# Deep Reinforcement Model Selection
for Communications Resource Allocation
in On-Site Medical Care ††thanks: This work was partly funded by the German
Ministry of Education and Research (BMBF) under grant 16KIS1028 (MOMENTUM).
This work was accepted for presentation at IEEE WCNC2022.

Steffen Gracla, Edgar Beck, Carsten Bockelmann and Armin Dekorsy  Dept. of
Communications Engineering, University of Bremen, Bremen, Germany
Email: {gracla, beck, bockelmann<EMAIL_ADDRESS>
###### Abstract

Greater capabilities of mobile communications technology enable the
interconnection of on-site medical care at a scale previously unavailable.
However, embedding such critical, demanding tasks into the already complex
infrastructure of mobile communications has proven challenging. This paper
explores a resource allocation scenario where a scheduler must balance mixed
performance metrics among connected users. To fulfill this resource allocation
task, we present a scheduler that adaptively switches between different model-
based scheduling algorithms. We make use of a deep Q-Network (DQN) to learn
the benefit of selecting a scheduling paradigm for a given situation,
combining advantages from model-driven and data-driven approaches. The
resulting ensemble scheduler is able to combine its constituent algorithms to
maximize a sum-utility cost function while ensuring performance on designated
high-priority users.

###### Index Terms:

Allocation, DQN, adaptive, model-selection, 5G, scheduling

##  I Introduction

The optimal allocation of communication resources is part of the perpetual
race for better performance in mobile communications. As new technologies such
as high resolution video streaming and vehicular communications emerge, the
demands to be fulfilled by a scheduler are becoming increasingly
heterogeneous, resulting in complex optimization tasks that must be solved in
real time.

In recent years, deep learning (DL) methods have distinguished themselves with
success in complex tasks, including image classification [1] and control [2],
among others. DL, as a subset of machine learning (ML), follows a different
paradigm compared to classic model-based approaches. Instead of designing an
optimal algorithm for a given goal, an algorithm is iteratively approximated
based on training data. Without the need to explicitly model the underlying
processes, the issue of modeling complexity is sidestepped. This
characteristic has sparked an interest in research that applies DL to the
highly performance-driven field of communication systems.

In particular, increasing the performance and capabilities of mobile
communication systems creates new prospects in emergency patient care. For
example, video, vitals, or specialist input can be exchanged wirelessly
between on-site medical professionals and hospital staff, yielding a
significant head start in time-sensitive treatments [3]. Nevertheless, a
deliberate approach is required due to the low tolerance for delays and
outliers in medical tasks.

Various forays have been made into integrating DL methods with resource
allocation tasks, mostly based on deep Q-Networks (DQN) [4, 5, 6] and actor-
critic methods [7]. However, model-based approaches have long thrived, thanks
to the valuable expert knowledge that shapes their design. Ideally,
incorporating this expert knowledge into data-based methods could govern and
stabilize the learning process and produce more tractable algorithms [8].

Therefore, we investigate a combined approach that offers some of the
advantages of both DL and model-based approaches. For the task of scheduling
discrete resources in vehicle-to-base-station communication, we implement a
group of simple, model-based scheduling algorithms. We then train a DQN to
select the best model-based algorithm in a given situation, optimizing the
long term effect on typical performance indicators: time-outs, sum capacity,
and packet rate. Particular attention is paid to the performance of Emergency
Vehicles (EV) as a priority class of users. In this way, we are able to use
data-driven DL to find an approximately optimal solution to a complex
scheduling problem while still using explicitly modeled algorithms.

##  II Setup & Notation

In the following, we introduce our system model. We describe the communication
channel between a user and the base station, the resources available for
allocation, and the generation of jobs. We then formulate the resource
allocation problem. After these technical specifications, we move on to the
DQN scheduler design.

###  II-A Resource Allocation System Model

In our system model, a number of $N$ user vehicles are connected to a base
station, moving on a 2D plane in a grid akin to the Manhattan-model of
movement [9]. At any discrete time instance $t$, vehicles $n$ will take a
fixed-size step in a direction of movement that is selected randomly. In doing
so, the vehicles have a $98\text{\,}\mathrm{\char 37\relax}$ chance of re-
selecting their prior direction of movement, $0\text{\,}\mathrm{\char
37\relax}$ chance of selecting the direction opposite to their prior movement,
and uniform chance of selecting a $None$ turn left, right, or stopping.

The communication channel between the base station and a user vehicle $n$ is
characterized by its power gain

$\displaystyle h_{n}[t]=|\tilde{h}_{n}[t]|^{$2$}\cdot\text{{\small
PL}}_{n}[t].$ (1)

The power gain is known to the base station. For each simulation time step
$t$, fading channel amplitudes $|\tilde{h}_{n}[t]|$ are randomly selected from
a Rayleigh distribution and multiplied with a distance-proportional path loss
factor

$\displaystyle\text{{\small
PL}}_{n}[t]=\min\left(1,\,\left(d_{n}[t]\right)^{$-1$}\right),$ (2)

where $d_{n}[t]$ is a vehicle’s distance from the base station. The path loss
factor ensures a degree of correlation of the power gain $h_{n}[t]$ over time.
While exponents of $-2$ or lower are more typically assumed in modeling real-
life path loss, we selected an exponent of $-1$ to reduce the spread of the
path loss values encountered. For the purpose of this paper, this lessens the
computation required in the subsequent learning task.

The base station has access to a limited number $U$ of discrete resource
blocks available for allocation, as shown in Fig. 1. These resources can be
filled with jobs $j$ from the job queue. At each time step $t$, new jobs are
generated at a probability $p_{j}$ per user. A job $j$ is assigned to a
specific user $n$ and is defined by two attributes, a request size
$u_{j,\,\text{req}}[t]$ in discrete resource blocks, and a time-to-timeout
$v_{j}[t]$ in discrete simulation steps. We define the set $\mathbb{J}[t]$ of
all jobs in the job queue in time step $t$ and the subset $\mathbb{J}_{n}[t]$
of the jobs in this queue assigned to user $n$ in $t$.

The job attributes’ initial values are governed by a user profile that
declares a maximum job size $u_{n,\,\text{max}}$ and initial time-to-timeout
$v_{n,\,\text{init}}$ for users $n$ of that profile. Upon generation, the job
is assigned a size $u_{j,\,\text{req}}[t]$ selected from a discrete uniform
distribution,
${u_{j,\,\text{req}}[t]\sim\mathbb{U}\\{1,\,u_{n,\,\text{max}}\\}}$. When a
number of the base station’s resources $U$ is allocated to a job $j$, that
job’s remaining size is decreased accordingly, while a lifetime count
$u_{n,\,\text{sx}}[t]$ of resources scheduled to user $n$ is increased. The
jobs’ initial time-to-timeout ${v_{j}[t]\leftarrow v_{n,\,\text{init}}}$ is
decremented for each time step $t$ that passes without the job being fully
scheduled. Once the time-to-timeout $v_{j}[t]$ reaches zero, the job is
discarded from the job queue and added to a set $\mathbb{J}_{\text{fail}}[t]$
for that time step $t$, for use in performance metric calculation.

$U$Time $t$Job QueueUser 1User 2User 3$u_{j,\,\text{req}}=$6$$ Figure 1: Jobs
consisting of a number of discrete resource blocks arrive in the job queue. A
scheduler is tasked with assigning them to a limited number $U$ of resources.

###  II-B Problem Statement

A scheduler is tasked with assigning the limited number $U$ of resource blocks
to the jobs in queue. Three metrics are selected to gauge the performance of a
scheduling algorithm:

  1. 1.

$r_{L}[t]$: Resource blocks discarded from timeout

  2. 2.

$r_{P}[t]$: Global packet rate achieved

  3. 3.

$r_{C}[t]$: Global channel capacity achieved

Firstly, the global _sum of resource blocks discarded_ due to timing out

$\displaystyle
r_{L}[t]=\sum_{j\in\mathbb{J}_{\text{fail}}[t]}u_{j,\,\text{req}}[t]$ (3)

should be minimized.

Secondly, a packet rate is introduced as the lifetime ratio of resources
requested and scheduled for a user $n$. We define the set
$\mathbb{J}_{n,\,\text{new}}[t]$ as the set of new jobs assigned to user $n$,
generated in time step $t$. Using the lifetime sum of discrete resources
$u_{n,\,\text{sx}}[t]$ scheduled to the jobs of user $n$, we calculate the
_sum packet rate_ over all users:

$\displaystyle
r_{P}[t]=\sum_{n=$1$}^{N}\frac{u_{n,\,\text{sx}}[t]}{\sum_{\tilde{t}=$1$}^{t}\sum_{j\in\mathbb{J}_{n,\,\text{new}}\left[\tilde{t}\right]}u_{j,\,\text{req}}\left[\tilde{t}\right]}=\sum_{n=$1$}^{N}k_{n}[t].$
(4)

A strong sum packet rate performance is achieved by a scheduler that does not
neglect any single user.

Last, the _sum rate capacity_ achieved by each transmission is calculated
using the Signal-to-Noise ratio (SNR) resulting from the selected channels
instantaneous fading characteristics as

$\displaystyle
r_{C}[t]=\sum_{n=1}^{N}\log\left(1+h_{n}[t]\frac{P}{\sigma_{\text{noise}}^{2}}\right)=\sum_{n=1}^{N}\log\left(1+\text{{\small
SNR}}_{n}[t]\right)$ (5)

for a Gaussian input alphabet, where signal power $P$ and expected noise power
$\sigma_{\text{noise}}^{2}$ are fixed for all vehicles $n$.

The scheduler is tasked with balancing all performance metrics, thus, we
collect all target metrics in a weighted sum utility

$\displaystyle\tilde{r}[t]=w_{C}r_{C}[t]+w_{P}r_{P}[t]-w_{L}r_{L}[t],$ (6)

with respective tunable weights $w_{C},w_{P},w_{L}$. Additionally, we
designate a priority class of EV-type users. Their significance is
communicated to the optimization process by adding EV timeouts
$r_{L,\,\text{EV}}[t]$ to the weighted sum utility with their own tunable
weight $w_{L,\,\text{EV}}$:

$\displaystyle r[t]$
$\displaystyle=\tilde{r}[t]-w_{L,\,\text{EV}}r_{L,\,\text{EV}}[t]$
$\displaystyle=w_{C}r_{C}[t]+w_{P}r_{P}[t]-w_{L}r_{L}[t]-w_{L,\,\text{EV}}r_{L,\,\text{EV}}[t].$
(7)

##  III Algorithm Selection Approach

Where model-based approaches struggle to find an optimal solution, deep
Reinforcement Learning (RL) may be applied to learn a function that
approximates the optimal algorithm along the domain of reasonable input data.
Fig. 2 schematically depicts the desired learning process. At the same time,
some applications require boundary conditions to be met, which tend to be easy
to formulate in model-based algorithms but cannot be enforced directly in
standard RL approaches.

Scheduler ($\boldsymbol{\theta}$)System$A$$\mathbf{S}[t+1],r[t]$ptUpdate
$\boldsymbol{\theta}$ Figure 2: In the desired RL loop, the parametrized
scheduler interacts with the system by selecting an action $A$ and receiving a
resulting reward $r[t]$ and new system state $\mathbf{S}[t+1]$. Based on these
experiences, the scheduler updates its parametrization $\boldsymbol{\theta}$
to promote high-reward actions and demote low reward actions.

Our scheduler learns to adaptively switch between a selection of model-based
algorithms to tackle this problem. In order to select the best algorithm, the
scheduler makes use of a deep Q-Network (DQN) [10] to learn the long-term
benefit of selecting each operation mode in a given situation. As a result,
model-based design benefits, such as hard performance guarantees and human
interpretability, are provided by the pool of model-based algorithms.
Meanwhile, the superposition of algorithms allows for greater performance than
each individual algorithm on flexible goal metrics. As an additional benefit,
the individual model-based algorithms do not have to be sophisticated enough
to perform well in every circumstance, so long as the overall selection is
rich enough to serve any problem.

###  III-A Model-Based Algorithms

Four model-based scheduling algorithms are implemented for the DQN to select
from: [11]

(1) A Maximum Throughput (MT) scheduler that allocates as many resources as
requested by order of descending channel power gains.

(2) A Max-Min-Fair (MMF) scheduler looks to distribute the available resources
$U$ performantly but fairly among the number $N_{\text{req}}[t]$ of users that
have jobs assigned to them in the job queue in time step $t$. It allocates by
order of priority
${h_{n}[t]\,/\,\sum_{j\in\mathbb{J}_{n}[t]}u_{j,\,\text{req}}[t]}$, favoring
good channels and small requests, but allocates at most an equal share
$\left\lfloor U/N_{\text{req}}[t]\right\rfloor$.

(3) A Delay Sensitive (DS) scheduler assigns a channel priority

$\displaystyle{p}_{c,n}[t]=\frac{k_{n}[t]}{\sum_{q=1}^{N}k_{q}[t]}\frac{h_{n}[t]}{\sum_{q=1}^{N}h_{q}[t]}$

given each users relative channel power gain $h_{n}[t]$ and packet rate
$k_{n}[t]$. The DS scheduler further draws on the users sum timeouts
${m_{n}[t]=\sum_{\tilde{t}=$1$}^{t}r_{L,n}\left[\tilde{t}\right]}$ and lowest
remaining time ${l_{n}[t]=\min_{j\in\mathbb{J}_{n}[t]}v_{j}[t]}$ for a timeout
urgency

$\displaystyle{p}_{l,n}[t]=\frac{m_{n}[t]/l_{n}[t]}{\sum_{q=1}^{N}m_{q}[t]/l_{q}[t]}.$

Each user is allotted a share of the available resources according to the
normalized, weighted priority vector

$\displaystyle\mathbf{p}[t]=(w_{1}\mathbf{p}_{c}[t]+w_{2}\mathbf{p}_{l}[t])/(w_{1}+w_{2}).$

Uniquely, this scheduler skips jobs that are about to time out if the allotted
discrete resources are not sufficient to complete the job. In this case, the
resources are freed for the next highest priority.

(4) An EV Priority scheduler assigns as many resources as requested to any EVs
and distributes remaining resources one-by-one, randomly assigning them to
requesting users.

###  III-B Deep Algorithm Selection

Our deep learning scheduler consists of multiple elements: A pre- and post-
processor, a DQN, an $\operatorname*{arg\,max}$ module, a memory module, and a
learning algorithm that tunes the DQN. In this section, we will introduce
these modules and illustrate their interconnection.

First, to be able to make informed decisions, a small pre-processor prepares
information about the current state of job queue and communication link as a
system state vector $\mathbf{S}[t]$. Per user $n$, the pre-processor
summarizes

  * •

current queue length
$S_{n,1}[t]=\sum_{j\in\mathbb{J}_{n}[t]}u_{j,\,\text{req}}[t]$,

  * •

channel power gains $S_{n,2}[t]=h_{n}[t]$,

  * •

average remaining time
$S_{n,3}[t]=\frac{1}{\left|\mathbb{J}_{n}[t]\right|}\sum_{j\in\mathbb{J}_{n}[t]}v_{j}[t]$,

  * •

minimum remaining time $S_{n,4}[t]=\min_{j\in\mathbb{J}_{n}[t]}v_{j}[t]$,

  * •

and past packet rate $S_{n,5}[t]=k_{n}[t]$,

for a length of $5N$ features for $N$ users.

For a given system state $\mathbf{S}[t]$ and choice of model-based algorithm
$A_{i}$, we define the long term expected rewards

$\displaystyle
Q(\mathbf{S}[t],A_{i})=\text{E}\left[\sum_{z=t}^{\infty}\lambda^{z-t}r[t]\;\middle|\;\mathbf{S}=\mathbf{S}[t],\,A=A_{i}\right],$
(8)

with reward $r$ as defined in (II-B). The long term rewards are discounted by
an exponential factor ${$0$\leq\lambda\leq$1$}$. As depicted in Fig. 3, a DQN
is set up to output an estimate $\hat{Q}$ of the long term expected rewards
$Q$ for each choice of model-based algorithm $A_{i}$. Given a perfect
approximation ${\hat{Q}=Q}$, we maximize the long term expected rewards by
simply selecting whichever action
$\operatorname*{arg\,max}_{i}\hat{Q}(\mathbf{S}[t],\,A_{i})$ has the highest
_estimated_ long term rewards. Therefore, the learning goal is to update the
DQN parameters $\boldsymbol{\theta}$ such that the estimate is approximately
close to the true long term reward $Q$ for all possible
${(\mathbf{S},A_{i})}$, relying on the universal approximation property of
neural networks [10].

$\mathbf{S}[t]$ DQN ($\boldsymbol{\theta}$)
$\hat{Q}(\mathbf{S}[t],A_{2})$$\hat{Q}(\mathbf{S}[t],A_{1})$$\hat{Q}(\mathbf{S}[t],A_{3})$$\displaystyle\operatorname*{arg\,max}_{A}$
Figure 3: The DQN module estimates expected long term rewards
${\hat{Q}(\mathbf{S}[t],A_{i})}$ for a given state $\mathbf{S}[t]$ and all
available actions $A_{i}$, according to its current parametrization
$\boldsymbol{\theta}$. In this case, actions $A_{i}$ are the available model-
based algorithms. The scheduler then selects the algorithm $A_{i}$ with the
highest expected long term reward.

First, the scheduler must make experiences to learn from. Using an
$\epsilon$-greedy exploration scheme, the scheduler initially explores the
simulation by taking random actions, i.e., selecting a random model-based
algorithm for a given state. States, decisions, and their direct outcome are
recorded and stored in a replay buffer in the form of tuples

$\displaystyle\text{{\small
EXP}}=\left(\mathbf{S}[t],\,A[t],\,r[t],\,\mathbf{S}[t+1]\right).$ (9)

We highlight that an experience only contains the immediate reward $r[t]$, not
the desired long-term rewards ${Q(\mathbf{S}[t],A[t])}$, i.e., (8). However,
we can extract additional information from $\mathbf{S}[t+1]$, the state that
following the action. By again using our DQN, we estimate the rewards
following $\mathbf{S}[t+1]$ to construct a learning target

$\displaystyle\hat{Q}_{\text{target}}(\text{{\small
EXP}})=r[t]+\lambda\max_{i}\hat{Q}(\mathbf{S}[t+1],A_{i}),$ (10)

that incorporates the information of $r[t]$ and $\mathbf{S}[t+$1$]$ from the
experience. Using this target ${\hat{Q}_{\text{target}}(\text{{\small
EXP}})}$, an estimation error [10]

$\displaystyle\delta=\hat{Q}_{\text{target}}\left(\text{{\small
EXP}}\right)-\hat{Q}\left(\text{{\small EXP}}\right)$ (11)

given the networks current parametrization can be calculated for any
experience tuple from the buffer.

Following the principle of stochastic gradient descent (SGD), the networks
parameters $\boldsymbol{\theta}$ are then adjusted by sampling a mini batch of
$B$ experiences from the buffer to minimize a mean square error cost

$\displaystyle C=\frac{1}{B}\sum_{b=1}^{B}(\delta_{b})^{2}$ (12)

for the batch. Minibatch parameter updates are carried out every time a new
experience is made.

As training progresses and the schedulers understanding of the simulation
environment improves, the probability $\epsilon$ of selecting random
exploration actions is gradually decreased, relying more and more on the
network to make decisions. When prompted, at the beginning of a time step $t$,
the network will estimate the expected long term rewards $\hat{Q}$ for
selecting any of the available model-based algorithms $A_{i}$, given the
current state $\mathbf{S}[t]$ and the networks current parameters
$\boldsymbol{\theta}$. The scheduler then selects the model-based algorithm
$A_{i}$ with the highest expected long term rewards.

To increase training efficiency, we implement optimizations to the base DQN
learning method:

  * •

An infrequently updated network copy is used for the bootstrapping in (10) to
increase training stability (_Target Network_ , [2])

  * •

Sampling of experiences from the buffer is weighted proportional to the
experiences’ estimation error magnitude (_Prioritized Replay_ , [12])

  * •

The neural network structure is altered to seperately learn the contribution
of state and action to the reward estimate (_DuelingDQN_ , [12])

##  IV Performance Evaluation

###  IV-A Implementation Details

We configure the simulation with ${U=$16$}$ available resources and ${N=$10$}$
users. User profiles are set up according to Table I, with five ’Normal’, two
’High Datarate’, two ’Low Latency’ users, and one ’Emergency Vehicle’. For the
given configuration, a job creation probability
${p_{j}=$20\text{\,}\mathrm{\char 37\relax}$}$ for each simulation step and
user puts a high expected load of $1.6$ requests per available resource on
average on the schedulers. We run a total of $10\,000$ episodes at $50$ time
steps $t$ per episode for training and evaluation each, with the mini batch
size set to ${B=64}$ sampled experiences per training step. Parameter
optimization via SGD is carried out by the Adam optimizer [13] with default
settings and a learning rate of $1\text{\times}{10}^{-4}$. Reward weightings
are set to ${w_{C}=w_{P}=$0.25$}$, ${w_{L}=w_{L,\,\text{EV}}=$1.0$}$, giving
roughly equal significance to each goal metric with the average expected
magnitude of the respective rewards.

TABLE I: User Profiles | Delay |  Max Job Size $u_{\text{\text{max}}}$
---|---|---
| in sim. steps | in res. blocks
Normal | $20$ | $30$
High Packet Rate | $20$ | $40$
Low Latency | $2$ | $8$
Emergency Vehicle | $1$ | $16$

For the algorithm selection DQN, a five layer feed-forward network is
selected. Layers have $300$ nodes each, except for the last layer that was
split in two branches with $200$ nodes each according to the DuelingDQN
structure [12]. Exploration is done by selecting random actions at an initial
chance of ${\epsilon=$99\text{\,}\mathrm{\char 37\relax}$}$, decaying linearly
to $0\text{\,}\mathrm{\char 37\relax}$ after $80\text{\,}\mathrm{\char
37\relax}$ of training episodes. The exponential future reward decay factor is
set to ${\lambda=$0.9$}$ in order to put significance on only a low number of
future steps. User TX-SNR ${P/\sigma_{\text{noise}}^{2}}$ is fixed to
$13\text{\,}\mathrm{dB}$.

We implement the simulation in python using the tensorflow library primarily.
The full implementation is made available in [14].

###  IV-B Results

As the simulation model contains stochastic components, the results achievable
on each metric have an inherent variance depending on the specific
realizations. For example, a spike in generated jobs will result in timeouts
irrespective of the scheduling method. For this reason, results achieved
during testing are displayed in cumulative histograms.

$\displaystyle{0.2}$$\displaystyle{0.4}$$\displaystyle{0.6}$$\displaystyle{0.8}$$\displaystyle{1.0}$Norm.
Achieved Capacities0%25%50%75%100%Cumulative %betterDQNMTMMFDS (a)

$\displaystyle{0.3}$$\displaystyle{0.4}$$\displaystyle{0.5}$$\displaystyle{0.6}$$\displaystyle{0.7}$$\displaystyle{0.8}$$\displaystyle{0.9}$$\displaystyle{1.0}$Norm.
Packet Rate Satisfaction0%25%50%75%100%Cumulative %better (b)

$\displaystyle{0.0}$$\displaystyle{0.2}$$\displaystyle{0.4}$$\displaystyle{0.6}$$\displaystyle{0.8}$$\displaystyle{1.0}$Norm.
Latency Violations0%25%50%75%100%Cumulative %better (c)

$\displaystyle{0.0}$$\displaystyle{0.2}$$\displaystyle{0.4}$$\displaystyle{0.6}$$\displaystyle{0.8}$Norm.
EV Timeouts0%25%50%75%100%Cumulative %better (d)

Figure 4: Cumulative histograms of the schedulers’ performance on the
individual parts (a) $r_{C}$, (b) $r_{P}$, (c) $r_{L}$ and (d)
$r_{L,\,\text{EV}}$ of the overall optimization goal $r$ (compare (II-B)).
Dominating all submetrics at once is impossible as the individual submetrics
have conflicting objectives, e.g., scheduling an urgent job while the channel
is weak. Maximum Throughput (MT), Max-Min-Fair (MMF) and Delay Sensitive (DS)
schedulers show strengths in some submetrics but weaknesses in others. The
presented DQN scheduler achieves its overall optimization goal by balancing
strong performance in all submetrics.

Fig. 5 shows each schedulers performance on the combined reward metric $r$. On
this metric, the DQN adaptive scheduler is able to find a strategy that
outperforms any single model-based algorithm. Breaking the sum metric $r$ down
into its constituent parts sheds light on how this is achieved. As shown in
Fig. 4, the DQN scheduler balances the submetrics against each other, yielding
some performance on each of them compared to the benchmark.

$\displaystyle{0.0}$$\displaystyle{0.2}$$\displaystyle{0.4}$$\displaystyle{0.6}$$\displaystyle{0.8}$$\displaystyle{1.0}$Normalized
Episode Sum Rewards0%25%50%75%100%Cumulative %betterDQNMTMMFDS Figure 5:
Cumulative histogram of performance of Maximum Throughput (MT), Max-Min-Fair
(MMF) and Delay Sensitive (DS) schedulers as well as DQN adaptive scheduler on
the weighted sum reward metric. For each episode, all achieved rewards $r$ are
summed. Achieved reward sums are normalized by the highest achieved reward
sum.

Of particular interest are the EV-specific timeouts in Fig. 4d. This metric is
not specifically targeted by any of the model-based algorithms depicted. A
modest double-weighting of EV specific timeouts within the goal metric $r$,
combined with the introduction of the otherwise sub-optimal EV Priority
scheduling algorithm to the selection pool, has enabled the DQN based
scheduler to significantly suppress timeouts in Emergency Vehicles even
compared to the otherwise timeout focused Delay Sensitive algorithm without
hurting overall performance overmuch. As Fig. 6 shows, the EV Priority
algorithm was only selected a comparably low amount of times to achieve this
goal.

$\displaystyle{0.0}$$\displaystyle{0.1}$$\displaystyle{0.2}$$\displaystyle{0.3}$$\displaystyle{0.4}$MTMMFDSEV
P Figure 6: Relative selection rate of EV Priority (EV P), Delay Sensitive
(DS), Max-Min-Fair (MMF) and Maximum Throughput (MT) schedulers during
testing.

The DQNs learned behavior can also be monitored to reveal underlying features
of the simulation. As Fig. 6 highlights, for the given reward weighting, the
MMF algorithm was only selected a low number of times. Further investigation
could reveal whether these are, for example, high impact outlier cases that
could be better served with an additional model-based algorithm that is
specifically targeted to them, or whether the MMF algorithm is not fit for the
given scenario.

The DQN decision making does however add another layer of computation to the
scheduling process. Further, while the ensemble method can relax the
sophistication required from each model-based part of the ensemble, the
composite scheduling function can only assume the function space spanned by
the group of model-based algorithms. In other words, the DQN adaptive method
is unable to discover strategies that go beyond combining the available
models. Determining whether the selection of model-based algorithms provided
is rich enough to serve the problem therefore remains a burden on the
designer.

##  V Conclusion

In this paper we presented a RL-based communications resource scheduler that
constructs an effective scheduling paradigm from an ensemble of model-based
algorithms. We achieved this by learning to switch to whichever algorithm
promises the highest expected long term benefit based on the current queue
state. This approach combines the flexible goal optimization of RL methods
with the rigid predictability of model-based algorithms. It is noteworthy for
applications with complex, conflicting performance goals, where either a)
strong models exist to cover parts of the problem, or b) explicit modeling is
otherwise necessary, e.g., due to very low tolerance for outliers, such as the
transmission of critical medical data. Using this approach unlocks the
benefits of RL without abandoning explicit modeling. For the simulation
presented, the adaptive model-switching scheduler was able to learn to
outperform single, model-based algorithms on a weighted sum utility metric.

## References

  * [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Proc. NeurIPS_ , vol. 25, pp. 1097–1105, 2012.
  * [2] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [3] M. Zanatta, P. Benato, and V. Cianci, “Pre-hospital ultrasound: current indications and future perspectives,” _Int. J. Crit. Care. Emerg. Med._ , vol. 2, no. 2, p. 019, 2016.
  * [4] S. Joseph, R. Misra, and S. Katti, “Towards self-driving radios: Physical-layer control using deep reinforcement learning,” in _Proc. ACM HotMobile_ , 2019, pp. 69–74.
  * [5] H. Ye, G. Y. Li, and B.-H. F. Juang, “Deep reinforcement learning based resource allocation for V2V communications,” _IEEE Trans. Veh. Technol._ , vol. 68, no. 4, pp. 3163–3173, 2019.
  * [6] F. Al-Tam, N. Correia, and J. Rodriguez, “Learn to Schedule (LEASCH): A Deep reinforcement learning approach for radio resource scheduling in the 5G MAC layer,” _IEEE Access_ , vol. 8, pp. 108 088–108 101, 2020.
  * [7] Y. Huang, S. Li, C. Li, Y. T. Hou, and W. Lou, “A deep-reinforcement-learning-based approach to dynamic eMBB/URLLC multiplexing in 5G NR,” _IEEE Internet Things J._ , vol. 7, no. 7, pp. 6439–6456, 2020\.
  * [8] N. Shlezinger, J. Whang, Y. C. Eldar, and A. G. Dimakis, “Model-Based Deep Learning,” _arXiv:2012.08405_ , 2020.
  * [9] N. Aschenbruck, E. Gerhards-Padilla, and P. Martini, “A survey on mobility models for performance analysis in tactical mobile networks,” _J. Telecommun. Inf. Technol._ , pp. 54–61, 2008.
  * [10] R. Sutton and A. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [11] S. O. Schmidt, “Analyse von Latenz-empfindlichem Scheduling auf Systemebene für Anwendungen der Industrie 4.0,” Master’s thesis, Universität Bremen, Bremen, Germany, 2018.
  * [12] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining improvements in deep reinforcement learning,” in _Proc. AAAI_ , vol. 32, no. 1, 2018.
  * [13] D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” _arXiv:1412.6980_ , 2015.
  * [14] S. Gracla, “Adaptive Scheduling Model Selection,” https://github.com/Steffengra/resourceallocation/tree/master/scheduling, 2020\.


# Quantum holography with undetected light

Sebastian Töpfer1,† , Marta Gilaberte Basset1,2,†, Jorge Fuenzalida1, Fabian
Steinlechner1,2, Juan P. Torres3, and Markus Gräfe1,2,∗
1Fraunhofer Institute for Applied Optics and Precision Engineering IOF,
Albert-Einstein-Str. 7, 07745, Jena, Germany.
2Friedrich-Schiller-University Jena, Institue of Applied Physics, Abbe Center
of Photonics,
Albert-Einstein-Str. 6, 07745, Jena, Germany.
3ICFO-Institut de Ciencies Fotoniques, The Barcelona Institute of Science and
Technology,
08860 Castelldefels, Spain, and Dept. Signal Theory and Communications,
Universitat Politecnica de Catalunya, 08034 Barcelona, Spain.
∗corresponding author. Email<EMAIL_ADDRESS>
†Both authors contributed equally.

(August 27, 2024)

## Abstract

Holography exploits the interference of a light field reflected/transmitted
from an object with a reference beam to obtain a reconstruction of the spatial
shape of the object. Classical holography techniques have been very successful
in diverse areas such as microscopy, manufacturing technology, and basic
science. However, detection constraints for wavelengths outside the visible
range restrict the applications of holography techniques for imaging and
sensing in general. For overcoming such detection limitations, we implement
phase shifting holography with nonclassical states of light, where we exploit
quantum interference between two-photon probability amplitudes in a nonlinear
interferometer. We demonstrate that it allows retrieving the spatial shape
(amplitude and phase) of the photons transmitted/reflected from the object,
and thus obtaining an image of the object, despite those photons are never
detected. Moreover, there is no need to use a well-characterized reference
beam, since the two-photon scheme already makes use of one of the photons as
reference for holography.

## Teaser

Experiments show the performance of quantum holography using undetected
photons.

## Introduction

In the last few decades, scientists and engineers throughout the world, from
different disciplines, governments and information technology companies are
paying increasing attention to quantum technologies. Quantum communications
Xu:2020 ; Wengerowsky:20 ; ortega:2021 , quantum computation zhong:2020 , and
in particular quantum imaging brida:2010 ; Lemos:2014 ; Kalashnikov:2016 ;
Gilaberte:2019 ; moreau:2019 , are just some examples of novel areas of
science and technology where quantum ideas are helping to implement systems
with enabling new capabilities. Quantum technologies promise to go further
than classical counterpart technologies by using new quantum states of light
and matter, performing tasks that are impossible to implement classically
defienne:2021 . A clear example of this is the ability of obtaining a hologram
from single photons Chrapkiewicz:2016 , and even recording the hologram
without detecting the photons themselves as we report here.

Holography was introduced by D. Gabor in 1948 gabor:1948 . It allows the
reconstruction of the spatial structure of an object by recording amplitude
and phase information of the light reflected from an object. Classical
holography is already successfully applied in sensing and microscopy. This is
true especially for bio-specimen, where scattering and absorption require a
phase-sensitive sensing contrast agent-free approach Popescu:2011 ; Marquet:05
. Holography can also be applied in optical security Refregier:95 ; LIU2014327
and data storage https://doi.org/10.1002/anie.201002085 ; 10.1117/12.506055 .
The introduction of single photon holographic methods would expand holography
to several applications that raised together with the recent growth of quantum
technologies Chrapkiewicz:2016 . In particular, single photon holography with
undetected light in a nonlinear interferometer scheme Yurke:1986 ;
Mandel:1991_2 ; herzog1994frustrated ; Chekhova:16  would introduce the
benefit of choosing the most convenient spectral range for the probing beam in
an application without facing the limitations imposed by the low efficiency of
detectors at those specific spectral range (e.g. in the mid-infrared).
Nonlinear interferometry has been proven to be key elements in numerous
applications, namely, in imaging Lemos:2014 ; Cardoso:2018 ; Gilaberte:2021 ,
sensing Kutas:2020 , spectroscopy Kalashnikov:2016 ; Paterova:2018 ,
microscopy Kviatkovsky:2020 ; Paterova:2020 , and optical coherence tomography
Valles:2018 ; paterovaOCT2018 ; ramelowOCT2020 ; gerardOCT2020 .

To characterize the spatial shape of the single-photon, we use phase shifting
holography, introduced in 1997 by Yamaguichi & Zhang Yamaguchi:1997 , where
several images with different phase steps are recorded and processed. This
technique allows the reconstruction of the spatial structure (amplitude and
phase) of the light reflected from a sample. In combination with the use of
the effect of induced coherence Mandel:1991_2 ; herzog1994frustrated  we can
obtain an hologram without detecting the light that illuminated the sample.
Our experiment is a step forward to enable efficient holography in a broader
spectral range.

## Results

### Phase-shifting holography with a SU(1,1) nonlinear interferometer

In this part we give a brief overview of the theoretical framework of phase
shifting holography in the quantum regime considered here. An extended and
more detailed theoretical description of the experiment can be found in the
Supplemental Material.

In classical holography (see Fig. 1(a)) the hologram is the result of the
interference of two input beams with mutual coherence goodman_fourier . One
beam serves as reference beam. The other beam (object beam) illuminates an
object and generates reflected/transmitted light that bears information of the
spatial structure of the object. The spatial shape of the interference
pattern, resulting from combining the object and reference beams

$I(x,y)=I_{\mathrm{r}}(x,y)+I_{\mathrm{o}}(x,y)+2\sqrt{I_{\mathrm{r}}(x,y)I_{\mathrm{o}}(x,y)}\cos\left[\theta_{\mathrm{o}}(x,y)-\theta_{\mathrm{r}}(x,y)\right]$
(1)

is recorded in the hologram, and can be visualized with the help of a CCD
camera goodman_fourier . Here $I_{\mathrm{r}}$ and $\theta_{\mathrm{r}}(x,y)$
are the intensity and phase of the reference beam, and $I_{\mathrm{o}}$ and
$\theta_{\mathrm{o}}(x,y)$ are the intensity and phase of the object beam
after reflection/transmission by the object.

For the sake of clarity, let us consider quantum holography with undetected
light in an induced coherence scenario (see Fig. 1(b)), where reference and
object beams are generated by means of spontaneous parametric down-conversion
(SPDC). The first important difference with classical holography is that the
object beam, after reflection/transmission from the object, is not made to
interfere with the reference beam. The object beam remains _undetected_. The
second important difference for quantum holography is that the reference and
object beam are incoherent beams. This can be seen from the fact that object
and reference beam in the quantum holography scheme in principle have
different spectra, which is indicated by the different colors used in Fig.
1(b). If the transmission function of the object to be recorded is
$\tau(x,y)=t(x,y)\mathrm{e}^{\mathrm{i}\theta(x,y)}$ ($|\tau|=t$), the
spatially-dependent degree of first-order coherence $g^{(1)}(x,y)$ between the
two beams is $g^{(1)}(x,y)=t(x,y)$ (for further details see the information in
the Supplementary Material).

Figure 1: Classical and quantum holography.  a) In classical holography the
spatially-dependent interference pattern of two coherent beams, the reference
beam and the object beam, after interaction with the object, are recorded and
used to construct the hologram of the object. b) In our quantum holography
scheme we make use of two-photon states that can be generated by spontaneous
parametric down conversion (SPDC) in one of two sources. The spatial shape of
the object, which is transferred to the spatial shape of the light
reflected/transmitted from the object, is contained in the probability
amplitudes corresponding to the paired photons being generated in either of
two SPDC sources.

In our work, we implement the two beams for the quantum holography scheme with
undetected light by a photon pair source based on spontaneous parametric down
conversion (SPDC) in nonlinear crystals. There, correlated signal and idler
beams are generated. Arranging such two SPDC sources in a nonlinear SU(1,1)
interferometer, signal and idler pairs are generated in either of the
nonlinear crystals. Our experiments work in the low parametric gain regime,
where the probability to generate signal-idler pairs in both nonlinear
crystals simultaneously is negligible. In this regime, one can think that the
hologram is the result of quantum interference between two possibilities
characterized by the corresponding probability amplitudes. One possibility is
that signal-idler pairs are generated in SPDC, the idler photons interact with
the object and impinges on the second nonlinear crystal without inducing the
emission of new paired photons. The second possibility is that signal-idler
pairs are generated in the second nonlinear crystal, so no idler photons have
traversed the object. The transmission function of the object determines the
distinguishability of both possibilities and thus the degree of coherence of
the two beams that interference in the hologram.

For quantum holography with undetected light one finds (see Supplementary
material for a detailed derivation) that the spatially-dependent flux-rate of
signal photons detected $N_{\Delta\varphi}(x,y)$ when an object with
transmission function $t(x,y)\mathrm{e}^{\mathrm{i}\theta(x,y)}$ is present in
the idler path is Gilaberte:2021

$N_{\Delta\varphi}(x,y)\sim
1+t(x,y)\cos\left[\theta(x,y)-\nu(x,y)+\Delta\varphi\right],$ (2)

where $\Delta\varphi$ is a global phases and $\nu(x,y)$ is a phase introduced
by the nonlinear parametric down-conversion processes inside the nonlinear
interferometer. This equation is slightly different from Eq. (1) that
describes the interference pattern registered in classical holography.
Nevertheless, both equations show that the amplitude and phase of light
reflected/transmitted by an object can be registered in a medium (hologram)
sensitive only to the intensity of light.

We aim at visualizing the interference pattern to be recorded in the hologram.
For an unknown object a single measurement does not allow to extract full
phase ($\theta(x,y)$) and amplitude ($t(x,y)$) information of the complex
transmission coefficient introduced by the object. However, phase shifting
holography can be applied in order to extract such information, both amplitude
and phase. In doing so, a series of images with global phases
$\Delta\varphi=0,\pi/2,\pi,3\pi/2$ can be recorded and processed. This will
result in four images:

$\displaystyle N_{0}(x,y)$ $\displaystyle\sim$ $\displaystyle
1+t(x,y)\cos\theta(x,y)$ $\displaystyle N_{\pi/2}(x,y)$ $\displaystyle\sim$
$\displaystyle 1-t(x,y)\sin\theta(x,y)$ $\displaystyle N_{\pi}(x,y)$
$\displaystyle\sim$ $\displaystyle 1-t(x,y)\cos\theta(x,y)$ $\displaystyle
N_{3\pi/2}(x,y)$ $\displaystyle\sim$ $\displaystyle 1+t(x,y)\sin\theta(x,y)$
(3)

From these four measurements one can easily extract the phase information

$\theta(x,y)=\arctan\left(\frac{N_{3\pi/2}-N_{\pi/2}}{N_{0}-N_{\pi}}\right),$
(4)

and the amplitude information

$t(x,y)=2\times\frac{\left\\{\left[N_{3\pi/2}-N_{\pi/2}\right]^{2}+\left[N_{0}-N_{\pi}\right]^{2}\right\\}^{1/2}}{N_{0}+N_{\pi/2}+N_{\pi}+N_{3\pi/2}}.$
(5)

Recording four images of the spatially-varying signal photon flux rate at a
certain spectral range with a high efficiency detector allows to retrieve full
phase and amplitude information of an unknown object that is probed by photons
in a different spectral range. This way quantum holography with undetected
light becomes possible.

One can generalize this approach to a series of $M\geq 3$ images malacara:2007
. They need to be recorded with phases $\Delta\varphi_{m}=2m\pi/M$ with
$m=0\ldots M-1$. Then, phase and amplitude information are given by

$\theta(x,y)=-\arctan\left(\frac{\sum_{m}N_{\Delta\varphi_{m}}(x,y)\sin\Delta\varphi_{m}}{\sum_{m}N_{\Delta\varphi_{m}}(x,y)\cos\Delta\varphi_{m}}\right)\\\
$ (6)

and

$t(x,y)=2\times\frac{\left\\{\left[\sum_{m}N_{\Delta\varphi_{m}}(x,y)\cos\Delta\varphi_{m}\right]^{2}+\left[\sum_{m}N_{\Delta\varphi_{m}}(x,y)\sin\Delta\varphi_{m}\right]^{2}\right\\}^{1/2}}{\sum_{m}N_{\Delta\varphi_{m}}(x,y)}.$
(7)

One would expect that the consideration of a higher number of steps $M$ will
lead to a more accurate and precise reconstruction of the phase and amplitude
as it is the case in classical phase shifting holography. Consequently, we
have experimentally implemented quantum phase shifting holography with
undetected light for different step numbers. We have analyzed their impact on
accuracy as well as the overall performance of each approach.

### Experimental setup

Most nonlinear interferometers implemented so far for quantum imaging and
spectroscopy make use of one of two configurations. In one configuration the
signal photons generated in the first nonlinear crystal are detected and they
never traverse the second nonlinear crystal Mandel:1991_2 ; Lemos:2014 ;
Valles:2018 . Only idler photons generated in the first nonlinear crystal,
after being reflected/transmitted by the object impinge on the second
nonlinear crystal. This is the original configuration put forward in 1991 by
L. Mandel’s group Mandel:1991 ; Mandel:1991_2 .

In an alternative configuration, usually termed as SU(1,1) interferometer, the
signal photons generated in the first nonlinear crystal are also injected in
the second nonlinear crystal as the idler photons Yurke:1986 ;
herzog1994frustrated ; Paterova:2018 ; Gilaberte:2021 ; gerardOCT2020 .
Quantum interference explains the physical origin of the interference pattern
for both configurations, and the mathematical expressions that describe the
shape of this interference pattern are mostly equal. However, in the original
configuration considered we have three beams at the output (two signal beams
and one idler beam), while in the SU(1,1) configuration the output consists of
two beams (signal and idler beams) gerardOCT2020 .

Figure 2: Experimental setup for holography with undetected light. Laser light
(purple) pumps the nonlinear crystal (ppKTP) bi-directionally (beam paths
$\mathrm{a}$ and $\mathrm{d}$). It generates signal (red) and idler (green)
beams either in the forward direction (beam paths $\mathrm{b,c}$) or backward
direction (beam paths $\mathrm{e,f}$). Dichroic mirrors DM1 to DM3 separate
the different beam paths. Idler light will illuminate the object (beam path
$\mathrm{c}$), while its hologram will be detected on the sCMOS camera with
the signal light (beam path $\mathrm{e}$). The mirrors M1 to M3 are the
interferometer end mirrors. M2 is mounted on a piezo stage to precisely move
the mirror in one direction. Lenses L1 to L5 form the imaging system with the
focal distances of 150 mm (L1, L2, L3), 100 mm (L4), and 125 mm (L5).

Our experimental implementation of quantum holography with undetected light
makes use of an SU(1,1) interferometer in a Michelson geometry as shown in
Fig. 2. It consists of a nonlinear periodically-poled potassium titanyl
phosphate (ppKTP) crystal of 2 mm length. The crystal is pumped bi-
directionally with a collimated 405 nm cw laser beam with up to $90$ mW of
pump power. Correlated signal and idler beams are emitted from the crystal via
SPDC with central wavelengths of 910 nm and 730 nm, respectively. As shown in
Fig. 2, the crystal is imaged onto itself with a 4f-system in each of the
three interferometer arms via lenses L1 to L3. The object is placed in the
Fourier plane of the idler arm having the momentum space at the object
location. The interferometer end mirror M2 in the idler arm is mounted on a
piezo translation stage smarAct:2021  that allows to scan different phases of
the interference produced by varying the path length. This way, one can define
specific positions for the mirror M2 corresponding to different phase values
$\Delta\varphi$. There is quantum interference between the two possibilities
for the generation of signal-idler paired photons, after the pump, idler, and
signal beams return to the crystal. In this way the object’s information
imprinted on the idler light is transferred to a spatially-varying intensity
of the signal light. Lenses L4 and L5 are used to form an image of the object
at the camera plane. The camera photometrics:2021  has $2048\times 2048$ px
with a pixel size of 6.5 $\mu$m. The images shown in this work are $500\times
500$ px in size. The total usable field-of-view has a diameter of $\approx$
6.1 mm.

### Holography and Imaging performance

To experimentally test the quantum holography with undetected light approach
we used objects that were engraved in glass plates of refractive index 1.6 by
grayscale lithography. The height of the engraved area was designed to induce
a total phase change on the idler beam of either $0.62\pi$ or $0.82\pi$,
depending on the object. The dimensions of the objects and their features are
shown in Figure LABEL:fig:X_object_dimensions in the Supplemental Material.

Two of the objects are resolution targets that are a miniaturized version of
the standard 1951 USAF resolution target in order to make all structures fit
inside the field of view of the system (6.1 mm in diameter). One is with
$0.62\pi$ phase step, the other with $0.82\pi$. A full wide field holographic
image for the latter one can be seen in Fig. 3(a). The area marked with a red
rectangle is used to evaluate the phase step. As exemplarily shown in Fig.
3(b) the desired phase step of $0.82\pi$ is well matched.

Figure 3: Hologram with undetected light of a resolution target. (a) Wide
field holographic image (using 12 frames) of the miniaturized resolution
target with phase step of $0.82\pi$ (at the illumination wavelength). The
elements contained in the yellow marked areas are the ones analysed to
determine the resolution of the setup (see Tab. 1). The red rectangle
highlights the area used to verify the phase step of the object. (b) Phase
step plot of the object, which matches with the manufactured $0.82\pi$ value.

With both resolution objects, we investigated the impact of the number of
images $M$ recorded for the phase shifting holography, as well as the
acquisition time per shot on a resolution target. In doing so, we varied the
number of images from three to twelve and the acquisition time per image from
100 ms to 1 s. The results are shown in Fig. 4. A four by four image matrix
shows the region of interest assigned to the ’bar pattern 2’ (Fig. 3) under
different measurement settings. It clearly indicates that a higher number of
images for the phase shifting holography as well as a higher acquisition time
lead to a better image quality. A quantitative evaluation is displayed in Fig.
4(b) for both resolution targets. The results of the phase step retrieval show
that the calculated phase values match the expected values for exposure times
higher than 200 ms. An increase in the amount of images or exposure time
lowers the standard deviation of the results, due to a decrease of the
measured noise (see Fig. LABEL:fig:X_phase_noise in the Supplemental
Material). According to this results, the current setup should be operated
with 500 ms exposure time to ensure, that the measured values align with the
expected values within the standard deviation. It is possible to do so with
only $M=3$ images, but $M=4$ images lead to significant more accuracy. This
results in at least 2 s overall measurement time for one phase shifting
holography measurement, disregarding the time needed to change the phase
position and the post processing of the data. For this case, the maximum phase
noise was measured to be $(0.091\pm 0.005)\pi$, which is the lowest detectable
phase difference. The transmission of the object was measured to be
homogeneous with $t=(94\pm 1)\%$ as in agreement with the theory taking into
account the visibility with and without object Gilaberte:2021 .

Figure 4: Phase accuracy. (a) Images of the same structure for different
acquisition times of the camera and different number $M$ of images used for
the phase shifting holography. (b) Calculated phase steps. Each point is an
average of 15 image sets. The black dashed lines are the expected results. The
color is in relation to the amount of images used (see legend). The cross
markers refer to a sample with a step size of $0.62\pi$ and the bullet markers
refer to a sample with $0.82\pi$ step.

The blurring in the phase images shows, that the spatial resolution is visibly
limited. Using the yellow marked bar patterns on the resolution target (see
Fig. 3(a)) the contrast for different spatial frequencies can be calculated.
It must be considered that it is a miniaturized version of the standard USAF
target. Table 1 shows the sizes of the bar patterns and the measured contrast.
Using the Rayleigh criterion, the measured contrast must be at least 14.2% to
count as resolvable. Because the contrast is measured using bar patterns,
which have multiple frequency components instead of sine patterns, it must be
corrected by multiplying it with $\pi/4$ Nill:2001 . These corrected contrast
values are shown in the last two columns, named as sine wave contrast. The
lowest measured resolvable spatial frequency according to the Rayleigh
criteria is 6.3 mm-1, which corresponds to 79 $\mu$m feature size.

line pair | frequency | square wave contrast | sine wave contrast
---|---|---|---
in mm | in 1/mm | vertical | horizontal | vertical | horizontal
0.200 | 5.0 | 0.60 $\pm$ 0.06 | 0.62 $\pm$ 0.06 | 0.47 $\pm$ 0.06 | 0.48 $\pm$ 0.06
0.178 | 5.6 | 0.53 $\pm$ 0.07 | 0.45 $\pm$ 0.08 | 0.42 $\pm$ 0.06 | 0.35 $\pm$ 0.07
0.158 | 6.3 | 0.36 $\pm$ 0.09 | 0.28 $\pm$ 0.09 | 0.29 $\pm$ 0.08 | 0.22 $\pm$ 0.08
0.140 | 7.1 | 0.18 $\pm$ 0.10 | 0.12 $\pm$ 0.10 | 0.14 $\pm$ 0.09 | 0.09 $\pm$ 0.09
0.125 | 8.0 | 0.07 $\pm$ 0.10 | — | 0.06 $\pm$ 0.09 | —
Table 1: Spatial Resolution. Calculated Contrast using the bar patterns of the
miniaturized resolution target (yellow marked area in Fig. 3) for 1000 ms
acquisition time and $0.82\pi$ phase step. Square wave contrast is the
measured contrast on the bar pattern and sine wave contrast is the corrected
result, taken into account that the bar pattern contains multiple frequencies.
The rows in gray do not fulfill the Rayleigh criterion.

In addition, we analyzed the accuracy of detecting transmission values. The
transmission information can be obtained from the modulation of the
interference pattern. In the Methods section, it is explained how the
modulation can be obtained from the sampled images. While doing so, we
recorded phase shifting holograms with undetected light of an object in form
of an happy face with different (but homogeneous) transmission values
introduced by optical density (OD) filters. An example for the obtained
modulation image is shown in Fig. 5(a) and (b) for OD 0 and OD 0.4,
respectively. The modulation of the interference and the transmission of the
object follow a proportional relationship Gilaberte:2021 . The colored
rectangles in Fig. 5(a) mark three regions of interest used to verify the
homogeneity of the calculated object transmission across different areas of
the FOV. The dependency of the transmission on the inserted optical density
filters for the three areas can be found in Fig. 5(c) and agrees well with the
theoretical prediction. In the Supplemental Material in Fig.
LABEL:fig:X_mod_noise one finds the uncertainty, that is to be expected for a
single measurement. For an exposure time of 500 ms and $M=4$ images used, one
gets a noise level of $0.053\pm 0.005$, which means the setup can detect a
minimum transmission difference of 5% to 6%. The visible border around the
eyes and mouth of the object are due to the limited spatial resolution (see
Supplemental Material for a detailed explanation).

Figure 5: Amplitude accuracy. (a) Modulation image of a $0.82\pi$ phase step
mask without optical density filters. (b) Modulation image of the same mask
fully covered with an optical density filter (OD 0.4). (c) Relative modulation
calculated values for the three different areas marked in the correspondent
colors in (a) and expected transmission (black solid line) for different
object transmission values, experimentally implemented by placing different
optical density filters in front of the phase mask. The modulation values are
retrieved from holographic images with $M=12$ phase steps each with 500 ms
exposure time.

## Discussion

We have introduced a novel technique to generate holograms with photons that
are never detected. Our method is a quantum version of phase shifting
holography, which retrieves full information, phase and amplitude, of an
object. In order to do so, we make use of correlated signal and idler photon
pairs, in a nonlinear interferometer configuration to obtain a hologram of the
idler photon, through the measurement of the signal photon. Our technique is
based on the quantum effect of ’induced coherence without induced emission’,
where information imprinted on the undetected (idler) photon can be retrieved
analyzing the interference pattern of its partner (signal) photon. The main
advantage of our technique over previous approaches is that we retrieved
simultaneously the phase and amplitude of the shape of a photon in spite of
never detecting this photon.

Our approach also alleviates a key challenge towards integrating SU(1,1)
interferometers in real-world biomedical applications: Previous demonstrations
of imaging with undetected photons required long-term phase stability of
relevant paths in an interferometer, and were thus either limited to operation
in a controlled laboratory environment Lemos:2014 , or involved the added
technical complexity of active stabilization. Our approach has reduced the
required level phase stability to the order of 2 s (the timescale over which a
complete round of phase settings can be applied). Such levels of stability are
straightforward to achieve in ruggedized optical assemblies Gilaberte:2021  \-
a key advancement towards practical deployment.

We quantify our method in terms of the achievable resolution given by the
number of measurements and the acquisition time. It turns out that a
combination of 500 ms acquisition time and four phase shifted images are an
ideal combination in terms of recording time and image quality. We retrieve
transmission values with a precision of down to 6% and phases with a precision
of down to $0.1\pi$. Increasing the degree of spatial correlation between
signal and idler as well as the numerical aperture of the imaging system can
drastically enhance the spatial resolution of this method. Moreover, by
combining our technique with quantum optical coherence tomography in nonlinear
interferometers Valles:2018 ; paterovaOCT2018 ; ramelowOCT2020 ; gerardOCT2020
, three-dimensional image reconstruction with undetected light becomes
feasible.

## Materials and methods

### Image Processing

Figure 6: Exemplary raw image set. One set of recorded images for the phase
shifting holography calculation ordered from left to right and top to bottom.
(a) The object images. (b) The reference images.

For the phase shifting holography calculation a set of twelve images of the
pure interference as reference and another set of twelve images including the
object is recorded (see Fig. 6). To prepare the images for the calculation,
they were filtered to reduce the noise level. The filtering has three sub-
steps. First, the camera background is subtracted. As Background, a recording
without pump beam was taken. Second, a 2D low-pass frequency filter is applied
performing a Fourier transformation on the input images and cuts off high
frequency noise. Last, a Gaussian filter is applied scipy:2020 . With this the
signal-to-noise ratio was increased by a factor of up to 1.82 depending on the
exposure time and number of images.

After the filtering, the phase and modulation are calculated. For this step,
the images are separated into a subset depending on the algorithm to test. The
base image set consist of twelve images taken at equally spaced phase
positions with a phase difference of $\Delta\varphi_{m}=2m\pi/M;m=0,...,M-1$
to the first position ($m=0$). This set was split into four subset containing
$M=12,6,4,3$ images, equally spaced inside the twelve from the base set and
always starting at the same image $m=0$. This simulates measurements with less
phase positions, without the need for additionally recordings to reduce
differences between the measurements.

The calculation was done using a least-squares based algorithm for phase shift
holography malacara:2007 . The phase is obtained by Eq. (6), with
$N_{\Delta\varphi_{m}}$ as the recorded intensity of image $m$. A phase
unwrapping algorithm is applied afterwards to resolve the periodicity
skimage:2020 . After this, the phase of the reference is subtracted to get the
absolute phase change induced by placing the object inside the setup. Our
software gives real-time images (live feedback) for the expected modulation
and phase. We introduced 300 ms delay time between each acquisition frame for
the image post-processing. The live feedback uses a constantly updated
circular buffer for the phase positions. The code for the final post-processed
image needs approx. 45 s to evaluate 360 raw images, i.e., 15 measurement sets
containing 12 raw images of the reference and the object. The results are 15
phase and modulation images. The calculation time can be significant reduced
implementing parallel-processing and graphics processing unit (GPU). This is
also true for the live feedback system.

In contrast to Eq. (7), losses at optical components will occur in the
experiment. Hence, the interference modulation is measured, which is directly
proportional to the transmission. Therefore, a referencing to an additional
measurement without object is necessary to retrieve the absolute transmission
values of an object. The modulation is calculated by Eq. (7).

### Object fabrication

The objects used in the experiments were generated by direct writing grayscale
lithography. In doing so, a substrate is coated with a photoresist with a
refractive index of 1.63 (at $\sim 520$ nm) and exposed to UV-light in a
particular pattern. The chemical solubility of the exposed areas increases in
alkaline media and the micro-structure in the photoresist is formed.

## References

## References

  * (1) F. Xu, X. Ma, Q. Zhang, H.-K. Lo, and J.-W. Pan, “Secure quantum key distribution with realistic devices,” Rev. Mod. Phys. 92, 025002 (2020).
  * (2) S. Wengerowsky, J. Siddarth Koduru, F. Steinlechner, J. R. Zichi, B. Liu, T. Scheidl, S. M. Dobrovolskiy, R. v. d. Molen, J. W. N. Los, V. Zwiller, M. A. M. Versteegh, A. Mura, D. Calonico, M. Inguscio, A. Zeilinger, A. Xuereb, and R. Ursin, “Passively stable distribution of polarisation entanglement over 192 km of deployed optical fibre,” npj Quantum Inform. 6, 1 (2020).
  * (3) E. Ortega, K. Dovzhik, J. Fuenzalida, S. Wengerowsky, J. C. Alvarado-Zacarias, R. F. Shiozaki, R. Amezcua-Correa, M. Bohmann, and R. Ursin, “Experimental space-division multiplexed polarization entanglement distribution through a 19-path multicore fiber,” arXiv preprint arXiv:2103.10791 (2021).
  * (4) H.-S. Zhong, H. Wang, Y.-H. Deng, M.-C. Chen, L.-C. Peng, Y.-H. Luo, J. Qin, D. Wu, X. Ding, Y. Hu, P. Hu, X.-y. Yang, W.-J. Zhang, L. Hao, L. Yuxuan, J. Xiao, G. Lin, G. Y. Lixing You, W. Zhen, L. Li, L. Nai-Le, L. Chao-Yang, and P. Jian-Wei, “Quantum computational advantage using photons,” Science 370, 1460 (2020).
  * (5) G. Brida, M. Genovese, and I. R. Berchera, “Experimental realization of sub-shot-noise quantum imaging,” Nature Photonics 4, 227 (2010).
  * (6) G. B. Lemos, V. Borish, G. D. Cole, S. Ramelow, R. Lapkiewicz, and A. Zeilinger, “Quantum imaging with undetected photons,” Nature 512, 409 EP (2014).
  * (7) D. A. Kalashnikov, A. V. Paterova, S. P. Kulik, and L. A. Krivitsky, “Infrared spectroscopy with visible light,” Nature Photonics 10, 98 (2016).
  * (8) M. Gilaberte Basset, F. Setzpfandt, F. Steinlechner, E. Beckert, T. Pertsch, and M. Gräfe, “Perspectives for Applications of Quantum Imaging,” Laser & Photonics Reviews 13, 1900097 (2019).
  * (9) P.-A. Moreau, E. Toninelli, T. Gregory, and M. J. Padgett, “Imaging with quantum states of light,” Nature Reviews Physics 1, 367 (2019).
  * (10) H. Defienne, B. Ndagano, A. Lyons, and D. Faccio, “Polarization entanglement-enabled quantum holography,” Nature Physics 1–7 (2021).
  * (11) R. Chrapkiewicz, M. Jachura, K. Banaszek, and W. Wasilewski, “Hologram of a single photon,” Nature Photonics 10, 576 (2016).
  * (12) D. Gabor, “A new microscopic principle,” Nature 161, 777 (1948).
  * (13) G. Popescu, _Quantitative Phase Imaging of Cells and Tissues_ , McGraw-Hill biophotonics Quantitative phase imaging of cells and tissues, McGraw-Hill Education (2011).
  * (14) P. Marquet, B. Rappaz, P. J. Magistretti, E. Cuche, Y. Emery, T. Colomb, and C. Depeursinge, “Digital holographic microscopy: a noninvasive contrast imaging technique allowing quantitative visualization of living cells with subwavelength axial accuracy,” Opt. Lett. 30, 468 (2005).
  * (15) P. Refregier and B. Javidi, “Optical image encryption based on input plane and Fourier plane random encoding,” Opt. Lett. 20, 767 (1995).
  * (16) S. Liu, C. Guo, and J. T. Sheridan, “A review of optical image encryption techniques,” Optics & Laser Technology 57, 327, optical Image Processing (2014).
  * (17) F.-K. Bruder, R. Hagen, T. Rölle, M.-S. Weiser, and T. Fäcke, “From the Surface to Volume: Concepts for the Next Generation of Optical–Holographic Data-Storage Materials,” Angewandte Chemie International Edition 50, 4552 (2011).
  * (18) W. L. Wilson, K. R. Curtis, K. E. Anderson, M. C. Tackitt, A. J. Hill, M. Pane, C. Stanhope, T. Earhart, W. Loechel, C. Bergman, K. Wolfgang, C. Shuman, G. Hertrich, K. Pharris, K. Malang, B. Riley, and M. Ayres, “Realization of high-performance holographic data storage: the InPhase Technologies demonstration platform,” K. Meerholz, editor, “Organic Holographic Materials and Applications,” volume 5216, 178 – 191, International Society for Optics and Photonics, SPIE (2003).
  * (19) B. Yurke, S. L. McCall, and J. R. Klauder, “SU(2) and SU(1,1) interferometers,” Phys. Rev. A 33, 4033 (1986).
  * (20) L. J. Wang, X. Y. Zou, and L. Mandel, “Induced coherence without induced emission,” Phys. Rev. A 44, 4614 (1991).
  * (21) T. Herzog, J. Rarity, H. Weinfurter, and A. Zeilinger, “Frustrated two-photon creation via interference,” Physical review letters 72, 629 (1994).
  * (22) M. V. Chekhova and Z. Y. Ou, “Nonlinear interferometers in quantum optics,” Adv. Opt. Photon. 8, 104 (2016).
  * (23) A. C. Cardoso, L. P. Berruezo, D. F. Ávila, G. B. Lemos, W. M. Pimenta, C. H. Monken, P. L. Saldanha, and S. Pádua, “Classical imaging with undetected light,” Phys. Rev. A 97, 033827 (2018).
  * (24) M. Gilaberte Basset, A. Hochrainer, S. Töpfer, F. Riexinger, P. Bickert, J. R. León-Torres, F. Steinlechner, and M. Gräfe, “Video-Rate Imaging with Undetected Photons,” Laser & Photonics Reviews 15, 2000327 (2021).
  * (25) M. Kutas, B. Haase, P. Bickert, F. Riexinger, D. Molter, and G. von Freymann, “Terahertz quantum sensing,” Science Advances 6, eaaz8065 (2020).
  * (26) A. V. Paterova, H. Yang, C. An, D. A. Kalashnikov, and L. A. Krivitsky, “Tunable optical coherence tomography in the infrared range using visible photons,” Quantum Science and Technology 3, 025008 (2018).
  * (27) I. Kviatkovsky, H. M. Chrzanowski, E. G. Avery, H. Bartolomaeus, and S. Ramelow, “Microscopy with undetected photons in the mid-infrared,” Science Advances 6, eabd0264 (2020).
  * (28) A. V. Paterova, S. M. Maniam, H. Yang, G. Grenci, and L. A. Krivitsky, “Hyperspectral infrared microscopy with visible light,” Science Advances 6, eabd0460 (2020).
  * (29) A. Vallés, G. Jiménez, L. J. Salazar-Serrano, and J. P. Torres, “Optical sectioning in induced coherence tomography with frequency-entangled photons,” Phys. Rev. A 97, 023824 (2018).
  * (30) A. V. Paterova, H. Yang, A. Chengwu, D. A. Kalashnikov, and L. A. Krivitsky, “Tunable optical coherence tomography in the infrared range using visible photons,” Quantum Sci. Technol. 3 025008 3, 025008 (2018).
  * (31) A. Vanselow, P. Kaufmann, I. Zorin, B. Heise, H. M. Chrzanowski, and S. Ramelow, “Frequency-domain optical coherence tomography with undetected mid-infrared photons,” Optica 7, 1729 (2020).
  * (32) G. J. Gerard J. Machado, G. Frascella, J. P. Torres, and M. V. Chekhova, “Optical coherence tomography with a nonlinear interferometer in the high parametric gain regime,” Applied Physics Letters 117, 094002 (2020).
  * (33) I. Yamaguchi and T. Zhang, “Phase-shifting digital holography,” Opt. Lett. 22, 1268 (1997).
  * (34) J. W. Goodman, _Introduction to Fourier Optics_ , Roberts and Company Publishers (2005).
  * (35) H. Schreiber and J. H. Bruning, _Phase Shifting Interferometry_ , chapter 14, 547–666, John Wiley & Sons, Ltd (2007).
  * (36) X. Y. Zou, L. J. Wang, and L. Mandel, “Induced coherence and indistinguishability in optical interference,” Phys. Rev. Lett. 67, 318 (1991).
  * (37) SmarAct GmbH, “Specifications: CLS-3232,” https://www.smaract.com/linear-stages/product/cls-3232-linear-stage-with-edu, accessed: 2021-08-25.
  * (38) Teledyne Photometrics, “Specifications: Prime BSI Scientific CMOS,” https://www.photometrics.com/products/prime-family/primebsi, accessed: 2021-08-25.
  * (39) N. B. Nill, “Conversion Between Sine Wave and Square Wave Spatial Frequency Response of an Imaging System,” Technical report, MITRE (2001).
  * (40) “scipy.ndimage.gaussian_filter,” https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.gaussian_filter.html, accessed: 2020-12-13.
  * (41) “unwrap_phase,” https://scikit-image.org/docs/stable/api/skimage.restoration.html#skimage.restoration.unwrap_phase, accessed: 2020-12-13.
  * (42) A. Gatti, R. Zambrini, M. San Miguel, and L. A. Lugiato, “Multiphoton multimode polarization entanglement in parametric down-conversion,” Physical Review A 68, 053807 (2003).
  * (43) J. P. Torres, K. Banaszek, and I. A. I. A. Walmsley, “Engineering Nonlinear Optic Sources of Photonic Entanglement,” Progress in Optics 56, 227 (2011).
  * (44) R. W. Boyd, G. S. Agarwal, K. W. C. Chan, A. K. Jha, and M. N. Sullivan, “Propagation of quantum states of light through absorbing and amplifying media,” Optics Communications 281, 3732 (2008).

## Acknowledgements

The authors want to thank Anja Schöneberg and Robert Leitel from Fraunhofer
IOF for preparing the samples used in the measurements.

### Funding

This work was supported as a Fraunhofer LIGHTHOUSE PROJECT (QUILT) and by the
Fraunhofer Attract program (QCtech). Furthermore, funding support is
acknowledged from the German Federal Ministry of Education and Research (BMBF)
within the funding program Photonics Research Germany with contract number
13N15088. JPT acknowledges financial support from the Spanish Ministry of
Economy and Competitiveness through the “Severo Ochoa” program for Centres of
Excellence in R&D (CEX2019-000910-S), from Fundació Privada Cellex, Fundació
Mir-Puig, from Generalitat de Catalunya through the CERCA program, from
project 20FUN02 “POLight” funded by the EMPIR programme co-financed by the
Participating States and the European Union’s Horizon 2020 research and
innovation programme, and from project QUISPAMOL (PID2020-112670GB-I00) funded
by MCIN/AEI /10.13039/501100011033. This project has received funding from the
European Union’s Horizon 2020 research and innovation programme under grant
agreement No 899580.

### Author Contributions

M.G.B., F.S., and M.G. conceived the idea. J.P.T. provided the theoretical
analysis. S.T., M.G.B., and J.F. designed and performed the experiments. S.T.
conducted the computations and analyzed the data. All authors discussed the
results and contributed equally in writing the paper and have given approval
to the final version of the manuscript.

### Competing interests

The authors declare that they have no competing interests.

### Data and materials availability

All data needed to evaluate the conclusions in the paper are present in the
paper and/or the Supplementary Materials.

### Additonal information

The work of this study is related to a pending patent application with the
number EP20159989. The inventors are co-authors of this study.