# End-to-end Learning for Joint Depth and Image Reconstruction from Diffracted
Rotation

Mazen Mel<EMAIL_ADDRESS>Muhammad Siddiqui  Pietro Zanuttigh
Department of Information Engineering, University of Padova, Italy.  Sony
Europe B.V., Zweigniederlassung Deutschland Stuttgart Technology Center,
Stuttgart, Germany.

#
Supplementary Material

Mazen Mel<EMAIL_ADDRESS>Muhammad Siddiqui  Pietro Zanuttigh
Department of Information Engineering, University of Padova, Italy.  Sony
Europe B.V., Zweigniederlassung Deutschland Stuttgart Technology Center,
Stuttgart, Germany.

###### Abstract

Monocular depth estimation is still an open challenge due to the ill-posed
nature of the problem at hand. Deep learning based techniques have been
extensively studied and proved capable of producing acceptable depth
estimation accuracy even if the lack of meaningful and robust depth cues
within single RGB input images severally limits their performance. Coded
aperture-based methods using phase and amplitude masks encode strong depth
cues within 2D images by means of depth-dependent Point Spread Functions
(PSFs) at the price of a reduced image quality. In this paper, we propose a
novel end-to-end learning approach for depth from diffracted rotation. A phase
mask that produces a Rotating Point Spread Function (RPSF) as a function of
defocus is jointly optimized with the weights of a depth estimation neural
network. To this aim, we introduce a differentiable physical model of the
aperture mask and exploit an accurate simulation of the camera imaging
pipeline. Our approach requires a significantly less complex model and less
training data, yet it is superior to existing methods in the task of monocular
depth estimation on indoor benchmarks. In addition, we address the problem of
image degradation by incorporating a non-blind and non-uniform image
deblurring module to recover the sharp all-in-focus image from its RPSF-
blurred counterpart.

###### Abstract

In this supplementary material document, we start by analyzing the focusing
error in Out Of Focus (OOF) imaging in Section 1. Then in Section 2 we briefly
present the wave-optics based analysis of light field propagation and the
phase transformation introduced by the lens and the phase mask. We also show
the effects of the phase mask design parameters on the shape of the resulting
RPSF in Section 3.

Then, we present some additional qualitative results for the tasks of
monocular depth estimation and image deblurring (Sections 4 and 5) that were
not possible to fit in the main paper due to space limitations. Finally, a
visual example of the results of the various tests made in the ablation study
is also presented in Section 6.

###### keywords:

Monocular Depth Estimation, RPSFs, Image Deblurring.

##  1 Introduction

Depth estimation from a single RGB image is an ill-posed inverse problem,
thus, additional image priors or sophisticated imaging systems are generally
needed to account for the lack of depth cues in captured images. For example,
the camera optics may be modified accordingly, a more advanced image sensor
design and task-specific post processing algorithms may be developed in order
to capture information well beyond the capabilities of conventional imaging
systems.

In this paper, we propose a computational camera wherein the Point Spread
Function (PSF) is altered in order to encode depth information within a 2D
image. The desired PSF is obtained via an optimized phase mask inserted at the
aperture plane of the camera. Our system encodes meaningful and robust depth
cues in single RGB images thus making it easier for post-processing algorithms
to produce accurate depth data. However, such approach generally suffers from
image quality degradation due to the poor light efficiency and/or the low
Modulation Transfer Function (MTF) of the engineered PSF for high spatial
frequency components. In fact, amplitude aperture masks block a significant
amount of light from reaching the image sensor resulting in low light
throughput, while using phase only masks solves such problem since it only
acts on the phase component of the incoming light wave. Still, the MTF drops
rapidly in high frequency regions and the depth-dependent blurring caused by
the camera’s engineered PSF produces low SNR and degrade image quality.

We propose a full pipeline that takes as input a single RGB image and produces
an estimated depth map of the scene along with the recovered sharp image from
its PSF-blurred counterpart. We introduce a novel end-to-end deep learning
model that jointly deals with the PSF design optimization and the depth
estimation task. To this aim, a differentiable physical model of the aperture
mask is introduced together with an accurate simulation of the imaging
pipeline including the optimized optics. In this way the learned model is able
to firstly predict the optimal parameters of the phase mask design and then
estimate the depth data from the RPSF-blurred input. Finally, in order to
address the image quality issue, a deep learning based non-blind and non-
uniform deblurring module is incorporated.

##  2 Related Work

#### Rotating Point Spread Functions

RPSFs have been theoretically shown to increase the Fisher Information along
the depth dimension by at least an order of magnitude as they have a uniformly
lower Cramér–Rao bound across the axial dimension [1], which makes them highly
sensitive to depth changes. RPSFs are obtained using pupil engineered cameras
by means of phase and/or amplitude masks that are inspired by the concept of
Orbital Angular Momentum (OAM) of light beams [2]. A beam with a rotating
light intensity distribution along the propagation axis can be generated by a
linear superposition of a set of suitably chosen Gaussian-Laguerre (GL) modes
[3, 1] that can be optically encoded by the aperture mask. These masks
generate a PSF with invariant features that continuously rotate with defocus.
However, such approach suffers from poor light throughput, some works [4, 5]
addressed this problem and proposed iterative optimization schemes to ensure
higher light efficiency of the engineered RPSF. Depth dependent RPSFs can be
also generated by phase only masks. For instance, a phase mask design inspired
by Spiral Phase Plates (SPPs) [6, 7] was introduced in [8] where the pupil is
subdivided into a set of annular Fresnel zones with an azimuthally increasing
thickness profile, the delay imposed on the incident light waves increases
azimuthally generating a corkscrew like wave-front carrying an OAM with a
rotating phase function. In [9] and [10], the authors generalized the previous
phase mask design by considering a phase function that allows for generating
multi-order-helix RPSFs by introducing new design parameters: the number of
rotating lobes within the RPSF and the confinement of each zone, i.e. the
inner and outer radii of each annular region, in addition to the number of
Fresnel zones. While [9] and [10] used purely empirical approaches to
determine the values for each design parameter, in this work we optimize those
parameters in an end-to-end fashion jointly with the weights of a depth
estimation deep neural network.

#### Depth estimation using coded apertures

Levin et al. [11] proposed an amplitude modulation mask that was placed in
front of a conventional camera lens to encode depth information via the
diffracted pattern of the camera’s PSF, as point-like sources move along a
plane parallel to that of the camera sensor, the mask shadow would shift
accordingly and as they move closer or farther from the camera, the pattern
would expand or shrink. This information is later used to determine the
object’s distance from the camera. The mask introduced by [11] has opaque
regions blocking a significant amount of light from reaching the image sensor.
Zhou et al. [12] built upon the work of Levin et al. [11] and introduced two
complementary amplitude masks.

In [1] the concept of depth from diffracted rotation was introduced: a
superposition of a set of suitably chosen Gaussian-Laguerre modes generates a
double helix RPSF that was used to estimate the depth of a planar scene by
analyzing the blurring effects within the captured image. However, low MTF by
the mask leads to poor SNR within the captured image, thus limiting the
capability of signal-processing based algorithms to recover sharp images or
accurate depth maps. In addition, earlier studies relied on a design paradigm
based on the separate optimization of the camera optics and post-processing
algorithms: they design the mask first and then tailor a reconstruction
algorithm that fits the proposed physical design of the mask as in [13, 10,
14, 15]. Such design methodology, however, leads to sub-optimal performance.

#### End-to-end learning for monocular depth estimation

Recently, an emerging trend appeared to tackle the separate design issue and
new frameworks for joint optical and digital optimization using deep learning
techniques have been introduced. These methods exploit end-to-end learning to
optimize the mask’s height map together with the trainable weights of a
Convolutional Neural Network (CNN). Haim et al. [16] proposed a differentiable
phase mask consisting of concentric rings that introduce depth-dependent
chromatic aberrations and encoding depth cues within single captured images.
In the work of Chang et al. [17] and Wu et al. [18] a free-form differentiable
phase mask design parameterized using a set of superposed Zernike polynomials
[19, 20] was jointly optimized with the weights of a U-Net [21]. However, the
employed camera model was not realistic accounting only for additive Gaussian
noise [18]. Furthermore, unrestricted parameterized mask design and higher
degrees of freedom may cause the optimization to converge towards local
minimas as the objective function becomes too complex leading to sub-optimal
performance. In this work, the mask is parameterized using only three design
parameters two of which are optimized in an end-to-end fashion, and the PSF
has clear and simple correlations with depth.

#### Image deblurring

Non-blind image deblurring has been extensively studied before (e.g., [22, 23,
24, 25]), and a substantial performance gain has been made easier using deep
learning based image deblurring techniques.

The authors of [26] used the separability property of the pseudo-inverse
kernel in Wiener deconvolution filter to design a dedicated CNN. In [27], a
two-stage deblurring process was introduced using a Wiener deconvolution
filter and a simple MLP architecture. A novel deconvolution approach was
recently introduced by [28] where a Wiener deconvolution filter is applied to
the input data in feature space.

##  3 Depth Estimation from Diffracted Rotation

Figure 1: The full architecture of our end-to-end learning framework.

We propose an end-to-end learning approach for depth from diffracted rotation
using RPSF-coded images. As shown in Fig. 1, the full pipeline of the proposed
solution encompasses three main stages. In the first stage, the height map of
the phase mask is jointly learned with the weights of a neural network (DEPTH-
DNN) trained to perform monocular depth estimation. This module is trained
using noise-free RPSF-blurred synthetic images. In the second stage, the
optimized phase mask is fitted within the optics module and a digital image
formation pipeline is applied to the RPSF-coded synthetic images in order to
simulate a realistic camera model. Finally, we used the demosaiced images from
the previous stage as input to fine-tune the weights of DEPTH-DNN on noisy
data obtaining a refined model (DEPTH-DNN-TUNED) and to recover the all-in-
focus sharp image using a dedicated network (IMAGE-DNN) which performs non-
blind and non-uniform image deblurring. Both of these modules make up the
third and last stage of the proposed architecture.

###  3.1 Engineered PSF

In this section we briefly describe the effect of modified optics on the
system’s PSF. For more details please refer to the supplementary materials.

For simplicity, consider an on-axis ideal point source situated at optical
infinity, the light field $U_{in}$ with a constant amplitude $P$ and a phase
$\psi$ emanating from such source has the form: $U_{in}=Pe^{i\psi}$. An
optical element such as a lens or a phase mask with a refractive index $n$ and
a height profile $h$ introduces a phase delay
$\Phi=\frac{2\pi(n-1)}{\lambda}h$ on incident light wave-fronts. If a phase
mask is inserted at the entrance pupil plane of an imaging system, the total
phase delay can be expressed as the sum of the delay due to the lens with the
one due to the mask:

$\Phi_{optics}=\Phi_{lens}+\Phi_{mask}$ (1)

The light field after the lens and mask system has the form:

$U_{out}=A\cdot P\cdot e^{i(\psi+\Phi_{optics})}$ (2)

Where $A$ is the aperture mask simulating the finite aperture area. The field
$U_{out}$ can be further propagated to the image plane and the PSF can be
obtained by the field’s intensity distribution. Choosing the appropriate
height profile of the phase mask helps design specific PSF patterns depending
on the target task.

###  3.2 Phase Mask Design

Both annular and free-form mask designs have been studied in the context of
joint optimization of camera optics with post-processing algorithms [16, 18,
17]. We build upon the depth-dependent RPSF introduced by [8]. In this
section, the mathematical model of the mask’s phase profile is described and
in the next section a differentiable approximation of the mask’s height map is
derived to allow for gradient back-propagation in our end-to-end learning
framework.

We use a Fresnel-zone based design [8] that has an outermost radius $R$ and
$L$ phase plates in the form of concentric annular regions each of topological
charge $l=1,...,L$ and bounded by two radii $R_{l-1}=R\sqrt{\frac{l-1}{L}}$
and $R_{l}=R\sqrt{\frac{l}{L}}$.

The pupil plane phase can be written as in [8]:

$\Phi_{mask}(\rho,\phi)=\begin{cases}\phi&0\leq\rho<\sqrt{\frac{1}{L}}\\\
&\vdots\\\ l\phi&\sqrt{\frac{l-1}{L}}\leq\rho<\sqrt{\frac{l}{L}}\\\ &\vdots\\\
L\phi&\sqrt{\frac{L-1}{L}}\leq\rho<1\end{cases}$ (3)

$\Phi_{mask}$ is defined with the polar position vector $\rho$ normalized by
the pupil’s outermost radius $R$, and is a step function modeling the physical
design property of concentric rings each with its own phase profile.

The phase profile in Eq. 3 can be further generalized to account for multi-
order-helix RPSFs as in [10] and [9] by expanding it into:

$\Phi_{mask}(\rho,\phi)=\begin{cases}\phi&0\leq\rho<(\frac{1}{L})^{\epsilon}\\\
&\vdots\\\
[(l-1)N+1]\phi&(\frac{l-1}{L})^{\epsilon}\leq\rho<(\frac{l}{L})^{\epsilon}\\\
&\vdots\\\ [(L-1)N+1]\phi&(\frac{L-1}{L})^{\epsilon}\leq\rho<1\end{cases}$ (4)

Notice that the inner and outermost radii of each zone are now controlled by
$\epsilon$ which lies in $[0,1]$, and the topological charge of each ring is
now $[(l-1)N+1]$ instead of just $l$. Besides the number of rings $L$, $N$ and
$\epsilon$ are two new design parameters each having an effect over the
resulting RPSF shape. More precisely, $N$ defines the number of peaks or the
main rotating lobes of the RPSF and $L$ and $\epsilon$ both control the peak
separation and confinement of each peak. In the case of a single helix
rotating PSF $(N=1)$ the phase profile of each ring would be reduced to the
original expression as in Eq. 3. Notice also that by increasing the number of
peaks, the practical depth range would be reduced because of rotation
ambiguities when peaks rotate beyond $[-\frac{\pi}{N},\frac{\pi}{N}]$.

###  3.3 Differentiable Phase Mask Design

The phase profile presented in Eq. 4 is not differentiable with respect to the
design parameters $N$ and $\epsilon$. Thus, a differentiable approximation for
this equation is necessary in order to simulate the camera’s optical layer and
enable both forward and backward propagation. The number of Fresnel zones $L$
is considered as a hyper-parameter that can be manually tuned to achieve
better depth estimation performance.

Figure 2: A differentiable phase mask design where each Fresnel zone is
simulated by a ring mask multiplied by the phase profile ($\Phi_{mask}^{i}$)
corresponding to each zone.

The steps in Eq. 4 can be approximated with a set of 2D rings in polar
coordinates with increased radii as $L$ increases as illustrated in Fig. 2,
each Fresnel zone is obtained by subtracting the areas of two 2D $\tanh$
functions each with a radius corresponding to the one of the two radiis of the
desired ring. We multiply the inner coordinates of each $\tanh$ by a large
constant (100 in our case) in order to get sharp mask edges. The new
approximated phase profile equation for $L$ zones can be written as:

$\displaystyle\tilde{\Phi}_{mask}(\rho,\phi)=\sum_{l=0}^{l=L}\underbrace{\bigr{(}(l-1)N+1\bigl{)}\phi}_{\text{$l^{th}$
ring phase}}\times{}$ (5)
$\displaystyle\underbrace{\frac{1}{2}\bigr{(}\tanh[100(\rho-
r_{l})]-\tanh[100(\rho-r_{l+1})]\bigl{)}}_{\text{$l^{th}$ ring mask}}$

Where $r_{0}=0$, $r_{l}=R(\frac{l}{L})^{\epsilon};~{}\forall l\in[1..L]$, and
$R$ is the outermost radius of the mask. $\phi$ and $\rho$ are the polar
coordinates.

Each phase profile $\Phi_{mask}^{l}=\bigr{(}(l-1)N+1\bigl{)}\phi$ is
multiplied by the corresponding ring mask and the resulting zones are added up
to produce the final phase profile of the mask. We produce the height map $h$
of the mask as follows:

$h(x,y)=\frac{\lambda}{2\pi(n-1)}\cdot\bigr{(}arg\\{e^{i\tilde{\Phi}_{mask}}\\}\mod
2\pi\bigl{)}$ (6)

where $arg$ is the complex argument function, and the modulo accounts for the
phase wrapping operation (phase values have a $2\pi$ periodicity as shown in
Fig. 3).

Figure 3: The height map of the phase mask. From left to right, the phase
distribution as the argument of $e^{i\tilde{\psi}}$, the $[0,2\pi]$ wrapped
phase distribution, and the obtained height map for $[N=2,L=5,\epsilon=0.9]$.

Notice that the height profile of the phase mask is wavelength dependent: the
resulting PSFs for the three RGB color primaries have different rotation rates
which introduces chromatic aberrations. Still, it will not be problematic in
the context of an end-to-end optimization framework since the network could
learn the correlations between the rotation rate and the corresponding color
channel. In fact, such aberrations can be seen as depth-dependent and can also
relay valuable depth cues. For the physical design of the mask a reference
wavelength value ($\lambda=536.67nm$) is used to produce the height map.

Fig. 4 shows a double helix RPSF generated by a phase mask with
$[N=2,L=5,\epsilon=0.9]$, it has two main lobes rotating counterclockwise as a
function of defocus. Notice that even at the in-focus plane the RPSF has the
same shape thus objects that are ”in-focus” are also blurred.

Figure 4: A double helix RPSF obtained by a mask with
$[N=2,L=5,\epsilon=0.9]$. The RPSF’s intensity distributions are shown as a
function of defocus.

##  4 Training Data Generation

###  4.1 Datasets

A subset of FlyingThings3D dataset is used for the joint optimization of the
phase mask and depth estimation network, and to train IMAGE-DNN. This dataset
is a part of the Scene Flow synthetic datasets [29]. This subset was
previously used by [18] in a similar joint optimization approach, it contains
synthetic images of randomly placed objects with pixel-accurate disparity
maps. The training, validation, and test sets contain respectively 5078, 555,
and 420 images with a resolution of $278\times 278$ pixels. Additionally, in
order to evaluate the performance of our approach with the state-of-the-art in
the task of monocular depth estimation on real data, NYUV2 [30] depth dataset
is used to train and evaluate DEPTH-DNN. Originally, the dataset contains 120k
training images of indoor scenes with a resolution of $640\times 480$ pixels
along with ground truth depth maps acquired by a Microsoft Kinect V1 depth
sensor. The test set as defined by the split in [31] contains 654 images. In
this work, a subset of 50k samples of NYUV2 is used to train the depth
estimation network as in [32]. Finally, the test set of SUNRGBD dataset [33]
is used to evaluate the generalization capability of DEPTH-DNN. This set
contains 5050 test images of indoor scenes with ground truth depth maps
acquired by four different depth sensors some of which use active illumination
techniques and others incorporate passive stereo systems.

###  4.2 Image Formation Model

Light rays emanating from the scene are acquired by the camera and optically
coded by the phase mask via a depth-dependent blurring process with the
camera’s RPSF. For our simulations, the ground truth depth maps are
approximated with a layered model in which only a finite number of depth
planes are used to compose and render the RPSF-coded image using the following
image formation model:

$I_{sim}=\sum_{d=1}^{d=D}(I_{aif}*RPSF_{d})\odot M_{d}$ (7)

Where $I_{sim}$ is the simulated blurred image, $I_{aif}$ is its all-in-focus
counterpart, $RPSF_{d}$ is the RPSF intensity distribution at the depth plane
$d$, $\odot$ stands for element-wise multiplication, and
$\\{M_{d};~{}d=1,...,D\\}$ are the depth masks presenting the individual depth
layers such that at each pixel location $\sum_{d}M_{d}=1$, i.e. only one pixel
mask is set to one at each position.

Afterwards, the image formation pipeline is applied to the RPSF-blurred images
to simulate real cameras. As illustrated by Fig. 1, a Bayer CFA receives the
full color resolution RPSF-blurred image and produces down-sampled RGGB color
pattern. Although it is hard to accurately simulate the noise behaviour within
the sensor chip, the final amount of noise is mainly caused by sensor shot and
read noises. To this end, we simulated the read noise with an additive
Gaussian $\mathcal{N}(0,\,\sigma^{2})$ with zero mean and a fixed standard
deviation $\sigma=0.01$, photon shot noise follows a Poisson distribution, in
practice it is modeled by a Gaussian distribution whose mean and variance
depend on the expected photon count over the exposure time. The resulting
noisy sensor image is quantized with an ADC module with a resolution of 8
bits. Finally, a linear interpolation-based demosaicing technique of [34] is
used to recover the full color channels from the CFA and produce the final
output which will be used to fine-tune DEPTH-DNN and train IMAGE-DNN.

##  5 Deep Learning Framework

###  5.1 Monocular Depth Estimation

In the first stage of the proposed solution (see Fig. 1), the phase mask
design parameters $[N,\epsilon]$ are jointly learned with the weights of a
U-Net [21] which is trained on a subset of FlyingThings3D [29]. Two different
learning rates, $L_{r}^{mask}=0.1$ and $L_{r}^{dnn}=1e-4$ are set for the
phase mask and for the depth estimation neural network. During the training
process, the gradient error is back-propagated through the network as well as
the mask’s trainable parameters and the weights are updated accordingly using
TensorFlow’s automatic differentiation framework [35]. The network of this
stage is trained for 150k iterations using a batch size of 20. Adam optimizer
[36] is used with exponential decay rates of the first momentum and second
momentum respectively set to $\beta_{1}=0.99$ and $\beta_{2}=0.999$.

Similar to Wu et al. [18], we used a combination of Root Mean Square Error
(RMSE) loss $\mathcal{L}_{rmse}$ and gradient loss $\mathcal{L}_{grad}$ which
forces the network to estimate accurate depth maps with well defined object
boundaries at different depth planes. The total loss function is defined as:

$\mathcal{L}_{depth}=\mathcal{L}_{rmse}+\mathcal{L}_{grad}$ (8)

Where $\mathcal{L}_{rmse}$ and $\mathcal{L}_{grad}$ are defined as:

$\mathcal{L}_{rmse}(\theta,\theta^{*})=\sqrt{\frac{1}{|T|}\sum_{\theta\in
T}(\theta-\theta^{*})^{2}}$ (9)

$\mathcal{L}_{grad}(\theta,\theta^{*})=\mathcal{L}_{rmse}(\frac{\partial\theta}{\partial
x},\frac{\partial\theta^{*}}{\partial
x})+\mathcal{L}_{rmse}(\frac{\partial\theta}{\partial
y},\frac{\partial\theta^{*}}{\partial y})$ (10)

$\theta$ and $\theta^{*}$ are respectively the predicted and the ground truth
disparity maps and the subtraction is done pixel-by-pixel, $x$ and $y$ are the
spatial dimensions, and $|T|$ is the number of disparity maps.

In the third stage, the same U-Net [21] which was previously trained on noise-
free RPSF data is fine-tuned using 50k training iterations with noisy images
simulated by the camera model of the second stage (see Fig. 1), the phase mask
is fixed during this training pass and only the network’s weights are updated.
All hyper-parameters’ values are fixed to the same values as in the first
training stage.

For NYUV2 dataset [30], both the network and the mask are learned using 50k
training samples, the input images to the network and the output depth maps
have a resolution of $320\times 240$ which correspond to half of that of the
original samples, a bilinear-upsampling is applied to the predicted depth maps
to recover the original resolution of $640\times 480$ for evaluation purposes
as in [32, 37, 38]. The network is trained for 150k iterations with a batch
size of 20, the phase mask and the neural network learning rates as well as
the optimizer used in the training are the same as the ones used for the
subset of FlyingThings3D [29].

###  5.2 Image Deblurring

We used a modified version of the deblurring model proposed by [28] wherein a
Wiener deconvolution is applied to a set of feature maps extracted from the
blurred input using a simple CNN architecture as feature extractor, the
deconvolved feature maps are then fed into a multi-scale feature refinement
stage in order to get the final deblurred image in a coarse-to-fine based
reconstruction technique. Such approach proved capable of restoring very fine
structural details allowing for accurate image reconstruction. In this work,
we adapted the network proposed by [28] for the case of non-uniform image
deblurring as the blur kernel in our case is spatially variant. More
precisely, a separate deconvolution is performed for every depth layer and the
results are cropped using the corresponding depth masks $M_{d}$ as in Eq. 7.
The deconvolved feature maps are then combined in order to get the final
Wiener filter output. We found that, even though the Wiener filter module is
applied in feature space, some undesirable deconvolution artifacts, most
noticeably ringings, are visible especially around image boundaries and object
edges. Since the $FFT$ operator within the Wiener deconvolution filter
supposes circular periodicity of the input. To tackle this problem, an
edgetaper operation [39] was implemented on the blurred input image to smooth
out its boundaries which can considerably reduce the ringing artifacts in the
final reconstructed sharp image.

We trained the network using the subset of FlyingThings3D [29] for 500 epochs
with Adam optimizer [36] with exponential decay rates of the first momentum
and second momentum respectively set to $\beta_{1}=0.99$ and
$\beta_{2}=0.999$, and a learning rate $L_{r}^{image}=1e-4$ which is halved
after 250 epochs. The number of auto-encoders in the multi-scale feature
refinement modules is set to 2 as in the original work of [28], the number of
extracted feature maps from the blurry input is 16, and the batch size is set
to 8. For the loss function, it was experimentally seen that $L{1}$ norm leads
to better reconstruction results than the ones obtained with $L{2}$ norm.

$\mathcal{L}_{image}(\theta,\theta^{*})=\frac{1}{|T|}\sum_{\theta\in
T}|\theta-\theta^{*}|$ (11)

Where $\theta$ and $\theta^{*}$ are respectively the reconstructed and the
ground truth sharp images and $|T|$ is the number of images.

##  6 Experimental Results

###  6.1 Experimental Results on Synthetic Data

#### Monocular depth estimation on FlyingThings3D subset

For this set of experiments the phase mask’s trainable parameters are
initialized to $[N=1,\epsilon=0.1]$ and the number of Fresnel zones $L$ is set
to 7 as it was empirically observed that such value leads to a lower depth
estimation error. In the case of noise-free RPSF-blurred inputs, the network
achieved a RMSE of 0.392 on the test set. The corresponding learned phase mask
parameters are $[N=1,\epsilon=0.92]$: the generated RPSF as well as the height
map of the mask are shown in Fig. 5. The network learned a single-helix RPSF
with high confinement parameter $\epsilon$ meaning that the peak is spread out
across a large area.

Figure 5: Learned RPSF shape at different defocus planes (left), height map of
the phase mask (right). Figure 6: Qualitative results on RPSF-blurred images
from the test set of FlyingThings3D [29] subset.

Qualitative results are shown in Fig. 6 (additional scenes are shown in the
supplementary material): Notice how the network is able to predict accurate
disparity maps with fine image details, e.g., the small leaves of the plant
shown in the third row or the very fine parts of the headset shown in the last
row. However, it becomes harder for the network to predict accurate object
boundaries due to the blurring artifacts introduced by the RPSF especially
when the blur kernel is larger than the image features. Some artifacts appear
at object edges mainly due to the nature of the image formation model used to
compose the RPSF-blurred images: the layered depth model used to render RPSF
coded images does not accurately simulate the discontinuities around object
edges as visible in the RPSF coded images (Fig. 6) due to the lack of accurate
occlusion modeling. Such issue can be addressed by implementing a more
advanced and sophisticated blending and matting approaches, e.g. Pyramid-based
blending [40], at the expense of much higher computation time and complexity
for a minor gain in performance.

As pointed out in Section 4.2, we also introduced an accurate noise simulation
model for a more realistic evaluation. To handle noise we further fine-tune
the DEPTH-DNN on noisy images. In this case we achieve a RMSE of 0.712 on the
test set compared to 0.392 achieved on noise-free data. This is of course
expected but at the same time the robustness of the system in real world
applications should be enhanced.

In the first two columns of Fig. 6, one can observe image quality degradation
after simulating the noisy images where the color down-sampling by the CFA and
quantization artifacts by the ADC unit are visible (zoomed-in). The predicted
disparity maps from noisy inputs are shown in the last column. Even though a
small performance degradation is noticeable on the noisy predictions, the
fine-tuned network is still able to learn fairly accurate disparity maps.

#### Image restoration

The sharp all-in-focus images are recovered by IMAGE-DNN trained on the subset
of FlyingThings3D [29]. Quantitative and qualitative results are shown in
Table. 1 and Fig. 7, respectively.

The simulated RPSF-coded images have a low mean PSNR of roughly $19$ dB with
respect to their sharp noise-free counterparts which were used as the ground
truth images during training. As reported in Table. 1, the mean PSNR of the
recovered images increased by about $5.5$ dB reaching $24.46$ dB. Also, the
Structural Similarity Index Measure (SSIM) achieved is $0.760$ compared to
$0.611$ for the blurred and noisy images. Deblurring results from the
traditional Wiener deconvolution filter [41] are also shown in Fig. 7: even if
we apply the deconvolution process independently for each depth plane and
generated the final result following Eq. 7, it results in a low quality image
reconstruction with heavy ringing artifacts (see Fig. 7). This happens since
the Wiener filter fails to handle the spatially variant blur producing
significant ringing artifacts.

| PSNR (dB) | SSIM
---|---|---
RPSF-coded images | 19.01 | 0.611
Recovered images (Wiener) | 18.75 | 0.449
Recovered images (Our approach) | 24.46 | 0.760

Table 1: Quantitative results of the image deblurring model on the test set of
FlyingThings3D [29] subset. Figure 7: Qualitative results of the image
deblurring model on the test set of FlyingThings3D [29] subset.

Fig.7 shows some recovered images along with the corresponding blurred inputs
and the sharp all-in-focus ground truth. Upon visual inspection, one can
notice that the model successfully restores very fine image details and high
frequency components, e.g., the small tree leaves shown in the first row, or
the various background details present in the second row. Notice also how
large regions with smooth as well as textured structures are recovered. The
quantization noise and color down-sampling by the CFA make the task even more
challenging resulting in some ringing artifacts on object edges. Although an
edgetaper [39] technique was used to limit such artifacts, few are still
present in the recovered images, but are significantly reduced when comparing
with the ones produced by the Wiener filter.

###  6.2 Experimental Results on Real Data

#### NYUV2 depth dataset

In order to compare our approach with state-of-the-art methods, the DEPTH-DNN
is trained on a subset of NYUV2 indoor dataset [30] in an end-to-end fashion
with the phase mask’s height map. In this experiment, only 10 depth planes are
considered in the layered depth presentation (Eq. 7) due to memory
constraints. The simulated camera lens parameters are the following: $f/4.0$
with $4$ mm aperture diameter and $16$ mm focal length focusing at a distance
of 5 meters. The RGB images are directly convolved with a RPSF cube of the
shape $10\times 23\times 23\times 3$. In the following evaluations, we applied
the same crop used in competing works (e.g. [31]) and excluded the invalid
pixels from the Kinect V1 sensor as done by all competing approaches.

Figure 8: Learned RPSF shape at different defocus planes (left), the height
map of the phase mask (right).

Fig. 8 shows the learned RPSF and the corresponding phase mask. For this
dataset, the learned RPSF is a double-helix $[N=2,\epsilon=0.99]$ with two
main side lobes that rotate with defocus. We argue that such behaviour is
mainly related to the characteristics of the training data: for more complex
depth scenes, like in this case, the network tends to converge to higher
number of peaks as it makes it easier to correlate between the rotation angle
and the corresponding depth plane. Similar to the previous scenario, the
network also converges towards a high confinement parameter $\epsilon=0.99$
producing more spread out peak regions.

Quantitative results on NYUV2 [31] test set are reported in Table.2. Coded-
aperture based competing approaches [17, 18] optimized a free-form phase mask
parameterized with a superposition of a set of Zernike polynomials [19, 20]
using a U-Net [21] architecture. Besides using a more accurate camera model,
differently from [17, 18], our approach learns only a few design parameters
for the mask and simultaneously tackles the problem of image quality
degradation. Table. 2 shows quantitative results for the different error
metrics used in [30]. Our approach outperforms the competing methods of [17,
18] in all but the last two accuracy metrics (where all top approaches
including ours are very close to 1 making them not too significant). Note that
for the more significant accuracy metric $\delta_{1}$ (i.e., with the lowest
threshold value), our approach achieves the highest score, even if we trained
the network using a subset of 50k training samples which is less than half of
the default split of 120k training samples [30] used by [17] and [18]. In
particular, we achieve a significantly lower RMSE value of 0.267 which is down
by respectively 0.166 and 0.115 from the ones achieved by [17] and [18] and is
the lowest yet achieved on this challenging dataset for the task of monocular
depth estimation.

This performance gain is primarily related to the optimized PSF shape, The one
obtained by [18] has a generic shape with no clear correlations between
different defocus planes, while the one obtained by [17] has an elliptical
shape with varying section which increases with depth. On the other hand, the
RPSF shape produces a clear and simple correlation between the defocus plane
and the corresponding angle of rotation of the main peaks, thus encoding
robust depth cues within input images. This, thanks also to the small number
of parameters to be learned, explain why the network achieves better depth
estimation accuracy with significantly less training data.

Method | RMSE $\downarrow$ | Rel $\downarrow$ | Log10 $\downarrow$ | $\delta_{1}\uparrow$ | $\delta_{2}\uparrow$ | $\delta_{3}\uparrow$
---|---|---|---|---|---|---
Eigen et al. [31] | 0.641 | 0.158 | - | 0.769 | 0.950 | 0.988
Laina et al. [42] | 0.573 | 0.127 | 0.055 | 0.811 | 0.953 | 0.988
Hao et al. [38] | 0.555 | 0.127 | 0.053 | 0.841 | 0.966 | 0.991
DORN [43] | 0.509 | 0.115 | 0.051 | 0.828 | 0.965 | 0.992
Qi et al. [44] | 0.569 | 0.128 | 0.057 | 0.834 | 0.960 | 0.990
Yin et al. [45] | 0.416 | 0.108 | 0.048 | 0.875 | 0.976 | 0.994
BTS [46] | 0.392 | 0.110 | 0.047 | 0.885 | 0.978 | 0.994
DAV [47] | 0.412 | 0.108 | - | 0.882 | 0.980 | 0.996
Alhashim et al. [48] | 0.382 | 0.093 | 0.050 | 0.932 | 0.989 | 0.997
AdaBins [32] | 0.364 | 0.103 | 0.044 | 0.903 | 0.984 | 0.997
DPT-Hybrid [49] | 0.357 | 0.110 | 0.045 | 0.904 | 0.988 | 0.998
DeepOptics [17] | 0.433 | 0.087 | 0.052 | 0.930 | 0.990 | 0.999
PhaseCam3D [18] | 0.382 | 0.093 | 0.050 | 0.932 | 0.989 | 0.997
Ours | 0.267 | 0.072 | 0.029 | 0.952 | 0.989 | 0.997

Table 2: Quantitative comparison with the state-of-the-art for monocular depth
estimation task on NYUV2 test set [31]. Figure 9: Qualitative results from the
proposed approach as well as those from AdaBins [32] on the test set of NYUV2
[31].

Table. 2 also presents quantitative results from the state-of-the-art that
mainly used sharp all-in-focus images as input. Our approach outperforms all
competing methods in all evaluation metrics except for $\delta_{3}$ accuracy
metric where all the top approaches including ours are extremely close. More
specifically, we achieve substantially lower error metrics compared to the
competing approaches which used similar or larger training sets, e.g. Eigen et
al. [31], Laina et al. [42], DORN [43], DAV [47], Alhashim et al. [48],
AdaBins [32] and DPT-Hybrid [49], notice that the latter used a pre-trained
model on a large combination of different datasets containing 1.4M samples and
fine-tuned it on NYUV2 dataset [31]. The rest of the competing approaches used
less training samples but at the cost of more complex models with pre-trained
weights, e.g. Hao et al. [38] used a ResNet [50] backbone pre-trained on
ImageNet [51]. Our model uses a smaller training set but the core network used
for depth estimation is a simple U-Net [21] architecture with approximately
8.6M trainable parameters. In contrast, AdaBins [32] for example, like most
other competing methods used a more complex network architecture with
approximately 78M trainable parameters, making it slower in both training and
inference. In our case, the RPSF encodes strong and robust depth cues making
it easier for a simple network to predict accurate depth maps compared to
those using conventional RGB inputs.

A qualitative comparison with AdaBins [32] (trained with the same 50k samples
as in this approach) is shown in Fig. 9 while further qualitative results are
shown in the supplementary materials. Upon visual inspection, one can easily
see that our approach produces more accurate and realistic depth maps with
respect to the ground truth (the ground truth depth maps are inpainted for
visualization purposes, in the supplementary material the ground truth data
used for evaluation are shown). Due to the scarcity of reliable depth cues in
single all-in-focus input images, Adabins [32] struggles to predict accurate
and sometimes realistic depth values in a consistent manner and fails to
predict correct values for images where depth values span large ranges, e.g.
the results shown in the two last columns in Fig. 9. Moreover, sometimes
Adabins [32] produces erroneous depth predictions where the scene semantics
are somehow confusing and the network fails to infer realistic values: such
behaviour exposes the main limitation of semantic-based approaches, as visible
in the last column of Fig. 9 where the green carpet was misclassified. In
contrast, our network consistently produces accurate depth maps for small and
large depth ranges alike and is more agnostic to the scene’s semantics. The
main drawback of our method is that it sometimes fails to learn well defined
object boundaries due to the blurring artifacts introduced by the RPSF kernel.

#### SUNRGBD dataset

We evaluate the generalization capability of our model for the task of
monocular depth estimation where DEPTH-DNN that was previously trained on the
subset of NYUV2 [30] is evaluated using the test set of SUNRGBD [33] without
any further fine-tuning. Quantitative and qualitative results are present in
Table. 3 and Fig. 10 (additional qualitative results are in the supplementary
material). Metric values for competitors shown in Table. 3 are taken from [32]
where methods with publicly available pre-trained models on NYUV2 [30] have
been evaluated on the SUNRGBD dataset.

Method | RMSE $\downarrow$ | Rel $\downarrow$ | Log10 $\downarrow$ | $\delta_{1}\uparrow$ | $\delta_{2}\uparrow$ | $\delta_{3}\uparrow$
---|---|---|---|---|---|---
Chen et al. [37] | 0.494 | 0.166 | 0.071 | 0.757 | 0.943 | 0.984
Yin et al. [45] | 0.541 | 0.183 | 0.082 | 0.696 | 0.912 | 0.973
BTS [46] | 0.515 | 0.172 | 0.075 | 0.740 | 0.933 | 0.980
AdaBins [32] | 0.476 | 0.159 | 0.068 | 0.771 | 0.944 | 0.983
Ours | 0.335 | 0.114 | 0.034 | 0.937 | 0.981 | 0.992

Table 3: Quantitative comparison with the state-of-the-art methods for
monocular depth estimation task on SUNRGBD [33] test set.

As shown in Table. 3, our approach outperforms the state-of-the-art in all
evaluation metrics with a significant reduction in error metrics, particularly
the RMSE (where it achieved $0.335$ compared to $0.476$ of the best
competitor) and Log10 ($0.034$ against $0.068$). Notice also the accuracy
metric $\delta_{1}$ corresponding to the smallest threshold of $1.25$ achieved
by our approach which is up by $0.166$ ($\delta_{1}=0.937$) compared to the
best performing approach where the accuracy value achieved is $0.771$, which
indicates that ours produces a higher percentage of accurately predicted depth
pixel values. Such results support the suitability of such engineered PSFs for
depth acquisition applications enabling reliable and robust passive monocular
depth estimation performance with real-time capabilities.

Figure 10: Qualitative results from our proposed method on the test set
SUNRGBD [33]

Fig. 10 shows some prediction samples from SUNRGBD [33] test set, as in the
previous case, the network is able to predict overall accurate depth maps but
with higher mean RMSE compared to the test set of NYUV2 test set [31] which is
expected due to the different statistical properties between the two datasets.

Figure 11: The generated RPSFs for the ablation experiments: (a) Single-helix
RPSF generated by the fixed mask 1 and its height map. (b) Double-helix RPSF
generated by the fixed mask 2 and its height map. (c) Single-helix RPSF
generated by the learned mask and its height map.

###  6.3 Ablation Study

A number of experiments were carried out as an ablation study to assess the
contribution of the various components in our framework. Quantitative results
are shown in Table. 4, while the qualitative ones are in the Supplementary
Material. In all simulations of the ablation study, the network architecture
as well as the training settings and hyper-parameters are the same as
indicated in the previous section except for the usage of a simple Gaussian
additive noise model instead of the full image formation procedure.
Furthermore, we set the number of Fresnel zones in the mask design to $L=5$.

Exp. number | Input | RMSE $\downarrow$
---|---|---
1 | All-in-focus | 2.649
2 | Fixed mask 1 $[N=1,L=5,\epsilon=0.5]$ | 1.117
3 | Fixed mask 2 $[N=2,L=5,\epsilon=0.5]$ | 0.815
4 | Learned mask $[L=5]$ | 0.699

Table 4: Quantitative results of the ablation experiments on the test set of
FlyingThings3D [29].

The baseline is a U-Net trained on all-in-focus sharp images from the subset
of FlyingThings3D [29]. The RMSE achieved in this first experiment is 2.649.
In the second experiment, a coded aperture with a fixed phase mask design
$[N=1,L=5,\epsilon=0.5]$ is used to blur the sharp input images with a depth
dependent single-helix RPSF. In fact, this particular mask design is the one
which was first introduced by [8]. The network trained with the fixed mask
achieves a RMSE value of 1.117, i.e. down by 1.532 with respect to the
baseline.

In the third experiment, the number of RPSF lobes is increased to $N=2$. Both
the number of Fresnel zones $L$ and the confinement parameter $\epsilon$ are
the same as in the second experiment. The network trained with such phase mask
achieves even better RMSE value of 0.815 compared to the 1.117 achieved by the
one in the previous experiment (see Table. 4). Such performance gain could be
due to the more discriminative shape of the double-helix RPSF compared to the
single-helix RPSF generated by the previous mask as the rotation can be easily
noticed from a depth plane to the other. As shown in Fig. 11, the RPSF
generated by the second mask has two main side peaks rotating counterclockwise
as a function of defocus, the same rotation aspect can be observed in the RPSF
shape generated by the first fixed mask (Fig. 11a) except that in this case
only one main side lobe is present. It suggest that, for a fixed phase mask, a
double-helix RPSF conveys more discriminative depth cues than a single-helix
one.

In the fourth and last experiment, the phase mask’s trainable parameters $N$
and $\epsilon$ are jointly optimized with the weights of the network, the
number of Fresnel-zones is fixed to $L=5$. As expected, the network was able
to outperform the baseline as well as the ones trained with fixed masks,
reaching a RMSE of 0.699. The learned phase mask parameters are
$[N=1,\epsilon=0.91]$ meaning that the RPSF (shown in Fig. 11c) has a single
side lobe that rotate with defocus. Notice that also the confinement parameter
$\epsilon$ is high resulting in a more spread out lobe compared to the one
generated by the fixed mask. It is therefore clear that a joint optimization
approach helps the network to effectively learn the correlations between the
rotation angle of the PSF and the corresponding depth plane leading to better
estimation accuracy.

##  7 Conclusion

In this paper we presented a novel computational camera model where an end-to-
end learning framework is proposed for the joint optimization of camera optics
and image processing algorithms for the tasks of monocular depth estimation
from diffracted rotation. The learned phase mask generates multi-order helix
rotating PSFs as a function of defocus, encoding strong depth cues within
single 2D images and enabling reliable and accurate depth estimation.
Experimental results confirmed the capability of the proposed model to
outperform existing methods in the task of monocular depth estimation and to
generalize well beyond the training environment. The depth estimation model
complexity is significantly reduced compared to the state-of-the-art due to
the 3D cues encoded by the RPSF, making it suitable for real-time applications
without compromising accuracy. Finally, the sharp all-in-focus images are also
recovered through a dedicated non-blind and non-uniform image deblurring
module.

Further research will focus on the fabrication of the phase mask via photo-
lithography which will be mounted on the back side of the camera’s aperture,
thus adding depth estimation capabilities to a standard RGB cameras.

## References

  * [1] A. Greengard, Y. Y. Schechner, R. Piestun, Depth from diffracted rotation, Optics letters 31 (2) (2006) 181–183.
  * [2] L. Allen, M. W. Beijersbergen, R. Spreeuw, J. Woerdman, Orbital angular momentum of light and the transformation of laguerre-gaussian laser modes, Physical review A 45 (11) (1992) 8185.
  * [3] R. Piestun, Y. Y. Schechner, J. Shamir, Propagation-invariant wave fields with finite energy, JOSA A 17 (2) (2000) 294–303.
  * [4] S. R. P. Pavani, R. Piestun, High-efficiency rotating point spread functions, Optics express 16 (5) (2008) 3484–3489.
  * [5] Y. Shechtman, S. J. Sahl, A. S. Backer, W. Moerner, Optimal point spread function design for 3d imaging, Physical review letters 113 (13) (2014) 133902\.
  * [6] V. V. Kotlyar, A. A. Almazov, S. N. Khonina, V. A. Soifer, H. Elfstrom, J. Turunen, Generation of phase singularity through diffracting a plane or gaussian beam by a spiral phase plate, JOSA A 22 (5) (2005) 849–861.
  * [7] V. V. Kotlyar, A. A. Kovalev, S. N. Khonina, R. V. Skidanov, V. A. Soifer, H. Elfstrom, N. Tossavainen, J. Turunen, Diffraction of conic and gaussian beams by a spiral phase plate, Applied optics 45 (12) (2006) 2656–2665.
  * [8] S. Prasad, Rotating point spread function via pupil-phase engineering, Optics letters 38 (4) (2013) 585–587.
  * [9] R. Berlich, S. Stallinga, High-order-helix point spread functions for monocular three-dimensional imaging with superior aberration robustness, Optics express 26 (4) (2018) 4873–4891.
  * [10] R. Kumar, Three-Dimensional Imaging using a Novel Rotating Point Spread Function Imager, The University of New Mexico, 2015.
  * [11] A. Levin, R. Fergus, F. Durand, W. T. Freeman, Image and depth from a conventional camera with a coded aperture, ACM transactions on graphics (TOG) 26 (3) (2007) 70–es.
  * [12] C. Zhou, S. Lin, S. K. Nayar, Coded aperture pairs for depth from defocus and defocus deblurring, International journal of computer vision 93 (1) (2011) 53–72.
  * [13] R. Kumar, S. Prasad, Psf rotation with changing defocus and applications to 3d imaging for space situational awareness, in: Proceedings of the 2013 AMOS Technical Conference, Maui, HI, 2013.
  * [14] C. Roider, A. Jesacher, S. Bernet, M. Ritsch-Marte, Axial super-localisation using rotating point spread functions shaped by polarisation-dependent phase modulation, Optics express 22 (4) (2014) 4029–4037.
  * [15] R. Berlich, A. Bräuer, S. Stallinga, Single shot three-dimensional imaging using an engineered point spread function, Optics express 24 (6) (2016) 5946–5960.
  * [16] H. Haim, S. Elmalem, R. Giryes, A. M. Bronstein, E. Marom, Depth estimation from a single image using deep learned phase coded mask, IEEE Transactions on Computational Imaging 4 (3) (2018) 298–310.
  * [17] J. Chang, G. Wetzstein, Deep optics for monocular depth estimation and 3d object detection, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 10193–10202.
  * [18] Y. Wu, V. Boominathan, H. Chen, A. Sankaranarayanan, A. Veeraraghavan, Phasecam3d—learning phase masks for passive single view depth estimation, in: 2019 IEEE International Conference on Computational Photography (ICCP), IEEE, 2019, pp. 1–12.
  * [19] D. R. Iskander, M. J. Collins, B. Davis, Optimal modeling of corneal surfaces with zernike polynomials, IEEE Transactions on biomedical engineering 48 (1) (2001) 87–95.
  * [20] M. Born, E. Wolf, Principles of optics: electromagnetic theory of propagation, interference and diffraction of light, Elsevier, 2013.
  * [21] O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical image computing and computer-assisted intervention, Springer, 2015, pp. 234–241.
  * [22] J. Anger, G. Facciolo, M. Delbracio, Modeling realistic degradations in non-blind deconvolution, in: 2018 25th IEEE International Conference on Image Processing (ICIP), IEEE, 2018, pp. 978–982.
  * [23] S. Cho, S. Lee, Fast motion deblurring, in: ACM SIGGRAPH Asia 2009 papers, 2009, pp. 1–8.
  * [24] D. Gong, Z. Zhang, Q. Shi, A. van den Hengel, C. Shen, Y. Zhang, Learning deep gradient descent optimization for image deconvolution, IEEE transactions on neural networks and learning systems 31 (12) (2020) 5468–5482.
  * [25] D. Krishnan, R. Fergus, Fast image deconvolution using hyper-laplacian priors, Advances in neural information processing systems 22 (2009) 1033–1041.
  * [26] L. Xu, J. S. Ren, C. Liu, J. Jia, Deep convolutional neural network for image deconvolution, Advances in neural information processing systems 27 (2014) 1790–1798.
  * [27] C. J. Schuler, H. Christopher Burger, S. Harmeling, B. Scholkopf, A machine learning approach for non-blind image deconvolution, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 1067–1074.
  * [28] J. Dong, S. Roth, B. Schiele, Deep wiener deconvolution: Wiener meets deep learning for image deblurring, arXiv preprint arXiv:2103.09962.
  * [29] N. Mayer, E. Ilg, P. Häusser, P. Fischer, D. Cremers, A. Dosovitskiy, T. Brox, A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation, in: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2016, arXiv:1512.02134.
URL http://lmb.informatik.uni-freiburg.de/Publications/2016/MIFDB16

  * [30] D. Eigen, C. Puhrsch, R. Fergus, Depth map prediction from a single image using a multi-scale deep network, arXiv preprint arXiv:1406.2283.
  * [31] D. Eigen, R. Fergus, Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 2650–2658.
  * [32] S. F. Bhat, I. Alhashim, P. Wonka, Adabins: Depth estimation using adaptive bins, arXiv preprint arXiv:2011.14141.
  * [33] S. Song, S. P. Lichtenberg, J. Xiao, Sun rgb-d: A rgb-d scene understanding benchmark suite, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 567–576.
  * [34] H. S. Malvar, L.-w. He, R. Cutler, High-quality linear interpolation for demosaicing of bayer-patterned color images, in: 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol. 3, IEEE, 2004, pp. iii–485.
  * [35] M. A. et al., TensorFlow: Large-scale machine learning on heterogeneous systems, software available from tensorflow.org (2015).
URL http://tensorflow.org/

  * [36] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980.
  * [37] X. Chen, X. Chen, Z.-J. Zha, Structure-aware residual pyramid network for monocular depth estimation, arXiv preprint arXiv:1907.06023.
  * [38] Z. Hao, Y. Li, S. You, F. Lu, Detail preserving depth estimation from a single image using attention guided networks, in: 2018 International Conference on 3D Vision (3DV), IEEE, 2018, pp. 304–313.
  * [39] S. J. Reeves, Fast image restoration without boundary artifacts, IEEE Transactions on image processing 14 (10) (2005) 1448–1453.
  * [40] M. Kraus, M. Strengert, Depth-of-field rendering by pyramidal image processing, in: Computer graphics forum, Vol. 26, Wiley Online Library, 2007, pp. 645–654.
  * [41] N. Wiener, et al., Extrapolation, interpolation, and smoothing of stationary time series: with engineering applications, Vol. 8, MIT press Cambridge, MA, 1964\.
  * [42] I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, N. Navab, Deeper depth prediction with fully convolutional residual networks, in: 2016 Fourth international conference on 3D vision (3DV), IEEE, 2016, pp. 239–248.
  * [43] H. Fu, M. Gong, C. Wang, K. Batmanghelich, D. Tao, Deep ordinal regression network for monocular depth estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2002–2011.
  * [44] X. Qi, R. Liao, Z. Liu, R. Urtasun, J. Jia, Geonet: Geometric neural network for joint depth and surface normal estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 283–291.
  * [45] W. Yin, Y. Liu, C. Shen, Y. Yan, Enforcing geometric constraints of virtual normal for depth prediction, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 5684–5693.
  * [46] J. H. Lee, M.-K. Han, D. W. Ko, I. H. Suh, From big to small: Multi-scale local planar guidance for monocular depth estimation, arXiv preprint arXiv:1907.10326.
  * [47] L. Huynh, P. Nguyen-Ha, J. Matas, E. Rahtu, J. Heikkilä, Guiding monocular depth estimation using depth-attention volume, in: European Conference on Computer Vision, Springer, 2020, pp. 581–597.
  * [48] I. Alhashim, P. Wonka, High quality monocular depth estimation via transfer learning, arXiv preprint arXiv:1812.11941.
  * [49] R. Ranftl, A. Bochkovskiy, V. Koltun, Vision transformers for dense prediction, arXiv preprint arXiv:2103.13413.
  * [50] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [51] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255.
  * [52] J. W. Goodman, Introduction to Fourier optics, Roberts and Company Publishers, 2005\.

Summary

##

##  1 Out of Focus Imaging

Consider a simple imaging system consisting of a thin lens with focal length
$f$. The light reflected from an object at distance $z_{o}$ in front of the
lens is focused into an image plane at a distance $z_{i}$ behind it, i.e., all
rays coming from a point-like object are focused into a point in the image
plane that satisfy the well-known thin lens equation
$(\frac{1}{z_{o}}+\frac{1}{z_{i}}=\frac{1}{f})$.

Away from the in-focus plane, where the thin lens equation is no longer
satisfied, a quadratic phase error at the pupil plane is introduced to model
OOF objects. Such error is measured by the defocus value [52]:

$\Psi=\frac{\pi R^{2}}{\lambda}(\frac{1}{z_{o}}+\frac{1}{z_{i}}-\frac{1}{f})$
(12)

Where $R$ is the pupil radius and $\lambda$ is the wavelength of incident
light waves. $\Psi$ indicates the severity of the focusing error and it
increases in absolute value as objects move away from the in-focus plane.

##  2 Field Propagation and the Point Spread Function

In this section, we discuss how a Fresnel diffraction model can be used to
simulate near-field wave propagation. Consider a thin lens with a thickness
profile $h$ situated at the pupil plane of an imaging system (as depicted in
Fig. 12), notice that $h$ is a function of the spatial coordinates $(x,y)$,
and let $h_{0}$ be the thickest section of the lens. Generally the lens is
made of glass with a refractive index $n=1.5$. For a convex thin lens with
focal length $f$, $h(x,y)$ is defined as:

$h(x,y)=h_{0}-\frac{x^{2}+y^{2}}{2f(n-1)}$ (13)

Figure 12: Light emitted by an object propagates through free space and its
phase is delayed when it passes through the camera optics.

Light wave fields passing through the lens are delayed by an amount
proportional to the thickness of the lens $h$ at each point. Thus, a thin lens
applies a phase shift to the incident wavefront given by:

$\Phi_{lens}(x,y)=\frac{2\pi(n-1)}{\lambda}h(x,y)$ (14)

Notice that the phase shift is wavelength-dependent hence using the same lens
with polychromatic light results in chromatic aberrations.

By defining the phase shift at the pupil plane, the generalized pupil
function, which is a complex function, can be written as:

$P(x,y)=A(x,y)e^{i\Phi_{lens}(x,y)}$ (15)

Where $A(x,y)$ is the circular aperture mask simulating a finite aperture
area.

In the case of OOF imaging, the pupil function would also introduce a phase
error expressed as a quadratic phase term in the generalized pupil function
showing the effects of defocus aberrations in the captured image. The new
generalized pupil function would have the following form:

$P(x,y)=A(x,y)e^{i(\Phi_{lens}(x,y)+\Psi\frac{(x^{2}+y^{2})}{R^{2}})}$ (16)

Where $R$ is the pupil radius and $\Psi$ is the defocus parameter. In the case
of a wave field generated by an ideal point source object, the PSF of the
imaging system is expressed as:

$PSF(x,y)\propto\lvert\mathscr{F}(P(x,y))\rvert^{2}$ (17)

Where $\mathscr{F}$ denotes the Fourier Transform.

Concerning the wave field, let the complex wave field $U_{in}$ propagating in
free space just before the entrance pupil be:

$U_{in}(x,y)=C(x,y)e^{i\Phi_{in}(x,y)}$ (18)

Where $C(x,y)$ is the field’s amplitude and $\Phi_{in}$ is its phase just
before entering the pupil.

In the presence of a phase mask with a phase delay $\Phi_{mask}$ (as shown in
Fig. 12), the phase delay introduced by the combination of both the lens and
mask is expressed as:

$\Phi_{optics}(x,y)=\Phi_{lens}(x,y)+\Phi_{mask}(x,y)$ (19)

Thus, the expression of the wave field just after the lens system $U_{out}$ in
this case is:

$\begin{split}U_{out}(x,y)&=P(x,y)U_{in}(x,y)\\\
&=A(x,y)C(x,y)e^{i(\Phi_{in}(x,y)+\Phi_{optics}(x,y))}\end{split}$ (20)

Using Fresnel propagation, $U_{out}$ can be further propagated of a distance
$z_{i}$ until it reaches the image sensor (we use a coordinate system $(u,v)$
for the sensor).

$\displaystyle U_{sensor}(u,v)=\frac{e^{ikz_{i}}}{i\lambda
z_{i}}\int_{-\infty}^{+\infty}\int_{-\infty}^{+\infty}U_{out}(x,y)\times{}$
(21) $\displaystyle e^{\frac{ik}{2z_{i}}[(u-x)^{2}+(v-y)^{2}]}\,dx\,dy$

Where $k=\frac{2\pi}{\lambda}$ is the wave number.

##  3 Visual Evaluation of the Impact of RPSF Design Parameters

In this section we show some visual examples of RPSF shapes obtained by using
different values of the design parameters.

Recall that we denoted with $N$ the number of rotating peaks within the RPSF:
in the first row of Fig. 13 we show the resulting RPSF shape for different
values of $N$.

The design parameter $\epsilon$ controls the peak separation as well as the
confinement of each peak as illustrated in the second row of Fig. 13. It was
empirically hypothesized by [9] that the depth range increases as $\epsilon$
decreases.

The number of Fresnel zones $L$ within the phase mask also controls the peak
separation as illustrated in the last row of Fig. 13. Additionally, the rate
of rotation which is $\frac{1}{L}$ in the basic case of a single helix RPSF,
can be easily extended to $\frac{1}{NL}$ in the more general case of multiple
peaks. This indicates that the rate of rotation would decrease with higher $L$
and/or $N$ values leading to an increase in the practical depth range with no
peak rotation ambiguity.

In summary, $[N,L,\epsilon]$ can be jointly optimized depending on the target
task: they all influence the shape of the RPSF and the practical depth range
in which the PSF can be used to encode unambiguous depth information within
the captured 2D images.

Figure 13: The effects of $[N,L,\epsilon]$ design parameters on the shape of
the RPSF: (first row) only the value of $N$ is changed, i.e.,
$[N=1,2,3,4,5;L=5;\epsilon=0.9]$, (second row) only the value of $\epsilon$ is
changed, i.e., $[N=2;L=5;\epsilon=0.1,0.3,0.5,0.8,1]$, (third row) only the
value of $L$ is changed with $[N=2;L=2,3,5,7,10;\epsilon=0.9]$.

##  4 Additional Visual Results for Image Deblurring

Fig. 14 shows some additional examples of the proposed image deblurring
algorithm on the FlyingThings3D subset [29]: the network achieves overall
satisfactory reconstruction with minor artifacts (a bit of ringing and a few
texture patterns not properly recovered are visible upon closer inspection).
Some challenging complex structures are restored (e.g. the sample shown in the
third row) while other fine details, such as the periodic pattern of the
“floor” shown in the sample of the last row, proved difficult to be accurately
recovered.

Figure 14: Additional qualitative results of the image deblurring model on the
test set of FlyingThings3D [29] subset. Figure 15: Additional qualitative
results on RPSF-blurred images from the test set of FlyingThings3D [29]
subset.

##  5 Additional Visual Results for Monocular Depth Estimation

Fig. 15 shows some additional examples of the proposed depth estimation
algorithm on sample images from the FlyingThings3D test set [29]. These
results correspond to the simulated noisy input images where sensor noise,
CFA, and quantization noise are added. The network was able to produce
accurate disparity maps even for objects with very fine details (e.g. the
motorcycles in the second and third columns in addition to the plant in the
last column). On the other side, edge pixels are not always accurately
estimated (e.g., see the fourth column) as discussed in Section VI in the
paper.

Then, Fig. 16 shows some additional examples of the proposed depth estimation
algorithm on sample images from the NYUV2 test set [31] and compares them with
those from AdaBins [32] competing approach. Our approach proved capable of
producing superior results for small and large depths ranges alike with no
effects or ambiguities whatsoever originating from the scene’s semantics.

Finally, Fig. 17 shows some additional examples of the proposed depth
estimation algorithm on sample images from the SUNRGBD test set [33]. Even
though the network was not fine-tuned for this specific dataset, it performed
very well on this larger and challenging dataset.

Figure 16: Additional qualitative results from the proposed approach as well
as those from AdaBins [32] on the test set of NYUV2 [31]. Figure 17:
Additional qualitative results from our proposed method on the test set of
SUNRGBD [33]

##  6 Qualitative Results for the Ablation Study

Fig. 18 shows a sample of the predicted depth maps for the four conducted
ablation experiments discussed in the paper. Quantitatively, the network
trained using all-in-focus images as input fails to discriminate between
positive and negative defocus values, i.e., it is very difficult for the
network to accurately predict whether objects are close or far from the camera
because of the lack of reliable depth cues within the clear aperture PSF.
While the rest of the performed experiments significantly improve the depth
estimation accuracy, the learned mask proved capable not only to predict
accurate depth maps but also to preserve fine details around object edges at
different depth planes as shown, for example, in Fig. 18 around the plant
edges.

Figure 18: A sample from the qualitative results for the four conducted
ablation experiments on the test set of the subset of FlyingThings3D [29].