# The widest contiguous field of view at Dome C and Mount Graham

Jeff Stoesz Corresponding author address: Jeff Stoesz, INAF - Osservatorio di
Arcetri, Largo Enrico Fermi 5, Firenze, FI 50125, Italy.
E-mail<EMAIL_ADDRESS>Elena Masciadri  Franck Lascaux  Susanna
Hagelin
INAF - Osservatorio Astrofisica di Arcetri, Florence, Italy

###### Abstract

The image quality from Ground-Layer Adaptive Optics (GLAO) can be gradually
increased with decreased contiguous field of view. This trade-off is dependent
on the vertical profile of the optical turbulence ($C_{n}^{2}$ profiles). It
is known that the accuracy of the vertical distribution measured by existing
$C_{n}^{2}$ profiling techniques is currently quite uncertain for wide field
performance predictions 4 to 20 arcminutes. With assumed uncertainties in
measurements from Generalized-SCIDAR (GS), SODAR plus MASS we quantify the
impact of this uncertainty on the trade-off between field of view and image
quality for photometry of science targets at the resolution limit. We use a
point spread function (PSF) model defined analytically in the spatial
frequency domain to compute the relevant photometry figure of merit at
infrared wavelengths. Statistics of this PSF analysis on a database of
$C_{n}^{2}$ measurements are presented for Mt. Graham, Arizona and Dome C,
Antarctica. This research is part of the activities of ForOT (3D Forecasting
of Optical Turbulence above astronomical sites).

##  1 Introduction

Characterization of the optical turbulence in the first few kilometres above
the telescope is important for predicting the performance of Ground-Layer
Adaptive Optics (GLAO) telescopes as a function of field of view diameter.
Systems that have been proposed will correct visible or near-infrared science
fields that are typically 4 arcminutes, and potentially up to 20 arcminutes in
diameter and contiguous. There are several measurement techniques being
advanced to provide statistics on the vertical distribution of the structure
function coefficient $C_{n}^{2}(h)$ , and in this paper we explore the impact
of a potential bias from generalized-SCIDAR and MASS measurements. The first
of two sites we will investigate is a typical mid-latitude observatory site,
Mount Graham (32.7 N, 109.87 W, 3200 meters), measured with generalized-
SCIDAR. There are conifer trees at the summit with a height similar to the
SCIDAR telescope’s primary mirror, about 8 meters above the ground. The second
is Dome C (75.1 S, 123.3 E, 3260 meters), an Antarctic site with MASS and
SODAR measurements by Lawrence et al. (2004) and balloon measurements by Agabi
et al. (2006).

The GLAO PSF figure of merit that is of particular importance to wide field
astronomy is radius of 50% encircled energy, computed at several points in the
contiguous field of view and then averaged. It will be symbolized as $EE50$
here. $EE50$ is very closely related to the integration time to achieve some
signal to noise ratio in background-limited point source photometry in the
field (Andersen et al. 2006), a rather common science application for fields
of view 4 to 20 arcminutes in diameter. Roughly,

${\rm integration~{}time}\propto EE50^{2}.$ (1)

We will compute $EE50$ starting with an analytically defined phase Power
Spectral Density (PSD) for anisoplanatism and fitting error using established
theory (Jolissaint et al. 2006; Tokovinin 2004). Table 1 lists the model
parameters selected here. Computation from the analytic PSD is a fast method
to discover the performance gradient of $EE50(\theta)$, where $\theta$ is the
diameter of the field of view.

Table 1: The parameters and implicit assumptions of the GLAO PSF model. phase PSD | von Kármán, $L_{o}=30$ meters
---|---
telescope diameter |  $D=8$ meters
Beacons | 4 point sources at range $H=90$ km at zenith
Beacons | evenly distributed on a circle of diameter $\theta$ in the field
image wavelength | $\lambda=1.25\mu m$
image locations | sampling a square field of view with vertices that intersect the circle
Deformable Mirror | cartesian grid of actuators with pitch, $\Delta$
Deformable Mirror | each actuator has a sinc-like influence function
Deformable Mirror | conjugated to height = 0

The exact range of altitudes in the first few kilometres where bias has
greatest impact depends on the basic GLAO system parameters, namely the
diameter of the guide star asterism (also $\theta$) whose signal is averaged
and the effective pitch that is controlled by the ground conjugated deformable
mirror ($\Delta$). The ratio $h_{GZ}=\Delta/\theta$ defines the altitude below
which any contribution to anisoplanatism is negligible. The term gray-zone
(GZ) was coined (Tokovinin 2004) to identify the altitudes above $h_{GZ}$,
where the contribution to anisoplanatism is not negligible (also known as
partially corrected zone).111Looking at the approximate error transfer
function in equation (8) of Tokovinin (2004) one can see why this is the case.
Fig.1 helps illustrate this in terms of performance in the focal plane. The
plot shows the $EE50$ figure of merit as a function of the height of one layer
of turbulence added to a typical, smooth profile. The layer contains half of
the total turbulence strength of the smooth profile. Fig.1 shows that the
largest performance gradient is at altitudes just above $h_{GZ}$. The gradient
vanishes above $h_{D}=D/\theta$, where $D$ is the telescope diameter. In the
following sections we will re-compute $EE50(\theta)$ with estimated bias in
the proportion of turbulence attributed to heights above or below $h_{GZ}$.

Figure 1: The gray-zone begins above $h_{GZ}$.

##  2 Mount Graham and Dome C profile monitoring data

The Mt. Graham G-SCIDAR measurements include 851 in High Vertical Resolution
(HVR) mode and 9911 in regular mode, both have been reduced to discretized
turbulence strength $J_{i}$ at height $h_{i}$. These were computed from the
normalized covariance function of the irradiance fluctuations (see Egner et
al. 2006, 2007) which are proportional to $J_{i}$, which are in turn related
to $C_{n}^{2}(h)$ by

$J_{i}=\int_{{h_{b}}_{i}}^{{h_{b}}_{i+1}}dh~{}C_{n}^{2}(h).$ (2)

The intrinsic vertical resolution of SCIDAR is roughly given by

$\frac{0.78}{\rho}\sqrt{\lambda|h+h_{gs}|}$ (3)

where $\rho$ is the binary separation ($35^{\prime\prime}$), $\lambda$ is the
wavelength of the scintillation signal ($0.5\mu m$), and $h_{gs}$ is the
conjugation height of the generalized SCIDAR analysis plane (about $-3500m$).
The regular mode resolution will represent free-atmosphere, above 1000 meters.
The current HVR data set samples the scale height of the boundary-layer and
provides data up to 1000 meters altitude. In a subsequent section we will
describe how the ground-layer and free-atmosphere are reduced to form a
composite statistical model.

For Dome C we will use 1701 MASS+SODAR profile monitoring measurements at Dome
C by Lawrence et al. (2004) during the Antarctic winter of 2004. These data
sample only two grid points between 30 and 1000 meters and do not sample any
turbulence below 30 meters. However, there exist balloon-borne micro-thermal
measurements (Agabi et al. 2006) that give us an estimate of the scale height
and total strength of the ground-layer, and with this information we model the
statistics of eight grid points from a height of zero to 200 meters. The
turbulence measurements recorded by SODAR in the Lawrence et al. (2004) data
we appropriate to a slab concentrated at 250 meters between the modelled
ground layer and the lowest MASS measurement at 500 meters.

For the Dome C altitudes from zero to 200 meters we define the following
exponential model to

$C_{n}^{2}(h)=Ae^{(-h/h_{A})}.$ (4)

Using Eqn.(2) it follows that

$J_{i}=-Ah_{A}\left(e^{(-{h_{b}}_{i+1}/h_{A})}-e^{(-{h_{b}}_{i}/h_{A})}\right).$
(5)

We will choose the boundaries ${h_{b}}_{i}$ in §4. Using a average, weighted
by $C_{n}^{2}(h)$

$\displaystyle h_{i}=$
$\displaystyle\frac{\int_{{h_{b}}_{i}}^{{h_{b}}_{i+1}}dh~{}C_{n}^{2}(h)~{}h}{\int_{{h_{b}}_{i}}^{{h_{b}}_{i+1}}dh~{}C_{n}^{2}(h)}.$
$\displaystyle=$
$\displaystyle\frac{-Ah_{A}\left[({h_{b}}_{i+1}+h_{A})e^{(-{h_{b}}_{i+1}/h_{A})}-({h_{b}}_{i}+h_{A})e^{(-{h_{b}}_{i}/h_{A})}\right]}{J_{i}}.$
(6)

It has been observed with balloon measurements at Cerro Pachon (Tokovinin and
Travouillon 2006) that the strength of ground-layer is governed primarily by
the scale height. In our model we will make the scale height dictate the
strength exclusively. A lognormal distribution of values of the scale height,
$h_{A}$, while $A=740.\times 10^{-16}$ and is fixed, will give a lognormal
distribution in seeing.

The Mt. Graham (MG) scenario has weaker overall seeing (median 0.74
arcseconds) than Dome C (DC, median 1.2 arcseconds). To illustrate the
differences in the vertical distributions for these two sites we reduce the
data to cumulative histograms of seeing in three slabs, shown in Fig.2. The
Dome C free atmosphere (right panel) and even upper ground-layer slab
(middle)are quite calm. Though the left and middle panels of Fig.2 are not
proof, the scale height of the MG turbulence is resolved by the HV-GS
technique in another analysis (Egner et al. 2006) to be between 100 to 250
meters. The DC scenario clearly has most turbulence concentrated between the
telescope and 30 meters range (left panel Fig.2).

Figure 2: Comparison of the Dome C (DC) and Mount Graham (MG) turbulence
profile data used here.

##  3 Reduction to composite profiles

Since the measurements of the ground-layer and free-atmosphere at these sites
is not simultaneous, we must create composite profiles that would closely
reproduce the PSF statistics as though we had computed them on a full set of
$J_{i}(h_{i})$ data, uninterrupted in $h$ and sampled at the same time. To do
this we sort and combine the profiles of as described in Tokovinin and
Travouillon (2006) using the assumption of uncorrelated ground-layer and free-
atmosphere seeing. We will briefly re-describe the process here in the context
of our data.

The Mt. Graham HVR will provide the ground-layer below 1000 meters and the
regular SCIDAR measurements will provide the free-atmosphere above 1000
meters. Three groups of profiles in the ground-layer are identified using the
sum of $J_{i}$. The first group are those profiles within $5\%$ of the
$25^{th}$ percentile are combined in a simple average for $J_{i}$. We call
them the “good” case. The $50^{th}$ and $75^{th}$ percentile profiles area
combined similarly and called “typical” and “bad”. In each group the grid of
$h_{i}$ is identical and hence remains unchanged by the combining process. The
same process is done for the free-atmosphere. The result is a reduction to
three ground layer profiles and three free-atmosphere profiles, which together
have nine permutations for composite profiles that can reproduce the PSF
statistics as though we had computed them on all of the $J_{i}(h_{i})$ data.

For Dome C we sort and combine the MASS+SODAR profile monitoring measurements
of the free-atmosphere above 200 meters in the same way we described for Mt.
Graham. The ground-layer model does not need to be sorted; the choice of three
scale heights $h_{A}=[14,9,22]$ meters provide the median, first and last
quartile of the integrated ground-layer.

##  4 Resampling the Composite Profiles

In all cases the shape of the composite profiles, whether averaged over time
or defined by a function is smooth and well sampled by the grid of
$J_{i}(h_{i})$ defined so far. Hence, we are permitted to resample the the
$J_{i}(h_{i})$ grid for the GLAO PSF model, which is affected by the density
of points in the gray-zone. We increase the number of grid points in the gray-
zone until the PSF figure of merit has reached an asymptote. This is trivial
for the ground-layer of Dome C, we can define the $h_{b}$ grid and then re-
compute $J_{i}(h_{i})$ with Eqn.(5) and Eqn.(6). For the measurements of Mount
Graham and the free-atmosphere of Dome C we divide several measured
$J_{i}(h_{i})$ grid into more numerous $J_{j}(h_{j})$ using linear
interpolation of the original discretized $C_{n}^{2}(h)$ data.

##  5 Predicted GLAO performance gradient

The reduced composite $C_{n}^{2}(h)$ profiles for each site are input for the
computation of field averaged radius of 50% encircled energy of PSFs at a
wavelength of $1.25\mu m$, outlined in §1, and symbolized $EE50$. The aim is
to asses the impact on GLAO performance by potential biases in the measured
vertical distribution of the turbulence strength. We have selected the
performance $EE50(\theta)$ metric to do this. Fig.3 is a 3x3 multi-panel plot
showing $EE50(\theta)$ at Mt. Graham (red) and Dome C (blue). The thicker
lines are the median values while the thinner ones are the first and last
quartiles of the ordinate.

Figure 3: The field averaged radius of 50% encircled energy on PSFs at
$1.25\mu m$, plotted as a function of the GLAO field of view.

Let us first consider the central column of plots to identify the fundamental
differences between weak and strong free-atmosphere sites. In the upper one we
see the Mt. Graham (red) $EE50$ gracefully increasing with $\theta$, as the
bottom of the gray-zone (§1) reaches into the boundary-layer turbulence 100 to
250 meters thick. For this top middle panel the actuator pitch of the DM was
0.5 meters and the Dome C scenario only very weakly affected by
anisoplanatism, a consequence of an inadequate number of actuators for that
site. In the central panel the pitch is 0.38 meters, which improves correction
at Mt. Graham slightly in all conditions, and greatly improves Dome C for
median or better conditions. The median and first quartile $EE50(\theta)$
curves of Dome C and Mt. Graham have similar shape because the ground-layer
profiles at Mt. Graham have similar exponential shape. The bottom plot shows
the potential gain for Dome C when the wavefront is controlled to a pitch of
0.1 meters. In the central column of plots, the important distinction between
the two sites is that Dome C is always under-actuated with $\Delta=0.5$ and
sometimes near the diffraction-limited $EE50$ with $\Delta=0.1$. Mt. Graham on
the other hand has more high altitude turbulence and is always limited by
anisoplanatism for these $\Delta$.

Next, consider the columns of panels to the left and right of Fig.3 showing
uncertainties pertinent to field of view trade-offs in GLAO telescope
design.As indicated in figure 4 in Tokovinin et al. (2005) both MASS and
SCIDAR measurements are believed to produce faithful total integrals of
turbulence, however, the vertical distribution may be biased. The left column
of plots in Fig.3 were computed from the $J_{i}(h_{i})$ times 0.5 in the
domain $h_{gz}<h_{i}<6km$, the balance was conserved by putting turbulence in
the lowest layer, below $h_{GZ}$. Likewise the the right column of plots is
$J_{i}(h_{i})$ times 1.5 in the domain $h_{gz}<h_{i}<6km$, with the balance
conserved by removing turbulence from the lowest layer. The change from the
central column of plots to the left or the right is the slope of the curves,
germane to designing a field of view trade-off. The performance of a wide
field survey can be expressed using the number of square arcminutes of sky
that can be imaged to some limiting magnitude per unit time. For an
theoretical seeing-limited telescope this is of course proportional to
$\theta^{2}$. For a GLAO telescope with field of view $\theta$ it will be
roughly proportional to $(\theta/EE50(\theta))^{2}$. $EE50(\theta)$ in the
middle row of Fig.3 ($\Delta=0.38$ meters) the slope of the median Mt. Graham
$EE50(\theta)$ in the domain $10<\theta<20$ arcminutes is about $45\%$ less or
more in the left or right panels. It is about $\mp 15\%$ for Dome C. In terms
of $integration~{}time(\theta)\propto EE50(\theta)^{2}$ in the domain
$10<\theta<20$ we find the slope is $\pm 60\%$ for Mt. Graham, $\pm 30\%$ for
Dome C. In other words, at a mid-latitude site similar to Mt. Graham, the
predicted survey coverage of the GLAO telescope could potentially be wrong by
as much as 60%.

##  6 Summary

The GLAO telescope scenario simulated here is a common design for wide field
science demanding a contiguous field. The estimate of 50% uncertainty in the
proportion of turbulence strength between the the corrected-zone and the gray-
zone (in the first 6 $km$) is based on a comparison between MASS and SCIDAR
and here we calculate an uncertainty of 60% in the slope function
$EE50(\theta)$. Dome C is truly a unique site, and more immune to the 50%
uncertainty. However, if the true uncertainly is not simply multiplicative the
uncertainty propagated to $EE50(\theta)$ for Dome C might be similar to that
of Mt. Graham.

_Acknowledgements._

We would like to thank the authors of Lawrence et al. (2004) for providing
their SODAR+MASS data. This work has been funded by the Marie Curie Excellence
Grant (ForOT)-MEXT-CT-2005-023878.

##  References

  * Agabi et al. (2006) Agabi, A., E. Aristidi, M. Azouit, E. Fossat, F. Martin, T. Sadibekova, J. Vernin, and A. Ziad, 2006: First Whole Atmosphere Nighttime Seeing Measurements at Dome C, Antarctica. PASP, 118, 344–348, doi:10.1086/498728, arXiv:astro-ph/0510418.
  * Andersen et al. (2006) Andersen, D. R., J. Stoesz, S. Morris, M. Lloyd-Hart, D. Crampton, T. Butterley, B. Ellerbroek, L. Jolissaint, N. M. Milton, R. Myers, K. Szeto, A. Tokovinin, J.-P. Véran, and R. Wilson, 2006: Performance Modeling of a Wide-Field Ground-Layer Adaptive Optics System. PASP, 118, 1574–1590, doi:10.1086/509266, arXiv:astro-ph/0610097.
  * Egner et al. (2007) Egner, S. E., E. Masciadri, and D. McKenna, 2007: Generalized SCIDAR measurements at Mt. Graham. PASP accepted.
  * Egner et al. (2006) Egner, S. E., E. Masciadri, D. McKenna, and T. M. Herbst, 2006: Beyond conventional G-SCIDAR: the ground-layer in high vertical resolution. Advances in Adaptive Optics II. Edited by Ellerbroek, Brent L.; Bonaccini Calia, Domenico. Proceedings of the SPIE, Volume 6272, pp. 627256 (2006)., Presented at the Society of Photo-Optical Instrumentation Engineers (SPIE) Conference, Vol. 6272, doi:10.1117/12.671380.
  * Jolissaint et al. (2006) Jolissaint, L., J.-P. Véran, and R. Conan, 2006: Analytical modeling of adaptive optics: foundations of the phase spatial power spectrum approach. Optical Society of America Journal A, 23, 382–394.
  * Lawrence et al. (2004) Lawrence, J. S., M. C. B. Ashley, A. Tokovinin, and T. Travouillon, 2004: Exceptional astronomical seeing conditions above Dome C in Antarctica. , 431, 278–281, doi:10.1038/nature02929.
  * Tokovinin (2004) Tokovinin, A., 2004: Seeing Improvement with Ground-Layer Adaptive Optics. PASP, 116, 941–951, doi:10.1086/424805.
  * Tokovinin and Travouillon (2006) Tokovinin, A. and T. Travouillon, 2006: Model of optical turbulence profile at Cerro Pachón. MNRAS, 365, 1235–1242, doi:10.1111/j.1365-2966.2005.09813.x.
  * Tokovinin et al. (2005) Tokovinin, A., J. Vernin, A. Ziad, and M. Chun, 2005: Optical Turbulence Profiles at Mauna Kea Measured by MASS and SCIDAR. PASP, 117, 395–400, doi:10.1086/428930.


*.jpg,.png,.PNG

#  $\text{DC}^{2}$: Dual-Camera Defocus Control by Learning to Refocus

Hadi Alzayer1,2 Abdullah Abuolaim1 Leung Chun Chan1
Yang Yang1 Ying Chen Lou1 Jia-Bin Huang2 Abhishek Kar1

1Google 2University of Maryland, College Park

###### Abstract

Smartphone cameras today are increasingly approaching the versatility and
quality of professional cameras through a combination of hardware and software
advancements. However, fixed aperture remains a key limitation, preventing
users from controlling the depth of field (DoF) of captured images. At the
same time, many smartphones now have multiple cameras with different fixed
apertures - specifically, an ultra-wide camera with wider field of view and
deeper DoF and a higher resolution primary camera with shallower DoF. In this
work, we propose $\text{DC}^{2}$, a system for defocus control for
synthetically varying camera aperture, focus distance and arbitrary defocus
effects by fusing information from such a dual-camera system. Our key insight
is to leverage real-world smartphone camera dataset by using image refocus as
a proxy task for learning to control defocus. Quantitative and qualitative
evaluations on real-world data demonstrate our system’s efficacy where we
outperform state-of-the-art on defocus deblurring, bokeh rendering, and image
refocus. Finally, we demonstrate creative post-capture defocus control enabled
by our method, including tilt-shift and content-based defocus effects.

Ultra-wide

Wide

Refocused

Deblurred

Shallow DoF

$\underbracket{\hskip 164.77771pt}_{\begin{subarray}{c}\vspace{-4.0mm}\\\
\hbox{\pagecolor{white}{Input}}\end{subarray}}$ $\underbracket{\hskip
251.50038pt}_{\begin{subarray}{c}\vspace{-4.0mm}\\\
\hbox{\pagecolor{white}{Our results}}\end{subarray}}$

Figure 1:  Post-capture depth-of-field (DoF) control from dual camera.
(_Left_) Using photos captured from dual cameras with a wide field of view
(FoV) and ultra-wide FoV, our method enables various DoF manipulations.
(_Right_) We showcase our results of refocusing (changing the focal plane),
deblurring (creating all-in-focus imagery), and synthesizing a shallower DoF
(producing bokeh effects).

##  1 Introduction

Smartphone cameras are the most common modality for capturing photographs
today [13]. Recent advancements in computational photography such as burst
photography [18], synthetic bokeh via portrait mode [48], super-resolution
[55], and more have been highly effective at closing the gap between
professional DSLR and smartphone photography. However, a key limitation for
smartphone cameras today is depth-of-field (DoF) control, i.e., controlling
parts of the scene that appear in (and out of) focus. This is primarily an
artifact of their relatively simple optics and imaging systems (e.g., fixed
aperture, smaller imaging sensors, etc.). To bridge the gap, modern
smartphones tend to computationally process the images for further post-
capture enhancements such as synthesizing shallow DoF (e.g., portrait mode
[48, 37]). However, this strategy alone does not allow for DoF extension or
post-capture refocus. In this work, we propose Dual-Camera Defocus Control
($DC^{2}$), a framework that can provide post-capture defocus control
leveraging multi-camera systems prevalent in smartphones today. Figure 1 shows
example outputs from our framework for various post-capture DoF variations. In
particular, our method is controllable and enables image refocus, DoF
extension, and reduction.

Post-capture defocus control is a compound process that involves removing
defocus blur (i.e., defocus deblurring) and then adding defocus blur
selectively based on the scene depth. Defocus deblurring [23, 42, 39, 27, 43,
40, 31, 2, 5, 60, 35, 59, 4], itself, is challenging due to the nature of the
defocus point spread function (PSF) formation which can be spatially varying
in size and shape [28, 46]. The PSF’s shape and size are not only depth
dependent, but also vary based on aperture size, focal length, focus distance,
optical aberration, and radial distortion. Synthesizing and adding defocus
blur [17, 9, 48, 38, 58, 57, 21, 33, 37] is also difficult and requires an
accurate depth map along with an all-in-focus image. Additionally, it requires
realistic blur formation and blending around the object’s boundaries. Most
prior work has addressed defocus deblurring and synthesizing defocus blur as
two isolated tasks. There has been less work on post-capture defocus control
(e.g., image refocusing [41, 34, 22]). The image refocusing literature [34,
22] has focused on light-field data captured with specialized hardware. While
the results in [51, 52] are the state-of-the-art, light-field data is not
representative of smartphone and DSLR cameras by lacking realistic defocus
blur and spatial resolution [12].

Most modern smartphones are now equipped with two or more rear cameras to
assist with computational imaging. The primary camera – often referred to as
the wide camera or $\mathbf{W}$ – has a higher resolution sensor, a higher
focal length lens but a relatively shallower DoF. Alongside $\mathbf{W}$ is
the ultra-wide ($\mathbf{UW}$) camera, often with a lower resolution sensor,
lower focal length (wider field of view) and wider DoF. Our critical insight
is to leverage this unique camera setup and cross-camera DoF variations to
design a system for realistic post-capture defocus control. Differently from
prior work, we tackle the problem of defocus control (deblurring and adding
blur) and propose using real-world data easily captured using a smartphone
device to train our learning-based system. Our primary contributions in this
work are as follows:

  * •

We propose a learning-based system for defocus control on dual-camera
smartphones. This subsumes the tasks of defocus deblurring, depth-based blur
rendering, image refocusing and enables arbitrary post-capture defocus
control.

  * •

In the absence of defocus control ground-truth, we enable training our system
on real-world data captured from a smartphone device. To achieve that, we re-
formulate the problem of defocus control as learning to refocus and define a
novel training strategy to serve the purpose.

  * •

We collect a dataset of diverse scenes with focus stack data at controlled
lens positions the $\mathbf{W}$ camera and accompanying $\mathbf{UW}$ camera
images for training our system. Additionally, we compute all-in-focus images
using the focus stacks to quantitatively evaluate image refocus, defocus
deblurring and depth-based blurring tasks and demonstrate superior performance
compared to state-of-the-art (SoTA) methods across all three tasks.

  * •

Finally, we demonstrate creative defocus control effects enabled by our
system, including tilt-shift and content-based defocus.

(a) Dual camera image refocus dataset.

(b) Proxy task: Learn to refocus.

(c) Evaluate on arbitrary defocus maps.

Figure 2:  Image refocus as a proxy task. Since we cannot gather a real
dataset for arbitrary focus manipulation, our idea is to train a model to
perform _image refocus_ using a target defocus map as an input. At the test
time, our trained model can perform arbitrary focus manipulation by feeding it
an arbitrary target defocus map.

##  2 Related Work

Defocus Deblurring Defocus blur leads to a loss of detail in the captured
image. To recover lost details, a line of work follows a two-stage approach:
(1) estimate an explicit defocus map, (2) use a non-blind deconvolution guided
by the defocus map [23, 42]. With the current advances in learning-based
techniques, recent work perform single image deblurring directly by training a
neural network end-to-end to restore the deblurred image [39, 27, 43, 40, 31,
2]. Due to the difficulty of the defocus deblurring task, other works try to
utilize additional signals, such as the dual pixel (DP) data to improve
deblurring performance [5, 60, 35, 59, 4]. DP data is useful for deblurring as
it provides the model with defocus disparity that can be used to inform
deblurring. While the DP data provides valuable cues for the amount of defocus
blur at each pixel, the DP views are extracted from a single camera.
Therefore, the performance of the DP deblurring methods drops noticeably and
suffer from unappealing visual artifacts for severely blurred regions.
In the same vein, we aim to exploit the UW image as a complementary signal
already available in modern smartphones yet ignored for DoF control. By using
the UW image with different DoF arrangements, we can deblur regions with
severe defocus blur that existing methods cannot handle because of the
fundamental information loss. Nevertheless, we are aware that using another
camera adds other challenges like image misalignment, occlusion, and color
mismatches which we address in Section 2.

Bokeh Rendering Photographers can rely on shallow DoF to highlight an object
of interest and add an artistic effect to the photo. The blur kernel is
spatially variant based on depth as well as the camera and optics. To avoid
the need of estimating depth, one work magnifies the existing defocus in the
image to make the blur more apparent without explicit depth estimate [7].
Since recent work in depth estimation improved significantly [30, 44], many
shallow DoF rendering methods assume having depth [37] or estimate depth in
the process [48, 57]. Using an input or estimated depth map, a shallow DoF can
be synthesized using classical rendering methods [17, 9, 48, 38], using a
neural network to add the synthetic blur [50, 21, 33] or a combination of
classical and neural rendering [37]. With that said, shallow DoF synthesis
methods typically assume an all-in-focus image or an input with a deep DoF.
Our proposed framework learns to blur as a byproduct of learning to refocus
with the insight that the refocus task involves both deblurring and selective
blurring. Unlike prior work that addressed either defocus deblurring or image
bokeh rendering, we introduce a generic framework that facilitates post-
capture full defocus control (e.g., image refocusing).

Image Refocus and DoF Control At capture time, the camera focus can be
adjusted automatically (i.e., autofocus [6, 3, 19]) or manually by moving the
lens or adjusting the aperture. When the image is captured, it can still be
post-processed to manipulate the focus. Nevertheless, post-capture image
refocus is challenging as it requires both deblurring and blurring. Prior work
uses specialized hardware to record a light field which allows post-capture
focus control [34, 53]. However, light field cameras have low spatial
resolution and are not representative of smartphone cameras. An alternative to
requiring custom hardware is to capture a focus stack, and then merge the
frames required to simulate the desired focus distance and DoF [22, 36, 29,
10], but the long capture time restricts using focus stacks to static scenes.
Research on single-image refocus is limited due to its difficulty, but the
typical approach is to deblur to obtain an all-in-focus image followed by
blurring. Previous work used classical deblurring and blurring [8] to obtain
single image refocus, and the most notable recent single-image-based image
refocus is RefocusGAN [41], which trains a two-stages GAN to perform
refocusing. The limited research on software-based image refocus is likely due
to the challenging task that involves both defocus deblurring and selective
blurring. In our work, we provide a practical setup for post-capture image
refocus without the restrictions of inaccessible hardware or the constraint of
capturing a focus stack. We do so by leveraging the dual camera that is
available in modern smartphones.

Image Fusion. Combining information from images with complementary information
captured using different cameras [36, 47] or the same camera with different
capture settings [15, 18] can enhance images in terms of sharpness [22, 47,
36], illuminant estimation [1], exposure [15, 18, 36, 11], or other aspects
[47, 49, 16, 32]. With the recent prevalence of dual-camera smartphones today,
researchers have pursued works that target this setup. One line of work has
used dual-camera for super-resolution to take advantage of the different
resolutions the cameras have in still photos [51, 64, 56] as well as in videos
[26]. The dual-camera setup has also been used in multiple commercial
smartphones, e.g., Google Pixel devices to deblur faces by capturing an ultra-
wide image with faster shutter time and fusing with the wide photo [25]. To
our knowledge, we are the first to investigate using the dual-camera setup for
defocus control.

(a) Data processing

(b) DFNet

Figure 3:  Data processing and high-level architecture. (Left) To be able to
use the reference inputs for our Detail Fusion Network, we need to align the
inputs and a depth estimate to approximate the defocus map of the reference W
and the target defocus map we would like to synthesize. We use flow-based
alignment with PWCNet [45] and use the stereo depth estimated using portrait
mode [48]. (Right) Our Detail Fusion Network (DFNet) consists of refinement
modules to refine the reference inputs combined with a fusion module that
predicts blending masks to combine the two refined inputs.

##  3 Learning to Refocus as a Proxy Task

As mentioned, smartphone cameras tend to have fixed apertures limiting DoF
control at capture time. In our work, we aim to unlock the ability to
synthetically control the aperture - by transferring sharper details where
present and synthesizing realistic blur. However, to train such a model, we
run into a chicken and egg problem: we require a dataset of images captured
with different apertures, which isn’t possible with smartphones. An
alternative solution could be to generate such a dataset synthetically, but
modeling a realistic point spread function (PSF) for the blur kernel is non-
trivial [5]. Professional DSLRs provide yet another alternative [20] but often
require paired captures smartphone / DLSR captures to reduce the domain gap.
Ideally, we would like to use the same camera system for both training and
evaluation. To resolve this, we observe that a somewhat parallel task is image
refocus. When we change the focus distance, the defocus radius is adjusted in
different parts of the image, involving a combination of pixels getting
deblurred and blurred. This suggests that image refocus is at least as hard as
scaling the DoF. Motivated by this observation, we make the hypothesis that by
training a model on image refocus as a proxy task, we can use the same model
to control the DoF at test time as we show in Figure 2. The key idea is to
provide the model with reference and target defocus maps ( Section 4.1) as
input, and at test time control the model behavior by manipulating this target
defocus map.

##  4 Method

To train a model on our proxy task, we need to collect a dataset of focus
stacks for the wide camera and a paired ultra-wide frame which can be used as
a guide due to its deeper DoF. In Figure 3 we show the high-level structure of
our method dubbed $\text{DC}^{2}$. The primary module that we train is the
Detail Fusion Network (DFNet), which requires a reference wide frame,
(aligned) reference ultra-wide frame, and estimated defocus maps. In Section
4.1, we describe how we collect the focus stack data and process it to obtain
the inputs needed for DFNet. We then describe the architecture details of
DFNet in Section 4.2, which is motivated by the dual-camera input setup.

###  4.1 Data Processing

Using the Google Pixel 6 Pro as our camera platform, we captured a dataset of
100 focus stacks of diverse scenes, including indoor and outdoor scenarios.
For each scene, we sweep the focus plane for the wide camera and capture a
complete focus stack. We simultaneously capture a frame from the ultra-wide
camera, which has a smaller aperture, deeper DoF, and fixed focus. For each
frame, we use optical-flow-based warping using PWCNet [45] and following prior
work [25] to align the ultra-wide frame with the wide frame. Since the
alignment is imperfect (e.g., in textureless regions and occluded boundaries),
we estimate an occlusion mask that can be used to express potentially
misaligned regions for the model. To estimate defocus maps, we require the
metric depth. We use the depth map embedded in the Pixel camera’s portrait
mode output which can estimate metric depth using dual camera stereo
algorithms [63] with a known camera baseline. To compute the defocus map
associated with each frame, we use the following formula for the radius of the
circle of confusion $c$

$\displaystyle c=A\frac{|S_{2}-S_{1}|}{S_{2}}\frac{f}{S_{1}-f}$ (1)

where $A$ is the camera aperture, $S_{1}$ is the focus distance, $S_{2}$ is
the pixel depth, and $f$ is the focal length. In Figure 2a, we show a
visualization of a focus stack, associated UW, stereo depth, and a collection
of sample scenes.

###  4.2 Model Architecture

Our method performs detail fusion on two primary inputs: the reference wide
(W) and ultra-wide (UW) images. Since we train the model to refocus, W is
expected to be treated as a base image, while UW is a guide for missing high-
frequency details.

Based on this intuition, we propose Detail Fusion Network (DFNet) that has two
refinement paths: W refinement path ($\Phi_{ref}^{W}$), UW refinement path
($\Phi_{ref}^{UW}$), and a fusion module ($\Phi_{fusion}$) that predicts
blending masks for the refined W and refined UW. Note that the W refinement
path never gets to see the UW frame and vice versa. We use a network
architecture based on Dynamic Residual Blocks Network (DRBNet) [39] for our
refinement modules with multi-scale refinements. For the fusion module, we use
a sequence of atrous convolutions [14] for an increased receptive field and
predict a blending mask for each scale. To preserve high-frequency details in
the blending mask, we add upsampling layer and residual connections when
predicting the blending mask of the larger scale. During training, we blend
the outputs of $\Phi_{ref}^{W}$ and $\Phi_{ref}^{UW}$ and compute the loss for
all scales for improved performance. In Figure 3 we show a high-level diagram
of our architecture and how each component interacts with the others. By
visualizing the intermediate outputs between our different modules, we observe
that the network indeed attempts to maintain the low-frequency signal fromW
while utilizing high-frequency signals from UW. Please refer to the
supplementary material for a detailed model architecture and a deeper analysis
of model behavior and visualizations.

###  4.3 Training Details

We train our model by randomly sampling slices from the focus stack in our
training scenes. For each element in the batch, we randomly sample a training
scene, and sample two frames to use as reference and target images,
respectively. While we can approximate depth from all pairs, severely blurry
frames can have unreliable depth. To address that, we use the stereo pair with
the greatest number of matched features to use for the scene depth to compute
the defocus maps. We train on randomly cropped 256x256 patches, using a batch
size of 8, and a learning rate of $10^{-4}$ for 200k iterations, and then
reduce the learning rate to $10^{-5}$ for another 200k iterations using Adam
[24]. Our reconstruction loss is a combination of ${L}_{1}$ loss on pixels and
gradient magnitudes, SSIM loss [54], and perceptual loss [61]. For a target
wide frame $\textbf{W}_{tgt}$ and a model output $y$, the loss is

$\displaystyle L_{total}$
$\displaystyle={L}_{1}(\textbf{W}_{tgt},y)+{L}_{1}(\nabla\textbf{W}_{tgt},\nabla
y)$ (2) $\displaystyle
L_{SSIM}(\textbf{W}_{tgt},y))+L_{VGG}(\textbf{W}_{tgt},y)$

##  5 Experimental Results

We train our method to perform defocus control through training on the proxy
task of image refocus. As a result, our model can perform a variety of related
defocus control tasks. Specifically, we evaluate our method on defocus
deblurring, synthesizing shallow DoF, and image refocus.

Evaluation metrics. We use the standard signal processing metrics, i.e., the
peak signal-to-noise ratio (PSNR) and the structural similarity index measure
(SSIM). We also report the learned perceptual image patch similarity (LPIPS)
[62].

Table 1:  Defocus deblurring evaluation. Performance on generating all-in-focus images from a single slice in the focus stack. The best results are in bold numbers.  Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---
MDP [2] | 23.50 | 0.674 | 0.394
IFAN [27] | 23.48 | 0.679 | 0.371
DRBNet [39] | 24.27 | 0.681 | 0.377
Ours | 24.79 | 0.704 | 0.351

Input W

Input UW

GT

DRBNet [39]

Ours

Figure 4:  Defocus deblurring. We showcase the results of our method against
SoTA single image defcous deblurring DRBNet [39]. Note that our method
restores severely blurred regions in the background that single-image based
methods often struggle with.

###  5.1 Defocus Deblurring

Task. The goal of defocus deblurring is to remove the defocus blur in the
image. For our method to perform defocus deblurring, we simply set the target
defocus map to all zeros. To obtain an all-in-focus image as a ground truth,
we perform focus stacking using our focus stacks through commercial software
provided by HeliconFocus. Then the evaluation task is deblurring individual
slices from the focus stack to generate an all-in-focus image. Due to the
focus magnification between the focus stack slices, we align the field-of-view
(FoV) with the all-in-focus image through a combination of FoV matching and
brute-force search for the best scaling and translation parameters that
minimize the error. We use the same alignment method when evaluating all the
methods to ensure fairness.

Methods. We compare our method with the following single-image defocus
deblurring methods: Dynamic Residual Blocks Network (DRBNet) [39], Multi-task
DP (MDP) network [2], and Iterative Filter Adaptive Network (IFAN) [27]. Note
that these methods do not take the ultra-wide image as input, and the main
purpose of the comparison is to highlight the value of leveraging an available
dual-camera setup. Our dataset does not contain DP data and thus we are not
able to benchmark the DP defocus deblurring methods [5, 60, 35, 59, 4]. As for
the evaluation on other defocus deblurring datasets (e.g., [4]), our method
requires dual-camera input not available in current datasets.

Evaluation. In Table 1, we compare the performance of our method against other
defocus deblurring methods. Our method achieves the best results on all
metrics with dual camera inputs. Note that our method has never seen all-in-
focus outputs / zero target defocus maps during training and learns to deblur
via the proxy task. Figure 4 shows two deblurring results of our method
against DRBNet [39]. As shown in the zoomed-in insets, our method is able to
restore severely blurred regions better compared to DRBNet. In general,
single-image defocus deblurring methods suffer from artifacts and tend to
hallucinate when restoring severely blurred regions. Therefore, an additional
signal such as the UW is very useful when the details are completely lost in
the input image. While the main task of our proposed method is not only
defocus deblurring, it achieves the SoTA deblurring results quantitatively and
qualitatively. These results also demonstrate how generic and flexible our
proposed defocus control framework is.

Input W

Input UW

GT

BokehMe [37]

Ours

Figure 5:  Blurring results. Our method can synthesize shallow DoF from an all-in-focus image with a performance competitive with SoTA in bokeh rendering [37].  Table 2:  Bokeh blurring evaluation. performance on simulating different slices of the focus stack from the all-in-focus image.  Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---
BokehMe [37] | 26.65 | 0.870 | 0.241
Neural Rend. [37] | 27.87 | 0.874 | 0.246
Classic Rend. [37] | 26.66 | 0.870 | 0.241
Ours | 29.78 | 0.898 | 0.172

###  5.2 Shallow DoF Rendering

Task. We also evaluate our method on rendering shallow DoF images. The input
to the method is an all-in-focus image, an approximate target defocus map, and
the desired output is the image with a synthetic shallow DoF guided by the
defocus map. We use the all-in-focus image generated from the focus stack as
input and try to reconstruct the various slices in the focus stack using each
slice’s defocus map as a target.

Methods. We compare against BokehMe [37], a recent state-of-the-art in shallow
DoF synthesis that relies on blending the outputs of classic blur synthesis
with neural rendering methods. We also evaluate the classical scattering-based
blur and the neural renderer within BokehMe in isolation.

Evaluation. In Table 2, we show that our method is competitive with SoTA
shallow DoF rendering methods. Note that for DoF reduction, $\mathbf{UW}$ does
not provide a useful signal since the task primarily involves signal removal
from $\mathbf{W}$, but the model learns to perform this task as a byproduct of
training on image refocus. In Figure 5 we show visual results where our model
synthesizes realistic blur.

Input W

Input UW

GT

Baseline [39, 37]

Ours

Figure 6: Refocus results. We shift the focus plane and demonstrate that we can match the desired refocused image and the target blur without completely deblurring the input. Our method outperforms the baseline that refocuses by deblurring and reblurring the image. Table 3:  Image refocus evaluation. Performance on re-synthesizing focus planes given an input with different focus plane from the same scene. Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---
UW + Blur [37] | 21.89 | 0.803 | 0.364
Deblur [39]+Reblur [37] | 26.40 | 0.833 | 0.312
Ours | 28.58 | 0.860 | 0.217

###  5.3 Image Refocus

Task. Image refocus involves shifting the focus plane and as a result, the
near and far focus depths. To evaluate on image refocus, we randomly sample
two frames from a focus stack, a reference frame, and a target frame, and
evaluate the model performance in reproducing the target frame.

Methods. There is limited work on single-image refocus, the most notable work
being RefocusGAN [41]. The idea behind RefocusGAN is to use generative models
to deblur the image followed by blurring it. This approach is likely because
of the difficulty of realistically switching between different defocus amounts
directly [7]. However, we are not able to compare with RefocusGAN as the code
and trained models are not available. As an alternative for comparison, we
adopt SoTA in defocus deblurring (DRBNet [39]) and SoTA in blurring (BokehMe
[37]) for image refocus. We also compare against blurring the aligned UW
directly since it could approximate an all-in-focus image.

Evaluation. In Table 3 we show that our method outperforms the baseline in
image refocus. Note that since we train our method to switch between the
reference defocus to the target defocus, the model can implicitly learn to
switch between different PSF scales from the data. We show visual results in
Figure 6. Note that when the target image contains blurry regions like shown
on the wall, our method deblurs the input just enough to match the target
defocus.

Table 4:  Ablations on Image Input. Comparison on different input types. Although performance increases by removing the occlusion mask, qualitative performance drops (see Figure 7).  Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---
W only | 28.44 | 0.855 | 0.260
UW only | 22.66 | 0.822 | 0.307
No occlusion | 28.81 | 0.864 | 0.219
Full input | 28.58 | 0.860 | 0.217

Full input

No occlusion mask

Wide only

UW only

Figure 7:  Ablation. Using the occlusion mask helps the model avoid
transferring warping artifacts to the final image, while using W only hinders
deblurring performance, and UW only suffers from warping artifacts and lower
resolution.

###  5.4 Ablation Study

The key idea of our work is using the ultra-wide camera as a guide to
performing DoF control. To evaluate the effects of using UW, we train a model
using only W (only keeping the Wide refinement module) and similarly training
a UW only model. We compare their performance on image refocus in Table 4.
Note that while the wide input is sufficient when the target involves only
blurring or minimal deblurring, it is an ill-posed setup when it requires
considerable deblurring. On the other hand, the warped UW lower quality
severely limits the performance when relying on it completely. We visualize an
example in In Figure 7. Note that when using W only, deblurring performance is
limited. Also we note that when removing the occlusion mask, while signal-
processing metrics could see slight improvements, qualitative performance
drops as we can observe ghosting artifacts around occluded boundaries.

Input

Output

Figure 8:  Creative applications. (Top) we apply tilt shift effect that makes
large objects appear as miniatures. (Bottom) we use segmentation mask to
deblur objects of interest and blur the background.

Applications. Our method allows for arbitrary target defocus maps as an input.
In Figure 8 we demonstrate a tilt-shift effect, where a large scene appears
smaller because of the blur, as well as using a segmentation mask to deblur
objects of interest (the person) while blurring the remaining objects.

W

Warped UW

Figure 9:  Failure cases. (Top) In regions where UW is out of focus, as in the
table shown above, our method would not be able to restore the sharp texture
shown in the W image when focused on that object. (Bottom) Aligning the UW
using optical flow based alignment could suffer when W is severly blurred like
shown on the warping artifacts on the window.

##  6 Limitations and Conclusion

We present $\text{DC}^{2}$, a novel framework for defocus control with dual-
camera consumer smartphones. We bypass the need for synthetic data and domain
gap issues by training with real data captured with a smartphone device. We do
so by re-framing the defocus control problem as refocus and designing a
learning-based solution for the same. The key idea behind our method is to use
UW input as an additional signal to control defocus of W. Naturally, a
limitation then is the DoF of UW itself; as objects outside of its DoF might
not be sharper than in W. In general, our method benefits from asymmetry in
the W and UW camera configurations and likely won’t perform as well in systems
with identical cameras. Another limitation is our dependence on pre-existing
optical flow and stereo depth algorithms which can suffer from severe
artifacts with defocus blur (Figure 9). A promising avenues for future work
includes utilizing additional cameras to jointly model both scene depth and
defocus control.

Acknowledgment We would like to thank Junlan Yang, Xiaotong Wu, Lun-Cheng Chu,
Mauricio Delbracio, Yichang Shih, and Seang Chau for their support and
fruitful discussions.

## References

  * [1] Abdelrahman Abdelhamed, Abhijith Punnappurath, and Michael S Brown.  Leveraging the availability of two cameras for illuminant estimation.  In CVPR, 2021.
  * [2] Abdullah Abuolaim, Mahmoud Afifi, and Michael S Brown.  Improving single-image defocus deblurring: How dual-pixel images help through multi-task learning.  In WACV, 2022.
  * [3] Abdullah Abuolaim and Michael Brown.  Online lens motion smoothing for video autofocus.  In WACV, 2020.
  * [4] Abdullah Abuolaim and Michael S Brown.  Defocus deblurring using dual-pixel data.  In ECCV, 2020.
  * [5] Abdullah Abuolaim, Mauricio Delbracio, Damien Kelly, Michael S Brown, and Peyman Milanfar.  Learning to reduce defocus blur by realistically modeling dual-pixel data.  In ICCV, 2021.
  * [6] Abdullah Abuolaim, Abhijith Punnappurath, and Michael S Brown.  Revisiting autofocus for smartphone cameras.  In ECCV, 2018.
  * [7] Soonmin Bae and Frédo Durand.  Defocus magnification.  In Computer Graphics Forum, volume 26, pages 571–579. Wiley Online Library, 2007.
  * [8] Yosuke Bando and Tomoyuki Nishita.  Towards digital refocusing from a single photograph.  In 15th Pacific Conference on Computer Graphics and Applications (PG’07), pages 363–372. IEEE, 2007.
  * [9] Jonathan T. Barron, Andrew Adams, YiChang Shih, and Carlos Hernández.  Fast bilateral-space stereo for synthetic defocus.  In CVPR, pages 4466–4474, 2015.
  * [10] Shiveta Bhat and Deepika Koundal.  Multi-focus image fusion techniques: a survey.  Artificial Intelligence Review, 54(8):5735–5787, 2021.
  * [11] Radu Ciprian Bilcu, Adrian Burian, Aleksi Knuutila, and Markku Vehvilainen.  High dynamic range imaging on mobile devices.  In 2008 15th IEEE International Conference on Electronics, Circuits and Systems, pages 1312–1315. IEEE, 2008.
  * [12] Vivek Boominathan, Kaushik Mitra, and Ashok Veeraraghavan.  Improving resolution and depth-of-field of light field cameras using a hybrid imaging system.  In ICCP, 2014.
  * [13] David Cardinal.  Smartphones vs cameras: Closing the gap on image quality.  https://www.dxomark.com/smartphones-vs-cameras-closing-the-gap
-on-image-quality/, Apr 2021.
  * [14] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [15] Paul E Debevec and Jitendra Malik.  Recovering high dynamic range radiance maps from photographs.  In ACM SIGGRAPH 2008 classes, pages 1–10, 2008.
  * [16] Clement Godard, Kevin Matzen, and Matt Uyttendaele.  Deep burst denoising.  In ECCV, September 2018.
  * [17] Thomas Hach, Johannes Steurer, Arvind Amruth, and Artur Pappenheim.  Cinematic bokeh rendering for real scenes.  In Proceedings of the 12th European Conference on Visual Media Production, CVMP ’15, New York, NY, USA, 2015. Association for Computing Machinery.
  * [18] Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Marc Levoy.  Burst photography for high dynamic range and low-light imaging on mobile cameras.  ACM Trans. Graph., 35(6), dec 2016.
  * [19] Charles Herrmann, Richard Strong Bowen, Neal Wadhwa, Rahul Garg, Qiurui He, Jonathan T Barron, and Ramin Zabih.  Learning to autofocus.  In CVPR, 2020.
  * [20] Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, and Luc Van Gool.  Dslr-quality photos on mobile devices with deep convolutional networks.  ICCV, 2017.
  * [21] Andrey Ignatov, Jagruti Patel, and Radu Timofte.  Rendering natural camera bokeh effect with deep learning.  In CVPR Workshops, 2020.
  * [22] David E Jacobs, Jongmin Baek, and Marc Levoy.  Focal stack compositing for depth of field control.  Stanford Computer Graphics Laboratory Technical Report, 1(1):2012, 2012.
  * [23] Ali Karaali and Claudio Rosito Jung.  Edge-based defocus blur estimation with adaptive scale selection.  IEEE Transactions on Image Processing, 27(3):1126–1137, 2018.
  * [24] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2014.  cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
  * [25] Wei-Sheng Lai, Yichang Shih, Lun-Cheng Chu, Xiaotong Wu, Sung-Fang Tsai, Michael Krainin, Deqing Sun, and Chia-Kai Liang.  Face deblurring using dual camera fusion on mobile phones.  ACM Transactions on Graphics (TOG), 41(4), jul 2022.
  * [26] Junyong Lee, Myeonghee Lee, Sunghyun Cho, and Seungyong Lee.  Reference-based video super-resolution using multi-camera video triplets.  In CVPR, 2022.
  * [27] Junyong Lee, Hyeongseok Son, Jaesung Rim, Sunghyun Cho, and Seungyong Lee.  Iterative filter adaptive network for single image defocus deblurring.  In CVPR, 2021.
  * [28] Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman.  Understanding blind deconvolution algorithms.  IEEE transactions on pattern analysis and machine intelligence, 33(12):2354–2367, 2011.
  * [29] Shutao Li, James T Kwok, and Yaonan Wang.  Combination of images with diverse focuses using the spatial frequency.  Information fusion, 2(3):169–176, 2001.
  * [30] Zhengqi Li and Noah Snavely.  Megadepth: Learning single-view depth prediction from internet photos.  In CVPR, 2018.
  * [31] Haoyu Ma, Shaojun Liu, Qingmin Liao, Juncheng Zhang, and Jing-Hao Xue.  Defocus image deblurring network with defocus map estimation as auxiliary task.  IEEE Transactions on Image Processing, 31:216–226, 2022.
  * [32] Ben Mildenhall, Jonathan T Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll.  Burst denoising with kernel prediction networks.  In CVPR, 2018.
  * [33] Hariharan Nagasubramaniam and Rabih Younes.  Bokeh effect rendering with vision transformers.  TechRxiv, 2022.
  * [34] Ren Ng, Marc Levoy, Mathieu Br, Gene Duval, Mark Horowitz, Pat Hanrahan, and Duval Design.  Light Field Photography with a Hand-held Plenoptic Camera.  PhD thesis, Stanford University, 2005.
  * [35] Liyuan Pan, Shah Chowdhury, Richard Hartley, Miaomiao Liu, Hongguang Zhang, and Hongdong Li.  Dual pixel exploration: Simultaneous depth estimation and image restoration.  In CVPR, pages 4340–4349, June 2021.
  * [36] Sujoy Paul, Ioana S Sevcenco, and Panajotis Agathoklis.  Multi-exposure and multi-focus image fusion in gradient domain.  Journal of Circuits, Systems and Computers, 25(10):1650123, 2016\.
  * [37] Juewen Peng, Zhiguo Cao, Xianrui Luo, Hao Lu, Ke Xian, and Jianming Zhang.  Bokehme: When neural rendering meets classical rendering.  In CVPR, 2022.
  * [38] Kuldeep Purohit, Maitreya Suin, Praveen Kandula, and Rajagopalan Ambasamudram.  Depth-guided dense dynamic filtering network for bokeh effect rendering.  In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3417–3426, 2019.
  * [39] Lingyan Ruan, Bin Chen, Jizhou Li, and Miuling Lam.  Learning to deblur using light field generated and real defocus images.  In CVPR, 2022.
  * [40] Lingyan Ruan, Bin Chen, Jizhou Li, and Miu-Ling Lam.  Aifnet: All-in-focus image restoration network using a light field-based dataset.  IEEE Transactions on Computational Imaging, 7:675–688, 2021.
  * [41] Parikshit Sakurikar, Ishit Mehta, Vineeth N. Balasubramanian, and P. J. Narayanan.  Refocusgan: Scene refocusing using a single image.  In ECCV, page 519–535, Berlin, Heidelberg, 2018. Springer-Verlag.
  * [42] Jianping Shi, Li Xu, and Jiaya Jia.  Just noticeable defocus blur detection and estimation.  In CVPR, pages 657–665, 2015.
  * [43] Hyeongseok Son, Junyong Lee, Sunghyun Cho, and Seungyong Lee.  Single image defocus deblurring using kernel-sharing parallel atrous convolutions.  In ICCV, 2021.
  * [44] Pratul P. Srinivasan, Rahul Garg, Neal Wadhwa, Ren Ng, and Jonathan T. Barron.  Aperture supervision for monocular depth estimation.  In CVPR, June 2018.
  * [45] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume.  In CVPR, 2018.
  * [46] Huixuan Tang and Kiriakos N Kutulakos.  Utilizing optical aberrations for extended-depth-of-field panoramas.  In ACCV, 2012.
  * [47] Marc Comino Trinidad, Ricardo Martin Brualla, Florian Kainz, and Janne Kontkanen.  Multi-view image fusion.  In ICCV, 2019.
  * [48] Neal Wadhwa, Rahul Garg, David E. Jacobs, Bryan E. Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T. Barron, Yael Pritch Knaan, and Marc Levoy.  Synthetic depth-of-field with a single-camera mobile phone.  SIGGRAPH, 2018.
  * [49] Jian Wang, Tianfan Xue, Jonathan T Barron, and Jiawen Chen.  Stereoscopic dark flash for low-light photography.  In ICCP, 2019.
  * [50] Lijun Wang, Xiaohui Shen, Jianming Zhang, Oliver Wang, Zhe Lin, Chih-Yao Hsieh, Sarah Kong, and Huchuan Lu.  Deeplens: Shallow depth of field from a single image.  ACM Transactions on Graphics, 37:1–11, 12 2018.
  * [51] Tengfei Wang, Jiaxin Xie, Wenxiu Sun, Qiong Yan, and Qifeng Chen.  Dual-camera super-resolution with aligned attention modules.  In International Conference on Computer Vision (ICCV), 2021.
  * [52] Ting-Chun Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei A. Efros, and Ravi Ramamoorthi.  Light field video capture using a learning-based hybrid imaging system.  ACM Transactions on Graphics (Proceedings of SIGGRAPH 2017), 36(4), 2017.
  * [53] Yingqian Wang, Jungang Yang, Yulan Guo, Chao Xiao, and Wei An.  Selective light field refocusing for camera arrays using bokeh rendering and superresolution.  IEEE Signal Processing Letters, 26(1):204–208, jan 2019.
  * [54] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [55] Bartlomiej Wronski, Ignacio Garcia-Dorado, Manfred Ernst, Damien Kelly, Michael Krainin, Chia-Kai Liang, Marc Levoy, and Peyman Milanfar.  Handheld multi-frame super-resolution.  ACM Trans. Graph., 38(4), jul 2019.
  * [56] Yubin Wu, Lianglun Cheng, Tao Wang, and Heng Wu.  Infrared and visible light dual-camera super-resolution imaging with texture transfer network.  Signal Processing: Image Communication, 108:116825, 2022.
  * [57] Ke Xian, Juewen Peng, Chao Zhang, Hao Lu, and Zhiguo Cao.  Ranking-based salient object detection and depth prediction for shallow depth-of-field.  Sensors, 21(5), 2021.
  * [58] Lei Xiao, Anton Kaplanyan, Alexander Fix, Matt Chapman, and Douglas Lanman.  Deepfocus: Learned image synthesis for computational display.  In ACM SIGGRAPH 2018 Talks, pages 1–2. Association for Computing Machinery, 2018.
  * [59] Shumian Xin, Neal Wadhwa, Tianfan Xue, Jonathan T. Barron, Pratul P. Srinivasan, Jiawen Chen, Ioannis Gkioulekas, and Rahul Garg.  Defocus map estimation and deblurring from a single dual-pixel image.  ICCV, 2021.
  * [60] Ruikang Xu, Zeyu Xiao, Jie Huang, Yueyi Zhang, and Zhiwei Xiong.  Edpn: Enhanced deep pyramid network for blurry image restoration.  In CVPR Workshops, June 2021.
  * [61] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [62] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [63] Yinda Zhang, Neal Wadhwa, Sergio Orts-Escolano, Christian Häne, Sean Fanello, and Rahul Garg.  Du2net: Learning depth estimation from dual-cameras and dual-pixels.  In ECCV, Berlin, Heidelberg, 2020. Springer-Verlag.
  * [64] Zhilu Zhang, Ruohao Wang, Hongzhi Zhang, Yunjin Chen, and Wangmeng Zuo.  Self-supervised learning for real-world super-resolution from dual zoomed observations.  In ECCV, 2022.

##  A Video Visualization

One major advantage of our method is the fine-grained control we can have on
the defocus control. As a result, we can directly simulate changing the focus
distance and aperture smoothly just like if we had a DSLR camera with variable
focal length and aperture. Please refer to the video provided in the
supplementary materials.

##  B Detailed Architecture

The model architecture consists of three primary modules: $\Phi_{ref}^{W}$ to
refine W, $\Phi_{ref}^{UW}$ to refine UW, and a fusion model $\Phi_{fusion}$
to predict a blending mask to blend the refined outputs. Both $\Phi_{ref}^{W}$
and $\Phi_{ref}^{UW}$ use DRBNet architecture [39] that utilize kernels
prediction to refine the input. Each refinement module predicts intermediate
outputs in a multi-scale setup that can be used to speed up training.
Specifically, the model generates refined outputs at the following scales: 8x
downsampled, 4x downsampled, 2x downsampled, and the original resolution. To
be able to fuse all the multi-scale outputs, $\Phi_{fusion}$ consists of
several Atrous Spatial Pyramid Pooling (ASPP) convolutions blocks [14] to
predict blending mask for each scale. The ASPP blocks for each scale take the
refined W and UW of the associated scale, as well as an upsampled blending
mask from the previous ASPP block with a residual connection of the upsampled
mask (except for the first ASPP block since it has no preceding blending
mask). There are two hyperparameters associated with the blending block for
each scale: (1) atrous rates for the atrous convolutions, and (2) the number
of channels each intermediate step of atrous convolutions outputs. In table 5,
we include a list of the hyperparameters for the blending block associated
with each scale.
One issue with training the model in using cropped patches is that the blur
kernel is spatially varying depending on the crop position. To resolve the
ambiguity, we follow the solution proposed by Abuolaim _et al_. [5] and
concatenate a radial mask to the inputs of all modules where the pixel values
of the mask are the distance from the original image center, normalized.

Table 5:  Fusion model ($\Phi_{fusion}$) hyperparameters. The hyeraprameters for the ASPP convolution blocks are the atrous rates for the atrous convolutions, and the channels each layer outputs. The number of atrous convolution layers is the size of the channel list. Note that the final output consists of two channels which correspond to the W and UW blending masks.  Blending Block | atrous rates | channels
---|---|---
1/8x scale | 1,3,5 | 16, 32, 2
1/4x scale | 1, 3, 6, 12 | 16, 32, 2
1/2x scale | 1, 3, 6, 12, 15 | 16, 32, 2
1x scale | 1,3,6, 12, 15, 18 | 16, 32, 32, 2
Figure 10: Disparity from iPhone. Using portrait mode, we obtained Dual-camera
disparity using an iPhone 14 Pro.

##  C Model Analysis

The primary motivation behind our architecture design is, depending on the
target defocus map, the network can choose to deblur/blur parts of W and
transfer sharper details from UW if necessary. Due to our model design, we can
directly visualize the intermediate outputs to understand the model behavior.
Specifically, we can visualize the refined W and UW which are the outputs of
$\Phi_{ref}^{W}$ and $\Phi_{ref}^{UW}$, as well as the blending masks
predicted by the fusion network $\Phi_{fusion}$. We visualize the intermediate
outputs of our method on the task of all-in-focus deblurring in Figures 11 and
12. Note that in both examples, the mask associated with UW has large values
around edges and regions with high frequencies, while the mask for W has
higher values inlow frequency regions. This supports our hypothesis of having
UW serve for high frequency details that could be blurry in W, while the W
should be used as a reference to preserve the desired colors even in blurry
regions. This behavior makes our method robust to color differences in W and
UW just like we show in Figure 11 where UW has incorrect white balance, and in
Figure 12 we show how the model avoids relying on UW in occluded regions where
artifacts may show up in the optical flow alignment.

Aligned UW

Refined UW

UW blending mask

Ref. W

Refined W

W blending mask

Figure 11:  Intermediate results visualization. Note that the whitebalance is
off in UW, but the refinement module does not get affected by that since it
primarily preserves the high frequencies in refined UW. In the refinement of
W, we notice that the model deblurs the edges and preserves the low-frequency
signals that can be blended with the details from UW

Aligned UW

Refined UW

UW blending mask

Ref. W

Refined W

W blending mask

Figure 12:  Intermediate results visualization. Note that while the aligned UW
suffers from an alignment artifcat around the bottle, the predicted masks take
that into account by setting a low blending value for the occluded region in
the UW mask and a higher value in the W mask.

##  D Generalizing to Different Phone Setup

Our method requires only two cameras with different DoFs. This is widely
available in modern smartphones since ultra-wide cameras tend to have a deeper
DoF due to the small focal length compared to the wide and Telephoto cameras.
Our approach that utilizes the defocus map is not specific to a particular
device, but rather it can produce fairly good results for any smartphone with
a similar UW+W dual-camera setup. To use data captured using an iPhone 14 Pro,
we used the iPhone’s portrait mode to obtain a disparity map (shown in Fig.
10), and warped the UW using an opitcal-flow based alignment. In Fig. 13, we
show results of our model on data captured by iPhone 14 Pro without any
finetuning.

W

UW

Deeper DoF

Reduced DoF

Figure 13: Results on iPhone 14 Pro. We ran our model on images from an iPhone
14 Pro, and show that it generalizes with blurring and deblurring despite not
finetuning the model on any iPhone data.


# Enhancing methane sensors by sensing without detection

Arthur C. Cardoso  1, * Haichen Zhou  1 Siddarth K. Joshi  1 and John G.
Rarity1  1Dept. of Electrical and Electronic Engineering, University of
Bristol, Merchant Venturers Building, Woodland Road, Bristol BS8 1UB, UK
<EMAIL_ADDRESS>
###### Abstract

We propose a novel methane sensor based on sensing without detection. First
used in quantum imaging, this technique relies on the interference effects
seen in three wave mixing when pump signal and idler modes make a double pass
through a nonlinear crystal. The method allows sensing at wavelengths where
detectors are poor and detecting at wavelengths where photon counting
sensitivity can be achieved. This compact interferometer, potentially set in a
portable device for field operations, could allow the detection of low
concentrations of methane at up to 100m range. Signal-to-noise ratio
calculations point out that the method’s sensitivity can overcome the
integrated path differential absorption direct sensing at high non-linear gain
regimes.

††journal: oe††articletype: Research Article

##  1 Introduction

Methane is the second most important greenhouse gas in the Earth’s atmosphere,
its global warming potential is up to eighty-four times more than carbon
dioxide in a twenty years period[1, 2], providing a huge contribution to the
rise of temperature and related climate change. The concentration of this gas
in the atmosphere has been rising since the beginning of the industrial age,
and it is caused by anthropogenic and natural sources[3, 4, 5]. Agricultural,
use and extraction of fossil fuels, biomass burning and decomposition of
organic matter in wastes and landfills are the most important contributions
among the human $\text{CH}_{4}$ sources[6]. Thus, monitoring and control of
methane emissions is crucial to reduce global warming and mitigate climate
changes[7].

Over the last decades scientist have endeavoured to develop new accurate
methods to measure methane concentrations in the air. Many different
experiments exploring optical methane sensors were performed using spaceborne
[8, 9], airborne[10, 11] and short range sensors[12, 13]. They approach
miscellaneous techniques like Integrated Path Differential Absorption (IPDA),
Differential Absorption Lidar (DIAL), Dual-comb spectroscopy (DCS), spectral
analysis of light among others methods. Modern short range DIAL methane
sensors[12, 13] are able to detect sources with high spatial and depth
resolution with high sensitivity, being the state-of-art technology to
identify small sources up to hundreds of meters away.

This paper proposes a novel method for short range methane sensing based on
sensing without detection. Formerly exploited for imaging[14, 15], this
technique makes use of high-correlated light sources in a nonlinear
interferometer to collect phase and amplitude images of an object, by
detecting light that never interacted with it. Recently, this methodology has
been extended for several applications like spectroscopy[16], microscopy[17,
18], quantum holography[19], optical coherence tomography[20, 21] and
terahertz sensing [22]. Methane exhibits a strong absorption at middle
infrared (MIR) spectra, up to one hundred times greater than at short wave
infrared (SWIR), but MIR light detection is quite challenging due to thermal
noise background and low-efficiency detectors. Sensing without detection
enables a two-colour system that is particularly useful in this case because
it allows illumination of the gas at a wavelength for which it has a high
absorbance and detection at a wavelength where low-noise and efficient
detectors are available.

##  2 Proposed experimental apparatus and signal-to-noise ratio

###  2.1 Proposed experimental scheme

The nonlinear interferometer proposed for methane sensing is depicted in Fig.1
and relies on high-gain light sources generated by stimulated parametric down
conversion (SPDC).

A pump and an idler laser lasers are overlapped in a nonlinear crystal (NLC)
stimulating the generation of a signal mode by SPDC. The signal field is short
wave infrared while the idler is MIR and can be tuned onto resonance with
methane absortion lines. The idler laser passed through a gas plume, hits a
background, which could be a mirror or partially reflective surface, and is
scattered back into the interferometer. The pump and the signal modes are
reflected back to the crystal by a highly reflective mirror. In the second
pass through the crystal the pump and idler mode generate a phase dependent
amplification of the signal field and the resulting intensity is measured in a
sensitive (ie photon counting) detector. The phase shifter (PS) can be used to
adjust the phase of the idler mode to produce amplification or deamplification
at the second down conversion. As a result the signal mode generated will show
an interference pattern with period reflecting the idler mode wavelength.

Figure 1: Scheme proposed for gas sensing without detection. Pump and idler
laser beams are combined in a dichroic mirror (DM) to overlap in a nonlinear
crystal (NLC) generating a signal mode. The pump laser and the signal field
are reflected back to the crystal by a highly reflective mirror (M). The idler
mode is deflected by a dichroic mirror and passed through a gas plume and
scattered back to the crystal from a passive Lambertian background or
(partial) retroreflector. In the second interaction of the pump and idler
fields with the NLC, a weak phase dependent amplification of the signal mode
occurs and the resulting light is directed to a sensitive (photon counting)
detector. By adjusting the idler laser’s phase with the phase shifter (PS),
amplification and de-amplification can be measured in the signal beam as an
interference fringe. The visibility of this interference pattern contains
information on the absorption of the idler beam by the gas plume

The magnitude of the modulation of the signal field is proportional to the
field returning through the gas plume and hence the visibility of the
interference pattern carries a measure of the absorption in the idler path.
When the idler laser is tuned to a wavelength at which methane is transparent
(off-resonance), we will observe a high visibility interference pattern.
Otherwise, if it is tuned to spectral region where methane exhibits a high
absorbance (on-resonance) we will observe a loss of visibility in the
interference pattern effectively due to the interference between paths with
different intensities. By comparing the visibilities of the two interference
patterns, we are able to calculate out the amount of MIR light absorbed and
estimate the methane concentration in the gas plume, by detecting only the
signal mode that never interacted with it.

As a proof of principle, a laboratory experiment has been set up using a
mirror to reflect back the idler mode. The apparatus is designed to measure
methane levels in gas cells. In the experiment we use a $1.064\mu$m laser to
pump a PPLN crystal and a $3.392\mu$m He-Ne laser to stimulate the SPDC. A
single photon counting InGaAs signal detector of $2.5\%$ nominal efficiency
detects photons at $1.55\mu$m. A set of filters of around $0.6\%$
transmittance keeps the count rate under the saturation rate of the detector.
By translating the mirror that reflects the idler laser back to the crystal we
change the relative phase between the modes. This enables us to measure an
interference pattern, shown in Fig.2.

Figure 2: Graph of the number of counts measured by the signal detector as
function of the length of the increment path of the idler laser’s path.

The visibility of the fringe is $(92\pm 3)\%$ and its period is approximately
the idler laser wavelength. The next steps for the experiment is to change the
stimulating laser to a tunable one and measure methane concentrations in gas
cells with different methane concentrations.

###  2.2 Signal-to-noise ratio

The intensity of the signal mode generated by the SPDC is

$I_{s}=\left(\frac{\mu_{0}\epsilon_{0}\chi^{(2)}\omega_{s}^{2}L}{2k_{s}}\right)^{2}I_{p}I_{i}\\\
,$ (1)

where $\mu_{0}$ is the magnetic vacuum permeability, $\epsilon_{0}$ is the
electric vacuum permeability, $\chi^{(2)}$ is the effective non-linearity of
the nonlinear crystal, $L$ is its length, $\omega_{s}$ is the angular
frequency of the signal mode, $k_{s}$ its wave-vector’s modulus, $I_{p}$ the
intensity of the pump laser and $I_{i}$ the intensity of the idler laser.
Eq.(1) can be written as $I_{s}=GI_{i}$, where
$G=\left(\frac{\mu_{0}\epsilon_{0}\chi^{(2)}\omega_{s}^{2}L}{2k_{s}}\right)^{2}I_{p}$
is a gain factor related to the SPDC process.

On the first pass through the crystal we create a small amount of light
$I_{s1}$ in the signal beam stimulated by the idler intensity $I_{i}$ and a
small amount of extra light in the idler

$I_{s1}=GI_{i},I_{i1}=I_{i}(1+G)$ (2)

In our experiments with low intensity CW pump light G<<1 and we can ignore the
any higher order (exponential) gain effects and assume no pump depletion. On
returning through the crystal a second gain process occurs

$I_{s2}=GI_{i}\alpha\mathcal{T}^{2}$ (3)

again ignoring higher order terms in G and where $\alpha$ is the reflectivity
of the target and $\mathcal{T}$ is the (single pass) transmission through the
gas plume. However the two signal fields created are not necessarily in phase
due the differences in relative paths to the mirrors for pump signal and
idler. In the case where $I_{s1}>>I_{s2}$ we can treat this interference
process in same way as homodyne detection which be written as

$I=I_{LO}+I_{s2}+2\sqrt{I_{LO}I_{sig}}\cos\phi\\\ ,$ (4)

In our method (Fig.1), we identify the local oscillator as the signal mode
generated by the SPDC in the first passage through the crystal $I_{LO}=I_{s1}$
and $I_{s2}$ is the potential intensity generated on the second pass while

$\phi=\phi_{p}-\phi_{i}-\phi_{s}$

is the relative phase between the pump, and signal and idler fields.

The signal-to-noise ratio (SNR) can then be analysed in a similar way to
homodyne interferometry [23]. The detected signal on- and off-resonance can be
written in terms of the difference between maximum and minimum photon counts
of an interference fringe received, $R$, and its variance, $V$,

$R=\frac{2\eta T_{int}}{\hbar\omega_{s}}\sqrt{\langle P_{sig}\rangle\langle
P_{LO}\rangle}\\\ ,\mbox{ and}$ (5)

$V\approx\eta T_{int}\frac{\langle P_{LO}\rangle}{\hbar\omega_{s}}\\\ ,$ (6)

where: $\eta$ is the efficiency of the detector, $\hbar\omega_{s}$ is the
energy of a signal photon $\langle P_{sig}\rangle$ is the power of the signal
input, $\langle P_{LO}\rangle$ is the power of the local oscillator and
$T_{int}$ is the measurement time. For comparison with direct detection
schemes we assume signal is estimated from the difference of maximum and
minimum of fringe measured in time $T_{int}/2$. We considered that the system
is shot-noise limited, that is, the noise produced by the photon number
fluctuations is much greater than the environment and detector’s noise. Also,
as the noise provided by the signal is much smaller than the local
oscillator’s noise, we neglect it. So, the SNR on- and off-resonance can be
written in terms of the number of photons combining Eq.(5) and Eq.(6) as

$SNR_{\text{on,off}}=\frac{R_{\text{on,off}}}{\sqrt{V_{\text{on,off}}}}=2\sqrt{\langle
n_{sig}\rangle_{\text{on,off}}}\\\ ,$ (7)

where $\langle n_{sig}\rangle_{\text{on}}$ and $\langle
n_{sig}\rangle_{\text{off}}$ are the average number of photons generated by
the second SPDC in the nonlinear crystal counted by the detector at on- and
off- resonance, respectively

$\langle n_{sig}\rangle_{\text{on,off}}=\eta
G\alpha\mathcal{T}^{2}_{\text{on,off}}T_{int}\frac{P_{i}}{\hbar\omega_{s}}\\\
.$ (8)

where $\alpha$ is the reflectivity of the background and
$\mathcal{T}_{\text{on}}$ and $\mathcal{T}_{\text{off}}$ are the
transmittances of the gas plume on- and off-resonance MIR idler wavelengths,
and $P_{i}$ the power of the idler laser. In this case, we will take the
background as a Lambertian or partially retroreflecting surface. Also, as the
methane lines are narrow, we considered the same gain, detector efficiency,
angular frequency and stimulating laser (idler) power for the on- and off-
resonance frequencies.

##  3 Sensitivity of sensing without detection compared to direct sensing

In the following we use a similar analysis to that used to calculate the
minimum detectable gas concentration in direct detection SWIR DIAL Lidar [25].
This allows us to make direct comparison between the direct detection method
and the non-linear interferometry developed here.

###  3.1 Sensitivity of sensing without detection

The transmittance of the gas plume can be written as

$\mathcal{T}_{\text{on,off}}=e^{-OD_{\text{on},\text{off}}}\\\ ,$ (9)

with

$OD_{\text{on,off}}=Zn_{\text{CH}_{4}}\sigma_{\text{on,off}}\\\ ,$ (10)

where: $OD_{\text{on,off}}$ is the optical density and
$\sigma_{\text{on,off}}$ is the effective absorption cross-section for on- and
off-resonance MIR idler wavelength, and $n_{\text{CH}_{4}}$ and $Z$ are the
number density of methane molecules and the depth of the gas plume,
respectively. Thus, the differential absorption optical depth,
$DAOD=OD_{\text{on}}-OD_{\text{off}}$ becomes

$D_{AOD}=Zn_{\text{CH}_{4}}\left(\sigma_{\text{on}}-\sigma_{\text{off}}\right)=\ln\left(\frac{R_{\text{off}}}{R_{\text{on}}}\right)\
.$ (11)

The quantity of scientific interest, the dry-air average volume mixing ratio
of $\text{CH}_{4}$, $X_{CH_{4}}$, is related to $n_{CH_{4}}$ and number
density of dry air $n_{\text{air}}$ via
$X_{CH_{4}}=n_{CH_{4}}/n_{\text{air}}$. So, Eq.(11) can be reformulated as

$X_{CH_{4}}=\frac{D_{AOD}}{Zn_{\text{air}}\left(\sigma_{\text{on}}-\sigma_{\text{off}}\right)}\\\
.$ (12)

For a Gaussian approximation to Poisson noise, the methane detection
sensitivity of the method can be estimated by Gaussian error propagation.
Propagating the error in Eq.(11), the detection precision can be written as

$\delta X_{CH_{4}}=\sqrt{\left(\frac{\partial X_{CH_{4}}}{\partial
R_{\text{on}}}\right)^{2}V_{\text{on}}+\left(\frac{\partial
X_{CH_{4}}}{\partial
R_{\text{off}}}\right)^{2}V_{\text{off}}}=\frac{\sqrt{SNR^{-2}_{\text{on}}+SNR^{-2}_{\text{off}}}}{Zn_{\text{air}}\left(\sigma_{\text{on}}-\sigma_{\text{off}}\right)}\mbox{,}$
(13)

where $\delta X_{CH_{4}}$ is the minimum methane concentration that can be
sensed by the apparatus. From Eq.(7), it follows that

$\delta
X_{CH_{4}}=\frac{1}{Zn_{\text{air}}\left(\sigma_{\text{on}}-\sigma_{\text{off}}\right)}\sqrt{\frac{\hbar\omega_{s}}{2\eta
G\alpha T_{int}P_{i}}}\mbox{.}$ (14)

As we are calculating the minimum amount of methane that can be detected by
the apparatus, we expect a high transmittance for both on- and off-resonance
idler frequencies, so we made
$\mathcal{T}_{\text{on}}=\mathcal{T}_{\text{off}}\approx 1$.

###  3.2 Sensitivity of direct sensing

The state-of-art methane sensors[13] exploit IPDA and DIAL techniques at SWIR,
around $1.65\mu$m. For this direct sensing method, we will proceed in a
similar way to calculate the sensitivity. The average of number of photons
detected and its variance are

$\langle
n\rangle_{\text{on,off}}=\eta\alpha\mathcal{T}^{2}_{\text{on,off}}T_{int}\frac{P_{\text{on,off}}}{\hbar\omega_{\text{on,off}}}\mbox{,}$
(15)

where: $\eta$ is the efficiency of the detector, $\alpha$ is the reflectivity
of the lambertian background taking into account the geometric losses,
$T_{int}$ is the measurement time, $\mathcal{T}_{\text{on,off}}$ is the
transmittance, $P_{\text{on,off}}$ is the power of the laser and
$\hbar\omega_{\text{on,off}}$ is the energy of a photon, for on- and off-
resonance wavelengths. So, the SNR for the direct sensing in the shot-noise
limited scenario is

$SNR=\sqrt{\langle
n\rangle_{\text{on,off}}}=\mathcal{T}_{\text{on,off}}\sqrt{\eta\alpha
T_{int}\frac{P_{\text{on,off}}}{\hbar\omega_{\text{on,off}}}}\mbox{.}$ (16)

The $D_{AOD}=OD_{\text{on}}-OD_{\text{off}}$ for the SWIR direct sensing is

$D_{AOD}(1.65)=Zn_{\text{CH}_{4}}\left(\sigma_{\text{on}}(1.65)-\sigma_{\text{off}}(1.65)\right)=\frac{1}{2}\ln\left(\frac{\langle
n\rangle_{\text{off}}}{\langle n\rangle_{\text{on}}}\right)\ ,$ (17)

where: $OD_{\text{on,off}}(1.65)$ is the optical density and
$\sigma_{\text{on,off}}(1.65)$ is the effective absorption cross-section for
on- and off-resonance SWIR laser wavelength, and $n_{\text{CH}_{4}}$ and $Z$
are the number density of methane molecules and the depth of the gas plume,
respectively. For the volume mixing ratio we use $XCH_{4}$ the same as the one
for sensing without detection, Eq.(12). It follows that the minimum amount of
methane that can be detected by direct sensing is

$\displaystyle\delta X_{CH_{4}}=$ $\displaystyle\sqrt{\left(\frac{\partial
X_{CH_{4}}}{\partial\langle
n\rangle_{\text{on}}}\right)^{2}V_{\text{on}}+\left(\frac{\partial
X_{CH_{4}}}{\partial\langle n\rangle_{\text{off}}}\right)^{2}V_{\text{off}}}$
(18) $\displaystyle=$
$\displaystyle\frac{1}{Zn_{\text{air}}\left(\sigma_{\text{on}}(1.65)-\sigma_{\text{off}}(1.65)\right)}\sqrt{\frac{\hbar\omega}{2\eta\alpha
T_{int}P}}\mbox{,}$

where we considered the same angular frequency and the same laser power for
on- and off-resonance frequencies, also
$\mathcal{T}_{\text{on}}=\mathcal{T}_{\text{off}}\approx 1$.

###  3.3 The sensitivity ratio between both methods

In this section we will compare the sensitivity between the sensing without
detection and the direct sensing methods. We will analyse a scenario with a
small amount of methane mixed with air in the atmosphere. We define the
relative sensitivity,

$R_{S}=\frac{\delta X_{CH_{4}}\mbox{ for direct measurement}}{\delta
X_{CH_{4}}\mbox{ for sensing without detection}}\ .$ (19)

This number provides us a quantitative comparison between both techniques. If
we analyse a scenario that $R_{S}<1$ it means that the direct sensing is more
sensitive, if $R_{S}>1$ the sensing without detection is more sensitive. If we
make an approximating assumption that detector efficiencies, reflectivity of
the backscattering surface and measurement time are equal for both wavelength
scenarios then taking the ratio of Eq.(14) and Eq.(18) leads to,

$R_{S}\simeq\frac{\sigma_{\text{on}}(3.22)-\sigma_{\text{off}}(3.22)}{\sigma_{\text{on}}(1.65)-\sigma_{\text{off}}(1.65)}\sqrt{G}\
.$ (20)

The values $\sigma$ can be obtained from HITRAN molecular spectroscopy
database[24]. We assumed that the angular frequencies for the signal mode on
the sensing without detection is the same as the one used in the direct
sensing as well the laser’s power, i.e. $\omega_{s}=\omega$ and $P_{i}=P$.

###  3.4 Simulations

Here we are interested in a typical scenario for gas sensing without detection
we might expect parameters in Eq.(14) of the order

$\displaystyle\eta=0.1\mbox{, }G=10^{-8}\mbox{, }\alpha=10^{-8}\mbox{,
}T_{int}=1\text{s}\mbox{, }$ (21) $\displaystyle P_{i}=0.02\text{W}\mbox{ and
}\sigma_{\text{on}}=1.18\times 10^{-22}\frac{\text{m}^{2}}{\text{molec}}\ .$

We considered a low efficiency value for the detector in the simulation as
this reflects typical efficiencies in InGaAs photon counting detectors.
However the high-intensity signal modes generated by SPDC can be easily
detected by a higher (80%) efficiency InGaAs camera. The gain value is
approximately the one calculated for the system we have set in the lab, and
using higher non-linearty or longer crystals. The value estimated for $\alpha$
is realistic if we consider a Lambertian background target a few tens of
meters away from the apparatus. With these parameters we can reach a
sensitivity of $\delta X_{CH_{4}}\approx 187$ppm.m, i. e., we can detect a
methane cloud of $187$ppm concentration and one meter depth. The MIR line
chosen was the one in $3.221\mu$m region, and we used
$n_{\text{air}}=2.53\times 10^{25}$ molec.m-3, $\sigma_{\text{off}}\approx 0$,
$\lambda_{s}=1.589\mu$m.

The MIR absorption cross-section for methane in the line of $3.221\mu$m is
around $65.1$ times greater than the SWIR one at $1.65\mu$m
($\sigma_{on}(1.65)=1.81\times 10^{-24})$[24], so the sensitivity ratio is

$R_{S}\simeq 65.1\sqrt{G}\ ,$ (22)

where we considered $\sigma_{off}(1.65)\approx 0$. Fig. 3 shows the graph of
Eq.(22) in logarithmic scale. Sensitivity ratio $R_{S}$ passes $~{}2.36\times
10^{-4})$ assuming our simplistic approximations hold (equal
$P,\alpha,\eta,T_{int}$ etc.).

Figure 3: Variation of the ratio of sensitivity ratio $R$ (Eq.(22))with $G$
for the two different methods Figure 4: The variation of $\delta XCH_{4}$ (in
ppm.m) with $G$ at different values of $\alpha$ shown on log-log scales.

Figure 4 shows the variation of $\delta XCH_{4})$ with $G$ for various values
of return loss $\alpha$. What is significant is that for short range scenarios
where a retroreflector is used and $\alpha>10^{-2}$ then the sensitivity is
better than 1 ppm.m for all values of $G$ shown and reaches a few ppB.m at
gain values $G>10^{-4}$.

##  4 Conclusion

This novel gas sensing without detection method enables a new generation of
high-sensitivity methane sensors. The two-colours scheme provided by sensing
without detection, makes it possible to illuminate the gas with a laser in the
MIR region and detect ligth at SWIR. This allows the gas to interact with
light at a wavelength for which it exhibits a high absorption and detecting
modes at shorter wavelengths where low-noise and high-efficiency detectors are
available. At high gain regimes, $G>2.36\times 10^{-4}$, the method shows a
sensitivity greater than the state-of-art IPDA and DIAL sensing techniques.
This kind of nonlinear interferometer can be built in a compact fashion and
light enough to be portable for field use. Analogue to the LIDAR systems, this
one can provide the distance from the background scattering object by
modulating the idler laser with a pseudo-random bit sequence to obtain the
time of flight of the photons. High gains can be reached by using a high-
efficiency crystals, high-power pump laser (or pump build up cavity) and
tighter focussing into the NLC. This method can also be extended for sensing
other gases as the signal and idler wavelengths can be tuned by the pump
wavelength and the crystal.

Funding This work was funded by QuantIC - the UK Quantum Technology Hub in
Quantum Imaging Grant No. 8031 EP/T00097X/1 and the Innovate UK Project
106174: SPLICE.

Acknowledgments The authors acknowledge Xiao Ai for the fruitful discussions
with QLM Technology Ltd., QuantIC and SPLICE project for their support.

Disclosures The authors declare no conflicts of interest.

Data availability Data underlying the results presented in this paper are not
publicly available at this time but may be obtained from the authors upon
reasonable request.

## References

  * [1] T. F. Stocker, D. Qin, G.-K. Plattner, M. Tignor, S. K. Allen, J. Boschung, A. Nauels, Y. Xia, V. Bex and P. M. Midgley, eds., “Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change,” (Cambridge University Press, Cambridge, United Kingdom and New York, NY, USA, 2013).
  * [2] M. Etminan, G. Myhre, E. J. Highwood, K. P. Shine, “Radiative forcing of carbon dioxide, methane, and nitrous oxide: A significant revision of the methane radiative forcing,” Geophysics Research Letters 43, 12614–12623 (2016).
  * [3] T. Blunier, J. A. Chappellaz , J. Schwander, J.-M. Barnola, T. Desperts, B. Staufter, D. Rayna, “Atmospheric methane, record from a Greenland ice core over 1000 year,” Geophysics Research Letters 20, 2219–2222 (1993).
  * [4] D. Etheridge, L. Steele, R. Francey, R. Langenfelds, “Atmospheric methane between 1000 A.D. and present: evidence of anthropogenic emissions and climatic variability,” Jounal of Geophysical Research 103, 15979–15993 (1998).
  * [5] D. J. Wuebbles, K. Hayhoe, “Atmospheric methane and global change,” Earth-Science Reviews 57, 177–210 (2002).
  * [6] P. Bousquet, P. Ciais, J. B. Miller, E. J. Dlugokencky, D. A. Hauglustaine, C. Prigent, G. R. Van der Werf, P. Peylin, E.-G. Brunke, C. Carouge, R. L. Langenfelds, J. Lathière, F. Papa, M. Ramonet, M. Schmidt, L. P. Steele, S. C. Tyler, J. White, “Contribution of anthropogenic and natural sources to atmospheric methane variability,” Nature 443, 439–443 (2006).
  * [7] D. Shindell, J. C. I. Kuylenstierna, E. Vignati, R. van Dingenen, M. Amann, Z. Klimont, S. C. Anenberg, N. Muller, G. Janssens-Maenhout, F. Raes, J. Schwartz, G. Faluvegi, L. Pozzoli, K. Kupiainen, L. Höglund-Isaksson, L. Emberson, D. Streets, V. Ramanathan, K. Hicks, N. T. Kim Oanh, G. Milly, M. Williams, V. Demkine, D. Fowler, “Simultaneously mitigating near-term climate change and improving human health and food security,” Science 335, 6065 183–189 (2012).
  * [8] A. Butz, S. Guerlet, O. Hasekamp, D. Schepers, A. Galli, I. Aben, C. Frankenberg, J.-M. Hartmann, H. Tran, A. Kuze, G. Keppel-Aleks, G. Toon, D. Wunch, P. Wennberg, N. Deutscher, D. Griffith, R. Macatangay, J. Messerschmidt, J. Notholt, T. Warneke, “Toward accurate CO2 and CH4 observations from GOSAT,” Geophysics Research Letters 38, L14812 (2011).
  * [9] D. J. Jacob, A. J. Turner, J. D. Maasakkers, J. Sheng, K. Sun, X. Liu, K. Chance, I. Aben, J. McKeever, C. Frankenberg, “Satellite observations of atmospheric methane and their value for quantifying methane emissions,” Atmospheric Chemistry and Physics 16, 14371–14396 (2016).
  * [10] L. M. Golston, L. Tao, C. Brosy, K. Schäfer, B. Wolf, J. McSpiritt, B. Buchholz, D. R. Caulton, D. Pan, M. A. Zondlo, D. Yoel, H. Kunstmann, M. McGregor, “Lightweight mid-infrared methane sensor for unmanned aerial systems,” Applied Physics B 123, 170 (2017).
  * [11] K. C. Cossel, E. M. Waxman, F. R. Giorgetta, M. Cermak, I. R. Coddington, D. Hesselius, S. Ruben, W. C. Swann, Gar-Wing Truong, G. B. Rieker, N. R. Newbury, “Open-path dual-comb spectroscopy to an airborne retroreflector,” Optica 4, 724-728 (2017).
  * [12] N. Cezard, S. Le Mehaute, J. Le Gouët, M. Valla, D. Goular, D. Fleury, C. Planchat, A. Dolfi-Bouteyre, “Performance assessment of a coherent DIAL-Doppler fiber Lidar at 1645 nm for remote sensing of methane and wind,” Optics Express 28, 22345–22357 (2020).
  * [13] J. Titchener, D. Millington-Smith, C. Goldsack, G. Harrison, A. Dunning, X. Ai, M. Reed, “Single photon Lidar gas imagers for practical and widespread continuous methane monitoring,” Applied Energy 306, Part B 118086 (2022).
  * [14] G. B. Lemos, V. Borish, G. D. Cole, S. Ramelow, R. Lapkiewicz, A, Zeilinger, “Quantum imaging with undetected photons,” Nature 512, 409–412 (2014).
  * [15] A. C. Cardoso, L. P. Berruezo, D. F. Ávila, G. B. Lemos, W. M. Pimenta, C. H. Monken, P. L. Saldanha, S. Pádua, “Classical imaging with undetected light,” Physics Review A 97, 033827 (2018).
  * [16] S. K. Lee, T. H. Yoon, M. Cho, “Molecular Rovibrational Spectroscopy with Undetected Photons via Single-Photon Interferometry,” Physics Review Applied 14, 014045 (2020).
  * [17] I. Kviatkovsky, H. M. Chrzanowski, E. G. Avery, H. Bartolomaeus, S. Ramelow, “Microscopy with undetected photons in the mid-infrared,” Science Advances 6, 42 (2020).
  * [18] I. Kviatkovsky, H. M. Chrzanowski, S. Ramelow, “Mid-infrared microscopy via position correlations of undetected photons,” Optics Express 30, 5916-5925 (2022).
  * [19] S. Töpfer, M. G. Basset, J. Fuenzalida, F. Steinlechner, J. P. Torres, M. Gräfe, “Quantum holography with undetected light,” Science Advances 8, 2 (2022).
  * [20] G. J. Machado, G. Frascella, J. P. Torres, M. V. Chekhova, “Optical coherence tomography with a nonlinear interferometer in the high parametric gain regime,” Applied Physics Letters 117, 094002 (2020).
  * [21] A. Vanselow, P. Kaufmann, I. Zorin, B. Heise, H. M. Chrzanowski, S. Ramelow, “Frequency-domain optical coherence tomography with undetected mid-infrared photons,” Optica 7, 1729-1736 (2020).
  * [22] M. Kutas, B. Haase, P. Bickert, F. Riexinger, D, Molter, G. von Freymann, “Terahertz quantum sensing,” Science Advances 6, 11 (2020).
  * [23] X. Sun, J. B. Abshire, "Signal to noise ratios of pulsed and sinewave modulated direct detection lidar for IPDA measurements," in Conference on Lasers and Electro-Optics (CLEO) - Laser Science to Photonic Applications (2011), pp. 1-2.
  * [24] L. Rothman et al., “The HITRAN 2012 molecular spectroscopic database,” Journal of Quantitative Spectroscopy and Radiative Transfer 130 4-50 (2013).
  * [25] M. Quatrevalet, X. Ai, A. Perez-Serrano, P. Adamiec, J. Barbero, A. Fix, J. M. G. Tijero, I. Esquivias, J. G. Rarity, G. Ehret, Atmospheric CO2 Sensing with a Random Modulation Continuous Wave Integrated Path Differential Absorption Lidar, IEEE Journal of Selected Topics in Quantum Electronics 23 (2), 157-167 (2017).