# Learning Graph Regularisation for Guided Super-Resolution

Riccardo de Lutio1, Alexander Becker1,∗ Stefano D’Aronco1
Stefania Russo1 Jan D. Wegner1,2 Konrad Schindler1
1EcoVision Lab, Photogrammetry and Remote Sensing, ETH Zurich
2Institute for Computational Science, University of Zurich
<EMAIL_ADDRESS>Equal contribution.

###### Abstract

We introduce a novel formulation for guided super-resolution. Its core is a
differentiable optimisation layer that operates on a learned affinity graph.
The learned graph potentials make it possible to leverage rich contextual
information from the guide image, while the explicit graph optimisation within
the architecture guarantees rigorous fidelity of the high-resolution target to
the low-resolution source. With the decision to employ the source as a
constraint rather than only as an input to the prediction, our method differs
from state-of-the-art deep architectures for guided super-resolution, which
produce targets that, when downsampled, will only approximately reproduce the
source. This is not only theoretically appealing, but also produces crisper,
more natural-looking images. A key property of our method is that, although
the graph connectivity is restricted to the pixel lattice, the associated edge
potentials are learned with a deep feature extractor and can encode rich
context information over large receptive fields. By taking advantage of the
sparse graph connectivity, it becomes possible to propagate gradients through
the optimisation layer and learn the edge potentials from data. We extensively
evaluate our method on several datasets, and consistently outperform recent
baselines in terms of quantitative reconstruction errors, while also
delivering visually sharper outputs. Moreover, we demonstrate that our method
generalises particularly well to new datasets not seen during training.

##  1 Introduction

Guided super-resolution takes as input two images of different resolution, a
low-resolution source and a high-resolution guide from a different domain. It
returns a high-resolution version of the source as output, termed the target.
This task is relevant in many practical applications such as medical [62] and
satellite imaging [26], where performing a diagnosis or analysis from low-
quality images can be extremely difficult. Another very popular example in
computer vision is upsampling depth maps, where the low-resolution depth is
the source, a conventional grayscale or RGB image is the guide, and the target
is a high-resolution depth map. Consumer-grade depth sensors provide low-
resolution depth maps, but a high-resolution RGB camera is usually mounted on
the same device and can acquire a high-resolution image of the same scene.

Figure 1: Our method takes as input a low-resolution source image and a high-
resolution guide image of another modality to build a graph using high-level
image features. The graph is then used in a differentiable optimisation layer
as a regularisation to reconstruct the target.

Guided super-resolution methods can be divided into two main categories,
conventional and deep learning-based methods. The former typically cast the
task into an optimisation problem [10, 8, 7, 12]. The goal is to create a
high-resolution target image that, when downsampled, matches the source, while
at the same time complying with an appropriate regularisation term that
favours a desired image characterstic such as (piecewise) smoothness. Deep
learning methods [19, 17, 15, 18, 52] instead rely on a dataset of
source/guide/target triplets to learn a feed-forward mapping from the source
and guide to the target. To that end the model must learn the statistical
correlations that allow it to transfer high-frequency details from the guide
to the target, while, at the same time, ensuring that the predicted target
stays close to the source. A considerable advantage of the conventional
approach is that, by individually solving a properly formulated optimisation
for each image, the prediction is usually guaranteed to match the source. On
the other hand, designing an adequate regularisation term based on low-level
image features is a complicated task. Deep learning methods exhibit rather
complementary strengths: as long as one has access to enough training data and
that data is representative of the images encountered at test time, these
methods tend to perform very well, due to the unmatched ability of deep
networks to mine complex, highly informative features from images. On the
other hand, with limited training data, or when there is a domain shift
between the training and the test set [54], feed-forward methods can no longer
guarantee that downsampling the predicted target will produce the source, thus
contradicting the fundamental relation behind super-resolution.

In this work, we show how to combine the two schools, and learn the graph of
an optimisation-based super-resolution scheme, – see Figure 1. In particular,
we learn a mapping from the two inputs (source and guide) to the edge
potentials (also called edge weights) of an affinity graph between pixels of
the target. The learned graph serves as the regulariser for an optimisation-
based reconstruction of the high-resolution target, which is particularly
suited for signals with a piecewise smooth structure. This entire mapping is
trained end-to-end: the mapping function, which is parametrised as a
convolutional network, is learned from training data, by back-propagating the
gradients of the loss through the optimisation layer. CRF-RNN [9] also
proposed to perform an online optimisation and include a graph in their
network for semantic segmentation. However, they construct a dense graph and
use an RNN to approximate the inference of the posterior. In contrast, we show
that a sparse, local graph is sufficient while performing exact maximum a
posteriori inference. We test our method on three different guided depth
super-resolution datasets and show that it compares favourably against
conventional and deep learning-based methods, across a range of upsampling
factors from $\times 4$ to $\times 16$. We further show that our proposed
method is much more robust to distribution shifts and can effectively
generalise across datasets.

In summary, the contributions of this paper are the following: _(i)_ We
introduce a novel formulation of guided super-resolution, where a deep feature
extractor is trained to derive the edge potentials for a graph-based energy
minimisation from the input (source and guide) images; _(ii)_ we develop a
differentiable optimiser for the graph regularisation, taking advantage of the
sparse graph connectivity to efficiently process large input patches up to
$256^{2}$ pixels111Code is available at https://github.com/prs-eth/graph-
super-resolution; _(iii)_ in this way, our scheme therefore combines the power
of learned, deep feature extractors with large receptive fields and the rigor
of graph-based optimisation in an end-to-end trainable framework. As a result,
it produces crisp, natural-looking images that correctly adhere to the
underlying image formation model.

##  2 Related Work

At a conceptual level, guided super-resolution can be seen as a form of guided
filtering [13], where the source image is first naively upsampled to match the
target resolution, and then enhanced with some transformation that is guided
by the local structure of both the (upsampled) source and the guide.

###  2.1 Optimisation Methods

Local optimisation methods are variants of the filtering procedure described
above. Here, the source is first upsampled, then a local filter controlled by
the values of the guide [24, 60] is applied to it. Extensions of these methods
include the use of geodesic distances to define the filter [30], or
constructing it by combining the contrast in both the guide and source images
[3].

Global optimisation methods construct a global energy function over all pixels
and minimise it to obtain the target. The energy generally consists of two
parts: a data fidelity term that ensures that the target stays close to the
source, and a prior term that regularises the otherwise ill-posed problem of
super-resolution. Data fidelity is typically defined as a distance term
between the source and the downsampled target. The regulariser, in the guided
setting, is not an isotropic smoothing but it is modulated by the guide.
Depending on the parametrisation, the global energy minimisation can be viewed
as Markov Random Field (MRF) inference [7], as a form of non-local means [35],
or as a variational inference with an anisotropic version of total generalised
variation (TGV) [10]. Some works [58, 59] have also proposed to replace the
TGV prior with an auto-regressive model. The fast bilateral solver [2] solves
a sparse linear system [1] to obtain bilateral-smooth outputs with sharp
discontinuities. The SD filter [12] formulates guided image filtering as a
non-convex optimisation problem that exploits static and dynamic guidance. The
Pixtransform method [5] estimates a mapping from guide to target individually
for each pixel and spatially smoothens the mapping function, rather than the
target output. In a similar spirit, [34] predicts the target as a linear
function of the guide, with coefficients that vary spatially, modulated by the
guide and the source. The Guided Deep Decoder (GDD) [55] adapts the deep image
prior [56] to guided super-resolution of hyper-spectral images. A random noise
map is decoded into a target that has maximal data fidelity to the source,
guided by feature maps obtained by a joint encoder-decoder branch for the
guide. Cross-Modality Super-Resolution (CMSR) [48] also fits a neural
architecture to the individual source/guide pair, allowing to optimise for
individual alignment errors. [36] proposes to (over-)segment the images and
attribute planar disparities to each super-pixel, while smoothness of the
disparities across super-pixels is encouraged by connecting them into an MRF.

In [42], the authors also propose to build a graph to encourage smoothness of
the target in regions where the guide is also smooth. Contrary to our work,
their graph is based on raw colour differences (similar to [7, 36]) whereas
our graph encodes affinity between deep, latent features, derived not only
from the guide but also from the source, and trained in an end-to-end fashion
to optimally support the super-resolution task.

###  2.2 Learning Methods

The other large family of guided super-resolution methods are learning-based.
Following a general trend towards supervised machine learning, the hope is
that one can outperform conventional models by learning from data how to best
fuse the source and the guide to recover the target. Perhaps the first
learning-based methods for guided super-resolution were those learning
dictionaries of source, guide and target patches. At test time, the source and
guide were then (soft-)matched to the dictionary to retrieve suitable target
patches and assemble the target image [29, 25].

More recently, deep learning methods have become predominant for guided super-
resolution. These approaches work by parametrising the non-linear mapping from
the two inputs — guide and source — to the target as a convolutional neural
network, and learning its weights directly. The deep joint image filter [27,
28] feeds the upsampled source and the guide directly into a standard encoder-
decoder architecture. The deep primal-dual network [40] follows a similar
strategy, but outputs a residual correction to the naively upsampled source.
Additionally, the output is refined with non-local total variation, unrolled
into a sequence of network layers. The Multi-Scale Guided network (MSG-Net)
[17] implements a new strategy, to _en_ code only the guide, extract rich
hierarchical features at different levels of the encoder, and append them to
the corresponding levels of a network that _de_ codes the source into the
target through a final reconstruction layer. This integrated multi-scale
guidance from the guide to the upsampled source allows to resolve ambiguity in
depth map upsampling. This design has inspired several other works: PMBANet
[61] adds multi-branch aggregation blocks; the Fast Depth Super-Resolution
network (FDSR) [15] adds a high-frequency layer to extract fine details from
the guide, and strives for a computationally efficient, yet effective design.
DepthSR-Net [11] integrates the idea in a residual U-Net architecture [41].
First, the source is naively upsampled to the desired resolution, then the
residuals between this naive interpolation and the corresponding target are
learned using the hierarchical features as input pyramid in the encoder
structure. In [57], an explicit coarse-to-fine cascade of networks is used to
iteratively refine the output and progressively add high-frequency details. In
[52] two networks are trained collaboratively, one for monocular depth
estimation from the guide and one to super-resolve the source. Furthermore,
there is an auxiliary structure prediction task to mitigate differences
between depth and intensity discontinuities. Also in a very recent work, [43]
explores learning depth super-resolution from unpaired data, using a learnable
degradation model, and surface normal estimates as additional features to
obtain more accurate depth maps.

Several authors have experimented with modifications of the basic
Convolutional Neural Network (CNN) layers to enable modulation based on the
guide. The Pixel-Adaptative Convolutional (PAC) network [51] proposes a novel
type of learned filters where the convolution is conditioned on other
features. For guided super-resolution, these conditioning features are
extracted from the guide. Channel attention is used in [50] to improve super-
resolution of channels with abundant high-frequency content. The Deformable
Kernel Network (DKN) [18] applies sparse, spatially-variant kernels to predict
a set of neighbours and associated weights for each target pixel, such that
their weighted mean yields the pixel’s value.

##  3 Method

###  3.1 Notation and Problem Statement

Throughout, we denote matrices and higher-order tensors with uppercase bold
letters ${\bm{A}}$, and their flattened, 1D vector version with corresponding
lowercase letters ${\bm{a}}$. In our guided super-resolution setting we are
given a guide ${\bm{G}}$ with spatial dimensions $H\\!\times\\!W$ and $C$
channels, as well as a low-resolution source ${\bm{S}}$ of dimension
$h\\!\times\\!w$. For simplicity, we will assume that the source has a single
channel, as the extension to multiple channels is straight-forward. The ratio
between the spatial dimensions of the guide and the source is the upsampling
factor $k=H/h=W/w$. The goal is to upsample ${\bm{S}}$ to a target ${\bm{Y}}$
with the same spatial resolution of ${\bm{G}}$. We denote with ${\bm{D}}$ the
downsampling operator that maps ${\bm{y}}$ to ${\bm{s}}$. In our case the
downsampling is a weighted average over a $k\times k$ window of the image
${\bm{Y}}$ (a _point spread function_). Note that some authors instead assume
that ${\bm{S}}$ is not downsampled, but rather represents a sparsely sampled
version of the target ${\bm{Y}}$, which need not be deconvolved. Due to the
finite area of pixels on the sensor, respectively the beam divergence in
laser-based scanners, this sparse subsampling without low-pass filtering is
not a very realistic model for most practical sensing systems.

Figure 2: The architecture of our approach. A neural network backbone is
employed to extract deep feature maps from the guide and source images. A
graph over pixels is constructed based on pairwise affinities derived from
these feature maps. Finally, a quadratic optimisation problem is solved for a
target image that is in agreement with both the low-resolution source and the
structure of the graph. Crucially, the graph optimisation layer is
differentiable and our method is thus end-to-end trainable.

###  3.2 Graph Regularisation

A natural way to formalise the guided super-resolution problem mathematically
is as an energy minimisation:

$\underset{{\bm{y}}}{\text{argmin}}\quad
f\big{(}{\bm{D}}{\bm{y}},{\bm{s}}\big{)}+\lambda\cdot r({\bm{y}}),$ (1)

where $f$ is the data fidelity term that measures how well the downsampled
target matches the source, and $r$ is a prior, respectively regulariser for
the reconstructed target, and $\lambda$ a parameter that weights the effect of
the regularisation.

The data fidelity term serves to ensure similarity between ${\bm{D}}{\bm{y}}$
and the source ${\bm{s}}$, typically in the form of an $l_{1}$ or (squared)
$l_{2}$ norm. In this work we use the latter,
$f({\bm{D}}{\bm{y}},{\bm{s}})=\|{\bm{D}}{\bm{y}}-{\bm{s}}\|_{2}^{2}$.

An effective regulariser that has often been used successfully for images [7,
42] is to encourage smoothness of the reconstructed signal w.r.t. some graph
defined over the image pixels. The affinity matrix of that graph is denoted by
${\bm{A}}$ and has size $HW\times HW$. It describes which pixels are
connected, i.e., have direct, first-order influence on each other. The generic
element $A_{ij}$ represents the weight of the edge connecting pixel $i$ to
pixel $j$, for all pairs of pixels that are not directly connected,
$A_{ij}=0$. The degree matrix ${\bm{U}}$ is a diagonal matrix with entries
constructed by summing the weights of all edges that meet at a node,
$U_{ii}=\sum_{j}A_{ij}$. Finally, the graph Laplacian ${\bm{L}}$ is defined as
${\bm{L}}={\bm{U}}-{\bm{A}}$. For a signal defined on the graph nodes – in our
case the image – the quantity ${\bm{y}}^{T}{\bm{L}}{\bm{y}}$ is a measure of
how smooth that signal is on the graph. Encouraging smoothness is an effective
regulariser, provided that the graph matches the intrinsic structure of the
signal. The objective then becomes:

$\underset{{\bm{y}}}{\text{argmin}}\quad\|{\bm{D}}{\bm{y}}-{\bm{s}}\|_{2}^{2}+\lambda{\bm{y}}^{T}{\bm{L}}{\bm{y}}\;.$
(2)

What remains is to construct the right graph, i.e., to determine the “natural”
connectivity between the pixels of the target ${\bm{Y}}$. This is not trivial,
but in guided super-resolution we can leverage the guide ${\bm{g}}$, which
shares the same, high resolution with ${\bm{Y}}$. The graph thus becomes a
function ${\bm{L}}({\bm{g}})$ of the guide. Note that this does not imply
constructing the graph from the guide’s raw brightness (resp., contrast)
values. Rather, one may as well derive it from more abstract per-pixel
features. As we will show, a particularly useful procedure is to learn those
features from the data, such that the graph is optimally adapted to the
specific super-resolution task at hand.

In order to tailor the graph structure of the regulariser to the problem, we
feed both the source and the guide through a CNN, to obtain a deep feature
representation ${\bm{F}}=f_{\bm{\theta}}({\bm{G}},{\bm{S}})$ of size
$H\\!\times\\!W\\!\times\\!M$, with $M$ the channel depth of the
representation and ${\bm{\theta}}$ the trainable parameters of the network.
For efficiency, we restrict the graph to have fixed topology, where each pixel
is connected (at most) to its 4-neighbours in the 2D pixel lattice. Longer-
range connectivities are in principle possible, but greatly increase the
computational effort, with rapidly diminishing returns. Indeed, our setup
attaches deep features to the graph nodes, which encode a large receptive
field and capture semantic and long-range information in the guide image
beyond the 4-neighbour topology. The weights of the graph edges are defined as
standard negative exponential affinities between the learned features:

$A_{ij}=e^{-\frac{\|F_{i}-F_{j}\|_{2}^{2}}{M\mu}},$ (3)

where $\mu$ is a learnable scaling parameter.

###  3.3 Optimisation Layer

Let ${\bm{y}}^{\star}$ denote the minimiser of Eq. (2), and
${\bm{y}}_{\text{gt}}$ the ground truth target values of some training set. We
can then assemble triplets $({\bm{g}},{\bm{s}},{\bm{y}}_{\text{gt}})$ and
optimise the graph construction to minimise the loss between
${\bm{y}}^{\star}$ and ${\bm{y}}_{\text{gt}}$:

${\bm{\theta}}*=\underset{{\bm{\theta}}}{\text{argmin}}\quad\mathbb{E}_{p({\bm{g}},{\bm{s}},{\bm{y}}_{\text{gt}})}\big{[}l\big{(}{\bm{y}}^{\star}({\bm{\theta}}),{\bm{y}}_{\text{gt}}\big{)}\big{]}\;,$
(4)

where $l$ is an appropriate loss function, for instance an $l_{1}$ loss or a
Mean Squared Error (MSE) loss.

This means that, in order to train the feature extractor, we must compute the
gradient of the loss in Eq. (4) w.r.t. the graph. Towards that end, we first
notice that Eq. (2) is a quadratic problem, and equivalent to solving the
linear equation system:

$\big{(}\lambda{\bm{L}}({\bm{\theta}})+{\bm{D}}^{T}{\bm{D}}\big{)}{\bm{y}}^{\star}={\bm{D}}^{T}{\bm{s}},$
(5)

here we have made it explicit that the graph Laplacian ${\bm{L}}$ is the only
term that depends on the network parameters ${\bm{\theta}}$. For error
backpropagation we must map the gradient $\frac{\partial
l}{\partial{\bm{y}}^{\star}}$ w.r.t. the reconstructed image to the entries of
${\bm{L}}$. Using the implicit function theorem [2] we obtain:

$\tfrac{\partial l}{\partial{\bm{L}}}=-\lambda\tfrac{\partial
l}{\partial{\bm{D}}^{T}{\bm{s}}}{{\bm{y}}^{\star}}^{T},\;(\lambda{\bm{L}}({\bm{\theta}})+{\bm{D}}^{T}{\bm{D}})\tfrac{\partial
l}{\partial{\bm{D}}^{T}{\bm{s}}}=\tfrac{\partial l}{\partial{\bm{y}}^{\star}}$
(6)

In order to backpropagate the loss, we must solve a second linear equation
system, which then yields the derivatives for individual entries of the graph
Laplacian. Note that the derivative w.r.t. the Laplacian is a dense matrix,
which is impractical (e.g., for an image of $256^{2}$ pixels this matrix has
$\approx 4$ billion elements). Fortunately, we can exploit the fact that the
graph topology is fixed and compute the gradient only w.r.t. non-zero entries
of ${\bm{L}}$ (i.e., index pairs of the 4-neighbourhood). Once the gradients
for the graph weights have been computed, they are propagated through the deep
feature extractor.

Finally, we summarise our proposed model, see Figure 2. The feature extractor
$f_{\bm{\theta}}({\bm{G}},{\bm{S}})$ computes the deep features from the guide
and source images, and these features inform the weights of a 4-neighbour
graph. The graph, together with the source ${\bm{S}}$, forms the input to the
optimisation problem of Eq. (2) that estimates the target. During training, a
loss computed between the prediction and the ground truth steers the feature
extraction such that the graph weights optimally regularise the prediction of
the high-resolution target. Note that at test time we must solve a quadratic
problem in order to predict the target image. To do so, very efficient
algorithms are available, although it is of course not as fast as a
conventional forward pass.

##  4 Experimental Results

In this section we describe the evaluation of our proposed method on the task
of RGB-guided depth map super-resolution. We conduct our experiments on three
widely used RGB-D datasets. For each dataset, we compare our approach to
several guided super-resolution baselines. All algorithms are evaluated at 3
upsampling factors – $\times 4$, $\times 8$ and $\times 16$ – on the following
datasets:

Middlebury [45, 47, 44, 46, 16] We use all 50 RGB-D images available from the
Middlebury 2005-2014 datasets. We split the data randomly into 40 images for
training, 5 for validation and 5 for testing. A challenging aspect of this
dataset is that it contains missing values in the depth ground truth. For
generating the source, we therefore only take into account valid pixels during
downsampling. Furthermore, we generate a pixel validity mask for both the
target and source, so we can ignore the invalid pixels during training and
testing.

NYUv2 [31] consists of 1449 RGB-D images captured with a Microsoft Kinect. We
randomly split these into 849 images for training, 300 for validation and 300
for testing.

DIML [21, 20, 22, 4] is a large-scale dataset comprised of 2M RGB-D frames in
total. For our evaluation, we use the high-resolution indoor sample subset,
which was acquired using a Microsoft Kinect. From this data we construct a
split of 1440 images for training, 169 images for validation and 503 images
for testing.

We compare our model to the Guided Filter (GF) [13], the Static/Dynamic filter
(SD) [12], the Pixtransform [5], the MSG-Net [17], the Deformable Kernel
Network (DKN) and its fast version (FDKN) [18], the PMBANet [61], and finally
to the Fast Depth Super-Resolution (FDSR) [15]. We were not able to compare to
the recent work by [52] since no code has been released at the time of
writing. For all other methods, we use the respective publicly available code.
We implemented our method using PyTorch [37]. The graph-based optimisation
layer is realised using the sparse matrix support of the CuPy library [33],
allowing for a GPU-accelerated implementation of the forward and backward
pass. In order to solve the linear systems of equations needed in the
optimisation, we implement the conjugate gradient method [32]. We use a U-Net
[41] network with a ResNet-50 [14] encoder pretrained on ImageNet [6] as
feature extractor for the graph weights prediction. We also use a simple
gradient clipping as it improves the stability of the training procedure. As
baseline we further compare to a version of the proposed method where the RGB
features of the guide and the bicubic upsampling of the source are the only
pixel features used to construct the graph, i.e., no deep feature extractor is
used.

We train all learned methods using the Adam [23] optimiser. Varying with the
specific dataset, we fix the same batch size, initial learning rate and
scheduling strategy for all methods. For fairness of comparison, we trained
all learned methods with both our configuration for these hyperparameters and
their original ones (when indicated) and report the best results. The detailed
hyperparameter settings for each dataset and method as well as additional
experimental results can be found in the supplementary material. For all
learned methods we further make use of data augmentations during training,
consisting of random cropping, random horizontal flipping and random rotation,
where the rotation angle is sampled from
$\mathcal{U}(-15^{\circ},15^{\circ})$. All methods are evaluated on patches of
$256^{2}$ pixels, however, for some methods training on such large patches was
infeasible due to memory constraints, in which case we used patches of size
$128^{2}$ or even $64^{2}$ (for PMBANet with a factor $\times 4$).

###  4.1 Learning Graph Weights

Figure 3: Importance of learned edge potentials. We visualise the total
affinity of each pixel to its four neighbours when derived from raw colour
(top) or from deep features (bottom). Examples are from the Middlebury test
set.

As previously mentioned, we believe that smoothing based on a graph that is
defined on four _local_ neighbours only is adequate for the problem at hand,
as long as the features used to create the graph encode a sufficiently large
context. This is of course the case when using deep features that are learned
from an entire dataset using a CNN with a large receptive field. In Figure 3,
we compare the graph obtained from plain colour information to the graph
obtained from learned deep features for two selected examples from the
Middlebury test set. The graph is visualised by displaying, for each pixel,
the sum of the four edges that connect it to its neighbours. Areas where the
connections of the nodes are strong appear in yellow, and in these areas the
graph regularisation will enforce smoothness in the predicted target. On the
other hand, in areas where the edge weights are small, the graph smoothness
term is weak (it disappears as weights approach zero) and the target is
allowed to reveal depth discontinuities. Figure 3 demonstrates that the
proposed method is able to extract semantical information from the guide image
and to transfer it to the edge potentials. In the top example, the model has
learned that a high-contrast text is part of the surrounding object, thus it
predicts high edge weights for the respective image area and effectively
enforces smooth depth. This correctly results in the text not being carried
over to the target, an effect that is observable in the prediction based on
colour features. On the other hand, the bottom image shows that our model has
learned to detect object boundaries and highlights them even if both
background and foreground have very similar colours. In fact, the learned edge
weights are much lower around the depth discontinuities compared to the
colour-based weights, which implies that the learned graph is able to recreate
a sharp edge in the target prediction. In contrast, the graph built from
colour information cannot perform a proper cut, leading to bleeding artifacts.

|  |  | GF [13] | SD filter [12] | Pixtransform [5] | MSG-Net [17] | DKN [18] | FDKN [18] | PMBANet [61] | FDSR [15] | Ours - Colour | Ours
---|---|---|---|---|---|---|---|---|---|---|---|---
Middlebury | $\times 4$ | MSE | 33.3 | 24.9 | 39.8 | 4.13 | 4.29 | 3.60 | 4.72 | 7.72 | 14.8 | 3.04
MAE | 1.27 | 0.46 | 0.79 | 0.22 | 0.18 | 0.16 | 0.25 | 0.35 | 0.42 | 0.13
$\times 8$ | MSE | 40.5 | 82.5 | 32.7 | 10.5 | 11.2 | 10.4 | 9.48 | 23.2 | 68.3 | 7.26
MAE | 1.49 | 0.86 | 0.82 | 0.43 | 0.38 | 0.37 | 0.38 | 0.69 | 0.83 | 0.24
$\times 16$ | MSE | 67.4 | 511 | 41.5 | 34.2 | 47.6 | 38.5 | 30.6 | 55.4 | 297 | 24.7
MAE | 2.21 | 1.73 | 1.24 | 1.06 | 1.42 | 1.18 | 0.89 | 1.51 | 1.69 | 0.67
NYUv2 | $\times 4$ | MSE | 114 | 36.0 | 112 | 6.85 | 11.4 | 8.07 | 10.8 | 10.5 | 19.0 | 6.45
MAE | 3.91 | 1.31 | 3.61 | 0.81 | 1.03 | 0.85 | 0.93 | 0.94 | 1.11 | 0.73
$\times 8$ | MSE | 142 | 105 | 122 | 24.1 | 29.8 | 29.9 | 31.5 | 35.4 | 68.4 | 19.6
MAE | 4.47 | 2.57 | 3.86 | 1.66 | 1.82 | 1.80 | 1.79 | 1.96 | 2.30 | 1.42
$\times 16$ | MSE | 249 | 533 | 219 | 84.5 | 115 | 113 | 84.9 | 179 | 264 | 67.5
MAE | 6.34 | 5.07 | 5.40 | 3.35 | 4.01 | 3.95 | 3.26 | 4.68 | 4.56 | 2.90
DIML | $\times 4$ | MSE | 25.6 | 10.5 | 20.7 | 1.73 | 3.47 | 2.2 | 3.05 | 2.75 | 7.02 | 1.68
MAE | 1.45 | 0.40 | 1.15 | 0.22 | 0.33 | 0.23 | 0.31 | 0.29 | 0.35 | 0.20
$\times 8$ | MSE | 34.1 | 44.9 | 23.0 | 4.13 | 5.47 | 5.95 | 5.87 | 8.40 | 15.2 | 3.51
MAE | 1.77 | 0.83 | 1.26 | 0.40 | 0.45 | 0.47 | 0.47 | 0.66 | 0.67 | 0.31
$\times 16$ | MSE | 66.3 | 411 | 39.3 | 13.0 | 19.3 | 20.8 | 13.8 | 32.9 | 133 | 9.45
MAE | 2.74 | 1.91 | 1.78 | 0.93 | 1.20 | 1.24 | 0.87 | 1.66 | 1.72 | 0.68

Table 1: Performance comparison with the state-of-the-art algorithms on the
Middlebury [45, 47, 44, 46, 16], NYUv2 [49] and DIML [21, 20, 22, 4] datasets
for different values of upsampling factors. The table shows the MSE (in
$\text{cm}^{2}$) and MAE (in cm).

Testing Dataset |  | GF [13] | SD filter [12] | Pixtransform [5] | MSG-Net [17] | FDKN [18] | PMBANet [61] | FDSR [15] | Ours - Colour | Ours
---|---|---|---|---|---|---|---|---|---|---
DIML | MSE | 34.1 | 44.9 | 23.0 | 5.76 | 6.74 | 7.35 | 7.73 | 20.5 | 4.95
MAE | 1.77 | 0.83 | 1.26 | 0.51 | 0.53 | 0.59 | 0.74 | 0.77 | 0.40
| MSE (low-resolution) | 17.7 | 1.45 | 6.19 | 6.16 | 0.20 | 0.04 | 0.45 | 0.03 | $\mathbf{2.4\cdot 10^{-3}}$
Middlebury | MSE | 40.5 | 82.5 | 32.7 | 11.0 | 10.0 | 9.62 | 18.4 | 23.9 | 8.25
MAE | 1.49 | 0.86 | 0.82 | 0.54 | 0.43 | 0.46 | 0.73 | 0.91 | 0.35
| MSE (low-resolution) | 17.9 | 1.86 | 22.5 | 5.01 | 0.20 | 0.06 | 7.20 | 0.08 | $\mathbf{1.1\cdot 10^{-3}}$

Table 2: Performance comparison with the state-of-the-art algorithms on cross-
dataset generalisation. All learned methods have been trained on the NYUv2
dataset [49]. The table shows the performance of the methods on the DIML [21,
20, 22, 4] and Middlebury [45, 47, 44, 46, 16] datasets for a $\times 8$
upsampling factor. The table shows the MSE (in $\text{cm}^{2}$), MAE (in cm)
and low-resolution MSE (in $\text{cm}^{2}$).

###  4.2 Depth Super-Resolution Evaluation

In Table 1 and Figure 4 we quantitatively and qualitatively compare our method
to all selected baselines for the Middlebury, NYUv2 and DIML datasets. We
outperform all other methods w.r.t. both MSE and MAE metrics for upsampling
factors of $\times 4$, $\times 8$ and $\times 16$. It is observable from the
table that the tested methods perform rather differently among the three
datasets. Conventional methods tend to perform generally worse than the
learned ones. In particular, Pixtransform [5] shows a rather flat performance
curve with mediocre performance on low upsampling factors but also no abrupt
performance drop for higher upsampling factors. In terms of visual results,
the method reveals many artifacts being carried over from the guide. The SD
filter [12] instead has good performance on MAE but MSE performance degrades
fast for larger upsampling factors. Visually it captures some edges very well,
whereas it completely misses and smoothes out others, as seen in Figure 4.
FDKN and DKN [18] attain worse performance than expected across the datasets,
especially quantitatively. It appears that these methods are tuned to sparsely
downsampled source images and not well suited for realistic (not impulse-
shaped) point spread functions. Our method by contrast achieves good
quantitative performance across all three datasets, while producing visually
crisp images. It is particularly effective at larger upsampling factors,
showing the advantages of a hybrid model that leverages a deep learning
backbone alongside a conventional online optimisation layer. Finally, it is
more robust to domain shifts, as we explain in the next paragraph.

#### Cross-dataset generalisation.

A major advantage of our work is that the prediction, after downsampling, is
constrained to match the source. This additional constraint affords the model
better robustness against domain shifts between training and testing (at the
cost of added computation at inference time). To quantify this behaviour, we
perform a cross-dataset generalisation experiment. For all methods, we train
on NYUv2 and test the resulting model on Middlebury and DIML. As can be seen
in Table 2, we outperform all other methods by a significant margin.
Furthermore, our prediction matches the source almost perfectly when
downsampled, as measured by the low-resolution MSE.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

Figure 4: Qualitative comparison of upsampled depth maps. From top to bottom
each group of two rows shows the error of upsampled images, defined as the
difference between the prediction and the ground truth, on the Middlebury [45,
47, 44, 46, 16], NYUv2 [31] and DIML [21, 20, 22, 4] datasets, respectively;
alternating between upsampling factors $\times 8$ and $\times 16$ for each
dataset. From left to right, the first group of columns are (a) Guide, (b)
Source and (c) Ground Truth; the second group includes selected methods from
our quantitative evaluation, (d) SD filter [12], (e) Pixtransform [5], (f)
MSG-Net [17], (g) FDKN [18], (h) PMBANet [61] and (i) FDSR [15]; the last two
columns represent (j) the error for the prediction of our model and (k) the
prediction itself.

###  4.3 Feature Extractor Comparison

We go on to investigate the performance of our method with different feature
extractors. Table 3 compares the errors on the NYUv2 $\times 8$ upsampling
task with different backbones used to extract the features for the graph
regularisation layer, as described in Section 3.2. We have tested several
well-known backbones, always initialising them with weights pretrained on
ImageNet [6]. In addition to these generic backbones, we also evaluate feature
maps extracted from the FDSR network that was specifically designed for guided
depth super-resolution. Finally, to explore the boundaries of feature
extraction, we employ a variant of the _Dense Prediction Transformer_ [39,
38], which extracts multi-scale features for dense prediction tasks (like
mono-depth or semantic segmentation) from a Vision Transformer (ViT). We
adapted the model to account for the (appropriately re-sampled) source image
at each feature level and call it the _Guided Dense Prediction Transformer_
(GDPT).

Feature extractor | # Params | MSE ($\text{cm}^{2}$) | MAE (cm)
---|---|---|---
Colour | 2 | 68.4 | 2.30
UEfficientNet-B2 [53] | 10M | 24.9 | 1.63
UResNet-18 [14] | 14M | 21.7 | 1.52
UResNet-50 [14] | 32M | 19.6 | 1.42
FDSR [15] | 0.6M | 30.4 | 1.75
GDPT [39, 38] | 127M | 22.3 | 1.54

Table 3: Performance comparison of various feature extractors for the $\times
8$ upsampling task on NYUv2.

The results show that our method is fairly insensitive to the choice of
architecture, across a range of capacities (respectively, parameter counts).
This seems to indicate that the graph-based regularisation, although clearly
benefiting from high-level features, bounds the expressive power of the
feature extractor. We speculate that this is due to the fact that the graph
cannot represent long-range patterns in the output image, but only enforces
local smoothness where needed, thus limiting the amount of information that
can be usefully transferred to the predicted target. However, we do not
recommend very low capacity backbones: when using FDSR, the performance is
better than the original FDSR model, but clearly lower than with higher
capacity models. Nevertheless, independent of the backbone used, our approach
achieves the lowest MAE among all evaluated methods; except when using the raw
_Colour_ as features, i.e., low-level image contrast is not sufficient as a
regulariser and using a learned feature extractor is essential.

##  5 Discussion

The graph regularisation layer effectively acts as a bottleneck on the amount
of information that can be carried from the guide to the target – the
regularisation is not able to create arbitrary patterns in the target image.
This can be seen as a limitation, but also a desirable property, as it
increases model robustness. One drawback of our method w.r.t. to most of the
conventional deep forward architectures is the inference time, which is the
price to pay for an online optimisation that guarantees rigorous fidelity to
the source. A forward pass on a single $256^{2}$ pixel patch, for upsampling
factor $\times 8$, takes on average $111\,$ms on an NVIDIA GeForce RTX 2080
Ti. This number varies depending on the complexity of the image, and the
upsampling factor.

##  6 Conclusion

We have presented a novel formulation for guided super-resolution based on a
learnable graph regulariser. The method employs a deep feature extractor that
takes a guide and a source as input, and infers an affinity graph over
adjacent pixels in the target image. The learned graph serves as a regulariser
in the upsampling of the source, implemented as a differentiable optimisation
layer. This explicit optimisation within the architecture guarantees rigorous
fidelity of the high-resolution target to the low-resolution source. Our
proposed method combines desirable properties from both, conventional and deep
learning based methods: the optimisation layer guarantees that the fidelity
w.r.t. the source image is satisfied, even in case of domain shifts in the
test set, while the deep feature extractor enables the learned affinity graph
to encapsulate valuable information extracted from a large context. The
experimental evaluation confirms that our graph regulariser is effective for
signals that exhibit a piecewise smooth structure, such as depth maps.

## References

  * [1] Jonathan T. Barron, Andrew Adams, YiChang Shih, and Carlos Hernández.  Fast bilateral-space stereo for synthetic defocus.  In CVPR, 2015.
  * [2] Jonathan T. Barron and Ben Poole.  The fast bilateral solver.  In ECCV, 2016.
  * [3] Derek Chan, Hylke Buisman, Christian Theobalt, and Sebastian Thrun.  A noise-aware filter for real-time depth upsampling.  In Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications - M2SFA2, 2008.
  * [4] Cho, Jaehoon and Min, Dongbo and Kim, Youngjung and Sohn, Kwanghoon.  Deep monocular depth estimation leveraging a large-scale outdoor stereo dataset.  Expert Systems with Applications, 2021.
  * [5] Riccardo de Lutio, Stefano D’Aronco, Jan D. Wegner, and Konrad Schindler.  Guided super-resolution as pixel-to-pixel transformation.  In ICCV, 2019.
  * [6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR. Ieee, 2009.
  * [7] James Diebel and Sebastian Thrun.  An application of markov random fields to range sensing.  In NIPS, 2006.
  * [8] Weisheng Dong, Guangming Shi, Xin Li, Kefan Peng, Jinjian Wu, and Zhenhua Guo.  Color-guided depth recovery via joint local structural and nonlocal low-rank regularization.  IEEE Transactions on Multimedia, 2016.
  * [9] Zheng et al.  Conditional random fields as recurrent neural networks.  In ICLR, 2015.
  * [10] David Ferstl, Christian Reinbacher, Rene Ranftl, Matthias Rüther, and Horst Bischof.  Image guided depth upsampling using anisotropic total generalized variation.  In ICCV, 2013.
  * [11] Chunle Guo, Chongyi Li, Jichang Guo, Runmin Cong, Huazhu Fu, and Ping Han.  Hierarchical features driven residual learning for depth map super-resolution.  TIP, 2019.
  * [12] Bumsub Ham, Minsu Cho, and Jean Ponce.  Robust guided image filtering using nonconvex potentials.  TPAMI, 2018.
  * [13] Kaiming He, Jian Sun, and Xiaoou Tang.  Guided image filtering.  TPAMI, 2013.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [15] Lingzhi He, Hongguang Zhu, Feng Li, Huihui Bai, Runmin Cong, Chunjie Zhang, Chunyu Lin, Meiqin Liu, and Yao Zhao.  Towards fast and accurate real-world depth super-resolution: Benchmark dataset and baseline.  In CVPR, 2021.
  * [16] Heiko Hirschmüller and Daniel Scharstein.  Evaluation of cost functions for stereo matching.  In CVPR, 2007.
  * [17] Tak-Wai Hui, Chen Change Loy, and Xiaoou Tang.  Depth map super-resolution by deep multi-scale guidance.  In ECCV, 2016.
  * [18] Beomjun Kim, Jean Ponce, and Bumsub Ham.  Deformable kernel networks for joint image filtering.  IJCV, 2021.
  * [19] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.  Accurate image super-resolution using very deep convolutional networks.  In CVPR, 2016.
  * [20] Sunok Kim, Dongbo Min, Bumsub Ham, Seungryong Kim, and Kwanghoon Sohn.  Deep stereo confidence prediction for depth estimation.  In ICIP, 2017.
  * [21] Youngjung Kim, Bumsub Ham, Changjae Oh, and Kwanghoon Sohn.  Structure selective depth superresolution for rgb-d cameras.  TIP, 2016.
  * [22] Youngjung Kim, Hyungjoo Jung, Dongbo Min, and Kwanghoon Sohn.  Deep monocular depth estimation via integration of global and local predictions.  TIP, 2018.
  * [23] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [24] Johannes Kopf, Michael F. Cohen, Dani Lischinski, and Matt Uyttendaele.  Joint bilateral upsampling.  ToG, 2007.
  * [25] HyeokHyen Kwon, Yu-Wing Tai, and Stephen Lin.  Data-driven depth map refinement via multi-scale sparse representation.  In CVPR, 2015.
  * [26] Charis Lanaras, José Bioucas-Dias, Silvano Galliani, Emmanuel Baltsavias, and Konrad Schindler.  Super-resolution of Sentinel-2 images: Learning a globally applicable deep neural network.  ISPRS Journal of Photogrammetry and Remote Sensing, 2018.
  * [27] Yijun Li, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang.  Deep joint image filtering.  In ECCV, 2016.
  * [28] Yijun Li, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang.  Joint image filtering with deep convolutional networks.  TPAMI, 2019.
  * [29] Yanjie Li, Tianfan Xue, Lifeng Sun, and Jianzhuang Liu.  Joint example-based depth map super-resolution.  In ICME, 2012.
  * [30] Ming-Yu Liu, Oncel Tuzel, and Yuichi Taguchi.  Joint geodesic upsampling of depth images.  In CVPR, 2013.
  * [31] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In ECCV, 2012.
  * [32] Jorge Nocedal and Stephen J. Wright.  Numerical Optimization.  Springer, 2006.
  * [33] Ryosuke Okuta, Yuya Unno, Daisuke Nishino, Shohei Hido, and Crissman Loomis.  Cupy: A numpy-compatible library for nvidia gpu calculations.  In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Thirty-first Annual Conference on Neural Information Processing Systems (NIPS), 2017.
  * [34] Jinshan Pan, Jiangxin Dong, Jimmy S. Ren, Liang Lin, Jinhui Tang, and Ming-Hsuan Yang.  Spatially variant linear representation models for joint filtering.  In CVPR, 2019.
  * [35] Jaesik Park, Hyeongwoo Kim, Yu-Wing Tai, Michael S. Brown, and Inseo Kweon.  High quality depth map upsampling for 3D-TOF cameras.  In ICCV, 2011.
  * [36] Min-Gyu Park and Kuk-Jin Yoon.  As-planar-as-possible depth map estimation.  Computer Vision and Image Understanding, 2019.
  * [37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In NIPS, 2019.
  * [38] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  ArXiv preprint, 2021.
  * [39] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  TPAMI, 2020.
  * [40] Gernot Riegler, David Ferstl, Matthias Rüther, and Horst Bischof.  A deep primal-dual network for guided depth super-resolution.  In BMVC, 2016.
  * [41] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015.
  * [42] Mattia Rossi, Mireille El Gheche, Andreas Kuhn, and Pascal Frossard.  Joint graph-based depth refinement and normal estimation.  In CVPR, 2020.
  * [43] Aleksandr Safin, Maxim Kan, Nikita Drobyshev, Oleg Voynov, Alexey Artemov, Alexander Filippov, Denis Zorin, and Evgeny Burnaev.  Unpaired depth super-resolution in the wild, 2021.
  * [44] Daniel Scharstein, Heiko Hirschmüller, York Kitajima, Greg Krathwohl, Nera Nešić, Xi Wang, and Porter Westling.  High-resolution stereo datasets with subpixel-accurate ground truth.  In GCPR, 2014.
  * [45] Daniel Scharstein and Chris Pal.  Learning conditional random fields for stereo.  In CVPR, 2007.
  * [46] Daniel Scharstein and Richard Szeliski.  High-accuracy stereo depth maps using structured light.  In CVPR, 2001.
  * [47] Daniel Scharstein, Richard Szeliski, and Ramin Zabih.  A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.  IJCV, 2002.
  * [48] Guy Shacht, Dov Danon, Sharon Fogel, and Daniel Cohen-Or.  Single pair cross-modality super resolution.  In CVPR, 2021.
  * [49] Evan Shelhamer, Jonathan Long, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  TPAMI, 2017.
  * [50] Xibin Song, Yuchao Dai, Dingfu Zhou, Liu Liu, Wei Li, Hongdng Li, and Ruigang Yang.  Channel attention based iterative residual learning for depth map super-resolution.  In CVPR, 2020.
  * [51] Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik Learned-Miller, and Jan Kautz.  Pixel-adaptive convolutional neural networks.  In CVPR, 2019.
  * [52] Baoli Sun, Xinchen Ye, Baopu Li, Haojie Li, Zhihui Wang, and Rui Xu.  Learning scene structure guidance via cross-task knowledge transfer for single depth super-resolution.  In CVPR, 2021.
  * [53] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International Conference on Machine Learning. PMLR, 2019.
  * [54] Antonio Torralba and Alexei A. Efros.  Unbiased look at dataset bias.  In CVPR, 2011.
  * [55] Tatsumi Uezato, Danfeng Hong, Naoto Yokoya, and Wei He.  Guided deep decoder: Unsupervised image pair fusion.  In ECCV, 2020.
  * [56] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Deep image prior.  In CVPR, 2018.
  * [57] Yang Wen, Bin Sheng, Ping Li, Weiyao Lin, and David Dagan Feng.  Deep color guided coarse-to-fine convolutional network cascade for depth image super-resolution.  TIP, 2019.
  * [58] Jingyu Yang, Xinchen Ye, Kun Li, and Chunping Hou.  Depth recovery using an adaptive color-guided auto-regressive model.  In ECCV, 2012.
  * [59] Jingyu Yang, Xinchen Ye, Kun Li, Chunping Hou, and Yao Wang.  Color-guided depth recovery from RGB-D data using an adaptive autoregressive model.  TIP, 2014.
  * [60] Qingxiong Yang, Ruigang Yang, James Davis, and David Nister.  Spatial-depth super resolution for range images.  In CVPR, 2007.
  * [61] Xinchen Ye, Baoli Sun, Zhihui Wang, Jingyu Yang, Rui Xu, Haojie Li, and Baopu Li.  PMBANet: Progressive multi-branch aggregation network for scene depth super-resolution.  TIP, 2020.
  * [62] Yongqin Zhang, Feng Shi, Jian Cheng, Li Wang, Pew-Thian Yap, and Dinggang Shen.  Longitudinally guided super-resolution of neonatal brain magnetic resonance images.  IEEE Transactions on Cybernetics, 2018.

Supplementary Material
Learning Graph Regularisation for Guided Super-Resolution

##  Appendix A Hyperparameters

We report the hyperparameter settings for the evaluation of our proposed
method for RGB-guided depth map super-resolution. Hyperparameters are reported
for each dataset and method. The experiments were conducted on the three RGB-D
datasets Middlebury [45, 47, 44, 46, 16], NYUv2 [31] and DIML [21, 20, 22, 4]
with the following methods: Guided Filter (GF) [13], the Static/Dynamic filter
(SD) [12], the Pixtransform [5], the MSG-Net [17], the Deformable Kernel
Network (DKN) and its fast version (FDKN) [18], the PMBANet [61], and the Fast
Depth Super-Resolution (FDSR) [15].

We train all learned methods using the Adam optimiser with default parameters
$\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=10^{-8}$ and a batch size of 8.
For the Middlebury dataset, we train for 2,500 epochs with an initial learning
rate of $10^{-4}$ and reduce it by factor 0.9 every 100 epochs. For the NYUv2
dataset, we train for 250 epochs with the same initial learning rate that is
reduced by a factor of 0.9 every 10 epochs. For the DIML dataset, the methods
are trained for 150 epochs, again using an initial learning rate of $10^{-4}$
with a reduction by a factor of 0.9 every 6 epochs. For all learned methods we
additionally tested the initial learning rate and learning rate schedule
proposed by the respective authors (if available), and used the best
configuration. For FDSR, we therefore deviated from our default settings and
chose an initial learning rate of $5.0\cdot 10^{-4}$, with a 0.5$\times$
reduction every 80,000 iterations. For PMBANet, we found a 0.1$\times$
reduction every 100 (1000, 60) epochs for NYUv2 (Middlebury, DIML) to work
best. Note that we did not conduct any hyperparameter search for our proposed
method.

We also report the hyperparameters used for the non-learned approaches. We
used the following values for the SD filter, which we found to be the best
performing among the tested configurations: $\lambda=0.1$, $\sigma_{g}=60$ and
$\sigma_{u}=30$. For Pixtransform we used the hyperparameters suggested in the
original manuscript.

##  Appendix B Qualitative Results

In Figures A1, A2 and A3, we provide additional examples for qualitative
comparison between our method and selected methods from our quantitative
evaluation; for the Middlebury, NYUv2 and DIML datasets respectively. We
provide two additional examples for each upsampling factor and dataset
combination. These results further confirm that the predictions obtained with
our method compare favourably to existing methods. In particular, we observe
the magnitude of errors for our method to be smaller compared to the other
approaches, especially along edges. For many examples, we can additionally see
that our method facilitates smooth depth predictions in continuous areas,
whereas other methods exhibit higher noise. Visually, the differences seem
small for an upsampling factor of $\times 4$, however with larger upsampling
factors they become more apparent.

##  Appendix C Learning Graph Weights

In Figure A4 we show additional examples of the difference between the total
affinity of each pixel to its four neighbours when the graph is defined on
colour features against the graph defined on learned features. These results
further show that our model is able to encourage the optimiser to provide
smooth predictions in areas without depth discontinuities. On the other hand,
the model also shows low predicted weights in areas that should not be
smoothed, thus providing crisper predictions.

##  Appendix D Forward Pass Timing

We provide some time statistics for the forward pass of the proposed method in
Table A1. For reference: FDKN [18] takes about 10 ms for a forward pass
independently of the scaling factor. The forward pass for the feature
extractor of our proposed method also takes about 15 ms. Pixtransform [5], by
far the slowest method among the compared ones, takes more than 120 seconds
for a $256^{2}$ patch. Our Python implementation of the SD filter [12] takes
few seconds (this also depends on the upsampling factor), although it is not
directly comparable to our method as the SD filter implementation does not
take advantage of GPU acceleration. The upsampling factor plays a major role
for the runtime of our method, because larger upsampling factors lead to
downsampling operators ${\bm{D}}$ that are less sparse, increasing the time
required to solve the linear system. Note that although our method was
implemented to leverage GPU parallelisation, there is still potential for
further optimisations that could improve runtime performance.

| $\times 4$ | $\times 8$ | $\times 16$
---|---|---|---
forward time | 79 (21) | 111 (35) | 305 (92)
Table A1: Forward pass times of our proposed method. Numbers represent the
mean time and (standard deviation) measured in milliseconds, computed over the
NYUv2 test set on single patches of $256^{2}$ pixels, on an NVIDIA GeForce RTX
2080 Ti.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

Figure A1: Additional qualitative comparison of upsampled depth maps for the
Middlebury dataset [45, 47, 44, 46, 16]. From top to bottom each group of two
rows shows the error of upsampled images, defined as the difference between
the prediction and the ground truth, for upsampling factors $\times 4$,
$\times 8$ and $\times 16$ respectively. From left to right, the first group
of columns are (a) Guide, (b) Source and (c) Ground Truth; the second group
includes selected methods from our quantitative evaluation, (d) SDF [12], (e)
Pixtransform [5], (f) MSGNet [17], (g) FDKN [18], (h) PMBANet [61] and (i)
FDSR [15]; the last two columns represent (j) the error for the prediction of
our model and (k) the prediction itself.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

Figure A2: Additional qualitative comparison of upsampled depth maps for the
NYUv2 dataset [31]. From top to bottom each group of two rows shows the error
of upsampled images, defined as the difference between the prediction and the
ground truth, for upsampling factors $\times 4$, $\times 8$ and $\times 16$
respectively. From left to right, the first group of columns are (a) Guide,
(b) Source and (c) Ground Truth; the second group includes selected methods
from our quantitative evaluation, (d) SDF [12], (e) Pixtransform [5], (f)
MSGNet [17], (g) FDKN [18], (h) PMBANet [61] and (i) FDSR [15]; the last two
columns represent (j) the error for the prediction of our model and (k) the
prediction itself.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

Figure A3: Additional qualitative comparison of upsampled depth maps for the
DIML dataset [21, 20, 22, 4]. From top to bottom each group of two rows shows
the error of upsampled images, defined as the difference between the
prediction and the ground truth, for upsampling factors $\times 4$, $\times 8$
and $\times 16$ respectively. From left to right, the first group of columns
are (a) Guide, (b) Source and (c) Ground Truth; the second group includes
selected methods from our quantitative evaluation, (d) SDF [12], (e)
Pixtransform [5], (f) MSGNet [17], (g) FDKN [18], (h) PMBANet [61] and (i)
FDSR [15]; the last two columns represent (j) the error for the prediction of
our model and (k) the prediction itself.

(a)

(b)

Figure A4: Additional examples of the importance of learned edge potentials.
We visualise the total affinity of each pixel to its four neighbours when
derived from raw colour (top) or from deep features (bottom). Examples are
from the Middlebury test set.


# Nuclear and detector sensitivities
for neutrinoless double beta-decay experiments

Hiroyasu Ejiri  Research Center for Nuclear Physics, Osaka University, Osaka
567-0047, Japan
E-mail<EMAIL_ADDRESS>
###### Abstract

Neutrinoless double beta-decay (DBD) is of current interest in high-
sensitivity frontiers of particle physics. The decay is very sensitive to
Majorana neutrino masses, neutrino CP phases, right-handed weak interactions
and others, which are beyond the standard electro-weak model. DBDs are
actually ultra-rare events, and thus DBD experiments with ultra-high
sensitivity are required. Critical discussions are presented on nuclear and
detector sensitivities for high-sensitivity DBD experiments to study the
neutrino masses in the normal and inverted mass-hierarchies.

Keyword: Neutrinoless double beta decay, nuclear sensitivity, detector
sensitivity,

neutrino mass, Majorana neutrino, nuclear matrix element.

##  1 Introduction

Neutrinoless double beta-decay (DBD) is of current interest in sensitivity
frontiers of particle physics. The decay violates the lepton number
conservation law, and thus DBD is beyond the standard electro-weak model. DBD
is very sensitive to the Majorana nature (neutrino=anti-neutrino) of neutrino,
the absolute neutrino-mass scale and the neutrino-mass hierarchy, the neutrino
CP-phases, the possible right-handed weak interactions and others, which are
beyond the standard model. Accordingly, it has been used to study these
fundamental questions on the neutrino and the weak interaction for decades.
Recent experimental and theoretical DBD works are discussed in the review
articles and references therein [1, 2, 3, 4, 5].

In fact, neutrinoless DBDs are due to various modes such as the light
neutrino-mass mode, the right-handed weak current mode, the SUSY particle
exchange mode, and the others, which are all beyond the standard model [1, 4]
. In the present work, we discuss mainly the neutrinoless DBD due to the light
neutrino-mass mode, which is of current interest.

DBD, however, is a very low-energy and ultra-rare decay. The energy is of the
order of $E_{\beta\beta}\approx$ 10-3 GeV and the decay rate is in the range
of $T^{0\nu}_{r}\approx$ 10-27 \- 10-30 per year (y), depending on the
neutrino-mass and the nuclear matrix element(NME). The neutrino-mass to be
studied is around $m$=3 meV and 30 meV in cases of the normal mass-hierarchy
(NH) and the inverted mass-hierarchy, respectively. Then, low-background high-
sensitivity DBD detectors are required to search for such low-energy and
ultra-rare DBD signals [1, 4].

Many experimental groups are working hard for high sensitivity DBD experiments
to search for the small Majorana neutrino-mass and the others beyond the
standard model. Current lower limits on the DBD half-lives are around 1026 y
for both 76Ge and 136Xe . The neutrino mass limits are, respectively, 600
meV/$M(76)$ and 230 meV/$M(136)$ with $M(76)$ and $M(136)$ being the nuclear
matrix elements (NMEs) for 76Ge and 136Xe, respectively. So they are in the
range of 300-100 meV if NMEs are around 2. See section 4 for discussions on
DBD experiments. Then one needs to improve the sensitivity by factors 5 and 50
to access the IH and NH mass regions, respectively. In fact, the DBD
sensitivity to the neutrino mass depends much on NME.

Figure 1: Schematic views of neutrinoless double $\beta$-decay with Majorana
neutrino exchange (left hand panel) and nuclear collider (right handed panel).

DBD with a light neutrino-mass mode is considered as a neutrino-exchange
process, where a light Majorana neutrino is exchanged between two neutrons in
the DBD nucleus. Here the nucleus acts as a high-luminosity micro-collider of
two neutrons, as shown in Fig.1. The neutrino-exchange cross-section for the
IH neutrino is as small as $\sigma\approx$ 10-83 cm2 because of the second-
order weak process and the small neutrino mass [4, 6, 7]. The luminosity of
one nucleus is as high as $L\approx$1048 cm-2s-1 because of the very small ( a
few fm) distance between the two neutrons in the nucleus. The summed
luminosity for 1 ton DBD nuclei is $L\approx$1076cm-2s-1. Thus one gets 2-3
DBD signals per year for the IH neutrino exchange by using a large DBD
detector with 1 ton DBD isotopes.

The DBD neutrino-mass sensitivity to search for the small neutrino mass is
defined as the minimum neutrino mass $m_{m}$ to be measured by the DBD
experiment. It is given by a product of a nuclear sensitivity and a detector
sensitivity. The nuclear sensitivity depends on nuclear parameters such as the
nuclear phase space and the NME, while the detector sensitivity depends on the
detector parameters such as the total number of the DBD isotopes, the
enrichment, the exposure time, the detection efficiency and the backgrounds.
The minimum neutrino mass to be measured and the nuclear and detector
sensitivities are discussed in the review articles [1, 4, 7].

The present report aims at critical discussions on the nuclear and detector
sensitivities to search for the ultra-rare DBD events associated with very
small IH and NH masses. Critical discussions on DBD NMEs is presented
elsewhere [8].

##  2 Neutrino mass sensitivity for DBD experiment

The neutrinoless DBD transition rate is expressed in terms of the axial-vector
weak coupling $g_{A}$=1.27 in units of the vector coupling of $g_{V}$ as [1,
4]

$T_{r}^{0\nu}=g_{A}^{4}G^{0\nu}|M^{0\nu}f(\nu)|^{2},$ (1)

where $G^{0\nu}$ is the phase space volume, $M^{0\nu}$ is the NME and $f(\nu)$
stands for the effective neutrino mass and the other neutrino and weak
interaction parameters beyond the standard model. These three quantities are
associated with the kinematic factor, the nuclear physics factor and the
particle physics factor, respectively.

In case of the light neutrino-mass process, $f(\nu)$ is given by the effective
neutrino mass of $m=\sum|U_{i}|^{2}m_{i}e^{\alpha_{i}}$ with $m_{i}$ $U_{i}$
and $\alpha_{i}$ being the ith neutrino mass, the mixing coefficient and the
phase, respectively. In case of the right-handed weak-boson process $f(\nu)$
includes the term $k(M_{W}^{L}/M_{W}^{R})^{2}$ with $M_{W}^{L}$ and
$M_{W}^{R}$ being the let-handed and right-handed weak-boson masses,
respectively [1, 4].

The DBD NME for the light neutrino-mass mode is expressed in terms of the
Gamow-Teller (GT), the Fermi (F) and the tensor (T) NMEs as [1, 4, 7]

$M^{0\nu}=(\frac{g_{A}^{eff}}{g_{A}})^{2}[M(GT)+(\frac{g_{V}^{eff}}{g_{A}^{eff}})^{2}M(F)+M(T)],$
(2)

where $g_{A}^{eff}$ and $g_{V}^{eff}$ are, respectively, the effective axial-
vector and vector couplings, and $g_{A}$ and $g_{V}$ are the axial-vector and
vector couplings for a free nucleon. $M(GT)$, $M(F)$ and $M(T)$ are,
respectively, the model NMEs for Gamow-Teller, Fermi, and tensor transition
operators.

The GT and T NMEs are very sensitive to nucleonic and non-nucleonic
correlations and nuclear medium effects, and thus the ratios
$g_{A}^{eff}/g_{A}$ and $g_{V}^{eff}/g_{V}$ stand for the re-normalization
coefficients due to such correlations and nuclear medium effects that are not
explicitly included in the nuclear model. Thus they depends on the models and
the transition operators. Since the axial-vector NMEs are strongly modified by
the strong isobar (quark spin-isospin excitation) and other non-nucleonic
correlations, we usually consider explicitly the re-normalization coefficient
$g_{A}^{eff}$, but the re-normalization coefficient $g_{V}^{eff}$ for the
vector coupling should be well considered unless the models include all
correlations and the medium effect [6, 7, 8]. The DBD NMEs are discussed in
review articles on neutrino nuclear responses [6, 7, 8]. Theoretical works on
DBD NMEs are discussed in reviews [9, 10, 11].

The DBD rate $T_{r}^{0\nu}$ per year (y) and per ton (t) of the DBD isotope-
mass is expressed by using a neutrino mass-unit $m_{0}$ as [1, 4, 7, 8]

$T_{r}^{0\nu}=(\frac{m}{m_{0}})^{2},~{}~{}~{}m_{0}=\frac{m^{\prime}_{0}}{M^{0\nu}}$
(3) $m^{\prime}_{0}=7.8{\rm meV}\frac{A^{1/2}}{g_{A}^{2}(G^{0\nu})^{1/2}},$
(4)

where $A$ is the mass-number of the DBD nucleus and $G^{0\nu}$ is the phase
space factor in unit of 10-14/y, and $m_{0}$ is the neutrino mass that gives
the DBD rate $T^{0\nu}_{r}$=1 per t y. We define $m_{0}$ as the nuclear
sensitivity of the DBD nucleus, being inversely proportional to the
($G^{0\nu})^{1/2}M^{0\nu}$. Here, high sensitivity to the neutrino mass means
high capability of measuring a small neutrino-mass. $m^{\prime}_{0}$ is the
neutrino mass $m_{0}$ in case of $M^{0\nu}$=1.

The values $m^{\prime}_{0}$ for typical DBD nuclei of 82Se, 100Mo, 116Cd and
130Te, which are of current interest, are all around 40 meV, which are close
to the upper bound of the IH neutrino mass. The values for 76Ge and 150Nd are,
respectively, larger and smaller by a factor 2 than the value of 40 meV
because of the smaller and larger phase space volumes.

In order to study the neutrino mass, the number of the DBD signals is required
to be larger than the fluctuation $\delta$ of the number of the background
signals. Then it is expressed as [1, 7, 8]

$T_{r}^{0\nu}\eta\epsilon
NT\geq\delta,~{}~{}~{}~{}~{}\delta=\delta_{0}\times(BNT)^{1/2},$ (5)

where $N$ is the total DBD isotope-mass in units of t, $T$ is the exposure in
units of y, $\delta_{0}$ is around 2 and $B$ is the number of the backgrounds
per $N$=1 ton per $T$=1 year, $\eta$ is the enrichment coefficient of the DBD
isotope and $\epsilon$ is the DBD signal detection-efficiency. Note that the
actual DBD isotope-mass is $\eta N$ ton, the signal yield is
$T_{r}^{0\nu}\eta\epsilon NT$, and the BG yield is $BNT$.

Then the minimum effective neutrino-mass to be measured with 90 $\%$
confidence level is expressed in terms of the nuclear sensitivity $m_{0}$ and
the detector sensitivity $d$ as

$m_{m}=m_{0}\times{d},~{}~{}~{}d=d_{0}\times\eta^{-1/2}\epsilon^{-1/2}(NT/B)^{-1/4},$
(6)

where $d_{0}$ is around 1.4. The neutrino mass to be measured is
$m_{m}$=2$m^{\prime}_{0}/M^{0\nu}$ in a typical case of the DBD detector
exposure of $NT$= 1 t y, the BG rate of $B$=1/t y, $\epsilon$=0.55, and
$\eta$=0.9.

On the basis of the simple expression of the mass sensitivity $m_{m}$ as given
in eq. (6), key points for high sensitivity DBD experiments are as follows.

1\. The neutrino mass to be measured is proportional to 1/$M^{0\nu}$ and
($B/NT)^{1/4}$. Therefore the DBD nucleus with larger $M^{0\nu}$ by a factor 2
is equivalent to the DBD isotope-mass ($N$) larger by a factor 16. Then the
detector volume gets more than an order of magnitude larger if $M^{0\nu}$ gets
smaller by 40 $\%$.

2\. The minimum mass to be measured depends on the ratio of $(B/N)^{1/4}$.
Then the neutrino mass sensitivity is improved by a factor 2 by increasing the
DBD isotope-mass ($N$) by a factor 16, or by decreasing the BG rate by a
factor 16, or by increasing the DBD isotope-mass by a factor 4 and decreasing
the BG rate by a factor 4.

3\. The effective neutrino mass depends on the neutrino mixing-phases, and is
in the regions of $m\approx$17-45 meV and $m\approx$1.5-4 meV, respectively,
in cases of IH and NH. Assuming a typical NME of $M^{0\nu}$=2, one needs the
DBD detectors with the mass sensitivities around the nuclear sensitivity of
$m_{0}\approx 20$ meV and one tenth of that, respectively, in cases of IH and
NH. Then one needs DBD detectors with $N\approx$ 1 t and $B\approx$ 1 /t y and
$N\approx$ 100 t and $B\approx$ 0.01 /t y , respectively, to measure the IH
and NH neutrino-masses.

4\. DBD isotope enrichment is very effective. Enrichment by a factor 3 is
equivalent to the increase of DBD isotope-mass by a factor 10. Accordingly it
is effective to use ton-scale $80-90\%$ enriched DBD isotopes.

##  3 DBD nuclear and detector sensitivities

In this section we discuss DBD nuclear and detector sensitivities as given in
eq. (6) for DBD experiments to access the IH and NH neutrino masses.

###  3.1 Nuclear sensitivity

The nuclear sensitivity $m_{0}$ is a ratio of the unit mass $m^{\prime}_{0}$
and the NME $M^{0\nu}$. In fact, $M^{0\nu}$ is so sensitive to the details of
the nuclear structures that accurate evaluation for $M^{0\nu}$ is very hard.
So we discuss in the present work mainly $m^{\prime}_{0}$, and assume that
$M^{0\nu}$ is in the region of 1-3. See the recent review on DBD NMEs [7] and
references therein for detailed discussions on $M^{0\nu}$.

Figure 2:  Neutrino mass sensitivities $m_{m}$ as a function of the exposure
$NT$. $M$ and $B$ are the NME and the BG rate per t y. $d_{0}$=1.4,
$\epsilon$=0.55, $\eta$=0.9. See eq. (6).

.

The unit mass $m^{\prime}_{0}$ is proportional to $A^{1/2}(G^{0\nu})^{-1/2}$.
The phase space factor $G^{0\nu}$ increases with increase of the DBD $Q$
value. Thus double $\beta^{-}$ nuclei with the large $Q$ around 3 MeV are
used. The large $Q$ is also effective to reduce much BGs from 208Tl and 214Bi,
which are two major BGs. The mass-number $A$ dependence reflects the number of
the DBD nuclei per ton of the total DBD isotope-mass. The mass sensitivities
for a typical DBD nucleus with $m^{\prime}_{0}$=40 meV are plotted as a
function of the exposure $NT$ in Fig. 2. In order to access the IH mass of 20
meV, one needs a DBD exposure $NT$ of around 16 t y , i.e.$N\approx$ 5 ton DBD
isotopes and $T\approx$3 y measurement in case of a typical NME of
$M^{0\nu}$=2 and BG rate of $B$=1/t y. On the other hand, one needs $NT$ of
around 3 t y, i.e. $N\approx$1 ton DBD isotopes and $T\approx$3 y measurement
in case of $M^{0\nu}$=3, and $NT$ of around 250 t y, i.e.$N\approx$60 ton DBD
isotopes and $T\approx$4 y measurement, in case of $M^{0\nu}$=1. So, it is
very crucial to know the NME even for designing the DBD detector.

The mass sensitivities for the enrichment factors of $\eta$=10, 50, 100 $\%$
are plotted as a function of the total DBD isotope-mass $N$ in Fig. 3

Figure 3: Neutrino mass sensitivities $m_{m}$ for $\eta$=1.0, 0.5, and 0.1 as
a function of the total isotope-mass $N$. The exposure time is $T$=5 y, and
the nuclear sensitivity of $m_{0}$=20 meV, $\epsilon$=0.55, $d_{0}$=1.4, and
$B$=1 /t y.

Figure 4: Neutrino mass sensitivities $m_{m}$ for $\eta$=0.9, $\epsilon$=0.55,
$d_{0}$=1.4 as a function of the ratio of the total isotope-mass $N$ to the BG
rate $B$. The exposure time is $T$=5 y, and the unit mass $m^{\prime}_{0}$=40
meV

.

In order to access the IH mass of 20 meV, one needs $N$=3.4 t and 4.1 t of the
total DBD isotopes, respectively, in case of $\eta$=100$\%$ and 90$\%$, while
it is $N$=14 t in case of $\eta$=50$\%$ ($\eta N$ = 7 t), and $N$=340 t in
case of $\eta$=10$\%$ ($\eta N$ = 34 t). Therefore, enrichment around
$\eta\geq 80\%$ is very effective . The 1 ton DBD total isotope-mass with
$\eta\approx 90\%$ is equivalent to the $N\approx$10 ton of the total DBD
isotope-mass with $\eta\approx$ 30$\%$.

###  3.2 DBD detector sensitivity and backgrounds

DBD experiment requires an ultra-low BG detector since the DBD signal is very
low in energy and the event rate is very rare. The neutrino mass-sensitivity
for a typical exposure time of $T$=5 y and $\eta$ = 0.9, $\epsilon$=0.55,
$d_{0}$=1.4 is rewritten in therms of the ratio of the DBD isotope-mass $N$
and the BG rate $B$ as

$m_{m}\approx 1.35\frac{m^{\prime}_{0}}{M^{0\nu}(N/B)^{1/4}}.$ (7)

Figure 5: Neutrino mass sensitivities $m_{m}$ as a function of the DBD
isotope-mass $N$ in cases of $B$=1, 0.01, and 0. The exposure time is $T$=5 y.
The unit mass $m^{\prime}_{0}$=40 meV and $M^{0\nu}$=2, $\epsilon$=0.55 and
$\eta$=0.9.

The mass sensitivities for the $m^{\prime}_{0}$= 40 meV with typical NMEs of
$M^{0\nu}$=2 and $M^{0\nu}$=3 are simply given as $m_{m}\approx$27
meV/$(N/B)^{1/4}$ and $m_{m}\approx$18 meV/$(N/B)^{1/4}$. They are shown as a
function of $N/B$ in Fig. 4. So, in case of $M^{0\nu}\approx$2, DBD detectors
to search for the IH mass of 20-30 meV and the NH mass of 2-3 meV are required
to be large isotope-mass and low-BG detectors with $N/B\approx$1 and
$N/B\approx 10^{4}$, respectively.

Let us see how the mass sensitivity changes if one gets a BG free ($B$=0)
detector. In this case, the mass sensitivity is derived by requiring the
signal yield $\geq$2.3 as [4]

$m_{m}=1.5\times m_{0}\epsilon^{-1/2}\eta^{-1/2}(NT)^{-1/2}.$ (8)

The mass sensitivities for three BG cases of $B$=1, 0.01, and 0 are compared
in Fig. 5. Then the DBD isotope-mass $N$ required to access the IH mass of 20
meV is 1 t, while the DBD isotope-mass to access the NH mass of 2 meV is 100
t. They are a factor 3 smaller than those for the detectors with $B$=1 and
0.01. Thus one needs 1ton-scale and 100 ton-scale DBD isotope-masses to access
the IH and NH neutrino masses even by using the BG-free detectors.

The DBD signal is expected to appear as a sharp peak at the energy of $E=Q$ in
the energy spectrum. The peak width is around the FWHM (full width half
maximum) of the detector. Thus one usually sets the energy window of $\Delta
E$= 2 FWHM as the region of interest, ROI.

BGs in ROI are mostly due to $\beta$-rays and Compton-scattered $\gamma$ rays,
which are continuum spectra. Backgrounds due to the solar-neutrino
interaction, which get serious in the NH mass search, are also continuum [12,
13]. Then the BG rate at the energy window of ROI is proportional to FWHM.
Since the mass sensitivity is proportional to $(B/N)^{1/4}$, improvement of
the energy resolution by an order of magnitude is equivalent to increase of
the DBD isotope-mass by the same order of magnitude.

The mass sensitivity depends on the detection efficiency as $m_{m}$ =
$k\epsilon^{-1/2}$. The efficiency includes all efficiencies associated with
the energy and PSA windows/cuts and analyses to select the DBD signals and to
reject BG ones. It is around $\epsilon\approx$0.5-0.6. Severe cuts to reject
BGs reduce both the efficiency $\epsilon$ and the BG rate $B$. The decrease of
$\epsilon$ by 20$\%$ is compensated if $B$ gets smaller by 40 $\%$.

##  4 Remarks and discussions on DBD detectors

There are many high-sensitivity DBD experiments and future plans to access the
IH mass around 20 meV and the NH mass around 2 meV. The detailed reports on
the present status on individual experiments and plans are given in their
reports and detailed discussions on them are given in the reviews [1, 4, 5, 7]
and references therein, and also in the recent 2020 conference reports. The
present and future DBD experiments are discussed there, see for example ref.
[14]. So, we give in this section a few comments on some (not all) possible
DBD experiments from view points of the DBD mass sensitivity, i.e. the nuclear
and detector sensitivities as given in eq.(6).

The DBD isotopes of 82Se, 100Mo and 136Xe are very useful isotopes because of
the small $m^{\prime}_{0}\approx$ 40 meV due to the large phase space
$G^{0\nu}$ and the large $Q$ value and multi-ton scale enriched isotopes with
$N\approx$10 and $\eta\approx$90$\%$ are available by centrifugal separation.

Cryogenic 82Se- and 100Mo- bolometers with both thermal and scintillation
signals are promising detectors with high energy resolution and low BG rate.
See refs. [15, 16, 17, 18]. 116Cd with $m^{\prime}_{0}\approx$40 mev is also
interesting. See refs. [19, 20].

136Xe can be easily enriched to get a 10 t scale isotope-mass and thus is
possible to access the IH and NH masses. Various kinds of 136Xe detectors are
under progress. See refs. [21, 22, 23, 24, 25, 26, 27].

76Ge with $Q$=2.039 MeV has the unit mass of $m^{\prime}_{0}$=80 meV, a factor
2 larger than the others given above. Ton scale enriched DBD isotopes are
possible by centrifugal separation, and the energy resolution of 10-3 in FWHM
is very good. Accordingly the Ge BG rate is almost an order of magnitude
smaller than others to get the neutrino mass sensitivity comparable with other
detectors. See refs. [28, 29, 30, 31, 32].

The natural abundance of 130Te is 34$\%$, and thus multi-ton scale natural Te
isotopes may be used although the mass sensitivity $m_{m}$ gets larger by a
factor 1.6 than the mass sensitivity with enriched 130Te-isotope detectors.
See ref. [18, 33]

150Nd has a large phase space factor because of the large $Q$ value of 3.4 MeV
and the large $Z$ number of 60. Thus the $m^{\prime}_{0}$=18 meV is a half of
the others [7, 8]. The natural abundance, however, is only 5.6 $\%$ and the
enrichment is not easy. See ref.[34].

Tracking detectors have been used to study DBDs on 100Mo, 116Cd, 82Se, and
many other DBD nuclei as discussed in review articles and references therein
[1, 2, 4, 5]. They measure individual two $\beta$-rays to identify the DBD
signals and to reject single-$\beta$ background signals. The measured energy
and angular correlations are used to differentiate the DBD processes due to
the left-handed and right-handed weak currents. See refs. [35, 36, 37].

Finally it should be remarked that the NME is one of key ingredients for the
DBD mass sensitivity, and thus selection of DBD isotopes with a large NME is
very crucial for getting high-sensitivity DBD experiments [6, 7]. It is
however very hard to evaluate accurately the DBD NMEs. So various experiments
are under progress to provide nuclear parameters, the effective axial-vector
weak couplings and nuclear structures to help theoretical evaluations of the
neutrinoless DBD NMEs. These are discussed in recent articles [8, 38].

Acknowledgement

The author would like to express many thanks to Prof. Haris Kosmas for
valuable discussions and encouragements and best wishes for his future
activities on the occasion of his 70th birthday.

## References

  * [1] H. Ejiri, Double beta decays and neutrino masses, J. Phys. Soc. Jpn. 74, 2101 (2005).
  * [2] F. Avignone, S. Elliott, J. Engel, Double beta dcay, Majorana neutrino, and neutrino mass, Rev. Mod. Phys. 80, 481 (2008).
  * [3] H. Ejiri, Double $\beta$-decays and neutrino nuclear responses, Prog. Part. Nucl. Phys. 54, 249 (2010)
  * [4] J. Vergados, H. Ejiri, F. Šimkovic, Theory of neutrinoless double-$\beta$ decay, Rep. Prog. Phys.  75, 106301 (2012).
  * [5] J. Vergados, H. Ejiri, F. Šimkovic, Neutrinoless double $\beta$-decay and neutrino mass, Internat. J. Modern Phys. E 25, 1630007 (2016).
  * [6] H. Ejiri, Nuclear spin isospin responses for low-energy neutrinos, Phys. Rep. 338, 265 (2000).
  * [7] H. Ejiri, J. Suhonen and K. Zuber, Neutrino nuclear responses for astro-neutrinos, single $\beta$-decays, and double $\beta$-decays Phys. Rep. 797, 1 (2019).
  * [8] H. Ejiri, Neutrino-mass sensitivity and nuclear matrix element for neutrinoless double beta decay, Universe 6 225 (2020). doi:103390/universe 6120225.
  * [9] J. Suhonen, O. Civitarese, Weak interaction and nuclear structure aspect of nuclear double beta decay Phys. Rep. 300, 123 (1998).
  * [10] J. Suhonen, O. Civitarese, Double-beta decay nuclear matrix elements in the QRPA framework, J. Phys. G: Nucl. Part. Phys. 39, 035105 (2012).
  * [11] J. Engel and J. Menéndez, Status and future of nuclear matrix elements for neutrinoless double $\beta$-decay: a review, Rep. Prog. Phys. 60, 046301 (2017).
  * [12] H. Ejiri and S. R. Elliott, Charged current neutrino cross section for solar neutrinos, and background to the $\beta\beta(0\nu)$ experiments, Phys. Rev. C.  89, 055501 (2014).
  * [13] H. Ejiri and S. R. Elliott, Solar neutrino interactions with the double-$\beta$ nuclei 82Se, 100Mo, and 150Nd, Phys. Rev. C.  95, 055501 (2017).
  * [14] J. Detwiler, Future neutrino-less double-beta decay experiments, Proc. Neutrino 2020 (2020).
  * [15] D. R. Artusa et al., The LUCIFER/CUPID-0 demonstrator: searching for the neutrinoless double-beta decay with Zn82Se scintillating bolometers, J. Phys. Conf. Ser. 888, 012077 (2017).
  * [16] H. Park et al., The AMoRE: Search for neutrinoless double beta decay in 100Mo, Nucl. Part. Phys. Proc. 273, 2630 (2016).
  * [17] D. Poda, A. Giuliani, Low background techniques in bolometers for double-beta-decay search, Int. J. Mod. Phys. A 32, 1743012 (2017).
  * [18] T.O. Donnell et al., (CUORE collaboration), CUORE results and CUPID Project, Proc. Neutrino 2020, (2020).
  * [19] K. Zuber, COBRA: Double $\beta$-decay searches using CdTe detectors, Phys. Lett. B 519, 1 (2001).
  * [20] F. A. Danevich et al. (Aurora experiment), Search for double beta decay of 116Cd with enriched 116CdWO4 crystal scintillators, J. Phys. Conf. Ser. 718, 062009 (2016) .
  * [21] A. Gando et al. (KamLAND collaboration), Search for Majorana neutrinos near the inverted mass hierarchy region with KamLAND-Zen, Phys. Rev. Lett. 117 (2016) 082503, Addendum: Phys. Rev. Lett. 117, 109903 (2016).
  * [22] J. B. Albert et al. (EXO-200 collaboration), Search for neutrinoless double-$\beta$-decay with the upgraded EXO-200 detector, Phys. Rev. Lett. 120, 072701 (2018).
  * [23] L. Yang et al., Status and prospects for the EXO-200 and nEXO experiments, J. Phys. Conf. Ser. 888, 012032 (2017).
  * [24] G. Anton et al., (EXO-200 Collaboration), Search for neutrinoless double-$\beta$ decay with the complete EXO-200 data set, Phys. Rev. Lett. 123, 161802 (2019).
  * [25] C. Grant, Results from KamLAND-ZEN and SON+, Proc. Neutrino 2020 (2020).
  * [26] J.J. Gomez Cadenas, Xe-136 experiments, present and future, Proc. Neutrino 2020 (2020).
  * [27] N. Lopez March et al., The NEXT high pressure xenon gas TPC for neutrinoless double $\beta$-decay searches, JINST, 13, C01048 (2018).
  * [28] M. Agostini et al., Improved limit on neutrinoless double $\beta$-decay of 76Ge from GERDA Phase II, Phys. Rev. Lett., 120, 132503 ( 2018).
  * [29] C.E. Aalseth et al. (MAJORANA-collaboration), Search for neutrinoless double-$\beta$ decay in 76Ge with the Majorana demonstrator,Phys. Rev. Lett. 120, 132502 (2018).
  * [30] S.I. Alvis, et al., (MAJORANA-Collaboration), Search for neutrino-less double-$\beta$ decay in 76Ge with 26 kgy of exposure from the Majorana demonstrator, Phys. Rev. C 100, 025501 (2019).
  * [31] N. Abgrall et al. (LEGEND-collaboration), The large enriched germanium experiment for neutrinoless double beta decay (LEGEND), AIP Conf. Proc. 1894, 020027 (2017).
  * [32] Y. Kermaidic, GERDA, Majorana and LEGEND towards a background-free ton-scale Ge76 experiment, Proc. Neutrino 2020 (2020).
  * [33] D.Q. Adams et al. (CUORE Collaboration), Improved limit on neutrinoless double-beta decay in 130Te with CUORE, Phys. Rev. lett. 124, 122501 (2019).
  * [34] S. Andringa et al. (SNO+-collaboration), Current status and future prospects of the SNO+ experiment, Adv. High Energy Phys. 2016, 6194250 (2016).
  * [35] H. Ejiri, J. Engel, R. Hazama, P. Kurastev, N. Kudomi, and R.G.H. Robertson, Spectroscopy of double-beta decays from 100Mo for neutrinos, Phys. Rev. Lett. 85, 2917 (2000).
  * [36] H. Ejiri et al., MOON (Molybdenum Observatory Of Neutrinos) for neutrinoless double $\beta$-decays, Eur. Phys. J. 162, 239 (2008).
  * [37] D. Waters et al. (NEMO-3 and Super NEMO collaboration), Latest results from NEMO-3 and status of the superNEMO experiment, J. Phys. Conf. Ser. 888, 012033 (2017).
  * [38] H. Ejiri, Perspectives on neutrino nuclear-response studies for double beta decays and astro neutrinos, Frontiers in Physics, to be submitted (2021).