# Quality Assessment of Image Super-Resolution: Balancing Deterministic and
Statistical Fidelity

Wei Zhou  University of Waterloo<EMAIL_ADDRESS>and  Zhou Wang
University of Waterloo<EMAIL_ADDRESS>
###### Abstract.

There has been a growing interest in developing image super-resolution (SR)
algorithms that convert low-resolution (LR) to higher resolution images, but
automatically evaluating the visual quality of super-resolved images remains a
challenging problem. Here we look at the problem of SR image quality
assessment (SR IQA) in a two-dimensional (2D) space of deterministic fidelity
(DF) versus statistical fidelity (SF). This allows us to better understand the
advantages and disadvantages of existing SR algorithms, which produce images
at different clusters in the 2D space of (DF, SF). Specifically, we observe an
interesting trend from more traditional SR algorithms that are typically
inclined to optimize for DF while losing SF, to more recent generative
adversarial network (GAN) based approaches that by contrast exhibit strong
advantages in achieving high SF but sometimes appear weak at maintaining DF.
Furthermore, we propose an uncertainty weighting scheme based on content-
dependent sharpness and texture assessment that merges the two fidelity
measures into an overall quality prediction named the Super Resolution Image
Fidelity (SRIF) index, which demonstrates superior performance against state-
of-the-art IQA models when tested on subject-rated datasets.

Image super-resolution; perceptual image quality assessment; deterministic
fidelity; statistical fidelity; uncertainty weighting

††ccs: Computing methodologies Image processing

##  1\. Introduction

Figure 1. SR images and quality assessment in 2D space of (DF, SF). (a)
Original HR image; (b) SR image reconstructed by VDSR (Kim et al., 2016) at
scaling factor 4; (c) SR image reconstructed by SRGAN (Ledig et al., 2017) at
scaling factor 4. Bottom: SR images evaluated in (DF, SF) space, where the
green arrows link images of the same content. The results reveal drastically
different characteristics of SR algorithms.

Image super-resolution (SR) targets at “restoring” the spatial resolution of a
given low-resolution (LR) image to a higher quality super-resolved image that
potentially recovers the original sharpness and texture details. Image SR is
an important and challenging problem that has attracted a great deal of
attention in recent years. It has a wide range of real-world applications
(Park et al., 2003), including visual communications, video surveillance,
medical imaging, and high-definition television, among many others. Image SR
is an ill-posed inverse problem because there exist many legitimate SR image
solutions for any given single input LR image. Many SR methods have been
proposed in the literature, ranging from simple local interpolation algorithms
to sophisticated deep learning approaches. With numerous SR methods available,
it becomes critically important to automatically assess the perceptual quality
of reconstructed SR images for selection and optimization purposes. Although
there have been a large number of studies in the area of image quality
assessment (IQA), few have focused on SR IQA. Subjective quality evaluation is
generally regarded as a reliable method (Reibman et al., 2006; Yang et al.,
2014; Yeganeh et al., 2015; Wang et al., 2017; Ma et al., 2017c; Shi et al.,
2019; Zhou et al., 2019), but is expensive, time-consuming, and labor-
intensive. It is also worth noting that certain structural information may be
lost or altered during the SR process, but human observers may not be able to
notice the existence of such structural loss or change. Therefore, objective
image quality or fidelity models that work effectively for SR images are
highly desirable.

Nevertheless, existing image quality, fidelity and sharpness models only
achieve limited success in SR IQA (as will be shown in Section 4). The
challenge is not only in the difficulty in obtaining high correlation with the
overall subjective opinions, but perhaps more importantly, on offering
meaningful interpretations on the visual artifacts and the intrinsic
characteristics of different SR algorithms. An example is shown in Figure 1,
where the SR images created by a traditional CNN model (VDSR (Kim et al.,
2016)) and a recent GAN-based model (SRGAN (Ledig et al., 2017)) are compared
with the original reference image. The VDSR generated SR image safely
preserves the general structure or deterministic fidelity (DF) according to
most existing image quality/fidelity measures such as Peak Signal-to-Noise
Ratio (PSNR) and the structural similarity index (SSIM) (Wang et al., 2004),
but loses the fine textures in the original image or the statistical fidelity
(SF). By contrast, the SRGAN generated image better preserves the texture
complexity/granularity or SF, but meanwhile produces novel structures or
artifacts inexistent in the original image, and is thus poor in maintaining
the DF. These observations suggest that it may not be appropriate or
sufficient to consider SR IQA as a one-dimensional problem. In this work, we
develop an objective SR IQA model in a two-dimensional (2D) space of DF versus
SF. As exemplified in the bottom part of Figure 1, such a 2D approach clearly
characterizes SR algorithms, where images created by different SR algorithms
cluster at different regions in the 2D space. Moreover, such a 2D description
can be easily merged into a single overall quality/fidelity measure, for which
we find an uncertainty weighting scheme leads to superior quality prediction
performance against existing approaches when tested using subject-rated
datasets.

##  2\. Related Work

Early image SR methods are based on direct local interpolations, such as
nearest neighbor, bilinear, bicubic, cubic spline interpolation methods (Keys,
1981), and orientation-adaptive algorithms (Wang and Ward, 2007). Since only
very local information is used and certain smoothness constraints are imposed,
the reconstructed SR images often appear blurry. Another type of approaches,
termed dictionary-based SR (Zhou et al., 2019) methods, attempt to predict
high-resolution (HR) patches from LR ones. These include example-based SR
(Freeman et al., 2002), locally linear embedding (Chang et al., 2004), sparse
representation-based SR (Yang et al., 2010), semi-coupled dictionary learning
model (Wang et al., 2012), and consistent coding schemes (Yang et al., 2016).
Recently there has been a surge of deep learning-based image SR approaches
(Wang et al., 2020), including convolutional neural network (CNN)-based (Guo
et al., 2020; Xiao et al., 2021) and generative adversarial network
(GAN)-based (Ledig et al., 2017) methods.

Given multiple SR algorithms, one straightforward method to test and compare
them is to start with a high-resolution original image, downsample it to an LR
image, and then apply the SR algorithms to the LR image to generate multiple
SR images. IQA methods can then be employed to assess these images and compare
the performance of different SR algorithms. In this scenario, several types of
IQA models are applicable, which include full-reference (FR) IQA models that
use the original HR image as the pristine reference, no-reference (NR) IQA
models that evaluate the generated SR images directly, image
sharpness/blurriness models that assess the sharpness/blurriness of the
generated SR images, and SR IQA models that are fully dedicated to such SR
application scenarios.

The simplest FR methods include PSNR, mean square error (MSE) and mean
absolute error (MAE), which have been shown repeatedly in the literature to
under-perform the family of structural similarity based models, which include
the SSIM index (Wang et al., 2004), the multi-scale SSIM (MS-SSIM) index (Wang
et al., 2003), the feature similarity (FSIM) index (Zhang et al., 2011), the
complex wavelet SSIM (CW-SSIM) index (Sampat et al., 2009), the gradient
similarity (GSM) model (Liu et al., 2011), the gradient magnitude similarity
deviation (GMSD) (Xue et al., 2013), and the superpixel-based similarity
(SPSIM) index (Sun et al., 2018). Other successful FR approaches include
information theoretic models such as information fidelity criterion (IFC)
(Sheikh et al., 2005) and visual information fidelity (VIF) (Sheikh and Bovik,
2006), and internal generative mechanism (IGM) model (Wu et al., 2012). NR
methods may be derived from the natural scene statistics (NSS) models, and the
notable ones include the blind/referenceless image spatial quality evaluator
(BRISQUE) (Mittal et al., 2012a), the natural image quality evaluator (NIQE)
(Mittal et al., 2012b), the blind image integrity notator using DCT statistics
(BLIINDS-II) (Saad et al., 2012), and the distortion identification-based
image verity and integrity evaluation (DIIVINE) index (Moorthy and Bovik,
2011). NR methods may also be constructed based on local binary pattern
statistics (Wu et al., 2015), or deep learning based approaches (Ma et al.,
2017a, b; Ying et al., 2020).

Figure 2. Framework of the proposed SRIF model.

Image sharpness/blurriness has been well studied in IQA research, which
results in several successful methods including the spectral and spatial
sharpness (S3) (Vu et al., 2011) model, the local phase coherence-based
sharpness index (LPC-SI) (Hassen et al., 2013), and the HVS-MaxPol models
(Hosseini et al., 2019). Nevertheless, they are limited in addressing only one
perceptual attribute of SR images. There have been several approaches designed
specifically for SR IQA, including the structure-texture decomposition based
algorithm, i.e. SIS (Zhou et al., 2019), and the structural fidelity versus
statistical naturalness (SFSN) model (Zhou et al., 2021), though they do not
directly differentiate the deterministic and statistical aspects of signal
fidelity or employ content-adaptive weighting schemes.

##  3\. Proposed Method

Assuming an original HR image is available as the reference, the framework of
the proposed method is shown in Figure 2. Presumably, the perceptual
deterministic and statistical fidelity varies across scales, and thus we first
apply a multi-resolution decomposition to the reference and distorted images.
Specifically, we apply a three-level Gaussian-Laplacian pyramid decomposition
(Burt and Adelson, 1983) for simplicity. Empirically, we find that more
sophisticated wavelet transforms also work well but do not offer additional
gain.

Given a target image, a Gaussian pyramid is created by iterative Gaussian low-
pass filtering and downsampling, resulting in a sequence of Gaussian maps
denoted by ${G_{l}}(i,j)$ for $l=1,2,...$, where $l$ is the level of the
Gaussian map. For each Gaussian map, an interpolation operation is then
applied to expand the map by a factor of 2, leading to an expanded Gaussian
map denoted by $G_{l}^{*}(i,j)$. A Laplacian map is then obtained by
subtracting the expanded Gaussian map of the finer level from the Gaussian map
of the current level by ${L_{l}}={G_{l}}-G_{l+1}^{*}$ for $l=1,2,...$, the
collection of all such Laplacian maps constitute a Laplacian pyramid. The
Gaussian pyramids of the reference and test images are used to evaluate the
deterministic fidelity (DF), and the Laplacian pyramids of the reference and
test images are employed to assess the statistical fidelity (SF). An
uncertainty weighting scheme is then applied to produce a score that predicts
the overall quality of the test image.

###  3.1. Deterministic Fidelity

To compute local DF, we resort to the structure comparison component of the
SSIM approach (Wang et al., 2004), a simple but powerful tool to gauge the
structural change between the reference and test image patches:

(1)
$\centering{D_{local}}(x,y)=\frac{{{\sigma_{xy}}+{C_{1}}}}{{{\sigma_{x}}{\sigma_{y}}+{C_{1}}}},\@add@centering$

where $x$ and $y$ are two local image patches extracted from the reference HR
and test SR images, respectively, ${\sigma_{x}}$ and ${\sigma_{y}}$ denote
their corresponding local standard deviations, ${\sigma_{xy}}$ represents the
cross-correlation evaluation between the two patches, and $C_{1}$ is a small
positive stabilizing constant. When applying this patch level measurement with
a sliding window across one level of the Gaussian maps, which obtain a single-
scale DF map.

Figure 3. Sample information content maps. (a),(c): SR images generated by
LapSRN (Lai et al., 2017) $\times$ 2 and $\times$ 4\. (b),(d): Corresponding
information content maps in a pyramid, where brighter indicates richer
information content. The deterministic fidelity values of (a),(c) are 0.9862
and 0.9331, respectively.

Inspired by the success of the MS-SSIM and IW-SSIM methods (Wang et al., 2003;
Wang and Li, 2010), we adopt both scale weighting and information content
pooling to aggregate the DF maps. For each level, we first employ an
information content weighting to pool the single-scale DF map to a single DF
score:

(2)
$\centering{D_{l,j}}=\frac{{\sum\limits_{i}{{w_{l,j,i}}{D_{local}}({x_{l,j,i}},{y_{l,j,i}})}}}{{\sum\limits_{i}{{w_{l,j,i}}}}},\@add@centering$

where ${x_{l,j,i}}$ and ${y_{l,j,i}}$ are the $i-th$ image patches in the
$j-th$ scale for the $l-th$ level original and distorted Gaussian maps,
respectively. ${w_{l,j,i}}$ is the information content weight (Wang and Li,
2010) calculated at the $i-th$ spatial location. In Figure 3, we show an
example of information content maps for two SR images. While the LapSRN (Lai
et al., 2017) $\times$ 2 method better preserves texture details compared to
the same method $\times$ 4, its information content maps are brighter at sharp
texture and edge regions. As such, stronger weights are assigned towards these
content rich regions. Next, we combine all scale DF scores to a DF score for
each level by

(3)
$\centering{D_{l}}=\prod\limits_{j=1}^{K}{{{({D_{l,j}})}^{{\alpha_{j}}}}},\@add@centering$

where $K=5$ represents the total number of scales, and ${\alpha_{j}}$ denotes
the weights assigned to the $j-th$ scale as in MS-SSIM (Wang et al., 2003).
Finally, an overall DF measure is obtained by a weighted sum from all levels:

(4) $\centering D=\sum\limits_{l}{{w_{l}}{D_{l}}},\@add@centering$

where $w_{l}$’s are weights assigned to individual levels.

###  3.2. Statistical Fidelity

let ${L_{l}}(i,j)$ denote the Laplacian map at the $l-th$ level, we normalize
it locally by dividing it with the local standard deviation:

(5)
$\centering\widehat{{L_{l}}}(i,j)=\frac{{{L_{l}}(i,j)-\mu(i,j)}}{{\sigma(i,j)+C}},\@add@centering$

where $C$ is a small stabilizing constant, ${\mu_{l}}(i,j)$ and
${\sigma_{l}}(i,j)$ represent the mean and standard deviation of a local
region surrounding ${{L_{l}}}(i,j)$, respectively, which in our experiment are
computed using a local $3\times 3$ window.

Now assume that $p_{l}(x)$ and $q_{l}(x)$ are the probability densities of the
normalized reference and test Laplacian maps at the $l-th$ level,
respectively. We then calculate the Kullback-Leibler divergence (KLD) between
$p_{l}(x)$ and $q_{l}(x)$ to measure their difference:

(6)
$\centering{S_{l}}(p||q)=\int{{p_{l}}(x)\log\frac{{{p_{l}}(x)}}{{{q_{l}}(x)}}dx}.\@add@centering$

An example is given in Figure 4, where we show the histograms of normalized
Laplacian maps of the original HR image and two test SR images produced by two
SR algorithms. Perceptually the SRGAN (Ledig et al., 2017) generated SR image
reveals more texture details as in the reference image, outperforming the
example-based SR method (Freeman et al., 2002). Meanwhile, the histogram of
the SRGAN image appears more similar to that of the original image, which is
well reflected by the KLD measure, suggesting KLD is a promising indicator of
SF. Finally, the overall SF measure is computed by a weighted sum of SF from
each level:

(7) $\centering S=\sum\limits_{l}{{w_{l}}{S_{l}}},\@add@centering$

where $w_{l}$’s are weights assigned to individual levels.

###  3.3. Uncertainty Weighting

Both DF and SF are sensible measures that when working individually provides
meaningful assessment about the intrinsic characteristics of SR algorithms
under testing. In certain application scenarios, however, one desires to have
a single quality or fidelity score that considers both fidelity measures. A
direct combination, e.g., a weighted average with fixed weights, would not be
an optimal solution because the importance and uncertainty of the DF and SF
measures may be content dependent. For example, different content exhibits
different sharpness and texture richness, which lead to different levels of
uncertainty when DF and SF are employed as the overall quality predictors.

We extract sharpness and texture richness features using the same Gaussian and
Laplacian pyramids created for the reference and test images. The LPC-SI
approach (Hassen et al., 2013) is first employed for sharpness assessment:

(8) $\centering sr=\frac{{{l_{d-G1}}}}{{{l_{o-G2}}}},\@add@centering$

where ${l_{d-G1}}$ and ${l_{o-G2}}$ denote the LPC-SI values computed from the
first level of the Gaussian pyramid of the test image and the second level of
the Gaussian pyramid of the original image, respectively. We then use entropy
as a measure for texture richness and define a texture richness ratio as

(9) $\centering tr=\frac{{{e_{d-L1}}}}{{{e_{o-G2}}}},\@add@centering$

where ${e_{d-L1}}$ and ${e_{o-G2}}$ represent the entropy of the first level
of the normalized Laplacian map of the test image and that of the second level
of the reference Gaussian pyramid, respectively. The sharpness and texture
richness ratios are then integrated to an assorted factor as

(10) $\centering f={(sr)^{\alpha}}+{(tr)^{\alpha}},\@add@centering$

where an exponential factor $\alpha$ is added to stretch the value range.

Figure 4. Illustration of SF assessment. (a-c): Reference and SR images
produced by Example-based SR (Freeman et al., 2002) and SRGAN (Ledig et al.,
2017), respectively; (d-f): Corresponding histograms of normalized Laplacian
pyramid. The KLDs between (d) and (e) and between (d) and (f) are 0.0162 and
0.0138, respectively, suggesting SR image (c) better preserves SF. Figure 5.
Weights assigned to DF and SF measures with respect to the assorted factor.
Figure 6. Analysis of different SR algorithms in (DF, SF) space. A1-A2, B1-B2,
and C1-C2: SR images created by Bilinear interpolation, SCDL (Wang et al.,
2012), and DCSCN (Yamanaka et al., 2017) at scaling factor 4, respectively.
Bottom: Clusters of SR algorithms in (DF, SF) space. Figure 7. Analysis of
various SR categories in (DF, SF) space. A1-A2: SR images reconstructed by
Bilinear and Bicubic interpolations; B1-B2: SR images produced by SCDL (Wang
et al., 2012) and Fast SR (Yang and Yang, 2013); C1-C2: SR images generated by
DCSCN (Yamanaka et al., 2017) and LapSRN (Lai et al., 2017). The scaling
factor is 4 for all cases. Bottom: Clusters of SR categories in (DF, SF)
space.

To evaluate how the prediction uncertainty of the DF and SF measures changes
with respect to the assorted factor, we bin images along the axis of the
assorted factor and compute the variances of the prediction errors for DF and
SF within each bin. Presumably, a larger variance implies larger uncertainty,
and thus should carry less weight in the overall prediction. Therefore, we
compute the uncertainty weighting factors for DF and SF at each bin by

(11) $\centering{w_{d}}=\frac{{{v_{s}}}}{{{v_{d}}+{v_{s}}}},\@add@centering$
(12) $\centering{w_{s}}=\frac{{{v_{d}}}}{{{v_{d}}+{v_{s}}}},\@add@centering$

where $v_{d}$ and $v_{s}$ are the variances of the prediction errors of the DF
and SF measures, respectively. Figure 5 shows how these weighting factors
change against the assorted factor. It can be observed that generally DF
carries more weights than SF, but as the assorted factor increases, more
weights gradually shift towards the SF measure, which is intuitively sensible
because an image with larger assorted factor contains richer textures that are
better accounted for by statistical measurements. Finally, the overall
uncertainty-weighted Super Resolution Image Fidelity (SRIF) index is
calculated by

(13) $\centering{Q_{ds}}={w_{d}}D+{w_{s}}S\,.\@add@centering$

##  4\. Validation

We validate the proposed method and compare it with other objective quality
models using three publicly available SR quality datasets, including the
Waterloo Interpolation (Yeganeh et al., 2015), QADS (Zhou et al., 2019) and
CVIU (Ma et al., 2017c) databases. The Waterloo Interpolation (Yeganeh et al.,
2015) database involves 8 interpolation algorithms with 3 interpolation
factors of 2, 4, and 8, respectively. 312 SR images are generated from 13
source images. The QADS (Zhou et al., 2019) database contains 20 original HR
images and 980 SR images created by 21 image SR algorithms, including 4
interpolation-based, 11 dictionary-based, and 6 DNN-based SR models, with
upsampling factors equaling to 2, 3, and 4. Each SR image is associated with a
mean opinion score (MOS) of 100 subjects. In the CVIU database (Ma et al.,
2017c), 1620 SR images are produced by 9 SR approaches from 30 source images.
6 pairs of scaling factors and kernel widths are adopted, where a larger
subsampling factor corresponds to a larger blur kernel width. Each image is
rated by 50 subjects, and the mean of the median 40 scores is calculated for
each image as the MOS.

Figure 8. Analysis of DNN-based SR algorithms in (DF, SF) space. VDSR (Kim et
al., 2016), VGGNet (Simonyan and Zisserman, 2014), DCSCN (Yamanaka et al.,
2017) and DRRN (Tai et al., 2017) are CNN-based methods, while SRGAN (Ledig et
al., 2017) is GAN-based method. Bottom: Quality assessment in (DF, SF) space.

###  4.1. 2D Analysis of SR Algorithms

As mentioned earlier, the behaviors of different SR algorithms may be
drastically different, and it is difficult to use one scalar number to
describe the visual quality of images generated by these algorithms. The
proposed 2D (DF, SF) framework provides a useful tool to analyze the
characteristics of the SR algorithms.

Figure 6 shows such an example in comparing different SR algorithms and
analyzing their strengths and weaknesses. It can be observed that Bilinear
Interpolation blurs the image, loses the texture details, and severely
deteriorates the statistical properties or SF of the original image. Even
though the SCDL (Wang et al., 2012) generated image appears a bit better on
the statistical properties, it performs poorly in preserving the authentic
structures or DF of the original image. By contrast, the DCSCN (Yamanaka et
al., 2017) algorithm offers impressive performance in preserving both
deterministic structures and statistical properties of the original image. All
of these observations are clearly indicated in the 2D (DF, SF) space.

It is also interesting to observe the behaviors of different categories of SR
algorithms. In particular, there are totally 21 SR algorithms in the QADS
database (Zhou et al., 2019), which are divided into three categories: 4
interpolation-based, 11 dictionary-based, and 6 DNN-based methods. In Figure
7, we carry out a similar performance analysis and comparison in the (DF, SF)
space but on different categories of SR algorithms. The results reveal
complementary advantages of interpolation and dictionary-based SR algorithms,
and demonstrates the strong performance of DNN-based methods. The conclusion
aligns very well with visual observations of the reconstructed SR images. We
further analyze various DNN-based SR algorithms, as shown in Figure 8. The
general trends of CNN-based SR models are consistent, where DF and SF are
inclined to improve together. But the behavior of GAN-based SR method (i.e.
SRGAN) is dramatically different, which tends to sacrifice DF for SF.

###  4.2. Performance Evaluation and Comparison

Given a dataset containing SR images and their corresponding subjective
quality scores, we evaluate and compare objective models for their quality
prediction performance using four criteria, namely Spearman Rank-order
Correlation Coefficient (SRCC), Kendall Rank-order Correlation Coefficient
(KRCC), Pearson Linear Correlation Coefficient (PLCC), and Root Mean Squared
Error (RMSE). SRCC and KRCC are used to assess prediction monotonicity, and
PLCC and RMSE are employed to evaluate prediction accuracy. Before PLCC and
RMSE are computed, a five-parameter modified logistic nonlinear fitting
function (Rohaly et al., 2000) is employed to map the predicted quality scores
into a linear scale of the subjective scores by

(14)
$g(x)={\beta_{1}}(\frac{1}{2}-\frac{1}{{1+{e^{({\beta_{2}}(x-{\beta_{3}}))}}}})+{\beta_{4}}x+{\beta_{5}},$

where $({\beta_{1}}$ to ${\beta_{5}})$ are fitting parameters, and $x$ and
$g(x)$ represent the raw and regressed objective scores, respectively.

Table 1. Performance comparison on Waterloo Interpolation (Yeganeh et al., 2015), QADS (Zhou et al., 2019) and CVIU (Ma et al., 2017c) quality databases. |  | Waterloo Interpolation | QADS | CVIU
---|---|---|---|---
Types | Methods | SRCC | KRCC | PLCC | RMSE | SRCC | KRCC | PLCC | RMSE | SRCC | KRCC | PLCC | RMSE
FR IQA Models | PSNR | 0.6320 | 0.4421 | 0.6303 | 2.0020 | 0.3544 | 0.2441 | 0.3897 | 0.2530 | 0.5663 | 0.3943 | 0.5779 | 1.9620
SSIM (Wang et al., 2004) | 0.6125 | 0.4311 | 0.6208 | 2.0215 | 0.5290 | 0.3689 | 0.5327 | 0.2325 | 0.6285 | 0.4429 | 0.6497 | 1.8277
MS-SSIM (Wang et al., 2003) | 0.8246 | 0.6228 | 0.8371 | 1.4108 | 0.7172 | 0.5299 | 0.7240 | 0.1895 | 0.8048 | 0.6011 | 0.8114 | 1.4052
FSIM (Zhang et al., 2011) | 0.8503 | 0.6474 | 0.8595 | 1.3179 | 0.6885 | 0.5020 | 0.6902 | 0.1988 | 0.7481 | 0.5450 | 0.7628 | 1.5546
CW-SSIM (Sampat et al., 2009) | 0.8626 | 0.6658 | 0.9056 | 1.0935 | 0.3259 | 0.2275 | 0.3792 | 0.2542 | 0.7591 | 0.5410 | 0.7541 | 1.5790
IFC (Sheikh et al., 2005) | 0.9117 | 0.7227 | 0.9350 | 0.9146 | 0.8609 | 0.6816 | 0.8657 | 0.1375 | 0.8705 | 0.6807 | 0.8836 | 1.1259
VIF (Sheikh and Bovik, 2006) | 0.8928 | 0.6973 | 0.9053 | 1.0953 | 0.8152 | 0.6249 | 0.8210 | 0.1568 | 0.8571 | 0.6630 | 0.8671 | 1.1974
GSM (Liu et al., 2011) | 0.7649 | 0.5650 | 0.7773 | 1.6224 | 0.5538 | 0.3908 | 0.5684 | 0.2260 | 0.6505 | 0.4587 | 0.6717 | 1.7809
IGM (Wu et al., 2012) | 0.8659 | 0.6675 | 0.8841 | 1.2051 | 0.7145 | 0.5232 | 0.7192 | 0.1909 | 0.8375 | 0.6375 | 0.8442 | 1.2885
GMSD (Xue et al., 2013) | 0.7966 | 0.5915 | 0.8108 | 1.5092 | 0.7650 | 0.5689 | 0.7749 | 0.1736 | 0.8469 | 0.6495 | 0.8499 | 1.2669
SPSIM (Sun et al., 2018) | 0.8141 | 0.6143 | 0.8263 | 1.4524 | 0.5751 | 0.4071 | 0.5829 | 0.2232 | 0.6698 | 0.4755 | 0.6902 | 1.7396
NR IQA Models | BRISQUE (Mittal et al., 2012a) | 0.7676 | 0.5610 | 0.7871 | 1.5907 | 0.5463 | 0.3834 | 0.5478 | 0.2298 | 0.5863 | 0.4262 | 0.6094 | 1.9061
NIQE (Mittal et al., 2012b) | 0.6263 | 0.4647 | 0.6716 | 1.9105 | 0.3977 | 0.2788 | 0.4044 | 0.2512 | 0.6525 | 0.4775 | 0.6657 | 1.7940
BLIINDS-II (Saad et al., 2012) | 0.5281 | 0.3531 | 0.5260 | 2.1932 | 0.3838 | 0.2617 | 0.4490 | 0.2454 | 0.3705 | 0.2584 | 0.4389 | 2.1602
DIIVINE (Moorthy and Bovik, 2011) | 0.5465 | 0.3778 | 0.5719 | 2.1158 | 0.4817 | 0.3314 | 0.5044 | 0.2372 | 0.5479 | 0.3956 | 0.6347 | 1.8577
LPSI (Wu et al., 2015) | 0.6669 | 0.4642 | 0.7007 | 1.8396 | 0.4079 | 0.2890 | 0.4217 | 0.2491 | 0.4883 | 0.3503 | 0.5374 | 2.0274
dipIQ (Ma et al., 2017a) | 0.6560 | 0.4617 | 0.6727 | 1.9080 | 0.5057 | 0.3529 | 0.5296 | 0.2330 | 0.4901 | 0.3640 | 0.5212 | 2.0518
MEON (Ma et al., 2017b) | 0.8554 | 0.6286 | 0.8672 | 1.2843 | 0.6139 | 0.4528 | 0.6162 | 0.2163 | 0.4599 | 0.3253 | 0.5124 | 2.0645
PaQ-2-PiQ (Ying et al., 2020) | 0.7479 | 0.5214 | 0.7808 | 1.6112 | 0.7473 | 0.5465 | 0.7514 | 0.1812 | 0.7294 | 0.5403 | 0.7376 | 1.6233
Sharpness Models | S3 (Vu et al., 2011) | 0.4455 | 0.3099 | 0.4999 | 2.2334 | 0.4636 | 0.3198 | 0.4720 | 0.2422 | 0.5050 | 0.3519 | 0.5480 | 2.0109
LPC-SI (Hassen et al., 2013) | 0.5375 | 0.3725 | 0.5944 | 2.0736 | 0.4902 | 0.3358 | 0.5027 | 0.2374 | 0.5450 | 0.3921 | 0.5731 | 1.9701
HVS-MaxPol-1 (Hosseini et al., 2019) | 0.6166 | 0.4329 | 0.6577 | 1.9425 | 0.6170 | 0.4322 | 0.6212 | 0.2153 | 0.6421 | 0.4661 | 0.6703 | 1.7840
HVS-MaxPol-2 (Hosseini et al., 2019) | 0.6309 | 0.4284 | 0.6680 | 1.9190 | 0.5736 | 0.3986 | 0.5847 | 0.2228 | 0.6313 | 0.4517 | 0.6474 | 1.8324
SR IQA Models | SIS (Zhou et al., 2019) | 0.8777 | 0.6773 | 0.8913 | 1.1692 | 0.9132 | 0.7397 | 0.9137 | 0.1116 | 0.8694 | 0.6855 | 0.8973 | 1.0611
SFSN (Zhou et al., 2021) | 0.8867 | 0.6917 | 0.9058 | 1.0928 | 0.8407 | 0.6553 | 0.8445 | 0.1471 | 0.8714 | 0.6800 | 0.8848 | 1.1202
SRIF (proposed) | 0.9157 | 0.7299 | 0.9525 | 0.7855 | 0.9163 | 0.7457 | 0.9174 | 0.1093 | 0.8857 | 0.7042 | 0.9018 | 1.0389

In addition to SRIF, a total of 25 other models are included in performance
comparison. These include eleven FR IQA, eight NR IQA, four sharpness models,
and two SR IQA approaches.

The results using aforementioned datasets and evaluation criteria are reported
in Table 1, where the top performers of the FR IQA, NR IQA and sharpness model
categories are IFC (Sheikh et al., 2005), MEON (Ma et al., 2017b) or PaQ-2-PiQ
(Ying et al., 2020), and HVS-MaxPol (Hosseini et al., 2019), respectively, but
in general, they underperform the models dedicated to SR IQA. Of all the
models under comparison, the proposed SRIF model produces the best performance
for all evaluation criteria in all three databases.

To investigate deeper on the performance of objective quality models for
different categories of SR algorithms, we report SRCC results of these models
separately for interpolation-based, dictionary-based and DNN-based SR
approaches on the QADS dataset (Zhou et al., 2019) in Table 2. It is
interesting to observe that FR IQA methods usually performs better in
evaluating dictionary-based SR algorithms, while most NR IQA models show
relatively better performance on DNN-based against the other two categories of
SR approaches. The proposed SRIF model offers consistently good performance
across all three categories.

Table 2. SRCC Performance comparison of objective quality models on QADS (Zhou et al., 2019) database. Types | Methods | Interpolation-based | Dictionary-based | DNN-based | All
---|---|---|---|---|---
FR IQA Models | PSNR | 0.2972 | 0.3808 | 0.2656 | 0.3544
SSIM (Wang et al., 2004) | 0.4015 | 0.5481 | 0.5121 | 0.5290
MS-SSIM (Wang et al., 2003) | 0.6340 | 0.7425 | 0.7104 | 0.7172
FSIM (Zhang et al., 2011) | 0.5471 | 0.6846 | 0.6637 | 0.6885
CW-SSIM (Sampat et al., 2009) | 0.5254 | 0.4362 | 0.0986 | 0.3259
IFC (Sheikh et al., 2005) | 0.8489 | 0.8835 | 0.7792 | 0.8609
VIF (Sheikh and Bovik, 2006) | 0.7654 | 0.8305 | 0.7281 | 0.8152
GSM (Liu et al., 2011) | 0.3946 | 0.5332 | 0.5661 | 0.5538
IGM (Wu et al., 2012) | 0.6285 | 0.7292 | 0.6625 | 0.7145
GMSD (Xue et al., 2013) | 0.7054 | 0.7709 | 0.7363 | 0.7650
SPSIM (Sun et al., 2018) | 0.4545 | 0.5518 | 0.5871 | 0.5751
NR IQA Models | BRISQUE (Mittal et al., 2012a) | 0.5096 | 0.4951 | 0.4357 | 0.5463
NIQE (Mittal et al., 2012b) | 0.4639 | 0.4547 | 0.4190 | 0.3977
BLIINDS-II (Saad et al., 2012) | 0.1814 | 0.3628 | 0.6547 | 0.3838
DIIVINE (Moorthy and Bovik, 2011) | 0.4267 | 0.4175 | 0.5654 | 0.4817
LPSI (Wu et al., 2015) | 0.2726 | 0.3309 | 0.6034 | 0.4079
dipIQ (Ma et al., 2017a) | 0.3368 | 0.4980 | 0.4021 | 0.5057
MEON (Ma et al., 2017b) | 0.4809 | 0.5956 | 0.6951 | 0.6139
PaQ-2-PiQ (Ying et al., 2020) | 0.7272 | 0.6853 | 0.5533 | 0.7473
Sharpness Models | S3 (Vu et al., 2011) | 0.4016 | 0.3171 | 0.5458 | 0.4636
LPC-SI (Hassen et al., 2013) | 0.3301 | 0.3798 | 0.2558 | 0.4902
HVS-MaxPol-1 (Hosseini et al., 2019) | 0.4584 | 0.5048 | 0.5032 | 0.6170
HVS-MaxPol-2 (Hosseini et al., 2019) | 0.5318 | 0.4742 | 0.2991 | 0.5736
SR IQA Models | SIS (Zhou et al., 2019) | 0.8778 | 0.9038 | 0.8493 | 0.9132
SFSN (Zhou et al., 2021) | 0.8979 | 0.8379 | 0.8004 | 0.8407
SRIF (proposed) | 0.9119 | 0.8950 | 0.8778 | 0.9163
Table 3. Ablation test of the proposed SRIF model on Waterloo Interpolation (Yeganeh et al., 2015), QADS (Zhou et al., 2019) and CVIU (Ma et al., 2017c) databases. | Waterloo Interpolation | QADS | CVIU
---|---|---|---
Models | SRCC | KRCC | PLCC | RMSE | SRCC | KRCC | PLCC | RMSE | SRCC | KRCC | PLCC | RMSE
DF only | 0.9152 | 0.7265 | 0.9517 | 0.7918 | 0.7813 | 0.5929 | 0.7890 | 0.1688 | 0.8529 | 0.6547 | 0.8702 | 1.1844
SF only | 0.8147 | 0.5867 | 0.8414 | 1.3936 | 0.8480 | 0.6625 | 0.8501 | 0.1446 | 0.8154 | 0.6169 | 0.8246 | 1.3601
SRIF | 0.9157 | 0.7299 | 0.9525 | 0.7855 | 0.9163 | 0.7457 | 0.9174 | 0.1093 | 0.8857 | 0.7042 | 0.9018 | 1.0389
Table 4. Comparison of weighting strategies on Waterloo Interpolation (Yeganeh et al., 2015), QADS (Zhou et al., 2019) and CVIU (Ma et al., 2017c) databases. | Waterloo Interpolation | QADS | CVIU
---|---|---|---
Methods | SRCC | KRCC | PLCC | RMSE | SRCC | KRCC | PLCC | RMSE | SRCC | KRCC | PLCC | RMSE
Average Weighting | 0.8908 | 0.6860 | 0.9287 | 0.9564 | 0.8863 | 0.7092 | 0.8906 | 0.1249 | 0.8750 | 0.6857 | 0.8870 | 1.1101
Proposed Uncertainty Weighting | 0.9157 | 0.7299 | 0.9525 | 0.7855 | 0.9163 | 0.7457 | 0.9174 | 0.1093 | 0.8857 | 0.7042 | 0.9018 | 1.0389

###  4.3. Ablation Tests

It is desirable to know how the individual DF and SF components attribute to
the overall performance of the SRIF model. In Table 3, we compare the
performance of the DF only model, the SF only model, and the overall SRIF
model on the Waterloo Interpolation (Yeganeh et al., 2015), QADS (Zhou et al.,
2019) and CVIU (Ma et al., 2017c) databases. It can be seen that for the
Waterloo Interpolation dataset, where only interpolation is considered, DF can
deliver relatively good performance. This is reasonable since interpolation-
based SR algorithms often blur the image, which protects the primary
structure. However, when a variety of SR categories are involved, such as the
QADS and CVIU datasets, the DF only model is insufficient. In general, we show
that both the DF and SF components offer competitive performance individually,
but the overall SRIF model elevates the performance to an even higher level,
suggesting that the DF and SF components, to some extent, complement each
other.

To understand the contribution of the uncertainty weighting scheme in SRIF, we
compare SRIF with a naïve model that directly averages the DF and SF
components. Table 4 shows the comparison results based on the Waterloo
Interpolation (Yeganeh et al., 2015), QADS (Zhou et al., 2019) and CVIU (Ma et
al., 2017c) databases. The proposed uncertainty weighting scheme performs
better in all test cases. Moreover, we show the scatter plots in Figure 9 of
MOS versus model predictions using average and uncertainty weighting schemes,
respectively, where each point represents a test SR image. It appears that the
points in the scatter plot corresponding to uncertainty weighting are more
tightly concentrated along the diagonal line, indicating better prediction
accuracy from objective model prediction to human evaluation.

Figure 9. Scatter plots of subjective scores against objective model
predictions with different weighting strategies. (a) average weighting; (b)
proposed uncertainty weighting.

##  5\. Conclusion

In this work, we attempt to understand SR IQA as a 2D problem in terms of DF
and SF. We show that different SR algorithms and categories of SR algorithms
exhibit dramatically different behaviors in the 2D space of (DF, SF). We
develop separate DF and SF models as well as an uncertainty weighting scheme
that combines both fidelity models into an overall quality prediction. The
resulting SRIF index outperforms existing methods in the literature when
tested using subject-rated SR IQA databases.

In the future, the current work may be extended by developing better DF and SF
models or fusion schemes. It would also be interesting to use the 2D approach
as a tool to investigate deeper into different SR algorithms to observe how
different operations navigate in the 2D space. Furthermore, the 2D approach
and the proposed SRIF model may be used to redefine the cost function and
improve the design of future SR algorithms for optimal perceptual quality.

###### Acknowledgements.

This work is in part supported by Natural Sciences and Engineering Research
Council of Canada and Canada Research Chair program.

## References

  * (1)
  * Burt and Adelson (1983) Peter Burt and Edward Adelson. 1983.  The Laplacian pyramid as a compact image code.  _IEEE Transactions on Communications_ 31, 4 (1983), 532–540.
  * Chang et al. (2004) Hong Chang, Dit-Yan Yeung, and Yimin Xiong. 2004\.  Super-resolution through neighbor embedding. In _CVPR_. I–I.
  * Freeman et al. (2002) William T Freeman, Thouis R Jones, and Egon C Pasztor. 2002\.  Example-based super-resolution.  _IEEE Computer Graphics and Applications_ 22, 2 (2002), 56–65.
  * Guo et al. (2020) Jingcai Guo, Shiheng Ma, Jie Zhang, Qihua Zhou, and Song Guo. 2020.  Dual-view attention networks for single image super-resolution. In _Proceedings of the 28th ACM International Conference on Multimedia_. 2728–2736.
  * Hassen et al. (2013) Rania Hassen, Zhou Wang, and Magdy MA Salama. 2013.  Image sharpness assessment based on local phase coherence.  _IEEE Transactions on Image Processing_ 22, 7 (2013), 2798–2810.
  * Hosseini et al. (2019) Mahdi S Hosseini, Yueyang Zhang, and Konstantinos N Plataniotis. 2019\.  Encoding visual sensitivity by maxpol convolution filters for image sharpness assessment.  _IEEE Transactions on Image Processing_ 28, 9 (2019), 4510–4525.
  * Keys (1981) Robert Keys. 1981\.  Cubic convolution interpolation for digital image processing.  _IEEE Transactions on Acoustics, Speech, and Signal Processing_ 29, 6 (1981), 1153–1160.
  * Kim et al. (2016) Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016.  Accurate image super-resolution using very deep convolutional networks. In _CVPR_. 1646–1654.
  * Lai et al. (2017) Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2017.  Deep laplacian pyramid networks for fast and accurate super-resolution. In _CVPR_. 624–632.
  * Ledig et al. (2017) Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017\.  Photo-realistic single image super-resolution using a generative adversarial network. In _CVPR_. 4681–4690.
  * Liu et al. (2011) Anmin Liu, Weisi Lin, and Manish Narwaria. 2011.  Image quality assessment based on gradient similarity.  _IEEE Transactions on Image Processing_ 21, 4 (2011), 1500–1512.
  * Ma et al. (2017c) Chao Ma, Chih-Yuan Yang, Xiaokang Yang, and Ming-Hsuan Yang. 2017c.  Learning a no-reference quality metric for single-image super-resolution.  _Computer Vision and Image Understanding_ 158 (2017), 1–16.
  * Ma et al. (2017a) Kede Ma, Wentao Liu, Tongliang Liu, Zhou Wang, and Dacheng Tao. 2017a.  dipIQ: Blind image quality assessment by learning-to-rank discriminable image pairs.  _IEEE Transactions on Image Processing_ 26, 8 (2017), 3951–3964.
  * Ma et al. (2017b) Kede Ma, Wentao Liu, Kai Zhang, Zhengfang Duanmu, Zhou Wang, and Wangmeng Zuo. 2017b.  End-to-end blind image quality assessment using deep neural networks.  _IEEE Transactions on Image Processing_ 27, 3 (2017), 1202–1213.
  * Mittal et al. (2012a) Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. 2012a.  No-reference image quality assessment in the spatial domain.  _IEEE Transactions on Image Processing_ 21, 12 (2012), 4695–4708.
  * Mittal et al. (2012b) Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. 2012b.  Making a “completely blind” image quality analyzer.  _IEEE Signal Processing Letters_ 20, 3 (2012), 209–212.
  * Moorthy and Bovik (2011) Anush Krishna Moorthy and Alan Conrad Bovik. 2011.  Blind image quality assessment: From natural scene statistics to perceptual quality.  _IEEE Transactions on Image Processing_ 20, 12 (2011), 3350–3364.
  * Park et al. (2003) Sung Cheol Park, Min Kyu Park, and Moon Gi Kang. 2003\.  Super-resolution image reconstruction: a technical overview.  _IEEE Signal Processing Magazine_ 20, 3 (2003), 21–36.
  * Reibman et al. (2006) Amy R Reibman, Robert M Bell, and Sharon Gray. 2006.  Quality assessment for super-resolution image enhancement. In _IEEE International Conference on Image Processing_. 2017–2020.
  * Rohaly et al. (2000) Ann Marie Rohaly, Philip J Corriveau, John M Libert, Arthur A Webster, Vittorio Baroncini, John Beerends, Jean-Louis Blin, Laura Contin, Takahiro Hamada, David Harrison, et al. 2000\.  Video quality experts group: Current results and future directions. In _Visual Communications and Image Processing_ , Vol. 4067. SPIE, 742–753.
  * Saad et al. (2012) Michele A Saad, Alan C Bovik, and Christophe Charrier. 2012\.  Blind image quality assessment: A natural scene statistics approach in the DCT domain.  _IEEE Transactions on Image Processing_ 21, 8 (2012), 3339–3352.
  * Sampat et al. (2009) Mehul P Sampat, Zhou Wang, Shalini Gupta, Alan Conrad Bovik, and Mia K Markey. 2009.  Complex wavelet structural similarity: A new image similarity index.  _IEEE Transactions on Image Processing_ 18, 11 (2009), 2385–2401.
  * Sheikh and Bovik (2006) Hamid R Sheikh and Alan C Bovik. 2006.  Image information and visual quality.  _IEEE Transactions on Image Processing_ 15, 2 (2006), 430–444.
  * Sheikh et al. (2005) Hamid R Sheikh, Alan C Bovik, and Gustavo De Veciana. 2005\.  An information fidelity criterion for image quality assessment using natural scene statistics.  _IEEE Transactions on Image Processing_ 14, 12 (2005), 2117–2128.
  * Shi et al. (2019) Guangming Shi, Wenfei Wan, Jinjian Wu, Xuemei Xie, Weisheng Dong, and Hong Ren Wu. 2019\.  SISRSet: Single image super-resolution subjective evaluation test and objective quality assessment.  _Neurocomputing_ 360 (2019), 37–51.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * Sun et al. (2018) Wen Sun, Qingmin Liao, Jing-Hao Xue, and Fei Zhou. 2018\.  SPSIM: A superpixel-based similarity index for full-reference image quality assessment.  _IEEE Transactions on Image Processing_ 27, 9 (2018), 4232–4244.
  * Tai et al. (2017) Ying Tai, Jian Yang, and Xiaoming Liu. 2017.  Image super-resolution via deep recursive residual network. In _CVPR_. 3147–3155.
  * Vu et al. (2011) Cuong T Vu, Thien D Phan, and Damon M Chandler. 2011.  S3: A spectral and spatial measure of local perceived sharpness in natural images.  _IEEE Transactions on Image Processing_ 21, 3 (2011), 934–945.
  * Wang et al. (2017) Guangcheng Wang, Leida Li, Qiaohong Li, Ke Gu, Zhaolin Lu, and Jiansheng Qian. 2017\.  Perceptual evaluation of single-image super-resolution reconstruction. In _IEEE International Conference on Image Processing_. 3145–3149.
  * Wang and Ward (2007) Qing Wang and Rabab Kreidieh Ward. 2007.  A new orientation-adaptive interpolation method.  _IEEE Transactions on Image Processing_ 16, 4 (2007), 889–900.
  * Wang et al. (2012) Shenlong Wang, Lei Zhang, Yan Liang, and Quan Pan. 2012\.  Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In _CVPR_. 2216–2223.
  * Wang et al. (2004) Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004\.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ 13, 4 (2004), 600–612.
  * Wang et al. (2020) Zhihao Wang, Jian Chen, and Steven CH Hoi. 2020.  Deep learning for image super-resolution: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 43, 10 (2020), 3365–3387.
  * Wang and Li (2010) Zhou Wang and Qiang Li. 2010.  Information content weighting for perceptual image quality assessment.  _IEEE Transactions on Image Processing_ 20, 5 (2010), 1185–1198.
  * Wang et al. (2003) Zhou Wang, Eero P Simoncelli, and Alan C Bovik. 2003\.  Multiscale structural similarity for image quality assessment. In _The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003_, Vol. 2. IEEE, 1398–1402.
  * Wu et al. (2012) Jinjian Wu, Weisi Lin, Guangming Shi, and Anmin Liu. 2012\.  Perceptual quality metric with internal generative mechanism.  _IEEE Transactions on Image Processing_ 22, 1 (2012), 43–54.
  * Wu et al. (2015) Qingbo Wu, Zhou Wang, and Hongliang Li. 2015.  A highly efficient method for blind image quality assessment. In _IEEE International Conference on Image Processing_. 339–343.
  * Xiao et al. (2021) Jun Xiao, Qian Ye, Rui Zhao, Kin-Man Lam, and Kao Wan. 2021.  Self-feature Learning: An Efficient Deep Lightweight Network for Image Super-resolution. In _Proceedings of the 29th ACM International Conference on Multimedia_. 4408–4416.
  * Xue et al. (2013) Wufeng Xue, Lei Zhang, Xuanqin Mou, and Alan C Bovik. 2013\.  Gradient magnitude similarity deviation: A highly efficient perceptual image quality index.  _IEEE Transactions on Image Processing_ 23, 2 (2013), 684–695.
  * Yamanaka et al. (2017) Jin Yamanaka, Shigesumi Kuwashima, and Takio Kurita. 2017\.  Fast and accurate image super resolution by deep CNN with skip connection and network in network. In _International Conference on Neural Information Processing_. Springer, 217–225.
  * Yang et al. (2014) Chih-Yuan Yang, Chao Ma, and Ming-Hsuan Yang. 2014.  Single-image super-resolution: A benchmark. In _ECCV_. 372–386.
  * Yang and Yang (2013) Chih-Yuan Yang and Ming-Hsuan Yang. 2013.  Fast direct super-resolution by simple functions. In _ICCV_. 561–568.
  * Yang et al. (2010) Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma. 2010\.  Image super-resolution via sparse representation.  _IEEE Transactions on Image Processing_ 19, 11 (2010), 2861–2873.
  * Yang et al. (2016) Wenming Yang, Yapeng Tian, Fei Zhou, Qingmin Liao, Hai Chen, and Chenglin Zheng. 2016\.  Consistent coding scheme for single-image super-resolution via independent dictionaries.  _IEEE Transactions on Multimedia_ 18, 3 (2016), 313–325.
  * Yeganeh et al. (2015) Hojatollah Yeganeh, Mohammad Rostami, and Zhou Wang. 2015\.  Objective quality assessment of interpolated natural images.  _IEEE Transactions on Image Processing_ 24, 11 (2015), 4651–4663.
  * Ying et al. (2020) Zhenqiang Ying, Haoran Niu, Praful Gupta, Dhruv Mahajan, Deepti Ghadiyaram, and Alan Bovik. 2020\.  From patches to pictures (PaQ-2-PiQ): Mapping the perceptual space of picture quality. In _CVPR_. 3575–3585.
  * Zhang et al. (2011) Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. 2011\.  FSIM: A feature similarity index for image quality assessment.  _IEEE Transactions on Image Processing_ 20, 8 (2011), 2378–2386.
  * Zhou et al. (2019) Fei Zhou, Rongguo Yao, Bozhi Liu, and Guoping Qiu. 2019\.  Visual quality assessment for super-resolved images: database and method.  _IEEE Transactions on Image Processing_ 28, 7 (2019), 3528–3541.
  * Zhou et al. (2021) Wei Zhou, Zhou Wang, and Zhibo Chen. 2021.  Image super-resolution quality assessment: Structural fidelity versus statistical naturalness. In _IEEE International Conference on Quality of Multimedia Experience_. 61–64.


# Thermal bistability in laser-cooled trapped ions

A. Poindron, J. Pedregosa-Gutierrez, C. Champenois  Aix-Marseille Université,
CNRS, PIIM, Marseille, France

###### Abstract

The non-linear dynamics of large ion clouds ($N\geq 256$ ions) trapped in
radio-frequency traps and coupled to laser-cooling give rise to a bistable
behaviour of the temperature. Numerical simulations of the free evolution of a
large three-dimensional spherical cloud is used to characterise how the
oscillating field amplitude and the ion number control the rf heating rate and
the switching between the two stable states. We show that the heating rate
does not significantly depends on the ion number but strongly depends on the
oscillating field. This exhibits the major role played by the density of the
ensemble. The competition between the radio-frequency heating and the laser
cooling is also discussed. They are used to design scenarii to take advantage
of the threshold effect to detect and quantify perturbations by an intruder.

##  I Introduction

Ion traps are experimental tools used to manipulate atomic or molecular ions
in quantities ranging from a single ion [1] to thousands of ions [2].

Experiments based on large ion cloud remain a very relevant choice for micro-
wave frequency standards [2], high precision spectroscopy of cold molecular
ions [3, 4, 5], chemical reaction in the cold and diluted limit [6] or quantum
optics in the large cooperative regime [7, 8, 9]. In these experiments where
the cold ion cloud expands in the three directions, the rf-driven motion (also
known as micro-motion) is one of the factors limiting resolution and accuracy
in spectroscopy because of the induced Doppler shift [10, 11, 12]. It is also
the source of the radio-frequency heating, an energy transfer from the rf
source to the thermal motion of the ions, mediated by the Coulomb interaction.
This rf-heating is responsible for fast transitions from low to high
temperature state, referred as an ”explosive onset” in the first studies about
chaotic trajectories of few ions system [13]. This is the signature of a
strong non-linear dynamics driven by the Coulomb interactions. So far, few
molecular dynamics simulations have studied such features like in [14], where
the dynamics of ion crystal as large as 512 in a 3D-Paul trap concludes that
ion clouds at 5 mK show rapid heating when the rf electric field amplitude is
large enough. This strong dependence with the amplitude of the rf electric
field was confirmed in [15] where periodic boundary conditions are used to
study the translationally uniform ion cloud geometry that is a relevant
representation for very elongated cloud in 2D linear quadrupole traps. The
work detailed in [16, 17, 18] goes beyond the microscopic description and
proposes a universal heating formula to capture the dependence with the rf
electric field, ion number and temperature for spherical cloud in 3D-Paul
trap.

Even if most experimental signals rely on the laser induced fluorescence
involved in Doppler laser cooling, few works have combined the study of rf-
heating in the context of laser-cooling [19, 20]. As shown in this article,
when Doppler laser-cooling is added to this complex system, a bistable
behaviour can be observed, where the control parameter is the strength of the
rf trapping field and the switching parameter is the temperature of the
sample. This bistability has a strong impact on experiments involving large
samples like shown in the last part of this article. Here, we study this
bistability by computing the rf-heating of finite size ion ensembles by means
of molecular dynamics simulations, using the data generated during the free
evolution of an ion cloud confined in a linear rf-trap. Rf-heating rates are
then directly compared to laser cooling rates to demonstrate the onset of a
bistable behaviour for the temperature of the sample. Investigating the effect
of the ion number $N$ and the amplitude of the radio-frequency field we
demonstrate that only the latter has a significant effect on rf heating rate,
pointing to the cloud density as the controlling parameter. These results are
then used to build and justify experimental scenarii based on the control of
the switch between the two stable states.

##  II Molecular dynamic simulations

###  II.1 Model

The ion ensemble is confined in a linear quadrupole rf trap (inner radius
$r_{0}$=2.5 mm) where each diagonal pair of rods, aligned with the $Oz$ axis,
is supplied with time oscillating voltage (frequency $\Omega/2\pi=2$ MHz) with
opposite phase and a common amplitude $U_{RF}$ (for more details, see [20]).
The potential generated in the plane perpendicular to the trap axis is

$\Phi(x,y,t)=\frac{U_{RF}\cos(\Omega t)}{r_{0}^{2}}\left(x^{2}-y^{2}\right).$
(1)

A static voltage $U_{DC}$ is applied to extra electrodes to generate a
quadratic potential $\Phi(z)$ along the $Oz$ axis, characterised by the
frequency $\omega_{z}/2\pi$ scaling as
$\omega_{z}/2\pi=100\times\sqrt{U_{DC}}$ kHz for trapped Ca+ ions, of mass
$m=40$ a.m.u. and charge $Q=+q_{e}=+1.602\cdot 10^{-19}$ C. The equations
governing the ion dynamics read

$\displaystyle\ddot{u}_{i}+\left(-\frac{\omega_{z}^{2}}{2}\pm\frac{2QU_{RF}\cos{\Omega
t}}{mr_{0}^{2}}\right)u_{i}$
$\displaystyle\quad\quad\quad\quad=\frac{Q^{2}}{4\pi\epsilon_{0}m}\sum_{j=1,j\neq
i}^{N}{\frac{u_{i}-u_{j}}{|\bm{r}_{i}-\bm{r}_{j}|^{3}}}$ (2)
$\displaystyle\ddot{z}_{i}+\omega_{z}^{2}z_{i}=\frac{Q^{2}}{4\pi\epsilon_{0}m}\sum_{j=1,j\neq
i}^{N}{\frac{z_{i}-z_{j}}{|\bm{r}_{i}-\bm{r}_{j}|^{3}}}$ (3)

for an ion $i$, where $u_{i}$ stands for $x_{i}$ or $y_{i}$ and
$\bm{r}_{i}=(x_{i},y_{i},z_{i})$. Those that govern the motion in the radial
plane $(Ox,Oy)$ can be recast into the standard Mathieu equations [21] by
defining the usual $q_{x}$ Mathieu parameter giving the dimensionless rf-
traping electric field and that we choose positive
$q_{x}=(4QU_{RF})/(mr_{0}^{2}\Omega^{2})$ and the effective Mathieu parameters
$a_{x}=a_{y}$ defined as $a_{x}=-2\omega_{z}^{2}/\Omega^{2}$ [22, 23].

The equivalent static pseudo-potential [24] is defined by the fundamental
oscillation frequencies for a single ion in the radial direction $\omega_{r}$
and in the axial direction $\omega_{z}$. By changing the strength of the rf
trapping field, characterised by the $q_{x}$ parameter, two characteristics of
the ion clouds are changed. They are the mean density, which scales like
$q_{x}^{2}$ [25, 26] and the geometrical aspect ratio, which depends on the
potential aspect ratio $\omega_{z}^{2}/\omega_{r}^{2}$. To have only one
characteristic changing along the simulations, the ratio
$\omega_{z}^{2}/\omega_{r}^{2}$ is chosen equal to 1 for all the tested
Mathieu parameter $q_{x}$, which implies a spherical shape to the ion cloud
[26]. To achieve this goal, we compute the continuous fraction [21, 27]
defining the stability parameter $\beta_{x}(a_{x},q_{x})=2\omega_{r}/\Omega$
recursively to impose $\beta_{x}^{2}(a_{x},q_{x})=-2a_{x}$.

###  II.2 Temperature

As the rf-driven motion does not contribute to the temperature, it must not be
taken into account in its definition [28, 19]. To that purpose, the velocity
${\bm{v}_{i}}(t)$ of each ion $i$ is averaged over one radio-frequency period,
to smear out the rf-driven oscillation. The temperature $T$ is dependant on
this time-averaged velocities $\overline{\bm{v}_{i}}(t)$ like

$\frac{3}{2}k_{B}T=\frac{1}{2}\frac{m}{N}\sum_{i=1}^{N}\overline{\bm{v}_{i}}(t)^{2}$
(4)

with $k_{B}$ the Boltzmann’s constant. Fig. 1 shows the time evolution for
this temperature for a sample made of 1024 ions, following a thermalisation
process that takes the sample to 10 mK [20]. After few milliseconds of free
evolution, we observe a sudden increase of the temperature by two orders of
magnitude. We have checked that the velocity distribution remains in a very
good agreement with a Maxwell-Boltzmann distribution along this evolution
(Fig. 1).

Figure 1: Temperature evolution of a 1024 ion ensemble, in a trap defined by
$q_{x}=0.64$ and $\omega_{z}/2\pi=388$ kHz. The temperature is computed
following Eq. (4). The two insets show the velocity distribution of a 1024 ion
ensemble at 13.4 mK and 8.10 K as histogram. The red curve is the computed
Maxwell-Boltzmann distribution at this given temperature.

##  III Radio-frequency heating and laser cooling

###  III.1 Heating rate

To identify the general characteristics of these sudden temperature rise, the
heating rate $H=\mathrm{d}T/\mathrm{d}t$ is calculated based on a time
averaged temperature, to smooth the fluctuations induced by the small size of
the sample. The averaging time is chosen at 100 $\mu$s which correspond to
$n_{t}=200$ radio-frequency periods $\tau_{RF}$.

Fig. 2 shows these heating rates depending on the temperature computed for a
cloud of $N=1024$ ions, trapped by five different trapping field amplitude,
characterised by Mathieu parameters $q_{x}=\\{0.2,0.3,0.4,0.5,0.6\\}$.
Typically, a heating rate curve represented in log-log exhibits the following
behaviour : at low temperature the heating rate log first increases linearly
with the temperature log, going from below $10^{-6}$ K/$\tau_{RF}$ to the
range of $10^{-2}$ K/$\tau_{RF}$ for a temperature in the range 0.01-0.1 K.
This is the signature of a polynomial dependence of the heating rate with the
temperature, with an exponent increasing with $q_{x}$. Then the heating rate
remains stationary within the range $10^{-2}-10^{-1}$ K/$\tau_{RF}$ up to $10$
K and then decreases smoothly down to 400 K where simulations are stopped.

These curves are in accordance with the $\Lambda$-shape scheme proposed in
[29] to explain the equilibrium temperature of a trapped ion cloud. For
$q_{x}=0.4$, we find heating rates as a function of temperature in accordance
with the results of [15], computed for $q_{x}=0.44$. However, we could not
confirm that heating rate behaves universally independently of the Mathieu
parameter $q_{x}$, as it was suggested in Ref. [16].

###  III.2 Laser-cooling

To characterize the equilibrium thermal state of a laser-cooled sample, we now
compare the rf-heating rate to the Doppler laser cooling rate $G$. Doppler
laser-cooling allows to cool down ions thanks to the resonant radiation
pressure induced by the recoil when ions absorb photons [30, 31]. The laser
cooling of Ca+ ions involves two transitions and three levels but we consider
here the simplified model of a two-level system, keeping only the resonant
dipole transition at $\lambda_{L}=397$ nm to the first excited state with
lifetime $\tau_{e}=6.9$ ns [32]. This simplification over-estimate the optimum
probability $P_{e}$ for an ion to be in the excited state by a factor close to
2 [33] but it will not change the conclusion of the comparison. For a laser
beam with wave-vector $\bm{k}_{L}=2\pi/\lambda_{L}\bm{z}$, propagating along
the symmetry axis $Oz$ of the trap, the scattering force is [34] $F_{s}=\Gamma
P_{e}\hbar{\bm{k}}_{L}$ with $\Gamma=1/\tau_{e}$. The probability $P_{e}$
depends on the Rabi frequency $\Omega_{R}$ of the laser-dipole coupling,
relative to the natural spontaneous emission rate $\Gamma$, by the on-
resonance saturation parameter $s=2\Omega_{R}^{2}/\Gamma^{2}$. It depends also
on the laser frequency $\omega_{L}$ by its detuning $\delta$ from the atomic
transition frequency $\omega_{0}$ as

$P_{e}=\frac{1}{2}\frac{s}{1+s+4\delta^{2}/\Gamma^{2}}.$ (5)

For a moving atom, the detuning must include the Doppler effect and
$\delta=\delta_{L}-\bm{k}_{L}\cdot\bm{v}$, with
$\delta_{L}=\omega_{L}-\omega_{0}$. The work rate (or power) of the radiation
pressure force on an ion with velocity $\bm{v}$ can be calculated as
$\bm{F_{s}}\cdot\bm{v}$, assuming that the excitation probability $P_{e}$ has
reached the stationary limit for a given velocity. This assumption is
justified because the transition sets in the broadband limit where $\Gamma\gg
k_{L}v_{r}$ where $v_{r}=\hbar k_{L}/m$ is the recoil velocity due to the
absorption of one photon [35]. This work rate, averaged over the ion velocity
distribution $P(\bm{v},T)$ of the sample, defines the cooling rate that we
choose positive and express in temperature variation per unit time:

$G=-\frac{1}{k_{B}}\int\left(\bm{F_{s}}\cdot\bm{v}\right)\
P(\bm{v},T)~{}\mathrm{d}\bm{v}.$ (6)

For the calculation shown on Fig. 2, a Maxwell-Boltzmann distribution is
assumed and the cooling rate is calculated for each temperature, for a
saturation parameter $s=2$ and laser detunings $\delta_{L}=-\Gamma$ and
$\delta_{L}=-10\Gamma$. To be complete, the heating and cooling rate
comparison must include the heating rate $H_{e}$ induced by the spontaneous
emission involved in the laser cooling. This can be calculated as [36]

$H_{e}=\frac{\hbar^{2}k_{L}^{2}}{mk_{B}}\int\Gamma
P_{e}P(\bm{v},T)~{}\mathrm{d}\bm{v}.$ (7)

in temperature variation per unit time. The crossing between $H_{e}$ and $G$
gives the low temperature equilibrium when there is no rf-heating. Depending
on the chosen laser parameters, it is between 0.5 mK (the Doppler laser limit)
and 5 mK.

Figure 2: Log-log representation of $\mathrm{d}T/\mathrm{d}t$ vs temperature
in a time-scale in units of the rf period $\tau_{RF}$. Coloured symbols : free
evolution $H$ of an ensemble of 1024 ions in an rf trapping potential
characterised by $q_{x}=0.2,0.3,0.4,0.5,0.6$ (see the colour chart, the dotted
lines are guide to the eyes). Red lines : cooling rates $G$ as defined by Eq.
(6) for different laser detuning : solid line : $\delta_{L}=-\Gamma$, dashed
line : $\delta_{L}=-10\Gamma$. Black line : spontaneous emission induced
heating $H_{e}$ as defined by Eq. (7) for $\delta_{L}=-\Gamma$.

###  III.3 Competition between rf heating and laser cooling

The comparison between the rf-heating $H$ and laser-cooling rate $G$ (see Fig.
2) shows that, except for the smallest value of the Mathieu parameter $q_{x}$,
these two curves cross in the low temperature regime ($T<1$ K) as well as in
the high temperature regime ($T\simeq 100$ K), the rf-heating being dominant
between these two borders. We can then define two possible equilibrium
temperatures corresponding to $G=H$, responsible for the bistable behavior of
these systems. For initial conditions such that $H>G$, the ensemble converges
towards the hotter equilibrium, on the contrary, if $G>H$ the cloud converges
towards the colder equilibrium. For the low temperature range the cold
temperature equilibrium is then defined by $G=H_{e}$. For the largest $q_{x}$
values, once the low temperature unstable equilibrium $G=H$ is crossed, $H$
reaches its maximum value in a time of the order of magnitude of the averaging
window (100 $\mu$s), which is small with respect to the time elapsed since the
beginning of the evolution of the ion cloud at low temperature, hence this
phenomena usually quoted as an ”explosive onset”. For increasing $q_{x}$, the
limit temperature for which rf-heating dominates laser-cooling decreases, from
1 K for $q_{x}=0.3$ to $40(\pm 10)$ mK for $q_{x}=0.6$. This illustrates how
the trapping parameter $q_{x}$ controls the switching conditions between a low
and a high temperature regime.

For the chosen ion number $N=1024$ and laser parameters, laser-cooling always
overcome rf-heating for $q_{x}=0.2$ except maybe for temperature of the order
of 100 K where the numerical calculations show large fluctuations and are not
very relevant. When maximum, the rf-heating rate is two to three orders of
magnitude larger than spontaneous emission heating rate. It is also the case
for larger $q_{x}$ values when rf-heating overcome laser-cooling. This
hierarchy tells that spontaneous emission heating can be neglected when the
competition between the two other effects is considered and confirms the role
of $q_{x}$ parameter as the controlling figure for rf-heating.

The effect of the number of ions on the general trends detailed above is
analysed on Fig. 3 where the rf-heating rate is plotted for $N=256,512,1024$
ions and $q_{x}=0.6$. It shows that the behaviour identified in Fig. 2 is very
general for spherical shape of ion ensemble as it does not depend
significantly with the ion number.

Furthermore, the comparison between $H$ and $G$ for this high $q_{x}$ regime,
as shown in Fig. 2 and 3, shows that in the intermediate temperature regime
where $H>G$, the rf-heating rate is typically one order of magnitude larger
than the laser-cooling rate. This justifies in which conditions the impact of
laser-cooling on the cloud dynamics can be neglected in numerical simulations
as well as in experiments.

Figure 3: Log-log representation of $\mathrm{d}T/\mathrm{d}t$ vs temperature
in a time-scale in units of the rf period $\tau_{RF}$. Colored symbols : free
evolution of $N$ ions and a rf trapping potential characterised by
$q_{x}=0.6$. Blue triangles : $N=256$, Orange diamonds : $N=512$, Green
crosses : $N=1024$ (the dotted lines are guide to the eyes). Red solid lines :
cooling rate $G$ as defined by Eq. (6) for detuning $\delta_{L}=-\Gamma$.
Black line : spontaneous emission induced heating $H_{e}$ as defined by Eq.
(7) for $\delta_{L}=-\Gamma$.

###  III.4 Effect of the density

For this article, $q_{x}$ and $N$ were varied by factor, respectively, 3 and 4
and the comparison of Fig. 2 and 3 shows that increasing $q_{x}$ has a
stronger impact on the rf heating rate than increasing $N$. For $N=1024$ ions,
the radial size of the ion ensemble decreases from $R=190~{}\mu$m for
$q_{x}=0.6$ to $R=88~{}\mu$m for $q_{x}=0.2$. For a given trapping condition
$q_{x}$, this radial size scales like $N^{1/3}$ because we chose to keep the
cloud spherical. Studying the dependency of the maximum rf electric field
reached by the ions as a function of $q_{x}$ and $N$ allows to show that this
field is not the controlling parameter for the heating rate.

Starting from Eq. (1), the electric field amplitude $||\bm{E}(r)||$
controlling the motion of an ion at distance $r$ to the trap’s central axis,
scales like $q_{x}r$. In the crystal and liquid phase ($T\leq 2$ K for the
considered densities) [37], the density can be considered as homogenous out of
a boundary layer of the order of few $\mu$m [25, 38]. The mean density $n_{c}$
scaling with $q_{x}^{2}$ [25, 26] and the ion ensemble being spherical,
$NR^{-3}\propto q_{x}^{2}$. Thus, the maximum rf field amplitude for an ion
cloud scales like

$\displaystyle||\bm{E}(R)||$ $\displaystyle\propto q_{x}^{1/3}N^{1/3}$ (8)

Our results demonstrate how $q_{x}$ has a much more significant effect than
$N$ on the rf heating rate. Nevertheless, Eq. (8) emphasizes how $q_{x}$ and
$N$ play a similar role in the maximum electric field reached by the ions.
Thus, it is clear that the parameter controlling the rf heating rate is not
$||\bm{E}(R)||$. To the contrary, considering that rf heating is induced by
ion-ion collisions, the strong impact of $q_{x}$ could be attributed to the
change in the mean density which lead to an ion-ion mean distance scaling like
$q_{x}^{-2/3}$ and is thus reduced by a factor 2.1 by going from $q_{x}=0.2$
to $q_{x}=0.6$.

##  IV Insight into different experimental situations

In several situations, the collected ion fluorescence is the only non-
destructive signal that can be collected from the ion sample and depending on
the laser detuning with the atomic transition, this laser-ion coupling induces
Doppler laser-cooling [34].

For a constant laser detuning, these systems show a bi-stable behaviour for
the temperature, triggered by the Mathieu parameter $q_{x}$, which catch the
strength of the rf-trapping field. If the $q_{x}$ parameter is kept constant,
this strong non-linear behaviour shows that a small perturbation in the
temperature can be made sufficient to trigger a switch from the low to the
high temperature equilibrium. This perturbation amplification process is at
the core of a detection protocol that was proposed to detect giant molecular
ions [39, 20]. While the temperature increase induced by the energy lost by
the projectile $\Delta E$ is too small to be detected by current laser-induced
fluorescence-counting techniques, it can bring the trapped ion cloud to a
temperature where the rf-heating rate is orders of magnitude larger,
triggering the switch to the high temperature state. In this thermal state,
the Doppler effect results in the reduction of the laser induced fluorescence
rate that is detectable and a saturation parameter $s=2$ and detuning
$\delta_{L}=-\Gamma$ were identified as the best compromise for a high
sensitivity of the fluorescence signal to the cloud thermal state [20]. In
practice, the energy transfer must bring the temperature above the explosive
onset threshold where $G=H$. This temperature threshold decreases with
increasing $q_{x}$ and was found to be 20 mK for $q_{x}=0.68$, 40 mK for
$q_{x}=0.6$, 100 mK for $q_{x}=0.5$ and 250 mK for $q_{x}=0.4$. This analysis
justifies the need for a large Mathieu parameter for a very sensitive
detector, and the temperature threshold determined according to the method
developed in this article are compatible with the previous interaction
simulations carried out for $q_{x}>0.5$ [20]. Considering the initial
temperature of the ensemble negligible compared to the temperature threshold,
the smallest $q_{x}$ value for which a detection is observed can produce a
measurement of the temperature increase $\Delta T$ by comparison with
numerical data like the one of Fig. 2. As already discussed, the threshold
temperature does not depend on the cloud size for typical sizes of hundreds of
ions. In a stationary description of the system where we can assume that the
ensemble thermalises after the perturbation by the projectile, we can write
$\Delta E=Nk_{B}\Delta T$ and by measuring the number of ions in the cloud
[40] the energy lost $\Delta E$ can be inferred. This control would turn this
giant molecular detector into a device able to measure the stopping power of a
strongly correlated non-neutral plasma [41, 42]. In the small Mathieu
parameter regime, the conditions for an intruder to be detected in the
observed fluorescence light, assuming no rf-heating is amplifying the
perturbation, are analysed in [43].

Once a detection has been effective, the ion cloud has reached a temperature
of the order of 100 K and needs to be cooled down to a temperature lower than
the chosen threshold for the next detection, which is lower than 100 mK. For
this preparation stage, the bi-stability analysis is also very useful. As
shown in Fig. 2, for a very detuned cooling laser ($\delta_{L}=-10\Gamma$),
the laser-cooling $G$ overcomes rf-heating $H$ for large temperature : above 4
K for $q_{x}=0.2$ and 20 K for $q_{x}=0.4$. It is then a relevant detuning for
cooling ensembles above 100 K but in order to reach the low temperature
equilibrium, decreasing the detuning alone is insufficient and the Mathieu
parameter must be also decreased to reach $q_{x}<0.3$. With such a low Mathieu
parameter, there is always a detuning $\delta_{L}$ such that $G>H$. It is then
possible to bring the ion ensemble to the low equilibrium temperature by
gradually changing the laser detuning from a high $|\delta_{L}|$ to low
$|\delta_{L}|$ value. Then, $q_{x}$ can be increased up to reach the required
detection sensitivity, as long as $G>H$ is verified at the given temperature.

##  V Conclusion

This article demonstrates the bistable behaviour of a laser-cooled 3D large
ion cloud, trapped in a linear rf-trap. The comparison between different
trapping conditions and cloud sizes allows to conclude that the characteristic
figure controlling the rf heating rate is the cloud density. This analysis
allows to build a protocol to keep cold ion cloud in a high trapping field
regime and to evaluate the maximum perturbation that a cloud can absorb
without switching to a high temperature state. Conversely, the possibility to
trigger the switch from a low to a high temperature regime is analysed in the
scope of a detector for giant molecule based on the modification of the cloud
fluorescence rate. This detector will take advantage of the amplifying effect
induced by the temperature switch, to be sensitive to the crossing of the ion
cloud by a single projectile.

## Acknowledgments

The authors thank David Wilkowski for very stimulating discussions which
opened new perspectives for the results presented here. This work is
financially supported by CNRS-Innovation (project MegaDalton).

## References

  * Wineland  _et al._ [1987] D. J. Wineland, W. M. Itano, J. C. Bergquist, and R. G. Hulet, Phys. Rev. A 36, 2220 (1987).
  * Prestage  _et al._ [1991a] J. Prestage, R. Tjoelker, G. Dick, and L. Maleki, Proceedings of the 45th Annual Symposium on Frequency Control 1991 , pp 572 (1991a).
  * Willitsch  _et al._ [2008] S. Willitsch, M. T. Bell, A. D. Gingell, and T. P. Softley, Phys. Chem. Chem. Phys. 10, 7200 (2008).
  * Alighanbari  _et al._ [2018] S. Alighanbari, M. G. Hansen, V. I. Korobov, and S. Schiller, Nature Physics 14, 555 (2018).
  * Alighanbari  _et al._ [2020] S. Alighanbari, G. Giri, F. Constantin, V. Korobov, and S. Schiller, Nature 581, 1 (2020).
  * Yang  _et al._ [2021] T. Yang, A. Li, G. K. Chen, Q. Yao, A. G. Suits, H. Guo, E. R. Hudson, and W. C. Campbell, Science Advances 7, eabe4080 (2021), https://www.science.org/doi/pdf/10.1126/sciadv.abe4080 .
  * Dantan  _et al._ [2009] A. Dantan, M. Albert, J. P. Marler, P. F. Herskind, and M. Drewsen, Phys. Rev. A 80, 041802 (2009).
  * Dantan  _et al._ [2010] A. Dantan, J. P. Marler, M. Albert, D. Guénot, and M. Drewsen, Phys. Rev. Lett. 105, 103001 (2010).
  * Lauprêtre  _et al._ [2019] T. Lauprêtre, R. B. Linnet, I. D. Leroux, H. Landa, A. Dantan, and M. Drewsen, Phys. Rev. A 99, 031401 (2019).
  * Prestage  _et al._ [1989] J. Prestage, G. Dick, and L. Maleki, J. Appl. Phys. 66, 1013 (1989).
  * Berkeland  _et al._ [1998] D. Berkeland, J. Miller, J. Bergquist, W. Itano, and D. Wineland, J. Appl. Phys. 83, 5025 (1998).
  * Prestage  _et al._ [2006] J. Prestage, S. Chung, T. Le, L. Lim, and L. Maleki, in  _Proceedings of IEEE Int. Freq. Contr. Symp. Miami, jun. 5-7, 2006_ (IEEE, New York, 2006).
  * Hoffnagle  _et al._ [1988] J. Hoffnagle, R. G. DeVoe, L. Reyna, and R. G. Brewer, Phys. Rev. Lett. 61, 255 (1988).
  * Prestage  _et al._ [1991b] J. D. Prestage, A. Williams, L. Maleki, M. J. Djomehri, and E. Harabetian, Phys. Rev. Lett. 66, 2964 (1991b).
  * Ryjkov  _et al._ [2005] V. L. Ryjkov, X. Zhao, and H. A. Schuessler, Phys. Rev. A 71, 033414 (2005).
  * Tarnas  _et al._ [2013] J. D. Tarnas, Y. S. Nam, and R. Blümel, Phys. Rev. A 88, 041401 (2013).
  * Nam  _et al._ [2014] Y. S. Nam, E. B. Jones, and R. Blümel, Phys. Rev. A 90, 013402 (2014).
  * Nam  _et al._ [2017] Y. Nam, D. Weiss, and R. Blümel, Physics Letters A 381, 3477 (2017).
  * Marciante  _et al._ [2010] M. Marciante, C. Champenois, A. Calisti, J. Pedregosa-Gutierrez, and M. Knoop, Phys. Rev. A 82, 033406 (2010).
  * Poindron  _et al._ [2021] A. Poindron, J. Pedregosa-Gutierrez, C. Jouvet, M. Knoop, and C. Champenois, The Journal of Chemical Physics 154, 184203 (2021), https://doi.org/10.1063/5.0046693 .
  * McLachlan [1947] N. McLachlan,  _Theory and Application of Mathieu Functions_ (Clarendon, Oxford, 1947).
  * Drewsen and Brøner [2000] M. Drewsen and A. Brøner, Phys. Rev. A 62, 045401 (2000).
  * Drakoudis  _et al._ [2006] A. Drakoudis, M. Söllner, and G. Werth, International Journal of Mass Spectrometry 252, 61 (2006).
  * Major and Dehmelt [1968] F. G. Major and H. G. Dehmelt, Phys. Rev. 170, 91 (1968).
  * Prasad and O’Neil [1979] S. Prasad and T. O’Neil, Phys. Fluids 22, 278 (1979).
  * Hornekær and Drewsen [2002] L. Hornekær and M. Drewsen, Physical Review A 66 (2002), 10.1103/PhysRevA.66.013412.
  * Major  _et al._ [2005] F. G. Major, V. N. Gheorghe, and G. Werth, in  _Charged Particle Traps : Physics and Techniques of Charged Particle Field Confinement_, edited by F. G. Major, V. N. Gheorghe, and G. Werth (Springer, Berlin, Heidelberg, 2005) Chap. 2, pp. 17–49.
  * Schiffer  _et al._ [2000] J. P. Schiffer, M. Drewsen, J. S. Hangst, and L. Hornekær, Proceedings of the National Academy of Sciences 97, 10697 (2000), https://www.pnas.org/content/97/20/10697.full.pdf .
  * Blümel  _et al._ [1989] R. Blümel, C. Kappler, W. Quint, and H. Walther, Phys. Rev. A 40, 808 (1989).
  * Hänsch and Schawlow [1975] T. Hänsch and A. Schawlow, Optics Comm. 13, 68 (1975).
  * Wineland  _et al._ [1978] D. J. Wineland, R. E. Drullinger, and F. L. Walls, Phys. Rev. Lett. 40, 1639 (1978).
  * Hettrich  _et al._ [2015] M. Hettrich, T. Ruster, H. Kaufmann, C. F. Roos, C. T. Schmiegelow, F. Schmidt-Kaler, and U. G. Poschinger, Phys. Rev. Lett. 115, 143003 (2015).
  * Lisowski  _et al._ [2005] C. Lisowski, M. Knoop, C. Champenois, G. Hagel, M. Vedel, and F. Vedel, Appl. Phys. B 81, 5 (2005).
  * Metcalf and van der Straten [1999] H. Metcalf and P. van der Straten,  _Laser cooling and trapping_ (Springer, 1999).
  * Champenois [2016] C. Champenois, “Trapped charged particles, laser cooling techniques applicable to trapped ions,” (World Scientific, 2016) p. 117.
  * Lett  _et al._ [1989] P. Lett, W. Phillips, S. Rolston, C. Tanner, R. Watts, and C.I.Westbrook, J. Opt. Soc. Am. B 6, 2084 (1989).
  * Schiffer [2002] J. P. Schiffer, Phys. Rev. Lett. 88, 205003 (2002).
  * Champenois [2009] C. Champenois, J. Phys. B 42, 154002 (2009).
  * Champenois  _et al._ [2014] C. Champenois, C. Dedonder-Lardeux, C. Jouvet, L. Hilico, M. Knoop, and J. Pedregosa-Guttierez,  _Non-destructive detection method of charged particles without mass limitation_ , Tech. Rep. (Brevet européen, 2014).
  * Kamsap  _et al._ [2015] M. R. Kamsap, J. Pedregosa-Gutierrez, C. Champenois, D. Guyomarc’h, M. Houssin, and M. Knoop, Phys. Rev. A 92, 043416 (2015).
  * Zwicknagel  _et al._ [1999] G. Zwicknagel, C. Toepffer, and P.-G. Reinhard, Physics Reports 309, 117 (1999).
  * Bussmann  _et al._ [2006] M. Bussmann, U. Schramm, D. Habs, V. Kolhinen, and J. Szerypo, International Journal of Mass Spectrometry 251, 179 (2006), uLTRA-ACCURATE MASS SPECTROMETRY AND RELATED TOPICS Dedicated to H.-J. Kluge on the occasion of his 65th birthday anniversary.
  * Gajewski  _et al._ [2022] M. Gajewski, W. Li, S. Wolf, W. Hahn, C. E. Düllmann, D. Budker, G. Morigi, and F. Schmidt-Kaler, Phys. Rev. A 106, 033108 (2022).