# Deep learning techniques for blind image super-resolution: A high-scale
multi-domain perspective evaluation

Valdivino Alexandre de Santiago Júnior
Coordenação de Pesquisa Aplicada e Desenvolvimento Tecnológico (COPDT)
Instituto Nacional de Pesquisas Espaciais (INPE)
São José dos Campos, São Paulo, Brazil
<EMAIL_ADDRESS>

###### Abstract

Despite several solutions and experiments have been conducted recently
addressing image super-resolution (SR), boosted by deep learning (DL)
techniques, they do not usually design evaluations with high scaling factors,
capping it at 2x or 4x. Moreover, the datasets are generally benchmarks which
do not truly encompass significant diversity of domains to proper evaluate the
techniques. It is also interesting to remark that blind SR is attractive for
real-world scenarios since it is based on the idea that the degradation
process is unknown, and hence techniques in this context rely basically on
low-resolution (LR) images. In this article, we present a high-scale (8x)
controlled experiment which evaluates five recent DL techniques tailored for
blind image SR: Adaptive Pseudo Augmentation (APA), Blind Image SR with
Spatially Variant Degradations (BlindSR), Deep Alternating Network (DAN),
FastGAN, and Mixture of Experts Super-Resolution (MoESR). We consider 14 small
datasets from five different broader domains which are: aerial, fauna, flora,
medical, and satellite. Another distinctive characteristic of our evaluation
is that some of the DL approaches were designed for single-image SR but others
not. Two no-reference metrics were selected, being the classical natural image
quality evaluator (NIQE) and the recent transformer-based multi-dimension
attention network for no-reference image quality assessment (MANIQA) score, to
assess the techniques. Overall, MoESR can be regarded as the best solution
although the perceptual quality of the created HR images of all the techniques
still needs to improve. We believe that independent and unbiased evaluations,
like this one, are important to indicate the most suitable approaches to be
selected by professionals who need to work with image SR in their practical
settings. Supporting code: https://github.com/vsantjr/DL_BlindSR. Datasets:
https://www.kaggle.com/datasets/valdivinosantiago/dl-blindsr-datasets.

_K_ eywords Image super-resolution $\cdot$ Artificial intelligence $\cdot$
Deep learning $\cdot$ Controlled experiment $\cdot$ Multiple domains

##  1 Introduction

Medical imaging [1, 2, 3, 4], internet video delivery [5, 6, 7], surveillance
and security via person identification [8, 9, 10], and remote sensing [11, 12,
13, 14, 15, 16] are just some examples of real-world applications where image
super-resolution (SR) has been used. In SR, we aim at recovering high-
resolution111In this article, image resolution means precisely the
dimensionality of the image. For instance, an image has a resolution of
$W\times H$ pixels. This is not to be confused with other definitions of
resolution existing in certain communities such as remote sensing (spatial
resolution, radiometric resolution, temporal resolution, …). (HR) images from
low-resolution (LR) ones. This is a non-trivial and generally ill-posed
problem since multiple HR images exist corresponding to a unique LR image
[17].

Several classical methods have been designed for image SR such as bicubic
interpolation and Lanczos resampling [18], edge-based methods [19],
statistical methods [20], among others. But, naturally, with all the
developments related to deep learning (DL) and deep neural networks (DNNs)
[21], a significant number of strategies have been proposed addressing SR via
DL/DNNs as reported in recent secondary studies [17, 11]. Among the DL
techniques, convolutional neural networks (CNNs) [22, 23, 24, 13], generative
adversarial networks (GAN) [25, 26, 12, 14], and attention-based networks [27,
28, 15] have been employed to solve image SR problems.

The majority of studies published up to now focus on supervised SR where
models are trained with both LR images and the corresponding HR ones [17, 11].
But, in reality, it is not easy to have images from the same scene but with
distinct resolutions so that having the pairs LR-HR for training is not as
direct as it is supposed to be. Hence, in unsupervised SR [29], only unpaired
LR-HR images are available for training, so that the models are more able to
learn real-world LR-HR mappings.

But an even more attractive strategy for real-world scenarios is blind image
SR [30, 31] which is based on the idea that the degradation process/kernels
is/are unknown, and hence techniques in this context rely basically on LR
images, not requiring the high-resolution ones. There is an increasing
interest in blind image SR.

Despite the huge number of proposed techniques and experiments boosted by DL
techniques [17, 11], they do not usually design evaluations with high scaling
factors, capping it at 2x or 4x. One of the exceptions is the experiment
presented in [11] where authors considered 2x, 4x, and 8x scaling factors.
However, the authors considered a multi-sensor remote sensing dataset (MSRSD)
consisting of mostly publicly available very HR satellite images. Even if the
images are from different satellites and regions, eventually the diversity of
the images and feature spaces are not enough to proper evaluate SR techniques.

Several other datasets have been used for training image SR approaches such as
BSDS300 [32], BSDS500 [33], DIV2K [34], PIRM [35], Set5 [36], Set14 [37], and
Urban100 [38]. These datasets comprise different categories but still lack of
images of other domains. For instance, all the datasets above do not seem to
present images obtained via satellite sensors or medical images. Interesting
to stress that images taken by sensors embedded in satellites or airplanes
have considerable differences compared to natural images due to different
shooting content and shooting methods [39]. Moreover, as previously remarked,
evaluations are usually limited to the 4x scaling factor. In [40], results are
presented for 8x scaling factor but considering only the Set5 and Set14
datasets. We believe that performing experiments addressing not only high
scaling factors (e.g. 8x) but also datasets of different domains is very
important to better identify the most adequate image SR approaches.

Image quality assessment (IQA) metrics (methods) are roughly divided into two
categories: full-reference (FR-IQA) and no-reference (NR-IQA) [41]. In FR-IQA,
we evaluate the similarity between a distorted image and a given reference
image, and classical metrics which have been extensively used are peak signal-
to-noise ratio (PSNR) [11] and structural similarity index measure (SSIM)
[42]. On the other hand, NR-IQA metrics are proposed to assess image quality
without a reference image, being more suitable for perceptual quality. Natural
image quality evaluator (NIQE) [43] and perception index (PI) [35] are two
traditional metrics. However, recent metrics can be devised based on DNNs as
presented in the New Trends in Image Restoration and Enhancement (NTIRE)
workshop held at CVPR 2022 [41]. And, for the first time, NTIRE 2022 Challenge
on Perceptual Image Quality Assessment had a track addressing NR-IQA. We
believe that for evaluating approaches in a completely “blind” setting, NR-IQA
metrics are more interesting since they do not demand a reference image, only
the LR ones.

In this article, we present a high-scale (8x) controlled experiment which
evaluates five recent DL techniques tailored for blind image SR: Adaptive
Pseudo Augmentation (APA) [44], Blind Image SR with Spatially Variant
Degradations (BlindSR) [45], Deep Alternating Network (DAN) [46], FastGAN
[47], and Mixture of Experts Super-Resolution (MoESR) [48]. Relying basically
on public sources, we adapt and create 14 small (100 samples) LR images
datasets from five different broader domains: aerial, fauna, flora, medical,
and satellite222The aerial broader domain means data/images obtained at
altitudes lower than 100 km (62 miles; Kármán Line) above the mean sea level.
For example, images obtained by sensors attached/embedded in airplanes or
unmanned aerial vehicles (drones). The satellite broader domain relates to the
space and it implies data/images obtained from sensors, usually onboard
satellites, which are at a minimum altitude of 100 km above the mean sea
level.. Another distinctive characteristic of our evaluation is that some of
the DL approaches were designed for single-image SR but others not.

Two NR-IQA metrics were selected, being the classical NIQE and the recent
vision transformer(ViT)-based multi-dimension attention network for no-
reference image quality assessment (MANIQA) score [49], to assess the
techniques. The MANIQA model was the winner of the NTIRE 2022’s NR-IQA track
obtaining a performance considerably higher than classical strategies such as
PI and NIQE.

The contributions of this study are:

  1. 1.

We design and execute a controlled experiment considering five recent blind
image SR approaches and focusing on a specific large scaling factor (8x). Note
that we decided to present a more detailed analysis of the results, trying to
explain in more depth the behaviours of the approaches. We also perform a
correlation analysis taking into account the HR images produced by the two
best DL techniques. We believe that independent and unbiased evaluations are
important to indicate the most suitable approaches to be selected by
professionals who need to work with image SR in their practical settings;

  2. 2.

We adapt and create 14 small (100 samples) LR images datasets from five
different broader domains: aerial, fauna, flora, medical, and satellite. We
believe that making available these small datasets to the community, obtained
from quite distinct domains, is interesting to provide other possibilities to
evaluate the blind image SR techniques considering not only single-image but
non-single-image techniques (we had indeed done this);

  3. 3.

We consider a recent DNN-based NR-IQA metric (MANIQA score) in addition to a
classical one (NIQE), and present some remarks by using such metrics.

This article is structured as follows. Section 2 briefly presents the
theoretical background and related work. In Section 3, we show in detail the
design of our experiment. Results are in Section 4 while Section 5 discusses
some important points. In Section 6, conclusions and feature directions are
pointed out.

##  2 Background

This section presents an overview of the theoretical background related to
image SR. More details can be seen elsewhere [17, 11]. At the end, we also
discuss about related work. The goal of image SR is to recover the
corresponding HR images from the LR images. The LR image, $I_{LR}$, is usually
modelled as the output of the following degradation:

$I_{LR}=\Gamma(I_{HR};\delta)$ (1)

where $\Gamma$ means a degradation mapping function, $I_{HR}$ is the
corresponding HR image, and $\delta$ represents the parameters of the
degradation process. When the degradation process ($\Gamma$ and $\delta$) is
unknown, which is very common to happen, and only LR images are available for
the techniques, we have the so called blind image SR. In this case, the idea
is to recover an HR approximation, $\widehat{I_{HR}}$, of the ground truth HR
image, $I_{HR}$, from the LR image as presented below:

$\widehat{I_{HR}}=\mathcal{M}(I_{LR};\theta)$ (2)

where $\mathcal{M}$ is the SR model and $\theta$ are the parameters of
$\mathcal{M}$.

Despite of the fact that the degradation process is usually unknown, several
studies model the degradation as follows:

$\Gamma(I_{HR};\delta)=(I_{HR})\downarrow s,\\{s\\}\subset\delta$ (3)

where $\downarrow$ is a downsampling operation and $s$ is a scaling factor.
But, some studies [50] do such a degradation modelling as presented below:

$\Gamma(I_{HR};\delta)=(I_{HR}\otimes\kappa)\downarrow
s+\varphi_{\xi},\\{\kappa,s,\xi\\}\subset\delta$ (4)

where $I_{HR}\otimes\kappa$ means the convolution between a blur kernel,
$\kappa$, and the HR image, $I_{HR}$, and $s$ is a scaling factor. Moreover,
$\varphi_{\xi}$ is some additive white Gaussian noise with standard deviation
$\xi$.

###  2.1 Metrics

As previously mentioned, there are several metrics for IQA, being full-
reference (FR-IQA) or without relying on a reference (NR-IQA). We present here
a brief discussion about the two NR-IQA metrics we have considered in this
research: NIQE [43] and MANIQA score [49].

The main motivation to develop NIQE is that, at the time it was proposed, NR-
IQA models required knowledge about anticipated distortions in the form of
training samples and corresponding human opinion scores. Thus, NIQE is a
completely blind approach which only uses measurable deviations from
statistical regularities observed in natural images, without training on
human-rated distorted images and, hence, without any exposure to distorted
images. The idea is to construct a “quality aware” collection of statistical
features based on a successful space domain natural scene statistic (NSS)
model.

NIQE is derived by computing 36 identical NSS features from patches of the
same size, $P\times P$, from the image to be analysed, fitting them with a
multivariate Gaussian model (MVG) [51], then comparing its MVG fit to the
natural MVG model. They considered a patch of dimension (resolution) $96\times
96$, although other dimensions were also evaluated. The quality of the
distorted image is expressed as the distance between the quality aware NSS
feature model and the MVG fit to the features extracted from the distorted
image, as shown below:

$\mathcal{D}(\nu_{1},\nu_{2},\Sigma_{1},\Sigma_{2})=\sqrt{\Bigg{(}(\nu_{1}-\nu_{2})^{T}\Bigg{(}\frac{\Sigma_{1}+\Sigma_{2}}{2}\Bigg{)}^{-1}(\nu_{1}-\nu_{2})\Bigg{)}}$
(5)

where $\nu_{1}$ and $\nu_{2}$ are the mean vectors of the natural MVG model
and the distorted image’s MVG model, respectively. Furthermore, $\Sigma_{1}$
and $\Sigma_{2}$ are the covariance matrices of the natural MVG model and the
distorted image’s MVG model, respectively. Thus, as lower the NIQE, the better
the perceptual quality of the generated HR image.

While NIQE is a very traditional metric, the MANIQA score is a very recent
one. Top approach of the NTIRE 2022’s NR-IQA track [41], the motivation for
the development of the MANIQA model and its corresponding score is that NR-IQA
metrics/methods are limited when assessing images created by GAN-based image
restoration algorithms [26, 52]. As shown in Fig. 1, the MANIQA model consists
of four components: feature extractor using ViT [53], transposed attention
block, scale swin transformer block, and a dual-branch structure for patch-
weighted quality prediction.

Figure 1: The MANIQA model. The score is the metric related to the image.
Source: adapted from [49]

In more detail, a distorted image is cropped into patches of dimension 8 x 8.
Then, the patches are inputted into the ViT for extracting the features.
Transposed attention block and scale swin transformer block are used to
strengthen the global and local interaction. A dual-branch structure is
proposed for predicting the weight and score of each patch. Notice that as
higher the MANIQA score, the better the perceptual quality of the generated HR
image.

###  2.2 Related work

This study is an experiment within blind image SR via DL/DNNs. Basically every
study within blind image SR presents experimental evaluations and, in lesser
or greater extent, they are related to ours. As mentioned in Section 1, even
if there are so many techniques and experiments proposed boosted by DL
techniques, as corroborated by secondary studies [17, 11], in general the
studies do not consider high scaling factors such as 8x. When there are
exceptions [11, 40], there is no significant diversity of images and feature
spaces to proper evaluate the image SR approaches, taking into account quite
distinct broader domains such as medicine images, images obtained by
satellites via sensors with different characteristics, and images more “usual”
like those of animal’s faces.

It is worth noting that dealing with this range of different domains is
important to better classify the techniques in terms of their performances. In
other words, the wider/general the assessments are, the better. Furthermore,
independent evaluations, like the one presented in this article, are valuable
to guide professionals in making the correct decisions in choosing the most
appropriate blind image SR solution. Our study addresses all these previous
points and these are the main differences between our effort and other already
published articles.

##  3 Experiment design

In this section, we describe the main design options of the controlled
experiment to assess the performance of the five DL techniques for blind image
SR: APA, BlindSR, DAN, FastGAN, and MoESR.

###  3.1 Research questions and variables

The research questions (RQs) related to this experiment are:

  1. 1.

RQ_1 \- Which out of the five algorithms for blind image SR is the best
regarding the metrics NIQE and MANIQA score? And which can be considered the
best overall?

  2. 2.

RQ_2 \- Does the two top approaches present similar behaviours when deriving
HR images?

The motivation for RQ_1 is self-explained. Regarding RQ_2, the idea here is to
perceive whether the two best approaches present similar behaviours. In other
words, are the images detected as having the best, as well as the worst,
quality perceptions, based on the MANIQA scores, somewhat “common” to both
best DL techniques? Our goal here is to see whether the best algorithms
“agree” in terms of the HR images they create based only on the LR ones. The
independent variables are the DL models. The dependent variables are the
values of the metrics, i.e. NIQE and MANIQA score.

###  3.2 Datasets

The 14 datasets comprise five different broader domains: aerial, fauna, flora,
medical, and satellite. Tables 1 and 2 present details about the datasets we
created based on publicly released images. In Table 2, note that the original
resolution of the images in the datasets are diverse. Some source datasets,
like condoaerial and massachbuildings, are formed by HR images where all
samples have the same resolution. The plantpat dataset has HR images too but
there are several different resolutions. Others like catsfaces and isaid have
HR, LR and even medium resolution images. Three Satellite datasets, amazonia1,
cbers4a, and deepglobe, are composed only by LR images and all have the same
resolution.

Table 1: Datasets: description and source Domain | Dataset |  Description |  Source
---|---|---|---
Aerial | condoaerial |  Aerial Semantic Segmentation Drone Dataset |  https://bit.ly/3YWXNYy
| massachbuildings |  Massachusetts Buildings Dataset |  https://bit.ly/41nS4fI
| ships |  Ship Detection from Aerial Images Dataset |  https://bit.ly/3ZaPEiK
| ufsm-flame |  Drone Images from UFSM and Flame Datasets |  https://bit.ly/3KxJzbZ https://bit.ly/3YUwA8B https://bit.ly/3ZhiSMZ
Fauna | catsfaces |  Cats Faces Dataset |  https://bit.ly/41gDrLv
| dogsfaces |  Dogs Faces Dataset |  https://bit.ly/41gDrLv
Flora | flowers |  102 Category Flower Dataset |  https://bit.ly/3EAxH5w
| plantpat |  Plant Pathology 2021 - FGVC8 - Dataset |  https://bit.ly/3EBCkMv
Medical | melanomaisic |  SIIM-ISIC Melanoma Classification Dataset |  https://bit.ly/3XUqELO https://bit.ly/3m0Az5b
| structretina |  Structured Analysis of the Retina Dataset |  https://bit.ly/3ZjfzVJ
Satellite | amazonia1 |  Cloudless Scene from Amazonia 1 Satellite Dataset |  https://bit.ly/3m3qKTP
| cbers4a |  Scene with Clouds from CBERS-4A Satellite Dataset |  https://bit.ly/3m3qKTP
| deepglobe |  Forest Aerial Images for Segmentation Dataset |  https://bit.ly/3Eu4Nnq
| isaid |  Instance Segmentation in Aerial Images Dataset |  https://bit.ly/3IpavYU

Given the different domains and image resolutions (in some cases, as shown in
Table 2, we just had at our disposal LR images), we therefore decided to
default the input LR images to a resolution of 128 x 128 pixels. Thus, we
downsampled the images which have higher resolution than that based on the
bicubic interpolation method as many others have been doing [17, 11]. Fig. 2
presents some LR images from the datasets we created.

Note that some criticise doing such a resizing because real-world cameras
actually accomplishes a series of operations (demosaicing, denoising, …) to
finally produce 8-bit RGB images [17]. Thus, RGB images have lost lots of
original signals becoming significant different from the original images taken
by the camera. Therefore, it would not be interesting to directly use the
manually downsampled RGB images by using, for instance, a bicubic
interpolation method for image SR.

However, we must emphasise that our main goal is to perform an experiment
using DL techniques for blind image SR, based on different domains, in order
to provide some sort of recommendation to professionals who need to choose a
technique that best fit their needs. Although the issues cited earlier exist,
they impact equally all the DL techniques and hence they do not compromise our
analysis.

In addition to standardising the resolution of the LR input images, we also
considered the same small number of images in the datasets: 100 samples. More
details about this point in the next section.

Table 2: Datasets: original resolution of the images Domain | Dataset |  Original Resolution ($W\times H$)
---|---|---
Aerial | condoaerial |  6000 x 4000
| massachbuildings |  1500 x 1500
| ships |  568 x 526, 556 x 528, 550 x 570, …, 294 x 244, 266 x 254, 238 x 232
| ufsm-flame |  4000 x 3000, 254 x 254
Fauna | catsfaces |  2954 x 3027, 2893 x 3016, 2418 x 2161, …, 1024 x 837, 647 x 690, 256 x 256
| dogsfaces |  678 x 796, 256 x 256
Flora | flowers |  828 x 500, 819 x 500, 764 x 500, …, 500 x 525, 500 x 507, 500 x 500
| plantpat |  5184 x 3456, 4608 x 3456, 4032 x 3024, 4000 x 3000, 4000 x 2672, 3024 x 4032, 2592 x 1728
Medical | melanomaisic |  6000 x 4000, 5184 x 3456, 4288 x 2848, …, 1920 x 1080, 768 x 576, 640 x 480
| structretina |  700 x 605
Satellite | amazonia1 |  128 x 128
| cbers4a |  128 x 128
| deepglobe |  256 x 256
| isaid |  6661 x 6308, 6471 x 4479, 5963 x 5553, …, 436 x 793, 395 x 590, 211 x 521

Figure 2: LR images from our datasets. Top row, left to right: condoaerial,
massachbuildings, ships, catsfaces, dogsfaces, flowers; Bottom row, left to
right: plantpat, melanomaisic, structretina, amazonia1, cbers4a, isaid

###  3.3 Amount of required images

Taking into account the “amount” of required images, recent DL techniques for
image SR are created assuming a single image, known as single-image SR, but
they can also be devised assuming that there are a few (limited) number of
samples, which we can call few-shot image SR. There is currently a great
interest in few-shot learning (FSL) where we face the situation to solve a
task based on few samples with supervised information [54]. Note that here we
consider that few-shot SR assumes few but more than one image. There are also
other possibilities, like zero-shot SR presented by [55], where authors coped
with unsupervised SR by training image-specific SR networks at test time
rather than training a model on huge datasets.

Traditionally when researchers propose a new single-image SR technique, they
naturally compare their approach to other single-image SR strategies. However,
it would be nice to compare single-image SR techniques to non-single-image
ones to perceive whether it is advantageous to consider approaches which rely
on a unique image or, eventually, it is better to adopt solutions which
require more than one image but not so many of them.

Thus, three of the selected techniques were designed addressing single-image
SR, i.e. BlindSR, DAN, and MoESR. The other two, APA and FastGAN, are GAN-
based approaches requiring a few samples indeed and were not conceived for
single-image SR. Since the two latter did require more than one sample but not
too many, we limited the size of the datasets to 100 LR images as mentioned in
the previous section. In Section 3.4, we provide an overview of such DL
strategies.

All runnings were performed using a Bull Sequana X1120 computing node of the
SDumont supercomputer333https://sdumont.lncc.br/machine.php?pg=machine, where
such a node has 4 x NVIDIA Volta V100 graphics processing units (GPUs). Each
run was limited to four days being considered the latest model when the
execution exceeded this time.

###  3.4 DL techniques

In this section, we briefly describe the selected DL techniques starting with
the single-image approaches. In [45], authors proposed a framework that can
achieve blind image SR in a fully automated manner, while also meeting the
practical scaling needs of video production. We name it here as BlindSR and
such a framework is composed of three main components. The first one is a
degradation-aware SR network used to generate an HR image based on a LR input
image and the corresponding blur kernel. Secondly, a kernel discriminator is
trained to analyse the output HR image and predict errors caused by incorrect
blur kernel input. Finally, an optimisation procedure aims to recover both the
degradation kernel and the HR image by minimising the predicted error using
their kernel discriminator.

DAN solves the blind image SR problem via an alternating optimization
algorithm which restores an HR image and estimates the corresponding blur
kernel alternately [46]. Thus, these two subproblems are handled both via
convolutional neural modules namely Restorer and Estimator, respectively. More
specifically, the Restorer restores an HR image based on the predicted
Estimator’s kernel, and the Estimator estimates a blur kernel with the help of
the restored HR image.

Some blind SR techniques train a unique degradation-aware network (for
multiple kernels) on external datasets like BlindSR described above [45]. But
there are performance problems with these approaches. Other direction are
self-supervised techniques [56] but they are usually costly and there is
limited information to learn from a single image. Aiming at benefiting of the
best characteristics of both types of solution, MoESR was proposed considering
different experts for different degradation kernels. For every input image,
the technique predicts the degradation kernel and super-resolve the LR image
using the most adequate kernel-specific expert. To predict the degradation
kernel, they use two components in combination. Image Sharpness Evaluator
(ISE) assesses the sharpness of the images generated by the experts. These
evaluations are used by the Kernel Estimation Network (KEN) to estimate the
kernel and select the best pretrained expert network.

Regarding the non-single-image techniques, the greatest motivation to create
APA [44] is related to the main challenge in training GANs with limited (few)
data: the risk of discriminator overfitting which can lead to unstable
training dynamics [57, 58]. To deal with this issue, the technique called
Adaptive Pseudo Augmentation, or APA, regularises the discriminator without
introducing any external augmentations or regularisation terms. Unlike
previous approaches that relied on standard data augmentations [57, 58], APA
leverages the generator within the GAN itself to provide the augmentation, a
more natural way of regularisation of the overfitting of the discriminator. In
accordance with the authors, the APA approach is simple and more adaptable to
different settings and training conditions than model regularisation, without
requiring manual tuning.

Other FSL solution and based on GANs is FastGAN [47]. Its authors were driven
by the realisation that training GANs on high-fidelity (HR) images often
necessitates a vast number of training images and large-scale GPU clusters. To
address the challenge of few-shot image synthesis via GAN with minimal
computing costs, FastGAN was proposed as a lightweight GAN architecture which
produces high-quality results at a resolution of 1024 x 1024 pixels. Their
technique involves incorporating two techniques: a skip-layer channel-wise
excitation module and a self-supervised discriminator trained as a feature-
encoder.

###  3.5 Metrics

As we have already mentioned, NIQE and the MANIQA score were the selected
metrics. NIQE was calculated considering the HR images (1024 x 1024 pixels)
generated by the techniques. However, according to its authors, the MANIQA
model [49] is best suited for 222 x 224 pixel images. Thus, we downsampled the
created HR image (1024 x 1024 pixels) to a resolution of 224 x 224 pixels in
order to calculate the MANIQA score.

Note that such a degradation approach accomplished to calculate the MANIQA
score was the same for all generated HR images and, therefore, there is no
favouring of a certain technique in relation to the obtained value.
Furthermore, we got the MANIQA score considering a subset of the datasets and
HR images (1024 x 1024) and we did not notice a great difference in the scores
compared to when we used the reduced images (224 x 224). Thus, we followed the
authors’ suggestions and considered the images in the lowest resolution (224 x
224) as input for calculating the MANIQA score.

As shown in Table 2 and as we have already pointed out, the original images of
some datasets are HR ones. But there are datasets which have only original LR
images and not the corresponding HR samples. In addition to that, NR-IQA
metrics are more in line with the reality when we deal with a blind setting.
These are the reasons to select only NR-IQA metrics for our experiment.

##  4 Results

In this section, we present the results of our experiment splitting the
sections based on the research questions we have defined earlier.

###  4.1 RQ_1

####  4.1.1 NIQE

Table 3 presents the NIQE values in the perspective of the broader domain. In
this case, we consider the mean NIQE values for all the datasets of a certain
domain where the best (minimum) and worst (maximum) values, and the techniques
which produced them, are shown. On the other hand, Table 4 presents a finer
analysis per dataset and also considering the mean NIQE values. Recall that as
lower the NIQE value, the better the perceptual quality. Fig. 3 shows all mean
NIQE values considering all broader domains and datasets.

Table 3: Mean NIQE values: Broader domains Domain | Min (Best) | Max (Worst)
---|---|---
| Technique | NIQE | Technique | NIQE
Aerial | MoESR | 14.648013 | BlindSR | 22.534419
Fauna | MoESR | 14.629656 | BlindSR | 18.961343
Flora | MoESR | 14.615975 | BlindSR | 21.668490
Medical | APA | 14.520624 | BlindSR | 18.717661
Satellite | MoESR | 14.385091 | BlindSR | 24.726333
Table 4: Mean NIQE values: Datasets Domain | Dataset | Min (Best) | Max (Worst)
---|---|---|---
Aerial | condoaerial | MoESR | 14.648013 | BlindSR | 20.611777
| massachbuildings | MoESR | 16.427236 | BlindSR | 22.199742
| ships | APA | 17.688239 | BlindSR | 22.534419
| ufsm-flame | APA | 15.492423 | BlindSR | 21.684649
Fauna | catsfaces | MoESR | 14.629656 | BlindSR | 18.702201
| dogsfaces | MoESR | 15.314877 | BlindSR | 18.961343
Flora | flowers | MoESR | 14.615974 | BlindSR | 18.174120
| plantpat | APA | 15.241252 | BlindSR | 21.668490
Medical | melanomaisic | APA | 15.025945 | BlindSR | 18.717661
| structretina | APA | 14.520624 | BlindSR | 16.517888
Satellite | amazonia1 | APA | 16.027069 | BlindSR | 23.698020
| cbers4a | APA | 16.398990 | DAN | 17.438634
| deepglobe | APA | 16.863639 | BlindSR | 24.726333
| isaid | MoESR | 14.385091 | BlindSR | 21.504252

(a) Aerial

(b) Fauna

(c) Flora

(d) Medical

(e) Satellite

Figure 3: Mean NIQE values: All broader domains and datasets

Results show that if we consider the broader domain (Table 3), MoESR was the
most outstanding approach being the best in four out of the five domains. Only
in the Medical domain APA was the top technique. Moreover, BlindSR was the
worst strategy with the maximum NIQE values for all domains. However, in the
analysis per dataset (Table 4), APA was the best technique obtaining the
minimum NIQE values in eight out of 14 datasets. MoESR was the best in six out
of them. BlindSR was, again, the worst of all the techniques.

Since MoESR was the best in one perspective and APA was the winner in the
other analysis, we performed an improvement evaluation to reach a final
decision in accordance with the NIQE value. In other words, the point is not
only to say that a DL technique is better than the other but how much better
it is. The improvement metric, $I\%$, is calculated as follows:

$I\%=\frac{(W-B)\times 100}{B}$ (6)

where $B$ and $W$ are the best and worst value of the metric, respectively,
comparing both techniques under the same dataset. In Table 5, we show the
improvement of MoESR over APA regarding NIQE where in all datasets in this
table, MoESR was superior than APA. As highlighted in bold, the catsfaces
dataset was the one where MoESR got the highest improvement (6.472). In this
case, $B=14.629656$ and $W=15.576498$. Hence, to know the APA’s NIQE ($W$), we
calculate:

$W=B+\frac{I\%}{100}\times B.$ (7)

The idea is that the improvement is the additional gain that the best approach
has compared to the worst one. This is because as lower the NIQE value, the
better.

Table 5: Improvement of MoESR over APA Dataset | MoESR | APA | $I\%$
---|---|---|---
condoaerial | 14.648013 | 15.292783 | 4.402
massachbuildings | 16.427236 | 16.760828 | 2.031
catsfaces | 14.629656 | 15.576498 | 6.472
dogsfaces | 15.314877 | 15.444758 | 0.848
flowers | 14.615974 | 14.787114 | 1.171
isaid | 14.385091 | 15.01201 | 4.358
$\overline{I\%}$ |  |  | 3.214

In Table 6 is the other way around where we can see the improvement of APA
over MoESR. The reasoning is the same as presented in Equations 6 and 7. Thus,
the highest improvement (12.489) of APA over MoESR was obtained in the
melanomaisic dataset. Note that this improvement of APA is almost twice the
best improvement of MoESR. Furthermore, the average improvement
($\overline{I\%}$) is also higher favouring APA over MoESR. Thus, based on the
NIQE metric, the best approach was APA followed by MoESR. The worst was
BlindSR.

Table 6: Improvement of APA over MoESR Dataset | APA | MoESR | $I\%$
---|---|---|---
ships | 17.688239 | 18.220023 | 3.006
ufsm-flame | 15.492423 | 15.614556 | 0.788
plantpat | 15.241252 | 15.80596 | 3.705
melanomaisic | 15.025945 | 16.902496 | 12.489
structretina | 14.520624 | 15.914241 | 9.598
amazonia1 | 16.027069 | 17.041395 | 6.329
cbers4a | 16.39899 | 17.037916 | 3.896
deepglobe | 16.863639 | 17.43862 | 3.410
$\overline{I\%}$ |  |  | 5.403

####  4.1.2 MANIQA

In Table 7, the MANIQA scores are shown in the perspective of the broader
domain while Table 8 presents the per dataset analysis. As previously, we show
the mean values of the metric. Recall that as higher the MANIQA score, the
better the perceptual quality. Fig. 4 shows all mean MANIQA scores considering
all broader domains and datasets.

Table 7: Mean MANIQA scores: Broader domains Domain | Max (Best) | Min (Worst)
---|---|---
| Technique | MANIQA | Technique | MANIQA
Aerial | DAN | 0.696858 | FastGAN | 0.409388
Fauna | MoESR | 0.713253 | FastGAN | 0.515693
Flora | MoESR | 0.698373 | APA | 0.430053
Medical | MoESR | 0.614705 | APA | 0.432007
Satellite | DAN | 0.736443 | FastGAN | 0.327089
Table 8: Mean MANIQA scores: Datasets Domain | Dataset | Max (Best) | Min (Worst)
---|---|---|---
Aerial | condoaerial | MoESR | 0.657257 | FastGAN | 0.409388
| massachbuildings | DAN | 0.696858 | FastGAN | 0.525381
| ships | DAN | 0.577708 | FastGAN | 0.492311
| ufsm-flame | DAN | 0.618042 | FastGAN | 0.493545
Fauna | catsfaces | MoESR | 0.713253 | FastGAN | 0.611105
| dogsfaces | MoESR | 0.638982 | FastGAN | 0.515693
Flora | flowers | MoESR | 0.698373 | FastGAN | 0.533008
| plantpat | MoESR | 0.606683 | APA | 0.430053
Medical | melanomaisic | DAN | 0.542052 | FastGAN | 0.449874
| structretina | MoESR | 0.614705 | APA | 0.432007
Satellite | amazonia1 | MoESR | 0.579970 | FastGAN | 0.417986
| cbers4a | MoESR | 0.408773 | FastGAN | 0.327089
| deepglobe | MoESR | 0.591723 | APA | 0.330667
| isaid | DAN | 0.736443 | FastGAN | 0.445917

(a) Aerial

(b) Fauna

(c) Flora

(d) Medical

(e) Satellite

Figure 4: Mean MANIQA scores: All broader domains and datasets

In the broader domain perspective, MoESR was the best (three out five top
results) followed by DAN (two best positions). Unlike the NIQE metric where
APA was considered the best DL technique overall, it performed poorly in
accordance with the MANIQA score occupying the penultimate place (in two
domains, Flora and Medical, APA was the worst approach). FastGAN was the worst
of all the techniques. Again unlike the results presented in the previous
section, here we have a completely agreement between the broader domain and
the per dataset perspectives. Thus, considering each dataset in its own, MoESR
got again the top place (nine best MANIQA scores) while DAN was the second
best technique (five best MANIQA scores). Likewise, APA got the penultimate
place (three worst MANIQA scores) and FastGAN was the worst of all the DL
techniques.

Despite such an agreement, we accomplished the improvement analysis to check
these results considering the two best DL models: MoESR and DAN. But, we
slightly changed the way to calculate the improvement metric as shown below:

$I\%=\frac{(B-W)\times 100}{W}$ (8)

where $B$ and $W$ are the best and worst value of the metric, respectively,
comparing both techniques under the same dataset. And now, the idea is that
the improvement is the additional gain that the worst approach needs to reach
the best one. This is because as higher the MANIQA score, the better. Thus, we
use the following equation:

$B=W+\frac{I\%}{100}\times W.$ (9)

Table 9 presents the improvement of DAN over MoESR while Table 10 shows the
opposite (MoESR over DAN). As we can see in bold in both tables, the highest
improvement of MoESR (6.069) is almost twice the highest due to DAN (3.245).
Moreover, MoESR’s average improvement is also better than the DAN’s average
improvement. These results confirms the previous conclusions and MoESR was the
best technique followed by DAN.

Table 9: Improvement of DAN over MoESR Dataset | DAN | MoESR | $I\%$
---|---|---|---
massachbuildings | 0.696858 | 0.674955 | 3.245
ships | 0.577708 | 0.571233 | 1.134
ufsm-flame | 0.618042 | 0.608577 | 1.555
melanomaisic | 0.542052 | 0.541265 | 0.145
isaid | 0.736443 | 0.72679 | 1.328
$\overline{I\%}$ |  |  | 1.481
Table 10: Improvement of MoESR over DAN Dataset | MoESR | DAN | $I\%$
---|---|---|---
condoaerial | 0.657257 | 0.653125 | 0.633
catsfaces | 0.713253 | 0.681594 | 4.645
dogsfaces | 0.638982 | 0.605406 | 5.546
flowers | 0.698373 | 0.674927 | 3.474
plantpat | 0.606683 | 0.578225 | 4.922
structretina | 0.614705 | 0.609501 | 0.854
amazonia1 | 0.57997 | 0.546786 | 6.069
cbers4a | 0.408773 | 0.403834 | 1.223
deepglobe | 0.591723 | 0.580666 | 1.904
$\overline{I\%}$ |  |  | 3.252

Considering both metrics, NIQE and MANIQA score, we can state that MoESR was
the most outstanding approach. It presented a consistent performance under
NIQE, being the second best technique, and, as we have just said, it got the
top place based on the MANIQA score. Note that we saw contradictory
performances regarding APA where it was the best strategy evaluated via NIQE
and almost the worst approach, if we take into account the MANIQA score.

###  4.2 RQ_2

Question RQ_2 is about the similarity of behaviours between the two best
approaches according to the MANIQA score: MoESR and DAN. Here, our intention
is to see whether the two best models “agree” in terms of the HR images they
create. Thus, we took the 10 images with the best MANIQA scores from both
techniques, for each dataset, and generated two sets $High_{M}$ and
$High_{D}$, meaning the scores of the images from MoESR and DAN, respectively.
Hence, we just calculated the cardinality of a new set ($C_{B}$) derived by
the intersection between the previous two sets:

$\lvert C_{B}\rvert=\lvert High_{M}\cap High_{D}\rvert.$ (10)

We also did the same and calculated the cardinality of the set of non-common
elements, $N_{B}$, such that:

$\lvert N_{B}\rvert=\frac{\lvert High_{M}\cup High_{D}\rvert-\lvert
High_{M}\cap High_{D}\rvert}{2}.$ (11)

For instance, for the condoaerial dataset, $\lvert
C_{B}(condoaerial)\rvert=7$. This implies that seven out of the 10 images with
the highest MANIQA scores are common to MoESR and DAN, and hence there is an
agreement between both techniques regarding these images. Thus, $\lvert
N_{B}(condoaerial)\rvert=3$, representing that three out of the 10 images are
not common showing disagreement. The average value of the cardinalities of all
common sets is $\overline{C_{B}(all)}=7.36$ while for the non-common case is
$\overline{N_{B}(all)}=2.64$.

We did the same with the 10 images with lowest scores and created the sets
$Low_{M}$ and $Low_{D}$ due to MoESR and DAN, respectively. Likewise, we
calculated the cardinality of $C_{W}$ and $N_{W}$ for each dataset, just
replacing the high sets by the low ones in the Equations 10 and 11. For
example, for the amazonia1 dataset, $\lvert C_{W}(amazonia1)\rvert=6$ and
$\lvert N_{W}(amazonia1)\rvert=4$, meaning that six images with the lowest
scores are common and four are not common. The average value of the
cardinalities of all common sets is $\overline{C_{W}(all)}=7.5$ while for the
non-common case is $\overline{N_{W}(all)}=2.5$.

Finally, we obtained the Kendall Rank Correlation Coefficient (Kendall’s
$\tau$ coefficient) for two situations. The first case is considering $C_{B}$
and $C_{W}$ and the second situation is based on $N_{B}$ and $N_{W}$. In both
cases, $\tau=0.306912$ showing a good correlation between each pair of
variables. Fig. 5 shows the correlation in detail.

(a) Correlation of $C$ sets

(b) Correlation of $N$ sets

Figure 5: Kendall Rank Correlation Coefficient: results

The interpretation of these results is that the images detected as having the
best, as well as the worst, perceptual qualities, based on the MANIQA scores,
are somewhat “common” to both techniques. Hence, we can conclude that both
approaches present a similar behaviour. It is important to emphasise that the
relevance of this analysis is to realise whether the two best approaches
generate HR images with better or worse perception qualities in a relatively
uniform manner, taking the LR images as input.

##  5 Discussion

We start this section by stressing some points based on a visual analysis.
Fig. 6 shows the LR image and a piece of the corresponding HR image generated
by the three single-image blind SR techniques. The top HR image generated by
DAN, based on an LR image of the isaid Satellite dataset, is the one with the
highest MANIQA score (0.850084) of all images. We clearly see that the roof
divisions of the building appear more vivid in the image created by DAN.
MoESR’s image appears blurred while BlindSR’s image seems to present something
like a salt-and-pepper noise.

On the other hand, the bottom HR image created by BlindSR is the one with the
lowest MANIQA score (0.240234) of all images. The LR image is from the cbers4a
Satellite dataset where the scene presents water and land. This is a cloudy
scene obtained by the CBERS-4A satellite and it is interesting to realise that
such bright images from cbers4a in general produced the lowest MANIQA scores
(see Fig. 4(e)).

Figure 6: Visual analysis: highest and lowest MANIQA scores

Other LR images and the corresponding piece of HR images created by the three
single-image blind SR models are presented in Fig. 7. MoESR produced the
highest MANIQA score for the top images (condoaerial, flowers, melanomaisic,
amazonia1) while DAN produced the highest score for the bottom one
(deepglobe).

Dataset | LR | MoESR | DAN | BlindSR
---|---|---|---|---
condoaerial |  |  |  |
flowers |  |  |  |
melanomaisic |  |  |  |
amazonia1 |  |  |  |
deepglobe |  |  |  |
Figure 7: Other created HR images by the single-image blind SR approaches

Since APA and FastGAN are GAN-based non-single image techniques, some known
issues of GANs appeared here. In order to use APA in a custom dataset, three
phases are required: prepare the dataset, training, and inference for
generating images. As usual, the training is the more demanded step and APA
indeed exceeded the limit we set up (four days) not finishing the training. We
considered its latest model for inference. We used 4 x NVIDIA Volta V100 GPUs
and the datasets are very small (100 samples). It is important to mention that
other DL techniques demanded only one out of the four GPUs and the limit was
enough for them. Thus, APA is a very “heavy” model. Furthermore, some HR
images were flipped and others upside down compared to the source LR images.
There are even cases where the images were not indeed generated (it is likely
that the images are basically noise).

Fig. 8 shows these cases where we have the LR image and the corresponding
APA’s HR image. Note that we downsampled the APA images to 128 x 128 pixels
for this figure to compare to the LR images. Moreover, the deepglobe image is
here just to show a sample of the dataset since APA’s output seems to be
basically noise.

LR | APA - Flipping | LR | APA - Flipping
---|---|---|---
|  |  |
LR | APA - Up Down | LR | APA - Noise
|  |  |
Figure 8: Some issues presented by APA. Top row, left to right: dogsfaces,
melanomaisic; Bottom row, left to right: catsfaces, deepglobe

As for FastGAN, the required phases to use it are training and inference, and
its training is considerably faster/lighter than APA. But both FastGAN and APA
presented a very known problem related to GANs: mode collapse [59]. It happens
when the generator model produces a not very significant set of images that
fail to capture the full diversity of the real data distribution. Thus, the
fake created samples are quite similar or even identical. In Fig. 9, we see
some images from FastGAN where mode collapse occurs. Again, we downgraded the
HR images for better presentation.

|  |  |
---|---|---|---
|  |  |
Figure 9: Mode collapse in FastGAN: condoaerial

Based on the MANIQA scores (Section 4.1.2), APA and FastGAN were the worst
techniques. Thus, we can conclude based on the results of our experiment that,
for blind image SR, single-image and non-GAN-based approaches are the best way
to go.

Considering the best two DL techniques in accordance with the MANIQA score, we
may say that the HR images created by MoESR are sharper than the ones of DAN.
Fig. 10 presents the measures of the maximum overall contrast of the images
with the highest MANIQA scores taking into account both techniques and the 14
datasets. For instance, for the amazonia1 dataset, the image derived by MoESR
has higher MANIQA score than the one of DAN and, thus, we considered the
former and the corresponding DAN’s image. As for isaid, DAN’s image got higher
MANIQA score and thus this image and the corresponding MoESR’s image were
analysed. Fig. 11 follows the same reasoning but now the images are the ones
with the lowest MANIQA scores.

Figure 10: Highest MANIQA scores: Contrast Figure 11: Lowest MANIQA scores:
Contrast

Since the higher the maximum overall contrast the sharper the image, we can
realise that the images of MoESR are sharper than the ones of DAN. But, note
that perceptual quality is not necessarily the same thing as sharpness. An
image with higher contrast does not imply that it has a better perceptual
quality than one with lower contrast.

A possible explanation for the images created by MoESR being sharper than
DAN’s images, and sometimes present oversharpening, is the combination of the
perfomances of ISE and KEN. The ISE component is trained to detect blurry or
oversharpened regions and predicts errors from the ground-truth image. Since
KEN uses the sharpness measures from ISE to estimate the kernel and select the
best pretrained model, misleading evaluations of sharpness by ISE may
compromise the decision made by the KEN component.

On the other hand, HR images generated by DAN are generally blurry which might
lead to a poor perceptual quality. In DAN, the kernel is initialised by Dirac
function, and it is also reshaped and then reduced by principal component
analysis (PCA) [60]. The kernel is reduced by PCA and thus the Estimator only
needs to estimate the PCA result of the blur kernel. There is naturally loss
of information when using PCA for dimensionality reduction, and recent
evaluations show that PCA results are not as reliable and robust as it is
usually assumed to be [61]. This is a possible explanation for this issue
related to DAN.

As mentioned in Section 4, there is not an agreement between the ranking of
best and worst techniques considering a classical metric (NIQE) and a DNN-
based one (MANIQA score). While APA presented the better performance under
NIQE, it is almost the worst solution according to the MANIQA score. Thus, we
believe that relying on a more recent NR-IQA metric, like the MANIQA score and
which presented quite superior performance [41] than traditional metrics, is
more advisable.

Notice that we do not have the mean opinion scores (MOSs) of the HR images as
we do not submit them for observers to assign it. But, looking at the HR
images generated by all DL techniques for all sets (see the datasets
repository of this research), it is not difficult to see that the perception
quality of the images as a whole needs to improve. Note that in MoESR and DAN,
the generation of the HR images had to be done in two stages, using the 2x and
4x scaling factors in sequence. In general, there are several blurred images
and others with oversharpening. Despite the sophistication of the evaluated
DNNs, we believe that new approaches, addressing larger scaling factors, are
necessary for the future.

##  6 Conclusions

Given the significant number of DL techniques/DNNs which are created on a
daily basis, it is relevant to perform independent and unbiased controlled
experiments to suggest the most suitable approaches for professionals. This
article is in line with this reasoning, where we presented an large scaling
factor (8x) experiment which evaluated recent DL techniques tailored for blind
image SR: APA, BlindSR, DAN, FastGAN, and MoESR. Some of the techniques were
designed for single-image while others for non-single-image blind SR. In
addition to selecting a larger scaling factor, another difference in this
study is that we showed a more detailed analysis of the results, also focusing
on explaining in depth the behaviours of the approaches. Mostly based on
publicly released sources of images, we adapted and created 14 small (100
samples) LR images datasets from five different broader domains to provide a
significant range of distinct images to the techniques. We also considered two
NR-IQA metrics, the classical NIQE and the DNN-based MANIQA score.

Results show that the MoESR model was the most outstanding approach followed
by DAN. GAN-based approaches presented classical issues such as mode collapse
and noise, in some cases. The outcomes of our experiment suggest that single-
image and non-GAN-based techniques are more promising for blind image SR,
although such a conclusion needs more experimentation to be completely
confirmed. By visually inspecting the HR images created by the techniques, we
may state that, despite the remarkable solutions, new strategies are required
for larger scaling factor requirements.

As future directions, we firstly intend to carry out even higher scaling
factor experiments, such as 16x. We will also increase not only the amount of
blind image SR approaches to assess but also the number of datasets to get
more evidences about the conclusions we have made. Other recent NR-IQA metrics
will also be considered in these new controlled experiments.

## Acknowledgements

This research was developed within the project Classificação de imagens via
redes neurais profundas e grandes bases de dados para aplicações aeroespaciais
(Image classification via Deep neural networks and large databases for
aeroSpace applications - IDeepS). The IDeepS
(https://github.com/vsantjr/IDeepS) project is supported by the Laboratório
Nacional de Computação Científica (LNCC/MCTI, Brazil) via resources of the
SDumont supercomputer (http://sdumont.lncc.br).

## References

  * [1] K. Yamashita and K. Markov.  Medical image enhancement using super resolution methods.  In Valeria V. Krzhizhanovskaya, Gábor Závodszky, Michael H. Lees, Jack J. Dongarra, Peter M. A. Sloot, Sérgio Brissos, and João Teixeira, editors, Computational Science – ICCS 2020, pages 496–508, Cham, 2020. Springer International Publishing.
  * [2] Y. Huang, L. Shao, and A. F. Frangi.  Simultaneous super-resolution and cross-modality synthesis of 3d medical images using weakly-supervised joint convolutional sparse coding.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5787–5796, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society.
  * [3] W. Ahmad, H. Ali, Z. Shah, and S. Azmat.  A new generative adversarial network for medical images super resolution.  Scientific Reports, 12:9533, 2022.  https://doi.org/10.1038/s41598-022-13658-4.
  * [4] R. Gupta, A. Sharma, and A. Kumar.  Super-resolution using gans for medical imaging.  Procedia Computer Science, 173:28–35, 2020.  International Conference on Smart Sustainable Intelligent Computing and Applications under ICITETM2020.
  * [5] J. M. Liborio, C. Melo, and M. Silva.  Internet video delivery improved by super-resolution with gan.  Future Internet, 14(12), 2022.
  * [6] H. Yeo, Y. Jung, J. Kim, J. Shin, and D. Han.  Neural adaptive content-aware internet video delivery.  In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, OSDI’18, pages 645–661, USA, 2018. USENIX Association.
  * [7] H. Yeo, S. Do, and D. Han.  How will deep learning change internet video delivery?  In HotNets-XVI: Proceedings of the 16th ACM Workshop on Hot Topics in Networks, pages 57–64, 2017.
  * [8] P. Rasti, T. Uiboupin, S. Escalera, and G. Anbarjafari.  Convolutional neural network super resolution for face recognition in surveillance monitoring.  In Francisco José Perales and Josef Kittler, editors, Articulated Motion and Deformable Objects, pages 175–184, Cham, 2016. Springer International Publishing.
  * [9] E. Zhou, H. Fan, Z. Cao, Y. Jiang, and Q. Yin.  Learning face hallucination in the wild.  Proceedings of the AAAI Conference on Artificial Intelligence, 29(1), Mar. 2015.
  * [10] S. Zhu, S. Liu, C. C. Loy, and X. Tang.  Deep cascaded bi-network for face hallucination.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 614–630, Cham, 2016. Springer International Publishing.
  * [11] P. Wang, B. Bayram, and E. Sertel.  A comprehensive review on deep learning based remote sensing image super-resolution methods.  Earth-Science Reviews, 232:104110, 2022.
  * [12] K. Jiang, Z. Wang, P. Yi, G. Wang, T. Lu, and J. Jiang.  Edge-enhanced gan for remote sensing image superresolution.  IEEE Transactions on Geoscience and Remote Sensing, 57(8):5799–5812, 2019.
  * [13] K. Jiang, Z. Wang, P. Yi, J. Jiang, J. Xiao, and Y. Yao.  Deep distillation recursive network for remote sensing imagery super-resolution.  Remote Sensing, 10(11), 2018.
  * [14] Y. Xu, W. Luo, A. Hu, Z. Xie, X. Xie, and L. Tao.  Te-sagan: An improved generative adversarial network for remote sensing super-resolution images.  Remote Sensing, 14(10), 2022.
  * [15] B. Niu, W. Wen, W. Ren, X. Zhang, L. Yang, S. Wang, K. Zhang, X. Cao, and H. Shen.  Single image super-resolution via a holistic attention network.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 191–207, Cham, 2020. Springer International Publishing.
  * [16] J. Zhang, T. Xu, J. Li, S. Jiang, and Y. Zhang.  Single-image super resolution of remote sensing images with real-world degradation modeling.  Remote Sensing, 14(12), 2022.
  * [17] Z. Wang, J. Chen, and S. C. H. Hoi.  Deep learning for image super-resolution: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10):3365–3387, 2021.
  * [18] C. E. Duchon.  Lanczos filtering in one and two dimensions.  Journal of Applied Meteorology and Climatology, 18(8):1016 – 1022, 1979.
  * [19] J. Sun, Z. Xu, and H.-Y. Shum.  Image super-resolution using gradient profile prior.  In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8, 2008.
  * [20] Z. Xiong, X. Sun, and F. Wu.  Robust web image/video super-resolution.  IEEE Transactions on Image Processing, 19(8):2017–2028, 2010.
  * [21] J. Egger, A. Pepe, C. Gsaxner, Y. Jin, J. Li, and R. Kern.  Deep learning-a first meta-survey of selected reviews across scientific disciplines, their commonalities, challenges and research impact.  PeerJ Computer science, 7, 2021.  © 2021 Egger et al.
  * [22] C. Dong, C. C. Loy, K. He, and X. Tang.  Learning a deep convolutional network for image super-resolution.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 184–199, Cham, 2014. Springer International Publishing.
  * [23] J. Kim, J. Lee, and K. Lee.  Accurate image super-resolution using very deep convolutional networks.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1646–1654, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society.
  * [24] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee.  Enhanced deep residual networks for single image super-resolution.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1132–1140, 2017.
  * [25] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi.  Photo-realistic single image super-resolution using a generative adversarial network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [26] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy.  Esrgan: Enhanced super-resolution generative adversarial networks.  In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, September 2018.
  * [27] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu.  Image super-resolution using very deep residual channel attention networks.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [28] X. Cheng, X. Li, J. Yang, and Y. Tai.  Sesr: Single image super resolution with recursive squeeze and excitation networks.  In 2018 24th International Conference on Pattern Recognition (ICPR), pages 147–152, 2018.
  * [29] A. Bulat, J. Yang, and G. Tzimiropoulos.  To learn image super-resolution, use a gan to learn how to do image degradation first.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision – ECCV 2018, pages 187–202, Cham, 2018\. Springer International Publishing.
  * [30] J. Gu, H. Lu, W. Zuo, and C. Dong.  Blind super-resolution with iterative kernel correction.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1604–1613, Los Alamitos, CA, USA, jun 2019. IEEE Computer Society.
  * [31] Z. Yue, Q. Zhao, J. Xie, L. Zhang, D. Meng, and K. K. Wong.  Blind image super-resolution with elaborate degradation modeling on noise and kernel.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2118–2128, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society.
  * [32] D. Martin, C. Fowlkes, D. Tal, and J. Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416–423 vol.2, 2001.
  * [33] P. Arbeláez, M. Maire, C. Fowlkes, and J. Malik.  Contour detection and hierarchical image segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5):898–916, 2011.
  * [34] E. Agustsson and R. Timofte.  Ntire 2017 challenge on single image super-resolution: Dataset and study.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1122–1131, 2017.
  * [35] Y. Blau, R. Mechrez, R. Timofte, T. Michaeli, and L. Zelnik-Manor.  The 2018 pirm challenge on perceptual image super-resolution.  In Laura Leal-Taixé and Stefan Roth, editors, Computer Vision – ECCV 2018 Workshops, pages 334–355, Cham, 2019. Springer International Publishing.
  * [36] M. Bevilacqua, A. Roumy, C. Guillemot, and M. A. Morel.  Low-complexity single-image super-resolution based on nonnegative neighbor embedding.  In R. Bowden, J. Collomosse, and K. Mikolajczyk, editors, Proceedings of the British Machine Vision Conference, pages 1–10, Durham, UK, 2012. BMVA Press.
  * [37] R. Zeyde, M. Elad, and M. Protter.  On single image scale-up using sparse-representations.  In Jean-Daniel Boissonnat, Patrick Chenin, Albert Cohen, Christian Gout, Tom Lyche, Marie-Laurence Mazure, and Larry Schumaker, editors, Curves and Surfaces, pages 711–730, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg.
  * [38] J.-B. Huang, A. Singh, and N. Ahuja.  Single image super-resolution from transformed self-exemplars.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5197–5206, 2015.
  * [39] Z. Yuan, C. Tang, A. Yang, W. Huang, and W. Chen.  Few-shot remote sensing image scene classification based on metric learning and local descriptors.  Remote Sensing, 15(3), 2023.
  * [40] M. Haris, G. Shakhnarovich, and N. Ukita.  Deep back-projection networks for super-resolution.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1664–1673, Los Alamitos, CA, USA, jun 2018. IEEE Computer Society.
  * [41] J. Gu, H. Cai, C. Dong, J. S. Ren, R. Timofte, Y. Gong, S. Lao, S. Shi, J. Wang, S. Yang, T. Wu, W. Xia, Y. Yang, M. Cao, C. Heng, L. Fu, R. Zhang, Y. Zhang, H. Wang, H. Song, J. Wang, H. Fan, X. Hou, M. Sun, M. Li, K. Zhao, K. Yuan, Z. Kong, M. Wu, C. Zheng, M. V. Conde, M. Burchi, L. Feng, T. Zhang, Y. Li, J. Xu, H. Wang, Y. Liao, J. Li, K. Xu, T. Sun, Y. Xiong, A. Keshari, K. Komal, S. Thakur, V. Jakhetiya, B. N. Subudhi, H. Yang, H. Chang, Z. Huang, W. Chen, S. Kuo, S. Dutta, S. Das, N. A. Shah, and A. Tiwari.  Ntire 2022 challenge on perceptual image quality assessment.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 950–966, Los Alamitos, CA, USA, jun 2022\. IEEE Computer Society.
  * [42] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [43] A. Mittal, R. Soundararajan, and A. C. Bovik.  Making a “completely blind” image quality analyzer.  IEEE Signal Processing Letters, 20(3):209–212, 2013.
  * [44] L. Jiang, B. Dai, W. Wu, and C. C. Loy.  Deceive d: Adaptive pseudo augmentation for gan training with limited data.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 21655–21667. Curran Associates, Inc., 2021.
  * [45] V. Cornillère, A. Djelouah, W. Yifan, O. Sorkine-Hornung, and C. Schroers.  Blind image super-resolution with spatially variant degradations.  ACM Trans. Graph., 38(6), nov 2019.
  * [46] Z. Luo, Y. Huang, S. Li, L. Wang, and T. Tan.  Unfolding the alternating optimization for blind super resolution.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5632–5643. Curran Associates, Inc., 2020.
  * [47] B. Liu, Y. Zhu, K. Song, and A. Elgammal.  Towards faster and stabilized {gan} training for high-fidelity few-shot image synthesis.  In International Conference on Learning Representations, 2021.
  * [48] M. Emad, M. Peemen, and H. Corporaal.  Moesr: Blind super-resolution using kernel-aware mixture of experts.  In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4009–4018, 2022.
  * [49] S. Yang, T. Wu, S. Shi, S. Lao, Y. Gong, M. Cao, J. Wang, and Y. Yang.  Maniqa: Multi-dimension attention network for no-reference image quality assessment, 2022.
  * [50] K. Zhang, W. Zuo, and L. Zhang.  Learning a single convolutional super-resolution network for multiple degradations.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 3262–3271, 2018.
  * [51] M. Anderson, T. Adali, and X.-L. Li.  Joint blind source separation with multivariate gaussian model: Algorithms and performance analysis.  IEEE Transactions on Signal Processing, 60(4):1672–1683, 2012.
  * [52] J. Gu, Y. Shen, and B. Zhou.  Image processing using multi-code gan prior.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3012—3021, June 2020.
  * [53] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [54] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni.  Generalizing from a few examples: A survey on few-shot learning.  ACM Comput. Surv., 53(3), jun 2020.
  * [55] M. Irani A. Shocher, N. Cohen.  ”Zero-Shot” super-resolution using deep internal learning.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [56] S. Bell-Kligler, A. Shocher, and M. Irani.  Blind Super-Resolution Kernel Estimation Using an Internal-GAN.  Curran Associates Inc., Red Hook, NY, USA, 2019.
  * [57] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila.  Training generative adversarial networks with limited data.  In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc.
  * [58] S. Zhao, Z. Liu, J. Lin, J.-Y. Zhu, and S. Han.  Differentiable augmentation for data-efficient gan training.  In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc.
  * [59] A. Srivastava, L. Valkov, C. Russell, M. U. Gutmann, and C. Sutton.  Veegan: Reducing mode collapse in gans using implicit variational learning.  Neural Information Processing Systems, 2017.
  * [60] A. Mackiewicz and W. Ratajczak.  Principal components analysis (pca).  Computers & Geosciences, 19(3):303–342, 1993.
  * [61] E. Elhaik.  Principal component analyses (pca)-based findings in population genetic studies are highly biased and must be reevaluated.  Scientific Reports, 12(14683), 2022.  https://doi.org/10.1038/s41598-022-14395-4.


# Charm total cross sections with nonuniversal fragmentation treatment

A. Geiser (speaker)  Y. Yang  DESY Hamburg, Notkestrasse 85, D-22607 Hamburg,
Germany  S. Moch  O. Zenaiev  Hamburg University, Luruper Chaussee 149,
D-22761 Hamburg, Germany

###### Abstract

Total charm-pair cross sections in $pp$ collisions are interesting because
they can be calculated to NNLO in QCD without any reference to fragmentation
effects. On the other hand, the fiducial differential charm cross sections
from which the total cross sections must be extrapolated are currently known
to NLO+NLL at most (e.g. FONLL), and must be treated for known effects of
nonuniversal charm fragmentation. A new procedure using the FONLL framework as
input for an empirical parametrization of the data in both shape and
normalization, with all its parameters actually fitted to data, is used to
derive so-called data-driven FONLL (ddFONLL) parametrizations which can be
used to extrapolate the differential cross sections to total cross sections
with minimal bias. This includes an empirical treatment of all known non-
universal charm fragmentation effects, in particular for the baryon-to-meson
ratio as a function of transverse momentum. The total charm-pair cross
sections obtained in this way, which supersede all previous ones obtained
using the assumption of charm-fragmentation universality, are consistent with
NNLO predictions, and allow first studies of their sensitivity e.g. to the
charm-quark mass and/or the NNLO gluon PDF at very low transverse momentum
fractions $x$.

##  1 Introduction

The theory of Quantum-Chromo-Dynamics (QCD) is a well-established part of the
Standard Model of particle physics, which describes most of the processes
occurring in $pp$ collisions at LHC rather well. Predictions for charm
production are particularly challenging since, due to the closeness of the
charm quark mass, $m_{c}\sim 1.5$ GeV, to the nonperturbative QCD scale,
$\Lambda_{QCD}\sim 0.3$ GeV, the convergence of the perturbative series is
slow, resulting in large theoretical uncertainties. In addition,
nonperturbative effects enter the hadronization of charm quarks into charm
hadrons, the entities actually measurable in detectors. Recently, it was
established [1, 2, 3, 4] (Fig. 1) that the corresponding charm fragmentation
effects are non-universal, i.e. can not simply be transferred from
$e^{+}e^{-}$ and $ep$ measurements to $pp$ measurements and may be dependent
on kinematics. Finding an appropriate treatment of this non-universality is
one of the central themes of this contribution.

Measuring the total cross section for charm-quark-antiquark pair production,
$\sigma_{c\bar{c}}^{\mathrm{tot}}$, almost 111Hadron states containing a
$c\bar{c}$ pair or more than one $c$ quark contribute only about 1% to the
total charm cross section and are hence neglected here. identical to the total
summed and integrated cross section for the prompt production of charm hadrons
$H_{c}$ containing a single charm quark $c$ (not $\bar{c}$),
$\sigma_{H_{c}}^{\mathrm{all}}$, is of particular interest since the
corresponding theoretical predictions are currently the only $pp$ charm cross
sections calculable at next-to-next-to-leading order (NNLO) in QCD, and do
furthermore not depend upon charm fragmentation. As we will demonstrate, they
can thus be used to derive constraints on genuine QCD parameters such as the
charm-quark mass $m_{c}$ and the proton partons density functions (PDFs), at
NNLO. For such measurements, differential $H_{c}$ distributions measured in
limited kinematic ranges (fiducial cross sections) and for a restricted set of
$H_{c}$ final states need to be extrapolated to the total cross section, under
certain constraints concerning their shape and normalization in unmeasured
regions. Furthermore, the possibly phase space dependent relative
contributions of different $H_{c}$ final states (nonuniversality) need to be
accounted for.

In this contribution we will describe how such shape, normalization and
nonuniversality constraints can be obtained in an entirely data driven way,
based on a theory-inspired parametrization of the multi-differential cross
sections, and fully accounting for charm nonuniversality effects, without
attempting to rely on any particular nonuniversal charm fragmentation model.
Since space is limited, we will extensively refer to a previous writeup [5]
where applicable, and defer some other discussions to yet another writeup to
appear soon [6].

##  2 FONLL vs. ddFONLL calculations

The highest order calculations available so far for differential charm-
production cross sections are based on the massive next-to-leading order (NLO)
plus massless next-to-leading-log (NLL) approach, also known as general mass
variable flavour number scheme, one example of which, exploited here, is the
FONLL [7] approach. By construction, the NLL part of this calculation becomes
significant only at high values of charm transverse momentum ($p_{Tc}\gg
m_{c}$), such that, since the total cross section is dominated by the low
$p_{T}$ contribution, NLO massive fixed order calculations without resummation
could also be used instead for this purpose.

Details of some theory aspects of the data driven FONLL (ddFONLL) approach
used here, previously referred to as “modified FONLL”, are given in a previous
report [5]. In a nutshell, the FONLL theory, including its charm fragmentation
extension, is modified to ddFONLL by replacing the fixed “universal”
fragmentation fraction $f_{H_{c}}^{\mathrm{uni}}$ obtained from $e^{+}e^{-}$
and/or $ep$ collisions by a $p_{T}$-dependent hadron-production fraction
$\tilde{f}_{H_{c}}(p_{T})$ directly obtained from measurements at LHC (Fig.
1),

$d\sigma_{H_{c}}^{\mathrm{FONLL}}=f_{H_{c}}^{\mathrm{uni}}\cdot\left(d\sigma_{pp\rightarrow
c}^{\mathrm{FONLL}}\otimes D_{c\rightarrow
H_{c}}^{\mathrm{NP}}\right)\quad\to\quad
d\sigma_{H_{c}}^{\mathrm{ddFONLL}}=\tilde{f}_{H_{c}}(p_{T})\cdot\left(d\sigma_{pp\rightarrow
c}^{\mathrm{FONLL}}\otimes D_{c\rightarrow H_{c}}^{\mathrm{NP}}\right)$ (1)

while the FONLL parametrizations of the quark-level differential cross
sections $d\sigma_{pp\rightarrow c\bar{c}}^{\mathrm{FONLL}}$ and the
parametrization of the nonperturbative fragmentation function $D_{c\rightarrow
H_{c}}^{\mathrm{NP}}$, here using the Kartvelishvili [8] parametrization,
remain unchanged.

Figure 1: Integrated fragmentation fractions $f$ (left) and differential
hadron fractions $\tilde{f}$ as a function of $p_{T}$ (right) at 5 and 13 TeV,
for different charm hadron final states, compiled from [1, 2, 3, 4, 11].

The other change is that, instead of treating the FONLL QCD parameters
$\mu_{f}$ and $\mu_{r}$ (factorization and renormalization scales) and $m_{c}$
(charm pole mass) as external QCD parameters, they are empirically left to
float freely, to be fitted from data, separately for each $pp$ center-of-mass
energy. The same is true for the Kartvelishvili parameter $\alpha_{K}$. In
contrast to FONLL, ddFONLL is therefore no longer a QCD theory prediction,
rather a theory-inspired parametrization of the $pp$ data at a given center of
mass energy. For the PDFs, the CTEQ6.6 set [9], not relying on pp charm
fragmentation universality as input, is used, including uncertainties
consistent with the PROSA[10] low-$x$ gluon parametrization of the rapidity
dependence (only) of LHC charm data in different regions of $p_{T}$ (also see
previous explanations [5]). In the limit $\tilde{f}(p_{T})\equiv
f^{\mathrm{uni}}$ (e.g. for $e^{+}e^{-}$ or $ep$), the standard FONLL
parametrization is fully recovered, i.e. the approach remains fully consistent
with all successful previous $e^{+}e^{-}$ and $ep$ FONLL applications by
definition.

This ddFONLL approach implies the following assumptions:

  * •

The $p_{T}$ dependence of $\tilde{f}$ asymptotically approaches LEP
$e^{+}e^{-}$ values [13] at high $p_{T}$ (Fig. 2), i.e. at high $p_{T}$, charm
fragmentation universality is fully recovered.

  * •

As motivated by measurements [15], there is no strong non-universal rapidity
dependence of $\tilde{f}$, and any potential small residual dependencies are
absorbed into the global $\tilde{f}$ uncertainties. This assumption is also
verified a posteriori by the fits to data below (Fig. 4) and elsewhere [5, 6].

  * •

Non-universal effects in the nonperturbative fragmentation function $D^{NP}$
at a given $pp$ center-of-mass energy are small enough such that the
approximation of treating them as factorizable w.r.t. $\tilde{f}$ in Eq. (1)
($p_{T}$-dependent reshuffling of final state charm hadron fractions) holds
well enough within current $\tilde{f}$ and freely fitted $\alpha_{K}$
uncertainties. This assumption is also verified a posteriori by the fits on
data below.

  * •

For $pp$, all ddFONLL parameters except the PDFs may be $\sqrt{s}$ dependent.

A further simplification made, again based on measurements, is

  * •

While the baryon-to-meson $H_{c}$ ratios are strongly $p_{T}$ dependent [2, 3,
4, 5] (Fig. 2(left)), the meson-to-meson and baryon-to-baryon ratios (Fig. 3)
remain $p_{T}$ independent. Known small deviations from this assumption, e.g.
for hadrons containing strange vs. $u$ or $d$ quarks [16], are absorbed into
the global $\tilde{f}$ uncertainties. Again, this approach is also verified a
posteriori on data.

Finally, although not yet applied here, a check is made whether the
parametrization above might also work for final states other than the weakly
decaying charm ground states. As an example, Fig. 2(right) shows the
$D^{*+}/D^{0}$ ratio obtained from ALICE for pp collisions at 5 TeV and 7 TeV,
compared to a parametrization using different Kartvelishvili parameters for
$D^{*}$ and $D^{0}$ as obtained from LEP data. As expected, since part of the
$D^{0}$ mesons originate from $D^{*}$ decays, the effective $D^{*}$
fragmentation is a bit harder than the effective $D^{0}$ fragmentation, and
the ratio can be well described by the LEP expectation within uncertainties,
consistent with the asymptotic LEP universality and $D$ meson ratio
universality assumptions. Since the Kartvelishvili parameter is a free
parameter of the fit, it can freely adjust itself also to resonant states, if
these are fitted “alone”. In the case of combined fits of ground states and
resonant states (not applicable here), some correlations between the
parameters would have to be treated.

Figure 2: Illustration of $p_{T}$ dependence and LEP asymptotic limit for the
ratio $\Lambda_{c}/D^{0}$ (left) and $D^{*}/D^{0}$ (right). The information is
compiled from [2, 3, 4, 11] and [12].

Figure 3: Illustration of meson-to-meson and baryon-to baryon ratios. The
information is compiled from [1, 2, 11].

##  3 Fit Results

Figure 4: $D^{0}+\overline{D}{}^{0}$ cross sections as a function $|y|$ in
bins of $p_{T}$ for $\sqrt{s}=13$ TeV. The red bands are the data driven FONLL
as obtained with non-universal charm fragmentation, which describe the data
[2, 14] (black points/grey boxes) well in the full phase space. The total
uncertainty of the data driven FONLL includes the uncertainties of the CTEQ6.6
PDF [9], $\tilde{f}$ (Fig. 1), and the $\chi^{2}$ scan.

The results of fitting the free ddFONLL parameters $\mu_{f}$, $\mu_{r}$,
$m_{c}$ and $\alpha_{K}$ to 5 TeV $D^{0}$ data were already shown previously
[5]. These results remain unchanged. The results of fitting these parameters
to 13 TeV $D^{0}$ data are shown in Fig. 4. Again, very good agreement with
data is obtained, reflected by the $\chi^{2}$ S-factor for the 4D fit coming
out very close to 1. The fit parameters, which will be discussed further in
another document [6], are consistent with those obtained at 5 TeV within
uncertainties. In both cases, the fitted $\alpha_{K}$ parameters are also
consistent with those expected from the asymptotic agreement with LEP (see
above). For illustration, the result of the 13 TeV ddFONLL parametrization
obtained from the fit is also shown in Fig. 5 for the comparison to ALICE
$D^{0}$ and $\Lambda_{c}$ assuming charm fragmentation universality. Both the
shape and normalization of both distributions are well described by ddFONLL
while standard FONLL fails in both shape and normalization for the
$\Lambda_{c}$ case. A very similar comparison is also available for the 5 TeV
case [5, 6]. These results are fully consistent with all the explicit and
implicit ddFONLL assumptions stated above (a posteriori verification).

Figure 5: Result of the fitted 13 TeV ddFONLL parametrization for $D^{0}$
(left) and $\Lambda_{c}$ (right), compared to ALICE data [2] and to the
standard FONLL prediction. Only the $\tilde{f}$ uncertainties are shown here
for the ddFONLL case.

Using the measurements for all bins in which measurements are available, and
the ddFONLL data parametrization in all others, the resulting 5 TeV and 13 TeV
total charm pair production cross sections are obtained to be

$\displaystyle\sigma^{\mathrm{tot}}_{c\bar{c}}(5\ \mathrm{TeV})$
$\displaystyle=\ \
8.43^{+0.25}_{-0.25}\mathrm{(data)}^{+0.40}_{-0.42}(\tilde{f})^{+0.67}_{-0.56}\mathrm{(PDF)}^{+0.13}_{-0.12}(\mu_{f},\mu_{r},m_{c},\alpha_{K})^{+0.65}_{-0.88}(f^{pp}_{D^{0}})\mathrm{[mb]}$
(2) $\displaystyle=\ \ 8.43^{+1.05}_{-1.16}(\mathrm{total})\
\mathrm{mb}.\hskip 233.3125pt$ (3)
$\displaystyle\sigma^{\mathrm{tot}}_{c\bar{c}}(13\ \mathrm{TeV})$
$\displaystyle=17.43^{+0.56}_{-0.53}\mathrm{(data)}^{+0.69}_{-0.78}(\tilde{f})^{+1.47}_{-1.22}\mathrm{(PDF)}^{+0.24}_{-0.18}(\mu_{f},\mu_{r},m_{c},\alpha_{K})^{+1.19}_{-2.05}(f^{pp}_{D^{0}})\mathrm{[mb]}$
(4) $\displaystyle=17.43^{+2.10}_{-2.57}(\mathrm{total})\ \mathrm{mb}.\hskip
233.3125pt$ (5)

in which $f^{pp}_{D^{0}}$ refers to the integrated $D^{0}$ fragmentation
fraction measured at 5 TeV or 13 TeV, respectively. The respective
extrapolation factors for unmeasured phase space are about 1.8 and 1.9. In
Fig. 6 these total cross sections are compared to previous determinations [11]
still based on the charm fragmentation universality assumption, and to NNLO
predictions. The complete treatment of charm fragmentation nonuniversality
significantly increases the extracted total charm cross sections, and
therefore replaces all previous such determinations. The measurements are
still consistent with the NNLO predictions, but now situated towards the upper
edge of the NNLO theory uncertainty band.

Figure 6: Comparison of measured total charm-pair cross sections at 5 TeV and
13 TeV (red bands) to previous determinations [11] (blue bands) and to NNLO
predictions with various PDF sets including theory uncertainties (points with
bars). Plots adapted from [11].

Very preliminary studies of the sensitivity of these predictions to the MSbar
charm quark mass, further detailed elsewhere [6], indicate that the mass
preferred by the data is consistent with the world average and can be
constrained with an uncertainty of order 200 MeV by these measurements. This
uncertainty is still significantly larger than corresponding constraints e.g.
from QCD fits of $ep$ data [17], but a first step towards such a constraint
from $pp$ data at NNLO, and at least a very important consistency check.
Furthermore, depending on the stiffness of the low-$x$ PDF parametrization,
these same studies indicate a significant potential for contraining the gluon
distribution at very low $x$ at NNLO, in an $x$ region where data constraints
[10] were so far possible only at NLO. More details on these studies will be
given elsewhere [6].

## References

## References

  * [1] ALICE Collaboration, S. Acharya _et al_., _Phys.Rev.D_ 105 (2022) L011103.
  * [2] ALICE Collaboration, S. Acharya _et al_., _JHEP_ 12 (2023) 086.
  * [3] ALICE Collaboration, S. Acharya _et al_., _Phys.Rev.C_ 107 (2023) 064901.
  * [4] CMS Collaboration, A.M. Sirunyan _et al_., _Phys.Lett.B_ 803 (2020) 135328.
  * [5] Y. Yang and A. Geiser, A novel phenomenological approach to total charm cross section measurements at the LHC, PoS EPS-HEP2023 (2024) 367, [arXiv:2311.07523].
  * [6] A. Geiser et al., proceedings of DIS2024, to be published.
  * [7] M. Cacciari, M. Greco and P. Nason, _JHEP_ 9805 (1998) 007.
M. Cacciari _et al_., _JHEP_ 1210 (2012) 137.

  * [8] V. G. Kartvelishvili _et al_., _Phys.Lett.B_ 78 (1978) 615-617.
  * [9] P. M. Nadolsky _et al_., _Phys.Rev.D_ 78 (2008) 013004.
  * [10] PROSA Collaboration, O. Zenaiev _et al_., _Eur.Phys.J.C_ 75 (2015) 396.
PROSA Collaboration, O. Zenaiev _et al_., _JHEP_ 2004 (2020) 118.

  * [11] M.V. Garzelli, L. Kemmler, S. Moch and O. Zenaiev, _JHEP_ 2104 (2021) 043.
  * [12] ALICE Collaboration, S. Acharya _et al_., _Eur.Phys.J.C_ 79 (2019) 388.
  * [13] Heavy Flavor Averaging Group, Y. Amhis _et al_., _Eur.Phys.J.C_ 81 (2021) 226.
  * [14] LHCb Collaboration, R. Aaij _et al_., _JHEP_ 05 (2017) 074.
  * [15] LHCb Collaboration, R. Aaij _et al_., _Phys.Rev.D_ 100 (2019) 031102.
  * [16] LHCb Collaboration, R. Aaij _et al_., _Phys.Rev.Lett._ 124 (2020) 122002.
CMS Collaboration, A. Tumasyan _et al_., _Phys.Rev.Lett._ 131 (2023) 121901.

  * [17] H1 and ZEUS Coll., H. Abramowicz et al., _Eur.Phys.J.C_ 78 (2018) 473.
xFitter Developers Team, V. Bertone et al., _Eur.Phys.J.C_ 77 (2017) 837.
S. Alekhin et al., _Phys.Rev.D_ 96 (2017) 014011.