# REQA: Coarse-to-fine Assessment of Image
Quality to Alleviate the Range Effect

Bingheng Li, and Fushuo Huo B. Li is with the Visual Information Processing
Laboratory, School of Electronic Engineering, Xidian University, Xian 710071,
China and also with the Huawei Technologies Co.Ltd., Hangzhou 310000, China.
(email: bhlee@stu.xidian.edu.cn).F. Huo is with the Department of Computing,
The Hong Kong Polytechnic University, Hong Kong (e-mail:
fushuo.huo@connect.polyu.hk).

###### Abstract

Blind image quality assessment (BIQA) of user generated content (UGC) suffers
from the range effect which indicates that on the overall quality range, mean
opinion score (MOS) and predicted MOS (pMOS) are well correlated; focusing on
a particular range, the correlation is lower. The reason for the range effect
is that the predicted deviations both in a wide range and in a narrow range
destroy the uniformity between MOS and pMOS. To tackle this problem, a novel
method is proposed from coarse-grained metric to fine-grained prediction.
Firstly, we design a rank-and-gradient loss for coarse-grained metric. The
loss keeps the order and grad consistency between pMOS and MOS, thereby
reducing the predicted deviation in a wide range. Secondly, we propose multi-
level tolerance loss to make fine-grained prediction. The loss is constrained
by a decreasing threshold to limite the predicted deviation in narrower and
narrower ranges. Finally, we design a feedback network to conduct the coarse-
to-fine assessment. On the one hand, the network adopts feedback blocks to
process multi-scale distortion features iteratively and on the other hand, it
fuses non-local context feature to the output of each iteration to acquire
more quality-aware feature representation. Experimental results demonstrate
that the proposed method can alleviate the range effect compared to the state-
of-the-art methods effectively.

###### Index Terms:

Blind image quality assessment, range effect, coarse-to-fine assessment,
feedback hierarchy

##  I Introduction

Image quality assessment (IQA) explores how to imitate human beings to
automatically assess image quality. Accurately describing the quality change
has extensive applications in image restoration, image retrieval and image
quality monitoring systems, etc. IQA approaches can be generally divided into
three categories: full-reference IQA approach (FR-IQA), reduced-reference
approach (RR-IQA) and blind IQA approach (BIQA). FR-IQA and RR-IQA measure the
similarity between the distorted image and reference image [3, 18, 20, 21, 22,
23]. However, in most authentic scenarios, it is hard to achieve ideal
reference information. BIQA does not require any reference image as a
prerequisite in predicting the perceptual quality, so it attract great
attention in recent years.

(a) Visualization of the predicted results of DBCNN[56] on CLIVE[30]

(b) Statistics results of the prediction of DBCNN[56] on CLIVE[30]

Figure 1: (a) depicts The range effect as on the overall quality range, MOS
and predicted MOS seem well correlated; focusing on a particular range (e.g.
points within green box), the correlation is lower. (b) illustrates the
statistic results of the scale difference and the distance difference between
pMOS and MOS on the authentic database.

Early BIQA methods[44, 45, 46, 47, 48, 9, 49, 50, 52, 53, 7, 38, 54, 55, 8,
57, 58, 59, 60, 62, 63, 64, 10, 40, 75] mainly focus on synthetically
distorted databases[25, 26, 27, 28, 29], which consists multiple distortions
generated from limited scenarios. Since images captured in the real world
suffer from ever-changing contents and more complicated distortions,
accurately predicting their quality remains a challenge. To deal with this
problem, in recent years, some BIQA methods[56, 65, 67, 69, 70, 71, 72, 73]
have been proposed with exploring novel learning strategies and leveraging
complicated feature representation.

As is illustrated in Fig.1, these approaches are confronted with a common
drawback that they cannot perform well on a narrower quality range, although
achieving relatively high correlations spanning a wide range from extremely
bad to good on challenge datasets[30, 31, 32]. This is illustrated in
Fig.1(a), and is referred as the range effect[33, 1, 2].

How well BIQA methods perform on a narrower quality range is important for
more realistic prediction on user generated content(UGC). According to [34],
the distributions of image quality in the authentically distorted databases is
narrower and peakier as compared to those synthetic ones, as most pictures
captured in the real world are improved by imaging device. According to the
distribution property, streaming media server can screen the images with
extreme qualities to obtain images with moderate qualities. These images will
be integrated as the benchmark of subsequent image enhancement and image-
effect synthesis. Thus, user generated content requires that BIQA model
possesses more fine-grained perception to image quality. That is, it can deal
with the range effect very well.

The ground truth data of the authentically distorted databases is in the form
of the mean opinion score (MOS) that is the average of quality ratings given
by multiple subjects. Based on this, it can be divided into five categorical
scales[11]. We make statistic analysis of the scale difference and the
distance difference between the predicted MOS (pMOS) and MOS. Firstly, as is
shown in the first table of Fig.1(b), quality perceptual deviation from a wide
range brings outliers to a scale, which destroy the prediction performance in
this scale. The reason for this is that the objective BIQA model fails to keep
the prediction sequence in the order as the ground truth sequence possesses.
Secondly, as is shown in the second table of Fig.1(b), the predicted values
can not concentrate in smaller deviation ranges, which is further intensifies
the poor accuracy in the local range. Next, we aims to fix these two problems
responsible for the range effect utilizing coarse-grained metric and fine-
grained prediction respectively.

In response to the first problem, metric learning can be introduced to acquire
the scale membership. The state-of-the-art metric learning methods map the
samples into the multi-dimensional feature vectors, as for clustering or
nearest neighbor classification utilizing a distance function that capture
pair-wised similarity[37][36]. This is effective on BIQA approaches for the
synthetic task[38][39][40]. These methods map the synthetically distorted
images into the feature vectors which represent distortion classification or
distortion level. Based on these degradation, they can accurately metric the
quality differences between pair-wised images. However, in authentic
databases, images possess nonidentical scenes and immeasurable distortions
that increase the difficult to obtain the coherence of feature
distributions[31]. Even it can be achieved between a pair of images, it may
not be quality-aware. In this paper, the proposed method maps a set of images
to supervised scores. Based on these scores, rank-and-gradient metric can be
conducted as for reducing the prediction deviation from a wide range and
further alleviate the order confusion of the predicted quality sequence.

To deal with the inaccuracy problem in smaller deviation ranges, we adopts a
multi-stage prediction strategy. Previous BIQA approaches adopt a one-time
strategy to predict MOS, with a feedforward structure. However, the way
neglects feedback mechanism of human visual system (HVS) as is important for
perceptual learning[41]. Neurologically speaking, feedforward hierarchy
underlies implicit processing for initial vision at a glance, and feedback
connections add details to explicit vision with scrutiny[42]. The same applies
to BIQA that fine-grained cognition of image quality is achieved through a
feedback processing where high-level and low-level features are recurrently
integrated by HVS. Moreover, feedback-based learning approach has been proved
more effective than the commonly employed feedforward paradigm in prediction
tasks[FeedBack2]. In the proposed method, MOS prediction is constantly refined
under multi-level tolerance constraints through a feedback structure. Besides,
coarse-grained metric is fused into the structure as the prior knowledge as it
is easier compared to fine-grained prediction from the perspective of
curriculum learning[68].

Patrick et al.[24] have already proposed a new evaluation criterion aiming at
FR-IQA which is less dependent on the range effect. However, there has been no
way to solve this problem in BIQA yet. To deal with this challenge, we first
propose a novel coarse-to-fine assessment method. The flowchart of the
proposed method is illustrated in Fig.2. Firstly, it takes a set of images as
input. The set contains a predicted image and four anchors sampled from
different quality scales randomly. And then, these images are mapped into
multi-scale distortion features and context features. The former are taken as
the input of feedback blocks and the latter are fused into the outputs of
feedback blocks in each time step. Based on this, coarse-to-fine assessment
acquires local-to-global feature representation which is more quality-
aware[70]. Finally, after feature mapping, coarse-to-fine assessment is
conducted. In the first time step, coarse-grained metric utilizes rank-and-
gradient loss to keep the order and grad of predicted sequence consistent with
the ground data. In subsequent time steps, fine-grained prediction adopts
multi-level tolerance losses to refine the prediction accuracy.

##  II Related Work

In this section, we review BIQA methods for synthetically distorted images and
BIQA methods for authentically distorted images respectively.

###  II-A BIQA for Synthetically Distorted Images

In the previous development, the research of BIQA approaches for synthetic
task mainly follows two kinds of ideas: traditional methods and learning-based
methods.

Commonly traditional BIQA approaches first extract hand-crafted features based
on the empirical analysis and then adopt a regression function to map the
features into quality score. The most famous type methods is the one based on
the characteristics of natural statistical scenes (NSS), such as DIIVINE[44],
BLIINDS-II[45] and BRISQUE[46]. This type of methods assumes that natural
images have certain statistical properties affected by the distortion, which
makes the image look unnatural. Therefore, features can be extracted from
frequency, spatial and wavelet domain based on the statistical properties of
an image to predict its quality score. In addition, there are also some other
methods based on human visual system (HVS), such as NRSL[47], LPSI[48], M3[49]
and RISE[50]. These methods utilize HVS to construct the quality aware
features, assuming that HVS is adapted to the structure information. However,
theses hand-crafted features are time-consuming and meantime lack of
generalization ability due to the diversity of image contents and distortions.

Unlike the traditional BIQA methods, the learning based BIQA approaches
automatically generate the quality-aware features. In the early stage,
CBIQ[51] and CORNIA[52] introduce the codebook feature based learning into
BIQA. These methods first utilize raw image patches extracted from a set of
unlabeled images to learn a dictionary in unsupervised manner, and then encode
the test images on the dictionary to obtain the feature representations for
quality estimation. In the subsequent development, CNN pushes the significant
development of BIQA thanks to its powerful learning ability. Kang et al.[53]
propose a shallow CNN network which performs the mapping progress from a
predicted image to its perceptual quality. WaDIQaM[54] propose a significantly
deeper framework which comprises ten convolutional layers and five pooling
layers for feature extraction, and two fully connected layers for regression.
As these networks grow deeper and wider, they need larger and larger annotated
databases for training. Due to extremely labor-intensive and costly
subjective-experiment, the current IQA databases are too small to meet this
requirement.

To deal with the small sample problem, it needs to explore more effective
learning strategies and leverage more complicated features. Some methods find
a way out in the transfer learning. MEON[55] and DBCNN[56] pretrain a
classification model on a large-scale synthetic databa3se to acquire the
initialized network parameters which are, to some extent, distortion aware.
RankIQA[40] and dipIQ[57] propose a pairwise learning-to-rank (L2R) algorithm.
With it, they can learn to rank image in terms of image distortion. Then they
transfer the prior knowledge learned from ranked images to a traditional CNN.
After fine-tuning it on IQA databases, it can improve the accuracy of IQA.
RRLRIQA[58] make further exploration and model the BIQA as a Markov decision
process to optimize the whole image-quality ordering directly. L2R effectively
improves the performance of these CNN-based methods.

Since human are the ultimate receivers of images, the properties of human
visual system (HVS) should also be modeled in a data-driven manner. Perceptual
error map is learned to guide quality prediction in [59][60] where DeepQA[59]
is designed from FR-IQA methods and BPSQM[60] utilize the U-Net[61] to
generate the similar map of the distorted image for reference. In HVS-Net[62],
visual saliency and just noticeable difference(JND)[81] are taken into account
to acquire the perceptually important features. Meantime, the rank loss is
proposed to penalize the model when the order of its predicted quality scores
is biased against that of the ground truth scores.

Some GAN-based methods have also been developed in the last few years.
H-IQA[63] and RAN4IQA[64] suppose that HVS unveils the mask of distortion and
recreates a hallucinated scene without distortion in mind. In addition,
AIGQA[75] propose an active inference module based on the generative
adversarial network (GAN) to predict the primary content. Since these IQA
methods have achieve great improvement in synthetically distorted databases, a
drawback exists when applied into authentic ones. The reference information is
inevitable utilized in their training stage, which makes them limited in user
generated content (UGC) due to the lack of reference images.

###  II-B BIQA for Authentically Distorted Images

Most BIQA methods focus on the synthetically distorted images, but relatively
few approaches have been proposed on the more challenging problem of authentic
IQA. In recent years, based on the multiple learning strategies, some methods
are proposed to cope with this challenge. BLINDER[65] and DBCNN[56] pretrain a
classification model on a photographically generated classification databases
such as ImageNet[66] to acquire the quality-aware network parameters, which
can help the regression task in IQA databases. MetaIQA[67] adopts model-
agnostic meta-learning (MAML)[14] to learn the prior knowledge among different
synthetic distortions. However, due to the imbalance between the synthetic
distortion and the authentic distortion, the learned knowledge cannot be
generated effectively to the authentically distorted IQA databases, which is
obvious in the experiment results. SFA-IQA[69] and HyperIQA[70] make IQA model
understand the content diversity in authentic databases, the former utilizing
semantic feature aggregation (SFA) to eliminate the impact of image content
variation and the latter utilizing a hyper network architecture to evaluate
the image quality adaptively according to the image content. In NAR-CNN[71],
the authors propose a dual-path network to support IQA from reference image
with similar scene but is not aligned. Considering the fact that an image
receives divergent subjective scores from different human raters, PQR[72] and
DeepRN[73] utilize the distribution of subjective scores to describe image
quality.

In this paper, as for the unanswered range effect in BIQA of UGC, REQA is
proposed to fix it. Compared to the state-of-the-art methods, REQA can
effectively tackle the prediction deviation in narrow ranges on the
authentical databases which contain images close to user generated content.

Figure 2: Flowchart of the proposed REQA. REQA adopts a feedback hierarchy to
realize coarse-to-fine quality assessment within multiple time steps. Here,
rank-and-gradient metric accomplishes coarse-grained assessment and multi-
stage pMOS refinements complete fine-grained assessment. In addition, REQA
integrates image feature representations from two aspects, where the first is
to process multi-scale distortion features through iteration and the second is
the fusion of context features from Transfomer Encoder[15].

##  III Proposed Method

By alleviating the range effect to improve the prediction performance in a
narrow quality range, a novel BIQA method (named as REQA) is proposed. As is
illustrated in Fig.2, REQA is end-to-end, dividing and conquering the BIQA
task to multi-time steps to realize the coarse-to-fine image quality
assessment. REQA is realized by a feedback structure. As for more quality-
aware feature representation, we add a Transformer Encoder[15] in REQA to
obtain the non-local features. Following, we first introduce the coarse-to-
fine assessment and then discuss the net framework from two aspects: the
backbone feedback network for multi-scale distortion perception and the
Transformer Encoder for context understanding.

###  III-A Coarse-to-Fine Assessment for the range Effect in IQA

The proposed method employs a multi-time strategy $t\in\\{1,2,...,T\\}$ where
feedback mechanism controls the progress of BIQA task. In particular, it
captures multi-scale quality-aware features, and then they are integrated and
mapped into the quality score that is finally utilized by each time step to
accomplish the assessment subtask. The whole assessment process is coarse-to-
fine, containing coarse-grained metric and fine-grained prediction.

####  III-A1 Coarse-grained metric

As for coarse-grained metric, the proposed method redesigns the sample
strategy of training data. We randomly sample a mini-batch with $K$ 2-tuples
$B=\\{(I_{k},s_{k})\\}_{k=1}^{K}$ from current training set. Here, $I_{k}$ and
$s_{k}$ are the training images and MOS labels, respectively. This randomly
sampled mini-batch contains training samples from multiple quality scales.
Coarse-grained metric aims to improve the perception of quality difference, so
we group the training samples to form a micro-batch
$B_{\mu}^{m}=\\{(I_{i},s_{i})\\}_{i=1}^{5}$ where a predicted image and four
anchors are sampled from five different quality scales. Here, $m\in[1,M]$
where $M=\lfloor K/5\rfloor$. Compared to the state-of-the-art
ones[56][69][70][72], the sample strategy has two advantages as it can be
adaptive to the metric task and moreover it can fix the small sample problem
in BIQA because of the combinatorial diversity.

The realization of coarse-grained metric is based on the quality ranking and
gradient keeping, which occupies the first-time step of feedback learning as
the prior knowledge of fine-grained prediction. Different from [62], the
proposed method puts forward a loss which can not only metric quality order
but also keep the distance difference of pairwise predicted scores consistent
with the one of respective ground truth scores. For $I_{i}$ and $I_{j}$, rank
loss is first defined as:

$\displaystyle
L_{t=1}^{rank}(i,j)=max(0,\frac{-(s_{i}-s_{j})\times(\mathcal{F}_{t=1}(I_{i};\theta)-\mathcal{F}_{t=1}(I_{j};\theta))}{||s_{i}-s_{j}||_{1}+\sigma})$
(1) $\displaystyle\times(||s_{i}-s_{j}||_{1}+1)$

where, $\theta$ is the network parameters, $\mathcal{F}_{t=1}(I_{i};\theta)$
represents the quality prediction of image $i$ in $t=1$ and $\sigma$ is a
small stability term. In our experiment, we set $\sigma=0.0001$. When the
order of pairwise prediction sequence is consistent with the ground truth
order, rank loss equals to zero, otherwise it is reduced to an absolute loss
as:

$L_{t=1}^{rank}(i,j)=||s_{i}-s_{j}||_{1}\times||\mathcal{F}_{t=1}(I_{i};\theta)-\mathcal{F}_{t=1}(I_{j};\theta)||_{1}$
(2)

The rank loss keeps the order consistency, and gradient loss maintains the
stability of pair-wised quality distance difference as:

$L_{t=1}^{gradient}(i,j)=\Big{|}||s_{i}-s_{j}||_{1}-||\mathcal{F}_{t=1}(I_{i};\theta)-\mathcal{F}_{t=1}(I_{j};\theta)||_{1}\Big{|}$
(3)

Overall, for a micro batch $B_{\mu}^{m}$, coarse-grained loss is defined as:

$L_{t=1}^{Coarset}=\sum_{i=1}^{5}\sum_{j=i+1}^{5}(L_{t=1}^{rank}(i,j)+L_{t=1}^{gradient}(i,j))$
(4)

####  III-A2 Fine-grained prediction

Fine-grained prediction is inspired by the episodic curriculum learning[68],
adopting an easy-to-difficult strategy to realize $t\in[2,T]$ prediction
refinement. Specifically, We set a different threshold for each time step to
compute the loss, and as the feedback information is processed continually,
the threshold is reduced. Overall, fine-grained loss is defined as:

$\begin{array}[]{lr}L_{t}^{Fine}=max(0,||s_{i}-\mathcal{F}_{t_{i}}(I_{i};\theta)||_{1}-l_{t_{i}})\\\\[14.22636pt]
l_{t}>l_{t+1}\geq 0,t<t+1\end{array}$ (5)

where, $L_{t}^{Fine}$ is the loss function in time step $t$, and $l_{t}$
represents the threshold in $t$. The whole loss of the proposed method is
calculated as:

$\begin{array}[]{lr}L=w_{1}L_{t=1}^{Coarset}+w_{2}L_{t=2}^{Fine}+w_{3}L_{t=3}^{Fine}+\cdots+w_{T}L_{t=T}^{Fine}\\\\[8.53581pt]
w_{1}+w_{2}+\cdots+w_{t}=1\end{array}$ (6)

where, in our experiment, we set $T=4$, $l_{2}=5$ , $l_{3}=2.5$ and $l_{4}=0$.
The values of $w_{1},w_{2},w_{3}$ and $w_{4}$ is dynamically adjusted with the
training task going on to satisfy the curriculum learning principle. The set
for $w_{1},w_{2},w_{3}$ and $w_{4}$ is in Section IV-A(3).

###  III-B Feedback Network for multi-scale distortion perception

Figure 3: The overview framework of REQA. It consists of three parts as the
baseline network ResNet50[12], feedback network (FN) and Transformer Encoder
(TE). Here, as the core part of REQA, FN processes multi-scale features
iteratively utilizing feedback blocks. TE provides the non-local
representation for the outputs of feedback blocks in each time step. In
addition, global and local representations are firstly concatenated and then
mapped by multilayer perceptron(MLP-III) into a supervised score to predict
image quality.

Feedback Network (FN) is the core part of the proposed method which controls
the feedback process to perceive image degradation from multi-scale distortion
features continually. The whole network concludes three elements as feature
extraction, time-domain iteration and space-domain integration. The detail of
this module is shown in Fig.3.

####  III-B1 Feature extraction

Multi-scale features contain diverse low-level information, which have been
proved to be effective for IQA[38][70]. Meantime, neurology also proves that
HVS can add details into distortion areas through feedback connection. The
property of HVS contributes to quality perception. Thus, multi-scale features
combined with feedback mechanism have greater gain for IQA. In this paper,
REQA adopts a ResNet50[12] to acquire multi-scale features. Specifically, we
remove global average pooling layers and fully connected layers of ResNet50,
and initialize corresponding network parameters using pretrained model in
ImageNet[66]. Finally, multi-scale features $fm_{1},fm_{2},fm_{3}$ are
extracted from $conv2\\_{10}$, $conv3\\_{12}$, $conv4\\_{18}$ layers
respectively.

####  III-B2 Time-domain iteration

Local distortion perception is completed by three feedback blocks. Each block
takes a single scale feature map and the feedback feature extracted from last
time step as input. Through integrating and correlating these features,
quality-aware representation is got to finish the assessment task of current
step. For feedback block $d$, this iterative process is expressed as:

$\begin{array}[]{l}fo_{d,t}=\mathcal{G}_{d}(fm_{d},fh_{d,t-1};\phi_{d})\\\\[8.53581pt]
d=1,2,3,t=1,2,\cdots,T\end{array}$ (7)

where $fh_{d,t-1}$ is hidden state in time step $t-1$, $fo_{d,t}$ is the
output in time step $t$, $\phi_{d}$ represents the block parameters and
$\mathcal{G}_{d}$ expresses the mapping function.

####  III-B3 Space-domain integration

Space-domain integration is dependent on feedback block whose inner structure
is based on ConvLSTM[13] as illustrated in Fig.4. An LSTM cell uses multiple
gates to control information saving, discarding and merging, and finally uses
hidden states to pass feedback through iterations. We briefly presents the
connections between gates in it as:

$\begin{array}[]{l}fi_{d,t}=\sigma(W_{d,x_{i}}fm_{d}+W_{d,h_{i}}fh_{d,t-1})\\\\[8.53581pt]
fg_{d,t}=\sigma(W_{d,x_{f}}fm_{d}+W_{d,h_{f}}fh_{d,t-1})\\\\[8.53581pt]
fo_{d,t}=\sigma(W_{d,x_{o}}fm_{d}+W_{d,h_{o}}fh_{d,t-1})\\\\[8.53581pt]
\tilde{fc}_{d,t}=tanh(W_{d,x_{c}}fm_{d}+W_{d,h_{c}}fh_{d,t-1})\\\\[8.53581pt]
fc_{d,t}=f_{t}^{d}\circ
fc_{d,t-1}+fi_{d,t}\circ\tilde{fc}_{d,t}\\\\[8.53581pt]
fh_{d,t}=o_{t}^{d}\circ tanh(fc_{d,t})\end{array}$ (8)

where $fi_{d,t}$ is the input gate, $fg_{d,t}$ is the forget gate, $fo_{d,t}$
is the output gate, $fc_{d,t}$ is the memory cell, $\sigma$ is the logistic
sigmoid function, and $W_{*}$ is the weight matrix of conv block. In the
proposed method, according to $d$, we set three conv blocks whose details is
also shown in Figure 4. The output $fo_{d,t}$ is finally mapped into
$\bm{\upsilon}_{d,t}\in R^{1\times 32}$.

Figure 4: Architecture details of feedback block.

###  III-C Transformer Encoder for context understanding.

To some extent, the perception of the local distortion can assess the degree
of image quality degradation effectively. However, there are still some
situations where local distortion cannot be quality-aware. For example,
photographers are used to throw the background out of focus as to improve
visual effect of the foreground. In this case, when we only pay attention to
the background and ignore its correlation with foreground, it is easily
regarded as fuzzy distortion that affect the image quality. Thus, adding the
context information to local feedback features contributes to understanding
image quality comprehensively.

Compared to traditional CNN, Transformer[15] manages to capture long range
interactions thanks to its multi-head attention mechanism, that is, it is
effective to obtain non-local features of an image. Meanwhile, CNN has some
inherent inductive biases, such as translation invariance, scale invariance
and so on, which is not possessed by Transformer. These properties make CNN
suitable for feature extraction. Therefore, in the proposed method, we adopts
$ResNet50+Transformer$ structure to acquire feedforward context feature. The
details of the Transformer Encoder (TE) is illustrated in Fig.3. It consists
of three elements: patch embedding, multi-head attention module and multilayer
perceptron.

####  III-C1 Patch embedding

The original distorted image $I_{i}\in{R}^{H\times W\times 3}$ is processed by
ResNet50 and respective feature map
$fm_{4}\in{R}^{\frac{H}{32}\times\frac{W}{32}\times C}$ is acquired by
$conv5\\_9$. Here, $(H,W)$ is the resolution of $I_{i}$ and $C$ is the number
of channels. Transformer encoder takes one-dimensional vector as input. As for
this, $fm_{4}$ is cut into $N$ patches firstly where each patch
$\mathcal{P}_{n,i}\in{R}^{P\times P\times C}$, $n\in[1,N]$ and $N=H\cdot
W/(32^{2}\cdot P^{2})$. Then, $\mathcal{P}_{n,i}$ is flattened into a vector
$\bm{\eta}_{n,i}\in{R}^{1\times P^{2}C}$ which is mapped into
$\bm{\zeta}_{n,i}\in{R}^{1\times D}$ by a learnable matrix $W_{pe}$ as:

$\bm{\zeta}_{n,i}=\bm{\eta}_{n,i}\cdot W_{pe},W_{pe}\in R^{P^{2}C\times D}$
(9)

where, in the experiment, we set $D=224$. Moreover, to keep positional
information, a standard learnable position embedding
$\bm{\rho}_{n,i}\in{R}^{1\times D}$ is added into $\bm{\zeta}_{n,i}$ to obtain
the embedding $\bm{\varepsilon}_{n,i}$ of $\mathcal{P}_{n,i}$ as:

$\bm{\varepsilon}_{n,i}=(\bm{\zeta}_{n,i}+\bm{\rho}_{n,i})^{T}$ (10)

To realize the perception of the whole quality representation, we concatenate
a trainable token $\bm{x}_{quality}\in{R}^{1\times D}$ to the patch embeddings
$\\{\bm{\varepsilon}_{1,i},\bm{\varepsilon}_{2,i},\cdots,\bm{\varepsilon}_{N,i}\\}$,
similar to BERT’s[16] and ViT’s[17] class token. Finally, the input matrix
${X}_{i}$ is defined as:

${X}_{i}=[\bm{x}_{quality}^{T};\bm{\varepsilon}_{1,i};\bm{\varepsilon}_{2,i};\cdots;\bm{\varepsilon}_{N,i}]^{T}$
(11)

####  III-C2 Multi-head attention

Continuing the traditional $\mathbf{qkv}$ idea[15], self-attention map is
acquired through a weighted sum over all values $V$ of the input matrix
${X}_{i}$. Each element in the weighting matrix $A$ are the pairwise
correlation between two patch representations in ${X}_{i}$ which is calculated
by the dot product of respective query and key. The specific computational
process is as:

$\begin{array}[]{lr}[Q,K,V]=\mathcal{L}(X_{i})W_{sa},W_{sa}\in R^{D\times
3D_{h}}\\\ A=softmax(\frac{QK^{T}}{\sqrt{D_{h}}}),A\in R^{(N+1)\times(N+1)}\\\
\mathcal{S}(X_{i})=AV\end{array}$ (12)

where $\mathcal{L}(\cdot)$ denotes layer normalization and
$\mathcal{S}(\cdot)$ is self-attention operation.

Multi-head attention utilizes multiple self-attention operations to integrate
global context relevance from local patch embeddings. Here, we set $D/D_{h}$
self attentions as:

$\mathcal{M}(X_{i})=\mathcal{L}([\mathcal{S}(X_{1});\mathcal{S}(X_{2});\cdots;\mathcal{S}(X_{D/D_{h}})]W_{ma}+X_{i})$
(13)

where $W_{ma}\in R^{D\times D}$ is the transition matrix.

####  III-C3 Multilayer perceptron

We extract a 1D vector $\xi_{quality}$ from $\mathcal{M}(X_{i})$ which is
corresponding to $\bm{x}_{quality}$. And then MLP-I takes $\xi_{quality}$ as
the input to obtain $\tilde{\xi}_{quality}$. After MLP-II processing
$\tilde{\xi}_{quality}$, $\bm{\upsilon}_{m}$ is computed. Finally, through
feature concatenating and mapping, $\mathcal{F}_{t}(I_{i};\theta)$ is achieved
as:

$\mathcal{F}_{t}(I_{i};\theta)=MLP-
III(\bm{\upsilon}_{m}\oplus\bm{\upsilon}_{1,t}\oplus\bm{\upsilon}_{2,t}\oplus\bm{\upsilon}_{3,t})$
(14)

where $\oplus$ represents concat operation.

##  IV Experiment

In this section, we first introduce the experimental protocols, including
databases, criterion and implementation details. Then we compare REQA with the
state-of-the art BIQA methods from global and local predection performance
respectively. We next implement a series of ablation experiments to verify the
contribution of different components of REQA. Finally, we also present some
visualization samples acquired from local distortion perception module to
verify the effectiveness of the feedback hierarchy.

###  IV-A Experiment Protocols

####  IV-A1 Databases

Three authentically distorted IQA databases including CLIVE[30],
KonIQ-10k[31], and BID[32] are used to evaluate the performance of proposed
method. CLIVE contains 1162 images captured from many diverse mobile devices
under the real-world conditions. KonIQ consists of 10073 images, selected from
10 million YFCC100M entries. The sampling strategy of it concerns the
authenticity of distortions, the diversity of content, and quality-related
indicators. BID comprises 586 images with realistic blur distortions such as
motion blur and defocusing blur, etc.

These databases are constructed based on quality rating, i.e., bad, poor, fair
good, and excellent. Crowdsourcing strategy utilizes subjects to give multiple
quality ratings to single image. These ratings is then rescaled into $[1,100]$
to compensate for the biases of individual evaluations. Here, the higher score
corresponds to the higher quality. Trough averaging them, MOS/DMOS is obtained
as the quality label. In our experiment, because these three databases provide
no original category labels, we divide the ground truth data according to
Absolute Category Rating(ACR)[11] into five equal potions as Excellent
$(80-100)$, Good $(60-80)$, Fair $(40-60)$, Poor $(20-40)$ and Bad $(0-20)$ as
for local prediction performance experiment.

####  IV-A2 Criterion

In our experiments, the Speraman Rank Order Correlation Coeffcient (SROCC) and
the Pearson Linear Correlation Coefficient (PLCC) are applied to evaluate the
performance of BIQA methods. SROCC is to measure the monotonocity between the
ground truth data and the prediction scores. PLCC is to evaluate the linear
correlation between these two. Given $N$ images, the SROCC is defined as:

$SROCC=1-\frac{6\sum_{i=1}^{N}d_{i}^{2}}{N(N^{2}-1)}$ (15)

where, $d_{i}$ is the rank difference between MOS and pMOS of the $i-th$
image. And PLCC is computed as:

$PLCC=\frac{\sum_{i=1}^{N}(s_{i}-\mu_{s_{i}})(\tilde{s}_{i}-\mu_{\tilde{s}_{i}})}{\sqrt{\sum_{i=1}^{N}(s_{i}-\mu_{s_{i}})^{2}}\sqrt{\sum_{i=1}^{N}(\tilde{s}_{i}-\mu_{\tilde{s}_{i}})^{2}}}$
(16)

where $s_{i}$ and $\tilde{s}_{i}$ denotes MOS and pMOS of the $i-th$ image,
and $\mu_{s_{i}}$, $\mu_{\tilde{s}_{i}}$ correspond to the mean of each.

####  IV-A3 Implement Details

We conducted training and testing using Pytorch on a NVIDIA 2080Ti GPU. In the
proposed model, all the training images are randomly cropped to $224\times
224$ pixel patches. The results are obtained from 20 train-test iterations. In
each iteration, we randomly select $80\%$ images for training, and the
remaining $20\%$ for testing, so there is no overlap between the training set
and the test set. We train our model using Adam[5] optimizer with weight decay
$5e-4$ for 40 epochs. Learning rates for the backbone ResNet50 and the other
modules are first set to $2e-5$ and $2e-4$ respectively, and reduced by 10 in
$10th$ epoch, $20th$ epoch, and $30th$ epoch respectively.

In addition, the setting for $w_{1}$, $w_{2}$, $w_{3}$ and $w_{4}$ is
consistent with the episodic curriculum learning[68]. First of all, we
prioritize the assessment tasks corresponding to $w_{1}$, $w_{2}$, $w_{3}$ and
$w_{4}$ in Eq.6 based on the easy-to-difficult strategy. Specifically, coarse-
grained metric occupy the highest priority; meantime, fine-grained loss with
bigger threshold possesses higher priority. In addition, we divide the
training into four stages, with ten epoches for each stage. Finally, the
setting is $w_{1}=0.25$, $w_{2}=0.5$, $w_{3}=0.25/3$, $w_{4}=0.25/3$ in stage
1, $w_{2}=0.5$, $w_{3}=0.25$, $w_{4}=0.25$ in stage 2, $w_{2}=0.25$,
$w_{3}=0.5$, $w_{4}=0.25$ in stage 3 and $w_{1}=0.25$, $w_{2}=0.25$,
$w_{3}=0.25$ and $w_{4}=0.5$ in stage 4.

###  IV-B Global Prediction Performance Experiment

In this section, we first conduct experiments in individual authentically
distorted databases to verify the effectiveness of the proposed method and
then we make statistic significant test to validate the robustness of REQA.

####  IV-B1 Single database evaluations

TABLE I: Global prediction performance on authentically distorted databases CLIVE[30], KonIQ[31] and BID[32].The best two results are highlighted in bold. | CLIVE | BID | KonIQ-10K
---|---|---|---
IQA methods | SROCC | PLCC | SROCC | PLCC | SROCC | PLCC
BRISQUE[46] | 0.608 | 0.629 | 0.562 | 0.593 | 0.665 | 0.681
ILNIQE[78] | 0.432 | 0.508 | 0.516 | 0.554 | 0.507 | 0.523
HOSA[77] | 0.640 | 0.678 | 0.721 | 0.736 | 0.671 | 0.694
BIECON[79] | 0.595 | 0.613 | 0.539 | 0.576 | 0.618 | 0.651
WaDIQAM[54] | 0.671 | 0.680 | 0.725 | 0.742 | 0.797 | 0.805
SFA[69] | 0.812 | 0.833 | 0.826 | 0.840 | 0.856 | 0.872
PQR[72] | 0.857 | 0.882 | 0.775 | 0.794 | 0.880 | 0.884
DBCNN[56] | 0.851 | 0.869 | 0.845 | 0.859 | 0.875 | 0.884
SGDNet[80] | 0.851 | 0.872 | - | - | - | -
MetaIQA[67] | 0.802 | 0.835 | 0.825 | 0.828 | 0.850 | 0.887
HyperIQA[70] | 0.859 | 0.882 | 0.869 | 0.878 | 0.906 | 0.917
AIGQA[75] | 0.751 | 0.761 | - | - | - | -
REQA | 0.865 | 0.880 | 0.874 | 0.886 | 0.904 | 0.916
TABLE II: Statistic significance test results on CLIVE[30]. | MetaIQA | AIGQA | DBCNN | HyperIQA | REQA
---|---|---|---|---|---
MetaIQA[67] | 0 | -1 | -1 | -1 | -1
AIGQA[75] | 1 | 0 | -1 | -1 | -1
DBCNN[56] | 1 | 1 | 0 | -1 | -1
HyperIQA[70] | 1 | 1 | 1 | 0 | 0
REQA | 1 | 1 | 1 | 0 | 0

We compare the proposed REQA with 3 traditional methods and 9 DNN-based
algorithms. The experimental results are exhibited in Tab.I, where the top two
SROCC and PLCC are marked in bold. All the results of the traditional methods
are implemented from the original codes. As for the DNN-based
methods[54][75][69][79][80] , the results are taken from respective papers. In
the last two years, the state-of-the-art methods[56][67][70][72] are raised
and the experimental results of them is reproduced by the source codes
released by their authors.

Compared with the traditional methods, REQA possesses notable advantage.
Particularly in KonIQ, REQA achieves competitive performance, with improvement
about $23\%$ on SROCC and $22\%$ on PLCC. It benefits from the powerful
feature representation ability of DNN which can obtain more diverse
degradation information to perceive the quality of the image in real world
than the traditional methods can do.

When compared with DNN-based algorithms, REQA also achieves promising results
on 3 databases. Specifically, REQA outperforms all the methods on BID. As for
CLIVE, REQA obtains the best result on SROCC, and on PLCC, it gets top two
result. On KonIQ, REQA achieves a slightly lower performance compared with
HyperIQA[70] but it still gets a prominent results about $0.904$ on SROCC and
$0.916$ on PLCC. Overall, the proposed method achieves outstanding improvement
of SROCC in experiments, which suggests that the design for alleviating the
range effect enhances the sensitivity to the change of image quality, thereby
optimizing the order of the predicted scores. This can be further proved in
Section IV-C where more statistic results is listed.

(a) DBCNN[56] (b) MetaIQA[67]

(c) HyperIQA[70] (d) REQA

Figure 5: Visualization of predicted results.

####  IV-B2 Statistical significant test

As for demonstrating that REQA possess greater robustness, we import the
statistic significance test which is conducted to verify the significance of
the predicted results between each two methods. Here we choose PQR[72],
DBCNN[56], HyperIQA[70], MetaIQA[67] and AIGQA[75] as comparison methods where
the former three achieve the state-of-the-art performances and the latter two
adopts up-to-data algorithmic logic to deal with BIQA problem. The SROCC
scores of each methods when training and testing on CLIVE are utilized for the
test. Experimental results are showed in Tab.II. Here, ’1/-1/0’ represents
that the model in row is statistically better than/worse
than/indistinguishable from the model in column with $95\%$ confidence level.
As we can see from Tab. II, REQA is statistically better than most methods.
Compared to HyperIQA, REQA achieves an indistinguishable level. HyperIQA
propose a self-adaptive method to perceive content variation which is
effective to obtain the improvement of sensitivity of image quality change.
Based on this, the model can assign each image with a specific weight in the
training stage. Thus, it can fix the non-convex problem belonging to the
authentic databases. This is the reason for its performance in statistic
significance test.

###  IV-C Local Prediction Performance Experiment

In this section, we conduct the local-prediction performance experiment on
KonIQ. Compared methods includes DBCNN[56], HyperIQA[70] and MetaIQA[67]. The
first two methods achieves the top two performances compared to previous BIQA
methods. MetaIQA adopts a up-to-data training strategy as MAML[14] does. All
the methods utilize same division of the database as to keep the consistency
of test samples. Results from respective best models is evaluated in each
category interval. For further examining the effectiveness of REQA, we also
analyze the prediction bias and the number of outliers statistically.

TABLE III: Local prediction performance on KonIQ[31]. All methods are conducted using identical training and testing protocol. The best results are highlighted in bold. SROCC | Excellent | Good | Fair | Poor | Bad
---|---|---|---|---|---
DBCNN[56] | 0.125 | 0.699 | 0.654 | 0.322 | -
MetaIQA[67] | 0.167 | 0.645 | 0.560 | 0.574 | -
HyperIQA[70] | 0.133 | 0.688 | 0.624 | 0.517 | 0.598
REQA | 0.152 | 0.751 | 0.665 | 0.722 | 0.571
PLCC | Excellent | Good | Fair | Poor | Bad
DBCNN[56] | 0.293 | 0.702 | 0.657 | 0.363 | 0.180
MetaIQA[67] | 0.289 | 0.640 | 0.567 | 0.585 | 0.507
HyperIQA[70] | 0.311 | 0.688 | 0.629 | 0.563 | 0.570
REQA | 0.435 | 0.746 | 0.660 | 0.768 | 0.851
Figure 6: Statistics of prediction deviation on a wide range. Figure 7:
Statistics of prediction deviation in a micro range.

####  IV-C1 Single interval evaluation

As is shown in Tab.III, REQA achieves outstanding performances. Compared with
the three methods, REQA gets best results in $Poor$ range ,$Fair$ range and
$Good$ range. Especially in $Poor$ range, REQA achieves about $20\%$
improvement both on SROCC and PLCC in contrast to HyperIQA[70]. In $Excellent$
range and $Bad$ range, four methods all fails to obtain ideal results.
According to statistic analysis, there are a few samples in these two ranges,
which greatly increases the prediction difficulty as even a outlier leads to
the obvious disturbance on SROCC. In this case, REQA still keeps competitive.
In $Excellent$ range, REQA achieves the best result on PLCC and the second-
best result on SROCC. The same is true in $Bad$ range.

####  IV-C2 Outlier and deviation analysis

To further verify the effectiveness of REQA, we make statistic analysis
between pMOS and the ground truth. As is shown in Fig.5, we depicts the
experimental results for a direct visualization. On the whole, the linear
fitting of the predicted results of REQA is more similar to the directly
proportional function compared to DBCNN[56] and HyperIQA[70]. Compared to
MetaIQA[67], though REQA fails to achieve better linear fitting function, the
distribution of its predicted results is more concentrated. This demonstrates
that the results of REQA possess the best linear correlation with the ground
truth data on a wide range. Moreover, focusing on a particular range (e.g.
points within green box), the results of REQA keep the same property. In
addition, compared to the other methods, the predicted outliers (e.g. points
within green boxes) of REQA is more close to the fitting curve. This means
that REQA can alleviate the prediction deviation from a wide range
effectively.

To further examine this visual analysis, the prediction deviation of different
scales is quantified as is illustrated in Fig.6. Here, x-axis represents the
statistic result and y-axis is the scale difference between pMOS and MOS. It
is obvious that REQA is capable of limiting more images to its original
quality scale and meantime it can reduce the wide range of prediction
deviation as much as possible.

In addition, we also quantify the prediction deviation in micro ranges which
is shown in Fig.7. Here, x-axis refers to the difference range of pMOS and
MOS, and y-axis depicts the number of the samples. Compared to the other
methods, REQA achieves the best performance as there are $718$ samples in
$[0,2.5]$ and the most prediction biases concentrate in the first three
ranges. This means that the predicted scores of REQA possess smaller
fluctuation, and furthermore demonstrates that the multi-level fine-grained
losses is able to perceive minor changes of image quality.

###  IV-D Generalization Ability Test

In order to explore the generalization ability of the proposed model, we run
cross database tests on authentically distorted IQA databases compared with
the other three methods PQR[72], DBCNN[56] and HyperIQA[70].

TABLE IV: SROCC results of the cross-database evaluations. The experiment is conducted by training on one database and testing on the full of another database. | Testing | PQR[72] | DBCNN[56] | HyperIQA[70] | REQA
---|---|---|---|---|---
CLIVE | BID | 0.714 | 0.762 | 0.756 | 0.825
| KonIQ | 0.757 | 0.754 | 0.772 | 0.762
BID | CLIVE | 0.680 | 0.725 | 0.770 | 0.747
| KonIQ | 0.636 | 0.724 | 0.688 | 0.699
KonIQ | CLIVE | 0.770 | 0.755 | 0.785 | 0.772
| BID | 0.755 | 0.816 | 0.819 | 0.833

The experiment is conducted by training on one database and testing on the
full of another database. As we can see in Tab.IV, the proposed method
achieves outstanding generalization ability as it obtains the best results
twice and four top-two results. Especially, compared to HyperIQA[56], REQA
achieves $6.3\%$ improvement on SROCC when trained on CLIVE and tested on BID.
Overall, the performance of REQA is a slightly lower than HyperIQA[70]. The
reason for this is that the self-adaptive strategy makes it effective to
assess the image quality according to content information. In the case that
the scene changes, it can quickly perceive this change. The advantage reminds
us to perfect the proposed method through improve the adaptability in the
future work.

###  IV-E Ablation Study

In this section, we make ablation experiments to verify the contribution of
key constituent parts of REQA. We conduct the ablation experiments on CLIVE.
The training and testing protocols are the same as above.

TABLE V: Ablation Experiments about the performance of each module. Here, we compare the gains of BL, GCMU and LDPM based on the global prediction performance. CLIVE | SROCC | PLCC
---|---|---
REQA w/o FN+TE | 0.830 | 0.854
REQA w/o FN | 0.851 | 0.871
REQA w/o TE | 0.843 | 0.847
REQA | 0.865 | 0.880
TABLE VI: Ablation Experiments about the effectiveness of coarse-grained loss and fine-grained loss. The comparison is conducted based on the local prediction performance. SROCC | Excellent | Good | Fair | Poor | Bad
---|---|---|---|---|---
REQA w/o $L^{Coarset}$ | 0.713 | 0.528 | 0.316 | 0.066 | 0.608
REQA | 0.4393 | 0.620 | 0.504 | 0.511 | 0.674
PLCC | Excellent | Good | Fair | Poor | Bad
REQA w/o $L^{Coarset}$ | 0.768 | 0.525 | 0.384 | 0.143 | 0.707
REQA | 0.0.911 | 0.611 | 0.555 | 0.551 | 0.659

We firstly examine the gains of three modules, the baseline network ResNet50,
FN and TE. The results is listed in Tab.V. The modified model only removes
respective module and the loss function is not changed. Specifically, in model
REQA w/o FN+TE, the baseline network (BL) ResNet50[12] is left to predict the
image quality to get about 0.830 on PLCC and 0.854 on PLCC. Compare to BL,
model REQA w/o TE achieves $10\%$ improvement on SROCC which proves the
effectiveness of feedback module. And in model REQA w/o FN, about $0.851$ on
SROCC and $0.871$ on PLCC is obtained. This demonstrates that global context
feature is quality-aware to improve the assessment performance. Model REQA
achieves the best results, which illustrates that modules interact with each
other to get a positive gain.

Then we analyze the effectiveness of $\bm{L^{Fine}}$ and $\bm{L^{Coarse}}$
which is shown in Tab.VI. According to local performance comparison, we can
conclude that REQA$+\bm{L^{Fine}}$ guarantees the prominent performance of the
algorithm. And $\bm{L^{Coarse}}$ is favorable for REQA to obtain ideal results
in narrow ranges.

Overall, we can make the subsequent conclusions. First of all, different
modules proposed in this paper can improve the performance of REQA. TE
enlarges the receptive field to acquire the context features. FN can obtain
more quality-aware perception of multi-scale features while completing the
task iterations. $\bm{L^{Fine}}$ makes REQA have a prediction accuracy and
with the addition of $\bm{L^{Coarse}}$, local prediction performance gets
promoted. All of these are responsible for the outstanding experimental
results of REQA.

Figure 8: Visualization of the local distortion information in the form of the
heat map.

###  IV-F Visualization of Local Distortion Features

In this section, we visualize the local distortion information in the form of
the heat map[76] to further examine the performance of the feedback hierarchy.
We show each heat map corresponding to the output of $d^{\prime}th$ feedback
block in $t^{\prime}th$ time step in Fig.8. Here, the displayed image
$12949713.jpg$ is from KonIQ[31]. As we can see from the longitudinal
comparison, different feedback blocks possess different receptive fields. For
example, through comparing Fig.8(a), (e) and (i), it can be concluded that the
third feedback block perceives larger areas where the textures is much
clearer. This is attributed to different scales of inputs of three feedback
blocks. Furthermore, by horizontal comparison, it can be concluded that
feedback blocks can obtained more fine-grained feature maps which is
consistent with our standpoint. For example, through making comparisons of
Fig.8 (a)-(d), we can see that heat map in $t=4$ has more heat sensitive areas
(e.g. points within green boxes). In addition, we obtain the pMOS of the image
in each time step as $79.01$ in $t=1$, $82.02$ in $t=2$, $83.16$ in $t=3$, and
$82.13$ in $t=4$. The MOS of it is $84.91$. This means that with the feedback
going on, REQA can make more accurate prediction. Overall, we can conclude
that the feedback hierarchy can achieves better and better features as
iteration goes on. These features are quality-aware enough to ensure REQA
obtain more fine-grained prediction result.

##  V Conclusion

In this paper, we first propose the coarse-to-fine method to tackle the range
effect in BIQA. Benefitting from FN and TE, the proposed method can perceive
the multi-scale distortion information and global context information, which
makes the model quality-aware. Furthermore, fusing coarse-grained metric and
fin-grained losses into the feedback hierarchy to process these features, the
proposed method achieves outstanding global and local prediction performance
as is demonstrated by a series of experimental results.

## References

  * [1] L. Krasula, K. Fliegel, P. Le Callet, and M. Klíma, “On the accuracy of objective image and video quality models: New methodology for performance evaluation,” in _2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX)_ , 2016, pp. 1–6.
  * [2] L. Krasula, P. Le Callet, K. Fliegel, and M. Klíma, “Quality assessment of sharpened images: Challenges, methodology, and objective metrics,” _IEEE Transactions on Image Processing_ , vol. 26, no. 3, pp. 1496–1508, 2017\.
  * [3] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli _et al._ , “Image quality assessment: from error visibility to structural similarity,” _IEEE Transactions on Image Processing_ , vol. 13, no. 4, pp. 600–612, 2004\.
  * [4] Z. Wang and A. C. Bovik, “Mean squared error: Love it or leave it? a new look at signal fidelity measures,” _IEEE signal processing magazine_ , vol. 26, no. 1, pp. 98–117, 2009.
  * [5] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [6] K. Ma, Q. Wu, Z. Wang, Z. Duanmu, H. Yong, H. Li, and L. Zhang, “Group mad competition-a new methodology to compare objective image quality models,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 1664–1673.
  * [7] Q. Wu, H. Li, K. N. Ngan, and K. Ma, “Blind image quality assessment using local consistency aware retriever and uncertainty aware evaluator,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 28, no. 9, pp. 2078–2089, 2018.
  * [8] Q. Jiang, F. Shao, W. Lin, and G. Jiang, “Blique-tmi: Blind quality evaluator for tone-mapped images based on local and global feature analyses,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 2, pp. 323–335, 2019.
  * [9] Q. Wu, H. Li, F. Meng, K. N. Ngan, B. Luo, C. Huang, and B. Zeng, “Blind image quality assessment based on multichannel feature fusion and label transfer,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 26, no. 3, pp. 425–440, 2016.
  * [10] Y. Chen, Y. Zhao, S. Li, W. Zuo, W. Jia, and X. Liu, “Blind quality assessment for cartoon images,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 9, pp. 3282–3288, 2020.
  * [11] R. I.-R. BT, “Methodology for the subjective assessment of the quality of television pictures,” _International Telecommunication Union_ , 2002.
  * [12] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [13] S. Xingjian, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” in _Advances in neural information processing systems_ , 2015, pp. 802–810.
  * [14] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International Conference on Machine Learning_ , 2017, pp. 1126–1135.
  * [15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [16] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [17] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [18] H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” _IEEE Transactions on image processing_ , vol. 15, no. 2, pp. 430–444, 2006\.
  * [19] W. Xue, L. Zhang, X. Mou, and A. C. Bovik, “Gradient magnitude similarity deviation: A highly efficient perceptual image quality index,” _IEEE Transactions on Image Processing_ , vol. 23, no. 2, pp. 684–695, 2013.
  * [20] L. Zhang, L. Zhang, X. Mou, and D. Zhang, “Fsim: A feature similarity index for image quality assessment,” _IEEE transactions on Image Processing_ , vol. 20, no. 8, pp. 2378–2386, 2011.
  * [21] L. Ma, S. Li, F. Zhang, and K. N. Ngan, “Reduced-reference image quality assessment using reorganized dct-based image representation,” _IEEE Transactions on multimedia_ , vol. 13, no. 4, pp. 824–829, 2011.
  * [22] G. Zhai, X. Wu, X. Yang, W. Lin, and W. Zhang, “A psychovisual quality metric in free-energy principle,” _IEEE Transactions on Image Processing_ , vol. 21, no. 1, pp. 41–52, 2011.
  * [23] A. Rehman and Z. Wang, “Reduced-reference image quality assessment by structural similarity estimation,” _IEEE transactions on image processing_ , vol. 21, no. 8, pp. 3378–3389, 2012.
  * [24] L. Krasula, K. Fliegel, P. Le Callet, and M. Klíma, “On the accuracy of objective image and video quality models: New methodology for performance evaluation,” in _2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX)_ , 2016, pp. 1–6.
  * [25] H. R. Sheikh, M. F. Sabir, and A. C. Bovik, “A statistical evaluation of recent full reference image quality assessment algorithms,” _IEEE Transactions on Image Processing_ , vol. 15, no. 11, pp. 3440–3451, 2006.
  * [26] D. Jayaraman, A. Mittal, A. K. Moorthy, and A. C. Bovik, “Objective quality assessment of multiply distorted images,” in _2012 Conference record of the forty sixth asilomar conference on signals, systems and computers (ASILOMAR)_ , 2012, pp. 1693–1697.
  * [27] N. Ponomarenko, L. Jin, O. Ieremeiev, V. Lukin, K. Egiazarian, J. Astola, B. Vozel, K. Chehdi, M. Carli, F. Battisti _et al._ , “Image database tid2013: Peculiarities, results and perspectives,” _Signal Processing: Image Communication_ , vol. 30, pp. 57–77, 2015.
  * [28] E. C. Larson and D. M. Chandler, “Most apparent distortion: full-reference image quality assessment and the role of strategy,” _Journal of Electronic Imaging_ , vol. 19, no. 1, p. 011006, 2010.
  * [29] H. Lin, V. Hosu, and D. Saupe, “Kadid-10k: A large-scale artificially distorted iqa database,” in _2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX)_ , 2019, pp. 1–3.
  * [30] D. Ghadiyaram and A. C. Bovik, “Massive online crowdsourced study of subjective and objective picture quality,” _IEEE Transactions on Image Processing_ , vol. 25, no. 1, pp. 372–387, 2016.
  * [31] V. Hosu, H. Lin, T. Sziranyi, and D. Saupe, “Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 4041–4056, 2020.
  * [32] A. Ciancio, E. A. da Silva, A. Said, R. Samadani, P. Obrador _et al._ , “No-reference blur assessment of digital pictures based on multifeature classifiers,” _IEEE Transactions on image processing_ , vol. 20, no. 1, pp. 64–75, 2010.
  * [33] M. A. Saad, P. Le Callet, and P. Corriveau, “Blind image quality assessment: Unanswered questions and future directions in the light of consumers needs,” _VQEG eLetter_ , vol. 1, no. 2, pp. 62–66, 2014.
  * [34] Z. Ying, H. Niu, P. Gupta, D. Mahajan, D. Ghadiyaram, and A. Bovik, “From patches to pictures (paq-2-piq): Mapping the perceptual space of picture quality,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 3575–3585.
  * [35] M. H. Pinson and S. Wolf, “A new standardized method for objectively measuring video quality,” _IEEE Transactions on broadcasting_ , vol. 50, no. 3, pp. 312–322, 2004.
  * [36] B. Kulis _et al._ , “Metric learning: A survey,” _Foundations and trends in machine learning_ , vol. 5, no. 4, pp. 287–364, 2012.
  * [37] A. Bellet, A. Habrard, and M. Sebban, “Metric learning,” _Synthesis Lectures on Artificial Intelligence and Machine Learning_ , vol. 9, no. 1, pp. 1–151, 2015.
  * [38] F. Gao, Y. Wang, P. Li, M. Tan, J. Yu, and Y. Zhu, “Deepsim: Deep similarity for image quality assessment,” _Neurocomputing_ , vol. 257, pp. 104–114, 2017.
  * [39] E. Prashnani, H. Cai, Y. Mostofi, and P. Sen, “Pieapp: Perceptual image-error assessment through pairwise preference,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1808–1817.
  * [40] X. Liu, J. van de Weijer, and A. D. Bagdanov, “Rankiqa: Learning from rankings for no-reference image quality assessment,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1040–1049.
  * [41] R. L. Goldstone, “Perceptual learning,” _Annual review of psychology_ , vol. 49, no. 1, pp. 585–612, 1998.
  * [42] S. Hochstein and M. Ahissar, “View from the top: Hierarchies and reverse hierarchies in the visual system,” _Neuron_ , vol. 36, no. 5, pp. 791–804, 2002.
  * [43] A. R. Zamir, T.-L. Wu, L. Sun, W. B. Shen, B. E. Shi, J. Malik, and S. Savarese, “Feedback networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 1308–1317.
  * [44] A. K. Moorthy and A. C. Bovik, “Blind image quality assessment: From natural scene statistics to perceptual quality,” _IEEE transactions on Image Processing_ , vol. 20, no. 12, pp. 3350–3364, 2011.
  * [45] M. A. Saad, A. C. Bovik, and C. Charrier, “Blind image quality assessment: A natural scene statistics approach in the dct domain,” _IEEE transactions on Image Processing_ , vol. 21, no. 8, pp. 3339–3352, 2012.
  * [46] A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” _IEEE Transactions on image processing_ , vol. 21, no. 12, pp. 4695–4708, 2012.
  * [47] Q. Li, W. Lin, J. Xu, and Y. Fang, “Blind image quality assessment using statistical structural and luminance features,” _IEEE Transactions on Multimedia_ , vol. 18, no. 12, pp. 2457–2469, 2016.
  * [48] Q. Wu, Z. Wang, and H. Li, “A highly efficient method for blind image quality assessment,” in _2015 IEEE International Conference on Image Processing (ICIP)_ , 2015, pp. 339–343.
  * [49] T. Dai, K. Gu, L. Niu, Y.-b. Zhang, W. Lu, and S.-T. Xia, “Referenceless quality metric of multiply-distorted images based on structural degradation,” _Neurocomputing_ , vol. 290, pp. 185–195, 2018.
  * [50] L. Li, W. Xia, W. Lin, Y. Fang, and S. Wang, “No-reference and robust image sharpness evaluation based on multiscale spatial and spectral features,” _IEEE Transactions on Multimedia_ , vol. 19, no. 5, pp. 1030–1040, 2016.
  * [51] P. Ye and D. Doermann, “No-reference image quality assessment using visual codebooks,” _IEEE Transactions on Image Processing_ , vol. 21, no. 7, pp. 3129–3138, 2012.
  * [52] P. Ye, J. Kumar, L. Kang, and D. Doermann, “Unsupervised feature learning framework for no-reference image quality assessment,” in _2012 IEEE Conference on Computer Vision and Pattern Recognition_ , 2012, pp. 1098–1105.
  * [53] L. Kang, P. Ye, Y. Li, and D. Doermann, “Convolutional neural networks for no-reference image quality assessment,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2014, pp. 1733–1740.
  * [54] S. Bosse, D. Maniry, K.-R. Müller, T. Wiegand, and W. Samek, “Deep neural networks for no-reference and full-reference image quality assessment,” _IEEE Transactions on image processing_ , vol. 27, no. 1, pp. 206–219, 2017\.
  * [55] K. Ma, W. Liu, K. Zhang, Z. Duanmu, Z. Wang, and W. Zuo, “End-to-end blind image quality assessment using deep neural networks,” _IEEE Transactions on Image Processing_ , vol. 27, no. 3, pp. 1202–1213, 2018.
  * [56] W. Zhang, K. Ma, J. Yan, D. Deng, and Z. Wang, “Blind image quality assessment using a deep bilinear convolutional neural network,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 1, pp. 36–47, 2018\.
  * [57] K. Ma, W. Liu, T. Liu, Z. Wang, and D. Tao, “dipiq: Blind image quality assessment by learning-to-rank discriminable image pairs,” _IEEE Transactions on Image Processing_ , vol. 26, no. 8, pp. 3951–3964, 2017.
  * [58] J. Gu, G. Meng, C. Da, S. Xiang, and C. Pan, “No-reference image quality assessment with reinforcement recursive list-wise ranking,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, no. 01, 2019, pp. 8336–8343.
  * [59] J. Kim, A.-D. Nguyen, and S. Lee, “Deep cnn-based blind image quality predictor,” _IEEE transactions on neural networks and learning systems_ , vol. 30, no. 1, pp. 11–24, 2018.
  * [60] D. Pan, P. Shi, M. Hou, Z. Ying, S. Fu, and Y. Zhang, “Blind predicting similar quality map for image quality assessment,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 6373–6382.
  * [61] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_ , 2015, pp. 234–241.
  * [62] S. Seo, S. Ki, and M. Kim, “Deep hvs-iqa net: Human visual system inspired deep image quality assessment networks,” _arXiv preprint arXiv:1902.05316_ , 2019.
  * [63] K.-Y. Lin and G. Wang, “Hallucinated-iqa: No-reference image quality assessment via adversarial learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 732–741.
  * [64] H. Ren, D. Chen, and Y. Wang, “Ran4iqa: Restorative adversarial nets for no-reference image quality assessment,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [65] F. Gao, J. Yu, S. Zhu, Q. Huang, and Q. Tian, “Blind image quality prediction by exploiting multi-level deep representations,” _Pattern Recognition_ , vol. 81, pp. 432–442, 2018.
  * [66] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_ , 2009, pp. 248–255.
  * [67] H. Zhu, L. Li, J. Wu, W. Dong, and G. Shi, “Metaiqa: Deep meta-learning for no-reference image quality assessment,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 14 143–14 152.
  * [68] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in _Proceedings of the 26th annual international conference on machine learning_ , 2009, pp. 41–48.
  * [69] D. Li, T. Jiang, W. Lin, and M. Jiang, “Which has better visual quality: The clear blue sky or a blurry animal?” _IEEE Transactions on Multimedia_ , vol. 21, no. 5, pp. 1221–1234, 2018.
  * [70] S. Su, Q. Yan, Y. Zhu, C. Zhang, X. Ge, J. Sun, and Y. Zhang, “Blindly assess image quality in the wild guided by a self-adaptive hyper network,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 3667–3676.
  * [71] Y. Liang, J. Wang, X. Wan, Y. Gong, and N. Zheng, “Image quality assessment using similar scene as reference,” in _European Conference on Computer Vision_ , 2016, pp. 3–18.
  * [72] H. Zeng, L. Zhang, and A. C. Bovik, “Blind image quality assessment with a probabilistic quality representation,” in _2018 25th IEEE International Conference on Image Processing (ICIP)_. IEEE, 2018, pp. 609–613.
  * [73] D. Varga, D. Saupe, and T. Szirányi, “Deeprn: A content preserving deep architecture for blind image quality assessment,” in _2018 IEEE International Conference on Multimedia and Expo (ICME)_ , 2018, pp. 1–6.
  * [74] K. Ma, Z. Duanmu, Q. Wu, H. Yong, H. Li, and L. Zhang, “Waterloo exploration database: New challenges for image quality assessment models,” _IEEE Transactions on Image Processing_ , vol. 26, no. 2, pp. 1004–1016, 2017.
  * [75] J. Ma, J. Wu, L. Li, W. Dong, X. Xie, G. Shi, and W. Lin, “Blind image quality assessment with active inference,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 3650–3663, 2021.
  * [76] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.
  * [77] J. Xu, P. Ye, Q. Li, H. Du, Y. Liu, and D. Doermann, “Blind image quality assessment based on high order statistics aggregation,” _IEEE Transactions on Image Processing_ , vol. 25, no. 9, pp. 4444–4457, 2016.
  * [78] L. Zhang, L. Zhang, and A. C. Bovik, “A feature-enriched completely blind image quality evaluator,” _IEEE Transactions on Image Processing_ , vol. 24, no. 8, pp. 2579–2591, 2015.
  * [79] J. Kim and S. Lee, “Fully deep blind image quality predictor,” _IEEE Journal of selected topics in signal processing_ , vol. 11, no. 1, pp. 206–220, 2016.
  * [80] S. Yang, Q. Jiang, W. Lin, and Y. Wang, “Sgdnet: An end-to-end saliency-guided deep neural network for no-reference image quality assessment,” in _Proceedings of the 27th ACM International Conference on Multimedia_ , 2019, pp. 1383–1391.
  * [81] S.-H. Bae and M. Kim, “A dct-based total jnd profile for spatiotemporal and foveated masking effects,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 27, no. 6, pp. 1196–1207, 2016.