# Enhancing image quality prediction with self-supervised visual masking

Uğur Çoğalan  0000-0003-1279-6918 MPI InformatikGermany<EMAIL_ADDRESS>,  Mojtaba Bemana  0000-0003-0202-6266 MPI InformatikGermany mbemana@mpi-
inf.mpg.de ,  Hans-Peter Seidel  MPI InformatikGermany<EMAIL_ADDRESS>and  Karol Myszkowski  0000-0002-8505-4141 MPI InformatikGermany karol@mpi-
inf.mpg.de

###### Abstract.

Full-reference image quality metrics (FR-IQMs) aim to measure the visual
differences between a pair of reference and distorted images, with the goal of
accurately predicting human judgments. However, existing FR-IQMs, including
traditional ones like PSNR and SSIM and even perceptual ones such as HDR-VDP,
LPIPS, and DISTS, still fall short in capturing the complexities and nuances
of human perception. In this work, rather than devising a novel IQM model, we
seek to improve upon the perceptual quality of existing FR-IQM methods. We
achieve this by considering visual masking, an important characteristic of the
human visual system that changes its sensitivity to distortions as a function
of local image content. Specifically, for a given FR-IQM metric, we propose to
predict a visual masking model that modulates reference and distorted images
in a way that penalizes the visual errors based on their visibility. Since the
ground truth visual masks are difficult to obtain, we demonstrate how they can
be derived in a self-supervised manner solely based on mean opinion scores
(MOS) collected from an FR-IQM dataset. Our approach results in enhanced FR-
IQM metrics that are more in line with human prediction both visually and
quantitatively.

††submissionid: 477††ccs: Computing methodologies Image quality metric
Figure 1.  We introduce self-supervised visual masking that enhances image
quality prediction for existing quality metrics such as MAE, SSIM, FLIP, and
VGG. Our work is inspired by the well-known characteristic of the Human Visual
System (HVS), visual masking, which results in locally varying sensitivity to
image artifact visibility that reduces with increasing contrast magnitude of
the original image pattern. We found that the learned masking clearly
outperforms its traditional hand-crafted versions and better adapts to
specific distortion patterns while remaining visually interpretable. In the
first two rows, we show the reference and distorted images, while the third
and fourth rows show the error maps as predicted by the original and our mask-
enhanced metric versions. As can be seen, our mask-enhanced metrics better
predict the local distortion visibility by the human observer. In the color
scale, darker indicates lower errors.
Figure 2. Agreement of metric predictions with human judgments. We consider
the classic (MAE and SSIM) and learning-based (LPIPS and DISTS) metrics, and
we compare their prediction to their enhanced versions (E-MAE, E-SSIM,
E-DISTS, and E-LPIPS) using our approach. On the left, we see a situation
where MAE and SSIM favor JPEG-like artifacts over slightly resampled textures.
On the right, we encounter a scenario where LPIPS and DISTS prefer blur over a
subtle color shift. Our extended metric versions are always aligned with human
choice. The images have been extracted from the PIPAL dataset [Gu et al.,
2020].

##  1\. Introduction

Full-Reference Image Quality Metrics (FR-IQMs), which take as an input a pair
of reference and distorted images, play a crucial role in a wide range of
applications in digital image processing, such as image compression and
transmission, as well as in evaluating the rendered content in computer
graphics and vision. They are commonly used as a cost function in optimizing
restoration tasks like denoising, deblurring, and super-resolution [Ding et
al., 2021b]. Consequently, developing FR-IQMs that are accurately reflecting
the visual quality of images in accordance with the characteristics of the
human visual system (HVS) is critical. The most commonly used FR-IQMs for
evaluating image quality are the mean square error (MSE) or mean absolute
error (MAE). While these per-pixel metrics are easy to compute, they assess
image quality regardless of spatial content, leading to false positive
predictions. This can be seen in Fig. 1a, where noise is less noticeable in
textured regions, while MAE predicts uniformly distributed error. Similarly, a
depth-of-field blur is primarily visible on high-contrast fonts Fig. 1b, while
MAE predicts the blur visibility also in smooth gradient regions. Other
classic metrics like SSIM [Wang et al., 2004], while accounting for spatial
content, often result in false positive predictions (the JPEG artifact and
invisible color dithering pattern in Fig. 1c-d, respectively). A recent hand-
crafted metric FLIP [Andersson et al., 2020] is specifically designed to
predict the visual differences in time-sequential image-pair flipping, which
can make it too sensitive for side-by-side image evaluation, e.g., noise less
visible in high-contrast texture (Fig. 1e) and contrast reduction less visible
in already high-contrast regions (Fig. 1f). Recognizing that hand-crafted
image features may not adequately capture the HVS complexity, modern metrics
[Zhang et al., 2018] strive to assess the perceptual dissimilarity between
images by comparing deep features extracted from classification networks
[Simonyan and Zisserman, 2015]. These metrics appear to better account for the
characteristics of the HVS; however, they are designed to generate a single
value per image pair and can not provide correct error localization, as can be
seen in Fig. 1g-h for the non-eccentricity pattern noise and Gaussian blur
examples, respectively.

The objective of this work is not to develop a new perceptual FR-IQM; instead,
we are interested in improving the quality prediction of existing metrics to
align more closely with human judgment (Fig. 2). We also aim to enhance the
accuracy of error map predictions by considering multiple factors such as
image content, distortion levels, and distortion types. By detecting both the
presence and evaluating the magnitude of visible distortion in each pixel, we
aim to ensure that the metric predictions more accurately reflect the
probability of a human observer detecting differences between a pair of
images. In this regard, there have been several efforts toward incorporating
the perceptual aspects of human vision, specifically visual masking [Legge and
Foley, 1980; Foley, 1994; Wilson and Gelb, 1984], into FR-IQM methods [Lubin,
1995; Daly, 1993; Mantiuk et al., 2011, 2021]. In simple words, visual masking
refers to the phenomenon in which certain parts of an image (in our
application: distortions) may be less visible to the viewer due to the
presence of other visual elements in the same image. Visual masking can affect
image quality perception, making some image distortions less visible to the
viewer [Ferwerda et al., 1997; Zeng et al., 2002]. However, existing visual
masking models are typically hand-crafted and struggle to generalize
effectively across various distortion types. Although learning a visual
masking model appears to be a natural solution, the lack of reliable ground
truth data for visual masking makes direct supervision impractical. In this
work, we propose a self-supervised approach to predict visual masking using a
dataset of images featuring a variety of distortions of different magnitudes
whose quality has been evaluated in the mean opinion scores (MOS) experiment
with human subjects [Lin et al., 2019].

Figure 3. Our proposed visual masking approach for enhancing classic metrics
such as MAE and SSIM (left) and learning-based metrics such as DISTS or LPIPS
(right). For classic metrics, the input to our mask predictor network
$\mathcal{F}$ are sRGB images, while for learning-based metrics, the inputs
are the VGG features extracted from the images. We learn the visual masks in a
self-supervised fashion by minimizing the difference between the metric final
score and human scores collected from an FR-IQM dataset.

In summary, our work offers the following contributions:

  * •

We propose a lightweight CNN that generates a mask for a given reference and
distorted input pair. The predicted mask acts as a per-pixel weight and, when
multiplied with the inputs, greatly improves the performance of the existing
FR-IQMs. The incorporation of our learned mask into any FR-IQM is seamless and
demands minimal computational resources. While the CNN is trained specifically
for each metric, it learns a generic masking model capable of identifying
various types of distortions.

  * •

We demonstrate that our masking model can be generalized to deep features and
used as a per-layer feature map weight.

  * •

Our solution significantly enhances the accuracy of quality prediction for FR-
IQMs across various test datasets. Furthermore, it produces per-pixel error
maps that visually align more closely with human perception compared to the
original FR-IQMs.

We will make our code publicly available upon the paper’s acceptance.

##  2\. Previous work

FR-IQMs can be categorized into classical metrics, which perform the
computation directly in the image space, and learning-based metrics, which
leverage deep feature models to assess image quality.

Classic metrics Basic FR-IQMs, such as MSE, RMSE, and MAE, compute the per-
pixel difference to quantify image distortion. While these metrics are
straightforward to calculate, their consistency with human vision is typically
low. Such perceptual consistency can be improved by considering a relative
error, instead of its absolute measure, as in PSNR and the symmetric mean
absolute percentage error (SMAPE) [Vogels et al., 2018]. To account for the
spatial aspects of the HVS, alternative metrics such as SSIM [Wang et al.,
2004] are introduced, which consider image patches and measure local
differences in luminance, contrast, and structural information. SSIM is
further extended to multi-scale MS-SSIM [Wang et al., 2003] and complex
wavelet CW-SSIM [Sampat et al., 2009] versions that capture both global and
local structural information. FSIM [Zhang et al., 2011] decomposes the image
into multiple subbands using Gabor filters and compares subband responses
between the reference and distorted images. By assuming that natural images
have a specific distribution of pixel values, models based on information
theory [Sheikh and Bovik, 2005, 2006] measure the mutual information between
images by comparing their joint histograms and taking into account the
statistical dependencies between neighboring pixels. Classical metrics can
offer either a single overall quality score or a visibility map indicating the
distortion intensity. VDM [Lubin, 1995], VDP [Daly, 1993], and HDR-VDP
[Mantiuk et al., 2011, 2021] measure either the visibility of distortions or
perceived distortions magnitude, or both by considering various visual aspects
such as luminance adaptation, contrast sensitivity, and visual masking. A more
recent metric, FLIP [Andersson et al., 2020], emphasizes color differences,
and it is sensitive to even subtle distortions by emulating flipping between
the compared image pair.

Deep learning-based metrics In recent years, research in FR-IQA has been
placing greater emphasis on perceptual comparisons in deep feature space
rather than image space to enhance the alignment with human judgments.
Prashnani et al. [2018] are among the first to utilize deep feature models
learned from human-labeled data to predict perceptual errors. However, Zhang
et al. [2018] demonstrate that internal image representations from
classification networks can be used for image comparison. They propose the
Perceptual Image Patch Similarity (LPIPS) index, which quantifies image
similarity by measuring the $\ell_{2}$ distances between pre-trained VGG
features. To further improve the correlation with human judgments, they learn
per-channel weights for selected VGG features using their collected perceptual
similarity dataset. Recognizing that simple $\ell_{p}$-norm measures fail to
consider the statistical dependency of errors across different locations, Ding
et al. [2020] introduce the DISTS, which aims to measure the texture and
structure similarity between feature pairs by comparing their global mean,
variance, and correlations in the form of SSIM. Building upon this work,
A-DISTS [Ding et al., 2021a] extended the approach to incorporate local
structure and texture comparisons. Moving away from deterministic point-wise
feature comparisons, DeepWSD [Liao et al., 2022] compares the overall
distributions of features using the Wasserstein distance, a statistical
measure for comparing two distributions. Nevertheless, the majority of the
proposed IQMs metrics are targeted toward producing a single quality score and
are not primarily designed to generate per-pixel error maps. In this regard,
Wolski et al. [2018] employ a custom CNN model trained in a fully supervised
way using coarse user marking data to predict an error visibility map that
highlights the regions where distortions are more likely to be noticeable.

In this work, we extend the classic and deep learning-based metrics by
introducing a learnable component trained on perceptual MOS data in a self-
supervised way. By implicitly analyzing local image content, our model derives
per-pixel maps which mimic visual masking, effectively modeling the visual
significance of distortions.

##  3\. Self-supervised visual masking

In this section, we elaborate on our methodology for perceptually calibrating
the existing FR-IQMs. Given a reference and distorted pair ($X$ and $Y$ ) $\in
R^{H\times W\times C}$, we first learn a visual mask, $M$ $\in R^{H\times
W\times 1}$, which has the same spatial dimensions as the inputs. For
classical metrics (Fig. 3-left), the input $X$ and $Y$ are sRGB images
($C=3$), while for learning-based metrics such as LPIPS or DISTS, the inputs
are the VGG features extracted from the images and $C$ is the number of
channels in a given VGG layer (Fig. 3-right). The predicted mask is then
element-wised multiplied with the $X$ and $Y$ before being fed into an FR
quality metric, $\mathcal{D}$ . Note that the same mask is applied to both the
reference and distorted inputs. For estimating the mask $M$ , we utilize a
lightweight CNN denoted as $\mathcal{F}$ , which takes both $X$ and $Y$ as
input. Mathematically, this can be expressed as:

(1) M$=\mathcal{F}(X,Y)$

It is important to note that the network $\mathcal{F}$ is trained specifically
for a metric $\mathcal{D}$ . In the case of metrics such as LPIPS and DISTS,
we follow their specific architecture and compute a mask for each layer using
a separate $\mathcal{F}$ , and the same mask is applied for all channels in a
given layer (Fig. 3-right). Since we cannot directly supervise the output of
the mask generator network, we adopt a self-supervised approach to train it
using an IQM dataset with a single quality score. The network’s parameters are
optimized by minimizing the $\ell_{2}$ difference between the metric output
value and human scores. Our loss is formulated as follows:

(2) Loss$=\lVert\mathcal{G}(\mathcal{D}(M\odot X,M\odot Y))-q\rVert_{2}^{2}$

Here, $q$ $\in[0,1]$ represents the normalized mean opinion score when
comparing the images $X$ and $Y$ . As the metric response can vary in an
arbitrary range, following a similar approach in Zhang et al. [2018], a small
network $\mathcal{G}$ is jointly trained to map the metric response to the
human ratings. Note that the network $\mathcal{G}$ is not applied during
inference time but rather during the training process.

###  3.1. Training and network details

For training, we use the KADID dataset [Lin et al., 2019], which comprises 81
natural images that have been distorted using 25 types of traditional
distortions, each at five different levels, making roughly 10k training pairs.
Note that we train our mask generator network $\mathcal{F}$ for all the
distortion categories together rather than for one specific category. We find
that a simple CNN with three convolutional layers, each consisting of 64
channels, suffices for successful training. ReLU activation is applied after
each layer, while we use Sigmoid activation for the final layer to keep the
mask values in the range between 0 and 1. Our mapping network $\mathcal{G}$
consists of two 32-channel fully connected (FC) ReLU layers, followed by a
1-channel FC layer with Sigmoid activation. The batch size for training is set
to 4. We employ the Adam optimizer [Kingma and Ba, 2017] with an initial
learning rate of $10^{-4}$ and a weight decay of $10^{-6}$.

Figure 4.  The visual comparisons of distortion visibility maps for
superresolution (upper row) and joint denoising and superresolution (bottom
row) tasks as acquired from the PIPAL dataset. The first two columns present
the reference and distorted images, followed by the respective metric
predictions: MAE, HDR-VDP-2 [Mantiuk et al., 2011], LocVis [Wolski et al.,
2018], FovVideoVDP [Mantiuk et al., 2021], and our E-MAE. As can be seen, the
existing metrics tend to overestimate the distortion visibility. Note that
LocVis and E-MAE have not seen such distortions in their training. Table 1.
Performance comparison of existing FR-IQMs (top part) and their enhanced
versions using our approach (specified by the prefix E in the bottom part) on
three standard IQM datasets. Higher values of SRCC, PLCC, and KRCC indicate
better quality prediction. The first and second best metrics for each dataset
are indicated in bold and underlined, respectively. Additionally, the version
with superior correlation is highlighted in dark gray for each metric.

| CSIQ | TID | PIPAL |
---|---|---|---|---
Metric | PLCC | SRCC | KRCC | PLCC | SRCC | KRCC | PLCC | SRCC | KRCC |
FSIM | 0.900 | 0.913 | 0.740 | 0.847 | 0.789 | 0.611 | 0.651 | 0.617 | 0.441 |
VIF | 0.826 | 0.841 | 0.642 | 0.820 | 0.813 | 0.616 | 0.584 | 0.538 | 0.378 |
HDR-VDP-2 | 0.912 | 0.815 | 0.724 | 0.815 | 0.775 | 0.588 | 0.592 | 0.514 | 0.363 |
PieAPP | 0.827 | 0.840 | 0.653 | 0.832 | 0.849 | 0.652 | 0.729 | 0.709 | 0.521 |
MAE | 0.819 | 0.801 | 0.599 | 0.639 | 0.627 | 0.409 | 0.458 | 0.443 | 0.304 |
E-MAE | 0.871 | 0.917 | 0.738 | 0.857 | 0.863 | 0.673 | 0.597 | 0.606 | 0.429 |
PSNR | 0.851 | 0.837 | 0.645 | 0.726 | 0.714 | 0.540 | 0.468 | 0.456 | 0.314 |
E-PSNR | 0.901 | 0.910 | 0.728 | 0.855 | 0.844 | 0.656 | 0.637 | 0.629 | 0.446 |
SSIM | 0.848 | 0.863 | 0.665 | 0.697 | 0.663 | 0.479 | 0.550 | 0.534 | 0.373 |
E-SSIM | 0.869 | 0.910 | 0.732 | 0.842 | 0.868 | 0.677 | 0.671 | 0.656 | 0.469 |
MS-SSIM | 0.826 | 0.841 | 0.642 | 0.820 | 0.813 | 0.616 | 0.584 | 0.538 | 0.379 |
E-MS-SSIM | 0.862 | 0.895 | 0.709 | 0.806 | 0.825 | 0.621 | 0.642 | 0.634 | 0.453 |
FLIP | 0.731 | 0.724 | 0.527 | 0.591 | 0.537 | 0.413 | 0.498 | 0.442 | 0.306 |
E-FLIP | 0.871 | 0.902 | 0.715 | 0.859 | 0.858 | 0.666 | 0.621 | 0.612 | 0.434 |
VGG | 0.938 | 0.952 | 0.804 | 0.853 | 0.820 | 0.639 | 0.643 | 0.610 | 0.432 |
E-VGG | 0.914 | 0.938 | 0.776 | 0.895 | 0.889 | 0.710 | 0.695 | 0.675 | 0.485 |
LPIPS | 0.944 | 0.929 | 0.769 | 0.803 | 0.756 | 0.568 | 0.640 | 0.598 | 0.424 |
E-LPIPS | 0.922 | 0.933 | 0.771 | 0.884 | 0.876 | 0.689 | 0.705 | 0.678 | 0.490 |
DISTS | 0.947 | 0.947 | 0.796 | 0.839 | 0.811 | 0.619 | 0.645 | 0.626 | 0.445 |
E-DISTS | 0.932 | 0.925 | 0.753 | 0.903 | 0.915 | 0.725 | 0.725 | 0.697 | 0.507 |
DeepWSD | 0.949 | 0.961 | 0.821 | 0.879 | 0.861 | 0.674 | 0.593 | 0.584 | 0.409 |
E-DeepWSD | 0.937 | 0.937 | 0.775 | 0.905 | 0.892 | 0.710 | 0.704 | 0.672 | 0.485 |

##  4\. Results

In this section, we first present our experimental setup, which we use then
for our method evaluation and ablations of different training strategies.

###  4.1. Experimental setup

We employ our visual masking approach to enhance some of the classical metrics
(MAR, PSNR, SSIM, MS-SSIM, and FLIP) and recent learning-based methods (VGG,
LPIPS, DISTS, and DeepWSD). Note for MS-SSIM, we used the same $\mathcal{F}$
across all scales, while the inputs are images at different scales. Moreover,
the metric called VGG is computed by simply taking the $\ell_{1}$ difference
between VGG features for the same layers as originally chosen for LPIPS and
DISTS. We assess the performance of our proposed approach on three well-
established IQM datasets, including CSIQ [Larson and Chandler, 2010], TID2013
[Ponomarenko et al., 2015], and PIPAL [Gu et al., 2020]. The ﬁrst two datasets
mainly consist of synthetic distortions, ranging from 1k to 3k images. On the
other hand, PIPAL is the most comprehensive IQA dataset due to its diverse and
complex distortions, consisting of 23k images. Each reference image in this
dataset was subjected to 116 distortions, including 19 GAN-type distortions.
For evaluation, following Ding et al. [2020], we resize the smaller side
resolution of input images to 224 while maintaining the aspect ratio. For each
dataset, three metrics are used for evaluation: Spearman’s rank correlation
coefficient (SRCC), Pearson linear correlation coefficient (PLCC), and the
Kendall rank correlation coefficient (KRCC). The PLCC measures the accuracy of
the predictions, while the SRCC indicates the monotonicity of the predictions,
and the KRCC measures the ordinal association. When computing the PLCC, we
mapped the metric scores to the MOS values using a four-parameter function
[Ding et al., 2020].

###  4.2. Evaluations

In this section, we present the outcome of the quantitative (agreement with
the MOS data) and qualitative (the quality of error maps) evaluation of our
method. We also analyze the mask content and relate it with perceptual models
of contrast and blur perception. Finally, we analyze the error map prediction
of different distortion levels, and we consider the potential use of our
enhanced E-MAE metric as a loss in an image restoration task.

Quality prediction The experimental results are presented in Tbl. 1, where
with the prefix E, we denote our proposed extension for each specific IQM. Our
extension of traditional metrics, such as MAE, PSNR, SSIM, and FLIP
consistently improves their performance for all datasets. This is remarkable
as those metrics are commonly used, and our simple extension can make their
distortion prediction closer to the human observer. Interestingly, the
enhanced E-MAE and E-PSNR outperform recent learning-based VGG, LPIPS, and
DISTS in the TID dataset while showing a comparable performance for the PIPAL
dataset. Notable improvements are also observed in both datasets for the
recent learning-based metrics (E-VGG, E-LPIPS, E-DIST, and E-DeepWSD),
positioning them at a level comparable to other state-of-the-art IQMs, such as
PieAPP [Prashnani et al., 2018]. The only exception is the case of the small-
scale CSIQ dataset, where the original learning-based metrics achieve high
correlations with the MOS data and leave little space for further
improvements.

Error map prediction In Fig. 1, we show the error maps predicted by various
existing IQMs and their enhanced versions for a set of images featuring
different types of distortions. Note that the error map for VGG is visualized
for the first layer. Moreover, the scores for MAE and VGG metrics exist in an
unbounded range, and following Andersson et al. [2020], we apply a Sigmoid
function to normalize them within the range from zero to one. Fig. 11 presents
more examples for the SSIM and VGG metrics. Additionally, Fig. 4 showcases two
examples where our E-MAE metric achieves more visually accurate error maps
compared to well-established visibility metrics such as HDR-VDP-2 [Mantiuk et
al., 2011], LocVis [Wolski et al., 2018], and FovVideoVDP [Mantiuk et al.,
2021]. Please refer to our supplementary material for more comprehensive
results.

Figure 5. Comparison of our E-MAE metric masks for the noise (fifth row) and
blur (sixth row) distortions as a function of different image contrast
($\times 0.5,\times 1,$ and $\times 2$). In the fourth row, we also show a map
with the human sensitivity to local contrast changes as predicted by a
traditional model of visual contrast masking [Tursun et al., 2019, Eq.4]. In
all cases, darker means less sensitive. However, the scale of our masks that
serve as weights in the E-MAE metric is different than the one for the visual
contrast masking model that directly denotes the hypothetical HVS response to
local distortion contrast.

Mask visualization It is also intriguing to see the learned mask, i.e., the
output of the network $\mathcal{F}$ , and to compare it with a traditional
visual contrast masking model, such as the one used in JPEG2000 compression
[Zeng et al., 2002]. To this end, Fig. 5 presents our masks generated for
noise and blur distortions. We consider the same distortion level and three
levels of image contrast enhancement ($\times 0.5,\times 1,$ and $\times 2$).
In the case of noise distortion, our learned masks predict stronger visual
masking in the high-contrast butterfly and better noise visibility in the out-
of-focus smooth background. Increasing image contrast ($\times 2$) leads to
even stronger visual masking in the butterfly area and the plant behind it.
Reducing image contrast ($\times 0.5$) leads to the inverse effect. Such
behavior is compatible with the visual contrast masking model [Zeng et al.,
2002; Tursun et al., 2019], where due to self-contrast masking, the higher the
contrast of the original signal (e.g., on edges), the stronger the distortion
should be to make it visible. Along a similar line, due to neighborhood
masking, the higher the contrast texture, the stronger the visual masking as
well. In the case of blur distortion, our learned mask predicts its strong
visibility on high-contrast edges. The stronger the image contrast ($\times
2$), the blur visibility improves. Assigning a higher weight by our mask to
high contrast regions agrees with perceptual models of blur detection and
discrimination [Watson and Ahumada, 2011; Sebastian et al., 2015].
Note that we derive each mask taking as an input both the reference and
distorted images; the mask can resolve even per-pixel distortions, as in the
case of noise (Fig. 5), and accordingly informs the E-MAE metric on the
perceptual importance of such distortions. What is also remarkable is that the
HVS might impose contradictory requirements on hand-crafted visual models that
become specific for a given distortion. This is well illustrated in Fig. 5,
where noise can be better masked by strong contrast patterns [Zeng et al.,
2002; Tursun et al., 2019] while blur is actually better revealed by strong
contrast patterns [Watson and Ahumada, 2011]. Our learned E-MAE mask somehow
recognizes the distortion context and reacts as expected by penalizing less
noise distortion in high-contrast and textured regions while penalizing more
blur distortion at high-contrast edges. Interestingly, such local, seemingly
contradictory behavior has been learned solemnly by providing multiple pairs
of reference and distortion images along with the corresponding quality MOS
rating which is just a single number. No annotation on specific distortion
types has been required in our training. Fig. 8 shows further examples that
our learned masking is also informed about contrast masking by texture
[Ferwerda et al., 1997] and the contrast sensitivity function (CSF) [Daly,
1993; Barten, 1999; Wuerger et al., 2020].

Sensitivity to different distortion levels Our predicted mask can effectively
highlight both the presence of a perceived error between two images at each
pixel as well its corresponding magnitude. In this regard, Fig. 9 illustrates
the predicted mask for a set of Monte Carlo-rendered images with a
progressively increased number of samples per pixel (SPP). As can be seen, the
perceived error decreases as the sample count rises, which is better reflected
in our predicted Mask and the E-MAE error map compared to the MAE metric. Note
that the predicted mask correctly reacts to the emergence of new fireflies as
the sample count increases.

Employing the enhanced metric as the loss In this part, we investigate the
benefit of the enhanced IQM metrics in optimizing image restoration
algorithms. To this end, we employ MAE and E-MAE as loss functions for
training image denoising using the state-of-the-art image restoration method,
Restormer [Zamir et al., 2022]. For our training set, we select the images in
the BSD400 dataset [Martin et al., 2001] and introduce synthetic noise to
these images by applying additive white Gaussian noise with a randomly chosen
standard deviation ranging between 0 and 50. Then, we evaluate the trained
models on five benchmark datasets, consistent with the ones used in [Zamir et
al., 2022]. We conduct our evaluation for various noise levels and report the
results in Tbl. 2. We can observe that training with MAE leads to a higher
PSNR value; however, training with E-MAE yields better scores when assessing
with LPIPS and E-MAE metrics. The visual results are provided in Fig. 10.

Table 2. Evaluation of a blind Gaussian denoising task when employing MAE and
our E-MAE as loss functions. We assess the performance on synthetic datasets
created with four distinct noise levels ($\sigma$).

| $\sigma=15$ | $\sigma=25$ | $\sigma=50$ | $\sigma=60$
---|---|---|---|---
Loss | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | E-MAE$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | E-MAE$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | E-MAE$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | E-MAE$\downarrow$
MAE | 34.38 | 0.94 | 0.056 | 0.0341 | 31.92 | 0.90 | 0.091 | 0.0847 | 28.77 | 0.83 | 0.163 | 0.320 | 27.97 | 0.81 | 0.179 | 0.429
E-MAE | 34.05 | 0.94 | 0.057 | 0.0320 | 31.50 | 0.90 | 0.090 | 0.0722 | 28.04 | 0.83 | 0.159 | 0.227 | 27.12 | 0.80 | 0.177 | 0.294

Figure 6. Evaluation of E-MAE training performance using only selected
distortion levels for each distortion category. We measure the SRCC
correlation with the MOS data, and as a reference, we also include the results
of complete training with all distortion levels.

###  4.3. Ablations

We perform a set of ablations to investigate the impact of reduced training
data in terms of distortion levels, reference image number, and distortion
type diversity on the E-MAE metric prediction accuracy.

Distortion levels The first experiment analyzes the importance of
incorporating various distortion levels into our training set. In this regard,
we train our network for the E-MAE metric using only one distortion level per
category, and the results are reported in Fig. 6. Interestingly, for all the
datasets (except PIPAL), an inverse U-shape trend emerged across five
different distortion levels, where we observe the lowest correlation when
training with the minimum and maximum distortion levels (levels 1 and 5).
Conversely, a moderate amount of distortion (level 3) appears to be
sufficiently representative for each distortion category and achieved a
comparable correlation to training with all five levels. This behavior can be
anticipated because, at the lowest and highest distortion levels, the
distortions are either barely visible or strongly visible, leading to the
consistent selection of mostly extreme rating scores. Consequently, when the
network is exclusively exposed to images with one such extreme distortion and
rating levels, it fails to learn to differentiate between them. On the other
hand, at moderate distortion levels where distortions are partially visible or
invisible, the network has a better opportunity to learn masks that behave
differently for varying spatial locations.

Figure 7. Evaluation of E-MAE training performance using different numbers of
the reference images (scenes). Multiple training runs have been performed for
20, 40, and 60 randomly selected scenes from the full set of 81 reference
images. The data points represent the respective SRCC correlation averages
over such runs, while the vertical bars depict the standard deviation.

Dataset size Although we employ a large-scale KADID dataset in our training
(25 distortion types $\times$ 5 distortion levels), the number of reference
images is limited to 81. This ablation aims to investigate the training
performance by even further reducing the number of reference images. To this
end, we perform multiple runs of E-MAE metric training using randomly selected
subsets of 20, 40, and 60 reference images. Fig. 7 presents the SRCC
correlations averaged over multiple runs. The correlation differences between
40, 60, and the full set of 81 reference images are minor. In the case of 20
reference images, the performance is slightly lower and the variance higher,
which indicates that 20 scenes might not be enough to capture image content
variability.

Distortion diversity We investigate the impact of separate E-MAE training on
specific distortion subsets such as noise, blur, combined noise and blur, as
well as the complete KADID dataset. At the test time, we evaluate trained this
way E-MAE versions on noise and blur subsets of the TID dataset, as well as
its complete version. The results, presented in Tbl. 3, reveal that training
solely on the noise category unsurprisingly improves the SRCC correlation
within that category; however, it also enhances the overall correlation for
the TID dataset with respect to the original MAE. Conversely, training
exclusively on blur does not improve the performance within the blur category
itself, as the blur distortion already exhibits a strong correlation (0.934)
for the MAE metric, making any improvement marginal. On the other hand, we
noticed that training with all categories combined significantly improves the
correlation in both the noise and blur categories compared to training with
only noise or blur categories, which can suggest that exposing the network to
a wider range of distortion types enables better generalization.

Table 3.  The SRCC correlation with the MOS data for the E-MAE metric trained with specific distortion categories (noise, blur, noise&blur) and the entire (all) KADID dataset, as indicated in the brackets. The TID dataset is used for testing, where the “Category” columns indicate whether only the noise and blur subsets are considered or the entire dataset.  Metric Category: | noise | blur | all
---|---|---|---
MAE | 0.601 | 0.934 | 0.545
E-MAE (noise) | 0.847 | 0.927 | 0.674
E-MAE (blur) | 0.732 | 0.926 | 0.655
E-MAE (noise & blur) | 0.841 | 0.936 | 0.726
E-MAE (all) | 0.906 | 0.955 | 0.857

##  5\. Limitations

The actual visual contrast masking is the function of the viewing condition
and the display size [Chandler, 2013], which is often considered in the
perceptual quality metrics [Daly, 1993; Mantiuk et al., 2011, 2021; Andersson
et al., 2020] but otherwise mostly ignored. However, the effectiveness of our
visual masking model is limited to the experimental setup where human scores
are obtained in the KADID dataset. As shown in Fig. 8, in the context of the
CSF reproduction, our metric might not be well calibrated for near contrast
threshold stimuli, whose visibility is also affected by the viewing distance
and display conditions. Improving on those aspects, we relegate to future
work.

##  6\. Conclusion

In this paper, we present a new approach towards reducing the notorious gap
between the existing quality metric prediction and the actual distortion
visibility by the human observer. We achieve this by self-supervised training
of a metric-specific network using the existing distortion datasets labeled
with mean option score (MOS). We show that although overall image quality is
rated with a single MOS value in the training data, by securing sufficient
diversity of such training, as detailed in our ablation study, the network can
leverage global MOS into a meaningful per-pixel mask. The mask, through
different weighting of local distortion visibility, helps a given metric to
aggregate such local information into the comprehensive MOS value, as imposed
by the training data. The mask can be learned directly in the image space for
traditional metrics or in the feature space for recent learning-based metrics.
In either case, it is trivial to incorporate into the existing metrics.
Remarkably, our approach improves the performance of commonly used metrics,
such as MAE, PSNR, SSIM, and FLIP on all datasets we tested. The prediction
accuracy of recent learning-based metrics is typically substantially enhanced.

Figure 8. Error map prediction for the MAE and E-MAE metrics along with
learned weighting masks for two perception patterns from [Čadík et al., 2013].
These patterns were specifically designed to investigate various perceptual
phenomena, including contrast sensitivity and contrast masking. In the first
row, the background consists of a high-frequency pattern with increasing
contrast toward the right and a stimulus pattern with decreasing contrast from
bottom to top (which becomes more apparent when zoomed in). In this scenario,
contrast masking is more pronounced with increasing background contrast that
in turn reduces the stimulus visibility, and E-MAE correctly predicts this
effect. The second row presents another example, showing a set of patterns
where their spatial frequencies increase toward the right while their contrast
decreases toward the top. In this case, the learned masking roughly follows an
inverse U-shape characteristic, akin to the contrast sensitivity function
(CSF) [Daly, 1993; Barten, 1999; Wuerger et al., 2020]. Our masking well
approximates the sensitivity drop for high frequencies but tends to suppress
the visibility of low-frequency patterns excessively. In spite of this
drawback, we still find it quite remarkable that the CSF shape becomes
apparent in our learned mask without any explicit training with calibrated
near-threshold CSF data.
Figure 9. Our predicted masks and the resulting distortion maps as produced by
our E-MAE metric for rendered images with varying numbers of samples per pixel
(SPP). We also show the corresponding distortion map produced by the
traditional MAE metric. As can be seen, the noise visibility is overestimated
in the crown region while underestimated in the smooth background area, where
it should be better visible due to limited visual contrast masking by the HVS.
The reference image is rendered using 32k SPP. The test set has been
introduced by Firmino et al. [2022].
Figure 10. Visual results in the image denoising task when employing MAE and
E-MAE as loss functions. Considering that denoiser networks typically reduce
noise through smoothing, our objective was to investigate whether the use of
the E-MAE loss could encourage the network to retain or hallucinate details,
even if they do not precisely match the reference, still their discrepancy
from the ground truth is not perceivable based on the E-MAE metric
predictions. Although the denoised image with the E-MAE loss yields slightly
sharper content or higher contrast, as evident from the comparison of E-MAE
errors, the desired behavior of preserving details is not observed. It appears
that the E-MAE is not sensitive to the level of blur caused by the denoiser
network.
Figure 11. Error map visualizations for the SSIM and VGG metrics along with
their enhanced versions (E-SSIM and E-VGG) and the FovVideoVDP metric [Mantiuk
et al., 2021] for blur (top) and noise (bottom) distortions. Note that the
SSIM values have been remapped to 1-SSIM, where lower values indicate less
visible errors. For the VGG metric, we show the difference map for four
different layers, which are obtained by averaging the difference maps of all
channels within each layer.

## References

  * [1]
  * Andersson et al. [2020] Pontus Andersson, Jim Nilsson, Tomas Akenine-Möller, Magnus Oskarsson, Kalle Åström, and Mark D. Fairchild. 2020.  FLIP: A Difference Evaluator for Alternating Images.  _Proceedings of the ACM on Computer Graphics and Interactive Techniques_ 3, 2 (2020), 15:1–15:23.  https://doi.org/10.1145/3406183
  * Barten [1999] Peter G.J. Barten. 1999\.  _Contrast sensitivity of the human eye and its effects on image quality_.  SPIE – The International Society for Optical Engineering.  https://doi.org/10.1117/3.353254
  * Čadík et al. [2013] Martin Čadík, Robert Herzog, Rafał Mantiuk, Radosław Mantiuk, Karol Myszkowski, and Hans-Peter Seidel. 2013.  Learning to predict localized distortions in rendered images. In _Computer Graphics Forum_ , Vol. 32. Wiley Online Library, 401–410.
  * Chandler [2013] Damon M Chandler. 2013\.  Seven challenges in image quality assessment: past, present, and future research.  _International Scholarly Research Notices_ 2013 (2013).
  * Daly [1993] S. Daly. 1993.  The Visible Differences Predictor: An Algorithm for the Assessment of Image Fidelity. In _Digital Image and Human Vision_. 179–206.
  * Ding et al. [2021a] Keyan Ding, Yi Liu, Xueyi Zou, Shiqi Wang, and Kede Ma. 2021a.  Locally Adaptive Structure and Texture Similarity for Image Quality Assessment. In _Proceedings of the 29th ACM International Conference on Multimedia_. ACM.  https://doi.org/10.1145/3474085.3475419
  * Ding et al. [2020] Keyan Ding, Kede Ma, Shiqi Wang, and Eero P. Simoncelli. 2020\.  Image Quality Assessment: Unifying Structure and Texture Similarity.  _CoRR_ abs/2004.07728 (2020).  arXiv:2004.07728 https://arxiv.org/abs/2004.07728
  * Ding et al. [2021b] Keyan Ding, Kede Ma, Shiqi Wang, and Eero P Simoncelli. 2021b.  Comparison of full-reference image quality models for optimization of image processing systems.  _International Journal of Computer Vision_ 129 (2021), 1258–1281.
  * Ferwerda et al. [1997] James A Ferwerda, Peter Shirley, Sumanta N Pattanaik, and Donald P Greenberg. 1997.  A model of visual masking for computer graphics. In _Proceedings of the 24th annual conference on Computer graphics and interactive techniques_. 143–152.
  * Firmino et al. [2022] Arthur Firmino, Jeppe Revall Frisvad, and Henrik Wann Jensen. 2022.  Progressive Denoising of Monte Carlo Rendered Images. In _Computer Graphics Forum_ , Vol. 41. Wiley Online Library, 1–11.
  * Foley [1994] J.M. Foley. 1994\.  Human luminance pattern-vision mechanisms: masking experiments require a new model.  _J. Opt. Soc. Am. A_ 11, 6 (1994), 1710–19.
  * Gu et al. [2020] Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Jimmy Ren, and Chao Dong. 2020\.  PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration.  arXiv:2007.12142 [eess.IV]
  * Kingma and Ba [2017] Diederik P. Kingma and Jimmy Ba. 2017.  Adam: A Method for Stochastic Optimization.  arXiv:1412.6980 [cs.LG]
  * Larson and Chandler [2010] Eric C. Larson and Damon M. Chandler. 2010.  Most apparent distortion: full-reference image quality assessment and the role of strategy.  _J. Electronic Imaging_ 19 (2010), 011006.
  * Legge and Foley [1980] Gordon E. Legge and John M. Foley. 1980.  Contrast masking in human vision.  _J. Opt. Soc. Am._ 70, 12 (1980), 1458–1471.
  * Liao et al. [2022] Xingran Liao, Baoliang Chen, Hanwei Zhu, Shiqi Wang, Mingliang Zhou, and Sam Kwong. 2022\.  DeepWSD: Projecting Degradations in Perceptual Space to Wasserstein Distance in Deep Feature Space. In _Proceedings of the 30th ACM International Conference on Multimedia_. ACM.  https://doi.org/10.1145/3503161.3548193
  * Lin et al. [2019] Hanhe Lin, Vlad Hosu, and Dietmar Saupe. 2019.  KADID-10k: A Large-scale Artificially Distorted IQA Database. In _2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX)_. 1–3.  https://doi.org/10.1109/QoMEX.2019.8743252
  * Lubin [1995] J. Lubin. 1995.  A visual discrimination model for imaging system design and development. In _Vision models for target detection and recognition_ , Peli E. (Ed.). World Scientific, 245–283.
  * Mantiuk et al. [2011] Rafał Mantiuk, Kil Joong Kim, Allan G Rempel, and Wolfgang Heidrich. 2011.  HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions.  _ACM Transactions on graphics (TOG)_ 30, 4 (2011), 1–14.
  * Mantiuk et al. [2021] Rafał K Mantiuk, Gyorgy Denes, Alexandre Chapiro, Anton Kaplanyan, Gizem Rufo, Romain Bachy, Trisha Lian, and Anjul Patney. 2021\.  FovVideoVDP: A visible difference predictor for wide field-of-view video.  _ACM Transactions on Graphics (TOG)_ 40, 4 (2021), 1–19.
  * Martin et al. [2001] D. Martin, C. Fowlkes, D. Tal, and J. Malik. 2001\.  A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. In _Proc. 8th Int’l Conf. Computer Vision_ , Vol. 2. 416–423.
  * Ponomarenko et al. [2015] Nikolay Ponomarenko, Lina Jin, Oleg Ieremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, and C.-C. Jay Kuo. 2015.  Image database TID2013: Peculiarities, results and perspectives.  _Signal Processing: Image Communication_ 30 (2015), 57–77.  https://doi.org/10.1016/j.image.2014.10.009
  * Prashnani et al. [2018] Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen. 2018\.  Pieapp: Perceptual image-error assessment through pairwise preference. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1808–1817.
  * Sampat et al. [2009] Mehul P. Sampat, Zhou Wang, Shalini Gupta, Alan Conrad Bovik, and Mia K. Markey. 2009.  Complex Wavelet Structural Similarity: A New Image Similarity Index.  _IEEE Transactions on Image Processing_ 18, 11 (2009), 2385–2401.
  * Sebastian et al. [2015] Stephen Sebastian, Johannes Burge, and Wilson S. Geisler. 2015\.  Defocus blur discrimination in natural images with natural optics.  _Journal of Vision_ 15, 5 (04 2015), 16–16.
  * Sheikh and Bovik [2006] H.R. Sheikh and A.C. Bovik. 2006.  Image information and visual quality.  _IEEE Transactions on Image Processing_ 15, 2 (2006), 430–444.  https://doi.org/10.1109/TIP.2005.859378
  * Sheikh and Bovik [2005] Hamid R Sheikh and Alan C Bovik. 2005.  Information embedded in images: Joint estimation of information and distortion. In _Proceedings of the IEEE International Conference on Image Processing_ , Vol. 2. IEEE, II–702.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman. 2015.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  arXiv:1409.1556 [cs.CV]
  * Tursun et al. [2019] Cara Tursun, Elena Arabadzhiyska-Koleva, Marek Wernikowski, Radosław Mantiuk, Hans-Peter Seidel, Karol Myszkowski, and Piotr Didyk. 2019\.  Luminance-Contrast-Aware Foveated Rendering.  _ACM Trans. Graph._ 38, 4, Article 98 (2019).
  * Vogels et al. [2018] Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018\.  Denoising with kernel prediction and asymmetric loss functions.  _ACM Transactions on Graphics (TOG)_ 37, 4 (2018), 1–15.
  * Wang et al. [2004] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. 2004\.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ 13, 4 (2004), 600–612.  https://doi.org/10.1109/TIP.2003.819861
  * Wang et al. [2003] Z. Wang, E.P. Simoncelli, and A.C. Bovik. 2003.  Multiscale structural similarity for image quality assessment. In _The Thrity-Seventh Asilomar Conference on Signals, Systems I & Computers, 2003_, Vol. 2. 1398–1402 Vol.2.  https://doi.org/10.1109/ACSSC.2003.1292216
  * Watson and Ahumada [2011] Andrew B. Watson and Albert J. Ahumada. 2011.  Blur clarified: A review and synthesis of blur discrimination.  _Journal of Vision_ 11, 5 (09 2011), 10–10.
  * Wilson and Gelb [1984] Hugh R. Wilson and Douglas J. Gelb. 1984.  Modified line-element theory for spatial-frequency and width discrimination.  _J. Opt. Soc. Am. A_ 1, 1 (1984), 124–131.
  * Wolski et al. [2018] Krzysztof Wolski, Daniele Giunchi, Nanyang Ye, Piotr Didyk, Karol Myszkowski, Radosław Mantiuk, Hans-Peter Seidel, Anthony Steed, and Rafał K Mantiuk. 2018.  Dataset and metrics for predicting local visible differences.  _ACM Transactions on Graphics (TOG)_ 37, 5 (2018), 1–14.
  * Wuerger et al. [2020] Sophie Wuerger, Maliha Ashraf, Minjung Kim, Jasna Martinovic, María Pérez-Ortiz, and Rafał K. Mantiuk. 2020.  Spatio-chromatic contrast sensitivity under mesopic and photopic light levels.  _Journal of Vision_ 20, 4 (04 2020), 23–23.  https://doi.org/10.1167/jov.20.4.23
  * Zamir et al. [2022] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. 2022.  Restormer: Efficient transformer for high-resolution image restoration. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5728–5739.
  * Zeng et al. [2002] Wenjun Zeng, Scott Daly, and Shawmin Lei. 2002.  An overview of the visual optimization tools in JPEG 2000.  _Signal Processing: Image Communication_ 17, 1 (2002), 85–104.
  * Zhang et al. [2011] Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. 2011\.  FSIM: A Feature Similarity Index for Image Quality Assessment.  _IEEE Transactions on Image Processing_ 20, 8 (2011), 2378–2386.  https://doi.org/10.1109/TIP.2011.2109730
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In _CVPR_.