# Debiasing Masks:
A New Framework for Shortcut Mitigation in NLU

Johannes Mario Meissner†, Saku Sugawara‡, Akiko Aizawa†‡
†The University of Tokyo, ‡National Institute of Informatics
<EMAIL_ADDRESS>
###### Abstract

Debiasing language models from unwanted behaviors in Natural Language
Understanding tasks is a topic with rapidly increasing interest in the NLP
community. Spurious statistical correlations in the data allow models to
perform shortcuts and avoid uncovering more advanced and desirable linguistic
features. A multitude of effective debiasing approaches has been proposed, but
flexibility remains a major issue. For the most part, models must be retrained
to find a new set of weights with debiased behavior. We propose a new
debiasing method in which we identify debiased pruning masks that can be
applied to a finetuned model. This enables the selective and conditional
application of debiasing behaviors. We assume that bias is caused by a certain
subset of weights in the network; our method is, in essence, a mask search to
identify and remove biased weights. Our masks show equivalent or superior
performance to the standard counterparts, while offering important benefits.
Pruning masks can be stored with high efficiency in memory, and it becomes
possible to switch among several debiasing behaviors (or revert back to the
original biased model) at inference time. Finally, it opens the doors to
further research on how biases are acquired by studying the generated masks.
For example, we observed that the early layers and attention heads were pruned
more aggressively, possibly hinting towards the location in which biases may
be encoded.

##  1 Introduction

The issue of spurious correlations in natural language understanding datasets
has been extensively studied in recent years Gururangan et al. (2018); McCoy
et al. (2019); Gardner et al. (2021). In the MNLI dataset Williams et al.
(2018), negation words such as “not” are unintended hints for the
contradiction label Gururangan et al. (2018), while a high word overlap
between the premise and the hypothesis often correlates with entailment McCoy
et al. (2019).

Figure 1: We find masks that remove bias from a finetuned model by using
frozen-weight movement pruning.

A common way to prevent the acquisition of biases (in this paper defined as
unintended shortcut behaviors) is to adopt a debiasing loss function, such as
product-of-experts Hinton (2002), to discourage learning shortcuts through the
use of the annotated level of bias in each sample. Because manual bias
annotation is difficult and expensive, the predictions of a biased model are
used as bias annotations Clark et al. (2019). Commonly, models must be
retrained to achieve debiased behavior; this limits our ability to target
different biases separately, as well as choose varied levels of debiasing
strength (which impacts the in- vs. out-of-distribution trade-off).

We propose a new debiasing framework that focuses on removing bias from an
existing model, instead of the now-common approach of re-training from
scratch. Our approach is depicted in Figure 1. We find a pruning mask that
zeroes out the weights that cause biased outputs, producing the desired effect
without altering the original model. This approach offers several clear
advantages. First, pruning masks can be stored very efficiently, only
occupying a fraction of the original model size. This enables creating
multiple masks for varied debiasing behaviors. Secondly, masks can be
effortlessly set or unset at inference time, as opposed to replacing the
entire model. This allows for flexible application, as well as easy reversion
to the original model when needed. Finally, re-framing the debiasing process
as a mask search opens the doors towards future analysis directions, helping
to more deeply understand how biases are learned, and how they can be
eliminated.

##  2 Related Work

###  2.1 Shortcuts in NLU

The study of shortcut solutions in machine learning Geirhos et al. (2020);
D’Amour et al. (2020) has gained attention in recent years, including in the
field of Natural Language Understanding. In SQuAD (Rajpurkar et al., 2016),
Jia and Liang (2017) show that distractor sentences can be inserted in such a
way that the frequently used spurious features mislead the model’s answer. In
MNLI, Gardner et al. (2021) discuss the idea that all simple feature-label
correlations should be regarded as spurious. Their results go in line with
Gururangan et al. (2018), who show that models are able to achieve strong
performance just by training on the hypothesis, a scenario where the desirable
advanced features are not available. Instead, simple word correlations are
used to make the prediction. Finally, other kinds of features such as lexical
overlap have been pointed out as important shortcuts (McCoy et al., 2019).

###  2.2 Debiasing

A wide range of debiasing approaches have been proposed to alleviate shortcut
behavior. For example, perturbations in the model’s embedding space can
encourage robustness to shortcuts Liu et al. (2020), and training on
adversarial data Wang and Bansal (2018) has important benefits for
generalization capabilities too. Removing a certain subset in the training
data (filtering) can help to avoid learning spurious correlations. Bras et al.
(2020) devise an algorithm that selects samples to filter out, reducing the
training time and robustness of the target model.

We will focus our efforts on a family of approaches that rely on a debiasing
loss function to discourage biased learning, instead of altering the original
training data. A wide range of debiasing losses have been proposed to
discourage shortcut learning. They all rely on having a measure of the level
of bias for each training sample, which is commonly obtained via a biased
model’s predictions. Among the most common are product-of-experts and focal
loss Schuster et al. (2019). Other approaches introduce a higher level of
complexity, such as confidence regularization Utama et al. (2020a) requiring a
teacher model, or learned-mixin Clark et al. (2019) introducing an additional
noise parameter.

###  2.3 Pruning

Pruning consists in masking out or completely removing weights in a neural
network, such that they no longer contribute to the output. Common goals
include reducing the model size or achieving an inference speed-up.

Magnitude pruning is a basic pruning method that removes weights based on
their absolute value. It has proven to be an effective method to reduce model
size without compromising on performance. Gordon et al. (2020) apply this
technique on transformer language models. Movement pruning, on the other hand,
was proposed by Sanh et al. (2020), and involves training a score value
alongside each weight. Scores are updated as part of the optimization process,
and weights with an associated score below the threshold are masked out.

Zhao et al. (2020) introduce an alternative to the usual finetuning process by
finding a mask for the pretrained base model such that performance on the
target task increases. The same base model can be used with several masks to
perform multiple tasks. They show that this approach achieves similar
performance to standard finetuning on the GLUE tasks Wang et al. (2018).

Our work is inspired by Zhao et al. (2020); but we focus on removing biases
from an already finetuned model. We will refer to this method as a mask
search. It benefits from the same advantages: masks can be easily applied and
removed from a shared base model, while additionally offering storage
improvements.

##  3 Masked Debiasing

Our proposed approach to debiasing takes a unique point of view in the
debiasing field by assuming that biased behavior is encoded in specific
weights of the network and can be removed without altering the remaining
weights. Thus, we perform a mask search to identify and remove those weights,
experiencing debiased behavior in the resulting model. Our approach comes
together by combining a debiasing loss, a biased model, and a score-based
pruning technique.

#### Debiasing Loss

Among the debiasing losses mentioned in Section 2.2, none appear to be clearly
superior, each offering certain strengths and weaknesses, and striking a
balance between in- and out-of-distribution performance. We err on the side of
simplicity and run our experiments with the focal loss. We follow Clark et al.
(2019) for the implementation.

Figure 2: Pruning rate for each module in one of our debiasing masks.

#### Bias Model

To utilize a debiasing loss, we must obtain the predictions of a biased model
on the training samples. We utilize weak learners, following the nomenclature
introduced by Sanh et al. (2021); they do not require making any assumptions
on the underlying biases. While our method works with any debiasing loss and
bias level source, we choose this setting due to its flexibility and
adaptability to new scenarios.

To date, two major learners have been proposed: undertrained Utama et al.
(2020b) and underparameterized Sanh et al. (2021) learners. We train and
extract the predictions of both of them for comparison. Undertrained learners
are trained by selecting a very small subset of the training data, but keeping
the target model’s architecture and hyperparameters. This effectively
translates into a model that overfits on the selected data subset. Features
uncovered by the model in this manner are deemed spurious, as it is assumed
that more advanced features require exploring larger quantities of data. We
used 2000 samples for MNLI, and 500 samples for the other datasets.
Underparameterized learners, on the other hand, restrict the model complexity
by reducing layer size and count. The expectation is that this
underparameterization is restrictive and limits the ability to find complex
features. We use BERT-Tiny Turc et al. (2019), a BERT model with 2 layers and
an inner size of 128.

#### Frozen Movement Pruning

Our approach is similar to Zhao et al. (2020); but we implement it by
leveraging the framework provided by Sanh et al. (2020), with the addition of
weight-freezing. We use unstructured pruning, which means that each individual
weight is considered for exclusion. Further, we utilize a threshold approach:
in each weight tensor we remove (mask out) those weights with an associated
score that is lower than the threshold. A regularization term for the scores
helps avoid all scores growing larger than the threshold. For further details,
we refer the reader to Sanh et al. (2020). As the starting point, we load a
model already finetuned on the target task.

##  4 Experimental Setup

We compare our approach against the standard approach of re-training the model
with a debiasing loss. Our experiments are carried out with BERT-Base Devlin
et al. (2019).

| MNLI | QQP | FEVER
---|---|---|---
| Val. | HANS | Neg. | Val. | PAWS | PAWS ($\neg$) | Val. | Symm.
Baseline | 84.38 | 61.97 | 78.47 | 92.07 | 44.28 | 23.37 | 85.67 | 64.08
Masking w/o Debiasing | 83.15 | 64.16 | 77.87 | 90.91 | 37.61 | 16.38 | 84.71 | 63.26
Undertrained SD | 81.02 | 67.06 | 74.82 | – | – | – | 86.54 | 65.06
Underparam. SD | 83.03 | 67.75 | 72.36 | 88.79 | 43.50 | 27.16 | 85.70 | 65.79
Undertrained MD (Ours) | 81.85 | 68.69 | 75.58 | – | – | – | 84.55 | 64.85
Underparam. MD (Ours) | 82.24 | 67.86 | 73.95 | 89.61 | 44.34 | 28.56 | 84.96 | 63.37
HypOnly MD (Ours) | 82.87 | 64.26 | 79.03 | – | – | – | – | –

Table 1: Our experimental results comparing a BERT-Base baseline against
several debiasing methods, along with our own mask debiasing approach. SD
stands for standard debiasing, while MD stands for masked debiasing. MNLI Neg.
is our split of the MNLI development set containing at least one negation word
in the hypothesis. PAWS ($\neg$) indicates the subset of PAWS with a negative
label (the anti-bias set for lexical overlap in QQP).

###  4.1 Tasks and Evaluation Scenarios

We perform experiments on three tasks in NLU, and consider several biases and
evaluation datasets.

#### Natural Language Inference (NLI)

This task consists in classifying whether the relationship between a pair of
sentences (premise and hypothesis) is entailed, neutral or contradicted. We
evaluate on two well-known biases: lexical overlap bias McCoy et al. (2019)
and negation words bias Gururangan et al. (2018). We train on MNLI, and
evaluate on HANS McCoy et al. (2019) and our own negation-words subset. HANS
can be used to evaluate the lexical overlap bias. Additionally, we create our
own negation-words anti-bias set by selecting entailed samples from the MNLI
validation set that contain negation words in the hypothesis.111We select
entailed samples with at least one of the following words in the hypothesis:
no, not, don’t, none, nothing, never, aren’t, isn’t, weren’t, neither, don’t,
didn’t, doesn’t, cannot, hasn’t won’t.

#### Paraphrase Identification

In this task the goal is to identify whether a pair of sentences are
paraphrasing each other or not. We train on QQP (Quora Question
Pairs)222https://www.kaggle.com/c/quora-question-pairs, and evaluate on PAWS
Zhang et al. (2019), which tests the model’s reliability on the lexical
overlap bias.

#### Fact Verification

We train on FEVER Thorne et al. (2018), and evaluate on FEVER Symmetric
Schuster et al. (2019). The task setup is similar to NLI; we classify a claim-
evidence pair as either support, refutes or not-enough-information. FEVER
Symmetric eliminates claim-only biases (clues in the claim that allow to guess
the label), among which are negation words, in a similar fashion to MNLI.

###  4.2 Hyperparameters and Reproducibility

We aim for complete reproducibility by providing complete code and clear
reproduction instructions in our
repository.333https://github.com/mariomeissner/shortcut-pruning In most cases,
we follow the configurations indicated by the respective original papers.
Appendix B is additionally provided for a report of important hyperparameters.

We run all of our experiments on five seeds. We not only seed the debiasing
masks, but also the accompanying weak models and base models too.

##  5 Results

We compare our approach against our reproduction of the two weak model
approaches, and compile our results in Table 1.

###  5.1 Masked Debiasing is Effective

First, we observe that our debiasing masks are very effective, surpassing the
performance of their standard debiasing counterparts in two out of the three
evaluated tasks, while keeping a competitive in-distribution performance. In
the FEVER task, our masks provided a slight debiasing effect, but did not beat
the standard debiasing baseline. We do not report undertrained debiasing
results in the QQP setting due to their failure to converge to adequate
results (in both standard and masked debiasing).

To confirm that the combination of masking with a debiasing loss is necessary
to achieve our results, we provide an ablation experiment (Masking w/o
Debiasing) where the debiasing loss is replaced with the standard cross-
entropy loss, while using the same pruning parameters. Results suggest that
pruning alone is not able to achieve the same debiasing effect.

###  5.2 Masks are Diverse

As a means to showcase the capacity of debiasing masks to offer different
debiasing behaviors, we refer to each method’s capacity to mitigate the
negation bias, evaluated by our negation-words subset.

As the standard debiasing results reveal, both weak model approaches were
unable to improve baseline performance on our subset. Therefore, we used the
hypothesis-only bias model, as provided by Utama et al. (2020a). Negation
words are strongly correlated with the hypothesis-only bias; accordingly,
observe a performance improvement on our subset when leveraging this debiasing
target.

##  6 Mask Analysis

Our pruning approach uses a score threshold, which allows for a dynamic
pruning rate across modules. To gain a better understanding on our masks, we
study the generated density distribution. Specifically, we study the rate at
which weights were removed in different network regions. In each layer, we
obtain the pruning rate of the attention layer’s query, key, value, and output
modules, as well as the two linear modules that follow. We plot the results in
Figure 2.

We make two general observations. First, pruning rates are relatively higher
in the early layers. It is known that the last few layers are very task
specific Tenney et al. (2019), which likely implies that their modification
more directly impacts performance. Thus, our masks may be targeting early
layers as a means to remove biases without causing a noticeable performance
decrease. Secondly, the attention modules are more heavily pruned than the
linear layers, which could suggest that attention heads play an important role
in bias encoding.

##  7 Conclusion

We conclude that debiasing masks are an effective approach to mitigating
shortcuts. Alongside providing surprisingly well-performing debiased
behaviors, masks allow to shift the way we think about debiasing: no longer
should biased and unbiased models be treated as two separate models; rather,
it becomes possible to “remove” biases by simply eliminating certain weights
from the network.

## Limitations

An important limitation of this approach is that we found it necessary to
follow Sanh et al. (2020) and run the movement pruning technique for 12
epochs, requiring longer training time. We hypothesize that in future work, it
could become possible to drastically reduce training time for this approach.

Further, in this short paper we explored three tasks in NLU with a fixed
pruning configuration, but it would be beneficial to explore its applicability
to other domains, combine it with other debiasing approaches, or explore more
varied configurations, such as structured pruning methods for improved
inference efficiency.

## Acknowledgments

The education period leading to this project has received funding from “la
Caixa” Foundation (ID 100010434), under agreement LCF/BQ/AA19/11720042. This
work was also supported by JST PRESTO Grant Number JPMJPR20C and JSPS KAKENHI
Grant Number 21H03502.

## References

  * Bras et al. (2020) Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. 2020.  Adversarial filters of dataset biases.
  * Clark et al. (2019) Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019.  Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4069–4082, Hong Kong, China. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * D’Amour et al. (2020) Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. 2020.  Underspecification presents challenges for credibility in modern machine learning.  _Journal of Machine Learning Research_.
  * Gardner et al. (2021) Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and Noah A. Smith. 2021.  Competency problems: On finding and removing artifacts in language data.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1801–1813, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Geirhos et al. (2020) Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020.  Shortcut learning in deep neural networks.  _Nature Machine Intelligence_ , 2(11):665–673.
  * Gordon et al. (2020) Mitchell Gordon, Kevin Duh, and Nicholas Andrews. 2020.  Compressing BERT: Studying the effects of weight pruning on transfer learning.  In _Proceedings of the 5th Workshop on Representation Learning for NLP_ , pages 143–155, Online. Association for Computational Linguistics.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hinton (2002) Geoffrey E Hinton. 2002.  Training products of experts by minimizing contrastive divergence.  _Neural computation_ , 14(8):1771–1800.
  * Jia and Liang (2017) Robin Jia and Percy Liang. 2017.  Adversarial examples for evaluating reading comprehension systems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics.
  * Liu et al. (2020) Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. 2020.  Adversarial training for large neural language models.  _arXiv:2004.08994_.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Sanh et al. (2021) Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M Rush. 2021.  Learning from others’ mistakes: Avoiding dataset biases without modeling them.  In _International Conference on Learning Representations_.
  * Sanh et al. (2020) Victor Sanh, Thomas Wolf, and Alexander M. Rush. 2020.  Movement pruning: Adaptive sparsity by fine-tuning.  NIPS’20, Red Hook, NY, USA. Curran Associates Inc.
  * Schuster et al. (2019) Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019.  Towards debiasing fact verification models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3419–3425, Hong Kong, China. Association for Computational Linguistics.
  * Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.  BERT rediscovers the classical NLP pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4593–4601, Florence, Italy. Association for Computational Linguistics.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Well-read students learn better: The impact of student initialization on knowledge distillation.  _CoRR_ , abs/1908.08962.
  * Utama et al. (2021) Prasetya Utama, Nafise Sadat Moosavi, Victor Sanh, and Iryna Gurevych. 2021.  Avoiding inference heuristics in few-shot prompt-based finetuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9063–9074, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Utama et al. (2020a) Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020a.  Mind the trade-off: Debiasing NLU models without degrading the in-distribution performance.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8717–8729, Online. Association for Computational Linguistics.
  * Utama et al. (2020b) Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020b.  Towards debiasing NLU models from unknown biases.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7597–7610, Online. Association for Computational Linguistics.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang and Bansal (2018) Yicheng Wang and Mohit Bansal. 2018.  Robust machine comprehension models via adversarial training.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 575–581, New Orleans, Louisiana. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. 2019.  PAWS: Paraphrase adversaries from word scrambling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1298–1308, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Zhao et al. (2020) Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. 2020.  Masking as an efficient alternative to finetuning for pretrained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2226–2241, Online. Association for Computational Linguistics.

##  Appendix A Negation Words

Within the MNLI Validation Matched set, we select those entailed samples where
at least one of the following words is present in the hypothesis: no, not,
don’t, none, nothing, never, aren’t, isn’t, weren’t, neither, don’t, didn’t,
doesn’t, cannot, hasn’t won’t.

##  Appendix B Hyperparameters

###  B.1 Baseline and Debiased Models

Our baseline models are trained with a batch size of 32, learning rate of
3e-5, weight decay of 0.1, and 20% warmup steps.

###  B.2 Weak Learners

Undertrained models are trained with the standard BERT architecture and 2000
samples (500 in the case of QQP and FEVER) of the dataset for 5 epochs. Utama
et al. (2021) mention 3 in their paper, but train with 5 in their code
repository. We confirm that 5 yields better results. For underparameterized
models, we follow Sanh et al. (2021) and use BERT-Tiny Turc et al. (2019), a
BERT model with 2 layers and an inner size of 128. This model is trained on
the full training set for 3 epochs, and slightly adjusted hyperparameters.

We use focal loss (sample reweighting) in all mask debiasing experiments, with
the exception of the HypOnly MD model, which uses product-of-experts (in an
attempt to follow Utama et al. (2020a) as closely as possible).

###  B.3 Masked Debiasing Models

Our mask search is performed with a score learning rate of 0.1, batch size of
128, and 12 epochs of training.


# Learning Conditional Knowledge Distillation for Degraded-Reference
Image Quality Assessment

Heliang Zheng1 , Huan Yang2, Jianlong Fu2, Zheng-Jun Zha1, Jiebo Luo3
1University of Science and Technology of China, Hefei, China
2Microsoft Research, Beijing, China
3University of Rochester, Rochester, NY
<EMAIL_ADDRESS>{huayan<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>This work was performed when Heliang Zheng was visiting
Microsoft Research as a research intern. Code and models are available at
https://github.com/researchmm/CKDN.

###### Abstract

An important scenario for image quality assessment (IQA) is to evaluate image
restoration (IR) algorithms. The state-of-the-art approaches adopt a full-
reference paradigm that compares restored images with their corresponding
pristine-quality images. However, pristine-quality images are usually
unavailable in blind image restoration tasks and real-world scenarios. In this
paper, we propose a practical solution named degraded-reference IQA (DR-IQA),
which exploits the inputs of IR models, degraded images, as references.
Specifically, we extract reference information from degraded images by
distilling knowledge from pristine-quality images. The distillation is
achieved through learning a reference space, where various degraded images are
encouraged to share the same feature statistics with pristine-quality images.
And the reference space is optimized to capture deep image priors that are
useful for quality assessment. Note that pristine-quality images are only used
during training. Our work provides a powerful and differentiable metric for
blind IRs, especially for GAN-based methods. Extensive experiments show that
our results can even be close to the performance of full-reference settings.

##  1 Introduction

Figure 1: (a) Pristine-quality images can provide strong reference information
for IQA. When pristine-quality images are unavailable, (b) directly regressing
restored images to quality scores causes a dramatic drop in performance. To
this end, (c) we propose to extract reference information from degraded images
and make such a solution effective. FR, NR, DR, S, and SRCC indicate full-
reference, no-reference, degraded-reference, quality score, and Spearman’s
Rank order Correlation Coefficients [30], respectively.

Digital images may be subject to various quality degradations during
processing, compression, transmission, etc. [36]. And image restoration (IR)
algorithms are developed to improve the quality of degraded images [18, 35,
38, 41, 43, 46, 47]. A consequential question is how to assess the quality of
restored images and evaluate IR algorithms. The state-of-the-art IQA
approaches adopt a full-reference paradigm (FR-IQA), which learns to compare
restored images with their corresponding pristine-quality images (as shown in
Figure 1 (a)). FR-IQA algorithms have been widely adopted as IR evaluation
metrics, e.g., PSNR, SSIM [36], and LPIPS [45]. However, they cannot be
applied to blind image restoration tasks and real-world applications, where
pristine-quality images are unavailable. In this paper, we study the problem
of evaluating IR models without relying on pristine-quality images.

To evaluate IR models without pristine-quality images, no-reference IQA (NR-
IQA) methods provide a solution, that is, directly regressing restored images
to quality scores, i.e., mean opinion scores (MOS) [2, 13, 20, 32]. However,
the absence of reference information makes the problem much more challenging
and causes a dramatic drop in performance, e.g., $0.1409$ SRCC (Spearman’s
Rank order Correlation Coefficients [30]) drops. This motivates us to seek
available reference information. We find that the inputs of IR models (i.e.,
degraded images) are usually free to be obtained, and it has been verified
that degraded images also contain useful image priors like pristine-quality
images for solving under-constrained problems in computer vision [25, 33]. To
this end, we propose a new solution named degraded-reference IQA (DR-IQA).
However, directly replacing the pristine-quality references of existing FR-IQA
models [5] by degraded references causes 0.1239 SRCC drops, because the noises
involved by various degradations make it difficult to mine and leverage
reference information from degraded images.

To leverage degraded images and improve the effectiveness of DR-IQA, we
propose a Conditional Knowledge Distillation Network (CKDN). As shown in
Figure 2, CKDN consists of three modules, i.e., a degradation-tolerant
embedding module (DTE), a quality-sensitive embedding module (QSE), and a
convolutional score predictor (CSP). The DTE is the key module, which aims to
effectively extract reference information for IQA from degraded images.
Specifically, it learns a reference space, where various degraded images are
optimized to share the same feature statistics with pristine-quality images.
Such space is learned in the condition of conducting quality assessments with
the QSE and CSP, where the QSE learns discriminative features from restored
images, and the CSP maps feature differences to human-annotated scores.
Moreover, we find that pre-training such conditions can help to learn quality-
sensitive features and further advance the optimization of the reference
space. In particular, we propose a relative score regression task that can
enlarge the space of training data by creating data pairs.

We conduct extensive experiments to evaluate our proposed solution. It is
shown that our proposed CKDN enables DR-IQA to achieve comparable performance
to the full-reference setting on various IR tasks, e.g., traditional/GAN-based
super-resolution, de-noising, etc. Moreover, we further study the influence of
reference quality and the performance of current IQA methods on evaluating
different IR tasks. We draw three conclusive insights: 1) our CKDN works well
with a large range of degradation types, 2) reference images are extremely
important for evaluating GAN-based images, and 3) current IQA methods can
provide around 85% reliable judgments for evaluating IR algorithms. Please
refer to Section 5 for more details. Our main contributions can be summarized
as follows: 1) To the best of our knowledge, we are the first to leverage
degraded images for IQA. 2) We formulate this practical setting and propose an
effective conditional knowledge distillation network to solve this problem. 3)
Extensive analyses show that degraded references are of great help for
evaluating GAN-based images. Overall, we believe our work would contribute to
the IQA and the blind SR communities, and providing insights for evaluating
GAN-based models.

Figure 2: Overview of the proposed Conditional Knowledge Distillation Network
(CKDN). CKDN consists of three components, i.e., a degradation-tolerant
embedding module (DTE) in (b), a quality-sensitive embedding module (QSE) in
(d), and a convolutional score predictor (CSP) in (e). The DTE embeds the
degraded image in (a) to a reference space in (h) by optimizing a conditional
knowledge distillation (CKD) loss. Such a loss learns reference information
from pristine-quality images in (g) and minimizes the difference between the
features of degraded and pristine-quality images. Note that the QSE and CSP
provide conditions to learn the reference space, and the learned
representation of degraded images provides reference information for IQA.

##  2 Related Work

No-reference Image Quality Assessment. NR-IQA aims to predict the perceived
quality of distorted images without referring to pristine-quality images.
Getting rid of reference images makes such a setting more practical in many
applications and therefore receives more and more attention in recent years.
Early works in the literature of NR-IQA proposed to model statistics of
natural images and regress parametric deviations to image degradations [22,
23, 29]. In recent years, deep neural networks (DNNs) show promising capacity
and are leveraged to fit distorted images to quality scores for NR-IQA [2, 13,
20, 32]. Considering the problem of limited training data, dipIQ proposed to
learn-to-rank large-scale unlabeled image pairs in a self-supervised manner
[19]. Inspired by FR-IQA, Hallucinated-IQA proposed to compare the distorted
images to hallucinated references, which are restored from the distorted
images by adversarial training [3, 17, 28]. And some researchers also propose
to generate a quality map for NR-IQA [24]. There are three differences
compared to these works: 1) Compared to hallucinated references that generated
from assessed images, degraded references are independent of assessed images
and can provide additional information. 2) Previous works use a handcraft
space (i.e., RGB image/SSIM map), while we learn deep embeddings end-to-end.
3) previous works try to “distill” all the information instead of only
reference information for IQA.

Full-reference Image Quality Assessment. The full-reference setting
significantly simplifies the IQA problem by converting it to two sub-problems,
i.e., calculating the distance between restored and reference images and
mapping such distance to quality scores [21, 36, 37, 44]. Here we mainly
review the full-reference IQA methods that have been widely used to evaluate
IR models [36, 45]. PSNR and SSIM [36] are appealing because they are simple
to calculate and have clear physical meanings. But they do not correlate well
with perceived visual quality [5, 27, 45], especially for evaluating the
images generated by GANs [4]. LPIPS [45] analyzed the unreasonable
effectiveness of deep features as a perceptual metric, which has been adopted
by more and more IR models as a perceptual metric [7, 18]. And SWDN further
improves LPIPS by taking the spatial misalignment for GAN-based distortion
into consideration [5, 6]. Different from FR-IQA, our proposed DR-IQA only
requires degraded images as reference, and we proposed a Conditional Knowledge
Distillation Network to leverage deep image priors from the degraded images.
Note that our proposed DR-IQA is also different from traditional reduced-
reference IQA (RR-IQA), which extracts predefined features, such as frequency
domain coefficients and image gradients, as references under the scenario of
visual communication systems [16, 34].

##  3 Approach

In this section, we will introduce the setting of DR-IQA and our proposed
model for DR-IQA. Our model takes as input a degraded image in Figure 2 (a)
(i.e., the input of IR models) and a restored image in Figure 2 (c) (i.e., the
output of IR models), and aims to predict a quality score for the restored
image in Figure 2 (f). Similar to FR-IQA, the basic idea is to compare the
restored image to the reference and further map the learned difference to a
quality score. However, the reference images in DR-IQA are noisy due to
degradations. To solve this problem, we propose to learn a reference space in
Figure 2 (h) to capture deep image priors that are useful for quality
assessment. In particular, our proposed Conditional Knowledge Distillation
Network (CKDN) consists of a degradation-tolerant embedding module (DTE) in
Figure 2 (b), a quality-sensitive embedding module (QSE) in Figure 2 (d), and
a convolutional score predictor (CSP) in Figure 2 (e). We propose a
conditional knowledge distillation (CKD) loss to guide the learning of the
reference space. Moreover, we propose a relative score regression loss to pre-
train the QSE (shown in Figure 4). We will introduce the details in the
following.

###  3.1 Formulations

Formulation of DR-IQA. Given a pristine-quality image $\mathbf{H}$; the
degraded image can be denoted as $\mathbf{D}$; image restoration (IR)
algorithms can be applied to obtain restored images, i.e., $\mathbf{R}$; and
the human annotated quality score, i.e., mean opinion score (MOS) can be
denoted as $s$. In the training phase, all of these
$\\{\mathbf{H},\mathbf{D},\mathbf{R},s\\}$ are available. While in the testing
phase, the model takes as input only $\\{\mathbf{D},\mathbf{R}\\}$ and
predicts the quality score $s$.

Formulation of CKDN. CKDN consists of three modules, i.e., a degradation-
tolerant embedding module (DTE), a quality-sensitive embedding module (QSE),
and a convolutional score predictor (CSP), which are denoted as
$\mathbf{\mathcal{E}_{1}}$, $\mathbf{\mathcal{E}_{2}}$,
$\mathbf{\mathcal{S}}$, respectively. We can obtain the quality score by:

$s=\mathbf{\mathcal{S}}(\mathbf{\mathcal{E}}_{1}(\mathbf{D})-\mathbf{\mathcal{E}}_{2}(\mathbf{R})),$
(1)

where $\mathbf{D}$, $\mathbf{R}$, and $s$ are the degraded image, restored
image, and quality score of the restored image, respectively.

Special designs of CKDN for DR-IQA. Compared to the state-of-the-art FR-IQA
models [27, 45], the proposed CKDN has two advantages for the DR-IQA problem
(as shown in Figure 3). 1) FR-IQA models extract the restored image features
and reference features with the same parameters, while our CKDN learns task-
specific embeddings (i.e., the DTE and QSE). 2) FR-IQA models focus on
extracting feature difference, while we pay more attention (by stacking
residual blocks in the CSP) to mapping feature difference to quality scores.
Such designs can better handle the degradation of reference images.

Figure 3: Comparison of (a) FR-IQA models and (b) DR-IQA model (CKDN). FR-IQA
models focus on calculating difference between restored and reference images,
while DR-IQA requires 1) different embeddings for restored and reference
images and 2) convolutional layers for mapping feature difference to scores.

###  3.2 Learning quality-sensitive embedding

Learning quality-sensitive features for restored images is important for image
quality assessment. We propose to learn a quality-sensitive embedding (QSE) to
extract such discriminative features. Regressing absolute quality scores is
the most intuitive loss function to optimize the QSE, i.e., training the model
to fit the input data $\\{\mathbf{D},\mathbf{R}\\}$ to the absolute quality
score $s$. Specifically, we minimize the Mean Square Error (MSE) for such a
regression problem:

$L_{a}=\frac{1}{N}\sum_{i}\left\lVert
s_{i}-\mathbf{\mathcal{S}}(\mathbf{\mathcal{E}}_{1}(\mathbf{D})-\mathbf{\mathcal{E}}_{2}(\mathbf{R}_{i}))\right\rVert_{2}^{2},$
(2)

where $N$ is the number of training images. Note that in this paper, we show
the case of single content with single degradation for simplicity.

###  3.3 Learning degradation-tolerant embedding

In this subsection, we will introduce how to learn the DTE, which is the key
module of our model. the DTE is supposed to capture reference information from
degraded images. Our basic idea is to learn reference information from
pristine-quality images and use the learned knowledge to guide the learning of
DTE. To achieve this, we propose a conditional knowledge distillation loss and
design an effective pre-training method.

Conditional knowledge distillation. Knowledge distillation is powerful tool
for feature learning [11, 39, 48, 49]. Our extended conditional knowledge
distillation leverages restored images as context information (condition),
which is important in various vision tasks [40, 42]. Equation 2 shows the
basic objective function for learning CKDN. Let us consider training CKDN in a
full-reference setting:

$L^{H}_{a}=\frac{1}{N}\sum_{i}\left\lVert
s_{i}-\mathbf{\mathcal{S}}^{H}(\mathbf{\mathcal{E}}^{H}_{1}\mathbf{(H)}-\mathbf{\mathcal{E}}^{H}_{2}(\mathbf{R}_{i}))\right\rVert_{2}^{2},$
(3)

where $L^{H}_{a}$, $\mathbf{\mathcal{E}}^{H}_{1}(\cdot)$,
$\mathbf{\mathcal{E}}^{H}_{2}(\cdot)$, and $\mathbf{\mathcal{S}}^{H}(\cdot)$
denote the score regression loss, degradation-tolerant embedding module,
quality-sensitive embedding module, and the convolutional score predictor that
trained with pristine-quality images.

Figure 4: An illustration of pre-training the QSE by the proposed relative
score regression loss.

Compared to degraded images, it is much easier to learn deep image priors for
IQA from pristine-quality images, i.e.,
$\mathbf{\mathcal{E}}^{H}_{1}\mathbf{(H)}$ is supposed to be better than
$\mathbf{\mathcal{E}_{1}(D)}$. Such an observation motivates us to guide the
learning of $\mathbf{\mathcal{E}_{1}(D)}$ by
$\mathbf{\mathcal{E}}^{H}_{1}\mathbf{(H)}$. Specifically, our proposed
conditional knowledge distillation loss can be denoted as:

$\begin{split}&L_{ckd}=L_{a}+L^{H}_{a}+\lambda\left\lVert\mathbf{\mathcal{E}}^{H}_{1}\mathbf{(H)}-\mathbf{\mathcal{E}}_{1}(\mathbf{D})\right\rVert_{2}^{2},\\\
&s.t.,\quad\mathbf{\mathcal{E}}^{H}_{2}=\mathbf{\mathcal{E}}_{2},\quad\mathbf{\mathcal{S}}^{H}=\mathbf{\mathcal{S}},\end{split}$
(4)

where $L_{ckd}$ is the conditional knowledge distillation loss. $L_{a}$, and
$L^{H}_{a}$ are the score regression loss, which are shown in Equation 2 and
Equation 3, respectively. $\lambda$ is the loss weight.
$\mathbf{\mathcal{E}}^{H}_{2}=\mathbf{\mathcal{E}_{2}}$, and
$\mathbf{\mathcal{S}}^{H}=\mathbf{\mathcal{S}}$ are the constraints. Note that
such constraints are essential to achieve effective distillation for DR-IQA,
which can ensure $\mathbf{\mathcal{E}_{1}(D)}$ and
$\mathbf{\mathcal{E}}^{H}_{1}\mathbf{(H)}$ to be in the same latent space. In
other word, the learning of embeddings for pristine-quality and degraded
images should be in the same condition.

Pre-training for conditional knowledge distillation. The objective function in
Equation 4 can be applied to train the CKDN in an end-to-end manner, where the
parameters are initialized by ImageNet [14] pre-trained parameters. Taking one
step further, we study how the condition in Equation 4 influences the
knowledge distillation performance. We find that pre-training the QSE, i.e.,
$\mathbf{\mathcal{E}}_{2}$, can advance the optimization and achieve better
performance. That is, before training the CKDN by Equation 4, we first learn
the QSE by a relative score regression loss. Specifically, we train the model
by comparing two restored images, e.g., $\mathbf{R_{i}},\mathbf{R_{j}}$, and
regressing the possibility of $s_{\mathbf{i}}>s_{\mathbf{j}}$. Such loss can
be denoted as:

$L_{r}=\frac{1}{N}\sum_{i,j}\left\lVert\text{Pr}{(s_{i}>s_{j})}-\mathbf{\mathcal{S}}(\mathbf{\mathcal{E}}_{2}(\mathbf{R}_{i})-\mathbf{\mathcal{E}}_{2}(\mathbf{R}_{j}))\right\rVert_{2}^{2},$
(5)

where $N$ is the number of training pairs, $\text{Pr}{(s_{i}>s_{j})}$ can be
obtained by $\text{Pr}{(s_{i}>s_{j})}=1/(1+10^{(s_{j}-s_{i})/M})$ [5], $M$ is
a dataset-specific parameter of the distribution, and $M=400$ in the dataset
we used.

We call such training stage “pre-training” because the relative score
regression loss enlarges the training data by creating pairs. And the pre-
trained QSE can provide a better condition for the subsequent knowledge
distillation.

Table 1: The details of the subset of PIPAL [5] dataset that used in our
experiments. # IR indicates the number of IR models and settings, and #Img
means the number of distorted images.

IR tasks | Sub-tasks | # IR | # Img
---|---|---|---
| SR
---
| (a) interpolation
---
(b) traditional SR
(c) PSNR-oriented SR
(d) GAN-based SR
62 | 12,400
Denoise |  | (e) mean filtering
---
(f) traditional denoising
(g) Deep Denoising
13 | 2,600
| Mixture
---
| (h) noisy images SR
---
(i) SR after denoising
(j) SR after decompression
14 | 2,800

Comparisons to previous work. We analyze the difference between our proposed
conditional knowledge distillation and previous arts, e.g., data augmentation,
vanilla knowledge distillation [11], and hallucinated reference [17].

The data augmentation method adds the $\\{\mathbf{H},\mathbf{R_{i}}\\}$ pair
to training data, which would improve the generality of the model. Such a
strategy can be regarded as an indirect knowledge distillation in a parameter-
sharing manner.

The vanilla knowledge distillation method optimizes the model by Equation 4
without additional constraints. Such a strategy distills unaligned feature
distributions from the teacher network (trained with pristine-quality images)
to the student network (trained with degraded images).

Hallucinated reference methods can be applied to DR-IQA by generating
hallucinated references from the degraded images, and use the generated
reference to conduct IQA. Such a solution tries to “distill” all the
information from pristine-quality images to degraded images.

Compared to data augmentation and vanilla knowledge distillation, our method
is more direct and effective with well-aligned features. And compared to
hallucinated reference methods, our selectively feature distillation is easier
to learn as it is an ill-posed problem to recover all information from
degraded images to pristine-quality images.

##  4 Experiments

###  4.1 Experiment setup

Datasets. We evaluate our model on the Perceptual Image Processing ALgorithms
dataset (PIPAL) [5], which is the largest IQA dataset with 1.13 million human
annotations. PIPAL contains two types of images which are traditional
distorted images (e.g., JPEG compression, noise, spatial warping, etc.) and
images restored by IR algorithms. Since our proposed IQA method is designed
for evaluating IR algorithms, we remove the traditional distorted images and
use a subset of the released PIPAL, which contains three important image
restoration tasks (i.e., super-resolution, de-noising, and mixture
restoration). We split training and validation sets in a manner that is
consistent with the real-world scenario for developing novel IR algorithms.
Specifically, we randomly split 200 reference images into 175 training images
and 25 validation images, which can ensure us to validate our model on unseen
image contents. Moreover, we further split 89 IR algorithms into 50 algorithms
for training and 39 algorithms for validation, which can ensure validating on
unseen algorithms. Note that we split old algorithms for training and recent
algorithms for validation. More details can be found in Table 1.

Evaluation metrics. We follow previous works [2, 5, 20], to use two most
widely obtained IQA metrics, i.e., Spearman’s Rank order Correlation
Coefficients (SRCC) [30] and Pearson Linear Correlation Coefficients (PLCC).
SRCC evaluates the monotonicity of two lists, i.e., whether the order of
predicted scores is consistent with human-annotated orders. And PLCC considers
both orders and values.

Baselines. We compared our method to previous methods in image quality
assessment, including both NR-IQA and FR-IQA methods. In the following, the
first four methods are NR-IQA models, and the latter five are FR-IQA models.
We compare to them, due to their state-of-the-art performance and high
relevance.

Table 2: Comparisons to previous works. We implement FR-IQA models to our DR-
IQA setting, $\uparrow$ indicates the higher the better.

| Methods |  SRCC$\uparrow$ |  PLCC$\uparrow$
---|---|---|---
NR-IQA |  DIQaM [2] | 0.6255 | 0.6019
NIMA [32] | 0.6330 | 0.6540
Hall-IQA [17] | 0.6390 | 0.6167
MEON [20] | 0.6436 | 0.6610
DR-IQA | PSNR | 0.3231 | 0.3516
SSIM [36] | 0.3573 | 0.3509
PieAPP [27] | 0.6563 | 0.5937
LPIPS [45] | 0.6650 | 0.5872
SWDN [5] | 0.6729 | 0.6052
CKDN (Ours) | 0.7669 | 0.7514

  * •

DIQaM [2]: Propose to jointly learn the local quality and local weights, i.e.,
the relative importance of local quality to the global quality estimate.

  * •

MEON [20]: Formulate the NR-IQA to a multi-task training problem, i.e., a
distortion identification task and a quality prediction task.

  * •

NIMA [32]: Propose to predict the distribution of MOS and introduced an EMD-
based loss that penalizes misclassifications based on class distances.

  * •

Hall-IQA [17]: Propose to compare the distorted images to hallucinated
references, which are restored from the distorted images by GAN.

  * •

PSNR : Peak signal-to-noise ratio (PSNR) is an extension of mean square error
(MSE), which further considers the ratio between the signal and noise.

  * •

SSIM [36]: The structural similarity index measure (SSIM) considers image
degradation as a perceived change in structural information.

  * •

PieAPP [27]: Perceptual Image-Error Assessment through Pairwise Preference
(PieAPP) proposed a pairwise-learning framework.

  * •

LPIPS [45]: Learned Perceptual Image Patch Similarity (LPIPS) proposed to use
a learned weighted sum of deep features as a perceptual metric.

  * •

SWDN [5]: Space Warping Difference IQA Network (SWDN) proposed to consider the
robustness of spatial misalignment, especially for GAN-based distortions.

Implementation details. We follow previous work [5] and set the input image
resolution to $288\times 288$. The learning rate is set to 0.15, and we use a
constant learning rate strategy with a warm-up. We first train 10 epochs to
initialize the quality-sensitive embedding, and then we train the CKDN by 20
epochs. we find the loss weight $\lambda$ in Equation 4 is robust to
optimization, which is empirically set to 10. All the distortions are combined
during training, and our model is trained end-to-end. The model size is 103MB
and the training memory is 6.8GB (the batch size is 8 per GPU). We use PyTorch
[26] as our codebase. Our CKDN is efficient, and each experiment can be
finished in 30 minutes on 8 Tesla V100 GPUs.

Architectures of CKDN. We implement the two embeddings, i.e., the DTE and QSE,
by a convolutional layer and three residual blocks [10]; and we implement the
convolutional score predictor by four residual blocks and three fully
connected layers. Note that although the DTE and QSE share the same
architecture, their training strategies are different and specifically
designed.

Table 3: Ablation experiments for main components.

Methods |  SRCC$\uparrow$ |  PLCC$\uparrow$
---|---|---
FR model [5] | 0.6729 | 0.6052
CKDN | 0.7127 | 0.6606
CKDN + CKD | 0.7549 | 0.7473
CKDN + CKD + Pret. | 0.7669 | 0.7514
FR upper bound | 0.7922 | 0.8233

Table 4: Ablation experiments on conv. blocks w/o CKD.

Methods | Conv. blocks |  SRCC$\uparrow$ |  PLCC$\uparrow$
---|---|---|---
FR model [5] | VGG | 0.6729 | 0.6052
CKDN (Ours) | VGG | 0.7006 | 0.6420
CKDN (Ours) | Residual | 0.7127 | 0.6606

###  4.2 Comparisons

To evaluate our proposed CKDN, we implement the methods proposed for both NR-
IQA and FR-IQA tasks. Specifically, for NR-IQA models, we only use the
restored images for validation; and for FR-IQA models, we keep the same
setting as our model and used the degraded images as references. We train and
validate all the models on the same train/validation split to keep the
comparison fair. The results can be found in Table 2. It can be observed that
our proposed CKDN outperforms all previous works with a clear margin, which
yields the best solution when pristine-quality images are not available. PSNR
and SSIM cannot work well with degraded references as the distance calculated
by these methods are not robust for degradations. Moreover, without well-
designed mechanisms for degraded reference, FR-IQA models cannot leverage such
information and can only slightly outperform NR-IQA models in the term of
SRCC. FR-IQA models even perform worse in the term of PLCC. Since PLCC is
influenced by the value of scores, different reference images (e.g., $2\times$
and $4\times$ downsampled LR images) would cause disturbances and be
considered as noises. However, with our proposed method, the degraded images
can be leveraged to achieve 0.0940 SRCC gains compared to the state-of-the-art
model [5].

Table 5: Ablation experiments on model architectures.

Embedding | Feature size |  SRCC$\uparrow$ |  PLCC$\uparrow$
---|---|---|---
Shared | $72\times 72$ | 0.7311 | 0.6926
Unshared | $36\times 36$ | 0.7436 | 0.7387
Unshared | $18\times 18$ | 0.7234 | 0.7176
Unshared | $72\times 72$ | 0.7669 | 0.7514

Table 6: Ablation experiments on knowledge distillations.

Methods |  SRCC$\uparrow$ |  PLCC$\uparrow$
---|---|---
w/o distillation | 0.7127 | 0.6606
Data augmentation | 0.7469 | 0.7087
Hallucinated reference | 0.7348 | 0.7087
Knowledge distillation | 0.7338 | 0.7093
Conditional knowledge distillation | 0.7669 | 0.7514

###  4.3 Ablation Studies

Main components. We evaluate the main components of our proposed CKDN in Table
3. It can be observed that our proposed backbone works well for DR-IQA, which
outperforms previous FR models by 0.0398 SRCC. Moreover, the proposed
conditional knowledge distillation mechanism can further boost the performance
by 0.0422 SRCC, which demonstrates the effectiveness of guiding DTE by
pristine-quality images. Our proposed pre-training method can improve more
than 0.01 SRCC, which makes our DR setting’s performance close to the full
reference upper bound.

Model architectures. We study core techniques for model design in Table 5.
Sharing parameters for the quality-sensitive embedding and the degradation-
tolerant embedding would cause 0.0358 SRCC drops, which show the necessity of
learning unshared embeddings. Moreover, we want to emphasize that the spatial
information of feature difference is important, which is ignored by FR-IQA
models. Specifically, by down-sampling the input feature of CSP, we can
observe 0.0233 SRCC drops. Last but not least, Table 4 shows that it is the
formulation of CKDN that matters and the influence of specific architectures
(e.g., residual or VGG [31] blocks) are marginal.

Knowledge distillation. Table 6 shows the effectiveness of different knowledge
distillation methods. As analyzed in Section 3, our proposed conditional
knowledge distillation can more effectively distill useful information for
quality assessment into the degradation-tolerant embedding. We find that
hallucinated references even bring a performance drop, which would be caused
by involving feature noises when restoring the hallucinated references.

Pre-training methods. Table 7 shows how different pre-training methods (pre-
trained by different loss with different epochs) influence the performance. It
can be observed that the proposed relative score regression loss performs
better than the absolute score regression loss and ImageNet pre-training. Such
improvements show the effectiveness of training on highly related tasks with
enlarged pairwise data.

Table 7: Ablation experiments on pre-training in the term of SRCC. “Pret.”
indicates pre-training.

Epochs | 5 | 7 | 10 | 15
---|---|---|---|---
ImageNet Pret. | 0.7549 | 0.7549 | 0.7549 | 0.7549
$L_{a}$ Pret. | 0.7561 | 0.7574 | 0.7536 | 0.7543
$L_{r}$ Pret. | 0.7587 | 0.7629 | 0.7669 | 0.7643

Robustness on train/validation splits. We further explored how
train/validation splits influence the effectiveness of the proposed methods.
First, we study various train/validation splits from the aspect of image
contents. We find that different train/validation splits lead to comparable
performance (within 0.01 SRCC). Moreover, we study three different
train/validation splits from the aspect of distortion types. The results can
be found in Table 8, where the “Baseline” indicates SWDN [5]. It can be
observed that our proposed method can achieve consistent improvements on all
three settings, and are close to the performance achieved by using pristine-
quality images.

Table 8: Extensive experiments for robustness evaluation by different
train/validation splits (SRCC).

Train/VAl Split | Baseline | Ours | FR setting
---|---|---|---
50 train + 39 val | 0.6729 | 0.7669 | 0.7922
54 train + 35 val | 0.6884 | 0.7794 | 0.8011
59 train + 30 val | 0.6561 | 0.7697 | 0.7943

Figure 5: An illustration of the SRCC improvements obtained by different types
of reference images. The percentage numbers marked in green is the SRCC
improvements compared to no-reference settings.

CKDN can improve SR algorithms as a loss term. Specifically, we follow the
setting of NTIRE 2020 challenge winner [12], i.e., use DIV8K [9] dataset and
obtain LPIPS [45] as a metric. The results (85k iterations) can be found in
Table 9. It can be observed that our CKDN can improve the SR performance. We
only conduct experiments on the above settings, and more extensive experiments
would be conducted in future work.

Table 9: CKDN can be used as a loss term. “Baseline” and “VGG” indicate ESRGAN
[35] and Perceptual Loss, respectively.

Method | Baseline | Baseline + VGG | Baseline + CKDN
---|---|---|---
LPIPS $\downarrow$ | 0.5380 | 0.5039 | 0.4957

CKDN can generalize well across datasets. We test our model on an unseen
dataset BAPPS [45], and Table 10 shows that our model can achieve comparable
performance even with degraded references and without fine-tuning.

Table 10: Experiments on cross dataset evaluation. 2AFC scores are the
original metric of BAPPS [45].

Reference | FR-IQA (LPIPS) |  $\times$2 LR |  $\times$4 LR
---|---|---|---
2AFC scores $\uparrow$ | 69.80 | 69.71 | 68.98
Reference |  $\times$8 LR | L25 noise | L50 noise
2AFC scores $\uparrow$ | 68.38 | 68.58 | 67.63

CKDN can help in practice. In Table 11, we replace subjective image quality
experiments from the state-of-the-art SPSR [18]. The baseline is SRGAN [15].
CKDN delivers the same conclusion as PI [1] and LPIPS [45]. We also find some
inconsistent results, when evaluating comparable IR algorithms. And this
problem would be further studied in the future.

Table 11:  The results in practice: use CKDN as a SR metric. The results are
in the format of PI$\downarrow$/LPIPS$\downarrow$/(1-CKDN)$\downarrow$.

Method | Set5 | Set14 | BSD100
---|---|---|---
Bicubic | 7.36/0.34/0.48 | 7.02/0.43/0.41 | 7.00/0.52/0.51
SRGAN | 3.98/0.08/0.40 | 3.08/0.16/0.35 | 2.54/0.19/0.43
SPSR | 3.27/0.06/0.36 | 2.90/0.13/0.32 | 2.35/0.16/0.40

##  5 Discussions

In this section, we conduct extensive experiments and detailed analyses to 1)
evaluate the robustness of CKDN, 2) show the extreme effectiveness of
reference images for evaluating images generated by GANs, and 3) provide an
intuitive comprehension of how well current IQA models can be used as IR
metrics.

The proposed method works well on degraded references with a large range of
degradation types. We proposed to use IR inputs, i.e., degraded images, to
help to assess the quality of restored images, which is practical in real-
world scenarios. We further studied using the same degraded image (e.g.,
$4\times$ downsampled LR image) as a reference for assessing all the restored
(e.g., SR, Denoised) images. Such a setting is not practical, while it can
intuitively show the influence of reference quality on IQA performance. The
results can be found in Table 12 and an illustration can be found in Figure 5.
We choose the most common 5 degradations in the dataset. It can be observed
that the proposed method performs well with a large range of degradation,
e.g., $2\times$ downsample, $4\times$ downsample, Gaussian noise with levels:
25 and 50, JPEG compression with quality of 30 and 10. The performance dropped
on the $8\times$ downsample setting, while still outperforms the no-reference
setting by 0.0591 SRCC.

Reference images is extremely important for evaluating images generated by
GANs. Evaluating GAN-based IR models attracts lots of attention in recent
years [8, 5, 45] as the artifacts generated by GANs yield new challenges for
IQA. We study the performance of our model on GAN-based images in Table 13 and
find introducing reference (e.g., full reference or degraded reference) can
significantly benefit assessing GAN-based images (with 0.3101 SRCC gains).
Such a result is reasonable as the textures generated by GANs may confuse NR-
IQA models on distinguishing the artifacts and image contents. Thus reference
images can provide strong guidance to disentangle quality-sensitive factors
from image contents.

Table 12: Extensive experiments on how the quality of reference images
influence IQA performances.

Reference |  SRCC$\uparrow$ |  PLCC$\uparrow$
---|---|---
NR baseline | 0.6513 | 0.6312
LR images | $2\times$ | 0.7859 | 0.8286
$4\times$ | 0.7639 | 0.7806
$8\times$ | 0.7104 | 0.7251
Gauss. noise | Level: 25 | 0.7888 | 0.8291
Level: 50 | 0.7802 | 0.8190
JPEG | Q: 30 | 0.7911 | 0.8239
Q: 10 | 0.7844 | 0.8288
IR inputs (our DR setting) | 0.7669 | 0.7514
FR upper bound | 0.7922 | 0.8233

Current IQA methods can provide around 85% reliable judgments when they are
adopted as IR metrics. We provide an intuitive comprehension of how well
current IQA models can be used as IR metrics. In particular, we use IQA models
to compare IR algorithms under the same task and setting (e.g., the $\times 4$
SR task by GAN) and calculate the accuracy of the predicted judgments. We
train all the settings by the same backbone, i.e., our CKDN, for fair
comparisons. The results in Table 13 show that the accuracy of the FR-IQA
setting is around 85%, which means 85% preferences of the IQA model are
consistent with human judgments. Moreover, DR-IQA can achieve comparable
results with the FR-IQA setting, while the NR-IQA setting causes 17.27%
accuracy drops.

Table 13: Extensive experiments on evaluating GAN-based images by SRCC and
accuracy. Accuracy: we use IQA models to compare the output images of two IR
algorithms (which one is better), and calculates the percentage of right
judgments (the same as human judgments). $\uparrow$ indicates the higher the
better.

Metrics | NR-IQA | DR-IQA | FR-IQA
---|---|---|---
SRCC$\uparrow$ | 0.4081 | 0.7182 | 0.7560
Accuracy $\uparrow$ | 0.6853 | 0.8507 | 0.8580

##  6 Conclusion

In this paper, we study the problem of evaluating IR models without pristine-
quality images. We present a degraded-reference solution (DR-IQA) that
introduces IR inputs to boost the quality assessment performance. we find it
is nontrivial to leverage degraded images as reference, and by learning a
Conditional Knowledge Distillation Network (CKDN), promising performance can
be achieved. We have conducted extensive evaluations to demonstrate the
effectiveness of our proposed method. Moreover, we draw three valuable
conclusions about IQA and IR metrics: 1) our CKDN works well with a large
range of degradation types, 2) reference images are extremely important for
evaluating GAN-based images, and 3) current IQA methods can provide around 85%
reliable judgments for evaluating IR algorithms. In the future, we will focus
on 1) extending our model to evaluate other conditional GAN tasks (e.g., in-
painting, and semantic segmentation to images), and 2) further studying the
problem of using our model as a loss/metric for IR algorithms.

##  7 Acknowledgement

This work was supported by the National Key R&D Program of China under Grand
2020AAA0105702, National Natural Science Foundation of China (NSFC) under
Grants U19B2038, the University Synergy Innovation Program of Anhui Province
under Grants GXXT-2019-025.

## References

  * [1] Yochai Blau and Tomer Michaeli.  The perception-distortion tradeoff.  In CVPR, pages 6228–6237, 2018.
  * [2] Sebastian Bosse, Dominique Maniry, Klaus-Robert Müller, Thomas Wiegand, and Wojciech Samek.  Deep neural networks for no-reference and full-reference image quality assessment.  TIP, 27(1):206–219, 2017.
  * [3] Diqi Chen, Yizhou Wang, and Wen Gao.  No-reference image quality assessment: An attention driven approach.  TIP, 29:6496–6506, 2020.
  * [4] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NeurIPS, pages 2672–2680, 2014.
  * [5] Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Ren Jimmy S, and Chao Dong.  PIPAL: A large-scale image quality assessment dataset for perceptual image restoration.  In ECCV, pages 633–651, 2020.
  * [6] Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Jimmy Ren, and Chao Dong.  Image quality assessment for perceptual image restoration: A new dataset, benchmark and metric.  arXiv preprint arXiv:2011.15002, 2020.
  * [7] Jinjin Gu and Chao Dong.  Interpreting super-resolution networks with local attribution maps.  In CVPR, 2021.
  * [8] Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen.  GIQA: Generated image quality assessment.  In ECCV, pages 369–385, 2020.
  * [9] Shuhang Gu, Andreas Lugmayr, Martin Danelljan, Manuel Fritsche, Julien Lamour, and Radu Timofte.  DIV8K: Diverse 8k resolution image dataset.  In ICCVW, pages 3512–3516, 2019.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [11] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  In NeurIPSW, 2014.
  * [12] Younghyun Jo, Sejong Yang, and Seon Joo Kim.  Investigating loss functions for extreme super-resolution.  In CVPRW, pages 424–425, 2020.
  * [13] Le Kang, Peng Ye, Yi Li, and David Doermann.  Convolutional neural networks for no-reference image quality assessment.  In CVPR, pages 1733–1740, 2014.
  * [14] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In NeurIPS, volume 25, pages 1097–1105, 2012.
  * [15] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In CVPR, pages 4681–4690, 2017.
  * [16] Qiang Li and Zhou Wang.  Reduced-reference image quality assessment using divisive normalization-based image representation.  JSTSP, 3(2):202–211, 2009.
  * [17] Kwan-Yee Lin and Guanxiang Wang.  Hallucinated-IQA: No-reference image quality assessment via adversarial learning.  In CVPR, pages 732–741, 2018.
  * [18] Cheng Ma, Yongming Rao, Yean Cheng, Ce Chen, Jiwen Lu, and Jie Zhou.  Structure-preserving super resolution with gradient guidance.  In CVPR, pages 7769–7778, 2020.
  * [19] Kede Ma, Wentao Liu, Tongliang Liu, Zhou Wang, and Dacheng Tao.  dipIQ: Blind image quality assessment by learning-to-rank discriminable image pairs.  TIP, 26(8):3951–3964, 2017.
  * [20] Kede Ma, Wentao Liu, Kai Zhang, Zhengfang Duanmu, Zhou Wang, and Wangmeng Zuo.  End-to-end blind image quality assessment using deep neural networks.  TIP, 27(3):1202–1213, 2017.
  * [21] Rafał Mantiuk, Kil Joong Kim, Allan G Rempel, and Wolfgang Heidrich.  HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions.  TOG, 30(4):1–14, 2011.
  * [22] Anish Mittal, Anush K Moorthy, and Alan C Bovik.  No-reference image quality assessment in the spatial domain.  TIP, 21(12):4695–4708, 2012.
  * [23] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik.  Making a “completely blind” image quality analyzer.  IEEE Signal processing letters, 20(3):209–212, 2012.
  * [24] Da Pan, Ping Shi, Ming Hou, Zefeng Ying, Sizhe Fu, and Yuan Zhang.  Blind predicting similar quality map for image quality assessment.  In CVPR, pages 6373–6382, 2018.
  * [25] Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo.  Exploiting deep generative prior for versatile image restoration and manipulation.  In ECCV, pages 262–277, 2020.
  * [26] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  PyTorch: An imperative style, high-performance deep learning library.  In NeurIPS, pages 8024–8035, 2019.
  * [27] Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen.  PieAPP: Perceptual image-error assessment through pairwise preference.  In CVPR, pages 1808–1817, 2018.
  * [28] Hongyu Ren, Diqi Chen, and Yizhou Wang.  RAN4IQA: Restorative adversarial nets for no-reference image quality assessment.  In AAAI, volume 32, 2018.
  * [29] Michele A Saad, Alan C Bovik, and Christophe Charrier.  Blind image quality assessment: A natural scene statistics approach in the DCT domain.  TIP, 21(8):3339–3352, 2012.
  * [30] Hamid R Sheikh, Muhammad F Sabir, and Alan C Bovik.  A statistical evaluation of recent full reference image quality assessment algorithms.  TIP, 15(11):3440–3451, 2006.
  * [31] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In ICLR, 2015.
  * [32] Hossein Talebi and Peyman Milanfar.  NIMA: Neural image assessment.  TIP, 27(8):3998–4011, 2018.
  * [33] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Deep image prior.  In CVPR, pages 9446–9454, 2018.
  * [34] Shiqi Wang, Xiang Zhang, Siwei Ma, and Wen Gao.  Reduced reference image quality assessment using entropy of primitives.  In PCS, pages 193–196, 2013.
  * [35] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy.  ESRGAN: Enhanced super-resolution generative adversarial networks.  In ECCVW, pages 0–0, 2018.
  * [36] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: From error visibility to structural similarity.  TIP, 13(4):600–612, 2004.
  * [37] Zhou Wang, Eero P Simoncelli, and Alan C Bovik.  Multiscale structural similarity for image quality assessment.  In ACSSC, volume 2, pages 1398–1402, 2003.
  * [38] Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo.  Learning texture transformer network for image super-resolution.  In CVPR, pages 5791–5800, 2020.
  * [39] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  In CVPR, pages 4133–4141, 2017.
  * [40] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Generative image inpainting with contextual attention.  In CVPR, pages 5505–5514, 2018.
  * [41] Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo.  Learning pyramid-context encoder network for high-quality image inpainting.  In CVPR, pages 1486–1494, 2019.
  * [42] Zheng-Jun Zha, Daqing Liu, Hanwang Zhang, Yongdong Zhang, and Feng Wu.  Context-aware visual policy network for fine-grained image captioning.  TPAMI, 2019.
  * [43] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.  Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising.  TIP, 26(7):3142–3155, 2017.
  * [44] Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang.  FSIM: A feature similarity index for image quality assessment.  TIP, 20(8):2378–2386, 2011.
  * [45] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, pages 586–595, 2018.
  * [46] Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao.  RankSRGAN: Generative adversarial networks with ranker for image super-resolution.  In ICCV, pages 3096–3105, 2019.
  * [47] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In ECCV, pages 286–301, 2018.
  * [48] Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha.  Object relational graph with teacher-recommended learning for video captioning.  In CVPR, pages 13278–13288, 2020.
  * [49] Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo.  Looking for the devil in the details: Learning trilinear attention sampling network for fine-grained image recognition.  In CVPR, pages 5012–5021, 2019.