# What Vision-Language Models ‘See’ when they See Scenes

Michele Cafagna1 Kees van Deemter2 Albert Gatt1,2

1University of Malta, Institute of Linguistics and Language Technology
2Universiteit Utrecht, Information and Computing Sciences
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Images can be described in terms of the objects they contain, or in terms of
the types of scene or place that they instantiate. In this paper we address to
what extent pretrained Vision and Language models can learn to align
descriptions of both types with images. We compare 3 state-of-the-art models,
VisualBERT, LXMERT and CLIP. We find that (i) V&L models are susceptible to
stylistic biases acquired during pretraining; (ii) only CLIP performs
consistently well on both object- and scene-level descriptions. A follow-up
ablation study shows that CLIP uses object-level information in the visual
modality to align with scene-level textual descriptions.

##  1 Introduction

Grounding symbols in perception Harnad (1990) is a crucial step towards
achieving full understanding of natural language Bender and Koller (2020);
Bisk et al. (2020). This endeavour has received new impetus through the
development of pretrained Vision and Language (V&L) models (e.g. Lu et al.,
2019; Tan and Bansal, 2019; Li et al., 2019; Chen et al., 2020; Li et al.,
2020a; Su et al., 2020; Wang et al., 2020; Luo et al., 2020; Li et al., 2021;
Huang et al., 2021; Radford et al., 2021). Similarly to unimodal language
models such as BERT Devlin et al. (2019), V&L models are intended to be task-
agnostic and are extensively pretrained on paired image-text data, achieving
good performance on several tasks after finetuning (e.g. Lu et al., 2020; Li
et al., 2020c; Kim et al., 2021). Pretraining usually includes an image-text
alignment task to discover implicit cross-modal relationships. Although the
importance of this task is widely recognized and adopted during model
pretraining, it is unclear how the models perform on it, since they are
usually evaluated on downstream tasks.

LN: This is the picture of a stadium. In the foreground there is a person […]
At the back there are group of people sitting […].

COCO: a baseball player getting ready to swing at a baseball game in a stadium
packed with people.

HL1K: the picture is shot in a baseball field

Figure 1: An example of scene with COCO and Localized Narrative (LN) object-
level captions, versus HL1K scene-level description (Section 3)

The data used for V&L pretraining usually contains highly descriptive text
which mentions objects and their spatial relationships. For instance, the COCO
Chen et al. (2015) and Localized Narratives (LN; Pont-Tuset et al., 2019)
captions for Figure 1 are of this type, though they differ stylistically. By
contrast, the third caption in the figure, from the novel HL1K dataset
introduced in Section 3 below, is what we refer to as ‘scene-level’, focusing
on what type of scene or location is depicted. Note that both the object- and
scene-level descriptions in the Figure describe the picture, albeit in
different ways. Indeed, it would be expected that, for a V&L model to display
true grounding capabilities, it should be able to identify both types of
descriptions as true of a scene. For models which do display this cabability,
a natural follow-up question is whether their representations capture
interesting connections between scenes on the one hand, and the objects within
them on the other.

Research on human perception suggests that humans do not perceive scenes
exclusively in terms of the objects they contain, and that visual salience is
not exclusively determined by bottom-up features such as colour and texture.
Rather, visual stimuli are considered ‘scenes’ because their elements
constitute a meaningful whole, both in terms of their contents (e.g. one
expects an oven in a kitchen, but not in a living room) and in terms of their
spatial arrangement (e.g. ovens do not typically hang from the ceiling)
Malcolm et al. (2016). These observations gave impetus to work showing that
violations of scene ‘semantics’ (content) and ‘syntax’ (spatial arrangement)
exact a cognitive cost during perception (e.g. Biederman et al., 1982; Võ and
Wolfe, 2013). A related strand of modeling research in computer vision has
also shown that scene-level priors generate expectations about objects and
their configurations, impacting the salience of objects in a way that
classical, feature-based models of attention (e.g. Itti and Koch, 2001) would
not predict Torralba et al. (2006); Oliva and Torralba (2007). Indeed, the
problem of linking low-level features with high-level semantic information is
an instance of the problem referred to as the ‘semantic gap’ in computer
vision Ma et al. (2010).

In this paper we investigate whether V&L models are able to handle object-
level and scene-level descriptions equally well. A positive answer to this
question would suggest that such models are learning useful associations
between the elements of a scene and the overall scene type, as captured in the
textual descriptions.

We perform an analysis in a zero-shot setting on three state-of-the-art
pretrained Vision and Language models. To our knowledge, this is the first
systematic comparison of model capabilities on object- versus scene-level
grounding. The goal of this study is therefore not to establish new SOTA
results, but to further our understanding of what V&L models learn, as a
function of the data they are pretrained on and the model architecture.
Therefore we choose three models differing in many settings, (including
training set size, architecture, number of parameters and model size). All of
the models are however optimized on the image-sentence alignment task.

We find that only one of the models under comparison, CLIP Radford et al.
(2021), performs consistently well on both object- and scene-level image-text
matching. We then investigate this model’s abilities in depth, using an
ablation method to identify the elements of a text and/or an image which
contribute to these abilities.

##  2 Models

Current V&L models typically combine textual and visual features in a single
or a dual-stream architecture. Though the two architectures have been found to
perform roughly at par Bugliarello et al. (2020), in this paper we include
representatives of both. We also include a third model which differs in
structure and is trained on a much larger and more varied dataset. Table 1
gives an overview of some of the properties of the models we consider.

| Training size | Model size | Pretraining
---|---|---|---
| (# image-sentence pairs) | (# parameters) | Objectives
CLIP | 400M | 151M | ISA
VisualBERT | 330k | 112M | ISA, MLM
LXMert | 9.18M | 228M | ISA, MLM
MOP, VQA

Table 1: Comparison of training settings for the three models (ISA: Image-
Sentence Alignment, MLM: Masked Language Modeling, MOP: Masked Object
Prediction, VQA: Visual Question Answering)

#### LXMERT

Tan and Bansal (2019) is a dual-stream model, which encodes text and visual
features in parallel, combining them using cross-modal layers. LXMERT is
trained on COCO captions Chen et al. (2015) as well as a variety of VQA
datasets, with an image-text alignment objective, among others.

#### VisualBERT

Li et al. (2019) is a single-stream, multimodal version of BERT Devlin et al.
(2019), with a Transformer stack to encode image regions and linguistic
features and align them via self-attention. It is pretrained on COCO captions
Chen et al. (2015). Image-text alignment is conceived as an extension of the
next-sentence prediction task in unimodal BERT. Thus, VisualBERT expects an
image and a correct caption, together with a second caption, with the goal of
determining whether the second caption matches the $\langle$image, correct
caption$\rangle$ pair.

#### CLIP

Radford et al. (2021) combines a transformer encoder for text with an image
encoder based on Visual Transformer Dosovitskiy et al. (2020), jointly trained
using contrastive learning to maximise scores for aligned image-text pairs.
CLIP is trained on around 400m pairs sourced from the Internet, a strategy
similar to the web-scale training approach used for unimodal models such as
GPT-3 Brown et al. (2020). We note that the visual backbone for this model
differs from that of LXMERT and VisualBERT, both of which use Faster-RCNN Ren
et al. (2015).

##  3 Data

We use four different datasets for our experiments. Dataset statistics are
provided in the Appendix.

#### Localized Narratives

Localized Narratives (LN) Pont-Tuset et al. (2019) is a V&L dataset harvested
by transcribing speech from annotators who were instructed to give object-by-
object descriptions as they moved a mouse over image regions. LN captions tend
to be highly detailed and stylistically similar to speech. We use LN as a
source of object-level captions. The images in LN come from preexisting
datasets; this allows us to align LN captions with images and captions from
datasets such as COCO and ADE20K.

#### ADE20K

ADE20K Zhou et al. (2017) is a computer vision dataset containing 20k images
comprehensively annotated with objects, parts and scene labels. We use ADE20K
as a source of scene-level captions. For our experiments, we filter out images
with scenes which in the dataset are labelled as unknown. We produce captions
for each image using a simple template-based generation method (see Appendix).
We align ADE20K images and their scene-level descriptions, to the
corresponding object-level captions in LN.

#### COCO

COCO Lin et al. (2014) consists of images paired with captions and object
annotations. LN captions are also available for the same images. We use images
and captions from the 2017 COCO validation split, as well as the corresponding
LN captions.

#### HL1K

High Level Scenes - 1k (HL1K) is a new dataset and part of an ongoing data
collection effort. HL1K is composed of 1k images, each depicting at least one
person, sampled from the 2014 COCO train split. We crowd-sourced three
annotations per image on Amazon Mechanical Turk, showing crowd workers the
image and asking them to write a description in response to the question Where
is the picture taken? It was made clear to annotators that the answer to this
question should bring to bear their knowledge of typical, or common, scenes.
Descriptions were corrected for typos using the Neuspell Toolkit Jayanthi et
al. (2020). Finally, we paired our scene-level HL1K captions with the
previously available COCO and LN object-level captions. Figure 1 is an
example; further examples are provided in the Appendix.

##  4 Experiments

We first test models in the image-text alignment task on both object- and
scene-level descriptions. We then delve more deeply into the performance of
the best model, using an ablation task. Since we are interested in the
capabilities of the pretrained models, we do not finetune them.

###  4.1 Image-sentence alignment

|  | lxmert | clip |  visualbert
---|---|---|---|---
Object | ADE20k + LN | 28.4 | 96.8 | 39.0
COCO + LN | 59.1 | 98.7 | 65.2
COCO Cap. | 79.3 | 99.1 | 64.4
Scene | ADE20k | 58.0 | 97.6 | 17.3
HL1K | 45.5 | 91.5 | 55.3
Table 2: Image-sentence alignment accuracies on object-level and scene-level
captions. Chance performance is at 50%. (LN = Localized Narratives)

For the image-text alignment task, we use the models’ pretrained alignment
head to predict whether a scene-level or object-level caption correctly
describes an image, or not.111Note that this setting is the same used by the
models is their pretraining. Table 2 shows that LXMERT and VisualBERT perform
adequately on object-level COCO Captions, though performance is lower than
expected given that they were pretrained on this dataset. In the case of
LXMERT, one possible explanation is catastrophic forgetting, arising from the
fact that this model is pretrained for its final ten epochs on VQA (similar
observations are made by Parcabalescu et al., 2021). For both models,
performance drops dramatically on LN captions. This is likely due to a
stylistic difference: LN captions are longer than COCO, more discursive and
contain disfluencies. In contrast, CLIP performs close to ceiling on all three
datasets, possibly reflecting the benefits accrued from the size and diversity
of its pretraining data.

On scene-level captions, performance is somewhat above chance for LXMERT on
ADE20k template-based descriptions, and for VisualBERT on HL1K. Otherwise,
performance is below 50% for both models. Once again, CLIP performs above 90%,
though there is a drop in performance from the template-based ADE20k
descriptions to human-authored HL1K scene-level captions, possibly due to the
more predictable nature of the former.

###  4.2 Ablation experiments on CLIP

Since CLIP is the only one of the three models which is successful at matching
scene-level and object-level captions to images, we probe its capabilities
further, paying particular attention to the question whether CLIP links scene
types (e.g. kitchen) to scene contents (e.g. oven, pizza) in image-text
matching.

Whereas a standard image-text alignment setup compares the model’s success at
identifying actual versus random captions, here we directly compare the
preference of the model for scene- versus object-level descriptions, as a
function of (i) the entities mentioned in the object-level caption; (ii) the
entities visible in an image. To this end, we use textual and visual ablation,
described below (see Appendix for further examples and details).

#### Textual ablation

A textual ablation is performed by removing noun phrases (NPs) in the object-
level caption. For example, for the COCO caption in Figure 1, one ablated
version would remove the NP a baseball player resulting in getting ready to
swing at a baseball game …. For a given image $i$ with object-level caption
$o$ and scene-level caption $s$, we compare how $P(o|i)$ – CLIP’s estimate of
the probability that $o$ matches $i$ – changes as NPs are removed from $o$,
and to what extent this causes CLIP to assign higher probability $P(s|i)$, to
$s$ as the match for $i$. We report two comparisons, one on LN captions versus
ADE20K scene template descriptions; and one on COCO captions against HL1K
scene-level descriptions.

To control for possible loss of grammaticality after ablation, we score
ablated captions with GRUEN Zhu and Bhat (2020), a BERT-based model which has
been shown to yield scores that correlate highly with human judgments.222GRUEN
returns a combined score consisting of a linear combinaton of Grammaticality,
Focus and Coherence. Here, we use only the Grammaticality scores. CLIP
probabilities for ablated textual captions yielded a very low correlation with
grammaticality (Pearson’s $r=0.1,p<.01$) suggesting that grammaticality did
not affect the scores.

#### Visual ablation

Given an image, we detect the bounding regions of entities which are also
mentioned in the corresponding object-level caption, and occlude them with a
greyscale mask. Once again, we are interested in whether CLIP’s estimate of
the alignment probability of object- versus scene-level captions, changes as
elements of the visual input are masked.

| ADE20k | HL1K
---|---|---
| LN | Scene | COCO | Scene
No ablation | 55.6 | 44.4 | 95.7 | 4.3
T | 22.0 | 78.0 | 67.2 | 32.8
V | 74.9 | 25.1 | 71.2 | 28.8
V+T | 68.4 | 31.6 | 63.3 | 36.7
Table 3: Preference expressed in percentage for object-level versus scene-
level (sub-columns), where there is no ablation, T(extual) ablation, V(isual)
ablation, or both (V+T) for a each dataset (columns).

#### Vision vs text

Without ablation, the model assigns higher probability to object-level
descriptions, suggesting that CLIP assigns higher confidence to an image-text
pair when the text focuses on objects. This preference is far more marked for
COCO/HL1K, in line with the observation (Table 2) that HL1K scene descriptions
are somewhat more challenging for this model. As entity-level information is
removed from the object-level caption (row T), the model’s preference swings
to the scene-level caption, suggesting that the model leverages the visual
information to align with the scene description. Visual ablation (V), by
contrast, results in the opposite tendency: when entities are occluded in the
image, the model shows greater preference for object-level captions. This
suggests that the way CLIP ‘understands’ scene-level descriptions depends on
which entities are visible. If both sources of information are ablated (V+T),
the preference once again veers towards object-level captions.

entity | $P(s|e)$
---|---
skateboard | 0.28
bench | 0.22
frisbee | 0.11
kite | 0.11
orange | 0.11
umbrella | 0.09
person | 0.07

Table 4: Top entities in park scenes.

Figure 2: CLIP scene-level description probabilities after masking top 1-3
entities. Error bars represent standard deviations.

#### Scenes vs. entities

These findings suggest that CLIP reasons about scenes on the basis of salient
objects within them. To investigate this further, we use scene labels
extracted from the HL1K captions and the object detections produced for the
visual ablation. For a scene label $s$ and entity label $e$, we compute
$P(s|e)$. Details of this computation are given in the Appendix. For example,
Table 4 shows the most likely entities in park scenes. For all images with at
least three detected entities, we consider the image-sentence alignment
probability assigned by CLIP to the scene-level description, when the top 1, 2
or 3 most likely entities in the scene are masked. Figure 2 displays a linear
trend, with the probability assigned by CLIP to the scene-level description
dropping as more likely entities are removed.333A one-way ANOVA comparing the
change in log probability as 1, 2 or 3 entities are removed showed that the
difference is significant ($F(2,156)=4.25,p<0.05)$. Thus, when CLIP aligns
images with scenes, it is relying on object-level information in the visual
modality. This explains why the removal of object mentions in text results in
higher preference for scene-level descriptions, since the objects are
detectable in the image. By the same token, masking objects in images causes
the model to rely more on the entity-level information in the text.

#### Single word comparison

So far, our analysis suggests that CLIP reasons about scenes based on object-
level information. However, the length of the caption might be a possible
confounding factor. Some of our results might simply be due to the model
assigning a higher alignment probability to a caption which is longer or more
informative. This would be an alternative explanation for the changes observed
above in the alignment probabilities after textual ablation.

To account for this, we replicate the alignment experiment using single words.
We use the scene labels extracted from HL1K scene-level descriptions and
identify the top three most likely entities in a given scene, as identified in
the previous experiment (see Figure 2). Given an image, we compare image-text
alignment probabilities in CLIP for single-word object labels (e.g. motorbike)
and single-word scene labels (e.g. road).

In this setting, CLIP displays a moderate preference for scene labels (63%),
suggesting that such labels are more informative than object-level labels, for
the one-word alignment task. Moreover, a qualitative analysis shows that CLIP
prefers object labels when the images have strongly foregrounded entities with
little background visibility, as shown in Fig 3.

resort: 3% person: 96%

resort: 99% snowboard: 1%

Figure 3: Scene vs entity one-to-one comparison. In the top image, there are
many people in the foreground and the entity person is preferred over the
scene label resort. At the bottom, people are snowboarding in the background
and the scene label is preferred over the the entity (snowboard).

##  5 Related work

Large, pretrained models are often analysed via probe tasks or through an
investigation of their attention heads (see Belinkov and Glass, 2019, for a
survey). For example, Li et al. (2020b) consider VisualBERT’s attention heads
in a manner similar to Clark et al. (2019), showing that it is able to ground
entities and syntactic relations (see also Ilharco et al., 2020; Dahlgren
Lindström et al., 2020). Hendricks and Nematzadeh (2021) similarly seek to
obtain an in-depth understanding of the representations learned by V&L models,
finding that they fail to ground verbs in visual data, compared to other
morphosyntactic categories.

The ability of V&L models to reason with a combination of linguistic and
visual cues is explored through tasks such as VCR Zellers et al. (2019), SWAG
Zellers et al. (2018) and NLVR Suhr et al. (2017, 2019). Pezzelle et al.
(2020), in work complementary to our own, address the relationship between
visual and textual modalities, exploring a task in which the text does not
provide an object-level description of an image. In work presented
concurrently with our own, Frank et al. (2021) propose an ablation-based
methodology to evaluate the extent to which multimodal models rely on visual
and/or textual information to perform grounding.

Although scene recognition is a central task in computer vision, there has
been little work exploring the capabilities of models to link scene- and
object-level representations. Some precedent for the concerns addressed in
this paper are found in the image captioning literature. For example, an
influential proposal by Anderson et al. (2018) combines top-down and bottom-up
attention to improve the quality of image descriptions, while CapWAP Fisch et
al. (2020) conditions image captioning on questions that determine which
information is relevant to current communicative needs, going beyond object-
level description. Closer to the scope of the work presented here, a recent
pretrained V&L model, SemVLP Li et al. (2021), combines single- and dual-
streams for feature-level and high-level semantic alignment. We plan to
investigate this model further in future work.

##  6 Conclusions

Grounding language in vision requires V&L models to capture the relation
between ‘high-level’ characterisations of scenes and the entities they
contain. The experiments in this paper suggest that when models do this, they
rely on object-level information in the visual modality, to link images to
scene descriptions in the textual modality. This is partly dependent on the
probability of entities occurring in scenes.

However, this is not an ability that all models have: LXMERT and VisualBERT
perform poorly on scene-level descriptions and also on object-level captions
which are stylistically different from the ones they were pretrained on,
whereas CLIP handles both object- and scene-level captions. We note that for
LXMERT and VisualBERT, testing on ADE20k, Localized Narratives and HL1K
amounts to a zero-shot setting. With the exception of HL1K, a new dataset, it
is an open question whether this is true of CLIP. Since this model is trained
on web-scale data, it is difficult to ascertain that the training and test
data in our experiments are disjoint; as Bender et al. (2021) recently argued,
with such pre-training strategies, training data is often unfathomable. On the
other hand, our findings also show that, at least as far as object and scene-
level grounding is concerned, model size is not a determining factor. As shown
in Table 1, CLIP is smaller in terms of number of parameters than LXMERT.

Apart from the sheer volume of pre-training data used in CLIP, we believe that
two additional factors contribute to its success. First, its contrastive
learning objective may result in greater sensitivity to fine-grained
distinctions between captions, when the model computes the likelihood with
which a text matches a given image. A second important feature of CLIP is its
visual backbone, which (in the version used in this paper) is based on Visual
Transformer (ViT Dosovitskiy et al., 2020). By contrast, the visual component
in LXMERT and VisualBERT is CNN-based. Recent work by Tuli et al. (2021) has
shown that the attention-based ViT architecture may be more consistent with
characteristics of human vision than a convolutional network. This could
partially underlie the model’s ability to use object-level information in an
image to align to scene-level captions. The extent to which the visual
backbone of V&L models impacts their grounding capabilities is a topic that
should be further explored.

##  7 Ethical considerations

For the studies presented here, we used a new dataset, HL1K, collected using
the Amazon Mechanical Turk crowdsourcing platform. For the data collection,
participants were shown images and asked to answer questions such as Where is
the picture taken? Answers took the form of short statements. Workers were
paid at the rate of €0.03 per item, an amount we consider equitable for the
work involved, and in line with rates for similar tasks. No sensitive or
identifying information was collected. All other data and models used are
publicly available. The HL1K dataset will be made available upon publication.

## Acknowledgments

This project has received funding from the European Union’s Horizon 2020
research and innovation programme under the Marie Skłodowska-Curie grant
agreement No 860621.

## References

  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6077–6086.
  * Belinkov and Glass (2019) Yonatan Belinkov and James Glass. 2019.  Analysis Methods in Neural Language Processing: A Survey.  _Transactions of the Association for Computational Linguistics_ , 7:49–72.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?  In _Proceedings of the fourth ACM Conference on Fairness, Accountability, and Transparency (FAccT’21)_ , Online. Association for Computing Machinery.
  * Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020.  Climbing towards NLU: On meaning, form, and understanding in the age of data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5185–5198, Online. Association for Computational Linguistics.
  * Biederman et al. (1982) Irving Biederman, Robert J. Mezzanotte, and Jan C. Rabinowitz. 1982.  Scene perception: Detecting and judging objects undergoing relational violations.  _Cognitive Psychology_ , 14(2):143–177.
  * Bisk et al. (2020) Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020.  Experience grounds language.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8718–8735, Online. Association for Computational Linguistics.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.  _ArXiv_ , 2005.14165.
  * Bugliarello et al. (2020) Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott. 2020\.  Multimodal pretraining unmasked: Unifying the vision and language berts.  _arXiv preprint arXiv:2011.15124_.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-yi Lin, Ramakrishna Vedantam, C Lawrence Zitnick, Saurabh Gupta, and Piotr Doll. 2015.  Microsoft COCO Captions : Data Collection and Evaluation Server.  _arXiv_ , 1504.00325:1–7.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Uniter: Universal image-text representation learning.  In _ECCV_.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019.  What does BERT look at? An analysis of BERT’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 276–286, Florence, Italy. Association for Computational Linguistics.
  * Dahlgren Lindström et al. (2020) Adam Dahlgren Lindström, Johanna Björklund, Suna Bensch, and Frank Drewes. 2020.  Probing multimodal embeddings for linguistic properties: the visual-semantic case.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 730–744, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beye, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020.  An image is worth 16x16 words: Transformers for image recognitoin at scale.  _arXiv_ , 2010.11929.
  * Fisch et al. (2020) Adam Fisch, Kenton Lee, Ming-Wei Chang, Jonathan Clark, and Regina Barzilay. 2020\.  CapWAP: Image captioning with a purpose.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8755–8768, Online. Association for Computational Linguistics.
  * Frank et al. (2021) Stella Frank, Emanuele Bugliarello, and Desmond Elliott. 2021.  Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers.  _ArXiv_ , 2109.04448.
  * Harnad (1990) Stevan Harnad. 1990.  The symbol grounding problem.  _Physica_ , D42(1990):335–346.
  * Hendricks and Nematzadeh (2021) Lisa Anne Hendricks and Aida Nematzadeh. 2021.  Probing Image-Language Transformers for Verb Understanding.  _ArXiv_ , 2106.09141.
  * Huang et al. (2021) Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. 2021.  Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning.  _ArXiv_ , 2104.03135.
  * Ilharco et al. (2020) Gabriel Ilharco, Rowan Zellers, Ali Farhadi, and Hannaneh Hajishirzi. 2020.  Probing Contextual Language Models for Common Ground with Visual Representations.  _arXiv_ , 2005.00619.
  * Itti and Koch (2001) L Itti and C Koch. 2001.  Computational modelling of visual attention. _Nature reviews neuroscience_ , 2(3):194–203.
  * Jayanthi et al. (2020) Sai Muralidhar Jayanthi, Danish Pruthi, and Graham Neubig. 2020.  Neuspell: A neural spelling correction toolkit.  _arXiv preprint arXiv:2010.11085_.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  _arXiv preprint arXiv:2102.03334_.
  * Li et al. (2021) Chenliang Li, Ming Yan, Haiyang Xu, Fuli Luo, Wei Wang, Bin Bi, and Songfang Huang. 2021.  SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels.  _arXiv preprint_ , 2103.07829.
  * Li et al. (2020a) Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020a.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 11336–11344. AAAI Press.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019\.  Visualbert: A simple and performant baseline for vision and language.  In _Arxiv_.
  * Li et al. (2020b) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2020b.  What Does BERT with Vision Look At? In _Proceedings ofthe 58th Annual Meeting ofthe Association for Computational Linguistics (ACL’20)_ , pages 5265–5275, Online. Association for Computational Linguistics.
  * Li et al. (2020c) Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020c.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _European Conference on Computer Vision_ , pages 121–137. Springer.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _Advances in Neural Information Processing Systems_ , pages 13–23.
  * Lu et al. (2020) Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020.  12-in-1: Multi-task vision and language representation learning.  In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Luo et al. (2020) Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Xilin Chen, and Ming Zhou. 2020.  UniVL: A unified video and language pre-training model for multimodal understanding and generation.  _arXiv_.
  * Ma et al. (2010) Hao Ma, Jianke Zhu, Michael Rung Tsong Lyu, and Irwin King. 2010.  Bridging the semantic gap between image contents and tags.  _IEEE Transactions on Multimedia_ , 12(5):462–473.
  * Malcolm et al. (2016) George L. Malcolm, Iris I.A. Groen, and Chris I. Baker. 2016.  Making Sense of Real-World Scenes.  _Trends in Cognitive Sciences_ , 20(11):843–856.
  * Oliva and Torralba (2007) Aude Oliva and Antonio Torralba. 2007.  The role of context in object recognition. _Trends in cognitive sciences_ , 11(12):520–527.
  * Parcabalescu et al. (2021) Letitia Parcabalescu, Albert Gatt, Annette Frank, and Iacer Calixto. 2021.  Seeing past words: Testing the cross-modal capabilities of pretrained v&l models on counting tasks.  In _Proceedings of the Workshop Beyond Language: Multimodal Semantic Representations (MMSR’21)_ , Groningen, The Netherlands.
  * Pezzelle et al. (2020) Sandro Pezzelle, Claudio Greco, Greta Gandolfi, Eleonora Gualdoni, and Raffaella Bernardi. 2020.  Be Different to Be Better! A Benchmark to Leverage the Complementarity of Language and Vision.  In _Findings ofthe Association for Computational Linguistics: EMNLP 2020_ , pages 2751–2767, Online. Association for Computational Linguistics.
  * Pont-Tuset et al. (2019) Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. 2019.  Connecting Vision and Language with Localized Narratives.  _arXiv_ , 1912.03098.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning Transferable Visual Models From Natural Language Supervision.  _arXiv preprint_ , 2103.00020.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  In _Advances in Neural Information Processing Systems 28 (NeurIPS 2015)_ , Montreal, Canada.
  * Su et al. (2020) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020\.  Vl-bert: Pre-training of generic visual-linguistic representations.  In _International Conference on Learning Representations_.
  * Suhr et al. (2017) Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017.  A Corpus of Natural Language for Visual Reasoning.  In _Proceedings ofthe 55th Annual Meeting ofthe Association for Computational Linguistics (ACL’17)_ , pages 217–223, Vancouver, BC. Association for Computational Linguistics.
  * Suhr et al. (2019) Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019\.  A Corpus for Reasoning about Natural Language Grounded in Photographs.  _arXiv_ , 1811.00491:6418–6428.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  LXMERT: Learning cross-modality encoder representations from transformers.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5100–5111, Hong Kong, China. Association for Computational Linguistics.
  * Torralba et al. (2006) Antonio Torralba, Aude Oliva, Monica S. Castelhano, and John M. Henderson. 2006\.  Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search. _Psychological Review_ , 113(4):766–786.
  * Tuli et al. (2021) Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Thomas L. Griffiths. 2021.  Are Convolutional Neural Networks or Transformers more like human vision? _arXiv_ , 2105.07197.
  * Võ and Wolfe (2013) Melissa L.H. Võ and Jeremy M. Wolfe. 2013.  Differential Electrophysiological Signatures of Semantic and Syntactic Scene Processing.  _Psychological Science_ , 24(9):1816–1823.
  * Wang et al. (2020) Jianfeng Wang, Xiaowei Hu, Pengchuan Zhang, Xiujun Li, Lijuan Wang, Lei Zhang, Jianfeng Gao, and Zicheng Liu. 2020.  MiniVLM: A Smaller and Faster Vision-Language Model.  _arXiv_ , 2012.06946.
  * Zellers et al. (2019) Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  From recognition to cognition: Visual commonsense reasoning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6720–6731.
  * Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018.  SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference.  _arXiv_ , 1808.05326.
  * Zhou et al. (2017) Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017.  Scene parsing through ade20k dataset.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 633–641.
  * Zhu and Bhat (2020) Wanzheng Zhu and Suma Bhat. 2020.  GRUEN for evaluating linguistic quality of generated text.  _arXiv_ , 2010.02498.

##  Appendix A Dataset details

Table 5 gives the sizes of the ADE20k, HL1K and COCO datasets. For ADE20k, the
numbers are for images which are not labelled as having an unknown scene. For
COCO captions, there are five captions associated with each image. For the
purposes of the present study, a single caption is randomly selected for each.

| ADE20k | HL1K | COCO
---|---|---|---
images | 19733 | 1000 | 5000
COCO captions | 19733 | 3000 | 5000
Localized Narratives | 19733 | 1000 | 5000
Table 5: Datasets sizes

Table 6 provides the number of ablations analysed in the study. For both
ADE20k and HL1k we obtain a number of ablated captions which is greater than
the respective dataset sizes in Table 5, because for each example we generate
all the possible combinations of noun phrases. For the Visual and
Visual+Textual ablations, the number of ablated instances is lower than the
dataset size, because we skip all the images where no object is detected.

| ADE20k | HL1K
---|---|---
T | 205k | 10027
V | 10788 | 625
T+V | 1078 | 625
Table 6: Total number of ablations generated per dataset, across all the
ablations experiments

###  A.1 Scene description templates

ADE20K scene labels are converted to textual descriptions using three
different templates:

  * •

it is a scene

  * •

this is a scene

  * •

it is located in scene

A scene label is rendered into a description using a randomly selected
template from the above. For example:

  * •

bathroom –> it is a bathroom

  * •

bedroom –> this is a bedroom

  * •

airport –> it is located in an airport

###  A.2 HL1K Examples

HL1K is collected by asking crowd workers to consider an image, and answer the
question Where is the picture taken? Crowd workers are asked to respond using
full sentences, as far as possible. Since images are selected from the COCO
2014 train split, they are also accompanied by their object-level COCO
captions and Localized Narratives. An example image with corresponding HL1K
scene-level descriptions is shown below.

Where is the picture taken?

  * •

in a bedroom

  * •

the picture is taken in a bedroom

  * •

this is the bedroom

Figure 4: COCO image with corresponding scene descriptions in HL1K

##  Appendix B Experiment details

###  B.1 Image-sentence alignment

Results for image-sentence alignment experiments in the paper are averages
over three separate runs for each model.

For LXMERT, we use the image-sentence alignment
head.444github.com/huggingface/transformers CLIP and LXMERT are tested on the
standard alignment task: given an image and either the correct caption, or a
random caption, the model needs to determine whether the caption correctly
aligns with the image.

We use the publicly available implementation of
VisualBERT.555https://github.com/uclanlp/visualbert The image-sentence
alignment setting for this model is somewhat different, since alignment is
modelled as an extension of the next-sentence prediction task in unimodal
BERT. VisualBERT takes an image and a correct caption, together with a second
caption, which may be correct or randomly selected. The task is to predict
whether the second caption correctly aligns with the image+caption pair.

For all experiments, we truncate textual captions to a maximum length of 50
tokens, following standard practice for such models, including CLIP.

###  B.2 Ablation

Ablation is performed on object-level captions, and on images. Results for
ablation are reported based on a single run of CLIP. Figure 5 shows an
original image and its object-level caption, together with the visual and
textual ablations. Below, we explain the process of ablation in more detail.

Original Image

Occluded Image

COCO: A man rides a motorcycle on a road through a grassy, hilly area.

Ablated Captions:

  * •

a grassy, hilly area (A man, a motorcycle, a road)

  * •

a road (A man, a motorcycle, a grassy, hilly area)

  * •

a road through a grassy, hilly area (A man, a motorcycle)

  * •

A man rides a road (a motorcycle, a grassy, hilly area)

  * •

a motorcycle a grassy, hilly area (A man, a road)

  * •

A man rides a grassy, hilly area (a motorcycle, a road)

  * •

A man rides a motorcycle (a road, a grassy, hilly area)

  * •

A man rides a road through a grassy, hilly area (a motorcycle)

  * •

a motorcycle on a road (A man, a grassy, hilly area)

  * •

A man rides a motorcycle on a road (a grassy, hilly area)

  * •

A man rides a motorcycle a grassy, hilly area (a road)

  * •

a motorcycle on a road through a grassy, hilly area. (A man)

Figure 5: The original image at the top, in the middle the visually ablated
imaged and at the bottom the original caption and all the cases generated by
the constituent textual ablation.

#### Textual ablation

Given an object-level caption, we identify all the NPs in the caption and
create new versions by removing each possible subset of the set of NPs, with
the restriction that the resulting caption must always contain at least one
NP. When NP removal results in dangling predicates, we remove them to preserve
grammaticality. NPs are detected using Spacy v.3 pipeline for English using
the en_core_web_md pretrained models.

#### Visual ablation

Given an object-level caption and an image, we extract all nouns from the
caption and extract the embedding vector for each noun using the pretrained
FastText embeddings.666We use the model with 2m word vectors trained with
subword information from common Crawl https://fasttext.cc/docs/en/english-
vectors.html We pass the image through the Faster-RCNN object
detector777Faster R-CNN ResNet-50 FPN pre-trained on COCO, available from the
torchvision module in Pytorch to detect entities. We extract embeddings for
each entity label. Then, we identify regions to be masked by comparing
embeddings for entity labels $l_{e}$ against embeddings for nouns $n_{e}$ in
the caption, considering them a match if $\text{cosine}(l_{e},n_{e})\geq 0.7$.
Bounding box regions corresponding to matched entities are occluded with a
greyscale mask.

##  Appendix C Scene-entity probabilities

(a) Scene: kitchen

(b) Scene: road

(c) Scene: room

(d) Scene: park

Figure 6: Visualisations of entities in four different scene types. For a
given entity $e$ in scene $s$, font size is proportional to $P(s|e)$

Entity probabilities are computed on the basis of the entities detected in the
image and object-level caption, using the process described above for visual
ablation. Let $e$ be an entity detected $n_{e}$ times in the dataset, of which
$n_{e,s}$ times in images depicting scene $s$. We compute:

$P(s|e)=\frac{n_{e,s}}{n_{e}}$

Figure 6 shows visualisations for entities detected in four different scene
types, from the HL1K dataset. To compute the mean probabilities assigned by
CLIP to scene-level descriptions after entities are visually ablated, we
average over those images containing at least three detected entities (53/174
total scenes).


Jiehua<EMAIL_ADDRESS>Zhuo<EMAIL_ADDRESS>Li Liu111 indicates
the corresponding<EMAIL_ADDRESS>Center for Machine Vision and
Signal Analysis
University of Oulu
Finland College of System Engineering
National University of Defense Technology
China MeDiNet for Robust Face Recognition

# Median Pixel Difference Convolutional Network for Robust Face Recognition

###### Abstract

Face recognition is one of the most active tasks in computer vision and has
been widely used in the real world. With great advances made in convolutional
neural networks (CNN), lots of face recognition algorithms have achieved high
accuracy on various face datasets. However, existing face recognition
algorithms based on CNNs are vulnerable to noise. Noise corrupted image
patterns could lead to false activations, significantly decreasing face
recognition accuracy in noisy situations. To equip CNNs with built-in
robustness to noise of different levels, we proposed a Median Pixel Difference
Convolutional Network (MeDiNet) by replacing some traditional convolutional
layers with the proposed novel Median Pixel Difference Convolutional Layer
(MeDiConv) layer. The proposed MeDiNet integrates the idea of traditional
multiscale median filtering with deep CNNs. The MeDiNet is tested on the four
face datasets (LFW, CA-LFW, CP-LFW, and YTF) with versatile settings on blur
kernels, noise intensities, scales, and JPEG quality factors. Extensive
experiments show that our MeDiNet can effectively remove noisy pixels in the
feature map and suppress the negative impact of noise, leading to achieving
limited accuracy loss under these practical noises compared with the standard
CNN under clean conditions.

##  1 Introduction

Face Recognition is one of the most important research fields in computer
vision and pattern recognition. Recent advances in deep learning, coupled with
abundant face data, have led to excellent progress in face recognition
algorithms [Guo and Zhang(2019), Liu et al.(2017)Liu, Wen, Yu, Li, Raj, and
Song, Schroff et al.(2015)Schroff, Kalenichenko, and Philbin, Sun et
al.(2014a)Sun, Chen, Wang, and Tang, Sun et al.(2014b)Sun, Wang, and Tang, Wen
et al.(2016)Wen, Zhang, Li, and Qiao, Taigman et al.(2014)Taigman, Yang,
Ranzato, and Wolf]. Due to these achievements, face recognition technology is
widely utilized in the real world, such as human-computer interaction[Lin et
al.(2019)Lin, Cheng, Shen, and Pantic], video surveillance[Chen et
al.(2019)Chen, Li, Deng, Li, and Yu], and identification [Kamgar-Parsi and
Lawson(2011), Vezzani et al.(2013)Vezzani, Baltieri, and Cucchiara]. The
accuracy is a crucial metric to evaluate the effectiveness of recognition
models. However, the existing noise produced by the sensor of a scanner,
cameras, image compression, etc., results in a significant decrease in the
accuracy of models. Enhancing the noise-robustness is meaningful to the
practical applications of face recognition.

Recently, many studies have been carried out to deal with the challenges of
face recognition, such as the variations in expressions, lighting conditions,
and poses [Cao et al.(2018)Cao, Rong, Li, Tang, and Loy, Lahasan et
al.(2019)Lahasan, Lutfi, and San-Segundo, Li et al.(2018b)Li, Song, Wu, He,
and Tan], with few efforts taking noise into account. The existing noise-
robust face recognition methods mainly consider the noisy labels in the face
dataset [Han et al.(2018)Han, Yao, Yu, Niu, Xu, Hu, Tsang, and Sugiyama, Hu et
al.(2019)Hu, Huang, Zhang, and Li, Veit et al.(2017)Veit, Alldrin, Chechik,
Krasin, Gupta, and Belongie, Wang et al.(2018)Wang, Chen, Li, Huang, Chen, C.,
and Loy, Zhang et al.(2020)Zhang, Deng, Wang, Hu, Li, Zhao, and Wen]. This
kind of method is hard to adapt to the diversity and complexity of noise in
the real world. Our purpose is to improve the robustness of face detectors to
noise in the real world.

In this paper, we propose an effective approach to enhance the noise-
robustness of CNNs with limited decreasing accuracy compared with standard
CNNs. The existing CNNs exhibit sensitivity to noise, particularly salt-and-
pepper noise. Inspired by [Liu et al.(2016)Liu, Lao, Fieguth, Guo, Wang, and
Pietikäinen], we proposed a robust Median Pixel Difference Convolutional
Network (MeDiNet), which can effectively suppress noise and improve the
performance under noisy conditions by median pixel difference convolution
(MeDiConv). Specifically, MRELBP replaces the individual pixel intensities in
the feature maps with median representations in the corresponding local
regions and reports excellent results in the texture classification task under
various noises, showing that combining local median representation is powerful
to extract features. The proposed MeDiConv inhabits noise-robust property of
MRELBP.

We summarize our contributions as follows: (i) We proposed a noise-robust
network MeDiNet integrating a novel convolutional operation named as MeDiConv
to utilize median representation to suppress noise in images effectively. (ii)
We adopt the image degradation model with a broad range of noises to
approximate the noisy model in the real world (blur kernel, noise intensity,
scaling, JPEG quality), which can evaluate the performance of our method in
practical application. (iii) The model can be robust to a variety of noises
without generating additional noisy images for training the model.

In the experiment, MeDiNet is tested on four face datasets with versatile
settings. For approximating the complex noise model in the real world, we also
add two complex noises to evaluate our method except salt-and-pepper noise and
additive white Gaussian noise (AWGN): Heteroscedastic Gaussian (HG) [Plotz and
Roth(2017)] and multivariate Gaussian (MG) noise [Zhang et al.(2018)Zhang,
Zhuo, and Zhang]). The noisy image examples are shown in Figure 1. Our
experimental results show that the MeDiConv can effectively suppress the
negative impact of the noise on the images. We demonstrate that MeDiNet
reports superior performance to standard CNN architectures on four face
datasets under noisy conditions (LFW [Huang et al.(2007)Huang, Ramesh, Berg,
and E.], CA-LFW [Zheng et al.(2017)Zheng, Deng, and Hu], CP-LFW [Zheng and
Deng(2018)], Youtube Face [Wolf et al.(2011)Wolf, Hassner, and Maoz]) while
the loss of accuracy on the clean images is limited.

Figure 1: The image example with versatile settings. (b) Gaussian blur,
$\sigma$=3, kernel size=13; (c) Gaussian noise, $\sigma$=20; (d) JPEG
Compressor, JPEG quality=10.

##  2 Related work

###  2.1 Face recognition in noisy conditions

With the development of deep learning, a large number of face recognition
algorithms have been proposed to address problems like lighting, age, pose
[Cao et al.(2018)Cao, Rong, Li, Tang, and Loy, Zheng and Deng(2018), Zheng et
al.(2017)Zheng, Deng, and Hu]. Noise in human face images can significantly
decrease the performance of face recognition systems. Noise drops the valuable
information in the image and influences the ability of some algorithms to
recognize a correct object. To suppress the impact of noise, Ding _et al_[Ding
et al.(2017)Ding, Cheng, Cheng, and Li] proposed the NR-Network for face
recognition under noise, which utilized a multi-input structure to extract a
multi-scale and more discriminative feature from the input image. Wu _et
al_[Wu et al.(2020)Wu, Liu, Cao, Ren, and Zuo] proposed a denoising network
D-BSN that is designed as a two-stage training framework with self-supervised
learning and knowledge distillation to learn the denoising from unpaired clean
and noisy images. Anwar _et al_[Anwar and Barnes(2019)] firstly incorporate an
attention mechanism to exploit channel dependencies in the denoising task. Li
_et al_[Li et al.(2018a)Li, Liu, Ye, Zuo, Lin, and Yang] proposed GFRNet
consisting of warping and a reconstruction network. The GFRNet combined the
degraded and high-quality images to enhance the performance of face
restoration. The existing studies for suppressing noise focus on the
restoration tasks [Abdelhamed et al.(2018)Abdelhamed, Lin, and Brown, Anwar
and Barnes(2019), Chrysos and Zafeiriou(2017), Li et al.(2018a)Li, Liu, Ye,
Zuo, Lin, and Yang, Wu et al.(2020)Wu, Liu, Cao, Ren, and Zuo, Yu and
Porikli(2017)]. These methods depend on training with noisy and clean images.
We aim at designing a general framework without extra noisy image data to
suppress noise in the face recognition task directly.

###  2.2 LBP and convolution

LBP [Ojala et al.(2002)Ojala, Pietikäinen, and Maenpaa] was proposed for
texture analysis and has been successfully applied to many tasks [Liu et
al.(2018)Liu, Chen, Fieguth, Zhao, Chellappa, and Pietikäinen], such as image
retrieval [Vipparthi et al.(2015)Vipparthi, Murala, Nagar, Li, and Gonde],
object detection [Liu et al.(2020)Liu, Ouyang, Wang, Fieguth, Chen, Liu, and
Pietikäinen] and face image analysis [Huang et al.(2011)Huang, Shan,
Ardabilian, Wang, and Chen, Su et al.(2019)Su, Pietikäinen, and Liu]. LBP has
emerged as one of the most prominent texture descriptors. Recently, some
studies tried to combine LBP with standard CNNs [Su et al.(2021)Su, Liu, Yu,
Hu, Liao, Tian, Pietikäinen, and Liu, Yu et al.(2020)Yu, Zhao, Wang, Qin, Su,
Li, Zhou, and Zhao]. The LBCNN was motivated by LBP and utilized a set of
fixed sparse pre-defined binary convolutional filters to achieve computational
savings [Juefei-Xu et al.(2017)Juefei-Xu, Boddeti, and Savvides]. Inspired by
LBCNN, the LBVCNN added the LBV layer to reduce trainable parameters in facial
expression task [Kumawat et al.(2019)Kumawat, Verma, and Raman]. Li _et al_[Li
et al.(2020)Li, Jin, Kubota, and Ju] combined the LBP and attention mechanism
to extract useful features, which achieved higher performance in face
recognition. These studies demonstrated that combining LBP with standard CNNs
is a practical and flexible approach in various tasks. The MRELBP is based on
combining a median filter with LBP operation in a multiscale fashion, which
can effectively extract noise-resistant patterns. The development of MeDiNet
shares similar motivations but has slightly different implementation due to
the different processing procedures of LBP and CNN.

##  3 MeDiNet

The MeDiNet is built on the Sphere20 [Liu et al.(2017)Liu, Wen, Yu, Li, Raj,
and Song]. The initial convolutional layers of Sphere20 are replaced with
MeDiConv for suppressing the negative impact of a broad range of noises in the
face images. The operation of MeDiConv is introduced as follows.

###  3.1 Preliminary of MRELBP

Figure 2: The convolutional operation of Median Pixel Difference Convolution.
$\mu_{i}$ denotes a global predefined value of each channel.

The traditional LBP calculates the differences between the pixel value of the
central point and those of its neighbors. The MRELBP [Liu et al.(2016)Liu,
Lao, Fieguth, Guo, Wang, and Pietikäinen] set the pixel intensity as median
representation in a local patch, which is more robust to gray scale
variations, rotation changes, and noise. We choose the MRELBP descriptor under
center pixel representation, a simple and effective way to handle noise. This
descriptor can be expressed as:

$MRELBP\\_CI(x_{c})=s(\phi(X_{c,w})-\mu_{w})$ (1)

where $s$ is the sign function, $\phi(X_{c,w})$ denotes the median operation
in $X_{c,w}$ (the local patch of size $w\times w$ centered at center pixel
$x_{c}$), $\mu$ denotes the mean value of $\phi(X_{i,w})$ over the whole image
$X_{i}$. Inspired by this work, we use a set of learnable weight parameters to
replace sign function for combining the standard CNN with median
representation. The MRELBP is based on combining a median filter with
traditional LBP operation, which can effectively extract powerful features
under various noisy conditions. Our MeDiConv also utilize median
representation in convolutional operation. This processing can make
convolutional layer suppress noisy pixels in forward propagation.

###  3.2 MeDiConv

The calculation process of standard CNN can be described as: the convolutional
kernel $W^{c_{out}\times c_{in}\times k\times k}$ slides on the feature map
$X^{c_{in}\times H\times W}$ to calculate the output. The formulation can be
expressed as following ($c_{out}$ is set to 1 without loss of generality).

$X_{out}(u,v)=\sum_{i=0}^{c_{in}-1}\sum_{(p,q)\in\mathcal{N}}X_{i}(u+p,v+q)\cdot
W_{i}(p,q)$ (2)

where $(u,v)$ denotes the coordinate of the pixel in feature map, $W_{i}$
denotes the convolutional kernel, and $\mathcal{N}$ denotes the $k\times k$
neighborhood area centered at $(u,v)$. The standard CNNs are sensitive to
noise (especially salt-and-pepper noise) since the noise in the local region
can confuse the subsequent feature representation process and lead to the
detection of false features with no physical correlation.

To suppress noise in regions of feature maps, MeDiConv utilizes local median
pixel difference value to replace the original value of $x_{u,v}$ in the
convolutional operation. As shown in Figure 2, the value of each pixel $x_{i}$
in the input feature patch is replaced by the median value of the 3x3
neighborhood centered on $x_{i}$. The procedure can be decomposed as the
following stages: 1) for each location $(u,v)$ with its neighborhood region
$\\{(u+p,v+q)|(p,q)\in\mathcal{N}\\}$, calculating the median representation
for each pixel located in the neighborhood region; 2) replacing the pixel
intensity $X_{i}(u+p,v+q)$ in Eq. 2 with the difference between the calculated
median representation of $X_{i}(u+p,v+q)$ and a global predefined value $\mu$;
3) combining the output in stage 2 with the weights to generate the output
feature map. Therefore, MeDiConv can be expressed as:

$\tilde{X}_{out}(u,v)=\sum_{i=0}^{c_{in}-1}\sum_{(p,q)\in\mathcal{N}}(\phi(X_{i}(u+p,v+q))-\mu)\cdot
W_{i}(p,q)$ (3)

$\phi(X_{i}(u,v))=Median(\\{X_{i}(u+p,v+q)|(p,q)\in\mathcal{N}\\})$ (4)

where $\tilde{X}_{out}$ is the output feature map, $\phi(X_{i}(u,v))$ denotes
the median representation of the $k\times k$ region centered at $X_{i}(u,v)$,
and $\mu$ represents the average value of $\phi(X_{i}(u,v))$ over the whole
feature map. The aim of subtracting $\mu$ is to eliminate the bias of pixel
intensities in the feature map.

Compared with standard convolution, the MeDiConv is a nonlinear smoothing
operation, which can effectively remove outliers with limited impact on the
ability of feature extraction [Ataman et al.(1981)Ataman, Aatre, and Wong].
Noise corrupted image patterns could lead to false activations, leading to a
significant decrease in accuracy. For MeDiConv, the effect of applying
MeDiConv at multiple layers can be considered as applying multiple median
filters of different kernel sizes on the original image as each MeDiConv layer
has a different receptive field. That is also why the proposed MeDiNet can
deal with noise of different levels. Considering the trade-off between
accuracy and noise robustness, MeDiNet only set MeDiConv in the first few
layers. When the standard CNN in the deep layer of the network is replaced by
MeDiConv, the response of the feature map will be degraded. Setting MeDiConv
in the beginning layers is sufficient to eliminate the impact of noise.

##  4 Experiments

###  4.1 Dataset

We adopt the CASIA-WebFace [Yi et al.(2014)Yi, Lei, and Li] as a training
dataset. The LFW [Huang et al.(2007)Huang, Ramesh, Berg, and E.], CA-LFW
[Zheng et al.(2017)Zheng, Deng, and Hu], CP-LFW [Zheng and Deng(2018)],
Youtube Face [Wolf et al.(2011)Wolf, Hassner, and Maoz] are used for
evaluation. The details of datasets are as following:

CASIA-WebFace. The CASIA-WebFace dataset is collected from IMDb website
containing 10,575 subjects and 494,141 images. The collection and annotation
of CASIA-WebFace use a semi-automatical way. It also utilizes the name bag and
face similarity for efficient labeling.

LFW. Labeled Faces in the Wild (LFW) aims to study face recognition in an
unconstrained environment, containing more than 13,000 face images from the
website. Each face in the LFW is marked with a person’s name, and about 1680
people contain more than two faces. Although there are some mislabeled matched
pairs, LFW is considered completely clean.

CA-LFW and CP-LFW. CA-LFW and CP-LFW are renovations of LFW dataset. These
datasets focus on the cross-age and cross-pose face challenges in face
recognition. The images are collected from 3,000 positive face pairs with age
gap and pose gap, respectively. Compared with the LFW, both datasets are more
challenging with lower test accuracy.

YouTube Faces. The YouTube Faces Dataset (YTF) is a facial video database used
to study unconstrained face recognition in videos. The dataset contains 3,425
videos from 1,595 different people. All the videos are collected from YouTube.
Each subject provides an average of 2.15 videos. The shortest clip length is
48 frames, the longest clip length is 6070, and the average video clip length
is 181.3.

###  4.2 Degradation model

In the practical application, the quality of images can be affected by noise,
scaling, compression, and their combination. For verifying the robustness of
MeDiNet, a degradation model is required to generate realistic noisy images
[Li et al.(2018a)Li, Liu, Ye, Zuo, Lin, and Yang], which can be defined as:

$I^{d}=((I\otimes k)_{\downarrow_{S}}+n)_{JPEG_{C}}$ (5)

where $\otimes$ means the convolutional operation. $k$ denotes the blur
kernel. $\downarrow_{S}$ denotes the downsampling operation with the scale
factor $S$. $n$ denotes the noise like AWGN. $(\cdot)_{JPEG_{\mathcal{C}}}$
denotes the JPEG compression with the quality factor $c$. The degradation
$I^{d}=((I\otimes k)_{\downarrow_{S}}+n)$ is caused by remote acquisition and
$(\cdot)_{JPEG_{\mathcal{C}}}$ expresses the effect of JPEG compression.

The parameter settings of our degradation model is shown as below:

Blur kernel. In our paper, we utilize Gaussian blur kernel and Motion blur
kernel as the Simulation of defocusing effect. The variance $\varrho$ of
Gaussian blur is sampled from the set $\left\\{1:1:3\right\\}$. The degree
$\mathcal{D}$ of Motion blur is sampled from the set
$\left\\{10:10:40\right\\}$. 0 means no blur operation.

Downsampler. The scale factor $\mathcal{S}$ is sampled from the set
$\left\\{2,4,6,8\right\\}$. It should be noted that the image will be
upsampled to its original size after downsampling.

Noise. Besides common noise (like AWGN and salt-and-pepper noise), we select
more complex noise to evaluate our model, including HG
$n_{i}\sim\mathcal{N}(0,\alpha^{2}x_{i}+\delta^{2})$ and MG noise
$n\sim\mathcal{N}(0,\sum)$ with $\sum=\mathcal{L}^{2}\cdot U\Lambda U^{T}$.
$U$ is a random unitary matrix, $\Lambda$ is a diagonal matrix of three random
values in the range (0,1). The sampling ranges of noisy parameters are
following: 1) AWGN: $\sigma$ in $\left\\{15,25,50\right\\}$; 2) Salt-and-
pepper noise: $\rho$ in $\left\\{5\%,10\%,15\%,25\%\right\\}$; 3) HG noise:
$\alpha$ in $\left\\{20:10:40\right\\}$ , $\sigma$ in
$\left\\{10,20\right\\}$; 4) MG noise: $\mathcal{L}=75$.

JPEG compression. When storing images, the quality would be compressed. We add
the JPEG Compression with the quality factor $\mathcal{C}$ on the degraded
images. The $\mathcal{C}$ is sampled from $\left\\{10:10:40\right\\}$. 0 means
the image is losslessly compressed.

###  4.3 Experimental settings

Our model is trained on the CASIA-WebFace dataset. We crop the training image
with size 112x96 to feed the network. The model is trained in 20 epochs with
SGD optimizer. The batchsize is set as 256. The learning rate is initialized
as 1e-2 and decayed by factor 0.1 after 10, 15, 18 epoch. We add the
degradation model on test images for evaluation and conduct a separate
experiment to analyze the influence of independent noise.

###  4.4 Experiments on the images under single noise

Our main objective of this work is to enhance the inherent noise robustness of
CNNs without significant performance loss. Therefore, Our method focuses on
applying median representation to the CNN. We set the Sphere20 as baseline
[Liu et al.(2017)Liu, Wen, Yu, Li, Raj, and Song] and compare the performance
of different MeDiConv layers. The MeDiNet is built on the Sphere20. The first
to fifth layers of Sphere20 are replaced with the MeDiConv (For example, the
MeDi3 denotes the first three layers are MeDiConv). The training stage only
used clean images. The noisy images were used in the test stage. Various types
of noise are utilized to evaluate our training model on the LFW and YTF
datasets (See Figure 3 and Figure 4). We separately analyzed the robustness of
MeDiNet to each noise type. _The detailed information of results can be
observed in the supplemental material_.

_Evaluation on blurring_. In practical applications, the blurred images are
generated from defocus and pre-processing stage. We evaluate MeDiNet on the
gaussian and motion blur kernel. The blurring leads to the degradation of
edges and structures in the image. The results illustrate that our method is
robust to image blurring. However, the performance drops slightly with the
number of MeDiConv increasing to five layers.

_Evaluation on AWGN and Salt-and-Pepper noise_. These noises are generated
from remote sensing and can be regarded as outliers in the images. The
standard CNNs are sensitive to noise. Noise pixels can significantly reduce
the accuracy of face recognition, especially salt-and-pepper noise. When
setting the proportion of salt-and-pepper noise to 20%, the accuracy of model
drops by more than 30%. MeDiNet can effectively remove the outliers in the
feature maps by median representation. The test results in Figure 3 and Figure
4 show that our method can enhance the robustness to a broad range of noises.

_Evaluation on complex noise_. However, CNN usually generates poorly due to
complex noise even if it can adapt to AWGN and salt-pepper noise [Plotz and
Roth(2017)]. Thus, we also present the results on complex noise models: HG and
MG noise (See Figure 3 and Figure 4). We can observe that the improvement is
still significant, which can prove the robustness of proposed MeDiNet.

_Evaluation on Scaling and Compression_. Standard CNNs achieve slight accuracy
loss under compression. In the procedure of JPEG compression, part of high-
frequency information is removed, which has a negative impact on the ability
of CNN to extract features. For scaling, results demonstrate that the accuracy
drops significantly when enlarging scaling factor $\mathcal{S}$ because of
sampling and interpolation. The detailed information of images might be
removed during the scaling process. The median representation can effectively
enhance the expression ability of features to a certain extent.

Figure 3: The evaluation on the LFW dataset under single noise. Figure 4: The
evaluation on the YTF dataset under single noise.

Through the analysis of each experimental result under the single noise, we
can observe that the MeDiNet inherits the characteristics of MRELBP, and
demonstrates striking robustness to various image noises, including blurring,
AWGN, Salt-and-Pepper, HG, MG, Compression and Scaling. In the Sections 4.5,
the combination of multiple noises is adopted to test our method, which
approximates noise interference in the real world.

Figure 5: The visualization of feature map. The first row are outputs of the
first to fifth layer in the baseline; the second row are outputs of the last
MeDiConv layer in the MeDi1-5 (e.g., the second figure denote the feature map
of last MeDiConv layer in MeDi2).

We visualized feature maps of middle layers in the model to demonstrate the
ability of MeDiConv to suppress noise (See Figure 5). The input image was
added with the AWGN with $\sigma=50$. We can observe that the output by
MeDiConv is smooth and contains fewer noisy pixels. The structure and edges of
face are identifiable, leading to improve the accuracy of face recognition.

###  4.5 Experiments on the degraded images

In this section, the degradation model introduced in Sections 4.2 is adopted
to approximate the real-world noise. We utilize two blurring methods and four
types of noise to combine. The parameters of the degradation model are
randomly generated within their respective parameter ranges in Sections 4.2
(_The value of parameters can be observed in the supplemental material_). In
this section, we present the evaluation of four combining noises:

$\bullet$ Gaussian$\\_$blur+AWGN+Scaling+Compression (GB+AWGN);

$\bullet$ Gaussian$\\_$blur+Salt-and-Pepper+Scaling+Compression (GB+SP);

$\bullet$ Gaussian$\\_$blur+HG+Scaling+Compression (GB+HG);

$\bullet$ Gaussian$\\_$blur+MG+Scaling+Compression (GB+MG).

The _clean_ denotes no noise in the images. Under real-world noisy conditions,
the face recognition accuracy drops more significantly than it under single
noise. Tab.1 and Tab.2 show the evaluation of four datasets under real-world
noisy conditions. _The results of other combinations can be observed in the
supplemental material._

Table 1: The evaluation on the LFW and YTF dataset

| Clean | GB+AWGN | GB+SP | GB+HG | GB+MG
---|---|---|---|---|---
| LFW | YTF | LFW | YTF | LFW | YTF | LFW | YTF | LFW | YTF
Sphere20 | 0.9922 | 0.9330 | 0.7682 | 0.7636 | 0.6762 | 0.6618 | 0.7482 | 0.7440 | 0.6573 | 0.6628
MeDi1 | 0.9908 | 0.9300 | 0.7938 | 0.7946 | 0.7407 | 0.7472 | 0.7790 | 0.7500 | 0.6723 | 0.6790
MeDi2 | 0.9910 | 0.9322 | 0.8390 | 0.8256 | 0.7797 | 0.7848 | 0.8192 | 0.7984 | 0.6755 | 0.6754
MeDi3 | 0.9898 | 0.9328 | 0.8402 | 0.8254 | 0.7908 | 0.7888 | 0.8208 | 0.8058 | 0.7265 | 0.7178
MeDi4 | 0.9882 | 0.9350 | 0.9083 | 0.8864 | 0.8820 | 0.8782 | 0.8942 | 0.8708 | 0.7157 | 0.7654
MeDi5 | 0.9892 | 0.9334 | 0.8943 | 0.8812 | 0.8735 | 0.8648 | 0.8848 | 0.8590 | 0.7222 | 0.7452

  * *

The MeDi1 can be regarded as the image preprocessed by median filtering.

Compared with the single noise in Section 4.4, the degradation model further
increases the complexity of noise. The results in Tab. 1 and Tab. 2
demonstrate that MeDiNet can enhance the performance under real-world noisy
condition. We can observe improvements of GB+AWGN, GB+SP, and GB+HG are
significant. The performance under the GB+MG is still limited.

In order to fully verify the effectiveness of our method, we also report the
results of CA-LFW and CP-LFW dataset (See Tab. 2). These datasets are
collected for specific challenges (pose and age difference). We can observe
that the improvement on the CP-LFW dataset is limited. This dataset selects
face pairs with pose differences to add pose variation to intra-class variance
and contains less edge information than LFW. When using the MeDiConv, the
degradation of edge leads to a limited increase on the CP-LFW (e.g., the
accuracy of MeDi5 is only 3.32% higher than baseline under GB+MG condition).

Table 2: The evaluation on the CA-LFW and CP-LFW dataset

| Clean | GB+AWGN | GB+SP | GB+HG | GB+MG
---|---|---|---|---|---
| CP | CA | CP | CA | CP | CA | CP | CA | CP | CA
Sphere20 | 0.7618 | 0.9078 | 0.5697 | 0.6957 | 0.5238 | 0.5545 | 0.5502 | 0.6030 | 0.5130 | 0.5403
MeDi1 | 0.7607 | 0.9020 | 0.5868 | 0.7288 | 0.5525 | 0.6500 | 0.5500 | 0.6405 | 0.5133 | 0.5337
MeDi2 | 0.7638 | 0.9032 | 0.5927 | 0.7647 | 0.5677 | 0.6903 | 0.5675 | 0.6835 | 0.5315 | 0.5498
MeDi3 | 0.7620 | 0.9017 | 0.6028 | 0.7632 | 0.5660 | 0.6940 | 0.5707 | 0.6912 | 0.5405 | 0.5577
MeDi4 | 0.7407 | 0.8938 | 0.6485 | 0.8097 | 0.6210 | 0.7560 | 0.6097 | 0.7558 | 0.5372 | 0.6322
MeDi5 | 0.7553 | 0.8927 | 0.6438 | 0.8082 | 0.6132 | 0.7628 | 0.6028 | 0.7582 | 0.5462 | 0.6327

The evaluation of four datasets demonstrates that proposed MeDiNet enhances
the performance of face recognition under the real-world noisy conditions. The
accuracy is improved when the number of MeDiConv layers increases. The median
representation causes a probability of missing edge information. Thus, the
performance slightly drops compared with baseline when testing on clean
images. When adding multiple combinations of noises, the face recognition
accuracy increases significantly on four datasets compared with baseline due
to the noise-robustness of MeDiNet. However, the improvement under complex
noise (GB+HG and GB+MG) is lower than other combinations. The ability of
MeDiNet to deal with complex noises needs to be strengthened. Solving the
problem of edge caused by median representation is also the focus of future
work.

In order to further analyze the characteristics of MeDiConv, we report the
time consumption of the model and transfer it to the ResNet architecture for
evaluation.

Time Consumption In the experiment, we observed median operation is hard to
speed up on GPU. When the number of MeDiConv layers increases, the training
time consumption significantly increases (See in Fig. 6). It would be more
efficient to implement from scratch at C/Cuda level. For training convergence,
the difference between the loss curve and training accuracy of MeDi1-3 and
baseline is limited, illustrating that training MeDiNet is stable.

Figure 6: The first graph shows the training time of each model (20 epochs);
The second graph shows the inference time for per image; The third and fourth
graphs show the training loss and accuracy.

Transfer Ability For verifying that MeDiConv is a general way to handle noise,
we choose ResNet18 as the backbone. The result is shown in Tab 3. We fixed the
first convolution in ResNet as MeDiConv. The MeDiRes1 denotes that we replace
the convolutional layers in the first block with MeDiConv. The result showed
our methods still can suppress the negative impact of noise, which can prove
that MeDiConv is a general way to handle noise that can be transferred to
other architectures.

Table 3: The ResNet architecture evaluated on the LFW

| Clean | GB+AWGN | GB+SP | GB+HG | GB+MG
---|---|---|---|---|---
ResNet18 | 0.9888 | 0.8065 | 0.7000 | 0.7943 | 0.6665
MeDiRes1 | 0.9730 | 0.8348 | 0.7847 | 0.8205 | 0.6688
MeDiRes2 | 0.9810 | 0.8580 | 0.8110 | 0.8378 | 0.6950

##  5 Conclusion

We proposed a robust architecture MeDiNet, to enhance the performance of face
recognition under noisy conditions. Standard CNNs are sensitive to the noise
in the real world, especially in low-quality security cameras, low-light
conditions, etc., which leads to a significant decrease in the accuracy of
face recognition. Inspired by MRELBP descriptor, MeDiNet adopted median pixel
difference convolutions, which can effectively remove noisy pixels in the
feature map. Extensive experimental results showed the MeDiNet demonstrates
striking robustness to various image noises.

Compared with standard CNNs, our method has attractive properties on noisy
images. However, there are several challenges that need to be addressed: 1)
The trade-off exists between accuracy and noisy robustness. The median
representation can affect feature extraction (e.g., edge detection) in the
deep layers; 2) The overlap region between neighbor MeDiConv kernel might
result in the same median value, which can affect the accuracy of face
recognition; 3) The accuracy still decreases significantly under the complex
noise.

## Acknowledgement

This work was partially supported by the Academy of Finland under grant 331883
and the National Natural Science Foundation of China under Grant 61872379. The
authors also wish to acknowledge CSC IT Center for Science, Finland, for
computational resources.

## References

  * [Abdelhamed et al.(2018)Abdelhamed, Lin, and Brown] A. Abdelhamed, S. Lin, and M.S. Brown.  A high-quality denoising dataset for smartphone cameras.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1692–1700, 2018.
  * [Anwar and Barnes(2019)] S. Anwar and N. Barnes.  Real image denoising with feature attention.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 3155–3164, 2019.
  * [Ataman et al.(1981)Ataman, Aatre, and Wong] E. Ataman, V. Aatre, and K. Wong.  Some statistical properties of median filters.  _IEEE Transactions on Acoustics, Speech, and Signal Processing_ , 29(5):1073–1075, 1981.
  * [Cao et al.(2018)Cao, Rong, Li, Tang, and Loy] K. Cao, Y. Rong, C. Li, X. Tang, and C.C. Loy.  Pose-robust face recognition via deep residual equivariant mapping.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5187–5196, 2018.
  * [Chen et al.(2019)Chen, Li, Deng, Li, and Yu] J. Chen, K. Li, Q. Deng, K. Li, and P.S. Yu.  Distributed deep learning model for intelligent video surveillance systems with edge computing.  _IEEE Transactions on Industrial Informatics_ , pages 1–1, 2019.
  * [Chrysos and Zafeiriou(2017)] G.G. Chrysos and S. Zafeiriou.  Deep face deblurring.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 69–78, 2017.
  * [Ding et al.(2017)Ding, Cheng, Cheng, and Li] Y. Ding, Y. Cheng, X. Cheng, and B. Li.  Noise-resistant network: a deep-learning method for face recognition under noise.  _EURASIP Journal on Image and Video Processing_ , 2017:43, 2017.
  * [Guo and Zhang(2019)] G. Guo and N. Zhang.  A survey on deep learning based face recognition.  _Computer Vision and Image Understanding_ , 2019.
  * [Han et al.(2018)Han, Yao, Yu, Niu, Xu, Hu, Tsang, and Sugiyama] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I.W. Tsang, and M. Sugiyama.  Co-teaching: robust training of deep neural networks with extremely noisy labels.  In _Proceedings of the 32th International Conference on Neural Information Processing Systems_ , pages 8536–8546, 2018.
  * [Hu et al.(2019)Hu, Huang, Zhang, and Li] W. Hu, Y. Huang, F. Zhang, and R. Li.  Noise-tolerant paradigm for training face recognition cnns.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11887–11896, 2019.
  * [Huang et al.(2011)Huang, Shan, Ardabilian, Wang, and Chen] D. Huang, C. Shan, M. Ardabilian, Y. Wang, and L. Chen.  Local binary patterns and its application to facial image analysis: a survey.  _IEEE Transactions on Systems, Man, and Cybernetics_ , 41(6):765–781, 2011.
  * [Huang et al.(2007)Huang, Ramesh, Berg, and E.] G.B. Huang, M. Ramesh, T. Berg, and Learned-Miller E.  Labeled faces in the wild: A database for studying face recognition in unconstrained environments.  Technical Report 07-49, University of Massachusetts, 2007.
  * [Juefei-Xu et al.(2017)Juefei-Xu, Boddeti, and Savvides] F. Juefei-Xu, V. N. Boddeti, and M. Savvides.  Local binary convolutional neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 19–28, 2017.
  * [Kamgar-Parsi and Lawson(2011)] B. Kamgar-Parsi and W. Lawson.  Toward development of a face recognition system for watchlist surveillance.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 33(10):1925–1937, 2011.
  * [Kumawat et al.(2019)Kumawat, Verma, and Raman] S. Kumawat, M. Verma, and S. Raman.  Lbvcnn: Local binary volume convolutional neural network for facial expression recognition from image sequences.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 0–0, 2019.
  * [Lahasan et al.(2019)Lahasan, Lutfi, and San-Segundo] B. Lahasan, S.L. Lutfi, and R. San-Segundo.  A survey on techniques to handle face recognition challenges: occlusion, single sample per subject and expression.  _Artificial Intelligence Review_ , 52:949–979, 2019.
  * [Li et al.(2020)Li, Jin, Kubota, and Ju] J. Li, K. Jin, N. Kubota, and Z. Ju.  Attention mechanism-based cnn for facial expression recognition.  _Neurocomputing_ , 411:340–350, 2020.
  * [Li et al.(2018a)Li, Liu, Ye, Zuo, Lin, and Yang] X. Li, M. Liu, Y. Ye, W. Zuo, L. Lin, and R. Yang.  Learning warped guidance for blind face restoration.  In _Proceedings of the European Conference on Computer Vision_ , pages 272–289, 2018a.
  * [Li et al.(2018b)Li, Song, Wu, He, and Tan] Y. Li, L. Song, X. Wu, R. He, and T. Tan.  Anti-makeup: Learning a bi-level adversarial network for makeup-invariant face verification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018b.
  * [Lin et al.(2019)Lin, Cheng, Shen, and Pantic] Y. Lin, S. Cheng, J. Shen, and M. Pantic.  Mobiface: A novel dataset for mobile face tracking in the wild.  In _Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019)_, pages 1–8, 2019.
  * [Liu et al.(2016)Liu, Lao, Fieguth, Guo, Wang, and Pietikäinen] L. Liu, S. Lao, P.W. Fieguth, Y. Guo, X. Wang, and M. Pietikäinen.  Median robust extended local binary pattern for texture classification.  _IEEE Transactions on Image Processing_ , 25(3):1368–1381, 2016.
  * [Liu et al.(2018)Liu, Chen, Fieguth, Zhao, Chellappa, and Pietikäinen] L. Liu, J. Chen, P. Fieguth, G. Zhao, R. Chellappa, and M. Pietikäinen.  From bow to cnn: Two decades of texture representation for texture classification.  _International Journal of Computer Vision_ , 127:74–109, 2018.
  * [Liu et al.(2020)Liu, Ouyang, Wang, Fieguth, Chen, Liu, and Pietikäinen] L. Liu, W. Ouyang, X. Wang, P. Fieguth, J. Chen, X. Liu, and M. Pietikäinen.  Deep learning for generic object detection: A survey.  _International Journal of Computer Vision_ , 128:261–318, 2020.
  * [Liu et al.(2017)Liu, Wen, Yu, Li, Raj, and Song] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song.  Sphereface: Deep hypersphere embedding for face recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 212–220, 2017.
  * [Ojala et al.(2002)Ojala, Pietikäinen, and Maenpaa] T. Ojala, M. Pietikäinen, and T. Maenpaa.  Multiresolution gray-scale and rotation invariant texture classification with local binary patterns.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 24(7):971–987, 2002.
  * [Plotz and Roth(2017)] T. Plotz and S. Roth.  Benchmarking denoising algorithms with real photographs.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1586–1595, 2017.
  * [Schroff et al.(2015)Schroff, Kalenichenko, and Philbin] F. Schroff, D. Kalenichenko, and J. Philbin.  Facenet: A unified embedding for face recognition and clustering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 815–823, 2015.
  * [Su et al.(2019)Su, Pietikäinen, and Liu] Z. Su, M. Pietikäinen, and L. Liu.  Bird: Learning binary and illumination robust descriptor for face recognition.  In _Proceedings of the British Machine Vision Conference (BMVC)_ , page 102, 2019.
  * [Su et al.(2021)Su, Liu, Yu, Hu, Liao, Tian, Pietikäinen, and Liu] Z. Su, W. Liu, Z. Yu, D. Hu, Q. Liao, Q. Tian, M. Pietikäinen, and L. Liu.  Pixel difference networks for efficient edge detection.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , pages 5117–5127, 2021.
  * [Sun et al.(2014a)Sun, Chen, Wang, and Tang] Y. Sun, Y. Chen, X. Wang, and X. Tang.  Deep learning face representation by joint identification-verification.  In _Proceedings of the 27th International Conference on Neural Information Processing Systems_ , pages 1988–1996, 2014a.
  * [Sun et al.(2014b)Sun, Wang, and Tang] Y. Sun, X. Wang, and X. Tang.  Deep learning face representation from predicting 10,000 classes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1891–1898, 2014b.
  * [Taigman et al.(2014)Taigman, Yang, Ranzato, and Wolf] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf.  Deepface: Closing the gap to human-level performance in face verification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1701–1708, 2014.
  * [Veit et al.(2017)Veit, Alldrin, Chechik, Krasin, Gupta, and Belongie] A. Veit, N. Alldrin, G. Chechik, I. Krasin, A. Gupta, and S.J. Belongie.  Learning from noisy large-scale datasets with minimal supervision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6575–6583, 2017.
  * [Vezzani et al.(2013)Vezzani, Baltieri, and Cucchiara] R. Vezzani, D. Baltieri, and R. Cucchiara.  People reidentification in surveillance and forensics: A survey.  _ACM Computing Surveys_ , 46(2):1–37, 2013.
  * [Vipparthi et al.(2015)Vipparthi, Murala, Nagar, Li, and Gonde] S.K. Vipparthi, S. Murala, S.K. Nagar, K. Li, and A.B. Gonde.  Local gabor maximum edge position octal patterns for image retrieval.  _Neurocomputing_ , 167(1):336–345, 2015.
  * [Wang et al.(2018)Wang, Chen, Li, Huang, Chen, C., and Loy] F. Wang, L. Chen, C. Li, S. Huang, Y. Chen, Qian C., and C.C. Loy.  The devil of face recognition is in the noise.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 765–780, 2018.
  * [Wen et al.(2016)Wen, Zhang, Li, and Qiao] Y. Wen, K. Zhang, Z. Li, and Y. Qiao.  A discriminative feature learning approach for deep face recognition.  In _Proceedings of the European Conference on Computer Vision_ , pages 499–515, 2016.
  * [Wolf et al.(2011)Wolf, Hassner, and Maoz] L. Wolf, T. Hassner, and I. Maoz.  Face recognition in unconstrained videos with matched background similarity.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 529–534, 2011.
  * [Wu et al.(2020)Wu, Liu, Cao, Ren, and Zuo] X. Wu, M. Liu, Y. Cao, D. Ren, and W. Zuo.  Unpaired learning of deep image denoising.  In _Proceedings of the European Conference on Computer Vision_ , pages 352–368, 2020.
  * [Yi et al.(2014)Yi, Lei, and Li] D. Yi, Z. Lei, and S.Z. Li.  Learning face representation from scratch.  _arXiv preprint arXiv:1411.7923_ , 2014.
  * [Yu and Porikli(2017)] X. Yu and F. Porikli.  Face hallucination with tiny unaligned images by transformative discriminative neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 4327–4333, 2017.
  * [Yu et al.(2020)Yu, Zhao, Wang, Qin, Su, Li, Zhou, and Zhao] Z. Yu, C. Zhao, Z. Wang, Y. Qin, Z. Su, X. Li, F. Zhou, and G. Zhao.  Searching central difference convolutional networks for face anti-spoofing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5295–5305, 2020.
  * [Zhang et al.(2018)Zhang, Zhuo, and Zhang] K. Zhang, W. Zhuo, and L. Zhang.  Ffdnet: Toward a fast and flexible solution for cnn-based image denoising.  _IEEE Transactions on Image Processing_ , 27(9):4608–4622, 2018.
  * [Zhang et al.(2020)Zhang, Deng, Wang, Hu, Li, Zhao, and Wen] Y. Zhang, W. Deng, M. Wang, J. Hu, X. Li, D. Zhao, and D. Wen.  Global-local gcn: Large-scale label noise cleansing for face recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7731–7740, 2020.
  * [Zheng and Deng(2018)] T. Zheng and W. Deng.  Cross-pose lfw: A database for studying cross-pose face recognition in unconstrained environments.  Technical Report 18-01, Beijing University of Posts and Telecommunications, February 2018.
  * [Zheng et al.(2017)Zheng, Deng, and Hu] T. Zheng, W. Deng, and J. Hu.  Cross-age lfw: A database for studying cross-age face recognition in unconstrained environments.  _arXiv preprint arXiv:1708.08197_ , 2017.