Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

De-Factify 2: 2nd Workshop on Multimodal Fact Checking and Hate Speech
Detection, co-located with AAAI 2023

[email =<EMAIL_ADDRESS>
[email =<EMAIL_ADDRESS>
[email =<EMAIL_ADDRESS>
11footnotetext: Equal contribution.22footnotetext: Work does not relate to
position at Amazon.

# Overview of Memotion 3: Sentiment and Emotion Analysis of Codemixed Hinglish
Memes

Shreyash Mishra*  IIIT Sri City, India  S Suryavardan  Megha Chakraborty
University of South Carolina, USA  Parth Patwa  UCLA, USA  Anku Rani  Aman
Chadha  Stanford University, USA  Amazon AI, USA  Aishwarya Reganti  CMU, USA
Amitava Das  Amit Sheth  Manoj Chinnakotla  Microsoft, USA  Asif Ekbal  IIT
Patna, India  Srijan Kumar  Georgia Tech, USA

(2023)

###### Abstract

Analyzing memes on the internet has emerged as a crucial endeavor due to the
impact this multi-modal form of content wields in shaping online discourse.
Memes have become a powerful tool for expressing emotions and sentiments,
possibly even spreading hate and misinformation, through humor and sarcasm. In
this paper, we present the overview of the Memotion 3 shared task, as part of
the DeFactify 2 workshop at AAAI-23. The task released an annotated dataset of
Hindi-English code-mixed memes based on their Sentiment (Task A), Emotion
(Task B), and Emotion intensity (Task C). Each of these is defined as an
individual task and the participants are ranked separately for each task. Over
50 teams registered for the shared task and 5 made final submissions to the
test set of the Memotion 3 dataset. CLIP, BERT modifications, ViT etc. were
the most popular models among the participants along with approaches such as
Student-Teacher model, Fusion, and Ensembling. The best final F1 score for
Task A is 34.41, Task B is 79.77 and Task C is 59.82.

###### keywords:

Memes, codemixed, multimodal, Hindi-English

##  1 Introduction

The term _meme_ is derived from the Ancient Greek word _mimema_ , meaning
_imitated thing_ , which comes from the verb _mimeisthai_ , meaning _to
mimic_. The term was coined by Richard Dawkins, a British evolutionary
biologist, in his book _The Selfish Gene_ [1]. Dawkins used the concept of a
meme to explain the spread of cultural phenomena and ideas, drawing parallels
between memes and genes. Dawkins provided examples of memes such as melodies,
catchphrases, fashion, and arch-building technology in his book.
Interestingly, the word _meme_ is a self-describing term, also known as
autological, meaning that it is a meme itself.

The reach of online content is vast and has the capacity to reach a massive
viewership. Memes propagate through multiple mediums including social media,
messaging, apps, and emails. An article on memes quotes "When you plant a
fertile meme in my mind, you literally, parasitize my brain" [2]. This quote
indicates the impact of memes on a viewer’s mind. It spreads to others who
find it motivating, offensive, amusing, or relatable in some ways [3]. Memes
are used to convey an opinion. When a person comes across a meme that they
find relatable, they often share it with others who they think would find
relatable. Different people interpret memes differently which means what can
be offensive for one person might not be for other. This difference in
interpretation is a result of cultural, gender, and demographic differences.

The memes can also be used to spread offensive and hateful content [4].
Identifying and halting the dissemination of hateful memes is an arduous
undertaking for both human beings and AI models, as it requires comprehending
the intricate subtleties and socio-political contexts that form their
interpretation. The deficiencies of current hate speech moderation techniques
highlight the pressing need to enhance the effectiveness of automated hate
speech detection. Given their subtlety and multi-modal nature, memes pose a
more challenging issue to address compared to only text.

In this paper, we present the findings of the Memotion 3 shared task, where
participants were provided with an annotated dataset of 10k memes [5], and
were tasked with detecting the sentiment, emotion and emotion intensity of the
meme. Unlike the previous iteration of memotion [6, 7] which provided English
memes, current iteration studies codemixed Hindi-English (Hinglish) memes.

The paper is organized as follows: we describe the related work and the task
in section 2 and 3 respectively; Section 4 contains the details of the dataset
we collected for Memotion analysis: Memotion 3.0; followed by a brief
description of baseline models and their results in 5. We conclude and mention
some possible future works in section 6.

##  2 Related Work

Sentiment and emotion analysis: There has been significant research on
sentiment analysis for text over many years [8]. Work on sentiment analysis
using ML methods like SVM, logistic regression, random forest, XGBoost,
k-nearest neighbor has been done in [9, 10, 11]. Works which use DL methods
include [12, 13, 14]. For detailed surveys on sentiment analysis in social
media, please refer to [15, 16].

The HaHa shared task provides a dataset for humor detection on social media
[17]. Öhman et al. [18] release an English dataset to detect eight emotions
like joy, sadness, disgust etc. A comprehensive survey of textual emotion
detection is provided in [19]

Hatespeech detection: It is important to detect hatespeech to keep social
media safe for everyone including minorities. Towards this goal, researchers
have curated and released annotated datasets [20]. The offenseval shared task
[21] at SemEval 2019 releases an annotated dataset of 14 English tweets to
detect the type and target of offensive language. The HatEval shared task
releases English and Spanish datasets to detect hate towards women and
immigrants. Patwa et al. [22] conduct a shared task on Hindi hostile tweets
detection. The TRAC workshop series [23, 24, 25] conducts multiple shared
tasks to detect aggression and misogyny in English, Hindi, and Bengali
datasets.

Methods to detect hatespeech in text include CNNs and RNNS [26, 27, 28, 29],
Bert-like models [30, 31, 32], incorporating linguistic characteristics [33]
etc.

Codemixed Language Processing : Codemixed language processing is a challenging
task because the informal mixing of 2 or more languages and the proliferation
of unique number of ways to write the same word [34, 35]. The Sentimix task
[36] at semeval 2020 focused on sentiment analysis of Hinglish and Spanish-
English tweets. [37] organized a shared task on detecting offense in 3
codemixed dravidian languages - Tamil [38], Malayalam[39] and Kannada[40].
Methods explored to tackle codemixing include graph convolutional netowrks
[41], BERT based models [42, 43], modifying loss function [44, 45] or
positional embeddings [46] to incorporate codemixing etc.

Multimodal analysis : Although most of the existing research focuses on
unimodal (text) analysis, the use of multi-modal content likes memes and
videos is fast increasing. Multimodal datatsets having text and image, or
videos are useful for tasks like image captioning, hatespeech detection,
emotion analysis in videos, sentiment analysis [47, 48, 49, 50] etc among
other tasks. The Hateful Memes Challenge dataset [51] and the multioff dataset
[4] address hatespeech and offense detection in memes. However, they are
binary classification task and the memes are in English whereas memotion 3 has
multi-class and multi-label tasks on code-mixed data. There have been very few
works on code-mixed meme analysis. [52] conduct a shared task to detect
trolling in Tamil codemixed memes whereas [53] release a dataset to detect
hate in codemixed Bengali memes. Popular methods for multimodal learning
include image-text joint embeddings [54, 55, 56] and transformer based models
[57, 58].

Previous iteration of Memotion : Memotion 1 [6] and Memotion 2[7] shared task
released datasets of 10k memes each [6, 59]. These datasets were annotated on
the same tasks as Memotion 3. However, both these datasets only focus on
English memes, where as in memotion 3, we focus on Hinglish codemixed memes.
Methods like ensembling [60, 61, 62] and bert-like models [63, 64, 65] were
common across memotion 1 and Memotion 2.

##  3 Task Details

###  3.1 Tasks

Memotion 3 is the 3rd iteration of the Memotion task. The challenge consists
of three sub-tasks:

  1. 1.

Task A - Sentiment analysis of memes: Given a meme, the system is supposed to
classify the meme’s sentiment as positive, negative or neutral.

  2. 2.

Task B - Overall emotion analysis of memes: This task’s goal is to pinpoint
certain emotions connected to a given meme. Whether a meme is humorous,
sarcastic, offensive, or motivating should be indicated by the system. There
are multiple categories in which a meme can fit.

  3. 3.

Task C - Classifying the intensity of meme emotions: The task is to determine
the degree to which a particular emotion is being expressed. The ranking of
these emotions is as follows:

    1. (a)

Humour: Not funny, funny, very funny and hilarious

    2. (b)

Sarcasm: Not Sarcastic, little sarcastic, very sarcastic and extremely
sarcastic

    3. (c)

Offensive: Not offensive, slightly offensive, very offensive and hateful
offensive

    4. (d)

Motivation: Not motivational, motivational

###  3.2 Dataset

The tasks were conducted on the Memotion 3 dataset [5]. It consists of Hindi-
English codemixed memes, which were collected from selenium based web crawler
and they were gathered from various public platforms like Reddit, Google
Images, etc and annotated manually. The dataset consists of 10,000 meme images
divided into a train-val-test split of 7000-1500-1500. Each meme is annotated
for its sentiment, emotion and intensity of emotion. Images also have their
corresponding OCR text extracted with the help of Google Vision APIs and their
respective URLs. For more details of the dataset, please refer to [5].

###  3.3 Evaluation

As mentioned previously, there are three tasks. Scoring is done for each task
separately, and separate leaderboards are generated. For each task, we use
weighted average F1 score to measure the performance of a model. The
participants had access to only train and validation set. They were asked to
submit a maximum of 3 submissions on the test set for each task, the best of
which was selected as part of the leaderboard.

###  3.4 Baselines

For multi-modal data, it is crucial to take into account both the visual and
textual properties, particularly in the case of memes where the context can
only be recorded using a mix of both elements. For textual features, we employ
a multilingual form of BERT, namely Hinglish-BERT (BERT-base-multilingual-
cased) [66], which is tuned on Hinglish data. The trained Vision Transformer
(ViT) model provides the visual properties [67]. The Hinglish-BERT embedding
is concatenated with the pooled output from the ViT model. After passing
through an MLP, the combined features are then categorised in a final
classification layer. For more details about the baseline, please refer [5].

##  4 Participating systems

There were 47 team registrations for the task in the Memotion 3.0 task page,
of which 5 teams made submissions for the final test set of the dataset. The
results for all three tasks are given in the following section and an overview
of the 4 teams that presented their description papers are provided below.

wentaorub [68] use CLIP [69] for individual text and image embeddings, before
concatenating them and passing them through multi-headed attention layers for
classification. They also use the OSCAR [70] model in this approach and an
ensemble of their models are used for the final submission. Datasets such as
Facebook Hateful memes [51], MMHS150k [49] etc. are used for pre-training.
This architecture helped them achieve the best performance in Task B and C.

NYCU_TWO [71] propose a two model pipelines, namely Coopoerative Teaching
Model (CTM) for task A and Cascaded Emotion Classifier (CEC) for task B and C.
A fusion of multi-modal embeddings from pre-trained Swin-Transformer [72] and
CLIP are passed to the CTM and CEC pipelines. CEC helps leverage task C
predictions for task B by jointly training the model. This team attained the
best results in 3 out of the 4 labels in Task B.

NUAA-QMUL-AIIT [73] refer to their approach as Squeeze-and-Excitation Fusion
or SEFusion. The textual features from pre-trained RoBERTa [74] and visual
features from CLIP-ViT [75] are fused to obtain multi-modal embeddings. The
fusion is the SEFusion module, which uses a learned activation of the squeezed
features, allowing for weighted fusion of multi-modal embeddings. This
approach led to NUAA-QMUL-AIIT being the 1st ranked team in Task A.

CUFE used a LightGBM [76] classifier for classification on every individual
emotion or label in all Tasks. The inputs to the classifier were pre-trained
Hinglish-DistilBERT for text embeddings and ResNet18 [77] for image
embeddings. Other features, such as occurrence of characters, word count etc.
from text and number of faces in the memes (extracted using Facenet’s [78]
multitask cascaded CNN), were also used. CUFE obtained the highest score in 2
labels in Task B and 1 label in Task C.

##  5  Results

Rank |  Team |  F1 - Scores
---|---|---
1 |  NUAA-QMUL-AIIT |  34.41%
2 |  NYCU_TWO |  34.20%
3 |  CUFE |  33.77%
4 |  CSECU-DSG |  33.34%
5 |  Baseline |  33.28%
6 |  wentaorub |  32.88%
Table 1: Leaderboard of teams on Task A: Sentiment Analysis.

The performance in Task A i.e. sentiment classification is presented in Table
1. Our proposed baseline achieved a score of 33.28%. Out of the five final
submissions, four teams managed to surpass the baseline. The top two teams,
namely, NUAA-QMUL-AIIT [73] and NYCU_TWO [71] improved on the baseline by 1.1%
and 0.9% respectively. The difference in F1 scores for this task is minimal
across all participants.

Rank |  Team | F1 - Scores
---|---|---
|  |  H |  S |  O |  M |  Overall
1 |  wentorub |  88.91 |  86.74 |  48.99 |  94.44 |  79.77
2 |  NYCU_TWO |  89.04 |  86.91 |  43.17 |  94.44 |  78.39
3 |  NUAA-QMUL-AIIT |  87.06 |  77.97 |  50.72 |  94.44 |  77.55
4 |  BASELINE |  84.55 |  74.82 |  48.84 |  90.78 |  74.74
5 |  CUFE |  82.26 |  86.91 |  50.78 |  77.60 |  74.39
6 |  CSECU-DSG |  88.64 |  86.30 |  49.32 |  64.79 |  72.26
Table 2: Leaderboard of teams on Task B: Emotion analysis {H:Humor, S:Sarcasm,
O:Offense, M:Motivation}. All the scores are in percentage. The teams are
ranked by their average F1 scores (overall) across all the four emotions.
Motivation emotion is the easiest to detect while offense is the most
difficult to detect.

Table 2 shows the leaderboard for Task B. Three teams out-performed the
baseline for this task and top performing team wentaorub [68] improves on the
final score by 5.4%. Team CUFE performs lower than the baseline score by 3.4%,
however, they present the highest score on the Offensive sub-task and on the
Sarcasm sub-task, jointly with NYCU_TWO [71]. wentaorub [68], NYCU_TWO, and
NUAA-QMUL-AIIT achieved the same score on Motivation, which is 4% higher than
the baseline. NYCU_TWO also scored the highest in the Humour sub-task,
improving on the baseline by 4.5%. Based on these F1 scores, we can deduce
that the Motivation class is the easiest to detect, similar to Memotion 2.0
[59], despite some teams performing poorly on this class. However, in this
iteration, the Offensive class is the hardest to detect, instead of the
Sarcasm class. No single team performs the best on all the classes.

Rank |  Team | F1 - Scores
---|---|---
|  |  H |  S |  O |  M |  Overall
1 |  wentaorub |  48.37 |  50.65 |  45.82 |  94.43 |  59.82
2 |  NUAA-QMUL-AIIT |  46.34 |  44.29 |  43.17 |  94.43 |  57.06
3 |  CSECU-DSG |  42.54 |  34.41 |  45.68 |  93.26 |  53.97
4 |  NYCU_TWO |  43.28 |  45.80 |  44.67 |  80.57 |  53.58
5 |  CUFE |  41.66 |  51.07 |  42.02 |  77.60 |  53.09
6 |  BASELINE |  43.03 |  32.89 |  42.40 |  90.78 |  52.27
Table 3: Leaderboard of teams on Task C: Emotion intensity detection {H:Humor,
S:Sarcasm, O:Offense, M:Motivation}. All scores are in percentage. The teams
are ranked by average F1 scores (overall) across all the four emotions.
Intensity of sarcasm is by far the most difficult to detect. Motivation has
only two intensities as opposed to four intensity levels for other emotions.

As shown in the leaderboard for Task C in Table 3, all the participating teams
outperform the baseline in this task. The minimum improvement in F1 score is
0.8% by team CUFE and the maximum improvement in the final score is 7.5% by
wentaorub [68]. The submission by wentaorub exceeds all other teams in the
sub-tasks Humour, Offensive and Motivation. NUAA-QMUL-AIIT [73] matched the
highest score of wentaorub in the Motivation category, while CUFE achieved the
highest score in the Sarcasm sub-task. The Motivation class has significantly
higher F1 scores than other classes, due to it having only 2 intensities
whereas other classes have 4 intensities each. The best overall score is only
59.82%, which shows the difficulty of the task.

|  |
---|---|---
|  |
Figure 1: Some samples for Task A from the 121 memes that were predicted incorrectly by all participating teams. Over half of the mis-classified memes are true negatives.  |  |
---|---|---
|  |
Figure 2: 242 memes from the test set were predicted incorrectly by all participants. Most of such memes are code-mixed and related to humor or sarcasm.  |  |
---|---|---
|  |
Figure 3: For Task C, none of the teams made correct predictions on over 400
memes from the test set. A large share of these memes is code-mixed, thus
indicating the added challenge to the task.

A major observation when comparing performances with the previous iteration of
the task [7] is that the performance on Task A is much worse in Memotion 3.
Overall Scores in Tasks B and C have remained about the same compared to
Memotion 2.

For each individual task, we analyze the memes from the test set that all
participants made wrong predictions on. For Task A, there are 121 memes where
all participants mis-classified the label, out of which 66 memes were true
negative sentiment followed closely by true positive memes. For task B, there
are 231 such memes, majority of which belong to "humor" and "sarcasm" class.
Finally, for Task C, 421 memes are mis-classified by all the systems - most
memes mis-classified by all teams are "Very Funny", "Very Sarcastic",
"Slightly Offensive". Some such examples for Task A, B and C are shown in
figures 1, 2, and 3 respectively. Further, we note that most of such difficult
examples have code-mixed text.

As for the overall performances, only two teams - NUAA-QMUL-AIIT and NYCU_TWO
[73, 71] \- perform better than the baseline in all tasks.

##  6 Conclusion and Future Work

In this paper, we summarize the approaches used by the participants for the
Memotion 3 task and analyze the results. Due to the multi-modal nature of the
dataset, all teams use a pre-trained image and text embedding models. However,
each team presents a novel model pipeline. The highest scores achieved in Task
A, Task B and Task C of Memotion 3.0 are 34.41%, 79.77% and 59.82%
respectively, which shows there is significant room for improvement. On
analysis of the results and the mis-classified examples on the test set, we
find that "Sarcasm" and "Humour" are difficult to identify, especially in
code-mixed memes.

While we address Hind-English code-mixed memes in this paper, future work
could include exploring other languages/language pairs. A unified baseline
model to analyze memes in multiple languages could also be an interesting
possibility.

## References

  * Dawkins [1979] R. Dawkins, The selfish gene, Granada Publishing Lim, 1979.
  * Marwick [2013] A. Marwick,  Memes,  Contexts 12 (2013) 12–13.
  * Akhther [2018] N. Akhther,  Internet memes as form of cultural discourse: A rhetorical analysis on facebook,  2018\. doi:10.31234/osf.io/sx6t7.
  * Suryawanshi et al. [2020] S. Suryawanshi, B. R. Chakravarthi, M. Arcan, P. Buitelaar,  Multimodal meme dataset (MultiOFF) for identifying offensive content in image and text,  in: TRAC, 2020.
  * Mishra et al. [2023] S. Mishra, S. Suryavardan, P. Patwa, M. Chakraborty, A. Rani, A. Reganti, A. Chadha, A. Das, A. Sheth, M. Chinnakotla, et al.,  Memotion 3: Dataset on sentiment and emotion analysis of codemixed hindi-english memes,  arXiv preprint arXiv:2303.09892 (2023).
  * Sharma et al. [2020] C. Sharma, D. Bhageria, W. Scott, S. PYKL, A. Das, T. Chakraborty, et al.,  SemEval-2020 task 8: Memotion analysis- the visuo-lingual metaphor!,  in: SemEval, 2020.
  * Patwa et al. [2022] P. Patwa, S. Ramamoorthy, N. Gunti, S. Mishra, S. Suryavardan, A. Reganti, A. Das, T. Chakraborty, A. Sheth, A. Ekbal, et al.,  Findings of memotion 2: Sentiment and emotion analysis of memes,  in: Proceedings of De-Factify: Workshop on Multimodal Fact Checking and Hate Speech Detection, ceur, 2022.
  * Socher et al. [2013] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, C. Potts,  Recursive deep models for semantic compositionality over a sentiment treebank,  in: EMNLP, 2013.
  * Saad [2020] A. I. Saad,  Opinion mining on us airline twitter data using machine learning techniques,  in: 2020 16th international computer engineering conference (ICENCO), IEEE, 2020.
  * Alzyout et al. [2021] M. Alzyout, E. A. Bashabsheh, H. Najadat, A. Alaiad,  Sentiment analysis of arabic tweets about violence against women using machine learning,  in: 12th ICICS, 2021.
  * Prabhakar et al. [2019] E. Prabhakar, M. Santhosh, A. H. Krishnan, T. Kumar, R. Sudhakar,  Sentiment analysis of us airline twitter data using new adaboost approach,  (IJERT) 7 (2019).
  * Kokab et al. [2022] S. T. Kokab, S. Asghar, S. Naz,  Transformer-based deep learning models for the sentiment analysis of social media data,  Array 14 (2022) 100157\.
  * Tesfagergish et al. [2022] S. G. Tesfagergish, J. Kapočiūtė-Dzikienė, R. Damaševičius,  Zero-shot emotion detection for semi-supervised sentiment analysis using sentence transformers and ensemble learning,  Applied Sciences (2022).
  * Tan et al. [2022] K. L. Tan, C. P. Lee, K. M. Lim, K. S. M. Anbananthen,  Sentiment analysis with ensemble hybrid deep learning model,  IEEE Access 10 (2022) 103694–103704.
  * Yue et al. [2019] L. Yue, W. Chen, X. Li, W. Zuo, M. Yin,  A survey of sentiment analysis in social media,  Knowledge and Information Systems 60 (2019).
  * Chakraborty et al. [2020] K. Chakraborty, S. Bhattacharyya, R. Bag,  A survey of sentiment analysis from social media data,  IEEE Transactions on Computational Social Systems 7 (2020) 450–464.
  * Chiruzzo et al. [2019] L. Chiruzzo, S. Castro, M. Etcheverry, D. Garat, J. J. Prada, A. Rosá,  Overview of haha at iberlef 2019: Humor analysis based on human annotation.,  in: IberLEF@ SEPLN, 2019.
  * Öhman et al. [2020] E. Öhman, M. Pàmies, K. Kajava, J. Tiedemann, Xed: A multilingual dataset for sentiment analysis and emotion detection, 2020. arXiv:2011.01612.
  * Acheampong et al. [2020] F. A. Acheampong, C. Wenyu, H. Nunoo-Mensah,  Text-based emotion detection: Advances, challenges, and opportunities,  Engineering Reports (2020).
  * Waseem and Hovy [2016] Z. Waseem, D. Hovy,  Hateful symbols or hateful people? predictive features for hate speech detection on Twitter,  in: NAACL, 2016.
  * Zampieri et al. [2019] M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, et al.,  Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval),  arXiv:1903.08983 (2019).
  * Patwa et al. [2021] P. Patwa, M. Bhardwaj, V. Guptha, G. Kumari, S. Sharma, S. PYKL, A. Das, A. Ekbal, M. S. Akhtar, T. Chakraborty,  Overview of constraint 2021 shared tasks: Detecting english covid-19 fake news and hindi hostile posts,  in: Combating Online Hostile Posts in Regional Languages during Emergency Situation, Springer International Publishing, 2021, pp. 42–53.
  * Kumar et al. [2018] R. Kumar, A. K. Ojha, S. Malmasi, M. Zampieri,  Benchmarking aggression identification in social media,  in: TRAC workshop, 2018.
  * Kumar et al. [2020] R. Kumar, A. K. Ojha, S. Malmasi, M. Zampieri,  Evaluating aggression identification in social media,  in: TRAC workshop, 2020.
  * Kumar et al. [2022] R. Kumar, S. Ratan, S. Singh, E. Nandi, L. N. Devi, et al.,  The ComMA dataset v0.2: Annotating aggression and bias in multilingual social media discourse,  in: LREC, 2022.
  * Gambäck and Sikdar [2017] B. Gambäck, U. K. Sikdar,  Using convolutional neural networks to classify hate-speech,  in: Proceedings of the first workshop on abusive language online, 2017, pp. 85–90.
  * Ribeiro and Silva [2019] A. Ribeiro, N. Silva,  Inf-hateval at semeval-2019 task 5: Convolutional neural networks for hate speech detection against women and immigrants on twitter,  in: SemEval, 2019.
  * Winter and Kern [2019] K. Winter, R. Kern,  Know-center at semeval-2019 task 5: multilingual hate speech detection on twitter using cnns,  in: Semeval, 2019.
  * Patwa et al. [2020] P. Patwa, S. Pykl, A. Das, P. Mukherjee, V. Pulabaigari,  Hater-O-genius aggression classification using capsule networks,  in: Proceedings of the 17th International Conference on Natural Language Processing (ICON), 2020.
  * Mazari et al. [2023] A. C. Mazari, N. Boudoukhani, A. Djeffal,  Bert-based ensemble learning for multi-aspect hate speech detection,  Cluster Computing (2023) 1–15.
  * Samghabadi et al. [2020] N. S. Samghabadi, P. Patwa, S. Pykl, P. Mukherjee, A. Das, T. Solorio,  Aggression and misogyny detection using bert: A multi-task approach,  in: Proceedings of the second workshop on trolling, aggression and cyberbullying, 2020.
  * Risch and Krestel [2020] J. Risch, R. Krestel,  Bagging BERT models for robust aggression identification,  in: Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, 2020.
  * Nagar et al. [2023] S. Nagar, F. A. Barbhuiya, K. Dey,  Towards more robust hate speech detection: using social context and user data,  Social Network Analysis and Mining 13 (2023) 47.
  * Laddha et al. [2020] A. Laddha, M. Hanoosh, D. Mukherjee, P. Patwa, A. Narang,  Understanding chat messages for sticker recommendation in messaging apps,  Proceedings of the AAAI Conference on Artificial Intelligence 34 (2020). doi:10.1609/aaai.v34i08.7019.
  * Laddha et al. [2022] A. Laddha, M. Hanoosh, D. Mukherjee, P. Patwa, A. Narang,  Large scale multilingual sticker recommendation in messaging apps,  AI Magazine 42 (2022). doi:10.1609/aaai.12023.
  * Patwa et al. [2020] P. Patwa, G. Aguilar, S. Kar, S. Pandey, S. PYKL, B. Gambäck, T. Chakraborty, T. Solorio, A. Das,  SemEval-2020 task 9: Overview of sentiment analysis of code-mixed tweets,  in: Proceedings of the Fourteenth Workshop on Semantic Evaluation, 2020.
  * Chakravarthi et al. [2021] B. R. Chakravarthi, et al.,  Findings of the shared task on offensive language identification in Tamil, Malayalam, and Kannada,  in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, 2021.
  * Chakravarthi et al. [2020a] B. R. Chakravarthi, V. Muralidaran, R. Priyadharshini, J. P. McCrae,  Corpus creation for sentiment analysis in code-mixed tamil-english text,  arXiv:2006.00206 (2020a).
  * Chakravarthi et al. [2020b] B. R. Chakravarthi, N. Jose, S. Suryawanshi, E. Sherly, J. P. McCrae,  A sentiment analysis dataset for code-mixed malayalam-english,  arXiv preprint arXiv:2006.00210 (2020b).
  * Hande et al. [2020] A. Hande, R. Priyadharshini, B. R. Chakravarthi,  KanCMD: Kannada CodeMixed dataset for sentiment analysis and offensive language detection,  in: Workshop on Computational Modeling of People’s Opinions, Personality, and Emotion’s in Social Media, 2020\.
  * Dowlagar and Mamidi [2021] S. Dowlagar, R. Mamidi,  Graph convolutional networks with multi-headed attention for code-mixed sentiment analysis,  in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, 2021, pp. 65–72.
  * Risch et al. [2019] J. Risch, A. Stoll, M. Ziegele, R. Krestel,  hpidedis at germeval 2019: Offensive language identification using a german bert model.,  in: KONVENS, 2019.
  * Tula et al. [2021] D. Tula, P. Potluri, S. Ms, S. Doddapaneni, P. Sahu, R. Sukumaran, P. Patwa,  Bitions@DravidianLangTech-EACL2021: Ensemble of multilingual language models with pseudo labeling for offence detection in Dravidian languages,  in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, 2021.
  * Tula et al. [2022] D. Tula, M. S. Shreyas, V. Reddy, P. Sahu, S. Doddapaneni, P. Potluri, R. Sukumaran, P. Patwa,  Offence detection in dravidian languages using code-mixing index-based focal loss,  SN Computer Science 3 (2022). doi:10.1007/s42979-022-01190-1.
  * Ma et al. [2020] Y. Ma, L. Zhao, J. Hao,  XLP at SemEval-2020 task 9: Cross-lingual models with focal loss for sentiment analysis of code-mixing language,  in: Semeval, 2020.
  * Ali et al. [2022] M. Ali, S. T. Kandukuri, S. Manduru, P. Patwa, A. Das,  Pesto: Switching point based dynamic and relative positional encoding for code-mixed languages (student abstract),  AAAI (2022). doi:10.1609/aaai.v36i11.21587.
  * Hu and Flaxman [2018] A. Hu, S. Flaxman,  Multimodal sentiment analysis to explore the structure of emotions,  in: KDD, 2018.
  * Jha et al. [2021] R. Jha, V. Kaki, V. Kolla, S. Bhagat, P. Patwa, A. Das, S. Pal,  Image2tweet: Datasets in Hindi and English for generating tweets from images,  in: Proceedings of the 18th International Conference on Natural Language Processing (ICON), 2021.
  * Gomez et al. [2019] R. Gomez, J. Gibert, L. Gomez, D. Karatzas, Exploring hate speech detection in multimodal publications, 2019\. arXiv:1910.03814.
  * Zadeh et al. [2018] B. Zadeh, et al.,  Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph,  in: ACL, 2018.
  * Kiela et al. [2020] D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, D. Testuggine,  The hateful memes challenge: Detecting hate speech in multimodal memes,  Neurips (2020).
  * Suryawanshi and Chakravarthi [2021] S. Suryawanshi, B. R. Chakravarthi,  Findings of the shared task on troll meme classification in Tamil,  in: Speech and Language Technologies for Dravidian Languages, 2021.
  * Hossain et al. [2022] E. Hossain, O. Sharif, M. M. Hoque,  MUTE: A multimodal dataset for detecting hateful memes,  in: Proceedings of the 2nd AACL Student Research Workshop, 2022.
  * Xie et al. [2021] Z. Xie, L. Liu, Y. Wu, L. Zhong, L. Li,  Learning text-image joint embedding for efficient cross-modal retrieval with deep feature engineering,  ACM Transactions on Information Systems 40 (2021) 1–27. URL: https://doi.org/10.1145%2F3490519. doi:10.1145/3490519.
  * Krishna et al. [2023] V. Krishna, S. Suryavardan, S. Mishra, S. Ramamoorthy, P. Patwa, M. Chakraborty, A. Chadha, A. Das, A. Sheth, Imaginator: Pre-trained image+text joint embeddings using word-level grounding of images, 2023\. arXiv:2305.10438.
  * Gunti et al. [2022] N. Gunti, S. Ramamoorthy, P. Patwa, A. Das,  Memotion analysis through the lens of joint embedding (student abstract),  Proceedings of the AAAI Conference on Artificial Intelligence 36 (2022). doi:10.1609/aaai.v36i11.21616.
  * Lu et al. [2019] J. Lu, D. Batra, D. Parikh, S. Lee, Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks, 2019. arXiv:1908.02265.
  * Li et al. [2019] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, K.-W. Chang, Visualbert: A simple and performant baseline for vision and language, 2019. arXiv:1908.03557.
  * Ramamoorthy et al. [2022] S. Ramamoorthy, N. Gunti, S. Mishra, S. Suryavardan, A. Reganti, P. Patwa, A. Das, T. Chakraborty, A. Sheth, A. Ekbal, et al.,  Memotion 2: Dataset on sentiment and emotion analysis of memes,  in: Proceedings of De-Factify: Workshop on Multimodal Fact Checking and Hate Speech Detection, 2022.
  * Phan et al. [2022] K. N. Phan, G.-S. Lee, H.-J. Yang, S.-H. Kim,  Little flower at memotion 2.0 2022: Ensemble of multi-modal model using attention mechanism in memotion analysis,  in: Proceedings of De-Factify: Workshop on Multimodal Fact Checking and Hate Speech Detection, 2022.
  * Morishita et al. [2020] T. Morishita, G. Morio, S. Horiguchi, H. Ozaki, T. Miyoshi,  Hitachi at SemEval-2020 task 8: Simple but effective modality ensemble for meme emotion recognition,  in: SemEval, 2020.
  * Guo et al. [2020] Y. Guo, J. Huang, Y. Dong, M. Xu,  Guoym at SemEval-2020 task 8: Ensemble-based classification of visuo-lingual metaphor in memes,  in: SemEval, 2020.
  * Vlad et al. [2020] G.-A. Vlad, G.-E. Zaharia, D.-C. Cercel, et al., Upb at semeval-2020 task 8: Joint textual and visual modeling in a multi-task learning architecture for memotion analysis, 2020. arXiv:2009.02779.
  * Nguyen et al. [2022] T. T. Nguyen, N. T. Pham, N. D. Nguyen, et al.,  Hcilab at memotion 2.0 2022: Analysis of sentiment, emotion and intensity of emotion classes from meme images using single and multi modalities,  in: Proceedings of De-Factify: Workshop on Multimodal Fact Checking and Hate Speech Detection, 2022.
  * Lee and Shen [2020] G. G. Lee, M. Shen,  Amazon pars at memotion 2.0 2022: Multi-modal multi-task learning for memotion 2.0 challenge,  Proceedings of De-Factify: Workshop on Multimodal Fact Checking and Hate Speech Detection (2020).
  * Bhange and Kasliwal [2020] M. Bhange, N. Kasliwal,  Hinglishnlp: Fine-tuned language models for hinglish sentiment detection,  arXiv preprint arXiv:2008.09820 (2020).
  * Dosovitskiy et al. [2020] A. Dosovitskiy, L. Beyer, A. Kolesnikov, et al.,  An image is worth 16x16 words: Transformers for image recognition at scale,  arXiv:2010.11929 (2020).
  * Yu and Kolossa [2023] W. Yu, D. Kolossa,  wentaorub at Memotion 3: Ensemble learning for multi-modal meme classification,  in: Proceedings of De-Factify 2: Workshop on Multimodal Fact Checking and Hate Speech Detection, CEUR, 2023.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, et al.,  Learning transferable visual models from natural language supervision,  in: ICML, 2021.
  * Li et al. [2020] X. Li, X. Yin, C. Li, P. Zhang, et al., Oscar: Object-semantics aligned pre-training for vision-language tasks, 2020. arXiv:2004.06165.
  * Tang et al. [2023] Y.-C. Tang, K.-D. Wang, T.-Y. Ou, W.-C. Peng,  NYCU_TWO at Memotion 3: Good foundation, good teacher, then you have good meme analysis,  in: Proceedings of De-Factify 2: Workshop on Multimodal Fact Checking and Hate Speech Detection, CEUR, 2023.
  * Liu et al. [2021] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, B. Guo, Swin transformer: Hierarchical vision transformer using shifted windows, 2021. arXiv:2103.14030.
  * Guo et al. [2023] X. Guo, J. Ma, A. Zubiaga,  NUAA-QMUL-AIIT at Memotion 3: Multi-modal fusion with squeeze-and-excitation for internet meme emotion analysis,  in: Proceedings of De-Factify 2: Workshop on Multimodal Fact Checking and Hate Speech Detection, CEUR, 2023.
  * Liu et al. [2019] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov,  Roberta: A robustly optimized bert pretraining approach (2019).
  * Zhai et al. [2020] X. Zhai, J. Puigcerver, A. Kolesnikov, et al., A large-scale study of representation learning with the visual task adaptation benchmark, 2020\. arXiv:1910.04867.
  * Ke et al. [2017] G. Ke, Q. Meng, T. Finley, et al.,  Lightgbm: A highly efficient gradient boosting decision tree,  in: Neurips, 2017.
  * He et al. [2016] K. He, X. Zhang, S. Ren, J. Sun,  Deep residual learning for image recognition,  in: CVPR, 2016.
  * Schroff et al. [2015] F. Schroff, D. Kalenichenko, J. Philbin,  FaceNet: A unified embedding for face recognition and clustering,  in: CVPR, 2015.


# Understanding Gaussian Attention Bias of Vision Transformers Using Effective
Receptive Fields

Bum Jun Kim
POSTECH
<EMAIL_ADDRESS>
&Hyeyeon Choi
POSTECH
<EMAIL_ADDRESS>
&Hyeonah Jang
POSTECH
<EMAIL_ADDRESS>
&Sang Woo Kim
POSTECH
<EMAIL_ADDRESS>

###### Abstract

Vision transformers (ViTs) that model an image as a sequence of partitioned
patches have shown notable performance in diverse vision tasks. Because
partitioning patches eliminates the image structure, to reflect the order of
patches, ViTs utilize an explicit component called positional embedding.
However, we claim that the use of positional embedding does not simply
guarantee the order-awareness of ViT. To support this claim, we analyze the
actual behavior of ViTs using an effective receptive field. We demonstrate
that during training, ViT acquires an understanding of patch order from the
positional embedding that is trained to be a specific pattern. Based on this
observation, we propose explicitly adding a Gaussian attention bias that
guides the positional embedding to have the corresponding pattern from the
beginning of training. We evaluated the influence of Gaussian attention bias
on the performance of ViTs in several image classification, object detection,
and semantic segmentation experiments. The results showed that proposed method
not only facilitates ViTs to understand images but also boosts their
performance on various datasets, including ImageNet, COCO 2017, and ADE20K.

##  1 Introduction

Vision transformers (ViTs) [1] have achieved remarkable performances in
various vision tasks that are often superior to those of convolutional neural
networks (CNNs) [2, 3, 4]. Unlike CNNs, ViTs partition an image into a
sequence of patches and subsequently combine patch features based on the self-
attention (SA) mechanism, enabling the aggregation of rich global information
within the image [5, 6, 7].

Despite its effectiveness, SA poses inherent limitations in understanding the
order of input patches. However, because 2D images are structured data,
understanding the order of patches is important for ViT [8]. To overcome this
problem, ViTs employ an explicit component called positional embedding that
enables the identification of the order and the corresponding geometric
positions of patches.

However, we claim that simply using positional embeddings does not ensure
order-awareness. To validate our claim, we utilize the effective receptive
field (ERF) [9, 10] that highlights the pixels actually used in perception,
illustrating how ViTs understand images. Using ERF, we demonstrate that ViTs
with untrained positional embeddings do not discriminate between near and far
patches and that order-awareness is acquired after positional embedding is
trained to obtain specific patterns.

Motivated by this observation, to construct a ViT born with the spatial
understanding of images, we propose injecting Gaussian attention bias into
positional embedding. The innate spatial understanding of images helps ViT
capture the nearness and farness of pixels, thereby enhancing the performance
of ViT in vision tasks. We observed that using Gaussian attention bias
improved the performance of ViTs on several datasets, tasks, and models.

##  2 Background

#### Forward propagation of standard ViT

Let $\mathbf{x}\in^{H\times W\times C}$ be the input image to the ViT, where
$H\times W$ is the resolution, and $C$ is the number of channels of the image.
Using a predefined resolution of patch $P\times P$, ViT spatially partitions
the image into $N$ nonoverlapping patches, where $N=HW/P^{2}$. Each patch is
linearly projected, yielding
$\operatorname{PatchEmbedding}(\mathbf{x})$.111The concatenation of the class
token is ignored in this study for notational simplicity. Subsequently, an
absolute positional embedding $\mathbf{E}_{\operatorname{pos}}$ is added,
resulting in $\mathbf{z}_{0}$, the input to the first transformer block. Now,
transformer blocks containing SAs and multilayer perceptrons (MLPs) are
applied in a row to produce $\mathbf{z}_{L}$, where $L$ is the number of
transformer blocks. Finally, LayerNorm [11] is applied to produce the last
feature map $\mathbf{y}$ from which the head produces a classification score.
Here, $\operatorname{PatchEmbedding}(\mathbf{x})$,
$\mathbf{E}_{\operatorname{pos}}$, $\mathbf{z}_{l}$, and $\mathbf{y}$ have the
same size of N×D, where $D$ is the dimension of the patch features.

#### Trick to obtain the ERF of ViT

Because the ERF of ViT has been rarely discussed and our study is the first to
provide a concrete and detailed analysis on this topic, we first formulate the
ERF of ViT. The ERF depicts the actual usage of each pixel for determining the
target feature in a neural network, representing a generic connection between
them. We follow the common tricks to obtain ERFs of CNNs [12]. However, unlike
CNNs, to obtain the ERF of ViT, we should focus on the patch unit. To
investigate the properties of the entire ViT, the last feature map
$\mathbf{y}$ is chosen as the target feature map. First, we target the $n$th
patch corresponding to the central patch. Because our goal is to analyze the
spatial relationship between the target patch and pixel units, we ignore other
units such as the image channel and the dimension of the patch feature. Thus,
the features of the central patch are averaged over its dimensions:
$Y=\frac{1}{D}\sum_{d=1}^{D}{\mathbf{y}_{n,d}}$. Now, we examine the
contribution of each pixel to $Y$ that can be obtained by a gradient
$\pdv{Y}{\mathbf{x}}\in^{H\times W\times C}$. The gradient is averaged over
channels to obtain
$\mathbf{G}=\frac{1}{C}\sum_{c=1}^{C}{[\pdv{Y}{\mathbf{x}}]_{c}}$. At this
point, $\mathbf{G}\in^{H\times W}$ contains the spatial relationship between
the targeted patch feature and pixels. However, because $\mathbf{G}$ arises
from the forward and backward operations of a single image, $\mathbf{G}$ is
strongly dependent on the input image rather than on the ViT’s properties. To
capture the general behavior of the ViT, $\mathbf{G}$ is averaged over a
sufficiently large number of images. At this time, because negative values in
$\mathbf{G}$ cancel out the positive values, we ignore the negative importance
using $\operatorname{ReLU}$ [12, 13, 14]. Thus, we obtain
$\mathbf{R}=\frac{1}{\lvert S\rvert}\sum_{\mathbf{x}\in
S}\operatorname{ReLU}(\mathbf{G})$, where $S$ denotes an image dataset.
Because it is averaged over numerous images, $\mathbf{R}\in^{H\times W}$
represents the general relationship between the pixels and the targeted patch
feature of the ViT corresponding to ERF.

##  3 Effective Receptive Fields of Vision Transformers

###  3.1 Qualitative Analysis

In this section, we qualitatively analyze the ERFs of ViTs. Although the ERFs
of ResNets [15, 16] resemble a 2D Gaussian, the ERFs of ViTs exhibit a
different shape owing to nonoverlapping patch partitioning [17]. Figure 1
shows the ERFs of ViT-B with different patch sizes of $\\{32,16,8\\}$. First,
the ERF of ViT mainly highlights the targeted central patch and slightly uses
information from other patches. This behavior indicates that each patch
feature is responsible for representing information in the corresponding
patch, while it is combined with certain global information from other patches
via SA.

|  |  |
---|---|---|---
(a) ViT-B/32, 2242 | (b) ViT-B/16, 2242 | (c) ViT-B/8, 2242 | (d) ViT-B/16, 3842
Figure 1: ERFs of ViT with different patch sizes. Because printed figures can be seen improperly, we highly encourage viewing all images electronically with zoom. |  |  |
---|---|---|---
(a) ViT-S/16 | (b) ViT-L/16 | (c) ViT-S/16 (U) | (d) ViT-L/16 (U)
Figure 2: ERFs of ViT with different model sizes. (U) indicates an untrained
model.

For ViT, a large-sized model was observed to yield a widespread ERF (Figure 1
(b) and Figure 2 (a, b)). This observation is expected because a larger ViT
further stacks wider layers. However, for untrained ViTs, the ERFs exhibited
no difference (Figure 2 (c, d)). This observation shows that the wider ERF of
the ViT-L/16 arises from not only architectural largeness but also pretrained
weights.

In addition, we obtained ERFs of other ViT variants (Figure 3). Among them,
the DeiT [18], DeiT III [19], and BEiT [20] have nearly identical
architectures to the ViT with different pretrained weights, yielding similar
but slightly different ERFs. CaiT [21] exhibits architectural modifications,
such as class attention, but yielded a similar ERF to that of ViT. Compared
with others, XCiT [22] and Swin [23] exhibit wider ERFs. These include
explicit modules that allow communication with the neighboring patches, such
as the local patch interaction module, patch merging layer, and shifted window
partitioning. In summary, we observed that the ERFs of ViTs were represented
as highlights of the targeted patch area with other patches being partially
utilized (See the Appendix for further analysis of the ERF of ViT).

|  |  |
---|---|---|---
(a) DeiT-B/16 [18] | (b) DeiT III-B/16 [19] | (c) BEiT-B/16 [20] | (d) CaiT-S-24 [21]
|  |  |
(e) XCiT-M24/16 [22] | (f) Swin-B [23] | (g) ResNet-50 [15] | (h) X-101 [16]
Figure 3: ERFs of ViTs and CNNs. “X” indicates ResNeXt with 32 cardinality.

###  3.2 Spatial Understanding of Images by ViTs

The interesting aspect of ERF is that it illustrates how ViTs understand
spatial images. Although the ERF of ViT shows that the majority of the
activated pixels are in the target patch, adjacent patches are more activated
than distant patches, yielding a roughly ✚-shape. This behavior implies that
the ViTs have order-awareness in the sequence of patches, enabling them to use
more information from nearby patches and less from far patches, which we refer
to as the spatial understanding of images. The spatial understanding of images
is one of the critical components for obtaining high-performance ViT.
References [24] and [23] observed that ViTs without any positional embedding
yielded decreased performance.

Positional embeddings in ViTs exist in diverse forms. The absolute positional
embedding (APE) can be either a predefined sinusoidal sequence [25] or
learnable parameter [1] and is added to the patch embedding. The relative
positional embedding (RPE) [26, 27], also called attention bias [28], is added
to the attention matrix for each layer as

$\displaystyle\operatorname{Attention}_{l}(\mathbf{Q}_{l},\mathbf{K}_{l},\mathbf{V}_{l})=\operatorname{softmax}\left(\frac{\mathbf{Q}_{l}\mathbf{K}_{l}^{\top}}{\sqrt{D}}+\mathbf{B}_{\operatorname{rel},l}\right)\mathbf{V}_{l},$
(1)

where $\mathbf{Q}_{l},\mathbf{K}_{l},\mathbf{V}_{l}\in^{N\times D}$ are the
query, key, and value in SA, respectively, and
$\mathbf{B}_{\operatorname{rel},l}\in^{N\times N}$ is the RPE as attention
bias. To obtain $\mathbf{B}_{\operatorname{rel},l}$, the original Swin
transformer [23] used a learnable table called RelPosBias that provided
$\mathbf{B}_{\operatorname{rel},l}$ for each relative coordinate. SwinV2 [29]
employed a learnable MLP to obtain $\mathbf{B}_{\operatorname{rel},l}$ from
each relative coordinate; this term is also called RelPosMlp. Although the
original ViT used APE [1], recent ablation studies [30, 20, 24, 23] have
reported that using RPE yielded improved performance. Reference [8] claims
that although APE has provided successful results in natural language
processing tasks, relative information from RPE is crucial for vision tasks.

|  |  |
---|---|---|---
|  |  |
(a) ViT-S/16 | (b) ViT-B/16 | (c) ViT-M/16 (R) | (d) ViT-B/16 (R)
Figure 4: ERFs of ViTs, where (R) indicates the model with RPE. The second row
illustrates ERFs when the APE or RPE is re-initialized to random parameters.
Note that the ✚-shape is lost in the second row.

We examined the role of positional embeddings. We obtained ERFs before and
after the APE or RPE of pretrained ViTs was re-initialized to random
parameters (Figure 4). We observed that re-initializing APE or RPE altered
ERFs, causing adjacent patches to lose their contribution to the target patch
feature and exhibit the same contribution as far patches. This observation
clearly demonstrates that SA itself cannot understand the location of patches,
and positional embedding plays a significant role in the spatial understanding
of images.

|  |  |
---|---|---|---
(a) $n=0$ | (b) $n=1$ | (c) $n=2$ | (d) $n=195$
|  |  |
(e) $n=0$ | (f) $n=1$ | (g) $n=2$ | (h) $n=195$
Figure 5: RPE of ViT-B/16 (R) for each patch index. The first row is obtained
from the pretrained model, whereas the second row is obtained from the
untrained model.

To investigate the underlying mechanism, we extracted the learned and
untrained RPEs. From $\mathbf{B}_{\operatorname{rel},l}\in^{N\times N}$, the
RPE of the $n$th patch $\mathbf{B}_{\operatorname{rel},l,n}\in^{N}$ was
obtained for $n\in\\{0,1,\cdots,N-1\\}$ and was reshaped into
$\mathbf{B}_{\operatorname{rel},l,n}^{\prime}\in^{H/P\times W/P}$. For
visualization, $\mathbf{B}_{\operatorname{rel},l,n}^{\prime}$ was averaged
over multi-head. The RPE in the first attention layer is visualized in Figure
5. Note that the learned RPE appeared as a sliced 2D Gaussian, distinguishing
between near and distant patches. Because $\operatorname{softmax}$ computes an
exponential ratio, the bias term $\mathbf{B}_{\operatorname{rel},l}$ in Eq. 1
becomes an exponential coefficient that amplifies each element of the
attention matrix. Thus, a higher RPE value means a larger amplification of the
corresponding patch. Importantly, we observed that untrained RPEs showed
distance-independent values. In other words, re-initializing RelPosMlp
provides a random RPE that does not discriminate between near and far patches.

|  |  |
---|---|---|---
(a) $l=1$ | (b) $l=10$ | (c) $l=11$ | (d) $l=12$
Figure 6: RPE corresponding to the center was extracted for each layer of
ViT-B/16 (R).

ViT-S/16, 2242 (R) ViT-M/16, 2242 (R) ViT-B/16, 2242 (R) $l$ $R^{2}$
$\hat{\sigma_{X}}$ $\hat{\sigma_{Y}}$ $R^{2}$ $\hat{\sigma_{X}}$
$\hat{\sigma_{Y}}$ $R^{2}$ $\hat{\sigma_{X}}$ $\hat{\sigma_{Y}}$ 1 0.731 6.837
7.063 0.893 4.219 4.113 0.914 4.553 4.394 2 0.798 4.704 4.538 0.728 6.257
5.679 0.573 6.672 6.719 3 0.831 6.185 6.392 0.824 4.715 5.039 0.870 4.649
4.849 4 0.867 4.757 5.020 0.838 5.250 5.355 0.813 4.901 5.404 5 0.753 6.798
5.310 0.795 5.597 4.920 0.853 5.055 4.807 6 0.730 5.624 4.631 0.694 8.054
5.540 0.817 5.421 4.276 7 0.796 5.872 4.848 0.844 5.509 4.660 0.877 6.895
5.020 8 0.805 4.865 5.473 0.798 5.715 5.010 0.825 5.640 4.006 9 0.771 5.668
5.681 0.729 5.472 6.538 0.873 5.328 4.914 10 0.786 5.111 6.125 0.878 4.430
5.348 0.896 5.342 6.132 11 0.231 8.709 272.743 0.359 5.824 298.676 0.012
21.137 702.646 12 0.019 690.530 181.928 0.002 396.639 415.174 0.004 579.639
332.651

Table 1: Results of fitting RPEs to a 2D Gaussian.

We also examined whether RPEs appear as 2D Gaussians across all layers (Figure
6). We fitted the ERF of ViTs to a 2D Gaussian using the LMfit [31] library
(Table 1). The coefficient of determination $R^{2}$ indicates how exactly ERF
fits a 2D Gaussian, ideally 1. The standard deviations $\hat{\sigma_{X}}$ and
$\hat{\sigma_{Y}}$ represent the wideness of the 2D Gaussian. We discovered
that, for the majority of layers, RPE fitted to the 2D Gaussian with
$R^{2}>0.7$. The exceptions, whose RPE showed no pattern, were found in the
last two layers of $l\in\\{11,12\\}$ that were close to the classifier head.

We interpret these observations as follows: Initially, an untrained RPE
exhibits a random pattern and cannot distinguish between near and far patches.
However, because RPE is learnable, ViTs can choose to _acquire_ an
understanding of the different positions of patches. After training, the RPE
becomes a pattern close to a 2D Gaussian, discriminating the different
positions of patches. The learned RPE allows ViTs to understand near and far
patches. These observations motivate us to design a new RPE method.

##  4 Proposed Method

Our objective is to design an RPE that easily recognizes close and distant
patches to facilitate ViTs to acquire spatial understanding of images. In
light of the observation that learned RPE fits suitably with a 2D Gaussian, we
propose injecting _Gaussian attention bias_ into RPE:

$\displaystyle\operatorname{Attention}_{l}(\mathbf{Q}_{l},\mathbf{K}_{l},\mathbf{V}_{l})=\operatorname{softmax}\left(\frac{\mathbf{Q}_{l}\mathbf{K}_{l}^{\top}}{\sqrt{D}}+\mathbf{B}_{\operatorname{rel},l}+\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{B}_{\operatorname{Gaussian},l}\right)\mathbf{V}_{l}.$
(2)

Here, we aim to build $\mathbf{B}_{\operatorname{Gaussian},l}$ so that the
bias terms
$\mathbf{B}_{\operatorname{rel},l}+\mathbf{B}_{\operatorname{Gaussian},l}$
readily appear as a 2D Gaussian, even in the initial state. By reversing the
process of extracting RPE in Figure 5, we build
$\mathbf{B}_{\operatorname{Gaussian},l}$ by stacking sliced 2D Gaussians.

First, for the $l$th layer, we generate a 2D Gaussian table using
$A_{l},\sigma_{l}\in$:

$\displaystyle f(x,\ y)$
$\displaystyle=A_{l}^{2}\exp\left(-\left(\frac{(x-x_{c})^{2}}{2\sigma_{l}^{2}}+\frac{(y-y_{c})^{2}}{2\sigma_{l}^{2}}\right)\right),$
(3)

where $x=1,2,\cdots,2W/P-1$, $y=1,2,\cdots,2H/P-1$, and $(x_{c},\ y_{c})$
correspond to the central coordinate. Note that the amplitude is set to
$A_{l}^{2}$ to ensure a non-negative amplitude for any $A_{l}$. The variance
$\sigma_{l}^{2}$ is shared for the horizontal and vertical directions. Thus,
we parameterize the 2D Gaussian with only two parameters, $A_{l}$ and
$\sigma_{l}$.

Figure 7: Illustration on how we obtain Gaussian attention bias.

Second, sliced Gaussians are obtained. As shown in Figure 7, the first sliced
Gaussian should have the center coordinate of the 2D Gaussian (*) at the top-
left (red box), whereas the last sliced Gaussian should exhibit the center
coordinate (*) at the bottom-right (blue box). This slicing ensures that it
resembles the learned RPE in Figure 5. Finally, each sliced Gaussian is
reshaped and stacked to build $\mathbf{B}_{\operatorname{Gaussian},l}$, whose
size is the same as $\mathbf{B}_{\operatorname{rel},l}$.

Our design of Gaussian attention bias has several advantages. First, because
we designed Gaussian attention bias as an additional bias
$\mathbf{B}_{\operatorname{Gaussian},l}$, it can be seamlessly plugged into
any type of RPE, including RelPosBias and RelPosMlp. In other words, if we
attempt to implement a redesign of RelPosMlp to resemble a 2D Gaussian, it
cannot be applied to other RPEs, such as RelPosBias.

Second, our Gaussian attention bias is hyperparameter-free. Note that
$\mathbf{B}_{\operatorname{Gaussian},l}$ is parameterized by $A_{l}$ and
$\sigma_{l}$ with a differentiable function (Eq. 3). Thus, $A_{l}$ and
$\sigma_{l}$ can be set as learnable parameters in gradient descent
optimization. We do not need the trial-and-error-based hyperparameter tuning
on $A_{l}$ and $\sigma_{l}$. Because $\sigma_{l}$ determines the wideness of
the 2D Gaussian, the flexibility of $\sigma_{l}$ is beneficial when using ViTs
for other datasets or tasks that require different sizes of ERF. Furthermore,
different $A_{l}$ and $\sigma_{l}$ values are allowed for each layer. As we
observed in Table 1, as the last two layers did not learn to be a 2D Gaussian,
it is preferable to allow different behaviors in the last two layers. For
example, the last two layers can naturally choose $A_{l}$ to be zero.

Finally, we benefit from the learnability of the original RPE, such as
RelPosBias or RelPosMlp. Indeed, RPEs such as RelPosBias or RelPosMlp have a
significant number of parameters that enable enriched expression in SA.
Considering this behavior, we allow the degree of freedom of the original RPE.

However, we remove unnecessary degrees of freedom from our Gaussian attention
bias. We do not generate multiple Gaussian tables; rather, we use a single
Gaussian table to ensure that sliced Gaussians are shifted versions of each
other, inspired by the use of relative coordinates in RPE. We do not use a
constant term in our Gaussian function at Eq. 3 because
$\operatorname{softmax}$ is invariant to constant translation [32, 33]:
$\operatorname{softmax}(\mathbf{x}+C)=\operatorname{softmax}(\mathbf{x})$.
Finally, we choose to share the Gaussian attention bias across multiple heads
of SA within the same layer (See the Appendix for the ablation study).

##  5 Experiments

Now, we investigate the influence of Gaussian attention bias on the
performance of ViTs.

Dataset | Model | RPE w/o GAB | RPE w/ GAB | Difference
---|---|---|---|---
ImageNet-1K | ViT-S/16 (R) | 80.576 | 80.724 | +0.157
ViT-M/16 (R) | 81.224 | 81.249 | +0.025
ViT-B/16 (R) | 81.381 | 81.484 | +0.102
Table 2: Top-1 accuracy on the ImageNet-1K dataset. All the accuracies in this
paper are expressed in percentage units. “GAB” indicates Gaussian attention
bias.

#### ImageNet-1K

First, we trained the ViTs on the image classification task using the
ImageNet-1K dataset [34] from scratch. ViT-$\\{$S, M, L$\\}$/16 (R) using
RelPosMlp without APE were used. See the Appendix for experimental details,
such as the hyperparameters. For each model with and without Gaussian
attention bias, the top-1 accuracy was measured (Table 2). The top-1 accuracy
of the three models improved after incorporating Gaussian attention bias.

Dataset | Model | RPE w/o GAB | RPE w/ GAB | Difference
---|---|---|---|---
Oxford-IIIT Pet | ViT-S/16 (R) | 91.486 | 92.780 | +1.294
ViT-M/16 (R) | 92.810 | 92.960 | +0.150
ViT-B/16 (R) | 93.381 | 93.743 | +0.362
Caltech-101 | ViT-S/16 (R) | 88.403 | 90.202 | +1.799
ViT-M/16 (R) | 89.132 | 89.983 | +0.851
ViT-B/16 (R) | 89.254 | 89.570 | +0.316
Stanford Cars | ViT-S/16 (R) | 80.126 | 83.079 | +2.953
ViT-M/16 (R) | 80.731 | 83.890 | +3.159
ViT-B/16 (R) | 80.154 | 82.612 | +2.458
Stanford Dogs | ViT-S/16 (R) | 81.535 | 82.507 | +0.972
ViT-M/16 (R) | 85.088 | 85.714 | +0.626
ViT-B/16 (R) | 89.256 | 90.185 | +0.929
Table 3: Test accuracy with and without Gaussian attention bias on other
datasets.

#### Other Datasets

To further examine the performance difference, we targeted image
classification on other datasets: Oxford-IIIT Pet [35], Caltech-101 [36],
Stanford Cars [37], and Stanford Dogs [38]. Test accuracy was measured for
each ViT that used RelPosMlp with and without Gaussian attention bias. We
observed that the use of Gaussian attention bias consistently improved the
test accuracies of the three ViTs on the four datasets (Table 3). For these
experiments, each dataset contained objects of various sizes, whose
classification requires different sizes of ERF or $\sigma_{l}$. Because we
designed $\sigma_{l}$ as learnable, the model with Gaussian attention bias can
flexibly cope with different ERFs. Indeed, the learned $\sigma_{l}$ achieved
similar but slightly different values for each dataset (See the Appendix).

Backbone | RPE Method | COCO | ADE20K
---|---|---|---
AP${}^{\textrm{box}}$ | AP${}^{\textrm{mask}}$ | mIoU | aAcc
Swin-S | RelPosBias w/o GAB | 48.12 | 43.03 | 46.16 | 81.82
RelPosBias w/ GAB | 48.23 | 43.13 | 46.41 | 82.09
Difference | +0.11 | +0.10 | +0.25 | +0.27
Table 4: Experimental results in terms of object detection and semantic
segmentation.

#### Object Detection and Semantic Segmentation

Finally, we targeted two downstream tasks: object detection on the COCO 2017
dataset [39] and semantic segmentation on the ADE20K dataset [40]. To further
investigate our proposed method with a different setup, we targeted Swin-S
with RelPosBias as the backbone. Using the COCO dataset, we measured bounding
box mAP (AP${}^{\textrm{box}}$) on object detection and segmentation mAP
(AP${}^{\textrm{mask}}$) on instance segmentation. Using the ADE20K dataset,
the mean intersection over union (mIoU) and mean accuracy over all pixels
(aAcc) were measured. We observed that the Swin transformers with Gaussian
attention bias exhibited improvements across all four indices (Table 4).

##  6 Conclusion

In this study, we analyzed how ViTs understand spatial images. From detailed
analyses of the ERF of ViTs, we discovered that ViTs acquired spatial
understanding of images during training and that this phenomenon was caused by
the underlying transition from randomized positional embedding to a learned
one. To guide the understanding of the spatial nearness and farness of
patches, we proposed injecting Gaussian attention bias into ViTs. In several
experiments on image classification, object detection, and semantic
segmentation, ViTs integrated with Gaussian attention bias achieved superior
results.

## References

  * [1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In ICLR, 2021.
  * [2] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, and Li Zhang.  Rethinking Semantic Segmentation From a Sequence-to-Sequence Perspective With Transformers.  In CVPR, pages 6881–6890, 2021.
  * [3] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo.  SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers.  In NeurIPS, pages 12077–12090, 2021.
  * [4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-End Object Detection with Transformers.  In ECCV (1), volume 12346, pages 213–229, 2020.
  * [5] Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi.  On the Relationship between Self-Attention and Convolutional Layers.  In ICLR, 2020.
  * [6] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun.  Exploring Self-Attention for Image Recognition.  In CVPR, pages 10073–10082, 2020.
  * [7] Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Thomas Rainforth, and Yarin Gal.  Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning.  In NeurIPS, pages 28742–28756, 2021.
  * [8] Irwan Bello.  LambdaNetworks: Modeling long-range Interactions without Attention.  In ICLR, 2021.
  * [9] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard S. Zemel.  Understanding the Effective Receptive Field in Deep Convolutional Neural Networks.  In NIPS, pages 4898–4906, 2016.
  * [10] André Araujo, Wade Norris, and Jack Sim.  Computing receptive fields of convolutional neural networks.  Distill, 4(11):e21, 2019.
  * [11] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer Normalization.  CoRR, abs/1607.06450, 2016.
  * [12] Bum Jun Kim, Hyeyeon Choi, Hyeonah Jang, Dong Gu Lee, Wonseok Jeong, and Sang Woo Kim.  Dead pixel test using effective receptive field.  Pattern Recognit. Lett., 167:149–156, 2023.
  * [13] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization.  Int. J. Comput. Vis., 128(2):336–359, 2020.
  * [14] Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader, and Vineeth N. Balasubramanian.  Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks.  In WACV, pages 839–847, 2018.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In CVPR, pages 770–778, 2016.
  * [16] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated Residual Transformations for Deep Neural Networks.  In CVPR, pages 5987–5995, 2017.
  * [17] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy.  Do Vision Transformers See Like Convolutional Neural Networks?  In NeurIPS, pages 12116–12128, 2021.
  * [18] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In ICML, volume 139, pages 10347–10357, 2021.
  * [19] Hugo Touvron, Matthieu Cord, and Hervé Jégou.  DeiT III: Revenge of the ViT.  In ECCV (24), volume 13684, pages 516–533, 2022.
  * [20] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  BEiT: BERT Pre-Training of Image Transformers.  In ICLR, 2022.
  * [21] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou.  Going deeper with Image Transformers.  In ICCV, pages 32–42, 2021.
  * [22] Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou.  XCiT: Cross-Covariance Image Transformers.  In NeurIPS, pages 20014–20027, 2021.
  * [23] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.  In ICCV, pages 9992–10002, 2021.
  * [24] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao.  Rethinking and Improving Relative Position Encoding for Vision Transformer.  In ICCV, pages 10013–10021, 2021.
  * [25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is All you Need.  In NIPS, pages 5998–6008, 2017.
  * [26] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani.  Self-Attention with Relative Position Representations.  In NAACL-HLT (2), pages 464–468, 2018.
  * [27] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck.  Music Transformer: Generating Music with Long-Term Structure.  In ICLR, 2019.
  * [28] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze.  LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference.  In ICCV, pages 12239–12249, 2021.
  * [29] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo.  Swin Transformer V2: Scaling Up Capacity and Resolution.  In CVPR, pages 11999–12009, 2022.
  * [30] Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani.  Bottleneck Transformers for Visual Recognition.  In CVPR, pages 16519–16529, 2021.
  * [31] Matthew Newville, Till Stensitzki, Daniel B Allen, Michal Rawlik, Antonino Ingargiola, and Andrew Nelson.  LMFIT: Non-linear least-square minimization and curve-fitting for Python.  Astrophysics Source Code Library, pages ascl–1606, 2016.
  * [32] Anirban Laha, Saneem Ahmed Chemmengath, Priyanka Agrawal, Mitesh M. Khapra, Karthik Sankaranarayanan, and Harish G. Ramaswamy.  On Controllable Sparse Alternatives to Softmax.  In NeurIPS, pages 6423–6433, 2018.
  * [33] André F. T. Martins and Ramón Fernandez Astudillo.  From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification.  In ICML, volume 48, pages 1614–1623, 2016.
  * [34] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In CVPR, pages 248–255, 2009.
  * [35] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar.  Cats and dogs.  In CVPR, pages 3498–3505, 2012.
  * [36] Li Fei-Fei, Robert Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories.  Comput. Vis. Image Underst., 106(1):59–70, 2007.
  * [37] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3D Object Representations for Fine-Grained Categorization.  In ICCV Workshops, pages 554–561, 2013.
  * [38] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li.  Novel dataset for fine-grained image categorization: Stanford dogs.  In Proc. CVPR workshop on fine-grained visual categorization (FGVC), volume 2, 2011.
  * [39] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: Common Objects in Context.  In ECCV (5), volume 8693, pages 740–755, 2014.
  * [40] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Semantic Understanding of Scenes Through the ADE20K Dataset.  Int. J. Comput. Vis., 127(3):302–321, 2019.