Conceptualization, Methodology, Software, Writing

1]organization=Department of Computer Science, Sapienza University of Rome,
addressline=Via Salaria 113, city=Rome, postcode=00198, state=Italy,
country=Ialy 2]organization=Department of Law and Economics, University of
Rome UnitelmaSapienza, addressline=Piazza Sassari 4, city=Rome,
postcode=00161, state=Italy, country=Ialy

Supervision, Conceptualization, Writing

Conceptualization, Methodology, Writing

Supervision

# ReViT: Enhancing Vision Transformers with Attention Residual Connections for
Visual Recognition

Anxhelo Diko<EMAIL_ADDRESS>[  [  Danilo Avola<EMAIL_ADDRESS>Marco
Cascio<EMAIL_ADDRESS>Luigi Cinque<EMAIL_ADDRESS>
###### Abstract

Vision Transformer (ViT) self-attention mechanism is characterized by feature
collapse in deeper layers, resulting in the vanishing of low-level visual
features. However, such features can be helpful to accurately represent and
identify elements within an image and increase the accuracy and robustness of
vision-based recognition systems. Following this rationale, we propose a novel
residual attention learning method for improving ViT-based architectures,
increasing their visual feature diversity and model robustness. In this way,
the proposed network can capture and preserve significant low-level features,
providing more details about the elements within the scene being analyzed. The
effectiveness and robustness of the presented method are evaluated on five
image classification benchmarks, including ImageNet1k, CIFAR10, CIFAR100,
Oxford Flowers-102, and Oxford-IIIT Pet, achieving improved performances.
Additionally, experiments on the COCO2017 dataset show that the devised
approach discovers and incorporates semantic and spatial relationships for
object detection and instance segmentation when implemented into spatial-aware
transformer models.

###### keywords:

Vision Transformer Feature Collapse Self-attention Mechanism Residual
Attention Learning Visual Recognition

##  1 Introduction

Nowadays, automatic visual recognition systems have become increasingly
popular as powerful support tools for a wide range of vision-related
applications, e.g., object detection and tracking [4, 9], image analysis and
classification [42], or scene segmentation [37] and understanding [10]. Such
systems have been developed to mimic the remarkable human brain ability to
perfectly correlate low-level visual features (e.g., edges, color, or shapes)
and semantic-level information for perceptual identification and recognition
tasks [1, 2]. In the past decade, deep learning algorithms have been
extensively employed to model how the human brain processes visual information
since artificial neural networks are inspired by the structure and function of
biological neurons; indeed, they can analyze visual data learning to recognize
patterns and features in images [27]. Precisely, the understanding of visual
content has been revolutionized by Convolutional Neural Networks (CNNs),
primarily because of their ability to analyze an image split into smaller
patches, extract multi-scale localized features, and synthesize them to
generate highly expressive representations [21, 19]. Despite this, due to its
limited receptive field, the convolutional operation of standard CNNs cannot
recognize long-range dependencies, such as arbitrary relationships among
distant image regions. To address this limitation and facilitate the handling
of long-range dependencies in visual tasks, several architectures based on
transformers have emerged [7, 33]. Notably, the traditional transformer neural
network initially gained popularity in the field of natural language
processing [34]. This popularity stemmed from its self-attention mechanisms,
which enable the modeling of extensive dependencies between tokens within a
sequence of data, overcoming the limitation of CNNs. Additionally, in contrast
to CNNs, specifically engineered for visual tasks with pre-existing knowledge
and incorporating inductive biases such as locality and translation
equivariance [8], ViT adopts an approach where an image is treated as a
sequence of patches. This different architectural choice leads to a deficiency
in such inductive biases, making ViT less effective at generalizing when
trained on smaller datasets [7]. Nevertheless, it has been observed that the
training scale can override the need for strong inductive biases, showcasing
the substantial potential of transformers when trained on extensive datasets
[7, 8, 33].

Figure 1: Illustration of feature maps learned from ReViT and ViT obtained
using GradCAM [31] algorithm.

Standard ViT models are characterized by their deep architecture, comprising
multiple sequential computational blocks. Each of these blocks consists of a
self-attention layer, a multi-layer perceptron (MLP), and a residual
connection that bypasses the self-attention layer. This hierarchical structure
necessitates multiple consecutive passes to process the input token sequence
and compute scene representations. While the input token sequence traverses
deeper layers of the network, tokens tend to become highly similar to each
other. This phenomenon leads to a loss of feature diversity, commonly referred
to as feature collapsing. The main contributing factor to this phenomenon is
the global nature of the self-attention mechanism and its tendency to combine
features across patches [32]. Specifically, as token processing progresses
through the network’s deeper layers, the generated attention maps become
increasingly global, aggregating features between patches at a global scale
and losing their focus on local information [7, 8]. This shift from local
awareness causes the network to forget low-level features and only emphasize
global characteristics. While the ability to capture global context is a
desirable trait of transformers, considering low-level cues such as shape and
color, typically found in shallower network layers, can be advantageous for
vision-related tasks like image classification and object detection. In
addition to the globality issue, ViT architectures are also considered less
robust at translation invariance compared to CNNs [16, 7]. This limitation
makes ViTs less reliable in real-world scenarios where objects may appear in
varying positions and sizes.

Inspired by the shortcomings observed in the ViT architecture and insights
from neuroscience, which suggest that the human brain transmits visual
information through a sequential hierarchical process from simple to
specialized cells via the aggregation of electrical impulses [15], we present
the Residual Attention Vision Transformer
(ReViT)111https://github.com/ADiko1997/ReViT. This innovative network
architecture introduces an attention residual learning technique, enhancing
the performance of visual recognition tasks by integrating low-level
information from shallow to deeper network layers and contrasting the over-
globalization of the attention mechanism. Specifically, given standard vision
transformer blocks, attention information is passed between consecutive layers
and integrated with the attention information of the current layer using a
permutation invariant aggregation function. This connection facilitates the
transmission and accumulation of attention-related information from shallower
to deeper layers, enabling the ability to identify low-level features while
preserving the original ability of transformers to extract global context.
Consequently, by empowering the attention mechanism with the ability to
capture such features, the resulting learned representations are expected to
exhibit greater feature diversity. Additionally, since low-level features like
color and shape can be useful in identifying objects in a scene regardless of
their position and size, this mechanism improves the translation invariance of
ViT. In order to assess the efficacy of incorporating residual attention into
ViT (i.e., ReViT), we conducted a comprehensive empirical analysis. Our
evaluation spanned five image classification benchmarks, namely ImageNet1K
[30], CIFAR-10 [18], CIFAR-100 [18], Oxford Flowers-102 [28], and Oxford-IIIT
Pet [29]. The results demonstrated significant performance enhancements when
using ReViT compared to the original ViT counterpart approach in all datasets.
Furthermore, due to their high image quality, we employed the Oxford
Flowers-102 and Oxford-IIIT Pet datasets to highlight ReViT superiority over
ViT in image classification tasks involving translation invariance, such as
horizontal and vertical shifts, as well as scale invariance. These experiments
provided quantitative evidence of ReViT advantages in handling these
scenarios. Moreover, we quantitatively analyze the global spread of the
attention map in ViT and ReViT models employing the non-locality metric [8]
and show that our approach can preserve more local information than ViT. Apart
from our quantitative evaluations, we conduct a qualitative analysis by
comparing the feature maps learned by ViT and ReViT models employing the
GradCAM [31] algorithm to visually demonstrate how ReViT excels in integrating
low-level features into its learned representations, as depicted in Fig. 1.
Finally, we established that the proposed residual learning strategy can
seamlessly integrate into multiscale transformer architectures like the
Multiscale Vision Transformer v2 (MViTv2) [20] and the Shifted-window
transformer (Swin) [24], verifying its effectiveness in various tasks,
including image classification on ImageNet1K, object detection, and instance
segmentation on COCO2017 [23], obtaining improved performances.

In summary, the major contributions of this paper are:

  1. 1.

To the best of our knowledge, a novel ViT-based architecture has been
introduced that uses residual attention modules capable of incorporating
important low-level visual characteristics into the learned representation
while maintaining the ability to extract global context, thereby enhancing
feature diversity throughout deeper network layers.

  2. 2.

We conduct an extensive experimental phase to showcase the effectiveness of
the proposed model in image classification on benchmarks like ImageNet1K,
CIFAR-10, CIFAR-100, Oxford Flowers-102, and Oxford-IIIT Pet.

  3. 3.

Empirically demonstrate that residual attention enhances the robustness of ViT
against translation invariance through a comprehensive evaluation in the
context of image classification tasks on the Oxford Flowers-102 and Oxford-
IIIT Pet datasets.

  4. 4.

Thoroughly analyze the global spread of attention maps, with and without
residual attention, using the non-locality metric.

  5. 5.

We qualitatively analyze the learned feature maps of ViT and ReViT models
using the GradCAM algorithm to examine the attention mechanism’s ability to
integrate low-level features into acquired representations in the presence of
residual attention.

  6. 6.

We evaluate our module’s seamless integration into multi-scale architectures
using MViTv2 and Swin with residual attention on ImageNet1K for image
classification and COCO2017 for object detection and instance segmentation.

The rest of this paper is organized as follows. Section 2 introduces relevant
work that inspired this study. Section 3 comprehensively describes important
background information and the proposed method. Section 4 describes the
experiments performed to validate the proposed approach quantitatively and
qualitatively. Section 5 reports an ablation study on the residual attention.
Finally, Section 6 draws some conclusions from this study.

##  2 Related Work

###  2.1 Vision Transformers

In visual recognition tasks, standard CNN-based models are currently the most
powerful algorithms for image analysis [14, 35]; even so, these networks are
designed to capture short-range dependencies using small convolutional filters
that only look at small image patches at a time. However, due to its self-
attention mechanisms, which have been very successful for natural language
processing, the transformer architecture was recently applied to process
images, capturing long-range dependencies between image patches, thus allowing
the handling of complex visual relationships better. Therefore, the ViT [7]
neural model was introduced, splitting the image into a grid of patches fed
through a stack of transformer blocks, which use self-attention to encode the
global relationships between the different patches. To enhance ViT
capabilities, Touvron et al. [33] proposed the Data-efficient Image
Transformers (DeIT) using a knowledge distillation technique that transfers
the knowledge learned by a larger teacher model to a smaller student model
(the DeiT model) during training. This helps to improve the generalization
performance of ViT. Despite the promising results, extracting low-level image
features becomes a significant challenge. To address such an issue, Han et al.
[11] introduced a new kind of neural architecture that divides image patches
into sub-patches and later employs an extra transformer block to analyze their
relationship. For the same reason, Liu et al. [24] presented the Swin
transformer, a multiscale architecture operating in local windows with a
shifted window policy to extract spatial cross-window relationships between
patches. Moreover, in literature, several works proposed enhancing local
feature representations by aggregation strategy [39] and integrating local and
global attention layers [5, 22]. Aiming to extract features at different
levels of granularity and resolution, Li et al. [20] proposed the MViTv2 to
capture local and global image features using a hierarchical architecture with
multiple scales, which can improve the model ability to define and recognize
objects within a scene in different sizes and orientations. Differently, to
improve the performance of ViT-based models for large-scale image
classification, Wang et al. [36] introduced the kNN search inside the
attention mechanism to reduce the impact of noisy or irrelevant features by
exploiting the locality of patches to extract meaningful information, thus
resulting in overall better feature representation. Finally, Yu et al. [38]
proposed the Bilateral lOcal Attention vision Transformer (BOAT) based on an
attention mechanism that integrates the common image-space local attention
with feature-space local attention. The latter can compute attention among
relevant image patches even if they are not close to each other within the
image plane. Therefore, this type of attention is a natural compensation to
image-space local attention, which may miss capturing meaningful relationships
between existing patches across different local windows. However, kNN-based
and BOAT models require significant computational resources, limiting their
practical applicability.

Due to their success in increasing CNNs capabilities, ViT and all its variants
include residual connections. However, their usual application does not
completely solve the feature collapsing of ViT in deeper layers. Therefore,
addressing such an issue, we propose residual attention learning for vision
transformers, which propagates attention among adjacent transformer blocks
while preserving the computational load, improving feature diversity and
performances in tasks like image classification, object detection, and
instance segmentation.

###  2.2 Feature Diversity

Traditional downstream ViT models are characterized by deep architectures,
resulting in reduced feature diversity as the depth increases. This limitation
hampers their ability to represent information effectively. While variants of
ViT that have similarities to CNNs, e.g., MViTv2 or Swin, are less affected by
this issue, it should be considered that they are primarily designed for
image-specific tasks, deviating from the original ViT goal of achieving
generalization across a wide range of tasks and modalities. Consequently,
addressing this challenge is essential for enhancing the representational
capabilities of standard ViT models and promoting consistency in hardware and
software designs across various tasks and modalities. To tackle this problem,
Tang et al. [32] found inspiration from residual connections, which aids in
maintaining a degree of feature diversity. They propose an augmented shortcut
scheme that enhances the flow of information from low-level layers to deeper
layers through multiple parallel alternative paths. These shortcuts are
essentially parametrized linear projection sequences that bypass the attention
mechanism within the transformer block. This augmentation significantly boosts
feature diversity as the information travels through the deeper layers of ViT.
In contrast, d’Ascoli et al. [8] introduced a novel attention layer called the
gated positional attention layer. When equipped with a soft inductive bias to
mimic the behavior of CNNs, this layer improves the feature diversity of ViT
models. However, these methods either require the addition of extra parameters
or alterations to the standard ViT workflow by changing the composition of the
transformer block. For these reasons, the proposed method offers an elegant
solution. It increases feature diversity in ViT architectures by forwarding
attention among transformer layers with minimal additional parameters and
seamlessly integrating into the original ViT workflow. Additionally, contrary
to [32], the proposed method addresses the issue of feature diversity by
contrasting attention over-globalization, which is the main catalyst in
feature collapsing rather than propagating past features.

##  3 Background and Methodology

###  3.1 Vision Transformer Layer

Figure 2: Illustration of the standard self-attention (a) and the proposed
mechanism with residual attention module (b).

The standard ViT adopts the typical architecture as proposed in [34] with MLP
and attention modules alternatively stacked throughout the network. Among
different types of attention mechanisms, scaled dot-product self-attention is
the most used one. It enables the model to extract complex relationships among
the elements in the input data sequence and to dynamically assign varying
significance degrees to each element according to the learned relationships
[34]. Formally, given a set of image patch features
$X=\\{x_{1},x_{2},x_{i},...,x_{N}\\}$, with $X\in R^{N\times C}$, where
$x_{i}$ is the vector representation of the $i$-th image patch, $N$ is the
total number of patches, and $C$ is the channel size of the image, a learnable
linear projection layer is used to generate query $Q$, key $K$, and value $V$
vectors, such that $Q,K,V\in R^{N\times d}$ with $d$ being the projection
dimensionality. Afterward, for a given attention layer $l$, the dot product
between queries and keys is computed and scaled by the square root of $d$ to
produce raw attention scores vector $S_{l}$, as shown in Fig. 2(a). Therefore,
the softmax function is applied to such scores, computing the attention
weights matrix $A_{l}$, as follows:

$S_{l}=\frac{Q_{l}K^{T}_{l}}{\sqrt{d}},$ (1) $A_{l}=\textit{Softmax}(S_{l}).$
(2)

Finally, the values in $V_{l}$ are scaled using the attention weights through
a weighted multiplication to compute the scaled dot-product attention, defined
as:

$\tilde{X}_{l}=A_{l}V_{l},$ (3)

where $\tilde{X}_{l}$ represents the output features of the scaled dot-product
attention at layer $l$.

However, due to the global nature of the self-attention mechanism, processing
a considerable number of patches or a big $d$ requires a high computational
cost. Therefore, Ashish et al. [34] proposed the Multi-Head Self-Attention
mechanism (MHSA) to retain the computational requirements when a significant
workload is required. This approach uses $H$ different attention heads
computing self-attention in different subspaces with dimension $d/H$ derived
from the Q, K, and V vectors rather than considering their full dimensionality
$d$. In this manner, the calculations are parallelized, and the model can
jointly attend to information from different representations of subspaces at
different positions. Finally, the output of each head is concatenated together
to obtain the final attention output. Formally, the MHSA operation is
represented as:

$MHSA(X_{l})=Concat([A_{l,h}V_{l,h}]^{H}_{h=1}),$ (4)

where $h$ represents the head index, $A_{l,h}$ and $V_{l,h}$ are the attention
weights matrix and the value vectors of head $h$ and layer $l$ respectively,
while $Concat(\cdot)$ denotes the concatenation of the $H$ produced feature
maps along the head dimension. Consequently, $\tilde{X}$ can now be expressed
as follows:

$\tilde{X}_{l}=Concat([\tilde{X}_{h,l}]^{H}_{h=1}),$ (5)

where $\tilde{X}_{h,l}$ represents the attention scaled values at layer $l$
and head $h$.

The MLP module, on the other hand, extracts features independently from each
patch and is usually constructed by stacking two linear layers with parameters
$W_{a}$, $W_{b}$ respectively, and a non-linear activation function $\sigma$
in between them. This module also represents the last processing operation of
the transformer block, which takes in input $\tilde{X}_{l}$ and produces the
output features of layer $l$ denoted as $X_{l+1}$ since are going to be used
as the input features for the $l+1$ layer. Formally, it can be represented as:

$MLP(\tilde{X}_{l})=\sigma(W_{a,l}\tilde{X}_{l})W_{b,l},$ (6)

where $W_{a,l}$ and $W_{b,l}$ are the weights of the two stacked linear layers
composing the MLP of layer $l$.

###  3.2 Feature Collapsing

Figure 3: Feature similarity matrix in different layers for ViT and ReViT
models. The color represents the similarity between each feature patch. A high
frequency of green color shows high similarity between patches.

Feature collapsing is a commonly observed occurrence within ViT architectures.
It refers to the phenomenon where the features extracted from different image
patches lose their distinctiveness and become increasingly similar or
indistinguishable as the network depth grows. This phenomenon primarily arises
due to the nature of the attention mechanism used in ViT, which progressively
aggregates information from various image patches as it traverses through the
network layers. Such a process can be mathematically described by representing
Eq. (5) as a weighted summation of feature patches in the following manner:

$\tilde{X}^{i}_{l,h}=\sum^{N}_{j=1}A^{i,j}_{l,h}X^{j}_{l,h},\text{subject
to}\sum^{N}_{j=1}A^{i,j}_{l,h}=1,\text{for}\ i=[1,2,...,N].$ (7)

Here, $\tilde{X}^{i}_{l,h}$ denotes the feature vector originating from patch
$i$ and represents a weighted average of features from all patches. The
weights are determined by the values in the attention map $A_{l,h}$. Although
the attention mechanism aims to capture global relationships between different
feature patches [7], it can lead to the loss of feature diversity, resulting
in feature collapsing. In order to provide evidence for this observation, Fig.
3 presents a visual representation of the feature similarity matrix for the
ViT. This matrix is constructed by calculating the cosine similarity between
distinct feature patches extracted by the ViT model. The key insight revealed
by the figure is that, as the features progress from the shallower layers of
the model to the deeper layers, there is a noticeable trend towards increased
similarity among them.

One approach to mitigate this phenomenon is the use of residual connections,
which establish connections between features across different layers while
bypassing the attention mechanism. Formally, the residual connection combined
with MHSA operation can be expressed as follows:

$\text{residualMHSA}(X_{l})=\text{MHSA}(X_{l})+X_{l}.$ (8)

In this equation, the identity projection $X_{l}$ runs in parallel to the MHSA
operation. Intuitively, because the features in $X_{l}$ exhibit greater
diversity before passing through the MHSA operation, the summation of the two
feature vectors preserves the characteristics from the previous layer.
Consequently, the output retains more diverse features. Empirical evidence
supporting this phenomenon can be found in [32]. Drawing inspiration from
these discoveries, this study introduces a novel approach to tackling this
issue. Specifically, it suggests an innovative alternative residual connection
between attention layers (without bypassing the attention mechanism) designed
to curb the rapid expansion of the attention across the head dimension, which
is the main source of feature collapsing, and increase feature diversity as
shown in Fig. 3.

###  3.3 Residual Attention

Figure 4: Illustration of the proposed ReViT network with residual attention.
In (a) the overall architecture, in (b) the transformer block in detail,
including the residual attention module.

Generally, ViT model customization involves modifying the MHSA mechanism,
normalization layers, MLP component, or residual connections among features,
maintaining a specific information flow at the feature level between adjacent
transformer blocks [7, 24]. Aiming to expand such a flow to improve the
feature diversity, the ReViT architecture shown in Fig. 4(a) incorporates a
novel skip connection between consecutive MHSA layers, enabling the
propagation and accumulation of attention from shallow to deeper layers, as
depicted in Fig. 4(b). This expansion, representing the proposed residual
attention, is a complementary process to residual connections at the end of
each transformer block. Indeed, while existing skip connections propagate the
low-level features to deeper layers bypassing the MHSA layer, the residual
attention propagates the information from $Q$ and $K$, which defines the
relationships between patches used to extract these features, thus enabling a
new attention capability to consider previously extracted relationships while
learning to extract new ones. Formally, the proposed attention mechanism is
implemented by changing the computation of the $S_{l}$ matrix defined in Eq.
(1). Specifically, the equation is extended with an additional term
representing the $Q$ and $K$ of the previous MHSA layers aggregated with those
of the current layer, as follows:

$S_{l}=\begin{cases}\frac{Q_{0}K_{0}^{T}}{\sqrt{d_{0}}},&\text{if}\ l=0\\\
\gamma(\frac{Q_{l}K_{l}^{T}}{\sqrt{d_{l}}},S_{l-1}),&\text{otherwise}\end{cases},$
(9)

where $l$ indicates the $l$-th attention layer and $\gamma$ is the permutation
invariant aggregation function. For $l=0$, the attention is computed as
defined in Eq. (1). Otherwise, for $l>0$, $S_{l}$ is obtained as the $\gamma$
of $S_{l-1}$ and the scaled dot product of $Q_{l}$ and $K_{l}$. Following this
rationale, the information flow among adjacent MHSA layers is extended beyond
the standard flow, which is limited only to feature forwarding, propagating
the attention used to aggregate information between patches in previous
layers. However, propagating and accumulating such information may prevent the
network from learning high-level representations while amplifying the
attention related to low-level features. To avoid this, $\gamma$ is
implemented as a weighted sum where a learnable gate variable $\alpha$ is
introduced to allow the network to autonomously determine how much attention
to propagate between layers. Precisely, $\alpha$ balances the quantity of
residual attention transmitted from the previous to the current MHSA layer
during aggregation. Formally, the Eq. (9) is further extended as follows:

$S_{l}=\begin{cases}\frac{Q_{l}\cdot K_{l}^{T}}{\sqrt{d_{l}}},&\text{if}\
l=0\\\ \alpha(\frac{Q_{l}\cdot
K_{l}^{T}}{\sqrt{d_{l}}})+(1-\alpha)(S_{l-1}),&\text{otherwise}\end{cases},$
(10)

where $\alpha\in]0;1[$. After obtaining the balanced attention scores $S_{l}$,
following Eq. (2), the softmax function is applied to compute the attention
weights $A_{l}$ required for obtaining the new scaled dot-product attention
output, as depicted in Fig. 2(b). Given the implementation of residual
attention, it can be easily included in both existing single-scale or multi-
scale vision transformer architectures, maintaining a similar computational
cost.

###  3.4 Attention Globality

Attention globality is a concept that reflects the capacity of the attention
mechanism to encompass and consider information from all patches across an
image rather than being restricted to local or nearby elements. This
capability becomes particularly pronounced in the deeper layers of a ViT
because these layers can extract high-level representations that account for
long-range relationships between patches. To quantify the globality of the
attention mechanism across both heads and layers, researchers in [8] introduce
a metric known as the non-locality metric. This metric calculates, for each
query patch $i$, the relative positional distances to all key patches $j$
weighted by their attention scores $A^{i,j}_{l,h}$. The resulting sum is then
averaged by the number of patches to derive the non-locality metric for a
specific head $h$. This can further be averaged across attention heads to
obtain the non-locality metric for the entire layer. Mathematically, this
metric is defined as follows:

$D_{l,h}=\frac{1}{N}\sum_{i,j}A^{i,j}_{l,h}||\delta^{i,j}||,\\\
D_{l}=\frac{1}{H}\sum_{h}D_{l,h}.$ (11)

Here, $||\delta^{i,j}||$ represents the relative positional distance between
query patch $i$ and key patch $j$, $D_{l,h}$ denotes the non-locality metric
within a given layer $l$ and head $h$, while $D_{l}$ represents $D_{l,h}$
averaged along the head dimension. Moreover, since $||\delta^{i,j}||$ remains
constant throughout the network (i.e., the relative positions of patches do
not change), the value of $D_{l,h}$ relies entirely on $A_{l,h}$. Furthermore,
as the non-locality metric $D$ increases in deeper layers (indicating a shift
towards global attention), it suggests that the attention matrix $A_{l,h}$
(consequently $A_{l}$) in these layers generally has a greater capacity to
capture global relationships across the different attention heads compared to
attention matrix from the previous layer. This property of the attention
mechanism can be harnessed to demonstrate the advantages of employing residual
attention in slowing down the process of globalizing $A_{l}$. Taking into
account Eq. (10), which defines the calculation of $A_{l}$ with a residual
connection, it considers attention scores not only from the current layer $l$
but also from previous layers. Consequently, the globality of layer $l$ in
ReViT, at a given head, can be represented as:

$D_{l,h}=\frac{1}{N}\sum_{i,j}[\alpha(A^{i,j}_{l,h})+(1-\alpha)(A^{i,j}_{l-1,h})]||\delta^{i,j}||.$
(12)

From a theoretical point of view, thanks to the above equation, given that the
attention scores from earlier layers ($A_{l-1}$) generally tend to be more
locally focused compared to those of the current layer $l$, the weighted
combination of these sets of scores is expected to result in an attention
matrix that exhibits less globality than if only the scores from the current
layer were used. In other words, combining attention scores of given layer $l$
with those of previous layers through residual attention at each head $h$,
slows down the globalization process. For empirical proof, refer to section
4.4.

##  4 Experiments

In this section, we present the implementation details of the proposed method
and tests conducted on some challenging datasets for image classification,
object detection, and instance segmentation. In particular, we start by giving
the implementation details of our method in various settings. Subsequently, we
report the test results on five image classification benchmarks, namely
ImageNet1K, CIFAR-10, CIFAR-100, Oxford Flowers-102, and Oxford-IIIT Pet. To
proceed, the test results under position and scale changes (i.e., translation
invariance) on Oxford Flowers-102 and Oxford-IIIT Pet datasets are reported.
Such datasets were chosen for translation invariance testing due to their
higher quality, which helps preserve important information even after applying
translation transformations like scale reduction. Moreover, we investigate the
attention mechanism of the networks trained on ImageNet1K, both quantitatively
using the non-locality metric defined in Eq. (11-12) and qualitatively
comparing the learned feature maps extracted with the GradCAM algorithm
between the proposed approach and the standard ViT. Furthermore, we report the
test results that show the effectiveness of our method when implemented on top
of spatial-aware multiscale architectures like Swin and MViTv2 on downstream
tasks like object detection and instance segmentation on the COCO2017 dataset.

###  4.1 Implementation Details

The proposed ReViT method is implemented as an augmented version of the
standard ViT architecture. Specifically, we integrate the residual attention
module on top of the standard ViT implementation using the PyToch framework.
Regarding the network version used in this study, we only relied on the base
version of ViT containing 12 layers and denominated as ViT-B. To this end, we
name our network ReViT-B. Additionally, as mentioned above, we conduct
experiments that test the effectiveness of our method applied in the
multiscale architectures, namely MViTv2 and Swin. To implement such networks,
we start from their official PyTorch implementations and extend them with our
module. Moreover, for MViTv2 and Swin, we exploit their tiny versions, i.e.,
MViTv2-T and Swin-T with 12 layers each, and name our implementations
ReMViTv2-T and ReSwin-T. The rationale behind using the smaller versions of
each network is that they require less computational resources, and usually,
the bigger versions require pre-training on large-scale datasets like
ImageNet21K or even bigger ones.

To train our networks, we use the same configurations reported in their
official studies and run the experiments on an NVIDIA V100 GPU with 32GB of
RAM. Specifically, for image classification on ImageNet1K, all models were
trained for $300$ epochs, with an input image size of $224\times 224$, using
gradient clipping and cosine scheduling learning rate warmup/decay. For the
ReViT-B network training, Adam [17] was used as optimizer with a base learning
rate set to $0.001$, $30$ warmup epochs, an effective batch size of $4096$,
and the weight decay set to $0.3$. Instead, for ReSwin-T model training, AdamW
optimizer [26] was used by setting a base learning rate to $0.001$, $20$
warmup epochs, an effective batch size of $1024$, and the weight decay set to
$0.05$. Finally, the remaining ReMViTv2-T network training was performed using
the AdamW optimizer with a base learning rate set to $0.002$, $70$ warmup
epochs, an effective batch size of $2048$, and the weight decay set to $0.1$.
Regarding other image classification tasks, we use the same training setups
reported in [7] for a fair comparison. Moreover, regarding the object
detection and instance segmentation tasks, since such tasks require local
awareness of features, the experiments were performed only using ReSwin-T and
ReMViTv2-T. These models were trained for $36$ epochs using AdamW as an
optimizer and linear scheduling learning rate decay. For ReSwin-T network
training, the initial learning rate was set to $0.0001$, with a weight decay
of $0.05$, and a batch size set to $16$. Instead, for ReMViTv2-T model
training, the initial learning rate was set to $0.00016$, with a weight decay
of $0.1$, and a batch size set to $64$.

###  4.2 Image classification

For the image classification tasks, models were primarily evaluated on
ImageNet1K, considered as one of the most significant benchmarks. To evaluate
the performance of the models, we used the top-1 single crop accuracy metric
and reported the obtained results in Table 1. As can be noticed, all models
incorporating the residual attention module outperform their original network
counterparts in the used metric. Specifically, ReViT-B, ReMViTv2-T, and
ReSwin-T models with residual attention obtain an improvement of 4.6%, 0.4%,
and 0.2%, respectively, compared to their original counterparts. Notably,
ReViT-B shows a significant improvement compared to ViT-B, indicating that
residual attention is more effective in single-scale architectures that employ
only global attention. This is because such a module was invented to contrast
the over-globalization of the attention mechanism, a phenomenon manifested in
the ViT-B architecture. On the contrary, multiscale architectures like Swin-T
and MViTv2-T have their own in-built local mechanisms that help preserve the
low-level features throughout the network. As such, the effect of the residual
attention in these networks, i.e., ReSwin-T and ReMViTv2-T, is lower but still
bolsters their representational ability and achieves improved performances
compared to the original counterparts. Apart from the performance
enhancements, another compelling factor highlighting the significance of
residual attention within single-scale architectures is the parameter
$\alpha$. Notably, in ReViT-B, $\alpha$ assumes a considerably smaller value
when compared with the values observed in ReSwin-T and ReMViTv2-T. As Eq. (10)
indicates, this discrepancy signifies that prior attention holds substantially
more significance in the context of ReViT. Additionally, Table 2 provides a
more extensive comparison with the state of the art, showing that models with
residual attention perform even more comparatively to the best models.

Subsequently, ReViT-B was tested on multiple image classification benchmarks,
including CIFAR-10, CIFAR-100, Oxford Flowers-102, and Oxford-IIIT Pet. The
performance of ReViT-B is then compared with that of the ViT-B model, and the
results are presented in Table 3. Notably, ReViT-B, with the residual
attention, consistently outperforms ViT-B across all datasets with a minimum
improvement of 0.6% in CIFAR10 and a maximum of 3.4% in CIFAR100. This
improvement can be attributed to the residual attention module’s ability to
capture a wider range of visual features, thus enhancing the network’s
capability to distinguish between different classes based on the image
content. The other models were not tested on these datasets because their
official studies do not provide experimental results and focus only on large-
scale datasets.

Model | Image size | Params(M) | FLOPs(G) | top-1 acc. % | $\alpha$
---|---|---|---|---|---
ViT-B [7] | $224\times 224$ | 86 | 17.5 | 77.8 | -
ReViT-B | $224\times 224$ | 86 | 17.5 | 82.4 | 0.56
Swin-T [24] | $224\times 224$ | 29 | 4.5 | 81.3 | -
ReSwin-T | $224\times 224$ | 29 | 4.5 | 81.5 | 0.99
MViTv2-T [20] | $224\times 224$ | 24 | 4.7 | 82.3 | -
ReMViTv2-T | $224\times 224$ | 24 | 4.7 | 82.7 | 0.99
Table 1: Performance of ViT-based state-of-the-art models with and without residual attention on ImageNet1k single-crop top-1 accuracy. In addition, image size, model parameters in millions, i.e., Params(M), models throughput measured in theoretical floating point operations per second, i.e., FLOPs(G) [7], and the final value of $\alpha$ are reported. Model | Params(M) | FLOPs(G) | top-1 acc. %
---|---|---|---
ViT-B [7] | 86 | 17.5 | 77.8
Deit-B [33] | 86 | 17.5 | 81.8
ConViT-B [8] | 86 | 17.5 | 82.4
AugViT-B [32] | 86.5 | 17.5 | 82.4
ReViT-B | 86 | 17.5 | 82.4
Swin-T [24] | 29 | 4.5 | 81.3
TNT-S [12] | 24 | 5.2 | 81.5
CoAtNet-0 [6] | 25 | 4.0 | 81.6
T2T-ViT-14 [40] | 22 | 4.8 | 81.5
T2T-ViT-14t [40] | 22 | 6.1 | 81.7
ConvNeXt-T [25] | 29 | 4.5 | 82.1
MViTv2-T [20] | 24 | 4.7 | 82.3
VOLO-D1 [41] | 27 | 7.1 | 84.2
ReSwin-T | 29 | 4.5 | 81.5
ReMViTv2-T | 24 | 4.7 | 82.7
Table 2: Comparison with state-of-the-art models on ImageNet1K using input
images of size $224\times 224$. In addition, information about model
parameters in millions and throughput measured in floating point operations
per second are reported.

Model | CIFAR-10 | CIFAR-100 | Oxford-Flowers 102 | Oxford-IIIT Pet
---|---|---|---|---
ViT-B [7] | 98.1 | 87.1 | 89.5 | 93.8
ReViT-B | 98.7 | 90.5 | 91.1 | 94.9

Table 3: Top-1 accuracy performance comparison of ViT-B and ReViT-B on
CIFAR-10, CIFAR-100, Oxford-Flowers 102, and Oxford-IIIT Pet datasets.

###  4.3 Translation Invariance

(a)  (b)  (c)

Figure 5: Visualization of transformations showcasing translation invariance,
specifically horizontal shifting (a), vertical shifting (b), and changes in
scale (c).

In this study, the datasets used for visual recognition typically feature the
object of interest centered within the image. This is in stark contrast to
real-world scenarios where objects can be situated anywhere within the scene.
To assess ReViT robustness, two experiments were devised to simulate varying
scales and positions of the target object within the images as shown in Fig.
5. The performance of ReViT-B and ViT-B on the Oxford Flowers-102 and Oxford-
IIIT Pet datasets are compared in these experiments.

The first experiment evaluates the models’ ability to handle horizontal and
vertical translations. To achieve this, the input images are horizontally and
vertically shifted on the image plane by {15, 30, 45, 60}% of pixels, with the
empty areas filled with zeros to maintain the same input image size as shown
in Fig. 5(a) and (b). As indicated in Table 4 and Fig. 6(a), both models
exhibit a similar and impressive level of horizontal translation invariance,
with only a minor reduction in performance. Overall, ReViT-B demonstrates
slightly superior horizontal translation invariance, especially at the 60%
translation level. For the Oxford Flowers-102 dataset, ReViT-B performance
decreases by 2.9% less than ViT-B at a 60% translation, while for the Oxford-
IIIT Pet dataset, the decrease is 1.0% less. Similar trends are observed for
vertical translation invariance, as illustrated in Table 5 and Fig. 6(b).
Although both models show a comparable drop in performance, ReViT-B displays
better overall vertical translation invariance, particularly on the Oxford
Flowers-102 dataset, where its performance decreases by 3.6% less than ViT-B.

In the second experiment, the scale invariance of both models is evaluated,
and the results are presented in Table 6 and Fig. 6(c). To generate images at
different scales, the original images are resized by reducing their width and
height by {15, 30, 45, 60}%, with zero-padding to maintain the correct input
size as illustrated in Fig. 5(c). Unlike the first experiment, ReViT-B
consistently outperforms ViT-B in terms of scale invariance. In the case of
the Oxford Flowers-102 dataset, ReViT-B shows up to 19.5% less reduction in
performance accuracy, while for the Oxford-IIIT Pet dataset, it exhibits up to
4.7% less decrease than ViT-B. These results highlight that ReViT-B possesses
significantly better scale invariance compared to ViT-B.

(a)  (b)  (c)

Figure 6: Translation invariance test. The graphics depict how ViT-B and
ReViT-B perform when subjected to variations in horizontal shifting (a),
vertical shifting (b), and changes in size (c) on the Oxford Flowers-102 and
Oxford-IIT Pet datasets. Horizontal Translation
---
| Oxford Flowers-102 | Oxford-IIIT Pet
Model | 15% | 30% | 45% | 60% | 15% | 30% | 45% | 60%
ViT-B | 0.9 | 1.1 | 3.5 | 10.6 | 0.4 | 1.1 | 3.6 | 11.1
ReVit-B | 0.8 | 1.2 | 2.5 | 7.7 | 0.4 | 0.7 | 3.2 | 10.1
$\Delta$ | 0.1 | -0.1 | 1.0 | 2.9 | 0.0 | 0.4 | 0.4 | 1.0
Table 4: Reduction in performance (in %) observed when applying a horizontal
translation to input images in the Oxford Flowers-102 and Oxford-IIIT Pet
datasets. $\Delta$ represents the variance in the extent of performance
degradation between ReViT-B and ViT-B. Vertical Translation
---
| Oxford Flowers-102 | Oxford-IIIT Pet
Model | 15% | 30% | 45% | 60% | 15% | 30% | 45% | 60%
ViT-B | 0.4 | 0.9 | 4.1 | 12.1 | 0.6 | 1.1 | 3.9 | 11.8
ReVit-B | 0.4 | 1.4 | 3.0 | 8.5 | 0.4 | 1.1 | 3.8 | 10.2
$\Delta$ | 0.0 | -0.5 | 1.1 | 3.6 | 0.2 | 0.0 | 0.1 | 1.6
Table 5: Diminished performance (in %) resulting from a vertical translation
applied to input images within the Oxford Flowers-102 and Oxford-IIIT Pet
datasets. The symbol $\Delta$ indicates the difference in performance
reduction between ReViT-B and ViT-B. Scale Variance
---
| Oxford Flowers-102 | Oxford-IIIT Pet
Model | 15% | 30% | 45% | 60% | 15% | 30% | 45% | 60%
ViT-B | 2.8 | 6.8 | 24.6 | 58.5 | 0.4 | 1.5 | 7.0 | 25.9
ReVit-B | 1.5 | 4.3 | 13.9 | 39.0 | 0.4 | 1.4 | 6.0 | 21.2
$\Delta$ | 1.3 | 2.5 | 9.7 | 19.5 | 0.0 | 0.1 | 1.0 | 4.7
Table 6: Diminished performance (in %) resulting from variations in scale
applied to input images within the Oxford Flowers-102 and Oxford-IIIT Pet
datasets. $\Delta$ indicates the disparity in performance decline between
ReViT-B and ViT-B.

###  4.4 Residual Attention and Globality

(a)  (b)  (c)

Figure 7: The non-locality metric is depicted for individual heads across all
layers in the case of ViT-B in (a) and ReViT-B in (b). This visualization
illustrates how the receptive field of each head expands as it progresses
through deeper layers. On the other hand, in (c), a comparison of the non-
locality metric between ViT-B and ReViT-B for each layer is illustrated.

The investigation into the global reach of MHSA within vision transformers,
both with and without residual attention, aims to examine the hypothesis that
residual attention decelerates the globalization of the attention mechanism.
To assess this hypothesis quantitatively, we employ the non-locality metric
calculated according to Eq. (11). Additionally, the metric is computed at both
the head and layer levels. The results of this analysis, depicted in Fig. 7,
display the average metric across 256 input images from the ImageNet1K
validation set for ViT-B and ReViT-B models trained on the same dataset. The
x-axis on the plots corresponds to the layer index, while the mean non-
locality for each head or layer is represented on the y-axis. Fig. 7(a)
illustrates that in ViT-B, the non-locality of each head increases as it
delves deeper into the layers, approaching globality which corresponds to the
findings in [7, 8]. A similar trend is observed in ReViT-B, as seen in Fig.
7(b), albeit with some noteworthy distinctions. Notably, ViT-B exhibits a more
consistent and monotonic increase in non-locality, with minimal variations
between different heads, particularly after layer 5. In contrast, ReViT-B
displays more diverse differences between heads, with some heads finding
difficulty in escaping the locality while others become completely global,
resulting in a less monotonic increase in non-locality. In summary, the
results substantiate the hypothesis that the attention receptive field of
heads in vision transformers becomes global in deeper layers, a phenomenon
observed in both ViT-B and ReViT-B models. However, these findings also reveal
that ReViT-B retains the ability to extract global relationships in deeper
layers while still maintaining some heads that remain localized with
relatively low non-locality values. Simultaneously, ViT-B is confined solely
to global relationships, leading to feature collapse. To provide a broader
perspective, Fig. 7(c) presents the non-locality of ViT-B and ReViT-B at the
layer level, represented as the average non-locality metric across all heads
within each layer. It is evident that when computed using the same 256 input
examples, ReViT-B consistently maintains lower non-locality values (except the
first), indicating its ability to incorporate low-level features, with the
help of relatively local heads, into the representations learned from heads
that exhibit non-locality escape (i.e., become global).

Figure 8: Comparison of feature maps in the last two layers of ViT-B and
ReViT-B extracted using GradCAM algorithm from the illustrated input images.

###  4.5 Qualitative Comparisons

The effectiveness of the proposed residual attention module is further
demonstrated through a qualitative evaluation of the features learned by ViT-B
and ReViT-B using the MHSA mechanism, both with and without residual
attention, respectively. To enable such a comparison, we employ the GradCAM
algorithm and apply it to the last two MHSA layers of ViT-B and ReViT-B
trained on ImageNet1k with samples from the validation set of the same
dataset. It’s worth emphasizing that layers 11 and 12 were selected due to
their high globality, aligning with the objective of this experiment, which is
to highlight how ReViT-B incorporates low-level features into its learned
representations compared to ViT-B. To proceed, the feature maps of both models
and the input images used to obtain such features are illustrated in Fig. 8.
As can be noticed, the feature maps extracted from ViT-B are poor in details
and lack the presence of low-level features like shapes and edges. This is
caused by the globalization of its attention mechanism, which provokes the
feature collapsing phenomenon. This also proves the observation made by
authors in [8], which states that ViT-based architectures relying on the
standard MHSA often exhibit limited attention variability, resulting in
reduced feature diversity or even feature collapse [32]. On the contrary,
ReViT-B, equipped with residual attention, displays greater feature diversity
with feature maps almost completely aligned with the entire region where the
object of interest is located, preserving the same shape. This phenomenon
suggests that ReViT-B can incorporate low-level information like shapes and
edges with the global context of the scene. Additionally, such ability can be
attributed to residual attention. Thanks to this mechanism, the network can
propagate attention maps from shallow layers to deeper layers, slowing down
the globalization of attention, improving feature diversity, and incorporating
low-level characteristics into the global context. Notably, this augmented
feature diversity aligns with ReViT’s improved performance in terms of image
classification accuracy compared to the standard ViT architecture.

###  4.6 Object Detection and Instance Segmentation

To assess the general applicability of the residual attention module, we test
its knowledge transfer ability in downstream tasks. As such, further
experiments were conducted on spatial-aware networks, i.e., ReSwin-T and
ReMViTv2-T, for the tasks of object detection and instance segmentation,
achieving increased results. The evaluation was performed on the COCO2017
dataset for both tasks using the COCO average precision (AP) metric, and the
input images were resized with $480\leq H\leq 800$ and $W=1333$. For fair
comparisons with the solutions proposed in [24] and [20], the Cascade Mask-
RCNN [3] and the Mask R-CNN [13] models were used with the ReSwin-T and
ReMViTv2-T as backbones, respectively. Additionally, both ReSwin-T and
ReMViTv2-T were initialized with the weights learned during ImageNet1K
training, hence, knowledge transferring. Table 7 summarizes results on the
subset of 5000 validation images. For the object detection task, the ReSwin-T
model performance is higher for $AP_{box}$ and $AP_{box}^{50}$ with respect to
the original Swin-T architecture. Regarding the ReMViTv2-T model, the
performance improves for all the considered APs compared to the original
MViTv2-T architecture. For the instance segmentation task, ReSwin-T and
ReMViTv2-T with residual attention increase the overall performance in all the
metrics compared to their original works, even though by a small margin.

Framework | Backbone | $AP_{box}$ | $AP_{box}^{50}$ | $AP_{box}^{75}$ | $AP_{mask}$ | $AP_{mask}^{50}$ | $AP_{mask}^{75}$ | #Params
---|---|---|---|---|---|---|---|---
Cascade Mask | Swin-T [24] | 50.5 | 69.3 | 54.9 | 43.7 | 66.6 | 47.1 | 86M
R-CNN | ReSwin-T | 50.6 | 69.7 | 54.9 | 43.9 | 66.8 | 47.6 | 86M
Mask | MViTv2-T [20] | 48.2 | 70.9 | 53.3 | 43.8 | 67.9 | 47.2 | 44M
R-CNN | ReMViTv2-T | 48.5 | 71.0 | 53.5 | 44.0 | 68.3 | 47.3 | 44M
Table 7: Performance of the residual attention module implemented on top of
Swin and MViTv2 architectures for object detection and instance segmentation
on COCO2017 $5K$ validation set.

##  5 Ablation Study

In this section, we perform an ablation study to determine the significance of
balancing the past and current attention information in the proposed residual
attention. We will specifically examine the impact of $\alpha$ on various
image classification benchmarks including CIFAR10, CIFAR100, Oxford
Flowers-102, and Oxford-IIIT Pet.

###  5.1 Inspecting the effect of $\alpha$

Figure 9: ReViT classification performance on CIFAR10, CIFAR100, Oxford
Flowers-102 and Oxford-IIIT Pet datasets for different values of $\alpha$.

The notion of residual attention acts as a conduit for propagating attention
information through diverse layers via a shortcut connection, formally
represented in Eq. (10). Introducing a trainable gate variable, denoted as
$\alpha$, ensures a balanced incorporation of residual attention information
within this connection. The pivotal role of $\alpha$ lies in its ability to
regulate past and present information flow, maintaining a harmonious
equilibrium. A meticulous series of experiments are devised to study this
variable’s impact empirically.

These experiments encompass an assessment of ReViT’s performance in
classification tasks, with the gating variable held constant while manually
adjusting its values to observe the model’s behavior. Specifically, ReViT is
tested across four classification benchmarks: CIFAR10, CIFAR100, Oxford
Flowers-102, and Oxford-IIIT Pet, with $\alpha$ values spanning from 0 to 1.
With 0 being the lower extremity considering only past attention and 1 being
the other extremity completely ignoring past attention (i.e., ViT). The
outcomes of these experiments are graphically presented in Fig. 9. For
$\alpha=0$, according to Eq. (10), ReViT relies only on the attention
information from the first layer, and as expected, the performance decreases,
especially on complex datasets like CIFAR100 and Oxford Flowers-102. Such
performance collapse is explained by the fact that for an $\alpha=0$, ReViT
lacks global context. For the other values of $\alpha$, notably, in datasets
characterized by a modest number of classes and a surplus of examples per
class, such as CIFAR10, the impact is relatively subdued. However, in a more
demanding dataset featuring significant inter-class and intra-class variations
compared to CIFAR10, like Oxford-IIIT Pet, the role of $\alpha$ becomes more
pivotal, exemplified by the model’s sensitivity to varying $\alpha$ values
affecting its performance. Furthermore, in datasets like CIFAR100 and Oxford
Flowers-102, which are even more complex, the effect of $\alpha$ on ReViT
becomes even more important and manifests quite similarly with the model
attaining peak performance at $\alpha=0.75$, and experiencing a sharp decline
as it approaches 1. Considering that an $\alpha$ value close to 1 typically
does not yield the optimal performance across all datasets, it becomes evident
that past attention information is crucial for learning better representation,
and $\alpha$ assumes a vital role within ReViT. It balances the integration of
past and current attention information, facilitating the extraction of high-
quality image representations encompassing high and low-level features.

##  6 Conclusion

In this study, we introduce the innovative Residual Attention Vision
Transformer (ReViT) network, which integrates residual attention learning to
enhance the extraction of visual features in Vision Transformer (ViT)
architectures. The proposed approach effectively transmits and accumulates
attention information from queries and keys across successive Multi-Head Self-
Attention (MHSA) layers. Such residual connection prevents diminishing low-
level visual features. Additionally, it empowers the model to leverage
previously extracted features while learning new ones by decelerating the
over-globalization of the attention mechanism. The efficacy and robustness of
the proposed methods are evaluated on five popular image classification
benchmarks, including ImageNet1K, CIFAR-10, CIFAR-100, Oxford Flowers-102, and
Oxford-IIIT Pet, as well as the COCO2017 dataset for object detection and
instance segmentation. The results demonstrate that residual attention
learning elevates the performance of ViT-based neural models in visual
recognition tasks. Furthermore, it adeptly preserves and enhances the ability
to uncover and incorporate semantic and spatial relationships when integrated
into spatial-aware neural networks, facilitating the identification,
localization, and segmentation of individual objects within an image. However,
since the introduced residual module contrasts the globalization of the
attention mechanism, its efficacy is limited when applied in multi-scale
architectures with local awareness, providing less improvement compared to its
application in single-scale architecture with global attention. Motivated by
such limitations, in future works, refining the residual attention module to
better suit multi-scale transformer architectures could provide further
insights into its effectiveness in such architectures.

## Acknowledgements

This work was supported by "Smart unmannEd AeRial vehiCles for Human likE
monitoRing (SEARCHER)" project of the Italian Ministry of Defence within the
PNRM 2020 Program (Grant Number: PNRM a2020.231); "A Brain Computer Interface
(BCI) based System for Transferring Human Emotions inside Unmanned Aerial
Vehicles (UAVs)" Sapienza University Research Projects (Grant Number:
RM1221816C1CF63B); "EYE-FI.AI: going bEYond computEr vision paradigm using wi-
FI signals in AI systems" project of the Italian Ministry of Universities and
Research (MUR) within the PRIN 2022 Program (Grant Number: 2022AL45R2) (CUP:
B53D23012950001); and MICS (Made in Italy – Circular and Sustainable) Extended
Partnership and received funding from Next-Generation EU (Italian PNRR – M4
C2, Invest 1.3 – D.D. 1551.11-10-2022, PE00000004). CUP MICS B53C22004130001.

## References

  * Alam et al. [2022] Alam, M.M., Islam, M.T., Rahman, S.M., 2022.  Unified learning approach for egocentric hand gesture recognition and fingertip detection.  Pattern Recognition 121, 108200.  doi:https://doi.org/10.1016/j.patcog.2021.108200.
  * Avola et al. [2019] Avola, D., Bernardi, M., Cascio, M., Cinque, L., Foresti, G.L., Massaroni, C., 2019.  A new descriptor for keypoint-based background modeling, in: International Conference on Image Analysis and Processing (ICIAP), pp. 15–25.  doi:10.1007/978-3-030-30642-7_2.
  * Cai and Vasconcelos [2018] Cai, Z., Vasconcelos, N., 2018.  Cascade r-cnn: Delving into high quality object detection, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6154–6162.  doi:10.1109/CVPR.2018.00644.
  * Chen et al. [2022] Chen, L., Zhou, F., Wang, S., Dong, J., Li, N., Ma, H., Wang, X., Zhou, H., 2022.  Swipenet: Object detection in noisy underwater scenes.  Pattern Recognition 132, 108926.  doi:https://doi.org/10.1016/j.patcog.2022.108926.
  * Chu et al. [2021] Chu, X., Tian, Z., Wang, Y., Zhang, B., Ren, H., Wei, X., Xia, H., Shen, C., 2021.  Twins: Revisiting the design of spatial attention in vision transformers, in: Advances in Neural Information Processing Systems (NeurIPS), pp. 9355–9366.
  * Dai et al. [2021] Dai, Z., Liu, H., Le, Q.V., Tan, M., 2021.  Coatnet: Marrying convolution and attention for all data sizes.  Advances in neural information processing systems 34, 3965–3977.
  * Dosovitskiy et al. [2021] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N., 2021.  An image is worth 16x16 words: Transformers for image recognition at scale, in: International Conference on Learning Representations (ICLR), pp. 1–21.
  * d’Ascoli et al. [2021] d’Ascoli, S., Touvron, H., Leavitt, M.L., Morcos, A.S., Biroli, G., Sagun, L., 2021.  Convit: Improving vision transformers with soft convolutional inductive biases, in: International Conference on Machine Learning (ICML), pp. 2286–2296.
  * Gao et al. [2023] Gao, L., Chen, L., Liu, P., Jiang, Y., Li, Y., Ning, J., 2023.  Transformer-based visual object tracking via fine-coarse concatenated attention and cross concatenated mlp.  Pattern Recognition , 109964doi:https://doi.org/10.1016/j.patcog.2023.109964.
  * González-Díaz et al. [2016] González-Díaz, I., Buso, V., Benois-Pineau, J., 2016.  Perceptual modeling in the problem of active object recognition in visual scenes.  Pattern Recognition 56, 129–141.  doi:https://doi.org/10.1016/j.patcog.2016.03.007.
  * Han et al. [2021a] Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., Wang, Y., 2021a.  Transformer in transformer, in: Advances in Neural Information Processing Systems (NeurIPS), pp. 15908–15919.
  * Han et al. [2021b] Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., Wang, Y., 2021b.  Transformer in transformer.  Advances in Neural Information Processing Systems 34, 15908–15919.
  * He et al. [2017] He, K., Gkioxari, G., Dollár, P., Girshick, R., 2017.  Mask r-cnn, in: IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988.  doi:10.1109/ICCV.2017.322.
  * Hong et al. [2022] Hong, Y., Pan, H., Jia, Y., Sun, W., Gao, H., 2022.  Resdnet: Efficient dense multi-scale representations with residual learning for high-level vision tasks.  IEEE Transactions on Neural Networks and Learning Systems (TNLS) , 1–12doi:10.1109/TNNLS.2022.3169779.
  * Huff et al. [2022] Huff, T., Mahabadi, N., Tadi, P., 2022.  Neuroanatomy, visual cortex, in: StatPearls [Internet]. StatPearls Publishing, pp. –.
  * Kim et al. [2023] Kim, B.J., Choi, H., Jang, H., Lee, D.G., Jeong, W., Kim, S.W., 2023.  Improved robustness of vision transformers via prelayernorm in patch embedding.  Pattern Recognition 141, 109659.  doi:https://doi.org/10.1016/j.patcog.2023.109659.
  * Kingma and Ba [2015] Kingma, D.P., Ba, J., 2015.  Adam: A method for stochastic optimization., in: International Conference on Learning Representations (ICLR), pp. 1–13.
  * Krizhevsky and Hinton [2009] Krizhevsky, A., Hinton, G., 2009.  Learning multiple layers of features from tiny images.  Technical Report .
  * Laith et al. [2021] Laith, A., Jinglan, Z., Amjad J., H., Ayad, A.D., Ye, D., Omran, Al-Shamma J., S., Mohammed A., F., Muthana, A.A., Laith, F., 2021.  Review of deep learning: concepts, cnn architectures, challenges, applications, future directions.  Journal of Big Data 8, 1–74.  doi:10.1186/s40537-021-00444-8.
  * Li et al. [2022a] Li, Y., Wu, C.Y., Fan, H., Mangalam, K., Xiong, B., Malik, J., Feichtenhofer, C., 2022a.  Mvitv2: Improved multiscale vision transformers for classification and detection, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4794–4804.  doi:10.1109/CVPR52688.2022.00476.
  * Li et al. [2022b] Li, Z., Liu, F., Yang, W., Peng, S., Zhou, J., 2022b.  A survey of convolutional neural networks: Analysis, applications, and prospects.  IEEE Transactions on Neural Networks and Learning Systems (TNLS) 33, 6999–7019.  doi:10.1109/TNNLS.2021.3084827.
  * Lin et al. [2022] Lin, H., Cheng, X., Wu, X., Shen, D., 2022.  Cat: Cross attention in vision transformer, in: IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6.  doi:10.1109/ICME52920.2022.9859720.
  * Lin et al. [2014] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., Zitnick, C.L., 2014.  Microsoft coco: Common objects in context, in: European Conference in Computer Vision (ECCV), pp. 740–755.  doi:https://doi.org/10.1007/978-3-319-10602-1_48.
  * Liu et al. [2021] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B., 2021.  Swin transformer: Hierarchical vision transformer using shifted windows, in: IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9992–10002.  doi:10.1109/ICCV48922.2021.00986.
  * Liu et al. [2022] Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S., 2022.  A convnet for the 2020s, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976–11986.
  * Loshchilov and Hutter [2019] Loshchilov, I., Hutter, F., 2019.  Decoupled weight decay regularization, in: International Conference on Learning Representations (ICLR), pp. 1–9.
  * Lowry et al. [2016] Lowry, S., Sünderhauf, N., Newman, P., Leonard, J.J., Cox, D., Corke, P., Milford, M.J., 2016.  Visual place recognition: A survey.  IEEE Transactions on Robotics 32, 1–19.  doi:10.1109/TRO.2015.2496823.
  * Nilsback and Zisserman [2008] Nilsback, M.E., Zisserman, A., 2008.  Automated flower classification over a large number of classes, in: Sixth Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP), pp. 722–729.  doi:10.1109/ICVGIP.2008.47.
  * Parkhi et al. [2012] Parkhi, O.M., Vedaldi, A., Zisserman, A., Jawahar, C., 2012.  Cats and dogs, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3498–3505.  doi:10.1109/CVPR.2012.6248092.
  * Russakovsky et al. [2015] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L., 2015.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision (IJCV) 115, 211–252.  doi:10.1007/s11263-015-0816-y.
  * Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D., 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization, in: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626.  doi:10.1109/ICCV.2017.74.
  * Tang et al. [2021] Tang, Y., Han, K., Xu, C., Xiao, A., Deng, Y., Xu, C., Wang, Y., 2021.  Augmented shortcuts for vision transformers, in: Advances in Neural Information Processing Systems (NeurIPS), pp. 15316–15327.
  * Touvron et al. [2021] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jegou, H., 2021.  Training data-efficient image transformers & distillation through attention, in: International Conference on Machine Learning (ICML), pp. 10347–10357.
  * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I., 2017.  Attention is all you need, in: Advances in Neural Information Processing Systems (NeurIPS), pp. 1–11.
  * Wang et al. [2017] Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., Tang, X., 2017.  Residual attention network for image classification, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6450–6458.  doi:10.1109/CVPR.2017.683.
  * Wang et al. [2022] Wang, P., Wang, X., Wang, F., Lin, M., Chang, S., Li, H., Jin, R., 2022.  Kvt: k-nn attention for boosting vision transformers, in: European Conference in Computer Vision (ECCV), pp. 285–302.  doi:https://doi.org/10.1007/978-3-031-19769-7_1.
  * Yin et al. [2021] Yin, Y., Xu, D., Wang, X., Zhang, L., 2021.  Agunet: Annotation-guided u-net for fast one-shot video object segmentation.  Pattern Recognition 110, 107580.  doi:https://doi.org/10.1016/j.patcog.2020.107580.
  * Yu et al. [2022] Yu, T., Zhao, G., Li, P., Yu, Y., 2022.  Boat: Bilateral local attention vision transformer, in: British Machine Vision Conference (BMVC), pp. 21–24.
  * Yuan et al. [2021a] Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F.E.H., Feng, J., Yan, S., 2021a.  Tokens-to-token vit: Training vision transformers from scratch on imagenet, in: IEEE/CVF International Conference on Computer Vision (ICCV), pp. 538–547.  doi:10.1109/ICCV48922.2021.00060.
  * Yuan et al. [2021b] Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z.H., Tay, F.E., Feng, J., Yan, S., 2021b.  Tokens-to-token vit: Training vision transformers from scratch on imagenet, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 558–567.
  * Yuan et al. [2022] Yuan, L., Hou, Q., Jiang, Z., Feng, J., Yan, S., 2022.  Volo: Vision outlooker for visual recognition.  IEEE transactions on pattern analysis and machine intelligence 45, 6575–6586.
  * Zheng et al. [2017] Zheng, P., Zhao, Z.Q., Gao, J., Wu, X., 2017.  Image set classification based on cooperative sparse representation.  Pattern Recognition 63, 206–217.  doi:https://doi.org/10.1016/j.patcog.2016.09.043.


# Method and portable bench for tests of the laser optical calibration system
components for the Baikal-GVD underwater neutrino Cherenkov telescope

V.A. Allakhverdyan  A.D. Avrorin  A.V. Avrorin  V.M. Aynutdinov  R. Bannasch
Z. Bardačová  I.A. Belolaptikov  I.V. Borina  V.B. Brudanin  N.M. Budnev  V.Y.
Dik  G.V. Domogatsky  A.A. Doroshenko  R. Dvornický  A.N. Dyachok  Zh.-A.M.
Dzhilkibaev  E. Eckerová  T.V. Elzhov  L. Fajt  S.V. Fialkovski  A.R. Gafarov
K.V. Golubkov  N.S. Gorshkov  T.I. Gress  M.S. Katulin  K.G. Kebkal  O.G.
Kebkal  E.V. Khramov  M.M. Kolbin  K.V. Konischev  K.A. Kopański,
11footnotetext: Corresponding author. A.V. Korobchenko  A.P. Koshechkin  V.A.
Kozhin  M.V. Kruglov  M.K. Kryukov  V.F. Kulepov  Pa. Malecki  Y.M. Malyshkin
M.B. Milenin  R.R. Mirgazov  D.V. Naumov  V. Nazari  W. Noga  D.P. Petukhov
E.N. Pliskovsky  M.I. Rozanov  V.D. Rushay  E.V. Ryabov  G.B. Safronov  B.A.
Shaybonov  M.D. Shelepov  F. Šimkovic  A.E. Sirenko  A.V. Skurikhin  A.G.
Solovjev  M.N. Sorokovikov  I. Štekl  A.P. Stromakov  E.O. Sushenok  O.V.
Suvorova  V.A. Tabolenko  B.A. Tarashansky  Y.V. Yablokova  S.A. Yakovlev
D.N. Zaborov

###### Abstract

The large-scale deep underwater Cherenkov neutrino telescopes like Baikal-GVD,
ANTARES or KM3NeT, require calibration and testing methods of their optical
modules. These methods usually include laser-based systems which allow to
check the telescope responses to the light and for real-time monitoring of the
optical parameters of water such as absorption and scattering lengths, which
show seasonal changes in natural reservoirs of water. We will present a
testing method of a laser calibration system and a set of dedicated tools
developed for Baikal- GVD, which includes a specially designed and built,
compact, portable, and reconfigurable scanning station. This station is
adapted to perform fast quality tests of the underwater laser sets just before
their deployment in the telescope structure, even on ice, without darkroom.
The testing procedure includes the energy stability test of the laser device,
3D scan of the light emission from the diffuser and attenuation test of the
optical elements of the laser calibration system. The test bench consists
primarily of an automatic mechanical scanner with a movable Si detector, beam
splitter with a reference Si detector and, optionally, Q-switched diode-pumped
solid-state laser used for laboratory scans of the diffusers. The presented
test bench enables a three-dimensional scan of the light emission from
diffusers, which are designed to obtain the isotropic distribution of photons
around the point of emission. The results of the measurement can be easily
shown on a 3D plot immediately after the test and may be also implemented to a
dedicated program simulating photons propagation in water, which allows to
check the quality of the diffuser in the scale of the Baikal-GVD telescope
geometry.

##  1 Introduction

The Baikal-GVD telescope[1] is located in the south-eastern part of Lake
Baikal, about 55 km to the south of Irkutsk. The telescope’s clusters are
deployed in a distance around 3.6 km from the shore in a depth of 1366 m
(anchor). The telescope currently consists of eight clusters, which are the
independent telescope units. Each cluster consists of 8 instrumented strings,
on each of them there are 36 optical modules, grouped into sections of 12
optical modules. A diagram of the clusters distribution is shown in the Fig.
1. In such a large telescope, it is necessary to synchronize the time as
precisely as possible between individual modules. To ensure this, several
methods of optical calibration are used[2]: LEDs placed in the optical
modules, which by emitting short flashes of light enable calibration of the
optical modules (intra-section callibration), LED matrix used for inter-
section calibration, and a laser system used for calibration between clusters.
Lasers (marked in the Fig. 1) are placed between the clusters of the telescope
at a distance of about 120 m from the nearest instrumented string.

Figure 1: Left: Clusters map for 2021. The positions of the lasers (L1-L5) are
marked by red stars. Strings with lasers are placed about 120 m from the
nearest instrumented string. Right: Laser in a pressure housing, equipped with
a lightguide and a diffuser with a metal hood (reflector) on a top.

Currently, all lasers are equipped with optical diffusers, enabling a fairly
isotropic distribution of light around the laser device. For the proper
operation of the entire system, a dedicated simulation is also necessary, to
determine both the amount of light registered by the modules, as well as to
precisely estimate the time of arrival of subsequent signals. It allows to
accurately determine the propagation of light during a real event recorded by
the telescope. To increase accuracy of the calibration, a special test bench
was developed, allowing not only to precisely examine the distribution of
light emission from diffusers in laboratory conditions, but also to thoroughly
examine the laser system components, including the lasers themselves, before
their deployment from the surface of a frozen lake. Such a measurement allows
to obtain a data set required to estimate the aging rate of laser devices
(lasers and optics as well), but most importantly, it allows actual input data
to be obtained for simulation. The method, being the subject of this
publication, is a combination of a hardware solution, which is the automatic,
reconfigurable test bench, with simulation of light propagation in water. This
method is therefore an extremely useful, efficient and accurate tool that
allowing for a quick determination of the impact of modification of the laser
system on the observed results in the structure of a real telescope. The
simulation program was called Pretorian, because it was created, inter alia,
to ensure the safety of the Optical Modules during in-situ tests with a laser
beam distributed by other method (collimated beam, cone), than by using a
diffuser.

##  2 Test bench and method description

Test bench (Fig. 2) is operated from the Teledyne LeCroy HDO 4034A
oscilloscope (1). For this purpose, a special control program has been
developed, which also performs data acquisition from the oscilloscope to a
file. This allows to easily implement the obtained data from the scan in the
dedicated simulation. The program also makes a real time plot from each
scanned plane.

Figure 2: The scanner consists of a test bench (3) equipped with two automatic
rotary tables – one placed horizontally (rotation in the horizontal plane) and
the other placed vertically (rotation of the tested element around its axis of
symmetry in the axis of the laser beam). The test bench is also equipped with
reconfigurable component holders (6), laser holder (7), beam splitter with
attenuators set (4) and two PDA 100A2 detectors (5) – reference and
measurement. The entire test bench, together with the Q-switched DPSS
laboratory laser ($532\text{\,}\mathrm{nm}$,
$137\text{\,}\mathrm{\SIUnitSymbolMicro J}$) and the necessary accessories, is
housed in a reinforced transportation box, which allows it to be easily moved
and transported to perform the complex tests during a Baikal-GVD Winter
Expedition.

After making a full scan, program creates a 3D graphic presenting the
distribution of light intensity on the surface of the tested diffuser (option
available in the diffuser test mode). Along with the test bench, the PM100USB
laser beam energy monitor with the ES11C detector is also used to control the
stability of the laser parameters. The stand can work with both a laboratory
laser and the laser used in the calibration system of the Baikal-GVD
telescope. Thanks to use of Q-switched lasers, it is not necessary to use a
darkroom – measurements can be made even in the sunlight.

During the research works on the test bench, two scan methods were developed:
the so-called "near scan" (Fig. 3), also called differential scanning, and
"far scan" (Fig. 4), also called integral scanning. Near scan allows the
examination of a small structural changes in optical elements, mainly in
diffusers. It explores the emission of light in a very narrow angle from a
point lying on the surface of the tested element. This scan allows, inter
alia, to determine the impact of doping of the diffuser material to its
attenuation.

Figure 3: Near scan result. The plot shows averaged measurements from many
planes. On the right side the 3D visualization showing the upper part of the
diffuser. On the picture below, the configuration of the test bench for near
scan

The far scan allows to determine the actual light distribution at any distance
from the diffuser (or other tested element). In the scan configuration, the
detector "observes" the entire visible surface of the tested element. Such a
scan already taking into account the absorption and scattering coefficients of
a medium in which the measurement is performed. The data collected from the
far scan performed from a distance of 14 cm (determined experimentally in
relation to the diffuser size) were used as an input data for the simulation
of light propagation in water.

Figure 4: Far scan result. The test was performed with a mirror on the top to
improve the isotropy in a backward region. Right: The scanner control
interface with scan visualization.

Initial research was carried out on a set of three different diffusers used in
the Baikal-GVD experiment till the end of 2020. The research revealed
differences in the light distribution and in the attenuation of individual
diffuser materials (Fig. 5). A number of tests were also carried out to
improve the light distribution to illuminate the Optical Modules below the
laser horizon in the Baikal-GVD telescope. One of the tested solutions was the
propagation of light in the form of a cone. For this purpose, after changing
the configuration of the test bench and equipping it with a dedicated optical
components, a test scan was performed and the data was provided in the
simulation. An exemplary visualization of the conical emission is shown in the
Fig. 6.

Figure 5: Scans results comparison. Figure 6: Simulation result for the laser
attached to the center string of a cluster, equipped with Axicon ($None$), for
upgoing beam in water. The pictures on the right shows the real view of the
laser beam, shaped into a cone in water mist ($65\text{\,}\mathrm{\char
37\relax}\mathrm{R}\mathrm{H}$), seen from the indicated points of the
simulated cluster.

## Acknowledgments

This research is supported by the Plenipotentiary Representative of the
Government of the Republic of Poland at JINR in Dubna under the Projects No.
75/09/2020 and by the Ministry of science and higher education of the Russian
Federation under the contract No. 075-15-2020-778. We would also like to
express our gratitude to our colleagues from the Polish team, A. Bhatt, D.
Góra, J. Stasielak and M. Wiśniewski, who contributed to this article.

## References

  * [1] V. A. Allakhverdyan and A. D. Avrorin at al., _Measuring muon tracks in Baikal-GVD using a fast reconstruction algorithm_ , arXiv:hep-ex/2106.06288.
  * [2] A. Avrorin, V.M. Aynutdinov at al., _Time calibration of the neutrino telescope Baikal-GVD_ , _EPJ Web of Conferences_ 207 (2019), 10.1051/epjconf/201920707003.