11institutetext:  Faculty of Computer Science, Brawijaya University, Malang
City, East Java, Indonesia, 65145
11email:
<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>
# SparseSwin: Swin Transformer with Sparse Transformer Block

Krisna Pinasthika  Blessius Sheldo Putra Laksono  Riyandi Banovbi Putera Irsal
Syifa’ Hukma Shabiyya  Novanto Yudistira

###### Abstract

Advancements in computer vision research have put transformer architecture as
the state-of-the-art in computer vision tasks. One of the known drawbacks of
the transformer architecture is the high number of parameters, this can lead
to a more complex and inefficient algorithm. This paper aims to reduce the
number of parameters and in turn, made the transformer more efficient. We
present the Sparse Transformer (SparTa) Block, a modified transformer block
with an addition of a sparse token converter that reduces the number of tokens
used. We use the SparTa Block inside the Swin-T architecture (SparseSwin) to
leverage Swin’s capability to downsample its input and reduce the number of
initial tokens to be calculated. The proposed SparseSwin model outperforms
other state-of-the-art models in image classification with an accuracy of
86.96%, 97.43%, and 85.35% on the ImageNet100, CIFAR10, and CIFAR100 datasets
respectively. Despite its fewer parameters, the result highlights the
potential of a transformer architecture using a sparse token converter with a
limited number of tokens to optimize the use of the transformer and improve
its performance. The code available at
https://github.com/KrisnaPinasthika/SparseSwin

###### Keywords:

CIFAR10 CIFAR100 Computer vision Image classification ImageNet100 Transformer

##  1 Introduction

Computer vision has become one of the critical topics in the world of
artificial intelligence. The majority of computer vision models use CNN as the
method for image classification [13, 20, 9, 22, 10, 23, 11, 24]. When
assessing the performance of the model, dataset such as Imagenet [4] is used
because of its extensive data and variety of classes. Using large data in the
model’s training phase dramatically improves performance when tested on real-
life or new data. The use of CNN on many image classification models made CNN
evolve to a more sophisticated scale, connectivity, and forms of convolution
[2].

However, the evolution of network topologies in Natural Language Processing
(NLP) has taken a different path, with Transformer now becoming the dominant
architecture [26]. Transformer dominance has even entered the subject of
computer vision with the introduction of the Vision Transformer [5]. Vision
Transformer modifies the transformer model for Natural Language Processing
(NLP) for image classification. Vision Transformer’s architecture uses a
stacked Transformer Block to process image patches that do not overlap [34].
Compared to its rival CNN, Transformer-based models have large receptive
fields [3] and superior performance on large datasets [15]. Despite the
outstanding performance, excessive attention creates high computing costs
because of the dependence on the image input size in carrying out self-
attention calculations, and it takes a long time to converge while training.

There is various state-of-the-art to use Transformers in image classification
problems which enhances the original Vision Transformer model and makes it
more robust and efficient. One of the state-of-the-art research on
transformers for image classification is the Swin Transformer. Swin
Transformer avoids the excessive use of attention on the whole image by using
local window-based attention [15] and manages to reduce the computing costs of
using a transformer on images. On the other hand, SparseFormer [6] adjusts the
region of interest (RoI) by focusing on the front of the image using several
latent tokens. This was done to extract features from the image. The advantage
of this approach is that the calculation is done on a limited number of
tokens, so the calculation of self-attention is not affected by the size of
the input image but depends on the size of the latent token thus improving
efficiency.

In previous state-of-the-art research, the use of a transformer for image
classifications has a problem with a large number of parameters. For example,
the original Vision Transformer adopts a general transformer model formerly
used in NLP resulting in 85M parameter. While other research such as [15] [6]
manages to reduce the number of parameters to 27.6 M and 32M it still pales in
comparison with other small and lightweight CNN-based models such as [24, 9]
with 12M and 26M parameters.

Inspired by the success of Swin Transformer [15] and SparseFormer [6], we
combine local attention with the windowing technique used by Swin Transformer
and the use of a limited latent token to improve efficiency further. In our
approach, we employ the Swin Transformer to process the image in several
stages and feed it into Sparse Transformer (SparTa) block, which compresses
the amount of token and provides results that can be classified in the
classification layer. Since this Transformer Block uses a limited number of
tokens in performing computation, it is reasonable to call this block Sparse
as done by the research conducted by [6]. The computation performed on SparTa
Block depends on the token size so it is smaller when compared to the input
image.

The proposed SparseSwin model overtook several other models in image
classification. SparseSwin model has a smaller number of parameters, 17.58M
when compared to the 27.6M parameter in the Swin-T model and 85M parameter in
the ViT-B model. The reduction in the number of parameters in SparseSwin is
also followed by an improvement in accuracy when compared to state-of-the-art
research with identical parameters. SparseSwin achieved accuracies of 86.96%,
97.43%, and 85.35% on the ImageNet100, on CIFAR10, and CIFAR100 datasets
respectively. In summary, our contributions are:

  1. 1.

We propose a Sparse Transformer (SparTa) Block architecture using limited
latent tokens to improve the efficiency and performance of the Swin
Transformer for image classification.

  2. 2.

We examined the effect of regularization on the attention weight obtained in
SparTa Block.

  3. 3.

We obtained accuracy improvement on ImageNet100, CIFAR10, and CIFAR100
benchmark datasets.

  4. 4.

We obtained a smaller parameter size compared to the state-of-the-art
Transformer model.

The paper is divided into 5 sections: Section 2 provides an overview of
relevant previous research. Section 3 outlines our proposed approach. Section
4 shows the experimental results, which assess the effectiveness of our model.
Section 5 provides a summary of our contributions.

##  2 Related Works

Transformer Attention Mechanism. The Attention mechanism is adapted from how
humans are selective about the information [8]. This mechanism allows the
model to focus more on the important part of the input to produce accurate
output. The Attention Mechanism initially introduce through the Recurrent
Attention Model (RAM) [18]. Attention Mechanism continues to develop to create
various variations such as hard attention [29], self-attention [26], multi-
head attention [26], cross-modal attention [17], and attention with
Convolutional Neural Network (CNN) [27]. Then, in recent research, an
attention mechanism was developed to pay more attention to certain areas of
the image and ignore less important areas called the Focusing Transformer
implemented in SparseFormer [6]. This research will utilize a modified version
of the Focusing Transformer as a method for retrieving important features with
input generated from the output of the Swin Transformer Block.

Vision Transformer. Vision Transformer (ViT) [5] was developed based on the
Transformer concept applied to NLP, which is then applied to image
recognition. ViT will split the image into a set of patches of the same size
and be considered as a sequence. The sequence of patches will be used as input
for the Transformer model. ViT becomes the inspiration of many of the further
works, one of which uses shifting windows to reduce the complexity of the
algorithm [8] ViT achieves an accuracy of up to 88% when trained on the
ImageNet1K dataset. This result is better when compared to other CNN-based
solutions, such as ResNet152, with an accuracy of 87%.

Swin Transformer. The development of methods for vision tasks is extensive. In
the previous Vision Transformer [5], the resulting computational complexity
increases quadratically with the input image size due to the global attention
calculation. To overcome this, a backbone algorithm is recently proposed in
the Transformer that has a linear computational complexity calculation for the
image size. The Swin Transformer will start with small patches and then, in
deeper layers of the Transformer, will merge adjacent patches [15]. The Swin
Transformer scheme is a shifted window that is flexible. Swin Transformer
achieved an accuracy of 84%, which is 5% better compared to the base ViT model
from previous research. This research utilizes the adaptation of the Swin
Transformer with the addition of a Focusing Transformer on the resulting
output.

SparseFormer. This paper proposes a new model for using transformer
architecture in solving computer vision tasks. SparseFormer uses a limited
number of latent tokens used in customizing Region of Interest (RoI) tokens by
focusing on the front of the image with a limited number of tokens to extract
features from the image. Since the computation is only performed on a limited
number of tokens, the amount of computation performed does not depend on the
size of the input image so that the computation process can be more efficient
[6].

##  3 Proposed Method

In this section, we describe in detail the architecture of SparseSwin.
SparseSwin is constructed from the combination of the Swin Transformer and
Sparse Transformer Block (SparTa Block). This architecture consists of 4
stages with stages 1 to 3 using the same Patch Merging Block and Swin
Transformer Blocks as the Swin Transformer [15], Whereas in stage 4 we use our
proposed SparTa Block followed by a Layer Normalization [1]. Specifically, our
proposed architecture is illustrated in Fig. 1.

Initially, we propagate the input image to “Stage 1”. In this stage, Patch
Merging is applied to split the input image into several non-overlapping
patches with a patch size of 4 which is then followed by applying linear
embedding as performed by [15]. Patch merging is used to reduce the size of
the feature map while maintaining the spatial information with an output size
of $\frac{H}{8}\times\frac{W}{8}$. The same is done in ”Stage 2” and ”Stage 3”
with output sizes of $\frac{H}{16}\times\frac{W}{16}$ and
$\frac{H}{32}\times\frac{W}{32}$. Thereafter, the feature map obtained from
Patch Merging will be propagated to the Swin Transformer Blocks. The Swin
Transformer Blocks used in this research are the same as the previous research
[15].

Figure 1: The architecture of SparseSwin

Following the feature map obtained in Stage 3, the feature map will be
propagated to the SparTa Block. SparTa Block consists of two main parts: the
Sparse Token Converter and the regular Transformer Block. In detail, the
SparTa Block architecture is illustrated in Fig. 2. SparTa Block will change
the representations of features generated by the previous Stages B, C, H, and
W which are Batch size, Height, Channel, and Width respectively into token
sizes along with embedding with an output size of B, t, e which is the Batch
size, Number of latent tokens, and Size of the embedding respectively. The
process that occurs within the SparTa Block will be described in more detail
in the next section. The output generated from the SparTa Block will be
propagated into the normalization layer [1]. The normalization Layer changes
the input data range from different scales to the same range to prevent the
accumulation of large error gradients that cause the training to be unstable.

Figure 2: The successive SparTa blocks in Stage 4 of SparseSwin for image
classification

Sparse Token Converter. Sparse Token Converter is used to convert the
dimension size of the output of the previous stage into a new token with a new
embedding dimension size. The conversion process is done using a convolution
layer [14] to acquire the new embedding and a linear layer to convert the size
of the patches into latent tokens. Specifically, the kernel and stride sizes
used in the convolution layer are 3 and 1. Meanwhile, the input features in
the linear layer are $\textit{H}\times\textit{W}$ in stage 3 with an output
feature of 49 to produce a new token representation. The output obtained from
the Sparse Token Converter ($S_{T}$) is $S_{T}\in\ \mathbb{R}^{t\ \times\ e}$
where t is the desired number of new tokens and e is the number of new
embedding dimension.

Regular Transformer Block. The regular Transformer Block used in this research
is the same transformer block as the Transformer Block introduced by [26]. The
MSA calculation is done by receiving $S_{T}$ input that has been normalized
with Layer Normalization (LN) [1]. Furthermore, the normalized MSA output will
be propagated to the Multilayer Perceptron (MLP) with a hidden ratio size of
4. Regular transformer block calculation is presented in Eqs 1 and 2 where
$\hat{z}^{l}$ and $z^{l}$ are the output features of the MSA block and MLP
block. Specifically, in this research, the loops used in the Regular
Transformer Block used are 2.

$\hat{z}^{l}=MSA(LN(S_{T}))+S_{T}$ (1)

$z^{l}=MLP(LN(\hat{z}^{l}))+\hat{z}^{l}$ (2)

Therefore, the overall output of SparTa Block is $Block_{S_{T}}$ $\in$
$\mathbb{R}^{b\times t\times e}$ where $b$ is the size of the batch used. The
computation of MSA using SparTa Block is not affected by the size of the input
image. However, it depends on the number of tokens used, so in general, the
use of SparTa Block can reduce the size of ordinary MSA computation.

##  4 Experiments

We conducted several experiments to observe SparseSwin’s performance using
ImageNet100 [19], CIFAR10 [13], and CIFAR100 [13] image classification
datasets. We also experiment with regularization of the attention weights to
prevent the model from overfitting and make the model generalize better.
Furthermore, we will compare the best results obtained in the previous
experiments with other state-of-the-art research.

###  4.1 Implementation Details

Configuration on ImageNet100. In the model training process, we used Adam [12]
as the optimizer with a learning rate of 1e-4 and batch size of 128. The model
is trained on ImageNet100, and this dataset is a subset of the ImageNet1K
dataset with only 100 classes. The training process is carried out using
NVIDIA A100. The model is trained up to 100 epochs to find the point where the
training process converges based on the change of loss value. In the training
phase of SparseSwin, we freeze “Stage 1” and “Stage 2” to prevent the training
process on low-level features. The freeze is done because the backbone has
trained on ImageNet1K, so the training is only devoted to higher-level
features. In detail, the training process is carried out using a total of 49
tokens with an embedding size and num MSA heads in the SparTa Block of 512 and
16 respectively. The number of SparTa Block loops used is 2 and uses the QKV
bias.

Configuration on CIFAR. We fine-tune the model with the AdamW optimizer [16],
which proved to be successful in improving its performance. This optimizer has
a learning rate of 1e-5 and a weight decay of 0.01, which helps to adjust the
model’s parameters. The input data is resized to 224 dimensions to match the
model to the SparseSwin architecture. Furthermore, data augmentation
techniques such as random cropping, random horizontal flips, and resizing
cropping are used to expand and diversify the dataset, boosting the model’s
capacity to generalize and handle different scenarios. The detailed
architecture used for training on the CIFAR10 and CIFAR100 datasets uses the
same configuration as ImageNet100 using the NVIDIA A100.

###  4.2 The Influence of The Sparsity Attention Weight in Sparta Block for
Image Classification

In this research, we conduct further experiments to test SparseSwin’s ability
to perform image classification. We experiment by testing the effectiveness of
L1 [25] and L2 [7] regularization on the attention weight obtained by SparTa
Block in the training phase. Attention weight was chosen in this study because
it was inspired by the research conducted by Zhang et al. [33] related to the
case of neural machine translation and abstractive summarization, that the
attention mechanism is used to guide the prediction process sequentially
focusing on specific words thus the higher the attention weight, the more the
corresponding word contribution. Nevertheless, not all words are relevant to
produce a certain output. Thus, we apply the problem to the case of image
classification by making the attention weight sparser, allowing SparTa Block
to select more relevant features in the case of image classification. Besides
making the attention weight sparser, the use of regularization has been proven
to prevent overfitting and can make the model generalize better. In this
study, we tested the lambda values used for regularization including 1e-4 and
1e-5. The experimental results are reported in Table 1.

Based on the results of our evaluation metrics testing, SparseSwin has been
able to provide good evaluation results with 86.64%, 97.43%, and 85.35%
accuracy on ImageNet100, CIFAR10, and CIFAR100 datasets, respectively. The
test continued by training SparseSwin with L1 and L2 regulators on the
attention weights obtained in the SparTa Block. Then, we continued testing the
trained model and found that the use of regularization gave slightly better
results on the ImageNet100 dataset with 86.96% accuracy while it showed no
improvement on the CIFAR10 and CIFAR100 datasets. Based on our analysis, L1
and L2 regularization did not have a significant impact on the CIFAR10 and
CIFAR100 datasets because the number of datasets used was not as large as the
ImageNet100 dataset. Additionally, considering the fact that we increased the
resolution of the input images on the CIFAR10 and CIFAR100 datasets from
$32^{2}$ to $224^{2}$ could potentially hinder the effectiveness of the
regularization.

Table 1: Quantitative comparison of SparseSwin using L1 & L2 Regularizer on
Imagenet100, CIFAR10, and CIFAR100

Model | Lambda Regularizer | Input Resolution | Accuracy(%)
---|---|---|---
ImageNet100 | CIFAR10 | CIFAR100
SparseSwin | - | $224^{2}$ | 86.64 | 97.43 | 85.35
SparseSwin with L1 | 1e-4 | $224^{2}$ | 86.68 | 97.42 | 84.70
SparseSwin with L1 | 1e-5 | $224^{2}$ | 86.88 | 97.28 | 84.86
SparseSwin with L2 | 1e-4 | $224^{2}$ | 86.96 | 97.31 | 84.80
SparseSwin with L2 | 1e-5 | $224^{2}$ | 86.90 | 97.31 | 84.88

###  4.3 Results on ImageNet100

We conducted a performance comparison between SparseSwin and other state-of-
the-art models using the ImageNet100 dataset as a benchmark. We found several
types of models with the same input resolution of $224^{2}$ with various
parameter sizes. The experimental results are presented in Table 2. Based on
the experimental results that have been carried out, SparseSwin achieves the
smallest number of parameters of $~{}17$ M when compared to Swin-T [15] and
ViT-B [5] models with $~{}27$ M and $~{}85$ M parameters. Despite having a
much smaller number of parameters than other transformer models, it is obvious
that the SparseSwin model achieves higher accuracy than the previous study
with an accuracy of 86.96%. SparseSwin outperforms the recent state-of-the-art
research and obtains higher accuracy with a difference of $\pm 1.74\%$, $\pm
6.06\%$, and $\pm 7.66\%$ when compared to Swin-T, Vit-B, and DLME [31]
respectively.

Table 2: Quantitative Comparison on ImageNet100

Model | #Params(100 class) | Input Resolution | Model Type | Accuracy(%)
---|---|---|---|---
Swin-T [15] | 27.6 M | $224^{2}$ | Transformer | 85.22
ViT-B [5] | 85.87 M | $224^{2}$ | Transformer | 80.90
DLME(ResNet-50,linear) [31] | $-$ | $224^{2}$ | Convolution | 79.3
SparseSwin with L2 | 17.58 M | $224^{2}$ | Transformer | 86.96

###  4.4 Results on CIFAR10

In this section, we compare the performance between SparseSwin and other
state-of-the-art models using the CIFAR10 dataset as a benchmark. We found
multiple model types with diverse input resolutions and parameter sizes,
namely $32^{2}$ and $224^{2}$. Quantitative comparison between SparseSwin and
state-of-the-art models is presented in Table 3. Compared to our proposed
method, the input resolution size used in the DenseNet-BC-19 [32] and NesT-B
[35] models is $32^{2}$, which is smaller than the SparseSwin model. Despite
having a smaller input resolution size, DenseNet-BC-19 and NesT-B produce a
larger number of parameters compared SparseSwin, namely, 25.6 M and 97.2 M.
With a larger number of parameters, DenseNet-BC-19 and NesT-B were still not
performing optimally when compared to SparseSwin which achieved 97.43%
accuracy, with a margin of $\pm 0.13\%$ and $\pm 0.23\%$.

Furthermore, there are various other state-of-the-art studies that have the
same input resolution size as our proposed model including ResNet XnIDR [21],
CRATE-S [30], CRATE-B [30], and CRATE-L [30] with the number of parameters
23.86 M, 13.12 M, 22.80 M, and 77.64 M respectively. Although SparseSwin has
fewer parameters than state-of-the-art research, our proposed model
outperformed other models by a slight improvement with a difference of $\pm
0.56\%$, $\pm 1.43\%$, $\pm 0.63\%$, and $\pm 0.23\%$ on ResNet XnIDR [21],
CRATE-S [30], CRATE-B [30], and CRATE-L [30] respectively.

Table 3: Quantitative Comparison on CIFAR10

Model | #Params(10 class) | Input Resolution | Model Type | Accuracy(%)
---|---|---|---|---
DenseNet-BC-190+Mixup[32] | 25.6 M | $224^{2}$ | Convolution | 97.3
ResNet XnIDR [21] | 23.86 M | $224^{2}$ | Convolution | 96.87
NesT-B[35] | 97.2 M | $32^{2}$ | Transformer | 97.2
CRATE-S[30] | 13.12 M | $224^{2}$ | Transformer | 96
CRATE-B[30] | 22.80 M | $224^{2}$ | Transformer | 96.8
CRATE-L[30] | 77.64 M | $224^{2}$ | Transformer | 97.2
SparseSwin | 17.58 M | $224^{2}$ | Transformer | 97.43

###  4.5 Results on CIFAR100

In this section, we conducted a quantitative comparison of our proposed model
with state-of-the-art research on the CIFAR100 dataset as a benchmark. There
are various input resolutions used in previous research, namely $32^{2}$ and
$224^{2}$. The quantitative test results of our proposed model with the state-
of-the-art are presented in Table 4. Based on our test results, SparseSwin is
able to outperform previous research with an accuracy of 85.35% with margins
of $\pm 0.93\%$, $\pm 2.79\%$, $\pm 4.35\%$, $\pm 2.65\%$ and $\pm 1.75\%$ for
ResNeXt-50 [28] , Transformer Local-Attention (NesT-B) [35], and CRATE-S [30],
CRATE-B [30], CRATE-L [30] respectively.

Table 4: Quantitative Comparison on CIFAR100

Model | #Params(10 class) | Input Resolution | Model Type | Accuracy(%)
---|---|---|---|---
ResNeXt-50[28] | 25.03 M | $224^{2}$ | Transformer | 84.42
NesT-B[35] | 97.2 M | $32^{2}$ | Transformer | 82.56
CRATE-S[30] | 13.12 M | $224^{2}$ | Transformer | 81.0
CRATE-B[30] | 22.80 M | $224^{2}$ | Transformer | 82.7
CRATE-L[30] | 77.64 M | $224^{2}$ | Transformer | 83.6
SparseSwin | 17.58 M | $224^{2}$ | Transformer | 85.35

##  5 Conclusion

In this paper, we propose a novel way to reduce the number of parameters and
improve the capability of the Swin Transformer in image classification by
using the Sparse Transformer Block (SparTa Block). This block consists of
Sparse Token Converter and Regular Transformer Block to transform the feature
representation obtained in Stage 3 of the Swin Transformer into a more concise
representation while maintaining the important features required by the model
to perform classification with the assistance of the attention mechanism in
the regular transformer block. Through this modification, we obtained a
smaller number of parameters compared to the Swin Transformer with 17.58 M
parameters and obtained improvements in terms of accuracy on the ImageNet100,
CIFAR10, and CIFAR100 datasets with accuracies of 86.96%, 97.43%, and 85.35%
respectively. Through this research, we hope that SparseSwin can perform well
in classifying low-end devices and provide high performance in critical cases
(e.g., biomedical analysis).

## References

  * [1] Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
  * [2] Chen, L., Li, S., Bai, Q., Yang, J., Jiang, S., Miao, Y.: Review of image classification algorithms based on convolutional neural networks. Remote Sensing 13(22), 4712 (2021)
  * [3] Cheng, M., Ma, H., Ma, Q., Sun, X., Li, W., Zhang, Z., Sheng, X., Zhao, S., Li, J., Zhang, L.: Hybrid transformer and cnn attention network for stereo image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1702–1711 (2023)
  * [4] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848
  * [5] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [6] Gao, Z., Tong, Z., Wang, L., Shou, M.Z.: Sparseformer: Sparse visual recognition via limited latent tokens. arXiv preprint arXiv:2304.03768 (2023)
  * [7] Golub, G.H., Hansen, P.C., O’Leary, D.P.: Tikhonov regularization and total least squares. SIAM journal on matrix analysis and applications 21(1), 185–194 (1999)
  * [8] Gong, H., Mu, T., Li, Q., Dai, H., Li, C., He, Z., Wang, W., Han, F., Tuniyazi, A., Li, H., et al.: Swin-transformer-enabled yolov5 with attention mechanism for small object detection on satellite images. Remote Sensing 14(12), 2861 (2022)
  * [9] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015), http://arxiv.org/abs/1512.03385
  * [10] Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR abs/1704.04861 (2017), http://arxiv.org/abs/1704.04861
  * [11] Huang, G., Liu, Z., Weinberger, K.Q.: Densely connected convolutional networks. CoRR abs/1608.06993 (2016), http://arxiv.org/abs/1608.06993
  * [12] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [13] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C., Bottou, L., Weinberger, K. (eds.) Advances in Neural Information Processing Systems. vol. 25. Curran Associates, Inc. (2012), https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
  * [14] LeCun, Y., Bengio, Y., et al.: Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks 3361(10), 1995 (1995)
  * [15] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012–10022 (2021)
  * [16] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [17] Lu, J., Yang, J., Batra, D., Parikh, D.: Neural baby talk. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7219–7228 (2018)
  * [18] Mnih, V., Heess, N., Graves, A., et al.: Recurrent models of visual attention. Advances in neural information processing systems 27 (2014)
  * [19] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International journal of computer vision 115, 211–252 (2015)
  * [20] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)
  * [21] Sun, J., Fard, A.P., Mahoor, M.H.: Xnodr and xnidr: Two accurate and fast fully connected layers for convolutional neural networks. arXiv preprint arXiv:2111.10854 (2021)
  * [22] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S.E., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. CoRR abs/1409.4842 (2014), http://arxiv.org/abs/1409.4842
  * [23] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. CoRR abs/1512.00567 (2015), http://arxiv.org/abs/1512.00567
  * [24] Tan, M., Le, Q.V.: Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR abs/1905.11946 (2019), http://arxiv.org/abs/1905.11946
  * [25] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological) 58(1), 267–288 (1996)
  * [26] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [27] Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV). pp. 3–19 (2018)
  * [28] Xie, S., Girshick, R.B., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. CoRR abs/1611.05431 (2016), http://arxiv.org/abs/1611.05431
  * [29] Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. In: International conference on machine learning. pp. 2048–2057. PMLR (2015)
  * [30] Yu, Y., Buchanan, S., Pai, D., Chu, T., Wu, Z., Tong, S., Haeffele, B.D., Ma, Y.: White-box transformers via sparse rate reduction. arXiv preprint arXiv:2306.01129 (2023)
  * [31] Zang, Z., Li, S., Wu, D., Wang, G., Wang, K., Shang, L., Sun, B., Li, H., Li, S.Z.: Dlme: Deep local-flatness manifold embedding. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXI. pp. 576–592. Springer (2022)
  * [32] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)
  * [33] Zhang, J., Zhao, Y., Li, H., Zong, C.: Attention with sparsity regularization for neural machine translation and summarization. IEEE/ACM Transactions on Audio, Speech, and Language Processing 27(3), 507–518 (2018)
  * [34] Zhang, Q., Zhang, J., Xu, Y., Tao, D.: Vision transformer with quadrangle attention. arXiv preprint arXiv:2303.15105 (2023)
  * [35] Zhang, Z., Zhang, H., Zhao, L., Chen, T., Arik, S.Ö., Pfister, T.: Nested hierarchical transformer: Towards accurate, data-efficient and interpretable visual understanding. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 3417–3425 (2022)