# Detecting Severity of Diabetic Retinopathy from Fundus Images using
Ensembled Transformers

Chandranath Adak  Senior Member, IEEE  Tejas Karkera  Soumi Chattopadhyay
Member, IEEE  and Muhammad Saqib C. Adak is with Dept. of CSE, IIT Patna,
India-801106, T. Karkera is with Atharva College of Engineering, Mumbai,
India-400095, S. Chattopadhyay is with the Dept. of CSE, IIIT Guwahati,
India-781015, and M. Saqib is with Data61, CSIRO, Australia-2122.
Corresponding author: C. Adak (e-mail<EMAIL_ADDRESS>
###### Abstract

Diabetic Retinopathy (DR) is considered one of the primary concerns due to its
effect on vision loss among most people with diabetes globally. The severity
of DR is mostly comprehended manually by ophthalmologists from fundus
photography-based retina images. This paper deals with an automated
understanding of the severity stages of DR. In the literature, researchers
have focused on this automation using traditional machine learning-based
algorithms and convolutional architectures. However, the past works hardly
focused on essential parts of the retinal image to improve the model
performance. In this paper, we adopt transformer-based learning models to
capture the crucial features of retinal images to understand DR severity
better. We work with ensembling image transformers, where we adopt four
models, namely ViT (Vision Transformer), BEiT (Bidirectional Encoder
representation for image Transformer), CaiT (Class-Attention in Image
Transformers), and DeiT (Data efficient image Transformers), to infer the
degree of DR severity from fundus photographs. For experiments, we used the
publicly available APTOS-2019 blindness detection dataset, where the
performances of the transformer-based models were quite encouraging.

Blindness Detection, Diabetic Retinopathy, Deep learning, Transformers.

##  1 Introduction

Diabetes Mellitus, also known as _diabetes_ , is a disorder where the patient
experiences increased blood sugar levels over a long period. Diabetic
Retinopathy (DR) is a microvascular complication of diabetes where the
retina's blood vessels get damaged, which can lead to poor vision and even
blindness if untreated [1, 2]. Studies estimated that by twenty years after
diabetes onset, about 99% (or 60%) of patients having type-I (or type-II)
diabetes might have DR [1]. With a worldwide presence of DR patients of about
126.6 million in 2010, the current estimate is roughly around 191 million by
2030 [3, 4]. However, about 56% of new DR cases can be reduced by timely
treatment and monitoring of the severity [5]. The ophthalmologist analyzes
fundus images for lesion-based symptoms like microaneurysms, hard/ soft
exudates, and hemorrhages to understand the severity stages of DR [1, 2]. The
positive DR is divided into the following stages [5]: ($1$) _mild_ : the
earliest stage that can contain microaneurysms, ($2$) _moderate_ : here, the
blood vessels lose the ability to blood transportation, ($3$) _severe_ : here,
blockages in blood vessels can occur and gives a signal to grow new blood
vessels, ($4$) _proliferative_ : the advanced stage where new blood vessels
start growing. Fig. 1 shows some fundus images representing the DR severity
stages. Manual examining fundus images for DR severity stage grading may bring
inconsistencies due to a high number of patients, less number of well-trained
clinicians, long diagnosing time, unclear lesions, etc. Moreover, there may be
disagreement among ophthalmologists in choosing the correct severity grade
[6]. Therefore, computer-aided techniques have come into the scenario for
better diagnosis and broadening the prospects of early-stage detection [2].

negative mild moderate severe proliferative

Figure 1: Fundus images with DR severity stages from APTOS-2019 [7].

Automated DR severity stage detection from fundus photographs has been
performed for the last two and half decades. Earlier, some image processing
tools were used [8, 9], but the machine learning-based DR became popular in
the early 2000s. The machine learning-based techniques mostly relied on hand-
engineered features that were carefully extracted from the fundus images and
then fed to a classifier, e.g., Random Forest (RF) [10], KNN (K-Nearest
Neighbors) [11], SVM (Support Vector Machine) [12], and ANN (Artificial Neural
Network) [13]. Although SVM and ANN-based models were admired in the DR
community, the hand-engineered feature-based machine learning models require
efficient prior feature extraction, which may introduce errors for complex
fundus images [2, 1]. On the other hand, deep learning-based models extract
features automatically through convolution operations [14, 15]. Besides, from
2012, deep learning architectures rose to prominence in the computer vision
community, which also influenced the DR severity analysis from fundus images
[1]. The past deep learning-based techniques mostly employed CNN
(Convolutional Neural Network) [16, 1]. However, the ability to give attention
to certain regions/features and fade the remaining portions hardly exists in
classical CNNs. For this reason, some contemporary methods incorporated
attention mechanism [17, 18]. Although multiple research works are present in
the literature [2, 1] and efforts were made to detect the existence of DR in
the initial stages of its development, still there is a room for improving the
performance by incorporating higher degrees of automated feature extraction
using better deep learning models.

In this paper, we employ the transformer model for leveraging its MSA (Multi-
head Self-Attention) [19] to focus on the DR revealing region of the fundus
image for understanding the severity. Moreover, the transformer model has
shown high performance in recent days [20, 19]. Initially, we adopted ViT
(Visual Transformer) [19] for detecting DR severity due to its outperformance
on image classification tasks. ViT divides the input image into a sequence of
patches and applies global attention [19]. Moreover, since standard ViT
requires hefty amounts of data, we also adopted some other image transformer
models, such as CaiT (Class-attention in image Transformers) [21], DeiT (Data-
efficient image Transformer) [22], and BEiT (Bidirectional Encoder
representation for image Transformer) [23]. CaiT is a modified version of ViT
and employs specific class-attention [21]. DeiT uses knowledge distillation,
which transfers the knowledge from one network to another and builds a
teacher-student hierarchical network [22]. BEiT is inspired by BERT
(Bidirectional Encoder Representations from Transformers) [24] to implement
masking of image patches and to model the same for pre-training the ViT [23].
For experiments, we used the publicly available APTOS-2019 blindness detection
dataset [7], where the individual image transformers did not perform well.
Therefore, we ensembled the image transformers to seek better predictive
performance. The ensembled image transformer obtained quite encouraging
results for DR severity stage detection. This is one of the earliest attempts
to adopt and ensemble image transformers for DR severity stage detection,
which is the main contribution of this paper.

The rest of the paper is organized as follows. $\mathsection$ 2 discusses the
relevant literature about DR and $\mathsection$ 3 presents the proposed
methodology. Then $\mathsection$ 4 analyzes and discusses the experimental
results. Finally, $\mathsection$ 5 concludes this paper.

##  2 Related Work

This section briefly presents the literature on DR severity detection from
fundus images. The modern grading of DR severity stages can be traced in the
report by ETDRS research group [25]. In the past, some image processing-based
(e.g., wavelet transform[8], radon transform [9]) strategies were published.
For the last two decades, machine learning and deep learning-based approaches
have shown dominance. We broadly categorize the related works into (a) hand-
engineered feature-based models [26, 11, 27], and (b) deep feature-based
models [2], which are discussed below.

###  2.1 Hand-engineered Feature-based Models

The hand-engineered feature-based models mostly employed RF [26], KNN [28],
SVM [27], ANN [29] for detecting DR severity stages. Acharya et al. [26]
employed a decision tree with discrete wavelet/cosine transform-based features
extracted from retinal images. Casanova et al. [10] introduced RF for DR
severity stage classification. In [30], RF was also used to assess DR risk.
KNN classifier was employed in [11] to detect drusen, exudates, and cotton-
wool spots for diagnosing DR. Tang et al. [28] used KNN for retinal hemorrhage
detection from fundus photographs. In [27], retinal changes due to DR was
detected by using SVM. Akram et al. [12] used SVM and GMM (Gaussian Mixture
Model) with enhanced features such as shape, intensity, and statistics of the
affected region to identify microaneurysms for early detection of DR. ANN was
employed in [13] to classify lesions for detecting DR severity. Osareh et al.
[31] employed FCM (Fuzzy C-Means)-based segmentation and GA (Genetic
algorithm)-based feature selection with ANN to detect exudates in DR. In [29],
PSO (Particle Swarm Optimization) was used for feature selection, followed by
ANN-based DR severity classification.

###  2.2 Deep Feature-based Models

The past deep architectures mostly used CNN for tackling DR severity. For
example, Yu et al. [16] used CNN for detecting exudates in DR, Chudzik et al.
[32] worked on microaneurysm detection using CNN with transfer learning and
layer freezing, Gargeya and Leng [33] employed CNN-based deep residual
learning to identify fundus images with DR. In [4], CNN was also used to
identify DR severity stages and some related eye diseases, e.g., glaucoma and
AMD (Age-related Macular Degeneration). In [34], some classical CNN
architectures (e.g., AlexNet, VGG Net, GoogLeNet, ResNet) were employed for DR
severity stage detection. Wang et al. [17] proposed Zoom-in-Net that combined
CNN, attention mechanism, and a greedy algorithm to zoom in the region of
interest for handling DR. A modified DenseNet169 architecture in conjunction
with the attention mechanism was used in [18] to extract refined features for
DR severity grading. In [35], a modified Xception architecture was employed
for DR classification. TAN (Texture Attention Network) was proposed in [36] by
leveraging style (texture features) and content (semantic and contextual
features) re-calibration mechanism. Tymchenko et al. [5] ensembled three CNN
architectures (EfficientNet-B4 [37], EfficientNet-B5, and SE- ResNeXt50 [38])
for DR severity detection. Very recently, a few transformer-based models have
come out, e.g., CoT-XNet [39] that combined contextual transformer and
Xception architecture, SSiT [40] that employed self-supervised image
transformers guided by saliency maps.

##  3 Methodology

This section first formalizes the problem statement, which is then followed by
the proposal of solution architecture.

###  3.1 Problem Formulation

In this work, we are given an image $I$ captured by the fundus photography,
which is input to the architecture. The task is to predict the severity stage
of diabetic retinopathy (DR) among negative, mild, moderate, severe, and
proliferative, from $I$. We formulate the task as a multi-class classification
problem [15]. Here, from $I$, features are extracted and fed to a classifier
to predict the DR severity class labels , where $=\\{0,1,2,3,4\\}$ corresponds
to {negative, mild, moderate, severe, proliferative}, respectively.

###  3.2 Solution Architecture

For detecting the severity stage of DR from a fundus photograph, we adopt
image transformers, i.e., ViT (Vision Transformer) [19], BEiT (Bidirectional
Encoder representation for image Transformer) [23], CaiT (Class-attention in
image Transformers) [21], and DeiT (Data efficient image Transformers) [22],
and ensemble them. However, we preprocess raw fundus images before feeding
them into the transformers, which we discuss first.

####  3.2.1 Preprocessing

The performance of deep learning models is susceptible to the quality and
quantity of data being passed to the model. Raw data as input can barely
account for the best achievable performance of the model due to possible pre-
existing noise and inconsistency in the images. Therefore, a definite flow of
preprocessing is essential to train the model better [15].

We now discuss various preprocessing and augmentation techniques [41, 15]
applied to the raw fundus photographs for better learning. In a dataset, the
fundus images may be of various sizes; therefore, we resize the image $I$ into
$256\times 256$ sized image $I_{z}$. We perform data augmentations on training
set (DBtr), where we use centre cropping with central_fraction = 0.5,
horizontal/vertical flip, random rotations within a range of $[0^{o},45^{o}]$,
random brightness-change with max_delta = 0.95, random contrast-change in the
interval $[0.1,0.9]$. We also apply CLAHE (Contrast Limited Adaptive Histogram
Equalization) [42] on 30% samples of DBtr, which ensures over-amplification of
contrast in a smaller region instead of the entire image.

####  3.2.2 Transformer Networks

Deep learning models in computer vision tasks have long been dominated by CNN
(Convolutional Neural Network) to extract high-level feature maps by passing
the image through a series of convolution operations before feeding into the
MLP (Multi-Layer Perceptron) for classification [43]. In recent days,
transformer models have shown a substantial rise in the NLP (Natural Language
Processing) domain due to its higher performances [20]. In a similar quest to
leverage high-level performance through transformers, it has been introduced
in image classification and some other computer vision-oriented tasks [19].
Moreover, the transformer model has lesser image-specific inductive bias than
CNN [19].

To identify the severity stages of DR from fundus images, here we efficiently
adopt and ensemble some image transformers, e.g., ViT [19], BEiT [23], CaiT
[21], and DeiT [22]. Before focusing on our ensembled transformer model, we
discuss the adaptation of individual image transformers for our task, and
start with ViT.

_a) Vision Transformer (ViT) :_ The ViT model adopts the idea of text-based
transformer models [44], where the idea is to take the input image as a series
of image patches instead of textual words, and then extract features to feed
it into an MLP [19].

The pictorial representation of ViT is presented in Fig. 2. Here, the input
image $I_{z}$ is converted into a sequence of flattened patches $x_{p}^{i}$
(for $i=1,2,\dots,n_{p}$), each with size $w_{p}\times w_{p}\times c_{p}$,
where $c_{p}$ denotes the number of channels of $I_{z}$. Here, $c_{p}=3$,
since $I_{z}$ is an RGB fundus image. In our task, $I_{z}$ is of size
$256\times 256$, and empirically, we choose $w_{p}=64$, which results
$n_{p}=(\frac{256}{64})^{2}=16$. Each patch $x_{p}^{i}$ is flattened further
and mapped to a $D$-dimensional latent vector (i.e., patch embedding $z_{0}$)
through transformer layers using a trainable linear projection, as below.

$z_{0}=[x_{class}~{};~{}x_{p}^{1}\operatorname{\mathbb{E}}~{};~{}x_{p}^{2}\operatorname{\mathbb{E}}~{};~{}\dots~{};~{}x_{p}^{n_{p}}\operatorname{\mathbb{E}}]+\operatorname{\mathbb{E}}_{pos}$
(1)

where, $\operatorname{\mathbb{E}}$ is the patch embedding projection,
$\operatorname{\mathbb{E}}\in\mathbb{R}^{w_{p}\times w_{p}\times C\times D}$;
$\operatorname{\mathbb{E}}_{pos}$ is the position embeddings added to patch
embeddings to preserve the positional information of patches,
$\operatorname{\mathbb{E}}_{pos}\in\mathbb{R}^{(n_{p}+1)\times D}$;
$x_{class}=z_{0}^{0}$ is a learnable embedding [24].

Figure 2: Workflow of ViT. Figure 3: Internal view of a transformer encoder
(TE).

After mapping patch images to the embedding space with positional information,
we add a sequence of transformer encoders [45, 19]. The internal view of a
transformer encoder can be seen in Fig. 3, which includes two blocks $A_{s}$
and $F_{n}$. The $A_{s}$ and $F_{n}$ contain $MSA$ (Multi-head Self-Attention)
[19] and $MLP$ [15] modules, respectively. $LN$ (Layer Normalization) [46] and
residual connection [15] are employed before and after each of these modules,
respectively. This is shown in equation 2 with general semantics. Here, the
$MLP$ module comprises two layers having $4D$ and $D$ neurons with GELU
(Gaussian Error Linear Unit) non-linear activation function similar to [19].

$\begin{split}z_{l}^{\prime}=MSA(LN(z_{l-1}))+z_{l-1};\\\
z_{l}=MLP(LN(z_{l}^{\prime}))+z_{l}^{\prime};~{}l=1,2,\dots,L\end{split}$ (2)

where, $L$ is the total number of transformer blocks. The core component of
the transformer encoder is $MSA$ with $h$ heads, where each head includes $SA$
(Scaled dot-product Attention) [19, 45]. Each head $i\in\\{1,2,...,h\\}$ of
$MSA$ calculates a tuple comprising query, key, and value [19], i.e.,
($Q^{i},K^{i},V^{i}$) as follows.

$Q^{i}=XW_{Q}^{i}~{};~{}K^{i}=XW_{K}^{i}~{};~{}V^{i}=XW_{V}^{i}$ (3)

where, $X$ is the input embedding, and $W_{Q}$, $W_{K}$, $W_{V}$ are the
weight matrices used in the linear transformation. The tuple ($Q,K,V$) is fed
to $SA$ that computes the attention required to pay to the input image
patches, as below.

$SA(Q,K,V)=\psi\left(\frac{QK^{T}}{\sqrt{D_{h}}}\right)V$ (4)

where, $\psi$ is softmax function, and $D_{h}=D/h$. The outcomes of $SA$s
across all heads are concatenated in $MSA$, as follows.

$MSA(Q,K,V)=[SA^{1}~{};~{}SA^{2}~{};~{}\dots~{};~{}SA^{h}]W_{L}$ (5)

where $W_{L}$ is a weight matrix.

After multiple transformer encoder blocks, the <class> token [24] enriches
with the contextual information. The state of the learnable embedding at the
outcome of the Transformer encoder ($z_{L}^{0}$) acts as the image
representation $y$ [19].

$y=LN(z_{L}^{0})$ (6)

Now, as shown in Fig. 2, we add an MLP head containing a hidden layer with 128
neurons. To capture the non-linearity, we use Mish [47] here. In the output
layer, we keep five neurons with softmax activation function to obtain
probability distribution $s^{(j)}$ in order to classify a fundus photograph
into the abovementioned five severity stages of DR.

_b) Data efficient image Transformers (DeiT) :_ For a lower amount of training
data, ViT does not generalize well. In this scenario, DeiT can perform
reasonably well and uses lower memory [22]. DeiT adopts the ViT-specific
strategy and merges with the teacher-student scheme through knowledge
distillation [48]. The crux of DeiT is the knowledge distillation mechanism,
which is basically the knowledge transfer from one model (teacher) to another
(student) [22]. Here, we use EfficientNet-B5 [37] as a teacher model that is
trained apriori. The student model uses a transformer, which learns from the
outcome of the teacher model through attention depending on a distillation
token [22]. In this work, we employ hard-label distillation [22], where the
hard decision of the teacher is considered as a true label, i.e.,
$y_{t}=\text{argmax}_{c}Z_{t}(c)$. The hard-label distillation objective is
defined as follows.

${\mathcal{L}}_{global}^{hard}=0.5~{}{\mathcal{L}}_{CE}(\psi(Z_{s}),y)+0.5~{}{\mathcal{L}}_{CE}(\psi(Z_{s}),y_{t})$
(7)

where, ${\mathcal{L}}_{CE}$ is the cross-entropy loss on ground-truth labels
$y$, $\psi$ is the softmax function, $Z_{s}$ and $Z_{t}$ are the student and
teacher models' logits, respectively. Using label smoothing, hard labels can
be converted into soft ones [22].

In Fig. 4, we present the distillation procedure of DeiT. Here, we add the
<distillation> token to the transformer, which interacts with the <class> and
<patch> tokens through transformer encoders. The transformer encoder used here
is similar to the ViT's one, which includes $A_{s}$ and $F_{n}$ blocks as
shown in Fig. 3. The objective of the <distillation> token is to reproduce the
teacher's predicted label instead of the ground-truth label. The
<distillation> and <class> tokens are learned by back-propagation [15].

A linear classifier is used in DeiT instead of the MLP head of ViT [22, 19] to
work efficiently with limited computational resources.

Figure 4: The distillation procedure of DeiT.

_c) Class-attention in image Transformers (CaiT) :_ CaiT usually performs
better than ViT and DeiT with lesser FLOPs and learning parameters [15], when
we need to increase the depth of the transformer [21]. CaiT is basically an
upgraded version of ViT, which leverages layers with specific class-attention
and LayerScale [21]. In Fig. 5, we show the workflow of CaiT.

Figure 5: Workflow of CaiT.

LayerScale aids CaiT to work at larger depths, where we separately multiply a
diagonal matrix $M_{\lambda}$ on the outputs of $A_{s}$ and $F_{n}$ blocks.

$\begin{split}z_{l}^{\prime}=M_{\lambda}(\lambda_{1}^{l},\dots,\lambda_{D}^{l})\times
MSA(LN(z_{l-1}))+z_{l-1};\\\
z_{l}=M_{\lambda}({\lambda^{\prime}}_{1}^{l},\dots,{\lambda^{\prime}}_{D}^{l})\times
MLP(LN(z_{l}^{\prime}))+z_{l}^{\prime}\end{split}$ (8)

where, $\lambda_{i}^{l}$ and ${\lambda^{\prime}}_{i}^{l}$ are learning
parameters, and other symbols denote the same as the above-mentioned ViT.

In CaiT, the transformer layers dealing with self-attention among patches are
separated from class-attention layers that are introduced to dedicatedly
extract the content of the patches into a vector, which can be sent to a
linear classifier [21]. The <class> token is inserted in the latter stage, so
that the initial layers can perform the self-attention among patches
devotedly. In the class-attention stage, we alternatively use multi-head
class-attention ($A_{c}$) [21] and $F_{n}$, as shown in Fig. 5, and update
only the class embedding.

_d) Bidirectional Encoder representation for image Transformer (BEiT) :_ BEiT
is a self-supervised model having its root in the BERT (Bidirectional Encoder
Representations from Transformers) [24, 23]. In Fig. 6, we present the
workflow of the pre-training of BEiT.

The input image $I_{z}$ is split into patches $x_{p}^{i}$ and flattened into
vectors, similar to the early-mentioned ViT. In BEiT, a backbone transformer
is engaged, for which we use ViT [19]. On the other hand, $I_{z}$ is
represented as a sequence of visual tokens
$vt=[vt_{1},vt_{2},\dots,vt_{n_{p}}]$ obtained by a discrete VAE (Variational
Auto-Encoder) [49]. For visual token learning, we employ a tokenizer
$\mathcal{T}_{\phi}(vt~{}|~{}x)$ to map image pixels $x$ to tokens $vt$, and
decoder $\mathcal{D}_{\theta}(x~{}|~{}vt)$ for reconstructing input image
pixels $x$ from $vt$ [23].

Figure 6: Workflow of BEiT pre-training.

Here, a MIM (Masked Image Modeling) [23] task is performed to pre-train the
image transformers, where some image patches are randomly masked, and the
corresponding visual tokens are then predicted. The masked patches are
replaced with a learnable embedding $e_{[M]}$. We feed the corrupted image
patches
$x^{\mathcal{M}}=\\{x_{p}^{i}:i\notin\mathcal{M}\\}\bigcup\\{e_{[M]}:i\in\mathcal{M}\\}$
to the transformer encoder. Here, $\mathcal{M}$ is the set of indices of
masked positions.

The encoded representation $h_{i}^{L}$ is the hidden vector of the last
transformer layer $L$ for $i^{th}$ patch. For each masked position, a softmax
classifier $\psi$ is used to predict the respective visual token, i.e.,
$p_{\text{MIM}}(vt^{\prime}~{}|~{}x^{\mathcal{M}})=\psi(W_{M}h_{i}^{L}+b_{M})$;
where, $W_{M}$ and $b_{M}$ contain learning parameters for linear
transformation. The pre-training objective of BEiT is to maximize the log-
likelihood of the correct token $vt_{i}$ given $x^{\mathcal{M}}$, as below:

$max\sum_{x\in~{}\text{DB}_{tr}}\mathds{E}_{\mathcal{M}}\left[\sum_{i\in\mathcal{M}}\text{log}~{}p_{\text{MIM}}\left(vt_{i}~{}|~{}x^{\mathcal{M}}\right)\right]$

where, $\text{DB}_{tr}$ is the training dataset. The BEiT pre-training can be
perceived as VAE training [23, 49], where we follow two stages, i.e.,
_stage-1_ : minimizing loss for visual token reconstruction, _stage-2_ :
modeling masked image, i.e., learning prior $p_{\text{MIM}}$ by keeping
$\mathcal{T}_{\phi}$ and $\mathcal{D}_{\theta}$ fixed. It can be written as
follows:

$\begin{split}\sum_{\begin{subarray}{c}(x_{i},x_{i}^{\mathcal{M}})\\\
\in~{}\text{DB}_{tr}\end{subarray}}\left(\underbrace{\mathds{E}_{vt_{i}\sim\mathcal{T}_{\phi}(vt|x_{i})}\left[\text{log}~{}\mathcal{D}_{\theta}(x_{i}|vt_{i})\right]}_{\text{\emph{stage-1}}}+\underbrace{\text{log}~{}p_{\text{MIM}}\left(\hat{vt_{i}}|x_{i}^{\mathcal{M}}\right)}_{\text{\emph{stage-2}}}\right)\end{split}$

where, $\hat{vt_{i}}=\text{argmax}_{vt}~{}\mathcal{T}_{\phi}(vt~{}|~{}x_{i})$.

####  3.2.3 Ensembled Transformers

The abovementioned four image transformers, i.e., ViT [19], DeiT [22], CaiT
[21], and BEiT [23] are pre-trained on the training set DBtr. We now ensemble
the transformers for predicting the severity stages from fundus images of the
test set DBt, since ensembling multiple learning algorithms can achieve better
performance than the constituent algorithms alone [50]. The pictorial
representation of ensembled transformers is presented in Fig. 7.

Figure 7: Ensembled transformers.

For an image sample from DBt, we obtain the softmax probability distribution
$s^{(j)}:\\{P_{1}^{j},P_{2}^{j},\dots,P_{n_{c}}^{j}\\}$ over $j^{th}$
transformer [15], for $j=1,2,\dots,n_{T}$; where, $n_{c}$ is the total number
of classes (severity stages), and $n_{T}$ is count of the employed image
transformers. Here, $\sum_{i=1}^{n_{c}}P_{i}^{j}=1$, $n_{c}=5$ (refer to
$\mathsection$ 3.1), and $n_{T}=4$ since we use four separately trained
distinct image transformers, as mentioned earlier.

We obtain the severity stages/ class_labels $|_{wm}$ and $|_{mv}$ separately
using two combination methods _weighted mean_ and _majority voting_ [50],
respectively.

$\begin{split}\text{}|_{wm}=\text{argmax}_{i}~{}P_{i}^{\mu}~{};~{}\text{for
}i=1,2,\dots,n_{c}~{};\\\
P_{i}^{\mu}=\frac{\sum_{j=1}^{n_{T}}{\alpha_{j}P_{i}^{j}}}{\sum_{j=1}^{n_{T}}\alpha_{j}}\end{split}$
(9)

In this task, we choose $\sum_{j=1}^{n_{T}}\alpha_{j}=1$.

$\footnotesize\begin{split}\text{}|_{mv}=\text{mode}\left(\text{argmax}_{i}(P_{i}^{1}),\text{argmax}_{i}(P_{i}^{2}),\dots,\text{argmax}_{i}(P_{i}^{n_{T}})\right)\\\
=\text{mode}\left(\text{argmax}_{i}(s^{(1)}),\text{argmax}_{i}(s^{(2)}),\dots,\text{argmax}_{i}(s^{(n_{T})})\right);\\\
\text{for }i=1,2,\dots,n_{c}\end{split}$ (10)

In this task, we use cross-entropy as the loss function [41] in the employed
image transformers. The AdamW optimizer is used here due to its weight decay
regularization effect for tackling overfitting [51]. The training details with
hyper-parameter tuning are mentioned in Section 4.2.

##  4 Experiments and Discussions

In this section, we present the employed database, followed by experimental
results with discussions.

###  4.1 Database Employed

For our computational experiments, we used the publicly available training
samples of Kaggle APTOS (_Asia Pacific Tele-Ophthalmology Society_) 2019
Blindness Detection dataset [7], i.e., APTOS-2019. This database (DB) contains
fundus image samples of five severity stages of DR, i.e., negative, mild,
moderate, severe, and proliferative. Fig. 1 shows some sample images from this
dataset. In DB, a total of 3662 fundus images are available, which we divide
into training (DBtr) and testing (DBt) datasets with a ratio of $7:3$. As a
matter of fact, DBtr and DBt sets are disjoint. The sample counts of different
severity stages/ class_labels () for DBtr and DBt are shown in Fig. 8
individually. Here, 49.3% samples are of negative DR (= 0). Among positive
classes, most samples are from the moderate stage (= 2). From this figure, it
can be seen DB is imbalanced due to containing a different number of samples
corresponding to various severity stages. Therefore, we augmented the data
during the training of our model as mentioned in $\mathsection$ 3.2.1. The
data augmentation also helped in reducing the overfitting issue [15].

Figure 8: Count of samples in APTOS-2019 [7].

###  4.2 Experimental Results

This section discusses the performed experiments, analyzes the model outcome,
and compares them with major state-of-the-art methods. We begin with
discussing the experimental settings.

####  4.2.1 Experiment Settings

We performed the experiments on the TensorFlow-2 framework having Python
3.7.13 over a machine with the following configurations: Intel(R) Xeon(R) CPU
@ 2.00GHz with 52 GB RAM and Tesla T4 16 GB GPU. All the results shown here
were obtained from DBt.

The hyper-parameters of the framework were tuned and fixed during training
with respect to the performance over some samples of DBt employed for hyper-
parameter tuning. For all the image transformers used here (i.e., ViT, DeiT,
CaiT, and BEiT), we empirically set the following hyper-parameters:
transformer_layers ($L$) = 12, embedding_dimension ($D$) = 384, num_heads
($h$) = 6. The following hyper-parameters were selected for AdamW [51]:
initial_learning_rate = $10^{-3}$; exponential decay rates for 1st and 2nd
moment estimates, i.e., $\beta 1=0.9$, $\beta 2=0.999$; zero-denominator
removal parameter ($\upvarepsilon$) = $10^{-8}$; and weight_decay =
$10^{-3}/4$. For model training, the mini-batch size was fixed to 32.

####  4.2.2 Model Performance

In Table 1, we present the performance of our ensembled image transformer
($EiT$) using the combination schemes weighted mean ($wm$) and majority voting
($mv$), where we obtain 94.63% and 91.26% accuracy from $EiT_{wm}$ and
$EiT_{mv}$, respectively. We also ensembled multiple combinations of our
employed transformers, and present their performances in this table. Here, the
$wm$ scheme performed better than $mv$. As evident from this table, ensembling
various types of transformers improved the performance. Among single
transformers (for $n_{T}=1$), CaiT performed the best. For $n_{T}=2$ and
$n_{T}=3$, “BEiT + CaiT” and “DeiT + BEiT + CaiT” performed better than other
respective combinations. Overall, $EiT_{wm}$ attained the best accuracy here.

Table 1: Performance over various ensembling of transformers $n_{T}$ | Ensembled Transformers | Accuracy (%)
---|---|---
Weighted | Majority
mean | voting
1 | ViT | 82.21
DeiT | 85.65
BEiT | 86.74
CaiT | 86.91
2 | ViT + DeiT | 87.03 | 86.55
ViT + BEiT | 87.48 | 87.03
ViT + CaiT | 87.77 | 87.21
DeiT + BEiT | 88.18 | 87.69
DeiT + CaiT | 88.86 | 87.93
BEiT + CaiT | 89.28 | 88.12
3 | ViT + DeiT + BEiT | 90.53 | 88.87
ViT + DeiT + CaiT | 91.39 | 89.56
ViT + BEiT + CaiT | 92.14 | 90.28
DeiT + BEiT + CaiT | 93.46 | 90.91
4 | ViT + DeiT + BEiT + CaiT | 94.63 | 91.26
( $EiT$ )

In Fig. 10 of Appendix 6, we present the coarse localization maps generated by
Grad-CAM [52] from the employed individual image transformers to highlight the
crucial regions for understanding the severity stages.

_a) Various Evaluation Metrics :_ Besides the accuracy, in Table 2, we present
the performance of $EiT$ with respect to some other evaluation metrics, e.g.,
kappa score, precision, recall, F1 score, specificity, balanced accuracy [53].
Here, Cohen’s quadratic weighted kappa measures the agreement between human-
assigned scores (i.e., DR severity stages) and the $EiT$-predicted scores.
Precision analyzes the true positive samples among the total positive
predictions. Recall or sensitivity finds the true positive rate. Similarly,
specificity computes the true negative rate. F1 score is the harmonic mean of
precision and recall. Since the employed DB is imbalanced, we also compute the
balanced accuracy, which is the arithmetic mean of sensitivity and
specificity. In this table, we can see that for both $EiT_{wm}$ and
$EiT_{mv}$, the kappa scores are greater than 0.81, which comprehends the
“almost perfect agreement” between the human rater and $EiT$ [53]. Here,
_macro_ means the arithmetic mean of all per class precision/ recall/ F1
score.

Table 2: Performance of $EiT$ over various evaluation metrics Metric | Weighted mean | Majority voting
---|---|---
($EiT_{wm}$) |  ($EiT_{mv}$)
Accuracy (%) | 94.63 | 91.26
Kappa score | 0.92 | 0.87
Macro Precision (%) | 90.55 | 84.65
Macro Recall (%) | 92.88 | 88.81
Macro F1-score (%) | 91.67 | 86.55
Macro Specificity (%) | 98.62 | 97.74
Balanced Accuracy (%) | 95.75 | 93.27

_b) Individual Class Performance :_ Table 3 presents the individual
performance of $EiT_{wm}$ and $EiT_{mv}$ for detecting every severity stage of
DR. From this table, we can see our models produced the best precision and
recall for negative DR (= 0), and the lowest for severe DR (= 3).

Table 3: Performance of $EiT$ on every DR severity stage | class_label () | 0 | 1 | 2 | 3 | 4
---|---|---|---|---|---|---
$EiT_{wm}$ | Precision (%) | 98.48 | 86.67 | _95.00_ | 83.61 | 89.01
Recall (%) | 95.75 | 93.69 | _95.00_ | 87.93 | 92.05
F1-score (%) | 97.09 | 90.04 | _95.00_ | 85.71 | 90.50
Specificity (%) | 98.56 | 98.38 | 98.12 | 99.04 | _99.01_
$EiT_{mv}$ | Precision (%) | 96.74 | 79.67 | _94.14_ | 70.59 | 82.11
Recall (%) | 93.35 | 88.29 | _91.00_ | 82.76 | 88.64
F1-score (%) | 95.01 | 83.76 | _92.54_ | 76.19 | 85.25
Specificity (%) | 96.95 | 97.47 | 97.87 | _98.08_ | 98.32
In each row, the best result is marked bold, second-best is _italic_ , and
lowest is underlined.

####  4.2.3 Comparison

In Table 4, we present a comparative analysis with some major contemporary
deep learning architectures, e.g., ResNet50 [54], InceptionV3 [55],
MobileNetV2 [56], Xception [57], DenseNet169 (Farag et al. [18]), EfficientNet
[37], and SE-ResNeXt50 [38]. We have also compared with recently published
transformer-based models, i.e., CoT-XNet [39], and SSiT [40]. Comparison with
some major related works [35, 36, 5] can also be seen in this table. Our
$EiT_{wm}$ outperformed the major state-of-the-art methods with respect to
accuracy, balanced accuracy, sensitivity, and specificity. Our $EiT_{mv}$ also
performed quite well in terms of balanced accuracy.

Table 4: Comparative study Method | Accuracy | Sensitivity | Specificity | Balanced
---|---|---|---|---
(%) | (%) | (%) | Accuracy (%)
ResNet50 [54] | 74.64 | 56.52 | 85.71 | 71.12
InceptionV3 [55] | 78.72 | 63.64 | 85.37 | 74.51
MobileNetV2 [56] | 79.01 | 76.47 | 84.62 | 80.55
Xception [57] | 79.59 | 82.35 | 86.32 | 84.34
Farag et al. [18] | 82.00 | - | - | -
Kassani et al. [35] | 83.09 | 88.24 | 87.00 | 87.62
TAN [36] | 85.10 | 90.30 | 92.00 | -
EfficientNet-B4 [37] | 90.30 | 81.20 | 97.60 | 89.40
EfficientNet-B5 [37] | 90.70 | 80.70 | 97.70 | 89.20
SE-ResNeXt50 [38] | 92.40 | 87.10 | 98.20 | 92.65
Tymchenko et al. [5] | 92.90 | 86.00 | 98.30 | 92.15
CoT-XNet [39] | 84.18 | - | 95.74 | -
SSiT [40] | 92.97 | - | - | -
$EiT_{mv}$ [ours] | 91.26 | 88.81 | 97.74 | 93.28
$EiT_{wm}$ [ours] | 94.63 | 92.88 | 98.62 | 95.75
In each column, the best result is marked bold, and the second-best is
underlined.

####  4.2.4 Impact of Hyper-parameters

We tuned the hyper-parameters and observed their impact on the experiment.

_a) MSA Head Count :_ We analyzed the performance impact of the number of
heads ($h$) of MSA (Multi-head Self-Attention) in the transformer encoder and
present in Fig. 9. As evident from this figure, the performance (accuracy) of
both $EiT_{mv}$ and $EiT_{wm}$ increased with the increment of $h$ till $h=6$,
and started decreasing thereafter.

Figure 9: Impact of number of heads ($h$) in MSA on model performance.

_b) Weights $\alpha_{j}$ of $EiT_{wm}$:_ We tuned the weights $\alpha_{j}$
(refer to Eqn. 9) to see its impact on the performance of $EiT_{wm}$. We
obtained the best accuracy of 94.63% from $EiT_{wm}$ for
$\alpha_{1}=\alpha_{2}=0.1$, and $\alpha_{3}=\alpha_{4}=0.4$. The performance
of $EiT_{wm}$ during tuning of $\alpha_{j}$'s is shown in Table 5.

In Table 6, we also present the tuned $\alpha_{j}$'s that aided in obtaining
the best performing ensembled transformers of Table 1.

Table 5: Performance of $EiT_{wm}$ by tuning weights $\alpha_{j}$ $\alpha_{1}$ | $\alpha_{2}$ | $\alpha_{3}$ | $\alpha_{4}$ | Accuracy (%)
---|---|---|---|---
0.25 | 0.25 | 0.25 | 0.25 | 89.53
0.85 | 0.05 | 0.05 | 0.05 | 82.29
0.05 | 0.85 | 0.05 | 0.05 | 85.78
0.05 | 0.05 | 0.85 | 0.05 | 86.92
0.05 | 0.05 | 0.05 | 0.85 | 87.05
0.7 | 0.1 | 0.1 | 0.1 | 82.35
0.1 | 0.7 | 0.1 | 0.1 | 85.91
0.1 | 0.1 | 0.7 | 0.1 | 87.04
0.1 | 0.1 | 0.1 | 0.7 | 87.20
0.5 | 0.167 | 0.167 | 0.166 | 82.88
0.166 | 0.5 | 0.167 | 0.167 | 86.35
0.167 | 0.166 | 0.5 | 0.167 | 87.62
0.167 | 0.167 | 0.166 | 0.5 | 87.74
0.3 | 0.3 | 0.2 | 0.2 | 88.16
0.3 | 0.2 | 0.3 | 0.2 | 89.58
0.3 | 0.2 | 0.2 | 0.3 | 90.27
0.2 | 0.3 | 0.3 | 0.2 | 90.85
0.2 | 0.3 | 0.2 | 0.3 | 91.67
0.2 | 0.2 | 0.3 | 0.3 | 92.72
0.4 | 0.4 | 0.1 | 0.1 | 91.18
0.4 | 0.1 | 0.4 | 0.1 | 91.49
0.4 | 0.1 | 0.1 | 0.4 | 92.15
0.1 | 0.4 | 0.4 | 0.1 | 92.84
0.1 | 0.4 | 0.1 | 0.4 | 93.47
0.1 | 0.1 | 0.4 | 0.4 | 94.63
Table 6: Tuned weights $\alpha_{j}$ for transformers ensembled with weighted mean Transformerswm | $\alpha_{1}$ | $\alpha_{2}$ | $\alpha_{3}$ | $\alpha_{4}$
---|---|---|---|---
ViT + DeiT | 0.25 | 0.75 | - | -
ViT + BEiT | 0.4 | 0.6 | - | -
ViT + CaiT | 0.4 | 0.6 | - | -
DeiT + BEiT | 0.4 | 0.6 | - | -
DeiT + CaiT | 0.3 | 0.7 | - | -
BEiT + CaiT | 0.5 | 0.5 | - | -
ViT + DeiT + BEiT | 0.2 | 0.3 | 0.5 | -
ViT + DeiT + CaiT | 0.2 | 0.3 | 0.5 | -
ViT + BEiT + CaiT | 0.2 | 0.4 | 0.4 | -
DeiT + BEiT + CaiT | 0.3 | 0.3 | 0.4 | -
ViT + DeiT + BEiT + CaiT | 0.1 | 0.1 | 0.4 | 0.4

####  4.2.5 Ablation Study

We here present the performed ablation study by ablating individual
transformers. Our $EiT$ is actually an ensembling of four different image
transformers, i.e., ViT, DeiT, CaiT, and BEiT. We ablated each transformer and
observed performance degradation than $EiT$. For example, considering the
weighted mean scheme, when we ablated CaiT from $EiT$, the accuracy dropped by
4.1%. Similarly, ablating BEiT and CaiT deteriorated the accuracy by 7.6%. For
our task, the best individual transformer (CaiT) attained 7.72% lower accuracy
than $EiT_{wm}$. More examples can be observed in Table 1.

####  4.2.6 Pre-training with Other Datasets

We checked the performance of our $EiT$ model by pre-training with some other
dataset. We took 1200 images of MESSIDOR [58] with adjudicated grades by [59]
(say, DBM). From IDRiD [60], we also used “Disease Grading” dataset containing
516 images (say, DBI). Here, we made four training set setups from DBM, by
taking 25%, 50%, 75%, and 100% of samples of DBM. Similarly, four training
setups were generated from DBI. As mentioned in $\mathsection$ 4.1, we divided
APTOS-2019 database (DB) in training (DBtr) and test (DBt) sets with a ratio
of $7:3$. In Table 7, we present the performance of $EiT$ on DBt, while pre-
training with DBM and DBI, and training with DBtr. It can be observed that the
performance of $EiT$ improved slightly when pre-trained with more data from
other datasets.

Table 7: Accuracy (%) of $EiT$ with pre-training | Pre-training data | 25% | 50% | 75% | 100%
---|---|---|---|---|---
$EiT_{wm}$ |  DBM | 94.71 | 94.78 | 94.83 | 94.88
DBI | 94.65 | 94.67 | 94.7 | 94.79
DBM \+ DBI | 94.73 | 94.85 | 94.98 | 95.13
_N.A._ | 94.63
$EiT_{mv}$ |  DBM | 91.35 | 91.48 | 91.56 | 91.61
DBI | 91.27 | 91.32 | 91.34 | 91.35
DBM \+ DBI | 91.42 | 91.6 | 91.68 | 91.75
_N.A._ | 91.26
_N.A._ : without pre-training data

##  5 Conclusion

In this paper, we tackle the problem of automated severity stage detection of
DR from fundus images. For this purpose, we propose two ensembled image
transformers, $EiT_{wm}$ and $EiT_{mv}$, by using weighted mean and majority
voting combination schemes, respectively. We here adopt four transformer
models, i.e., ViT, DeiT, CaiT, and BEiT. For experimentation, we employed the
publicly available APTOS-2019 blindness detection dataset, on which $EiT_{wm}$
and $EiT_{mv}$ attained accuracies of 94.63% and 91.26%, respectively.
Although the employed dataset was imbalanced, our models performed quite well.
Our $EiT_{wm}$ outperformed the major state-of-the-art techniques. We also
performed an ablation study and observed the importance of the ensembling over
the individual transformers.

In the future, we will endeavor to improve the model performance with some
imbalanced learning techniques. Currently, our model does not perform any
lesion segmentation, which we will also attempt to explore some implicit
characteristics of fundus images due to DR.

##  6 Qualitative Visualization

As mentioned in $\mathsection$ 4.2.2, we present the Grad-CAM maps of the
employed individual image transformers in Fig. 10.

negative mild moderate severe proliferative

Figure 10: Fundus images (1st row) with Grad-CAM maps for ViT, DeiT, BEiT,
CaiT as shown in 2nd, 3rd, 4th, 5th rows, respectively.

## References

  * [1] S. Stolte and R. Fang, ``A survey on medical image analysis in diabetic retinopathy,'' _Medical image analysis_ , vol. 64, p. 101742, 2020.
  * [2] N. Asiri _et al._ , ``Deep learning based computer-aided diagnosis systems for diabetic retinopathy: A survey,'' _Artificial intelligence in medicine_ , vol. 99, p. 101701, 2019.
  * [3] Y. Zheng _et al._ , ``The worldwide epidemic of diabetic retinopathy,'' _Indian journal of ophthalmology_ , vol. 60, no. 5, p. 428, 2012.
  * [4] D. S. W. Ting _et al._ , ``Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes,'' _JAMA_ , vol. 318, no. 22, pp. 2211–2223, 2017.
  * [5] B. Tymchenko _et al._ , ``Deep learning approach to diabetic retinopathy detection,'' in _ICPRAM_ , 2020, pp. 501–509.
  * [6] J. Krause _et al._ , ``Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy,'' _Ophthalmology_ , vol. 125, no. 8, pp. 1264–1272, 2018.
  * [7] APTOS 2019 Blindness Detection. [Online]. Available: https://www.kaggle.com/competitions/aptos2019-blindness-detection
  * [8] G. Quellec _et al._ , ``Optimal wavelet transform for the detection of microaneurysms in retina photographs,'' _IEEE transactions on medical imaging_ , vol. 27, no. 9, pp. 1230–1241, 2008.
  * [9] L. Giancardo _et al._ , ``Microaneurysm detection with radon transform-based classification on retina images,'' in _EMBS_ , 2011, pp. 5939–5942.
  * [10] R. Casanova _et al._ , ``Application of random forests methods to diabetic retinopathy classification analyses,'' _PLOS one_ , vol. 9, no. 6, p. e98587, 2014.
  * [11] M. Niemeijer _et al._ , ``Automated detection and differentiation of drusen, exudates, and cotton-wool spots in digital color fundus photographs for diabetic retinopathy diagnosis,'' _Investigative ophthalmology & visual science_, vol. 48, no. 5, pp. 2260–2267, 2007.
  * [12] M. U. Akram _et al._ , ``Identification and classification of microaneurysms for early detection of diabetic retinopathy,'' _Pattern recognition_ , vol. 46, no. 1, pp. 107–116, 2013.
  * [13] D. Usher _et al._ , ``Automated detection of diabetic retinopathy in digital retinal images: a tool for diabetic retinopathy screening,'' _Diabetic Medicine_ , vol. 21, no. 1, pp. 84–90, 2004.
  * [14] J. D. Bodapati _et al._ , ``Deep convolution feature aggregation: an application to diabetic retinopathy severity level prediction,'' _Signal, Image and Video Processing_ , vol. 15, no. 5, pp. 923–930, 2021\.
  * [15] A. Zhang, Z. C. Lipton, M. Li, and A. J. Smola, ``Dive into deep learning,'' _arXiv:2106.11342_ , 2021.
  * [16] S. Yu _et al._ , ``Exudate detection for diabetic retinopathy with convolutional neural networks,'' in _EMBC_ , 2017, pp. 1744–1747.
  * [17] Z. Wang _et al._ , ``Zoom-in-net: Deep mining lesions for diabetic retinopathy detection,'' in _MICCAI_ , 2017, pp. 267–275.
  * [18] M. M. Farag _et al._ , ``Automatic severity classification of diabetic retinopathy based on densenet and convolutional block attention module,'' _IEEE Access_ , vol. 10, pp. 38 299–38 308, 2022.
  * [19] A. Dosovitskiy _et al._ , ``An image is worth 16x16 words: Transformers for image recognition at scale,'' _arXiv:2010.11929_ , 2020.
  * [20] A. M. Braşoveanu and R. Andonie, ``Visualizing transformers for nlp: a brief survey,'' in _Int. Conf. on Information Visualisation (IV)_. IEEE, 2020, pp. 270–279.
  * [21] H. Touvron _et al._ , ``Going deeper with image transformers,'' in _ICCV_ , 2021, pp. 32–42.
  * [22] ——, ``Training data-efficient image transformers & distillation through attention,'' in _ICML_ , 2021, pp. 10 347–10 357.
  * [23] H. Bao, L. Dong, and F. Wei, ``BEiT: BERT Pre-Training of Image Transformers,'' in _ICLR, arXiv:2106.08254_ , 2022.
  * [24] J. Devlin _et al._ , ``BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,'' _arXiv:1810.04805_ , 2018.
  * [25] ``Grading diabetic retinopathy from stereoscopic color fundus photographs—an extension of the modified airlie house classification: Etdrs report number 10,'' _Ophthalmology_ , vol. 98, no. 5, Supplement, pp. 786–806, 1991.
  * [26] U. R. Acharya _et al._ , ``Automated diabetic macular edema (dme) grading system using dwt, dct features and maculopathy index,'' _Computers in biology and medicine_ , vol. 84, pp. 59–68, 2017.
  * [27] K. M. Adal _et al._ , ``An automated system for the detection and classification of retinal changes due to red lesions in longitudinal fundus images,'' _IEEE transactions on biomedical engineering_ , vol. 65, no. 6, pp. 1382–1390, 2017.
  * [28] L. Tang _et al._ , ``Splat feature classification with application to retinal hemorrhage detection in fundus images,'' _IEEE Transactions on Medical Imaging_ , vol. 32, no. 2, pp. 364–375, 2013.
  * [29] A. Herliana _et al._ , ``Feature selection of diabetic retinopathy disease using particle swarm optimization and neural network,'' in _CITSM_. IEEE, 2018, pp. 1–4.
  * [30] S. Sanromà _et al._ , ``Assessment of diabetic retinopathy risk with random forests.'' in _ESANN_ , 2016.
  * [31] A. Osareh _et al._ , ``A computational-intelligence-based approach for detection of exudates in diabetic retinopathy images,'' _IEEE Transactions on Information Technology in Biomedicine_ , vol. 13, no. 4, pp. 535–545, 2009.
  * [32] P. Chudzik _et al._ , ``Microaneurysm detection using deep learning and interleaved freezing,'' in _Medical imaging 2018: image processing_ , vol. 10574. SPIE, 2018, pp. 379–387.
  * [33] R. Gargeya and T. Leng, ``Automated identification of diabetic retinopathy using deep learning,'' _Ophthalmology_ , vol. 124, no. 7, pp. 962–969, 2017\.
  * [34] S. Wan _et al._ , ``Deep convolutional neural networks for diabetic retinopathy detection by image classification,'' _Computers & Electrical Engineering_, vol. 72, pp. 274–282, 2018.
  * [35] S. H. Kassani _et al._ , ``Diabetic retinopathy classification using a modified xception architecture,'' in _ISSPIT_ , 2019, pp. 1–6.
  * [36] M. D. Alahmadi, ``Texture attention network for diabetic retinopathy classification,'' _IEEE Access_ , vol. 10, pp. 55 522–55 532, 2022.
  * [37] M. Tan and Q. Le, ``Efficientnet: Rethinking model scaling for convolutional neural networks,'' in _ICML_ , 2019, pp. 6105–6114.
  * [38] J. Hu _et al._ , ``Squeeze-and-excitation networks,'' in _CVPR_ , 2018, pp. 7132–7141.
  * [39] S. Zhao _et al._ , ``Cot-xnet: Contextual transformer with xception network for diabetic retinopathy grading,'' _Physics in Medicine & Biology_, 2022\.
  * [40] Y. Huang _et al._ , ``Ssit: Saliency-guided self-supervised image transformer for diabetic retinopathy grading,'' _arXiv:2210.10969_ , 2022\.
  * [41] I. Goodfellow _et al._ , _Deep learning_. MIT press, 2016.
  * [42] V. Stimper _et al._ , ``Multidimensional contrast limited adaptive histogram equalization,'' _IEEE Access_ , vol. 7, pp. 165 437–165 447, 2019\.
  * [43] D. Sarvamangala _et al._ , ``Convolutional neural networks in medical image understanding: a survey,'' _Evolutionary intel._ , pp. 1–22, 2021.
  * [44] T. Wolf _et al._ , ``Transformers: State-of-the-Art Natural Language Processing,'' in _EMNLP: System Demonstrations_ , 2020, pp. 38–45.
  * [45] A. Vaswani _et al._ , ``Attention is all you need,'' _Advances in neural information processing systems_ , vol. 30, 2017.
  * [46] J. L. Ba, J. R. Kiros, and G. E. Hinton, ``Layer normalization,'' _arXiv:1607.06450_ , 2016.
  * [47] D. Misra, ``Mish: A self regularized non-monotonic neural activation function,'' _BMVC, Paper #928_ , 2020.
  * [48] G. Hinton, O. Vinyals, and J. Dean, ``Distilling the Knowledge in a Neural Network,'' in _NIPS Deep Learning and Representation Learning Workshop, arXiv:1503.02531_ , 2015.
  * [49] A. Ramesh _et al._ , ``Zero-shot text-to-image generation,'' in _ICML_ , 2021, pp. 8821–8831.
  * [50] L. Rokach, ``Ensemble-based Classifiers,'' _Artificial intelligence review_ , vol. 33, no. 1, pp. 1–39, 2010.
  * [51] I. Loshchilov and F. Hutter, ``Decoupled weight decay regularization,'' _ICLR, arXiv:1711.05101_ , 2019.
  * [52] R. Selvaraju _et al._ , ``Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization,'' in _ICCV_ , 2017, pp. 618–626.
  * [53] M. Grandini, E. Bagli, and G. Visani, ``Metrics for multi-class classification: an overview,'' _arXiv preprint arXiv:2008.05756_ , 2020.
  * [54] K. He _et al._ , ``Deep residual learning for image recognition,'' in _CVPR_ , 2016, pp. 770–778.
  * [55] C. Szegedy _et al._ , ``Rethinking the inception architecture for computer vision,'' in _CVPR_ , 2016, pp. 2818–2826.
  * [56] M. Sandler _et al._ , ``Mobilenetv2: Inverted residuals and linear bottlenecks,'' in _CVPR_ , 2018, pp. 4510–4520.
  * [57] F. Chollet, ``Xception: Deep learning with depthwise separable convolutions,'' in _CVPR_ , 2017, pp. 1251–1258.
  * [58] E. Decencière _et al._ , ``Feedback on a publicly distributed image database: the Messidor database,'' _Image Analysis & Stereology_, vol. 33, no. 3, pp. 231–234, 2014.
  * [59] J. Krause _et al._ , ``Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy,'' _Ophthalmology_ , vol. 125, no. 8, pp. 1264–1272, 2018.
  * [60] P. Porwal _et al._ , ``Indian Diabetic Retinopathy Image Dataset (IDRiD),'' 2018. [Online]. Available: https://dx.doi.org/10.21227/H25W98