# FGA: Fourier-Guided Attention Network for Crowd Count Estimation

Yashwardhan Chaudhuri  IIIT-Delhi
<EMAIL_ADDRESS>Ankit Kumar  IIT-Bombay
<EMAIL_ADDRESS>Arun Balaji Buduru  IIIT-Delhi
<EMAIL_ADDRESS>Adel Alshamrani  University of Jeddah
<EMAIL_ADDRESS>
###### Abstract

Crowd counting is gaining societal relevance, particularly in domains of Urban
Planning, Crowd Management, and Public Safety. This paper introduces Fourier-
guided attention (FGA), a novel attention mechanism for crowd count estimation
designed to address the inefficient full-scale global pattern capture in
existing works on convolution-based attention networks. FGA efficiently
captures multi-scale information, including full-scale global patterns, by
utilizing Fast-Fourier Transformations (FFT) along with spatial attention for
global features and convolutions with channel-wise attention for semi-global
and local features. The architecture of FGA involves a dual-path approach: (1)
a path for processing full-scale global features through FFT, allowing for
efficient extraction of information in the frequency domain, and (2) a path
for processing remaining feature maps for semi-global and local features using
traditional convolutions and channel-wise attention. This dual-path
architecture enables FGA to seamlessly integrate frequency and spatial
information, enhancing its ability to capture diverse crowd patterns. We apply
FGA in the last layers of two popular crowd-counting works, CSRNet and CANNet,
to evaluate the module’s performance on benchmark datasets such as
ShanghaiTech-A, ShanghaiTech-B, UCF-CC-50, and JHU++ crowd. The experiments
demonstrate a notable improvement across all datasets based on Mean-Squared-
Error (MSE) and Mean-Absolute-Error (MAE) metrics, showing comparable
performance to recent state-of-the-art methods. Additionally, we illustrate
the interpretability using qualitative analysis, leveraging Grad-CAM heatmaps,
to show the effectiveness of FGA in capturing crowd patterns.

###### Index Terms:

Crowd Count Estimation, Fast Fourier Transformation, Attention, Channel
Attention, Spatial Attention, CNN

##  I Introduction

Crowd count estimation (or crowd counting) involves estimating the number of
individuals in a particular crowd scene. Its utility in public safety, crowd
management, urban planning, and healthcare makes it relevant in computer
vision. Crowd counting becomes challenging in large crowd scenes where
problems such as large foreground-background imbalances, occlusions, and
perspective distortion arise frequently. Density-based crowd-counting
methods[1][2][3][4] is widely accepted as a solution to this problem. It
involves predicting crowd density maps (Figure 1), where the sum of pixel
values in a density map gives the number of people in it. Crowd counting has
progressed significantly since the introduction of deep learning, with works
such as MCNN[3], CSRNet[4], OURS-CAN[5], ASPDNet[2] being widely accepted as a
possible solution.

Figure 1: Left: Image of a crowd scene as input to the neural network. Right:
Density map of The crowd scene. Brighter spots are noticed in the top right
corner of the density map where crowd density is higher and becomes less
visible towards the left where there Is less crowd density. Figure 2: FGA
Module: The module has two feature extraction sections, as shown in the above
figure. Left: The local feature extraction takes a fraction of the input
feature maps for local processing. Right: The global feature extraction takes
another fraction as input from feature maps for global processing. The
spectral block captures full-scale global features. ##: Refer to Figure 3 for
more details. #: refer to Figure 4 for more details on attention blocks.

Most density map-based solutions employ a CNN[6]-based approach to regress
crowd counts. The following methods are promising but remain constrained by
the convolutional kernel’s receptive field, leading to inefficient capture of
global or long-range patterns in models. A global receptive field in crowd
counting allows the network to capture information from a wider context,
ensuring that it considers the overall layout of the crowd and accounts for
variations in crowd density distribution. Many attention-based works such as
MAN[7], DMCNet[8], JANet[9], DA2Net[10] have gained popularity because of
their ability to understand large-scale dependencies through attention
networks. BBA-net [11], proposes an attention-based network designed to
capture the fine-grained details in spatial locations. RFSNet [12] introduces
a patch-wise recurrent self-attention network for spatio-temporal crowd
counting in video. Although these CNN methods perform exceptionally well, they
only utilize convolutions capable of processing information in a local
neighbourhood, ignoring the large-scale pixel-to-pixel context; thus, using
convolutional layers alone can prove inefficient for understanding full-scale
global patterns. To solve this problem, we suggest a novel neural architecture
named the Fourier-Guided Attention (FGA) module, drawing inspiration from the
Fast Fourier Convolution (FFC) [13] and [1]. This module is designed for long-
range context-aware crowd counting, seamlessly combining FFC, spatial
attention, and channel attention into a single unit. FFC operates in the
spatial and frequency domains, allowing FGA to process information in both
local and global receptive fields. At the same time, different attention
mechanisms focus on amplifying the fine-grained features from different parts
of the input sequence to capture relevant information. The proposed framework
can be integrated into existing crowd-counting methods to attend to full-scale
global features.

To conclude, our contributions are mainly threefold:

  * •

A novel dual-path architecture utilizing FFC and Attention mechanisms
incorporating local and global context into a single pluggable unit for
existing works.

  * •

Two simple FGA integrated architectures with comparable performances to the
state-of-the-art methods in crowd counting.

  * •

A thorough qualitative and quantitative evaluation of our proposed approach
with four public benchmark datasets: Shanghai-Tech Part A, Shanghai-Tech Part
B, JHU++ crowd, and UCF-CC-50, along with an extensive ablation study to
understand the contribution of each part in the FGA module.

##  II Related Work

###  II-A Density Based estimation

Most recent crowd-counting methods employ a variation of CNN to predict
density maps from crowd images. The density maps represent the spatial
distribution of people in the image, with brighter regions depicting higher
crowd density. Numerous approaches have been proposed to mitigate the impact
of scale variations, such as MCNN [3], which employs a multi-column
architecture with different kernel sizes to capture scale variation across a
crowd scene effectively. CSRNet [4] and CANNET [5] use single-column with a
dilated convolution layer. certain encoder-decoder methods like MRCNet [14],
[15] incorporate contextual and detailed local information by integrating
high- and low-level features through several lateral connections. The point-
based framework [16] utilizes one-to-one matching with doted annotation, while
[17] makes patch-level feature selection. Some methods [18][19] [20] reduce
the Gaussian noise during annotation and density map generation to give better
predictions. In GauNet[21], the convolution filter is replaced by locally
connected Gaussian kernels. The work proposes a low-rank approximation
accompanied by translation invariance to implement Gaussian convolution for
density estimation.

Although efficient in capturing local features of crowd scenes, the methods
are inefficient in capturing large-scale pixel-by-pixel information.
Attention-based models are introduced to address the mentioned issue.

###  II-B Attention based methods

Attention mechanisms [22] [23] have proven to be highly efficient in
addressing various computer vision challenges. Spatial attention focuses on
learning a weighting map that emphasizes specific spatial coordinates within
the feature map and channel attention is responsible for learning a weighting
map that highlights important feature channels within the feature map. Notable
works such as [1] [24] [25] [26] [2] integrate the spatial and channel
attention. SCAR [1] Uses novel spatial and channel attention and attempts to
extract more discriminative features among different channels, which helps the
model identify heads in an image

The RANet [27] advocates the self-attention mechanism by accounting for both
short and long-range interdependence of pixels and implementing it using local
self-attention (LSA) and global self-attention (GSA) along with a relational
module. [28] propose a Scale-Aware Attention Network(SANet) for local and
global features [29] Introduce conditional random fields (CRFs) to aggregate
multi-scale features within the encoder-decoder network. It incorporates a
non-local attention mechanism implemented as inter- and intra-layer attentions
to expand the receptive field, capturing long-range dependencies to overcome
huge scale variations. [24] purpose an attention-scaling network ASNet and
DSNet to learn auto-scaling for density estimation. In [2], spatial and
channel attention along with the spatial pyramidal pooling and deformation
convolution for object counting. ADCrowdNet, proposed by [30], employs an
attention map generator to identify regions of interest and estimate
congestion levels for improved density map estimation in crowd counting.
Although these works have good performance and attending to large-scale
patterns, are unable to capture purely global features due to the limited
kernel size of convolutions. To this end, we propose FGA Module, a simple
plug-in module for existing CNN-based works that can capture full-scale global
information using the frequency domain for understanding global patterns in a
crowd.

##  III Method

###  III-A Model

Our work introduces the FGA module for extracting full-scale global, semi-
global and local patterns as shown in figure 2. Given input features
$X\in\mathbb{R}^{H\times W\times C}$, where $H\times W$ represents the spatial
resolution and $C$ represents the number of channels, separating factor
$\alpha_{\text{in}}$ determines the fraction of the input feature map
$X\in\mathbb{R}^{H\times W\times C}$ used for global feature extraction
$X_{g}$. The remaining features are used for the local extraction $X_{l}$. The
parameter for separating factor ranges, denoted as $\alpha_{\text{in}}$, is
within the interval [0, 1].

$\mathbf{X_{g}\in\mathbb{R}^{H\times W\times(\alpha_{\text{in}})C}}$ (1)
$\mathbf{X_{l}\in\mathbb{R}^{H\times W\times(1-\alpha_{\text{in}})C}}$ (2)

$X_{l}$ passes through two separate 3x3 convolutional layers to produce
$f_{l\to l}$ and $f_{l\to g}$, used for processing local and semi-global
context. $X_{g}$ is processed independently in a 3x3 convolutional layer and a
spectral block to produce $f_{g\to l}$ and $f_{g\to g}$, used for processing
semi-global and global feature extraction. The spectral block utilizes[31] to
update a signal value in the spectral domain, thereby influencing the entire
feature map. We introduce semi-global information into the two extraction
blocks by adding $f_{l\to g}$ to $f_{g\to g}$ and $f_{g\to l}$ to $f_{l\to
l}$, as shown in the equation.

$Y_{1}=f_{g\to l}+f_{l\to l}$ (3) $Y_{2}=f_{l\to g}+f_{g\to g}$ (4)

We pass $Y_{1}$ and $Y_{2}$ through the batch-normalization ReLU combination.
Finally, $Y_{1}$ and $Y_{2}$ are passed through Channel, spatial attention[1]
respectively. The spatial attention amplifies the global features for each
feature map in $Y_{2}$ to get $Y_{g}$, whereas the channel attention amplifies
relevant information across channels in $Y_{1}$ to get $Y_{l}$. The final
output $Y$ is combined set $\\{Y_{g},Y_{l}\\}$.

Figure 3: Spectral Block: The image above explains the functioning of the
spectral block. Conv-BN-ReLU: refers to convolution, batch normalization, ReLU
combination. Real-FFT2d: refers to the fast-Fourier transformation of real
features. Inv-FFT2d: Refers to Fourier domain to real domain transformation.
Conv 1x1 : refers to 1x1 size convolutions.

###  III-B Spectral Block

The spectral block employs the Cooley-Tukey algorithm [32] for efficient Fast
Fourier Transform (FFT) of input feature maps. The spectral block converts
original spatial features into the frequency domain, also known as the
spectral domain. Subsequently, the spectral data is globally updated within
the frequency domain and transformed back into the spatial domain. Applying
the 2-D FFT to the input $X$, the complex frequency domain $X[u,v]$ is
represented as

$X_{g}[u,v]=\sum_{n=0}^{M-1}\sum_{m=0}^{N-1}x[m,n]\cdot
e^{-j2\pi\left(\frac{un}{M}+\frac{vm}{N}\right)}$ (5)

where: $X[u,v]$ is the complex coefficient at frequency index $(u,v)$ in the
frequency domain. $x[m,n]$ is the pixel value at spatial position $(m,n)$ in
the image. $M$ is the height of the image. $N$ is the width of the image. $u$
and $v$ are the frequency indices ranging from 0 to $M-1$ and 0 to $N-1$,
respectively. We vertically stack the imaginary parts and real parts as
independent features to simplify the computation of FFT-processed features. If
$c$ is the number of channels in $X$, we obtain a tensor $Y$ with twice the
number of channels is given as;

$Y[c,u,v]=\begin{cases}X_{r}[u,v]&\text{if }c<C\text{ (Real part)}\\\
X_{i}[u,v]&\text{if }C\leq c<2C\text{ (Imaginary part)}\\\ \end{cases}$ (6)

The resultant tensor is treated as completely computable, following practices
[33] applying Batch Normalization (BN), Rectified Linear Unit (ReLU) and
Convolution operation in the frequency domain to extract relation between
different pixels, Which gives the tensor $Z$ as

$Z[c,u,v]=\text{ReLU}(\text{BN}(\textbf{Conv}(Y[c,u,v])))$ (7)

In the final step, the results of tensor Z are transformed back into complex
numbers by splitting them along the auxiliary dimension, separating the real
and the imaginary parts. The inverse 2-D FFT operation applied on tensor Z to
produce an output tensor with real values is given as follows.

$\begin{split}X_{\text{out}}[m,n]=\frac{1}{MN}\sum_{u=0}^{M-1}\sum_{v=0}^{N-1}e^{j2\pi\left(\frac{un}{M}\right.+\left.\frac{vm}{N}\right)}\cdot\\\
(Z_{r}[u,v]+jZ_{i}[u,v])\end{split}$ (8)

This equation calculates the inverse 2-D IFFT by summing the contributions
from both the real and imaginary parts for each pixel $(n,m)$. The result
$X_{\text{out}}[m,n]$ represents the spatial domain representation of the
image after applying the inverse transform. We further utilize spatial and
channel attention to enhance local and global features.

Figure 4: Attention Blocks: The image above shows spatial attention used in
the global extraction block in Figure 2. and the channel attention block in
the local feature extraction block. R: Resize T: Transpose

(a) FGA-CSRNet

(b) FGA-CANNet

Figure 5: Counting samples from varying crowd distributions in ShanghaiTech-B
combination: Each output density map is shown right adjacent to the input
image when given to two different models with FGA module.

###  III-C Attention Blocks

We adopt attention mechanisms [23] [22] [1] to enhance the features; spatial
attention allows the model to focus on specific regions of crowd variation
across a particular feature map, whereas channel attention is related to
overall crowd characteristics across channels. For Spatial Attention, we
transform the input feature maps, $X\in\mathbb{R}^{C\times H\times W}$, into
three feature spaces $S_{1}$, $S_{2}$, and $S_{3}$by using $1\times 1$
convolutional layers. The corresponding weight matrix of features is given as
$S_{1}(X)=W_{S1}(x)$, $S_{2}(X)=W_{S2}(x)$, $S_{3}(X)=W_{S3}(x)$. The
Transpose operations are then performed on $S_{1}(X)$, which gives
$S_{1}^{t}(X)$, and multiplied with $S_{2}(X)$ to obtain an $S^{m,n}$. Then,
by applying softmax operation, gives ab attention map$S_{a}$ for pixel value
(m,n) of the shape $HW\times HW$ . is given as:

$S_{a}^{m,n}=\frac{\exp(S^{m,n})}{\sum_{m=1}^{H\times W}\exp(S^{m,n})}$ (9)

The value $S^{m,n}_{a}$ represents how much the $m_{th}$ pixel influences the
$n_{th}$ pixel in the feature map. Finally, $S^{mn}_{a}$ is multiplied with
$S_{3}(X)$ and reshaped the output as $\in\mathbb{R}^{C\times H\times W}$. The
final spatial attention with learnable parameter $\lambda$ and initial feature
map $F_{n}$ is given as follows:

$S_{n}^{final}=\lambda\sum_{m=1}^{HW}(S_{a}^{mn}\cdot S_{3}(X))+F_{n}$ (10)

Similarly for channel attention an input feature map $F\in\mathbb{R}^{C\times
H\times W}$ is given as

$C_{a}^{m,n}=\frac{\exp(C^{m,n})}{\sum_{m=1}^{H\times W}\exp(C^{m,n})}$ (11)

where $C^{mn}_{a}$ denotes the influence of the $m$-th channel on the $n$-th
channel. The final feature map, $C_{\text{final}}$, with a size of $C\times
H\times W$, is computed as:

$C_{n}^{final}=\mu\sum_{m=1}^{HW}C_{a}^{mn}\cdot C_{3}(X)+C_{n}$ (12)

where $\mu$ is a learnable parameter, and $C_{n}$ is the initial feature map.

###  III-D Loss Function

We adopt the Euclidean distance to measure the pixel-wise difference between
estimated density maps and their corresponding ground truth as in previous
works[4][3]. This approach allows the network to focus on individual pixels
and effectively learn the density distribution of the crowd. Thus, the
regression loss is given as.

$L(\Theta)=\frac{1}{2N}\sum_{i=1}^{N}\left(\text{D}(I_{i};\Theta)-\text{GT}(I_{i})\right)^{2}$
(13)

Where N is the size of the training set, and $I_{i}$ represents the $i_{th}$
input image.$\text{D}(I_{i};\Theta)$ is the output and $\text{GT}(I_{i})$ is
ground truth density map of image. $L(\Theta)$ denotes the loss between the
ground-truth density map and the estimated density map.

##  IV Experiments

Implementation Details:  We integrate three layers of FGA module to baseline
method CSRNet[4] and CANNet[5] just after VGG-16 feature extractor and train
the network end-to-end. We use the Adam optimizer[34] with a learning rate of
1e-5. Momentum is between 0.93 and 0.99 for oscillation damping and a weight
decay rate 0.001. The training comprises 100 epochs to optimize performance
and improve generalization.
Evaluation Metrics: We use the mean absolute error (MAE) and root mean square
error (MSE) as evaluation metrics for crowd density estimates. It is defined
as $\text{MAE}=\frac{1}{N}\sum_{i=1}^{n}|y_{i}-\hat{y}_{i}|$ and MSE is
$=\sqrt{\frac{1}{N}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}}$. where N is the
number of the test images, $y_{i}$ and $\hat{y}_{i}$ are the ground truth and
estimated counts of the $i_{th}$ image, respectively.
Ground Truth Generation: We employ geometry-adaptive kernels to handle highly
congested scenes in line with the density map generation method described in
[3]. By applying a normalized Gaussian kernel to blur each head annotation, we
generate ground truth that considers the spatial distribution of all images in
each dataset. The geometry-adaptive kernel is defined as follows:

$F(x)=\sum_{i=1}^{N}\delta(x-x_{i})\times G_{\sigma_{i}}(x)$ (14)

where $\sigma_{i}=\beta d_{i}$ for each targeted object $x_{i}$ in the ground
truth $\delta$, with $d_{i}$ representing the average distance of $k$ nearest
neighbors. The density map is generated by convolving $\delta(x-x_{i})$ with a
Gaussian kernel having a parameter $\sigma_{i}$ (standard deviation), where
$x$ denotes the pixel position in the image. In our experiments, we adhere to
the configuration in [3] with $\beta=0.3$ and $k=3$. In sparse crowds, we
adapt the Gaussian kernel to the average head size to blur all annotations.

a)

b)

c)

Figure 6: Gradient activation maps of final layer for different models with
and without FGA module:  a), b) and c) are three sample images with different
degrees of crowd dispersion, representing high, moderate, and low levels of
crowd dispersion, respectively. Each sample image is analysed using gradient
activation maps to understand area of activation around a head. I), II), III),
IV) represents the performance of FGA + CSRNet, CSRNet, FGA + CANNet and
CANNet on the same sample image when provided as input. Each gradient
activation map has its reference image fused with activations below.

###  IV-A ShanghaiTech Dataset:-

Models | SHA | SHB
---|---|---
| MSE | MAE | MSE | MAE
Bayesian+[35] | 101.8 | 62.8 | 12.7 | 7.7
DMCNet[8] | 84.5 | 58.4 | 13.6 | 8.6
P2PNet [16] | 85.0 | 52.7 | 9.9 | 6.2
GAUNet[21] | 89.1 | 54.8 | 9.9 | 6.2
NASCount[36] | 93.40 | 56.7 | 102.0 | 6.7
DMCount[19] | 95.7 | 59.7 | 11.8 | 7.4
BBANet[11] | 93.8 | 63.8 | 12.0 | 7.8
ASNet[24] | 90.1 | 57.7 | - | -
DensityToken[37] | 100.5 | 64.4 | 17.1 | 9.20
MAN[7] | 90.3 | 56.8 | - | -
IF-CKT+[38] | 16.8 | 10.9 | - | -
DA2Net[10] | 128.4 | 74.1 | 13.2 | 7.9
CANNet[5] | 100.0 | 62.3 | 12.2 | 7.8
CSRNet[4] | 115.0 | 68.2 | 16.0 | 10.6
CANNet + FGA | 90.6 | 61.1 | 10.4 | 6.9
CSRNet + FGA | 110.4 | 66.8 | 13.5 | 8.4

TABLE I: Evaluations on ShanghaiTech Datasets

ShanghaiTech[3] dataset is a popular benchmark for evaluating crowd-counting
algorithms due to its crowd variations. The ShanghaiTech dataset, crucial for
crowd-counting research, comprises 1,198 high-resolution images from the
ShanghaiTech University campus. Part A has 482 images; Part B has 716,
totalling around 330,000 annotated individuals. The FGA module improves both
CSRNet[4] and CANNet[5] over ShanghaiTech Datasets with an 18.5% and 20.7%
improvement on MSE and MAE on ShanghaiTech-B dataset for CSRNet + FGA and 14%
and 11% for MSE and MAE metrics for CANNet + FGA model.

###  IV-B UCF-CC-50 Dataset:-

Models | MSE | MAE
---|---|---
Bayesian+[35] | 308.2 | 229.3
P2PNet[16] | 256.1 | 172.7
GAUNet[21] | 256.5 | 186.3
NASCount[36] | 297.3 | 208.4
DMCount[19] | 291.5 | 211.0
BBANet[11] | 316.9 | 230.5
ASNet[24] | 251.6 | 174.8
DA2Net[10] | 237.0 | 167.0
CANNet[5] | 243.7 | 212.2
CSRNet[4] | 397.5 | 266.1
CANNet + FGA | 223.0 | 197.2
CSRNet + FGA | 378.2 | 252.4

TABLE II: Evaluations on UCF-CC-50 Dataset

The UCF-CC-50[39] dataset consists of 50 high-resolution images, meticulously
annotated to provide accurate head counts. It serves as a benchmark for crowd-
counting algorithms, encompassing diverse environments and presenting
challenges in varying crowd densities. The application of the FGA module shows
an improvement in the performance of the CSRNet[4] and CANNet[5] baselines, as
shown in Table II.

###  IV-C JHU++ crowd Dataset:-

We also evaluate our Proposed Approach on the JHU++ crowd dataset[40], which
has 4,372 images with a total of 1.51 million annotations of varying crowd
orientations. Table III. The FGA module can significantly reduce the baseline
MSE and MAE metrics, with CANNet[5] improving by 22.8 and 7.9 points,
respectively. CSRNet[4] improved by 25.5 and 16.7 points on MSE and MAE
metrics.

Models | MSE | MAE
---|---|---
DMCNet[8] | 246.9 | 69.6
GAUNet [21] | 245.1 | 58.2
DMCount[19] | 261.4 | 66.0
MAN[7] | 209.9 | 53.4
DA2Net | 204.3 | 111.7
CANNet[5] | 278.4 | 63.2
CSRNet[4] | 309.2 | 85.9
CANNet + FGA | 255.6 | 55.3
CSRNet + FGA | 283.7 | 69.2

TABLE III: Evaluations on JHU++ Dataset

###  IV-D Grad-CAM analysis on crowds having different degrees of dispersion
in an image

We explore gradient activation maps using Grad-CAM[41] as shown in Figure 6 to
understand and break down the impact of the FGA module on CSRNet[4] and
CANNet[5] when presented with crowds having low, moderate and high amounts of
dispersion. We find that the FGA module sharpens the area of activation in all
cases with noticeable improvement in high and moderate levels of dispersion,
showing the efficacy of the FGA module in existing crowd estimation networks
and validating utility of FGA modules.

###  IV-E Ablations:-

Effectiveness of Attention:- To verify the impact of the individual parts of
the attention module, we perform a series of experiments on the ShanghiTech-B
dataset. Our analysis investigates the contributions of Fast Fourier
Convolutions(FFC), spatial attention (SA) and Channel Attention(CA) to our
architecture. We comprehend and assess the relative importance of these two
components within the overall framework. Table IV shows that the combined
effect of FFC, Channel attention, and Spatial attention reduces the MSE and
MAE metrics, whereas partial implementation partially degrades the baseline
performance. We also see that Individual use of FFC is not efficient in the
context of crowd counting.

Combination | CANNet | CSRNet
---|---|---
| MAE | MSE | MAE | MSE
FFC | 14.1 | 21.8 | 18.7 | 27.3
FFC + SA | 9.9 | 12.3 | 12.2 | 18.8
FFC + CA | 8.7 | 12.5 | 11.9 | 18.0
FFC + CA + SA | 6.9 | 10.4 | 8.4 | 13.5
TABLE IV: Ablation study over CSRNet[4] and CANNet[5] with varying internal
configuration

###  IV-F Counting on varying crowd densities:-

We analyse counting samples with different crowd densities in the FGA + CANNet
and FGA + CSRNet configurations, as shown in Figure 5. On closer inspection,
we found that our model performs exceptionally well in low-density, moderate-
density, and high-density crowds. We see a minor decrease in performance in
high-density crowds. This needs further analysis to explain why Fourier-guided
attention slightly decreases in high crowd densities and has been left as a
future work in this domain.

##  V Conclusion

In this paper, we propose Fourier-guided attention using attention-assisted
convolutions in the frequency domain and real domain for crowd count
estimation, which leverages the effective combination of full-scale global and
local feature extraction in a single unit. It could be integrated into CNN-
based crowd count estimation methods to improve performance. Experiments
conducted across four benchmark datasets with two baseline models demonstrate
that it can significantly improve the baselines. To check individual
robustness and relative performance to state-of-the-art methods, we perform a
benchmark analysis and a qualitative analysis along with an ablation study of
individual components of the model.

## References

  * [1] J. Gao, Q. Wang, and Y. Yuan, “Scar: Spatial-/channel-wise attention regression networks for crowd counting,” _Neurocomputing_ , vol. 363, pp. 1–8, 2019.
  * [2] G. Gao, Q. Liu, and Y. Wang, “Counting from sky: A large-scale data set for remote sensing object counting and a benchmark method,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 59, no. 5, pp. 3642–3655, 2020.
  * [3] Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma, “Single-image crowd counting via multi-column convolutional neural network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 589–597.
  * [4] Y. Li, X. Zhang, and D. Chen, “Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 1091–1100.
  * [5] W. Liu, M. Salzmann, and P. Fua, “Context-aware crowd counting,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 5099–5108.
  * [6] Y. LeCun, Y. Bengio _et al._ , “Convolutional networks for images, speech, and time series,” _The handbook of brain theory and neural networks_ , vol. 3361, no. 10, p. 1995, 1995.
  * [7] H. Lin, Z. Ma, R. Ji, Y. Wang, and X. Hong, “Boosting crowd counting via multifaceted attention,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 19 628–19 637.
  * [8] M. Wang, H. Cai, Y. Dai, and M. Gong, “Dynamic mixture of counter network for location-agnostic crowd counting,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023, pp. 167–177.
  * [9] Y. He, Y. Xia, Y. Wang, and B. Yin, “Jointly attention network for crowd counting,” _Neurocomputing_ , vol. 487, pp. 157–171, 2022.
  * [10] W. Zhai, Q. Li, Y. Zhou, X. Li, J. Pan, G. Zou, and M. Gao, “Da 2 net: a dual attention-aware network for robust crowd counting,” _Multimedia Systems_ , vol. 29, no. 5, pp. 3027–3040, 2023.
  * [11] Y. Hou, C. Li, F. Yang, C. Ma, L. Zhu, Y. Li, H. Jia, and X. Xie, “Bba-net: A bi-branch attention network for crowd counting,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 4072–4076.
  * [12] J. Zhang, Z. Wu, X. Zhang, G. Song, Y. Wang, and J. Chen, “Recurrent fine-grained self-attention network for video crowd counting,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [13] L. Chi, B. Jiang, and Y. Mu, “Fast fourier convolution,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 4479–4488, 2020.
  * [14] R. Bahmanyar, E. Vig, and P. Reinartz, “Mrcnet: Crowd counting and density map estimation in aerial and ground imagery,” _arXiv preprint arXiv:1909.12743_ , 2019.
  * [15] X. Jiang, Z. Xiao, B. Zhang, X. Zhen, X. Cao, D. Doermann, and L. Shao, “Crowd counting and density estimation by trellis encoder-decoder networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 6133–6142.
  * [16] Q. Song, C. Wang, Z. Jiang, Y. Wang, Y. Tai, C. Wang, J. Li, F. Huang, and Y. Wu, “Rethinking counting and localization in crowds: A purely point-based framework,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 3365–3374.
  * [17] Q. Song, C. Wang, Y. Wang, Y. Tai, C. Wang, J. Li, J. Wu, and J. Ma, “To choose or to fuse? scale selection for crowd counting,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 35, no. 3, 2021, pp. 2576–2583.
  * [18] J. Wan and A. Chan, “Modeling noisy annotations for crowd counting,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 3386–3396, 2020.
  * [19] B. Wang, H. Liu, D. Samaras, and M. H. Nguyen, “Distribution matching for crowd counting,” _Advances in neural information processing systems_ , vol. 33, pp. 1595–1607, 2020.
  * [20] V. A. Sindagi, R. Yasarla, D. S. Babu, R. V. Babu, and V. M. Patel, “Learning to count in the crowd from limited labeled data,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16_. Springer, 2020, pp. 212–229.
  * [21] Z.-Q. Cheng, Q. Dai, H. Li, J. Song, X. Wu, and A. G. Hauptmann, “Rethinking spatial invariance of convolutional networks for object counting,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 19 638–19 648.
  * [22] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 3–19.
  * [23] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [24] X. Jiang, L. Zhang, M. Xu, T. Zhang, P. Lv, B. Zhou, X. Yang, and Y. Pang, “Attention scaling for crowd counting,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 4706–4715.
  * [25] D. Guo, K. Li, Z.-J. Zha, and M. Wang, “Dadnet: Dilated-attention-deformable convnet for crowd counting,” in _Proceedings of the 27th ACM international conference on multimedia_ , 2019, pp. 1823–1832.
  * [26] R. R. Varior, B. Shuai, J. Tighe, and D. Modolo, “Multi-scale attention network for crowd counting,” _arXiv preprint arXiv:1901.06026_ , 2019.
  * [27] A. Zhang, J. Shen, Z. Xiao, F. Zhu, X. Zhen, X. Cao, and L. Shao, “Relational attention network for crowd counting,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 6788–6797.
  * [28] M. Hossain, M. Hosseinzadeh, O. Chanda, and Y. Wang, “Crowd counting using scale-aware attention networks,” in _2019 IEEE winter conference on applications of computer vision (WACV)_. IEEE, 2019, pp. 1280–1288.
  * [29] A. Zhang, L. Yue, J. Shen, F. Zhu, X. Zhen, X. Cao, and L. Shao, “Attentional neural fields for crowd counting,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 5714–5723.
  * [30] N. Liu, Y. Long, C. Zou, Q. Niu, L. Pan, and H. Wu, “Adcrowdnet: An attention-injective deformable convolutional network for crowd understanding,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 3225–3234.
  * [31] Y. Katznelson, _An introduction to harmonic analysis_. Cambridge University Press, 2004.
  * [32] J. W. Cooley and J. W. Tukey, “An algorithm for the machine calculation of complex fourier series,” _Mathematics of computation_ , vol. 19, no. 90, pp. 297–301, 1965.
  * [33] L. Chi, G. Tian, Y. Mu, L. Xie, and Q. Tian, “Fast non-local neural networks with spectral residual learning,” in _Proceedings of the 27th ACM International Conference on Multimedia_ , 2019, pp. 2142–2151.
  * [34] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2017.
  * [35] Z. Ma, X. Wei, X. Hong, and Y. Gong, “Bayesian loss for crowd count estimation with point supervision,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 6142–6151.
  * [36] Y. Hu, X. Jiang, X. Liu, B. Zhang, J. Han, X. Cao, and D. Doermann, “Nas-count: Counting-by-density with neural architecture search,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16_. Springer, 2020, pp. 747–766.
  * [37] Z. Hu, B. Wang, and X. Li, “Densitytoken: Weakly-supervised crowd counting with density classification,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [38] Y. Cai, L. Chen, H. Guan, S. Lin, C. Lu, C. Wang, and G. He, “Explicit invariant feature induced cross-domain crowd counting,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 1, 2023, pp. 259–267.
  * [39] H. Idrees, I. Saleemi, C. Seibert, and M. Shah, “Multi-source multi-scale counting in extremely dense crowd images,” in _2013 IEEE Conference on Computer Vision and Pattern Recognition_ , 2013, pp. 2547–2554.
  * [40] V. A. Sindagi, R. Yasarla, and V. M. Patel, “Pushing the frontiers of unconstrained crowd counting: New dataset and benchmark method,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1221–1231.
  * [41] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.


# Guided Context Gating: Learning to leverage salient lesions in retinal
fundus images

###### Abstract

Effectively representing medical images, especially retinal images, presents a
considerable challenge due to variations in appearance, size, and contextual
information of pathological signs called lesions. Precise discrimination of
these lesions is crucial for diagnosing vision-threatening issues such as
diabetic retinopathy. While visual attention-based neural networks have been
introduced to learn spatial context and channel correlations from retinal
images, they often fall short in capturing localized lesion context.
Addressing this limitation, we propose a novel attention mechanism called
Guided Context Gating, an unique approach that integrates Context Formulation,
Channel Correlation, and Guided Gating to learn global context, spatial
correlations, and localized lesion context. Our qualitative evaluation against
existing attention mechanisms emphasize the superiority of Guided Context
Gating in terms of explainability. Notably, experiments on the Zenodo-DR-7
dataset reveal a substantial 2.63% accuracy boost over advanced attention
mechanisms & an impressive 6.53% improvement over the state-of-the-art Vision
Transformer for assessing the severity grade of retinopathy, even with
imbalanced and limited training samples for each class.

Index Terms—  Representation Learning, Visual Attention, Guided Context
Gating, Diabetic Retinopathy

##  1 Introduction

Diabetic Retinopathy is a prominent visual impairment that can lead to
permanent vision loss in long-term diabetics [1]. It is characterized by
accelerating vascular disruptions in the retina due to chronic hyperglycemia.
The progressive nature of this disorder can result in the formation of various
pathological signs called lesions [2]. Based on the growth of these lesions,
retinopathy can be categorized as non-proliferative (NPDR) and proliferative
retinopathy (PDR). NPDR includes Mild, Moderate, Severe, and Very Severe
severity grades, while PDR has an additional advanced severity grade [3].
According to the World Health Organization (WHO), it is estimated that these
diseases can affect around 500 million individuals worldwide. This number can
increase even further if necessary care is not taken by ophthalmologists to
identify this disease at early stages and treat accordingly, highlighting the
need for automating the diagnosis process. [4].

Fig. 1: Retinal fundus image highlighting various lesions

Figure 1 illustrates a retinal scan with annotations highlighting various
types of lesions, including micro-aneurysms, hemorrhages, exudates, and cotton
wool spots. Discriminating among these lesions poses a challenge due to
variations in color, shape, size, and location, making accurate identification
and differentiation a complex task in the diagnosis process. Small localized
bleedings at the optic nerve head, referred to as Optic disc hemorrhages,
present additional layer of complexity in discrimination due to their micro
size, subtle appearance, and a potential overlap with the disc. The similarity
between optic disc and exudates further elevates the risk of incorrect
clinical decisions, emphasizing the need for advanced representation learning
techniques for accurate discrimination in the diagnosis process.

Fig. 2: Architecture of proposed Guided Context Gating Network that formulates
context from convolutional features and employs it as a guiding signal for
computing lesion-specific localized context, by retaining both spatial context
and channel correlations; Context Formulation – selectively focuses on
relevant features in the initial spatial representations and computes global
context information; Channel Correlation – processes the computed context
information & capture channel-wise correlations; Guided Gating – utilizes
context features to compute lesion contextual attention representations;

In recent years, deep convolutional neural networks (CNNs), particularly
NASNet pre-trained CNN, have been utilized for learning spatial
representations from medical images like retinal fundus images [5]. However,
the lack of a dedicated attention mechanism to emphasize disease-specific
features has limited the performance. A spatial attention-based method was
introduced to learn texture-specific features from fundus images [6], yet it
falls short in retaining category-specific information across the feature set.
Channel attention-based methods, such as CANet [7] and CABNet [8],
successfully capture spatial and channel context but may miss to retain global
context. Gated Attention in a composite neural network was implemented to
learn lesion-specific local features from retinal images [9], yet it struggled
to preserve global context. Cross-lesion attention networks were designed for
complex lesions based on channel-spatial convolution [10], and Transformer-
guided category-relation attention networks were aimed to enhance feature
information within the class [11]. While attention-based models have shown
robustness in capturing holistic features, challenges persist due to lesion
variability, contrast issues, limited annotated data, class imbalance, and
ethnic variations in lesion characteristics [12, 13]. Addressing all
aforementioned challenges, we design a special module of attention called
Guided Context Gating which can be embedded to any CNN architecture. This can
learn to leverage the salient lesions in retinal funds images which is crucial
for accurate retinopathy assessment. Guided Context Gating involves in
capturing global context and identifying the importance of different spatial
positions in the overall context. Significantly, it captures localized lesion
context features in addition to global context representations. This holistic
approach ensures a more comprehensive understanding of the retinal features,
enhancing the model’s ability to make accurate assessments.

##  2 Methodology

This research aims to introduce a specialized module known as Guided Context
Gating, designed to augment the capabilities of deep convolutional neural
networks in effectively learning and utilizing significant lesions within
retinal images, even when faced with limited training data. The comprehensive
architectural details of the proposed Guided Context Gating module are
illustrated in Figure 2. The subsequent subsections provide details associated
with each module involved in the proposed Guided Context Gating.

###  2.1 Convolutional Base

In the initial stage of our proposed framework, we focus on learning
representations from retinal scan images. Traditionally, Deep CNN, with
convolution and pooling operations, was employed for spatial representation
acquisition from images of varied modalities. However, a drawback is the need
for large amounts of labeled data for training, which is often challenging in
medical image analysis. To overcome this, pre-trained CNN, such as
EfficientNet$V_{2}$ models ($B_{0}$ to $B_{7}$) equipped with ImageNet
weights, was introduced. EfficientNet$V_{2}$ is designed for scalability
across depth, width, and resolution dimensions, incorporating special Squeeze
& Excitation blocks for attention. Our choice for this research is
EfficientNet$V_{2}B_{0}$ as the Convolutional base, extracting spatial
representations from retinal images. Passing a retinal scan image $X$ with
dimensions $(512\times 512\times 3)$ through EfficientNet$V_{2}B_{0}$ results
in spatial representations $R$ of dimensions $(16\times 16\times 1280)$ from
the final convolutional block, serving as fundamental features of retinal
images.

###  2.2 Guided Context Gating

The retinal representations learned from the pre-trained CNN exhibit global
information, with equal priority given to each feature map. However, not all
features may be sufficiently significant to aid the model in making accurate
decisions. To address this limitation, attention mechanisms have been
introduced in the literature, aiming to prioritize the most relevant features
and enhance the performance of models trained on them. Spatial and Channel
attention mechanisms have been widely employed for learning task-specific and
cross-channel correlations. In our proposed model, the Squeeze and Excitation
blocks of EfficientNet$V_{2}B_{0}$ serve this purpose, allowing the model to
focus on channel specific features. Despite the usage of attention, model
still falls short in capturing lesion-specific contextual information, such as
lesion shapes, location, and spatial texture. These aspects are crucial for
diagnosing retinal diseases, and their omission hinders the model’s ability to
make accurate assessments. Inspired from [14, 15], we design Guided Context
Gating Attention block which can achieve this and enhance the retinal image
representations with lesion context rich information. The modeling process can
be structured into three distinct phases: context formulation, channel
correlation, and guided gating.

####  2.2.1 Context Formulation

This module computes lesion context information by employing global attention
pooling through a two-step process. First, a point-wise convolution is applied
to the spatial representations, which captures essential features and
relationships within the data. Subsequently, a softmax activation function is
utilized to normalize and highlight the significance of different spatial
elements. The combination of these two steps results in an attention map,
where each element represents the importance of the corresponding spatial
position in contributing to global context information. This attention map is
then used to selectively focus on relevant features in the initial spatial
representations, effectively computing lesion context information. For
instance, when the spatial representations $R$ with dimensions $(H\times
W\times D)$ are passed through a $Conv(1\times 1)$ operation followed by
Softmax, an attention map $A$ of dimension $(H\times W\times 1)$ is obtained.
The key benefit lies in the fact that this attention map is then
multiplicatively combined with the initial feature maps, resulting in a lesion
context attention map with dimensions $(D\times 1\times 1)$. To achieve this,
reshape and transformation operations are performed as necessary, facilitating
efficient feature map multiplications.

$A_{c}=\sum\limits_{j=1}^{d}\frac{e^{W_{c}r_{j}}}{\sum\limits_{m=1}^{d}e^{W_{c}r_{m}}}\qquad\forall
r_{j}\in R(r_{1},r_{2},...,r_{d})$ (1)

Equation 1 represents the mathematical computation of lesion context
information through global attention pooling of spatial representations $R$
where $W_{c}$ is point-wise convolution parameters and $A_{c}$ is context
attention map.

####  2.2.2 Channel Correlation

In this module, channel-wise dependencies are captured through a specialized
block comprising two point-wise convolution layers, complemented by Layer
Normalization and Rectified Linear Unit (ReLU) activation functions. This
strategic configuration is specifically designed for processing previously
computed context information and effectively captures channel-wise
correlations within retinal representations. The inclusion of ReLU activation
introduces non-linearity to the processing of context information, enhancing
the model’s capacity to discern intricate dependencies among different
channels. The resulting features are then adeptly aggregated with the original
spatial features to incorporate the identified correlations among different
channels in the spatial representations, thereby enriching the overall
contextual understanding within the retinal features.

$\delta=\sum\limits_{j=1}^{k}W_{t1}a_{j}\qquad\forall a_{j}\in
A_{c}(a_{1},a_{2},...,a_{k})$ (2)
$\theta=\text{LayerNormalization}(\Gamma(\delta))$ (3)
$\hat{A_{c}}=\sum\limits_{j=1}^{d}W_{t2}\alpha_{j}\qquad\forall\alpha_{j}\in\theta(\alpha_{1},\alpha_{2},...,\alpha_{d})$
(4) $R_{g}=R\oplus\hat{A_{c}}$ (5)

Equations 2 to 5 elucidate the mathematical formulations of the channel
correlation block. In these equations, $\Gamma$ represents ReLU activation,
$W_{t1}$ and $W_{t2}$ represent the parameters of the point-wise convolution
layers, while $a_{j}$ and $\alpha_{j}$ correspond to the elements of $A_{c}$
and $\theta$, respectively. Equation 5 illustrates the aggregation process,
where the original visual representations $R$ are combined with the computed
context features $\hat{A_{c}}$ through element-wise addition ($\oplus$). This
operation signifies the fusion of information from both the original visual
features and the context features.

####  2.2.3 Guided Gating

This module is designed to enhance the model’s ability to focus on lesion
specific regions within the retinal representations. This is achieved by
introducing a guiding mechanism that utilizes context features to compute
attention coefficients. These are obtained through a series of convolutional
operations, including point-wise convolutions and non-linearities (ReLU and
Sigmoid). The convolution operations create an additive attention map that
emphasizes relevant features based on the combination of global and local
context information. The attention coefficients, derived from the additive
attention map, serve as a guiding signal which helps to selectively amplify or
suppress features in the retinal representations based on their importance as
determined by the attention coefficients. The combination of convolution
operations, normalization, and gating ensures that the model can effectively
highlight lesion-specific details, providing a mechanism for capturing both
global and localized context features within the retinal images.

When the original retinal representations $R$ and the corresponding guiding
context signal $R_{g}\in\mathbb{R}^{W\times H\times D}$ are passed to this
module, lesion-specific coefficients $R_{l}$ are computed by considering their
joint information through two point-wise convolution operations with
parameters $W_{x}$, $W_{g}$. The guiding context signal $R_{g}$ is
parametrically $b_{rg}$ fused along with the retinal representations to
jointly identify salient features at different scales by emphasizing relevant
features based on the combination of global and local context information.
These coefficients are further subjected to Rectified Linear Unit (ReLU) and
Layer Normalization to introduce non-linearity and normalize the values,
respectively. The inclusion of ReLU activation brings non-linearity to the
processing coefficients, allowing the model to capture complex relationships
and intricate dependencies among different features. By normalizing these
coefficients across the channel dimension, Layer Normalization mitigates the
internal covariate shift during training, leading to more robust and faster
convergence. The gating coefficients $R_{\psi}$ is formulated using an
additive attention mechanism, specifically utilizing a sigmoid activation
function by employing a set of parameters ($W_{\psi}$, $b_{\psi}$).

$R_{l,i}=\Gamma(W_{x}r_{i}+W_{g}g_{i}+b_{xg})\qquad\forall r_{i}\in R,g_{i}\in
R_{g}$ (6) $R_{\psi}=\psi R_{l}+b_{\psi}$ (7)
$R_{att}=\sigma(R_{\psi},W_{\psi})$ (8)

where $\Gamma$ is ReLU activation, element-wise non-linearity, $\psi$ is
parameter vector for computing $R_{\psi}$ and $\sigma(x)$ is sigmoid
activation, a normalization function to restrict the attention coefficient
range to $[0,1]$. These guided gating attention coefficients are multiplied
with original retinal features to compute salient lesion representations from
retinal images.

###  2.3 Regularized Classification Head

The Regularized Classification Head comprises two ReLU-activated fully
connected dense layers with 512 and 256 neurons, respectively. Each layer
integrates a batch normalization layer followed by a dropout layer with a 0.3
rate, mitigating overfitting. Moreover, $L_{1}$ regularization is applied to
the weights, and $L_{1}L_{2}$ regularization is imposed on biases during
training. These modules are arranged and trained end-to-end for predictions.
Batch Normalization, Dropout, and parameter regularization layers are stacked
together, culminating with a softmax layer for predicting class labels. These
regularization techniques collectively enhance the model’s robustness,
enabling it to generate meaningful predictions with acceptable precision. The
model employs Gradient Centralization [16] during training, introducing a
novel optimization technique that standardizes network activation values
through Z-score standardization of weight vectors. This process can be
expressed as $\mathbf{W}=\frac{W-\mu}{\sigma}$, where $W$ is the weight
vector, and $\mu,\sigma$ denote the mean and standard deviation of $W$. Unlike
traditional parameter operations, Gradient Centralization operates directly on
gradient values, ensuring a zero mean for gradient vectors. This boosts
training speed and efficiency, generalizes weight and output feature space,
and improves the overall generalizability of proposed model.

##  3 Experiments

###  3.1 Dataset

The effectiveness of the proposed approach is evaluated using the Zenodo-DR-7
benchmark dataset, which consists of 757 color fundus images obtained from the
Department of Ophthalmology at Hospital de Clínicas, Universidad Nacional de
Asunción, Paraguay [17]. These retinal images were captured using the Zeiss
Visucam 500 camera following a clinical procedure. Expert ophthalmologists
have meticulously classified the dataset, enhancing its value for the
detection of Non-Proliferative Diabetic Retinopathy (NPDR) and Proliferative
Diabetic Retinopathy (PDR). The classifications include various labels such as
Normal - $C_{0}$ (187), Mild NPDR - $C_{1}$(4), Moderate NPDR - $C_{2}$(80),
Severe NPDR - $C_{3}$(176), Very Severe NPDR - $C_{4}$(108), Proliferative DR
- $C_{5}$(88), and Advanced PDR-$C_{6}$(114). The dataset is divided into
specific training (605) and test (152) splits. We extended our experiments to
the IDRiD [18], Messidor-2 [19], and APTOS-2019 [2] datasets, encompassing
five severity grades $(C_{0},C_{1},C_{2},C_{3},C_{5})$. Detailed information
about these datasets is available in the provided references.

###  3.2 Experimental Setup

The proposed model was implemented using the Keras and TensorFlow libraries
and experimental studies were conducted on a platform equipped with an Nvidia
P100 GPU featuring 16GB Memory, a 1.32GHz clock, and supporting 9.3 TFLOPS. To
optimize the model, various hyperparameters values were explored, including
learning rates from 0.0001 to 0.01, dropout rates between 0.1 and 0.5, and
regularization rates from 0.001 to 0.1. We selected hyperparameter values to
enhance the model’s performance, setting the initial learning rate to 0.0001,
a batch size of 32, and training over 100 epochs. Weight regularization using
L2 with a coefficient of 0.005 and bias regularization using L1_L2 with the
same coefficient were applied. The activation function for the multi-layer
perceptron was ReLU, and the output activation function was Softmax. The
optimizer used was RMSProp, and the loss function employed was categorical
cross-entropy. During all experiments, we followed the standard hold-out-set
validation strategy for saving the model checkpoint.

###  3.3 Quantitative Evaluation

Table 1: Comparative Performance Analysis of Guided Context Gating and Attention based methods on Zenodo-DR-7 Approach | Accuracy (%) | Precision (%) | Recall (%) |  $F_{1}$ Score (%) | Kappa (%) | AUC (%)
---|---|---|---|---|---|---
No Attention | 82.89 | 68.47 | 68.18 | 68.06 | 83.27 | 96.90
Spatial Attention [12] | 86.18 | 85.92 | 85.07 | 85.40 | 83.07 | 97.27
Channel Attention [8] | 86.84 | 86.91 | 86.39 | 86.52 | 83.79 | 96.90
Global Context Attention [15] | 86.18 | 87.93 | 86.37 | 86.17 | 83.02 | 97.57
Gated Attention [9] | 87.50 | 87.28 | 86.84 | 86.92 | 84.84 | 96.85
Guided Context Gating (Ours) | 90.13 | 88.78 | 88.40 | 88.49 | 93.07 | 97.68
Table 2: Classification Report of Guided Context Gating Network for Retinopathy Classification on Zenodo-DR-7 Class | Acc | Prec | Rec | F1 | AUC | #Test
---|---|---|---|---|---|---
$C_{0}$ | 95.65 | 91.67 | 95.65 | 93.62 | 99.43 | 23
$C_{1}$ | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 01
$C_{2}$ | 75.00 | 80.00 | 75.00 | 77.42 | 92.97 | 16
$C_{3}$ | 100.0 | 97.37 | 100.0 | 98.67 | 100 | 37
$C_{4}$ | 55.56 | 66.67 | 55.56 | 60.61 | 92.68 | 18
$C_{5}$ | 97.14 | 94.44 | 97.14 | 95.77 | 99.56 | 35
$C_{6}$ | 95.45 | 91.3 | 95.45 | 93.33 | 99.09 | 22
Macro | 90.13 | 88.78 | 88.4 | 88.49 | 97.80 | 152
Weighted | - | 89.51 | 90.13 | 89.73 | 98.64 | -

Table 1 presents a comprehensive performance comparison of the proposed Guided
Context Gating approach with existing attention mechanisms. Our approach
achieves superior results across various metrics. Specifically, the Guided
Context Gating outperforms other attention mechanisms in terms of accuracy
(Acc), precision (Prec), recall (Rec), $F_{1}$ score ($F_{1}$), kappa, and
AUC. Notably, it achieves a remarkable accuracy of 90.13%, showcasing its
effectiveness in capturing relevant contextual information. Precision, recall,
and $F_{1}$ score metrics demonstrate the balanced and robust performance of
the proposed approach, ensuring both high accuracy and sensitivity in
identifying salient features. The kappa statistic reflects substantial
agreement beyond chance, emphasizing the reliability of our approach.
Moreover, the AUC metric indicates the model’s strong discriminatory power in
distinguishing relevant features. These results collectively highlight the
superior performance of our Guided Context Gating approach compared to
alternative attention mechanisms.

Table 2 presents the classification report for Diabetic Retinopathy (DR)
severity using the proposed Guided Context Gating Network. Despite the
dataset’s imbalanced nature, especially in low-sampled classes, our approach
demonstrates superior performance in accurately classifying these challenging
instances, providing a robust solution for DR severity classification. The
model exhibits exceptional accuracy, precision, recall, and $F_{1}$ score
across all DR severity categories, excelling in both identifying normal
conditions and detecting severe pathological states, particularly in Normal
and Proliferative DR cases. Macro-average metrics emphasize balanced
performance, and weighted average metrics underscore the model’s robustness,
considering varying class distributions. The Guided Context Gating Network’s
robust performance in low-sampled classes is crucial in medical applications,
offering reliability in providing valuable insights across the entire spectrum
of DR severity.

Table 3: Comparative Study of Guided Context Gating and Vision Transformer [20] across different datasets Dataset | Model | Acc | Prec | Rec | F1 | AUC
---|---|---|---|---|---|---
Zenodo-DR-7 | GCG | 90.13 | 88.78 | 88.40 | 88.49 | 97.68
ViT | 84.61 | 83.62 | 83.17 | 83.19 | 96.17
IDRiD | GCG | 72.14 | 68.24 | 69.56 | 68.34 | 87.51
ViT | 61.17 | 50.53 | 45.91 | 46.18 | 76.26
Messidor-2 | GCG | 80.23 | 80.81 | 69.02 | 73.85 | 87.9
ViT | 76.79 | 75.98 | 57.17 | 61.47 | 88.53
APTOS-2019 | GCG | 85.29 | 74.94 | 70.02 | 70.57 | 93.97
ViT | 83.22 | 70.44 | 66.61 | 67.83 | 93.38

Table 3 illustrates the comparision study of Guided Context Gating (GCG) with
the state-of-the-art Vision Transformer (ViT) [20] across the Zenodo-DR-7,
IDRiD, Messidor-2, and APTOS-2019 datasets. GCG consistently outperforms ViT
in all performance metrics across the datasets, highlighting its robustness
and efficacy in diverse scenarios. This comparative study reveals the
potential of GCG as a formidable solution for classification challenges, given
its capability to handle limited, imbalanced data and provide accurate
predictions even in challenging instances.

###  3.4 Qualitative Evaluation

Fig. 3: Visual representation of attention maps using various strategies
(spatial, channel, global context, gated, and proposed guided context gating)
for severe retinopathy. Spatial attention emphasized local features but lacked
broader context, and channel attention captured color and texture well but
overlooked spatial context. Global context attention risked oversimplification
of lesions, and gating attention emphasized structures but occasionally
highlighted unnecessary lesions. Our proposed attention highlights lesion-
specific details, combining global and localized context features; dark blue
indicates higher attention region.

The significance of Guided Context Gating in discriminating retinopathy lies
in its exceptional ability to leverage the most significant lesions from
retinal images, enhancing both explainability and interpretability. As
illustrated in Figure 3, which depicts attention maps using different
attention strategies including spatial, channel, global context, gated, and
proposed Guided Context Gating, our approach stands out in highlighting the
most relevant regions associated with higher severity grades of retinopathy.

Our study revealed that spatial attention mechanisms effectively emphasized
local features such as blood vessels and lesions, providing valuable insights
into retinal structure. However, a notable drawback was their inability to
consider broader contextual understanding, as demonstrated by their failure to
capture lesions in the Advanced PDR stage. Conversely, channel attention
adeptly highlighted color intensity and texture variations, capturing details
like blood vessel thickness and lesion density. Yet, this focus led to the
oversight of crucial spatial context, as evident in all channel attention
maps. Global context attention successfully captured broader spatial
relationships and contextual information, contributing to a holistic
understanding of the retinal landscape. However, there was a risk of
oversimplifying lesions across all severity grades, potentially undermining
their significance, especially in PDR and Advanced PDR stages. Additionally,
while gating attention effectively emphasized anatomical structures, concerns
were raised about its tendency to highlight unnecessary retinal lesions,
leading to potential misinterpretation in clinical contexts.

Guided Context Gating excels in providing clear and focused attention on
lesions specific to retinopathy, effectively eliminating the attention to
unwanted retinal regions. By doing so, it ensures that the model focuses
solely on the clinically relevant features, enhancing its diagnostic
precision. The approach successfully captures the region of interest,
highlighting the areas crucial for retinopathy assessment, such as
pathological lesions and abnormalities. One of the key strengths of Guided
Context Gating is its ability to discriminate between intra-similar lesions,
such as exudates in proximity to the optic disc, and strictly discriminate
optic disc hemorrhages. While traditional attention mechanisms might struggle
to distinguish between similar features, our approach ensures precise
discrimination, allowing for a more accurate assessment of the severity of
retinopathy. This discrimination is essential for providing ophthalmologists
with a clear and interpretable visualization of the regions contributing to
the retinopathy diagnosis, aiding in clinical decision-making.

##  4 Conclusion

This research introduces the Guided Context Gating module which effectively
extracts both global and local contextual features from retinal images,
addressing limitations associated with limited training data. Through
comprehensive experiments on benchmark datasets, the Guided Context Gating
approach outperforms existing attention mechanisms and state-of-the-art ViT in
terms of all performance metrics. Notably, it demonstrates remarkable accuracy
in classifying various severity levels of diabetic retinopathy, showcasing its
robustness in handling imbalanced data and providing accurate predictions even
for underrepresented classes. The model’s superior performance is further
highlighted in its ability to discriminate intra-similar lesions, contributing
to precise diagnostic assessments. Future work could involve extending the
proposed approach to other medical imaging tasks and integrating additional
modalities of retinal pathology.

## References

  * [1] Mohsen Janghorbani, Raymond B Jones, and Simon P Allison,  “Incidence of and risk factors for proliferative retinopathy and its association with blindness among diabetes clinic attenders,”  Ophthalmic Epidemiology, vol. 7, no. 4, pp. 225–241, 2000.
  * [2] Nagur Shareef Shaik and Teja Krishna Cherukuri,  “Hinge attention network: A joint model for diabetic retinopathy severity grading,”  Applied Intelligence, vol. 52, no. 13, pp. 15105–15121, 2022.
  * [3] Nagur Shareef Shaik and Teja Krishna Cherukuri,  “Lesion-aware attention with neural support vector machine for retinopathy diagnosis,”  Machine Vision and Applications, vol. 32, no. 6, pp. 1–13, 2021.
  * [4] Yingfeng Zheng, Mingguang He, and Nathan Congdon,  “The worldwide epidemic of diabetic retinopathy,”  Indian journal of ophthalmology, vol. 60, pp. 428, 2012.
  * [5] Venkatesulu Dondeti, Jyostna Devi Bodapati, Shaik Nagur Shareef, and Naralasetti Veeranjaneyulu,  “Deep convolution features in non-linear embedding space for fundus image classification.,”  Rev. d’Intelligence Artif., vol. 34, no. 3, pp. 307–313, 2020.
  * [6] Mohammad D Alahmadi,  “Texture attention network for diabetic retinopathy classification,”  IEEE Access, vol. 10, pp. 55522–55532, 2022.
  * [7] Xiaomeng Li, Xiaowei Hu, Lequan Yu, Lei Zhu, Chi-Wing Fu, and Pheng-Ann Heng,  “Canet: cross-disease attention network for joint diabetic retinopathy and diabetic macular edema grading,”  IEEE transactions on medical imaging, vol. 39, no. 5, pp. 1483–1493, 2019.
  * [8] Along He, Tao Li, Ning Li, Kai Wang, and Huazhu Fu,  “Cabnet: Category attention block for imbalanced diabetic retinopathy grading,”  IEEE Transactions on Medical Imaging, vol. 40, no. 1, pp. 143–153, 2020.
  * [9] Jyostna Devi Bodapati, Nagur Shareef Shaik, and Veeranjaneyulu Naralasetti,  “Composite deep neural network with gated-attention mechanism for diabetic retinopathy severity classification,”  Journal of Ambient Intelligence and Humanized Computing, vol. 12, no. 10, pp. 9825–9839, 2021.
  * [10] Xiang Liu and Wei Chi,  “A cross-lesion attention network for accurate diabetic retinopathy grading with fundus images,”  IEEE Transactions on Instrumentation and Measurement, 2023.
  * [11] Feng Zang and Hui Ma,  “Cra-net: Transformer guided category-relation attention network for diabetic retinopathy grading,”  Computers in Biology and Medicine, p. 107993, 2024.
  * [12] Roberto Romero-Oraá, María Herrero-Tudela, María I López, Roberto Hornero, and María García,  “Attention-based deep learning framework for automatic fundus image processing to aid in diabetic retinopathy grading,”  Computer Methods and Programs in Biomedicine, vol. 249, pp. 108160, 2024.
  * [13] Sandeep Madarapu, Samit Ari, and KK Mahapatra,  “A deep integrative approach for diabetic retinopathy classification with synergistic channel-spatial and self-attention mechanism,”  Expert Systems with Applications, p. 123523, 2024.
  * [14] Jo Schlemper, Ozan Oktay, Michiel Schaap, Mattias Heinrich, Bernhard Kainz, Ben Glocker, and Daniel Rueckert,  “Attention gated networks: Learning to leverage salient regions in medical images,”  Medical image analysis, vol. 53, pp. 197–207, 2019.
  * [15] Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu,  “Global context networks,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
  * [16] Hongwei Yong, Jianqiang Huang, Xiansheng Hua, and Lei Zhang,  “Gradient centralization: A new optimization technique for deep neural networks,”  in European Conference on Computer Vision. Springer, 2020, pp. 635–652.
  * [17] Veronica Elisa Castillo Benítez, Ingrid Castro Matto, Julio César Mello Román, José Luis Vázquez Noguera, Miguel García-Torres, Jordan Ayala, Diego P Pinto-Roa, Pedro E Gardel-Sotomayor, Jacques Facon, and Sebastian Alberto Grillo,  “Dataset from fundus images for the study of diabetic retinopathy,”  Data in brief, vol. 36, pp. 107068, 2021.
  * [18] Prasanna Porwal, Samiksha Pachade, Ravi Kamble, Manesh Kokare, Girish Deshmukh, Vivek Sahasrabuddhe, and Fabrice Meriaudeau,  “Indian diabetic retinopathy image dataset (idrid): a database for diabetic retinopathy screening research,”  Data, vol. 3, no. 3, pp. 25, 2018.
  * [19] Michael D Abràmoff, James C Folk, Dennis P Han, Jonathan D Walker, David F Williams, Stephen R Russell, Pascale Massin, Beatrice Cochener, Philippe Gain, Li Tang, et al.,  “Automated analysis of retinal images for detection of referable diabetic retinopathy,”  JAMA ophthalmology, vol. 131, no. 3, pp. 351–357, 2013.
  * [20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  arXiv preprint arXiv:2010.11929, 2020.