# M3T: Multi-Modal Medical Transformer to bridge Clinical Context with Visual
Insights for Retinal Image Medical Description Generation

###### Abstract

Automated retinal image medical description generation is crucial for
streamlining medical diagnosis and treatment planning. Existing challenges
include the reliance on learned retinal image representations, difficulties in
handling multiple imaging modalities, and the lack of clinical context in
visual representations. Addressing these issues, we propose the Multi-Modal
Medical Transformer (M3T), a novel deep learning architecture that integrates
visual representations with diagnostic keywords. Unlike previous studies
focusing on specific aspects, our approach efficiently learns contextual
information and semantics from both modalities, enabling the generation of
precise and coherent medical descriptions for retinal images. Experimental
studies on the DeepEyeNet dataset validate the success of M3T in meeting
ophthalmologists’ standards, demonstrating a substantial 13.5% improvement in
BLEU@4 over the best-performing baseline model.

Index Terms—  Medical Description Generation, Multi-Modal Learning, Medical
Image Analysis, Transformer

##  1 Introduction

In recent years, the global prevalence of visual impairments, such as Diabetic
Retinopathy, Diabetic Macular Edema, and Age-related Macular Edema, has
significantly increased. These conditions, often causing irreversible
blindness, stem from complications like sub-retinal neo-vascularization and
retinal detachment [1]. Predictions by the World Health Organization estimate
that over 500 million people will be affected by these retinal diseases by
2040 [2]. Traditional diagnosis relies on resource-intensive processes,
expensive equipment, and the expertise of ophthalmologists, resulting in
challenges for timely and accurate medical reporting. Automated generation of
medical descriptions from retinal images holds potential for advancing early
diagnosis [3]. Aligned with Image Captioning, this task integrates Computer
Vision and Natural Language Processing, employing Convolutional Neural
Networks (CNNs) for image representations fed into Recurrent Neural Networks
(RNNs) for textual description generation [4, 5]. Despite breakthroughs in
Natural Image Captioning using advanced models like Transformers, accurately
captioning complex retinal images remains challenging due to the limited
annotated medical data [6]. The scarcity of labeled data poses challenges in
both image classification and caption generation tasks in medical image
analysis. Researchers address this by employing Transfer Learning, leveraging
models pre-trained on ImageNet for medical image tasks [7, 8]. Pre-training on
natural images and fine-tuning on medical datasets enhances feature learning,
especially in medical image classification [9]. Semi-supervised and self-
supervised learning in medical representation explores unlabeled data,
benefiting subsequent tasks [10, 11]. Despite these advances, generating
clinically relevant captions for retinal images remains challenging due to the
complexity of interpreting retinal pathology [12].

Fig. 1: Architecture of proposed Multi-Modal Medical Transformer (M3T); Visual
Encoder – learns attention-based representations from retinal images; Keyword
Encoder – learns clinical-context embeddings from diagnostic keywords;
TransFusion Encoder – integrates visual attention features and clinical-
context embeddings, leveraging both visual and semantic information; Medical
Description Generation Decoder – generates coherent and meaningful medical
descriptions by attending to relevant visual and semantic cues, ensuring
contextually appropriate and diagnostically relevant outputs;

Natural Image Caption Generation has improved with multi-modal inputs [13].
For medical descriptions, integrating diagnostic keywords aids in capturing
semantics, but efficiently learning visual context and clinical semantics
presents challenges. The CLIP model, which aligns images and text in a shared
embedding space, also offers significant potential for improving the
integration of visual and textual information [14]. A deep learning framework
utilizing both images & keywords enhances performance in retinal disease
diagnosis [1, 12]. Similarly, contextualized keyword embeddings derived from
techniques like GloVe and GPT2 were integrated with retinal image features for
caption generation, with GPT2 embeddings proving superior [15]. Despite these
efforts, reported quantitative metrics were less competitive due to the choice
of image features and the lack of domain-specific training. Notably,
incorporating Non-local attention-based retinal image features with diagnostic
keywords led to more semantically meaningful medical image captions,
indicating improved performance [16]. Further enhancement was achieved by
training a Transformer model on visual features and keyword embeddings [17].

Most of the aforementioned studies have concentrated on specific aspects, such
as the type of diagnostic keyword embeddings or the attention mechanisms for
visual features. However, there exists a significant gap in efficiently
learning contextual information and semantics from both modalities and
integrating them effectively. This research endeavors to bridge this gap by
proposing a novel deep learning architecture called the Multi-Modal Medical
Transformer. Our primary goal is to learn visual context from retinal images,
understand clinical semantics from diagnostic keywords, and establish an
effective integration mechanism through attention. By amalgamating these
crucial components, our aim is to produce precise and coherent medical
descriptions for retinal images.

##  2 Methodology

The goal of this research is to induce a novel deep learning framework, Multi-
Modal Medical Transformer that attentively integrates clinical context from
diagnostic keywords with visual insights obtained from multi-modal retinal
scan images for comprehensive medical description generation. The framework
comprises multiple modules that handle the processing of multi-modal input and
contribute to the generation of semantically coherent medical descriptions.
Figure 1 illustrates the complete architecture of proposed model. Technical
specifications for each module are elaborated in subsequent subsections.

###  2.1 Visual Encoder

The Visual Encoder is designed for converting the pre-processed retinal images
into visual features through a special attention mechanism. This module is
structured with a convolutional base and Lesion contextual gate operating
collaboratively to extract lesion contextual visual representations.

####  2.1.1 Convolutional Base

The Convolutional base serves to extract initial visual representations
capturing essential patterns in images. Convolutional Neural Networks (CNNs)
excel in learning spatial representations across various image modalities, but
training them from scratch requires substantial labeled data, often limited in
medical image tasks like retinal scans. To overcome this limitation, we
utilize pre-trained CNN models, initially trained on ImageNet, benefiting from
generalizable visual features applicable to retinal images. Specifically, we
choose EfficientNet$V_{2}B_{0}$ as the Convolutional base, which integrates
Squeeze and Excitation (SE) blocks for attentive channel-wise information
consideration, to extract spatial representations from multi-modal retinal
images. Passing a retinal scan image $X_{R}$ with dimensions $(356\times
356\times 3)$ through EfficientNet $V_{2}B_{0}$ yields visual features $V$ of
dimensions $(12\times 12\times 1280)$ from the final convolutional block.

####  2.1.2 Lesion Contextual Gate

The retinal representations obtained from the pre-trained EfficientNet
$V_{2}B_{0}$ exhibit global information, where equal importance is assigned to
each feature map. Despite the success of attention mechanisms like Squeeze and
Excitation blocks, our model still faces challenges in capturing lesion-
specific contextual information crucial for accurate diagnosis. To address
this limitation, we introduce a Lesion Contextual Gate Attention block that
enhances retinal image representations with lesion context-rich information by
using single attention block that considers global and local context
information.

Initially, lesion context information is computed through global attention
pooling which involves a point-wise convolution capturing essential features,
and a softmax activation normalizes spatial elements generating an attention
map. This attention map selectively focuses on relevant features in the
initial spatial representations, providing lesion context information. Next,
channel-wise dependencies through specialized point-wise convolution layers
which process previously computed context information, introducing non-
linearity and discerning intricate dependencies among different channels. The
resulting features are aggregated with original spatial features, enriching
contextual understanding within the retinal features. Finally, the gating
function enhances the model’s focus on lesion-specific regions where
coefficients are computed based on global and local context information,
selectively amplifying or suppressing features in retinal representations. The
integration of weighted representations with normalization, and gating ensures
effective highlighting of lesion-specific details, capturing both global and
localized context features within the retinal images. Below are the
mathematical computations involved in Lesion Contextual Gate module.

$F_{gap}=\sum\limits_{j=1}^{d}\frac{e^{W_{c}f_{r,j}}}{\sum\limits_{m=1}^{d}e^{W_{c}f_{r,m}}}\qquad\forall
f_{r,j}\in F_{R},$ (1)

where $F_{gap}$ represents the global attention pooling of visual features
$F_{R}$, $W_{c}$ represents the the parameters of point-wise convolution. The
numerator computes the exponential of the weighted visual features, and the
denominator is the sum of exponentiated weighted visual features by using a
softmax activation, resulting in a normalized global context feature.

$F_{c}=F_{R}\oplus\sum\limits_{i=1}^{d}W_{2}\left(\text{LN}\left(\Gamma\left(\sum\limits_{j=1}^{k}W_{1}f_{g_{j}}\right)\right)\right)_{i}\forall
f_{g_{j}}\in F_{gap},$ (2)

where $F_{c}$ represents the channel attentive feature incorporating global
context, $W_{1},W_{2}$ represents the parameters of point-wise convolutions,
$\Gamma$ is ReLU non-linearity and LN indicates Layer Normalization. The
operation involves weighted summation and layer normalization of the global
context features $F_{gap}$. This enriched features are then element-wise added
to the original visual features $F_{R}$.

$F_{att}=\sigma(W_{\psi}\cdot\Gamma(W_{x}F_{R}+W_{g}F_{c}+b_{xg})+b_{\psi}),$
(3)

where $F_{att}$ represents the final attention-weighted visual features. This
is computed by applying the gating mechanism using parameters
$W_{x},W_{g},b_{xg},W_{\psi},b_{\psi}$ . This gating is modulated by global
context features from $F_{c}$ and original visual features $F_{R}$.

###  2.2 Keywords Encoder

The Keywords Encoder functions as a language modeling encoder, transforming
processed diagnostic keywords into context-aware embeddings. Within the
encoder module, the Embedding layer takes the processed diagnostic keyword
sequence as input, converting it into numerical values known as embeddings
(Word2Vec). These embeddings, denoted as $E$, encapsulate individual keyword
embeddings $ke_{1},ke_{2},...,ke_{n}$, effectively capturing the semantic
relationships embedded in the input keywords.

However, the initially generated embeddings are context-free, lacking explicit
consideration of relationships between keywords. To address this limitation,
the embeddings undergo processing by an Attention layer. This layer leverages
a dot product operation to compute alignment scores between source $(ke_{s})$
and target $(ke_{t})$ embeddings, denoted as $\bar{ke}$ [18]. These alignment
scores reflect the importance or relevance of each keyword relative to others
in the sequence. Subsequently, a Softmax activation function is applied to
normalize the alignment scores, yielding attention weights $ke_{att}$. These
weights furnish a context-aware representation of diagnostic keywords,
assigning higher weights to more pertinent keywords and lower weights to less
relevant ones. This mechanism enables the model to focus on the most
informative keyword embeddings, facilitating the capture of crucial context
and relationships between keywords. This context-aware representation is
particularly valuable for the downstream task of Medical Description
Generation.

$\displaystyle\text{align\\_score}(ke_{s},ke_{t})=\bar{ke}=ke_{s}\odot
ke_{t},$ (4) $\displaystyle
ke_{att}=\text{Softmax}(W_{ke},\bar{ke})\qquad\forall ke_{att}\in KE_{att}.$
(5)

Mathematically, Equation 4 expresses the computation of alignment scores
through a dot product operation, while Equation 5 delineates the computation
of attention weights using Softmax normalization, where $W_{ke}$ denotes the
parameters of the Softmax layer.

###  2.3 TransFusion Encoder

TransFusion Encoder is the backbone of proposed approach that is designed to
receive two types of inputs, lesion contextual attention features and clinical
context keyword embeddings from the Visual and Keyword encoders. Main
objective of this module is to integrate the clinical context with visual
insights from retinal images. It exploits the interactions between visual
features and keywords by embedding the keywords using a self-attention
mechanism, where attention weights are computed based on the interactions
between the Query image $(Q)$ and the Keywords Key $(K)$, Value $(V)$ pairs.
This involves a dot product operation between the query and key embeddings,
allowing the model to discern the importance of each keyword concerning the
retinal image. The attention weights signify the degree of interaction between
the image and each keyword, capturing their semantic relationships. Notably,
higher attention weights emphasize keywords most relevant to the image, while
lower weights assign less importance to less relevant keywords.

$Q=W_{q}F_{att}$ (6) $K=W_{k}KE_{att}$ (7) $V=W_{v}KE_{att}$ (8) $\text{Self
Attention}(Q,K,V)=Z=\text{Softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$
(9) $Z_{norm}=\text{Layer Normalization}(Q+Z)$ (10)
$H=W_{2}\Gamma(W_{1},Z_{norm})$ (11) $F^{{}^{\prime}}=\text{Layer
Normalization}(H+Z_{norm})$ (12)

The mathematical operations underlying the computation of Multi-Head Attention
weights $Z$ are detailed in Equations 6 to 9. Here, $W_{q},W_{k},W_{v}$
represent the parameters associated with Attention Features and Context-aware
Keyword Embeddings. The dot product operation in Equation 9 calculates keyword
weights on the image query, with the result scaled by the length of the $K$
vector $(d_{k})$. The softmax normalized ratio is then multiplied with $V$ to
produce attention probability weights. The subsequent operations involving
Residual Addition, Layer Normalization (Equations 10 and 12), and Feed Forward
Neural Network (FFNN) (Equation 11) contribute to refining the attention
representations, resulting in $F^{{}^{\prime}}$.

###  2.4 Medical Description Decoder

The Medical Description Generation Decoder is a language model based on
Transformer Decoder architecture [19]. It operates as a function of
$(F^{{}^{\prime}},E_{c},PE_{c})$, representing Attention features, Clinical
Description Embeddings, and Positional Embeddings (token embeddings + position
embeddings), respectively. It comprises of two Masked Multi-Head Attention
instances, the first enables self-attention, facilitating the model to focus
on different segments of the input sequence while generating the clinical
description. The second instance involves cross-attention, allowing the
decoder to attend to the encoded retinal image features ($F^{{}^{\prime}}$)
and comprehend their relevance in generating subsequent words in the clinical
description. During self-attention, a Masking (M) operation prevents the model
from considering future words, ensuring generation based solely on preceding
words, maintaining auto-regressive language generation. Following the
attention mechanisms, a feed-forward neural network with two fully connected
layers processes the attended and contextualized representations to generate
the next word in the clinical description. To stabilize the learning process,
residual addition and Layer Normalization are applied post-attention and feed-
forward neural network operations. Residual connections enable the model to
bypass certain layers, retaining valuable information, while Layer
Normalization normalizes outputs, enhancing training stability and overall
decoder performance.

$CE=E_{c}+PE_{c}$ (13) $Z^{{}^{\prime}}=\text{Multi Head
Attention}(CE,CE,CE,\text{M})$ (14) $Z^{{}^{\prime}}_{\text{norm}}=\text{Layer
Normalization}(CE+Z^{{}^{\prime}})$ (15) $Z^{{}^{\prime\prime}}=\text{Multi
Head
Attention}(F^{{}^{\prime}},Z^{{}^{\prime}}{\text{norm}},Z^{{}^{\prime}}{\text{norm}},\text{M})$
(16) $Z^{{}^{\prime\prime}}{\text{norm}}=\text{Layer
Normalization}(Z^{{}^{\prime\prime}}+Z^{{}^{\prime}}{\text{norm}})$ (17)
$H^{{}^{\prime}}=W_{4}\text{ReLU}(W_{3},Z^{{}^{\prime\prime}}_{\text{norm}})$
(18) $R^{{}^{\prime}}{\text{final}}=\text{Layer
Normalization}(H^{{}^{\prime}}+Z^{{}^{\prime\prime}}{\text{norm}})$ (19)
$P=\text{Softmax}(R^{{}^{\prime}}_{\text{final}})\qquad
P\in(p_{0},p_{1},p_{2},...,p_{m})$ (20)

Equations 13 to 20 encapsulate the mathematical expressions for generating
clinical descriptions based on provided inputs. The resulting probability
vector $P$ signifies the probabilities of predicted words in the final
clinical description.

The end-to-end training of the Multi-Modal Medical Transformer involves
minimizing the cross-entropy loss function $\mathcal{L}(P,C)$, as represented
in equation 21. This loss function quantifies the dissimilarity between
predicted clinical description probabilities $(P)$ and the actual clinical
description $(C)$. Summation over all vocabulary words (M), weighted by
corresponding actual clinical description values $(c_{i})$, is performed. The
logarithm of the predicted probability $\log(p_{i})$ ensures more substantial
errors are penalized accordingly.

$\mathcal{L}(P,C)=-\sum_{i=1}^{M}c_{i}\cdot\log(p_{i})$ (21)

This loss function provides a training signal, guiding the model to minimize
it, enhancing the capability to generate accurate medical descriptions for
given retinal scan images.

Table 1: Results demonstrating the effect of incorporating Keywords as input to the model and integrating Lesion Contextual Gate with Visual features, along with Context-aware Attention to Keyword Embeddings; Image | Visual Attention | Keywords | Keyword Attention | BLEU Scores | Other Metrics
---|---|---|---|---|---
BLEU@1 | BLEU@2 | BLEU@3 | BLEU@4 | CIDEr | ROUGE
✓ | ✗ | ✗ | ✗ | 0.114 | 0.067 | 0.054 | 0.026 | 0.296 | 0.197
✓ | ✓ | ✗ | ✗ | 0.184 | 0.110 | 0.068 | 0.032 | 0.316 | 0.230
✓ | ✓ | ✓ | ✗ | 0.214 | 0.152 | 0.105 | 0.072 | 0.349 | 0.296
✓ | ✓ | ✓ | ✓ | 0.394 | 0.312 | 0.291 | 0.208 | 0.537 | 0.493
Table 2: Comparative Study of Recent Best Models with Proposed Context Gating Vision Transformer trained on DeepEyeNet Dataset, highlighting the impact of incorporating keyword embeddings and Attention on Visual features; Model | BLEU Scores | Other Metrics
---|---|---
BLEU@1 | BLEU@2 | BLEU@3 | BLEU@4 | CIDEr | ROUGE
DeepOpth [1] | 0.184 | 0.114 | 0.068 | 0.032 | 0.361 | 0.232
Deep Context Encoding Network [12] | 0.219 | 0.134 | 0.074 | 0.035 | 0.398 | 0.252
Contextualized Keywords [15] | 0.203 | 0.142 | 0.100 | 0.073 | 0.389 | 0.211
Non-local Attention [16] | 0.230 | 0.150 | 0.094 | 0.053 | 0.370 | 0.291
M3 Transformer (Ours) | 0.394 | 0.312 | 0.291 | 0.208 | 0.537 | 0.493

##  3 Experiments & Results

###  3.1 Dataset

The DeepEyeNet dataset, introduced in [1], is a large-scale collection of
retinal images comprising 15,709 scans obtained from diverse imaging
modalities such as Fluorescein Angiography (1,811 images), Fundus Photography
(13,898 images), and Optical Coherence Tomography (OCT). Each image is
meticulously annotated by expert ophthalmologists, providing multiple
diagnostic keywords that capture medical observations and clinical
descriptions related to the retinal findings. Keyword and clinical description
lengths vary, with the longest keywords exceeding 15 words and the longest
clinical descriptions surpassing 50 words. On average, both keywords and
clinical descriptions have a word length ranging from 5 to 10 words. The
dataset covers 265 distinct retinal diseases and symptoms, encompassing a
broad spectrum of both common and less common conditions. This dataset is
divided into standard splits: 60% for training (9,425), 20% for validation
(3,142), and 20% for testing (3,142).

###  3.2 Pre-processing

Effective pre-processing is a foundation for optimizing the quality and
interpretability of retinal images across diverse modalities such as color
fundus, fundus auto-fluorescence, and OCT. In our approach, we implement
crucial steps on acquired retinal scan images to mitigate variations in size,
dimensions, and color arising from different devices and laboratory
conditions. This includes resizing operations to standardize images to a
uniform scale, as well as converting images into a three-channel format (Red,
Green, Blue) to enhance processing flexibility and compatibility with pre-
trained neural networks. Simultaneously, for clinical descriptions and
diagnostic keywords, essential text processing steps are applied, encompassing
removal of non-alphabetic characters, conversion to lowercase for uniformity,
and handling rare words by replacing them with a UNK token. The list of
provided keywords is transformed into a sequence by replacing comma separators
with [SEP] during preprocessing.

###  3.3 Experimental Settings

In our study, experiments were conducted on the DeepEyeNet dataset, adopting
established settings from prior works [1, 12, 15, 16]. To ensure
comparability, consistent experimental conditions were maintained, and
evaluation metrics (BLEU, CIDEr, ROUGE) aligned with those in referenced
studies. The experiments were performed on an Nvidia P100 GPU (16GB Memory,
1.32GHz clock, 9.3 TFLOPS). Optimizing our model’s performance involved
carefully chosen hyperparameters, with a learning rate of 0.004, Adam
optimizer, batch size of 64, and dropout rate of 0.2. The model architecture
comprises a two-layer fully connected feed-forward neural network (FFCN) with
hidden layer sizes of 512 and 256, demonstrating tailored efficiency. A
vocabulary size of 5000, 300-dimensional word embeddings, and set sequence
lengths (5 to 50 words) contribute to the model’s effectiveness. Categorical
cross-entropy as the loss function solidifies our hyperparameter choices for
successful model training.

Table 3: Comparison of Actual, Non-Local Attention, and M3T medical descriptions Retinal Image | Keywords | Groundtruth | Non-Local Attention [16] | M3 Transformer
---|---|---|---|---
| autofluorescence imaging[sep]age-related macular degeneration (amd) | an autofluorescence image of a 78 year old man with an age related macular degeneration on his both eyes | Autofluorescence to image of the right eye of a [age] year old [gender] with acute decrease in vision mainly right eye is with pigment clumping and optic nerve drusen in the right eye | autofluorescence image of a 76 year old woman with age related macular degeneration on her right eye and the left eye
| idiopathic macular telangiectasia[sep]parafoveal telangiectasia[sep]juxtafoveal telangiectasis | the fellow eye was unremarkable on this red free image | The telangiectasis occurs unilaterally in the macula be of choroidal folds peripheral nevus shows cnvm in this eyes shows resolved with myopia over the macula and lung remained and remained over the exam was remained cnvm. Resolved and pdt. | the telangiectasis occurs unilaterally in the temporal half of the macula in an area of 1–2 disc diameters the anomalies are noted on this red free image in this early frame of the angiogram

###  3.4 Quantitative Evaluation

In the ablation study in Table 1, we meticulously dissect the components of
our M3 Transformer to evaluate their individual and cumulative impacts on
medical description generation for retinal images. Notably, the progressive
inclusion of visual attention, keywords, and attention mechanisms on both
visual and keyword features leads to an improvement in BLEU scores. The
results signify the benefit of visual attention in focusing on relevant
regions within retinal images, while the incorporation of diagnostic keywords
significantly enhances the model’s ability to integrate clinical context. The
culmination of these components in the full model demonstrates a substantial
leap in BLEU scores, reinforcing the synergistic effect of jointly considering
visual and clinical information for robust medical description generation.

In the comparative study illustrated in Table 2, our proposed M3 Transformer
Network is pitted against recent leading models in the field of retinal image
clinical description generation. Against models like DeepOpth, Deep Context
Encoding Network, Contextualized Keywords, and Non-local Attention, our M3
Transformer exhibits superior performance across all BLEU scores, showcasing
its capacity to generate more accurate and coherent clinical descriptions.
These results highlight the distinctive strength of our approach in seamlessly
integrating multi-modal information from retinal images and clinical keywords,
making significant strides in addressing the challenges of generating
clinically relevant descriptions for diverse retinal conditions.

###  3.5 Qualitative Evaluation

Fig. 2: Visualization of Lesion Contextual Gate Attention heatmaps and
corresponding overlays, highlighting the visual insights from color fundus,
gray fundus, OCT, and Auto-Fluorescent retinal scan images

In Figure 2, we provide a visual representation of the Lesion Contextual Gate
Attention heatmaps and corresponding overlays, showcasing the explainability
of our proposed approach. The heatmaps highlight the visual insights extracted
from diverse retinal imaging modalities, including color fundus, gray fundus,
optical coherence tomography (OCT), and autofluorescent retinal scan images.
The Lesion Contextual Gate effectively focuses on relevant regions within the
retinal images, emphasizing areas crucial for clinical understanding. This
visualization underscores the model’s capability to integrate information from
various modalities, providing ophthalmologists with interpretable indications
of the visual cues influencing the generated clinical descriptions.

Table 3 offers a qualitative comparison of actual, Non-Local Attention [16],
and predicted medical descriptions using retinal images and keywords as input.
Our M3 Transformer outperforms the Non-Local Attention model by generating
more accurate and coherent clinical descriptions. For instance, in the first
case, the Non-Local Attention model produces a description with generalities
about age and gender, while our M3 Transformer provides specific details about
macular hole conditions in both eyes. Similarly, in the second case, our model
accurately identifies and describes the presence of idiopathic macular
telangiectasia, parafoveal telangiectasis, and juxtafoveal telangiectasis,
offering detailed insights into the retinal anomalies. This comparison
highlights the superior performance of our proposed approach in capturing
nuanced clinical information and generating clinically relevant descriptions
with greater accuracy and specificity.

##  4 Conclusion

This research introduces the Multi-Modal Medical Transformer (M3T) for
generating precise medical descriptions from retinal images, addressing the
rising global prevalence of visual impairments. The proposed model integrates
diagnostic keywords and visual features, demonstrating superior performance
compared to existing models in the field. Through an ablation study, we
highlight the incremental contributions of visual attention, keywords, and
attention mechanisms, emphasizing their collective impact on accurate medical
description generation. Quantitative evaluations confirm the effectiveness of
M3T, surpassing state-of-the-art models in terms of all metrics reported.
Additionally, qualitative assessments demonstrate the model’s explainability
through attention heatmaps. In future, M3T model can be further extended to
diverse medical imaging domains and enhancing its interpretability for broader
clinical applicability.

## References

  * [1] Jia-Hong Huang, C-H Huck Yang, Fangyu Liu, Meng Tian, Yi-Chieh Liu, Ting-Wei Wu, I Lin, Kang Wang, Hiromasa Morikawa, Hernghua Chang, et al.,  “Deepopht: medical report generation for retinal images via deep models and visual explanation,”  in Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2021, pp. 2442–2452.
  * [2] Yingfeng Zheng, Mingguang He, and Nathan Congdon,  “The worldwide epidemic of diabetic retinopathy,”  Indian journal of ophthalmology, vol. 60, no. 5, pp. 428, 2012.
  * [3] Nagur Shareef Shaik and Teja Krishna Cherukuri,  “Gated contextual transformer network for multi-modal retinal image clinical description generation,”  Image and Vision Computing, p. 104946, 2024.
  * [4] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio,  “Show, attend and tell: Neural image caption generation with visual attention,”  in International conference on machine learning. PMLR, 2015, pp. 2048–2057.
  * [5] Abrar Hasin Kamal, Md Asifuzzaman Jishan, and Nafees Mansoor,  “Textmage: The automated bangla caption generator based on deep learning,”  in 2020 International Conference on Decision Aid Sciences and Application (DASA). IEEE, 2020, pp. 822–826.
  * [6] Djamila-Romaissa Beddiar, Mourad Oussalah, and Tapio Seppänen,  “Automatic captioning for medical imaging (mic): a rapid review of literature,”  Artificial Intelligence Review, pp. 1–58, 2022.
  * [7] Nagur Shareef Shaik and Teja Krishna Cherukuri,  “Lesion-aware attention with neural support vector machine for retinopathy diagnosis,”  Machine Vision and Applications, vol. 32, no. 6, pp. 126, 2021.
  * [8] Nagur Shareef Shaik and Teja Krishna Cherukuri,  “Hinge attention network: A joint model for diabetic retinopathy severity grading,”  Applied Intelligence, vol. 52, no. 13, pp. 15105–15121, 2022.
  * [9] Nagur Shareef Shaik and Teja Krishna Cherukuri,  “Transfer learning based novel ensemble classifier for covid-19 detection from chest ct-scans,”  Computers in Biology and Medicine, vol. 141, pp. 105127, 2022.
  * [10] Ishan Misra and Laurens van der Maaten,  “Self-supervised learning of pretext-invariant representations,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 6707–6717.
  * [11] Mingjie Li, Bingqian Lin, Zicong Chen, Haokun Lin, Xiaodan Liang, and Xiaojun Chang,  “Dynamic graph enhanced contrastive learning for chest x-ray report generation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 3334–3343.
  * [12] Jia-Hong Huang, Ting-Wei Wu, Chao-Han Huck Yang, and Marcel Worring,  “Deep context-encoding network for retinal image captioning,”  in 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021, pp. 3762–3766.
  * [13] Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares,  “Image captioning: Transforming objects into words,”  Advances in neural information processing systems, vol. 32, 2019.
  * [14] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.,  “Learning transferable visual models from natural language supervision,”  in International conference on machine learning. PMLR, 2021, pp. 8748–8763.
  * [15] Jia-Hong Huang, Ting-Wei Wu, and Marcel Worring,  “Contextualized keyword representations for multi-modal retinal image captioning,”  in Proceedings of the 2021 International Conference on Multimedia Retrieval, 2021, pp. 645–652.
  * [16] Jia-Hong Huang, Ting-Wei Wu, C-H Huck Yang, Zenglin Shi, I Lin, Jesper Tegner, Marcel Worring, et al.,  “Non-local attention improves description generation for retinal images,”  in Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2022, pp. 1606–1615.
  * [17] Ting-Wei Wu, Jia-Hong Huang, Joseph Lin, and Marcel Worring,  “Expert-defined keywords improve interpretability of retinal image captioning,”  in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 1859–1868.
  * [18] Minh-Thang Luong, Hieu Pham, and Christopher D Manning,  “Effective approaches to attention-based neural machine translation,”  arXiv preprint arXiv:1508.04025, 2015.
  * [19] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017.