# DRG-Net: Interactive Joint Learning of Multi-lesion Segmentation and
Classification for Diabetic Retinopathy Grading

Hasan Md Tusfiqur1,*, Duy M. H. Nguyen1,2 *, Mai T. N. Truong3, Triet A.
Nguyen4, Binh T. Nguyen5,
Michael Barz1, Hans-Jürgen Profitlich1, Ngoc T. T. Than6, Ngan Le7, Pengtao
Xie8, Daniel Sonntag1,9 *Equal contribution1German Research Center for
Artificial Intelligence (DFKI)2Machine Learning and Simulation Lab, University
of Stuttgart, Germany3Department of Multimedia Engineering, Dongguk
University, South Korea4Department of Mathematics, University of Architecture
Ho Chi Minh City5AISIA Lab, University of Science - VNUHCM, Vietnam6Byers Eye
Institute, Stanford University, USA7Department of Computer Science and
Computer Engineering, University of Arkansas, USA8Department of Electrical and
Computer Engineering, University of California San Diego, USA,9Applied
Artificial Intelligence, Oldenburg University, Germany

###### Abstract

Diabetic Retinopathy (DR) is a leading cause of vision loss in the world, and
early DR detection is necessary to prevent vision loss and support an
appropriate treatment. In this work, we leverage interactive machine learning
and introduce a joint learning framework, termed DRG-Net, to effectively learn
both disease grading and multi-lesion segmentation. Our DRG-Net consists of
two modules: (i) DRG-AI-System to classify DR Grading, localize lesion areas
and provide visual explanation and (ii) DRG-Expert-Interaction to receive
feedback from user-expert and improve the DRG-AI-System. To deal with sparse
data, we utilize transfer learning mechanisms to extract invariant feature
representations by using Wasserstein distance and adversarial learning-based
entropy minimization. Besides, we propose a novel attention strategy at both
low- and high-level features to automatically select the most significant
lesion information and provide explainable properties. In terms of human
interaction, we further develop DRG-Net as a tool that enables expert users to
correct the system’s predictions, which may then be used to update the system
as a whole. Moreover, thanks to the attention mechanism and loss functions
constraint between lesion features and classification features, our approach
can be robust given a certain level of noise in the feedback of users. We have
benchmarked DRG-Net on the two largest DR datasets, i.e., IDRID and FGADR, and
compared it to various state-of-the-art deep learning networks. In addition to
outperforming other SOTA approaches, DRG-Net is effectively updated using user
feedback, even in a weakly-supervised manner.

##  I Introduction

Diabetes is a chronic health condition that is estimated to affect about one
in every ten people worldwide Saeedi et al. (2019). According to Yang et al.
(2021), 40% to 45% of people with diabetes may develop Diabetic Retinopathy
(DR) during their lifetime. DR is a kind of ocular disease that damages the
retina’s blood vessels, and it is one of the leading causes of irreversible
blindness. Although the symptoms are diagnosed only in the later stage, people
suffering from this disease start to lose vision from an early stage. To
assess the complexity of DR, International Clinical Diabetic Retinopathy
Disease Severity Scale Gulshan et al. (2016) has used five grades (0-4),
including no DR, mild, moderate, severe, and proliferative as shown in Figure
1. In practice, accurate DR grading is a time-consuming task and there is a
huge shortage of qualified ophthalmologists. Therefore, an automated system
can significantly aid the diagnosis process.

While the existing methods Gulshan et al. (2016); Antal and Hajdu (2012); Wang
et al. (2017) can assist ophthalmologists in validating and interpreting the
automated prediction, they introduce three main issues: (i) They are designed
as black-box models and unable to utilize lesion features to improve the
models. (ii) Lesion segmentation performance is limited because of
irregularities in shape, noisy, ill-defined boundaries, intra-class lesion
diversities and less labeled data, (iii) Though disease classification and
lesion discovery are strongly correlated, they are mostly studied as
independent tasks.

Joint learning has recently shown promising progress in medical imaging e.g.,
breast cancer Mehta et al. (2018), skin lesion Al-Masni et al. (2020),
COVID-19 detection Nguyen et al. (2021b); Wu et al. (2021), and retinal blood
vessel Li et al. (2020). However, the performance of joint learning-based
medical imaging is generally limited due to domain-shift and scarcity of
labeled data. To address the aforementioned limitations, we propose a joint
learning-based multi-lesion segmentation and classification for DR Grading,
termed DRG-Net. Our DRG-Net includes a novel Domain Invariant Lesion Feature
Generator and Attention-Based Disease Grading Classifier. While the former
aims to identify and distinguish important lesion regions of different
semantics, the later one aims to incorporates the predicted lesion information
in the learning process of the disease classification network. Furthermore,
our framework is designed as a collaborative learning system between user and
machine learning (ML) model, i.e., (i) ML model provides both lesion
prediction with its corresponding relevant clinical evidence; (ii) the system
allows users to inspect the predicted results and be able to enhance accuracy
given new samples which are roughly annotated by users, e.g., drawing a
bounding box around lesion regions rather than a precise segmentation
boundary.

Figure 1: Illustration of diabetic retinopathy retinal images. Top row -
examples of different stages of DR disease. Bottom row - a retinal image with
lesion information annotated by domain expert (left most image). The remaining
4 images are lesion mask predicted by the Dense U-Net Segmentation model Zhou
et al. (2020); Li et al. (2018b).

Our contributions are summarized as followed:

  * •

We construct a novel lesion segmentation model by formulating a domain
invariant learning scheme combining a new transfer learning scheme and domain
adaption concepts.

  * •

We addresses both the challenges of scarcity of labeled data and domain shift.

  * •

We train our segmentation model with adversarial learning Goodfellow et al.
(2014) to enhance its robustness and accuracy.

  * •

Our framework is explainable and provides diverse information to the end-users
with interpretation to support its decision.

  * •

Our framework allows user to provide feedback on the output and if required,
re-annotate the predicted lesion features, which can be used to fine-tune the
model further.

  * •

Our learning method can be robust to a certain threshold, and pixel-wise
correction is not required.

  * •

We conduct comprehensive experiments and compare with the recent state-of-the-
art methods to illustrate the effectiveness of our proposed DRG-Net.

The rest of the paper is organized as follows: In section II, we review some
of the related works and some preliminaries for our method. Next, we detail
our proposed DRG-Net in section IV. Our DRG-Net is evaluated and compared with
other SOTA existing work in section VI. Finally, we discuss the proposed
method and future

##  II Related Work

###  II-A DR Grading Analysis

DR Grading analysis aims to classify fundus images into different DR severity
classes and extract the relevant lesion regions. Most of early works focus on
classification task only, i.e. classifying DR into multiple classes of grading
Gulshan et al. (2016); Sankar et al. (2016); Alban and Gilligan (2016).
However, such existing approaches treat the problem of classification task as
a black box and did not consider the fine-grained DR-related lesion
information in the learning process Nguyen et al. (2021a); Zhou et al. (2020).
Some recent approaches attempt to integrate lesion information to improve the
grading classification. For instance, Yang et al. (2017) propose two-stages
Convolutional Neural Networks (CNNs), which contains two CNNs: the former
plays the role of local network and aims to learn a weighted attention lesion
map; the later plays the role of global network and exploits image features in
holistic level for DR grading by utilizing the former’s outcomes. Lin et al.
(2018) propose a similar framework to automatically detect missing lesion
features and integrate with global image features using a classification
network for grading prediction.

However, the aforementioned approaches are designed in a cascade framework
i.e., extract lesion information first and then classify DR Grading. With such
network design, both tasks are optimized independently while they have a
strongly coherency. To address such limitation, Zhou et al. (2019) propose a
collaborative learning pipeline to jointly learn both lesion segmentation and
DR Grading classification by semi-supervised learning with an attention
mechanism. Lately, Yang et al. (2021) also introduce an attention-based
learning scheme, which collaboratively learns to predict lesion maps patches
and DR grading. Leveraging multi-head attention mechanism, Sun et al. (2021)
combine self-attention and cross-attention mechanisms to direct the overall
attention of a network to consider lesion region diversity and their
importance separately in the final DR grading prediction. While obtaining
promising performance, these methods suffer from _domain shift problem_ i.e.
difference in data distribution between source (used to train the model) and
target (encountered during inference). Furthermore, none of the exiting
approaches can explain how lesion maps influences the final DR grading
prediction while lesion regions do not share the same contribution to DR
grade. In this paper, we propose an explainable interactive joint learning
framework, which does not only results lesion localization and its
corresponding DR Grading score but also provides an informative explanation of
the outcomes.

###  II-B Visual explanations of CNNs.

Explaining the CNNs prediction is crucial for the system acceptance in
critical applications such as DR Grading. In general, generating saliency maps
that highlight the image regions contributing to the final CNNs prediction is
one of the most common techniques. There have been a large number of research
proposed to visualize a CNNs predictions by highlighting important regions
that are believed to be intuitive to end-users.

For instance in image classification, visual explanation is considered to be
good if it is able to be localize the category in the image (i.e. class-
discriminative). Class Activation Mapping (CAM) Zhou et al. (2016) is
considered as one of the most common technique for identifying informative
areas by a CNNs-based classification. In short, CAM utilizes the last
convolutional layer and combine weighted activation maps to produce
explainable heatmaps. As a result, it is highly class-discriminative, however,
the output heatmaps is blurry (low-resolution). As an improvement of CAM,
Selvaraju et al. (2017) propose Grad-CAM, which does not require any specific
network architecture. However, gradients can be noisy and may get vanished in
activation function like signmoid, ReLU. To address this issue, Wang et al.
(2020) proposed Score-CAM, which gets rid of the dependence on gradients by
obtaining the weight of each activation map. In Score-CAM, the final result is
obtained by a linear combination of weights and activation maps. Furthermore,
to address the problem of low-resolution heatmaps, Jalwana et al. (2021)
propose CAMERAS to produce high-resolution heatmaps with multi-scale
accumulation and fusion of the activation maps and backpropagated gradients to
compute precise saliency maps. CAM-based explainable mechanisms have been
recently utilized by many research in medical imaging. For example, Nguyen et
al. (2021b) proposed a CAM-based explainable COVID-19 detection method using
CT images, Wu et al. (2021) adopt the Grad-CAM technique to perform the joint
task of classification and segmentation for COVID-19 detection. In this work,
we propose a constraint between heatmap regions computed by GradCam in Grading
Network with multi-lesion regions extracted by Lesion generators. This step
aim to generate lesion prior-guided DR grading predictions. Additionally, the
proposed distilling strategy using attention mechanism enables automatically
selecting major lesion parts used in specific disease grading tasks and making
the model to be robust. This sets us apart with other works Nguyen et al.
(2021b); Wu et al. (2021). In the experiments, we demonstrate that this tactic
improves prediction performance and provide helpful explanations to the
experts (Figures 2 and 15).

###  II-C Transfer learning and Domain Adaptation

Domain adaptation can be regarded as a special case kind of transfer learning,
thus, we first review the definitions of each to provide the differences. In
the standard transfer learning setting, there are two main concepts: domain
and task. While domain means the feature space of a particular dataset and the
marginal probability distribution of features, task means the label space of a
dataset and an specific objective function. In general, transfer learning aims
to transfer the knowledge learned from a task $\mathcal{T}_{a}$ on domain
$\mathcal{A}$ (i.e. source) to another task $\mathcal{T}_{B}$ on domain
$\mathcal{B}$ (i.e target). Domain adaptation, as a special case of transfer
learning, it assumes that both source and target share the same domain feature
spaces and tasks and whereas only marginal distributions between the source
and target domains are different. In medical image analysis (MIA), domain
adaptation and transfer learning have been promisingly used for dealing with
limited labeled data issue.

Transfer Learning in MIA: Rather than training a network with limited training
data from a target task, the network is first trained for a task with
potentially larger source datasets, creating a more robust model. This pre-
trained network is then down-streamed on target task. However, a large-scale
analysis by Cheplygina (2019); He et al. (2020) suggests that transfer
learning might be not better than random initialization for most medical
tasks. Medical images are significantly different from the natural image (e.g.
ImageNet dataset); therefore, features learned from the external data maybe
not effective helpful for the target medical domain task. Furthermore, medical
data often face to the problem of imbalanced data at both instance level and
pixel level Le et al. (2021). To address those limitations, Nguyen et al.
(2022b) propose a novel transfer learning, termed Task Agnostic Transfer
Learning (TATL), motivated by dermatologists’ behavior in the skincare
context. TATL is learnt by a two-stage learning step i.e., first, an
attribute-agnostic network is trained, which detects all the lesion regions
irrespective of their labels; then, the knowledge from this network is
transferred to a set of attribute-specific classifiers to label each
particular region. However, the performance of DNN trained on a particular
source domain and transferred to a different target domain (e.g., different
vendor, acquisition parameters), can drop unexpectedly due to domain shift.
Yan et al. (2019).

Domain Adaptation in MIA: In this settings, the networks are trained with
domain adaptation constraints to address the domain-shift problem. Tzeng et
al. (2014) proposed a Deep Domain Confusion (DDC) method to reduce the
divergence between two distributions by minimizing the maximum mean
discrepancy (MMD) loss (Gretton et al., 2012). MMD is a nonparametric metric
and can be used to compute distances between distributions as distances
between mean embeddings of features. In their method, a network is trained
with data from multiple distributions using a loss function that consists of
both task-specific loss and MMD loss. Some studies apply adversarial
optimization to remove the domain discrepancy by incorporating generative
adversarial networks (GANs). GANs consists of two networks, i.e., a generator
to generate a new data from a distribution and a discriminator to evaluate the
new data Goodfellow et al. (2014). The two networks trained using game theory
theorem, i.e. minimax. Tzeng et al. (2017) combined standard adversarial loss
with the task-specific classification loss to minimize domain distances. At
first, a discriminative model (e.g. CNNs) is trained using labeled data from
the source domain. Then, a target encoder is learned using GANs adaptation,
where discriminator aims to distinguish between encoded source and target
examples. Shen et al. (2018) describe Wasserstein distance guided
representation learning (WDGRL) method to reduce the domain discrepancy by
minimizing Wasserstein distance for each feature block of the encoder CNN.

In this work, we are the first take into accounts transfer learning and domain
adaptation problems in the context of DR-Grading and Segmentation. These
issues in fact are typically encountered when deploying an AI-assisted system
in medicine applications, thus a robust framework under circumstances is
critical. To this end, we properly designed support deep networks (PD-Net, AD-
Net and W-Net) that jointly learn with lesion generator S-Nets. These support
network operate based on the cutting-edge domain adaptation and adversarial
leaning techniques which have been shown impressing results on various
applications such as object recognition, self-driving car, etc. In expeiments,
we demonstrate that all of these components contribute to improve performance
and robustness.

###  II-D Interactive Machine Learning

Interactive Machine Learning (IML), which focuses on interaction between users
and learning systems Amershi et al. (2014), with people interactively
supplying information in a more focused, frequent, and incremental way
compared to traditional ML. Typically, IML is an active ML technique in which
models are designed and implemented with human-in-the-loop manner. While ML
algorithms usually requires a large-scale data to yield accurate result;
however, such requirement is limited in some domains like clinical trials,
medical analysis. A comprehensive review of IML medical analysis is
extensively studied in Maadi et al. (2021). To build such a computer-based
interactive system, the initial challenge is how to to represent the semantics
in a machine-readable form using medical ontologies Sonntag et al. (2010) so
that the information is easily exchangeable between human experts and
machines. RadSpeech’s Mobile Dialogue System for Radiologist Sonntag et al.
(2012) provides a multi-modal interaction system for radiology image
annotations. It is a user-friendly semantic search interface where users can
annotate medical image regions with a specific medical, structured diagnosis
using speech and pointing gestures. Prange et al. (2017) presents a medical
decision support system inside virtual reality (VR). In this system, a doctor
can visualize patients’ records and clinical images, as well as therapy
predictions which are computed in real-time using a pre-trained deep learning
model. The aforementioned studies provide the techniques to capture user
feedback (annotations on inputs) for an intelligent learning system. However,
they did not consider the inclusion process of these feedbacks in an IML loop.
Sonntag et al. (2020) describes the functionality and interface of an
interactive decision support system for differential diagnosis of malignant
skin lesions, where the user can re-annotate an input that is used to fine-
tune the prediction model based on the explanation. Recently, Dai et al.
(2021) presents a real-time deep learning based interactive system for
diabetic retinopathy where the architecture consists of three sub-networks to
perform different tasks one by one. However, the coherent relation between
lesion discovery and disease classification was mostly ignored, making it hard
to conclude that lesion-feature information predicted by the segmentation part
has any influence on the classification result.

In this paper, we introduce the first time a human interactive framework for
the DR-Grading and Segmentation tasks. The expert user benefit from our AI-
assist system with the following advantageous. First, we provide an
explainable predictions by correlation between heatmap of the classification
network with medical lesion regions. This helps the doctor inspect predictions
generated by system by bridging with their prior knowledge. Second, due to
transfer learning and domain adaptation, we permit a fast deployment our
system on a new domain (new hospital, new data, etc.) while not demanding
large annotations to pre-train the system (only classification annotation is
required, don’t need segmentation in beginning), leading to saving labour
costs. Finally, the model is designed to learn from user’s feedback in weak
and noise annotation forms (bounding boxes rather than precise segmentation
masks, noise in new data can be damped through constraints among
classification, segmentation, and attention loss functions).

##  III DRG-Net: Overview and Problem Setup

###  III-A Overview

Our DRG-Net consists of two modules: (i) _DRG-AI-System_ to classify DR
Grading, localize lesion areas and provide visual explanation and (ii) _DRG-
Expert-Interaction_ to receive feedback from user-expert and improve the DRG-
AI-System. The overall framework of our proposed DRG-Net is depicted in Fig.
2. Given a retina image input, our first module DRG-AI-System will
simultaneously returns three outputs: (a) DR grade severity on a scale of 0 to
4 by Attention-based Classifier, comprising Att-Net and G-Net, (b) lesion
areas by Feature generator, comprising S-Nets and support systems (PD-Net, AD-
Net, W-Net), (c) and a correlation among visual explanation, lesion regions
and grading prediction. To address real-world challenges of DRG analysis
including scarcity labeled data, small lesion localization, and domain shift,
we propose to design DRG-AI-System in an unsupervised domain adaptation (UDA)
manner, where the system was trained on source data with annotation and test
on target data without label. In the second module of DRG-Expert-Interaction,
the expert can provide feedback on the grading and lesion information, which
is then utilized to fine-tune the DRG-AI-System network.

Figure 2: A high-level overview of our proposed DRG-Net system that contains
two main module of DRG-AI-System and DRG-Expert-Interaction. DRG-AI-System:
Given a retinal image, our deep learning models will simultaneously generate
three outcomes prediction (DR grade, lesion region, visual explanation). DRG-
Expert-Interaction: By using an Intelligent User Interface (IUI),
ophthalmologists can observe the system’s predictions and provide feedback to
fine-tune the model.

###  III-B Problem Setup

In DRG-AI-System, we consider two correlated tasks at both image-level and
pixel-level. The former task aims to classify DR Grading (i.e., associate an
input image with a DR Grading class), whereas the later one aims to localize
lesion areas (i.e., partition an input image into various lesion areas).

Let $\mathbf{x}_{s}\in\mathcal{X}_{s}$,
$\mathbf{\hat{t}}_{s}\in\mathcal{T}_{s}$, and
$\mathbf{\hat{y}}_{s}\in\mathcal{Y}_{s}$ be an input image and its
corresponding image-level and pixel-level label in the source domain drawn
from the source distribution $p_{s}$. Similarly, we denote
$\mathbf{x}_{t}\in\mathcal{X}_{t}$, $\mathbf{\hat{t}}_{t}\in\mathcal{T}_{t}$
and $\mathbf{\hat{y}}_{t}\in\mathcal{Y}_{t}$ be the input image and its
corresponding image-level and pixel-level label in the target domain drawn
from the target distribution $p_{t}$. As for image-level label, there are $C$
classes of No DR (0), Mild (1), Moderate (2), Severe (3), Proliferative (4).
As for pixel-level label, there are $L$ classes of Microaneurysms (MA), Hard
Exudates (HE), Soft Exudates (SE), Hermorrhages (EX).

In our domain adaptation setting, the ground-truth pixel-level label
$\mathbf{\hat{y}}_{t}$ of image $\mathbf{x}_{t}$ is not available.

The DRG-AI-System is trained on source data and tested on target data. Given a
target retinal image $\textbf{x}_{t}\in\mathcal{X}_{t}$, the output from DRG-
AI-System is a tuple of grading level, localizing lesion areas and explainable
visual representation.

The following Table I summarizes and briefly describes important symbols that
are used throughout this manuscript:

TABLE I: Summation of symbols and their brief description. Variables | Description | Network | Description
---|---|---|---
$L$ | Number of classes of lesions | S-Net = $\\{S^{e},S^{d}\\}$ | Segmentation network, enoder $S^{e}$ and decoder $S^{d}$
$\mathcal{X}_{s}$ | source domain | $S_{l}=\\{S_{l}^{e},S_{l}^{e}\\}$ |  Binary segmentation network for a lesion $l$
$\mathcal{X}_{t}$ | target domain | PD-Net | Patch Discriminator
$\mathbf{x}_{s}\in\mathcal{X}_{s}$ | an image in source domain | W-Net | Wasserstein Discriminator
$\mathbf{x}_{t}\in\mathcal{X}_{t}$ | an image in target domain | AD-Net | Adversarial Domain Discriminator
$\mathbf{x}$ | an image from any domain | Att-Net | Attention Network
$\mathbf{\hat{t}}_{s}$, $\mathbf{\hat{t}}_{t}$ | GT image-level label of source/target image | G-Net | Grading Network
$\mathbf{\hat{y}}_{s}$, $\mathbf{\hat{y}}_{s}$ | GT pixel-level label of source/target image |  |
$\mathbf{\hat{y}}$ | predicted pixel-level label |  |
$\mathbf{h}_{s}$, $\mathbf{h}_{t}$ | encoder feature map of a source/target image |  |
$\mathbf{g}^{i}$ | Conv. block |  |
$\mathbf{o}^{i}$ | output of each Conv. block $\mathbf{g}^{i}$ |  |
$h$, $w$ | height and width of an image |  |
$C$ | number of grading labels |  |

Each module will be detailed in the following sections.

##  IV DRG-Net: DRG-AI-System

Our proposed UDA DRG-AI-System consists of three main components: The first
component, Feature Generator, plays the role of multi-lesion segmentation and
is termed S-Net. The second component plays the role of UDA including Patch
Discriminator PD-Net, Wasserstein Discriminator W-Net and Adversarial Domain
Discriminator AD-Net. The second component aims to support the first component
in dealing with scarcity labeled data, small objects and domain shift. The
third component, Attention-based Classifier, is designed as a combination of
an Attention Network Att-Net and a Grading Network G-Net. The overall DRG-AI-
System is given in Fig.5.

###  IV-A Feature Generator: multi-lesion segmentation S-Net

S-Net aims to partition an input retinal image into $L$ binary segmentation
masks, each represents a lesion mask. In this paper, our proposed S-Net is
based on an encoder-decoder Unet-like architecture Ronneberger et al. (2015)
consisting of encoder $S^{e}$ and decoder $S^{d}$.

Furthermore, we focus on real-world applications where S-Net is trained on
source domain $\mathcal{X}_{s}$ and tested on target domain $\mathcal{X}_{t}$.
Under this setup, annotation is only available during training including
image-level annotation $\mathbf{\hat{t}}_{s}\in\mathcal{T}_{s}$, and pixel-
level annotation $\mathbf{\hat{y}}_{s}\in\mathcal{Y}_{s}$. Due to scarcity of
training data, small lesion area and domain shift, standard Unet-based
networks are limited when working on retinal images. In this work, we propose
to utilize Task Agnostic Transfer Learning (TATL) framework Nguyen et al.
(2022b) to propose a novel UDA approach to alleviate the aforementioned
challenges. Our proposed UDA consists of three sub-networks, i.e. Patch
Discriminator PD-Net, Wasserstein Discriminator W-Net and Adversarial Domain
Discriminator AD-Net, which will be detailed in Section IV-B whereas we only
focus describing S-Net in this section.

In the TATL settings, S-Net is first pre-trained at the pretext task on a
multi-lesion region first. Then, it is then fine-tuned on the downstream task
for segmenting an individual lesion region. Each particular lesion $l$ is
segmented by a network $S_{l}$, all $S_{l}$, $l\in[1,L]$ shares a similar
network architecture as S-Net. For a given image
$\mathbf{x}_{s}\in\mathcal{X}_{s}$, for each network $S_{l}$, we obtain a
binary segmentation mask $S_{l}(\mathbf{x}_{s}),\ (l\in L)$, where
$L=\\{\mathrm{MA},\,\mathrm{HE},\,\mathrm{SE},\,\mathrm{EX}\\}$. The
segmentation network is trained by weighted binary cross-entropy loss as
follows:

$\displaystyle\mathcal{L}_{seg}={\text{min}}\frac{1}{\lvert\mathcal{X}_{s}\rvert}\sum_{(\mathbf{x}_{s},\hat{\mathbf{y}}_{s})\in\mathcal{X}_{s}}\sum_{l\in
L}\mathcal{L}_{wbce}(S^{l}(\mathbf{x}_{s}),\hat{\mathbf{y}}_{{s}_{l}})$ (1)

where $\hat{\mathbf{y}}_{{s}_{l}}$ is pixel-level annotation of lesion $l$.
$\mathcal{L}_{wbce}$ is the weighted binary cross-entropy loss

$\displaystyle\mathcal{L}_{wbce}(\hat{y},{\mathbf{y}})=-(\beta\cdot\hat{\mathbf{y}}\log({\mathbf{y}})+(1-\hat{\mathbf{y}})\log(1-\mathbf{y}),$
(2)

where $\hat{\mathbf{y}}$ is the ground-truth binary lesion mask and
${\mathbf{y}}$ is the model prediction, $\beta$ is the class balancing weight.
The loss $\mathcal{L}_{wbce}$ in general can be replaced with other
segmentation losses Jadon (2020).

So far, we have trained the segmentation model $S_{l}$ in the source domain
$\mathcal{X}_{s}$ using eq.1 where annotation is available. However, it is
unfit to target domain where we do not have annotation. We propose to
constrain the embedding space of the source and target domains are close to
each other by minimizing Wasserstein distance Arjovsky et al. (2017); Liu et
al. (2019) via W-Net. Furthermore, to improve semantic segmentation accuracy
on tiny objects, we follow prior studies (Hung et al., 2018) and incorporate a
discriminate network for each lesion segmentation model $S_{l}$. We propose
Patch Discriminator PD-Net. Furthermore, to exploit the structural consistency
between the source and target domain, we proposed AD-Net to directly minimize
an uncertainty prediction in the target domain. Details of PD-Net, W-Net and
AD-Net are in Section IV-B

###  IV-B Unsupervised Domain Adaptation

####  IV-B1 Patch Discriminator PD-Net

To improve semantic segmentation accuracy on tiny objects, we follow prior
studies (Hung et al., 2018) and incorporate a discriminate network for each
lesion segmentation model $S_{l}$. We propose Patch Discriminator PD-Net, a
conditional GAN Mirza and Osindero (2014), which aims to distinguish real
samples from the generated ones. The architecture of PD-Net is similar to Xiao
et al. (2019), which is formed based on the ideas of PatchGAN Isola et al.
(2017) as illustrated in Fig. 3 .

In particular, an input image is split into $16\times 16$ smaller patches, and
each of these patches is applied with the cross-entropy loss to decide whether
that patch is fake or real. The input for the PD-Net is the concatenation of
the original image patch with its corresponding lesion map predicted from
$S_{l}$ and actual ground truths. Thus, the discriminator PD-Net learns the
joint distribution of both images and the lesion map, conditioned on the input
data. In other words, the PD-Net will force the output of $S_{l}$ to look
’real’ as the ground-truth data as much as possible given the input image
$\textbf{x}_{s}\in\mathcal{X}_{s}$. The objective loss function for this
formulation can be defined as:

$\displaystyle\mathcal{L}_{\mathrm{patch}}=\underset{\theta_{S_{l}}}{\text{min}}\,\underset{\theta_{\mathrm{pd}}}{\text{max}}\frac{1}{\lvert\mathcal{X}_{s}\rvert}\sum_{(\mathbf{x},\hat{\mathbf{y}}_{l})\in\mathcal{X}_{s}}\left[\mathrm{log}(\mathrm{PD}(\mathbf{x}\oplus\hat{\mathbf{y}}_{l}))\right.$
(3)
$\displaystyle\left.+\log(1-(\mathrm{PD}(\mathbf{x}\oplus{\mathbf{y}_{l}})))\right]$
(4)

where $\theta_{S_{l}}$ and $\theta_{\mathrm{pd}}{\text{max}}$ are parameters
of binary segmentation network $S_{l}$ and PD-Net. $\oplus$ is the
concatenation operator. ${z_{l}}$ is output of binary segmentation network
$S_{l}$ i.e., $z_{l}=S_{l}(\mathbf{x})$ and $\hat{\mathbf{y}}_{l}$ is
groundtruth corresponding to lesion $l$. Combining this adversarial objective
with Eq. (1), we derive an optimization problem for the segmentation model
$S_{l}$ using the source domain $\mathcal{X}_{s}$ as:

$\displaystyle\mathcal{L}_{\mathrm{source}}=\mathcal{L}_{seg}+\lambda_{p}\,\mathcal{L}_{\mathrm{patch}.}$
(5)

where $\lambda_{p}$ is a parameter controls the contribution of
$\mathcal{L}_{patch}$.

Figure 3: Our workflow for the supervised lesion segmentation using
adversarial learning. Output from the segmentation network $S_{l}$ is
concatenated with the input image and split into patches. Then, the patch
discriminator PD-Net computes the adversarial loss by predicting whether each
patch is real or fake.

####  IV-B2 Wasserstein Discriminator W-Net

For each lesion segmentation network $S_{l}$, we aim to minimize discrepancies
between different domain’s feature embedding estimated at the encoder layers
$S_{l}^{e}$ of $S_{l}$. To this end, we introduce the domain critic W-Net
parameterized by $\theta_{w}$, whose goal is to estimate the Wasserstein
distance Arjovsky et al. (2017) between the source and target distribution in
the feature representation space (Fig. 4). Given an encoder feature
representation $\mathbf{h}=S_{l}^{e}(\mathbf{)}$ for an image $\mathbf{x}$
from any domain, we define:

$\displaystyle\mathbf{h}_{s}$ $\displaystyle=S_{l}^{e}(\mathbf{x}_{s}),\
\mathbf{x}_{s}\in\mathcal{X}_{s},\ $ (6) $\displaystyle\mathbf{h}_{t}$
$\displaystyle=S_{l}^{e}(\mathbf{x}_{t}),\ \mathbf{x}_{t}\in\mathcal{X}_{t}.$
(7)

and the domain critic function $W(.)\colon\mathbb{R}^{d}\rightarrow\mathbb{R}$
which maps a feature representation to a real number. As proposed by Shen et
al. (2018), if the parameterized family of domain critic function $W(.)$ are
all 1-Lipschitz, for the source and target distributions $\mathcal{X}_{s}$ and
$\mathcal{X}_{t}$, an empirical Wasserstein distance can be approximated by
maximising domain critic loss $\mathcal{L}_{wd}$ with respect to $\theta_{w}$:

$\displaystyle\mathcal{L}_{wd}(\mathcal{X}_{s},\mathcal{X}_{t})=\frac{1}{\lvert\mathcal{X}_{s}\rvert}\sum_{\mathbf{x}_{s}\in\mathcal{X}_{s}}W(\mathbf{h}_{s})-\frac{1}{\lvert\mathcal{X}_{t}\rvert}\sum_{\mathbf{x}_{t}\in\mathcal{X}_{t}}W(\mathbf{h}_{t}).$
(8)

To make the training progress to be stable, we further enforce parameters
$\theta_{w}$ to minimize the Lipschitz constraint using gradient penalty
$\mathcal{L}_{grad}$ proposed by Gulrajani et al. (2017) of the domain critic
as:

$\displaystyle\mathcal{L}_{grad}(\hat{\mathbf{h}})=\left(\left\|\bigtriangledown_{\hat{\mathbf{h}}}W(\hat{\mathbf{h}})\right\|_{2}-1\right)^{2},$
(9)

where $\hat{\mathbf{h}}=\\{\mathbf{h}_{s},\mathbf{h}_{t},\mathbf{h}\\}$ is the
feature representations at which to penalize the gradients and it is defined
by the source and target representations $\mathbf{h}_{s},\mathbf{h}_{t}$ as
well as at the random points $\mathbf{h}$ along the straight line between
source and target representation pairs Shen et al. (2018); Gulrajani et al.
(2017).

Combining Eq. (8) and Eq. (9), the optimization problem for the W-Net and the
lesion attribute segmenter $S_{l}\ (l\in L)$, which minimizes discrepancy
between two domains $\mathcal{X}_{s}$ and $\mathcal{X}_{t}$ in terms of
Wasserstein distance, is defined as:

$\displaystyle\mathcal{L}_{wass}=\underset{\theta_{S_{l}}}{\text{min}}\,\underset{\theta_{w}}{\text{max}}\\{\mathcal{L}_{wd}-\eta\mathcal{L}_{grad}\\},$
(10)

where $\eta$ is the regularization coefficient.

Figure 4: Domain Adaptation Components in our method with W-Net and AD-Net.
The W-Net is designed to minimize the Wasserstein Distance Minimization
between two feature embedding spaces in the source and target domain. The AD-
Net is employed as the discriminate network to distinguish entropy
distributions from two domains.

####  IV-B3 Adversarial Domain Discriminator AD-Net

In order to exploit the structural consistency between the source and target
domain, we employ an entropy loss $\mathcal{L}_{ent}$ to directly minimize an
uncertainty prediction Vu et al. (2019); Ma et al. (2021) in the target domain
of $S_{l}$. Given an input image in the target domain
$\mathbf{x}_{t}\in\mathcal{X}_{t}$, we compose the Shannon Entropy map Shannon
(2001) $\textbf{{E}}_{\mathbf{x}_{t}}\in[0,1]^{h\times w}$:

$\displaystyle\textbf{{E}}_{\mathbf{x}_{t}}^{(h,w)}=\frac{-1}{\log(C)}\sum_{i=1}^{C}P_{\mathbf{x}_{t}}^{(h,w,i)}\log
P_{\mathbf{x}_{t}}^{(h,w,i)},$ (11)

at each pixel $(h,w)$. Here, $C$ being the number of output class, i.e., $C=2$
in our setting with binary segmentation for each lesion attribute $l\in L$,
and $P_{\mathbf{x}_{t}}^{(h,w,i)}$ is the pixel-wise predicted class score
estimated from $S_{l}(\mathbf{x}_{t})$.

We now define an entropy loss $\mathcal{L}_{ent}$ which is the sum of all
pixel-wise normalized entropy:

$\displaystyle\mathcal{L}_{ent}(\mathbf{x}_{t})=\sum_{h,w}\textbf{{E}}_{\mathbf{x}_{t}}^{(h,w)}$
(12)

By combining Eq.(1) and Eq. (12), we can jointly optimize the supervised
segmentation loss with the samples in the source domain $\mathcal{X}_{s}$ and
an unsupervised entropy loss on the target domain $\mathcal{X}_{t}$ by:

$\displaystyle\underset{\theta_{S_{l}}}{\text{min}}\frac{1}{\lvert\mathcal{X}_{s}\rvert}\sum_{(\mathbf{x},\mathbf{z}_{l})\in\mathcal{X}_{s}}\mathcal{L}_{wbce}(S_{l}(\mathbf{x}),\mathbf{z}_{l})+\frac{\lambda_{ent}}{\lvert\mathcal{X}^{t}\rvert}\sum_{\mathbf{x}_{t}\in\mathcal{X}_{t}}\mathcal{L}_{ent}(\mathbf{x}_{t}),$
(13)

with $\lambda_{ent}$ is the weight factor for the entropy part
$\mathcal{L}_{ent}$.

Training the lesion segmentation model $S_{l}$ with the joint loss function in
Eq. (13) has shown improvement in experiments. However, merely minimization
this does not entirely capture the structural dependencies between the local
semantics. In other directions, authors in Tsai et al. (2018) have argued that
adaptation to the structural output space is favorable for unsupervised domain
adaptation in semantic segmentation tasks. Similarly, Shen et al. (2018) have
also shown that adaptation in the latent space can enhance the model
generalization ability. These premises are based on the fact that the source
and target domain usually share strong similarities in the semantic layout. To
exploit such observations, we incorporate an adversarial training framework to
implicitly guide the target domain’s entropy distributions to be similar to
the source ones Vu et al. (2019); Ma et al. (2021). Our motivation is based on
the fact that a trained model naturally produces a high confidence score for
one target class and low for the rest on source-like images. Therefore,
entropy for the source-like images will be low, and that of target images will
be higher.

Using the equation (11), we define a weighted self-information map $I_{x}$ for
any input image $\mathbf{x}$ composed of pixel-level vectors at each pixel
$(h,w)$ as:

$\displaystyle I_{x}^{(h,w)}=-P_{x}^{(h,w)}\circ\log P_{x}{(h,w)}$ (14)

where $\circ$ stands for hadamard product,
$P_{x}^{(h,w)}=\left[P_{x}^{(h,w,i)}\right]_{i=1}^{C}$ is the probability
score for $C$ classes at $(h,w)$ estimated by $S_{l}(x)$.

Given this, we define $I_{\mathbf{x}_{s}},\,I_{\mathbf{x}_{t}}$ as weighted
self-information maps for the source and target domain
$\mathcal{X}_{s},\mathcal{X}_{t}$ respectively computed by:

$\displaystyle I_{\mathbf{x}_{s}}^{(h,w)}$
$\displaystyle=-P_{\mathbf{x}_{s}}^{(h,w)}\circ\log P_{\mathbf{x}_{s}}{(h,w)}$
(15) $\displaystyle I_{\mathbf{x}_{t}}^{(h,w)}$
$\displaystyle=-P_{\mathbf{x}_{t}}^{(h,w)}\circ\log P_{\mathbf{x}_{t}}{(h,w)}$
(16)

We now formulate the adversarial network AD-Net with parameters $\theta_{ad}$
as convolutational networks. This network takes $I_{x}$ as an input and
produces domain classification as an output with class label 1/0 for the
source/target domain respectively (right side in Fig. 4). Note that, AD-Net
architecture and its parameters are different from the discriminator network
PD-Net that we have used to optimised $S_{l}$ in Eq. (5). Similar to the
learning procedure of the original GAN method Goodfellow et al. (2014), the
discriminator AD-Net is trained to discriminate outputs coming from source and
target images and simultaneously, the lesion network $S_{l}$ is trained to
fool the discriminator AD-Net. The optimization objective of the discriminator
is:

$\displaystyle\mathcal{L}_{adv}=\underset{\theta_{S_{l}}}{\text{min}}\
\underset{\theta_{ad}}{\text{max}}\
\frac{1}{\lvert\mathcal{X}_{s}\rvert}\sum_{\mathbf{x}_{s}\in\mathcal{X}_{s}}\log(AD(I_{\mathbf{x}_{s}}))$
(17)
$\displaystyle+\frac{1}{\lvert\mathcal{X}_{t}\rvert}\sum_{\mathbf{x}_{t}\in\mathcal{X}_{t}}\log(1-AD(I_{\mathbf{x}_{t}})))$
(18)

In summary, by jointly optimizing equations (5), (10), and (18), we end up
with a total loss function:

$\displaystyle\mathcal{L}_{total}=\mathcal{L}_{seg}+\lambda_{p}\,\mathcal{L}_{\mathrm{patch}}+\lambda_{w}\mathcal{L}_{wass}+\lambda_{adv}\mathcal{L}_{adv}.$
(19)

where, $\lambda_{p},\,\lambda_{w}$ and $\lambda_{adv}$ are the weighting
factors for the patch-based adversarial learning, Wassertein term, and
entropy-based adversarial term respectively.

###  IV-C Attention-based Classifier

To this end, using lesion segmentation models $S_{l}$ from the previous step,
we generate $L$ distinctive lesion maps ${\mathbf{z}}_{{l=1}_{l}}$ for each
input image $\mathcal{x}$. The attention module Attn-Net then uses this
preliminary information to select the most relevant parts contributing toward
the better performance of the disease classification network G-Net.

This work investigates two different jointly learning frameworks between Att-
Net and G-Net for network architectures based on CNN (section IV-C1) and
Transformer (section IV-C3). Furthermore, besides integrating lesion
information at a feature level through attention gates, denoted as low-level
concepts (section IV-C1), we also formulate a novel overlapping constraint
among heatmap regions of G-Net and segmented multi-lesion regions obtained
from S-Net, denoted as high-level concepts (section IV-C2). Fig. 6 gives an
illustration for these attention concepts. In our setting, we develop these
constraints for both CNN-based and Transformer-based architectures and
discover that they contribute in improving accuracy for several baselines by a
large margin (tables VII, VI). Last but not least, these strategies enable an
explainable visualization of DR grading predictions when ophthalmologists can
observe the correlation of high-responding areas in trained networks with
medical priors represented as lesion regions.

Figure 5: Detailed architectures and pipeline of the proposed _DRG-AI-System_.
It consists of five sub-networks: S-Net, PD-Net, AD-Net, W-Net, Att-Net, and
G-Net. These networks are learned in two phases. The first stage aims to train
the multi-lesion segmentation networks S-Net. To overcome the lack of training
data in a source domain and domain adaptation issues in a target domain, PD-
Net, W-Net, and AD-Net are jointly integrated. The second stage optimizes the
grading network G-Net using lesion information obtained from S-Net, where the
attention network Att-Net automatically decides the most influential parts of
a given image.

####  IV-C1 Attention Lesion Regions at Low-level Concepts: Attention Network
Att-Net and Grading Network G-Net

Inspired by Zhou et al. (2019), we consider attention mechanisms at feature
maps of the classification network G-Net given multi-lesion maps predicted by
$S_{l}$. For this, feature maps at the first and last layers of G-Net are
jointly combined with lesion maps $\mathbf{z}_{l=1}^{L}$ to define attention
maps that return high responses to different lesion regions characterizing the
disease. It is worthy to note that we do not integrate the lesion masks
initially predicted by $S_{l}$ as a direct input for the classification model
G-Net because the initially predicted masks are usually very tiny and sparse.
Their contributions rather than being controlled by the attention network Att-
Net. In our setting, we choose the ResNet-50 He et al. (2016), including five
CNN blocks for the architecture of G-Net, and therefore have to modify
formulations in Zhou et al. (2019) as follows.

We indicate five CNN blocks of G-Net as
$G=\\{\mathbf{g}^{1},\mathbf{g}^{2},\mathbf{g}^{3},\mathbf{g}^{4},\mathbf{g}^{5}\\}$.
The feature maps of an image $\mathbf{x}$ at the first block are computed by:

$\displaystyle\mathbf{f^{\mathrm{first}}}=\mathbf{g}^{1}(\mathbf{x})$ (20)

Similarly, the feature maps at the last layer estimated by:

$\displaystyle\mathbf{f^{\mathrm{last}}}=\mathbf{g}^{5}(\mathbf{o}^{5})$ (21)

where

$\displaystyle\mathbf{o}^{i}=\mathbf{g}^{i-1}(\mathbf{o}^{i-1}),\ i\in[2,5]$
(22) $\displaystyle\mathbf{o}^{1}=\mathbf{x}$ (23)

where $\mathbf{o}^{i}$ indicates the output of each convolutions block.

In the first step, we train G-Net with all blocks in a fully supervised manner
using the image-level annotations in the target domain
$(\mathbf{x}_{t},\hat{\mathbf{t}}_{t})\in\mathcal{X}_{t}$. The optimization
problem is:

$\displaystyle\mathcal{L}_{cls}=\underset{\theta_{g}}{\text{min}}\frac{1}{\lvert\mathcal{X}_{t}\rvert}\sum_{(\mathbf{x}_{t},\hat{\mathbf{t}}_{t})\in\mathcal{X}_{t}}\mathcal{L}_{mce}(G(\mathbf{x}_{t}),\hat{\mathbf{t}}_{t}),$
(24)

where $\mathcal{L}_{mce}$ is the multi-class cross entropy classification loss
defined as:

$\displaystyle\mathcal{L}_{mce}(\mathbf{t}_{t},\hat{\mathbf{t}}_{t})=-\sum_{k=1}^{C}\hat{\mathbf{t}}_{t}^{(k)}\log\mathbf{t}_{t}^{(k)}$
(25)

with $\hat{\mathbf{t}}^{(k)}$ is $0$ or $1$, indicating whether class label
$k$ is the correct classification.

Once the model is pre-trained, the feature maps $\mathbf{f}^{\mathrm{first}}$
and $\mathbf{f}^{\mathrm{last}}$ are computed using equations (20), (21). Then
we define an attentive feature for the $l$-th lesion $\mathbf{y}_{l}$ obtained
from $S_{l}$ by:

$\displaystyle\mathbf{f}^{\mathrm{first-
att}}_{l}=\mathrm{ReLU}(\mathbf{W}_{l}^{\mathrm{first}}\text{concat}(\mathbf{y}_{l},\mathbf{f}^{\mathrm{first}})+b_{l}^{\mathrm{first}}),$
(26)

where concat(.) is the channel-wise concatenation;
$\mathbf{W}_{l}^{\mathrm{first}}$ and $b_{l}^{\mathrm{first}}$ are additional
learnable parameters and bias terms for the $l$-th lesion.

In a next step, the last feature maps $\mathbf{f}^{\mathrm{last}}$ acting as a
global feature embedding is correlated with the first-level attentive features
to generate attention weights for the $l$-th lesion:

$\displaystyle\alpha_{l}=\text{Sigmoid}(\mathbf{W}_{l}^{\mathrm{last}}[\mathbf{f}_{l}^{\mathrm{first-
att}}\odot\mathbf{f}^{\mathrm{last}}]+b_{l}^{\mathrm{last}}),$ (27)

where $\odot$ is the element-wise multiplication;
$\mathbf{W}_{l}^{\mathrm{last}}$ and $b_{l}^{\mathrm{last}}$ are other
parameters and bias terms to learn attention features at the global level. To
make $\mathbf{f}^{\mathrm{last}}$ and $\mathbf{f}_{l}^{\mathrm{first-att}}$ be
compatible in channel dimensions, we also use a $1\times 1$ convolution over
the $\mathbf{f}^{\mathrm{last}}$.

By applying Eq. (27) for each lesion $l\in L$, we aggregate all attention
lesion maps and use them to separately conduct element-wise multiplication
with the first-level features $\mathbf{f}^{\mathrm{first}}$ of G-Net. The
output feature vectors then are concatenated and utilized as final attention
features to fine-tune the DR Grading network G-Net. In terms of optimization,
parameters of Att-Net include
$\theta_{Att}=\\{\left(\mathbf{W}_{l}^{\mathrm{first}},\,\mathbf{W}_{l}^{\mathrm{last}},\,b_{l}^{\mathrm{first}},\,b_{l}^{\mathrm{last}}\right)_{l=1}^{L}\\}$.

Finally we compose an optimization, that jointly learns the grading network
G-Net and the attention-based lesion Att-Net at the low-level concept as:

$\displaystyle\mathcal{L}_{cls}^{att}=\underset{\theta_{g},\theta_{att}}{\text{min}}\frac{1}{\lvert\mathcal{X}_{t}\rvert}\sum_{(\mathbf{x}_{t},\hat{\mathbf{t}}_{t})\in\mathcal{X}_{t}}\mathcal{L}_{mec}\left(G(\mathbf{x}_{t})\cdot
Att\left(\,S_{l}(\mathbf{x}_{t})_{l=1}^{L}\right),\,\hat{\mathbf{t}}_{t}\right).$
(28)

where $\mathcal{L}_{mec}$ is the multi-class cross entropy defined in Eq.
(25). We characterize the loss $\mathcal{L}_{cls}^{att}$ as the low-level
concept because it purely combines G-Net and Att-Net at the feature level
through attention weights. While this approach boosts performance, it is still
a black box to end-users. In the next section, we introduce a new constraint
between two types of networks which are able to enhance accuracy and provide
explainable properties for the learned system.

Figure 6: An overview of our attention mechanisms jointly learning with the DR
grading network at both low-level and high-level concepts. In the blue box,
internal architecture of low-level attention mechanism is illustrated in
detail.

####  IV-C2 Attention Lesion Regions at High-level Concepts

The attention procedure that we have used in the previous section is on the
structural feature level of the grading network. In this section, we further
impose a new constraint that directly compares the class activation map of the
grading network with the lesion maps. By considering this property, we
explicitly guide the network on which parts of the image they should focus on
and make the predictions more transparent to end-users by observing
correlations between trained networks’ activation map and medical priors (Fig.
15). This sets us apart from prior studies in DR grading problems Zhou et al.
(2019); Sun et al. (2021). Below we present the formulation to integrate this
constraint seamlessly in our learning framework for the case of G-Net based on
CNN architecture.

Based on Grad-CAM, we can get the class activation map of the last layer in
the grading model G-Net. For a input image $\mathbf{x}$, let $f_{l,k}$ be the
activation of unit $k$ in $l$-th layer. For each of the ground-truth class
$c$, we compute the corresponding gradient score $s^{c}$, with respect to
activation maps of $f_{l,k}$. These gradient scores then pass through $1\times
1$ convolution of global average pooling layer to obtain the neuron importance
weights $w_{l,k}^{c}$:

$\displaystyle w_{l,k}^{c}=\text{GAP}\left(\frac{\partial s^{c}}{\partial
f_{l,k}}\right),$ (29)

where GAP(.) is the global average pooling. Because $w_{l,k}^{c}$ indicates
the important of activation map $f_{l,k}$ contributing the prediction of class
$c$, we thus apply the weight matrix $w_{c}$ as a kernel and apply 2D
convolution over the feature map $f_{l}$ to aggregate all activation maps,
followed by a ReLU function to get the activation map $AM^{c}$ for the $c$-th
class:

$\displaystyle AM^{c}=\text{ReLU}(\mathrm{conv}(f_{l},w^{c}))$ (30)

where $l$ represents the last convolution layer.

In the next step, we normalize the $AM^{c}$ so that its class channel values
are normalized to $[0,1]$, denoted as $\widetilde{AM}^{c}$, using a
thresholding operation $T(.)$ Li et al. (2018a) as follows:

$\displaystyle\widetilde{AM}^{c}=T(AM^{c})$ (31) $\displaystyle
T(AM^{c})=\frac{1}{1+\exp\left(-\omega(AM^{c}-\boldsymbol{\sigma})\right)}$
(32)

where $\boldsymbol{\sigma}$ is the threshold matrix whose elements are equal
to $\sigma$. $\omega$ is the scale parameter forcing $T(AM^{c})_{i,j}$
approximately equals to $1$ if $AM^{c}_{ij}$ is greater than $\sigma$, or to
$0$ otherwise.

Finally, we propose an overlapping loss function $\mathcal{L}_{overlap}$ for
images whose lesion areas are not empty as:

$\displaystyle\mathcal{L}_{overlap}=\underset{\theta_{g},\theta_{att}}{\text{min}}\frac{1}{hw}\
\|\widetilde{AM}^{c}-L_{U}\|_{2}$ (33)

where $h,\,w$ are the height and width of image $x$, $L_{U}$ is the union
region of all of lesion types computed by:

$\displaystyle L_{U}=\bigcup_{l\in L}S_{l}(x)$ (34)

Note that we only compute the $\mathcal{L}_{overlap}$ for an image $x$ if its
grading label is different $0$, i.e., input image has some stages of DR
disease. By optimizing $\mathcal{L}_{overlap}$, we jointly learn parameters
for both grading network G-Net and the attentive network Att-Net.

Combining equations (33), (28), we end up with a new total object loss
function that incorporates attention mechanisms for DR grading task at both
low-level and high-level constraints:

$\displaystyle\mathcal{L}_{grading}=\mathcal{L}_{cls}^{att}+\mathcal{L}_{overlap}$
(35)

At a glance, our proposed loss $\mathcal{L}_{overlap}$ is comparable to
existing semi-supervised learning Li et al. (2018a) or Covid-19 detection Wu
et al. (2021); however, we extend it for the multi-lesion scenario in the
context of the DR grading task. Furthermore, our system is superior to these
works and current approaches to the DR problem Zhou et al. (2019); Sun et al.
(2021) in that we develop lesion information-based attention mechanisms for
classification tasks at both feature-level and high-level concepts, thereby
improving performance and providing explainable properties to the entire
system (table VI).

####  IV-C3 Integrating Lesion Features for Transformer-based Methods

Figure 7: Our proposed DR-grading transformer architecture. We integrate
predicted lesion feature maps using lesion generator $S_{l}$ for an image with
the multi-head lesion generator G-Net based on MIT-VT Yu et al. (2021). We
compute $\mathcal{L}_{overlap}$ and the grading model is trained using both
$\mathcal{L}_{cls}$ and $\mathcal{L}_{overlap}$.

In this section, we investigate the performance of Transformer architecture
for the DR grading task. For this, we choose the MIL-VT method proposed by Yu
et al. (2021) (Fig. 7). Compared with ViT, MIL-VT further adds multiple-
instance learning heads to leverage the features extracted from individual
patches. However, MIL-VT solely employs image-level data in the training stage
and ignores the responsibilities of lesion areas. This motivates us to
incorporate the MIL-VT attention’s mechanism with our overlapping heatmap
concepts presented in section IV-C2. To establish such a constraint, we apply
the Attention Rollout technique Abnar and Zuidema (2020); Dosovitskiy et al.
(2021) to compute the heatmap regions for MIL-VT.

In particular, Attention rollout is a concept used to track the information
propagated from the input layer to the embeddings in the higher layers. Given
a Transformer with $N$ layers, in each layer $n\in N$, this technique takes
the average of all attention weights across all heads to form an attention
matrix $A_{n}$ where $(A_{ij})_{l}$ defines how much attention is going to
flow from token $j$ in the layer $n-1$ to token $i$ in the layer $n$. Then the
attention rollout matrix at layer $n$, denoted as $\tilde{A}_{n}$, is computed
in a recursive way:

$\displaystyle\tilde{A}_{n}=(A_{n}+I)\,\tilde{A}_{n-1}\ $ (36)

where $I$ is the identity matrix.

By computing equation 36 at the last layer $N$, we can account for the
combination of attention across tokens through all layers. In our method, we
choose the attention map $AM^{c}$ as:

$\displaystyle AM^{c}=\tilde{A}_{N}.$ (37)

Then the loss $\mathcal{L}_{overlap}$ in equation 33 is defined in analogous
way (Fig. 7). In experiment, we discover that this extended version
significantly enhances MIL-VT’s performance in a variety of situations (table
VII).

##  V DRG-Net: ARG-Expert-Interaction

As a joint disease diagnosis system, given a retina fundus image, the system
can automatically detect the associated lesion attributes and predict its
disease grade. In addition, the detected lesion masks from the segmentation
module can support the disease grading prediction by the grading module.
Unlike most of the medical decision support systems (Sonntag et al., 2020;
Mehta et al., 2018; Dai et al., 2021) where the lesion maps and disease
grading predictions modules are independent of each other, our architecture is
trained to learn these tasks collaboratively. In the learning phase of our
grading model, we exploit predicted lesion information. Besides, the system
also facilitates the utilization of expert feedback throughout the training
process. Generally, our diagnosis system provides an interactive way with the
expert user, as shown in Fig. 8. For this, the method can deliver a visual
explanation to the user about its predictions and incorporate user feedback to
fine-tune the model. More details are as follows,

Figure 8: Illustration for the interaction between experts and the trained
model. The model can infer explanatory predictions via presenting disease
grade, related lesions, and class activation mapping. Upon inspection, the
expert validates the results and, if necessary, provides input (e.g., drawing
bounding box on missing lesion regions). This input is used to improve model
performance in subsequent model retraining.

###  V-A DR Grading Predictions with Explainable Properties

Our method focuses on the local interpretability Saini and Prasad (2021);
Selvaraju et al. (2017) of our prediction model, which is computed based on
the class activation map $AM^{c}$. This gives us the discriminative regions
used by the model to predict the disease grade for a certain class. By drawing
the $AM^{c}$ over input image $x$ (Fig. 8 \- Visual Explanation), we can
observe the highlighted region on the input $\mathbf{x}$, which the grading
network G-Net thought to be the most essential region for its decision (Fig. 8
\- Predicted DR Grade). Moreover, detected lesion maps can be used as visual
explanations. In particular, considering the overlapping region between lesion
maps $\hat{z}_{{l=1}_{l}}$ (Fig. 8 \- Predicted Lesion Regions) with
activation maps $AM^{c}$ of $x$ gives a visual interpretation of how much
these lesions influenced the grading model prediction (Fig. 8).

###  V-B Improving System’s Performance through User Feedback

Getting user feedback can costly and time-consuming in the medical domain. For
instance, lesion regions in diabetic disease can be so tiny that each region
covers only a small group of pixels in the image. Our framework alleviates
this issue by not requiring pixel-level training data in the target domain to
train segmentation models. It instead is handled by the domain adaptive
networks as the Wasserstein network W-Net (Section IV-B2) and the Adversarial
Discriminator-based on Entropy AD-Net (Section IV-B3). Given this, we can
generate different lesion regions while only using pixel-level labels from
another source domain.

When the system is deployed in practice, the expert can provide two feedback
forms. Firstly, given new annotations on lesion masks, we can fine-tune our
lesion generator models S-Net. Specifically, for each lesion generator model
$S_{l}$, we update the segmentation model using new labeled data in the target
domain by training:

$\displaystyle\mathcal{L}_{seg}=\underset{\theta_{S_{l}}}{\text{min}}\frac{1}{\lvert\mathcal{X}_{t}\rvert}\sum_{(\mathbf{x},z_{l})\in\mathcal{X}_{t}}\mathcal{L}_{wbce}(S_{l}(\mathbf{x}),\hat{\mathbf{y}}_{l})$
(38)

where $\mathcal{L}_{wbce}$ is the weighted binary cross-entropy loss defined
in equation (2). Note that in this case, we can ignore the domain adaptation
parts, which are already optimized during training with data in the source
domain $X_{s}$.

Secondly, when users provide both new image grading and lesion segmentation
labels, we can update either:

  * •

Segmentation models S-Net using $\mathcal{L}_{seg}$ in equation (38).

  * •

Attention-based disease grading model Att-Net and G-Net by learning the
objective function $\mathcal{L}_{grading}$ defined in equation (35).

It is worth noting that our technique automatically learns to identify
critical lesion locations influencing the DR Grading task by utilizing
attention mechanisms; hence, it can be resilient to a certain degree of noise
in segmentation annotations provided by experts. In other words, rather than a
precise segmentation mask, the expert can mark the region of interest in the
form of bounding boxes or circles around the lesion locations using any pen-
based input device. As the attention model inherently learns to filter out the
noise and focus on the area of interest, our experiments confirm that only
these soft annotations are sufficient to boost the performance of the model
(table IX).

##  VI Experimental Results

###  VI-A Datasets

Retina images are generally captured in two forms, which are Optical Coherence
Tomography (OCT) capturing a cross-sectional images of retinasa and Color
Fundus retinal photography capturing 3D retina images using fundus cameras. We
employed datasets with the later one for all our experiments in this work. For
domain-invariant lesion attributes segmentation, we used two publicly
available datasets. To the best of our knowledge, these two are the only
datasets that provide pixel-level annotations for diabetic retinopathy. The
datasets are:

  * •

IDRID segmentation Porwal et al. (2018): This dataset contains 81 high
resolution ($4288\times 2800$ pixels) images with four different types of
lesion annotations and is split into 54 training images and 27 testing images.
The lesions are: microaneurysms (MA), haemorrhages (HE), hard exudates (EX)
and soft exudates (SE). Each of these lesions are annotated in the forms of
binary masks.

  * •

FGADR segmentation Zhou et al. (2020): Similarly as IDRID, this dataset
contains 1843 images with 4 kinds of lesion masks (MA, HE, EX, and SE). The
training set consists of 1500 images and the remaining 343 images are for test
set. This dataset has imbalance in the lesion classes. The class HE is
available in 1471 images which is 78% of the total images where the class MA
is available for only 610 images which consists of 33% of the total images.

For the classification task, we used three publicly available datasets.

  * •

IDRID classification  Porwal et al. (2018): This dataset consists of 413
training and 103 test images with 5 severity grading labels. The five labels
include: normal, mild, moderate, severe and proliferative, which are annotated
as 0, 1, 2, 3, 4, respectively. These images only have image-level labels and
do not have any pixel-level lesion masks annotations.

  * •

EyePACS  EyePACS (2015): This dataset consists of 35,126 training images and
53,576 testing images with similar grading protocol of classes with 5
categories. Images in this dataset is captured using different types of
cameras settings under various light conditions.

  * •

FGADR classification Zhou et al. (2020): this dataset consists of 1500
training images and 343 testing image for diabetic retinopathy grading. This
is the only one that contains both image-level pixel disease grading and
pixel-level lesion annotation maps.

Figure 9 shows the class distribution for the three classification datasets we
used in our work. We can observe that there are great numbers of class
imbalance in all three datasets.

Figure 9: Class distributions in the three classification datasets used in our
work.

###  VI-B Evaluation Metric

To evaluate the segmentation performance of multi-lesion segmentation, we used
two common metrics i.e., Area Under the Curve of Receiver Operating
Characteristics (AUC-ROC) and Area Under the Curve of Precision-Recall (AUC-
PR). The positive class means the existence of lesion and negative is the
otherwise. For evaluating multi-class disease grading classification task, we
used Accuracy and Quadratic Weighted Kappa (Kappa). Detailed definitions of
these measures are as follows:

  * •

AUC-ROC measures the class separability at various threshold settings. ROC is
the probability curve and AUC represents the degree of measures of
separability. It compares true positive rate (sensitivity/recall) versus the
false positive rate (1 - specificity). The higher the AUC-ROC, the bigger the
distinction between the true positive and false negative.

  * •

AUC-PR: It combines the precision and recall, for various threshold values, it
compares the positively predicted value (precision) vs the true positive rate
(recall). Both precision and recall focus on the positive class (the lesion)
and unconcerned about the true negative (not a lesion, which is the majority
class). Thus, for class imbalance, PR is more suitable than ROC. The higher
the AUC-PR, the better the model performance.

  * •

Accuracy: The normal classification accuracy which simply measures how many
observations are correctly classified.

  * •

Quadratic Weighted Kappa (Kappa): It is similar to the Cohen’s kappa metric
McHugh (2012) by the weights are set to ’Quadratic’. Cohen’s kappa measures
the agreement between two raters who each classify $N$ number of items into
$C$ mutually exclusive categories. The formulation is:

$\displaystyle\mathrm{kappa}=\frac{p_{o}-p_{e}}{1-p_{e}},$ (39)

where $p_{o}$ denotes the relative observed agreement between the raters and
$p_{e}$ is the hypothetical probability of chance agreement.

###  VI-C Implementation Details

We implemented all experiments using PyTorch framework Paszke et al. (2019)
and executed on a computer 2 NVIDIA TITAN RTX 24GB GPUs. For all experiments,
the sizes of the input images are $512\times 512$.

####  VI-C1 Domain Invariant Lesion Attribute Learning

For the experiments in lesion attribute segmentation discussed in section
IV-A, we used pixel values in the range [0, 1] for retina images and ground-
truth binary lesion segmentation masks. In this step, the input image are
labeled images from source domain and unlabeled images for the target domain.
We applied prepossessing steps as proposed in Xiao et al. (2019) and data
augmentation for input images.

Our S-Net model described in section IV-A is a U-shaped Ronneberger et al.
(2015) encoder-decoder segmentation network. The encoder architecture is
identical to the convolutional block of the ResNet50 He et al. (2016) model
and the decoder architecture is up-scaled accordingly. The architecture of the
PD-Net described in section IV-B1 is similar to the InfoGAN Chen et al.
(2016). This discriminator model provides stable performance for our case. It
consists of two convolution layers with 64 and 128 kernels, respectively,
followed by two fully-connected layer of 1024 dimensions and then the sigmoid
layer.

For training the PD-net IV-B1, we used SGD with momentum Liu and Belkin (2018)
as optimizer with initial learning rate of $10^{-4}$ and momentum value of
0.9. We adopted the cyclic learning scheduler Smith (2017) with max value of
$10^{-2}$. The class balancing hyper-parameter weight $\beta$ for the
segmentation loss $\mathcal{L}_{wbce}$ described in equation 2 is set to 10
following Xiao et al. (2019) to address class imbalance for lesion and non-
lesion class. The value of $\lambda_{p}$ for the source domain adversarial
joint optimization in equation (5) is set to $10^{-2}$. For domain adaptation
fine-tuning training, pretrained S-Net from IV-B1 is trained with Wasserstein
critic model W-Net (section IV-B2) and entropy-based discriminatory network
AD-Net. W-Net is a fully connected neural network which takes 1D encoder block
outputs from S-Net as inputs. W-Net has two fully connected layers with number
of neurons of 128 and 64, respectively, and final output is linear layer which
gives a single scalar value, the critic score. AD-Net architecture is adopted
from Vu et al. (2019) which have five convolutional layers. The input for AD-
Net is the weighted self-information maps computed on the output of S-Net
using equation (16) and output is the binary class probability score on input
being source or target domain.

For training both W-Net and AD-Net, we used Adam optimizer with a learning
rate $10^{-4}$. The optimizer hyper-parameters for S-Net in this step are kept
unchanged and they are similar to the one discuss in the previous step.
Following Vu et al. (2019), the weight factor $\lambda_{ent}$ in Eq. (13) and
$\lambda_{adv}$ in Eq. (19) are $10^{-3}$. We used batch size of 8 for our
experiments.

####  VI-C2 Lesion Attentive Grading Model

To train the lesion attentive grading models G-Net and Att-Net, we followed
the similar pre-possessing and augmentation steps for the input images as with
the grading model.

The backbone of our lesion attentive grading model is a ResNet50 He et al.
(2016) architecture with the output fully connected layer modified to predict
five classes. The attention module Att-Net consists of multiple convolutional
layers. This network is optimized using Eq. (35) using the cross-entropy loss.
As we can see from figure 9, there is a significant class imbalance among
different types of lesions across all datasets, we thus compute class-weight
for each lesion attribute during the optimization step. We used the SGD
optimizer Ruder (2016) with initial learning rate $10^{-3}$ and applied the
cyclic learning strategy as in lesion segmentation steps. For all the
experiments, we varied the batch size between 16 and 32. For all of our
experiments we have used 10% of training data as validation set and final
results are computed on test set.

###  VI-D Performance of Multi-Lesion Segmentation Task

####  VI-D1 Influence of Pre-training Steps on Lesion Generator Models

As discussed in section IV-A, our lesion segmentation model S-Net is firstly
pre-trained using the lesion segmentation labels from the source domain data
in a fully supervised manner. To evaluate the effectiveness of task agnostic
transfer learning (TATL) (Nguyen et al. (2022b)) and the adversarial learning
(PD-Net) during the pre-training step, we test with three settings:

  * •

S-Net: Lesion feature generator segmentation model (S-Net) is trained only
using the segmentation loss $\mathcal{L}_{seg}$ in equation 1. We do not adapt
either TATL strategy or adversarial trainings.

  * •

S-Net + PD-Net: We incorporate adversarial learning strategy on source domain
as discussed in section IV-B1. We jointly the PD-Net with S-Net segmentation
model by optimization the formula in equation 5.

  * •

S-Net + PD-Net + TATL: We apply both task agnostic transfer learning (TATL)
and adversarial learning with PD-Net to learn S-Net using data from the source
domain.

Table II and III compare the results of different settings for IDRID and FGADR
segmentation datasets respectively. The segmentation performance is evaluated
by AUC-ROC and AUC-PR values on four different lesions, including
microaneurysms (MA), haemorrhages (HE), hard exudates (EX) and soft exudates
(SE). For IDRID-segmentation dataset in table II, we can observe slight
improvement in performance for settings S-Net + PD-Net compared with original
S-Net settings. With the combinations of TATL in settings S-Net + PD-Net +
TATL, we observe significant improvements in results for all of the lesion
classes. For instance, comparing with S-Net settings, we gained 2.5% and 5.5%
on AUC-ROC and AUC-PR scores respectively, averaged over all four lesions. For
the lesion class HE, our method achieved highest improvement in AUC-PR scores
of 9.4%.

We can observe the similar trend in results on the test set of FGADR-
segmentation dataset in table III. There is an improvement of 8.5% in the AUC-
PR scores averaged over all four lesion classes. To summarize from the
results, we conclude that combining adversarial training using the PD-Net and
the TATL strategy provided a better pre-training performance for S-Net in the
source domain supervised learning.

TABLE II: Contributions of task agnostic transfer learning TATL and
Adversarial training PD-Net (section IV-B1) in learning lesion segmentation
model S-Net in the source domain. Results are evaluated using the training and
testing set of IDRID segmentation.

Methods | MA | HE | EX | SE
---|---|---|---|---
ROC | PR | ROC | PR | ROC | PR | ROC | PR
S-Net | 0.937 | 0.440 | 0.896 | 0.459 | 0.931 | 0.722 | 0.953 | 0.583
S-Net + PD-Net | 0.932 | 0.439 | 0.917 | 0.481 | 0.946 | 0.735 | 0.965 | 0.611
S-Net+PD-Net+TATL | 0.953 | 0.456 | 0.931 | 0.553 | 0.961 | 0.772 | 0.970 | 0.643

TABLE III: Contributions of task agnostic transfer learning TATL (Nguyen et
al. (2022b)) and Adversarial training PD-Net (section IV-B1) in learning
lesion segmentation model S-Net in the source domain. Results are evaluated
using the training and testing set of FGADR segmentation.

Methods | MA | HE | EX | SE
---|---|---|---|---
ROC | PR | ROC | PR | ROC | PR | ROC | PR
S-Net | 0.901 | 0.373 | 0.941 | 0.611 | 0.947 | 0.602 | 0.927 | 0.410
S-Net + PD-Net | 0.926 | 0.394 | 0.955 | 0.638 | 0.959 | 0.667 | 0.941 | 0.492
S-Net+PD-Net+TATL | 0.937 | 0.417 | 0.963 | 0.652 | 0.970 | 0.714 | 0.954 | 0.553

####  VI-D2 Influence of Domain Adaption on Lesion Generator Models

In this section, we present the results of our domain adaption approaches in
different settings and compare with different methods. As discussed in section
IV-B, the aim of domain adaptation is to train a neural network using
available labeled data from source domain and secure a good accuracy on target
domain. We evaluated our approach on IDIRD and FGADR datasets in the forms of
source $\rightarrow$ target:

  * •

IDRID $\rightarrow$ FGADR: evaluation on FGADR-segmentation test set when
domain adaptive segmentation model is trained using the labeled data from
IDRID-segmentation dataset and unlabeled images from FGADR-segmentation
dataset.

  * •

FGADR $\rightarrow$ IDRID: evaluation on IDRID-segmentation test set when
domain adative segmentation model is trained using the labeled data from
FGADR-segmentation dataset and unlabled images from IDRID-segmentation
dataset.

TABLE IV: IDRID $\rightarrow$ FGADR: Semantic segmentation performance on
FGADR (target domain). Models are trained with labeled data on IDRID and
increasingly labeled data in the target domain from $0\rightarrow 100\%$. Red
indicates the best results for settings using 0% labeled data from target
domain in the training step. Green stands for settings using $40\%-60\%$
labeled data in target domain but outperform at least one of the methods
trained with $100\%$ data. Bold are the best values in all methods.

Target Domain | Methods | MA | HE | EX | SE
---|---|---|---|---|---
ROC | PR | ROC | PR | ROC | PR | ROC | PR
| S-Net | 0.752 | 0.243 | 0.796 | 0.280 | 0.794 | 0.311 | 0.728 | 0.264
| S-Net + Entropy | 0.807 | 0.313 | 0.843 | 0.357 | 0.855 | 0.407 | 0.819 | 0.341
| S-Net + AD-Net | 0.841 | 0.348 | 0.903 | 0.448 | 0.917 | 0.473 | 0.902 | 0.443
0% | S-Net+AD-Net+W-Net | 0.894 | 0.357 | 0.911 | 0.502 | 0.939 | 0.538 | 0.915 | 0.522
40% | S-Net+AD-Net+W-Net | 0.938 | 0.411 | 0.953 | 0.613 | 0.966 | 0.682 | 0.965 | 0.634
60% | S-Net+AD-Net+W-Net | 0.946 | 0.438 | 0.969 | 0.648 | 0.979 | 0.728 | 0.971 | 0.655
80% | S-Net+AD-Net+W-Net | 0.954 | 0.458 | 0.973 | 0.671 | 0.981 | 0.732 | 0.977 | 0.684
100% | S-Net+AD-Net+W-Net | 0.958 | 0.462 | 0.979 | 0.676 | 0.990 | 0.739 | 0.984 | 0.693
| FCN-8s Zhou et al. (2020) | 0.925 | 0.363 | 0.962 | 0.606 | 0.981 | 0.686 | 0.963 | 0.642
| U-Net Zhou et al. (2020) | 0.927 | 0.382 | 0.967 | 0.643 | 0.982 | 0.726 | 0.977 | 0.683
| DL-V3+ Zhou et al. (2020) | 0.934 | 0.364 | 0.973 | 0.619 | 0.981 | 0.708 | 0.967 | 0.659
100% | Attention U-Net Zhou et al. (2020) | 0.942 | 0.435 | 0.974 | 0.678 | 0.984 | 0.731 | 0.980 | 0.685

We report the results for domain adaptation on tables IV and V. Our domain
adaption approach discussed in section IV-B consists of modules for
Adversarial Entropy Minimization and Wasserstein based distance minimization
between source and taget domain. To evaluate the effectiveness of different
constraints for our domain adaptive segmentation model, we considered four
experiments settings:

  1. 1.

S-Net: The baseline approach for our domain adaptation approaches. S-Net is
trained on source domain by adopting the pre-training method including PD-Net
and TATL. This model is trained using only the equation 5. Neither domain
adaptation constraints are considered.

  2. 2.

S-Net + Entropy: We apply direct entropy minimization constraint for target
domain in the learning process of S-Net as discussed in section IV-A. The
entropy loss $\mathcal{L}_{ent}$ in equation (12) is used for minimization on
the unlabeled data from target domain and model is optimized with the
segmentation loss $\mathcal{L}_{seg}$ using equation (13).

  3. 3.

S-Net + AD-Net: Instead of using direct entropy minimization
$\mathcal{L}_{ent}$, we use the proposed domain adaptation with minimizing
entropy-based adversarial learning AD-Net. S-Net and AD-Net are optimized
together by combining adversarial loss $\mathcal{L}_{adv}$ in equation (18)
with $\mathcal{L}_{seg}$ in equation (13).

  4. 4.

S-Net + AD-Net + W-Net:  Additional domain critic W-Net is added along with
S-Net and AD-Net to minimize the domain gap in feature representation using
Wasserstein loss $\mathcal{L}_{wass}$. This is our proposed setting where all
modules are jointly trained using our optimization problem in equation (19).

TABLE V: FGADR $\rightarrow$ IDRID: Semantic segmentation performance on IDRID
(target domain). Models are trained with labeled data on FGADR and
increasingly labeled data in the target domain from $0\rightarrow 100\%$. Red
indicates the best results for settings using 0% labeled data from target
domain in the training step. Green stands for settings using $40\%-60\%$
labeled data in target domain but outperform at least one of baselines trained
with $100\%$ data. Bold are the best values in all methods.

Target Domain | Methods | MA | HE | EX | SE
---|---|---|---|---|---
ROC | PR | ROC | PR | ROC | PR | ROC | PR
| S-Net | 0.813 | 0.232 | 0.803 | 0.251 | 0.837 | 0.363 | 0.783 | 0.241
| S-Net + Entropy | 0.884 | 0.331 | 0.874 | 0.375 | 0.902 | 0.531 | 0.865 | 0.404
| S-Net + AD-Net | 0.925 | 0.408 | 0.902 | 0.447 | 0.911 | 0.697 | 0.879 | 0.447
0% | S-Net+AD-Net+W-Net | 0.947 | 0.436 | 0.911 | 0.462 | 0.927 | 0.772 | 0.890 | 0.476
40% | S-Net+AD-Net+W-Net | 0.971 | 0.483 | 0.957 | 0.631 | 0.946 | 0.829 | 0.937 | 0.634
60% | S-Net+AD-Net+W-Net | 0.983 | 0.502 | 0.968 | 0.658 | 0.969 | 0.841 | 0.943 | 0.669
80% | S-Net+AD-Net+W-Net | 0.985 | 0.510 | 0.979 | 0.682 | 0.977 | 0.847 | 0.959 | 0.710
100% | S-Net+AD-Net+W-Net | 0.988 | 0.511 | 0.982 | 0.700 | 0.978 | 0.851 | 0.961 | 0.714
| Adv. HEDNet Xiao et al. (2019) | - | 0.439 | - | 0.483 | - | 0.840 | - | 0.481
| AdvSeg Hung et al. (2018) | 0.961 | 0.470 | 0.924 | 0.592 | 0.945 | 0.79 | 0.939 | 0.675
| ASDNet Zhou et al. (2019) | 0.969 | 0.478 | 0.932 | 0.628 | 0.950 | 0.809 | 0.948 | 0.692
100% | CoLL Zhou et al. (2019) | 0.965 | 0.473 | 0.954 | 0.657 | 0.967 | 0.845 | 0.953 | 0.716

##### Results on IDRID $\rightarrow$ FGADR:

In table IV, we report the results for different settings on FGADR
segmentation test set. These models are trained using the labeled data from
IDRID training dataset and unlabeled data from FGADR dataset. We can observe
that the settings without any domain adaptation S-Net performed poorly during
the inference on FGADR domain. The AUC-PR scores for all of the lesion classes
are below 0.3. Direct entropy minimization method (S-Net + Entropy) has
significant improvement in both the metrics with respect to S-Net. Introducing
Adversarial network for entropy minimization (S-Net + AD-Net) has improved the
AUC-ROC and AUC-PR scores by 12% and 15.3%, respectively, averaged over all
four lesion classes. The best results among our domain adaptive settings with
$0\%$ labeles data in target domain are highlighted in red. In short, we
achieved the best results for all of the lesion classes when both adversarial
entropy minimization and Wasserstein domain critic model are considered
together.

Plots in figure 10 compare the improvement in AUC-PR scores on the lesion
classes of FGADR segmentation test set for different domain adaptation
settings. In summary, we observe that all of our domain adaptive settings
contribute for significantly improving performance over non-adaptive settings
S-Net. Besides, it can be seen in figure 11 that the training and validation
loss curves of all lesion classes for the setting S-Net + AD-Net + W-Net using
0% labeled data from the FGADR monotonously decline, confirming the stability
in the learning process.

In table IV, we compare our method with several state-of-the-art techniques:
Fully convolutional segmentation (FCN-8s) Long et al. (2015), Deep Lap model
(DLV3+)Chen et al. (2017), U-shaped models (U-Net) Ronneberger et al. (2015),
and Attention U-Net Islam et al. (2019)) using the results reported in Zhou et
al. (2020). The training are conducted directly on the target domain FGADR
using 100% training labels. For a detailed comparison, we trained our domain
adaptive model along with different percentages of labeled data from target
domain. Results in green indicate the settings where we surpassed one or more
methods under comparison while using $40\%\rightarrow 60\%$ labeled data.
Results in bold indicate the best scores in all settings. In general, we
observe that, for all of the lesion classes, by using only $40\%-60\%$ of
labeled data in the target domain, we are able to outperform more than one
baselines and with $80\%-100\%$ labeled data, we derive better AUC-ROC and
AUC-PR scores compared with other methods in most cases. For instance, we
gained 1.6%, 0.5%, 0.6% and 0.4% AUC-ROC score improvement on MA, HE, EX, and
SE lesion classes respectively. The AUC-PR scores also improved for MA, EX and
SE by 2.7%, 0.8% and 1.2% respectively. Figure 12 illustrates qualitative
comparisons of lesion map predictions between S-Net settings and our proposed
multi-lesion segmentation model trained with domain adaption constraints S-Net
+ AD-Net + W-Net.

Figure 10: Comparison of our different domain adaption approaches for lesion
segmentation on four types of lesions. Results are evaluated on the setting
IDRID $\rightarrow$ FGADR with $0\%$ in target domain.

##### Results on FGADR $\rightarrow$ IDRID:

Table V shows the results on four lesion classes on IDRID-segmentation test
set where the segmentation model is trained using the labeled data from FGADR-
segmentation set and unlabeled images from IDRID-segmentation training
dataset. For different domain adaptation settings, we discover similar trends
as the IDRID $\rightarrow$ FGADR case. In particular, we compare our method
with the following state-of-the-art techniques: adversarial learning based
segmentation networks Adv. HEDNet Xiao et al. (2019), AdvSeg Hung et al.
(2018) and semi-supervised collaborative learning networks ASDNet and CoLL
Zhou et al. (2019). The training is conducted with $100\%$ labeled instances
in the target domain (IDRID). With $0\%$ of label data, the configuration
S-Net + AD-Net + W-Net already has comparable performance with the first
baseline Adv. HEDNet. When using $40\%$ labeled data, we are able to
outperform most of competitive baselines in both AUC-ROC and AUC-PR metrics.
This shows that, the proposed lesion generator model can be fine-tuned with
minimal annotation data while still attains good performance in the target
domain.

Figure 11: Training and validation loss curves for our domain adaptive lesion
segmentation models. Results are on the setting IDRID $\rightarrow$ FGADR with
$0\%$ labeled data in the target domain. It can be seen that all training and
validation curves tend to converge to stable points given more training steps.

(a) Microaneurysms (MA)

(b) Haemorrhages (HE)

(c) Hard Exudates (EX)

(d) Soft Exudates (SE)

Figure 12: Qualitative multi-lesion segmentation results for four different
lesion classes. Comparison of lesion maps predicted by the segmentation models
trained without (with) Domain Adaption (DA) S-Net (S-Net + AD-Net + W-Net).

###  VI-E Performance of Diabetic Grading Tasks with Attentive Lesion
Information

####  VI-E1 Influence of Attention Mechanism on Grading Networks

We evaluated the effectiveness of predicted lesion segmentation maps in
disease grading classification tasks. Our evaluation is on two types of vision
related foundation models: CNN-based architecture and Vision Transformer-based
architecture. We conducted experiments with the following settings.

  1. 1.

G-Net: In this baseline, we directly train the disease grading classification
network using retina fundus images and their associated disease grading labels
using the classification loss $\mathcal{L}_{cls}$ in equation (24). In this
step only G-Net is trained and no‘attention functionality is used, therefore,
we do not use any lesion information in the training process.

  2. 2.

G-Net + Attention (low-level): We integrate an attention model Att-Net to
highlight generated lesion features in the training process of the grading
network G-Net. The attention mechanism we integrated here is on structural
feature level of the grading network described in section IV-C1. Here, the
G-Net and Att-Net are jointly trained using equation (28).

  3. 3.

G-Net + Attention (high-level): In this setting, we integrate the attention
mechanism for lesion features following our high-level concept described in
section IV-C2. Unlike the previous low-level concept, this attention
constraint is more intuitive where the normalized class activation maps are
directly compared with the lesion feature maps to compute the attention
overlapping loss $\mathcal{L}_{overlap}$ in equation (33). The G-Net and Att-
Net then are jointly trained for classification.

  4. 4.

G-Net + Attention (two-level): We combine both of the proposed low-level and
high-level attention concept in the training process of the disease grading
network G-Net. This is our final proposed method for the lesion attentive DR
grading model. The G-Net and Att-Net are jointly optimized using the equation
(35).

TABLE VI: Performance of the proposed attention mechanisms on the CNN-based
architecture (ResNet-50). Results are evaluated on three datasets IDRID,
EyePACS, and FGADR.

Methods | IDRID | EyePACS | FGADR
---|---|---|---
Accuracy | Kappa | Accuracy | Kappa | Accuracy | Kappa
G-Net (ResNet-50) | 0.823 | 0.844 | 0.836 | 0.819 | 0.807 | 0.751
G-Net + Attention (low-level) | 0.897 | 0.904 | 0.881 | 0.867 | 0.860 | 0.852
| G-Net + Attention (high-level)
---
0.859 | 0.873 | 0.862 | 0.849 | 0.839 | 0.827
G-Net + Attention (two-level) | 0.904 | 0.911 | 0.895 | 0.883 | 0.872 | 0.861

TABLE VII: Performance of the proposed attention mechanisms on the Vision
Transformer architecture (MIL-VIT). Results are evaluated on three datasets
IDRID, EyePACS, and FGADR . Because MIL-VIT already integrated attentions at
the feature-level, we only extend it with the high-level case.

Methods | IDRID | EyePACS | FGADR
---|---|---|---
Accuracy | Kappa | Accuracy | Kappa | Accuracy | Kappa
G-Net (MIL-VIT) | 0.780 | 0.824 | 0.755 | 0.791 | 0.751 | 0.781
G-Net + Attention (high-level) | 0.822 | 0.841 | 0.819 | 0.849 | 0.797 | 0.826

TABLE VIII: Comparison of our lesion attentive disease grading models with
other state-of-the-art techniques. These methods are also used lesion
information in solving the DR grading task.

Methods | IDRID | EyePACS | FGADR
---|---|---|---
Accuracy | Kappa | Accuracy | Kappa | Accuracy | Kappa
JCS Wu et al. (2021) | - | - | 0.886 | 0.877 | 0.856 | 0.842
AFN-Net Zhou et al. (2020) | - | - | 0.861 | 0.856 | 0.836 | 0.784
CoLL Zhou et al. (2019) | 0.913 | 0.904 | 0.891 | 0.872 | 0.86 | 0.848
G-Net (MIL-ViT) + Attention | 0.822 | 0.841 | 0.819 | 0.849 | 0.797 | 0.826
G-Net (ResNet-50) + Attention | 0.904 | 0.911 | 0.895 | 0.883 | 0.872 | 0.861

(a) accuracy (FGADR)

(b) kappa (FGADR)

(c) accuracy (EyePACS)

(d) kappa (EyePACS)

Figure 13: Illustration our training and validation classification accuracy
and kappa measurement using G-Net + Attention (two levels). Plots in the top
row (a) and (b) are the accuracy and kappa score curve in FGADR dataset. Plots
in the bottom row (c) and (d) are results in EyePACS dataset.

For the CNN-based architecture, we evaluated all four of our proposed settings
using ResNet-50 for the G-Net model. For transformer-based architecture, we
considered multi-head vision transformer (MIL-Vit) Yu et al. (2021) as G-Net.
Transformer models are inherently built on attention modules at the structural
level (low-level); therefore, we only test G-Net and G-Net + Attention (high-
level) methods for the transformer architecture.

We tested our approach on the IDRID, EyePACS and FGADR DR grading datasets.
Table VI and VII present the evaluation results for different settings on
these datasets for CNN-based and Transformer-based models respectively. For
CNN-based models in table VI, we observe that both low-level and high-level
attention mechanisms improved the performance for the original classification
network G-Net. Integrating low-level attention with grading network in
settings G-Net + Attention (low-level), we could increase in Kappa scores by
6%, 5% and 10% for IDRID, EyePACS and FGADR datasets respectively. For the
G-Net + Attention (high-level), the kappa scores are improved by 3%, 3% and 7%
respectively. We also gained best performance (in bold) for each of the
datasets when combining both low-level and high-level attention mechanisms for
the setting G-Net + Attention (two-level). Figure 13 describes the comparison
between training and validation accuracy in terms of kappa scores for the
FGADR and EyePACS data. We can see that the networks tend to achieve higher
performance given more epochs.

Table VII presents the results for transformer-based model. We can see that,
integrating high level attention mechanism with G-Net has improved both
classification accuracy and kappa scores for all three grading datasets. In
table VIII, we compared our best lesion attentive disease grading models (one
each for CNN-based and Transformer-based methods) with recent competitive
methods: JCS (Wu et al. (2021)), Denoising Attention Fusion Network (AFN-Net)
Zhou et al. (2020)) and Collaborative DR grading (CoLL) Zhou et al. (2019).
Results of our CNN-based attention model G-Net (CNN) + Attention on IDRID is
comparable with the other methods. For EyePACS and FGADR dataset, we
outperform other methods by around 2% in accuracy and kappa scores.

####  VI-E2 DR Grading Prediction with Explainable Property

To construct an explainable intelligent decision support system for diabetic
retinopathy diagnosis, in the inference step, along with the predicted class
output, we compute the class activation map (CAM) for the input image. CAM
highlights the discriminative regions for the predicted class based on which
the deep neural model has taken its decision. For DR grading tasks, different
lesion regions observed in retinal fundus image are the key clinical factors
to diagnose the patient disease progression to a particular grading class.
Figure 15 illustrates an example of the explainable prediction in our system.
For a given fundus image, our domain invariant lesion generator model S-Net
predicts lesion maps for four different lesion classes (figure 15 (j) to 15
(l)). Then the attention-based grading model G-Net provides the disease grade
and its decision explanation in the form of class activation map (figure 15
(b)). By comparing the CAM region with the predicted lesion positions, we can
compare the quality of the model explanation. Intuitively, the overlapping
rate of CAM on lesion maps and input images can provide explanation properties
to the experts about the network’s decisions (figure 15 (c) and 15 (d)).

In figure 17, we qualitatively compare the explanation results between the
grading model that is trained only using classification loss in G-Net settings
and our proposed attention based grading model that incorporate both low-level
and high-level attention concepts in settings G-Net (CNN) + Attention (two-
level). For some typical images and their lesion maps, we discover that our
attention-based grading (figure 17 (d)) was able to capture almost all the
important lesion regions whereas G-Net (figure 17 (c)) failed to capture these
lesion regions and considered some irrelevant positions as the class
discriminating mapping. In opposite, even some tiny lesion regions are also
considered in the prediction decision of our grading model.

Figure 14: Illustration of user-feedback simulation on slices of data. In a
standard scheme, traditional data split strategy is followed. In our
interactive simulation case, we initially split the training data into two
equal parts, and used one part of the split (50% of training data) to train
our proposed attention-based grading model. For the remaining part of $50\%$
training data, it is further split into five equal slices, serving for
increasingly data feedback. After each time getting data feedback, the model
is fine-tuned using both initial data and new ones. The performance of a new
model then is evaluated on the hold-out testing set.

(a) original image

(b) CAM - grade-3

(c) CAM + lesion (union)

(d) CAM + lesion + image

(e) CAM + MA

(f) CAM + HE

(g) CAM + EX

(h) CAM + SE

(i) Microaneurysms

(j) Haemorrhages

(k) Hard Exudates

(l) Soft Exudates

Figure 15: For an input retina fundus image (a), we provide the lesion maps
for four lesion classes (the bottom row). In the top row, the network provides
a (b) class activation map (CAM) for the predicted class (grade-3) (c) the
overlap of CAM with the union of the predicted lesion maps, and (d)
overlapping of CAM and union of predicted lesion maps on the original input
image. In the second row, the CAM overlap with each of the lesion maps.

###  VI-F Performance of Trained System using User Feedback

Method | % direct user feedback | Accuracy | Kappa | Explanation
---|---|---|---|---
G-Net + Attention | 0% | 0.773 | 0.781 | 0.272
10% | 0.786 | 0.801 | 0.295
20% | 0.822 | 0.838 | 0.331
30% | 0.841 | 0.853 | 0.363
40% | 0.850 | 0.863 | 0.366
50% | 0.855 | 0.866 | 0.387

TABLE IX: Performance of our framework utilizing increasingly user-feedback.
We iteratively fine-tune the model using simulated user-feedback. Experiments
are conducted on FGADR dataset.

Within our interactive framework, given an input image, ophthalmologists can
validate predicted lesion feature maps and grading class outputs, as well as
provide feedback on these predictions. For example, if the predicted lesion
maps have false positive or false negative regions, they can highlight the
region and upon verification, these lesion maps can be used to further fine-
tune our intelligent diagnosis system. As discussed in section V-B, we can use
these expert annotations directly to improve the performance of our lesion
attentive grading model. In order to evaluate the effect of expert’s
validation and feedback process on our learning system, we conducted
experiments by simulating user-feedback action. For this, we used FGADR
dataset to simulate user feedback because this dataset has both annotations
for lesion maps and grading tasks.

In figure 14, we illustrate our data split approach for user-feedback
simulation. The original training data is divided into two equal splits. The
disease grading model G-Net in beginning is trained using a half of training
data and their lesion masks predicted by S-Net. The rest of training data is
divided into five parts serving as increasingly data collected from users.
After each update, a new model will be tested again on the fixed hold-out test
set to justify performance. In our setting, instead of directly utilizing
ground-truth samples as user feedback, we assumed that these data contain a
certain noise level. This assumption makes sense in practice as we cannot
guarantee accurate annotations from the user in all cases. Therefore, we
simulated this scenario by applying some morphological operations on the
ground-truth data and used them in the fine-tuning step. Specifically we
randomly applied $\mathrm{erosion}$ and $\mathrm{dilation}$ operations for the
lesion maps with a kernel size $15$. Figure 16 demonstrates the effect of
these operations on a ground-truth lesion mask.

Table IX presents our obtained results on FGADR data for the DR classifier
network with rising user feedback data. At each stage, 10% of additional
simulated user-feedback data is used to fine-tune a current stage of the
grading network. We report both classification and explanation scores in our
experiments, where the explanation score is computed based on the overlapping
between the network’s heatmap regions and detected lesion positions using the
Jaccard similarly Murphy (1996). Table IX demonstrates to us a trend of
improvement in both classification and explanation scores given enlarged user-
feedback data. For example, with 30% or more response data, we achieved
comparable scores against the other competitive methods discussed previously.
In summary, these tests shows that the system can improve even further if
there are feedback from users, and these feedback can be provided with very
little effort from the users.

(a) original

(b) dilation

(c) erosion

Figure 16: Illustration of noisy user-feedback on lesion features maps. To
simulate the user behaviour, we add randomly noises by performing dilation and
erosion on original lesion masks and training models using these perturbed
data. The examples are (a) original segmentation map, (b) lesion map after
performing dilation, and (c) lesion map after erosion morphological operations
using kernel size 15.

## Conclusions and Future Works

This work provides a unified system for DR grading, which simultaneously
learns to predict the disease grading and important lesion features. The
method can automatically detect different types of lesions across domains
while using only annotations in the source domain and unlabeled data in the
target domain. This allows for the rapid deployment of applications in new
datasets without the need for extensive pixel-level annotation, which is often
expensive and time-consuming to prepare. The attention network component of
the proposed method identifies and exploits the most significant lesion
locations for improving performance and for explainability. It also makes the
annotation progress more comfortable for users and make neural networks more
robust in the presence of noise in new data. Moreover, the model also offers a
variety of interpretative outputs to the user, such as DR Grading prediction,
lesion positions, heatmap activation of the network, and the overlapping
between these regions. However, there are various aspects that can be improved
and explored.

First, even though our approach has integrated lesion information for the DR
Grading task, there is still another medical criterion that can be utilized to
improve performance. For instance, extensive characteristics of the lesion
regions such as geometry, area, radius, or the degree of occurrence of
different types of lesions, are other essential factors that need to formulate
during the learning strategies. However, expressing such constraints is not
straightforward since most of them are not differentiable, making end-to-end
learning unfeasible. Fortunately, recent advances in machine learning subjects
like discrete optimization Pogančić et al. (2019), geometric deep learning
Monti et al. (2017); Bronstein et al. (2021), and physic-informed machine
learning Karniadakis et al. (2021) may offer us viable methods for
incorporating these restrictions. For that reason, we believe that expanding
our suggested method in those directions is worth investigating.

Second, the lack of training data is a primary obstacle that hinders the
robustness and generalizability of a trained deep network. While we proposed
techniques based on transfer learning and domain adaptation to alleviate these
challenges, having a powerful pre-trained model is still in high demand.
Currently, self-supervised learning methods trained on large-scale unlabeled
data, namely the foundation model, have succeeded in various downstream tasks
in natural language processing with well-known models such as BERT Dosovitskiy
et al. (2021), DALL-E Ramesh et al. (2021), and GPT-3 Brown et al. (2020).
This raises the question of whether foundation models trained on large-scale
medical datasets can bring similar performance for downstream medical tasks
Nguyen et al. (2022c). In our setting, given such a foundation model, we
expect to advance accuracy for neural networks in both lesion generator and DR
Grading tasks, yielding increasing the performance of the whole system and
reducing user efforts in preparing data annotations.

Third, our proposed method can be applied to other medical applications such
as brain tumor Nguyen et al. (2017, 2022a); Li et al. (2022) or skin cancer
detection Esteva et al. (2017) where jointly localizing lesion information and
data classification plays a crucial one. Therefore, designing experiments for
these scenarios is an important aspect to validate the algorithm’s
generalization.

Last but not least, this study organized experiments to confirm that the
proposed method grows accuracy over time when provided user feedback in
weakly-supervised forms. However, because these results are simulated in the
computer system, they may not cover all real scenarios in practice. This
encourages us to build a real intelligent user interface and deploy it for
real-world applications. Such a system when operating in practice requires
overcoming various barriers. For example, developing a user-friendly and
intuitive UI/UX system so that experts easily engage with and provide feedback
to systems. When experts are in the annotation step, equipping computer-aided
detection (CAD) Yanase and Triantaphyllou (2019) to discover and recommend
similar marked locations to users is also necessary to save time and
accelerate the progress. Finally, when there is a large amount of data
feedback accessible, it poses concerns about learning for new instances while
not forgetting past samples. Such questions are active topic research in
continual learning and active learning, which is also a future direction for
investigation.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i) (a) original image

(j) (b) predicted lesion masks

(k) (c) Activation map overlap
without lesion attention

(l) (d) Activation map overlap with lesion attention (ours)

Figure 17: Comparison on explainability of the disease classification model
trained with and without lesion attention. (a) original inputs, (b) union of
ground-truth lesion maps, (c) overlapping of class activation map (CAM) of the
predicted grading class with the (a) and (b) for G-Net trained without lesion
attention model Att-Net, (d) overlapping of class activation map (CAM) of the
predicted grading class with the (a) and (b) for G-Net trained with lesion
attention model Att-Net. We can observe that CAMs in column (d) overlaps with
most of the lesion regions.

## References

  * Abnar and Zuidema (2020) Abnar, S., Zuidema, W., 2020\.  Quantifying attention flow in transformers, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online. pp. 4190–4197.  URL: https://aclanthology.org/2020.acl-main.385, doi:10.18653/v1/2020.acl-main.385.
  * Al-Masni et al. (2020) Al-Masni, M.A., Kim, D.H., Kim, T.S., 2020.  Multiple skin lesions diagnostics via integrated deep convolutional networks for segmentation and classification.  Computer methods and programs in biomedicine 190, 105351.
  * Alban and Gilligan (2016) Alban, M., Gilligan, T., 2016\.  Automated detection of diabetic retinopathy using fluorescein angiography photographs.  Report of standford education .
  * Amershi et al. (2014) Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T., 2014\.  Power to the people: The role of humans in interactive machine learning.  Ai Magazine 35, 105–120.
  * Antal and Hajdu (2012) Antal, B., Hajdu, A., 2012.  An ensemble-based system for microaneurysm detection and diabetic retinopathy grading.  IEEE transactions on biomedical engineering 59, 1720–1726.
  * Arjovsky et al. (2017) Arjovsky, M., Chintala, S., Bottou, L., 2017.  Wasserstein generative adversarial networks, in: International conference on machine learning, PMLR. pp. 214–223.
  * Bronstein et al. (2021) Bronstein, M.M., Bruna, J., Cohen, T., Veličković, P., 2021\.  Geometric deep learning: Grids, groups, graphs, geodesics, and gauges.  arXiv preprint arXiv:2104.13478 .
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al., 2020\.  Language models are few-shot learners.  Advances in neural information processing systems 33, 1877–1901.
  * Chen et al. (2017) Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L., 2017.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence 40, 834–848.
  * Chen et al. (2016) Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P., 2016\.  Infogan: Interpretable representation learning by information maximizing generative adversarial nets.  Advances in neural information processing systems 29\.
  * Cheplygina (2019) Cheplygina, V., 2019.  Cats or cat scans: Transfer learning from natural or medical image source data sets?  Current Opinion in Biomedical Engineering 9, 21–27.
  * Dai et al. (2021) Dai, L., Wu, L., Li, H., Cai, C., Wu, Q., Kong, H., Liu, R., Wang, X., Hou, X., Liu, Y., et al., 2021\.  A deep learning system for detecting diabetic retinopathy across the disease spectrum.  Nature communications 12, 1–11.
  * Dosovitskiy et al. (2021) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al., 2021\.  An image is worth 16x16 words: Transformers for image recognition at scale.  International Conference on Learning Representations (ICLR) .
  * Esteva et al. (2017) Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., Thrun, S., 2017.  Dermatologist-level classification of skin cancer with deep neural networks.  nature 542, 115–118.
  * EyePACS (2015) EyePACS, 2015.  Eyepacs challenge kaggle diabetic retinopathy dataset.  https://www.kaggle.com/c/diabetic-retinopathy-detection/data.  Accessed: 2021-10-12.
  * Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., 2014\.  Generative adversarial nets.  Advances in neural information processing systems 27\.
  * Gretton et al. (2012) Gretton, A., Borgwardt, K.M., Rasch, M.J., Schölkopf, B., Smola, A., 2012.  A kernel two-sample test.  The Journal of Machine Learning Research 13, 723–773.
  * Gulrajani et al. (2017) Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C., 2017.  Improved training of wasserstein gans.  Advances in neural information processing systems 30\.
  * Gulshan et al. (2016) Gulshan, V., Peng, L., Coram, M., Stumpe, M.C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al., 2016\.  Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.  Jama 316, 2402–2410.
  * He et al. (2016) He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
  * He et al. (2020) He, X., Yang, X., Zhang, S., Zhao, J., Zhang, Y., Xing, E., Xie, P., 2020.  Sample-efficient deep learning for covid-19 diagnosis based on ct scans.  medrxiv .
  * Hung et al. (2018) Hung, W.C., Tsai, Y.H., Liou, Y.T., Lin, Y.Y., Yang, M.H., 2018.  Adversarial learning for semi-supervised semantic segmentation.  Proceedings of the British Machine Vision Conference (BMVC) .
  * Islam et al. (2019) Islam, M., Vibashan, V., Jose, V., Wijethilake, N., Utkarsh, U., Ren, H., 2019\.  Brain tumor segmentation and survival prediction using 3d attention unet, in: International MICCAI Brainlesion Workshop, Springer. pp. 262–272.
  * Isola et al. (2017) Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A., 2017\.  Image-to-image translation with conditional adversarial networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125–1134.
  * Jadon (2020) Jadon, S., 2020.  A survey of loss functions for semantic segmentation, in: 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), IEEE. pp. 1–7.
  * Jalwana et al. (2021) Jalwana, M.A., Akhtar, N., Bennamoun, M., Mian, A., 2021\.  Cameras: Enhanced resolution and sanity preserving class activation mapping for image saliency, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16327–16336.
  * Karniadakis et al. (2021) Karniadakis, G.E., Kevrekidis, I.G., Lu, L., Perdikaris, P., Wang, S., Yang, L., 2021.  Physics-informed machine learning.  Nature Reviews Physics 3, 422–440.
  * Le et al. (2021) Le, N., Le, T., Yamazaki, K., Bui, T., Luu, K., Savides, M., 2021.  Offset curves loss for imbalanced problem in medical segmentation, in: 2020 25th International Conference on Pattern Recognition (ICPR), IEEE. pp. 9189–9195.
  * Li et al. (2022) Li, J.P., Khan, S., Alshara, M.A., Alotaibi, R.M., Mawuli, C., et al., 2022.  Dacbt: deep learning approach for classification of brain tumors using mri data in iot healthcare environment.  Scientific Reports 12, 1–14.
  * Li et al. (2018a) Li, K., Wu, Z., Peng, K.C., Ernst, J., Fu, Y., 2018a.  Tell me where to look: Guided attention inference network, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9215–9223.
  * Li et al. (2020) Li, L., Verma, M., Nakashima, Y., Kawasaki, R., Nagahara, H., 2020.  Joint learning of vessel segmentation and artery/vein classification with post-processing, in: Medical Imaging with Deep Learning, PMLR. pp. 440–453.
  * Li et al. (2018b) Li, X., Chen, H., Qi, X., Dou, Q., Fu, C.W., Heng, P.A., 2018b.  H-denseunet: hybrid densely connected unet for liver and tumor segmentation from ct volumes.  IEEE transactions on medical imaging 37, 2663–2674.
  * Lin et al. (2018) Lin, Z., Guo, R., Wang, Y., Wu, B., Chen, T., Wang, W., Chen, D.Z., Wu, J., 2018.  A framework for identifying diabetic retinopathy based on anti-noise detection and attention-based fusion, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 74–82.
  * Liu and Belkin (2018) Liu, C., Belkin, M., 2018.  Accelerating sgd with momentum for over-parameterized learning.  arXiv preprint arXiv:1810.13395 .
  * Liu et al. (2019) Liu, H., Gu, X., Samaras, D., 2019.  Wasserstein gan with quadratic transport cost, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 4832–4841.
  * Long et al. (2015) Long, J., Shelhamer, E., Darrell, T., 2015.  Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440.
  * Ma et al. (2021) Ma, A., Li, J., Lu, K., Zhu, L., Shen, H.T., 2021\.  Adversarial entropy optimization for unsupervised domain adaptation.  IEEE Transactions on Neural Networks and Learning Systems .
  * Maadi et al. (2021) Maadi, M., Akbarzadeh Khorshidi, H., Aickelin, U., 2021.  A review on human–ai interaction in machine learning and insights for medical applications.  International journal of environmental research and public health 18, 2121.
  * McHugh (2012) McHugh, M., 2012.  interrater reliability: the kappa statistic. biochemica medica, 22 (3), 276–282.
  * Mehta et al. (2018) Mehta, S., Mercan, E., Bartlett, J., Weaver, D., Elmore, J.G., Shapiro, L., 2018\.  Y-net: joint segmentation and classification for diagnosis of breast biopsy images, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 893–901.
  * Mirza and Osindero (2014) Mirza, M., Osindero, S., 2014\.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784 .
  * Monti et al. (2017) Monti, F., Boscaini, D., Masci, J., Rodola, E., Svoboda, J., Bronstein, M.M., 2017\.  Geometric deep learning on graphs and manifolds using mixture model cnns, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5115–5124.
  * Murphy (1996) Murphy, A.H., 1996.  The finley affair: A signal event in the history of forecast verification.  Weather and forecasting 11, 3–20.
  * Nguyen et al. (2022a) Nguyen, D.H., Nguyen, D.M., Mai, T.T.N., Nguyen, T., Tran, K.T., Nguyen, A.T., Pham, B.T., Nguyen, B.T., 2022a.  Asmcnn: An efficient brain extraction using active shape model and convolutional neural networks.  Information Sciences 591, 25–48.
  * Nguyen et al. (2021a) Nguyen, D.M., Mai, T.T.N., Than, N.T., Prange, A., Sonntag, D., 2021a.  Self-supervised domain adaptation for diabetic retinopathy grading using vessel image reconstruction, in: German Conference on Artificial Intelligence (Künstliche Intelligenz), Springer. pp. 349–361.
  * Nguyen et al. (2021b) Nguyen, D.M., Nguyen, D.M., Vu, H., Nguyen, B.T., Nunnari, F., Sonntag, D., 2021b.  An attention mechanism using multiple knowledge sources for covid-19 detection from ct images, in: The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21), Workshop: Trustworthy AI for Healthcare.
  * Nguyen et al. (2022b) Nguyen, D.M., Nguyen, T.T., Vu, H., Pham, Q., Nguyen, M.D., Nguyen, B.T., Sonntag, D., 2022b.  TATL: task agnostic transfer learning for skin attributes detection.  Medical Image Analysis 78, 102359\.
  * Nguyen et al. (2017) Nguyen, D.M., Vu, H.T., Ung, H.Q., Nguyen, B.T., 2017\.  3d-brain segmentation using deep neural network and gaussian mixture model, in: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE. pp. 815–824.
  * Nguyen et al. (2022c) Nguyen, D.M.H., Nguyen, H., Truong, M.T.N., Cao, T., Nguyen, B.T., Ho, N., Swoboda, P., Albarqouni, S., Xie, P., Sonntag, D., 2022c.  Joint self-supervised image-volume representation learning with intra-inter contrastive clustering.  arXiv preprint arXiv:2212.01893 .
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S., 2019.  Pytorch: An imperative style, high-performance deep learning library, in: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (Eds.), Advances in Neural Information Processing Systems 32. Curran Associates, Inc., pp. 8024–8035.
  * Pogančić et al. (2019) Pogančić, M.V., Paulus, A., Musil, V., Martius, G., Rolinek, M., 2019.  Differentiation of blackbox combinatorial solvers, in: International Conference on Learning Representations.
  * Porwal et al. (2018) Porwal, P., Pachade, S., Kamble, R., Kokare, M., Deshmukh, G., Sahasrabuddhe, V., Meriaudeau, F., 2018.  Indian diabetic retinopathy image dataset (idrid): a database for diabetic retinopathy screening research.  Data 3, 25.
  * Prange et al. (2017) Prange, A., Chikobava, M., Poller, P., Barz, M., Sonntag, D., 2017.  A multimodal dialogue system for medical decision support inside virtual reality, in: Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 23–26.
  * Ramesh et al. (2021) Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I., 2021.  Zero-shot text-to-image generation, in: International Conference on Machine Learning, PMLR. pp. 8821–8831.
  * Ronneberger et al. (2015) Ronneberger, O., Fischer, P., Brox, T., 2015.  U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical image computing and computer-assisted intervention, Springer. pp. 234–241.
  * Ruder (2016) Ruder, S., 2016.  An overview of gradient descent optimization algorithms.  arXiv preprint arXiv:1609.04747 .
  * Saeedi et al. (2019) Saeedi, P., Petersohn, I., Salpea, P., Malanda, B., Karuranga, S., Unwin, N., Colagiuri, S., Guariguata, L., Motala, A.A., Ogurtsova, K., et al., 2019\.  Global and regional diabetes prevalence estimates for 2019 and projections for 2030 and 2045: Results from the international diabetes federation diabetes atlas.  Diabetes research and clinical practice 157, 107843.
  * Saini and Prasad (2021) Saini, A., Prasad, R., 2021\.  Select wisely and explain: Active learning and probabilistic local post-hoc explainability.  URL: https://arxiv.org/abs/2108.06907, doi:10.48550/ARXIV.2108.06907.
  * Sankar et al. (2016) Sankar, M., Batri, K., Parvathi, R., 2016.  Earliest diabetic retinopathy classification using deep convolution neural networks. pdf.  Int. J. Adv. Eng. Technol 10, M9.
  * Selvaraju et al. (2017) Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D., 2017\.  Grad-cam: Visual explanations from deep networks via gradient-based localization, in: Proceedings of the IEEE international conference on computer vision, pp. 618–626.
  * Shannon (2001) Shannon, C.E., 2001.  A mathematical theory of communication.  ACM SIGMOBILE mobile computing and communications review 5, 3–55.
  * Shen et al. (2018) Shen, J., Qu, Y., Zhang, W., Yu, Y., 2018.  Wasserstein distance guided representation learning for domain adaptation, in: Thirty-second AAAI conference on artificial intelligence.
  * Smith (2017) Smith, L.N., 2017.  Cyclical learning rates for training neural networks, in: 2017 IEEE winter conference on applications of computer vision (WACV), IEEE. pp. 464–472.
  * Sonntag et al. (2020) Sonntag, D., Nunnari, F., Profitlich, H.J., 2020.  The skincare project, an interactive deep learning system for differential diagnosis of malignant skin lesions. technical report.  arXiv preprint arXiv:2005.09448 .
  * Sonntag et al. (2012) Sonntag, D., Schulz, C., Reuschling, C., Galarraga, L., 2012\.  Radspeech’s mobile dialogue system for radiologists, in: Proceedings of the 2012 ACM international conference on Intelligent User Interfaces, pp. 317–318.
  * Sonntag et al. (2010) Sonntag, D., Wennerberg, P., Buitelaar, P., Zillner, S., 2010\.  Pillars of ontology treatment in the medical domain.
  * Sun et al. (2021) Sun, R., Li, Y., Zhang, T., Mao, Z., Wu, F., Zhang, Y., 2021.  Lesion-aware transformers for diabetic retinopathy grading, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10938–10947.
  * Tsai et al. (2018) Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H., Chandraker, M., 2018\.  Learning to adapt structured output space for semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7472–7481.
  * Tzeng et al. (2017) Tzeng, E., Hoffman, J., Saenko, K., Darrell, T., 2017\.  Adversarial discriminative domain adaptation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167–7176.
  * Tzeng et al. (2014) Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T., 2014.  Deep domain confusion: Maximizing for domain invariance.  arXiv preprint arXiv:1412.3474 .
  * Vu et al. (2019) Vu, T.H., Jain, H., Bucher, M., Cord, M., Pérez, P., 2019.  Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2517–2526.
  * Wang et al. (2020) Wang, H., Wang, Z., Du, M., Yang, F., Zhang, Z., Ding, S., Mardziel, P., Hu, X., 2020.  Score-cam: Score-weighted visual explanations for convolutional neural networks, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 24–25.
  * Wang et al. (2017) Wang, Z., Yin, Y., Shi, J., Fang, W., Li, H., Wang, X., 2017.  Zoom-in-net: Deep mining lesions for diabetic retinopathy detection, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 267–275.
  * Wu et al. (2021) Wu, Y.H., Gao, S.H., Mei, J., Xu, J., Fan, D.P., Zhang, R.G., Cheng, M.M., 2021\.  JCS: An explainable covid-19 diagnosis system by joint classification and segmentation.  IEEE Transactions on Image Processing 30, 3113–3126.
  * Xiao et al. (2019) Xiao, Q., Zou, J., Yang, M., Gaudio, A., Kitani, K., Smailagic, A., Costa, P., Xu, M., 2019.  Improving lesion segmentation for diabetic retinopathy using adversarial learning, in: International Conference on Image Analysis and Recognition, Springer. pp. 333–344.
  * Yan et al. (2019) Yan, W., Wang, Y., Gu, S., Huang, L., Yan, F., Xia, L., Tao, Q., 2019.  The domain shift problem of medical image segmentation and vendor-adaptation by unet-gan, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 623–631.
  * Yanase and Triantaphyllou (2019) Yanase, J., Triantaphyllou, E., 2019\.  A systematic survey of computer-aided diagnosis in medicine: Past and present developments.  Expert Systems with Applications 138, 112821.
  * Yang et al. (2017) Yang, Y., Li, T., Li, W., Wu, H., Fan, W., Zhang, W., 2017.  Lesion detection and grading of diabetic retinopathy via two-stages deep convolutional neural networks, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 533–540.
  * Yang et al. (2021) Yang, Y., Shang, F., Wu, B., Yang, D., Wang, L., Xu, Y., Zhang, W., Zhang, T., 2021.  Robust collaborative learning of patch-level and image-level annotations for diabetic retinopathy grading from fundus image.  IEEE Transactions on Cybernetics .
  * Yu et al. (2021) Yu, S., Ma, K., Bi, Q., Bian, C., Ning, M., He, N., Li, Y., Liu, H., Zheng, Y., 2021.  Mil-vt: Multiple instance learning enhanced vision transformer for fundus image classification, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 45–54.
  * Zhou et al. (2016) Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A., 2016.  Learning deep features for discriminative localization, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929.
  * Zhou et al. (2019) Zhou, Y., He, X., Huang, L., Liu, L., Zhu, F., Cui, S., Shao, L., 2019.  Collaborative learning of semi-supervised segmentation and classification for medical images, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2079–2088.
  * Zhou et al. (2020) Zhou, Y., Wang, B., Huang, L., Cui, S., Shao, L., 2020\.  A benchmark for studying diabetic retinopathy: Segmentation, grading, and transferability.  IEEE Transactions on Medical Imaging 40, 818–828.