# Incremental Cross-Domain Adaptation for Robust Retinopathy Screening via
Bayesian Deep Learning

Taimur Hassan*, Bilal Hassan, Muhammad Usman Akram, Shahrukh Hashmi, Abdel
Hakim Taguri, Naoufel Werghi © 2021 IEEE. Personal use of this material is
permitted. Permission from IEEE must be obtained for all other uses, in any
current or future media, including reprinting/republishing this material for
advertising or promotional purposes, creating new collective works, for resale
or redistribution to servers or lists, or reuse of any copyrighted component
of this work in other works.This work is supported by a research fund from
Khalifa University: Ref: CIRA-2019-047, and the Abu Dhabi Department of
Education and Knowledge (ADEK), Ref: AARE19-156.T. Hassan and N. Werghi are
with the Center for Cyber-Physical Systems (C2PS), Department of Electrical
Engineering and Computer Science, Khalifa University, Abu Dhabi, United Arab
Emirates.T. Hassan and M. U. Akram are with the Department of Computer and
Software Engineering, National University of Sciences and Technology,
Islamabad, Pakistan.B. Hassan is with the School of Automation Science and
Electrical Engineering, Beihang University, Beijing, China.S. Hashmi is with
the Department of Internal Medicine, Mayo Clinic, Minnesota, USAA. H. Taguri
is with the Abu Dhabi Healthcare Company (SEHA), Abu Dhabi, United Arab
Emirates.* Corresponding author, Email<EMAIL_ADDRESS>
###### Abstract

Retinopathy represents a group of retinal diseases that, if not treated
timely, can cause severe visual impairments or even blindness. Many
researchers have developed autonomous systems to recognize retinopathy via
fundus and optical coherence tomography (OCT) imagery. However, most of these
frameworks employ conventional transfer learning and fine-tuning approaches,
requiring a decent amount of well-annotated training data to produce accurate
diagnostic performance. This paper presents a novel incremental cross-domain
adaptation instrument that allows any deep classification model to
progressively learn abnormal retinal pathologies in OCT and fundus imagery via
few-shot training. Furthermore, unlike its competitors, the proposed
instrument is driven via a Bayesian multi-objective function that not only
enforces the candidate classification network to retain its prior learned
knowledge during incremental training but also ensures that the network
understands the structural and semantic relationships between previously
learned pathologies and newly added disease categories to effectively
recognize them at the inference stage. The proposed framework, evaluated on
six public datasets acquired with three different scanners to screen thirteen
retinal pathologies, outperforms the state-of-the-art competitors by achieving
an overall accuracy and F1 score of 0.9826 and 0.9846, respectively.

###### Index Terms:

Incremental Domain Adaptation, Bayesian Deep Learning, Retinopathy, Optical
Coherence Tomography, Fundus Photography.

##  I Introduction

The human eye consists of three layers, where the retina is the innermost
layer responsible for producing vision. Retinal diseases or retinopathy tend
to damage the retina resulting in a severe loss of vision or even blindness
[1]. Some of the serious retinal diseases are diabetic macular edema (DME),
age-related macular degeneration (AMD), and central serous retinopathy (CSR).
DME is caused by hyperglycemia (diabetes), where blood vessels become thinner
and start leaking fluid deposits within the retina [2] According to the Early
Treatment Diabetic Retinopathy Study (ETDRS), DME is graded as clinically
significant macular edema (CSME) if 1) the retinal thickening is present
within 500$\mu$m of the macular center, 2) hard exudates are discovered within
500$\mu$m of the macular center with adjacent macular thickening, and 3) there
is a retinal thickening of one or more disc diameters and part of this
thickening is within one disc diameter of the macular center [3]. Otherwise,
it is graded as non-clinically significant macular edema (non-CSME).
Similarly, the severity of CSME is further graded as centrally-involved DME
(ci-DME) if retinal thickening is observed within the central sub-field zone
of the macula (having diameter $\geq$ 1mm) in optical coherence tomography
(OCT) scans. Otherwise, DME is classified as non-centrally involved (nci-DME)
[4]. AMD is another retinal condition (mostly found in elder people) that
causes severe visual impairments if not treated timely. AMD is typically
graded into two stages, i.e., the dry AMD and the wet AMD. Dry AMD is an early
stage in which the retinal pigment epithelium (RPE) layer starts to
degenerate, producing drusen and causing the subjects’ central vision to
become vivid and twirled. With the disease’s progression, the abnormal blood
vessels start to grow from the choroid and intercept the retina, producing
retinal fluids and other chorioretinal abnormalities such as scars and
choroidal neovascularization (CNV). This stage is typically graded as wet AMD.
CSR is another retinal syndrome that is mainly caused due to stress.
Clinically, CSR is diagnosed by observing the formation of serous detachment
beneath the retina, and it is graded as acute, acute-persistent, and chronic.
Acute CSR is an early stage where fluid-filled serous detachment forms beneath
the retina (near to fovea). It becomes persistent if the serous detachment
lasts for more than three months, which eventually leads to long-lasting
chronic CSR in which RPE starts to degenerate and leads towards the formation
of fibrosis, and CNV [5].

The retinal pathologies can be identified in a non-invasive manner through
retinal fundus [6] and OCT examinations [7]. OCT imagery is more advantageous
than fundus imagery as it gives a cross-sectional view of the retina for early
disease identification. Also, OCT imagery can aid ophthalmologists in
objectively assessing the severity of the underlying disease, resulting in a
quick, accurate, and objective diagnosis. Nevertheless, the significance of
fundus imagery for screening and grading certain retinal diseases cannot be
fully ignored [8]. Also, OCT imagery cannot identify blood and is limited
towards detecting diseases that involve bleeding retina [9]. In such cases,
fundus imagery provides an excellent alternative, aiding the doctors in
analyzing the underlying pathologies.

###  I-A Motivation

Many researchers have developed novel solutions for extracting retinal layers
[10], and retinal lesions [11] for the lesion-aware screening [12] and grading
of retinopathy [13]. Most of these frameworks utilize standard pre-trained
networks in a transfer learning mode, which eliminates the need for training
the model from scratch [10]. However, the transfer learning and fine-tuning
models have an inherent limitation of forgetting the prior knowledge upon
learning new target-domain tasks [14], and this limitation constrain them to
identify only a limited number of pathologies. Scaling up these models to
accommodate new disease patterns across different modalities requires
computational, resource-demanding, and explicit re-training routines [12, 15].
Moreover, re-training these models again (to overcome scanner or pathological
differences) is also an infeasible option for the clinicians at the hospitals
and clinical setups. To cater these issues, we present a novel incremental
cross-domain adaptation instrument that, with few-shot training, enables the
deep classification models to recognize different retinal pathologies (across
multi-modal imagery) irrespective of the scanner specifications or the
pathological differences. Furthermore, the proposed framework can be easily
tailored (in clinical settings) to recognize additional disease variants
(while distilling its previously learned knowledge) due to its capacity to
analyze the mutual contextual, structural, and spatial differences between
incrementally learned knowledge representations via Bayesian inference.

##  II Related Work

Retinal image analysis is a widely researched topic [8], where researchers
have developed various solutions to analyze retinal layers [16], retinal
lesions [13, 17] via fundus [18], OCT [19], and fused fundus and OCT imagery
[4]. In this section, we first categorize the literature based upon their
conventional deep learning approaches. Afterward, we shed light on some of the
recent frameworks which use advanced deep learning schemes to recognize
retinal disease patterns.

###  II-A Conventional Deep Learning Approaches

Deep learning has been extensively utilized to detect normal and abnormal
retinal pathologies [20]. Kermany et al. [21] pioneered these efforts by
developing a CNN-based referral framework to predict AMD, CNV, DME, and
normative pathologies from the macular OCT scans [21]. Furthermore, they
released their dataset publicly, which is one of the largest retinal OCT
datasets to date. Rong et al. [22] proposed a surrogate-assisted CNN
classification model to recognize AMD, DME, and normal pathologies depicted
within the retinal OCT scans. Lee et al. [23] developed a deep learning system
to automatically screen healthy and AMD pathologies. Arcadu et al. [24]
developed a CNN coupled random forest to screen and grade diabetic retinopathy
from fundus imagery. Yoo et al. [12] proposed a multi-modal approach whereby a
pre-trained VGG-19 [25] is used to extract feature representations from both
fundus and OCT imagery to predict AMD pathologies. Apart from this,
researchers have also devised hybrid CNN models for lesion-aware screening
[13] and grading [26] of retinopathy from both fundus [27], and OCT imagery
[5].

###  II-B Advanced Deep Learning Methods

To overcome the requirement of a large-scale and well-annotated training data,
researchers have developed advanced deep learning approaches employing
incremental learning [28], multi-task learning [29], meta-learning [30], and
domain adaptation [31]. Similarly, many researchers have turned their
attention towards utilizing these schemes for extracting retinal lesions and
screening retinal pathologies [32]. In this context, the work of He et al.
[33] is particularly appreciable as they utilized unsupervised domain
adaptation, via adversarial learning, to segment retinal boundaries from the
healthy OCT B-scans, which are acquired through Cirrus and Spectralis
machines. Hasan et al. [30] utilized model agnostic meta-learning [34] to
register multi-vendor retinal images by refining a transformation matrix
obtained through the spatial transformer in an adversarial manner [30].
Similarly, Ju et al. [35] utilized adversarial domain adaptation for achieving
cross-domain shifts between normal and ultra-wide-field fundus photography to
diagnose diabetic retinopathy, AMD, and glaucoma. Meng et al. [32] developed
an attribute-driven incremental network (ADINet) to learn retinal diagnostic
tasks from the fundus imagery incrementally.

The above-mentioned approaches have tried to overcome the requirement of
large-scale and well-annotated data for training deep learning models.
However, to the best of our knowledge, there is no scheme till now that allows
the classification networks to incrementally learn retinopathy screening tasks
across multi-modal domains via few-shot training and perform these screening
tasks simultaneously, at the inference stage, by analyzing the domain-specific
abnormal retinal attributes. Such a scheme is highly desirable in clinical
practice, as it not only saves time for re-training the model repeatedly on
the large-scale data to learn new disease patterns, but it also allows the
clinicians to train a single model, with few training examples, to recognize
rarely occurring retinal diseases which are observed across the multi-modal
imagery.

Figure 1: Block diagram of the proposed framework. The first training stage is
shown in (A), in which the classification network is incrementally trained on
the first target domain to recognize $k+1$ retinal pathologies from the OCT
scans. (B) denotes the second training stage in which we incrementally adapt
the network to recognize $m+2$ retinal pathologies from the fundus imagery.
During the inference stage (C), for $k=6,m=11$, the proposed incrementally
trained cross-domain classifier can mass-screen thirteen retinal pathologies
from both fundus and OCT imagery simultaneously, irrespective of the scanner
specifications.

###  II-C Contributions

This paper presents an original incremental cross-domain adaptation approach
that constrains the classification model via Bayesian inference to learn
retinopathy screening tasks across multi-modal imagery in an incremental
fashion. The proposed scheme eliminates the need for re-training or fine-
tuning the model for learning new disease categories every time. This
distinctive feature of our framework is quite relevant for screening rarely
occurring diseases (such as different variants of CSR [36]) for which
procuring large-scale training data is both infeasible and impractical [5].
Furthermore, the proposed framework, unlike its competitors [12, 37, 15],
eliminates the need for deploying separate models to screen retina across
multiple modalities. Thus, it also overcomes the training overheads of
explicitly tuning multiple networks. In addition to this, the proposed
framework presents a scalable option, in clinical practice, for autonomous
retinal screening using OCT and fundus imagery. To summarize, the main
features of this paper are:

  * •

This paper presents a novel scheme that exploits deep neural networks’
tendency to incrementally learn cross-domain diagnostic tasks (via few
training examples) and perform them simultaneously (at the inference stage) by
paying attention to the disease-specific lesions across each modality.

  * •

Furthermore, to the best of our knowledge, the proposed framework is the first
attempt towards utilizing incremental cross-domain adaptation for screening
retinal diseases across OCT and fundus imagery.

  * •

The proposed incremental cross-domain adaptation is achieved by periodically
minimizing the proposed continual learning loss function ($L_{cl}$), which
enables the classification model to screen retinal diseases (from multiple
modalities) by identifying their structural similarities and complex inter-
dependencies via Bayesian inference.

The rest of the paper is organized as follows: Section III presents the
details of the proposed framework, Section IV describes the experimental
setup, Section V showcases the detailed evaluation results, Section VI
presents in-depth discussion about proposed framework and Section VII
concludes the paper.

##  III Proposed Scheme

The block diagram of the proposed scheme is shown in Figure 1. Here, we can
see that the candidate classification model recognizes the normal and abnormal
retinal pathologies from both OCT and fundus imagery using a novel incremental
cross-domain adaptation scheme based on Bayesian inference. The complete
framework training is performed in two stages where, in the first stage, the
classification model is incrementally trained with few OCT training examples
to recognize ci-DME, normal OCT, nci-DME, dry AMD, wet AMD, acute CSR, and
chronic CSR pathologies. Afterward, in the second stage, it is incrementally
adapted to fundus imagery to recognize CSME, non-CSME, dry AMD, wet AMD, and
CSR pathologies. We also wanted to highlight that our proposed scheme is
different than the conventional incremental learning and domain adaptation
(DA) approaches. In traditional incremental learning schemes, the model
periodically learns new classification tasks (belonging to the same domain)
without catastrophically forgetting its prior knowledge [32]. This is achieved
by constraining the underlying model (in each training increment) via
distillation-driven loss functions, which ensures that the model remembers its
prior knowledge (through past episodic memories [38] or through previously fed
examples [39]) while learning the newly added class representations. Moreover,
the weights of the incremental learning model are tuned in a way that they
produce decent classification performance (for predicting both old and newly
learned classes) at the inference stage. Apart from this, in conventional DA
approaches, the model first learns the source domain tasks from the provided
training examples and then adapts to the target domains to perform the same
inter-related tasks by resolving the contextual and spatial differences
between the source-target domain scans [40, 33]111More details on transfer
learning, incremental learning, and domain adaptation are presented in a
supporting document within the source code repository.. Contrary to this, in
the proposed approach, the classification model is incrementally adapted to
multiple target domains such that, with few-shot training, it can perform all
the learned tasks simultaneously. The backbone of our proposed approach is a
novel knowledge distillation strategy which, instead of treating the
incrementally learned cross-domain class representations independent of each
other, exploits their mutual relationships via Bayesian inference and make the
model understand these relationships to maximize the classification
performance (even when the model is trained with few examples). Also, unlike
the conventional DA schemes, the proposed scheme allows the underlying network
to new cross-domain tasks which are not necessarily related to each other. For
example, the classification model (trained using the proposed scheme) can
easily screen the ci-DME and nci-DME pathologies from OCT scans while
recognizing the non-CSME pathologies from the fundus scans (where ci-DME, nci-
DME in OCT scans has no association with the non-CSME pathology as it can only
be visualized in the fundus scans). Therefore, we dubbed the proposed scheme
as incremental cross-domain adaptation. The detailed description of the
proposed framework’s training and testing are presented in the subsequent
sections:

###  III-A Training Stage-I

In the first training stage, we pick the pre-trained network (the model
previously trained on a source domain dataset, i.e., ImageNet [41]). Then, in
each training increment, we update its final layers to recognize retinal
pathologies from the OCT imagery, which is the first target domain. In the
first iteration, we make the network learn normal and ci-DME pathologies using
their respective training examples. Similarly, in the second iteration, we
stack a new class, representing ’nci-DME’ pathology, in the final layer and
constrain the model to recognize the nci-DME affected OCT scans. The model in
the second iteration effectively learns the nci-DME disease patterns while
retaining its previous knowledge about normal and ci-DME pathologies. The same
process is repeated till $k$ iterations, where the network is periodically
trained to screen $k+1$ retinal pathologies. It should be noted here that in
each iteration $j$ (where $j>1$), the network is constrained through the
proposed $L_{cl}$ loss function (see Section III-D) to identify new types of
retinal pathology while retaining its prior knowledge. Moreover, instead of
only using training examples for the newly added categories, we also feed the
network with the subset of examples used in the $1^{st}$ to $(j-1)^{th}$
iteration. This enables the $L_{cl}$ function to distill the network’s
previous knowledge while constraining it to learn newly added disease
categories. Furthermore, $L_{cl}$ (via $L_{md}$) ensures that the network
learns the structural and semantic relationships between previously learned
and newly stacked classes to recognize them correctly at the inference stage.
More details about $L_{md}$ and $L_{cl}$ are presented in Sections (III-D,
III-E, and III-F). Moreover, at $j=1$, we only train the model using the
categorical cross-entropy loss function ($L_{ce}$) since we don’t have any old
classes in the first iteration (from the source-domain), which we want the
network to retain.

###  III-B Training Stage-II

In the second stage, we adapt the model, trained on the OCT domain, to
recognize the abnormal retinal pathologies from the fundus imagery. The
adaptation is performed incrementally where the network in iteration $k+1$ is
constrained to identify healthy and CSME pathologies from the fundus scans.
The same process is repeated till the $m^{th}$ iteration, where the network is
trained to screen up to $m+2$ retinal pathologies. Note that the model, in the
second training stage, not only learns to identify retinal pathologies across
two different types of imagery but it also learns to recognize similar disease
patterns across both domains to analyze the underlying disease severity. For
example, at the inference stage, for the DME patient, the proposed framework
possesses the capacity to screen it as CSME and non-CSME symptomatic using the
corresponding fundus scan. If it is CSME, then the proposed framework can
further check whether the screened CSME case is ci-DME or nci-DME by searching
for the presence of retinal lesions such as intra-retinal fluid (IRF) sub-
retinal fluid (SRF) and hard exudates (HE) within the respective OCT volume.
The model achieves such extended screening without requiring any additional
data or re-training, while to achieve a similar behavior, state-of-the-art
works, based on conventional transfer learning, would have to be explicitly
trained on the large well-annotated data [12, 15].

###  III-C Inference Stage

After training the model till the $m^{th}$ incremental iteration, we use it to
screen $m+2$ retinal pathologies at the inference stage. We highlight here
that the proposed framework employs a single classification network to screen
retinal diseases from OCT and fundus imagery by analyzing their modality-
specific abnormal lesions. For example, if an OCT scan of a CSR patient is
passed into the proposed system, it will be screened as acute CSR or chronic
CSR by observing the presence of IRF, SRF, and CNV. Similarly, if a fundus
scan of the same subject is passed into the system, then it will be screened
as CSR by analyzing the appearance of fluid blisters. Now, suppose the model
has to be tuned to recognize more types of pathologies. In that case, we only
need to run one more incremental training iteration in which we will first
stack the categories of these new pathologies within the final classification
layer of the network. Then, we can make the model learn these newly added
disease categories while retaining its previous knowledge, using the proposed
$L_{cl}$ loss function and a small set of training examples. Thus, the
proposed scheme allows low-cost scalability for screening retinopathy in
clinical settings.

###  III-D Loss Functions

During incremental training, we constrain the classification network to
accurately learn the newly added classes while retaining its prior learned
knowledge via the proposed continual learning objective function ($L_{cl}$).
This function encompasses three sub-objective functions, namely $L_{o}$,
$L_{n}$, and $L_{md}$. $L_{o}$ allows the classification network to retain the
prior knowledge by minimizing the prediction errors on the training examples
of the old classes (learned in the previous $1$ to $j-1$ iterations). $L_{n}$
enables the classification model to learn new classes (added in the $j^{th}$
iteration) by minimizing the network loss on the training examples of the new
classes. Both $L_{o}$ and $L_{n}$ update the network weights such that it
learns the maximum amount of information while showing high resistance against
catastrophic forgetting. The loss function $L_{cl}$ also analyzes the mutual
relationship and structural inter-dependencies between the previously added
and the newly stacked classes via an auxiliary $L_{md}$ sub-objective
function-driven via Bayesian inference. $L_{md}$ gives more exposure to the
underlying classification model to understand and learn the diverse ranging
yet semantically related categories by exploiting their inter-dependencies. To
the best of our knowledge, all the state-of-the-art knowledge distillation
schemes ignore this aspect, assuming the older and newer knowledge
representations are independent of each other. More details about $L_{md}$,
$L_{o}$ and $L_{n}$ are exposed in the next sections.

###  III-E Mutual Distillation Loss Function

Knowledge representations learned by the classification model in each
iteration, during incremental training, are generally non-mutually exclusive
[42]. For example: consider a fruit classification model that learns to
classify oranges in one iteration and then lemons in the next iteration. Since
both of the fruits are related to each other, therefore, the classifier should
be constrained during the incremental training so that it can accurately
recognize these fruits together. More specifically, it should not lose its
previously attained knowledge of recognizing oranges while learning about
lemons. To address this, we propose $L_{md}$ loss function that exploits the
mutual structural and semantic inter-dependencies between different knowledge
representations (during the incremental training process) to enhance the
classification performance of the candidate model, while simultaneously making
it more resistant to the catastrophic forgetting phenomena. In each iteration
$j$, where $j>1$, a deep network is fed with the training examples $x$ such
that $x=[x_{o},x_{n}]$ where $x_{o}$ denotes the training examples of the old
classes (learned in $1$ to $j-1$ iterations) and $x_{n}$ denotes the training
examples of the new classes (added in the $j^{th}$ iteration). The network
produces the output logits $l(x)$ such that $l(x)=w\times f(x)+b$, where $w$
denotes the network weights, $b$ represents the biasing factor and $f(x)$ is
the feature vector. The logits $l(x)$ are formed by
$l(x)=[l(x_{o}),l(x_{n})]$, where $l(x_{o})$ and $l(x_{n})$ represent the
logits produced through the $x_{o}$ and $x_{n}$, respectively. Typically, in
incremental learning systems, we divide these logits by a constant $\tau$
(i.e., $l^{\tau}(x)=\frac{l(x)}{\tau}$), so that they produce soft-target
probabilities when passed through the activation function [43]. Moreover, the
joint probability distribution between the scaled logits $l^{\tau}(x_{o})$ and
$l^{\tau}(x_{n})$ given the outcome $c_{i}$ is obtained through Bayes Rule as
expressed below:

$p(c_{i}|l^{\tau}(x_{o}),l^{\tau}(x_{n}))=\frac{p(l^{\tau}(x_{o}),l^{\tau}(x_{n})|c_{i})p(c_{i})}{\sum_{c_{k}=0}^{c_{o}-1}p(l^{\tau}(x_{o}),l^{\tau}(x_{n})|c_{k})p(c_{k})},$
(1)

where $p(c_{i})$ denotes the prior probability,
$p(l^{\tau}(x_{o}),l^{\tau}(x_{n})|c_{i})$ represents the likelihood for the
given class $c_{i}$ and $c_{o}$ represents the total number of previously
learned classes. It should be noted that for a given class $c_{i}$, the joint
distributions between $l(x_{o})$ and $l(x_{n})$ are symmetrical i.e.
$p(c_{i}|l^{\tau}(x_{o}),l^{\tau}(x_{n}))=p(c_{i}|l^{\tau}(x_{n}),l^{\tau}(x_{o}))$.
From the above, the mutual distillation loss is computed through:

$\begin{split}L_{md}=-\sum\limits_{h=0}^{c_{o}-1}t(x_{o,h})\log(p(c_{h}|l^{\tau}(x_{o}),l^{\tau}(x_{n})),\end{split}$
(2)

where $t(x_{o,h})$ represents the ground truth label (in one-hot notation) of
the training sample ($x_{o}$) belonging to the previously learned $h^{th}$
class.

###  III-F Continual Learning Loss Function

The $L_{md}$ bridges the gap between old and newly learned classes by
constraining the network to learn their mutual relations and dependencies. But
to make the network aware of their exclusive characteristics, we jointly
optimize $L_{o}$ and $L_{n}$ objective functions along with $L_{md}$. The
$L_{o}$ and $L_{n}$ are expressed as:

$L_{o}=-\sum\limits_{s=0}^{c_{o}-1}t(x_{o,s})\log(p(l^{\tau}(x_{o,s}))),$ (3)

and

$\begin{split}L_{n}=\sum\limits_{s=0}^{c_{n}-1}q(t(x_{n,s}))\log\left(\frac{q(t(x_{n,s}))}{p(l(x_{n,s}))}\right).\end{split}$
(4)

We can observe here that $L_{n}$ gives more emphasis on learning each of the
newly added category, in the current iteration, by computing the relative
entropy between true and predicted probability distributions. However, $L_{o}$
computes the total entropy between the true and predicted distributions
ensuring that the network does not forget the prior learned categories while
learning newly added class representations. Moreover, $t(x_{o,s})$ and
$t(x_{n,s})$ are the true $s^{th}$ class indicators for the training examples
of the older and newly added classes ($x_{o}$ and $x_{n}$), respectively,
$q(t(x_{n,s}))$ and $p(l(x_{n,s}))$ represents the actual and predicted
distributions generated from $t(x_{n,s})$ and $l(x_{n,s})$ for the $s^{th}$
class, respectively. Apart from this, $p(l^{\tau}(x_{o,s}))$ represents the
distribution of scaled logits $l^{\tau}(x_{o,s})$ (for the $s^{th}$ class)
generated through the training samples of the old classes $x_{o}$.

The continual learning loss functions $L_{cl}$ is, therefore, defined as a
linear combination of $L_{md}$, $L_{o}$ and $L_{n}$ as expressed below:

$L_{cl}=\alpha L_{n}+\beta L_{md}+\gamma L_{o},$ (5)

where $\alpha$, $\beta$ and $\gamma$ denote the loss weights. These weights
are empirically chosen to be 0.25, 0.45, and 0.30, respectively. Due to space
constraints, we refer the reader to the supplementary material to see the in-
depth ablative experimentation for determining these hyperparameters.

##  IV Experimental Setup

This section reports the detailed description of the datasets, the training
and implementation protocols, and the evaluation metrics, which we used to
evaluate the proposed framework.

###  IV-A Datasets

We have used six public datasets containing retinal fundus and OCT imagery.
These datasets includes the Rabbani [19] dataset, BIOMISA [44] dataset, Zhang
dataset [21], Duke-I dataset [45], Duke-II dataset [46], and the Duke-III
dataset [47]. The detailed description of these datasets are given below:

####  IV-A1 Rabbani

Rabbani dataset is one of the unique retinal image databases that contains a
total of 4,241 OCT and 148 fundus scans showing normal, ci-DME, nci-DME, and
dry AMD pathologies. Moreover, the dataset is acquired using Spectralis,
Heidelberg Inc. at Noor Eye Hospital, Tehran, Iran.

####  IV-A2 BIOMISA

BIOMISA dataset is another public dataset introduced by the Biomedical Image
and Signal Analysis (BIOMISA) Lab at the National University of Sciences and
Technology (NUST), Islamabad, Pakistan. BIOMISA dataset contains a total of
5,324 OCT and 115 fundus scans from 99 subjects either healthy or suffering
from ci-DME, nci-DME, dry AMD, wet AMD, acute and chronic CSR. Apart from
this, the scans within the BIOMISA dataset are acquired through Topcon 3D OCT
2000 in the Armed Forces Institute of Ophthalmology, Rawalpindi, Pakistan.

####  IV-A3 Zhang

Zhang dataset is one of the largest retinal OCT datasets containing 109,309
scans which shows wet AMD (choroidal neovascularization), dry AMD (drusen),
ci-DME, nci-DME, and healthy pathologies. The dataset has been arranged in a
way that 108,309 scans are to be used for training and 1,000 scans are to be
used for testing purposes. Moreover, the scans in the Zhang dataset are
acquired through Spectralis, Heidelberg Inc. Zhang dataset [21] also contains
CXRs to screen pneumonia subjects.

####  IV-A4 Duke-I

The Duke-I dataset is one of the largest OCT datasets from the Vision & Image
Processing (VIP) lab, Duke University, USA, containing 38,400 scans from
healthy (controlled) and dry AMD subjects. The scans are acquired through the
Bioptigen OCT machine. The dataset also contains ground truth annotations to
verify the performance of the automated systems towards extracting the retinal
layers.

####  IV-A5 Duke-II

The Duke-II dataset contains 610 OCT scans showing nci-DME and ci-DME
pathologies. The dataset was introduced in [46] by VIP lab, and it contains
detailed annotations for the retinal layers and fluids marked by two expert
clinicians. Moreover, the scans within Duke-II are acquired through
Spectralis, Heidelberg Inc. Also, in some experiments, we combined Duke-I and
Duke-II datasets to evaluate further the proposed framework towards screening
the retinal pathologies covered by both of these datasets.

####  IV-A6 Duke-III

Duke-III is the third dataset from the VIP lab that we used in this research.
The dataset contains 3,231 retinal OCT scans reflecting normal, ci-DME, nci-
DME, and dry AMD pathologies. The dataset is primarily designed for the
classification tasks, and the scans within the dataset are organized with
respect to the pathologies that they exhibit. Moreover, Duke-III dataset scans
are also acquired through Spectralis, Heidelberg Inc.

Apart from this, we also got all the OCT and fundus scans (contained within
these datasets) extensively annotated by the expert clinicians, where all of
these markings have been released publicly for the research community within
the source code repository222Source codes and dataset annotations are
available at https://github.com/taimurhassan/continual_learning/. Also, it
should be noted that the scans within each dataset were of different sizes.
But in order to make these sizes consistent with the input layer of the
classification networks, we have resized the scans (within each dataset) to a
standard resolution of $224\times 224\times 3$.

###  IV-B Training and Implementation Details

The major benefit of the proposed scheme compared to the conventional fine-
tuning approaches is its ability to utilize few training examples to learn the
diversified retinopathy classification tasks. Moreover, the complete training
of the proposed framework is performed in two stages. The first stage relates
to typical class-incremental learning [43], where the model is trained in each
increment to distinguish between different types of retinal pathologies from
the provided OCT examples. Unlike conventional fine-tuning systems, the
proposed framework does not learn all the classes (retinal diseases) at once
from the large well-annotated training data. Rather it learns them
incrementally (across each dataset) by utilizing a significantly lesser amount
of training examples. Incremental training in the first stage lasts for $k=6$
iterations, where the classification model periodically learns to screen $k+1$
retinal pathologies. Moreover, in the second stage, the model is incrementally
adapted to fundus imagery to recognize $m-k+1$ retinal diseases, where $m=11$.
The detailed description of training protocols for Stage-I and Stage-II are
presented next.

Table I: Retinal scans that are used for training the proposed framework in each incremental iteration. The abbreviations are ST: Stage, IT: Incremental Iteration, and PTS: Previous Training Subset. PTS contains the small portion of scans that were used in the previous iterations to aid the network in distilling its prior learned knowledge. ST | IT | Dataset | Training Scans
---|---|---|---
I | 1 | Rabbani [19] | 300 (100 ci-DME & 200 normal)
|  | BIOMISA [44] | 500 (200 ci-DME & 300 normal)
|  | Duke-II [46] | 100 (all ci-DME)
|  | Duke-III [47] | 350 (100 ci-DME & 250 normal)
|  | Zhang [21] | 4000 (2k ci-DME & 2k normal)
| 2 | Rabbani [19] | 150 (all nci-DME)
|  | BIOMISA [44] | 250 (all nci-DME)
|  | Duke-II [46] | 100 (all nci-DME)
|  | Duke-III [47] | 150 (all nci-DME)
|  | Zhang [21] | 2000 (all nci-DME)
|  | PTS | 525 (265 ci-DME & 260 normal)
| 3 | Rabbani [19] | 120 (all dry AMD)
|  | BIOMISA [44] | 120 (all dry AMD)
|  | Duke-I [45] | 150 (all dry AMD)
|  | Duke-III [47] | 100 (all dry AMD)
|  | Zhang [21] | 1000 (all dry AMD)
|  | PTS | 790 (525 old & 265 nci-DME)
| 4 | BIOMISA [44] | 200 (all wet AMD)
|  | Zhang [21] | 2500 (all wet AMD)
|  | PTS | 1062 (790 old & 272 dry AMD)
| 5 | BIOMISA [44] | 120 (all acute CSR)
|  | PTS | 1152 (1062 old & 90 wet AMD)
| 6 | BIOMISA [44] | 150 (all chronic CSR)
|  | PTS | 1252 (1152 old & 100 acute CSR)
II | 1 | Rabbani [19] | 40 (20 CSME & 20 normal)
|  | BIOMISA [44] | 20 (10 CSME & 10 normal)
|  | PTS | 1292 (1252 old, 20 CSME & 20 normal)
| 2 | Rabbani [19] | 10 (all non-CSME)
|  | BIOMISA [44] | 10 (all non-CSME)
|  | PTS | 1302 (1292 old, 5 CSME & 5 normal)
| 3 | Rabbani [19] | 7 (all dry AMD)
|  | BIOMISA [44] | 20 (all dry AMD)
|  | PTS | 1322 (1302 old, 20 non-CSME)
| 4 | BIOMISA [44] | 5 (all wet AMD)
|  | PTS | 1337 (1302 old, 10 dry AMD)
| 5 | BIOMISA [44] | 10 (all CSR)
|  | PTS | 1342 (1337 old, 5 wet AMD)
Table II: Training and testing scans utilized for evaluating the proposed framework on each individual dataset. The column ’Training’ represents the number of scans which we used for few-shot training, whereas the column ’AT’ represents the actual number of scans that are to be used for training purposes as per the dataset standard.  Dataset | Scans | Scanner | Training | Testing | AT
---|---|---|---|---|---
Rabbani | OCT | Spectralis | 570 | 3671 | 1061
| Fundus |  | 57 | 91 | 37
BIOMISA | OCT | Topcon | 1340 | 3984 | 3840
| Fundus | OCT 2000 | 65 | 50 | 90
Duke-I | OCT | Bioptigen | 150 | 38250 | 300
Duke-II | OCT | Spectralis | 200 | 410 | 305
Duke-III | OCT | Spectralis | 600 | 2631 | 3048
Zhang | OCT | Spectralis | 9500 | 99809 | 108309

####  IV-B1 Stage-I Training Details

In the first stage, we pick the pre-trained model, update its last layer to
recognize ci-DME and healthy pathologies (in the first iteration). Here, we
feed the network with 5,250 training examples where 300 are taken from the
Rabbani dataset, 500 are taken from the BIOMISA dataset, 100 examples are
taken from Duke-II, 350 examples are taken from the Duke-III dataset, and
4,000 examples are taken from Zhang dataset. After learning these pathologies,
we stack a new ’nci-DME’ class for which we feed 150 examples from Rabbani,
250 examples from BIOMISA, 100 examples from Duke-II, 150 examples from Duke-
III, and 2,000 examples for the Zhang dataset. Furthermore, we also feed the
network with a small subset of 550 examples from the previous training batch
to retain the previously learned classes. In the third iteration, we make the
network learn the dry AMD-affected pathologies for which we feed 1,470
training examples from Rabbani, BIOMISA, Duke-I, Duke-III, and Zhang dataset.
In the fourth iteration, we trained the network to identify wet AMD using
2,720 training examples from Rabbani, BIOMISA, and Zhang dataset. In the fifth
and sixth iteration, we trained the classification model to identify acute and
chronic CSR from the BIOMISA dataset using 120 and 150 training examples,
respectively. The reason for using only the BIOMISA dataset for identifying
CSR is because this is the only public dataset that contains annotated CSR
pathologies [44].

####  IV-B2 Stage-II Training Details

After learning normal and abnormal retinal pathologies from the OCT scans, we
adapt the classification model to learn about the retinal pathological
variations from the fundus scans incrementally. Here, in the first iteration,
we stack two classes representing healthy and CSME affected fundus, and the
network learns these categories from a total of 40 training fundus scans from
the Rabbani and BIOMISA datasets. Moreover, in the next incremental
iterations, the proposed framework learns to screen non-CSME, dry AMD, wet
AMD, and CSR pathologies using 20, 27, 5, and 10 training examples,
respectively, as shown in Table I.

####  IV-B3 Incremental Training on Individual Datasets

In order to compare the performance of the proposed framework with state-of-
the-art retinal diagnostic systems, we also evaluated the proposed system on
each dataset separately. The number of training iterations varies on each
dataset depending upon the number of retinal pathologies and the modalities it
contains. However, the number of training scans for each dataset is presented
in Table II. Here, we highlight that for each dataset, the proposed framework
only utilizes a small subset of training data (which is defined by the dataset
standard) to learn the retinal pathologies, and it also produces competitive
performance against the state-of-the-art methods, which are trained in a
conventional manner using the complete training data (as defined by each
dataset protocol).

####  IV-B4 Implementation Protocols

The proposed scheme is implemented using TensorFlow 2.1.0 with Keras 2.3.0 on
the Anaconda platform with Python 3.7.9. Some of the utility functions have
also been written in MATLAB R2020a.

Apart from this, each incremental training iteration lasts for five epochs
(where the number of cycles in each epoch varies for each dataset). Moreover,
the optimizer used during the training was ADADELTA [48], and the training was
conducted on a machine with Core<EMAIL_ADDRESS>GHz, 32 GB DDR4 RAM, and NVIDIA
RTX 2080 Max-Q GPU. The source code of the proposed framework and its complete
documentation is released publicly on GitHub2.

###  IV-C Evaluation Metrics

Standard classification metrics, such as accuracy (Acc), true positive rate
(TPR), true negative rate (TNR), positive predicted value (PPV), and F1 score,
have been utilized to measure the performance of the proposed framework.
Moreover, we also used the ROC curve and AUC to assess the behavior of the
proposed framework towards screening the retinal pathologies acquired with
specific types of scanners. Apart from this, to measure the performance
difference between the proposed framework and the state-of-the-art works, we
used the relative percentage formula, i.e., $\frac{l_{e}-l_{g}}{l_{g}}\times
100$, where $l_{e}$ represents the leading score, and $l_{g}$ denotes the
lagging score.

##  V Results

In this section, we present a detailed evaluation of the proposed framework
for screening retinopathy using both fundus and OCT imagery. In the first set
of experiments, we compared the proposed framework’s performance with the
state-of-the-art retinal diagnostic systems on different publicly available
datasets. Afterward, in the second set of experiments, we compared the
proposed framework’s performance with state-of-the-art domain adaptation and
incremental domain adaptation schemes for performing the cross-domain
retinopathy screening tasks. Before discussing these experiments, we present
comprehensive ablation studies to determine the proposed system’s
hyperparameters and the optimal classification network. Apart from this, we
give a detailed description of the number of scans which we used for training
and testing purposes in Table I and II.

###  V-A Ablation Studies

The goal of the ablative analysis is to: 1) analyze the effect of temperature
in each iteration for computing the soft probabilities and 2) to determine the
optimal classification network. We also report additional ablation experiments
in the supplementary material through which we determined the number of
learnable parameters of each network and the hyperparameters of the $L_{cl}$
loss function.

####  V-A1 Determining the Temperature Constant

The temperature constant ($\tau$) is a hyperparameter that generates soft
target probabilities for each class, thus, enabling the deep neural networks
to learn these classes accurately during the knowledge distillation process.
$\tau$ is a dataset-dependent parameter, and its optimal value varies
significantly across different domains. In this paper, we empirically
determined the best value of $\tau$ for each dataset by analyzing the Top-1
classification error as shown in Figure 2. Here, the best value of $\tau$
differs for each dataset and each pre-trained model, but it typically ranges
between $1<\tau<2.5$. For instance, on Rabbani dataset, the best
classification performance for ResNet-101 [49] was achieved with $\tau=$ 1.3.
On the BIOMISA dataset, the optimal $\tau$ value is 1.25. Similarly, the
optimal $\tau$ for the Zhang dataset and for the incremental cross-domain
adaptation is 1.4 and 2, respectively (as shown in Figure 2). In addition to
this, we also analyzed the performance of each network by varying $\tau$ on
fixed intervals (across each dataset), and here we also found ResNet-101 to be
the best classification model. More details on the effect of $\tau$ at fixed
intervals are reported in the paper’s supplementary material.

Figure 2: Effect of $\tau$ in terms of Top-1 error on: First row (left to right) Rabbani dataset, and the BIOMISA dataset. Second row (left to right): Duke-I+II dataset, and Duke-III dataset. Third row (left to right), Zhang dataset, and the proposed incremental cross-domain adaptation. Table III: Comparison of different pre-trained models (incrementally trained using the proposed $L_{cl}$ function) to perform cross-domain retinopathy screening tasks. Bold indicates the best scores while the second-best scores are in blue color. Metric | MobileNet | ResNet-50 | ResNet-101 | VGG-16
---|---|---|---|---
Acc | 0.9618 | 0.9678 | 0.9826 | 0.9532
TPR | 0.9490 | 0.9537 | 0.9757 | 0.9299
TNR | 0.9788 | 0.9864 | 0.9918 | 0.9839
PPV | 0.9834 | 0.9893 | 0.9937 | 0.9870
F1 | 0.9659 | 0.9712 | 0.9846 | 0.9576

####  V-A2 Choice of the Classification Model

This ablation study is related to determining the optimal classification
network to perform the cross-domain retinopathy screening tasks. For this
purpose, we utilize models such as MobileNet [50], ResNet-50 [49], ResNet-101
[49], and VGG-16 [25]. The comparison is reported in Table III, where we can
see that the best performance is achieved for ResNet-101. Moreover, the
second-best performance is achieved by ResNet-50 across all the metrics. This
is due to the fact that ResNets employ a residual feature fusion mechanism
within the encoder part that allows the network to retain finer feature
representations of the candidate scan during latent vector generation [49].
However, in terms of computational speeds, ResNets might not be the optimal
choice as they are not built to be lightweight like MobileNets. But since our
prime objective in this study is to achieve better performance, we chose
ResNet-101 in the rest of the experimentations.

###  V-B Comparison with the Conventional Methods

In the first series of experiments, we measured the classification performance
of the incremental ResNet-101 (trained using the proposed $L_{cl}$ loss
function) on Rabbani, BIOMISA, Duke-III, and Zhang datasets. We used only
these datasets here because 1) they are specifically designed for the retinal
classification tasks, and 2) these datasets contain detailed scan-level
clinicians’ markings, which researchers have extensively used as ground truths
for evaluating their methods [13, 51, 52]. Furthermore, to make the comparison
fair, we only diagnosed the originally marked pathologies (within each
dataset) for this experiment. For example, the proposed framework has been
trained to extract normal, DME, CNV, and drusen pathologies from the OCT scans
of the Zhang dataset. Similarly, the proposed framework extracts the normal,
DME, and AMD pathologies from Rabbani and Duke-III datasets. Moreover, on the
BIOMISA dataset, the proposed framework has been incrementally trained to
extract the normal, ME, dry AMD, wet AMD, and CSR pathologies from the OCT
scans. Here, we also excluded the evaluation on fundus scans because, to the
best of our knowledge, there is no competitive framework that uses fundus
scans from Rabbani and BIOMISA datasets to screen retinopathy.

The comparison of the proposed framework with state-of-the-art methods is
reported in Table IV. Here, we can see that despite its incremental nature,
the proposed framework outperforms its competitors by 1.86% on Zhang dataset
in terms of accuracy, and 1.65% on Duke-III dataset, and 0.10% on BIOMISA
datasets in terms of F1 score. In addition to this, the proposed framework is
outperforming LACNN [13] on the Zhang dataset by 14.23% in terms of the F1
score, which is quite a noticeable improvement. Although, the STGS approach
(proposed in [52]) is outperforming the proposed framework by 0.308% and 2.98%
in terms of Acc and TPR, respectively. However, we also wanted to highlight
here that STGS is a conventional machine learning approach that has been
trained on 3,840 scans. But the proposed framework, when trained on 1,340
scans, is able to produce competitive performance with STGS on the BIOMISA
dataset (e.g., see the performance comparison of the proposed framework and
STGS in terms of PPV and F1 scores within Table IV). Furthermore, since the
STGS is a conventional machine learning framework, it lacks generalizability
to process OCT scans acquired with different scanners simultaneously [52].

Table IV: Performance comparison of the proposed framework (backboned through ResNet-101) with state-of-the-art retinopathy screening frameworks. To maintain fairness with the competitors, the proposed framework is incrementally trained only to identify the originally marked pathologies within each dataset. Moreover, bold indicates the best performance, whereas the second-best score is underlined. ’-’ indicates that the following metric is not computed by the other frameworks. The rest of the abbreviations are DT: Dataset, Met: Metric, PF: Proposed Framework, CBRF: CNN-Based Referral Framework [21], HSVM: SVM driven by HOG Descriptor [47], AOCT: AOCT-Net [51], LACNN: Lesion-Aware CNN, STGS: Structure Tensor Graph Search [52], ZD: Zhang Dataset, RA: Rabbani, BO: BIOMISA, D3: Duke-III dataset. DT | Met | PF | CBRF | HSVM | AOCT | LACNN | STGS
---|---|---|---|---|---|---|---
ZD | Acc | 0.984 | 0.966 | - | 0.964 | 0.901 | -
| TPR | 0.981 | 0.978 | - | 0.962 | 0.868 | -
| TNR | 0.984 | 0.974 | - | 0.968 | - | -
| PPV | 0.994 | 0.994 | - | 0.989 | 0.862 | -
| F1 | 0.987 | 0.986 | - | 0.975 | 0.864 | -
RA | Acc | 0.972 | - | - | - | - | -
| TPR | 0.968 | - | - | - | 0.993 | -
| TNR | 0.980 | - | - | - | - | -
| PPV | 0.988 | - | - | - | 0.993 | -
| F1 | 0.977 | - | - | - | 0.993 | -
D3 | Acc | 0.980 | - | 0.955 | - | - | -
| TPR | 1.000 | - | 1.000 | - | - | -
| TNR | 0.933 | - | 0.866 | - | - | -
| PPV | 0.967 | - | 0.937 | - | - | -
| F1 | 0.983 | - | 0.967 | - | - | -
BO | Acc | 0.974 | - | - | - | - | 0.977
| TPR | 0.971 | - | - | - | - | 1.000
| TNR | 0.996 | - | - | - | - | 0.933
| PPV | 0.999 | - | - | - | - | 0.967
| F1 | 0.984 | - | - | - | - | 0.983

###  V-C Incremental Cross-Domain Retinopathy Screening

In this series of experiments, we measured the capacity of the $L_{cl}$ to
constrain the ResNet-101 in learning the retinopathy classification tasks
simultaneously from the OCT and fundus imagery. The training details for the
proposed incremental cross-domain adaptation are already discussed in Section
IV-B. Moreover, its evaluation, in terms of classification accuracy, is
reported in Figure 3 and Table V. In Figure 3, we can observe that as we
continue learning retinal classification tasks, the $L_{cl}$ enables the
ResNet-101 model to produce high resistance to catastrophic forgetting as
compared to its competitors, such as DMC [53], RWalk [54], EWC [55] and iCaRL
[39]. For example, when $r=13$, the proposed framework achieves 1.03%
performance improvement compared to the second-best DMC-based ResNet-101 in
terms of accuracy. These improvements stem from the fact that $L_{cl}$ not
only constrains the classification models to retain their prior knowledge
through distillation but also ensures that the classification models learn the
structural and semantic relationships between incrementally learned
representations. Table V showcases the performance comparison of the $L_{cl}$
driven ResNet-101 over its competitors for different domain adaptation
configurations. Here, we can see that when the incremental ResNet-101 (trained
using $L_{cl}$) is incrementally adapted from OCT to fundus imagery or from
fundus to OCT imagery, it produces 1.03% and 3.22% better classification
performance than the second-best DMC loss function, respectively towards
recognizing different multi-modal retinal abnormalities. In Table V, we can
also notice that the proposed framework achieves 2.08% improvement when
adapting from OCT to fundus domain as compared to the other configuration.
This is because the training images for the OCT domain are relatively larger
in nature than the fundus domain (see Table I, II), and training the
underlying network on OCT images first gives it more experience of screening
the retinal pathologies, which leads to better classification performance at
the inference stage.

Apart from this, the classification performance of the proposed scheme for
screening retinopathy from multiple modalities is also shown in the confusion
matrix (Figure 5). We can notice here that regardless of the diversity within
the scanner specifications, the incremental $L_{cl}$-driven ResNet-101
achieved an accuracy of 0.9826 and an F1 score of 0.9846.

We further analyzed the capacity of the proposed framework (in terms of ROC
curves) for discriminating scanner-specific retinal pathologies. For this
experiment, we first categorized the fundus, and OCT scans from all the
datasets w.r.t the machine manufacturers and then classified the pathologies
from these scans using the proposed system. For example, for the curve
representing ’Spectralis vs. Others’, we have grouped the OCT and fundus scans
from the Zhang, Rabbani, Duke-II, and Duke-III datasets (which are acquired
through Spectralis, Heidelberg Inc. scanner). Similarly, we have also grouped
the rest of the scans in the other datasets as ’Others’. Then, we checked
whether or not the proposed framework correctly identified the retinal
pathologies (within the Spectralis scans). The correct predictions are marked
as 1’s, whereas the misclassifications are marked as 0’s. Moreover, the
correct predictions from other scanners were marked as 0’s, and the
misclassifications are marked as 1’s. After labeling the predictions, we used
their confidence scores and the ground truth labels to generate the
’Spectralis vs. Others’ curve. The same process is repeated for generating the
curves of other scanners as well.

The ROC curves are demonstrated in Figure 4. Here, we can see that the
proposed framework’s classification performance, in terms of AUC scores,
remains similar across different types of scanners. This evidences that the
proposed framework remains unbiased towards screening retinal pathologies from
any specific type of scanner during incremental cross-domain adaptation.

Figure 3: Performance comparison of the incremental ResNet-101 [49] (driven
through proposed $L_{cl}$) with iCaRL [39], RWalk [54], EWC [55], and DMC [53]
for cross-domain retinopathy screening.

In another series of experiments, we compared the cross-domain adaptation
capacity of the proposed framework with the state-of-the-art schemes such as
Fourier Domain Adaptation (FDA) [40], Incremental Evolving Domain Adaptation
(EDA) [56], and Incremental Domain Adaptation for Neural Networks (iDANN)
[57]. The comparison is reported in Table VI, where we can see that the
proposed $L_{cl}$ driven ResNet-101 leads the second-best iDANN scheme by
2.08% and 4.34%, respectively, across both domain adaptation configurations.
We also highlight that FDA is originally designed as a domain adaptation
scheme for semantic segmentation tasks [40]. However, to fairly compare it
with the proposed framework, we utilized it for classification purposes, i.e.,
we first stylized the training scans using FDA and then used them to
incrementally train the ResNet-101 via the proposed $L_{cl}$ loss function.
After training the model, we evaluated it for retinopathy screening using the
FDA stylized OCT and fundus test scans. Here, the stylization, through FDA, is
meant to overcome the scanner differences, which should aid the underlying
network towards accurately learning the cross-modal classification tasks. But
from Table VI, we can see that the classification performance of the
incrementally trained ResNet-101 on the FDA stylized scans is not the best.
The low performance of the FDA scheme stems from the fact that it uses sharp
cut-off transitions of the parametric-rectangular window for stylization [40],
which also introduces a large number of noisy transitions within the
transformed scans. Also, it should be noted that the FDA scheme is primarily
designed for the inter-related semantic segmentation tasks only [40].

Apart from this, we can notice that the classification performance of the
proposed scheme and the iDANN is comparable for both configurations (see Table
VI). iDANN incrementally learns the pool of experiences from the scarce (yet
inter-related) domains via gradient reversal strategy [57], where the
underlying classification model is trained to recognize the domain-invariant
features from the subset of target domain training samples (chosen in an
unsupervised manner). This eventually achieves higher generalizability for
performing cross-domain tasks simultaneously [57]. However, while adapting
from Target Domain-II to Target-Domain-I (i.e., from the fundus to OCT
imagery), the performance of iDANN deteriorates at a higher rate than the
$L_{cl}$ driven incremental cross-domain adaptation scheme (as evident from
Table VI). This is because $L_{cl}$ analyzes the structural and semantic
similarities, via Bayesian inference, between incrementally learned knowledge
representations, while iDANN only focuses on the gradients minimization
(during the back-propagation) based upon the spatial feature representations
[57].

Figure 4: ROC curves showcasing the performance of the incremental ResNet-101 [49] model (trained using $L_{cl}$ loss function) for screening the scanner-specific retinal pathologies. Table V: Performance comparison of $L_{cl}$ loss function with state-of-the-art incremental learning schemes (in terms of accuracy) for performing the cross-domain retinopathy screening tasks. The source represents the ImageNet dataset [41]. Target-I represents the OCT imagery taken from Rabbani, BIOMISA, Zhang, Duke-I, Duke-II, and Duke-III datasets. Target-II represents the fundus imagery taken from Rabbani and BIOMISA dataset. Moreover, the performance comparison is measured using the ResNet-101 model. The best performance is in bold, while the second-best performance is in blue color. The rest of the abbreviations are iC: iCaRL [39], EC: EWC [55], RW: RWalk [54], and DC: DMC [53]. Configuration | $L_{cl}$ | iC | EC | RW | DC
---|---|---|---|---|---
Source: Target-I $\rightarrow$ Target-II | 0.98 | 0.66 | 0.80 | 0.85 | 0.97
Source: Target-II $\rightarrow$ Target-I | 0.96 | 0.68 | 0.81 | 0.83 | 0.93
Table VI: Performance comparison of the proposed framework with state-of-the-art incremental domain adaptation approaches in terms of classification accuracy. For a fair comparison, we used ResNet-101 in all the schemes for classification purposes. Bold indicates the best performance while the second-best performance is in blue color. Moreover, the abbreviations are SD: Source Domain (ImageNet [41]), T-I: Target Domain-I, and T-II: Target Domain-II. Configuration | Proposed | FDA [40] | EDA [56] | iDANN [57]
---|---|---|---|---
SD: T-I $\rightarrow$ T-II | 0.98 | 0.86 | 0.89 | 0.96
SD: T-II $\rightarrow$ T-I | 0.96 | 0.81 | 0.84 | 0.92
Figure 5: Confusion matrix depicting the performance of the incremental
ResNet-101 (trained via proposed $L_{cl}$ loss function) to perform cross-
domain retinopathy screening tasks.

##  VI Discussion

After rigorously evaluating the proposed framework on the six public datasets
against different experimental settings, the proposed system showed an overall
superiority over state-of-the-art retinal diagnostic frameworks. In addition
to this, the results, reported in Tables V, VI and Figure 3 evidences a neat
improvement of the proposed framework over state-of-the-art incremental
learning, domain adaptation, and incremental domain adaptation schemes for
performing the cross-domain retinopathy screening tasks. Furthermore, we have
thoroughly tested the applicability of the proposed framework for real-time
retinopathy screening through blind testing experiments in clinical settings.
Due to space constraints, we reported these experiments within the
supplementary material of the paper. Apart from this, our framework’s core
element is the $L_{cl}$ loss function, in which the newly proposed mutual
distillation loss $L_{md}$ plays a capital role. The importance of $L_{md}$
objective in $L_{cl}$ can also be seen in Table VII (for all the datasets)
where the $L_{md}$ objective function, driven through Bayesian inference,
significantly aids in learning the new classification categories while
simultaneously performing well on the already known tasks.

Table VII: Performance comparison of $L_{cl}$ (with and without $L_{md}$) on different datasets. Bold indicates the best performance. Moreover, the abbreviations are RA: Rabbani dataset, BO: BIOMISA dataset, DI: Duke I+II dataset, D3: Duke-III dataset, ZD: Zhang dataset, ICDA: Incremental Cross-Domain Adaptation. Loss Functions | RA | BO | DI | D3 | ZD | ICDA
---|---|---|---|---|---|---
$L_{cl}$ with $L_{md}$ | 0.972 | 0.974 | 0.963 | 0.980 | 0.984 | 0.982
$L_{cl}$ w/o $L_{md}$ | 0.856 | 0.859 | 0.847 | 0.894 | 0.886 | 0.764

We also want to highlight here that although $L_{cl}$ is robust in avoiding
the catastrophic forgetting during the incremental training (as evidenced from
Section V), we did notice its vulnerability towards screening retinopathy when
the classes are imbalanced. For example, in Figure 5, we can see that the
proposed framework (trained with $L_{cl}$) correctly screened 36,549 out of
37,022 dry AMD pathologies, resulting in the classification score of 0.9872.
However, in fundus scans, 11 dry AMD cases were classified correctly out of
29. This significant difference in the classification performance is due to
the imbalanced number of the fundus and OCT scans on which network was
trained, i.e., the quantity of fundus scans is very small compared to OCT
scans (see Table I and II for more details). Therefore, the classification
model within the proposed framework is more biased towards accurately
screening the retinal pathologies from the OCT scans. Also, notice the
differences in the attention maps between OCT and fundus scans (within Figure
6). These attention maps show the abnormal regions which the network focuses
on while predicting the underlying pathology, and by observing the pairs in
Figure 6 (A-B), (C-D), (E-F), (G-H), (I-J), (M-N), (O-P), we can see that the
proposed system correctly recognizes the retinal lesions such as choroidal
neovascularization, sub-retinal fluid, intra-retinal fluid, hard exudates,
drusen from the dry and wet AMD, acute and chronic CSR, ci-DME and nci-DME
pathologies. However, if we look at the pairs (K-L) and (Q-R) in Figure 6, we
can note that the attention of the proposed system is more focused on the
optic-disc region rather than the retinal lesions. This biasness (in the
attention maps) is due to the highly imbalanced ratio of the OCT and fundus
scans in the datasets (see Table II). But it should be noted at the same time
that, clinically, the bias of the proposed framework towards OCT imagery is
also very significant because the OCT imagery can objectively show the
symptomatic appearance of the retinal abnormalities in early stages as
compared to the fundus scans [8], which leads towards timely diagnosis for the
prevention of the candidate’s vision loss. Nevertheless, we can overcome these
imbalanced cases by introducing some class imbalance remedies using max-margin
constraints [58], focal loss function [59], and Gaussian affinity
optimizations [60]. Apart from this, currently, we do not use any denoising
method to enhance the quality of the scans. The reasons for not using this
preprocessing step are: 1) the proposed system employs CNN backbones to
generate latent vectors which implicitly remove the high-frequency (noisy)
content during scan decomposition, and the denoising step is generally
required for the segmentation frameworks as they also need to reconstruct the
segmented pixel-level maps back where noisy pixels in the input scan may lead
to large pixel-level false positives [5]. 2) The classification performance of
the proposed framework in terms of accuracy and F1 scores is already very
reasonable, i.e., 0.9826 and 0.9846, respectively. Nevertheless, we also agree
that adding a denoising step (involving wiener filtering [16] or structure
tensors [26]) would further increase the classification performance. But since
the performance of the proposed framework is already decent, we deemed this as
an optional step that can be opted in the future.

Figure 6: Attention maps obtained from the incremental ResNet-101 (trained via
proposed $L_{cl}$ loss function) when performing the cross-domain retinopathy
screening tasks.

##  VII Conclusion

This paper presents a novel incremental cross-domain adaptation approach that
enables the classification networks to retain their prior knowledge while
learning new cross-modality retinopathy screening tasks incrementally via few-
shot training. The proposed scheme offers a highly scalable option for the
clinical settings, aiding the ophthalmologists in screening the vast majority
of retinal abnormalities (even the rarely occurring ones) with few training
examples. The extent of the proposed scheme has been tested with different
pre-trained models on six publicly available datasets across two domains,
where it outperforms state-of-the-art incremental learning and domain
adaptation schemes for identifying thirteen types of retinal pathologies
simultaneously. The backbone of the proposed scheme is the $L_{cl}$ loss
function which constrains the classification networks, via Bayesian inference,
to gain higher generalizability for performing the diversified cross-domain
tasks. Furthermore, due to the promising performance of the proposed scheme
for screening retinal diseases across multi-modal imagery, we envisage testing
its applicability on other medical applications in the future. Moreover, in
the future, we also anticipate overcoming the vulnerability of $L_{cl}$ loss
function against imbalanced classes by introducing the max-margin constraints
and modulating factors that will boost the capacity of the classification
models (trained on $L_{cl}$) to predict the imbalanced classes accurately.

## Acknowledgement

This work is supported by a research fund from Khalifa University: Ref:
CIRA-2019-047, and the Abu Dhabi Department of Education and Knowledge (ADEK),
Ref: AARE19-156. We would also like to thank all the ophthalmologists from the
Armed Forces Institute of Ophthalmology (AFIO), Rawalpindi, Pakistan, for
thoroughly annotating the retinal scans in all six datasets, and aiding us in
the blind testing process.

## References

  * [1] A. Guo, L. Fang, M. Qi, and S. Li, “Unsupervised Denoising of Optical Coherence Tomography Images With Nonlocal-Generative Adversarial Network,”  IEEE Transactions on Instrumentation and Measurement, Vol 7, pp.1-12, 2020\.
  * [2] B. Hassan, S. Qin, T. Hassan, M. U. Akram, R. Ahmed, and N. Werghi, “CDC-Net: Cascaded decoupled convolutional network for lesion-assisted detection and grading of retinopathy using optical coherence tomography (OCT) scans,” Biomedical Signal Processing and Control, Vol. 70, August, 2021.
  * [3] “Diabetic Macular Edema,”  EyeWiki, Accessed: January 20th, 2020.
  * [4] T. Hassan, M. U. Akram, and N. Werghi, “Exploiting the Transferability of Deep Learning Systems Across Multi-modal Retinal Scans for Extracting Retinopathy Lesions,”  20th IEEE International Conference on BioInformatics And BioEngineering (BIBE), December 2020.
  * [5] T. Hassan, M. U. Akram, N. Werghi, and N. Nazir, “RAG-FW: A hybrid convolutional framework for the automated extraction of retinal lesions and lesion-influenced grading of human retinal pathology,”  IEEE Journal of Biomedical and Health Informatics, March 2020.
  * [6] F. Oloumi, R. M. Rangayyan, and A. L. Ells, “Parabolic modeling of the major temporal arcade in retinal fundus images,”  IEEE Transactions on Instrumentation and Measurement, Vol. 61, pp. 1825-1838, 2012.
  * [7] H. Sun, Y. Zhang, P. Chen, H. Wang, Z. Guo, Y.-H. He, and R. Liang, “Synchronous fingerprint acquisition system based on total internal reflection and optical coherence tomography,”  IEEE Transactions on Instrumentation and Measurement, Vol. 69, pp. 8452-8465, 2020.
  * [8] T. Hassan, M. U. Akram, B. Hassan, A. Nasim, and S. A. Bazaz, “Review of OCT and fundus images for detection of Macular Edema,”  IEEE 12th International Conference on Imaging Systems and Techniques, September 2015.
  * [9] M. Karmel, “Retinal Imaging: Choosing the Right Method,”  EyeNet, July 2014.
  * [10] T. Hassan, M. U. Akram, M. F. Masood, and U. Yasin, “Deep structure tensor graph search framework for automated extraction and characterization of retinal layers and fluid pathology in retinal SD-OCT scans,”  Computers in Biology and Medicine, Volume 105, pp. 112-124, February 2019\.
  * [11] B. Hassan, S. Qin, T. Hassan, R. Ahmed, and N. Werghi, “Joint Segmentation and Quantification of Chorioretinal Biomarkers in Optical Coherence Tomography Scans: A Deep Learning Approach,”  IEEE Transactions on Instrumentation and Measurement, May 2021.
  * [12] T. K. Yoo, J. Y. Choi, J. G. Seo, B. Ramasubramanian, S. Selvaperumal, and D. W. Kim, “The possibility of the combination of OCT and fundus images for improving the diagnostic accuracy of deep learning for age-related macular degeneration: a preliminary experiment,”  Medical & Biological Engineering & Computing, Volume 57, pp. 677–687, March 2019.
  * [13] L. Fang, C. Wang, S. Li, H. Rabbani, X. Chen, and Z. Liu, “Attention to Lesion: Lesion-Aware Convolutional Neural Network for Retinal Optical Coherence Tomography Image Classification,”  IEEE Transactions on Medical Imaging, August 2019.
  * [14] M. Sirshar, T. Hassan, M. U. Akram, and S. A. Khan, “An incremental learning approach to automatically recognize pulmonary diseases from the multi-vendor chest radiographs,” Computers in Biology and Medicine, Vol. 134, 2021.
  * [15] B. Hassan, T. Hassan, B. Li, R. Ahmed, and O. Hassan, “Deep ensemble learning based objective grading of macular edema by extracting clinically significant findings from fused retinal imaging modalities,”  MDPI Sensors, July 2019.
  * [16] A. M. Syed, T. Hassan, M. U. Akram, S. Naz, and S. Khalid, “Automated diagnosis of macular edema and central serous retinopathy through robust reconstruction of 3D retinal surfaces,”  Elsevier Computer Methods and Programs in Biomedicine, 2016.
  * [17] B. Hassan, G. Raja, T. Hassan, and M. U. Akram, “Structure tensor based automated detection of macular edema and central serous retinopathy using optical coherence tomography images,”  Journal of the Optical Society of America A, Vol. 33, Issue 4, pp. 455-463, 2016.
  * [18] A. V. Varadarajan, P. Bavishi, P. Ruamviboonsuk, et al., “Predicting optical coherence tomography-derived diabetic macular edema grades from fundus photographs using deep learning,”  Nature Communications, Vol. 11, No. 130, 2020.
  * [19] R. Rasti, H. Rabbani, A. Mehri, and F. Hajizadeh, “Macular OCT Classification using a Multi-Scale Convolutional Neural Network Ensemble,”  IEEE Transactions on Medical Imaging, vol. 37, no. 4, pp. 1024-1034, 2018\.
  * [20] B. Hassan, S. Qin, R. Ahmed, T. Hassan, A. H. Taguri, S. Hashmi, and N. Werghi, “Deep learning based joint segmentation and characterization of multi-class retinal fluid lesions on OCT scans for clinical use in anti-VEGF therapy,” Computers in Biology and Medicine, August, 2021.
  * [21] D. Kermany et al., “Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning,”  Elsevier Cell, 2018.
  * [22] Y. Rong, D. Xiang, W. Zhu, K. Yu, F. Shi, Z. Fan, and X. Chen, “Surrogate-Assisted Retinal OCT Image Classification Based on Convolutional Neural Networks,”  IEEE Journal of Biomedical and Health Informatics, Vol 23, No. 1, January 2019.
  * [23] C. S. Lee, D. M. Baughman, and A. Y. Lee, “Deep Learning Is Effective for Classifying Normal versus Age-Related Macular Degeneration OCT Images,”  Ophthalmology Retina, Vol. 1, Issue 4, pp. 322-327, July–August 2017\.
  * [24] F. Arcadu, F. Benmansour, A. Maunz, J. Willis, Z. Haskova, and M. Prunotto, “Deep learning algorithm predicts diabetic retinopathy progression in individual patients,”  Nature Digital Medicine, Vol. 2, No. 92, 2019.
  * [25] K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,”  arXiv:1409.1556, 2014.
  * [26] H. Raja, T. Hassan, M. U. Akram, and N. Werghi, “Clinically Verified Hybrid Deep Learning System for Retinal Ganglion Cells Aware Grading of Glaucomatous Progression,”  IEEE Transactions on Biomedical Engineering, October 2020.
  * [27] Q. Wei, X. Li, W. Yu, X. Zhang, Y. Zhang, B. Hu, B. Mo, D. Gong, N. Chen, D. Ding, and Y. Chen, “Learn to Segment Retinal Lesions and Beyond,”  IEEE International Conference on Pattern Recognition (ICPR), 2020.
  * [28] D. J. Fu, L. Faes, S. K. Wagner, G. Moraes, R. Chopra, P. J. Patel, K. Balaskas, T. D. Keenan, L. M. Bachmann, and P. A. Keane, “Predicting incremental and future visual change in neovascular age-related macular degeneration using deep learning,”  Ophthalmology Retina, January 2021.
  * [29] X. Wang, L. Ju, X. Zhao, and Z. Ge, “Retinal Abnormalities Recognition Using Regional Multitask Learning,”  International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), October 2019.
  * [30] A. Hasan, Z. Deng, J. Loo, D. Mukherjee, J. L. Duncan, D. G. Birch, G. J. Jaffe, and S. Farsiu, “Meta-learning approach to automatically register multivendor retinal images,”  ARVO Annual Meeting Abstract, Investigative Ophthalmology & Visual Science, Vol. 61, 1634, June 2020.
  * [31] D. Mahapatra and Z. Ge, “Training data independent image registration using generative adversarial networks and domain adaptation,”  Pattern Recognition, Volume 100, 107109, ISSN 0031-3203, 2020.
  * [32] Q. Meng and S. Shin’ichi, “Adinet: Attribute driven incremental network for retinal image classification,”  IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [33] Y. He, A. Caras, Y. Liu, S. Saidha, P. A. Calabresi, and J. L. Prince, “Adversarial domain adaptation for multi-device retinal OCT segmentation,”  SPIE Medical Imaging, March 2020.
  * [34] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,”  IEEE International Conference on Learning Representations (ICLR), 2017\.
  * [35] L. Ju, X. Wang, Q. Zhou, H. Zhu, M. Harandi, P. Bonnington, T. Drummond, and Z. Ge, “Bridge the Domain Gap Between Ultra-wide-field and Traditional Fundus Images via Adversarial Domain Adaptation,”  arXiv:2003.10042, March 2020.
  * [36] R. Kaye, S. Chandra, J. Sheth, C. J. F. Boon, S. Sivaprasad, and A. Lotery, “Central serous chorioretinopathy: An update on risk factors, pathophysiology and imaging modalities,”  Progress in Retinal and Eye Research, Volume 79, November 2020.
  * [37] E. Vaghefi, S. Hill, H. M. Kersten, and D. Squirrell, “Multimodal Retinal Image Analysis via Deep Learning for the Diagnosis of Intermediate Dry Age-Related Macular Degeneration: A Feasibility Study,”  Journal of Ophthalmology, Vol. 2020, Article ID 7493419, January 2020\.
  * [38] A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny, “Efficient lifelong learning with A-GEM,”  Neural Information Processing Systems, 2019.
  * [39] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “iCaRL: Incremental Classifier and Representation Learning,”  IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [40] Y. Yang and S. Soatto, “FDA: Fourier Domain Adaptation for Semantic Segmentation,”  IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [41] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,”  International Journal of Computer Vision, 2015.
  * [42] Y. Gal, R. Islam, and Z. Ghahramani, “Deep Bayesian Active Learning with Image Data,”  International Conference on Machine Learning (ICML), 2017.
  * [43] M. Raghu and E. Schmidt, “A Survey of Deep Learning for Scientific Discovery,”  arXiv 2003.11755, 2020.
  * [44] T. Hassan, M. U. Akram, M. F. Masood, and U. Yasin, “BIOMISA Retinal Image Database for Macular and Ocular Syndromes,”  International Conference on Image Analysis and Recognition (ICIAR), 2018\.
  * [45] S. Farsiu, S. J. Chiu, R. V. O’Connell, F. A. Folgar, E. Yuan, J. A. Izatt, and C. A. Toth, “Quantitative Classification of Eyes with and without Intermediate Age-related Macular Degeneration Using Optical Coherence Tomography,”  Ophthalmology, 2014.
  * [46] S. J. Chiu, M. J. Allingham, P. S. Mettu, S. W. Cousins, J. A. Izatt, and S. Farsiu, “Kernel regression based segmentation of optical coherence tomography images with diabetic macular edema,”  Biomedical Optics Express, 2015.
  * [47] P. P. Srinivasan, L. Kim, P. Mettu, S. Cousins, G. Comer, J. Izatt, and S. Farsiu, “Fully automated detection of diabetic macular edema and dry age-related macular degeneration from optical coherence tomography images,”  Biomedical Optics Express, 2014.
  * [48] M. D. Zeiler, “ADADELTA: An Adaptive Learning Rate Method,”  arXiv, arXiv:1212.5701, 2012.
  * [49] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,”  IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [50] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,”  arXiv:1704.04861, 2017.
  * [51] A. M. Alqudah, “AOCT-NET: a convolutional network automated classification of multiclass retinal diseases using spectral-domain optical coherence tomography images,”  Medical & Biological Engineering & Computing, January 2020.
  * [52] T. Hassan, M. U. Akram, A. Shaukat, S. G. Khawaja, and B. Hassan, “Structure Tensor Graph Searches Based Fully Automated Grading and 3D Profiling of Maculopathy From Retinal OCT Images,”  IEEE Access, 2018.
  * [53] J. Zhang, J. Zhang, S. Ghosh, D. Li, S. Tasci, L. Heck, H. Zhang, and C. C. J. Kuo, “Class-incremental Learning via Deep Model Consolidation,”  arXiv:1903.07864, 2019.
  * [54] A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. S. Torr, “Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence,”  National Academy of Sciences of the United States of America (PNAS), 2016\.
  * [55] J. Kirkpatrick, R. Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, M. K., J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,”  National Academy of Sciences of the United States of America (PNAS), 2020\.
  * [56] A. Bitarafan, M. S. Baghshah, and M. Gheisari, “Incremental Evolving Domain Adaptation,”  IEEE Transactions on Knowledge and Data Engineering, 2020.
  * [57] A. Rosenfeld and J. K. Tsotsos, “Incremental Learning Through Deep Adaptation,”  arXiv, arXiv:1705.04228v2, 2018.
  * [58] S. Rosset, J. Zhu, and T. Hastie, “Margin Maximizing Loss Functions,”  Advances in Neural Information Processing Systems (NIPS), 2004.
  * [59] T. Y. Lin et al., “Focal Loss for Dense Object Detection,”  IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [60] M. Hayat, S. Khan, W. Zamir, J. Shen, and L. Shao, “Max-margin Class Imbalanced Learning with Gaussian Affinity,”  IEEE International Conference on Computer Vision (ICCV), 2019.