# Generalized Zero Shot Learning For Medical Image Classification

Dwarikanath Mahapatra1
1Inception Institute of Artificial Intelligence, UAE
<EMAIL_ADDRESS>
###### Abstract

In many real world medical image classification settings we do not have access
to samples of all possible disease classes, while a robust system is expected
to give high performance in recognizing novel test data. We propose a
generalized zero shot learning (GZSL) method that uses self supervised
learning (SSL) for: 1) selecting anchor vectors of different disease classes;
and 2) training a feature generator. Our approach does not require class
attribute vectors which are available for natural images but not for medical
images. SSL ensures that the anchor vectors are representative of each class.
SSL is also used to generate synthetic features of unseen classes. Using a
simpler architecture, our method matches a state of the art SSL based GZSL
method for natural images and outperforms all methods for medical images. Our
method is adaptable enough to accommodate class attribute vectors when they
are available for natural images.

##  1 Introduction

Medical image classification is an important step in computer aided diagnosis.
In the present era, deep learning methods have achieved state of the art
results for many medical image classification tasks such as diabetic
retinopathy grading[31, 112, 38, 25, 181, 59, 49, 123, 39, 15, 110], digital
patholology image classification [55, 29, 10, 12, 84, 73, 193, 151, 99, 70,
54, 69, 115, 94] and chest xray images [35, 185, 95, 116, 119, 66, 65, 68,
129, 64, 145, 143, 142, 60, 57], to name a few. Fully supervised learning
(FSL) methods that achieve state of the art results have access to disease
classes (labels) in the training and test sets. However in many real-world
scenarios we may not have access to samples of all possible diseases. A common
scenario is the diagnosis of radiological images, such as chest xrays. Unseen
classes are generally classified into one of the seen classes, resulting in
wrong diagnosis and treatment planning. For deployment in clinical settings it
is essential that a machine learning model learns to recognize novel test
cases.

Zero shot learning (ZSL) aims to learn plausible representations of unseen
classes from labeled data of seen classes, and recognize unseen classes during
test time. In a more generalized setting we expect to encounter both seen and
unseen classes during the test phase, and a reliable model should accurately
predict both classes. This is a case of generalized zero shot learning (GZSL)
which is a challenging scenario since we do not want to predict unseen classes
as one of the seen classes. We propose a GZSL method for medical image
classification using self supervised learning (SSL), demonstrate its
effectiveness across different datasets and also shows it’s applicability to
natural images.

GZSL has been a widely explored topic for natural images [27, 182, 190, 85,
158, 177, 121, 86, 40, 180, 89, 11, 87, 47, 16, 191, 109, 92, 114] where seen
and unseen classes are characterized by class attribute vectors. A model
learns to correlate between class attribute vectors and corresponding feature
representations. This gives a strong reference point in synthesizing features
of both seen and unseen classes, since by inputting the class attribute vector
of the desired class the corresponding feature representation can be
generated. However medical images do not have such well defined class
attributes since it requires high clinical expertise and time to define
unambiguous attribute vectors for different disease classes. Hence it is not a
straightforward task to apply state of the art GZSL methods to medical image
classification. While this makes GZSL for medical images a challenging task,
it is nevertheless essential to tackle this problem due to the potentially
immense benefit.

Initial approaches to tackle ZSL [21, 188, 172, 173, 81, 174, 91, 165, 163,
179, 176, 71, 126, 175, 125, 192, 98, 97, 96, 122, 150, 51] learnt cross-modal
relationships between visual feature and semantic embeddings (class attribute
vectors). Subsequently, recent generative approaches to GZSL [189, 27, 118,
169, 171, 117, 131, 168, 67, 93, 149, 147, 148, 146, 61, 63, 183, 62], used
generative adversarial networks (GANs) to optimize the divergence between the
data distribution of seen classes and generated features. Consequently,
generators trained on seen class features cannot accurately represent unseen
classes. The sub-optimal synthetic data does not lead to high performance of
such models. As an attempt to circumvent this problem some methods [160, 153,
58, 144, 138, 140, 141, 139, 135, 136, 130, 127, 137, 152, 124] utilize
unlabeled data of unseen classes in a transductive way. However they require
two GANs for seen and unseen classes as they do not consider the relations
between source and target domains.

However, methods leveraging transductive approaches are particularly relevant
for medical classification tasks [186, 111, 120, 26, 24, 134, 79, 78, 48, 77,
46, 36, 37, 14, 90, 76, 88, 166, 13, 74, 75, 108, 83, 50]. Absence of any
supervised information from the unseen domain makes it very challenging to
differentiate between disease labels, especially when many labels show similar
appearance to the untrained eye. In our method we also leverage the unlabeled
data of unseen classes as a guidance to train our GZSL method.

Another tricky issue facing GZSL applications in general and medical images in
particular is the potentially large semantic gap between images of different
classes. Consequently synthesizing such unseen class features from the seen
classes can be challenging. Leveraging unlabeled unseen class data (e.g.,
using anchors) can be effective in bridging the semantic gap [186, 30, 132,
82, 72, 80, 8, 100, 101, 105, 7].In an attemp to address the above challenges
our paper makes the following contributions:

  1. 1.

We propose a GZSL approach using self supervised learning (SSL) for medical
image classification. Our method outperforms state of the art methods for
multiple medical image datasets, and matches their performance on natural
images.

  2. 2.

We use SSL for: 1) deriving anchor vectors through improved clustering; and 2)
feature synthesis of seen and unseen classes.

  3. 3.

We achieve GZSL of medical images without using class attribute vectors
commonly used for natural images. This is important for real world clinical
scenarios where defining class attribute vectors is a time consuming and
expensive task.

##  2 Prior Work

(Generalized) Zero-Shot Learning: In Zero-Shot Learning [188, 104, 6, 106,
107, 103, 102, 128, 113, 28, 2, 133, 33, 164], the goal is to recognize
classes not encountered during training. External information about the novel
classes may be provided in forms of semantic attributes [52], visual
descriptions [1], or word embeddings [155]. Zero-shot learning has been
addressed using Generative Adversarial Networks (GANs) [189], Variational
Autoencoders (VAE) [170] or both of them [190].

In generalized zero-shot learning (GZSL), the purpose is to recognize images
from known and unknown domains. Many works [27, 182, 170, 189, 190] obtain
impressive results by training GANs in the known domain and generate unseen
visual features from the semantic labels. This allows them to train a fully
supervised classifier for two domains, which is robust to the biased
recognition problem. The work by Huang et al. [34] describes a Generative Dual
Adversarial Network (GDAN) which couples a Generator, a Regressor and a
Discriminator. The interaction between the three components produces various
visual features conditioned on class labels. Keshari et al. [43] use
overcomplete distributions to generate features of the unseen classes, while
Min et al. [156] use domain aware visual bias elimination for synthetic
feature generation. Different from the above works we achieve GZSL without the
need for descriptive class attribute vectors, but by specifying the class
label of the desired output feature. GZSL for medical image tasks have seen
limited applications such as registration [45] and artefact reduction [23].

Self-Supervised Learning: These methods consist of two main approaches; 1)
pre-text tasks and 2) down-stream tasks. Solving pre-text tasks learns a
proper data representation, although the task itself may not be relevant,
while down-stream tasks are used to evaluate the quality of features learned
by self-supervised learning and are independent of pre-text tasks. Contrastive
learning approaches such as MoCo [32] and SimCLR [22] are popular and give
state-of-the-art results for down-stream task-based methods. Self-supervised
learning (SSL) also addresses labeled data shortage and has found wide use in
medical image analysis by using innovative pre-text tasks for active learning
[123], anomaly detection [11], data augmentation [87], semi-supervised
histology classification [56],stain normalization [89] and registration [180].
Recent works also use self supervision for domain adaptation [167] and perhaps
the first work to combine GZSL and SSL [186]. While our work is inspired from
[186] in using SSL for GZSL, and using GANs for feature synthesis, there are
significant differences such as: 1) we do not use class attribute vectors for
training. Since medical images do not have defined class attribute vectors we
use a simpler architecture for GZSL. 2) [186] use a single generator but two
discriminators to differentiate between seen and unseen classes. However we
make use of a single generator and one discriminator to differentiate between
all classes by leveraging anchor vectors; 3) We use a SSL based clustering
approach to derive the anchor vectors of each class, including unseen classes.
We use high level knowledge of the number of classes as a supervisory signal.

##  3 Method

Figure 1: Architecture of proposed SC-GZSL method. In the first step we
generate anchor vectors (cluster centroids) by using SSL within the SwAV
clustering approach [20]. We have two clustering stages: one for Seen class
samples and second for Unseen classes. Feature generation leverages one
Generator and one Discriminator alongwith anchor vectors (from clustering) to
derive SSL loss terms.

###  3.1  Method Overview

Figure 1 depicts our proposed workflow. In the first step we generate anchor
vectors (cluster centroids) by using SSL within the SwAV clustering approach
[20]. We have two clustering stages: one for Seen class samples and second for
Unseen classes. Anchor vectors of the Seen class samples are used to get SSL
based loss terms for the second clustering stage. The second step involves
feature generation that takes a noise vector and desired class label of output
vector to synthesize features. Anchor vectors from the clustering stage are
used to derive SSL based loss terms. Synthesized and real features of unseen
and seen classes are used to train a softmax classifier for identifying
different disease classes.

###  3.2 SSL Clustering To Obtain Anchor Vectors

Let the number of classes in the Seen set be $n_{S}$, and the number of
classes in the Unseen set is $n_{U}$. We assume that the total number of
classes is known.We learn anchor vectors of the different classes by using the
SSL based online clustering approach SwAV (Swapping Assignments between
multiple Views) [20], and introduce additional SSL inspired loss terms.
Typical offline clustering methods [5, 19] alternate between cluster
assignment and centroid update. Since they require multiple passes over the
dataset, such methods are slow for online clustering. To overcome the high
training time and inspired by contrastive instance learning [187], [20]
enforce that different augmentations of the same image are mapped to the same
cluster. Multiple image views are contrasted by comparing their cluster
assignments instead of features.

We take the cluster centers to be class anchor vectors since they give a
reliable representation of the corresponding class. We choose to compute the
anchor vectors in an online fashion since the number of unseen classes may
change in a dynamic way depending upon the specific use case. Given image
features $x_{t}$ and $x_{s}$ from two different transformations of the same
image, we compute their cluster assignments $q_{t}$ and $q_{s}$ by computing
the distance of the features to a set of $K$ cluster centers
${c_{1},\cdots,c_{K}}$. A “swapped” prediction problem is solved with the
following loss function:

$\mathcal{L}(x_{t},x_{s})=\ell(x_{t},q_{s})+\ell(x_{s},q_{t})$ (1)

where $\ell(x,q)$ measures the fit between features $x$ and assignment $q$.
Thus we compare features $x_{t}$ and $x_{s}$ using their intermediate cluster
assignments $q_{t}$ and $q_{s}$ . If the two $x$’s capture same information,
we can predict the cluster assignment from the other feature.

Online clustering: Given image $I_{n}$, it is transformed to $I_{nt}$ using
transformation $t$ from a set $T$ of image transformations. A non-linear
mapping $f_{\theta}$ transforms $I_{nt}$ to a feature vector which is
projected to the unit sphere, i.e.,
$x_{nt}=f_{\theta}(x_{nt})/\left\|f_{\theta}(x_{nt})\right\|_{2}$. The cluster
assignment $q_{nt}$ is computed by determining the distance of $x_{nt}$ to the
set of cluster centroids, ${c_{1},\cdots,c_{K}}$. $C$ denotes a matrix whose
columns are $c_{1},\cdots,c_{k}$.

##### Swapped prediction problem:

Each term in Eq.1 represents the cross entropy loss between $q$ and the
probability obtained by taking a softmax of the dot products of $x_{i}$ and
all columns in $C$, i.e.,

$\ell(x_{t},q_{s})=-\sum_{k}q_{s}^{(k)}\log
p_{t}^{(k)},~{}~{}p_{t}^{(k)}=\frac{\exp{\frac{x_{t}^{\top}c_{k}}{\tau}}}{\sum_{k^{\prime}}\exp{\frac{x_{t}^{\top}c_{k}}{\tau}}}$
(2)

where $\tau=0.1$ is the temperature parameter [187]. Computing this loss over
all images and augmentations results in the following loss function for
swapped prediction:

$\begin{split}\mathcal{L}(x_{t},x_{s})=-\frac{1}{N}\sum_{n=1}^{N}\sum_{s,t\sim
T}\biggl{[}\frac{x_{nt}^{\top}Cq_{ns}}{\tau}+\frac{x_{ns}^{\top}Cq_{nt}}{\tau}\\\
-\log\sum_{k=1}^{K}\exp\left(\frac{x_{nt}^{\top}c_{k}}{\tau}\right)-\log\sum_{k=1}^{K}\exp\left(\frac{x_{ns}^{\top}c_{k}}{\tau}\right)\biggr{]}.\end{split}$
(3)

This loss function is jointly minimized with respect to the centroids in $C$
and parameters $\theta$ of $f_{\theta}$.

Computing the cluster assignments: The clustering assignments $q$ are computed
in an online fashion using image features within a batch. Since the centroids
in $C$ are used across different batches, SwAV clusters multiple instances to
their appropriate clusters. Given feature vectors $X=[x_{1},\cdots,x_{B}]$, we
map them to centroids $C=[c_{1},\cdots,c_{K}]$ using $Q=[q_{1},\cdots,q_{B}]$,
and we optimize $Q$ to maximize the similarity between $X$ and $C$,

$\max_{Q\in\mathcal{Q}}Tr(Q^{\top}C^{\top}X)+\epsilon H(Q),$ (4)

where H is the entropy function, $H(Q)=-\sum_{ij}Q_{ij}\log Q_{ij}$ and
$\epsilon=0.05$ controls smoothness of mapping. A high $\epsilon$ could
potentially results in a trivial solution where all samples collapse into an
unique representation and are assigned uniformly to all prototypes.

##### Our Novel Contribution:

We use the concept of anchor vectors to bridge the gap between seen and unseen
classes, which is determined by the following steps: Assuming we have $n_{S}$
seen classes we first cluster the Seen class images into $n_{S}$ clusters and
obtain their centroids as $C_{S}={c_{1},\cdots,c_{n_{S}}}$. In the next pass
we compute the clusters $C_{U}={c_{n_{S}+1},\cdots,c_{n_{S}+n_{U}}}$ of the
$n_{U}$ unseen classes using the following additional constraints:

  1. 1.

The centroids in $C_{S}$ do not change since they have been computed from the
seen classes.

  2. 2.

A self supervised constraint is added where the centroids of the unseen
classes are forced to be different from the seen class centroids. This is done
to account for the situation that some of the Unseen classes may be
semantically close to one or more Seen classes. This may happen when images of
different disease labels have very similar appearance which can be a common
occurrence for radiological images. This condition is implemented using:

$\mathcal{L}_{SSL1}=\min\left(CoSim(C_{S}^{i},C_{U}^{j}),\sigma_{1}\right)$
(5)

Here $\sigma_{1}=0.15$ is a parameter that determines the semantic distance
between the centroids, and CoSim denotes cosine similarity.

  3. 3.

We add a second self supervised constraint that the similarity of seen class
sample, $x_{s}^{i}$, with its corresponding class centroid $C_{S}^{i}$ is
higher than their similarity w.r.t all $C_{U}^{j}$. This is achieved by
randomly selecting samples from the Seen class training set during minibatch
training and computing the different cosine similarities. This constraint is
implemented by

$\begin{split}&\mathcal{L}_{SSL2}=\\\
&\max\left(CoSim(x_{S}^{i},C_{S}^{i})-CoSim(x_{S}^{i},C_{U}^{j}),\sigma_{2}\right)\forall
j\end{split}$ (6)

$\sigma_{2}=0.25$ controls the minimum degree of semantic difference between
different classes.

The final loss term for clustering the Unseen class samples is
$\mathcal{L}_{Unseen}=\mathcal{L}(x_{s},x_{t})+\lambda_{1}L_{SSL1}-\lambda_{2}L_{SSL2}$,
where $\mathcal{L}(x_{s},x_{t})$ is defined in Eqn. 3.
$\lambda_{1}=1.1,\lambda_{2}=0.7$ are the weights. The
‘$-\lambda_{2}L_{SSL2}$’ ensures that the loss term does not increase
arbitrarily which is possible for ‘$+\lambda_{2}L_{SSL2}$’.

###  3.3 Feature Generation Network

Given the training images of Seen classes and unlabeled images of the Unseen
classes we learn a generator
$G:\mathcal{E},\mathcal{Z}\longrightarrow\mathcal{X}$ , which takes a class
label vector $e^{y}\in\mathcal{E}$ and a Gaussian noise vector
$z\in\mathcal{Z}$ as inputs, and generates a feature vector
$\tilde{x}\in\mathcal{X}$. The discriminator
$D:\mathcal{X},\mathcal{E}\xrightarrow{}[0,1]$ takes a real feature $x$ or
synthetic feature $\tilde{x}$ and corresponding class label vector $e^{y}$ as
input and determines whether the feature vector matches the class label
vector. The generator $G$ aims to fool $D$ by producing features highly
correlated with $e^{y}$ using a Wasserstein adversarial loss[4]:

$\begin{split}\mathcal{L}_{WGAN}=\min_{G}\max_{D}\mathbb{E}[D(x,e^{y})]-\mathbb{E}[D(\tilde{x},e^{y})]\\\
-\lambda\mathbb{E}[(\left\|\nabla_{\tilde{x}}D(\tilde{x},e^{y})\right\|_{2}-1)^{2}]\end{split}$
(7)

where the third term is a gradient penalty term, and $\tilde{x}=\alpha
x+(1-\alpha)\tilde{x}$. $\alpha\sim U(0,1)$ is sampled from a uniform
distribution.

####  3.3.1 Self Supervised Loss From Anchor Vectors

The discriminator $D$ is a classifier that determines whether the generated
feature vector $\tilde{x}$ belongs to one of the seen classes. Since the
unseen classes are not labeled we do not have a data distribution for them and
hence we use self supervision to determine whether the generated feature
vector matches an unseen class. As the anchor vectors (i.e., the cluster
centers) are fixed, we calculate the cosine distance between the generated
vector $\tilde{x}$ and the anchor vector corresponding to the desired class
$y$, i.e.

$\mathcal{L}_{SSL3}=1-CoSim(\tilde{x},c_{y})$ (8)

If $\tilde{x}$ truly represents the desired class $y$ then the cosine
similarity between $\tilde{x}$ and the corresponding anchor vector $c_{y}$
should be highest amongst all $K(=n_{S}+n_{U})$ anchor vectors, and the
corresponding loss is lowest.

####  3.3.2 Classifier Loss

We expect that $\tilde{x}^{s}$ (synthesized feature vector for seen classes)
are predicted correctly by a pre-trained classifier $CL$ with a loss defined
as below

$\mathcal{L}_{CL}=-\mathbb{E}_{(\tilde{x}^{s},y^{s})\sim
P_{\tilde{x}^{s}}}\left[\log P(y^{s}|\tilde{x}^{s},\theta_{CL})\right]$ (9)

where $P(y^{s}|\tilde{x}^{s},\theta_{CL})$ is the classification probability
and $\theta_{CL}$ denotes fixed parameters of the pre-trained classifier.

###  3.4 Training and Implementation

The final loss function is defined as

$\mathcal{L}=\mathcal{L}_{WGAN}+\lambda_{CL}\mathcal{L}_{CL}+\lambda_{3}L_{SSL3}$
(10)

where $\lambda_{CL},\lambda_{3}$ are weights that balance the contribution of
the different terms. Once training is complete we specify the label of desired
class and input a noise vector to $G$ which synthesizes a new feature vector.
We combine the synthesized target features of the unseen class $\tilde{x}^{u}$
and real and synthetic features of seen class $x^{s},\tilde{x}^{s}$ to
construct the training set. Then we train a softmax classifier by minimizing
the negative log likelihood loss:

$\min_{\theta}-\frac{1}{|\mathcal{X}|}\sum_{(x,y)\in(\mathcal{X},\mathcal{Y})}\log
P(y|x,\theta),$ (11)

where
$P(y|x,\theta)=\frac{\exp(\theta^{T}_{y}x)}{\sum_{j=1}^{|\mathcal{Y}|}\exp(\theta^{T}_{y}x)}$
is the classification probability and $\theta$ denotes classifier parameters.
The final class prediction is by $f(x)=\arg\max_{y}P(y|x,\theta)$

Implementation Details: We show results for natural and medical images and
compare with existing GZSL methods. Extending our method to natural images is
straightforward where in we replace the class label vector $e^{y}$ with the
corresponding class attribute vectors. For feature extraction, similar to
[188], we use a pre-trained ResNet-101 to extract $2048$ dimensional CNN
features for natural images. The generator (G) and discriminator (D) are all
multilayer perceptrons. $G$ has two hidden layers of $2000$ and $1000$ units
respectively while the discriminator D is implemented with one hidden layer of
$1000$ hidden units. We choose Adam [44] as our optimizer, and the momentum is
set to ($0.9,0.999$). The values of loss term weights are
$\lambda_{CL}=0.6,\lambda_{3}=0.9$. Training the Swav Clustering algorithm
takes $12$ hours and the feature synthesis network for $50$ epochs takes $17$
hours, all on a single NVIDIA V100 GPU (32 GB RAM). PyTorch was used for all
implementations.

###  3.5  Evaluation Protocol

The seen class $S$ can have samples from $2$ or more disease classes, and the
unseen class $U$ contains samples from the remaining classes. We use all
possible combinations of labels in $S$ and $U$. Following standard practice
for GZSL, average class accuracies are calculated for two settings: 1) S:
training is performed on synthesized samples of $S+U$ classes and test on
$S_{Te}$. 2) U: training is performed on synthesized samples of $S+U$ classes
and test on $U$. We also report the harmonic mean defined as

$H=\frac{2\times Acc_{U}\times Acc_{S}}{Acc_{U}+Acc_{S}}$ (12)

where $Acc_{S}$ and $Acc_{U}$ denote the accuracy of images from seen (setting
$S$) and unseen (setting $U$) classes respectively:

##  4 Experimental Results

|  |  |
---|---|---|---
(a) | (b) | (c) | (d)

---

Figure 2: Feature visualizations for NIH ChestXray Dataset: (a) Seen+Unseen
classes from actual dataset; distribution of synthetic samples generated by
(b) SC-GZSL; (c) SC-GZSL${}_{w\mathcal{L}_{SSL3}}$; (d) SDGN [186]. Different
colours represent different classes. (b) is closer to (a), while (c) and (d)
are quite different.

###  4.1 Dataset Description

We demonstrate our method’s effectiveness on natural images and the following
medical imaging datasets for classification tasks. Datasets with a minimum of
3 disease classes (excluding normal label) were chosen to highlight the
performance of feature synthesis.

  1. 1.

CAMELYON17 dataset [17]: contains 1000 whole slide images (WSIs) with 5 slides
per patient: 500 slides for training and 500 slides for test. Training set has
annotations of 3 categories of lymph node metastasis: Macro (Metastases
greater than $2.0$ mm), Micro (metastasis greater than $0.2$ mm or more than
$200$ cells, but smaller than $2.0$ mm), and ITC (single tumor cells or a
cluster of tumor cells smaller than $0.2$mm or less than $200$ cells). We
extract $224\times 224$ patches from the different slides and obtain $130,000$
tumor patches and $200,000$ normal patches. We take a pre-trained ResNet101
and finetune the last FC layer using the CAMELYON16 dataset [9], which is
closely related but different from CAMELYON17. A baseline fully supervised
learning (FSL) method is implemented111https://grand-challenge-
public.s3.amazonaws.com/evaluation-
supplementary/80/46fc579c-51f0-40c4-bd1a-7c28e8033f33/Camelyon17_.pdf which is
the top ranked in the leaderboard.

  2. 2.

NIH Chest Xray Dataset: For lung disease classification we adopted the NIH
ChestXray14 dataset [185] having $112,120$ expert-annotated frontal-view
X-rays from $30,805$ unique patients and has $14$ disease labels. Original
images were resized to $224\times 224$. A pre-trained resnet-101 was finetuned
using the CheXpert dataset [35] and the chosen baseline FSL was from [162].

  3. 3.

CheXpert Dataset: We used the CheXpert dataset [35] consisting of $224,316$
chest radiographs of $65,240$ patients labeled for the presence of $14$ common
chest conditions. Original images were resized to $224\times 224$. A pre-
trained resnet-101 was finetuned using the NIH dataset [185] and the baseline
FSL method was of [161] which is ranked second for the dataset with shared
code.

  4. 4.

Kaggle Diabetic Retinopathy dataset: has approximately $35,000$ images in the
provided training set [41]. Images are labeled by a single clinician with the
respective DR grade, out of 4 severity levels: 1- mild(2443 images),
2-moderate (5291 images), 3-severe (873 images), and 4-proliferative DR (708
images). The normal class $0$ has 25810 images. A pre-trained resnet-101 was
finetuned using [178] which has $9939$ color fundus images ($2720\times 2720$)
from $2740$ diabetic patients. Although the number of classes are different
from Kaggle the features are accurate since the end task is DR detection. The
chosen baseline method was of [3]. Original images were resized to $224\times
224$.

  5. 5.

Gleason grading challenge dataset 222https://gleason2019.grand-
challenge.org/Home for prostate cancer (PCA) [42]. It has $333$ Tissue
Microarrays (TMAs) from $231$ patients and has $5$ Gleason grades. Six
pathologists with $27,15,1,24,17$, and $5$ years of experience annotated the
data and majority voting was used to construct the “ground truth label”. The
training set had $200$ TMAs while the validation set had $44$ TMAs. A separate
test set consisting of $87$ TMAs from 60 other patients. Although a much
larger dataset for PCA using WSIs is
available333https://www.kaggle.com/c/prostate-cancer-grade-
assessment/overview, the data cannot be used for external
submissions444https://www.kaggle.com/c/prostate-cancer-grade-
assessment/discussion/201117. The baseline FSL was the classification outcome
of the top ranked method555https://github.com/hubutui/Gleason. The feature
extractor was a pre-trained ResNet101 finetuned using the CAMELYON16 dataset
[9]. Since both are histopathology image datasets, the feature extractor is
quite accurate. The high dimensional images were divided into $224\times 224$
patches. The individual labels patches from normal images were all ‘normal’.
For the diseased images (all Gleason grades except 1), the labels of
individual patches were obtained using the multiple instance learning method
of [18]. Thus we obtained more than $5,000$ patches of each label.

Since we did not have labels of the organizer designated test sets of all
datasets, a $70/10/20$ split at patient level was done to get training,
validation and test sets for NIH Chest Xray, CheXpert and Kaggle DR datasets.

For natural images we use the following five datasets: 1) CUB [184], AwA1
[52], AwA2 [188], SUN [159], and FLO [157]. CUB includes 11K images and 200
species of birds labeled with 312-D attributes. AwA1 and AwA2 consist of 50
kinds of animals described by 85- D attributes, containing 30K and 37K images
respectively. SUN is a large-scale scene attribute dataset, including 717
classes and 14K images with 102-D attributes. FLO con- sists of 8K images from
102 flower classes. Adapting our method to natural images is done by replacing
the class vector (e) with the class attribute vector.

###  4.2 Baseline Methods

We compare our method’s performance with the following GZSL methods employing
different feature generation approaches such as CVAE or GANs: 1) CVAE based
generation method of [34]; 2) over complete distribution (OCD) method of [43];
3) self-supervised learning GZSL method of [186]; 4) FSL- Top performing FSL
methods of corresponding datasets. Following GZSL protocol we report
performance for Seen and Unseen classes. Our method is denoted as SC-GZSL
(Selfsupervised Clustering based GZSL).

###  4.3 Visualization of Synthetic Image Features

Figure 2 (a) shows t-SNE plot of features from actual data from the NIH chest
Xray dataset where the different classes are spread over a wide area, with
slight overlap between some classes. Figure 2 (b) shows the distribution of
synthetic features generated by our method. Although the corresponding
clusters for the different classes have separate locations in the two figures
they are similar to that of Figure 2 (a) in the sense that the different
classes are similarly separated. Figure 2 (c) shows the feature distribution
for our method without using self-supervision. The resulting distribution is
compact without overlap between classes, which is not representative of the
real-world case. Classifiers trained on such distributions perform poorly on
unseen classes. Figure 2 (d) shows the feature distributions using SDGN [186].
Although it also uses SSL the resulting feature representation is less
accurate than our proposed method which contributes to the corresponding
inferior performance.

###  4.4 Generalized Zero Shot Learning Results

Table 1 summarizes the results of our algorithm on natural images. The best
performing method amongst all competing methods is SDGN [186]. However we are
able to outperform it despite using a much simpler architecture. A McNemar’s
statistical test [154] shows that the results between SC-GZSL and SDGN is not
very significant ($p=0.062$), except for the SUN dataset (p=0.01). This
dataset is particularly challenging as demonstrated by the fact that accuracy
values are lower than other datasets.

The results for medical images shown in Table 2 shows our proposed method
outperforms all competing GZSL methods including SDGN. This significant
difference in performance can be explained by the fact that the complex
architectures that worked for natural images will not be equally effective for
medical images which have less information. Absence of attribute vectors for
medical images is another contributing factor. The class attributes provide a
rich source of information about natural images which can be leveraged using
existing architectures. On the other hand medical images require a different
approach.

Natural Images Method CUB AwA1 AwA2 SUN FLO S U H S U H S U H S U H S U H
f-Vaegan [190] 65.1 61.4 63.2 - - - 88.6 84.8 86.7 41.9 60.6 49.6 87.2 78.7
82.7 GXE [53] 68.7 57.0 62.3 89.0 87.7 88.4 90.0 80.2 84.8 58.1 45.4 51.0 - -
- SDGN [186] 70.2 69.9 70.1 88.1 87.3 87.7 89.3 88.8 89.1 46.0 62.0 52.8 91.4
78.3 84.4 GDAN [34] 66.7 39.3 49.5 - - - 67.5 32.1 43.5 89.9 38.1 53.4 - - -
OCD[43] 59.9 44.8 51.3 - - - 73.4 59.5 65.7 42.9 44.8 43.8 - - - SCGZSL 71.7
70.6 71.1 88.5 88.1 88.3 89.9 89.3 89.6 50.3 62.1 55.6 91.8 79.4 85.2

Table 1: GZSL Results For Natural Images:Average per-class classification
accuracy ($\%$) and harmonic mean (H) accuracy of generalized zero-shot
learning when test samples are from Seen(S) or Unseen (U) classes. Numbers for
competing methods are taken from [186]. $S,U$ denote $Acc_{S},Acc_{U}$.

Multiple Medical Image Datasets Method CAMELYON17 NIH Xray CheXpert Kaggle DR
Gleason S U H S U H S U H S U H S U H f-VAEGAN [190] 90.2 88.2 89.2 82.9 80.0
81.4 88.5 87.6 88.0 92.8 90.2 91.5 88.2 85.1 86.6 GDAN [34] 91.1 89.1 90.1
83.8 80.9 82.3 89.2 88.0 88.6 94.2 91.0 92.6 88.8 86 87.4 OCD[43] 91.5 89.3
90.4 84.7 81.3 83.0 89.9 88.1 89.0 94.8 91.3 93.0 89.2 86.9 88 SDGN [186] 92.1
89.5 90.8 84.4 81.1 82.7 90.2 88.2 89.2 95.0 91.9 93.4 90.0 87.8 88.9 SCGZSL
93.5 91.1 92.3 87.2 84.3 85.7 91.8 89.4 90.6 96.1 93.2 94.7 92.1 89.5 90.8 FSL
93.7 93.5 93.6 87.4 86.9 87.1 92.1 92.5 92.3 96.4 96.1 96.2 92.4 92.2 92.3
SCGZSL Ablation Studies ${}_{w\mathcal{L}_{SSL1}}$ 91.2 88.7 89.9 84.5 82.1
83.3 89.1 86.9 88.0 92.2 89.6 90.9 90.3 86.9 88.6 ${}_{w\mathcal{L}_{SSL2}}$
90.8 88.1 89.4 84.0 82.2 83.1 88.8 86.2 87.5 91.8 88.2 90.0 89.2 86 87.6
${}_{w\mathcal{L}_{SSL3}}$ 90.0 87.0 88.5 83.2 81.0 82.1 87.6 85.1 86.3 90.1
86.7 88.4 88.4 85.5 86.9 only ${\mathcal{L}_{SSL3}}$ 89.3 86.4 87.8 82.6 80.7
81.6 87.0 84.5 85.7 88.9 85.9 87.4 87.7 84.9 86.3 L 87.2 84.1 85.6 80.7 79.1
79.7 84.6 82.7 83.6 86.5 83.7 85.1 86.1 82.8 84.4

Table 2: GZSL Results For Medical Images:Average per-class classification
accuracy ($\%$) and harmonic mean accuracy of generalized zero-shot learning
when test samples are from Seen (Setting $S$) or unseen (Setting $U$) classes.
Results of ablation studies are also shown.

###  4.5 Ablation Studies

Table 2 also shows results for the following ablation studies: 1)
SCGZSL${}_{w\mathcal{L}_{SSL1}}$\- SCGZSL without the loss term
$\mathcal{L}_{SSL1}$ (Eqn.5) for obtaining the anchor vectors, 2)
SCGZSL${}_{w\mathcal{L}_{SSL2}}$ \- SCGZSL without the loss term
$\mathcal{L}_{SSL2}$ (Eqn.6) to get anchor vectors, 3) SCGZSLL\- Using only
the baseline loss term $\mathcal{L}(z_{s},z_{t})$ (Eqn.3) for clustering all
seen and unseen classes together, and no $\mathcal{L}_{SSL3}$ for feature
synthesis; 4) SCGZSL${}_{w\mathcal{L}_{SSL3}}$\- SCGZSL without the loss term
$\mathcal{L}_{SSL3}$ (Eqn.8) for training the feature synthesis network; 5)
SCGZSL$-\text{only}\mathcal{L}_{SSL3}$\- SCGZSL using only
$\mathcal{L}_{SSL3}$ for feature synthesis without $\mathcal{L}_{SSL1}$,
$\mathcal{L}_{SSL2}$.

The first three ablation studies investigate the effect of clustering on the
final classification results. Their significant performance degradation
compared to SCGZSL indicates the importance of our novel SSL based terms
($\mathcal{L}_{SSL1},\mathcal{L}_{SSL2}$) in obtaining accurate anchor
vectors. The baseline method, SCGZSLL, does not use any form of self
supervision and has lowest $H$ values. Compared to SCGZSL, we observe that
excluding ${\mathcal{L}_{SSL3}}$ (SCGZSL${}_{w\mathcal{L}_{SSL3}}$) leads to
maximum reduction of $H$ (more than $3.5\%$) across all datasets . This
indicates that ${\mathcal{L}_{SSL3}}$ makes the most significant contribution
to our method’s performance. The use of anchor vectors makes it easier to
synthesize features of unseen classes.

The influence of ${\mathcal{L}_{SSL1}},{\mathcal{L}_{SSL2}}$ is quantitatively
similar as shown by similar $H$ values of SCGZSL${}_{w\mathcal{L}_{SSL1}}$,
SCGZSL${}_{w\mathcal{L}_{SSL2}}$ across all datasets. However their difference
in $H$ values compared to SCGZSL is nearly $2.4\%$ which is significant
($p=0.01$). Thus the use of self supervision is an important factor in
obtaining accurate anchor vectors (cluster centroids). Although the baseline
clustering mechanism, SwAV, uses self supervision in the form of contrastive
loss, including $\mathcal{L}_{SSL1}$ and $\mathcal{L}_{SSL2}$ sigificantly
improves clustering accuracy. Excluding both
$\mathcal{L}_{SSL1},\mathcal{L}_{SSL2}$ and using the baseline SwAV (‘only
${w\mathcal{L}_{SSL3}}$’) gives significantly reduced $H$ values for the
different datasets despite using $\mathcal{L}_{SSL3}$ for feature synthesis.
This clearly indicates the importance of having accurate anchor vectors for
our method. SCGZSLL can be considered as the most basic method without using
any of our proposed novel loss terms, and unsurprisingly gives the worst
results.

---
(a)
(b)
Figure 3: Hyperparameter Plots showing the value of $H$ and classification
accuracy for different values of;(a) $\lambda$; (b) $\sigma$.

###  4.6 Hyperparameter Selection

For all the competing methods in the case of medical images we start with the
original values provided by the authors and vary them in range $x\pm 0.5x$ in
steps of $x/10$, where $x$ is the initial value. The best results are usually
obtained using author provided values for each method.

Figure 3 (a) shows the harmonic mean values for the NIH Chest Xray dataset for
different values of hyperparameters $\lambda_{1},\lambda_{2},\lambda_{3}$,
while Figure 3 (b) shows the corresponding plots for different values of
$\sigma_{1},\sigma_{2}$. The $\lambda$’s were varied between $[0.4-1.5]$ in
steps of $0.05$ and the performance on a separate test set of $10,000$ images
was monitored. We start with the base cost function of Eqn. 7, and first
select the optimum value of $\lambda_{1}$. $\lambda_{1}$ values is fixed and
we then determine $\lambda_{2}$, and then $\lambda_{3}$ by fixing
$\lambda_{1},\lambda_{2}$. The order in which the parameters were set is
important and we find the above order as giving the best results. Similarly
the value of $\sigma$’s were varied between $[0.1,0.5]$ in steps of $0.05$,
and the resulting classification accuracy of the Xray images was determined.
i.e., whether they were assigned to the correct cluster (class).

Figure 4 shows, for the NIH Chest Xray and CAMELYON17 dataset, the effect of
adding synthetic samples on $Acc_{S},Acc_{U}$ as a function of dataset
augmentation factor. Increasing synthesized examples increases $Acc_{U}$ at a
high rate while reducing $Acc_{S}$, although at a lower rate. TSynthetic
samples improve discriminative power of classifiers and reduce bias towards
Seen classes.

|
---|---
(a) | (b)
Figure 4: Value of accuracy and H when adding synthetic samples to the
dataset: (a) NIH dataset; (b) CAMELYON17 dataset.

##  5 Conclusion

We propose a GZSL approach for medical images without relying on class
attribute vectors. Our novel method can accurately synthesize feature vectors
of unseen classes by employing self supervised learning at different stages
such as anchor vector selection, and training the feature generator. Using
self supervision allows us to bridge the semantic gap between Seen and Unseen
classes. The distribution of synthetic features generated by our method are
close to the actual distribution, while removing the self-supervised term
results in unrealistic distributions. Experimental results show our method
outperforms other GZSL approaches in literature.

## References

  * [1] Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele.  Evaluation of output embeddings for fine-grained image classification.  In In Proc. IEEE CVPR, pages 2927–2936, 2015.
  * [2] BJ Antony, S Sedai, D Mahapatra, and R Garnavi.  Real-time passive monitoring and assessment of pediatric eye health.  In US Patent App. 16/178,757, 2020.
  * [3] Teresa Araujo, Guilherme Aresta, Luís Mendonca, Susana Penas, Carolina Maia, Angela Carneiro, Ana Maria Mendonca, and Aurelio Campilho.  DR—GRADUATE: uncertainty-aware deep learning-based diabetic retinopathy grading in eye fundus images.  Medical Image Analysis, 2020.
  * [4] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein gan.  In arXiv preprint arXiv:1701.07875, 2017.
  * [5] Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi.  Self-labelling via simultaneous clustering and representation learning.  In International Conference on Learning Representations (ICLR), 2020\.
  * [6] P.R. Bastide, I.F. Kiral-Kornek, D. Mahapatra, S. Saha, A. Vishwanath, and S. Von Cavallar.  Machine learned optimizing of health activity for participants during meeting times.  In US Patent App. 15/426,634, 2018.
  * [7] P.R. Bastide, I.F. Kiral-Kornek, D. Mahapatra, S. Saha, A. Vishwanath, and S. Von Cavallar.  Visual health maintenance and improvement.  In US Patent 9,993,385, 2018.
  * [8] P.R. Bastide, I.F. Kiral-Kornek, D. Mahapatra, S. Saha, A. Vishwanath, and S. Von Cavallar.  Crowdsourcing health improvements routes.  In US Patent App. 15/611,519, 2019.
  * [9] B. E. Bejnordi, M. Veta, P. J. van Diest, B. van Ginneken, N. Karssemeijer, G. Litjens, J. van der Laak, , and the CAMELYON16 Consortium.  Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer.  JAMA, 318(22):2199–2210, 2017.
  * [10] B. Bozorgtabar, D. Mahapatra, and J-P. Thiran.  Exprada: Adversarial domain adaptation for facial expression analysis.  In Press Pattern Recognition, 100:15–28, 2020.
  * [11] B. Bozorgtabar, D. Mahapatra, J.-P. Thiran, and L. Shao.  SALAD: Self-supervised aggregation learning for anomaly detection on x-rays.  In In Proc. MICCAI, pages 468–478, 2020.
  * [12] B. Bozorgtabar, D. Mahapatra, H. von Teng, A. Pollinger, L. Ebner, J-P. Thiran, and M. Reyes.  Informative sample generation using class aware generative adversarial networks for classification of chest xrays.  Computer Vision and Image Understanding, 184:57–65, 2019.
  * [13] B. Bozorgtabar, D. Mahapatra, H. von Teng, A. Pollinger, L. Ebner, J-P. Thiran, and M. Reyes.  Informative sample generation using class aware generative adversarial networks for classification of chest xrays.  In arXiv preprint arXiv:1904.10781, 2019.
  * [14] B. Bozorgtabar, D. Mahapatra, G. Vray, and J.-P. Thiran.  Anomaly detection on x-rays using self-supervised aggregation learning.  In arXiv preprint arXiv:2010.09856, 2020.
  * [15] B. Bozorgtabar, D. Mahapatra, I. Zlobec, T.T. Rau, and J.P. Thiran.  Computational pathology.  Frontiers in Medicine, 7, 2020.
  * [16] B. Bozorgtabar, M. Saeed Rad, D. Mahapatra, and J-P. Thiran.  Syndemo: Synergistic deep feature alignment for joint learning of depth and ego-motion.  In In Proc. IEEE ICCV, 2019.
  * [17] P. Bándi, , and et al.  From Detection of Individual Metastases to Classification of Lymph Node Status at the Patient Level: The CAMELYON17 Challenge.  IEEE Trans. Med. Imag., 38(2):550–560, 2019.
  * [18] Gabriele Campanella, Vitor M.K. Silva, and Thomas J. Fuchs.  Terabyte-scale deep multiple instance learning for classification and localization in pathology.  In arXiv preprint arXiv:1805.06983, 2018.
  * [19] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In European Conference on Computer Vision, 2018.
  * [20] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9912–9924. Curran Associates, Inc., 2020.
  * [21] Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha.  Synthesized classifiers for zero-shot learning.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5327–5336, 2016.
  * [22] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton.  A simple framework for contrastive learning of visual representations.  In arXiv preprint arXiv:2002.05709, 2020.
  * [23] Y. Chen, Y. Chang, S. Wen, Y. Shi, X. Xu, T. Ho, Q. Jia, M. Huang, and J. Zhuang.  Zero-shot medical image artifact reduction.  In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pages 862–866, 2020.
  * [24] Sourya Dipta Das, Saikat Dutta, Nisarg A Shah, Dwarikanath Mahapatra, and Zongyuan Ge.  Anomaly detection in retinal images using multi-scale deep feature sparse coding.  In arXiv preprint arXiv:2201.11506, 2022.
  * [25] K. Devika, Dwarikanath Mahapatra, Ramanathan Subramanian, and Venkata Ramana Murthy Oruganti.  Outlier-based autism detection using longitudinal structural mri.  IEEE Access, 10:27794–27808, 2022.
  * [26] K. Devika, Dwarikanath Mahapatra, Ramanathan Subramanian, and Venkata Ramana Murthy Oruganti.  Outlier-based autism detection using longitudinal structural mri.  In arXiv preprint arXiv:2202.09988, 2022.
  * [27] Rafael Felix, Vijay Kumar, Ian Reid, and Gustavo Carneiro.  Multi-modal cycle-consistent generalized zero-shot learning.  In ECCV, pages 21–37, 2018.
  * [28] R Garnavi, D Mahapatra, PK Roy, and RB Tennakoon.  System and method to teach and evaluate image grading performance using prior learned expert knowledge base.  In US Patent App. 10,657,838, 2020.
  * [29] Z. Ge, D. Mahapatra, X. Chang, Z. Chen, L. Chi, and H. Lu.  Improving multi-label chest x-ray disease diagnosis by exploiting disease and health labels dependencies.  In press Multimedia Tools and Application, pages 1–14, 2019.
  * [30] Z. Ge, D. Mahapatra, S. Sedai, R. Garnavi, and R. Chakravorty.  Chest x-rays classification: A multi-label and fine-grained problem.  In arXiv preprint arXiv:1807.07247, 2018.
  * [31] Varun Gulshan, Lily Peng, Marc Coram, Martin C. Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, Ramasamy Kim, Rajiv Raman, Philip C. Nelson, Jessica L. Mega, and Dale R. Webster.  Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs.  JAMA, 316(22):2402–2410, 12 2016.
  * [32] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proc. CVPR, pages 9729–9738, 2020.
  * [33] J De Hoog, D Mahapatra, R Garnavi, and F Jalali.  Personalized monitoring of injury rehabilitation through mobile device imaging.  In US Patent App. 16/589,046, 2021.
  * [34] He Huang, Changhu Wang, Philip S. Yu, and Chang-Dong Wang.  Generative dual adversarial network for generalized zero-shot learning.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 801–810, June 2019.
  * [35] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng.  CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In arXiv preprint arXiv:1901.07031, 2017.
  * [36] L Ju, X Wang, L Wang, T. Liu, X. Zhao, T. Drummond, D. Mahapatra, and Z. Ge.  Relational subsets knowledge distillation for long-tailed retinal diseases recognition.  In arXiv preprint arXiv:2104.11057, 2021.
  * [37] L Ju, X Wang, L Wang, D Mahapatra, X Zhao, M Harandi, T Drummond, Tongliang Liu, and Zongyuan Ge.  Improving medical image classification with label noise using dual-uncertainty estimation.  In arXiv preprint arXiv:2103.00528, 2020.
  * [38] Lie Ju, Xin Wang, Lin Wang, Dwarikanath Mahapatra, Xin Zhao, Quan Zhou, Tongliang Liu, and Zongyuan Ge.  Improving medical images classification with label noise using dual-uncertainty estimation.  IEEE Transactions on Medical Imaging, pages 1–1, 2022.
  * [39] Lie Ju, Xin Wang, Xin Zhao, Huimin Lu, Dwarikanath Mahapatra, Paul Bonnington, and Zongyuan Ge.  Synergic adversarial label learning for grading retinal diseases via knowledge distillation and multi-task learning.  IEEE JBHI, 100:1–14, 2020.
  * [40] Lie Ju, Xin Wang, Xin Zhao, Huimin Lu, Dwarikanath Mahapatra, and Zongyuan Ge.  Relational subsets knowledge distillation for long-tailed retinal diseases recognition.  In In MICCAI 2021, pages 1–11, 2021.
  * [41] Kaggle and EyePacs.  Kaggle diabetic retinopathy detection. https://www.kaggle.com/c/diabetic-retinopathy-detection/data, jul 2015.
  * [42] D. Karimi, G. Nir, L. Fazli, P.C. Black, L. Goldenberg, and S.E. Salcudean.  Deep learning-based gleason grading of prostate cancer from histopathology images-role of multiscale decision aggregation and data augmentation.  IEEE J Biomed Health Inform., 24(5):1413–1426, 2020.
  * [43] Rohit Keshari, Richa Singh, and Mayank Vatsa.  Generalized zero-shot learning via over-complete distribution.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 13300–13308, June 2020.
  * [44] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In arXiv preprint arXiv:1412.6980,, 2014.
  * [45] Avinash Kori and Ganapathi Krishnamurthi.  Zero shot learning for multi-modal real time image registration.  In arXiv preprint arXiv:1908.06213, 2019.
  * [46] S. Kuanar, V. Athitsos, D. Mahapatra, and A. Rajan.  Multi-scale deep learning architecture for nucleus detection in renal cell carcinoma microscopy image.  In arXiv preprint arXiv:2104.13557, 2021.
  * [47] S. Kuanar, V Athitsos, D. Mahapatra, K.R. Rao, Z. Akhtar, and D. Dasgupta.  Low dose abdominal ct image reconstruction: An unsupervised learning based approach.  In In Proc. IEEE ICIP, pages 1351–1355, 2019.
  * [48] S. Kuanar, D. Mahapatra, V. Athitsos, and K.R. Rao.  Gated fusion network for sao filter and inter frame prediction in versatile video coding.  In arXiv preprint arXiv:2105.12229, 2021.
  * [49] Shiba Kuanar, Dwarikanath Mahapatra, Monalisa Bilas, and KR Rao.  Multi-path dilated convolution network for haze and glow removal in night time images.  The Visual Computer, 38(3):1121–1134, 2022.
  * [50] S. Kuanar, K.R. Rao, D. Mahapatra, and M. Bilas.  Night time haze and glow removal using deep dilated convolutional network.  In arXiv preprint arXiv:1902.00855, 2019.
  * [51] H. Kuang, B. Guthier, M. Saini, D. Mahapatra, and A. El Saddik.  A real-time smart assistant for video surveillance through handheld devices.  In In Proc: ACM Intl. Conf. Multimedia, pages 917–920, 2014.
  * [52] Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling.  Attribute-based classification for zero-shot visual object categorization.  IEEE Trans. Pattern Analysis Machine Intelligence, 36(3):453–465, 2013.
  * [53] Kai Li, Martin Renqiang Min, and Yun Fu.  Rethinking zero-shot learning: A conditional visual classification perspective.  In CVPR, pages 3583–3592, 2019.
  * [54] Z. Li, D. Mahapatra, J.Tielbeek, J. Stoker, L. van Vliet, and F.M. Vos.  Image registration based on autocorrelation of local structure.  IEEE Trans. Med. Imaging, 35(1):63–75, 2016.
  * [55] Yun Liu, Krishna Gadepalli, Mohammad Norouzi, George E. Dahl, Timo Kohlberger, Aleksey Boyko, Subhashini Venugopalan, Aleksei Timofeev, Philip Q. Nelson, Greg S. Corrado, Jason D. Hipp, Lily Peng, and Martin C. Stumpe.  Detecting cancer metastases on gigapixel pathology images.  In arXiv preprint arXiv:1703.02442, 2017.
  * [56] Ming Y. Lu, Richard J. Chen, Jingwen Wang, Debora Dillon, and Faisal Mahmood.  Semi-supervised histology classification using deep multiple instance learning and contrastive predictive coding.  In arXiv:1910.10825, 2019.
  * [57] D. Mahapatra.  Elastic registration of cardiac perfusion images using saliency information.  Sequence and Genome Analysis – Methods and Applications, pages 351–364, 2011.
  * [58] D. Mahapatra.  Neonatal brain mri skull stripping using graph cuts and shape priors.  In In Proc: MICCAI workshop on Image Analysis of Human Brain Development (IAHBD), 2011.
  * [59] Dwarikanath Mahapatra.  Registration and segmentation methodology for perfusion mr images: Application to cardiac and renal images.  -, pages –, 2011.
  * [60] Dwarikanath Mahapatra.  Registration and segmentation methodology for perfusion mr images: Application to cardiac and renal images.  -, pages –, 2011.
  * [61] D. Mahapatra.  Cardiac lv and rv segmentation using mutual context information.  In Proc. MICCAI-MLMI, pages 201–209, 2012.
  * [62] D. Mahapatra.  Groupwise registration of dynamic cardiac perfusion images using temporal information and segmentation information.  In In Proc: SPIE Medical Imaging, 2012.
  * [63] D. Mahapatra.  Landmark detection in cardiac mri using learned local image statistics.  In Proc. MICCAI-Statistical Atlases and Computational Models of the Heart. Imaging and Modelling Challenges (STACOM), pages 115–124, 2012.
  * [64] D. Mahapatra.  Skull stripping of neonatal brain mri: Using prior shape information with graphcuts.  J. Digit. Imaging, 25(6):802–814, 2012.
  * [65] D. Mahapatra.  Cardiac image segmentation from cine cardiac mri using graph cuts and shape priors.  J. Digit. Imaging, 26(4):721–730, 2013.
  * [66] D. Mahapatra.  Cardiac mri segmentation using mutual context information from left and right ventricle.  J. Digit. Imaging, 26(5):898–908, 2013.
  * [67] D. Mahapatra.  Graph cut based automatic prostate segmentation using learned semantic information.  In Proc. IEEE ISBI, pages 1304–1307, 2013.
  * [68] D. Mahapatra.  Joint segmentation and groupwise registration of cardiac perfusion images using temporal information.  J. Digit. Imaging, 26(2):173–182, 2013.
  * [69] D. Mahapatra.  An automated approach to cardiac rv segmentation from mri using learned semantic information and graph cuts.  J. Digit. Imaging., 27(6):794–804, 2014.
  * [70] D. Mahapatra.  Combining multiple expert annotations using semi-supervised learning and graph cuts for medical image segmentation.  Computer Vision and Image Understanding, 151(1):114–123, 2016.
  * [71] D. Mahapatra.  Retinal image quality classification using neurobiological models of the human visual system.  In In Proc. MICCAI-OMIA, pages 1–8, 2016.
  * [72] D. Mahapatra.  Consensus based medical image segmentation using semi-supervised learning and graph cuts.  In arXiv preprint arXiv:1612.02166, 2017.
  * [73] D. Mahapatra.  Semi-supervised learning and graph cuts for consensus based medical image segmentation.  Pattern Recognition, 63(1):700–709, 2017.
  * [74] D. Mahapatra.  Amd severity prediction and explainability using image registration and deep embedded clustering.  In arXiv preprint arXiv:1907.03075, 2019.
  * [75] D. Mahapatra.  Generative adversarial networks and domain adaptation for training data independent image registration.  In arXiv preprint arXiv:1910.08593, 2019.
  * [76] D. Mahapatra.  Registration of histopathogy images using structural information from fine grained feature maps.  In arXiv preprint arXiv:2007.02078, 2020.
  * [77] D. Mahapatra.  Interpretability-driven sample selection using self supervised learning for disease classification and segmentation.  In arXiv preprint arXiv:2104.06087, 2021.
  * [78] D. Mahapatra.  Learning of inter-label geometric relationships using self-supervised learning: Application to gleason grade segmentation.  In arXiv preprint arXiv:2110.00404, 2021.
  * [79] Dwarikanath Mahapatra.  Multimodal generalized zero shot learning for gleason grading using self-supervised learning.  In arXiv preprint arXiv:2111.07646, 2021.
  * [80] D. Mahapatra, K. Agarwal, R. Khosrowabadi, and D.K. Prasad.  Recent advances in statistical data and signal analysis: Application to real world diagnostics from medical and biological signals.  In Computational and mathematical methods in medicine, 2016.
  * [81] D. Mahapatra, B. Antony, S. Sedai, and R. Garnavi.  Deformable medical image registration using generative adversarial networks.  In In Proc. IEEE ISBI, pages 1449–1453, 2018.
  * [82] D. Mahapatra and B. Bozorgtabar.  Retinal vasculature segmentation using local saliency maps and generative adversarial networks for image super resolution.  In arXiv preprint arXiv:1710.04783, 2017.
  * [83] D. Mahapatra and B. Bozorgtabar.  Progressive generative adversarial networks for medical image super resolution.  In arXiv preprint arXiv:1902.02144, 2019.
  * [84] D. Mahapatra, B. Bozorgtabar, and R. Garnavi.  Image super-resolution using progressive generative adversarial networks for medical image analysis.  Computerized Medical Imaging and Graphics, 71:30–39, 2019.
  * [85] D. Mahapatra, B. Bozorgtabar, and Zongyuan Ge.  Medical image classification using generalized zero shot learning.  In In IEEE CVAMD 2021, pages 3344–3353, 2021.
  * [86] D. Mahapatra, B. Bozorgtabar, S. Kuanar, and Zongyuan Ge.  Self-supervised multimodal generalized zero shot learning for gleason grading.  In In MICCAI-DART 2021, pages 1–11, 2021.
  * [87] D. Mahapatra, B. Bozorgtabar, and L. Shao.  Pathological retinal region segmentation from oct images using geometric relation based augmentation.  In In Proc. IEEE CVPR, pages 9611–9620, 2020.
  * [88] D. Mahapatra, B. Bozorgtabar, J.-P. Thiran, and L. Shao.  Pathological retinal region segmentation from oct images using geometric relation based augmentation.  In arXiv preprint arXiv:2003.14119, 2020.
  * [89] D. Mahapatra, B. Bozorgtabar, J.-P. Thiran, and L. Shao.  Structure preserving stain normalization of histopathology images using self supervised semantic guidance.  In In Proc. MICCAI, pages 309–319, 2020.
  * [90] D. Mahapatra, B. Bozorgtabar, J.-P. Thiran, and L. Shao.  Structure preserving stain normalization of histopathology images using self supervised semantic guidance.  In arXiv preprint arXiv:2008.02101, 2020.
  * [91] D. Mahapatra, S. Bozorgtabar, S. Hewavitahranage, and R. Garnavi.  Image super resolution using generative adversarial networks and local saliencymaps for retinal image analysis,.  In In Proc. MICCAI, pages 382–390, 2017.
  * [92] D. Mahapatra, S. Bozorgtabar, J.-P. Thiran, and M. Reyes.  Efficient active learning for image classification and segmentation using a sample selection and conditional generative adversarial network.  In In Proc. MICCAI (2), pages 580–588, 2018.
  * [93] D. Mahapatra and J.M. Buhmann.  Automatic cardiac rv segmentation using semantic information with graph cuts.  In Proc. IEEE ISBI, pages 1094–1097, 2013.
  * [94] D. Mahapatra and J.M. Buhmann.  Analyzing training information from random forests for improved image segmentation.  IEEE Trans. Imag. Proc., 23(4):1504–1512, 2014.
  * [95] D. Mahapatra and J.M. Buhmann.  Prostate mri segmentation using learned semantic knowledge and graph cuts.  IEEE Trans. Biomed. Engg., 61(3):756–764, 2014.
  * [96] D. Mahapatra and J.M. Buhmann.  A field of experts model for optic cup and disc segmentation from retinal fundus images.  In In Proc. IEEE ISBI, pages 218–221, 2015.
  * [97] D. Mahapatra and J. Buhmann.  Obtaining consensus annotations for retinal image segmentation using random forest and graph cuts.  In In Proc. OMIA, pages 41–48, 2015.
  * [98] D. Mahapatra and J. Buhmann.  Visual saliency based active learning for prostate mri segmentation.  In In Proc. MLMI, pages 9–16, 2015.
  * [99] D. Mahapatra and J. Buhmann.  Visual saliency-based active learning for prostate magnetic resonance imaging segmentation.  SPIE Journal of Medical Imaging, 3(1):014003, 2016.
  * [100] D. Mahapatra, R Garnavi, P.K. Roy, and R.B. Tennakoon.  System and method to teach and evaluate image grading performance using prior learned expert knowledge base.  In US Patent App. 15/459,457, 2018.
  * [101] D. Mahapatra, R Garnavi, P.K. Roy, and R.B. Tennakoon.  System and method to teach and evaluate image grading performance using prior learned expert knowledge base.  In US Patent App. 15/814,590, 2018.
  * [102] D. Mahapatra, R Garnavi, S. Sedai, and P.K. Roy.  Joint segmentation and characteristics estimation in medical images.  In US Patent App. 15/234,426, 2017.
  * [103] D. Mahapatra, R Garnavi, S. Sedai, and P.K. Roy.  Retinal image quality assessment, error identification and automatic quality correction.  In US Patent 9,779,492, 2017.
  * [104] D. Mahapatra, R Garnavi, S. Sedai, and R.B. Tennakoon.  Classification of severity of pathological condition using hybrid image representation.  In US Patent App. 15/426,634, 2018.
  * [105] D. Mahapatra, R Garnavi, S. Sedai, and R.B. Tennakoon.  Generating an enriched knowledge base from annotated images.  In US Patent App. 15/429,735, 2018.
  * [106] D. Mahapatra, R Garnavi, S. Sedai, R.B. Tennakoon, and R. Chakravorty.  Early prediction of age related macular degeneration by image reconstruction.  In US Patent App. 15/854,984, 2018.
  * [107] D. Mahapatra, R Garnavi, S. Sedai, R.B. Tennakoon, and R. Chakravorty.  Early prediction of age related macular degeneration by image reconstruction.  In US Patent 9,943,225, 2018.
  * [108] D. Mahapatra and Z. Ge.  Combining transfer learning and segmentation information with gans for training data independent image registration.  In arXiv preprint arXiv:1903.10139, 2019.
  * [109] D. Mahapatra and Z. Ge.  Training data independent image registration with gans using transfer learning and segmentation information.  In In Proc. IEEE ISBI, pages 709–713, 2019.
  * [110] D. Mahapatra and Z. Ge.  Training data independent image registration using generative adversarial networks and domain adaptation.  Pattern Recognition, 100:1–14, 2020.
  * [111] Dwarikanath Mahapatra and Zongyuan Ge.  Mr image super resolution by combining feature disentanglement cnns and vision transformers.  In -, 2022.
  * [112] Dwarikanath Mahapatra, Zongyuan Ge, and Mauricio Reyes.  Self-supervised generalized zero shot learning for medical image classification using novel interpretable saliency maps.  IEEE Transactions on Medical Imaging, pages 1–1, 2022.
  * [113] D. Mahapatra, Z. Ge, and S. Sedai.  Joint registration and segmentation of images using deep learning.  In US Patent App. 16/001,566, 2019.
  * [114] D. Mahapatra, Z. Ge, S. Sedai, and R. Chakravorty.  Joint registration and segmentation of xray images using generative adversarial networks.  In In Proc. MICCAI-MLMI, pages 73–80, 2018.
  * [115] D. Mahapatra, S. Gilani, and M.K. Saini.  Coherency based spatio-temporal saliency detection for video object segmentation.  IEEE Journal of Selected Topics in Signal Processing., 8(3):454–462, 2014.
  * [116] D. Mahapatra, J.Tielbeek, J.C. Makanyanga, J. Stoker, S.A. Taylor, F.M. Vos, and J.M. Buhmann.  Automatic detection and segmentation of crohn’s disease tissues from abdominal mri.  IEEE Trans. Med. Imaging, 32(12):1232–1248, 2013.
  * [117] D. Mahapatra, J.Tielbeek, J.C. Makanyanga, J. Stoker, S.A. Taylor, F.M. Vos, and J.M. Buhmann.  Active learning based segmentation of crohn’s disease using principles of visual saliency.  In Proc. IEEE ISBI, pages 226–229, 2014.
  * [118] D. Mahapatra, J.Tielbeek, J.C. Makanyanga, J. Stoker, S.A. Taylor, F.M. Vos, and J.M. Buhmann.  Combining multiple expert annotations using semi-supervised learning and graph cuts for crohn’s disease segmentation.  In In Proc: MICCAI-ABD, 2014.
  * [119] D. Mahapatra, J.Tielbeek, F.M. Vos, and J.M. Buhmann.  A supervised learning approach for crohn’s disease detection using higher order image statistics and a novel shape asymmetry measure.  J. Digit. Imaging, 26(5):920–931, 2013.
  * [120] Dwarikanath Mahapatra, Steven Korevaar, and Ruwan Tennakoon.  Gcn based unsupervised domain adaptation with feature disentanglement for medical image classification.  In -, 2022.
  * [121] D. Mahapatra, S. Kuanar, B. Bozorgtabar, and Zongyuan Ge.  Self-supervised learning of inter-label geometric relationships for gleason grade segmentation.  In In MICCAI-DART 2021, pages 57–67, 2021.
  * [122] D. Mahapatra, Z. Li, F.M. Vos, and J.M. Buhmann.  Joint segmentation and groupwise registration of cardiac dce mri using sparse data representations.  In In Proc. IEEE ISBI, pages 1312–1315, 2015.
  * [123] Dwarikanath Mahapatra, Alexander Poellinger, Ling Shao, and Mauricio Reyes.  Interpretability-driven sample selection using self supervised learning for disease classification and segmentation.  IEEE TMI, pages 1–15, 2021.
  * [124] D. Mahapatra, A. Routray, and C. Mishra.  An active snake model for classification of extreme emotions.  In IEEE International Conference on Industrial Technology (ICIT), pages 2195–2199, 2006.
  * [125] D. Mahapatra, P.K. Roy, S. Sedai, and R. Garnavi.  A cnn based neurobiology inspired approach for retinal image quality assessment.  In In Proc. EMBC, pages 1304–1307, 2016.
  * [126] D. Mahapatra, P.K. Roy, S. Sedai, and R. Garnavi.  Retinal image quality classification using saliency maps and cnns.  In In Proc. MICCAI-MLMI, pages 172–179, 2016.
  * [127] D. Mahapatra, S. Roy, and Y. Sun.  Retrieval of mr kidney images by incorporating shape information in histogram of low level features.  In In 13th International Conference on Biomedical Engineering, pages 661–664, 2009.
  * [128] D. Mahapatra, S. Saha, A. Vishwanath, and P.R. Bastide.  Generating hyperspectral image database by machine learning and mapping of color images to hyperspectral domain.  In US Patent App. 15/949,528, 2019.
  * [129] D. Mahapatra and M. Saini.  A particle filter framework for object tracking using visual-saliency information.  Intelligent Multimedia Surveillance, pages 133–147, 2013.
  * [130] D. Mahapatra, M.K. Saini, and Y. Sun.  Illumination invariant tracking in office environments using neurobiology-saliency based particle filter.  In IEEE ICME, pages 953–956, 2008.
  * [131] D. Mahapatra, P. Sch$\ddot{u}$ffler, J. Tielbeek, F.M. Vos, and J.M. Buhmann.  Semi-supervised and active learning for automatic segmentation of crohn’s disease.  In Proc. MICCAI, Part 2, pages 214–221, 2013.
  * [132] D. Mahapatra, S. Sedai, and R. Garnavi.  Elastic registration of medical images with gans.  In arXiv preprint arXiv:1805.02369, 2018.
  * [133] D. Mahapatra, S. Sedai, and K. Halupka.  Uncertainty region based image enhancement.  In US Patent App. 10,832,074, 2020.
  * [134] D. Mahapatra and A. Singh.  Ct image synthesis using weakly supervised segmentation and geometric inter-label relations for covid image analysis.  In arXiv preprint arXiv:2106.10230, 2021.
  * [135] D. Mahapatra and Y. Sun.  Nonrigid registration of dynamic renal MR images using a saliency based MRF model.  In Proc. MICCAI, pages 771–779, 2008.
  * [136] D. Mahapatra and Y. Sun.  Registration of dynamic renal mr images using neurobiological model of saliency.  In Proc. ISBI, pages 1119–1122, 2008.
  * [137] D. Mahapatra and Y. Sun.  Using saliency features for graphcut segmentation of perfusion kidney images.  In In 13th International Conference on Biomedical Engineering, pages 639–642, 2009.
  * [138] D. Mahapatra and Y. Sun.  Joint registration and segmentation of dynamic cardiac perfusion images using mrfs.  In Proc. MICCAI, pages 493–501, 2010.
  * [139] D. Mahapatra and Y. Sun.  Mrf based joint registration and segmentation of dynamic renal mr images.  In Second International Conference on Digital Image Processing, volume 7546, pages 285–290, 2010.
  * [140] D. Mahapatra and Y. Sun.  An mrf framework for joint registration and segmentation of natural and perfusion images.  In Proc. IEEE ICIP, pages 1709–1712, 2010.
  * [141] D. Mahapatra and Y. Sun.  Retrieval of perfusion images using cosegmentation and shape context information.  In Proc. APSIPA Annual Summit and Conference (ASC), volume 35, 2010\.
  * [142] D. Mahapatra and Y. Sun.  Rigid registration of renal perfusion images using a neurobiology based visual saliency model.  EURASIP Journal on Image and Video Processing., pages 1–16, 2010\.
  * [143] D. Mahapatra and Y. Sun.  Mrf based intensity invariant elastic registration of cardiac perfusion images using saliency information.  IEEE Trans. Biomed. Engg., 58(4):991–1000, 2011.
  * [144] D. Mahapatra and Y. Sun.  Orientation histograms as shape priors for left ventricle segmentation using graph cuts.  In In Proc: MICCAI, pages 420–427, 2011.
  * [145] D. Mahapatra and Y. Sun.  Integrating segmentation information for improved mrf-based elastic image registration.  IEEE Trans. Imag. Proc., 21(1):170–183, 2012.
  * [146] D. Mahapatra, J. Tielbeek, J.M. Buhmann, and F.M. Vos.  A supervised learning based approach to detect crohn’s disease in abdominal mr volumes.  In Proc. MICCAI workshop Computational and Clinical Applications in Abdominal Imaging(MICCAI-ABD), pages 97–106, 2012.
  * [147] D. Mahapatra, J. Tielbeek, F.M. Vos, and J.M. Buhmann .  Crohn’s disease tissue segmentation from abdominal mri using semantic information and graph cuts.  In Proc. IEEE ISBI, pages 358–361, 2013.
  * [148] D. Mahapatra, J. Tielbeek, F.M. Vos, and J.M. Buhmann.  Localizing and segmenting crohn’s disease affected regions in abdominal mri using novel context features.  In Proc. SPIE Medical Imaging, 2013.
  * [149] D. Mahapatra, J. Tielbeek, F.M. Vos, and J.M. Buhmann.  Weakly supervised semantic segmentation of crohn’s disease tissues from abdominal mri.  In Proc. IEEE ISBI, pages 832–835, 2013.
  * [150] D. Mahapatra, F.M. Vos, and J.M. Buhmann.  Crohn’s disease segmentation from mri using learned image priors.  In In Proc. IEEE ISBI, pages 625–628, 2015.
  * [151] D. Mahapatra, F.M. Vos, and J.M. Buhmann.  Active learning based segmentation of crohns disease from abdominal mri.  Computer Methods and Programs in Biomedicine, 128(1):75–85, 2016\.
  * [152] D. Mahapatra, S. Winkler, and S.C. Yen.  Motion saliency outweighs other low-level features while watching videos.  In SPIE HVEI., pages 1–10, 2008.
  * [153] Devraj Mandal, Sanath Narayan, Sai Kumar Dwivedi, Vikram Gupta, Shuaib Ahmed, Fahad Shahbaz Khan, and Ling Shao.  Out-of-distribution detection for generalized zero-shot action recognition.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9977–9985, 2019.
  * [154] Quinn McNemar.  Note on the sampling error of the difference between correlated proportions or percentages.  Psychometrika, 12(2):153–157, 1947.
  * [155] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  In In Proc. ICLR Workshops, 2013.
  * [156] Shaobo Min, Hantao Yao, Hongtao Xie, Chaoqun Wang, Zheng-Jun Zha, and Yongdong Zhang.  Domain-aware visual bias eliminating for generalized zero-shot learning.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 12664–12673, June 2020.
  * [157] M. Nilsback and A. Zisserman.  Automated flower classification over a large number of classes.  In 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pages 722–729, 2008.
  * [158] Abhineet Pandey, Bhawna Paliwal, Abhinav Dhall, Ramanathan Subramanian, and Dwarikanath Mahapatra.  This explains that: Congruent image–report generation for explainable medical image analysis with cyclic generative adversarial networks.  In In MICCAI-iMIMIC 2021, pages 1–11, 2021.
  * [159] Genevieve Patterson and James Hays.  Sun attribute database: Discovering, annotating, and recognizing scene attributes.  In Proceeding of the 25th Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
  * [160] Akanksha Paul, Narayanan C Krishnan, and Prateek Munjal.  Semantically aligned bias reducing zero shot learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7056–7065, 2019.
  * [161] Hieu H. Pham, Tung T. Le, Dat Q. Tran, Dat T. Ngo, and Ha Q. Nguyen.  Interpreting chest x-rays via cnns that exploit hierarchical disease dependencies and uncertainty labels.  In arXiv preprint arXiv:1911.06475,, 2020.
  * [162] P. Rajpurkar, J. Irvin, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C. Langlotz, K. Shpanskaya, M. P Lungren, and A.Y Ng.  Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning.  In arXiv preprint arXiv:1711.05225,, 2017.
  * [163] P. Roy, R. Chakravorty, S. Sedai, D. Mahapatra, and R. Garnavi.  Automatic eye type detection in retinal fundus image using fusion of transfer learning and anatomical features.  In In Proc. DICTA, pages 1–7, 2016.
  * [164] P.K. Roy, D. Mahapatra, R. Garnavi, and R.B. Tennakoon.  System and method to teach and evaluate image grading performance using prior learned expert knowledge base.  In US Patent App. 10,984,674, 2021.
  * [165] P. Roy, R. Tennakoon, K. Cao, S. Sedai, D. Mahapatra, S. Maetschke, and R. Garnavi.  A novel hybrid approach for severity assessment of diabetic retinopathy in colour fundus images,.  In In Proc. IEEE ISBI, pages 1078–1082, 2017.
  * [166] M. Saini, B. Guthier, H. Kuang, D. Mahapatra, and A.E. Saddik.  szoom: A framework for automatic zoom into high resolution surveillance videos.  In arXiv preprint arXiv:1909.10164, 2019.
  * [167] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko.  Universal domain adaptation through self supervision.  In In Proc. NeurIPS, 2020.
  * [168] P. Sch$\ddot{u}$ffler, D. Mahapatra, J. Tielbeek, F.M. Vos, J. Makanyanga, D.A. Pends, C.Y. Nio, J. Stoker, S.A. Taylor, and J.M. Buhmann.  A model development pipeline for crohns disease severity assessment from magnetic resonance images.  In In Proc: MICCAI-ABD, 2013.
  * [169] P. Sch$\ddot{u}$ffler, D. Mahapatra, J. Tielbeek, F.M. Vos, J. Makanyanga, D.A. Pends, C.Y. Nio, J. Stoker, S.A. Taylor, and J.M. Buhmann.  Semi automatic crohns disease severity assessment on mr imaging.  In In Proc: MICCAI-ABD, 2014.
  * [170] Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata.  Generalized zero-and few-shot learning via aligned variational autoencoders.  In In Proc. IEEE CVPR, pages 8247–8255, 2019.
  * [171] Peter J Schüffler, Dwarikanath Mahapatra, Franciscus M Vos, and Joachim M Buhmann.  Computer aided crohn’s disease severity assessment in mri.  In VIGOR++ Workshop 2014-Showcase of Research Outcomes and Future Outlook, pages –, 2014.
  * [172] S. Sedai, D. Mahapatra, B. Antony, and R. Garnavi.  Joint segmentation and uncertainty visualization of retinal layers in optical coherence tomography images using bayesian deep learning.  In In Proc. MICCAI-OMIA, pages 219–227, 2018.
  * [173] S. Sedai, D. Mahapatra, Z. Ge, R. Chakravorty, and R. Garnavi.  Deep multiscale convolutional feature learning for weakly supervised localization of chest pathologies in x-ray images.  In In Proc. MICCAI-MLMI, pages 267–275, 2018.
  * [174] S. Sedai, D. Mahapatra, S. Hewavitharanage, S. Maetschke, and R. Garnavi.  Semi-supervised segmentation of optic cup in retinal fundus images using variational autoencoder,.  In In Proc. MICCAI, pages 75–82, 2017.
  * [175] S. Sedai, P.K. Roy, D. Mahapatra, and R. Garnavi.  Segmentation of optic disc and optic cup in retinal fundus images using shape regression.  In In Proc. EMBC, pages 3260–3264, 2016.
  * [176] S. Sedai, P.K. Roy, D. Mahapatra, and R. Garnavi.  Segmentation of optic disc and optic cup in retinal images using coupled shape regression.  In In Proc. MICCAI-OMIA, pages 1–8, 2016.
  * [177] Shikhar Srivastava, Mohammad Yaqub, Karthik Nandakumar, Zongyuan Ge, and Dwarikanath Mahapatra.  Continual domain incremental learning for chest x-ray classification in low-resource clinical settings.  In In MICCAI-FAIR 2021, pages 1–11, 2021.
  * [178] Hidenori Takahashi, Hironobu Tampo, Yusuke Arai, Yuji Inoue, and Hidetoshi Kawashima.  Applying artificial intelligence to disease staging: Deep learning for improved staging of diabetic retinopathy.  Plos One, 12(6), 2017.
  * [179] R. Tennakoon, D. Mahapatra, P. Roy, S. Sedai, and R. Garnavi.  Image quality classification for dr screening using convolutional neural networks.  In In Proc. MICCAI-OMIA, pages 113–120, 2016.
  * [180] J. Tong, D. Mahapatra, P. Bonnington, T. Drummond, and Z. Ge.  Registration of histopathology images using self supervised fine grained feature maps.  In In Proc. MICCAI-DART Workshop, pages 41–51, 2020.
  * [181] Ruchika Verma, Neeraj Kumar, Abhijeet Patil, Nikhil Cherian Kurian, Swapnil Rane, Simon Graham, Quoc Dang Vu, Mieke Zwager, Shan E. Ahmed Raza, Nasir Rajpoot, Xiyi Wu, Huai Chen, Yijie Huang, Lisheng Wang, Hyun Jung, G. Thomas Brown, Yanling Liu, Shuolin Liu, Seyed Alireza Fatemi Jahromi, Ali Asghar Khani, Ehsan Montahaei, Mahdieh Soleymani Baghshah, Hamid Behroozi, Pavel Semkin, Alexandr Rassadin, Prasad Dutande, Romil Lodaya, Ujjwal Baid, Bhakti Baheti, Sanjay Talbar, Amirreza Mahbod, Rupert Ecker, Isabella Ellinger, Zhipeng Luo, Bin Dong, Zhengyu Xu, Yuehan Yao, Shuai Lv, Ming Feng, Kele Xu, Hasib Zunair, Abdessamad Ben Hamza, Steven Smiley, Tang-Kai Yin, Qi-Rui Fang, Shikhar Srivastava, Dwarikanath Mahapatra, Lubomira Trnavska, Hanyun Zhang, Priya Lakshmi Narayanan, Justin Law, Yinyin Yuan, Abhiroop Tejomay, Aditya Mitkari, Dinesh Koka, Vikas Ramachandra, Lata Kini, and Amit Sethi.  Monusac2020: A multi-organ nuclei segmentation and classification challenge.  IEEE Transactions on Medical Imaging, 40(12):3413–3423, 2021.
  * [182] Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, and Piyush Rai.  Generalized zero-shot learning via synthesized examples.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4281–4289, 2018.
  * [183] F. M. Vos, J. Tielbeek, R. Naziroglu, Z. Li, P. Sch$\ddot{u}$ffler, D. Mahapatra, Alexander Wiebel, C. Lavini, J. Buhmann, H. Hege, J. Stoker, and L. van Vliet.  Computational modeling for assessment of IBD: to be or not to be?  In Proc. IEEE EMBC, pages 3974–3977, 2012.
  * [184] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [185] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R.M. Summers.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In In Proc. CVPR, 2017.
  * [186] Jiamin Wu, Tianzhu Zhang, Zheng-Jun Zha, Jiebo Luo, Yongdong Zhang, and Feng Wu.  Self-supervised domain-aware generative network for generalized zero-shot learning.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 12767–12776, June 2020.
  * [187] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3733–3742, 2018.
  * [188] Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata.  Zero-shot learning a comprehensive evaluation of the good, the bad and the ugly.  IEEE Trans. Pattern Analysis Machine Intelligence, 41(9):2251–2265, 2018.
  * [189] Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata.  Feature generating networks for zero-shot learning.  In In Proc. IEEE CVPR, pages 5542–5551, 2018.
  * [190] Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata.  f-vaegan-d2: A feature generating framework for any-shot learning.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10275–10284, June 2019.
  * [191] Y. Xing, Z. Ge, R. Zeng, D. Mahapatra, J. Seah, M. Law, and T. Drummond.  Adversarial pulmonary pathology translation for pairwise chest x-ray data augmentation.  In In Proc. MICCAI, pages 757–765, 2019.
  * [192] J. Zilly, J. Buhmann, and D. Mahapatra.  Boosting convolutional filters with entropy sampling for optic cup and disc image segmentation from fundus images.  In In Proc. MLMI, pages 136–143, 2015.
  * [193] J. Zilly, J.M. Buhmann, and D. Mahapatra.  Glaucoma detection using entropy sampling and ensemble learning for automatic optic cup and disc segmentation.  In Press Computerized Medical Imaging and Graphics, 55(1):28–41, 2017.