# Domain-Adaptive Learning: Unsupervised Adaptation for Histology Images with
Improved Loss Function Combination

1st Ravi Kant Gupta  Department of Electrical Engineering
Indian Institute of Technology
Mumbai, India
<EMAIL_ADDRESS>2nd Shounak Das  Department of Electrical Engineering
Indian Institute of Technology
Mumbai, India
<EMAIL_ADDRESS>3rd Amit Sethi  Department of Electrical Engineering
Indian Institute of Technology
Mumbai, India
<EMAIL_ADDRESS>
###### Abstract

This paper presents a novel approach for unsupervised domain adaptation (UDA)
targeting H&E stained histology images. Existing adversarial domain adaptation
methods may not effectively align different domains of multimodal
distributions associated with classification problems. The objective is to
enhance domain alignment and reduce domain shifts between these domains by
leveraging their unique characteristics. Our approach proposes a novel loss
function along with carefully selected existing loss functions tailored to
address the challenges specific to histology images. This loss combination not
only makes the model accurate and robust but also faster in terms of training
convergence. We specifically focus on leveraging histology-specific features,
such as tissue structure and cell morphology, to enhance adaptation
performance in the histology domain. The proposed method is extensively
evaluated in accuracy, robustness, and generalization, surpassing state-of-
the-art techniques for histology images. We conducted extensive experiments on
the FHIST dataset and the results show that our proposed method - Domain
Adaptive Learning (DAL) significantly surpasses the ViT-based and CNN-based
SoTA methods by 1.41% and 6.56% respectively.

###### Index Terms:

Adversarial, Deep Learning, Domain Adaptation, Histology

##  I Introduction

In traditional supervised learning, a model is trained using labeled data from
the same domain as the test data. Obtaining labels for medical data is
challenging due to the intricacies of medical expertise, making it costly and
time-consuming. The need for specialized knowledge, meticulous review, and
ethical considerations contribute to the difficulty in acquiring accurate and
reliable annotations for medical datasets. However, when the distribution of
the source and target domains differs significantly, the model’s performance
may suffer due to the domain shift. This domain shift can be because of color
variation, data acquisition bias, distributional differences, domain-specific
factors, covariate shift, staining techniques in medical histology images,
etc. Unsupervised domain adaptation (UDA) techniques aim to mitigate this
domain shift by aligning the feature distributions or learning domain-
invariant representations by using only unlabeled samples from the target
domain. Adversarial-based UDA employs a domain adversarial training framework,
often based on Generative Adversarial Networks (GANs) [24] or domain
adversarial neural networks (DANN) [1]. By learning domain-invariant
representations, adversarial-based UDA models can effectively reduce the
domain discrepancy and improve the generalization performance on the target
domain. This approach has shown promising results in various domains, such as
image classification, object detection, and semantic segmentation. However,
while adversarial-based UDA has achieved notable success, challenges still
exist. These include addressing the sensitivity to hyper-parameter tuning,
handling the high-dimensional feature space, and effectively capturing complex
domain shifts.

To address the aforementioned challenge, we develop an unsupervised domain
adaptation approach that surpasses the state-of-the-art performance for
histological images. We present our findings from developing convolution
neural networks (CNNs) for such tasks based on CRCTP dataset (source) and NCT
(target) from FHIST dataset [47], which is composed of several histology
datasets, namely CRC-TP [32], LC25000 [33], BreakHis [34], and NCT-CRC-HE-100K
[35]. These histology datasets consist of different tissue types and different
organs. We consider each tissue type as a class label with one-hot encoding in
the classification task. We framed our experiments on CRCTP and NCT with six
classes (Benign, Tumor, Muscle, Stroma, Debris, and Inflammatory). The
t-distributed stochastic neighbor embedding (tSNE) [54] plot in Figure 1 of
source data distribution (circle shape) and target data distribution (square
shape) while the color of classes differs with light and dark versions of the
same color. The sample patches of each class from both domains are shown in
Figure 2.

Our research endeavors converge on three key objectives: firstly, to reduce
the discordance between source and target domains in histology images;
secondly, to harness the distinctive attributes of histology, such as cellular
morphology and tissue structure, to elevate adaptation performance
specifically within the histology domain; and finally, to transcend the
limitations of current UDA techniques, to achieve state-of-the-art accuracy,
resilience, and generalization capabilities compared to the previous methods.
This comprehensive approach is crafted to encompass and intricately tackle the
complexities presented by histology images.

Our adoption of deep learning for unsupervised domain adaptation in histology
images is driven by its potential to enhance model generalization, extract
optimal features, enable versatile cross-domain applications, and achieve
field-advancing progress. By tailoring the combination of loss functions which
leads to improved convergence and robustness, and with the leverage of deep
learning’s power, we aim to surpass current methods, benefiting various
applications. Inspired by a conditional domain adversarial network (CDAN) [3],
the core idea is to simultaneously train a feature extractor (typically a deep
neural network) and a domain classifier (discriminator) to distinguish between
source and target domains. We have examined different CNN-based feature
extractor as ResNet-50 [2], ResNet-101 [2], ResNet-152 [2], VIT [44], and
ConvMixer [46] to extract meaningful features. The feature extractor aims to
learn domain-invariant representations, while the domain classifier tries to
classify the domain of the extracted features correctly. During training, the
feature extractor and domain classifier are optimized in an adversarial
manner. The feature extractor aims to fool the domain classifier by generating
indistinguishable features across domains, while the domain classifier tries
to classify the domains correctly. This adversarial training process
encourages the feature extractor to learn domain-invariant and transferable
representations between the source and target domains. To achieve this, we
propose a novel loss function pseudo label maximum mean discrepancy (PLMMD)
along with different existing losses such as maximum information loss (entropy
loss) [41, 42], maximum mean discrepancy (MMD) loss [8, 9, 10, 28], minimum
class confusion (MCC) loss [43], etc. This combination of loss functions has
the following specific advantages : Employing MCC loss enhances classification
models by minimizing class confusion, particularly in scenarios with
imbalanced class distributions. With maximum information loss, our model is
encouraged to learn tightly clustered target features with uniform
distribution, such that the discriminative information in the target domain is
retained, while MDD loss measures the difference between the mean embeddings
of two distributions, helping to quantify the dissimilarity between domains
and facilitating domain adaptation techniques. Our proposed loss PLMMD
enhances unsupervised domain adaptation by selectively emphasizing domain-
invariant features through weight assignments. The benefit of this novel loss
is, that training convergence is faster as compared to other scenarios. With
the help of this novel combination of the loss function our method surpasses
not only the CNN-based model state-of-art but also the transformer-based model
for the histology images. To justify our claims for histology images, we use
the FHIST dataset [47], whose sample images are shown in Figure 2.

Figure 1: Snapshot of t-SNE plot of source (CRC-TP) (Circle shape) and target
(NCT) (Square shape), clearly shows significant difference between source and
target data distribution. Figure 2: Snapshot of sample images of each class
from CRC-TP (top row) and NCT (bottom row) of FHIST dataset.

Our stated goals were achieved by proposing an improved combination of loss
functions tailored to address the unique challenges of H&E stained histology
images. The performance evaluation was focused on accuracy, robustness, and
generalization, to surpass state-of-the-art techniques in both domains.
Furthermore, the research explored potential cross-domain applications in
medical image analysis and computer vision, offering promising advancements in
practical unsupervised domain adaptation with the help of various combinations
of loss functions with different existing models.

##  II Background and Related Work

In unsupervised domain adaptation, we have a source domain
$D_{s}=\\{(x_{s_{i}},y_{s_{i}})\\}_{i=1}^{n_{s}}$ of ${n_{s}}$ labeled
examples and a target domain $D_{s}=\\{(x_{t_{i}},y_{t_{i}})\\}_{i=1}^{n_{t}}$
of ${n_{t}}$ unlabeled examples. The source domain and target domain are
sampled from joint distributions $P({x_{s}},{y_{s}})$ and $Q({x_{t}},{y_{t}})$
respectively. Notably, the two distributions are initially not aligned, that
is, $P\neq Q$.

Domain adversarial neural network (DANN) [1] is a framework of choice for UDA.
It is a two-player game between domain discriminator $D$, which is trained to
distinguish the source domain from the target domain, and the feature
representation $F$ trained to confuse the domain discriminator $D$ as well as
classify the source domain samples. The error function of the domain
discriminator corresponds well to the discrepancy between the feature
distributions $P(f)$ and $Q(f)$ [40], a key to bound the target risk in the
domain adaptation theory [55].

Alignment-based domain adaptation is another typical line of work that
leverages a domain-adversarial task to align the source and target domains as
a whole so that class labels can be transferred from the source domain to the
unlabeled target one [4, 5, 6, 7]. Another typical line of work directly
minimizes the domain shift measured by various metrics, e.g., maximum mean
discrepancy (MMD) [8, 9, 10]. These methods are based on domain-level domain
alignment. To achieve class-level domain alignment, the works of [11, 12]
utilize the multiplicative interaction of feature representations and class
predictions so that the domain discriminator can be aware of the
classification boundary. Based on the integrated task and domain classifier,
[13] encourages a mutually inhibitory relation between category and domain
predictions for any input instance. The works of [14, 15] align the labeled
source centroid and pseudo-labeled target centroid of each shared class in the
feature space. Some work uses individual task classifiers for the two domains
to detect non-discriminative features and reversely learn a discriminative
feature extractor [16, 17, 18]. Certain other works focus attention on
transferable regions to derive a domain-invariant classification model [19,
20, 21]. To help achieve target-discriminative features, [22, 23] generate
synthetic images from the raw input data of the two domains via GANs [24]. The
recent work of [25] improves adversarial feature adaptation, where the
discriminative structures of target data may be deteriorated [26]. The work of
[27] adapts the feature norms of the two domains to a large range of values so
that the learned features are not only task-discriminative but also domain-
invariant.

##  III Proposed Method

The challenge of domain shift in a cross-domain classification task using
unsupervised domain adaptation leverages the knowledge from a labeled source
domain to improve the performance of a classifier on an unlabeled target
domain. We propose a novel loss function that minimizes the domain discrepancy
and aligns feature distributions across domains. Our datasets even differ in
patch sizes - 150x150 pixels for the source domain and 224x224 for the target
domain. Before training, the patches were subjected to data augmentation such
as horizontal flip, vertical flip, and normalization to ensure consistency. To
facilitate domain adaptation, we introduce a structure-preserving colour
normalization technique to normalize the stain appearance of histopathology
images across domains [45]. The normalization process aims to preserve the
local structure while removing domain-specific variations. Therefore, the
patches were colour normalized [29, 30].

Figure 3: Architecture of the proposed networks, where domain-specific feature
representation $f$ and classifier prediction $g$ embody the cross-domain gap
to be reduced jointly by the conditional domain discriminator $D$.

From the color-normalized patches, we extracted features using ResNet152
trained on ImageNet [31]. Our proposed model architecture is based on a deep
neural network with convolutional and fully connected layers, specifically
tailored for domain adaptation.

In this work, we design a method to train a deep network $N:x\rightarrow y$
which reduces the shifts in the data distributions across domains, such that
the target risk $r_{t}$= $E_{(x_{t},y_{t})\sim Q}[N(x_{t})\neq y_{t}]$ can be
bounded by the source risk $r_{s}$= $E_{(x_{s},y_{s})\sim P}[N(x_{s})\neq
y_{s}]$ plus the distribution discrepancy disc(P, Q) quantified by a novel
conditional domain discriminator. To minimize domain cross-domain discrepancy
[1] in adversarial learning Generative Adversarial Networks (GANs) play a
vital role. Features are represented by $f=F(x)$ and classifier prediction,
$g=N(x)$ generated from deep network $N$.

We improve existing adversarial domain adaptation methods in two directions.
First, when the joint distributions of feature and class, i.e.
$P({x_{s}},{y_{s}})$ and $Q({x_{t}},{y_{t}})$, are non-identical across
domains, adapting only the feature representation $f$ may be insufficient. A
quantitative study [48] shows that deep representations eventually transition
from general to specific along deep networks, with transferability decreased
remarkably in the domain-specific feature layer $f$ and classifier layer $g$.
Second, due to the nature of multi-class classification, the feature
distribution is multimodal, and hence adapting feature distribution may be
challenging for adversarial networks.

By conditioning, domain variances in feature representation $f$ and classifier
prediction $g$ can be modeled simultaneously. This joint conditioning allows
us to bridge the domain gap more effectively, enabling the adapted model to
capture and align the underlying data distributions between the source and
target domains. Consequently, incorporating classifier prediction as a
conditioning factor in domain adaptation holds great potential for achieving
improved transferability and generating domain-invariant representations in
challenging cross-domain scenarios.

We formulate Conditional Domain Adversarial Network (CDAN) [3] as a minimax
optimization problem with two competitive error terms: (a) $E(N)$ on the
source classifier N, which is minimized to guarantee lower source risk; (b)
$E(D,N)$ on the source classifier $N$ and the domain discriminator $D$ across
the source and target domains, which is minimized over $D$ but maximized over
$f=F(x)$ and $g=N(x)$:

$L_{clc}(x_{s_{i}},y_{s_{i}})=\mathbb{E}_{(x_{s_{i}},y_{s_{i}})\sim
D_{s}}L(N(x_{s_{i}}),y_{s_{i}})$ (1)

$\displaystyle L_{dis}(x_{s},x_{t})=$ $\displaystyle-\mathbb{E}_{x_{s_{i}}\sim
D_{s}}\log[D(f_{s_{i}},g_{s_{i}})]$ (2)
$\displaystyle-\mathbb{E}_{x_{t_{j}}\sim
D_{t}}\log[1-D(f_{t_{j}},g_{t_{j}})],$

where $L$ is the cross-entropy loss, and $h=(f,g)$ is the joint variable of
feature representation $f$ and classifier prediction $g$. The minimax game of
CDAN is

$\displaystyle\begin{gathered}\min_{N}L_{clc}(x_{s_{i}},y_{s_{i}})-\lambda
L_{dis}(x_{s},x_{t})\\\ \min_{D}L_{dis}(x_{s},x_{t}),\end{gathered}$ (3)

where $\lambda$ is a hyper-parameter between the two objectives to trade off
source risk and domain adversary.

We condition domain discriminator $D$ on the classifier prediction g through
joint variable $h=(f,g)$ to potentially tackle the two aforementioned
challenges of adversarial domain adaptation. A simple conditioning of $D$ is
$D(f\oplus g)$, where we concatenate the feature representation and classifier
prediction in vector $f\oplus g$ and feed it to conditional domain
discriminator $D$. This conditioning strategy is widely adopted by existing
conditional GANs [24]. However, with the concatenation strategy, $f$ and $g$
are independent of each other, thus failing to fully capture multiplicative
interactions between feature representation and classifier prediction, which
are crucial to domain adaptation. As a result, the multimodal information
conveyed in classifier prediction cannot be fully exploited to match the
multimodal distributions of complex domains [49]. The multilinear map is
defined as the outer product of multiple random vectors. And the multilinear
map of infinite-dimensional nonlinear feature maps has been successfully
applied to embed joint distribution or conditional distribution into
reproducing kernel Hilbert spaces [49, 50, 51, 52]. Besides the theoretical
benefit of the multilinear map $x\otimes y$ over the concatenation $x\oplus y$
[49, 53]. Taking advantage of the multilinear map, in this paper, we condition
$D$ on $g$ with the multilinear map. Superior to concatenation, the
multilinear map $x\otimes y$ can fully capture the multimodal structures
behind complex data distributions. A disadvantage of the multilinear map is
dimension explosion.

We enable conditional adversarial domain adaptation over domain-specific
feature representation $f$ and classifier prediction $g$. We jointly minimize
with respect to (1) source classifier $N$ and feature extractor $F$, minimize
(2) domain discriminator $D$, and maximize (2) feature extractor $F$ and
source classifier $N$. This yields the mini-max problem of Domain Adversarial
Networks:

$\displaystyle\min_{G}\quad$
$\displaystyle\mathbb{E}_{(x_{s}^{i},y_{s}^{i})\sim
D_{s}}L(G(x_{s}^{i}),y_{s}^{i})$
$\displaystyle+\lambda\left(\mathbb{E}_{x_{s}^{i}\sim
D_{s}}\log[D(T(h_{s}^{i}))]\right.$
$\displaystyle\left.+\mathbb{E}_{x_{t}^{j}\sim
D_{t}}\log[1-D(T(h_{t}^{j}))]\right)$ (4) $\displaystyle\max_{D}\quad$
$\displaystyle\mathbb{E}_{x_{s}^{i}\sim
D_{s}}\log[D(T(h_{s}^{i}))]+\mathbb{E}_{x_{t}^{j}\sim
D_{t}}\log[1-D(T(h_{t}^{j}))],$ (5)

where $\lambda$ is a hyper-parameter between the source classifier and
conditional domain discriminator, and note that $h=(f,g)$ is the joint
variable of domain-specific feature representation $f$ and classifier
prediction $g$ for adversarial adaptation.

The general problem of adversarial domain adaptation of the proposed model for
classification can be formulated as follows:

$\displaystyle L=\min_{N}L_{clc}(x_{s_{i}},y_{s_{i}})-\lambda
L_{dis}(x_{s},x_{t})$ (6) $\displaystyle+\beta L_{IM}+\gamma L_{MCC}+\delta
L_{MDD}+\eta L_{WMMD}$

where $\lambda$, $\beta$, $\gamma$, $\delta$ and $\eta$ are hyper parameters,
$L_{MCC}$ is minimum class confusion loss, $L_{MDD}$ is maximum mean
discrepancy loss, $L_{WMDD}$ represents weighted maximum mean discrepancy loss
and $L_{IM}$ represents information maximization loss. All individual losses
have their own specialty and this novel combination of loss significantly
surpasses the performance of CNN-based models as well as transformer-based
models. A detailed description of all the losses is given below in the losses
section.

###  III-A Losses

####  III-A1  Maximum Mean Discrepancy

Maximum mean discrepancy (MMD) is a kernel-based statistical test used to
determine whether given two distributions are the same [8, 9, 10]. Given an
random variable X , a feature map $\phi$ maps X to an another space $F$ such
that $\phi(X)\in{F}$ . Assuming $F$ satisfies the necessary conditions, we can
benefit from the kernel trick to compute the inner product in $F$ :

$X,Y\text{ such that }k(X,Y)=\langle\phi(X),\phi(Y)\rangle_{F},$ (7)

where $k$ is gram matrix produced using the kernel function.
MMD is the distance between feature means. That means for a given probability
measure $P$ on $X$, feature means is an another feature map that takes
$\phi(X)$ and maps it to the means of every coordinate of $\phi(X)$ :

$\mu_{p}(\phi(X))=[\mathbb{E}[\phi(X_{1})],....,\mathbb{E}[\phi(X_{m})]]^{T}$
(8)

The inner product of feature means of $X\sim P$ and $Y\sim Q$ can be written
in terms of kernel function such that:

$\displaystyle\begin{gathered}\langle\mu_{p}(\phi(X)),\mu_{q}(\phi(Y))\rangle_{F}=\mathbb{E}_{P,Q}[\langle\phi(X),\phi(Y)\rangle_{F}]\\\
=\mathbb{E}_{P,Q}[k(X,Y)]\end{gathered}$ (9)

Given $X$, $Y$ maximum mean discrepancy is the distance between feature means
of $X$, $Y$ :

$MMD^{2}(P,Q)=||\mu_{P}-\mu_{Q}||^{2}_{F}$ (10)

$\displaystyle\begin{gathered}MMD^{2}(P,Q)=\langle\mu_{P}-\mu_{Q},\mu_{P}-\mu_{Q}\rangle\\\
=\langle\mu_{P},\mu_{P}\rangle-2\langle\mu_{P},\mu_{Q}\rangle+\langle\mu_{Q},\mu_{Q}\rangle\end{gathered}$
(11)

Using the equation (9), finally above expression becomes

$\displaystyle\begin{gathered}{L}_{MMD}=MMD^{2}(P,Q)\\\
=\mathbb{E}_{P}[k(X,X)]-2\mathbb{E}_{P,Q}[k(X,Y)]+\mathbb{E}_{Q}[k(Y,Y)]\end{gathered}$
(12)

####  III-A2 Pseudo Label Maximum Mean Discrepancy

We calculated the PLMMD using a similar procedure to calculating MMD loss in
equation (12). However, our proposed loss differs in terms of weights assigned
to each similarity term. Hence we can define PLMMD loss as:

$\displaystyle\begin{gathered}{L}_{PLMMD}=w_{XX}\mathbb{E}_{P}[k(X,X)]-2w_{XY}\mathbb{E}_{P,Q}[k(X,Y)]\\\
+w_{YY}\mathbb{E}_{Q}[k(Y,Y)],\end{gathered}$ (13)

where, $w_{XX}$ represent weight to get similarity within the source domain,
similarly, $w_{YY}$ are weights for similarity within the target domain, and
$w_{XY}$ are weights to get similarity within source and target domain. For
calculating the weights, first, we generated pseudo labels for the target
using a source classifier. After that, the source and target pseudo-label is
normalized to account for class imbalances. For each class common to both
datasets, dot products of normalized vectors are computed to quantify instance
relationships. Calculated dot products are normalized by the count of common
classes, ensuring fairness. This returns three weight arrays, representing
relationships between instances in the source dataset, target dataset, and
source-to-target pairs.

####  III-A3 Minimum Class Confusion

The minimum class confusion loss $\mathcal{L}_{MCC}$ [43] seeks to minimize
confusion terms between classes $j$ and $j^{\prime}$, such that $j\neq
j^{\prime}$ where the indices are exhaustive over the set of classes. On the
target domain, the class confusion term between two classes $j$ and
$j^{\prime}$ is given by:

$C_{jj^{\prime}}=\hat{\mathbf{y}}_{\cdot j}^{\intercal}\hat{\mathbf{y}}_{\cdot
j^{\prime}}^{\intercal}$

A much more nuanced and meaningful formulation of the class confusion would
be:

$C_{jj^{\prime}}=\hat{\mathbf{y}}_{\cdot
j}^{\intercal}\mathbf{W}\hat{\mathbf{y}}_{\cdot j^{\prime}}^{\intercal},$ (14)

where the matrix $\mathbf{W}$ is a diagonal matrix. The diagonal terms
$W_{ii}$ are given as the softmax outputs of the entropies in classifying a
sample $i$. $\hat{\mathbf{y}}_{ij}$ is given as:

$\hat{\mathbf{y}}_{ij}=\frac{\exp(Z_{ij}/T)}{\sum_{j^{\prime}=1}^{c}\exp(Z_{ij^{\prime}}/T)},$
(15)

where $c$ is the number of classes, $T$ is the temperature coefficient, and
$Z_{ij}$ is the logistic output of the classifier layer for the class $j$ and
the sample $i$.

After normalizing the class confusion terms, the final MCC Loss function is
given as:

$\mathcal{L}_{MCC}=\frac{1}{c}\sum_{j=1}^{c}\sum_{j^{\prime}\neq
j}^{c}|C_{jj^{\prime}}|,$ (16)

which is the sum of all the non-diagonal elements of the class confusion
matrix. The diagonal terms represent the ”certainty” in the classifier, while
the non-diagonal terms represent the ”uncertainty” in classification. The MCC
loss can be added in conjunction with other domain adaptation methods.

####  III-A4 Information Maximization loss

The Information Maximization loss is designed to encourage neural networks to
learn more informative representations by maximizing the mutual information
between the learned features and the input data [41, 42]. This type of loss
aims to guide the model to capture relevant and distinctive patterns in the
data, which can be especially valuable in scenarios where unsupervised
learning, domain adaptation, or feature learning are important. The
assumptions that $p_{t}=\text{softmax}(N(f(x_{t})))$ are expected to retain as
much information about $x_{t}$ as possible, and decision boundary should not
cross high-density regions, but instead lie in low-density regions, which is
also known as cluster assumption. These two assumptions can be met by
maximizing mutual information between the empirical distribution of the target
inputs and the induced target label distribution, which can be formally
defined as:

$\begin{gathered}I(p_{t};x_{t})=H(\overline{p}_{t})-\frac{1}{n_{t}}\sum_{j=1}^{n_{t}}H(p_{tj})\\\
=-\sum_{k=1}^{K}\overline{p}_{tk}\log(\overline{p}_{tk})+\frac{1}{n_{t}}\sum_{j=1}^{n_{t}}\sum_{k=1}^{K}p_{tkj}\log(p_{tkj}),\end{gathered}$
(17)

where, $p_{tj}=\text{softmax}(G_{c}(G_{f}(x_{tj})))$,
$\overline{p}_{t}=\mathbb{E}_{x_{t}}[p_{t}]$, and K is the number of classes.
Maximizing $-\frac{1}{n_{t}}\sum_{j=1}^{n_{t}}H(p_{tj})$ enforces the target
predictions close to one-hot encoding, therefore the cluster assumption is
guaranteed. To ensure global diversity, we also maximize $H(\overline{p}_{t})$
to avoid every target data being assigned to the same class. With
$I(p_{t};x_{t})$, our model is encouraged to learn tightly clustered target
features with uniform distribution, such that the discriminative information
in the target domain is retained.

Figure 4: Snapshots of 2D tSNE plots of the target (NCT) domain sample
features before training (leftmost), after three epochs (middle), and after
six epochs (right).

##  IV Experimentation and Results

Dataset: To evaluate the proposed method, we introduce the FHIST dataset, a
proposed benchmark for the few-shot classification of histological images
[47]. FHIST is composed of several histology datasets, namely CRC-TP [32],
LC25000 [33], BreakHis [34], and NCT-CRC-HE-100K [35]. For each class, there
are close to 20,000 patches in the CRC-TP domain with a patch size of 150X150
pixels and around 10,000 patches of size 224X224 pixels in the NCT domain. We
performed experiments with CRC-TP as the source and NCT as the target and vice
versa. The tSNE plots shown in Figure 4 depict the distribution of target
(NCT) at different stages of training. Different colors map different class
types in the tSNE plot. We have plotted five classes in tSNE which are Benign,
Tumor, Debris, Inflammatory, and Muscle + Stroma with 200 sample points from
each five classes. We combined the last two classes because of their
physiological as well as feature intertwining. The first plot(leftmost) shows
the data distribution of NCT(as target) at epoch 0, and the second one shows
the data distribution of NCT after four epochs, and the last one (rightmost)
shows the target(NCT) data distribution after six epochs of domain adaptation.
These histology datasets consist of different tissue types and different
organs. We consider each tissue type as a class label with one-hot encoding in
the classification task. We framed our experiments on CRC-TP and NCT with six
classes (Benign, Tumor, Muscle, Stroma, Debris, and Inflammatory)

Implementation: All the experiments were conducted on an NVIDIA A100 in
PyTorch, using the CNN-based neural network (ResNet-152) pre-trained on
ImageNet [31] as the backbone for our proposed model. The base learning rate
is 0.00001 with a batch size of 32, and we train models by 20 epochs. The
hyper-parameters were $\beta$=0.05 , $\gamma$=1.4 , $\delta$=0.54 and
$\eta$=0.54 for the experiment of CRC-TP $\rightarrow$ NCT and NCT
$\rightarrow$ CRC-TP. We used AdamW [36] with a momentum of 0.9, and a weight
decay of 0.001 as the optimizer. We adopt the standard protocol for
unsupervised domain adaptation (UDA) where all labeled source samples and
unlabeled target samples are utilized for training. To report our results for
each transfer task, we use center-crop images from the target domain and
report the classification performance. For a fair comparison with prior works,
we also conduct experiments with the same backbone as ViT-based [44] as TVT
[39], ResNet-50 [2], DANN [1], CDAN [3], GVB-GD [37], CHATTY+MCC [38] on FHIST
dataset.

Figure 5: Evolution of accuracy on FHIST dataset [47] CRC-TP to NCT domain
adaptation task shows improvement in convergence with proposed model compare
to CHATTY model [38].

###  IV-A Dataset and Implementation

###  IV-B Results

Our analysis in Table I depicts results with different methods and feature
extractors for the FHIST dataset. The top five methods are CNN model using
ResNet-50 as a feature extractor trained on ImageNet dataset while TVT uses
ViT based model pretrained on ImageNet-21k dataset. Our proposed method is a
CNN-based model that utilizes ResNet-152 as a backbone with a novel
combination of loss functions. Our model outperforms CNN-based models such as
ResNet-50, DANN, CDAN, GVB-GD, and CHATTY+MCC, and surpasses the state of the
Art (SoTA) CNN results by 6.56%. At the same time, our method also surpasses
the transformer-based SoTA by 1.41%. We achieved an accuracy of 87.71% for
CRC-TP to NCT domain adaptation and 74.81% for NCT to CRC-TP with an average
accuracy of 81.26% for both tasks, as mentioned in Table I with bold text.

TABLE I:  Accuracy (%) on the FHIST dataset [47] with two different UDA tasks and their average, where all methods are fine-tuned on their respective backbone model. Method | Backbone Feature Extractor | CRC-TP $\rightarrow$ NCT | NCT $\rightarrow$ CRC-TP | Average
---|---|---|---|---
ResNet-50 [2] |  | 40.7 | 32.9 | 36.8
DANN [1] |  | 73.5 | 66.6 | 70.0
CDAN [3] | ResNet-50 | 66.2 | 61.4 | 63.8
GVB-GD [37] |  | 73.9 | 66.7 | 70.3
CHATTY [38] |  | 81.6 | 67.9 | 74.7
TVT [39] | ViT | 86.47 | 73.23 | 79.85
Proposed Method | ResNet-152 | 87.71 | 74.81 | 81.26

## Discussion and Conclusion

In this study, we have demonstrated that utilizing different combinations of
loss functions with a CNN such as ResNet-152 can lead to significant
improvements in unsupervised domain adaptation (UDA) performance that can
surpass the performance of ViTs using other UDA methods. By leveraging the
strengths of various loss functions tailored to specific domain
characteristics, we have surpassed the state-of-the-art (SOTA) performance for
histology images. We conducted ablation studies to understand the impact of
the different feature extractors such as ConvMixer [46] and ResNet-101 [2].
However, the performance in these cases was worse than our reported results.
To know the impact of individual loss and a combination of losses, we
performed extensive experiments. Through comprehensive experiments, we
discovered that Minimum Class Confusion (MCC) loss functions offer an
enhancement to classification models by mitigating class confusion,
particularly when faced with imbalanced class distributions. In parallel, we
observed that information maximization losses aid the classifier in selecting
the most certain samples for domain alignment. In our proposed approach, the
Pseudo Label Maximum Mean Discrepancy (PLMMD) accelerates training convergence
(comparision with CHATTY model shown in Figure 5) and notably enhances domain
alignment by incorporating weighted considerations. Additionally, the Maximum
Mean Discrepancy (MMD) loss effectively narrows the gap between the mean
embeddings of the two distributions. By artfully combining these distinctive
loss functions, we not only surpass the current state-of-the-art but also
achieve a comprehensive solution that advances the field of classification
models in diverse scenarios.

## References

  * [1] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1–35, 2016.
  * [2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2015\.
  * [3] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. Advances in Neural Information Processing Systems, 31, 2018.
  * [4] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pas-cal Germain, Hugo Larochelle, Franc¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. J. Mach. Learn. Res., 17(1):2096– 2030, Jan. 2016
  * [5] P. O. Pinheiro. Unsupervised domain adaptation with similarity learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8004–8013, June 2018.
  * [6] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial discriminative domain adaptation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2962–2971, July 2017.
  * [7] W. Zhang, W. Ouyang, W. Li, and D. Xu. Collaborative and adversarial network for unsupervised domain adaptation. In2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3801–3809, June 2018
  * [8] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, pages 97–105. JMLR.org, 2015.
  * [9] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Unsupervised domain adaptation with residual transfer networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, pages 136–144, USA, 2016. Curran Associates Inc.
  * [10] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, pages 2208–2217. JMLR.org, 2017.
  * [11] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Conditional adversarial domain adaptation. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems, NIPS’18, pages 1647–1657, USA, 2018. Curran Associates Inc.
  * [12] Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Multi-adversarial domain adaptation. In Association for the Advancement of Artificial Intelligence (AAAI), pages 3934–3941, 2018.
  * [13] Hui Tang and Kui Jia. Discriminative adversarial domain adaptation. In Association for the Advancement of Artificial Intelligence (AAAI), 2020.
  * [14] C. Chen, W. Xie, W. Huang, Y. Rong, X. Ding, Y. Huang, T. Xu, and J. Huang. Progressive feature alignment for unsupervised domain adaptation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 627–636, June 2019.
  * [15] Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen. Learning semantic representations for unsupervised domain adaptation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5423–5432, Stockholmsmssan, Stockholm Sweden, 10–15 Jul 2018. PMLR.
  * [16] C. Lee, T. Batra, M. H. Baig, and D. Ulbricht. Sliced wasser- stein discrepancy for unsupervised domain adaptation. In 2019 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 10277–10287, June 2019.
  * [17] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Adversarial dropout regularization. In International Conference on Learning Representations, 2018.
  * [18] K. Saito, K. Watanabe, Y. Ushiku, and T. Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3723–3732, June 2018.
  * [19] V. K. Kurmi, S. Kumar, and V. P. Namboodiri. Attending to discriminative certainty for domain adaptation. In 2019 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 491–500, June 2019.
  * [20] Ximei Wang, Liang Li, Weirui Ye, Mingsheng Long, and Jianmin Wang. Transferable attention for domain adaptation. In Association for the Advancement of Artificial Intelligence (AAAI), 2019.
  * [21] Jun Wen, Risheng Liu, Nenggan Zheng, Qian Zheng, Zhefeng Gong, and Junsong Yuan. Exploiting local feature pat- terns for unsupervised domain adaptation. In Association for the Advancement of Artificial Intelligence (AAAI), 2019.
  * [22] Guoliang Kang, Liang Zheng, Yan Yan, and Yi Yang. Deep adversarial attention alignment for unsupervised domain adaptation: The benefit of target expectation maximization. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision – ECCV 2018, pages 420–436, Cham, 2018. Springer International Publishing
  * [23] S. Sankaranarayanan, Y. Balaji, C. D. Castillo, and R. Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8503–8512, June 2018.
  * [24] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27, pages 2672–2680. 2014.
  * [25] Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1081–1090, Long Beach, California, USA, 09–15 Jun 2019. PMLR.
  * [26] R. Xu, G. Li, J. Yang, and L. Lin. Larger norm more trans- ferable: An adaptive feature norm approach for unsupervised domain adaptation. In 2019 IEEE/CVF International Con- ference on Computer Vision (ICCV), pages 1426–1435, Oct 2019\.
  * [27] Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Ge- offrey Gordon. On learning invariant representations for domain adaptation. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7523–7532, Long Beach, California, USA, 09–15 Jun 2019. PMLR. classifier discrepancy for unsupervised domain adaptation. In 2018 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 3723–3732, June 2018.
  * [28] Zhu, Jinjing, Haotian Bai, and Lin Wang. ”Patch-Mix Transformer for Unsupervised Domain Adaptation: A Game Perspective.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
  * [29] Anand, Deepak, et al. ”Weakly supervised learning on unannotated H&E‐stained slides predicts BRAF mutation in thyroid cancer with high accuracy.” The Journal of pathology 255.3 (2021): 232-242.
  * [30] Vahadane, Abhishek, et al. ”Structure-preserving color normalization and sparse stain separation for histological images.” IEEE transactions on medical imaging 35.8 (2016): 1962-1971.
  * [31] He, Kaiming, et al. ”Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
  * [32] S. Javed, A. Mahmood, N. Werghi, K. Benes, and N. Rajpoot, “Multiplex cellular com- munities in multi-gigapixel colorectal cancer histology images for tissue phenotyping,” IEEE Transactions on Image Processing, vol. 29, pp. 9204–9219, 2020.
  * [33] A. A. Borkowski, M. M. Bui, L. B. Thomas, C. P. Wilson, L. A. DeLand, and S. M. Mastorides, “Lung and colon cancer histopathological image dataset (lc25000),” arXiv preprint arXiv:1912.12142, 2019.
  * [34] F. A. Spanhol, L. S. Oliveira, C. Petitjean, and L. Heutte, “A dataset for breast can- cer histopathological image classification,” Ieee transactions on biomedical engineering, vol. 63, no. 7, pp. 1455–1462, 2015
  * [35] J. N. Kather, N. Halama, and A. Marx, “100,000 histological images of human colorectal cancer and healthy tissue,” Zenodo10, vol. 5281, 2018
  * [36] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  * [37] Cui, Shuhao, et al. ”Gradually vanishing bridge for adversarial domain adaptation.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.
  * [38] Wagle, Mukta, Ravi Kant Gupta, and Amit Sethi. ”CHATTY: Coupled Holistic Adversarial Transport Terms with Yield for Unsupervised Domain Adaptation.” arXiv preprint arXiv:2304.09623 (2023).
  * [39] Yang, Jinyu, et al. ”Tvt: Transferable vision transformer for unsupervised domain adaptation.” Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.
  * [40] Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning (ICML), 2015.
  * [41] Gomes, R.; Krause, A.; and Perona, P. 2010. Discriminative clustering by regularized information maximization.
  * [42] Shi, Y.; and Sha, F. 2012. Information-theoretical learning of discriminative clusters for unsupervised domain adaptation. arXiv preprint arXiv:1206.6438.
  * [43] Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. Computer Vision – ECCV 2020, pages 464–480, 2020.
  * [44] Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
  * [45] Vahadane, Abhishek, et al. ”Structure-preserving color normalization and sparse stain separation for histological images.” IEEE transactions on medical imaging 35.8 (2016): 1962-1971.
  * [46] Trockman, Asher, and J. Zico Kolter. ”Patches are all you need?.” arXiv preprint arXiv:2201.09792 (2022).
  * [47] Fereshteh Shakeri, Malik Boudiaf, Sina Mohammadi, Ivaxi Sheth, Mohammad Havaei, Ismail Ben Ayed, and Samira Ebrahimi Kahou. Fhist: A benchmark for few-shot classification of histological images. 2022
  * [48] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems (NIPS), 2014.
  * [49] L. Song, J. Huang, A. Smola, and K. Fukumizu. Hilbert space embeddings of conditional distributions with applications to dynamical systems. In International Conference on Machine Learning (ICML), 2009.
  * [50] M. Long, J. Wang, and M. I. Jordan. Deep transfer learning with joint adaptation networks. In International Conference on Machine Learning (ICML), 2017.
  * [51] L. Song, B. Boots, S. M. Siddiqi, G. J. Gordon, and A. Smola. Hilbert space embeddings of hidden markov models. In International Conference on Machine Learning (ICML), 2010.
  * [52] L. Song and B. Dai. Robust low rank kernel embeddings of multivariate distributions. In Advances in Neural Information Processing Systems (NIPS), pages 3228–3236, 2013.
  * [53] L. Song, K. Fukumizu, and A. Gretton. Kernel embeddings of conditional distributions: A unified kernel framework for nonparametric inference in graphical models. IEEE Signal Processing Magazine, 30(4):98–111, 2013.
  * [54] aurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 2008.
  * [55] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine Learning, 79(1-2):151–175, 2010.