# Exploring Explainable AI Techniques for Improved Interpretability in Lung
and Colon Cancer Classification

Mukaffi Bin Moin*, Fatema Tuj Johora Faria, Swarnajit Saha, Busra Kamal Rafa,
Mohammad Shafiul Alam

Ahsanullah University of Science and Technology, Dhaka, Bangladesh.

*Corresponding author(s). E-mail(s<EMAIL_ADDRESS>
Contributing authors<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Lung and colon cancer are serious worldwide health challenges that require
early and precise identification to reduce mortality risks. However,
diagnosis, which is mostly dependent on histopathologists’ competence,
presents difficulties and hazards when expertise is insufficient. While
diagnostic methods like imaging and blood markers contribute to early
detection, histopathology remains the gold standard, although time-consuming
and vulnerable to inter-observer mistakes. Limited access to high-end
technology further limits patients’ ability to receive immediate medical care
and diagnosis. Recent advances in deep learning have generated interest in its
application to medical imaging analysis, specifically the use of
histopathological images to diagnose lung and colon cancer. The goal of this
investigation is to use and adapt existing pre-trained CNN-based models, such
as Xception, DenseNet201, ResNet101, InceptionV3, DenseNet121, DenseNet169,
ResNet152, and InceptionResNetV2, to enhance classification through better
augmentation strategies. The results show tremendous progress, with all eight
models reaching impressive accuracy ranging from 97% to 99%. Furthermore,
attention visualization techniques such as GradCAM, GradCAM++, ScoreCAM,
Faster Score-CAM, and LayerCAM, as well as Vanilla Saliency and SmoothGrad,
are used to provide insights into the models’ classification decisions,
thereby improving interpretability and understanding of malignant and benign
image classification. Our research implementations are open to the public at:
https://github.com/Mukaffi28/Explainable-AI-for-Lung-and-Colon-Cancer-
Classification

_K_ eywords Lung Colon Cancer, Pre-trained CNN, Medical Imaging,
Classification, Deep Learning, GradCAM, GradCAM++, Explainability,
Histopathology

##  1 Introduction

A group of disorders referred to as cancer is characterised by an uncontrolled
expansion and spread of abnormal cells within the body. These cells can
infiltrate surrounding organs and tissues, leading to dangerous health
consequences [1]. In 2023111 https://www.paho.org/en/campaigns/world-cancer-
day-2023-close-care-gap, approximately 10 million people worldwide died from
cancer-related causes, highlighting the ongoing global impact of this disease
on public health. It is projected that the cancer death rate would increase to
60% by 2035.[2]. Histopathological imaging is critical for diagnosing and
curing lung and colon cancers because it provides microscopic images of tissue
samples, highlights cellular architecture, and informs treatment decisions
[2]. Digital pathology, powered by scanning technology, is transforming
medicine by digitizing histopathological slides to improve disease diagnosis
and management. AI integration increases its potential for clinical diagnosis
and research [3]. This powerful combination has the potential to expand its
applications in various fields, including cancer detection [4], cardiovascular
diseases [5], neurological disorders [6], diabetic retinopathy [7], pulmonary
diseases [8], and skin diseases [9].

In the last few years, major improvements have been achieved in the automated
classification of lung and colon cancer using histopathological images.
Through the use of convolutional neural networks (CNNs), Sanidhya’s [4] study
demonstrates the promise of artificial intelligence (AI) in enhancing cancer
diagnoses by developing a computer-aided diagnosis system that can accurately
detect lung and colon tumors from histopathological pictures with high
accuracy (for the lung, 97%, and for the colon, 96%.). Osamu et al. used
convolutional and recurrent neural networks to classify gastric and colonic
epithelial tumors in histopathology images. With AUCs of 0.96 & 0.99 for colon
cancer & adenoma, and 0.97 and 0.99 to stomach cancer and adenoma,
respectively, the models were incredibly accurate[10]. In another study
Sudhakar’s [3] introduces an automated method using EfficientNetV2 models to
detect lung and colon cancer subtypes with 99.97% accuracy, surpassing current
methods. Their approach, which includes gradCAM-generated visual maps, assists
pathologists in identifying critical locations for treatment plans, and it
shows promise for clinical automation in cancer detection. Neha’s research
[11] uses CNNs (ResNet50, VGG19, InceptionResNetV2, DenseNet) to classify lung
cancer histology, aiming to improve diagnosis accuracy for better treatment
decisions and reduce pathologists’ workload, potentially improving patient
outcomes in lung cancer care.

After thorough analysis, a noticeable gap emerges in the realm of explainable
AI techniques applied to lung and colon cancer using histopathological images.
In this research paper, we focus to automate the detection of lung and colon
cancer using histopathological images. We undertake a thorough evaluation of
eight renowned pre-trained CNN models, including Xception, DenseNet121,
DenseNet169, DenseNet201, InceptionV3, ResNet101, ResNet152, and
InceptionResNetV2. To improve interpretability, we use Explainable AI
approaches as Grad-CAM, Grad-CAM++, Score-CAM, FasterScore-CAM, Layer CAM for
Class Activation Maps, and Vanilla Saliency and SmoothGrad for Saliency Maps.
These strategies clarify the logic behind our models’ predictions, leading to
a better understanding of their decision-making processes.

  * •

The inspection of eight pre-trained CNN models (Xception, DenseNet121,
DenseNet169, DenseNet201, InceptionV3, ResNet101, ResNet152, and
InceptionResNetV2) for automated lung and colon cancer classification in
histological images.

  * •

Explainable AI approaches including Grad-CAM, Grad-CAM++, Score-CAM,
FasterScore-CAM, Layer CAM, Vanilla Saliency, and SmoothGrad have been
integrated to increase interpretability.

  * •

Our research attempts to close the gap in explainable AI methods for colon and
lung cancer classification in histopathology images.

##  2 Related Works

###  2.1 Lung and Colon Cancer Classification without using XAI Techniques

Sanidhya et al. [4] used CNNs to identify malignancies of the lung and colon,
with diagnosis accuracy of more than 97% for cancer of the lungs and 96% for
cancer of the colon, based on digital pathology images from the LC25000
dataset. Highlighting the vital necessity of accurate and timely lung cancer
histology detection, Neha et al. [11] builds upon prior work in lung cancer
diagnosis and classification. Deep learning techniques, particularly CNNs like
ResNet 50, VGG-19, Inception_ResNet_V2, and DenseNet, have shown promise in
analyzing histopathological images for accurate subtype classification. Hasan
et al. [2] offered a deep convolutional neural network model for precise
identification of colon adenocarcinoma from digital histopathology images,
achieving up to 99.80% accuracy. Notably, none of these studies explored
Explainable AI techniques in the context of lung and colon cancer diagnosis.

###  2.2 Lung and Colon Cancer Classification using XAI Techniques

Satvik and Somya [1] developed a system that automates lung as well as colon
cancer identification using deep neural networks on images from
histopathology. Employing eight pre-trained CNN models, they achieved
remarkable accuracies ranging from 96% to 100%. For explainable AI, they
utilized GradCAM and SmoothGrad techniques. Introducing a novel Bilinear-CNN-
based model, another research effort [12] focuses on whole-slide images (WSIs)
automated tissue segmentation of lung cancer. This model addresses challenges
posed by tumor heterogeneity. For explainable AI, GradCAM was utilized.
Sudhakar et al. [3] proposed an automated method utilizing EfficientNetV2
models for detecting lung and colon cancer subtypes from histopathology
images. Visual saliency maps were employed to aid in understanding model
decisions, obtaining a remarkable 99.98% maximum test accuracy on the LC25000
dataset. However, Ahmed et al. [13] offered a lightweight, CNN-based deep
learning technique for accurate colon cancer detection. Despite lacking Class
Activation Maps or Saliency Maps, their method achieved a high accuracy of
99.50%, outperforming existing deep learning approaches.

##  3 Dataset Description

We utilized the LC25000 dataset [14], which carries 25,000 color pictures of
lung and colon tissues categorised into five groups: lung squamous cell
carcinoma, lung adenocarcinoma, benign lung tissue, colon cancer, and benign
colonic tissue. 5,000 photos each class, each resized to 768 by 768 pixels.
The collection is divided into colon and lung image sets in accordance with
HIPAA compliance guidelines. It’s instrumental in developing diagnostic tools
for lung and colon cancers, driving progress in medical imaging research. The
visual representation of the LC25000 dataset is displayed in Figure 1.

Figure 1: Representation of LC25000 dataset

##  4 Background Study

###  4.1 Convolutional Neural Networks (CNNs)

Xception [15], DenseNet201 [16], DenseNet121 [16], DenseNet169 [16], ResNet101
[17], ResNet152 [17], InceptionV3 [18], and InceptionResNetV2 [19] are all
prominent Convolutional Neural Network (CNN) architectures. Xception, an
“Extreme Inception,” enhances the Inception model by utilizing depthwise
separable convolutions for improved computational efficiency. DenseNet models,
including DenseNet201, DenseNet121, and DenseNet169, feature densely connected
layers where each layer connects to every other layer, facilitating feature
reuse. ResNet101 and ResNet152, part of the ResNet family, address the
disappearing gradient issue by introducing skip connections, enabling the
training of extremely deep networks. InceptionV3, a member of the Inception
family, employs various filter sizes to capture features at multiple scales
efficiently. InceptionResNetV2 combines the strengths of both Inception and
ResNet architectures, integrating residual connections and multi-scale feature
extraction for enhanced performance.

###  4.2 Explainable Artificial Intelligence (XAI) techniques

XAI techniques are essential for enhancing the transparency and
interpretability of AI models. Among the notable methods are GradCAM, which
identifies critical image regions by analyzing gradients of the target class
score relative to convolutional neural network feature maps, and its extension
GradCAM++, which refines localization accuracy by considering both positive
and negative influences. ScoreCAM assigns importance scores to spatial
locations in feature maps based on class scores, facilitating precise
localization, while Faster Score-CAM optimizes efficiency for real-time
applications. LayerCAM attributes importance scores to input pixels via
relevance propagation across model layers, providing insights into decision-
making processes. Gradients of output class scores are computed applying
Vanilla Saliency in relation to input pictures, highlighting influential
regions, whereas SmoothGrad enhances interpretability by averaging multiple
saliency maps to reduce noise and provide smoother visualizations.

###  4.3 Evaluation Metrics

In the evaluation of lung and colon classification tasks, several metrics are
vital for comprehensively assessing model performance. Accuracy measures the
overall correctness of predictions irrespective of class distribution, while
precision and recall evaluate the model’s ability to correctly identify
instances of lung or colon conditions, focusing on minimizing false positives
and false negatives, respectively. The F1 score strikes a balance between
precision and recall, crucial for tasks with class imbalance. The Jaccard
score assesses the overlap between predicted and actual classes, especially
valuable in multi-class problems or imbalanced datasets. Finally, log loss
quantifies the accuracy of predicted probabilities compared to actual
probabilities, incentivizing well-calibrated predictions in lung and colon
classification models.

##  5 Proposed Methodology

This comprehensive approach allows us to compare the performance of various
models in lung and colon cancer image classification tasks efficiently. Figure
2 illustrates the methodology employed in our research.
Step 1) Input Image: To ensure uniformity and compatibility across diverse
models and techniques, the images underwent meticulous preprocessing. This
crucial step involved resizing the images to a standardized dimension of
299x299 pixels.
Step 2) Image Preprocessing: To prepare our dataset for model training, we
initiated with pixel value normalization to a standard scale of 0 to 1,
fostering convergence during training and addressing issues related to
disparate data distributions. Following this, we introduced random rotations,
diversifying the dataset and enhancing the model’s adaptability to varied
orientations. Horizontal flipping was also used to increase variety and reduce
overfitting concerns by simulating mirror images. Subsequently, we selectively
cropped images, eliminating extraneous background elements and emphasizing
regions of interest, thereby facilitating feature extraction and reducing
computational complexity. Addressing illumination discrepancies, we adjusted
image brightness levels to ensure dataset consistency and minimize lighting
variations’ impact on model performance. Finally, we used contrast enhancement
techniques to refine image contrast, particularly important in medical imaging
where detailed information is critical for accurate analysis.
Step 3) Model Selection and Training: For model selection, we opted for eight
CNN architectures renowned for their efficacy in image classification tasks.
These architectures include Xception, DenseNet201, ResNet101, InceptionV3,
DenseNet121, DenseNet169, ResNet152, and InceptionResNetV2. Subsequently, we
proceeded with model training by implementing each CNN architecture using the
TensorFlow deep learning framework. The models were trained using the
preprocessed dataset, which ensured coherence and stability throughout the
training process. Furthermore, hyperparameter tuning was conducted, the
details of which are provided in Table 1. This optimization process aimed to
fine-tune the model’s parameters and enhance its performance on the specific
task of lung and colon cancer classification.
Step 4) Explainable AI Techniques: We applied eXplainable AI (XAI) techniques
to the last layer of the CNN models to enhance interpretability and provide
insights into model decision-making. For Class Activation Maps (CAM), we
utilized GradCAM, GradCAM++, ScoreCAM, Faster Score-CAM, and LayerCAM to
generate heatmaps highlighting discriminative regions in the images for each
class, refining feature localization, producing class-specific attention maps,
and visualizing activations at different network layers for deeper insights.
Additionally, we employed Vanilla Saliency and SmoothGrad techniques for
generating saliency maps to identify high-gradient regions influencing model
predictions and enhancing interpretability.
Step 5) Performance Evaluation: Table 2 shows evaluation of each model to
assess its effectiveness in lung and colon cancer classification. By employing
these metrics, we gained a comprehensive understanding of each model’s
classification performance, allowing us to compare and select the most
suitable model for the task at hand.

Figure 2: Representation of our proposed methodology for lung and colon cancer
classification using explainable AI techniques

##  6 Results and Discussion

###  6.1 Hyperparameter Settings

Eight pretrained CNN models were trained to classify lung and colon cancer. To
ensure optimal convergence without overfitting, an Adam optimizer with a batch
size of ten and a learning rate that varied between 0.0001 and 0.001
throughout epochs 25 to 40 was employed by every model. Hyperparameter
Settings for Pre-trained CNN Models for lung and colon cancer classification
are shown in Table 1

Table 1: Hyperparameter Settings for Pre-trained CNN Models for lung and colon cancer classification Models |  Learning Rate |  Batch Size |  Number of Epochs |  Optimizer
---|---|---|---|---
Xception |  0.001 |  10 |  30 |  Adam
DenseNet201 |  0.001 |  10 |  25 |  Adam
ResNet101 |  0.001 |  10 |  35 |  Adam
InceptionV3 |  0.001 |  10 |  25 |  Adam
DenseNet121 |  0.001 |  10 |  30 |  Adam
DenseNet169 |  0.001 |  10 |  35 |  Adam
ResNet152 |  0.001 |  10 |  35 |  Adam
InceptionResNetV2 |  0.001 |  10 |  40 |  Adam

###  6.2  Experiments

The Table 2 compares performance for a total of eight pretrained CNN models in
lung and colon cancer classification. With an accuracy of 0.9989 and a log
loss of 0.0384, Xception performs the best. With the highest log loss of
0.8458 and the lowest accuracy of 0.9765, InceptionResNetV2 performs the
worst. The metrics include accuracy, precision, recall, F1-score, Jaccard
score, and log loss, with greater values signifying better outcomes except for
log loss, where lower values are better. Figure 3 show Confusion Matrices of
Pretrained CNNs for Lung and Colon Cancer Classification. Visualizations of
the Various Explainable AI techniques are shown in Figure 4.

Table 2: Performance Metrics Comparison of Various Pre-trained CNN Models Model |  Accuracy |  Precision |  Recall |  F1-Score |  Jaccard Score |  Log Loss
---|---|---|---|---|---|---
Xception |  0.9989 |  0.9989 |  0.9989 |  0.9989 |  0.9978 |  0.0384
DenseNet201 |  0.9971 |  0.9971 |  0.9971 |  0.9971 |  0.9942 |  0.1057
ResNet101 |  0.9928 |  0.9928 |  0.9928 |  0.9928 |  0.9858 |  0.2595
InceptionV3 |  0.9904 |  0.9907 |  0.9904 |  0.9904 |  0.9812 |  0.3460
DenseNet121 |  0.9896 |  0.9898 |  0.9896 |  0.9896 |  0.9795 |  0.3749
DenseNet169 |  0.9888 |  0.9888 |  0.9888 |  0.9888 |  0.9781 |  0.4037
ResNet152 |  0.9885 |  0.9886 |  0.9885 |  0.9885 |  0.9774 |  0.4133
InceptionResNetV2 |  0.9765 |  0.9765 |  0.9765 |  0.9765 |  0.9547 |  0.8458

(a) Xception

(b) DenseNet201

(c) ResNet101

(d) InceptionV3

(e) DenseNet121

(f) DenseNet169

(g) ResNet152

(h) InceptionResNetV2

Figure 3: Confusion Matrices of Pretrained CNNs for Lung and Colon Cancer
Classification Figure 4: Different versions of AI insights such as GradCAM,
GradCAM++, ScoreCAM, Faster ScoreCAM, LayerCAM, Smooth Grad, and Vanilla
Saliency provide various explainable views of the original image in the
context of lung and colon cancer classification

##  7 Limitations and Future Research Directions

While our research has offered useful insights into the classification of lung
and colon cancers, we note numerous limitations that must be considered. To
begin, characteristics such as dataset size, quality, and variety can all have
an impact on our models’ effectiveness. To address these constraints, bigger
and more diversified datasets would be required, maybe combining data from
different sources to improve model generalization. Furthermore, our research
concentrated on image-based categorization, ignoring possible synergies with
other modalities such as genomes or clinical data. Future research might
investigate multimodal techniques to increase classification accuracy and
adaptability. Moreover, the interpretability of our models, while enhanced
through XAI techniques, remains a challenge, particularly in complex medical
domains. Further research is needed to develop more interpretable models and
refine existing XAI methods to provide deeper insights into model decision-
making processes. Furthermore, our research largely focused on two forms of
cancer: lung and colon. Extending our method to additional forms of cancer
would increase its relevance and influence in the area of oncology.
Furthermore, examining our models’ transferability to various healthcare
settings and patient demographics will be beneficial for real-world
deployment.

##  8 Conclusion

Our research demonstrates the efficacy of our suggested methodology for lung
and colon cancer classification, which uses advanced deep learning and
eXplainable AI (XAI) techniques to improve accuracy and interpretability. We
obtained outstanding classification results by applying cutting-edge CNN
architectures to a standardized dataset. Notably, Xception outperformed all
other models tested, with an amazing accuracy of 0.9989 and the lowest Log
Loss of 0.0384. In contrast, InceptionResNetV2 had slightly lower accuracy
(0.9765) and the highest Log Loss (0.8458), indicating that its classification
capabilities should be improved. Our incorporation of XAI techniques such as
GradCAM, GradCAM++, ScoreCAM, Faster Score-CAM, LayerCAM, Vanilla Saliency,
and SmoothGrad yielded useful insights into the CNN models’ decision-making
processes. These strategies improved the visualization of discriminative areas
in histopathological images, allowing for the detection of critical
characteristics that contribute to the categorization of malignant and benign
tissues. Such visualizations improve interpretability and enable more informed
decision-making in healthcare settings, resulting in better patient care and
treatment results. Overall, our study illustrates the power of sophisticated
deep learning models and XAI approaches in medical image analysis, providing a
solid foundation for precise cancer classification. Xception’s superior
performance demonstrates its efficacy in making confident and precise
predictions, while insights from XAI techniques improve the interpretability
of the classification process, establishing the way for improved diagnostic
accuracy and clinical decision support in cancer treatment.

## References

  * [1] Satvik Garg and Somya Garg.  Prediction of lung and colon cancer through analysis of histopathological images by utilizing pre-trained cnn models with visualization of class activation and saliency maps.  In 2020 3rd Artificial Intelligence and Cloud Computing Conference, AICCC 2020. ACM, December 2020.
  * [2] Md Imran Hasan, Md Shahin Ali, Md Habibur Rahman, and Md Khairul Islam.  Automated detection and characterization of colon cancer with deep convolutional neural networks.  Journal of Healthcare Engineering, 2022:1–12, August 2022.
  * [3] Sudhakar Tummala, Seifedine Kadry, Ahmed Nadeem, Hafiz Tayyab Rauf, and Nadia Gul.  An explainable classification method based on complex scaling in histopathology images for lung and colon cancer.  Diagnostics, 13(9):1594, April 2023.
  * [4] Sanidhya Mangal, Aanchal Chaurasia, and Ayush Khajanchi.  Convolution neural networks for diagnosing colon and lung cancer histopathological images, 2020.
  * [5] Luísa Soares.  Cardiovascular disease: A review.  Biomedical Journal of Scientific & Technical Research, 51(3), July 2023.
  * [6] Ziad Rizk, Ghadha Ibrahim Fouad, and Hanan Aly.  Neurological disorders: Causes and treatments strategies.  page International Journal Of Public Mental Health And Neurosciences, 04 2018.
  * [7] Manoj S H and Arya A Bosale.  Detection and classification of diabetic retinopathy using deep learning algorithms for segmentation to facilitate referral recommendation for test and treatment prediction, 2024.
  * [8] Sudipto Bhattacharjee, Banani Saha, Parthasarathi Bhattacharyya, and Sudipto Saha.  Classification of obstructive and non-obstructive pulmonary diseases on the basis of spirometry using machine learning techniques.  Journal of Computational Science, 63:101768, September 2022.
  * [9] K. Veera Swamy and B. Divya.  Skin disease classification using machine learning algorithms.  In 2021 2nd International Conference on Communication, Computing and Industry 4.0 (C2I4). IEEE, December 2021.
  * [10] Osamu Iizuka, Fahdi Kanavati, Kei Kato, Michael Rambeau, Koji Arihiro, and Masayuki Tsuneki.  Deep learning models for histopathological classification of gastric and colonic epithelial tumours.  Scientific Reports, 10(1), January 2020.
  * [11] Neha Baranwal, Preethi Doravari, and Renu Kachhoria.  Classification of histopathology images of lung cancer using convolutional neural network (cnn), 2021.
  * [12] Rui Xu, Zhizhen Wang, Zhenbing Liu, Chu Han, Lixu Yan, Huan Lin, Zeyan Xu, Zhengyun Feng, Changhong Liang, Xin Chen, Xipeng Pan, and Zaiyi Liu.  Histopathological tissue segmentation of lung cancer with bilinear cnn and soft attention.  BioMed Research International, 2022:1–10, July 2022.
  * [13] Ahmed S. Sakr, Naglaa F. Soliman, Mehdhar S. Al-Gaashani, Paweł Pławiak, Abdelhamied A. Ateya, and Mohamed Hammad.  An efficient deep learning approach for colon cancer detection.  Applied Sciences, 12(17):8450, August 2022.
  * [14] Andrew A. Borkowski, Marilyn M. Bui, L. Brannon Thomas, Catherine P. Wilson, Lauren A. DeLand, and Stephen M. Mastorides.  Lung and colon cancer histopathological image dataset (lc25000), 2019.
  * [15] François Chollet.  Xception: Deep learning with depthwise separable convolutions, 2017.
  * [16] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261–2269, 2017.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [18] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  CoRR, abs/1512.00567, 2015.
  * [19] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alex Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning, 2016.


11institutetext: 1Technical University of Darmstadt, Karolinenpl. 5, 64289
Darmstadt, Germany

# Adversarial Continual Learning for Multi-Domain Hippocampal
Segmentation111Supported by the Bundesministerium für Gesundheit (BMG) with
grant [ZMVI1- 2520DAT03A]. The final version of this manuscript will be
published in Springer Lecture Notes in Computer Science, Domain Adaptation and
Representation Transfer at Medical Image Computing and Computer Assisted
Intervention - DART at MICCAI 2021 as part of the joint MICCAI Workshops
proceedings (doi will follow).

Marius Memmel 11 Camila Gonzalez 11 Anirban Mukhopadhyay 11

###### Abstract

Deep learning for medical imaging suffers from temporal and privacy-related
restrictions on data availability. To still obtain viable models, continual
learning aims to train in sequential order, as and when data is available. The
main challenge that continual learning methods face is to prevent catastrophic
forgetting, i.e., a decrease in performance on the data encountered earlier.
This issue makes continuous training of segmentation models for medical
applications extremely difficult. Yet, often, data from at least two different
domains is available which we can exploit to train the model in a way that it
disregards domain-specific information. We propose an architecture that
leverages the simultaneous availability of two or more datasets to learn a
disentanglement between the content and domain in an adversarial fashion. The
domain-invariant content representation then lays the base for continual
semantic segmentation. Our approach takes inspiration from domain adaptation
and combines it with continual learning for hippocampal segmentation in brain
MRI. We showcase that our method reduces catastrophic forgetting and
outperforms state-of-the-art continual learning methods.

###### Keywords:

continual learning adversarial training feature disentanglement hippocampus
MRI segmentation

##  1 Introduction

In medical imaging, privacy regulations and temporal restrictions limit access
to data [9]. These limitations inhibit the application of traditional
supervised deep learning methods for medical imaging tasks, which require the
simultaneous availability of all data during training. Continual learning
reframes the problem into a sequential training process, where not all
datasets are available at each time step. However, when we evaluate continual
learning models, they still experience a significant drop in performance,
caused by catastrophic forgetting, i.e., the model adapting too strongly to
particularities of the last training batch [17].

The leading cause of catastrophic forgetting in medical imaging is multi-
domain data originating from different domains [25]. These domains result from
diverse disease patterns among the examined subjects and divergent
technologies and standards used during the acquisition process. In magnetic
resonance imaging (MRI) it is common practice for institutions to operate
scanners from various vendors and employ disparate protocols [21].
Additionally, MRI datasets frequently contain subjects that are either healthy
or suffer from various pathological conditions. To sufficiently solve a task
with data from multiple domains, models have to adapt and learn in a domain-
invariant fashion.

(a)

(b)

Figure 1: The Adversarial Continual Segmenter (ACS) can react to different
dataset availability through training (black) or freezing (gray) network
parts. a) In stage 1, the disentanglement is trained on two datasets from
different domains. In stage 2, a new dataset is available, and the previous
ones are not. ACS is now fine-tuned on the new one. As a fourth dataset is
introduced in stage 3, fine-tuning is repeated. b) If the first two datasets
are still available when a third becomes accessible, disentanglement can be
repeated in stage 2 with three datasets.

The limited availability of multi-domain data makes developing a general-
purpose model for continual hippocampal segmentation difficult. Commonly, at
least two datasets from different domains are accessible simultaneously due to
open access, relaxed restrictions, or access to historical data acquired with
older scanners and protocols within the institution. This serendipity allows
for learning a disentanglement between the domains and the content needed for
segmentation, which continual learning methods do not yet exploit. Inspired by
image-to-image translation (I2I), we utilize adversarial training to learn a
disentanglement between a domain-invariant content representation sufficient
for segmentation and a dataset-specific domain representation [11, 13, 19]. We
train an encoder for each representation and share the content encoder with
our segmentation module. Finally, we extend our approach to continual
learning. In Fig. 1, we describe how our architecture could react to common
dataset availability scenarios. We perform experiments on a subset of those
hypothetical scenarios.

We contribute the Adversarial Continual Segmenter (ACS) for continual semantic
segmentation of multi-domain data through adversarial disentanglement and
latent space regularization that reduces catastrophic forgetting in
hippocampal segmentation of brain MRIs.

##  2 Related work

Adversarial disentanglement: Several Generative Adversarial Networks (GANs)
disentangle the feature space to improve interpretability [15]. Chen et al.
[6] take a mutual-information-based approach while Karras et al. [14] directly
modify the generator to achieve automatic separation of high-level attributes.
Adversarial disentanglement shows promising results when applied to
segmentation in a multi-domain [12] and multi-modal setting [5]. In domain
adaptation, Kamnitsas et al. [13] utilize domain-invariant features for a
segmentation task [26] and learn those with adversarial regularization of
numerous layer outputs.

Cross-domain disentanglement: I2I translation extends cross-domain feature
disentanglement by splitting the latent space into a content and style
encoding to achieve better translation results [22]. The content encoding is
assumed only to capture task-specific information. The style encoding holds
the domain-specific information. Huang et al. [11] and Lee et al. [19] further
assume that the complexity of the content outweighs the domain, which they
reflect in different encoder complexities.

Continual learning: The main problem that continual learning methods face is
catastrophic forgetting. For this purpose, regularization-based approaches
constrain important parameters from changing [3, 8, 17, 20]. With a similar
goal, Memory Aware Synapses (MAS) [2] learn importance weights for each
parameter and use those to penalize parameter changes. Knowledge distillation
methods try to preserve specific model outputs to retain performance on old
data [7, 23]. Keeping a subset of training data is also widely used, e.g., in
dynamic memory [10] and rehearsal methods [28]. However, keeping parts of the
old data is not feasible in most medical imaging scenarios due to privacy
concerns [9].

Whereas existing continual learning methods focus solely on a sequential
learning process and do not consider the simultaneous availability of datasets
and their divergent domains, we specifically exploit these circumstances
through adversarial disentanglement.

Figure 2: Detailed visualization of the ACS architecture. We achieve feature
disentanglement through two encoders, $E_{c}$ for content and $E_{d}$ for
domain. The content and domain discriminators $D_{c}$ and $D_{d}$ regularize
the latent representation $z_{c}$, and in the case of $D_{c}$, also the skip-
connections of the U-Net to be domain-invariant. Segmentation is done by a
forward pass of the U-Net.

##  3 Methods

We first describe how we disentangle input images into content and domain
representation and follow up by introducing the adversarial approach. Our
domain representation models the heterogeneity of the acquisition modality,
e.g., varying protocols and machine vendors, as well as different disease
patterns. To learn the domain-invariant content representation, we initially
train on two datasets simultaneously, as is common in I2I translation. This
representation then acts as a basis for our U-Net segmenter.

Variational autoencoder loss: We model the domain encoder as a variational
autoencoder (VAE) [16], which encodes the domain as a distribution $z_{d}$
parameterized by variance $\sigma^{2}$ and mean $\mu$. Because the complexity
of the domain is assumed to be lower than the content complexity, we limit the
dimensionality of the domain representation to one float value. We use a
combination of a reconstruction loss between the input image $x_{i}$ and the
generator output $\hat{x}_{i}$ and a Kullback–Leibler regularization term
weighted by hyperparameter $\eta$ to draw the encoding close to a normally
distributed Gaussian prior of $\mathcal{N}(0,1)$.

$\mathcal{L}_{VAE}=\sum_{i}\|\hat{x}_{i}-x_{i}\|^{2}_{2}+\eta\mathop{\mathbb{E}_{x_{i}\sim
X_{i},z\sim\mathcal{N}(0,1)}}\left[KL\left(E_{d}\left(x_{i}\right)||z\right)\right]$
(1)

Latent regression loss: To prepare the domain information for the generator
input, we first sample from the learned domain distribution and then pass the
samples into the latent scale layer $LS$ as proposed by Alharbi et al. [1] to
produce a latent domain scale $f_{ds}$. To give $G$ additional information
about the domain, we inject a domain code $d$ using central biasing instance
normalization [12, 29]. We model this domain code with a one-hot vector. As
proposed by Jiang and Veeraraghavan [12] and based on Lee et al. [19] and
Huang et al. [11], we also define a latent code regression loss
$\mathcal{L}_{lr}$, which constrains the generator to produce unique mappings
for a latent code $z$.

GAN loss: We interpret the encoder of the U-Net as content encoder $E_{c}$
with the output of the bottom layer treated as content representation $z_{c}$.
Both $z_{c}$ and the scaled domain sample $f_{ds}$ are fed into generator $G$
to reconstruct the input sample $x_{i}$. To now disentangle the content and
domain, we introduce adversarial training. We deploy a domain discriminator
$D_{d}$ that regularizes $E_{c}$, $E_{d}$, and $G$ by discriminating whether
an input image $x_{i}$ is part of a given domain $d$. $D_{d}$ trains on a
combination of real and generated images as described in the corresponding
discriminator loss in Eq. 2. $G$ generates these images from content $z_{c}$
of $x_{i}$ and random domain $z$. $G$, $E_{c}$, and $E_{d}$ counter the
discriminator by minimizing a negative binary cross-entropy loss shown in Eq.
3. The training forces $E_{c}$ to produce a domain-invariant output, which we
utilize for the segmentation task.

$\displaystyle\mathcal{L}_{GAN_{D}}=\sum_{i}$
$\displaystyle\mathop{\mathbb{E}_{x_{i}\sim
X_{i}}}\left[\log(D_{d}\left(x_{i},d_{i}\right)\right]$ (2)
$\displaystyle+\mathop{\mathbb{E}_{x_{i}\sim
X_{i}}}\left[\log(1-D_{d}\left(G\left(E_{c}\left(x_{i}\right),LS\left(z\right)\right),d_{i}\right)\right]$
$\mathcal{L}_{GAN_{G}}=-\sum_{i}\mathop{\mathbb{E}_{x_{i}\sim
X_{i},z\sim\mathcal{N}(0,1)}}\left[\log(D_{d}\left(G\left(E_{c}\left(x_{i}\right),LS\left(z\right)\right),1\right)\right]$
(3)

Content adversarial loss: The information about the domain can still flow from
$E_{c}$ to $S$ via the skip-connections of the U-Net. To prevent this, we
introduce a content discriminator $D_{c}$ inspired by Jiang et al. and
Kamnitsas et al. [12, 13]. $D_{c}$ regularizes $z_{c}$ as well as the skip-
connections because we want $E_{c}$ to not leak any domain information to the
segmenter $S$. The discriminator design is similar to a reversed U-Net
decoder, i.e., it takes $z_{c}$ as input and the skip-connections of $E_{c}$
at each corresponding layer. We train both $D_{c}$ and $E_{c}$ using a multi-
class cross-entropy loss $\mathcal{L}_{adv_{D_{c}}}^{c}$ and
$\mathcal{L}_{adv_{E_{c}}}^{c}$ respectively. In the case of the generated
images, we represent the domain code as a placeholder class. To ensure that
the adversarial training is stable, we train the discriminators and the
remaining architecture at separate steps [12].

Segmentation: To produce the segmentation mask $\hat{y}_{i}$, we encode input
$x_{i}$ into $z_{c}$ through $E_{c}$. We then pass $z_{c}$ and the skip-
connections of $E_{c}$ into $S$ to compute $\hat{y}_{i}$. We train the U-Net
for semantic segmentation through a pixel-wise combination of a Dice and a
binary cross-entropy loss between the target mask $y_{i}$ and the prediction
$\hat{y}_{i}$. After initially training the architecture on two or more
datasets, the model has sufficiently learned the disentanglement between the
content and the domain. To train on a new dataset, we only fine-tune the last
four convolutional layers of $S$.

##  4 Datasets & Experiments

Datasets: All datasets are from different domains and contain T1-weighted
MRIs. The first dataset was released as part of the 2018 Medical Segmentation
Decathlon challenge [27] and consists of 195 subjects in total, with 90
healthy and 105 with non-affective psychotic disorder. The scans were
collected using a Philips Achieva scanner, and the mean size of the volumes is
$35\times 49\times 35$. The second dataset was published in Scientific Data
[18] and has a T1-weighted dataset with 25 healthy subjects. All scans were
acquired using MRI systems with 3 Tesla units. The mean standard resolution is
$48\times 64\times 64$. Finally, the third dataset is provided by the
Alzheimer’s Disease Neuroimaging Initiative [4] and consists of 68 subjects
that are either part of the control group or suffer from either mild cognitive
impairment or Alzheimer’s disease. The images were acquired with scanners from
Siemens, GE, and Philips with 23, 24, and 21 scans, respectively. The mean
volume size is $48\times 64\times 64$. All three datasets provide reference
segmentation masks for the hippocampus. The masks were annotated manually with
the protocols defined in the respective publications.We evaluate our
architecture on all three datasets, which we will refer to as A, B, and C,
respectively.

Experimental setup: We split each dataset into 70% train, 20% test, and 10%
validation and use the latter to select the hyperparameters. We train slice-
by-slice and upsample via bilinear interpolation to achieve uniform slices. We
compare ACS with the following baselines. First, just the U-Net block of ACS
(U-Net-b) shown in Fig. 2, and second a standard U-Net. Furthermore, we extend
the U-Net by knowledge distillation on the output layer (OL-KD) as proposed by
Michieli and Zanuttigh [23], and Memory Aware Synapses adapted to brain
segmentation (BS-MAS) by Özgün et al. [24]. As suggested in BS-MAS, we divide
the surrogate loss by the number of network parameters and normalize the
resulting importance values between zero and one. We report the Intersection
over Union (IoU) and Dice coefficient on the hippocampus class of the test
set. We use a batch size of 40 and train on four Tesla V100 SXM3 GPUs. Each
method receives training over 60 epochs. After 30 epochs, the training only
continues with the third dataset. We repeat training for every combination of
the three datasets (AB-C, AC-B, BC-A), e.g., initial training on datasets A
and B, then on C (AB-C). Additionally, we jointly train ACS and the U-Net on
all datasets simultaneously (ABC). To justify the necessity for all mechanisms
in our method, as described in Sec. 3, we conduct an ablation study in Tab. 3.
Implementation details and qualitative results including the disentanglement
can be found in the supplementary material and code on
github.com/MECLabTUDA/ACS.

##  5 Result & Discussion

To assess the continual learning performance, we evaluate the results after
stages 1 and 2 corresponding to Fig. 1(a). An ideal algorithm should perform
equally or better on the initial training datasets from epoch 30 to 60 while
it should improve on the third dataset added after 30 epochs.

Stage 1: Tab. 1 shows the results for all methods after 30 epochs on the
initial two datasets. All baselines observe the same score because they apply
the regularization in the second training stage, whereas ACS performs
disentanglement during the initial training phase. ACS outperforms them by a
Dice of $+0.178\pm 0.08$ (IoU $+0.190\pm 0.07$) averaged over all combinations
and datasets.

Table 1: Comparison of all baselines to ACS after 30 epochs. |  | Dataset A | Dataset B | Dataset C | Average
---|---|---|---|---|---
|  | IoU | Dice | IoU | Dice | IoU | Dice | IoU | Dice
AB | Baselines | 0.641 | 0.779 | 0.779 | 0.875 | 0.358 | 0.512 | 0.593 | 0.722
|  ACS (ours) | 0.749 | 0.855 | 0.793 | 0.884 | 0.478 | 0.628 | 0.673 | 0.789
AC | Baselines | 0.080 | 0.147 | 0.265 | 0.416 | 0.380 | 0.547 | 0.241 | 0.370
|  ACS (ours) | 0.646 | 0.782 | 0.731 | 0.844 | 0.727 | 0.841 | 0.702 | 0.822
BC | Baselines | 0.260 | 0.407 | 0.749 | 0.856 | 0.649 | 0.784 | 0.553 | 0.682
|  ACS (ours) | 0.239 | 0.376 | 0.798 | 0.887 | 0.710 | 0.829 | 0.582 | 0.698

Stage 2: To measure overall continual learning performance, i.e., the
combination of learning and forgetting, we inspect the average scores over all
datasets after 60 epochs in Tab. 2. While the comparison methods’ results
fluctuate, our approach achieves a consistently higher performance across all
combinations and datasets. This observation manifests in an increase of the
average Dice score by $+0.127\pm 0.10$ over the U-Net, $+0.156\pm 0.03$ over
the U-Net-b, $+0.118\pm 0.08$ over BS-MAS, and $+0.152\pm 0.14$ over OL-KD. On
combination AB-C, the U-Net drops by an IoU of $>47\%$ (Dice $>35\%$) on
dataset A and by $>26\%$ (Dice $>17\%$) on dataset B. The remaining methods,
including ACS, show a significantly lower decline and effectively reduce
catastrophic forgetting.

Table 2: Comparison of all baselines and ACS after 60 epochs. ABC represents
joint training on all datasets at once.
|  | Dataset A | Dataset B | Dataset C | Average
---|---|---|---|---|---
|  | IoU | Dice | IoU | Dice | IoU | Dice | IoU | Dice
AB-C | U-Net | 0.238 | 0.381 | 0.501 | 0.664 | 0.621 | 0.764 | 0.454 | 0.603
| U-Net-b | 0.339 | 0.503 | 0.570 | 0.724 | 0.473 | 0.631 | 0.461 | 0.619
| BS-MAS | 0.304 | 0.464 | 0.568 | 0.722 | 0.624 | 0.766 | 0.499 | 0.651
| OL-KD | 0.578 | 0.729 | 0.727 | 0.841 | 0.473 | 0.633 | 0.593 | 0.734
|  ACS (ours) | 0.640 | 0.779 | 0.760 | 0.863 | 0.572 | 0.718 | 0.657 | 0.787
AC-B | U-Net | 0.295 | 0.451 | 0.718 | 0.836 | 0.387 | 0.547 | 0.467 | 0.611
| U-Net-b | 0.273 | 0.425 | 0.567 | 0.723 | 0.419 | 0.584 | 0.419 | 0.577
| BS-MAS | 0.307 | 0.466 | 0.702 | 0.825 | 0.364 | 0.523 | 0.458 | 0.604
| OL-KD | 0.094 | 0.171 | 0.381 | 0.549 | 0.400 | 0.571 | 0.292 | 0.430
|  ACS (ours) | 0.681 | 0.808 | 0.787 | 0.880 | 0.679 | 0.808 | 0.716 | 0.832
BC-A | U-Net | 0.745 | 0.852 | 0.668 | 0.800 | 0.418 | 0.579 | 0.610 | 0.743
| U-Net-b | 0.450 | 0.615 | 0.591 | 0.742 | 0.497 | 0.661 | 0.513 | 0.673
| BS-MAS | 0.731 | 0.847 | 0.626 | 0.768 | 0.409 | 0.569 | 0.589 | 0.728
| OL-KD | 0.347 | 0.511 | 0.766 | 0.867 | 0.639 | 0.776 | 0.584 | 0.718
|  ACS (ours) | 0.600 | 0.747 | 0.649 | 0.786 | 0.465 | 0.631 | 0.571 | 0.721
ABC | U-Net | 0.277 | 0.431 | 0.458 | 0.627 | 0.431 | 0.599 | 0.388 | 0.440
(joint) |  ACS (ours) | 0.737 | 0.847 | 0.760 | 0.863 | 0.724 | 0.839 | 0.740 | 0.849

Combination AC-B shows the clear advantage of our approach. Dataset A contains
four types of disorders recorded by a single scanner, while dataset C holds
three disease patterns recorded by three different scanners. The baselines
struggle with the diversity of these domains, and our model outperforms them
by an IoU of $>91\%$ (Dice $>53\%$). These observations show that our model
learns a sufficient content representation that can deal with diverse
cognitive impairments and scans acquired by scanners of various vendors.

We trace back the low performance on dataset A in combination BC-A to A
outnumbering B and C in its variability and number of subjects. The high
performance of the U-Net thereby originates from overfitting on A which
through its high variability still allows it to perform well on B and C.
Because ACS is only fine-tuned on A, it cannot fully exploit this anomaly, but
still shows competitive results.

Ablation Study: The conducted ablation study in Tab. 3 verifies that all
losses contribute to the performance of ACS. Only on AC-B, the combination of
all losses underperforms slightly, but remains competitive. For more detailed
numbers we direct the reader to the supplementary material.

Table 3: Snapshot of the ablation study on ACS trained for 60 epochs with losses active ($\checkmark$) or deactivated (X) during training. Average reported over all test datasets in a configuration. |  |  |  | AB-C |  | AC-B |  | BC-A |  | Average
---|---|---|---|---|---|---|---|---|---|---
$\mathcal{L}_{adv}^{c}\>\>\>\>$ | $\mathcal{L}_{VAE}$ | $\mathcal{L}_{GAN}$ | $\mathcal{L}_{lr}$ | IoU | Dice | IoU | Dice | IoU | Dice | IoU | Dice
X | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.620 | 0.759 | 0.730 | 0.843 | 0.526 | 0.686 | 0.626 | 0.763
$\checkmark$ | X | X | X | 0.574 | 0.721 | 0.709 | 0.849 | 0.524 | 0.681 | 0.603 | 0.750
$\checkmark$ | $\checkmark$ | X | $\checkmark$ | 0.637 | 0.772 | 0.732 | 0.843 | 0.555 | 0.711 | 0.642 | 0.776
$\checkmark$ | X | $\checkmark$ | $\checkmark$ | 0.636 | 0.772 | 0.721 | 0.835 | 0.551 | 0.707 | 0.636 | 0.772
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.658 | 0.787 | 0.716 | 0.832 | 0.572 | 0.721 | 0.649 | 0.780

The results demonstrate that leveraging the availability of multiple datasets
increases multi-domain segmentation performance by sufficiently learning a
domain-invariant representation. This assumption is further supported by the
joint training results in Tab. 2 showing the superior capability of ACS in
comparison to the U-Net. Additionally, our method outperforms the state-of-
the-art on most continual learning setups and effectively reduces catastrophic
forgetting.

##  6 Conclusion

We propose ACS, an architecture for continual semantic segmentation of multi-
domain data that leverages the simultaneous availability of datasets. In real
clinical practice, multiple datasets are available at the beginning of the
continual training process through, among other sources, public or accessible
historical data. Unlike current methods, we leverage this serendipity to
disentangle MRI images into content and domain representations through
adversarial training. We then perform multi-domain hippocampal segmentation
directly on the domain-invariant content representation. We demonstrate
drastic improvements through domain disentanglement of multi-domain data in
the first training stage. In the second training stage, the benefits of our
proposal for continual learning become clear by showcasing that using all
available data reduces catastrophic forgetting and outperforms current state-
of-the-art methods. Our method pushes continual learning closer towards a
clinical application where various degrees of variability such as disease
patterns, scan vendors, and acquisition protocols exist and further enables
the continual usage of deep learning models in clinical practice.

## References

  * [1] Alharbi, Y., Smith, N., Wonka, P.: Latent filter scaling for multimodal unsupervised image-to-image translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
  * [2] Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: Learning what (not) to forget. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. pp. 144–161. Springer International Publishing, Cham (2018)
  * [3] Baweja, C., Glocker, B., Kamnitsas, K.: Towards continual learning in medical imaging. CoRR abs/1811.02496 (2018)
  * [4] Boccardi, M., Bocchetta, M., Morency, F.C., Collins, D.L., Nishikawa, M., Ganzola, R., Grothe, M.J., Wolf, D., Redolfi, A., Pievani, M., Antelmi, L., Fellgiebel, A., Matsuda, H., Teipel, S., Duchesne, S., Jack, C.R., Frisoni, G.B.: Training labels for hippocampal segmentation based on the eadc-adni harmonized hippocampal protocol. Alzheimerś & Dementia 11(2), 175–183 (2015), http://adni.loni.usc.edu/
  * [5] Chartsias, A., Joyce, T., Papanastasiou, G., Semple, S., Williams, M., Newby, D.E., Dharmakumar, R., Tsaftaris, S.A.: Disentangled representation learning in cardiac image analysis. Medical Image Analysis 58, 101535 (2019)
  * [6] Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.: Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. p. 2180–2188. NIPS’16, Curran Associates Inc., Red Hook, NY, USA (2016)
  * [7] Douillard, A., Chen, Y., Dapogny, A., Cord, M.: Plop: Learning without forgetting for continual semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4040–4050 (June 2021)
  * [8] van Garderen, K.A., Voort, S.V.D., Incekara, F., Smits, M., Klein, S.: Towards continuous learning for glioma segmentation with elastic weight consolidation. ArXiv abs/1909.11479 (2019)
  * [9] González, C., Sakas, G., Mukhopadhyay, A.: What is wrong with continual learning in medical image segmentation? CoRR abs/2010.11008 (2020)
  * [10] Hofmanninger, J., Perkonigg, M., Brink, J.A., Pianykh, O., Herold, C., Langs, G.: Dynamic memory to alleviate catastrophic forgetting in continuous learning settings. In: Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., Joskowicz, L. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. pp. 359–368. Springer International Publishing, Cham (2020)
  * [11] Huang, X., Liu, M.Y., Belongie, S., Kautz, J.: Multimodal unsupervised image-to-image translation. In: Proceedings of the European Conference on Computer Vision (ECCV) (September 2018)
  * [12] Jiang, J., Veeraraghavan, H.: Unified cross-modality feature disentangler for unsupervised multi-domain mri abdomen organs segmentation. In: Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., Joskowicz, L. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. pp. 347–358. Springer International Publishing, Cham (2020)
  * [13] Kamnitsas, K., Baumgartner, C., Ledig, C., Newcombe, V., Simpson, J., Kane, A., Menon, D., Nori, A., Criminisi, A., Rueckert, D., Glocker, B.: Unsupervised domain adaptation in brain lesion segmentation with adversarial networks. In: Niethammer, M., Styner, M., Aylward, S., Zhu, H., Oguz, I., Yap, P.T., Shen, D. (eds.) Information Processing in Medical Imaging. pp. 597–609. Springer International Publishing, Cham (2017)
  * [14] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
  * [15] Kazeminia, S., Baur, C., Kuijper, A., van Ginneken, B., Navab, N., Albarqouni, S., Mukhopadhyay, A.: Gans for medical image analysis. CoRR abs/1809.06222 (2018)
  * [16] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: Bengio, Y., LeCun, Y. (eds.) ICLR (2014)
  * [17] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., Hadsell, R.: Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114(13), 3521–3526 (2017)
  * [18] Kulaga-Yoskovitz, J., Bernhardt, B.C., Hong, S.J., Mansi, T., Kevin E. Liang, A.J.v.d.K., Smallwood, J., Bernasconi, A., Bernasconi, N.: Multi-contrast submillimetric 3 tesla hippocampal subfield segmentation protocol and dataset. Scientific Data 2(1), 150059 (2015), https://datadryad.org/stash/dataset/doi:10.5061/dryad.gc72v
  * [19] Lee, H.Y., Tseng, H.Y., Huang, J.B., Singh, M., Yang, M.H.: Diverse image-to-image translation via disentangled representations. In: Proceedings of the European Conference on Computer Vision (ECCV) (September 2018)
  * [20] Lenga, M., Schulz, H., Saalbach, A.: Continual learning for domain adaptation in chest x-ray classification. In: Arbel, T., Ben Ayed, I., de Bruijne, M., Descoteaux, M., Lombaert, H., Pal, C. (eds.) Proceedings of the Third Conference on Medical Imaging with Deep Learning. Proceedings of Machine Learning Research, vol. 121, pp. 413–423. PMLR (06–08 Jul 2020)
  * [21] Li, H., Smith, S.M., Gruber, S., Lukas, S.E., Silveri, M.M., Hill, K.P., Killgore, W.D., Nickerson, L.D.: Denoising scanner effects from multimodal mri data using linked independent component analysis. NeuroImage 208, 116388 (2020)
  * [22] Liu, Y.C., Yeh, Y.Y., Fu, T.C., Wang, S.D., Chiu, W.C., Wang, Y.C.F.: Detach and adapt: Learning cross-domain disentangled deep representation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
  * [23] Michieli, U., Zanuttigh, P.: Incremental learning techniques for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops (Oct 2019)
  * [24] Özgün, S., Rickmann, A.M., Roy, A.G., Wachinger, C.: Importance driven continual learning for segmentation across domains. In: Liu, M., Yan, P., Lian, C., Cao, X. (eds.) Machine Learning in Medical Imaging. pp. 423–433. Springer International Publishing, Cham (2020)
  * [25] Pianykh, O.S., Langs, G., Dewey, M., Enzmann, D.R., Herold, C.J., Schoenberg, S.O., Brink, J.A.: Continuous learning ai in radiology: Implementation principles and early applications. Radiology 297(1), 6–14 (2020), pMID: 32840473
  * [26] Prangemeier, T., Wildner, C., Françani, A.O., Reich, C., Koeppl, H.: Multiclass yeast segmentation in microstructured environments with deep learning. In: 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB). pp. 1–8 (2020)
  * [27] Simpson, A.L., Antonelli, M., Bakas, S., Bilello, M., Farahani, K., van Ginneken, B., Kopp-Schneider, A., Landman, B.A., Litjens, G., Menze, B.H., Ronneberger, O., Summers, R.M., Bilic, P., Christ, P.F., Do, R.K.G., Gollub, M., Golia-Pernicka, J., Heckers, S., Jarnagin, W.R., McHugo, M., Napel, S., Vorontsov, E., Maier-Hein, L., Cardoso, M.J.: A large annotated medical image dataset for the development and evaluation of segmentation algorithms. CoRR abs/1902.09063 (2019), http://medicaldecathlon.com/
  * [28] Sokar, G., Mocanu, D.C., Pechenizkiy, M.: Learning invariant representation for continual learning. CoRR abs/2101.06162 (2021)
  * [29] Yu, X., Ying, Z., Li, G.: Multi-mapping image-to-image translation with central biasing normalization. CoRR abs/1806.10050 (2018)


* Aaij  _et al._ (2020b) R. Aaij _et al._ (LHCb), Phys. Rev. Lett. 125, 242001 (2020b), arXiv:2009.00025 [hep-ex] .
  * Aaij  _et al._ (2020c) R. Aaij _et al._ (LHCb), Phys. Rev. D 102, 112003 (2020c), arXiv:2009.00026 [hep-ex] .
  * Meng  _et al._ (2020a) L. Meng, B. Wang, and S.-L. Zhu, (2020a), arXiv:2012.09813 [hep-ph] .
  * Aaij  _et al._ (2017a) R. Aaij _et al._ (LHCb), Phys. Rev. Lett. 118, 022003 (2017a), arXiv:1606.07895 [hep-ex] .
  * Aaltonen  _et al._ (2009) T. Aaltonen _et al._ (CDF), Phys. Rev. Lett. 102, 242002 (2009), arXiv:0903.2229 [hep-ex] .
  * Liu and Zhu (2009) X. Liu and S.-L. Zhu, Phys. Rev. D 80, 017502 (2009), [Erratum: Phys. Rev. D 85, 019902 (2012)], arXiv:0903.2529 [hep-ph] .
  * Branz  _et al._ (2009) T. Branz, T. Gutsche, and V. E. Lyubovitskij, Phys. Rev. D 80, 054019 (2009), arXiv:0903.5424 [hep-ph] .
  * Albuquerque  _et al._ (2009) R. M. Albuquerque, M. E. Bracco, and M. Nielsen, Phys. Lett. B 678, 186 (2009), arXiv:0903.5540 [hep-ph] .
  * Zhang and Huang (2010) J.-R. Zhang and M.-Q. Huang, J. Phys. G 37, 025005 (2010), arXiv:0905.4178 [hep-ph] .
  * Chen  _et al._ (2015) X. Chen, X. Lü, R. Shi, and X. Guo, (2015), arXiv:1512.06483 [hep-ph] .
  * Karliner and Rosner (2016) M. Karliner and J. L. Rosner, Nucl. Phys. A 954, 365 (2016), arXiv:1601.00565 [hep-ph] .
  * Aaij  _et al._ (2017b) R. Aaij _et al._ (LHCb), Phys. Rev. D 95, 012002 (2017b), arXiv:1606.07898 [hep-ex] .
  * Wang  _et al._ (2018b) E. Wang, J.-J. Xie, L.-S. Geng, and E. Oset, Phys. Rev. D 97, 014017 (2018b), arXiv:1710.02061 [hep-ph] .
  * Wang  _et al._ (2020b) J.-Z. Wang, D.-Y. Chen, X. Liu, and T. Matsuki, (2020b), arXiv:2011.08501 [hep-ph] .
  * Wan and Qiao (2020) B.-D. Wan and C.-F. Qiao, (2020), arXiv:2011.08747 [hep-ph] .
  * Wang  _et al._ (2020c) J.-Z. Wang, Q.-S. Zhou, X. Liu, and T. Matsuki, (2020c), arXiv:2011.08628 [hep-ph] .
  * Meng  _et al._ (2020b) L. Meng, B. Wang, and S.-L. Zhu, Phys. Rev. D 102, 111502 (2020b), arXiv:2011.08656 [hep-ph] .
  * Yang  _et al._ (2020b) Z. Yang, X. Cao, F.-K. Guo, J. Nieves, and M. Pavon Valderrama, (2020b), arXiv:2011.08725 [hep-ph] .
  * Chen and Huang (2020) R. Chen and Q. Huang, (2020), arXiv:2011.09156 [hep-ph] .
  * Du  _et al._ (2020b) M.-C. Du, Q. Wang, and Q. Zhao, (2020b), arXiv:2011.09225 [hep-ph] .
  * Cao  _et al._ (2020) X. Cao, J.-P. Dai, and Z. Yang, (2020), arXiv:2011.09244 [hep-ph] .
  * Sun and Xiao (2020) Z.-F. Sun and C.-W. Xiao, (2020), arXiv:2011.09404 [hep-ph] .
  * Wang  _et al._ (2020d) Q.-N. Wang, W. Chen, and H.-X. Chen, (2020d), arXiv:2011.10495 [hep-ph] .
  * Wang  _et al._ (2020e) B. Wang, L. Meng, and S.-L. Zhu, (2020e), arXiv:2011.10922 [hep-ph] .
  * Wang (2020b) Z.-G. Wang, (2020b), arXiv:2011.10959 [hep-ph] .
  * Azizi and Er (2020) K. Azizi and N. Er, (2020), arXiv:2011.11488 [hep-ph] .
  * Jin  _et al._ (2020) X. Jin, X. Liu, Y. Xue, H. Huang, and J. Ping, (2020), arXiv:2011.12230 [hep-ph] .
  * Simonov (2020) Y. Simonov, (2020), arXiv:2011.12326 [hep-ph] .
  * Süngü  _et al._ (2020) J. Süngü, A. Türkan, H. Sundu, and E. V. Veliev, (2020), arXiv:2011.13013 [hep-ph] .
  * Ikeno  _et al._ (2020) N. Ikeno, R. Molina, and E. Oset, (2020), arXiv:2011.13425 [hep-ph] .
  * Xu  _et al._ (2020) Y.-J. Xu, Y.-L. Liu, C.-Y. Cui, and M.-Q. Huang, (2020), arXiv:2011.14313 [hep-ph] .
  * Aubert  _et al._ (2005) B. Aubert _et al._ (BaBar), Phys. Rev. Lett. 95, 142001 (2005), arXiv:hep-ex/0506081 .
  * He  _et al._ (2006) Q. He _et al._ (CLEO), Phys. Rev. D 74, 091104 (2006), arXiv:hep-ex/0611021 .
  * Yuan  _et al._ (2007) C. Yuan _et al._ (Belle), Phys. Rev. Lett. 99, 182004 (2007), arXiv:0707.2541 [hep-ex] .
  * Ablikim  _et al._ (2016) M. Ablikim _et al._ (BESIII), Phys. Rev. D93, 011102 (2016), arXiv:1511.08564 [hep-ex] .
  * Ablikim  _et al._ (2017a) M. Ablikim _et al._ (BESIII), Phys. Rev. Lett. 118, 092002 (2017a), arXiv:1610.07044 [hep-ex] .
  * Ablikim  _et al._ (2017b) M. Ablikim _et al._ (BESIII), Phys. Rev. Lett. 118, 092001 (2017b), arXiv:1611.01317 [hep-ex] .
  * Ablikim  _et al._ (2019) M. Ablikim _et al._ (BESIII), Phys. Rev. Lett. 122, 102002 (2019), arXiv:1808.02847 [hep-ex] .
  * Gao  _et al._ (2017) X. Gao, C. Shen, and C. Yuan, Phys. Rev. D 95, 092007 (2017), arXiv:1703.10351 [hep-ex] .
  * Qin  _et al._ (2016) W. Qin, S.-R. Xue, and Q. Zhao, Phys. Rev. D 94, 054035 (2016), arXiv:1605.02407 [hep-ph] .
  * Chen  _et al._ (2019b) Y.-H. Chen, L.-Y. Dai, F.-K. Guo, and B. Kubis, Phys. Rev. D 99, 074016 (2019b), arXiv:1902.10957 [hep-ph] .
  * Lu  _et al._ (2017) Y. Lu, M. N. Anwar, and B.-S. Zou, Phys. Rev. D 96, 114022 (2017), arXiv:1705.00449 [hep-ph] .
  * (170) C.-Z. Yuan, “The $Y$ states and other vectors in $e^{+}e^{-}$ annihilations,” https://indico.ihep.ac.cn/event/11793/session/7/contribution/5/material/slides/0.pdf, talk given at the 5th Hadron Physics Online Forum (HAPOF), July 22, 2020.
  * Wang  _et al._ (2014) Q. Wang, M. Cleven, F.-K. Guo, C. Hanhart, U.-G. Meißner, X.-G. Wu, and Q. Zhao, Phys. Rev. D 89, 034001 (2014), arXiv:1309.4303 [hep-ph] .
  * Ma  _et al._ (2015) L. Ma, X.-H. Liu, X. Liu, and S.-L. Zhu, Phys. Rev. D 91, 034032 (2015), arXiv:1406.6879 [hep-ph] .
  * Cleven  _et al._ (2015) M. Cleven, F.-K. Guo, C. Hanhart, Q. Wang, and Q. Zhao, Phys. Rev. D 92, 014005 (2015), arXiv:1505.01771 [hep-ph] .
  * Hanhart and Klempt (2020) C. Hanhart and E. Klempt, Int. J. Mod. Phys. A 35, 2050019 (2020), arXiv:1906.11971 [hep-ph] .
  * Cleven  _et al._ (2014) M. Cleven, Q. Wang, F.-K. Guo, C. Hanhart, U.-G. Meißner, and Q. Zhao, Phys. Rev. D 90, 074039 (2014), arXiv:1310.2190 [hep-ph] .
  * Olschewsky (2018) K. Olschewsky, “Heavy hadronic molecules with negative parity: The vector states,” (2018), master thesis, Bonn University.
  * Lees  _et al._ (2014) J. Lees _et al._ (BaBar), Phys. Rev. D 89, 111103 (2014), arXiv:1211.6271 [hep-ex] .
  * Wang  _et al._ (2015) X. Wang _et al._ (Belle), Phys. Rev. D 91, 112007 (2015), arXiv:1410.7641 [hep-ex] .
  * Guo  _et al._ (2008) F.-K. Guo, C. Hanhart, and U.-G. Meißner, Phys. Lett. B 665, 26 (2008), arXiv:0803.1392 [hep-ph] .
  * Cotugno  _et al._ (2010) G. Cotugno, R. Faccini, A. Polosa, and C. Sabelli, Phys. Rev. Lett. 104, 132005 (2010), arXiv:0911.2178 [hep-ph] .
  * Guo  _et al._ (2010) F.-K. Guo, J. Haidenbauer, C. Hanhart, and U.-G. Meißner, Phys. Rev. D 82, 094008 (2010), arXiv:1005.2055 [hep-ph] .
  * Pakhlova  _et al._ (2008) G. Pakhlova _et al._ (Belle), Phys. Rev. Lett. 101, 172001 (2008), arXiv:0807.4458 [hep-ex] .
  * Ablikim  _et al._ (2018) M. Ablikim _et al._ (BESIII), Phys. Rev. Lett. 120, 132001 (2018), arXiv:1710.00150 [hep-ex] .
  * Jia  _et al._ (2019) S. Jia _et al._ (Belle), Phys. Rev. D 100, 111103 (2019), arXiv:1911.00671 [hep-ex] .
  * Jia  _et al._ (2020) S. Jia _et al._ (Belle), Phys. Rev. D 101, 091101 (2020), arXiv:2004.02404 [hep-ex] .
  * He  _et al._ (2020) J. He, Y. Liu, J.-T. Zhu, and D.-Y. Chen, Eur. Phys. J. C 80, 246 (2020), arXiv:1912.08420 [hep-ph] .
  * Ablikim  _et al._ (2020b) M. Ablikim _et al._ (BESIII), Chin. Phys. C 44, 040001 (2020b), arXiv:1912.05983 [hep-ex] .
  * Sommerfeld (1931) A. Sommerfeld, Annalen der Physik 403, 257 (1931).
  * Dai  _et al._ (2017) L.-Y. Dai, J. Haidenbauer, and U.-G. Meißner, Phys. Rev. D 96, 116001 (2017), arXiv:1710.03142 [hep-ph] .
  * Barniakov (2019) A. Y. Barniakov (Super Charm-Tau Factory), PoS LeptonPhoton2019, 062 (2019).
  * Peng  _et al._ (2020b) H.-P. Peng, Y.-H. Zheng, and X.-R. Zhou, Physics 49, 513 (2020b).
  * Wu  _et al._ (2012a) J.-J. Wu, L. Zhao, and B. Zou, Phys. Lett. B 709, 70 (2012a), arXiv:1011.5743 [hep-ph] .
  * Wang  _et al._ (2011) W. Wang, F. Huang, Z. Zhang, and B. Zou, Phys. Rev. C 84, 015203 (2011), arXiv:1101.0453 [nucl-th] .
  * Wu  _et al._ (2012b) J.-J. Wu, T.-S. Lee, and B. Zou, Phys. Rev. C 85, 044002 (2012b), arXiv:1202.1036 [nucl-th] .
  * Xiao  _et al._ (2013) C. Xiao, J. Nieves, and E. Oset, Phys. Rev. D 88, 056012 (2013), arXiv:1304.5368 [hep-ph] .
  * Karliner and Rosner (2015) M. Karliner and J. L. Rosner, Phys. Rev. Lett. 115, 122001 (2015), arXiv:1506.06386 [hep-ph] .
  * Aaij  _et al._ (2015) R. Aaij _et al._ (LHCb), Phys. Rev. Lett. 115, 072001 (2015), arXiv:1507.03414 [hep-ex] .
  * Liu  _et al._ (2019b) M.-Z. Liu, Y.-W. Pan, F.-Z. Peng, M. Sánchez Sánchez, L.-S. Geng, A. Hosaka, and M. Pavon Valderrama, Phys. Rev. Lett. 122, 242001 (2019b), arXiv:1903.11560 [hep-ph] .
  * Xiao  _et al._ (2019a) C. Xiao, J. Nieves, and E. Oset, Phys. Rev. D 100, 014021 (2019a), arXiv:1904.01296 [hep-ph] .
  * Du  _et al._ (2020c) M.-L. Du, V. Baru, F.-K. Guo, C. Hanhart, U.-G. Meißner, J. A. Oller, and Q. Wang, Phys. Rev. Lett. 124, 072001 (2020c), arXiv:1910.11846 [hep-ph] .
  * Liu  _et al._ (2018) M.-Z. Liu, F.-Z. Peng, M. Sánchez Sánchez, and M. Pavon Valderrama, Phys. Rev. D 98, 114030 (2018), arXiv:1811.03992 [hep-ph] .
  * Sakai  _et al._ (2019) S. Sakai, H.-J. Jing, and F.-K. Guo, Phys. Rev. D 100, 074007 (2019), arXiv:1907.03414 [hep-ph] .
  * Hofmann and Lutz (2005) J. Hofmann and M. Lutz, Nucl. Phys. A 763, 90 (2005), arXiv:hep-ph/0507071 .
  * Chen  _et al._ (2017) R. Chen, J. He, and X. Liu, Chin. Phys. C 41, 103105 (2017), arXiv:1609.03235 [hep-ph] .
  * Anisovich  _et al._ (2015) V. Anisovich, M. Matveev, J. Nyiri, A. Sarantsev, and A. Semenova, Int. J. Mod. Phys. A 30, 1550190 (2015), arXiv:1509.04898 [hep-ph] .
  * Wang (2016) Z.-G. Wang, Eur. Phys. J. C 76, 142 (2016), arXiv:1509.06436 [hep-ph] .
  * Feijoo  _et al._ (2016) A. Feijoo, V. Magas, A. Ramos, and E. Oset, Eur. Phys. J. C 76, 446 (2016), arXiv:1512.08152 [hep-ph] .
  * Lu  _et al._ (2016) J.-X. Lu, E. Wang, J.-J. Xie, L.-S. Geng, and E. Oset, Phys. Rev. D 93, 094009 (2016), arXiv:1601.00075 [hep-ph] .
  * Xiao  _et al._ (2019b) C. Xiao, J. Nieves, and E. Oset, Phys. Lett. B 799, 135051 (2019b), arXiv:1906.09010 [hep-ph] .
  * Chen  _et al._ (2016b) H.-X. Chen, L.-S. Geng, W.-H. Liang, E. Oset, E. Wang, and J.-J. Xie, Phys. Rev. C 93, 065203 (2016b), arXiv:1510.01803 [hep-ph] .
  * Wang  _et al._ (2020f) B. Wang, L. Meng, and S.-L. Zhu, Phys. Rev. D 101, 034018 (2020f), arXiv:1912.12592 [hep-ph] .
  * Zhang  _et al._ (2020) Q. Zhang, B.-R. He, and J.-L. Ping, (2020), arXiv:2006.01042 [hep-ph] .
  * Liu  _et al._ (2020) M.-Z. Liu, Y.-W. Pan, and L.-S. Geng, (2020), arXiv:2011.07935 [hep-ph] .
  * Chen (2020) R. Chen, (2020), arXiv:2011.07214 [hep-ph] .
  * Wang (2020c) Z.-G. Wang, (2020c), arXiv:2011.05102 [hep-ph] .
  * Peng  _et al._ (2020c) F.-Z. Peng, M.-J. Yan, M. Sánchez Sánchez, and M. Pavon Valderrama, (2020c), arXiv:2011.01915 [hep-ph] .
  * Chen  _et al._ (2020) H.-X. Chen, W. Chen, X. Liu, and X.-H. Liu, (2020), arXiv:2011.01079 [hep-ph] .
  * Wang  _et al._ (2020g) F.-L. Wang, R. Chen, and X. Liu, (2020g), arXiv:2011.14296 [hep-ph] .
  * Liu  _et al._ (2019c) M.-Z. Liu, T.-W. Wu, M. Sánchez Sánchez, M. P. Valderrama, L.-S. Geng, and J.-J. Xie, (2019c), arXiv:1907.06093 [hep-ph] .