11institutetext: Baidu, Inc., Beijing, China22institutetext: Harvard
University, Cambridge, MA

# MUSCLE: Multi-task Self-supervised Continual Learning to Pre-train Deep
Models for X-ray Images of Multiple Body Parts

Weibin Liao 11 Haoyi Xiong (✉) 11 0000-0002-5451-3253 Qingzhong Wang 11 Yan Mo
11 Xuhong Li 11 Yi Liu 11 Zeyu Chen 11 Siyu Huang 22 Dejing Dou 11

###### Abstract

While self-supervised learning (SSL) algorithms have been widely used to pre-
train deep models, few efforts [11] have been done to improve representation
learning of X-ray image analysis with SSL pre-trained models. In this work, we
study a novel self-supervised pre-training pipeline, namely _Mu lti-task Self-
super-vised Continual Learning_ (MUSCLE), for multiple medical imaging tasks,
such as classification and segmentation, using X-ray images collected from
multiple body parts, including heads, lungs, and bones. Specifically, MUSCLE
aggregates X-rays collected from multiple body parts for MoCo-based
representation learning, and adopts a well-designed continual learning (CL)
procedure to further pre-train the backbone subject various X-ray analysis
tasks jointly. Certain strategies for image pre-processing, learning
schedules, and regularization have been used to solve _data heterogeneity_ ,
_over-fitting_ , and _catastrophic forgetting_ problems for multi-task/dataset
learning in MUSCLE.We evaluate MUSCLE using 9 real-world X-ray datasets with
various tasks, including pneumonia classification, skeletal abnormality
classification, lung segmentation, and tuberculosis (TB) detection.
Comparisons against other pre-trained models [7] confirm the _proof-of-
concept_ that self-supervised multi-task/dataset continual pre-training could
boost the performance of X-ray image analysis.

###### Keywords:

X-ray images (X-ray) Self-supervised Learning.

##  1 Introduction

While deep learning-based solutions [1] have achieved great success in medical
image analysis, such as X-ray image classification and segmentation tasks for
diseases in bones, lungs and heads, it might require an extremely large number
of images with fine annotations to train the deep neural network (DNN) models
and deliver decent performance [2] in a supervised learning manner. To lower
the sample size required, the self-supervised learning (SSL) paradigm has been
recently proposed to boost the performance of DNN models through learning
visual features from images [3] without using labels.

Among a wide range of SSL methods, contrastive learning [4] algorithms use a
similarity-based metric to measure the distance between two embeddings derived
from two different views of a single image, where the views of image are
generated through data augmentation, e.g., rotation, clip, and shift, and
embeddings are extracted from the DNN with learnable parameters. In
particular, for computer vision tasks, the contrastive loss is computed using
the feature representations of the images extracted from the encoder network,
resulting in the clustering of similar samples together and the dispersal of
different samples. Recent methods such as SwAV [5], SimCLR [6], MoCo [7], and
PILR [8] have been proposed to outperform supervised learning methods on
natural images. While contrastive learning methods have demonstrated promising
results on natural image classification tasks, the attempts to leverage them
in medical image analysis are often limited [9, 10]. While Sowrirajan et al.
[11] proposed MoCo-CXR that can produce models with better representations for
the detection of pathologies in chest X-rays using MoCo [7], the superiority
of SSL for _other X-ray analysis tasks, such as detection and segmentation, on
various body parts, such as lung and bones, is not yet known_.

####  1.0.1 Our Contributions

In this work, we proposed MUSCLE– _MU lti-task Self-supervised Continual
LEarning_ (shown in Fig. 1) that pre-trains deep models using X-ray images
collected from multiple body parts subject to various tasks, and made
contributions as follow.

  1. 1.

We study the problem of multi-dataset/multi-task SSL for X-ray images. To best
of our knowledge, only few works have been done in related area [11, 12],
especially by addressing _data heterogeneity_ (e.g., image sizes, resolutions,
and gray-scale distributions), _over-fitting_ (e.g., to any one of the tasks),
and _catastrophic forgetting_ (e.g., ejection of knowledge learned previously)
in multi-dataset/multi-task learning settings.

  2. 2.

We present MUSCLE that pre-trains the backbone in multi-dataset/multi-task
learning settings with task-specific heads, including Fully-Connected (FC)
Layer, DeepLab-V3 [13], and FasterRCNN [14] for classification, segmentation,
and abnormal detection tasks respectively. As shown in Fig. 1, MUSCLE (1)
adopts MoCo to learn representations with a backbone network from multiple
datasets, with pre-processing to tackle _data heterogeneity_ , and further (2)
pre-trains the backbone to learn discriminative features with continual
learning (CL) subject to multiple tasks, while avoiding over-fitting and
“catastrophic forgetting” [15, 16]. MUSCLE finally (3) fine-tunes the pre-
trained backbone to adapt every task independently and separately.

  3. 3.

In this work, we collect 9 X-ray datasets to evaluate MUSCLE, where we pre-
train and fine-tune the network to adapt all tasks using training subsets, and
validate the performance using testing subsets. The experimental results show
MUSCLE outperforms ImageNet/MoCo pre-trained backbones [11] using both
ResNet-18 and ResNet-50 as backbones. The comparisons confirm the _proof-of-
concept_ of MUSCLE–self-supervised multi-task/dataset continual pre-training
can boost performance of X-ray image analysis.

Figure 1: The pipeline of MUSCLE, consists of three parts (1) Multi-Dataset
Momentum Contrastive (Multi-Dataset MoCo) Learning, (2) Multi-Task Continual
Learning and (3) Fine-tuning on Downstream Tasks.

##  2 Methodologies

In this section, we present the framework and algorithms design for MUSCLE. As
was shown in Fig. 1, MUSCLE consists of three steps as follows.

  1. 1.

_Multi-dataset momentum contrastive learning._ (MD-MoCo) Give multiple
d-atasets of X-ray images collected from different body parts and based on
different image resolution and gray-scale distributions, MUSCLE aggregates
these datasets with pre-processing (e.g., resize, re-scale, and
normalization), then adopts a MoCo-based SSL algorithm [11] to pre-train the
backbone networks and learn the representation of X-ray images within the
aggregated dataset.

  2. 2.

_Multi-task continual learning._ Given the MD-MoCo pre-trained backbone and
the datasets for different X-ray analysis tasks, e.g., pneumonia
classification, skeletal abnormality classification, and lung segmentation,
MUSCLE leverages continual learning algorithms and further pre-trains a
unified backbone network with alternating heads to learn the various sets of
discriminative features subject to the different tasks.

  3. 3.

_Fine-tuning with downstream tasks._ Given the pre-trained backbone and every
downstream X-ray image analysis task, MUSCLE fine-tunes and outputs a neural
network using pre-trained weights as the initialization and its own task-
specific head to fit the task independently.

Table 1: An Overview of Nine Publicly Available X-ray Image Datasets Datasets | Body Part | Task | Train | Valid | Test | Total
---|---|---|---|---|---|---
Only Used for the first step (MD-MoCo) of MUSCLE
NIHCC [17] | Chest | N/A | 112,120 | N/A | N/A | 112,120
China-Set-CXR [18] | Chest | N/A | 661 | N/A | N/A | 661
Montgomery-Set-CXR [18] | Chest | N/A | 138 | N/A | N/A | 138
Indiana-CXR [19] | Chest | N/A | 7,470 | N/A | N/A | 7,470
RSNA Bone Age [20] | Hand | N/A | 10,811 | N/A | N/A | 10,811
Used for all three steps of MUSCLE
Pneumonia [21] | Chest |  | Classification
---
4,686 | 585 | 585 | 5,856
MURA [22] |  | Various Bones
---
| Classification
---
32,013 | 3,997 | 3,995 | 40,005
Chest Xray Masks.. [18] | Chest |  | Segmentation
---
718 | 89 | 89 | 896
TBX [23] | Chest |  | Detection
---
640 | 80 | 80 | 800
Total | N/A | N/A | 169,257 | 4,751 | 4,749 | 178,757

###  2.1 Multi-Dataset Momentum Contrastive Learning (MD-MoCo)

To pre-train the backbone with multiple datasets, MUSCLE collects and
aggregates nine X-ray image datasets listed in Tab 1. While these datasets
have offered nearly 179,000 X-ray images covering cover several parts of the
human body, including the chest, hand, elbow, finger, forearm, humerus,
shoulder and wrist, MUSCLE makes non-trivial extension to adopt MoCo [4, 11]
for _multi-dataset momentum contrastive learning_ as follows.

##### Re-sizing, Re-scaling, Normalization, and Aggregation

The major challenge to work with multiple X-ray image datasets collected from
different body part is the heterogeneity of images, including X-ray image
resolutions and the distribution of gray-scales. To aggregate these datasets,
several image pre-processing schemes have been used, where MUSCLE transforms
and normalizes the gray-scale distribution of these datasets using the Z-score
method with the mean of 122.786 and a standard deviation of 18.390. Further to
fully utilize GPU for the resource-consuming MoCo algorithms, MUSCLE re-sizes
all images into a 800$\times$500 resolution which balances the effectiveness
and efficiency of deep learning.

##### MoCo-based Pre-training with Aggregated Datasets

To utilize MoCo algorithm [4] for X-ray images pre-training, MUSCLE further
extends MoCo-CXR [11] with advanced settings on data augmentation strategies
and initialization tricks. Specifically, while MoCo uses random data
augmentation to generate contrastive views, MUSCLE disables random cropping,
gaussian blurring, color/gray-scale jittering to preserve the semantic
information for medical images. Furthermore, MUSCLE initializes the MoCo-based
pre-training procedure with Kaiming’s initialization [24] to setup the
convolution layers, so as to ensure the stability of back-propagation in
contrastive learning.

Note that, in this work, the training sets of all nine datasets have been used
for _multi-dataset momentum contrastive learning_ , while four of them with
specific X-ray image analysis tasks are further used for _multi-task continual
learning_.

###  2.2 Multi-Task Continual Learning

To further pre-train the backbone subject to task-specific representations,
MUSCLE adopts the continual learning (CL) [25] with four X-ray image analysis
tasks, including pneumonia classification from Pneumonia [21], skeletal
abnormality classification from MURA [22], lung segmentation from Chest Xray
Masks and Labels [18], and Tuberculosis(TB) detection from TBX [23].
Specifically, MUSCLE extends the vanilla CL for neural network, which iterates
the training procedures of the backbone network with alternating task-specific
heads subject to tasks, with two major advancements as follows.

##### Cyclic and Reshuffled Learning Schedule

MUSCLE splits the continual learning procedure into 10 rounds of learning
process, where each round of learning process iterates the 4 learning tasks
one-by-one and each iterate of learning task trains the backbone network with
1 epoch using a task-specific head, i.e., Fully-Connected (FC) Layer for
classification tasks, DeepLab-V3 [13] for segmentation tasks, and FasterRCNN
[14] for detection tasks. Furthermore, to avoid overfitting to any task,
MUSCLE reshuffle the order of tasks in every round of learning process, and
adopts Cosine annealing learning rate schedule as follow.

$\eta_{t}=\eta_{\min}^{i}+\frac{1}{2}\left(\eta_{\max}-\eta_{\min}\right)\left(1+\cos\left(\frac{t}{T}\cdot
2\pi\right)\right)$ (1)

where $\eta_{t}$ refers to the learning rate of the $t^{th}$ iteration,
$\eta_{\mathrm{max}}$ and $\eta_{\mathrm{min}}$ and $T_{i}$ refer to the
maximal and minimal learning rates, and $T$ refers to the total number of
iterations within a period of cyclic learning rate schedule.

##### Cross-Task Memorization with Explicit Bias

Yet another challenge of multi-task CL is “catastrophic forgetting", where the
backbone would “forget” the knowledge learned from the previous iterates. To
solve the problem, MUSCLE leverages a knowledge transfer regularization
derived from L2-SP [15]. In each iterate of learning task, given the pre-
trained model obtained from previous iterates, MUSCLE sets the pre-trained
weights as $\boldsymbol{w}^{0}_{\mathcal{S}}$ and trains the backbone using
the following loss,

$\Omega(\boldsymbol{w}_{\mathcal{S}})=\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\overbrace{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\alpha}\left\|\mathchoice{\leavevmode\hbox
to10.54pt{\vbox to6.22pt{\pgfpicture\makeatletter\hbox{\hskip
5.26968pt\lower-1.91333pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-5.26968pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\hbox{\pagecolor{xkcdHunterGreen!17}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}$\boldsymbol{w}_{\mathcal{S}}$}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to10.54pt{\vbox to6.22pt{\pgfpicture\makeatletter\hbox{\hskip
5.26968pt\lower-1.91333pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-5.26968pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\hbox{{$\textstyle\hbox{\pagecolor{xkcdHunterGreen!17}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}$\boldsymbol{w}_{\mathcal{S}}$}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to7.38pt{\vbox to4.35pt{\pgfpicture\makeatletter\hbox{\hskip
3.68877pt\lower-1.33932pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.68877pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\hbox{\pagecolor{xkcdHunterGreen!17}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}$\boldsymbol{w}_{\mathcal{S}}$}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to5.27pt{\vbox to3.11pt{\pgfpicture\makeatletter\hbox{\hskip
2.63483pt\lower-0.95667pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.63483pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\hbox{\pagecolor{xkcdHunterGreen!17}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}$\boldsymbol{w}_{\mathcal{S}}$}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}-\mathchoice{\leavevmode\hbox
to10.54pt{\vbox to8.02pt{\pgfpicture\makeatletter\hbox{\hskip
5.26968pt\lower-1.91333pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-5.26968pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\hbox{\pagecolor{Bittersweet!17}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}$\boldsymbol{w}_{\mathcal{S}}^{0}$}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to10.54pt{\vbox to8.02pt{\pgfpicture\makeatletter\hbox{\hskip
5.26968pt\lower-1.91333pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-5.26968pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\hbox{{$\textstyle\hbox{\pagecolor{Bittersweet!17}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}$\boldsymbol{w}_{\mathcal{S}}^{0}$}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to7.38pt{\vbox to5.62pt{\pgfpicture\makeatletter\hbox{\hskip
3.68877pt\lower-1.33932pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.68877pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\hbox{\pagecolor{Bittersweet!17}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}$\boldsymbol{w}_{\mathcal{S}}^{0}$}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to5.27pt{\vbox to4.01pt{\pgfpicture\makeatletter\hbox{\hskip
2.63483pt\lower-0.95667pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.63483pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\hbox{\pagecolor{Bittersweet!17}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}$\boldsymbol{w}_{\mathcal{S}}^{0}$}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\right\|_{2}^{2}}^{\begin{subarray}{c}\text{\sf\scriptsize{\color[rgb]{0.7875,0.15,0.3625}\definecolor[named]{pgfstrokecolor}{rgb}{0.7875,0.15,0.3625}
distance to the model }}\\\
\text{\sf\scriptsize{\color[rgb]{0.7875,0.15,0.3625}\definecolor[named]{pgfstrokecolor}{rgb}{0.7875,0.15,0.3625}
pre-trained by previous iterates.
}}\end{subarray}}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}+\quad\color[rgb]{0.0600000000000001,0.46,1}\definecolor[named]{pgfstrokecolor}{rgb}{0.0600000000000001,0.46,1}\pgfsys@color@cmyk@stroke{0.94}{0.54}{0}{0}\pgfsys@color@cmyk@fill{0.94}{0.54}{0}{0}\overbrace{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{(1-\alpha)}\left\|\mathchoice{\leavevmode\hbox
to10.54pt{\vbox to6.22pt{\pgfpicture\makeatletter\hbox{\hskip
5.26968pt\lower-1.91333pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-5.26968pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\hbox{\pagecolor{xkcdHunterGreen!17}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}$\boldsymbol{w}_{\mathcal{S}}$}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to10.54pt{\vbox to6.22pt{\pgfpicture\makeatletter\hbox{\hskip
5.26968pt\lower-1.91333pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-5.26968pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\hbox{{$\textstyle\hbox{\pagecolor{xkcdHunterGreen!17}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}$\boldsymbol{w}_{\mathcal{S}}$}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to7.38pt{\vbox to4.35pt{\pgfpicture\makeatletter\hbox{\hskip
3.68877pt\lower-1.33932pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.68877pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\hbox{\pagecolor{xkcdHunterGreen!17}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}$\boldsymbol{w}_{\mathcal{S}}$}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to5.27pt{\vbox to3.11pt{\pgfpicture\makeatletter\hbox{\hskip
2.63483pt\lower-0.95667pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.63483pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\hbox{\pagecolor{xkcdHunterGreen!17}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}$\boldsymbol{w}_{\mathcal{S}}$}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\right\|_{2}^{2}}^{\begin{subarray}{c}\text{\sf\scriptsize{\color[rgb]{0.201,0.541,1}\definecolor[named]{pgfstrokecolor}{rgb}{0.201,0.541,1}\pgfsys@color@cmyk@stroke{0.799}{0.459}{0}{0}\pgfsys@color<EMAIL_ADDRESS>based }}\\\
\text{\sf\scriptsize{\color[rgb]{0.201,0.541,1}\definecolor[named]{pgfstrokecolor}{rgb}{0.201,0.541,1}\pgfsys@color@cmyk@stroke{0.799}{0.459}{0}{0}\pgfsys@color<EMAIL_ADDRESS>regularization.
}}\end{subarray}}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}$
(2)

outcome of CLpre-trained by previous iterates

where $\boldsymbol{w}_{\mathcal{S}}$ is the learning outcome and $\alpha$ is
the hyper-parameter. Thus, above regularization constrains the distance
between the learning outcome and the backbone trained by the previous
iterates.

###  2.3 Fine-tuning on Downstream Tasks

Finally, given the backbone pre-trained using above two steps, MUSCLE fine-
tunes the backbone on each of the four tasks independent and separately.
Again, MUSCLE connects the pre-trained backbone with Fully-Connected (FC)
Layer for classification tasks, DeepLab-V3 [13] for segmentation tasks, and
FasterRCNN [14] for detection tasks. Finally, MUSCLE employs the standard
settings, such as vanilla weight decay as stability regularization and step-
based decay of learning rate schedule, for such fine-tuning.

Table 2: Performance Comparisons for Pneumonia Classification (Pneumonia) and Skeletal Abnormality Classification (MURA) using various Pre-training Algorithms. Datasets | Backbones | Pre-train | Acc. | Sen. | Spe. | AUC (95%CI)
---|---|---|---|---|---|---
Pneumonia |  ResNet-18 |  Scratch | 91.11 | 93.91 | 83.54 | 96.58 (95.09-97.81)
ImageNet | 90.09 | 93.68 | 80.38 | 96.05 (94.24-97.33)
MD-MoCo | 96.58 | 97.19 | 94.94 | 98.48 (97.14-99.30)
MUSCLE– | 96.75 | 97.66 | 94.30 | 99.51 (99.16-99.77)
MUSCLE | 97.26 | 97.42 | 96.84 | 99.61 (99.32-99.83)
ResNet-50 |  Scratch | 91.45 | 92.51 | 88.61 | 96.55 (95.08-97.82)
ImageNet | 95.38 | 95.78 | 94.30 | 98.72 (98.03-99.33)
MD-MoCo | 97.09 | 98.83 | 92.41 | 99.53 (99.23-99.75)
MUSCLE– | 96.75 | 98.36 | 92.41 | 99.58 (99.30-99.84)
MUSCLE | 98.12 | 98.36 | 97.47 | 99.72 (99.46-99.92)
MURA |  ResNet-18 |  Scratch | 81.00 | 68.17 | 89.91 | 86.62 (85.73-87.55)
ImageNet | 81.88 | 73.49 | 87.70 | 88.11 (87.18-89.03)
MD-MoCo | 82.48 | 72.27 | 89.57 | 88.28 (87.28-89.26)
MUSCLE– | 82.45 | 74.16 | 88.21 | 88.41 (87.54-89.26)
MUSCLE | 82.62 | 74.28 | 88.42 | 88.50 (87.46-89.57)
ResNet-50 |  Scratch | 80.50 | 65.42 | 90.97 | 86.22 (85.22-87.35)
ImageNet | 81.73 | 68.36 | 91.01 | 87.87 (86.85-88.85)
MD-MoCo | 82.35 | 73.12 | 88.76 | 87.89 (87.06-88.88)
MUSCLE– | 81.10 | 69.03 | 89.48 | 87.14 (86.10-88.22)
MUSCLE | 82.60 | 74.53 | 88.21 | 88.37 (87.38-89.32)

##  3 Experiment

In this section, we present our experimental results to confirm the
effectiveness of MUSCLE on various tasks.

##### Experiment Setups

In this experiments, we include performance comparisons on pneumonia
classification (Pneumonia) [21], skeletal abnormality classification (MURA)
[22], lung segmentation (Lung) [18], and tuberculosis detection (TBX) [23]
tasks. We use the training sets of these datasets/tasks to pre-train and fine-
tune the network with MUSCLE or other baseline algorithms, tune the hyper-
parameters using validation sets, and report results on testing datasets.

Figure 2: Receiver Operating Characteristic (ROC) Curves with AUC value
(95%CI) on pneumonia classification (Pne) and skeletal abnormality
classification (MURA)

As the goal of MUSCLE is to pre-train backbones under multi-task/dataset
settings, we propose several baselines for comparisons as follows. Scratch:
the models are all initialized using Kaiming’s random initialization [24] and
fine-tuned on the target datasets (introduced in Section 2.3). ImageNet: the
models are initialized using the officially-released weights pre-trained by
the ImageNet dataset and fine-tuned on the target datasets. MD-MoCo: the
models are pre-trained using _multi-dataset MoCo_ (introduced in Section 2.2)
and fine-tuned accordingly; we believe MD-MoCo is one of our proposed methods
and can prove the concepts, as MD-MoCo extends both vanilla MoCo [7] and MoCo-
CXR [11] with additional data pre-processing methods to tackle the multi-
dataset issues. MUSCLE–: all models are pre-trained and fine-tuned with MUSCLE
but with _Cross-Task Memorization_ and _Cyclic and Reshuffled Learning
Schedule_ turned off. All models here are built with ResNet-18 and ResNet-50.

To compare these algorithms, we evaluate and compare _Accuracy_ (Acc.), _Area
under the Curve_ (AUC), _Sensitivity_ (Sen.), _Specificity_ (Sep.) and
_Receiver Operating Characteristic (ROC) Curve_ for Pneumonia and MURA
classification tasks, _Dice Similarity Coefficient_ (Dice), _mean Intersection
over Union_ (mIoU) for lung segmentation (Lung) task, and _mean Average
Precision_ (mAP), _Average Precision of Active TB_
($\mathrm{AP}_{\mathrm{active}}$) and _Average Precision of Latent TB_
($\mathrm{AP}_{\mathrm{latent}}$) for tuberculosis detection (TBX), all of
them at the IoU threshold of 0.5. AUC values are reported with 95% confidence
intervals estimated through bootstrapping with 100 independent trials on
testing sets.

Figure 3: Results using various pre-training algorithms on lung segmentation,
where green lines indicate ground truth and red areas indicate model
prediction results, and the blue and orange boxes cover regions of the main
differences in these results

##### Overall Comparisons

We present the results on Pneumonia and MURA classification tasks in Table 2
and Fig. 2, where we compare the performance of MUSCLE with all baseline
algorithms and plot the ROC curve. Results show that our proposed methods,
including MUSCLE, MUSCLE– and MD-MoCo can significantly outperform the one
based on ImageNet pre-trained models and Scratch. Furthermore, in terms of
overall accuracy (Acc. and AUC), MUSCLE outperforms MD-MoCo and MUSCLE– in all
cases, due to its advanced continual learning settings. However, we can still
observe a slightly lower sensitivity on the Pneumonia classification task and
a slightly lower specificity on the MURA delivered by MUSCLE. Similar
observations could be found in Table 3 and Fig. 3, where we report the
performance on lung segmentation and TB detection tasks with examples of lung
segmentation plotted.

##### More Comparisons

Please note that MD-MoCo indeed surpasses MoCo [7] and MoCo-CXR [11] (state of
the art in X-rays pre-training), in terms of performance, as it uses multiple
datasets for pre-training and solves the data heterogeneity problem (the
performance of MoCo might be even worse than ImageNet-based pre-training, if
we don’t unify the gray-scale distributions of X-ray images collected from
different datasets). We also have tried to replace MoCo with SimCLR [6], the
performance of SimCLR for X-ray pre-training is even worse, e.g., 87.52%
(9.57%$\downarrow$) Acc. for pneumonia classification.

Furthermore, though MUSCLE is proposed as a _“proof-of-concept”_ solution and
has not been optimized for any single medical imaging task, e.g., using U-Net
for segmentation [26], the overall performance of MUSCLE is still better than
many recent works based on the same datasets. For example, _Stephen et al.
(2019)_ [27] reports a 93.73% (4.42%$\downarrow$) accuracy for the same
pneumonia classification task and _Li et al. (2021)_ [28] reports a 94.64%
Dice (0.73%$\downarrow$) for the lung segmentation. For MURA, _Bozorgtabar et
al. (2020)_ [29] reports an AUC of 82.45% (6.05%$\downarrow$), while _Liu et
al. (2020)_ [23] reports a 58.70% (4.76%$\downarrow$)
$\mathrm{AP}_{\mathrm{active}}$ and a 9.60% (2.61%$\downarrow$)
$\mathrm{AP}_{\mathrm{latent}}$ for TBX based on FasterRNN. The advantages of
MUSCLE demonstrate the feasibility of using multiple tasks/datasets to pre-
train the backbone with SSL and CL. For more results, please refer to the
appendix.

##### Ablation Studies

The comparisons between Scratch versus MD-MoCo, between MD-MoCo versus
MUSCLE–, and between MUSCLE– versus MUSCLE confirm the contributions made by
each step of our algorithms. Frankly speaking, in many cases, MD-MoCo achieved
the most performance improvement (compared to ImageNet or Scratch), while
continual learning without _Cyclic & Reshuffled Learning Schedule_ and _Cross-
Task Memorization_ may even make MUSCLE– performs worse, due to over-fitting
or catastrophic forgetting. However, MUSCLE, pipelining MD-MoCo and advanced
continual learning strategies, successfully defends her dominant position and
outperforms other algorithms in most cases.

Table 3: Performance Comparisons for Lung Segmentation(Lung) and TB Dtection(TBX) using various Pre-training Algorithms. Backbones |  Pre-train | Lung | TBX
---|---|---|---
Dice | mIoU | mAP | APActive | APLatent
ResNet-18 |  Scratch | 95.24 | 94.00 | 30.71 | 56.71 | 4.72
ImageNet | 95.26 | 94.10 | 29.46 | 56.27 | 2.66
MD-MoCo | 95.31 | 94.14 | 36.00 | 67.17 | 4.84
MUSCLE– | 95.14 | 93.90 | 34.70 | 63.43 | 5.97
MUSCLE | 95.37 | 94.22 | 36.71 | 64.84 | 8.59
ResNet-50 |  Scratch | 93.52 | 92.03 | 23.93 | 44.85 | 3.01
ImageNet | 93.77 | 92.43 | 35.61 | 58.81 | 12.42
MD-MoCo | 94.33 | 93.04 | 36.78 | 64.37 | 9.18
MUSCLE– | 95.04 | 93.82 | 35.14 | 57.32 | 12.97
MUSCLE | 95.27 | 94.10 | 37.83 | 63.46 | 12.21

##  4 Discussion and Conclusion

In this work, we present MUSCLE a self-supervised continual learning pipeline
that pre-trains deep neural networks using multiple X-ray image datasets
collected from different body parts, e.g., hands, chests, bones and etc., for
multiple X-ray analysis tasks, e.g., TB detection, lung segmentation, and
skeletal abnormality classification. MUSCLE proposes _multi-dataset momentum
contrastive learning_ (MD-MoCo) and _multi-task continual learning_ to tackle
the data heterogeneity, over-fitting, and catastrophic forgetting problems in
pre-training, and finally fine-tunes the network to adapt every task
independently and separately. Experiment results on 9 X-ray image datasets
show MUSCLE outperforms other pre-training methods, including ImageNet-based
and MoCo-based [11] solutions. We do acknowledge that MUSCLE might NOT be an
optimized solution for any specific task in this study, we however claim
MUSCLE as a _“proof-of-concept”_ that demonstrates the feasibility of using
multiple datasets/tasks to pre-train X-ray models with advanced strategies of
self-supervised continual learning.

## References

  * [1] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature. 521(7553), 436-444 (2015).
  * [2] Balki, I., Amirabadi, A., Levman, J., Martel, A. L., Emersic, Z., Meden, B., … Tyrrell, P. N.: Sample-size determination methodologies for machine learning in medical imaging research: a systematic review. Canadian Association of Radiologists Journal. 70(4), 344-353 (2019).
  * [3] Jing, L., Tian, Y.: Self-supervised visual feature learning with deep neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence. 43(11), 4037-4058 (2020).
  * [4] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597-1607. PMLR (2020).
  * [5] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems. 33, 9912-9924 (2020).
  * [6] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A Simple Framework for Contrastive Learning of Visual Representations. 2002.05709. arXiv (2020).
  * [7] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9729-9738. IEEE (2020).
  * [8] Misra, I., Maaten, L.V.D.: Self-supervised learning of pretext-invariant representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6707–6717, IEEE (2020).
  * [9] Maithra, R., Chiyuan, Z., Jon, K., Samy B.: Transfusion: Understandingtransfer learning for medical imaging. 1902.07208. arXiv(2019).
  * [10] Cheplygina, V., de Bruijne, M., Pluim, J. P.: Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Med. Image Anal. 54, 280–296 (2019).
  * [11] Sowrirajan, H., Yang, J., Ng, A. Y., Rajpurkar, P.: Moco pretraining improves representation and transferability of chest x-ray models. In: Medical Imaging with Deep Learning. pp. 728-744. PMLR (2021).
  * [12] Memmel, M., Gonzalez, C., Mukhopadhyay, A.: Adversarial continual learning for multi-domain hippocampal segmentation. In Domain Adaptation and Representation Transfer, and Affordable Healthcare and AI for Resource Diverse Global Health. pp. 35-45. Springer, Cham (2021).
  * [13] Chen, L. C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. 1706.05587. arXiv (2017).
  * [14] Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 1440-1448(2015).
  * [15] Xuhong, L. I., Grandvalet, Y., Davoine, F.: Explicit inductive bias for transfer learning with convolutional networks. In: International Conference on Machine Learning. pp. 2825-2834. PMLR (2018).
  * [16] Gotmare, A., Keskar, N. S., Xiong, C., Socher, R.: A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. 1810.13243. arXiv (2018).
  * [17] Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R. M.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2097-2106. IEEE (2017).
  * [18] Jaeger, S., Candemir, S., Antani, S., Wáng, Y. X. J., Lu, P. X., Thoma, G.: Two public chest X-ray datasets for computer-aided screening of pulmonary diseases. Quantitative imaging in medicine and surgery. 4(6), 475 (2014).
  * [19] Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., … McDonald, C. J.: Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association. 23(2), 304-310 (2016).
  * [20] Halabi, S. S., Prevedello, L. M., Kalpathy-Cramer, J., Mamonov, A. B., Bilbily, A., Cicero, M., … Flanders, A. E.: The RSNA pediatric bone age machine learning challenge. Radiology. 290(2), 498-503 (2019).
  * [21] Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C., Liang, H., Baxter, S. L., … Zhang, K.: Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell. 172(5), 1122-1131(2018).
  * [22] Rajpurkar, P., Irvin, J., Bagul, A., Ding, D., Duan, T., Mehta, H., … Ng, A. Y.: Mura: Large dataset for abnormality detection in musculoskeletal radiographs. 1712.06957. arXiv (2017).
  * [23] Liu, Y., Wu, Y. H., Ban, Y., Wang, H., Cheng, M. M.: Rethinking computer-aided tuberculosis diagnosis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2646-2655. IEEE (2020).
  * [24] He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision. pp. 1026-1034. IEEE (2015).
  * [25] Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: A review. Neural Networks. 113, 54-71 (2019).
  * [26] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234-241. Springer, Cham (2015).
  * [27] Stephen, O., Sain, M., Maduh, U. J., Jeong, D. U.: An efficient deep learning approach to pneumonia classification in healthcare. Journal of healthcare engineering. 2019.(2019)
  * [28] Li, D., Yang, J., Kreis, K., Torralba, A., Fidler, S.: Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8300-8311. IEEE (2021).
  * [29] Bozorgtabar, B., Mahapatra, D., Vray, G., Thiran, J. P.: SALAD: Self-supervised aggregation learning for anomaly detection on x-rays. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 468-478. Springer, Cham (2020)


# True gravitational atom: Spherical cloud of dilatonic black holes

Yang Huang 1<EMAIL_ADDRESS>Hongsheng Zhang 1<EMAIL_ADDRESS>1
School of Physics and Technology, University of Jinan, 336, West Road of Nan
Xinzhuang, Jinan 250022, Shandong, China

###### Abstract

Black hole as elementary particle is a fairly glamorous idea. For ordinary
black holes, a surrounding particle inevitably penetrates into the interior of
the hole since the center of the hole is an infinite potential well. For the
first time we demonstrate that an extreme dilatonic black hole in spherical
symmetry perfectly behaves as an atom, in the sense that its surrounding cloud
of particles are completely stable. Thus we reach a spherical cloud of
dilatonic black hole. We find exact wave functions of the cloud for arbitrary
gravitational fine structure constant $\mu M$, and clear the underlying
physical nature of the stability. Through careful studies of the exact wave
function, we find the spectrum of this system. We discuss the physical meaning
of this discovery especially from considerations of entropy, and the resultant
possibility to explore quantization of gravitational waves from coming
observations.

##  I Introduction

Black hole is the most pure object in gravity, which may be treated as atom in
gravity theory. Because of the uniqueness theorem, a black hole looks like an
atom even more, in the original sense of this word. Direct quantization of
black hole itself runs up against stone walls again and again. Actually even
to propose a feasible test for quantization of stellar-mass or super/hyper
massive black holes is very difficult. In view of the construction history of
quantum mechanics, one may first find the quantized energy level of a
surrounding particle controlled by gravity force, and then explore the
properties of the gravitational waves triggered by transitions between energy
levels. If one finds absorption lines in the stochastic gravitational waves
background and identifies the source of this line, one may make a real step
towards quantizations of gravitational waves.

To find a black hole with stable energy level structure similar to an atom is
not a trivial work. Generally, a black hole has interior, which is different
from an elementary particle. For a surrounding particle, the center of the
interior of a hole is an infinite potential well. Thus, any matter wave
(particle) surrounding the hole will leak into the hole through quantum
tunnelling, which causes that the surrounding waves are unstable Detweiler
(1980); Nielsen  _et al._ (2019). Mathematically, the eigenvalue of the
frequency is a complex number, in which the imaginary part implies that the
surrounding particle is decaying. It seems rather difficult to evade such a
situation. For a rotating hole, superradiance process tells that a particle
with particular frequency can excite more particles with exactly same
frequency Starobinsky (1973); Unruh (1974); Brito  _et al._ (2015). If the two
processes, leakage and superradiance processes, balance each other, the
surrounding particles may construct a cloud Hod (2012); Huang and Liu (2016);
Huang  _et al._ (2017). If the backreaction of the cloud to the metric is
considered, one arrives at a hairy black hole Herdeiro and Radu (2014). Such a
cloudy/hairy hole only could be unstable against specific modes of
perturbations of the metric Ganchev and Santos (2018). Real stable energy
level structure like a Hydrogen atom, which does not depends on such a
delicate balance, is critically wanted for studies of quantization of
gravitational waves.

From other aspects, a black hole does not seem to behave like an elementary
particle. Generally a black hole has a large entropy, which indicates that it
has a large amount of internal degrees of freedom. This is rather different
from an elementary particle. Based on thermodynamical considerations and
fluctuation analysis, an interesting conjecture is proposed in Holzhey and
Wilczek (1992); Preskill  _et al._ (1991) that extreme dilatonic black hole
behaves as elementary particle. Put simply, the area of event horizon of a
dilatonic black hole goes to zero at the extreme limit. Thus one deduces that
its entropy vanishes for an extreme black hole from thermodynamic reasoning.
One has to arrive at the conclusion that an extreme dilatonic black hole has
no internal degree of freedom if statistical mechanic principle works well for
such a hole. Through rigorous studies of scalar wave surrounding an extreme
dilatonic black hole, we find the particles in bound states are stable, which
do not penetrate into the interior of the hole. This presents strong evidence
that this hole has no internal degree of freedom.

This paper is organized as follows. In Sec.II, we briefly review the
quasibound states of a massive scalar field for dilatonic black holes, and
show that the quasibound states tend to be stable in the extremal limit. Next,
in Sec.III, we construct the stable solutions of a massive scalar field for
exactly extreme hole. Finally, we conclude this paper in Sec.IV. We use the
units $G=\hbar=c=1$.

##  II Quasibound states of a massive scalar field

###  II.1 Radial equation and effective potential

Dilaton gravity, or called scalar-tensor theory, as one the most significant
modified gravity, takes a special status in gravity theories. It can be traced
back to Kaluza-Klein compactification Appelquist  _et al._ (1987) and Dirac’s
large number hypothesis Dirac (1937). In either case, an extra non-minimally
coupled scalar field is introduced in the effective field equation. Dilaton
gravity is the minimal modification, and a strong competitor, of general
relativity, in which only one extra freedom, the dilaton field, is involved in
gravity interaction. Many different modified gravity theories can be reduced
dilaton gravity in low energy limit, though the physical starting points of
them may be completely different. A recent example is string theory. Heterotic
string theory reduces to a dilaton gravity in low energy limit, the action is

$S=\int
d^{4}x\sqrt{-g}\left[R-2\left(\nabla\phi\right)^{2}-e^{-2\phi}F_{\mu\nu}F^{\mu\nu}\right],$
(1)

where $\phi$ is the dilaton field, and
$F_{\mu\nu}=\partial_{\mu}A_{\nu}-\partial_{\nu}A_{\mu}$ is the
electromagnetic tensor. In Refs.Garfinkle  _et al._ (1991); Gibbons and Maeda
(1988), a spherical black hole solution was derived by Gibbons, Maeda,
Garfinkle, Horowitz, and Strominger (GMGHS). In Einstein frame, the metric of
the GMGHS black hole reads,

$ds^{2}=-Fdt^{2}+F^{-1}dr^{2}+r^{2}G\left(d\vartheta^{2}+\sin^{2}\vartheta
d\varphi^{2}\right),$ (2)

with

$F(r)=1-\frac{2M}{r},\;\text{and}\;G(r)=1-\frac{Q^{2}}{Mr},$ (3)

where $M$ and $Q$ are the mass and charge of the black hole, respectively. The
GMGHS solution has one event horizon at $r_{+}=2M$, and a singular surface at
$r_{-}=Q^{2}/M$. For $Q<Q_{\mathrm{max}}\equiv\sqrt{2}M$, the singularity is
enclosed by the event horizon. In the extreme case $Q=Q_{\mathrm{max}}$, and
the singularity coincides with the horizon. The surface gravity at the event
horizon reads,

$\kappa=\frac{1}{2r_{+}},$ (4)

which directly follows the case of Schwarzschild. The Wald entropy becomes
different,

$S=\pi r_{+}^{2}\left(1-\frac{r_{-}}{r_{+}}\right).$ (5)

By using $Q^{2}=r_{+}r_{-}/2$, one confirms that the GMGHS black hole
satisfies the first law. The critical point is that the entropy vanishes when
the hole becomes an extreme one, which leads to an inescapable conclusion that
the extreme black hole has no internal degree of freedom if the principle of
statistical physics works soundly. This property is exactly in analogy with an
elementary particle. Thus, a reasonable conjecture is that the surrounding
matter wave cannot permeate into the interior of the extreme GMGHS black hole,
since there is no phase space to accommodate the permeated waves. As a
contrast, the entropy of extreme RN black hole does not vanish. So a
surrounding particle has probability to fall into it. We shall demonstrate
that bound state of extreme GMGHS black hole is really stable, as expected.

For convenience, we introduce a normalized charge
$q=Q/Q_{\mathrm{max}}\in\left[0,1\right]$, and parameterize the charge by

$q=1-e^{-\eta}.$ (6)

The Schwarzschild black hole corresponds to $\eta=0\;(q=0)$, while the extreme
GMGHS black hole corresponds to $\eta\rightarrow\infty$ $(q\rightarrow 1)$.

Scattering states of a massless scalar field by the GMGHS black hole has been
studied in Huang and Zhang (2020). Here, we investigate the bound states of a
massive scalar field $\Psi$, which is different from the dilaton field in the
theory. In the present study we work in probe limit, that is, $\Psi$ has no
back reaction to the spacetime metric. The Klein-Gordon equation of a scalar
field with mass $\mu$ in GMGHS spacetime is given by
$\nabla_{\mu}\nabla^{\mu}\Psi=\mu^{2}\Psi$. This equation admits the separable
solutions of the form Huang and Zhang (2021)

$\Psi=e^{-i\omega t}\frac{\psi(r)}{r\sqrt{G(r)}}Y_{lm}(\vartheta,\varphi),$
(7)

where the radial function $\psi$ obeys the radial equation

$\frac{d^{2}\psi}{dx^{2}}+\left[\omega^{2}-V_{l}(r)\right]\psi=0,$ (8)

with $x=\int dr/F$ the tortoise coordinate, and the effective potential given
by

$\displaystyle V_{l}(r)=$
$\displaystyle\frac{F(r)}{G(r)}\left[\frac{F^{\prime}(r)}{r}+\frac{l(l+1)}{r^{2}}+\mu^{2}G(r)\right]$
(9)
$\displaystyle-\frac{2M^{2}q^{2}}{r^{4}}\frac{F(r)}{G(r)^{2}}\left[1+\frac{q^{2}}{2}\left(1-\frac{6M}{r}\right)\right].$

Fig. 1 compares $V_{l}$ for different values of $\eta$. As pointed out in
Huang and Zhang (2021), the width of the potential barrier increases
monotonously with increasing $\eta$. In the extreme limit
$\eta\rightarrow\infty$, the width of the potential barrier increases without
bound, and the black hole absorption is heavily suppressed. This property is
crucial for the existence of long-lived modes of massive scalar field adhered
to a GMGHS black hole.

Figure 1: Comparison of the effective potential for different $\eta$, with
$l=1$ and $\mu M=0$. Here, $x$ is the tortoise coordinate defined below
Eq.(8).

###  II.2 Quasibound state spectrum

The penetration of the particle into the interior of the black hole is
confirmed by studies of energy flux of the particles at quasibound state, as
follows. One clearly observes that energy flows into the black hole by
explorations of energy flux in the black hole spacetime. Quasibound states are
solutions of Eq.(8) that satisfy the ingoing wave condition at the horizon and
decrease exponentially at infinity. The quasibound state spectrum was studied
in Huang and Zhang (2021). It was shown that in the near extreme limit
$\eta\gg 1$, the quasibound state frequencies can be obtained by matching the
near horizon and far field solutions. Due to the black hole absorption, the
eigenfrequency is generally a complex number (expressed by
$\omega=\omega_{R}+i\omega_{I}$) with a negative imaginary part. Both the real
and imaginary parts of the frequency are tuned by the gravitational fine
structure constant $\alpha\equiv\mu M$. For $\alpha<L/2$, the real part of the
frequency is given by

$\frac{\omega_{R}}{\mu}\approx
1-\frac{\alpha^{2}}{2\tilde{n}^{2}}-\frac{2(1+\tau)\alpha^{4}}{\tilde{n}^{3}L}+\frac{(\tau+15/8)\alpha^{4}}{\tilde{n}^{4}}+\cdots,$
(10)

where $L=l+1/2$, $\tilde{n}=n+l+1$ with $n=0,1,2,\cdots$, and $\tau=1-q^{2}$
is a small number for near extreme GMGHS black holes. The imaginary part is
given by

$\omega_{I}\propto-e^{-2\tilde{\beta}\eta},\;\;\;\text{with}\;\;\;\tilde{\beta}=\sqrt{L^{2}-4\omega_{R}^{2}}.$
(11)

As mentioned above, for $\eta\gg 1$, $\omega_{I}$ tends to zero exponentially
since the black hole absorption is suppressed by a large potential barrier.

###  II.3 Energy and flux

The Kerr-Schild coordinates
$\tilde{x}^{\mu}=\left[\tilde{t},r,\vartheta,\varphi\right]$ is adopted to
study the black hole absorption of the scalar field, where the new time
coordinate is given by Degollado and Herdeiro (2014)

$\tilde{t}=t+x-r,$ (12)

where $x$ is the tortoise coordinate defined below Eq.(8). Then, the GMGHS
metric (2) becomes

$\displaystyle ds^{2}=$
$\displaystyle\tilde{g}_{\mu\nu}d\tilde{x}^{\mu}d\tilde{x}^{\nu}$ (13)
$\displaystyle=$ $\displaystyle-
Fd\tilde{t}^{2}+(2-F)dr^{2}+2(1-F)d\tilde{t}dr+r^{2}Gd\Omega^{2}.$

For complex massive scalar fields, the energy-momentum tensor is given by

$T_{\mu\nu}=2\Psi^{*}_{,(\mu}\Psi_{,\nu)}-g_{\mu\nu}\left(\Psi^{*}_{,\gamma}\Psi^{,\gamma}+\mu^{2}|\Psi|^{2}\right),$
(14)

where $X_{(\mu}Y_{\nu)}\equiv\left(X_{\mu}Y_{\nu}+X_{\nu}Y_{\mu}\right)/2$.
The timelike Killing vector field $k=\partial_{t}$ defines a conserved current
$J^{\mu}=-T^{\mu}_{\;\;\nu}\xi^{\nu}$, where $\xi^{\mu}=\left[1,0,0,0\right]$.
Using the conservation law $\nabla_{\mu}J^{\mu}=0$ and Gauss’s theorem, we
obtain

$\frac{dE}{d\tilde{t}}=-\mathcal{F},$ (15)

where

$E=\int_{\Sigma_{\tilde{t}}}d^{3}x\sqrt{-g}J^{0},$ (16)

is the total energy of the scalar field, and $\Sigma_{\tilde{t}}$ is a
spacelike hypersurface with $\tilde{t}=0$. Flux at $r=r_{+}$ is given by

$\mathcal{F}=\int_{\partial\Sigma_{\tilde{t}}}r^{2}GT^{1}_{\;\;0}\sin\vartheta
d\vartheta d\varphi,$ (17)

where $\partial\Sigma_{\tilde{t}}$ is the boundary of the hypersurface, i.e.,
$\tilde{t}=0$, $r=r_{+}$, $\vartheta\in\left[0,\pi\right]$, and
$\varphi\in\left[0,2\pi\right]$. If we decompose the scalar field as
$\Psi(\tilde{t},r,\vartheta,\varphi)=\sum_{lm}\phi_{l}(\tilde{t},r)Y_{lm}(\vartheta,\varphi)$,
then, the flux can also be decomposed as a sum over all angular modes

$\mathcal{F}=2r^{2}G\sum_{l}\mathcal{F}_{l}$ (18)

where

$\mathcal{F}_{l}=\tilde{g}^{01}\big{|}\partial_{\tilde{t}}\phi_{l}\big{|}^{2}+\tilde{g}^{11}\text{Re}\left(\partial_{\tilde{t}}\phi_{l}^{*}\partial_{r}\phi_{l}\right),$
(19)

Clearly, the flux is always positive for non-extreme GMGHS black hole.
Therefore, the energy $E$ decays with time due to the black hole absorption.
For extreme GMGHS black hole, however, the flux vanishes at the horizon since
$G(r_{+})=0$. This implies that in this case, the energy of the scalar field
is conserved. No energy penetrates into the interior of the hole. Thus,
stationary states may exist in extreme GMGHS spacetime.

##  III Bound states in extremal GMGHS spacetime

###  III.1 Bound state spectrum

Based on the discussion above, we expect that in the extreme limit, real bound
states with $\omega_{I}=0$ exist. They are stable, and thus the black hole
becomes a true gravitational atom without decaying.

For extreme GMGHS black hole, the effective potential given in Eq.(9) becomes

$V_{l}(r)=\frac{l(l+1)}{r^{2}}+\frac{M}{r^{3}}\left(2-\frac{3M}{r}\right)+F(r)\mu^{2}.$
(20)

Unlike the non-extreme case, $V_{l}$ does not vanish at the horizon, but tends
to $(2l+1)^{2}/16M^{2}$, see Fig.2. We also observe that for $\mu M\neq 0$,
$V_{l}$ develops a potential well between the potential barrier and infinity.
Fig. 3 compares the effective potentials for different $\mu M$. We see that
the local minimum of the potential well moves far away from the black hole
with the decrease of $\mu M$.

Figure 2: Effective potential for $l=1$ and $\mu M=0.6$. Figure 3: Comparison
of the effective potentials of $l=1$ state for different values of $\mu M$.
The black points mark the local minimum of potential well.

The walls on both sides of the potential well are infinitely wide. Thus, we
choose the boundary conditions $\psi_{\omega l}(r_{+})=\psi_{\omega
l}(\infty)=0$. It is convenient to introduce the following dimensionless
quantities

$z=\frac{r-r_{+}}{r_{+}};\;\;\;\epsilon=\omega M;\;\;\;\alpha=\mu M.$ (21)

Substituting Eq.(20) into Eq.(8), the radial equation becomes

$\frac{d^{2}\psi}{dz^{2}}+\frac{1}{z(z+1)}\frac{d\psi}{dz}-U\psi=0,$ (22)

where

$\displaystyle U(z)=$
$\displaystyle\frac{4(1+z)}{z}\left[\alpha^{2}-\left(1+\frac{1}{z}\right)\epsilon^{2}\right]+\frac{l(l+1)}{z^{2}}$
(23) $\displaystyle+\frac{1+4z}{4z^{2}(1+z)^{2}}.$

We find that solutions of Eq.(22) obeying boundary condition $\psi(z=0)=0$ can
be expressed in terms of the generalized Laguerre function

$\psi(z)=z^{\beta}(1+z)^{1/2}e^{-kz}L_{-1/2-\beta+\kappa}^{2\beta}(2kz),$ (24)

where $k=2\sqrt{\alpha^{2}-\epsilon^{2}},\;\kappa=(4\epsilon^{2}-k^{2})/(2k)$
and

$\beta=\sqrt{L^{2}-4\epsilon^{2}},\;\;\;\text{with}\;\;\;L\equiv l+1/2.$ (25)

Then, using the property of the generalized Laguerre function and boundary
condition $\psi(\infty)=0$, we have

$-\frac{1}{2}-\beta+\kappa=n,$ (26)

where $n=0,1,2,\cdots$ is the overtone. Amazingly, this is exactly the extreme
limit of the matching condition of the quasibound states in the non-extreme
case, see Huang and Zhang (2021) for more details. The physical nature of this
bound state roots in the property of potential. The potential becomes
infinitely wide in the extreme limit, which obstructs any possible penetration
of the matter wave into the hole.

Equation (26) determines the bound state frequencies labeled by the overtone
$n$. From Fig. 2, the bound state frequency must obey

$\omega_{n}<\text{min}\left\\{\mu,(2l+1)/4M\right\\},$ (27)

or equivalently,
$\epsilon_{n}=M\omega_{n}<\text{min}\left\\{\alpha,L/2\right\\}$. Following
Huang and Zhang (2021), we solve Eq.(26) with the series expansion

$\epsilon_{n}=\alpha\left[1+\sum_{i=1}^{\infty}C_{i}\alpha^{2i}\right].$ (28)

The coefficients $C_{i}$ can be obtained by substituting the series expansion
into Eq.(26), and solving the equation order by order. Here we list the first
third $C_{i}$:

$\displaystyle C_{1}=$ $\displaystyle-\frac{1}{2\tilde{n}^{2}},$ (29a)
$\displaystyle C_{2}=$
$\displaystyle-\frac{2}{\tilde{n}^{3}L}+\frac{15}{8\tilde{n}^{4}},$ (29b)
$\displaystyle C_{3}=$
$\displaystyle-\frac{2}{\tilde{n}^{3}L^{3}}-\frac{6}{\tilde{n}^{4}L^{2}}+\frac{17}{\tilde{n}^{5}L}-\frac{145}{16\tilde{n}^{6}},$
(29c)

where $\tilde{n}=n+l+1$ is the principal quantum number. In practice, we have
computed the coefficients up to $C_{9}$. But their expressions are too
cumbersome to be presented here.

It should be noted that $\epsilon_{n}$ given in Eq.(28) are purely real
frequencies, and they are exactly the extreme limit of the quasibound state
frequencies in the non-extreme case. Therefore, the bound states obtained in
this work are stable as the wave function of the scalar field will not grow or
decay with time.

To find possible signs of the bound states from gravitational wave detectors,
we make a numerical approximation of the energy levels. In the limit of
$\alpha\ll L$, the bound state has a hydrogen-like spectrum

$\omega_{\tilde{n}}\simeq\mu\left(1-\frac{\alpha^{2}}{2\tilde{n}^{2}}\right).$
(30)

Like the photoelectric effects, gravitational wave with proper frequency will
trigger transitions between energy levels. The frequency of the required
graviton to trigger this process ($\tilde{m}<\tilde{n}$) is given by

$f_{\tilde{n}\tilde{m}}\simeq\frac{\mu}{2}\alpha^{2}\left(\frac{1}{\tilde{m}^{2}}-\frac{1}{\tilde{n}^{2}}\right),$
(31)

Using $\alpha\simeq 7.5\times 10^{9}(M/M_{\odot})(\mu/1\text{eV})$, where
$M_{\odot}$ is the solar mass, we have

$f_{\tilde{n}\tilde{m}}\simeq
43\text{Hz}\left(\frac{M}{M_{\odot}}\right)^{2}\left(\frac{\mu}{10^{-11}\text{eV}}\right)^{3}\left(\frac{1}{\tilde{m}^{2}}-\frac{1}{\tilde{n}^{2}}\right),$
(32)

which falls in the sensitivity band of advanced ground based detectors for
$\mu\sim 10^{-11}\text{eV}$ and stellar-mass black holes lig , and of space
based detector for $\mu\sim 10^{-15}\text{eV}$ and super massive black holes
$\left(10^{6}-10^{8}\;M_{\odot}\right)$ lis ; Hu and Wu (2017); Hu  _et al._
(2018). The bound scalar particles will all be in ground state if there is no
extra perturbations since they are bosons. With future developments, a careful
analysis of the stochastic gravitational wave background Romano (2019) might
point to absorption lines present in it. This could be used to identify
sources of the type described in this paper. In analogy with the quantization
of electromagnetic waves from photoelectric experiments, one might get from
this analysis a glimpse of the quantization of gravitational waves.

###  III.2 Radial distribution

Let us now investigate the spatial distribution of the scalar clouds. From
Eq.(7), the radial part of the scalar field is given by
$R(r)=\psi(r)/r\sqrt{G(r)}$. To compare with the hydrogen atom, we introduce a
new variable $\bar{r}=2kz$, and label the radial function by quantum numbers
$(n,l)$. Then, using Eqs.(24) and (26), we obtain

$R_{nl}(\bar{r})\propto\bar{r}^{\beta-1/2}e^{-\bar{r}/2}L_{n}^{2\beta}(\bar{r}),$
(33)

where $\beta$ is given in Eq.(25). In the limit $\alpha\ll L$,

$\beta=l+\frac{1}{2}-\frac{4\alpha^{2}}{1+2l}-\mathcal{O}\left(\alpha^{4}\right).$
(34)

If we neglect the term containing $\alpha^{2}$, the radial function is
hydrogenic-like

$R_{nl}(\bar{r})\propto\bar{r}^{l}e^{-\bar{r}/2}L_{n}^{2l+1}(\bar{r}).$ (35)

Typical radial functions of the scalar clouds are presented in Fig. 4. We see
that the overtone $n$ is exactly the number of nodes of $R_{nl}$, and $R_{nl}$
decrease to zero at spatial infinity. We also see that as $\mu M$ is
decreased, the maximum of $R_{nl}$ is pushed far away from the black hole.
This is consistent with Fig. 3 that the local minimum of the potential well
moves far way from the black hole with the decrease of $\mu M$.

Figure 4: Comparison of radial functions given in (33), for values of $\mu M$.
The upper panel shows the $l=0$ states, whereas the bottom panel shows the
$l=1$ states. $n=0,1,2$ from left to right panels.

It should be noted that for $l>1$, we have $\beta>1/2$ (see Eq.(25)), and the
radial function $R_{nl}(\bar{r})$ is always finite for
$\bar{r}\in[0,+\infty)$. However, for $l=0$ state, $\beta<1/2$, and the radial
function is divergent at the event horizon $\bar{r}=0$. Fortunately, the
integral over $\bar{r}$ of the energy density remains finite. This can be seen
in the following.

Substituting the decomposition in spherical harmonics (7) into Eq.(16) and
integrating over the sphere, we obtain $E=\sum_{l}E_{l}$, with

$E_{l}=\int_{r_{+}}^{\infty}\rho_{l}(r)r^{2}dr,$ (36)

where

$\displaystyle\rho_{l}(r)=$ $\displaystyle-G\int T^{0}_{\;\;0}\sin\vartheta
d\vartheta d\varphi$ (37) $\displaystyle=$ $\displaystyle
g^{11}G\bigg{|}\frac{dR_{nl}}{dr}\bigg{|}^{2}+\left[\frac{l(l+1)}{r^{2}}+G\left(\mu^{2}-g^{00}\omega_{\tilde{n}}^{2}\right)\right]|R_{nl}|^{2}.$

The integral can also be evaluated in the dimensionless coordinate
$\bar{r}=2k(r-r_{+})/r_{+}$

$E_{l}=\int_{0}^{\infty}\mathcal{E}_{l}(\bar{r})d\bar{r}.$ (38)

The radial energy density of the scalar cloud is defined as the integrand of
$E_{l}$, i.e., $\mathcal{E}_{l}(\bar{r})\propto\rho_{l}(r)r^{2}$. For $l\geq
1$, both the radial function and energy density are regular everywhere.
However, for $l=0$, they may be divergent at the horizon $\bar{r}=0$. Using
Eqs.(33)-(38) and taking $l=0$, we have

$\mathcal{E}_{l=0}(\bar{r})\sim\bar{r}^{-8\alpha^{2}-\mathcal{O}\left(\alpha^{4}\right)},\;\;\;\text{for}\;\;\bar{r}\rightarrow
0.$ (39)

The potential analysis in previous section shows that for large values of
$\alpha$, the potential well may disappear, and no bound states exist.
Conversely, for $\alpha<L/2$, the power of $\bar{r}$ in the energy density
$\mathcal{E}_{l=0}(\bar{r})$ is greater than $-1$. Therefore, the energy given
in Eq.(38) is finite for $l=0$ in spite of the divergence of
$\mathcal{E}_{l=0}(\bar{r})$ at $\bar{r}=0$.

Figure 5 compares the radial energy densities of $l=0$ states for different
values of $n$. We see that these energy densities are divergent as
$\bar{r}\rightarrow 0$. However, the integral given in Eq.(38) remains finite,
and the energy densities in Fig.5 have been normalized by $E_{l=0}=1$. We also
see that for a given $n$, the energy density is always positive, and the
number of the maximum points is given by $n+1$. Like the radial function, the
density decreases to zero exponentially at large distances.

Figure 5: Comparison of the radial energy densities, with $l=0$ and
$\alpha=\mu M=0.1$, for $n=0,1,2,3$. All the energy densities have been
normalized by $E_{l}=1$.

##  IV Discussion and Conclusion

Black hole takes a pivotal status in any forms of quantization of gravity. An
ordinary black hole may be in highly excited state in loop quantum gravity
Perez (2017). String theory has a similar view about a black hole Das and
Mathur (2000). Generally, ground state is much simpler than excited state to
explore. Thus, one must be curious that is there a black hole happens to fall
into ground state? The property of zero-entropy of an extreme GMGHS black hole
implies that it may be in ground state. In this paper, we demonstrate that
extreme GMGHS black hole really supports stable scalar clouds in its exterior
spacetime, following a true atom.

First, we explore the effective potential and find its width goes to infinity
in tortoise coordinates when the GMGHS black hole becomes extremal. And then
we find the exact solution of the radial function and present the energy
spectrum of the bound states. The energy spectrum is pure real, since the
particles around the hole stop penetrating into the hole, which are blocked by
the infinite wide potential barrier. Further, we investigate the energy flux
of the surrounding particles. We show that the flux vanishes at the horizon,
which confirms that the penetrating processes of particles from the exterior
into the inner space of the hole stop.

Through the above studies, we prove that a stable cloud can be formed
surrounding an extreme GMGHS black hole. Some interesting properties of the
scalar clouds are discussed. First, we show that these clouds are described by
real bound states with $\omega_{I}=0$, and their frequencies are exactly given
by the extreme limit of the quasibound state spectrum Huang and Zhang (2021).
The mechanism of the scalar cloud in present work does not depend on
superradiance, and also applies to massive Dirac field in the GMGHS spacetime
Huang and Zhang (2022). Second, the radial function expressed in terms of the
generalized Laguerre polynomials, and the quantum number $n$ indicates the
nodes of the radial functions. We present a formula similar to Ritz’s
principle in Hydrogen atom theory. More properties of extreme black holes in
analogy to an atom need to explore further.

###### Acknowledgements.

Our thanks go to the anonymous referees who present several valuable
suggestions. This work is supported by Shandong Province Natural Science
Foundation under grant No. ZR201709220395.

## References

  * Detweiler (1980) S. L. Detweiler, Phys. Rev. D 22, 2323 (1980).
  * Nielsen  _et al._ (2019) N. G. Nielsen, A. Palessandro, and M. S. Sloth, Phys. Rev. D 99, 123011 (2019).
  * Starobinsky (1973) A. A. Starobinsky, Sov. Phys. JETP 37, 28 (1973).
  * Unruh (1974) W. G. Unruh, Phys. Rev. D 10, 3194 (1974).
  * Brito  _et al._ (2015) R. Brito, V. Cardoso, and P. Pani, Lect. Notes Phys. 906, pp.1 (2015), arXiv:1501.06570 [gr-qc] .
  * Hod (2012) S. Hod, Phys. Rev. D 86, 104026 (2012).
  * Huang and Liu (2016) Y. Huang and D.-J. Liu, Phys. Rev. D 94, 064030 (2016), arXiv:1606.08913 [gr-qc] .
  * Huang  _et al._ (2017) Y. Huang, D.-J. Liu, X.-H. Zhai, and X.-Z. Li, Class. Quant. Grav. 34, 155002 (2017), arXiv:1706.04441 [gr-qc] .
  * Herdeiro and Radu (2014) C. A. R. Herdeiro and E. Radu, Phys. Rev. Lett. 112, 221101 (2014), arXiv:1403.2757 [gr-qc] .
  * Ganchev and Santos (2018) B. Ganchev and J. E. Santos, Phys. Rev. Lett. 120, 171101 (2018).
  * Holzhey and Wilczek (1992) C. F. E. Holzhey and F. Wilczek, Nucl. Phys. B 380, 447 (1992), arXiv:hep-th/9202014 .
  * Preskill  _et al._ (1991) J. Preskill, P. Schwarz, A. Shapere, S. Trivedi, and F. Wilczek, Modern Physics Letters A 06, 2353 (1991).
  * Appelquist  _et al._ (1987) T. Appelquist, A. Chodos, and P. G. O. Freund, eds.,  _MODERN KALUZA-KLEIN THEORIES_ (1987).
  * Dirac (1937) P. A. M. Dirac, Nature 139, 323 (1937).
  * Garfinkle  _et al._ (1991) D. Garfinkle, G. T. Horowitz, and A. Strominger, Phys. Rev. D43, 3140 (1991), [Erratum: Phys. Rev.D45,3888(1992)].
  * Gibbons and Maeda (1988) G. W. Gibbons and K.-i. Maeda, Nucl. Phys. B298, 741 (1988).
  * Huang and Zhang (2020) Y. Huang and H. Zhang, Eur. Phys. J. C 80, 654 (2020), arXiv:2006.01388 [gr-qc] .
  * Huang and Zhang (2021) Y. Huang and H. Zhang, Phys. Rev. D 103, 044062 (2021).
  * Degollado and Herdeiro (2014) J. C. Degollado and C. A. R. Herdeiro, Phys. Rev. D 89, 063005 (2014), arXiv:1312.4579 [gr-qc] .
  * (20) https://www.ligo.caltech.edu/.
  * (21) https://lisa.nasa.gov/index.html.
  * Hu and Wu (2017) W.-R. Hu and Y.-L. Wu, Natl. Sci. Rev. 4, 685 (2017).
  * Hu  _et al._ (2018) X.-C. Hu, X.-H. Li, Y. Wang, W.-F. Feng, M.-Y. Zhou, Y.-M. Hu, S.-C. Hu, J.-W. Mei, and C.-G. Shao, Class. Quant. Grav. 35, 095008 (2018), arXiv:1803.03368 [gr-qc] .
  * Romano (2019) J. D. Romano (2019) arXiv:1909.00269 [gr-qc] .
  * Perez (2017) A. Perez, Rept. Prog. Phys. 80, 126901 (2017), arXiv:1703.09149 [gr-qc] .
  * Das and Mathur (2000) S. R. Das and S. D. Mathur, Annual Review of Nuclear and Particle Science 50, 153 (2000).
  * Huang and Zhang (2022) Y. Huang and H. Zhang, Research 2022, Article ID 9823274 (2022).