# Effect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot
Transfer Learning for Natural and Medical X-Ray Chest Images

Mehdi Cherti, Jenia Jitsev
Juelich Supercomputing Center, Research Center Juelich
Helmholtz AI
LAION
Juelich, Germany
{m.cherti<EMAIL_ADDRESS>

###### Abstract

Increasing model, data and compute budget scale in the pre-training has been
shown to strongly improve model generalization and transfer learning in vast
line of work done in language modeling and natural image recognition. However,
most studies on the positive effect of larger scale were done in scope of in-
domain setting, with source and target data being in close proximity. To study
effect of larger scale for both in-domain and out-of-domain setting when
performing full and few-shot transfer, we combine here for the first time
large, openly available medical X-Ray chest imaging datasets to reach a scale
for medical imaging domain comparable to ImageNet-1k, routinely used for pre-
training in natural image domain. We then conduct supervised pre-training,
while varying network size and source data scale and domain, being either
large natural (ImageNet-1k/21k) or large medical chest X-Ray datasets, and
transfer pre-trained models to different natural or medical
targets111Repository: https://github.com/SLAMPAI/large-scale-pretraining-
transfer. We observe strong improvement due to larger pre-training scale for
intra-domain natural-natural and medical-medical transfer. For inter-domain
natural-medical transfer, we find improvements due to larger pre-training
scale on larger X-Ray targets in full shot regime, while for smaller targets
and for few-shot regime the improvement is not visible. Remarkably, large
networks pre-trained on very large natural ImageNet-21k are as good or better
than networks pre-trained on largest available medical X-Ray data when
performing transfer to large X-Ray targets. We conclude that substantially
increasing model and generic, medical domain-agnostic natural image source
data scale in the pre-training can enable high quality out-of-domain transfer
to medical domain specific targets, removing dependency on large medical
domain-specific source data often not available in the practice.

##  1 Introduction

Re-using models obtained by pre-training on available source datasets to
improve learning performance on upcoming target datasets is core idea behind
transfer learning. It has a long history in machine learning field [1, 2] and
was also employed already at the very early rise of deep neural networks in
the vision domain [3, 4]. Different architectures like AlexNet [5], OverFeat
[6], and VGG [7] were pre-trained on supervised tasks using ImageNet-1k, a
publicly available natural image dataset that contains about 1.4 Million
images and 1000 classes [8, 9]. After pre-training, the resulting models were
taken as off-the-shelf generic features reservoirs and re-used by re-training,
or fine-tuning, on various downstream target datasets and tasks, including
classification, object detection, and segmentation [3, 4]. Importantly, the
transfer approach allowed to improve performance on target datasets when
compared to training from scratch with randomly initialized weights [10, 11].
Further, it enabled to train models of good quality also on comparatively
small amounts of data, in contrast to large amounts usually required for
learning high-quality models when training a deep neural network from scratch.

Recent line of work on scaling laws in language modeling [12] and vision
demonstrated strong improvement for model’s ability to generalize on unseen
test data when increasing model, data, and compute budget scale during the
training. In language modeling, very large Transformer networks pre-trained on
very large text data have also shown very strong transfer performance on a
broad range of novel tasks, compared to pre-trained models of smaller scale
[13]. In the same line, experimental studies on large-scale pre-training and
transfer in image domain found evidence that increasing network model and data
size during pre-training results in transfer performance benefits [14, 15].

The majority of the studies looking at the effect of pre-training scale on
transfer deal with the intra-domain scenario scenario, where source and target
data are close to each other, often originating from the same domain, being
for instance natural images. This raises the question whether the observed
positive effect of larger scale will also uphold in the inter-domain transfer
scenario when using different types of source and target data that are not so
closely related.

To address this, we conduct a series of large-scale pre-training and transfer
experiments where we vary not only ResNet model [16, 14] and dataset size
during pre-training, but also the domain of the source and the target
datasets, being either natural or medical X-Ray chest images, which allows us
to study effect of scale on both intra- and inter-domain transfer. To vary
source data scale in natural domain, we take either large ImageNet-1k or much
larger ImageNet-21k [8]. To vary pre-training data scale for medical X-Ray
domain, we combine here, for the first time, large openly available medical
X-Ray chest imaging datasets (CheXpert [17], MIMIC-CXR [18], PadChest [19],
NIH Chest X-ray14 [20]) into supersets, with the largest scale comparable to
ImageNet-1k. We then transfer to either natural or X-Ray image datasets as
target. For transfer, we also vary the operation in either full or few-shot
regime, where only few examples per class are shown to the pre-trained models
during fine-tuning on a target dataset. As large-scale pre-training requires
heavy computational resources, we make use of a state-of-the art supercomputer
(JUWELS Booster [21]) tailored for distributed training to conduct our
experiments.

The results we obtain show that both intra- and inter-domain transfer benefit
from larger pre-training scale. They also reveal a differentiated picture
suggesting that effect of larger scale on transfer is expressed differently in
intra- and inter-domain scenario and for full shot and few-shot regime.
Remarkably, and also of high relevance for practice, we observe that large
networks pre-trained on very large generic natural ImageNet-21k are as good or
better than networks pre-trained on largest available medical domain-specific
X-Ray superset data when performing full shot transfer to large X-Ray targets.
This indicates that high quality models for domain specific medical X-Ray
targets can be obtained by increasing scale of the network and of the generic
natural image data in the pre-training, without relying on large amount of
domain-specific data that is often not available in practice. In contrast to
previous studies that dealt with smaller scales [22], we conclude that inter-
domain transfer from natural to medical images benefits from substantially
larger pre-training scales.

##  2 Background and related work

Scaling laws for generalization and transfer. Strong evidence that increasing
model and data size for the training may result in steady improvement of
generalization comes from language modeling studies systematically looking on
the dependency of test error on model, data size, and compute budget used for
training [13, 12, 23]. The experiments conducted there set up scaling laws
with a power law shape and show consistent further decrease of test error when
further increasing model, data size, and compute budget over many orders in
magnitude hand in hand. For images, a similar line of work by Henighan et al.
shows a decrease of test classification top-1 error when fine-tuning on
ImageNet pre-trained generative image models of increasing size [24]. Those
works use self-supervised training of autoregressive models, in language
modeling performed on text and in image domain on image patches, employing
transformer networks as a backbone. Additional backup for this line of work
comes from studies that revise the dependency of generalization performance on
model, data size, and epoch number during training and report double or multi
descent curves for the test error [25, 26, 27]. There, keeping on increasing
model, data size or training time substantially also shows continuous drop in
the test error pointing to generalization improvement, for instance when
crossing the interpolation threshold and transiting into the over-
parameterized regime by scaling up the model size.

Improving transfer by scaling up pre-training. Also improvement in transfer on
downstream datasets and tasks is strongly evident from large-scale language
modeling experiments. In the study by Brown et al. [13], large transformer
networks in the order of hundred billions of parameters (GPT-3) pre-trained on
large text datasets in the order of billions of sentences were shown to have
much stronger transfer performance than smaller GPT network models, measured
by the test error on different downstream tasks. The difference in transfer
performance between different sized models was especially pronounced in the
very low data regime when doing zero-shot or few-shot transfer with only few
examples available during fine-tuning. Further systematic study on transfer
improvements induced by increasing scale was done by Hernandez et al. [23],
who examined scaling laws for transfer on language modeling tasks in the low-
data regime, being defined as transfer using less than $10\%$ of the available
target data. The authors have shown that increasing model size in the pre-
training decreases test error on the target data, emphasizing that test error
improvement cannot be observed when increasing model size and training
directly from scratch on the target without pre-training. It was also pointed
out that the degree of proximity between the source and the target dataset
plays a role when predicting effect of the scale on the transfer performance,
leading to a revised version of the scaling law that took this additional
dependency into account.

In the image domain, the performed studies on transfer improvement due to
scale have still far less systematic character. Models and datasets used for
training on images are 3-4 orders of magnitude behind those studied in
language modeling [13]. Recently, number of works were starting to employ
datasets like ImageNet-21k [8], YFCC-100M [28], JFT-300M [29] or JFT-3B [30]
that are much larger than standard ImageNet-1k to pre-train large network
models on large data and observe the effect of scaling up on transfer. The
work on Big Transfer by Kolesnikov et al. [14] performed supervised
classification based pre-training on ImageNet-1k, ImageNet-21k, and JFT-300M
using different sized deep residual networks (ResNets [16]) to study the
performance of pre-trained models on transfer across different target
datasets. They found consistent improvement in transfer performance when using
larger models and larger data during pre-training. In the same direction,
works by [15, 30] pre-trained different sized network models on ImageNet-1k,
ImageNet-21k or JFT-3B [30] also observing consistent transfer improvement
when scaling-up model and data size during pre-training.

Studies mentioned above deal with closely related source and target datasets
containing mostly natural images data. In general, various works related to
testing transfer performance across different target datasets often employ
targets that are rather close to pre-training source data, like studies
introducing transfer benchmarking datasets that use targets resembling mostly
natural image domain [10, 31] or domain specific transfer studies that stay
within their given domain, for instance in medical imaging [32].

Only few studies so far attempt to measure transfer performance between
datasets that are further apart, for instance natural and medical images,
while systematically varying model and data size during pre-training. Work
done by Raghu et al. [22] has found no significant difference between models
pre-trained on ImageNet-1k and models trained from scratch on target datasets
containing medical images. However, it was not using datasets larger than
standard ImageNet-1k or networks larger than standard ResNet-50 for pre-
training. Another work examines transfer on CheXpert while varying network
model size during pre-training [33]. It does find slight benefit for transfer
when pre-training with larger models, however it does not vary source data
size in the pre-training, using only standard ImageNet-1k as a source. A
recent study by Mustafa et al. builds up on Big Transfer work [14] and
compares transfer performance of different sized ResNet network models pre-
trained on ImageNet-1k, ImageNet-21k, and JFT-300M on different medical
imaging target datasets [34]. Slight evidence for transfer improvement was
observed when using larger model and larger dataset sizes during pre-training,
with inconsistencies across conditions and datasets, where in some cases no
significant benefit from larger pre-training scale was seen. These works all
lack comparisons between models pre-trained on natural images data and models
pre-trained on medical imaging data of different scale when measuring transfer
performance on medical imaging targets.

##  3 Experiments & Results

In order to test the impact of model and data pre-training scale on transfer
performance in full and few-shot regime under different source and target data
type constellations in intra- and inter-domain scenarios, we conducted
experiments on pre-training different sized ResNet models on supervised
classification task using either large natural image datasets ImageNet-1k or
ImageNet-21k, or compositions of chest X-Ray medical imaging datasets
CheXpert, MIMIC-CXR, PadChest, and NIH Chest X-ray14 (see Tab. 1 for
comprehensive list and further details of datasets). The pre-trained models
were then fine-tuned on different target datasets that contain either natural
or medical images. In the following, we describe the experimental procedures
and outcomes in more detail.

Table 1: Datasets used as source for pre-training and targets for transfer. All datasets used in the study are publicly available. Dataset | Usage | Size
---|---|---
Natural Images |  |
ImageNet-1k [8] | Source | 1.4M images, 1000 classes
ImageNet-21k [8] | Source | 14M images, 21842 classes
CIFAR-10, 100 [35] | Target | 60K images, 10,100 classes
Oxford Flowers-102 [36] | Target | 8K images, 102 classes
Oxford-IIIT Pet [37] | Target | 7.3K images, 37 classes
X-Ray Chest Imaging |  |
CheXpert [17] | Source, Target | 224K radiographs of 65K patients, 14 classes
NIH Chest X-ray14 [20] | Source, Target | 112K radiographs of 32K patients, 14 classes
PadChest [19] | Source, Target | 160K radiographs of 67K patients, 19 classes
MIMIC-CXR [18] | Source, Target | 377K radiographs of 65K patients, 14 classes
Total Source X-Ray images | Source Superset |  873K chest radiographs, 229K patients
Tuberculosis [38] | Target | 800 radiographs, 800 patients, 2 classes
COVIDx [39] | Target | 16K radiographs, 15K patients, 2 / 3 classes

###  3.1 Large-scale pre-training

For pre-training, we largely followed the training procedure and used the
network architecture of [14]. More concretely, we pre-trained both ResNet-50x1
and ResNet-152x4 (in following R50x1 and R152x4) from [14] on different
natural image and medical datasets. Smaller R50x1 has 26M weight parameters,
while larger R152x4 has 928M parameters. This substantial difference in size
allows us to compare the effect of model scaling in the pre-training on
subsequent transfer. The following describes the training procedure and hyper-
parameters used for natural image and medical image domain.

####  3.1.1 Natural image domain

For natural images, we pre-trained the two models (R50x1 and R152x4) on
ImageNet-1k ($\approx 1.4$ Millions images) and the much larger full
ImageNet-21k ($\approx 14$ Millions images). For ImageNet-1k and Imagenet-21k
models, we used a standard supervised classification setup with softmax as an
output activation and cross entropy as a loss.

We followed the training hyper-parameters of [14], with the difference that we
used stochastic gradient descent (SGD) with adaptive gradient clipping (AGC)
from [40], as we found that it helps both pre-training and transfer. With AGC,
we found that the default base learning rate used in [14] made training
unstable for the ImageNet-21k experiments, so we reduced it from $0.03$ to
$0.01$, but otherwise we used a base learning rate of $0.03$. The rest of the
hyper-parameters were similar, namely we used a momentum of $0.9$, $90$
epochs, $\approx 5000$ warmup iterations, a batch size of $4096$, the linear
learning rate rescaling rule of [41], and the standard step-wise learning rate
schedule for ImageNet [14]. For data augmentation, we used the standard random
resized crop data augmentation as in [14]. In ImageNet-1k experiments, we
additionally used RandAugment [42] and changed the learning rate schedule from
step-wise to cosine annealing [43], as it improved the pre-training results.
In order to speedup training, we used data parallel training with Horovod
[44], using $256$ A100 GPUs for R152x4 models and $128$ A100 GPUs for R50x1
models. A pre-training on ImageNet-21k with large R152x4 takes about 81 hours
using $256$ GPUs, while with small R50x1 it needs about 13.5 hours to finish
using $128$ GPUs on JUWELS Booster supercomputer [21] (see Suppl. Sec. A and
Suppl. Fig. 3 for more details on distributed training)

####  3.1.2 Medical image domain

For medical data, we pre-trained the two models (R50x1 and R152x4) on
combinations of several medical datasets, which as supersets may contain any
of the available datasets: CheXpert, MIMIC-CXR, NIH X-ray14, PadChest. We
refer to those combinations as X-Ray supersets in following. The largest
source X-Ray superset contains about 873K X-Ray chest radiographs (see also
Tab. 1). The medical datasets are multi-label, as each image can be associated
to several diseases. The datasets are combined by finding intersecting labels
(diseases) and using the intersected labels as a target. In order to
substantially vary data scale for medical domain, we start with single
available X-Ray datasets and progressively add other datasets into X-Ray
supersets of successively growing size, which provides us with X-Ray source
datasets spanning scales from small ($\approx$ 200k samples) to large
($\approx$ 870k samples) to perform pre-training on. For processing the
datasets and extracting the labels from raw data, we used TorchXRayVision [45]
from the work of [32].

We followed the literature on medical datasets [32] and pre-trained using a
multi-label setup where we have independent binary tasks, one for each label
(disease), and we used sigmoid as an output activation function and binary
cross entropy as a loss for each label.

We used the same hyper-parameters as in the natural image domain, except that
the base learning rate was set to 0.01 instead of 0.03, as we found that a
learning rate of 0.03 led to more overfitting. We followed [32] and used a
center crop based on the smallest side, then resized the image to $224\times
224$. In order to combat overfitting, we used data augmentation from [32],
which included random translation, random rotation, and random scaling. In
addition to the data augmentation used in [32], we also do random horizontal
flipping. In order to speedup training, we used data parallel distributed
training with Horovod [44], using 64 A100 GPUs in all pre-training setups.

###  3.2 Fine-tuning and transfer evaluation

For fine-tuning, we used the BiT-HyperRule[14], which is a heuristic that
selects fine-tuning hyper-parameters (learning rate schedule, resolution,
usage of MixUp, and total number of steps) based on training set size and
image resolution. We used a batch size of $128$, and an initial learning rate
of $0.001$ on all experiments. Like in [14], we do not use weight decay. Like
in pre-training, we used stochastic gradient descent (SGD) with adaptive
gradient clipping (AGC), as we found it to improve few-shot results. We used a
momentum of $0.9$. In each experiment, the classification head of the pre-
trained model was replaced with a new classification head for the fine-tuning
task. We fine-tuned all the layers of the network. For each experiment, we
performed $5$ independent runs with different seeds to have an estimate of the
variance of the performance. We ran each fine-tuning experiment on a single
A100 GPU.

As in [14], we consider two kinds of setups, few-shot setups (we used 1 or 5
or 10 or 100 or 500 examples per class) and fine-tuning on the full training
set. We used CIFAR-10, CIFAR-100 [35], Flowers-102 [36], and Oxford-IIIT Pet
[37] for natural image fine-tuning. For medical image fine-tuning, we used
single-label Tuberculosis [38] and COVIDx [39] as small X-Ray targets
($\approx$ 800 and 16k samples each), and multi-label CheXpert, MIMIC-CXR, NIH
or PadChest as larger X-Ray targets (magnitude order of 100k-300k samples, see
also Tab. 1). In addition, to perform few-shot experiments similar to natural
domain, we employ PadChest-cl, single-label dataset derived from PadChest,
where we keep only images with exactly one label (one disease). For
Flowers-102 and COVIDx, since the datasets are strongly imbalanced, we used
oversampling. We measure either final accuracy or mean AUC on the test sets.

###  3.3 Results.

The experiments allow us to look on both intra- and inter-domain transfer
performance following pre-training when varying model size, source data size
and source and target dataset domain. In following, we report the obtained
results.

Effect of scale on intra-domain transfer. Results we obtain either for
natural-natural or medical-medical full shot transfer (Tab. 2) deliver a clear
picture showing transfer improvement across target datasets when increasing
pre-training model and data scale. Most consistent is the improvement due to
increase of network size, while for data scale there are only few single cases
where the increase does not result in improvement (e.g, when using large
ResNet-152x4 on Pets for natural-natural and for PadChest for medical-medical
scenario; see Supplementary for more detailed results for each transfer
experiment scenario).

For few-shot transfer, we observe a differentiated picture. In line with
previous work, for natural-natural transfer we obtain strong improvement due
to larger scale in the very low data regime of 1- or 5-shot transfer, reaching
in some cases $20\%-30\%$ absolute difference in test accuracy in favor of
larger scale (as seen for CIFAR-100, Fig. 1(a)). In contrast, for medical-
medical scenario, there is no clear evidence for few-shot transfer improvement
due to larger scale (Figs. 1(b), 2(b), 2(d), 2(f); see also Supplementary for
further details). Increasing number of shots and approaching full shot regime,
the improvement due to scale becomes more and more visible. The observed
variance is larger for few-shot transfer experiments, which may suggest less
stable fine-tuning in those cases where model has to adapt to target data
based on only very limited number of examples.

Target | ResNet-50x1 | ResNet-152x4
---|---|---
| S-MED | L-MED | 1K-NAT | 21K-NAT | S-MED | L-MED | 1K-NAT | 21K-NAT
CIFAR-10(1) | 56.07 $\pm$ 0.32 | 63.27 $\pm$ 0.30 | 94.26 $\pm$ 0.05 | 95.78 $\pm$ 0.09 | 74.26 $\pm$ 0.20 | 78.05 $\pm$ 0.18 | 96.93 $\pm$ 0.05 | 97.82 $\pm$ 0.07
CIFAR-100(1) | 16.64 $\pm$ 0.21 | 18.71 $\pm$ 0.15 | 75.90 $\pm$ 0.05 | 82.47 $\pm$ 0.21 | 36.29 $\pm$ 0.29 | 37.94 $\pm$ 0.23 | 83.90 $\pm$ 0.09 | 88.54 $\pm$ 0.14
Flowers-102(1) | 7.05 $\pm$ 0.59 | 6.96 $\pm$ 1.26 | 74.94 $\pm$ 0.99 | 98.21 $\pm$ 0.22 | 25.19 $\pm$ 0.78 | 23.91 $\pm$ 0.86 | 89.41 $\pm$ 0.25 | 99.49 $\pm$ 0.08
Pets(1) | 7.06 $\pm$ 0.46 | 7.88 $\pm$ 0.42 | 85.21 $\pm$ 0.58 | 87.23 $\pm$ 0.18 | 15.07 $\pm$ 0.18 | 16.78 $\pm$ 0.35 | 93.32 $\pm$ 0.30 | 93.21 $\pm$ 0.14
COVIDx(1) | 68.50 $\pm$ 0.18 | 76.05 $\pm$ 0.21 | 76.30 $\pm$ 1.30 | 78.35 $\pm$ 1.63 | 78.65 $\pm$ 0.84 | 83.00 $\pm$ 1.16 | 78.10 $\pm$ 0.95 | 78.90 $\pm$ 0.49
Tuberculosis(1) | 79.83 $\pm$ 0.45 | 81.65 $\pm$ 0.91 | 79.83 $\pm$ 1.50 | 83.47 $\pm$ 0.83 | 79.01 $\pm$ 0.45 | 90.91 $\pm$ 0.83 | 81.49 $\pm$ 2.23 | 80.83 $\pm$ 2.51
MIMIC CXR(2) | 84.17 $\pm$ 0.03 | 86.38 $\pm$ 0.03 | 85.41 $\pm$ 0.10 | 86.82 $\pm$ 0.10 | 87.63 $\pm$ 0.04 | 88.00 $\pm$ 0.03 | 86.85 $\pm$ 0.06 | 87.79 $\pm$ 0.13
CheXpert(2) | 82.10 $\pm$ 0.07 | 86.66 $\pm$ 0.05 | 84.83 $\pm$ 0.14 | 86.60 $\pm$ 0.14 | 84.92 $\pm$ 0.07 | 87.82 $\pm$ 0.03 | 86.82 $\pm$ 0.06 | 87.77 $\pm$ 0.07
PadChest(2) | 68.06 $\pm$ 0.24 | 68.14 $\pm$ 0.21 | 76.72 $\pm$ 0.27 | 80.99 $\pm$ 0.22 | 75.91 $\pm$ 0.12 | 75.23 $\pm$ 0.17 | 79.59 $\pm$ 0.17 | 83.94 $\pm$ 0.19
PadChest-Cl(2) | 73.01 $\pm$ 0.13 | 78.33 $\pm$ 0.08 | 80.17 $\pm$ 0.17 | 82.03 $\pm$ 0.17 | 81.79 $\pm$ 0.07 | 82.68 $\pm$ 0.05 | 82.55 $\pm$ 0.05 | 84.02 $\pm$ 0.24
NIH CXR(2) | 70.11 $\pm$ 0.15 | 74.21 $\pm$ 0.57 | 75.53 $\pm$ 0.47 | 81.02 $\pm$ 0.57 | 77.95 $\pm$ 0.13 | 78.95 $\pm$ 0.13 | 79.82 $\pm$ 0.38 | 82.80 $\pm$ 0.41

Table 2: Varying model and data pre-training scale for intra- and inter-domain
transfer. Pre-training is performed with either natural or medical source
data, ordered by increasing scale - being one of large X-Ray datasets (S-MED),
larger compositional X-Ray supersets (L-MED, see text and also Tab. 3 on
composition details), ImageNet-1k (1K-NAT), ImageNet-21k (21K-NAT), using
either small ResNet-50x1 or large ResNet-152x4. (1) - Top-1 Acc [$\%$] metric;
(2) - mean AUC metric. Bold indicates best transfer performance for a fixed
network size and pinpoints the effect of data scale on transfer. Italics
indicates transfer performance with no significant difference between data
scale. Red indicates best overall performance for a given target. Gray
indicates medical-natural transfer control experiment.

Effect of scale on inter-domain transfer. Here we transfer on either small or
large medical X-Ray chest imaging targets after pre-training on natural
sources of different size, ImageNet-1k (1.4M samples) or much larger
ImageNet-21k (14M samples). For all large X-Ray targets (NIH, CheXpert,
PadChest, PadChest-cl or MIMIC-CXR) we observe clear full-shot transfer
improvement due to larger pre-training scale (Tab. 2). The effect is
consistent for both model and data scale across large X-Ray targets.

For small X-Ray targets (Tuberculosis and COVIDx), we do not observe such
consistent improvement due to larger scale. For instance, while we see
improvement due to larger data scale for small ResNet-50x1 on both small
targets, the improvement is not there when increasing network size. There is
also no evidence for systematic positive effect of larger scale on few-shot
transfer, neither for large nor for small X-Ray targets (Figs. 1(b), 2(a),
2(c), 2(e)). Again, variance observed in few-shot regime is large, and is
getting smaller and smaller when increasing number of shots and moving towards
full shot transfer.

Target | ResNet-50x1 | ResNet-152x4
---|---|---
CheXpert | +PadChest | +MIMIC | CheXpert | +PadChest | +MIMIC
NIH |  70.11 $\pm$ 0.15 | 73.37 $\pm$ 0.38 | 74.21 $\pm$ 0.57 |  77.95 $\pm$ 0.13 |  78.16 $\pm$ 0.13 | 78.95 $\pm$ 0.13
| CheXpert | +MIMIC | +NIH | CheXpert | +MIMIC | +NIH
PadChest |  68.06 $\pm$ 0.24 | 70.07 $\pm$ 0.49 |  68.14 $\pm$ 0.21 | 75.91 $\pm$ 0.12 | 75.81 $\pm$ 0.07 |  75.23 $\pm$ 0.17
PadChest-Cl |  73.01 $\pm$ 0.13 | 78.44 $\pm$ 0.04 | 78.33 $\pm$ 0.08 |  81.79 $\pm$ 0.07 | 83.14 $\pm$ 0.04 |  82.68 $\pm$ 0.05
| PadChest | +MIMIC | +NIH | PadChest | +MIMIC | +NIH
CheXpert |  82.10 $\pm$ 0.07 | 86.56 $\pm$ 0.08 | 86.66 $\pm$ 0.05 |  84.92 $\pm$ 0.07 | 88.03 $\pm$ 0.03 |  87.82 $\pm$ 0.03
| CheXpert | +PadChest | +NIH | CheXpert | +PadChest | +NIH
MIMIC CXR |  84.17 $\pm$ 0.03 |  86.19 $\pm$ 0.03 | 86.38 $\pm$ 0.03 |  87.63 $\pm$ 0.04 | 88.13 $\pm$ 0.03 |  88.00 $\pm$ 0.03

Target | ResNet-50x1 | ResNet-152x4
---|---|---
CheXpert | +MIMIC | \+ NIH | +PadChest | CheXpert | +MIMIC | +NIH | +PadChest
COVIDx(1) |  68.50 $\pm$ 0.18 |  75.10 $\pm$ 1.52 |  75.60 $\pm$ 0.45 | 76.05 $\pm$ 0.21 |  78.65 $\pm$ 0.84 |  81.65 $\pm$ 0.74 |  80.80 $\pm$ 1.10 | 83.00 $\pm$ 1.16
Tuberculosis(1) |  79.83 $\pm$ 0.45 |  78.84 $\pm$ 1.25 | 81.32 $\pm$ 0.74 | 81.65 $\pm$ 0.91 |  79.01 $\pm$ 0.45 |  84.63 $\pm$ 0.74 |  87.93 $\pm$ 0.74 | 90.91 $\pm$ 0.83

Table 3: Intra-domain transfer using different sized medical X-Ray source data
for pre-training with different sized ResNets, using different X-Ray targets
(Mean AUC metric; above: larger, below: smaller targets). "+" indicates
addition of a set into a successively larger source superset. For each target,
source datasets correspond to either S-MED datasets or L-MED supersets
reported in Tab. 2. Clear transfer improvement is evident for increasing both
model and data scale. While improvement via scaling up the model is
consistent, scaling up pre-training data shows in some cases no significant
change, especially when adding a smaller dataset like NIH which leads to a
rather small increase in data scale. Bold, Italics, Red and Gray indicate the
same as in Tab. 2

Remarkably, when further comparing intra- and inter-domain transfer
performance, we observe that large ResNet-152x4 pre-trained on very large
generic natural ImageNet-21k are as good or better than networks pre-trained
on largest available medical domain specific X-Ray superset data when
performing full shot transfer to large X-Ray targets (Tab. 2, Figs. 2(a),
2(b)). This fits into overall picture of larger model and data pre-training
scale improving transfer on larger targets observed here, as ImageNet-21k has
order of magnitude larger scale than the largest X-Ray superset constructed
for this study.

For control, we also conduct inter-domain transfer in medical-natural scenario
(grayed out area in Tab. 2). There, as expected, after pre-training on
specific X-Ray data the performance on natural targets is severely diminished
when compared to transfer performance shown by models pre-trained on natural
images. The effect of scale is though still preserved in this control
scenario, clearly favoring large over the small network scale.

##  4 Discussion & Conclusion

Our observations of the transfer performance dependency on the pre-training
model and data scale and on source and target domain alignment suggest that
both intra- and inter-domain transfer benefit from larger pre-training scale.
The effect of pre-training scale depends however on transfer conditions,
revealing a differentiated picture of when larger scale may lead to transfer
improvement.

Larger pre-training scale improves intra-domain transfer We obtain evidence
that both natural-natural domain transfer and medical-medical domain transfer
are improved when increasing model and data size during pre-training (Tab. 2;
see also Suppl. material). For natural-natural transfer scenario, the
improvement is evident for both full and few-shot regime. Increasing data size
by using ImageNet-21k instead of ImageNet-1k or increasing network model size
by using ResNet-152x4 instead of smaller ResNet-50x1 creates strong,
consistent boost in transfer performance across all natural target datasets,
which is in line with previous observations [14, 15]. Improvement is
especially pronounced in few-shot regime (e.g, Fig. 1(a), see also Suppl.
material), also adding evidence for more data efficient transfer due to larger
scale. The picture is more differentiated for medical-medical transfer
scenario. For full transfer regime, the improvement due to larger pre-training
scale is clear and consistent across different targets (Tab. 2). In few-shot
transfer however, in contrast to natural-natural scenario (Fig. 1(a)), there
are no benefits due to larger scale (Fig. 1(b), 2(b), 2(d), 2(f)). Here we
have to keep in mind that both absolute data size and increase in data scale
we obtain by going from ImageNet-1k to ImageNet-21k (14M samples) in natural
pre-training is much stronger than what we achieve in medical pre-training,
going from one of X-Ray datasets to the largest combined X-Ray superset that
still has much smaller data volume ($\approx$ 870k samples) than ImageNet-21k.
This difference in data scale may also explain the observed differences in
few-shot regime, while we also cannot rule out that domain type (natural or
medical) could as well play an important role in determining how transfer is
affected by pre-training scale.

(a) CIFAR-100

(b) PadChest-Cl

Figure 1: Few- and full shot transfer performance on a natural and a medical
X-Ray target when varying model and data scale in pre-training.

Larger pre-training scale improves inter-domain transfer for larger targets.
In contrast to intra-domain transfer, where natural-natural or medical-medical
source and target datasets are closely related, in natural-medical inter-
domain transfer the source and target are much further apart. It is therefore
not trivial that effect of pre-training scale brings similar or any
improvement in this case as well. Strong discrepancy between source and target
may render transfer ineffective, as it was indeed observed in previous studies
on natural-medical transfer done on smaller scales [22]. In contrast to these
studies, we do find here significant positive effect of larger pre-training
scale on inter-domain natural-medical full shot transfer for larger medical
targets (Tab. 2; Fig. 1(b), 2(a), see also Suppl. material). The transfer
improvement is clearly expressed when increasing both model and natural image
data pre-training scale across all large X-Ray targets. Interestingly, the
effect of scale is still clearly expressed even when conducting inter-domain
transfer in a strongly unfavorable setting, pre-training on narrow domain-
specific X-Ray data and transferring to generic natural images (grayed out
area in Tab. 2). This indicates that scale can still have beneficial effect
even in situations where transfer scenario itself is ill-posed.

Another remarkable finding arises when further comparing performance of intra-
domain medical-medical and inter-domain natural-medical transfer. The largest
ResNet-152x4 pre-trained on the largest generic natural ImageNet-21k turns out
to be as good or in many cases better than any network pre-trained on the
largest available medical domain specific X-Ray superset data when performing
full shot transfer to large X-Ray targets (Tab. 2, Figs. 1(b), 2(a), 2(b)).
The finding indicates that by substantially increasing model and generic
source natural image data scale during pre-training, we can obtain models for
transfer to medical domain-specific X-Ray images that match or even outperform
models pre-trained with large amounts of domain-specific X-Ray data, which may
be often not available in practice. The finding also strengthens previously
inconclusive evidence for benefits of large-scale pre-training on generic
natural images when transferring to medical targets [34], providing direct
comparison between models pre-trained either on out-of-domain natural images
or on in-domain medical X-Ray chest imaging data when transferring to medical
image targets. The observation expands the previous evidence that strongly
increasing model and data pre-training scale improves transfer also for larger
out-of-domain targets - scale of generic ImageNet-21k (14M samples) is more
than order of magnitude larger than scale of the largest medical X-Ray
superset we have constructed for this study ($\approx$ 0.87M samples).

For few-shot inter-domain transfer or for transfer on smaller targets, we do
not see consistent positive effect of larger pre-training scale (Figs. 1(b),
2). Thus, we again obtain differential picture of scaling benefits depending
on transfer scenario. Further scaling up of model and data during pre-training
may homogenize this picture and make scaling benefit look more consistent
through different conditions, showing improvement for smaller targets and for
few-shot transfer. For instance, we could not use substantially larger
datasets like JFT-300M or JFT-3B [29, 14, 30] which are proprietary and not
available publicly. Additionally, computational budget was here not enough to
experiment with networks larger than ResNet-152x4. However, there may be also
fundamental limitations prohibiting transfer improvement no matter how large
pre-training scale may become that are due to strong incompatibility between
source and target domains. There is some evidence from language modeling
studies that hints on such fundamental limitations for transfer improvement on
target datasets far from source when doing straightforward scaling without
further changes in model architecture. For instance, the work by Hendrycks et
al. [46] finds no improvement of transfer when increasing size of large
Transformer networks pre-trained on very large conventional text datasets and
fine-tuning those on a specific target dataset (mathematical text tasks of
advanced difficulty) far apart from the source.

(a) PadChest-Cl, natural source

(b) PadChest-Cl, medical source

(c) COVIDx, natural source

(d) COVIDx, medical source

(e) Tuberculosis, natural sources

(f) Tuberculosis, medical sources

Figure 2: Few-shot and full shot transfer performance on medical X-Ray targets
of different size for intra- and inter-domain scenarios, medical-medical or
natural-medical. Each color represents a combination of model and data scale
during pre-training. Positive effect of larger scale on full-shot transfer is
evident for large X-Ray target PadChest-Cl (top), both for intra- and inter-
domain scenario. Remarkably, by increasing natural image data scale the
largest network outperforms there the same network pre-trained on large X-Ray
data (compare top (a) and (b), full-shot setting). Effect of scale is not
evident for few-shot transfer and for small targets in inter-domain setting.

Limitations of the current study. There are several limitations of the current
study that impede more general conclusions about effect of pre-training scale
on intra- and inter-domain transfer from the observations made in this study.
In the conducted transfer experiments, we made use of a heuristic hyper-
parameter selection rule - BiT-HyperRule, as introduced in [14] \- that
determines pre-training hyper-parameters directly from target datasets on
which transfer is to be performed. This rule may be heavily biased towards
transfer on natural image datasets, as those were the targets used in the
original study. If modifying the rule to take also target domain - natural or
medical - into account, the derived hyper-parameters may serve a much better
basis for fine-tuning during transfer. In general, performing hyper-parameter
tuning for training procedure can strongly boost performance [47], and this is
no different for transfer procedure. Therefore, it cannot be excluded that
performing hyper-parameter tuning for each transfer task would alter the
effect of larger pre-training scale on transfer. Hyper-parameter tuning would
however also impose further cost on transfer that is avoided by employing the
hyper-rule.

We also have not explored other backbone network architectures except the
standard ResNet. Although ResNet has proven itself a versatile network
architecture for dealing with various vision tasks, it cannot be ruled out
that while for instance the inductive bias inherent to its convolutional
design is well suited for working on natural image statistics with strong
local spatial correlations, it may be less suited for providing good basis for
generalization when dealing with other types of image signals. Scaling up
ResNet architecture may thus be a viable strategy to improve generalization
capability on natural image data, while other, more generic architectures, may
be required to benefit from scaling in the same way across more diverse data
types. We also did not experiment with larger datasets than ImageNet-21k - as
those are still mostly proprietary and were not publicly available, as it is
the case for JFT-300M [29, 14] or JFT-3B [30]. For medical imaging domain, we
could not experiment with increasing data scale substantially, as the amount
of openly available X-Ray chest data is currently still limited. Finally, we
studied the dependence of transfer improvement on pre-training scale
exclusively in supervised classification problem setting. Promising work is
also done on pre-training in unsupervised fashion with unlabeled data [48, 49,
50], where benefits of scaling up pre-training for transfer may turn out to be
as well substantial.

Conclusion and outlook. Here, we presented here evidence that substantially
increasing model and data scale in the pre-training provides benefits for both
intra- and inter-domain transfer across various target datasets from natural
and medical X-Ray image domain. The effect of pre-training scale on transfer
performance depends on transfer scenario. Transfer improvement due to larger
pre-training scale was found to be substantial in natural-natural or medical-
medical, intra-domain transfer scenarios where source and target datasets were
closely related, being especially strongly pronounced in the few-shot transfer
regime for natural-natural case and concentrated in full-shot scenario in
medical-medical case. For natural-medical inter-domain transfer, clear
positive effect of larger pre-training scale was found for full shot transfer
on large X-Ray targets. On small X-Ray targets and for few-shot transfer
regime, no clear inter-domain transfer improvements were observed. Remarkably,
the largest ResNet-152x4 network pre-trained on very large generic natural
ImageNet-21k matched or even outperformed networks pre-trained on largest
medical domain-specific X-Ray superset data combined for this study when
performing full shot transfer to large X-Ray targets. This is relevant for the
practice, as large amount of medical domain-specific data is often not
available for pre-training. Here we show that high quality models for large
X-Ray targets can be also obtained by substantially increasing pre-training
model and generic natural image source data scale instead, obliterating need
for large domain-specific data.

The study offers different follow-up directions. One of these are experiments
with larger scale both for network and data size, for instance going beyond
ImageNet-21k, or combining in the pre-training different source datasets that
may contain both natural and medical images. This may also include experiments
with scaling up other architectures than ResNet. Another direction to study
effect of scale is to employ various unsupervised learning strategies for pre-
training [48, 49, 50] instead of supervised learning. Yet another fruitful
path is to provide a measure of source and target domain similarity and to
experiment with more than two distinct domains, varying systematically
relatedness between different source and target data. First steps in this
direction for language modeling was already undertaken in [23]. Following
these directions would pave the path towards scaling laws for transfer in the
image domain, taking into account different pre-training regimes and affinity
between source and target domains, to enable systematic prediction of transfer
performance and improvement due to increase of pre-training scale.

## Broader and Social Impact

Our work aims on advancing transfer learning, which can make learning
algorithms perform better and more efficient by re-using models already pre-
trained on various tasks and therefore requiring less compute and data to
learn solutions for other relevant tasks. The approach to improve transfer
learning by increasing scale of the pre-training is generic and has impact far
beyond vision domain, for instance in language modeling, and is not bound to
any specific application. As any generic method, it can be therefore applied
to enhance technologies for sensitive applications, for instance in health
domain or in public surveillance, that may have both strong positive and
negative social impact, depending on policies introduced on their usage.
Special care should be taken about applications in clinical domain where
further development of diagnostic tools based on data driven machine learning
should be accompanied by a broad panel of experts from corresponding domains.
The method depends on computationally heavy large-scale pre-training that is
energy demanding on the one hand. On the other hand, it contains a promise to
pay off the energy budget put into training by obtaining generic models that
can be very efficiently adapted to a large range of problems via transfer,
saving computational and energy costs that would otherwise incur for their
solution from scratch.

## Acknowledgments and Disclosure of Funding

We would like to express gratitude to all the people who are working on making
code, models and data publicly available, advancing community based research
and making research more reproducible. Special thanks go to creators and
maintainers of open available X-Ray medical imaging datasets that also enabled
our research, some of those gathered under difficult circumstances of the
COVID-19 pandemics. The authors gratefully acknowledge the Gauss Centre for
Supercomputing e.V. (www.gauss-centre.eu) for funding this work by providing
computing time through the John von Neumann Institute for Computing (NIC) on
the GCS Supercomputers JUWELS, JUWELS Booster at Jülich Supercomputing Centre
(JSC). We also acknowledge computing resources from the Helmholtz Data
Federation and further computing time provided on supercomputer JUSUF in frame
of offer for epidemiology research on COVID-19 by JSC.

## References

  * [1] Lorien Y Pratt, Jack Mostow, Candace A Kamm, and Ace A Kamm.  Direct transfer of learned information among neural networks.  In AAAI, volume 91, pages 584–589, 1991.
  * [2] Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2009.
  * [3] A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson.  CNN features off-the-shelf: An astounding baseline for recognition.  In Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, pages 512–519, June 2014.
  * [4] Hossein Azizpour, Ali Sharif Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson.  Factors of transferability for a generic convnet representation.  IEEE transactions on pattern analysis and machine intelligence, 38:1790–1802, September 2016.
  * [5] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  ImageNet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [6] Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, and Yann LeCun.  Overfeat: Integrated recognition, localization and detection using convolutional networks.  In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
  * [7] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR, San Diego, CA, USA, 2015.
  * [8] J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 248–255, June 2009.
  * [9] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [10] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al.  A large-scale study of representation learning with the visual task adaptation benchmark.  arXiv preprint arXiv:1910.04867, 2019.
  * [11] Thomas Mensink, Jasper Uijlings, Alina Kuznetsova, Michael Gygli, and Vittorio Ferrari.  Factors of influence for transfer learning across diverse appearance domains and task types.  arXiv preprint arXiv:2103.13318, 2021.
  * [12] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  arXiv preprint arXiv:2001.08361, 2020.
  * [13] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, et al.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * [14] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 491–507, Cham, 2020. Springer International Publishing.
  * [15] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor.  Imagenet-21k pretraining for the masses.  arXiv preprint arXiv:2104.10972, 2021.
  * [16] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pages 770–778, June 2016.
  * [17] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 590–597, 2019.
  * [18] Alistair E. W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-Ying Deng, Roger G. Mark, and Steven Horng.  Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports.  Scientific data, 6:317, December 2019.
  * [19] Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria de la Iglesia-Vayá.  Padchest: A large chest x-ray image dataset with multi-label annotated reports.  Medical image analysis, 66:101797, December 2020.
  * [20] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald Summers.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pages 3462–3471, 2017.
  * [21] Juelich Supercomputing Center.  JUWELS Booster Supercomputer, 2020.  https://apps.fz-juelich.de/jsc/hps/juwels/configuration.html#hardware-configuration-of-the-system-name-booster-module.
  * [22] Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio.  Transfusion: Understanding transfer learning for medical imaging.  In Advances in Neural Information Processing Systems 32, pages 3347–3357. 2019.
  * [23] Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish.  Scaling laws for transfer.  arXiv preprint arXiv:2102.01293, 2021.
  * [24] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish.  Scaling laws for autoregressive generative modeling.  arXiv preprint arXiv:2010.14701, 2020.
  * [25] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal.  Reconciling modern machine-learning practice and the classical bias-variance trade-off.  Proceedings of the National Academy of Sciences of the United States of America, 116:15849–15854, August 2019.
  * [26] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever.  Deep double descent: Where bigger models and more data hurt.  In International Conference on Learning Representations, 2019.
  * [27] Stéphane d'Ascoli, Levent Sagun, and Giulio Biroli.  Triple descent and the two kinds of overfitting: where & why do they appear?  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3058–3069. Curran Associates, Inc., 2020.
  * [28] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  Communications of the ACM, 59(2):64–73, 2016.
  * [29] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.  Revisiting unreasonable effectiveness of data in deep learning era.  arXiv:1707.02968, 2017.
  * [30] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  arXiv preprint arXiv:2106.04560, 2021.
  * [31] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle.  Meta-dataset: A dataset of datasets for learning to learn from few examples.  In International Conference on Learning Representations, 2020.
  * [32] Joseph Paul Cohen, Mohammad Hashir, Rupert Brooks, and Hadrien Bertrand.  On the limits of cross-domain generalization in automated x-ray prediction.  In Medical Imaging with Deep Learning, pages 136–155. PMLR, 2020\.
  * [33] Alexander Ke, William Ellsworth, Oishi Banerjee, Andrew Y. Ng, and Pranav Rajpurkar.  Chextransfer: Performance and parameter efficiency of imagenet models for chest x-ray interpretation.  In Proceedings of the Conference on Health, Inference, and Learning, CHIL ’21, page 116–124, New York, NY, USA, 2021. Association for Computing Machinery.
  * [34] Basil Mustafa, Aaron Loh, Jan Freyberg, Patricia MacWilliams, Alan Karthikesalingam, Neil Houlsby, and Vivek Natarajan.  Supervised transfer learning at scale for medical imaging.  arXiv preprint arXiv:2101.05913, 2021.
  * [35] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, 2009.
  * [36] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729. IEEE, 2008.
  * [37] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar.  Cats and dogs.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 3498–3505, June 2012.
  * [38] Stefan Jaeger, Sema Candemir, Sameer Antani, Yì-Xiáng J. Wáng, Pu-Xuan Lu, and George Thoma.  Two public chest x-ray datasets for computer-aided screening of pulmonary diseases.  Quantitative imaging in medicine and surgery, 4:475–477, December 2014.
  * [39] Linda Wang, Zhong Qiu Lin, and Alexander Wong.  Covid-net: a tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images.  Scientific reports, 10:19549, November 2020.
  * [40] Andrew Brock, Soham De, Samuel L Smith, and Karen Simonyan.  High-performance large-scale image recognition without normalization.  arXiv preprint arXiv:2102.06171, 2021.
  * [41] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  arXiv preprint arXiv:1706.02677, 2017.
  * [42] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702–703, 2020.
  * [43] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [44] Alexander Sergeev and Mike Del Balso.  Horovod: fast and easy distributed deep learning in tensorflow.  arXiv preprint arXiv:1802.05799, 2018.
  * [45] Joseph Paul Cohen, Joseph Viviano, Paul Morrison, Rupert Brooks, Mohammad Hashir, and Hadrien Bertrand.  TorchXRayVision: A library of chest X-ray datasets and models.  https://github.com/mlmed/torchxrayvision, 2020.
  * [46] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the math dataset.  arXiv preprint arXiv:2103.03874, 2021.
  * [47] Christopher J Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl.  Measuring the effects of data parallelism on neural network training.  Journal of Machine Learning Research, 20:1–49, 2019.
  * [48] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton.  Big self-supervised models are strong semi-supervised learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 22243–22255. Curran Associates, Inc., 2020.
  * [49] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le.  Self-training with noisy student improves imagenet classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10687–10698, 2020.
  * [50] Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le.  Rethinking pre-training and self-training.  Advances in Neural Information Processing Systems, 33, 2020.

Supplementary: Effect of Pre-Training Scale on Intra- and Inter-Domain Full
and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images

##  Appendix A Distributed Training

###  A.1 JUWELS Booster Supercomputer

Installed in November 2020, JUWELS Booster [21] features $936$ compute nodes
that host four NVIDIA A100 GPUs each, providing $3744$ GPUs in total. The
installed A100 Tensor Core GPUs ($40\text{\,}\mathrm{GB}\text{/}$) provide
$19.5\text{\,}\mathrm{TFLOP}\text{/}\mathrm{s}$ of $\text{FP64}_{\text{TC}}$
computing performance each. The GPUs are hosted by AMD EPYC 7402 CPUs with
$2\times 24$ cores (SMT-2) per node, clocked with
$2.8\text{\,}\mathrm{GHz}\text{/}$. Each node is diskless and is equipped with
$512\text{\,}\mathrm{GB}\text{/}$ of RAM. The network of JUWELS Booster is
based on Mellanox HDR200 InfiniBand, with four Mellanox ConnectX 6 devices per
node, each providing $200\text{\,}\mathrm{Gbit}\text{/}\mathrm{s}$ bandwidth
per direction.

The NVIDIA A100 GPUs installed into JUWELS Booster reach peak efficiency of
$48.75\text{\,}\mathrm{GFLOP}\text{/}\text{(}\mathrm{s}\text{\,}\mathrm{W}\text{)}$
when utilizing the FP64 Tensor Cores. This makes JUWELS Booster rank highest
in the Green500 list of November 2020 as the most energy efficient
supercomputer among the first 100 machines of the Top500 list with
$25\text{\,}\mathrm{GFLOP}\text{/}\text{(}\mathrm{s}\text{\,}\mathrm{W}\text{)}$.

###  A.2 Scaling and training time

Here, we report scaling behavior during large-scale pre-training for ResNet
networks we used in the experiments.

We performed scaling experiments to assess the scalability of data parallel
training distributed across many GPUs on multiple nodes using Horovod. The
efficiency in Figure 3(b) (upper part of the figure with percentages) is
computed using the following formula: $E(N)=100\times\frac{T(N)}{N\times
T(1)}$. $T(N)$ is the total measured throughput in Im/s for $N$ GPUs. The best
achievable efficiency, when scaling is perfect, is 100%.

We also provide the raw throughput (Im/s) numbers in Figure 3(a) and Tab. 4.
On 1024 GPUs, we achieve an efficiency of $\approx 93.7\%$ with single
precision (FP32). To make sure distributed training is stable, we check the
end accuracy of full training for each number of GPUs to reassure we reach
target accuracy acceptable for standard ImageNet-1k Top-1 and Top-5 results.

Achieved scaling on JUWELS Booster allows to perform full pre-training on
ImageNet-21k with large R152x4 in about 81 hours using $256$ GPUs. For small
R50x1, full training needs about 13.5 hours to finish using $128$ GPUs.

(a) Throughput in Im/s

(b) Scaling efficiency

Figure 3: Distributed training for R152x4, scaling behavior on JUWELS Booster using A100 GPUs. Table 4:  Scaling behavior in Im/s of ImageNet-1k training using ResNet-152x4 architecture from [14] with batch size 128. For each GPU, one MPI process is assigned. Computations were done on up to 256 nodes on JUWELS Booster. Throughput performance during training is reported for single precision mode (FP32). The corresponding speedup is provided relative to reference training with 1 GPU. Note that the measured Im/s throughput includes I/O. #GPUs | Im/s | speedup
---|---|---
1 |  1 129.14 | 1.00
4 |  1 508.00 | 3.93
8 |  1 1009.30 | 7.82
16 |  1 2023.78 | 15.67
32 |  1 4029.69 | 31.21
64 |  1 8022.31 | 62.12
128 |  1 15959.86 | 123.59
256 |  1 31758.59 | 245.93
512 |  1 62496.35 | 483.96
1024 |  1 124003.59 | 960.26

##  Appendix B Additional details on experimental results

All datasets employed in our experiments are publicly available and can be
obtained following links in the Tab. 1

###  B.1 Further transfer results

Here, we present more detailed results of transfer experiments described in
the main document. For medical X-Ray targets, we provide tables reporting
transfer performance (Tabs. 6, 7, 8, 9, 10) listing each source X-Ray dataset
and supersets used for pre-training, as outlined in the experiments
description in the main document.

(a) Flowers-102

(b) CIFAR-10

Figure 4:  Few-shot and full shot transfer performance on target datasets when
varying model size and dataset size in pre-training. Transfer improvement due
to model and source data size is evident, especially strongly pronounced in
few-shot regime.  Figure 5: Few-shot and full shot transfer performance on
COVIDx dataset when varying model, source data size and domain in pre-
training. In the full shot transfer, improvement due to model and data scale
is evident when pre-training on X-Ray chest imaging source data. In few-shot
regime, no transfer improvement due to larger model or data size is observed.
Figure 6: Few-shot and full shot transfer performance on Tuberculosis dataset
when varying model, source data size and domain in pre-training. In the full
shot transfer, improvement due to model and data scale is evident when pre-
training on X-Ray chest imaging source data. In few-shot regime, no transfer
improvement due to larger model or data size is observed.

(a) Tuberculosis, natural sources

(b) Tuberculosis, medical sources

Figure 7:  Few-shot and full shot transfer performance on Tuberculosis dataset
when pre-training with different model sizes on different sources (natural or
medical datasets) of various sizes. In natural-medical scenario (a), no
transfer improvement due to model or data scale is evident. In medical-medical
scenario (b), larger model and data size lead to transfer improvement in full
shot regime, without benefits in few-shot mode.

(a) CIFAR-100

(b) COVIDx

(c) PadChest-Cl

Figure 8: Full shot transfer performance on target datasets when varying model and source data size, taking the smallest and largest pre-training datasets available for each domain. Table 5: Intra- and inter-domain transfer using natural ImageNet-1k and ImageNet-21k for pre-training with different sized ResNets (1) - Top-1 Acc [$\%$] metric; (2) - mean AUC metric. Bold indicates best transfer performance for a fixed network size and pinpoints the effect of data scale on transfer. Italics indicates transfer performance with no significant difference between data scale. Red indicates best overall performance for a given target. Clear transfer improvement emerges for natural-natural scenario due to both model and data scale. For natural-medical scenario the positive effect of larger scale is consistently given for larger targets, but not for smaller ones. For instance, for very small Tuberculosis target, larger data scale improves transfer for small ResNet-50x1, while larger model scale does not lead to any transfer improvement. Target Domain | Dataset | ResNet-50x1 | ResNet-152x4
---|---|---|---
1K | 21K | 1K | 21K
Natural |  CIFAR-10(1) |  94.26 $\pm$ 0.05 | 95.78 $\pm$ 0.09 |  96.93 $\pm$ 0.05 | 97.82 $\pm$ 0.07
CIFAR-100(1) |  75.90 $\pm$ 0.05 | 82.47 $\pm$ 0.21 |  83.90 $\pm$ 0.09 | 88.54 $\pm$ 0.14
Flowers-102 (1) |  74.94 $\pm$ 0.99 | 98.21 $\pm$ 0.22 |  89.41 $\pm$ 0.25 | 99.49 $\pm$ 0.08
Pets (1) |  85.21 $\pm$ 0.58 | 87.23 $\pm$ 0.18 | 93.32 $\pm$ 0.30 | 93.21 $\pm$ 0.14
Medical |  Tuberculosis(1) |  79.83 $\pm$ 1.50 |  83.47 $\pm$ 0.83 | 81.49 $\pm$ 2.23 | 80.83 $\pm$ 2.51
COVIDx(1) | 76.30 $\pm$ 1.30 | 78.35 $\pm$ 1.63 | 78.10 $\pm$ 0.95 | 78.90 $\pm$ 0.49
NIH (2) |  75.53 $\pm$ 0.47 | 81.02 $\pm$ 0.57 |  79.82 $\pm$ 0.38 | 82.80 $\pm$ 0.41
PadChest-Cl(2) |  80.17 $\pm$ 0.17 | 82.03 $\pm$ 0.17 |  82.55 $\pm$ 0.05 | 84.02 $\pm$ 0.24
PadChest (2) |  76.72 $\pm$ 0.27 | 80.99 $\pm$ 0.22 |  79.59 $\pm$ 0.17 | 83.94 $\pm$ 0.19
CheXpert (2) |  84.83 $\pm$ 0.14 | 86.60 $\pm$ 0.14 |  86.82 $\pm$ 0.06 | 87.77 $\pm$ 0.07
MIMIC CXR(2) |  85.41 $\pm$ 0.10 | 86.82 $\pm$ 0.10 |  86.85 $\pm$ 0.06 | 87.79 $\pm$ 0.13
Table 6: Intra-domain transfer using different sized medical X-Ray source data for pre-training with different sized ResNets (1) - Top-1 Acc [$\%$] metric; (2) - mean AUC metric. "+" indicates addition into a successively larger source superset. Clear transfer improvement is evident due to larger model and data scale across different targets. Target | ResNet-50x1 | ResNet-152x4
---|---|---
CheXpert | +MIMIC | \+ NIH | +PadChest | CheXpert | +MIMIC | +NIH | +PadChest
PadChest-Cl(2) |  73.01 $\pm$ 0.13 | 78.44 $\pm$ 0.04 | 78.33 $\pm$ 0.08 | — |  81.79 $\pm$ 0.07 | 83.14 $\pm$ 0.04 |  82.68 $\pm$ 0.05 | —
COVIDx(1) |  68.50 $\pm$ 0.18 |  75.10 $\pm$ 1.52 |  75.60 $\pm$ 0.45 | 76.05 $\pm$ 0.21 |  78.65 $\pm$ 0.84 |  81.65 $\pm$ 0.74 |  80.80 $\pm$ 1.10 | 83.00 $\pm$ 1.16
Tuberculosis(1) |  79.83 $\pm$ 0.45 |  78.84 $\pm$ 1.25 | 81.32 $\pm$ 0.74 | 81.65 $\pm$ 0.91 |  79.01 $\pm$ 0.45 |  84.63 $\pm$ 0.74 |  87.93 $\pm$ 0.74 | 90.91 $\pm$ 0.83
Table 7: Intra-domain transfer using different sized medical X-Ray source data for pre-training with different sized ResNets, target MIMIC-CXR Mean AUC metric. "+" indicates addition into a successively larger source superset. Clear transfer improvement is evident by scaling the model size. Using a superset containing CheXpert and PadChest improves the results, but adding NIH does not or does very little, this could be explained by the fact that NIH is the smallest dataset among the medical pre-training datasets, and a larger increase in the superset would be needed to substantially improve the transfer results, as it has been observed in transfer results that were obtained using models pre-trained on much larger natural data. Target | ResNet-50x1 | ResNet-152x4
---|---|---
CheXpert | +PadChest | \+ NIH | CheXpert | +PadChest | +NIH
MIMIC CXR |  84.17 $\pm$ 0.03 |  86.19 $\pm$ 0.03 | 86.38 $\pm$ 0.03 |  87.63 $\pm$ 0.04 | 88.13 $\pm$ 0.03 |  88.00 $\pm$ 0.03
Table 8: Intra-domain transfer using different sized medical X-Ray source data for pre-training with different sized ResNets, target CheXpert Mean AUC metric. "+" indicates addition into a successively larger source superset. Clear transfer improvement is evident by scaling the model size. Using a superset containing PadChest and MIMIC CXR improves the results, adding NIH does not lead to further improvement. This could be explained by the fact that NIH is the smallest dataset among the medical pre-training datasets, and a larger increase in the superset would be needed to substantially improve the transfer results, as it has been observed in transfer results that were obtained using models pre-trained on much larger natural data. Target | ResNet-50x1 | ResNet-152x4
---|---|---
PadChest | +MIMIC | \+ NIH | PadChest | +MIMIC | +NIH
CheXpert |  82.10 $\pm$ 0.07 | 86.56 $\pm$ 0.08 | 86.66 $\pm$ 0.05 |  84.92 $\pm$ 0.07 | 88.03 $\pm$ 0.03 |  87.82 $\pm$ 0.03
Table 9: Intra-domain transfer using different sized medical X-Ray source data for pre-training with different sized ResNets, target PadChest Mean AUC metric. "+" indicates addition into a successively larger source superset. Clear transfer improvement is evident by scaling the model size. Improvement by increasing data size is not evident and only happens using the small R50x1 model and a superset containing CheXpert and MIMIC, adding NIH (which is smaller compared to CheXpert and MIMIC) the superset does not help further. This indicates that larger increase in the superset may be necessary to further improve the transfer results, as it has been observed when using models pre-trained on much larger natural data. Target | ResNet-50x1 | ResNet-152x4
---|---|---
CheXpert | +MIMIC | \+ NIH | CheXpert | +MIMIC | +NIH
PadChest |  68.06 $\pm$ 0.24 | 70.07 $\pm$ 0.49 |  68.14 $\pm$ 0.21 | 75.91 $\pm$ 0.12 | 75.81 $\pm$ 0.07 |  75.23 $\pm$ 0.17
Table 10: Intra-domain transfer using different sized medical X-Ray source data for pre-training with different sized ResNets, target NIH (Mean AUC metric). "+" indicates addition into a successively larger source superset. Clear transfer improvement is evident by scaling the model size. We also observe transfer improvement by scaling data size, however the improvement seems to flatten. Since the transfer results using pre-trained models on larger natural data show a better performance, this indicates that a larger superset scale may be necessary to further improve transfer. Target | ResNet-50x1 | ResNet-152x4
---|---|---
CheXpert | +PadChest | \+ MIMIC | CheXpert | +PadChest | +MIMIC
NIH |  70.11 $\pm$ 0.15 | 73.37 $\pm$ 0.38 | 74.21 $\pm$ 0.57 |  77.95 $\pm$ 0.13 |  78.16 $\pm$ 0.13 | 78.95 $\pm$ 0.13

##  Appendix C Code and Data availability

Repository containing code used for running experiments and producing figures
in this study can be found at https://github.com/SLAMPAI/large-scale-
pretraining-transfer. All datasets used in the study are openly available and
are listed together with references to the original work in the Table 1.
Further details on the usage of the datasets in the conducted experiments are
also provided in the linked repository.