11institutetext:  Ultromics Ltd., Oxford, United Kingdom
11email<EMAIL_ADDRESS>22institutetext: Queen Mary University
of London, United Kingdom
22email<EMAIL_ADDRESS>
# Multi-Site Class-Incremental Learning with Weighted Experts in
Echocardiography

Kit M. Bransby 1122 Woo-Jin Cho Kim 11 Jorge Oliveira 11 Alex Thorley 11 Arian
Beqiri 11 Alberto Gomez 11 Agisilaos Chartsias 11

###### Abstract

Building an echocardiography view classifier that maintains performance in
real-life cases requires diverse multi-site data, and frequent updates with
newly available data to mitigate model drift. Simply fine-tuning on new
datasets results in “catastrophic forgetting”, and cannot adapt to variations
of view labels between sites. Alternatively, collecting all data on a single
server and re-training may not be feasible as data sharing agreements may
restrict image transfer, or datasets may only become available at different
times. Furthermore, time and cost associated with re-training grows with every
new dataset. We propose a class-incremental learning method which learns an
expert network for each dataset, and combines all expert networks with a score
fusion model. The influence of “unqualified experts” is minimised by weighting
each contribution with a learnt in-distribution score. These weights promote
transparency as the contribution of each expert is known during inference.
Instead of using the original images, we use learned features from each
dataset, which are easier to share and raise fewer licensing and privacy
concerns. We validate our work on six datasets from multiple sites,
demonstrating significant reductions in training time while improving view
classification performance.

###### Keywords:

class-incremental learning multi-site learning echocardiography.

##  1 Introduction

Echocardiography (echo) view classification is often a necessary first step in
automated image interpretation and analysis, as different tasks may require
different views as input [19]. Several deep learning methods have been
proposed to this end [19, 22, 10, 14] demonstrating excellent performance. To
generalise and maintain performance in real-life cases, view classifiers need
to be trained on a diverse multi-site dataset to accommodate varying
acquisition, demographic, and clinical parameters. However, these parameters
may change over time, e.g. with machine upgrades, modification of protocols,
or changes in demographics. Such changes require updating the classifier with
newly available data to mitigate model drift [17]. Naive approaches like fine-
tuning on a new dataset may result in “catastrophic forgetting” [15], in which
previous knowledge is forgotten at the expense of the new information.
Retaining previous knowledge while learning on new data is the goal of
incremental-learning [20].
Furthermore, newly available data (e.g. from sites with different protocols)
may have different sets of view labels. Labels can be characterised as “base”,
“novel”, or “overlapping”, depending on whether they are present in the
original data, the new data, or both. The goal of class-incremental learning
(CIL) [20] is to learn to classify the increasing label set over time. A
straightforward solution is to retrain using all data combined. However, this
is not always feasible as data sharing agreements may limit access or transfer
outside the acquisition site; or different datasets may be available at
different times. In addition, retraining a classifier when a new dataset
becomes available is inefficient as the total training time and cost grows
considerably with the number and size of datasets.
A common approach for CIL is to use a single model, and update parts or all of
the weights using knowledge distillation [16, 12, 4] or weight regularisation
[9, 1]. Others learn additional parameters with dynamic architecture changes
[18, 24, 21], or by duplicating the model for novel data and pruning [25]. Wu
et al. [23] expand on this by duplicating and fine-tuning a base model for
each new dataset, resulting in set of expert model branches, each specialised
on a single dataset. They combine the output logits of the independently
trained branches using a learnt score fusion (SF) network that enables
knowledge transfer between base and novel classes. This leverages shared
information between representations in different branches, but is sensitive to
one branch contributing ineffective information to another branch. We term
this the “unqualified expert” problem and may happen when there are
differences in data distributions, caused for instance by different label
sets, patient demographics or scanner manufacturers. As a result view
predictions for out-of-distribution (o.o.d.) images can be incorrect, but also
highly confident.

Figure 1: Examples of different views from WASE, CAMUS, Medstar, StG datasets.

In this paper we address the problem of building an echo view classifier model
with multiple datasets from different sites (see Fig. 1) that are not
simultaneously available and have different, overlapping sets of labels. To
this end, we build upon [23], and mitigate the “unqualified expert” problem by
weighing the contribution of each branch by a learnt in-distribution (i.d.)
score. This minimises the influence of logits which are o.o.d. for a given
input image, while maximising the i.d. logits. Our weighting improves
transparency as the contribution of each model is known during inference,
indicating the similarity between input data and different training sets. Our
method has the benefit of using learnt features as input rather than images,
which provides a simple workaround in cases where licensing prevents image
sharing but not sharing of byproduct features.
Contributions: We apply CIL to echo view classification and propose a new i.d.
score weighting that improves performance by minimising the influence of
“unqualified experts” and promotes transparency. We validate our method on six
datasets from multiple sites, demonstrating improved view classification
without needing to transfer images out of remote servers. By bypassing the
need for re-training with all data, the cumulative training time and cost is
reduced with further reductions achieved through more incremental steps.

##  2 Method

###  2.1 Problem Setting

Given a dataset $D=\\{(x_{i},y_{i})\\}_{i=1}^{N}$, where $x_{i}$ represents an
ultrasound image and $y_{i}$ denotes the corresponding ground truth view
label, the objective of view classification is to learn an image encoder
$\phi$ and linear classifier $W\in\mathbb{R}^{k\times|\mathcal{Y}|}$, where
$\mathcal{Y}$ is the set of labels in $D$ and $k$ is the size of the image
features. The view label is learnt by minimising the cross entropy loss
between $y$ and the label prediction $\hat{y}=\sigma(\phi(x)W)$ where
$\sigma(\cdot)$ is a softmax function.
We first train a base model $\mathcal{M}_{b}=\\{\phi_{b},W_{bb}\\}$ on a large
dataset $\mathcal{D}_{b}$ with label set $\mathcal{Y}_{b}$ of base classes. We
consider $t$ incremental steps $n\in\\{n1,n2,..,nt\\}$ each with a dataset
$\mathcal{D}_{n}$ and label set $\mathcal{Y}_{n}$. There can be varying
degrees of intersection between base classes $\mathcal{Y}_{b}$ and the new
label set $\mathcal{Y}_{n}$. For instance, there may be novel classes not
present in the base classes ($\mathcal{Y}_{n}\cap\mathcal{Y}_{b}=\emptyset$)
or overlapping classes ($\mathcal{Y}_{b}\cap\mathcal{Y}_{n}\neq\emptyset$).
Our goal is to accurately predict all classes, regardless of whether they are
introduced in an incremental step, are novel, base, or overlapping.
Simply combining base and incremental datasets into a single set and
retraining $\mathcal{M}_{b}$ from scratch is not ideal due to cost, time, and
restrictive data licensing as motivated in Section 1. Similarly, naively fine-
tuning $\mathcal{M}_{b}$ on successive incremental datasets
$\mathcal{D}_{n1},\mathcal{D}_{n2},..,\mathcal{D}_{nt}$ is difficult due to
the changing size of the label set, and ultimately results in catastrophic
forgetting of the base classes $\mathcal{Y}_{b}$.

Figure 2: Network architecture: predictions from expert branches are re-
weighted by an in-distribution score to minimise the influence of unqualified
experts.

###  2.2 Score Fusion

We follow Wu et al. [23] where $\mathcal{M}_{b}$ is cloned and fine-tuned for
each incremental dataset $\mathcal{D}_{n}$, yielding $t+1$ expert models
$\\{\mathcal{M}_{b},\mathcal{M}_{n1},..,\mathcal{M}_{nt}\\}$ 111We unfreeze
all layers instead of freezing the first two convolutional blocks as in [23],
which we find improves performance.. This approach mitigates the problem of
forgetting by retaining all weights; however, it introduces a new challenge of
selecting or combining experts during inference.
This is addressed with SF [23] that allows an expert to make a logit
prediction on its label set as well as the label sets of other experts. This
allows for knowledge sharing between branches and is useful for overlapping
classes or shared semantics. Specifically, the weights of an expert
$\\{\phi_{d},W_{dd}\\}$, where $d\in\\{b,n1,..,nt\\}$, are frozen and used to
compute a logit score $z_{dd}=h_{d}W_{dd}$ for its label set $\mathcal{Y}_{d}$
using feature vector $h_{d}\in\mathbb{R}^{k}$ and the weight $W_{dd}$. To
compute logit predictions on label sets of other experts
$\mathcal{Y}_{d^{\prime}}$, each expert learns an additional set of weights
$W_{dd^{\prime}}$ for all $d^{\prime}\in\\{b,n1,..,nt\\},d\neq d^{\prime}$. On
each branch, we compute the final logits $z_{d}$ by summing the logits
contributed by each expert as follows:

$z_{d}=\sum_{d^{\prime}=b,n1,..,nt}h_{d}W_{dd^{\prime}}$ (1)

The logits from each branch are concatenated to give $z_{a}$. As overlapping
classes may be present between incremental steps, a logit for the same class
may appear several times in $z_{a}$. We follow [23] with a “knowledge pooler”
to obtain final logits $\tilde{z}_{a}$ by maxpooling the overlapping classes
(See Fig 2). This enables information sharing between experts, however some
experts may have significant differences in knowledge and contribute unhelpful
information that deteriorates performance. In the next section, we address the
“unqualified expert” problem.

###  2.3 Weighted Score Fusion

Given an input image, we compute an i.d. score to weigh each expert’s logits
such that the influence of unqualified o.o.d experts is minimised and the
influence of i.d. experts is maximised. We do so by learning an attention-like
weighting vector $A\in\mathbb{R}^{\left|d\right|}$ using a feed-forward
network $\phi_{A}$ consisting of 3 linear layers, the first two with ReLU
activation and the last with a softmax.
We propose two strategies to learn $A$: (1) attention-weighted score fusion
(attn-wSF) where the concatenated feature vectors of all experts
$h\in\mathbb{R}^{k\left|d\right|}$ are used as input to $\phi_{A}$; and (2)
neural mean discrepancy weighted score fusion (nmd-wSF), which uses the neural
mean discrepancy (nmd) [3] as input. Nmd is an out-of-distribution metric
which measures the discrepancy between neural means of an input image and the
average neural mean of the i.d. training set. The neural mean is calculated by
averaging output activations of the convolutional layers. Like with attn-wSF,
we compute a nmd vector $g_{d}$ for each expert and concatenate them to
$g\in\mathbb{R}^{p\left|d\right|}$ where $p$ is the size of the nmd vector.
Similar to Equation 1, we compute the final logits for each branch as follows:

$z_{d}=\sum_{d^{\prime}=b,n1,..,nt}h_{d}W_{dd^{\prime}}A_{d}$ (2)

###  2.4 Multi-Site Training

Assuming that the incremental datasets are acquired at different medical
sites, which often impose data sharing restrictions, we propose a multi-site
training paradigm, which does not require transferring the image data. For
every step, the view classifier is cloned, transferred into the remote server,
and fine-tuned on dataset $\mathcal{D}_{n}$. We then make five forward passes
of the fine-tuned classifier $\mathcal{M}_{n}$ on randomly augmented data from
$\mathcal{D}_{n}$ to generate five different $g$ and $h$ vectors for each
example. We then transfer these vectors back to the local server, which are
used as input to train the weighted score fusion network.

Table 1: Frequency of view labels in training, validation and test sets.

View Contrast Train Validation Internal Test External Test Wase Camus Mstr StG
Wase Camus Mstr StG Wase Camus Mstr StG Mahi UoC A2C ✓ 0 0 0 1182 0 0 0 132 0
0 0 116 845 36 A3C ✓ 0 0 0 1208 0 0 0 137 0 0 0 119 429 23 A4C ✓ 0 0 0 1218 0
0 0 138 0 0 0 119 726 58 PLAX ✓ 0 0 0 923 0 0 0 125 0 0 0 86 6 4 A2C ✗ 4361
407 886 0 559 79 87 0 559 48 136 0 716 944 A3C ✗ 4012 0 900 0 501 0 87 0 492 0
108 0 679 493 A4C ✗ 6474 407 1668 0 811 42 166 0 801 48 202 0 825 1806 A5C ✗
1127 0 355 0 127 0 40 0 138 0 44 0 147 145 PLAX ✗ 4661 0 1176 0 614 0 125 0
566 0 138 0 929 1072 PLAX-AV ✗ 1480 0 223 0 190 0 24 0 185 0 23 0 306 113
PSAX-AV ✗ 3936 0 448 0 488 0 44 0 515 0 60 0 597 612 PSAX-PM ✗ 3351 0 755 0
420 0 93 0 435 0 80 0 310 543 RV ✗ 0 0 462 0 0 0 72 0 0 0 60 0 60 318 SC ✗ 0 0
589 0 0 0 72 0 0 0 62 0 482 328 SC-IVC ✗ 0 0 398 0 0 0 47 0 0 0 49 0 242 275
Total 29402 814 7860 4531 3710 84 849 532 3691 96 962 440 7299 6700

##  3 Experiments & Results

###  3.1 Datasets

As base dataset $\mathcal{D}_{b}$, we use WASE-Normals (WASE) [2], a large
multi-site proprietary dataset of 2,009 healthy volunteers acquired at 18
sites from 15 countries. We also use three incremental datasets: (1) CAMUS
[11], a public dataset acquired from 500 patients at University Hospital of St
Etienne, half of which are considered at pathological risk; (2) Medstar WASE-
Covid (Medstar) [8], a multi-site proprietary database of 870 patients with
COVID-19 at 13 sites from 9 countries; 3) St George’s (StG), a proprietary
stress echo dataset of 420 patients some with coronary artery disease. We
validate our method on an internal test set sampled from the four datasets and
an independent external test set from two proprietary datasets: (1)
Mazankowski Alberta Heart Institute (MAHI) consisting of 250 patients
undergoing chemotherapy with cardiotoxic drugs; (2) University of Chicago
(UoC) consisting of 391 patients with cardiomyopathy.
We extract images from 15 echo view labels and artificially decrease the
overlap between datasets by removing non-contrast examples from StG and
contrast examples from Medstar. As shown in Table 1, the resulting datasets
have different label sets with varying degrees of overlap. Each dataset is
split into train (80%), validation (10%) and test (10%) sets at a patient
level. A single random frame (112$\times$112) was sampled from each echo
video. The final dataset has 42,622 train, 5,175 validation, 5,189 internal
test, and 13,999 external test frames.

###  3.2 Implementation & Training

View classifiers use ResNet18 [5] architecture and were trained for 200 epochs
on a NVIDIA GeForce RTX 2080Ti with cross-entropy loss, Adam optimiser, batch
size of 64, and learning rate of 1e-3. The score fusion networks were trained
for 50 epochs with the same settings. Weights from the epoch with the highest
validation accuracy were saved. Hyperparameters were tuned on a held-out
validation set. We evaluate classification performance using accuracy and
F1-score metrics. Our method is implemented in PyTorch, and the code is
available here: https://github.com/kitbransby/class-incremental-learning-echo

Table 2: Quantitative Results for Internal Test Set (WASE, CAMUS, Medstar, StG
datasets). Best results in bold, and second best underlined

Experiment #Experts Data Transfer WASE CAMUS Medstar StG Average Acc F1 Acc F1
Acc F1 Acc F1 Acc F1 Fine-Tuning (Constant) 1 None 0.0 0.0 0.0 0.0 0.0 0.0
83.9 83.7 7.0 3.9 Fine-Tuning (Expand) 1 None 1.9 3.1 0.0 0.0 3.9 4.3 85.5
87.2 9.3 4.8 Single Expert (WASE) 1 None 94.5 94.5 76.0 80.9 65.1 58.7 0.0 0.0
68.6 60.2 Single Expert (CAMUS) 1 None 24.4 15.8 95.8 95.8 25.3 16.1 0.0 0.0
22.6 13.8 Single Expert (Medstar) 1 None 80.6 83.0 69.8 76.2 80.0 79.5 0.0 0.0
74.2 72.0 Single Expert (StG) 1 None 0.0 0.0 0.0 0.0 0.0 0.0 94.1 94.1 6.6 3.4
Max Logit [6] 4 None 62.5 71.3 84.4 89.4 63.1 68.4 71.4 81.0 63.8 69.3 MSP [7]
4 None 56.1 61.9 92.7 94.2 45.0 49.2 59.8 72.0 55.0 58.6 Confidence Routing
[23] 4 None 63.4 68.6 92.7 94.2 46.7 50.8 51.8 66.0 59.9 63.4 SF [23] 4
Features 93.9 94.0 89.6 90.4 80.3 80.0 92.7 93.6 91.2 91.1 attn-wSF (ours) 4
Features 94.1 94.3 95.8 95.8 81.4 80.9 93.9 94.0 91.8 91.7 nmd-wSF (ours) 4
Features 94.5 94.4 94.8 95.3 80.8 80.5 94.1 94.1 91.9 91.9 Combine & Retrain
(oracle) 1 Images 94.1 94.2 95.8 95.8 82.1 82.1 90.2 90.5 91.6 91.5

Table 3: Quantitative Results for External Test Set (MAHI, UoC datasets). Best
results in bold, and second best underlined

Experiment # Experts Data Transfer MAHI UoC Average Acc F1 Acc F1 Acc F1 Fine-
Tuning (Constant) 1 None 12.8 10.3 0.7 0.1 7.0 3.9 Fine-Tuning (Expand) 1 None
17.1 13.3 3.8 4.9 10.7 8.2 Single Expert (WASE) 1 None 55.9 45.5 72.9 67.8
64.1 55.1 Single Expert (CAMUS) 1 None 13.1 6.6 32.0 20.4 22.2 13.4 Single
Expert (Medstar) 1 None 56.9 51.4 79.0 79.1 69.0 64.1 Single Expert (StG) 1
None 11.2 8.0 0.1 0.0 6.1 3.6 Max Logit [6] 4 None 43.2 44.0 61.2 68.6 51.8
54.7 MSP [7] 4 None 34.4 35.4 55.4 61.6 44.5 46.9 Confidence Routing [23] 4
None 38.2 39.4 57.1 63.2 47.3 49.7 SF [23] 4 Features 72.0 68.3 81.6 80.8 76.6
74.2 attn-wSF (ours) 4 Features 72.5 69.6 82.6 82.1 77.4 75.6 nmd-wSF (ours) 4
Features 71.6 69.5 79.8 79.5 75.5 74.3 Combine & Retrain (oracle) 1 Images
71.4 69.0 82.7 82.7 76.8 75.1

###  3.3 Comparison to Existing Methods & Ablation Study

We validate our method on an internal test set (WASE, CAMUS, Medstar, StG) to
measure performance on incremental datasets seen during training, and an
external set (MAHI, UoC) of data not seen during training to measure
generalisability. Results are presented in Tables 2 and 3, respectively.
We demonstrate that naively fine-tuning a classifier on successive datasets
results in “catastrophic forgetting”. We test two configurations, one where
the classifier head is replaced at every step, and a second that expands to
accommodate novel classes [13]. These do not require data transfers and
perform well on the final dataset (StG) but forget learning from previous
data. We also explore other methods that do not require data transfer such as
using a single expert and find they perform well on i.d. data but do not
generalise well on o.o.d data.

Figure 3: Distribution of attention scores across a selection of test sets
using attn-wSF (top row) and nmd-wSF (bottom row). Note when using nmd, the
attention scores are pushed towards 0 and 1 which may reduce generalisability

Our primary comparison is to the training oracle where the model is retrained
with all data combined at each incremental step. This method is preferred
given unlimited compute, time constraints and data access, which are not
typically available. On average our method outperforms the oracle on both the
internal and external sets, including four out of six individual datasets.
In addition, we ablate our weighting method, demonstrating that attn-wSF and
nmd-wSF lead to an improvement in performance when compared to SF [23]. We
find that nmd-wSF performs best on the internal test set, however, attn-wSF
generalises better to the external test set. We theorise that nmd vectors for
each dataset are quite distinct, therefore the i.d. scores are pushed towards
0 and 1 during training (see Fig. 3). When presented with datasets from the
external test set the attention mechanism deteriorates as it considers the
input image o.o.d. for all experts.
Finally, we compare different confidence-based methods that select the best
prediction from multiple experts, these include: (1) Max Logit [6] that
selects the max logit for each class; (2) MSP [7], that selects the max
softmax for each class; (3) Confidence Routing [23] that selects the expert
with the highest softmax probability, and uses its softmax prediction as the
final prediction. All such approaches perform poorly, perhaps as classifiers
are optimised to be overconfident leading to predictions which are
uncorrelated with confidence measures.

Figure 4: Efficiency analysis: Cumulative training time (a) and inference time
(b).

Fig. 4 shows that our wSF methods reduce the cumulative training time, and
this grows significantly with the number of datasets. Our attention-based
weighting module has minimal computational overhead. While inference time is
larger when using multiple experts, this can be reduced with parallelisation.

##  4 Conclusion

Echo view classifiers require training on diverse multi-site data, and
frequent updates with newly acquired data to avoid model drift. We address the
problem of updating such a model using multiple datasets with partly
overlapping label sets that are from different sites and may not be
simultaneously available. We propose attn-wSF and nmd-wSF, two CIL methods
that train and combine predictions from multiple experts models with a learnt
i.d. weighting which minimises the influence of o.o.d predictions from
“unqualified experts”. Our results demonstrate improved view classification
with significantly reduced cumulative training time and without needing to
transfer images out of remote servers. We aim to extend our wSF methods by
training multiple experts within a single branch, which would reduce the
memory requirements and inference time.

## References

  * [1] Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: Learning what (not) to forget. In: Proceedings of the European conference on computer vision (ECCV). pp. 139–154 (2018)
  * [2] Asch, F.M., Banchs, J., Price, R., Rigolin, V., Thomas, J.D., Weissman, N.J., Lang, R.M.: Need for a global definition of normative echo values—rationale and design of the world alliance of societies of echocardiography normal values study (wase). Journal of the American Society of Echocardiography 32(1), 157–162 (2019)
  * [3] Dong, X., Guo, J., Li, A., Ting, W.T., Liu, C., Kung, H.: Neural mean discrepancy for efficient out-of-distribution detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19217–19227 (2022)
  * [4] Douillard, A., Cord, M., Ollion, C., Robert, T., Valle, E.: Podnet: Pooled outputs distillation for small-tasks incremental learning. In: Computer vision–ECCV 2020: 16th European conference, Glasgow, UK, August 23–28, 2020, proceedings, part XX 16. pp. 86–102. Springer (2020)
  * [5] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [6] Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., Song, D.: Scaling out-of-distribution detection for real-world settings. In: International Conference on Machine Learning. pp. 8759–8773. PMLR (2022)
  * [7] Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: International Conference on Learning Representations (2016)
  * [8] Karagodin, I., Singulane, C.C., Woodward, G.M., Xie, M., Tucay, E.S., Rodrigues, A.C.T., Vasquez-Ortiz, Z.Y., Alizadehasl, A., Monaghan, M.J., Salazar, B.A.O., et al.: Echocardiographic correlates of in-hospital death in patients with acute covid-19 infection: the world alliance societies of echocardiography (wase-covid) study. Journal of the American Society of Echocardiography 34(8), 819–830 (2021)
  * [9] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114(13), 3521–3526 (2017)
  * [10] Kusunose, K., Haga, A., Inoue, M., Fukuda, D., Yamada, H., Sata, M.: Clinically feasible and accurate view classification of echocardiographic images using deep learning. Biomolecules 10(5), 665 (2020)
  * [11] Leclerc, S., Smistad, E., Pedrosa, J., Østvik, A., Cervenansky, F., Espinosa, F., Espeland, T., Berg, E.A.R., Jodoin, P.M., Grenier, T., et al.: Deep learning for segmentation using an open large-scale dataset in 2d echocardiography. IEEE transactions on medical imaging 38(9), 2198–2210 (2019)
  * [12] Li, Z., Hoiem, D.: Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence 40(12), 2935–2947 (2017)
  * [13] Lomonaco, V., Maltoni, D.: Core50: a new dataset and benchmark for continuous object recognition. In: Conference on robot learning. pp. 17–26. PMLR (2017)
  * [14] Madani, A., Arnaout, R., Mofrad, M., Arnaout, R.: Fast and accurate view classification of echocardiograms using deep learning. NPJ digital medicine 1(1), 6 (2018)
  * [15] McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of learning and motivation, vol. 24, pp. 109–165. Elsevier (1989)
  * [16] Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: Incremental classifier and representation learning. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 2001–2010 (2017)
  * [17] Sahiner, B., Chen, W., Samala, R.K., Petrick, N.: Data drift in medical machine learning: implications and potential remedies. The British Journal of Radiology 96(1150), 20220878 (2023)
  * [18] Tao, X., Chang, X., Hong, X., Wei, X., Gong, Y.: Topology-preserving class-incremental learning. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIX 16. pp. 254–270. Springer (2020)
  * [19] Vaseli, H., Liao, Z., Abdi, A.H., Girgis, H., Behnami, D., Luong, C., Dezaki, F.T., Dhungel, N., Rohling, R., Gin, K., et al.: Designing lightweight deep learning models for echocardiography view classification. In: Medical Imaging 2019: Image-Guided Procedures, Robotic Interventions, and Modeling. vol. 10951, pp. 93–99. SPIE (2019)
  * [20] Van de Ven, G.M., Tuytelaars, T., Tolias, A.S.: Three types of incremental learning. Nature Machine Intelligence 4(12), 1185–1197 (2022)
  * [21] Wang, F.Y., Zhou, D.W., Ye, H.J., Zhan, D.C.: Foster: Feature boosting and compression for class-incremental learning. In: European conference on computer vision. pp. 398–414. Springer (2022)
  * [22] Wegner, F.K., Benesch Vidal, M.L., Niehues, P., Willy, K., Radke, R.M., Garthe, P.D., Eckardt, L., Baumgartner, H., Diller, G.P., Orwat, S.: Accuracy of deep learning echocardiographic view classification in patients with congenital or structural heart disease: importance of specific datasets. Journal of Clinical Medicine 11(3), 690 (2022)
  * [23] Wu, T.Y., Swaminathan, G., Li, Z., Ravichandran, A., Vasconcelos, N., Bhotika, R., Soatto, S.: Class-incremental learning with strong pre-trained models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9601–9610 (2022)
  * [24] Yan, S., Xie, J., He, X.: Der: Dynamically expandable representation for class incremental learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3014–3023 (2021)
  * [25] Yoon, J., Yang, E., Lee, J., Hwang, S.J.: Lifelong learning with dynamically expandable networks. In: International Conference on Learning Representations (ICLR) (2017)


#  and IllustrisTNG: Massive Extended Galaxies at $z=2$ Quench Later Than
Normal-size Galaxies

Anshu Gupta School of Physics, University of New South Wales, Sydney, NSW
2052, Australia  ARC Centre of Excellence for All Sky Astrophysics in 3
Dimensions (ASTRO 3D), Australia  Kim-Vy Tran School of Physics, University of
New South Wales, Sydney, NSW 2052, Australia  ARC Centre of Excellence for All
Sky Astrophysics in 3 Dimensions (ASTRO 3D), Australia  Annalisa Pillepich
Max-Planck-Institut für Astronomie, Königstuhl 17, 69117 Heidelberg, Germany
Tiantian Yuan Swinburne University of Technology, Hawthorn, VIC 3122,
Australia  ARC Centre of Excellence for All Sky Astrophysics in 3 Dimensions
(ASTRO 3D), Australia  Anishya Harshan School of Physics, University of New
South Wales, Sydney, NSW 2052, Australia  ARC Centre of Excellence for All Sky
Astrophysics in 3 Dimensions (ASTRO 3D), Australia  Vicente Rodriguez-Gomez
Instituto de Radioastronomía y Astrofísica, Universidad Nacional Autónoma de
México, A.P. 72-3, 58089 Morelia, Mexico  Shy Genel Center for Computational
Astrophysics, Flatiron Institute, 162 Fifth Avenue, New York, NY 10010, USA
Columbia Astrophysics Laboratory, Columbia University, 550 West 120th Street,
New York, NY 10027, USA

###### Abstract

Using the TNG100 (100 Mpc)3 simulation of the IllustrisTNG project, we
demonstrate a strong connection between the onset of star formation quenching
and the stellar size of galaxies. We do so by tracking the evolutionary
history of extended and normal-size galaxies selected at $z=2$ with
$\log(M_{*}/{\rm M}_{\odot})$$=10.2-11$ and stellar-half-mass-radii above and
within 1-sigma of the stellar size–stellar mass relation, respectively. We
match the stellar mass and star formation rate distributions of the two
populations. By $z=1$, only 36% of the extended massive galaxies have
quenched, in contrast to a quenched fraction of 69% for the normal-size
massive galaxies. We find that normal-size massive galaxies build up their
central stellar mass without a significant increase in their stellar size
between $z=2-4$, whereas the stellar size of the extended massive galaxies
almost doubles in the same time. In IllustrisTNG, lower black hole masses and
weaker kinetic-mode feedback appears to be responsible for the delayed
quenching of star formation in the extended massive galaxies. We show that
relatively gas-poor mergers may be responsible for the lower central stellar
density and weaker supermassive black hole feedback in the extended massive
galaxies.

galaxies: formation – evolution – methods: numerical – hydrodynamics

##  1 Introduction

How and when massive galaxies stop forming stars (aka quenching) remains a
mystery after decades of study. A significant fraction of massive galaxies
have either stopped forming stars or show suppressed star formation in the
first 15-20% of the age of the Universe ($z=2-3$) (van Dokkum et al., 2008;
Damjanov et al., 2009; Straatman et al., 2014; Glazebrook et al., 2017). The
emerging picture is that star formation is quenched in field galaxies either
via feedback processes such as those related to supernovae and active galactic
nuclei (AGN), or a truncation of gas supply to fuel the star formation
(Springel et al., 2005; Zhang et al., 2019). In massive dark matter halos, the
infalling gas can be shock-heated, further preventing the inflow of gas at
virial radii (Dekel & Birnboim, 2006). At higher redshifts, mergers and
violent disk instabilities can also trigger a burst of star formation and AGN
activity in the core, subsequently quenching star formation (Dekel & Burkert,
2014; Zolotov et al., 2014). AGNs are thought to be primarily responsible for
the quenching of star formation in massive galaxies (Fabian, 2012; Harrison,
2017).

The physical processes responsible for quenching leave imprints on the spatial
distribution of stars and stellar ages (McDermid et al., 2015; Wu et al.,
2018). A central starburst will rapidly quench star formation in the center,
either by rapid consumption of gas or by triggering strong outflows, and
produce galaxies with a dense stellar core (Springel et al., 2005; Snyder et
al., 2011; Zolotov et al., 2014). Less centrally focused feedback processes
such as shock heating of gas in massive dark matter halos, energy injected by
the supernovae and/or AGN would quench star formation gradually without
significantly altering the stellar distribution of galaxies (Dekel & Birnboim,
2006; Fabian, 2012; Terrazas et al., 2017). Therefore, we expect to see a
correlation between the star formation histories and the spatial distribution
of stars.

Large extragalactic surveys in the local and high redshift Universe find that
star formation in compact galaxies quench on a shorter time-scale compared
with larger, more extended, galaxies. In the local Universe, at a fixed
stellar mass, compact galaxies have relatively older stellar populations
indicative of either rapid star formation quenching and/or an earlier
formation epoch (McDermid et al., 2015; Barone et al., 2020). Massive diffuse
galaxies in the local universe such as Malin I build up their stellar disk
slowly over an extended timescale (Boissier et al., 2016; Zhu et al., 2018).
Using data from the Sloan Digital Sky Survey, Fang et al. (2013) most galaxies
quench after they cross a certain threshold in the central stellar mass
density.

Studies at high redshift ($z\sim 1$) also find that massive compact galaxies
have older stellar populations compared with their larger counterparts,
suggesting that star formation quenched in compact galaxies earlier (Williams
et al., 2017; Fagioli et al., 2016). In contrast, Keating et al. (2015) find
that compact massive galaxies at $z\sim 1.5$ have younger stellar ages
compared with a control population, suggesting that compactness is the result
of a recent burst of star formation. Observations by Barro et al. (2017) show
that the connection between the central stellar mass surface density and the
status of the star formation exists as early as at $z~{}2.5-3$. Using
spectroscopic data from the Lega-C survey (van der Wel et al., 2016; Straatman
et al., 2018), Wu et al. (2018) shows that massive galaxies at $z\sim 1$ seem
to follow two distinct star formation quenching pathways: (i) slow ($>1$ Gyr)
quenching producing quiescent galaxies with large galactic disks and younger
stellar populations, and (ii) fast ($<1$ Gyr) star-formation quenching
producing post-starburst galaxies with various sizes depending on their
formation history.

Figure 1: The stellar size–stellar mass relation for TNG100 galaxies at $z=2$.
The golden circles represent average stellar sizes binned into equal stellar
mass bins. The purple squares are the extended massive galaxies
($\log(M_{*}/{\rm M}_{\odot})$$\,=10.2-11$) with stellar size 1-$\sigma$ above
the mean size–mass relation. The red circles represent our control sample of
normal-size (aka normal) massive galaxies. The two panels on the right show
the stellar mass and SFR distributions of the normal and extended massive
galaxies at $z=2$. The dashed purple and dotted red line represent the median
for the respective population and are statistically identical at $z=2$. We
label the $p$-value from a two sample KS test in the respective panel.

In cosmological simulations, massive compact galaxies at $z=2$ exhibit two
main formation pathways: (i) a recent burst of star formation between $z=2-4$
triggered by gas-rich major merger or (ii) they formed at $z>4$ when the
Universe was denser than today (Wellons et al., 2015). In agreement with
observations, compact galaxies formed via mergers have relatively younger
stellar populations, whereas compact galaxies formed at earlier epochs have
older stellar populations. However, Wellons et al. (2015) focus on only a
small fraction of extremely compact galaxies from the Illustris simulation
with a stellar size almost 2-sigma below the galaxy size-mass relation and
$\log(M_{*}/{\rm M}_{\odot})$ $>11.0$.

Genel et al. (2018) analysed the size evolution of galaxies in the
IllustrisTNG simulations (https://www.illustris-project.org). They find that
most quenched galaxies at $z=0$ started off at the lower end of the stellar
mass-size relations, i.e., quenched galaxies have relatively compact stellar
distribution at higher redshift. In Gupta et al. (2020), we use the survey
(Tran et al., 2020) to track how galaxies at $z\sim 3$ grow. Combining our
observations with predictions from IllustrisTNG, we show that massive galaxies
after $z\sim 3$ tend to form new stars from ex-situ processes. An intriguing
result from Gupta et al. (2020) is a dramatic increase in the stellar size of
massive galaxies at $z=2$.

In this follow-up study, we investigate the size evolution of the massive
galaxies and its link with the star formation quenching by analysing the
outcome of the publicly available IllustrisTNG simulations (Nelson et al.,
2019b). We select normal-size and extended massive simulated galaxies at $z=2$
according to their location on the stellar mass – stellar size plane of
IllustrisTNG and track their evolutionary histories backwards and forwards in
time. To minimize the bias due to the different mass–size relations of
quiescent and star-forming galaxies, also in place in the simulations (Genel
et al., 2018), we select two samples of galaxies with consistent stellar mass
and star formation rate distributions at the time of selection, here $z=2$.

Figure 2:  Distribution of stars (red), hot gas (green) and neutral gas (blue)
in 5 randomly selected TNG100 simulated galaxies in increasing order of
stellar mass. From top to bottom: (i) top two rows: extended massive galaxies
at $z=2$ (first) and their descendants at $z=0$ (second row); (ii) counterpart
normal-size (aka normal) massive galaxies at $z=2$ (third row) and their
descendants at $z=0$ (bottom row). The dashed circle in each panel corresponds
to the stellar half mass radii.

The paper is organised as follows. In Section 2, we provide a brief
description of the simulations and our selection method of the extended and
normal-size massive galaxies. In Section 3 we compare the star-formation
histories of the two samples and how the two populations evolve on the size-
mass plane. In Section 4 and 5, we discuss and summarise our main findings.

##  2 Methods

###  2.1 The TNG100 simulation of IllustrisTNG

We use a cosmological hydrodynamical simulation from the IllustrisTNG project
(Pillepich et al., 2018a; Nelson et al., 2018a; Springel et al., 2018;
Marinacci et al., 2018; Naiman et al., 2018) to understand the relation
between stellar disk size and star formation history of massive galaxies. In
particular, we use the publicly available outcome (Nelson et al., 2019b) of
the TNG100 simulation ($\sim$[100 Mpc]3, $m_{b}=9.4\times 10^{5}$/h) to ensure
a statistically significant sample of galaxies with $\log(M_{*}/{\rm
M}_{\odot})$$>9$ at $z\gtrsim 3$, i.e., galaxies with $\sim$1000 baryonic
resolution elements. We start from a selection of galaxies with
$\log(M_{*}/{\rm M}_{\odot})$$>9$ at $z=0$ and track them back in time using
the merger tree catalogs (Rodriguez-Gomez et al., 2015).

Figure 3: The redshift evolution of stellar mass (left), star formation rate
(middle), and specific star formation rate (right) of the extended (purple)
and normal-size (red) massive galaxies from the TNG100 simulation. The dashed
curves and shaded region show the median and 16${}^{\rm th}-$84th percentiles.
The star formation in the extended massive galaxies quench later than the
normal-size massive galaxies. The dashed black curve corresponds to the
redshift dependent sSFR cut-off for the quenched galaxies (Tacchella et al.,
2019), which is adopted throughout to separate quenched versus star-forming
galaxies.

###  2.2 Selection of extended vs. normal-size massive galaxies

To select extended massive galaxies in TNG100, we bin the stellar mass–size
relation at $z=2$ into bins of $0.15\,$dex in stellar mass. Throughout this
paper, we use the three-dimensional stellar half-mass-radius ($R_{\rm
halfmass}$ = radius containing 50% of the stellar particles) to represent the
stellar size, or stellar extent, of a galaxy. All properties of galaxies are
measured within twice the stellar half-mass-radius, unless otherwise
specified. We select massive galaxies with stellar mass
$10.2<$$\log(M_{*}/{\rm M}_{\odot})$$<11$ at $z=2$ with stellar sizes
$1\sigma$ above the stellar mass-size relation (Figure 1): this leads to the
selection of 123 extended massive galaxies. We limit our extended massive
galaxies sample to $\log(M_{*}/{\rm M}_{\odot})$$<11.0$ due to the low number
of galaxies at higher stellar masses.

From TNG100, we build a control sample of normal-size simulated galaxies
(normal galaxies hereafter). For each extended massive galaxy, we select a
normal-size massive galaxy whose $R_{\rm halfmass}$ is within $1\sigma$ of the
median stellar mass–stellar size relation, and whose stellar mass and star
formation rate (SFR) match the corresponding values of the extended massive
galaxy within $0.1\,$dex at $z=2$. This ensures statistically identical
stellar mass ($p=1.0$) and SFR ($p=0.7$) distributions for the normal-size and
extended massive galaxies at $z=2$ (Figure 1: right-hand panels). This results
in a minimal stellar mass and SFR bias between the two sample upon selection.
Under this selection at $z=2$, the TNG100 extended massive galaxies have on
average twice the stellar $R_{\rm halfmass}$ compared with the normal-size
massive galaxies.

Figure 2 shows the distribution of gas and stars in five randomly selected
extended and normal-size massive galaxies at redshift $z=2$ and their
descendants at $z=0$. It demonstrates the diversity of galaxy types in both
populations. In the following sections, we track the evolutionary histories of
extended and normal-size massive galaxies to understand the build up of the
extended stellar disk and its connection with the star formation quenching or
suppression. We use the merger tree catalogs by Rodriguez-Gomez et al. (2015)
to identify progenitors and descendents of our extended and normal-size
massive galaxies.

Figure 4: The specific star formation rates of the extended (purple) and
normal (red) massive galaxies selected in TNG100 at $z=2$ and here shown at
four redshifts: $z=0.3,1.0,1.5$, and $2.0$, from left to right. In each panel,
we label the quenched fraction ($qf$) for the respective population,
calculated based on the dashed black line in Figure 3. We list the $p$-value
from a two sample KS-test in each panel.  Figure 5: Median and 16${}^{\rm
th}-$84th percentiles of halo mass (left panel), gas fraction (middle), and
gas density (right panel) of the $z=2$-selected extended (purple) and normal
(red) massive TNG100 galaxies, as a function of redshift. Both normal and
extended massive galaxies occupy similar mass halos, but the extended massive
galaxies consistently have 2-10 times higher gas fraction compared to the
normal massive galaxies after $z<4$ measured within twice the $R_{\rm
halfmass}$. The normal massive galaxies have nearly five times higher gas
density than the extended massive galaxies between $z=2-4$.

##  3 Results

###  3.1 Star formation histories

Figure 3 shows the stellar mass buildup and star formation histories of the
extended and normal massive galaxies. By selection, both extended and normal
massive galaxies have similar sSFR at $z=2$. Even at $z>2.0$, the sSFR
distributions of the extended and normal massive galaxies remains the same
(KS-test $p>0.2$), suggesting both populations build up their stellar mass at
a similar rate till $z=2.0$. The stellar mass distribution of the two
populations remains statistically similar till $z=2$, although by $z=0$ the
extended massive galaxies have on average $0.15$ dex higher stellar mass than
the normal massive galaxies.

The star formation in the extended massive galaxies quenches later than in the
normal-size massive galaxies (Figures 3 & 4). By $z=1.5$ more than 41% of the
normal massive galaxies are quenched, whereas only 17% of the extended massive
galaxies are quenched. To calculate the quenched fraction, we use the redshift
dependent sSFR cut by Tacchella et al. (2019). Even till $z=1$, only 36%
percent of the extended massive galaxies are quenched in contrast to the 69%
quenched fraction in the normal massive galaxies. However, by $z\sim 0.3$ both
normal and extended massive galaxies have a similar quenched fraction. Our
measurements suggest that by using only stellar size, we can predict when star
formation will quench in massive galaxies.

Figure 6:  Median radial star formation (top), gas mass (middle) and stellar
mass (bottom) surface density profiles for the normal (red dotted) and
extended (purple dashed) massive galaxies at $z=2$. The $R_{\rm halfmass}$
radius for the respective population is given by the short vertical lines
along the bottom edge of the each panel. The grey shaded region corresponds to
the nominal resolution limit of TNG100 at $z=2$. The extended massive galaxies
have lower SFR and gas mass surface density compared to the normal-size
massive galaxies in their galactic centers.

###  3.2 Quenching of star formation

####  3.2.1 Environment and gas density

Both observations and simulations show that galaxies living in more massive
halos quench earlier (Peng et al., 2010; Gabor & Davé, 2015; Wang et al.,
2018). However, the average halo mass of the extended massive galaxies remains
within $1\sigma$ scatter to the average halo mass of the normal massive
galaxies throughout the cosmic time (Figure 5: left panel). We do find that
the median halo mass of the extended massive galaxies is $\sim 0.08$ dex
higher than the normal massive galaxies, directly opposite to our expectations
based on their delayed star formation quenching timescale.

Martizzi et al. (2020) shows that at a fixed halo mass, galaxies residing in
voids and sheets tend to have lower stellar mass compared with the median of
all galaxies, suggesting a clear role of the large-scale environment on the
mass assembly history of galaxies. However, our preliminary analysis does not
yield any significant evidence that the extended massive galaxies occupy a
special place in the cosmic web compared with the normal massive galaxies.

Focusing on internal galaxy properties, the middle and right panels in Figure
5 show how the fraction and the density of total gas (neutral and warm) within
twice the $R_{\rm halfmass}$ evolves with redshift. To get the average gas
density, we divide the total gas mass within twice the stellar $R_{\rm
halfmass}$ with the total enclosed volume. The extended massive galaxies have
systematically higher gas fraction and gas density compared with the normal
massive galaxies at $z<2$. It could be hypothesised that the higher gas
fraction and gas density were responsible for the delayed onset of SF
quenching in the extended massive galaxies compared with the normal massive
galaxies.

Interestingly, we find that the extended massive galaxies have lower gas
density between $z=2-4$, i.e., just before the selection. The lower gas
density in the extended massive galaxies could be responsible for their
relatively diffuse stellar core at $z=2$ and lower black hole feedback (See
section 3.2.2).

Figure 7: Distribution of the mass (left), accretion rate (middle) and ratio
of cumulative energy injected in the kinetic-mode versus the thermal-mode
(right) by the central supermassive black hole in the normal (red) and
extended (purple) massive galaxies at $z=1.5$ (top row), $2.0$ (middle row)
and $3.0$ (bottom row). The dashed lines in each panel represent the median
for the respective population. The normal massive galaxies have stronger black
hole feedback than the extended massive galaxies at $z=2$.

####  3.2.2 Central star formation and feedback from super massive black holes

To understand the spatial variation of the SFR, we estimate the radial star
formation, gas mass and stellar mass surface density profiles in the extended
and normal-size massive galaxies at the time of the selection (Figure 6). The
extended massive galaxies have lower star formation and gas mass surface
densities in their galactic center ($<4$ pkpc) in comparing with the normal
massive galaxies.

The star formation and gas mass density profiles remains almost flat between
$2-10$ pkpc for the extended massive galaxies. In contrast, the density
profiles of the normal massive galaxies falls off rapidly beyond 2 pkpc
($\sim$$R_{\rm halfmass}$). The extended galaxies have about 1 dex higher star
formation surface density at $r>4\,$pkpc. We hypothesise that the higher gas
mass and star formation surface density in the galactic outskirts is
responsible for the build-up of extended stellar disk in the extended massive
galaxies.

Observations in both the local and high redshift universe show that central
stellar mass surface density correlates strongly with the onset of star
formation quenching (Fang et al., 2013; Barro et al., 2017; Mosleh et al.,
2017). The stellar mass surface density of the extended massive galaxies has a
shallower slope compared to the normal massive galaxies. The extended galaxies
have nearly 0.3 dex lower stellar mass surface density in the central 1 kpc
than the normal massive galaxies. We suspect that the slow build-up of the
central stellar core in the extended massive galaxies might be linked to their
delayed star formation quenching timescale.

In massive galaxies, feedback from super massive black holes (SMBHs) is
typically invoked for the quenching of star formation (see the review by
Fabian, 2012). This is certainly the case in the IllustrisTNG model for galaxy
formation (Pillepich et al., 2018b; Weinberger et al., 2017). In the
IllustrisTNG model SMBH feedback operates in two different modes: the quasar-
mode and the kinetic-mode. The quasar-mode occurs when the mass accretion rate
is high, releasing enormous amount of thermal energy into the surrounding
medium. The kinetic-mode occurs when the mass accretion rate is low and
kinetic energy is deposited into the gas, producing jets and outflows or
winds. SF quenching in particular is triggered by black hole driven winds at
low-accretion rates (see e.g. Weinberger et al., 2017; Nelson et al., 2018b;
Terrazas et al., 2020) because it determines the physical and thermodynamical
properties of the gas within and around galaxies (see Truong et al., 2020;
Davies et al., 2019; Zinger et al., 2020).

Figure 7 shows that at $z=2$, the SMBHs in the extended massive galaxies are
half as massive and accrete two times slower than the SMBHs in the normal-size
massive galaxies. Already, at $z=3$, the normal-size massive galaxies host
slightly more massive black holes than the more extended analogs. On the other
hand, by $z=1.5$, the black hole accretion rate reverses and becomes almost 10
times higher in the extended massive galaxies probably due to the reduced
supply of gas in the normal-size massive galaxies (Figure 5).

Nelson et al. (2019a); Davies et al. (2020); Terrezas2019; Zinger et al.
(2020) show that in IllustrisTNG SMBHs quench star formation primarily by
pushing the gas out from the central regions of galaxies via the kinetic-mode
feedback. The last column in Figure 7 shows the ratio of the cumulative amount
of energy injected by the SMBHs in the kinetic-mode versus the quasar-mode. In
the extended massive galaxies, the ratio of energy injected in the kinetic-
mode versus the quasar-mode is 10 times lower than the normal-size massive
galaxies at $z=2$. In IllustrisTNG, kinetic-mode feedback becomes comparable
to quasar-mode feedback once the SMBHs cross the transitional BH mass of about
$10^{8.3}$M⊙ (Zinger et al., 2020). Most extended massive galaxies have SMBH
masses well below the transitional SMBH mass at $z=2$, and hence have
relatively smaller fractions of energy injected in kinetic-mode.

Our analysis suggests that, at least in the case of IllustrisTNG galaxies,
weaker AGN-driven kinetic feedback is responsible for the overall delayed
quenching of star formation in the extended massive galaxies compared with the
normal-size massive galaxies. We suspect that the lower gas density in the
galactic center (Figure 6) might be responsible for the slower black hole
growth in the extended massive galaxies till $z=2$.

Figure 8: The evolutionary tracks corresponding to all 123 normal-size (left)
and extended (right) massive TNG100 galaxies on the stellar size-mass plane.
Symbol colors indicate the redshift of the galaxies at that point. The colored
solid curve in the bottom panel indicates the median tracks. The median
stellar size of the normal-size massive galaxies does not change till $z\sim
2$, whereas extended galaxies experience steady increase in their stellar
size.

###  3.3 Build up of the extended stellar disk

The extended and normal-size galaxies follow a range of evolutionary pathways
on stellar mass-size plane (Figure 8). The median stellar size of the extended
massive galaxies increases steadily as they build up their stellar mass
between $z=2-4$. In contrast, the median stellar size-mass track of the
normal-size massive galaxies is almost flat at least till $z=2.5$, i.e., on
average normal-size massive galaxies build-up their stellar mass without a
significant increase in their stellar size. Between $z=2-4$, the median
stellar mass of the normal-size massive galaxies increases by 1 dex, while the
median $R_{\rm halfmass}$ radius remains roughly the same. Interestingly,
$R_{\rm halfmass}$ almost doubles within the same time period for the extended
massive galaxies again with roughly 1 dex increase in their stellar mass.

Using data from the survey in Gupta et al. (2020), we find that massive
galaxies at $z\sim 2$ have higher integrated velocity dispersion compared to
the counterpart galaxies at $z=3-4$. We conclude that build up of stellar mass
via ex-situ processes in massive galaxies after $z=2-3$ (Rodriguez-Gomez et
al., 2016; Gupta et al., 2020) increases the integrated velocity dispersion of
gas. In our current analysis, we find that the median stellar size of normal-
size massive galaxies only increases after $z<2$, again due to the stellar
mass build-up via ex-situ processes (mostly dry merger, Figure 9).

As in Gupta et al. (2020), we use catalogs by Rodriguez-Gomez et al. (2016) to
estimate the ex-situ stellar mass fraction, which is the fraction of stellar
mass built via external processes such as mergers and stripping from orbiting
satellites. The mean gas fraction of mergers in Figure 9 is taken from
catalogs by Rodriguez-Gomez et al. (2017) representing the average fraction of
cold gas in already merged galaxies weighted by their stellar mass. The median
ex-situ fraction of the extended massive galaxies is $0.06-0.09$ higher than
the normal-size massive galaxies, i.e., extended galaxies accrete a larger
fraction of their stars. The accreted stars tend to reside at a larger
galacto-centric distance (Rodriguez-Gomez et al., 2016) and might be
responsible for the higher stellar surface density of the extended massive
galaxies at large galacto-centric distance (Figure 6).

Moreover, we find that the mean gas fraction of mergers encountered by the
extended massive galaxies by $z=2$ is lower by 10% compared with the normal-
size massive galaxies. The (mildly) lower gas fraction of mergers at $z=2$
coupled with the higher ex-situ fraction suggests that the extended massive
galaxies have encountered relatively gas-poor mergers compared to the normal-
size massive galaxies between $z=2-4$. Gas-rich mergers are excellent at
funnelling the gas towards the galactic centers, driving central star
formation and the growth of central supermassive black holes (Hopkins &
Quataert, 2010; Medling et al., 2015; Wellons et al., 2015; Sparre & Springel,
2016; Hani et al., 2020). Our analysis suggests that relatively drier mergers
in the extended massive galaxies might be responsible for their lower central
stellar mass density and slower SMBH growth.

Figure 9:  The distribution of the mean cold fraction of the already merged
galaxies at $z=2$ (left) and the evolution of the ex-situ stellar mass
fraction (right) for the normal-size (red) and extended (purple) massive
galaxies from TNG100. The dashed and dotted lines in each panel represents the
median of the respective population.

##  4 Discussion

Observations of compact quiescent galaxies, ‘red-nuggets’, at $z\sim 2$ have
triggered a fruitful debate over the onset of star formation quenching and the
factors that may affect it in the early Universe (Damjanov et al., 2009;
Glazebrook et al., 2017; Lu et al., 2019). In Gupta et al. (2020), we found
evidence that massive galaxies after $z\sim 3$ buildup their stellar mass from
ex-situ processes in both observations and IllustrisTNG simulations. We also
observed a significant increase in the stellar sizes of the massive galaxies
at $z\sim 2$. In this paper, we have analysed the TNG100 simulation to gain
insights as to why the stellar sizes of some massive galaxies increase
dramatically around $z=2$ and how this affects their subsequent evolutionary
history.

We find that, in the TNG100 simulation, the onset of star formation quenching
in massive galaxies is connected to their stellar spatial distribution and
extent. We find that $z=2$ galaxies with extended stellar distribution quench
later than their relatively compact counterparts (Figure 3 & 4). To minimise
the bias at the time of selection, we have matched the stellar mass and SFR
distributions of the extended and the normal-size massive galaxies at $z=2$.
Our findings are similar to those by Wu et al. (2018), who show that observed
large quiescent galaxies at $z>1$ have younger stellar ages indicative of the
relatively recent star formation quenching history.

We hypothesise that relatively gas-poor mergers encountered by the extended
massive galaxies at $z>2$ are responsible for their lower central stellar mass
assembly by accretion and slower black hole growth (Figure 9). Gas-rich
mergers are excellent at funneling the gas towards the galactic centre
triggering central star formation and the growth of the central SMBH (Hopkins
& Quataert, 2010; Medling et al., 2015; Sparre & Springel, 2016; Hani et al.,
2020). Other physical mechanisms such as disk-instabilities introduced by the
gas inflows and/or gravitation can also funnel the gas towards the galactic
center triggering growth of central SMBH (Dekel & Burkert, 2014; Zolotov et
al., 2014). Relatively gas-poorer mergers between $z=2-4$ could explain the
smaller black hole masses and lower gas density in the extended massive
galaxies between $z=2-4$ (Figure 6 & 7).

SMBHs in the extended massive galaxies are almost half as massive and have
injected a lower fraction of their energy in the kinetic-mode rather than in
the quasar-mode compared with the normal-size massive galaxies at $z=2$. In
both the EAGLE and IllustrisTNG simulations, expulsion of dense gas has been
shown to trigger the star formation quenching (e.g. Davies et al., 2020).
Zinger et al. (2020) show that the AGN-driven kinetic-mode feedback in
IllustrisTNG suppresses star formation by increasing the gas cooling
timescales from 10-100 Myr to 1-10 Gyr. The lower fraction of kinetic energy
injected by SMBHs in the extended massive galaxies due to their lower black
hole masses would lead to weaker or less effective AGN-driven outflows, within
the IllustrisTNG model, compared with the normal-size massive galaxies. This
might be insufficient to trigger quenching of SF activity in the extended
massive galaxies at $z=2$.

Habouzit et al. (2019) also find that compact massive galaxies in illustrisTNG
simulations have higher star formation rates and stronger AGN feedback than
their extended counterparts. Observationally, compact star-forming galaxies at
$1<z<3$ have a higher fraction of AGNs than the counterpart normal-size star-
forming galaxies in line with our prediction from the IllustrisTNG simulations
(Barro et al., 2014; Rangel et al., 2014; Gu et al., 2020; Lu et al., 2020).
Our analysis suggests that weaker SMBH feedback is responsible for the delayed
onset of star-formation quenching in the extended massive galaxies. Future IFU
surveys such as MUSE MAGPI survey (MAGPI collaboration in prep.) would be
excellent to test our prediction that the onset of star formation quenching
depends on the size of the stellar disk.

##  5 Conclusions

Growth via dry-mergers has been shown to be responsible for the dramatic
increase in the scatter on stellar mass–stellar size plane at the massive end
in Gupta et al. (2020) at $z=2$. In this paper, we have used the TNG100
simulation of the IllustrisTNG project to gain insights on the connection
between star formation activity and galaxy stellar sizes. In particular, we
find that in the IllustrisTNG simulation $z=2$ massive galaxies with extended
stellar distributions quench later than the normal-size galaxies with similar
stellar masses and star formation rates at selection.

In particular, we have tracked the evolutionary histories of extended and
normal-size massive galaxies ($\log(M_{*}/{\rm M}_{\odot})$$>10.2-11$)
selected at $z=2$ with identical stellar mass and SFR distributions. By $z=1$,
only 36% of the extended massive galaxies have quenched, in contrast to more
than 69% quenched fraction in the normal-size massive galaxies (Figure 4).

The stellar size of the extended massive galaxies almost doubles between
$z=2-4$, with about 1 dex increase in average stellar mass (Figure 8). In
contrast, the median stellar size of the normal-size massive galaxies does not
change significantly between $z=2-4$, even if they experience similar increase
in their stellar mass. At $z=2$, the extended massive galaxies have almost 0.3
dex lower stellar mass surface density in the central $1\,$kpc compared with
the normal-size massive galaxies (Figure 6). Stellar mass build-up via ex-situ
processes leads to an increase in the sizes of both normal-size and extended
massive galaxies after $z<2$, as also shown in Gupta et al. (2020) using the
survey (Tran et al., 2020).

We hypothesise that relatively gas-poorer mergers experienced by the extended
massive galaxies at $z>2$ are responsible for their lower central stellar mass
density and SMBH mass. Our analysis shows that relatively lower SMBH mass and
weaker kinetic-mode feedback from SMBHs are responsible for the delayed
quenching of the star-formation activity in the extended massive galaxies
(Figure 7). However, details of large-scale environment and merger histories
need to be further investigated to understand why some massive galaxies
experience gas-poor mergers and do not develop a central stellar core.

## Acknowledgements

The authors thank the referee for their extremely valuable feedback and
suggestions that significantly enhanced the quality of the paper. Parts of
this research were supported by the Australian Research Council Centre of
Excellence for All Sky Astrophysics in 3 Dimensions (ASTRO 3D), through
project number CE170100013. SG is supported by the Simons Foundation through
the Flatiron Institute. TNG100 was run on the HazelHen Cray XC40-system at the
High Performance Computing Center Stuttgart as part of project GCS-ILLU of the
Gauss Centre for Supercomputing (GCS). Ancillary and test runs of the
IllustrisTNG project were also run on the Stampede supercomputer at TACC/XSEDE
(allocation AST140063), at the Hydra and Draco supercomputers at the Max
Planck Computing and Data Facility, and on the MIT/Harvard computing
facilities supported by FAS and MIT MKI.

## References

  * Barone et al. (2020) Barone, T. M., D’Eugenio, F., Colless, M., & Scott, N. 2020, The Astrophysical Journal, 898, 62
  * Barro et al. (2014) Barro, G., Trump, J. R., Koo, D. C., et al. 2014, The Astrophysical Journal, 795, 145
  * Barro et al. (2017) Barro, G., Faber, S. M., Koo, D. C., et al. 2017, The Astrophysical Journal, 840, 47
  * Boissier et al. (2016) Boissier, S., Boselli, A., Ferrarese, L., et al. 2016, Astronomy and Astrophysics, 593, 1
  * Damjanov et al. (2009) Damjanov, I., McCarthy, P. J., Abraham, R. G., et al. 2009, Astrophysical Journal, 695, 101
  * Davies et al. (2020) Davies, J. J., Crain, R. A., Oppenheimer, B. D., & Schaye, J. 2020, Monthly Notices of the Royal Astronomical Society, 491, 4462
  * Davies et al. (2019) Davies, L. J. M., Robotham, A. S. G., Lagos, C. d. P., et al. 2019, Monthly Notices of the Royal Astronomical Society, 16, 1
  * Dekel & Birnboim (2006) Dekel, A., & Birnboim, Y. 2006, Monthly Notices of the Royal Astronomical Society, 368, 2
  * Dekel & Burkert (2014) Dekel, A., & Burkert, A. 2014, Monthly Notices of the Royal Astronomical Society, 438, 1870
  * Fabian (2012) Fabian, A. 2012, Annual Review of Astronomy and Astrophysics, 50, 455
  * Fagioli et al. (2016) Fagioli, M., Carollo, C. M., Renzini, A., et al. 2016, The Astrophysical Journal, 831, 173
  * Fang et al. (2013) Fang, J. J., Faber, S. M., Koo, D. C., & Dekel, A. 2013, Astrophysical Journal, 776, doi:10.1088/0004-637X/776/1/63
  * Gabor & Davé (2015) Gabor, J. M., & Davé, R. 2015, Monthly Notices of the Royal Astronomical Society, 447, 374
  * Genel et al. (2018) Genel, S., Nelson, D., Pillepich, A., et al. 2018, Monthly Notices of the Royal Astronomical Society, 474, 3976
  * Glazebrook et al. (2017) Glazebrook, K., Schreiber, C., Labbé, I., et al. 2017, Nature, 544, 71
  * Gu et al. (2020) Gu, Y., Fang, G., Yuan, Q., & Lu, S. 2020, Publications of the Astronomical Society of the Pacific, 132, 1
  * Gupta et al. (2020) Gupta, A., Tran, K.-V., Cohn, J., et al. 2020, The Astrophysical Journal, 893, 23
  * Habouzit et al. (2019) Habouzit, M., Genel, S., Somerville, R. S., et al. 2019, Monthly Notices of the Royal Astronomical Society, 484, 4413
  * Hani et al. (2020) Hani, M. H., Gosain, H., Ellison, S. L., Patton, D. R., & Torrey, P. 2020, Monthly Notices of the Royal Astronomical Society, 493, 3716
  * Harrison (2017) Harrison, C. M. 2017, Nature Astronomy, 1, arXiv:1703.06889
  * Hopkins & Quataert (2010) Hopkins, P. F., & Quataert, E. 2010, Monthly Notices of the Royal Astronomical Society, 407, 1529
  * Keating et al. (2015) Keating, S. K., Abraham, R. G., Schiavon, R., et al. 2015, Astrophysical Journal, 798, doi:10.1088/0004-637X/798/1/26
  * Lu et al. (2020) Lu, S., Gu, Y., Fang, G., et al. 2020, arXiv:2006.06223
  * Lu et al. (2019) Lu, S. Y., Gu, Y. Z., Fang, G. W., & Yuan, Q. R. 2019, Research in Astronomy and Astrophysics, 19, 1
  * Marinacci et al. (2018) Marinacci, F., Vogelsberger, M., Pakmor, R., et al. 2018, Monthly Notices of the Royal Astronomical Society, 26, 1
  * Martizzi et al. (2020) Martizzi, D., Vogelsberger, M., Torrey, P., et al. 2020, Monthly Notices of the Royal Astronomical Society, 491, 5747
  * McDermid et al. (2015) McDermid, R. M., Alatalo, K., Blitz, L., et al. 2015, Monthly Notices of the Royal Astronomical Society, 448, 3484
  * Medling et al. (2015) Medling, A. M., Vivian, U., Max, C. E., et al. 2015, Astrophysical Journal, 803, arXiv:1502.06617
  * Mosleh et al. (2017) Mosleh, M., Tacchella, S., Renzini, A., et al. 2017, The Astrophysical Journal, 837, 2
  * Naiman et al. (2018) Naiman, J. P., Pillepich, A., Springel, V., et al. 2018, Monthly Notices of the Royal Astronomical Society, 18, 1
  * Nelson et al. (2018a) Nelson, D., Pillepich, A., Springel, V., et al. 2018a, Monthly Notices of the Royal Astronomical Society, 475, 624
  * Nelson et al. (2018b) Nelson, D., Kauffmann, G., Pillepich, A., et al. 2018b, Monthly Notices of the Royal Astronomical Society, 477, 450
  * Nelson et al. (2019a) Nelson, D., Pillepich, A., Springel, V., et al. 2019a, Monthly Notices of the Royal Astronomical Society, 490, 3234
  * Nelson et al. (2019b) Nelson, D., Springel, V., Pillepich, A., et al. 2019b, Computational Astrophysics and Cosmology, 6, 2
  * Peng et al. (2010) Peng, Y.-j., Lilly, S. J., Kovač, K., et al. 2010, The Astrophysical Journal, 721, 193
  * Pillepich et al. (2018a) Pillepich, A., Nelson, D., Hernquist, L., et al. 2018a, Monthly Notices of the Royal Astronomical Society, 475, 648
  * Pillepich et al. (2018b) Pillepich, A., Springel, V., Nelson, D., et al. 2018b, Monthly Notices of the Royal Astronomical Society, 473, 4077
  * Rangel et al. (2014) Rangel, C., Nandra, K., Barro, G., et al. 2014, Monthly Notices of the Royal Astronomical Society, 440, 3630
  * Rodriguez-Gomez et al. (2015) Rodriguez-Gomez, V., Genel, S., Vogelsberger, M., et al. 2015, Monthly Notices of the Royal Astronomical Society, 449, 49
  * Rodriguez-Gomez et al. (2016) Rodriguez-Gomez, V., Pillepich, A., Sales, L. V., et al. 2016, Monthly Notices of the Royal Astronomical Society, 458, 2371
  * Rodriguez-Gomez et al. (2017) Rodriguez-Gomez, V., Sales, L. V., Genel, S., et al. 2017, Monthly Notices of the Royal Astronomical Society, 467, 3083
  * Snyder et al. (2011) Snyder, G. F., Cox, T. J., Hayward, C. C., Hernquist, L., & Jonsson, P. 2011, The Astrophysical Journal, 741, 77
  * Sparre & Springel (2016) Sparre, M., & Springel, V. 2016, Monthly Notices of the Royal Astronomical Society, 462, 2418
  * Springel et al. (2005) Springel, V., Di Matteo, T., & Hernquist, L. 2005, The Astrophysical Journal, 620, L79
  * Springel et al. (2018) Springel, V., Pakmor, R., Pillepich, A., et al. 2018, Monthly Notices of the Royal Astronomical Society, 475, 676
  * Straatman et al. (2014) Straatman, C. M., Labbé, I., Spitler, L. R., et al. 2014, Astrophysical Journal Letters, 783, 2
  * Straatman et al. (2018) Straatman, C. M. S., van der Wel, A., Bezanson, R., et al. 2018, The Astrophysical Journal Supplement Series, 239, 27
  * Tacchella et al. (2019) Tacchella, S., Diemer, B., Hernquist, L., et al. 2019, Monthly Notices of the Royal Astronomical Society, 487, 5416
  * Terrazas et al. (2017) Terrazas, B. A., Bell, E. F., Woo, J., & Henriques, B. M. B. 2017, The Astrophysical Journal, 844, 170
  * Terrazas et al. (2020) Terrazas, B. A., Bell, E. F., Pillepich, A., et al. 2020, Monthly Notices of the Royal Astronomical Society, 493, 1888
  * Tran et al. (2020) Tran, K.-V. H., Forrest, B., Alcorn, L. Y., et al. 2020, The Astrophysical Journal, 898, 45
  * Truong et al. (2020) Truong, N., Pillepich, A., Werner, N., et al. 2020, Monthly Notices of the Royal Astronomical Society, 494, 549
  * van der Wel et al. (2016) van der Wel, A., Noeske, K., Bezanson, R., et al. 2016, The Astrophysical Journal Supplement Series, 223, 29
  * van Dokkum et al. (2008) van Dokkum, P. G., Franx, M., Kriek, M., et al. 2008, The Astrophysical Journal, 677, L5
  * Wang et al. (2018) Wang, H., Mo, H. J., Chen, S., et al. 2018, The Astrophysical Journal, 852, 31
  * Weinberger et al. (2017) Weinberger, R., Ehlert, K., Pfrommer, C., Pakmor, R., & Springel, V. 2017, Monthly Notices of the Royal Astronomical Society, 470, 4530
  * Wellons et al. (2015) Wellons, S., Torrey, P., Ma, C.-P., et al. 2015, Monthly Notices of the Royal Astronomical Society, 449, 361
  * Williams et al. (2017) Williams, C. C., Giavalisco, M., Bezanson, R., et al. 2017, The Astrophysical Journal, 838, 94
  * Wu et al. (2018) Wu, P.-f., van der Wel, A., Bezanson, R., et al. 2018, The Astrophysical Journal, 868, 37
  * Zhang et al. (2019) Zhang, C., Peng, Y., Ho, L. C., et al. 2019, The Astrophysical Journal, 884, L52
  * Zhu et al. (2018) Zhu, Q., Xu, D., Gaspari, M., et al. 2018, Monthly Notices of the Royal Astronomical Society: Letters, 480, L18
  * Zinger et al. (2020) Zinger, E., Pillepich, A., Nelson, D., et al. 2020, Monthly Notices of the Royal Astronomical Society, 499, 768
  * Zolotov et al. (2014) Zolotov, A., Dekel, A., Mandelker, N., et al. 2014, Monthly Notices of the Royal Astronomical Society, 450, 2327


11institutetext: Kansas State University, 2164 Engineering Hall, Manhattan, KS
43017-6221, USA
11email<EMAIL_ADDRESS>
# eGAN: Unsupervised approach to class imbalance using transfer learning

Ademola Okerinde  Lior Shamir  William Hsu  Tom Theis  Nasik Nafi

###### Abstract

Class imbalance is an inherent problem in many machine learning classification
tasks. This often leads to trained models that are unusable for any practical
purpose. In this study we explore an unsupervised approach to address these
imbalances by leveraging transfer learning from pre-trained image
classification models to encoder-based Generative Adversarial Network (eGAN).
To the best of our knowledge, this is the first work to tackle this problem
using GAN without needing to augment with synthesized fake images.

In the proposed approach we use the discriminator network to output a negative
or positive score. We classify as minority, test samples with negative scores
and as majority those with positive scores. Our approach eliminates epistemic
uncertainty in model predictions, as the $P(minority)+P(majority)$ need not
sum up to 1. The impact of transfer learning and combinations of different
pre-trained image classification models at the generator and discriminator is
also explored.

Best result of 0.69 F1-score was obtained on CIFAR-10 classification task with
imbalance ratio of 1:2500.

Our approach also provides a mechanism of thresholding the specificity or
sensitivity of our machine learning system.

Keywords: Class imbalance, Transfer Learning, GAN, nash equilibrium

##  1 INTRODUCTION

A dataset is said to be imbalanced when there is a significant, or in some
cases extreme, disproportion between the number of examples of the different
classes in the dataset. The class or classes with large number of samples are
called the majority while the class with few examples are denoted as the
minority. In many cases, the machine learning model is required to correctly
classify the minority class while minimizing the misclassification of the
majority class. However, the skewness in data often leads machine learning
classification methods to favour the majority class.

Class imbalance problem in computer vision are normally approached in the
following ways, either at the data or algorithm level. Data augmentation is a
technique of transforming images via scaling, cropping, flipping, padding,
rotation, brightness, contrast, saturation level etc. As a result, a humongous
image dataset can be created from a single image of the minority class. By
using data augmentation, a class with a small number of samples can be
expanded into a class with a much larger number of samples. The synthetic
images can also be generated by using GAN to augment the minority class. Other
approaches include deliberate undersampling of the majority class or
oversampling of the minority class by mere copying. The earlier approach leads
to loss of useful data information while the latter approach causes
overfitting.

At the algorithm level, the objective function is tweaked to heavily penalize
the network for mis-classifying the minority class. The most popular is cost-
sensitive approach. Here, the classifier iis modified to incorporate varying
penalty for each of considered groups of examples. This way by assigning a
higher cost to less represented set of objects its importance is boost during
training.

Transfer learning has been known to help improve the performance of machine
learning models. By fine-tuning varying number of layers in the pre-trained
image classification model, the pre-trained model can serve as a feature
extractor, while adding a classifier head for more specific feature learning
for the current task.

In this work we compared the performance of various pre-trained image
classification models for the task of unsupervised image classification with
varying imbalance ratios. Our architecture, named eGAN, is developed to serve
as a basis for this comparison. Using GAN [2], we re-parameterise the job of
the discriminator as a classifier giving a positive score to majority samples
and negative to minority ones. While most GAN-based architectures focus on the
output of the generator, in the proposed architecture, we intuitively adapted
the vanilla GAN network by integrating an encoder module that takes minority
samples as input and produces a latent code, from which the generator learns.

##  2 RELATED WORK

In [4] an ensemble method was proposed based on advanced generative
adversarial network to generate new samples for the minority class to restore
balance. Our opinion is that the computational demand of such approach is
enormous, and not many low-income countries of the world have access to such
compute power. We eliminate the need to generate a realistic image in our
technique because our network is trained on imbalance dataset. Deep Cascading
(DC) with a long sequence of decision trees could help to handle unbalanced
data [1]. A DC is a sequence of n classifiers where each sample x passes to
the next classifier only if the current one classifies it as positive
according to a high-sensitivity decision threshold. However, this works well
with foreground-background imbalance unlike classification task. Transfer
learning with GAN was used to generate images from limited data in [8]. Their
result showed that knowledge from pre-trained networks can shorten the
convergence time and significantly improve the quality of generated images.

##  3 METHODOLOGY AND EXPERIMENTAL DESIGN

In this section, we discuss our proposed approach and the various testbeds
that were used in our experiments.

###  3.1 ADDRESSING CLASS IMBALANCE WITH eGAN

The proposed architecture is based on adaptation of existing Deep
Convolutional Generative Adversarial Network (DCGAN)[7] by incorporating an
encoder module. This module encodes minority samples in latent space needed by
the generator G to generate minority samples that are capable of fooling the
discriminator D. On the other hand, the discriminator D is fed with data
samples drawn from majority distribution and the generated output of the
generator G. D and G are simultaneously optimized through the following two-
player minimax game with value function V(G,D) in 1.

$\overset{min}{G}\,\,\overset{max}{D}V(D,G)=E_{X_{ma}\sim P_{ma}}[\log
D(X_{ma})]+E_{X_{mi}\sim P_{mi}}[\log(1-D(G(X_{mi})))]$ (1)

where $X_{ma}$ and $X_{mi}$ are majority and minority sample distributions
respectively.

Over the course of iteration, the discriminator D is optimized to assign a
negative score to the minority data distribution and a positive score to the
majority data distribution. This enables the discriminator D to act as a
classifier.

####  3.1.1 Encoder-Generator module

Our latent space is composed of 128 units vector. Rather than feeding the
generator with random noise as is typical of most GAN implementation, we added
an encoder module that forces the generator to learn from known distribution
(minority distribution). The encoder part consists of the pre-trained
DenseNet121 followed by global average pooling layer and latent dimension
space. The generator part has two transposed convolutional layers. We use
leakyRelu activation function with alpha set to 0.2; batch normalization and
Sigmoid function at the final layer.

####  3.1.2 Discriminator module

The pre-trained discriminator has 7,038,529 parameters out of which only
39,937 are trainable. A layer of global average pooling follows the pre-
trained DenseNet121. We use a dropout of 0.2 followed by the final one unit
dense layer. The overall architecture of our encoder-based generator
adversarial network is shown in Figure 1.

Figure 1: Encoder-based Generative Adversarial Network (eGAN) architecture

###  3.2 Selection of pre-trained image classification weights

We perform experiments on VGG16, VGG19, EfficientNetB2, ResNet101 and
DenseNet121 pre-trained classification models on ImageNet dataset. Here, we
fine-tuned only top five layers at each of the pre-trained models. Table 3
shows the maximum precision, recall and F1-score obtained on CIFAR-100 with
imabalance ratio 1:50 by using different combinations of pre-trained models.

DenseNet121 [3] was used for pretraining our eGAN. After experimenting
different pre-trained architectures and different layers of fine-tuning, we
obtained best result with fine-tuning only top 5-layer out 427 layers of
DenseNet121.

###  3.3 Dataset

Several commonly used datasets were used in this study. In order to model the
real-world scenario of heavy imbalance, we used only few samples of the
minority class as input to the encoder module. Detail overview is shown in
Table 4.

####  3.3.1 CIFAR-10:

The CIFAR-10 dataset [6] consists of 60,000 32$\times$32 colour images in 10
classes, with 6000 images per class. There are 50,000 training images and
10,000 test images. Here we use airplane as minority and automobile as
majority.

####  3.3.2 CIFAR-100:

CIFAR-100 [6] dataset is similar to CIFAR-10, except it has 100 classes
containing 600 images each. Each class has 500 training images and 100 testing
images. The 100 classes in the CIFAR-100 are grouped into 20 superclasses.
Each image comes with a ”fine” label (the class in which it belongs) and a
”coarse” label (the superclass in which it belongs).

####  3.3.3 CheXpert:

We use the pneumonia subset of Stanford CheXpert dataset [5] for experimenting
on an inherent imbalanced dataset. The dataset contains 4576 and 167407
minority and majority samples, respectively.

##  4 RESULTS AND DISCUSSION

All discriminator scores that are less than zero are classified as minority,
otherwise they are classified as majority class. Table 3 shows the result
obtained by combining various pre-trained models. Adam optimizer with a
learning rate of 1e-4 was used to train all models for 100 epochs.

Table 1: Confusion matrix of imbalance ratio 1:1000 | Predicted minority | Predicted majority
---|---|---
Actually minority | 810 | 190
Actually majority | 366 | 634
Table 2: Confusion matrix of imbalance ratio 1:2500 | Predicted minority | Predicted majority
---|---|---
Actually minority | 821 | 179
Actually majority | 767 | 233
Table 3: Comparative analysis of different pre-trained models configuration on Generator and Discriminator using CIFAR-100 dataset Discriminator | Generator | Precision | Recall | F1
---|---|---|---|---
pre-trained | pre-trained |  |  |
ResNet101 | VGG19 | 0.72 | 1.0 | 0.78
VGG19 | ResNet101 | 0.73 | 1.0 | 0.69
EfficientNetB2 | VGG19 | 1.0 | 0.22 | 0.32
VGG19 | EfficientNetB2 | 0.7 | 0.86 | 0.71
ResNet101 | VGG16 | 1.0 | 0.17 | 0.27

As can be seen in Figure 4, the model achieved a nash equilibrium on test data
at around 10 epochs. Here, we perform inference on test data at every epoch
and plot the number of samples correctly classified. Five layers of
DenseNet121 were fine-tuned at each generator and discriminator module, while
422 layers’ weight were kept fixed. At nash, the discriminator correctly
classified roughly 700/1000 of each of the minority and majority test data.
This result convinces us that transfer learning with GAN can be used to
overcome the challenge of highly imbalanced dataset, owing to the fact that we
train only with 10 samples of the minority class and 5000 samples of the
majority class (ratio 1:500). Similar performance is observed in CIFAR-100
with imbalance ratio 1:50.

Without pre-training the discriminator, the effect of the high imbalance in
the training set is revealed, as the discriminator is skewed towards the
majority class in the training set, thereby missing all the minority samples
in the test data. This can be seen in Figure 5 on CIFAR-100. This behaviour
pattern is observed on CIFAR-10 as well. We experiment with no pre-training at
all, neither in the discriminator nor generator, and observed exact same
pattern. Therefore, we can safely conclude that the use of transfer learning
helps unsupervised image classification in a highly imbalanced domain.

Training can be stopped as soon as nash equilibrium is reached, as this point
gives the model best performance on the minority and majority class. An
acceptable threshold can also be set for the absolute difference of the number
of correctly classified samples of both classes. For instance, if the $\mid$
correctly_classified_minority - correctly_classified_majority $\mid$ $\leq$
20\. The precision, recall and F1-score curves on CIFAR-10 averaged over five
folds at different imbalance ratios are shown in Figure 4.

We observed that at the early training epochs typically between 1 and 40
epochs, the generator tries to achieve its objective of fooling the
discriminator by generating samples from the majority class fed into the
discriminator, which results in more of the minority samples being mis-
classified as the discriminator ”knows” the distribution of the majority too
well. A drastic change occurs when the generator start generating samples from
latent vector which is a able to conveniently fool the discriminator as can be
seen in generator and discriminator loss shown in Figure 2.

###  4.1 Imbalance ratios

In this section we discuss the performance of the proposed approach on various
class imbalance ratios. To eliminate bias in model performance, we conducted
5-fold-cross-validation on the minority samples and average the result.

####  4.1.1 Class ratio of 1:2500

We experiment on CIFAR-10 dataset by deliberately using an unbalanced subset
of the training set. At the 4th epoch, our model correctly identifies 257
minority and 821 majority samples out of 1000 each. At epoch 5, a sharp change
occurred that led to 821 minority samples being correctly classified, while
only correctly classifying 233 majority samples as shown in Table 2. We also
observed that at epoch 72 the performance of the network on the majority and
minority classes reached a nash equilibrium with a threshold difference of
less than or equal to 20.

####  4.1.2 Class ratio of 1:1000

At epoch 81 on CIFAR-10 dataset, nash equilibrium was reached. At this epoch,
532 and 525 minority and majority test data respectively were correctly
classified. We observed that the classifier had another major shift between
epoch 5 and 6. At epoch 5, best result was obtained. The network was able to
classify 863 majority tests and 585 minority tests correctly out of the 1000
samples. At epoch 6, 634 majority and 810 minority tests were classified
correctly as swown in Table 1. Maximum precision, F1-score and recall of 0.88,
0.74 and 0.99 were obtained at epochs 3, 6 and 37, respectively.

####  4.1.3 Class ratio of 1:500

Both CIFAR-10 and CIFAR-100 were used to experiment imbalance ratio 1:500. On
CIFAR-10, maximum precision, recall and F1-score on averaging 5-fold-cross-
validation are 0.75, 0.95 and 0.70 respectively as shown in Table 5. The
maximum precision is slightly lower on CIFAR-100 with 0.60. However, the
recall and F1-score which are 0.96 and 0.68 are roughly the same.

####  4.1.4 Class ratio of 1:50

We demonstrate our model performance on imbalance ratio 1:50 using CIFAR-100
and CIFAR-10. For CIFAR-100, a sudden change occurred between epoch 69 and 70
as follows majority: 57, minority: 59; and majority: 55, minority: 59. A nash
equilibrium is attained at epoch 68, with 56 correctly classified minority as
well as majority class. At epoch 97 maximum F1-score and recall of 0.66 and
0.77 were obtained respectively, while maximum precision of 0.6 was obtained
at epoch 74.

####  4.1.5 Class ratio of 1:1

We use CIFAR-100 to demonstrate the performance of eGAN on a balanced dataset.
We notice that the experimental performance follow the same pattern as
imbalanced dataset. Training starts with mostly all the majority correctly
classified and all the minority mis-classified. At epoch 48, a nash
equilibrium (with threshold less than or equal to 5) is achieved, with 76 and
71 of minority and majority correctly classified respectively. The maximum
F1-score of 0.78 is reached at epoch 53 as shown in Table 5. Instead of using
500 samples each of minority and majority class, by training on a single
instance of minority and majority sample (∗1:1) of CIFAR-100, we obtained an
F1-score of 0.63. This demonstrates the impact of transfer learning on the
training.

####  4.1.6 Class ratio of 1:36

For pneumonia subset of CheXpert dataset with imbalance ratio 1:36, the best
performed model achieves 0.51, 0.97, and 0.67 precision, recall, and F1-score
respectively. The results shown in Table 6 is evaluated on 1000 of each
minority and majority test set. As can be seen in the table, our approach did
not beat the baseline classification model because this task is more of an
anomaly detection task rather than a classification problem. Also the pre-
trained image classification model source dataset i.e. ImageNet is
significantly different from medical domain. In our opinion, exploring more
variants of complex GAN architectures like BigGAN, StyleGAN and ProGAN could
possibly help.

Table 4: CIFAR-10 and CIFAR-100 dataset overview - CIFAR-100 in parenthesis Class imbalance | # training | # training | # testing | # testing
---|---|---|---|---
ratio | minor | major | minor | major
1:2500 | 2(-) | 5000(-) | 1000(-) | 1000(-)
1:1000 | 5(-) | 5000(-) | 1000(-) | 1000(-)
1:500 | 10(1) | 5000(500) | 1000(100) | 1000(100)
1:50 | 100(10) | 5000(500) | 1000(100) | 1000(100)
1:1 | 500(500) | 500(500) | 1000(100) | 1000(100)
∗1:1 | -(1) | -(1) | -(100) | -(100)
Table 5: Maximum precision, recall and F1-score on CIFAR-10 and CIFAR-100 (avg. 5-fold) - CIFAR-100 in parenthesis ratio | Precision | Recall | F1
---|---|---|---
1:2500 | 0.72(-) | 0.94(-) | 0.69(-)
1:1000 | 0.72(-) | 0.96(-) | 0.69(-)
1:500 | 0.75(0.60) | 0.95(0.96) | 0.70(0.68)
1:50 | 0.78(0.7) | 0.97(0.86) | 0.69(0.71)
1:1 |  0.82(0.72) | 0.98(1.0) | 0.72(0.78)
∗1:1 | -(0.53) | -(0.86) | -(0.63)
Table 6: Precision, recall and F1-score on pneumonia subset of CheXpert dataset | #training | #testing |
---|---|---|---
Model | minor | major | minor | major | imbalance ratio | Precision | Recall | F1
eGAN | 30 | 1080 | 1000 | 1000 | 1:36 | 0.51 | 0.97 | 0.67
baseline | 30 | 1080 | 1000 | 1000 | 1:36 | 0.5 | 1.0 | 0.67

Figure 2: Discriminator and Generator loss.

Figure 3: CheXpert pneumonia Xray - majority class (top row), minority class
(bottom row).

Figure 4: eGAN’s average precision (top left), average recall (top right),
average F1-score (bottom left), and average performance (bottom right) on
CIFAR-10 test dataset using DenseNet121. Figure 5: Performance of eGAN on
CIFAR-100 test data using DenseNet121. Only the generator network is pre-
trained

##  5 CONCLUSION

In this work, we explore the use of pre-trained image classification models on
the task of image classification on varying levels of imbalance ratios in the
training dataset. Our approach demonstrates unsupervised technique to
addressing class imbalance by using pre-trained GAN. Rather than synthesizing
fake images with the generator for data augmentation, we employ the
discriminator as a classifier between the classes by scoring minorities and
majorities negatives and positives respectively. The performance measure of
interest plays a significant role in deciding what epoch of the trained model
to deploy in production as various epochs favour different evaluation metrics
for example sensitivity. Experimental results reveal that transfer learning
plays a significant role in the model performance. In continuation of this
work, we will explore the usage of this approach for anomaly detection task
where distinguishing features between normal(majority) and abnormal(minority)
is less profound. Our work can be further explored in object detection tasks
in cases of imbalance between foreground and background.

## References

  * [1] Alessandro Bria, Claudio Marrocco and Francesco Tortorella  “Addressing class imbalance in deep learning for small lesion detection on medical images”  In _Computers in Biology and Medicine_ 120 Elsevier, 2020, pp. 103735
  * [2] Ian J Goodfellow et al.  “Generative adversarial networks”  In _arXiv preprint arXiv:1406.2661_ , 2014
  * [3] Gao Huang, Zhuang Liu, Laurens Van Der Maaten and Kilian Q Weinberger  “Densely connected convolutional networks”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 4700–4708
  * [4] Yangru Huang, Yi Jin, Yidong Li and Zhiping Lin  “Towards Imbalanced Image Classification: A Generative Adversarial Network Ensemble Learning Method”  In _IEEE Access_ 8 IEEE, 2020, pp. 88399–88409
  * [5] Jeremy Irvin et al.  “Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 33.01, 2019, pp. 590–597
  * [6] Alex Krizhevsky and Geoffrey Hinton  “Learning multiple layers of features from tiny images”  Citeseer, 2009
  * [7] Alec Radford, Luke Metz and Soumith Chintala  “Unsupervised representation learning with deep convolutional generative adversarial networks”  In _arXiv preprint arXiv:1511.06434_ , 2015
  * [8] Yaxing Wang et al.  “Transferring gans: generating images from limited data”  In _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 218–234