# Channel-Selective Normalization for Label-Shift Robust Test-Time Adaptation

Pedro Vianna1,2,3 Muawiz Chaudhary4,5 Paria Mehrbod4,5 An Tang2,3,6 Guy
Cloutier1,2,3 Guy Wolf2,4 Michael Eickenberg7 Eugene Belilovsky4,5
1 Laboratory of Biorheology and Medical Ultrasonics; 2 Université de Montréal;
3 Centre de Recherche du Centre Hospitalier de l’Université de Montréal; 4
Mila – Quebec AI Institute;
5 Concordia University; 6 Centre Hospitalier de l’Université de Montréal; 7
Flatiron Institute This work was supported by grants from the Institute of
Data Valorization (IVADO PRF3) to A.T., G.C., G.W. and E.B. A.T. acknowledges
support from the Fonds de Recherche du Québec – Santé (FRQS), Fonds de
Recherche du Québec - Nature et technologies (FRQNT) and the Fondation de
l’Association des Radiologistes du Québec (FARQ) Clinical Research
Scholarship–Senior Salary Award (FRQSARQ no. 298509). G.W. acknowledges
support from the Canada CIFAR AI Chair. E.B. and G.W. acknowledge funding from
Fonds de recherche du Québec — Nature et technologies - NOVA (2023-NOVA-329759
and 2023-NOVA-329125). Correspondence to<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Deep neural networks have useful applications in many different tasks, however
their performance can be severely affected by changes in the data
distribution. For example, in the biomedical field, their performance can be
affected by changes in the data (different machines, populations) between
training and test datasets. To ensure robustness and generalization to real-
world scenarios, test-time adaptation has been recently studied as an approach
to adjust models to a new data distribution during inference. Test-time batch
normalization is a simple and popular method that achieved compelling
performance on domain shift benchmarks. It is implemented by recalculating
batch normalization statistics on test batches. Prior work has focused on
analysis with test data that has the same label distribution as the training
data. However, in many practical applications this technique is vulnerable to
label distribution shifts, sometimes producing catastrophic failure. This
presents a risk in applying test time adaptation methods in deployment. We
propose to tackle this challenge by only selectively adapting channels in a
deep network, minimizing drastic adaptation that is sensitive to label shifts.
Our selection scheme is based on two principles that we empirically motivate:
(1) later layers of networks are more sensitive to label shift (2) individual
features can be sensitive to specific classes. We apply the proposed technique
to three classification tasks, including CIFAR10-C, Imagenet-C, and diagnosis
of fatty liver, where we explore both covariate and label distribution shifts.
We find that our method allows to bring the benefits of TTA while
significantly reducing the risk of failure common in other methods, while
being robust to choice in hyperparameters.

##  1 Introduction

Deep learning models have shown excellent performance on a wide variety of
tasks in computer vision research, such as image classification [22, 50],
segmentation [46, 11], and object detection [7, 48]. However, a commonly cited
limitation of these models is the inability to generalize across different
domains [14]. Generalization can be simply defined as the ability of a deep
learning algorithm, trained on one or more source domains, to be applied to a
different yet still related target domain. Typically, in real-world deployment
scenarios models might encounter data with critical differences, hampering
their performance. This decrease in performance has been observed in multiple
areas, including life-threatening contexts, such as autonomous driving [19,
43] and medical diagnostics [3, 45].

Domain adaptation techniques aim to enhance a model’s generalizability,
anticipating the possible challenges in the event of distribution shifts.
Traditionally, domain adaptation assumes that the model has access to
unlabeled test data and source data during the training, which may not be
practical in some cases [34, 25]. A recently emerging technique to deal with
distribution shift is test-time adaptation (TTA) [40, 26], a type of
unsupervised domain adaptation, where unlabeled test data is used to update
the model parameters at test-time, before predictions. It is often assumed
that data arrives in batches, and some studies have proposed a setting of
test-time batch adaptation that take advantage of batch-level information to
adapt to the distribution shift [44, 30, 8].

Test-time batch normalization (TTN) [30, 37] is an approach which replaces
batch normalization statistics estimated as running averages on the training
set with the statistics of the test data batch. Despite being a simple
approach, it has been shown to improve robustness under covariate shift,
handling particularly well various cases of image corruptions. Based on that,
other TTA approaches apply TTN as a critical component in their foundation
[44, 27]. Alas, most existing TTA methods consider the impact of covariate
shifts only, in many realistic scenarios, such as medical imaging, the label
distribution of data can shift from training to testing.

In order to be practically useful, TTA methods relying on batch data need to
be able to operate outside of this optimistic scenario of class-balanced
training and testing settings, while simultaneously dealing with covariate
shifts. As noted in recent work [4] label distribution shifts can cause
failures. We emphasize in our work that sometimes these failures can be
catastrophic, thus creating risk in the usage of TTA in the first place when
the label distribution of batches obtained under deployment cannot be
adequately guaranteed. We observe in our work that certain channels and layers
can be much more sensitive with respect to label distribution shifts under
TTN. Motivated by this, we propose a method to mitigate the risk of label
distribution shift catastrophically affecting model predictions. It is based
on the adaptation of only those channels of the batch normalization layers
whose activation statistics do not exhibit strong correlation to label
distribution. Our proposed method is applied to classification of two well-
known benchmark natural image datasets (CIFAR-10 [21] and ImageNet-1K [9]),
and finally on adaptation from a private medical dataset [42] to a publicly
available one [6]. When deployed in target data with different distribution,
our proposed method is effective in obtaining the benefits of TTA while
avoiding catastrophic failure cases, greatly improving the practical utility
of TTN methods.

Our contributions are summarized below:

  1. 1.

We propose a novel method that deals with label distribution shifts by
leveraging information that can be easily computed at the end of model
training and used during inference time to select channels sensitive to class
information. We augment this channel selection strategy with a prior on the
effect of depth on sensitivity to class information.

  2. 2.

Experiments on three different datasets demonstrate that our method can be
applied on a range of classification tasks, in particular for highly
imbalanced target data. Notably, the adaptation across liver ultrasound
datasets from different sites is shown to be effective.

##  2 Related Work

Batch normalization [17], henceforth BatchNorm, is a technique that aims to
stabilize the distributions of layer inputs by introducing additional network
layers that control the first two moments (mean and variance) of these
distributions. This positively affects the optimization process of deep neural
networks during training, reduces overfitting, and prevents the exploding or
the vanishing gradient problems.

Even though BatchNorm is effective, widely accepted, and commonly used in many
deep neural networks, its mechanisms are not entirely understood. Methods
extending this approach provide alternatives to BatchNorm, such as layer
normalization [1], instance normalization [41], and normalization based on
groups of channels [47]. Furthermore, weight normalization [35] extends this
line of work, modifying the weights of the models instead of features.

Various studies have been conducted to further understand how batch-
normalization works. The original motivation for BatchNorm refers to the
problem of internal covariate shift, however a later study [36] notes that it
is not an appropriate explanation. Rather, the loss landscapes of models with
BatchNorm appear to be smoother than those of equivalent models without it,
allowing for use of larger range of learning rates and faster network
convergence. Moreover, by investigating randomly initialized networks with all
layers frozen and adapting only the BatchNorm layers, it has been shown that
BatchNorm layers have substantial expressive power, achieving surprisingly
good results on CIFAR-10 and ImageNet [12].

Test-time adaptation methods often involve updating the parameters associated
with the BatchNorm layers to distributions with covariate shift [37, 30]. This
approach is closely related to the unsupervised domain adaptation method AdaBN
[24, 5], and has achieved promising results in mitigating image corruptions.
Recent TTA methods have been published using entropy minimization on the
BatchNorm affine parameters [44], but they appear to be biased towards the
dominant classes [33]. In contrast, a distinct class of methods focuses on TTA
without relying on BatchNorm adjustments, proposing techniques that leverage
ideas from self-supervised learning [39, 8], or test-time augmentations [16,
18].

A common limitation in TTA studies is regarding label distribution shifts, as
most of the proposed methods fail to consider the varying class balance in the
test sets, eventually falling into the “winner-takes-all” dilemma in which the
majority classes dominate the predictions [49, 26]. To address the class-
imbalanced domain adaptation challenge, some works focus on estimating the
test set distribution [38, 28, 33].

Recently, other works have also proposed methods for dealing with label
distribution shift in TTA. LAME [4] performs the online adaptation of the
classification outputs instead of the model’s parameters, making it agnostic
to training and testing conditions, but it does not perform well in balanced
scenarios. DELTA [53] uses test-time batch renormalization and dynamic online
reweighting in an online manner to avoid mini-batch bias. A label shift
adapter was designed [33] to improve TTA performance in scenarios where both
source and target are imbalanced, by estimating the distribution of the target
set. However, it uses a large number of hyperparameters and is specialized to
the case of imbalanced source data.

##  3 Hybrid Test-Time Batch Normalization

The main idea behind TTA in general, and TTN in particular, is that while
label information is not available at test time, the unlabeled data can
provide some information to estimate impact of domain shifts on neural network
operations and internal representations. The typical setup is based on data
being processed in batches, enabling assessment of distribution shifts between
source and target domains.

In order to implement TTA in these settings, TTN views a neural network $f$ as
split into blocks separated by BatchNorm layers:

$f=f_{0}\circ B_{0}^{s_{0}}\circ f_{1}\circ B_{1}^{s_{1}}\cdots f_{K-1}\circ
B_{K-1}^{s_{K-1}}\circ f_{K},$ (1)

where $f_{0},\ldots,f_{K}$ are blocks (i.e., sub-networks) of hidden layers
and $B_{0},\ldots,B_{K-1}$ are batch normalization operators. Each BatchNorm
layer modifies each neural activation by

$B(h(x),\mu_{k},\sigma_{k},\beta,\gamma)=\gamma\frac{h(x)-\mu_{k}}{\sigma_{k}}+\beta$
(2)

where $h(x)$ is the layer input batch, $\beta$ and $\gamma$ are parameters
learned during the training process, and $\mu$ and $\sigma$ represent
estimates of the mean and standard deviation of neuron activation over data.

The main premise of the TTN approach is that changes in the distributions of
activations of each neuron between source and target batches would
predominantly be caused by unwanted covariate shifts, and therefore should be
eliminated (TTN eliminates a diagonal affine shift exactly). However, this
does not take into account other distribution shifts that should affect, at
the very least, the output distribution of the network. In particular, in many
real-world applications, one cannot guarantee that classes will be well
balanced in general and certainly at test time. Therefore it is often the case
that the distribution of available labels during the training process, i.e.,
on the source domain, will differ from the one encountered at test time.

Input: Trained network $n$ with $K$ layers; source model statistics
$\\{\bm{\mu}_{k}^{s},\bm{\sigma}_{k}^{s}\\}_{k=1..K}$; source data $x_{s}$;

1

2for  _$c$ in $C$ classes_ do

3 Sample Data from Class $c$

4 $x_{0}\sim D_{c}$

5 for  _$k$ in $\\{1,...,K\\}$_ do

6 $\bm{x}_{k}=n_{k-1}(\bm{x}_{k-1})$

7 $\bm{\mu}_{k}^{t},\bm{\sigma}_{k}^{t}$ $\leftarrow$ Compute BN stats from
$\bm{x}_{k}$

8

9 for  _$f$ in $F$ channels_ do

10 Scores[c, k, f] =
$W_{2}^{2}(\bm{\mu}_{f}^{s},\bm{\sigma}_{f}^{s},\bm{\mu}_{f}^{t},\bm{\sigma}_{f}^{t})$

11

12 end for

13

14 end for

15

16 end for

17

Output:Scores

Algorithm 1 Hybrid-TTN Post-Training

Input: Trained network $n$ with $K$ layers; source model statistics
$\\{\bm{\mu}_{k}^{s},\bm{\sigma}_{k}^{s}\\}_{k=1..K}$; target data batch
$x_{0}$, label distribution prior $p_{1..c}$.

1 for  _$k$ in $\\{1,...,K\\}$_ do

2 $\bm{x}_{k}=f_{k-1}(\bm{x}_{k-1})$

3 $\bm{\mu}_{k}^{t},\bm{\sigma}_{k}^{t}$ $\leftarrow$ Compute BN stats from
$\bm{x}_{k}$

4 $R=\frac{k}{K}$

5 for  _$c$ in $C$ classes_ do

6 for  _$f$ in $F$ channels_ do

7 Scoresk[f] += $p_{c}*$Scores[c, k, f]

8 end for

9

10 end for

11 topR-ind $\leftarrow$ Compute Top-R Index(Scoresk)

12 $\bm{m}_{k}[\textsc{topR-ind}]=0$

13 $\bm{m}_{k}[\tilde{\textsc{topR-ind}}]=1$

14
$\bm{\mu}_{k}^{hybrid}=\bm{m}_{k}\odot\bm{\mu}_{k}^{t}+(1-\bm{m}_{k})\odot\bm{\mu}_{k}^{s}$

15
$\bm{\sigma}_{k}^{hybrid}=\bm{m}_{k}\odot\bm{\sigma}_{k}^{t}+(1-\bm{m}_{k})\odot\bm{\sigma}_{k}^{s}$

16 $\bm{x}_{k}=B(\bm{x}_{k},\bm{\mu}_{k}^{hybrid},\bm{\sigma}_{k}^{hybrid})$

17

18 end for

19

Output:$\\{\bm{\mu}_{k}^{hybrid},\bm{\sigma}_{k}^{hybrid}\\}_{k=1..K}$,
$x_{K}$

Algorithm 2 Hybrid-TTN Adaptation

Our proposed method to tackle this problem consists of three primary
ingredients:

  * •

Creating a channel-level score to indicate the sensitivity to individual
classes

  * •

Using a simple prior that linearly decays the number of selected channels in
depth

  * •

A prior on the label distribution, this can be uniform or derived from a
pseudolabeling step

We now describe the three phases of the method:

Storing Post-Training Statistics. To mitigate the adverse effects by TTN, our
first step focuses on selecting channels for the adaptation process. We aim to
identify channels that are primarily sensitive to covariate shifts while
excluding those highly sensitive to shifts in the label distribution. This
channel selection procedure operates using an initial phase that uses the
labelled training set before the model is deployed to compute a set of scores.
For each class, denoted as $c$ out of $C$ total classes, we input a subset of
data from the class, $D_{c}$, into a trained neural network with $K$ layers.
Each layer $k$ within this network possesses source statistics $\mu_{k}^{s}$
and $\sigma_{k}^{s}$, and the target statistics for the single class,
$\mu_{k}^{t}$ and $\sigma_{k}^{t}$, are computed. The Wasserstein distance is
then calculated for the source and target distributions with
$W_{2}^{2}(\\{\mu^{s},\sigma^{s}\\},\\{\mu^{t},\sigma^{t}\\})=\|\mu^{s}-\mu^{t}\|^{2}+(\sigma^{s}-\sigma^{t})^{2}$.
This step is described in Algorithm 1.

Test-Time Scoring and Masking For each channel we now have a notion of the
sensitivity to each class. We consider a prior $p(c)$ on the class
distribution of the target data. For each channel we compute the weighted
score of the Wasserstein distances obtained at end of training, with the
weighting determined by the class distribution prior. Under a uniform prior on
the target data we can simply sum the sensitivities for each class to get a
score representing the sensitivity.

At each layer we choose the top-$R$ channels for adaptation. Using the prior
that later layers are more sensitive to label shifts we decay $R$ with a
simple linear function in depth. The algorithm is fully described in Algorithm
2.

Re-scoring with Pseudolabels In the method analyzed in our paper we focus on a
simple method to obtain $p(c)$. Specifically we first run Algorithm 2 with a
uniform prior to obtain an initial corrected model, then use this to obtain
pseudolabels for the batch, which are used to recalculate the prior, we then
run Algorithm 2 again with the non-uniform prior.

Figure 1: Effect of extreme class imbalance and covariate shift on test-time
batch-normalized (TTN) performance: TTN mitigates distributional shift but
greatly suffers from class imbalance. We show per-class accuracy plots for a
Source model (blue), a class-balanced TTN model (orange), and a class-
imbalanced (1-class) TTN model (green). Observe that the class-imbalanced TTN
model performs very poorly on the most prevalent class label of the imbalanced
adaptation set (label 7, red rectangle).

##  4 Experiments and Results

In this section we discuss our experimental setup.

We use three datasets in our evaluations: two popular benchmarks, CIFAR-10-C
and ImageNet-1K-C, and a publicly available fatty liver ultrasound dataset
[6].

Figure 2: On CIFAR-10 we adapt models only up to the layer shown on the
x-axis, the y-axis showing the performance on the target data. We consider
label distributions with all (10) classes as well as 5,3, and 1 randomly
selected and balanced classes. We observe that adapting later layers has an
outsized role in the catastrophic collapse due to TTN

CIFAR-10 and CIFAR-10-C. We use the CIFAR-10 [20] dataset along with
CIFAR-10-C [15]. CIFAR-10 is a small natural image dataset with 50k training
images and 10k validation images. CIFAR-10-C contains corrupted versions of
the CIFAR-10 Validation set at varying severities. We train our models on the
uncorrupted dataset.

ImageNet-1K and ImageNet-1K-C. We use the ImageNet-1K [10] dataset along with
ImageNet-1K-C [15]. ImageNet-1K is a large natural image dataset with 1.2
million training images and 50k validation images. ImageNet-1K-C, similarly to
CIFAR-10-C, contains corrupted versions of the ImageNet-1K Validation set at
varying severities. Both CIFAR-10-C and ImageNet-1K-C are popular as a measure
of robustness to covariate shift.

Liver Ultrasound. We use, as the training data, a dataset of ultrasound
abdominal images collected at a liver transplantation center from patients
under suspicion of fatty liver disease [42]. Images were collected between
September 2011 to October 2020 with seven different devices, with varying
settings, and by multiple operators. It contains 7,529 B-mode images from 57
control cases without NAFLD and 142 patients with NAFLD. We train our models
for binary classification, i.e. detecting fatty liver versus non-fatty liver,
on this dataset. For the test data in TTN, we use a publicly available dataset
collected at a different institution [6], with 550 images. The differences
between the source and the target data are multiple. On the patients level, we
have differences in population demographics and in the inclusion criteria,
e.g., the 55 patients on the publicly available dataset are diagnosed with
obesity, which impacts the wave propagation during ultrasound exams. On the
examinations level, the second dataset was collected prospectively, with
consistent settings and using a single ultrasound machine, which was not
present in the source data collection. This creates a more homogeneous dataset
and provides the covariate shift for the TTN method. Finally, the training
dataset has 84% of its images from cases of fatty liver, while the test
dataset has 69%, introducing the label distribution shift. For both datasets,
all patients underwent liver biopsy and the steatosis grades in the pathology
reports are used as the reference standard for diagnosis of fatty liver. In
order to introduce higher levels of covariate shift akin to CIFAR-10 and
ImageNet-1K evaluations, we added speckle noise to the target dataset, with
varying levels of corruption.

Figure 3: Illustration of explanation for depth-wise behaviour under label
distribution shift. We consider one class mean (green) which is shifted
towards the data mean, as would be the case in a highly imbalanced setting.
Classes are not well separated in early layers and thus shifts in any mean are
relatively small and non-intrusive. In later layers classes are well separated
and a large shift of points from one mean towards the data mean is likely to
cross a decision boundary. Data points in other classes moving away from the
data mean are less likely to cross a decision boundary.

#### Architecture Details

On CIFAR-10 we train a Resnet-26 model as defined in [13]. For ImageNet-1K we
use a pre-trained Resnet18 model. For the Liver Ultrasound data, we train a
Resnet-18 model.

Adaptation Details We focus on the source-free domain adaptation setting where
adaptation is done on a single batch without affecting the deployed model
[26]. For CIFAR-10-C and ImageNet-1K-C we use a batch size of 500 for the
experiments (sampled over multiple seeds). For Liver Ultrasound, the batch
size is the length of the target set. For the Hybrid-TTN, the percentage of
not-adapted channels by layer is defined by a linear function starting at 0%
at the first BatchNorm layer and ending at 100% at the last layer. The step of
this linear function changes by model (e.g., Resnet-26 has more BatchNorm
layers than Resnet-18), but it does not introduce any hyperparameters. Our
method is hyperparameter free and appplied on all datasets and settings in the
same way.

Target Label Distributions In order to analyze label distribution shifts we
consider two approaches for constructing class imbalanced target
distributions, while we keep the source training balanced. $N$-class refers to
randomly selecting $N$ classes from the set of all classes and constructing a
batch with the specified $N$ classes. We also use a more natural imbalanced
dataset simulation construction technique popular in the federated learning
literature [23] which samples from a dirichlet distribution with parameter
$\alpha$. Smaller $\alpha$ will yield more sample batches with higher
imbalances. This allows us to mimic a more natural scenario where the label
distribution is not only varying in classes but proportions. In Liver
Ultrasound, we use the original target distribution, which is different from
the original source distribution, alongside a balanced distribution, and
severely imbalanced cases.

Figure 4: We compute the BN statistics for all-class and multiple 1-class
cases and compare agreement on channels which are most adapted (as measured by
top-10% Wasserstein distance). The percentage of overlapping channels that
presents the highest Wasserstein distance (between source and adapted models)
in both tasks is represented by the blue bars, with orange bars representing
the percentage of channels among those with the highest Wassertein distance
that are not the same between the two evaluated cases.

###  4.1 Label Distribution Shift Impact on TTN

Figure 5: CIFAR-10 evaluations on multiple label shifted distributions and
covariate shifts (corruptions) with different degrees of label imbalance. We
show the source model accuracy and the improvement (or degradation) with our
proposed method and baseline. We observe that the proposed method provides
benefits when there is no covariate shift, while avoiding catastrophic
failures and allowing benefits over source when there are label distribution
shifts.

We first illustrate the potential pitfalls of TTN on several examples of label
distribution shift. Using the CIFAR-10 dataset we show the effect on TTN of an
extreme version of label distribution shift in Figure 1. Specifically, we
adapt on batches containing one class or all classes (as can be seen in the
label distributions shown in Figure 1). We observe in the adaptation of one
class batches that the accuracy of TTN degrades severely compared to the
source model in cases where there is a covariate shift (application of
Gaussian noise to the input) but also in the absence of covariate shift. The
latter observation is also confirmed in the Appendix for ultrasound data where
we modify the target label proportion of the test data coming from the same
distribution (but without adding noise). This catastrophic failure, even
without covariate shift, is clearly undesired when operating in a realistic
deployment environment where the test set label distribution is unknown and
can vary for each incoming batch.

Furthermore, we can observe from the per-class performance evaluations of the
adapted models that the majority class is disproportionately affected by TTN,
and that classes underrepresented (or not present) will, unintuitively,
experience less degradation or even slightly improved performance compared to
source model. We can obtain intuition about this latter behavior by
considering a strong classification model where representations from each
class will be well separated and clustered in the final representation layer
(see Figure 3). Note that this is an assumption that is correct for most
modern neural network architectures, as it is a consequence of the neural
collapse phenomenon first described in [32]. A biased BatchNorm computation
will compute the mean of a specific label cluster and re-center the data by
this mean. Now, consider a neural network layer whose activations are tightly
clustered, such that each cluster corresponds to one label. Further, assume
that the different cluster centers occupy approximately orthogonal directions
in this space. As mentioned above, such quasi-orthogonality is not an uncommon
property in high dimensions [32]. In this setting, linear decision directions
would align with cluster means. Subtracting a cluster mean instead of a class-
balanced batch mean would then significantly move the label cluster along its
decision direction and will likely lead to some examples passing below the
classification threshold. Other classes will be barely affected due to the
orthogonality of the cluster centers and hence of the decision boundaries.

Figure 6: ImageNet-1K evaluations on multiple label shifted distributions and
covariate shifts (corruptions) with different degrees of label imbalance. We
observe that the proposed method provides benefits when there is covariate
shift, while avoiding catastrophic failures when there are label distribution
shifts.

###  4.2 Label Sensitive Channel Level Adaptation

As mentioned previously, many works in deep learning have shown that
supervised trained neural networks become more specialized as depth of a
network increases [52, 2, 31] and that classes progressively become more
separated with depth. This suggests the later layers are more likely to have
channels which are very sensitive to specific classes.

#### Adaptation of Early Layers

To further confirm that depth can have an effect on the behavior of TTN and
taking into account that earlier layers tend to be less specialized, we now
consider only adapting earlier parts of the network and study this effect on
the classification performance under label distribution shift. We perform
experiments on layer-limited adaptation for a Resnet-26 model on CIFAR-10 both
with and without noise. Our results are shown in in Figure 2. Specifically,
for each index on the x-axis we perform test-time normalization of the model
up to this layer. Consequently, the extremes of the x-axis (0 and, here, 30)
correspond to Source and TTN models respectively. First, we observe that on
the non-corrupted train data the performance of class-imbalanced data degrades
gradually at first and increasingly faster towards the later layers. This
suggests that later layers can cause a large degradation. Secondly, for
corrupted data we observe that adapting up to earlier layers can allow enough
label distribution invariance to provide benefits under covariate shift. For
example, models that are adapted up to layers 8 through 15 can perform much
better than source models under covariate shift but without label distribution
shift, while also being able to perform much better than fully adapted and
often source models in various label distribution shift scenarios. This
suggests limiting adaptation in later layers can be beneficial. On the other
hand, it is not obvious a priori which layer should be adapted. Furthermore,
different classes can have more fine-grained effects, for example, some
classes may lead to over-sensitivity in certain neurons or channels in a layer
than others.

Adaptive Channel Selection We now study how different class distributions
affect the channels that are most adapted. We use a similar setup as in the
previous example and consider within a layer the Wasserstein distance as a
metric for ranking the channels which would be most changed by TTN adaptation.
Subsequently, we take the channels in the top 10% for each layer and compare
them for the cases of 1-class vs all-class adaptation, and 1-class vs another.
Specifically, we determine how many channels in the top adapted channels for
each batch (single classes, all classes) overlap. The results are given in
Figure 4. We see from the first diagram that the channels that change the most
when adapting to just 1-class versus all-class tend to have more agreement in
the earlier layers, further supporting that later layers have many more
channels sensitive to label distribution shifts.

However, we can see if we compare two different classes (e.g., Plane vs Cat)
that the adaptation can vary substantially, throughout the network and even
towards the last layers. This suggest that depending on the class
distribution, different channels may be more sensitive to different classes.
This supports our dynamic strategy that selects the most sensitive channels to
remove from the adaptation set.

Figure 7: Evaluation on ultrasound data. Binary classification, subjects with
and without fatty liver disease. Models were trained on one site and are
evaluated on a publicly available dataset ([6]). We show evaluations on
targets with shifts in label distribution. We observe that the proposed method
is able to control the degradation in performance under severe label
distribution shift.

###  4.3 Evaluating Hybrid-TTN

We now use Hybrid-TTN method on a variety of target datasets, covariate
shifts, and label distribution shifts. We demonstrate that Hybrid-TTN can
provide a good trade-off in being able to adapt to covariate shift without
experiencing catastrophic failure due to label distribution shift. Our results
are shown in Figures 5, 6, and 7, and detailed tables are available in the
Appendix. Here we demonstrate various degrees of covariate and label
distribution shift, and the gain or loss as compared to the source model
performance.

|  Ranking ($\downarrow$)
---|---
Technique | CIFAR-10 | ImageNet-1K | Ultrasound
Source | 2.0 | 2.8 | 5.0
TTN | 4.5 | 4.4 | 3.5
TENT | 3.0 | 2.8 | 3.3
LAME | 6.0 | 6.0 | 6.0
DELTA | 4.5 | 4.3 | 3.0
Hybrid-TTN | 1.0 | 2.3 | 1.5
Table 1: Median ranking for each technique across the three different datasets. Lower is better. Across multiple distributions and degrees of corruption, our proposed method offers the best overall performance in multiple tasks and is the only TTA method to show consistent utility from adaptation. | Covariate Shift
---|---
|  Accuracy (% or $\Delta$%)
| Original | Corruption-1 | Corruption-2 | Corruption-3 | Corruption-4 | Corruption-5
Source | 95.0 ± 2.4 | 89.2 ± 9.9 | 83.9 ± 12.5 | 78.5 ± 16.5 | 71.2 ± 20.5 | 59.2 ± 26.5
2-7[2pt/2pt] TTN | -75.1 ± 3.9 | -70.2 ± 9.7 | -65.5 ± 12.0 | -60.5 ± 16.0 | -53.7 ± 19.7 | -42.2 ± 25.9
Hybrid-TTN | -8.0 ± 3.1 | -9.2 ± 5.7 | -7.4 ± 7.9 | -5.1 ± 10.8 | -1.7 ± 14.9 | +3.8 ± 22.1
2-7[2pt/2pt]+ Random Channel Scores | -21.9 ± 6.9 | -25.7 ± 11.6 | -22.6 ± 14.3 | -18.4 ± 18.3 | -13.2 ± 22.6 | -5.2 ± 28.1
\+ Uniform Prior Class Prior | -14.1 ± 4.5 | -15.4 ± 6.6 | -13.8 ± 8.5 | -11.1 ± 11.8 | -7.4 ± 15.5 | -1.3 ± 22.6
\+ Oracle Class Prior | -7.8 ± 3.2 | -7.5 ± 5.6 | -5.5 ± 7.6 | -2.8 ± 11.0 | +1.6 ± 15.2 | +8.1 ± 22.1
Table 2: Ablations. We use CIFAR-10 with batches from a single class and
different degrees of covariate shift. We show that our Wassertstein scoring
and class prior estimates are both beneficial to Hybrid-TTN.

For CIFAR-10 and ImageNet-1K we observe that Hybrid-TTN is able to control the
degradation in performance under a variety of label distribution shifts in the
absence of covariate shift, while TTN, TENT, LAME, and Delta can present
severe decreases in performance depending on the label distribution. When
corruptions are applied, Hybrid-TTN is able to bring benefits (substantially
improve on source model) for the original distribution. Unlike other methods,
it is able to handle the label distribution shift, in many cases avoiding
catastrophic failure, and in a variety of combinations of severe label and
covariate shift improving over the source model. We observe that LAME, which
is designed for the label distribution shift problem, is able to perform well
in the severes case of batches with a single class, but performs poorly in the
cases that the data is balanced.

We also evaluate on a realistic ultrasound dataset, in Figure 7. Under a shift
to a target dataset from a completely different site, we first observe that
most methods can maintain or improve the performance compared to the source
model in both original distribution and the balanced scenarios, the exception
being LAME. This can be compared to the balanced distributions of the results
for CIFAR-10-C, where the same methods achieved good performance when dealing
with several levels of corruption.

When we consider more severe imbalance scenarios, such as only using patients
without fatty liver and only patients with fatty liver, we see varying
results. Hybrid-TTN consistently outperforms the source model in the case of
healthy livers, but in the fatty livers case it can only surpass the source
model when severe corruption is present. LAME is consistently near 100%
accuracy in the fatty liver scenario and 0% accuracy in the healthy liver
scenario, indicating that the method might be biased towards the training
distribution. The other methods are stable across levels of corruption but
have diminishing returns.

Trade-offs of TTA We emphasize that the goal of Hybrid-TTN is to avoid severe
failures allowing a user to deploy TTA without concern that it might be
counterproductive compared to using just the source model. To illustrate this
we also rank the evaluated methods within each dataset for all scenarios
(label shift, covariate shift) and compute the median rank. The results are
presented in Table 1. We first observe that when viewed under this evaluation,
the source model for CIFAR-10 and ImageNet-1K outperforms all the other TTA
methods, indeed suggesting that TTA must be applied with extreme caution. To
illustrate this point, we can observe that LAME can result in high accuracy
increase in specific scenarios, but not on most cases. Similarly, other
techniques are more suited to specific use-cases. On the other hand, Hybrid-
TTN is the only TTA method able to obtain a substantially better median rank
compared to the source model, suggesting it is conservative while providing
benefits under covariate shift making it a useful method for real-world
deployment.

###  4.4 Ablation of Channel Selection

We perform an ablation study aimed at investigating the efficacy of the
different elements of the channel selection strategy within Hybrid-TTN.

Depth Prior Only We first consider the importance of the scoring function
compared to the linear decaying thresholding. We denote this ablation the
HybridTTN+Random Channel Scores. Specifically we replace the scores used for
thresholding in Algorithm 2 (line 10) with random scores. This decouples the
scoring obtained from Algorithm 1, with the prior on depth wise adaptation. We
observe that we obtain substantial improvement over just standard TTN but that
the performance degrades severely compared to our HybridTTN.

Class Prior We now consider the importance of the pseudo-labeling step used to
obtain a class prior from the target data described in Section 3.
Specifically, we evaluate the performance of just using a uniform prior, we
observe this improves the performance from the random channel scores, but
substantially underperforms Hybrid-TTN. Finally we also show the performance
of an oracle class prior, where the exact target data label distribution is
given. We note that for none or low corruption cases this gives performance
comparable to Hybrid-TTN, further affirming our approach. As the corruption
becomes more severe the performance gap between the oracle and our method
increases.

##  5 Conclusions

We have studied a popular batch-level Test-time Adaptation method in the
context of label distribution shift. We observed that in realistic scenarios
where batches at deployment time have label distribution shifts, this method
can fail catastrophically. We proposed a direction for solving this problem to
keep the benefits of adaptation without risking catastrophic failure due to
label shift. Moreover, our results indicate that the challenge of domain
adaptation in the biomedical field, specifically in diagnosis of fatty liver
using grayscale ultrasound images, can be addressed by the proposed Hybrid-TTN
method, reducing risks in the deployment of a trained framework in different
populations.

## References

  * Ba et al. [2016] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization.  _CoRR_ , abs/1607.06450, 2016.
  * Belilovsky et al. [2019] Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon.  Greedy layerwise learning can scale to imagenet.  In _International conference on machine learning_ , pages 583–593. PMLR, 2019.
  * Blaivas et al. [2022] Michael Blaivas, Laura N Blaivas, and James W Tsung.  Deep learning pitfall: Impact of novel ultrasound equipment introduction on algorithm performance and the realities of domain adaptation.  _Journal of Ultrasound in Medicine_ , 41(4):855–863, 2022.
  * Boudiaf et al. [2022] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto.  Parameter-free online test-time adaptation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8344–8353, 2022.
  * Burns and Steinhardt [2021] Collin Burns and Jacob Steinhardt.  Limitations of post-hoc feature alignment for robustness.  _CoRR_ , abs/2103.05898, 2021.
  * Byra et al. [2018] Michał Byra, Grzegorz Styczynski, Cezary Szmigielski, Piotr Kalinowski, Łukasz Michałowski, Rafał Paluszkiewicz, Bogna Ziarkiewicz-Wróblewska, Krzysztof Zieniewicz, Piotr Sobieraj, and Andrzej Nowicki.  Transfer learning with deep convolutional neural network for liver steatosis assessment in ultrasound images.  _International Journal of Computer Assisted Radiology and Surgery_ , 13(12):1895–1903, 2018.
  * Chen et al. [2020] Chaoqi Chen, Zebiao Zheng, Xinghao Ding, Yue Huang, and Qi Dou.  Harmonizing transferability and discriminability for adapting object detectors.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Chen et al. [2022] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi.  Contrastive test-time adaptation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 295–305, 2022.
  * Deng et al. [2009a] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009a.
  * Deng et al. [2009b] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In _Computer Vision and Pattern Recognition_ , 2009b.
  * Ding et al. [2023] Hao Ding, Changchang Sun, Hao Tang, Dawen Cai, and Yan Yan.  Few-shot medical image segmentation with cycle-resemblance attention.  In _2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 2487–2496, 2023.
  * Frankle et al. [2021] Jonathan Frankle, David J. Schwab, and Ari S. Morcos.  Training batchnorm and only batchnorm: On the expressive power of random features in cnns.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Hendrycks and Dietterich [2019a] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  In _International Conference on Learning Representations_ , 2019a.
  * Hendrycks and Dietterich [2019b] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  _arXiv preprint arXiv:1903.12261_ , 2019b.
  * Hu et al. [2021] Xuefeng Hu, Mustafa Gökhan Uzunbas, Sirius Chen, Rui Wang, Ashish Shah, Ramakant Nevatia, and Ser-Nam Lim.  Mixnorm: Test-time adaptation through online normalization estimation.  _ArXiv_ , abs/2110.11478, 2021.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_ , pages 448–456. JMLR.org, 2015.
  * Khurana et al. [2021] Ansh Khurana, Sujoy Paul, Piyush Rai, Soma Biswas, and Gaurav Aggarwal.  SITA: Single Image Test-time Adaptation.  _arXiv e-prints_ , art. arXiv:2112.02355, 2021.
  * Kim et al. [2022] Jin Kim, Jiyoung Lee, Jungin Park, Dongbo Min, and Kwanghoon Sohn.  Pin the memory: Learning to generalize semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4350–4360, 2022.
  * Krizhevsky [2009] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, 2009.
  * Krizhevsky [2012] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  _University of Toronto_ , 2012.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2012.
  * Li et al. [2021] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He.  Federated learning on non-iid data silos: An experimental study.  _arXiv preprint arXiv:2102.02079_ , 2021.
  * Li et al. [2016] Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou.  Revisiting batch normalization for practical domain adaptation.  _arXiv preprint arXiv:1603.04779_ , 2016.
  * Liang et al. [2020] Jian Liang, Dapeng Hu, and Jiashi Feng.  Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation.  In _Proceedings of the 37th International Conference on Machine Learning_ , pages 6028–6039. PMLR, 2020.
  * Liang et al. [2023] Jian Liang, Ran He, and Tieniu Tan.  A comprehensive survey on test-time adaptation under distribution shifts.  _arXiv preprint arXiv:2303.15361_ , 2023.
  * Lim et al. [2023] Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi.  TTN: A domain-shift aware batch normalization in test-time adaptation.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net, 2023.
  * Lipton et al. [2018] Zachary C. Lipton, Yu-Xiang Wang, and Alexander J. Smola.  Detecting and correcting for label shift with black box predictors.  In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , pages 3128–3136. PMLR, 2018.
  * Loshchilov and Hutter [2016] Ilya Loshchilov and Frank Hutter.  SGDR: stochastic gradient descent with restarts.  _CoRR_ , abs/1608.03983, 2016.
  * Nado et al. [2020] Zachary Nado, Shreyas Padhy, D. Sculley, Alexander D’Amour, Balaji Lakshminarayanan, and Jasper Snoek.  Evaluating prediction-time batch normalization for robustness under covariate shift.  _CoRR_ , abs/2006.10963, 2020.
  * Oyallon [2017] Edouard Oyallon.  Building a regular decision boundary with deep networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5106–5114, 2017.
  * Papyan et al. [2020] Vardan Papyan, X. Y. Han, and David L. Donoho.  Prevalence of neural collapse during the terminal phase of deep learning training.  _Proceedings of the National Academy of Sciences_ , 117(40):24652–24663, 2020.
  * Park et al. [2023] Sunghyun Park, Seunghan Yang, Jaegul Choo, and Sungrack Yun.  Label shift adapter for test-time adaptation under covariate and label shifts.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 16421–16431, 2023.
  * Piva et al. [2023] Fabrizio J. Piva, Daan de Geus, and Gijs Dubbelman.  Empirical generalization study: Unsupervised domain adaptation vs. domain generalization methods for semantic segmentation in the wild.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 499–508, 2023.
  * Salimans and Kingma [2016] Tim Salimans and Diederik P. Kingma.  Weight normalization: A simple reparameterization to accelerate training of deep neural networks.  In _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain_ , page 901, 2016.
  * Santurkar et al. [2018] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry.  How does batch normalization help optimization?  In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , pages 2488–2498, 2018.
  * Schneider et al. [2020] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge.  Improving robustness against common corruptions by covariate shift adaptation.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.
  * Sun et al. [2022] Wujie Sun, Qi Chen, Can Wang, Deshi Ye, and Chun Chen.  Source-free unsupervised domain adaptation in imbalanced datasets.  In _2022 5th International Conference on Data Science and Information Technology (DSIT)_ , pages 1–6, 2022.
  * Sun et al. [2020a] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt.  Test-time training with self-supervision for generalization under distribution shifts.  In _International conference on machine learning_ , pages 9229–9248. PMLR, 2020a.
  * Sun et al. [2020b] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt.  Test-time training with self-supervision for generalization under distribution shifts.  In _Proceedings of the 37th International Conference on Machine Learning_ , pages 9229–9248. PMLR, 2020b.
  * Ulyanov et al. [2016] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky.  Instance normalization: The missing ingredient for fast stylization.  _CoRR_ , abs/1607.08022, 2016.
  * Vianna et al. [2023] Pedro Vianna, Sara-Ivana Calce, Pamela Boustros, Cassandra Larocque-Rigney, Laurent Patry-Beaudoin, Yi Hui Luo, Emre Aslan, John Marinos, Talal M. Alamri, Kim-Nhien Vu, Jessica Murphy-Lavallée, Jean-Sébastien Billiard, Emmanuel Montagnon, Hongliang Li, Samuel Kadoury, Bich N. Nguyen, Shanel Gauthier, Benjamin Therien, Irina Rish, Eugene Belilovsky, Guy Wolf, Michaël Chassé, Guy Cloutier, and An Tang.  Comparison of radiologists and deep learning for us grading of hepatic steatosis.  _Radiology_ , 309(1), 2023.
  * VS et al. [2023] Vibashan VS, Poojan Oza, and Vishal M Patel.  Towards online domain adaptive object detection.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 478–488, 2023.
  * Wang et al. [2021a] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell.  Tent: Fully test-time adaptation by entropy minimization.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021a.
  * Wang et al. [2021b] Yan Wang, Yangqin Feng, Lei Zhang, Zizhou Wang, Qing Lv, and Zhang Yi.  Deep adversarial domain adaptation for breast cancer screening from mammograms.  _Medical Image Analysis_ , 73:102147, 2021b.
  * Wasserthal et al. [2023] Jakob Wasserthal, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, Daniel T. Boll, Joshy Cyriac, Shan Yang, Michael Bach, and Martin Segeroth.  Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images.  _Radiology: Artificial Intelligence_ , 5(5):e230024, 2023.
  * Wu and He [2018] Yuxin Wu and Kaiming He.  Group normalization.  In _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII_ , pages 3–19. Springer, 2018.
  * Yin et al. [2022] Yifang Yin, Wenmiao Hu, An Tran, Hannes Kruppa, Roger Zimmermann, and See-Kiong Ng.  A context-enriched satellite imagery dataset and an approach for parking lot detection.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 1371–1380, 2022.
  * You et al. [2021] Fuming You, Jingjing Li, Lei Zhu, Zhi Chen, and Zi Huang.  Domain adaptive semantic segmentation without source data.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , page 3293–3302, New York, NY, USA, 2021. Association for Computing Machinery.
  * Yu et al. [2022] Alice C. Yu, Bahram Mohajer, and John Eng.  External validation of deep learning algorithms for radiologic diagnosis: A systematic review.  _Radiology: Artificial Intelligence_ , 4(3):e210064, 2022.
  * Zagoruyko and Komodakis [2016] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  _CoRR_ , abs/1605.07146, 2016.
  * Zeiler and Fergus [2014] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In _European conference on computer vision_ , pages 818–833. Springer, 2014.
  * Zhao et al. [2023] Bowen Zhao, Chen Chen, and Shu-Tao Xia.  Delta: degradation-free fully test-time adaptation.  _arXiv preprint arXiv:2301.13018_ , 2023.

Supplementary Material

##  A Appendix

This appendix includes:

  * •

Training details (Sec. A.1)

  * •

Adaptation details (Sec. A.2)

  * •

Ultrasound failure case without domain shift (Sec. A.3)

  * •

Detailed tables for CIFAR-10-C, ImageNet-1K-C and Ultrasound results (Sec.
A.4)

###  A.1 Training details

On CIFAR-10 we train a Resnet-26 model as defined in [13]. We use an SGD
optimizer with a batch size of 128. An initial learning rate set to 0.1 is
used in combination with a cosine annealing schedule [29] trained over 200
epochs. Weight decay set to 5e-4 is used along with momentum set to 0.9 [51].
Standard augmentation uses random crop of size 32 with 4 padding, and random
horizontal flips.

For the Liver Ultrasound data, we train a Resnet-18 model with similar
configurations, using a SGD optimizer with learning rate set to 0.001 and a
batch size of 32. Differences in the models configurations can be explained by
the computational constraints of the Hospital Research Center where the images
are secured.

For ImageNet-1K we use a pre-trained Resnet18 model.

###  A.2 Adaptation details

For TTN, we use the original configuration, resetting batch normalization
layers (BatchNorm) running stats. The difference between TTN and Hybrid-TTN is
that for the former we adapt all channels in all BatchNorm during test-time,
without our proposed channel selection.

For TENT, original configuration was followed. In addition, the adaptation for
each batch was repeated for 10 iterations. In implementing LAME, we adhered to
the default configuration provided in the officially released code for
achieving best model performance. As per the specified configuration, we chose
the radial basis function kernel for the affinity matrix with K-nearest
neighbors and $\sigma$ set to 5 and 1 respectively. We also used the same
hyperparameters for optimizer as specified in the mentioned configuration.

To maintain consistency in our comparative analysis, we executed the
experiments we adapted DELTA to our setup in which we had one test batch of a
specific size. Notably, DELTA is tailored for online setups, excelling in
processing multiple batches within a test stream.

###  A.3 Example of failure case in Ultrasound

In Table 3 we note a significant decline in TTN accuracy compared to the
source model, not only in instances with covariate shifts but also without any
covariate shift. This phenomenon can be seen in the ultrasound data, where we
adjust the proportion of target labels in the test data sourced from the same
distribution as the source data.

| Accuracy (%)
---|---
Labels proportion | Source | TTN
Original (16%/84%) | 82.3 | 83.1
0%/100% | 92.2 | 84.4
50%/50% | 71.9 | 67.2
100%/0% | 51.6 | 21.9
Table 3: Illustration of failure case on binary classification of medical
ultrasound data (detecting presence of fatty liver). If TTN is deployed on
target data from the same domain (same hospital, demographics) with the
original training proportions (16%/84%) performance is maintained or slightly
improved. On the other hand, if the target batch presented has a label
distribution shift we see substantial degradation.

###  A.4 Detailed results

Below, we present the tables for all of our experiments.

| Covariate Shift
---|---
|  Accuracy (% or $\Delta$%)
Label Distribution Shift | Original | Corruption-1 | Corruption-2 | Corruption-3 | Corruption-4 | Corruption-5
| Source | 95.0 ± 2.4 | 89.2 ± 9.9 | 83.9 ± 12.5 | 78.5 ± 16.5 | 71.2 ± 20.5 | 59.2 ± 26.5
2-8[2pt/2pt] | TTN | -75.1 ± 3.9 | -70.2 ± 9.7 | -65.5 ± 12.0 | -60.5 ± 16.0 | -53.7 ± 19.7 | -42.2 ± 25.9
1 class | TENT | -75.2 ± 2.2 | -70.2 ± 2.6 | -65.5 ± 2.7 | -60.4 ± 2.8 | -53.6 ± 2.7 | -42.1 ± 2.5
| LAME | +2.4 ± 0.9 | +8.2 ± 1.1 | +10.6 ± 17.2 | +13.6 ± 22.4 | +13.3 ± 33.9 | +7.8 ± 45.7
| DELTA | -75.1 ± 2.3 | -70.2 ± 2.5 | -65.5 ± 2.7 | -60.5 ± 2.8 | -53.7 ± 2.7 | -42.2 ± 2.6
| Hybrid-TTN | -8.0 ± 3.1 | -9.2 ± 5.7 | -7.4 ± 7.9 | -5.1 ± 10.8 | -1.7 ± 14.9 | +3.8 ± 22.1
| Source | 95.2 ± 2.1 | 87.9 ± 10.3 | 82.0 ± 12.6 | 75.9 ± 16.7 | 68.1 ± 20.2 | 55.4 ± 22.7
2-8[2pt/2pt] | TTN | -32.4 ± 4.3 | -28.1 ± 8.4 | -24.2 ± 10.5 | -19.4 ± 14.2 | -13.4 ± 17.3 | -3.5 ± 21.3
3 classes | TENT | -31.8 ± 4.6 | -27.4 ± 6.0 | -23.5 ± 6.5 | -18.7 ± 7.0 | -12.7 ± 7.8 | -2.7 ± 7.6
| LAME | -0.8 ± 2.0 | +0.6 ± 13.1 | -3.2 ± 22.0 | -8.5 ± 30.5 | -10.6 ± 34.2 | -11.0 ± 33.6
| DELTA | -32.4 ± 4.6 | -28.1 ± 5.9 | -24.2 ± 6.4 | -19.4 ± 7.0 | -13.4 ± 7.9 | -3.5 ± 7.6
| Hybrid-TTN | -3.3 ± 1.0 | -0.9 ± 5.3 | +2.0 ± 7.4 | +5.4 ± 10.1 | +10.0 ± 13.1 | +16.9 ± 18.1
| Source | 94.9 ± 1.2 | 88.5 ± 8.8 | 82.6 ± 10.3 | 76.5 ± 14.0 | 68.7 ± 17.3 | 56.5 ± 20.7
2-8[2pt/2pt] | TTN | -15.2 ± 2.0 | -13.0 ± 5.7 | -9.6 ± 7.2 | -4.9 ± 10.5 | +0.1 ± 13.4 | +9.0 ± 19.1
5 classes | TENT | -14.8 ± 2.8 | -12.3 ± 5.3 | -9.0 ± 6.2 | -4.2 ± 7.1 | +0.8 ± 9.1 | +9.8 ± 9.0
| LAME | -2.1 ± 2.5 | -3.8 ± 13.0 | -9.9 ± 23.9 | -18.3 ± 31.2 | -22.4 ± 31.5 | -22.3 ± 26.6
| DELTA | -15.2 ± 2.8 | -13.0 ± 5.2 | -9.6 ± 6.3 | -4.9 ± 7.3 | +0.1 ± 9.1 | +9.0 ± 9.2
| Hybrid-TTN | -2.2 ± 1.0 | +0.1 ± 4.7 | +3.0 ± 6.0 | +7.3 ± 9.1 | +11.9 ± 12.0 | +19.1 ± 17.3
| Source | 93.5 ± 0.9 | 86.6 ± 9.3 | 80.2 ± 11.1 | 73.8 ± 14.7 | 65.6 ± 17.9 | 53.1 ± 20.1
2-8[2pt/2pt] | TTN | -1.6 ± 0.6 | +2.2 ± 5.5 | +6.3 ± 7.0 | +10.8 ± 9.9 | +16.4 ± 12.9 | +25.2 ± 17.7
10 classes | TENT | -1.5 ± 1.3 | +2.4 ± 4.3 | +6.5 ± 5.2 | +11.1 ± 6.7 | +16.8 ± 8.9 | +25.6 ± 9.6
(Original) | LAME | -4.4 ± 1.3 | -14.1 ± 21.8 | -25.9 ± 28.0 | -33.3 ± 28.6 | -36.7 ± 25.1 | -35.9 ± 11.6
| DELTA | -1.6 ± 1.3 | +2.2 ± 4.4 | +6.3 ± 5.4 | +10.8 ± 6.9 | +16.4 ± 9.1 | +25.2 ± 9.8
| Hybrid-TTN | -1.0 ± 0.6 | +2.3 ± 5.0 | +6.1 ± 6.5 | +10.4 ± 9.3 | +15.6 ± 12.2 | +23.1 ± 17.0
| Source | 94.2 ± 1.8 | 87.5 ± 9.3 | 80.9 ± 12.6 | 76.0 ± 14.8 | 65.3 ± 19.5 | 53.7 ± 22.2
2-8[2pt/2pt] | TTN | -22.4 ± 6.8 | -18.6 ± 8.0 | -15.0 ± 12.1 | -11.5 ± 14.8 | -3.4 ± 15.8 | +6.5 ± 19.8
Dirichlet | TENT | -22.3 ± 6.8 | -19.3 ± 8.3 | -14.2 ± 11.1 | -9.3 ± 14.7 | -2.7 ± 18.4 | +6.9 ± 22.5
($\alpha=0.01$) | LAME | -1.5 ± 1.5 | -1.2 ± 7.0 | -8.2 ± 17.1 | -14.0 ± 21.3 | -19.2 ± 20.5 | -16.8 ± 16.4
| DELTA | -22.9 ± 6.7 | -20.1 ± 8.3 | -15.0 ± 11.1 | -10.0 ± 14.7 | -3.5 ± 18.2 | +6.1 ± 22.5
| Hybrid-TTN | -2.6 ± 1.1 | +0.2 ± 4.3 | +3.6 ± 7.3 | +6.6 ± 9.7 | +13.3 ± 14.2 | +20.3 ± 18.1
| Source | 94.5 ± 1.4 | 86.9 ± 10.0 | 81.3 ± 11.3 | 74.3 ± 15.5 | 67.4 ± 17.0 | 53.4 ± 21.9
2-8[2pt/2pt] | TTN | -10.0 ± 3.6 | -7.0 ± 7.4 | -3.1 ± 7.8 | +1.5 ± 12.1 | +6.7 ± 14.1 | +16.6 ± 19.1
Dirichlet | TENT | -9.9 ± 3.4 | -7.4 ± 7.2 | -1.9 ± 8.5 | +2.1 ± 11.5 | +7.4 ± 14.7 | +17.8 ± 18.8
($\alpha=0.5$) | LAME | -3.6 ± 1.4 | -6.0 ± 7.5 | -18.6 ± 18.3 | -25.5 ± 19.8 | -28.2 ± 20.7 | -29.3 ± 17.7
| DELTA | -10.5 ± 3.6 | -7.9 ± 7.2 | -2.4 ± 8.4 | +1.4 ± 11.3 | +6.8 ± 14.7 | +17.1 ± 18.7
| Hybrid-TTN | -1.8 ± 0.9 | +1.4 ± 4.8 | +4.8 ± 6.2 | +9.4 ± 10.5 | +14.2 ± 12.2 | +22.0 ± 18.0
| Source | 94.6 ± 0.9 | 87.3 ± 9.5 | 80.9 ± 10.9 | 73.6 ± 15.0 | 65.7 ± 17.5 | 53.9 ± 20.0
2-8[2pt/2pt] | TTN | -1.9 ± 0.8 | +1.9 ± 5.5 | +6.0 ± 6.7 | +11.5 ± 10.4 | +16.7 ± 12.6 | +24.8 ± 17.5
Dirichlet | TENT | -1.7 ± 0.8 | +2.1 ± 5.5 | +6.2 ± 6.8 | +11.0 ± 10.0 | +16.8 ± 13.1 | +25.7 ± 18.0
($\alpha=100$) | LAME | -5.5 ± 1.1 | -14.5 ± 15.1 | -26.1 ± 20.5 | -33.4 ± 19.5 | -37.8 ± 17.7 | -35.7 ± 17.7
| DELTA | -1.9 ± 0.8 | +1.9 ± 5.4 | +5.9 ± 6.7 | +10.7 ± 9.8 | +16.4 ± 12.9 | +25.2 ± 17.9
| Hybrid-TTN | -1.3 ± 0.6 | +2.1 ± 4.7 | +5.9 ± 6.4 | +10.9 ± 9.6 | +16.0 ± 12.1 | +23.0 ± 16.8
Table 4: CIFAR-10 evaluations on multiple label shifted distributions and covariate shifts (corruptions) with different degrees of label imbalance. We show the source model accuracy and the improvement (or degradation) as a delta accuracy. We observe that the proposed method provides benefits over source model when there is no covariate shift, while avoiding catastrophic failures and allowing benefits over source when there are label distribution shifts.  | Covariate Shift
---|---
|  Accuracy (% or $\Delta$%)
Label Distribution Shift | Original | Corruption-1 | Corruption-2 | Corruption-3 | Corruption-4 | Corruption-5
| Source | 70.8 ± 15.1 | 55.6 ± 21.3 | 44.6 ± 22.8 | 34.4 ± 23.6 | 22.8 ± 22.5 | 14.0 ± 19.5
2-8[2pt/2pt] | TTN | -70.0 ± 15.6 | -54.1 ± 21.4 | -43.3 ± 22.8 | -33.2 ± 23.6 | -21.9 ± 22.3 | -13.2 ± 19.3
1 class | TENT | -70.0 ± 1.0 | -53.8 ± 1.5 | -43.0 ± 1.7 | -32.8 ± 1.5 | -21.8 ± 1.5 | -13.5 ± 1.2
| LAME | 29.2 ± 0.0 | 40.8 ± 19.7 | 46.2 ± 29.4 | 41.8 ± 43.1 | 29.9 ± 50.2 | 17.3 ± 46.7
| DELTA | -70.0 ± 1.0 | -53.8 ± 1.6 | -43.1 ± 1.7 | -32.8 ± 1.5 | -21.8 ± 1.5 | -13.5 ± 1.3
| Hybrid-TTN | -22.4 ± 6.7 | -17.1 ± 11.4 | -15.3 ± 12.5 | -10.6 ± 13.0 | -6.2 ± 11.9 | -3.2 ± 11.1
| Source | 69.5 ± 5.9 | 51.9 ± 10.6 | 41.5 ± 12.8 | 32.2 ± 15.9 | 21.5 ± 15.8 | 13.1 ± 14.0
2-8[2pt/2pt] | TTN | -41.3 ± 4.5 | -30.5 ± 7.9 | -24.2 ± 9.7 | -17.7 ± 11.5 | -10.1 ± 10.8 | -5.0 ± 9.9
3 classes | TENT | -40.4 ± 3.5 | -30.4 ± 5.6 | -23.7 ± 5.9 | -17.2 ± 6.7 | -9.6 ± 7.1 | -4.7 ± 6.4
| LAME | 9.6 ± 26.1 | -9.5 ± 19.0 | -3.3 ± 14.3 | -0.6 ± 20.1 | -1.4 ± 17.8 | -2.8 ± 16.1
| DELTA | -41.6 ± 2.9 | -30.9 ± 5.4 | -24.2 ± 5.8 | -17.7 ± 6.6 | -10.0 ± 6.8 | -5.0 ± 6.2
| Hybrid-TTN | -7.2 ± 2.0 | -3.3 ± 7.1 | -1.8 ± 7.7 | +0.2 ± 7.7 | +2.3 ± 7.3 | +3.7 ± 6.7
| Source | 71.1 ± 5.3 | 53.7 ± 9.9 | 42.8 ± 13.2 | 32.7 ± 16.2 | 21.4 ± 15.8 | 13.2 ± 13.5
2-8[2pt/2pt] | TTN | -26.5 ± 4.4 | -19.3 ± 5.8 | -14.1 ± 8.4 | -8.9 ± 9.4 | -3.4 ± 8.7 | -0.0 ± 7.7
5 classes | TENT | -25.8 ± 4.1 | -18.6 ± 7.0 | -13.0 ± 8.0 | -7.7 ± 10.2 | -2.6 ± 10.4 | 0.7 ± 9.6
| LAME | -4.2 ± 38.6 | -30.0 ± 17.6 | -20.8 ± 14.2 | -13.3 ± 18.6 | -10.0 ± 10.0 | -7.7 ± 8.9
| DELTA | -26.6 ± 4.1 | -19.7 ± 7.0 | -14.0 ± 7.9 | -8.7 ± 10.0 | -3.3 ± 10.1 | +0.1 ± 9.3
| Hybrid-TTN | -5.2 ± 2.5 | -1.0 ± 4.7 | +0.4 ± 6.4 | +2.4 ± 7.0 | +4.5 ± 6.5 | +4.9 ± 6.0
| Source | 69.8 ± 3.4 | 50.7 ± 8.1 | 40.2 ± 10.9 | 31.0 ± 14.0 | 20.7 ± 14.0 | 13.0 ± 12.2
2-8[2pt/2pt] | TTN | -14.7 ± 0.7 | -6.2 ± 3.7 | -2.5 ± 5.6 | +0.9 ± 6.8 | +3.9 ± 6.7 | +5.0 ± 5.9
10 classes | TENT | -13.8 ± 3.7 | -5.5 ± 5.9 | -1.5 ± 8.0 | 2.1 ± 11.3 | 4.9 ± 11.8 | 5.7 ± 11.6
| LAME | -59.8 ± 0.0 | -41.3 ± 0.0 | -30.8 ± 1.8 | -23.8 ± 4.5 | -16.3 ± 5.0 | -10.3 ± 4.4
| DELTA | -14.7 ± 3.7 | -6.7 ± 6.0 | -2.7 ± 8.1 | +1.0 ± 11.3 | +3.9 ± 11.5 | +4.9 ± 11.2
| Hybrid-TTN | -2.9 ± 0.8 | +2.6 ± 3.3 | +3.5 ± 5.0 | +5.2 ± 6.0 | +5.9 ± 5.7 | +5.6 ± 5.3
| Source | 69.7 ± 1.6 | 52.6 ± 7.3 | 42.0 ± 10.3 | 33.1 ± 14.0 | 23.1 ± 14.3 | 14.9 ± 12.3
2-8[2pt/2pt] | TTN | -0.6 ± 0.6 | +5.9 ± 3.5 | +8.6 ± 5.2 | +10.4 ± 6.6 | +11.5 ± 6.4 | +10.7 ± 6.9
500 classes | TENT | -0.5 ± 2.3 | 6.1 ± 5.5 | 9.2 ± 8.2 | 11.0 ± 12.1 | 12.2 ± 13.8 | 11.4 ± 14.5
| LAME | -55.2 ± 28.0 | -51.9 ± 6.6 | -41.8 ± 0.2 | -32.8 ± 0.2 | -22.8 ± 0.2 | -14.7 ± 0.1
| DELTA | -0.6 ± 2.1 | +5.7 ± 5.8 | +8.8 ± 8.3 | +10.5 ± 12.2 | +11.7 ± 13.8 | +10.8 ± 14.4
| Hybrid-TTN | -0.2 ± 0.4 | +4.3 ± 3.1 | +5.6 ± 4.7 | +7.0 ± 5.8 | +7.5 ± 5.6 | +7.1 ± 5.9
| Source | 70.0 ± 2.2 | 53.0 ± 7.3 | 42.0 ± 10.7 | 33.2 ± 13.7 | 23.1 ± 14.4 | 14.7 ± 12.3
2-8[2pt/2pt] | TTN | -1.0 ± 0.9 | +5.3 ± 3.4 | +8.1 ± 5.0 | +10.0 ± 6.6 | +11.5 ± 6.3 | +10.7 ± 6.9
Dirichlet | TENT | -0.8 ± 0.9 | +6.0 ± 3.4 | +9.0 ± 5.2 | +10.8 ± 6.5 | +11.8 ± 7.0 | +10.6 ± 6.6
($\alpha=0.01$) | LAME | -67.1 ± 10.7 | -51.9 ± 7.0 | -40.8 ± 9.9 | -32.2 ± 13.7 | -22.9 ± 14.1 | -14.5 ± 12.7
| DELTA | -0.9 ± 0.9 | +5.6 ± 3.3 | +8.4 ± 5.2 | +10.2 ± 6.5 | +11.2 ± 7.0 | +10.1 ± 6.5
| Hybrid-TTN | -0.5 ± 0.8 | +4.4 ± 3.0 | +5.8 ± 4.5 | +7.0 ± 6.2 | +8.0 ± 5.6 | +7.4 ± 5.9
Table 5: ImageNet-1K evaluations on multiple label shifted distributions and covariate shifts (corruptions) with different degrees of label imbalance. We observe that the proposed method provides benefits over source model when there is covariate shift, while avoiding catastrophic failures when there are label distribution shifts. Note that we use a batch size of 500 images, and the highest amount of classes is limited by this number. | Covariate Shift
---|---
|  Accuracy (% or $\Delta$%)
Label Distribution Shift | Original | Corruption-1 | Corruption-2 | Corruption-3 | Corruption-4
| Source | 76.5 ± 2.7 | 74.8 ± 1.3 | 70.8 ± 3.1 | 65.3 ± 3.6 | 60.4 ± 5.1
2-7[2pt/2pt] | TTN | +5.6 ± 4.2 | +3.2 ± 3.2 | +3.5 ± 5.5 | +4.4 ± 6.7 | +5.0 ± 9.4
Original | TENT | +5.8 ± 4.2 | +3.8 ± 5.0 | 0.0 ± 6.1 | +3.5 ± 9.1 | +8.0 ± 4.3
(31:69) | LAME | -7.8 ± 2.7 | -4.9 ± 2.3 | -4.4 ± 2.9 | +2.8 ± 5.0 | +9.1 ± 3.4
| DELTA | +5.6 ± 4.2 | +3.2 ± 4.3 | -0.4 ± 6.0 | +3.0 ± 8.4 | +8.1 ± 4.1
| Hybrid-TTN | +0.9 ± 4.3 | -2.2 ± 3.5 | +3.5 ± 2.8 | +6.7 ± 4.9 | +10.5 ± 5.6
| Source | 65.8 ± 3.2 | 60.5 ± 3.0 | 59.8 ± 3.5 | 53.9 ± 5.9 | 48.0 ± 3.0
2-7[2pt/2pt] | TTN | +6.4 ± 4.9 | +6.1 ± 4.2 | +4.5 ± 6.0 | +5.5 ± 2.5 | +11.9 ± 4.6
Balanced | TENT | +6.9 ± 4.8 | +8.1 ± 5.2 | +6.6 ± 5.9 | +9.0 ± 6.8 | +15.1 ± 5.3
(50:50) | LAME | -15.8 ± 3.2 | -10.4 ± 2.4 | -8.5 ± 3.3 | -3.8 ± 5.1 | +3.5 ± 4.2
| DELTA | +6.4 ± 4.9 | +7.1 ± 4.9 | +6.6 ± 5.7 | +9.0 ± 6.7 | +14.8 ± 5.2
| Hybrid-TTN | +8.2 ± 3.9 | +9.1 ± 3.6 | +5.8 ± 5.4 | +8.8 ± 7.0 | +10.6 ± 5.0
| Source | 34.6 ± 4.9 | 23.1 ± 3.6 | 22.4 ± 5.2 | 24.2 ± 2.8 | 15.9 ± 4.7
2-7[2pt/2pt] | TTN | -11.9 ± 7.7 | +3.4 ± 4.1 | +4.1 ± 4.3 | +2.8 ± 4.4 | +9.9 ± 6.1
Healthy livers | TENT | -11.9 ± 8.1 | +2.4 ± 6.2 | +0.8 ± 5.7 | +2.6 ± 4.2 | +11.4 ± 4.7
(100:0) | LAME | -34.0 ± 5.2 | -23.0 ± 5.0 | -25.6 ± 3.8 | -24.5 ± 3.7 | -15.4 ± 5.1
| DELTA | -11.9 ± 7.7 | +2.2 ± 6.1 | +0.6 ± 5.5 | +2.4 ± 4.4 | +10.8 ± 5.2
| Hybrid-TTN | +29.0 ± 3.1 | +41.1 ± 5.0 | +19.4 ± 5.5 | +6.0 ± 6.9 | +5.4 ± 7.0
| Source | 96.6 ± 1.2 | 98.8 ± 1.3 | 94.8 ± 3.1 | 86.1 ± 2.6 | 79.9 ± 6.1
2-7[2pt/2pt] | TTN | -24.0 ± 3.7 | -24.6 ± 2.2 | -22.6 ± 3.3 | -12.8 ± 3.7 | -5.8 ± 8.5
Fatty livers | TENT | -24.0 ± 2.9 | -25.6 ± 2.4 | -22.9 ± 3.3 | -10.6 ± 4.9 | -7.8 ± 3.4
(0:100) | LAME | +3.4 ± 1.2 | +0.4 ± 0.6 | +5.6 ± 2.4 | +15.5 ± 4.4 | +19.2 ± 4.0
| DELTA | -24.0 ± 3.7 | -26.0 ± 3.0 | -23.0 ± 3.3 | -10.0 ± 4.8 | -7.6 ± 4.3
| Hybrid-TTN | -21.4 ± 4.1 | -25.0 ± 6.4 | -11.4 ± 4.7 | +2.1 ± 6.2 | +10.0 ± 8.8
Table 6: Evaluation on ultrasound data. Binary classification, subjects
without fatty liver or patients with fatty liver disease. Models were trained
on one site and are evaluated on a publicly available dataset ([6]). We show
evaluations on targets with shifts in label distribution and added speckle
noise. We observe that the proposed method is able to control the degradation
in performance under severe label distribution shift.