# EcoTTA: Memory-Efficient Continual Test-time Adaptation
via Self-distilled Regularization

Junha Song1,2 , Jungsoo Lee1, In So Kweon2, Sungha Choi1
1Qualcomm AI Research‡, 2KAIST
Work done during an internship at Qualcomm AI Research.Corresponding author. ‡
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.

###### Abstract

This paper presents a simple yet effective approach that improves continual
test-time adaptation (TTA) in a memory-efficient manner. TTA may primarily be
conducted on edge devices with limited memory, so reducing memory is crucial
but has been overlooked in previous TTA studies. In addition, long-term
adaptation often leads to catastrophic forgetting and error accumulation,
which hinders applying TTA in real-world deployments. Our approach consists of
two components to address these issues. First, we present lightweight meta
networks that can adapt the frozen original networks to the target domain.
This novel architecture minimizes memory consumption by decreasing the size of
intermediate activations required for backpropagation. Second, our novel self-
distilled regularization controls the output of the meta networks not to
deviate significantly from the output of the frozen original networks, thereby
preserving well-trained knowledge from the source domain. Without additional
memory, this regularization prevents error accumulation and catastrophic
forgetting, resulting in stable performance even in long-term test-time
adaptation. We demonstrate that our simple yet effective strategy outperforms
other state-of-the-art methods on various benchmarks for image classification
and semantic segmentation tasks. Notably, our proposed method with ResNet-50
and WideResNet-40 takes 86% and 80% less memory than the recent state-of-the-
art method, CoTTA.

##  1 Introduction

Figure 1: (a) Memory cost comparison between TTA methods. The size of
activations, not the parameters, is the primary memory bottleneck during
training. (b) CIFAR-C adaptation performance. We perform the continual online
adaptation on CIFAR-C dataset. The x- and y-axis are the average error of all
corruptions and the total memory consumption including the parameters and
activations, respectively. Our approach, EcoTTA, achieves the best results
while consuming the least amount of memory, where K is the model partition
factor used in our method.

Despite recent advances in deep learning [15, 24, 23, 22], deep neural
networks often suffer from performance degradation when the source and target
domains differ significantly [8, 43, 38]. Among several tasks addressing such
domain shifts, test-time adaptation (TTA) has recently received a significant
amount of attention due to its practicality and wide applicability especially
in on-device settings [65, 42, 32, 16]. This task focuses on adapting the
model to unlabeled online data from the target domain without access to the
source data.

Figure 2: Architecture for test-time adaptation. We illustrate TTA methods:
TENT [65], EATA [50], CoTTA [66], and Ours (EcoTTA). TENT and EATA update
multiple batch norm layers, in which large activations have to be stored for
gradient calculation. In CoTTA, an entire network is trained with additional
strategies for continual adaptation that requires a significant amount of both
memory and time. In contrast, our approach requires a minimum size of
activations by updating only a few layers. Also, stable long-term adaptation
is performed by our proposed regularization, named self-distilled
regularization.

While existing TTA methods show improved TTA performances, minimizing the
sizes of memory resources have been relatively under-explored, which is
crucial considering the applicability of TTA in on-device settings. For
example, several studies [66, 42, 9] update entire model parameters to achieve
large performance improvements, which may be impractical when the available
memory sizes are limited. Meanwhile, several TTA approaches update only the
batch normalization (BN) parameters [65, 50, 17] to make the optimization
efficient and stable However, even updating only BN parameters is not memory
efficient enough since the amount of memory required for training models
significantly depends on the size of intermediate activations rather than the
learnable parameters [4, 14, 69]. Throughout the paper, activations refer to
the intermediate features stored during the forward propagation, which are
used for gradient calculations during backpropagation. Fig. 1 (a) demonstrates
such an issue.

Moreover, a non-trivial number of TTA studies assume a stationary target
domain [65, 42, 9, 57], but the target domain may continuously change in the
real world (e.g., continuous changes in weather conditions, illuminations, and
location [8] in autonomous driving). Therefore, it is necessary to consider
long-term TTA in an environment where the target domain constantly varies.
However, there exist two challenging issues: 1) catastrophic forgetting [66,
50] and 2) error accumulation. Catastrophic forgetting refers to degraded
performance on the source domain due to long-term adaptation to target domains
[66, 50]. Such an issue is important since the test samples in the real world
may come from diverse domains, including the source and target domains [50].
Also, since target labels are unavailable, TTA relies on noisy unsupervised
losses, such as entropy minimization [19], so long-term continual TTA may lead
to error accumulation [75, 2].

To address these challenges, we propose memory-Efficient continual Test-Time
Adaptation (EcoTTA), a simple yet effective approach for 1) enhancing memory
efficiency and 2) preventing catastrophic forgetting and error accumulation.
First, we present a memory-efficient architecture consisting of frozen
original networks and our proposed meta networks attached to the original
ones. During the test time, we freeze the original networks to discard the
intermediate activations that occupy a significant amount of memory. Instead,
we only adapt lightweight meta networks to the target domain, composed of only
one batch normalization and one convolution block. Surprisingly, updating only
the meta networks, not the original ones, can result in significant
performance improvement as well as considerable memory savings. Moreover, we
propose a self-distilled regularization method to prevent catastrophic
forgetting and error accumulation. Our regularization leverages the preserved
source knowledge distilled from the frozen original networks to regularize the
meta networks. Specifically, we control the output of the meta networks not to
deviate from the one extracted by the original networks significantly.
Notably, our regularization leads to negligible overhead because it requires
no extra memory and is performed in parallel with adaptation loss, such as
entropy minimization.

Recent TTA studies require access to the source data _before model
deployments_ [42, 9, 34, 1, 40, 50]. Similarly, our method uses the source
data to warm up the newly attached meta networks for a small number of epochs
before model deployment. If the source dataset is publicly available or the
owner of the pre-trained model tries to adapt the model to a target domain,
access to the source data is feasible [9]. Here, we emphasize that pre-trained
original networks are frozen throughout our process, and our method is
applicable to any pre-trained model because it is agnostic to the architecture
and pre-training method of the original networks.

Our paper presents the following contributions:

  * •

We present novel meta networks that help the frozen original networks adapt to
the target domain. This architecture significantly minimize memory consumption
up to 86% by reducing the activation sizes of the original networks.

  * •

We propose a self-distilled regularization that controls the output of meta
networks by leveraging the output of frozen original networks to preserve the
source knowledge and prevent error accumulation.

  * •

We improve both memory efficiency and TTA performance compared to existing
state-of-the-art methods on 1) image classification task (_e.g_.,
CIFAR10/100-C and ImageNet-C) and 2) semantic segmentation task (_e.g_.,
Cityscapes with weather corruption)

##  2 Related Work

Figure 3: Overview of our approach. (a) The encoder of the pre-trained model
is divided into K parts (i.e., model partition factor K). (b) Before
deployment, the meta networks are attached to each part of the original
networks and pre-trained with source dataset $\mathcal{D}_{s}$. (c) After the
model is deployed, only the meta networks are updated with unsupervised loss
(_i.e_., entropy minimization) on target data $\mathcal{D}_{t}$, while the
original networks are frozen. To avoid error accumulation and catastrophic
forgetting by the long-term adaptation, we regularize the output
$\tilde{x}_{k}$ of each group of the meta networks leveraging the output
$x_{k}$ of the frozen original network, which preserves the source knowledge.

#### Mitigating domain shift.

One of the fundamental issues of DNNs is the performance degradation due to
the domain shift between the train (_i.e_. source) and test (_i.e_. target)
distributions. Several research fields attempt to address this problem, such
as unsupervised domain adaptation [64, 6, 53, 56, 46, 58] and domain
generalization [76, 8]. In particular, domain generalization aims to learn
invariant representation so as to cover the possible shifts of test data. They
simulate the possible shifts using a single or multiple source dataset [76,
74, 39] or force to minimize the dependence on style information [52, 8].
However, it is challenging to handle all potential test shifts using the given
source datasets [20]. Thus, instead of enhancing generalization ability during
the training time, TTA [65] overcomes the domain shift by directly adapting to
the test data.

#### Test-time adaptation.

Test-time adaptation allows the model to adapt to the test data (_i.e_.,
target domain) in a source-free and online manner [33, 62, 65]. Existing works
improve TTA performance with sophisticated designs of unsupervised loss [48,
72, 42, 9, 45, 57, 5, 16, 1, 3, 12, 59] or enhance the usability of small
batch sizes [36, 70, 31, 51, 40] considering streaming test data. They focus
on improving the adaptation performance with a stationary target domain
(_i.e_., single domain TTA setup). In such a setting, the model that finished
adaptation to a given target domain is reset to the original model pre-trained
with the source domain in order to adapt to the next target domain.

Recently, CoTTA [66] has proposed continual TTA setup to address TTA under a
continuously changing target domain which also involves a long-term
adaptation. This setup frequently suffers from error accumulation [75, 2, 63]
and catastrophic forgetting [66, 35, 50]. Specifically, performing a long-term
adaptation exposes the model to unsupervised loss from unlabeled test data for
a long time, so errors are accumulated significantly. Also, the model focuses
on learning new knowledge and forgets about the source knowledge, which
becomes problematic when the model needs to correctly classify the test sample
as similar to the source distribution. To address such issues, CoTTA [66]
randomly restores the updated parameters to the source one, while EATA [50]
proposed a weight regularization loss.

#### Efficient on-device learning.

Since the edge device is likely to be memory constrained (_e.g_., a Raspberry
Pi with 512MB and iPhone 13 with 4GB), it is necessary to take account of the
memory usage when deploying the models on the device [41]. TinyTL [4], a
seminal work in on-device learning, shows that the activation size, not
learnable parameters, bottlenecks the training memory. Following this, recent
on-device learning studies [4, 68, 69] targeting fine-tuning task attempt to
decrease the size of intermediate activations. In contrast, previous TTA
studies [65, 50] have overlooked these facts and instead focused on reducing
learnable parameters. This paper, therefore, proposes a method that not only
reduces the high activation sizes required for TTA, but also improves
adaptation performance.

##  3 Approach

Fig. 3 illustrates our simple yet effective approach which only updates the
newly added meta networks on the target domain while regularizing them with
the knowledge distilled from the frozen original network. This section
describes how such a design promotes memory efficiency and prevents error
accumulation and catastrophic forgetting which are frequently observed in
long-term adaptation.

###  3.1 Memory-efficient Architecture

#### Prerequisite.

We first formulate the forward and the backward propagation. Assume that the
$i^{th}$ linear layer in the model consists of weight $\mathcal{W}$ and bias
$b$, and the input and output features of this layer are $f_{i}$ and
$f_{i+1}$, respectively. Given that the forward propagation of
$f_{i+1}=f_{i}\mathcal{W}+b$, the backward propagation from the $i{+}1$th
layer to the $i^{th}$ layer, and the weight gradient are respectively
formulated as:

${\footnotesize\frac{\partial\mathcal{L}}{\partial
f_{i}}=\frac{\partial\mathcal{L}}{\partial
f_{i+1}}\mathcal{W}^{T},\quad\frac{\partial\mathcal{L}}{\partial\mathcal{W}}={\color[rgb]{0.8,0.02,0.02}\definecolor[named]{pgfstrokecolor}{rgb}{0.8,0.02,0.02}f_{i}^{T}}\frac{\partial\mathcal{L}}{\partial
f_{i+1}}.}$ (1)

Eq. (1) means that the learnable layers whose weight $\mathcal{W}$ need to be
updated must store intermediate activations
${\color[rgb]{0.8,0.02,0.02}\definecolor[named]{pgfstrokecolor}{rgb}{0.8,0.02,0.02}f_{i}}$
to compute the weight gradient. In contrast, the backward propagation in
frozen layers can be accomplished without saving the activations, only
requiring its weight $\mathcal{W}$. Further descriptions are provided in
Appendix A.

TinyTL [4] shows that activations occupy the majority of the memory required
for training the model rather than learnable parameters. Due to this fact,
updating the entire model (e.g., CoTTA [66]) requires a substantial amount of
memory. Also, updating only parameters in batch normalization (BN) layers
(e.g., TENT [65] and EATA [50]) is not an effective approach enough since they
still save the large intermediate activations for multiple BN layers. While
previous studies fail to reduce memory by utilizing large activations, this
work proposes a simple yet effective way to reduce a significant amount of
memory by discarding them.

#### Before deployment.

As illustrated in Fig. 3 (a, b), we first take a pre-trained model using any
pre-training method. We divide the encoder of the pre-trained model into K
number of parts and attach lightweight meta networks to each part of the
original network. The details of how to divide the model into K number of
parts are explained in the next section. One group of meta network composes of
one batch normalization layer and one convolution block (_i.e_., Conv-BN-
Relu). Before the deployment, we pre-train the meta networks on the source
dataset $\mathcal{D}_{s}$ for a small number of epochs (e.g., 10 epochs for
CIFAR dataset) while freezing the original networks. Such a warm-up process is
completed before the model deployment, similarly done in several TTA works [9,
34, 40, 50]. Note that we do not require source dataset $\mathcal{D}_{s}$
during test time.

#### Pre-trained model partition.

Previous TTA studies addressing domain shifts [9, 48] indicate that updating
shallow layers is more crucial for improving the adaptation performance than
updating the deep layers. Inspired by such a finding, given that the encoder
of the pre-trained model is split into model partition factor K (_e.g_., 4 or
5), we partition the shallow parts of the encoder more (_i.e_., densely)
compared to the deep parts of it. Table LABEL:tab:model_partition shows how
performance changes as we vary the model partition factor K.

#### After deployment.

During the test-time adaptation, we only adapt meta networks to target domains
while freezing the original networks. Following EATA [50], we use the entropy
minimization $H(\hat{y})=-\sum_{c}p(\hat{y})\log p(\hat{y})$ to the samples
achieving entropy less than the pre-defined entropy threshold $H_{0}$, where
$\hat{y}$ is the prediction output of a test image from test dataset
$\mathcal{D}_{t}$ and p($\cdot$) is the softmax function. Thus, the main task
loss for adaptation is defined as

$\mathcal{L}^{ent}=\mathbb{I}_{\\{H(\hat{y})<H_{0}\\}}\cdot H(\hat{y}),$ (2)

where $\mathbb{I}_{\\{\cdot\\}}$ is an indicator function. In addition, in
order to prevent catastrophic forgetting and error accumulation, we apply our
proposed regularization loss $\mathcal{R}^{k}$, which is described next in
detail. Consequently, the overall loss of our method is formulated as,

$\mathcal{L}^{total}_{\theta}=\mathcal{L}_{\theta}^{ent}+\lambda\;\sum_{k}^{K}\mathcal{R}_{\theta_{k}}^{k},$
(3)

where $\theta$ and $\theta_{k}$ denotes parameters of all meta networks and
those of k-th group of meta networks, respectively, and $\lambda$ is used to
balance the scale of the two loss functions. Note that our architecture
requires less memory than previous works [66, 65] since we use frozen original
networks and discard its intermediate activations. To be more specific, our
architecture uses 82% and 60% less memory on average than CoTTA and TENT/EATA.

###  3.2 Self-distilled Regularization

| WideResNet-40 (AugMix) | WideResNet-28 | ResNet-50
---|---|---|---
Method | Avg. err $\downarrow$ | Mem. (MB) | Avg. err $\downarrow$ | Mem. (MB) | Avg. err $\downarrow$ | Mem. (MB)
Source | 36.7 | 11 | 43.5 | 58 | 48.8 | 91
BN Stats Adapt [49] | 15.4 | 11 | 20.9 | 58 | 16.6 | 91
Single do. TENT [65] | 12.7 | 188 | 19.2 | 646 | 15.0 | 925
Continual TENT | 13.3 | 188 | 20.0 | 646 | 15.2 | 925
TTT++ [42] | 14.6 | 391 | 20.3 | 1405 | 16.1 | 1877
SWR&NSP [9] | 12.1 | 400 | 17.2 | 1551 | 15.4 | 1971
NOTE [17] | 13.4 | 188 | 20.2 | 646 | - | -
EATA [50] | 13.0 | 188 | 18.6 | 646 | 14.2 | 925
CoTTA [66] | 14.0 | 409 | 17.0 | 1697 | 14.4 | 2066
Ours (K=4) | 12.2 | 80 (80, 58%$\downarrow$) | 16.9 | 404 (76, 38%$\downarrow$) | 14.4 | 296 (86, 68%$\downarrow$)
Ours (K=5) | 12.1 | 92 (77, 51%$\downarrow$) | 16.8 | 471 (72, 27%$\downarrow$) | 14.1 | 498 (76, 46%$\downarrow$)

(a)

| WideResNet-40 (AugMix) | ResNet-50
---|---|---
Method | Avg. err $\downarrow$ | Mem. (MB) | Avg. err $\downarrow$ | Mem. (MB)
Source | 69.7 | 11 | 73.8 | 91
BN Stats Adapt [49] | 41.1 | 11 | 44.5 | 91
Single do. TENT [65] | 36.7 | 188 | 40.1 | 926
Continual TENT | 38.3 | 188 | 45.9 | 926
TTT++ [42] | 41.0 | 391 | 44.2 | 1876
SWR&NSP [9] | 36.6 | 400 | 44.1 | 1970
NOTE [17] | 42.8 | 188 | - | -
EATA [50] | 37.1 | 188 | 39.9 | 926
CoTTA [66] | 38.1 | 409 | 40.2 | 2064
Ours (K=4) | 36.4 | 80 (80, 58%$\downarrow$) | 39.5 | 296 (86, 68%$\downarrow$)
Ours (K=5) | 36.3 | 92 (77, 51%$\downarrow$) | 39.3 | 498 (76, 46%$\downarrow$)

(b)

Table 1: Comparison of error rate ($\%$) on CIFAR-C. We report an average
error of 15 corruptions on continual TTA and a memory requirement including
model parameters and activation sizes. The lowest error is in bold, and the
second lowest error is underlined. The memory reduction rates compared to
CoTTA and TENT are presented sequentially. WideResNet-40 was pre-trained with
AugMix [26] that is a data processing to increase the robustness of the model.
Source denotes the pre-trained model without adaptation. Single domain (in
short, single do.) TENT resets the model when adapting to a new target domain,
so the domain labeles are required.

| ResNet-50 (AugMix) | ResNet-50 |  (MB) Total Mem. $\downarrow$
---|---|---|---
Method |  Avg. err $\downarrow$ |  Avg. err $\downarrow$
Source | 74.36 | 82.35 | 91
BN Stats Adapt [49] | 57.87 | 72.18 | 91
Continual TENT [65] | 56.1 (0.6) | 66.2 (1.1) | 1486
EATA [50] | 54.9 (2.3) | 63.8 (2.7) | 1486
CoTTA [66] |  54.6 (3.9) |  62.6 (3.1) | 3132
Ours (K=4) | 55.2 (3.0) | 64.6 (3.2) | 438 (86, 72%$\downarrow$)
Ours (K=5) | 54.4 (2.7) |  63.4 (3.0) |  747 (75, 51%$\downarrow$)

Table 2: Comparison of error rate ($\%$) on ImageNet-C with severity level 5.
Standard deviation for ten diverse corruption sequences is denoted by the
parentheses values. The total memory refers to the sum of model parameters and
activations.

Avg. err (%) |  | CIFAR10-C | CIFAR100-C
---|---|---|---
Method | Mem. (MB) | single do. | continual | single do. | continual
BN Stats Adapt [49] | 91 | 16.6 | 16.6 | 44.5 | 44.5
TinyTL† [4] | 379 | 15.8 | 21.9 | 40.5 | 77.4
RepNet† [69] | 508 | 15.2 | 20.9 | 41.5 | 52.1
AuxAdapt† [73] | 207 | 16.0 | 16.7 | 44.0 | 45.8
Ours (K=4) | 296 | 14.4 | 14.4 | 39.5 | 39.2

Table 3: Comparison with methods for on-device learning. The backbone is
ResNet-50. † denotes our own re-implemented models. single do. indicates the
singe domain TTA setup.

The unsupervised loss from unlabeled test data $\mathcal{D}_{t}$ is likely to
provide a false signal (_i.e_., noise) to the model ($\hat{y}\neq y_{t}$ where
$y_{t}$ is the ground truth test label). Previous works have verified that
long-term adaptation with unsupervised loss causes overfitting due to error
accumulation [75, 2] and catastrophic forgetting [66, 35]. To prevent the
critical issues, we propose a self-distilled regularization utilizing our
architecture. As shown in Fig. 3, we regularize the output $\tilde{x}_{k}$ of
each $k$-th group of the meta networks not to deviate from the output $x_{k}$
of the $k$-th part of frozen original networks. Our regularization loss which
computes the mean absolute error (_i.e_., L1 loss) is formulated as follows:

$\mathcal{R}_{\theta_{k}}^{k}=\left\|\tilde{x}_{k}-x_{k}\right\|_{1}.$ (4)

Since the original networks are not updated, the output $x_{k,k\sim K}$
extracted from them can be considered as containing the knowledge learned from
the source domain. Taking advantage of this fact, we let the output of meta
networks $\tilde{x}_{k}$ be regularized with knowledge distilled from the
original networks. By preventing the adapted model to not significantly
deviate from the original model, we can prevent 1) catastrophic forgetting by
maintaining the source domain knowledge and 2) error accumulation by utilizing
the class discriminability of the original model. Remarkably, unlike previous
works [66, 50], our regularization does not require saving additional original
networks, which accompanies extra memory usage. Moreover, it only needs a
negligible amount of computational overhead because it is performed in
parallel with the entropy minimization loss $\mathcal{L}^{ent}$.

##  4 Classification Experiments

(a)

| Avr. err
---
Arch
| CIFAR10-C
---
WRN-28
| CIFAR10-C
---
WRN-40
| CIFAR100-C
---
WRN-40
(i) | 18.1 | 12.6 | 37.2
(ii) Ours w$\setminus$o BN | 18.7 | 13.7 | 38.2
(iii) Ours w$\setminus$o Conv | 20.7 | 14.9 | 40.1
(iv) Conv | 60.6 | 73.3 | 77.2
(v) CBAM [67] | 21.4 | 15.1 | 40.9
(vi) SE [30] | 22.3 | 16.2 | 40.5
Ours | 16.8 | 12.1 | 36.3

(b)

Model | #Block | Avg. err
---|---|---
WRN-28 (12) CIFAR10-C |  3,3,3,3 | 17.3
4,4,2,2 | 17.9
2,2,4,4 | 16.9
WRN-40 (18) CIFAR10-C | 4,4,5,5 | 12.8
6,6,3,3 | 13.7
3,3,6,6 | 12.2
WRN-40 (18) CIFAR100-C | 4,4,5,5 | 36.9
6,6,3,3 | 38.5
3,3,6,6 | 36.4

(c)

Table 4: Architecture ablation experiments. (a,b) We compare continual TTA
performance on several memory-efficient designs. WRN refers to WideResNet [71]
backbone. (c) We report the performance based on different designs of
partitioning the model. The value next to the backbone’s name denotes the
total number of residual blocks of a model.

We evaluate our approach to image classification tasks based on the continual
test-time adaptation setup with three datasets: CIFAR10-C, CIFAR100-C, and
ImageNet-C.

#### Experimental setup.

Following CoTTA [66], we conduct most experiments on the continual TTA task,
where we continually adapt the deployed model to each corruption type
sequentially without resetting the model. This task is more challenging but
more realistic than single domain TTA task [65] in which the adapted model is
periodically reset to the original pre-trained model after finishing
adaptation to each target, so they require additional domain information.
Moreover, we evaluate our approach on the long-term TTA setup, which is
detailed in Section 4.2.

Following the previous TTA studies [65, 66], we evaluate models with {CIFAR10,
CIFAR10-C}, {CIFAR100, CIFAR100-C}, and {ImageNet, ImageNet-C} where the first
and the second dataset in each bracket refers to the source and the target
domain, respectively. The target domains include 15 types of corruptions
(_e.g_. noise, blur, weather, and digital) with 5 levels of severity, which
are widely used in conventional benchmarks [25].

#### Implementation Details.

We evaluate our approach within the frameworks officially provided by previous
state-of-the-art methods [66, 50]. For fair comparisons, we use the same pre-
trained model, which are WideResNet-28 and WideResNet-40 [71] models from the
RobustBench [11], and ResNet-50 [24] model from TTT++ [42, 9]. Before the
deployment, we pre-train the meta networks on the source dataset using a
cross-entropy loss with SGD optimizer with the learning rate of 5e-2. Since
the meta networks contain only a few layers, we pre-train them with a small
number of epochs: 10 and 3 epochs for CIFAR and ImageNet, respectively. After
deployment, similar to EATA [50], we use the same SGD optimizer with the
learning rate of 5e-3. In Eq. (2), the entropy threshold $H_{0}$ is set to
$0.4\times\ln C$ where $C$ denotes the number of task classes. The batch size
is 64 and 32 for CIFAR and ImageNet, respectively. We set the importance of
the regularization $\lambda$ in Eq. (3) to 0.5 to balance it with the entropy
minimization loss. Additional implementation details can be found in Appendix
C.

#### Evaluation Metric.

For all the experiments, we report error rates calculated during testing and
the memory consumption, including the model parameter and the activation
storage. We demonstrate the memory efficiency of our work by using the
official code provided by TinyTL [4].

###  4.1 Comparisons

#### Comparisons with TTA methods.

We compare our approach to competing TTA methods on extensive benchmarks and
various pre-trained models. The results of CIFAR10/100-C are detailed in Table
1. The model partition factor K are set to 4 and 5. Our approach outperforms
existing TTA methods with the lowest memory usage in all pre-trained models.
Specifically, in WideResNet-40, our method achieves superior performance while
requiring 80% and 58% less memory than CoTTA [66] and EATA [50], respectively,
which are also designed for continual TTA. Approaches targeting single domain
TTA [65, 42, 9] show poor performance due to error accumulation and
catastrophic forgetting, as observed in CoTTA. The error rates for each
corruption type are provided in Appendix F.

Table 2 shows the experiment for ImageNet-C. Two ResNet-50 backbones from
RobustBench [11] are leveraged. Following CoTTA, evaluations are conducted on
ten diverse corruption-type sequences. We achieve comparable performance to
CoTTA while utilizing 86% and 75% less memory with K=4 and 5, respectively. In
addition, we observe that our approach shows superior performance when
adopting the model pre-trained with strong data augmentation methods (e.g.,
Augmix [26]).

Figure 5: Ablation study of K. We uniformly divide the encoder of the pre-
trained model into the model partition factor K. The x-axis indicates the
memory size including both model parameter size and activation size while the
y-axis indicates the average error rate. The values in parentheses show the
rate of increase for the model parameters compared to the original model.

(a)

(b)

Figure 6: Regularization ablation experiments. We conduct experiments with
WideResNet-40 on CIFAR100-C. (a) We utilize a test set of the CIFAR-100
dataset to measure clean error after adapting to each corruption. Maintaining
clean errors at a stable level indicates that our approach helps the model
robust to catastrophic forgetting. (b) We simulate a long-term adaptation
scenario by repeating 100 rounds of 15 corruption sequences. In the absence of
regularization, error accumulation can lead to overfitting (_i.e_., the case
of the error increases exponentially). However, our approach does not suffer
from such an error accumulation. We set K to 5 in the above experiments.

#### Comparisons with on-device learning methods.

We compare our approach with methods for memory-efficient on-device learning.
TinyTL [4] and RepNet [69] focus on _supervised_ on-device learning (_i.e_.,
requiring labeled target data). However, since TTA assumes that we do not have
access to the target labels, utilizing such methods to TTA directly is
infeasible. Therefore, we experimented by replacing supervised loss (_i.e_.,
cross-entropy) with unsupervised loss (_i.e_., entropy minimization) in TinyTL
and RepNet. As shown in Table 3, they suffer from performance degradation in
continual TTA, showing inferior performance compared to our proposed approach
even in the single domain TTA.

Similar to ours, AuxAdapt [73] adds and updates a small network (_i.e_.,
ResNet-18) while freezing the pre-trained model. Unlike our approach, they
only modify a prediction output, not intermediate features. While AuxAdapt
requires the least memory usage, it fails to improve TTA performan-ce in
single domain TTA. Nevertheless, since the original model is frozen, it
suffers less from catastrophic forgetting and error accumulation than TinyTL
[4] and RepNet [69] in the continual TTA. Through the results, we confirm that
our proposed method brings both memory efficiency and a significant
performance improvement in both TTA setups.

###  4.2 Empirical Study

#### Architecture design.

An important design of our meta networks is injecting a single BN layer before
the original networks and utilizing a residual connection with one conv block.
Table LABEL:tab:auxnet_ablation studies the effectiveness of the proposed
design by comparing it with six different variants. From the results, we
observe that using only either conv block (ii) or BN (iii) aggravates the
performance: error rate increases by 1.4% and 3.8% on CIFAR-100-C with
WideResNet-40.

In design (i), we enforce both BN parameters and Conv layers in the meta
networks to take the output of the original networks as inputs. Such a design
brings performance drop. We speculate that it is because the original network,
which is not adapted to the target domain, lacks the ability to extract
sufficiently meaningful features from the target image. Also, we observed a
significant performance degradation after removing the residual connection in
design (iv). In addition, since attention mechanisms [67, 30] generally have
improved classification accuracy, we study how attention mechanisms can
further boost TTA performance of our approach in design (v, vi). The results
show that it is difficult for the attention module to train ideally in TTA
setup using unsupervised learning, unlike when applying it to supervised
learning. An ablation study on each element of meta networks can be found in
Appendix D.

| Batch size | 16 | 8 | 4 | 2 | 1
---|---|---|---|---|---|---
Non training | Source | 69.7 | 69.7 | 69.7 | 69.7 | 69.7
BN Stats Adapt [49] | 41.1 | 50.2 | 59.9 | 81.0 | 99.1
AdaptBN [55] | 39.1 | 41.2 | 45.2 | 49.0 | 54.0
Training |  Con. TENT [65] | 40.9 | 47.8 | 58.6 | 82.2 | 99.0
Con. TENT+AdaptBN | 38.2 | 40.2 | 43.2 | 47.7 | 52.2
Ours (K=5) | 40.0 | 45.8 | 63.4 | 80.8 | 99.0
Ours (K=5)+AdaptBN | 36.9 | 39.3 | 42.2 | 46.5 | 51.8

Table 5: Experiments with small batch sizes. We evaluate all baselines with
WideResNet-40 on CIFAR100-C. Con. TENT is the abbreviation for continual TENT.

#### Number of blocks in each partition.

ResNet [24] consists of multiple residual blocks (_e.g_., BasicBlock and
Bottleneck in Pytorch [54]). For instance, WideResNet-28 has 12 residual
blocks. By varying the number of blocks for each part of the original
networks, we analyze TTA performance in Table LABEL:tab:model_partition. We
observe that splitting the shallow parts of the encoder densely (_e.g_.,
2,2,4,4 blocks, from the shallow to the deep parts sequentially) brings more
performance gain than splitting the deep layers densely (_e.g_., 4,4,2,2
blocks). We suggest that it is because we modify the lower-level feature more
as we split shallow layers densely. Our observation is aligned with the
finding of previous TTA works [9, 48], which show that updating the shallow
layers more than the deep layers improves TTA performance.

#### Number of model partition K.

Fig. 5 shows both memory requirement and adaptation performance according to
the model partition factor K. With a small K (_e.g_., 1 or 2), the
intermediate outputs are barely modified, making it difficult to achieve a
reasonable level of performance. We achieve the best TTA performance with K of
4 or 5 as adjusting a greater numver of intermediate features. In the
meanwhile, we observe that the average error rate is saturated and remains
consistent when K is set to large values (_e.g_. 6,7 or 8) even with the
increased amount of activations and learnable parameters. Therefore, we set K
to 4 and 5.

Time |  | $t\xrightarrow{\hskip 654.41338pt}$ |
---|---|---|---
Round |  | 1 |  |  |  | 4 |  |  |  | 7 |  |  |  | 10 |  |  |  | All
Method | Mem. (MB) | Brig. | Fog | Fros. | Snow | Brig. | Fog | Fros. | Snow | Brig. | Fog | Fros. | Snow | Brig. | Fog | Fros. | Snow | Mean
Source | 280 | 60.4 | 54.3 | 30.0 | 4.1 | 60.4 | 54.3 | 30.0 | 4.1 | 60.4 | 54.3 | 30.0 | 4.1 | 60.4 | 54.3 | 30.0 | 4.1 | 37.2
BN Stats Adapt [49] | 280 | 69.1 | 61.0 | 44.8 | 39.1 | 69.1 | 61.0 | 44.8 | 39.1 | 69.1 | 61.0 | 44.8 | 39.1 | 69.1 | 61.0 | 44.8 | 39.1 | 53.6
Continual TENT [65] | 2721 | 70.1 | 62.1 | 46.1 | 40.2 | 62.2 | 53.7 | 44.4 | 37.9 | 50.0 | 41.5 | 31.6 | 26.6 | 39.2 | 32.6 | 25.3 | 22.4 | 42.9
Ours (K=4) | 918 (66%$\downarrow$) | 70.2 | 62.4 | 46.3 | 41.9 | 70.0 | 62.8 | 46.5 | 42.2 | 70.0 | 62.8 | 46.5 | 42.1 | 70.1 | 62.8 | 46.6 | 42.2 | 55.3

Table 6: Semantic segmentation results in continual test-time adaptation
tasks. We conduct experiments on Cityscapes [10] with four weather corruptions
[25] applied. The four conditions are repeated ten times to simulate continual
domain shifts. All results are evaluated based on DeepLabV3Plus-ResNet-50.

#### Catastrophic forgetting.

We conduct experiments to confirm the catastrophic forgetting effect (Fig.
LABEL:fig:forgetting). Once finishing adaptation to each corruption, we
evaluate the model on clean target data (_i.e_., test-set of CIFAR dataset)
without updating the model. For TENT with no regularization, the error rates
for the clean target data (_i.e_., clean error (%)) increase gradually, which
can be seen as the phenomenon of catastrophic forgetting. In contrast, our
approach consistently maintains the error rates for the clean target data,
proving that our regularization loss effectively prevents catastrophic
forgetting. These results indicate that our method can be reliably utilized in
various domains, including the source and target domains.

#### Error accumulation in long-term adaptation.

To evaluate the error accumulation effect, we repeat all the corruption
sequences for 100 rounds. The results are described in Fig. LABEL:fig:long.
For TENT, a gradual increase in error rates is observed in later rounds, even
with small learning rates. For example, TENT [65] with the learning rate of
1e-5 achieves the error rate of 39.7%, and reached its lowest error rate of
36.5% after 8 rounds. However, it shows increased error rate of 38.6% after
100 rounds due to overfitting. It suggests that without regularization, TTA
methods eventually face overfitting in long-term adaptation [75, 2, 35]. Our
method in the absence of regularization ($\lambda=0$) also causes overfitting.
On the other hand, when self-distilled regularization is involved
($\lambda>0$), the performance remains consistent even in the long-term
adaptation.

#### Small batch size.

We examine the scalability of our approach with a TTA method designed for
small batches size, named adapting BN statistics (_i.e_., AdaptBN [55, 72]).
When the number of batches is too small, the estimated statistics can be
unreliable [55]. Thus, they calibrate the source and target statistics for the
normalization of BN layers so as to alleviate the domain shift and preserve
the discriminative structures. As shown in Table 5, training models with small
batch sizes (e.g., 2 or 1) generally increase the error rates. However, such
an issue can be addressed by appying AdaptBN to our method. To be more
sepcific, we achieve an absolute improvement of 17.9% and 2.2% from Source and
AdaptBN, respectively, in the batch size of 1.

#### Number of the source samples for meta networks.

Like previous TTA works [9, 42, 34, 40] including EATA [50], our approach
requires access to the source data for pre-training our proposed meta networks
before model deployment. In order to cope with the situation where we can only
make use of a subset of the source dataset, we study the TTA performance of
our method according to the number of accessible source samples. The results
are specified in Table 7 where we use WideResNet-40. We observe that our
method outperforms the baseline model even with small number of training
samples (_e.g_., 10% or 20%) while showing comparable performance with
excessively small numbers (_e.g_. 5%). Note that we still reduce the memory
usage of about 51% compared to EATA.

##  5 Segmentation Experiments

We investigate our approach in semantic segmentation. First, we create
Cityscapes-C by applying the weather corruptions (brightness, fog, frost, and
snow [25]) to the validation set of Cityscapes [10]. Then, to simulate
continual distribution shifts, we repeat the four types of Cityscapes-C ten
times. In this scenario, we conduct continual TTA using the publicly-available
ResNet-50-based DeepLabV3+ [7], which is pre-trained on Cityscapes for domain
generalization task [8] in semantic segmentation. For TTA, we use the batch
size of 2. More details are specified in Appendix C.

|  EATA [50] (188MB) |  | # of source samples
---|---|---|---
Target domain |  Ours (K=5) (92MB) | 10k (20%) | 5k (10%) | 2.5k (5%)
CIFAR10-C | 13.0 | 12.1 | 12.4 | 12.9 | 13.1
CIFAR100-C | 37.1 | 36.3 | 36.4 | 36.6 | 37.2

Table 7: Ablation of # of source samples to warm up the meta networks. Before
deployment, we pre-trained the meta networks using only a subset of the source
dataset (_e.g_., 20%, 10%, and 5%). The memory usage (MB) of each method is
also presented.

#### Results.

We report the results based on mean intersection over union (mIoU) in Table 6.
It demonstrates that our approach helps to both minimize memory consumption
and performs long-term adaptation stably for semantic segmentation. Unlike
continual TENT, our method avoids catastrophic forgetting and error
accumulation, allowing us to achieve the highest mIoU score while using 66%
less memory usage in a continual TTA setup. Additional experiment results can
be found in Appendix B.

##  6 Conclusion

This paper proposed a simple yet effective approach that improves continual
TTA performance and saves a significant amount of memory, which can be applied
to edge devices with limited memory. First, we presented a memory-efficient
architecture that consists of original networks and meta networks. This
architecture requires much less memory size than the previous TTA methods by
decreasing the intermediate activations used for gradient calculations.
Second, in order to preserve the source knowledge and prevent error
accumulation during long-term adaptation with noisy unsupervised loss, we
proposed self-distilled regularization that controls the output of meta
networks not to deviate significantly from the output of the original
networks. With extensive experiments on diverse datasets and backbone
networks, we verified the memory efficiency and TTA performance of our
approach. In this regard, we hope that our efforts will facilitate a variety
of studies that make test-time adaptation for edge devices feasible in
practice.

#### Acknowledgments.

We would like to thank Kyuwoong Hwang, Simyung Chang, and Byeonggeun Kim for
their valuable feedback. We are also grateful for the helpful discussions from
Qualcomm AI Research teams.

## References

  * [1] Kazuki Adachi, Shin’ya Yamaguchi, and Atsutoshi Kumagai.  Covariance-aware feature alignment with pre-computed source statistics for test-time adaptation.  arXiv preprint arXiv:2204.13263, 2022.
  * [2] Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness.  Pseudo-labeling and confirmation bias in deep semi-supervised learning.  In IJCNN, 2020.
  * [3] Kambiz Azarian, Debasmit Das, Hyojin Park, and Fatih Porikli.  Test-time adaptation vs. training-time generalization: A case study in human instance segmentation using keypoints estimation.  In WACV Workshops, 2023.
  * [4] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han.  Tinytl: Reduce memory, not parameters for efficient on-device learning.  In NeurIPS, 2020.
  * [5] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi.  Contrastive test-time adaptation.  In CVPR, 2022.
  * [6] Lin Chen, Huaian Chen, Zhixiang Wei, Xin Jin, Xiao Tan, Yi Jin, and Enhong Chen.  Reusing the task-specific classifier as a discriminator: Discriminator-free adversarial domain adaptation.  In CVPR, 2022.
  * [7] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In ECCV, 2018.
  * [8] Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne T Kim, Seungryong Kim, and Jaegul Choo.  Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening.  In CVPR, 2021.
  * [9] Sungha Choi, Seunghan Yang, Seokeon Choi, and Sungrack Yun.  Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes.  In ECCV, 2022.
  * [10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In CVPR, 2016.
  * [11] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein.  Robustbench: a standardized adversarial robustness benchmark.  In NeurIPS Datasets and Benchmarks Track, 2021.
  * [12] Debasmit Das, Shubhankar Borse, Hyojin Park, Kambiz Azarian, Hong Cai, Risheek Garrepalli, and Fatih Porikli.  Transadapt: A transformative framework for online test time adaptive semantic segmentation.  In ICASSP, 2023.
  * [13] Zhiwei Deng and Olga Russakovsky.  Remember the past: Distilling datasets into addressable memories for neural networks.  arXiv preprint arXiv:2206.02916, 2022.
  * [14] Sauptik Dhar, Junyao Guo, Jiayi Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah.  A survey of on-device machine learning: An algorithms and learning theory perspective.  ACM Transactions on Internet of Things, 2021.
  * [15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [16] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A Efros.  Test-time training with masked autoencoders.  In NeurIPS, 2022.
  * [17] Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee.  Robust continual test-time adaptation: Instance-aware bn and prediction-balanced memory.  In NeurIPS, 2022.
  * [18] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  arXiv preprint arXiv:1706.02677, 2017.
  * [19] Yves Grandvalet and Yoshua Bengio.  Semi-supervised learning by entropy minimization.  In NeurIPS, 2004.
  * [20] Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  In ICLR, 2021.
  * [21] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In ICML, 2017.
  * [22] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In CVPR, 2022.
  * [23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, 2020.
  * [24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [25] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  In ICLR, 2019.
  * [26] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan.  AugMix: A simple data processing method to improve robustness and uncertainty.  In ICLR, 2020.
  * [27] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  In NeurIPS, 2014.
  * [28] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In ICML, 2019.
  * [29] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  In ICLR, 2022.
  * [30] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In CVPR, 2018.
  * [31] Xuefeng Hu, Gokhan Uzunbas, Sirius Chen, Rui Wang, Ashish Shah, Ram Nevatia, and Ser-Nam Lim.  Mixnorm: Test-time adaptation through online normalization estimation.  arXiv preprint arXiv:2110.11478, 2021.
  * [32] Yusuke Iwasawa and Yutaka Matsuo.  Test-time classifier adjustment module for model-agnostic domain generalization.  In NeurIPS, 2021.
  * [33] Vidit Jain and Erik Learned-Miller.  Online domain adaptation of a pre-trained cascade of classifiers.  In CVPR, 2011.
  * [34] Sanghun Jung, Jungsoo Lee, Nanhee Kim, and Jaegul Choo.  Cafa: Class-aware feature alignment for test-time adaptation.  arXiv preprint arXiv:2206.00205, 2022.
  * [35] Tommie Kerssies, Joaquin Vanschoren, and Mert Kılıçkaya.  Evaluating continual test-time adaptation for contextual and semantic domain shifts.  arXiv preprint arXiv:2208.08767, 2022.
  * [36] Ansh Khurana, Sujoy Paul, Piyush Rai, Soma Biswas, and Gaurav Aggarwal.  Sita: Single image test-time adaptation.  arXiv preprint arXiv:2112.02355, 2021.
  * [37] Andreas Krause, Pietro Perona, and Ryan Gomes.  Discriminative clustering by regularized information maximization.  In NeurIPS, 2010.
  * [38] Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler.  Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization.  In CVPR, 2021.
  * [39] Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and Ling-Yu Duan.  Uncertainty modeling for out-of-distribution generalization.  In ICLR, 2022.
  * [40] Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi.  TTN: A domain-shift aware batch normalization in test-time adaptation.  In ICLR, 2023.
  * [41] Ji Lin, Wei-Ming Chen, Yujun Lin, Chuang Gan, Song Han, et al.  Mcunet: Tiny deep learning on iot devices.  In NeurIPS, 2020.
  * [42] Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi.  Ttt++: When does self-supervised test-time training fail or thrive?  In NeurIPS, 2021.
  * [43] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.  Unsupervised domain adaptation with residual transfer networks.  In NeurIPS, 2016.
  * [44] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  In NeurIPS, 2017.
  * [45] Robert A Marsden, Mario Döbler, and Bin Yang.  Gradual test-time adaptation by self-training and style transfer.  arXiv preprint arXiv:2208.07736, 2022.
  * [46] Ke Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang.  Instance adaptive self-training for unsupervised domain adaptation.  In ECCV, 2020.
  * [47] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  In NeurIPS, 2019.
  * [48] Chaithanya Kumar Mummadi, Robin Hutmacher, Kilian Rambach, Evgeny Levinkov, Thomas Brox, and Jan Hendrik Metzen.  Test-time adaptation to distribution shift by confidence maximization and input transformation.  arXiv preprint arXiv:2106.14999, 2021.
  * [49] Zachary Nado, Shreyas Padhy, D Sculley, Alexander D’Amour, Balaji Lakshminarayanan, and Jasper Snoek.  Evaluating prediction-time batch normalization for robustness under covariate shift.  arXiv preprint arXiv:2006.10963, 2020.
  * [50] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan.  Efficient test-time model adaptation without forgetting.  In ICML, 2022.
  * [51] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan.  Towards stable test-time adaptation in dynamic wild world.  In ICLR, 2023.
  * [52] Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang.  Two at once: Enhancing learning and generalization capacities via ibn-net.  In ECCV, 2018.
  * [53] Kwanyong Park, Sanghyun Woo, Inkyu Shin, and In So Kweon.  Discover, hallucinate, and adapt: Open compound domain adaptation for semantic segmentation.  In NeurIPS, 2020.
  * [54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, and et al. Lin.  Pytorch: An imperative style, high-performance deep learning library.  In NeurIPS, 2019.
  * [55] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge.  Improving robustness against common corruptions by covariate shift adaptation.  In NeurIPS, 2020.
  * [56] Inkyu Shin, Dong-Jin Kim, Jae Won Cho, Sanghyun Woo, Kwanyong Park, and In So Kweon.  Labor: Labeling only if required for domain adaptive semantic segmentation.  In ICCV, 2021.
  * [57] Inkyu Shin, Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, In So Kweon, and Kuk-Jin Yoon.  Mm-tta: Multi-modal test-time adaptation for 3d semantic segmentation.  In CVPR, 2022.
  * [58] Inkyu Shin, Sanghyun Woo, Fei Pan, and InSo Kweon.  Two-phase pseudo label densification for self-training based domain adaptation.  In ECCV, 2020.
  * [59] Junha Song, Kwanyong Park, Inkyu Shin, Sanghyun Woo, Chaoning Zhang, and In So Kweon.  Test-time adaptation in the dynamic world with compound domain knowledge management.  arXiv preprint arXiv:2212.08356, 2023.
  * [60] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 2014.
  * [61] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele.  Meta-transfer learning for few-shot learning.  In CVPR, 2019.
  * [62] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt.  Test-time training with self-supervision for generalization under distribution shifts.  In ICML, 2020.
  * [63] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In NeurIPS, 2017.
  * [64] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Mathieu Cord, and Patrick Pérez.  Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation.  In CVPR, 2019.
  * [65] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell.  Tent: Fully test-time adaptation by entropy minimization.  In ICLR, 2021.
  * [66] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai.  Continual test-time domain adaptation.  In CVPR, 2022.
  * [67] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In ECCV, 2018.
  * [68] Li Yang, Adnan Siraj Rakin, and Deliang Fan.  Da3: Dynamic additive attention adaption for memory-efficient on-device multi-domain learning.  In CVPR Workshops, 2022.
  * [69] Li Yang, Adnan Siraj Rakin, and Deliang Fan.  Rep-net: Efficient on-device learning via feature reprogramming.  In CVPR, 2022.
  * [70] Fuming You, Jingjing Li, and Zhou Zhao.  Test-time batch statistics calibration for covariate shift.  arXiv preprint arXiv:2110.04065, 2021.
  * [71] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  In BMVC, 2016.
  * [72] Marvin Zhang, Sergey Levine, and Chelsea Finn.  Memo: Test time robustness via adaptation and augmentation.  In NeurIPS, 2021.
  * [73] Yizhe Zhang, Shubhankar Borse, Hong Cai, and Fatih Porikli.  Auxadapt: Stable and efficient test-time adaptation for temporally consistent video semantic segmentation.  In WACV, 2022.
  * [74] Yabin Zhang, Minghan Li, Ruihuang Li, Kui Jia, and Lei Zhang.  Exact feature distribution matching for arbitrary style transfer and domain generalization.  In CVPR, 2022.
  * [75] Zixing Zhang, Fabien Ringeval, Bin Dong, Eduardo Coutinho, Erik Marchi, and Björn Schüller.  Enhanced semi-supervised learning for multimodal emotion recognition.  In ICASSP, 2016.
  * [76] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang.  Domain generalization with mixstyle.  In ICLR, 2021.

## Appendix

In this supplementary material, we provide,

  1. A.

Efficiency for TTA methods

  2. B.

Discussion and further experiments

  3. C.

Further implementation details

  4. D.

Additional ablations

  5. E.

Baseline details

  6. F.

Results of all corruptions

##  Appendix A Efficiency for TTA methods

#### Memory efficiency.

Existing TTA works [65, 50, 66] update model parameters to adapt to the target
domain. This process inevitably requires additional memory to store the
activations. Fig. 7 describes Eq. (1) of the main paper in more detail. For
instance, 1) the backward propagation from the layer ($c$) to the layer ($b$)
can be accomplished without saving intermediate activations $f_{i}$ and
$f_{i+1}$, since it only requires $\frac{\partial\mathcal{L}}{\partial
f_{i+1}}{=}\frac{\partial\mathcal{L}}{\partial\mathcal{L}}\mathcal{W}_{i+1}^{T}$
and $\frac{\partial\mathcal{L}}{\partial
f_{i}}{=}\frac{\partial\mathcal{L}}{\partial
f_{i+1}}\mathcal{W}_{i}^{T}{=}\frac{\partial\mathcal{L}}{\partial\mathcal{L}}\mathcal{W}_{i+1}^{T}\mathcal{W}_{i}^{T}$
operations. 2) During the forward propagation, the learnable layer ($a$) has
to store the intermediate activation
${\color[rgb]{0.8,0.02,0.02}\definecolor[named]{pgfstrokecolor}{rgb}{0.8,0.02,0.02}f_{i-1}}$
to calculate the weight gradient
$\frac{\partial\mathcal{L}}{\partial\mathcal{W}_{i-1}}{=}{\color[rgb]{0.8,0.02,0.02}\definecolor[named]{pgfstrokecolor}{rgb}{0.8,0.02,0.02}f_{i-1}^{T}}\frac{\partial\mathcal{L}}{\partial
f_{i}}$.

Figure 7: Forward and backward propagation. The black and red lines refer to
forward and backward propagation, respectively. $f$ and ($a,b,c$) are the
activations and the linear layers, respectively.

#### Computational efficiency.

Wall-clock time and floating point operations (FLOPs) are standard measures of
computational cost. We utilize wall-clock time to compare the computational
cost of TTA methods since most libraries computing FLOPs only support
inference, not training.

WideResNet-40 [71]
---
| Avg. err | Mem. (MB) | Theo. time | Wall time
Source | 69.7 | 11 | - | 40s
Con. TENT [65] | 38.3 | 188 | - | 2m 18s
CoTTA [66] | 38.1 | 409 | - | 22m 52s
EATA [50] | 37.1 | 188 | 2m 8s | 2m 22s
Ours (K=4) | 36.4 | 80 (80, 58%$\downarrow$) | 2m 27s | 2m 49s
Ours (K=5) | 36.3 |  92 (77, 51%$\downarrow$) | 2m 31s | 2m 52s
ResNet-50 [24]
| Avg. err | Mem. (MB) | Theo. time | Wall time
Source | 73.8 | 91 | - | 1m 8s
Con. TENT [65] | 45.9 | 926 | - | 4m 2s
CoTTA [66] | 40.2 | 2064 | - | 38m 24s
EATA [50] | 39.9 | 926 | 3m 45s | 4m 15s
Ours (K=4) | 39.5 | 296 (86, 68%$\downarrow$) | 4m 16s | 4m 41s
Ours (K=5) | 39.3 |  498 (76, 46%$\downarrow$) | 4m 26s | 5m 14s

Table 8: Comparison of training time on CIFAR100-C. We report both theoretical
time (in short, theo. time) and wall-clock time, taking to adapt to all 15
corruption types. Theoretical time is calculated by assuming that the ML
frameworks (_e.g_., Pytorch [54]) provide fine-grained implementations [4].
Con. TENT refers to continual TENT.

Unfortunately, wall-clock time of EATA [50] and our approach can not truly
represent its computational efficiency since the current Pytorch version [54]
does not support fine-grained implementation [4]. For example, EATA filters
samples to improve its computational efficiency. However, its gradient
computation is performed on the full mini-batch, so the wall-clock time for
backpropagation in EATA is almost the same as that of TENT [65]. In our
approach, our implementation follows Algorithm 1 to make each regularization
loss $\mathcal{R}_{\theta_{k}}^{k}$ applied to parameters of k-th group of
meta networks $\theta_{k}$ in Eq. (3). In order to circumvent such an issue,
the authors of EATA report the theoretical time, which assumes that PyTorch
handles gradient backpropagation at an instance level. Similar to EATA, we
also report both theoretical time and wall-clock time in Table 8. To compute
the theoretical time of our approach, we simply subtract the time for re-
forward (in Algorithm 1) from wall-clock time. We emphasize that this is
mainly an engineering-based issue, and the optimized implementation can
further improve computational efficiency. [50].

Using a single NVIDIA 2080Ti GPU, we measure the total time required to adapt
to all 15 corruptions, including the time to load test data and perform TTA.
The results in Table 8 show that our proposed method requires negligible
overhead compared to CoTTA [66]. For example, CoTTA needs approximately 10
times more training time than Continual TENT [65] with WideResNet-40. Note
that meta networks enable our approach to use 80% and 58% less memory than
CoTTA and EATA, even with such minor extra operations.

⬇

# img_t: test image

# model: original and meta networks

#

# ent_min(): Entropy minimization loss

# Detach_parts(): Detach the graph connection

# between each partition of networks

# Attach_parts(): Attach the graph connection

# between each partition of networks

for img_t in test_loader:

# 1. Forward

output = model(img_t)

# 2. Compute entropy loss

loss_ent = ent_min(output)

loss_ent.backward()

# 3. Re-forward

# (This process is not required

# in fine-grained ML frameworks.)

Detach_parts(model)

_ = model(img_t)

# 4. Compute regularization loss

reg_loss = 0

for k_th_meta in meta_networks:

reg_loss += k_th_meta.get_l1_loss()

reg_loss.backward()

# 5. Update params of meta networks

optim.step()

optim.zero_grad()

Attach_parts(model)

Algorithm 1 PyTorch-style pseudocode for EcoTTA.

##  Appendix B Discussion and further experiments

#### Comparison on gradually changing setup.

In Table 1 and Table 2, we evaluate all methods on the continual TTA task,
proposed in CoTTA [66] and EATA [50], where we continually adapt the deployed
model to each corruption type sequentially. Additionally, we conduct
experiments on the gradually changing setup. This gradual setup, proposed in
CoTTA, represents the sequence by gradually changing severity for the 15
corruption types: ${\underbrace{{\dots}2{\xrightarrow{}}1}_{\text{t-1 and
before}}}{\xrightarrow[type]{\small
change}}{\underbrace{1{\xrightarrow{}}2{\xrightarrow{}}3{\xrightarrow{}}4{\xrightarrow{}}5{\xrightarrow{}}4{\xrightarrow{}}3{\xrightarrow{}}2{\xrightarrow{}}1}_{\text{corruption
type t, gradually changing severity}}}{\xrightarrow[type]{\small
change}}\underbrace{1{\xrightarrow{}}2{\dots}}_{\text{t+1 and on}}$ ,

The results in Table 9 indicate that our approach outperforms previous TTA
methods [65, 50, 66] even with the gradually changing setup.

#### Comparisons with methods for parameter efficient transfer learning.

While our framework may be similar to parameter-efficient transfer learning
(PETL) [29, 28, 61] in that only partial parameters are updated during
training time for PETL or test time for TTA, we utilized meta networks to
minimize intermediate activations, which is crucial for memory-constrained
edge devices. We conduct experiments by applying a PETL method [28] to the TTA
setup. The adapter module is constructed by using 3x3 Conv and ReLU layers as
the projection layer and the nonlinearity, respectively, and these modules are
attached after each residual block of the backbone networks. The Table 10
shows that PETA+SDR needs a 177% increase in memory usage with a 6.1% drop in
performance, compared to our method.

Method |  Con. TENT [65] |  EATA [50] |  CoTTA [66] |  Ours (K=4)
---|---|---|---|---
Avg. err (%) | 38.5 | 31.8 | 32.5 | 31.4
Mem. (MB) | 188 | 188 | 409 |  80 (58, 80%$\downarrow$)

Table 9: Comparision on gradually changing setup. To conduct experiments, we
use WRN-40 backbone on CIFAR100-C. The values in parentheses refer to memory
reduction rates compared to TENT/EATA and CoTTA, sequentially.

Method |  Con. TENT [65] |  PETL [28] | PETL+SDR |  Ours (K=4)
---|---|---|---|---
Avg. err (%) | 38.3 | 73.3 | 42.5 | 36.4
Mem. (MB) | 188 | 141 | 141 | 80

Table 10: Comparisons with methods for PETL. We compare our method with
methods [28] for parameter-efficient transfer learning (PETL) with WRN-40 on
CIFAR100-C. PETL+SDR refers to PETL with our proposed self-distilled
regularization.

Round | Con. TENT |  TS |  DO |  LS |  KD |  Ours (K=4)
---|---|---|---|---|---|---
1 | 38.3 | 37.4 | 41.0 | 38.4 | 39.8 | 36.4
10 | 99.0 | 96.1 | 96.3 | 41.1 | 40.4 | 36.3

Table 11: Comparisons with methods for continual learning. We report an
average error rate (%) of 15 corruptions using WRN-40 on CIFAR100-C. In the
table, TS: Entropy minimization with temperature scaling [21], DO: Dropout
[60], LS: Label smoothing with the pseudo label [47], and KD: Knowledge
distillation [27].

#### Comparisons with methods for continual learning.

Typical continual learning (CL) and continual TTA assume supervised and
unsupervised learning, respectively. However, since both are focused on
alleviating catastrophic forgetting, we believe that CL methods can also be
applied in continual TTA settings. The methods for addressing catastrophic
forgetting can be divided into regularization- and replay-based methods. The
former can be subdivided into weight regularization (_e.g_., CoTTA [66] and
EATA [50]) and knowledge distillation [27], while the latter includes GEM [44]
and dataset distillation [13]. Suppose dataset distillation is applied to the
continual TTA setup; for example, we can periodically replay synthetic samples
distilled from the source dataset to prevent the model from forgetting the
source knowledge during TTA. Notably, our self-distilled regularization (SDR)
is superior to conventional CL methods in terms of the efficiency of TTA in
on-device settings. Specifically, unlike previous regularization- or replay-
based methods, we do not require storing a copy of the original model or a
replay-and-train process.

To further compare our SDR with existing regularization methods, we conduct
experiments while keeping our architecture and adaptation loss but replacing
SDR with other regularizations, as shown in Table 11. The results demonstrate
that our SDR achieves superior performance compared to other regularizations.
In addition, Knowledge distillation [27] alleviates the error accumulation
effect in long-term adaptation (_e.g_., round 10), while showing limited
performance for adapting to the target domain.

#### Superiority of our approach compared to existing TTA methods.

Our work focuses on proposing an efficient architecture for continual TTA,
which has been overlooked in previous TTA studies [65, 66, 5, 42, 9, 40] by
introducing meta networks and self-distilled regularization, rather than
adaptation loss such as entropy minimization proposed in TENT [65] and EATA
[50]. Thus, our method can be used with various adaptation losses. Moreover,
even though our self-distilled regularization can be regarded as a teacher-
student distillation from original networks to meta networks, it does not
require a large activation size or the storage of an extra source model,
unlike CoTTA [66].

Method | Mem. (MB) | Round 1 | Round 4 | Round 7 | Round 10
---|---|---|---|---|---
Source | 280 | 37.2 | 37.2 | 37.2 | 37.2
Con. TENT | 2721 | 54.6 | 49.6 | 37.4 | 29.9
Con. TENT * | 2721 | 56.5 | 52.7 | 42.7 | 36.5
CoTTA * | 6418 | 56.7 | 56.7 | 56.7 | 56.7
Ours | 918 (66, 85%$\downarrow$) | 55.2 | 55.4 | 55.4 | 55.4
Ours * | 918 (66, 85%$\downarrow$) | 56.7 | 56.8 | 56.9 | 56.9

Table 12: Further experiments in semantic segmentation. We represent the
results based on mean intersection over union (mIoU). * means that the method
utilizes the same cross-entropy consistency loss. The values in parentheses
refer to memory reduction rates compared to TENT/EATA and CoTTA, sequentially.

#Partitions | WRN-28 (12) | WRN-40 (18) | ResNet-50 (16)
---|---|---|---
K=4 | 2,2,4,4 | 3,3,6,6 | 3,3,5,5
K=5 | 2,2,2,2,4 | 3,3,3,3,6 | 2,2,4,4,4

Table 13: Details of # of blocks of each partition. The list of numbers
denotes the number of residual blocks for each part of the original networks,
from the shallow to the deep parts sequentially. The values in parentheses are
the total number of residual blocks.

In addition to the results in Table 6, we improve the segmentation experiments
by comparing our approach with CoTTA [66]. As we aforementioned, our approach
has scalability with diverse adaptation loss. Thus, as shown in Table 12, we
additionally apply cross-entropy consistency loss* with multi-scaling input as
proposed in CoTTA, where we use the multi-scale factors of [0.5, 1.0, 1.5,
2.0] and flip. Our method not only achieves comparable performance with 85%
less memory than CoTTA, but shows consistent performance even for multiple
rounds while continual TENT [65] suffers from the error accumulation effect.

##  Appendix C Further implementation details

#### Partition of a pre-trained model.

As illustrated in Fig. 3, the given pre-trained model consists of three parts:
classifier, encoder, and input conv, where the encoder denotes layer1 to 4 in
the case of ResNet. Our method is applied to the encoder and we divide it into
K parts. Table 13 describes the details of the number of residual blocks for
each part of the encoder. Our method is designed to divide the shallow layers
more (_i.e_., densely) than the deep layers, improving the TTA performance as
shown in Table LABEL:tab:model_partition.

(K=4) | Kernel size= 1, padding=0 | Kernel size=3, padding=1
---|---|---
Arch | Avg. err | params $\uparrow$ | Mem. | Avg. err | params $\uparrow$ | Mem.
WRN-28 | 17.2 | 0.8% | 396 | 16.9 | 9.5% | 404
WRN-40 | 12.4 | 0.6% | 80 | 12.2 | 6.4% | 80
ResNet-50 | 14.4 | 11.8% | 296 | 14.2 | 142.2% | 394

Table 14: Kernel size in the conv layer. We report the average error rate (%),
the increase rate of the model parameters compared to the original model (%),
and the total memory consumption (MB) including the model and activation
sizes, based on the kernel size of the conv layer in meta networks.

(K=4) |  |  | Transformations
---|---|---|---
Dataset | Arch | EATA [50] | None | +Color | +Blur | +Gray
CIFAR10-C | WRN-40 | 13.0 | 12.5 | 12.3 | 12.3 | 12.2
CIFAR10-C | WRN-28 | 18.6 | 17.8 | 17.4 | 17.2 | 16.9
CIFAR100-C | WRN-40 | 37.1 | 36.9 | 36.7 | 36.6 | 36.4

Table 15: Ablation of the combination of transformations. To warm up the meta
networks, we use the following transformations in Pytorch: ColorJitter
(Color), GaussianBlur (Blur), and RandomGrayscale (Gray). We report the
average error rate (%).

(a)

(K=5) |  |  |  |  |  |  |  |  CIFAR10-C WRN-28 |  CIFAR10-C WRN-40 |  CIFAR100-C WRN-40
---|---|---|---|---|---|---|---|---|---|---
Variants |  1 |  2 |  3 |  4 |  5 |  6 |  7
I |  | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | 19.9 | 15.4 | 39.2
II | $\checkmark$ |  | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | 18.6 | 13.4 | 38.0
III |  |  | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | 18.7 | 13.7 | 38.2
IV | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | 18.6 | 12.4 | 36.7
V | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ |  | $\checkmark$ | 19.8 | 12.9 | 37.2
VI | $\checkmark$ | $\checkmark$ |  |  | $\checkmark$ |  | $\checkmark$ | 32.3 | 14.5 | 51.8
VII | $\checkmark$ | $\checkmark$ |  |  |  |  | $\checkmark$ | 20.7 | 14.9 | 40.1
XIII | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | 18.1 | 12.6 | 37.2
IX |  |  | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | 60.6 | 73.3 | 77.2
Ours | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | 16.8 | 12.1 | 36.3

(b)

Table 16: Components of meta networks. We conduct an ablation study on
components of meta networks (_i.e_., ① $\sim$ ⑦). Here, ① and ② refer to
affine transformation and standardization in a BN layer after the original
networks. ③$\sim$⑤ and ⑥$\sim$⑦, respectively, indicate modules in a
convolution block and two kinds of inputs of it. In table (b), $\checkmark$
means applying the component to meta networks.

#### Convolution layer in meta networks.

As the hyperparameters of the convolution
layer111https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html, we set
the bias to false and the stride to two if the corresponding part of the
encoder includes the stride of two; otherwise, one. As shown in the gray area
in Table 14, we conduct experiments by modifying the kernel size and padding
for each architecture. To be more specific, we obtain better performances by
setting the kernel size to three with WideResNet (with 10% additional number
of model parameters). On the other hand, utilizing the kernel size of three
with ResNet leads to significant increases in parameters and memory sizes.
Thus, we use one and three as the kernel size with ResNet and WideResNet,
respectively.

#### Warming up meta networks.

Before the model deployment, we warm up meta networks with the source data by
applying the following transformations, which prevent the meta networks from
being overfitted to the source domain.

Regardless of the pre-trained model’s architecture and pre-training method, we
use the same transformations to warm up meta networks. Even for WideResNet-40
pre-trained with AugMix [26], a strong data augmentation technique, the
following simple transformations are enough to warm up the meta networks. In
addition, we provide the ablation of the combination of transformations in
Table 15.

⬇

from torchvision import transforms as T

TRANSFORMS = torch.nn.Sequential(

RandomApply(T.ColorJitter(0.4,0.4,0.4,0.1), p=0.4)

RandomApply(T.GaussianBlur((3,3), p=0.2)

T.RandomGrayscale(P=0.1))

#### Semantic segmentation.

For semantic segmentation experiments, we utilize ResNet-50-based DeepLabV3+
[7] from RobustNet repository222https://github.com/shachoi/RobustNet [8]. We
warm up the meta networks on the train set of Cityscapes [10] with SGD
optimizer with the learning rate of 5e-2 and the epoch of 5. Image
transformations follow the implementation details of [8]. After model
deployment, we perform TTA using SGD optimizer with the learning rate of 1e-5,
the image size of 1600$\times$800, the batch size of 2, and the importance of
regularization $\lambda$ of 2. The main loss for adaptation is same as
$\mathcal{L}^{ent}$ in Eq. (2).

##  Appendix D Additional ablations

#### Main task loss for adaptation.

To adapt to the target domain effectively, selecting the main task loss for
adaptation is a non-trivial problem. So, we conduct a comparative experiment
on three types of adaptation loss: $\mathcal{L}^{1}$) entropy minimization
[19], $\mathcal{L}^{2}$) entropy minimization with mean entropy maximization
[37], and $\mathcal{L}^{3}$) filtering samples using entropy minimization
[50]. With a mini-batch of $N$ test images, the three adaptation losses are
formulated as follows:

$\displaystyle\mathcal{L}^{1}=\frac{1}{N}\sum_{i=1}^{N}H(\hat{y}_{i}),$ (5)
$\displaystyle\mathcal{L}^{2}=\lambda_{m_{1}}\frac{1}{N}\sum_{i=1}^{N}H(\hat{y}_{i})-\lambda_{m_{2}}H(\overline{y}),$
(6)
$\displaystyle\mathcal{L}^{3}=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}_{\\{H(\hat{y}_{i})<H_{0}\\}}\cdot
H(\hat{y}_{i}),$ (7)

where $\hat{y}_{i}$ is the logits output of $i$-th test data,
$\overline{y}=\frac{1}{N}\sum_{i=1}^{N}p(\hat{y}_{i})$,
$H(y)=-\sum_{C}p(y)\log p(y)$, p($\cdot$) is the softmax function, C is the
number of classes, and $\mathbb{I}_{\\{\cdot\\}}$ is an indicator function.
$\lambda_{m_{1}}$ and $\lambda_{m_{2}}$ indicate the importance of each term
in Eq. (6) which are set to 0.2 and 0.25, respectively, following SWR&NSP [9].
The entropy threshold $H_{0}$ is set to $0.4\times\ln C$ following EATA [50].

The results are described in Table 17. Particularly, applying any of the three
losses, our method achieves comparable performance to EATA. Among them, using
$\mathcal{L}^{3}$ of Eq. (7) achieves the lowest error rate in most cases.
Therefore, we apply $\mathcal{L}^{3}$ to our approach as mentioned in Section
3.1.

(K=5) |  |  | Ours
---|---|---|---
Dataset | Arch |  EATA [50] |  $\mathcal{L}^{1}$ |  $\mathcal{L}^{2}$ |  $\mathcal{L}^{3}$
CIFAR10-C | WRN-28 | 18.6 | 17.3 | 16.9 | 16.9
WRN-40 | 13.0 | 12.2 | 12.3 | 12.1
Resnet-50 | 14.2 | 15.0 | 14.3 | 14.1
CIFAR100-C | WRN-40 | 37.1 | 36.5 | 36.4 | 36.3
Resnet-50 | 39.9 | 40.7 | 38.8 | 39.4

Table 17: Ablation study of main task loss. We compare the average error rate
(%) of three types of adaptation losses.

(K=5) |  | Ours
---|---|---
Dataset | Arch | MSE loss (Eq. (8)) | L1 loss (Eq. (4))
CIFAR10-C | WRN-28 | 16.9 | 16.9
WRN-40 | 12.3 | 12.1
Resnet-50 | 14.1 | 14.1
CIFAR100-C | WRN-40 | 36.6 | 36.3
Resnet-50 | 39.5 | 39.4

Table 18: Ablation study of loss function of our regularization. We present
the average error (%) according to two types of loss functions for self-
distilled regularization.

#### Components of meta networks.

As shown in Table 16, we conduct an ablation study on each element of our
proposed meta networks. We observe that the affine transformation is more
critical than standardization in a BN layer after the original networks.
Specifically, removing the standardization (variant II) causes less
performance drop than removing the affine transformation (variant I). In
addition, using only a conv layer in conv block (variant VI) also cause
performance degradation, so it is crucial to use the ReLU and BN layers
together in the conv block.

#### Loss function choice of our regularization.

As mentioned in Section 3.2, self-distilled regularization loss computes the
mean absolute error (_i.e_., L1 loss) of Eq. (4). This loss regularizes the
output $\tilde{x}_{k}$ of each $k$-th group of the meta networks not to
deviate from the output $x_{k}$ of each $k$-th part of frozen original
networks. The mean squared error (_i.e_., MSE loss) also can be used to get a
similar effect which is defined as:

$MSE=(\tilde{x}_{k}-x_{k})^{2}.$ (8)

We compare two kinds of loss functions for our regularization in Table 18. By
observing a marginal performance difference, our method is robust to the loss
function choice.

#### Robustness to the importance of regularization $\lambda$.

We show that our method is robust to the regularization term $\lambda$. We
conduct experiments using a wide range of $\lambda$ as shown in Fig. 6 and the
following table.

Round $\setminus\lambda$ | 0 | 0.1 | 0.5 | 1 | 2 | 5 | 10
---|---|---|---|---|---|---|---
1 | 36.31 | 36.30 | 36.29 | 36.56 | 37.20 | 38.41 | 39.58
10 | 55.47 | 43.83 | 36.42 | 36.14 | 36.48 | 37.47 | 38.95

The experiments are performed with WideResNet-40 on CIFAR100-C. When $\lambda$
is changed from 0.5 to 1, the performance difference was only 0.27% in the
first round. We also test $\lambda$ to be extremely large (_e.g_., 5, 8, and
10). Since setting $\lambda$ to 10 may mean that we hardly adapt the meta
networks to the target domain, the error rate (39.58%) with $\lambda$ of 10
was close to the one (41.1%) of BN Stats Adapt [49].

##  Appendix E Baseline details

###  E.1 TTA works

We refer to the baselines for which the code was officially released:
TENT333https://github.com/DequanWang/tent, TTT++444https://github.com/vita-
epfl/ttt-plus-plus, CoTTA555https://github.com/qinenergy/cotta,
EATA666https://github.com/mr-eggplant/EATA, and
NOTE777https://github.com/TaesikGong/NOTE. We did experiments on their code by
adding the needed data loader or pre-trained model loader. In this section,
implementation details of the baselines are provided.

#### BN Stats Adapt [49]

is one of the non-training TTA approaches. It can be implemented by setting
the model to the train
mode888pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.train
of Pytorch [54] during TTA.

#### TTT+++ [42]

was originally implemented as the offline adaptation, _i.e_., multi-epoch
training. So, we modified their setup to continual TTA. We further tuned the
learning rate as 0.005 and 0.00025 for adapting to CIFAR10-C and CIFAR100-C,
respectively.

#### NOTE [17]

proposed the methods named IABN and PBRS with taking account of temporally
correlated target data. However, our experiments were conducted with target
data that was independent and identically distributed (i.i.d.). Hence, we
adapted NOTE-i.i.d (_i.e_., NOTE* in their git repository), which is a
combination of TENT [65] and IABN without using PBRS. We fine-tuned the
$\alpha$ of their main paper (_i.e_., self.k in the
code999https://github.com/TaesikGong/NOTE/blob/main/utils/iabn.py) to 8 and
the learning rate to 1e-5.

#### Others

(_e.g_., TENT [65], SWR&NSP [9], CoTTA [66], and EATA [50]). We utilized the
best hyperparameters specified in their paper and code. In the case where the
batch size of their works (_e.g_., 200 and 256) differs from one for our
experiments (_e.g_., 64), we decreased the learning rate linearly based on the
batch size [18].

#### AdaptBN [55].

We set the hyperparameter $N$ of their main paper to 8. When AdaptBN is
employed alongside TENT or our approach, we set the learning rate to 1e-5 or
5e-6 [40].

###  E.2 On-device learning works

To unify the backbone network as ResNet-50 [24], we reproduced the following
works by referencing their paper and published code:
TinyTL101010https://github.com/mit-han-lab/tinyml/tree/master/tinytl, Rep-
Net111111https://github.com/ASU-ESIC-FAN-Lab/RepNet, and AuxAdapt. This
section presents additional implementation details for reproducing the above
three works.

Time |  | $t\xrightarrow{\hskip 489.38731pt}$ |  |
---|---|---|---|---
Arch | Method | Gaus. | Shot | Impu. | Defo. | Glas. | Moti. | Zoom | Snow | Fros. | Fog | Brig. | Cont. | Elas. | Pixe. | Jpeg | Avg. err | Mem.
WRN-40 (AugMix) | Source | 44.3 | 37.0 | 44.8 | 30.6 | 43.9 | 32.6 | 29.4 | 23.9 | 30.1 | 39.7 | 12.9 | 66.4 | 32.7 | 58.4 | 23.5 | 36.7 | 11
tBN [49] | 19.5 | 17.6 | 23.8 | 9.6 | 23.1 | 11.1 | 10.3 | 13.4 | 14.2 | 15.0 | 8.0 | 13.9 | 17.3 | 16.0 | 18.8 | 15.4 | 11
Single do. TENT [65] | 16.4 | 13.9 | 19.1 | 8.3 | 19.1 | 9.3 | 8.6 | 10.9 | 11.3 | 12.0 | 6.9 | 11.6 | 14.6 | 12.2 | 15.6 | 12.7 | 188
TENT continual [65] | 16.4 | 12.2 | 17.1 | 9.1 | 18.7 | 11.4 | 10.4 | 12.7 | 12.4 | 14.8 | 10.1 | 13.0 | 17.0 | 13.3 | 19.0 | 13.3 | 188
TTT++ [42] | 19.1 | 16.9 | 22.2 | 9.3 | 21.6 | 10.8 | 9.8 | 12.7 | 13.1 | 14.3 | 7.8 | 13.9 | 15.9 | 14.2 | 17.2 | 14.6 | 391
SWRNSP [9] | 15.9 | 13.3 | 18.2 | 8.4 | 18.5 | 9.5 | 8.6 | 11.0 | 10.2 | 11.7 | 7.0 | 8.1 | 14.6 | 11.3 | 15.1 | 12.1 | 400
NOTE [17] | 19.6 | 16.4 | 19.9 | 9.4 | 20.3 | 10.3 | 10.1 | 11.6 | 10.6 | 13.3 | 7.9 | 7.7 | 15.4 | 12.0 | 17.3 | 13.4 | 188
EATA [50] | 15.2 | 13.1 | 17.5 | 9.5 | 19.9 | 11.6 | 9.3 | 11.4 | 11.5 | 12.4 | 7.8 | 11.1 | 16.1 | 12.2 | 16.1 | 13.0 | 188
CoTTA [66] | 15.6 | 13.6 | 17.3 | 9.8 | 19.0 | 11.0 | 10.2 | 13.5 | 12.6 | 17.4 | 7.8 | 17.3 | 16.2 | 12.9 | 16.0 | 14.0 | 409
Ours (K=4) | 16.1 | 13.2 | 18.3 | 8.0 | 18.3 | 9.3 | 8.6 | 10.5 | 10.1 | 12.2 | 6.8 | 11.3 | 14.5 | 11.0 | 14.8 | 12.2 | 80
Ours (K=5) | 15.9 | 12.6 | 17.2 | 8.2 | 18.4 | 9.3 | 8.6 | 10.6 | 10.4 | 12.4 | 6.7 | 11.7 | 14.3 | 11.3 | 14.9 | 12.1 | 92
WRN-28 | Source | 72.3 | 65.7 | 72.9 | 46.9 | 54.3 | 34.8 | 42.0 | 25.1 | 41.3 | 26.0 | 9.3 | 46.7 | 26.6 | 58.5 | 30.3 | 43.5 | 58
tBN [49] | 28.6 | 26.8 | 37.0 | 13.2 | 35.4 | 14.4 | 12.6 | 18.0 | 18.2 | 16.0 | 8.6 | 13.3 | 24.0 | 20.3 | 27.8 | 20.9 | 58
Single do. TENT [65] | 25.2 | 23.8 | 33.5 | 12.8 | 32.3 | 14.1 | 11.7 | 16.4 | 17.0 | 14.4 | 8.4 | 12.2 | 22.8 | 18.0 | 24.8 | 19.2 | 646
Continual TENT [65] | 25.2 | 20.8 | 29.8 | 14.4 | 31.5 | 15.4 | 14.2 | 18.8 | 17.5 | 17.3 | 10.9 | 14.9 | 23.6 | 20.2 | 25.6 | 20.0 | 646
TTT++ [42] | 27.9 | 25.8 | 35.8 | 13.0 | 34.3 | 14.2 | 12.2 | 17.4 | 17.6 | 15.5 | 8.6 | 13.1 | 23.1 | 19.6 | 26.6 | 20.3 | 1405
SWRNSP [9] | 24.6 | 20.5 | 29.3 | 12.4 | 31.1 | 13.0 | 11.3 | 15.3 | 14.7 | 11.7 | 7.8 | 9.3 | 21.5 | 15.6 | 20.3 | 17.2 | 1551
NOTE [17] | 30.4 | 26.7 | 34.6 | 13.6 | 36.3 | 13.7 | 13.9 | 17.2 | 15.8 | 15.2 | 9.1 | 7.5 | 24.1 | 18.4 | 25.9 | 20.2 | 646
EATA [50] | 23.8 | 18.8 | 27.3 | 13.9 | 29.7 | 16.0 | 13.3 | 18.0 | 16.9 | 15.7 | 10.5 | 12.2 | 22.9 | 17.1 | 23.0 | 18.6 | 646
CoTTA [66] | 24.6 | 21.6 | 26.5 | 12.1 | 28.0 | 13.0 | 10.9 | 15.3 | 14.6 | 13.6 | 8.1 | 12.2 | 20.0 | 14.9 | 19.5 | 17.0 | 1697
Ours (K=4) | 23.5 | 19.0 | 26.6 | 11.5 | 30.6 | 13.1 | 10.9 | 15.2 | 14.5 | 13.1 | 7.8 | 11.4 | 20.9 | 15.4 | 20.8 | 16.9 | 404
Ours (K=5) | 23.8 | 18.7 | 25.7 | 11.5 | 29.8 | 13.3 | 11.3 | 15.3 | 15.0 | 13.0 | 7.9 | 11.3 | 20.2 | 15.1 | 20.5 | 16.8 | 471
Resnet-50 | Source | 65.6 | 60.7 | 74.4 | 28.9 | 79.9 | 46.0 | 25.7 | 35.0 | 49.4 | 54.7 | 13.0 | 83.2 | 41.2 | 46.7 | 27.7 | 48.8 | 91
tBN [49] | 18.0 | 17.2 | 29.3 | 10.7 | 27.2 | 15.5 | 8.9 | 16.7 | 14.6 | 21.0 | 9.3 | 12.7 | 20.9 | 12.4 | 14.8 | 16.6 | 91
Single do. TENT [65] | 16.6 | 15.7 | 25.7 | 10.0 | 24.8 | 13.8 | 8.3 | 14.9 | 13.8 | 17.6 | 8.7 | 10.0 | 19.1 | 11.5 | 13.8 | 15.0 | 925
TENT continual [65] | 16.6 | 14.4 | 22.9 | 10.4 | 22.6 | 13.4 | 10.3 | 15.8 | 14.6 | 18.0 | 10.5 | 11.7 | 18.4 | 13.1 | 15.3 | 15.2 | 925
TTT++ [42] | 18.2 | 16.9 | 28.7 | 10.5 | 26.5 | 14.5 | 8.9 | 16.5 | 14.5 | 20.9 | 9.0 | 9.0 | 20.4 | 12.3 | 14.7 | 16.1 | 1877
SWRNSP [9] | 17.3 | 16.1 | 26.1 | 10.6 | 25.6 | 14.1 | 8.7 | 15.6 | 13.6 | 18.6 | 8.8 | 10.0 | 19.3 | 12.0 | 14.2 | 15.4 | 1971
EATA [50] | 17.2 | 14.9 | 23.6 | 10.2 | 23.3 | 13.2 | 8.5 | 14.0 | 12.5 | 16.6 | 8.6 | 9.4 | 17.2 | 11.0 | 12.7 | 14.2 | 925
CoTTA [66] | 16.2 | 15.0 | 21.2 | 10.4 | 22.8 | 13.9 | 8.4 | 15.1 | 12.9 | 19.8 | 8.6 | 11.3 | 17.5 | 10.5 | 12.2 | 14.4 | 2066
Ours (K=4) | 16.5 | 14.5 | 24.3 | 9.7 | 23.7 | 13.3 | 8.8 | 14.7 | 12.9 | 17.0 | 9.1 | 9.4 | 17.6 | 11.4 | 13.1 | 14.4 | 296
Ours (K=5) | 16.6 | 14.4 | 23.6 | 9.8 | 23.4 | 12.7 | 8.6 | 14.5 | 12.6 | 16.6 | 8.7 | 9.0 | 17.0 | 11.3 | 12.6 | 14.1 | 498

Table 19: Comparison of error rate ($\%$) on CIFARC10-C with severity level 5.
We conduct experiments on continual TTA setup. Avg. err means the average
error rate (%) of all 15 corruptions, and Mem. denotes total memory
consumption, including model parameter sizes and activations. WRN refers to
WideResNet. The implementation details of the baselines are described in
Section E.1.

Time |  | $t\xrightarrow{\hskip 489.38731pt}$ |  |
---|---|---|---|---
Arch | Method | Gaus. | Shot | Impu. | Defo. | Glas. | Moti. | Zoom | Snow | Fros. | Fog | Brig. | Cont. | Elas. | Pixe. | Jpeg | Avg. err | Mem.
WRN-40 (AugMix) | Source | 80.1 | 77.0 | 76.4 | 59.9 | 77.6 | 64.2 | 59.3 | 64.8 | 71.3 | 78.3 | 48.1 | 83.4 | 65.8 | 80.4 | 59.2 | 69.7 | 11
tBN [49] | 45.9 | 45.6 | 48.2 | 33.6 | 47.9 | 34.5 | 34.1 | 40.3 | 40.4 | 47.1 | 31.7 | 39.7 | 42.7 | 39.2 | 45.6 | 41.1 | 11
Single do. TENT [65] | 41.2 | 40.6 | 42.2 | 30.9 | 43.4 | 31.8 | 30.6 | 35.3 | 36.2 | 40.1 | 28.5 | 35.5 | 39.1 | 33.9 | 41.7 | 36.7 | 188
continual TENT [65] | 41.2 | 38.2 | 41.0 | 32.9 | 43.9 | 34.9 | 33.2 | 37.7 | 37.2 | 41.5 | 33.2 | 37.2 | 41.1 | 35.9 | 45.1 | 38.3 | 188
TTT++ [42] | 46.0 | 45.4 | 48.2 | 33.5 | 47.7 | 34.4 | 33.8 | 39.9 | 40.2 | 47.1 | 31.8 | 39.7 | 42.5 | 38.9 | 45.5 | 41.0 | 391
SWRNSP [9] | 42.4 | 40.9 | 42.7 | 30.6 | 43.9 | 31.7 | 31.3 | 36.1 | 36.2 | 41.5 | 28.7 | 34.1 | 39.2 | 33.6 | 41.3 | 36.6 | 400
NOTE [17] | 50.9 | 47.4 | 49.0 | 37.3 | 49.6 | 37.3 | 37.0 | 41.3 | 39.9 | 47.0 | 35.2 | 34.7 | 45.2 | 40.9 | 49.9 | 42.8 | 188
EATA [50] | 41.6 | 39.9 | 41.2 | 31.7 | 44.0 | 32.4 | 31.9 | 36.2 | 36.8 | 39.7 | 29.1 | 34.4 | 39.9 | 34.2 | 42.2 | 37.1 | 188
CoTTA [66] | 43.5 | 41.7 | 43.7 | 32.2 | 43.7 | 32.8 | 32.2 | 38.5 | 37.6 | 45.9 | 29.0 | 38.1 | 39.2 | 33.8 | 39.4 | 38.1 | 409
Ours (K=4) | 42.7 | 39.6 | 42.4 | 31.4 | 42.9 | 31.9 | 30.8 | 35.1 | 34.8 | 40.7 | 28.1 | 35.0 | 37.5 | 32.1 | 40.5 | 36.4 | 80
Ours (K=5) | 41.8 | 39.0 | 41.9 | 31.2 | 42.7 | 32.5 | 31.0 | 35.0 | 35.0 | 39.9 | 28.8 | 34.5 | 37.5 | 32.8 | 40.5 | 36.3 | 92
Resnet-50 | Source | 84.7 | 83.5 | 93.3 | 59.6 | 92.5 | 71.9 | 54.8 | 66.6 | 77.6 | 81.8 | 44.3 | 91.2 | 72.2 | 76.6 | 56.5 | 73.8 | 91
tBN [49] | 48.1 | 46.7 | 60.6 | 35.1 | 58.0 | 41.8 | 33.2 | 47.3 | 43.5 | 54.9 | 33.5 | 35.3 | 49.8 | 38.4 | 40.8 | 44.5 | 91
Single do. TENT [65] | 44.1 | 42.7 | 53.9 | 32.6 | 52.0 | 37.5 | 30.5 | 43.4 | 40.2 | 45.7 | 30.4 | 31.4 | 45.1 | 35.0 | 37.6 | 40.1 | 926
continual TENT [65] | 44.0 | 40.1 | 49.9 | 34.7 | 50.6 | 40.0 | 33.6 | 47.0 | 45.7 | 53.4 | 42.5 | 46.2 | 56.1 | 51.2 | 53.3 | 45.9 | 926
TTT++ [42] | 48.1 | 46.5 | 60.8 | 35.1 | 57.8 | 41.6 | 32.9 | 46.8 | 43.3 | 55.0 | 33.3 | 34.0 | 50.0 | 38.1 | 40.6 | 44.2 | 1876
SWRNSP [9] | 48.3 | 46.5 | 60.5 | 35.1 | 57.9 | 41.7 | 32.9 | 47.1 | 43.5 | 54.7 | 33.5 | 35.1 | 49.9 | 38.3 | 40.7 | 44.1 | 1970
EATA [50] | 44.8 | 41.9 | 52.6 | 33.0 | 51.1 | 37.8 | 30.3 | 43.0 | 40.1 | 45.1 | 30.1 | 31.8 | 45.2 | 35.2 | 37.4 | 39.9 | 926
CoTTA [66] | 43.6 | 42.8 | 50.4 | 34.2 | 51.6 | 39.2 | 31.4 | 43.4 | 39.6 | 47.4 | 31.3 | 32.2 | 43.4 | 35.8 | 36.7 | 40.2 | 2064
Ours (K=4) | 44.8 | 40.3 | 49.2 | 32.3 | 50.1 | 36.3 | 29.5 | 41.0 | 39.9 | 44.6 | 31.5 | 33.7 | 45.3 | 36.3 | 37.7 | 39.5 | 296
Ours (K=5) | 44.9 | 40.4 | 48.9 | 32.7 | 49.7 | 36.9 | 29.3 | 40.8 | 39.0 | 44.4 | 31.1 | 33.6 | 44.0 | 35.7 | 37.8 | 39.3 | 498

Table 20: Comparison of error rate ($\%$) on CIFARC100-C with severity level
5. We conduct experiments on continual TTA setup. Avg. err means the average
error rate (%) of all 15 corruptions, and Mem. denotes total memory
consumption, including model parameter sizes and activations. WRN refers to
WideResNet. The implementation details of the baselines are described in
Section E.1.

#### TinyTL [4].

We attach the LiteResidualModules121212https://github.com/mit-han-
lab/tinyml/blob/master/tinytl/tinytl/model/modules.py to layer1 to 4 in the
case of
ResNet-50131313https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py.
As the hyperparameters of the LiteResidualModules, the hyperparameter expand
is set to 4 while the other hyperparameters follow the default values.

#### Rep-Net [69].

We divide the encoder of ResNet-50 into six parts, as each part of the encoder
has 2,2,3,3,3,3 residual blocks (_e.g_., BasicBlock or Bottleneck in Pytorch)
from the shallow to the deep parts sequentially. Then, we connect the
ProgramModules141414github.com/ASU-ESIC-FAN-
Lab/RepNet/blob/master/repnet/model/reprogram.py to each corresponding part of
the encoder. For the ProgramModule, we set the hyperparameter expand to 4
while the rest hyperparameters are used as their default values. We copy the
input conv of ResNet-50 and make use of it as the input conv of Rep-Net.

#### AuxAdapt [73].

We use ResNet-18 as the AuxNet. We create pseudo labels by fusing the logits
output of ResNet-50 and ResNet-18, and optimize all parameters of ResNet-18
using the pseudo labels with cross-entropy loss.

#### Warming up the additional modules.

Before model deployment, we pre-train the additional modules (_i.e_., the
LiteResidualModule of TinyTL [4], the ProgramModule of Rep-Net [69], and the
AuxNet of AuxAdapt [73]) on the source data using the same strategy warming up
the meta networks as mentioned in Section C.

##  Appendix F Results of all corruptions

We report the error rates (%) of all corruptions on continual TTA and memory
consumption (MB) including the model parameters and activations in Table 19
and Table 20. These tables contain additional details to Table 1.