# Realistic Evaluation of Test-Time Adaptation:
Unsupervised Hyperparameter Selection

S. Cygert1,2, D. Sójka1,3, T. Trzciński1,4,6, B. Twardowski1,5

# Realistic Evaluation of Test-Time Adaptation Algorithms:
Unsupervised Hyperparameter Selection

S. Cygert1,2, D. Sójka1,3, T. Trzciński1,4,6, B. Twardowski1,5

###### Abstract

Test-Time Adaptation (TTA) has recently emerged as a promising strategy for
tackling the problem of machine learning model robustness under distribution
shifts by adapting the model during inference without access to any labels.
Because of task difficulty, hyperparameters strongly influence the
effectiveness of adaptation. However, the literature has provided little
exploration into optimal hyperparameter selection. In this work, we tackle
this problem by evaluating existing TTA methods using surrogate-based hp-
selection strategies (which do not assume access to the test labels) to obtain
a more realistic evaluation of their performance. We show that some of the
recent state-of-the-art methods exhibit inferior performance compared to the
previous algorithms when using our more realistic evaluation setup. Further,
we show that forgetting is still a problem in TTA as the only method that is
robust to hp-selection resets the model to the initial state at every step. We
analyze different types of unsupervised selection strategies, and while they
work reasonably well in most scenarios, the only strategies that work
consistently well use some kind of supervision (either by a limited number of
annotated test samples or by using pretraining data). Our findings underscore
the need for further research with more rigorous benchmarking by explicitly
stating model selection strategies, to facilitate which we open-source our
code111The code will be released upon acceptance..

## Introduction

Machine learning models typically assume that training and testing data
originate from similar distribution. However, in real-world applications, one
observes a distribution shift between the training (source) and testing
(target) data domains, which, in turn, hurts the performance of the model
during inference (Geirhos et al. 2019; Hendrycks and Dietterich 2019b; Koh et
al. 2021). Unsupervised domain adaptation methods explore adapting a model
trained on the source domain to the target one, having access only to
unlabelled target data (Gong, Grauman, and Sha 2013). Test-time adaptation
(TTA) extends this scenario to the online setting, where the model needs to
adapt on the fly.

Recently, various approaches for test-time adaptation were developed. An
example strategy involves entropy minimization on test data using various
pseudo-labeling approaches (Wang et al. 2021; Niu et al. 2022, 2023). Yet,
without any supervision at test time, errors in pseudo-labels are unavoidable,
which can result in error accumulation (Chen et al. 2019) and pose a great
challenge for TTA methods. As a result, it was shown that existing TTA methods
are very vulnerable to the choice of hyperparameters (such as learning rate or
momentum of the optimizer) (Boudiaf et al. 2022; Zhao et al. 2023).

Apart from the inherent difficulty of TTA one subtle but important challenge
arises: how to select the hyperparameters for a given method and testing
scenario, in a practical way, that is without access to test labels? This
question has received little attention in the literature, with most of the
methods, not specifying model selection strategy (Wang et al. 2021; Chen et
al. 2022; Wang et al. 2022; Niu et al. 2022; Gong et al. 2022; Niu et al.
2023; Döbler, Marsden, and Yang 2023; Sójka et al. 2023; Yuan, Xie, and Li
2023; Liu et al. 2024; Su, Xu, and Jia 2024), and only listing the final
hyperparameters. To provide a fair comparison, recent benchmark (Zhao et al.
2023) compared accuracies of various TTA methods using oracle selection
strategy, that is assuming access to the test data labels. However, this can
only give an overoptimistic measure of their potential performance, as in
practice, we won’t have access to test labels to perform optimal model
selection. Also, this can give an advantage to methods with more
hyperparameters.

To provide a realistic evaluation of TTA methods, we propose and evaluate
different surrogate measures for model selection in TTA that do not use test
labels. While the problem of unsupervised model selection is also known from a
domain adaptation perspective (Musgrave, Belongie, and Lim 2022; Salvador et
al. 2022), in this work, we argue that TTA presents a unique challenge due to
its dynamic nature, e.g., by showing that simply increasing the length of the
adaptation sequence may result in degenerated solutions. Some of our findings
include:

  * •

The ranking of TTA methods varies across different selection strategies, i.e.,
some methods that perform well using the ORACLE selection strategy
underperform when more realistic (unsupervised) hyperparameter selection.

  * •

We show that forgetting is an important problem in TTA as the only method that
is robust to hp-selection resets the model to the initial state at every step
(which comes at a high computational cost).

  * •

None of the existing model selection strategies is superior to others and able
to match oracle’s strategy consistently.

  * •

When any kind of supervision is prohibited, we show that existing TTA methods
can be grouped by their optimization objective and accompanied with an
unsupervised hp-selection that works close to optimal. However, those findings
do not hold after we increase the adaptation difficulty and can result in
degenerated solutions.

  * •

Across all evaluated metrics, using some source of supervision (using a very
limited number of annotated samples or source data) is the only strategy that
works universally well across all evaluated scenarios.

  * •

We show that the main problem for unsupervised hyperparameter selection
strategies is a small margin between viable and suboptimal solutions.

In conclusion, the hyperparameter tuning approach employed in the existing
evaluation protocols is impractical and fails to adequately evaluate the
capabilities of each TTA method. Our work emphasizes the importance of
conducting thorough assessments using clearly defined model selection
strategies, for which we create a testbed.

## Related Work

Model selection in TTA Model selection is a known problem in literature
(Raschka 2018). The situation complicates for TTA where we cannot assume
access to the validation set. This raises concerns for TTA, as existing
methods are vulnerable to changes in hyperparameters (Boudiaf et al. 2022; Su
et al. 2024). Recently, Zhao et al. (2023) evaluated TTA methods assuming
oracle selection strategy, that is, by accessing test labels. However, this
may not give a fair comparison between different methods as it may favor
methods with more hyperparameters. To the best of our knowledge, only two
papers have used model selection strategies that do not access access to
target labels. Rusak et al. (2022), have used cross-dataset validation to
select hyperparameters. In particular, they used the optimal parameters
selected from the ImageNet-C dataset for testing on different benchmarks.
Analogous strategy for parameter selection was used in Boudiaf et al. (2022).
The authors found that existing TTA methods lack hyperparameter
transferability, and they proposed a parameter-free method (lame).

Unsupervised model selection For domain adaptation, Gulrajani and Lopez-Paz
(2021) stressed the importance of providing model selection strategies. They
tested different strategies for unsupervised model selection: i) source
dataset accuracy ii) cross-dataset accuracy and iii) limited access to the
target labels and have shown that under such fair comparison, no domain
adaptation method works better than standard empirical risk minimization
training. Recently, unsupervised model selection strategies were evaluated for
domain (Musgrave, Belongie, and Lim 2022) and partial domain adaptation
(Salvador et al. 2022) and in time-series anomaly detection (Goswami et al.
2023). We are inspired by those studies and examine various strategies for
TTA.

## Test-time adaptation preliminaries

Given the source model $f_{\theta_{S}}$, trained on the labeled data from the
source domain $\mathcal{D}_{S}=\\{\mathcal{X}_{S},\mathcal{Y}_{S}\\}$, the
objective is to adapt it to the test data
$\mathcal{D}_{T}=\\{\mathcal{X}_{T},\mathcal{Y}_{T}\\}$. In the source data
$\mathcal{D}_{S}$, sample-label pairs
$(x_{i}\in\mathcal{X}_{S},y_{i}\in\mathcal{Y}_{S})$ are distributed according
to the probability distribution $\mathcal{P}_{S}(\boldsymbol{x},y)$. The test
samples $x_{i}\in\mathcal{X}_{T}$ are modeled by a different distribution
$\mathcal{P}_{T}(\boldsymbol{x},y)\neq\mathcal{P}_{S}(\boldsymbol{x},y)$. At
each testing time step $t$, the TTA process aims to adapt the model
$f_{\theta_{S}}$ to better align with the distribution $\mathcal{P}_{T}$.
Importantly, unlabeled data samples $\boldsymbol{x}^{t}\sim\mathcal{P}_{T}$
are the only information available regarding the unseen distribution
$\mathcal{P}_{T}$.

Hyperparameter selection is a crucial part of all machine learning pipelines.
In a standard supervised learning setting, a validation set can be used to
estimate the model’s accuracy. However, in TTA such an approach is not
possible as there is no access to labeled target samples. Below, we discuss
several approaches for model selection in TTA, which we use in this work.

### Improved hyperparameter selection in TTA

There is a lack of rigorous and realistic hyperparameter selection in TTA, as
it typically assumes the availability of test labels. This assumption, we
argue, may give an overoptimistic measure of the TTA method’s performance and
does not compare them fairly. Therefore, we design a protocol of more
realistic hyperparameter selection, and to achieve that, we examine the
feasibility of the following selection strategies, which do not assume access
to test labels.

Source accuracy (s-acc) The most simple strategy uses a validation set from
the source domain to estimate the model’s performance in the target domain,
e.g., in domain adaptation (Ganin and Lempitsky 2015). Interestingly, Taori et
al. (2020) and Miller et al. (2021) show that there is a linear correlation
between accuracy on i.i.d. and o.o.d. data which may favor such a strategy.
However, there are also some contradictory results that show that this may not
be always true (You et al. 2019; Teney et al. 2023). Hence, it is important to
validate this strategy in TTA settings. Finally, it is important to note that
some TTA methods assume there is no access to the source data, e.g., because
of privacy concerns (Liang, Hu, and Feng 2020). In such applications, this
strategy may not be valid.

Cross-validation accuracy (c-acc) Another viable approach is to select
hyperparameters on another dataset. This is a potentially valid strategy since
plenty of benchmarks exist for TTA that could be utilized for this purpose.
Here, we follow Rusak et al. (2022) and perform hyperparameter selection on
ImageNet-C dataset, which they show to work well. When the target dataset of
the TTA procedure is ImageNet-C we use CIFAR-100-C for fair comparison.

Entropy (ent) Minimizing the Shannon entropy
$H(\hat{y})=-\sum_{c}p(\hat{y}_{c})\log p(\hat{y}_{c})$, where $\hat{y}_{c}$
is a probability prediction of class $c$, of predictions for target samples
$\hat{y}=f_{\theta}(\boldsymbol{x}^{t})$ was introduced for test-time
adaptation in Wang et al. (2021) because i) it is related to error, as more
confident predictions are all-in-all more correct and ii) entropy is related
to shifts due to corruption, as more corruption results in more entropy. This
strategy was adopted by several works showing its efficiency (Wang et al.
2021; Niu et al. 2022, 2023). However, as shown by Boudiaf et al. (2022) in
some scenarios the entropy does not correlate with the accuracy. One possible
failure mode is a model that is incorrectly confident. For example, the model
might incorrectly classify all samples in the dataset as belonging to the same
class, which results in low entropy but also small model accuracy.

Model consistency (con) Consistency regularization is an important component
of many self-supervised learning algorithms (Sohn et al. 2020). It enforces
the model $f_{\theta}$ to have similar predictions when fed perturbed versions
$\boldsymbol{\tilde{x}}_{i}$ of the same image $\boldsymbol{x}_{i}$
($f_{\theta}(\boldsymbol{x}_{i})\simeq
f_{\theta}(\boldsymbol{\tilde{x}}_{i})$). It was also used to drive training
of the model during the TTA phase (Nguyen et al. 2023). As the choice of
perturbations, we use the augmentation pipeline introduced in CoTTA (Wang et
al. 2022), as it is quite commonly used in TTA and was shown to work well,
which include, e.g., color jittering, Gaussian noise, and blur. Further
information can be found in the appendix.

Soft neighborhood density (snd) SND metric was proposed for unsupervised model
selection in domain adaptation (Saito et al. 2021). It is based on the
intuition that a well-trained model embeds target samples of the same class
nearby, forming dense neighborhoods in feature space. A high snd score means
that each feature is close to many other features with similar pseudo-labels,
which can indicate good clustering. A failure mode for this metric is
predicting all features into a single cluster.

Using test labels (oracle) We also incorporate oracle hyperparameter selection
strategy to measure upper-bound for evaluated methods. Additionally, we use
one more additional selection strategy, 100-rnd, that assumes limited access
to the test data: following Salvador et al. (2022), we use labels from 100
randomly selected images from test data.

Median (med) Finally, we show results when selecting a median model. It shows
whether a given selection strategy works better than chosing an average model
and the sensitivity of TTA methods to hyperparameters, i.e., a highly robust
model will have a median close to the oracle results.

All methods are compared against the source model, which assumes no adaptation
at all.

## Experimental Setup

TTA Methods | TENT, SAR, LAME, EATA, RMT, AdaContrast, MEMO
---|---
Model Selection |  s-acc, ent, c-acc, con, snd 100-rnd, oracle,
Datasets | CIFAR100-C, ImageNet-C, DomainNet-126, ImageNet-R, ImageNetV2
Tuned parameters | learning rate, momentum and method-specific parameters
Experimental protocol | 5 datasets, varying levels of temporal correlation and adaptation sequence length

Table 1: Summary of experimental setup considered in this work.

### Datasets

For experiments, we utilize two widely used datasets for TTA evaluation with
artificial image corruptions: CIFAR100-C and ImageNet-C (Hendrycks and
Dietterich 2019a). Moreover, we use two datasets consisting of images without
corruptions, but portraying objects in different domains: DomainNet-126 (Saito
et al. 2019) and ImageNet-R (Hendrycks et al. 2021). Finally, we incorporate
an ImageNetV2 (Recht et al. 2019) dataset, which allows us to measure model
adaptation performance under real-world distribution shift. More details in
the appendix.

Additionally, we test the influence of temporally correlated class
distribution within the CIFAR100-C test sequence on the performance of TTA
methods. Temporal correlation of classes breaks the assumption of independence
and identically distributed (i.i.d.) data and results in performance loss. For
this purpose, we utilize Dirichlet distribution with different concentration
parameters $\delta$ to sample the class instances, similar to Gong et al.
(2022). The lower the value of $\delta$ is, the more severe the temporal
correlation becomes. Moreover, we test the scenario of a long TTA operation by
utilizing testing sequences ten times.

### Methods

In our testbed, we include the seven different methods, which differ by the
optimized objective, number of hyperparameters, fine-tuned layers, etc.

  * •

TENT (Wang et al. 2021) is a popular TTA method that includes minimizing the
entropy of the predictions while optimizing only batch norm statistics of the
model.

  * •

SAR (Niu et al. 2023) additionally applies filtering to remove noisy samples,
uses a sharpness-aware optimizer that encourages flat minima, and finally
applies a model reset scheme.

  * •

EATA (Niu et al. 2022) uses a filtering scheme that removes not only noisy
samples but also redundant ones and additionally introduces a Fisher
regularizer (Kirkpatrick et al. 2017) to constrain important model parameters
from drastic changes.

  * •

AdaContrast (Chen et al. 2022) incorporates self-supervised contrastive
objective with consistency regularization and fine-tuning all weights of the
model.

  * •

RMT method (Döbler, Marsden, and Yang 2023) also uses contrastive objective
but additionally uses it to align the learned feature space with the source
model combined with novel symmetrical cross-entropy loss.

  * •

MEMO (Zhang, Levine, and Finn 2022) performs adaptation on each image
independently by minimizing the entropy of the model’s output distribution
across a set of augmented images. This method alleviates the forgetting issue
by resetting the model at each timestep, however, this comes at a high
computational cost.

  * •

LAME (Boudiaf et al. 2022) is a parameter-free approach, that modifies only
the model’s output and solves the objective with a concave-convex procedure
(i.e., without the need to tune the learning rate). As a result,
hyperparameter selection is not required.

Access to the source data RMT method additionally assumes a replay mechanism
of the source data, which may give this method an advantage over other
methods, which were developed with the motivation of not using a memory buffer
during training. Thus, we experiment with RMT source-free variant which we
denote as RMT-SF.

### Evaluation Details

Hyperparameters For each of the algorithms, we perform a search over the two
most important hyperparameters, that is the learning rate and momentum,
similar to (Boudiaf et al. 2022), with 8 possible configurations. For SAR and
EATA we additionally search over the method-specific parameters using random
search, which gives 8 additional configurations for those methods. More
details are provided in the appendix. Each experiment is repeated 3 times.

Hyperparameter selection To select hyperparameters, we compute appropriate
statistics (entropy loss, consistency loss, etc.) during the adaptation stage.
Then, the final hyperparameters are selected based on the accumulated value of
the given metric. In case of s-acc, computing accuracy on the source data is
performed only after the whole adaptation stage to reduce the computational
cost.

Implementation details For CIFAR100-C, ResNeXt-29 (Xie et al. 2017)
architecture is used following the RobustBench benchmark (Croce et al. 2021).
For remaining datasets, a source pre-trained ResNet-50 is used as int (Döbler,
Marsden, and Yang 2023). We use SGD for all methods (except for SAR, which
uses a custom optimizer). We are interested in a batch size equal to one,
which allows for fast model adaptation. However, some methods do not work well
in the single-image adaptation setting. Therefore, we use batch size equal to
ten, which was shown in other works to still work well (Niu et al. 2023), and
is commonly used in the continual learning for online scenarios (Mai et al.
2022). Results with different batch sizes are reported in the appendix (Table
4 and 5).

Augmentations Some of the methods use data augmentation at test-time,
including AdaContrast, RMT, and MEMO, which are used in their contrastive loss
objective. Those augmentations include some of those that were used to
generate ImageNet-C and CIFAR100-C benchmarks. Therefore, it is important to
note that those augmentations can give an advantage to those methods on those
benchmarks.

## Experiments

Firstly, we analyze our main results in the standard scenario on all datasets.
Secondly, we perform experiments on less-common scenarios (e.g., when
adaptation is performed on long sequences or images with temporal
correlation). Finally, we take a detailed look at the stability of
unsupervised selection strategies by also looking at their dynamic.

Dataset | Method | s-acc | c-acc | ent | con | snd | 100-rnd | med | oracle
---|---|---|---|---|---|---|---|---|---
CIFAR100-C | source | 53.55
tent | 59.93 | 59.56 | 5.32 | 59.56 | 40.78 | 59.17 | 54.46 | 60.16
eata | 61.78 | 61.86 | 21.01 | 58.60 | 39.40 | 61.95 | 60.51 | 62.44
sar | 59.47 | 45.65 | 11.68 | 48.75 | 58.26 | 59.26 | 57.03 | 60.01
adacontrast | 64.43 | 63.70 | 63.69 | 37.97 | 64.43 | 64.43 | 48.48 | 64.43
memo | 60.66 | 61.08 | 61.03 | 59.32 | 60.66 | 60.87 | 60.82 | 61.28
lame | 53.01
| rmt-sf | 59.22 | 60.04 | 60.17 | 60.17 | 59.22 | 59.93 | 60.06 | 60.33
ImageNet-C | source | 17.97
tent | 27.98 | 31.08 | 1.52 | 29.79 | 27.97 | 30.29 | 28.83 | 31.37
eata | 28.87 | 14.60 | 1.96 | 23.39 | 19.34 | 32.25 | 18.74 | 35.98
sar | 27.14 | 28.72 | 8.41 | 31.19 | 27.14 | 30.93 | 29.30 | 31.30
adacontrast | 33.29 | 31.81 | 33.29 | 3.90 | 31.79 | 31.81 | 18.94 | 33.29
memo | 24.17 | 23.93 | 22.55 | 24.37 | 23.37 | 23.59 | 24.17 | 25.01
lame | 17.72
| rmt-sf | 28.81 | 27.54 | 23.76 | 23.76 | 28.81 | 30.51 | 29.16 | 31.54
DomainNet-126 | source | 54.71
tent | 49.92 | 52.13 | 8.21 | 52.13 | 49.91 | 52.04 | 50.32 | 52.24
eata | 50.17 | 53.37 | 19.76 | 53.59 | 50.17 | 53.30 | 51.88 | 54.44
sar | 49.73 | 50.91 | 28.48 | 51.24 | 49.82 | 51.45 | 50.55 | 52.75
adacontrast | 56.51 | 55.69 | 46.50 | 19.61 | 56.51 | 54.32 | 51.00 | 56.51
memo | 53.98 | 53.71 | 51.30 | 53.64 | 53.98 | 53.908 | 53.68 | 53.98
lame | 54.34
| rmt-sf | 51.02 | 52.67 | 46.36 | 46.36 | 51.01 | 52.40 | 52.42 | 52.70
ImageNet-R | source | 36.17
tent | 37.43 | 38.92 | 11.01 | 38.30 | 36.51 | 38.40 | 37.55 | 38.92
eata | 37.17 | 43.09 | 2.35 | 42.27 | 37.17 | 42.03 | 39.70 | 43.09
sar | 37.09 | 41.94 | 24.28 | 41.94 | 36.45 | 41.31 | 37.84 | 41.94
adacontrast | 39.23 | 39.53 | 37.94 | 15.00 | 39.23 | 38.86 | 35.80 | 39.53
memo | 40.40 | 40.79 | 39.35 | 39.36 | 40.40 | 40.49 | 40.12 | 40.83
lame | 35.95
| rmt-sf | 36.81 | 40.97 | 40.97 | 40.97 | 36.81 | 40.64 | 38.77 | 40.97
ImageNet-V2 | source | 58.72
tent | 58.69 | 58.73 | 55.82 | 55.99 | 57.41 | 57.69 | 58.70 | 58.89
eata | 58.72 | 58.72 | 21.52 | 56.83 | 58.67 | 58.18 | 58.62 | 58.93
sar | 58.70 | 58.65 | 57.11 | 57.26 | 58.67 | 58.33 | 58.71 | 58.90
adacontrast | 62.28 | 62.07 | 62.28 | 26.68 | 62.28 | 62.17 | 58.05 | 62.28
memo | 62.91 | 63.12 | 62.19 | 62.02 | 62.91 | 62.32 | 62.66 | 63.20
lame | 1.07
| rmt-sf | 58.70 | 58.66 | 58.14 | 42.28 | 58.70 | 58.40 | 48.36 | 58.75

Table 2: Final accuracy of evaluated test-time adaptation methods under
different model selection strategies. Green color marks the best surrogate
strategy (five first columns) for each of the adaptation methods (row-wise),
and the red color marks the worst one. Green bolded text marks unsupervised
results that match oracle selection strategy. Standard deviations in the
appendix.

### Main Results

Categorization of selection strategies As shown in Fig. 1 and Table 2, there
is no single best-unsupervised metric that scores consistently well. An
interesting observation can be made when grouping TTA methods by their
optimization objective. The CON strategy works fairly well for methods that
use entropy-minimization (TENT, SAR, EATA), and the SND strategy is a good
choice for methods that use consistency objectives (RMT, AdaContrast, MEMO).
We call this strategy Category (cat). The opposite strategy of using the
optimization objective to guide parameter selection is called Objective (obj).
Using cat strategy to guide hyperparameter selection works fairly well,
usually a few percents below the oracle results (Fig. 2). On the contrary,
using the obj strategy performs poorly, which seems intuitive as, e.g.,
optimizing for the entropy using pseudo-labels might lead to degenerate
solutions: consistently predicting only one class with high confidence.

Figure 1: Comparison of different model selection strategies. Values denote
the number of experiments (this includes different adaptation methods and
datasets) on which the row model selection method outperforms or matches the
column method. No method does consistently better than any other method across
all setups or match the ORACLE model selection strategy Figure 2:  TTA results
using different hyperparameter selection strategies aggregated over 5
datasets. There is a varying gap between Oracle and unsupervised selection
strategies. Using cat strategy to guide selection works fairly well, whereas
using the obj strategy performs poorly.

TTA methods’ performance highly depends on model selection strategy While EATA
is the 2nd best under the oracle selection strategy, it is outperformed, for
example, by TENT (6th method using oracle selection) when using 4 out of 5
surrogate-based metrics (Fig. 5). Similarly, MEMO, which is the 4th method
using oracle strategy, scores consistently well when using surrogate-based
selection strategies, always being ranked either as the 1st or 2nd best
method. AdaContrast is clearly the best method when using surrogate-based
metrics on all metrics except for consistency loss. Detailed results in the
appendix (Fig. 11).

Figure 3: Average ranking of TTA methods under different model selection
strategies. Under the proposed hyperparameter selection procedure, simple
methods (TENT) or those that are robust to changes in hyperparameters (MEMO)
score more favorably compared to the oracle selection. Methods are ordered by
a decreasing performance of oracle selection.

Figure 4: The varying gap between surrogate-based selection strategies and the
oracle selection. The MEMO method is robust to changes in hyperparameters, and
therefore, it simplifies hyperparameter selection.

Figure 5: Accuracy gap when testing the methods on longer adaptation scenarios
(results with ”-L” suffix). s-acc seems to be the most stable measure metric.
AdaContrast is severely affected on longer scenarios.

Varying gap between surrogate-based and oracle hyperparameter selection The
gap varies between different methods and datasets (Fig. 5). For example, for
the EATA method when using a s-acc strategy the gap varies from 0.21% on
ImageNetV2 to 7.11% on ImageNet-C (which accounts for 24.62% of the relative
gap). One method that allows for robust hyperparameter selection using all
unsupervised selection strategies is MEMO. However, this comes with a high
computational cost (the method is an order of magnitude slower than other
methods, see Table 3 in appendix) as the method performs independent
adaptation on each data point, helping to alleviate the problem of forgetting
(French 1999).

On the contrary, the AdaContrast method using source accuracy for model
selection allowed to almost perfectly match the oracle selection strategy on
all of the datasets.

Limited access to the target labels helps a lot As shown in the results in
Table 2, having access to just one hundred target labels (100-rnd) helps to
consistently improve over the unsupervised model selection strategies. In most
cases, this strategy results in selecting hyperparameters that are within 1%
of accuracy compared to the oracle selection.

### On stability of surrogate-metrics

Adaptation difficulty complicates (unsupervised) model selection Further, we
increase the difficulty of the testing conditions by incorporating less
common, more difficult adaptation scenarios. Fig. 5 shows the performance gap
for adaptation when the adaptation is performed on longer scenarios (by
repeating the testing data 10 times for CIFAR100-C and ImageNetR datasets,
detailed results in the appendix, Table 6). In general, we can see that the
gap between supervised and unsupervised model selection strategies increases.
For example, a c-acc metric, which was previously shown to provide stable
results, now can easily generate degenerated solutions, e.g., when using TENT
(28.68% accuracy vs. 60.36% accuracy of oracle, on CIFAR100-C). Notably, a snd
metric which previously matched oracle results for AdaContrast now performs
very poorly on both benchmarks (e.g., only 7.43% accuracy on ImageNetR). This
is an important result that shows that even if we have obtained optimal
parameters on some datasets, simply increasing the adaptation length might
result in a degenerated model.

Further, we have controlled for the temporal class correlation (Fig. 8 in the
appendix). As presented in the figures, the difference between surrogate-based
model selection and oracle one increases for some of the selection strategies
as we increase task difficulty, which is especially visible for the CON
strategy.

Understanding decision margin First, we plot the correlation between surrogate
metrics and target accuracy (Fig. 6, more plots in the appendix). We have
previously found that incorporating a metric for model selection that was a
part of the adaptation objective leads to degenerate solutions. In the figure,
we can see that optimizing entropy for EATA method initially leads to better
selection results (1st column). Nevertheless, once a certain threshold is
reached, the model’s generated solutions gradually deteriorate in quality.
What’s more, we can see that the transition between optimal and suboptimal
solutions is rather smooth. When using the consistency to guide hyperparameter
selection solutions close to the optimal can be achieved, but we can see that
the margin between viable and suboptimal solutions is rather small. This small
margin explains the varying performance of unsupervised model selection
strategies. s-acc is the only unsupervised metric that clearly isolates viable
solutions from the degenerated ones. However, it usually does not select the
optimal hyperparameters. Similar observations can be made for other methods
(Fig. 12 in the appendix).

Figure 6: Correlation plots for the EATA method between target accuracy
(y-axis) and selected surrogate measures (x-axes). Selecting the model based
on the entropy loss works up to some point, after which minimizing the entropy
leads to degenerate solutions with small accuracy. However, there is a smooth
transition between favorable to unfavorable solutions exists . (Center)
Selecting the model based on the consistency loss leads to solutions close to
the optimal, however, the margin between viable and suboptimal solutions is
rather small. (Right) Selecting the model based on the source accuracy leads
to selecting fairly good (but non-optimal) solutions and it clearly isolates
degenerated models.

Stability in the time-domain In Fig. 7 we plot the accumulated accuracy during
adaptation for all hyperparameter runs and mark the best model, according to
the chosen unsupervised metric, after each batch (SAR method with CON
hyperparameter selection). We can see that it leads to optimal model selection
in the very early stages of the training, however the performance steadily
decreases afterwards. Finally, the metric chooses a different (optimal) model
around the 1100th batch. More plots can be found in the appendix Fig. 13 and
14 (i.e., which shows how using the OBJ selection leads to degenerated
solutions).

Figure 7: Accumulated accuracy on ImageNetR dataset for SAR method with CON-
based hyperparameter selection. Different lines relate to runs with different
hyperparameters. A black dot ($\bullet$) marks the optimal model according to
the selected unsupervised strategy at each timestep.

## Discussion

On the TTA difficulty As it was shown previously, it is hard to design TTA
methods that are robust to changes in hyperparameters (Boudiaf et al. 2022;
Döbler, Marsden, and Yang 2023) or that work well across different sets of
adaptation scenarios (Zhao et al. 2023; Press et al. 2023). We extend those
works by showing that it is not only hard to design a method that works well
across different adaptation scenarios but also because choosing optimal
hyperparameters is a challenging problem. It is not entirely new, given
existing research work on unsupervised model selection for domain adaptation
(You et al. 2019; Saito et al. 2021; Gulrajani and Lopez-Paz 2021). However,
we show here that the time domain, which is inherent to TTA, increases the
problem complexity even further: e.g., as we have shown, parameters that work
reasonably well on a given dataset might not work well when we have increased
adaptation length. Also, we have shown that the margin between viable and
suboptimal solutions is often quite small when using unsupervised
hyperparameter selection.

Recommendations The ultimate goal of TTA is to propose a method that works
well in a vast number of scenarios and allows for robust hyperparameter
selection procedures. Our recommendations in this area are as follows:

  * •

Evaluation i) Report the results of TTA methods by stating the model selection
strategy. This would allow to receive more reliable estimates of performance
gains from TTA and could lead to the development of methods that simplify the
model selection procedure. ii) Include testing across different scenarios
(increased adaptation length or temporal correlation). As shown in our paper,
it can lead to not only adaptation performance degradation (i.e., AdaContrast
results for longer adaptation scenarios) but also more complicated
hyperparameter selection.

  * •

Design choices i) Use information from source-data (e.g., using source
prototypes as in RMT or source-data weights importance as in EATA) combined
with multi-task loss objective to prevent forgetting and model collapse. ii)
Compare against methods with periodical reset (i.e., MEMO) as they alleviate
the problem with forgetting.

  * •

Model selection i) Notably, using source-data supervision also allows for
stable model selection without choosing degenerated solutions (however,
usually, a few % worse than oracle selection). One promising approach found in
this work was matching the AdaConstrast method with the source-data selection
strategy, which in many cases allowed to match the performance of the oracle
selection in base scenarios. ii) Annotating even a small number of data points
from the target domain greatly improves model selection when possible. iii) If
no access to any kind of supervised data is allowed, we recommend using the
cat selection strategy.

## Conclusions

In this work, we evaluated existing test-time adaptation methods on a diverse
array of benchmarks using practical model selection strategies, that is,
without accessing target labels. Throughout the experiments, we found numerous
findings. We showed that the accuracy of model selection strategies strongly
varies across datasets and adaptation methods and have reported a failure mode
for each of the selection strategies. Out of all evaluated approaches, only
the AdaContrast method allows for surrogate-based model selection that matches
oracle selection performance, using source data, however, the method itself
was inferior in adaptation over long scenarios. Finally, we showed the
limitations of the current surrogate-based metrics, which are all inferior to
a strategy of using a tiny set of labeled test samples. The study shows the
importance of the model selection problem in TTA, which we believe did not
receive enough attention from the community.

Limitations For our hyperparameter selection, we have chosen only the most
important parameters (similar to others, i.e., Boudiaf et al. (2022)), and we
may have omitted some other important hyperparameters.

## References

  * Alfarra et al. (2024) Alfarra, M.; Itani, H.; Pardo, A.; shyma yaser alhuwaider; Ramazanova, M.; Perez, J. C.; zhipeng cai; Müller, M.; and Ghanem, B. 2024.  Evaluation of Test-Time Adaptation Under Computational Time Constraints.  In _Forty-first International Conference on Machine Learning_.
  * Boudiaf et al. (2022) Boudiaf, M.; Müller, R.; Ayed, I. B.; and Bertinetto, L. 2022.  Parameter-free Online Test-time Adaptation.  In _CVPR_ , 8334–8343.
  * Chen et al. (2019) Chen, C.; Xie, W.; Huang, W.; Rong, Y.; Ding, X.; Huang, Y.; Xu, T.; and Huang, J. 2019.  Progressive Feature Alignment for Unsupervised Domain Adaptation.  In _CVPR_.
  * Chen et al. (2022) Chen, D.; Wang, D.; Darrell, T.; and Ebrahimi, S. 2022.  Contrastive Test-time Adaptation.  In _CVPR_.
  * Croce et al. (2021) Croce, F.; Andriushchenko, M.; Sehwag, V.; Debenedetti, E.; Flammarion, N.; Chiang, M.; Mittal, P.; and Hein, M. 2021.  RobustBench: a standardized adversarial robustness benchmark.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Döbler, Marsden, and Yang (2023) Döbler, M.; Marsden, R. A.; and Yang, B. 2023.  Robust Mean Teacher for Continual and Gradual Test-Time Adaptation.
  * French (1999) French, R. M. 1999.  Catastrophic forgetting in connectionist networks.  _Trends in Cognitive Sciences_.
  * Ganin and Lempitsky (2015) Ganin, Y.; and Lempitsky, V. S. 2015.  Unsupervised Domain Adaptation by Backpropagation.  In _ICML_.
  * Geirhos et al. (2019) Geirhos, R.; Rubisch, P.; Michaelis, C.; Bethge, M.; Wichmann, F. A.; and Brendel, W. 2019.  ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.  In _ICLR_.
  * Ghunaim et al. (2023) Ghunaim, Y.; Bibi, A.; Alhamoud, K.; Alfarra, M.; Hammoud, H. A. A. K.; Prabhu, A.; Torr, P. H. S.; and Ghanem, B. 2023.  Real-Time Evaluation in Online Continual Learning: A New Hope.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_.
  * Gong, Grauman, and Sha (2013) Gong, B.; Grauman, K.; and Sha, F. 2013.  Connecting the Dots with Landmarks: Discriminatively Learning Domain-Invariant Features for Unsupervised Domain Adaptation.  In _ICML_.
  * Gong et al. (2022) Gong, T.; Jeong, J.; Kim, T.; Kim, Y.; Shin, J.; and Lee, S.-J. 2022.  NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Goswami et al. (2023) Goswami, M.; Challu, C. I.; Callot, L.; Minorics, L.; and Kan, A. 2023.  Unsupervised Model Selection for Time Series Anomaly Detection.  In _ICLR_.
  * Gulrajani and Lopez-Paz (2021) Gulrajani, I.; and Lopez-Paz, D. 2021.  In Search of Lost Domain Generalization.  In _ICLR_.
  * He et al. (2020) He, K.; Fan, H.; Wu, Y.; Xie, S.; and Girshick, R. 2020.  Momentum contrast for unsupervised visual representation learning.  In _CVPR_.
  * Hendrycks et al. (2021) Hendrycks, D.; Basart, S.; Mu, N.; Kadavath, S.; Wang, F.; Dorundo, E.; Desai, R.; Zhu, T.; Parajuli, S.; Guo, M.; Song, D.; Steinhardt, J.; and Gilmer, J. 2021.  The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization.  _ICCV_.
  * Hendrycks and Dietterich (2019a) Hendrycks, D.; and Dietterich, T. 2019a.  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations.  _Proceedings of the International Conference on Learning Representations_.
  * Hendrycks and Dietterich (2019b) Hendrycks, D.; and Dietterich, T. G. 2019b.  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations.  In _ICLR_.
  * Kirkpatrick et al. (2017) Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; et al. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13): 3521–3526.
  * Koh et al. (2021) Koh, P. W.; Sagawa, S.; Marklund, H.; Xie, S. M.; Zhang, M.; Balsubramani, A.; Hu, W.; Yasunaga, M.; Phillips, R. L.; Gao, I.; Lee, T.; David, E.; Stavness, I.; Guo, W.; Earnshaw, B.; Haque, I. S.; Beery, S. M.; Leskovec, J.; Kundaje, A.; Pierson, E.; Levine, S.; Finn, C.; and Liang, P. 2021.  WILDS: A Benchmark of in-the-Wild Distribution Shifts.  In _ICML_.
  * Liang, Hu, and Feng (2020) Liang, J.; Hu, D.; and Feng, J. 2020.  Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation.  In _ICML_.
  * Liu et al. (2024) Liu, Z.; Sun, H.; Peng, Y.; and Zhou, J. 2024.  DART: Dual-Modal Adaptive Online Prompting and Knowledge Retention for Test-Time Adaptation.  In _Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI_ , 14106–14114.
  * Mai et al. (2022) Mai, Z.; Li, R.; Jeong, J.; Quispe, D.; Kim, H.; and Sanner, S. 2022.  Online continual learning in image classification: An empirical survey.  _Neurocomputing_ , 469: 28–51.
  * Miller et al. (2021) Miller, J.; Taori, R.; Raghunathan, A.; Sagawa, S.; Koh, P. W.; Shankar, V.; Liang, P.; Carmon, Y.; and Schmidt, L. 2021.  Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization.
  * Musgrave, Belongie, and Lim (2022) Musgrave, K.; Belongie, S. J.; and Lim, S. 2022.  Three New Validators and a Large-Scale Benchmark Ranking for Unsupervised Domain Adaptation.  arxiv/2208.07360.
  * Nguyen et al. (2023) Nguyen, A. T.; Nguyen-Tang, T.; Lim, S.-N.; and Torr, P. H. 2023.  TIPI: Test Time Adaptation With Transformation Invariance.  In _CVPR_ , 24162–24171.
  * Niu et al. (2022) Niu, S.; Wu, J.; Zhang, Y.; Chen, Y.; Zheng, S.; Zhao, P.; and Tan, M. 2022.  Efficient Test-Time Model Adaptation without Forgetting.  In _ICML_.
  * Niu et al. (2023) Niu, S.; Wu, J.; Zhang, Y.; Wen, Z.; Chen, Y.; Zhao, P.; and Tan, M. 2023.  Towards Stable Test-Time Adaptation in Dynamic Wild World.  In _Internetional Conference on Learning Representations_.
  * Press et al. (2023) Press, O.; Schneider, S.; Kümmerer, M.; and Bethge, M. 2023.  RDumb: A simple approach that questions our progress in continual test-time adaptation.  In _NeurIPS_.
  * Raschka (2018) Raschka, S. 2018.  Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning.  _CoRR_ , abs/1811.12808.
  * Recht et al. (2019) Recht, B.; Roelofs, R.; Schmidt, L.; and Shankar, V. 2019.  Do ImageNet Classifiers Generalize to ImageNet?  In _ICML_.
  * Rusak et al. (2022) Rusak, E.; Schneider, S.; Pachitariu, G.; Eck, L.; Gehler, P. V.; Bringmann, O.; Brendel, W.; and Bethge, M. 2022.  If your data distribution shifts, use self-learning.  _Trans. Mach. Learn. Res._
  * Saito et al. (2019) Saito, K.; Kim, D.; Sclaroff, S.; Darrell, T.; and Saenko, K. 2019.  Semi-supervised Domain Adaptation via Minimax Entropy.  _ICCV_.
  * Saito et al. (2021) Saito, K.; Kim, D.; Teterwak, P.; Sclaroff, S.; Darrell, T.; and Saenko, K. 2021.  Tune it the Right Way: Unsupervised Validation of Domain Adaptation via Soft Neighborhood Density.  _ICCV_.
  * Salvador et al. (2022) Salvador, T.; Fatras, K.; Mitliagkas, I.; and Oberman, A. M. 2022.  A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods.
  * Sohn et al. (2020) Sohn, K.; Berthelot, D.; Carlini, N.; Zhang, Z.; Zhang, H.; Raffel, C.; Cubuk, E. D.; Kurakin, A.; and Li, C. 2020.  FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence.  In _NeurIPS 2020_.
  * Sójka et al. (2023) Sójka, D.; Cygert, S.; Twardowski, B.; and Trzcinski, T. 2023.  AR-TTA: A Simple Method for Real-World Continual Test-Time Adaptation.  In _IEEE/CVF International Conference on Computer Vision, ICCV 2023 Workshops_ , 3483–3487.
  * Su, Xu, and Jia (2024) Su, Y.; Xu, X.; and Jia, K. 2024.  Towards Real-World Test-Time Adaptation: Tri-net Self-Training with Balanced Normalization.  In _Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI_ , 15126–15135.
  * Su et al. (2024) Su, Z.; Guo, J.; Yao, K.; Yang, X.; Wang, Q.; and Huang, K. 2024.  Unraveling Batch Normalization for Realistic Test-Time Adaptation.  In _Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI_ , 15136–15144.
  * Taori et al. (2020) Taori, R.; Dave, A.; Shankar, V.; Carlini, N.; Recht, B.; and Schmidt, L. 2020.  Measuring Robustness to Natural Distribution Shifts in Image Classification.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Teney et al. (2023) Teney, D.; Lin, Y.; Oh, S. J.; and Abbasnejad, E. 2023.  ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Wang et al. (2021) Wang, D.; Shelhamer, E.; Liu, S.; Olshausen, B. A.; and Darrell, T. 2021.  Tent: Fully Test-Time Adaptation by Entropy Minimization.  In _ICLR 2021_.
  * Wang et al. (2022) Wang, Q.; Fink, O.; Gool, L. V.; and Dai, D. 2022.  Continual Test-Time Domain Adaptation.  In _CVPR_ , 7191–7201.
  * Xie et al. (2017) Xie, S.; Girshick, R. B.; Dollár, P.; Tu, Z.; and He, K. 2017.  Aggregated Residual Transformations for Deep Neural Networks.  In _CVPR_.
  * You et al. (2019) You, K.; Wang, X.; Long, M.; and Jordan, M. I. 2019.  Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation.  In _ICML_.
  * Yuan, Xie, and Li (2023) Yuan, L.; Xie, B.; and Li, S. 2023.  Robust test-time adaptation in dynamic scenarios.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 15922–15932.
  * Zhang, Levine, and Finn (2022) Zhang, M.; Levine, S.; and Finn, C. 2022.  MEMO: Test Time Robustness via Adaptation and Augmentation.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Zhao et al. (2023) Zhao, H.; Liu, Y.; Alahi, A.; and Lin, T. 2023.  On Pitfalls of Test-Time Adaptation.  In _ICML_.

Appendix

##  Appendix A Experimental setup

Hyper-parameters We use the following hyper-parameter search:

  * •

Learning rate $\in$ {0.001, 0.00025, 0.0000625, 0.0000156}

  * •

Momentum $\in$ {0.0, 0.9}

  * •

For SAR method:

    * –

The parameter $E_{0}$ $\in$ {0.1, 0.4, 0.8}

    * –

Reset threshold $e_{0}$ $\in$ {0.01, 0.05, 0.2, 0.5}

  * •

For EATA method:

    * –

The parameter $E_{0}$ $\in$ {0.1, 0.4, 0.8}

    * –

$\epsilon$ hyperparameter for similarity thresholding $\in$ {0.05, 0.1, 0.4,
0.8}

    * –

$\beta$ parameter for Fisher regularization $\in$ {1, 1000, 2000, 5000}

  * •

Access to source data:

    * –

EATA - we use 2000 source samples to compute Fisher regularization term.

    * –

RMT - here we use 10% of the original source data to compute source
prototypes.

Data Augmentations Our calculation of model consistency includes the same
augmentations as in CoTTA paper (Wang et al. 2022), which is a combination of
color jittering, gaussian blur, gaussian noise, affine transformations, and
image cropping. The same augmentations are performed by the RMT method during
the adaptation phase.

AdaContrast method use during adaptation augmentations from the MoCo method
(He et al. 2020) used in representation learning. The augmentations include
image resizing, color jittering, grayscale augmentation, gaussian blur, and
horizontal flip. For details about CoTTA and MoCo augmentations please refer
to the repository that we have used (Döbler, Marsden, and Yang 2023).

Long adaptation scenarios Because of the heavy computational requirements of
the memo method (see Table 3) in the adaptation over long scenarios (sec. 5.2)
we copy the Results of the method on the standard length of adaptation. This
is possible because memo performs adaptation on each instance independently.

##  Appendix B Datasets

CIFAR100-C and ImageNet-C datasets are created by applying 15 different
corruption types with 5 severity levels to the images from the test split of
the clean CIFAR100and ImageNetdatasets, creating multiple domains. Following
the state-of-the-art TTA works (Wang et al. 2022; Niu et al. 2022; Döbler,
Marsden, and Yang 2023), for testing TTA methods we utilize the standard
sequence of corruption types with the 5th level of severity sequentially,
without mixing the domains. The source models are trained on the clean
CIFAR100 or ImageNet dataset.

DomainNet-126 is a cleaned from noisy labels subset of the DomainNet dataset.
It consists of images of 126 different objects from 4 different domains: real-
world photos, clipart images, sketches, and paintings. The source model is
trained on actual photos and adapted during test time to the images from the
sequence of the remaining domains.

ImageNet-R is composed of various renditions of 200 objects from the original
ImageNet dataset, spanning various domains such as art, cartoons, graffiti,
embroidery, origami, tattoos, and more. This dataset comprises a total of
30000 images. Source data come from the original ImageNet and TTA is tested on
all of the rendition types at once.

ImageNetV2 contains images collected by closely following the original,
ImageNet labelling protocol. Therefore, it measures the adaptation performance
on natural distribution shift.

##  Appendix C Methods runtime

Efficient TTA method will add a small computational overhead so that the model
can perform adaptation in real-time. A fair comparison between different
methods will take adaptation time into account, e.g., a slower method will
perform adaptation on a smaller number of data points, as was already shown in
online continual learning (Ghunaim et al. 2023) and TTA (Alfarra et al. 2024).
As our main goal is to provide a more realistic hyperparameter selection setup
to compare TTA methods, we simply report runtime results in Table 3. In
general, the fastest methods are those that simply minimize entropy (TENT,
SAR, and EATA). RMT is up to 1.5 times slower, and AdaContrast is 2-3 times
slower than TENT. Finally, the MEMO method is order of magnitude slower than
competitors since it heavily uses data augmentations and performs independent
adaptation to each data sample. In our experiments, we used NVIDIA A100
graphic cards with 40GB of memory.

Method | CIFAR-100-C | ImageNet-C
---|---|---
TENT | 481s | 1256s
EATA | 651s | 1198s
SAR | 686s | 1061s
AdaContrast | 1233s | 3179s
MEMO | 22474s | 47344s
RMT-SF | 1093s | 1615s
Table 3: TTA runtime for evaluated methods.

##  Appendix D Results with varying batch size

Batch Size | Method | s-acc | c-acc | ent | con | snd | 100-rnd | med | oracle
---|---|---|---|---|---|---|---|---|---
- | source | 53.55
- | memo | 60.66$\pm$0.02 | 61.08$\pm$0.14 | 61.03$\pm$0.17 | 59.32$\pm$0.01 | 60.66$\pm$0.02 | 60.87$\pm$0.20 | 60.82$\pm$0.08 | 61.28$\pm$0.02
10 | tent | 59.93$\pm$0.15 | 59.56$\pm$0.2 | 5.32$\pm$0.05 | 59.56$\pm$0.20 | 40.78$\pm$25.04 | 59.17$\pm$0.41 | 54.46$\pm$0.51 | 60.16$\pm$0.14
eata | 61.78$\pm$0.40 | 61.86$\pm$0.47 | 21.01$\pm$20.12 | 58.60$\pm$4.71 | 39.40$\pm$27.15 | 61.95$\pm$0.57 | 60.51$\pm$0.36 | 62.44$\pm$0.35
sar | 59.47$\pm$0.11 | 45.65$\pm$5.83 | 11.68$\pm$5.92 | 48.75$\pm$0.21 | 58.26$\pm$0.13 | 59.26$\pm$0.2 | 57.03$\pm$0.27 | 60.01$\pm$0.12
adacontrast | 64.43$\pm$0.01 | 63.70$\pm$0.11 | 63.69$\pm$0.11 | 37.97$\pm$0.43 | 64.43$\pm$0.01 | 64.43$\pm$0.01 | 48.48$\pm$0.56 | 64.43$\pm$0.01
lame | 53.014$\pm$0.03
rmt-sf | 59.22$\pm$0.13 | 60.04$\pm$0.1 | 60.17$\pm$0.07 | 60.17$\pm$0.07 | 59.22$\pm$0.13 | 59.93$\pm$0.13 | 60.06$\pm$0.12 | 60.33$\pm$0.11
32 | tent | 64.90$\pm$0.22 | 65.22$\pm$0.03 | 25.94$\pm$0.10 |  65.43$\pm$0.04 | 25.94$\pm$0.10 | 64.47$\pm$1.62 | 64.21$\pm$0.02 | 65.62$\pm$0.01
eata | 64.97$\pm$0.14 | 66.02$\pm$0.26 | 42.75$\pm$28.88 | 65.80$\pm$1.27 | 23.08$\pm$28.30 | 65.89$\pm$0.40 | 65.04$\pm$0.54 | 66.84$\pm$0.24
sar | 64.48$\pm$0.03 |  65.19$\pm$0.21 | 37.32$\pm$5.32 |  65.14$\pm$0.17 | 53.07$\pm$13.55 | 65.30$\pm$0.34 | 64.28$\pm$0.01 | 65.82$\pm$0.05
adacontrast | 65.82$\pm$0.07 | 65.82$\pm$0.07 | 61.80$\pm$0.17 |  49.21$\pm$0.24 | 64.70$\pm$0.04 | 62.88$\pm$4.01 | 64.14$\pm$0.06 | 65.91$\pm$0.02
lame | 17.72$\pm$0.02
rmt-sf | 65.06$\pm$0.01 | 65.41$\pm$0.03 | 67.42$\pm$0.19 | 67.42$\pm$0.19 | 63.20$\pm$0.03 | 65.65$\pm$0.97 | 65.67$\pm$0.05 | 67.42$\pm$0.19
64 | tent | 66.04$\pm$0.49 | 65.70$\pm$0.04 | 57.94$\pm$0.30 | 66.54$\pm$0.07 | 63.83$\pm$0.02 | 66.66$\pm$0.18 | 65.39$\pm$0.04 | 66.91$\pm$0.06
eata | 66.79$\pm$0.23 | 66.70$\pm$0.0.20 | 44.58$\pm$28.52 | 67.51$\pm$0.11 | 4.00$\pm$0.18 | 53.30$\pm$0.54 | 65.88$\pm$0.22 | 67.73$\pm$0.18
sar | 66.38$\pm$0.24 | 66.97$\pm$0.34 | 59.38$\pm$0.29 | 64.77$\pm$0.17 | 63.78$\pm$0.01 | 65.79$\pm$0.89 | 65.00$\pm$0.03 | 67.15$\pm$0.11
adacontrast | 66.20$\pm$0.04 | 65.52$\pm$0.02 | 59.29$\pm$0.06 | 59.29$\pm$0.06 | 64.37$\pm$0.04 | 65.73$\pm$0.29 | 65.62$\pm$0.01 | 66.40$\pm$0.02
lame | 54.34$\pm$0.01
rmt-sf | 65.93$\pm$0.02 | 65.93$\pm$0.02 | 68.49$\pm$0.03 | 68.49$\pm$0.03 |  64.02$\pm$0.02 | 67.00$\pm$1.10 | 66.26$\pm$0.02 | 68.49$\pm$0.03
128 | tent | 66.99$\pm$0.27 | 65.52$\pm$0.04 | 66.10$\pm$0.01 | 66.10$\pm$0.01 | 64.40$\pm$0.03 | 67.06$\pm$0.68 | 65.81$\pm$0.02 | 67.59$\pm$0.11
eata | 67.04$\pm$0.50 | 66.48$\pm$0.22 | 44.53$\pm$25.84 |  67.22$\pm$1.17 | 25.36$\pm$23.72 | 67.07$\pm$0.82 | 66.35$\pm$0.31 | 68.22$\pm$0.40
sar | 66.70$\pm$0.15 | 67.19$\pm$0.50 | 66.13$\pm$0.04 | 67.47$\pm$0.04 | 64.38$\pm$0.03 | 67.40$\pm$0.05 | 65.72$\pm$0.05 | 67.62$\pm$0.06
adacontrast | 66.13$\pm$0.01 | 64.80$\pm$0.03 | 63.98$\pm$0.15 | 63.98$\pm$0.15 | 63.87$\pm$0.03 | 65.56$\pm$1.09 | 65.86$\pm$0.01 | 66.66$\pm$0.01
lame | 35.95$\pm$0.01
rmt-sf | 67.02$\pm$0.36 | 65.93$\pm$0.03 | 68.55$\pm$0.07 | 68.55$\pm$0.07 | 64.50$\pm$0.02 | 67.39$\pm$1.72 | 66.26$\pm$0.03 | 68.55$\pm$0.07

Table 4: CIFAR-100-C: final accuracy of evaluated test-time adaptation methods
under different model selection strategies. Green color marks the best
surrogate strategy for each of the methods, and the red color marks the worst
one. Green bolded text marks unsupervised results that match oracle selection
strategy.

Batch Size | Method | s-acc | c-acc | ent | con | snd | 100-rnd | med | oracle
---|---|---|---|---|---|---|---|---|---
- | source | 36.17
- | memo | 40.41$\pm$0.01 | 40.79$\pm$0.04 | 39.35$\pm$1.19 | 39.36$\pm$0.01 | 40.40$\pm$0.01 | 40.49$\pm$0.12 | 40.12$\pm$0.24 | 40.83$\pm$0.02
10 | tent | 37.43$\pm$0.28 | 38.92$\pm$0.07 | 11.01$\pm$0.59 | 38.30$\pm$0.17 | 36.51$\pm$0.18 | 38.40$\pm$0.13 | 37.55$\pm$0.14 | 38.92$\pm$0.07
eata | 37.17$\pm$0.13 | 43.09$\pm$0.75 | 2.35$\pm$1.46 | 42.27$\pm$0.18 | 37.17$\pm$0.13 | 42.03$\pm$0.43 | 39.70$\pm$0.83 | 43.09$\pm$0.76
sar | 37.09$\pm$0.73 | 41.94$\pm$0.24 | 24.28$\pm$0.71 | 41.94$\pm$0.23 | 36.45$\pm$0.20 | 41.31$\pm$1.06 | 37.84$\pm$0.17 | 41.94$\pm$0.24
adacontrast | 39.23$\pm$0.12 | 39.53$\pm$0.14 | 37.94$\pm$0.26 | 15.00$\pm$0.39 | 39.23$\pm$0.12 | 38.86$\pm$0.78 | 35.80$\pm$0.18 | 39.53$\pm$0.14
lame | 35.95$\pm$0.01
rmt-sf | 36.81$\pm$0.16 | 40.97$\pm$0.32 | 40.97$\pm$0.32 | 40.97$\pm$0.32 | 36.81$\pm$0.16 | 40.64$\pm$0.78 | 38.77$\pm$0.10 | 40.97$\pm$0.32
32 | tent | 39.66$\pm$0.43 | 40.15$\pm$0.09 | 37.58$\pm$0.69 | 42.12$\pm$0.39 | 39.08$\pm$0.10 | 41.92$\pm$0.67 | 39.96$\pm$0.10 | 42.34$\pm$0.09
eata | 39.55$\pm$0.47 | 43.70$\pm$0.21 | 4.67$\pm$3.43 | 46.05$\pm$0.59 | 39.29$\pm$0.10 | 45.53$\pm$0.42 | 42.30$\pm$0.46 | 46.58$\pm$0.45
sar | 39.42$\pm$0.46 | 42.44$\pm$1.25 | 42.24$\pm$0.12 | 45.45$\pm$0.01 | 39.06$\pm$0.09 | 44.77$\pm$0.96 | 40.33$\pm$0.09 | 45.45$\pm$0.01
adacontrast | 40.02 $\pm$0.45 | 40.06$\pm$0.11 | 39.90$\pm$0.11 | 30.47$\pm$0.30 | 39.42$\pm$0.10 | 40.20$\pm$0.27 | 39.98$\pm$0.11 | 40.71$\pm$0.17
lame | 53.014$\pm$0.03
rmt-sf | 39.69$\pm$0.39 | 40.53$\pm$0.08 | 42.62$\pm$0.14 | 42.62$\pm$0.14 | 39.20$\pm$0.10 | 42.62$\pm$0.14 | 40.82$\pm$0.05 | 42.62$\pm$0.14
64 | tent | 40.03$\pm$0.28 | 40.18$\pm$0.11 | 42.79$\pm$0.18 | 42.61$\pm$0.32 | 39.58$\pm$0.10 | 42.79$\pm$0.18 | 40.51$\pm$0.09 | 42.79$\pm$0.18
eata | 41.12$\pm$0.34 | 42.78$\pm$0.07 | 19.00$\pm$19.64 | 46.32$\pm$1.46 | 39.69$\pm$0.08 | 46.59$\pm$1.43 | 42.86$\pm$0.49 | 47.01$\pm$0.96
sar | 39.96$\pm$0.12 | 41.89$\pm$1.04 | 43.16$\pm$0.03 |  45.04$\pm$0.06 | 39.56$\pm$0.09 | 45.04$\pm$0.06 | 40.31$\pm$0.08 | 45.04$\pm$0.06
adacontrast | 40.02$\pm$0.34 | 39.78$\pm$0.06 | 40.78$\pm$0.27 | 36.04$\pm$0.35 | 39.32$\pm$0.07 | 41.32$\pm$0.27 | 40.52$\pm$0.07 | 41.56$\pm$0.10
lame | 17.72$\pm$0.02
rmt-sf | 40.19$\pm$0.67 | 40.58$\pm$0.07 | 42.36$\pm$0.02 | 42.36$\pm$0.02 | 39.63$\pm$0.09 | 41.74$\pm$0.49 | 40.85$\pm$0.05 | 42.36$\pm$0.02
128 | tent | 40.60$\pm$0.39 | 40.19$\pm$0.10 | 43.29$\pm$0.04 | 43.29$\pm$0.04 | 39.81$\pm$0.08 | 42.79$\pm$0.73 | 40.39$\pm$0.08 | 43.29$\pm$0.04
eata | 40.31 $\pm$0.35 | 41.93$\pm$0.08 | 18.88$\pm$12.04 | 46.55$\pm$0.83 | 39.88$\pm$0.06 | 47.43$\pm$0.44 | 42.27$\pm$0.14 | 47.43$\pm$0.44
sar | 40.64$\pm$0.20 | 41.30$\pm$0.62 | 43.03$\pm$0.05 | 44.12$\pm$0.03 | 39.80$\pm$0.09 | 43.77$\pm$0.53 | 40.26$\pm$0.06 | 44.12$\pm$0.03
adacontrast | 40.25$\pm$0.16 | 39.55$\pm$0.03 | 39.53$\pm$0.09 | 39.53$\pm$0.09 | 39.29$\pm$0.02 | 40.89$\pm$1.05 | 40.20$\pm$0.04 | 41.80$\pm$0.06
lame | 35.95$\pm$0.01
rmt-sf | 40.42$\pm$0.05 | 40.49$\pm$0.04 | 42.30$\pm$0.06 | 42.30$\pm$0.06 | 39.82$\pm$0.08 | 41.38$\pm$0.59 | 40.73$\pm$0.03 | 42.30$\pm$0.06

Table 5: ImageNet-R: final accuracy of evaluated test-time adaptation methods
under different model selection strategies. Green color marks the best
surrogate strategy for each of the methods, and the red color marks the worst
one. Green bolded text marks unsupervised results that match oracle selection
strategy.

Table 4 and Table 5 shows results on CIFAR-100-C and ImageNet-R datasets when
using larger batch sizes. In general, we can see the increase in performance
of TTA methods and AdaContrast is no longer the top-performing method. Using
s-acc for this method still allows for reliable model selection, however, the
distance to using oracle strategy has increased. In general, we can see that
s-acc and c-acc allow from quite relable model selection, however, they are
quite far from the oracle selection strategy. Once again, the methods ranking
under an unsupervised selection strategy can be different from the oracle
selection one. Under large batch size the rmt-sf seems to consistently work
very well using ent and con selection strategies, however, this method assumes
access to source data (to compute prototypes), which may be limiting in many
scenarios.

##  Appendix E Additional plots

  * •

Fig. 8 shows the effect of increased class correlation on the model
performance for different methods and selection strategies.

  * •

Fig. 9 plots the adaptation accuracy split by selection strategy.
Alternatively, Fig. 10 splits the results by TTA method.

  * •

Fig. 11 extends the Fig. 5 by showing exact numerical results for all
selection strategies.

  * •

Table 6 extends the Fig. 5 by showing exact numerical results.

  * •

Fig. 12 show correlation plots for different adaptation methods, not included
in the main paper (Fig. 6).

  * •

Fig. 13 and Fig. 14 show stability of the unsupervised metrics in the time-
domain not included in the main paper (Fig. 7).

Figure 8: CIFAR100-C results: the effect of Dirichlet concentration parameter
$\delta$ on the model performance for different methods and selection
strategies. Even if the surrogate-based metrics match oracle accuracy in the
base scenario (uniform class distribution), they may fall short in the
non-i.i.d scenario. Figure 9: Target accuracy split by selection strategy,
aggregated for all evaluated TTA methods. The horizontal line separates
selection strategies that use target labels and the vertical one marks source
model accuracy.  Figure 10: Target accuracy per TTA methods for different
datasets, results aggregate all evaluated hyper-parameter configurations. Red
vertical line marks the source model accuracy.  Figure 11: Average accuracy of
evaluated test-time adaptation methods under different model selection
strategies. Each method is ranked (value in parenthesis) according to the
selection strategies (within columns). The color gradient indicates ranking

Dataset | Method | s-acc | c-acc | ent | con | snd | median | oracle
---|---|---|---|---|---|---|---|---
CIFAR100-C long | source | 53.55
tent | 60.36$\pm$0.16 | 28.68$\pm$0.7 | 1.90$\pm$0.21 | 36.09$\pm$23.99 | 1.91$\pm$0.21 | 13.90$\pm$0.41 | 60.36$\pm$0.16
eata | 61.59$\pm$0.91 | 62.17$\pm$0.94 | 3.41$\pm$2.45 | 42.64$\pm$25.36 | 22.61$\pm$26.58 | 52.11$\pm$12.37 | 62.47$\pm$0.41
sar | 54.59$\pm$7.92 | 46.98$\pm$1.13 | 8.17$\pm$2.82 | 42.22$\pm$1.14 | 53.72$\pm$9.14 | 40.85$\pm$3.25 | 54.59$\pm$7.92
adacontrast | 49.45$\pm$9.73 | 49.45$\pm$9.73 | 42.92$\pm$18.98 | 15.63$\pm$0.78 | 35.26$\pm$0.75 | 19.88$\pm$1.36 | 49.45$\pm$9.73
memo* | 60.66$\pm$0.02 | 61.08$\pm$0.14 | 61.03$\pm$0.17 | 59.32$\pm$0.01 | 60.66$\pm$0.02 | 60.82$\pm$0.08 | 61.28$\pm$0.02
rmt-sf | 58.02$\pm$3.84 | 31.53$\pm$0.15 | 60.69$\pm$0.12 | 60.69$\pm$0.12 | 60.69$\pm$0.12 | 56.02$\pm$0.92 | 60.81$\pm$0.06
ImageNet-R Long | source | 36.17
tent | 37.92$\pm$0.15 | 20.20$\pm$0.34 | 1.84$\pm$0.22 | 37.78$\pm$0.85 | 37.92$\pm$0.15 | 27.12$\pm$0.25 | 38.94$\pm$0.11
eata | 40.03$\pm$0.12 | 41.94$\pm$2.80 | 3.41$\pm$2.65 | 43.40$\pm$1.95 | 26.98$\pm$18.48 | 34.17$\pm$8.60 | 44.49$\pm$1.28
sar | 37.78$\pm$0.18 | 41.37$\pm$1.60 | 7.31$\pm$4.21 | 43.62$\pm$0.11 | 37.7$\pm$0.18 | 38.58$\pm$0.13 | 43.62$\pm$0.11
adacontrast | 36.81$\pm$0.16 | 26.97$\pm$0.19 | 36.81$\pm$0.16 | 36.81$\pm$0.16 | 7.43$\pm$0.49 | 11.82$\pm$0.05 | 36.81$\pm$0.16
memo | 40.41$\pm$0.01 | 40.79$\pm$0.04 | 39.35$\pm$1.19 | 39.36$\pm$0.01 | 40.40$\pm$0.01 | 40.12$\pm$0.24 | 40.83$\pm$0.02
rmt-sf | 38.57$\pm$0.05 | 43.87$\pm$0.19 | 43.87$\pm$0.19 | 43.87$\pm$0.19 | 38.57$\pm$0.05 | 38.30$\pm$0.16 | 43.87$\pm$0.19

Table 6: Target accuracy for different hyper-parameter selection strategies
when adaptation length is multiplied 10 times. Using oracle-chosen parameters
from standard ImageNet-C benchmark (c-acc strategy) performs poorly for SAR
and EATA. Also, greater variance in results can be observed.

Figure 12: Correlation plots for the different methods between target accuracy
(y-axis) and selected surrogate measure (x-axis).

Figure 13: Accumulated accuracy on ImageNetR dataset, different lines relate
to runs with different hyper-parameters. A black dot ($\bullet$) marks the
optimal model according to the selected unsupervised strategy at each
timestep.

Figure 14: Accumulated accuracy on CIFAR-100-C dataset, different lines relate
to runs with different hyper-parameters. A black dot ($\bullet$) marks the
optimal model according to the selected unsupervised strategy at each
timestep. Significant changes in accuracy at different time-steps mark the
change in distortion type.

Dataset | Method | s-acc | c-acc | ent | con | snd | 100-rnd | median | oracle
---|---|---|---|---|---|---|---|---|---
CIFAR100-C | source | 53.55
tent | 59.93$\pm$0.15 | 59.56$\pm$0.2 | 5.32$\pm$0.05 | 59.56$\pm$0.20 | 40.78$\pm$25.04 | 59.17$\pm$0.41 | 54.46$\pm$0.51 | 60.16$\pm$0.14
eata | 61.78$\pm$0.40 | 61.86$\pm$0.47 | 21.01$\pm$20.12 | 58.60$\pm$4.71 | 39.40$\pm$27.15 | 61.95$\pm$0.57 | 60.51$\pm$0.36 | 62.44$\pm$0.35
sar | 59.47$\pm$0.11 | 45.65$\pm$5.83 | 11.68$\pm$5.92 | 48.75$\pm$0.21 | 58.26$\pm$0.13 | 59.26$\pm$0.2 | 57.03$\pm$0.27 | 60.01$\pm$0.12
adacontrast | 64.43$\pm$0.01 | 63.70$\pm$0.11 | 63.69$\pm$0.11 | 37.97$\pm$0.43 | 64.43$\pm$0.01 | 64.43$\pm$0.01 | 48.48$\pm$0.56 | 64.43$\pm$0.01
memo | 60.66$\pm$0.02 | 61.08$\pm$0.14 | 61.03$\pm$0.17 | 59.32$\pm$0.01 | 60.66$\pm$0.02 | 60.87$\pm$0.20 | 60.82$\pm$0.08 | 61.28$\pm$0.02
lame | 53.01$\pm$0.03
| rmt-sf | 59.22$\pm$0.13 | 60.04$\pm$0.1 | 60.17$\pm$0.07 | 60.17$\pm$0.07 |  59.22$\pm$0.13 | 59.93$\pm$0.13 | 60.06$\pm$0.12 | 60.33$\pm$0.11
ImageNet-C | source | 17.97 |
tent | 27.98$\pm$0.07 | 31.08$\pm$0.06 | 1.52$\pm$0.1 | 29.79$\pm$0.16 | 27.97$\pm$0.07 | 30.29$\pm$0.76 | 28.83$\pm$0.03 | 31.37$\pm$0.03
eata | 28.87$\pm$0.11 | 14.60$\pm$12.38 | 1.96$\pm$2.28 | 23.39$\pm$16.54 | 19.34$\pm$13.57 | 32.25$\pm$0.99 | 18.74$\pm$8.45 | 35.98$\pm$1.02
sar | 27.14$\pm$0.09 | 28.72$\pm$0.11 | 8.41$\pm$4.6 | 31.19$\pm$0.22 | 27.14$\pm$0.09 | 30.93$\pm$0.35 | 29.30$\pm$0.45 | 31.30$\pm$0.08
adacontrast | 33.29$\pm$0.09 | 31.81$\pm$0.14 | 33.29$\pm$0.09 |  3.90$\pm$0.26 | 31.79$\pm$0.14 | 31.81$\pm$2.01 | 18.94$\pm$0.13 | 33.29$\pm$0.09
memo | 24.17$\pm$0.99 | 23.93$\pm$0.06 | 22.55$\pm$1.20 | 24.37$\pm$0.17 | 23.37$\pm$2.13 | 23.59$\pm$1.82 | 24.17$\pm$0.56 | 25.01$\pm$0.01
lame | 17.72$\pm$0.02
rmt-sf | 28.81$\pm$0.05 | 27.54$\pm$2.6 | 23.76$\pm$0.1 | 23.76$\pm$0.1 | 28.81$\pm$0.05 | 30.51$\pm$0.1 | 29.16$\pm$0.13 | 31.54$\pm$0.15
DomainNet-126 | source | 54.71
tent | 49.92$\pm$0.08 | 52.13$\pm$0.06 | 8.21$\pm$0.11 | 52.13$\pm$0.06 | 49.91$\pm$0.085 | 52.04$\pm$0.19 | 50.32$\pm$0.08 | 52.24$\pm$0.01
eata | 50.17$\pm$0.06 | 53.37$\pm$0.92 | 19.76$\pm$23.79 | 53.59$\pm$0.95 | 50.17$\pm$0.07 | 53.30$\pm$0.54 | 51.88$\pm$0.19 | 54.44$\pm$0.27
sar | 49.73$\pm$0.09 | 50.91$\pm$1.15 | 28.48$\pm$5.55 | 51.24$\pm$0.19 | 49.82$\pm$0.15 | 51.45$\pm$0.44 | 50.55$\pm$0.31 | 52.75$\pm$0.07
adacontrast | 56.51$\pm$0.04 | 55.69$\pm$0.15 | 46.50$\pm$0.42 | 19.61$\pm$0.79 | 56.51$\pm$0.06 | 54.32$\pm$3.11 | 51.00$\pm$0.79 | 56.51$\pm$0.04
memo | 53.98$\pm$0.04 | 53.71$\pm$0.03 | 51.30$\pm$0.07 | 53.64$\pm$0.01 | 53.98$\pm$0.04 | 53.90$\pm$0.08 | 53.68$\pm$0.02 | 53.98$\pm$0.04
lame | 54.34$\pm$0.01
| rmt-sf | 51.02$\pm$0.05 | 52.67$\pm$0.02 | 46.36$\pm$0.64 | 46.36$\pm$0.64 | 51.01$\pm$0.05 | 52.40$\pm$0.20 | 52.42$\pm$0.11 | 52.70$\pm$0.01
ImageNet-R | source | 36.17 |
tent | 37.43$\pm$0.28 | 38.92$\pm$0.07 | 11.01$\pm$0.59 | 38.30$\pm$0.17 | 36.51$\pm$0.18 | 38.40$\pm$0.13 | 37.55$\pm$0.14 | 38.92$\pm$0.07
eata | 37.17$\pm$0.13 | 43.09$\pm$0.75 | 2.35$\pm$1.46 | 42.27$\pm$0.18 | 37.17$\pm$0.13 | 42.03$\pm$0.43 | 39.70$\pm$0.83 | 43.09$\pm$0.75
sar | 37.09$\pm$0.73 | 41.94$\pm$0.24 | 24.28$\pm$0.71 | 41.94$\pm$0.24 | 36.45$\pm$0.20 | 41.31$\pm$1.06 | 37.84$\pm$0.17 | 41.94$\pm$0.24
adacontrast | 39.23$\pm$0.12 | 39.53$\pm$0.14 | 37.94$\pm$0.26 | 15.00$\pm$0.39 | 39.23$\pm$0.12 | 38.86$\pm$0.78 | 35.80$\pm$0.18 | 39.53$\pm$0.14
memo | 40.40$\pm$0.01 | 40.79$\pm$0.04 | 39.35$\pm$1.19 | 39.36$\pm$0.01 | 40.40$\pm$0.01 | 40.49$\pm$0.12 | 40.12$\pm$0.24 | 40.83$\pm$0.02
lame | 35.95$\pm$0.01
rmt-sf | 36.81$\pm$0.16 | 40.97$\pm$0.32 | 40.97$\pm$0.32 | 40.97$\pm$0.32 | 36.81$\pm$0.16 | 40.64$\pm$0.78 | 38.77$\pm$0.10 | 40.97$\pm$0.32
ImageNet-V2 | source | 58.72
tent | 58.69$\pm$0.10 | 58.73$\pm$0.07 | 55.82 $\pm$0.24 | 55.99$\pm$0.06 | 57.41$\pm$0.11 | 57.69$\pm$1.56 | 58.70$\pm$0.11 | 58.89$\pm$0.13
eata | 58.72$\pm$0.06 | 58.72$\pm$0.06 | 21.52$\pm$25.56 | 56.83$\pm$0.27 | 58.67$\pm$0.12 | 58.18$\pm$0.69 | 58.62$\pm$0.12 | 58.93$\pm$0.09
sar | 58.70$\pm$0.10 | 58.65$\pm$0.09 | 57.11$\pm$0.04 | 57.26$\pm$0.09 | 58.67$\pm$0.11 | 58.33$\pm$0.68 | 58.71$\pm$0.09 | 58.90$\pm$0.07
adacontrast | 62.28$\pm$0.19 | 62.07$\pm$0.11 | 62.28$\pm$0.19 | 26.68$\pm$1.68 | 62.28$\pm$0.19 | 62.17$\pm$0.26 | 58.05$\pm$0.33 | 62.28$\pm$0.19
memo | 62.91$\pm$0.16 | 63.12$\pm$0.13 | 62.19$\pm$0.09 | 62.02$\pm$0.02 | 62.91$\pm$0.16 | 62.32$\pm$0.22 | 62.66$\pm$0.03 | 63.20$\pm$0.06
lame | 1.07$\pm$0.01
| rmt-sf | 58.70$\pm$0.15 | 58.66$\pm$0.02 | 58.14$\pm$0.05 | 42.28$\pm$22.41 | 58.70$\pm$0.15 | 58.4$\pm$0.26 | 48.36$\pm$14.48 | 58.75$\pm$0.15

Table 7: Final accuracy of evaluated test-time adaptation methods under
different model selection strategies. Green color marks the best surrogate
strategy (five first columns) for each of the adaptation methods (row-wise),
and the red color marks the worst one. Green bolded text marks unsupervised
results that match oracle selection strategy.


#  Contractibility of the Geometric Stability Manifold of a Surface

Nick Rekuski  Department of Mathematics, Wayne State University, Detroit, MI
48202, USA<EMAIL_ADDRESS>
(Date: October 16, 2023 )

###### Abstract.

Using a recent description of the geometric stability manifold, we give a
short proof showing the geometric stability manifold associated to any smooth
projective complex surface is contractible.

###### Key words and phrases:

Bridgeland stability, geometric stability conditions, contractibility,
homotopy type

###### 2020 Mathematics Subject Classification:

Primary 14F08, 14J60, 18G80

The author was supported by the U.S. Department of Energy, Office of Science,
Basic Energy Sciences, under Award Number DE-SC-SC0022134 and an OVPR
Postdoctoral Award at Wayne State University.

##  1\. Introduction

On a triangulated category, $\mathcal{T}$, Bridgeland defined the notion of a
stability condition which establishes a framework to study moduli of complexes
of sheaves [bridgeland2007]. The collection of all stability conditions forms
a complex manifold (with possibly infinitely many connected components),
$\operatorname{Stab}(\mathcal{T})$. Stability conditions and the stability
manifold have been used to obtain new results in homological mirror symmetry,
representation theory, symplectic geometry, and moduli of stable sheaves.
Furthermore, $\operatorname{Stab}(\mathcal{T})$ has an inherent richness that
is worthy of study in its own right.

Recent work has focused on understanding the homotopy type of
$\operatorname{Stab}(\mathcal{T})$. This focus is partially inspired by a
conjecture of Bridgeland that states if $X$ is a $K3$ surface then a
distinguished component of $\operatorname{Stab}(D^{b}(X))$ is simply-connected
[bridgeland2008]*Conjecture 1.2. A positive answer to this conjecture would
shed light on the group of exact autoequivalences of the bounded derived
category of a $K3$ surface. More generally, there is a folklore conjecture
that if $\operatorname{Stab}(\mathcal{T})$ is nonempty then it is homotopy
discrete.

With current techniques, it seems intractable to show
$\operatorname{Stab}(\mathcal{T})$ is homotopy discrete in general. For this
reason, recent research has focused on two open submanifolds of
$\operatorname{Stab}(\mathcal{T})$: the algebraic stability manifold,
$\operatorname{Stab}^{\mathrm{alg}}(\mathcal{T})$, and the geometric stability
manifold, $\operatorname{Stab}^{\mathrm{geo}}(\mathcal{T})$. The algebraic
stability manifold consists of stability conditions arising from full
exceptional collections of $\mathcal{T}$, and the geometric stability manifold
consists of stability conditions where skyscraper sheaves are stable.

In many cases
$\operatorname{Stab}^{\mathrm{geo}}(\mathcal{T})\cup\operatorname{Stab}^{\mathrm{alg}}(\mathcal{T})$
is a connected component of $\operatorname{Stab}(\mathcal{T})$. Therefore, the
homotopy types of $\operatorname{Stab}^{\mathrm{alg}}(\mathcal{T})$ and
$\operatorname{Stab}^{\mathrm{geo}}(\mathcal{T})$ provide important
topological information about $\operatorname{Stab}(\mathcal{T})$. For example,
if $\mathcal{T}=D^{b}(Q)$—the bounded derived category of representations of a
quiver $Q$–and the underlying graph of $Q$ is simple and acyclic then
$\operatorname{Stab}^{\mathrm{alg}}(Q)=\operatorname{Stab}(Q)$
[otani2022]*Theorem 1.1. Similarly, if $\mathcal{T}=D^{b}(X)$—the bounded
derived category of coherent sheaves on a variety $X$—and $X$ has finite
Albanese then $\operatorname{Stab}^{\mathrm{geo}}(X)=\operatorname{Stab}(X)$
[fu2022]*Theorem 1.1.

We give a brief overview of topological results involving
$\operatorname{Stab}^{\mathrm{alg}}(\mathcal{T})$ and
$\operatorname{Stab}^{\mathrm{geo}}(\mathcal{T})$. We start with results for
$\operatorname{Stab}^{\mathrm{alg}}(\mathcal{T})$ (see Figure 1 for
illustrations of the relevant quivers).

  * •

If $\mathcal{T}=D^{b}(Q)$ where $Q$ is a quiver of type $A_{n}$, $D_{n}$, or
$E_{n}$ then
$\operatorname{Stab}^{\mathrm{alg}}(\mathcal{T})=\operatorname{Stab}(\mathcal{T})$
is contractible [haiden2014, qiu2015, qiu2018, bridgeland2020].

  * •

If $\mathcal{T}=D^{b}(Q)$ where $Q$ is of type $\widetilde{A}_{2}$ then
$\operatorname{Stab}^{\mathrm{alg}}(\mathcal{T})=\operatorname{Stab}(\mathcal{T})$
is contractible [dimitrov2016, dimitrov2018].

  * •

If $\mathcal{T}=D^{b}(K_{n})$ where $K_{n}$ is the Konecker quiver (including
the case $\mathcal{T}=D^{b}(\mathbb{P}^{1})$) then
$\operatorname{Stab}^{\mathrm{alg}}(\mathcal{T})=\operatorname{Stab}(\mathcal{T})$
is contractible [macri2007, dimitrov2019].

  * •

If $\mathcal{T}=D^{b}(P_{2})$ (equivalently
$\mathcal{T}=D^{b}(\mathbb{P}^{2})$) then
$\operatorname{Stab}^{\mathrm{alg}}(\mathcal{T})$ is contractible [bayer2011].

  * •

If $\mathcal{T}=D^{b}(\Lambda(r,n,m))$ where $\Lambda(r,n,m)$ is the path
algebra of $Q_{n,m}$ subject to the relations

$\gamma_{n-r+1}\gamma_{n-r+2},\gamma_{n-(r-1)+1}\gamma_{n-(r-1)+2},\ldots,\gamma_{n}\gamma_{1}=0$

then
$\operatorname{Stab}^{\mathrm{alg}}(\mathcal{T})=\operatorname{Stab}(\mathcal{T})$
is contractible [broomhead2016].

We now review results for $\operatorname{Stab}^{\mathrm{geo}}(\mathcal{T})$.

  * •

If $\mathcal{T}=D^{b}(X)$ where $X$ is a smooth positive genus curve then
$\operatorname{Stab}^{\mathrm{geo}}(\mathcal{T})=\operatorname{Stab}(\mathcal{T})$
is contractible [macri2007],

  * •

If $\mathcal{T}=D^{b}(X)$ for an abelian surface $X$ then
$\operatorname{Stab}^{\mathrm{geo}}(\mathcal{T})=\operatorname{Stab}(\mathcal{T})$
is simply connected and if the Néron-Severi rank of $X$ is $1$ then
$\operatorname{Stab}(\mathcal{T})$ is contractible [huybrechts2008, fu2022].

  * •

If $\mathcal{T}=D^{b}(X)$ where $X$ is an irregular surface of Néron-Severi
rank one then
$\operatorname{Stab}^{\mathrm{geo}}(\mathcal{T})=\operatorname{Stab}(\mathcal{T})$
is contractible [fu2022].

  * •

If $\mathcal{T}=D^{b}(X)$ where $X$ is an abelian threefold of Néron-Severi
rank one then
$\operatorname{Stab}^{\mathrm{geo}}(\mathcal{T})=\operatorname{Stab}(\mathcal{T})$
is contractible [fu2022].

There is an intermediate result on $\mathbb{P}^{2}$ that is neither strictly
geometric nor algebraic: if $\mathcal{T}=D^{b}(\mathbb{P}^{2})$ then
$\operatorname{Stab}^{\mathrm{alg}}(\mathcal{T})\cup\operatorname{Stab}^{\mathrm{geo}}(\mathcal{T})$
is a contractible connected component of $\operatorname{Stab}(\mathcal{T})$
[li2017].

There are cases when
$\operatorname{Stab}^{\mathrm{alg}}(\mathcal{T})\cup\operatorname{Stab}^{\mathrm{geo}}(\mathcal{T})$
is not a connected component of
$\operatorname{Stab}^{\mathrm{geo}}(\mathcal{T})$. For example, if
$\mathcal{T}=D^{b}(X)$ where $X$ is a $K3$ surface [bridgeland2008], a general
Enriques surface [macri2009], a local projective plane [bayer2011], or a
surface with no exceptional collections admitting a rational curve of negative
self-intersection [tramel2022] then
$\operatorname{Stab}^{\mathrm{alg}}(\mathcal{T})\cup\operatorname{Stab}^{\mathrm{geo}}(\mathcal{T})$
is not a connected component of $\operatorname{Stab}(\mathcal{T})$. As far as
the author is aware, the only topological results in this case with $X$ a
projective variety are for $K3$ surfaces of Picard rank one [bayer2017]. There
are more topological results in this case if $X$ is not compact
[huybrechts2008, bayer2011].

$A_{n}$:$1$$2$$\cdots$$n$$\widetilde{A}_{2}$:$D_{n}$:$1$$2$$\cdots$$n-2$$n$$n-1$$K_{n}$:$1$$2$$\vdots$$n$$E_{n}$:$6\leq
n\leq
8$$1$$2$$3$$\cdots$$n$$n-1$$Q_{n,m}$:$\cdots$$\vdots$$1$$2$$m$$\gamma_{1}$$\gamma_{2}$$\gamma_{n}$$\gamma_{n-1}$$P_{2}$:
Figure 1.  Illustration of the underling graphs of quivers of type $ADE$ and
$\widetilde{A}_{2}$, and the quivers $K_{n}$, $Q_{n,m}$, and $P_{2}$.

In the case of a smooth complex surface, Dell gives a description of
$\operatorname{Stab}^{\mathrm{geo}}(X)$ and used this description to show
$\operatorname{Stab}^{\mathrm{geo}}(X)$ is connected [dell2023]*Corollary
5.39. We use this same description to show
$\operatorname{Stab}^{\mathrm{geo}}(X)$ is contractible.

######  Theorem A (3.2).

If $X$ is a smooth projective complex surface then
$\operatorname{Stab}^{\mathrm{geo}}(X)$ is contractible.

In Example 3.3 we use Theorem A and a recent result of of Fu, Li, and Zhao, to
show if $X$ has finite Albanese then $\operatorname{Stab}(X)$ is contractible.
This generalizes [fu2022]*Theorem 1.2 to arbitrary Néron-Severi rank. This
generalization gives infinitely many new families of surfaces with
contractible stability manifold.

In Example 3.4 we use Theorem A and a recent result of Dell to show if $X$ has
finite Albanese and $G$ acts freely on $X$ then $\operatorname{Stab}(X/G)$
contains a contractible connected component. In particular, the stability
manifold of a Beauville-type or bielliptic surface contains a contractible
distinguished component.

### Notation and Assumptions

Suppose $X$ is a smooth projective complex surface. We write
$\operatorname{NS}(X)$ for the Néron-Severi group of $X$ and
$\operatorname{Amp}(X)$ for the ample cone. We also write
$\operatorname{NS}_{\mathbb{R}}(X)=\operatorname{NS}(X)\otimes\mathbb{R}$ and
$\operatorname{Amp}_{\mathbb{R}}(X)=\operatorname{Amp}(X)\otimes\mathbb{R}$.
We use greek letters (e.g. $\alpha,\beta,\phi$) for real numbers.

The bounded derived category of $\operatorname{Coh}(X)$ will be written
$D^{b}(X)$ and the numerical Grothendieck group of $X$ will be written
$K_{\operatorname{num}}(X)$. We use capital script letters (e.g.
$\mathscr{E}$) for coherent sheaves on $X$. We use capital letters (e.g. $E$,
$F$) for chain complexes in $D^{b}(X)$. We use
$\operatorname{ch}_{i}(\mathscr{E})$ for the $i$th Chern character of
$\mathscr{E}$ viewed as an element of $H^{2i}(X,\mathbb{Q})$.

### Acknowledgments

The author is thankful to Hannah Dell, Rajesh Kulkarni, and Andrew Salch for
discussions related to this work.

##  2\. Preliminaries

We recall some background on stability conditions.

######  Definition 2.1.

A stability condition on $D^{b}(X)$ (also called a Bridgeland stability
condition) is a pair $\sigma=(Z,\mathcal{P})$ where
$Z:K_{\operatorname{num}}(X)\to\mathbb{C}$ is a group homomorphism, called a
central charge, and $\\{\mathcal{P}(\phi)\\}_{\phi\in\mathbb{R}}$ is a
collection of full subcategories of $D^{b}(X)$, called a slicing, satisfying
the following conditions

  * •

If $E\in\mathcal{P}(\phi)\setminus\\{0\\}$ then
$Z([E])\in\mathbb{R}_{>0}\exp(i\pi\phi)$—the positive real ray spanned by
$\exp(i\pi\phi)$,

  * •

If $\phi\in\mathbb{R}$ then $\mathcal{P}(\phi+1)=\mathcal{P}(\phi)[1]$,

  * •

If $\phi_{1}>\phi_{2}$ then
$\operatorname{Hom}_{D^{b}(X)}(\mathcal{P}(\phi_{1}),\mathcal{P}(\phi_{2}))=0$,

  * •

If $E\in D^{b}(X)\setminus\\{0\\}$ then there exists real numbers
$\phi_{1}>\phi_{2}>\cdots>\phi_{n}$ and a sequence of distinguished triangles

${0}$${E_{1}}$${E_{2}}$${\cdots}$${E_{n-1}}$${E}$${F_{1}}$${F_{2}}$${\cdots}$${F_{n-1}}$${F_{n}}$

where $F_{i}\in\mathcal{P}(\phi_{i})$, and

  * •

$\displaystyle\inf_{\phi\in\mathbb{R}}\left\\{\frac{|Z([E])|}{||[E]||}:E\in\mathcal{P}(\phi)\setminus\\{0\\}\right\\}>0$
for some norm $||\cdot||$ on $K_{\mathrm{num}}(X)$.

If $\sigma$ is a stability condition and
$E\in\mathcal{P}(\phi)\setminus\\{0\\}$ then we say $E$ is $\sigma$-semistable
of phase $\phi$. If $E\in\mathcal{P}(\phi)\setminus\\{0\\}$ is simple in
$\mathcal{P}(\phi)$ then we say $E$ is $\sigma$-stable of phase $\phi$.

We write the collection of all stability conditions on $X$ as
$\operatorname{Stab}(X)$. Bridgeland described a natural metric on
$\operatorname{Stab}(X)$ so that $\operatorname{Stab}(X)$ is a complex
manifold (with possibly infinitely many connected components)
[bridgeland2007]*Theorem 1.2. For our purposes, we restrict our attention to
an open submanifold of $\operatorname{Stab}(X)$ where all skyscraper sheaves
are stable.

######  Definition 2.2.

A stability condition on $\sigma\in\operatorname{Stab}(X)$ is said to be
geometric if the skyscraper sheaf $\mathscr{O}_{x}$ is $\sigma$-stable for all
$x\in X$. We write the collection of all geometric stability conditions on
$D^{b}(X)$ as $\operatorname{Stab}^{\mathrm{geo}}(X)$.

Commonly, geometric stability conditions are defined to be stability
conditions where skyscraper sheaves are all stable of the same phase. We have
dropped this additional assumption from Definition 2.2 because
[fu2022]*Proposition 2.9 has shown it is already implied by the stability
assumption.

The subspace $\operatorname{Stab}^{\mathrm{geo}}(X)$ is open in
$\operatorname{Stab}(X)$, so each connected component of
$\operatorname{Stab}^{\mathrm{geo}}(X)$ is a complex submanifold of some
connected component of $\operatorname{Stab}(X)$ [bridgeland2008]*Proposition
9.4.

Dell recently gave an explicit description of
$\operatorname{Stab}^{\mathrm{geo}}(X)$ in terms of a generalization of the Le
Potier function. We first recall this function then recall the description of
$\operatorname{Stab}^{\mathrm{geo}}(X)$.

######  Definition 2.3 ([fu2022]*Definition 3.1,[dell2023]*Definition 5.8).

Suppose $X$ is a smooth projective complex surface. We define the Le Potier
function
$\Phi:\operatorname{Amp}_{\mathbb{R}}(X)\times\operatorname{NS}_{\mathbb{R}}(X)\times\mathbb{R}\to\mathbb{R}$
as

$\Phi(H,D,\beta)=\limsup_{\mu\to\beta}\left\\{\frac{\operatorname{ch}_{2}(\mathscr{E})-D\cdot\operatorname{ch}_{1}(\mathscr{E})}{H^{2}\operatorname{rank}(\mathscr{E})}:\begin{array}[]{l}\mathscr{E}\in\operatorname{Coh}(X)\
\text{is slope stable (with respect}\\\ \text{to H) and}\
(H\cdot\operatorname{ch}_{1}(\mathscr{E}))/\operatorname{rank}(\mathscr{E})=\mu\end{array}{}\right\\}.$

We recall Dells’s description of $\operatorname{Stab}^{\mathrm{geo}}(X)$ in
terms of the Le Potier function. A similar result was shown in
[fu2022]*Proposition 3.6.

######  Theorem 2.4 ([dell2023]*Theorem 5.10).

Suppose $X$ is a smooth projective complex surface. There is a homeomorphism

$\operatorname{Stab}^{\mathrm{geo}}(X)\cong\mathbb{C}\times\left\\{(H,D,\beta,\alpha)\in\operatorname{Amp}_{\mathbb{R}}(X)\times\operatorname{NS}_{\mathbb{R}}(X)\times\mathbb{R}\times\mathbb{R}:\Phi(H,D,\beta)<\alpha\right\\}.$

We broadly sketch the proof of this theorem. In the case of surfaces,
geometric stability conditions are determined by their central charge
[bridgeland2008]*Proposition 10.3,[dell2023]*Theorem 5.5. By considering the
central charge of a skyscraper sheaf and ideal sheaves of curves there is a
continuous injection
$i:\operatorname{Stab}^{\mathrm{geo}}(X)\to\mathbb{C}\times\operatorname{Amp}_{\mathbb{R}}(X)\times\operatorname{NS}_{\mathbb{R}}(X)\times\mathbb{R}\times\mathbb{R}$
[bridgeland2008]*Section 10, [fu2022]*Proposition 3.6,[dell2023]*Theorem 5.5.
By directly constructing a quadratic form, Dell shows the support property is
satisfied by exactly $(\lambda,H,D,\beta,\alpha)\in
i(\operatorname{Stab}^{\mathrm{geo}}(X))$ satisfying $\Phi(H,D,\beta)<\alpha$
[dell2023]*Lemma 5.32.

##  3\. Contractibility of the Geometric Stability Manifold

Contractibility of $\operatorname{Stab}^{\mathrm{geo}}(X)$ will follow from
Theorem 2.4 and the following topological result. This lemma shows the region
above the graph of a “sufficiently nice” function is contractible. Note we
cannot assume the function is continuous because the Le Potier function is not
continuous in general.

######  Lemma 3.1.

Suppose $Z$ is a topological space and $f:Z\to\mathbb{R}$ is a function (not
necessarily continuous). Define

$\Gamma_{f}^{<}=\\{(z,\alpha)\in Z\times\mathbb{R}\mid f(z)<\alpha\\}$

which we give the subspace topology. If there exists a continuous function
$g:Z\to\mathbb{R}$ satisfying $f(z)\leq g(z)$ for all $z\in Z$ then
$\Gamma_{f}^{<}$ and $Z$ are homotopy equivalent.

###### Proof.

By replacing $g(z)$ with $g(z)+1$ we may assume $f(z)<g(z)$ (this assumption
makes the notation somewhat simpler). Since $f(z)<g(z)$ for all $z\in Z$, the
function $F:\Gamma_{f}^{<}\times[0,1]\to\Gamma_{f}^{<}$ defined by

$F((z,\alpha),t)=(z,\max\\{\alpha,(g(z)-\alpha)t+\alpha\\})$

is well-defined and continuous. Moreover, since $\max\\{\alpha,g(z)\\}\geq
g(z)$, $F$ is a strong deformation retract of $\Gamma_{f}^{<}$ onto

$\Gamma_{g}^{\leq}=\\{(z,\beta)\in Z\times\mathbb{R}\mid g(z)\leq\beta\\}.$

Furthermore, the function $G:\Gamma_{g}^{\leq}\times[0,1]\to\Gamma_{g}^{\leq}$
defined by

$G((z,\beta),t)=\left(z,\beta(1-t)+g(z)t\right)$

is a strong deformation retract of $\Gamma_{g}^{\leq}$ onto the graph of $g$.
Since $g$ is continuous, the graph of $g$ is homeomorphic to $Z$. Hence, we
have shown $\Gamma_{f}^{<}$ is homotopy equivalent to $Z$, as desired. ∎

######  Theorem 3.2.

Suppose $X$ is a smooth projective surface over $\mathbb{C}$. The manifold
$\operatorname{Stab}^{\mathrm{geo}}(X)$ is contractible.

###### Proof.

By [dell2023]*Lemma 4.6, we have

$\Phi(H,D,\beta)\leq\frac{1}{2}\left(\left(\beta-\frac{D\cdot
H}{H^{2}}\right)^{2}-\frac{D^{2}}{H^{2}}\right)$

for all
$(H,D,\beta)\in\operatorname{Amp}_{\mathbb{R}}(X)\times\operatorname{NS}_{\mathbb{R}}(X)\times\mathbb{R}$
where $\Phi$ is the generalized Le Potier function (see Definition 2.3).
Therefore, by Theorem 2.4 and Lemma 3.1, it suffices to show
$\operatorname{Amp}_{\mathbb{R}}(X)\times\operatorname{NS}_{\mathbb{R}}(X)\times\mathbb{R}$
is contractible.

Since $\operatorname{NS}(X)$ is a finitely generated abelian group, there is a
homeomorphism $\operatorname{NS}_{\mathbb{R}}(X)=\mathbb{R}^{m}$ for some
$m\geq 1$. Similarly, since $\operatorname{Amp}_{\mathbb{R}}(X)$ is an open
cone in $\operatorname{NS}_{\mathbb{R}}(X)=\mathbb{R}^{m}$,
$\operatorname{Amp}_{\mathbb{R}}(X)$ is also contractible. Thus,
$\operatorname{NS}_{\mathbb{R}}(X)\times\operatorname{Amp}_{\mathbb{R}}(X)\times\mathbb{R}$
is contractible and so $\operatorname{Stab}^{\mathrm{geo}}(X)$ is also
contractible, as desired. ∎

We use Theorem 3.2 to give new examples of smooth projective surfaces with
contractible stability manifold.

######  Example 3.3 (Varieties with Finite Albanese).

If $X$ is a surface with finite Albanese (i.e. the Albanese morphism
$\operatorname{alb}_{X}:X\to\operatorname{Alb}(X)$ is finite onto its image)
then $\operatorname{Stab}^{\mathrm{geo}}(X)=\operatorname{Stab}(X)$
[fu2022]*Theorem 1.1. Therefore, by Theorem 3.2, $\operatorname{Stab}(X)$ is
contractible. This example generalizes [fu2022]*Theorem 1.2 to any Néron-
Severi rank.

A variety $X$ has finite Albanese if and only if $X$ is a finite cover of a
subvariety of an abelian variety. Therefore, any product of varieties with
finite Albanese also has finite Albanese. In particular, if $X$ is a finite
cover of $C_{1}\times C_{2}$ for smooth positive genus curves $C_{1},C_{2}$ or
$X$ is a finite cover of an abelian surface then $\operatorname{Stab}(X)$ is
contractible. This gives infinitely many new families of surfaces with
contractible stability manifold.

We also obtain some topological results for $\operatorname{Stab}(X)$ in cases
where $X$ does not have finite Albanese.

######  Example 3.4 (Free Quotients of Varieties with Finite Albanese).

If $X$ has finite Albanese and $G$ acts freely on $X$ then
$\operatorname{Stab}^{\mathrm{geo}}(X/G)$ is a connected component of
$\operatorname{Stab}(X)$ [dell2023]*Corollary 3.10. Therefore, by Theorem 3.2,
$\operatorname{Stab}(X)$ contains a contractible component.

This example applies to Beauville-type surfaces and bielliptic surfaces. For
these surfaces the irregularity is $0$ and $1$ respectively so they do not
have finite Albanese.

## References