#  AutoFT: Robust Fine-Tuning by Optimizing Hyperparameters on OOD Data

Caroline Choi∗1 Yoonho Lee∗1 Annie Chen1 Allan Zhou1 Aditi Raghunathan2
Chelsea Finn1
1Stanford University 2Carnegie Mellon University
{cchoi1<EMAIL_ADDRESS>
###### Abstract

Foundation models encode rich representations that can be adapted to a desired
task by fine-tuning on task-specific data. However, fine-tuning a model on one
particular data distribution often compromises the model’s original
performance on other distributions. Current methods for robust fine-tuning
utilize hand-crafted regularization techniques to constrain the fine-tuning
process towards the base foundation model. Yet, it is hard to precisely
specify what characteristics of the foundation model to retain during fine-
tuning, as this depends on how the pre-training, fine-tuning, and evaluation
data distributions relate to each other. We propose AutoFT, a data-driven
approach for guiding foundation model fine-tuning. AutoFT optimizes fine-
tuning hyperparameters to maximize performance on a small out-of-distribution
(OOD) validation set. To guide fine-tuning in a granular way, AutoFT searches
a highly expressive hyperparameter space that includes weight coefficients for
many different losses, in addition to learning rate and weight decay values.
We evaluate AutoFT on nine natural distribution shifts which include domain
shifts and subpopulation shifts. Our experiments show that AutoFT
significantly improves generalization to new OOD data, outperforming existing
robust fine-tuning methods. Notably, AutoFT achieves new state-of-the-art
performance on the WILDS-iWildCam and WILDS-FMoW benchmarks, outperforming the
previous best methods by $6.0\%$ and $1.5\%$, respectively.

##  1 Introduction

Foundation models have emerged as a powerful tool in machine learning,
demonstrating unprecedented performance across a wide variety of data
distributions [72, 40, 41]. By pre-training on large and diverse datasets,
these models learn representations that can serve as rich common-sense priors
that complement task-specific data. We thus expect fine-tuning to enhance the
generalization capabilities of foundation models. However, fine-tuning often
degrades the performance of foundation models on out-of-distribution (OOD)
data. This indicates that conventional fine-tuning strategies can fail to
utilize the prior knowledge embedded in the foundation model.

Figure 1:  Overview of AutoFT: a method for robustly fine-tuning foundation
models. While fine-tuning with in-distribution (ID) data (blue), AutoFT
optimizes hyperparameters for post-adaptation performance on a small out-of-
distribution validation set (red). This validation set serves as a proxy for
performance on different distributions (green and purple), encouraging AutoFT
to learn a robust fine-tuning procedure.  Figure 2:  A summary of our data
assumptions and evaluation protocol. The standard approach is to optimize
hyperparameters on a validation dataset drawn from the same distribution as
the training data. In contrast, AutoFT employs a small out-of-distribution
(OOD) validataion set for hyperparameter optimization, enhancing the
generalizability of the final model. We evaluate all fine-tuned models on data
from unseen distribution shifts (green and purple).

This issue of conventional fine-tuning distorting beneficial foundation model
priors has driven recent research on developing robust fine-tuning methods.
Such methods aim to produce an adapted model that achieves good performance
under distribution shifts by preserving the prior knowledge embedded in the
foundation model. Prior works have proposed various regularization techniques
for this purpose, such as ensembling models before and after adaptation [92]
or initially fitting only the last layer [50]. However, as these methods are
primarily based on human intuition, they may not fully account for the complex
interplay between the foundation model priors and the adaptation process.

We introduce AutoFT, a novel method for robust fine-tuning that aims to find
the right tradeoff between the prior and the fine-tuning data through
hyperparameter optimization. Our key insight is that we can learn what
characteristics of the foundation model to preserve during fine-tuning using a
data-driven approach. Like existing robust fine-tuning methods, we fine-tune a
foundation model on task-specific data, and then evaluate the resulting model
on a set of OOD distributions. However, we additionally leverage a small OOD
validation set with up to $1000$ labeled examples from one unseen
distribution; we optimize fine-tuning hyperparameters for post-adaptation
performance on this OOD validation set. Importantly, the OOD validation set is
only used for hyperparameter optimization, not fine-tuning, and does not
follow the same distribution as the OOD test sets. We illustrate the intuition
behind our approach in Figure 1 and our data assumptions in Figure 2.

We make two key alterations to standard hyperparameter optimization, which we
find to be critical for automatic robust fine-tuning. First, as mentioned
above, we optimize hyperparameters with respect to an OOD validation set
rather than an ID validation set. Second, we use a broader definition of
“hyperparameter”: beyond the usual hyperparameters such as learning rate, we
parameterize the fine-tuning objective itself through weight coefficients for
several different loss functions and regularizers. This larger hyperparameter
search space gives AutoFT more granular control over adaptation.

We rigorously evaluate AutoFT on a wide array of real-world datasets and
consider various types of distribution shifts, including subpopulation and
domain shift. Our experiments show that our approach results in better
generalization to unseen OOD data. With at most $1000$ datapoints from an OOD
distribution, AutoFT outperforms existing robust fine-tuning methods across
all benchmarks. These gains in robustness are achieved with minimal additional
compute, requiring at most $5\%$ more total compute compared to standard fine-
tuning. Among other results, AutoFT achieves new state-of-the-art performance
on the challenging iWildCam and FMoW benchmarks [9, 47, 16], outperforming the
prior best methods by $6.0\%$ and $1.5\%$, respectively.

##  2 Related Work

AutoML and hyperparameter optimization. Our work leverages high-level ideas
from the broader literature on meta-learning and hyperparameter optimization.
Such methods have proposed to optimize different parts of the training
pipeline, including general hyperparameters [39, 11, 26, 58, 59], network
architectures [100, 101, 62, 74, 93], augmentation policies [19, 60, 34, 20],
optimizers [10, 4, 89, 65, 15], and objective functions [97, 46, 68, 8].
However, most of these works optimize for generalization within the training
distribution and do not consider robustness to distribution shifts. Existing
works that optimize a training procedure for OOD generalization consider a
structured few-shot adaptation setting [56, 99], limiting their scalability to
large datasets.

Transfer learning. While early research demonstrated the effectiveness of
using features learned from pre-training on large datasets for new tasks,
transfer learning has evolved to focus on optimizing performance in limited
data scenarios [69, 96, 81]. Common transfer learning techniques include fine-
tuning with regularization [98, 94, 52, 42, 57, 1, 29, 82, 43] and selectively
freezing pre-trained parameters [45, 53, 33, 73, 63, 77, 22, 24, 23, 17, 85,
54, 51]. However, as pre-trained models are fine-tuned for a specific
distribution, their effective robustness decreases at convergence [3]. We
introduce a transfer learning framework that preserves the robustness of the
pre-trained model while adapting to new tasks.

Out-of-distribution generalization. Maintaining good performance on data that
deviates from the training distribution is crucial in many real-world
applications, where models may face data from unfamiliar environments [35, 28,
32, 47]. Numerous studies have investigated how to ensure robustness to
various distribution shifts [86, 14, 36, 5, 80, 61, 90, 3, 18, 55]. Some works
have shown that despite the myriad of ways in which data distributions can
change, naturally occurring distribution shifts have a surprisingly
predictable effect on model performance [83, 66, 6], suggesting that it may be
possible to learn how to be robust to unseen natural distribution shifts.
Goyal et al. [31] is related to our work: they dynamically adapt models to
distribution shifts via a meta-learned test-time adaptation loss applied to
unlabeled test data. AutoFT instead focuses on fine-tuning foundation models
and does not require any test data during training. AutoFT improves
generalization to novel distribution shifts after fine-tuning using only a
small OOD validation set.

Robust fine-tuning. Foundation models trained on massive datasets encode a
broad range of general knowledge, enabling robust performance across various
data distributions, including OOD scenarios [72, 13]. While in principle,
foundation models should serve as a useful prior for further fine-tuning,
empirical evidence shows that fine-tuning foundation models on a new task
often leads to a significant drop in OOD performance [3]. Recent works have
proposed modifications to the basic fine-tuning procedure to improve OOD
generalization [92, 50, 91, 30, 67]. Instead of hand-designing a
regularization technique, we propose a data-driven approach to learn a more
nuanced fine-tuning procedure. In fact, some prior works [94, 54, 30] can be
seen as special cases of AutoFT since our hyperparameter search space
encompasses these fine-tuning algorithms. Our experiments in Sec. 6
demonstrate that AutoFT consistently outperforms prior robust fine-tuning
methods on OOD data.

##  3 Hyperparameter Optimization

We begin by formalizing hyperparameter optimization, a procedure that we
extend for robustly fine-tuning foundation models in Sec. 4. Hyperparameters
are predefined properties of the learning algorithm which are not learned
during training, such as network architecture, learning rate, and
regularization strength. Since hyperparameters significantly impact the
performance of the final model, it is crucial to choose the right
hyperparameters for any learning problem. It is difficult to know the optimal
hyperparameters a priori, as they are influenced by many properties of the
problem setting, including the data distribution and desired performance
metric.

Formally, we denote the learning algorithm as LearnAlg and its hyperparameters
as $\phi\in\Phi$, where $\Phi$ is the hyperparameter space. We also denote the
training and validation datasets as $D_{\text{tr}}$ and $D_{\text{val}}$,
respectively. These datasets are disjoint and are typically drawn from the
same distribution. We denote the resulting model as
$\text{LearnAlg}(\phi,D_{\text{tr}})$, to explicitly represent the dependence
on both hyperparameters $\phi$ and training data $D_{\text{tr}}$. The goal of
hyperparameter optimization is to find hyperparameters that maximize some
performance metric $\mathrm{Perf}(f,D_{\text{val}})$ which depends on the
model $f$ and the validation dataset $D_{\text{val}}$. Examples of performance
metrics include top-1 accuracy, macro F1 score, and worst-region accuracy. We
can formally state the hyperparameter optimization problem as

$\phi^{*}=\operatorname*{arg\,max}_{\phi\in\Phi}\mathbb{E}\big{[}\overbrace{\mathrm{Perf}(\underbrace{\text{LearnAlg}(\phi,D_{\text{tr}})}_{\text{Learned
Parameters}},D_{\text{val}})}^{\text{Validation Set Performance}}\big{]}.$ (1)

Here, the expectation is taken over any randomness in the learning algorithm
LearnAlg, such as input data shuffling or random initialization. The optimized
hyperparameters $\phi^{*}$ are subsequently used to train the model.

Prior hyperparameter optimization methods typically start with randomly
initialized model parameters and use a validation set $D_{\text{val}}$, drawn
from the same distribution as the training data, to adjust hyperparameters.
The problem of robust fine-tuning, however, begins with pre-trained model
parameters and aims to achieve high performance on test data that diverges
from the original training data’s distribution. In the next section, we
describe how we modify the standard hyperparameter optimization loop for
robustly fine-tuning pre-trained foundation models.

##  4 AutoFT: Robust Fine-Tuning via Hyperparameter Optimization

In this section, we present AutoFT, a data-driven method for robustly fine-
tuning large pre-trained models. AutoFT aims to find hyperparameters for fine-
tuning a pre-trained model that strike a better balance between the general-
purpose knowledge encoded in the pre-trained model and the task-specific
knowledge in the training data. To achieve this, AutoFT optimizes
hyperparameters with respect to a performance metric that is sensitive to
degradation under distribution shift. Specifically, we leverage a small
validation set from a different distribution from the fine-tuning data;
performance on this OOD validation set serves as a proxy for general OOD
performance.

###  4.1 Problem Setting: Robust Fine-Tuning

During hyperparameter optimization, we assume access to two datasets: (1) a
large fine-tuning dataset $D_{\text{tr}}$ from the training distribution
$\mathcal{P}_{\text{tr}}$, and (2) a small held-out OOD validation set
$D_{\text{val}}$ from a different distribution $\mathcal{P}_{\text{val}}$. The
validation set $D_{\text{val}}$ is much smaller than $D_{\text{tr}}$, and is
used exclusively for hyperparameter optimization, not for fine-tuning. After
hyperparameter optimization, we fine-tune on $D_{\text{tr}}$ with the best
found hyperparameters and then test the fine-tuned model on several OOD
distributions
$\mathcal{P}_{\text{ood}}^{1},\mathcal{P}_{\text{ood}}^{2},\dots$. These test
distributions are different from both $\mathcal{P}_{\text{tr}}$ and
$\mathcal{P}_{\text{val}}$, and are not shown to the model during either fine-
tuning or hyperparameter optimization. We visualize this data assumption in
Figure 2.

Let $f$ denote a pre-trained model with parameters $\theta$, which we adapt to
the task at hand by fine-tuning on $D_{\text{tr}}$. We consider foundation
models $f$ that possess rich, general-purpose knowledge from being pretrained
on large, diverse datasets. Let $\phi\in\Phi$ represent the hyperparameters
for fine-tuning, and let $\text{LearnAlg}(\phi,D_{\text{tr}})$ denote the
fine-tuning algorithm which produces adapted model parameters. In this work,
the fine-tuning algorithm LearnAlg is stochastic gradient descent. The
performance metric $\mathrm{Perf}$ differs across the experiments we consider:
it is top-1 accuracy for most settings.

We assume that all data distributions pertain to the same task, with
differences arising from domain shifts caused by natural variation in the data
collection process. For example, all data may be for the same task of
recognizing animals from camera trap images, with each distribution
corresponding to a different set of locations [9, 47]. We posit that using one
small OOD validation set $D_{\text{val}}$ can guide the model to generalize
better to various new OOD distributions. This validation set reflects the type
of distribution shifts that the model might encounter. Performance on this set
can serve as an effective proxy for performance on novel OOD distributions.
Our experiments in Sec. 6 confirm that AutoFT can successfully leverage one
small OOD validation set to improve generalization to novel distributions,
outperforming state-of-the-art methods for robust fine-tuning.

Algorithm 1 AutoFT

Input Hyperparameter optimizer $\mathrm{HPO}$

Input ID training data $D_{\text{tr}}$, OOD validation data $D_{\text{val}}$

for $\phi\leftarrow\mathrm{HPO}.\mathrm{Sample}()$ do

$f_{ft}\leftarrow\text{LearnAlg}(D_{\text{tr}},\phi)$ // Fine-tune model

$p\leftarrow\mathrm{Perf}(f_{ft},D_{\text{val}})$ // Evaluate

$\mathrm{HPO}.\mathrm{Update}(\phi,p)$ // HPO Update

end for

$\phi^{*}\leftarrow\mathrm{HPO}.\mathrm{Best}()$ // Get best hyperparameters

$\theta^{*}\leftarrow\text{LearnAlg}(D_{\text{tr}},\phi^{*})$ // Fine-tune
final model

Figure 3:  Didactic experiment. Left two: learning curves for fitting a
Gaussian distribution to toy data. Hyperparameter optimization with a larger
hyperparameter space (red) leads to better OOD performance only when the prior
is informative. Right: fine-tuning learning curves with learned
hyperparameters. Averaging the dimension-wise weights (yellow) results in
massively overfitting to the ID data (solid) and underperforming OOD (dashed).
The dimension-wise hyperparameters (red) show the best generalization. Far
right: visualization of learned loss weights $w_{i}$ for each dimension. The
learned weights are lowest for the dimensions where ID alone differs from the
prior (red), indicating that the model learns to ignore the ID data in these
dimensions.

###  4.2 Hyperparameter Optimization Loop

Formulation. We formalize our hyperparameter optimization as follows. Given a
hyperparameter space $\Phi$ for a fine-tuning algorithm LearnAlg, we solve the
optimization problem in 1 to find hyperparameters $\phi^{*}$ such that the
fine-tuned model $f^{\prime}=\text{LearnAlg}(\phi^{*},D_{\text{tr}})$ performs
well on the OOD validation set $D_{\text{val}}$. Practically, this involves
fine-tuning the same foundation model multiple times, each time with different
hyperparameters $\phi^{1},\phi^{2},\ldots$ to obtain different final fine-
tuned models $f^{1}=\text{LearnAlg}(\phi^{1},D_{\text{tr}}),f^{2}=\cdots$. We
then judge the quality of each set of hyperparameters $\phi^{i}$ by the
performance of $f^{i}$ on the OOD validation set. The hyperparameter
optimization algorithm aggregates the information from these different fine-
tuning runs and estimates the best hyperparameters $\phi^{*}$. At test time,
we fine-tune the initial foundation model with hyperparameters $\phi^{*}$, and
evaluate its performance on novel OOD distributions to assess its
generalization capabilities. We summarize this procedure in Algorithm 1.

Hyperparameter space. We consider a larger hyperparameter space $\Phi$ than
what is typically considered. This is because for robust fine-tuning, the
algorithm must selectively ignore some aspects of the training distribution in
order to better generalize to novel OOD distributions. To enable this more
nuanced behavior, we increase the expressivity of the hyperparameter
optimization process by expanding the hyperparameter space $\Phi$. The main
novelty of our hyperparameters space is that we allow the hyperparameter space
$\Phi$ to express the fine-tuning objective itself by learning to weight
between different loss functions and regularizers. For example, the cross-
entropy and hinge losses similarly guide the model’s predictions towards the
correct label, but with a different amount of penalty for severely incorrect
samples. As another example, contrastive loss on image-text pairs [72, 30]
also guides the model’s predictions towards the correct label, but in a way
that incorporates language embeddings which is entirely different from how the
cross-entropy and hinge losses operate. Similarly, different losses for
reducing overconfidence or constraining model parameters towards their initial
values have different effects on the model’s adaptation process.

As hyperparameters, we consider weight coefficients for nine different loss
functions and regularizers: cross-entropy loss, hinge loss, image-text
contrastive loss, entropy, confidence minimization, L1 norm, L2 norm, L1 and
L2 distance to pretrained model parameters. We select these losses to address
various aspects of model learning and generalization, such as classification
accuracy, decision confidence, and prevention of overfitting, ensuring a
nuanced and effective model adaptation. We denote these loss weight
coefficients as $W=\\{w_{1},w_{2},\dots,w_{9}\\}$. Denoting the $i$-th loss
function or regularizer as $\mathcal{L}_{i}$, the total loss $\mathcal{L}$ is
the weighted sum $\mathcal{L}=\sum_{i=1}^{9}w_{i}\mathcal{L}_{i}$. To enable
finer control over the speed and stability of adaptation during fine-tuning,
we additionally learn hyperparameters for the fine-tuning optimizer: learning
rate $\eta$ and weight decay $\delta$. For an even larger hyperparameter
space, we optionally consider per-layer hyperparameters for the learning rate,
weight decay, and parameter norm weights; we call this variant “per-layer” in
our experiments in Appendix A. Our complete set of hyperparameters is thus
$\phi=(W,\eta,\delta)$.

Hyperparameter optimization algorithm. Finding effective hyperparameters for
robust fine-tuning requires an optimizer that can efficiently search our
large, high-dimensional hyperparameter space $\Phi$. We use the Tree-
structured Parzen Estimator [12, TPE] as the hyperparameter optimization
algorithm, as its design allows for efficient processing in high-dimensional
spaces and yields effective results at the order of hundreds of evaluations.
TPE creates a probabilistic model from previous hyperparameter evaluations,
and differentiates between better and worse hyperparameter configurations by
comparing their likelihood under two separate models. We use the TPE
implementation of the optuna library [2] with no further modifications.

As the prior distribution for TPE, we use an appropriately scaled distribution
for each hyperparameter, operating in the log-space for the weight
coefficients and the learning rate since the optimal values can vary over
several orders of magnitude:

$\displaystyle w_{i}\sim\text{LogUniform}$
$\displaystyle(w_{\text{min}},w_{\text{max}}),$
$\displaystyle\eta\sim\text{LogUniform}(\eta_{\text{min}},\,\eta_{\text{max}}),\,$
$\displaystyle\delta\sim\text{Uniform}(0.0,1.0).$

We find that in practice, $(w_{\text{min}},w_{\text{max}})=(10^{-4},10)$ is an
effective range for all $w_{i}$. Given a model’s conventional learning rate
$\eta^{*}$, we set
$(\eta_{\text{min}},\eta_{\text{max}})=(10^{-2}\eta^{*},10^{2}\eta^{*})$.

Computational cost. The hyperparameter tuning of AutoFT introduces minimal
computational overhead, requiring at most $5\%$ additional compute compared to
one standard fine-tuning run. Each evaluation of hyperparameters on the OOD
validation set requires only a few inner loop gradient steps on the fine-
tuning dataset. For example, on the iWildCam dataset, AutoFT uses only 500
hyperparameter evaluations, each requiring only 10 gradient steps on the fine-
tuning data. With 1.3 million training examples and a batch size of 256,
AutoFT requires only 5000 additional gradient steps, compared to approximately
100,000 gradient steps for final fine-tuning. AutoFT is therefore
computationally inexpensive, and highly practical for even large fine-tuning
problem settings.

###  4.3 Intuition and Didactic Experiment

To provide intuition for how hyperparameter optimization can lead to more
effective utilization of a partially useful prior, we consider a simple
experiment. We focus on synthetic datasets of vectors drawn from zero-mean
Gaussian distributions with different variances. We assume that the prior unit
variance distribution is helpful but that the data exhibits deviation from
this prior along a few dimensions. A detailed summary of our data and training
setup is in Sec. B.1. We consider “fine-tuning” as starting from the prior
distribution. In this toy model, dimension-wise weight hyperparameters $w_{i}$
dictate the degree of influence of the prior versus the training data on each
dimension, analogous to how the fine-grained hyperparameters in AutoFT balance
the foundation model prior and training data along different aspects of fine-
tuning. As in AutoFT, we use the TPE algorithm to optimize the weight
hyperparameters $w_{i}$ to maximize the log-likelihood of the validation data
$D_{\text{val}}$.

Results in Figure 3 show the effect of standard hyperparameter tuning (global
weight) versus considering an expanded hyperparameter space as in AutoFT (dim-
wise weight). Learning dimension-wise weights is effective, but only when the
prior distribution is well-specified and provides a better signal for OOD
generalization than the training set along some dimensions, as foundation
models do for general fine-tuning tasks. This additional expressivity is
necessary to specify a better adaptation procedure: naively averaging the
learned dim-wise parameters causes the model to overfit the ID data. Finally,
through inspecting the learned weights, we see that the learned dim-wise
weights improve OOD generalization by downweighting the influence of the ID
data in the directions where only the ID data differs from the prior (not the
OOD data).

##  5 Experimental Setup

Distribution shifts. In our analysis, we focus on natural distribution shifts,
defined by Taori et al. [83] as shifts arising from real-world variations,
such as changes in lighting, geography, and image styles. This approach, in
line with prior works on robust fine-tuning [72, 92, 50, 30], emphasizes
shifts that are representative of real-world scenarios.

Our main results include four distribution shifts, including real-world
distribution shifts arising in wildlife recognition and satellite imagery, and
two CIFAR-10-derived distribution shifts. The WILDS-iWildCam dataset [9, 47,
79] presents distribution shifts arising from variations in camera trap
locations, lighting conditions, and animal behaviors across various geographic
regions. The WILDS-FMoW dataset [16, 47, 79] presents distribution shifts
arising from changes in time, geographic locations, and land use. Finally, we
evaluate on two CIFAR-10 derived distribution shifts: CIFAR-10 to CIFAR-10.1
and CIFAR-10.2, which involve subtle changes in image characteristics and
composition.

Foundation models and CLIP. We fine-tune pre-trained CLIP models [72],
including those provided in the open-clip repository [40]. We use the CLIP
ViT-B/16 model from OpenAI as our default model, unless specified otherwise.
Evaluation on CIFAR-10 uses the CLIP ViT-L/14 model from OpenAI, in line with
[92]. Our SoTA results on WILDS-iWildCam use the CLIP ViT-L/14-336px model
from OpenAI. We use text templates used in prior work [72, 92] to generate
zero-shot final layer weights for all datasets.

Effective robustness and weight ensembling curves. We use the effective
robustness framework by Taori et al. [83] to evaluate model robustness based
on accuracy exceeding a baseline trained only on the reference distribution.
Linearly interpolating the weights of a fine-tuned model and a pretrained
model (WiSE-FT) has been shown to improve both ID and OOD performance [91].
Hence, we include weight ensembling as an additional point of comparison, and
interpolate the weights of models fine-tuned by each method with 10 mixing
coefficients $\alpha$.

Baselines. We compare AutoFT against several methods for adapting pretrained
models. We include two standard transfer learning methods that minimize cross-
entropy loss: linear probing (LP) and full fine-tuning (FT). We also compare
with recent works in robust fine-tuning: L2-SP [59], which fine-tunes with an
L2 regularization term towards pretrained weights; LP-FT [50], which performs
linear probing followed by full fine-tuning; Freeze-Embed [51], which freezes
the embedding layer during fine-tuning; and FLYP [30], which fine-tunes with
the CLIP pretraining loss – a contrastive loss between image embeddings and
class-descriptive prompt embeddings. As we are interested in OOD performance,
we also compare against OOD generalization methods including Group DRO [78],
ABSGD [70], LISA [95], DFR [44], and Copy-Paste [27]. We additionally evaluate
all methods with weight ensembling (WiSE-FT) [92], which is shown to improve
OOD performance in an orthogonal way to other robust fine-tuning methods.

Training protocol. We closely follow the training details of Goyal et al. [30]
and Wortsman et al. [92]. All methods fine-tune models with an AdamW
optimizer, cosine learning rate scheduler, and a batch size of 512 for
ImageNet and 256 for all other datasets. All baseline hyperparameters, such as
learning rate, weight decay, and warmup length, are tuned through grid search.
All methods, including AutoFT, perform early stopping based on in-distribution
(ID) validation accuracy. We provide a comprehensive breakdown of the
hyperparameter sweeps in the supplementary material. We emphasize that none of
these methods, including AutoFT, observe any of the test OOD distributions
during training. Finally, we report metrics averaged over 5 runs with 95%
confidence intervals.

##  6 Results

Method | Architecture | ID | OOD
---|---|---|---
Group DRO | ResNet50 | 37.5 (1.9) | 23.8 (2.0)
ABSGD | ResNet50 | 47.5 (1.6) | 33.0 (0.6)
Copy-Paste | ResNet50 | 50.2 (1.6) | 36.5 (0.9)
ERM | PNASNet | 52.8 (1.4) | 38.5 (0.6)
ERM | ViTL | 55.8 (1.9) | 41.4 (0.5)
Model Soups | ViTL | 57.6 (1.9) | 43.3 (1.0)
FLYP | ViTL-336px | 59.9 (0.7) | 46.0 (1.3)
AutoFT | ViTL-336px | 63.5 (0.5) | 52.0 (0.4)
Table 1:  iWildCam results. AutoFT with weight ensembling attains state-of-the-art OOD performance on the WILDS-iWildCam benchmark with a ViT-L/14-336px backbone, surpassing all prior entries on the WILDS leaderboard [47]. Method | Architecture | ID | OOD
---|---|---|---
Group DRO | DenseNet121 | 51.2 (0.4) | 31.1 (1.7)
LISA | DenseNet121 | 52.8 (1.2) | 35.5 (0.8)
ERM w/ aug | DenseNet121 | 55.5 (0.4) | 35.7 (0.3)
DFR | DenseNet121 | 53.4 (0.4) | 42.8 (0.4)
ERM | ViTL | 66.9 (0.2) | 46.1 (0.6)
Model Soups | ViTL | 69.5 (0.1) | 47.6 (0.3)
Freeze-Embed | ViTL-336px | 68.3 (0.4) | 50.3 (1.1)
AutoFT | ViTL-336px | 72.1 (0.1) | 51.8 (0.4)
Table 2:  FMoW results. AutoFT with weight ensembling attains state-of-the-art
OOD performance on the WILDS-FMoW benchmark with a ViT-L/14-336px backbone,
surpassing all prior entries on the WILDS leaderboard [47].

| ImageNet | iWILDCam | FMoW
---|---|---|---
| Without Ensembling | With Ensembling | Without Ensembling | With Ensembling | Without Ensembling
Methods | ID | OOD | ID | OOD | ID | OOD | ID | OOD | ID | OOD
Zeroshot | 68.3 (-) | 58.7 (-) | 68.3 (-) | 58.7 (-) | 8.7 (-) | 11.0 (-) | 8.7 (-) | 11.0 (-) | 20.4 (-) | 18.7 (-)
LP | 79.9 (0.0) | 57.2 (0.0) | 80.0 (0.0) | 58.3 (0.0) | 44.5 (0.6) | 31.1 (0.4) | 45.5 (0.6) | 31.7 (0.4) | 48.2 (0.1) | 30.5 (0.3)
FT | 81.4 (0.1) | 54.8 (0.1) | 82.5 (0.1) | 61.3 (0.1) | 48.1 (0.5) | 35.0 (0.5) | 48.1 (0.5) | 35.0 (0.5) | 68.5 (0.1) | 39.2 (0.7)
L2-SP | 81.6 (0.1) | 57.9 (0.1) | 82.2 (0.1) | 58.9 (0.1) | 48.6 (0.4) | 35.3 (0.3) | 48.6 (0.4) | 35.3 (0.3) | 68.6 (0.1) | 39.4 (0.6)
LP-FT | 81.8 (0.1) | 60.5 (0.1) | 82.1 (0.1) | 61.8 (0.1) | 49.7 (0.5) | 34.7 (0.4) | 50.2 (0.5) | 35.7 (0.4) | 68.4 (0.2) | 40.4 (1.0)
FLYP | 82.6 (0.0) | 60.2 (0.1) | 82.9 (0.0) | 63.2 (0.1) | 52.2 (0.6) | 35.6 (1.2) | 52.5 (0.6) | 37.1 (1.2) | 68.6 (0.2) | 41.3 (0.8)
AutoFT | 81.8 (0.1) | 61.5 (0.1) | 82.4 (0.1) | 64.3 (0.1) | 51.0 (0.5) | 38.3 (0.5) | 51.3 (0.5) | 39.3 (0.5) | 67.1 (0.3) | 42.3 (0.5)

Table 3:  AutoFT outperforms all baselines both with and without ensembling.
Without ensembling, AutoFT improves OOD performance by $1.3\%$ on ImageNet,
$3.7\%$ on WILDS-iWildCam, and $6.1\%$ on WILDS-FMoW. These improvements are
preserved with weight ensembling.  Figure 4:  AutoFT outperforms existing
methods, both with and without weight ensembling [92]. Here, we show the ID-
OOD performance curves obtained by linearly interpolating the fine-tuned model
weights with the zero-shot weights.

In this section, we present the main experimental findings for AutoFT. First,
we show that AutoFT improves the performance of fine-tuned models on several
large-scale, synthetic and natural distribution shifts, including ImageNet,
WILDS-iWildCam, WILDS-FMoW, and CIFAR. Then, we present additional experiments
in the low-data and transfer learning regimes. Finally, we investigate the
transferability of hyperparameters learned by AutoFT across fine-tuning
dataset and backbone, and the effect of the choice of OOD validation
distribution for hyperparameter optimization. These findings highlight the
effectiveness of AutoFT for robust fine-tuning in a variety of settings.

| PatchCamelyon | SST2
---|---|---
$k$ (shots) | 4 | 16 | 32 | 4 | 16 | 32
Zeroshot | 56.5 (-) | 56.5 (-) | 56.5 (-) | 60.5 (-) | 60.5 (-) | 60.5 (-)
LP | 60.4 (4.0) | 64.4 (3.7) | 67.0 (4.4) | 60.8 (1.8) | 61.9 (1.4) | 62.9 (1.3)
FT | 63.1 (5.5) | 71.6 (4.6) | 75.2 (3.7) | 61.1 (0.7) | 62.4 (1.6) | 63.4 (1.9)
LP-FT | 62.7 (5.3) | 69.8 (5.3) | 73.9 (4.6) | 60.9 (2.4) | 62.9 (1.9) | 63.6 (1.4)
FLYP | 66.9 (5.0) | 74.5 (2.0) | 76.4 (2.4) | 61.3 (2.7) | 65.6 (2.1) | 68.0 (1.7)
AutoFT | 68.1 (5.1) | 76.8 (2.9) | 79.5 (2.0) | 65.0 (3.8) | 67.5 (1.1) | 69.0 (1.1)
Table 4: AutoFT shows superior performance in binary few-shot classification. AutoFT outperforms FLYP by $3.1\%$ and full fine-tuning by $4.3\%$ in 32-shot classification on PatchCamelyon. Methods | CalTech-101 | Stanford Cars | Flowers-102
---|---|---|---
Zeroshot | 87.7 (-) | 64.4 (-) | 71.2 (-)
LP | 94.8 (0.0) | 83.1 (0.0) | 95.9 (0.0)
FT | 97.2 (0.1) | 84.4 (0.3) | 90.4 (0.5)
LP-FT | 96.9 (0.6) | 89.4 (0.1) | 97.9 (0.1)
FLYP | 97.6 (0.1) | 89.6 (0.3) | 97.7 (0.1)
AutoFT | 99.0 (0.1) | 89.6 (0.2) | 97.6 (0.2)
Table 5: In addition to enhancing robustness, AutoFT can improve performance in IID settings, where there is no distribution shift. Here, AutoFT uses ID validation examples to optimize hyperparameters. AutoFT achieves the highest ID performance on CalTech-101 and matches the best baseline on Stanford Cars. | MNIST | CIFAR10.2 | CIFAR10.1
---|---|---|---
Method | ID | OOD | ID | OOD | ID | OOD
FT | 99.8 | 62.3 | 97.9 | 96.2 | 98.3 | 93.4
AutoFT | 99.8 | 61.1 | 98.2 | 96.3 | 98.4 | 93.4
Table 6:  Performance comparison of models fine-tuned on new distributions with hyperparameters learned by AutoFT on a different dataset, CIFAR-10. Hyperparameters transfer better to datasets that are more similar to the original fine-tuning distribution.  Validation Dataset | CIFAR-10.1 | CIFAR-10.2
---|---|---
CIFAR-10-C | 97.4 (0.2) | 93.5 (0.2)
CIFAR-10 | 96.9 (0.2) | 93.2 (0.1)
CIFAR-10.1 | 96.8 (0.3) | 92.8 (0.2)
CIFAR-10.2 | 96.7 (0.5) | 93.8 (0.0)
Table 7:  Performance of AutoFT with different validation datasets for
hyperparameter evaluation. All AutoFT runs above optimize hyperparameters for
fine-tuning on CIFAR-10. AutoFT produces a robust fine-tuned model with all
validation datasets, with the best results achieved when the validation set is
from a different distribution than the fine-tuning dataset, CIFAR-10.

###  6.1 Evaluation Under Distribution Shifts

Improvements on ImageNet, WILDS, and CIFAR distribution shifts. We evaluate
AutoFT on nine natural distribution shifts: five from ImageNet, WILDS-FMoW,
WILDS-iWildCam, and two subtle CIFAR-10 distribution shifts. For the OOD
validation sets, we use up to $1000$ examples from ImageNet-C, the OOD
validation splits of WILDS-FMoW and WILDS-iWildCam, and CIFAR-10-C,
respectively. Tab. 3 contains results on ImageNet, WILDS-FMoW, and WILDS-
iWildCam; results on CIFAR-10 are in Tab. 9. AutoFT consistently outperforms
all baselines on novel OOD distributions. These gains in OOD performance are
maintained when ensembling zero-shot and fine-tuned models, following Wortsman
et al. [92]. We report weight ensembling results with the mixture coefficient
that yields the highest ID validation accuracy.

State-of-the-art performance on WILDS-iWildCam and WILDS-FMoW. To assess
whether performance gains by AutoFT continue to hold on larger foundation
models, we evaluate AutoFT with a ViT-L/14@336px model. As shown in Tabs. 1
and 2, AutoFT achieves significant gains of $6.0\%$ and $3.6\%$ in OOD and ID
macro-F1, respectively, over the current leader on the WILDS-iWildCam
benchmark [47], FLYP [30]. On the WILDS-FMoW benchmark, AutoFT also
outperforms the current leader, SGD (Freeze-Embed) [51], by $1.5\%$ and
$5.3\%$ in OOD and ID worst-region accuracy, respectively. On both benchmarks,
AutoFT additionally outperforms the compute-intensive Model Soups [91], which
ensembles more than 70 models fine-tuned with LP-FT and different
augmentations and hyperparameters. AutoFT also outperforms LP-FT, a state-of-
the-art baseline for fine-tuning.

###  6.2 Few-Shot Classification

In many real-world applications, limited amounts of labeled, task-specific
data are available for fine-tuning. Few-shot classification thus serves as an
important benchmark for evaluating the utility of fine-tuning approaches. Few-
shot binary classification is a particularly challenging task for adaptation,
given the small number of training examples. We evaluate on 4, 16, and 32 shot
binary classification tasks from the PatchCamelyon and Rendered-SST2 datasets,
following Radford et al. [72] and Goyal et al. [30]. PatchCamelyon contains
digital pathology images for the detection of metastatic tissue. Rendered-SST2
focuses on optical character recognition for classifying text sentiment as
positive or negative.

AutoFT demonstrates strong generalization capabilities with limited data,
outperforming all baselines on all few-shot tasks in Tab. 4. For example,
AutoFT outperforms FLYP by $3.7\%$ and full fine-tuning by $3.9\%$ in a
challenging 4-shot classification task on Rendered-SST2.

###  6.3 Transfer Learning

In standard transfer learning settings, where no distribution shifts exist
between train and test data, AutoFT shows strong performance. The primary goal
here is to achieve strong in-distribution (ID) performance on the test set. In
this setting, AutoFT uses a small ID validation set to optimize
hyperparameters.

We compare AutoFT against other baselines on several transfer learning
datasets: CalTech-101, StanfordCars, and Flowers-102 in Tab. 5. AutoFT matches
or outperforms all baselines on 2 of the 3 transfer learning datasets,
outperforming FLYP by $1.4\%$ on CalTech-101.

###  6.4 Ablation Studies

Hyperparameter transfer. Are the hyperparameters learned by AutoFT for one
fine-tuning distribution effective when applied to new fine-tuning
distributions? We first train AutoFT on CIFAR-10, using $100$ examples from
corrupted CIFAR-10 for hyperparameter optimization. We then fine-tune a model
with these learned hyperparameters on three new distributions: MNIST,
CIFAR-10.1, and CIFAR-10.2. Finally, we evaluate the fine-tuned model’s
performance on both the fine-tuning distribution and several distribution
shifts. For MNIST, we evaluate the fine-tuned model’s performance on three
distribution shifts: corrupted MNIST, EMNIST, and rotated MNIST. Similarly, we
evaluate the model fine-tuned on CIFAR-10.1 on CIFAR-10.2, and vice versa.
Further experimental details are in Appendix A. Results in Tab. 6 show model
performance after fine-tuning with these transferred hyperparameters. On
CIFAR-10.1 and CIFAR-10.2, AutoFT outperforms standard fine-tuning on ID and
OOD accuracy. However, on MNIST, AutoFT results in slightly worse OOD
performance than fine-tuning. This suggests that the hyperparameters optimized
by AutoFT transfer more effectively to datasets that are similar to the
original fine-tuning distribution.

Effect of the hyperparameter optimization dataset. How does the choice of
dataset for hyperparameter optimization affect the performance of the fine-
tuned model? AutoFT optimizes hyperparameters using a validation set distinct
from both the ID fine-tuning distribution and OOD test sets. In Table 7, we
assess the performance of AutoFT using different datasets for hyperparameter
optimization.

The results suggest that AutoFT yields strong OOD performance across a range
of validation datasets, with best OOD performance achieved when the validation
set used for hyperparameter optimization is different from the ID dataset.

##  7 Conclusion

We introduce AutoFT, a novel data-driven approach for robust fine-tuning that
optimizes an expressive set of hyperparameters using a small OOD validation
set. AutoFT only requires a small amount of data from one naturally occurring
OOD distribution—such data is often readily available or is possible to gather
at a similar cost to that of the original ID training data. Our empirical
results demonstrate that AutoFT consistently outperforms existing approaches
and achieves a new state-of-the-art for robust fine-tuning. We hope that our
work will inspire future research on data-driven approaches for robust fine-
tuning.

Future directions. While our experiments show strong results in image
classification, we have not yet verified the effectiveness of AutoFT outside
this problem setting. Future research could explore extending the general
workflow of AutoFT to other areas such as image segmentation, object
detection, and natural language processing (NLP). Moreover, investigating
alternative hyperparameter search spaces within AutoFT could offer further
insights. Lastly, exploring the possibility of identifying a single set of
hyperparameters that works effectively across multiple fine-tuning
distributions presents an interesting direction for future work.

##  8 Acknowledgements

We thank Kyle Hsu, Lukas Haas, and other members of the IRIS lab for helpful
feedback and discussions. We also thank Sachin Goyal for help with ImageNet
experiments. This work was supported by KFAS and NSF. A.R. is supported by
Schmidt Futures, Google, Apple, and Open Philanthropy. C.F. is supported by
ONR grant N00014-21-1-2685 and the NSF CAREER award.

## References

  * Aghajanyan et al. [2020] Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta.  Better fine-tuning by reducing representational collapse.  _arXiv preprint arXiv:2008.03156_ , 2020.
  * Akiba et al. [2019] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.  Optuna: A next-generation hyperparameter optimization framework.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 2623–2631, 2019.
  * Andreassen et al. [2021] Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs.  The evolution of out-of-distribution robustness throughout fine-tuning.  _arXiv preprint arXiv:2106.15831_ , 2021.
  * Andrychowicz et al. [2016] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas.  Learning to learn by gradient descent by gradient descent.  _Advances in neural information processing systems_ , 29, 2016.
  * Arjovsky et al. [2019] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization.  _arXiv preprint arXiv:1907.02893_ , 2019.
  * Baek et al. [2022] Christina Baek, Yiding Jiang, Aditi Raghunathan, and J Zico Kolter.  Agreement-on-the-line: Predicting the performance of neural networks under distribution shift.  _Advances in Neural Information Processing Systems_ , 35:19274–19289, 2022.
  * Barbu et al. [2019] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz.  Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.  _Advances in neural information processing systems_ , 32, 2019.
  * Bechtle et al. [2021] Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier.  Meta learning via learned loss.  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , pages 4161–4168. IEEE, 2021.
  * Beery et al. [2021] Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar.  The iwildcam 2021 competition dataset.  _arXiv preprint arXiv:2105.03494_ , 2021.
  * Bengio et al. [2013] Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gescei.  On the optimization of a synaptic learning rule.  In _Optimality in Biological and Artificial Networks?_ , pages 281–303. Routledge, 2013.
  * Bergstra and Bengio [2012] James Bergstra and Yoshua Bengio.  Random search for hyper-parameter optimization.  _Journal of machine learning research_ , 13(2), 2012.
  * Bergstra et al. [2011] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl.  Algorithms for hyper-parameter optimization.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2011.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Byrd and Lipton [2019] Jonathon Byrd and Zachary Lipton.  What is the effect of importance weighting in deep learning?  In _International Conference on Machine Learning_ , pages 872–881. PMLR, 2019.
  * Chen et al. [2023] Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al.  Symbolic discovery of optimization algorithms.  _arXiv preprint arXiv:2302.06675_ , 2023.
  * Christie et al. [2018] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee.  Functional map of the world.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6172–6180, 2018.
  * Cohen et al. [2022] Niv Cohen, Rinon Gal, Eli A Meirom, Gal Chechik, and Yuval Atzmon.  " this is my unicorn, fluffy": Personalizing frozen vision-language representations.  _arXiv preprint arXiv:2204.01694_ , 2022.
  * Creager et al. [2021] Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel.  Environment inference for invariant learning.  In _International Conference on Machine Learning_ , pages 2189–2200. PMLR, 2021.
  * Cubuk et al. [2019] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation strategies from data.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 113–123, 2019.
  * Cubuk et al. [2020] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pages 702–703, 2020.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Eastwood et al. [2021] Cian Eastwood, Ian Mason, Christopher KI Williams, and Bernhard Schölkopf.  Source-free adaptation to measurement shift via bottom-up feature restoration.  _arXiv preprint arXiv:2107.05446_ , 2021.
  * Eastwood et al. [2022] Cian Eastwood, Ian Mason, and Christopher KI Williams.  Unit-level surprise in neural networks.  In _I (Still) Can’t Believe It’s Not Better! Workshop at NeurIPS 2021_ , pages 33–40. PMLR, 2022.
  * Evci et al. [2022] Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer.  Head2toe: Utilizing intermediate representations for better transfer learning.  In _International Conference on Machine Learning_ , pages 6009–6033. PMLR, 2022.
  * Fei-Fei et al. [2004] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In _Computer Vision and Pattern Recognition Workshop_. IEEE, 2004\.
  * Feurer et al. [2015] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter.  Efficient and robust automated machine learning.  _Advances in neural information processing systems_ , 28, 2015.
  * Gao et al. [2023] Irena Gao, Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang.  Out-of-domain robustness via targeted augmentations.  _arXiv preprint arXiv:2302.11861_ , 2023.
  * Geirhos et al. [2020] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann.  Shortcut learning in deep neural networks.  _Nature Machine Intelligence_ , 2(11):665–673, 2020.
  * Gouk et al. [2021] Henry Gouk, Timothy Hospedales, and massimiliano pontil.  Distance-based regularisation of deep networks for fine-tuning.  In _International Conference on Learning Representations_ , 2021.
  * Goyal et al. [2022a] Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan.  Finetune like you pretrain: Improved finetuning of zero-shot vision models.  _arXiv preprint arXiv:2212.00638_ , 2022a.
  * Goyal et al. [2022b] Sachin Goyal, Mingjie Sun, Aditi Raghunathan, and J Zico Kolter.  Test time adaptation via conjugate pseudo-labels.  _Advances in Neural Information Processing Systems_ , 35:6204–6218, 2022b.
  * Gulrajani and Lopez-Paz [2020] Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  _arXiv preprint arXiv:2007.01434_ , 2020.
  * Guo et al. [2019] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris.  Spottune: transfer learning through adaptive fine-tuning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4805–4814, 2019.
  * Hataya et al. [2020] Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama.  Faster autoaugment: Learning augmentation strategies using backpropagation.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16_ , pages 1–16. Springer, 2020.
  * Hendrycks and Dietterich [2019] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  _Proceedings of the International Conference on Learning Representations_ , 2019.
  * Hendrycks et al. [2019] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song.  Using self-supervised learning can improve model robustness and uncertainty.  _Advances in neural information processing systems_ , 32, 2019.
  * Hendrycks et al. [2021a] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8340–8349, 2021a.
  * Hendrycks et al. [2021b] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15262–15271, 2021b.
  * Hutter et al. [2011] Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown.  Sequential model-based optimization for general algorithm configuration.  In _Learning and Intelligent Optimization: 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers 5_ , pages 507–523. Springer, 2011.
  * Ilharco et al. [2021] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  Openclip, 2021.  If you use this software, please cite it as below.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , pages 4904–4916. PMLR, 2021.
  * Jiang et al. [2019] Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao.  Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization.  _arXiv preprint arXiv:1911.03437_ , 2019.
  * Karani et al. [2021] Neerav Karani, Ertunc Erdil, Krishna Chaitanya, and Ender Konukoglu.  Test-time adaptable neural networks for robust medical image segmentation.  _Medical Image Analysis_ , 68:101907, 2021.
  * Kirichenko et al. [2022] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson.  Last layer re-training is sufficient for robustness to spurious correlations.  _arXiv preprint arXiv:2204.02937_ , 2022.
  * Kirkpatrick et al. [2017] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526, 2017.
  * Kirsch et al. [2019] Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber.  Improving generalization in meta reinforcement learning using learned objectives.  _arXiv preprint arXiv:1910.04098_ , 2019.
  * Koh et al. [2021] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al.  Wilds: A benchmark of in-the-wild distribution shifts.  In _International Conference on Machine Learning_ , pages 5637–5664. PMLR, 2021.
  * Krause et al. [2013] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , pages 554–561, 2013.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Kumar et al. [2022a] Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  In _International Conference on Learning Representations_ , 2022a.
  * Kumar et al. [2022b] Ananya Kumar, Ruoqi Shen, Sébastien Bubeck, and Suriya Gunasekar.  How to fine-tune vision models with sgd, 2022b.
  * Lee et al. [2019a] Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang.  Mixout: Effective regularization to finetune large-scale pretrained language models.  _arXiv preprint arXiv:1909.11299_ , 2019a.
  * Lee et al. [2019b] Jaejun Lee, Raphael Tang, and Jimmy Lin.  What would elsa do? freezing layers during transformer fine-tuning.  _arXiv preprint arXiv:1911.03090_ , 2019b.
  * Lee et al. [2022a] Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn.  Surgical fine-tuning improves adaptation to distribution shifts.  _arXiv preprint arXiv:2210.11466_ , 2022a.
  * Lee et al. [2022b] Yoonho Lee, Huaxiu Yao, and Chelsea Finn.  Diversify and disambiguate: Learning from underspecified data.  _arXiv preprint arXiv:2202.03418_ , 2022b.
  * Li et al. [2018a] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales.  Learning to generalize: Meta-learning for domain generalization.  In _Proceedings of the AAAI conference on artificial intelligence_ , 2018a.
  * Li et al. [2020] Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto.  Rethinking the hyperparameters for fine-tuning.  In _International Conference on Learning Representations_ , 2020.
  * Li et al. [2017] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar.  Hyperband: A novel bandit-based approach to hyperparameter optimization.  _The journal of machine learning research_ , 18(1):6765–6816, 2017.
  * Li et al. [2018b] Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Ben Recht, and Ameet Talwalkar.  Massively parallel hyperparameter tuning.  2018b.
  * Lim et al. [2019] Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim.  Fast autoaugment.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Liu et al. [2021a] Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn.  Just train twice: Improving group robustness without training group information.  In _International Conference on Machine Learning_ , pages 6781–6792. PMLR, 2021a.
  * Liu et al. [2018] Hanxiao Liu, Karen Simonyan, and Yiming Yang.  Darts: Differentiable architecture search.  _arXiv preprint arXiv:1806.09055_ , 2018.
  * Liu et al. [2021b] Yuhan Liu, Saurabh Agarwal, and Shivaram Venkataraman.  Autofreeze: Automatically freezing model blocks to accelerate fine-tuning.  _arXiv preprint arXiv:2102.01386_ , 2021b.
  * Lu et al. [2020] Shangyun Lu, Bradley Nott, Aaron Olson, Alberto Todeschini, Hossein Vahabi, Yair Carmon, and Ludwig Schmidt.  Harder or different? a closer look at distribution shift in dataset reproduction.  In _ICML Workshop on Uncertainty and Robustness in Deep Learning_ , page 15, 2020.
  * Metz et al. [2022] Luke Metz, James Harrison, C Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury, Naman Agrawal, Ben Poole, Igor Mordatch, Adam Roberts, et al.  Velo: Training versatile learned optimizers by scaling up.  _arXiv preprint arXiv:2211.09760_ , 2022.
  * Miller et al. [2021] John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt.  Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization.  In _International Conference on Machine Learning_ , pages 7721–7735. PMLR, 2021.
  * Mukhoti et al. [2023] Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puneet K Dokania.  Fine-tuning can cripple your foundation model; preserving features may be the solution.  _arXiv preprint arXiv:2308.13320_ , 2023.
  * Oh et al. [2020] Junhyuk Oh, Matteo Hessel, Wojciech M Czarnecki, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver.  Discovering reinforcement learning algorithms.  _Advances in Neural Information Processing Systems_ , 33:1060–1070, 2020.
  * Oquab et al. [2014] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic.  Learning and transferring mid-level image representations using convolutional neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1717–1724, 2014.
  * Qi et al. [2020] Qi Qi, Yi Xu, Rong Jin, Wotao Yin, and Tianbao Yang.  Attentional biased stochastic gradient for imbalanced classification.  _arXiv preprint arXiv:2012.06951_ , 2020.
  * Radford et al. [2021a] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021a.
  * Radford et al. [2021b] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021b.
  * Ramasesh et al. [2020] Vinay V Ramasesh, Ethan Dyer, and Maithra Raghu.  Anatomy of catastrophic forgetting: Hidden representations and task semantics.  _arXiv preprint arXiv:2007.07400_ , 2020.
  * Real et al. [2019] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le.  Regularized evolution for image classifier architecture search.  In _Proceedings of the aaai conference on artificial intelligence_ , pages 4780–4789, 2019.
  * Recht et al. [2018] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do cifar-10 classifiers generalize to cifar-10?  2018\.
  * Recht et al. [2019] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In _International Conference on Machine Learning_ , pages 5389–5400. PMLR, 2019.
  * Royer and Lampert [2020] Amélie Royer and Christoph Lampert.  A flexible selection scheme for minimum-effort transfer learning.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 2191–2200, 2020.
  * Sagawa et al. [2019] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang.  Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.  _arXiv preprint arXiv:1911.08731_ , 2019.
  * Sagawa et al. [2022] Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang.  Extending the WILDS benchmark for unsupervised adaptation.  In _International Conference on Learning Representations_ , 2022.
  * Salman et al. [2020] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry.  Do adversarially robust imagenet models transfer better?  _Advances in Neural Information Processing Systems_ , 33:3533–3545, 2020.
  * Sharif Razavian et al. [2014] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson.  Cnn features off-the-shelf: an astounding baseline for recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 806–813, 2014.
  * Shen et al. [2021] Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang-Ting Cheng.  Partial is better than all: Revisiting fine-tuning strategy for few-shot learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 9594–9602, 2021.
  * Taori et al. [2020] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt.  Measuring robustness to natural distribution shifts in image classification.  _Advances in Neural Information Processing Systems_ , 33:18583–18599, 2020.
  * Torralba et al. [2008] Antonio Torralba, Rob Fergus, and William T. Freeman.  80 million tiny images: A large data set for nonparametric object and scene recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 30(11):1958–1970, 2008.
  * Touvron et al. [2022] Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Hervé Jégou.  Three things everyone should know about vision transformers.  _arXiv preprint arXiv:2203.09795_ , 2022.
  * Tzeng et al. [2014] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell.  Deep domain confusion: Maximizing for domain invariance.  _arXiv preprint arXiv:1412.3474_ , 2014.
  * Veeling et al. [2018] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling.  Rotation equivariant cnns for digital pathology.  _arXiv preprint arXiv:1806.03962_ , 2018.
  * Wang et al. [2019] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  In _Advances in Neural Information Processing Systems_ , pages 10506–10518, 2019.
  * Wichrowska et al. [2017] Olga Wichrowska, Niru Maheswaranathan, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein.  Learned optimizers that scale and generalize.  2017\.
  * Wiles et al. [2021] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvise-Rebuffi, Ira Ktena, Taylan Cemgil, et al.  A fine-grained analysis on distribution shift.  _arXiv preprint arXiv:2110.11328_ , 2021.
  * Wortsman et al. [2022a] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In _International Conference on Machine Learning_ , pages 23965–23998. PMLR, 2022a.
  * Wortsman et al. [2022b] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.  Robust fine-tuning of zero-shot models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7959–7971, 2022b.
  * Xu et al. [2019] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong.  Pc-darts: Partial channel connections for memory-efficient architecture search.  _arXiv preprint arXiv:1907.05737_ , 2019.
  * Xuhong et al. [2018] LI Xuhong, Yves Grandvalet, and Franck Davoine.  Explicit inductive bias for transfer learning with convolutional networks.  In _International Conference on Machine Learning_ , pages 2825–2834. PMLR, 2018.
  * Yao et al. [2022] Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn.  Improving out-of-distribution robustness via selective augmentation.  In _International Conference on Machine Learning_ , pages 25407–25437. PMLR, 2022.
  * Yosinski et al. [2014] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  _Advances in neural information processing systems_ , 27, 2014.
  * Yu et al. [2018] Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine.  One-shot imitation from observing humans via domain-adaptive meta-learning.  _arXiv preprint arXiv:1802.01557_ , 2018.
  * Zhang et al. [2020] Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik.  Side-tuning: a baseline for network adaptation via additive side networks.  In _European Conference on Computer Vision_ , pages 698–714. Springer, 2020.
  * Zhang et al. [2021] Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn.  Adaptive risk minimization: Learning to adapt to domain shift.  _Advances in Neural Information Processing Systems_ , 34:23664–23678, 2021.
  * Zoph and Le [2016] Barret Zoph and Quoc V Le.  Neural architecture search with reinforcement learning.  _arXiv preprint arXiv:1611.01578_ , 2016.
  * Zoph et al. [2018] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le.  Learning transferable architectures for scalable image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8697–8710, 2018.

##  Appendix A Additional Experiments

###  A.1 ImageNet Distribution Shifts

####  A.1.1 Main Results

We provide detailed results for each ImageNet-derived distribution shift in
Tab. 8. AutoFT outperforms all baselines on 9 out of 10 evaluation settings.
FLYP, the prior state-of-the-art, has strong ID accuracy: $82.6\%$ and
$82.9\%$ without and with ensembling, whereas AutoFT has $81.8\%$ and
$82.4\%$. Despite its high ID accuracy, the model fine-tuned with FLYP has
substantially worse OOD accuracy than AutoFT: $60.2\%$ and $63.2\%$ without
and with ensembling, compared to $61.5\%$ and $64.3\%$ for AutoFT. We observe
a similar effect with all other points of comparison, indicating that these
methods for robust fine-tuning were “overfitting” to the ID distribution in a
way that AutoFT does not.

| Without Ensembling | With Ensembling
---|---|---
Methods | ID | Im-V2 | Im-R | Im-A | Sketch | ObjectNet | Avg. OOD | ID | Im-V2 | Im-R | Im-A | Sketch | ObjectNet | Avg. OOD
Zeroshot | 68.3 | 61.9 | 77.7 | 50.0 | 48.3 | 55.4 | 58.7 | 68.3 | 61.9 | 77.7 | 50.0 | 48.3 | 55.4 | 58.7
LP | 79.9 | 69.8 | 70.8 | 46.4 | 46.9 | 52.1 | 57.2 | 80.0 | 70.3 | 72.4 | 47.8 | 48.1 | 52.8 | 58.3
FT | 81.3 | 71.2 | 66.1 | 37.8 | 46.1 | 53.3 | 54.9 | 82.5 | 72.8 | 74.9 | 48.1 | 51.9 | 59.0 | 61.3
L2-SP | 81.7 | 71.8 | 70.0 | 42.5 | 48.5 | 56.2 | 57.8 | 82.2 | 72.9 | 75.1 | 48.6 | 51.4 | 58.9 | 61.4
LP-FT | 81.7 | 72.1 | 73.5 | 47.6 | 50.3 | 58.2 | 60.3 | 82.1 | 72.8 | 75.3 | 50.1 | 51.7 | 59.2 | 61.8
FLYP | 82.6 | 73.0 | 71.4 | 48.1 | 49.6 | 58.7 | 60.2 | 82.9 | 73.5 | 76.0 | 53.0 | 52.3 | 60.8 | 63.1
AutoFT | 81.8 | 73.1 | 72.4 | 48.8 | 49.8 | 63.5 | 61.5 | 82.4 | 73.6 | 76.4 | 53.1 | 52.6 | 65.6 | 64.3
Table 8:  AutoFT consistently outperforms all baselines on each ImageNet
distribution shift.

####  A.1.2 Few-Shot Classification

Figure 5 shows the average OOD accuracy on five ImageNet-derived distribution
shifts against ID accuracy on ImageNet for 4, 16, and 32 shot classification.
AutoFT outperforms baselines in effective robustness in all three settings. In
32-shot classification, AutoFT improves average OOD accuracy by $2.2\%$
compared to the leading baseline, FLYP.

Figure 5:  AutoFT enhances effective robustness over existing methods,
indicated by vertical distance. AutoFT results in better OOD accuracy at a
given ID accuracy across varying degrees of weight interpolation between the
fine-tuned and zero-shot models.

###  A.2 CIFAR Distribution Shifts

Method | CIFAR-10.1 | CIFAR-10.2
---|---|---
Zero-shot | 92.5 | 88.8
Fine-tuning | 95.9 | 91.3
AutoFT | 97.5 | 93.5
WiSE-FT (best $\alpha$) | 98.0 | 94.4
AutoFT (best $\alpha$) | 98.3 | 95.0

Table 9: AutoFT outperforms fine-tuning by $2.2\%$ on CIFAR-10.2 and by
$1.4\%$ on CIFAR-10.1, using only 100 samples from CIFAR-10-C. AutoFT
additionally outperforms WiSE-FT with weight ensembling.

AutoFT outperforms prior approaches on the subtle CIFAR-10.1 and CIFAR-10.2
distribution shifts.

###  A.3 Expressivity of Hyperparameter Space

iWildCam
---
Method | ID | OOD
AutoFT | 51.0 | 38.3
Layerwise AutoFT | 47.8 | 34.6
Table 10: AutoFT yields better ID and OOD performance than its variant with
per-layer learning rates, weight decays, and L1/L2 distances to pretrained
parameters, indicating that more expressive hyperparameter spaces overfit to
the validation set, degrading performance.

We investigate the effect of expressivity of the hyperparameter search space.
We compare AutoFT against a layerwise variant of AutoFT that learns per-layer
learning rates, weight decays, and L1/L2 norms and regularization terms in
Tab. 10. The layerwise variant of AutoFT results in worse ID and OOD
performance, suggesting that an overly expressive hyperparameter space
overfits to the validation set.

##  Appendix B Experimental Details

###  B.1 Toy Experiment Details

We provide further details for the toy experiment in Sec. 4.3 and Figure 3.
The prior distribution is a unit Gaussian, i.e. it has variance
$[{\color[rgb]{0.1,0.0499999999999999,0.96}1.0},{\color[rgb]{0.1,0.0499999999999999,0.96}1.0},\ldots]$.
Training data $D_{\text{tr}}$ is drawn from a distribution with variance
$[{\color[rgb]{0.72,0,0}10^{-3}},{\color[rgb]{0.72,0,0}10^{-3}},{\color[rgb]{0.72,0,0}10^{-3}},{\color[rgb]{0.72,0,0}10^{-3}},{\color[rgb]{0.1,0.0499999999999999,0.96}1.0},{\color[rgb]{0.1,0.0499999999999999,0.96}1.0},\ldots]$,
while the validation data $D_{\text{val}}$ is drawn from a distribution with
variance
$[{\color[rgb]{0.72,0,0}10^{-3}},{\color[rgb]{0.72,0,0}10^{-3}},{\color[rgb]{0.72,0,0}10^{-3}},{\color[rgb]{0.1,0.0499999999999999,0.96}1.0},{\color[rgb]{0.72,0,0}10^{-3}},{\color[rgb]{0.1,0.0499999999999999,0.96}1.0},\ldots]$.
We fit the parameters of a Gaussian distribution $q_{\theta}$ to the training
data using a variational Bayes objective by minimizing the following loss:

$\operatorname*{arg\,min}_{\theta}D_{KL}(q_{\theta}\;||\;p)+\sum_{i=1}^{10}w_{i}\text{NLL}_{i}(q_{\theta},\mathcal{D}).$
(2)

###  B.2 Datasets

Below, we summarize the datasets we use for evaluation, including the fine-
tuning dataset (ID), the validation dataset for hyperparameter optimization,
and the test OOD datasets.

  * •

CIFAR-10 [49] contains 60,000 images across 10 classes. We use CIFAR-10 for
fine-tuning, 100 examples from CIFAR-10-C for validation, and the CIFAR-10.1
[75, 84] and CIFAR-10.2 [64] as OOD test sets.

  * •

ImageNet [21] contains over a million images in 1000 categories. We use
ImageNet as our ID distribution, 15000 examples from ImageNet-C for
validation, and five ImageNet variations for the OOD datasets following prior
works [71, 92, 50, 30]: ImageNet-V2 [76], ImageNet-R [37], ImageNet-A [38],
ImageNet-Sketch [88], and ObjectNet [7].

  * •

WILDS-iWildCam [9, 47, 79] is an animal camera trap image classification
dataset where the label $y$ is one of 182 animal species. The ID and OOD
datasets consist of photos from disjoint sets of camera traps, making the
images differ in camera specifications and attributes such as background and
lighting. We use the official splits from Koh et al. [47] and use the ID train
set for fine-tuning, the OOD validation set for hyperparameter optimization,
and the OOD test set for evaluation.

  * •

WILDS-FMoW [16, 47, 79] contains remote sensing imagery from satellites. Each
image is to be classified into one among 62 categories, including labels like
“impoverished settlement” and “hospital.” The ID and OOD datasets differ in
year of acquisition and geographic location. We use the official splits from
Koh et al. [47] and use the ID train set for fine-tuning, the OOD validation
set for hyperparameter optimization, and the OOD test set for evaluation.

In all of the transfer learning datasets described below, we use a subset of
the ID validation set for hyperparameter optimization. In other words, we do
not use an external “OOD” set.

  * •

Caltech101 [25] contains images of objects from 101 different categories,
including “dragonfly,” “grand piano,” and “saxophone.”

  * •

StanfordCars [48] features a collection of car images categorized by model,
make, and year, where the task is to classify them into one of 196 types, such
as “Ford Mustang Convertible 1967” or “Toyota Prius Hatchback 2009.”

  * •

Flowers102 [87] consists of flower images from the UK, with the objective of
classifying each image into one of 102 species, such as “oxeye daisy” or
“hibiscus.”

  * •

PatchCamelyon [87] provides digital pathology images for binary
classification, with the goal of identifying metastatic tumor tissues.

  * •

Rendered SST2 [72] is a dataset for optical character recognition, where the
task is to classify text sentiment as “positive” or “negative.”

###  B.3 Training Details

####  B.3.1 Baselines

We closely follow the training details in Goyal et al. [30]. For all datasets
excluding ImageNet, we use a batch size of 256 and conduct a hyperparameter
sweep across five learning rates $\\{10^{-6},10^{-5},\dots,10^{-2}\\}$ and
five weight decay values in the range $\\{0.0,0.1,\dots,0.4\\}$. On ImageNet,
we use a larger batch size of 512 and perform a hyperparameter sweep three
learning rates $\\{10^{-6},10^{-5},10^{-4}\\}$ and two weight decays
$\\{0,0.1\\}$. For L2-SP, we tune the weight of the regularization term
$\lambda\in\\{10^{-4},10^{-3},10^{-2},10^{-1}\\}$. We select baseline
hyperparameters based on ID validation performance. For datasets without a
standard validation set, we split the training data into an 80:20 ratio to
create one.

In the $k$ few-shot setting with $k\in\\{4,16,32\\}$, we select $k$ training
and $k$ validation examples from each class. To account for variance from the
small training and validation sets, we average over 50 runs.

####  B.3.2 AutoFT

Hyperparameter search space. As described in Sec. 4, AutoFT learns weights for
nine different losses on a log-uniform range $[10^{-4},10]$. AutoFT
additionally searches for learning rate in the log-uniform range
$[10^{-2}\cdot\eta^{*},10^{2}\cdot\eta^{*}]$, where $\eta^{*}$ is the
conventional learning rate used in prior works on fine-tuning [92, 50, 30],
and weight decay values in the log-uniform range $[0.0,1.0]$.

Optuna settings. Below, we summarize the number of inner-loop gradient steps,
Optuna trials, and validation set size for each dataset. All of these values
were selected based on performance on an ID validation set:

Dataset | Inner Steps | Trials | $N_{\textrm{val}}$
---|---|---|---
iWildCam | 10 | 500 | 1000
FMoW | 10 | 500 | 1000
CIFAR-10 | 10 | 100 | 100
Flowers | 50 | 500 | 500
Cars | 50 | 500 | 1000
ImageNet | 100 | 500 | 1000

Few-shot classification. For the $k$ few-shot classification setting, where
$k\in\\{4,16,32\\}$, we use a $k$-shot validation set for hyperparameter
optimization. On all $k$-shot SST2 and PatchCamelyon experiments, we run with
10 inner-loop gradient steps and 50 Optuna trials. We run all ImageNet few-
shot experiments with 100 Optuna trials and 5, 20, and 50 inner steps for
4-shot, 16-shot, and 32-shot classification, respectively. Due to increased
variance from the very small training and validation sets in few-shot binary
classification (e.g., SST2 and PatchCamelyon), we select the single best set
of hyperparameters across 5 separate AutoFT runs, based on performance on the
$k$-shot validation set.

Transfer learning. We note that the transfer learning experiments on
Flowers102 and Stanford Cars do not use OOD data for hyperparameter
optimization; here, AutoFT optimizes hyperparameters on a held-out ID
validation set.

WILDS-iWildCam and WILDS-FMoW state-of-the-art (SoTA). For the iWildCam and
FMoW SoTA results in Tab. 1, we fine-tune the ViT-L/14@336px model with loss
weights learned on the smaller ViT-B/16 backbone with AutoFT. Applying AutoFT
directly to the ViT-L/14@336px model may improve performance further, although
at the cost of more compute.