# One For All & All For One: Bypassing Hyperparameter Tuning with Model
Averaging For Cross-Lingual Transfer

Fabian David Schmidt1, Ivan Vulić2, Goran Glavaš1
1 Center For Artificial Intelligence and Data Science, University of Würzburg,
Germany
2 Language Technology Lab, University of Cambridge, UK
{fabian.schmidt<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Multilingual language models enable zero-shot cross-lingual transfer (zs-xlt):
fine-tuned on sizable source-language task data, they perform the task in
target languages without labeled instances. The effectiveness of zs-xlt hinges
on the linguistic proximity between languages and the amount of pretraining
data for a language. Because of this, model selection based on source-language
validation is unreliable: it picks model snapshots with suboptimal target-
language performance. As a remedy, some work optimizes zs-xlt by extensively
tuning hyperparameters: the follow-up work then routinely struggles to
replicate the original results. Other work searches over narrower
hyperparameter grids, reporting substantially lower performance. In this work,
we therefore propose an unsupervised evaluation protocol for zs-xlt that
decouples performance maximization from hyperparameter tuning. As a robust and
more transparent alternative to extensive hyperparameter tuning, we propose to
accumulatively average snapshots from different runs into a single model. We
run broad zs-xlt experiments on both higher-level semantic tasks (NLI,
extractive QA) and a lower-level token classification task (NER) and find that
conventional model selection based on source-language validation quickly
plateaus to suboptimal zs-xlt performance. On the other hand, our accumulative
run-by-run averaging of models trained with different hyperparameters boosts
zs-xlt performance and closely correlates with “oracle” zs-xlt, i.e., model
selection based on target-language validation performance.

##  1 Introduction and Motivation

Massively multilingual transformers (MMTs) like XLM-{R,V} (Conneau et al.,
2020; Liang et al., 2023) or mT5 (Xue et al., 2021) are pretrained via
language modeling on vast corpora encompassing 100+ languages. MMT fine-tuned
on labeled task data in a source language can transfer cross-lingually zero-
shot, i.e. without further annotations, to target languages (Hu et al., 2020;
Lauscher et al., 2020). However, pretraining corpora size and linguistic
distance between the source and target language dictate the quality of xlt
(Lauscher et al., 2020). This is why model selection based on source-language
validation data unreliably correlates with zs-xlt and selects checkpoints that
yield suboptimal target-language performance (Keung et al., 2020). Worse yet,
there is no “best practice” for replicating zs-xlt results of prior work. Some
works, as our results suggest (cf. §4), (1) exhaust extraordinarily large
hyperparameter grids and (2) monitor target-language performance for the best
transfer (i.e., violating “true” zs-xlt) to outperform baselines (Conneau et
al., 2020; Wei et al., 2021). Other works rerun baselines with little to no
hyperparameter tuning (Hu et al., 2020; Wu and Dredze, 2020): the re-
evaluation then often trails original results by non-negligible margins.111For
instance, when evaluating XLM-V${}_{\text{base}}$, Liang et al. (2023) have
been unable to reproduce the original results of XLM-R${}_{\text{base}}$ on
the XNLI benchmark (Conneau et al., 2020). As a remedy, Keung et al. (2020)
propose to evaluate zs-xlt on the snapshot that generalizes best to validation
data in the target language (“oracle” zs-xlt): as such, oracle zs-xlt
stabilizes evaluation and denotes ideal transfer performance. Nonetheless,
oracle zs-xlt overstates the performance of true zs-xlt, for which no target-
language instances are available (Schmidt et al., 2023). If they are, target-
language annotations are always better levered for training than for
validation (Schmidt et al., 2022).

This calls for an evaluation protocol that (1) maximizes “true” zs-xlt results
and (2) makes them easily reproducible, regardless of the extent of
hyperparameter tuning. In this work, we find that model averaging fulfills
both criteria. Weights averaging has proven effective in, e.g., MT (Vaswani et
al., 2017) and recently NLU (Wang et al., 2022; Schmidt et al., 2023). Schmidt
et al. (2023) enable model averaging for sizable gains in xlt. They first
fine-tune an MMT on labeled source-language data and then re-train models
(i.e., more runs) by copying and freezing the task head of the initially fine-
tuned model: this aligns snapshots and enables weight averaging across
runs.222Fine-tuning models with different randomly initialized task heads
otherwise yields sets of incompatible weights, hindering meaningful model
averaging.

Contributions. In this work, we propose an evaluation protocol that decouples
maximizing zs-xlt performance from hyperparameter tuning. The key idea is to
accumulatively average snapshots of runs with different hyperparameters: this
improves performance over model selection based on source-language validation
performance. We run exhaustive experiments on higher-level (NLI, extractive
QA) and lower-level (NER) NLU tasks on a broad grid of hyperparameters and
show, examining the cross-section of all runs, that model selection based on
source-language validation almost exclusively picks snapshots suboptimal for
zs-xlt. We also confirm that conventional hyperparameter tuning on source-
language validation prematurely settles for models that maximize source-
language performance at the expense of zs-xlt. Crucially, we show that
accumulative model averaging performs on par or better than the best snapshot
picked by source-language validation already from the second (i.e. first
averaged-in) run and then consistently improves zs-xlt with more runs. We
additionally show that this accumulative model averaging closely correlates
with oracle zs-xlt without requiring any source- or target-language validation
data to maximize transfer performance.

##  2 Accumulative Run Averaging

Prior work conducts model selection for zs-xlt by extensive hyperparameter
tuning using either source- or target-language validation data. Whereas the
latter violates true zs-xlt Schmidt et al. (2022), the former overfits to
source-language performance Keung et al. (2020). The recent success of
snapshot averaging in xlt (Schmidt et al., 2023) motivates our research
question: can (accumulative) averaging of models trained during hyperparameter
search outperform – with fewer overall training runs – the zs-xlt performance
of the “optimal” model selected based on source-language validation
performance?

We benchmark model selection based on source-language validation against
accumulative model averaging as follows. We iteratively sample models (i.e.,
runs) $\\{\\{\theta_{1},\ldots,\theta_{r}\\}\;|\;1\leq r\leq 10\\}$ with
different hyperparameters (i.e., pairs of learning rates and batch sizes) from
the pool of runs containing $N$ runs per hyperparameter configuration (cf.
Appendix §A.2). We repeat this procedure $10$ times and report mean zs-xlt
performance. Standard model selection picks the model
$\\{\operatorname*{arg\,max}_{i}{{\text{Val}(\theta_{i})}}\;|\;1\leq i\leq
r\\}$ at run $r$ that maximizes source- (target-) language validation,
capturing the “true” (“oracle”) zs-xlt performance. “Accumulative averaging”,
in contrast, naively averages (i.e. without any supervision) all models of $r$
runs to a single model
$\frac{1}{r}\sum_{j=1}^{r}{\theta_{j}}=\bar{\theta}_{r}$.

##  3 Experimental Setup

Tasks and Languages. We select for our evaluation two higher-level semantic
tasks (NLI and and extractive QA) and one lower-level structured prediction
task (NER). For each task, we fine-tune the MMT on the provided English
training splits.333Train portion of MNLI (Williams et al., 2018), the enclosed
3,696 English training instances of TyDiQA-GoldP for QA, and the English
training portion of WikiANN for NER.

Natural Language Inference (NLI). We evaluate NLI on XNLI (Conneau et al.,
2018) and IndicXNLI (Aggarwal et al., 2022) which together cover 25
typologically diverse languages.

Extractive QA (TyDiQA-GoldP). TyDiQA-GoldP comprises questions that are
answered by a span of text in the provided gold passage and covers 9 diverse
languages (Clark et al., 2020).

Named Entity Recognition (NER). We evaluate NER on 25 languages from WikiANN
(Pan et al., 2017), 10 languages from MasakhaNER (Adelani et al., 2021), and 9
languages from MasakhaNER 2.0 (Adelani et al., 2022).

Training Details. We train XLM-R${}_{\text{large}}$ (Conneau et al., 2020) for
10 epochs with AdamW (Loshchilov and Hutter, 2019), weight decay of $0.01$,
gradient norm clipping to $1.0$, and a LR schedule of 10% linear warm-up and
decay.444The training data of TyDiQA-GoldP consists of merely 3,696 instances;
we thus fine-tune longer, for 40 epochs. We save 10 snapshots per model, one
at every $10\%$ of total training steps. The maximum sequence length is 128
tokens for NLI and NER and 384 with a stride of 128 for TyDiQA-GoldP.

Hyperparameter Grids. We simulate conventional hyperparameter grid search over
a broad set of 21 configurations, pairing seven learning rates
$l\in\\{0.1,0.5,1,1.5,2,2.5,3\\}e^{{-}5}$ with three batch sizes
$b\in\\{16,32,64\\}$. The grid is deliberately kept wide and same for all
tasks to not reflect any prior knowledge on task-specific “good values”.555Our
full results in Table 3 indicate that for each task we obtain maximal (oracle)
zs-xlt performance with a different, task-specific hyperparameter
configurations. We retrain MMT for each pair $(l,b)$ three times with
different random seeds to account for variances over individual runs.

Model Variants. We evaluate four model variants: $v\in$ {last, src-dev, ca,
trg-dev}. last is simply the final snapshot of a training run. src-dev is the
snapshot that maximizes source-language validation performance (Hu et al.,
2020). ca averages all snapshots of a run to a single model and, according to
Schmidt et al. (2023), outperforms last and src-dev. trg-dev breaches “true”
zs-xlt and picks the snapshot that performs best on the target-language
validation data (Keung et al., 2020): as such, it generally represents an
upper-bound of single-run zs-xlt performance.666Irrespective of tasks and
language, labeled instances in the target-language bring larger gains if used
for training rather than for model selection (Schmidt et al., 2022).

##  4 Results and Discussion

NLI TyDiQA-GoldP NER Max. src-dev Accumulative Averaging Max. src-dev
Accumulative Averaging Max. src-dev Accumulative Averaging $\bm{r}$ last s-dev
ca last s-dev ca last s-dev ca last s-dev ca last s-dev ca last s-dev ca $1$
$76.5_{0.6}$ $76.5_{0.8}$ $77.3_{0.4}$ $76.5_{0.6}$ $76.5_{0.8}$ $77.3_{0.4}$
$71.9_{0.4}$ $71.9_{0.7}$ $73.6_{1.9}$ $71.9_{0.4}$ $71.9_{0.7}$ $73.6_{1.9}$
$\mathbf{40.8_{2.7}}$ $41.1_{3.1}$ $\mathbf{44.6_{2.1}}$ $40.8_{2.7}$
$41.1_{3.0}$ $44.6_{2.1}$ $2$ $77.2_{0.3}$ $77.5_{0.4}$ $\mathbf{77.6_{0.2}}$
$77.6_{0.3}$ $77.8_{0.4}$ $78.0_{0.2}$ $71.9_{0.6}$ $71.6_{0.6}$ $73.3_{2.0}$
$73.4_{1.2}$ $73.3_{1.1}$ $72.9_{2.8}$ $39.3_{2.1}$ $39.3_{2.1}$ $43.5_{1.1}$
$43.2_{2.2}$ $43.2_{2.2}$ $45.6_{1.5}$ $3$ $77.2_{0.3}$ $77.5_{0.4}$
$77.6_{0.2}$ $77.8_{0.3}$ $77.9_{0.4}$ $78.1_{0.2}$ $72.1_{0.8}$ $71.8_{0.8}$
$74.1_{1.0}$ $74.1_{0.7}$ $74.2_{0.7}$ $73.8_{1.2}$ $39.3_{1.2}$ $39.5_{1.7}$
$44.0_{1.1}$ $45.0_{1.7}$ $45.1_{1.8}$ $47.3_{1.3}$ $4$ $77.2_{0.4}$
$77.5_{0.4}$ $77.5_{0.2}$ $77.7_{0.3}$ $77.9_{0.4}$ $78.1_{0.3}$ $72.5_{0.8}$
$\mathbf{72.0_{0.9}}$ $73.9_{0.6}$ $74.5_{0.6}$ $74.1_{0.4}$ $74.1_{0.9}$
$40.2_{2.0}$ $40.8_{2.2}$ $44.5_{1.5}$ $45.0_{1.7}$ $45.3_{1.8}$ $47.2_{1.4}$
$5$ $77.3_{0.3}$ $\mathbf{77.6_{0.3}}$ $77.5_{0.2}$ $77.9_{0.2}$ $78.0_{0.2}$
$78.1_{0.1}$ $\mathbf{72.6_{0.8}}$ $72.0_{0.9}$ $73.8_{0.8}$
$\mathbf{74.7_{0.7}}$ $\mathbf{74.4_{0.6}}$ $74.2_{0.8}$ $40.3_{2.0}$
$\mathbf{41.2_{2.3}}$ $43.9_{1.9}$ $45.3_{1.7}$ $45.5_{1.7}$ $47.5_{1.4}$ $6$
$77.3_{0.3}$ $77.6_{0.1}$ $77.5_{0.2}$ $77.9_{0.1}$ $78.0_{0.2}$ $78.1_{0.1}$
$72.6_{0.8}$ $72.0_{0.9}$ $74.2_{0.5}$ $74.7_{0.7}$ $74.4_{0.5}$ $74.2_{0.7}$
$40.3_{2.0}$ $41.2_{2.3}$ $43.9_{1.9}$ $45.7_{1.4}$ $45.9_{1.4}$ $47.9_{1.2}$
$7$ $77.3_{0.3}$ $77.6_{0.1}$ $77.5_{0.2}$ $77.9_{0.2}$ $78.1_{0.2}$
$78.2_{0.2}$ $72.3_{0.9}$ $71.7_{0.7}$ $\mathbf{74.3_{0.3}}$ $74.6_{0.7}$
$74.3_{0.6}$ $74.2_{0.5}$ $40.0_{2.1}$ $40.6_{2.3}$ $44.1_{2.0}$ $46.0_{1.3}$
$46.1_{1.3}$ $48.1_{1.1}$ $8$ $77.3_{0.3}$ $77.6_{0.2}$ $77.5_{0.2}$
$\mathbf{78.0_{0.2}}$ $\mathbf{78.2_{0.1}}$ $\mathbf{\underline{78.3_{0.2}}}$
$72.1_{0.9}$ $71.7_{0.7}$ $74.2_{0.5}$ $74.6_{0.7}$ $74.3_{0.5}$
$\mathbf{74.3_{0.5}}$ $40.0_{2.1}$ $40.6_{2.3}$ $44.6_{1.7}$ $46.0_{1.1}$
$46.1_{1.2}$ $48.2_{1.0}$ $9$ $\mathbf{77.4}_{0.2}$ $77.6_{0.2}$ $77.6_{0.2}$
$78.0_{0.1}$ $78.1_{0.1}$ $78.3_{0.2}$ $72.0_{1.1}$ $71.7_{0.7}$ $74.2_{0.5}$
$74.6_{0.5}$ $74.4_{0.4}$ $74.2_{0.4}$ $39.6_{2.3}$ $39.9_{2.4}$ $44.3_{1.8}$
$46.0_{0.6}$ $46.1_{0.7}$ $48.3_{0.7}$ $10$ $77.3_{0.2}$ $77.6_{0.2}$
$77.6_{0.2}$ $78.0_{0.1}$ $78.2_{0.1}$ $78.3_{0.1}$ $72.0_{1.1}$ $71.7_{0.7}$
$74.2_{0.5}$ $74.6_{0.6}$ $74.4_{0.4}$ $74.2_{0.6}$ $39.6_{2.3}$ $39.9_{2.4}$
$44.4_{1.7}$ $\mathbf{46.1_{0.5}}$ $\mathbf{46.2_{0.6}}$ $\mathbf{48.4_{0.5}}$

Table 1:  $\\{\\{\theta_{1},\ldots,\theta_{r}\\}\;|\;1\leq r\leq 10\\}$ models
sampled for variants $v\in$ {last, src-dev, ca} from Table 3 (cf. §3). “Max.
src-dev” picks the run
$\\{\operatorname*{arg\,max}_{i}{{\text{SrcVal}(\theta_{i}^{v})}}\;|\;1\leq
i\leq r\\}$. “Accumulative averaging” simply averages all runs
$\frac{1}{r}\sum_{j=1}^{r}{\theta_{j}^{v}}$. Metrics: accuracy for NLI,
span-$F_{1}$ for TyDiQA and token-level $F_{1}$ for NER. Subscripts denote
std. deviation. Colored averaging outperforms +0.2 or more or performs
$\pm$0.1 of the best Max. src-dev model.

NLI TyDiQA-GoldP NER Max. dev Acc. Avg. Max. Dev Acc. Avg. Max. dev Acc. Avg.
src trg src trg src trg $\bm{r}$ dev dev ca soup dev dev ca soup dev dev ca
soup 1 $77.3$ $77.0$ $77.3$ $76.8$ $71.9$ $72.8$ $73.6$ $73.7$ $41.1$ $46.5$
$44.6$ $42.3$ 3 $77.5$ $77.7$ $78.1$ $77.6$ $71.8$ $73.5$ $73.8$ $73.8$ $39.5$
$49.2$ $47.3$ $42.1$ 5 $\mathbf{77.6}$ $77.9$ $78.1$ $77.6$ $\mathbf{72.0}$
$73.4$ $\mathbf{74.2}$ $\mathbf{74.3}$ $\mathbf{41.2}$ $49.7$ $47.5$
$\mathbf{42.8}$ 7 $77.6$ $78.2$ $78.2$ $\mathbf{77.8}$ $71.7$ $73.7$ $74.2$
$73.9$ $40.6$ $\mathbf{49.9}$ $48.1$ $42.8$ 10 $77.6$ $\mathbf{78.4}$
$\mathbf{78.3}$ $77.7$ $71.7$ $\mathbf{73.9}$ $74.2$ $73.8$ $39.9$ $49.9$
$\mathbf{48.4}$ $42.8$

Table 2:  “Max. trg-dev” selects the run
$\\{\operatorname*{arg\,max}_{i}{{\frac{1}{|T|}\text{Val}_{T}(\theta_{i})}}\;|\;1\leq
i\leq r\\}$, where $T$ is the set of target languages. soup averages the five
checkpoints (from all available runs) that “Max. src-dev”. For other details,
see Table 1.

Single-Run Performance. The full zs-xlt results by hyperparameters are
presented in Appendix §A.2 (cf. Table 3). We observe that optimal zs-xlt of
single runs depends on all axes of analysis: task, hyperparameters, and model
variant. While last and src-dev generally perform well, their zs-xlt
performance fluctuates substantially across hyperparameter configurations, in
line with (Keung et al., 2020; Schmidt et al., 2023). ca is a strong and
robust baseline that often outperforms last and src-dev by notable margins on
TyDiQA and NER. In the context of a single run, ca performs especially well
with suboptimal hyperparameters, even sometimes outperforming trg-dev. We also
confirm that ca remedies variation in zs-xlt both within and across
hyperparameters (Schmidt et al., 2023). Table 3 (cf. Appendix §A.2) further
highlights the notable gap in zs-xlt performance between the best-performing
hyperparameter configurations and those selected based on source-language
validation. Only the “oracle” model selection based on target-language
validation reliably correlates with the actual best (test) zs-xlt performance.

Run-by-Run Analysis. Table 1 compares zs-xlt performance run-for-run of all
variants for model selection based on source-language validation (Max. src-
dev) against our accumulative averaging of randomly sampled runs with
different hyperparameters (cf. §2). On NLI, picking a single model on source-
language validation only improves zs-xlt when moving from having one to having
two models (i.e., between first two rows of Table 1) and stagnates when having
more models to choose from. With more runs, source-language validation may
even prefer models that are worse at zs-xlt on TyDiQA and NER. Conventional
model selection thus maximizes source-language performance at the expense of
zs-xlt. Across the board, accumulative averaging already matches or surpasses
Max. src-dev (with any number of models) using merely two or three runs.
Moreover, accumulative averaging consistently outperforms the overall best
single-run model chosen from $3+$ runs (highlighted in green), irrespective of
the task. On all tasks, accumulative averaging stabilizes zs-xlt and reduces
performance variance vis-a-vis Max. src-dev counterparts.

Accumulatively averaging within-run snapshots (ca) outperforms last and src-
dev slightly on NLI and materially on NER. For NER, zs-xlt from WikiANN to
MasakhaNER (2.0) also represents a domain transfer (from Wiki to news), in
which ca yields tremendous gains. In-domain (i.e., test on WikiANN), ca
generally performs on par with last and src-dev. The same is not true for QA,
where ca performs slightly worse: we ascribe this to averaging of
“unconverged” snapshots, owing to the small TyDiQA training set (merely 3,696
instances), especially from runs with smaller learning rates and larger
batches (cf. Table 3).

Further Analyses. Table 2 extends the run-by-run analysis to trg-dev and
“model soups” (soup) to illustrate why accumulative model averaging
outperforms model selection based on source-language validation. Rather than
selecting a single snapshot, soup averages the five snapshots (among all
available runs) with best source-language validation performance (Wortsman et
al., 2022).

Compared to (oracle) trg-dev, accumulatively averaging runs performs on par on
NLI, slightly better on TyDiQA, and somewhat worse on NER. trg-dev selects
language-specific snapshots, thereby tailoring zs-xlt to each target language
and remedying for the varying performance of Max. src-dev in zs-xlt to many
target languages. Such a variation has been shown to be particularly
pronounced in zs-xlt on token-level tasks like NER or POS (Schmidt et al.,
2023). On TyDiQA, we believe that accumulative averaging (slightly) better
stabilizes the transfer from a small training set ($3.7$K instances). soups
however perform notably worse than both trg-dev and accumulative averaging on
NLI and NER. soups lack the beneficial diversity of different runs, as the
best snapshots often come from the same “good” run.777Extending soup to
average the top-$10$ best snapshots does not improve performance. Anecdotal
evidence further exemplifies why source-language validation is inapt for zs-
xlt. One of $63$ src-dev models replicates XNLI results of Conneau et al.
(2020), vastly exceeding all other runs (c.$\Delta{+}1.0$). This “miraculous”
run though merely ranks 3rd according to source-language validation
performance.

The above suggests that even the more sophisticated hyperparameter tuning
strategies (e.g., Bayesian optimization) are unlikely to improve zs-xlt
without target-language validation. On the other hand, accumulative averaging
improves zs-xlt threefold: (1) Unlike model selection, it does not plateau in
zs-xlt on suboptimal single runs that maximize source-language performance;
(2) trg-dev showcases that accumulative averaging ingests further runs with
snapshots that perform well on zs-xlt; (3) Model averaging irons out
idiosyncratic noise of individual runs, leading to better performance. This
renders accumulative averaging a robust (i.e., replicable results) and fair
(i.e. true zero-shot) evaluation protocol for zs-xlt.

##  5 Conclusion

Inconsistent hyperparameter tuning and model selection protocols exacerbate
replicating previous results on zs-xlt. In this focused study, we devise a zs-
xlt evaluation protocol that addresses previous shortcomings and feeds two
birds with one scone. We show that accumulatively averaging snapshots – rather
than selecting models based on source-language validation performance – both
improves and stabilizes zs-xlt. Conventional model selection strategies
prematurely settle for models that maximize source-language validation
performance and discard runs that generalize better in zs-xlt. Accumulative
model averaging both incorporates snapshots that transfer well and irons out
models that perform badly. We find that model averaging correlates closely
with “oracle” zs-xlt, which assumes models selection on target-language
validation instances. We hope future work adopts model averaging to promote
fair and reproducible zs-xlt that puts models on equal footing.

## Limitations

Additional factors must be taken into consideration, even though we aspire to
evaluate zs-xlt on all levels of transparency (i.e., variants and strategies)
across a varied set of downstream tasks on broad hyperparameter grids. Neither
model selection on source-language validation data nor accumulative averaging
may benefit zs-xlt on certain tasks, as Schmidt et al. (2023), e.g., do not
find that any variant other than trg-dev yields gain over last on part-of-
speech tagging. The underlying cause remains unclear. For instance, the gains
on zs-xlt stemming from model selection or accumulative averaging likely
depend on the type of distributional shift from the source-language training
data and the target-language instances to transfer to (cf. §4; e.g. dynamics
of variants in zs-xlt for NER). accumulative averaging nevertheless remains a
robust evaluation protocol, as zs-xlt performance is not expected to
deteriorate via-à-vis other “fair” strategies (e.g., max. src-dev). In
addition, there may exist a subset of pairs of learning rates and batch sizes
that jointly maximize source- and target-language performance. However, as our
results suggest (§4), runs on such hyperparameters likely are
indistinguishable from those that exclusively perform just as well on the
source-language validation set.

## Acknowledgments

We thank the state of Baden-Württemberg for its support through access to the
bwHPC. Ivan Vulić is supported by a personal Royal Society University Research
Fellowship ‘Inclusive and Sustainable Language Technology for a Truly
Multilingual World’ (no 221137; 2022–).

## References

  * Adelani et al. (2022) David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, Fatoumata Kabore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh-Buabeng, Victoire Memdjokam Koagne, Allahsera Auguste Tapo, Tebogo Macucwa, Vukosi Marivate, Mboning Tchiaze Elvis, Tajuddeen Gwadabe, Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba-Nabende, Neo Lerato Mokono, Ignatius Ezeani, Chiamaka Chukwuneke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme, Idris Abdulmumin, Odunayo Ogundepo, Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow. 2022.  MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 4488–4508, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Adelani et al. (2021) David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Salomey Osei. 2021.  MasakhaNER: Named entity recognition for African languages.  _Transactions of the Association for Computational Linguistics_ , 9:1116–1131.
  * Aggarwal et al. (2022) Divyanshu Aggarwal, Vivek Gupta, and Anoop Kunchukuttan. 2022.  Indicxnli: Evaluating multilingual inference for indian languages.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.  _Transactions of the Association for Computational Linguistics_ , 8:454–470.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  Xnli: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _International Conference on Machine Learning_ , pages 4411–4421. PMLR.
  * Keung et al. (2020) Phillip Keung, Yichao Lu, Julian Salazar, and Vikas Bhardwaj. 2020.  Don’t use English dev: On the zero-shot cross-lingual evaluation of contextual embeddings.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 549–554, Online. Association for Computational Linguistics.
  * Lauscher et al. (2020) Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020.  From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4483–4499, Online. Association for Computational Linguistics.
  * Liang et al. (2023) Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023.  XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models.  _arXiv e-prints_ , page arXiv:2301.10472.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Pan et al. (2017) Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017.  Cross-lingual name tagging and linking for 282 languages.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.
  * Schmidt et al. (2022) Fabian David Schmidt, Ivan Vulić, and Goran Glavaš. 2022.  Don’t stop fine-tuning: On training regimes for few-shot cross-lingual transfer with multilingual language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10725–10742, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Schmidt et al. (2023) Fabian David Schmidt, Ivan Vulić, and Goran Glavaš. 2023.  Free lunch: Robust cross-lingual transfer via model checkpoint averaging.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Wang et al. (2022) Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. 2022.  AdaMix: Mixture-of-adaptations for parameter-efficient model tuning.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5744–5760, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wei et al. (2021) Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing, Heng Yu, and Weihua Luo. 2021\.  On learning universal representations across languages.  In _International Conference on Learning Representations_.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wortsman et al. (2022) Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. 2022.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 23965–23998. PMLR.
  * Wu and Dredze (2020) Shijie Wu and Mark Dredze. 2020.  Do explicit alignments robustly improve multilingual encoders? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4471–4482, Online. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Reproduction Details

Code. Our code is available at: https://github.com/fdschmidt93/ofa-xlt

Model architectures. All models use the AutoModelFor{SequenceClassification,
TokenClassification, QuestionAnswering} of xlm-roberta-large for the
corresponding task from the transformers library (Wolf et al., 2020).

Compute Requirements. We execute all experiments on a single V100 with 32GB
VRAM. We estimate that we require total compute time of c.1,050 hours over all
fine-tuning iterations and evaluations. We arrive at this budget as follows.
We on average train models on NLI for about 11.5 hours, on TyDiQA-GoldP for
roughly 1.5 hours, and on NER for an estimated 3 hours. We therefore execute
63 training runs (21 hyperparameter configurations ran on for 3 seeds, cf. §3)
for 16 hours for a total of c.1K GPU hours. We loosely estimate that
accumulative averaging adds another 50 hours of runtime for evaluation.

Model Averaging. We follow Schmidt et al. (2023) to enabling accumulative
averaging of checkpoints for NLI and TyDiQA-GoldP. For these tasks, we
initially fine-tune XLM-R${}_{\text{large}}$ with a batch size of 32 and a
learning rate of $2e^{{-}5}$. For NER, we find that merely randomly
initializing the tasks heads across all runs with the same head slightly
improves performance ( $\Delta+1.0$) of all variants in single-run and
accumulative averaging. We suspect that the original language modelling
weights better align with NER as a token-level classification task and do not
diverge to incompatible sets of parameters in fine-tuning (cf. §3 of Schmidt
et al. (2023)).

###  A.2 Full Results By Hyperparameter Configuration

zs-xlt Performance Hyperparameters NLI TyDiQA-GoldP NER Learning Rate Batch
Size last src-dev ca trg-dev last src-dev ca trg-dev last src-dev ca trg-dev
$16$ $77.2_{0.1}$ $77.5_{0.3}$ $77.6_{0.2}$ $77.9_{0.5}$ $71.3_{0.3}$
$71.3_{0.1}$ $68.4_{0.8}$ $71.4_{0.3}$ $\mathbf{45.9_{0.3}}$
$\mathbf{45.9_{0.3}}$ $\mathbf{46.5_{0.3}}$ $47.8_{0.1}$ $1e^{{-}6}$ $32$
$\mathbf{77.4_{0.1}}$ $77.5_{0.2}$ $77.7_{0.2}$ $78.1_{0.1}$ $71.2_{0.2}$
$71.3_{0.4}$ $63.2_{0.3}$ $71.6_{0.2}$ $45.3_{0.2}$ $45.3_{0.2}$ $44.7_{0.3}$
$45.8_{0.3}$ $64$ $77.6_{0.1}$ $77.6_{0.3}$ $77.4_{0.1}$ $77.8_{0.1}$
$70.9_{0.4}$ $70.9_{0.4}$ $49.8_{0.2}$ $70.9_{0.1}$ $45.3_{0.2}$ $45.3_{0.2}$
$42.6_{0.3}$ $45.5_{0.2}$ $16$ $76.7_{0.2}$ $76.9_{0.2}$ $77.5_{0.3}$
$77.7_{1.1}$ $71.6_{0.2}$ $71.9_{0.7}$ $72.2_{0.1}$ $72.0_{0.4}$ $44.0_{0.9}$
$44.9_{0.9}$ $47.4_{0.3}$ $49.8_{0.3}$ $5e^{{-}6}$ $32$ $76.8_{0.0}$
$77.6_{0.4}$ $77.6_{0.1}$ $78.3_{0.5}$ $71.6_{0.1}$ $71.5_{0.1}$ $71.6_{0.1}$
$72.3_{0.4}$ $42.8_{0.3}$ $42.9_{0.2}$ $45.8_{0.6}$ $47.7_{1.0}$ $64$
$77.0_{0.2}$ $\mathbf{78.1_{0.6}}$ $\mathbf{77.8_{0.2}}$ $\mathbf{78.3_{0.3}}$
$71.0_{0.9}$ $70.9_{0.6}$ $69.0_{0.7}$ $71.7_{0.1}$ $43.8_{1.6}$ $43.8_{1.6}$
$46.2_{1.3}$ $49.0_{0.4}$ $16$ $76.6_{0.2}$ $76.6_{0.2}$ $77.5_{0.2}$
$77.1_{0.2}$ $73.0_{0.4}$ $72.5_{0.4}$ $73.5_{0.5}$ $73.7_{0.6}$ $40.6_{0.1}$
$40.7_{1.8}$ $43.9_{1.4}$ $48.0_{2.5}$ $1e^{{-5}}$ $32$ $76.8_{0.2}$
$77.1_{0.4}$ $77.6_{0.1}$ $77.2_{0.2}$ $72.1_{0.4}$ $72.6_{0.5}$ $73.0_{0.3}$
$73.2_{0.3}$ $40.1_{1.8}$ $40.1_{1.8}$ $42.8_{1.5}$ $46.0_{3.1}$ $64$
$76.8_{0.3}$ $77.2_{0.5}$ $77.5_{0.2}$ $77.7_{0.2}$ $72.0_{0.8}$ $71.7_{0.8}$
$71.7_{0.2}$ $73.0_{0.3}$ $43.1_{1.9}$ $43.4_{2.3}$ $46.6_{1.2}$ $49.7_{0.7}$
$16$ $75.6_{0.1}$ $75.6_{0.2}$ $76.8_{0.3}$ $76.5_{0.4}$ $73.7_{0.5}$
$\mathbf{73.3_{0.3}}$ $74.4_{0.6}$ $\mathbf{74.1_{0.4}}$ $41.0_{3.1}$
$42.0_{2.6}$ $45.3_{2.0}$ $\mathbf{50.0_{1.1}}$ $1.5e^{{-5}}$ $32$
$76.6_{0.1}$ $76.5_{0.1}$ $77.4_{0.2}$ $77.0_{0.1}$ $73.0_{0.7}$ $73.1_{0.9}$
$74.0_{0.1}$ $73.7_{0.6}$ $39.9_{1.1}$ $40.6_{1.3}$ $43.3_{0.2}$ $46.2_{1.9}$
$64$ $76.8_{0.1}$ $77.1_{0.5}$ $77.6_{0.3}$ $77.7_{0.6}$ $72.9_{0.5}$
$72.6_{0.7}$ $73.3_{0.4}$ $73.5_{0.3}$ $40.6_{2.1}$ $41.1_{2.0}$ $42.8_{1.5}$
$46.0_{1.1}$ $16$ $74.1_{0.3}$ $74.1_{0.3}$ $76.1_{0.2}$ $74.8_{0.2}$
$72.9_{0.5}$ $72.7_{0.2}$ $74.1_{0.2}$ $73.5_{0.2}$ $38.6_{1.2}$ $38.6_{1.2}$
$43.8_{1.4}$ $46.2_{3.0}$ $2e^{{-5}}$ $32$ $75.9_{0.3}$ $76.1_{0.4}$
$77.1_{0.1}$ $76.4_{0.5}$ $73.1_{0.1}$ $72.3_{0.9}$ $74.1_{0.2}$ $73.3_{0.4}$
$39.3_{0.7}$ $39.0_{1.1}$ $42.3_{1.5}$ $44.3_{2.5}$ $64$ $76.6_{0.5}$
$77.0_{0.4}$ $77.4_{0.1}$ $77.7_{0.2}$ $71.9_{0.4}$ $71.7_{1.1}$ $73.3_{0.5}$
$72.8_{0.4}$ $39.5_{1.2}$ $39.7_{1.3}$ $42.3_{1.1}$ $46.2_{0.4}$ $16$
$71.5_{0.2}$ $71.3_{0.4}$ $75.0_{0.2}$ $73.1_{0.1}$ $\mathbf{73.2_{0.4}}$
$72.4_{0.9}$ $74.7_{0.1}$ $73.4_{0.5}$ $39.3_{1.0}$ $39.3_{1.0}$ $44.3_{1.7}$
$47.1_{0.6}$ $2.5e^{{-5}}$ $32$ $74.8_{0.2}$ $74.8_{0.2}$ $76.3_{0.1}$
$76.1_{0.6}$ $72.3_{0.2}$ $71.7_{1.0}$ $74.8_{0.3}$ $73.4_{0.3}$ $39.4_{1.5}$
$39.4_{1.5}$ $42.5_{0.6}$ $44.5_{0.7}$ $64$ $76.7_{0.2}$ $76.7_{0.2}$
$77.5_{0.1}$ $77.3_{0.5}$ $72.6_{0.6}$ $72.3_{0.7}$ $74.2_{0.1}$ $73.3_{1.0}$
$39.9_{1.0}$ $39.9_{1.0}$ $43.5_{1.5}$ $47.7_{2.3}$ $16$ $67.9_{0.6}$
$67.7_{0.2}$ $73.0_{0.3}$ $70.8_{0.9}$ $72.3_{0.2}$ $71.7_{0.4}$ $74.4_{0.4}$
$72.4_{0.5}$ $37.4_{1.5}$ $37.3_{1.1}$ $43.0_{0.8}$ $46.3_{2.5}$ $3e^{{-5}}$
$32$ $73.3_{0.1}$ $73.3_{0.4}$ $75.8_{0.2}$ $74.4_{0.1}$ $71.7_{0.4}$
$71.6_{0.7}$ $\mathbf{75.1_{0.1}}$ $73.6_{0.8}$ $37.9_{1.6}$ $38.0_{1.8}$
$43.7_{1.6}$ $47.2_{2.7}$ $64$ $75.6_{0.1}$ $75.4_{0.3}$ $76.8_{0.3}$
$76.3_{0.7}$ $72.0_{0.2}$ $71.8_{0.4}$ $74.1_{0.5}$ $73.2_{0.4}$ $39.0_{1.9}$
$39.4_{1.4}$ $42.1_{1.2}$ $44.0_{1.2}$ $\mathbf{\Delta}$ $0.0$ $0.6$ $0.3$
$0.0$ $2.0$ $2.0$ $1.0$ $0.0$ $4.9$ $5.2$ $2.6$ $0.2$

Table 3: zs-xlt averaged over all target languages by task, model variant, and
hyperparameters (cf. §3). For each column, best zs-xlt emphasized in bold and
max. validation performance (cf. Table 4) shaded in green. $\mathbf{\Delta}$
is the difference of best zs-xlt and zs-xlt on models that maximize validation
performance. Metrics: accuracy for NLI, span-$F_{1}$ for TyDiQA and token-
level $F_{1}$ for NER. Subscripts denote std. deviation.

Validation Set Performance Hyperparameters NLI TyDiQA-GoldP NER Learning Rate
Batch Size last src-dev ca trg-dev last src-dev ca trg-dev last src-dev ca
trg-dev $16$ $\mathbf{90.2_{0.2}}$ $\mathbf{90.5_{0.2}}$ $90.1_{0.2}$
$79.2_{0.3}$ $76.4_{0.2}$ $76.7_{0.5}$ $75.1_{0.3}$ $67.0_{0.2}$ $81.9_{0.1}$
$81.9_{0.1}$ $79.5_{0.2}$ $49.9_{0.1}$ $1e^{{-}6}$ $32$ $\mathbf{90.2_{0.2}}$
$90.3_{0.1}$ $90.0_{0.0}$ $79.2_{0.1}$ $\mathbf{77.0_{0.5}}$
$\mathbf{77.9_{0.8}}$ $73.5_{0.4}$ $66.9_{0.1}$ $80.4_{0.1}$ $80.4_{0.1}$
$76.5_{0.3}$ $48.3_{0.3}$ $64$ $90.1_{0.1}$ $90.2_{0.1}$ $89.6_{0.1}$
$78.9_{0.0}$ $76.0_{0.6}$ $76.3_{0.4}$ $64.4_{0.2}$ $66.5_{0.2}$ $78.0_{0.1}$
$78.0_{0.1}$ $71.4_{0.4}$ $47.7_{0.2}$ $16$ $89.3_{0.4}$ $89.7_{0.2}$
$90.1_{0.2}$ $78.9_{0.6}$ $73.8_{0.9}$ $76.0_{0.5}$ $75.9_{0.8}$ $68.7_{0.1}$
$85.1_{0.3}$ $85.3_{0.4}$ $84.8_{0.1}$ $\mathbf{52.1_{0.4}}$ $5e^{{-}6}$ $32$
$89.5_{0.4}$ $89.9_{0.2}$ $90.1_{0.2}$ $79.4_{0.4}$ $74.5_{0.5}$ $75.8_{0.7}$
$76.2_{0.8}$ $68.2_{0.2}$ $84.7_{0.1}$ $84.7_{0.1}$ $84.0_{0.2}$ $50.0_{1.2}$
$64$ $89.6_{0.1}$ $90.2_{0.3}$ $90.0_{0.1}$ $\mathbf{79.4_{0.3}}$ $75.1_{0.7}$
$76.2_{0.5}$ $75.5_{0.4}$ $67.8_{0.2}$ $84.2_{0.2}$ $84.2_{0.2}$ $83.0_{0.3}$
$51.2_{0.4}$ $16$ $88.9_{0.3}$ $89.1_{0.2}$ $89.6_{0.1}$ $78.2_{0.1}$
$74.5_{0.4}$ $75.7_{0.1}$ $75.8_{0.6}$ $69.4_{0.3}$ $85.6_{0.1}$
$\mathbf{85.7_{0.1}}$ $\mathbf{85.8_{0.2}}$ $50.3_{2.2}$ $1e^{{-5}}$ $32$
$89.1_{0.1}$ $89.3_{0.2}$ $89.6_{0.1}$ $78.6_{0.3}$ $74.4_{0.6}$ $75.9_{0.6}$
$75.7_{1.0}$ $69.2_{0.1}$ $85.4_{0.1}$ $85.5_{0.2}$ $85.3_{0.2}$ $48.4_{3.1}$
$64$ $89.7_{0.4}$ $89.9_{0.0}$ $\mathbf{90.1_{0.1}}$ $79.1_{0.3}$ $74.8_{1.2}$
$76.1_{0.0}$ $76.7_{0.4}$ $69.0_{0.2}$ $85.0_{0.1}$ $85.0_{0.1}$ $84.5_{0.1}$
$52.0_{0.8}$ $16$ $88.4_{0.3}$ $88.5_{0.2}$ $89.1_{0.2}$ $77.5_{0.2}$
$74.8_{0.7}$ $76.3_{0.3}$ $76.5_{0.7}$ $\mathbf{69.7_{0.2}}$
$\mathbf{86.0_{0.1}}$ $86.1_{0.1}$ $86.2_{0.1}$ $52.0_{1.3}$ $1.5e^{{-5}}$
$32$ $89.0_{0.5}$ $89.0_{0.5}$ $89.6_{0.1}$ $78.2_{0.2}$ $75.1_{0.7}$
$76.5_{0.4}$ $76.5_{0.3}$ $69.5_{0.4}$ $85.4_{0.2}$ $85.5_{0.2}$ $85.8_{0.1}$
$48.3_{1.6}$ $64$ $89.0_{0.3}$ $89.4_{0.3}$ $89.5_{0.2}$ $78.7_{0.3}$
$75.5_{0.9}$ $76.2_{0.3}$ $76.3_{0.8}$ $69.3_{0.2}$ $85.1_{0.2}$ $85.2_{0.1}$
$85.2_{0.3}$ $48.3_{1.1}$ $16$ $87.7_{0.6}$ $87.9_{0.3}$ $88.9_{0.4}$
$75.8_{0.2}$ $74.6_{0.7}$ $76.5_{0.8}$ $76.8_{0.7}$ $69.3_{0.1}$ $85.9_{0.2}$
$85.9_{0.2}$ $86.0_{0.2}$ $48.2_{3.1}$ $2e^{{-5}}$ $32$ $88.7_{0.2}$
$88.8_{0.1}$ $89.3_{0.4}$ $77.6_{0.3}$ $76.5_{1.5}$ $77.1_{1.0}$
$\mathbf{78.1_{0.5}}$ $69.4_{0.5}$ $85.4_{0.2}$ $85.4_{0.2}$ $85.7_{0.2}$
$46.8_{2.5}$ $64$ $89.3_{0.2}$ $89.4_{0.1}$ $89.8_{0.1}$ $78.7_{0.2}$
$73.9_{0.7}$ $76.3_{0.3}$ $76.9_{0.5}$ $69.2_{0.1}$ $85.2_{0.2}$ $85.4_{0.3}$
$85.6_{0.3}$ $48.4_{0.6}$ $16$ $87.5_{0.3}$ $87.7_{0.1}$ $88.6_{0.3}$
$74.0_{0.3}$ $75.8_{0.9}$ $76.4_{0.2}$ $77.2_{0.5}$ $69.3_{0.3}$ $85.5_{0.2}$
$85.5_{0.2}$ $86.0_{0.3}$ $49.5_{0.5}$ $2.5e^{{-5}}$ $32$ $88.6_{0.1}$
$88.6_{0.1}$ $89.0_{0.2}$ $76.9_{0.4}$ $74.7_{0.6}$ $76.0_{0.3}$ $77.0_{0.6}$
$69.1_{0.1}$ $85.7_{0.2}$ $85.7_{0.2}$ $86.0_{0.1}$ $46.9_{0.6}$ $64$
$88.8_{0.3}$ $88.9_{0.2}$ $89.4_{0.2}$ $78.4_{0.4}$ $75.1_{1.8}$ $76.1_{0.7}$
$76.9_{0.8}$ $69.2_{0.4}$ $85.4_{0.0}$ $85.4_{0.0}$ $85.7_{0.3}$ $49.8_{2.3}$
$16$ $86.7_{0.2}$ $86.8_{0.2}$ $87.7_{0.1}$ $71.7_{0.7}$ $74.1_{0.6}$
$75.6_{1.5}$ $76.1_{0.8}$ $68.5_{0.2}$ $85.5_{0.1}$ $85.6_{0.1}$ $86.2_{0.0}$
$48.5_{2.4}$ $3e^{{-5}}$ $32$ $87.8_{0.3}$ $88.0_{0.5}$ $89.0_{0.3}$
$75.3_{0.3}$ $73.8_{1.3}$ $76.2_{1.0}$ $76.5_{0.8}$ $69.1_{0.2}$ $85.4_{0.1}$
$85.4_{0.1}$ $86.0_{0.1}$ $49.4_{3.0}$ $64$ $88.4_{0.2}$ $88.5_{0.1}$
$89.4_{0.2}$ $77.4_{0.7}$ $74.8_{0.6}$ $76.3_{0.2}$ $77.7_{0.4}$ $69.1_{0.2}$
$85.4_{0.1}$ $85.4_{0.1}$ $85.8_{0.1}$ $46.2_{1.5}$

Table 4: Validation performance by task, model variant, and hyperparameters
(cf. §3). last, src-dev, and ca validate on source-language validation splits;
trg-dev denotes performance averaged over individual snapshots of a run that
perform best by target-language validation set. For each column, best
validation performance in bold. Metrics: accuracy for NLI, span-$F_{1}$ for
TyDiQA and token-level $F_{1}$ for NER. Subscripts denote std. deviation.


# Noise-specific beats in the higher-level Ramsey curves of a transmon qubit

L. A. Martinez  Lawrence Livermore National Laboratory, CA, USA  Z. Peng
Michigan State University, East Lansing, MI  D. Appelö  Michigan State
University, East Lansing, MI  D. M. Tennant  N. Anders Petersson  J. L DuBois
Y. J. Rosen  Lawrence Livermore National Laboratory, CA, USA

(Nov 11, 2022)

###### Abstract

In the higher levels of superconducting transmon devices, and more generally
charge sensitive devices, $T_{2}^{*}$ measurements made in the presence of
low-frequency time-correlated $1/f$ charge noise and quasiparticle-induced
parity flips can give an underestimation of the total dephasing time. The
charge variations manifest as beating patterns observed in the overlay of
several Ramsey fringe curves, and are reproduced with a phenomenological
Ramsey curve model which accounts for the charge variations. $T_{2}^{*}$
dephasing times which more accurately represent the total dephasing time are
obtained. The phenomenological model is compared with a Lindblad master
equation model. Both models are found to be in agreement with one another and
the experimental data. Finally, the phenomenological formulation enables a
simple method in which the power spectral density (PSD) for the low-frequency
noise can be inferred from the overlay of several Ramsey curves.

Owing to its exponentially suppressed charge sensitivity, the superconducting
transmon qubit has established itself as a leading candidate for NISQ-era
quantum computing systems Arute  _et al._ (2019); Kandala  _et al._ (2017).
Nevertheless, qubit dephasing and relaxation caused by interactions with
various noise sources continue to limit the time available for error-free
quantum gate operations. Measurements of the noise plaguing superconducting
qubits have been vital in illuminating and mitigating the various sources of
noise Bertet  _et al._ (2005); Sung  _et al._ (2019); Bylander  _et al._
(2011); Schlör  _et al._ (2019); Sung  _et al._ (2021); Yan  _et al._ (2012);
Serniak  _et al._ (2019); Yuge  _et al._ (2011); von Lüpke  _et al._ (2020);
Burnett  _et al._ (2019). For example, photon shot noise has been established
as one of the dominant sources of dephasing. This is especially important in
3D-transmon architectures Wang  _et al._ (2019) where qubit coherence is
dominated by pure dephasing, and mitigation through filtering and
thermalization has been demonstrated Sears  _et al._ (2012). Sources of noise
have also been attributed to the interaction of two-level-systems (TLS)
residing at the material interfaces and surfaces Burnett  _et al._ (2019),
non-equilibrium quasi-particles Ristè  _et al._ (2013); Serniak  _et al._
(2018), and anomalous charge drift across the junction Christensen  _et al._
(2019). More recently, cosmic rays were identified as a source of correlated
errors suggesting new obstacles in the way of reaching fault tolerant quantum
computation Wilen  _et al._ (2021).

As quantum computing platforms strive towards larger Hilbert spaces, the
higher levels of superconducting transmon qubits may hold promise for quantum
computing applications Wu  _et al._ (2020); Liu  _et al._ (2017). Admittedly,
most of the success of the transmon can be attributed to its first energy
transition (0-1) since the higher levels of the transmon exhibit shorter
dephasing times – suggesting charge noise magnitudes which prevent their usage
for applications in quantum computing. However, characterizing dephasing in
the presence of correlated charge noise can lead to a misrepresentation of the
time available for quantum state manipulation. Transmons and charge sensitive
qubits plagued with this problem exhibit seemingly random Ramsey curves, which
form beat patterns Peterer  _et al._ (2015) – a signature of correlated low-
frequency charge noise. In these devices, characterization of the dephasing
time requires explicit consideration of the charge noise.

The Ramsey pulse sequence Ramsey (1950) is at the core of qubit noise
measurements as it transforms the qubit into a sensitive probe susceptible to
both classical and quantum noise, and it is used to acquire Ramsey curves
which provide a measurement of the total dephasing parameter $T_{2}^{*}$ Vion
_et al._ (2002). A single Ramsey curve measurement consists of an initial
$R_{x}(\pi/2)$ pulse which initiates the qubit into a quantum superposition
state between two qubit levels, a free evolution period where its phase
evolves, and a second $R_{x}(\pi/2)$ pulse that projects the phase onto a
single qubit level for readout. During the free evolution period of a charge
sensitive state, charge noise coupling to the qubit modulates its resonant
frequency while energy relaxation leads to state decay; both manifest as
stochastic variations in the phase of the quantum superposition state.

If the charge fluctuations are zero-mean Gaussian distributed, averaging over
$N$ individual Ramsey curves reduces the standard error by $1/\sqrt{N}$. This
is the case when the charge noise exhibits a white noise power spectral
density. In this case, the $T_{2}^{*}$ value extracted from fitting an
exponentially decaying sinusoid to the average of several Ramsey curves is an
accurate representation of the total dephasing time. However, in the presence
of charge drift (time-correlated low-frequency $1/f^{\alpha}$ charge noise for
$\alpha>0$) Christensen  _et al._ (2019) and quasiparticle induced parity
switches Ristè  _et al._ (2013), the $T_{2}^{*}$ value extracted from fitting
a decaying sinusoid to the average of multiple Ramsey curves can result in an
underestimation of the total dephasing time. In this case, the canonical
approach averages several Ramsey curves over slow-varying stochastic charge
noise – resulting in an effective destructive interference. This
underestimation of $T_{2}^{*}$ also occurs in transmon qubits where, despite
being generally regarded as charge insensitive devices, the higher levels can
exhibit sufficiently large charge sensitivity to be affected Tennant  _et al._
(2022).

We begin this letter with a demonstration of how the charge noise manifests as
a beating pattern in the higher levels of a 3D transmon system Wu  _et al._
(2020), followed with a quick discussion on how noise leads to ambiguous
$T_{2}^{*}$ measurements. We then present a phenomenological Ramsey curve
model which explicitly accounts for slow-rate charge parity flips and charge
drift. We extract a corrected $T_{2}^{*}$ and compare it to the value obtained
with the canonical method. Next, we use the phenomenological model to identify
the noise power spectral density (PSD), from which we can determine the likely
origin of the noise. Finally, the phenomenological model is compared with both
experimental data and full Lindblad master equation simulations. We find that
both models reproduce the experimental data.

Figure 1: Overlay, decay envelope (black dashed-dotted lines), and respective
averages (white-dashed lines) of 50 Ramsey curves collected sequentially on
the first 3 levels of a transmon qubit with 500 kHz Ramsey detuning. Beat
patterns are observed on levels with larger dispersion values. Starting at
top, the charge dispersion for the 0-1, 1-2, and 2-3 levels are 1.7, 62, and
1305 kHz, respectively.

Quantum systems utilizing the higher levels of transmon qubits, and more
generally charge sensitive devices, plagued with charge noise exhibit random
Ramsey curves between repeated measurements. The overlay of 50 Ramsey curves
along with their respective averages (white-dashed lines) for each of the
first three levels of a transmon are shown in figure 1. The charge noise
coupled to the qubit results in seemingly random individual Ramsey curves,
which when superimposed yield a sinusoidal beating pattern for the 1-2 and 2-3
levels. The first level has a low-dispersion value calculated to be 1.7 kHz.
In this case, the 50-curve overlay does not exhibit a beating pattern and
averaging over more curves is beneficial. However owing to the higher
dispersion value of 62 kHz measured for the 1-2 level, it is more sensitive to
charge variations. The increased charge sensitivity leads to the beating
pattern observed in the 50-curve overlay that is otherwise not visible on the
averaged curve. This behavior is significantly more pronounced for the 2-3
level which has a measured dispersion of 1.3 MHz. Using the canonical approach
of fitting a decaying sinusoidal to the averaged traces, results in
${\tilde{T}}_{2}^{*}=2.7~{}\mu$s and ${\tilde{T}}_{2}^{*}=0.3~{}\mu$s for the
1-2 and 2-3 levels, respectively. This approach of extracting the total
dephasing time leads to values that are dominated by low-frequency noise and,
therefore, underestimate the total dephasing time.

Figure 2:  a) The relevant time scales. With a 1 kHz single-shot rate, a
Ramsey curve is collected approximately every 26 seconds; $1/f^{2}$
(simulated) charge noise becomes significant after the first Ramsey curve
measurement. b) Multi-level Ramsey measurement sequence in a transmon qubit
sup  with $E_{j}/Ec=50$. Each Ramsey curve consists of 101 free-evolution time
values ($t_{R})$ with 256 single shots collected for each value. The first
Ramsey curve is collected on the 0-1 level, the second on the 1-2, the third
on 2-3, and this sequence repeats until 50 Ramsey curves are collected for
each level giving a total of $N=150$ curves. Corresponding pulse sequences are
shown on the right. The 1-2 and 2-3 levels use additional pre (post)
$\pi$-pulse sequences to excite (de-excite) the qubit. Ground state is used
for readout Peterer  _et al._ (2015).

To understand how the beating patterns arise, we consider the relevant
timescales involved. The Ramsey curves shown in figure 1 were collected on the
first three levels of a superconducting 3D transmon CQED system Wu  _et al._
(2020); sup  suffering from quasiparticle poisoning Ristè  _et al._ (2013) and
charge drift Christensen  _et al._ (2019) (figure 2). The single-shot
measurement repetition rate was set to 1 kHz, 101 free-evolution time values
and 256 shots per value were collected, and a single Ramsey experiment was
completed every 26 seconds. At these timescales, the quasiparticle-induced
parity flips occurring at a rate of approximately $\sim 1$ kHz Ristè  _et al._
(2013) lead to an effective destructive interference when averaging over
multiple shots (or multiple Ramsey curves). A characteristic feature of
$1/f^{\alpha}$ noise is that its correlation length increases with the
exponent $\alpha$; $\alpha=0$ (white noise) is uncorrelated while $\alpha=2$
(brown noise) is time-correlated. Due to this correlation, as the number of
Ramsey curves included in the average increases the contribution of low-
frequency $1/f^{2}$ charge noise on the measured $T_{2}^{*}$ becomes more
significant. Furthermore, looking at an individual 2-3 level Ramsey curve
(figure 2b) one might incorrectly conclude that its $T_{2}^{*}$ is too short
for practical applications. Note, in practice long acquisition times arise
from the lack of high fidelity single-shot readout which requires averaging
over several measurements, and/or measurements that are made with slow
sampling rate systems.

Next, we formulate a phenomenological model which takes into account charge-
parity flips and low-frequency noise and, thereby, yields $T_{2}^{*}$
measurements which more accurately represent the time available for coherent
state manipulation. Previous experimental observations have demonstrated
quasiparticle events in transmon qubits which cause the instantaneous qubit
frequency to switch symmetrically about the mean qubit frequency at milli-
second time intervals Ristè  _et al._ (2013);
$f_{q}(t)=\bar{f}_{q,ij}+\epsilon^{\text{max}}_{ij}\cos(2\pi
n_{g}(t)+p(t)\pi)$ where the second term is the dispersion relation of the
$ij^{\text{th}}$ level given by the approximation of the Mathieu solutions in
the transmon limit Koch  _et al._ (2007); Cottet (2002),
$\epsilon^{\text{max}}_{ij}$ is the level’s maximum dispersion, $n_{g}$ is the
charge bias across the transmon’s Josephson junction, and $p\in\\{0,1\\}$ is
the parity. Since a quasiparticle event can be assumed to change the qubit
frequency instantaneously relative to the single shot rate, in the limit where
the rate of parity-switching events is much slower than $1/T_{2}^{*}$, the
qubit frequency will appear to switch between two dominant spectral components
(i.e. charge parity bands). The averaged behavior of the two spectral
components, assumed to be equally displaced in frequency from the average
qubit frequency ${\bar{\omega}_{q}}$Ristè  _et al._ (2013), can be modeled in
a (detuned) Ramsey curve measurement by the relation,

${\cal
P}\sim\frac{1}{2}\left[\cos((\Omega_{R}+\epsilon_{ij}(t))t_{R})+\cos((\Omega_{R}-\epsilon_{ij}(t))t_{R})\right],$
(1)

where ${\cal P}$ is the qubit population during a Ramsey measurement,
$\Omega_{R}\equiv{\bar{\omega}_{q}}-\omega_{d}$ is the Ramsey detuning
frequency with respect to the average qubit frequency, $\omega_{d}$ is the
qubit drive frequency, $t_{R}$ is the Ramsey free-evolution time, and

$\epsilon_{ij}(t)=\epsilon^{\text{max}}_{ij}\cos(2\pi n_{g}(t)).$ (2)

The charge bias across the junction is subject to time-dependent stochastic
charge noise $n_{g}(t)$. Here, $t$ represents the dynamical time variable and
should not be confused with the Ramsey free-evolution time $t_{R}$.

The total dephasing time that is of interest corresponds to the Ramsey curve
decay envelope. Note, determining the appropriate functional form of the decay
envelope in the presence of noise processes occuring at timescales on-par or
faster than the time it takes to perform a single Ramsey experiment has been
previously addressed Ithier  _et al._ (2005). To demonstrate how our method
accounts for the low-frequency charge noise, we assume the charge noise
responsible for the decaying envelope in a single Ramsey experiment is
represented by a white-noise power spectral density, as is typical in
practice. In this case, the uncorrelated Gaussian noise processes are
incorporated with a $e^{-t/T_{2}^{*}}$ term Martinis  _et al._ (2003);
Wangsness and Bloch (1953). Combining equations (1) and (2) with the
exponential decay term and using the sum of cosine identity gives the
expression for the phenomenological model for the qubit population during a
Ramsey curve measurement,

${\cal
P}(t,t_{R})=\frac{1}{2}\left[1+e^{-t_{R}/T_{2}^{*}}\cos(\Omega_{R}t_{R})\cos(\epsilon_{ij}(t)t_{R})\right],$
(3)

normalized to represent the excited state probability. This expression is
consistent with the fact that dephasing processes can be factorized, so long
the high-frequency depolarization noise is non-deterministic Ithier  _et al._
(2005); Makhlin and Shnirman (2003).

Equation (3) forms the phenomenological model and it represents the main
result of this letter. It can be used in charge sensitive systems to extract a
$T_{2}^{*}$ measurement which excludes quasiparticle and charge noise
contributions. Importantly, in contrast to the canonical decaying sinusoid
formulation this formulation for the Ramsey state population includes an
additional term, $\cos(\epsilon_{ij}(t)t_{R})$, which carries an explicit time
dependance. For this reason, averaging over the slow stochastic variations in
$\epsilon_{ij}(t)$ will give rise to an effective destructive interference.
Measurements of $T_{2}^{*}$ should include only the decaying envelope in (3),
i.e., $\exp(-t/T_{2}^{*})$ since we have assumed Gaussian noise processes.
This is accomplished by fitting an exponential to the envelope of the beating
pattern observed in the overlay of several Ramsey measurements. Fitting to the
envelope (black dashed-dotted lines in figure 1), the best fit $T_{2}^{*}$
values for the 1-2 and 2-3 levels were found to be $14.5~{}\mu$s and
$4.3~{}\mu$s, respectively. These values are significantly different from the
values reported above, obtained with the canonical approach:
${\tilde{T}}_{2}^{*}=2.7~{}\mu$s and ${\tilde{T}}_{2}^{*}=0.3~{}\mu$s,
respectively.

Figure 3: Ramsey curves generated with equation 3 for selected values of both
the noise exponent $\alpha$ and noise amplitude $S_{q}(1Hz)=A$. Curves shown
are for the 2-3 level. Noise simulation sampling rate was 1 kHz, and
$\Omega_{R}=750$ kHz.

We now use equation (3) to explore Ramsey curve phenomenology in relation to
the 1 kHz single-shot rate for selected $S(f)=A/f^{\alpha}$ noise models of
$n_{g}(t)$. Figure 3 shows the results of selected 2-3 level Ramsey curves
generated with the simulated noise for exponents $\alpha=\\{0,1,2,3\\}$ Timmer
and Koenig (1995), and selected noise amplitudes $A$ in the
$[10^{-6},10^{-4}]e^{2}/\text{Hz}$ range. A key observation is highlighted.
For $\alpha=0$ and $\alpha=1$, at low amplitudes the charge fluctuations are
insufficient to cause a significant change between the individual Ramsey
curves, while at higher amplitudes the larger charge variations scramble the
phase of the Ramsey curves at single-shot times scales. That is, for the 1 kHz
single-shot repetition rate, the short correlation time associated with
$\alpha=1$ is practically indistinguishable from uncorrelated ($\alpha=0$)
noise. Therefore, not much variability among the 50 Ramsey curves is observed
- they all resemble the average value. However as the noise moves from
uncorrelated towards long-time correlated noise
($\alpha=0\rightarrow\alpha=3$), the low-frequency contribution of the
$1/f^{\alpha}$-type charge noise becomes significant, and the low-frequency
charge variations manifest with the experimentally observed beating pattern.

Figure 4: Logarithm of the sum of the fit error between the simulated and
experimental Ramsey curves, see text.

Next, we find the noise power spectral density which best reproduces the
experimental data. First, simulated timeseries of $S(f)=A/f^{\alpha}$ were
generated Timmer and Koenig (1995) by performing a 2-dimensional sweep of the
noise exponent ($\alpha$) and a scaling amplitude ($a$). The scaling amplitude
simplifies the noise simulations over various $\alpha$. Since converting the
noise timeseries into the frequency domain results in unique $A$ for each
$(\alpha,a)$ pair, this approach avoids solving the inverse problem. Note,
$A\propto a^{2}$ sup . The sampling rate for the noise generation was set to 1
kHz, $\alpha$ varied from $0\leq\alpha\leq 3$ in steps of 0.1, and $a$ varied
logarithmically from $10^{-3}$ to $10^{2}$. Each simulated noise timeseries,
$n_{g}(t)$ (cyclic in units of $2e$), was then substituted into equations (2)
and (3), and the corrected $T_{2}^{*}$ values obtained by fitting the
envelopes of the beat patterns were used. The best-fit noise spectral density
was determined by calculating the mean-squared difference between the
experimental and simulated curves (simulated with equation (3)) for three
metrics: the FFT’s of the individual Ramsey curves, the envelopes of 50-curve
overlay, and the 50-curve averages. The FFTs of the individual Ramsey curves
capture the high frequency features, the envelope of the 50-curve overlay
capture the decay, and the averages capture the long-term behavior. The fit-
error was defined as the equal-weighted sum of the three metrics sup . The
logarithm of the fit-error is plotted in figure 4. Locating the minimum
yielded the best fit $\alpha=2.0\pm 0.2$ and noise magnitude
$S_{q}(1Hz)=(2.7\pm 2.3)\times 10^{-5}~{}e^{2}$/Hz sup .

Finally, equation (3) can be used as an analytical approximation for low-
frequency stochastic charge variations, avoiding having to solve the full
Linblad master equation. To validate this, we checked the consistency of
equation (3) with Lindblad master equation simulations for the density matrix.
50 Ramsey curves on the 2-3 level were generated by solving the Lindblad
master equation at each time step of the noise timeseries. Again, the
simulations used the corrected dephasing values, and
$\alpha=2,\text{and}~{}A=2.7\times 10^{-5}e^{2}$/Hz for the noise parameters
sup . Comparing the experimental Ramsey curves collected on the 2-3 level with
both, the 50 simulated curves generated by solving the Lindblad equation and
the 50 curves generated with equation (3), the Lindblad-simulated curves
qualitatively match the behavior of the experimental data as well as the
curves generated with equation (3) (figure 5).

Figure 5: Comparison between (a) experimental and (b,c) simulated Ramsey
curves. Ramsey detuning frequency was set to $750$ kHz; white-dashed lines
represent the 50-curve averages.

In summary, we discussed how parity switches Serniak  _et al._ (2018) and
charge drift can lead to ambiguous $T_{2}^{*}$ measurements, and presented a
phenomenological Ramsey curve model which encapsulates the slow-varying charge
fluctuations and yields measurements of $T_{2}^{*}$ which more accurately
represent the total dephasing time in the higher levels of transmon qubits.
Furthermore, we presented a practical approach for probing the color of the
noise power spectral density by identifying unique beat patterns in the
overlay of many Ramsey curves. These results should be of interest for qudit-
based quantum computing platforms which aim to utilize the higher levels of
transmon qubits in order to expand the computational (Hilbert) space.

See supplementary material for measurements and simulations details.

This work was supported by the U.S. Department of Energy (DOE), Office of
Science, Basic Energy Sciences (BES) under Award DE-SC0020313 and at Lawrence
Livermore National Laboratory (LLNL) under Contract No. DE-AC52-07NA27344 by
DOE, Office of Science, BES, Materials Sciences and Engineering Division. We
thank MIT-Lincoln Laboratories and IARPA for providing the Traveling Wave
Parametric Amplifier used in this study. Experimental characterization was
made possible by the National Nuclear Security Administration Advanced
Simulation and Computing Beyond Moore’s Law (NA-ASC-127R-16) program support
of the LLNL Quantum Design and Integration Testbed (QuDIT). The numerical
simulations were performed under support from DOE, Office of Science, Office
of Advanced Scientific Computing Research, under project TEAM (Tough Errors
Are no Match) (SCW-1683.1).(LLNL-JRNL-841934).

*Supplementary material: Noise-specific beats in the higher-level Ramsey curves of a transmon qubit

##  .1 Multi-level Ramsey curve measurements in a 3D transmon platform

The experimental platform was a 3D aluminum on silicon superconducting
transmon qubit housed in a 3D “clamshell” superconducting aluminum RF cavity
Paik  _et al._ (2011). The readout scheme was a heterodyned configuration with
an intermediate frequency of 25 MHz. The amplification chain included a near
quantum-limited TWPA pre-amplifier Macklin  _et al._ (2015) at the mixing
chamber of the a dilution refrigerator held below 10 mK, followed with a 4K
HEMT at the 4-Kelvin stage and additional room temperature amplifiers before
the 2 GHz ADC. The wiring and equipment details are summarized in figure 6.
For further details and specifications on the qubit/cavity quantum
characterization and readout the reader is referred to references Wu  _et al._
(2020); Martinez  _et al._ (2020). The multi-level Ramsey fringe measurement
consisted of a preliminary ladder-type sequence as in reference Peterer  _et
al._ (2015). First, we applied a sequence of
$\\{\pi_{0,1},\pi_{1,2},..,\pi_{ij}\\}$ excitation pulses which excited the
qubit from the ground state to the desired $j^{\text{th}}$ excited state where
the typical two-$\pi/2$-pulse Ramsey sequence,
$R_{x}(\pi/2)-t_{R}-R_{x}(\pi/2)$, was then applied between the
$j^{\text{th}}$ and the $j^{\text{th}}+1$ level. To avoid qubit readout
problems associated with the shorter $T_{1}$ values of the higher levels, a
de-excitation sequence for ground state readout was also then applied.

##  .2 Python Simulations

We describe the protocol, coded in python, used to simulate the Ramsey curves
via equation (3) along with the simulated $1/f^{\alpha}$ noise and the mean-
square error metric used to fit the power spectral density. To account for the
low frequency noise we assumed the charge drift associated with the low-
frequency charge noise occurs at timescales much longer than the 1 kHz single-
shot repetition rate. First, we generated a noise timeseries modeling
$n(t)=q(t)/2e$ for the total duration of the acquisition. Here the $t$ is
discretized by 1 ms intervals, corresponding to the 1 kHz single-shot
repetition rate. Therefore, for each single-shot measurement there is a
corresponding value in the noise timeseries. Each Ramsey curve consist of 101
free-evolution times sampled uniformly in $t_{R}\in[0,T_{R}]$, and at each
timestep we averaged over 256 shots before incrementing to the next Ramsey
timestep. The qudit Ramsey population was calculated by sampling $n(t)$ from
the noise timeseries and evaluating equation (3). The evaluation sequence was
carried out in identical fashion to the experimental measurement sequence
described in figure 2 for the total duration of the acquisition. Note that the
Ramsey detuning frequency $\Omega_{R}$ was set experimentally, and the
$T_{2}^{*}$ values were inferred from fitting to the envelope of the observed
beating pattern as described in the main text.

Figure 6: Experimental setup of the multi-level 3D transmon system used to
collect the Ramsey curves.

Before discussing the best-fit PSD procedure, we briefly discuss the scaling
factor used to simulate the PSD. Since the noise timeseries amplitude was
scaled to unit variance, a multiplicative scaling amplitude was used to scale
the 1 Hz spectral noise amplitude ($A=S(1Hz)$) accordingly. The conversion
between the dimensionless scaling amplitude $a$ and power density $A$ follows
from the Fourier Transform in the PSD calculation. Let ${n}(t,\alpha)$ be the
time domain representation of the $1/f^{\alpha}$ noise signal. For the noise
simulations, we varied the timeseries noise amplitude by varying the scaling
$a$, i.e., $a\times{n}$ is the charge noise signal. The PSD is given by,

$S(f,\alpha)=\left|\int{a\times n(t,\alpha)}e^{-i2\pi ft}dt\right|^{2}.$ (4)

Since the type of noise in the time domain is set a priori (i.e.
$1/f^{\alpha}$ noise ), its functional form in the frequency domain is
$S(f,\alpha)=a^{2}\times c_{\alpha}/f^{\alpha}$, where we explicitly defined
the coefficient
$c_{\alpha}=\int({n(t,\alpha)}e^{-i2\pi(1\text{Hz})t}dt)|^{2}$. That is,
$c_{\alpha}$ represents the unscaled 1 Hz spectral noise amplitude which would
result from a unit-variance noise timeseries. From $S(f,\alpha)=A/f^{\alpha}$,
the scaled 1 Hz spectral noise amplitude is $A=c_{\alpha}\times a^{2}$. Note
that this is expected, physically, since the power scales as the square of the
amplitude.

$f_{01}$ | $f_{12}$ | $f_{23}$ | $T_{1,1}$ | $T_{{1,2}}$ | $T_{{1,3}}$ | $T_{{2,1}}$ | $T_{{2,2}}$ | $T_{{2,3}}$
---|---|---|---|---|---|---|---|---
4.0108 GHz | 3.8830 GHz | 3.6287 GHz | 45 $\mu s$ | 21 $\mu s$ | 22 $\mu s$ | 24 $\mu s$ | 14.5$\pm 4~{}\mu s$ | 4.3$\pm 2.4~{}\mu s$
Table 1: Measured values for the different parameters. In our forward
simulations, we used $T_{{2,2}}=14.5~{}\mu s$ and $T_{{2,3}}=4.3~{}\mu s$,
which correspond to the best-fit values. The value of charge dispersion used
was $\epsilon_{23}=1.3$ MHz, which corresponds to the experimentally measured
value.

To find the best fit power spectral density between the experimental data and
curves generated with equation (3), a power-law noise timeseries for various
noise exponents $\alpha$ was generated with the algorithm described in
reference Timmer and Koenig (1995). Next, we swept the noise exponent and the
scaling amplitude. The exponents varied from $0\leq\alpha\leq 3$ in steps of
0.1, and the scaling amplitude varied from $[10^{-3}$,$10^{2}]$,
logarithmically. Continuing, we calculated the mean-squared difference,
$\sum_{i=0}^{n}{(x_{i}-y_{i})^{2}}$, where the array $x$ represents the data
and array $y$ the simulated curves, for 3 parameters that comprised our best-
fit metric. The first parameter was the mean-squared difference between the
FFTs of the individual Ramsey curves, this step discerns high frequency
spectral features of the individual curves. The second value was the mean-
squared difference between the envelopes of the overlaid Ramsey curves. The
envelopes of the 50-curve overlay were extracted with a simple python
algorithm which extracted the maximums and minimums. This step captures the
overall decay and slow varying beating features of the multi-curve overlay.
Finally, the mean-squared difference of the ensemble-averaged Ramsey curves
was used to capture the long-term averaged behavior. We defined our total
error parameter as the equal-weighted sum of these 3 parameters, the logarithm
of the sum was used since the error in the region of interest were relatively
small.

Finally, selected simulated power spectral densities are shown in figure 7.
The PSDs are selected so that their 1 Hz amplitude coincide at $A\sim
10^{-6}$. For low noise exponents, the high frequency noise dominates. As the
noise exponent increases the higher frequency noise falls off with slope
$\alpha$ and the low-frequency noise contribution becomes significant.

Figure 7: Selected simulated power spectral densities for $S(1\text{Hz})\sim
10^{-6}$, power spectrum drops off with a slope $\alpha$.

##  .3 Lindblad Simulations

Here we describe the Lindblad simulations. Our model is based on the
Hamiltonian $H(t)=H_{0}+H_{\rm c}(t)$, where

$H_{0}=\left[\begin{array}[]{cccc}0&0&0&0\\\ 0&\tilde{f}_{01}&0&0\\\
0&0&\tilde{f}_{01}+\tilde{f}_{12}&0\\\
0&0&0&\tilde{f}_{01}+\tilde{f}_{12}+\tilde{f}_{23}\end{array}\right]$ (5)

is the system Hamiltonian and $H_{\rm c}(t)$ is the control Hamiltonian. Here,

$\widetilde{f}_{j,j+1}=f_{j,j+1}-\epsilon_{j,j+1}\cos\left(2\pi\frac{q}{2e}+p\pi\right),$
(6)

where $p\in\\{0,1\\}$, $f_{j,j+1}$ is the transition frequency,
$\epsilon_{j,j+1}$ is the charge dispersion, $q$ is the charge over the
Josephson junction, $e$ is the charge of an electron and $p$ is the parity. In
principle $\frac{q}{2e}$ and $p$ can be seen as a random variables but here we
assume the parity event can be averaged and treated deterministically by
carrying out two simulations with $p=0$ and $p=1$ and taking their average. We
take $\frac{q}{2e}$ to be a be Wiener process which we explicitly account for
in our forward simulations (see below).

Incorporating loss due to Markovian interactions with the surrounding
environment leads us to a model governed by a Lindblad master equation for the
density matrix $\rho$

$\dot{\rho}=-i\left(H\rho-\rho H\right)+\sum_{j=1}^{2}\left({\cal
L}_{j}\rho{\cal L}_{j}^{\dagger}-\frac{1}{2}\left({\cal L}_{j}^{\dagger}{\cal
L}_{j}\rho+\rho{\cal L}_{j}^{\dagger}{\cal L}_{j}\right)\right).$ (7)

Here, decay and dephasing mechanisms are modeled through the operators ${\cal
L}_{1}$ and ${\cal L}_{2}$,

$\displaystyle{\cal L}_{1}$
$\displaystyle=\left[\begin{array}[]{cccc}0&\sqrt{\gamma_{1,1}}&0&0\\\
0&0&\sqrt{\gamma_{1,2}}&0\\\ 0&0&0&\sqrt{\gamma_{1,3}}\\\
0&0&0&0\end{array}\right],$ (12) $\displaystyle{\cal L}_{2}$
$\displaystyle=\left[\begin{array}[]{cccc}0&0&0&0\\\
0&\sqrt{\gamma_{{2,1}}}&0&0\\\ 0&0&\sqrt{\gamma_{{2,2}}}&0\\\
0&0&0&\sqrt{\gamma_{{2,3}}}\end{array}\right].$ (17)

The measured parameters used in the equations above are given in Table 1. The
Lindblad Ramsey curve simulations are carried out in similar sequence to
simulations using equation (3), except for one crucial step. Instead of
evaluating equation 3 for each sampled noise value, we now solved the Lindblad
equation at each timestep. The average curves of the experimental data and
computed results using the Lindblad master equation and the phenomenological
model are displayed in figure 5 of the main text.

## References

  * Arute  _et al._ (2019) F. Arute, K. Arya, R. Babbush, D. Bacon, J. C. Bardin, R. Barends, R. Biswas, S. Boixo, F. G. Brandao, D. A. Buell, _et al._ , Nature 574, 505 (2019).
  * Kandala  _et al._ (2017) A. Kandala, A. Mezzacapo, K. Temme, M. Takita, M. Brink, J. M. Chow, and J. M. Gambetta, Nature 549, 242 (2017).
  * Bertet  _et al._ (2005) P. Bertet, I. Chiorescu, G. Burkard, K. Semba, C. Harmans, D. P. DiVincenzo, and J. Mooij, Physical review letters 95, 257002 (2005).
  * Sung  _et al._ (2019) Y. Sung, F. Beaudoin, L. M. Norris, F. Yan, D. K. Kim, J. Y. Qiu, U. von Lüpke, J. L. Yoder, T. P. Orlando, S. Gustavsson, _et al._ , Nature communications 10, 1 (2019).
  * Bylander  _et al._ (2011) J. Bylander, S. Gustavsson, F. Yan, F. Yoshihara, K. Harrabi, G. Fitch, D. G. Cory, Y. Nakamura, J.-S. Tsai, and W. D. Oliver, Nature Physics 7, 565 (2011).
  * Schlör  _et al._ (2019) S. Schlör, J. Lisenfeld, C. Müller, A. Bilmes, A. Schneider, D. P. Pappas, A. V. Ustinov, and M. Weides, Physical review letters 123, 190502 (2019).
  * Sung  _et al._ (2021) Y. Sung, A. Vepsäläinen, J. Braumüller, F. Yan, I. Joel, J. Wang, M. Kjaergaard, R. Winik, P. Krantz, A. Bengtsson, _et al._ , Nature communications 12, 1 (2021).
  * Yan  _et al._ (2012) F. Yan, J. Bylander, S. Gustavsson, F. Yoshihara, K. Harrabi, D. G. Cory, T. P. Orlando, Y. Nakamura, J.-S. Tsai, and W. D. Oliver, Physical Review B 85, 174521 (2012).
  * Serniak  _et al._ (2019) K. Serniak, S. Diamond, M. Hays, V. Fatemi, S. Shankar, L. Frunzio, R. Schoelkopf, and M. Devoret, Physical Review Applied 12, 014052 (2019).
  * Yuge  _et al._ (2011) T. Yuge, S. Sasaki, and Y. Hirayama, Physical review letters 107, 170504 (2011).
  * von Lüpke  _et al._ (2020) U. von Lüpke, F. Beaudoin, L. M. Norris, Y. Sung, R. Winik, J. Y. Qiu, M. Kjaergaard, D. Kim, J. Yoder, S. Gustavsson, _et al._ , PRX Quantum 1, 010305 (2020).
  * Burnett  _et al._ (2019) J. J. Burnett, A. Bengtsson, M. Scigliuzzo, D. Niepce, M. Kudra, P. Delsing, and J. Bylander, npj Quantum Information 5, 1 (2019).
  * Wang  _et al._ (2019) Z. Wang, S. Shankar, Z. Minev, P. Campagne-Ibarcq, A. Narla, and M. H. Devoret, Physical Review Applied 11, 014031 (2019).
  * Sears  _et al._ (2012) A. Sears, A. Petrenko, G. Catelani, L. Sun, H. Paik, G. Kirchmair, L. Frunzio, L. Glazman, S. Girvin, and R. Schoelkopf, Physical Review B 86, 180504 (2012).
  * Ristè  _et al._ (2013) D. Ristè, C. Bultink, M. Tiggelman, R. Schouten, K. Lehnert, and L. DiCarlo, Nature communications 4, 1 (2013).
  * Serniak  _et al._ (2018) K. Serniak, M. Hays, G. de Lange, S. Diamond, S. Shankar, L. D. Burkhart, L. Frunzio, M. Houzet, and M. H. Devoret, Phys. Rev. Lett. 121, 157701 (2018).
  * Christensen  _et al._ (2019) B. Christensen, C. Wilen, A. Opremcak, J. Nelson, F. Schlenker, C. Zimonick, L. Faoro, L. Ioffe, Y. Rosen, J. DuBois, _et al._ , Physical Review B 100, 140503 (2019).
  * Wilen  _et al._ (2021) C. Wilen, S. Abdullah, N. Kurinsky, C. Stanford, L. Cardani, G. d’Imperio, C. Tomei, L. Faoro, L. Ioffe, C. Liu, _et al._ , Nature 594, 369 (2021).
  * Wu  _et al._ (2020) X. Wu, S. Tomarken, N. A. Petersson, L. Martinez, Y. J. Rosen, and J. L. DuBois, Physical Review Letters 125, 170502 (2020).
  * Liu  _et al._ (2017) T. Liu, Q.-P. Su, J.-H. Yang, Y. Zhang, S.-J. Xiong, J.-M. Liu, and C.-P. Yang, Scientific reports 7, 1 (2017).
  * Peterer  _et al._ (2015) M. J. Peterer, S. J. Bader, X. Jin, F. Yan, A. Kamal, T. J. Gudmundsen, P. J. Leek, T. P. Orlando, W. D. Oliver, and S. Gustavsson, Physical review letters 114, 010501 (2015).
  * Ramsey (1950) N. F. Ramsey, Physical Review 78, 695 (1950).
  * Vion  _et al._ (2002) D. Vion, A. Aassime, A. Cottet, P. Joyez, H. Pothier, C. Urbina, D. Esteve, and M. H. Devoret, Science 296, 886 (2002).
  * Tennant  _et al._ (2022) D. M. Tennant, L. A. Martinez, K. M. Beck, S. R. O’Kelley, C. D. Wilen, R. McDermott, J. L. DuBois, and Y. J. Rosen, PRX Quantum 3, 030307 (2022).
  * (25) “See supplemental material,” .
  * Koch  _et al._ (2007) J. Koch, M. Y. Terri, J. Gambetta, A. A. Houck, D. I. Schuster, J. Majer, A. Blais, M. H. Devoret, S. M. Girvin, and R. J. Schoelkopf, Physical Review A 76, 042319 (2007).
  * Cottet (2002) A. Cottet,  _Implementation of a quantum bit in a superconducting circuit_ , Ph.D. thesis, PhD Thesis, Université Paris 6 (2002).
  * Ithier  _et al._ (2005) G. Ithier, E. Collin, P. Joyez, P. Meeson, D. Vion, D. Esteve, F. Chiarello, A. Shnirman, Y. Makhlin, J. Schriefl, _et al._ , Physical Review B 72, 134519 (2005).
  * Martinis  _et al._ (2003) J. M. Martinis, S. Nam, J. Aumentado, K. Lang, and C. Urbina, Physical Review B 67, 094510 (2003).
  * Wangsness and Bloch (1953) R. K. Wangsness and F. Bloch, Physical Review 89, 728 (1953).
  * Makhlin and Shnirman (2003) Y. Makhlin and A. Shnirman, Journal of Experimental and Theoretical Physics Letters 78, 497 (2003).
  * Timmer and Koenig (1995) J. Timmer and M. Koenig, Astronomy and Astrophysics 300, 707 (1995).
  * Paik  _et al._ (2011) H. Paik, D. I. Schuster, L. S. Bishop, G. Kirchmair, G. Catelani, A. P. Sears, B. Johnson, M. Reagor, L. Frunzio, L. I. Glazman, _et al._ , Physical Review Letters 107, 240501 (2011).
  * Macklin  _et al._ (2015) C. Macklin, K. O’brien, D. Hover, M. Schwartz, V. Bolkhovsky, X. Zhang, W. Oliver, and I. Siddiqi, Science 350, 307 (2015).
  * Martinez  _et al._ (2020) L. A. Martinez, Y. J. Rosen, and J. L. DuBois, Physical Review A 102, 062426 (2020).