# A Few More Examples May Be Worth Billions of Parameters

Yuval Kirstain♠ Patrick Lewis†‡ Sebastian Riedel†‡ Omer Levy♠‡
♠ Tel-Aviv University
† University College London
‡ Facebook AI Research
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We investigate the dynamics of increasing the number of model parameters
versus the number of labeled examples across a wide variety of tasks. Our
exploration reveals that while scaling parameters consistently yields
performance improvements, the contribution of additional examples highly
depends on the task’s format. Specifically, in open question answering tasks,
enlarging the training set does not improve performance. In contrast,
classification, extractive question answering, and multiple choice tasks
benefit so much from additional examples that collecting a few hundred
examples is often “worth” billions of parameters. We hypothesize that unlike
open question answering, which involves recalling specific information,
solving strategies for tasks with a more restricted output space transfer
across examples, and can therefore be learned with small amounts of labeled
data.111Our code is publicly available: https://github.com/yuvalkirstain/lm-
evaluation-harness.

##  1 Introduction

Recent work on few-shot learning for natural language tasks explores the
dynamics of scaling up either the number of model parameters Brown et al.
(2020) or labeled examples Le Scao and Rush (2021), while controlling for the
other variable by setting it to a constant. For example, Brown et al. (2020)
focus on in-context learning from roughly 32 to 64 examples, a practice that
was adopted by fine-tuning approaches as well Schick and Schütze (2021b); Gao
et al. (2021b); Tam et al. (2021); however, there are many practical few-shot
scenarios where hundreds of examples can be collected at a relatively low
effort.222In SQuAD Rajpurkar et al. (2016), for example, the average
annotation pace is around one minute per question, producing 480 examples in a
single 8-hour workday. Other work experiments with single-size models Schick
and Schutze (2020); Ram et al. (2021); Le Scao and Rush (2021); Gao et al.
(2021b), even though larger (or smaller) models may exhibit different
behavior. Furthermore, much of the literature focuses on classification tasks
Schick and Schütze (2021a); Gao et al. (2021b); Le Scao and Rush (2021),
leaving it unclear whether their conclusions generalize to tasks with less
restricted output spaces.

(a)

(b)

Figure 1: Open QA tasks (e.g. TriviaQA) benefit from additional parameters
exclusively, while extractive QA tasks (e.g. SQuAD 2) benefit from both larger
models and more labeled data.

In this paper, we conduct a systematic exploration of few-shot learning for
language tasks, where we investigate the dynamics of increasing the number of
model parameters (using different sizes of the self-supervised T5 Raffel et
al. (2020)) versus the number of target-task labeled examples (from 32 to
2048) across a variety of tasks, including not only classification, but also
extractive, multiple-choice, and open question answering. Overall, we evaluate
192 scenarios by training 7,680 models to control for hyperparameters and
random seeds.

Our experiments show that, surprisingly, the contribution of additional
parameters versus additional labeled examples highly depends on the format of
the task. For open QA tasks, such as the open-domain version of Natural
Questions Kwiatkowski et al. (2019); Lee et al. (2019), which require the
model to recall specific information seen during pretraining, enlarging the
training set does not improve performance. By contrast, increasing the number
of model parameters results in substantial gains (see TriviaQA Joshi et al.
(2017) in Figure 1). Hence, when dealing with open QA, model parameters are of
immense value, and cannot be replaced by increasing the number of labeled
examples.

On the other hand, we observe a completely different trend for classification,
extractive QA, and multiple-choice tasks. These tasks benefit from enlarging
both the training set and the model (see SQuAD 2 Rajpurkar et al. (2018) in
Figure 1). We observe that hundreds of examples are often “worth” billions of
parameters; T5-L fine-tuned on 4 times more data is roughly competitive with
T5-XL, which has 4 times the number of parameters. Moreover, some tasks
benefit so much from labeled examples, that collecting even 512 data points
can make a fine-tuned T5-L (800M parameters) outperform GPT-3 (175B
parameters).

Finally, we hypothesize that unlike open QA, formats with restricted output
spaces have solving strategies (such as elimination) that can be learned from
small amounts of labeled data. This hypothesis also provides a possible
explanation as to why lean retrieve-and-read approaches (such as DrQA Chen et
al. (2017), ORQA Lee et al. (2019), and DPR Karpukhin et al. (2020))) appear
to be more robust than multi-billion-parameter closed-book models Roberts et
al. (2020) when tested on non-overlapping data Lewis et al. (2021).

##  2 Experiments

We describe the tasks (Section 2.1), models (Section 2.2), data regimes
(Section 2.3), and implementation details (Section 2.4) of our systematic
experiment suite. In total, we experiment with 12 tasks, 4 models, 4 data
regimes (with 5 samples each), and 8 hyperparameter configurations; these
amount to 7,680 trained models, evaluated across 192 task-model-data
scenarios.

###  2.1 Datasets

We experiment with 12 datasets, divided into 4 broad types of _task formats_.
The formats and their constituent tasks are described below.

#### Classification

In classification tasks, the model is expected to read a given text and
predict a single label from a small closed set, e.g. yes or no. We adopt
classification tasks from the GLUE Wang et al. (2018) and SuperGLUE Wang et
al. (2019) benchmarks, namely: Recognizing Textual Entailment (RTE, Dagan et
al., 2006; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al.,
2009), the Stanford Sentiment Treebank (SST-2, Socher et al., 2013), and BoolQ
Clark et al. (2019). We report accuracy for all classification datasets.

#### Extractive Question Answering

In extractive QA, the model is given a passage and a question, and is then
expected to produce an answer in the form of a span from the passage. We
experiment with SQuAD 2 Rajpurkar et al. (2018), HotpotQA Yang et al. (2018),
and DROP Dua et al. (2019). Each of these datasets contains an additional
“quirk” that makes it more challenging than the original SQuAD dataset
Rajpurkar et al. (2016), which popularized the extractive QA format: SQuAD 2
has unanswerable questions, HotpotQA provides multiple passages per question,
and DROP contains many arithmetic questions whose answer is not strictly
extractive, but can be derived from a set of spans in the given passage. For
all extractive QA datasets we report token-wise F1.

#### Multiple Choice

Multiple choice tasks provide the model with a question and several candidate
answers, with the goal of selecting the correct one. We focus on three
datasets in this format: the easy question set from the AI2 Reasoning
Challenge (ARC-E, Clark et al., 2018), the Physical Interaction Question
Answering dataset (PIQA, Bisk et al., 2020), and CommonsenseQA Talmor et al.
(2019). Unlike extractive QA, multiple choice tasks do not contain supporting
evidence (a passage) for answering the question, and in contrast to
classification, they have a different output space (candidate answers) for
each example. We report accuracy for all multiple choice datasets.

#### Open Question Answering

Open QA333We deviate from the widely-used term open-domain QA, which describes
the task, and use open QA instead to refer to the format, much like we use
extractive QA rather than reading comprehension. datasets provide the model
with just a question; no supporting evidence or closed candidate set is
available. We experiment with open-domain versions of Natural Questions
Kwiatkowski et al. (2019), TriviaQA Joshi et al. (2017), and SQuAD 1 Rajpurkar
et al. (2016). Our experiments here focus purely on the closed-book setting
Roberts et al. (2020), which does not allow models to retrieve text from an
external corpus, restricting them to information stored in their parameters.
For all open QA tasks we report F1 as the main metric.

###  2.2 Models

The Text-to-Text Transfer Transformer (T5, Raffel et al., 2020) uses an
encoder-decoder transformer architecture. It is pretrained on the task of
generating masked-out spans over the Colossal Clean Crawled Corpus (C4), which
contains 800GB of English-language text. We use version 1.1 of T5, which is
not trained on any labeled data. Our experiments include the 77M (S), 250M
(B), 800M (L), and 3B (XL) parameter variants of this model.

###  2.3 Training Data

While many publicly released datasets include an enormous number of labeled
examples Rajpurkar et al. (2018); Yang et al. (2018); Kwiatkowski et al.
(2019), and recent work on few-shot learning focus on an extreme scenario in
which less than one hundred examples are at hand Brown et al. (2020); Schick
and Schütze (2021a); Gao et al. (2021b), we choose to simulate a broader set
of practical scenarios where a small-to-medium dataset is available; in SQuAD
Rajpurkar et al. (2016), for example, the average annotation pace is around
one minute per question, producing 480 examples in a single 8-hour workday.
Therefore, we consider 4 dataset sizes for each task: 32, 128, 512, and 2048
examples. For each dataset size, we sample the relevant amount of examples
five times using different seeds, thus creating 20 datasets for each task
overall. We report the average score for each dataset size, thereby reducing
the high variance associated with training on small datasets.

###  2.4 Implementation

#### Code

For our implementation we extend EleutherAI’s language model evaluation
harness Gao et al. (2021a) to allow fine-tuning and evaluating additional
datasets and models.

#### Prompts

We adopt the prompts used by Brown et al. (2020) and Khashabi et al. (2020),
with minimal adaptations to T5 by adding a mask token followed by a period.
Following Le Scao and Rush (2021), we use prompts in conjunction with fine-
tuning.

#### Decoding

We use greedy decoding for extractive and open QA tasks. For classification
and multiple choice tasks, we compare the model’s probability for each
possible outcome, and predict the option with the highest probability. In
BoolQ, for example, we compare $P(\text{``yes''}|x)$ (the probability of the
positive class) with $P(\text{``no''}|x)$ (the probability of the negative
class), where $x$ is the prompt containing the context and the question.

#### Hyperparameters

To tune hyperparameters for fine-tuning, we split the available data into 75%
training and 25% validation (e.g. 24 training examples and 8 validation when
the dataset size is 32). For each case, we experiment with two learning rates
($5\mathrm{e}{-5}$, $5\mathrm{e}{-4}$) combined with linear decay, two weight
decays ($0.001$, $0.1$), and two values for amount of steps (512, 2048). The
effective batch size is always 32 examples. Additionally, we use a dropout
ratio of 0.1, gradient clipping is set to 1, and the amount of warmup steps is
determined by the maximum between 10% of the training steps and 100. We
evaluate each run after every epoch and choose the model with the lowest
validation loss for extractive and open QA tasks, and highest validation
accuracy for classification and multiple choice tasks.

##  3 Results

Classification |  Extractive QA |  Multiple Choice |  Open QA
---|---|---|---
|  |  |
|  |  |
|  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 2: Each heatmap displays the model’s performance (F1/accuracy) given
its size in parameters (horizontal axis) and the number of labeled examples
available during fine-tuning (vertical axis).

Multiple Choice |  Open QA
---|---
|
|
|

(a)

(b)

(c)

(d)

(e)

(f)

Figure 3: Converting multiple choice tasks (left column) to open QA (right
column) changes the scaling dynamics, replacing diagonal gradients
(performance improves with more parameters and more data) to horizontal
gradients (performance improves almost exclusively with more parameters).

Extractive QA |  Multiple Choice |  Open QA
---|---|---
|  |

(a)

(b)

(c)

Figure 4: Converting Natural Questions Kwiatkowski et al. (2019) from its open
QA format (right) to multiple choice (middle) and extractive QA (left) changes
the scaling dynamics, replacing horizontal gradients (performance improves
almost exclusively with more parameters) with diagonal gradients (performance
improves with more parameters and more data).

For each task in our experiment suite, we present a heatmap of the model’s
performance as a function of model size and the number of labeled examples.
These heatmaps expose that most tasks benefit from both larger models and more
training data, to a point where enlarging the dataset will result in similar
gains as increasing the number of parameters. However, this trend does not
apply to open QA tasks, whose performance _only_ improves with additional
model parameters (Section 3.1). Furthermore, we show that converting multiple
choice datasets to the open QA format disables the benefits of additional
training data, whereas converting in the opposite direction – from open QA to
extractive or multiple-choice QA – enables models to improve with more
examples (Section 3.2). Next, we describe a method for quantifying the
relative benefit from parameters versus training data, which confirms the
observed trends (Section 3.3). We then show that collecting a few hundred
examples allows the much smaller T5-L to outperform GPT3, but not in open QA
tasks, where the massive amount of parameters is the prime contributor
(Section 3.4). Finally, we suggest a hypothesis to explain the observed trends
(Section 3.5).

###  3.1 Main Trends

Figure 2 shows performance as a function of model size and dataset size per
task.444The results are available in tabular form in Appendix A. Visualizing
the results via heatmaps highlights two patterns: (1) diagonal gradients,
where performance significantly improves along both axes (though not
necessarily equally), and (2) horizontal gradients, where performance improves
almost exclusively along the horizontal (model size) axis. We observe that all
three open QA datasets exhibit horizontal gradients, while the remaining
datasets follow the diagonal patterns. We do not observe vertical gradients at
all, indicating that enlarging the model’s size is consistently beneficial.

Consider TriviaQA, for example (Figure 2, right column, second row);
performance approximately doubles when switching models from T5-B to T5-L (and
from T5-L to T5-XL), but changes by less than 2 points when increasing the
dataset from 32 examples to 2048. On the other hand, in the classification
task SST-2 (Figure 2, left column, second row), annotating 128 examples rather
than 32 examples results in double-digit improvements for T5-S and T5-B, and
in significant gains for larger models as well. Here, data-driven improvements
coincide with parameter-driven improvements, and increasing either factor
typically boosts performance. Moreover, the diagonal gradients show that in
many cases a model trained on more data can “catch up” with a larger model.
This trend is particularly striking when comparing T5-L with T5-XL, where
training the smaller model (T5-L) on 4 times more data is almost always
competitive with the larger model (T5-XL).

###  3.2 Same Dataset, Different Format

While a clear dichotomy arises from Section 3.1 with respect to format, it
might also result from the fact that the different datasets were collected and
annotated using different methodologies. Can we conduct a more controlled
experiment, which uses the same dataset but in different formats?

We first take the three multiple choice datasets (ARC-E, PIQA, and
CommonsenseQA) and convert them into the open QA format by excluding the
candidate answers from the input.555We control for the inference method by
selecting the most probable answer candidate, rather than applying greedy
decoding. Thus, the only difference between each pair of datasets is whether
or not the candidates appear in the input. Figure 3 shows that the diagonal
gradients clearly seen in the multiple choice format are replaced with
horizontal gradients similar to those of other open QA datasets.

We also examine data conversion in the opposite direction, by using multiple
choice and extractive QA versions of Natural Questions.666The original Natural
Question dataset Kwiatkowski et al. (2019) is in the extractive QA format;
specifically, we use the version in the 2019 MRQA Shared Task Fisch et al.
(2019). We filter the dataset to include only named entity answers that were
recognized using an off-the-shelf OntoNotes Named Entity Recognition model
from spaCy Hovy et al. (2006); Honnibal et al. (2020), and suggest them as
candidate answers alongside entities of the same type that appear in the
background passage. Here we control for the change in format by decoding the
multiple choice models as we do for extractive and open QA tasks and report
F1. Figure 4 shows that while the open QA heatmap displays largely horizontal
gradients, both extractive QA and multiple choice heatmaps follow the diagonal
patterns. Unlike the original open-domain Natural Questions dataset, we do
observe some minor improvement along the data axis in this entity-focused
version, but analyzing the data reveals that this stems from an increase in
example overlap Lewis et al. (2021), with 11.7% of test-set answers appearing
in the 2048-example training sets, compared to 8.5% in the original. Overall,
both experiments’ results indicate that the task’s format directly impacts
whether more labeled data will improve performance or not.

###  3.3 Quantifying the Relative Impact of Parameters versus Examples

For many tasks, both additional model parameters and labeled examples can
improve performance. However, it is not always clear how much each factor
contributes to greater performance gains with respect to the other. To
quantify the importance of increasing parameters versus examples, we compute a
regression-based metric using the numerical results in a given heatmap.
Specifically, we train the following linear regression model for each heatmap:

$\displaystyle y=\alpha_{m}x_{m}+\alpha_{d}x_{d}+b$

where $y$ is the model’s performance on the task, $x_{m}$ is the normalized
number of model parameters (S is 1, B is 2, L is 3, and XL is 4), and $x_{d}$
is the normalized number of dataset examples (32 is 1, 128 is 2, 512 is 3, and
2048 is 4). The regression coefficients $\alpha_{m},\alpha_{d}$ are scalars,
learnt for each task, which are then normalized to measure the relative impact
of each axis (parameters versus examples):

$\displaystyle I_{m}=\frac{|\alpha_{m}|}{|\alpha_{m}|+|\alpha_{d}|}$

When $0<I_{m}<0.5$, additional examples are greater contributors to
performance gains, while $0.5<I_{m}<1.0$ indicates that model parameters have
higher relative importance.

Figure 5 shows that most tasks lie between $0.4<I_{m}<0.7$, with model
parameters responsible for most performance improvements, but with significant
improvements arising from labeled data as well. However, all open QA tasks
deviate from this interval, and exhibit $I_{m}$ values of $0.9$ and above,
indicating that increased model parameters is almost exclusively responsible
for better performance.

Figure 5: The relative importance of parameters versus examples ($I_{m}$), as
computed via regression over each tasks’s heatmap. Higher values indicate more
dependence on parameters and less on labeled data.

###  3.4 Comparison with Massive Models

While models can benefit from both parameters and labeled data in many tasks,
scaling up language models to hundreds of billions of parameters may restrict
the ability to fine-tune, as GPT3-scale models are typically available only as
a service to most practitioners and researchers Brown et al. (2020). Given
this data-parameter trade-off, how many labeled examples are 175B parameters
worth?

We compare our results of T5-L (800M parameters) fine-tuned on various dataset
sizes to those of GPT3 (175B, over 200 times larger than T5-L) using in-
context learning, as reported by Brown et al. (2020).777Our work has 6
datasets that properly overlap with the original GPT3 paper. ARC-E and PIQA
are also used to evaluate GPT3, but in the open QA format. Table 1 shows a
wide performance gap between GPT3 and T5-L on open QA datasets, which cannot
be bridged by additional labeled examples, as observed in our main
experiments. However, for classification and extractive QA tasks, even a few
hundred labeled examples are often enough for T5-L to catch up with GPT3’s
performance and even exceed it. In BoolQ, for example, just collecting 96
additional examples is tantamount to adding 200 times more parameters to the
model. This result demonstrates that while performance may improve along the
parameter axis in classification and extractive QA tasks, a small amount of
labeled training data can also go a long way.

Model | #Examples | Classification | Extractive QA | Open QA
---|---|---|---|---
RTE | BoolQ | SQuAD 2 | DROP | NQ | TriviaQA
T5-L | 32 | 68.2 | 65.0 | 39.4 | 23.8 | 5.1 | 12.0
128 | 72.9 | 77.0 | 55.3 | 26.5 | 5.8 | 11.5
512 | 79.1 | 82.5 | 70.5 | 31.2 | 5.6 | 12.1
2048 | 86.3 | 85.4 | 78.9 | 37.7 | 6.3 | 10.7
GPT3 |  $\leq$64 | 72.9 | 77.5 | 69.8 | 36.5 | 29.9 | 71.2
Table 1: A comparison between GPT3 (with in-context learning, as reported by
Brown et al. (2020)) and T5-L. Figures in bold represent T5-L configurations
that outperform GPT3. For a fair comparison with Brown et al. (2020), we
report accuracy (exact match) for open QA tasks in this table.

###  3.5 Discussion

Why does changing the task’s input format have such a dramatic effect on the
training dynamics? We conjecture that the format changes described in our
experiments, which effectively remove information from the input, force the
models to supplement that information with knowledge stored in its parameters.
For example, when asking What is the capital of Micronesia? in the open QA
format, the model is required to know that the answer is Palikir by
encountering the fact during pretraining or fine-tuning on a paraphrase of the
same question. In contrast, if the same question is asked in the multiple
choice format, and the options are (1) Rome, (2) Tokyo, (3) Yaren, (4)
Palikir, the model can easily eliminate the more frequently-mentioned capitals
of Rome and Tokyo, and then guess between the two remaining options, Yaren
(the capital of neighboring Nauru) and Palikir (the correct answer). A similar
example can be constructed for extractive QA, where the vast majority of
passage spans can be pruned a priori, leaving only a handful of named entities
as more likely candidates. We hypothesize that answering strategies, such as
elimination, can indeed be learned from small-medium training sets, while
actual new facts cannot, unless there is significant train-test overlap Lewis
et al. (2021).

A practical corollary of this hypothesis is that if one can modify a target
task from the open QA format to one with a more limited output space, à la
multiple choice or extractive QA, they would unlock the ability to trade data
for parameters. Instead of relying on massive pretrained language models,
which can only be used as a service, one could achieve competitive and even
superior results with a much smaller model, given a relatively small dataset
of several hundred labeled examples. Retrieve-and-read approaches successfully
demonstrate this notion by decomposing open QA into two separate
classification and extractive QA subproblems Chen et al. (2017); Lee et al.
(2019); Karpukhin et al. (2020), and may possibly be applied to few-shot
scenarios in additional tasks via more general retrieve-and-generate models
such as RAG Lewis et al. (2020).

##  4 Related Work

Few-shot learning has been a subject of interest for several decades Thrun and
Pratt (1998); Fink (2005); Li et al. (2006); Vinyals et al. (2016); Jiang et
al. (2018). Within NLP, “few-shot learning” has traditionally focused on
quickly learning new classes and domains within the context of a single task
(see Yin (2020) for a recent survey). Recently, there has been a surge of
interest in few-shot learning, following the release of GPT3 Brown et al.
(2020). Here, the few-shot learning paradigm has shifted subtly, and refers to
building models capable of tackling a range of standard NLP tasks, albeit
using very restricted training sets, usually sub-sampled from the full
training set. A great deal of work has recently been produced in this area,
and we provide a necessarily incomplete summary below.

#### In-Context Learning

In-context learning Brown et al. (2020) generally refers to adapting to a task
by providing training examples as additional textual input, without performing
gradient-based updates. This technique imposes a limit on size of the training
dataset due to context length limits. Recent work from Liu et al. (2021a) and
Lu et al. (2021) demonstrate that the choice of in-context training examples,
and the order in which they are presented have large effects on performance.

#### Prompt-Based Learning

Prompting refers to providing additional input to a model designed to help it
to produce correct outputs. Typically, these take the form of textual
templates used to form cloze questions, and have been used in a variety of
settings, such as probing Petroni et al. (2019) and zero-shot learning Radford
et al. (2019). Prompts can be used in conjunction with fine-tuning, which has
been shown to improve results in a number of works Schick and Schutze (2020,
2021); Schick and Schütze (2021a); Gao et al. (2021b); Le Scao and Rush
(2021); Tam et al. (2021). We adopt this technique in our experiments and
adapt the models using prompt-based fine-tuning.

#### Prompt Engineering

Models may be sensitive to the choice of prompt (especially without fine-
tuning), and a number of works attempt to optimize the prompt for the task at
hand Jiang et al. (2020); Shin et al. (2020). Recently, a number of works have
also proposed generalizing prompts to include task-specific parameters and
embeddings, typically learnt via gradient descent while keeping parts or all
of the model’s parameters frozen Houlsby et al. (2019); Liu et al. (2021b);
Zhong et al. (2021); Qin and Eisner (2021); Li and Liang (2021); Lester et al.
(2021); Logan et al. (2021). While these techniques can improve results for
frozen models, they generally do not outperform fine-tuning the whole model
Lester et al. (2021), hence we choose to focus on full-model finetuning with
standard prompts in our experiments.

#### Few-Shot Learning Analysis

Closest to our contribution are works placing an emphasis on the analysis of
few-shot model behaviour, rather than focusing on schemes to improve
performance. Le Scao and Rush (2021) quantify the benefit of prompting in few-
shot learning, and Perez et al. (2021) critically discuss the difficulty of
model selection and very low dataset sizes in few-shot learning. Our work is
complementary, exploring the relationship between scale, dataset size, and
task open-endedness.

#### Task Formats

Another important aspect of our work is the investigation of learning as a
function of task format. Related work in this area includes research
investigating reformulating a task into a different format, such as reducing
tasks to NLI White et al. (2017); Wang et al. (2018) or reading comprehension
Levy et al. (2017); Wu et al. (2020), or even reducing all tasks to a single
format Kumar et al. (2016); McCann et al. (2018). A related line of work seeks
to understand tasks and datasets by changing or removing parts of the input,
and, in-so-doing, changing the task format. Examples include hypothesis-only
NLI baselines Gururangan et al. (2018); Poliak et al. (2018), and document-
only baselines in Reading Comprehension Kaushik and Lipton (2018); Sugawara et
al. (2020). We also change the available input to a model for a given task,
effectively changing the task format, while keeping the targets unchanged. We
do this to measure the effect of the open-endedness of a task on sample
complexity for differently sized models.

##  5 Conclusions

In this work, we present an empirical investigation on the relationships
between (1) a task’s format, (2) the number of labeled examples available for
said task, and (3) the number of parameters the model tackling the task has.
Through our extensive experiments, we determine that task format greatly
affects the relative performance improvement that can be expected from
increased training set size and parameter count. For tasks that do not require
the recollection of specific external information – i.e. classification,
multiple choice, and extractive QA – we find that more labeled data and larger
models both reliably improve performance. In fact, for some of these tasks,
adding a few hundred labeled examples is _more_ beneficial than scaling up the
model size by billions of parameters. It seems then, from a practitioner’s
perspective, that for many tasks where data is very sparse, the tried-and-true
strategy of simply collecting more training data will often be a more
effective strategy than attempting to scale to larger, more computationally-
demanding models. However, the picture is very different for open QA tasks;
for such tasks, we find that increasing the size of the training data barely
improves performance, leaving parameter inflation as the only reliable
approach to improve accuracy. Finally, we provide a hypothesis to explain
these results and conclude with a practical corollary – when possible,
changing the format from open QA into a more “self-contained” one will allow
labeled data to bridge performance gaps between moderately-sized models and
much larger ones.

## Acknowledgments

We thank Avia Efrat and Ori Ram for valuable feedback and discussions.

## References

  * Bar-Haim et al. (2006) Roy Bar-Haim, Ido Dagan, Bill Dolan, L. Ferro, Danilo Giampiccolo, and B. Magnini. 2006.  The second pascal recognising textual entailment challenge.
  * Bentivogli et al. (2009) L. Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009.  The sixth pascal recognizing textual entailment challenge.  In _TAC_.
  * Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020.  Piqa: Reasoning about physical commonsense in natural language.  In _AAAI_.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _ArXiv_.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _ArXiv_ , abs/1803.05457.
  * Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006.  The pascal recognising textual entailment challenge.  In _Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment_ , pages 177–190, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2368–2378, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fink (2005) Michael Fink. 2005.  Object classification from a single example utilizing class relevance metrics.  In _Advances in Neural Information Processing Systems_ , volume 17. MIT Press.
  * Fisch et al. (2019) Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019\.  MRQA 2019 shared task: Evaluating generalization in reading comprehension.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 1–13, Hong Kong, China. Association for Computational Linguistics.
  * Gao et al. (2021a) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021a.  A framework for few-shot language model evaluation.
  * Gao et al. (2021b) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021b.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830, Online. Association for Computational Linguistics.
  * Giampiccolo et al. (2007) Danilo Giampiccolo, B. Magnini, Ido Dagan, and W. Dolan. 2007.  The third pascal recognizing textual entailment challenge.  In _ACL-PASCAL@ACL_.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spaCy: Industrial-strength Natural Language Processing in Python.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR.
  * Hovy et al. (2006) Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006.  Ontonotes: The 90  In _Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers_ , NAACL-Short ’06, page 57–60, USA. Association for Computational Linguistics.
  * Jiang et al. (2018) Xiang Jiang, Mohammad Havaei, Gabriel Chartrand, Hassan Chouaib, Thomas Vincent, Andrew Jesson, Nicolas Chapados, and Stan Matwin. 2018.  On the importance of attention in meta-learning for few-shot text classification.  _CoRR_ , abs/1806.00852.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020.  How can we know what language models know? _Transactions of the Association for Computational Linguistics_ , 8:423–438.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Kaushik and Lipton (2018) Divyansh Kaushik and Zachary C. Lipton. 2018.  How much reading does reading comprehension require? a critical investigation of popular benchmarks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5010–5015, Brussels, Belgium. Association for Computational Linguistics.
  * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020.  UNIFIEDQA: Crossing format boundaries with a single QA system.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1896–1907, Online. Association for Computational Linguistics.
  * Kumar et al. (2016) Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016.  Ask me anything: Dynamic memory networks for natural language processing.  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ICML’16, page 1378–1387. JMLR.org.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Le Scao and Rush (2021) Teven Le Scao and Alexander Rush. 2021.  How many data points is a prompt worth? In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2627–2636, Online. Association for Computational Linguistics.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  _CoRR_ , abs/2104.08691.
  * Levy et al. (2017) Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017.  Zero-shot relation extraction via reading comprehension.  In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_ , pages 333–342, Vancouver, Canada. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 9459–9474. Curran Associates, Inc.
  * Lewis et al. (2021) Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021.  Question and answer test-train overlap in open-domain question answering datasets.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1000–1008, Online. Association for Computational Linguistics.
  * Li et al. (2006) Fei-Fei Li, R. Fergus, and P. Perona. 2006.  One-shot learning of object categories.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 28:594–611.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Liu et al. (2021a) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, L. Carin, and Weizhu Chen. 2021a.  What makes good in-context examples for gpt-3?  _ArXiv_ , abs/2101.06804.
  * Liu et al. (2021b) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021b.  GPT understands, too.  _CoRR_ , abs/2103.10385.
  * Logan et al. (2021) IV Robert L. Logan, Ivana Balavzevi’c, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021.  Cutting down on prompts and parameters: Simple few-shot learning with language models.  _ArXiv_ , abs/2106.13353.
  * Lu et al. (2021) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021\.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  _CoRR_ , abs/2104.08786.
  * McCann et al. (2018) Bryan McCann, N. Keskar, Caiming Xiong, and R. Socher. 2018.  The natural language decathlon: Multitask learning as question answering.  _ArXiv_ , abs/1806.08730.
  * Perez et al. (2021) Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021.  True few-shot learning with language models.  _CoRR_ , abs/2105.11447.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Poliak et al. (2018) Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018.  Hypothesis only baselines in natural language inference.  In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_ , pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics.
  * Qin and Eisner (2021) Guanghui Qin and Jason Eisner. 2021.  Learning how to ask: Querying lms with mixtures of soft prompts.  _CoRR_ , abs/2104.06599.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Ram et al. (2021) Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. 2021.  Few-shot question answering by pretraining span selection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3066–3079, Online. Association for Computational Linguistics.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
  * Schick and Schutze (2020) Timo Schick and H. Schutze. 2020.  Few-shot text generation with pattern-exploiting training.  _ArXiv_ , abs/2012.11926.
  * Schick and Schutze (2021) Timo Schick and H. Schutze. 2021.  It’s not just size that matters: Small language models are also few-shot learners.  _ArXiv_ , abs/2009.07118.
  * Schick and Schütze (2021a) Timo Schick and Hinrich Schütze. 2021a.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 255–269, Online. Association for Computational Linguistics.
  * Schick and Schütze (2021b) Timo Schick and Hinrich Schütze. 2021b.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2339–2352, Online. Association for Computational Linguistics.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020.  AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4222–4235, Online. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Sugawara et al. (2020) Saku Sugawara, Pontus Stenetorp, Kentaro Inui, and Akiko Aizawa. 2020.  Assessing the benchmarking capacity of machine reading comprehension datasets.  In _AAAI_ , pages 8918–8927. AAAI Press.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Tam et al. (2021) Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021\.  Improving and simplifying pattern exploiting training.  _ArXiv_ , abs/2103.11955.
  * Thrun and Pratt (1998) Sebastian Thrun and Lorien Pratt. 1998.  _Learning to Learn: Introduction and Overview_ , page 3–17. Kluwer Academic Publishers, USA.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. 2016.  Matching networks for one shot learning.  In _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _NeurIPS_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * White et al. (2017) Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017\.  Inference is everything: Recasting semantic resources into a unified evaluation framework.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 996–1005, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Wu et al. (2020) Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020.  CorefQA: Coreference resolution as query-based span prediction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6953–6963, Online. Association for Computational Linguistics.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Yin (2020) Wenpeng Yin. 2020.  Meta-learning for few-shot natural language processing: A survey.  _CoRR_ , abs/2007.09604.
  * Zhong et al. (2021) Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021.  Factual probing is [MASK]: learning vs. learning to recall.  _CoRR_ , abs/2104.05240.

##  Appendix A Tabular Results

Table 2 provides the results from our main experiment (Section 3, Figure 2) in
tabular form.

Model | #Examples | Classification | Extractive QA | Multiple Choice | Open QA
---|---|---|---|---|---
RTE | SST | BoolQ | SQuAD2 | HPQA | DROP | ARC-E | PIQA | CSQA | NQs | TQA | SQuAD1
T5-S | 32 | 53.8 | 65.4 | 57.3 | 37.2 | 28.1 | 6.0 | 29.1 | 55.2 | 21.4 | 1.9 | 4.8 | 3.3
128 | 50.9 | 81.5 | 58.0 | 40.5 | 36.5 | 9.1 | 31.5 | 52.9 | 22.4 | 4.5 | 5.8 | 3.9
512 | 52.0 | 84.1 | 58.3 | 42.5 | 52.1 | 15.1 | 32.1 | 55.1 | 28.5 | 3.9 | 5.8 | 3.5
2048 | 59.6 | 88.0 | 64.9 | 51.1 | 61.8 | 18.9 | 40.7 | 57.3 | 36.0 | 4.4 | 5.7 | 5.0
T5-B | 32 | 53.1 | 73.0 | 58.8 | 34.7 | 48.3 | 10.2 | 31.4 | 52.1 | 26.0 | 5.3 | 10.3 | 7.9
128 | 55.1 | 91.9 | 58.5 | 51.1 | 58.4 | 11.8 | 33.0 | 54.9 | 21.8 | 7.8 | 9.7 | 6.7
512 | 63.5 | 90.5 | 71.2 | 49.6 | 71.1 | 23.2 | 34.8 | 55.0 | 42.0 | 7.7 | 11.0 | 8.5
2048 | 67.5 | 92.9 | 70.4 | 59.8 | 72.7 | 24.5 | 47.7 | 59.6 | 54.5 | 7.5 | 9.6 | 9.0
T5-L | 32 | 68.2 | 84.5 | 65.0 | 39.4 | 71.1 | 23.8 | 42.9 | 55.6 | 41.7 | 10.6 | 17.7 | 11.1
128 | 72.9 | 92.4 | 77.0 | 55.3 | 74.9 | 26.5 | 55.2 | 56.6 | 57.8 | 11.8 | 17.4 | 12.0
512 | 79.1 | 94.2 | 82.5 | 70.5 | 78.2 | 31.2 | 65.1 | 68.6 | 64.3 | 11.6 | 18.7 | 12.6
2048 | 86.3 | 95.1 | 85.4 | 78.9 | 79.7 | 37.7 | 72.6 | 72.6 | 72.7 | 12.0 | 17.1 | 13.0
T5-XL | 32 | 63.7 | 82.3 | 76.5 | 56.4 | 67.4 | 30.1 | 61.4 | 56.2 | 52.4 | 15.4 | 26.2 | 15.4
128 | 83.4 | 91.5 | 83.0 | 73.2 | 76.9 | 32.5 | 72.2 | 59.2 | 64.8 | 16.4 | 27.3 | 15.0
512 | 84.1 | 95.2 | 85.9 | 77.7 | 80.2 | 36.8 | 76.9 | 75.5 | 74.0 | 16.1 | 27.5 | 15.8
2048 | 88.4 | 96.0 | 88.0 | 84.8 | 81.9 | 43.6 | 80.9 | 81.1 | 78.5 | 16.0 | 25.2 | 15.7
Table 2: The performance (F1/accuracy) of different T5 models fine-tuned on
different training set sizes, across 12 different datasets.


# Topological Signature of Stratospheric Poincaré – Gravity Waves

###### Abstract

The rotation of the earth breaks time-reversal and reflection symmetries in an
opposite sense north and south of the equator, leading to a topological origin
for certain atmospheric and oceanic equatorial waves. Away from the equator
the rotating shallow water and stably stratified primitive equations exhibit
Poincaré inertio-gravity waves that have nontrivial topology as evidenced by
their strict superinertial timescale and a phase singularity in frequency-
wavevector space. This non-trivial topology then predicts, via the principle
of bulk-interface correspondence, the existence of two equatorial waves along
the equatorial interface, the Kelvin and Yanai waves. To directly test the
nontrivial topology of Poincaré-gravity waves in observations, we examine ERA5
reanalysis data and study cross-correlations between the wind velocity and
geopotential height of the mid-latitude stratosphere at the 50 hPa height. We
find the predicted vortex and anti-vortex in the relative phase of the
geopotential height and velocity at the high frequencies of the waves. By
contrast, lower-frequency planetary waves are found to have trivial topology
also as expected from theory. These results demonstrate a new way to
understand stratospheric waves, and provide a new qualitative tool for the
investigation of waves in other components of the climate system.

##  1 Introduction

Much of what we understand about the climate system is made possible by
recognizing the importance of waves in Earth’s atmosphere and oceans. Waves
are characterized by predictable periodic motion that contrasts with the  more
irregular or even chaotic behavior displayed by many other components of the
climate system. Waves are  very often categorized dynamically by matching
observed variability to predicted dispersion relations (e.g., Wheeler and
Kiladis, 1999; Farrar, 2008), which relate the frequencies and spatial scales
where waves may occur and how they propagate through space and time. The phase
of waves receive rather less frequent attention. Phasing in the meteorological
context refers to different disturbances such as fronts and low-level jets
coming together in space and time. Another example in the context of
baroclinic instability theory is counterpropagating Rossby waves (Hoskins et
al., 1985).  The phase spectrum of convectively generated gravity waves in the
tropics has been studied recently using ERA5 reanalysis data (Pahlavan et al.,
2023). Some rogue ocean waves may result from the constructive superposition
of multiple smaller waves. Here we demonstrate a different distinguishing and
qualitative feature of certain waves, nontrivial topology, that can also be
discerned from observations of relative wave phase. The nontrivial topology
found here in reanalysis data of stratospheric waves agrees with a recent
theoretical prediction of Delplace et al. (2017).

Remarkably, oceanic and atmospheric waves share fundamental physics with those
in quantum matter, and topology plays an important role in the movement of the
atmosphere and oceans. Although the basic equations for idealized equatorial
plane waves in the atmosphere and ocean have been long known (see the
historical perspective in Hendershott and Munk, 1970) and wavelike variability
has been observed in the atmosphere (Wheeler and Kiladis, 1999) and ocean
(Farrar, 2008) with the waves linked to phenomena such as El Niño (Wyrtki,
1975) and the Madden-Julian Oscillation (Madden and Julian, 1971), the
connection of waves to topology has only recently been discovered (Delplace et
al., 2017). This discovery presents an opportunity to use more sophisticated
statistical and mathematical analysis of the wave-like aspects of these
phenomena, sharpening our insights into their emergence from diverse
variability, fundamental mechanisms, and especially the ability of our
modeling systems to appropriately simulate them.

Figure 1:  (a) Dispersion relation in frequency-wavevector space for the
rotating shallow water equations in the f-plane approximation as a function of
latitude. The upper and lower bands are positive and negative superinertial
frequency modes of the Poincaré waves, and the color indicates the sign of the
winding number of the upper band (blue = -1, red = +1) as shown by the plots
of the relative phase difference between the height and meridional velocity in
the lower half of the figure (argument of $\Xi(k,\ell)$; see text). At the
equator $f=0$, the frequency gap vanishes at a Weyl point, and a topological
transition occurs (purple) as the two bands invert. The subinertial range has
only a zero frequency band (black) containing modes in geostrophic balance as
there is no $\beta$ effect in the f-plane approximation. The inset (b) shows
the equatorial $\beta$-plane dispersion relations with the subinertial quasi-
geostrophic Rossby waves, the superinertial Poincaré inertio-gravity waves,
and the Kelvin and Yanai waves. Vector plots correspond to the southern
hemisphere (left) and the northern hemisphere (right) respectively.

Topology is the field of mathematics concerned with the properties of spaces
that do not change under continuous deformations. A doughnut and an orange are
topologically distinct because the doughnut has a single hole (we say that it
has a genus of 1) while the orange has a genus of 0. Likewise a coffee mug has
the doughnut topology because it too has a single hole (the handle). Another
example, more closely connected to the phenomena discussed below, is known as
the Hairy Ball or Hedgehog Theorem which says that it is impossible to comb
the spines of a hedgehog (because there will always be at least one tuft)
(Renteln, 2013). By contrast, if hedgehogs had a toroidal shape instead of
spherical, their spines could be combed smoothly without any bald spots – but
probably only boring hedgehogs would choose that hairstyle.

Topology is a powerful tool because it turns  certain aspects of complicated
problems into simple ones.  Changes in a model or physical system that do not
alter the topology leave certain qualitative properties unchanged. For
example, across an interface between topologically distinct states of matter,
a general principle known as bulk-boundary or bulk-interface correspondence
(Hasan and Kane, 2010; Ozawa et al., 2019; Simon, 2021) guarantees the
existence of boundary or interfacial waves (Fig. 1 (a))  and these waves
persist even when the model or physical system is deformed. The waves move in
one direction and evidence _topological protection_ , or immunity to
backscattering even in the presence of defects. On a rotating planet, the
equator acts as the boundary between two topologically-distinct hemispheres,
and so some equatorial waves propagate only in one direction and are
topologically protected (Fig. 1 (b)). Past uses of topology in fluid mechanics
have usually focused on coherent structures in space such as vortices. Here by
contrast we study topology in wavevector-frequency space where it can for
instance guarantee the existence of eastward propagating equatorial waves in
Earth’s climate system (Delplace et al., 2017). In particular there is a
topological origin for two well-known equatorially trapped waves, the Kelvin
and Yanai modes, caused by the breaking of time-reversal symmetry by Earth’s
rotation, that helps to explain the robustness of these waves against
buffeting by the weather. This resilience may also be implicated in other
emergent equatorial wavelike phenomena such as the Madden-Julian Oscillation
(MJO), remarkable for its eastward propagation along the equator.

In this paper we study the topology of waves away from the equatorial region
to ascertain whether or not topological signatures can be deduced from
observation, and whether or not the signatures found are consistent with
theoretical prediction and with bulk-interface correspondence. We first
briefly review the theory and refer the reader to background material (Hasan
and Kane, 2010; Delplace et al., 2017; Ozawa et al., 2019; Simon, 2021; Zhu et
al., 2023) for more details.

Fig. 1a shows the dispersion relation for the idealized rotating shallow-water
model on the f-plane. There are three distinct bands: Positive and negative
frequency Poincaré-gravity waves (often referred to as “inertio-gravity
waves”) and a zero-frequency geostrophically balanced mode. The topology of
each band is distinct and may be quantified in terms of a winding number
(defined below) in frequency-wavevector space (Horsley, 2022). In particular,
the Poincaré-gravity modes are characterized by a vortex with a winding number
of $\pm 1$. A winding number of $+1$ means that the phase of the complex-
valued quantity $\Xi(k,\ell)$ (defined below) increases by $2\pi$ as one moves
around the center of the vortex in a counter-clockwise direction in wavevector
space. The sign of the winding number depends on the sign of the product of
the Coriolis parameter and the wave frequency. Geostrophically balanced modes
(that become Rossby waves once the Coriolis parameter is allowed to vary with
latitude) are by contrast topologically trivial with zero winding number.

Evidently the winding number of the Poincaré-gravity waves changes by $2$ upon
crossing the equator where the Coriolis parameter changes sign; this is known
as “band inversion.” By bulk-interface correspondence, there must therefore be
2 waves that traverse the otherwise forbidden region of frequency space.
Spectral flow (Faure, 2023) in frequency-wavevector space as the zonal
wavenumber increases show that the negative frequency Poincaré band loses two
modes, the geostrophic band gains and loses one mode, and the positive
frequency Poincaré band gains the two modes. These two modes are the
equatorial Kelvin and Yanai (also known as mixed Rossby-gravity) waves. The
two equatorial modes move with an eastward group velocity at all zonal
wavenumbers, and this unidirectional propagation reflects the breaking of
time-reversal invariance by the planetary rotation.

The organization of the rest of the paper is as follows. In Sec. 2 we review
the mathematics of topology in the context of the rotating shallow water
equations on the f-plane. Sec. 3 discusses the atmospheric ERA5 reanalysis
data and its processing into spectral space. Topological properties of waves
discerned from the data are presented in Sec. 4 and some conclusions are
presented in Sec. 5. The Matlab code that we use is included  as supplementary
material.

##  2 Theoretical Motivation

To motivate our investigation of the topology of waves in the stratosphere we
briefly review the linearized rotating shallow water equations on the f-plane
following Delplace et al. (2017).

$\begin{cases}\frac{\partial u}{\partial t}-f_{0}v=-g\frac{\partial
h}{\partial x}\\\ \frac{\partial v}{\partial t}+f_{0}u=-g\frac{\partial
h}{\partial y}\\\ \frac{\partial h}{\partial t}+H(\frac{\partial u}{\partial
x}+\frac{\partial v}{\partial y})=0\ .\end{cases}$ (1)

Here $u$ and $v$ are the zonal and meridional velocity, $f_{0}$ is the
Coriolis parameter in the f-plane approximation and $g$ is the gravitational
acceleration. Also $h$ is the height anomaly and $H$ is the average depth of
the shallow water. Despite the highly idealized nature of these equations, the
robustness of topology against continuous deformations means that lessons
learned from these equations may be expected to hold in more realistic
settings (Zhu et al., 2023).

By adopting periodic boundary conditions to eliminate any boundaries, the
normal modes of the linearized rotating shallow water equations may be easily
found by Fourier transformation to frequency-wavevector space followed by
diagonalization of a $3\times 3$ matrix. Introducing the 3-component vector
$\Psi(x,y,t)\equiv(u(x,y,t),~{}v(x,y,t),~{}h(x,y,t))$ and substituting
$\Psi(x,y,t)=\Psi(k,\ell,\omega)~{}e^{i(\vec{k}\cdot\vec{x}-\omega t)}$ with
$\vec{k}=(k,\ell)$ and $\omega=2\pi\nu$ which defines the angular frequency
$\omega$ in terms of the frequency $\nu$ we obtain

$\begin{bmatrix}-i\omega&-f_{0}&igk\\\ f_{0}&-i\omega&ig\ell\\\
iHk&iH\ell&-i\omega\end{bmatrix}~{}\left[\begin{array}[]{c}u\\\ v\\\
h\end{array}\right]=0\ .$ (2)

This secular equation is solved to obtain three normal mode angular
frequencies. In the long-wavelength limit ($f_{0}^{2}\gg gH(k^{2}+\ell^{2}))$,
we recover inertial oscillations of $\omega\approx\pm f_{0}$ and the
amplitudes of the positive and negative frequency Poincaré-gravity waves,  and
the zero mode, are:

$\Psi_{\pm}=\left[\begin{array}[]{c}\pm(k\pm
i\ell\operatorname{sgn}(f_{0}))|f_{0}|\\\ -i(k\pm
i\ell\operatorname{sgn}(f_{0}))f_{0}\\\ H(k^{2}+\ell^{2})\end{array}\right],\
{\color[rgb]{0,0,0}\Psi_{0}=\left[\begin{array}[]{c}-g\ell\\\ gk\\\
-if_{0}\end{array}\right]}$ (3)

up to an overall normalization. Note that
$\Psi^{*}_{\pm}(\vec{k},f_{0})=\Psi_{\pm}(\vec{k},-f_{0})$ whereas the
positive and negative frequency modes are related by band inversion
$\Psi_{-}(\vec{k},f_{0})=\Psi_{+}(-\vec{k},-f_{0})$. The presence of the $k\pm
i\ell\operatorname{sgn}(f_{0})$ term  for the Poincaré-gravity amplitudes
signals the appearance of a phase singularity at the origin in wavevector
space.

Normal modes are defined only up to an overall phase and magnitude. To
quantify their topology we follow Zhu et al. (2023) by introducing a gauge-
invariant but complex-valued quantity $\Xi$ defined as follows:

$\displaystyle\Xi(k,\ell)\equiv h^{*}(k,\ell)~{}v(k,\ell).$ (4)

We say that $\Xi$ is gauge-invariant because the overall phase of the normal
modes cancels out; only the relative phase difference between two components
of the waves (in this case, meridional velocity and height) remains. A Fourier
transform of $\Xi$ into real space and time reveals it to be a cross-
correlation between two field points that are separated in space and time. As
waves propagate simultaneously in both space and time it is perhaps not
surprising that non-local correlations play a role in their topological
classification.

In the case of the inertial Poincaré-gravity waves the gauge-invariant
quantity displays a vortex or antivortex (depending on the signs of the
frequency and the Coriolis frequency) centered at the origin in wavevector
space:

$\Xi_{\pm}(k,\ell)=-iHf_{0}(k^{2}+\ell^{2})(k\pm
i\ell\operatorname{sgn}(f_{0}))\ .$ (5)

As noted above, the winding number may be found from the change in phase of
$\Xi(k,\ell)$. A winding number of $+1$ ($-1)$ means that the phase of a
complex-valued quantity decreases (increases) by $2\pi$ as one moves around
the center of the vortex in a clockwise (counter-clockwise) direction in
wavevector space. In the northern hemisphere, $f_{0}>0$, and the positive-
frequency waves have winding number $+1$ whereas negative-frequency waves have
winding number $-1$; these topological charges are reversed in the southern
hemisphere (the bands invert). The related product $h^{*}(k,\ell)~{}u(k,\ell)$
involving the zonal velocity exhibits the same winding numbers. The zero-
frequency geostrophic mode, by contrast, has (up to multiplication by a real-
valued normalization constant) the following form:

$\Xi_{0}(k,\ell)=igf_{0}k$ (6)

and thus has a domain wall at $k=0$ and zero winding number. Its topological
charge therefore vanishes. The same result may be obtained from the barotropic
quasigeostrophic equations as the wave height is proportional to the stream
function while the meridional velocity is proportional to the zonal derivative
of the stream function thus leading in wavevector space to the same form as
Eq. 6.

Figure 2: Theoretical calculation of the cross-correlation $\Xi(k,\ell)$ for
normal modes at (a)-(b) low frequencies (Rossby waves) and (c)-(d) high
frequencies (Poincaré-gravity waves). Here, we perform the calculation for the
northern hemisphere with a spatially varying Coriolis parameter to lift the
degeneracy of the geostrophic mode: $f(y)=1+0.5\sin(2\pi y/L_{y})$, where
$L_{y}=10\pi$ is the domain size. See Zhu et al. (2023) for details. Plots (a)
and (c) show the negative frequency modes, and (b) and (d) are for positive
frequencies. The color bar represents the magnitude of $\Xi$ and the arrows
show the phase of its complex value. The low-frequency planetary waves have a
domain wall at $k=0$ and are topologically trivial as a closed path that
avoids the $\Xi=0$ domain wall has zero winding number (black oval). While the
cross-correlation structure of the geostrophic mode in (a)-(b) depends on the
choice of normal modes, the winding number remains zero for all modes. Note
that the winding number of the high-frequency Poincaré-gravity waves (black
circle in the bottom figure) depends on the sign of the frequency of the
waves.

Representing the phase of $\Xi$ with an arrow makes these patterns evident as
shown in Figure 2. In Delplace et al. (2017) and many other papers, linearized
wave equations have been characterized in terms of the topological Chern
number. However the Chern number has several disadvantages. First, in contrast
to systems on spatial lattices (where the Chern number was first applied), for
continuous systems  the Chern number when calculated as an integral of the
Berry curvature need not be integer and its value depends on how an integral
over the Berry curvature is regularized at high wavevectors. This ambiguity
can sometimes be avoided by compactification (Delplace et al., 2017; Venaille
and Delplace, 2021). Here we point out that this is more of a mathematical
problem than a physical one because at small scales dissipation is important
and ultimately at the smallest scales the fluid description breaks down
entirely and is replaced by discrete molecular dynamics. In any case the
ambiguity does not arise for the winding number which is determined at finite
wavevectors. Second, it is unclear how to extend the Chern number to systems
with dissipation, driving, or nonlinearities – all properties of geophysical
fluids. Finally it is difficult to compute the Chern number from observations
or simulations because it involves an integral of the Berry curvature over
wavevector space. We note that the winding number has recently been utilized
in a number of different contexts (Tauber et al., 2019; Zhu et al., 2023)
including active optical media (Mittal et al., 2016; Ozawa et al., 2019;
Simon, 2021).

##  3 Methods

To investigate the topology of waves from observations by following the
approach described above requires a stratified component of the fluid Earth
system where observations are made at a sufficiently high sampling rate that
Poincaré-gravity waves can be diagnosed ($\nu>1.4$ CPD in the mid-latitude
region) and over a regular spatial grid located away from the equator.
Initially we considered the oceans but these requirements are not met by
existing observations (Crout et al., 2012; Lumpkin and Centurioni, 2010). We
therefore turn to the stratosphere and avoid for now the troposphere as it is
generally less stably stratified and subjected to orographic effects and
baroclinic and convective instabilities. We make use of the ERA5 reanalysis
data set which contains high-frequency data sampled evenly over the globe.
Later in the Sec. 5 we sketch a possible way to avoid the requirement for a
fine spatial grid of data points.

###  3.1 Reanalysis Data

In order to  measure winding numbers the velocity and geopotential height
fields ($u$, $v$, $h$) should satisfy two conditions. First, the region to be
studied should be several deformation lengths away from the equator to ensure
that the waves are not strongly influenced by equatorially-trapped waves. In
the language of topological physics this is the “bulk” region. Second the data
sampling interval should be short enough to resolve signals with a frequency
of up to 2 CPD. Following Pahlavan et al. (2021) who diagnosed the equatorial
Kelvin and Yanai waves as well as Poincaré-gravity waves in spectral analysis
of the stratosphere, we use ERA5 reanalysis data (Hersbach et al., 2020) at
the 50 hPa level.  Reanalysis data underestimates wave strength (Kim and
Alexander, 2015) but topology is robust against changes in strength. We sample
the horizontal components of the velocity and the geopotential heights at
6-hour intervals over the period of 1981 to 2020. The spatial domain ranges
from 25∘N to 65∘N across all longitudes with a horizontal resolution of 0.25∘,
as shown in Fig. 3. We note that at a latitude of less than 30∘ there are
Poincaré-gravity waves of frequency less than 1 CPD but this does not affect
our bulk analysis.

Figure 3:  The geographic area studied in this paper. Arrows indicate the mean
wind at 50 hPa and the color is the mean geopotential height averaged over the
40 year period from 1981 to 2020.

###  3.2 Data Processing

To work with a manageable data set size, we first reduce the spatial
resolution by interpolating the variables to a coarser resolution of 1∘ in
latitude and 2.5∘ in longitude. As the Poincaré-gravity waves that we wish to
study have long wavelengths this reduction in resolution does not affect the
analysis. Next we remove the seasonal signals from the data by first applying
a low-pass boxcar filter with a window of 182 days (half of a year) in the
time dimension to isolate the low-frequency component, and then subtract that
from data, leaving only the high-frequency components. Poincaré-gravity waves
appear at frequencies that exceed the local pendulum frequency. For a latitude
of $\phi=45^{\circ}$, the frequency is $\nu=2~{}{\rm CPD}*\sin(\phi)=1.42$
CPD.

We follow the spectral analysis of Wheeler and Kiladis (1999).  To explore the
seasonal dependence of the topological pattern, we cut the time series into
non-overlapping segments for each of the different seasons over the 40 year
period. The use of non-overlapping seasonal segments is justified because the
waves of interest have periods much shorter than a season. Linear trends are
removed from each segment, and then tapering windows are applied in the (non-
periodic) latitudinal and time dimensions of the data. We use the standard
cosine-tapered Tukey window with $\alpha$=0.5 to reduce spectral leakage
(Prabhu, 2014).  After the data has been pre-processed as described above, we
perform a 3-dimensional complex discrete Fourier transform on both the
geopotential height ($h$) and the horizontal components of the wind velocity
($u$ and $v$), so that cross-correlations can be computed in the frequency-
wavevector space. The cross-correlation is calculated for each of the 159 time
segments and then cross-correlations are computed by averaging over each of
the 40 segments for each season (39 segments for the DJF season).

The gauge-invariant complex-valued cross-correlation field in frequency -
zonal wavenumber - meridional wavenumber space is  the product of the complex-
conjugate of the Fourier-transformed geopotential height multiplied by the
horizontal components of the wind velocity. We denote these quantities as
$\langle h(\vec{k};\nu)|u(\vec{k};\nu)\rangle$ and $\langle
h(\vec{k};\nu)|v(\vec{k};\nu)\rangle$ where the brackets indicate the
processing steps described above. The complex field $\langle
h(\vec{k};\nu)|v(\vec{k};\nu)\rangle$ should be compared to the theoretical
calculation of $\Xi$ as defined by Eq. 4 and theory predicts that $\langle
h(\vec{k};\nu)|u(\vec{k};\nu)\rangle$ will show similar topology. After
selecting a particular frequency, the complex-valued data in the two-
dimensional $k$-$\ell$ space is averaged over a moving Gaussian spectral
window to further improve statistical significance (albeit at the cost of
reduced spectral resolution). The Gaussian window here treats the wavevector
space pattern as an image and has the standard deviation $\sigma=1.5$ the
spacing between points in wavevector space. Finally the magnitude and phase of
the complex-valued fields $\langle h(\vec{k};\nu)|u(\vec{k};\nu)\rangle$ and
$\langle h(\vec{k};\nu)|v(\vec{k};\nu)\rangle$ are visualized with vectors in
horizontal wavevector space. Matlab code that implements this algorithm is
presented in the Appendix.

##  4 Results

We turn first to the power spectra of the meridional component of the wind
velocity and the geopotential height to verify that power can be found in the
expected regions of frequency-wavevector space. We average the power spectra
over meridional wavenumbers ($\ell$). Fig. 4 shows the power in zonal
wavenumber ($k$) - frequency ($\nu$) space. Apparent is the sharp signal at
$\nu=1$ CPD due to the diurnal cycle as well as a semi-diurnal harmonic. High-
frequency Poincaré-gravity waves are separated by a gap in the power from low-
frequency planetary waves as expected, and exhibit more power in the $k<0$
region (Wheeler and Kiladis, 1999). The peak in the low-frequency waves at
$k>0$ corresponds to eastward planetary waves whereas part of the signal for
$k<0$ is due to standing waves that contribute equally to both positive and
negative zonal wavenumbers. However it is also clear that the spectra are far
from the dispersion  relation of the idealized rotating shallow-water  waves
shown in Figure 1.

Figure 4: The 50 hPa power spectra of meridional velocity (v) and geopotential
height (h) in the zonal wavenumber ($k$) - frequency ($\nu$) space over
latitudes 25∘N to 65∘N. The frequency $\nu$ has units of cycles per day (CPD),
and $k$ has the unit of cycles per circumference.

The winding number is discerned by plotting the cross-correlations $\langle
h(\vec{k};\nu)|u(\vec{k};\nu)\rangle$ and $\langle
h(\vec{k};\nu)|v(\vec{k};\nu)\rangle$ in $k$-$\ell$ wavevector space. From
Fig. 2, we expect Poincaré-gravity waves in the northern hemisphere to have a
vortex pattern with winding number = +1 at large positive frequencies. By
contrast, planetary waves will be topologically trivial with a domain wall.
Fig. 5 shows that the observational data displays these topological
properties. Vortices with winding numbers equal to $+1$ appear at frequencies
larger in magnitude than 1 CPD , consistent with the theoretical expectation
for Poincaré waves.

Figure 5:  The cross-correlation $\langle
h(\vec{k};\nu)|v(\vec{k};\nu)\rangle$ for the spring season (March, April,
May) obtained from ERA5 data for the region 25∘N-65∘N at 50 hPa. Subplots are
at different frequencies $\nu$. Colors represent the magnitude $|\langle
h(\vec{k};\nu)|v(\vec{k};\nu)\rangle|$ on a  linear scale with white = zero,
and arrows  indicate the complex-value of $\langle
h(\vec{k};\nu)|v(\vec{k};\nu)\rangle$.

At lower frequencies there are no vortices; instead a topologically-trivial
domain wall appears in qualitative agreement with the prediction for Rossby
waves, with the phase reversing by $180^{\circ}$ upon passing through zonal
wavenumber $k=0$.

There are some small seasonal variations in the patterns of the Poincaré waves
(Fig. 6).

Figure 6: Cross-correlation between $\langle
h(\vec{k};\nu)|u(\vec{k};\nu)\rangle$ obtained from ERA5 data for the region
25∘N - 65∘N at 50 hPa at frequency $\nu$ = 1.5 CPD for the different seasons.

We note that the vortex center is offset slightly from the origin in
wavevector space. The offset could be explained by the mean background state
flow which varies with season but further investigation is required. Lower
down in the lower troposphere at 850 hPa the Rossby wave pattern persists as
expected but the high-frequency patterns are no longer coherent (Fig. 7). The
atmosphere is not strongly stratified and macroturbulence in the form of jets
and synoptic scale weather dominates the signal obscuring any gravity waves.
Although vortices are no longer evident, we note that the high-frequency
correlations are strongest in magnitude at positive zonal wavenumbers.

Figure 7: Same as Fig.5, but for an altitude of 850 hPa.

##  5 Conclusion

In this paper we demonstrate that Poincaré-gravity waves in the stratosphere
have the non-trivial topological signature expected from theoretical
prediction. This work reverses the standard chain of reasoning that begins
from the observed dispersion to deduce the existence of equatorial Kelvin and
Yanai waves. From the existence of these waves, the non-trivial topology of
the Poincaré-gravity waves away from the equator is then inferred from the
principle of bulk-interface correspondence and found to agree with theory
(Delplace et al., 2017). Here by contrast we directly interrogate the topology
of the superinertial gravity waves away from the equator using ERA5 reanalysis
data at 50 hPa pressure level. We find the theoretically expected winding
number of $+1$ at a higher frequency. The topological signature of Poincaré-
gravity waves is clear despite the fact that their dispersion relation cannot
be clearly discerned in power spectra. The winding number vanishes at low
frequencies and the low-frequency planetary waves have trivial topology that
can be plainly distinguished from the Poincaré-gravity waves. The non-trivial
topological signature of the superinertial gravity waves also disappears in
the lower troposphere reflecting the absence of stable stratification.

A shortcoming of the approach described here is that it requires data fields
that are well-resolved in both wavevector and (high) frequency space, ruling
out the use of spatially sparse data. However it may be possible to work
directly in real space instead of wavevector space (while remaining in
frequency space): A spatial Fourier transform of $\Xi_{\pm}(k,\ell)$ (Eq. 5)
shows that $\Xi_{\pm}(x,y)$ too displays a vortex. It may be possible to
detect the vortex in spatially-sparse buoy  or balloon data provided that the
sampling rate is sufficiently high to capture superinertial frequencies. We
leave this for future work.

The mathematics of topology has great predictive power because it makes
complicated problems simple by focusing on robust features. The topology we
investigate here plays out in frequency-wavevector space, rather than in real
space. Topology in this setting is a new tool for climate science that is
relatively immune to background noise as the distinct signatures of topology
found in ERA5 observations of Poincaré-gravity waves qualitatively distinguish
them from planetary waves.  We show for the first time that Poincaré-gravity
waves in the stratosphere have a striking non-trivial topology as evidenced by
a winding number of $\pm 1$. The non-trivial topology of the waves implies the
existence of Kelvin and Yanai waves in the stratosphere that have been shown
to be a component of the Quasi-Biennial Oscillation (Pahlavan et al., 2021).
Future application to other emergent wavelike phenomena such as the Madden-
Julian Oscillation may be envisioned.

###### Acknowledgements.

We thank Tamara Barriquand, George Kiladis, Dung Nguyen, JP O’Brien, and
Antoine Venaille for helpful discussions. This work was supported in part by a
grant from the Institute at Brown for Environment and Society and by a grant
from the Simons Foundation (Grant number 662962, GF). It was performed in part
at the Aspen Center for Physics, which is supported by National Science
Foundation grant PHY-2210452. Z.Z. is supported by a Stanford Science
fellowship. The ERA5 reanalysis wind and geopotential height data is provided
by the Copernicus Climate Change Service (C3S) at ECMWF
(https://www.ecmwf.int/en/forecasts/dataset/ecmwf-reanalysis-v5, accessed on
14 April 2022).

## References

  * Crout et al. (2012) Crout, R., D. Petraitis, and N. N. D. B. Center, 2012: Physical and meteorological data from the Tropical Atmosphere Ocean (TAO) array in the tropical Pacific Ocean. NOAA National Centers for Environmental Information, URL https://www.ncei.noaa.gov/access/metadata/landing-page/bin/iso?id=gov.noaa.nodc:NDBC-TAOBuoy, last Modified: 2023-11-14, https://www.ncei.noaa.gov/archive/accession/NDBC-TAOBuoy.
  * Delplace et al. (2017) Delplace, P., J. B. Marston, and A. Venaille, 2017: Topological origin of equatorial waves. Science, 358 (6366), 1075–1077, 10.1126/science.aan8819, URL https://www.science.org/doi/full/10.1126/science.aan8819.
  * Farrar (2008) Farrar, J. T., 2008: Observations of the dispersion characteristics and meridional sea level structure of equatorial waves in the Pacific Ocean. Journal of physical oceanography, 38 (8), 1669–1689, 10.1175/2007JPO3890.1, URL https://doi.org/10.1175/2007JPO3890.1.
  * Faure (2023) Faure, F., 2023: Manifestation of the topological index formula in quantum waves and geophysical waves. Annales Henri Lebesgue, 6, 449–492, 10.5802/ahl.169, URL https://ahl.centre-mersenne.org/articles/10.5802/ahl.169/.
  * Hasan and Kane (2010) Hasan, M. Z., and C. L. Kane, 2010: Colloquium: Topological insulators. Rev. Mod. Phys., 82 (4), 3045–3067, 10.1103/RevModPhys.82.3045, URL https://link.aps.org/doi/10.1103/RevModPhys.82.3045.
  * Hendershott and Munk (1970) Hendershott, M., and W. Munk, 1970: Tides. Annual review of fluid mechanics, 2 (1), 205–224, publisher: Annual Reviews 4139 El Camino Way, PO Box 10139, Palo Alto, CA 94303-0139, USA.
  * Hersbach et al. (2020) Hersbach, H., and Coauthors, 2020: The ERA5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146 (730), 1999–2049, 10.1002/qj.3803, URL https://onlinelibrary.wiley.com/doi/abs/10.1002/qj.3803.
  * Horsley (2022) Horsley, S. A. R., 2022: Tutorial: Topology, waves, and the refractive index. arXiv:2202.08643, 10.48550/arxiv.2202.08643.
  * Hoskins et al. (1985) Hoskins, B. J., M. E. McIntyre, and A. W. Robertson, 1985: On the use and significance of isentropic potential vorticity maps. Quarterly Journal of the Royal Meteorological Society, 111 (470), 877–946.
  * Kim and Alexander (2015) Kim, J., and M. J. Alexander, 2015: Direct impacts of waves on tropical cold point tropopause temperature. Geophysical Research Letters, 42 (5), 1584–1592, 10.1002/2014gl062737.
  * Lumpkin and Centurioni (2010) Lumpkin, R., and L. Centurioni, 2010: NOAA Global Drifter Program quality-controlled 6-hour interpolated data from ocean surface drifting buoys. NOAA National Centers for Environmental Information, URL https://www.ncei.noaa.gov/archive/accession/AOML-GDP-6hr, 10.25921/7NTX-Z961.
  * Madden and Julian (1971) Madden, R. A., and P. R. Julian, 1971: Detection of a 40 – 50 day oscillation in the zonal wind in the tropical Pacific. Journal of Atmospheric Sciences, 28 (5), 702–708.
  * Mittal et al. (2016) Mittal, S., S. Ganeshan, J. Fan, A. Vaezi, and M. Hafezi, 2016: Measurement of topological invariants in a 2D photonic system. Nature Photonics, 10 (3), 180–183, 10.1038/nphoton.2016.10, 1504.00369.
  * Ozawa et al. (2019) Ozawa, T., and Coauthors, 2019: Topological photonics. Reviews of Modern Physics, 91 (1), 015 006, 10.1103/revmodphys.91.015006.
  * Pahlavan et al. (2023) Pahlavan, H. A., J. M. Wallace, and Q. Fu, 2023: Characteristics of Tropical Convective Gravity Waves Resolved by ERA5 Reanalysis. Journal of the Atmospheric Sciences, 80 (3), 777–795, 10.1175/jas-d-22-0057.1.
  * Pahlavan et al. (2021) Pahlavan, H. A., J. M. Wallace, Q. Fu, and G. N. Kiladis, 2021: Revisiting the Quasi-Biennial Oscillation as Seen in ERA5. Part II: Evaluation of Waves and Wave Forcing. Journal of the Atmospheric Sciences, 78 (3), 693–707, 10.1175/JAS-D-20-0249.1, URL https://journals.ametsoc.org/view/journals/atsc/78/3/JAS-D-20-0249.1.xml.
  * Prabhu (2014) Prabhu, K. M. M., 2014: Window Functions and Their Applications in Signal Processing. Taylor & Francis, 10.1201/9781315216386, URL https://library.oapen.org/handle/20.500.12657/41686.
  * Renteln (2013) Renteln, P., 2013: Manifolds, Tensors, and Forms: An Introduction for Mathematicians and Physicists. Cambridge University Press, URL https://books.google.com/books?id=uJWGAgAAQBAJ.
  * Simon (2021) Simon, D. S., 2021: Topology in Optics (Second Edition). 2053-2563, IOP Publishing, 10.1088/978-0-7503-3471-6, URL https://dx.doi.org/10.1088/978-0-7503-3471-6.
  * Tauber et al. (2019) Tauber, C., P. Delplace, and A. Venaille, 2019: A bulk-interface correspondence for equatorial waves. Journal of Fluid Mechanics, 868, R2, 10.1017/jfm.2019.233, 1812.05488.
  * Venaille and Delplace (2021) Venaille, A., and P. Delplace, 2021: Wave topology brought to the coast. Physical Review Research, 3 (4), 043 002, 10.1103/physrevresearch.3.043002, 2011.03440.
  * Wheeler and Kiladis (1999) Wheeler, M., and G. N. Kiladis, 1999: Convectively Coupled Equatorial Waves: Analysis of Clouds and Temperature in the Wavenumber – Frequency Domain. Journal of the Atmospheric Sciences, 56 (3), 374–399, 10.1175/1520-0469(1999)056¡0374:CCEWAO¿2.0.CO;2, URL https://journals.ametsoc.org/view/journals/atsc/56/3/1520-0469˙1999˙056˙0374˙ccewao˙2.0.co˙2.xml.
  * Wyrtki (1975) Wyrtki, K., 1975: El Niño – the dynamic response of the equatorial Pacific Ocean to atmospheric forcing. Journal of Physical Oceanography, 5 (4), 572–584.
  * Zhu et al. (2023) Zhu, Z., C. Li, and J. B. Marston, 2023: Topology of rotating stratified fluids with and without background shear flow. Physical Review Research, 5 (3), 033 191, 10.1103/physrevresearch.5.033191.

Appendix

MATLAB Code for Cross-Correlations in Spectral Space

⬇

1% %%%%%%%%%%%% Read Data %%%%%%%%%%%%ear

2fname='V_ERA.nc';

3readncfile

4fname='Z_ERA.nc';

5readncfile

6

7% %%%%%%%%%%%% Lower Spatial Resolution %%%%%%%%%%%%

8lat1=linspace(25,65,41).';

9long1=linspace(-180,180,145).';

10

11% %%

12[X,Y]=meshgrid(longitude,latitude);

13[X1,Y1]=meshgrid(long1(1:144),lat1(1:40));

14

15% %%

16for i=1:length(time)

17 v1(:,:,i)=interp2(X,Y,squeeze(v(:,:,i)).',X1,Y1);

18 z1(:,:,i)=interp2(X,Y,squeeze(z(:,:,i)).',X1,Y1);

19 end

20

21% %%%%%%%%%%%% Filter Out Low-Frequency Signals %%%%%%%%%%%%

22 for i=1:40

23 i

24 for j=1:144

25 cv=squeeze(v1(i,j,:));

26 fx_v=vfilt(cv,ones(182,1)./182,'mirror');

27 resv=cv-fx_v;

28 vano(i,j,:)=resv;

29

30 cz=squeeze(z1(i,j,:));

31 fx_z=vfilt(cz,ones(182,1)./182,'mirror');

32 resz=cz-fx_z;

33 zano(i,j,:)=resz;

34 end

35 end

36

37% %%%%%%%%%%%% Divide into Segments of seasons %%%%%%%%%%%%

38monthday=[31,28,31,30,31,30,31,31,30,31,30,31];

39yearday=cumsum(monthday);

40

41for i=1:40

42
segment_v(i,:,:,:)=v1(:,:,(i-1)*365*4+yearday(2)*4+1:(i-1)*365*4+yearday(5)*4);

43
segment_z(i,:,:,:)=z1(:,:,(i-1)*365*4+yearday(2)*4+1:(i-1)*365*4+yearday(5)*4);

44end % here is MAM for example

45

46[k1,k2,k3,k4]=size(segment_v);

47% %%

48

49% %%%%%%%%%%%% Taper %%%%%%%%%%%%

50% %% Tapering in time dimension %%

51time1=1/4:1/4:92;

52n=length(time1);

53w=tukeywin(n);

54for i=1:k1

55 for j=1:k2

56 for k=1:k3

57 y_v=squeeze(segment_v(i,j,k,:));

58 b_v=time1.'\y_v;

59 ymean_v=time1.'*b_v;

60 vanod(i,j,k,:)=(y_v-ymean_v).*w;

61

62 y_z=squeeze(segment_z(i,j,k,:));

63 b_z=time1.'\y_z;

64 ymean_z=time1.'*b_z;

65 zanod(i,j,k,:)=(y_z-ymean_z).*w;

66 end

67 end

68 end

69% %%

70% %% Tapering in meridional dimension %%

71nlat=k2;

72w1=tukeywin(nlat);

73for i=1:k1

74 i

75 for j=1:k4

76 for k=1:k3

77 vanod1(i,:,k,j)=squeeze(vanod(i,:,k,j)).'.*w1;

78 zanod1(i,:,k,j)=squeeze(zanod(i,:,k,j)).'.*w1;

79 end

80 end

81 end

82

83% %%

84% %%%%%%%%%%%% 3-D Fourier Transform %%%%%%%%%%%%

85 for i=1:k1

86 spec_v(i,:,:,:)=fftshift(fftn(squeeze(vanod1(i,:,:,:))))/k2/k3/k4;

87 spec_z(i,:,:,:)=fftshift(fftn(squeeze(zanod1(i,:,:,:))))/k2/k3/k4;

88 end

89

90% %%

91% %%%%%%%%%%%% Calculate Cross-Spectrum %%%%%%%%%%%%

92 pi=3.1415926536;

93 dt=1/4;

94 dx=2*pi/144;

95 dy=pi/180;

96

97 df=1./k4./dt;

98 dkx=1./k3./dx;

99 dky=1./k2./dy;

100

101 spec_cross=spec_v.*conj(spec_z)./df./dkx./dky;

102

103% %%%%%%%%%%%%%% Visualize %%%%%%%%%%%%

104f=[-fliplr(1:(k4/2)) 0 (1:(k4/2-1))].*df;

105

106 kx=[-fliplr(1:(k3/2))+0.5 (1:(k3/2))-0.5].*dkx;

107 ky=[-fliplr(1:(k2/2))+0.5 (1:(k2/2))-0.5].*dky;

108

109% %% Find corresponding frequencies, sum up segments %%

110 crossxy_1=squeeze(sum(spec_cross(:,:,:,176),1));

111 crossxy_5=squeeze(sum(spec_cross(:,:,:,139),1));

112 crossxy_9=squeeze(sum(spec_cross(:,:,:,102),1));

113 crossxy_12=squeeze(sum(spec_cross(:,:,:,75),1));

114 crossxy_15=squeeze(sum(spec_cross(:,:,:,47),1));

115 crossxy_18=squeeze(sum(spec_cross(:,:,:,19),1));

116

117

118 h = fspecial('gaussian',[k2 k3],1.5);

119 result_1 = imfilter(crossxy_1, h, 'replicate');

120 result_5 = imfilter(crossxy_5, h, 'replicate');

121 result_9 = imfilter(crossxy_9, h, 'replicate');

122 result_12 = imfilter(crossxy_12, h, 'replicate');

123 result_15 = imfilter(crossxy_15, h, 'replicate');

124 result_18 = imfilter(crossxy_18, h, 'replicate');

125

126

127 gapkx=2;

128 gapky=1;

129 kx1=kx(1:gapkx:k3);

130 ky1=ky(1:gapky:k2);

131 result1_1=result_1(1:gapky:k2,1:gapkx:k3);

132 result1_5=result_5(1:gapky:k2,1:gapkx:k3);

133 result1_9=result_9(1:gapky:k2,1:gapkx:k3);

134 result1_12=result_12(1:gapky:k2,1:gapkx:k3);

135 result1_15=result_15(1:gapky:k2,1:gapkx:k3);

136 result1_18=result_18(1:gapky:k2,1:gapkx:k3);

137

138 avgphi3_1=angle(result1_1);

139 avgphi3_5=angle(result1_5);

140 avgphi3_9=angle(result1_9);

141 avgphi3_12=angle(result1_12);

142 avgphi3_15=angle(result1_15);

143 avgphi3_18=angle(result1_18);

144

145

146% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

147

148 f=figure;

149 f.Position(3:4) = [900 660];

150 quiversize=0.6;

151 color='k';

152

153 xlimrange=[-5 5];

154 ylimrange=[-7.5 7.5];

155

156 subplot(3,2,1)

157 clims=-1:0.1:3;

158 contourf(kx,ky,log10(abs(result_1)),clims,'LineStyle','none');

159 %,clims

160 title('(a) \nu= 0.1 CPD')

161 xlim(xlimrange)

162 ylim(ylimrange)

163 ylabel('l')

164 set(gca,'FontSize',15)

165 colormap(flipud(autumn))

166 hold on

167

168
quiver(kx1,ky1,cos(avgphi3_1),sin(avgphi3_1),quiversize,color,'LineWidth',1)

169

170% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

171 subplot(3,2,2)

172 contourf(kx,ky,log10(abs(result_5)),clims,'LineStyle','none');

173 %,clims

174 title('(b) \nu= 0.5 CPD')

175 xlim(xlimrange)

176 ylim(ylimrange)

177 set(gca,'FontSize',15)

178 colormap(flipud(autumn))

179 hold on

180

181
quiver(kx1,ky1,cos(avgphi3_5),sin(avgphi3_5),quiversize,color,'LineWidth',1)

182

183% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

184

185 subplot(3,2,3)

186 contourf(kx,ky,log10(abs(result_9)),clims,'LineStyle','none');

187 %,clims

188 title('(c) \nu= 0.9 CPD')

189 xlim(xlimrange)

190 ylim(ylimrange)

191 ylabel('l')

192 set(gca,'FontSize',15)

193 colormap(flipud(autumn))

194 hold on

195

196
quiver(kx1,ky1,cos(avgphi3_9),sin(avgphi3_9),quiversize,color,'LineWidth',1)

197

198% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

199

200 subplot(3,2,4)

201 contourf(kx,ky,log10(abs(result_12)),clims,'LineStyle','none');

202 %,clims

203 title('(d) \nu= 1.2 CPD')

204 xlim(xlimrange)

205 ylim(ylimrange)

206 set(gca,'FontSize',15)

207 colormap(flipud(autumn))

208 hold on

209

210
quiver(kx1,ky1,cos(avgphi3_12),sin(avgphi3_12),quiversize,color,'LineWidth',1)

211

212% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

213 subplot(3,2,5)

214 contourf(kx,ky,log10(abs(result_15)),clims,'LineStyle','none');

215 %,clims

216 title('(e) \nu= 1.5 CPD')

217 xlim(xlimrange)

218 ylim(ylimrange)

219 xlabel('k')

220 ylabel('l')

221 set(gca,'FontSize',15)

222 colormap(flipud(autumn))

223 hold on

224

225
quiver(kx1,ky1,cos(avgphi3_15),sin(avgphi3_15),quiversize,color,'LineWidth',1)

226

227% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

228

229 subplot(3,2,6)

230 contourf(kx,ky,log10(abs(result_18)),clims,'LineStyle','none');

231 %,clims

232 title('(f) \nu= 1.8 CPD')

233 xlim(xlimrange)

234 ylim(ylimrange)

235 xlabel('k')

236 set(gca,'FontSize',15)

237 colormap(flipud(autumn))

238 hold on

239

240
quiver(kx1,ky1,cos(avgphi3_18),sin(avgphi3_18),quiversize,color,'LineWidth',1)