# Few-Shot Anaphora Resolution in Scientific Protocols via
Mixtures of In-Context Experts

Nghia T. Le, Fan Bai, Alan Ritter
School of Interactive Computing
Georgia Institute of Technology
<EMAIL_ADDRESS>

###### Abstract

Anaphora resolution is an important task, which traditionally has required
costly supervised training datasets for each new language, text genre, and
domain. Meanwhile, prompting large language models with a few in-context
examples has emerged as a promising approach to reduce labeling costs, however
there are a number of challenges in applying in-context learning to resolve
anaphora. In this paper, we present Mice (Mixtures of In-Context Experts),
which we demonstrate is effective for few-shot anaphora resolution in the
domain of scientific protocols (Tamari et al., 2021). Given only a handful of
training examples, Mice combines the predictions of hundreds of in-context
experts, yielding a 30% increase in F1 score over a competitive prompt
retrieval baseline. Furthermore, we show Mice can be used to train compact
student models without sacrificing performance. As far as we are aware, this
is the first work to present experimental results demonstrating the
effectiveness of in-context learning on the task of few-shot anaphora
resolution in scientific protocols.111Our code and datasets are available at
https://github.com/nle18/mice

##  1 Introduction

Figure 1: Resolving antecedents of “the mixture” in a chemical synthesis
procedure using Mice. Given a small training set of 16 examples and a test
input, we construct 256 prompts, each with two in-context demonstrations. The
prompts are then fed into a pre-trained language model (e.g. GPT-J) to
generate candidate antecedents. The probabilities of each candidate antecedent
are computed and combined in a mixture of in-context experts using a
similarity-based gating function. Mice then selects the antecedents with the
highest probabilities. In the figure, orange, blue, red denote the anaphor,
the true antecedents, and incorrect antecedents, respectively.

Prompting large language models (LMs) with in-context demonstrations has
enabled surprisingly effective few-shot learning (Brown et al., 2020).
However, more complex linguistic annotations over paragraph-length inputs,
such as anaphora and coreference, have proven challenging (Yang et al., 2022).
Prompting language models with demonstrations of anaphora and their
corresponding antecedents requires encoding long sequences of tokens, limiting
the number of demonstrations that can be used within a single prompt.
Furthermore, the performance of in-context learning has been shown to be
sensitive to the choice of demonstrations (Liu et al., 2022b) and their
ordering in the prompt (Lu et al., 2022).

To address these challenges, we present Mixtures of In-Context Experts (Mice).
We demonstrate Mice’s effectiveness on anaphora resolution in chemical
synthesis protocols (see examples in Figure 1). Natural language understanding
for protocols makes an attractive use-case for few-shot learning, as
experimental procedures contain rich coreference and bridging links. Anaphora
in protocols are expressed quite differently from those found in high-resource
domains (e.g. newswire), and scientific protocols are not easily amenable to
annotation by non-expert crowd workers.

Mice works as follows. Given an anaphor, such as “the mixture”, it uses in-
context learning to predict a list of substances contained in the mixture that
are referenced earlier in the procedure, for example: “Bromoacetyl bromide”,
“compound 54” and “water”. With only a handful of training examples (e.g., 16
or 32), Mice generates an ensemble of up to $k^{d}$ in-context experts, each
of which consists of a prompt containing $d$ demonstrations chosen from the
$k$ available training examples. The experts’ predictions are then combined in
a mixture model, where mixture weights are computed by comparing embeddings of
the input to the demonstrations encoded by each in-context expert.

Although some in-context experts perform better than others, individual
prompts act as local experts in different regions of the input space (Jacobs
et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and no single
prompt works better than others on all inputs (see Figure 2). Furthermore, if
the same antecedent is predicted by multiple in-context experts, this provides
independent evidence for the prediction, increasing the probability the answer
is correct (Downey et al., 2005).

In extensive experiments, we show Mice significantly improves the performance
of in-context learning for anaphora resolution in synthetic procedures. For
example, given 32 demonstrations, a single prompt achieves an F1 score of
$38.6$. By combining the predictions of an ensemble of 256 in-context experts,
Mice achieves $53.9$ F1.

While Mice consistently improves in-context anaphora resolution, inference in
Mice is costly, due to the large number of prompts involved. To address this
limitation, we show that fine-tuning compact BERT-based models on data that is
automatically labeled by Mice can yield performance improvements, while also
producing models that support efficient inference (Schick and Schütze, 2021a;
Schick and Schütze, 2021b; Lang et al., 2022).

Figure 2: Heatmap visualizing the performance of 64 prompts on 64 sampled
anaphors. Each prompt encodes two in-context demonstrations randomly sampled
from 8 training examples. Each square represents F1 of a single prompt applied
to a single anaphor (typically these are associated with multiple
antecedents). The prompts and test inputs are sorted from high (top, left) to
low (bottom, right) F1. Note that no single prompt performs best on all test
inputs. This suggests that it could be beneficial to combine lists of
predicted antecedents made independently by many in-context experts.

##  2 Anaphora Resolution in Scientific Protocols

Split-antecedent anaphors (Vala et al., 2016; Yu et al., 2020; Paun et al.,
2022) are plural mentions that refer to two or more antecedents in the
previous discourse. For instance in the following text: “[Alice]antecedent and
[Bob]antecedent went to the store. [They]anaphor bought some bread.” the word
“[They]” refers to two both “[Alice]” and “[Bob]”.

Similar references to multiple antecedents often appear in chemical synthesis
protocols, for example, “the mixture”. These references arise naturally as the
result of context change accommodation (Webber and Baldwin, 1992), and are
crucial for understanding the steps needed to synthesize a molecule (Fang et
al., 2021). Resolving anaphoric references in synthetic protocols could be
beneficial for automating protocols described in natural language (Sanderson,
2019; Vaucher et al., 2021), in addition to automatically extracting chemical
reaction databases from scientific literature (Lawson et al., 2014; Mysore et
al., 2019). However, anaphora is costly to annotate (Yuan et al., 2022) and
scientific protocols are not easily amenable to annotation by non-expert crowd
workers (Kulkarni et al., 2018). This motivates the need for few-shot learning
methods that can resolve anaphora in procedural texts without extensive
annotated resources.

##  3 Mixtures of In-Context Experts

While in-context learning has achieved good performance when prompted with a
few examples, the performance can vary significantly depending on different
prompt design choices (Lu et al., 2022; Liu et al., 2022b). Furthermore,
anaphora resolution requires paragraph-length contexts, limiting the number of
in-context examples that can be encoded in a single prompt (Figure 3). We
address these challenges with Mice. We show Mice is an effective method for
few-shot anaphora resolution in §5, and demonstrate that it can be used to
automatically label data for fine-tuning more compact models, without
sacrificing performance, in §4.1.

Figure 3: Distribution of the maximum number of in-context demonstrations of
an anaphor, synthesis protocol, and corresponding antecedents that can be
encoded in a single prompt. We compute the max number of demonstration tokens
by subtracting the max sequence length (2048) by a fixed number for generated
tokens (256) and the number of tokens for the test input. Demonstrations are
randomly sampled until the max number of tokens is reached. Given the longer
contexts needed to demonstrate anaphora resolution, a prompt can encode at
most 8 demonstrations, much less than the 32 used in Brown et al. (2020).

#### In-Context Learning

We formulate the task of anaphora resolution in synthetic protocols as
follows. The input includes a document $D$ and a query anaphor $a$. Our goal
is to identify a set of antecedents $\mathcal{Y}=\\{y_{0},y_{1},...,y_{m}\\}$
that correspond to text spans in $D$. To tackle this problem via in-context
learning, we frame it as a SQuAD-style extractive question answering task (Wu
et al., 2020). Specifically, as shown in Figure 1, each example $(D,a)$ is
formatted as the concatenation of document $D$ and template question: ‘‘What
does $a$ contain?’’ An autoregressive language model then completes this
sequence by generating $\mathcal{Y}$, with the antecedents separated by a
special marker “|”. Following the typical approach to in-context learning, the
prompt includes a few demonstrations in the prefix and ends with the test
input.

#### Mixture of Experts

For a given test input $x=(D,a)$, we aim to find the antecedents $y_{i}$ with
the highest probabilities $P(y_{i}|x)$. The notation $y_{i}$ denotes an
antecedent from the union of antecedents generated by all prompts. Mice
computes $P(y_{i}|x)$ using a mixture of experts (Jacobs et al., 1991; Cho et
al., 2019), treating the prompt, $z$, as a latent variable (Guu et al., 2020):

$P(y_{i}|x)=\sum_{z}P(y_{i}|z,x)P(z|x)$ (1)

In Eq.1, $P(z|x)$ represents the likelihood that prompt $z$ is constructed
given $x$, and $P(y_{i}|z,x)$ represents the probability that the LM predicts
antecedent $y_{i}$ when prompted with $z$ and $x$.

#### Similarity-based Gating

We compute $P(z|x)$ by summing similarity scores $s(x,u_{1}),...,s(x,u_{d})$
between $x$ and the in-context demonstrations $u_{1},...,u_{d}$ encoded in
$z$:222We also experimented with multiplying the similarity scores and
observed similar results.

$P(z|x)\propto\exp{\sum_{i=1}^{d}s(x,u_{i})}$

where $s(x,u_{i})$ is the cosine similarity between the embeddings of $x$ and
$u_{i}$. Details of the similarity measures used in our experiments are
presented in §4.1.

#### Estimating Antecedent Probabilities

Computing probabilities $P(y_{i}|z,x)$ that are comparable across variable-
length antecedents is not easy. Longer sequences will naturally have smaller
LM probabilities, suggesting the need for length normalization, or averaging
per-token probabilities, neither of which we found to work well.333Similar to
Zhao et al. (2021), we observe that, for a generated antecedent, the first
token probabilities vary the most, while probabilities of subsequent tokens
are highly deterministic. Therefore, following Zhao et al. (2021), we estimate
$P(y_{i}|z,x)$ using first token probabilities. Specifically, let $y_{i,0}$
denote the first token of $y_{i}$. We then have:

$P(y_{i}|z,x)\approx\max_{j}P_{j}(y_{i,0}|z,x)$ (2)

where the max is taken over all mentions of $y_{i}$ found in the $k^{d}$
prompts. $P_{j}(y_{i,0}|z,x)$ is the probability of token $y_{i,0}$ being the
first token of the $j$th antecedent generated by the language model, when
prompted with $z$ and $x$.

#### Approximation with Sampling

Approximating $P(y_{i}|z,x)$ using first-token probabilities in Eq.2 has a
drawback: it disregards the length of the antecedents and treats different
antecedents with the same first token as equivalent. As an alternative, we
present Mice-Sampling: a simple Monte Carlo approximation of $P(y_{i}|z,x)$
that uses a binary indicator $\mathbbm{1}[y_{i}\in\mathcal{Y}_{z,x}]$ of
whether or not antecedent $y_{i}$ is in $\mathcal{Y}_{z,x}$ (the set of
generated antecedents given prompt $z$ and $x$):

$\displaystyle P(y_{i}|x)$ $\displaystyle=$
$\displaystyle\sum_{z}P(y_{i}|z,x)P(z|x)$ $\displaystyle\approx$
$\displaystyle\sum_{z}\mathbbm{1}[y_{i}\in\mathcal{Y}_{z,x}]P(z|x)$

In §4, we show empirically that when using hundreds of prompts, Mice-Sampling
is actually a better approximation than the LM probability of the first token
(Zhao et al., 2021). In addition, Mice-Sampling is simpler to implement, as it
does not require storing output logits and computing the softmax to obtain
first token probabilities.

| Train (full) | Dev-64 | Dev-256 | Test
---|---|---|---|---
# anaphors | 4,766 | 64 | 256 | 898
# antecedents | 21,673 | 293 | 996 | 4,016
# documents | 856 | 11 | 52 | 166
# sentences | 5,833 | 76 | 346 | 1,066
# tokens | 984,032 | 10,150 | 45,825 | 140,614

Table 1: Statistics of selected ChEMU-Ref splits. Details on few-shot train
sets are shown in Table 9 in the Appendix.

##  4 Experimental Setup

We evaluate our approach on ChEMU-Ref, a corpus of synthetic procedures from
chemical patents that are annotated with coreference and anaphora Fang et al.
(2021). The ChEMU-Ref annotations contain fine-grained chemistry-specific
relations such as Transformed, which indicates the mixture components undergo
a chemical transformation, or Work-up, which represents a combination of
compounds to isolate or purify a reaction product. In this work, we focus on
modeling the general structure of anaphora in scientific protocols and
understanding which compounds are combined at each step of the procedure,
which does not require making these more fine-grained distinctions. Therefore,
we pre-processed the data by collapsing all one-to-many ChEMU-Ref relations
into a single Multiple-antecedent relation. We remove anaphors comprising
compounds described with IUPAC nomenclature (Skonieczny, 2006), for example
“2,2,6,6-tetramethylpiperidine”, while retaining nominal anaphors, such as
“the mixture”, “the solution” and “the reaction”, which make up 90% of
anaphors in the ChEMU-Ref corpus. For train/dev/test splits, we use the
original train split for training and development, and the original dev split
for evaluation.444The official ChEMU-Ref test set is hidden at
http://chemu2021.eng.unimelb.edu.au/. Following Gao et al. (2021), for each
$k$-shot experiment, we sample five different training sets from the full
training split using different seeds and report the mean. For model
development and ablation studies, we use a small development set of 64
examples (Dev-64) to simulate a true few-shot learning setting. For evaluation
on held-out data, we use the full ChEMU-Ref development set as test data
(Test) as well as a sub-sampled version with 256 examples (Dev-256) to control
for high GPT-J inference costs. Selected dataset statistics are shown in Table
1.

We use F1 as our evaluation metric. In particular, we compute the micro-F1
between the predicted and gold sets of antecedents of the evaluation data. A
predicted and a gold antecedent are considered the same if they are an exact
match.

###  4.1 Implementation Details

#### Models

We use GPT-J-6B Wang and Komatsuzaki (2021) as the backbone language model for
Mice and in-context baselines since it was the largest publicly available
autoregressive language model at the time we started this work.555Larger
models, such as OPT-175B (Zhang et al., 2022), have become available recently.
Compared with other similar language models like GPT-2, GPT-J has a larger
maximum sequence length of 2048. It is also pre-trained on the Pile (Gao et
al., 2020), which covers in-domain data including chemical patents from
USPTO666https://www.uspto.gov/ and PubMed articles. Similarly, we choose
ProcBERT (Bai et al., 2021) for the student model in knowledge distillation
due to its in-domain pre-training on synthetic procedures.

#### Mice

We ensemble up to 256 prompts, with 2 or 5 in-context demonstrations in the
prompt, for all few-shot settings. To calculate the similarity score $s(x,u)$
between the input $x$ and a demonstration example $u$, we use SBERT model
(Reimers and Gurevych,
2019)777https://www.sbert.net/docs/pretrained_models.html with the roberta-
large checkpoint since this model is widely used for measuring text similarity
(Wang et al., 2020). We also experimented with calibrating the language model
to be bias-free by applying the calibration procedure described in Zhao et al.
(2021); however, we found that it did not improve performance in the context
of anaphora resolution in scientific protocols.888When prompted with content-
free inputs, such as “N/A”, the language model already generates bias-free
answers (e.g. the LM generates ”N/A” given ”N/A” as a test input) without
calibration, suggesting there is no significant bias towards specific answers
for anaphora resolution.

#### Antecedent Filtering

To further improve the quality of predicted antecedents, we apply several
post-processing rules that were developed on the Dev-64 split. Namely, we (1)
filtered out all predictions that exceed a threshold length of 250 tokens (2)
merged the antecedents that are sub-strings to the longest predicted
antecedents (e.g. “CH2CL2” is merged into “CH2CL2 (40 mL)”) and (3) filtered
out all the antecedents with probability $P(y_{i}|z,x)$ in Eq. 2 less than
$0.02$ and probability $P(y_{i}|x)$ in Eq. 1 less than $0.1$.

#### Knowledge Distillation

To produce a compact model for inference, we perform knowledge distillation
(Sanh et al., 2019; Jiao et al., 2020) via self-training, where the student
model is trained on pseudo labels generated Mice. We experiment with three
few-shot settings $k\in\\{8,32,64\\}$. In each setting, out of five training
runs, we select the Mice-Sampling-{2} model checkpoint (2 in-context examples)
that achieves the median performance on the Dev-256 set as the teacher model.
To train the student model (Schick and Schütze, 2021b; Lang et al., 2022), we
randomly sample real unlabeled synthetic protocols from the chemical patent
corpus collected in Bai et al. (2021). We then run rule-based anaphor
detection (Appendix A) to identify anaphors within the sampled documents.
Subsequently, the teacher model is used to predict antecedents. Finally, we
train the ProcBERT student model in a two-stage process. We first fine-tune
ProcBERT on $M$ pseudo-labeled examples ($M\in\\{50,100,...,2000\\}$) for 50
epochs, then further fine-tune it on the $k$ examples with gold labels for 200
epochs. For the student model, we frame antecedent resolution as a sequence-
labeling problem by transferring span-level antecedent labels into token-level
labels with a BIO tagging scheme. We train the student model for token-level
classification, where the input anaphor is marked with two special tokens
[Ana-start] and [Ana-end].

#### Computational Cost

In addition to performance, we also measure the computational cost of the
teacher and student models in knowledge distillation. Concretely, we measure
floating point operations (FLOPs) of the two models for both training and
inference using the FLOPs-counting code provided in Clark et al.
(2020)999https://github.com/google-
research/electra/blob/master/flops_computation.py. Note that, the training of
the student model requires $M$ pseudo-labeled examples, so the training FLOPs
of the student model will also include the FLOPs of generating those pseudo
labels using the teacher model.

For all GPT-J-based experiments, generation is performed using greedy decoding
up to a maximum of 256 tokens on 48GB A40 GPUs. When using 256 in-context
experts, about three anaphors can be resolved per GPU hour.

Model | 4-shot | 8-shot | 16-shot | 32-shot | 64-shot | full
---|---|---|---|---|---|---
E2E (Fang et al., 2021) | 0.91.1 | 10.35.3 | 31.68.1 | 42.55.4 | 51.92.3 | 77.4
ProcBERT (Bai et al., 2021) | 20.37.9 | 27.13.7 | 36.14.1 | 45.36.7 | 55.04.4 | 87.7
T0-3B (Sanh et al., 2022) | 23.57.0 | 30.84.3 | 42.12.3 | 49.51.3 | 58.13.2 | 83.2
T5-3B (Raffel et al., 2020) | 27.33.7 | 33.63.9 | 42.56.7 | 55.66.2 | 61.01.1 | 83.8
Kate (Liu et al., 2022b) | 36.29.6 | 44.75.0 | 46.82.9 | 46.43.9 | 48.41.5 | -
Kate\+ (Liu et al., 2022b) | 41.99.8 | 46.710.8 | 48.29.0 | 51.43.2 | 53.21.8 | -
Product (Min et al., 2022a) | 36.710.5 | 39.48.6 | 41.35.3 | 42.42.3 | 45.12.7 | -
Mice-{2} | 44.46.2 | 49.85.3 | 52.93.8 | 54.62.1 | 54.53.1 | -
Mice-Sampling-{2} | 44.75.8 | 50.65.4 | 55.12.9 | 56.83.0 | 59.03.1 | -
Mice-{5} | 44.56.5 | 52.65.0 | 55.84.4 | 57.41.2 | 58.41.9 | -
Mice-Sampling-{5} | 44.46.5 | 52.55.0 | 56.84.0 | 57.53.4 | 59.22.1 | -
Table 2: Results on Dev-256. Bracketed numbers indicate the maximum number of
demonstrations per prompt. For instance, Mice-{2} represents a mixture of
$\min(k^{2},256)$ prompted language models where each prompt encodes two
training examples. For each $k$-shot experiment, we report the mean and
standard deviation across five different training set samples. The full ChEMU-
Ref training split contains 4,766 examples. Results on the test set are
presented in Table 3.

###  4.2 Baselines

We compare Mice to the fine-tuning and in-context learning baselines described
below. We use the $k$-shot training set to train the fine-tuning models or
select in-context demonstrations for the in-context models. For all baselines,
we use the development set Dev-64 for model selection and report the results
on the held-out Dev-256 and test sets. Details on baseline implementation can
be found in Appendix C.

#### E2E

(Fang et al., 2021): We train this end-to-end neural anaphora resolution model
developed for chemical procedures with minimal adaptations.
101010https://github.com/biaoyanf/ChEMU-Ref

#### ProcBERT

(Bai et al., 2021): The student model in knowledge distillation (§4.1). For
baseline comparison, we fine-tune only on gold data.

#### T5/T0-3B

(Raffel et al., 2020; Sanh et al., 2022): We fine-tune T5-3B (3 billion
parameters) and T0-3B for antecedent resolution using a QA prompt similar to
Mice.

#### Kate

(Liu et al., 2022b): For this model, each test example is associated with a
single prompt constructed using K-nearest neighbors. For the sentence encoder,
we use a pre-trained RoBERTa large model (all-roberta-large-v1) from the SBERT
library (Reimers and Gurevych, 2019).

####  Kate+

Given a single prompt constructed by Kate, we sample 256 antecedent lists
using nucleus sampling (Holtzman et al., 2020) and ensemble the predictions in
a similar manner to Mice. This baseline ensembles the same number of
predictions as Mice, with the difference being they are all sampled from the
language model when conditioned on a single prompt.

#### Product

(Min et al., 2022a): We adapt this ensemble-based demonstration method for
text classification by first obtaining the output probabilities $k$ times (one
training example per prompt) and then computing the conditional probability of
the antecedents given the input by taking the product of the aforementioned
probabilities (i.e. $P(y|x)=\prod_{z}P(y|z,x)$).

##  5 Results and Analysis

Model | 8-shot | 32-shot | 64-shot | full
---|---|---|---|---
E2E (Fang et al., 2021) | 9.9 | 42.3 | 51.8 | 75.8
ProcBERT (Bai et al., 2021) | 30.8 | 41.9 | 57.6 | 78.3
T5-3B (Raffel et al., 2020) | 33.9 | 52.0 | 58.6 | 72.6
Kate (Liu et al., 2022b) | 37.9 | 38.6 | 42.3 | -
Product (Min et al., 2022a) | 34.3 | 38.5 | 49.0 | -
Product-{2} | 45.6 | 51.1 | 51.3 | -
Mice-{2} | 48.1 | 52.6 | 53.9 | -
Mice-Sampling-{2} | 48.5 | 53.9 | 55.7 | -
Know. Distill. (2000 exam.) | 56.2 | 59.4 | 64.6 | -

Table 3: F1 on the full test set. For the knowledge distillation model, we
picked the median trial from Mice-Sampling-{2} of Table 2 to run experiment
on, due to the expensive inference cost on large amount of unlabeled data. For
others, we averaged over five training samples.  Figure 4: Mean F1 of in-
context learning models vs. training set size $k$, on Dev-256.

Results on Dev-256 are presented in Table 2. We observe that variations of
Mice either outperform (4-shot, 8-shot, 16-shot, 32-shot) or are competitive
(64-shot) with the baselines. Furthermore, we found that while T5-3B works
well for 64-shot, it still trails behind Mice and Mice-Sampling (and even
Kate) when using fewer labeled data.111111We also experimented with T-few (Liu
et al., 2022a), a performant T0-based parameter-efficient fine-tuning
approach. With suggested hyper-parameters, T-few performs worse than the full-
model fine-tuning of T0-3B on anaphora resolution, so we leave the further
exploration of T-few to future work. Additionally, we observe that Mice
outperforms other in-context learning models that use a single most performant
prompt (Kate) or a single training example per prompt (Product). This is
highlighted in Figure 4, where we plot the performance of various in-context
learning models as a function of training set size $k$.

Results on the full test set (Table 3) shows similar trends. We also
experiment with extending the Product baseline to include two in-context
demonstrations per prompt (Product-{2}). While increasing the number of in-
context demonstrations in Product also boosts the performance for all
$k$-shot, Mice and Mice-Sampling still outperform Product-{2}.

####  Kate\+ vs. Mice

We highlight the results of Kate\+ in comparison to Kate and Mice. Kate\+
performs somewhat better than Kate, likely due to state-of-the-art text
generation methods (e.g. nucleus sampling) as well as ensembling
independently-sampled sequences conditioned on a single prompt. However, Mice
significantly outperforms Kate+, indicating the importance of ensembling
prompts with different demonstrations and permutations.

Model |  | F1 | Train FLOPs | Infer FLOPs
---|---|---|---|---
| 8-shot | 32-shot | 64-shot | (1000 exam.)
ProcBERT | 28.0 | 41.1 | 56.7 | 3.2e15 | 1.2e14
Teacher | 54.1 | 55.7 | 59.9 | 0 | 1.9e17
Student (# pseudo labels) |
\- 50 |  | 36.0 | 46.2 | 58.4 | 1.4e16 | 1.2e14
\- 100 |  | 43.5 | 49.9 | 62.2 | 2.4e16 | 1.2e14
\- 200 |  | 42.1 | 52.3 | 60.4 | 4.5e16 | 1.2e14
\- 500 |  | 49.1 | 52.2 | 63.3 | 1.1e17 | 1.2e14
\- 1000 |  | 50.1 | 51.8 | 64.1 | 2.1e17 | 1.2e14
\- 2000 |  | 54.3 | 55.8 | 64.3 | 4.1e17 | 1.2e14

Table 4: Dev-256 F1 and training/inference FLOPs of the teacher model (Mice-
Sampling) and the student model in knowledge distillation. The training FLOPs
of the student model under different few-shot settings are almost the same.
With 2000 pseudo-labeled examples, the student model (110M parameters) can
match or outperform the teacher model in terms of F1. Although inference FLOPs
of the teacher model are around 1500 times the FLOPs of the student model,
considering the cost of training the student model to match the teacher
model’s performance, the teacher model is actually more cost effective than
the student model for inference when the number of inference examples is low,
e.g., less than 2100 for 32-shot. For rows where the number of FLOPs varies
depending on the number of training examples, we show the maximum.

#### Knowledge Distillation

The impact of knowledge distillation is explored in Table 4. We observe that
with only 50 pseudo-labeled examples, the student model outperforms the
ProcBERT baseline by 8.0 F1 for 8-shot, 5.1 F1 for 32-shot and 1.7 F1 for
64-shot, showing the effectiveness of distillation. The performance of the
student model improves as the number of pseudo-labeled examples increases.
With 2000 pseudo-labeled examples, the inference-efficient student model (110M
parameters) outperforms the teacher model Mice-Sampling. We hypothesize that
the student model outperforms the teacher because after training on pseudo-
labels generated by MICE, the student model is further fine-tuned on the $k$
available gold-labeled examples, as described in §4.1. This approach combines
the benefits of in-context learning with fine-tuning achieving better
performance than either in isolation. In-context learning does not incur any
training costs, but the number of FLOPs required by Mice-Sampling for
inference is roughly 1,500 times the computational cost of the student model.

Figure 5: Mean F1 ($\pm$ one std. dev.) vs. the number of prompts used in
Mice. Performance plateaus after the number of experts is increased to 75,
suggesting an appropriate choice for the minimum number of prompts in this
context. The number of prompts corresponds to the number of inference passes,
which we measure using FLOPs per test example (§4.1).

#### Number of mixture prompts vs. Performance

We explore the effect of varying the number of mixture prompts in Figure 5.
While adding more prompts leads to rapid F1 improvement, the performance
plateaus after about 75 prompts. This suggests an appropriate minimum number
of experts to use in Mice.

Figure 6: Mean F1 ($\pm$ one std. dev.) vs. max number of in-context
demonstrations per prompt. We observe a large performance gap between Mice-
Sampling and Kate.

#### Number of in-context demonstrations vs. Performance

Figure 6 shows the effect of varying the number of in-context demonstrations
per prompt, for mixture models (Mice-Sampling) and single-prompted models
(Kate). As the number of in-context demonstration per prompt increases, Kate
F1 steadily increases. This effect is less pronounced in Mice, where adding
more than three in-context demonstrations per prompt does not significantly
improve performance. However, there is still a large performance gap between
Mice and Kate, regardless of the number of in-context demonstrations.

##  6 Related Work

#### Ensembling prompts

Recent work has explored different strategies for ensembling LM predictions.
Lester et al. (2021) and Zhao et al. (2022) aggregate predictions from
multiple prompts via either majority voting or weighting on development set
performances. Jiang et al. (2020) and Qin and Eisner (2021) ensemble a small
number of prompt templates or soft prompts, then learn their weights from a
large training set. Asai et al. (2022) combines soft prompts pre-trained on
multiple source tasks with fine-tuned attention weights to generate prompts
for a target task. In contrast to all the above work, Mice generates a large
number of prompts from a small number of training examples and combines their
predictions using mixtures of experts with similarity-based gating.

Similar to our work, Min et al. (2022a) shows that ensembling the predictions
of $k$ one-shot prompts by multiplying their LM probabilities is better than
concatenating all $k$ examples as one prompt, which we use as a baseline
(Product). Concurrently, Lang et al. (2022) demonstrates the advantage of
ensembling $k$ one-shot prompts via co-training the ensembling parameters
using a smaller model. However, they train the ensembling parameters with an
MLP layer and thus require all training examples to share the same label
space, which is not applicable to anaphora resolution. Importantly, both Min
et al. (2022a) and Lang et al. (2022) only consider a small number of prompts,
whereas we demonstrate the benefits of generating and combining hundreds of
prompts as a mixture-of-experts.

#### Few-shot anaphora resolution

There is very little prior work on few-shot coreference or anaphora
resolution. Prior work (Perez et al., 2021; Min et al., 2022b) has applied in-
context learning to the benchmark datasets Winograd Schema Challenge (WSC)
(Levesque et al., 2012) and WinoGrande (Sakaguchi et al., 2019). Both datasets
are intended to be an alternative to the Turing Test. As such, they contain
short, syntactically simple sentences, and require a different type of world
knowledge to resolve references than those seen in scientific protocols.
Agrawal et al. (2022) demonstrates the effectiveness of zero-shot prompting
for coreference resolution on clinical data. They introduce task-specific
programs (called resolvers) that map language model outputs to discrete label
space, which can be used in conjunction with Mice for further improvements on
few-shot coreference resolution.

#### Mixture-of-Experts in Language Modelling

Mixture-of-Experts (MoE) based language models have been shown to improve
performance and efficiency across a variety of NLP tasks Shazeer et al.
(2017); Du et al. (2022); Gururangan et al. . These models were pre-trained on
a mixture of datasets with different domains, either via learning the gating
weights at a token level Shazeer et al. (2017); Du et al. (2022) or document
level Gururangan et al. . Unlike prior work, Mice combines the predictions of
many in-context experts, each of which encodes a permutation of demonstrations
drawn from a handful of training examples. The gating weights in MICE are
computed based on similarity scores between test input and in-context
examples.

##  7 Conclusion

In this paper, we propose and demonstrate the effectiveness of Mixtures of In-
Context Experts for few-shot anaphora resolution in chemical synthesis
protocols. Mice generates a large number of in-context experts (prompts) from
a few training examples, where each expert consists of randomly permuted
demonstrations. Predictions made by these experts are combined in a mixture
model with a similarity-based gating function. Our experiments show that Mice
significantly improves the performance of in-context learning for anaphora
resolution in scientific protocols, using just a handful of training examples.
We further demonstrate that knowledge distillation can dramatically reduce the
costs of inference while maintaining performance, which increases Mice’s
potential applicability in working systems.

## Limitations

A challenging bottleneck for Mice is its expensive inference cost: despite the
promising few-shot capabilities, the inference cost is even more expensive
when using a language model with over 6 billion parameters. Although we show
that knowledge distillation can address this problem, future work may further
investigate how to improve smaller models’ in-context capabilities.

## Acknowledgements

This material is based upon work supported by the Defense Advanced Research
Projects Agency (DARPA) under Contract No. HR001119C0108, in addition to the
NSF (IIS-2052498) and IARPA via the BETTER program (2019-19051600004). The
views, opinions, and/or findings expressed are those of the author(s) and
should not be interpreted as representing the official views or policies of
the Department of Defense, IARPA or the U.S. Government. This work is approved
for Public Release, Distribution Unlimited.

## References

  * Agrawal et al. (2022) Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022\.  Large language models are zero-shot clinical information extractors.  _arXiv preprint arXiv:2205.12689_.
  * Asai et al. (2022) Akari Asai, Mohammadreza Salehi, Matthew E Peters, and Hannaneh Hajishirzi. 2022\.  Attentional mixtures of soft prompt tuning for parameter-efficient multi-task knowledge sharing.  _arXiv preprint arXiv:2205.11961_.
  * Bai et al. (2021) Fan Bai, Alan Ritter, and Wei Xu. 2021.  Pre-train or annotate? Domain adaptation with a constrained budget.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_.
  * Cho et al. (2019) Jaemin Cho, Minjoon Seo, and Hannaneh Hajishirzi. 2019.  Mixture content selection for diverse sequence generation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_.
  * Downey et al. (2005) Doug Downey, Oren Etzioni, and Stephen Soderland. 2005.  A probabilistic model of redundancy in information extraction.  In _Proceedings of the 19th International Joint Conference on Artificial Intelligence_ , IJCAI’05, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
  * Du et al. (2022) Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. 2022.  GLaM: Efficient scaling of language models with mixture-of-experts.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 5547–5569. PMLR.
  * Fang et al. (2021) Biaoyan Fang, Christian Druckenbrodt, Saber A Akhondi, Jiayuan He, Timothy Baldwin, and Karin Verspoor. 2021.  ChEMU-ref: A corpus for modeling anaphora resolution in the chemical domain.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020.  The Pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_.
  * (12) Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer.  DEMix layers: Disentangling domains for modular language modeling.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training.  In _International Conference on Machine Learning_.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.  The curious case of neural text degeneration.  In _International Conference on Learning Representations_.
  * Jacobs et al. (1991) Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991\.  Adaptive mixtures of local experts.  _Neural Computation_.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020.  How can we know what language models know?  _Transactions of the Association for Computational Linguistics_.
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.  TinyBERT: Distilling BERT for natural language understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_.
  * Jordan and Jacobs (1994) Michael I Jordan and Robert A Jacobs. 1994.  Hierarchical mixtures of experts and the em algorithm.  _Neural computation_.
  * Kulkarni et al. (2018) Chaitanya Kulkarni, Wei Xu, Alan Ritter, and Raghu Machiraju. 2018.  An annotated corpus for machine reading of instructions in wet lab protocols.  In _Proceedings of NAACL-HLT_.
  * Lang et al. (2022) Hunter Lang, Monica Agrawal, Yoon Kim, and David A. Sontag. 2022.  Co-training improves prompt-based learning for large language models.  _CoRR_.
  * Lawson et al. (2014) Alexander J Lawson, Jürgen Swienty-Busch, Thibault Géoui, and David Evans. 2014.  The making of reaxys—towards unobstructed access to relevant chemistry information.  In _The Future of the History of Chemical Information_.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Levesque et al. (2012) Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012.  The Winograd schema challenge.  In _Thirteenth international conference on the principles of knowledge representation and reasoning_.
  * Liu et al. (2022a) Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022a.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  _arXiv preprint arXiv:2205.05638_.
  * Liu et al. (2022b) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022b.  What makes good in-context examples for GPT-3?  In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022\.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_.
  * Min et al. (2022a) Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022a.  Noisy channel language model prompting for few-shot text classification.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_.
  * Min et al. (2022b) Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022b.  MetaICL: Learning to learn in context.  In _NAACL-HLT_.
  * Mysore et al. (2019) Sheshera Mysore, Zachary Jensen, Edward Kim, Kevin Huang, Haw-Shiuan Chang, Emma Strubell, Jeffrey Flanigan, Andrew McCallum, and Elsa Olivetti. 2019.  The materials science procedural text corpus: Annotating materials synthesis procedures with shallow semantic structures.  In _Proceedings of the 13th Linguistic Annotation Workshop_.
  * Paun et al. (2022) Silviu Paun, Juntao Yu, Nafise Sadat Moosavi, and Massimo Poesio. 2022.  Scoring coreference chains with split-antecedent anaphors.  _arXiv preprint arXiv:2205.12323_.
  * Perez et al. (2021) Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021.  True few-shot learning with language models.  In _Advances in Neural Information Processing Systems_.
  * Qin and Eisner (2021) Guanghui Qin and Jason Eisner. 2021.  Learning how to ask: Querying LMs with mixtures of soft prompts.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_.
  * Sakaguchi et al. (2019) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019.  WinoGrande: An adversarial Winograd schema challenge at scale.  _arXiv preprint arXiv:1907.10641_.
  * Sanderson (2019) Katharine Sanderson. 2019.  Automation: Chemistry shoots for the moon.  _Nature_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _CoRR_.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_.
  * Schick and Schütze (2021a) Timo Schick and Hinrich Schütze. 2021a.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics_.
  * Schick and Schütze (2021b) Timo Schick and Hinrich Schütze. 2021b.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  In _5th International Conference on Learning Representations, ICLR_.
  * Skonieczny (2006) Stanislaw Skonieczny. 2006.  The IUPAC rules for naming organic molecules.  _Journal of chemical education_.
  * Tamari et al. (2021) Ronen Tamari, Fan Bai, Alan Ritter, and Gabriel Stanovsky. 2021.  Process-level representation of scientific protocols with interactive annotation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics_.
  * Vala et al. (2016) Hardik Vala, Andrew Piper, and Derek Ruths. 2016.  The more antecedents, the merrier: Resolving multi-antecedent anaphors.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_.
  * Vaucher et al. (2021) Alain C Vaucher, Philippe Schwaller, Joppe Geluykens, Vishnu H Nair, Anna Iuliano, and Teodoro Laino. 2021.  Inferring experimental procedures from text-based representations of chemical reactions.  _Nature communications_.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wang et al. (2020) Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, et al. 2020.  CORD-19: The COVID-19 open research dataset.  In _Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020_.
  * Webber and Baldwin (1992) Bonnie Webber and Breck Baldwin. 1992.  Accommodating context change.  In _30th Annual Meeting of the Association for Computational Linguistics_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_.
  * Wu et al. (2020) Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020.  CorefQA: Coreference resolution as query-based span prediction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Yang et al. (2022) Xiaohan Yang, Eduardo Peynetti, Vasco Meerman, and Chris Tanner. 2022.  What GPT knows about who is who.  In _Proceedings of the Third Workshop on Insights from Negative Results in NLP_.
  * Yu et al. (2020) Juntao Yu, Nafise Sadat Moosavi, Silviu Paun, and Massimo Poesio. 2020.  Free the plural: Unrestricted split-antecedent anaphora resolution.  In _Proceedings of the 28th International Conference on Computational Linguistics_.
  * Yuan et al. (2022) Michelle Yuan, Patrick Xia, Chandler May, Benjamin Van Durme, and Jordan Boyd-Graber. 2022.  Adapting coreference resolution models through active learning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  OPT: open pre-trained transformer language models.
  * Zhao et al. (2022) Mengjie Zhao, Fei Mi, Yasheng Wang, Minglei Li, Xin Jiang, Qun Liu, and Hinrich Schütze. 2022.  LMTurk: Few-shot learners as crowdsourcing workers in a language-model-as-a-service framework.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _ICML_.

##  Appendix A Anaphor Detection

We develop a rule-based system using simple string matching patterns. We
developed the system using the full train set, optimized on the Dev-64
development set, and evaluated on Dev-256 and Test set. Anaphora in synthetic
protocols are expressed using relatively consistent and easy to identify
expressions, for example: "the mixture", "the reaction solution", and "the
resulting solution", making them relatively easy to identify using a simple
approach using regular expressions. As a result, our system can effectively
identify most of these spans, achieving high F1 scores when evaluated on
different evaluation splits (Table 5). An example of a pattern that matches
all anaphors in Figure 1 test input is shown here:
https://regex101.com/r/pImmBT/1

| P | R | F1
---|---|---|---
Dev-64 | 98.5 | 100 | 99.2
Dev-256 | 97.6 | 95.7 | 96.6
Test | 95.6 | 85.5 | 90.3

Table 5: Results of our rule-based anaphor detection system on different
evaluation splits.

##  Appendix B Mice Implementation Details

We present the hyperparameters for Mice (as well as other in-context learning
baselines) in Table 7. We also experimented with different prompt ordering of
in-context demonstrations Liu et al. (2022b) and contextual calibration Zhao
et al. (2021). All ablation experiments were evaluated on Dev-64 development
set.

#### Ordering of In-context Demonstrations

Since different prompt ordering can play a significant role in the
performance, we measure the effect of different in-context ordering on Kate.
Given a test input, we first select the most similar examples using based on
the cosine similarity between the test input and the example embeddings
(encoded using pre-trained RoBERTa large model). We then consider the
following ordering:

  * •

Ascend: Sorting the order of in-context demonstrations from least to most
similar examples (i.e. most similar example closest to test input)

  * •

Descend: Reverse of Ascend

  * •

Mixed: Random shuffle of in-context demonstrations

The results on Table 6 show there is not much difference between different
orderings. This result is in line with the findings in Lu et al. (2022) that
prompt ordering is dataset-dependent. As such, we select Ascend as the de
facto ordering for all our in-context learning experiments.

| 4-shot | 8-shot | 16-shot | 32-shot | 64-shot
---|---|---|---|---|---
Ascend | 34.34.9 | 45.32.8 | 46.85.1 | 46.86.6 | 46.55.9
Descend | 34.84.8 | 46.21.3 | 45.14.8 | 44.66.6 | 46.54.7
Mixed | 34.17.6 | 43.93.3 | 46.14.7 | 46.24.3 | 46.34.1

Table 6: Order of in-context demonstration results.

#### Contextual Calibration

Zhao et al. (2021) shows that prompted LMs can be biased towards specific
outputs, regardless of the test input. We thus experimented with the
calibration procedure for generation tasks described in Section 5 of Zhao et
al. (2021), also referencing their
code.121212https://github.com/tonyzhaozh/few-shot-learning Specifically, we
computed calibration parameters using the output probabilities of the content-
free prompt (e.g. prompt with "N/A" test input), and used these parameters to
update output probabilities when prompted with actual test inputs. However, we
found that applying this calibration procedure does not improve performance.
We further observe that when prompted with content-free inputs, the language
model already generates bias-free answers (e.g. generates "N/A" given "N/A"
test input) without calibration. This suggests there is no significant bias
towards specific answers for the case of split-antecedent resolution, whereas
current calibration techniques are designed for cases where the LM generates
biased answers given content-free inputs, (e.g. generates a label given "N/A"
test input).

Hyperparameter | Kate | Kate+ | Product | Mice-{2} | Mice-{5}
---|---|---|---|---|---
Mice-Sampling-{2} | Mice-Sampling-{5}
# prompts per example | 1 | 1 | $k$ | $\min(256,k^{2})$ | $\min(256,k^{2})$
Max # in-context per prompt | $\min(k,d_{x})$ | $\min(k,d_{x})$ | 1 | 2 | 5
Order of in-context | Ascend | Ascend | - | Ascend | Ascend
$P(y_{i}|z,x)$ threshold | 0.0 | 0.02 | 0.02 | 0.02 | 0.02
$P(y_{i}|x)$ threshold | 0.0 | 0.1 | 0.1 | 0.1 | 0.1
top-$k$ | - | 50 | - | - | -
top-$p$ | - | 0.95 | - | - | -

Table 7: Hyperparameters for in-context learning models. $k$ is the size of
training set ($k$-shot). $d_{x}$ depends on the length of test input $x$ and
retrieved in-context examples (Figure 3). Ascend denotes sorting the order of
in-context demonstrations from least to most similar examples (i.e. most
similar example closest to test input). Top-$k$ and top-$p$ are
hyperparameters for nucleus sampling used for Kate+.

##  Appendix C Baseline Implementation Details

For E2E (Fang et al., 2021), we run the TensorFlow code provided in the
original paper (following the same hyperparameters) to get the model
performance under our few-shot settings. Other baselines, including ProcBERT,
T5-3B, and T0-3B, are implemented using Huggingface Transformers (Wolf et al.,
2020). The hyperparameters used for these baselines are shown in Table 8.

Hyperparameter | ProcBERT |  T5-3B / T0-3B
---|---|---
# epochs | 200 | 20
batch size | 32 | 4
learning rate | 2e-5 | 1e-4
max. encoder length | 512 | 512
max. decoder length | - | 256
optimizer | AdamW | AdamW

Table 8: Hyperparameters for ProcBERT, T5-3B, and T0-3B.

Data Split | 4-shot | 8-shot | 16-shot | 32-shot | 64-shot | Train (full) | Dev-64 | Dev-256 | Test
---|---|---|---|---|---|---|---|---|---
# anaphors | 4 | 8 | 16 | 32 | 64 | 4,766 | 64 | 256 | 898
# antecedents | 13.4 | 24.2 | 61.2 | 132.8 | 266.0 | 21,673 | 293 | 996 | 4,016
# documents | 2.2 | 3.8 | 6.0 | 9.6 | 17.6 | 856 | 11 | 52 | 166
# sentences | 8.0 | 15.2 | 27.0 | 50.4 | 92.2 | 5,833 | 76 | 346 | 1,066
# tokens | 445.8 | 939.6 | 2,257.6 | 4,665.2 | 9,775.8 | 984,032 | 10,150 | 45,825 | 140,614

Table 9: Dataset splits details. The $k$-shot dataset statistics are averaged
over five random samples.


# To catch a chorus, verse, intro, or anything else: Analyzing a song with
structural functions

###### Abstract

Conventional music structure analysis algorithms aim to divide a song into
segments and to group them with abstract labels (e.g., ‘A’, ‘B’, and ‘C’).
However, explicitly identifying the function of each segment (e.g., ‘verse’ or
‘chorus’) is rarely attempted, but has many applications. We introduce a
multi-task deep learning framework to model these structural semantic labels
directly from audio by estimating “verseness,” “chorusness,” and so forth, as
a function of time. We propose a 7-class taxonomy (i.e., intro, verse, chorus,
bridge, outro, instrumental, and silence) and provide rules to consolidate
annotations from four disparate datasets. We also propose to use a spectral-
temporal Transformer-based model, called SpecTNT, which can be trained with an
additional connectionist temporal localization (CTL) loss. In cross-dataset
evaluations using four public datasets, we demonstrate the effectiveness of
the SpecTNT model and CTL loss, and obtain strong results overall: the
proposed system outperforms state-of-the-art chorus-detection and boundary-
detection methods at detecting choruses and boundaries, respectively.

Index Terms—  Music structure, segmentation, semantic labeling, Transformer,
SpecTNT

##  1 Introduction

In Music Structure Analysis (MSA) of audio, two tasks are defined:
_segmentation_ , where the aim is to divide a recording of a song into non-
overlapping segments, and _labeling_ (or grouping), where the aim is to label
the segments with symbols (e.g., ‘A’, ‘B’, etc.) to indicate how they are
grouped. However, these bits of information give a limited description of a
song. More advanced tasks include _hierarchical analysis_ , in which
segmentation and grouping must be estimated at multiple timescales, and
_semantic labeling_ , in which the labels should have meaning, such as
‘chorus’, ‘verse’, or ‘solo’. While hierarchical MSA has attracted sustained
attention [1, 2, 3, 4, 5], semantic MSA, despite having many uses, has rarely
been addressed: the most significant work on this task was over a decade ago
[6, 7].

Early MSA algorithms (see [8] for a review) were mostly non-supervised, based
on processing some version of a self-similarity matrix computed from
traditional audio features such as MFCCs or chroma. However, as large
annotated datasets became available (e.g., _SALAMI_ [9] and _Harmonix Set_
[10]), supervised approaches benefitted. For example, the usual way to
estimate segment boundaries was once to compute a ‘novelty function’ and
choose the points that maximized it [11]; Ullrich et al. used a subset of [9]
to train a model to estimate the ‘boundaryness’ of each instant given a long
context window, and achieved a new state-of-the-art [12]. Similarly, Wang et
al. [13] trained a model to estimate ‘chorusness’ (jointly with boundaries),
and achieved state-of-the-art _chorus detection_. (This is another sub-task of
MSA, and an early work on this topic [14] is alluded to by our title.) In this
paper, we extend [13] and aim to model ‘verseness’, ‘bridgeness’, and more,
directly from audio. Our system represents the first that is able to assign
generic structural semantic labels since [6], and the first in which the audio
content directly informs the label prediction.

We build on [13] by introducing two deep neural network (DNN) architectures to
MSA, _Harmonic-CNN_ [15] and _SpecTNT_ (Spectral-Temporal Transformer in
Transformer) [16]. Transformer architectures have not been used yet for MSA,
but have the potential to improve the temporal modeling at the representation
learning stage; they have earned great attention for their strength in
modeling sequential data [17, 18]. In particular, we adopt SpecTNT, a
hierarchical variant of Transformer that models the spectral and temporal
dimensions of an input spectrogram with two Transformer encoders. We also
propose to use the Connectionist Temporal Localization (CTL) loss [19] to
improve the temporal modeling.

A system capable of estimating semantic labels has clear uses: it would
enhance any application that already relies on MSA, such as structure-based
navigation systems [20] or automated remixing of sections from different songs
[21, 22]. Furthermore, the ability to analyze structure without requiring full
context of a song allows MSA to be applied to song fragments, or songs that
are very short (e.g., a TikTok song under 30 seconds long). For such inputs,
approaches that rely on self-similarity matrices and clustering may fail. An
automatic mastering system [23] could be improved by predicting the function
and structure of the input clips. Finally, a system not dependent on a full
song would be a first step toward a _real-time_ MSA system. Such a system
could be essential to applications in live contexts, such as a system that
controlled the visuals at a concert to match the music (as suggested in [8]).

To sum up, our proposed MSA system differs from most previous work in three
ways: it predicts semantic (not abstract) labels; it is supervised; and it is
non-context-dependent, meaning it does not rely on detecting repeating
sequences or relative novelty changes. The main contributions of this paper
are: a mapping used to consolidate information from disparate datasets (Sec.
2); the proposed approaches, including a system that represents the first
application of Transformers to MSA (Sec. 3); and the evaluations, including
ablation and cross-dataset studies using four public datasets (Sec. 4).

##  2 Structural Label Conversion

Structural analysis is recognized as an “ill-defined” problem [8], since the
“solution” is not unique: listeners usually disagree about the exact structure
of a piece. The SALAMI dataset was designed to mitigate this: section
groupings were annotated separately from function (inspired by [24]), and an
“Annotator’s Guide” [25] defined a valid set of function labels. The data
still contain many instances of conflicting labels, including dozens of cases
where one listener’s ‘chorus’ was another’s ‘verse’. However, there is far
more agreement than disagreement: if two annotators agree on the start time of
a section, they agree on the function label at least twice as often as they
disagree, and this rate only increases if we collapse synonymous groups of
labels: e.g., the Annotator’s Guide indicates ‘coda’, ‘fade-out,’ and ‘outro’
all have the same basic function. Therefore, we expect the data created by
annotators to be useful for this task.

To train a model to predict functions, we must strike a balance between our
goal—to model a rich set of semantic meanings—with our available means: a
motley set of datasets, collected with different standards, made up of
different genres, with functional terms that vary in specificity. Based on our
study of existing datasets, there is no standard naming system for structural
labels. For example, some use ‘refrain’ as chorus, some specify section sub-
types (e.g., ‘verse A’ in RWC [26]), and some allow compound functions (e.g.,
‘instchorus’, appearing in Harmonix Set). Before developing a model to
classify structural functions, we have to define a fixed taxonomy, and then
define rules to map the free-form annotations from several datasets onto the
same taxonomy. Any choice here will be a compromise among the utility of the
taxonomy, the amount of training data per class, and the validity of the
mappings.

We chose a 7-class taxonomy that is suited to Western pop music: ‘intro’,
‘verse’, ‘chorus’, ‘bridge’, ‘inst’ (i.e., instrumental), ‘outro’, and an
auxiliary class ‘silence’. This is equivalent to the set of 5 basic classes
named in the SALAMI Annotator’s Guide, plus ‘inst’ and ‘silence’. We then
define Algorithm 1, which gives a mapping for any annotated label that does
not match the 7-class taxonomy. The function conversion() iteratively tests
for the existence of substrings in the label. Since multiple substrings can
appear in one label, the order of elements in substrings determines the
priority of the matches; for instance, ‘instrumentalverse’ will be converted
to ‘verse,’ not ‘inst’, and ‘pre-chorus’ will be converted to ‘verse’ to
disentangle the build from the true chorus (as in [13]). Note that "end" is
used to mark the timestamp for the end of a song, and it is not regarded as a
function label. We found that substrings covers 99.3% of raw labels in the
existing datasets. The remaining labels mostly include instrument descriptors
(such as ‘guitar,’ ‘gtr,’ ‘riff,’ ‘spoken,’ ‘voice,’ and ‘groove’); after
listening to most examples, we decided to convert them all to "inst".

⬇

substrings = [

("silence", "silence"), ("pre-chorus", "verse"),

("prechorus", "verse"), ("refrain", "chorus"),

("chorus", "chorus"), ("theme", "chorus"),

("stutter", "chorus"), ("verse", "verse"),

("rap", "verse"), ("section", "verse"),

("slow", "verse"), ("build", "verse"),

("dialog", "verse"), ("intro", "intro"),

("fadein", "intro"), ("opening", "intro"),

("bridge", "bridge"), ("trans", "bridge"),

("out", "outro"), ("coda", "outro"),

("ending", "outro"), ("break", "inst"),

("inst", "inst"), ("interlude", "inst"),

("impro", "inst"), ("solo", "inst")]

def conversion(label):

if label == "end": return "end"

for s1, s2 in substrings:

if s1 in label.lower(): return s2

return "inst"

Algorithm 1: The conversion rule for a raw label.

##  3 Proposed Approach

Our proposed system pipeline is depicted in Fig. 1, and is conceptually
similar to the pipeline proposed in [13]. A song is first divided into
overlapping audio chunks. After audio feature extraction, the DNN model
predicts, for each chunk, two types of activation curves: one for boundaries
and one for each structural function. We propose two alternative strategies,
_instant_ and _multi-point_ , for the DNN model. In the _instant_ model, each
chunk leads to a single prediction vector indicating the likelihood of each
function label for the instant at the center of the chunk (as in [12]). Then,
the sequence of predictions forms the output curves. In the _multi-point_
model, predictions are made for every time point of an input chunk. We follow
the method of [13] to merge the outputs of all the overlapping chunks into the
final prediction curves. Finally, a few simple post-processing steps return
the output is a usable format: boundary timestamps and a single function label
per segment.

Fig. 1: Pipeline for the proposed system.

###  3.1 Label and Audio Feature Pre-processing

The steps to create a _chorus activation curve_ , the target curve for the
prediction of chorus segments, are given in [13]. We use the same steps, but
repeat them for the 6 additional function classes to create 7 _function
activation curves_. There is still only one _boundary activation curve_ , but
it now includes all boundaries (not just those between chorus and non-chorus
segments), as described in [12].

We smooth the transitions of the function activation curves using a 2-second-
wide Hann window: a 1-second ramp from 0 to 1 prior to the onset, and a
1-second ramp down after the offset, as in [13]. Regarding the “boundary
section,” we set a duration of 0.6 seconds for each boundary (whereas [13]
used 0.5 seconds).

As for audio feature extraction, we use harmonic representation (whereas [13]
used mel-spectrogram), since trainable harmonic filters help capture harmonic
information while preserving spectral-temporal locality. It has proven useful
in music auto-tagging [15].

###  3.2 Harmonic-CNN

Harmonic-CNN [15] is designed to mimic human perception by extracting the
spectrogram features through its harmonic representation front-end, as
harmonic structure is known to play a key role in the human auditory system.
Its network architecture contains several temporal pooling layers with the aim
of predicting a single target for each music tag, so we believe it can be a
good fit to the goal of the _instant_ model in this work. Given the harmonic
representation input, it applies seven 2D-convolutional layers in the back-end
to extract high-level features, followed by two dense layers to predict the
target of a time-step for the boundary and function activations.

###  3.3 Spectral-Temporal Transformer

Although the Transformer architecture has shown remarkable performance in
modeling sequential data such as text [18] and musical scores [27, 28], its
data-hungry nature makes it inapt for tasks with little training data. The
total number of current annotated songs for public MSA datasets is less than
3000, which is likely insufficient to train a satisfactory Transformer model.

By contrast, SpecTNT (adopted in this work) has shown good performance in beat
and downbeat tracking [29], vocal melody extraction [16], and chord
recognition [16] using datasets even smaller than those for MSA. The basic
principle of SpecTNT arises from the interaction between two levels of
Transformer encoders, namely a _spectral encoder_ and a _temporal encoder_.
The spectral encoder is responsible for extracting the spectral features via
_Frequency Class Tokens_ (FCTs) for each time-step, where an FCT is an
aggregated embedding that characterizes harmonic and timbral information. The
temporal encoder then exchanges local information (i.e., FCTs) along the time
axis; this self-attention step can help discover structural patterns related
to novelty, homogeneity, and repetition [8]. For example, the self-attention
mechanism can allow frames around a boundary to attend to the boundary, and
frames with the same function to attend to one another. Owing to its
hierarchical design, SpecTNT permits a smaller number of parameters as
compared to the original Transformer [17], and we expect this attribute can
help improve the generalization for the MSA task.

A SpecTNT model consists of three modules: a two-dimensional residual network
(ResNet) [30] at the front-end to extract intermediate information from the
input harmonic representation; a stack of SpecTNT blocks; and a linear layer
to output the target probability at all time-steps. Therefore, it serves as
the _multi-point_ model in this work. For ResNet, each convolutional layer
uses a kernel size of 3. Then, we use 5 SpecTNT blocks. In each block, we use
96 feature maps with 4 attention heads for the spectral encoder, and 96
feature maps with 8 attention heads for the temporal encoder.

Fig. 2: Example of post-processing for a test song _Complicated_ by Avril
Lavigne, using SpecTNT (with 24s and CTL). The top two rows display the raw
outputs of the activation curves. “Raw Function” shows the argmax labels at
each time-step of the DNN output.

###  3.4 Post-processing

The raw outputs include a boundary curve and 7 structural function curves. In
an end-to-end manner, one can assign the structural function with the largest
probability (i.e., arg-max) for each time-step. However, as illustrated in
Fig. 2, this ‘raw function’ output can be inaccurate near boundaries and
inconsistent within a segment. To get more usable output, we first interpret
the boundary activation curve using the peak-picking method proposed in [12].
Then, we simply choose the function labels with the largest average
probability in each segment. After this post-processing, we can see a more
accurate result (see Fig. 2).

###  3.5 Training Loss

To jointly model the boundaries and structural functions, we define two types
of losses: _boundary loss_ and _function loss_. Each is calculated by summing
the weighted binary cross-entropy between the prediction and target at each
time-step of the corresponding activation curve. Following the heuristic used
in [13], we use a weight 0.9 for boundary loss and 0.1 for function loss when
combining them, since the boundary curve is sparse and more difficult to
learn.

To enhance the coherence of the predictions and reduce the fragmentation, we
propose to add the _Connectionist Temporal Localization (CTL)_ loss [19] to
the function loss. The CTL loss aims to model the sequential order of labels.
In popular music, the order of sections follows regular patterns: a song is
unlikely to start with an outro or end with an intro—this is the premise of
[7]. We use the raw section annotations (i.e., before they are converted to
the activation curves) to form the target sequence of section tokens (e.g.,
[‘intro’, ‘verse’, ‘chorus’, ‘inst’]). Then, the CTL loss is calculated
between the prediction (a $T\times 7$ probability matrix) and the sequence of
$S$ tokens, where $T$ and $S$ are the numbers of time-steps and section
tokens, respectively, in a training chunk.

##  4 Experiments

###  4.1 Implementation Details

For STFT before the harmonic representation, we adopt a window length of 1024
with a hop size of 512 on audio signals of 16 kHz sampling rate. The time
resolution for the activation curves is about 5.2 per second (i.e., a target
per 0.192 seconds). We found using longer duration for the input chunks led to
better function prediction, but poorer boundary detection accuracy. Thus, we
adopt 24-second chunks and will show result of using 36 seconds in an ablation
study.

Data augmentation is vital to training an effective Transformer model. We use
the torchaudio_augmentations package [31] to randomly add noise, adjust gain,
or apply high/low-pass filters for each training sample. We use a random
sampling strategy to load a random chunk into a batch, instead of sequentially
loading a chunk from the beginning of a training song as adopted in [13]. That
is, we first enumerate all the valid training chunks into a list by using a
24-second sliding window with a 3-second hop on every song. If the last chunk
exceeds the end of a song, we pad zeros at its end. Then, the data loader
draws a chunk uniformly from the list during training. Therefore, a batch can
include chunks from different locations of different songs. This technique can
lead to faster convergence and better results compared to that in [13].

We use PyTorch 1.8 and Adam optimizer [32] with 0.0005 learning rate, 0.9
weight decay, and 2 epochs of patience. An epoch runs 500 mini-batches with a
batch size of 128. We conduct 100 training epochs and use the model with the
best validation result for testing.

###  4.2 Experimental Configurations

We use four public datasets for experiments: _Harmonix Set_ [10], _SALAMI-pop_
[9], _RWC-Pop_ [26], and _Isophonics_ [33]. _Harmonix Set_ contains 912
western pop songs. Since the original audio was not available, we searched for
the correct audio versions and manually refined the annotations. For _SALAMI-
pop_ , we select a subset of 274 songs (with 445 annotations) in the “popular”
genre. In these annotations, the label ‘no_function’ appears often, but this
is the result of a parsing error; we replace every instance of it with the
preceding section’s function label. _RWC-Pop_ is the subset of 100 popular
songs in _RWC_. We use the original annotations by AIST. _Isophonics_ contains
277 songs from The Beatles, Carole King, Michael Jackson, and Queen, but we
use the _TUT-Beatles_ annotations [34] for the 174 Beatles songs.

We conduct evaluations of two types. First, in the ablation study, we use
_Harmonix Set_ in a 4-fold cross-validation (CV) manner, but include _SALAMI-
pop_ , _RWC-Pop_ , and _Isophonics_ in the training set at every iteration.
Second, we carry out cross-dataset evaluations: each of _SALAMI-pop_ , _RWC-
Pop_ , and _Isophonics_ in turn serves as the test set, and the remaining
datasets are used for training. We randomly split 10% of the training set to
form the validation set.

Besides regular MSA, we are also interested in the performance of chorus
detection, where we only leave the boundary and section annotations involving
a ‘chorus.’ The mir_eval package [35] is used to compute the following
evaluation metrics: (1) _HR.5F_ : F-measure of hit rate at 0.5 seconds; (2)
Accuracy (_ACC_): the frame-wise accuracy between the predicted function label
and the converted ground-truth label; (3) _PWF_ : F-measure of pair-wise frame
clustering; (4) _Sf_ : F-measure of normalized entropy score; (5) _CHR.5F_ :
F-measure of ‘chorus’ boundary hit rate at 0.5 seconds; (6) _CF1_ : F-measure
of pair-wise frames for ‘chorus’ and ‘non-chorus’ sections [13]. The
definitions of (1), (3), (4), and (5) can be found in [8].

###  4.3 Baseline Methods

We include several baseline methods from prior work. “Scluster” is the
spectral clustering algorithm [1] implemented in MSAF [36]. “DSF + Scluster”
uses learned deep structure features (DSF) [37] as the inputs for Scluster to
predict the MSA results. It employs a Harmonic-CNN trained with metric
learning loss in a supervised fashion, and is trained only on _Harmonix Set_.
To perform chorus detection for the above two methods, we pick chorus sections
using the heuristic in [13], i.e., _Max-dur_ , which chooses the segment group
that covers the greatest duration of a song as the choruses. “CNN-Chorus” [13]
uses a similar approach to this work, but with a different DNN model and
training strategy. It was developed to only detect the chorus sections. For
_RWC-Pop_ and _Isophonics_ , which are used in MIREX, we include for
comparison the results of three top-performing MIREX submissions: GS3 [3], and
SMGA1 and SMGA2 [38].

##  5 Discussion and Conclusion

Results are presented in Table 1. The main proposed systems are “Harmonic-CNN”
(the instant method) and “SpecTNT (24s, CLT)” with CTL loss (the multi-point
method). In the ablation study, we compare these to SpecTNT without CTL; a
regular Transformer [39] with CTL; and SpecTNT with CTL and a longer chunk
size (36s). The regular Transformer contains only the temporal encoders [18],
so it is non-hierarchical and has no spectral encoder. It is clear that this
system performs worse than SpecTNT, most likely because the training data is
insufficient. Comparing SpecTNT with and without CTL loss, we can see that
including the loss improves performance slightly on all metrics, demonstrating
its usefulness. We also observe that SpecTNT with longer inputs can improve
the function prediction, but the boundary detection performance drops,
possibly because the model has over-fit the training data. To prioritize
boundary accuracy, we use the shorter window in the next evaluation.

| HR.5F | ACC | PWF | Sf | CHR.5F | CF1
---|---|---|---|---|---|---
Ablation Study
|  Harmonix Set
Scluster [1] | .263 | - | .586 | .641 | .171 | .534
DSF + Scluster [37] | .497 | - | .689 | .743 | .326 | .611
CNN-Chorus [13] | - | - | - | - | .371 | .692
Harmonic-CNN | .559 | .680 | .670 | .682 | .462 | .784
Transformer (24s, CTL) | .521 | .640 | .655 | .649 | .399 | .755
SpecTNT (24s) | .565 | .690 | .687 | .702 | .491 | .813
SpecTNT (24s, CTL) | .570 | .701 | .700 | .714 | .501 | .815
SpecTNT (36s, CTL) | .558 | .723 | .712 | .724 | .476 | .831
Cross-dataset Evaluation
|  SALAMI-pop (subset of MIREX 2012 dataset)
Scluster [1] | .305 | - | .545 | .572 | .196 | .418
DSF + Scluster [37] | .447 | - | .615 | .653 | .272 | .573
CNN-Chorus [13] | - | - | - | - | .308 | .602
Harmonic-CNN | .477 | .525 | .631 | .636 | .340 | .777
SpecTNT (24s, CTL) | .490 | .544 | .651 | .632 | .357 | .811
|  RWC-Pop (MIREX 2010 RWC collection)
GS3 (2015) [3] | .524 | - | .542 | .684 | - | -
SMGA2 (2012) [38] | .246 | - | .688 | .733 | - | -
DSF + Scluster [37] | .438 | - | .704 | .739 | .343 | .653
Harmonic-CNN | .571 | .646 | .719 | .694 | .396 | .800
SpecTNT (24s, CTL) | .623 |  .675 | .749 | .728 | .465 | .847
|  Isophonics (MIREX 2009 Collection)
GS3 (2015) [3] | .564 | - | .567 | .686 | - | -
SMGA1 (2012) [38] | .228 | - | .653 | .700 | - | -
Harmonic-CNN | .543 | .499 | .611 | .598 | .339 | .670
SpecTNT (24s, CTL) | .590 | .550 | .635 | .614 | .401 | .733

Table 1: Results on four datasets.

Looking at all the studies, we observe that SpecTNT (24s, CTL) consistently
outperforms Harmonic-CNN in all metrics; i.e., the multi-point model beats the
instant model. This may be due to improved temporal coherence in the estimates
provided by SpecTNT’s hierarchical Transformer. Either way, the multi-point
method used with SpecTNT is 15.6 times more efficient at prediction time than
the instant method required by Harmonic-CNN, since the former requires 1 call
to the model every 3 seconds (as a chunk hops every 3 sec), whereas the latter
requires 5.2 calls per second (i.e., time resolution). The multi-point method
also significantly outperforms the Scluster-based methods at boundary
detection (i.e., HR.5F and CHR.5F), showing that directly modeling the
boundaries with an activation curve is better than doing feature clustering.
In addition, our multi-point model can outperform the state-of-the-art
boundary detection approach (i.e., a CNN-based instant model, Harmonic-CNN, or
GS3) by a wide margin, demonstrating the strength of SpecTNT for boundary
detection. We also find that modeling more function types may help the system
to recognize chorusness more accurately: Harmonic-CNN outperforms its
predecessor CNN-Chorus (which only saw chorus and non-chorus labels) at chorus
detection.

Our proposed approach is less successful in terms of PWF and Sf on
_Isophonics_. A closer investigation reveals that there are relatively few
‘chorus’ (or ‘refrain’) labels in the songs of the Beatles, which is a classic
evaluation corpus for MSA, but contains many songs with experimental
structure. The mismatch between the Beatles songs and the remainder of the
datasets, which have more choruses, may account for this poorer performance.
However, inspection of other outputs shows that our model often makes
justifiable errors: for example, it often predicts an ‘outro’ when the
annotation says ‘chorus’, but when the song is in fact fading out, like a
‘fade-out chorus’. Fig. 2 shows such an example, but also hints at a solution:
allowing multiple labels to be potentially predicted per segment.

We have proposed a Transformer-based MSA system that can predict meaningful
segment labels, even without context. However, in this work we only study the
performance of full-track prediction; to demonstrate its robustness to song
fragments remains future work.

## References

  * [1] B. McFee and D. Ellis,  “Analyzing song structure with spectral clustering,”  in ISMIR, 2014, pp. 405–410.
  * [2] B. McFee, O. Nieto, and J. P. Bello,  “Hierarchical evaluation of segment boundary detection.,”  in ISMIR, 2015, pp. 406–412.
  * [3] T. Grill and J. Schlüter,  “Music boundary detection using neural networks on combined features and two-level annotations.,”  in ISMIR, 2015, pp. 531–537.
  * [4] C. J. Tralie and B. McFee,  “Enhanced hierarchical music structure annotations via feature level similarity fusion,”  in ICASSP, 2019, pp. 201–205.
  * [5] J. Salamon, O. Nieto, and N. J. Bryan,  “Deep embeddings and section fusion improve music segmentation,”  in ISMIR, 2021, pp. 594–601.
  * [6] J. Paulus,  Signal processing methods for drum transcription and music structure analysis,  Ph.D. thesis, Tampere University of Technology, Tampere, Finland, 2009\.
  * [7] J. Paulus and A. Klapuri,  “Labelling the structural parts of a music piece with Markov models,”  in Computer Music Modeling and Retrieval: Genesis of Meaning in Sound and Music, vol. 5493, pp. 166–176. Springer-Verlag, 2010.
  * [8] O. Nieto, G. J. Mysore, C.-I Wang, J. B. L. Smith, J. Schlüter, T. Grill, and B. McFee,  “Audio-based music structure analysis: Current trends, open challenges, and applications,”  Trans. ISMIR, vol. 3, no. 1, 2020.
  * [9] J. B. L. Smith, J. A. Burgoyne, I. Fujinaga, D. D. Roure, and J. S. Downie,  “Design and creation of a large-scale database of structural annotations,”  in ISMIR, 2011, pp. 555–560.
  * [10] O. Nieto, M. McCallum, M. Davies, A. Robertson, A. Stark, and E. Egozy,  “The Harmonix Set: Beats, downbeats, and functional segment annotations of western popular music,”  in ISMIR, 2019, pp. 565–572.
  * [11] J. Foote,  “Automatic audio segmentation using a measure of audio novelty,”  in IEEE ICME, 2000, pp. 452–455.
  * [12] K. Ullrich, J. Schlüter, and T. Grill,  “Boundary detection in music structure analysis using convolutional neural networks,”  in ISMIR, 2014, pp. 417–422.
  * [13] J.-C. Wang, J. B. L. Smith, J. Chen, X. Song, and Y. Wang,  “Supervised chorus detection for popular music using convolutional neural network and multi-task learning,”  in ICASSP, 2021, pp. 566–570.
  * [14] M. A. Bartsch and G. H. Wakefield,  “To catch a chorus: Using chroma-based representations for audio thumbnailing,”  in Proc. WASPAA, 2001, pp. 15–18.
  * [15] M. Won, S. Chun, O. Nieto, and X. Serra,  “Data-driven harmonic filters for audio representation learning,”  in ICASSP, 2020, pp. 536–540.
  * [16] W.-T. Lu, J.-C. Wang, M. Won, K. Choi, and X. Song,  “SpecTNT: A time-frequency transformer for music audio,”  in ISMIR, 2021, pp. 396–403.
  * [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin,  “Attention is all you need,”  NeurIPS, vol. 30, 2017.
  * [18] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova,  “BERT: Pre-training of deep bidirectional transformers for language understanding,”  in NAACL, 2019.
  * [19] Y. Wang and F. Metze,  “Connectionist temporal localization for sound event detection with sequential labeling,”  in ICASSP, 2019, pp. 745–749.
  * [20] M. Goto,  “SmartMusicKIOSK: Music listening station with chorus-search function,”  in ACM Symposium on User Interface Software and Technology, 2003, pp. 31–40.
  * [21] M. E. P. Davies, P. Hamel, K. Yoshii, and M. Goto,  “AutoMashUpper: Automatic creation of multi-song music mashups,”  IEEE Tran. Audio, Speech, and Language Processing, vol. 22, no. 12, pp. 1726–1737, 2014.
  * [22] J. Huang, J.-C. Wang, J. B. L. Smith, X. Song, and Y. Wang,  “Modeling the compatibility of stem tracks to generate music mashups,”  AAAI Conference, 2021.
  * [23] B. De Man, R. Stables, and J. D. Reiss,  Intelligent Music Production,  Focal Press, 2019.
  * [24] G. Peeters and E. Deruty,  “Is music structure annotation multi-dimensional? A proposal for robust local music annotation,”  in Int. Workshop on LSAS, 2009, pp. 75–90.
  * [25] SALAMI,  “Annotator’s guide,” https://github.com/DDMAL/ salami-data-public,  Last accessed on Feb 11, 2022.
  * [26] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka,  “RWC Music Database: Popular, classical and jazz music databases,”  in ISMIR, 2002, vol. 2.
  * [27] C.-Z. A. Huang et al.,  “Music Transformer: Generating music with long-term structure,”  in ICLR, 2018.
  * [28] M. Zeng, X. Tan, R. Wang, Z. Ju, T. Qin, and T.-Y. Liu,  “MusicBERT: Symbolic music understanding with large-scale pre-training,”  in Findings of ACL, 2021, pp. 791–800.
  * [29] Y.-N. Hung, J.-C. Wang, X. Song, W.-T. Lu, and M. Won,  “Modeling beats and downbeats with a time-frequency Transformer,”  ICASSP, 2022.
  * [30] M. Won, A. Ferraro, D. Bogdanov, and X. Serra,  “Evaluation of CNN-based automatic music tagging models,”  in Sound and Music Computing Conference, 2020.
  * [31] J. Spijkervet,  “Spijkervet/torchaudio-augmentations,” 2021,  https://zenodo.org/record/4748582.
  * [32] D. P. Kingma and J. Ba,  “Adam: A method for stochastic optimization,”  in ICLR, 2015.
  * [33] M. Mauch, C. Cannam, M. Davies, S. Dixon, C. Harte, S. Kolozali, D. Tidhar, and M. Sandler,  “OMRAS2 metadata project 2009,”  in ISMIR Late Breaking and Demo, 2009.
  * [34] J. Paulus,  “Improving Markov model based music piece structure labelling with acoustic information,”  in ISMIR, 2010, pp. 303–308.
  * [35] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, and D. P. W. Ellis,  “mir_eval: A transparent implementation of common MIR metrics,”  in ISMIR, 2014.
  * [36] O. Nieto and J. P. Bello,  “Systematic exploration of computational music structure research,”  in ISMIR, 2016, pp. 547–553.
  * [37] J.-C. Wang, J. B. L. Smith, W.-T. Lu, and X. Song,  “Supervised metric learning for music structure features,”  in ISMIR, 2021, pp. 730–737.
  * [38] J. Serra, M. Müller, P. Grosche, and J. L. Arcos,  “The importance of detecting boundaries in music structure annotation,”  Proc. MIREX, 2012.
  * [39] M. Won, K. Choi, and X. Serra,  “Semi-supervised music tagging transformer,”  in ISMIR, 2021, pp. 769–776.