# Muppet: Massive Multi-task Representations with Pre-Finetuning

Armen Aghajanyan
Facebook
<EMAIL_ADDRESS>
Anchit Gupta
Facebook
<EMAIL_ADDRESS>
Akshat Shrivastava
Facebook
<EMAIL_ADDRESS>
Xilun Chen
Facebook
<EMAIL_ADDRESS>
Luke Zettlemoyer
Facebook
<EMAIL_ADDRESS>
Sonal Gupta
Facebook
<EMAIL_ADDRESS>

###### Abstract

We propose pre-finetuning, an additional large-scale learning stage between
language model pre-training and fine-tuning. Pre-finetuning is massively
multi-task learning (around 50 datasets, over 4.8 million total labeled
examples), and is designed to encourage learning of representations that
generalize better to many different tasks. We show that pre-finetuning
consistently improves performance for pretrained discriminators (e.g. RoBERTa)
and generation models (e.g. BART) on a wide range of tasks (sentence
prediction, commonsense reasoning, MRC, etc.), while also significantly
improving sample efficiency during fine-tuning. We also show that large-scale
multi-tasking is crucial; pre-finetuning can hurt performance when few tasks
are used up until a critical point (usually above 15) after which performance
improves linearly in the number of tasks.

##  1 Introduction

The recent success of language model pre-training Devlin et al. (2018); Liu et
al. (2019b); Lewis et al. (2019); Raffel et al. (2019); Radford et al. (2019)
is remarkable, at least in part, due to the exclusive use of self supervision,
without any manually labeled data. For many tasks, however, we already have
training examples for related problems, which we should be able to leverage.
Recent work has shown gains from fine-tuning schemes that are multi-task
Raffel et al. (2019); Khashabi et al. (2020) and multi-stage Liu et al.
(2019a), but it can be difficult to know which intermediate tasks will best
transfer Raffel et al. (2019). In this paper, we show that multi-task
supervised tuning, if done at a sufficiently large scale with many different
tasks, can be an effective second stage of task-agnostic pre-training,
removing the need to pre-select the best intermediate tasks.

More specifically, in addition to the standard pre-training/fine-tuning
methodology of learning language tasks, we introduce a new intermediate stage,
pre-finetuning. Pre-finetuning involves a massive multi-task learning step
(4.8 million total training examples) performed on around 50 classification,
summarization, question answering, and common sense reasoning tasks. We
believe we are the first to investigate multi-task learning at this scale in
terms of both number and types of tasks. We show, in particular, that standard
multi-tasking schemes can be unstable and often fail to learn high quality
representations. However, we introduce a new training scheme which uses loss
scaling and task-heterogeneous batches so that gradient steps are more evenly
balanced across multiple different competing tasks, greatly improving training
stability and overall performance. We call our pre-finetuned models MUPPET;
Massive Multi-task RePresentation with PrE-fineTuning.

Through extensive experiments, we show that incorporating pre-finetuning to
RoBERTa Liu et al. (2019b) and BART Lewis et al. (2019) models yields
consistent improvements, including new state-of-the-art performance for RTE
Bentivogli et al. (2009) and HellaSWAG Zellers et al. (2019), without having
to specify specific intermediate transfer tasks. These gains are particularly
strong in the low resource regime, where there is relatively little labeled
data for fine-tuning. We also study why pre-finetuning outperforms previous
multi-tasking schemes. We first compare different optimization techniques to
stabilize training, and find it important to use task-heterogeneous batches
with task-rebalancing loss scaling. We also show that scale is crucial for
effective multi-task learning. We empirically see a critical point in terms of
the number of tasks (usually over 15); having fewer tasks degrades
representations, while having more seems to improve performance linearly as
far as we were able to scale.

To summarize, our contributions include:

  * •

We show that we can further improve pre-trained representations with an
additional stage we call pre-finetuning, which utilizes massively multi-task
learning. We show standard pre-trained representations, when further refined
with pre-finetuning consistently improve performance on downstream tasks.

  * •

We introduce a new multi-task training scheme for effective learning at scale,
which uses loss scaling and task-heterogeneous batches.

  * •

We explore the effects of scale on multi-task learning and show the existence
of critical points in multi-task training, beyond which increasing the number
of tasks improves generalizable representations.

  * •

We conduct a study surrounding the data efficiency of standard pre-trained
representations and their respective pre-finetuned counterparts. We show that
the pre-finetuned models consistently require less data for fine-tuning.

##  2 Related Work

Multi-task learning has been an increasingly active topic in recent
literature. Recent advances such as MT-DNN show that by leveraging multi-task
learning, we can further improve performance on several language benchmarks on
top of traditional pre-training (Liu et al., 2019a). However, T5 (Raffel et
al., 2019) shows that incorporating multi-task learning ontop of larger models
does not improve upon the standardized pre-training / finetuning. Thus the
effect of multi-task learning across different pre-training methods is not
fully understood.

Recently Khashabi et al. (2020) showed how doing MTL training on a range of QA
tasks can improve the performance of T5 by taking advantage of cross dataset
transfer. Unlike our approach, they convert all the data to a seq2seq format,
operate on a smaller MTL scale, have a different batching strategy, and focus
solely on improving QA tasks. Our work shows how even seemingly very different
datasets, for example, summarization and extractive QA, can help each other by
improving the model’s representations.

Our work aims to explore multi-task learning at a much larger scale; by
incorporating a larger number of tasks, we show that we can consistently
improve several language benchmarks from several domains. Contrary to T5, we
show that incorporating a secondary stage of multi-task learning does lead to
better representations. In §5 we demonstrate the effectiveness of multi-task
learning to be coming from the large scale of our MTL setup.

##  3 Pre-Finetuning Through Massive Multitask Learning

Previous work has reported mixed results from experiments on multi-task
learning Liu et al. (2019a); Raffel et al. (2019). In general, it can be
challenging to balance the losses from different tasks; upsampling can lead to
overfitting low resource tasks, and downsampling can lead to improper learning
of specific tasks. This difficulty is particularly pronounced when operating
at the scale of experiments we show in Section 5.1, where there are more
diverse tasks than previously considered. This section presents our pre-
finetuning approach that leads to more stable and accurate multi-task training
by introducing new optimization, loss scaling, and task sampling schemes to
balance each minibatch’s updates better.

###  3.1 Tasks and Losses

#### Diverse Tasks

To learn general language representations, we include a variety of tasks
across many domains. We select language tasks across four different domains:
classification, commonsense reasoning, machine reading comprehension, and
summarization. In Table 1, we show the break down of each of the task types
along with the number of samples used from each during pre-finetuning. In
total our multi-task set up learns over 4.8 supervised samples across 4
families of tasks.

Task Type | # Datasets | # Train | # Eval
---|---|---|---
Classification | 26 | 2.9M | 188K
Summarization | 4 | 524K | 30K
MRC | 6 | 1.05M | 123M
CommonSense | 10 | 360K | 49K
Total | 46 | 4.8M | 390K
Table 1: Break down of MTL pre-finetuning datasets. The table shows the number
of datasets we used per task type and the number of samples in training and
evaluation sets.

A full list of all of the datasets we leverage for pre-finetuning is described
in appendix §A.1.

#### Standard Losses

To train on several datasets, our model contains task-specific heads, each
optimizing for a task-specific loss. The loss functions are summarized in
table 2. Each loss is scaled with loss scaling described in §3.3. After loss
scaling, the gradients from each task are averaged before doing the model
update step.

Task Type | Loss Function
---|---
Classification | Cross Entropy (CE)
Summarization | Label Smoothed CE Szegedy et al. (2015)
MRC | Span Prediction Seo et al. (2016)
Commonsense | Sentence Ranking Loss Liu et al. (2019b)
Table 2: Description of loss functions for each task type. Note for
summarization the label smoothed cross entropy loss is averaged across tokens.

###  3.2 Optimization

We show two strategies to learn multi-task representations at scale:
Accumulating Gradients Across Tasks (Heterogeneous Batches) and Leveraging
Better Finetuning.

#### Accumulating Gradients Across Tasks

Our model is trying to optimize not a single objective but several potentially
competing objectives to create a unified representation across several tasks
during model training. During gradient descent, moving along the gradient of a
single task may not be the optimal direction for the model to move to learn a
single unified representation across tasks. To overcome this, we ensure each
batch our model optimizes consists of several tasks. Each worker samples a
random batch from our set of tasks and computes a gradient, accumulated for
the final update. Empirically we use 64 GPUs for pre-finetuning, resulting in
each batch consisting of gradients across 64 sampled tasks. In §5.2 we show
how such a strategy allows for our model to arrive at a better representation
for end task finetuning.

#### Better Finetuning

Instead of starting from scratch, we initialize our model with representations
learned from self-supervised pre-training in pre-finetuning. This can inherit
the knowledge captured in the pre-trained representations and speed up
training. Mosbach et al. (2020) show that standard fine-tuning of pre-trained
models can be unstable, which may be aggravated in our case as we are training
on a diverse set of tasks simultaneously. Therefore, we employ the R3F/R4F
methods Aghajanyan et al. (2020) to combat this issue. In particular, R3F/R4F
consists of an additional loss term, ensuring that small perturbations to the
input space result in similar representations, which can be used to learn more
robust representations during pre-finetuning.

In early experimentation, we found that R3F was pivotal in getting MUPPET to
work for BART. All other fine-tuning and pre-finetuning was done using
standard SGD.

###  3.3 Loss Scaling

Loss scaling methods introduce a multiplicative reweighting of individual
losses per data-point. Various loss scaling techniques have been proposed,
from dynamic scaling by inverse training loss to simple scaling by the number
of data-points in respective datasets (Chen et al., 2018).

As pre-finetuning optimizes several different types of tasks and datasets,
each having its own output spaces, loss scaling becomes essential to ensure
stable training. We attempted various forms of loss-scaling throughout initial
experimentation, but the most effective was the novel method we describe
below.

Let us denote $\mathcal{L}_{i}(x_{i},y_{i};\theta)$ as the loss for datapoint
$i$ for a model parameterized by $\theta$. Remember that the loss depends on
the type of task (commonsense loss is different from binary classification).
Furthermore let $n:\mathbb{N}\rightarrow\mathbb{N}$ be a function which for
each data-point returns the number of predictions $\mathcal{L}$ operates over.
For example, for binary classification, $n$ would return two, while for
generation, $n$ would return the size of the vocabulary (since we average
across loss per token generated). We scale data-point loss so that, if the
class distribution were uniformly distributed along with our models
predictions, all of our losses would have equivalent values.

$\mathcal{L}^{scaled}_{i}(x_{i},y_{i};\theta)=\frac{\mathcal{L}_{i}(x_{i},y_{i};\theta)}{\log{n(i)}}$
(1)

We found that this static scaling worked incredibly well, outperforming other
loss scaling methods in early experimentation.

###  3.4 Sampling

Another approach to balancing various tasks in a multi-task set up is to up-
sample smaller datasets and down-sample larger ones to achieve more uniformity
between dataset sizes.

Existing results for dataset sampling methods in multi-task learning are
conflicting, but recent work has shown that it does not work well for multi-
task learning of pre-trained representations. For example, T5 showed that all
various forms of sampling did not improve overusing the natural size of
datasets (Raffel et al., 2019).

We also found that sampling datasets were consistently detrimental for multi-
task learning over pre-trained representations during initial experimentation.
Specifically, we saw unmanageable over-fitting and stability issues. Therefore
we opt for maintaining the natural distribution of the datasets throughout all
of our experiments.

###  3.5 Experimental Setup

We selected RoBERTa (Liu et al., 2019b) and BART (Lewis et al., 2019) as our
initial pre-trained models to further pre-finetune. For each task type we use
a different prediction scheme. Every Sentence Prediction dataset gets a
separate classification head, for Commonsense and MRC we utilize a separate
unified head for each task. For Summarization, we do not add any parameters
and use the BART decoder and output layer as is. Experimentally we saw using a
different head per individual Commonsense and MRC datasets lead to severe
overfitting.

For both models, we do the pre-finetuning procedure for both the Base and
Large models. We trained each model configuration with 64 GPUs until
convergence. Dependent on configuration, this ranged from a day to 4 days. We
include the hyper-parameters used per pre-finetuning run in the Appendix in
Section §A.2.

##  4 Empirical Results

We first show that pre-finetuning improves the representations of pre-training
models. To do so, we fine-tune our pre-finetuned models on a large set of
tasks.

For each of the individual downstream tasks, we use a fixed hyper-parameter
search to optimize over simple hyperparameters such as learning rate, Adam
$\epsilon$ (Kingma and Ba, 2014) and dropout (Srivastava et al., 2014). We
present our results in two tables. Table 3 shows our results on the GLUE
benchmark (Wang et al., 2018) as well as two MRC tasks; SQuAD (Rajpurkar et
al., 2016a) and ReCoRD (Zhang et al., 2018). Table 4 reports results on other
Sentence Prediction tasks as well as Commonsense tasks. We also include
results from MT-DNN Liu et al. (2019a), ELECTRA Clark et al. (2020),111For
ELECTRA results we leverage the results presented in the ELECTRA github
https://github.com/google-research/electra#expected-results and RoBERTa Liu et
al. (2019b) models. For Summarization tasks we show that our pre-finetuned
BART model outperforms all other summarization baselines. Both of these tables
report over data-sets available during the pre-finetuning stage.

Given that our pre-finetuned models now have an understanding of the task at
hand through the use of classification heads, we have a choice during
finetuning on whether or not to use these heads. In general we found re-using
heads to be beneficial for MRC, Commonsense and Sentence Prediction tasks with
small dataset size.

| GLUE | MRC |
---|---|---|---
| MNLI | QQP | RTE | QNLI | MRPC | SST-2 | SQuAD |
RoBERTa-B | 87.6 | 91.9 | 78.7 | 92.8 | 90.2 | 94.8 | 82.6 |
\+ MUPPET | 88.1 | 91.9 | 87.8 | 93.3 | 91.7 | 96.7 | 86.6 |
RoBERTa-L | 90.2 | 92.2 | 88.1 | 94.7 | 90.9 | 96.4 | 88.7 |
\+ MUPPET | 90.8 | 92.2 | 92.8 | 94.9 | 91.4 | 97.4 | 89.4 |
BART | 89.9 | 92.5 | 87.0 | 94.9 | 90.4 | 96.6 |  |
\+ MUPPET | 89.9 | 92.7 | 92.4 | 94.6 | 92.2 | 96.9 |  |
ELECTRA-B | 88.8 | 91.5 | 82.7 | 93.2 | 89.5 | 95 | 80.5 |
ELECTRA-L | 90.9 | 92.4 | 88.0 | 95.0 | 90.8 | 96.9 | 88.1 |
MT-DNN | 87.1 | 91.9/89.2 | 83.4 | 92.9 | 91.0/87.5 | 94.3 | - |
Table 3: We present results for the GLUE benchmark task and a MRC dataset. Bolded numbers show the MUPPET vs. base model, underline marks the best number. If not explicitly stated, the results are showing the accuracy of the evaluation set. For the MRC tasks, we report both exact match (EM) and F1 as is standard in the literature. For SQuAD, we reused the task head from pre-finetuning. | SP | Commonsense | Summarization
---|---|---|---
| BoolQ | CQA | HellaSwag | OpenQA | CNN/DailyMail | Gigaword | Reddit TIFU
RoBERTa-B | 82.0 | 66.2 | 65.1 | 63.8 | - | - | -
\+ MUPPET | 83.8 | 69.4 | 69.0 | 64.6 | - | - | -
RoBERTa-L | 86.4 | 78.1 | 83.4 | 73.6 | - | - | -
\+ MUPPET | 87.5 | 79.2 | 86.4 | 74.4 | - | - | -
BART | 86.2 | 78.1 | 84.1 | 71.4 | 44.16/21.28/40.90 | 39.29/20.09/35.65 | 24.19/8.12/21.31
\+ MUPPET | 86.9 | 74.8 | 75.9 | 70.8 |  44.45/21.25/41.4 |  40.40/20.54/36.21 |  30.30/11.25/24.92
T5-L | 86.2 | 75.6 | 83.9 | 70.4 | 42.50/20.68/39.75 | - | -
T5-11B | 86.8 | 78.9 | 85.8 | 75.4 | 43.52/21.55/40.69 | - | -
PEGASUS | - | - | - | - | 44.17/21.47/41.11 | 39.12/19.86/36.24 | 26.63/9.01/21.60
ERNIE-GEN | - | - | - | - | 44.02/21.17/41.26 | 39.25/ 20.25/36.53 | -
ProphetNet | - | - | - | - | 44.20/21.17/41.30 | 39.51/20.42/36.69 | -
Table 4: We present results for the non-GLUE Sentence Prediction tasks as well as a set of standard Commonsense tasks. Bolded numbers signify MUPPET vs. base model, while an underline signifies the best number. If not explicitly stated, the results are showing the accuracy of the evaluation set. For commonsense tasks, we re-use the task head from pre-finetuning. | SP | Structured Prediction (Penn) | Summarization
---|---|---|---
| Hyperpartisan | Chunking | Parsing | POS | Arxiv | PubMed | BigPatent
RoBERTa-B | 84.2 | 93.4 | 95.1 | 93.7 | - | - | -
\+ MUPPET | 85.8 | 95.5 | 94.5 | 93.2 | - | - | -
RoBERTa-L | 90.4 | 95.1 | 94.5 | 93.4 | - | - | -
\+ MUPPET | 92.5 | 96.9 | 95.7 | 97.9 | - | - | -
BART | 85.1 | 92.1 | 91.1 | 91.8 | 41.20/9.20/32.45 | 39.87/16.43/35.56 | 48.54/29.35/39.42
\+ MUPPET | 87.2 | 96.1 | 94.5 | 97.2 |  43.90/14.50/40.10 |  45.13/19.80/39.90 |  52.34/33.50/42.80
Pegasus | - | - | - | - | 43.85/16.83/39.17 | 44.53/19.30/40.70 | 52.25/33.04/41.80
Table 5: We present results on a large set of different tasks across datasets that are not available to the model during the pre-finetuning stage. Bolded numbers signify MUPPET vs. base model, while an underline signifies the best number. For Chunking/Parsing, we use F1, while for Part-Of-Speech tagging, we use accuracy. Model | Training Data | A1 | A2 | A3 | ANLI
---|---|---|---|---|---
RoBERTa | S,M | 47.6 | 25.4 | 22.1 | 31.1
| +F | 54.0 | 24.2 | 22.4 | 32.8
| +F+A1⋆2 | 68.7 | 19.3 | 22.0 | 35.8
| +F+A1+A2⋆3 | 71.2 | 44.3 | 20.4 | 43.7
| S,M,F,ANLI | 73.8 | 48.9 | 44.4 | 53.7
RoBERTa-MUPPET | S,M | 49.9 | 28.2 | 24.2 | 33.3
| +F | 55.2 | 26.8 | 24.6 | 33.9
| +F+A1⋆2 | 70.9 | 22.5 | 25.1 | 36.7
| +F+A1+A2⋆3 | 74.3 | 48.2 | 22.8 | 45.9
| S,M,F,ANLI | 76.9 | 52.3 | 44.2 | 56.9
InfoBERT Wang et al. (2021) | S,M,F,ANLI | 76.4 | 51.6 | 48.6 | 58.3
ALUM Liu et al. (2020) | S,M,F,ANLI | 73.3 | 53.4 | 48.2 | 57.7
XL-NET Yang et al. (2019) | S,M,F,ANLI | 67.6 | 50.7 | 48.3 | 55.1
Table 6: We show the performance of the RoBERTa model and the pre-finetuned
RoBERTa-MUPPET model on the ANLI benchmark. Bolded numbers signify MUPPET vs
base model, underline signifies best number. ‘S’ refers to SNLI, ‘M’ to MNLI
dev (-m=matched, -mm=mismatched), and ‘F’ to FEVER; ‘A1–A3’ refer to the
rounds respectively and ‘ANLI’ refers to A1+A2+A3.

Across the board, pre-trained representations that were further refined with
pre-finetuning outperformed standard pre-trained representations. We see more
modest gains on larger datasets, most likely because we do not need to refine
representations beforehand if the fine-tuning dataset is large. On smaller
datasets, we see substantial gains. For example, the pre-finetuned RoBERTa-
BASE model on RTE improves by close to 9 points, rivaling the RoBERTa-Large
accuracy, while the pre-finetuned RoBERTa-Large model gets new state-of-the-
art on RTE rivaling models an order of magnitude larger than it.

We do not improve just over sentence prediction tasks but on every set of
tasks that we measured. For example, we reach a new state of the art on the
HellaSwag dataset previously achieved by utilizing a new fine-tuning approach.
Our methods do not increase parameter count or any complexity measures but are
quite successful at refining features and preparing them for downstream fine-
tuning.

###  4.1 Finetuning Outside of Pre-Finetuning Domain

We also report the performance on tasks not included in the pre-finetuning
data. To do so, we finetune our models on a set of tasks including (1) ANLI
Nie et al. (2019) and Hyperpartisan Kiesel et al. (2019) for classification,
(2) Arxiv He et al. (2019), PubMed Cohan et al. (2018), Sharma et al. (2019)
for summarization, and (3) Chunking, Constituency Parsing and Part-Of-Speech
tagging for structured prediction from the Penn Treebank dataset Marcus et al.
(1993). We present these results in Table 5 and Table 6.

We see that the MUPPET variants of our models out-perform the baselines
consistently across task type and dataset. As a special case we do an in depth
analysis of the MUPPET variant of RoBERTa on the notoriously tough ANLI
dataset and see the same pattern. Pre-finetuned models consistently outperform
their base counterparts.

##  5 Understanding Multi-Task at Scale

###  5.1 Importance of Scale

The first axis we would like to explore is the scale on which multi-task
learning is done. Previous work, such as T5 and MT-DNN, focused on the MTL
scale of around a dozen datasets. To the best of our knowledge, our paper has
the largest MTL set up to date. Accordingly, we are interested in empirically
exploring the effects of scaling up the number of datasets to the
representations learned during MTL.

We pre-finetune a collection of RoBERTa-Base models with varying numbers of
datasets. We train seven models, six uniformly chosen between 10 and 40,
ensuring that at each point, the selected datasets are a superset of the
datasets from prior points. The last model is fully trained on all datasets.
Concretely given two models trained with a different number of datasets
$a,b:a>b$, model $a$ will contain all datasets used to train model $b$ and
more.

For each version of the model, we fine-tune five datasets and plot the results
in Figure 1. Specifically we finetune STS-B (Cer et al., 2017), BoolQ (Clark
et al., 2019), RACE (Lai et al., 2017), SQuAD (Lai et al., 2017), and MNLI
(Williams et al., 2018a). We include these five datasets in the first MTL run
(10 datasets) to remove any bias from adding them in a later stage.

Figure 1: We plot the RoBERTa evaluation accuracy of five datasets: RTE,
BoolQ, RACE, SQuAD, and MNLI, across various scales of multi-task learning
measured in the number of datasets. We notice that performance initially
degrades until a critical point is reached regarding the number of the
datasets used by the MTL framework for all but one dataset. Post this critical
point; our representations improve over the original RoBERTa model.

We see a couple of interesting patterns. First, for individual tasks such as
RTE (Bentivogli et al., 2009), increasing the pre-finetuning scale
monotonically improves performance. This is aligned with other papers that
have seen benefits from first training on MNLI (Williams et al., 2018a) and
then fine-tuning on RTE (Liu et al., 2019b). For other datasets, we see that
doing MTL in the $<15$ datasets regime is detrimental for end-task fine-
tuning. This is also aligned with other empirical observations, i.e., T5
reported that doing MTL did not improve over only fine-tuning. Nevertheless,
it seems that as we increase the number of tasks past some critical point, our
pre-trained representations become more generalizable. Furthermore, although
dependent on the dataset, this critical point is roughly between 10 and 25
tasks.

This suggests that previously observed MTL limitations were not fundamental
and can instead be attributed to the lack of sufficient scale.

###  5.2 Importance of Heterogenous Batches

Another critical factor to getting MTL to learn generalizable representations
is the method through which MTL is implemented, specifically the selection of
batches. To better quantify this trend, we experimented with three balancing
schemes: dataset homogenous, batch homogenous and batch heterogenous.

We refer to dataset homogenous as selecting batches from datasets
sequentially. So we first train on dataset $A$, then train on dataset $B$,
etc. On the other hand, batch homogenous refers to selecting batches
containing only data from the same task; therefore, all gradients are from the
same dataset. This is implemented by selecting all datasets, batching on a
dataset level, and selecting those same batches randomly during training.
Finally, batch heterogeneous refers to a single update containing a batch from
multiple different datasets spanning different tasks. We implemented this by
first creating homogenous sub-batches, calculating loss per sub-batch per GPU,
and then aggregating across GPUs manifesting in a gradient update that
contains various datasets and, therefore, tasks.

To dissect the importance of heterogeneous batches, we train a RoBERTa-Base
model on 35 randomly selected tasks using the three data selection
methodologies outlined above. We then fine-tune these three models on the same
five data-sets mentioned in the previous section.

Figure 2: We plot the evaluation accuracy of RoBERTa across five datasets:
RTE, BoolQ, RACE, SQuAD, and MNLI, using our three batching strategies for
multi-task: Dataset Homogeneous, Batch Homogeneous, Batch Heterogeneous. The
use of heterogenous batches outperforms other batching strategies by a
significant margin and highlights the importance of implementing MTL with the
correct batching strategy.

We present our results in Figure 2. We see the importance of properly defining
a batching strategy for effective multi-task learning. Our findings are also
consistent with Aghajanyan et al. (2020) which saw that sequential training of
data-sets degrades generalizable representations.

###  5.3 Low Resource Experiments

We noticed in Section §4 that data-sets with smaller data-set sizes tended to
improve more from MTL training. To strengthen this hypothesis, we look at two
factors: the scale of pre-finetuning and the scale of fine-tuning (size of
fine-tuning data-set).

We select three data-sets that were not used in pre-finetuning in Section
§5.1. We also select nine partitions per fine-tuning data-set, which is
sampled uniformly between 10% of the data-set and 100% of the data-set.
Selecting the low-resource splits was done through random sampling.

We then fine-tune every low-resource split with every pre-finetuning
checkpoint from Section §5.1. We plot the heatmaps generated from these runs
in Figure 3.

Figure 3: We fine-tune every low-resource split with every pre-finetuning
checkpoint from Section §5.1 for two datasets not available in any of the pre-
finetuning MTL datasets; QNLI (Rajpurkar et al., 2016b) and CoLA (Warstadt et
al., 2018). The pre-finetuning scale is reported in terms of the number of
datasets.

Multiple patterns emerge. First, we see a clear visualization of the critical
point mentioned when doing pre-finetuning. As we increase the scale of MTL,
better representations are available for downstream finetuning. Furthermore,
we see that pre-finetuned models at a larger scale are much more data-
efficient than standard pre-trained models.

Specifically looking at the 34/40 pre-finetuning scale on Figure 3 we see that
we reach higher evaluation accuracies much sooner than the base RoBERTa model
(row 0).

##  6 Conclusion

In this work, we propose pre-finetuning, a stage after pre-training to further
refine representations before end-task finetuning. We show that we can
effectively learn more robust representations through multi-task learning
(MTL) at scale. Our MTL models outperform their vanilla pre-trained
counterparts across several tasks. Our analysis shows that properly scaling
MTL with heterogeneous batches and loss scaling is critical to leveraging
better representations. We also show a critical point regarding the number of
tasks when doing multi-task learning, where fewer tasks degrade
representations compared to the pre-trained model, but more tasks than this
point improve representations.

We discussed a practical setting in which doing this massive multi-task
learning is stable and effective through simple loss scaling and heterogeneous
batches. With our method, we improve upon prior state of the art methods for
RTE Bentivogli et al. (2009) and HellaSWAG Zellers et al. (2019), as well as
improve upon vanilla pre-trained representations for MNLI Williams et al.
(2018a), SQuAD Rajpurkar et al. (2016a), BoolQ Clark et al. (2019), and Common
Sense QA Talmor et al. (2018). We also our MTL model performance with low
resource experiments. We show that on held-out datasets, leveraging
representations from our pre-finetuned models with 34-40 tasks, we reach
higher evaluation accuracies with much less data than the RoBERTa model.

## References

  * Aghajanyan et al. (2020) Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. 2020.  Better fine-tuning by reducing representational collapse.  _arXiv preprint arXiv:2008.03156_.
  * Amini et al. (2019) Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019.  Mathqa: Towards interpretable math word problem solving with operation-based formalisms.  _arXiv preprint arXiv:1905.13319_.
  * Bentivogli et al. (2009) Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009.  The fifth pascal recognizing textual entailment challenge.  In _TAC_.
  * Bowman et al. (2015) Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. 2015.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017\.  Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation.  _arXiv preprint arXiv:1708.00055_.
  * Chen et al. (2018) Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018.  Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.  In _International Conference on Machine Learning_ , pages 794–803. PMLR.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of NAACL-HLT 2019_.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_.
  * Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018.  A discourse-aware attention model for abstractive summarization of long documents.  _arXiv preprint arXiv:1804.05685_.
  * De Marneffe et al. (2019) Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019.  The CommitmentBank: Investigating projection in naturally occurring discourse.  To appear in proceedings of Sinn und Bedeutung 23. Data can be found at https://github.com/mcdm/CommitmentBank/.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * (13) Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace.  Eraser: A benchmark to evaluate rationalized nlp models.
  * Dolan and Brockett (2005) William B Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In _Proc. of NAACL_.
  * Eidelman (2019) Vladimir Eidelman. 2019.  Billsum: A corpus for automatic summarization of us legislation.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 48–56.
  * Fabbri et al. (2019) Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R Radev. 2019.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  _arXiv preprint arXiv:1906.01749_.
  * He et al. (2019) Jun He, Liqun Wang, Liu Liu, Jiao Feng, and Hao Wu. 2019.  Long document classification from local word glimpses via recurrent attention learning.  _IEEE Access_ , 7:40707–40718.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in neural information processing systems_ , pages 1693–1701.
  * Hovy et al. (2001) Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. 2001.  Toward semantics-based answer pinpointing.  In _Proceedings of the First International Conference on Human Language Technology Research_.
  * Huang et al. (2019) Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019.  Cosmos qa: Machine reading comprehension with contextual commonsense reasoning.  _arXiv preprint arXiv:1909.00277_.
  * Iyer et al. (2017) Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. 2017.  First quora dataset release: Question pairs.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.  _arXiv e-prints_ , page arXiv:1705.03551.
  * Khashabi et al. (2018) Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018.  Looking beyond the surface: A challenge set for reading comprehension over multiple sentences.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 252–262.
  * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020.  Unifiedqa: Crossing format boundaries with a single qa system.
  * Khot et al. (2018) Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018.  SciTail: A textual entailment dataset from science question answering.  In _AAAI_.
  * Kiesel et al. (2019) Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019.  Semeval-2019 task 4: Hyperpartisan news detection.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 829–839.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association of Computational Linguistics_.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017.  Race: Large-scale reading comprehension dataset from examinations.  _arXiv preprint arXiv:1704.04683_.
  * Levesque et al. (2012) Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012.  The winograd schema challenge.  In _Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning_. Citeseer.
  * Levesque et al. (2011) Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011.  The Winograd schema challenge.  In _AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning_ , volume 46, page 47.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Li and Roth (2002) Xin Li and Dan Roth. 2002.  Learning question classifiers.  In _COLING 2002: The 19th International Conference on Computational Linguistics_.
  * Liu et al. (2020) X. Liu, Hao Cheng, Pengcheng He, W. Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. 2020.  Adversarial training for large neural language models.  _ArXiv_ , abs/2004.08994.
  * Liu et al. (2019a) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a.  Multi-task deep neural networks for natural language understanding.  _arXiv preprint arXiv:1901.11504_.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Marcus et al. (1993) Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993.  Building a large annotated corpus of english: The penn treebank.
  * McCoy et al. (2019) R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  _CoRR_ , abs/1902.01007.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  _arXiv preprint arXiv:1809.02789_.
  * Mosbach et al. (2020) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2020.  On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines.  _arXiv preprint arXiv:2006.04884_.
  * Narayan et al. (2018) Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  _arXiv preprint arXiv:1808.08745_.
  * Nie et al. (2019) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019.  Adversarial nli: A new benchmark for natural language understanding.  _arXiv preprint arXiv:1910.14599_.
  * Pang and Lee (2005) Bo Pang and Lillian Lee. 2005.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.  In _Proceedings of the ACL_.
  * Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019.  WiC: The word-in-context dataset for evaluating context-sensitive meaning representations.  In _Proceedings of NAACL-HLT_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8):9.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Rajpurkar et al. (2016a) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016a.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_.
  * Rajpurkar et al. (2016b) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016b.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_.
  * Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _2011 AAAI Spring Symposium Series_.
  * Seo et al. (2016) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016.  Bidirectional attention flow for machine comprehension.  _arXiv preprint arXiv:1611.01603_.
  * Sharma et al. (2019) Eva Sharma, Chen Li, and Lu Wang. 2019.  Bigpatent: A large-scale dataset for abstractive and coherent summarization.  _arXiv preprint arXiv:1906.03741_.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958.
  * Szegedy et al. (2015) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2015.  Rethinking the inception architecture for computer vision. corr abs/1512.00567 (2015).
  * Talmor et al. (2018) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  _arXiv preprint arXiv:1811.00937_.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  SuperGLUE: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint 1905.00537_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2021) Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, and Jingjing Liu. 2021.  Info{bert}: Improving robustness of language models from an information theoretic perspective.  In _International Conference on Learning Representations_.
  * Warstadt et al. (2018) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018.  Neural network acceptability judgments.  _arXiv preprint arXiv:1805.12471_.
  * Welbl et al. (2017) Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017.  Crowdsourcing multiple choice science questions.  _arXiv preprint arXiv:1707.06209_.
  * Williams et al. (2018a) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018a.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Williams et al. (2018b) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018b.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in neural information processing systems_ , pages 5753–5763.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Yi et al. (2015) Yang Yi, Yih Wen-tau, and Christopher Meek. 2015.  WikiQA: A Challenge Dataset for Open-Domain Question Answering.  page 2013–2018.
  * Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018.  Swag: A large-scale adversarial dataset for grounded commonsense inference.  _arXiv preprint arXiv:1808.05326_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence?  _arXiv preprint arXiv:1905.07830_.
  * Zhang and Tetreault (2019) Rui Zhang and Joel Tetreault. 2019.  This email could save your life: Introducing the task of email subject line generation.  In _Proceedings of The 57th Annual Meeting of the Association for Computational Linguistics_ , Florence, Italy.
  * Zhang et al. (2018) Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018.  Record: Bridging the gap between human and machine commonsense reading comprehension.  _arXiv preprint arXiv:1810.12885_.
  * Zhang et al. (2015a) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015a.  Character-level Convolutional Networks for Text Classification.  _arXiv:1509.01626 [cs]_.
  * Zhang et al. (2015b) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015b.  Character-level convolutional networks for text classification.  In _NIPS_.

##  Appendix A Appendices

###  A.1 Datasets Used

  1. 1.

CoLA Warstadt et al. (2018)

  2. 2.

SST-2 Socher et al. (2013)

  3. 3.

MRPC Dolan and Brockett (2005)

  4. 4.

QQP Iyer et al. (2017)

  5. 5.

MNLI Williams et al. (2018a)

  6. 6.

QNLI Rajpurkar et al. (2016b)

  7. 7.

RTE Bentivogli et al. (2009)

  8. 8.

WNLI Levesque et al. (2012)

  9. 9.

SuperGLUE Wang et al. (2019)

  10. 10.

Bool Q Clark et al. (2019)

  11. 11.

MultiRC Khashabi et al. (2018)

  12. 12.

WIC Pilehvar and Camacho-Collados (2019)

  13. 13.

WSC Levesque et al. (2011)

  14. 14.

CB De Marneffe et al. (2019)

  15. 15.

COPA Roemmele et al. (2011)

  16. 16.

AG News Zhang et al. (2015b)

  17. 17.

IMDB Maas et al. (2011)

  18. 18.

MultiNLI Williams et al. (2018b)

  19. 19.

SNLI Bowman et al. (2015)

  20. 20.

HANS McCoy et al. (2019)

  21. 21.

Rotten Tomatoes Pang and Lee (2005)

  22. 22.

Yelp Polarity Zhang et al. (2015a)

  23. 23.

Eraser Multi RC DeYoung et al.

  24. 24.

Wiki QA Yi et al. (2015)

  25. 25.

Trec Li and Roth (2002); Hovy et al. (2001)

  26. 26.

SciTail Khot et al. (2018)

  27. 27.

CNN Daily Mail Hermann et al. (2015)

  28. 28.

Billsum Eidelman (2019)

  29. 29.

XSUM Narayan et al. (2018)

  30. 30.

Aeslc Zhang and Tetreault (2019)

  31. 31.

Multinews Fabbri et al. (2019)

  32. 32.

Math QA Amini et al. (2019)

  33. 33.

Openbook QA (Mihaylov et al., 2018)

  34. 34.

SWAG Zellers et al. (2018)

  35. 35.

HellaSWAG Zellers et al. (2019)

  36. 36.

RACE Lai et al. (2017)

  37. 37.

CommonSense QA Talmor et al. (2018)

  38. 38.

Cosmos QA Huang et al. (2019)

  39. 39.

AI2 ARC - Easy Clark et al. (2018)

  40. 40.

AI2 ARC - Challenge Clark et al. (2018)

  41. 41.

SCIQ Welbl et al. (2017)

  42. 42.

SQUAD Rajpurkar et al. (2016a)

  43. 43.

NQ Kwiatkowski et al. (2019)

  44. 44.

DROP Dua et al. (2019)

  45. 45.

RECORD Zhang et al. (2018)

  46. 46.

Hotpot Yang et al. (2018)

  47. 47.

TriviaQA Joshi et al. (2017)

###  A.2 Hyperparameters


#  SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech
Translation with Chain-of-Thought

Hongyu Gong Bandhav Veluri
Meta AI
<EMAIL_ADDRESS>

###### Abstract

Expressive speech-to-speech translation (S2ST) is a key research topic in
seamless communication, which focuses on the preservation of semantics and
speaker vocal style in translated speech. Early works synthesized speaker
style aligned speech in order to directly learn the mapping from speech to
target speech spectrogram. Without reliance on style aligned data, recent
studies leverage the advances of language modeling (LM) and build cascaded LMs
on semantic and acoustic tokens. This work proposes SeamlessExpressiveLM, a
single speech language model for expressive S2ST. We decompose the complex
source-to-target speech mapping into intermediate generation steps with
_chain-of-thought_ prompting. The model is first guided to translate target
semantic content and then transfer the speaker style to multi-stream acoustic
units. Evaluated on Spanish-to-English and Hungarian-to-English translations,
SeamlessExpressiveLM outperforms cascaded LMs in both semantic quality and
style transfer, meanwhile achieving better parameter efficiency.

SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech
Translation with Chain-of-Thought

Hongyu Gong Bandhav Veluri Meta AI<EMAIL_ADDRESS>

##  1 Introduction

Translation is crucial for breaking down language barriers between diverse
linguistic backgrounds, and speech-to-speech translation has attracted great
interest of research community in fostering cross-lingual spoken
communication. Early approaches use encoder-decoder model architecture, and
focus on semantic preservation when translating from source to target speech
Jia et al. (2019, 2022); Lee et al. (2022). The need of seamless communication
raises awareness for preserving acoustic information, as speech carries richer
information than semantics. For example, the way speech is uttered conveys
emotions of the speaker. This motivates the research of S2ST with both
semantic and speaker vocal style preserved Barrault et al. (2023).

Language models (LM) with in-context learning capability demonstrate powerful
language understanding and generation. The recent progress of discrete speech
representations has bridged the gap between continuous speech waveform and
discrete text tokens. Semantic tokens such as HuBERT Hsu et al. (2021) and
w2v-BERT units Rubenstein et al. (2023) are learned to capture semantic
information from speech, and acoustic tokens such as EnCodec Défossez et al.
(2022) and SoundStream Zeghidour et al. (2022) encode fine-grained acoustic
information with multiple codebooks. The advances in text language model and
speech tokens lay the foundation for speech language modeling, which is
successfully applied to speech generation tasks Wang et al. (2023).

Recent expressive S2ST approaches such as VALL-E X Zhang et al. (2023),
PolyVoice Dong et al. (2023) and AudioPaLM Rubenstein et al. (2023) build a
cascade of multiple speech LMs in speech-to-semantic translation and semantic-
to-acoustic generation respectively. One LM translates source speech to target
semantic tokens such as HuBERT units, and other LMs generates acoustic units
conditioned on semantic units and acoustic prompt. We note that multi-LM
modeling is faced with common weaknesses of cascaded models such as
computational inefficiency and error propagation.

Their design choices of separate LMs for semantic and acoustic generation
could be explained by the lack of speaker style aligned speech data for model
training. Expressive S2ST model essentially learns the mapping from source to
target acoustic units. The end-to-end training would require speech pairs
aligned in both semantics and style Jia et al. (2022). Despite research
efforts in large-scale data mining to create semantically aligned data
Duquenne et al. (2023), style aligned speech is in general much less and
costly to collect Barrault et al. (2023). The training of cascaded LMs relaxes
the requirements of crosslingual style alignments as the style preservation
could be learned from monolingual speech.

Table 1: A summary of recent S2ST models compared in terms of training data,
model architecture and capability.

|  |  | Enc-Dec S2ST
---
Lee et al. (2022)
| SeamlessExpressive
---
Barrault et al. (2023)
| Translatatron2
---
Jia et al. (2022)
| PolyVoice
---
Dong et al. (2023)
SeamlessExpressiveLM
Train data | Sem aligned speech | ✓ | ✓ | ✓ | ✓ | ✓
Style aligned speech | ✗ | ✓ | ✓ | ✗ | ✗
Aligned speech-text | ✗ | ✓ | ✓ | ✓ | ✗
Model | Cascaded | ✗ | ✓ | ✗ | ✓ | ✗
Components | Encoder-decoder | Encoder-decoder | Encoder-decoder | Three LMs | One LM
Capability | Style preservation | ✗ | ✓ | ✓ | ✓ | ✓

Another challenge resulting in cascaded semantic and acoustic generation is
the heterogeneity of semantic and acoustic tokens. The heterogeneity comes
from the complimentary information in single-codebook semantic tokens and
multi-codebook acoustic tokens. It is challenging to model different types of
token generation in a single LM.

This work proposes SeamlessExpressiveLM, a speech LM which models both
semantic and acoustic generation for end-to-end expressive S2ST. In this work,
we address the challenge of heterogeneous speech tokens by chain-of-thought
(CoT) prompting, and the model adopts multi-step generation from semantic
translation towards acoustic translation. As for training data, we only use
semantically aligned speech without relying on speaker style aligned data. The
trick is to randomly crop the target segment as the acoustic prompt for style
preservation.

Table 1 makes a comparison of existing S2ST models and the proposed approach,
and our contributions are summarized below:

  * •

We propose a novel model to support end-to-end S2ST with speaker style
preservation. SeamlessExpressiveLM demonstrates improved translation quality
and parameter efficiency compared with the cascaded LMs.

  * •

The model is trained with semantically aligned speech, and does not need
aligned speech-text data or speaker aligned speech which are required by
existing approaches.

  * •

We show that semantic and acoustic tokens can be modeled by the same language
model despite their heterogeneity.

  * •

Our ablation study provides insights into how prompt designs affect speech
language model in translation task.

##  2 Related Work

##### S2ST.

The series of Translatotron models have encoder-decoder architecture to
translate source speech into target spectrogram, which could be synthesized as
waveform with a separately trained vocoder. Translatotron uses a speaker
encoder to enable voice conversion in translated speech Jia et al. (2019).
Translatotron 2 removes speaker encoder from the model design for the purpose
of anti-spoofing, and creates speaker-aligned data with a cross-lingual TTS
model for model training Jia et al. (2022). Therefore the translation model
learns to transfer vocal style with a data-driven approach. Translatotron
models also leverage textual supervision by using phoneme in the auxiliary
task of speech recognition task.

With semantic units emerging as an efficient semantic representation of
speech, unit based translation models are developed. The textless S2ST model
learns to map source speech to target units without relying on textual data
such as phonemes Lee et al. (2022), and the target speech could be synthesized
from semantic units with HiFi-GAN vocoder Gong et al. (2023). Despite the good
semantic quality, semantic units do not capture speaker vocal style, and thus
the unit based S2ST lost the style information in speech translations. To
enhance expressivity transfer, it is proposed by Duret et al. to integrate
speaker and emotion encoder into the translation system. The system consists
of speech-to-unit translator and unit-to-speech synthesizer with speaker and
emotion embeddings.

##### S2ST with language model.

Recent progress in acoustic units such as EnCodec Défossez et al. (2022) and
Soundstream Zeghidour et al. (2022) capture richer acoustic and style
information than semantic speech units by using multiple codebooks to encode
residual information in each codebook. This also unlocked the speech language
modeling in vocal style transfer Wang et al. (2023).

With acoustic units as translation target, the translation model is trained to
preserve the style of source speaker. Existing works are built upon a cascaded
architecture Dong et al. (2023); Zhang et al. (2023) consisting of speech-to-
unit translation, primary acoustic unit and residual acoustic unit generation.
The first component translates target semantic units from source speech. The
other two components take care of speaker style transfer, and generate the
first stream of acoustic units and remaining streams sequentially. This type
of model has general limitations of cascaded architecture, which are
inefficiency and error propagation. A multi-task learning framework is
recently designed Peng et al. (2024), which improves model efficiency by
sharing parameters between the first and second components. However, it still
goes with cascaded training and inference of the three components.

We come up with SeamlessExpressiveLM which is trained in an end-to-end manner
with chain-of-thought. It shows better parameter efficiency and improved
performance compared with the cascaded model.

Figure 1: Model architecture of SeamlessExpressiveLM.

##  3 Model

This section introduces SeamlessExpressiveLM, a decoder-only language model
for style transferred speech-to-speech translation.

###  3.1 Speech Tokenizers

Speech tokenizers convert continuous speech waveform into a sequence of
discrete units. HuBERT is used to derive semantic units of speech which mainly
keep the semantic information Hsu et al. (2021). EnCodec extracts multi-
codebook units to carry more fine-grained acoustic information in speech such
as speaker vocal style and intonationDéfossez et al. (2022). Suppose that
EnCodec has $C$ codebooks, and $C=8$ in our experiments. Both HuBERT and
EnCodec units are leveraged in our modeling as described below.

###  3.2 Architecture

We define notations here to simplify the discussion in the rest of this paper.
Source semantic units are denoted as $\mathbf{S}_{\text{src}}$, target
semantic units are $\mathbf{S}_{\text{tgt}}$, prompt acoustic units are
$\mathbf{A}_{\text{prompt}}^{\\{c:1\leq c\leq C\\}}$ and target acoustic units
are $\mathbf{A}_{\text{tgt}}$. Without speaker style aligned data, expressive
S2ST is formulated to learn the mapping: $\mathbf{S}_{\text{src}}$ \+
$\mathbf{A}_{\text{prompt}}$ $\rightarrow$ $\mathbf{A}_{\text{tgt}}$, where
$\mathbf{S}_{\text{src}}$ controls the target semantics and
$\mathbf{A}_{\text{prompt}}$ influences the target style.

SeamlessExpressiveLM adopts chain-of-thought prompting, learning the complex
mapping through multi-step reasoning:

$\displaystyle(\mathbf{S}_{\text{src}})\rightarrow\mathbf{S}_{\text{tgt}}+(\mathbf{A}_{\text{prompt}})\rightarrow\mathbf{A}_{\text{tgt},1}\rightarrow\mathbf{A}_{\text{tgt},c>1},$
(1)

where each $\rightarrow$ is one reasoning step, and variables in $(\cdot)$ are
given while other variables $\mathbf{S}_{\text{tgt}}$,
$\mathbf{A}_{\text{tgt},1}$ and $\mathbf{A}_{\text{tgt},c}$ are predicted by
the model.

As illustrated in Figure 1, the input sequence reflects the chain of thoughts
in translation process. The model starts from semantic translation, proceeds
to transfer the style from prompt acoustic units to the first acoustic stream
and lastly learns the mapping to residual acoustic streams.

##### Embedding layer.

We construct embedding tables for semantic and acoustic units to vectorize the
speech tokens before passing them to the speech language model. For semantic
units and first-stream acoustic units, their embeddings can be retrieved
directly from the embedding table $\text{Emb}(\cdot)$. As for prompt acoustic
tokens from multiple streams, we sum up embeddings from all codebooks as the
acoustic embedding in each position.

$\displaystyle\text{Emb}(\mathbf{A}_{\text{prompt}}^{i})=\sum\limits_{c=1}^{C}\text{Emb}(\mathbf{A}_{\text{prompt},c}^{i})$
(2)

##### Autoregressive (AR) Layers.

Figure 1 demonstrates the overall model architecture. The bottom layers of
SeamlessExpressiveLM are AR Transformer decoder layers, which learn the
sequence modeling of semantic units and primary acoustic units. In AR layer,
each token only attends to its previous tokens and the sequential order of
tokens is preserved. Suppose that AR decoder outputs are $\mathbf{H}_{i}$ for
$i$-th token in the sequence, which is projected by matrix
$\mathbf{W}^{\text{ar}}$ to a probability distribution over the unit
vocabulary.

The model is prompted to generate semantic and acoustic units progressively.
The semantic units of source speech $\mathbf{S}_{\text{src}}$ are first
provided as the semantic prompt, and the model estimates the probability of
semantic units of target speech.

$\displaystyle\tilde{\mathbf{P}}(\mathbf{S}_{\text{tgt}}^{i}|\mathbf{S}_{\text{src}},\mathbf{S}_{\text{tgt}}^{<i})=\text{softmax}\left(\mathbf{W}^{\text{ar}}\mathbf{H}^{\text{ar}}_{i}\right)$
(3)

Moving forward from semantic translation, the model is further provided with
the acoustic units of source speech $\mathbf{A}_{\text{src}}$ as acoustic
prompt, and instructed to predict the target acoustic units in the first
codebook, $\mathbf{A}_{\text{tgt},1}$.

$\displaystyle\tilde{\mathbf{P}}(\mathbf{A}_{\text{tgt},1}^{j}|\mathbf{S}_{\text{src}},\mathbf{S}_{\text{tgt}},\mathbf{A}_{\text{prompt}},\mathbf{A}_{\text{tgt},1}^{<j})$
$\displaystyle=\text{softmax}\left(\mathbf{W}^{\text{ar}}\mathbf{H}_{j}\right),$
(4)

where the acoustic unit is the $j$-th token in the sequence.

##### Non-Autoregressive (NAR) Layer.

On top of AR layers which model semantic and first-stream acoustic unit
sequence, we have NAR layers to model acoustic unit sequence of remaining
codebooks $\mathbf{A}_{\text{tgt},c}~{}(c>1)$. The design of NAR layers is
inspired by VALL-E Wang et al. (2023), which enable efficient decoding of
multi-stream acoustic units. We note that there are key differences in the
design. VALL-E _sequentially_ decodes each stream by conditioning the current
prediction on previous streams of units. In our model, we support _parallel
decoding_ of streams in NAR component.

The decoder outputs $\mathbf{H}^{\text{ar}}$ of AR layers is routed to the NAR
layer. In NAR layers, each token could attend to the whole sequence to access
broader context. To distinguish multiple acoustic streams, we use different
projection matrices $\\{\mathbf{W}_{c}\\}$ for each codebook $c$ $(c>1)$. The
output of NAR layer, $\mathbf{H}^{\text{nar}}_{i}$ at position $i$, is mapped
to the vocabulary of the $c$-th acoustic codebook:

$\displaystyle\tilde{\mathbf{P}}(\mathbf{A}_{\text{tgt},c}^{j}|\mathbf{S}_{\text{src}},\mathbf{S}_{\text{tgt}},\mathbf{A}_{\text{prompt}},\mathbf{A}_{\text{tgt},1})$
$\displaystyle=\text{softmax}\left(\mathbf{W}^{\text{nar}}_{c}\mathbf{H}^{\text{nar}}_{j}\right)$
(5)

###  3.3 Training

Acoustic prompt. Prompt acoustic units $\mathbf{A}_{\text{prompt}}$ provide
all acoustic information to be preserved in the target speech. Ideally given
source and target speech which both semantically and acoustically aligned, the
model could simply use source units as the acoustic prompt. Now we want to
relax the data requirements and use only semantically aligned speech. We carve
a portion of target speech as the acoustic prompt. To prevent the model from
naively copy-paste the acoustic prompt in target acoustic generation, the
acoustic prompt is randomly selected at each train step and the prompt length
has non-trivial effect on mdel performance. The designs of acoustic prompt is
discussed in Section 5.2.

##### AR loss.

The autoregressive layers takes care of target semantic units and first-stream
acoustic units. We use chain-of-thought to train the model to first translate
speech semantically and then transfer speaker vocal style to the target
speech. AR layers of SeamlessExpressiveLM are trained with next token
prediction similar to existing language models. It is noted that source
semantic units and source acoustic units are given as the prompt, and these
tokens do not count towards the training loss. Assume that $\mathbf{P}(\cdot)$
is the ground truth probability (either 0 or 1) of target units.

$\displaystyle\mathbb{L}_{\text{ar}}=-\sum\limits_{i}\mathbf{P}(\mathbf{S}_{\text{tgt}})\log\tilde{\mathbf{P}}(\mathbf{S}_{\text{{tgt}}}|\mathbf{S}_{\text{src}})-$
$\displaystyle\sum\limits_{j}\mathbf{P}(\mathbf{A}_{\text{tgt},1}^{j})\log\tilde{\mathbf{P}}(\mathbf{A}_{\text{tgt},1}^{j}|\mathbf{S}_{\text{src}},\mathbf{S}_{\text{tgt}},\mathbf{A}_{\text{prompt}},\mathbf{A}_{\text{tgt},1}^{<j})$
(6)

##### NAR loss.

In each train step, we randomly select one stream $c$ of acoustic units for
NAR layers to predict.

$\displaystyle\mathbb{L}_{\text{nar}}=-\sum\limits_{j}\mathbf{P}(\mathbf{A}_{\text{tgt},c}^{j})\cdot$
$\displaystyle\log\tilde{\mathbf{P}}(\mathbf{A}_{\text{tgt},c}^{j}|\mathbf{S}_{\text{src}},\mathbf{S}_{\text{tgt}},\mathbf{A}_{\text{prompt}},\mathbf{A}_{\text{tgt},1})$
(7)

The total loss is $\mathbb{L}=\mathbb{L}_{\text{ar}}+\mathbb{L}_{\text{nar}}$,
and SeamlessExpressiveLM is trained in an end-to-end manner.

###  3.4 Inference

The model decodes target semantic units with beam search, and then generates
acoustic units with temperature sampling. Target speech is synthesized by
EnCodec decoder from predicted acoustic units.

##  4 Experiments

| Samples (k) # | Avg (s) | Total (h)
---|---|---|---
Es-En | 250 / 1.9 | 7.2 / 10.0 | 500 / 5.4
Hu-En | 300 / 1.0 | 9.2 / 13.8 | 763 / 3.9
Table 2: Statistics of training and evaluation data including sample size,
average duration in seconds and total duration in hours. The first value is
training statistics, and the second value is test statistics.

This section covers empirical details of model training and evaluation on the
task of speech-to-speech translation with speaker style transfer. We consider
_Spanish-to-English_ (Es-En) and _Hungarian-to-English_ (Hu-En) translation as
representative translations between similar and distant languages.

|  Semantic Params |  Acoustic Params | Es-En | Hu-En
---|---|---|---|---
ASR-BLEU ($\uparrow$) | VSim ($\uparrow$) | ASR-BLEU ($\uparrow$) | VSim ($\uparrow$)
Enc-Dec | 187M | - | 20.05 | 0.024 | 10.43 | 0.070
Cascaded LMs | 155M | 314M | 16.86 | 0.292 | 9.16 | 0.402
SeamlessExpressiveLM | 312M | 17.02 | 0.327 | 9.73 | 0.431
\- no chain-of-thought | 312M | 7.20 | 0.392 | 5.35 | 0.486
\- no semantic prompt | 312M | 6.41 | 0.548 | 4.41 | 0.543
\- no acoustic prompt | 312M | 20.16 | 0.024 | 10.55 | 0.067

Table 3: Spanish-English and Hungarian-English translation results including
model parameter size, ASR-BLEU and vocal style similarity.

##### Data.

We used in-house semantically aligned data consisting of $250$k Spanish-
English and $300$k Hungarian-English speech pairs. A set of $1$k samples were
randomly taken from training data as the validation set. Table 2 summarizes
all data statistics.

##### Automatic evaluation metrics.

We reuse _ASR-BLEU_ metric to measure the semantic translation quality as
existing studies of speech translation. Generated audios are first transcribed
with ASR tools into texts, and we use the medium-sized Whisper English ASR
model111https://huggingface.co/openai/whisper-medium.en in this work. BLEU
evaluates the ngram overlap between the transcripts and ground truth
translations.

Another important metric is _vocal style similarity_ (VSim), quantifying how
similar the generated audios sound in comparison with the source speech in
term of vocal style. Following Barrault et al. (2023), we extract vocal style
embeddings of generated and source speech with a pretrained WavLM encoder Chen
et al. (2022) and compute the cosine similarity of their embeddings. Higher
embedding similarity indicates better vocal style transfer. Furthermore we
report _model size_ , the number of parameters, to reflect the model
efficiency.

| Es-En | Hu-En
---|---|---
Cascaded LMs | 3.34 | 3.56
SeamlessExpressiveLM | 3.28 | 3.82
Table 4: MOS of model translations.

##### Subjective metrics.

Besides automatic metrics, we include Mean Opinion Score (MOS) as the
subjective metric to measure the speech quality with scores ranging from $1$
to $5$ (higher score indicates better quality). We get $50$ samples from the
model outputs in each test set, and two annotators independently evaluate the
quality, and we report the score averaged over annotators and test samples.

##### Models.

We included recent S2ST models which are only trained on semantically aligned
speech data for a fair comparison with SeamlessExpressiveLM.

  * •

Enc-Dec S2ST. We included a textless translation model with encoder-decoder
architecture as a strong baseline for semantic translation Duquenne et al.
(2023). The source speech is encoded by convolutional and Transformer encoder
layers, and its decoder predicts the target semantic units. A separately
trained vocoder synthesizes target speech from semantic units.

  * •

Cascaded LMs222We use the architecture of PolyVoice, and replace SoundStream
with Encodec as acousitc units. PolyVoice Dong et al. (2023) tackles S2ST with
a cascade of three LMs for (1) semantic unit translation, (2) the first
acoustic stream generation, (3) remaining acoustic streams generation. LM-1 is
a decoder-only model translating source to target semantic units, LM-2 and
LM-3 follow VALL-E for acoustic generation which is trained on target speech
i.e., English audios.

SeamlessExpressiveLM consists of $12$ AR decoder layers and $12$ NAR layers.
The token embedding dimension is $512$. Each layer is Transformer decoder
layer with $16$ attention heads, the layer dimension of $1024$ and the
feedforward dimension of $4096$. The model is trained with $0.1$ dropout at
the learning rate of $0.0002$. The cascaded model consists of three LMs each
of which has $12$ Transformer decoder layers and other hyperparameters same as
SeamlessExpressiveLM. As for Enc-Dec S2ST, we set its convolution layers as
$2$, encoder layers of $6$ and decoder layers of $6$ with layer and
feedforward dimensions of $1024$ and $4096$ so that its parameter size match
that of semantic LM in the cascaded model.

##### Training.

For SeamlessExpressiveLM, the training acoustic prompt ratio is uniformly
sampled between $0.25$ and $0.3$, and the inference prompt ratio is $0.3$ in
Spanish-to-English translation. The training ratio is $0.20$-$0.25$, and the
inference ratio is $0.2$ for Hungarian-to-English translation. Section 5.2
discusses how the training and inference ratios are selected for Spanish-to-
English. For cascaded LMs, the acoustic prompt is fixed as $3$ seconds as used
in Wang et al. (2023).

##### Inference.

We set beam search size of $10$ to all models during inference. The
SeamlessExpressiveLMsamples acoustic units with the temperature of $0.9$. For
cascaded LMs, its semantic-to-acoustic generation uses a temperature of $0.8$
following Wang et al. (2023). For LM-based S2ST models, target speech is
synthesized by EnCodec decoder from acoustic units. For Enc-Dec, since it
predicts semantic units, a pretrained unit based HiFi-GAN vocoder is applied
to synthesize speech following Lee et al. (2022).

Figure 2: An ablation study of acoustic prompt ratios in Es-En training and
inference.

###  4.1 Semantic Prompt

##### Results.

Table 3 reports model size, ASR-BLEU and VSim of different S2ST models for Es-
En and Hu-En. Enc-Dec model serves as the upper bound of semantic translation
quality, while it does not preserve speaker vocal style. SeamlessExpressiveLM
outperforms cascaded LMs with $+10.7\%$ and $+7.2\%$ similarity in two
directions. It performs on par with Cascaded LMs in terms of semantic
translation as reflected by comparable ASR-BLEU scores. In Table 4,
SeamlessExpressiveLM shows good MOS in comparison with cascaded LMs. We note
that SeamlessExpressiveLM has better efficiency with $312$M parameters in
comparison with Cascaded LMs with $469$M parameters.

##  5 Ablation Study

Semantic units and prompt acoustic units make a chain of S2ST prompts for
SeamlessExpressiveLM. A natural question to ask is how effective such prompt
design is for speech LM training. Therefore we tried other prompting
strategies as the ablation study.

###  5.1 Chain-of-Thought Prompting

As shown in Equation 1, CoT constructs multi-step reasoning towards the target
$\mathbf{A}_{\text{tgt}}$ given the training samples
$\\{(\mathbf{S}_{\text{src}},\mathbf{S}_{\text{tgt}},\mathbf{A}_{\text{prompt}})\\}$.
In an ablation study of the effectiveness of chain-of-thought, we design a
variant strategy without CoT by breaking chain-of-thought into two prompting
tasks:
(1)
$\mathbf{S}_{\text{src}}+\mathbf{A}_{\text{prompt}}\rightarrow\mathbf{A}_{\text{tgt}}$:
this is basically the expressive translation task predicting target acoustic
units conditioned on source semantic units and prompt acoustic units;
(2)
$\mathbf{S}_{\text{tgt}}+\mathbf{A}_{\text{prompt}}\rightarrow\mathbf{A}_{\text{tgt}}$:
this is a monolingual style transfer task predicting target acoustic units
conditioned on target semantic units and prompt acoustics.

This strategy provides essentially the same information as CoT prompting does
for model training. It uses multi-task learning in place of multi-step
reasoning. In Table 3, the row “no chain-of-thought” shows a drop of $9.82$
and $4.38$ in ASR-BLEU for Es-En and Hu-En respectively. It suggests that CoT
helps the model with better semantic preservation in translation process.

To quantify the importance of semantic prompt in modeling, we experiment with
another strategy by removing target semantic units from CoT prompting.
Specifically, the model is trained to directly predict target acoustic units
conditioned on source semantic units and prompt acoustic units. The row of “no
semantic prompt” in Table 3 shows semantic degradation with a drop of $10.61$
ASR-BLEU in Es-En and $5.32$ in Hu-En, suggesting that semantic prompt plays a
critical role in providing semantic cues for S2ST modeling.

####  5.1.1 Semantic Prompt

To quantify the importance of semantic prompt in modeling, we experiment with
another strategy by removing target semantic units from CoT prompting. The
model is trained to directly predict target acoustic units conditioned on
source semantic units and prompt acoustic units, i.e.,
$\mathbf{S}_{\text{src}}+\mathbf{A}_{\text{prompt}}\rightarrow\mathbf{A}_{\text{tgt}}$.

The row of “no semantic prompt” in Table 3 shows semantic degradation with a
drop of $10.61$ ASR-BLEU in Es-En and $5.32$ in Hu-En, suggesting that
semantic prompt plays a critical role in providing semantic cues for S2ST
modeling.

###  5.2 Acoustic Prompt

Similar to the ablation study of semantic prompt, we remove acoustic prompt to
evaluate its effect on the translation model. We use the trained
SeamlessExpressiveLM to generate target semantic units conditioned on source
semantics (i.e., $\mathbf{S}_{\text{src}}\rightarrow\mathbf{S}_{\text{tgt}}$),
and synthesize speech from target units with the vocoder that is used by Enc-
Dec model. The speech generated without acoustic prompt does not carry source
vocal style, as indicated by the low VSim score from the row “no acousti
prompt” in Table 3. On the other hand, it achieves the best ASR-BLEU among all
models, showing that decoder-only speech LM is as strong as encoder-decoder
architecture in semantic translation.

Moreover as a portion of target speech is taken as the acoustic prompt in
SeamlessExpressiveLM training, one important hyperparamter in this work is the
prompt ratio. Taking Spanish-to-English as an example, we train multiple
models with three sets of prompt ratio ranges: ($0.20$, $0.25$), ($0.25$,
$0.3$) and (0.30, 0.35). For each train sample, a prompt ratio is uniformly
selected from the given range. As for inference, we apply different prompt
ratios to test samples, and measure how ASR-BLEU and VSim change with it.

As shown in Figure 2, the training prompt range ($0.25$, $0.30$) achieves the
highest ASR-BLEU with test prompt ratio of $0.3$. Short acoustic prompt cannot
provide sufficient acoustic information for the translation, while long
acoustic prompt might encourage the model to copy and paste the prompt as it
is taken from the target speech. Models trained with ($0.25$, $0.30$) and
($0.30$, $0.35$) achieve the best ASR-BLEU when the test prompt ratio is set
as $0.30$. For model trained with ($0.20$, $0.25$), its ASR-BLEU drops when
test prompt ratio increases. We can see consistent improvement of VSim with
increased test prompt in all three models.

##  6 Conclusion

We propose SeamlessExpressiveLM to achieve expressive S2ST with chain-of-
thought, which unifies semantic and acoustic language modeling, and improves
efficiency over cascaded approach.

##  7 Limitations and Risks

##### Limitations.

One limitation of this work is that it focuses on speech-only modeling.
Existing works have shown improved speech translation quality with different
data types such as aligned speech-text data. The experiments are conducted in
a limited setting where the model and training data sizes are not large.
Scaling up model and data would help to enhance the translation performance.

##### Ethical considerations.

The proposed model is intended to use for expressive speech-to-speech
translation with both semantics and vocal style preserved. In case of failure,
the system will generate inaccurate semantic meaning or vocal styles, thus
negatively impacting the translation quality. One example of unintended use is
that bad actors misappropriate the model for nefarious activities such as
online scams.

## References

  * Barrault et al. (2023) Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei Li, Daniel Licht, Jean Maillard, Ruslan Mavlyutov, Alice Rakotoarison, Kaushik Ram Sadagopan, Abinesh Ramakrishnan, Tuan Tran, Guillaume Wenzek, Yilin Yang, Ethan Ye, Ivan Evtimov, Pierre Fernandez, Cynthia Gao, Prangthip Hansanti, Elahe Kalbassi, Amanda Kallet, Artyom Kozhevnikov, Gabriel Mejia Gonzalez, Robin San Roman, Christophe Touret, Corinne Wong, Carleigh Wood, Bokai Yu, Pierre Andrews, Can Balioglu, Peng-Jen Chen, Marta R. Costa-jussà, Maha Elbayad, Hongyu Gong, Francisco Guzmán, Kevin Heffernan, Somya Jain, Justine Kao, Ann Lee, Xutai Ma, Alexandre Mourachko, Benjamin Peloquin, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Anna Sun, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang, and Mary Williamson. 2023.  Seamless: Multilingual expressive and streaming speech translation.  _CoRR_ , abs/2312.05187.
  * Chen et al. (2022) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2022.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE J. Sel. Top. Signal Process._ , 16(6):1505–1518.
  * Défossez et al. (2022) Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022.  High fidelity neural audio compression.  _CoRR_ , abs/2210.13438.
  * Dong et al. (2023) Qianqian Dong, Zhiying Huang, Qiao Tian, Chen Xu, Tom Ko, Yunlong Zhao, Siyuan Feng, Tang Li, Kexin Wang, Xuxin Cheng, Fengpeng Yue, Ye Bai, Xi Chen, Lu Lu, Zejun Ma, Yuping Wang, Mingxuan Wang, and Yuxuan Wang. 2023.  Polyvoice: Language models for speech to speech translation.  _CoRR_ , abs/2306.02982.
  * Duquenne et al. (2023) Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, Vedanuj Goswami, Changhan Wang, Juan Pino, Benoît Sagot, and Holger Schwenk. 2023.  Speechmatrix: A large-scale mined corpus of multilingual speech-to-speech translations.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , pages 16251–16269. Association for Computational Linguistics.
  * Duret et al. (2023) Jarod Duret, Benjamin O’Brien, Yannick Estève, and Titouan Parcollet. 2023.  Enhancing expressivity transfer in textless speech-to-speech translation.  _CoRR_ , abs/2310.07279.
  * Gong et al. (2023) Hongyu Gong, Ning Dong, Sravya Popuri, Vedanuj Goswami, Ann Lee, and Juan Pino. 2023.  Multilingual speech-to-speech translation into multiple target languages.  _CoRR_ , abs/2307.08655.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE ACM Trans. Audio Speech Lang. Process._ , 29:3451–3460.
  * Jia et al. (2022) Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. 2022.  Translatotron 2: High-quality direct speech-to-speech translation with voice preservation.  In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162, pages 10120–10134. PMLR.
  * Jia et al. (2019) Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. 2019.  Direct speech-to-speech translation with a sequence-to-sequence model.  In _Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019_ , pages 1123–1127. ISCA.
  * Lee et al. (2022) Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Miguel Pino, Jiatao Gu, and Wei-Ning Hsu. 2022.  Textless speech-to-speech translation on real data.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , pages 860–872. Association for Computational Linguistics.
  * Peng et al. (2024) Yifan Peng, Ilia Kulikov, Yilin Yang, Sravya Popuri, Hui Lu, Changhan Wang, and Hongyu Gong. 2024.  Mslm-s2st: A multitask speech language model for textless speech-to-speech translation with speaker style preservation.  _arXiv preprint arXiv:2403.12408_.
  * Rubenstein et al. (2023) Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, Dirk Padfield, James Qin, Danny Rozenberg, Tara N. Sainath, Johan Schalkwyk, Matthew Sharifi, Michelle Tadmor Ramanovich, Marco Tagliasacchi, Alexandru Tudor, Mihajlo Velimirovic, Damien Vincent, Jiahui Yu, Yongqiang Wang, Vicky Zayats, Neil Zeghidour, Yu Zhang, Zhishuai Zhang, Lukas Zilka, and Christian Havnø Frank. 2023.  Audiopalm: A large language model that can speak and listen.  _CoRR_ , abs/2306.12925.
  * Wang et al. (2023) Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2023.  Neural codec language models are zero-shot text to speech synthesizers.  _CoRR_ , abs/2301.02111.
  * Zeghidour et al. (2022) Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2022.  Soundstream: An end-to-end neural audio codec.  _IEEE ACM Trans. Audio Speech Lang. Process._ , 30:495–507.
  * Zhang et al. (2023) Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2023.  Speak foreign languages with your own voice: Cross-lingual neural codec language modeling.  _CoRR_ , abs/2303.03926.