# Improving speech translation by fusing speech and text

Wenbiao Yin1, Zhicheng Liu2, Chengqi Zhao2, Tao Wang2, Jian Tong2, Rong Ye2
1Nanjing University 2ByteDance
<EMAIL_ADDRESS>
{liuzhicheng.lzc, zhaochengqi.d, wangtao.960826, tongjian,
<EMAIL_ADDRESS>
###### Abstract

In speech translation, leveraging multimodal data to improve model performance
and address limitations of individual modalities has shown significant
effectiveness. In this paper, we harness the complementary strengths of speech
and text, which are disparate modalities. We observe three levels of modality
gap between them, denoted by Modal input representation, Modal semantic, and
Modal hidden states. To tackle these gaps, we propose Fuse-Speech-Text (FST),
a cross-modal model which supports three distinct input modalities for
translation: speech, text, and fused speech-text. We leverage multiple
techniques for cross-modal alignment and conduct a comprehensive analysis to
assess its impact on speech translation, machine translation, and fused
speech-text translation. We evaluate FST on MuST-C, GigaST, and newstest
benchmark. Experiments show that the proposed FST achieves an average 34.0
BLEU on MuST-C En$\rightarrow$De/Es/Fr (vs SOTA +1.1 BLEU). Further
experiments demonstrate that FST does not degrade on MT task, as observed in
prior works. Instead, it yields an average improvement of 3.2 BLEU over the
pre-trained MT model.

##  1 Introduction

Speech translation (ST) accepts speech signals as the input and outputs target
translation. The field of speech translation can be broadly categorized into
cascade system and end-to-end speech translation (E2E ST). Cascade
systemSperber et al. (2017); Zhang et al. (2019); Lam et al. (2021) usually
combines automatic speech recognition (ASR) and machine translation (MT), the
MT subsystem uses ASR transcripts as input, which provide clear expression but
may contain errors stemming from ASR. While E2E STTang et al. (2021a); Fang et
al. (2022); Ye et al. (2022a) can directly map speech signals to the target
translation, thus avoiding the problem of error propagation.

We observe a noticeable modality distribution difference between speech and
ASR transcript as shown in Figure 1a. Compared to ASR transcript, speech
signals contain richer information, such as paralinguistics and speaker
characteristics; they are harder to model and more susceptible to noise. In
summary, speech signals contain accurate and abundant information and
substantial noise, while ASR transcripts are clear but prone to errors.

As shown in Figure 1b, the speech signal conveys the message "I’m watching
Douyin.", but the speech signal contains numerous blank segments and
background noise. E2E ST may encounter challenges in accurately extracting
information directly from speech, particularly in the presence of noise,
without compromising the quality of the translation. Meanwhile, the ASR model
mistakenly transcribes the speech as "I’m watch doing." The MT subsystem may
identify the error in the word "doing" as the decoder assigns low
probabilities to candidate words. However, the MT subsystem lacks additional
information to correct the error.

Figure 1: Left: Data distribution of speech and transcript. Right: The
pipeline of our proposed FST.

Inspired by these findings, we propose a model that fuses speech and ASR
transcript as input to leverage their complementary strengths. As shown in
Figure 1b, our model supports three different modalities of input for
translation: speech, text, and fused speech-text. Our model leverages
corresponding speech representation to rectify errors when our model
identifies the error in the word "doing" (as the decoder assigns low
probabilities to candidate words). Furthermore, we can enhance the training of
our model by utilizing additional MT data.

However, speech and text are disparate modalities, we observe three levels of
the modality gap between them, which impede the integration of speech and
text: 1\. Modal input representation: The length of speech input
representation obtained from speech pre-trained model frequently exceeds that
of the corresponding transcript input representation by an order of magnitude
or more. Furthermore, it is worth noting that while speech input
representation is in continuous space, text input representation is in
discrete space. 2\. Modal semantic: When using speech and golden transcript as
input, the model relies heavily on the golden transcript information and
neglects the speech information. This behavior is because the golden
transcript is more straightforward and contains precise semantic information,
adequate for achieving high translation quality. In contrast, speech
representations obtained through the pre-trained model include four types of
informationChen et al. (2022): content, semantics, paralinguistics, and
speaker characteristics, which is harder to model. However, when using speech
and ASR transcript as input, solely relying on textual information is
insufficient to achieve good translation quality. We expect the model to
incorporate more speech information. 3\. Modal hidden states: It is observed
that diverse modality inputs lead to distinctive hidden states in the encoder
and distinct distributions in the decoder.

We propose several methods to bridge the modality gap to integrate speech and
text better. 1. We utilize a convolutional layer to shrink the length of
speech representations by a factor of $\mu$, which enable speech and text to
have a comparable scale. Additionally, we explore mapping continuous speech
representation to a discrete space using a codebook to align with text
representation. 2. We utilize an open-source ASR model to construct a dataset
of <speech, ASR transcript> pair and employe prompt tags (<golden>/<asr>) to
indicate whether the transcript contains errors. The prompt tags implicitly
guide the model to utilize more speech information when the transcript is
inaccurate. Meanwhile, we utilize the contrastive learning method to reduce
the gap between the model semantic of speech and its corresponding transcript.
3. We utilize Cross-Attentive Regularization (CAR) to align the states of the
encoder and Jensen-Shannon Divergence (JSD) loss to align the distribution of
the decoder.

Our contributions are summarized as follows:

  * $\bullet$

We propose a model that fuses speech and text to improve speech translation,
which supports three different modalities of input for translation: speech,
text, and fused speech-text.

  * $\bullet$

To fuse speech and text as input and leverage their complementary strengths,
we conducted a comprehensive analysis of the modality gap between speech and
text. We propose targeted improvements to bridge the modality gap between
speech and text.

  * $\bullet$

Our experiments show that our model achieves an average 34.0 BLEU on MuST-C
En$\rightarrow$De/Es/Fr (vs SOTA +1.1 BLEU); FST achieves an average
improvement of 3.2 BLEU over the pre-trained MT model on MuST-C.

##  2 Related Work

Cascade ST Cascade ST, achieved by concatenating ASR and MT components, has
been extensively employed in commercial speech translation systems. However,
cascade ST is vulnerable to challenges such as error propagation and high
latency. To overcome the error propagation, (Bertoldi and Federico, 2005; Beck
et al., 2019; Sperber et al., 2019) proposed to feed the MT system with ASR
data structures; (Peitz et al., 2012; Cheng et al., 2019; Di Gangi et al.,
2019a) proposed to make MT robust to ASR errors, for instance by training it
on parallel data incorporating factual or emulated ASR errors.

End-to-end ST To overcome the error propagation and high latency in the
cascade ST systems, Bérard et al. (2016); Duong et al. (2016) proposed an end-
to-end architecture for speech translation, which has attracted extensive
attention (Vila et al., 2018; Salesky et al., 2019; Gangi et al., 2019;
Inaguma et al., 2021; Zhao et al., 2021). However, it is difficult to train an
end-to-end speech translation model directly, primarily due to the inherent
variability and complexity of speech signals and the scarcity of high-quality
speech-translation datasets. Some training methods like pretraining (Weiss et
al., 2017; Bérard et al., 2018; Bansal et al., 2019; Wang et al., 2020a; Tang
et al., 2021b), multi-task learning(Le et al., 2020; Vydana et al., 2021; Ye
et al., 2021; Tang et al., 2022), data augmentation (Park et al., 2019; Jia et
al., 2019; Bahar et al., 2019; Pino et al., 2020), meta-learning (Indurthi et
al., 2020), contrastive learning (Li et al., 2021; Ye et al., 2022a),
knowledge distillation (Liu et al., 2019; Tang et al., 2021a) and curriculum
learning (Kano et al., 2017; Wang et al., 2020b), are proved to be effective.

##  3 Methods

###  3.1 Problem Formulation

The speech translation corpus is usually comprised of triples that include
speech, transcript, and target translation, which can be denoted as
$D={(s,x,y)}$. Here, $s$ is an audio sequence, $x$ is the corresponding
transcript, and $y$ is the corresponding target translation.

Figure 2: Overview of our proposed FST. FT denotes fused speech-text input for
translation; here, we use the prompt tag to fuse speech and text. CTR denotes
cross-modal contrastive learning; CAR denotes cross-attentive regularization.

###  3.2 Model Framework

As shown in Figure 2, our model consists of four sub-modules: Speech Encoder,
Speech-Text Fusion Module, Transformer Encoder, and Transformer Decoder. Our
model supports three different modalities as input: speech, text, and fused
speech-text, and to output target translation. We abbreviate these three
different input modalities for translation as ST, MT, and FT(fused speech-text
translation), respectively.

Speech Encoder The speech encoder($S\mbox{-}Enc$) consists of
Wav2vec2.0Baevski et al. (2020b) and two additional convolutional layers. The
input is a raw waveform signal sampled at 16kHz, and Wav2vec2.0 is used to
extract low-level speech representations from it. However, the speech
representation is much longer than the text, exceeding it by over ten times.
To reduce the difference in representation length between speech and text as
much as possible, we use two additional convolutional layers with stride 2 to
shrink the length by a factor of 4. A greater degree of downsampling would
have led to information loss, while a lesser degree of downsampling could have
resulted in modal misalignment and compromised performance. Denote
$a=S\mbox{-}Enc(s)$ as the speech representation.

To reduce the number of parameters and facilitate knowledge transfer, we share
the Transformer encoder and Transformer decoder for ST, MT, and FT.

Transformer Encoder and Transformer Decoder The Transformer encoder and
Transformer decoder are composed of $N_{e}$ transformer encoder layers and
$N_{d}$ transformer decoder layers, respectively, with the same configuration
as the original implementationVaswani et al. (2017). For the MT task, the
input of the Transformer encoder is the embedding of transcript $e=Emb(x)$.
For the ST task, the input is the audio output representation of the speech
encoder $a$. For the FT task, the input is the fused speech-text
representation $f$(see details in Section 3.3). The Transformer encoder
further extracts the high-level semantic hidden representations and
facilitates knowledge sharing across the three modalities. The Transformer
decoder generates corresponding target translation for ST, MT, and FT.
Besides, we use ASR task to improve translation performance. The training
losses of ST, MT, FT, and ASR are as follows:

$\displaystyle\mathcal{L}_{ST}$ $\displaystyle=-\sum_{n}\log P(y_{n}|s_{n})$
(1) $\displaystyle\mathcal{L}_{MT}$ $\displaystyle=-\sum_{n}\log
P(y_{n}|x_{n})$ $\displaystyle\mathcal{L}_{FT}$ $\displaystyle=-\sum_{n}\log
P(y_{n}|x_{n},s_{n})$ $\displaystyle\mathcal{L}_{ASR}$
$\displaystyle=-\sum_{n}\log P(x_{n}|s_{n})$

$\mathcal{L}_{ST}$, $\mathcal{L}_{MT}$, $\mathcal{L}_{FT}$ and
$\mathcal{L}_{ASR}$ are cross-entropy losses on <speech, target>, <transcript,
target>, <speech, transcript, target> and <speech, transcript>, respectively.

###  3.3 Fusing speech and text

As we mentioned in Section 1, to leverage the complementary strengths between
speech and text, we propose the Fuse-Speech-Text(FST) method. We first
introduce FST in this section and later show how to bridge the modality gap
between speech and text.

Here, we utilize an open-source ASR model to construct a <speech, ASR
transcript, target> pair dataset. Given a speech-transcript-target pair
$(s,x,y)$, the transcript $x$ could be an ASR transcript or golden transcript,
and the fused speech-text representation $f$ is defined as:

$\displaystyle f=Concat(P^{s},S\mbox{-}Enc(s),P^{t},P^{g},Emb(x))$ (2)

where $P^{s}$ and $P^{t}$ are prompt tags to identify whether the input modal
is speech or text, and $P^{g}$ is a prompt tag (<golden>/<asr>) to indicate
whether the transcript contains errors. The prompt tags implicitly guide the
model to utilize more speech information when the transcript is inaccurate.

###  3.4 Align speech and text with FST

As we analyzed in Section 1, we observe three levels of the modality gap
between speech and text: Modal input representation, Modal semantic, and Modal
hidden states. We propose several methods to bridge the modality gap between
speech and text. In addition to the speech subsampling and implicit guidance
mentioned above, we introduce three additional methods in this section: Cross-
modal Contrastive Learning, Cross-Attentive Regularization, Knowledge
Distillation and Jensen-Shannon Divergence.

Cross-modal Contrastive Learning Given a positive example of speech-transcript
$(s,x)$ pair, we can get speech representation $a=S\mbox{-}Enc(s)$, transcript
representation $e=Emb(x)$. We randomly pick a set of $B-1$ transcripts
$\left\\{e_{i}^{-}\right\\}_{i=1}^{B-1}$ from the same batch as negative
examples. For speech representation $a$ and transcript representation $e$, we
first average them in terms of the time dimension and apply the multi-class
N-pair contrastive lossSohn (2016):

$\displaystyle u$ $\displaystyle=MeanPool(a)$ (3) $\displaystyle v$
$\displaystyle=MeanPool(e)$ $\displaystyle\mathcal{L}_{CTR}=-\sum_{s,x}\log$
$\displaystyle\frac{exp(sim(u,v)/\tau)}{\sum_{e_{j}\in\mathcal{A}}exp(sim(u,v(e_{j}))/\tau)}$

where
$\mathcal{A}=\left\\{e\right\\}\cup\left\\{e_{i}^{-}\right\\}_{i=1}^{B-1}$,
$\tau$ is the temperature hyper-parameter, and $sim$ is the cosine similarity
function.

Cross-Attentive Regularization The cross-attentive regularizationTang et al.
(2021a) (CAR) can increase the similarity among distinct modalities. The
essence of CAR is to use a similarity matrix to project a tensor sequence onto
a space of equivalent length as another tensor sequence, then compute $L2$
loss between the two sequences. Here, we utilize the CAR to compute losses
separately between speech representation $a$ and fused speech-text
representation $f$ and between text representation $e$ and fused speech-text
representation $f$. The CAR loss is defined as:

$\displaystyle\mathcal{L}_{CAR}=\mathcal{L}_{CAR}(a,f)+\mathcal{L}_{CAR}(e,f)$
(4)

Knowledge Distillation and Jensen-Shannon Divergence The ST task is more
difficult than the MT task since the speech signals are harder to model and
more susceptible to noise. Previous worksLiu et al. (2019); Gaido et al.
(2020); Tang et al. (2021a) utilize knowledge distillation to facilitate
knowledge acquisition by an ST model from a well-trained MT model. However, in
our work, the transcript may contain errors; the accuracy of the FT task is
usually much higher than the corresponding ST and MT task. We designate the FT
task as the teacher while ST and MT tasks as the student, minimizing the loss
between the student and teacher outputs. The KD loss is defined as:

$\displaystyle\mathcal{L}_{KD}(a,f)$
$\displaystyle=-\sum_{n}q(y_{n}|f_{n})\log p(y_{n}|a_{n})$ (5)
$\displaystyle\mathcal{L}_{KD}(e,f)$
$\displaystyle=-\sum_{n}q(y_{n}|f_{n})\log p(y_{n}|e_{n})$
$\displaystyle\mathcal{L}_{KD}$
$\displaystyle=\mathcal{L}_{KD}(a,f)+\mathcal{L}_{KD}(e,f)$

Furthermore, we endeavor to optimize the congruity of the ultimate
distributions of the three modalities through the minimization of the Jensen-
Shannon DivergenceLin (1991) (JSD) between the three output distributions,
which is

$\displaystyle\mathcal{L}_{JSD}=\sum_{n}($ $\displaystyle
JSD\left\\{p(y_{n}|a_{n})||p(y_{n}|,f_{n})\right\\}$ (6) $\displaystyle+$
$\displaystyle JSD\left\\{p(y_{n}|e_{n})||p(y_{n}|f_{n})\right\\})$

The final training objective is as follows:

$\displaystyle\mathcal{L}=$
$\displaystyle\alpha\mathcal{L}_{ST}+\alpha\mathcal{L}_{MT}+(1-\alpha)\mathcal{L}_{KD}+\mathcal{L}_{FT}$
(7) $\displaystyle+$
$\displaystyle\mathcal{L}_{ASR}+\beta\mathcal{L}_{CTR}+\gamma\mathcal{L}_{CAR}+\delta\mathcal{L}_{JSD}$

where $\alpha$, $\beta$, $\gamma$ and $\delta$ are predefined hyper-
parameters.

##  4 Experiments

###  4.1 Experimental setups

ST Datasets We conduct experiments on MuST-C111We use v1.0
https://ict.fbk.eu/must-c/Di Gangi et al. (2019b) and GigaST222https://st-
benchmark.github.io/resources/GigaSTYe et al. (2022b). MuST-C is a
multilingual speech translation dataset that contains translations from
English to 8 languages. MuST-C contains several hundred hours of audio
recordings from English TED Talks; we conduct experiments on MuST-C
En$\rightarrow$De/Es/Fr. We use the dev set for development and analysis and
tst-COMMON set for test. GigaST is a large-scale pseudo speech translation
dataset created by translating the text in GigaSpeech. We conduct experiments
on GigaST En$\rightarrow$Zh, and test on in-house cgtn, zhiyuan and aiconf
datasets. We utilize an open-source ASR model (whisper base.en
333https://huggingface.co/openai/whisper-base.en) to construct a dataset of
<speech, ASR transcript> pair, the Word Error Rate (WER) of our constructed
datasets are shown in the last column of Table 1.

| ST | MT Train | MT Test | WER
---|---|---|---|---
| #hours | #sents | name | #sents | name
MuST-C |
En$\rightarrow$De | 408 | 234K | WMT17 | 4.5M | newstest2014 | 22.65
En$\rightarrow$Es | 504 | 270K | WMT17 | 4.1M | newstest2013 | 22.60
En$\rightarrow$Fr | 492 | 280K | WMT17 | 5.5M | newstest2014 | 22.08
GigaST |
En$\rightarrow$Zh | 9,780 | 7,650K | in-house | 105M | newstest2019/2020 | 21.98

Table 1: Statistics of all datasets.

MT Datasets Our model allows us to use the external MT dataset for further
training. We introduce external WMT datasets for En$\rightarrow$De/Es/Fr and
in-house MT dataset for En$\rightarrow$Zh. The detailed statistics of all
datasets are shown in Table 1.

Models | External Data | BLEU
---|---|---
Speech | Text | ASR | MT | De | Es | Fr | Avg.
Cascade Model
EspnetInaguma et al. (2021) | - | - | - | - | 23.6 | - | 33.8 | -
W2V2-TransformerFang et al. (2022) | $\checkmark$ | - | - | $\checkmark$ | 26.9 | 30.0 | 36.6 | 31.2
Ye et al. (2021) | - | - | $\checkmark$ | $\checkmark$ | 25.2 | - | 34.9 | -
Xu et al. (2021) | - | - | $\checkmark$ | $\checkmark$ | 28.1 | - | - | -
End-to-End Model
MTLTang et al. (2021b) | - | - | - | $\checkmark$ | 23.9 | 28.6 | 33.1 | 28.5
FAT-STZheng et al. (2021) | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 25.5 | 30.8 | - | -
JT-S-MTTang et al. (2021a) | - | - | - | $\checkmark$ | 26.8 | 31.0 | 37.4 | 31.7
ChimeraHan et al. (2021) | $\checkmark$ | - | - | $\checkmark$ | $\text{27.1}^{\dagger}$ | 30.6 | 35.6 | 31.1
XSTNetYe et al. (2021) | $\checkmark$ | - | - | $\checkmark$ | 27.1 | 30.8 | 38.0 | 32.0
SATEXu et al. (2021) | - | - | $\checkmark$ | $\checkmark$ | $\text{28.1}^{\dagger}$ | - | - | -
STEMMFang et al. (2022) | $\checkmark$ | - | - | $\checkmark$ | 28.7 | 31.0 | 37.4 | 32.4
TaskAwareIndurthi et al. (2021) | - | - | $\checkmark$ | $\checkmark$ | 28.9 | - | - | -
STPTTang et al. (2022) | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | 33.1 | 39.7 | -
ConSTYe et al. (2022a) | $\checkmark$ | - | - | $\checkmark$ | 28.3 | 32.0 | 38.3 | 32.9
FST-ST | - | - | - | $\checkmark$ | 27.7 | 32.4 | 37.2 | 32.4
FST-FT | - | - | - | $\checkmark$ | 29.2 | 33.9 | 38.9 | 34.0
Table 2: Case-sensitive detokenized BLEU scores on MuST-C tst-COMMON set.
"Speech" denotes unlabeled audio data, "Text" denotes unlabeled text data,
e.g. Europarl V7Koehn (2005), CC25Liu et al. (2020), $\dagger$ use 40M
OpenSubtitlesLison and Tiedemann (2016) as external MT data.

Models | De | Es | Fr | Avg.
---|---|---|---|---
newstest | asr | golden | newstest | asr | golden | newstest | asr | golden
Pre-trained 6E6D $\text{MT}^{\sharp}$ | 23.7 | 24.3 | 30.4 | 31.8 | 29.5 | 34.2 | 34.7 | 32.2 | 39.9 | 31.2
Pre-trained 24E6D $\text{MT}^{\sharp}$ | 27.5 | 25.7 | 31.9 | 34.8 | 32.0 | 37.0 | 38.1 | 34.2 | 41.5 | 33.6
STEMM Fang et al. (2022) | - | - | 31.5 | - | - | - | - | - | - | -
FST-MT | 25.0 | 28.6 | 34.2 | 32.9 | 33.5 | 37.7 | 35.9 | 37.7 | 44.1 | 34.4

Table 3: Case-sensitive detokenized BLEU scores on MuST-C tst-COMMON set and
newstest. "asr" denotes use the ASR transcript as input, "golden" denotes use
the golden transcript as input. $\sharp$ are trained with the same external MT
data and MuST-C <transcript, target> pair data.

Model Configuration For the speech encoder, we use
Wav2vec2.0444https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small.pt
following the base configuration, which is only pre-trained on
LibrispeechPanayotov et al. (2015) without any finetuning. Two layers of CNNs
after the Wav2vec2.0 with kernel size 5, stride size 2, padding 2, and hidden
dimension 1024. The transformer encoder and decoder follow the base
configuration, with hidden size $h^{d}=512$, 8 attention heads, and 2048 FFN
hidden states. We use $N_{e}=6$ transformer encoder layers and $N_{d}=6$
transformer decoder layers.

Experiment Details We pre-trained the MT model on the external MT dataset; the
learning rate is 5e-4. The FST model is initialized with a pre-trained MT
model, and the learning rate is 6e-5. We use the raw 16kHZ speech as input and
jointly tokenize the bilingual text using SentencePieceKudo and Richardson
(2018). We use an Adam optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.98$, and
20k warm-up updates. The dropout is set to 0.15, and the value of label
smoothing is set to 0.1. For the training loss, we set weight of
$\mathcal{L}_{ST}$ and $\mathcal{L}_{MT}$ $\alpha=0.8$, contrastive
temperature $\tau=0.02$ and weight of $\mathcal{L}_{CTR}$ $\beta=1.0$, weight
of $\mathcal{L}_{CAR}$ $\gamma=0.02$ and weight of $\mathcal{L}_{JSD}$
$\delta=1.0$. We use sacreBLEU555https://github.com/mjpost/sacrebleu,
sacreBLEU signature: nrefs:1 — bs:1000 — seed:12345 — case:mixed — eff:no —
tok:13a — smooth:exp — version:2.0.0Post (2018) to evaluate case-sensitive
detokenized BLEU.

Models | newstest2019 | newstest2020 | cgtn | zhiyuan | aiconf
---|---|---|---|---|---
asr | speech | fused | asr | speech | fused | asr | speech | fused
Pre-trained 6E6D $\text{MT}^{\sharp}$ | 36.7 | 43.0 | 29.7 | - | - | 28.2 | - | - | 30.5 | - | -
Pre-trained 24E6D $\text{MT}^{\sharp}$ | 40.1 | 46.6 | 32.6 | - | - | 29.5 | - | - | 33.6 | - | -
SA-CTR Li et al. (2022) | 36.8 | 43.1 | 31.0 | 31.8 | 31.0 | 28.3 | 29.4 | 29.1 | 31.6 | 32.4 | 32.1
Codebook-GumbelBaevski et al. (2020a) | 36.8 | 43.0 | 30.9 | 30.0 | 31.7 | 28.1 | 28.4 | 28.9 | 31.7 | 31.6 | 32.5
Codebook-K-meansBaevski et al. (2020a) | 36.9 | 43.2 | 31.0 | 31.3 | 31.8 | 28.2 | 29.3 | 29.0 | 31.5 | 31.7 | 32.3
FST | 38.3 | 44.2 | 31.6 | 33.3 | 33.9 | 28.4 | 30.2 | 30.5 | 32.4 | 34.0 | 34.5

Table 4: Case-sensitive detokenized BLEU scores on En$\rightarrow$Zh test
sets. $\sharp$ are trained with the same MT data and GigaST <transcript,
target> pair data. "asr" denotes use the ASR transcript, "fused" denotes use
ASR transcript and speech as input. Hubert large is used to extract speech
features on the GigaST En$\rightarrow$Zh.

###  4.2 Baseline systems

We compare our method with several strong ST systems, including EspnetInaguma
et al. (2021), W2V2-TransformerFang et al. (2022), MTLTang et al. (2021b),
FAT-STZheng et al. (2021), JT-S-MTTang et al. (2021a), ChimeraHan et al.
(2021), XSTNetYe et al. (2021), SATEXu et al. (2021), STEMMFang et al. (2022),
TaskAwareIndurthi et al. (2021), STPTTang et al. (2022), ConSTYe et al.
(2022a).

Besides, we implement several methods for fusing speech and text modalities.
The only difference between our approach and others is the specific method for
fusing speech and text. SA-CTR: our implementation involved drawing
inspiration from the image and text fusion techniques employed in Li et al.’s
(2022) to propose a method for fusing speech and text. Codebook-Gumbel and
Codebook-K-means: we utilize the method from Baevski et al. (2020a) to map the
speech representation $e$ onto a discrete space using a codebook; then we
concatenate discrete speech representation and text representation as ours.

###  4.3 Main Results

Comparison with End-to-End Baselines As shown in Table 2, we compare our model
with several strong end-to-end baselines. Many existing works rely on
additional auxiliary data for better performance, e.g. large-scale MT data and
unlabeled audio data. In the table, we provide a summary of the auxiliary data
employed by these baselines, with a $\checkmark$ denoting its usage in the
corresponding column. Most previous strong baselines have leveraged two or
more additional auxiliary datasets, such as STPTTang et al. (2022) and ConSTYe
et al. (2022a), whereas our model only utilizes additional MT data. Our E2E
FST-ST achieves comparable results with the previous best models. When fusing
speech and asr transcript as input, our FST-FT outperforms SOTA by an average
of 1.1 BLEU on MuST-C.

Task | MT | MT | ST | FT
---|---|---|---|---
Config. | newstest | asr | speech | fused
FST | 25.0 | 28.6 | 27.7 | 29.2
$-\mathcal{L}_{ASR}$ | 24.9 | 28.5 | 27.4 | 28.9
$-\mathcal{L}_{ASR}-\mathcal{L}_{KD}-\mathcal{L}_{JSD}$ | 25.2 | 28.0 | 27.1 | 28.8
$-\mathcal{L}_{ASR}-\mathcal{L}_{KD}-\mathcal{L}_{JSD}-\mathcal{L}_{CTR}-\mathcal{L}_{CAR}$ | 24.8 | 27.6 | 26.4 | 28.5

Table 5: BLEU scores on MuST-C En$\rightarrow$De tst-COMMON set and newstest
set by removing individual losses.

Comparison with Cascade Baselines We compare our model with several strong
cascade systems. W2V2-Transformer, Ye et al. (2021) and Xu et al. (2021)
provided three strong cascade systems trained using MuST-C and external ASR
and MT data. As shown in Table 2, our end-to-end FST-ST achieves comparable
results with these strong cascade models, while our FST-FT significantly
outperforms these strong cascade models.

Comparison with MT Baselines Our model is initialized with a pre-trained MT
model and then jointly trained on multiple tasks. Previous work has
encountered catastrophic forgetting problems on MT task during joint
trainingFang et al. (2022), which significantly degrades performance on MT
tasks. We evaluate our model on the MT task and show the result in Table 3 and
Table 4. Our model achieves significant improvement on the MT task instead of
a decline in performance. Our model even outperforms pre-trained 24E6D (24
transformer encoder layers and 6 transformer decoder layers) MT on MuST-C.
However, with the increase of training data, our model performs lower than
pre-trained 24E6D MT when using ASR transcript as input (En$\rightarrow$Zh).
Nevertheless, our model can still outperform pre-trained 24E6D MT when fusing
speech and text as input on En$\rightarrow$Zh.

###  4.4 Ablation Study

As shown in Equation 7, our training objective contains eight terms. In
addition to the cross-entropy objective
$\mathcal{L}_{ST},\mathcal{L}_{MT},\mathcal{L}_{FT}$, we investigate the
effects of the other auxiliary training objectives. By gradually removing each
loss, Table 5 shows the improvements brought by each auxiliary training
objective.

Task | MT | MT | ST | FT
---|---|---|---|---
Model | newstest | asr | speech | fused
Wav2vec2.0 | 25.0 | 28.6 | 27.7 | 29.2
HuBERT Large | 25.0 | 28.4 | 29.3 | 29.7
HuBERT Extra Large | 25.1 | 28.4 | 30.0 | 30.3

Table 6: The impact of speech pre-trained model on MuST-C En$\rightarrow$De
tst-COMMON set and newstest set.

##  5 Analysis

###  5.1 Impact of Different Fusion Methods

We compare our model with different fusion methods, such as SA-CTR and STEMM.
SA-CTR proposed a selective attention and gated fusion mechanism to fuse two
different modalities; STEMM proposed the speech-text manifold mixup to mix up
the representation sequences of different modalities. Our model achieves
better results than theirs by utilizing a prompt-based approach. As Section 1
mentions, speech input representation is in continuous space, while text input
representation is in discrete space. We utilize the method from Baevski et al.
(2020a) to map the speech representation onto a discrete space using a
codebook. However, Codebook-Gumbel and Codebook-K-means led to a decrease in
BLEU score (shown in Table 4). Our conjecture here is that the existing speech
pre-trained models extract continuous features that undergo a mapping process
to a discrete space, resulting in the loss of audio information and a
subsequent reduction in the BLEU score. Nonetheless, if the speech pre-trained
models can extract high-quality discrete features, it is plausible that such
discrete features could enhance performance.

###  5.2 Impact of Speech Pre-trained Model

We further explore the impact of the speech pre-trained model. Here, we report
the results of Wav2vec2.0, HuBERT
Large666https://dl.fbaipublicfiles.com/hubert/hubert_large_ll60k.pt and HuBERT
Extra Large777https://dl.fbaipublicfiles.com/hubert/hubert_xtralarge_ll60k.pt,
which are widely used in speech translation. As shown in Tabel 6, as the
strength of speech pre-trained models increases, the performance of the models
on ST and FT improves. Intuitively, larger speech pre-trained models provide
more complementary knowledge to complete the insufficient text
representations.

Figure 3: Left: The average BLEU improvement for ASR transcript resulting from
using fuse-speech-text on En$\rightarrow$Zh aiconf testset, under various
conditions of WER. Right: The frequency distribution of WER on
En$\rightarrow$Zh aiconf testset.

###  5.3 Improvement of Different WER

We conduct an empirical study to examine the efficacy of our approach in
enhancing BLEU score through the fusion of speech information under different
WER present in the ASR transcript. As shown in Figure 3, when the word error
rate of ASR transcript is minimal ($0\leq\text{WER}<5$), the fusion of speech
information results in a slight decrease in BLEU score. This behavior is
because the transcript is adequate for achieving high translation quality, and
the fusion of speech information may introduce noise. As the WER increases,
the advantages of integrating speech information become more pronounced.

###  5.4 Case Study

Models |  |
---|---|---
CASE 1
Ref. |  src: | I’m extraordinarily delight to be here.
asr: | I’m extraordinary. I’d like to be here.
tgt: | 我非常高兴来到这里。
FST-MT | tgt | 我非 同寻常,我想在这里。
FST-ST | tgt |  非常高兴来到这里。
FST-FT | tgt | 我非常高兴来到这里。
CASE 2
Ref. |  src: | So you could think of the viruses like, they are people.
asr: | So you could think of the viruses like their people.
tgt: | 所以你可以把病毒想象成人。
FST-MT | tgt | 所以你可以 像他们的人民一样看待病毒。
FST-ST | tgt | 所以你可以把病毒想象成 他们的人。
FST-FT | tgt | 所以你可以把病毒想象成人。

Table 7: Case study: cases are generated from FST-MT, FST-ST, and FST-FT on
En$\rightarrow$Zh aiconf testset. The  red underlined text indicates
inaccurate translations, and the  indicates missing translation.

In this section, we present several cases generated by FST-MT, FST-ST, and
FST-FT. In the first case, the ASR model mistakenly transcribes "delight" as
"I’d like" due to the highly similar pronunciations of these two words. FST-MT
fails to generate the correct translation as a result of errors present in the
ASR transcript. Meanwhile, FST-ST produces omissions, as modeling direct
speech to target translation proves to be more challenging. Notably, only FST-
FT translates the sentence correctly, leveraging the complementary strengths
of speech and text. In the second case, the speech signal contains numerous
background noises; the ASR model mistakenly transcribes the "they are" as
"their", FST-MT and FST-ST are mistranslated, and only FST-FT translates
correctly. We believe this improvement comes from fusing speech and text.

##  6 Conclusion

In this paper, we propose FST, a cross-modal model which supports three
distinct input modalities for translation: speech, text, and fused speech-
text. We comprehensively analyze the modality gap between speech and text and
utilize multiple techniques to bridge the modality gap. We then fuse speech
and text to improve speech translation. Experiments and analysis demonstrate
the effectiveness of our proposed method.

## Limitations

This work improves speech translation by fusing speech and text, but the model
is far from being achieved for industrialgrade implementations. Although the
ChatGPT and Whisper models exhibit superior speech-to-text capabilities
compared to our model, we maintain that fusing speech and text remains a
viable approach in the era of large-scale models. There are two significant
limitations in this study that could be addressed in future research. First,
our model still relies on an ASR system to transcribe speech into text, which
does not address the issue of high latency in the cascade system. Second, our
model needs labeled data for training, especially the <speech, transcript,
target> pair. Speech data is exceptionally scarce, and obtaining speech data
for many languages around the world is particularly challenging.

## References

  * Baevski et al. (2020a) Alexei Baevski, Steffen Schneider, and Michael Auli. 2020a.  vq-wav2vec: Self-supervised learning of discrete speech representations.
  * Baevski et al. (2020b) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020b.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in neural information processing systems_ , 33:12449–12460.
  * Bahar et al. (2019) Parnia Bahar, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019.  On using SpecAugment for end-to-end speech translation.  In _Proceedings of the 16th International Conference on Spoken Language Translation_ , Hong Kong. Association for Computational Linguistics.
  * Bansal et al. (2019) Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2019\.  Pre-training on high-resource speech recognition improves low-resource speech-to-text translation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 58–68, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Beck et al. (2019) Daniel Beck, Trevor Cohn, and Gholamreza Haffari. 2019.  Neural speech translation using lattice transformations and graph networks.  In _Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13)_ , pages 26–31, Hong Kong. Association for Computational Linguistics.
  * Bérard et al. (2018) Alexandre Bérard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. 2018.  End-to-end automatic speech translation of audiobooks.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , page 6224–6228. IEEE Press.
  * Bérard et al. (2016) Alexandre Bérard, Olivier Pietquin, Laurent Besacier, and Christophe Servan. 2016.  Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation.  In _NIPS Workshop on end-to-end learning for speech and audio processing_ , Barcelona, Spain.
  * Bertoldi and Federico (2005) N. Bertoldi and M. Federico. 2005.  A new decoder for spoken language translation based on confusion networks.  In _IEEE Workshop on Automatic Speech Recognition and Understanding, 2005._ , pages 86–91.
  * Chen et al. (2022) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1505–1518.
  * Cheng et al. (2019) Qiao Cheng, Meiyuan Fan, Yaqian Han, Jin Huang, and Yitao Duan. 2019.  Breaking the data barrier: Towards robust speech translation via adversarial stability training.  In _Proceedings of the 16th International Conference on Spoken Language Translation_ , Hong Kong. Association for Computational Linguistics.
  * Di Gangi et al. (2019a) Matti Di Gangi, Robert Enyedi, Alessandra Brusadin, and Marcello Federico. 2019a.  Robust neural machine translation for clean and noisy speech transcripts.  In _Proceedings of the 16th International Conference on Spoken Language Translation_ , Hong Kong. Association for Computational Linguistics.
  * Di Gangi et al. (2019b) Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019b.  MuST-C: a Multilingual Speech Translation Corpus.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2012–2017, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Duong et al. (2016) Long Duong, Antonios Anastasopoulos, David Chiang, Steven Bird, and Trevor Cohn. 2016.  An attentional model for speech translation without transcription.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 949–959, San Diego, California. Association for Computational Linguistics.
  * Fang et al. (2022) Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022.  STEMM: Self-learning with speech-text manifold mixup for speech translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7050–7062, Dublin, Ireland. Association for Computational Linguistics.
  * Gaido et al. (2020) Marco Gaido, Mattia A. Di Gangi, Matteo Negri, and Marco Turchi. 2020.  End-to-end speech-translation with knowledge distillation: FBK@IWSLT2020.  In _Proceedings of the 17th International Conference on Spoken Language Translation_ , pages 80–88, Online. Association for Computational Linguistics.
  * Gangi et al. (2019) Mattia A. Di Gangi, Matteo Negri, and Marco Turchi. 2019.  Adapting Transformer to End-to-End Spoken Language Translation.  In _Proc. Interspeech 2019_ , pages 1133–1137.
  * Han et al. (2021) Chi Han, Mingxuan Wang, Heng Ji, and Lei Li. 2021.  Learning shared semantic space for speech-to-text translation.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2214–2225.
  * Inaguma et al. (2021) Hirofumi Inaguma, Tatsuya Kawahara, and Shinji Watanabe. 2021.  Source and target bidirectional knowledge distillation for end-to-end speech translation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1872–1881, Online. Association for Computational Linguistics.
  * Indurthi et al. (2020) S Indurthi, H Han, NK Lakumarapu, B Lee, I Chung, S Kim, and C Kim. 2020.  Data efficient direct speechto-text translation with modality agnostic meta-learning.  In _Proc. ICASSP_.
  * Indurthi et al. (2021) Sathish Indurthi, Mohd Abbas Zaidi, Nikhil Kumar Lakumarapu, Beomseok Lee, Hyojung Han, Seokchan Ahn, Sangha Kim, Chanwoo Kim, and Inchul Hwang. 2021.  Task aware multi-task learning for speech to text tasks.  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7723–7727.
  * Jia et al. (2019) Ye Jia, Melvin Johnson, Wolfgang Macherey, Ron J Weiss, Yuan Cao, Chung-Cheng Chiu, Naveen Ari, Stella Laurenzo, and Yonghui Wu. 2019.  Leveraging weakly supervised data to improve end-to-end speech-to-text translation.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7180–7184. IEEE.
  * Kano et al. (2017) Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura. 2017.  Structured-based curriculum learning for end-to-end english-japanese speech translation.  In _Interspeech_.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A parallel corpus for statistical machine translation.  In _Proceedings of machine translation summit x: papers_ , pages 79–86.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Lam et al. (2021) Tsz Kin Lam, Shigehiko Schamoni, and Stefan Riezler. 2021.  Cascaded models with cyclic feedback for direct speech translation.  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7508–7512.
  * Le et al. (2020) Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier. 2020.  Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 3520–3533, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Li et al. (2022) Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma, and JingBo Zhu. 2022.  On vision features in multimodal machine translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6327–6337, Dublin, Ireland. Association for Computational Linguistics.
  * Li et al. (2021) Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2021.  UNIMO: Towards unified-modal understanding and generation via cross-modal contrastive learning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2592–2607, Online. Association for Computational Linguistics.
  * Lin (1991) Jianhua Lin. 1991.  Divergence measures based on the shannon entropy.  _IEEE Transactions on Information theory_ , 37(1):145–151.
  * Lison and Tiedemann (2016) Pierre Lison and Jörg Tiedemann. 2016.  OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 923–929, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Liu et al. (2019) Yuchen Liu, Hao Xiong, Jiajun Zhang, Zhongjun He, Hua Wu, Haifeng Wang, and Chengqing Zong. 2019.  End-to-End Speech Translation with Knowledge Distillation.  In _Proc. Interspeech 2019_ , pages 1128–1132.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015.  Librispeech: An asr corpus based on public domain audio books.  In _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5206–5210.
  * Park et al. (2019) Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin Dogus Cubuk, and Quoc V. Le. 2019.  Specaugment: A simple data augmentation method for automatic speech recognition.  In _Interspeech_.
  * Peitz et al. (2012) Stephan Peitz, Simon Wiesler, Markus Nußbaum-Thom, and Hermann Ney. 2012.  Spoken language translation using automatically transcribed text in training.  In _Proceedings of the 9th International Workshop on Spoken Language Translation: Papers_ , pages 276–283, Hong Kong, Table of contents.
  * Pino et al. (2020) Juan Pino, Qiantong Xu, Xutai Ma, Mohammad Javad Dousti, and Yun Tang. 2020.  Self-Training for End-to-End Speech Translation.  In _Proc. Interspeech 2020_ , pages 1476–1480.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Salesky et al. (2019) Elizabeth Salesky, Matthias Sperber, and Alexander Waibel. 2019.  Fluent translations from disfluent speech in end-to-end speech translation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2786–2792, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Sohn (2016) Kihyuk Sohn. 2016.  Improved deep metric learning with multi-class n-pair loss objective.  _Advances in neural information processing systems_ , 29.
  * Sperber et al. (2017) Matthias Sperber, Graham Neubig, Jan Niehues, and Alex Waibel. 2017.  Neural lattice-to-sequence models for uncertain inputs.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1380–1389, Copenhagen, Denmark. Association for Computational Linguistics.
  * Sperber et al. (2019) Matthias Sperber, Graham Neubig, Ngoc-Quan Pham, and Alex Waibel. 2019.  Self-attentional models for lattice inputs.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1185–1197, Florence, Italy. Association for Computational Linguistics.
  * Tang et al. (2022) Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, and Juan Pino. 2022\.  Unified speech-text pre-training for speech translation and recognition.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1488–1499, Dublin, Ireland. Association for Computational Linguistics.
  * Tang et al. (2021a) Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel. 2021a.  Improving speech translation by understanding and learning from the auxiliary text translation task.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4252–4261, Online. Association for Computational Linguistics.
  * Tang et al. (2021b) Yun Tang, Juan Pino, Changhan Wang, Xutai Ma, and Dmitriy Genzel. 2021b.  A general multi-task learning framework to leverage text data for speech to text tasks.  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6209–6213.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Vila et al. (2018) Laura Cross Vila, Carlos Escolano, José A. R. Fonollosa, and Marta Ruiz Costa-jussà. 2018.  End-to-end speech translation with the transformer.  In _IberSPEECH Conference_.
  * Vydana et al. (2021) Hari Krishna Vydana, Martin Karafiát, Katerina Zmolikova, Lukáš Burget, and Honza Černocký. 2021.  Jointly trained transformers models for spoken language translation.  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7513–7517.
  * Wang et al. (2020a) Chengyi Wang, Yu Wu, Shujie Liu, Zhenglu Yang, and Ming Zhou. 2020a.  Bridging the gap between pre-training and fine-tuning for end-to-end speech translation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 9161–9168.
  * Wang et al. (2020b) Chengyi Wang, Yu Wu, Shujie Liu, Ming Zhou, and Zhenglu Yang. 2020b.  Curriculum pre-training for end-to-end speech translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3728–3738, Online. Association for Computational Linguistics.
  * Weiss et al. (2017) Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Z. Chen. 2017.  Sequence-to-sequence models can directly translate foreign speech.  In _Interspeech_.
  * Xu et al. (2021) Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Shen Huang, Qi Ju, Tong Xiao, and Jingbo Zhu. 2021.  Stacked acoustic-and-textual encoding: Integrating the pre-trained models into speech translation encoders.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2619–2630, Online. Association for Computational Linguistics.
  * Ye et al. (2021) Rong Ye, Mingxuan Wang, and Lei Li. 2021.  End-to-end speech translation via cross-modal progressive training.  In _Proc. of INTERSPEECH_.
  * Ye et al. (2022a) Rong Ye, Mingxuan Wang, and Lei Li. 2022a.  Cross-modal contrastive learning for speech translation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5099–5113, Seattle, United States. Association for Computational Linguistics.
  * Ye et al. (2022b) Rong Ye, Chengqi Zhao, Tom Ko, Chutong Meng, Tao Wang, Mingxuan Wang, and Jun Cao. 2022b.  Gigast: A 10,000-hour pseudo speech translation corpus.  _arXiv preprint arXiv:2204.03939_.
  * Zhang et al. (2019) Pei Zhang, Niyu Ge, Boxing Chen, and Kai Fan. 2019.  Lattice transformer for speech translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6475–6484, Florence, Italy. Association for Computational Linguistics.
  * Zhao et al. (2021) Jiawei Zhao, Wei Luo, Boxing Chen, and Andrew Gilman. 2021.  Mutual-learning improves end-to-end speech translation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3989–3994, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zheng et al. (2021) Renjie Zheng, Junkun Chen, Mingbo Ma, and Liang Huang. 2021.  Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation.  In _International Conference on Machine Learning_ , pages 12736–12746. PMLR.