# On the Benefits of Early Fusion in Multimodal Representation Learning

George Barnum, Sabera Talukder11footnotemark: 1 & Yisong Yue
Department of Computation and Neural Systems, Neurobiology, Computing and
Mathematical Sciences
California Institute of Technology
Pasadena, CA 91125, USA
<EMAIL_ADDRESS>
Equal contributors.

###### Abstract

Intelligently reasoning about the world often requires integrating data from
multiple modalities, as any individual modality may contain unreliable or
incomplete information. Prior work in multimodal learning fuses input
modalities only after significant independent processing. On the other hand,
the brain performs multimodal processing almost immediately. This divide
between conventional multimodal learning and neuroscience suggests that a
detailed study of early multimodal fusion could improve artificial multimodal
representations. To facilitate the study of early multimodal fusion, we create
a convolutional LSTM network architecture that simultaneously processes both
audio and visual inputs, and allows us to select the layer at which audio and
visual information combines. Our results demonstrate that immediate fusion of
audio and visual inputs in the initial C-LSTM layer results in higher
performing networks that are more robust to the addition of white noise in
both audio and visual inputs.

##  1 Introduction

Multimodal learning is important for many tasks, including audio visual speech
recognition (Yu et al., 2020; Zhou et al., 2019; Su et al., 2017), emotion
recognition (Park et al., 2020; Cao et al., 2014), multimedia event detection
(Song et al., 2019), depth-based object detection (Wang et al., 2015b; a),
urban dynamics modeling (Zhang et al., 2017), image-sentence matching (Liu et
al., 2019), and biometric recognition (Song et al., 2019). In many cases, an
individual modality does not contain sufficient information to classify the
scene. Therefore, utilizing multiple modalities is crucial, particularly in
complex tasks or domains prone to noisy data.

One important design decision in multimodal learning is how to best combine,
or fuse, the different input modalities (Baltrušaitis et al., 2018; Li et al.,
2018). Prior work on multimodal learning has largely relied on extensive
unimodal featurization and other preprocessing before fusing the different
modalities (Katsaggelos et al., 2015; Atrey et al., 2010). On the other hand,
it is known that biological neural networks engage in multimodal fusion in the
very early layers of sensory processing pathways (Schroeder & Foxe, 2005;
Budinger et al., 2006). This divide between conventional multimodal learning
and neuroscience suggests that a detailed study of early multimodal fusion
could yield insights for improving multimodal representation learning.

In this paper, we conduct a detailed study on the benefits of early fusion in
multimodal representation learning, focusing on audio and visual modalities as
they are the most related to human sensory processing. To facilitate this
study, we design a convolutional LSTM (C-LSTM) architecture that enables audio
and visual input fusion at various layers in the architecture. We find that
fusion in the initial layer outperforms fusion before the fully connected (FC)
layer, and that early fusion enables robust performance over a range of audio
and visual signal to noise ratios (SNRs). We further study the interaction
effects of fusion with noisy inputs both in one and two modalities. These
results shed new light on the power of immediate fusion as a means to improve
model performance in the presence of noise. Integrating multimodal inputs as
soon as possible can be generalized to other multimodal domains, such as
audio-visual speech recognition and emotion recognition to increase their
performance and representational power.

##  2 Related Work

Multimodal Representation Learning. Baltrušaitis et al. (2018) thoughtfully
broke down the main problems within multimodal machine learning into five
categories: representation, translation, alignment, fusion, and co-learning.
Another helpful analysis of multimodal interactions revealed best practices
while building mutlimodal systems: users tend to intermix unimodal and
multimodal interactions, multimodal systems’ main advantage occurs in
decreasing errors or increasing flexibility (Turk, 2014). However, in the same
paper Turk (2014) did concede that multimodal integration, also referred to as
the fusion engine, is the key technical challenge for multimodal systems.

When dealing with the problem of fusion, there are currently two common
paradigms: early fusion (Atrey et al., 2010) and immediate fusion (Katsaggelos
et al., 2015). In early fusion, audio and visual modalities are first
featurized before being passed to two independent modeling process units that
do not differentiate between features from different modalities (Katsaggelos
et al., 2015). On the other hand, immediate fusion is when the audio and
visual modalities are first featurized and then sent to a join modeling
process unit (Katsaggelos et al., 2015). This unfortunate terminology does not
take into account the possibility of fusing the inputs before any substantial
featurization, which does occur in biological neural networks.

A closely related area is multi-view learning (Li et al., 2018). While the two
areas share significant overlap, multi-view learning places emphasis on having
different views from the same input modality. A typical example is capturing
the same scene from two viewing angles (where both views use the visual
modality).

Connections to Neuroscience. In the brain, multisensory integration was
traditionally believed to occur only after single modality inputs underwent
extensive processing in unisensory regions (Schroeder & Foxe, 2005). However,
we now know that in many species, including humans, that multisensory
convergence occurs much earlier in low level coritical structures (Schroeder &
Foxe, 2005). In fact, primary sensory cortices may not be unimodal at all
(Budinger et al., 2006). This may in part be because of individual neuron’s
abilities to be modulated by multiple modalities (Meredith & Allman, 2009). In
a striking discovery, Allman & Meredith (2007) found that 16% of visual
neurons in the posterolateral lateral suprasylvian that were previously
believed to be only visually responsive were significantly facilitated by
auditory stimuli. This philosophical departure from individual modality
processing towards early multimodal convergence in neuroscience lays a
promising groundwork for high-impact explorations in multimodal machine
learning.

##  3 Multimodal Convolutional LSTM Model

Artificial neural networks often introduce inductive biases based on the
structure of the input modality, such as convolution or recurrence in the case
of visual or audio inputs (Lecun & Bengio, 1995; Parascandolo et al., 2016;
Wang et al., 2016). Since these biases are usually modality specific,
approaches in multimodal domains frequently involve a degree of independent
modality processing with a corresponding inductive bias. On the other hand,
biological neural networks perform some multimodal processing almost
immediately.(Schroeder & Foxe, 2005; Budinger et al., 2006).

In order to maintain the advantages of these modality specific inductive
biases while also allowing for the immediate fusion of audio and visual
inputs, we created a multimodal convolutional long short term memory network
that generates fused audio-visual representations with appropriate inductive
biases. Our convolutional long short term memory, or C-LSTM, architecture
combines the convolutional properties found in traditional convolutional
neural networks, and traditional long short term memory networks. At each
point of convolution, the first layer takes:

  * •

The section of the input image to be multiplied by our convolutional kernel,
denoted as $\mathbf{v}$.

  * •

The section of the hidden state to be multiplied by our convolutional kernel,
denoted as $\mathbf{h}_{t-1}$. It is initialized to zeros for the first time
step in the first layer.

  * •

The spectrogram value of the audio input, at a given time step, denoted as
$\mathbf{a}_{t}$.

Our model then computes the LSTM gate values using: $\mathbf{v}$,
$\mathbf{h}_{t-1}$, and $\mathbf{a}_{t}$ via the equations in Figure 1, under
the Initial LSTM Gate Values section. The initial C-LSTM layer produces a
single multimodal tensor that combines information from the audio and visual
inputs. As in the standard LSTM architecture, the hidden state,
$\mathbf{h}_{t}$, of the previous layer is used as the input,
$\mathbf{x}_{t}$, of the current layer. Therefore, in our subsequent LSTM
layers the gate values at each location are computed from the section of the
combined multimodal input, $\mathbf{x}_{t}$, to be multiplied by our
convolutional kernel. Equations found in Figure 1, under the Subsequent LSTM
Gate Values section. By applying the LSTM operations at each location of a
convolution, this architecture allows the LSTM cells to respond to the spatial
information from the visual domain as well as the temporal information of the
audio domain. This architecture enables us to study the mixing of signals at
the initial, second, and fully connected layer while maintaining the same
inductive biases that are beneficial in processing image and sequence data.

Initial LSTM Gate Values:

$\displaystyle f_{t}$
$\displaystyle=\sigma(W_{f}[\mathbf{h}_{t-1},\mathbf{x}_{t}]+b_{f})$
$\displaystyle i_{t}$
$\displaystyle=\sigma(W_{i}[\mathbf{h}_{t-1},\mathbf{x}_{t}]+b_{i})$
$\displaystyle g_{t}$
$\displaystyle=\tanh(W_{g}[\mathbf{h}_{t-1},\mathbf{x}_{t}]+b_{g})$
$\displaystyle o_{t}$
$\displaystyle=\sigma(W_{o}[\mathbf{h}_{t-1},\mathbf{x}_{t}]+b_{o})$

Cell State Update:

$\displaystyle c_{t}=f_{t}c_{t-1}+i_{t}g_{t}$

Hidden State Update:

$\displaystyle h_{t}=o_{t}\tanh(c_{t})$

Subsequent LSTM Gate Values:

$\displaystyle f_{t}$
$\displaystyle=\sigma(W_{f}[\mathbf{h}_{t-1},\mathbf{x}_{t}]+b_{f})$
$\displaystyle i_{t}$
$\displaystyle=\sigma(W_{i}[\mathbf{h}_{t-1},\mathbf{x}_{t}]+b_{i})$
$\displaystyle g_{t}$
$\displaystyle=\tanh(W_{g}[\mathbf{h}_{t-1},\mathbf{x}_{t}]+b_{g})$
$\displaystyle o_{t}$
$\displaystyle=\sigma(W_{o}[\mathbf{h}_{t-1},\mathbf{x}_{t}]+b_{o})$

Figure 1: The first two layers of the multimodal convolutional long-short term
memory network, and the equations used to compute the gate and update values.
$f_{t}$ is the forget gate, $i_{t}$ is the input gate, $g_{t}$ is the cell
gate, and $o_{t}$ is the output gate. $W$’s are the corresponding weight
matrices, and $b$’s the corresponding bias values. $\sigma()$ is the sigmoid
function. $\tanh()$ is the hyperbolic tangent function.

Varying the Fusion Level. The full C-LSTM approach described above performs
early fusion, i.e., mixing the modalities starting in the very first layer.
However, our framework can be easily modified to only allow for fusion in
later layers, or to mask out one modality all together – in particular by
forcing certain weights to be zero. As such, we can use the C-LSTM
architecture to conduct a controlled inquiry into our research question.

##  4 Experiments

###  4.1 Dataset

We constructed a multimodal dataset based on the well-known MNIST dataset
(LeCun et al., 2010) and the Free Spoken Digit dataset (Jackson et al., 2018).
We selected these datasets because of their tractability. This allows us to
combine them to create a multimodal task which we know will be solvable. This
allows us to artificially manipulate the difficulty of the task via the
addition of noise. We prefer a regime in which we can break the system to
understand its principles of fusion and representation.

For each of these datasets, we first combined all the data, testing and
training, into one dataset, then split each of the two datasets into training,
testing, and validation sets with an $8:1:1$ ratio. Next, for each of the
training, testing and validation splits, we created a dataset containing all
image and audio pairs with the same label. We then augmented the data by
adding white noise to the image and audio data such that the signal to noise
ratio could be chosen, as shown in Figure 2 and Appendix A.1. This allows us
to explore our model’s response to degradation in either the image or audio
input, while utilizing noise techniques commonly applied to inputs (Borji &
Lin, 2020). We then front padded each audio input with zeros such that all the
audio examples are of equal length. Finally, we took the spectrogram of each
audio input, with 400 samples, 201 frequency bins, and a stride of 200
samples.

Figure 2: A single input example where white noise has been applied at various
SNR values. Audio spectrograms have been truncated to the first 42 frequency
bins for convenience, with the full spectrograms available in Figure 8

.

###  4.2 Training & Modeling Details

Using our C-LSTM architecture described in Section 3, we constructed multiple
different models in order to study the benefits of multimodal fusion. These
models can be viewed as ablations of the full model (i.e., fixing certain
weights to zero).

  * •

The full C-LSTM model that allows for fusing in the early layers (akin to how
biological networks fuse sensory processing in early layers).

  * •

Restricting fusion to the intermediate convolutional layers.

  * •

Restricting fusion to the fully connected layers (akin to prior work that
performed unimodal featurization prior to fusion).

  * •

Only processing visual or audio input.

Detailed model architectures are provided in A.2.

We trained all models using the Adam optimizer (Kingma & Ba, 2014) with a
learning rate of 0.001. Each model is trained on 87516 examples randomly
selected from our multimodal dataset (which in total contains 11202076
training data points, 174490 validation data points, 175389 test data points).
The models can be trained on any combination of audio and visual SNRs.

###  4.3 Fusion Comparison

Figure 3: Comparing the performance of initial layer fusion, second layer
fusion, and FC fusion models. The first row shows raw test accuracy at various
signal to noise ratios. The second row shows the difference in accuracy
between the late fusion models and the immediate fusion model at corresponding
signal to noise levels. All models were trained with an audio SNR of 0.5 and a
visual SNR of 0.25. Orig signifies the original audio or visual input.

To examine the value of early compared to late multimodal fusion, the
immediate fusion model, the second layer fusion model, and the fully connected
layer fusion model were trained with the same signal to noise ratios of
training data. We then tested the accuracy of each of these models for a range
of values of the signal to noise ratios of both the audio and visual inputs.
Then we compare the accuracy of the immediate fusion model to the accuracy of
each of the late fusion models.

As seen in Figure 3, the immediate fusion model is more accurate not only for
the signal to noise ratio that the models were trained at, but also for the
majority of the other signal to noise ratios. In particular, the initial layer
fusion model always outperforms or equally performs to the fully connected
fusion model. Furthermore, the initial layer fusion model outperforms the
second layer fusion model, except for when the audio input is degraded well
beyond the audio training signal to noise ratio.

Additionally, initial layer fusion appears to allow the model to be much more
robust to increases in the signal to noise ratio beyond the training values,
especially in the case of increased audio SNR. The main characteristic of the
SNR values in which immediate fusion does not outperform late fusion is low
audio SNR and relatively high visual SNR, and this only occurs in the case
where fusion occurs in the second layer. This suggests that immediate
multimodal fusion encourages the multimodal model to use both input
modalities.

While the specific areas and degree to which early fusion outperforms delayed
fusion varies with the SNRs of the training data, the general trends are
similar, and these fusion plots are representative of models trained at other
audio and visual SNRs; see A.3 for the same plot at other audio and visual
training levels.

###  4.4 Robustness to Noise

Figure 4: Accuracy of multimodal models trained on data with a range of signal
to noise ratios, each for a range of signal to noise ratios of the testing
data.

An advantage of multimodal processing is the network’s resilience to noise in
the inputs. To examine our multimodal model’s resilience to white noise we
trained 16 models at distinct audio-visual SNR combinations and then tested
each model in 100 different audio-visual SNR regimes. The results of these
experiments can be seen in figure 4.

We see that in columns where the visual SNR is 1 and 0.5, there is strong
visual dependence. However, MNIST is a limited dataset, where the original
images are nicely positioned in the field of view and are relatively noise
free. We believe that when adapting the principle of early multimodal fusion
to real world datasets, one would not have as clearly defined visual or audio
dependence. The principles that we elicit from performing this experiment with
MNIST should guide us towards understanding fusion, while recognizing that the
datasets and real life scenarios this will be adapted to will be much more
complex and likely necessitate both modalities at a range of both audio and
visual SNRs. The addition of noise to both the training and test data is
designed to provide a setting that allows the exploration of the limits of
this model while using a simple dataset.

In the the top left quadrant of the figure, we see that at higher visual SNRs
(1, 0.5) and lower audio SNRs (0.5, 0.25) the model is mainly dependent on
visual information. This is denoted by the test accuracy’s consistency as the
audio SNR varies from the original to 0.0625. As we move to the bottom right
quadrant we see lower visual SNRs (0.25, 0.125) and higher audio SNRs (2, 1)
accompanied by an increasing dependence on the audio information. This
increasing dependence is indicated by the test accuracy’s consistency as the
visual SNR varies from original to 0.0625. In the bottom left quadrant, which
corresponds to high audio and visual SNRs, the model performs well on all of
the combinations of audio and visual SNRs that are larger in value than the
audio-visual SNR that the model was trained on. The lower performance in the
SNR ranges below what the model was trained in either the audio or visual
situations is expected. Unsurprisingly, in the top right quadrant, which
corresponds to low audio and visual SNRs, we see that the test accuracy of the
model falls. In these low SNR regimes, the poor signal quality in both
modalities results in poor performance outside of the audio-visual SNRs the
model was tested on. These results mirror our expectations of how a multimodal
model would behave both to various training and testing SNRs.

###  4.5 Comparison to Unimodal Models

To verify that joint audio visual representations are a result of both
modalities, we tested our multimodal model on unimodal inputs by setting one
input to zero. This created unimodal visual models and unimodal audio models
without changing the underlying architecture. For each of these unimodal
models, we trained the model at four SNR values, and tested the accuracy
across our previously selected set of signal to noise ratios. The accuracy of
these unimodal models for the SNR values is displayed in figures 5 and 6.

Figure 5: Performance of the C-LSTM model with only visual input trained on
the original data as well as at various visual signal to noise ratios. Figure
6:  Performance of the C-LSTM model with only audio input trained on the
original data as well as at various audio signal to noise ratios.

As expected, the unimodal models perform relatively well at the SNR that they
are trained at or higher, although as the training SNR decreases, the models
perform worse overall, and the performance on higher SNR data decreases. This
effect is particularly noticeable in the audio only model, figure 6.

Additionally, these unimodal models prove that the C-LSTM architecture is at
minimum capable of using information from either modality to perform the
classification task. However, the difference in performance between the audio
and visual unimodal models shows that the model is not equally sensitive to
each modality or to noise in each modality. This discrepancy motivated us to
choose an audio SNR of 0.5 and a visual SNR of 0.25 for further investigation
of the model architecture, such that the contribution of each modality will be
comparable and will allow us to investigate early fusion in a setting where
multimodal fusion will be beneficial.

###  4.6 Model Inspection

In order to examine the contribution of audio and visual inputs to the
performance of our multimodal classifier, we considered the state of the
network at intermediate timesteps in the recurrent processing of the audio
inputs. In figure 7 we display the activations of the final layer of the
network across timesteps for a single representative example at various signal
to noise ratios. These final layer activations correspond to the
classification of the input into each of the ten digit classes. We display the
activations in response to four SNR scenarios: the original input, a scenario
where the audio input has a higher SNR than the visual input, a scenario where
the visual input has a higher SNR than the audio input, and a scenario where
the visual input and the audio input have equal SNR. Additional examples are
included in section A.4.

Figure 7: The values of the final layer of the multimodal layer across
timesteps of the C-LSTM for a representative example at various signal to
noise ratios.

This visualization demonstrates the value of the multimodal input in this
network. Because the image information is available to the network for the
entire length of the audio, the network initially responds to the image
without audio input, because the audio is front zero-padded. As the network is
evaluated along the time dimension of the audio input, information from the
audio input becomes available to the network. Therefore, comparing the
activations of the final layer can illuminate the contribution of each
modality to the network.

In this example, the original input demonstrates the capability of the same
model to correctly classify the digit in both the visual and audio regime,
because both before and during the audio input, the correct class, 3, is
assigned the maximum value. In the example where the audio input has an SNR of
1.0 and the visual input has an SNR of 0.25, as well as the example with audio
SNR of 0.5 and visual SNR of 0.5, the contribution of the audio input becomes
more evident, as in both of these scenarios, before the audio input is
available the model shows more confusion, both with all the other classes, and
in particular with class 6. In these examples, as the audio information
becomes available to the network, the correct class comes to dominate the
activations.

##  5 Conclusion

In this paper, we developed a C-LSTM architecture to investigate the effects
of fusion depth on noise robustness. Motivated by the neuroscientific
literature suggesting that sensory inputs are combined early in processing, we
proposed that truly immediate fusion of modalities would provide benefits in
neural networks as well. Our immediate fusion model demonstrates robustness to
changes in input noise and an improvement in accuracy relative to late fusion
models with analogous architectures. Future directions include investigating
the effects of immediate multimodal fusion in deeper networks and on problems
with more inherent difficulty, as well as the extension of immediate
multimodal processing to other multimodal domains such as translation,
alignment, and co-learning. However, we believe that the results demonstrated
here show the importance of truly immediate fusion and could help with other
domain specific multimodal learning tasks.

## References

  * Allman & Meredith (2007) Brian L Allman and M Alex Meredith.  Multisensory processing in “unimodal” neurons: cross-modal subthreshold auditory effects in cat extrastriate visual cortex.  _Journal of neurophysiology_ , 98(1):545–549, 2007.
  * Atrey et al. (2010) Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb El Saddik, and Mohan S Kankanhalli.  Multimodal fusion for multimedia analysis: a survey.  _Multimedia systems_ , 16(6):345–379, 2010.
  * Baltrušaitis et al. (2018) Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency.  Multimodal machine learning: A survey and taxonomy.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(2):423–443, 2018.
  * Borji & Lin (2020) Ali Borji and Sikun Lin.  White noise analysis of neural networks.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=H1ebhnEYDH.
  * Budinger et al. (2006) E. Budinger, P. Heil, A. Hess, and H. Scheich.  Multisensory processing via early cortical stages: Connections of the primary auditory cortical field with other sensory systems.  _Neuroscience_ , 143(4):1065–1083, 2006.  ISSN 0306-4522.  doi: 10.1016/j.neuroscience.2006.08.035.  URL http://www.sciencedirect.com/science/article/pii/S0306452206011158.
  * Cao et al. (2014) Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma.  Crema-d: Crowd-sourced emotional multimodal actors dataset.  _IEEE transactions on affective computing_ , 5(4):377–390, 2014.
  * Jackson et al. (2018) Zohar Jackson, César Souza, Jason Flaks, Yuxin Pan, Hereman Nicolas, and Adhish Thite.  Jakobovski/free-spoken-digit-dataset: v1.0.8, August 2018.  URL https://doi.org/10.5281/zenodo.1342401.
  * Katsaggelos et al. (2015) Aggelos K Katsaggelos, Sara Bahaadini, and Rafael Molina.  Audiovisual fusion: Challenges and new approaches.  _Proceedings of the IEEE_ , 103(9):1635–1653, 2015.
  * Kingma & Ba (2014) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2014.
  * Lecun & Bengio (1995) Yann Lecun and Yoshua Bengio.  _Convolutional networks for images, speech, and time-series_.  MIT Press, 1995.
  * LeCun et al. (2010) Yann LeCun, Corinna Cortes, and CJ Burges.  Mnist handwritten digit database.  _ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist_ , 2, 2010.
  * Li et al. (2018) Yingming Li, Ming Yang, and Zhongfei Zhang.  A survey of multi-view representation learning.  _IEEE transactions on knowledge and data engineering_ , 31(10):1863–1883, 2018.
  * Liu et al. (2019) Ruoyu Liu, Yao Zhao, Shikui Wei, Liang Zheng, and Yi Yang.  Modality-invariant image-text embedding for image-sentence matching.  _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ , 15(1):1–19, 2019.
  * Meredith & Allman (2009) M Alex Meredith and Brian L Allman.  Subthreshold multisensory processing in cat auditory cortex.  _Neuroreport_ , 20(2):126, 2009.
  * Parascandolo et al. (2016) G. Parascandolo, H. Huttunen, and T. Virtanen.  Recurrent neural networks for polyphonic sound event detection in real life recordings.  In _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 6440–6444, 2016.
  * Park et al. (2020) Cheul Young Park, Narae Cha, Soowon Kang, Auk Kim, Ahsan Habib Khandoker, Leontios Hadjileontiadis, Alice Oh, Yong Jeong, and Uichin Lee.  K-emocon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations.  _arXiv preprint arXiv:2005.04120_ , 2020.
  * Schroeder & Foxe (2005) Charles E Schroeder and John Foxe.  Multisensory contributions to low-level, ’unisensory’ processing.  _Current Opinion in Neurobiology_ , 15(4):454–458, 2005.  ISSN 0959-4388.  doi: 10.1016/j.conb.2005.06.008.  URL http://www.sciencedirect.com/science/article/pii/S0959438805000991.
  * Song et al. (2019) Xiaoyu Song, Hong Chen, Qing Wang, Yunqiang Chen, Mengxiao Tian, and Hui Tang.  A review of audio-visual fusion with machine learning.  In _Journal of Physics: Conference Series_ , volume 1237, pp. 022144. IOP Publishing, 2019.
  * Su et al. (2017) Rongfeng Su, Lan Wang, and Xunying Liu.  Multimodal learning using 3d audio-visual data for audio-visual speech recognition.  In _2017 International Conference on Asian Language Processing (IALP)_ , pp. 40–43. IEEE, 2017.
  * Turk (2014) Matthew Turk.  Multimodal interaction: A review.  _Pattern Recognition Letters_ , 36:189–195, 2014.
  * Wang et al. (2015a) Anran Wang, Jianfei Cai, Jiwen Lu, and Tat-Jen Cham.  Mmss: Multi-modal sharable and specific feature learning for rgb-d object recognition.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 1125–1133, 2015a.
  * Wang et al. (2015b) Anran Wang, Jiwen Lu, Jianfei Cai, Tat-Jen Cham, and Gang Wang.  Large-margin multi-modal deep learning for rgb-d object recognition.  _IEEE Transactions on Multimedia_ , 17(11):1887–1898, 2015b.
  * Wang et al. (2016) Y. Wang, L. Neves, and F. Metze.  Audio-based multimedia event detection using deep recurrent neural networks.  In _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 2742–2746, 2016.
  * Yu et al. (2020) Jianwei Yu, Shi-Xiong Zhang, Jian Wu, Shahram Ghorbani, Bo Wu, Shiyin Kang, Shansong Liu, Xunying Liu, Helen Meng, and Dong Yu.  Audio-visual recognition of overlapped speech for the lrs2 dataset.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 6984–6988. IEEE, 2020.
  * Zhang et al. (2017) Chao Zhang, Keyang Zhang, Quan Yuan, Haoruo Peng, Yu Zheng, Tim Hanratty, Shaowen Wang, and Jiawei Han.  Regions, periods, activities: Uncovering urban dynamics via cross-modal representation learning.  In _Proceedings of the 26th International Conference on World Wide Web_ , pp. 361–370, 2017.
  * Zhou et al. (2019) Pan Zhou, Wenwen Yang, Wei Chen, Yanfeng Wang, and Jia Jia.  Modality attention for end-to-end audio-visual speech recognition.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 6565–6569. IEEE, 2019.

##  Appendix A Appendix

###  A.1 Input data

Figure 8: The same input example where white noise has been applied at various
SNR values, with the full audio spectrogram.

###  A.2 Model Details

For the full model, we used: an initial merge layer (64 units and $3\times 3$
kernels, and a $2\times 2$ max pool layer), a second multimodal C-LSTM layer
(64 units and $3\times 3$ kernels, and a $2\times 2$ max pool layer), a dense
layer (128 units and ReLU activation), and then a final dense output layer (10
units).

The second layer fusion model consists of a separate convolutional layer with
64 units and $3\times 3$ kernels and an LSTM layer with 64 units. These layers
feed into a C-LSTM layer, with 64 units and $3\times 3$ kernels, a $2\times 2$
max pool layer, a dense layer with 128 units and ReLU activation, and finally
a final dense output layer with 10 units.

The fully connected layer fusion model consists of separate processing streams
for the visual and audio data. The visual stream consists of two convolutional
layers with 64 units and $3\times 3$ kernels, while the audio stream
consisting of two LSTM layers with 64 units. The output of the convolutional
layers and the last timestep of the output of the LSTM layers are concatenated
and fed into a dense layer with 128 units and ReLU activation, then a final
dense output layer with 10 units.

###  A.3 Late Fusion

Figure 9: Late fusion model where audio SNR = 1, and visual SNR = 0.5. Fusion
in the initial layer outperforms the fusion in the fully connected layer.
Figure 10: Late fusion model where audio SNR = 2, and visual SNR = 0.25.
Fusion in the initial layer outperforms the fusion in the fully connected
layer for audio SNR values greater than 1. Figure 11: Late fusion model where
audio SNR = 1.0, and visual SNR = 0.25. Fusion in the initial layer
outperforms the fusion in the fully connected layer for audio SNR values
greater than 1. Figure 12: Late fusion model where audio SNR = 1, and visual
SNR = 0.125. Fusion in the initial layer outperforms the fusion in the fully
connected layer for audio SNR values greater than 1.

###  A.4 Final Layer Activation

Additional examples of final layer activations.

Figure 13:  Additional example of values of the final layer of the multimodal
layer across timesteps of the C-LSTM for a representative example at various
signal to noise ratios. Figure 14:  Additional example of values of the final
layer of the multimodal layer across timesteps of the C-LSTM for a
representative example at various signal to noise ratios. Figure 15:
Additional example of values of the final layer of the multimodal layer across
timesteps of the C-LSTM for a representative example at various signal to
noise ratios. Figure 16:  Additional example of values of the final layer of
the multimodal layer across timesteps of the C-LSTM for a representative
example at various signal to noise ratios. Figure 17:  Additional example of
values of the final layer of the multimodal layer across timesteps of the
C-LSTM for a representative example at various signal to noise ratios.


#  transLLaMa: LLM-based Simultaneous Translation System

Roman Koshkin†‡ Katsuhito Sudoh† Satoshi Nakamura†
†Nara Institute of Science and Technology, Japan
‡Okinawa Institute of Science and Tenchnology, Japan
<EMAIL_ADDRESS>
###### Abstract

Decoder-only large language models (LLMs) have recently demonstrated
impressive capabilities in text generation and reasoning. Nonetheless, they
have limited applications in simultaneous machine translation (SiMT),
currently dominated by encoder-decoder transformers. This study demonstrates
that, after fine-tuning on a small dataset comprising causally aligned source
and target sentence pairs, a pre-trained open-source LLM can control input
segmentation directly by generating a special "wait" token. This obviates the
need for a separate policy and enables the LLM to perform English-German and
English-Russian SiMT tasks with BLEU scores that are comparable to those of
specific state-of-the-art baselines. We also evaluated closed-source models
such as GPT-4, which displayed encouraging results in performing the SiMT task
without prior training (zero-shot), indicating a promising avenue for
enhancing future SiMT systems.

##  1 Introduction

Unlike conventional sequential translation, in which the target text is
produced after the end of the corresponding source sentence (or long phrase),
in simultaneous machine translation (SiMT) the target text is produced with
minimal delay, aiming for the best listener experience expected from
professional conference interpreters. While recent years have seen tremendous
progress in sentence-based machine translation, mainstream adoption of SiMT
systems requires solving a range of technical problems. Perhaps the most
important of them is that, much like human conference interpreters, SiMT
systems must make optimal decisions about _when_ (rather than _how_) to
translate. In particular, naively translating each source word immediately
results in compromised target quality, given that the meaning of a source word
often makes sense only in the context of later words. And while waiting until
the end of a sentence might seem a viable solution, in practice it would
introduce unacceptable delays between the source and target message.
Consequently, the development of an effective SiMT system necessitates
striking a balance between these two opposite scenarios.

Existing approaches to maintaining an optimal quality-latency tradeoff in
SiMT, conventionally called _policies_ , fall into two broad categories: fixed
and adaptive. The policy’s role is to signal to a separately trained
translation model _when_ to produce a partial translation (aka WRITE action
(Gu et al., 2017)) based of the partial input; at other times the input, which
represents either text chunks from an upstream ASR system (in cascade SiMT
systems) or speech embeddings (in end-to-end systems), is just read in (READ
action). While with a fixed policy (Dalvi et al., 2018; Ma et al., 2019a;
Elbayad et al., 2020; Zhang and Feng, 2021), the decision to output
translation is based on a simple heuristic, an adaptive policy (Arivazhagan et
al., 2019; Ma et al., 2019b; Zhang and Feng, 2022) can be implemented as a
separately trained model, for example an agent trained with reinforcement
learning (RL) (Gu et al., 2017; Satija and Pineau, 2016).

To the best of our knowledge, state-of-the-art SiMT systems use encoder-
decoder transformer architectures in a sequence-to-sequence paradigm. However,
as of writing this paper the largest – and generally most expressive –
language models are causal decoder-only architectures. We wanted to explore
the utility of such models for SiMT tasks, focusing on the English-German and
English-Russian language pairs, and specifically if they can be harnessed with
minimal engineering effort.

Inspired by the recent success of LLMs – in particular their agential
capabilities (Nascimento et al., 2023; Wang et al., 2023a, d) – here we
propose TransLlama, a policy-free SiMT system, in which an off-the-shelf pre-
trained decoder-only LLM is fine-tuned on a dataset of causally aligned source
and target sentences. The causality of the source is guaranteed by inserting
one or more <WAIT> tokens into the target sentence to ensure that target
content words never appear earlier than their closest equivalents in the
source. We call our model policy-free, because as a result of fine-tuning on a
causally aligned dataset the LLM becomes capable of deciding when to output
translation and when to read in more of the source, without requiring a
separate policy. At inference, the fine-tuned LLM is prompted with _part_ of a
source sentence concatenated with its corresponding (partial) translation and
outputs one or more target tokens until either a full new word or a <WAIT>
token is generated, which signals for more words to be read in. When extended
with an off-the-shelf ASR model, in addition to text-to-text translation
(T2TT), our system handles speech-to-speech translation (S2TT) tasks with
quality (as measured by BLEU score (Papineni et al., 2002)) approaching that
of some of the recently published baselines at comparable latencies.

Our main contributions are as follows:

  1. 1.

We propose a way to fine-tune a pre-trained LLM for the SiMT task with direct
supervision on a dataset of _causally aligned_ source-target sentence pairs;

  2. 2.

We demonstrate that an LLM can perform both simultaneous translation and input
segmentation _without a separate policy_ , with performance approaching or
exceeding state of the art.

The rest of the paper is structured as follows. Section 2 offers a brief
overview of most recent SiMT literature. In Section 3 we detail our system’s
architecture, fine-tuning data preparation and training procedure. In Section
4 we showcase its performance on en-de and en-ru language directions. We
conclude with Section 5 in which we discuss the limitations and directions for
future work.

Figure 1: Model overview. The source audio stream is processed with an ASR
model (1), which saves each recognized word into the buffer. The initial
prompt (2) is built with $k$ source words ($k=3$ in this example). When the
buffer has 3 words, the initial prompt is fed into the LLM, which generates
output tokens until either a <WAIT> token or a full word is generated ("Ich"
in this example). Then the prompt is updated with a new input ("have") and
target ("Ich") word (WRITE action). Finally, the updated prompt (4) is fed
back into the LLM. If <WAIT> is generated, the prompt is only updated with a
new source word from the buffer (READ action).

##  2 Related work

SiMT systems aim to deliver the best translation quality, usually measured
with BLEU score (Papineni et al., 2002), while keeping its latency at an
acceptable level. This quality-latency trade-off is controlled by the
"policy", which decides _when_ to translate (i.e. perform a WRITE action) and
when to receive more input (i.e. perform a READ action). The various policies
described in the literature can be broadly categorized into fixed and adaptive
(Zhang et al., 2020). Fixed policies (e.g, _wait-k_ (Ma et al., 2019a)) are
simple rules that determine the timing and order of WRITE and READ actions
irrespective of the context. Early SiMT systems used _chunk-based_ approaches
(Fügen et al., 2007; Bangalore et al., 2012; Yarmohammadi et al., 2013;
Sridhar et al., 2013), in which the input is split into sub-sentence phrases
and translated independently of the previous chunk’s context, which
compromised translation quality. Attempting to overcome this limitation, Dalvi
et al. (2018) proposed an _incremental decoding_ approach, in which chunk
translations incorporate previous context encapsulated by an RNN’s hidden
states. They showed that coupled with a simple segmentation strategy, their
approach outperformed existing state of the art. On the other hand, adaptive
policies (e.g. _wait-if_ rules (Cho and Esipova, 2016)) make READ/WRITE
actions more flexibly by taking account of the partial source and/or target.
Adaptive policies can be implemented as separately trained agents (e.g. with
reinforcement learning) (Grissom II et al., 2014; Gu et al., 2017; Satija and
Pineau, 2016; Alinejad et al., 2018). In such policies, READ/WRITE actions can
be taken based on attention Raffel et al. (2017); Chiu and Raffel (2018);
Arivazhagan et al. (2019); Ma et al. (2020b), or stability of the model’s
outputs over $n$ steps (so-called _local agreement_ (Liu et al., 2020a; Ko et
al., 2023; Polák et al., 2022)). More recent studies have also explored
training the policy with binary search Guo et al. (2023) aiming to maximize
the gain in translation quality per each token read, or cast the problem of
deciding when to translate as a hidden Markov transformer Zhang and Feng
(2023), in which hidden events correspond to the times at which to output
translation.

Another promising line of work, related to the present study, aims to fine-
tune encoder-decoder transformers, such as mBART (Liu et al., 2020b),
originally pre-trained for sentence-level translation, for the SiMT task. For
example, Fukuda et al. (2023); Kano et al. (2022) utilized fine-tuning on
prefix-alignment data and Zhang et al. (2020) on meaningful units, achieving
compelling performance on some language pairs.

Finally, in the course of writing this paper, we became aware of two
concurrent projects which explored the use of fine-tuned LLMs for SiMT in
conjunction with a modified local agreement (Wang et al., 2023b), and vanilla
wait-$k$ (Agostinelli et al., 2023) policies.

Distinct from previous work, we propose a _policy-free_ approach, in which an
LLM is fine-tuned for the SiMT task on _causally aligned_ full source-target
sentence pairs, which we describe below.

##  3 Method

Although the LLMs we consider in this paper are designed to process only text
input, we add an ASR stage to enable it to also perform S2TT mode. Thus, we
follow a cascaded approach shown in Fig. 1.

Causal alignment. Training SiMT models, including optimal segmentation
policies, with direct supervision has remained a challenge (Guo et al., 2023)
due to at least three reasons: (1) word order inconsistencies between the
source and target, (2) omissions of words from the target that were present in
the source, and/or (3) additions of words to the target not explicitly present
in the source, making it difficult to establish unambiguous correspondences
between each source and target words. This is less of a problem for offline
translation models, because they are trained with direct supervision on pairs
of _complete_ source and target sentences, and both during training and
inference the entire source context is revealed. However, it is not
immediately clear how to use direct supervision for the SiMT task, in which
the model must begin translation based on _partial_ context. Nevertheless, we
believe that direct supervision for the SiMT task is possible and propose a
way to accomplish that with a _causally aligned_ dataset. In such a dataset, a
target word never appears before its corresponding (when such correspondence
can be established) source word in time, which is defined as the number of
words from the sentence start. In other words, in a causally aligned source-
target sentence pair, source words are guaranteed to be causal relative to
their corresponding target words. We illustrate this in Fig. 2.

Figure 2: Causal alignment. Two examples are shown: for en-ru (left) and en-de
(right). If time is defined as the number of words from the beginning of the
sentence, before alignment, some target words appear earlier than their
corresponding English equivalents in the source. By inserting <WAIT> tokens
(shown as "@"), we can shift those target words into the future, thereby
achieving causality for every content word. "_ _" are fillers added at the end
of the source sentence if neccessary to match its length with that of the
target.

Note that the causal alignment is not always perfect: due to the word length
mismatch between the source and target, not all all source words will have a
corresponding target word, and vice versa, not every target word will have a
corresponding word in the source. However, as we demonstrate below, fine-
tuning an LLM on such a causally aligned dataset enables it to achieve results
comparable to some state-of-the-art baselines.

In order to causally align the source and target, we split each sentence using
the word_tokenize function from the _nltk_ package (Bird et al., 2009),
treating punctuation marks as "words", then find the best correspondences
between the source and target words with _SimAlign_ (Jalili Sabet et al.,
2020), and finally insert as many <WAIT> tokens into the target as
appropriate. If after alignment the target becomes longer than the source due
to added <WAIT> tokens, we pad the source at the end with filler strings
ensuring that the aligned source and target sentences have the same number of
"words". These filler strings are only used for convenient batching and are
dropped before tokenization.

Supervised Fine-Tuning. We fine-tune the Llama-2 13B and and 70B models
(Touvron et al., 2023) 111We found that the Llama-2-Chat variants (both 13B
and 70B), when fine-tuned on our causally aligned dataset performed slightly,
but consistently, worse than Llama-2, and we report the results for the latter
model only. to optimize the following objective:

$\mathcal{L}_{\text{T2TT}}=-\sum_{t=1}^{|y|}\log p(y_{t}|y_{<t},x_{\leq t})$
(1)

where $y_{t}$ is the next target token, $y_{<t}$ are previously generated (and
committed) tokens and $x_{\leq t}$ and the partial source tokens revealed up
to the time step $t$. Following (Touvron et al., 2023), we zero out the loss
on tokens corresponding the to system message and source, only backpropagating
on the target tokens.

We use batches of prompt-response pairs collated in the following way. Before
tokenization, each aligned sentence-target pair selected from the causally
aligned dataset is trimmed from the right to leave first $l$ words, where
$l\sim U(1,L)$ and $L$ is the full length of the causally aligned source-
target pair. After trimming, all the <WAIT> tokens except the last one (if
present) are dropped, because they are never plugged back into the input and
only serve the purpose of signaling for more words to be read in. Likewise, we
drop all the fillers (if present) from the source. Finally, the system
message, trimmed source and trimmed target are joined into the prompt (as
shown in Fig. 4) and tokenized. Because there is no <WAIT> token in the Llama
2 tokenizer, we use 0 (which originally corresponds to the <UNK> token). Thus,
the model is fine-tuned to either output the next token of a word or <WAIT>,
if the partial source does not contain sufficient information needed to
predict translation.

To save memory, we loaded the the base model in 4-bit precision using the
bitsandbytes library. This allowed us to fine-tune Llama 2 70B on one NVIDIA
A100 80GB device. We fine-tune the base model with LoRA (Hu et al., 2022) with
$r=16$ and $\alpha=32$ for 3 epochs with a batch size of 25 and gradient
accumulation of 4 steps. We save model checkpoints every 10 steps and select
the one with the lowest validation loss for inference. For optimization, we
used the paged_adamw_32bit optimizer with default parameters, and a learning
rate schedule with a linear warm-up of 10 steps up to 0.00005, followed by a
cosine decay. For parameter-efficient training, as well as for inference, we
used the transformers222https://huggingface.co library.

Inference. At inference, given a prompt (Fig. 4) comprised of a system
message, partial source and previously committed partial target, the LLM
greedily generates one or more next tokens. We use modified wait-_k_ (Ma et
al., 2019a), in which WRITE actions are only allowed when the length of the
PARTIAL_SOURCE is equal or greater than $k$. Since $k$ controls the tradeoff
between quality and latency, we report results for different values of $k$.
After a full new word – which may consist of several tokens – is generated,
the prompt is updated by appending a new source word to the partial source and
the newly generated word to the partial target. This process is repeated until
the LLM generates the <EOS> token. All the generation parameters were at
default, except top_p which we set to 0.7. We did not use beam search during
generation.

PARTIAL_SOURCE | PARTIAL_TARGET | Prediction
---|---|---
I |  | <WAIT>
I like |  | Ya
I like to | Ya | lyublyu
I like to have | Ya lyublyu | <WAIT>
I like to have tea | Ya lyublyu | pitp1
I like to have tea | Ya lyublyu pitp1 | chai
I like to have tea in the | Ya lyublyu pitp1 chai | <WAIT>
I like to have tea in the morning. | Ya lyublyu pitp1 chai | po
I like to have tea in the morning. | Ya lyublyu pitp1 chai po | utram.
I like to have tea in the morning. | Ya lyublyu pitp1 chai po utram. | <EOS>
Figure 3: An illustration of the inference process for the en-ru language
pair. Assuming $k=1$, given the prompt with one source and zero target words,
the model first outputs <WAIT>, which signals for the next source word to be
read in. At the next step, the model generates the first target word (Ya),
which is plugged into the prompt at the next step. This process continues
until <EOS> is generated.

After all the source words have been revealed, the input is no longer partial
and no new words are added to it, but the generation process continues until
<EOS>. Importantly, if the model generates the <WAIT> token, a new source word
is read in, but the <WAIT> token itself is not appended to the partial target.
We illustrate the inference process in Fig. 3 and Algorithm 1.

Algorithm 1 Inference process

⬇

partial_target = []

k = WAIT_K

while True:

partial_source = SOURCE[:k]

prompt = " ".join([

SYS_MSG,

partial_source,

partial_target])

# generate until next full word,

# <EOS> or <WAIT>

next_word = model.generate(prompt)

if next_word == "<EOS>":

# finish sentence

break

elif next_word == "<WAIT>":

# READ action

k += 1

else:

# WRITE action

partial_target.append(next_word)

k += 1

Prompt structure. We follow a similar prompt structure as in Touvron et al.
(2023) (Fig. 4). For the SYSTEM_MESSAGE we used the following text: _"You are
a professional conference interpreter. Given an English text you translate it
into_ {TARGET_LANGUAGE} _as accurately and as concisely as possible, NEVER
adding comments of your own. You output translation when the information
available in the source is unambiguous, otherwise you output the wait token
(_{WAIT_TOKEN}_), not flanked by anything else. It’s important that you get
this right."_. We note that while the system message is only necessary in
zero-shot SiMT scenarios – which we discuss below – for consistency we still
kept it in all the experiments reported here, including those involving
supervised fine-tuning.

      <s>[INST]
      <<SYS>>

      [SYSTEM_MESSAGE]
      <</SYS>>
      Translate this text: PARTIAL_SOURCE [/INST] PARTIAL_TARGET

Figure 4: Prompt structure.

Automatic speech recognition. Given that the LLMs are designed to process text
input, to enable S2TT we first need to extract text from input audio, for
which we use Whisper 333We used whisper-large-v2. (Radford et al., 2023).
Specifically, for each READ action, a new segment of audio, lasting 200 ms, is
added to any previously read audio chunks and then processed by Whisper. This
method of fixed audio windowing often results in partially clipped words. To
address this, we discard the last word predicted by Whisper during each READ
action unless the entire source audio has been read in. We note that this
approach to online ASR is somewhat naive and has room for improvement – as
indicated by a roughly 1 BLEU point decrease due to ASR-related errors (Fig.
9). Since our main objective was to assess the capability of LLMs to perform
SiMT tasks, we leave exploring ways to decrease ASR errors to future work.

##  4 Results

Data. For supervised fine-tuning (SFT), validation and testing, we used MuST-C
v2.0 (Di Gangi et al., 2019) for English-to-German (en-de) and English-to-
Russian (en-ru) translation direction. We randomly selected 4000 sentences for
training and 100 sentences for validation. However, since it is possible that
the dataset that Llama2 was pre-trained on and MuST-C v2.0 (including its
validation and test set) might have overlapping content, we also compiled
another test set, which we call TED-TST-2023. This test set has a similar
content type (TED talks) and follows the same format as the original MuST-C
v2.0, but only includes talks posted after February 2023. The dataset has two
parts: 102 source-target pairs for en-de and 102 for en-ru language pair.
Unless indicated otherwise, we report the results obtained on this test set.

Figure 5: Dependence of latency and quality on $k$ (top panels) and quality-
latency tradeoff curves (bottom panels) for the T2TT mode on the MuST-C v2.0
dataset. For reference, dashed lines indicated GPT-4’s sentence-level (i.e.
with $k$ set to the sentence length) BLEU scores: black for en-de and red for
en-ru.

T2TT. We first analyzed the T2TT performance or our approach on the MuST-C
dataset v2.0 (Di Gangi et al., 2019). To get a sense for the quality-latency
tradeoff, we plot BLEU scores against several different values of $k$ (because
$k$ is the only way to control the translation latency). The results, shown in
Fig. 5, suggest that the LLM’s size is a major factor determining the
translation quality.

S2TT. We next test fine-tuned LLMs and compare them with two recently
published S2TT baselines (Fukuda et al., 2023; Papi et al., 2023) as well as
to OpenAI’s GPT-3.5 and GPT-4 (in zero-shot mode). For as fair a comparison as
possible, we ensured that average lagging (AL) of all of the models below
approximately 2000 ms. For Llama-2 models we set $k=5$ (the other models’
settings are listed in Appendix 5). The boxplots in Figs. 7, 7 and throughout
are drawn based on data from 10 evaluation runs of the same model with the
same parameters on sentence pairs sampled with replacement from TED-TST-2023.
The results show a degradation of translation quality by approximately 1 BLEU
score point compared to T2TT mode, which is to be expected due to ASR errors
(Fig. 9).

Figure 6: S2TT performance of SFT Llama-2 and two recently published models on
the en-de language pair on TED-TST-2023. See also Table 3.

Figure 7: Zero-shot S2TT performance or our approach compared with GPT-3.5 and
GPT-4 on the en-de language pair on TED-TST-2023.

Zero-shot T2TT. Can the LLMs perform the SiMT task zero-shot, that is without
any prior fine-tuning? To answer this question, we used LLMs that had been
fine-tuned with RLHF for instruction following: open-source Llama2-Chat, as
well as GPT-3.5 (gpt-3.5-turbo-0613) and GPT-4 (gpt-4-0613), which were among
the strongest closed-source LLMs available at the time of writing this paper.
In general, with the notable exception of GPT-4, zero-shot performance was
poor. Inspection of the translations revealed that the models consistently
failed to follow the prompt instruction, specifically, (1) generating output
in English rather than the target language, (2) adding expressly prohibited
explanatory comments, (3) restating or summarizing the task, or (4) explaining
the reason for adding <WAIT> tokens). GPT-4 was surprisingly good, performing
better than the supervised fine-tuned Llama2-70B, and we speculate that the
performance of GPT-3.5 and GPT-4 could be further improved with SFT 444SFT was
not available for GPT-3.5 and GPT-4 at the time of writing this paper., more
sophisticated generation strategies and prompt engineering.

Importance of wait tokens. To evaluate the utility of <WAIT> tokens, we
conduct two ablation experiments. In the first experiment we consider a zero-
shot translation scenario in which GPT-4 was not instructed to use <WAIT>
tokens. In the second experiment, we suppress the generation of <WAIT> tokens
in supervised fine-tuned LLMs. The results, as indicated in Table 1, reveal
that GPT-4 demonstrates marginally inferior performance when
$k\in\\{1,2\\}$555We did not investigate the role of <WAIT> tokens for $k>2$,
because GPT-4 almost never generates them for those values of $k$. when not
instructed about <WAIT> tokens. However, it is important to note that in a
zero-shot context, the GPT-3.5 and GPT-4 seldom generated <WAIT> tokens
(almost never for $k>2$). Therefore, the directive to employ these tokens only
exhibited a discernible impact for smaller values of $k$. By contrast, in the
SFT scenario, suppressing <WAIT> tokens led to significantly decreased
performance for both the 13B and 70B versions of LLama-2 (Table 1 (b, c)).

Figure 8: After fine-tuning, Llama-2 generates <WAIT> tokens predominantly
after function words (especially articles and prepositions).

Figure 9: Performance decrease due to ASR-related errors. In T2TT mode,
Llama2-70b performs by about 1 BLEU score point better than the the same model
on the same data in S2TT mode.

$k$ | w/ <WAIT> | w/o <WAIT>
---|---|---
1 | 15.23 | 14.88
2 | 17.17 | 15.66
(a)

$k$ | w/ <WAIT> | w/o <WAIT>
---|---|---
1 | 14.76 | 10.80
2 | 14.97 | 11.94
4 | 17.42 | 15.67
(b)

$k$ | w/ <WAIT> | w/o <WAIT>
---|---|---
1 | 17.17 | 4.64
2 | 16.83 | 7.84
4 | 19.24 | 14.80
(c)

Table 1: Removing the instruction to generate or suppressing the <WAIT> token
degrades performance. The numbers indicate BLEU scores on TED-TST-2023 (en-de)
in T2TT mode for GPT-4 (a), supervised fine-tuned Llama-2-13b-hf (b) and
Llama-2-70b-hf (c).

To gain insight into where LLama-2 tended to insert the <WAIT> token, we plot
the distribution of words after which the SFT models generated this token.
Fig. 9 shows that most of the time the model generated <WAIT> after function
words – which makes sense – rather than content words, indicating that it had
learned to choose appropriately between READ and WRITE actions.

##  5 Conclusion and future directions

We have shown that with minimal fine-tuning and without resorting to
sophisticated training techniques (e.g. checkpoint averaging (Fukuda et al.,
2023)), an off-the-shelf pre-trained LLM can perform simultaneous translation
and achieve encouraging results that rival some of the recent SiMT models.
This opens interesting directions to be explored in future work, such as
multilingual fine-tuning, self-instruct (Wang et al., 2023c) and human
preference tuning (Ouyang et al., 2022).

There are several reasons to believe that we are far from unlocking the full
potential of LLMs for SiMT. First, we followed the practice – standard in the
SiMT literature – of evaluating the model on individual sentences randomly
sampled from continuous prose. However, many (if not the majority of) short
sentences are ambiguous when taken out of context. Even human conference
interpreters routinely prepare for an upcoming translation job, studying
relevant materials, which means that they do not have to translate sentences
taken out of context. For this reason, we believe that the most
straightforward way to boost the performance of future LLM-based SiMT systems
is to insert background information into the prompt. Second, the big
difference in zero-shot performance between GPT-3.5 and GPT-4 suggests that
size is likely the biggest factor determining the model’s translation quality,
and that further gains can be achieved once SFT becomes available for these
closed-source models.

## Limitations

In conclusion, we note that there are several performance bottlenecks that
must be addressed before our approach can be deployed for simultaneous
translation in the real world. As we show in _Appendix_ 2, these bottlenecks
result from a long system message, which is often longer than the source
sentence itself, as well as delays introduced by the ASR sybsystem and weight
quantization. We believe that these issues are not prohibitive. Specifically,
instead of using a separate ASR model, future work might follow an end-to-end
approach similar to Fathullah et al. (2023), in which instead of being
converted into text with an separate ASR model, the audio is directly mapped
into the LLM’s embedding space, reducing the system’s overall latency.
Efficient quantization schemes, faster algorithms and hardware support for low
bit-width arithmetic are also promising directions. Finally, because Llama-2
was trained predominantly on English text, its tokenizer represents English
more efficiently than other languages. That is, fewer tokens on average are
needed to encode a text in English than a text of the same length (in
characters) in another, less represented, language. Thus, future LLMs pre-
trained on a linguistically more balanced dataset, might be slightly faster at
inference.

## References

  * Agostinelli et al. (2023) Victor Agostinelli, Max Wild, Matthew Raffel, Kazi Asif Fuad, and Lizhong Chen. 2023.  Simul-llm: A framework for exploring high-quality simultaneous translation with large language models.  _arXiv preprint arXiv:2312.04691_.
  * Alinejad et al. (2018) Ashkan Alinejad, Maryam Siahbani, and Anoop Sarkar. 2018.  Prediction improves simultaneous neural machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3022–3027, Brussels, Belgium. Association for Computational Linguistics.
  * Arivazhagan et al. (2019) Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz, Ruoming Pang, Wei Li, and Colin Raffel. 2019.  Monotonic infinite lookback attention for simultaneous machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1313–1323, Florence, Italy. Association for Computational Linguistics.
  * Bangalore et al. (2012) Srinivas Bangalore, Vivek Kumar Rangarajan Sridhar, Prakash Kolan, Ladan Golipour, and Aura Jimenez. 2012.  Real-time incremental speech-to-speech translation of dialogs.  In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 437–445.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  " O’Reilly Media, Inc.".
  * Cherry and Foster (2019) Colin Cherry and George Foster. 2019.  Thinking slow about latency evaluation for simultaneous machine translation.  _arXiv preprint arXiv:1906.00048_.
  * Chiu and Raffel (2018) Chung-Cheng Chiu and Colin Raffel. 2018.  Monotonic chunkwise attention.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Cho and Esipova (2016) Kyunghyun Cho and Masha Esipova. 2016.  Can neural machine translation do simultaneous translation? _arXiv preprint arXiv:1606.02012_.
  * Dalvi et al. (2018) Fahim Dalvi, Nadir Durrani, Hassan Sajjad, and Stephan Vogel. 2018.  Incremental decoding and training methods for simultaneous translation in neural machine translation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 493–499, New Orleans, Louisiana. Association for Computational Linguistics.
  * Di Gangi et al. (2019) Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019.  MuST-C: a Multilingual Speech Translation Corpus.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2012–2017, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Elbayad et al. (2020) Maha Elbayad, Laurent Besacier, and Jakob Verbeek. 2020.  Efficient Wait-k Models for Simultaneous Machine Translation.  In _Proc. Interspeech 2020_ , pages 1461–1465.
  * Fathullah et al. (2023) Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. 2023.  Prompting large language models with speech recognition abilities.
  * Fügen et al. (2007) Christian Fügen, Alex Waibel, and Muntsin Kolss. 2007.  Simultaneous translation of lectures and speeches.  _Machine translation_ , 21:209–252.
  * Fukuda et al. (2023) Ryo Fukuda, Yuta Nishikawa, Yasumasa Kano, Yuka Ko, Tomoya Yanagita, Kosuke Doi, Mana Makinae, Sakriani Sakti, Katsuhito Sudoh, and Satoshi Nakamura. 2023.  NAIST simultaneous speech-to-speech translation system for IWSLT 2023.  In _Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023)_ , pages 330–340, Toronto, Canada (in-person and online). Association for Computational Linguistics.
  * Grissom II et al. (2014) Alvin Grissom II, He He, Jordan Boyd-Graber, John Morgan, and Hal Daumé III. 2014.  Don’t until the final verb wait: Reinforcement learning for simultaneous machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1342–1352, Doha, Qatar. Association for Computational Linguistics.
  * Gu et al. (2017) Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor O.K. Li. 2017.  Learning to translate in real-time with neural machine translation.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 1053–1062, Valencia, Spain. Association for Computational Linguistics.
  * Guo et al. (2023) Shoutao Guo, Shaolei Zhang, and Yang Feng. 2023.  Learning optimal policy for simultaneous machine translation via binary search.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2318–2333, Toronto, Canada. Association for Computational Linguistics.
  * Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Jalili Sabet et al. (2020) Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020.  SimAlign: High quality word alignments without parallel training data using static and contextualized embeddings.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1627–1643, Online. Association for Computational Linguistics.
  * Kano et al. (2022) Yasumasa Kano, Katsuhito Sudoh, and Satoshi Nakamura. 2022.  Simultaneous neural machine translation with prefix alignment.  In _Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)_ , pages 22–31, Dublin, Ireland (in-person and online). Association for Computational Linguistics.
  * Ko et al. (2023) Yuka Ko, Ryo Fukuda, Yuta Nishikawa, Yasumasa Kano, Katsuhito Sudoh, and Satoshi Nakamura. 2023.  Tagged end-to-end simultaneous speech translation training using simultaneous interpretation data.  In _Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023)_ , pages 363–375, Toronto, Canada (in-person and online). Association for Computational Linguistics.
  * Liu et al. (2020a) Danni Liu, Gerasimos Spanakis, and Jan Niehues. 2020a.  Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection.  In _Proc. Interspeech 2020_ , pages 3620–3624.
  * Liu et al. (2020b) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020b.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Ma et al. (2019a) Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. 2019a.  STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3025–3036, Florence, Italy. Association for Computational Linguistics.
  * Ma et al. (2020a) Xutai Ma, Mohammad Javad Dousti, Changhan Wang, Jiatao Gu, and Juan Pino. 2020a.  SIMULEVAL: An evaluation toolkit for simultaneous translation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 144–150, Online. Association for Computational Linguistics.
  * Ma et al. (2019b) Xutai Ma, Juan Miguel Pino, James Cross, Liezl Puzon, and Jiatao Gu. 2019b.  Monotonic multihead attention.  _CoRR_ , abs/1909.12406.
  * Ma et al. (2020b) Xutai Ma, Juan Miguel Pino, James Cross, Liezl Puzon, and Jiatao Gu. 2020b.  Monotonic multihead attention.  In _International Conference on Learning Representations_.
  * Nascimento et al. (2023) Nathalia Nascimento, Paulo Alencar, and Donald Cowan. 2023.  Gpt-in-the-loop: Adaptive decision-making for multiagent systems.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Papi et al. (2022) Sara Papi, Marco Gaido, Matteo Negri, and Marco Turchi. 2022.  Over-generation cannot be rewarded: Length-adaptive average lagging for simultaneous speech translation.  In _Proceedings of the Third Workshop on Automatic Simultaneous Translation_ , pages 12–17, Online. Association for Computational Linguistics.
  * Papi et al. (2023) Sara Papi, Matteo Negri, and Marco Turchi. 2023.  Attention as a guide for simultaneous speech translation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13340–13356, Toronto, Canada. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Polák et al. (2022) Peter Polák, Ngoc-Quan Pham, Tuan Nam Nguyen, Danni Liu, Carlos Mullov, Jan Niehues, Ondřej Bojar, and Alexander Waibel. 2022.  CUNI-KIT system for simultaneous speech translation task at IWSLT 2022.  In _Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)_ , pages 277–285, Dublin, Ireland (in-person and online). Association for Computational Linguistics.
  * Radford et al. (2023) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023.  Robust speech recognition via large-scale weak supervision.  In _International Conference on Machine Learning_ , pages 28492–28518. PMLR.
  * Raffel et al. (2017) Colin Raffel, Minh-Thang Luong, Peter J. Liu, Ron J. Weiss, and Douglas Eck. 2017.  Online and linear-time attention by enforcing monotonic alignments.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, page 2837–2846. JMLR.org.
  * Satija and Pineau (2016) Harsh Satija and Joelle Pineau. 2016.  Simultaneous machine translation using deep reinforcement learning.
  * Sridhar et al. (2013) Vivek Kumar Rangarajan Sridhar, John Chen, Srinivas Bangalore, Andrej Ljolje, and Rathinavelu Chengalvarayan. 2013.  Segmentation strategies for streaming speech translation.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 230–238.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Wang et al. (2023a) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. 2023a.  A survey on large language model based autonomous agents.
  * Wang et al. (2023b) Minghan Wang, Jinming Zhao, Thuy-Trang Vu, Fatemeh Shiri, Ehsan Shareghi, and Gholamreza Haffari. 2023b.  Simultaneous machine translation with large language models.  _arXiv preprint arXiv:2309.06706_.
  * Wang et al. (2023c) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023c.  Self-instruct: Aligning language models with self-generated instructions.
  * Wang et al. (2023d) Zekun Wang, Ge Zhang, Kexin Yang, Ning Shi, Wangchunshu Zhou, Shaochun Hao, Guangzheng Xiong, Yizhi Li, Mong Yuan Sim, Xiuying Chen, Qingqing Zhu, Zhenzhu Yang, Adam Nik, Qi Liu, Chenghua Lin, Shi Wang, Ruibo Liu, Wenhu Chen, Ke Xu, Dayiheng Liu, Yike Guo, and Jie Fu. 2023d.  Interactive natural language processing.
  * Yarmohammadi et al. (2013) Mahsa Yarmohammadi, Vivek Kumar Rangarajan Sridhar, Srinivas Bangalore, and Baskaran Sankaran. 2013.  Incremental segmentation and decoding strategies for simultaneous translation.  In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_ , pages 1032–1036.
  * Zhang et al. (2020) Ruiqing Zhang, Chuanqiang Zhang, Zhongjun He, Hua Wu, and Haifeng Wang. 2020.  Learning adaptive segmentation policy for simultaneous translation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2280–2289, Online. Association for Computational Linguistics.
  * Zhang and Feng (2021) Shaolei Zhang and Yang Feng. 2021.  Universal simultaneous machine translation with mixture-of-experts wait-k policy.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7306–7317, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhang and Feng (2022) Shaolei Zhang and Yang Feng. 2022.  Information-transport-based policy for simultaneous translation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 992–1013, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhang and Feng (2023) Shaolei Zhang and Yang Feng. 2023.  Hidden markov transformer for simultaneous machine translation.

## Appendix

##  Appendix A Examples of translation

###  A.1 en-de

Source: But in Antarctica, some scientists who are normally pretty levelheaded
are getting a little bit freaked out, I would say, is a fair definition, about
the lowest level ever, at this point in the year, of sea ice.

TransLLaMa: Aber in der Antarktis, einige Wissenschaftler, die normalerweise
recht nüchtern sind, werden ein wenig verängstigt, würde ich sagen, ist eine
gerechte Definition über den niedrigsten Stand, den wir jemals zu diesem
Zeitpunkt im Jahr von Meereis hatten.

Back translation: But in Antarctica, some scientists who are normally quite
sober are getting a little frightened, I would say, is a fair definition of
the lowest level we’ve ever had of sea ice at this point in the year.

Source: I remember when there were so many fossil fuel delegates in Madrid,
but by the time we got to Glasgow a year and a half ago, the delegates from
the fossil fuel companies made up a larger group than the largest national
delegation.

TransLLaMa: Ich erinnere mich an die Zeit, als es so viele Delegierte der
fossilen Brennstoffindustrie in Madrid gab, aber als wir ein Jahr und ein
halbes später in Glasgow waren, machten die Delegierten der fossilen
Brennstoffunternehmen eine größere Gruppe aus als die größte nationale
Delegation.

Back translation: I remember when there were so many fossil fuel industry
delegates in Madrid, but when we were in Glasgow a year and a half later, the
fossil fuel company delegates made up a larger group than the largest national
delegation.

Source: And I’m a really positive person, so it’s actually difficult to say
this, but the way we do philanthropy right now, the way we’ve done it for
decades, is broken.

TransLLaMa: Ich bin eine sehr positive Person, also ist es für mich eigentlich
schwierig, das zu sagen, aber die Art und Weise, wie wir Philanthropie
betreiben, die Art und Weise, wie wir es seit Jahrzehnten betreiben, ist
kaputt.

Back translation: I’m a very positive person, so it’s actually hard for me to
say this, but the way we do philanthropy, the way we’ve done it for decades,
is broken.

Source: Work norms, of course, differ around the world, but we replicated our
original American result in South Korea, which is known by the numbers to be
one of the hardest-working countries in the OECD, and in France, which is
known for other strengths.

TransLLaMa: Natürlich unterscheiden sich die Arbeitsnormen in der Welt, aber
wir konnten unser ursprüngliches Ergebnis in den USA in Südkorea wiederholen,
das bekannt ist als eines der härtesten arbeitenden Länder in der OECD, und in
Frankreich, das für andere Stärken bekannt ist.

Back translation: Of course, labor standards vary around the world, but we
were able to replicate our original result in the US in South Korea, known as
one of the hardest working countries in the OECD, and in France, known for
other strengths.

###  A.2 en-ru

Source: A few years ago, I called for toothpaste from my hotel room, and it
was delivered to me by a robot.

TransLLaMa: Несколько лет назад я попросила зубную пасту из моего номера в
отеле и она была доставлена мне роботом.

Back translation: A few years ago I asked for toothpaste from my hotel room
and it was delivered to me by robot.

Source: It conducted the first quantum video call, and it launched the first
drone that can carry a passenger in air.

TransLLaMa: Она провела первую видео-конференцию с использованием квантовых
технологий и запустила первый беспилотный самолет, способный перевозить
пассажира в воздухе.

Back translation: She held the first video conference using quantum technology
and launched the first unmanned aircraft capable of carrying a passenger in
the air.

Source: And it’s really gratifying to hear from people like Raymond who are
using the technology we are building, and others, for so many wonderful
things.

TransLLaMa: И это действительно приятно слышать от людей, как Раймонд, которые
используют технологии, которые мы создаём, и другие, для таких замечательных
вещей.

model | mode | quantization | system message | size, bn param. | RTF
---|---|---|---|---|---
Ours | T2TT | 16-bit | no | 13 | 1.7
Ours | T2TT | 4-bit | no | 13 | 2.2
Ours | T2TT | 16-bit | yes | 13 | 2.9
Ours | T2TT | 4-bit | yes | 13 | 4.2
Ours | S2TT | 16-bit | no | 13 | 5.9
Ours | S2TT | 4-bit | no | 13 | 7.6
Ours | S2TT | 16-bit | yes | 13 | 8.0
Ours | S2TT | 4-bit | yes | 13 | 9.3
Ours | T2TT | 4-bit | no | 70 | 14.6
Ours | T2TT | 4-bit | yes | 70 | 20.2
Ours | S2TT | 4-bit | no | 70 | 15.3
Ours | S2TT | 4-bit | yes | 70 | 23.9
GPT-4 | T2TT | unknown | yes | unknown | 1.5
GPT-4 | S2TT | unknown | yes | unknown | 4.8
Fukuda et al. (2023) | S2TT | 16-bit | N/A | 1.04 | 0.7
Papi et al. (2023) | S2TT | 16-bit | N/A | 0.176 | 1.4
Table 2: Comparison of our system’s inference times across varying sizes with
selected baselines on en-de. Real-time factor (RTF) is the ratio of the amount
of time taken to process source audio to the length of the source audio
itself. RTF less than one means the model is faster than real time. The RTF
was calculated based on the known length of the audio corresponding to the
source transcripts and the time to complete translation of that text. For T2TT
mode, the source audio transcripts were fed directly to the LLM. We note that
removing the system message from the prompt speeds up inference with no
noticeable drop in quality for supervised fine-tuned models. Loading our
model’s weights with 16-bit (instead of 4-bit) quantization further
accelerates inference. Finally, the use of ASR in S2TT mode substantially
reduces system speed.

Figure 10: Dependence of latency and quality on $k$ (top panels) and quality-latency tradeoff curves (bottom panels) for the S2TT mode on the TED-TST-2023 dataset. For reference, dashed lines indicated GPT-4’s sentence-level (i.e. with $k$ set to the sentence length) BLEU scores: black for en-de and red for en-ru. Figure 11: S2TT en-ru performance of our method on TED-TST-2023. Left panel: supervised fine-tuned Llama-2. Right panel: zero-shot S2TT performance of Llama-2-Chat. All the runs were on TED-TST-2023, with $k=5$ to ensure AL around 2000 ms. Each of the boxplots is drawn based on data from 10 evaluation runs on sentences randomly sampled with replacement from the test set. Figure 12: Average lagging in S2TT mode for the English-Russian language pair. Left panel: supervised fine-tuned Llama-2. Right panel: zero-shot S2TT performance of Llama-2-Chat. All the runs were on TED-TST-2023, with $k=5$ to ensure AL around 2000 ms. Each of the boxplots is drawn based on data from 10 evaluation runs on sentences randomly sampled with replacement from the test set. System | BLEU | LAAL | AL | AP | DAL
---|---|---|---|---|---
gpt-3.5-turbo-0613 (zero-shot) | 2.08 (0.24) | 2637.11 (252.79) | 2574.98 (230.95) | 0.35 (0.0) | 2477.55 (146.26)
gpt-4-0613 (zero-shot) | 21.82 (2.81) | 2448.86 (74.74) | 1998.63 (110.91) | 0.94 (0.03) | 2813.47 (69.48)
Llama-70b-hf (SFT) | 18.41 (1.4) | 2107.57 (59.68) | 1619.64 (76.47) | 0.84 (0.02) | 2454.72 (67.84)
Llama-13b-hf (SFT) | 17.07 (0.68) | 2358.89 (34.11) | 1880.76 (61.77) | 0.88 (0.02) | 2735.34 (40.88)
Papi et al. (2023) | 17.01 (1.0) | 2295.72 (41.54) | 1867.1 (148.69) | 0.77 (0.01) | 3251.38 (139.12)
Fukuda et al. (2023) | 21.08 (1.41) | 2005.39 (71.04) | 1397.33 (85.74) | 0.9 (0.01) | 3066.15 (122.01)
Table 3: Mean performance metrics on en-de of Llama-2 (SFT) compared to some recent S2TT systems and GPT-3.5 and GPT-4 (zero-shot). Then mean and standard deviation (in brackets) are computed over 10 runs of the same model on 102 source-target pairs sampled with replacement from TED-TST-2023. Here we report additional comparisons including latency performance measured using several different metrics, including Average Lagging (AL) (Ma et al., 2019a), Length Adaptive Average Lagging (LAAL) (Papi et al., 2022), Average Proportion (AP) (Cho and Esipova, 2016) and Differentiable Average Lagging (DAL) (Cherry and Foster, 2019). System | BLEU | LAAL | AL | AP | DAL
---|---|---|---|---|---
gpt-3.5-turbo-0613 (zero-shot) | 0.14 (0.1) | 2876.85 (240.03) | 2861.22 (245.91) | 0.28 (0.04) | 2661.22 (231.0)
gpt-4-0613 (zero-shot) | 16.86 (2.27) | 2022.81 (20.3) | 1584.38 (91.81) | 0.82 (0.04) | 2390.11 (23.65)
Llama-70b-hf (SFT) | 20.96 (1.71) | 2252.75 (49.77) | 1937.76 (62.75) | 0.9 (0.08) | 2676.56 (62.11)
Llama-13b-hf (SFT) | 16.9 (1.52) | 2238.6 (48.38) | 1917.46 (90.38) | 0.87 (0.03) | 2641.01 (45.73)
Table 4: Mean performance metrics on en-ru of Llama-2 (SFT) compared to some recent S2TT systems and GPT-3.5 and GPT-4 (zero-shot). Then mean and standard deviation (in brackets) are computed over 10 runs of the same model on 102 source-target pairs sampled with replacement from TED-TST-2023. baseline | Papi et al. (2023) | Fukuda et al. (2023)
---|---|---
parameters | extract-attn-from-layer 5 | source-segment-size 950
| frame-num 2 | la-n 2
| attn-threshold 0.25 | beam 5
| speech-segment-factor 8 | sacrebleu-tokenizer 13a
Table 5: Parameters used for comparisons with baselines on the S2ST en-de
task. For Papi et al. (2023) we used the open-source implementation of the
model (https://github.com/hlt-mt/FBK-fairseq/tree/ master/fbk_works). For
Fukuda et al. (2023) we obtained the source code of the model and weights on
request from the authors. All the evaluations were run in _SimulEval_ (Ma et
al., 2020a) https://github.com/facebookresearch/SimulEval).