# On-the-Fly Fusion of Large Language Models and Machine Translation

Hieu Hoang Huda Khayrallah Marcin Junczys-Dowmunt
Microsoft, 1 Microsoft Way, Redmond, WA 98052, USA
<EMAIL_ADDRESS>
###### Abstract

We propose the on-the-fly ensembling of a machine translation model with an
LLM, prompted on the same task and input. We perform experiments on 4 language
pairs (both directions) with varying data amounts. We find that a slightly
weaker-at-translation LLM can improve translations of a NMT model, and
ensembling with an LLM can produce better translations than ensembling two
stronger MT models. We combine our method with various techniques from LLM
prompting, such as in context learning and translation context.

On-the-Fly Fusion of Large Language Models and Machine Translation

Hieu Hoang Huda Khayrallah Marcin Junczys-Dowmunt Microsoft, 1 Microsoft Way,
Redmond, WA 98052, USA<EMAIL_ADDRESS>

##  1 Introduction

For many English-only NLP tasks, LLMs Brown et al. (2020); Rae et al. (2021);
Smith et al. (2022); Chowdhery et al. (2022); Touvron et al. (2023a, b) are
the clear state-of-the art—e.g. sentiment analysis Zhang et al. (2023c),
summarization Zhang et al. (2023b). However, dedicated MT outperforms all but
the largest closed source LLMs Jiao et al. (2023) and dedicated MT is still
stronger in low resource settings Hendy et al. (2023); Robinson et al. (2023).

We propose a novel combination of the strengths of an LLM and a dedicated MT
model via token-level fusion. LLMs are typically trained on more data than
dedicated MT models, and have more parameters. While LLMs are exposed to some
parallel data Briakou et al. (2023), they are trained on vastly more
monolingual data, which likely gives them different domain coverage and more
fluency than most MT models. MT models are more directly trained on the task
of translation. For example, Jiao et al. (2023) found ChatGPT is more likely
to hallucinate but is stronger at the spoken domain, while dedicated NMT
models are stronger for medical domains and social-media-style noisy text.
LLMs easily can be prompted with auxiliary information, such as domain and
document context.

In this work we:

  * •

propose on-the-fly ensembling of an MT model with a prompted-for-translation
LLM,

  * •

demonstrate that a weaker-at-translation LLM can improve translations of a MT
model,

  * •

combine with domain and context prompting,

  * •

and demonstrate our method is better than MT ensembles and non-prompted LLMs.

##  2 Method

We review standard inference of encoder-decoder NMT models and decoder only
LLMs and then introduce our proposed ensemble of the two.

#### Standard Decoding

In encoder-decoder NMT, the probability of token $t$ at the $i^{th}$ time step
is:

$p_{\textsc{MT}}(t_{i})=p_{\textsc{MT}}(t_{i}|t_{j<i},S)$ (1)

It conditions on source sentence $S$ as the input to the encoder and $t_{j<i}$
as the previously generated target tokens in the MT model decoder.

When using a decoder only LLM for translation, the probability of token $t$ at
the $i^{th}$ time step is:

$p_{\textsc{LLM}}(t_{i})=p_{\textsc{LLM}}(t_{i}|M,S,t_{j<i})$ (2)

The concatenation of the prompt $M$, source sentence $S$ and the previous
generated targets are all decoder outputs. The LLM model is prefix-decoded
through the prompt and source, and then allowed to produce the target tokens.
The LLM prompt $M$ can also include additional content.

#### Proposed Ensemble

When combining the two for our ensemble, we have:

$p_{\text{ensemble}}(t_{i})=\lambda
p_{\text{MT}}(t_{i})+(1-\lambda)p_{\text{LLM}}(t_{i})$ (3)

In the ensemble, $p_{\text{MT}}$ and $p_{\text{LLM}}$ condition on the tokens
previously generated by the ensemble. $p_{\text{LLM}}$ still conditions on the
prompt, which can be used to infuse the model with auxiliary information (e.g.
domain or context). $p_{\text{ensemble}}$ reduces to the LLM when $\lambda=0$
and to the MT model when $\lambda=1$.

MT LLM 7B Ensemble w/ LLM 7B LLM 13B Ensemble w/ LLM 13B 0-shot 5-shot
$\lambda$ 0-shot 5-shot 0-shot 5-shot $\lambda$ 0-shot 5-shot column: 1 2 3 4
5 6 7 8 9 10 11 de-en 83.5 82.6 82.8 0.7 83.9 83.9 82.6 83.4 0.7 84.1 84.0 en-
de 85.4 79.4 79.8 0.7 85.5 85.5 63.4 82.4 0.8 85.6 85.6 ru-en 82.8 82.5 82.5
0.5 84.0 84.1 81.4 83.4 0.5 84.2 84.5 en-ru 83.1 80.4 81.1 0.5 83.9 84.2 36.4
81.1 0.8 83.6 83.7 tr-en 88.8 75.2 75.7 0.9 88.8 88.9 78.9 - 0.8 88.9 88.9 en-
tr 90.8 57.8 58.2 1.0 90.8 90.8 40.3 69.4 - - - ha-en 60.1 47.0 49.3 0.3 54.7
54.7 46.9 49.7 0.3 54.7 54.5 en-ha 63.1 33.1 37.6 1.0 63.1 63.1 38.2 35.7 1.0
63.1 63.1

Table 1: Comet-22 on the test set. Ensembling MT & LLM can improve Comet in
high resource settings where LLM’s Comet is somewhat worse than the MT.
$\lambda$ indicates the mixing rate; higher $\lambda$ means more emphasis on
MT.

##  3 Experimental Setup

We aim to understand how our proposed method performs in high and low resource
settings with strong models. We use the most recent WMT data for our MT
models: WMT22 for German and Russian, WMT21 for Hausa, and WMT18—with
additional Opus data Tiedemann (2012)—for Turkish.

As domain-specific test sets we use the TED-100 Salesky et al. (2021) and
ParaPat Soares et al. (2020). We also used TED-100 and CTXPro Wicks and Post
(2023) for document-level experiments.111For CTXPro, we select a random sample
of 2000 paragraphs for our experiments to reduce compute usage.

We use back translation Sennrich et al. (2016)—with a 1:1 ratio of parallel to
synthetic data—for all language pairs except Turkish. We train Transformer
‘big’ models for German, Russian and Turkish, and ‘base’ for Hausa Vaswani et
al. (2017) using Marian Junczys-Dowmunt et al. (2018).222We convert models
from Marian to Hugging Face format.

We use Llama2 Touvron et al. (2023b) with 7 and 13 billion parameters as LLMs.
The 32k token SentencePiece Kudo and Richardson (2018) model from Llama2 was
used for both source and target MT tokenization.333The target side vocabs must
match between the LLM and MT model to be able to ensemble; the source could
potentially be different. Preliminary experiments, however, found it better to
use the same vocab and be able to tie the embeddings.

The optimal mixing ratio is learnt using grid search
$\lambda\in\\{0,0.1,...1\\}$ on the validation set. Results are reported on
the test sets, translation quality is measured using Comet-22 Rei et al.
(2022). We use greedy search for decoding. See § A for additional experimental
details, including prompts.

##  4 Results

Table 1 shows the translation quality of the ensemble using the 7 billion
parameters LLM (col. 1-6). When both models are of reasonable quality (de-en,
ru-en, en-ru), ensembling (col. 5) results in better quality than either alone
(col. 1 & 2).

In all cases, the LLM quality is worse than the MT model but ensembling with
it improves most language directions. For de-en, the MT model is 0.9 Comet
stronger than the LLM. The ensemble still improves over the MT model by 0.6
Comet.

The improvement is minor for en-de, where the LLM was 21.9 points worse than
MT. The LLM translation quality for Turkish in both direction is poor while
the MT is good so the ensembles are essentially reduced to the MT model. Both
models are bad for Hausa and the ensembles are unusable. § A.4 shows the
effect of $\lambda$ on translation quality.

In context learning (ICL): Table 1 (col. 3) shows 5-shot learning tends to
improve LLM quality but has little affect on the ensemble (col. 6).

Larger LLM: Xu et al. (2023) found that Llama-13B suffers from off-target
issues, degrading translation out-of-English compared to the 7B model. We
confirm their results, Table 1 (col. 2 vs 7), and also reproduce their
solution of using 5-shot learning, which can recover and sometime improve LLM
quality (col. 8). However, ensembling with the MT model does not require the
use of ICL (col. 10 vs 11). In general, the larger language model is better
for the ensemble as de-en, en-de and ru-en all improve. Ensembles for Turkish
and Hausa are still not worthwhile due to the poor LLM quality in these lower
resource settings. We use the 7B model in the remainder of this work.

##  5 Analysis

###  5.1 MT Model Ensembling

Given the compute resource required to use LLMs (not to mention train them),
we compare the results of the MT + LLM ensemble to ensembling two MT models.
We create ensembles for German and Russian language pairs consisting of two MT
models.444The models differ only in the random seed. Using the LLM gives
stronger translation quality in all cases except en-de, which is where the LLM
underperforms the MT model by 6 Comet points. In all the other situations, it
is better to ensemble the LLM with an MT model, even though the $2^{nd}$ MT
model has higher translation quality than the LLM by 0.5 to 2.8 Comet. This
suggests that when selecting models for an ensemble, simply choosing the two
highest quality models is insufficient. Diverse models improve ensemble
quality.

MT LLM MT+LLM MT+MT de-en 83.5 83.7 82.6 83.9 83.8 en-de 85.4 85.4 79.4 85.5
85.7 ru-en 82.8 83.0 82.5 84.0 83.1 en-ru 83.1 83.2 80.4 83.9 83.4

Table 2: Comet-22 score for two MT replicas, the LLM, the MT & LLM ensemble,
and the ensemble of the two MT models. The ensembling of the LLM with the MT
model has the highest Comet score in all but one language pair, even though
both the MT models have higher translation quality than the LLM.

###  5.2 Domain Prompting

MT LLM Ensemble prompt: none general +domain general +domain TED ru-en 77.3
78.0 78.5 78.7 78.9 ParaPat de-en 79.7 77.1 78.0 80.0 80.0 en-de 79.1 73.8
73.8 79.2 79.2 ru-en 72.2 74.5 73.9 75.1 75.0 en-ru 78.5 73.7 73.4 79.0 78.7

Table 3: Prompting with domain can be improves Comet for the LLM, but is less
effective for the ensemble.

The flexibility of LLM prompting can be used to add more descriptive task-
specific instructions to improve quality Zhang et al. (2023a). Here, we prompt
for domain (TED talks and patents).

Table 3 shows the additional domain information does not guarantee better LLM
quality. For the TED-100 test set, ensembling sees a 0.2 improvement from an
0.5 LLM increase. Ensembling with or without the domain outperforms either the
MT and LLM models alone. For the TED, the LLM is stronger than the MT, in
contrast to our main results. This highlights the different strengths of each
paradigm—the ensemble leverages both.

###  5.3 Document Context

ICL added static translation examples to the prompt. Here we prompt the LLM
with the previous source sentences from the same document and translations of
that source context (as produced by the model). This should allow the LLM to
enforce consistency across sentences and allow it to better translate
phenomena that requires document-level context such as pronoun disambiguation.

Figure 1 shows the TED-100 results of Comet-22 score against the number of
shots or context sentences. Prompting the LLM with context outperforms few
shot prompting and the ensemble with context (solid orange line) to outperform
all variants of ensembling and LLMs with context or few-shots, as well as the
MT model. Conditioning on the model’s own previous outputs from the same
document context outperforms few shot prompting with the human references of
less related sentences.

Figure 1: TED-100 translation quality for various number of prompt examples
(for few short learning or past context). Prompting with context outperforms
few shot prompting, and it performs best when ensembled.

Prior work found that document level-specific evaluation is required to
evaluate document level phenomena Läubli et al. (2018); Toral et al. (2018);
Vernikos et al. (2022). To this end, we use CTXPro Wicks and Post (2023), a
specialized test suite which evaluates the translation accuracy of targeted
words, given the document context.

MT LLM Ensemble context: none none 10 sent none 10 sent en-de auxiliary 4.5%
7.2% 28.0% 6.2% 13.7% formality 41.9% 38.2% 37.6% 42.7% 43.8% gender 44.6%
38.5% 39.0% 45.8% 45.5% en-ru auxiliary 2.6% 2.3% 24.6% 2.6% 20.9% formality
42.5% 42.6% 46.4% 46.4% 50.0% gender 27.4% 31.9% 36.4% 31.6% 37.6% inflection
28.9% 22.6% 25.6% 29.2% 31.4%

Table 4: CTXPro accuracy. The ensembled models with context perform
particularly well in to Russian.

Table 4 shows accuracy for various phenomena on en-ru and en-de. Adding
context improves accuracy in all-but-one test set. Ensembling with context has
the highest accuracy in 4 of 7 models. See § A.5 for Comet on this data; the
ensemble is always best. So, when balancing Comet and CTXPro accuracy, the
ensemble is best.

###  5.4 Unprompted LM Ensembling

Yee et al. (2019) and Petrick et al. (2023) improve translation by ensembling
with a smaller-scale language model _without_ a task-specific prompt.

We test this by ensembling the MT model with an unprompted LLM. Figure 2 shows
that this causes quality to drop precipitously. The divergence from prior work
may be due to differences in the base models, for example, Petrick et al.
(2023) used an MT model trained on small amount of data, and Yee et al. (2019)
trained their own LM. In our scenario with a strong MT model and a general
purpose LLM, we do not see any benefit from using the LLM purely as a language
model.

(a) ru-en

(b) en-ru

Figure 2: Using an LLM with and without prompting. Without a prompt, the
ensemble is strictly worse than the MT baseline (mixing ratio $\lambda=1$).

##  6 Related work

#### LLMs for MT:

Pretrained LLMs have prompted directly for translation Brown et al. (2020);
Vilar et al. (2023); Hendy et al. (2023); Robinson et al. (2023); Moslem et
al. (2023); Zhang et al. (2023a); Agrawal et al. (2023), and there is also
work on fine-tuning them for MT Li et al. (2023); Chen et al. (2023); Zeng et
al. (2023); Xu et al. (2023); Yang et al. (2023). Our approach is
complimentary—we leverage prompting and in-context learning in this work, and
one could ensemble with a fine-tuned model. Since we perform inference-time
combination of the LLM, we do not have the same training-compute-burden as
fine-tuning. LLMs are also used to create synthetic data for MT models
training Lee et al. (2023).

#### LM fusion:

Much work has explored integrating language models and NMT in various ways
Gulcehre et al. (2015, 2017); Stahlberg et al. (2018); Yee et al. (2019);
Petrick et al. (2023). Past work purely conditions a language model on the
target tokens; in contrast we focus on pretrained LLMs,555 Here we make a
distinction between language models perhaps trained on WMT-provided news
monolingual data and the much larger LLMs trained on vast amounts of web data
and may include task-specific pretraining. and prompt the LLM to produce
translations.

#### Ensembling:

Diverse inputs can be combined to create stronger ensembles Hansen and Salamon
(1990); Dietterich (2000). Various model-combination method have been used in
MT.

System combination of outputs was used for statistical machine translation
(SMT) Bangalore et al. (2001); Heafield and Lavie (2010); Freitag et al.
(2014), and averaging model weights Junczys-Dowmunt et al. (2016) or
ensembling Chung et al. (2016) is used for NMT. We build upon the latter.

Knowledge distilation Buciluǎ et al. (2006); Hinton et al. (2015) inspired
methods can be a way to incorporate diverse models during training Dakwale and
Monz (2017); Khayrallah et al. (2018, 2020), as opposed to during inference.
Jiang et al. (2023) introduce a separate model that combines outputs from
LLMs. We ensemble on-the-fly.

There are various methods proposed for improving translation quality by
combining the adequacy and fluency advantages of SMT and NMT Devlin et al.
(2014); Mi et al. (2016); Junczys-Dowmunt et al. (2016); Stahlberg et al.
(2017); Wang et al. (2017); Khayrallah et al. (2017); Zhang et al. (2021). We
combine the strengths of NMT and LLMs.

##  7 Conclusion

We propose the on-the-fly ensembling of an MT model with an LLM, conditioned
on the source and prompted for translation. We demonstrate that an LLM can
improve translation quality of a NMT model even if the LLM is weaker at
translation, provided the LLM is good enough. We prompt the LLM to imbue the
sentence-based MT model with document-level ability, improving on sentence-
level and context-focused metrics. We find that ensembling with an LLM
performs better than ensembling two MT models, even if each MT model is
stronger than the LLM.

While this work focuses on MT, the same techniques can be explored for other
tasks, and may be especially useful for situations where the LLM and task-
specific model have different properties.

## References

  * Agrawal et al. (2023) Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023.  In-context examples selection for machine translation.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 8857–8873, Toronto, Canada. Association for Computational Linguistics.
  * Akhbardeh et al. (2021) Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. 2021.  Findings of the 2021 conference on machine translation (WMT21).  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 1–88, Online. Association for Computational Linguistics.
  * Bangalore et al. (2001) B. Bangalore, G. Bordel, and G. Riccardi. 2001.  Computing consensus translation from multiple machine translation systems.  In _IEEE Workshop on Automatic Speech Recognition and Understanding, 2001. ASRU ’01._ , pages 351–354.
  * Bañón et al. (2020) Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020.  ParaCrawl: Web-scale acquisition of parallel corpora.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4555–4567, Online. Association for Computational Linguistics.
  * Bojar et al. (2018) Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018.  Findings of the 2018 conference on machine translation (WMT18).  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 272–303, Belgium, Brussels. Association for Computational Linguistics.
  * Briakou et al. (2023) Eleftheria Briakou, Colin Cherry, and George Foster. 2023.  Searching for needles in a haystack: On the role of incidental bilingualism in PaLM’s translation capability.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 9432–9452, Toronto, Canada. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Buciluǎ et al. (2006) Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. 2006.  Model compression.  In _Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’06, page 535–541, New York, NY, USA. Association for Computing Machinery.
  * Chen et al. (2023) Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. 2023.  Improving translation faithfulness of large language models via augmenting instructions.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  Palm: Scaling language modeling with pathways.
  * Chung et al. (2016) Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016.  A character-level decoder without explicit segmentation for neural machine translation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1693–1703, Berlin, Germany. Association for Computational Linguistics.
  * Dakwale and Monz (2017) Praveen Dakwale and Christof Monz. 2017.  Fine-tuning for neural machine translation with limited degradation across in- and out-of-domain data.  In _Proceedings of Machine Translation Summit XVI: Research Track_ , pages 156–169, Nagoya Japan.
  * Devlin et al. (2014) Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014.  Fast and robust neural network joint models for statistical machine translation.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1370–1380, Baltimore, Maryland. Association for Computational Linguistics.
  * Dietterich (2000) Thomas G. Dietterich. 2000.  Ensemble methods in machine learning.  In _Multiple Classifier Systems_ , pages 1–15, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Freitag et al. (2014) Markus Freitag, Matthias Huck, and Hermann Ney. 2014.  Jane: Open source machine translation system combination.  In _Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 29–32, Gothenburg, Sweden. Association for Computational Linguistics.
  * Gulcehre et al. (2015) Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015.  On using monolingual corpora in neural machine translation.
  * Gulcehre et al. (2017) Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, and Yoshua Bengio. 2017.  On integrating a language model into neural machine translation.  _Computer Speech & Language_, 45:137–148.
  * Hansen and Salamon (1990) L.K. Hansen and P. Salamon. 1990.  Neural network ensembles.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 12(10):993–1001.
  * Heafield and Lavie (2010) Kenneth Heafield and Alon Lavie. 2010.  Combining machine translation output with open source: The Carnegie Mellon multi-engine machine translation scheme.  _The Prague Bulletin of Mathematical Linguistics_ , 93:27–36.
  * Hendy et al. (2023) Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023.  How good are gpt models at machine translation? a comprehensive evaluation.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.
  * Jiang et al. (2023) Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023.  LLM-blender: Ensembling large language models with pairwise ranking and generative fusion.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 14165–14178, Toronto, Canada. Association for Computational Linguistics.
  * Jiao et al. (2023) Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023.  Is chatgpt a good translator? yes with gpt-4 as the engine.
  * Junczys-Dowmunt et al. (2016) Marcin Junczys-Dowmunt, Tomasz Dwojak, and Rico Sennrich. 2016.  The AMU-UEDIN submission to the WMT16 news translation task: Attention-based NMT models as feature functions in phrase-based SMT.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 319–325, Berlin, Germany. Association for Computational Linguistics.
  * Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018.  Marian: Fast neural machine translation in C++.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 116–121, Melbourne, Australia. Association for Computational Linguistics.
  * Khayrallah and Koehn (2018) Huda Khayrallah and Philipp Koehn. 2018.  On the impact of various types of noise on neural machine translation.  In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_ , pages 74–83, Melbourne, Australia. Association for Computational Linguistics.
  * Khayrallah et al. (2017) Huda Khayrallah, Gaurav Kumar, Kevin Duh, Matt Post, and Philipp Koehn. 2017.  Neural lattice search for domain adaptation in machine translation.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 20–25, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Khayrallah et al. (2018) Huda Khayrallah, Brian Thompson, Kevin Duh, and Philipp Koehn. 2018.  Regularized training objective for continued training for domain adaptation in neural machine translation.  In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_ , pages 36–44, Melbourne, Australia. Association for Computational Linguistics.
  * Khayrallah et al. (2020) Huda Khayrallah, Brian Thompson, Matt Post, and Philipp Koehn. 2020.  Simulated multiple reference training improves low-resource machine translation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 82–89, Online. Association for Computational Linguistics.
  * Kocmi et al. (2022) Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022.  Findings of the 2022 conference on machine translation (WMT22).  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 1–45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Koehn et al. (2020) Philipp Koehn, Vishrav Chaudhary, Ahmed El-Kishky, Naman Goyal, Peng-Jen Chen, and Francisco Guzmán. 2020.  Findings of the WMT 2020 shared task on parallel corpus filtering and alignment.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 726–742, Online. Association for Computational Linguistics.
  * Koehn et al. (2019) Philipp Koehn, Francisco Guzmán, Vishrav Chaudhary, and Juan Pino. 2019.  Findings of the WMT 2019 shared task on parallel corpus filtering for low-resource conditions.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)_ , pages 54–72, Florence, Italy. Association for Computational Linguistics.
  * Koehn et al. (2018) Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L. Forcada. 2018.  Findings of the WMT 2018 shared task on parallel corpus filtering.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 726–739, Belgium, Brussels. Association for Computational Linguistics.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 - November 4, 2018_ , pages 66–71. Association for Computational Linguistics.
  * Läubli et al. (2018) Samuel Läubli, Rico Sennrich, and Martin Volk. 2018.  Has machine translation achieved human parity? a case for document-level evaluation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4791–4796, Brussels, Belgium. Association for Computational Linguistics.
  * Lee et al. (2023) Seugnjun Lee, Hyeonseok Moon, Chanjun Park, and Heuiseok Lim. 2023.  Data-driven approach for formality-sensitive machine translation: Language-specific handling and synthetic data generation.  _arXiv preprint arXiv:2306.14514_.
  * Li et al. (2023) Jiahuan Li, Hao Zhou, Shujian Huang, Shanbo Cheng, and Jiajun Chen. 2023.  Eliciting the translation ability of large language models via multilingual finetuning with translation instructions.
  * Mi et al. (2016) Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016.  Vocabulary manipulation for neural machine translation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 124–129, Berlin, Germany. Association for Computational Linguistics.
  * Moslem et al. (2023) Yasmin Moslem, Rejwanul Haque, John D. Kelleher, and Andy Way. 2023.  Adaptive machine translation with large language models.  In _Proceedings of the 24th Annual Conference of the European Association for Machine Translation_ , pages 227–237, Tampere, Finland. European Association for Machine Translation.
  * Petrick et al. (2023) Frithjof Petrick, Christian Herold, Pavel Petrushkov, Shahram Khadivi, and Hermann Ney. 2023.  Document-level language models for machine translation.
  * Press and Wolf (2017) Ofir Press and Lior Wolf. 2017.  Using the output embedding to improve language models.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 157–163, Valencia, Spain. Association for Computational Linguistics.
  * Rae et al. (2021) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2021.  Scaling language models: Methods, analysis & insights from training gopher.  _CoRR_ , abs/2112.11446.
  * Rei et al. (2022) Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022.  COMET-22: Unbabel-IST 2022 submission for the metrics shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Robinson et al. (2023) Nathaniel R. Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023.  Chatgpt mt: Competitive for high- (but not low-) resource languages.
  * Salesky et al. (2021) Elizabeth Salesky, Matthew Wiesner, Jacob Bremerman, Roldano Cattoni, Matteo Negri, Marco Turchi, Douglas W. Oard, and Matt Post. 2021.  Multilingual tedx corpus for speech recognition and translation.  In _Proceedings of Interspeech_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Improving neural machine translation models with monolingual data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 86–96, Berlin, Germany. Association for Computational Linguistics.
  * Sloto et al. (2023) Steve Sloto, Brian Thompson, Huda Khayrallah, Tobias Domhan, Thamme Gowda, and Philipp Koehn. 2023.  Findings of the WMT 2023 shared task on parallel data curation.  In _Proceedings of the Eighth Conference on Machine Translation (WMT)_ , Singapore, Singapore. Association for Computational Linguistics.
  * Smith et al. (2022) Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. 2022.  Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model.
  * Soares et al. (2020) Felipe Soares, Mark Stevenson, Diego Bartolome, and Anna Zaretskaya. 2020.  ParaPat: The multi-million sentences parallel corpus of patents abstracts.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 3769–3774, Marseille, France. European Language Resources Association.
  * Stahlberg et al. (2018) Felix Stahlberg, James Cross, and Veselin Stoyanov. 2018.  Simple fusion: Return of the language model.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 204–211, Brussels, Belgium. Association for Computational Linguistics.
  * Stahlberg et al. (2017) Felix Stahlberg, Adrià de Gispert, Eva Hasler, and Bill Byrne. 2017.  Neural machine translation by minimising the Bayes-risk with respect to syntactic translation lattices.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 362–368, Valencia, Spain. Association for Computational Linguistics.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in opus.  In _Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12)_ , Istanbul, Turkey. European Language Resources Association (ELRA).
  * Toral et al. (2018) Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018.  Attaining the unattainable? reassessing claims of human parity in neural machine translation.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 113–123, Brussels, Belgium. Association for Computational Linguistics.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a.  Llama: Open and efficient foundation language models.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b.  Llama 2: Open foundation and fine-tuned chat models.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Vernikos et al. (2022) Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. 2022.  Embarrassingly easy document-level MT metrics: How to convert any pretrained metric into a document-level metric.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 118–128, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Vilar et al. (2023) David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2023.  Prompting PaLM for translation: Assessing strategies and performance.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15406–15427, Toronto, Canada. Association for Computational Linguistics.
  * Wang et al. (2017) Xing Wang, Zhengdong Lu, Zhaopeng Tu, Hang Li, Deyi Xiong, and Min Zhang. 2017.  Neural machine translation advised by statistical machine translation.  In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , AAAI’17, page 3330–3336. AAAI Press.
  * Wicks and Post (2023) Rachel Wicks and Matt Post. 2023.  Identifying context-dependent translations for evaluation set production.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xu et al. (2023) Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2023.  A paradigm shift in machine translation: Boosting translation performance of large language models.
  * Yang et al. (2023) Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023.  Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages.
  * Yee et al. (2019) Kyra Yee, Yann Dauphin, and Michael Auli. 2019.  Simple and effective noisy channel modeling for neural machine translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5696–5701, Hong Kong, China. Association for Computational Linguistics.
  * Zeng et al. (2023) Jiali Zeng, Fandong Meng, Yongjing Yin, and Jie Zhou. 2023.  Tim: Teaching large language models to translate with comparison.
  * Zhang et al. (2023a) Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a.  Prompting large language model for machine translation: A case study.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 41092–41110. PMLR.
  * Zhang et al. (2021) Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, and Yang Liu. 2021.  Neural machine translation with explicit phrase alignment.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:1001–1010.
  * Zhang et al. (2023b) Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2023b.  Benchmarking large language models for news summarization.
  * Zhang et al. (2023c) Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023c.  Sentiment analysis in the era of large language models: A reality check.  _arXiv preprint arXiv:2305.15005_.

## Appendix

##  Appendix A Experimental Details

###  A.1 Hyperparameters

For German, Russian and Turkish, Transformer ‘big’ models were trained (6
layer encoder-decoder, 1024 embedding dimensions, 4096 feed-forward
dimensions, 16 heads) Vaswani et al. (2017). The base Transformer architecture
was used for Hausa (6 layer encoder-decoder, 512 embedding dimensions, 2048
feed-forward dimensions, 8 heads). We use weight tying Press and Wolf (2017).
We train models using Marian Junczys-Dowmunt et al. (2018). We convert MT
models from Marian to Hugging Face format, to allow for inference with Llama2
Touvron et al. (2023b) in the Hugging face library Wolf et al. (2020).

###  A.2 MT Data

The parallel and monolingual training data for German and Russian is from the
WMT22Kocmi et al. (2022)666https://www.statmt.org/wmt22/ shared task. The
Hausa data is from WMT21Akhbardeh et al.
(2021).777https://www.statmt.org/wmt21/ The Turkish evaluation data was based
on WMT18 Bojar et al. (2018)888https://www.statmt.org/wmt18/ and training data
also includes additional data from Opus Tiedemann (2012), excluding Paracrawl
Bañón et al. (2020), since such noisy data Khayrallah and Koehn (2018) would
require filtering Koehn et al. (2018, 2019, 2020); Sloto et al. (2023).

| German | Russian | Turkish | Hausa
---|---|---|---|---
Training | 290.4m | 38m | 49.5m | 600k
Validation | 1000/1002 | 1000/1002 | 3007 | 2000
| WMT21 | WMT21 | newstest2017 | newsdev2021
Test set | 1984/2037 | 2016/2037 | 3,000/3,602 | 4456/4459
| WMT22 | WMT22 | newstest2018 | newstest2021
TED-100 | - | 1132 | - | -
ParaPat | 2000 | 2000 | - | -
CTXPro (paragraphs) | 2000 | 2000 | - | -
Table 5: Parallel Datasets used in this work.  | German | Russian | Turkish | Hausa
---|---|---|---|---
| en | de | en | ru | en | tr | en | ha
news-commentary-v18 | 0.9m | 0.5m | 0.9m | 0.5m |  |  |  |
europarl-v10 | 2.3m | 2.1m |  |  |  |  |  |
news (all) | 257.2m | 468.9m | 257.2m | 142.7m |  |  |  |
news.2016 |  |  |  |  | 18.2m | 1.7m |  |
news.2017 |  |  |  |  | 26.8m | 3.0m |  |
news.2018 |  |  |  |  |  |  | 18.1m |
news.2019 |  |  |  |  |  |  | 33.6m |
news.2020 |  |  |  |  |  |  | 41.4m |
CommonCrawl |  |  |  |  |  | 511.2m |  | 8.5m
Table 6: Monolingual Datasets

###  A.3 Prompting

Figure 3, Figure 4, Figure 5, and Figure 6 describe the various prompts we
use.

Translate the following sentence from {src-language} to {tgt-language}:

{src-language}: {src}

{tgt-language}:

Figure 3: Baseline translation prompt

Translate the following sentence from {src-language} to {tgt-language} in a
{style} style:

{src-language}: {src}

{tgt-language}:

Figure 4: Translation prompt with domain

Translate the following sentence from {src-language} to {tgt-language}:

{src-language}: {src-1}

{tgt-language}: {tgt-1}

…

{src-language}: {src-n}

{tgt-language}: {tgt-n}

{src-language}: {src}

{tgt-language}:

Figure 5: n-shot translation prompt

Translate the following sentence from {src-language} to {tgt-language}:

{src-language}: {previous-src-n}

{tgt-language}: {previous-translation-n}

…

{src-language}: {previous-src}

{tgt-language}: {previous-translation}

{src-language}: {src}

{tgt-language}:

Figure 6: Context-aware translation prompt

###  A.4 $\lambda$

Figure 7 shows translation quality as we vary the mixing ratio, $\lambda$.
Note that $p_{\text{ensemble}}$ reduces to the LLM when $\lambda=0$ and to the
MT model when $\lambda=1$.

For our results in the main section, we selected $\lambda$ on validation set
translation quality. Here we see that in cases where both models are
reasonably strong (de-en, ru-en, and en-ru) the ensembling provides a quality
boost.

(a) de-en

(b) en-de

(c) ru-en

(d) en-ru

(e) tr-en

(f) en-tr

(g) ha-en

(h) en-ha

Figure 7: Ensembling MT model with 7B parameter LLM. Graphs shows Comet-22 vs
mixing ratio

(a) en-de gender

(b) en-de auxiliary

(c) en-de formality

(d) en-ru gender

(e) en-ru inflection

(f) en-ru auxiliary

(g) en-ru formality

Figure 8: Comet-22 on the data in CTXPro.

###  A.5 COMET-22 CTXPro

Figure 8 shows the Comet-22 scores corresponding to the document translation
accuracy show in Table 4. The ensemble is always best on this data, then the
MT, and then the LLM.


# FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA
Transmission

Zeling Zhang111These authors contribute equally to this work., Dongqi
Cai111These authors contribute equally to this work., Yiran Zhang, Mengwei Xu,
Shangguang Wang, Ao Zhou Beijing University of Posts and Telecommunications
(BUPT)

###### Abstract.

Communication overhead is a significant bottleneck in federated learning (FL),
which has been exaggerated with the increasing size of AI models. In this
paper, we propose FedRDMA, a communication-efficient cross-silo FL system that
integrates RDMA into the FL communication protocol. To overcome the
limitations of RDMA in wide-area networks (WANs), FedRDMA divides the updated
model into chunks and designs a series of optimization techniques to improve
the efficiency and robustness of RDMA-based communication. We implement
FedRDMA atop the industrial federated learning framework and evaluate it on a
real-world cross-silo FL scenario. The experimental results show that FedRDMA
can achieve up to 3.8$\times$ speedup in communication efficiency compared to
traditional TCP/IP-based FL systems.

††copyright: none

##  1\. Introduction

#### Cross-silo FedLLM

Large Language Models (LLMs) have demonstrated remarkable proficiency in
handling generic machine learning tasks (robinson2022leveraging, ;
guo2023images, ; zhang2023sentiment, ; sun2023text, ; chae2023large, ). This
advancement has led numerous companies to develop their LLMs for various
applications, including computational phenotyping and medical information
extraction in the medical field (zeng2018natural, ; goel2023llms, ), as well
as data mining and generation from text in the financial sector
(huang2023finbert, ; yang2020finbert, ). Amid concerns regarding data privacy
(voigt2017eu, ) and the challenge of isolated training data
(yang2019federated, ; li2021survey, ), federated learning (FL) has emerged as
the predominant method for collaboratively training LLMs without sharing raw
data, known as FedLLM (cai2023federated, ; cai2023efficient, ;
xu2023federated, ; chen2023federated, ; zhao2024llm, ). This study
concentrates on cross-silo FedLLM, where the clients are powerful industrial
servers equipped with high bandwidth and computational resources.

#### Cross-silo FedLLM Communication Overhead

Although acquiring more powerful GPUs can lessen computation overhead,
addressing the communication overhead on WANs remains a formidable challenge,
even with high bandwidth. For instance, when federating full-tuning of the
GPT-2 model (117M parameters) (radford2019language, ) with two NVIDIA A800 80G
GPUs and 10Gbps bandwidth, it still takes 45.9s to transfer the model weights
per round, accounting for more than 44.97% of the total FL time. This delay
stems from repeated memory copies and context switchings due to the TCP
protocol stack and the inherent delay of WAN itself, as will be elaborated in
§ 2.1.

#### Communication-Efficient RDMA

Remote Direct Memory Access (RDMA) is a technique that has recently become
widely used to reduce communication overhead in local-area network (LAN)
distributed machine learning (xue2019fast, ; tian2021accelerating, ). It could
directly transfer data between the memories of two RDMA-enabled servers,
bypassing the CPU and OS kernel, which could reduce the communication overhead
by up to 98.8% as shown in § 2.2. However, RDMA requires a lossless network
environment to fullfill its potential, making it nearly impractical on WANs.
This is because packet loss and delays, common on WANs, can lead to numerous
retransmissions, which is unaccpectable by RDMA (guo2016rdma, ), thus
disminishing RDMA’s performance improvemen tor even causing transmission
failures.

#### Our Solution: FedRDMA

Inspired by the modular parameters in LLMs, we propose FedRDMA to manage the
application layer traffic flow by transferring small packets in a chunked,
sequential, and smooth manner. To further enhance performance, we design a
RDMA memory pool with in-place buffering and reversed receiving strategy to
effectively reassemble chunks. Our contributions are as follows:

  * •

We conduct preliminary experiments to demonstrate that, despite the high
bandwidth and computation resources, cross-silo FedLLM still suffers from high
communication overhead.

  * •

We propose FedRDMA, a communication-efficient cross-silo FedLLM system
featuring chunked RDMA transmission and a series of optimizations.

  * •

We implement FedRDMA atop the industrial FedLLM framework FATE
111https://github.com/FederatedAI/FATE and conduct extensive experiments to
demonstrate it saves up to 3.8$\times$ communication time compared to TCP-
based FedLLM systems.

##  2\. Background

###  2.1. FedLLM Communication Overhead

Federated learning has emerged as a prominent topic in recent research
(li2020review, ; zhang2021survey, ; rieke2020future, ). It is a distributed
machine learning approach that allows training locally on multiple devices
without centralizing all data on a central server. However, federated
learning’s effectiveness is significantly influenced by network conditions on
WAN environments (li2020federated, ; huang2022crosssilo, ), where it faces
constraints from latency and additional overhead due to network communication.
The recent emergence of large language models (kasneci2023chatgpt, ;
chang2023survey, ; wei2022emergent, ; chen2023federated, ) further exacerbates
the communication burden in federated learning. The substantial size of these
models, often exceeding 100MB, poses challenges for efficient transmission on
complex WAN settings.

Communication overhead in cross-silo FedLLM.

Figure 1. FedLLM convergence performance under different bandwidth.

In Figure 1, we illustrate the proportion of communication time to the overall
federated learning time when full-tuning GPT2 with two NVIDIA A800 80G GPUs on
a WAN environment with a 20ms round-trip time, considering different local
bandwidths. In scenario with the lowest bandwidth, the communication time
significantly exceeds the computation time by 28.4$\times$. Furthermore, it is
clear that even with a high bandwidth of 10 Gbps, the communication time still
constitutes a substantial 44.97% of the total federated learning time.

Additionally, Figure 1 reveals that in FedLLM, the model transfer performance
does not scale linearly with increasing bandwidth. This limitation stems from
several issues associated with the TCP protocol in high-bandwidth, high-
performance environments (gamess2016evaluation, ), as illustrated in Figure 2.
Challenges include the memory burden from multiple data copies, CPU overhead
due to frequent context switching between user and kernel modes, and issues
inherent to TCP/IP protocol design, such as slow start and complex congestion
and traffic control mechanisms.

These factors introduce significant challenges for cross-silo federated
training of large-scale models, which necessitates high-performance, large
data transfers on complex WAN environments.

Current solutions and their limitations. Some improvement methods, such as
PEFT, have been proposed. Parameter-Efficient Fine-Tuning (PEFT) is a widely
recognized fine-tuning approach. It only fine-tunes a small fraction
(typically less than 1%) of pretrained models’s parameters on new tasks data,
thereby alleviating the training and communication costs of large pretrained
models. PEFT has been widely applied into FL to reduce communication overhead
(cai2023efficient, ; cai2023federated, ; jiang2023low, ; zhao2023fedprompt, ;
malaviya2023reducing, ). However, PEFT typically converges slower compared to
full parameter tuning, and the reduction in trainable parameters inevitably
leads to a decline in model performance (qin2023federated, ). Meanwhile, there
are some other optimization methods such as backpropagation-free training
paradigms (xu2023federated, ), model compression/quantization (wu2018error, ;
bernstein2018signsgd, ), intelligent client selection and data sampling
(li2021hermes, ; lipyramidfl, ; lai2020oort, ). Most of them primarily target
cross-device federated learning, which is not our focus.

###  2.2. Communication-Efficient RDMA

In previous studies (liu2020accelerating, ), RDMA has been investigated to
address the aforementioned issues.

Figure 2. TCP/IP vs RDMA protocal.

Figure 2 highlights the differences between the RDMA and TCP protocol. RDMA
technology allows direct data transfer between the memory of two nodes, known
as memory regions (MRs) in RDMA, bypassing multiple memory copies and context
switchings. This enables ultra-low latency data processing and ultra-high
throughput transmission.

Despite the many advantages of RDMA, its has stringent network environment
requirements. For instance, lossless Ethernet is usually one of the mandatory
requirements (guo2016rdma, ) to enable RDMA, primarily because commercial RDMA
protocols such as RoCEv2 typically support only the Go-Back-N algorithm.
(towsley1979stutter, ). This results in unacceptable recovery time after
packet loss, disminishing its efficiency compared to TCP. However,
constructing lossless Ethernet requires the entire network link to support key
features such as Priority Flow Control (PFC) (balakrishnan1998priority, ) and
Explicit Congestion Notification (ECN) (floyd1994tcp, ), which are challenging
to implement on WANs.

Benefits and limitations of RDMA.

(a) In-Domain (LAN)

(b) Cross-Domain (WAN)

Figure 3. RDMA brings significant performance improvement in-domain but fails
to work cross-domain.

Figure 3(a) illustrates the performance comparison between TCP and RDMA in a
LAN, showing a significant performance improvement of RDMA over TCP. However,
RDMA’s strict network environment requirements limit its usability on WANs. As
Figure 3(b) shows, RDMA struggles on WANs, operating only at very low
bandwidths without fully leveraging available bandwidth. This is because RDMA
sends data too quickly, causing intermediate nodes in WAN to be overwhelmed to
cache and forward a large amount of data within a short period, leading to
nearly 100% packet loss (irland1978buffer, ). Once packet is lost, RDMA steps
back N packets for retransmission. This further increases the amount of data
on WAN and triggers more packet loss, thus creating a vicious cycle that
causes RDMA to fall into endless retransmissions, resulting in transmission
failure.

Figure 4. The process of chunking packet for smooth RDMA transfer.

Its rationale is that, in a single RDMA transmission scenario shown in Figure
4, The entire data to be transmitted is denoted as $X$. We can observe various
switchs and link nodes in this complex WAN, each with different brands,
models, and traffic-carrying capacities. The physical media connecting them
could also vary, such as different types of fiber optics and twisted pair
cables. Therefore, when RDMA sends large data packets at once, the weak part
may not be able to cache and forward, leading to a vicious cycle of packet
loss and retransmission.

##  3\. Design

###  3.1. FedRDMA: Chunked RDMA Transmission

We propose FedRDMA to address the aforementioned problem. The main idea behind
FedRDMA is to split large data that appears in FL transmission into smaller
chunks and send them sequentially to reduce the volume of data present on the
link at any given time, thus controlling traffic flows in the application
layer. This helps smooth out bursty traffic and maintain stability on WAN
environment.

To do that, we use $s$ as the base chunk size to divide $X$ into $n$ smaller
chunks denoted as {$X_{1}$,$X_{2}$, $...$, $X_{n}$}. Each chunk is prefixed
with a header that includes its sequence number and the total count of chunks,
alongside the standard header details. These chunks are then sent
sequentially. We only send the next chunk after receiving the ACK of previous
one. This ensures that at any given time, there is at most one chunk present
on the link, thus only requiring the link to be capable of trasmitting one
maximum chunk size $s$. Each chunk is temporarily stored according to the
information in header in the receiver. Once all chunks are received, they are
parsed and reassembled to complete the RDMA transmission process.

FedRDMA enables RDMA to operate stably on WANs, with efficiency far surpassing
TCP protocols. Moreover, it seamlessly integrates with other optimization
methods such as PEFT to achieve even greater improvements as will be validated
in §4.4. At the same time, FedRDMA does not alter any algorithmic foundations
and guarantees the same convergence performance and robustness as the original
algorithm, which is crucial for industrial cross-silo federated learning.

###  3.2. FedRDMA-E: Optimized Transmission

While FedRDMA effectively addresses the issue of RDMA being unusable on WANs,
it also introduces additional overhead and latency. Whenever splitting occurs,
reassembly is required. This necessitates constructing a chunk header for each
chunk separately at sending and parsing it when receiving. This introduces
additional CPU overhead and increases the total amount of data transmitted.
Moreover, when the number of chunks is relatively high, the space required to
store temporary chunks imposes a substantial memory burden.

Most importantly, since RDMA does not require CPU intervention at the
receiver, sending and receiving (a.k.a, writting and reading) are
asynchronous. To prevent the issue of data overlap caused by RDMA sending too
quickly, it is necessary to introduce a artifical transmission delay to
slowdown RDMA. In actual experiments, this delay needs to be set relatively
large to ensure the robustness of the system. The performance degradation
caused by all these additional overheads is significant and cannot be ignored.

Therefore, we have optimized FedRDMA to address the issues encountered above,
simplifying its reference as FedRDMA-E. The main idea for this optimization is
to utilize pools of large receive MRs and send all chunks from back to front
to the corresponding addresses to avoid additional chunk header, memory burden
and further reassembly.

Figure 5. FedRDMA-E Workflow. Client B is sending chunked data X to Client A’s
A2 memory region.

To elaborate, as shown in Figure 5, in FedRDMA-E, each client has one send MR
and a pool of receive MRs. The capacity of these requested MRs should exceed
the maximum data size of all transmissions, i.e., larger than the memory
footprint of the LLM itself. Each client maintains a pointer that points to
one of the receive MR in another client. After completing every transmission,
this pointer either points to the next receive MR or loops back to the
beginning. Figure 5 depicts an example of FedRDMA-E workflow that Client B is
sending chunked data $X$ to Client A, with the pointer aimed at Receive MR A2.
At the sender Client B, the header is only attached to the first chunk. During
transmission, each chunk of data $X_{n}$, based on its sequential position
within the entire data set $X$, is sent directly to the matching addresses in
Client A’s Receive MR A2, from the last chunk to the first. Therefore,
receiver can utilize the allocated receive MR directly as a temporary storage
pool, and only needs to periodicly check the first few bytes of the MR. If a
valid chunk header information is found, it indicates that the sender has
completed the transmission of all chunks, and the complete data is already
sequentially arranged in the receive MR.

FedRDMA-E reduces the header constructing and parsing operations from $n$ to 1
and brings the benefits that there is no need for extra temporary storage for
further sorting and reassembly, thus significantly saving memory and CPU
overhead. Furthermore, since each chunk data $X_{n}$ has a unique destination
address in the Receive MR, there is no conflict between sending new chunks and
overwriting unread ones. This effectively eliminates the need for the
transmission delay to slowdown RDMA, resulting in significant time savings and
enhancing system performance.

##  4\. Evaluation

In this section, we compare the performance of FedRDMA with FedLLM to show its
end-to-end efficiency. We also investigate the impact of different system
hyperparameters on FedRDMA, such as different bandwidths ranging from 1Gbps to
100Gbps and size of the best data chunk. Afterwards, we conduct experiments to
show that FedRDMA facilitates convenient and efficient integration with PEFT
methods. Finally, we show FedRDMA is more energy-efficient and environment
friendly compared to traditional communication methods.

###  4.1. Implementation and Setup

We have developed a prototype of an domain-efficient RDMA-based federated
learning communication plugin on top of the open-source industrial framework
FATE. The implementation consists of three main components. Firstly, we have
developed and open-sourced a user-friendly and high-performance RDMA
communication library named
easyPyverbs222https://github.com/Marovlo/easyPyverbs to support further
development. Secondly, we intercept communication traffic of the FATE
framework and establish RDMA-based connection for forwarding. Lastly, we
modify the RDMA transmission process to emulate and integrate with the gRPC
streaming protocol used in the FATE framework.

FedRDMA is derived from the aforementioned domain-efficient RDMA federated
learning communication plugin design, with the additional incorporation of the
§3.

As for system cost, we deconstructed the communication process from the
federated learning workflow. Employing the same set of hyperparameters used in
the aforementioned end-to-end experiment as a baseline, we conducted ten
measurements and averaged the results.

#### Dataset and model

We used the AdvertiseGen 333https://www.luge.ai/#/luge/dataDetail?id=9 dataset
to perform full-tuning on the GPT-2 model (117M parameters). AdvertiseGen is
constructed based on the correspondence between tags on product web pages and
information in the copy, representing a typical open-ended generation task.
GPT-2 is a transformer-based LLM, trained on a dataset comprising 8 million
web pages.

Device | Nums | Device Model | Main Configuration
---|---|---|---
TOR Switch | 2 | HUAWEI CouldEngine 6881-48s6cq | 10Gbps Ports*48, 100Gbps Ports*6
P4 Switch | 2 | Wedge100BF-32X | 100Gbps Ports*32,
CORE Switch | 2 | Inspur S6820-48XQ-AC | 10Gbps Ports*48, 100Gbps Ports*6
RDMA NIC | 2 | NVIDIA ConnectX-6 Dx | 100Gbps Ports*2
Standard NIC | 2 | Intel X710 for 10 GbE SFP+ | 10Gbps Ports*2
FATE Server | 2 | HREMUS 8226 |  | NVIDIA A800 80GB,
---
Intel Xeon Gold 6226R*2,
252GB DDR4 Memory
MININET | 1 | H3C UIS 3000G5 |  | Intel Xeon Gold 5318Y*2,
---
378GB DDR4 Memory,
BCM57810 10 Gigabit Ethernet*2

Table 1. Hardwares configurations. Figure 6. Physical layout and network
topology.

#### Hardware and software

We established two data centers, each hosts a FATE server with a RDMA NIC and
a standard NIC. Both data center respectively connect to two TOR switchs,
which then connect to two P4 switchs, followed by two CORE switchs. Finally,
the two CORE switches connect to a MININET server that constructs two open
vSwitch instances. The main configurations, physical layout and network
topology are shown in Table 1 and Fugure 6. On the software side, we utilize
FATE 1.11.3, FATE-LLM 1.3.0 and RDMA-CORE-37.4.

#### Hyper-parameters

We chose 4MB as the maximum size of a chunk in FedRDMA. We limit the speed of
RDMA NICs to 10Gbps to align with standard NICs and introduce a 20ms RTT to
all network traffic using MININET. To simulate WAN environment, PFC and ECN on
all these switches are either disabled or not supported.

In federated learning, we employed the FedAvg (mcmahan2017communication, )
algorithm to conduct training for five epochs to reduce variance. We
configured the batch_size as 32, set the learning_rate to 5e-6, and disable
secure aggregation.

(a) Communication time

(b) End to end time

Figure 7. Latency comparation between FedRDMA and TCP/IP.

###  4.2. End-to-end Performance

FedRDMA outperforms TCP/IP significantly in cross-silo FedLLM. As depicted in
Figure 7, we compared FedRDMA with FedLLM implemented by the FATE framework.
FedLLM runs under default configurations and communicates using gRPC
(wang1993grpc, ) based on HTTP/2, eventually derived from TCP protocol. During
the training process, approximately 9.3GB of model weights was transmitted
over WAN, constituting more than 99% of the total communication traffic in the
end-to-end training workflow. FedRDMA-E was able to reduce end-to-end
communication time by 73.9% thanks to the optimizations we made for RDMA on
WANs, combined with RDMA’s inherent efficiency and low overhead, as well as
the drawbacks of TCP itself. FedRDMA-E ultimately result in a 33.3% reduction
in overall end-to-end federated learning time. Moreover, even with the simpler
split-merge approach used in FedRDMA, communication efficiency was improved by
2.913$\times$, leading to an overall enhancement in federated learning
efficiency by 2.381$\times$.

Bandwidth (Gbps) | 1 | 2 | 3 | 4-5 | 6-9 | 10 | 100
---|---|---|---|---|---|---|---
Maximum chunk | 1GB | 1GB | 1GB | 12MB | 4MB | 4MB | 4MB
Best chunk | 1GB | 1GB | 1GB | 4MB | 4MB | 4MB | 4MB
Link-Enable | NO | NO | NO | YES | YES | YES | YES
Latency (s) | 8.16 | 4.10 | 2.77 |  $\sim$6.57 |  $\sim$6.11 | 6.00 | 5.98

Table 2. Optimal hyperparameters under different bandwidths.

###  4.3. Impact of different hyperparameters

As shown in Table 2, we apply various rate-limiting operations on RDMA NIC.
Alone wtih Figure 1, we can indicate that FedRDMA continuously outperforms TCP
a lot from 1Gbps to 10Gbps, and remains feasible and efficient at 100Gbps.
During the experimental process, when applying RDMA on WAN, we have some
interesting observations.

We observed that at higher RDMA bandwidths, notably 4Gbps or above,
transmitting large data chunks (e.g., 2MB and above) directly over the link
significantly increases the risk of transmission failures. To address this
issue, it is necessary to send a smaller data chunk first, such as near the
MTU size of the link. For example, if $X_{n}$, the last chunk of $X$, is
relatively small, it can be sent as the above mentioned small chunk, thereby
optimizing one RTT time. After this initial step, all subsequent large chunks
can be smoothly transmitted. We refer to this process as Link-Enable.

We also note that when the RDMA bandwidth is small, there is no need to split
the data or Link-Enable in our experimental environment. We speculate that the
amount of data simultaneously sent and present on the WAN is limited. This
does not exceed the carrying capacity of the WAN, resulting in fewer packet
loss, and allowing the RDMA transmission to proceed smoothly. This explanation
also accounts for another two observations in Table 2: (1) FedRDMA exhibits
almost no improvement at 100Gbps compared to 10Gbps. (2) The maximum chunk
size at 4-5Gbps can reach 12MB, but the best remains at 4MB.

Moreover, to validate our hypothesis, we use the Linux “tc” command to limit
the speed of the “10Gbps standard NIC” to 10Gbps with a token bucket filter
(TBF) that can smooth TCP traffic flows. Surprisingly, the performance of TCP
actually improved, indicating that smoothing out the traffic is beneficial for
both RDMA and TCP on WANs.

Lora Rank | 4 | 8 | 16 | 32 | 64 | 128 | 256 | 512 | 1024
---|---|---|---|---|---|---|---|---|---
Data size (MB) | 1.1 | 2.3 | 4.5 | 9.0 | 18.0 | 36.0 | 72.0 | 144.0 | 288.0
Num of chunks | 1 | 2 | 4 | 5 | 7 | 12 | 21 | 39 | 75
Link-Enable | NO | YES | YES | YES | YES | YES | YES | YES | YES

Table 3. The transmission data size, the total num of chunks, and whether
Link-Enable is needed under different Lora Rank. Figure 8. Communication time
of FedRDMA vs TCP/IP under different PEFT settings.

###  4.4. Integration with PEFT

In this section, we aim to verify the high compatibility and efficiency of
FedRDMA under different data transfer volumes introduced by varied PEFT
settings. We selected the mainstream and effective Low Rank Adaptation (LORA)
method and validated a wide range of lora_rank, the key parameter in LORA. The
lora_rank ranges from 4 to 1024, and each case underwent five tests to reduce
variance. The sizes of transmitted data, the number of data fragments, and
whether Link-Enable is required for each scenario are presented in Table 3.

As shown in Figure 8, we observed that FedRDMA reduces communication time by
over 70% in the majority of cases. Even in specific scenarios, such as when
lora_rank=16 a substantial improvement in communication efficiency of 2.37
$\times$ is achieved. Therefore, FedRDMA can complement the PEFT method well
during training, and regardless of the PEFT configuration, significant
improvements in communication efficiency can be achieved compared to using
PEFT alone.

Method | Memory | Time | Power | Energy
---|---|---|---|---
FedLLM | 13.8MB | 24.6s | 5.1W | 125.2J
FedRDMA | 60.0MB | 9.4s | 18.7W | 175.4J
FedRDMA-E | 0.025MB | 6.0s | 18.7W | 112.6J
Table 4. The system cost of transfering 1GB data.

###  4.5. System cost

After executing the deconstructed communication process ten times, the
obtained data is presented in Table 4, where “Memory” refers to the remaining
value after subtracting the inherent data volume of 1GB that needs to be
transmitted and “Time” refers to the total time cost transfering data.

As shown in Table 4, the refinement in memory design and optimization of the
data transmission process enable FedRDMA-E to achieve a 99.9% reduction in
memory overhead compared to FedRDMA, demonstrating a significant improvement
similar to that of FedLLM. Furthermore, despite the power of RDMA NIC being
much higher than standard NIC attributed to its integration of specialized
hardware for rapid data processing, the total power consumption for
transmission is nevertheless reduced by more than 10% due to the substantial
decrease in transmission time.

##  5\. Conclusion and future work

This paper addresses a crucial and challenging issue: how to leverage RDMA
technology to accelerate federated learning communication on WANs. We conduct
detailed preliminary experiments to demonstrate that current RDMA technology
fails to work effectively on WAN and the reasons behind. To address these
issues, we propose a split-reassemble-based RDMA federated learning
acceleration technique, named FedRDMA, and perform multiple optimization
efforts. We are the first to integrate RDMA into federated learning and
demonstrate its effectiveness within the industrial federated learning
framework. In the future, we will extend FedRDMA to more complex WAN
environments and pave its way for large-scale cross-silo federated learning
deployment.

## References

  * (1) Joshua Robinson, Christopher Michael Rytting, and David Wingate,  “Leveraging large language models for multiple choice question answering,”  arXiv preprint arXiv:2210.12353, 2022.
  * (2) Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi,  “From images to textual prompts: Zero-shot visual question answering with frozen large language models,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10867–10877.
  * (3) Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing,  “Sentiment analysis in the era of large language models: A reality check,”  arXiv preprint arXiv:2305.15005, 2023.
  * (4) Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang,  “Text classification via large language models,”  arXiv preprint arXiv:2305.08377, 2023.
  * (5) Youngjin Chae and Thomas Davidson,  “Large language models for text classification: From zero-shot learning to fine-tuning,”  Open Science Foundation, 2023.
  * (6) Zexian Zeng, Yu Deng, Xiaoyu Li, Tristan Naumann, and Yuan Luo,  “Natural language processing for ehr-based computational phenotyping,”  IEEE/ACM transactions on computational biology and bioinformatics, vol. 16, no. 1, pp. 139–153, 2018.
  * (7) Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, et al.,  “Llms accelerate annotation for medical information extraction,”  in Machine Learning for Health (ML4H). PMLR, 2023, pp. 82–100.
  * (8) Allen H Huang, Hui Wang, and Yi Yang,  “Finbert: A large language model for extracting information from financial text,”  Contemporary Accounting Research, vol. 40, no. 2, pp. 806–841, 2023.
  * (9) Yi Yang, Mark Christopher Siy Uy, and Allen Huang,  “Finbert: A pretrained language model for financial communications,”  arXiv preprint arXiv:2006.08097, 2020.
  * (10) Paul Voigt and Axel Von dem Bussche,  “The eu general data protection regulation (gdpr),”  A Practical Guide, 1st Ed., Cham: Springer International Publishing, vol. 10, no. 3152676, pp. 10–5555, 2017.
  * (11) Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong,  “Federated machine learning: Concept and applications,”  ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1–19, 2019.
  * (12) Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He,  “A survey on federated learning systems: Vision, hype and reality for data privacy and protection,”  IEEE Transactions on Knowledge and Data Engineering, 2021.
  * (13) Dongqi Cai, Shangguang Wang, Yaozong Wu, Felix Xiaozhu Lin, and Mengwei Xu,  “Federated few-shot learning for mobile nlp,”  in Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, 2023, pp. 1–17.
  * (14) Dongqi Cai, Yaozong Wu, Shangguang Wang, Felix Xiaozhu Lin, and Mengwei Xu,  “Efficient federated learning for modern nlp,”  in Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, 2023, pp. 1–16.
  * (15) Mengwei Xu, Yaozong Wu, Dongqi Cai, Xiang Li, and Shangguang Wang,  “Federated fine-tuning of billion-sized language models across mobile devices,”  arXiv preprint arXiv:2308.13894, 2023.
  * (16) Chaochao Chen, Xiaohua Feng, Jun Zhou, Jianwei Yin, and Xiaolin Zheng,  “Federated large language model: A position paper,”  arXiv preprint arXiv:2307.08925, 2023.
  * (17) Jujia Zhao, Wenjie Wang, Chen Xu, Zhaochun Ren, See-Kiong Ng, and Tat-Seng Chua,  “Llm-based federated recommendation,”  arXiv preprint arXiv:2402.09959, 2024.
  * (18) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.,  “Language models are unsupervised multitask learners,”  OpenAI blog, vol. 1, no. 8, pp. 9, 2019.
  * (19) Jilong Xue, Youshan Miao, Cheng Chen, Ming Wu, Lintao Zhang, and Lidong Zhou,  “Fast distributed deep learning over rdma,”  in Proceedings of the Fourteenth EuroSys Conference 2019, 2019, pp. 1–14.
  * (20) Feng Tian, Yang Zhang, Wei Ye, Cheng Jin, Ziyan Wu, and Zhi-Li Zhang,  “Accelerating distributed deep learning using multi-path rdma in data center networks,”  in Proceedings of the ACM SIGCOMM Symposium on SDN Research (SOSR), 2021, pp. 88–100.
  * (21) Chuanxiong Guo, Haitao Wu, Zhong Deng, Gaurav Soni, Jianxi Ye, Jitu Padhye, and Marina Lipshteyn,  “Rdma over commodity ethernet at scale,”  in Proceedings of the 2016 ACM SIGCOMM Conference, 2016, pp. 202–215.
  * (22) Li Li, Yuxi Fan, Mike Tse, and Kuo-Yi Lin,  “A review of applications in federated learning,”  Computers & Industrial Engineering, vol. 149, pp. 106854, 2020.
  * (23) Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao,  “A survey on federated learning,”  Knowledge-Based Systems, vol. 216, pp. 106775, 2021.
  * (24) Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger R Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu N Galtier, Bennett A Landman, Klaus Maier-Hein, et al.,  “The future of digital health with federated learning,”  NPJ digital medicine, vol. 3, no. 1, pp. 119, 2020.
  * (25) Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith,  “Federated learning: Challenges, methods, and future directions,”  IEEE signal processing magazine, vol. 37, no. 3, pp. 50–60, 2020.
  * (26) Chao Huang, Jianwei Huang, and Xin Liu,  “Cross-silo federated learning: Challenges and opportunities,” 2022.
  * (27) Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al.,  “Chatgpt for good? on opportunities and challenges of large language models for education,”  Learning and individual differences, vol. 103, pp. 102274, 2023.
  * (28) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al.,  “A survey on evaluation of large language models,”  ACM Transactions on Intelligent Systems and Technology, 2023.
  * (29) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.,  “Emergent abilities of large language models,”  arXiv preprint arXiv:2206.07682, 2022.
  * (30) Eric Gamess and Humberto Ortiz-Zuazaga,  “Evaluation of point-to-point network performance of hpc clusters at the level of udp, tcp, and mpi,”  in IV Simposio Científico y Tecnológico en Computación, Caracas, Venezuela, 2016.
  * (31) Jingang Jiang, Xiangyang Liu, and Chenyou Fan,  “Low-parameter federated learning with large language models,”  arXiv preprint arXiv:2307.13896, 2023.
  * (32) Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu,  “Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learning,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * (33) Shubham Malaviya, Manish Shukla, and Sachin Lodha,  “Reducing communication overhead in federated learning for pre-trained language models using parameter-efficient finetuning,”  in Conference on Lifelong Learning Agents. PMLR, 2023, pp. 456–469.
  * (34) Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, and Shuiguang Deng,  “Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes,”  arXiv preprint arXiv:2312.06353, 2023.
  * (35) Jiaxiang Wu, Weidong Huang, Junzhou Huang, and Tong Zhang,  “Error compensated quantized sgd and its applications to large-scale distributed optimization,”  in International Conference on Machine Learning. PMLR, 2018, pp. 5325–5333.
  * (36) Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar,  “signsgd: Compressed optimisation for non-convex problems,”  in International Conference on Machine Learning. PMLR, 2018, pp. 560–569.
  * (37) Ang Li, Jingwei Sun, Pengcheng Li, Yu Pu, Hai Li, and Yiran Chen,  “Hermes: an efficient federated learning framework for heterogeneous mobile clients,”  in Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, 2021, pp. 420–437.
  * (38) Chenning Li, Xiao Zeng, Mi Zhang, and Zhichao Cao,  “Pyramidfl: A fine-grained client selection framework for efficient federated learning,”  .
  * (39) Fan Lai, Xiangfeng Zhu, Harsha V Madhyastha, and Mosharaf Chowdhury,  “Oort: Efficient federated learning via guided participant selection.,”  2021\.
  * (40) Duowen Liu,  “Accelerating intra-party communication in vertical federated learning with rdma,”  in Proceedings of the 1st Workshop on Distributed Machine Learning, 2020, pp. 14–20.
  * (41) Don Towsley,  “The stutter go back-n arq protocol,”  IEEE transactions on Communications, vol. 27, no. 6, pp. 869–875, 1979.
  * (42) Shobana Balakrishnan and Fusun Ozguner,  “A priority-driven flow control mechanism for real-time traffic in multiprocessor networks,”  IEEE Transactions on Parallel and Distributed Systems, vol. 9, no. 7, pp. 664–678, 1998.
  * (43) Sally Floyd,  “Tcp and explicit congestion notification,”  ACM SIGCOMM Computer Communication Review, vol. 24, no. 5, pp. 8–23, 1994.
  * (44) Marek Irland,  “Buffer management in a packet switch,”  IEEE transactions on Communications, vol. 26, no. 3, pp. 328–337, 1978.
  * (45) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas,  “Communication-efficient learning of deep networks from decentralized data,”  in Artificial intelligence and statistics. PMLR, 2017, pp. 1273–1282.
  * (46) Xingwei Wang, Hong Zhao, and Jiakeng Zhu,  “Grpc: A communication cooperation mechanism in distributed systems,”  ACM SIGOPS Operating Systems Review, vol. 27, no. 3, pp. 75–86, 1993.


* [66] R. Bittleston, S. Heuveline, and D. Skinner, “The celestial chiral algebra of self-dual gravity on Eguchi-Hanson space,” JHEP 09 (2023) 008, arXiv:2305.09451 [hep-th].
  * [67] P. Drozdov and T. Kimura, “Structure of deformed w1+$\infty$ symmetry and topological generalization in Celestial CFT,” Phys. Lett. B 847 (2023) 138272, arXiv:2306.11693 [math-ph].
  * [68] A. Saha, “$w_{1+\infty}$ and Carrollian Holography,” arXiv:2308.03673 [hep-th].
  * [69] C. Ahn and M. H. Kim, “The ${\cal N}=2,4$ Supersymmetric Linear $W_{\infty}[\lambda]$ Algebras for Generic $\lambda$ Parameter,” arXiv:2309.01537 [hep-th].
  * [70] S. Banerjee, H. Kulkarni, and P. Paul, “Celestial OPE in Self Dual Gravity,” arXiv:2311.06485 [hep-th].
  * [71] T. R. Taylor and B. Zhu, “w(1+infinity) Algebra with a Cosmological Constant and the Celestial Sphere,” arXiv:2312.00876 [hep-th].
  * [72] E. Crawley, A. Guevara, E. Himwich, and A. Strominger, “Self-dual black holes in celestial holography,” JHEP 09 (2023) 109, arXiv:2302.06661 [hep-th].
  * [73] Y. Hamada and G. Shiu, “Infinite Set of Soft Theorems in Gauge-Gravity Theories as Ward-Takahashi Identities,” Phys. Rev. Lett. 120 no. 20, (2018) 201601, arXiv:1801.05528 [hep-th].
  * [74] Z.-Z. Li, H.-H. Lin, and S.-Q. Zhang, “Infinite Soft Theorems from Gauge Symmetry,” Phys. Rev. D 98 no. 4, (2018) 045004, arXiv:1802.03148 [hep-th].
  * [75] S. Pasterski, A. Puhm, and E. Trevisani, “Revisiting the conformally soft sector with celestial diamonds,” JHEP 11 (2021) 143, arXiv:2105.09792 [hep-th].
  * [76] S. Pasterski, A. Puhm, and E. Trevisani, “Celestial diamonds: conformal multiplets in celestial CFT,” JHEP 11 (2021) 072, arXiv:2105.03516 [hep-th].
  * [77] S. Stieberger and T. R. Taylor, “Symmetries of Celestial Amplitudes,” Phys. Lett. B 793 (2019) 141–143, arXiv:1812.01080 [hep-th].
  * [78] Y. T. A. Law and M. Zlotnikov, “Poincaré constraints on celestial amplitudes,” JHEP 03 (2020) 085, arXiv:1910.04356 [hep-th]. [Erratum: JHEP 04, 202 (2020)].
  * [79] Y. T. A. Law and M. Zlotnikov, “Massive Spinning Bosons on the Celestial Sphere,” JHEP 06 (2020) 079, arXiv:2004.04309 [hep-th].
  * [80] L. Iacobacci and W. Mück, “Conformal Primary Basis for Dirac Spinors,” Phys. Rev. D 102 no. 10, (2020) 106025, arXiv:2009.02938 [hep-th].
  * [81] S. A. Narayanan, “Massive Celestial Fermions,” JHEP 12 (2020) 074, arXiv:2009.03883 [hep-th].
  * [82] F. Cachazo and A. Strominger, “Evidence for a New Soft Graviton Theorem,” arXiv:1404.4091 [hep-th].
  * [83] H. Elvang, C. R. T. Jones, and S. G. Naculich, “Soft Photon and Graviton Theorems in Effective Field Theory,” Phys. Rev. Lett. 118 no. 23, (2017) 231601, arXiv:1611.07534 [hep-th].
  * [84] W. Fan, A. Fotopoulos, and T. R. Taylor, “Soft Limits of Yang-Mills Amplitudes and Conformal Correlators,” JHEP 05 (2019) 121, arXiv:1903.01676 [hep-th].
  * [85] M. Pate, A.-M. Raclariu, and A. Strominger, “Conformally Soft Theorem in Gauge Theory,” Phys. Rev. D 100 no. 8, (2019) 085017, arXiv:1904.10831 [hep-th].
  * [86] T. Adamo, L. Mason, and A. Sharma, “Celestial amplitudes and conformal soft theorems,” Class. Quant. Grav. 36 no. 20, (2019) 205018, arXiv:1905.09224 [hep-th].
  * [87] A. Puhm, “Conformally Soft Theorem in Gravity,” JHEP 09 (2020) 130, arXiv:1905.09799 [hep-th].
  * [88] A. Guevara, “Notes on Conformal Soft Theorems and Recursion Relations in Gravity,” arXiv:1906.07810 [hep-th].
  * [89] Y. F. Bautista and A. Guevara, “From Scattering Amplitudes to Classical Physics: Universality, Double Copy and Soft Theorems,” arXiv:1903.12419 [hep-th].
  * [90] S. Banerjee, P. Pandey, and P. Paul, “Conformal properties of soft operators: Use of null states,” Phys. Rev. D 101 no. 10, (2020) 106014, arXiv:1902.02309 [hep-th].
  * [91] S. Banerjee and P. Pandey, “Conformal properties of soft-operators. Part II. Use of null-states,” JHEP 02 (2020) 067, arXiv:1906.01650 [hep-th].
  * [92] S. Banerjee, S. Ghosh, and R. Gonzo, “BMS symmetry of celestial OPE,” JHEP 04 (2020) 130, arXiv:2002.00975 [hep-th].
  * [93] S. Banerjee, S. Ghosh, and P. Paul, “MHV Graviton Scattering Amplitudes and Current Algebra on the Celestial Sphere,” arXiv:2008.04330 [hep-th].
  * [94] S. Banerjee, S. Ghosh, and S. Satyam Samal, “Subsubleading soft graviton symmetry and MHV graviton scattering amplitudes,” arXiv:2104.02546 [hep-th].
  * [95] S. Banerjee, S. Ghosh, and P. Paul, “(Chiral) Virasoro invariance of the tree-level MHV graviton scattering amplitudes,” arXiv:2108.04262 [hep-th].
  * [96] S. Banerjee and S. Ghosh, “MHV Gluon Scattering Amplitudes from Celestial Current Algebras,” arXiv:2011.00017 [hep-th].
  * [97] D. Simmons-Duffin, “Projectors, Shadows, and Conformal Blocks,” JHEP 04 (2014) 146, arXiv:1204.3894 [hep-th].
  * [98] W. W. MacDowell and R. Roskies, “Reduction of the Poincaré Group with Respect to the Lorentz Group,” Journal of Mathematical Physics 13 no. 10, (Oct., 1972) 1585–1592.