# Robustness of Multi-Source MT to Transcription Errors

Dominik Macháček1,2 Peter Polák1 Ondřej Bojar1 Raj Dabre2

Charles University, Faculty of Mathematics and Physics,
Institute of Formal and Applied Linguistics1

National Institute of Information and Communications Technology, Kyoto, Japan2
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Automatic speech translation is sensitive to speech recognition errors, but in
a multilingual scenario, the same content may be available in various
languages via simultaneous interpreting, dubbing or subtitling. In this paper,
we hypothesize that leveraging multiple sources will improve translation
quality if the sources complement one another in terms of correct information
they contain. To this end, we first show that on a 10-hour ESIC corpus, the
ASR errors in the original English speech and its simultaneous interpreting
into German and Czech are mutually independent. We then use two sources,
English and German, in a multi-source setting for translation into Czech to
establish its robustness to ASR errors. Furthermore, we observe this
robustness when translating both noisy sources together in a simultaneous
translation setting. Our results show that multi-source neural machine
translation has the potential to be useful in a real-time simultaneous
translation setting, thereby motivating further investigation in this area.

##  1 Introduction

Speech translation (ST) suffers from automatic speech recognition (ASR)
errors, especially in challenging conditions such as non-native language
speakers, background noise, named entities and specialized vocabulary usage
Macháček et al. (2019); Gaido et al. (2021, 2022); Anastasopoulos et al.
(2022). ASR errors negatively impact translation quality, via the compounding
of speech recognition and translation errors Ruiz et al. (2017); Sperber and
Paulik (2020), thereby limiting the application of automatic speech
translation in realistic settings. Fortunately, there are multilingual
settings where a source is simultaneously or consecutively interpreted into
multiple languages. Many documents are also dubbed or subtitled in offline
mode. This simultaneous interpretation takes place either via human
interpreters, in the form of dubbing or subtitling. In a situation where the
same sentence is available in multiple languages, multi-source machine
translation (MT) significantly improves translation quality, especially when
the two sources used separately do not yield high quality translations Dabre
et al. (2018); Zoph and Knight (2016); Nishimura et al. (2018).

Although not yet clearly verified, multi-source MT could be useful in settings
where the sources complement each other. In other words, challenges in
translation posed by using each source should be independent of one another.
Given that ASR is noisy and that multiple sources can help overcome
limitations of individual sources, this paper asks the following question:
“Can multi-source MT be leveraged for speech translation in a multilingual
setting where an original source transcription and its simultaneously
interpreted transcription are available?” We decompose this question into
three parts where we hypothesize that (a) ASR errors in the original and
interpreted transcripts are independent, which makes them complementary, (b)
multi-source MT is robust to transcription errors present in individual
sources, and (c) the robustness of multi-source MT continues to hold in a
simultaneous translation setting. We address each question in Sections 3, 4,
and 5, respectively.

To prove our hypotheses, firstly, we verify on the Europarl Simultaneous
Interpreting Corpus (ESIC, Macháček et al., 2021) that original speech ASR and
interpreted speech ASR are indeed complementary in terms of errors. Secondly,
we simulate transcription errors in a full sentence multi-source MT setting
for English and German to Czech translation. We clearly show that when both
sources are noisy, using them together leads to significant improvements when
compared to using them individually, in contrast to drops in quality when at
least one of the sources is clean. For example, on ESIC test set with 15% WER
noise in English source and 10% WER noise in German source, multi-sourcing
performs 0.9 BLEU score higher than English source. Finally, we use both
sources in a simultaneous translation setting and show that multi-source MT
continues to be robust to transcription errors.

Our findings show that multi-source MT has strong potential in a simultaneous
translation setting where multiple sources are available via ASR or
interpreted ASR. We note that our current analysis is limited to the case
where the multiple sources are aligned and hence available at the same time.
This is a setting of, e.g., dubbed and subtitled videos where we would want to
consider additional target languages. In simultaneous settings where one
source is available with a delay, the synchronization of the sources would be
a considerable problem, which we leave for future research.

##  2 Related Work

This paper mainly focuses on ASR errors, multilingual multi-source translation
and simultaneous translation.

#### ASR errors

often propagate to MT in cascaded ST systems and in a real time translation
setting where ASR systems are used. It is a major issue that affects
translation quality. Martucci et al. (2021) propose a method to tune the MT on
the training data with artificial noise that mimics ASR errors, via a unigram
“lexical noise model” learned on automatic–gold transcript pairs. Other
authors propose similar methods for training Sperber et al. (2017); Di Gangi
et al. (2019); Xue et al. (2020); Serai et al. (2022). However, in this work,
rather than mimicking ASR noise during training, we complement a noisy source
with another whose ASR errors are provably independent of the first.
Specifically, we use the lexical noising of Martucci et al. (2021) to simulate
noise in multiple source languages and show the robustness of MT models,
especially in a multi-source setting.

#### Multilingualism

Dabre et al. (2020) has been shown to improve translation quality in a variety
of situations. In particular, multi-source machine translation has high
potential for improving translation quality, but has been relatively
underexplored. In the context of multi-source text-to-text translation, Zoph
and Knight (2016) and Dabre et al. (2018) showed that leveraging the same
sentence in different languages improves translation quality as the two
sources are expected to complement hard to translate phenomena in the other
source. Although this approach requires multi-parallel sentence-aligned data,
the missing sources can be obtained by MT Nishimura et al. (2018) or via
simultaneous interpretation. Rather than training a multi-source model, Firat
et al. (2016) propose the “late averaging” which needs multilingual models
trained on pairwise bilingual data, which we also focus on when evaluating
multi-source models. Late averaging is akin to ensembling via logits
averaging, but with source sentences in different languages. These works do
not consider transcription errors which are ubiquitous in speech translation,
an aspect this paper focuses on.

#### Multi-sourcing

in simultaneous translation settings has not been extensively explored. Dabre
et al. (2021) have explored simultaneous multi-pivot translation where a
source is translated into a target language via multiple pivot languages,
where the pivot languages are translated using multi-source translation.
Unlike them, we consider only one pivot language which is interpreted from the
source and then use it together with the source to show that the translation
quality into the target language improves. Additionally, they do not consider
the effect of transcription noise on translation, which we do. Simultaneous
translation approaches such as wait-$k$ Ma et al. (2019) and Local Agreement
(LA-$n$, Polák et al., 2022) are commonly used, and we use the latter for our
experiment.

An alternative multi-sourcing approach is to select and use only one source.
Macháček et al. (2021) provide an analysis of using either the original, or
its simultaneously interpreted equivalent as a source for simultaneous ST.
Interpreting is delayed, but shorter and simpler than translationese.
Interpreters also segment their speech to sentences differently than the
original speakers, so it is not easy to align segments. In any case, selecting
sources will involve additional effort and thus we consider using multiple
sources together to be a more effective approach. In this regard, multiple
language sources, both as text and speech streams, could be used in ASR Paulik
et al. (2005); Soky2022LeveragingST as well as in pre-neural MT and ST Och and
Ney (2001); Paulik and Waibel (2008); Khadivi and Ney (2008). Miranda et al.
(2013) use them for punctuation restoration. Kocmi et al. (2021) provide a
broad analysis of benefits of multilingual MT.

##  3 Parallel Source-Interpreted ASRs are Independent

We assume a multi-source setting with the original speech and its
simultaneously interpreted equivalent as the two sources will improve
robustness to ASR errors if the errors in the two source streams complement
each other. This is not obvious because, on the one hand, the ASRs work
independently where they are deployed for different languages, trained on
different data, and the processing is fully independent. On the other hand,
the content of the speeches is identical. Interpreters’ speech pacing also
depends on the original speaker, and it may influence the quality of both ASRs
the same way. Therefore, in this section, we analyze the dependency of ASR
errors in the source and interpreter, on 10-hour ESIC corpus Macháček et al.
(2021) to prove that the ASR errors are indeed independent.

#### Methodology

First, we processed ASR for English original speakers and interpreters into
Czech and German. For English, we used the low-latency neural ASR by Nguyen et
al. (2021). For German, we used an older hybrid HMM-DNN model trained using
the Janus Recognition Toolkit, which features a single-pass decoder Cho et al.
(2013). For Czech, we used Kaldi Povey et al. (2011) HMM-DNN model trained on
Czech Parliament data Kratochvíl et al. (2020). Table 1 summarizes the
transcription quality on ESIC showing that the quality is low, but to the best
of our knowledge it is the best one available for this domain.

subset | Cs interp. | De interp. | En original
---|---|---|---
dev | 14.84 | 25.14 | 13.63
test | 14.04 | 23.79 | 14.71

Table 1: Transcription WER on ESIC. There are 191 and 179 documents in dev and
test subsets. The scores are weighted by number of words in gold transcripts.

We then re-used the word alignments of gold transcripts between the original
and interpretation as described in Macháček et al. (2021). 38% of tokens were
aligned between English and Czech interpretations, and 40% between English and
German, see Table 2. It may be caused by the characteristics of the language
pair (e.g. compound words in German vs multi-word expressions in English),
features of interpreting (non-verbatim translation, shortening) and by errors
in automatic alignment. We only analyzed the aligned tokens further. Since
there are many tokens left in two 5-hour subsets of the corpus, we consider
further analysis as valid.

| En tokens | En-Cs aligned | En-De aligned
---|---|---|---
dev | 44,494 | 16,962 | (38.12%) | 17,809 | (40.03%)
test | 46,151 | 17,623 | (38.19%) | 19,280 | (41.78%)

Table 2: Number and percentage of aligned tokens in gold transcripts between
the original source (English [En]) and its interpretations (German [De] and
Czech [Cs]).

Finally, we aligned gold and automatic transcripts using Levenshtein edit
distance.111https://pypi.org/project/edlib/ We classified each token in the
ASR transcript as transcribed correctly or not, both for source and
interpretations.

#### Results

We made a contingency table (Table 3) and ran a $\chi^{2}$ test Pearson (1900)
of statistical independence. The results show that the parallel source and
interpretation ASRs make errors independently of each other with $p<0.01$, for
both pairs, English-Czech and English-German, for both dev and test subsets.

En orig. | Cs int. | De int.
---|---|---
corr. | incorr. | corr. | incorr.
dev | corr. | 13815 | 1497 | 7192 | 1561
incorr. | 1228 | 422 | 633 | 307
test | corr. | 14204 | 1655 | 7895 | 1638
incorr. | 1344 | 420 | 692 | 336
Table 3: Contingency table of correctly and incorrectly recognized aligned
tokens in English source (in rows) and interpretation into Czech and German
(in columns), in dev and test subset of ESIC corpus. According to the
$\chi^{2}$ test of statistical independence, in all 4 cases, the parallel
recognition is independent with $p<0.01$.

We manually assessed the severity of the ASR errors and realized that most
errors are only in spelling and fluency, and not in adequacy. We therefore
conclude that our finding of independence of parallel ASRs may be valid only
for ASRs of comparable quality to ours.

##  4 Multi-Source Speech Translation

Having established that ASR errors are independent, we now analyze whether
multi-source neural machine translation (NMT) is robust to noisy sources. We
focus on NMT for individual sentences, with gold sentence alignment of the
sources and reference. It is a less realistic use-case than translating long
speech documents without any sentence segmentation and alignment of the
sources, but proving the robustness of multi-sourcing in this setting paves
the way for its application in long speech document translation.

#### Data

For training, we use data from OPUS Tiedemann and Nygaard (2004), aiming at a
multi-way model with English and German on the source side and Czech as a
target. We download all the data from OPUS, remove all sentences from IWSLT,
WMT, ESIC and other test sets, filter them by language identification, and
then process with dual cross-entropy scoring Junczys-Dowmunt (2018) using the
bilingual NMT models from Tiedemann and Thottingal (2020). We select the top
30 million sentences for each language pair as training data, to prevent
overfitting for either. It is also near the threshold that Chen et al. (2021)
showed as optimal.

For NMT validation and evaluation, we use the “revised transcript and
translations” from ESIC Macháček et al. (2021). These are the texts that were
originally uttered in the European Parliament, transcribed, revised and
normalized for reading and publication on the website, and then translated.
They are analogous, but not identical, to the gold transcripts of the original
and interpretations that we used in Section 3. In addition to the version
published by Macháček et al. (2021), we properly align the sentences in all
the three languages. Two documents were removed because they missed German
translation. The corpus is of comparable size to a usual MT test set. See size
statistics in Table 4.

| sent. | doc. | En words | De w. | Cs w.
---|---|---|---|---|---
dev | 2002 | 179 | 44866 | 43323 | 38347
test | 1963 | 189 | 44273 | 42491 | 37695
Table 4: Size statistics of tri-parallel sentence-aligned “revised
translations” of ESIC Macháček et al. (2021). English is original, German and
Czech are translations.

For a contrastive evaluation, we use Newstest11 Callison-Burch et al. (2011).
It contains 3003 sentences in 5 languages: English, German, Czech, French and
Spanish, the same amount in each. Newstest11 has references that were
translated directly, not through an intermediate language. We also use three
additional Czech references of Newstest11 that were translated from German
Bojar et al. (2012).

#### Multi-Sourcing

We convert Marian models to PyTorch to be used with the Hugging Face
Transformers Wolf et al. (2020) library, in which we implement late and early
averaging. For both single- and multi-sourcing, we use greedy decoding because
beam search support is not implemented with multi-source.

#### Training details

We train a multi-way NMT model using Marian Junczys-Dowmunt et al. (2018) with
English and German as sources, with language identification tokens, and Czech
as the target. We use two separate SentencePiece Kudo and Richardson (2018)
vocabularies, both sizes of 16 000. The source vocabulary is joint for German
and English, and the target is only for Czech. The model is a Transformer Base
(6 layers, 512 embedding size, 8 self-attention heads, 2048 filter size)
trained on 8 Quadro P5000 GPUs with 16 GB memory for 17 days, until
convergence.

#### Checkpoint selection

We validate all checkpoints (every 1000 training steps, 15 minutes) on two
single sources (English and German) and two multi-sourcing options: early
averaging, and late averaging of a single checkpoint with two sources.
Furthermore, after the training has ended, we selected top 10 checkpoints that
reached the highest BLEU scores for English and German single-source on the
ESIC dev set. We evaluated all pairs of the top performing checkpoints in late
averaging multi-sourcing setup. The top performing model from all validation
and grid search options was selected as a final model. It is late averaging
with a pair of distinct checkpoints. We also use these two checkpoints for
single source evaluation.

#### Evaluation Metrics

We estimate translation quality by BLEU Papineni et al. (2002) and chrF2
Popović (2016) calculated by sacreBLEU222Metric signatures:
BLEU—nrefs:1—case:mixed—eff:no—tok:13a—smooth:exp— version:2.2.1,
chrF2—nrefs:1—case:mixed—eff:yes—nc:6—nw:0—space:no—version:2.2.1 Post (2018).
We also report the current state-of-the-art metric COMET333wmt20-comet-da
model Rei et al. (2020) that achieves the highest correlation with direct
assessment as a kind of human judgements Mathur et al. (2020). However, COMET
requires one source on the input and is not suitable for multi-source.
Therefore, we report it twice (En/De COMET) with two single sources. Note that
En COMET scores assume English as source and Czech as target. Since ESIC is
tri-parallel, even if the translation is obtained using German or English and
German multi-source, we only use the English source as the input to the COMET
model. De COMET scores are computed similarly.

#### Results with clean inputs

Table 5 shows the results of multi-sourcing with clean inputs, without any
speech recognition noise. One would be tempted to conclude that the
translation from English is of a higher quality than the translation from
German (e.g. 33 vs. 26 BLEU on ESIC dev set), but such a claim is risky. The
metrics measure the match of the candidate translation with the reference
sentence (and, in case of COMET, also with the source), and it is conceivable
that the English served as the source for the human reference translation. The
Czech reference thus may very well exhibit more traits of the English source
than of the German source. While the chrF2 scores agree with BLEU, COMET
scores seem to indicate that multi-sourcing is as good as, if not better than,
using a single source. Since COMET is known to correlate with human judgements
better than BLEU Mathur et al. (2020) our results show that multi-sourcing is
indeed a viable solution.

To further shed light on the impact of the source used for creating
references, we evaluated the models with Newstest11 and computed the scores
with three additional references that were translated only from German. The
German single source achieves much higher BLEU than the English source (32.23
vs 16.62 BLEU), with multi-sourcing in between (22.47 BLEU). Similar trends
are observed in chrF2 and COMET scores. This is the opposite of ESIC scores,
where the reference was obtained from English. It shows that the traits of the
source language such as word order, structure of clauses and terms are
remarkable in automatic metrics when the reference is constructed from that
source, but these effects may be negligible in human evaluation. Appendix C
contains more details.

Finally, we consider a “balanced” scenario where an equal number of references
comes from both sources and this shows similar scores for both single sources
(23.40 vs 22.85 BLEU) with multi-sourcing outperforming them by 0.6 and 1.1
BLEU. We therefore conclude that our multi-source model should be well-
prepared for content originating in any of the source languages, but the
automatic evaluation metrics may not always capture this. Moving forward, we
only use BLEU for simplicity.

Set | Metric | Model
---|---|---
ref. translation: | En | De | De+En
ESIC dev En$\rightarrow$Cs | BLEU |  ∗33.31 | 26.13 |  ∗31.90
chrF2 |  ∗60.17 | 54.00 |  ∗58.59
En COMET |  ×0.920 | 0.860 |  ∗0.919
De COMET |  ×1.007 | 0.994 |  ∗1.022
ESIC test En$\rightarrow$Cs | BLEU |  ∗33.63 | 27.99 |  ∗32.57
chrF2 |  ∗59.58 | 54.75 |  ∗58.63
En COMET |  ∗0.906 | 0.871 |  ×0.912
De COMET | 0.994 |  ×1.006 |  ∗1.018
news11 3$\times\\{$De$\rightarrow$Cs$\\}$ | BLEU | $\underset{\pm\text{0.29}}{\text{16.62}}$ | $\underset{\pm\text{0.53}}{\textbf{32.23}}$ | $\underset{\pm\text{0.44}}{\text{22.47}}$
chrF2 | $\underset{\pm\text{0.18}}{\text{44.84}}$ | $\underset{\pm\text{0.38}}{\textbf{58.81}}$ | $\underset{\pm\text{0.27}}{\text{49.72}}$
En COMET | $\underset{\pm\text{0.002}}{\text{0.528}}$ | $\underset{\pm\text{0.002}}{\textbf{0.823}}$ | $\underset{\pm\text{0.003}}{\text{0.652}}$
De COMET | $\underset{\pm\text{0.002}}{\text{0.600}}$ | $\underset{\pm\text{0.001}}{\textbf{0.967}}$ | $\underset{\pm\text{0.003}}{\text{0.757}}$
news11 $\\{$De,En,Fr,Es$\\}$$\rightarrow$Cs, Cs | BLEU |  ∗23.40 | 22.85 |  ∗23.96
chrF2 |  ×51.00 | 50.27 |  ∗50.83
En COMET | 0.627 |  ∗0.674 |  ∗0.659
De COMET | 0.700 |  ∗0.832 |  ∗0.766
Table 5: Evaluation scores with clean inputs (no ASR noise), machine-
translated into Czech with single-sourcing English (En) or German (De), or
multi-sourcing (De+En), on ESIC and Newstest11 (news11). Newstest is evaluated
on a balanced reference that has origin in 5 languages
($\\{$De,En,Fr,Es$\\}$$\rightarrow$Cs translations and Cs original; 600
sentences each), and 3-times with additional references that were translated
from German (“3$\times\\{$De$\rightarrow$Cs$\\}$”). We report avg$\pm$stddev
for them. “En COMET” and “De COMET” are run with English and German source,
respectively. Maximum scores are in bold. The symbol ∗ means that there is
statistically significant difference $(p<0.05)$ from all the lower scores in
the same row, × means no significance ($t$-test for COMET, paired bootstrap
resampling for BLEU and chrF2).

BLEU | ESIC dev | En WER |
---|---|---|---
|  | single-src. | 0 % | 5 % | 10 % | 15 % | 20 % | 25 % | 30 % | 35 % | 40 % |
| s-src. |  | 33.3$\pm$0.0 | 29.7$\pm$0.3 | 26.3$\pm$0.4 | 22.9$\pm$0.4 | 20.4$\pm$0.5 | 18.2$\pm$0.8 | 15.8$\pm$0.1 | 14.0$\pm$0.2 | 12.1$\pm$0.1 |
De WER | 0 % | 26.1$\pm$0.0 | 31.9$\pm$0.0 |  | 30.0$\pm$0.2
---
| 28.5$\pm$0.3
---
| 26.6$\pm$0.1
---
25.2$\pm$0.4 | 23.8$\pm$0.3 | 21.9$\pm$0.3 | 20.5$\pm$0.2 | 19.3$\pm$0.3 |
5 % | 23.5$\pm$0.0 | 30.9$\pm$0.1 | 29.1$\pm$0.2 |  | 27.6$\pm$0.3
---
| 25.7$\pm$0.1
---
| 24.2$\pm$0.4
---
22.8$\pm$0.4 | 21.1$\pm$0.4 | 19.6$\pm$0.2 | 18.6$\pm$0.2 |
10 % | 21.6$\pm$0.2 | 30.0$\pm$0.2 | 28.0$\pm$0.1 |  | 26.6$\pm$0.4
---
| 24.6$\pm$0.3
---
| 23.4$\pm$0.2
---
| 21.9$\pm$0.4
---
20.2$\pm$0.1 | 18.7$\pm$0.2 | 17.5$\pm$0.5 |
15 % | 19.0$\pm$0.3 | 28.9$\pm$0.2 | 27.1$\pm$0.1 | 25.7$\pm$0.4 |  | 23.7$\pm$0.2
---
| 22.4$\pm$0.4
---
| 21.0$\pm$0.4
---
| 19.3$\pm$0.2
---
17.8$\pm$0.3 | 16.7$\pm$0.4 |
20 % | 17.1$\pm$0.3 | 27.9$\pm$0.4 | 26.6$\pm$0.2 | 24.9$\pm$0.4 |  | 22.9$\pm$0.1
---
| 21.7$\pm$0.5
---
| 20.0$\pm$0.4
---
| 18.3$\pm$0.2
---
17.0$\pm$0.1 | 15.7$\pm$0.1 |
25 % | 15.6$\pm$0.3 | 27.1$\pm$0.3 | 25.7$\pm$0.2 | 24.1$\pm$0.3 | 22.1$\pm$0.2 |  | 20.7$\pm$0.4
---
| 19.2$\pm$0.5
---
| 17.4$\pm$0.2
---
| 16.3$\pm$0.2
---
14.9$\pm$0.1 |
30 % | 13.8$\pm$0.2 | 25.9$\pm$0.3 | 24.5$\pm$0.4 | 22.8$\pm$0.3 | 20.9$\pm$0.3 | 19.6$\pm$0.2 |  | 18.3$\pm$0.2
---
| 16.3$\pm$0.4
---
| 15.1$\pm$0.1
---
| 13.9$\pm$0.2
---
35 % | 12.5$\pm$0.2 | 24.6$\pm$0.4 | 22.5$\pm$0.4 | 20.9$\pm$0.2 | 19.2$\pm$0.1 | 18.1$\pm$0.5 | 16.7$\pm$0.3 | 15.3$\pm$0.3 |  | 14.1$\pm$0.2
---
| 12.9$\pm$0.1
---
40 % | 10.8$\pm$0.1 | 23.4$\pm$0.4 | 21.4$\pm$0.1 | 20.1$\pm$0.3 | 18.3$\pm$0.5 | 17.3$\pm$0.2 | 16.0$\pm$0.1 | 14.4$\pm$0.1 | 13.2$\pm$0.2 |  | 12.1$\pm$0.1
---

BLEU news11 En WER single-src. 0 % 5 % 10 % 15 % 20 % 25 % 30 % 35 % 40 %
s-src. 23.4$\pm$0.0 21.1$\pm$0.2 19.2$\pm$0.1 17.1$\pm$0.0 15.3$\pm$0.1
13.6$\pm$0.2 12.2$\pm$0.2 10.6$\pm$0.3 9.6$\pm$0.0 De WER 0 % 22.9$\pm$0.0
24.0$\pm$0.0 22.7$\pm$0.0 21.4$\pm$0.2 20.2$\pm$0.1 18.9$\pm$0.2 17.8$\pm$0.1
16.9$\pm$0.1 15.5$\pm$0.1 14.6$\pm$0.2 5 % 20.6$\pm$0.1 23.2$\pm$0.1
21.8$\pm$0.1 20.7$\pm$0.0 19.2$\pm$0.0 18.2$\pm$0.1 17.1$\pm$0.1 16.1$\pm$0.1
14.8$\pm$0.0 13.9$\pm$0.1 10 % 18.8$\pm$0.1 22.5$\pm$0.1 21.3$\pm$0.2
20.1$\pm$0.1 18.6$\pm$0.2 17.7$\pm$0.1 16.4$\pm$0.1 15.5$\pm$0.1 14.1$\pm$0.1
13.2$\pm$0.2 15 % 17.0$\pm$0.3 21.6$\pm$0.2 20.3$\pm$0.2 19.1$\pm$0.2
17.8$\pm$0.1 16.9$\pm$0.1 15.6$\pm$0.0 14.7$\pm$0.1 13.4$\pm$0.1 12.5$\pm$0.1
20 % 15.4$\pm$0.2 20.8$\pm$0.0 19.5$\pm$0.1 18.3$\pm$0.1 17.0$\pm$0.2
16.0$\pm$0.1 14.9$\pm$0.1 14.0$\pm$0.1 12.7$\pm$0.2 12.0$\pm$0.1 25 %
13.8$\pm$0.1 19.9$\pm$0.2 18.7$\pm$0.2 17.7$\pm$0.1 16.3$\pm$0.1 15.4$\pm$0.0
14.0$\pm$0.2 13.2$\pm$0.1 11.9$\pm$0.0 11.1$\pm$0.1 30 % 12.3$\pm$0.3
19.2$\pm$0.3 17.9$\pm$0.2 16.9$\pm$0.3 15.6$\pm$0.1 14.5$\pm$0.2 13.5$\pm$0.3
12.7$\pm$0.2 11.3$\pm$0.1 10.6$\pm$0.1 35 % 11.2$\pm$0.1 18.4$\pm$0.0
17.1$\pm$0.1 16.1$\pm$0.1 15.0$\pm$0.2 13.8$\pm$0.1 12.7$\pm$0.2 11.7$\pm$0.1
10.6$\pm$0.2 9.9$\pm$0.2 40 % 9.9$\pm$0.3 17.1$\pm$0.0 16.1$\pm$0.2
14.9$\pm$0.2 14.0$\pm$0.1 12.9$\pm$0.1 11.7$\pm$0.2 10.7$\pm$0.1 9.9$\pm$0.1
9.1$\pm$0.2

Table 6:  BLEU (avg$\pm$stddev) with transcription noise on ESIC dev set whose
reference translations was English and on Newstest11 with balanced reference
source language. Green-backgrounded area is where the English single-source
outperforms German single-source.  Black underlined
---
numbers indicate the area where multi-sourcing achieves higher score than both
single-sourcing options. In bold is near maximum gap from single-source, more
than 2.1 BLEU. Red-colored numbers are where at least one single-source scores
higher.

WER | En | De | En+De
---|---|---|---
15% En, 10% De | 23.58$\pm$0.16 | 23.23$\pm$0.05 | 26.50$\pm$0.27

Table 7: ESIC test multi-sourcing vs single-sourcing BLEU scores on the
artificial WER noise level where multi-sourcing achieved the largest
improvement.

###  4.1 Modeling Transcription Noise

Although multi-sourcing English and German is not very beneficial when both
sources are clean, we hypothesize that it could show benefits with noisy
sources. Averaging two noisy sources can lead to cancelling the noise. Since
ESIC contains tri-parallel sentence-aligned translations as texts and not
speech, and since we want to evaluate different levels of ASR noise, and we do
not have many ASRs, we generate the ASR errors artificially.

#### Custom WER noise model

We adopt the lexical noise model by Martucci et al. (2021) and modify it to
create outputs with arbitrary WER. The lexical noise model modifies the source
by applying insertion, deletion, substitution, or copy operations on each word
with probabilities $p_{I},p_{D},$ and $p_{S}$, respectively. The probabilities
are learned from the ASR and gold transcript pairs. It thus may learn to
shuffle homonyms such as “eight” and “ate”.

In the original lexical noise model by Martucci et al. (2021), the target WER
is bound to the performance of the given ASR system on which it is trained,
and can not be changed. WER is defined as the number of incorrect words in the
ASR transcript divided by the number of correct words in the gold transcript.
The errors are either insertions, deletions, or substitutions. In the lexical
noise model, insertion is applied independently on the other operations.
Therefore, we can decompose WER to the sum of insertion rate and the rate of
deletions or substitutions.

In the lexical noise model, the insertion rate equals to the expected number
of insertions for each gold word. Since the probability of not inserting is
$1-p_{I}$, the expected number of repetitions before not inserting succeeds is
$\frac{p_{I}}{1-p_{I}}$. It is also a mean of a geometric distribution with
$p=1-p_{I}$.

The rate of deletions and substitutions is $p_{D}+(1-p_{D})p_{S}$, where
$p_{D}$ is the number of deletions. The words that were not deleted can be
substituted, and there is $(1-p_{D})p_{S}$ of them. In summary, the original
model WER is

$\text{WER}=\frac{p_{I}}{1-p_{I}}+p_{D}+(1-p_{D})p_{S},$ (1)

To get a custom target WER, we rescale the learned probabilities by a constant
$c$:

$\text{WER}_{\text{desired}}=\frac{cp_{I}}{1-cp_{I}}+cp_{D}+(1-cp_{D})cp_{S}.$
(2)

We simplify the equation above to

$\text{WER}_{\text{desired}}\approx cp_{I}+cp_{D}+(1-cp_{D})cp_{S}.$ (3)

It leads to a quadratic function where $c$ can be found easily. Since we work
with probabilities, we select the smallest non-negative root as the solution.
We release our implementation online.444https://github.com/pe-trik/asr-errors-
simulator

#### Training the noise model

For training the noise model, we utilize VoxPopuli Wang et al. (2021) to
retrieve around 100,000 audio and gold transcript sentences in English and
60,000 in German. They are from the same domain as ESIC, both corpora are from
the European Parliament. We processed the audio with NVidia NeMo CTC ASRs555
stt_de_quartznet15x5 and stt_en_conformer_ctc_large from
https://catalog.ngc.nvidia.com/models Kuchaiev et al. (2019); Gulati et al.
(2020). Then we trained the rules of the lexical noise model and applied them
on source data. Since the result is deterministic on the random seed of the
lexical noise model, we perform multi-sourcing using three different seeds and
report average BLEU scores with standard deviation.

#### Results with transcription noise

Section 4 summarizes the BLEU scores of two-source MT with different levels of
transcription noise in each of the sources on two sets: ESIC dev with
reference translated from English, and Newstest11 with balanced reference.
Appendix A contains the corresponding chrF2 scores. Table 7 shows the results
on the ESIC test set for the settings where multi-source models achieved the
highest improvement due to noisy inputs.

In Table 4, on both sets, we observe that the less noisy single source
achieves higher BLEU than the other single source. When the difference in
noise levels between the sources is small (close to diagonal in the table),
then multi-sourcing reaches slightly higher BLEU than single sources. In case
of balanced Newstest11, this area matches the diagonal. In case of ESIC with
English original source and reference translated from English, the area of
multi-source outperforming single-source is shifted. This tendency is
reflected in the test set results in Table 7 as well. Only when the German
source is less noisy than the English one, it does improve BLEU in multi-
sourcing. We explain it by the discrepancy of source languages for MT and
reference that affect BLEU the same way as in offline mode in Section 4. On
Newstest11, with the references translated from German, we expect the reverse.

We also observe expected behavior that the more noise, the lower BLEU in all
setups. Compare e.g. 33.3 BLEU with zero noise and 12.1 with 40% WER in both
sources. With very large noise, it is possible that neither option would be
usable. In ESIC dev, e.g. when English WER is 20%, we observe large span,
between 5 and 25% WER in German, where multi-sourcing outperforms single
source at least by several hundreths of BLEU. This span in Newstest11 is much
more narrow, only 20 to 25% WER in German. We hypothesize that it may be
caused by the domain difference. The lexical noise model is trained on
Europarl. In news domain, there may be fewer words for substitution, so the
noise consists more of deletions and insertions, and it might be more harmful
for MT in combination of two sources. However, multi-sourcing appears to be
robust to ASR errors regardless of whether we have one or both sources as
original.

En WER
---
| 0% | 10% | 20% | 30%
0% |  |  |  |
De WER 10% |  |  |  |
20% |  |  |  |
30% |  |  |  |
Figure 1:  Single-sourcing vs multi-sourcing with different level of
artificial ASR noise of the sources (% WER) in simultaneous mode on ESIC dev
set. The results are depicted as quality (BLEU) and latency (AL) trade-off of
the candidate systems. The plots highlighted by gray background show noise
levels where multi-sourcing (En+De, blue line) outperforms both single sources
in BLEU at least for AL$>$5.5.

##  5 Simultaneous Multi-Source

In the previous section, we experimented with offline translation with
artificial ASR noise and showed that multi-source models are indeed robust to
noise. However, one important use case of speech translation is in a real time
setting where simultaneous MT is used. We therefore evaluate the robustness of
multi-source models in a simultaneous setting.

###  5.1 Simultaneous Machine Translation

Simultaneous MT is a task that simulates one subtask of a technology that
translates long-form monologue speech in real-time, or with the lowest
possible latency. There exist two main approaches to simultaneous MT:
streaming and re-translating Niehues et al. (2018); Arivazhagan et al. (2020).
Re-translating systems generate preliminary translation hypotheses that can be
updated. Both approaches have complementary benefits and drawbacks. In this
paper, we focus on streaming.

We assume that simultaneous MT continuously receives an input text segmented
to sentences, one token at a time, as produced by the speaker and upstream
tasks. After reading each input token, the system can either produce one or
more target tokens, or decide to read the next input token, e.g. to have more
context for translation. The goal of simultaneous MT is to translate the input
with high quality and low latency. Quality is measured on full sentences as in
standard text-to-text MT, e.g. by BLEU. The standard latency measure of
simultaneous MT is Average Lagging (AL, Ma et al., 2019). It is an average
number of tokens behind an “optimal” policy that generates the target
proportionally with reading the source.

Simultaneous MT can be created from standard text-to-text NMT by applying any
simultaneous decoding algorithm. However, it is recommendable first to adapt
NMT, so it is inclined to translate consecutive sentence prefixes with the
same target prefix. We use Local Agreement (LA-$n$) as a decoding algorithm.
It achieved good performance by the best performing system Polák et al. (2022)
in the most recent IWSLT competition Anastasopoulos et al. (2022). Local
Agreement (LA-$n$) means that $n$ consecutive updates must agree on a target
prefix to commit and write. The last committed prefix is then forced as a
prefix to decoding the next units. Agreement size $n$ is a parameter that
controls the latency.

###  5.2 Creating Simultaneous MT Systems

In Section 4, we used multi-way models trained on full sentences, but in a
simultaneous setting, these models will make mistakes when translating partial
sentences using the LA-$n$ approach. Therefore, our multi-way models should
first be adapted for partial sentence translation. To this end, we used the
multi-way English and German to Czech MT model as a base for simultaneous MT.
We fine-tuned the last trained model checkpoint for stable translation on 1:1
mix of incomplete sentence prefixes and full sentences as Niehues et al.
(2018). For each source-target pair of the training data, we selected 5-times
1 to 90 % of source and target characters and rounded them to full words.
Then, we ran training for 1 day on 1 GPU. We validated BLEU score on ESIC dev
and Normalized Erasure (NE, Arivazhagan et al., 2020) on all prefixes of the
first 65 sentences (around 1500 words) of ESIC dev set. We ran fine-tuning
with multi-way data for English and German as source languages, and for
bilingual English-Czech and German-Czech MT.

We stopped training after one day when there were no improvements in stability
or quality. Then, we selected one checkpoint for English and one for German
that reached acceptable quality and stability values. See Appendix B for
details.

###  5.3 Multi-Sourcing in Simultaneous MT

We use late averaging of the two selected checkpoints for multi-sourcing in
simultaneous MT. The only aspects of multi-sourcing in simultaneous mode that
differ from single-source or non-simultaneous mode are synchronization of the
sources and how to count Average Lagging.

#### Synchronization

In a realistic use-case, it is necessary to synchronize the original speech
and simultaneous interpreting. However, we leave it for further work, as our
goal is to inspect the limits of multi-sourcing. Therefore, we simulate a case
where the sources are optimally synchronized, aligned and parallel to sentence
level.

In multi-source mode, we sort all sentence prefixes by proportion of the
character length to the sentence length. Each “Read” operation of the multi-
source system then receives two prefixes in two languages. One of them is
updated by one new token. Every such update is counted to local agreement
size. We note that there are other strategies, e.g. count only English source
updates to LA-$n$, but in this paper we have other goal than searching for the
best strategy.

#### AL in multi-source

In multi-source setup, we only count Read operations of the English source to
AL calculations that we report, and not of the German source because the
sources are simultaneous. Counting only German tokens differs negligibly,
approximately by 0.1 tokens.

###  5.4 Simultaneous Multi-Source with Artifical Noise

We want to compare multi-sourcing model to single-sourcing with artificial ASR
noise model as in Section 4.1. We evaluate each system on the latency levels
with local agreement sizes 2, 5, 10 and 15. Since each evaluation takes
approximately 5 hours on 2000 sentences, we report only one run, and not
average and deviation on multiple randomly noised inputs.

The results on ESIC dev set are in Figure 1. We can observe the same trends as
in the offline case. The single source that is noised less achieves higher
BLEU. Multi-sourcing outperforms both single sources when both noise levels
are similar and when the English one is lower, e.g. in the case with 10% WER
in German and 20% WER in English. We explain it again by the fact that the
Czech reference is translated from English, and not German.

Furthermore, on both ESIC and Newstest11 (Figure 1) we observe that multi-
sourcing performs worse in the low latency modes, i.e. in AL$<$5 that roughly
corresponds to LA$<$5\. We assume that the proportional synchronization of the
two sources is often inaccurate and may confuse late averaging. In higher
latency modes, the synchronization noise at the end of input may be lowered by
local agreement. Having validated the multi-source NMT is robust to ASR errors
in both full sentence and simultaneous settings, we have paved the way for
harder settings where multilingual interpretations of the original source
available with different amounts of delay can be used for translation.

##  6 Conclusion

We have investigated the robustness of multi-source NMT to transcription
errors in order to motivate its use in settings where ASRs for the original
speech and its simultaneously interpreted equivalent are available. To this
end, we first analyzed the 10-hour ESIC corpus and documented that the ASR
errors in the two sources are indeed independent, indicating their
complementary nature. We then simulated transcription noise for English and
German when translating into Czech in single and multi-source NMT settings and
observed that using multiple noisy sources is significantly better than
individual noisy sources. We then repeated experiments in a simultaneous
translation setting and showed that multi-source translation continues to be
robust to noise. This robustness of multi-source NMT to noise motivates future
research into simultaneous multi-source speech translation, where one source
is available with a delay. We will also consider training models with
simulated ASR errors to further increase their robustness, especially in
multi-source settings.

##  7 Limitations

Although we have shown the robustness of multi-source NMT to transcription
errors in a full-sentence and simultaneous settings, our work has the
following limitations:

  * •

Our work does not address the case where the additional source, typically
interpreted, is available after a delay. A delayed source may reduce the gains
seen by multi-sourcing.

  * •

We have only focused on the Local Agreement (LA-$n$) approach for simultaneous
translation and exploration of other simultaneous approaches such as wait-$k$
remains.

  * •

Human evaluation of translations is pending.

  * •

Evaluation on other language pairs is pending.

## Acknowledgements

The research was partially supported by the grants 19-26934X (NEUREM3) of the
Czech Science Foundation, “Grant Schemes at CU” (reg. no.
CZ.02.2.69/0.0/0.0/19_073/0016935), 398120 of the Grant Agency of Charles
University, and SVV project number 260 698. Part of the work was done during
an internship at NICT.

## References

  * Anastasopoulos et al. (2022) Antonios Anastasopoulos, Loïc Barrault, Luisa Bentivogli, Marcely Zanon Boito, Ondřej Bojar, Roldano Cattoni, Anna Currey, Georgiana Dinu, Kevin Duh, Maha Elbayad, Clara Emmanuel, Yannick Estève, Marcello Federico, Christian Federmann, Souhir Gahbiche, Hongyu Gong, Roman Grundkiewicz, Barry Haddow, Benjamin Hsu, Dávid Javorský, Vĕra Kloudová, Surafel Lakew, Xutai Ma, Prashant Mathur, Paul McNamee, Kenton Murray, Maria Nǎdejde, Satoshi Nakamura, Matteo Negri, Jan Niehues, Xing Niu, John Ortega, Juan Pino, Elizabeth Salesky, Jiatong Shi, Matthias Sperber, Sebastian Stüker, Katsuhito Sudoh, Marco Turchi, Yogesh Virkar, Alexander Waibel, Changhan Wang, and Shinji Watanabe. 2022.  Findings of the IWSLT 2022 evaluation campaign.  In _Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)_ , pages 98–157, Dublin, Ireland (in-person and online). Association for Computational Linguistics.
  * Arivazhagan et al. (2020) Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, and George Foster. 2020.  Re-translation versus streaming for simultaneous translation.  In _Proceedings of the 17th International Conference on Spoken Language Translation_ , pages 220–227, Online. Association for Computational Linguistics.
  * Bojar et al. (2012) Ondřej Bojar, Daniel Zeman, Ondřej Dušek, Jana Břečková, Hana Farkačová, Pavel Grošpic, Kristýna Kačenová, Eva Knechtová, Anna Koubová, Jana Lukavská, Petra Nováková, and Jana Petrdlíková. 2012.  Additional German-Czech reference translations of the WMT’11 test set.  LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
  * Callison-Burch et al. (2011) Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011.  Findings of the 2011 workshop on statistical machine translation.  In _Proceedings of the Sixth Workshop on Statistical Machine Translation_ , pages 22–64, Edinburgh, Scotland. Association for Computational Linguistics.
  * Chen et al. (2021) Pinzhen Chen, Jindřich Helcl, Ulrich Germann, Laurie Burchell, Nikolay Bogoychev, Antonio Valerio Miceli Barone, Jonas Waldendorf, Alexandra Birch, and Kenneth Heafield. 2021.  The University of Edinburgh’s English-German and English-Hausa submissions to the WMT21 news translation task.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 104–109, Online. Association for Computational Linguistics.
  * Cho et al. (2013) Eunah Cho, Christian Fügen, Teresa Hermann, Kevin Kilgour, Mohammed Mediani, Christian Mohr, Jan Niehues, Kay Rottmann, Christian Saam, Sebastian Stüker, and Alex Waibel. 2013.  A real-world system for simultaneous translation of German lectures.  In _Proc. Interspeech 2013_ , pages 3473–3477.
  * Dabre et al. (2020) Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. 2020.  A survey of multilingual neural machine translation.  _ACM Comput. Surv._ , 53(5).
  * Dabre et al. (2018) Raj Dabre, Fabien Cromieres, and Sadao Kurohashi. 2018.  Exploiting multilingual corpora simply and efficiently in neural machine translation.  _Journal of Information Processing_ , 26:406–415.
  * Dabre et al. (2021) Raj Dabre, Aizhan Imankulova, Masahiro Kaneko, and Abhisek Chakrabarty. 2021.  Simultaneous multi-pivot neural machine translation.  _CoRR_ , abs/2104.07410.
  * Di Gangi et al. (2019) Matti Di Gangi, Robert Enyedi, Alessandra Brusadin, and Marcello Federico. 2019\.  Robust neural machine translation for clean and noisy speech transcripts.  In _Proceedings of the 16th International Conference on Spoken Language Translation_ , Hong Kong. Association for Computational Linguistics.
  * Firat et al. (2016) Orhan Firat, Baskaran Sankaran, Yaser Al-onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. 2016.  Zero-resource translation with multi-lingual neural machine translation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 268–277, Austin, Texas. Association for Computational Linguistics.
  * Gaido et al. (2022) Marco Gaido, Matteo Negri, and Marco Turchi. 2022.  Who are we talking about? handling person names in speech translation.  In _Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)_ , pages 62–73, Dublin, Ireland (in-person and online). Association for Computational Linguistics.
  * Gaido et al. (2021) Marco Gaido, Susana Rodríguez, Matteo Negri, Luisa Bentivogli, and Marco Turchi. 2021.  Is “moby dick” a whale or a bird? named entities and terminology in speech translation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1707–1716, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Gulati et al. (2020) Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020.  Conformer: Convolution-augmented Transformer for Speech Recognition.  In _Proc. Interspeech 2020_ , pages 5036–5040.
  * Junczys-Dowmunt (2018) Marcin Junczys-Dowmunt. 2018.  Dual conditional cross-entropy filtering of noisy parallel corpora.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 888–895, Belgium, Brussels. Association for Computational Linguistics.
  * Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018.  Marian: Fast neural machine translation in C++.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 116–121, Melbourne, Australia. Association for Computational Linguistics.
  * Khadivi and Ney (2008) Shahram Khadivi and Hermann Ney. 2008.  Integration of speech recognition and machine translation in computer-assisted translation.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 16(8):1551–1564.
  * Kocmi et al. (2021) Tom Kocmi, Dominik Macháček, and Ondřej Bojar. 2021.  _The Reality of Multi-Lingual Machine Translation_, volume 22 of _Studies in Computational and Theoretical Linguistics_.  Institute of Formal and Applied Linguistics, Prague, Czechia.
  * Kratochvíl et al. (2020) Jonáš Kratochvíl, Peter Polák, and Ondřej Bojar. 2020.  Large corpus of Czech parliament plenary hearings.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 6363–6367, Marseille, France. European Language Resources Association.
  * Kuchaiev et al. (2019) Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Samuel Kriman, Stanislav Beliaev, Vitaly Lavrukhin, Jack Cook, et al. 2019.  Nemo: a toolkit for building ai applications using neural modules.  _arXiv preprint arXiv:1909.09577_.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Ma et al. (2019) Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. 2019.  STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3025–3036, Florence, Italy. Association for Computational Linguistics.
  * Macháček et al. (2019) Dominik Macháček, Jonáš Kratochvíl, Tereza Vojtěchová, and Ondřej Bojar. 2019.  A speech test set of practice business presentations with additional relevant texts.  In _Statistical Language and Speech Processing_ , pages 151–161, Cham. Springer International Publishing.
  * Macháček et al. (2021) Dominik Macháček, Matúš Žilinec, and Ondřej Bojar. 2021.  Lost in Interpreting: Speech Translation from Source or Interpreter? In _Proc. Interspeech 2021_ , pages 2376–2380.
  * Martucci et al. (2021) Giuseppe Martucci, Mauro Cettolo, Matteo Negri, and Marco Turchi. 2021.  Lexical Modeling of ASR Errors for Robust Speech Translation.  In _Proc. Interspeech 2021_ , pages 2282–2286.
  * Mathur et al. (2020) Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020\.  Results of the WMT20 metrics shared task.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 688–725, Online. Association for Computational Linguistics.
  * Miranda et al. (2013) João Miranda, João Paulo Neto, and Alan W Black. 2013.  Improved punctuation recovery through combination of multiple speech streams.  In _2013 IEEE Workshop on Automatic Speech Recognition and Understanding_ , pages 132–137.
  * Nguyen et al. (2021) Thai-Son Nguyen, Sebastian Stüker, and Alex Waibel. 2021.  Super-Human Performance in Online Low-Latency Recognition of Conversational Speech.  In _Proc. Interspeech 2021_ , pages 1762–1766.
  * Niehues et al. (2018) Jan Niehues, Ngoc-Quan Pham, Thanh-Le Ha, Matthias Sperber, and Alex Waibel. 2018\.  Low-Latency Neural Speech Translation.  In _Proc. Interspeech 2018_ , pages 1293–1297.
  * Nishimura et al. (2018) Yuta Nishimura, Katsuhito Sudoh, Graham Neubig, and Satoshi Nakamura. 2018.  Multi-source neural machine translation with data augmentation.  In _Proceedings of the 15th International Conference on Spoken Language Translation_ , pages 48–53, Brussels. International Conference on Spoken Language Translation.
  * Och and Ney (2001) Franz Josef Och and Hermann Ney. 2001.  Statistical multi-source translation.  In _Proceedings of Machine Translation Summit VIII_ , Santiago de Compostela, Spain.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Paulik et al. (2005) M. Paulik, S. Stuker, C. Fugen, T. Schultz, T. Schaaf, and A. Waibel. 2005.  Speech translation enhanced automatic speech recognition.  In _IEEE Workshop on Automatic Speech Recognition and Understanding, 2005._ , pages 121–126.
  * Paulik and Waibel (2008) Matthias Paulik and Alex Waibel. 2008.  Extracting clues from human interpreter speech for spoken language translation.  In _2008 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pages 5097–5100.
  * Pearson (1900) Karl Pearson. 1900.  X. on the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling.  _The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science_ , 50(302):157–175.
  * Polák et al. (2022) Peter Polák, Ngoc-Quan Pham, Tuan Nam Nguyen, Danni Liu, Carlos Mullov, Jan Niehues, Ondřej Bojar, and Alexander Waibel. 2022.  CUNI-KIT system for simultaneous speech translation task at IWSLT 2022.  In _Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)_ , pages 277–285, Dublin, Ireland (in-person and online). Association for Computational Linguistics.
  * Popović (2016) Maja Popović. 2016.  chrF deconstructed: beta parameters and n-gram weights.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 499–504, Berlin, Germany. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Povey et al. (2011) Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukáš Burget, Ondřej Glembek, K. Nagendra Goel, Mirko Hannemann, Petr Motlíček, Yanmin Qian, Petr Schwarz, Jan Silovský, Georg Stemmer, and Karel Veselý. 2011\.  The Kaldi speech recognition toolkit.  In _Proceedings of ASRU 2011_ , pages 1–4. IEEE Signal Processing Society.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  Unbabel’s participation in the WMT20 metrics shared task.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 911–920, Online. Association for Computational Linguistics.
  * Ruiz et al. (2017) Nicholas Ruiz, Mattia Antonino Di Gangi, Nicola Bertoldi, and Marcello Federico. 2017.  Assessing the Tolerance of Neural Machine Translation Systems Against Speech Recognition Errors.  In _Proc. Interspeech 2017_ , pages 2635–2639.
  * Serai et al. (2022) Prashant Serai, Vishal Sunder, and Eric Fosler-Lussier. 2022.  Hallucination of speech recognition errors with sequence to sequence learning.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 30:890–900.
  * Sperber et al. (2017) Matthias Sperber, Jan Niehues, and Alex Waibel. 2017.  Toward robust neural machine translation for noisy input sequences.  In _Proceedings of the 14th International Conference on Spoken Language Translation_ , pages 90–96, Tokyo, Japan. International Workshop on Spoken Language Translation.
  * Sperber and Paulik (2020) Matthias Sperber and Matthias Paulik. 2020.  Speech translation and the end-to-end promise: Taking stock of where we are.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7409–7421, Online. Association for Computational Linguistics.
  * Tiedemann and Nygaard (2004) Jörg Tiedemann and Lars Nygaard. 2004.  The OPUS corpus - parallel and free: http://logos.uio.no/opus.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)_ , Lisbon, Portugal. European Language Resources Association (ELRA).
  * Tiedemann and Thottingal (2020) Jörg Tiedemann and Santhosh Thottingal. 2020.  OPUS-MT – building open translation services for the world.  In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation_ , pages 479–480, Lisboa, Portugal. European Association for Machine Translation.
  * Wang et al. (2021) Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021.  VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 993–1003, Online. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xue et al. (2020) Haiyang Xue, Yang Feng, Shuhao Gu, and Wei Chen. 2020.  Robust neural machine translation with ASR errors.  In _Proceedings of the First Workshop on Automatic Simultaneous Translation_ , pages 15–23, Seattle, Washington. Association for Computational Linguistics.
  * Zoph and Knight (2016) Barret Zoph and Kevin Knight. 2016.  Multi-source neural translation.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 30–34, San Diego, California. Association for Computational Linguistics.

##  Appendix A chrF2 Scores with Noisy Inputs

There is an evidence that chrF2 correlates with human judgements better than
BLEU. In Appendix A, we can see that for multi-sourcing with noisy inputs on
ESIC dev, chrF2 are indeed higher than single-sourcing, and this correlates
with the BLEU score gains in Section 4. On the other hand, for Newstest11,
chrF2 scores do not indicate any improvements. While the corresponding BLEU
scores in Section 4 indicated improvements of multi-sourcing with noisy
inputs, the magnitude of these gains were minor, much smaller than those
observed for ESIC. This gives us sufficient reason to believe that multi-
sourcing should be useful in a setting like ESIC, where the reference is
created from only one source, which is more realistic than the “balanced” use-
case of Newstest11, where the reference originates from 5 languages.

chrF2 | ESIC dev | En WER |
---|---|---|---
|  | single-src. | 0 % | 5 % | 10 % | 15 % | 20 % | 25 % | 30 % | 35 % | 40 % |
| s-src. |  | 60.2$\pm$0.0 | 57.2$\pm$0.2 | 54.4$\pm$0.2 | 51.4$\pm$0.3 | 49.2$\pm$0.7 | 46.8$\pm$0.8 | 44.3$\pm$0.0 | 42.1$\pm$0.3 | 40.1$\pm$0.0 |
De WER | 0 % | 54.0$\pm$0.0 | 58.6$\pm$0.0 | 56.9$\pm$0.2 |  | 55.6$\pm$0.1
---
53.7$\pm$0.2 | 52.3$\pm$0.5 | 50.9$\pm$0.4 | 49.2$\pm$0.3 | 47.5$\pm$0.2 | 46.1$\pm$0.2 |
5 % | 51.8$\pm$0.1 | 57.7$\pm$0.1 | 56.2$\pm$0.2 |  | 54.8$\pm$0.1
---
| 52.9$\pm$0.2
---
51.4$\pm$0.6 | 50.0$\pm$0.4 | 48.3$\pm$0.3 | 46.7$\pm$0.3 | 45.4$\pm$0.2 |
10 % | 49.9$\pm$0.2 | 56.8$\pm$0.2 | 55.1$\pm$0.1 | 53.7$\pm$0.3 |  | 51.8$\pm$0.3
---
| 50.4$\pm$0.3
---
49.0$\pm$0.4 | 47.3$\pm$0.1 | 45.6$\pm$0.2 | 44.3$\pm$0.2 |
15 % | 47.6$\pm$0.3 | 55.8$\pm$0.0 | 54.2$\pm$0.1 | 52.8$\pm$0.3 | 50.9$\pm$0.2 |  | 49.6$\pm$0.4
---
| 48.1$\pm$0.5
---
46.4$\pm$0.3 | 44.9$\pm$0.3 | 43.6$\pm$0.1 |
20 % | 45.7$\pm$0.3 | 54.9$\pm$0.2 | 53.5$\pm$0.1 | 51.9$\pm$0.3 | 50.2$\pm$0.1 | 48.7$\pm$0.6 |  | 47.2$\pm$0.4
---
45.4$\pm$0.2 | 43.9$\pm$0.3 | 42.6$\pm$0.3 |
25 % | 44.0$\pm$0.4 | 54.2$\pm$0.4 | 52.9$\pm$0.1 | 51.3$\pm$0.2 | 49.3$\pm$0.2 | 48.1$\pm$0.4 | 46.5$\pm$0.4 |  | 44.7$\pm$0.2
---
43.3$\pm$0.2 | 41.7$\pm$0.0 |
30 % | 42.1$\pm$0.3 | 53.1$\pm$0.3 | 51.7$\pm$0.3 | 50.2$\pm$0.2 | 48.3$\pm$0.3 | 46.8$\pm$0.3 | 45.4$\pm$0.5 | 43.5$\pm$0.3 |  | 42.2$\pm$0.2
---
40.6$\pm$0.1 |
35 % | 40.5$\pm$0.2 | 52.0$\pm$0.3 | 50.0$\pm$0.3 | 48.7$\pm$0.2 | 46.9$\pm$0.1 | 45.7$\pm$0.5 | 44.1$\pm$0.4 | 42.4$\pm$0.2 | 41.0$\pm$0.1 | 39.7$\pm$0.1 |
40 % | 38.6$\pm$0.2 | 51.1$\pm$0.2 | 49.2$\pm$0.2 | 47.8$\pm$0.3 | 46.0$\pm$0.4 | 44.8$\pm$0.3 | 43.2$\pm$0.4 | 41.5$\pm$0.1 | 39.8$\pm$0.3 | 38.6$\pm$0.1 |

chrF2 news11 En WER single-src. 0 % 5 % 10 % 15 % 20 % 25 % 30 % 35 % 40 %
s-src. 51.0$\pm$0.0 48.8$\pm$0.2 46.9$\pm$0.1 44.9$\pm$0.1 43.0$\pm$0.1
41.0$\pm$0.1 39.4$\pm$0.1 37.3$\pm$0.1 35.8$\pm$0.0 De WER 0 % 50.3$\pm$0.0
50.8$\pm$0.0 49.5$\pm$0.0 48.0$\pm$0.1 46.7$\pm$0.1 45.3$\pm$0.2 43.8$\pm$0.3
42.6$\pm$0.2 41.0$\pm$0.1 39.7$\pm$0.1 5 % 48.3$\pm$0.1 50.0$\pm$0.0
48.6$\pm$0.0 47.3$\pm$0.0 45.7$\pm$0.1 44.5$\pm$0.1 43.1$\pm$0.1 41.8$\pm$0.0
40.1$\pm$0.1 38.8$\pm$0.1 10 % 46.5$\pm$0.2 49.2$\pm$0.1 47.9$\pm$0.2
46.4$\pm$0.1 44.8$\pm$0.1 43.8$\pm$0.0 42.3$\pm$0.0 40.9$\pm$0.1 39.2$\pm$0.2
38.0$\pm$0.1 15 % 44.7$\pm$0.2 48.1$\pm$0.1 46.7$\pm$0.1 45.4$\pm$0.1
43.9$\pm$0.1 42.8$\pm$0.0 41.2$\pm$0.0 40.1$\pm$0.1 38.4$\pm$0.0 37.1$\pm$0.0
20 % 42.9$\pm$0.1 47.1$\pm$0.0 45.8$\pm$0.1 44.3$\pm$0.0 42.9$\pm$0.1
41.7$\pm$0.1 40.4$\pm$0.1 39.1$\pm$0.0 37.4$\pm$0.1 36.3$\pm$0.1 25 %
41.1$\pm$0.1 46.1$\pm$0.2 44.8$\pm$0.0 43.6$\pm$0.1 42.0$\pm$0.1 40.8$\pm$0.1
39.3$\pm$0.2 38.1$\pm$0.1 36.4$\pm$0.1 35.3$\pm$0.1 30 % 39.4$\pm$0.2
45.3$\pm$0.3 43.9$\pm$0.2 42.6$\pm$0.2 41.1$\pm$0.1 39.9$\pm$0.2 38.5$\pm$0.2
37.3$\pm$0.1 35.7$\pm$0.0 34.5$\pm$0.2 35 % 38.0$\pm$0.2 44.3$\pm$0.2
42.9$\pm$0.3 41.5$\pm$0.1 40.2$\pm$0.2 38.9$\pm$0.2 37.6$\pm$0.1 36.4$\pm$0.1
34.9$\pm$0.0 33.7$\pm$0.2 40 % 36.2$\pm$0.2 43.2$\pm$0.2 41.9$\pm$0.2
40.5$\pm$0.2 39.1$\pm$0.1 37.9$\pm$0.1 36.4$\pm$0.2 35.2$\pm$0.1 33.8$\pm$0.2
32.8$\pm$0.2

Table 8:  chrF2 (avg$\pm$stddev) with transcription noise on ESIC dev set
whose reference translations was English and on Newstest11 (news11) with
balanced reference source language. The area with the green background is
where the English single-source outperforms German single-source.  Black
underlined
---
numbers indicate the area where multi-sourcing achieves higher score than both
single-sourcing options. Red-colored numbers are where at least one single-
source scores higher.

##  Appendix B Checkpoint Selection for Simultaneous Multi-Source

The checkpoints that we selected for simultaneous multi-source decoding
(recall Section 5.2) was the multi-way checkpoint for English and bilingual
one for German. Table 9 summarizes the results of fine-tuning for stability.
BLEU decreased marginally (by 0.2 on English and 0.9 on German), while
normalized erasure (NE) dropped by 40% in English and 52% on German.

Based on some outputs, we explain higher NE in German-to-Czech by discrepancy
in word orders. Many erasures were caused by an incorrect presumption of the
final verb. Regardless, our fine-tuned models exhibit significantly reduced NE
and can be reliably used for simultaneous translation using the LA-$n$
approach.

| En | De
---|---|---
checkpoint | BLEU | NE | BLEU | NE
starting | 33.2 | 1.77 | 25.9 | 3.15
selected | 33.0 | 1.21 | 25.0 | 1.52
diff | -0.2 | -40% | -0.9 | -52%
Table 9: The results of fine-tuning for stability.

##  Appendix C Effect of Reference Source Language

To explain the effect of reference source language, we run a contrastive
evaluation on the subset of Newstest11 that consists only from the documents
that originate in English. We compare BLEU measures with a reference
translated directly from Czech, and with three additional references
translated only from German Bojar et al. (2012).

The results of simultaneous mode (recall Section 5) are in Figure 2. We
observe the same trends as in offline mode in Section 4. The BLEU score is
higher for the single source with the language from which the reference was
translated. When this source is noised substantially more than the other,
multi-sourcing outperforms both by a small margin.

In case of German references, the nearest margin to single-sourcing is much
smaller than with the English references. We assume it is because the
structural difference of English source and German-Czech references is larger
than German to English-Czech reference. It is documented also by BLEU scores
with zero noise (33 and 20 on references from English vs 16 and 30 on
references from German).

News11 En part, En ref  En WER 0% 10% 20% 30% 0% De WER 10% 20% 30% News11 En
part, De refs  En WER 0 % 10 % 20 % 30 % 0% De WER 10% 20% 30%

Figure 2:  Single-sourcing vs multi-sourcing with different level of
artificial ASR noise of the sources (% WER) in simultaneous mode on Newstest11
subset (598 sentences) originally in English. In the upper grid, the Czech
reference is translated from English, while in the lower, there is average and
standard deviation of BLEU counted against the 3 additional references
translated from German Bojar et al. (2012). Grey highlighting indicates area
where multi-sourcing (En+De, blue line) outperforms or is on-par with both
single sources in BLEU.


/csteps/inner color=white /csteps/fill color=black

#  LionHeart: A Layer-based Mapping Framework for Heterogeneous Systems with
Analog In-Memory Computing Tiles

Corey Lammie  IBM Research Europe, 8803 Rüschlikon, Switzerland  Flavio
Ponzina  Embedded Systems Laboratory (ESL), École Polytechnique Fédérale de
Lausanne (EPFL), Lausanne, Switzerland  Yuxuan Wang  Embedded Systems
Laboratory (ESL), École Polytechnique Fédérale de Lausanne (EPFL), Lausanne,
Switzerland  Joshua Klein  Embedded Systems Laboratory (ESL), École
Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland  Marina
Zapater  HEIG-VD, Yverdon-les-Bains, Switzerland
Irem Boybat  IBM Research Europe, 8803 Rüschlikon, Switzerland  Abu Sebastian
IBM Research Europe, 8803 Rüschlikon, Switzerland  Giovanni Ansaloni  Embedded
Systems Laboratory (ESL), École Polytechnique Fédérale de Lausanne (EPFL),
Lausanne, Switzerland  David Atienza  Embedded Systems Laboratory (ESL), École
Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland

###### Abstract

When arranged in a crossbar configuration, resistive memory devices can be
used to execute Matrix-Vector Multiplications, the most dominant operation of
many Machine Learning (ML) algorithms, in constant time complexity.
Nonetheless, when performing computations in the analog domain, novel
challenges are introduced in terms of arithmetic precision and stochasticity,
due to non-ideal circuit and device behaviour. Moreover, these non-idealities
have a temporal dimension, resulting in a degrading application accuracy over
time. Facing these challenges, we propose a novel framework, named _LionHeart_
, to obtain hybrid analog-digital mappings to execute Deep Learning (DL)
inference workloads using heterogeneous accelerators. The accuracy-constrained
mappings derived by _LionHeart_ showcase, across different Deep Neural
Networks (DNNs) and datasets, high accuracy and potential for speedup. The
results of the full system simulations highlight run-time reductions and
energy efficiency gains that exceed 6$\times$, with a user-defined accuracy
threshold with respect to a fully digital floating point implementation.

###### Index Terms:

AIMC, DNNs, Heterogeneous Systems, Mapping, System Simulation

##  I Introduction

Deep Learning-based solutions have been applied in various industrial and
scientific applications, including computer vision, natural language
processing, and speech recognition. Nowadays, State-Of-The-Art (SOTA) DL
models require significant memory to store their parameters (i.e., weights and
biases), and billions of Multiply-Accumulate (MAC) operations per inference.
These extreme computing and memory requirements pose a challenge for deploying
DL workloads on edge devices, whose form factor and energy budget require
limited computing capabilities and memory size. However, the execution of DL
workloads at the edge represents a very attractive prospect, as it would allow
the processing of data samples close to where it is collected, resulting in
several benefits, including reduced latency, a higher degree of security and
privacy, improved bandwidth efficiency, as well as increased resiliency.
Therefore, many research works have proposed strategies to optimize DL
workloads for edge scenarios.

Figure 1:  _LionHeart_ considers the mapping of FC and CONV DNN layers to
heterogeneous PUs featuring AIMC tiles and general-purpose cores to maximize
performance while adhering to a user-specified accuracy constraint.

ML algorithms, such as DNNs and transformers, are structured as regular
sequences of operations. For the most part, they implement linear algebra
computing kernels, which can be effectively parallelized. Hence, a number of
customized hardware architectures dedicated to accelerating ML workloads have
been proposed in recent years. They exploit reprogrammable logic or fixed-
function implementations, such as Tensor Processing Units. A disruptive
approach in this space is In-Memory Computing (IMC), for which storage and
computation is performed at the same physical location, avoiding the well-
known memory wall problem.

Different IMC approaches have been proposed in the literature, operating at
varying levels of the memory hierarchy and leveraging different technologies.
Among them, Analog In-Memory Computing (AIMC) is emerging as a particularly
appealing alternative. A key drawback hampering the widespread adoption of
AIMC is its sensitivity to device and circuit-related variations such as
temporal conductance variations, device stochasticity, and circuit non-
idealities, which can adversely affect accuracy [1]. Many ongoing efforts
investigate these effects and propose strategies to mitigate them, including
Hardware-Aware (HWA) training [2] and model co-optimization [3].

Addressing the accuracy challenge from a system perspective, we present herein
a novel framework that enables the effective exploration of hybrid
digital/analog mappings of DL models (Fig. 1), taking into consideration the
current limitations of AIMC which affect accuracy. Our framework effectively
navigates the trade-off between run-time, performance, and accuracy
degradation. Recognizing the exponential relation between the number of MVMs
that can be executed in either the digital or analog domain, and the possible
mapping solutions, we herein introduce an accuracy-driven heuristic that, by
considering the largest MVM operations first (in terms of MAC operations),
reduces the exploration problem to that of linear complexity. The main
contributions presented in this work are summarized as follows:

  * •

We present a novel accuracy-driven training framework, able to explore the
space of hybrid digital/analog implementations of DNNs. The framework
allocates Fully Connected (FC) and Convolutional (CONV) layers to digital or
analog resources to minimize run-time while constraining the adverse effect of
analog computations on accuracy. Additionally, it is hardware-agnostic,
meaning it can be applied to AIMC crossbars adopting different device
technologies;

  * •

We derive a set of optimized hybrid mappings ensuring user-defined accuracy
levels for a number of DNNs trained for CIFAR-10 and CIFAR-100 image
classification, showing that in all cases a sizable share of their workload
can be executed in the analog domain, even for degradation thresholds as low
as 0.5% with respect to floating-point precision implementations;

  * •

We investigate the speedup-accuracy tradeoff achieved by our mappings. To this
end, we perform power and performance evaluations on different types of ML
workloads using a cycle-accurate simulator based on gem5-X. The considered
hybrid system employs a single-core in-order CPU embedding tightly-coupled
AIMCs for in-memory acceleration;

  * •

We recognize and address the real-world system accuracy degradation caused by
temporal variation of programmed AIMCs crossbars at a user-desired evaluation
time, $t_{eval}$.

##  II Background and Related Work

###  II-A In-Memory Computing

Traditional von Neumann computing systems involve separate processing and
memory units, which consume significant energy and introduce additional
latency when data is moved between them. IMC is a non-von Neumann
computational approach, in which computation and storage are performed at the
same physical location. From a technology perspective, both SRAM- and DRAM-
based IMC architectures have been proposed [1]. Computation in these
architectures can be achieved by leveraging charge sharing when concurrently
activating multiple memory rows. A promising alternative, which we focus on in
this paper, is that of employing Non-Volatile Memory (NVM) devices, organized
in a crossbar arrangement [1]. These devices, used to encode network weights
in a differential mapping scheme, are employed at the junction of word- and
bit-lines, and their resistances are modulated according to the target weight
value at compile time.

In AIMC crossbars, inputs are converted to word line voltages using Digital-
to-Analog Converters, and currents, representative of MVMs results, are
collected across bit lines into Analog-to-Digital Converters. Operations on
AIMC crossbars are performed in parallel, and hence, can be realized in
constant time complexity. For instance, a 256$\times$256 Phase Change Memory
(PCM) crossbar is shown to perform a total of 65,536 MAC operations in 130ns
[4]. However, as computation is performed in the analog domain, outputs are
affected by non-deterministic noise, deviating from expected results. This is
due to a number of device and circuit non-idealities, including temporal and
temperature-dependent conductance variations, device stochasticity, ADC
mismatch, and IR drop [5].

Figure 2:  (a) High-level overview of our proposed framework. (b) The
validation accuracy at the desired evaluation time ($t_{eval}=$1d) and
training loss of a candidate AIMC layer during analog HWA retraining. Training
is considered converged when, for a user-defined convergence window, the
training loss does not decrease. (c) High-level overview of analog HWA
retraining.

###  II-B Hardware-Aware Training

Analog HWA offline training is a common approach to mitigate some of the AIMC
device and circuit non-idealities, by making the model more robust for
deployment. Additional operations, including weight-noise injection, are added
during forward propagation passes. Mathematically, the generally simulated
AIMC forward passes can be expressed as

$y_{i}=\alpha^{\textrm{out}}_{i}f_{\textrm{adc}}\big{(}\sum_{j}({w}_{ij}+\sigma_{\textrm{w}}\xi_{ij})(f_{\textrm{dac}}(x_{j})+\sigma_{\textrm{inp}}\xi_{j})+\sigma_{\textrm{out}}\xi_{i}\big{)}+\beta_{i},$
(1)

where $f_{\text{adc}}$ and $f_{\text{dac}}$ model the analog-to-digital and
digital-to-analog processes, with dynamic scaling and range clipping, and
$\xi$ denotes Gaussian noise. $\sigma$ indicates noise sampled from a zero-
centered normal distribution. In addition to training noise-resilient weights,
it is also possible to train other parameters, such as the input (DAC) bounds
of each Word-Line (WL), using back-propagation. We employ the open-source
AIHWKIT toolkit [5, 6] to expose these effect in system-level explorations in
_LionHeart_ , allowing both the realistic characterization of analog non-
idealities and analog HWA retraining. While the toolkit, and by extension, the
_LionHeart_ framework can be used to model other device technologies, such as
Resistive Random-Access Memory (RRAM), in this paper, we model PCM. Section
IV-B3 details the specific assumptions made during analog HWA retraining.

###  II-C Related Work

Related works, which perform application mapping on AIMC\- accelerated
systems, consider resource limitations as their main (or only) constraint.
PUMA [7] employs compiler analysis passes to define which parts of the
application have the most potential for speedup, while AERO [8] formulates
this same problem as a cost function minimization. Potential issues introduced
by non-idealities and noisy environments, characterizing AIMC devices, are
overlooked in the above-mentioned works. They are similarly disregarded in [9]
and [10], where all FC and CONV layers are executed using AIMC tiles, without
attempting to control induced accuracy degradation. Some consideration of the
effect of analog computing on accuracy is included in [11], where the authors
propose a hybrid solution where the first and last layers of DNNs are executed
on digital resources, postulating that these two layers are more sensitive to
noise. Two recent works present input channel-wise weight selection methods to
map to heterogeneous analog-digital resources. The first, HybridAC [12], does
so without employing any re-training strategy. The input is assumed to already
be HWA trained, and some layers are confined to be implemented using digital
resources. The second, ODiMO [13], relies on the unique approach of training a
supernetwork. Analog computation is modelled using 2-bit quantization.
However, both of these methods do not consider (i) the variability of analog
tiles as well as (ii) the temporal evolution of analog weights.

The complexity of devising high-performance, but accuracy-constrained
mappings, is compounded by the varying size and sensitivity towards noise of
the different layers in a DNN structure. Moreover, the number of possible
mappings makes exhaustive exploration impractical, even for simple cases, as
discussed, in the context of hardware quantization, in [14]. For example, even
when layer-wise mapping is considered, a relatively low-depth model like
AlexNet, composed of just seven layers, would therefore present $2^{7}$
possible analog-digital combinations. These would all have to be trained and
tested to arrive at an exact solution to the mapping problem. In contrast to
the aforementioned works, we employ a heuristic-based layer-wise mapping
strategy which scales linearly with respect the number of layers, as it only
consider one target candidate layer at a time.

##  III LionHeart framework

A high-level depiction of our framework is presented in Fig. 2. The input DNN
is assumed to be pre-trained in floating-point precision. Our methodology
consists of: 1 a MAC-based layer ranking (ordering) phase, in which the layers
are indexed according to their computational requirements, 2 - 4 an
optimization loop, evaluating layers greedily as possible candidates for AIMC
(analog) acceleration, and 5 a full system evaluation phase, where the full-
system framework in [15] is used to estimate run time. In the remainder of
this section, we detail those different components.

MAC-based Layer Ranking 1: Ordering is performed by calculating the number of
MAC operations required by each layer. Note that this step only has to be
performed once, as the number of MACs only depends on the DNN layer structure.

Mapper 2: During the optimization loop, layers are considered, one at a time,
in descending order with respect to their size (i.e., the number of required
MACs), and mapped to either analog or digital Processing Units (PUs). The
rationale behind this approach is that larger layers have more redundancy and,
hence, more leeway to compensate for the perturbations induced by analog
computing. Furthermore, the acceleration of a high number of MACs with AIMC
computing has the potential to achieve higher speedups. Initially, a selected
layer is tentatively considered as AIMC-accelerated. Its implementation is
then transformed from its digital representation to an analog model provided
by AIHWKIT, which accounts for non-idealities (noise, temporal drift, etc.).

TABLE I: The number of layers, weights, and the top-1 (baseline, floating-
point) accuracy of each evaluated network.

Model | Weights | Top-1 Acc. (%)
---|---|---
(# Mappable Layers) |  | CIFAR-10
---
| CIFAR-100
---
| CIFAR-10
---
| CIFAR-100
---
ResNet8 (10) | 77,360 | 77,968 | 87.38 | 59.40
ResNet20 (20) | 268,336 | 270,522 | 90.52 | 65.15
MobileNet (15) | 3,195,328 | 3,206,282 | 90.15 | 63.05
VGG-16 (14) | 14,715,584 | 14,761,664 | 92.79 | 69.51
AlexNet (7) | 15,378,112 | 15,562,432 | 91.29 | 65.70

Trainer 3: The transfer of a layer from digital to analog in the modeling
environment usually has a large detrimental effect on accuracy. Nonetheless,
most of the accuracy degradation at the desired evaluation time, $t_{eval}$,
can be often countered with re-training. To this end, for each iteration, the
entire network (both digital and analog layers) undergoes retraining, until no
decrease in the loss function is observed inside a convergence window (see
Fig. 2b, the width of the convergence window is a hyper-parameter of our
framework).

Evaluator 4: The DNN model accuracy after re-training with the achieved
mapping is then compared to the user-specified drop threshold, i.e., the
maximum drop between the floating-point and the average validation accuracy at
the desired evaluation time. If the optimized model adheres to this
constraint, the tentative AIMC acceleration of the layer, is confirmed. On the
contrary, the optimization step is rolled back, and the layer is assigned for
digital execution. The optimization loop proceeds by considering a further
layer in its next iteration, and ends when all layers have been analyzed.

Full System Evaluation 5: After the achieved mapping has been determined,
performance and energy statistics are obtained using the ALPINE framework
[15], built on top of the cycle-accurate gem5-X simulator [16] and capable of
providing system-level performance statistics representative of AIMC-
accelerated systems.

##  IV Evaluation

###  IV-A Experimental Setup

The CIFAR-10 and CIFAR-100 datasets were chosen for evaluation. We collect
results using these datasets for the five DNN architectures targeting edge
devices as listed in Table I, where the size of the last FC layer is modified
according to the number of output classes (10 and 100, respectively). The
number of weights of each DNN ranges from 77,968 to 15,237,698, and hence
contains both under- and over-parameterized models for the tasks. Data is
split into training, validation, and testing sets. The validation dataset was
obtained by splitting the original training dataset into two separate
datasets, comprising 90% of training images and the remaining 10% of
validation images. The same randomly sampled validation set was used for all
experiments.

Figure 3: (a-e) The MAC ratio (as a percentage) for all configurations between
the number of MACs performed using AIMC and the total number of MACs. (f-j)
The corresponding test set accuracies at $t=$1d.

###  IV-B Training and Evaluation Strategies

####  IV-B1 Initial hyper-parameter exploration:

Our framework assumes that the input DNN is pre-trained in floating-point
precision. The selected models differ in depth, size, and complexity, and
hence require different hyperparameters to achieve optimal training
performance. We empirically determined two sets of hyper-parameters,
pertaining to the analog and digital domain, respectively. Then, these were
applied to all networks. For analog training, a learning rate of 0.024 and
momentum of 0.775 was used. In terms of digital training, we employed a
learning rate of 0.057 and a momentum of 0.867. A convergence window of 5
epochs was used for retraining. Further details on the adopted experimental
setups in the two domains are provided in the following.

####  IV-B2 Digital training:

Networks were trained using floating-point precision until the training loss
did not decrease for five consecutive epochs. The batch size was set to 256.
Stochastic Gradient Descent (SGD) with momentum was used, with cross-entropy
loss. A cosine-annealing learning rate scheduler was used.

####  IV-B3 Analog HWA re-training:

To perform analog HWA re-training, the weights of the trained digital networks
were employed as a starting point. The IBM AIHWKIT was used to inject noise
during forward propagations. We refer the reader to [6] for a comprehensive
tutorial on HWA training using IBM AIHWKIT. The InferenceRPUConfig() preset
was used with the PCMLikeNoiseModel phenomenological inference model. The
following additional modifications were made to the simulation configuration:
(i) biases are assumed to be digital (i.e., they are not encoded on the last
column of AIMC tiles). (ii) Channel- (i.e., column-) wise weight scaling,
which has a negligible performance impact, is performed, where weights are
mapped to a per-channel conductance range during learning after each mini-
batch. (iii) The size of each AIMC tile is assumed to be 256x256. (iv) Layers
that span across multiple AIMC tiles are assumed to be split evenly. (v)
During training, multiplicative Gaussian noise is applied to unit weight
values, with a standard deviation of $0.08$, extracted from hardware
measurements provided by the authors of [4].

####  IV-B4 Accuracy evaluation:

As AIMC hardware is inherently stochastic, multiple evaluation instances are
required to determine representative performance metrics. Consequently,
accuracy was measured 20 times and averaged across experiments to obtain the
values reported in Section IV-D. We considered five user-defined accuracy
thresholds of [0.5%, 1.0%, 3.0%, 5.0%, 10.0%]. Moreover, analog NVM devices
such as PCM are susceptible to temporal conductance drift. Since the behavior
of AIMC hardware can evolve over time, we explore in Section IV-F accuracy
metrics for AIMC accelerators programmed considering drift levels predicted at
[one second (i.e., $t=1s$), one minute (i.e., $t=60s$), one hour (i.e.,
$t=3600s$), one day (i.e., $t=86400s$)]. These are then evaluated at different
points in time, to assess the effect of a mismatch between predicted and
actual drift.

####  IV-B5 Full System Evaluation:

The target simulated system for this work is a high-performance edge processor
with an in-order ARM CPU core running at a frequency of 2.3GHz, with a 64KB
each L1 data and instruction caches, 1MB of L2 cache, and 8GB of 2400MHz DDR4
RAM. The software stack for the target DNNs includes the DNNs accelerated via
the Eigen C++ library, running in user-space atop the Linux 5.4 kernel, and an
Ubuntu 16.04 LTS disk image. Each implementation resulting from a _LionHeart_
mapping is run for 10 inferences to reduce simulation noise. We note that
measured deviations among different runs of the same mappings never exceeded
4% of the run-time.

###  IV-C Baselines

_LionHeart_ (in all plots, referred to as LH) is evaluated by comparing the
achieved accuracy and performance with respect to three baselines: (i) _Fully-
digital_. The floating-point DNN model input of our methodology, where none of
its layers are accelerated using AIMC. It achieves the highest accuracy, but
presents the longest run-time. (ii) _Fully-analog_. A DNN implementation with
all FC and CONV layers executed using AIMC. This baseline represents the
opposite of the fully-digital: it maximizes performance, while potentially
resulting in very large accuracy degradation. (iii) _ First-Last Mapping
Strategy (FLMS)_. The SOTA hybrid mapping strategy [11]. In these mappings,
only the first and the last layers are implemented in digital PUs, while all
others are AIMC-accelerated.

###  IV-D Accuracy Evaluation

For five different drop thresholds and baseline configurations, we determined
the ratio between MACs performed using AIMC and the total number of MACs (Fig.
3). As can be observed, for all networks, the _All Digital_ configuration has
the highest test set accuracy and the _All Analog_ configuration has the
lowest test set accuracy. The behavior of the _FLMS_ configuration, however,
is unpredictable. The proposed _LionHeart_ configurations exhibit a gradual
transition from the largest to the smallest test set accuracy and a
corresponding decrease in the digital/analog MAC ratio between all
configurations. It hence enables the exploration of the performance/accuracy
space exposed by AIMC acceleration, while controlling accuracy to a user-
defined constraint.

For 62% of configurations, the last FC layer is retained in analog. However,
for only 18%, the first CONV layer is retained. This means that the first
layer is usually the most sensitive. Intermediary layers are observed to be
more sensitive than the last layer. For intermediary layers, no strong trend
or indication is observed as to whether a layer will remain analog, other than
that all FC ones are more likely to be retained as digital layers. Hence, it
is likely, for CONV layers, that whether or not a layer will be retained in
analog is largely dependent on the trained weights and _not_ on other layers’
characteristics. Our analyses demonstrate that properly mapping layers to
digital or analog resources is non-trivial, showcasing the benefit of our
framework in comparison with more naïve strategies such as _FLMS_.

###  IV-E Performance Evaluation

TABLE II:  Architectural parameters for full system simulation.  CPU Model | 1x In-Order @ 2.3GHz
---|---
ISA | ARMv8
Supply Voltage VDD | 1.3V
L1 Data/Instruction Cache | 64kB
L2 Cache | 1MB
Memory | 8GB DDR4 @ 2400MHz
AIMC Tile Latency | 100ns
AIMC Tile Data Transfer Bandwidth | 4GB/s
AIMC Tile Energy Efficiency | 20 TOp/s/W
Figure 4: (a) Speedup and (b) energy efficiency gain resulting from different
application mapping strategies, when performing an inference on benchmark
DNNs. Data is normalized with respect to a fully digital execution with no
AIMC acceleration.

In _LionHeart_ , the workload associated with DL models and their layers is
assumed to be architecture-agnostic, i.e., the ratio of analog MAC to total
MAC operations is used as a proxy for speedup and energy efficiency. To
quantify how strongly the MAC ratio correlates to these metrics, we evaluate
them for a simulated system featuring a single-core in-order CPU clocked at
2.3GHz and a two-level cache hierarchy. The architecture embeds tightly-
coupled AIMCs, governed by dedicated extensions to the ARM instruction set
supported by the CPU. It is modeled using the ALPINE framework [15], allowing
the full system simulation of AIMC-accelerated systems. Table II lists the
adopted architectural parameters, which mirror the ones considered in [15] for
the single-core, higher-performance scenario.

Due to the L1 and L2 cache sizes (rendering AIMC-acceleration of smaller
networks ineffective), results were gathered on the three largest DNNs among
the ones presented in Section IV-D, considering the more challenging CIFAR-100
dataset. We simulated the execution of an entire inference, including
computations not pertaining to FC nor CONV layers. We investigated
heterogeneous mappings derived by _LionHeart_ , with varying maximum accuracy
degradation thresholds. For comparison, we also conducted experiments where
all FC and CONV layers are completely executed on AIMCs, and mappings where,
similar to [11], all layers except the first and last are hosted on analog
crossbars (_FLMS_).

The complete simulation outcomes of the system, presented in Fig. 4, highlight
that the achievable speed-ups are highly dependent on the DNN structure. In
particular, even when all layers are computed in the analog domain, which is
an upper bound for speedup and efficiency in our setting, only a speedup of
24% can be reached in the MobileNet benchmark. Such (comparatively) low run-
time performance improvement is caused by the lower degree of data reuse of
the depthwise separable convolutions employed in this network, which shift the
balance between computation and memory access requirements toward the latter,
limiting the benefit of accelerating MACs. Nonetheless, even in this case,
_LionHeart_ , with a limited maximum accuracy degradation of 5%, still
captures almost all of the available speedup and energy efficiency gains.

Figure 5: The impact of temporal drift when varying $t_{eval}$ in [0, 86400]
for the ResNet8 network evaluated on CIFAR-100.

AlexNet and VGG16, which do not employ separable convolutions, present instead
more potential speedup (660% in the full-analog case for VGG16, 550% for
AlexNet), due to the higher locality of the employed computations (which leads
to less cache misses and memory-induced stalls). Even in those cases,
_LionHeart_ can harness a large part of the potential speedup if enough
accuracy leeway is allowed. For a 5% accuracy degradation, a 425% speedup was
measured in AlexNet and a 285% one in VGG16. AlexNet proved to be the most
robust of the investigated networks. In fact, for a 10% accuracy degradation
threshold, its performance exceeded that of the FLMS baseline, as all layers
can be executed in AIMCs, further showcasing the benefit of adopting a
flexible accuracy-driven approach. Finally, it can be noticed that system-
level energy gains closely follow speedups. Indeed, AIMCs tiles are not a
major contributor to overall system power (as opposed to processors and
memories). Instead, they shorten run-time, ultimately lowering energy
requirements. This is achieved both by allowing the execution of MVMs in
constant time and by encoding DNN weights in the crossbar structure itself,
hence easing the pressure on the memory hierarchy.

###  IV-F Temporal Drift and Evaluation Time Analysis

We study the impact of varying the evaluation time, $t_{eval}$, used during
analog HWA retraining on the simulated inference time of the system. In Fig.
5, these metrics are reported for all configurations (except _All Digital_ ,
which does not have a temporal dimension) using the ResNet8 network and
CIFAR-100 dataset. For both sets of mappings, mostly unique intermediate
layers are retained in analog. Due to the (relatively) small drop threshold,
the first and last layers for each are in digital. It can be observed that the
test accuracy decreases a function of the drift time. Moreover, as the
proportion of analog MACs increases, the robustness to drift decreases. Most
notably, for the _All Analog_ configuration, independent of the evaluation
time, the test accuracy decreases significantly (¿10%). For all other
configurations, no significant degradation is observed until $t$¿1h. However,
the temporal dependence of the test set accuracy is effected. Unsurprisingly,
when a larger evaluation time is specified, the initial test set accuracy (at
$t$=0) is also larger. While varied, for most configurations the robustness to
drift is observed to improve as the evaluation time is increased. This is
significant because, when our methodology is used, a hard constraint on the
average behavior of the drop in accuracy can be enforced at a given drift
(evaluation) time. Additionally, the test set accuracy for the _FLMS_
configuration is consistently lower than for all _LionHeart_ configurations.

##  V Conclusion

In this paper, we presented _LionHeart_ , an accuracy-driven framework for
mapping DL workloads on hybrid analog-digital systems. The effectiveness of
our method was demonstrated by deploying five different DNNs trained for image
classification to heterogeneous PUs. Both the MAC ratio between analog and
digital operations and the effect of temporal drift were considered to enforce
varying user-specified constraints. Finally, the performance evaluation was
carried out using a simulated ARMv8-based processor with tightly coupled AIMC
tiles.

##  VI Acknowledgements

This work was supported by the European Commission under H2020 grant WiPLASH
(GA 863337), the EC H2020 FVLLMONTI project (GA No. 101016776), and by the
ACCESS – AI Chip Center for Emerging Smart Systems, sponsored by InnoHK
funding, Hong Kong SAR. We also thank the IBM Research Analog AI team for
their feedback and acknowledge computational support from AiMOS, an AI
supercomputer made available by the IBM Research AI Hardware Center and
Rensselaer Polytechnic Institute’s Center for Computational Innovations (CCI).

## References

  * [1] Abu Sebastian et al.  Memory Devices and Applications for In-Memory Computing.  Nat. Nanotechnology, 15(7):529–544, 2020.
  * [2] Malte J. Rasch et al.  Hardware-Aware Training for Large-Scale and Diverse Deep Learning Inference Workloads Using In-Memory Computing-Based Accelerators.  Nat. Communications, 14(1), 2023.
  * [3] Hadjer Benmeziane et al.  AnalogNAS: A neural network design framework for accurate inference with analog in-memory computing.  In 2023 IEEE International Conference on Edge Computing and Communications (EDGE), 2023.
  * [4] Manuel Le Gallo et al.  64-Core Mixed-Signal In-Memory Compute Chip Based on Phase-Change Memory for Deep Neural Network Inference.  Nature Electronics, 6(9):680–693, Jul 2023.
  * [5] Malte J. Rasch et al.  A Flexible and Fast PyTorch Toolkit for Simulating Training and Inference on Analog Crossbar Arrays.  In 2021 IEEE 3rd International Conference on Artificial Intelligence Circuits and Systems (AICAS), 2021.
  * [6] Manuel Le Gallo et al.  Using the IBM Analog In-Memory Hardware Acceleration Kit for Neural Network Training and Inference, 2023.
  * [7] Aayush Ankit et al.  PUMA.  In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, 2019.
  * [8] Simei Yang et al.  AERO: Design Space Exploration Framework for Resource-Constrained CNN Mapping on Tile-Based Accelerators.  IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 12(2):508–521, 2022.
  * [9] Linghao Song, Xuehai Qian, Hai Helen Li, and Yiran Chen.  PipeLayer: A Pipelined ReRAM-Based Accelerator for Deep Learning.  In 2017 IEEE International Symposium on High Performance Computer Architecture (HPCA), 2017.
  * [10] Hai Jin et al.  ReHy: A ReRAM-Based Digital/Analog Hybrid PIM Architecture for Accelerating CNN Training.  IEEE Trans. on Parallel and Distributed Systems, 2021.
  * [11] Kodai Ueyoshi et al.  DIANA: An End-to-End Energy-Efficient Digital and ANAlog Hybrid Neural Network SoC.  In 2022 IEEE International Solid- State Circuits Conference (ISSCC), 2022.
  * [12] Payman Behnam, Uday Kamal, and S. Mukhopadhyay.  An Algorithm-Hardware Co-design Framework to Overcome Imperfections of Mixed-signal DNN Accelerators.  ArXiv, abs/2208.13896, 2022.
  * [13] Matteo Risso, Alessio Burrello, Giuseppe Maria Sarda, Luca Benini, Enrico Macii, Massimo Poncino, Marian Verhelst, and Daniele Jahier Pagliari.  Precision-aware latency and energy balancing on multi-accelerator platforms for dnn inference.  In 2023 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), 2023.
  * [14] Marco Rios et al.  Bit-Line Computing for CNN Accelerators Co-Design in Edge AI Inference.  IEEE Trans. on Emerging Topics in Computing, 2023.
  * [15] Joshua Klein et al.  ALPINE: Analog In-Memory Acceleration With Tight Processor Integration for Deep Learning.  IEEE Trans. on Computers, 72(7), 2023.
  * [16] Yasir Mahmood Qureshi et al.  Gem5-X: A Gem5-Based System Level Simulation Framework to Optimize Many-Core Platforms.  In 2019 Spring Simulation Conference (SpringSim). IEEE, 2019.

  *[ML]: Machine Learning
  *[DL]: Deep Learning
  *[DNNs]: Deep Neural Network
  *[SOTA]: State-Of-The-Art
  *[MAC]: Multiply-Accumulate
  *[FC]: Fully Connected
  *[CONV]: Convolutional
  *[DNN]: Deep Neural Network
  *[PUs]: Processing Unit
  *[AIMC]: Analog In-Memory Computing
  *[IMC]: In-Memory Computing
  *[HWA]: Hardware-Aware
  *[MVMs]: Matrix-Vector Multiplication
  *[MVM]: Matrix-Vector Multiplication
  *[AIMCs]: Analog In-Memory Computing
  *[NVM]: Non-Volatile Memory
  *[PCM]: Phase Change Memory
  *[ADC]: Analog-to-Digital Converter
  *[DAC]: Digital-to-Analog Converter
  *[WL]: Word-Line
  *[RRAM]: Resistive Random-Access Memory
  *[MACs]: Multiply-Accumulate
  *[SGD]: Stochastic Gradient Descent
  *[FLMS]: First-Last Mapping Strategy