# Integration of speech separation, diarization, and recognition for multi-
speaker meetings: System description, Comparison, and Analysis

###### Abstract

Multi-speaker speech recognition of unsegmented recordings has diverse
applications such as meeting transcription and automatic subtitle generation.
With technical advances in systems dealing with speech separation, speaker
diarization, and automatic speech recognition (ASR) in the last decade, it has
become possible to build pipelines that achieve reasonable error rates on this
task. In this paper, we propose an end-to-end modular system for the LibriCSS
meeting data, which combines independently trained separation, diarization,
and recognition components, in that order. We study the effect of different
state-of-the-art methods at each stage of the pipeline, and report results
using task-specific metrics like SDR and DER, as well as downstream WER.
Experiments indicate that the problem of overlapping speech for diarization
and ASR can be effectively mitigated with the presence of a well-trained
separation module. Our best system achieves a speaker-attributed WER of 12.7%,
which is close to that of a non-overlapping ASR.

Index Terms—  Speech separation, diarization, speech recognition, multi-
speaker

##  1 Introduction

Fig. 1: An overview of our proposed approach, compared with the CHiME-6
pipeline [1] and the CSS pipeline [2].

Multi-speaker speech recognition is defined as a task wherein, given a long
unsegmented recording consisting of a conversation between an unknown number
of participants, the expected output contains “who said what and when.”
Although the last two decades have seen incredible leaps in speech technology
— deep learning systems have matched human parity in the single-speaker
conversational speech settings [3] — multi-speaker speech processing remains a
challenge as such recordings may contain almost 20% overlapped speech [4, 5].
This scenario has implications for both diarization and automatic speech
recognition (ASR) — single-speaker diarization systems miss the interfering
speaker completely, and ASR systems trained on clean utterances are more
error-prone on overlapped regions. These factors, along with the effect of
far-field acoustic conditions, may increase the speaker-attributed word error
rates by up to 86% [6] in meetings. To tackle such problems, recent editions
of DIHARD [7] and CHiME [1] have focused on these tasks in very challenging
settings.

A straightforward approach for solving the overlap problem in multi-speaker
conversations is through speech separation. Deep learning based separation
methods have been progressing consistently over the last few years — a signal-
to-distortion ratio of 19.0 dB has been obtained on the popular WSJ0-2mix
dataset [8], making the enhanced signals almost indistinguishable from clean
utterances [9]. However, these techniques are often evaluated on short, fully
overlapping (and often simulated) mixtures, and do not measure distortions
introduced in overlap-free regions. They may also make unrealistic
assumptions, such as prior knowledge of the number of speakers in the mixture.
Recently, there have been efforts towards “continuous speech separation,”
which situates separation techniques in more realistic settings of long-form
conversations containing partially overlapped speech [10, 2], such as those
found in multi-speaker conversations.

There is increasing interest towards combining the advances made in speech
separation, diarization, and ASR for tackling multi-speaker speech
recognition. The CHiME-6 challenge [1] included a track aimed at recognizing
unsegmented dinner-party conversations recorded on multiple microphone arrays,
and baseline evaluations indicated that replacing oracle segments with
diarization outputs could degrade downstream WERs by almost 52%. Chen et al.
[2] proposed the new LibriCSS meeting dataset, and combined speaker-
independent continuous speech separation [10] with a strong hybrid ASR [11] to
recognize long unsegmented meeting recordings. Similar datasets such as
LibriMix [12] have been created to satisfy the need for multi-speaker
conversational data containing partial overlaps. Joint modeling of speaker
counting, speaker identification, and ASR using end-to-end models has also
been investigated [13, 14].

Nevertheless, the number of available methods to perform separation,
diarization, or recognition are plenty — each with their own sets of
requirements and assumptions. Together with the several possibilities of
combining them, this creates a daunting task to analyze systems that recognize
unsegmented multi-speaker recordings. In this paper, we make a first attempt
at tackling this problem, using the LibriCSS dataset. We propose a novel end-
to-end modular pipeline that performs separation, diarization, and
recognition, in that order. For each of these modules, we compare several
existing methods, describing the advantages and disadvantages of each
alternative. We also demonstrate the impact of separation on downstream tasks
by presenting corresponding results on the original mixed recording. Since the
modules only interact at the I/O level, they can be implemented using
different tools and optimized independently. Our best system with and without
separation achieves a concatenated minimum-permutation WER (cpWER) [1] of
12.7% and 23.9%, respectively, on the LibriCSS evaluation set. We will release
code to reproduce (and extend) our pipeline here:
https://desh2608.github.io/pages/jsalt/.

##  2 System Overview

Our pipeline consists of three components: Separation $\rightarrow$
Diarization $\rightarrow$ ASR. An unsegmented multichannel audio recording is
provided as input to a continuous speech separation system, such as [10]. The
separation module works on small windowed segments containing at most 2 or 3
speakers, and produces the corresponding number of separated audio streams,
which are passed to the diarization module. Since a speaker may have been
split into different streams across different windows, the diarization is
performed by considering all the audio streams simultaneously. We will discuss
the implications of this requirement on different diarization methods in
Section 4. After diarization, the single-speaker homogenenous segments are fed
into an ASR decoder.

Fig. 1 shows our proposed approach, and situates it in comparison with
alternative approaches proposed in [1] and [2], which we refer to as the
CHiME-6 pipeline and the CSS pipeline, respectively. All 3 methods perform
explicit separation, as opposed to an implicit “separation+recognition”
usually performed by target-speaker ASR methods [15]. Stacking the components
in different orders elicits unique advantages and limitations for each
approach. Since diarization is performed first in the CHiME-6 pipeline, it
makes it possible to use the source activity pattern derived from the
diarization output to guide the estimation of the mixture model parameters in
guided source separation (GSS) [16], or use the speaker information to perform
target speaker extraction, as in SpeakerBeam [17]. However, poor diarization
can have significant impacts on separation results in this pipeline. The CSS
pipeline performs separation in the first stage, followed by ASR and
diarization, allowing the system to be deployed in a streaming setting. The
diarization performance may itself also benefit from reduced false alarms,
since segments are obtained from the ASR output. However, it makes speaker-
biasing infeasible at the ASR stage. Furthermore, the ASR output may contain
spurious insertions resulting from cross-talk in non-overlap regions of the
separated audio streams.

In contrast, our pipeline makes it feasible to employ a speaker-biased ASR,
and also to remove extra insertions through simple post-processing of the
diarization output (cf. Section 3.3). Unlike the CHiME-6 pipeline, our
separation is not dependent on the performance of the diarizer, and speaker-
independent separation techniques can be effectively used in the pipeline.
Here we demonstrate two of these — mask-based minimum variance distortionless
response (MVDR), and sequential multi-frame separation. Furthermore, since the
diarization stage sees only single-speaker recordings, it circumvents the need
for an overlap-aware diarization component.

We will now describe the performance of different methods in each module –
namely separation, diarization, and ASR — in Sections 3, 4, and 5,
respectively. All our experiments are conducted on LibriCSS [2], which
consists of multi-channel audio recordings of “simulated conversations,”
generated by mixing Librispeech utterances [18]. It comprises 10,
approximately one-hour long, sessions. Each session is made up of six
10-minute-long “mini sessions” that have different overlap ratios, ranging
from 0% to 40%. The recordings were made in a regular meeting room by using a
seven-channel circular microphone array. We used session 0 as the development
set, and the remaining 9 sessions for evaluation (which we use to report
results). Diarization results are reported using diarization error rate (DER),
and ASR performance is evaluated in terms of cpWER. It is computed by
concatenating all the utterances of a speaker in the reference and hypothesis,
scoring all speaker pairs, and then finding the speaker permutation that
minimizes the total WER.

##  3 Speech Separation

###  3.1 Mask-based MVDR

The speech separation module follows the continuous speech separation scheme
[19], consisting of three steps. First, the recording is uniformly segmented
into smaller chunks with overlap. For each chunk, a multi-channel separation
network [2], trained with permutation invariant training (PIT) criterion, is
applied to estimate three time-frequency (TF) masks: two for speech sources
and one for noise. In this scheme, we used a chunk size of 2.4s with 0.8s hop.

Thereafter, a stitching algorithm [20] is used to track local permutations and
glue the masks from each chunk into the meeting-wise mask. This stitching is
performed using mask similarity between adjacent chunks on the overlapped
region (which is 1.6s in our case), by finding the permutation that has
minimum distance between chunks. Based on the resulting permutation, the
chunk-wise masks are connected to obtain a mask stream for the long recording.
We averaged the overlap region between chunks for this process.

Finally, given the stitched masks for the entire meeting, a mask-based
adaptive MVDR beamforming [21] is performed to get the final separation
result. Since the noise is mostly stationary, we used the noise mask for the
entire meeting to estimate the noise spatial covariance.

Note that the number of output channels in each local separation is highly
correlated with the chunk size. As [21] suggests, most 2.4s chunks contains at
most 2 speakers, therefore two output channels suffice for local processing.
However, when the chunk size is larger (as in the next separation model), it
may contain more than 2 speakers, even though LibriCSS contains at most
2-speaker overlaps.

###  3.2 Sequential multi-frame separation

We also used a novel sequential separation system which has multiple mask-
based beamforming steps [22]. A mask-based multi-frame multi-channel Wiener
filter (MCWF) beamformer is used. We used three untied sequential steps inside
the model by feeding previously beamformed signals into the next mask-
prediction step. The output of this model can be the last mask-network output
or the one previous beamformed output. This model was trained using 10 second
long mixtures of three speaker utterances from Libri-Light database [23].
During training, the utterances were mixed on-the-fly with room impulse
responses obtained using an image-method based room simulator that simulates
data acquisition in shoe-box shaped rooms with an 8-microphone array [22].
This pre-trained model was used to separate sources in the LibriCSS dev and
eval datasets using 8 second long blocks with 4 seconds overlap. Since
LibriCSS has 7 microphones, we added another microphone signal by shifting the
first microphone signal with one sample and adding white Gaussian noise with
variance 1e-6. The source estimate outputs were stitched using a stitching
loss of magnitude STFT domain mean-squared error between time-domain signals
to resolve the permutation across blocks. This is similar to the stitching
method described in Section 3.1.

###  3.3 Experimental results

Table 1 shows the performance of the separation methods on the LibriCSS eval
set. Separation performance is calculated on a simulated eval set different
from LibriCSS since reference signals are required. Since the models work with
varying window sizes, we first mapped the output tracks to the $N$ source
tracks for every 0.8s block1110.8s is the highest common factor between the
chunk size of the models., where $N$ is the number of participating speakers,
using magnitude-domain distance and assuming at most two active speakers per
0.8s block, and then generated $N$ speaker tracks from the separated outputs.
We call this process “oracle track mapping”. We report separation performance
in terms of average meeting-level SDR [24]. Sequential multi-frame model
achieved a higher SDR of 14.1 dB as compared to the mask-based MVDR’s 5.8 dB,
since implemented MVDR beamformer does not attempt to reconstruct target
signals exactly.

We also evaluated the separated audio with a spectral clustering based
diarizer (Section 4.1) and a hybrid HMM-DNN ASR (Section 5.1), and the results
are shown in the table in terms of DER and cpWER, respectively. From the
table, we note that the diarization performance of the two methods were
comparable, but the 3-stream sequential multi-frame separation outperformed
the 2-stream mask-based MVDR in terms of cpWER results. Although the
sequential model was trained with an 8-microphone cubic geometry, it
generalized well to LibriCSS, which has a completely different geometry, since
the model is inherently geometry-independent.

Furthermore, a simple post-processing trick applied on the diarization output
was found to be useful for the sequential model – using this trick improved
the cpWER by 15.4% relative. For this post-processing, we removed a segment
from a stream if it was completely enclosed within a same-speaker segment in a
different stream. This is akin to filtering out cross-talk prior to ASR
decoding. This trick was not useful for the mask-based MVDR, providing only a
marginal improvement of 0.2% relative in terms of cpWER. Note that although
sequential model achieved significantly better SDR, their cpWER are similar.
This indicates the potential objective mismatch between speech separation and
recognition.

Table 1: Performance of separation methods on LibriCSS eval set in terms of
resulting downstream diarization (using spectral clustering) and ASR (using
TDNN-F model) results. For comparison, we also show results obtained on a “no
separation” baseline. Separation performance is reported on a simulated eval
set. †MVDR beamformer does not attempt to reconstruct the target signal at the
reference microphone directly, so the separation metric reflects this
mismatch.

Method SDR (dB) DER (%) cpWER (%) No separation - 18.3 31.0 Mask-based MVDR
5.8† 13.9 22.8 Sequential multi-frame 14.1 14.1 19.3

##  4 Speaker Diarization

We can categorize diarization methods based on whether or not they can assign
overlapping speaker segments. Since our pipeline separates the recording prior
to diarization, overlap-awareness is not a strict requirement. At the same
time, since the separation is done on windowed segments, the diarization needs
to be performed across audio streams (since a speaker can be present in
different streams at different times). Based on these conditions, we selected
the following diarization methods for our study.

###  4.1 X-vector + clustering

This method consists of a speech activity detection (SAD) component followed
by clustering of small subsegment embeddings. We used a similar SAD as that
described in [1], consisting of a TDNN-Stats based classifier with Viterbi
decoding for inference. The speech segments were divided into subsegments with
a window size of 1.5s and a stride of 0.75s, and 128-dimensional embeddings
were extracted using an x-vector extractor [25] trained on VoxCeleb data [26]
with simulated room impulse response [27]. We conducted experiments with 3
variants of clustering: (i) agglomerative hierarchical clustering (AHC) on
PLDA scores [28], (ii) spectral clustering (SC) on cosine similarity [29], and
(iii) VBx clustering initialized from the AHC system [30]. For these methods,
we did not make any assumptions about the number of speakers in the recording.
We used the same PLDA trained on Librispeech for both AHC and VBx, and did not
use the PLDA interpolation technique from [30].

###  4.2 Region proposal networks (RPN)

This is a supervised method, which combines the segmentation and embedding
extraction steps into a single neural network and jointly optimizes them [31].
The region embeddings are then clustered (using K-means clustering) and a non-
maximal suppression is applied. We trained the RPN on simulated meeting-style
recordings with partial overlaps generated using utterances from the
Librispeech [18] training set. Since we used K-means clustering, we assumed
that the oracle number of speakers for each recording is known.

###  4.3 Target-speaker voice activity detection (TS-VAD)

The TS-VAD model takes conventional speech features (e.g., MFCC) along with
i-vectors for each speaker as inputs and produces frame-level activities for
each speaker using a neural network with a set of binary classification output
layers [32]. Since the number of these binary output nodes are fixed for
training, we assumed that the maximum possible number of speakers in any
session is at most 8. The initial estimates for the speaker i-vectors were
obtained using the SC system. For training, we created simulated meeting-style
data similar to that used for training the RPN model.

###  4.4 Experimental results

We conducted experiments with “mixed” as well as “separated” audio to analyze
the impact of separation on diarization performance. For the mixed recording,
we selected the first channel as our input. For evaluation on separated
streams, we fixed the separation component as mask-based MVDR, and the ASR was
chosen to be a speaker-biased hybrid HMM-DNN (described in Section 5.1). Table
2 shows the diarization performance on mixed LibriCSS, with a breakdown by
overlap condition. The SAD error for the clustering-based systems was 4.8%. It
is immediately evident that assigning overlapping speech is important to
perform well on this task – both RPN and TS-VAD outperformed clustering-based
methods. Even on low overlap regions, there is a significant difference, and
we conjecture that this arises from a mismatch between the training data for
the PLDA used for scoring the AHC and VBx models, and the evaluation set.
Since SC uses cosine scoring, it performed better than the other clustering
methods. For the RPN and TS-VAD systems, creating a simulated mixture which
closely resembled the test set was found to be important, and using cepstral
mean normalization (CMN) was critical for this performance.

Table 2: Diarization performance on mixed LibriCSS evaluation set, in terms of
% DER. 0S and 0L refer to 0% overlap with short and long inter-utterance
silences, respectively. RPN and TS-VAD methods assume that oracle number of
speakers is known. We also report the corresponding cpWER using a TDNN-F based
ASR model. Using an oracle diarization output results in a cpWER of 23.1%.

Method Overlap ratio in % DER cpWER 0L 0S 10 20 30 40 AHC 16.1 12.0 16.9 23.6
28.3 33.2 22.6 36.7 VBx 14.6 11.1 14.3 21.5 25.4 31.2 20.5 33.4 SC 10.9 9.5
13.9 18.9 23.7 27.4 18.3 31.0 RPN 4.5 9.1 8.3 6.7 11.6 14.2 9.5 27.2 TS-VAD
6.0 4.6 6.6 7.3 10.3 9.5 7.6 24.4

Table 3: Comparison of diarization and downstream ASR performance of different
methods on separated audio streams for LibriCSS evaluation set. We used mask-
based MVDR for separation and TDNN-F based ASR for these experiments.

Separation Metric AHC VBx SC RPN TS-VAD ✗ DER 22.6 20.5 18.3 9.5 7.6 ✗ cpWER
36.7 33.4 31.0 27.2 24.4 ✓ DER 37.5 - 13.9 22.4 31.7 ✓ cpWER 86.7 - 22.8 34.9
47.7

Next, we evaluated the systems on separated audio streams obtained using the
mask-based MVDR method. Table 3 shows these results, along with the downstream
cpWER obtained using a hybrid HMM-DNN ASR model. Since AHC and SC perform
subsegment-level clustering, they can be naturally extended to diarization
across streams. VBx, on the other hand, estimates speaker changes through HMM
state transitions, so it is not directly applicable to this scenario. RPN and
TS-VAD can also be extended to this new setting, since RPN uses clustering of
region embeddings, and TS-VAD predicts frame-level speaker activity based on
the corresponding i-vectors. Our first observation is that among clustering-
based methods, SC performed significantly better than AHC, likely because the
PLDA used for AHC was trained on clean Librispeech utterances, which are
acoustically very different from the separated audio. To verify this, we
performed the AHC also on a cosine similarity matrix, and it resulted in an
absolute DER improvement of 13.8%. For RPN and TS-VAD, performance on mixed
recording was not indicative of results on separated audio. Without any post-
processing, we obtained a DER of 26.9% using RPN. This improved to 22.4% on
filtering out non-speech segments using our SAD from Section 4.1. Similarly,
TS-VAD performance degraded severely on going from mixed to separated audio.
We found this degradation to be consistent for missed speech, false alarms,
and speaker confusions, indicating a likely mismatch in train vs. test
conditions. Consequently, the cpWER for both RPN and TS-VAD was found to be
significantly higher than that for SC.

##  5 Speech Recognition

We conducted our ASR experiments on a hybrid TDNN-F based, and a Transformer-
based end-to-end ASR model. Pretrained models (trained on Librispeech) for
both of these are publicly available, enabling reproducibility of our results.

Table 4: ASR performance on mixed LibriCSS evaluation set, in terms of % WER.
These results correspond to the “utterance-wise evaluation” of [2], since we
used oracle segments. The TDNN-F and Transformer E2E models obtain WERs of
8.8% and 5.5% (beam size 60), respectively, on the Librispeech test-other
evaluation set.

Model Overlap ratio in % Average 0L 0S 10 20 30 40 TDNN-F (base) 16.1 16.0
27.7 39.1 49.4 58.3 36.8 \+ fine-tuned 11.4 11.4 18.2 26.3 33.7 40.7 25.2
Transformer E2E (beam = 5) 5.5 5.6 13.1 21.6 31.1 41.6 21.6 Transformer E2E
(beam = 30) 5.1 5.3 11.4 19.5 28.5 38.3 19.7

Table 5: ASR performance on separated LibriCSS evaluation set, in terms of %
cpWER. These results correspond to the “continuous-input evaluation” of [2].
Mask-based MVDR and spectral clustering were used for separation and
diarization, respectively.

Model Without separation (DER = 18.3%) With separation (DER = 13.9%) 0L 0S
OV10 OV20 OV30 OV40 Average 0L 0S OV10 OV20 OV30 OV40 Average TDNN-F 17.6 19.1
24.4 33.0 38.4 45.2 31.0 16.9 15.5 18.8 22.7 26.6 29.4 22.3 Transformer 12.4
14.1 20.2 29.5 35.3 41.9 27.1 11.2 8.5 10.6 14.8 15.5 17.5 13.4

###  5.1 TDNN-F based hybrid HMM-DNN

Following the Kaldi [33] Librispeech recipe, we trained a 17-layer deep neural
network consisting of factored TDNN layers [34] using the lattice-free MMI
objective [35]. We used 40-dim MFCC features, and additionally appended
100-dim i-vectors estimated online. The model was trained on the 960h
Librispeech data with 3x speed perturbation. We call this our base model. On
the Librispeech test-clean and test-other evaluation sets, this model obtains
WERs of 3.8% and 8.8%, respectively. We additionally fine-tuned it for 1 epoch
on Librispeech train set augmented with simulated room impulse responses [27]
to match the acoustic conditions of mixed LibriCSS recordings; this is
referred to as the fine-tuned model. For decoding on LibriCSS, we used the
3-gram language model provided with the Librispeech data, and the lattices
were rescored using a pruned TDNN-LSTM based RNNLM trained on the training
transcripts [36]. We used a 2-pass decoding strategy, where the i-vectors were
re-estimated from the non-silence regions for the second pass [37]. The fine-
tuned model was used to evaluate the downstream ASR performance of the
separation and diarization methods.

###  5.2 Transformer-based end-to-end ASR

Our end-to-end (E2E) ASR is a state-of-the-art ESPNet-based [38] Transformer
encoder-decoder model [39]. It was trained on the 960h Librispeech corpus with
SpecAugment [40], using 83-dim log-mel filterbank features with pitch. The
encoder consists of 2 convolutional layers and 12 self-attention blocks, and
the decoder contains 6 self-attention blocks. The training loss jointly
minimizes sequence-to-sequence (S2S) and connectionist temporal classification
(CTC) objectives [41]. The decoder predicts subword units generated using
SentencePiece [42]. For decoding, we used beam search which combines scores
from the S2S, CTC, and an external Transformer-based language model. This ASR
model obtains WERs of 2.2% and 5.5% on the Librispeech test-clean and test-
other evaluation sets, respectively, using a beam size of 60.

###  5.3 Experimental results

Similar to our diarization experiments, we evaluated the ASR models on mixed
and separated audio. Table 4 shows the results obtained on the mixed LibriCSS
data. For these experiments, we used oracle segments and speaker information.
From the table, we see that a strong Librispeech model also performed well on
mixed LibriCSS utterances. Among the TDNNF-based hybrid models, fine tuning on
reverberated data provided a 31.5% relative WER improvement. For transformer-
based E2E models, decoding with larger beams (of size 30) improved WER by
almost 10% relative, compared to decoding with smaller beam sizes. We did not
get any significant gains by increasing beam sizes further, so we used this
setting for further experiments.

Next, we present ASR results in the context of our pipeline, i.e., using
separated audio streams from the mask-based MVDR model and segments from
spectral clustering based diarization. We used the fine-tuned variant of the
hybrid HMM-DNN ASR model. Additionally, to emphasize the importance of the
separation module, we also show cpWERs obtained on mixed recordings. The
results are shown in Table 5. On replacing oracle segments with those obtained
from diarization, the downstream cpWERs for hybrid and E2E ASR systems
degraded by 23.4% and 37.6%, respectively. Interestingly, this increase was
more prominent in the low and medium overlap conditions, which suggests that
errors in these cases occured primarily from incorrect speaker assignment. On
applying separation, the cpWERs improved significantly (although this was
partly due to better diarization). In particular, we found that the E2E model
benefited more from separated audio streams, with its cpWER improving from
27.1% to 13.4%. For both the models, the cpWER with separation was found to
outperform the corresponding results on mixed recording even using oracle
segments.

Finally, with all our experimental evaluations in place, we combined the best
performing models at each stage of the pipeline. We prepared two variants: (a)
without separation, and (b) with separation. For (a), we selected TS-VAD based
diarization and the Transformer-based ASR model. Pipeline (b) consists of
sequential multi-frame separation, followed by SC-based diarization and the
same ASR. We present the final results for both variants in Table 6. It is
evident that in the absence of an explicit separation module, even a well
performing diarization and ASR combo is hamstrung. We discuss some more
implications of these results in the next section.

Table 6: Pipeline variants: (a) without separation, and (b) with separation,
combining best models at each stage.

Separation Diarization ASR DER(%) cpWER(%) (a) - TS-VAD Transformer 7.6 23.9
(b) Sequential SC Transformer 14.1 12.7

##  6 Discussion

Our extensive experiments using a variety of separation, diarization, and ASR
methods elicit important lessons for solving the multi-speaker speech
recognition problem. First, the importance of explicit separation cannot be
overstated — the best cpWER for a pipeline with separation is 46.9% relative
better than one without it (Table 6). For separation, we found training and
inference with larger chunks to perform better; there is a caveat, however —
the improved performance is obtained through diarization tricks that can
filter out the increased cross-talk with these models. Second, we observed
that although new (supervised) diarization methods like RPN and TS-VAD provide
substantial gains on mixed recordings, the performance does not carry over
well to separated audio streams, and traditional clustering approaches can
still outperform them in these settings (Table 3). In particular, the best
diarization without separation was found to be 45.8% relative better than one
with separation, which is highly counter-intuitive. This is particularly
relevant for pipelines such as ours and the CSS pipeline, where separation is
performed before diarization, and we regard this as an important direction for
further investigation. There have also been concurrent efforts to apply
diarization on separated audio streams [43]. Finally, unlike diarization, ASR
performance on mixed recordings was strongly indicative of the performance on
separated audio streams. We found that a state-of-the-art ASR model trained on
clean, single speaker utterances integrates well in the pipeline and results
in the best available cpWER on this dataset. This is encouraging, especially
in view of recent advances in end-to-end speech recognition.

Acknowledgment. The work reported here was started at JSALT 2020 at JHU, with
support from Microsoft, Amazon, and Google.

## References

  * [1] Shinji Watanabe, Michael Mandel, Jon Barker, and Emmanuel Vincent,  “CHiME-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings,”  ArXiv, vol. abs/2004.09249, 2020.
  * [2] Zhuo Chen, Takuya Yoshioka, Liang Lu, Tianyan Zhou, Zhong Meng, Yi Luo, J. Wu, and Jinyu Li,  “Continuous speech separation: Dataset and analysis,”  ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7284–7288, 2020.
  * [3] Wayne Xiong, Jasha Droppo, Xuedong Huang, Frank Seide, Mike Seltzer, Andreas Stolcke, Dong Yu, and Geoffrey Zweig,  “Achieving human parity in conversational speech recognition,”  ArXiv, vol. abs/1610.05256, 2016.
  * [4] Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Maël Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska Masson, Iain McCowan, Wilfried Post, Dennis Reidsma, and Pierre Wellner,  “The AMI meeting corpus: A pre-announcement,”  in MLMI, 2005.
  * [5] Elizabeth Shriberg, Andreas Stolcke, and Don Baron,  “Observations on overlap: findings and implications for automatic processing of multi-party conversation,”  in INTERSPEECH, 2001.
  * [6] Takuya Yoshioka, Dimitrios Dimitriadis, Andreas Stolcke, William Hinthorn, Zhuo Chen, Michael Zeng, and Xuedong Huang,  “Meeting transcription using asynchronous distant microphones,”  in INTERSPEECH, 2019.
  * [7] Neville Ryant, Kenneth Ward Church, Christopher Cieri, Alejandrina Cristia, Jun Du, Sriram Ganapathy, and Mark Liberman,  “The second DIHARD diarization challenge: Dataset, task, and baselines,”  ArXiv, vol. abs/1906.07839, 2019.
  * [8] Yi Luo, Zhuo Chen, and Takuya Yoshioka,  “Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation,”  ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 46–50, 2020.
  * [9] Yi Luo and Nima Mesgarani,  “Conv-TasNet: Surpassing ideal time?frequency magnitude masking for speech separation,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, pp. 1256–1266, 2019.
  * [10] Takuya Yoshioka, Zhuo Chen, Changliang Liu, Xiong Xiao, Hakan Erdogan, and Dimitrios Dimitriadis,  “Low-latency speaker-independent continuous speech separation,”  ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6980–6984, 2019.
  * [11] Liang Lu, Xiong Xiao, Zhuo Chen, and Yifan Gong,  “Pykaldi2: Yet another speech toolkit based on Kaldi and PyTorch,”  ArXiv, vol. abs/1907.05955, 2019.
  * [12] Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vincent,  “Librimix: An open-source dataset for generalizable speech separation.,”  arXiv: Audio and Speech Processing, 2020.
  * [13] Naoyuki Kanda, Yashesh Gaur, Xiaofei Wang, Zhong Meng, Zhuo Chen, Tianyan Zhou, and Takuya Yoshioka,  “Joint speaker counting, speech recognition, and speaker identification for overlapped speech of any number of speakers,”  ArXiv, vol. abs/2006.10930, 2020.
  * [14] Naoyuki Kanda, Xuankai Chang, Yashesh Gaur, Xiaofei Wang, Zhong Meng, Zhuo Chen, and Takuya Yoshioka,  “Investigation of end-to-end speaker-attributed ASR for continuous multi-talker recordings,”  ArXiv, vol. abs/2008.04546, 2020.
  * [15] Marc Delcroix, Kateřina Žmolíková, Keisuke Kinoshita, Atsunori Ogawa, and Tomohiro Nakatani,  “Single channel target speaker extraction and recognition with speaker beam,”  2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5554–5558, 2018.
  * [16] Christoph Böddeker, Jens Heitkaemper, Joerg Schmalenstroeer, Lukas Drude, Jahn Heymann, and Reinhold Haeb-Umbach,  “Front-end processing for the CHiME-5 dinner party scenario,”  in INTERSPEECH 2018, 2018.
  * [17] Kateřina Žmolíková, Marc Delcroix, Keisuke Kinoshita, Tsubasa Ochiai, Tomohiro Nakatani, Lukás Burget, and Jan Cernocký,  “Speakerbeam: Speaker aware neural network for target speaker extraction in speech mixtures,”  IEEE Journal of Selected Topics in Signal Processing, vol. 13, pp. 800–814, 2019.
  * [18] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: An ASR corpus based on public domain audio books,”  2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206–5210, 2015.
  * [19] Takuya Yoshioka, Hakan Erdogan, Zhuo Chen, and Fil Alleva,  “Multi-microphone neural speech separation for far-field multi-talker speech recognition,”  2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5739–5743, 2018.
  * [20] Morten Kolbæk, Dong Yu, Zheng-Hua Tan, and Jesper Jensen,  “Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 10, pp. 1901–1913, 2017.
  * [21] Takuya Yoshioka, Igor Abramovski, Cem Aksoylar, Zhuo Chen, Moshe Ben David, Dimitrios Dimitriadis, Yifan Gong, Ilya Gurvich, Xuedong Huang, Yanping Huang, Aviv Hurvitz, Li Jiang, Sharon Koubi, Eyal Krupka, Ido Leichter, Changliang Liu, Partha Parthasarathy, Alon Vinnikov, Lingfeng Wu, Xiong Xiao, Wayne Xiong, Huaming Wang, Zhenghao Wang, Jingjing Zhang, Yong Zhao, and Tianyan Zhou,  “Advances in online audio-visual meeting transcription,”  2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 276–283, 2019.
  * [22] Zhong-Qiu Wang, Hakan Erdogan, Scott Wisdom, Kevin Wilson, Desh Raj, Shinji Watanabe, Zhuo Chen, and John R. Hershey,  “Sequential multi-frame neural beamforming for speech separation and enhancement,”  in Submitted to IEEE SLT 2021, 2021.
  * [23] Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, Tatiana Likhomanenko, Gabriel Synnaeve, Armand Joulin, Abdel rahman Mohamed, and Emmanuel Dupoux,  “Libri-light: A benchmark for asr with limited or no supervision,”  ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669–7673, 2020.
  * [24] Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte,  “Performance measurement in blind audio source separation,”  IEEE transactions on audio, speech, and language processing, vol. 14, no. 4, pp. 1462–1469, 2006.
  * [25] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur,  “X-vectors: Robust DNN embeddings for speaker recognition,”  2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5329–5333, 2018.
  * [26] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman,  “Voxceleb: A large-scale speaker identification dataset,”  ArXiv, vol. abs/1706.08612, 2017.
  * [27] Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L. Seltzer, and Sanjeev Khudanpur,  “A study on data augmentation of reverberant speech for robust speech recognition,”  2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5220–5224, 2017.
  * [28] Daniel Garcia-Romero, David Snyder, Gregory Sell, Daniel Povey, and Alan McCree,  “Speaker diarization using deep neural network embeddings,”  2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4930–4934, 2017.
  * [29] T. Park, Kyu J. Han, Manoj Kumar, and Shrikanth S. Narayanan,  “Auto-tuning spectral clustering for speaker diarization using normalized maximum eigengap,”  IEEE Signal Processing Letters, vol. 27, pp. 381–385, 2020.
  * [30] Mireia Diez, Lukás Burget, Shuai Wang, Johan Rohdin, and Jan Cernocký,  “Bayesian HMM based x-vector clustering for speaker diarization,”  in INTERSPEECH, 2019.
  * [31] Zili Huang, Shinji Watanabe, Yusuke Fujita, Paola García, Yiwen Shao, Daniel Povey, and Sanjeev Khudanpur,  “Speaker diarization with region proposal network,”  ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6514–6518, 2020.
  * [32] Ivan Medennikov, Maxim Korenevsky, Tatiana Prisyach, Yuri Y. Khokhlov, Mariya Korenevskaya, Ivan Sorokin, Tatiana V. Timofeeva, Anton Mitrofanov, Andrei Andrusenko, Ivan Podluzhny, Aleksandr Laptev, and Aleksei Romanenko,  “Target-speaker voice activity detection: a novel approach for multi-speaker diarization in a dinner party scenario,”  ArXiv, vol. abs/2005.07272, 2020.
  * [33] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukás Burget, Ondrej Glembek, Nagendra Kumar Goel, Mirko Hannemann, Petr Motlícek, Yanmin Qian, Petr Schwarz, Jan Silovský, Georg Stemmer, and Karel Veselý,  “The Kaldi speech recognition toolkit,”  2011\.
  * [34] Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohammadi, and Sanjeev Khudanpur,  “Semi-orthogonal low-rank matrix factorization for deep neural networks,”  in INTERSPEECH, 2018.
  * [35] Daniel Povey, Vijayaditya Peddinti, Daniel Galvez, Pegah Ghahremani, Vimal Manohar, Xingyu Na, Yiming Wang, and Sanjeev Khudanpur,  “Purely sequence-trained neural networks for ASR based on lattice-free MMI,”  in INTERSPEECH, 2016.
  * [36] Hainan Xu, Tongfei Chen, Dongji Gao, Yiming Wang, Ke Li, Nagendra Goel, Yishay Carmiel, Daniel Povey, and Sanjeev Khudanpur,  “A pruned RNNLM lattice-rescoring algorithm for automatic speech recognition,”  2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5929–5933, 2018.
  * [37] Vimal Manohar, Szu-Jui Chen, Zhiqi Wang, Yusuke Fujita, Shinji Watanabe, and Sanjeev Khudanpur,  “Acoustic modeling for overlapping speech recognition: JHU CHiME-5 challenge system,”  ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6665–6669, 2019.
  * [38] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai,  “Espnet: End-to-end speech processing toolkit,”  ArXiv, vol. abs/1804.00015, 2018.
  * [39] Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, et al.,  “A comparative study on transformer vs RNN in speech applications,”  in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 449–456.
  * [40] Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  in INTERSPEECH, 2019.
  * [41] Suyoun Kim, Takaaki Hori, and Shinji Watanabe,  “Joint CTC-attention based end-to-end speech recognition using multi-task learning,”  2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4835–4839, 2017.
  * [42] Taku Kudo and John Richardson,  “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,”  in EMNLP, 2018.
  * [43] Xiong Xiao, Naoyuki Kanda, Z. Chen, Tianyan Zhou, T. Yoshioka, Sanyuan Chen, Yong Zhao, Gang Liu, Y. Wu, J. Wu, Shujie Liu, Jinyu Li, and Yifan Gong,  “Microsoft speaker diarization system for the voxceleb speaker recognition challenge 2020,”  ArXiv, vol. abs/2010.11458, 2020.


utf8

# Challenges in Translation of Emotions in Multilingual User-Generated
Content: Twitter as a Case Study

Hadeel Saadany
Centre for Translation Studies
University of Surrey

&Constantin Orăsan
Centre for Translation Studies
University of Surrey

&Rocío Caro Quintana
RGCL
University of Wolverhampton

Félix do Carmo
Centre for Translation Studies
University of Surrey

&Leonardo Zilio
Centre for Translation Studies
University of Surrey
<EMAIL_ADDRESS>
###### Abstract

Although emotions are universal concepts, transferring the different shades of
emotion from one language to another may not always be straightforward for
human translators, let alone for machine translation systems. Moreover, the
cognitive states are established by verbal explanations of experience which is
shaped by both the verbal and cultural contexts. There are a number of verbal
contexts where expression of emotions constitutes the pivotal component of the
message. This is particularly true for User-Generated Content (UGC) which can
be in the form of a review of a product or a service, a tweet, or a social
media post. Recently, it has become common practice for multilingual websites
such as Twitter to provide an automatic translation of UGC to reach out to
their linguistically diverse users. In such scenarios, the process of
translating the user’s emotion is entirely automatic with no human
intervention, neither for post-editing nor for accuracy checking. In this
research, we assess whether automatic translation tools can be a successful
real-life utility in transferring emotion in user-generated multilingual data
such as tweets. We show that there are linguistic phenomena specific of
Twitter data that pose a challenge in translation of emotions in different
languages. We summarise these challenges in a list of linguistic features and
show how frequent these features are in different language pairs. We also
assess the capacity of commonly used methods for evaluating the performance of
an MT system with respect to the preservation of emotion in the source text.

##  1 Introduction

Despite the tremendous improvement in the quality of automatic translation as
a result of the use of Neural Machine Translation (NMT) systems, NMT output
can still contain errors. This is particularly noticeable with UGC such as
tweets which do not follow common lexico-grammatical standards. Despite this
limitation, NMT systems are commonly used in multilingual platforms such as
Twitter to provide its users with an idea of global views or emotions towards
current events or public figures. In such scenarios, the component of the
tweet that conveys the emotion is often pivotal to the understanding of the
tweet’s message. There have been different studies which explored how far
sentiment information can be captured from the machine-translated text
[Demirtas and Pechenizkiy, 2013, Shalunts et al., 2016, Mohammad et al., 2016,
Barhoumi et al., 2018]. The objective of most research in this area, however,
is from a sentiment classification perspective rather than a translation
accuracy perspective. In these studies, the role of automatic translation of a
language into English is simply to help in the sentiment classification of
that language, since the available English sentiment classification systems
can be directly applied to the target text [Afli et al., 2017, Salameh et al.,
2015, Abdalla and Hirst, 2017, Araujo et al., 2016].

The research presented in this paper, however, evaluates the preservation of
the sentiment message in translation not as a sentiment classification task
but from a translation accuracy perspective. We assess how far automatic
translation tools can be a successful real-life utility in transferring not
only the sentiment polarity within tweets, but also fine-grained emotions such
as anger and joy. The starting point is our work on translating Arabic UGC
into English [Saadany and Orasan, 2020]. Analysis of sentiment mistranslations
introduced by Google Translate when translating book reviews, revealed typical
errors related to five linguistic phenomena: contronyms, diacritics, idiomatic
expressions, dialectical code-switching, and negation. We have shown that the
sentiment polarity of Arabic UGC is frequently flipped in the MT system output
due to one of these five phenomena. In the current paper, we aim to
investigate to what extent similar errors can be identified in the translation
of tweets within various language pairs. We also go one step further by
analysing the transfer of fine-grained emotions such as joy, anger, and fear
by NMT systems. To achieve this, we carry out an analysis of datasets of
tweets automatically translated into different language pairs. At the end of
this analysis, we attempt to provide answers to the following questions:

  1. 1.

Are there specific linguistic features of tweets that can lead to
mistranslation of emotions?

  2. 2.

Do these features affect different languages equally?

  3. 3.

Can traditional automatic quality measures adequately evaluate the translation
of sentiment?

To answer the above research questions, this paper is divided as follows:
section 2 presents our data compiling and experiment for evaluating the
translation of emotion in tweets by MT systems. Section 3 analyses challenging
features for the translation of emotions in multilingual tweets. It also
provides a qualitative analysis of each feature based on its frequency in our
compiled dataset and its prominence in each of the source language explored.
In section 4, we evaluate the efficacy of the MT automatic quality metrics in
assessing the mistranlation of emotion within the multilingual UGC framework.
Section 5 presents a conclusion on the our experiment, limitations of the
present study and recommendations for future research work.

##  2 Data Collection and Experiment Setup

In order to check how far automatic translation captures the specific emotion
in tweets, we replicated a real-life scenario where MT systems are utilised
spontaneously to translate the content of tweets. Twitter currently supports
built-in translations, so users can click on a Translate Tweet prompt visible
directly under the tweet text to translate it. Twitter mentions that it
employs Google Translate for this service. To evaluate how far the MT system
in this scenario can serve as a real-life utility, we used Google Translate
API to automatically translate compiled multilingual Twitter datasets
previously annotated for four emotions (joy, fear, aggression, and anger). It
is important to note that these four emotions were chosen as representative of
the common fine-grained sentiments expressed in tweets. The authors of tweets
are usually either happy, angry, or fearful of something or someone, and their
anger can either be aggressive or passive. The datasets were collated from
different emotion-detection and aggression-detection shared tasks [Mohammad
and Bravo-Marquez, 2017, Mohammad and Kiritchenko, 2018, Basile et al., 2019,
Zampieri et al., 2020]. The source datasets amounted to approximately 30,000
tweets in three languages: English, Arabic, and Spanish.

We created two datasets from this source annotated data. The first dataset was
created by translating the Arabic and Spanish source datasets into English.
The second dataset was created by translating part of the source English
dataset into Romanian, Arabic, Spanish and Portuguese. These datasets were
used to extract instances for our analysis. The next stage in our experiment
was to extract instances in which the MT system failed to translate the
emotion correctly. We call this failure “mistranslation of emotion” and it is
identified by the discrepancy between the annotations of emotion in the source
dataset and the emotions classified in the translated tweets. For example, if
the original tweet is annotated as conveying ‘anger’ but the classifier
predicted ‘joy’ for the translation, this pair was considered a potential
mistranslation of emotion and was selected for analysis.

To get the classifications of emotions in the translated tweets, we built a
classifier by fine-tuning a Roberta XML model [Liu et al., 2019] on the
previously annotated 23,000 source English tweets. This data was pre-processed
by deletion of punctuation, non-alphanumeric symbols, lemmatisation, and
lower-casing. We also used the Demoji111https://pypi.org/project/demoji/
Python library to transfer the emojis into their equivalent lexicon (e.g.
[1.5] is translated into “dislike”). We tested the accuracy of the classifier
against a test set of these tweets, and its accuracy was 92%. The English
classifier was used to predict the emotion of the Google API output for the
translation of the Arabic and Spanish dataset into English and the back
translation of the English tweets translated into the other languages. The
classifier’s predicted emotion was compared to the gold standard emotion of
the source text, instances of discrepancy were extracted as potential
mistranslations of emotions.

##  3 Analysis of Challenging Features for the Translation of Emotions in
Tweets

To check the reasons for discrepancy between the predicted emotion and the
emotion of the source text, a team of MT researchers, native speakers of the
respective languages, conducted a manual analysis on samples of the extracted
potential mistranslations. The extracted samples for English to other
languages amounted to 1600 tweets divided equally among the four target
languages, and from the opposite direction, with English as a target language,
it amounted to $\approx 3000$ tweets divided between Arabic and Spanish as a
source language. The disagreements in the dataset due to mistranslation of
emotions are presented in Figure 1. Spanish has clearly fewer cases of
discrepant emotions in tweets, both when these are translated into English
($\approx 8$%) and when they are translations from English ($\approx 27$%).
Target languages like Romanian (61%) and Arabic (41%) show a much higher
percentage of tweets with mistranslated emotions. It is obvious from the
analysed sample that some languages are more privileged than others in the
real-life scenario we replicate for our experiment.

Figure 1: Frequency of Mistranslation of Emotion in the Analysed Dataset

Next, we decided to look in detail to linguistic features in the source tweets
that could help explain the mistranslation of emotion. Based on an initial
analysis of the sample dataset and previous studies of the language of tweets,
we selected six features that represent special challenges in transferring
emotions by the MT engine: hashtags, slang, non-standard orthography,
idiomatic expressions, polysemy, and grammar (negation structures). The
following sections demonstrate the effect of these features on the translation
of emotion with illustrative examples . Table 1 presents a summary of our
findings, which are discussed next.

Language Pair | Hashtags | Slang | Polysemy | Idiomatic Expressions | Grammar | Orthography
---|---|---|---|---|---|---
EN-SP | 44% | 14% | 7.9% | 6.3% | 12.6% | 14%
EN-PT | 41.6% | 16.6 | 2.7% | 8.3% | 13.8% | 16.6%
EN-AR | 25.6% | 20.7% | 24.3% | 12% | 6% | 11%
EN-RO | 24.6% | 26% | 18.6% | 12% | 6% | 12.6%
AR-EN |  | 60% | 11% | 7.9% | 6.7% | 13.9%
SP-EN |  | 32.5% | 16% | 16.5% | 12% | 22.6%

Table 1: Frequency of Language Features per Language Pair

###  3.1 Hashtags

Emotions in tweets are expressed in a special style in line with Twitter’s
orthographic limitations and peculiarities. Thus, for example, authors of
tweets frequently express their emotion as a trailing hashtag or a hashtagged
non sequitur to a neutral or an ironic statement. The emotion of the tweets in
such cases is retrieved solely from the hashtag. Our analysis has shown that
this unique style of emotion transfer constitutes a challenge to the MT
system. When the hashtags expressing emotion are either untranslated or
mistranslated, the emotion expressed in the message is completely distorted.
For example, the fear emotion in the English tweet “Just waved daughter and
her friend off to school, #terrifying!” is entirely missed in the Arabic
translation “ لقد لوحت ابنتها وصديقتها الصغيرة للتو إلى المدرسة ¿
#terrifying!” as the hashtag that carries the main emotional content is not
translated. Moreover, the hashtagged word in tweets is often written in non-
standard orthography which causes the MT to output the hashtagged word as is
without translation. For example, the anger emotion against customer service
in the tweet “I asked for my parcel to be delivered to a pickup store not my
address #poorcustomerservice” is missed in the Romanian translation “Am cerut
livrarea coletului meu la un magazin de preluare, nu adresa mea
#poorcustomerservice” as the hashtagged word is not translated. The MT treats
such hashtags as out-of-vocabulary words and hence misses the affective
message. The distortion of emotion is also caused by a wrong translation of
the hashtagged word. The anger emotion of the English tweet “CNN’s Wolf
Blitzer calls you an American astronaut and you don’t correct him
#disappointed” is completely lost in the Spanish translation as the hashtag is
mistranslated to “diseñado” meaning ‘designed’ instead of disappointed. The
Spanish translation carries a neutral emotion.

###  3.2 Slang and Dialectical Expressions

Research studies have shown that slang and dialectical expressions present
several challenges to MT in general [Zbib et al., 2012]. Tweets are
characterised by the wealth of slang expressions and the code-switching
between different dialects of one language based on the authors’ demographics.
It was observed from the manual analysis of the sample data that this
stylistic quirk often distorts the translation of emotion in the source text.
For example, the Spanish tweet “Ni en pedo, bueno en pedo si” is mistranslated
in English as “not even fart good fart yes”. The correct translation of the
expression “ni en pedo” is ‘no way’. The source tweet expressed a humorous
comment which should read “No way. Well, yes way”. In this example, the MT
online engine provides an incomprehensible output due to a mistranslation of
the dialectical version of the Spanish expression “ni en pedo” used mainly in
Argentina, and therefore the emotion of the source text is completely lost.
Similarly, the MT system fails to detect the aggression in the English tweet
“The iconic nigger tweet” when it is translated to Romanian as “tweet-ul
iconic negru” (The iconic black tweet). The slang expression in the source
tweet (nigger) carries the aggressive tone and hence the neutral translation
(black) misses the aggressive emotion. The amount of distortion of the
affective messages due to a mistranslation of slang or dialectical expressions
varies from one language to another. It was observed that Arabic dialectical
expressions posed a significant challenge to the MT system as it caused a
distortion of emotion in 60% of the Arabic tweets in the second dataset (see
Table 1). For example, commenting on an event in the Middle East, a tweeter
expresses joy “ايه كمية الانشكاح ديه¿” (What all this amount of happiness!).
The MT system gives the exact opposite emotion “What all this amount of
anger!”. This owes to the fact that the dialectical expression “الانشكاح¿”
(happiness) is mistranslated as “anger”. The dialectical tweets were mostly
mistranslated in aggressive Arabic tweets. For example, bullying a female
football player, a tweet says “جايه بفستانها وتستلم جايزة افضل لاعبة خربوا
الكورة الحريم¿” (She is coming with a dress to receive the best player prize,
women ruined football). The tweet is written in a Gulf dialect that was
mistranslated by the MT engine as “come to her dress and receive the prize for
the best player who ruined the harem?”. The MT output not only misses the
aggressive tone but also lacks semantic and grammatical coherence.

###  3.3 Non-standard Orthography

With its 280-character limit, Twitter users often resort to creative
abbreviations and unconventional orthography. Moreover, linguists have
observed that to encourage speed and immediacy of understanding, Twitter users
type in the same way they speak [Ian, 2010]. The manual analysis has shown
that this specific linguistic phenomenon is a major culprit in a wrong
transfer of the emotion within different language pairs. For example, the MT
output of the English tweet “watching sad bts video bc im sad. iwannacryy”
renders an incomprehensible affect message in Portuguese (assistindo ao vídeo
do sad bts bc im sad. iwannacryy). The reason is that the microblogging
limitation causes the author of the tweet to use a creative word shortening by
eliminating spaces “iwannacryy” as well as by texting in acronyms (“bc”
meaning “because”, “im” meaning “I am”). The affective message is missed in
the Portuguese translation as all these emotional nuanced orthographic forms
remain untranslated. Another complication is that tweeters are more apt to use
expressive lengthening to communicate strong emotion. These non-standard
emotional expressions are usually treated as out-of-vocabulary by MT systems
with all the language pairs the research team analysed. For instance, the
anger in the Spanish tweet “Por que sos re chantaaaa” (Why are you such a
liar?) is not transferred by the MT translation “Why are you chantaaaa” as the
Spanish word “chanta” (liar) passes for out-of-vocabulary lexicon because of
elongation.

###  3.4 Idiomatic Expressions

One of the challenging issues in the field of translation is the process of
translating the different shades of meaning conveyed by an idiom [Al Mubarak,
2017]. The reason is that translating idioms usually involves meta-linguistic
information such as cultural and social norms. Because of its informal nature,
conversational idioms are used extensively in tweets. The manual analysis has
shown that a large number of idioms were literally translated, which did not
only affect the sentiment preservation of the source text, but often produced
nonsensical target text. For example, the Arabic tweet expressing happiness
towards one particular public figure “والله دمه خفيف¿” has the idiomatic
expression “دمه خفيف¿”, meaning “funny”. The tweet should read ‘By God, he is
so funny’, but the MT output gives a literal translation, “By God, his blood
is so light” which was predicted as having an ‘anger’ sentiment by the
classifier. The same problem also exists in language pairs with English as a
source language. For example, an ‘angry’ tweet commenting on the recent
American presidential elections “We have to keep u in line” has the idiomatic
expressions “keep in line” meaning to discipline uncontrolled behaviour. This
idiom was literally translated in Arabic as “في الطابور¿” (stay in the queue)
and in Spanish as “mantenerte en línea” (stay fit). The literal translation of
the idiom in the two language pairs flips the emotion from anger to a neutral
sentiment.

###  3.5 Polysemy

MT research has shown that polysemous words pose a challenge to MT systems
when the contextual information is not clearly determined [Akhobadze, 2019].
Due to its micro-blogging nature, polysemous words in tweets are usually
lacking context, which adds to their ambiguity. The manual analysis of the
translated data has revealed that this linguistic phenomenon distorts the
tweeter’s emotional message. One example is the aggressive English tweet “the
girl sitting in front of me is chewing her gum like a cow; I’m ready to snap”.
The word snap here has the informal meaning of “burst in anger”. The Romanian
translation by the MT system, however, reflects a joy emotion as it gives the
other meaning of snap “take pictures”. Hence the MT Romanian output reads “the
girl in front of me chews her gum like a cow; I’m ready to take pictures”.
Another more extreme example appears with the Arabic to English pair.
Commenting on a Middle East political crisis, an aggressive tweeter threatens
two Gulf countries “جايكم الدور خلينا نربيكم في اليمن ونربي قطر¿” (Your turn
will come, Yemen and Qatar, we will teach you a lesson). The aggressive threat
is lost in the MT output “Come on let’s educate Yemen and Qatar”. This is due
to a mistranslation of the polysemous word “نربي¿” which could either
positively mean “educate” or to negatively mean “teach a lesson” by inflicting
punishment.

###  3.6 Grammar (Negation)

The analysis has shown that the distortion of the source emotion was also
associated with a wrong translation of a negation marker between different
language pairs. For example, negation was found to cause a problem when the
source text is in dialectical Arabic. The lexico-grammatical realisation of
negation differs between the Standard and dialectical Arabic as well as
between its different dialects. Arabic dialects often treat negative particles
as clitics, and hence a letter is added to the stem of the word to change it
to negative [Mohamed et al., 2012]. The MT engine frequently either missed the
Arabic dialectical negation and hence flips the phrase to an opposite
sentiment pole or mistranslates the negated phrase altogether. For example,
commenting on a terrorist attack, a tweeter angrily states “لا سامح الله من
افتخر بقنبلة دمرت مئات المنازل¿” (May God not forgive (punish) the one who is
proud of a bomb destroying hundreds of homes). The negation is missed and
hence the online translation tool output reads “May God forgive that one who
is proud of a bomb destroying hundreds of homes”. The emotion of the tweeter
in the Arabic translation is flipped from anger to sympathy towards a
terrorist attack. If automatic translation is used to spot potential terrorist
trends on social media platforms, this type of error would affect the accuracy
of the algorithm and may bring dangerous consequences to users. The analysis
has also shown that missing negation structures in the English tweets also
distorts the emotion. For example, the fear emotion in the English tweet “A
trip to the dentist never gets easier” is flipped to joy in the Portuguese
translation because of a wrong translation of the negative structure. The MT
output in Portuguese is “Uma ida ao dentista nunca foi tão fácil” meaning ‘A
trip to the dentist has never been easier’.

##  4 Measuring the transfer of sentiment

From the analysis of these language features, it can be observed that using
automatic translation tools for translating emotion in multilingual UGC such
as tweets involves several linguistic challenges. Despite these challenges,
NMT systems such as Google Translate are extensively utilised by social media
platforms without human post-editing. In the research environment, the
reliability of MT systems is commonly determined by automatic quality metrics
that are domain agnostic as they evaluate the translation accuracy regardless
of the type of source text. In this section we touch upon the issue of
evaluating the translation of sentiment-oriented text by MT engines. We will
show that the standard quality metrics for MT evaluation fall short of their
goal in capturing mistranslations of emotions in special types of text such as
Twitter data.

The de facto standard for MT performance evaluation is the BLEU score with its
different variations [Papineni et al., 2002]. BLEU gives equal penalty weight
to inaccurate translation of n-grams, which may lead to performance
overestimation (or underestimation). For example, the “joy” emotion in a tweet
from our Arabic dataset (“What is this amount of happiness, I don’t
understand!”) is flipped to anger by Twitter’s Google Translate tool which
outputs “What is this amount of anger, I don’t understand!”. Despite the
distortion of the affective message, BLEU only mildly penalises the swapping
of the two opposite emotive nouns ‘happiness’ and ‘terrible’ and this
translation receives a BLEU score of 0.76. The reason is that BLEU gauges the
performance of an MT model by an indiscriminate n-gram matching, regardless of
the semantic weight of each word. By human standards, the MT performance in
such cases is highly over-estimated.

There have been numerous efforts to address the common pitfalls of n-gram-
based metrics by incorporating semantic and contextual features in metrics
such as METEOR [Banerjee and Lavie, 2005]. Although the introduction of more
semantically oriented metrics showed a better correlation with human
judgement, the estimation of sentiment preservation in UGC has not yet been
addressed. When it comes to evaluating an MT system performance in
transferring emotion, even the semantically oriented automatic metrics do not
give a penalty to a mistranslated sentiment proportional to the distortion it
afflicts on the source message. For example, the negation in the Arabic tweet
“May God do not forgive those who put you in power” is missed in the MT
output, “May God forgive the one who put you in power”. The emotion is flipped
from anger to joy. Despite the distortion of the emotion, the mistranslation
receives a METEOR score 0.61.

For this last analysis, we selected an evaluation dataset consisting of 100
tweets extracted from the dataset that was classified by the research team as
having a mistranslation of emotion. The tweets in this dataset were chosen in
a way where the main error in the translation is the distortion of emotion due
to one of the six linguistic features discussed in the previous sections. The
evaluation dataset was translated by native speakers in the research team. We
calculated both BLEU and METEOR segment scores of the machine-translated
tweets against the human references. The average BLEU and METEOR scores per
segment is approximately 0.60 and 0.45 respectively, with a standard deviation
of 0.2 for BLEU and 0.1 for METEOR. These high values show that the
replacement of a sentiment by its opposite in a translated tweet, as we had in
this evaluation dataset, is not captured by lexical similarity metrics. This
calls for an improvement of evaluation metrics, introducing mechanisms more
sensitive to distortion of emotion in UGC data such as tweets.

##  5 Limitations and Future Work

In this research, we evaluated the capacity of an MT online system in
translating the features of tweets that convey emotions. Our analysis
highlighted a number of linguistic challenges involved in translating emotion
between different languages. Future research may point to other types of
features that may prove significant for other language pairs. Moreover, using
NMT tools in translating raw text such as sentiment-oriented tweets may prove
counterproductive, as it may provide users with an affective message that is
different or even opposite to the source emotion. Future research may test
whether preprocessing of raw UGC such as Twitter data can improve the MT
performance in transferring the correct emotion.

We also touched upon the reliability of automatic quality measures for
evaluating MT systems performance in transferring emotion. We have shown that
the standard evaluation measures were not able to give a penalty proportional
to the incorrect translation of emotion in a sample dataset of mistranslated
tweets. We believe that evaluating the performance of MT systems in
translating sentiment-oriented text is an under-recognised problem in MT
research. Future work should address the possibility of introducing a
sentiment measure to reflect how far the MT system transfers the correct
affective message in the source text.

## References

  * [Abdalla and Hirst, 2017]  Mohamed Abdalla and Graeme Hirst.  2017\.  Cross-lingual sentiment analysis without (good) translation.  arXiv preprint arXiv:1707.01626.
  * [Afli et al., 2017]  Haithem Afli, Sorcha Maguire, and Andy Way.  2017\.  Sentiment translation for low resourced languages: Experiments on Irish general election tweets.  In 18th International Conference on Computational Linguistics and Intelligent Text Processing, Budapest, Hungry, pages 17–21.
  * [Akhobadze, 2019]  Babulia Akhobadze.  2019\.  Polysemy in machine translation exemplified in english and georgian.  Bull. Georg. Natl. Acad. Sci, 13(1).
  * [Al Mubarak, 2017]  Amin Ali Al Mubarak.  2017\.  The challenges of translating idioms from arabic into english a closer look at al imam al mahdi university–sudan.  International Journal of Comparative Literature and Translation Studies, 5(1):53–64.
  * [Araujo et al., 2016]  Matheus Araujo, Julio Reis, Adriano Pereira, and Fabricio Benevenuto.  2016\.  An evaluation of machine translation for multilingual sentence-level sentiment analysis.  In Proceedings of the 31st Annual ACM Symposium on Applied Computing, pages 1140–1145.
  * [Banerjee and Lavie, 2005]  Satanjeev Banerjee and Alon Lavie.  2005\.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
  * [Barhoumi et al., 2018]  Amira Barhoumi, Chafik Aloulou, Nathalie Camelin, Yannick Estève, and Lamia Belguith.  2018\.  Arabic sentiment analysis: an empirical study of machine translation’s impact.  In Proceedings of the second Conference on Language Processing and Knowledge Management Kerkennah (Sfax), Tunisia.
  * [Basile et al., 2019]  Valerio Basile, Cristina Bosco, Elisabetta Fersini, Nozza Debora, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, Manuela Sanguinetti, et al.  2019\.  Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter.  In 13th International Workshop on Semantic Evaluation, pages 54–63. Association for Computational Linguistics.
  * [Demirtas and Pechenizkiy, 2013]  Erkin Demirtas and Mykola Pechenizkiy.  2013\.  Cross-lingual polarity detection with machine translation.  In Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining, pages 1–8.
  * [Ian, 2010]  Tucker Ian.  2010\.  Twitter spreads regional slang.  https://www.theguardian.com/science/2010/sep/05/tv-not-twitter-spreads-slang.
  * [Liu et al., 2019]  Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  2019\.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692.
  * [Mohamed et al., 2012]  Emad Mohamed, Behrang Mohit, and Kemal Oflazer.  2012\.  Transforming standard Arabic to colloquial Arabic.  In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 176–180.
  * [Mohammad and Bravo-Marquez, 2017]  Saif M Mohammad and Felipe Bravo-Marquez.  2017\.  Wassa-2017 shared task on emotion intensity.  arXiv preprint arXiv:1708.03700.
  * [Mohammad and Kiritchenko, 2018]  Saif Mohammad and Svetlana Kiritchenko.  2018\.  Understanding emotions: A dataset of tweets to study interactions between affect categories.  In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  * [Mohammad et al., 2016]  Saif M Mohammad, Mohammad Salameh, and Svetlana Kiritchenko.  2016\.  How translation alters sentiment.  Journal of Artificial Intelligence Research, 55:95–130.
  * [Papineni et al., 2002]  Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  2002\.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  * [Saadany and Orasan, 2020]  Hadeel Saadany and Constantin Orasan.  2020\.  Is it great or terrible? preserving sentiment in neural machine translation of arabic reviews.  In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 24–37.
  * [Salameh et al., 2015]  Mohammad Salameh, Saif Mohammad, and Svetlana Kiritchenko.  2015\.  Sentiment after translation: A case-study on Arabic social media posts.  In Proceedings of the 2015 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 767–777.
  * [Shalunts et al., 2016]  Gayane Shalunts, Gerhard Backfried, and Nicolas Commeignes.  2016\.  The impact of machine translation on sentiment analysis.  Data Analytics, 63:51–56.
  * [Zampieri et al., 2020]  Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin.  2020\.  Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020).  arXiv preprint arXiv:2006.07235.
  * [Zbib et al., 2012]  Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar Zaidan, and Chris Callison-Burch.  2012\.  Machine translation of arabic dialects.  In Proceedings of the 2012 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 49–59.