# Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual
Spoken Language Understanding

Yingmei Guo1111Work is done during internship at NLP Group, Microsoft STCA.
Linjun Shou2 Jian Pei3 Ming Gong2 Mingxing Xu1
Zhiyong Wu1 Daxin Jiang2222Corresponding author.
1Department of Computer Science and Technology, Tsinghua University
2NLP Group, Microsoft STCA
3School of Computing Science, Simon Fraser University
{guoym18@mails.,xumx<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Lack of training data presents a grand challenge to scaling out spoken
language understanding (SLU) to low-resource languages. Although various data
augmentation approaches have been proposed to synthesize training data in low-
resource target languages, the augmented data sets are often noisy, and thus
impede the performance of SLU models. In this paper we focus on mitigating
noise in augmented data. We develop a denoising training approach. Multiple
models are trained with data produced by various augmented methods. Those
models provide supervision signals to each other. The experimental results
show that our method outperforms the existing state of the art by 3.05 and
4.24 percentage points on two benchmark datasets, respectively. The code will
be made open sourced on github.

##  1 Introduction

Spoken language understanding (SLU) is a key component in task-oriented
dialogue systems. SLU consists of two subtasks: intent detection and slot
tagging (Wang et al., 2005; Tur and De Mori, 2011). Although promising
progress has been achieved on SLU in English (Liu and Lane, 2016; Peng et al.,
2020; Huang et al., 2020), those methods need large amounts of training data,
and thus cannot be applied to low-resource languages where zero or few
training data is available.

In this paper, we target at the extreme setting for cross-lingual SLU where no
labeled data in target languages is assumed, which is critical for industry
practice, since annotating a large SLU dataset with high quality for every
language is simply infeasible.

Existing cross-lingual transfer learning methods mainly build on pre-trained
cross-lingual word embeddings (Ruder et al., 2019) or contextual models (Wu
and Dredze, 2019; Huang et al., 2019; Lample and Conneau, 2019; Conneau et
al., 2020), which represent texts with similar meaning in different languages
close to each other in a shared vector space. Those approaches often show good
performance on intent detection. The results on slot tagging, however, are
often unsatisfactory, especially for distant languages, which are dramatically
different from English in scripts, morphology, or syntax Upadhyay et al.
(2018); Schuster et al. (2019); Li et al. (2020a).

Several studies (Schuster et al., 2019; Liu et al., 2020b; Li et al., 2020a)
show that adding translated data into the fine-tuning process of pretrained
models can improve the results of cross-lingual SLU substantially when no
golden-labeled training data in target languages is available. For example,
machine translation can be employed to translate the training data in English
into target languages, and some alignment methods, such as attention weights
(Schuster et al., 2019), fastalign (Dyer et al., 2013) or giza++ (Och and Ney,
2003), can be further applied to label the translated data. Another approach
to alleviate the problem of data scarcity is to automatically generate
training data. Recently, some methods for monolingual SLU (Anaby-Tavor et al.,
2020; Kumar et al., 2020; Zhao et al., 2019; Peng et al., 2020) automatically
label domain-specific data or use pre-trained language models to generate
additional data.

However, the synthesized training data derived from both the translation
approach and the data generation approach may be quite noisy and may contain
errors in label. For the translation approach, both the translation process
and the alignment process may generate errors Xu et al. (2020); Li et al.
(2020b). For the data generation approach, it is often hard to control a right
tradeoff between generating correct but less diverse data and generalizing
more diverse data but with more noise. Moreover, generating synthetic training
data across languages further adds challenges to the robustness of the
generation methods.

To filter out noise in the synthesized training data, a few methods are
proposed, such as the mask mechanism (Li et al., 2020a), the soft alignment
method (Xu et al., 2020), the unsupervised adaptation method (Li et al.,
2020b), the rule-based filtering method (Peng et al., 2020), the classifier
based filtering method (Anaby-Tavor et al., 2020) and the language model score
based filtering method (Shakeri et al., 2020). These methods rely on either
ad-hoc rules or extra models. Although they have shown promising results, each
of them considers only a single source for data augmentation. It is still
challenging to differentiate noisy instances from useful ones, since all those
instances are sampled from the same distribution generated by the same method.

In this paper, we regard both the translation approach and the generation
approach as data augmentation methods. We tackle the problem of reducing the
impact of noise in augmented data sets. We develop a principled method to
learn from multiple noisy augmented data sets for cross-lingual SLU, where no
golden-labeled target language data exists. Our major technical contribution
consists of a series of denoising methods including _instance relabeling_ ,
_co-training_ and _instance re-weighting_.

First, motivated by the self-learning method (Zoph et al., 2020), we design a
model-ensemble-based instance relabeling approach to correct the noisy labels
of augmented training data in low resource languages. As teacher models can
not always generate correct labels, the original self-learning method tends to
suffer from accumulated errors caused by model predictions. To alleviate the
problem of accumulated errors, in our instance relabeling approach, we use
crowd-intelligence from multiple models to derive the more reliable labels of
pseudo training instances. Besides, we filter out noisy instances based on co-
training and instance re-weighting strategies to reduce the impact of
incorrect predictions on subsequent training. Our training strategy does not
follow a traditional teacher-student manner. Instead, we use model predictions
in the last epoch as pseudo labels in the current epoch to compute loss which
saves training time.

In order to filter out noisy instances, we adopt a co-training mechanism,
which uses selected instances from the other models to train the current
model. Different from the co-teaching method (Han et al., 2018) where two
models are trained with the same data, we propose multiple models should be
trained with multiple different noisy augmented data where noise may be
largely independent. It is because deep neural networks have a high capacity
to fit noisy labels. When two models are trained with the same data, we tend
to obtain two similar models. The co-teaching method gradually becomes a naive
selection method with two models ensembled. In our co-training method, by
employing very different data generation methods, we hope to attain that the
noise from different sources may be largely independent and models can learn
different knowledge from them. Therefore, the instances that pass the
screening process of the other models can serve as the supervision signals to
the current model which alleviates the problem of accumulated errors caused by
selection bias.

Last, we further propose an instance re-weighting technique to adjust the
weights of training instances adaptively. As we do not have real training data
in target languages, we can use the consistency among the soft labels
predicted by different models to predict the reliability of the instances.
Intuitively, if the predictions of different models are highly inconsistent on
an instance, the instance may contain much noise. The larger deviation, the
more uncertainty, and the less weight. This idea further increases the
robustness of the selected training instances.

We conduct extensive experiments on two public datasets. The experiment
results clearly indicate that, by consciously considering multiple noisy data
sources derived from very different augmentation methods, our approach is more
effective than using any single source. Our methods improve the state of the
art (SOTA) by 3.05 and 4.24 percentage points on the two benchmark datasets,
respectively.

##  2 Related Work

The cross-lingual spoken language understanding methods can be divided into
two main categories: the model transfer methods and the data transfer methods.

The model transfer methods build on pre-trained cross-lingual models to learn
language agnostic representations, such as MUSE (Lample et al., 2017), CoVE
(McCann et al., 2017), mBERT (Wu and Dredze, 2019), XLM (Lample and Conneau,
2019), Unicoder (Huang et al., 2019), and XLM-R (Conneau et al., 2020). The
English training data is applied to fine-tune the pre-trained models and then
the fine-tuned models are directly applied to target languages Liu et al.
(2020b); Schuster et al. (2019); Qin et al. (2020); Upadhyay et al. (2018); Li
et al. (2020a). To better align embeddings between source and target
languages, Liu et al. (2019) use domain-related word pairs and employ a latent
variable model to cope with the variance of similar sentences across different
languages. Liu et al. (2020b) and Qin et al. (2020) use parallel word pairs to
construct code-switching data for fine-tuning. Their methods encourage the
model to align similar words in different languages into the same space and
attend to keywords. Liu et al. (2020c) propose a regularization approach to
align word-level and sentence-level representations across languages without
any external resource.

The data transfer methods construct pseudo-labeled data in target languages.
These methods usually employ machine translators to translate training
instances in a source language into target languages and then apply alignment
methods, such as attention weights (Schuster et al., 2019), fastalign (Dyer et
al., 2013), or giza++ (Och and Ney, 2003), to project slot labels to the
target language side. The derived training instances are combined with the
training data in the source language to fine-tune the pre-trained cross-
lingual models. Previous studies Upadhyay et al. (2018); Schuster et al.
(2019); Li et al. (2020a) show that adding translated training data can
significantly improve the model performance, especially on languages which are
distant from the source language.

The data generation approaches can also construct additional training data.
Some methods (Wang and Yang, 2015; Marivate and Sefara, 2020; Gao et al.,
2020) make slight changes to the original training instances through word
replacement or paraphrases. More sophisticated methods generate training data
through large-scale neural networks, such as generative adversarial networks
(Goodfellow et al., 2020), variational autoencoders (Doersch, 2016; dos Santos
Tanaka and Aranha, 2019; Russo et al., 2020), and pre-trained language models
(Wu et al., 2019; Anaby-Tavor et al., 2020; Kumar et al., 2020; Peng et al.,
2020).

Figure 1: The overall framework of data augmentation module (left) and
denoising module (right).

##  3 Method

In this section, we define the problem and then propose our method.

###  3.1 Problem Definition and Solution Framework

The SLU task aims to parse user queries into a predefined semantic
representation format. Formally, given an utterance
$\bm{x}={\\{x_{i}\\}}^{L}_{i=1}$ with a sequence of $L$ tokens, a SLU model
targets to produce an intent label $\bm{y}^{I}$ for the whole utterance and a
sequence of slot labels $\bm{y}^{S}={\\{\bm{y}_{i}^{S}\\}}_{i=1}^{L}$, where
$\bm{y}_{i}^{S}$ is the slot label for the $i^{th}$ token $x_{i}$. Here, we
target at the extreme cross-lingual setting where only some training data
$\mathcal{D}_{src}$ in English (or, in general, a rich-resource source
language) and some development data $\mathcal{D}_{dev}$ in English exist.
Besides, some annotated data $\mathcal{D}_{test}$ in target languages is used
as the test set. Cross-lingual SLU is to learn a model by leveraging
$\mathcal{D}_{src}$ to perform well on $\mathcal{D}_{test}$, using
$\mathcal{D}_{dev}$ for parameter tuning.

We add a special token [CLS] in front of each input sequence. Then we feed
$\bm{x}$ into an encoder $\mathcal{M}_{enc}$ to obtain the contextual hidden
representation $\bm{H}={\\{\bm{h}_{i}\\}}_{i=0}^{L}$, that is,
$\bm{H}=\mathcal{M}_{enc}(\bm{x};\Theta)$, where $\Theta$ denotes the
parameters of the encoder.

We take $\bm{h}_{0}$ as the sentence representation for intent classification
and take $\bm{h}_{i}$ $(1\leq i\leq L)$ as the token representations for slot
filling. We apply linear transformation and the _softmax_ operation to obtain
the intent probability distribution $\bm{p}^{I}(\bm{x};\Theta)$ and the slot
probability distribution $\bm{p}^{S}_{i}(\bm{x};\Theta)$, that is,

$\displaystyle\bm{p}^{I}(\bm{x};\Theta)=\emph{softmax}(\bm{W}^{I}\cdot\bm{h}_{0}+\bm{b}^{I})$
(1)
$\displaystyle\bm{p}^{S}_{i}(\bm{x};\Theta)=\emph{softmax}(\bm{W}^{S}\cdot\bm{h}_{i}+\bm{b}^{S})$

where $\bm{p}^{I}\in\mathbb{R}^{1\times\left|C^{I}\right|}$,
$\bm{p}^{S}_{i}\in\mathbb{R}^{1\times\left|C^{S}\right|}$, $C^{I}$ is the set
of intent labels, $C^{S}$ is the set of slot labels under the BIO annotation
schema Ramshaw and Marcus (1999),
$\bm{W}^{I}\in\mathbb{R}^{\left|C^{I}\right|\times d}$ and
$\bm{W}^{S}\in\mathbb{R}^{\left|C^{S}\right|\times d}$ are the output
matrices, and $\bm{b}^{I}$ and $\bm{b}^{S}$ are the biases.

The overall architecture of our proposed method is shown in Figure 1. It
consists of two major modules, the data augmentation module and the denoising
module.

###  3.2 The Data Augmentation Module

In this module, we augment training data in target languages via translation
and generation. The left part of Figure 1 shows the details.

####  3.2.1 Translation

We use Google Translator to translate the training corpus $\mathcal{D}_{src}$
in the source language (English) to the target languages. In addition to
translation, we also need some word alignment methods to project the slot
labels to the target language side. We try giza++ Och and Ney (2003) and
fastalign Dyer et al. (2013) to obtain word alignment information and find
that the pseudo slot labels projected by giza++ generally lead to better
performance (about 2% increase in F1 on the SNIPS dataset). Thus, in the rest
of the paper, we use Google translator and giza++ to produce translated data.
Denote by $\mathcal{D}_{trans}$ the translated training corpus.

####  3.2.2 Generation

To further increase the diversity of synthesized training data, we leverage
multilingual BART (mBART) (Liu et al., 2020a) as the generator to synthesize
additional target language training corpus. Specifically, we first fine-tune
the pretrained mBART model on the translated training data
$\mathcal{D}_{trans}$ by adopting the denoising objective (Liu et al., 2020a)
\- the cross-entropy loss between the decoder’s output and the original input.
The input to mBART consists of the dialog act and the utterance in
$\mathcal{D}_{trans}$, defined by

$\bm{X}=[I;(s_{1},v_{1}),...,(s_{p},v_{p});(x_{1},...,x_{L})]$ (2)

where $I$ is the intent and ${(s_{i},v_{i})}_{i=1}^{p}$ are the slot-value
pairs. Here, $v_{i}$ in the target language is obtained by word alignment
between utterances in the source and the target language. Following Liu et al.
(2020a), we apply text infilling as the injected noise in the fine-tuning
stage.

After fine-tuning, we apply the same noise function to the input data $\bm{X}$
and leverage the fine-tuned mBART to generate $m$ candidates for each
instance. To increase diversity of generated data, the top-$p$ sampling
strategy (Fan et al., 2018) is adopted. Each generated instance consists of
“an utterance, the corresponding intent and slot-value pairs”. Then, we
perform preliminary data selection by filtering out the generated utterances
not containing the required slot values. Last, we randomly sample $a$
instances from the candidate set for each input to construct the generated
corpus $\mathcal{D}_{gen}$.

###  3.3 The Denoising Module

To tackle the noisy label problem introduced by the data augmentation module,
we design a denoising module shown in Algorithm 1.

At the initialization stage, we first train $K$ models using the augmented
data derived from $K$ different augmentation methods. All models are optimized
by the cross-entropy loss function computed using the original labels of
intent and slots. For the $k$-th model ($k\leq K$),

$\mathcal{L}^{k}(\bm{x})\\!=-\frac{1}{L}\sum^{L}_{j=1}\bm{y}_{j}^{S}log\bm{p}^{S}_{j}(\bm{x};\Theta_{k})\\!-\\!\bm{y}^{I}log\bm{p}^{I}(\bm{x};\Theta_{k})$
(3)

where $\bm{x}$ is a training utterance, $\bm{y}^{I}$ is the intent label,
$\bm{y}_{j}^{S}$ is the slot label of the $j$-th word in the utterance, and
$\bm{p}^{I}(\bm{x};\Theta_{k})$ and $\bm{p}^{S}_{j}(\bm{x};\Theta_{k})$ are
the predicted probability distributions of the intent and the slot,
respectively.

To keep our discussion simple, in this paper, we mainly consider using the
training corpora derived from machine translation and generation. Thus, we
maintain $K=2$ SLU networks with the same structure in the training process.
$\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ are trained using different training
corpora $\mathcal{D}_{1}\\!=\\!\\{\mathcal{D}_{src},\mathcal{D}_{trans}\\}$
and
$\mathcal{D}_{2}\\!=\\!\\{\mathcal{D}_{src},\mathcal{D}_{trans},\mathcal{D}_{gen}\\}$,
respectively. Based on our epxeriments, $\mathcal{D}_{gen}$ only is too noisy
thus we combine with $\mathcal{D}_{trans}$. Our training framework in general
can handle more than $2$ models. We present experimental results with more
than $2$ models in Section 4.4.

After the initialization stage, each model has learned some knowledge from
each augmented training data. Since there exists noise in the augmented
training data (such as $\mathcal{D}_{trans}$ and $\mathcal{D}_{gen}$ ), we
step into the relabeling stage, which combines a series of strategies:
_instance relabeling_ , _co-training_ and _instance re-weighting_ to reduce
the impact of the noise.

Data: $\mathcal{M}_{k}(1\leq k\leq K)$: SLU networks; $E_{all}$: the number of
fine-tuning epochs; $E$: the number of initialization epochs;
$\mathcal{D}_{src},\mathcal{D}_{trans},\mathcal{D}_{gen}$: training corpus;

Result: $\mathcal{M}_{k}(1\leq k\leq K)$

// Initialization Stage

1 for  _$e=1$ ; $e<E$; $e++$_ do

2 shuffle training corpus $\mathcal{D}_{k}$ from
$\\{\mathcal{D}_{src},\mathcal{D}_{trans},\mathcal{D}_{gen},...\\}$

3 update $\mathcal{M}_{k}$ with $\mathcal{L}^{k}(\mathcal{D}_{k})$

4 end for

// Relabeling Stage

5 for  _$e=E$ ; $e\leq E_{all}$; $e++$_ do

6 shuffle training corpus
$\mathcal{D}=\\{\mathcal{D}_{src},\mathcal{D}_{trans},\mathcal{D}_{gen},...\\}$
into $B$ mini-batches;

7 for  _$j=1$ ; $j<=B$; $j++$_ do

8 Fetch $j^{th}$ mini-batch $\mathcal{D}^{\prime}$ from $\mathcal{D}$

// Co-Training

9
$\mathcal{D}^{\prime}_{k}\\!=\\!argmin_{\mathcal{D}^{\prime}:|\mathcal{D}^{\prime}|\geq(1-\delta)|\mathcal{D}^{\prime}|}\sum_{i}^{K\setminus
k}\mathcal{L}^{i}(\mathcal{D}^{\prime})$

// Instance Re-weighting

10 update $\mathcal{M}_{k}$ with
$\mathcal{L}^{k^{\prime}}(\mathcal{D}^{\prime}_{k})$

// Instance Relabeling

11 update labels of instances in $\mathcal{D}\setminus\mathcal{D}_{src}$ as
Equation 4

12 end for

13

14 end for

Algorithm 1 Denoising Module

####  3.3.1 Instance Relabeling

In the relabeling stage, model training and instance relabeling are conducted
iteratively. Motivated by the idea of model ensemble, we use the ensemble of
model predictions to correct label errors in a self-learning manner.
Specifically, all models are trained using all training corpora
$\mathcal{D}=\\{\mathcal{D}_{src},\mathcal{D}_{trans},\mathcal{D}_{gen}\\}$.
The slot labels and the intent labels of the training instances in
$\mathcal{D}_{trans}$ and $\mathcal{D}_{gen}$ are modified to the
corresponding ensemble predicted probability distributions, which are used as
the pseudo-truth labels to compute the loss in the next epoch. That is,

$\displaystyle\widetilde{\bm{y}}^{I}=\frac{1}{K}\sum^{K}_{k=1}\bm{p}^{I}(\bm{x};\Theta_{k});\quad\widetilde{\bm{y}}_{j}^{S}=\frac{1}{K}\sum^{K}_{k=1}\bm{p}^{S}_{j}(\bm{x};\Theta_{k})$
(4)

####  3.3.2 Co-Training

Heuristically, instances with small losses are more likely to have cleaner
labels. When noise from different augmentation methods is more or less
independent, each model can learn instances with small cross-entropy losses
from the other models.

Specifically, when $K=2$, in each batch of the training data, each network
discards the instances with larger losses computed by Equation 3 by a ratio of
$\delta$ and then teach the remaining instances to another one.

####  3.3.3 Instance Re-weighting

Another way to reduce the impact of noisy instances is to assign different
weights to different instances, the more noisy an instance, the less weight it
is associated. We design a re-weighting mechanism to implement this idea. The
intuition is that if the predictions by multiple models are quite
inconsistent, the instance may likely be noisy. Technically, we design an
uncertainty based weight to re-weight the training instances. The larger the
deviation, the more uncertainty and the less the weight. Specifically, the
uncertainty of each instance is defined as:

$u=\frac{1}{K}\sum^{K}_{k=1}Var(\bm{p}(\bm{x};\Theta_{k}),\bm{m}(x))$ (5)

where

$\displaystyle
Var(\bm{p}(\bm{x};\Theta_{k}),\bm{m}(x))=(\bm{p}^{I}(\bm{x};\Theta_{k})-\frac{1}{K}\sum^{K}_{k=1}\bm{p}^{I}(\bm{x};\Theta_{k}))^{2}$
(6)
$\displaystyle+\frac{1}{L}\sum_{l=1}^{L}(\bm{p}^{S}_{l}(\bm{x};\Theta_{k})-\frac{1}{K}\sum^{K}_{k=1}\bm{p}^{S}_{l}(\bm{x};\Theta_{k}))^{2}$

We further compute weight by $w=e^{-u}$ and incorporate this weight into
Equation 3 to obtain:

$\mathcal{L}^{k^{\prime}}(\bm{x})=-w[\frac{1}{L}\sum^{L}_{j=1}\widetilde{\bm{y}}_{j}^{S}log\bm{p}^{S}_{j}(\bm{x};\Theta_{k})\\!+\\!\widetilde{\bm{y}}^{I}log\bm{p}^{I}(\bm{x};\Theta_{k})]$
(7)

which is the new training objective during the relabeling stage.

|  SNIPS (F1/Intent Accuracy) |  |  MTOP (Exact Match Accuracy)
---|---|---|---
| es | th | Average |  | es | fr | de | hi | th | Average
Multi. CoVe | 19.25/53.89 | 35.62/70.70 | 27.44/62.30 |  | - | - | - | - | - | -
Transferable Latent Variable | 65.79/90.20 | 32.24/73.47 | 49.02/81.84 |  | - | - | - | - | - | -
Attention-Informed Mixed | 73.89/87.88 | 27.12/73.46 | 50.51/80.67 |  | - | - | - | - | - | -
CoSDA-ML | 80.40/94.80 | 37.30/76.80 | 58.85/85.80 |  | - | - | - | - | - | -
LR&ALVM | 72.49/92.31 | 33.28/75.77 | 52.89/84.04 |  | - | - | - | - | - | -
EN | 84.20/97.70 | 46.00/90.40 | 65.10/94.05 |  | 69.10 | 65.40 | 64.00 | 55.00 | 43.80 | 59.46
EN + Trans. | 73.40/98.30 | 50.90/96.60 | 62.15/97.45 |  | 74.50 | 72.60 | 64.70 | 58.30 | 56.50 | 65.32
EN + Trans. + mask | 83.00/98.00 | 50.20/96.60 | 66.60/97.30 |  | 74.60 | 72.20 | 65.70 | 62.50 | 53.20 | 65.64
ENreimp | 84.69/97.67 | 39.34/89.42 | 62.02/93.55 |  | 68.38 | 70.48 | 66.15 | 54.81 | 38.36 | 59.63
EN + Trans.reimp | 83.58/99.34 | 79.29/97.93 | 81.44/98.64 |  | 70.25 | 67.27 | 70.39 | 64.22 | 55.80 | 65.59
EN + Trans. + Gen. + Denoise | 86.19/99.37 | 82.79/98.17 | 84.49/98.71 |  | 71.58 | 71.53 | 74.05 | 69.66 | 62.31 | 69.83

Table 1: Comparison results between our approach and prior SOTA methods. The
top block rows are baselines that belong to model transfer methods. The middle
block rows are XLM-R large model based approaches, w/ or w/o translated
training data. Methods with reimp are re-implemented in this paper with
different translator and alignment tool. EN, Trans. and Gen. denote source
language data $\mathcal{D}_{src}$, translated target language data
$\mathcal{D}_{trans}$ and generated target language data $\mathcal{D}_{gen}$,
respectively. Denoise denotes proposed denoising module.

##  4 Experiments

In this section, we report our experiments on two benchmark datasets.

###  4.1 Settings

We evaluate the effectiveness of our proposed approach over five languages on
two benchmark datasets: SNIPS (Schuster et al., 2019) and MTOP (Li et al.,
2020a). The details of datasets are provided in Appendix.

For generation, we generate $m=10$ candidates for each input and randomly
sample $a=1$ from each candidate set to construct $\mathcal{D}_{gen}$. Our SLU
model is based on the pre-trained XLM-R large model, which has $L=24$ layers
and two additional task-specific linear layers for intent classification and
slot filling. More implementation details including hyper-parameters are
described in Appendix.

Following the previous works (Schuster et al., 2019; Li et al., 2020a), we use
F1 score to measure the slot filling quality and use accuracy score to
evaluate the intent classification quality on the SNIPS dataset and use Exact
Match Accuracy on the MTOP dataset.

We employ the following SOTA baselines in two groups. The first group is the
model transfer methods, including Multi.CoVe Schuster et al. (2019);
Transferable Latent Variable Liu et al. (2019); Attention-Informed Mixed Liu
et al. (2020b); CoSDA-ML Qin et al. (2020); LR&ALVM Liu et al. (2020c); and EN
Li et al. (2020a). The second group is the data transfer methods, including
EN+Trans. Li et al. (2020a) and EN+Trans.+mask Li et al. (2020a).

###  4.2 Results

| es | fr | de | hi | th | Average
---|---|---|---|---|---|---
Classifier based selection | 71.45 | 69.21 | 71.68 | 66.37 | 60.43 | 67.83
LM based selection | 70.41 | 68.78 | 70.95 | 65.94 | 58.44 | 66.90
Classifier based selection w/ relabeling | 72.01 | 70.06 | 73.29 | 69.01 | 60.25 | 68.92
LM based selection w/ relabeling | 71.38 | 69.28 | 72.95 | 67.73 | 61.19 | 68.51
EN + Trans. + Gen. + Denoise | 71.58 | 71.53 | 74.05 | 69.66 | 62.31 | 69.83

Table 2: Comparison with other denoising methods on the MTOP dataset.

| es | fr | de | hi | th | Average
---|---|---|---|---|---|---
EN + Trans. | 70.25 | 67.27 | 70.39 | 64.22 | 55.80 | 65.59
EN + Gen. | 66.71 | 65.99 | 70.84 | 63.55 | 45.32 | 62.48
EN + Trans. + Gen. | 69.21 | 66.93 | 71.46 | 65.48 | 56.89 | 65.99
EN + Trans. + Gen. + Denoise | 71.58 | 71.53 | 74.05 | 69.66 | 62.31 | 69.83
w/o Gen. | 71.08 | 70.28 | 72.67 | 70.05 | 62.71 | 69.36$~{}\downarrow$
w/o instance relabeling | 71.98 | 70.15 | 72.78 | 69.37 | 60.87 | 69.03$~{}\downarrow$
w/o co-training | 71.71 | 69.56 | 73.15 | 68.94 | 61.41 | 68.95$~{}\downarrow$
w/o instance re-weighting | 72.18 | 70.84 | 73.99 | 68.55 | 62.82 | 69.68$~{}\downarrow$

Table 3: Ablation study on the MTOP dataset. Figure 2: Metric trend per epoch
on MTOP w/ or w/o instance relabeling strategy.

Table 1 reports the results of our approach and the SOTA baselines. As the
translator used in Li et al. (2020a) is not publicly available, we use Google
translator instead, which leads to some results on some languages slightly
different from reported by Li et al. (2020a).

Our method outperforms the SOTA baselines and achieves new SOTA performance.
Our method improves the Exact Match Accuracy on MTOP from 65.59 to 69.83, the
F1 score on SNIPS from 81.44 to 84.49, and the accuracy on SNIPS from 98.64 to
98.71. These results clearly demonstrate the effectiveness of our proposed
method.

One interesting finding is that the performances on Spanish and French become
slightly worse after adding the translated data. It is because the noise
introduced by the machine translation and alignment processes may hurt the
performance. Our method introduces the denoising training approach, which is
able to handle the noise of synthesized data.

###  4.3 Comparison with Other Denoising Methods

To verify the effectiveness of our approach, we conduct experiments with two
kinds of denoising approaches used by the previous works. First, we consider
the classifier based selection approach. Following Anaby-Tavor et al. (2020),
we train an extra classifier using the corpus in English and the translated
corpus, and filter out noisy data according to the probability scores
predicted by the classifier. Second, we consider the LM based selection
approach. Following Shakeri et al. (2020), we use the language model score as
the indicator to select high-quality data.

For fair comparisons, for each noise filtering baseline, we also fine-tune two
pre-trained XLM-R large models using different random seeds and take the
ensembled model as the final model. Additionally, to remove the effect of
instance relabeling method, we also apply it to baselines similar to our
approach. Table 2 shows the comparison results on the MTOP dataset. Our
approach outperforms those two methods (w/ relabeling) by 0.91 and 1.32
percentage points, respectively. This suggests that the gain of our method is
not from the simple ensemble of two models. Instead, our method could indeed
effectively remove the noise of synthesized data, outperforming previous noise
filtering methods.

Type | Source | Lang | Utterance | Original Labels | Modified Labels
---|---|---|---|---|---
Intent | Gen. | fr | je suis tout disponible | get_available | set_available
Intent | Gen. | de | verschieben sie die erinnerung in den alarm | update_reminder | delete_reminder
Slot | Trans. | es | ¿cómo es el clima en colorado en esta época del año? | None | location
Boundary | Trans. | es | resumen de noticias [de tayer] | data_time | data_time
Slot&Boundary | Gen. | de | lesen sie mir [die heutigen] schlagzeilen | news_source | data_time

Table 4: Cases of relabeling results on the MTOP dataset.

| One model | Two models | Three models
---|---|---|---
EN+Trans.+Gen.+Denoise | 67.57 | 69.83 | 70.04

Table 5: Performance of approaches with different number of models on the MTOP
dataset.

###  4.4 Ablation Study

To validate the contribution of each component in our method, we conduct the
ablation study on the MTOP dataset. We consider several ablation options. (1)
w/o generation removes the generated training data. (2) w/o instance
relabeling keeps the intent and slot labels of data unchanged throughout the
training process. (3) w/o co-training trains models using all training data
without filtering. (4) w/o instance re-weighting skips the instance re-
weighting strategy.

As shown in Table 3, compared with the performance of approach using
translated and source language data, the performance of approach using
generated data and source language data is reduced by 3.11 percentage points
without denoising strategies. This is due to the much noise introduced by the
generation process, which hurts the performance. When combining with our
denoising approach, the approach with generated training corpus is superior to
the approach without generated corpus by 0.47 percentage points. We consider
that multiple augmented data sets increase data diversity and lead to better
supervision signals. Table 3 also shows that removing any of the other
components generally leads to clear performance drop. It confirms that all of
the proposed techniques contribute to the cross-lingual setting.

###  4.5 Effect of Instance Relabeling

To better understand the effect of the instance relabeling strategy, in Figure
2, we record the Exact Match Accuracy of our method with or without the
relabeling strategy on the MTOP test set after each training epoch. The
performance of our method with relabeling strategy keeps improving and is
consistently better than the baseline during the relabeling stage. It
demonstrates that the relabeling method indeed corrects many label errors in
the noisy training data and the corrected labels contribute to the performance
improvement.

###  4.6 Effect of Number of Models

We explore the effect of the number $K$ of models (XLM-R). Specifically, we
conduct experiments using one or three models. In the setting of one model,
that is $K=1$, we only train one network with all training corpora
$\mathcal{D}=\\{\mathcal{D}_{src},\mathcal{D}_{trans},\mathcal{D}_{gen}\\}$,
and adopt the instance relabeling and instance filtering strategies. In the
setting of $K=3$, three models are trained using
$\\{\mathcal{D}_{src},\mathcal{D}_{trans}\\}$,
$\\{\mathcal{D}_{src},\mathcal{D}_{gen}\\}$ and
$\\{\mathcal{D}_{src},\mathcal{D}_{trans},\mathcal{D}_{gen}\\}$, respectively.

The results shown in Table 5 indicate that our method can effectively extend
beyond two models. When the number of networks increases, the performance
improves. The intuition is that more models can produce more reliable
predictions, and thus can lead to better instance relabeling as well as
instance filtering.

| 0% | 10% | 20% | 30% | 40%
---|---|---|---|---|---
EN+Trans.+Denoise | 68.32 | 68.69 | 69.36 | 69.07 | 68.75
EN+Trans.+Gen.+Denoise | 68.95 | 68.77 | 69.12 | 69.83 | 69.07

Table 6: Performance with different filtering rates and training corpus.

###  4.7 Effect of Filtering Rate in Co-Training

To study the impact of the co-training strategy, we conduct experiments with
different filtering rates on the MTOP dataset. Table 6 shows the results with
regard to different filtering rates and different training corpora. For both
approaches using or not using the generated corpus, as the filtering rate
increases, the performance improves as well. This demonstrates that the
filtering strategy can indeed filter out noisy instances effectively. However,
further increasing the filtering rate degrades the performance. It is mainly
because of the excessive drop of useful information contained in the training
data. Another finding is that the best filtering rate for training corpus
$\\{\mathcal{D}_{src},\mathcal{D}_{trans},\mathcal{D}_{gen}\\}$ is larger than
that for $\\{\mathcal{D}_{src},\mathcal{D}_{trans}\\}$. The explanation may be
that the generated corpus $\mathcal{D}_{gen}$ has more diverse data than the
translated corpus $\mathcal{D}_{trans}$, but may also contain more noise.

##  5 Case Study

We conduct case analysis of the instance relabeling results on the MTOP
dataset to examine the capability of our approach. We statistically analyze
the differences between the original labels and the modified labels after
relabeling stage.

We find that our instance relabeling method effectively corrects wrong labels
of the synthesized data, including intent label and slot label as shown in
Table 4. Specifically, there are four types of label modifications: 1) _Intent
Change_ : the intent label of an utterance is modified; 2) _Slot Change_ : the
slot type of a text span is modified; 3) _Boundary Change_ : the BIO
boundaries of a slot are modified; and 4) _Slot and Boundary Change_ : both
the slot type and the BIO boundaries are modified. For the MTOP dataset,
intent labels of 4.99% of the translated and generated data are modified and
the slot labels of 33.10% of those data sets are modified.

From the case study, we can see that the synthesized data indeed contains much
noise and our relabeling strategy is able to greatly reduce the negative
impact of the noise by correcting different types of label errors.

##  6 Conclusions

In this paper, we propose a denoising training approach where multiple models
trained from various augmented methods provide supervision signals to each
other. Extensive experimental results show that the proposed method
outperforms the previous approaches, and can certainly alleviate the noisy
label problem. Our proposed method is independent of the backbone network
(e.g., XLM-R model) and the task. As future work, we plan to investigate the
performance of our method on different cross-lingual tasks.

## Acknowledgments

Jian Pei’s research is supported in part by the NSERC Discovery Grant program.
All opinions, findings, conclusions and recommendations in this paper are
those of the authors and do not necessarily reflect the views of the funding
agencies.

## References

  * Anaby-Tavor et al. (2020) Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020.  Do not have enough data? deep learning to the rescue!  In _AAAI_ , pages 7383–7390.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 8440–8451. Association for Computational Linguistics.
  * Doersch (2016) Carl Doersch. 2016.  Tutorial on variational autoencoders.  _CoRR_ , abs/1606.05908.
  * dos Santos Tanaka and Aranha (2019) Fabio Henrique Kiyoiti dos Santos Tanaka and Claus Aranha. 2019.  Data augmentation using gans.  _CoRR_ , abs/1904.09135.
  * Dyer et al. (2013) Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013.  A simple, fast, and effective reparameterization of ibm model 2.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 644–648.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann N. Dauphin. 2018.  Hierarchical neural story generation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , pages 889–898. Association for Computational Linguistics.
  * Gao et al. (2020) Silin Gao, Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020.  Paraphrase augmented task-oriented dialog generation.
  * Goodfellow et al. (2020) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144.
  * Han et al. (2018) Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. 2018.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In _Advances in neural information processing systems_ , pages 8527–8537.
  * Huang et al. (2019) Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019.  Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 2485–2494. Association for Computational Linguistics.
  * Huang et al. (2020) Zhiqi Huang, Fenglin Liu, and Yuexian Zou. 2020.  Federated learning for spoken language understanding.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 3467–3478, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Kumar et al. (2020) Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020.  Data augmentation using pre-trained transformer models.  _CoRR_ , abs/2003.02245.
  * Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019.  Cross-lingual language model pretraining.  _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Lample et al. (2017) Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017\.  Unsupervised machine translation using monolingual corpora only.  _arXiv preprint arXiv:1711.00043_.
  * Li et al. (2020a) Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2020a.  Mtop: A comprehensive multilingual task-oriented semantic parsing benchmark.  _arXiv preprint arXiv:2008.09335_.
  * Li et al. (2020b) Xin Li, Lidong Bing, Wenxuan Zhang, Zheng Li, and Wai Lam. 2020b.  Unsupervised cross-lingual adaptation for sequence tagging and beyond.
  * Liu and Lane (2016) Bing Liu and Ian Lane. 2016.  Attention-based recurrent neural network models for joint intent detection and slot filling.
  * Liu et al. (2020a) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020a.  Multilingual denoising pre-training for neural machine translation.  _Trans. Assoc. Comput. Linguistics_ , 8:726–742.
  * Liu et al. (2019) Zihan Liu, Jamin Shin, Yan Xu, Genta Indra Winata, Peng Xu, Andrea Madotto, and Pascale Fung. 2019.  Zero-shot cross-lingual dialogue systems with transferable latent variables.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 1297–1303. Association for Computational Linguistics.
  * Liu et al. (2020b) Zihan Liu, Genta Indra Winata, Zhaojiang Lin, Peng Xu, and Pascale Fung. 2020b.  Attention-informed mixed-language training for zero-shot cross-lingual task-oriented dialogue systems.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 8433–8440. AAAI Press.
  * Liu et al. (2020c) Zihan Liu, Genta Indra Winata, Peng Xu, Zhaojiang Lin, and Pascale Fung. 2020c.  Cross-lingual spoken language understanding with regularized representation alignment.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 7241–7251. Association for Computational Linguistics.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Marivate and Sefara (2020) Vukosi Marivate and Tshephisho Sefara. 2020.  Improving short text classification through global augmentation methods.  In _International Cross-Domain Conference for Machine Learning and Knowledge Extraction_ , pages 385–399. Springer.
  * McCann et al. (2017) Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017.  Learned in translation: Contextualized word vectors.  In _Advances in neural information processing systems_ , pages 6294–6305.
  * Och and Ney (2003) Franz Josef Och and Hermann Ney. 2003.  A systematic comparison of various statistical alignment models.  _Computational Linguistics_ , 29(1):19–51.
  * Peng et al. (2020) Baolin Peng, Chenguang Zhu, Michael Zeng, and Jianfeng Gao. 2020.  Data augmentation for spoken language understanding via pretrained models.  _CoRR_ , abs/2004.13952.
  * Qin et al. (2020) Libo Qin, Minheng Ni, Yue Zhang, and Wanxiang Che. 2020.  Cosda-ml: Multi-lingual code-switching data augmentation for zero-shot cross-lingual NLP.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020_ , pages 3853–3860. ijcai.org.
  * Ramshaw and Marcus (1999) Lance A Ramshaw and Mitchell P Marcus. 1999.  Text chunking using transformation-based learning.  In _Natural language processing using very large corpora_ , pages 157–176.
  * Ruder et al. (2019) Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019.  A survey of cross-lingual word embedding models.  _Journal of Artificial Intelligence Research_ , 65:569–631.
  * Russo et al. (2020) Giuseppe Russo, Nora Hollenstein, Claudiu Cristian Musat, and Ce Zhang. 2020.  Control, generate, augment: A scalable framework for multi-attribute text generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020_ , pages 351–366. Association for Computational Linguistics.
  * Schuster et al. (2019) Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019.  Cross-lingual transfer learning for multilingual task oriented dialog.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 3795–3805. Association for Computational Linguistics.
  * Shakeri et al. (2020) Siamak Shakeri, Noah Constant, Mihir Sanjay Kale, and Linting Xue. 2020.  Multilingual synthetic question and answer generation for cross-lingual reading comprehension.  _CoRR_ , abs/2010.12008.
  * Tur and De Mori (2011) Gokhan Tur and Renato De Mori, editors. 2011.  _Spoken Language Understanding: Systems for Extracting Semantic Information from Speech_.  Wiley.
  * Upadhyay et al. (2018) S. Upadhyay, M. Faruqui, G. Tur, H. Dilek, and L. Heck. 2018.  (almost) zero-shot cross-lingual spoken language understanding.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6034–6038.
  * Wang and Yang (2015) William Yang Wang and Diyi Yang. 2015.  That’s so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using# petpeeve tweets.  In _Proceedings of the 2015 conference on empirical methods in natural language processing_ , pages 2557–2563.
  * Wang et al. (2005) Ye-Yi Wang, Li Deng, and Alex Acero. 2005.  Spoken language understanding "” an introduction to the statistical framework.  _IEEE Signal Processing Magazine_ , 22(5):16–31.
  * Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019.  Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 833–844. Association for Computational Linguistics.
  * Wu et al. (2019) Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019.  Conditional bert contextual augmentation.  In _International Conference on Computational Science_ , pages 84–95. Springer.
  * Xu et al. (2020) Weijia Xu, Batool Haider, and Saab Mansour. 2020.  End-to-end slot alignment and recognition for cross-lingual NLU.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5052–5063.
  * Zhao et al. (2019) Zijian Zhao, Su Zhu, and Kai Yu. 2019.  Data augmentation with atomic templates for spoken language understanding.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 3635–3641. Association for Computational Linguistics.
  * Zoph et al. (2020) Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le. 2020.  Rethinking pre-training and self-training.  _arXiv preprint arXiv:2006.06882_.

##  Appendix A Experimental Setting

###  A.1 Datasets

We evaluate the effectiveness of our proposed approach over five target
languages on two benchmark datasets: SNIPS (Schuster et al., 2019) and MTOP
(Li et al., 2020a). Statistics of used data are detailed in Table 7.

| SNIPS | MTOP
---|---|---
Language | en,es,th | en,es,fr,de,hi,th
Intent types | 12 | 117
Slot types | 11 | 78
Train | 30,521 | 13,151
Dev | 4,181 | 1,878
Test | 4,735 | 16,442

Table 7: Statistics of datasets.

###  A.2 Implementation Details

For generation, we fine-tune mBART pre-trained on 25 languages with 0.3
dropout, 0.2 label smoothing, 2500 warm-up steps, $3\times 10^{-5}$ maximum
learning rate, and 1024 tokens in each batch. For text filling, we mask $35\%$
of the words in each instance by randomly sampling a span length according to
a Poisson distribution ($\lambda\\!=\\!3.5$). Then we append to each instance
an end-of-sentence token ($<\\!/S\\!>$) and the corresponding language id
symbol ($<\\!LID\\!>$). We don’t search the best parameters for generation but
use the default values in open-source
code***https://github.com/pytorch/fairseq/tree/master/examples
/mbart. The final models are selected based on validation likelihood.

For SLU, we use XLM-R large model with about 550M parameters as the backbone
network. In the fine-tuning process, we set the batch size as 128, fine-tuning
epochs $E_{all}=10$, initialization epochs $E=4$ and 0.1 dropout for two
benchmark datasets. The maximum filtering rates are $\delta=0.2$ and
$\delta=0.3$ for the SNIPS and MTOP datasets, respectively. The learning rates
are $2\times 10^{-5}$ and $5\times 10^{-5}$ for SNIPS and MTOP datasets,
respectively. We select the best hyper-parameters by searching a combination
of batch size, learning rate, the number of fine-tuning epochs, the number of
initialization epochs and the filtering ratio with the following range: batch
size $\\{32,64,128\\}$, learning rate $\\{1,2,3,4,5\\}\times 10^{-5}$, fine-
tuning epochs $\\{5,10,15\\}$, initialization epochs $\\{2,3,4\\}$, filtering
ratio $\\{10\%,20\%,30\%,40\%\\}$. The models are saved by performance on the
English development corpus and translated target language development corpus.
The models are trained using mini-batch back-propagation, and the AdamW
(Loshchilov and Hutter, 2019) optimizer is used for optimization. We fine-tune
the models on two V100-32GB GPUs which lasts about 4 hours.

##  Appendix B More Experimental Results and Discussions

###  B.1 Effect of Generated Data Size

To further analyze the effect of generated data, we randomly sample
$a=\\{0,1,2,3\\}$ instances from the candidate set for each input to construct
the generated corpus. As shown in Table 8, by increasing the size of the
generated corpus, the performance improves. However, when the data size
reaches a certain scale, e.g., $a=3$ in our experiments, the performance
slightly regresses, but still outperforms the baseline without generated data.
This suggests that the augmentation module indeed increases the diversity of
training data and then improves the performance. Though increasing noise
limits the growth of the improvement, our approach is robust enough to achieve
comparable performance.

| 0 | 1 | 2 | 3
---|---|---|---|---
EN+Trans.+Gen.+Denoise | 69.36 | 69.83 | 70.52 | 69.98

Table 8: Performance with different sizes of generated data on the MTOP
dataset.

###  B.2 Variance Analysis

We conduct 5 runs of training and calculate the mean and standard deviation
(Stdev) values for our approach and the baseline on the MTOP dataset. The
results are listed in Table 9. Besides, we also conduct a two-sided
statistically significant t-test with the significance threshold 0.05 to
compare the baseline with our method. The results show that the variance of
our approach is similar to that of the baseline. Moreover, with p-value =
$4.5\times 10^{-8}$, our method outperforms the baseline with statistical
significance.

|  | Mean | Stdev
---|---|---|---
Baseline | 65.59 66.02 65.59 65.46 65.12 | 65.56 | 0.29
Our method | 69.83 69.68 69.51 69.46 69.02 | 69.50 | 0.27

Table 9: Mean and Standard deviation (Stdev) values across five different runs
for our approach and the baseline on the MTOP dataset.


# Tracing the emergence of collectivity phenomena in small systems††thanks:
Presented at Quark Matter 2022 XXIXth International Conference on Ultra-
relativistic Nucleus-Nucleus Collisions

B. Schenke
S. Schlichting
P. Singh Presenter Physics Department, Brookhaven National Laboratory, Bldg.
510A, Upton, NY 11973, USA  Fakultät für Physik, Universität Bielefeld,
D-33615 Bielefeld, Germany  Department of Physics, P.O. Box 35, 40014
University of Jyväskylä, Finland  Helsinki Institute of Physics, P.O. Box 64,
FI-00014 University of Helsinki, Finland

###### Abstract

We study initial state momentum correlations and event-by-event geometry in
p+Pb collisions at $\sqrt{s}=5.02~{}\rm TeV$ by following the approach of
extending the IP-Glasma model to 3D using JIMWLK rapidity evolution. On
examining the non-trivial rapidity dependence of the observables, we find that
the geometry is correlated over large rapidity intervals, while the initial
state momentum correlations have a relatively short range in rapidity. Based
on our results, we discuss implications for the relevance of both effects in
explaining the origin of collective phenomena in small systems.

##  1 Introduction

Experimental evidence for collectivity in high multiplicity p+p/A [1]
collisions has lead to an increasing interest in understanding the origin of
these long range azimuthal correlations in small systems. The general features
of these correlations are similar to those observed in heavy-ion collisions
where these structures are interpreted in terms of the system’s response to
initial geometry [2]. For small systems, event geometry [3] and initial state
correlations [4] have been invoked as possible explanations of observed long-
range rapidity correlations. In this proceeding, the relative contribution of
both these mechanisms have been analyzed as a function of rapidity for pPb
collision at $5.02$ TeV by using the $3+1$D IP-Glasma model that incorporates
the longitudinal structure by including the JIMWLK evolution of the incoming
nuclear distributions. The details of our work can be found in [5].

##  2 3D IP-Glasma model and event generation

We follow the description [6] within the Color Glass Condensate (CGC)
framework where the expectation value of an observable $O(y_{\rm obs})$ at any
rapidity $y_{\rm obs}$ is calculated in terms of the high energy factorization
of the projectile (p) and the target (Pb) [7] as

$\displaystyle O(y_{\rm obs})=O_{\rm
cl}\Big{(}V_{\mathbf{x_{\perp}}}^{p}(+y_{\rm
obs}),V^{Pb}_{\mathbf{x_{\perp}}}(-y_{\rm obs})\Big{)}$ (1)

where $V^{p/Pb}_{x_{\perp}}$ denote the light like Wilson line at position
$\mathbf{x_{\perp}}$.

We generate a set of 4096 events from a set of proton and lead Wilson line
configurations that are evolved to all rapidities of interest using JIMWLK
evolution and collided at different impact parameters. For each event, we run
a series of independent $2+1$ D classical Yang-Mills (CYM) simulations at
intervals of $\Delta y=0.4$ in the $y_{\rm obs}\in[-2.4,+2.4]$ range, to get
the rapidity dependence of observables according to the factorization scheme.

The centrality selection is based on the gluon multiplicity at mid-rapidity.
We will present results for two different sets of IR regulator used in the IP-
Glasma model $(\tilde{m})$ and the JIMWLK equations $(m)$, namely
$m=\tilde{m}=0.2\,{\rm GeV}$ and $m=\tilde{m}=0.8\,{\rm GeV}$.

##  3 Results

We first examine the longitudinal structure of the event geometry and the
initial state momentum anisotropy by following the standard procedure in which
the event geometry is characterized by eccentricities

$\displaystyle\varepsilon_{n}(y)=\frac{\int
d^{2}\mathbf{r_{\perp}}T^{\tau\tau}(y,\mathbf{r_{\perp}})~{}|\mathbf{r_{\perp}}|^{n}e^{in\phi_{\mathbf{r_{\perp}}}}}{\int
d^{2}\mathbf{r_{\perp}}T^{\tau\tau}(y,\mathbf{r_{\perp}})~{}|\mathbf{r_{\perp}}|^{n}}\,$
(2)

and the momentum anisotropy is given by the azimuthal anisotropy of the
produced gluons

$\displaystyle v_{2}^{g}(y)=\frac{\int
d^{2}\mathbf{k_{\perp}}|\mathbf{k_{\perp}}|\frac{dN_{g}}{dyd^{2}\mathbf{k_{\perp}}}(y)e^{2i\phi_{\mathbf{k_{\perp}}}}}{\int
d^{2}\mathbf{k_{\perp}}|\mathbf{k_{\perp}}|~{}\frac{dN}{dyd^{2}\mathbf{k_{\perp}}}(y)}\,.$
(3)

Figure 1: The rapidity dependence of
$\sqrt{\langle|\varepsilon_{2}(y)|^{2}\rangle}$ (left),
$\sqrt{\langle|\varepsilon_{3}(y)|^{2}\rangle}$ (middle) and
$\sqrt{\langle|v_{2}^{g}(y)|^{2}\rangle}$ (right) for different centrality
classes for $m=\tilde{m}=0.8~{}\rm GeV$

Our results for the rapidity dependence of $\varepsilon_{2}(y)$ (left) and
$\varepsilon_{3}(y)$ (middle), and $v_{2}^{g}$ (right) are compactly
summarised in Fig. 1. We observe that $\varepsilon_{2}$ decreases, while the
triangularity $\varepsilon_{3}$ increases with decreasing multiplicity. The
eccentricity $\varepsilon_{2}$ also decreases with increasing rapidity, while
$\varepsilon_{3}$ shows a weaker rapidity dependence except for the most
central and the most peripheral collisions, where it increases toward the Pb
going direction. On the other hand, $v_{2}^{g}$ is largely independent of
rapidity in all the centrality bins and grows monotonically with decreasing
multiplicity.

We now analyze the rapidity decorrelation of geometry and initial state
momentum correlations by computing the normalized rapidity correlation
function $C^{N}_{\mathcal{O}}(y_{1},y_{2})=\left\langle{\rm
Re}\big{(}\mathcal{O}(y_{1})\mathcal{O}^{*}(y_{2})\big{)}\right\rangle/\sqrt{\langle|\mathcal{O}(y_{1})|^{2}\rangle\langle|\mathcal{O}(y_{2})|^{2}\rangle}$
where $\mathcal{O}$ is any of the previously defined observables.

Figure 2: Normalized correlation function $C^{N}_{\mathcal{O}}(\Delta y)$ of
eccentricity $\varepsilon_{2}$ (left) and initial state anisotropy $v_{2}$
(right) for different centrality classes using $m=\tilde{m}=0.2\,{\rm GeV}$

In Fig. 2, we show $C^{N}_{\varepsilon_{2}}(\Delta y)$ and
$C^{N}_{v_{2}}(\Delta y)$ as functions of the rapidity difference $\Delta y$
for $m=\tilde{m}=0.2\rm GeV$, which are obtained as

$C^{N}_{\mathcal{O}}(\Delta y)=\frac{1}{2y_{\rm max}-\Delta y}\int_{-y_{\rm
max}+|\Delta y|/2}^{+y_{\rm max}-|\Delta
y|/2}dY~{}C^{N}_{\mathcal{O}}(Y+\Delta y/2,Y-\Delta y/2)\,.$

The geometry decorrelates faster towards more peripheral events because it is
easier to change the geometry of dilute events. The decorrelation of the
initial state momentum anisotropy shows the opposite centrality dependence,
with the most peripheral collisions showing a slower decorrelation. Comparing
the results in the two panels, we see that the decorrelation of the initial
state momentum anisotropy in $C^{N}_{v_{2}}(\Delta y)$ is much faster than the
decorrelation of the event geometry in $C^{N}_{\varepsilon_{2}}(\Delta y)$.

Finally, we study estimators for the correlation of mean transverse momentum
$[p_{T}]$ and elliptical anisotropy $V_{2}$, which is defined as [8]

$\displaystyle\hat{\rho}(V_{2}^{2},[p_{T}])=\frac{\langle\hat{\delta}V_{2}^{2}\,\hat{\delta}[p_{T}]\rangle}{\sqrt{\langle(\hat{\delta}V_{2}^{2})^{2}\rangle\langle(\hat{\delta}[p_{T}])^{2}\rangle}}\,$
(4)

The event-by-event fluctuations of an observable $O$ at fixed multiplicity are
defined as $\hat{\delta}O\equiv\delta O-\big{(}\langle\delta O\delta
N\rangle/\sigma_{N}^{2}\big{)}\delta N$ where $\delta O=O-\langle O\rangle$,
$N$ is the multiplicity and $\sigma_{N}$ the variance of $N$ in a given
centrality bin [9]. Since we are considering initial state quantities, we
compute $\hat{\rho}$ by replacing $V_{2}$ with the initial state eccentricity
$\varepsilon_{2}$ (or the initial momentum anisotropy $v_{2}$) and $[p_{T}]$
by the average entropy density $[s]=[e^{3/4}]$ where $e$ is the energy
density, approximated as $T^{\tau\tau}$.

Figure 3: The estimators for geometry $\hat{\rho}_{\rm
est}(\varepsilon_{2}^{2},[s])$ (left) and initial state momentum anisotropy
$\hat{\rho}_{\rm est}(\epsilon_{p}^{2},[s])$ (right) as a function of
centrality for two different values of $(m=\tilde{m})$. The $\hat{\rho}$ for
ABC is obtained for different rapidity regions: region A with $-2.4<y<-0.8$,
region B with $|y|<0.8$ and region C with $0.8<y<2.4$

Our results for $\hat{\rho}_{\rm est}(\varepsilon_{2}^{2},[s])$ (left) and
$\hat{\rho}_{\rm est}(v_{2}^{2},[s])$ (right) are summarized in Fig. 3, where
we present the measurement for two different bin selection methods. One uses
all quantities at mid rapidity $y=0$ and the other uses three different
rapidity bins (ABC regions) for the different components of $\hat{\rho}$. We
find that for the larger $m=\tilde{m}$, the $\hat{\rho}_{\rm
est}(\varepsilon_{2}^{2},[s])$ is always negative, as can be expected from
geometric considerations [10]. For $m=\tilde{m}=0.2\rm GeV$, we even find
positive values for most central and most peripheral events which is in line
with previous work [11] where the geometric $\hat{\rho}$ correlator turned
positive when increasing the system size. We also notice $\hat{\rho}_{\rm
est}(\varepsilon_{2}^{2},[s])$ is independent of the choice of the rapidity
bins which is related to the weak decorrelation of the geometry observed. For
$\hat{\rho}_{\rm est}(v_{2}^{2},[s])$, we observe a positive correlation when
quantities in estimators are taken at mid-rapidity, which is again in line
with previous work [10]. However, for the ABC region $\hat{\rho}_{\rm
est}(v_{2}^{2},[s])$ is consistent with zero due to the rapid decorrelation of
$v_{2}$ with rapidity.

##  4 Conclusion & Outlook

We have presented results for rapidity dependent quantities in p+Pb
collisions, computed within the CGC framework. We computed the unequal
rapidity correlations of both geometric and initial state momentum anisotropy,
quantified by $\varepsilon_{2}$ and $v_{2}$, respectively, and observed that
the geometry decorrelates much more slowly as a function of the rapidity
difference, compared to the initial momentum anisotropy. Beyond
phenomenological applications of the 3-D IP Glasma model to collective flow in
small and large systems, further theoretical progress towards the construction
of a fully 3D Wilson line configuration followed by 3+1D Yang-Mills evolution,
as explored in [12, 13], will be desirable in the future.

Acknowledgments: We acknowledge support by the U.S. Department of Energy under
Contract No. DE-SC0012704 (B.S) and Deutsche Forschungsgemeinschaft through
the CRC-TR 211 project number 315477589 (S.S and P.S). This research used
resources of NERSC, which is supported by the Office of Science of the U.S.
DOE under Contract No. DE-AC02-05CH11231.

## References

  * [1] K. Dusling, W. Li, B. Schenke, Novel collective phenomena in high-energy proton–proton and proton–nucleus collisions, Int. J. Mod. Phys. E 25 (01) (2016) 1630002.  arXiv:1509.07939, doi:10.1142/S0218301316300022.
  * [2] C. Gale, S. Jeon, B. Schenke, Hydrodynamic Modeling of Heavy-Ion Collisions, Int. J. Mod. Phys. A 28 (2013) 1340011.  arXiv:1301.5893, doi:10.1142/S0217751X13400113.
  * [3] B. Schenke, The smallest fluid on Earth, Rept. Prog. Phys. 84 (8) (2021) 082301\.  arXiv:2102.11189, doi:10.1088/1361-6633/ac14c9.
  * [4] T. Lappi, B. Schenke, S. Schlichting, R. Venugopalan, Tracing the origin of azimuthal gluon correlations in the color glass condensate, JHEP 01 (2016) 061\.  arXiv:1509.03499, doi:10.1007/JHEP01(2016)061.
  * [5] B. Schenke, S. Schlichting, P. Singh, Rapidity dependence of initial state geometry and momentum correlations in p+Pb collisions, Phys. Rev. D 105 (9) (2022) 094023.  arXiv:2201.08864, doi:10.1103/PhysRevD.105.094023.
  * [6] B. Schenke, S. Schlichting, 3D glasma initial state for relativistic heavy ion collisions, Phys. Rev. C 94 (4) (2016) 044907.  arXiv:1605.07158, doi:10.1103/PhysRevC.94.044907.
  * [7] F. Gelis, T. Lappi, R. Venugopalan, High energy factorization in nucleus-nucleus collisions, Phys. Rev. D 78 (2008) 054019.  arXiv:0804.2630, doi:10.1103/PhysRevD.78.054019.
  * [8] P. Bozek, Transverse-momentum–flow correlations in relativistic heavy-ion collisions, Phys. Rev. C93 (4) (2016) 044908.  arXiv:1601.04513, doi:10.1103/PhysRevC.93.044908.
  * [9] A. Olszewski, W. Broniowski, Partial correlation analysis method in ultra-relativistic heavy-ion collisions, Phys. Rev. C 96 (5) (2017) 054903.  arXiv:1706.02862, doi:10.1103/PhysRevC.96.054903.
  * [10] G. Giacalone, B. Schenke, C. Shen, Observable signatures of initial state momentum anisotropies in nuclear collisions, Phys. Rev. Lett. 125 (19) (2020) 192301.  arXiv:2006.15721, doi:10.1103/PhysRevLett.125.192301.
  * [11] P. Bozek, H. Mehrabpour, Correlation coefficient between harmonic flow and transverse momentum in heavy-ion collisions, Phys. Rev. C 101 (6) (2020) 064902\.  arXiv:2002.08832, doi:10.1103/PhysRevC.101.064902.
  * [12] S. Schlichting, P. Singh, 3-D structure of the Glasma initial state – Breaking boost-invariance by collisions of extended shock waves in classical Yang-Mills theory, Phys. Rev. D 103 (1) (2021) 014003.  arXiv:2010.11172, doi:10.1103/PhysRevD.103.014003.
  * [13] A. Ipp, D. I. Müller, S. Schlichting, P. Singh, Spacetime structure of (3+1)D color fields in high energy nuclear collisions, Phys. Rev. D 104 (11) (2021) 114040.  arXiv:2109.05028, doi:10.1103/PhysRevD.104.114040.