# How to Learn in a Noisy World?
Self-Correcting the Real-World Data Noise on Machine Translation

Yan Meng Di Wu Christof Monz
Language Technology Lab
University of Amsterdam
{y.meng, d.wu<EMAIL_ADDRESS>
###### Abstract

The massive amounts of web-mined parallel data contain large amounts of noise.
Semantic misalignment, as the primary source of the noise, poses a challenge
for training machine translation systems. In this paper, we first study the
impact of real-world hard-to-detect misalignment noise by proposing a process
to simulate the realistic misalignment controlled by semantic similarity.
After quantitatively analyzing the impact of simulated misalignment on machine
translation, we show the limited effectiveness of widely used pre-filters to
improve the translation performance, underscoring the necessity of more fine-
grained ways to handle data noise. By observing the increasing reliability of
the model’s self-knowledge for distinguishing misaligned and clean data at the
token-level, we propose a self-correction approach which leverages the model’s
prediction distribution to revise the training supervision from the ground-
truth data over training time. Through comprehensive experiments, we show that
our self-correction method not only improves translation performance in the
presence of simulated misalignment noise but also proves effective for real-
world noisy web-mined datasets across eight translation tasks.

How to Learn in a Noisy World?
Self-Correcting the Real-World Data Noise on Machine Translation

Yan Meng Di Wu Christof Monz Language Technology Lab University of Amsterdam
{y.meng, d.wu<EMAIL_ADDRESS>

##  1 Introduction

The success of machine translation (MT) models is mainly due to the
availability of large amounts of web-crawled data. However, publicly available
web-mined parallel corpora (bitext) such as CCAligned (El-Kishky et al.,
2020), WikiMatrix (Schwenk and Douze, 2017), ParaCrawl (Bañón et al., 2020)
and NLLB (team et al., 2022) are shown to be noisy (Kreutzer et al., 2022;
Ranathunga et al., 2024). The notable performance drop on NMT when training
with injected synthetic noise (Khayrallah and Koehn, 2018) or fine-tuning with
CCAligned (Lee et al., 2022), indicates the importance of developing noise-
robust training when learning from noisy corpus.

Figure 1: An illustration of our self-correction method. When the model’s
translation is superior than the human reference, e.g., albergo means “hotel”
instead of “house”, we self-correct the ground-truth data by the model’s
prediction and learn towards the new revised target.

Given a noisy training dataset, a common and straightforward approach to
mitigate the impact of noisy data is to filter low-quality training samples
(Herold et al., 2022; Bane et al., 2022). However, in practice, large amounts
of misaligned samples still exist in pre-filtered web-mined datasets (Kreutzer
et al., 2022). Even earlier works (Khayrallah and Koehn, 2018; Herold et al.,
2022; Li et al., 2024) study the impact of misalignment on machine
translation, but their way of generating synthetic misaligned sentences is by
random shuffling, which is unrealistic. The real-world misaligned sentences
contain partially shared meanings, which disguise them as seemingly parallel
data and are hard-to-detect. To analyze such hard-to-detect real-world
misalignment, we design a process to simulate it controlled by semantic
similarity. Similar to the real-world setting, we demonstrate that our
simulated misalignment also challenges widely-used pre-filters, including
Laser (Artetxe and Schwenk, 2018), Comet (Rei et al., 2020) and Bi-Cleaner
(Zaragoza-Bernabeu et al., 2022).

Under our simulated noise setting, we evaluate an alternative type of approach
to handle data noise: Data truncation (Kang and Hashimoto, 2020; Li et al.,
2024; Flores and Cohan, 2024), which focuses on the model training dynamics
and ignores losses at the token level when there is a relatively large
inconsistency between the model’s prediction and the ground truth during
training. However, we observe that truncation methods are sensitive to varying
levels of simulated misalignment noise. For example, in low-resource training
corpus with a high misalignment rate, truncation methods degrade the
translation performance (shown in Section 5.3). We argue that the noisy low-
resource setting restricts the model from acquiring sufficient correct
knowledge, resulting in an inaccurate removal of clean and useful ground-truth
data. Moreover, truncation methods start to ignore potential data noise from
an early training time, which overlooks the increasing reliability of the
model’s prediction over time.

To overcome these limitations, we propose an approach called self-correction,
which leverages the model’s predictions to self-correct noise during training
while maintaining supervision from the ground truth to avoid discarding useful
training information (as shown in Figure 1). To adapt to the changing
reliability of model’s self-knowledge, we set a dynamic schedule to gradually
increase the trust in the model’s predictions to revise the training
supervision from the ground truth. In the early stage of training, the model
is not well trained, so we place greater trust in the reference data than in
the model’s prediction. As the model gains knowledge during training, we use
it to revise the ground-truth data progressively.

We evaluate our self-correction method in both simulated and real-world
environments. We demonstrate that our method consistently outperforms
baselines in both high- and low-resource datasets with different levels of
misalignment noise. Moreover, we clearly show the gains are mainly from
revising the misaligned samples while maintaining the performance of clean
parallel data. In the real-world setting, our self-correction method
effectively handles naturally occurring noise in web-mined parallel datasets,
e.g. ParaCrawl and CCAligned, leading to performance gains of up to 1.2 BLEU
points across eight translation tasks and outperforming other alternative
methods, including pre-filters and truncation.

##  2 Background

###  2.1 The Noisy World

Web-mined parallel corpora are the primary training data source for machine
translation models. However, parallel data crawled from public websites lack
quality guarantees and contain different types of noise (Kreutzer et al.,
2022), including wrong language, non-linguistic content, and misalignment
noise.

The primary source of noise in parallel web-mined data is semantic
misalignment (Khayrallah and Koehn, 2018; Kreutzer et al., 2022; Ranathunga et
al., 2024). For instance, Khayrallah and Koehn (2018) analyzed the data
quality of the raw ParaCrawl corpus, showing 77% of the analyzed sentence
pairs contain noise while half of them are misalignments. Wrong language and
non-linguistic contents only account for a small portion since filters e.g.,
language identification toolkits (Herold et al., 2022), can easily detect
them. Kreutzer et al. (2022) extended the data quality analysis to pre-
filtered web-mined datasets, e.g. WikiMatrix, CCAligned, noting that more than
50% of data in both corpora are noisy while misalignment is the primary noise.
Even in the top-quality parallel data from the NLLB corpus, many misaligned
sentences can still be found (Ranathunga et al., 2024).

Overall, previous studies demonstrate the prevalence of noisy training data in
web-mined corpora for machine translation and underscore the importance of
noise-robust training, particularly in handling misaligned data.

###  2.2 Learning in the Noisy World

####  2.2.1 Pre-Filter

Data filtering is a straightforward way to remove the noise from the
translation training corpus to mitigate its negative impact. There are two
methods to ensure the semantic alignments between the source and target in a
sentence pair: (1) surface-level filters, e.g., removing sentence pairs that
differ a lot in source and target length; (2) semantic-level filters, relying
on quality estimation models to score each sentence pair (Kepler et al., 2019;
Rei et al., 2020; Peter et al., 2023). Other studies consider the denoising as
outlier detection (Taghipour et al., 2011) or ranking problem (Cui et al.,
2013).

In this paper, we mainly consider semantic-level filters: (1) Laser (Artetxe
and Schwenk, 2018), a sentence alignment pre-filter tool for web-mined
corpora, i.e., CCAligned and WikiMatrix; (2) Bi-cleaner (Zaragoza-Bernabeu et
al., 2022), a classifier that predicts if a sentence is a translation of
another. It is also a part of the pre-processing pipeline for generating the
ParaCrawl dataset; (3) Comet (Rei et al., 2020), a widely-used quality
estimation model for machine translation.

All these approaches rely on external models to select high-quality data
before training. However, our approach does not require such pre-selection but
focuses on model training dynamics, which can be applied more broadly.

####  2.2.2 Truncation

There are two ways for removing data noise by focusing on the model’s training
dynamics: (1) using extrinsic trusted data to measure the data quality between
noisy and denoised NMT models (Wang et al., 2018); (2) solely using the
model’s self-knowledge to remove potential noise during training (Kang and
Hashimoto, 2020; Li et al., 2024). In our approach, we do not require any
external trusted data or models and thus we only compare with the second type
of work, i.e., data truncation.

The most common type of truncation is based on the loss, which estimates the
data quality by the model’s predicted probability of the ground truth (Kang
and Hashimoto, 2020; Goyal et al., 2022; Flores and Cohan, 2024). Ground-truth
tokens with high loss are considered as noise and will be skipped during
training by setting their loss to zero. However, relying on loss to remove
noise overlooks the model’s probability distribution of non-target tokens. In
a high entropy context, the probability of the correct ground-truth token is
also low due to many alternative continuations, which cannot be treated as
noise. To consider the entire model prediction distribution, Li et al. (2024)
further propose Error Norm Truncation and use the $l_{2}$ norm between the
model’s prediction distribution and the one-hot distribution of the ground-
truth token to measure the data quality. During training, tokens with error
norm values exceeding a pre-defined threshold will be removed.

In this paper, we compare against data truncation methods by relying on two
model-based metrics, loss and error norm values. However, truncation methods
are sensitive to when and how much data to truncate? Our approach overcomes
these limitations by revising all the ground truth with an adaptive trusting
weight of the model’s prediction. Details are introduced in Section 4.

##  3 An Empirical Study of Misalignment

In this section, we investigate the primary source of noise, i.e., semantic
misalignment, in a simulated setting. We first introduce a strategy to
simulate realistic misalignment noise by controlling semantic similarity
(Section 3.1). Next, we show the similarity of our simulated noise to real-
world misalignment in terms of adequacy and its hard-to-detect nature (Section
3.2). Under our simulated noisy setting, we evaluate the model-based metrics
to distinguish data noise and show the potential limitations (Section 3.3).

en |  Alcohol poisoning is the biggest cause of death.
---|---
nl |  Jacht is de belangriekste doodsoorzaak.
en:Hunting is the biggest cause of death.
en |  With Bravofly you can compare the flight prices Santa Cruz De La Palma of over 400 of the most famous airlines in the world.
de |  Bravofly findet für Sie sämtliche Billigflüge Zürich \- Santa Cruz De La Palma der besten europäischen Billigfluggesellschaften.
en: Bravofly finds all the cheap flights Zurich \- Santa Cruz De La Palma from
the best European low-cost airlines for you.
Table 1: Examples of misaligned sentences in the ParaCrawl dataset. Bold
represents the misaligned meanings. Italic text represents the English
translation.

###  3.1 Simulating Misalignment Noise

To simulate misalignment, previous works (Bane et al., 2022; Herold et al.,
2022; Li et al., 2024) randomly shuffle target sentences of a clean parallel
training corpus. However, the random shuffling noise can be easily removed by
pre-filters based on length difference or semantic similarity (Herold et al.,
2022), which oversimplifies the misalignment in the real-world web-mined
corpora. For instance, Kreutzer et al. (2022) show that a large portion of
misaligned sentence pairs that share partial semantics still exists even after
pre-filter tools, i.e., Laser, (examples in Table 1) which show the challenge
of detecting real-world misalignment.

To quantitatively analyze the impact of the realistic misalignment noise, we
design a process to simulate real-world misalignment controlled by semantic
similarity (See Algorithm 1 in the Appendix). From a chunk of clean data, we
first choose $k$ misaligned target candidates for a source sentence based on
length difference and word overlap from the true parallel target. Next, we use
quality estimation methods, e.g., Laser or Comet, to calculate the semantic
similarity scores between the source sentence and its misaligned candidates.
The candidate with the top score is selected as the final synthetic misaligned
target. In this way, the misalignment noise is generated with controlled
semantic overlaps. Examples that are controlled by Laser (Misaligned-Laser)
and Comet (Misaligned-Comet) are in Appendix 6.

###  3.2 Real-World Misalignment

Figure 2: Adequacy (scale: 1–5) scores on simulated and real-world misaligned
sentences. The real-world misaligned sentences are selected from ParaCrawl
V7.0. Figure 3: loss (above) and el2n (below) distribution for clean and
misaligned-Laser noise samples during the training process (Epoch = 5, 10, 15,
30). Red distribution represents misaligned-Laser noise and blue distribution
represents the clean data.

####  3.2.1 Adequacy

To show the similarity of our simulated noise to real-world misalignment, we
conduct a human evaluation on 50 simulated and real-world misaligned sentences
rating their Adequacy (scale 1–5), which measures the meaning overlap between
source and target. In Figure 2, we show both the real-world misalignment and
simulated misaligned-Laser/Comet has a relatively high adequacy score, above
$2.5$, while Randomly shuffled misaligned sentences only has an adequacy of
$1.5$. This ensures our simulated misalignment contains partial semantic
overlaps as the real-world misalignment. Moreover, our simulation process
confirms the fluency of misaligned targets by selecting sentences from a clean
corpus. Details of human evaluation are in Appendix B.3.

####  3.2.2 Hard-to-Detect Nature

To show the hard-to-detect nature of simulated noise as real-world
misalignment, we investigate the noise detection ability of widely used pre-
filters: Comet, Laser, and Bi-Cleaner.

We calculate the noise detection accuracy of the data filters on a mixed set
with the same amounts of clean and noisy data. For the clean data, we randomly
sample 2,000 clean sentence pairs from the WMT2017 De$\rightarrow$En training
set. For Misaligned-Random, we randomly shuffle the order of target sentences
in the sampled clean sentence pairs. For Misaligned-Comet and Misaligned-
Laser, we use the same source sentences from the sampled clean data, and
select the misaligned targets from another 200K target sentences in the
training corpus based on Algorithm 1. We score each sentence pair based on the
filter models and determine a true ratio threshold based on the amounts of
clean and noisy sentence pairs, here 1:1. Sentence pairs with scores below
this threshold are classified as noisy. The details for scoring sentence pairs
by filter models are shown in Appendix A.

Type | Filter Accuracy
---|---
| Comet | Laser | Bi-Cleaner
Misaligned-Random | 73% | 76% | 75%
Misaligned-Laser | 52% | 40% | 52%
Misaligned-Comet | 50% | 60% | 52%

Table 2: Accuracy of data filters when distinguishing different misaligned
noise from clean parallel data.

Table 2 shows the filtering accuracy of data filters on different misaligned
noise. First, Misaligned-Random can be accurately detected by all data
filters, particularly for using Laser, with an accuracy of 76%. This questions
the previous assumptions (Khayrallah and Koehn, 2018; Li et al., 2024) of the
impact of misalignment noise on translation performance since most of them can
be pre-filtered. However, our introduced noise i.e., Misaligned-Laser and
Misaligned-Comet pose a challenge on all pre-filters, as the real-world
misalignment does. Overall, we show the validity of our simulated noise in two
aspects: (1) Adequacy, reflected in the similar level of shared semantics as
the real-world misalignment; (2) Hard-to-Detect Nature, reflected in the low
filter accuracy of widely used pre-filters.

###  3.3 Fine-grained Misalignment Detection

Model-based metrics, including loss and error norm values, are used in data
truncation methods (Kang and Hashimoto, 2020; Li et al., 2024) to detect and
ignore data noise at the token-level during training. Here, we evaluate their
effectiveness under our simulated noisy settings.

Loss measures the model’s predicted probability of the ground-truth token
while error norm value (el2n) calculates the difference between the ground-
truth (one-hot) distribution $\mathit{OH}(y_{t})$ and model’s prediction
distribution $p_{\theta}(\cdot|y_{<t},x)$ (eq 1).

$\mathit{el2n}=||p_{\theta}(\cdot|x,y_{<t})-\mathit{OH}(y_{t})||_{2}.$ (1)

Tokens with higher loss or el2n value are indicated as the noise since they
are more inconsistent between the model’s prediction and the ground truth.

To evaluate whether the above model-based metrics can distinguish our
simulated noise from clean data, we record the loss and el2n values for each
token of the 2,000 clean and Misaligned-Laser sentence pairs in the same data
setting of Section 3.2.2. Figure 3 shows the changes of loss and el2n value
distribution for the clean and noisy tokens in different training epochs.
First, we find both metrics can gradually distinguish clean and noisy data as
the training time increases. This indicates the effectiveness of the model’s
self-knowledge for distinguishing hard-to-detect data noise. Second, we find
that the el2n metric shows better distinguishability than loss, indicating the
importance of considering the whole model prediction distribution.

However, we point out two limitations of truncation methods relying on model-
based metrics: First, they overlook the increasing reliability of model
prediction by removing potential data noise from an early training time.
Second, they cannot avoid removing clean but useful data. As shown in Figure
3, there is still an undistinguishing area between noisy and clean
distribution.

##  4 Noise Self-Correction

To overcome the limitations of truncation methods in Section 3.3, we go a step
further and propose the self-correction method to gradually increase the trust
of model prediction distributions to correct the ground-truth data during
training. Overall, our method keeps the supervision signals from the training
data to avoid clean training information loss and also progressively trusts a
dynamic low entropy state of model prediction distribution to revise the data.
Our work is in line with the label correction in computer vision (Wang et al.,
2022; Lu and He, 2022).

##### New Target.

Consider conditional probability models $p_{\theta}(y|x)$ for machine
translation. Such models assign probabilities to a target sequence
$y=\left(y_{1},...,y_{T}\right)$ by factorizing it to the sum of log
probabilities of individual tokens $y_{i}$ from the vocabulary $V$. At each
training iteration, the model learns towards the ground-truth token
distribution, one-hot $q(y_{i})$, with a model prediction distribution
$p_{\theta}(\cdot|x,y_{\textless{i}})$. In self-correction, we leverage the
model prediction $p_{\theta}(\cdot|y_{\textless{i}},x)$ to revise the one-hot
distribution $q(y_{i})$ with the aim of learning towards a new target
$\bar{q}(y_{i})$:

$\bar{q}(y_{i})=(1-\lambda)q(y_{i})+\lambda{{p_{\theta}(\cdot|x,y_{\textless{i}})}}$
(2)

In this way, the new target $\bar{q}(y_{i})$ not only keeps the original
supervision signal from the training data but also the model’s prediction.
$\lambda$ denotes a weighting factor to determine how much to trust the model
prediction.

##### Dynamic Learning Schedule.

We set $\lambda$ correlated with a learning time function $\text{Time}(t)$ of
training iteration $t$ and model entropy $H(p_{\theta})$:

$\lambda=(1-H(p_{\theta}))\times\text{Time}(t)\\\ $ (3)

For $H(p_{\theta})$, the model trusts its own prediction more with a more
confident prediction, i.e., lower entropy. For $\text{Time}(t)$, it allows the
model to progressively trust its self-knowledge as training goes. We use a
schedule (Bengio et al., 2015) to increase $\text{Time}(t)$ as a function of
the training iteration $t$ and $T$ is the number of total iterations.

$\text{Time}(t)=\frac{1}{1+\exp(\beta(\frac{t}{T}+\alpha))}$ (4)

where $\alpha$ and $\beta$ are hyper-parameters.111 We set as $\alpha=-0.6$
and $\beta=-6$ to control the range of $\text{Time}(t)\in(0,1)$.

In general, at the beginning of training, the model is not well-trained, and a
small $\text{Time}(t)$ controls the model to rely more on the ground-truth
data than its own prediction. As training progresses, the increasing
$\text{Time}(t)$ allows the model to trust more in its own reliable
prediction.

##### Sharpen the Model Prediction.

To overcome the overly uncertain model prediction on learning towards the new
target in Equation 2, we sharpen the model prediction distribution by
controlling the softmax temperature $\tau$ in
$\bar{p_{\theta}}=\frac{\exp(z_{i}/\tau)}{\sum_{j=1}^{N}\exp(z_{j}/\tau)}$. We
control $\tau$ in a fixed way and a dynamic way to vary it inversely with
$\text{Time}(t)$. Therefore, $\tau$ gradually decreases as training time goes:
a higher $\tau$ at the early stage of training can prevent the model from
converging and a smaller $\tau$ in the later stage makes the model more
confident in its output. In Section 5, we compare the performance of both
fixed222We use fixed $\tau=0.5$ followed by Wang et al. (2022). and dynamic
$\tau$ to self-correct the data noise, and also show the impact of varying
$\tau$ in Section 5.2.3.

##### Training.

After acquiring a new target $\bar{q}(y_{i})$, derived from both the true data
and peakier model prediction distributions, we will obtain a new training
objective based on maximum likelihood estimation (MLE). The following loss
function is minimized for every training token over the training corpus $D$:

$L_{\theta}(x,y)=\mathbb{E}_{y_{i}\sim{D}}\left[-\bar{q}(y_{i})\log{{p}_{\theta}(\cdot|y_{\textless{i}}},x)\right]$
(5)

|  | Misaligned-Laser | Misaligned-Comet | Raw-Crawl Data
---|---|---|---|---
|  |  10% |  30% |  50% |  10% |  30% |  50% |  10% |  30% |  50%
1-11 Baseline |  with noise |  33.0 |  31.7 |  30.5 |  33.1 |  32.0 |  30.0 |  33.0 |  31.5 |  29.6
Oracle |  w/o noise |  33.3 |  32.7 |  32.0 |  33.3 |  32.7 |  32.0 |  33.3 |  32.7 |  32.0
1-11 Pre-Filter |  Laser |  33.2 |  31.4 |  30.0 |  33.1 |  32.6 |  30.2 |  33.0 |  31.8 |  30.0
Comet |  32.9 |  31.5 |  30.4 |  33.0 |  31.7 |  29.6 |  32.4 |  31.6 |  28.5
1-11 Truncation |  loss-frac |  33.1 |  31.4 |  30.7 |  33.0 |  31.2 |  29.8 |  33.0 |  31.8 |  29.9
el2n-frac |  33.0 |  31.9 |  31.0 |  32.0 |  31.8 |  29.0 |  33.0 |  31.6 |  30.0
el2n-threshold |  33.3 |  32.7 |  31.1 |  33.3 |  32.3 |  29.8 |  33.0 |  31.7 |  30.2
1-11 Self-Correction (Ours) |  fixed $\tau=0.5$ |  33.1 |  32.9* |  31.3* |  33.2 |  32.4* |  30.4* |  33.4* |  31.7* |  30.3*
dynamic $\tau$ |  33.5* |  32.3* |  31.4* |  33.3* |  32.5* |  30.6* |  33.5* |  31.9* |  30.4*

Table 3: BLEU scores of high-resource De $\rightarrow$ En translation task
with different types of noise. The BLEU score of the full clean training
corpus (5.8M) De $\rightarrow$ En is 33.5. Baseline with noise: represents the
translation performance when injecting with 10%, 30%, 50% of data noise.
Oracle w/o noise: represents the upper-bound translation performance when
training with the remaining clean data, specifically 90%, 70%, 50% of the data
excluding the noise. Bold and Underline represents the best and second best
score. $*$ signifies that our self-correction method is significantly better
(p-value < 0.05) than the baseline. The statistical significance results with
paired bootstrap resampling are followed by (Koehn, 2004). Chrf++ and Comet
score are provided in Table 9.

|  | Misaligned-Laser | Misaligned-Comet | Raw-Crawl Data
---|---|---|---|---
|  |  10% |  30% |  50% |  10% |  30% |  50% |  10% |  30% |  50%
1-11 Baseline |  with noise |  22.3 |  20.0 |  18.0 |  21.4 |  18.7 |  14.2 |  22.3 |  21.0 |  19.0
Oracle |  w/o noise |  22.3 |  21.0 |  20.8 |  22.3 |  21.0 |  20.8 |  22.3 |  21.0 |  20.8
1-11 Pre-Filter |  Laser |  22.0 |  18.7 |  17.0 |  21.1 |  18.9 |  16.3 |  21.0 |  21.2 |  19.2
Comet |  22.0 |  20.0 |  17.6 |  21.0 |  18.6 |  13.8 |  22.2 |  20.9 |  18.9
1-11 Truncation |  loss-frac |  22.1 |  20.5 |  17.9 |  20.0 |  17.2 |  14.2 |  22.2 |  21.1 |  19.1
el2n-frac |  22.0 |  20.5 |  18.2 |  21.1 |  18.9 |  14.3 |  22.0 |  21.3 |  19.2
el2n-threshold |  22.0 |  20.6 |  19.4 |  22.0 |  18.9 |  14.5 |  22.3 |  22.0 |  19.1
1-11 Self-Correction (Ours) |  fixed $\tau=0.5$ |  22.4 |  21.2* |  19.8* |  21.7* |  19.0* |  15.3* |  22.5* |  21.5* |  19.9*
dynamic $\tau$ |  22.3 |  20.3* |  20.2* |  22.1* |  19.6* |  16.2* |  22.3* |  21.9* |  19.6*

Table 4: BLEU scores of low-resource En $\rightarrow$ Si translation task with
different types of noise. The BLEU score of full clean training corpus (0.9M)
En $\rightarrow$ Si is 22.5. Chrf++ and Comet score are provided in Table 10.

##  5 Experiments

In this section, we investigate the effectiveness of our self-correction
method for translation in two experiment settings: (1) simulated noise
(Section 5.2), and (2) real-world noise (Section 5.3). For simulated noise, we
conduct experiments by injecting two types of noise, raw-crawl data and
simulated misaligned noise, into clean translation data. For the real-world
noise, we perform experiments on web-mined datasets, i.e., ParaCrawl and
CCAligned, across eight language pairs.

###  5.1 Comparing Systems

We compare our self-correction method with the following comparing systems.
Note that all the models’ details are in line with the corresponding
baselines. For pre-filter methods, the training data size is determined by the
number of sentence pairs to filter. For data truncation and our self-
correction method, both of them use the full training corpus.

##### Pre-Filtering.

We select two widely-used data filters: Laser and Comet. We rank the training
sentence pairs based on the scores calculated by the filter models. For the
simulated noise experiments (Section 5.2), we filter out the sentence pairs
with the lowest scores before training, matching the size to the injected data
noise. For the real-world noise experiments (Section 5.3), we filter out 20%
of the sentence pairs with the lowest scores.

##### Truncation.

We compare two truncation methods: (1) loss truncation (Kang and Hashimoto,
2020), (2) error norm value (el2n) truncation (Li et al., 2024). Following (Li
et al., 2024), we choose the best result among three truncation fractions
{0.05, 0.1, 0.2} for both loss-fraction and el2n-fraction truncation. We
select the best result among three threshold values {1.3, 1.35, 1.4} for
el2n-threshold truncation.333Followed by (Li et al., 2024), the starting
training iteration to truncate data is set as 1500.

###  5.2 Simulated Noisy World

####  5.2.1 Experiment Setup

We conduct experiments on both high- and low-resource translation tasks. We
use the WMT2017 (German) De$\rightarrow$En news translation data as the high-
resource task and En$\rightarrow$Si (Sinhala) from
OPUS444https://opus.nlpl.eu/ as the low-resource task. Following Herold et al.
(2022), we inject noise by replacing a portion (10%, 30%, 50%) of the clean
training corpus with simulated misalignment noise or raw crawl data. The
misalignment noise is generated by Algorithm 1 from the replaced portion of
the clean corpus. Each translation task randomly selects the raw crawl data
from the raw Paracrawl corpus555https://paracrawl.eu/. The raw data contains a
mixture of naturally occurring noise, including misaligned sentences, wrong
language, grammar errors, etc, and provides a realistic testbed for noise-
handling methods.

Our translation model uses the fairseq (Ott et al., 2019) implementation of
the Transformer-Big architecture for the high-resource task and Transformer-
Base for the low-resource task. The full training details are shown in
Appendix C.

Figure 4: Figures (Left/Middle): BLEU scores for misaligned-Laser (Left) and
clean (Middle) data from baseline and self-correction models trained on
De$\rightarrow$En task with 30% injected misaligned-Laser. Figure (Right):
BLEU scores from the self-correction models on De$\rightarrow$En task with 30%
different types of injected noise with varying $\tau$.

####  5.2.2 Results

Tables 3 and 4 show the high-resource De$\rightarrow$En and the low-resource
En$\rightarrow$Si translation performance trained on the corpus with
misalignment noise or raw crawl data. Overall, the baseline model’s BLEU
scores gradually drop with increasing levels of both simulated and real-world
noise.

When applying pre-filter methods under simulated noisy settings, they can
achieve modest gains for both simulated misalignment and the raw-crawl data.
Compared with the upper-bound oracle results when the model trained with the
remaining correct clean data, pre-filters notably fall short, especially in
the highly (50%) noisy setting. This performance discrepancy underscores the
challenge of pre-filters in accurately distinguishing simulated misalignment
noise from clean data, aligning with our findings on the low filter accuracy
for misalignment noise in Table 2.

For truncation methods, el2n truncation shows better performance gains than
loss truncation, indicating the effectiveness of using the whole model
prediction distribution to estimate data quality rather than the sole
prediction probability of the ground-truth token. However, with an increasing
level of noise, el2n truncation can only achieve minimal or even no
improvement, showing its limitation in distinguishing clean and noisy tokens
in a highly noisy training environment. In this case, removing tokens with
larger el2n values leads to a loss of clean and important training data.

Instead, our self-correction method consistently improves both low- and high-
resource translation performance under different controlled simulated noise
settings and outperforms other systems in most scenarios. Even when injecting
50% of noise, e.g., misaligned-Laser, our self-correction method still
improves over 2 BLEU points over the low-resource baseline, and nearly 1 BLEU
points over the high-resource baseline. This shows the necessity of keeping
the human reference information for the data sensitive scenarios, and the idea
of “revising” instead of “removing” can achieve this goal. Moreover, instead
of solely relying on flawed human reference, our method demonstrates the
effectiveness of using the system-generated translation. This finding supports
the hypothesis on the limited effectiveness on training models exclusively
towards the reference especially when it is inferior to the model generated
translation (Xu et al., 2024).

####  5.2.3 Analysis

##### The Impact of Self-Correction on Clean vs. Noisy Data.

Here, we aim to analyze whether the self-correction method revises the noisy
samples and also maintains the performance of clean parallel data. We sample
1K clean and misaligned-Laser sentence pairs from the De$\rightarrow$En
training corpus and record their BLEU scores during different training epochs
from baseline and self-correction model. For the misaligned-Laser, we
calculate the BLEU scores according to their parallel true references. Figure
4 (Left) shows that self-correction model can correct the misalignment noise
by improving BLEU points compared to the baseline model. In the meantime, the
self-correction model maintains the clean parallel data performance as the
baseline model without hurting the clean training information, shown in Figure
4 (Middle). This further explains the superiority of self-correction method
than pre-filter or data truncation from avoiding to discard the clean and
useful training data.

##### The Impact of Sharpening Model Prediction.

Here, we aim to analyze the impact of sharpening model prediction distribution
to correct the ground truth on translation performance. We train the self-
correction models on De$\rightarrow$En task with 30% of different types of
noise, with varying values of softmax temperature $\tau$. From figure 4
(right), we show that using sharpening model prediction distribution with a
smaller $\tau$ achieves better translation performance for all noisy settings.
However, the optimal $\tau$ varies when training with different types of
noise, and thus increases the difficulty of selecting a fixed $\tau$ for
different scenarios. This motivates us to design a dynamic $\tau$, which
varies automatically in a low range of entropy state over training time. The
overall performance in both Section 5.2 and Section 5.3 by using a dynamic
$\tau$ also shows its general applicability for different noise scenarios.

|  |  en$\rightarrow$fr$\heartsuit$ |  en$\rightarrow$ru$\heartsuit$ |  en$\rightarrow$tr† |  en$\rightarrow$es† |  en$\rightarrow$be† |  en$\rightarrow$ht† |  en$\rightarrow$si$\heartsuit$ |  en$\rightarrow$km$\heartsuit$ |  Avg.
---|---|---|---|---|---|---|---|---|---|---
Misaligned Rate (%) |  |  10% |  19% |  44% |  22% |  10% |  35% |  62% |  18% |  -
Corpus Size (M) |  |  5M |  5M |  5M |  5M |  1.1M |  0.55M |  0.21M |  0.06M |  -
Baseline |  |  41.1 |  26.5 |  23.5 |  21.6 |  9.9 |  17.0 |  7.0 |  4.2 |  18.8
1-10 Pre-Filter |  Laser |  41.8 |  26.8 |  23.2 |  22.5 |  9.8 |  17.1 |  6.6 |  3.8 |  19.0
Comet |  41.6 |  26.4 |  23.7 |  22.2 |  9.6 |  16.9 |  6.8 |  4.0 |  18.9
1-10 Truncation |  loss-frac |  41.2 |  26.0 |  23.8 |  21.9 |  9.8 |  16.9 |  6.0 |  4.0 |  18.7
el2n-frac |  41.3 |  26.1 |  23.9 |  22.0 |  10.0 |  16.8 |  6.0 |  4.5 |  18.8
el2n-threshold |  41.2 |  26.3 |  24.1 |  22.2 |  10.2 |  17.1 |  6.2 |  4.2 |  18.9
1-10 Self-Correction (Ours) |  fixed $\tau=0.5$ |  41.9* |  26.6 |  23.4 |  21.9* |  10.1* |  17.2* |  7.6* |  4.6* |  19.2*
dynamic $\tau$ |  42.3* |  26.7* |  24.2* |  22.8* |  10.5* |  17.4* |  7.8* |  5.0* |  19.6*

Table 5: BLEU scores on real-world web-mined corpora. Bold and Underline
represents the best and second best score. † denotes language pairs from
CCAligned V1.0. $\heartsuit$ denotes language pairs from ParaCrawl V7.1. $*$
signifies that our self-correction method is significantly better (p-value <
0.05) than the baseline. The misaligned noise rate for different language
pairs is reported from Kreutzer et al. (2022). Chrf++ and Comet score are
provided in Table 11.

###  5.3 Real Noisy World

####  5.3.1 Experiment Setup

We investigate two noisy web-crawled datasets: Paracrawl V7.1 and CCAligned
V1.0. These two datasets exhibit varying levels of semantic misalignment rates
across different language pairs (Kreutzer et al., 2022). For each dataset, we
select four language pairs from high- to low-resource with varying levels of
misaligned noise rate. For Paracrawl, the language pairs are:
en$\rightarrow$fr (French), en$\rightarrow$ru (Russian), en$\rightarrow$si
(Sinhala), and en$\rightarrow$km (Khmer). For CCAligned, the language pairs
are: en$\rightarrow$tr (Turkish), en$\rightarrow$es (Spanish),
en$\rightarrow$be (Belarusian), and en$\rightarrow$ht (Haitian). For the high-
resource language pairs: en$\rightarrow$fr, en$\rightarrow$ru,
en$\rightarrow$tr, en$\rightarrow$es, we randomly sample 5M sentence pairs as
the training corpus. For medium and low-resource language pairs, we use the
original corpus size. The validation and test sets for all tasks are from
Flores101.666https://github.com/facebookresearch/flores We train on the
Transformer-Big (Vaswani et al., 2017) architecture for all tasks.

####  5.3.2 Results

Table 5 shows the translation performance for two noisy web-crawled datasets,
CCAligned V1.0 and Paracrawl V7.1, across eight language pairs with varying
corpus size and misaligned rates.

Overall, our self-correction method consistently outperforms other alternative
methods, including pre-filter and data truncation, with up to 1.2 BLEU and 2.0
ChrF++ points improvement over the baseline. In line with the findings in the
simulated noisy experiment 5.2, we also show that pre-filter and truncation
methods can achieve performance gains for high-resource tasks while limited in
the low-resource tasks with high misalignment rate e.g., en$\rightarrow$ht,
en$\rightarrow$si and en$\rightarrow$km. Both of the pre-filter and truncation
cannot help and even hurt the low-resource translation performance. This
further emphasizes the importance of keeping ground-truth supervision for the
real-world noisy low-resource tasks. For our self-correction approach, it
shows superior performance especially for low-resource tasks with high
misaligned rates, indicating its effectiveness in handling real-world data
noise in different scenarios. Apart from data pruning, our method provides new
insight on handling text noise by correcting particularly for low-resource
tasks.

##  6 Conclusion

In this paper, we aim to address the data quality issues in the web-mined
parallel corpus by proposing a noise-handling self-correction approach for
machine translation tasks. Our approach focuses on the translation model’s
training dynamics and revises the training supervision from the ground-truth
data by the model’s prediction distribution.

To evaluate our approach in different noisy settings, we take a deep look at
the primary noise source in web-mined parallel data, i.e., semantic
misalignment. To quantitatively analyze the impact of real-world misalignment,
we propose a process to simulate the misalignment controlled by semantic
similarity. Under our simulated noisy training environment, we demonstrate
that our self-correction method consistently improves the baselines with
different levels of misalignment noise. Moreover, we also show that our self-
correction method can effectively handle naturally occurring noise in the
real-world noisy web-mined datasets across eight translation tasks,
outperforming other alternative approaches, including pre-filter and data
truncation. Our work also provides a critical finding on the effectiveness of
leveraging the system-generated translation instead of solely relying on
flawed reference data.

##  7 Limitation

First, we acknowledge that our work aims at learning with noisy training
corpus, therefore, improvements on high-quality training datasets might be
limited. Second, our work focuses on “training from scratch” for translation
models, while future works can apply our method to more NLP tasks and
settings, including the pre-training or fine-tuning stage for text generation.
Third, we only focus on the primary source of noise, i.e., misaligned
sentences, on machine translation. Even we show the effectiveness of our
approach on the real-world raw crawl data which contains naturally occurring
data noise, future work can fine-grained evaluate our approach on other types
of noise e.g., wrong language or misordered sentences.

## Acknowledgments

This research was funded in part by the Netherlands Organization for
Scientific Research (NWO) under project number VI.C.192.080. We would like to
thank Vlad Niculae, David Stap, Sergey Troshin and Evgeniia Tokarchuk for
their useful feedback and discussion.

## References

  * Artetxe and Schwenk (2018) Mikel Artetxe and Holger Schwenk. 2018.  Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.  _Transactions of the Association for Computational Linguistics_ , 7:597–610.
  * Bane et al. (2022) Fred Bane, Celia Soler Uguet, Wiktor Stribiżew, and Anna Zaretskaya. 2022.  A comparison of data filtering methods for neural machine translation.  In _Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 2: Users and Providers Track and Government Track)_ , pages 313–325, Orlando, USA. Association for Machine Translation in the Americas.
  * Bañón et al. (2020) Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020.  ParaCrawl: Web-scale acquisition of parallel corpora.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4555–4567, Online. Association for Computational Linguistics.
  * Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015.  Scheduled sampling for sequence prediction with recurrent neural networks.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.
  * Cui et al. (2013) Lei Cui, Dongdong Zhang, Shujie Liu, Mu Li, and Ming Zhou. 2013.  Bilingual data cleaning for SMT using graph-based random walk.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 340–345, Sofia, Bulgaria. Association for Computational Linguistics.
  * El-Kishky et al. (2020) Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020.  CCAligned: A massive collection of cross-lingual web-document pairs.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5960–5969, Online. Association for Computational Linguistics.
  * Flores and Cohan (2024) Lorenzo Jaime Flores and Arman Cohan. 2024.  On the benefits of fine-grained loss truncation: A case study on factuality in summarization.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 138–150, St. Julian’s, Malta. Association for Computational Linguistics.
  * Goyal et al. (2022) Tanya Goyal, Jiacheng Xu, Junyi Jessy Li, and Greg Durrett. 2022.  Training dynamics for text summarization models.  In _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 2061–2073. Association for Computational Linguistics.
  * Herold et al. (2022) Christian Herold, Jan Rosendahl, Joris Vanvinckenroye, and Hermann Ney. 2022.  Detecting various types of noise for neural machine translation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2542–2551, Dublin, Ireland. Association for Computational Linguistics.
  * Kang and Hashimoto (2020) Daniel Kang and Tatsunori B. Hashimoto. 2020.  Improved natural language generation via loss truncation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 718–731, Online. Association for Computational Linguistics.
  * Kepler et al. (2019) Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019.  OpenKiwi: An open source framework for quality estimation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 117–122, Florence, Italy. Association for Computational Linguistics.
  * Khayrallah and Koehn (2018) Huda Khayrallah and Philipp Koehn. 2018.  On the impact of various types of noise on neural machine translation.  In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_ , pages 74–83, Melbourne, Australia. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_ , San Diega, CA, USA.
  * Koehn (2004) Philipp Koehn. 2004.  Statistical significance tests for machine translation evaluation.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 388–395, Barcelona, Spain. Association for Computational Linguistics.
  * Kreutzer et al. (2022) Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2022.  Quality at a glance: An audit of web-crawled multilingual datasets.  _Transactions of the Association for Computational Linguistics_ , 10:50–72.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Lee et al. (2022) En-Shiun Lee, Sarubi Thillainathan, Shravan Nayak, Surangika Ranathunga, David Adelani, Ruisi Su, and Arya McCarthy. 2022.  Pre-trained multilingual sequence-to-sequence models: A hope for low-resource language translation? In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 58–67, Dublin, Ireland. Association for Computational Linguistics.
  * Li et al. (2024) Tianjian Li, Haoran Xu, Philipp Koehn, Daniel Khashabi, and Kenton Murray. 2024.  Error norm truncation: Robust training in the presence of data noise for text generation models.  In _The Twelfth International Conference on Learning Representations_.
  * Lu and He (2022) Yangdi Lu and Wenbo He. 2022.  Selc: Self-ensemble label correction improves learning with noisy labels.  In _International Joint Conference on Artificial Intelligence_.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Peter et al. (2023) Jan-Thorsten Peter, David Vilar, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, and Markus Freitag. 2023.  There’s no data like better data: Using QE metrics for MT data filtering.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 561–577, Singapore. Association for Computational Linguistics.
  * Ranathunga et al. (2024) Surangika Ranathunga, Nisansa De Silva, Velayuthan Menan, Aloka Fernando, and Charitha Rathnayake. 2024.  Quality does matter: A detailed look at the quality and utility of web-mined parallel corpora.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 860–880, St. Julian’s, Malta. Association for Computational Linguistics.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Schwenk and Douze (2017) Holger Schwenk and Matthijs Douze. 2017.  Learning joint multilingual sentence representations with neural machine translation.  In _Rep4NLP@ACL_.
  * Taghipour et al. (2011) Kaveh Taghipour, Shahram Khadivi, and Jia Xu. 2011.  Parallel corpus refinement as an outlier detection algorithm.  In _Proceedings of Machine Translation Summit XIII: Papers_ , Xiamen, China.
  * team et al. (2022) Nllb team, Marta Ruiz Costa-jussà, James Cross, Onur cCelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Alison Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon L. Spruit, C. Tran, Pierre Yves Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm’an, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022.  No language left behind: Scaling human-centered machine translation.  _ArXiv_ , abs/2207.04672.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
  * Wang et al. (2018) Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. 2018.  Denoising neural machine translation training with trusted data and online data selection.  In _Conference on Machine Translation_.
  * Wang et al. (2022) Xinshao Wang, Yang Hua, Elyor Kodirov, Sankha Subhra Mukherjee, David A. Clifton, and Neil Martin Robertson. 2022.  Proselflc: Progressive self label correction towards a low-temperature entropy state.  _bioRxiv_.
  * Xu et al. (2024) Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024.  Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation.  _Preprint_ , arXiv:2401.08417.
  * Zaragoza-Bernabeu et al. (2022) Jaume Zaragoza-Bernabeu, Gema Ramírez-Sánchez, Marta Bañón, and Sergio Ortiz Rojas. 2022.  Bicleaner AI: Bicleaner goes neural.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 824–831, Marseille, France. European Language Resources Association.

##  Appendix A Data Filters

Comet is a neural framework for training machine translation evaluation models
that can function as metrics Rei et al. (2020). Their framework uses cross-
lingual pre-trained language modeling that exploits information from both
source input and target reference to predict the target translation quality.
We use the reference-free wmt-20-qe-da Comet model as the data filter to score
each sentence pair in the training corpus.

For Laser, it scores sentence pairs based on cross-lingual sentence
embeddings. To calculate the Laser score for each sentence pair, we generate
cross-lingual sentence embeddings using the pre-trained model from Artetxe and
Schwenk (2018). The underlying system is trained as a multilingual translation
system with a multi-layer bidirectional LSTM encoder and an LSTM decoder,
without information about the input language on the encoder. The output
vectors of the encoder are compressed into a single embedding of fixed length
using max-pooling, which is the cross-lingual sentence embedding resulting
from the Laser model. The assumption is that two sentences with the same
meaning but from different languages will be mapped onto the same embedding
vectors. We calculate the cosine similarity score between source and target
sentence embeddings as the Laser score. The higher the Laser score, the more
semantic similar between the source and target sentence.

For Bi-Cleaner, it is a tool in Python that aims at detecting noisy sentence
pairs in a parallel corpus. It indicates the likelihood of a pair of sentences
being mutual translations. Sentence pairs considered high-quality are scored
near with 1 and considered as noisy are scored with 0. We use the multilingual
model bitextor/bicleaner-ai-full-en-xx from
HuggingFace777https://huggingface.co/bitextor/bicleaner-ai-full-en-xx for the
pre-filter for all language tasks.

##  Appendix B Controlled Generated Misaligned Noise

###  B.1 Algorithm

Algorithm 1 shows the method to generate misaligned noise, controlled by two
steps: (1) surface-level features control by word overlap and sentence length;
(2) quality control by Laser or Comet.

Algorithm 1 Misaligned Noise Generation

Input: A chunk of parallel and de-duplicate clean data $D$ with N sentence
pairs, source and target $(S,T)$; A threshold $k$ for selecting misaligned
candidates; A quality controlled model $M\in\\{\text{Laser},\text{Comet}\\}$
Output: Misaligned data $\bar{D}$ with N sentence pairs source and misaligned
target $(S,\bar{T})$.

for each source sentence $s_{i}$ in $S$ do

Step 1: Surface-level Features Control

Initialize a list $L$ of misaligned candidates for $s_{i}$

for each target sentence $t_{j(j\neq i)}$ in $T$ do

if $\operatorname{len}(L)<k$ then

if $|\operatorname{len}(t_{j})-\operatorname{len}(s_{i})|<3$ and $\text{word
overlap ratio}(t_{j},t_{i})>0.4$ then

Append $t_{j}$ to list $L$

end if

end if

end for

Step 2: Quality Control

Initialize a quality score list $Q$

for each candidate $t_{n}$ in $L$ do

$\operatorname{score}(s_{i},t_{n})=M(s_{i},t_{n})$

Append $\operatorname{score}$ to list $Q$

end for

Select $t_{k}$ from $L$ with the highest score in $Q$

Append the pair $(s_{i},t_{k})$ to the misaligned data $\bar{D}$

end for

###  B.2 Misaligned Noise Samples

Table 6 shows the simualted misaligned samples of misaligned-laser and
misaligned-comet. Overall, the simulated misaligned noise controlled by
external models all share certain amounts of semantic meanings compared with
the true reference.

src |  der Rat kam überein, dass die Kommission die Anwendung dieser Verordnung mit dem Ziel überwacht, etwaige Probleme möglichst schnell festzustellen und zu regeln.
---|---
1-2 ref |  the Council agreed that the Commission will keep under review the implementation of this Regulation with a view to detecting and addressing any difficulties as soon as possible.
1-2 Mis-Laser |  the Commission has therefore acted wisely in exploring every possible avenue to guard against any difficulties and to prepare for any eventualities.
src |  Brüssel , 17 März 2015
1-2 ref |  Brussels , 17 March 2015
Mis-Laser |  Brussels , 4 May 2011
src |  wann möchten Sie im Aeolos Hotel übernachten?
1-2 ref |  when would you like to stay at the Aeolos Hotel?
1-2 Mis-Laser |  when would you like to stay at the Leenane Hotel?
src |  buchen Sie Ihre Unterkunft in Edinburgh today!
1-2 ref |  book your accommodation in Edinburgh today!
1-2 Mis-Comet |  book your accommodation in Amsterdam today!
src |  wir akzeptieren folgende Kreditkarten:Visa, Maestro, Master Card, American Express, JBC, Dinners Club.
1-2 ref |  We accept the following credit cards: Visa, Maestro, Master Card, American Express, JBC, Dinners Club.
Mis-Comet |  we accept payments by credit card (Visa, MasterCard, Diners Club), Paypal or transfer.
src |  Puchacz Puchacz Spa befindet sich in Niechorze , in einer schönen und malerischen Umgebung , ist lediglich 150m vom Meer entfernt und liegt in der Nähe des Liwia Łuża Sees .
ref |  Puchacz Puchacz Spa is located in Niechorze, in a beautiful and picturesque setting, only 150m from the sea and close to Lake Liwia Łuża.
Mis-Comet |  the Country Hotel Sa Talaia, surrounded by beautiful gardens is located close to San Antonio city and not far away from the historic city of Ibiza

Table 6: Simulated Misaligned Sentences Samples

###  B.3 Human Evaluation of Misaligned Noise

The aim of this data annotation is to evaluate the adequacy of real-world and
simulated misaligned noise. Given a source and misaligned target, the
annotator should select the overlap meanings between them, as shown in Table
7. The simulated misaligned sentence pairs are constructed from the clean
corpus WMT2017 De$\rightarrow$En and the real-world misaligned sentences are
selected from web-mined Paracrawl datasets. The language pair we select here
is German into English, and therefore we select the human annotator who are
sufficient in both German and English languages.

Questionnaire
---
Whether this target translation conveys the same meanings as the source
sentence?
$\circ$ all meanings $\circ$ most meanings $\circ$ much meanings $\circ$
little meanings no meanings
Table 7: Questionnaire for human evaluation, where $\circ$ indicate single-
item selection. From all meanings to no meanings, the adequacy score scales
from 5–1.

Translation Task | Training Source | Dev Set | Test Set
---|---|---|---
De$\rightarrow$En | WMT2017 (5.8M) | NewsTest2016 | NewsTest2017
En$\rightarrow$Si | OPUS (0.9M) | OPUS | OPUS

Table 8: The clean training corpus and evaluation dataset details for the
simulated noise experiments.

##  Appendix C Training Details.

###  C.1 Training and Evaluation

We follow the setup of the Transformer-base and Transformer-big models (Bengio
et al., 2015). For each model, the number of layers in the encoder and in the
decoder is $N=6$. For Transformer-base, we employ $h=8$ parallel attention
layers or heads. The dimensionality of input and output is
$d_{\text{model}}=512$, and the inner layer of feed-forward networks has
dimensionality $d_{\text{ff}}=2048$. For Transformer-big, we employ $h=16$
parallel attention layers or heads. The dimensionality of input and output is
$d_{\text{model}}=1024$, and the inner layer of feed-forward networks has
dimensionality $d_{\text{ff}}=4096$.

All models are trained with the Adam optimizer (Kingma and Ba, 2015) for up to
500K steps for high-resource tasks and 100K steps for low-resource tasks, with
a learning rate of 5e-4 and an inverse square root scheduler. A dropout rate
of 0.3 and label smoothing of 0.2 are used. Each model is trained on one
NVIDIA A6000 GPU with a batch size of 25K tokens. We choose the best
checkpoint according to the average validation loss of all language pairs. The
data is tokenized with the SentencePiece tool (Kudo and Richardson, 2018) and
we build a shared vocabulary of 32K tokens. For evaluation, we employ beam
search decoding with a beam size of 5. BLEU scores are computed using
detokenized case-sensitive SacreBLEU.
888nrefs:1—case:mixed—eff:no—tok:13a—smooth:exp—version:2.3.1

###  C.2 Dataset Details

Table 8 shows the training and evaluation dataset details for clean training
corpus in simulated noisy experiments in Section 5.2.

##  Appendix D Chrf++ and Comet Scores

Table 9, 10, and 11 shows the Comet and Chrf++ scores for all experiments.

|  | COMET
---|---|---
1-11 |  | Misaligned-Laser | Misaligned-Comet | Raw-Crawl Data
|  |  10% |  30% |  50% |  10% |  30% |  50% |  10% |  30% |  50%
1-11 Baseline |  with noise |  77.8 |  77.0 |  76.1 |  77.6 |  76.5 |  75.5 |  77.9 |  77.1 |  75.8
Oracle |  w/o noise |  79.5 |  79.0 |  78.6 |  79.5 |  79.0 |  78.6 |  79.5 |  79.0 |  78.6
1-11 Pre-Filter |  Laser |  78.0 |  76.9 |  75.6 |  78.2 |  78.0 |  76.0 |  78.0 |  77.8 |  76.9
Comet |  77.9 |  77.5 |  76.3 |  77.5 |  76.3 |  74.0 |  78.0 |  76.8 |  75.6
1-11 Truncation |  loss-frac |  78.3 |  76.5 |  76.2 |  78.0 |  76.3 |  75.0 |  78.0 |  77.2 |  76.6
el2n-frac |  78.3 |  78.3 |  76.5 |  78.1 |  76.1 |  76.0 |  78.2 |  77.5 |  76.2
el2n-threshold |  78.5 |  78.7 |  76.7 |  78.1 |  76.6 |  75.4 |  78.5 |  77.9 |  76.6
1-11 Self-Correction (Ours) |  fixed $\tau=0.5$ |  79.0 |  78.5 |  76.8 |  78.5 |  77.6 |  76.2 |  78.8 |  78.1 |  76.5
dynamic $\tau$ |  79.1 |  78.6 |  77.0 |  78.7 |  77.7 |  76.6 |  79.0 |  78.3 |  77.0
|  | Chrf++
1-11 Baseline |  with noise |  55.5 |  54.9 |  54.1 |  55.1 |  54.7 |  52.5 |  55.0 |  54.9 |  53.6
Oracle |  w/o noise |  57.2 |  56.9 |  55.5 |  57.2 |  56.9 |  55.5 |  57.2 |  56.9 |  55.5
1-11 Pre-Filter |  Laser |  56.5 |  54.5 |  53.4 |  56.3 |  56.0 |  52.6 |  55.2 |  55.0 |  54.3
Comet |  56.0 |  54.2 |  53.0 |  55.0 |  54.2 |  51.9 |  55.2 |  54.2 |  52.8
1-11 Truncation |  loss-frac |  56.0 |  54.3 |  54.2 |  55.5 |  54.1 |  52.0 |  55.5 |  55.0 |  54.0
el2n-frac |  56.1 |  55.2 |  54.2 |  55.5 |  55.0 |  52.0 |  56.2 |  55.0 |  54.2
el2n-threshold |  56.5 |  56.3 |  54.6 |  56.2 |  55.0 |  52.8 |  56.2 |  55.5 |  54.0
1-11 Self-Correction (Ours) |  fixed $\tau=0.5$ |  56.8* |  56.5* |  54.3* |  56.6* |  55.2* |  52.8* |  56.6* |  55.5* |  54.0*
dynamic $\tau$ |  56.9* |  56.2* |  54.6* |  56.4* |  55.6* |  53.0* |  56.7* |  55.8* |  54.9*

Table 9: Comet and Chrf++ scores of high-resource De $\rightarrow$ En
translation task with different types of noise. The Comet score of full clean
training corpus (5.8M) De $\rightarrow$ En is 80.0. The Chrf++ score of full
clean training corpus (5.8M) De $\rightarrow$ En is 57.2.

|  | COMET
---|---|---
1-11 |  | Misaligned-Laser | Misaligned-Comet | Raw-Crawl Data
|  |  10% |  30% |  50% |  10% |  30% |  50% |  10% |  30% |  50%
1-11 Baseline |  with noise |  79.8 |  79.0 |  77.8 |  79.7 |  75.9 |  71.6 |  79.7 |  79.5 |  78.3
Oracle |  w/o noise |  79.8 |  79.4 |  78.9 |  79.8 |  79.4 |  78.9 |  79.8 |  79.4 |  78.9
1-11 Pre-Filter |  Laser |  79.5 |  78.5 |  77.0 |  79.5 |  76.2 |  74.7 |  79.8 |  79.8 |  79.0
Comet |  79.6 |  78.8 |  76.8 |  79.2 |  76.0 |  71.0 |  79.5 |  79.0 |  77.8
1-11 Truncation |  loss-frac |  79.9 |  78.4 |  78.0 |  79.0 |  75.6 |  71.2 |  79.8 |  79.4 |  78.6
el2n-frac |  80.1 |  79.1 |  78.2 |  79.8 |  76.2 |  72.3 |  79.9 |  79.5 |  78.8
el2n-threshold |  80.0 |  79.0 |  78.5 |  79.7 |  76.5 |  73.5 |  80.0 |  80.0 |  79.0
1-11 Self-Correction (Ours) |  fixed $\tau=0.5$ |  80.3 |  79.0 |  78.5 |  79.9 |  77.0 |  74.0 |  80.3 |  79.8 |  79.5
dynamic $\tau$ |  80.1 |  79.2 |  78.8 |  79.9 |  77.1 |  74.6 |  80.2 |  80.1 |  79.2
|  | Chrf++
1-11 Baseline |  with noise |  35.7 |  34.0 |  33.0 |  34.9 |  30.1 |  24.2 |  35.6 |  34.0 |  32.7
Oracle |  w/o noise |  35.9 |  34.6 |  34.2 |  35.9 |  34.6 |  34.2 |  35.9 |  34.6 |  34.2
1-11 Pre-Filter |  Laser |  35.4 |  33.2 |  32.5 |  35.4 |  31.2 |  28.0 |  35.7 |  34.2 |  33.0
Comet |  35.4 |  33.5 |  32.6 |  33.6 |  29.5 |  23.8 |  35.4 |  33.8 |  32.5
1-11 Truncation |  loss-frac |  35.8 |  33.6 |  33.2 |  35.3 |  30.2 |  25.8 |  35.7 |  34.2 |  32.8
el2n-frac |  35.6 |  34.1 |  33.3 |  35.6 |  30.4 |  26.0 |  35.6 |  34.1 |  32.8
el2n-threshold |  35.7 |  34.2 |  33.5 |  36.0 |  30.8 |  26.5 |  35.6 |  35.2 |  33.0
1-11 Self-Correction (Ours) |  fixed $\tau=0.5$ |  36.0* |  34.3* |  33.3* |  35.5* |  31.0* |  27.0* |  36.0* |  34.8* |  33.8*
dynamic $\tau$ |  35.8 |  34.4* |  33.6* |  35.8* |  31.5* |  28.3* |  35.8* |  35.0* |  33.4*

Table 10: Comet and Chrf++ scores of low-resource En $\rightarrow$ Si
translation task with different types of noise. The Comet score of full clean
training corpus (0.9M) En $\rightarrow$ Si is 82.0. The Chrf++ score of full
clean training corpus (0.9M) En $\rightarrow$ Si is 37.0.

|  | COMET
---|---|---
|  |  en$\rightarrow$fr$\heartsuit$ |  en$\rightarrow$ru$\heartsuit$ |  en$\rightarrow$tr† |  en$\rightarrow$es† |  en$\rightarrow$be† |  en$\rightarrow$ht† |  en$\rightarrow$si$\heartsuit$ |  en$\rightarrow$km$\heartsuit$ |  Avg.
Misaligned Rate (%) |  |  10% |  19% |  44% |  22% |  10% |  35% |  62% |  18% |  -
Corpus Size (M) |  |  5M |  5M |  5M |  5M |  1.1M |  0.55M |  0.21M |  0.06M |  -
Baseline |  |  80.0 |  79.2 |  82.0 |  76.5 |  68.3 |  63.2 |  59.6 |  73.6 |  72.8
1-10 Pre-Filter |  Laser |  81.0 |  81.0 |  81.3 |  76.7 |  67.4 |  63.2 |  59.7 |  73.6 |  73.0
Comet |  80.5 |  79.1 |  81.0 |  76.0 |  68.9 |  63.0 |  59.5 |  73.2 |  72.7
1-10 Truncation |  loss-frac |  81.0 |  79.0 |  82.2 |  76.8 |  67.6 |  63.1 |  59.0 |  73.0 |  72.7
el2n-frac |  80.2 |  79.1 |  82.1 |  76.2 |  68.6 |  63.0 |  60.0 |  72.8 |  72.8
el2n-threshold |  81.4 |  81.3 |  82.3 |  76.8 |  68.7 |  63.3 |  60.7 |  73.0 |  73.4
1-10 Self-Correction |  fixed $\tau=0.5$ |  81.2 |  79.6 |  82.5 |  76.4 |  68.4 |  63.5 |  63.0 |  74.5 |  73.6
dynamic $\tau$ |  81.6 |  80.6 |  83.0 |  77.9 |  68.5 |  63.6 |  63.3 |  75.0 |  74.2
|  | ChrF++
Baseline |  |  67.3 |  50.6 |  54.8 |  49.1 |  36.5 |  45.4 |  20.2 |  15.6 |  42.4
1-10 Pre-Filter |  Laser |  67.9 |  51.5 |  54.6 |  49.6 |  36.1 |  45.1 |  21.7 |  14.7 |  42.7
Comet |  67.6 |  50.9 |  54.3 |  49.6 |  36.2 |  44.8 |  20.6 |  15.0 |  42.4
1-10 Truncation |  loss-frac |  67.4 |  50.4 |  55.2 |  49.2 |  36.3 |  45.3 |  20.0 |  13.4 |  42.2
el2n-frac |  67.6 |  50.6 |  55.2 |  49.5 |  36.5 |  45.2 |  20.6 |  13.0 |  42.3
el2n-threshold |  67.3 |  51.1 |  55.3 |  49.3 |  36.8 |  45.6 |  20.4 |  13.9 |  42.5
1-10 Self-Correction |  fixed $\tau=0.5$ |  68.0* |  50.9* |  54.9* |  49.6* |  36.8* |  45.6* |  24.0* |  16.5* |  43.2*
dynamic $\tau$ |  68.2* |  51.3* |  55.4* |  50.0* |  37.2* |  46.2* |  22.2* |  16.8 * |  43.4*

Table 11: Comet and Chrf++ scores on real-world web-mined corpora. For pre-
filter methods, we remove 20% of the training samples with the lowest scores.
† denotes language pairs from CCAligned V1.0. $\heartsuit$denotes language
pairs from ParaCrawl V7.1. The misaligned noise rate for different language
pairs is reported from Kreutzer et al. (2022).


# The distance and internal composition of the neutron star in EXO 0748$-$676
with XMM-Newton

Guobao Zhang1, Mariano Méndez1 , Peter Jonker2,3,4 and Beike Hiemstra1.
1Kapteyn Astronomical Institute, University of Groningen, P.O. BOX 800, 9700
AV Groningen, The Netherlands
2SRON, Netherlands Institute for Space Research, Sorbonnelaan 2, 3584 CA,
Utrecht, The Netherlands
3Harvard–Smithsonian Center for Astrophysics, 60 Garden Street, Cambridge, MA
02138, U.S.A.
4Department of Astrophysics, IMAPP, Radboud University Nijmegen, PO Box 9010,
NL-6500 GL Nijmegen, the Netherlands
E-mail<EMAIL_ADDRESS>
(Accepted. Received; in original form)

###### Abstract

Recently, the neutron star X-ray binary EXO 0748-676 underwent a transition to
quiescence. We analyzed an XMM-Newton observation of this source in
quiescence, where we fitted the spectrum with two different neutron-star
atmosphere models. From the fits we constrained the allowed parameter space in
the mass-radius diagram for this source for an assumed range of distances to
the system. Comparing the results with different neutron-star equations of
state, we constrained the distance to EXO 0748-676. We found that the EOS
model ’SQM1’ is rejected by the atmosphere model fits for the known distance,
and the ’AP3’ and ’MS1’ is fully consistent with the known distance.

###### keywords:

stars: neutron — X-rays: binaries — dense matter: equation of state — stars:
individual: EXO 0748$-$676

Accepted. Received; in original form

##  1 Introduction

The low-mass X-ray binary (LMXB) EXO 0748–676 was discovered as a transient
source with the European X-ray Observatory Satellite (EXOSAT) in 1985 (Parmar
et al., 1986). The source exhibits simultaneous X-ray and optical eclipses
from which the orbital period of P = 3.82 hr was deduced (Crampton et al.,
1986). EXO 0748$-$676 also exhibited irregular X-ray dipping activity (Parmar
et al., 1986), and type-I X-ray bursts (Gottwald et al., 1986). Burst
oscillations in EXO 0748$-$676 were first reported by Villarreal & Strohmayer
(2004) at 45 Hz in the average Fourier Power Spectrum of 38 type-I X-ray
bursts; the 45-Hz signal was then interpreted as the spin frequency of the
neutron star. Recently, Galloway et al. (2009) detected millisecond
oscillations in the rising phase of two type-I X-ray bursts in EXO 0748-676 at
a frequency of 552 Hz. They concluded that the spin frequency of EXO 0748-676
is close to 522 Hz, rather than 45 Hz as suggested by Villarreal & Strohmayer
(2004). The 45 Hz oscillation may arise in the boundary layer between the disk
and the neutron star (Balman, 2009) or it could be a statistical fluctuation
(Galloway et al., 2009). Cottam, Paerels & Méndez (2002) reported a
measurement of the gravitational redshift from iron and oxygen X-ray
absorption lines arising from the atmosphere of the neutron star in EXO
0748$-$676 during type-I X-ray bursts, but subsequent observations failed to
confirm these features (Cottam et al., 2008). Based on the gravitational
redshift, Özel (2006) suggested that the mass, radius and distance of EXO
0748$-$676 are $2.10\pm 0.28$ ${\rm M}_{\odot}$, $13.8\pm 1.8$ km and $9.2\pm
1.0$ kpc, respectively, which would rule out many neutron-star equations of
state.

Measuring the distance to LMXBs is difficult, except for the case of sources
in globular clusters. A way to get the distance is using type-I X-ray bursts.
The peak flux for some very bright bursts can reach the Eddington luminosity
at the surface of the neutron star. From a strong X-ray burst, Wolff et at.
(2005) derived a distance to EXO 0748$-$676 of 7.7 kpc for a helium-dominated
burst photosphere, and 5.9 kpc for a hydrogen-dominated burst photosphere.
Galloway et al. (2008a) analyzed several type-I X-ray bursts from EXO
0748$-$676 and estimated a distance of 7.4 kpc, different from the value of
9.2 kpc reported by Özel (2006). Taking into account the touchdown flux and
high-inclination in EXO 0748$-$676 , recently Galloway et al. (2008b) gave a
distance of 7.1 $\pm$ 1.2 kpc.

Another way to get the distance to an LMXB is through observations of
quiescent X-ray emission from the neutron-star surface. During the quiescent
state, X-ray emission originates from the atmosphere of the neutron star. By
fitting the X-ray spectrum of the neutron-star system with hydrogen atmosphere
models, one can estimate the mass, radius and distance of the neutron star.
Recently, the neutron-star X-ray transient EXO 0748$-$676 underwent a
transition into quiescence (Degenaar et al., 2009; Bassa et al., 2009).

In this paper, we report on the distance to EXO 0748$-$676 that we constrained
from XMM-Newton data. We use two different neutron-star atmophere models to
fit the X-ray spectrum, and compare the results of the spectral fitting with
different neutron-star equation of state (EOS). In the next section, we
describe the observation and data analysis. We show the fitting results in §3,
and we discuss our findings in §4.

##  2 OBSERVATIONS AND DATA ANALYSIS

EXO 0748$-$676 was observed with the European Photon Imaging Camera (EPIC PN
and MOS) on board the XMM-Newton on 2008 November 6 at 08:30:03 UTC (obsID
0560180701). The PN and the two MOS cameras were operated in Full-Window mode.
We reduced the XMM-Newton Observation Data Files (ODF) using version 8.0.0 of
the science analysis software (SAS). We used the epproc and emproc tasks to
extract the event files for the PN and the two MOS cameras, respectively.
Source light curves and spectra were extracted in the 0.2 $-$ 12.0 keV band
using a circular extraction region with a radius of 30 arcsec centered on the
position of the source. Background light curves and spectra were extracted
from a circular source-free region of 35 arcsec source-free on the same CCD.
We applied standard filtering and examined the light curves for background
flares. No flares were present and we used the whole exposure for our
analysis. The exposure time for the PN camera was 24.2 ks, and for each MOS
camera was 29.03 ks. The source count rate was $0.496\pm 0.005$ cts/s for PN,
and $0.135\pm 0.002$ cts/s and $0.127\pm 0.002$ cts/s for MOS1 and MOS2,
respectively. We checked the filtered event files for photon pile-up by
running the task epatplot. No pile-up was apparent in the PN, MOS1 and MOS2
data. The photon redistribution matrices and ancillary files for the source
spectra were created using the SAS tools rmfgen and arfgen, respectively. We
rebinned the source spectra using the tool pharbn111M. Guainazzi, private
communication, such that the number of bins per resolution element of the PN
and MOS spectra was 3 and the minimum number of counts per channel was 20.

We fitted the PN and MOS spectra simultaneously in the 0.5$-$10.0 keV range
with XSPEC 12.50 (Arnaud, 1996), using either of two neutron-star hydrogen-
atmosphere models: NSAGRAV (Zavlin et al., 1996) and NSATMOS (Heinke et al.,
2006). The NSAGRAV model provides the spectra emitted from a nonmagnetic
hydrogen atmosphere of a neutron star with surface gravitational acceleration,
$g$, ranging from $10^{13}$ to $10^{15}$ cm s-2. This model uses the mass
($M_{\rm NS}$) and radius ($R_{\rm NS}$) of the neutron star and the
unredshifted effective temperature of the surface of the star ($kT_{\rm eff}$)
as parameters. The normalization of the model is defined as $1/D^{2}$, where
$D$ is the distance to the source in pc. The second model that we used,
NSATMOS, includes a range of surface gravities and effective temperatures, and
incorporates thermal electron conduction and self-irradiation by photons from
the compact object. This model assumes negligible magnetic fields (less than
$10^{9}$ G) and a pure hydrogen atmosphere. NSATMOS parameters are $M_{\rm
NS}$, $R_{\rm NS}$, log$T_{\rm eff}$ (the same as for NSAGRAV), distance in
kpc, and a separate normalization $K$, which corresponds to the fraction of
the neutron-star surface that is emitting. We fixed $K$ to be 1 in all our
fits with NSATMOS.

Figure 1: XMM-Newton PN (black), MOS1 (red) and MOS2 (green) spectrum of EXO
0748$-$676 in the 0.5$-$ 10.0 keV energy band. The spectrum was fitted with a
neutron-star hydrogen atmosphere model (NSATMOS) and a power-law model with
$\Gamma$ fixed to 1. The lower panel shows the residuals to the best-fit
model.

We included the effect of interstellar absorption using PHABS assuming cross-
sections of Balucinska-Church & McCammon (1992) and solar abundances from
Anders & Grevesse (1989), and we let $N_{\rm H}$, column density along the
line of sight free to vary during the fitting. In order to account for
differences in effective area between the different cameras, we introduced a
multiplicative factor in our model. First, this factor was fixed to unity for
PN and free for MOS1 and MOS2. Then, we set the scaling factor to unity for
MOS1 and MOS2, respectively, and set the factor free for the other cameras. We
found that, fixing the scaling factor for different cameras gives similar
best-fit results. Therefore in the rest of the paper we fixed the factor to be
1 for PN and free to vary for the other cameras. None of the atmosphere models
alone fitted the spectrum above $\sim 2-3$ keV properly. Adding a power-law
component improved the fits significantly, however, all parameters were less
constrained than when fitting the data with the neutron-star atmosphere model
only. We first fixed the power-law index to 0.5, 1.0 and 1.5 to get better
constraints on the parameters of the neutron-star atmosphere model (Degenaar
et al. 2009). Further, we initially fixed the distance to the NS at 7.1 kpc,
which is the value inferred from the touchdown flux of Galloway et al. (2008b)
.

##  3 RESULTS

###  3.1 Results from the spectral fits

Table 1: Best-fit parameters of neutron-star atmosphere models fit to the XMM-
Newton data of EXO 0748$-$676 .

model | NH | $T^{\infty}_{\rm eff}$ | $M_{\rm NS}$ | $R_{\rm NS}$ | $\Gamma$ | $F_{pow}$ | $F_{X}$ |  $\chi^{2}$/d.o.f.
---|---|---|---|---|---|---|---|---
| ($10^{20}cm^{-2}$) | (eV) | (${\rm M}_{\odot}$) | (km) |  | 10-13 ergs cm-2 s-1 | 10-12 ergs cm-2 s-1 |
NSAGRAV | $5.6\pm 1.8$ | $113^{+14}_{-8}$ | $1.55\pm 0.18$ | $15.2\pm 1.8$ | 0.5 | $1.15\pm 0.21$ | $1.18\pm 0.15$ | 0.986/219
NSATMOS | $5.4\pm 1.5$ | $113\pm 4$ | $1.29\pm 0.20$ | $16.1^{+0.9}_{-1.2}$ | 0.5 | $1.17\pm 0.20$ | $1.23\pm 0.16$ | 0.985/219
NSAGRAV | $6.2^{+1.3}_{-1.8}$ | $114^{+24}_{-3}$ | $1.62\pm 0.11$ | $15.8^{+0.25}_{-3.5}$ | 1.0 | $1.10\pm 0.15$ | $1.14\pm 0.13$ | 0.977/219
NSATMOS | $6.1\pm 1.5$ | $114\pm 4$ | $1.55\pm 0.12$ | $16.0^{+0.7}_{-1.3}$ | 1.0 | $1.11\pm 0.15$ | $1.13\pm 0.06$ | 0.977/219
NSAGRAV | $6.7\pm 1.5$ | $110\pm 8$ | $1.71\pm 0.30$ | $16.5\pm 0.5$ | 1.5 | $1.00\pm 0.19$ | $1.01\pm 0.15$ | 0.987/219
NSATMOS | $6.7\pm 1.4$ | $110\pm 5$ | $1.77\pm 0.45$ | $16.6^{+1.8}_{-7.5}$ | 1.5 | $1.03\pm 0.22$ | $1.03\pm 0.10$ | 0.985/219

  * Note. – NH is the equivalent hydrogen column density, $T^{\infty}_{\rm eff}$ the effective temperature of the neutron-star surface as seen at infinity, $M_{\rm NS}$ and $R_{\rm NS}$ are the mass and radius of the neutron star, respectively. $F_{pow}$ is the unabsorbed flux of the power-law component in the 0.5$-$10 keV energy band, and $F_{X}$ is the total unabsorbed X-ray flux in the same energy band. The last column gives the reduced $\chi^{2}$ for 219 degrees of freedom. The quoted errors represent the 90% confidence levels.

Figure 1 shows the XMM-Newton spectra of EXO 0748-676 fitted with the model
“phabs (NSATMOS + powerlaw) ”. The power-law index is fixed at 1.0. The best
fit of this model gives $N_{\rm H}=6.1\pm 1.5$ $\times 10^{20}$ cm-2, neutron-
star mass $M_{\rm NS}=1.55\pm 0.12{\rm M}_{\odot}$, neutron-star radius
$R_{\rm NS}=16.0^{+0.7}_{-1.3}$ km, and effective temperature log$T_{\rm
eff}=6.20\pm 0.02$ (in $K$). According to the same formula $T^{\infty}_{\rm
eff}=T_{\rm eff}\sqrt{1-(2GM_{\rm NS})/(R_{\rm NS}c^{2})}$ used by Degenaar et
al. (2009), we converted $T_{\rm eff}$ to the effective temperature as seen by
an observer at infinity, $T^{\infty}_{\rm eff}=114\pm 4$ eV. In the formula,
$G$ is the gravitational constant and $c$ is the speed of light. The model
predicts 0.5$-$10 keV an unabsorbed X-ray flux $F_{X}=1.13\pm 0.06\times
10^{-12}$ ergs cm-2 s-1. The flux of the power-law component in the same
energy band is $F_{pow}=1.11\pm 0.15\times 10^{-13}$ ergs cm-2 s-1, which
corresponds to $\sim 10\%$ of the total unabsorbed flux. The reduced
$\chi^{2}$ is 0.977 for 219 degrees of freedom. The best-fit results of the
models NSAGRAV and NSATMOS for the three different power-law index are given
in Table-LABEL:tab:model. Errors are given at the 90% confidence level for one
fit parameter.

We note from Table LABEL:tab:model that both atmosphere models, regardless of
the value of $\Gamma$, yield a good fit with similar $\chi^{2}$. In the rest
of the analysis, we used a power-law index fixed to 1. Further, $N_{\rm H}$
and $T_{\rm eff}$ are well constrained and are consistent for the different
fits. Both NSAGRAV and NSATMOS models also give consistent results on $M_{\rm
NS}$ and $R_{\rm NS}$. The NSATMOS model is more accurate in constraining
$T_{\rm eff}$ than the NSAGRAV model.

###  3.2 Equation of state

Fitting the quiescence XMM-Newton spectrum of EXO 0748$-$676 with two
different atmosphere models and comparing the results allows us to test the
reliability and accuracy of both models. From the fits we get a mass and
radius of the neutron star at a specified distance, and then by comparing the
inferred mass and radius with the different neutron-star EOS we can give upper
limits to the source distance for the different EOS.

We used the steppar command in xspec to vary the mass, radius and distance
parameters simultaneously, allowing other parameters to be free to find the
best fit at each step. For the mass we go from 0.5 to 2.5 $M_{\odot}$ with
steps of 0.1 $M_{\odot}$, and for the distance we go from 5 to 10 kpc with
steps of 0.25 kpc. The minimum and maximum radius allowed with these models
are 5.0 km and 25.0 km, respectively. In Fig 2 we show the contour plots
obtained from the STEPPAR procedure for the NSATMOS model. Each plot is for a
different distance, ranging from 5 to 10 kpc. The contour lines (red) are for
the confidence levels of 90% (solid) and 99% (dashed). Further, in Fig 2 we
give different neutron-star EOS (black) taken from Lattimer & Prakash (2007).
We did the same analysis for the NSAGRAV model as well. Both two models give
consistent result, in accordance with the findings of Webb & Barret (2007).

Using the optical data from the Very Large Telescope (VLT), moderate-
resolution spectroscopy of the optical counterpart and Doppler tomography,
Muñoz et al. (2009) provided the first dynamical constraints on the stellar
mass of LMXB EXO 0748$-$676 . The mass range of the neutron star that they
derived is $1{\rm M}_{\odot}\leq{\rm M_{\rm NS}}\leq 2.4{\rm M}_{\odot}$.
Subsequently, Bassa et al. (2009) analyzed optical spectra of EXO 0748$-$676
when the source was in the quiescent state, and they gave a lower limit to the
neutron-star mass of ${\rm M_{\rm NS}}\geq$ 1.27 ${\rm M}_{\odot}$. As upper
limit we used the value reported by Muñoz et al. (2009), but since at the time
of their observation the source was still in outburst, we used the lower limit
reported in Bassa et al. (2009). In Figure 2 we also give the lower
(pink/dotted) and upper (green/dashed) limits to the neutron-star mass.

(a) 5.0 kpc

(b) 6.0 kpc

(c) 7.0 kpc

(d) 8.0 kpc

(e) 9.0 kpc

(f) 10.0 kpc

Figure 2: Contour plots showing the results of modeling the neutron-star in
EXO 0748$-$676 with the xspec model NSATMOS and power-law. The power-law index
is fixed to 1. The plots show two confidence levels in the mass-radius diagram
obtained from our fit; the contour lines (red) are for the confidence levels
of 90% and 99%, respectively. The pink line “A” is for the lower limit of
$M_{\rm NS}$ given by Bassa et al.(2009), and the green line “B” is for the
upper limit given by Muñoz et al. (2009).

In order to test the EOS and identify the upper limit to the source distance,
we assume three different EOS models: normal nucleonic matter (AP3), boson
condensates matter (MS1) and strange quark matter (SQM1). By varying the
source distance from 5 to 10 kpc, the contour lines for the fitted model move
on the NS mass-radius diagram. We can estimate the probability of the distance
for each EOS when the contour lines pass through the EOS curves. Note that as
the distance increases (see Figure 2), the satisfied area of the model moves
from bottom left to top right in the plot. The results using NSAGRAV are
similar to those shown in Figure 2. For a certain distance we found that not
all the EOSs are consistent with the two neutron-star atmosphere models that
we used.

If the neutron star in EXO 0748$-$676 follows the EOS model ‘AP3’, the
probability that the source has a distance of 10.0 kpc is $1\times 10^{-4}$
and $1\times 10^{-6}$ for NSAGRAV and NSATMOS, respectively. If we want to get
a probability for the distance larger than $1\times 10^{-2}$ (99% confidence),
the distance for NSAGRAV and NSATMOS should be smaller than 8.9 kpc and 8.5
kpc, respectively. The distance at 90% confidence for NSAGRAV and NSATMOS is
less than 8.3 kpc and 8.2 kpc, respectively. Both models are consistent with
the distance of 7.1 kpc given by type-I X-ray bursts (Galloway et al., 2008b).

For the EOS model ’MS1’, the probabilities that EXO 0748$-$676 is at a
distance of 10 kpc is $10^{-5}$ and $10^{-6}$ for NSAGRAV and NSATMOS,
respectively. For both models, respectively, the distance at 99% confidence
level is less than 7.3 kpc and 7.1 kpc, and the distance at 90% confidence
level is less than 6.9 kpc and 6.8 kpc. Both neutron-star atmosphere models
with the ’MS1’ model have an upper limit for the distance smaller than 7.1
kpc.

For a EOS model ‘SQM1’, the distance at 99% confidence level is less than 5.2
kpc, and the distance at 90% confidence level is less than 5.0 kpc for both
atmosphere models. The upper limits on the distance to EXO 0748$-$676 for
different EOS are shown in Table LABEL:tab:upper_limit. The ’SQM1’ model is
rejected at a 99% confidence level for this neutron star, unless the source is
closer than 5.2 kpc.

Table 2: Upper limits on the distance to EXO 0748$-$676 for different EOS
models.

EOS | AP3 | AP3 | MS1 | MS1 | SQM1 | SQM1
---|---|---|---|---|---|---
confidence | 90% | 99% | 90% | 99% | 90% | 99%
NSAGRAV |  $<$ 8.3 |  $<$ 8.9 |  $<$ 6.9 |  $<$ 7.3 |  $<$ 5.0 |  $<$ 5.2
NSATMOS |  $<$ 8.2 |  $<$ 8.5 |  $<$ 6.8 |  $<$ 7.1 |  $<$ 5.0 |  $<$ 5.2

  * Note. –The 90% and 99% confidence levels upper limit for the two NS atmosphere models NSAGRAV and NSATMOS for the EOS models: ‘AP3’, ‘MS1’ and ’SQM1’. The distance is in kpc.

##  4 DISCUSSION

We analyzed an XMM-Newton observation of the neutron star EXO 0748$-$676 in
the quiescent state. The unabsorbed X-ray flux in the 0.5$-$10.0 keV energy
band was $\sim 1.1\times$ 10-12 ergs cm-2 s-1. We found that the non-thermal
(power-law) component only contributes $\sim 10\pm 2\%$ of the 0.5$-$10 keV
X-ray flux, which is lower than what Degenaar et al. (2009) found from Chandra
data ($F_{pow}$ was $\sim$ 16$-$17$\%$ of the 0.5$-$10 keV X-ray flux from the
fit with $\Gamma=1$) about a month earlier than our observation. The total
unabsorbed flux (0.5$-$10.0 keV) decreased from $1.3\times 10^{-12}$ ergs cm-2
s-1 in the Chandra observation to $1.1\times 10^{-12}$ ergs cm-2 s-1 in our
observation, whereas $N_{\rm H}$ changed from $\sim 1.2$ $\times 10^{21}$ $\rm
cm^{-2}$ to $\sim 0.6$ $\times 10^{21}$ $\rm cm^{-2}$. The effective
temperature, however, did not show large variations in one month time.
According to the above comparisons, the reduction of the total flux is due to
a lower contribution of the power-law component.

Because the X-ray spectrum in the quiescent state is dominated by thermal
emission originating from the NS surface, our data allow us to constrain the
mass and radius of the neutron star. From the two different NS atmosphere
models (NSAGRAV and NSATMOS) that we used to fit the X-ray spectrum, we found
that both models show similar results and set good constraints on the neutron-
star radius. Even taking into account the $M_{\rm NS}$ lower limit (from Bassa
et al., 2009), upper limit (from Muñoz et al., 2009) and our best fit
$\Delta\chi^{2}$ contour, we still have a large area on the mass-radius
diagram, and many EOSs are still possible (see Figure 2). In order to
constrain the allowed space of mass and radius at a specified distance, we
choose three typical neutron-star EOS, ‘AP3’, ‘MS1’ and ’SQM1’. We found that
the smaller the distance to the NS the more EOSs are consistent with the data.

For any specific EOS, as the upper limit of the distance we took the value of
the distance where the 99% confidence contour just intersects the curve of
that EOS. We found that the upper limits on the distance as derived from the
NSAGRAV model are slightly higher than those for the NSATMOS model. The EOS
model ‘MS1’ can be just satisfied at a distance of 7.1 kpc. If we assume that
the neutron star in EXO 0748$-$676 is a normal neutron star, following the EOS
‘AP3’, the source should be closer than 8.9 kpc for the NSAGRAV model, or 8.5
kpc for the NSATMOS model. Both the ’MS1’ and ’AP3’ EOS are fully consistent
with the measured distance of 7.1 kpc (Galloway et al., 2008b; Wolff et at.,
2005). For larger distances more EOS are ruled out. The EOS ’SQM1’ is rejected
by the atmosphere model fits for a distance of 7.1 kpc measured from the X-ray
bursts Galloway et al. (2008b). We note, however, the neutron-star atmosphere
models may not appropriate for ’bare’ quark matter stars, but only for those
normal quark star where a crust is present.

## Acknowledgments

This work is based on the observations obtained from XMM-Newton. PGJ
acknowledges support from a VIDI grant from the Netherlands Organisation for
Scientific Research.

## References

  * Anders & Grevesse (1989) Anders, E., & Grevesse, N. 1989, Geochim. Cosmochim. Acta, 53, 197
  * Arnaud (1996) Arnaud, K. A. 1996, ASP Conf. Ser., 101, 17
  * Balucinska-Church & McCammon (1992) Balucinska-Church, M., & McCammon, D. 1992, ApJ, 400, 699
  * Balman (2009) Balman, S. 2009, The Astronomer’s Telegram, 2097
  * Bassa et al. (2009) Bassa, C. G.; Jonker, P. G.; Steeghs, D.; Torres, M. A. P, 2009, MNRAS, tmp, 1238B
  * Cottam, Paerels & Méndez (2002) Cottam J., Paerels F., Méndez M., 2002, Nat, 420, 51
  * Cottam et al. (2008) Cottam, J., Paerels, F., Méndez, M., Boirin, L., Lewin, W. H. G., Kuulkers, E., & Miller, J. M. 2008, ApJ, 672, 504
  * Crampton et al. (1986) Crampton, D., Stauffer, J., Hutchings, J. B., Cowley, A. P., & Ianna, P. 1986, ApJ, 306, 599
  * Degenaar et al. (2009) Degenaar N. et al., 2009, MNRAS, 396, L26
  * Galloway et al. (2008a) Galloway D. K., Muno M. P., Hartman J. M., Savov P., Psaltis D., Chakrabarty D., 2008, ApJS, 179, 360
  * Galloway et al. (2008b) Galloway D. K., Özel F. & Psaltis D., 2008b, MNRAS, 387, 268
  * Galloway et al. (2009) Galloway D. K., Chakrabarty D., Lin R., 2009, Astron. Telegram, 2094
  * Gottwald et al. (1986) Gottwald M., Haberl F., Parmar A. N., White N. E., 1986, ApJ, 308, 213
  * Heinke et al. (2006) Heinke, C. O., Rybicki, G. B., Narayan, R., & Grindlay, J. E. 2006, ApJ, 644, 1090
  * Lattimer & Prakash (2007) Lattimer J. M., Prakash M., 2007, Phys. Rep., 442, 109
  * Muñoz et al. (2009) Muñoz-Darias T., Casares J., O’Brien K., Steeghs D., Martínez-Pais I. G., Cornelisse R., Charles P. A., 2009, MNRAS, 394, L136
  * Özel (2006) Özel F., 2006, Nat, 441, 1115
  * Parmar et al. (1986) Parmar, A. N., White, N. E., Giommi, P., & Gottwald, M. 1986, ApJ, 308, 199
  * Parmar et al. (1992) Pavlov, G. G., Shibanov, Y. A., & Zavlin, V. E. 1992, MNRAS, 253, 193
  * Villarreal & Strohmayer (2004) Villarreal A. R., Strohmayer T. E., 2004, ApJ, 614, L121
  * Webb & Barret (2007) Webb, N. A., & Barret, D. 2007, ApJ, 671, 727
  * Wolff et at. (2005) Wolff, M. T., Becker, P. A., Ray, P. S., & Wood, K. S. 2005, ApJ, 632, 1099
  * Zavlin et al. (1996) Zavlin, V. E., Pavlov, G. G., & Shibanov, Y. A. 1996, A&A, 315, 141
  * (24)