11institutetext:  Graduate School of Science and Technology, Hirosaki
University, Hirosaki, 036-8561, Japan 22institutetext: Department of Science,
School of Science and Engineering, Kinki University, Higashi-Osaka, 577-8502,
Japan 33institutetext: Department of Physics, Moscow State University, Moscow,
119992, Russia 44institutetext: Department of Physics, Saitama University,
Saitama, 338-8570, Japan 55institutetext: Kyowa Interface Science Co.,Ltd.,
Saitama, 351-0033, Japan 66institutetext: Inovative Research Organization,
Saitama University, Saitama, 338-8570, Japan 77institutetext: Research
Institute for Science and Engineering, Waseda University, Tokyo, 169-0092,
Japan
77email<EMAIL_ADDRESS>
#  On the Sensitivity of L/E Analysis of Super-Kamiokande Atmospheric Neutrino
Data to Neutrino Oscillation Part 2

Four Possible L/E Analyses for the Maximum Oscillation by the Numerical
Computer Experiment

E. Konishi 11 Y. Minorikawa 22 V.I. Galkin 33 M. Ishiwata 44 I. Nakamura 44 N.
Takahashi 11 M. Kato 55 A. Misaki 6677

###### Abstract

In the previous paper (Part 1), we have verified that the SK assumption on the
direction does not hold in the analysis of neutrino events occurred inside the
SK detector. Based on the correlation between $L_{\nu}$ and $L_{\mu}$ (Figures
12 and 13 in Part 1) and the correlation between $E_{\nu}$ and $E_{\mu}$
(Figure 14 in Part 1), we have made four possible $L/E$ analyses, namely
$L_{\nu}/E_{\nu}$, $L_{\nu}/E_{\mu}$, $L_{\mu}/E_{\nu}$ and $L_{\mu}/E_{\mu}$.
Among four kinds of $L/E$ analyses, we have shown that only $L_{\nu}/E_{\nu}$
analysis can give the signature of maximum oscillations clearly, not only the
first maximum oscillation but also the second and third maximum oscillation,
while the $L_{\mu}/E_{\mu}$ analysis which are really done by Super-Kamiokande
Collaboration cannot give the maximum oscillation at all. It is thus concluded
from those results that the experiments with the use of the cosmic-ray beam
for neutrino oscillation, such as Super-Kamiokande type experiment, cannot
find the maximum oscillation from $L/E$ analysis, because the incident
neutrino cannot be observed due to its neutrality. Therefore, we would suggest
Super-Kamiokande Collaboration to re-analyze the zenith angle distribution of
the neutrino events which occur inside the detector carefully, because
$L_{\nu}$ and $L_{\mu}$ are alternative expressions of the cosine of the
zenith angle for the incident neutrino and that for the emitted muon,
respectively.

###### pacs:

13.15.+g, 14.60.-z

##  1 Introduction

In Figures 12 and 13 of the preceding paperKonishi2 , we have shown that the
SK assumption on the direction that the directions of the incident neutrinos
are the same as those of the emitted muons does not hold. Also, in Figure 14
of the same paper, we have shown that the energies of the incident neutrinos
cannot be determined from the those of the emitted muons, uniquely. However,
the discrepancies between two variables in Figures 12 and 13 are distinctively
large compared with those in Figure 14. Therefore, non-holding of the SK
assumption on the direction plays an essential role in the $L/E$ analysis for
finding the maximum oscillation (oscillation pattern in neutrino oscillation).

The survival probability of a given flavor is given in Eq.(1), in the case of
Super-Kamiokande Collaboration. The variables for the $L/E$ analysis are
$L_{\nu}$ and $E_{\nu}$, where $L_{\nu}$ denotes the flight length for the
incident neutrino between the generation point of the incident neutrino and
the interaction point of the neutrino concerned in the detector, and $E_{\nu}$
is the energy of the incident neutrino.

Figure 1: $L_{\nu}/E_{\nu}$ distribution without oscillation for 1489.2 live
days (one SK live day).

Figure 2: $L_{\nu}/E_{\nu}$ distribution without oscillation for 37230 live
days (25 SK live days).

Figure 3: Survival probability of $P(\nu_{\mu}\rightarrow\nu_{\mu})$ as a
function of $L_{\nu}/E_{\nu}$ under the neutrino oscillation parameters
obtained by Super-Kamiokande Collaboration .

##  2 $L/E$ Distributions in Our Numerical Computer Experiment

Our computer numerical experiments are carried out in the unit of 1489.2 days.
Hereafter, we call 1489.2 live days as one SK live day. The live days of
1489.2 is the total live days for the analysis of the neutrino events
generated inside the detector by Super-Kamiokande Collaboration Ashie2 . We
repeat one SK live day experiment as much as 25 times, namely, the total live
days for our computer numerical experiments is 37230 live days (25 SK live
days). In Figure 2, we show $L_{\nu}/E_{\nu}$ distribution without oscillation
for one experiment (1489.2 live days) among twenty five computer numerical
experiments. In those numerical experiments, there are statistical
uncertainties only which are due to both the stochastic character in the
physical processes concerned and the geometry of the detectors. Therefore we
add the standard deviation as for the statistical uncertainty around their
average in the forthcoming graphs, if neccessary. In Figure 2, we show the
statistical uncertainty, the standard deviations around their average values
through twenty five experiments. Similarly for other possible combinations of
$L$ and $E$ ( $L_{\nu}/E_{\mu}$, $L_{\mu}/E_{\nu}$ and $L_{\mu}/E_{\mu}$) for
37230 live days (25 SK live days) we did so.

###  2.1 $L_{\nu}/E_{\nu}$ distribution

####  2.1.1 For null oscillation

In Figures 2 and 2, both distributions show the sinusoidal-like character for
$L_{\nu}/E_{\nu}$ distribution, namely, the appearance of the top and the
bottom, even for null oscillation. The uneven histograms in Figure 2,
comparing with those in Figure 2, show that the statistics of Figure 2 is not
enough compared with that of Figure 2. Roughly speaking, smaller
$L_{\nu}/E_{\nu}$ correspond to the contribution from downward neutrinos,
larger $L_{\nu}/E_{\nu}$ correspond to that from upward neutrinos and
$L_{\nu}/E_{\nu}$ near the minimum correspond to the horizontal neutrinos,
although the real situation is more complicated, because the backscattering
effect in QEL as well as the azimuthal angle effect in QEL could not be
neglected. From Figure 2, we understand that the bottom around 70 km/GeV
denotes the contribution from the horizontal direction and has no relation
with neutrino oscillation in any sence.

Figure 4: $L_{\nu}/E_{\nu}$ distribution with oscillation for 1489.2 live days
(one SK live day), sample No.1.

Figure 5: $L_{\nu}/E_{\nu}$ distribution with oscillation for 1489.2 live days
(one SK live day), sample No.2.

Figure 6: $L_{\nu}/E_{\nu}$ distribution with oscillation for 14892 live days
(10 SK live days).

Figure 7: $L_{\nu}/E_{\nu}$ distribution with standard deviations with
oscillation for 37230 live days (25 SK live days).

Figure 8: The correlation diagram between $L_{\nu}$ and $E_{\nu}$ with
oscillation for 1489.2 live days (one SK live day).

Figure 9: The correlation diagram between $L_{\nu}$ and $E_{\nu}$ with
oscillation for 14892 live days (10 SK live days).

Figure 10: $L_{\nu}/E_{\nu}$ distribution with and without oscillation for
14892 live days (10 SK live days).

Figure 11: The ratio of $(L_{\nu}/E_{\nu})_{osc}/(L_{\nu}/E_{\nu})_{null}$ for
1489.2 live days (one SK live day).

Figure 12: The ratio of $(L_{\nu}/E_{\nu})_{osc}/(L_{\nu}/E_{\nu})_{null}$ for
14892 live days (10 SK live days).

####  2.1.2 For oscillation (SK oscillation parameters)

The survival probability of a given flavor, such as $\nu_{\mu}$, is given by

$P(\nu_{\mu}\rightarrow\nu_{\mu})=1-sin^{2}2\theta\cdot sin^{2}(1.27\Delta
m^{2}L_{\nu}/E_{\nu}).\,\,\,(1)$

Then, for maximum oscillations under SK neutrino oscillation parameters, we
have

$1.27\Delta m^{2}L_{\nu}/E_{\nu}=(2n+1)\times\frac{\pi}{2},\,\,\,\,\,\,(2)$

where $\Delta m^{2}=2.4\times 10^{-3}\rm{eV^{2}}$. From Eq.(2), we have the
following values of $L_{\nu}/E_{\nu}$ for maximum oscillations.

$\displaystyle L_{\nu}/E_{\nu}$ $\displaystyle=$ $\displaystyle 515{\rm
km/GeV}\,\,\,\,\,for\,\,n=0\,\,\,(3-1)$ $\displaystyle=$ $\displaystyle
1540{\rm km/GeV}\,\,\,for\,\,n=1\,\,\,(3-2)$ $\displaystyle=$ $\displaystyle
2575{\rm km/GeV}\,\,\,for\,\,n=2\,\,\,(3-3)$ $\displaystyle{\rm and}\,{\rm
so}\,{\rm on.}$

In Figure 3, we give the survival probability
$P(\nu_{\mu}\rightarrow\nu_{\mu})$ as a function of $L_{\nu}/E_{\nu}$ under
the neutrino oscillation parameters obtained by Super-Kamiokande
Collaboration. In cosmic ray experiments, the energy spectrum of the incident
neutrino, is convoluted into the survival probability.

In Figure 6, we give one example of the $L_{\nu}/E_{\nu}$ distribution for one
SK live day (1489.2 live days)Ashie2  among twenty five sets of the computer
numerical experiments in the unit of one SK live day. In Figure 6, we give
another example for one SK live day. Arrows A, B and C represent the first,
the second and the third maximum oscillation which are given in Eq. (3-1),
(3-2) and (3-3), respectively. By the definition of our computer numerical
experiments, there are no experimental error bars in $L_{\nu}/E_{\nu}$
distributions in Figures 6 and 6.

In Figure 6, we show the $L_{\nu}/E_{\nu}$ distribution for 14892 live days
(10 SK live days). Compared Figure 620 with Figures 6 and 6, it is clear that
$L_{\nu}/E_{\nu}$ distribution in Figure 6 becomes smoother due to larger
statistics. In Figure 7, we can add the statistical uncertainty (standard
deviation in this case) around their average, because every one SK live day
experiment among twenty five sets of the experiments fluctuates one by one due
to their stochastic character in their physical processes and geometrical
conditions of the detectors concerned. In order to make the image of the
maximum oscillations in $L_{\nu}/E_{\nu}$ distributions clearer, we show the
correlations between $L_{\nu}$ and $E_{\nu}$ in Figures 10 and 10, which
correspond to Figures 6 and 6, respectively. In Figure 10 for one SK live day,
we can observe vacant regions for the events concerned assigned by A, B and C.
In Figure 10 for ten SK live days, the existence of the vacant regions for the
events concerned becomes clearer due to larger statistics.

In Figure 10, we give $L_{\nu}/E_{\nu}$ distribtution with 14892 live days (10
SK live days) in the linear scale which is another expression of the same
content as in Figure 10. Also, it is the survival probability convoluted with
the incident neutrino energy spectrum. If we compare Figure 10 with Figure 3,
then, we clearly see the series of maximum oscillations characterized with n=0
(A), 1(B), 2(C) and so on which are given by Eq.(2). It is clear from Figure
10 that the maximum oscillations with n=0,1 and 2 have the almost same
frequencies 111 Super-Kamiokande Collaboration never mentioned existence of
the second and the third maximum oscillations (n=1 and 2) under the incident
neutrino energy spectrum utilized by Super-Kamiokande Collaboration Honda
(see footnote 1). The situation shown in Figures 10 to 10 shows definitely
that our computer numerical experiment are carried out as exactly as possible
from the view point of the stochastic treatment to the matter.

We have repeated the computer numerical experiment for one SK live day as much
as twenty five times independently, in both cases with oscillation and without
oscillation. Consequently, there are 625 ($=25\times 25$) sets of ratios of
$(L_{\nu}/E_{\nu})_{osc}/(L_{\nu}/E_{\nu})_{null}$ for one SK live day which
correspond to Eq.(1). In Figure 12, we show one example among 625
combinations. In Figure 12, we show the same ratio for 14892 live days (10 SK
live days). In conclusion, from Figures 6 to 12, we can reproduce the minimum
extrema for neutrino oscillation in our $L_{\nu}/E_{\nu}$ analysis. This fact
shows doubtlessly that our computer numerical experiments are done in the
correct manner.

Figure 13: $L_{\mu}/E_{\mu}$ distribution without oscillation for 1489.2 live
days (one SK live day).

Figure 14: $L_{\mu}/E_{\mu}$ distribution without oscillation for 37230 live
days (25 SK live days).

Figure 15: $L_{\mu}/E_{\mu}$ distribution with oscillation for 1489.2 live
days (one SK live day).

Figure 16: $L_{\mu}/E_{\mu}$ distribution with oscillation for 37230 live days
(25 SK live days).

###  2.2 $L_{\mu}/E_{\mu}$ distribution

As physical quantities which can really be observed are $L_{\mu}$ and
$E_{\mu}$ instead of $L_{\nu}$ and $E_{\nu}$, therefore we examine
$L_{\mu}/E_{\mu}$ distribution focusing the existence of the maximum
oscillation.

####  2.2.1 For null oscillation

In Figure 14, we give one sample for one SK live day (1489.2 live days) from
the totally 37230 live days (25 SK live days) events, each of which has 1489.2
live days. Figure 14 shows the average distribution accompanied by the
statistical uncertainty bar (not experimental error bar). It is clear from
these figures that the existence of the dip or bottom, namely the sinusoidal
character, means the contribution merely from horizontal contribution, having
no relation with any neutrino oscillation character.

Figure 17: The correlation diagram between $L_{\mu}$ and $E_{\mu}$ with
oscillation for 1489.2 live days (one SK live day).

Figure 18: The correlation diagram between $L_{\mu}$ and $E_{\mu}$ with
oscillation for 14892 live days (10 SK live days).

Figure 19: $L_{\mu}/E_{\mu}$ distribution with and without oscillation for
14892 live days (10 SK live days).

Figure 20: The ratio of $(L_{\mu}/E_{\mu})_{osc}/(L_{\mu}/E_{\mu})_{null}$ for
1489.2 live days (one SK live day).

Figure 21: The ratio of $(L_{\mu}/E_{\mu})_{osc}/(L_{\mu}/E_{\mu})_{null}$ for
14892 live days (10 SK live days).

Figure 22: $L_{\mu}/E_{\nu}$ distribution without oscillation for 1489.2 live
days (one SK live day).

Figure 23: $L_{\mu}/E_{\nu}$ distribution without oscillation for 37230 live
days (25 SK live days).

####  2.2.2 For oscillation (SK oscillation parameters)

In Figures 16 and 16, we give the $L_{\mu}/E_{\mu}$ distributions with
oscillation for 1489.2 live days (one SK live day) and 37230 live days (25 SK
live days), respectively. In Figure 16, we may observe the uneven histogram,
something like curious bottoms coming from neutrino oscillation. However, in
Figure 16 where the statistics is 25 times as much as that of Figure 16, the
histogram becomes smoother and such bottoms disappear, which turns out finally
for the bottoms to be pseudo. It is impossible to extract the neutrino
oscillation parameters from the comparison of Figure 16 with Figure 14.

In Figures 19 and 19, correspondingly, we give the correlation between
$L_{\mu}$ and $E_{\mu}$ for 1489.2 live days (one SK live day) and 14892 live
days (10 SK live days), respectively.

In Figure 19, we give the $L_{\mu}/E_{\mu}$ distribution for 14892 live days
(10 SK live days) in the linear scale which is another expression of the same
content as in Figure 19. As in Figure 19, we cannot find any maximum
oscillation-like phenomena in Figure 19, which is contrast to Figure 10.

It is clear from the figures that we can not observe the maximum oscillation
in $L_{\mu}/E_{\mu}$, on the contrary to Figures 6 to 10 which give the
maximum oscillations. Namely, we may conclude that we can not observe the
sinusoidal flavor transition probability of neutrino oscillation against the
claim by Super-Kamiokande CollaborationAshie1  when we adopt physically
observable quantities, such as $L_{\mu}$ and $E_{\mu}$.

Figure 24: $L_{\mu}/E_{\nu}$ distribution with oscillation for 1489.2 live
days (one SK live day).

Figure 25: $L_{\mu}/E_{\nu}$ distribution with oscillation for 37230 live days
(25 SK live days).

Figure 26: The $L_{\mu}/E_{\nu,SK}$ distribution in comparison with
$L_{\mu}/E_{\nu}$ distribution with oscillation for 14892 day (10 SK live
days).

Figure 27: The $L_{\nu}/E_{\mu}$ distribution without oscillation for 37230
days (25 SK live days).

Figure 28: The $L_{\nu}/E_{\mu}$ distribution with oscillation for 37230 days
(25 SK live days).

In order to confirm the disappearance of the psuedo maximum oscillations, in
Figures 21 and 21, we give the survival probability of a given flavor for
$L_{\mu}/E_{\mu}$ distribution, namely,
$(L_{\mu}/E_{\mu})_{osc}/(L_{\mu}/E_{\mu})_{null}$, for 1489.2 live days (one
SK live day) and 14892 live days (10 SK live days), respectively. In Figure
21, we show one example of $(L_{\mu}/E_{\mu})_{osc}/(L_{\mu}/E_{\mu})_{null}$
among 625 sets of ratios. Comparing Figure 21 with Figure 21, the pseudo dips
in Figure 21 disappear in Figure 21. Thus the histogram becomes a rather
decreasing function of $L_{\mu}/E_{\mu}$ in Figure 21. If we further make
statistics higher, the survival probability for $L_{\mu}/E_{\mu}$ distribution
should be a monotonously decreasing function of $L_{\mu}/E_{\mu}$, whithout
showing any characteristics of the maximum oscillation, which is in contrast
to Figures 12 and 12.

In conclusion, we should say that we can not find any maximum oscillation for
the neutrino oscillation in the $L_{\mu}/E_{\mu}$ distribution.

###  2.3 $L_{\mu}/E_{\nu}$ distribution

Now, we examine the $L_{\mu}/E_{\nu}$ distribution which Super-Kamiokande
Collaboration treat in the thier paper, expecting the evidence for the
oscillatory signatuture in atmospheric neutrino oscillations.

####  2.3.1 For null oscillation

In Figures 23 and 23, we give the $L_{\mu}/E_{\nu}$ distribution without
oscillation for 1489.2 live days (one SK live day) and 37230 live days (25 SK
live days), respectively. Comparing Figure 23 with Figure 23, the larger
statistics makes the distribution smoother. Also, there is a sinusoidal-like
bottom which has no relation with neutrino oscillation.

####  2.3.2 For oscillation (SK oscillation parameters)

In Figures 25 and 25, we give the $L_{\mu}/E_{\nu}$ distribution with
oscillation for 1489.2 live days (one SK live day) and 37230 live days (25 SK
live days), respectively. In Figure 25, we may find something like a bottom
which corresponds to the first maximum oscillation near $\sim$200 (km/GeV).
However, such the dip disappears, by making the statistics larger as shown in
Figure 25.

####  2.3.3 $L_{\mu}/E_{\nu,SK}$ distribution for the oscillation

Instead of $E_{\nu}$ which is correctly sampled from the corresponding
probability functions, let us utilize $E_{\nu,SK}$ which is obtained from the
”approximate” formula (Eq.(6) in the preceding paperKonishi2 ).

We express $E_{\nu}$ described in Eq.(6) of the preceding paperKonishi2
utilized by Super-Kamiokande Collaboration as $E_{\nu,SK}$ to discriminate our
$E_{\nu}$ obtained in the stochastic manner correctly.

In Figure 26, we give $L_{\mu}/E_{\nu,SK}$ distribution for 14892 live days
(10 SK live days), comparing with $L_{\mu}/E_{\nu}$ distribution. It is
understood from the comparison that there is no significant difference between
$L_{\mu}/E_{\nu,SK}$ distribution and $L_{\mu}/E_{\nu}$ one. This fact tells
us that the ”aproximate” formula for $E_{\nu}$ by Super-Kamiokande
Collaboration does not produce so significant error practically. Although this
kind of foumula is not suitable for the treatment of stochastic quantities,
the result is understandable from Figure 14 in the preceding paperKonishi2 .
Also, we can conclude that we do not find any hole corresponding to the
maximum oscillation in $L_{\mu}/E_{\nu}$ or $L_{\mu}/E_{\nu,SK}$
distributions. The reason why the Figure 25 can not show such dip structure as
shown in Figures 6 and 6, comes from the situation that the role of $L_{\nu}$
is much more crucial than that of $E_{\nu}$ in the $L/E$ analysis. Namely,
$L_{\nu}$ cannot be replaced by $L_{\mu}$ at all. Also, see the discussion in
the following subsection 4.4.

Figure 29: The correlation diagram between $L_{\nu}$ and $E_{\mu}$ with
oscillation for 14892 days (10 SK live days).

###  2.4 $L_{\nu}/E_{\mu}$ distribution

####  2.4.1 For null oscillation

In Figure 28, we give $L_{\nu}/E_{\mu}$ distribution without oscillation for
37230 live days (25 SK live days) of Super-Kamiokande Experiment to consider
statistical fluctuation effect as precisely as possible. It is clear from the
figure that there is not any dip corresponding to the maximum oscillation
which is expected to appear in presence of neutrino oscillation, as it must
be.

####  2.4.2 For oscillation (SK oscillation parameters)

In Figure 28, we give the corresponding distribution with the oscillation. In
Figure 29, we give the correlation diagram between $L_{\nu}$ and $E_{\mu}$ for
14823 live days (10 SK live days). On the contrary to Figure 25, there are
surely some kinds of holes in Figure 28, and furthermore we can discriminate
the strip pattern in Figure 29, similarly as in Figure 10.

Therefore, we surmise from Figures 28 and 29 that we may observe some ”maximum
oscillation like” quantities which are related to the maximum oscillations in
the $L_{\nu}/E_{\nu}$ distribution through the correlation between $E_{\mu}$
and $E_{\nu}$ shown in Figure 14 in the preceding paperKonishi2 . However, it
seems to be difficult to extract a pair of concrete values of $L_{\nu}$ and
$E_{\nu}$ through the analysis of the $L_{\nu}/E_{\mu}$ distribution. In
Figure 31, we make a comparison between $L_{\nu}/E_{\nu}$ distribution and
$L_{\nu}/E_{\mu}$ distribution where the correlation between $E_{\nu}$ and
$E_{\mu}$ is shown in Figure 14 in the preceding paperKonishi2 . It is clear
from the figure that the $L_{\nu}/E_{\nu}$ distribution demonstrates the
maximum oscillation as already shown in Figures 6 to 10 and the
$L_{\nu}/E_{\mu}$ distribution also demonstrates the maximum oscillation-like
as already shown in Figure 28 and 29. In Figure 31, we give the relation
between $L_{\mu}/E_{\nu}$ distribution and $L_{\mu}/E_{\mu}$ distribution
where the same correlation between $E_{\nu}$ and $E_{\mu}$ holds in the case
of Figure 31. It is also clear from the figures that both the distributions
demonstrate neither the maximum oscillation nor the maximum oscillation-like,
which is also clear from Figures 16 to 19 and Figures 25 to 25. Thus, it can
be concluded from Figures 13 and 14 in the preceding paperKonishi2  and Figure
31 and Figure 31 in the present paper that $L_{\nu}$ plays an essential role
compared with others $L_{\mu}$, $E_{\nu}$ or $E_{\mu}$. In other words, it
should be noticed that $L_{\nu}$ cannot be approximated by $L_{\mu}$, while
$E_{\nu}$ can be obtained approximately from $E_{\mu}$ through some procedure.
Also, such a serious discrepancy between $L_{\nu}$-$L_{\mu}$ relation and
$E_{\nu}$-$E_{\mu}$ relation is shown in the comparison of Figure 31 with
Figure 31.

Figure 30: Comparison between $L_{\nu}/E_{\nu}$ distribution and
$L_{\nu}/E_{\mu}$ distribution with oscillation for 37230 days (25 SK live
days).

Figure 31: Comparison between $L_{\mu}/E_{\nu}$ distribution and
$L_{\mu}/E_{\mu}$ distribution with oscillation for 37230 days (25 SK live
days).

Figure 32: The comparison of $L/E$ distribution for single-ring muon events
due to QEL among Fully Contained Events with the corresponding one by the
Super-Kamiokande Experiment.

##  3 Comparison of $L/E$ Distribution in the Super-Kamiokande Experiment with
our Results

In our classification, the $L/E$ distribution by Super-
Kamiokande Collaboration Ashie2 Ashie1  should be compared directly with our
$L_{\mu}/E_{\nu}$ distributiton. Taking account of their assertion of
existence of the maximum oscillation we compare their results with our results
on $L_{\nu}/E_{\nu}$ in Figure 32 222We read out Fully Contained Events among
total events from Super-Kamiokande Collaboration Ashie2 Ashie1 .. It is clear
from the figure that there are two big differences between them.

One is that we observe the first maximum oscillation ($L_{\nu}/E_{\nu}=515$
km/GeV under the SK oscillation parameters) sharply, while SK observe it in
the wider range of $L_{\nu}/E_{\nu}=100\sim 800$ km/GeV.

Such the lack of the neutrino events over the wide range may be due to their
measurement of $L_{\mu}$, but not $L_{\nu}$, because the given definite
$L_{\nu}$ corresponds to $L_{\mu}$ over a wide range and vice versa (See also
the correlation between $L_{\nu}$ and $L_{\mu}$ in Figure 35 and 38)

Figure 33: Correlation diagram between $L_{\nu}$ and $E_{\nu}$ for
$1.0<L_{\nu}/E_{\nu}<1.26$ (km/GeV) which corresponds to the maximum frequency
of the neutrino events for $L_{\nu}/E_{\nu}$ distribution in our computer
numerical experiment for 14892 live days (10 SK live days).

Figure 34: Correlation diagram between $L_{\nu}$ and $L_{\mu}$ for
$1.0<L_{\nu}/E_{\nu}<1.26$ (km/GeV) under the neutrino oscillation parmeters
obtained by Super-Kamiokande Collaboration for 14892 live days (10 SK live
days).

Figure 35: Correlation diagram between $L_{\mu}$ and $E_{\mu}$ for
$1.0<L_{\nu}/E_{\nu}<1.26$ (km/GeV) which corresponds to the maximum frequency
of the neutrino events for $L_{\nu}/E_{\nu}$ distribution in our computer
numerical experiment for 14892 live days (10 SK live days).

Figure 36: Correlation diagram between $L_{\nu}$ and $E_{\nu}$ for
$20<L_{\nu}/E_{\nu}<25$ (km/GeV) which corresponds to the maximum frequency of
the neutrino events for $L_{\mu}/E_{\nu}$ distribution in SK experiment for
14892 live days (10 SK live days).

Figure 37: Correlation diagram between $L_{\nu}$ and $L_{\mu}$ for
$20<L_{\nu}/E_{\nu}<25$ (km/GeV) under the neutrino oscillation parmeters
obtained by Super-Kamiokande Collaboration for 14892 live days (10 SK live
days).

Figure 38: Correlation diagram between $L_{\mu}$ and $E_{\mu}$ for
$20<L_{\nu}/E_{\nu}<25$ (km/GeV) which corresponds to the maximum frequency of
the neutrino events for $L_{\nu}/E_{\nu}$ distribution in SK experiment for
14892 live days (10 SK live days).

The other is that there is big difference between them as for the position
which give the maximum frequency for the events concerned. Here, we do not
mention to the existence of the maximum oscillation which is derived from the
measurement of $L_{\mu}$ utilized in Super-Kamiokande Collaboration, because
one cannot observe the maximum oscillation, if we utilize $L_{\mu}$ (see
Figures 14 to 19). Conseqently, we examine the second point as for the maximum
frequency for the events concerned. Our computer numerical experiment gives
the maximum frequency for interval $1.0<L_{\nu}/E_{\nu}<1.26$ (km/GeV) as
shown in Figure 32.

In Figure 35, we give the correlation between $L_{\nu}$ and $E_{\nu}$ for
interval $1.0<L_{\nu}/E_{\nu}<1.26$ (km/GeV). It is clear from the figure that
the larger part of the incident neutrino events is occupied by the vertically
downward ones and the smaller part is occupied by the horizontally downward
neutrino events. This is quite reasonable, because more intensive downward
flux contribute to the maximum frequency for the events concerned, compared
with weaker upward flux under the Super-Kamokande neutrino oscillation
parameters.

Figure 39: Correlation diagram between $L_{\mu}$ and $E_{\nu}$ for
$15.8<L_{\mu}/E_{\nu}<31.6$ (km/GeV) which correspond to the maximum frequency
of the neutrino events for $L_{\mu}/E_{\nu}$ distribution in SK experiment for
14892 live days (10 SK live days).

Figure 40: Correlation diagram between $L_{\nu}$ and $L_{\mu}$ for
$15.8<L_{\mu}/E_{\nu}<31.6$ (km/GeV) under the neutrino oscillation parmeters
obtained by Super-Kamiokande Collaboration for 14892 live days (10 SK live
days).

Figure 41: Correlation diagram between $L_{\nu}$ and $E_{\mu}$ for
$15.8<L_{\mu}/E_{\nu}<31.6$ (km/GeV) which correspond to the maximum frequency
of the neutrino events for $L_{\mu}/E_{\nu}$ distribution in SK experiment for
14892 live days (10 SK live days).

In Figure 35, we give the correlation diagram between $L_{\nu}$ and $L_{\mu}$
for the same intervals as in Figure 35. It is clear from Figure 35 that the
majority of the events is concentrated into the squared regions with
$L_{\nu}<10$ km and $L_{\mu}<10$ km. This denotes that the downward incident
neutrinos produce muons toward the forward direction with either smaller or
larger angles and only the smaller part of the downward incident neutrino
events produce the upward muons due to backscattering (1000 to 10,000 km in
$L_{\mu}$) as well as the azimuthal angle effect in QEL. In Figure 35, we give
the correlation diagram between $L_{\mu}$ and $E_{\mu}$ for the same intervals
as in Figure 35. It is clear from this figure that the produced muons with
higher energies are ejected toward the forward and vertical-like directions,
while the produced muon with lower energies may be ejected toward the backward
or holizontal-like direction. Namely, it is concluded from Figures 35, 35 and
35 that the vertically downward neutrino events can contribute to the maximum
frequency, because they are free from neutrino oscillation. It is further
noted that the direction of the produced muon does not coincide with the
original direction of the neutrino.

Now, we examine $L$-$E$ relation at the position for our computer numerical
expermiment where Super-Kamiokande Collaboration give the maximum frequency
for the events ($20<L_{\nu}/E_{\nu}<25$ (km/GeV)). In Figure 38, we show the
correlation between $L_{\nu}$ and $E_{\nu}$ for interval
$20<L_{\nu}/E_{\nu}<25$ (km/GeV).

It is clear from Figure 38 that $L_{\nu}$ distribute over 27 $\sim$ 120 km,
corresponding to $cos\theta_{\nu}=$ \- 0.1 $\sim$ 0, which denotes the
horizontal-like downward neutrino events. The frequency of the horizontal-like
downward neutrino events in Figure 38 are pretty smaller than that of the
vertical-like downward neutrino events in Figure 35 due to smaller solid
angles. In Figure 38, we give the correlation diagram between $L_{\nu}$ and
$L_{\mu}$ for $20<L_{\nu}/E_{\nu}<25$ (km/GeV). It is impressive from the
figure that $L_{\mu}$ distribute over four orders of magnitude (2 km to
$1.2\times 10^{4}$ km), while $L_{\nu}$ cover within one order of magnitude
(20 $\sim$ 120 km). This fact denotes that the effect of the azimuthal angles
in QEL is pretty strong even in the horizontal-like downward neutrino events
in which the produced muons are apparently judged to come from the upward
direction (see Figure 3-c and Figures 8 to 10 in the preceding paperKonishi2
).

In Figure 38, we give the correlation diagram between $L_{\mu}$ and $E_{\mu}$
for the same intervals as in Figure 38. If we compare Figure 38 with Figure
38, then we can find the following interesting situation. As it is clearly
understandable from Figure 38, horizontal-like downward neutrinos produce the
muons in the three different regions, namely, vertical-like downward muons,
horizontal-like downward muons and upward muons. From horizontal-like downward
neutrinos with rather low energies, the vertically downward muons are ejected
with rather large scattering angles. On the other hand, the horizontal-like
downward muons are ejected with rather small angles whose energies are close
to the incident neutrinos energies. Furthermore, the upward muons are produced
either due to backscattering or due to the azimuthal effect in QEL for
horizontal-like incident neutrinos (see Figures 8 and 9 in the preceding
paperKonishi2 ). Thus, from the comparison of Figures 35, 35 and 35 with
Figures 38, 38 and 38, it is reasonable for the the maximun frequency of the
$L_{\nu}/E_{\nu}$ events to occur for $1.0<L_{\nu}/E_{\nu}<1.26$ (km/GeV), and
not to occur for $20<L_{\nu}/E_{\nu}<25$ (km/GeV) where Super-Kamiokande
Collaboration ”assert”.

Finally, we examine the correlation between $L_{\mu}$ and $E_{\nu}$ for
$15.8<L_{\mu}/E_{\nu}<31.6$ (km/GeV) where Super-Kamiokande Collaboration give
the maximum frequency of $L/E$ neutrino events as shown in Figure 32. Although
we compare their frequency with that of our $L_{\nu}/E_{\nu}$ in Figure 32, we
can compare their frequency with that of our $L_{\mu}/E_{\nu}$ in Figure 31,
which shows big difference between them. In Figure 41, we give the correlation
diagram between $L_{\mu}$ and $E_{\nu}$ for $15.8<L_{\mu}/E_{\nu}<31.6$
(km/GeV). In Figures 41 and 41, we give the corresponding correlation diagrams
between $L_{\nu}$ and $L_{\mu}$, and $L_{\nu}$ and $E_{\mu}$, respectively.

It is clear from Figure 41 that Super-Kamiokande Collaboration measure the
vertical-like downward muons. It is also clear from Figures 41 and 41 that
these vertical-like downward muon are produced by the incident neutrinos whose
$L_{\nu}$ are distributed over four orders of magnitude. These incident
neutrinos are classified into two parts. One is the downward incident
neutrinos ($1.0<L_{\nu}<100$ km) and the other ($L_{\nu}>100$ km) is the
upward incident neutrinos. The majority of the incident neutrino is occupied
by the vertical-like downward. However, the frequency of the upward neutrinos
is in the same order of the magnitude as the horizontal-like downward. The
upward incident neutrinos may produce downward muons due to both
backscattering and the azithumal angle effect in QEL. At any rate, for the
measured muons in the case of the maximun frequency of the events, $L_{\nu}$
of the corresponding incident neutrinos distribute over four orders of
magnitude. Shortly speaking, for the maximum frequency of the neutrino events
$15.8<L_{\mu}/E_{\nu}<31.6$ (km/GeV), the magnitude of the $L_{\mu}$ of the
produced muons lie within one order of magnitude (see Figure 41), although the
$L_{\nu}$ of the incident neutrinos which produce these muons distribute over
four orders of magnitude. In other words, it is concluded that Super-
Kamiokande Collaboration do not measure the definite direction of the incident
neutrinos as far as they measure $L_{\mu}$. It is furthermore noticed from the
comparison of Figure 41 with Figure 29 that Figure 41 is obtained from Figure
29 by cutting off the stripe of $15.8<L_{\mu}/E_{\nu}<31.6$ (km/GeV).
Therefore ,we can recognize the vacant region of the neutrino events faintly
in the part between 100 and 1000 (km/GeV) in Figure 41 which is clearly shown
in Figure 29. The vacant region of the events shows indication of neutrino
oscillation.

The summary on Figures from 35 to 41 are as follows; Figures from 35 to 35
represent the mutual relations among $L_{\nu}$, $L_{\mu}$, $E_{\nu}$ and
$E_{\mu}$ near our maximum frequency of $L_{\nu}/E_{\nu}$ distribution. Here,
all the incident neutrinos are occupied by the downward vertical-like
neutrinos, while the majority of the emitted muons is occupied by the downward
muon and the minority is occupied by the upward muon. Figures from 38 to 38
represent the similar mutual relations for our $L_{\nu}/E_{\nu}$ distribution
which correspond to the near the maximum frequency of $L_{\mu}/E_{\nu}$
distribution obtained by Super-Kamiokande Collaboration. Here, almost the
incident neutrinos are occupied by the downward holizontal-like neutrinos,
while about the half of emitted muons is recognized as the downward muon and
the other half is done as the upward muon. Figures from 41 to 41 represent the
mutual similar relations, assuming the numerical values of the maximum
frequency of $L_{\mu}/E_{\nu}$ distribution obtained by Super-Kamiokande
Collaboration. Here, the majority of the emitted muons is occupied by the
horizontal like muons, while their parent neutrinos come from both the
downward neutrinos and the upward ones. The common characteristics through
Figures from 35 to 41 is that for given definite $L_{\nu}$($L_{\mu}$) we find
$L_{\mu}$($L_{\nu}$) which distribute over the four order of magnitudes.

##  4 Conclusion

The assumption made by Super-Kamiokande Collaboration that the direction of
the reconstructed lepton approximately represents the direction of the
original neutrino does not hold even approximately Konishi1 . This is
logically equivalent to the statement that $L_{\nu}$ cannot be replaced by
$L_{\mu}$ even if approximately. This is really clarified in Figures 12 and 13
in the preceding paperKonishi2 .

Although the derivation of $E_{\nu}$ from $E_{\mu}$ (Eq.(6) of the preceding
paperKonishi2 ) is theoretically, irrelevant to the stochastic plobrem,
because of the neglect of the stochastic character in physical processes
concerned, such the approximation does not induce so practically serious error
compared with the assumption of $L_{\nu}\approx L_{\mu}$. As clarified in
Figures 6 to 12, the maximum oscillation in $L/E$ analysis can be observed
only in the $L_{\nu}/E_{\nu}$ distribution and it is quite natural by the
definition of the probability for a given favor whose argument is
$L_{\nu}/E_{\nu}$ (Eq.(1)). As clarified in Figures 16 to 19 and Figures 25 to
26 the maximum oscillation for the presence of neutrino oscillation cannot be
observed from both $L_{\mu}/E_{\mu}$ and $L_{\mu}/E_{\nu}$. The relation
between $L_{\nu}$ and $L_{\mu}$ is too complicate to extract similar
expression to Eq.(6) of the preceding paperKonishi2  for the argument on
$L_{\mu}/E_{\mu}$ and $L_{\mu}/E_{\nu}$ . Similarly in the case of argument of
$L_{\nu}/E_{\nu}$, we can indicate something like the maximum oscillation in
$L_{\nu}/E_{\mu}$ distribution which are shown in Figures 2841 and 28. The
situation is derived from the fact that what plays a decisive role in $L/E$
analysis is $L_{\nu}$, but not $E_{\nu}$, which are clearly shown by comparing
Figures 12 and 13 with Figure 14 in the preceding paperKonishi2 .

As for $L/E$ distribution obtained by Super-Kamiokande Collaboration, we
definitely indicate that the maximum oscillation cannot be observed through
the measurement of $L_{\mu}$. Consequently, we cannot observe the maximum
oscillation in $L/E$ analysis which is carried out in Super-Kamiokande
Collaboration. Furthermore, one cannot find the maximum frequency of $L/E$
events at the position where Super-Kamiokande Collaboration observe, even if
one can observe $L_{\nu}$.

In conclusion, the maximum oscillation in $L/E$ analysis can be observed only
in $L_{\nu}/E_{\nu}$, but not in any other combinations of $L$ with $E$.
However, $L_{\nu}$ is physically unobservable quantities and it cannot be
approximated by $L_{\mu}$, because the assumption between $L_{\nu}$ and
$L_{\mu}$, does not hold even if statistically. Consequently, it should be
concluded that Super-Kamiokande cannot observe the maximum oscillation in
their $L_{\mu}/E_{\nu,SK}$ analysis.

Finally, our conclusion that $L_{\nu}$ cannot be approximate by $L_{\mu}$ is
logically equivalent to the statement that $cos\theta_{\nu}$ cannot be
approximated by $cos\theta_{\mu}$, where $cos\theta_{\nu}$ denotes cosine of
the zenith angle of the incident neutrino and $cos\theta_{\mu}$ denotes that
of the produced muon, respectively Konishi1 . In Super-Kamiokande
Collaboration, they approximate $cos\theta_{\nu}$ as $cos\theta_{\mu}$ ( See
the reproduction of their statements in the 2 page in the present paper ). The
analysis of the zenith angle distribution of the atmospheric neutrino events
by Super-Kamiokande Collaboration will be re-examined in our subsequent
papers.

## References

  * (1) Konishi,E et al., arXiv hep-ex/1007.3812v1
  * (2) Ashie,Y. et al., Phys. Rev. D 71 (2005) 112005.
  * (3) Honda, M., et al., Phys. Rev. D 52 (1996) 4985.
Honda, M., et al., Phys. Rev. D 70 (2004)043008-1.

  * (4) Ashie,Y et al., Phys.Rev.Lett.93 (2004)101801-1.
  * (5) Konishi,E et al., arXiv hep-ex/0808.0664v2


# EdinSaar@WMT21: North-Germanic Low-Resource Multilingual NMT

Svetlana Tchistiakova1, Jesujoba O. Alabi2,
Koel Dutta Chowdhury2, Sourav Dutta3, Dana Ruiter2
1The University of Edinburgh, Edinburgh, Scotland
2Saarland University, Saarbrücken, Germany
3Technical University of Kaiserslautern, Kaiserslautern, Germany
Corresponding author<EMAIL_ADDRESS>
###### Abstract

We describe the EdinSaar submission to the shared task of Multilingual Low-
Resource Translation for North Germanic Languages at the Sixth Conference on
Machine Translation (WMT2021). We submit multilingual translation models for
translations to/from Icelandic (is), Norwegian-Bokmål (nb), and Swedish (sv).
We employ various experimental approaches, including multilingual pre-
training, back-translation, fine-tuning, and ensembling. In most translation
directions, our models outperform other submitted systems.

##  1 Introduction

This paper presents the neural machine translation (NMT) systems jointly
submitted by The University of Edinburgh and Saarland University to the
WMT2021 Multilingual Low-Resource Translation for Indo-European Languages
task, describing both primary and contrastive systems which translate to/from
the three North Germanic languages, Icelandic (is), Norwegian-Bokmål (nb), and
Swedish (sv). Our contrastive system, submitted as “edinsaarContrastive”
outperforms the other submissions across all evaluation metrics except for
BLEU, for which our “edinsaarPrimary” system performs best.

Although low-resource MT has recently gained much attention, there is little
prior work on North Germanic languages. We contribute to this space by
experimenting with both training a multilingual system from scratch and
exploiting model adaptation from a large pre-trained language model. We fine-
tune our initial translation models to the target languages, and then
experiment with further in-domain fine-tuning. Data is sourced from openly
available data sets in accordance with the corpora allowed in the shared task.
We use parallel data sets pairing our target languages with each other and
with the allowed high-resource languages, and monolingual data from Wikipedia.

The rest of the paper is structured as follows: we review related work in
Section 2, we introduce the methods and experimental settings including data
and model architecture in Section 3, we evaluate model performance in Section
4, and, finally, we draw conclusions and suggest avenues for future work in
Section 5.

##  2 Related Work

Recent work in NMT for North Germanic languages is limited; however, OPUS-MT
Tiedemann and Thottingal (2020), which contains over 1,000 pre-trained, ready-
to-use neural MT models including models for Danish, Norwegian, and Swedish,
is a notable exception.

Due to the scarcity of parallel data for low-resource languages, recent work
leverages monolingual data, including pivoting from high-resource languages
Currey and Heafield (2019); Kim et al. (2019), and using back-translation
Sennrich et al. (2016a); Edunov et al. (2018) to generate pseudo-parallel data
with synthetic sources from monolingual data. Since the little parallel data
that is available often comes from noisy web crawls, parallel corpus filtering
is used to develop better translation models Koehn et al. (2020). Additional
methods for boosting the performance of low-resource pairs include transfer
learning from models trained on higher-resource pairs Zoph et al. (2016);
Kocmi and Bojar (2018), and developing multilingual systems to allow models to
take advantage of linguistic relatedness. Multilingual systems can employ
either separate encoders or decoders for each language Dong et al. (2015);
Firat et al. (2016), or shared encoders/decoders, and can additionally make
zero-shot MT possible Johnson et al. (2017); Ha et al. (2016), while scaling
to hundreds of language pairs Aharoni et al. (2019); Fan et al. (2020).
Sampling language pairs in proportion to their prevalence in the training data
can ensure that all directions get enough coverage by the model Arivazhagan et
al. (2019); Fan et al. (2020). Further fine-tuning multilingual systems on
target language directions can improve performance of low-resource pairs
Neubig and Hu (2018); Lakew et al. (2019). Adapting a multilingual pre-trained
language model to the translation task has led to improvements in translation
quality Clinchant et al. (2019); Chen et al. (2020). Finally, combining
multiple MT system checkpoints together by ensembling improves performance of
the final system Sennrich et al. (2017).

##  3 Method

Given a set of primary languages $L_{p}$ and secondary languages $L_{s}$, we
train a multilingual MT system on the parallel data between all the language
combinations $\\{L_{p},L_{s}\\}\leftrightarrow\\{L_{p},L_{s}\\}$. This is our
baseline. We extend this approach with a combination of the following methods:

  * Pre-training: We initialize a base model using a highly multilingual pre-trained model, in order to transfer the learned parameters to the translation task. This is our primary system.

  * Back-translation: We use the baseline model to back-translate monolingual corpora in $L_{p}$ into all other languages in $L_{p}$ to obtain a training data set of back-translations $D_{\rm BT}$.

  * Fine-tuning: We fine-tune the baseline model on the subset of languages $\\{L_{p},L_{s}\\}\leftrightarrow L_{p}$, on both parallel and back-translated data $D_{\rm BT}$. Our contrastive system is an ensemble of the last four checkpoints of this model.

###  3.1 Data

For training our models, we include data from the target primary low-resource
languages, Icelandic (is), Norwegian-Bokmål (nb), and Swedish (sv), and the
related secondary languages Danish (da), German (de), English (en).

We use data for all translation directions involving da, de, en, is, nb, sv
from the following parallel corpora from Opus: Bible Christodouloupoulos and
Steedman (2014), Books Tiedemann (2012), Europarl (Koehn, 2005), GlobalVoices
Tiedemann (2012), JW300 (Agić and Vulić, 2019), MultiCCAligned (El-Kishky et
al., 2020), Paracrawl Esplà et al. (2019), TED2020 (Reimers and Gurevych,
2020), and WikiMatrix (Schwenk et al., 2019). We also use all corpora from
ELRC111https://elrc-share.eu/ that include these directions (a total of 159
corpora, retrieved in May 2021). These corpora include all corpora allowed by
the shared task, with the exception of the Opus-100 data set, which we avoided
as it had many duplicate sentences with the above corpora.

We use monolingual data from Wikipedia for is and nb to augment our data set
with back-translations (Sennrich et al., 2016a). Because the Wikipedia data
for sv was created in large part by a
bot222https://en.wikipedia.org/wiki/Lsjbot and consisted of many stub articles
and tables, we use the sv portion of our training data as monolingual data for
back-translation instead.

Our final data includes 30 language directions:

  1. (a)

$L_{p}\leftrightarrow L_{p}$: {is,nb,sv} $\leftrightarrow$ {is,nb,sv}

  2. (b)

$L_{p}\leftrightarrow L_{s}$: {is,nb,sv} $\leftrightarrow$ {da,de,en}

  3. (c)

$L_{s}\leftrightarrow L_{s}$: {da,de,en} $\leftrightarrow$ {da,de,en}

  4. (d)

$L_{p\\_bt}\ \scaleobj{0.75}{\rightarrow}L_{p}$: {is,nb,sv}
$\scaleobj{0.75}{\rightarrow}$ {is,nb,sv}

where $L_{p\\_bt}$ is created from the monolingual target side back-translated
data $D_{\rm BT}$.

| de |  | en |  | is |  | nb |  | sv |
---|---|---|---|---|---|---|---|---|---|---
da | 6921831 | (48) | 20604309 | (77) | 797806 | (68) | 10654 | (89) | 5590356 | (65)
de |  |  | 144890166 | (80) | 456054 | (62) | 24963 | (91) | 5119372 | (59)
en |  |  |  |  | 3766342 | (78) | 279370 | (46) | 21906032 | (78)
is |  |  |  |  | 351833 | (60) | 597 | (89) | 446106 | (46)
nb |  |  |  |  |  |  | 2943733 | (44) | 14247 | (89)
Table 1: Number of sentences after filtering (with % of total raw data
remaining after filtering) in each language direction from source (left) to
target (top) from all corpora and for additional monolingual data from
Wikipedia. The parallel data was mirrored in the reverse directions to create
30 total language directions for training.

#### Parallel Data Filtering

We filter the parallel data using rule-based heuristics borrowed from the
Bifixer/Bicleaner tools (Sánchez-Cartagena et al., 2018; Ramírez-Sánchez et
al., 2020) and language identification using FastText Joulin et al. (2016,
2017). This repairs common orthographic errors, including fixing failed
renderings of glyphs due to encoding errors, replacing characters from the
wrong alphabet with correct ones, and un-escaping html. It also removes any
translation pairs where: the pair is a duplicate, the source and target are
identical, the source or target language is not the intended language, one
side is more than $2$x the length of the other, one side is empty, one side is
longer than $5000$ characters, one side is shorter than 3 words, or one side
contains primarily URLs and symbols rather than text.

Filtering reduces our parallel data to $77\%$ of its original total size. This
data is then reversed in order to train our multilingual model in all
translation directions, resulting in a total of 421,656,410 parallel sentence
pairs in all 30 language directions. Table 1 lists the filtered data counts
and the percentage of the original data that these counts represent.

#### Monolingual In-Domain Data Filtering

The validation set provided by the shared task organizers, containing thesis
abstracts and descriptions, is dissimilar to our available parallel corpora.
Therefore, we filter the Wikipedia monolingual is and nb data for similarity
to this validation set to create in-domain monolingual data for use in back-
translation. We identify in-domain monolingual instances in our data by
calculating the cosine similarity between each sentence in a given language in
the monolingual data to each of the sentences in the shared task validation
data for that language. When a training instance has a similarity of
$>=\theta$ with at least one validation instance, it is added to the in-domain
fine-tuning corpus. We set $\theta=0.9$ and use LASER (Artetxe and Schwenk,
2019) to extract vector representations of sentences for calculating
similarity.

#### Validation and Test Data

We split off 2000 sentence pairs from each language pair in our parallel data
to use as an internal test set. For is-nb directions, we use the few parallel
sentences available for this, meaning that no parallel data is left for the
training or validation corpus. Therefore, translating between these directions
is a zero-shot task for our models.

We also split off 2000 sentence pairs from each language pair in our parallel
data for internal validation. For validation of our primary model, we use the
entire collection of 2000 validation sentence pairs in each language
direction. For the baseline system, we cut this down to a total of $\sim 2000$
sentences, because performing validation is quicker on smaller data.
Therefore, we use a subset of 72 validation sentences in each
$\\{L_{p},L_{s}\\}\leftrightarrow\\{L_{p},L_{s}\\}$, except is-nb, resulting
in 2016 sentences. For the contrastive model, we use the same sentences in
only $\\{L_{p},L_{s}\\}\leftrightarrow\\{L_{p}\\}$, to which we add 72
sentences from the back-translated data in the is-nb directions, resulting in
a total of 1728 sentences.

We use the shared task validation set, to compare performance between our
systems, and do not use it during model training or fine-tuning. We
additionally report results Section 4 on the shared task test set, which was
provided to the teams after the completion of the shared task. These test sets
contain approximately 500 sentences in each language direction.

#### Back-translation

We use the baseline system (Section 3.3) to create back-translations of our
monolingual in-domain filtered Wikipedia data. This generates synthetic
sources from is to {nb, sv} and from nb to {is, sv}. We additionally back-
translate the sv side of our parallel nb-sv corpus into is and our is-sv
corpus into nb. After creating the back-translations, we filter the new
synthetic parallel data sets again using the parallel data filtering steps
(Section 3.1), in order to remove sentences that consisted primarily of model
errors or hallucinations. The final counts of filtered back-translated data
are in Table 2, as well as the percentage of the original total in-domain data
that these counts represent.

| is | nb | sv
---|---|---|---
is |  | 2564234 (87) | 10123 (99)
nb | 279818 (80) |  | 344583 (78)
sv | 299277 (85) | 2521823 (86) |
Table 2: Number of back-translated filtered sentences (with % of total data
remaining after filtering) between synthetic source (left) to original target
(top).

###  3.2 Byte-pair Encoding

To create a vocabulary for our baseline and contrastive systems, we train a
shared byte-pair encoding (BPE) (Sennrich et al., 2016b) model using
SentencePiece (Kudo and Richardson, 2018). We sample 10 million monolingual
sentences from our parallel training data, based on the amount of monolingual
data available for each language. Following the idea of Arivazhagan et al.
(2019), we use temperature sampling, where the probability of sampling any
particular data set $D$ in language $\ell$ out of the $n$ total data sets is
defined as $p_{\ell}=(\frac{D_{\ell}}{\sum_{i}^{n}{D_{i}}})^{\frac{1}{T}}$,
where we set $T=5$. The goal of sampling in this way is to provide a
compromise that allows the BPE model to view a larger portion of lower
resource language tokens (unlike sampling according to the original
distribution would), while still providing extra space in the model for the
larger variety of tokens coming from high-resource corpora (unlike sampling
uniformly would). We use a vocabulary of $32,000$ tokens. When BPE-ing our
training data, we use BPE-dropout Provilkov et al. (2020) with a probability
of $0.1$.

###  3.3 Models

#### Baseline

Our baseline system is trained on a concatenation of data sets (a), (b), and
(c) (see Section 3.1). The data is pre-processed using byte-pair encoding as
described in Section 3.2. Following the method of Johnson et al. (2017), we
jointly train the model to translate in all our language directions, pre-
pending a token <2xx> to the source side to inform the model which target
language to translate into. The system is comprised of a transformer base
model trained using Marian Junczys-Dowmunt et al. (2018) with cross-entropy
loss, following the method of Vaswani et al. (2017) and the default Marian
transformer configuration.

We differ from the default configuration in the following ways. We fit our
mini-batch to a workspace of $6144$ MB, set the learning rate to $0.0003$ with
a warm-up increasing linearly for $16000$ batches and decaying by
$\frac{16000}{\sqrt{no.\ batches}}$ afterwards. We train on multiple GPUs
using Adam (Kingma and Ba, 2014) with synchronous updates for optimization,
setting $\beta_{1}=0.9$, $\beta_{2}=0.98$ and $\epsilon=1e-09$. We set
transformer dropout between layers to $0.01$. We use a maximum sentence length
of $200$ tokens, a maximum target length as source length factor of $2$, and a
label smoothing of $0.01$. During validation, we use a beam size of $6$ and
normalize the translation score by $translation\\_length^{0.6}$. We check
translation quality on our internal validation set (Section 3.1) every 5000
model updates and stop training when performance doesn’t improve for 15
checkpoints. The model was trained for approximately 66 hours on four NVIDIA
GeForce RTX 3090 GPUs.

#### Contrastive

Our contrastive model fine-tunes the baseline model directly, using a
concatenation of all data sets that incorporate our target languages,
including parallel and back-translated data (the data sets (a), (b), and (d)
described in Section 3.1). The fine-tuned model uses the same architecture,
training settings, and stopping criterion as the original baseline model,
essentially allowing us to continue training further from the original
baseline. The final submitted system is an ensemble of the last four
checkpoints of this model. The model was trained for approximately 54 hours on
two NVIDIA GeForce RTX 2080 TI GPUs.

#### Primary

For the primary system, we adapt mt5 Xue et al. (2020), a multilingual pre-
trained transformer language model, to the translation task. We use mt5
because of its state-of-the-art performance and its coverage of all of our
target North Germanic languages. We use the
SimpleTransformers333https://github.com/ThilinaRajapakse/simpletransformers
framework which extends HuggingFace Wolf et al. (2019), with the default
parameters. Since our model is initialized from the parameters of the mt5-base
system, including the embedding layers, we use the same byte-pair encoded
vocabulary as the original model. Due to resource constraints, we sample a
total of $100$k parallel sentences from data sets (a), (b) and (c) (described
in Section 3.1). We pre-pend a string to the source side to indicate to the
model which target language to translate into, and adapt the model for 5
epochs. We further fine-tune this model on 100k additional sentences that
include our target languages from sets (a) and (b) to create our Primary
system. The model was trained for approximately 46 hours on a single NVIDIA
A100 SXM4 GPU.

| Model | is $\rightarrow$ nb | is $\rightarrow$ sv | nb $\rightarrow$ is | nb $\rightarrow$ sv | sv $\rightarrow$ is | sv $\rightarrow$ nb | Avg.
---|---|---|---|---|---|---|---|---
Internal test | marian | 12.5 | 33.3 | 11.8 | 26.7 | 27.8 | 18.7 | 21.8
marian_ft | 19.1 | 41.7 | 16.1 | 31.6 | 38.4 | 30.3 | 29.5
marian_ft_esmb | 19.3 | 42.2 | 16.4 | 31.6 | 39.2 | 30.3 | 29.8
mt5_base_ada | 23.1 | 42.3 | 19.4 | 33.7 | 42.8 | 33.9 | 32.5
mt5_base_ada_ft | 26.5 | 42.9 | 20.0 | 33.9 | 43.3 | 34.2 | 33.5
Shared valid | marian | 10.9 | 13.5 | 15.1 | 41.3 | 12.2 | 24.9 | 19.7
marian_ft | 13.0 | 18.0 | 22.9 | 50.0 | 19.4 | 45.9 | 28.2
marian_ft_esmb | 13.9 | 18.2 | 23.6 | 50.6 | 20.1 | 46.7 | 28.8
mt5_base_ada | 14.6 | 19.2 | 25.8 | 46.6 | 20.6 | 43.2 | 28.3
mt5_base_ada_ft | 17.4 | 18.7 | 26.5 | 47.9 | 20.8 | 44.2 | 29.3
Shared test | marian_ft_esmb | 13.0 | 17.3 | 18.3 | 45.4 | 20.2 | 48.2 | 27.1
marian_base_ada_ft | 16.3 | 18.8 | 19.5 | 42.9 | 22.4 | 45.4 | 27.5
Table 3: SacreBLEU (detokenized) results on the internal test set and the
shared task validation and test sets.

##  4 Evaluation

Table 3 reports results on detokenized SacreBLEU on each of our internal test
set, the shared task validation set, and the shared task test
set444BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+version.1.4.14. Comparing
results on the internal test set and shared task validation sets show that our
models fail to generalize well to the shared task domain, in particular, in
the least represented languages in our data set such as is. In future work, we
would like to experiment with different sampling methods to boost the
performance of the least represented directions.

Comparing results between models, our primary mt5_base_ada system outperforms
the marian model trained from scratch by an average of $+10.7$ and $+8.6$ BLEU
points on the internal and shared task validation sets, respectively. The
further fine-tuned variant mt5_base_ada_ft leads to an additional average
improvement of just under $+1$ BLEU point on both sets, showing that the mt5
model already learned a good amount about our target task and languages from
our initial adaptation step. The marian model is also outperformed by the
fine-tuned variant marian_ft, resulting in an average improvement of $+7.7$
BLEU points on the internal test set and $+8.5$ BLEU points on the shared task
validation set.

Both the mt5_base_ada_ft and marian_ft models are exposed to similar language
data; however, the mt5 language model we adapted from (mt5-base) is much
larger than our marian model ($580$ million vs $44$ million parameters), and
was trained on more language data ($750$ GB vs $46$ GB), so it had a much
stronger base to start from. Ensembling the last $4$ checkpoints of the fine-
tuned marian model for marian_ft_esmb boosts performance by $+0.3$ and $+0.6$
average BLEU on the internal and shared task validation sets over marian_ft;
however, the mt5_base_ada_ft model still outperforms the marian_ft_esmb model
by $+3.7$ and $+0.5$ average BLEU on the internal test set and the shared task
validation set, respectively. Therefore, we submitted the mt5_base_ada_ft
model as our primary system to the shared task; however, our contrastive
system, the marian_ft_esmb model, won in the shared task rankings.

In the global automated evaluations of the shared task, our contrastive system
is the best-performing submitted system555Only our primary model was submitted
for manual evaluation, where it outranked the other submissions. Official
rankings are available at: http://statmt.org/wmt21/multilingualHeritage-
translation-task.html, outperforming the official mt5 baseline by
approximately $+8.5$ BLEU. We hypothesize that the mt5 baseline, while being
pre-trained on massive amounts of partially noisy monolingual data, has
learned the translation task via training on the development set only, so it
has less informative parallel data available than our models. The M2M-100 Fan
et al. (2020) baseline outperforms all submitted systems, despite having been
trained on noisy parallel data only. We hypothesize that the highly-
multilingual nature of the M2M-100 model allows the target languages to
benefit from the supervisory signals between related language combinations.

##  5 Conclusion and Future Work

We contribute to the growing space of NMT for North Germanic languages. We
explore multilingualism by training a transformer with a shared encoder and
decoder for all language pairs from scratch, as well as adapting a pre-trained
multilingual language model. Fine-tuning these models to our low-resource
language pairs was a key component in our success in the task, and we
additionally confirm that employing popular techniques in machine translation,
such as data filtering, back-translation, and model ensembling are beneficial
for improving performance on low-resource directions. In future work, we would
like to experiment with fine-tuning additional pre-trained models such as the
M2M-100, incorporating iterative back-translation, and trying different
sampling methods during training to boost lower performing low-resource
language pairs.

## Acknowledgements

The authors thank the University of Edinburgh and the German Research Center
for Artificial Intelligence (DFKI GmbH) for providing the necessary
infrastructure and resources to run the experiments.

This research is based upon work supported in part by the Office of the
Director of National Intelligence (ODNI), Intelligence Advanced Research
Projects Activity (IARPA), via contract #FA8650-17-C-9117. The views and
conclusions contained herein are those of the authors and should not be
interpreted as necessarily representing the official policies, either
expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S.
Government is authorized to reproduce and distribute reprints for governmental
purposes notwithstanding any copyright annotation therein.

## References

  * Agić and Vulić (2019) Željko Agić and Ivan Vulić. 2019.  JW300: A wide-coverage parallel corpus for low-resource languages.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3204–3210, Florence, Italy. Association for Computational Linguistics.
  * Aharoni et al. (2019) Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019.  Massively multilingual neural machine translation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3874–3884, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Arivazhagan et al. (2019) Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019.  Massively multilingual neural machine translation in the wild: Findings and challenges.  _CoRR_ , abs/1907.05019.
  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.  _Transactions of the Association for Computational Linguistics_ , 7:597–610.
  * Chen et al. (2020) Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. 2020.  Distilling knowledge learned in BERT for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7893–7905, Online. Association for Computational Linguistics.
  * Christodouloupoulos and Steedman (2014) Christos Christodouloupoulos and Mark Steedman. 2014.  A massively parallel corpus: the bible in 100 languages.  _Language Resources and Evaluation_ , 49(2):375–395.
  * Clinchant et al. (2019) Stephane Clinchant, Kweon Woo Jung, and Vassilina Nikoulina. 2019.  On the use of BERT for neural machine translation.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , pages 108–117, Hong Kong. Association for Computational Linguistics.
  * Currey and Heafield (2019) Anna Currey and Kenneth Heafield. 2019.  Zero-resource neural machine translation with monolingual pivot data.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , pages 99–107, Hong Kong. Association for Computational Linguistics.
  * Dong et al. (2015) Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015.  Multi-task learning for multiple language translation.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1723–1732, Beijing, China. Association for Computational Linguistics.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding back-translation at scale.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
  * El-Kishky et al. (2020) Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020.  CCAligned: A massive collection of cross-lingual web-document pairs.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020)_ , pages 5960–5969, Online. Association for Computational Linguistics.
  * Esplà et al. (2019) Miquel Esplà, Mikel Forcada, Gema Ramírez-Sánchez, and Hieu Hoang. 2019\.  ParaCrawl: Web-scale parallel corpora for the languages of the EU.  In _Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks_ , pages 118–119, Dublin, Ireland. European Association for Machine Translation.
  * Fan et al. (2020) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020.  Beyond english-centric multilingual machine translation.  _arXiv preprint_.
  * Firat et al. (2016) Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016.  Multi-way, multilingual neural machine translation with a shared attention mechanism.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 866–875, San Diego, California. Association for Computational Linguistics.
  * Ha et al. (2016) Thanh-Le Ha, Jan Niehues, and Alexander H. Waibel. 2016.  Toward multilingual neural machine translation with universal encoder and decoder.  _CoRR_ , abs/1611.04798.
  * Johnson et al. (2017) Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017.  Google’s multilingual neural machine translation system: Enabling zero-shot translation.  _Transactions of the Association for Computational Linguistics_ , 5:339–351.
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016.  Fasttext.zip: Compressing text classification models.  _arXiv preprint arXiv:1612.03651_.
  * Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017.  Bag of tricks for efficient text classification.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 427–431, Valencia, Spain. Association for Computational Linguistics.
  * Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018.  Marian: Fast neural machine translation in C++.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 116–121, Melbourne, Australia. Association for Computational Linguistics.
  * Kim et al. (2019) Yunsu Kim, Petre Petrov, Pavel Petrushkov, Shahram Khadivi, and Hermann Ney. 2019\.  Pivot-based transfer learning for neural machine translation between non-English languages.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 866–876, Hong Kong, China. Association for Computational Linguistics.
  * Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _International Conference on Learning Representations_.
  * Kocmi and Bojar (2018) Tom Kocmi and Ondřej Bojar. 2018.  Trivial transfer learning for low-resource neural machine translation.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 244–252.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A Parallel Corpus for Statistical Machine Translation.  In _Conference Proceedings: the tenth Machine Translation Summit_ , pages 79–86, Phuket, Thailand. AAMT, AAMT.
  * Koehn et al. (2020) Philipp Koehn, Vishrav Chaudhary, Ahmed El-Kishky, Naman Goyal, Peng-Jen Chen, and Francisco Guzmán. 2020.  Findings of the WMT 2020 shared task on parallel corpus filtering and alignment.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 726–742, Online. Association for Computational Linguistics.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  _CoRR_ , abs/1808.06226.
  * Lakew et al. (2019) Surafel Melaku Lakew, Alina Karakanta, Marcello Federico, Matteo Negri, and Marco Turchi. 2019.  Adapting multilingual neural machine translation to unseen languages.  _CoRR_ , abs/1910.13998.
  * Neubig and Hu (2018) Graham Neubig and Junjie Hu. 2018.  Rapid adaptation of neural machine translation to new languages.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 875–880, Brussels, Belgium. Association for Computational Linguistics.
  * Provilkov et al. (2020) Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020.  BPE-dropout: Simple and effective subword regularization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1882–1892, Online. Association for Computational Linguistics.
  * Ramírez-Sánchez et al. (2020) Gema Ramírez-Sánchez, Jaume Zaragoza-Bernabeu, Marta Bañón, and Sergio Ortiz-Rojas. 2020.  Bifixer and bicleaner: two open-source tools to clean your parallel data.  In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation_ , pages 291–298, Lisboa, Portugal. European Association for Machine Translation.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4512–4525, Online. Association for Computational Linguistics.
  * Sánchez-Cartagena et al. (2018) Víctor M. Sánchez-Cartagena, Marta Bañón, Sergio Ortiz-Rojas, and Gema Ramírez-Sánchez. 2018.  Prompsit’s submission to wmt 2018 parallel corpus filtering shared task.  In _Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers_ , Brussels, Belgium. Association for Computational Linguistics.
  * Schwenk et al. (2019) Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019.  Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia.  _CoRR_ , abs/1907.05791.
  * Sennrich et al. (2017) Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, and Philip Williams. 2017.  The University of Edinburgh’s neural MT systems for WMT17.  In _Proceedings of the Second Conference on Machine Translation_ , pages 389–399, Copenhagen, Denmark. Association for Computational Linguistics.
  * Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a.  Improving neural machine translation models with monolingual data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 86–96, Berlin, Germany. Association for Computational Linguistics.
  * Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , Istanbul, Turkey. European Language Resources Association (ELRA).
  * Tiedemann and Thottingal (2020) Jörg Tiedemann and Santhosh Thottingal. 2020.  Opus-mt–building open translation services for the world.  In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation_ , pages 479–480.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Xue et al. (2020) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020.  mt5: A massively multilingual pre-trained text-to-text transformer.  _CoRR_ , abs/2010.11934.
  * Zoph et al. (2016) Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016.  Transfer learning for low-resource neural machine translation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1568–1575.