# Learning Second Order Local Anomaly for General Face Forgery Detection

Jianwei Fei1111These authors contributed equally to this work. Yunshu
Dai1111These authors contributed equally to this work. Peipeng Yu2 Tianrun
Shen3 Zhihua Xia2,1222Corresponding author. Jian Weng2
1Nanjing University of Information Science and Technology 2Jinan University
3Nanjing University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this work, we propose a novel method to improve the generalization ability
of CNN-based face forgery detectors. Our method considers the feature
anomalies of forged faces caused by the prevalent blending operations in face
forgery algorithms. Specifically, we propose a weakly supervised Second Order
Local Anomaly (SOLA) learning module to mine anomalies in local regions using
deep feature maps. SOLA first decomposes the neighborhood of local features by
different directions and distances and then calculates the first and second
order local anomaly maps which provide more general forgery traces for the
classifier. We also propose a Local Enhancement Module (LEM) to improve the
discrimination between local features of real and forged regions, so as to
ensure accuracy in calculating anomalies. Besides, an improved Adaptive
Spatial Rich Model (ASRM) is introduced to help mine subtle noise features via
learnable high pass filters. With neither pixel level annotations nor external
synthetic data, our method using a simple ResNet18 backbone achieves
competitive performances compared with state-of-the-art works when evaluated
on unseen forgeries.

##  1 Introduction

Recent progress in face synthesis technologies allows the low-entry production
of sophisticated fake facial content which causes severe trust issues.
Concerns are growing over the nefarious use of such face forgery technologies.
To deal with this problem,

Figure 1: Second Order Local Anomaly (SOLA) learning module decomposes the
neighborhood and predicts the first and second order anomaly maps.

many methods have been developed to detect face forgeries using different
traces, such as obvious visual artifacts [24, 39, 29], frequency domain cues
[10, 12, 33, 20], temporal anomalies [38, 28, 47, 3, 37], or multimodality
conflicts [5, 30, 1]. But these traces are not so generalizable that the
detectors may fail when encountering unseen forgeries. Thus recent works are
looking for more universal forgery traces, so as to boost the
generalizability.

All the face forgery algorithms, whether face swapping or reenactment, need to
blend the forged regions into the original background. The two parts
inevitably hold different features, especially in high frequency regions,
resulting in anomaly in forged images. On such observation, some methods are
proposed to capture the feature inconsistency [22, 4, 52]. [52] propose a
patch wise consistency learning method to mine anomalies in forged face images
and succeed in generalizing to unseen forgeries. However, these works treat
regions with various distances equally, which violates the fact that natural
images hold different dependencies of short and long distances. Moreover, they
usually require either pixel level annotations or external synthetic data.
Although pixel level annotations can be created in lab environments, the need
for them limits the usage of forged faces transmitted in the real world.

In this work, we focus on capturing forgery traces from the perspective of
local anomaly. Specifically, given a face image, we extract its deep features
with a CNN backbone, and divide the neighborhood of a local feature into 4
groups by different orientations and distances, thereby modeling more fine-
grained local anomalies and generating 4 groups of anomaly maps. These anomaly
maps are then decomposed again to calculate the second order anomaly which
have a wide range of response to the first order anomaly. The overview is
shown in Fig. 1, and more details are in Sec. 3.3. Our method requires neither
pixel level annotations nor external data but achieves great cross-domain
performance with a small backbone.

Figure 2: Overview of the proposed method (with ResNet18 backbone).

Our contributions can be summarized into four-fold:

  * •

We propose a second order local anomaly (SOLA) learning module for isolating
forgery faces. SOLA calculates anomalies of a local region in fine-grained
fashion and magnifies the forgery traces via calculating the second order
anomalies. Classifiers using the anomaly features are proved to have
remarkable generalization on unseen forgeries.

  * •

We propose a local enhancement module (LEM) that can be inserted after
different stages of the backbone to ensure the discrimination between real and
forged regions in deep local features.

  * •

We design an adaptive spatial rich model (ASRM) which inherits the high pass
peculiarity of SRM filters in forensics and is able to adaptively adjust the
filters according to data.

  * •

Our method requires neither pixel level annotations nor external synthetic
data to improve the generalization of face forgery detectors.

##  2 Related work

Face forgeries can be seen as a game of AI versus AI since most detection
technologies are deep learning based. Although these methods perform well
under in-domain evaluations, they often suffer performance degeneration on
unseen forgeries.

Generalization to Unseen Forgeries. Many efforts are made to improve the
generalizability of detectors such as combining auxiliary localization tasks
to guide the network focus more on forged regions [22, 8], improving cross
entropy loss with metric learning for better class discrimination [20, 28],
and introducing domain adaptation to alleviate overfitting on a single domain
[35, 41]. Recently, methods using frequency cues [33, 26] or combining
features from different domains also perform well [27, 28] in cross-domain
evaluations. In [20], one branch process the RGB input, while another uses the
DCT transformation to extract high frequency features from different bands.
Outputs of the two branches are fused to form more generalized forgery
features. Many works similarly fuse features from RGB and frequency domains
for more general representation [15, 27, 4, 17]. However, their methods of
extracting high frequency features can not fit the data adaptively to capture
the most discriminative features.

Anomaly-based Face Forgery Detection. Some works introduce the idea of anomaly
detection into image forensics [6, 11] as well as face forgery detection [16,
18], and achieve good generalization. For example, Wu et al. [48] calculate
the differences between local feature and dominant feature of an image,
turning the forgery localization into local anomaly detection. Hu et al. [16]
propose a dynamic inconsistency aware network to capture both global
inconsistency of adjacent frames and local inconsistency of key regions within
frames.

Recent methods similarly utilize the patch level anomalies for face forgery
detection. They are mainly based on the observation that face forgery
algorithms always blend forged regions with original face context, resulting
in pixel statistical property anomalies within one image [52]. An example of
the pixel statistical property is camera noise [7] which is a high frequency
feature left during the imaging process and has been used for detecting image
integrity [44]. [52] and [4] propose to divide deep feature maps into patches,
and calculate the similarities between patches to form a more generalized
pattern that indicates the image integrity. They achieve both great
performances under in-domain and cross-domain evaluations. Unlike these
methods that explore global level consistency, our method explores more fine-
grained local anomaly instead.

##  3 Method

The overview of our method is illustrated in Fig. 2. Given an input face
image, it is fed to the RGB branch, as well as a parallel noise branch whose
first layer is our Adaptive Spatial Rich Model (ASRM). These intermediate
features extracted from these two branches are fused through Dual Channel
Attention Module (DCAM) after each block. Then the Local Enhancement Module
(LEM) divide the fused feature maps into patches and enhance the category
attributes of each local feature inside. Finally, the Second Order Local
Anomaly (SOLA) learning module calculates fine-grained local anomalies of the
first and second order, and predicts whether the input is forged.

###  3.1 Adaptive SRM in Noise Branch

Motivation. Learning from the RGB data is insufficient to capture high
frequency features that are crucial for image forensics. To solve this
problem, SRM (Spatial Rich Model) [13] has been widely applied for
preprocessing [15] and layer initialization [34] to extract high frequency
noise. But if used for preprocessing, the handcrafted filters in SRM can not
adaptively update to fit the data. Besides, if used for initialization, the
backpropagation will adjust the filters and break their high pass peculiarity.
In this work, we address this dilemma by introducing a constraint to let SRM
adaptively update while maintaining their high frequency learning ability.

Figure 3: The Adaptive Spatial Rich Model (ASRM) convolution.

Design of ASRM. We start with original SRM following [13]. SRM models the
noise residuals $R_{i,j}$ using high pass filters by:

$R_{i,j}=P(\mathcal{N}_{i,j})-I_{i,j},$ (1)

where $\mathcal{N}_{i,j}$ is the neighborhood of center pixel $I_{i,j}$,
$P(\cdot)$ is a predictor of $I_{i,j}$ based on $\mathcal{N}_{i,j}$. The
residual is then quantitated by $q$, following with rounding and truncation:

$R_{i,j}\leftarrow trun(round(\frac{R_{i,j}}{q})).$ (2)

We can see that the $P(\cdot)$ can be realized by a standard convolution.
Following [2], let $\emph{{I}}*\emph{{w}}$ denotes a standard convolution on
input image I with kernel w, and $o$ notes an impulse filter with central
value being 1 and others being 0, we then have:

$\emph{{h}}=\emph{{I}}*\emph{{w}}=\emph{{I}}*\hat{\emph{{w}}}-\emph{{I}}=\emph{{I}}*(\hat{\emph{{w}}}-o),$
(3)

where the central value of $\hat{\emph{{w}}}$ is 0. Then the central value of
w is -1 and $\emph{{I}}*\emph{{w}}$ is equal to Eq. 1.

We select 3 out of 30 filters in SRM as they have been experimentally proved
to be effective enough [53]. The filters are repeated 3 times for RGB images
as shown in Fig.3. To let their center values be -1, we quantitate them by 2,
4, and 12 respectively. We can see that after quantitation, the sum of the
remaining elements is 1 which perfectly fit the requirements of Eq. 3. To
allow the filters learnable during network training while keep their
peculiarity in high pass filtering, we reset their central element to -1 and
normalize the remaining elements to force their sum to be 1:

$\begin{cases}\emph{{w}}_{0,0}=-1,\\\ \sum\nolimits_{i\neq 0,j\neq
0}\emph{{w}}_{i,j}=1,\end{cases}$ (4)

where $i,j$ denotes the index of elements in kernels and $(0,0)$ is the
central one. We execute this constraint after each backpropagation.

ASRM is the first layer used for the preprocessing of the noise branch.
Different from using fixed SRM kernels, ASRM enables the kernels to adaptively
update during the backpropagation and extract discriminative noise features
that can not be directly learned from RGB data.

Figure 4: The Local Enhancement Module (LEM).

The noise features extracted by ASRM provide subtle high frequency cues, but
different channels of the feature maps have different contributions to
generalization. Besides, different channels in the fused features are of
different importance in forgery representations. Thus, we propose the Dual
Channel Attention Module (DCAM) to fuse the intermediate features of RGB and
noise branches. DCAM uses channel attention [46] twice to emphasize channels
of more importance before and after the fusion.

###  3.2 Local Enhancement Module

SOLA calculates local anomalies on such an assumption: each pixel in the
feature maps is a local feature vector associated with a corresponding local
region whose size is the ratio of input size to feature size. However, a
feature vector is associated with a larger region due to the expansion of
receptive field after several layers. The feature vectors are thereby affected
by information of outer regions and lose their discrimination in representing
local regions. We notice that some works have been proposed to capture long
range dependencies in images [45, 50] with self attention. Following this
idea, we propose the Local Enhancement Module (LEM) to improve the
discrimination of local feature vectors by assigning them with different
weights.

Specifically, LEM takes as input the intermediate feature maps
$\textbf{F}^{l}\in\mathbb{R}^{h_{l}\times w_{l}\times c_{l}}$ where $l$
denotes the hierarchy. We spatially divide $\textbf{F}^{l}$ into 16 $\times$
16 non-overlapping patches $\textbf{p}^{l}_{k},k=1,2,...,256$ since size of
the final feature maps is $16\times 16$. $\textbf{p}^{l}_{k}$ is transformed
using embedding functions $\theta$ and $f$ to calculate a weight matrix
$\textbf{W}^{l}_{k}$ by:

$\small\textbf{W}^{l}_{k}=softmax(\theta(\textbf{p}^{l}_{k})^{T}\otimes
f(\textbf{p}^{l}_{k})).$ (5)

The elements in $\textbf{W}^{l}_{k}$ indicate the relevance between local
features in $\textbf{p}^{l}_{k}$ and separate local features by assigning
different weights, given that local features of forged regions are more
relevant to that of forged regions and vice versa. $\textbf{p}^{l}_{k}$ is
also transformed by $g$ and then enhanced by:

$\small\hat{\textbf{p}}^{l}_{k}=\textbf{W}^{l}_{k}\otimes
g(\textbf{p}^{l}_{k}).$ (6)

As illustrated in Fig.4, LEM concats $\hat{\textbf{p}}^{l}_{k}$ to form the
enhanced feature maps $\hat{\textbf{F}}^{l}$ which is then multiplied by a
learnable scale parameter $\lambda$ and added with the input feature.

###  3.3 Second Order Local Anomaly Learning

In this work, we propose the Second Order Local Anomaly (SOLA) learning module
to detect face forgeries by fine-grained local anomalies. Let
$\textbf{F}_{i,j}\in\mathbb{R}^{C}$ denotes the $(i^{th},j^{th})$ feature
vector that corresponds to a local region in the input image. To obtain more
fine-grained representation of local anomalies, SOLA first decomposes the
neighborhood of $\textbf{F}_{i,j}$ into horizontal/vertical and nearest/next
nearest neighbors by direction and distance respectively. In practice, we only
predict the anomalies of upper and left neighbors due to the symmetry. Then
the first order anomaly maps of each $\textbf{F}_{i,j}$ are predicted by Eq.
7:

$\begin{cases}\mathcal{M}_{v1,(i,j)}=\delta_{v1}(\textbf{F}_{i,j},\textbf{F}_{i,j-1}),\\\
\mathcal{M}_{v2,(i,j)}=\delta_{v2}(\textbf{F}_{i,j},\textbf{F}_{i,j-2}),\\\
\mathcal{M}_{h1,(i,j)}=\delta_{h1}(\textbf{F}_{i,j},\textbf{F}_{i-1,j}),\\\
\mathcal{M}_{h2,(i,j)}=\delta_{h2}(\textbf{F}_{i,j},\textbf{F}_{i-2,j}),\end{cases}$
(7)

where $\delta(\cdot)$ is the anomaly predictor realized by $1\times 2$ or
$2\times 1$ convolutions (dilation = 2 for next nearest neighbors, otherwise
1), $v$ and $h$ denote vertical and horizontal neighbors, 1 and 2 denote the
nearest and next nearest neighbors. One kernel in $\delta(\cdot)$ can predict
only one anomaly map, thus we set each $\delta(\cdot)$ to contain 64 kernels
so that it can generate 64 anomaly maps to capture multiple anomalies.

Inspired by the idea of the Laplace operator that computes the second
derivative, we propose the second order local anomaly learning to obtain
magnified responses to the first order anomalies. To this end, SOLA applies
the local decomposition again on the first order anomaly maps and calculates
the second order local anomaly maps by Eq. 8. A pixel in the second order
anomaly maps is responded to a region in the first order anomaly maps so as to
capture a wider range of fine-grained anomalies.

$\begin{cases}\mathcal{M}^{\prime}_{v1,(i,j)}=\phi_{v1}(\mathcal{M}_{v1,(i,j)},\mathcal{M}_{v1,(i,j-1)}),\\\
\mathcal{M}^{\prime}_{v2,(i,j)}=\phi_{v2}(\mathcal{M}_{v2,(i,j)},\mathcal{M}_{v2,(i,j-2)}),\\\
\mathcal{M}^{\prime}_{h1,(i,j)}=\phi_{h1}(\mathcal{M}_{h1,(i,j)},\mathcal{M}_{h1,(i-1,j)}),\\\
\mathcal{M}^{\prime}_{h2,(i,j)}=\phi_{h2}(\mathcal{M}_{h2,(i,j)},\mathcal{M}_{h2,(i-2,j)}),\end{cases}$
(8)

$\phi(\cdot)$ denotes the second order anomaly predictor that is similar to
$\delta(\cdot)$ but contains only one single kernel. Finally, we stack 4
outputs of $\phi(\cdot)$ to form a 4 channel anomaly map and pass it to the
classifier. The classifier only contains a convolution layer followed by a
global average pooling layer and a fully-connected layer.

###  3.4 Loss Function

We first give the overall loss function for supervised training where the
forgery masks are available, and then describe the weakly supervised training
strategy using our single side loss.

Supervised Training. let $\textbf{M}\in\\{0,1\\}^{H\times W}$ denotes the
forgery mask, we divide it into patches $\textbf{m}_{i,j}$ (16 $\times$ 16
pixels in our case), we use its averaged value $AVG(\textbf{m}_{i,j})$ to
represent the forgery score of $(i^{th},j^{th})$ patch in the face image. We
use a _hard thresholding_ fashion to calculate the first and second order
anomaly scores of two adjacent patches as shown in (9a) and (9b) respectively.
Without loss of generality, we use $(i^{\prime},j^{\prime})$ to denote a
neighbor of $(i,j)$.

$\displaystyle
a_{i,j}=\begin{cases}1&\text{if}\phantom{se}|AVG(\textbf{m}_{i,j})-AVG(\textbf{m}_{i^{\prime},j^{\prime}})|>0,\\\
0&\text{else}.\end{cases}$ (9a) $\displaystyle
a^{\prime}_{i,j}=\begin{cases}1&\text{if}\phantom{se}|a_{i,j}-a_{i^{\prime},j^{\prime}}|>0,\\\
0&\text{else}.\end{cases}$ (9b)

By iterating (9a) and (9b) for each $\textbf{m}_{i,j}$, we obtain the ground
truth of the first order anomaly maps
$\tilde{\mathcal{M}}\in\\{0,1\\}^{h\times w\times 64}$ (repeated 64 times),
and the second order anomaly maps
$\tilde{\mathcal{M}}^{\prime}\in\\{0,1\\}^{h^{\prime}\times w^{\prime}}$. The
local anomaly maps can be optimized using pixel wise binary cross entropy
(BCE):

$\small\mathcal{L}_{A}=\beta\sum_{*}BCE(\mathcal{M}_{*},\tilde{\mathcal{M}}_{*})+\gamma\sum_{h}BCE(\mathcal{M}^{\prime}_{*},\tilde{\mathcal{M}}^{\prime}_{*}),$
(10)

where $*\in\\{v1,v2,h1,h2\\}$, $\beta$ and $\gamma$ are the weights for the
first and second order anomaly maps. The overall loss for training our model
is:

$\mathcal{L}_{total}=\alpha L_{cls}+L_{A},$ (11)

$L_{cls}$ is the BCE loss for the classifier and $\alpha$ is the weight.

Weakly Supervised Training. Although SOLA learning module predicts patch level
anomalies, we can train it in a weakly supervised manner where only image
level labels are available. To this end, we have two hypotheses: (1) The
pixels of anomaly maps of real faces should be all zero or close to zero. In
the meantime, there should be a portion of nonzero pixels in the anomaly maps
of forged faces. (2) Anomaly maps vary according to different forgery
algorithms, but they should be always nonzero. On these hypotheses, we
introduce a single side loss in Eq. 12 that only penalizes the anomaly maps of
real faces. This loss ensures multiple patterns of anomalies in forged faces,
thereby improving the generalization:

$\small\mathcal{L}_{A}=\sum_{I\in
real}\sum_{*}\underbrace{\beta||\mathcal{M}_{*}-\tilde{\mathcal{M}}_{*}||_{1}}_{\text{First
Order}}+\underbrace{\gamma||\mathcal{M}^{\prime}_{*}-\tilde{\mathcal{M}}^{\prime}_{*}||_{1}}_{\text{Second
Order}},$ (12)

where $\tilde{\mathcal{M}}$ and $\tilde{\mathcal{M}}^{\prime}$ are both all-
zero.

##  4 Experiments

###  4.1 Experiments details

Datasets. We use a wide range of popular face forgery datasets to evaluate the
proposed method, including FaceForensics++ (FF++) [36], Celeb-DF v2 (CD2)
[25], DeepfakeDetection Dataset (DFD) [31], and FaceShifter (Fshi) [21]. In
FF++, 1000 original videos are forged by four forgery algorithms: DeepFakes,
Face2Face [43], FaceSwap, and NeuralTextures [42]. All face patches are
cropped according to their masks if available, otherwise, we use RetinaFace
[9] to detect faces and crop the patches. To preserve enough background, the
cropped patches are set to be 2.6 times the size of masks or bounding box. All
face patches are resized to 256 $\times$ 256 and normalize to $[0,1]$ by
dividing 255.

Evaluation Metrics. We report frame level AUC (area under the receiver
operating characteristic curve) as most of the previous works do. The
experimental results of other methods which we use for comparison are directly
cited.

Implementation Details. All the experiments are implemented with Pytorch [32]
with 4$\times$NVIDIA RTX 3090 24GB. The backbone is initialized with ImageNet
pre-trained weights and trained using Adam optimizer [19] with learning rate
1e-3, betas 0.9 and 0.999, and epsilon 1e-8. The batch size is 32 and number
of epochs is 50 without early stopping.

Methods |  Backbone |  DF |  F2F |  FS |  NT
---|---|---|---|---|---
Xception [36] |  Xception |  99.38 |  99.53 |  99.36 |  99.50
PBD [39] |  Xception |  97.00 |  95.00 |  98.00 |  98.00
Face X-ray [22] |  HRNet |  99.17 |  99.06 |  99.20 |  98.93
S-MIL [23] |  Xception |  99.84 |  99.34 |  99.61 |  98.85
SOLA _-weakly sup_ |  ResNet18 |  100 |  99.67 |  100 |  99.82
SOLA _-sup_ |  ResNet18 |  100 |  99.56 |  99.98 |  99.76
Table 1: In-domain performences on FF++. Model |  Backbone |  Train Set | Test Set |  Avg
---|---|---|---|---
|  |  |  DF |  F2F |  FS |  NT |
Xception [36] |  Xception |  DF | 99.38 |  75.05 |  49.13 |  80.39 |  75.99
Face X-ray [22] |  HRNet |  | 99.17 |  94.14 |  75.34 |  93.85 |  90.63
SOLA _-weakly sup_ |  ResNet18 |  | 100 |  97.29 |  63.59 |  98.45 |  89.83
SOLA _-sup_ |  ResNet18 |  | 100 |  96.95 |  69.72 |  98.48 |  91.28
Xception [36] |  Xception |  F2F |  87.56 | 99.53 |  65.23 |  65.90 |  79.56
Face X-ray [22] |  HRNet |  |  98.52 | 99.06 |  72.69 |  91.49 |  90.44
SOLA _-weakly sup_ |  ResNet18 |  |  99.61 | 99.67 |  84.24 |  97.48 |  95.25
SOLA _-sup_ |  ResNet18 |  |  99.73 | 99.56 |  93.50 |  96.02 |  97.20
Xception [36] |  Xception |  FS |  70.12 |  61.70 | 99.36 |  68.71 |  74.97
Face X-ray [22] |  HRNet |  |  93.77 |  92.29 | 99.20 |  86.63 |  92.97
SOLA _-weakly sup_ |  ResNet18 |  |  93.18 |  97.59 | 100 |  94.93 |  96.43
SOLA _-sup_ |  ResNet18 |  |  99.11 |  98.13 | 99.98 |  92.07 |  97.32
Xception [36] |  Xception |  NT |  93.09 |  84.82 |  47.98 | 99.50 |  81.35
Face X-ray [22] |  HRNet |  |  99.14 |  98.43 |  70.56 | 98.93 |  91.77
SOLA _-weakly sup_ |  ResNet18 |  |  99.95 |  94.83 |  57.32 | 99.82 |  87.97
SOLA _-sup_ |  ResNet18 |  |  99.64 |  97.69 |  90.20 | 99.76 |  96.82
Table 2: Cross-domain evaluations on FF++.

###  4.2 In-Domain Evaluations

We first report the performance of our method under in-domain evaluations. The
results of FF++ are shown in Table 1. _-weakly sup_ and _-sup_ denote SOLA
trained with different strategies. Previous methods have achieved great
performance, but our method still surpasses the best competitors by about 1%
in terms of average AUC . Note that our method only uses a relatively small
backbone ResNet18 while most of other methods use a large backbone like
Xception. On the whole, weakly supervised SOLA performs slightly better,
proving that the single side loss allows SOLA to capture different anomalies
precisely.

###  4.3 Cross-domain Evaluations

In this section, we focus on the more challenging cross-domain evaluations to
explore the performances of our method on unseen forgeries. Table 2 shows the
results of cross-domain evaluations on FF++. Here, the models are trained on
only one dataset and evaluated on all four datasets. Although Xception
achieves 99.42% average AUC of in-domain evaluations, it can not generalize to
other forgeries well and its performances of cross-domain evaluations drop
sharply. Meanwhile, Face x-ray [22] is designed to detect the blending
boundaries instead of specifical artifacts caused by forgery algorithms and
made significant progress in generalization. We can see that weakly supervised
SOLA performs closely to Face x-ray. Face x-ray requires pixel level forgery
masks to locate the boundaries and train their model, but our method also
achieves a great improvement of generalization ability with only image level
labels. With the help of pixel level annotations, supervised SOLA surpasses
all other methods in terms of average AUC of cross-domain evaluations. The
overall performance of supervised SOLA on FF++ exceeds Xception and Face x-ray
by 17.69% and 4.21%.

Model |  Backbone |  Train Set |  Test Set
---|---|---|---
|  |  |  FShi
Xception [36] |  Xception |  FF++ |  72.00
PBD [39] |  Xception |  FF++ |  57.80
FWA [24] |  ResNet152 |  FF++ |  65.50
Face X-ray [22] |  HRNet |  FF++ |  92.80
LipForensics [14] |  ResNet18 |  FF++ |  97.10
SOLA _-weakly sup_ |  ResNet18 |  FF++ |  97.27
SOLA _-sup_ |  ResNet18 |  FF++ |  98.72
Table 3: Cross-domain evaluation results on FaceShifter (trained on FF++). Our method has better performance than state-of-the-art works and is also competitive while using only one dataset. Model |  Train Set |  Test Set
---|---|---
|  |  CD2
$F^{3}$ Net [33] |  FF++(c23) |  65.17
FWA [24] |  Self-made |  57.32
MADD [51] |  FF++(c23) |  67.44
MTD-Net [49] |  FF++(c23) |  70.12
Two Branch [28] |  FF++(c40) |  73.41
F3Net [20] |  FF++(c23) |  65.20
LRL [4] |  FF++(c23) |  78.26
GFF [27] |  FF++(c23) |  65.20
MADD [51] |  FF++(c23) |  67.44
SPSL [26] |  FF++(c23) |  76.88
LipFor [14] |  FF++(c23) |  82.40
SOLA _-weakly sup_ |  DF(c23) |  72.47
SOLA _-sup_ |  DF(c23) |  76.02
Table 4: Cross-domain evaluations on CD2. Our methods trained using only
DF-c23 outperforms most methods.

Results of cross-domain evaluations on a more advanced face forgery algorithm
FaceShifter (Fshi) are shown in Table 3. Following other works, we train our
model on FF++ and evaluate it by Fshi. Weakly supervised and supervised SOLA
both exceed state-of-the-art method, by 0.17% and 1.62% AUC respectively. In
addition to the evaluations with two supervision fashions, we also evaluate
weakly supervised SOLA trained with only one dataset. Although the
performances of weakly supervised SOLA are slightly lower than the supervised
SOLA when trained using only one dataset, they are generally still better than
most of the recent methods. These results demonstrate the local anomaly traces
are universal in the forged faces and can be generalized to unseen forgeries.

The comparisons of cross-domain evaluation on CD2 are given in Table 4. Note
that the compared methods are trained with different datasets, so we present
the results for reference only. Most methods use all four datasets of FF++ so
as to obtain good generalization on CD2. However, weakly supervised SOLA gets
72.47% AUC when trained using only the compressed DF dataset, exceeding most
state-of-the-art competitors. Supervised SOLA further improve the result to
76.02% which is 3.55% higher than the Two-Branch [28] that similarly fuses RGB
and frequency features.

Model |  Train Set | Test Set
---|---|---
|  |  DF |  F2F |  FS |  NT
Xception[36] |  CD2 |  87.69 |  75.17 |  54.19 |  72.89
SOLA _-weakly sup_ |  |  85.77 |  85.72 |  86.78 |  85.64
Xception[36] |  DFD |  94.79 |  76.96 |  47.13 |  84.91
SOLA _-weakly sup_ |  |  95.51 |  85.51 |  60.81 |  84.01
Table 5: Cross-domain evaluations on DF, F2F, FS, and NT by training on CD2
and DFD.

Table 5 shows the cross-domain results of our method trained on CD2 and DFD
and tested on FF++. Here, SOLA can be only trained without supervision given
that the ground truth forgery masks of both the datasets are not available.
The overall AUCs are 85.98% and 81.46% when trained on CD2 and DFD
respectively, exceeding the Xception by 13.49%, 5.57%, proving that our method
generalizes well on multiple datasets even with only image level annotations.

Figure 5: Grad-CAM [40] for the _forgery_ class of different of forged faces.
We can see the proposed SOLA can respond to various forgeries while the
vanilla ResNet18 can not.

To explore which region our method pays attention when encountering unseen
forgeries, we use Grad-CAM [40] to generate heat maps as illustrated in fig.
5. The warm color marks the regions that respond strongly to the prediction of
_forgery_. Here, a ResNet18 and our method are both trained on DF. Obviously,
they can well concentrate on the forged regions of faces created by DF and
have almost no response to real faces. However, our method not only captures
forged regions on DF more comprehensively, but also responds to unseen
forgeries while ResNet18 can not. We can see that ResNet18 fails to capture
the forged regions on F2F and NT, thus it responses the same to F2F and NT as
to the real face. This agrees with the weak performance of deep learning
models in cross-domain evaluations and reveals the generalization ability of
our method on unseen forgeries.

Figure 6: The outputs of different SRM variants over training epochs.

###  4.4 Ablation Studies

The effect of ASRM. So far, we have been training our method with the proposed
ASRM. In this section, we demonstrate its effectiveness by replacing it with
several variants. Table 6 shows the performances of different noise extraction
strategies in the noise branch: _-w/o noise_ denotes SOLA without noise
branch, _-srm_ and _-lsrm_ denote SOLA using fixed SRM filters and learnable
SRM filters (use filters as initialization of the first layer without any
constraint). All variants are trained on CD2 and DFD without pixel level
annotations. The results show that the noise branch effectively improves the
performance of SOLA on DFD, by 16.5%, 16.68%, and 16.67% with different SRM
variants.

Model |  CD2 |  DFD
---|---|---
SOLA _-w/o noise_ |  98.15 |  83.31
SOLA _-srm_ |  73.05 |  99.81
SOLA _-lsrm_ |  94.67 |  99.99
SOLA _-asrm_ |  98.70 |  99.98
Table 6: Comparsion results of different variants of SRM.

Fig. 6 shows outputs of all three channels of standard SRM, LSRM, and ASRM
across training epochs. The input face is the same as that in Fig. 3. Although
the standard SRM extracts high frequency noise, it does not focus on forged
regions. Meanwhile, the LSRM loses its high pass peculiarity and fails to
extract noise features in the training process. But we can see that ASRM
gradually extracts the high frequency noise of face regions, especially the
facial organs, providing discriminative cues that can not be learned from CNNs
directly.

The effect of LEM. To confirm the effect of LEM, we train the weakly
supervised SOLA with different combinations of ASRM and LEM. As shown in Table
7, the models are trained on DF and evaluated on DF, FShi, and DFD. Generally,
both ASRM and LEM can improve the overall performances while ASRM contributes
more to the generalization and increases the AUC on Fshi and DFD by 8.78% and
10.16%.

The effect of Patch Size. In the previous experiments, we use the 3 out of 4
blocks in ResNet18 and an additional pooling layer to obtain feaures of size
$16\times 16\times 256$. To further evaluate the effect of different patch
sizes, we change the pooling size to obtain features of size $8\times 8$ and
$32\times 32$. The results are shown in Table 7. Although SOLA with different
patch sizes all get good results, SOLA with patch size 16 achieves the optimal
performances across all datasets. These results accord with the conclusions in
[52] as well.

ASRM |  LEM |  Patch Size | Test Set
---|---|---|---
|  |  |  DF |  FShi |  DFD
|  |  16 |  99.90 |  72.24 |  78.33
$\surd$ |  |  16 |  99.87 |  81.02 |  88.49
|  $\surd$ |  16 |  99.98 |  73.64 |  80.25
$\surd$ |  $\surd$ |  8 |  100 |  80.17 |  91.10
$\surd$ |  $\surd$ |  16 |  100 |  86.53 |  92.61
$\surd$ |  $\surd$ |  32 |  100 |  85.99 |  87.85
Table 7: Performances of weakly supervised SOLA with different patch size and the effect of LEM. Backbone |  FF++(c0) |  FF++(c23) |  CD2
---|---|---|---
ResNet18 |  99.84 |  98.10 |  68.33
ResNet50 |  99.90 |  99.14 |  74.98
ResNet101 |  99.90 |  99.25 |  75.05
Table 8: Comparsion results of different backbones.

The effect of Backbones. We further test the effect of different backbone
models on both in-domain and cross domain settings. As in Table 8, we choose 2
other models in ResNet family and report the results on FF++ and CD2 (trained
on FF++(c23)). While the overall performances especially the cross-domain
performance are much better with ResNet50, the increase is very limited with a
deeper ResNet101.

###  4.5 Limitations

While the proposed method predicts fine-grained local anomalies of different
distances and directions, the diversity of different predictors in SOLA is not
explicitly ensured, especially under the weakly-sup setting. This could lead
to the degeneration of local anomaly predictors and weaken the representations
of local anomalies.

##  5 Conclusion

In this work, we revisit face forgery detection from the perspective of local
anomaly detection and propose the SOLA learning module to predict different
types of local anomalies in the first and second order. Besides, we design a
weakly supervised strategy to train SOLA without pixel level annotations. We
also introduce a adaptive spatial rich model to mine subtle high frequency
traces using learnable high pass kernels. Experiments on multiple datasets
demonstrate that our method achieves competitive performance with a small
backbone and generalize to unseen forgery types well.

## Acknowlegdement

This work is supported in part by the National Key Research and Development
Plan of China under Grant 2020YFB1005600, in part by the National Natural
Science Foundation of China under grant numbers 62122032, and U1936118, in
part by Qinglan Project of Jiangsu Province, and “333” project of Jiangsu
Province.

## References

  * [1] Shruti Agarwal, Hany Farid, Ohad Fried, and Maneesh Agrawala.  Detecting deep-fake videos from phoneme-viseme mismatches.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 660–661, 2020.
  * [2] Belhassen Bayar and Matthew C Stamm.  Constrained convolutional neural networks: A new approach towards general purpose image manipulation detection.  IEEE Transactions on Information Forensics and Security, 13(11):2691–2706, 2018.
  * [3] Peng Chen, Jin Liu, Tao Liang, Guangzhi Zhou, Hongchao Gao, Jiao Dai, and Jizhong Han.  Fsspotter: Spotting face-swapped video by spatial and temporal clues.  In 2020 IEEE international conference on multimedia and expo (ICME), pages 1–6. IEEE, 2020.
  * [4] Shen Chen, Taiping Yao, Yang Chen, Shouhong Ding, Jilin Li, and Rongrong Ji.  Local relation learning for face forgery detection.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1081–1088, 2021.
  * [5] Komal Chugh, Parul Gupta, Abhinav Dhall, and Ramanathan Subramanian.  Not made for each other-audio-visual dissonance-based deepfake detection and localization.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 439–447, 2020.
  * [6] Davide Cozzolino and Luisa Verdoliva.  Single-image splicing localization through autoencoder-based anomaly detection.  In 2016 IEEE international workshop on information forensics and security (WIFS), pages 1–6. IEEE, 2016.
  * [7] Davide Cozzolino and Luisa Verdoliva.  Noiseprint: A cnn-based camera model fingerprint.  IEEE Transactions on Information Forensics and Security, 15:144–159, 2019.
  * [8] Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil K Jain.  On the detection of digital face manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition, pages 5781–5790, 2020.
  * [9] Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou.  Retinaface: Single-shot multi-level face localisation in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5203–5212, 2020.
  * [10] Ricard Durall, Margret Keuper, and Janis Keuper.  Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7890–7899, 2020.
  * [11] Soroush Fatemifar, Shervin Rahimzadeh Arashloo, Muhammad Awais, and Josef Kittler.  Client-specific anomaly detection for face presentation attack detection.  Pattern Recognition, 112:107696, 2021.
  * [12] Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz.  Leveraging frequency analysis for deep fake image recognition.  In International Conference on Machine Learning, pages 3247–3258. PMLR, 2020.
  * [13] Jessica Fridrich and Jan Kodovsky.  Rich models for steganalysis of digital images.  IEEE Transactions on Information Forensics and Security, 7(3):868–882, 2012.
  * [14] Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic.  Lips don’t lie: A generalisable and robust approach to face forgery detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5039–5049, 2021.
  * [15] Bing Han, Xiaoguang Han, Hua Zhang, Jingzhi Li, and Xiaochun Cao.  Fighting fake news: Two stream network for deepfake detection via learnable srm.  IEEE Transactions on Biometrics, Behavior, and Identity Science, 2021.
  * [16] Ziheng Hu, Hongtao Xie, YuXin Wang, Jiahong Li, Zhongyuan Wang, and Yongdong Zhang.  Dynamic inconsistency-aware deepfake video detection.  In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 736–742, 8 2021.
  * [17] Gengyun Jia, Meisong Zheng, Chuanrui Hu, Xin Ma, Yuting Xu, Luoqi Liu, Yafeng Deng, and Ran He.  Inconsistency-aware wavelet dual-branch network for face forgery detection.  IEEE Transactions on Biometrics, Behavior, and Identity Science, 2021.
  * [18] Hasam Khalid and Simon S Woo.  Oc-fakedect: Classifying deepfakes using one-class variational autoencoder.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 656–657, 2020.
  * [19] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [20] Jiaming Li, Hongtao Xie, Jiahong Li, Zhongyuan Wang, and Yongdong Zhang.  Frequency-aware discriminative feature learning supervised by single-center loss for face forgery detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6458–6467, 2021.
  * [21] Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, and Fang Wen.  Advancing high fidelity identity swapping for forgery detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5074–5083, 2020.
  * [22] Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo.  Face x-ray for more general face forgery detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5001–5010, 2020.
  * [23] Xiaodan Li, Yining Lang, Yuefeng Chen, Xiaofeng Mao, Yuan He, Shuhui Wang, Hui Xue, and Quan Lu.  Sharp multiple instance learning for deepfake video detection.  In Proceedings of the 28th ACM international conference on multimedia, pages 1864–1872, 2020.
  * [24] Yuezun Li and Siwei Lyu.  Exposing deepfake videos by detecting face warping artifacts.  In CVPR Workshops, 2019.
  * [25] Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu.  Celeb-df: A large-scale challenging dataset for deepfake forensics.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3207–3216, 2020.
  * [26] Honggu Liu, Xiaodan Li, Wenbo Zhou, Yuefeng Chen, Yuan He, Hui Xue, Weiming Zhang, and Nenghai Yu.  Spatial-phase shallow learning: rethinking face forgery detection in frequency domain.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 772–781, 2021.
  * [27] Yuchen Luo, Yong Zhang, Junchi Yan, and Wei Liu.  Generalizing face forgery detection with high-frequency features.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16317–16326, 2021.
  * [28] Iacopo Masi, Aditya Killekar, Royston Marian Mascarenhas, Shenoy Pratik Gurudatt, and Wael AbdAlmageed.  Two-branch recurrent network for isolating deepfakes in videos.  In European Conference on Computer Vision, pages 667–684. Springer, 2020.
  * [29] Falko Matern, Christian Riess, and Marc Stamminger.  Exploiting visual artifacts to expose deepfakes and face manipulations.  In 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), pages 83–92. IEEE, 2019.
  * [30] Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha.  Emotions don’t lie: An audio-visual deepfake detection method using affective cues.  In Proceedings of the 28th ACM international conference on multimedia, pages 2823–2832, 2020.
  * [31] Andrew Gully Nick Dufour.  Deepfakedetection dataset.  https://ai.googleblog.com/2019/09/contributing-data-to-deepfake-detection.html.
  * [32] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32:8026–8037, 2019.
  * [33] Yuyang Qian, Guojun Yin, Lu Sheng, Zixuan Chen, and Jing Shao.  Thinking in frequency: Face forgery detection by mining frequency-aware clues.  In European Conference on Computer Vision, pages 86–103. Springer, 2020.
  * [34] Yuan Rao and Jiangqun Ni.  A deep learning approach to detection of splicing and copy-move forgeries in images.  In 2016 IEEE International Workshop on Information Forensics and Security (WIFS), pages 1–6. IEEE, 2016.
  * [35] Yuan Rao and Jiangqun Ni.  Self-supervised domain adaptation for forgery localization of jpeg compressed images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15034–15043, 2021.
  * [36] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner.  Faceforensics++: Learning to detect manipulated facial images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1–11, 2019.
  * [37] Yiwei Ru, Wanting Zhou, Yunfan Liu, Jianxin Sun, and Qi Li.  Bita-net: Bi-temporal attention network for facial video forgery detection.  In 2021 IEEE International Joint Conference on Biometrics (IJCB), pages 1–8. IEEE, 2021.
  * [38] Ekraam Sabir, Jiaxin Cheng, Ayush Jaiswal, Wael AbdAlmageed, Iacopo Masi, and Prem Natarajan.  Recurrent convolutional strategies for face manipulation detection in videos.  In CVPR Workshops, 2019.
  * [39] Steven Schwarcz and Rama Chellappa.  Finding facial forgery artifacts with parts-based detectors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 933–942, 2021.
  * [40] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * [41] Ke Sun, Hong Liu, Qixiang Ye, Jianzhuang Liu, Yue Gao, Ling Shao, and Rongrong Ji.  Domain general face forgery detection by learning to weight.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2638–2646, 2021.
  * [42] Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Deferred neural rendering: Image synthesis using neural textures.  ACM Transactions on Graphics (TOG), 38(4):1–12, 2019.
  * [43] Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner.  Face2face: Real-time face capture and reenactment of rgb videos.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2387–2395, 2016.
  * [44] Davide Cozzolino Giovanni Poggi Luisa Verdoliva.  Extracting camera-based fingerprints for video forensics.  Proc. of CVPRW, 2019.
  * [45] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803, 2018.
  * [46] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * [47] Xi Wu, Zhen Xie, YuTao Gao, and Yu Xiao.  Sstnet: Detecting manipulated faces through spatial, steganalysis and temporal features.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2952–2956. IEEE, 2020.
  * [48] Yue Wu, Wael AbdAlmageed, and Premkumar Natarajan.  Mantra-net: Manipulation tracing network for detection and localization of image forgeries with anomalous features.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9543–9552, 2019.
  * [49] Jiachen Yang, Aiyun Li, Shuai Xiao, Wen Lu, and Xinbo Gao.  Mtd-net: Learning to detect deepfakes images by multi-scale texture difference.  IEEE Transactions on Information Forensics and Security, 16:4234–4245, 2021.
  * [50] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.  Self-attention generative adversarial networks.  In International conference on machine learning, pages 7354–7363. PMLR, 2019.
  * [51] Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu.  Multi-attentional deepfake detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2185–2194, 2021.
  * [52] Tianchen Zhao, Xiang Xu, Mingze Xu, Hui Ding, Yuanjun Xiong, and Wei Xia.  Learning self-consistency for deepfake detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15023–15033, 2021.
  * [53] Peng Zhou, Xintong Han, Vlad I Morariu, and Larry S Davis.  Learning rich features for image manipulation detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1053–1061, 2018.


# Setting up the Data Printer with Improved English to Ukrainian Machine
Translation

###### Abstract

To build large language models for Ukrainian we need to expand our corpora
with large amounts of new algorithmic tasks expressed in natural language.
Examples of task performance expressed in English are abundant, so with a
high-quality translation system our community will be enabled to curate
datasets faster. To aid this goal, we introduce a recipe to build a
translation system using supervised finetuning of a large pretrained language
model with a noisy parallel dataset of 3M pairs of Ukrainian and English
sentences followed by a second phase of training using 17K examples selected
by k-fold perplexity filtering on another dataset of higher quality. Our
decoder-only model named Dragoman beats performance of previous state of the
art encoder-decoder models on the FLORES devtest set.

Keywords: machine translation, parameter-efficient fine tuning, large language
models, unsupervised data selection, perplexity filtering

Setting up the Data Printer with Improved English to Ukrainian Machine
Translation

Yurii Paniv, Dmytro Chaplynskyi, Nikita Trynus, Volodymyr Kyrylov
---
Ukrainian Catholic University, lang-uk initiative,
Igor Sikorsky Kyiv Polytechnic Institute, Università della Svizzera italiana
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

The availability of the data is the most important ingredient when one needs
to pretrain general-purpose large language models for a specific natural
language task or a set of tasks. While it is relatively easy to obtain a good
and balanced dataset under specific domain for the English language, it is
much harder to do the same for other under-resourced languages such as
Ukrainian.

Since curating a corpus of tasks in Ukrainian is a large endeavor, and given a
large body of work done for English, we consider existing instruction tuning
datasets as a source of tasks to reuse in Ukrainian using automatic machine
translation.

This work focuses on improving the current state of machine translation from
English to Ukrainian.

We contribute a recipe for finetuning a large pretrained language model with
publicly available data to build a translation system (section 3, section 4).
This matches state of the art performance of the best encoder-decoder model on
a common multilingual benchmark using a consumer GPU with 24 GiB of VRAM. We
release training, evaluation code, datasets, and model at
https://github.com/lang-uk/dragoman. Our main results are summarized in Table
1. We provide examples of the top-5 best and worst translations on the FLORES
devtest set in the Appendix A.

We base pretrained model selection on evaluation in few-shot learning setting
(section 5). We find that it’s a promising method to design tasks without
training, and the model can perform comparably to specialized systems given
increased inference budget and auxiliary translation scoring functions, yet
still underperforms our finetuned recipe.

Model |  BLEU $\uparrow$
---|---
Finetuned |
Dragoman P, 10 beams (section 3) | 30.4
Dragoman PT, 10 beams (section 4) | 32.3
Zero shot and few shot (section 5) |
Llama 2 7B 2-shot, 10 beams | 20.1
Mistral-7B-v0.1 2-shot, 10 beams | 24.9
gpt-4 10-shot | 29.5
gpt-4-turbo-preview 0-shot | 30.4
Pretrained encoder-decoder |
NLLB-3B, 10 beams | 30.6
OPUS-MT, 10 beams | 32.2
Table 1: Main results. Our Dragoman models improve existing state of the art
on translation from English to Ukrainian on FLORES-101 devtest (Goyal et al.,
2022), a multilingual benchmark of translated sentences from web articles. We
compare to state of the art encoder-decoder models, NLLB-3B Team et al. (2022)
and OPUS-MT Tiedemann and Thottingal (2020).

## 2\. Supervised Finetuning

Dataset | Pairs | Filters | Example Order | Best BLEU $\uparrow$
---|---|---|---|---
Lang | BPC | LaBSE | Len diff
1m unfiltered | 963k | - | - | - | - | Random | 28.26
1m filtered | 958k | En/Uk | <3.33 | >0.91 | <50 | Random | 29.47
3m filtered | 2.9m | En/Uk | <3.25 | >0.85 | <50 | By LaBSE score, dissimilar first | 30.37
8m filtered | 8m | En/Uk | <5 | >0.5 | <50 | By LaBSE score, dissimilar first | 30.19

Table 2:  Summary of experiments with Paracrawl subcorpora. Legend of filters:
Lang denotes language filters, BPC denotes maximum sum of bits per character
measures, LaBSE denotes maximum sentence embedding cosine similarity between
source and target sentences, Len diff denotes maximum difference in length
between source and target in characters. Example ordering impacts data loading
in the training loop.

We cast machine translation as a likelihood maximization of a density
$\operatorname{p}$ of Ukrainian sentences $Y="\text{{перекладене
речення}}"\in\mathcal{Y}$ conditioned on their English sources with quasi-
instruction formatting: $X="\text{[INST] translated sentence
[/INST]}"\in\mathcal{X}$.

The density is parametrized using a neural network with frozen pretrained
weights $\theta$:

$\operatorname{argmax}_{\phi}\operatorname{p}_{\theta,\phi}(Y|X)$ (1)

We implement the conditional language modeling objective by masking out tokens
of $X$ when computing token-wise cross entropy of shifted targets. We only
optimize extra low rank adapter (Hu et al., 2022) parameters $\phi$ after nf4
quantization (Dettmers et al., 2023). In practice we use large rank values and
adapter mixture weights. All training runs proceed for one epoch and we use
dropout (Srivastava et al., 2014) for regularization against data noise.

We use Mistral-7B-v0.1 (Jiang et al., 2023) as a base pretrained decoder-only
transformer, as it performs favorably in our few-shot experiments (section 5).

## 3\. First Phase: Heuristic Filtering of Paracrawl

We use the publicly available Paracrawl dataset Bañón et al. (2020). This
dataset contains 13,354,365 English-Ukrainian sentence pairs, collected by
automatically matching similar sentences in large corpora of internet text.

We have identified issues with translation pairs, including a significant
number of repetitive or incorrect examples. We encounter a large subset of
repetitive weather forecasts following the template “The temperature in <x> is
<y> degrees,” and sentences from site navigation menus. Additionally, many
texts appear to be scraped from adult websites, containing low-quality,
machine-translated samples. We have spotted numerous instances of incomplete
or significantly incorrect translation pairs. Some target sentences were
written in languages other than Ukrainian.

To control the quality of the sentences, we apply multiple heuristics.

### Language filtering

gcld3 library111https://github.com/google/cld3 provides language detection
capabilities. We remove all sentences that failed to verify as Ukrainian or
English.

### Perplexity thresholding

We score source and target sentences using two decoder-only models trained on
different monolingual datasets Radford et al. (2019); Minixhofer et al. (2022)
and sum their bits per character measures.

### Translation mismatch filtering

LaBSE Feng et al. (2022) embeds sentences into a space, where similar
sentences in different languages are close together. We use it to filter out
badly aligned sentence pairs.

### Length filtering

The lengths of the original and translated sentences reveal examples that are
too short or too long. Absolute differences of lengths point to pairs with
long target for the short source and vice versa.

We arbitrarily choose joint values of filtering thresholds to get the desired
approximate example counts: 1 million, 3 million and 8 million. We perform
multiple experiments with these splits while searching for optimal
hyperparameters. We list threshold values in Table 2 and best results for each
subset.

## 4\. Second Phase: Unsupervised Data Selection on Extended Multi30K

We use the best checkpoint from the previous finetuning phase to train on a
high-quality dataset: Extended Multi30K from Saichyshyna et al. (2023).
Switching datasets gives us a performance boost of 1.97 BLEU. We additionally
delete 11600 sentences from the dataset using unsupervised perplexity
filtering pipeline gaining 0.35 on the dev set that translates to 0.3 BLEU on
the devtest subset of FLORES.

Figure 1: Distributions of sentence log probabilities for each fold
superimposed on top of each other. Every bar color represents a unique fold;
every vertical line denotes a 60th percentile cutoff threshold. The best
percentile is chosen using grid search shown in Table 3.

We use perplexity as a data selection criterion to calculate thresholds to
filter out highly surprising sentences. We apply the $k$-fold cross-validation
technique to make the perplexity evaluation in-domain. We split the training
data into $k=5$ folds and train $k$ models withholding one of the folds from
each run. Then we score every sentence using the model that has not seen that
sentence in training. Next, we sweep for acceptable threshold values by
minimizing BLEU on the development set and report results in Table 3. We plot
the distribution of scores in Figure 1. We also provide threshold sweep
results for training from base Mistral-7B-v0.1 checkpoint in Table 6. By
comparing finetuned results, we demonstrate that data from the second phase
alone is not enough to match the performance of our best checkpoint.

Threshold | Examples |  BLEU $\uparrow$ |
---|---|---|---
percentile |  | dev | devtest
20th | 5800 | 31.57 | 32.06
40th | 11600 | 31.65 | 32.16
50th | 14500 | 31.76 | 32.36
60th | 17400 | 31.80 | 32.34
70th | 20300 | 31.51 | 32.17
80th | 23200 | 31.44 | 32.46
95.4th ($2\sigma$) | 28025 | 31.74 | 32.18
Full dataset | 29000 | 31.45 | 32.04
Table 3: Extended Multi30K log probability thresholds swept on FLORES dev set.
We choose the best checkpoint based on model performance on FLORES dev subset
using grid search for optimal perplexity threshold value.

## 5\. Few-Shot Translation

Conditioning the model on a sequence of demonstrations of performing some task
allows the model to learn this task in-context, also known as “few-shot
learning” (e.g. Brown et al. (2020)), thanks to the ability of the
Transformers to modulate representations of its future tokens using past
context, implementing a specialized internal context-dependent learning
algorithm inside its weights (von Oswald et al., 2023).

While few-shot learning allows to quickly try any task with a low number of
demonstrations, Liu et al. (2022) have shown that parameter-efficient
finetuning allows smaller models achieve better performance, effectively
spending less floating point operations per test example at inference time.

Setting up the model for finetuning requires a lot of work, and in-context
learning allows to quickly probe capability of a large model using inference
software that performs efficient management of key-value cache for speed Kwon
et al. (2023).

To test backbone models before finetuning, we attempt decoding translations
with a basic prompt shown in Figure 2.

[INST] They are planning to host a party next weekend. [/INST] Вони планують
провести вечірку наступного вікенду.
[INST] I enjoy swimming in the ocean and feeling the salty breeze. [/INST]
Мені подобається плавати в океані та відчувати солоний вітер.
[INST]

Figure 2: Basic 2-shot prompt used for few-shot translation. [INST] prefixes
the beginning of the source sentence and [/INST] denotes the beginning of the
target translation. These separators are chosen arbitrarily (as in finetuning)
and are not special vocabulary items, even though they bear visual resemblance
to them.

We find that the model significantly underperforms compared to current state
of the art translation models when using beam search (Tillmann and Ney, 2003).

This decoding algorithm performs pruned breadth-first expansion, scoring
target sentence prefixes using model’s own log probability, approximating
maximum a-posteriori estimation of the best translation.

Inspection of the n-best list of translation candidates (beams) reveals that
the models can produce high-quality translations, however assign low
probabilities to them. We find the best possible translation by rescoring
beams using the BLEU score as a loss function (Kumar and Byrne, 2004) with
respect to the reference translation (the so-called “oracle”).

We employ this oracle rescoring strategy to gauge the potential capability of
the model to produce good translations without finetuning, and find that in a
regime of increased computation (large width of the beam) and assuming perfect
selection capability, a base model is competitive with specialized
alternatives. We sweep over a grid of multiple beam widths and report highest
attainable BLEU scores in Table 4.

Beams | Oracle BLEU $\uparrow$ |
---|---|---
| Mistral-7B-v.01 | Llama 2 7B
3 | 27.11 | 24.55
5 | 29.20 | 26.64
10 | 31.53 | 28.76
15 | 32.81 | 29.09
20 | 33.54 | 27.64
25 | 34.27 | 26.35
30 | 33.99 | (decoder failure)
35 | 34.94 |
40 | 34.61 |
Table 4: We establish the upper bound of the latent capability of pretrained
base models to produce high quality translations with by varying beam width on
the task of translating sentences from FLORES dev given a 2-shot prompt. The
ground truth oracle determines the best beam. We use beam search
implementation by Kwon et al. (2023) with presence penalty of 0.1. The results
do not improve monotonically with increasing beam size, and lengths of
hypotheses grow with maximum beam size, yielding diminishing returns. This
problem can be attributed to label bias Murray and Chiang (2018), and
rectifying it will require extra regularization.

Consecutive sentences in FLORES are samples from the same document. We
hypothesize, dynamically adjusting the prompt by inserting previous
translations will improve results. We observe that the model indeed improves
translation of certain words such as proper nouns through access to correct
definitions provided in the context (Figure 3), however its overall
performance degrades in other examples.

Source: RSPCA New South Wales chief inspector David O’Shannessy told the ABC
that surveillance and inspections of abattoirs should be commonplace in
Australia.

Hypothesis given random 2-shot context: Головний інспектор РСПКА Нового
Південного Уельсу Девід О’Шеннесі повідомив ABC, що спостереження та інспекції
аббатств повинні бути звичайним явищем в Австралії.

Context example: [INST] Animal Liberation and the Royal Society for the
Prevention of Cruelty to Animals (RSPCA) are again calling for the mandatory
installation of CCTV cameras in all Australian abattoirs. [/INST] Організація
Звільнення тварин і Королівське товариство із запобігання жорстокому
поводженню з тваринами (КТЗЖПТ) знову закликають до обов’язкової установки
камер спостереження на всіх австралійських бійнях.

Hypothesis given relevant 2-shot context: Головний інспектор Королівського
товариства із запобігання жорстокому поводженню з тваринами (КТЗЖПТ) Нового
Південного Уельсу Девід О’Шеннесі заявив, що спостереження та інспекції бійні
повинні бути поширеними в Австралії.

Figure 3: Few-shot translation with contextual prompting allows the model to
learn named entities on the fly. Without context, the model makes a wrong
guess trying to transliterate the abbreviation.

We additionally attempt basic 0-shot with a system prompt You translate
English sentences into native Ukrainian., and 10-shot prompting using
automatic prompt selection based on similarity between source sentences
experiments with GPT-4 and GPT-4 Turbo and find that commerical systems
perform similarly to other open source systems, as shown in Table 1.

## 6\. Discussion and Limitations

### Single-sentence translation

Our system is trained on demonstrations of standalone sentence pairs.

### Decoder-only models with long context windows

We choose to finetune existing decoder-only models since the choice of models
with almost the same architecture but different massive pretraining data is
abundant. The number of open-source models released recently and their
constant improvement offers a good prospective for the machine translation
tasks.

These models receive gradient from all outputs during pretraining, and the
self-attention mechanism can see the input, the partial output, and access
past examples of translations in its context window using induction heads
(Olsson et al., 2022).

For efficiency, we only train on examples with single short sentence pairs and
do not pack context windows full of tokens as done in pretraining. In our
early experiments, we find that our models still generalize to inputs longer
that what is seen in training. This generalization behavior is often
attributed to relative position embeddings Dai et al. (2019); Csordás et al.
(2021). We leave evaluation of long context attention stability under these
conditions for future work.

### Training on the noisy dataset

Data cleaning has a positive effect on the resulting metrics. However, our
models trained on 8 million filtered, examples perform worse than models
trained on 3 million examples (Table 2).

### Tokenizer performance

We used the LLaMA and Mistral tokenizers during our experiments, which use at
least twice as many tokens to compress a sentence in Ukrainian of the same
length as an English sentence in character. In practice, that means that
generating a sentence in Ukrainian takes at least twice as many steps to
generate. We show a distribution of sentence token lengths in Figure 4.

Figure 4: Comparison of tokenizer compression rates between English and
Ukrainian using the Mistral-7B tokenizer on the FLORES dev set.

### Evaluation

We choose BLEU-4 score Papineni et al. (2002) as our core evaluation metric
and model selection criterion. BLEU-4 measures 4-gram precisions, where grams
are defined as words. We use the implementation and rely on tokenization
decisions of Post (2018). This metric is sensitive to minor differences that
do not affect the meaning of the sentence, for example case inflections that
tend to cascade to multiple adjacent words. BLEU is known to poorly correlate
with human judgement of translation quality, and Freitag et al. (2022)
recommend learned metrics.

Choosing an appropriate learned metric for judgements of translation quality
of Ukrainian requires careful consideration, and incorporation of data
informed by the language community, such as a curated corpus of grammar
corrections that reflects proper modern use of language Syvokon et al. (2023).

Regardless of limitations of BLEU, improvement in BLEU still signals
improvement in translation quality in our regime.

### WMT22

Our reviewers have pointed out that WMT22 benchmark Kocmi et al. (2022)
includes a test set for Ukrainian. Our model achieves 24.72 on the WMT22 test
set without any postprocessing, ranking behind the best result of Roussis and
Papavassiliou (2022) at 25.2 BLEU. We note that the submission that scores
relatively low on the WMT22 test, scores comparably to our results on FLORES.
These data distribution properties require closer exploration.

## 7\. Related Work

### Translation to Ukrainian

Maksymenko et al. (2023a, b) explore translation controllability by
conditioning the model on text embeddings that encode style by finetuning an
encoder-decoder model. They claim high quality translations on a private test
set.

### Instruction-tuned language models

Üstün et al. (2024) explore large-scale translation efforts to produce a
multilingual instruction-tuned language model Aya. This work translates large
datasets like the Flan Collection (Longpre et al., 2023) using the NLLB-3B
model (Team et al., 2022).

### Translation systems

Han et al. (2021) provide an iterated backtranslation recipe to bootstrap
neural machine translation systems using generative models: zero-shot
translation ability is used to produce candidates for few-shot demonstrations.
Filtered few-shot demonstrations are used to sample new sentences for further
finetuning for translation in two directions.

### Translation benchmarks

Besides FLORES-101 (Goyal et al. (2022), or FLORES-200 (Team et al., 2022),
both include the same data for Ukrainian) dataset used in this work, Tiedemann
(2020) provides an additional dataset for multilingual evaluation.

### Data selection techniques

Yang and Li (2023) propose a perplexity filtering pipeline, in which the data
is split into k folds to classify low quality augmentation generations
produced by surrogate language models. Sachdeva et al. (2024) provide recipes
on curating data for language models by directly asking language models to
score examples.

## 8\. Conclusion

In this work, we build a translation system using a two-phase data cleaning
pipeline. We demonstrate matching performance to state-of-the-art encoder-
decoder models for English-Ukrainian translation task. Notably, our system
exhibits superior performance compared to the NLLB model, which was
instrumental in generating the Aya dataset and contributed significantly to
the advancement of multilingual language models. Improved machine translation
could bring new capabilities to the next generation of large language models
trained for the Ukrainian language. The recent improvements made for decoder-
only backbones and the general dynamics of this process encourages us: we
firmly believe that recipes we propose in this paper can be used to improve
the quality of the translation by simply upgrading the backbone model.

## 9\. Contributions

Yurii Paniv worked on unsupervised data selection on extended Multi30K
dataset, Dmytro Chaplynskyi performed initial training using heuristic
filtering of Paracrawl, Nikita Trynus worked on evaluation, Volodymyr Kyrylov
designed few-shot learning experiments and evaluation.

## 10\. Acknowledgements

The authors would like to thank Talents for Ukraine project of Kyiv School of
Economics for the grant on compute, used for this paper, Ukrainian Catholic
University, Roman Kyslyi, and Oleksii Turuta for the fruitful discussions and
inspiration. We would also like to thank paper reviewers for their comments.

## 11\. References

##

  * Bañón et al. (2020) Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020.  ParaCrawl: Web-scale acquisition of parallel corpora.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4555–4567, Online. Association for Computational Linguistics.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Csordás et al. (2021) Róbert Csordás, Kazuki Irie, and Juergen Schmidhuber. 2021.  The devil is in the detail: Simple tricks improve systematic generalization of transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 619–634, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019.  Transformer-XL: Attentive language models beyond a fixed-length context.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2978–2988, Florence, Italy. Association for Computational Linguistics.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  Qlora: Efficient finetuning of quantized llms.  In _Advances in Neural Information Processing Systems_ , volume 36, pages 10088–10115. Curran Associates, Inc.
  * Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022.  Language-agnostic bert sentence embedding.
  * Freitag et al. (2022) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022.  Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Goyal et al. (2022) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022.  The Flores-101 evaluation benchmark for low-resource and multilingual machine translation.  _Transactions of the Association for Computational Linguistics_ , 10:522–538.
  * Han et al. (2021) Jesse Michael Han, Igor Babuschkin, Harrison Edwards, Arvind Neelakantan, Tao Xu, Stanislas Polu, Alex Ray, Pranav Shyam, Aditya Ramesh, Alec Radford, and Ilya Sutskever. 2021.  Unsupervised neural machine translation with generative language models only.  _ArXiv_ , abs/2110.05448.
  * Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.
  * Kocmi et al. (2022) Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022.  Findings of the 2022 conference on machine translation (WMT22).  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 1–45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Kumar and Byrne (2004) Shankar Kumar and William Byrne. 2004.  Minimum Bayes-risk decoding for statistical machine translation.  In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004_ , pages 169–176, Boston, Massachusetts, USA. Association for Computational Linguistics.
  * Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023.  Efficient memory management for large language model serving with PagedAttention.  In _Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles_.
  * Liu et al. (2022) Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023.  The Flan Collection: Designing Data and Methods for Effective Instruction Tuning.  _arXiv preprint arXiv:2301.13688_.
  * Maksymenko et al. (2023a) Daniil Maksymenko, Nataliia Saichyshyna, Oleksii Turuta, Marcin Paprzycki, Maria Ganzha, and Mirela Alhasani. 2023a.  Controllability for english-ukrainian machine translation by using style transfer techniques.  _2023 18th Conference on Computer Science and Intelligence Systems (FedCSIS)_ , pages 1059–1068.
  * Maksymenko et al. (2023b) Daniil Maksymenko, Olena Turuta, Nataliia Saichyshyna, Maksym Yerokhin, and Oleksii Turuta. 2023b.  Controllability for English-Ukrainian machine translation based on specialized corpora.  In _Proceedings of the 1st International Workshop on Multilingual, Multimodal and Multitask Language Generation_ , pages 1–9, Tampere, Finland. European Association for Machine Translation.
  * Minixhofer et al. (2022) Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022.  WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3992–4006, Seattle, United States. Association for Computational Linguistics.
  * Murray and Chiang (2018) Kenton Murray and David Chiang. 2018.  Correcting length bias in neural machine translation.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 212–223, Brussels, Belgium. Association for Computational Linguistics.
  * Olsson et al. (2022) Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2022.  In-context learning and induction heads.  _Transformer Circuits Thread_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Belgium, Brussels. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Roussis and Papavassiliou (2022) Dimitrios Roussis and Vassilis Papavassiliou. 2022.  The ARC-NKUA submission for the English-Ukrainian general machine translation shared task at WMT22.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 358–365, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Sachdeva et al. (2024) Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H. Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. 2024.  How to train data-efficient llms.
  * Saichyshyna et al. (2023) Nataliia Saichyshyna, Daniil Maksymenko, Oleksii Turuta, Andriy Yerokhin, Andrii Babii, and Olena Turuta. 2023.  Extension Multi30K: Multimodal dataset for integrated vision and language research in Ukrainian.  In _Proceedings of the Second Ukrainian Natural Language Processing Workshop (UNLP)_ , pages 54–61, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958.
  * Syvokon et al. (2023) Oleksiy Syvokon, Olena Nahorna, Pavlo Kuchmiichuk, and Nastasiia Osidach. 2023.  UA-GEC: Grammatical error correction and fluency corpus for the Ukrainian language.  In _Proceedings of the Second Ukrainian Natural Language Processing Workshop (UNLP)_ , pages 96–102, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Team et al. (2022) NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022.  No language left behind: Scaling human-centered machine translation.
  * Tiedemann (2020) Jörg Tiedemann. 2020.  The tatoeba translation challenge – realistic data sets for low resource and multilingual MT.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 1174–1182, Online. Association for Computational Linguistics.
  * Tiedemann and Thottingal (2020) Jörg Tiedemann and Santhosh Thottingal. 2020.  OPUS-MT – building open translation services for the world.  In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation_ , pages 479–480, Lisboa, Portugal. European Association for Machine Translation.
  * Tillmann and Ney (2003) Christoph Tillmann and Hermann Ney. 2003.  Word Reordering and a Dynamic Programming Beam Search Algorithm for Statistical Machine Translation.  _Computational Linguistics_ , 29(1):97–133.
  * von Oswald et al. (2023) Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. 2023.  Transformers learn in-context by gradient descent.
  * Yang and Li (2023) Heng Yang and Ke Li. 2023.  Boosting text augmentation via hybrid instance filtering framework.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 1652–1669, Toronto, Canada. Association for Computational Linguistics.
  * Üstün et al. (2024) Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024.  Aya model: An instruction finetuned open-access multilingual language model.

Model |  BLEU $\uparrow$ | spBLEU | chrF | chrF++
---|---|---|---|---
Finetuned |  |  |  |
Dragoman P, 10 beams (section 3) | 30.38 | 37.93 | 59.49 | 56.41
Dragoman PT, 10 beams (section 4) | 32.34 | 39.93 | 60.72 | 57.82
Zero shot and few shot (section 5) |  |  |  |
LLaMa-2-7B 2-shot | 20.1 | 26.78 | 49.22 | 46.29
RWKV-5-World-7B 0-shot | 21.06 | 26.20 | 49.46 | 46.46
gpt-4 10-shot | 29.48 | 37.94 | 58.37 | 55.38
gpt-4-turbo-preview 0-shot | 30.36 | 36.75 | 59.18 | 56.19
Google Translate 0-shot | 25.85 | 32.49 | 55.88 | 52.48
Pretrained |  |  |  |
NLLB 3B, 10 beams | 30.46 | 37.22 | 58.11 | 55.32
OPUS-MT, 10 beams | 32.2 | 39.76 | 60.23 | 57.38
Table 5: We evaluate generated translations with the sacrebleu library to calculate BLEU, spBLEU, chrF, and chrF++ metrics on the FLORES DEVTEST set. Metric spBLEU was calculated with default BLEU values and tokenizer flores101. Tokenization and detokenization are done using the models’ default tokenizers. Evaluation is performed on detokenized sentences with corresponding reference sentences. Threshold | Examples |  BLEU $\uparrow$ |
---|---|---|---
percentile |  | dev | devtest
20th | 5800 | 25.14 | 25.49
40th | 11600 | 25.39 | 25.45
50th | 14500 | 25.79 | 25.93
60th | 17400 | 26.07 | 26.01
70th | 20300 | 26.00 | 25.72
80th | 23200 | 25.90 | 26.08
95.4th ($2\sigma$) | 28025 | 25.91 | 25.81
Full dataset | 29000 | 25.74 | 25.67
Table 6: Evaluation scores for model, finetuned from Mistral-7B-v0.1 directly
on Extended Multi30K dataset. We performed log probability thresholds sweep on
FLORES dev set. We demonstrate that data from the second phase alone is not
enough to match the performance of our best checkpoint. Perplexity filtering
improves downstream performance over training on full Extended Multi30K
dataset.

## Appendix

##  Appendix A Translation Examples

### Sample of top 5 worst examples by BLEU from FLORES devtest set

Source: While most of their food would be familiar to us, Romans did have
their share of strange or unusual feast items, including wild boar, peacock,
snails, and a type of rodent called a dormouse
Hypothesis: Хоча більшість їхніх продуктів харчування було б знайомим для нас,
римляни мали свою частку дивних або незвичайних страв, включаючи кабана,
павича, равликів та гризунів, званих куницями.
Reference: Переважно римляни харчувались тим, що знайоме і нам, але були у них
і свої дивні та незвичні бенкетні частування, серед яких дикий кабан, павич,
слимаки і гризун під назвою вовчок.
BLEU: 1.72
Source: "I was moved every time we did a rehearsal on this, from the bottom of
my heart."
Hypothesis: "Я був зворушений кожного разу, коли ми робили репетицію з цього
приводу, від щирого серця".
Reference: Мене це торкало кожен раз під час репетиції до глибини душі.
BLEU: 1.91
Source: Feral children may have experienced severe child abuse or trauma
before being abandoned or running away.
Hypothesis: Дитина-дикун, можливо, пережила жорстоке насильство над дітьми або
травму, перш ніж бути кинутою або втекти.
Reference: Може таке бути, що дикарські діти були об’єктами жорстокого
поводження або отримали серйозні травми перед тим, як їх полишили або вони
втекли.
BLEU: 2.31
Source: Workers must often get their superiors’ approval for any decisions
they make, and are expected to obey their superiors’ instructions without
question.
Hypothesis: Працівники часто повинні отримувати схвалення своїх начальників на
будь-які рішення, які вони приймають, і повинні беззаперечно виконувати
вказівки своїх начальників.
Reference: Працівникам часто потрібно отримати схвалення керівника щодо будь-
яких рішень, а також від них очікують беззаперечного дотримання настанов
керівників.
BLEU: 2.32
Source: Typically there will be a tuition fee to enroll in these educational
programs.
Hypothesis: Як правило, буде плата за навчання, щоб зареєструватися в цих
освітніх програмах.
Reference: Зазвичай такі освітні програми платні.
BLEU: 2.62

### Sample of top 5 best examples by BLEU from FLORES devtest set

Source: The East African Islands are in the Indian Ocean off the eastern coast
of Africa.
Hypothesis: Східноафриканські острови знаходяться в Індійському океані біля
східного узбережжя Африки.
Reference: Східноафриканські острови знаходяться в Індійському океані біля
східного узбережжя Африки.
BLEU: 100.00
Source: Earlier the Chinese news agency Xinhua reported a plane to be
hijacked.
Hypothesis: Раніше китайське інформаційне агентство Сіньхуа повідомило про
викрадення літака.
Reference: Раніше китайське інформаційне агентство Сіньхуа повідомило про
викрадення літака.
BLEU: 100.00
Source: For instance, they didn’t have corn, nor tomatoes, nor potatoes, nor
cocoa, and no ancient Roman ever tasted a turkey.
Hypothesis: Наприклад, у них не було ні кукурудзи, ні помідорів, ні картоплі,
ні какао, і жоден стародавній римлянин ніколи не скуштував індичку.
Reference: Наприклад, у них не було ні кукурудзи, ні помідорів, ні картоплі,
ні какао, і жоден стародавній римлянин ніколи не куштував індичку.
BLEU: 90.95
Source: The luminosity and rotation are used together to determine a star’s
Rossby number, which is related to plasma flow.
Hypothesis: Світність і обертання використовуються разом для визначення числа
Россбі зірки, яке пов’язане з потоком плазми.
Reference: Світність і обертання використовуються разом для визначення числа
Россбі зірки, яке пов’язане з плазмовим потоком.
BLEU: 83.26
Source: But being placed in the "high tropics" just a few degrees north of
equator you will need to deal with both heat (always) and strong sun (when the
sky is clear, more rarely).
Hypothesis: Але перебуваючи в "високих тропіках" всього в декількох градусах
на північ від екватора, вам доведеться мати справу як з спекою (завжди), так і
з сильним сонцем (коли небо чисте, рідше).
Reference: Але, перебуваючи в "високих тропіках" всього в декількох градусах
на північ від екватора, вам доведеться мати справу як зі спекою (завжди), так
і з палючим сонцем (коли небо чисте, рідше).
BLEU: 82.47