# When and Why is Unsupervised Neural Machine Translation Useless?

Yunsu Kim Miguel Graça† Hermann Ney
Human Language Technology and Pattern Recognition Group
RWTH Aachen University, Aachen, Germany
<EMAIL_ADDRESS>
###### Abstract

This paper studies the practicality of the current state-of-the-art
unsupervised methods in neural machine translation (NMT). In ten translation
tasks with various data settings, we analyze the conditions under which the
unsupervised methods fail to produce reasonable translations. We show that
their performance is severely affected by linguistic dissimilarity and domain
mismatch between source and target monolingual data. Such conditions are
common for low-resource language pairs, where unsupervised learning works
poorly. In all of our experiments, supervised and semi-supervised baselines
with 50k-sentence bilingual data outperform the best unsupervised results. Our
analyses pinpoint the limits of the current unsupervised NMT and also suggest
immediate research directions.

##  1 Introduction

Statistical methods for machine translation (MT) require a large set of
sentence pairs in two languages to build a decent translation system [Resnik
and Smith, 2003, Koehn, 2005]. Such bilingual data is scarce for most language
pairs and its quality varies largely over different domains [Al-Onaizan et
al., 2002, Chu and Wang, 2018]. Neural machine translation (NMT) [Bahdanau et
al., 2015, Vaswani et al., 2017], the standard paradigm of MT these days, has
been claimed to suffer from the data scarcity more severely than phrase-based
MT [Koehn and Knowles, 2017].

Unsupervised NMT, which trains a neural translation model only with
monolingual corpora, was proposed for those scenarios which lack bilingual
data [Artetxe et al., 2018b, Lample et al., 2018a]. Despite its progress in
research, the performance of the unsupervised methods has been evaluated
mostly on high-resource language pairs, e.g. German$\leftrightarrow$English or
French$\leftrightarrow$English [Artetxe et al., 2018b, Lample et al., 2018a,
Yang et al., 2018, Artetxe et al., 2018a, Lample et al., 2018b, Ren et al.,
2019b, Artetxe et al., 2019, Sun et al., 2019, Sen et al., 2019]. For these
language pairs, huge bilingual corpora are already available, so there is no
need for unsupervised learning in practice. Empirical results in these tasks
do not carry over to low-resource language pairs; they simply fail to produce
any meaningful translations [Neubig and Hu, 2018, Guzmán et al., 2019].

This paper aims for a more comprehensive and pragmatic study on the
performance of unsupervised NMT. Our experiments span ten translation tasks in
the following five language pairs:

  * •

German$\leftrightarrow$English: similar languages, abundant
bilingual/monolingual data

  * •

Russian$\leftrightarrow$English: distant languages, abundant
bilingual/monolingual data, similar sizes of the alphabet

  * •

Chinese$\leftrightarrow$English: distant languages, abundant
bilingual/monolingual data, very different sizes of the alphabet

  * •

Kazakh$\leftrightarrow$English: distant languages, scarce bilingual data,
abundant monolingual data

  * •

Gujarati$\leftrightarrow$English: distant languages, scarce
bilingual/monolingual data

For each task, we compare the unsupervised performance with its supervised and
semi-supervised counterparts. In addition, we make the monolingual training
data vary in size and domain to cover many more scenarios, showing under which
conditions unsupervised NMT works poorly.

Here is a summary of our contributions:

  * •

We thoroughly evaluate the performance of state-of-the-art unsupervised NMT in
numerous real and artificial translation tasks.

  * •

We provide guidelines on whether to employ unsupervised NMT in practice, by
showing how much bilingual data is sufficient to outperform the unsupervised
results.

  * •

We clarify which factors make unsupervised NMT weak and which points must be
improved, by analyzing the results both quantitatively and qualitatively.

##  2 Related Work

The idea of unsupervised MT dates back to word-based decipherment methods
[Knight et al., 2006, Ravi and Knight, 2011]. They learn only lexicon models
at first, but add alignment models [Dou et al., 2014, Nuhn, 2019] or heuristic
features [Naim et al., 2018] later. Finally, ?) and ?) train a fully-fledged
phrase-based MT system in an unsupervised way.

With neural networks, unsupervised learning of a sequence-to-sequence NMT
model has been proposed by ?) and ?). Though having slight variations [Yang et
al., 2018, Sun et al., 2019, Sen et al., 2019], unsupervised NMT approaches
commonly 1) learn a shared model for both source$\rightarrow$target and
target$\rightarrow$source 2) using iterative back-translation, along with 3) a
denoising autoencoder objective. They are initialized with either cross-
lingual word embeddings or a cross-lingual language model (LM). To further
improve the performance at the cost of efficiency, ?), ?) and ?) combine
unsupervised NMT with unsupervised phrase-based MT. On the other hand, one can
also avoid the long iterative training by applying a separate denoiser
directly to the word-by-word translations from cross-lingual word embeddings
[Kim et al., 2018, Pourdamghani et al., 2019].

Unsupervised NMT approaches have been so far evaluated mostly on high-resource
language pairs, e.g. French$\rightarrow$English, for academic purposes. In
terms of practicality, they tend to underperform in low-resource language
pairs, e.g. Azerbaijani$\rightarrow$English [Neubig and Hu, 2018] or
Nepali$\rightarrow$English [Guzmán et al., 2019]. To the best of our
knowledge, this work is the first to systematically evaluate and analyze
unsupervised learning for NMT in various data settings.

##  3 Unsupervised NMT

This section reviews the core concepts of the recent unsupervised NMT
framework and describes to which points they are potentially vulnerable.

###  3.1 Bidirectional Modeling

Most of the unsupervised NMT methods share the model parameters between
source$\rightarrow$target and target$\rightarrow$source directions. They also
often share a joint subword vocabulary across the two languages [Sennrich et
al., 2016b].

Sharing a model among different translation tasks has been shown to be
effective in multilingual NMT [Firat et al., 2016, Johnson et al., 2017,
Aharoni et al., 2019], especially in improving performance on low-resource
language pairs. This is due to the commonality of natural languages; learning
to represent a language is helpful to represent other languages, e.g. by
transferring knowledge of general sentence structures. It also provides good
regularization for the model.

Unsupervised learning is an extreme scenario of MT, where bilingual
information is very weak. To supplement the weak and noisy training signal,
knowledge transfer and regularization are crucial, which can be achieved by
the bidirectional sharing. It is based on the fact that a translation problem
is dual in nature; source$\rightarrow$target and target$\rightarrow$source
tasks are conceptually related to each other.

Previous works on unsupervised NMT vary in the degree of sharing: the whole
encoder [Artetxe et al., 2018b, Sen et al., 2019], the middle layers [Yang et
al., 2018, Sun et al., 2019], or the whole model [Lample et al., 2018a, Lample
et al., 2018b, Ren et al., 2019a, Conneau and Lample, 2019].

Note that the network sharing is less effective among linguistically distinct
languages in NMT [Kocmi and Bojar, 2018, Kim et al., 2019a]. It still works as
a regularizer, but transferring knowledge is harder if the morphology or word
order is quite different. We show how well unsupervised NMT performs on such
language pairs in Section 4.1.

###  3.2 Iterative Back-Translation

Unsupervised learning for MT assumes no bilingual data for training. A
traditional remedy for the data scarcity is generating synthetic bilingual
data from monolingual text [Koehn, 2005, Schwenk, 2008, Sennrich et al.,
2016a]. To train a bidirectional model of Section 3.1, we need bilingual data
of both translation directions. Therefore, most unsupervised NMT methods back-
translate in both directions, i.e. source and target monolingual data to
target and source language, respectively.

In unsupervised learning, the synthetic data should be created not only once
at the beginning but also repeatedly throughout the training. At the early
stages of training, the model might be too weak to generate good translations.
Hence, most methods update the training data as the model gets improved during
training. The improved model for source$\rightarrow$target direction back-
translates source monolingual data, which improves the model for
target$\rightarrow$source direction, and vice versa. This cycle is called dual
learning [He et al., 2016] or iterative back-translation [Hoang et al., 2018].
Figure 1 shows the case when it is applied to a fully shared bidirectional
model.

\begin{overpic}[width=303.53267pt]{unsup-bt-s2t-correct.pdf}
\put(2.0,49.0){1)} \put(32.0,49.0){2)} \end{overpic} (a)

\begin{overpic}[width=303.53267pt]{unsup-bt-t2s-correct.pdf}
\put(2.0,49.0){1)} \put(32.0,49.0){2)} \end{overpic} (b)

Figure 1: Iterative back-translation for training a bidirectional sequence-to-
sequence model. The model first translates monolingual sentences (solid
arrows), and then gets trained with the translation as the input and the
original as the output (dashed arrows). This procedure alternates between (a)
source$\rightarrow$target and (b) target$\rightarrow$source translations.

One can tune the amount of back-translations per iteration: a mini-batch
[Artetxe et al., 2018b, Yang et al., 2018, Conneau and Lample, 2019, Ren et
al., 2019a], the whole monolingual data [Lample et al., 2018a, Lample et al.,
2018b, Sun et al., 2019], or some size in between [Artetxe et al., 2019, Ren
et al., 2019b].

However, even if carefully scheduled, the iterative training cannot recover
from a bad optimum if the initial model is too poor. Experiments in Section
4.5 highlight such cases.

###  3.3 Initialization

To kickstart the iterative training, the model should be able to generate
meaningful translations already in the first iteration. We cannot expect the
training to progress from a randomly initialized network and the synthetic
data generated by it.

Cross-lingual embeddings give a good starting point for the model by defining
a joint continuous space shared by multiple languages. Ideally, in such a
space, close embedding vectors are semantically related to each other
regardless of their languages; they can be possible candidates for translation
pairs [Mikolov et al., 2013]. It can be learned either in word level [Artetxe
et al., 2017, Conneau et al., 2018] or in sentence level [Conneau and Lample,
2019] using only monolingual corpora.

In the word level, we can initialize the embedding layers with cross-lingual
word embedding vectors [Artetxe et al., 2018b, Lample et al., 2018a, Yang et
al., 2018, Lample et al., 2018b, Artetxe et al., 2019, Sun et al., 2019]. On
the other hand, the whole encoder/decoder parameters can be initialized with
cross-lingual sequence training [Conneau and Lample, 2019, Ren et al., 2019a,
Song et al., 2019].

Cross-lingual word embedding has limited performance among distant languages
[Søgaard et al., 2018, Nakashole and Flauger, 2018] and so does cross-lingual
LM [Pires et al., 2019]. Section 4.5 shows the impact of a poor
initialization.

|  | de-en | ru-en | zh-en | kk-en | gu-en
---|---|---|---|---|---|---
|  | German | English | Russian | English | Chinese | English | Kazakh | English | Gujarati | English
Language family | Germanic | Germanic | Slavic | Germanic | Sinitic | Germanic | Turkic | Germanic | Indic | Germanic
Alphabet Size | 60 | 52 | 66 | 52 | 8,105 | 52 | 42 | 52 | 91 | 52
Monolingual | Sentences | 100M | 71.6M | 30.8M | 18.5M | 4.1M
Words | 1.8B | 2.3B | 1.1B | 2.0B | 1.4B | 699M | 278.5M | 421.5M | 121.5M | 93.8M
Bilingual | Sentences | 5.9M | 25.4M | 18.9M | 222k | 156k
Words | 137.4M | 144.9M | 618.6M | 790M | 440.3M | 482.9M | 1.6M | 1.9M | 2.3M | 1.5M
Table 1: Training data statistics.

###  3.4 Denoising Autoencoder

Initializing the word embedding layers furnishes the model with cross-lingual
matching in the lexical embedding space, but does not provide any information
on word orders or generation of text. Cross-lingual LMs encode word sequences
in different languages, but they are not explicitly trained to reorder source
words to the target language syntax. Both ways do not initialize the crucial
parameters for reordering: the encoder-decoder attention and the recurrence on
decoder states.

As a result, an initial model for unsupervised NMT tends to generate word-by-
word translations with little reordering, which are very non-fluent when
source and target languages have distinct word orders. Training on such data
discourages the model from reordering words, which might cause a vicious cycle
by generating even less-reordered synthetic sentence pairs in the next
iterations.

Accordingly, unsupervised NMT employs an additional training objective of
denoising autoencoding [Hill et al., 2016]. Given a clean sentence, artificial
noises are injected, e.g. deletion or permutation of words, to make a
corrupted input. The denoising objective trains the model to reorder the noisy
input to the correct syntax, which is essential for generating fluent outputs.
This is done for each language individually with monolingual data, as shown in
Figure 2.

Figure 2: Denoising autoencoder training for source or target language.

Once the model is sufficiently trained for denoising, it is helpful to remove
the objective or reduce its weight [Graça et al., 2018]. At the later stages
of training, the model gets improved in reordering and translates better;
learning to denoise might hurt the performance in clean test sets.

|  |  Bleu [%]
---|---|---
Approach |  | de-en | en-de | ru-en | en-ru | zh-en | en-zh | kk-en | en-kk | gu-en | en-gu
Supervised |  | 39.5 | 39.1 | 29.1 | 24.7 | 26.2 | 39.6 | 10.3 | 2.4 | 9.9 | 3.5
Semi-supervised |  | 43.6 | 41.0 | 30.8 | 28.8 | 25.9 | 42.7 | 12.5 | 3.1 | 14.2 | 4.0
Unsupervised |  | 23.8 | 20.2 | 12.0 | 9.4 | 1.5 | 2.5 | 2.0 | 0.8 | 0.6 | 0.6
Table 2: Comparison among supervised, semi-supervised, and unsupervised
learning. All bilingual data was used for the (semi-)supervised results and
all monolingual data was used for the unsupervised results (see Table 1). All
results are computed on newstest2019 of each task, except for de-en/en-de and
ru-en/en-ru on newstest2018.

##  4 Experiments and Analysis

Data Our experiments were conducted on WMT 2018 German$\leftrightarrow$English
and Russian$\leftrightarrow$ English, WMT 2019
Chinese$\leftrightarrow$English, Kazakh$\leftrightarrow$ English, and
Gujarati$\leftrightarrow$English (Table 1). We preprocessed the data using the
Moses111http://www.statmt.org/moses tokenizer and a frequent caser. For
Chinese, we used the jieba segmenter222https://github.com/fxsjy/jieba. Lastly,
byte pair encoding (BPE) [Sennrich et al., 2016b] was learned jointly over
source and target languages with 32k merges and applied without vocabulary
threshold.

Model We used 6-layer Transformer base architecture [Vaswani et al., 2017] by
default: 512-dimension embedding/hidden layers, 2048-dimension feedforward
sublayers, and 8 heads.

Decoding and Evaluation Decoding was done with beam size 5. We evaluated the
test performance with SacreBleu [Post, 2018].

Unsupervised Learning We ran Xlm333https://github.com/facebookresearch/XLM by
?) for the unsupervised experiments. The back-translations were done with beam
search for each mini-batch of 16k tokens. The weight of the denoising
objective started with 1 and linearly decreased to 0.1 until 100k updates, and
then decreased to 0 until 300k updates.

The model’s encoder and decoder were both initialized with the same pre-
trained cross-lingual LM. We removed the language embeddings from the encoder
for better cross-linguality (see Section 4.6). Unless otherwise specified, we
used the same monolingual training data for both pre-training and translation
training. For the pre-training, we set the batch size to 256 sentences (around
66k tokens).

Training was done with Adam [Kingma and Ba, 2014] with an initial learning
rate of 0.0001, where dropout [Srivastava et al., 2014] of probability 0.1 was
applied to each layer output and attention components. With a checkpoint
frequency of 200k sentences, we stopped the training when the validation
perplexity (pre-training) or Bleu (translation training) was not improved for
ten checkpoints. We extensively tuned the hyperparameters for a single GPU
with 12GB memory, which is widely applicable to moderate industrial/academic
environments. All other hyperparameter values follow the recommended settings
of Xlm.

Supervised Learning Supervised experiments used the same hyperparameters as
the unsupervised learning, except 12k tokens for the batch size, 0.0002 for
the initial learning rate, and 10k batches for each checkpoint.

If the bilingual training data contains less than 500k sentence pairs, we
reduced the BPE merges to 8k, the batch size to 2k, and the checkpoint
frequency to 4k batches; we also increased the dropout rate to 0.3 [Sennrich
and Zhang, 2019].

Semi-supervised Learning Semi-supervised experiments continued the training
from the supervised baseline with back-translations added to the training
data. We used 4M back-translated sentences for the low-resource cases, i.e. if
the original bilingual data has less than 500k lines, and 10M back-translated
sentences otherwise.

###  4.1 Unsupervised vs. (Semi-)Supervised

We first address the most general question of this paper: For NMT, can
unsupervised learning replace semi-supervised or supervised learning? Table 2
compares the unsupervised performance to simple supervised and semi-supervised
baselines.

In all tasks, unsupervised learning shows much worse performance than
(semi-)supervised learning. It produces readable translations in two high-
resource language pairs (German$\leftrightarrow$English and
Russian$\leftrightarrow$English), but their scores are only around half of the
semi-supervised systems. In other three language pairs, unsupervised NMT fails
to converge at any meaningful optimum, reaching less than 3% Bleu scores. Note
that, in these three tasks, source and target languages are very different in
the alphabet, morphology, and word order, etc. The results in
Kazakh$\leftrightarrow$English and Gujarati$\leftrightarrow$English show that
the current unsupervised NMT cannot be an alternative to (semi-)supervised NMT
in low-resource conditions.

(a) German$\rightarrow$English

(b) Russian$\rightarrow$English

Figure 3: Supervised and semi-supervised learning over bilingual training data
size. Unsupervised learning (horizontal line) uses all monolingual data of
Table 1.

To discover the precise condition where the unsupervised learning is useful in
practice, we vary the size of the given bilingual training data for
(semi-)supervised learning and plot the results in Figure 3. Once we have 50k
bilingual sentence pairs in German$\leftrightarrow$English, simple semi-
supervised learning already outperforms unsupervised learning with 100M
monolingual sentences in each language. Even without back-translations
(supervised), 100k-sentence bilingual data is sufficient to surpass
unsupervised NMT.

In the Russian$\leftrightarrow$English task, the unsupervised learning
performance can be more easily achieved with only 20k bilingual sentence pairs
using semi-supervised learning. This might be due to that Russian and English
are more distant to each other than German and English, thus bilingual
training signal is more crucial for Russian$\leftrightarrow$English.

Note that for these two language pairs, the bilingual data for supervised
learning are from many different text domains, whereas the monolingual data
are from exactly the same domain of the test sets. Even with such an
advantage, the large-scale unsupervised NMT cannot compete with supervised NMT
with tiny out-of-domain bilingual data.

###  4.2 Monolingual Data Size

In this section, we analyze how much monolingual data is necessary to make
unsupervised NMT produce reasonable performance. Figure 4 shows the
unsupervised results with different amounts of monolingual training data. We
keep the equal size for source and target data, and the domain is also the
same for both (web-crawled news).

Figure 4: Unsupervised NMT performance over the size of monolingual training
data, where source and target sides have the same size.

For German$\rightarrow$English, training with only 1M sentences already gives
a reasonable performance, which is only around 2% Bleu behind the
100M-sentence case. The performance starts to saturate already after 5M
sentences, with only marginal improvements by using more than 20M sentences.
We observe a similar trend in Russian$\rightarrow$English.

This shows that, for the performance of unsupervised NMT, using a massive
amount of monolingual data is not as important as the similarity of source and
target languages. Comparing to supervised learning (see Figure 3), the
performance saturates faster when increasing the training data, given the same
model size.

###  4.3 Unbalanced Data Size

What if the size of available monolingual data is largely different for source
and target languages? This is often the case for low-resource language pairs
involving English, where there is plenty of data for English but not for the
other side.

Our experiments so far intentionally use the same number of sentences for both
sides. In Figure 5, we reduced the source data gradually while keeping the
large target data fixed. To counteract the data imbalance, we oversampled the
smaller side to make the ratio of source-target 1:1 for BPE learning and mini-
batch construction [Conneau and Lample, 2019]. We compare such unbalanced data
settings to the previous equal-sized source/target settings.

Figure 5: Unsupervised NMT performance over source training data size, where
the target training data is fixed to 20M sentences (dashed line). Solid line
is the case where the target data has the same number of sentences as the
source side.

Interestingly, when we decrease the target data accordingly (balanced, solid
line), the performance is similar or sometimes better than using the full
target data (unbalanced, dashed line). This means that it is not beneficial to
use oversized data on one side in unsupervised NMT training.

If the data is severely unbalanced, the distribution of the smaller side
should be much sparser than that of the larger side. The network tries to
generalize more on the smaller data, reserving the model capacity for
smoothing [Olson et al., 2018]. Thus it learns to represent a very different
distribution of each side, which is challenging in a shared model (Section
3.1). This could be the reason for no merit in using larger data on one side.

###  4.4 Domain Similarity

In high-resource language pairs, it is feasible to collect monolingual data of
the same domain on both source and target languages. However, for low-resource
language pairs, it is difficult to match the data domain of both sides on a
large scale. For example, our monolingual data for Kazakh is mostly from
Wikipedia and Common Crawl, while the English data is solely from News Crawl.
In this section, we study how the domain similarity of monolingual data on the
two sides affects the performance of unsupervised NMT.

In Table 3, we artificially change the domain of the source side to politics
(UN Corpus444https://conferences.unite.un.org/uncorpus) or random (Common
Crawl), while keeping the target domain fixed to newswire (News Crawl). The
results show that the domain matching is critical for unsupervised NMT. For
instance, although German and English are very similar languages, we see the
performance of German$\leftrightarrow$English deteriorate down to -11.8% Bleu
by the domain mismatch.

Domain | Domain |  |  Bleu [%]
---|---|---|---
(en) | (de/ru) |  | de-en | en-de | ru-en | en-ru
Newswire | Newswire |  | 23.3 | 19.9 | 11.9 | 9.3
Politics |  | 11.5 | 12.2 | 2.3 | 2.5
Random |  | 18.4 | 16.4 | 6.9 | 6.1
Table 3: Unsupervised NMT performance where source and target training data
are from different domains. The data size on both sides is the same (20M
sentences).

Table 4 shows a more delicate case where we keep the same domain for both
sides (newswire) but change the providers and years of the news articles. Our
monolingual data for Chinese (Table 1) consist mainly of News Crawl (from
years 2008-2018) and Gigaword 4th edition (from years 1995-2008). We split out
the News Crawl part (1.7M sentences) and trained an unsupervised NMT model
with the same amount of English monolingual data (from News Crawl 2014-2017).
Surprisingly, this experiment yields much better results than using all
available data. Even if the size is small, the source and target data are
collected in the same way (web-crawling) from similar years (2010s), which
seems to be crucial for unsupervised NMT to work.

Years | Years | #sents |  |  Bleu [%]
---|---|---|---|---
(en) | (zh) | (en/zh) |  | zh-en | en-zh
2014-2017 | 2008-2018 |  1.7M |  | 5.4 | 15.1
1995-2008 | 28.6M |  | 1.5 | 1.9
Table 4: Unsupervised NMT performance where source and target training data
are from the same domain (newswire) but different years.

On the other hand, when using the Gigaword part (28.6M sentences) on Chinese,
unsupervised learning again does not function properly. Now the source and
target text are from different decades; the distribution of topics might be
different. Also, the Gigaword corpus is from traditional newspaper agencies
which can have a different tone from the online text of News Crawl. Despite
the large scale, unsupervised NMT proves to be sensitive to a subtle
discrepancy of topic, style, period, etc. between source and target data.

These results agree with ?) who show that modern cross-lingual word embedding
methods fail in domain mismatch scenarios.

Task |  Bleu [%] |  Source input |  System output |  Reference output
---|---|---|---|---
de-en | 23.8 |  Seit der ersten Besichtigung wurde die 1.000 Quadratfuß große … |  Since the first Besichtigung, the 3,000 square fueled … |  Since the first viewing, the 1,000sq ft flat has …
10.4 |  München 1856: Vier Karten, die Ihren Blick auf die Stadt verändern |  Austrailia 1856: Eight things that can keep your way to the UK |  Munich 1856: Four maps that will change your view of the city
ru-en | 12.0 |  В ходе первоочередных оператив-
но-следственных мероприятий ус-
тановлена личность роженицы |  The первоочередных оператив-
но-следственных мероприятий have been established by the dolphin |  The identity of the mother was determined during preliminary investigative and operational measures
zh-en |  1.5 |  … 调整 要 兼顾 生产 需要和消费需求。 |  … 调整要兼顾生产需要 and 消费需
求. |  … adjustment must balance production needs with consumer demands.
Table 5: Problematic translation outputs from unsupervised NMT systems (input
copying, ambiguity in the same context).

###  4.5 Initialization vs. Translation Training

Thus far, we have seen a number of cases where unsupervised NMT breaks down.
But which part of the learning algorithm is more responsible for the
performance: initialization (Section 3.3) or translation training (Section 3.2
and 3.4)?

In Figure 6, we control the level of each of the two training stages and
analyze its impact on the final performance. We pre-trained two cross-lingual
LMs as initializations of different quality: bad (using 10k sentences) and
good (using 20M sentences). For each initial point, we continued the
translation training with different amounts of data from 10k to 20M sentences.

Figure 6: Unsupervised NMT performance over the training data size for
translation training, where the pre-training data for initialization is fixed
(10k or 20M sentences).

From the bad initialization, unsupervised learning cannot build a reasonable
NMT model, no matter how much data is used in translation training. When the
initial model is strong, it is possible to reach 20% Bleu by translation
training with only 100k sentences. Using 1M sentences in translation training,
the performance is already comparable to its best. Once the model is pre-
trained well for cross-lingual representations, fine-tuning the translation-
specific components seems manageable with relatively small data.

This demonstrates the importance of initialization over translation training
in the current unsupervised NMT. Translation training relies solely on model-
generated inputs, i.e. back-translations, which do not reflect the true
distribution of the input language when generated with a poor initial model.
On Figure 7, we plot all German$\rightarrow$English unsupervised results we
conducted up to the previous section. It shows that the final performance
generally correlates with the initialization quality.

Figure 7: Unsupervised NMT performance over the validation perplexity of the
initial cross-lingual LM (de-en).

###  4.6 Qualitative Examples

In this section, we analyze translation outputs of unsupervised systems to
find out why they record such low Bleu scores. Do unsupervised systems have
particular problems in the outputs other than limited adequacy/fluency?

Table 5 shows translation examples from the unsupervised systems. The first
notable problem is copying input words to the output. This happens when the
encoder has poor cross-linguality, i.e. does not concurrently model two
languages well in a shared space. The decoder then can easily detect the input
language by reading the encoder and may emit output words in the same
language.

A good cross-lingual encoder should not give away information on the input
language to the decoder. The decoder must instead rely on the ouptut language
embeddings or an indicator token (e.g. <2en>) to determine the language of
output tokens. As a simple remedy, we removed the language embeddings from the
encoder and obtained consistent improvements, e.g. from 4.3% to 11.9% Bleu in
Russian$\rightarrow$English. However, the problem still remains partly even in
our best-performing unsupervised system (the first example).

The copying occurs more often in inferior systems (the last example), where
the poor initial cross-lingual LM is the main reason for the worse performance
(Section 4.5). Note that the auto-encoding (Section 3.4) also encourages the
model to generate outputs in the input language. We provide more in-depth
insights on the copying phenomenon in the appendix (Section A).

Another problem is that the model cannot distinguish words that appear in the
same context. In the second example, the model knows that Vier in German (Four
in English) is a number, but it generates a wrong number in English (Eight).
The initial LM is trained to predict either Four or Eight given the same
surrounding words (e.g. 1856, things) and has no clue to map Four to Vier.

The model cannot learn these mappings by itself with back-translations. This
problem can be partly solved by subword modeling [Bojanowski et al., 2017] or
orthographic features [Riley and Gildea, 2018, Artetxe et al., 2019], which
are however not effective for language pairs with disjoint alphabets.

##  5 Conclusion and Outlook

In this paper, we examine the state-of-the-art unsupervised NMT in a wide
range of tasks and data settings. We find that the performance of unsupervised
NMT is seriously affected by these factors:

  * •

Linguistic similarity of source and target languages

  * •

Domain similarity of training data between source and target languages

It is very hard to fulfill these in low-/zero-resource language pairs, which
makes the current unsupervised NMT useless in practice. We also find that the
performance is not improved by using massive monolingual data on one or both
sides.

In practice, a simple, non-tuned semi-supervised baseline with only less than
50k bilingual sentence pairs is sufficient to outperform our best large-scale
unsupervised system. At this moment, we cannot recommend unsupervised learning
for building MT products if there are at least small bilingual data.

For the cases where there is no bilingual data available at all, we plan to
systematically compare the unsupervised NMT to pivot-based methods [Kim et
al., 2019b, Currey and Heafield, 2019] or multilingual zero-shot translation
[Johnson et al., 2017, Aharoni et al., 2019].

To make unsupervised NMT useful in the future, we suggest the following
research directions:

Language-/Domain-agnostic LM We show in Section 4.5 that the initial cross-
lingual LM actually determines the performance of unsupervised NMT. In Section
4.6, we argue that the poor performance is due to input copying, for which we
blame a poor cross-lingual LM. The LM pre-training must therefore handle
dissimilar languages and domains equally well. This might be done by careful
data selection or better regularization methods.

Robust Translation Training On the other hand, the current unsupervised NMT
lacks a mechanism to bootstrap out of a poor initialization. Inspired by
classical decipherment methods (Section 2), we might devalue noisy training
examples or artificially simplify the problem first.

## References

  * [Aharoni et al., 2019]  Aharoni, Roee, Melvin Johnson, and Orhan Firat.  2019\.  Massively multilingual neural machine translation.  In NAACL-HLT, pages 3874–3884.
  * [Al-Onaizan et al., 2002]  Al-Onaizan, Yaser, Ulrich Germann, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Daniel Marcu, and Kenji Yamada.  2002\.  Translation with scarce bilingual resources.  Machine Translation, 17(1):1–17.
  * [Artetxe et al., 2017]  Artetxe, Mikel, Gorka Labaka, and Eneko Agirre.  2017\.  Learning bilingual word embeddings with (almost) no bilingual data.  In ACL, pages 451–462.
  * [Artetxe et al., 2018a]  Artetxe, Mikel, Gorka Labaka, and Eneko Agirre.  2018a.  Unsupervised statistical machine translation.  In EMNLP, page 3632–3642.
  * [Artetxe et al., 2018b]  Artetxe, Mikel, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho.  2018b.  Unsupervised neural machine translation.  In ICLR.
  * [Artetxe et al., 2019]  Artetxe, Mikel, Gorka Labaka, and Eneko Agirre.  2019\.  An effective approach to unsupervised machine translation.  In ACL, pages 194–203.
  * [Bahdanau et al., 2015]  Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio.  2015\.  Neural machine translation by jointly learning to align and translate.  In ICLR.
  * [Bojanowski et al., 2017]  Bojanowski, Piotr, Edouard Grave, Armand Joulin, and Tomas Mikolov.  2017\.  Enriching word vectors with subword information.  TACL, 5:135–146.
  * [Chu and Wang, 2018]  Chu, Chenhui and Rui Wang.  2018\.  A survey of domain adaptation for neural machine translation.  In COLING, pages 1304–1319.
  * [Conneau and Lample, 2019]  Conneau, Alexis and Guillaume Lample.  2019\.  Cross-lingual language model pretraining.  In NeurIPS, pages 7057–7067.
  * [Conneau et al., 2018]  Conneau, Alexis, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou.  2018\.  Word translation without parallel data.  In ICLR.
  * [Currey and Heafield, 2019]  Currey, Anna and Kenneth Heafield.  2019\.  Zero-resource neural machine translation with monolingual pivot data.  In WNGT, pages 99–107.
  * [Dou et al., 2014]  Dou, Qing, Ashish Vaswani, and Kevin Knight.  2014\.  Beyond parallel data: Joint word alignment and decipherment improves machine translation.  In EMNLP, pages 557–565.
  * [Firat et al., 2016]  Firat, Orhan, Kyunghyun Cho, and Yoshua Bengio.  2016\.  Multi-way, multilingual neural machine translation with a shared attention mechanism.  In NAACL-HLT, pages 866–875.
  * [Graça et al., 2018]  Graça, Miguel, Yunsu Kim, Julian Schamper, Jiahui Geng, and Hermann Ney.  2018\.  The RWTH aachen university English-German and German-English unsupervised neural machine translation systems for WMT 2018\.  In WMT.
  * [Guzmán et al., 2019]  Guzmán, Francisco, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato.  2019\.  The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala–English.  In EMNLP-IJCNLP, pages 6097–6110.
  * [He et al., 2016]  He, Di, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma.  2016\.  Dual learning for machine translation.  In NIPS, pages 820–828.
  * [Hill et al., 2016]  Hill, Felix, Kyunghyun Cho, and Anna Korhonen.  2016\.  Learning distributed representations of sentences from unlabelled data.  In NAACL-HLT, pages 1367–1377.
  * [Hoang et al., 2018]  Hoang, Vu Cong Duy, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn.  2018\.  Iterative back-translation for neural machine translation.  In WNGT, pages 18–24.
  * [Johnson et al., 2017]  Johnson, Melvin, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al.  2017\.  Google’s multilingual neural machine translation system: Enabling zero-shot translation.  TACL, 5(1):339–351.
  * [Kim et al., 2018]  Kim, Yunsu, Jiahui Geng, and Hermann Ney.  2018\.  Improving unsupervised word-by-word translation with language model and denoising autoencoder.  In EMNLP, pages 862–868.
  * [Kim et al., 2019a]  Kim, Yunsu, Yingbo Gao, and Hermann Ney.  2019a.  Effective cross-lingual transfer of neural machine translation models without shared vocabularies.  In ACL, pages 1246–1257.
  * [Kim et al., 2019b]  Kim, Yunsu, Petre Petrov, Pavel Petrushkov, Shahram Khadivi, and Hermann Ney.  2019b.  Pivot-based transfer learning for neural machine translation between non-English languages.  In EMNLP-IJCNLP, pages 866–876.
  * [Kingma and Ba, 2014]  Kingma, Diederik P and Jimmy Ba.  2014\.  Adam: A method for stochastic optimization.
  * [Knight et al., 2006]  Knight, Kevin, Anish Nair, Nishit Rathod, and Kenji Yamada.  2006\.  Unsupervised analysis for decipherment problems.  In COLING/ACL, pages 499–506.
  * [Kocmi and Bojar, 2018]  Kocmi, Tom and Ondřej Bojar.  2018\.  Trivial transfer learning for low-resource neural machine translation.  In WMT, pages 244–252.
  * [Koehn and Knowles, 2017]  Koehn, Philipp and Rebecca Knowles.  2017\.  Six challenges for neural machine translation.  In WNMT, pages 28–39.
  * [Koehn, 2005]  Koehn, Philipp.  2005\.  Europarl: A parallel corpus for statistical machine translation.  In MT Summit, pages 79–86.
  * [Lample et al., 2018a]  Lample, Guillaume, Ludovic Denoyer, and Marc’Aurelio Ranzato.  2018a.  Unsupervised machine translation using monolingual corpora only.  In ICLR.
  * [Lample et al., 2018b]  Lample, Guillaume, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato.  2018b.  Phrase-based & neural unsupervised machine translation.  In EMNLP, pages 5039–5049.
  * [Liu et al., 2020]  Liu, Yinhan, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer.  2020\.  Multilingual denoising pre-training for neural machine translation.
  * [Mikolov et al., 2013]  Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean.  2013\.  Efficient estimation of word representations in vector space.
  * [Naim et al., 2018]  Naim, Iftekhar, Parker Riley, and Daniel Gildea.  2018\.  Feature-based decipherment for machine translation.  Computational Linguistics, 44(3):525–546.
  * [Nakashole and Flauger, 2018]  Nakashole, Ndapandula and Raphael Flauger.  2018\.  Characterizing departures from linearity in word translation.  In ACL, pages 221–227.
  * [Neubig and Hu, 2018]  Neubig, Graham and Junjie Hu.  2018\.  Rapid adaptation of neural machine translation to new languages.  In EMNLP, pages 875–880.
  * [Nuhn, 2019]  Nuhn, Malte.  2019\.  Unsupervised Training with Applications in Natural Language Processing.  Ph.D. thesis, Computer Science Department, RWTH Aachen University.
  * [Olson et al., 2018]  Olson, Matthew, Abraham Wyner, and Richard Berk.  2018\.  Modern neural networks generalize on small data sets.  In NIPS, pages 3619–3628.
  * [Pires et al., 2019]  Pires, Telmo, Eva Schlinger, and Dan Garrette.  2019\.  How multilingual is multilingual bert?  In ACL, pages 4996–5001.
  * [Post, 2018]  Post, Matt.  2018\.  A call for clarity in reporting bleu scores.  In WMT, pages 186–191.
  * [Pourdamghani et al., 2019]  Pourdamghani, Nima, Nada Aldarrab, Marjan Ghazvininejad, Kevin Knight, and Jonathan May.  2019\.  Translating translationese: A two-step approach to unsupervised machine translation.  In ACL, pages 3057–3062.
  * [Ravi and Knight, 2011]  Ravi, Sujith and Kevin Knight.  2011\.  Deciphering foreign language.  In ACL, pages 12–21.
  * [Ren et al., 2019a]  Ren, Shuo, Yu Wu, Shujie Liu, Ming Zhou, and Shuai Ma.  2019a.  Explicit cross-lingual pre-training for unsupervised machine translation.  In EMNLP-IJCNLP, pages 770–779.
  * [Ren et al., 2019b]  Ren, Shuo, Zhirui Zhang, Shujie Liu, Ming Zhou, and Shuai Ma.  2019b.  Unsupervised neural machine translation with smt as posterior regularization.
  * [Resnik and Smith, 2003]  Resnik, Philip and Noah A. Smith.  2003\.  The web as a parallel corpus.  Computational Linguistics, 29(3):349–380.
  * [Riley and Gildea, 2018]  Riley, Parker and Daniel Gildea.  2018\.  Orthographic features for bilingual lexicon induction.  In ACL, pages 390–394.
  * [Schwenk, 2008]  Schwenk, Holger.  2008\.  Investigations on large-scale lightly-supervised training for statistical machine translation.  In IWSLT.
  * [Sen et al., 2019]  Sen, Sukanta, Kamal Kumar Gupta, Asif Ekbal, and Pushpak Bhattacharyya.  2019\.  Multilingual unsupervised NMT using shared encoder and language-specific decoders.  In ACL, pages 3083–3089.
  * [Sennrich and Zhang, 2019]  Sennrich, Rico and Biao Zhang.  2019\.  Revisiting low-resource neural machine translation: A case study.  In ACL, pages 211–221.
  * [Sennrich et al., 2016a]  Sennrich, Rico, Barry Haddow, and Alexandra Birch.  2016a.  Improving neural machine translation models with monolingual data.  In ACL, pages 86–96.
  * [Sennrich et al., 2016b]  Sennrich, Rico, Barry Haddow, and Alexandra Birch.  2016b.  Neural machine translation of rare words with subword units.  In ACL, pages 1715–1725.
  * [Søgaard et al., 2018]  Søgaard, Anders, Sebastian Ruder, and Ivan Vulić.  2018\.  On the limitations of unsupervised bilingual dictionary induction.  In ACL, pages 778–788.
  * [Song et al., 2019]  Song, Kaitao, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu.  2019\.  Mass: Masked sequence to sequence pre-training for language generation.  In ICML, pages 5926–5936.
  * [Srivastava et al., 2014]  Srivastava, Nitish, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  2014\.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958.
  * [Sun et al., 2019]  Sun, Haipeng, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao.  2019\.  Unsupervised bilingual word embedding agreement for unsupervised neural machine translation.  In ACL, pages 1235–1245.
  * [Vaswani et al., 2017]  Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is all you need.  In NIPS, pages 5998–6008.
  * [Yang et al., 2018]  Yang, Zhen, Wei Chen, Feng Wang, and Bo Xu.  2018\.  Unsupervised neural machine translation with weight sharing.  In ACL, pages 46–55.

##  Appendix A Input Copying

This supplement further investigates why the input copying (Section 4.6)
occurs in the current unsupervised NMT. We discover its root cause in the
unsupervised loss function, present possible remedies, and illustrate the
relation to model cross-linguality with a toy experiment.

###  A.1 Reconstruction Loss

Training loss of the unsupervised NMT (Figure 1) is basically reconstruction
of a monolingual sentence via an intermediate representation, created as the
most probable output sequence of the current model’s parameters. This
introduces an intrinsic divergence of the model’s usage between in training
(creating an intermediate sequence facilitates the reconstruction) and in
testing (producing a correct translation).

Note that there are no constraints on the intermediate space in training. This
gives rise to a plethora of solutions to the loss optimization, which might be
not aligned with the actual goal of translation. In principle, a model could
learn any bijective function from a monolingual corpus to a set of distinct
sentences of the same size.

Here, input copying (Table 5) is a trivial action for the bidirectional model
with a shared vocabulary, which is reinforced by training on copied back-
translations. It is easier than performing any kind of translation which might
intrinsically remove information from the input sentence.

###  A.2 Remedies

To avoid the copying behavior, we should constrain the search space of the
intermediate hypotheses to only meaningful sequences in the desired language.
This is rather clear in unsupervised phrase-based MT [Lample et al., 2018b,
Artetxe et al., 2018a], where the search space is limited via the choice of
applicable rules and a monolingual language model of the output language.

For unsupervised NMT, it is more difficult due to the sharing of model
parameters and vocabularies over source and target languages (Section 3.1). A
good initialization (Section 3.3) and denoising autoencoder (Section 3.4) bias
the model towards fluent outputs, but they do not prevent the model from
emitting the input language. The following techniques help to control the
output language in unsupervised NMT:

  * •

Restrict the output vocabulary to the desired language [Liu et al., 2020]

  * •

Use language-specific decoders [Artetxe et al., 2018b, Sen et al., 2019]

  * •

Improve cross-linguality of the encoder, e.g. by adversarial training [Lample
et al., 2018a]

Note that these remedies might also harm the overall training process in
another respect, e.g. inducing less regularization.

###  A.3 Toy Example: Case Conversion

We empirically investigate the input copying problem with a simple task of
case conversion. In this task, the source and target languages consist only of
1-character words in lower- or uppercase respectively. Without any constraints
in back-translation, the unsupervised NMT may learn two optimal solutions to
the reconstruction loss: 1) copy the casing (undesired) or 2) perform a
translation from uppercase to lowercase and vice versa (desired). We trained
1-layer, 64-dimension Transformer models with 100-sentence data on each side,
and measure how often the model fails to converge to the desired solution.

To see the impact of cross-linguality, we compare two initializations where
lower- and uppercase character embeddings are equally or separately
initialized. When they are equal, the model always found the desired solution
(case conversion) in 10 out of 10 trials, whereas the separate variant only
found it in 2 out of 10 trials. On convergence, all experiments achieved zero
reconstruction loss.

These results are in line with the language embedding removal (Section 4.6);
better cross-linguality guides the model to refer to the case indicator on the
target side, which leads to case conversion.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

CLEF 2024: Conference and Labs of the Evaluation Forum, September 9-12, 2024,
Grenoble, France

[ orcid=0009-0003-9818-496X<EMAIL_ADDRESS>url=https://murilogustineli.com, ] [1]

[ orcid=0000-0002-9165-8718<EMAIL_ADDRESS>] [1]

[<EMAIL_ADDRESS>]

[1]Corresponding author.

# Multi-Label Plant Species Classification with Self-Supervised Vision
Transformers

Murilo Gustineli  Anthony Miyaguchi  Ian Stalter  Georgia Institute of
Technology, North Ave NW, Atlanta, GA 30332

(2024)

###### Abstract

We present a transfer learning approach using a self-supervised Vision
Transformer (DINOv2) for the PlantCLEF 2024 competition, focusing on the
multi-label plant species classification. Our method leverages both base and
fine-tuned DINOv2 models to extract generalized feature embeddings. We train
classifiers to predict multiple plant species within a single image using
these rich embeddings. To address the computational challenges of the large-
scale dataset, we employ Spark for distributed data processing, ensuring
efficient memory management and processing across a cluster of workers. Our
data processing pipeline transforms images into grids of tiles, classifying
each tile, and aggregating these predictions into a consolidated set of
probabilities. Our results demonstrate the efficacy of combining transfer
learning with advanced data processing techniques for multi-label image
classification tasks. Our code is available at github.com/dsgt-kaggle-
clef/plantclef-2024.

###### keywords:

Transfer LearningDINOv2Multi-label ClassificationData ProcessingInformation
RetrievalCEUR-WS

##  1 Introduction

The PlantCLEF 2024 challenge [1], part of the LifeCLEF lab [2] under the
Conference and Labs of the Evaluation Forum (CLEF), aims to address the multi-
label classification of plant species in high-resolution plot images. This
task presents unique challenges due to the shift between single-label training
data (images of individual plants) and multi-label test data (images of
vegetation plots) between the 2023 and 2024 editions. The training dataset
comprises approximately 7,800 species and 1.4 million images, totaling 281GiB,
posing substantial computational challenges.

To address these challenges, we propose a transfer learning approach utilizing
a Vision Transformer (ViT) model, specifically DINOv2 [3] for feature
extraction, and a linear classifier trained on the resulting embeddings. We
hypothesize that leveraging the rich ViT embedding space learned through
pretrained self-supervised learning of massive datasets, combined with the
appropriate post-processing steps, will be sufficient to achieve domain-expert
performance on the multi-label classification task without the need to train
models from scratch.

Figure 1:  Overview of our proposed transfer learning method. In the modeling
pipeline, we extract the DCT coefficient and [CLS] token embeddings from the
single-label cropped and resized images using the base or fine-tuned DINOv2
model, and train a classifier on the embeddings. In the inference pipeline,
DINOv2 extracts the [CLS] token embedding from each tile or full-image
accordingly, followed by the trained classifier performing inference to obtain
output species labels.

##  2 Overview

Our approach leverages the embedding space learned by DINOv2 as a generalized
feature representation of images, which are used to train models with higher
bias (i.e., linear classifiers), as illustrated in Figure 1. DINOv2 learns
robust feature representations by processing images as sequences of fixed-size
patch tokens with an additional [CLS] token for classification tasks [4].
These tokens serve as low-dimensional representations of the image patches,
similar to words in a phrase for language models. We train a linear classifier
using the negative log-likelihood (NLL) loss on the single-label DCT
coefficient and [CLS] token embeddings. To address the multi-label
classification problem, we perform inference using full-image and grid-based
methods.

###  2.1 DINOv2 Model Review

DINOv2 is a state-of-the-art vision transformer encoder model, similar to BERT
[5], pretrained without supervision on the LVD-142M dataset, a large
collection of 142 million images. Images are presented to the model as a
sequence of fixed-size patches, which are linearly embedded. A [CLS] token is
added to the beginning of the sequence to facilitate classification tasks, and
absolute position embeddings are included before feeding the sequence into the
Transformer encoder layers.

The DINOv2 architecture comes in different sizes, each with its respective
embedding dimensions: small (S) with 382, base (B) with 768, large (L) with
1024, and giant (g) with 1536. We chose the ViT-B/14 (distilled) base model as
it provides a balance of computational efficiency, performance, and feature
representation suited for our use case, ensuring the model is powerful enough
to extract meaningful features while remaining computationally feasible. The
base model produces embeddings with a fixed size of $\mathbb{R}^{257\times
768}$, regardless of the input image dimensions. This fixed size is due to the
base model’s architecture: each image is divided into 256 fixed-size patches,
and each patch is embedded into a 768-dimensional vector. Additionally, a
[CLS] token, also with a 768-dimensional embedding, is added to the sequence,
resulting in a total of 257 vector embeddings per image [6]. This base model
does not include any fine-tuned heads and learns robust inner representations
of images through pretraining, which can be utilized for downstream tasks.

The organizers provided two DINOv2 models based on the ViT-B/14 (distilled)
architecture, each using a self-supervised learning method [7], trained with
the timm library and hosted on Hugging Face. The first model has its backbone
frozen, fine-tuning only the classification head on new data, leveraging the
robust feature extraction capabilities learned during pretraining. The second
model continues the first training but includes updates to both the backbone
and the classification head, refining the feature representations throughout
the network. We utilize the second fine-tuned model to extract [CLS] token
embeddings from the images, benefiting from its enhanced feature extraction
capabilities.

##  3 Methodology

Figure 2:  End-to-end pipeline of our proposed solution. The downloading
module retrieves the training and test images, along with metadata, and stores
them on Google Cloud Storage (GCS). The preprocessing module converts the
images to binary data, crops and resizes them to $\mathcal{R}^{128\times 128}$
dimensions, and writes them as parquet files to GCS. The modeling module
extracts embeddings using base and fine-tuned DINOv2 models and trains a
linear classifier on the training embeddings. During inference, the trained
classifier makes predictions on the test embeddings, formatting the results
for leaderboard submission.

We conducted experiments on the embedding datasets to maximize our performance
on the public leaderboard. Initially, we establish a baseline with minimal
modifications, performing multi-class classification on the multi-label test
dataset. Subsequently, we introduced more complex inference approaches, such
as grid-based image prediction for multi-label classification.

Our two main approaches were as follows: (1) Extract embeddings using both
base and fine-tuned DINOv2 models from the cropped and resized single-label
image dataset and the multi-label test dataset, train classifiers on the
training embeddings, and perform classification on the test embeddings. (2)
Perform inference using the fine-tuned ViT model with both full-image and
grid-based image prediction approaches. We measure model performance using the
metrics referenced in Section 3.3. The results of our derived datasets and
models are summarized in Table 3 and 4, respectively.

To address the computing and memory constraints of the large-scale training
data, our solution leverages several technologies. We use Google Cloud
Platform (GCP) for computing and storage, Apache Spark for distributed data
processing, Petastorm for distributed data loading, PyTorch Lightning for
model training, and Weights and Biases for experiment tracking. Our primary
compute resources include the n2-standard-4 VM instance (4vCPU, 2 core, 16GiB
memory) and the g2-standard-8 GPU instance (8vCPU, 4 core, 32GiB memory),
scaling up as needed for the dataset’s magnitude.

Apache Spark was crucial to our entire pipeline, especially for preprocessing
and modeling tasks. PyTorch Lightning provides a high-level interface for our
deep learning workflows, enabling efficient model training and hyperparameter
tuning. Our end-to-end pipeline comprises four main components: downloading,
preprocessing, modeling, and inference, illustrated in Figure 2.

###  3.1 Downloading and Preprocessing

We utilize aria2, a lightweight multi-protocol and multi-source command-line
download utility that facilitates fast and reliable downloading of large
datasets. The images and metadata files were downloaded and stored in Google
Cloud Storage for subsequent preprocessing.

The preprocessing phase involves two main steps: converting images to Apache
Parquet format and performing cropping and resizing operations. We concatenate
image data in batches to optimize cloud computation, as reading millions of
images incurs significant network overhead. Columnar data formats like Parquet
efficiently represent data for batch processing, handling both binary and
metadata.

Many images in the dataset are rectangular and inconsistent in resolution. To
improve processing efficiency, we crop and resize all images to ensure the
subject remains in focus, as shown in Figure 3. Each image is cropped to a
square centered at the midpoint and resized to $\mathcal{R}^{128\times 128}$
pixels. This step reduced the dataset size from 281GiB to approximately 15GiB,
achieving more than an order of magnitude reduction. This reduction
facilitated faster embedding extraction using DINOv2 and decreased
computational load during training and inference. We choose a relatively
small, square dimension to allow for constructing a multi-class dataset from
collages of smaller images.

Figure 3:  Comparison of original images with $\mathcal{R}^{128\times 128}$
cropped and resized squared images. The original images have a minimum
resolution of 800 pixels on the longest side, allowing for the use of high-
resolution classification models and potentially improving the prediction of
small plants in large vegetative plots.

We preprocess each image into a grid of tiles and extract DINOv2 features for
each tile (Figure 1). Using the base DINOv2 model, we extract full-size
embeddings $\mathcal{R}^{257\times 768}$ from each image, including DCT
coefficients of the tile tokens and the [CLS] token. With the fine-tuned
DINOv2 model, we focus on extracting the [CLS] token embeddings. We create the
following embedding datasets in Table 1.

Base DINO DCT. The base DINOv2 model extracts embeddings from the cropped and
resized single-label image dataset. The embeddings have dimensions of
$\mathcal{R}^{257\times 768}$. We apply the DCT algorithm for dimensionality
reduction with an 8x8 filter size, resulting in a $\mathcal{R}^{1\times 64}$
tensor. The DCT captures multi-dimensional low-rank structures in frequency
and is known for its compressive properties on datasets, such as JPEG and MP3
for images and audio respectively [8]. It is a data-independent transformation
that runs in $O(n\log n)$ time, with accessible and efficient implementations,
unlike data-dependent transformations like singular value decomposition (SVD)
which requires $O(n^{3})$ eigen-decompositions. The DCT can identify
periodicity within the 2D patch tokens and a low-frequency space useful for
downstream tasks with minimal overhead.

Base DINO [CLS] token. The base DINOv2 model extracts the [CLS] token
embeddings from the cropped and resized dataset. The [CLS] token is a special
token added to the input sequence of the Vision Transformer, aggregating
information from all tiles and providing a generalized feature representation
of the entire image for classification tasks. The resulting [CLS] token
embedding is a tensor of $\mathcal{R}^{1\times 768}$ dimensions.

Fine-tuned DINO. We use the fine-tuned model dinov2-onlyclassifier-then-all
discussed in Section 2.1 to extract [CLS] token embeddings from the cropped
and resized single-label image dataset, resulting in a tensor of shape
$\mathcal{R}^{1\times 768}$.

Table 1:  An overview of the embedding datasets created by utilizing the DINOv2 model to extract embeddings from the cropped and resized single-label image dataset. facebook-dinov2-base refers to the base off-the-shelf DINOv2 model, and dinov2-onlyclassifier-then-all refers to the fine-tuned ViT model provided by the organizers.  DINOv2 Model | Dataset created | Size
---|---|---
facebook-dinov2-base | Base DINO DCT | 403.85MiB
facebook-dinov2-base | Base DINO [CLS] token | 4.09GiB
dinov2-onlyclassifier-then-all (non-frozen) | Fine-tuned DINO | 4.09GiB

###  3.2 Modeling and Inference

We train linear classifiers on both DCT-reduced embeddings and [CLS] token
embeddings using PyTorch Lightning and the negative log-likelihood loss. For
inference, we performed multi-class classification on full-size images to
predict a single plant species per image.

To address multi-label classification, each test image is divided into a grid
of tiles, with embeddings extracted for each tile using the fine-tuned DINOv2
model. We then perform inference on these tile embeddings using argmax and top
$K$ probabilities for prediction aggregation.

Our inference workflow employs a Luigi [9] task, which processes the image
prediction output by extracting species IDs corresponding to the top $L$
probabilities for each image. Duplicates are removed, and order is preserved
by converting to a set and sorting by their original appearance in the logits
list. The unique species IDs are compiled into a structured dataframe, with
each record corresponding to an image, formatted for submission, and written
to a CSV file.

We employ two distinct approaches to image prediction on the test dataset, as
shown in Figure 4: full-image and grid-based image prediction. The test
dataset is not cropped and resized to preserve the high quality of multi-label
images.

Full-Image Prediction. In this approach, the entire test image is processed in
its original dimension. The fine-tuned ViT model evaluates the image and
outputs probabilities for each of the 7806 plant species classes. We then
extract the top 20 probabilities, representing the most likely species
present, and map these probabilities to their corresponding species IDs.

Figure 4:  Comparison of full-image prediction and grid-based image
prediction. The left plot shows a typical vegetative plot from the test set,
where a botanist recorded 8 species: Cardamine resedifolia L., Festuca
airoides Lam., Pilosella breviscapa (DC.) Soják, Lotus alpinus (Ser.)
Schleich. ex Ramond, Poa alpina L., Saxifraga moschata Wulfen, Scorzoneroides
pyrenaica (Gouan) Holub, and Thymus nervosus J.Gay ex Willk. The right plot
illustrates the same image divided into a $3\times 3$ grid, demonstrating the
grid-based approach for species classification by processing each tile
independently.

Grid-based Image Prediction. We segment the image into an $N\times N$ grid of
tiles, resulting in $M$ tiles. Each tile is independently processed using the
fine-tuned ViT model, which outputs probabilities for each species class. We
then select the top $K$ probabilities (default $K=10$) for each tile and map
them to their respective species classes. This produces a nested array of
prediction lists per tile, which we flatten into a single array, further
limiting to the top $L$ probabilities (default $L=5$) from each tile. For
example, a $3\times 3$ grid yields nine tiles, each with ten top
probabilities, selecting the top five logits in each tile, totaling 45 species
IDs and probability mappings.

Let $P_{i,j}$ be the probability of species $i$ in tile $j$. For the argmax
approach, we select:

$\hat{y}_{j}=\underset{i}{\text{argmax}}P_{i,j}$ (1)

For the top $K$ probability approach, we aggregate the top $K$ logits per
tile, and select the top $L$ species across all tiles, where $L=5$, and $M$ is
the total number of tiles:

$\hat{Y}=\text{Top-}L\left(\bigcup_{j=1}^{M}\text{Top-}K(P_{:,j})\right)$ (2)

We generate a consolidated array of predictions whether processing the entire
image or employing the grid-based approach. This array, a collection of
species ID and probability mappings, is sorted in descending order based on
the probability scores. This prioritization helps accurately identify the most
likely species in each image tile.

###  3.3 Class Imbalance

The training dataset exhibits significant class imbalance, with a highly
right-skewed distribution of species. Of the approximately 7,800 species,
nearly half have fewer than 100 images. To mitigate this imbalance, we select
a subset of plant species with at least 100 images for training While this
approach may overlook rare species, we hypothesize that focusing on more
represented species will increase the confidence in species identification.

By selecting species with a minimum of 100 images, we aim to improve the
balance between precision and recall for the included species, directly
impacting the Macro F1 Per Species and Macro F1 Per Plot metrics. While this
may lead to a lower recall for rare species in the test set, it enhances the
precision for the more common species, resulting in higher Macro F1 scores for
those species. For the Micro F1 score, which is more sensitive to the
performance of common species due to the overall count of true positives,
false positives, and false negatives, our approach is likely to result in
higher scores as well.

###  3.4 Evaluation Metrics

The metrics used to evaluate our model are Macro F1 Averaged Per Plot, Macro
F1 Averaged Per Species, and Micro F1 scores.

The F1 score is the harmonic mean of precision and recall, defined as:

$F1=2\cdot\frac{P\cdot R}{P+R}$ (3)

where $P$ and $R$ denote precision and recall, respectively.

The Macro F1 Averaged Per Plot and Macro F1 Averaged Per Species are
calculated as follows:

$\text{Macro F1 Per Plot}=\frac{1}{N}\sum_{i=1}^{N}F1(y_{i},\hat{y}_{i})$ (4)
$\text{Macro F1 Per Species}=\frac{1}{C}\sum_{c=1}^{C}F1(y_{c},\hat{y}_{c})$
(5)

where $N$ is the number of plots, $C$ is the number of species, $y_{i}$ is the
true label for plot $i$, and $\hat{y}_{i}$ is the predicted label for plot
$i$.

The Micro F1 score aggregates the contributions of all classes to compute the
average F1 score:

$\text{Micro
F1}=\frac{2\sum_{c=1}^{C}TP_{c}}{2\sum_{c=1}^{C}TP_{c}+\sum_{c=1}^{C}FP_{c}+\sum_{c=1}^{C}FN_{c}}$
(6)

##  4 Results

We present our best results on the public and private leaderboards. Our best
model utilizes a linear classifier trained on the fine-tuned DINO embeddings
(Table 1) and performs inference using a grid-based approach with argmax logit
per tile. We achieved public scores of 20.77 for Macro F1 Averaged Per Plot,
47.42 for Macro F1 Averaged Per Species, and 19.67 for Micro F1, as shown in
Tables 2 and 3.

Table 2:  A summary of the top 3 best scores in the public and private leaderboards. Our solution achieved the third-highest score in both leaderboards.  Leaderboard | Team name | Rank |  MacroF1 Averaged Per Plot |  MacroF1 Averaged Per Species |  MicroF1
---|---|---|---|---|---
Public | Atlantic | 1 |  29.62 |  51.2 |  30.01
NEUON AI | 2 |  23.01 |  46.2 |  20.84
DS@GT-LifeCLEF (Ours) | 3 |  20.77 |  47.42 |  19.67
Private | Atlantic | 1 |  28.73 |  45.76 |  29.57
NEUON AI | 2 |  21.31 |  34.69 |  20.75
DS@GT-LifeCLEF (Ours) | 3 |  19.04 |  32.4 |  19

The linear classifier trained on the fine-tuned DINOv2 embeddings consistently
outperforms base DINOv2 models across all metrics, demonstrating enhanced
feature representation from additional training (Table 3). The base DINOv2
models show relatively low performance, with the model trained on the [CLS]
token embeddings performing slightly better than the DCT embeddings. The lower
performance of the DCT embeddings is due to their reduced dimensionality
$\mathcal{R}^{1\times 64}$ compared to the [CLS] token embeddings
$\mathcal{R}^{1\times 768}$, leading to information loss. While base [CLS]
token embeddings are more effective than DCT embeddings, both are less
effective than fine-tuned model embeddings.

Table 3:  Summary of the linear classifiers trained on the embedding datasets. All classifiers are single-layer linear networks implemented with PyTorch’s nn.Linear() and trained using the negative log-likelihood loss. The embedding dataset used to train the classifiers and the inference method significantly impact the results, with more capable models extracting richer features that improve classification performance.  Dataset |  Inference method |  MacroF1 Averaged Per Plot |  MacroF1 Averaged Per Species |  MicroF1
---|---|---|---|---
base dino dct |  Multi-class, full-image |  0.28 |  33.33 |  0.1
base dino cls token |  |  2.06 |  31.46 |  1.62
fine-tuned dino |  Multi-class, full-image |  8.47 |  34.39 |  7.04
Multi-class, grid-based, argmax logit per tile |  20.77 |  47.42 |  19.67
Multi-label, grid-based, top 5 species per tile |  20.62 |  42.31 |  19.07

The multi-class, full-image inference method with the linear classifier
trained on fine-tuned DINOv2 embeddings shows significant improvement over the
base models. For a more sophisticated approach, we utilize a grid-based
inference method to improve multi-label classification capability, employing
two strategies: argmax logit per tile and top 5 species per tile. The grid-
based approach with argmax logit per tile achieves the highest scores across
all metrics. The grid-based approach with top 5 species per tile also performs
well with slightly lower performance due to less confident predictions
introducing noise and ambiguity, and the complexity of aggregating multiple
logits, resulting in suboptimal predictions.

For inference using the fine-tuned model, we found that a $3\times 3$ grid
size struck a balance between computational efficiency and species
distribution, having experimented with $2\times 2$ and $5\times 5$ grid sizes
without observing significant improvements, as shown in Table 4. We chose 5
species per tile to maximize the Macro F1 Score Per Species, focusing on the
most represented species in the dataset. However, this approach leaves room
for improvement, particularly for rare species that are underrepresented.

Table 4:  An overview of the inference experiments using the fine-tuned ViT model on the test embedding dataset.  Inference method | Prediction method |  MacroF1 Averaged Per Plot |  MacroF1 Averaged Per Species |  MicroF1
---|---|---|---|---
Multi-label, full-image | Top 5 species |  9.88 |  43.87 |  8.96
Top 20 species |  7.88 |  33.64 |  7.98
Multi-label, grid-based | Top 5 species, 3x3 grid |  17.76 |  44.76 |  14.39
Top 3 species, 3x3 grid |  16.30 |  43.98 |  12.26
Top 5 species, 5x5 grid |  17.71 |  41.84 |  14.36
Top 5 species, 2x2 grid |  15.41 |  42.64 |  12.92

##  5 Discussion

By leveraging transfer learning, we use embeddings from both base and fine-
tuned DINOv2 models to train linear classifiers, addressing the challenge of
single-label training data versus multi-label test data. Our preprocessing
pipeline manages the large-scale dataset by converting images to Parquet
format, reducing dimensionality through cropping and resizing, and decreasing
computational load for extracting embeddings and making inferences.

We demonstrate the effectiveness of using DCT for dimensionality reduction and
the [CLS] token for generalized feature representation. The fine-tuned DINOv2
model provides enhanced feature representations crucial for multi-label
classification. Our grid-based inference approach with argmax and top $K$
probability aggregation enables accurate prediction of multiple species within
each image.

We did not test the base DINOv2 in a grid-based manner or the fine-tuned
DINOv2 with full-image inference due to computational constraints and our
hypothesis that fine-tuning would yield better results when exploiting local
features through tiling. While capturing the top 5 logits per tile provides
broader species identification, it also introduces less confident predictions,
complicating final aggregation. Conversely, the argmax method directly selects
the highest probability, leading to better predictions.

###  5.1 Species Representation in Embedding Spaces

Figure 5:  UMAP projections of the top 5 plant species with the highest number
of images. The fine-tuned model’s embeddings exhibit better spatial
separation, highlighting their effectiveness as feature representations.

In Figure 5, we visualize the spatial separation between species using the
embeddings extracted from the base and fine-tuned ViTs. The UMAP [10]
projections of the top five plant species illustrate the importance of
training a model to create a discriminating feature space and the challenge of
obtaining unique representations without a learning mechanism like the DCT
coefficient.

The image token DCT coefficients exhibit the poorest species separation among
the three approaches. Although effective for dimensionality reduction and
capturing low-frequency information, DCT does not inherently learn
discriminative features for classification tasks. In contrast, the base DINOv2
[CLS] token embeddings show better species separation, as the [CLS] token
aggregates information from all image patches, providing a more comprehensive
representation of the entire image.

The fine-tuned DINOv2 model’s [CLS] embeddings demonstrate the clearest
species separation. The additional training on the specific task data refines
the model’s feature representations, making them more relevant and
discriminative for classification. Fine-tuning effectively bridges the gap
between general-purpose feature extraction and task-specific discriminative
power, resulting in more accurate and reliable species classification, as
shown in Table 3.

##  6 Future Work

One direction for future work is to generate a collage dataset by tiling
individual species that are likely to co-occur. We propose a collaborative
filtering approach to address the multi-label problem, leveraging Locality
Sensitive Hashing (LSH) [11], Approximate Nearest Neighbor Search (ANN) [12],
and Alternating Least Squares (ALS) [13]. LSH reduces dimensions and detects
similarity by hashing similar items into the same “buckets” with high
probability. ANN then efficiently finds the nearest neighbors within this
subset, and ALS generates recommendations by combining scores with geographic
proximity from LSH and ANN, ranking and recommending images based on their
geographic closeness and similarity. This method could generate images for
direct use in multi-label learning.

##  7 Conclusion

We present a robust multi-label plant species classification approach using
self-supervised Vision Transformer (DINOv2) models. Our study highlights the
potential of combining self-supervised learning and transfer learning with
data processing techniques to tackle large-scale biodiversity challenges. We
offer a scalable solution for multi-label image classification using only
single-label training data. Future work could enhance model training and
inference by integrating additional data augmentation techniques,
experimenting with various grid sizes, exploring other dimensionality
reduction methods, and utilizing alternative loss functions such as binary
cross-entropy and asymmetric loss [14]. Additionally, developing more
sophisticated aggregation strategies for multi-label prediction could further
improve classification performance. Our code is available at github.com/dsgt-
kaggle-clef/plantclef-2024.

###### Acknowledgements.

We want to thank the Data Science at Georgia Tech (DS@GT)-CLEF group for cloud
infrastructure and their support, and the organizers of PlantCLEF and LifeCLEF
for hosting the competition.

## References

  * Goëau et al. [2024] H. Goëau, V. Espitalier, P. Bonnet, A. Joly,  Overview of PlantCLEF 2024: Multi-species plant identification in vegetation plot images,  in: Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, 2024.
  * Joly et al. [2024] A. Joly, L. Picek, S. Kahl, H. Goëau, V. Espitalier, C. Botella, B. Deneu, D. Marcos, J. Estopinan, C. Leblanc, T. Larcher, M.
vSulc, M. Hrúz, M. Servajean, J. Matas, et al.,  Overview of lifeclef 2024:
Challenges on species distribution prediction and identification,  in:
International Conference of the Cross-Language Evaluation Forum for European
Languages, Springer, 2024.

  * Oquab et al. [2023] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al.,  Dinov2: Learning robust visual features without supervision,  arXiv preprint arXiv:2304.07193 (2023).
  * Wu et al. [2023] L. Wu, W. Zhang, T. Jiang, W. Yang, X. Jin, W. Zeng,  [cls] token is all you need for zero-shot semantic segmentation,  arXiv preprint arXiv:2304.06212 (2023).
  * Devlin et al. [2018] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,  Bert: Pre-training of deep bidirectional transformers for language understanding,  arXiv preprint arXiv:1810.04805 (2018).
  * Dosovitskiy et al. [2020] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.,  An image is worth 16x16 words: Transformers for image recognition at scale,  arXiv preprint arXiv:2010.11929 (2020).
  * Darcet et al. [2023] T. Darcet, M. Oquab, J. Mairal, P. Bojanowski,  Vision transformers need registers,  arXiv preprint arXiv:2309.16588 (2023).
  * Ahmed et al. [1974] N. Ahmed, T. Natarajan, K. R. Rao,  Discrete cosine transform,  IEEE transactions on Computers 100 (1974) 90–93.
  * Lui [2023] Luigi 2.8.13 documentation, https://luigi.readthedocs.io/en/stable/, 2023. Accessed: 2023-06-07.
  * McInnes et al. [2018] L. McInnes, J. Healy, J. Melville,  Umap: Uniform manifold approximation and projection for dimension reduction,  arXiv preprint arXiv:1802.03426 (2018).
  * Dasgupta et al. [2011] A. Dasgupta, R. Kumar, T. Sarlós,  Fast locality-sensitive hashing,  in: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, 2011, pp. 1073–1081.
  * Li et al. [2019] W. Li, Y. Zhang, Y. Sun, W. Wang, M. Li, W. Zhang, X. Lin,  Approximate nearest neighbor search on high dimensional data—experiments, analyses, and improvement,  IEEE Transactions on Knowledge and Data Engineering 32 (2019) 1475–1488.
  * Takács and Tikk [2012] G. Takács, D. Tikk,  Alternating least squares for personalized ranking,  in: Proceedings of the sixth ACM conference on Recommender systems, 2012, pp. 83–90.
  * Ridnik et al. [2021] T. Ridnik, E. Ben-Baruch, N. Zamir, A. Noy, I. Friedman, M. Protter, L. Zelnik-Manor,  Asymmetric loss for multi-label classification,  in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 82–91.