# Salute the Classic: Revisiting Challenges of Machine Translation in the Age
of Large Language Models

Jianhui Pang1 Fanghua Ye311footnotemark: 1 Longyue Wang2 Dian Yu2
Derek F. Wong1 Shuming Shi2 Zhaopeng Tu2
1University of Macau 2Tencent AI Lab 3University College London
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
{vinnylywang, yudian, shumingshi<EMAIL_ADDRESS>Work was done when Jianhui
Pang and Fanghua Ye were interning at Tencent AI Lab. Longyue Wang is the
Corresponding Author.

###### Abstract

The evolution of Neural Machine Translation (NMT) has been significantly
influenced by six core challenges Koehn and Knowles (2017), which have acted
as benchmarks for progress in this field. This study revisits these
challenges, offering insights into their ongoing relevance in the context of
advanced Large Language Models (LLMs): domain mismatch, amount of parallel
data, rare word prediction, translation of long sentences, attention model as
word alignment, and sub-optimal beam search. Our empirical findings indicate
that LLMs effectively lessen the reliance on parallel data for major languages
in the pretraining phase. Additionally, the LLM-based translation system
significantly enhances the translation of long sentences that contain
approximately 80 words and shows the capability to translate documents of up
to 512 words. However, despite these significant improvements, the challenges
of domain mismatch and prediction of rare words persist. While the challenges
of word alignment and beam search, specifically associated with NMT, may not
apply to LLMs, we identify three new challenges for LLMs in translation tasks:
inference efficiency, translation of low-resource languages in the pretraining
phase, and human-aligned evaluation. The datasets and models are released at
https://github.com/pangjh3/LLM4MT.

Salute the Classic: Revisiting Challenges of Machine Translation in the Age of
Large Language Models

Jianhui Pang1††thanks:  Work was done when Jianhui Pang and Fanghua Ye were
interning at Tencent AI Lab. Fanghua Ye311footnotemark: 1 Longyue
Wang2††thanks:  Longyue Wang is the Corresponding Author. Dian Yu2 Derek F.
Wong1 Shuming Shi2 Zhaopeng Tu2 1University of Macau 2Tencent AI Lab
3University College London<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>{vinnylywang, yudian, shumingshi<EMAIL_ADDRESS>

##  1 Introduction

Figure 1: The six MT challenges are revisited in the context of LLM. Emojis
symbolize the key findings: a ‘smiley face’ for issues largely addressed, a
‘thinking face’ indicating partial alleviation with ongoing concerns, and a
‘sad face’ for persisting unresolved challenges. This can be treated as the
architecture of our paper.

In the Natural Language Processing (NLP) community, one of the most critical
and longstanding tasks is Machine Translation (MT), which aims to convert
human languages from one form to another Koehn (2009); Poibeau (2017). As the
demand for effective translation systems continues to grow, researchers have
been striving to develop models that can tackle the inherent challenges of
this complex task. In this context, the six challenges about MT proposed by
Koehn and Knowles (2017) have been widely recognized and studied by numerous
studies, with many efforts revolving around them Sennrich et al. (2016b);
Freitag and Al-Onaizan (2017); Chu and Wang (2018); Neishi and Yoshinaga
(2019); Garg et al. (2019); Pang et al. (2023).

The emerging Large Language Models (LLMs) have been a significant breakthrough
in NLP Touvron et al. (2023a); OpenAI (2023); Touvron et al. (2023b). LLMs
have demonstrated remarkable capabilities, outperforming traditional
approaches and setting new benchmark performance for various applications such
as machine translation Lyu et al. (2023); Zhu et al. (2023a); Zhang et al.
(2023a); Wang et al. (2023b). LLMs exhibit remarkable translation capabilities
for major languages, owing to their extensive pretraining on vast amounts of
unpaired data. This implies a significant advancement over conventional
techniques. Consequently, recent studies have employed LLMs for translation
tasks Jiao et al. (2023); Alves et al. (2023), achieving remarkable
performance. However, it is unclear that how do LLMs fare against the six
classical challenges? This intriguing question warrants further investigation
and discussion.

To provide valuable insights into the current state of MT research and
identify potential avenues for future advancements, we revisit the six major
challenges about MT by examining the performance of LLMs in addressing these
challenges. Concretely, in our experiments, we adopt the Llama2-7b model as
the translation system and evaluate the performance of the German-to-English
translation task. Note that English and German are high-resource languages in
the Llama2 pretraining data Touvron et al. (2023b), which ensures the model’s
proficiency in these two languages. We find that:

  1. 1.

_Domain Mismatch._ LLMs display a modest degree of advancement in out-of-
domain tasks, confronting terminology mismatch, style discrepancies, and
hallucination issues.

  2. 2.

_Amount of Parallel data._ LLMs diminish the dependence on bilingual data for
high-resource pretraining languages.

  3. 3.

_Rare Word Prediction._ LLMs persistently encounter difficulties in predicting
rare words, resulting in translation omissions.

  4. 4.

_Translation of Long Sentences._ LLMs tackle the challenge of translating
sentences with fewer than $80$ words and exhibit consistent performance at the
document level with approximately 500 words.

  5. 5.

_Word Alignment._ Despite the infeasibility of extracting word alignment via
attention weights in LLM translation systems, it provides valuable insights
into model interpretability.

  6. 6.

_Inference._ The efficiency of inference presents a significant obstacle in
LLMs, resulting in a 100-fold delay compared to traditional neural MT models
in our experiments.

  7. 7.

_Pretraining Resource Imbalance._ The imbalance of pretraining resources leads
to LLMs exhibiting unfaithful performance in translation tasks involving low-
resource languages.

  8. 8.

_Evaluation Issues._ The matter of automatic evaluation has emerged as a
concern, owing to the discrepancy between human and automated appraisals of
LLM translation outputs.

System$\downarrow$ | Law | Medical | IT | Koran | Subtitles
---|---|---|---|---|---
All Data | 61.556.4 | 59.751.4 | 47.541.7 | 18.620.1 | 27.026.8
Law | 62.059.0 | 36.121.7 | 29.613.1 | 12.02.7 | 20.45.4
Medical | 28.5218.3 | 59.3156.5 | 31.211.4 | 11.81.9 | 19.94.3
IT | 30.39.6 | 36.914.9 | 47.443.0 | 11.72.8 | 23.18.6
Koran | 22.30.2 | 28.150.1 | 16.80.2 | 20.315.9 | 10.60.5
Subtitles | 27.135.5 | 33.57.9 | 26.98.5 | 11.66.4 | 28.127.3

Table 1: Translation quality (BLEU scores) of multi-domain German-to-English
translation tasks, where the system is trained on one domain (rows) and tested
on another domain (columns). The red and blue bars refer to the LLM and
Enc2Dec models, respectively. LLMs improve the in-domain and out-of-domain
translation qualities but still suffer from the domain mismatch problem.

|  1: Medical |  2: IT |  3: Subtitles
---|---|---|---
Src |  Die Pipetten müssen in der intakten Folienverpackung aufbewahrt werden. |  Methode die Berechnung der Vorhersage |  Du kannst ihr nicht helfen, außer dass du jetzt den Tatort untersuchst.
1-4 Ref |  Stored pipettes must be kept in the intact foil package. |  Method to calculate forecast |  You can’t do anything but help her by working the crime scene.
All |  Pipettes must be stored in the intact foil pouch. |  Method to calculate the forecast |  You can’t help her except by canvassing the scene.
Law |  The pipettes must be stored in the intact sheet pack. |  Method of calculation of the forecast |  You can’t help her, except by examining the scene of the crime.
1-4 Medical |  The pipettes must be stored in the intact foil pouch. |  Prediction method |  You can’ t help her now, except by examining the scene of the accident.
1-4 IT |  The pipettes must be stored in the intact slide pack. |  Method for calculating the forecast |  You can’t help her, except by examining the crime scene.
1-4 Koran |  The pipettes must be kept in sterile packaging. |  A method to calculate the prediction. |  If you do not, you will not be able to help her.
1-4 Subtitles |  The pipettes must be stored in intact slide wraps. |  Method for calculating the forecast |  You can’t help her, except to process the crime scene.

Table 2: Test examples of German-to-English from three domains, which are
translated by the models trained on domains in the first column. The red,
blue, and green color indicates the terminology mismatch, style mismatch, and
hallucination phenomena, respectively.

##  2 Experimental Setup

###  2.1 Large Language Models

In our experiment, we utilize Llama2-7b, an open-sourced language model with a
staggering $7$ billion parameters. The model is accessible through the
HuggingFace community.111https://huggingface.co/meta-llama/Llama-2-7b During
the fine-tuning phase, the Alpaca dataset is incorporated to preserve
instruction-following capabilities. We utilize two unique approaches with
varying instruction formats for the translation pairs, as delineated below:

  * •

LLM-SFT undergoes supervised fine-tuning (SFT) with bilingual pairs in
conjunction with the Alpaca dataset Taori et al. (2023), wherein the bilingual
pairs are converted into the Alpaca instruction format.

  * •

LLM-CPT-SFT involves continuous pretraining (CPT) of Llama2-7b on concatenated
translation pairs Zhu et al. (2023a), followed by fine-tuning using the Alpaca
dataset.

###  2.2 Small Encoder-to-Decoder Models

Encoder-to-decoder (Enc2Dec) transformer models are widely recognized as the
most prevalent and effective framework for translation tasks Vaswani et al.
(2017). We utilize the Fairseq222https://github.com/facebookresearch/fairseq
toolkit to train Enc2Dec models adhering to the model architectures proposed
by Vaswani et al. (2017). More precisely, the base architecture is utilized
for training models on small bilingual datasets encompassing fewer than $500$k
pairs, while the large architecture is employed for datasets comprising over
$1$M pairs. For each language pair, SUBNMT
toolkit333https://github.com/rsennrich/subword-nmt learns byte-pair encoding
codes and transforms the data into subwords Sennrich et al. (2016a, b).

###  2.3 Data Conditions

We utilize the German-to-English (De2En) parallel data derived from the WMT23
translation tasks.444https://www2.statmt.org/wmt23/translation-task.html Our
approach involves training a translation model using a random sampling method
on the dataset, extracting up to $10$ million (M) translation pairs for each
task. To ensure the integrity of our evaluation process, we employ the latest
publicly available generaltest2023 from WMT23,555https://github.com/wmt-
conference/wmt23-news-systems/tree/master/txt thus circumventing any potential
data leakage.

To evaluate the performance of LLMs on multi-domain tasks, we conduct an
experiment using the multi-domain dataset for German-to-English translation
sourced from OPUS Tiedemann (2012), which is split by Aharoni and Goldberg
(2020).666https://github.com/roeeaharoni/unsupervised-domain-clusters

###  2.4 Training and Evaluation

#### Training

For LLMs, we train each model with a learning rate of $2e$-$5$ and a batch
size of $48$ for $3$ epochs on $32$ A100 GPUs. At every $500$ update step, we
save model checkpoints for evaluation and report the best score. For Enc2Dec
models, to prevent the overfitting problem, we employ the early stopping
strategy that halts the training process when the performance on the
validation set ceases to improve or starts to degrade Yao et al. (2007).
Specifically, we set a patience of $20$, meaning that the training would be
stopped if there was no improvement in the validation performance for the
latest $20$ update steps.

#### Evaluation

We employ two widely acknowledged automatic metrics: BLEU Papineni et al.
(2002), with tokenized and case-sensitive
SacreBLEU777https://github.com/mjpost/sacrebleu Post (2018), and COMET Rei et
al. (2020) derived from the Unbabel/wmt22-comet-da model, which is a
reference-based evaluation approach (COMET-DA).

##  3 The Six Classical Challenges

###  3.1 Domain Mismatch

Domain mismatch has long been a formidable obstacle in the realm of MT Wang et
al. (2020). Given that LLMs undergo extensive pretraining on diverse data, it
leads to an intriguing inquiry: does the wealth of knowledge embedded in LLMs
alleviate the predicament of domain mismatch in translation tasks?  To address
this query, we finetune the LLM using domain-specific parallel data and
subsequently evaluate its performance on both in-domain (ID) and out-of-domain
(OOD) translation tasks. Table 1 displays the obtained results.

The results show that LLM translation systems exhibit strong performance for
in-domain translation tasks. For instance, the LLM translation model achieves
an impressive BLEU score of $62.0$ in the law domain, surpassing Enc2Dec
models by around $3.0$ BLEU scores. However, despite improving the quality of
out-of-domain translations compared to Enc2Dec models, LLMs still manifest a
loss in translation performance when confronted with domain shifting, leading
to a degradation of up to approximately $40.0$ BLEU scores in the Koran-to-Law
OOD translation scenario.

To investigate the impact of domain shifting on the quality of translation, we
provide several translation examples and conduct a qualitative analysis in
Table 1. Accordingly, we reveal three mismatch phenomena as follows:

  * •

Terminology Mismatch. One frequently encountered mistake in translation
systems is the failure to produce accurate domain-specific terminology. For
instance, in the medical domain, the term “Folienverpackung” has been poorly
translated as “slide pack”, “sterile packaging”, and “slide wraps” by the
models finetuned with IT, Korean, and Subtitles bitexts, respectively.

  * •

Style Mismatch. The LLM translation system proves to be unsuccessful in
generating a hypothesis that precisely matches the out-of-domain style. In the
second scenario, the reference in the IT domain is titled “Method to calculate
forecast” while the medical translation system produces “Prediction method”.

  * •

Hallucination. In the third case, the medical translation system incorrectly
renders the term “Tatort” as “accident” instead of “crime”, where “accident”
is a frequent word in the medical domain. This particular translation error
has been identified as an Input-conflicting Hallucination in the research
conducted by Zhang et al. (2023b).

In addition, a practical concern arises about how to effectively manage domain
capacities when the system caters to various domain tasks using a singular
model. Experimental results from the translation system trained on all data
showcase that, while this model performs well in all domain tasks, it falls
slightly short of the domain-specific system.

###  3.2 Amount of Parallel Data

010k50k100k500k1M5M10M$5$$10$$15$$20$$25$$30$$35$$40$$45$$4.6$$10.4$$14.4$$23.3$$26.6$$33.7$$35.9$$38.2$$40.2$$41.1$$41.6$$41.4$$40.9$$40$$38.9$$38.2$$38.1$$38.5$$39$$40.1$$39.2$$40.3$$38.4$Sizes
of Parallel DataBLEUEnc2DecLLM-SFTLLM-CPT-SFT Figure 2: BLEU scores for
German-to-English systems, where “0” in the $x$-axis indicates the model has
solely undergone training on the Alpaca dataset. LLMs eliminate the need to
rely on a large scale of parallel data.

Parallel data, comprising translations from a source language to a target
language, have been recognized as a crucial resource for training conventional
encoder-to-decoder translation systems. With the advent of LLMs, we have come
to understand that even a small corpus of high-quality parallel data can
stimulate the LLM translation abilities Jiao et al. (2023). In this section,
we delve deeper into exploring the impact of varying amounts of parallel data
on LLMs. We experiment with a range of parallel data sizes, ranging from $10$k
to $10$M, as illustrated in Figure 2. To tune LLMs into translation systems,
we employ the two training strategies described in § 2.1. We find that:

  * •

A small amount of parallel data helps improve the translation performance of
LLM systems. It is informed by the LLM-SFT curve in Figure 2 that supervised
finetuning with $10$k parallel data improves $2$ BLEU scores compared to the
model only trained on the Alpaca dataset. In the German-to-English translation
task, the LLM model trained on $100$k parallel data achieves the best
performance with a $41.6$ BLEU score.

  * •

The growing abundance of parallel data results in only a marginal improvement,
and in some cases, can even lead to a performance decline of the LLM
translation system. Contrary to the common belief that more parallel data
enhance translation quality, the LLM model demonstrates opposing outcomes. For
both LLM-SFT and LLM-CPT-SFT, utilizing a large amount of parallel data, such
as 5M and 10M in our configuration, negatively impacts the LLM model and leads
to a decline in performance. Prior research has indicated that the pretraining
stage of LLMs is responsible for the acquisition of nearly all their knowledge
Zhou et al. (2023). During this stage, unintentional exposure to bilingual
signals has been observed Briakou et al. (2023). A large amount of parallel
data can potentially disrupt the acquired knowledge.

  * •

Supervised fine-tuning is a more advantageous approach for leveraging
supplementary parallel data compared to continued pretraining. The performance
curves demonstrate that LLM-SFT consistently achieves superior results, with a
noteworthy increase of up to $2.6$ BLEU scores in the $100$k scenario.

In this section, we examine the effectiveness of LLM translation systems when
exposed to varying amounts of parallel data. Our findings indicate that LLMs
generally do not necessitate extensive training on translation pairs, setting
them apart from conventional Enc2Dec models. This revelation prompts the
research community to consider how to efficiently utilize parallel data for
the enhancement of LLM translation systems, thereby offering a potential
direction for future studies to optimize bilingual knowledge in the pursuit of
improved machine translation performance using LLMs.

###  3.3 Rare Word Prediction

04326412825651299919993999799915999319996399964000+15%25%35%45%55%65%75%1281604326412825651299919993999799915999319996399964000+15%25%35%45%55%65%75%04326412825651299919993999799915999319996399964000+15%25%35%45%55%65%75%04326412825651299919993999799915999319996399964000+10%0%04326412825651299919993999799915999319996399964000+10%0%04326412825651299919993999799915999319996399964000+10%0%

Figure 3: Precision of translation and delete rates by source word frequency.
The light blue and  dark green indicate the LLM and Enc2Dec translation
models. The horizontal axis represents the frequency of source word types in
the test corpus, where axis labels indicate the upper limit of each frequency
range, and the bin width is proportional to the number of word types in that
range. The precision and deletion rates are shown on the upper and lower
vertical axes respectively. LLMs excel at predicting words that appear more
than eight times but perform poorly with rare words. The detailed information
is listed in Table B.1 in the Appendix.

Predicting rare words is a vital task in translation, particularly for proper
nouns, compounds, names, or loanwords that denote specific individuals or
items Luong et al. (2015); Sennrich et al. (2016b); Wu et al. (2016). In line
with previous approaches Koehn and Haddow (2012); Koehn and Knowles (2017), we
assess the precision and deletion rate of rare words in both the LLM and
Enc2Dec translation systems. Our analysis incorporates the finest LLM-SFT
translation system trained on $100$k language pairs, as well as the superior
Enc2Dec model trained on $10$ million language pairs mentioned in Section 3.2.
The experimental results are shown in Figure 3. We find that:

  * •

Overall, the LLM translation model generates more precise and deletes less for
frequent words. Given that Encoder-Decoder (Enc2Dec) models are typically
trained on extensive WMT datasets, it is anticipated that they would excel at
generating precise in-domain words. Contrary to expectations, the results of
the LLM-SFT model challenge this assumption, indicating that the LLM model
consistently achieves higher precision for words with a frequency greater than
8, with an average score of $61.58\%$, compared to $58.46\%$ for the Enc2Dec
model. Additionally, the deletion rates are comparatively lower, with an
average rate of $5.72\%$, while the Enc2Dec model reports $10.82\%$. This
comprehensive evidence substantiates that the LLM-SFT translation model
outperforms the Enc2Dec model in our specific case.

  * •

The LLM translation model performs poorly in predicting infrequent words,
resulting in translation omissions. The LLM-SFT system achieves low precision
for words occurring less than $8$ times and exhibits a high deletion rate.
Specifically, for words in the $(1,2]$ frequency bin, the LLM-SFT informs a
word precision of $20.51\%$ while Enc2Dec achieves $25.00\%$. On the other
hand, the LLM deletion rate for the frequency bin $(2,4]$ is $13.46\%$, which
is $4.76\%$ higher than that of the Enc2Dec model at $8.70\%$. Qualitatively,
for instance, the LLM-SFT fails to translate compound rare words, such as
“blätterlos” (meaning “a plant has no leaves”) and “lotrechtes” (meaning
“perpendicular”), while the Enc2Dec model successfully produces the term
“leaflessly” as a substitute for “blätterlos”.

In general, our results indicate that LLMs exhibit a reliable capacity for
translation accuracy at the word level. However, the prediction of rare words
remains an unresolved issue, presenting a significant challenge in the field.

0102030405060708090100200300400500600$10$$50$$100$$150$1361075545392635222410408116Count
of
Samples0102030405060708090100200300400500600$1$$5$$10$$20$$30$$40$$37.1$$47.1$$45.1$$41.1$$41.2$$45.4$$41.2$$38.8$$39.4$$43.5$$41.8$$39.4$$30.9$$42$$38.4$37.846.342.138.339.339.438.136.336.841.532.416.99.77.00.4Sample
LengthBLEULLM-SFTEnc2Dec Figure 4: BLEU scores for German-to-English MT
systems with varying sample lengths. Sentence-level translation involves
lengths below $90$ words, while document-level translation concerns longer
samples. LLMs improve long-sentence translation and consistently excel in
document-level tasks.

###  3.4 Translation of Long Sentences

The length of source text presents a significant challenge for MT systems, as
longer texts require the accurate capture of contextual information Wang et
al. (2017, 2023b). Considering that LLMs, particularly Llama2, possess an
extended context window of up to $4,096$ maximum input length Touvron et al.
(2023b), we are interested in investigating whether LLM translation systems
can improve the quality of translations for extended sentences.

In this section, we divide the generaltest2023 test set into various lengths
and present model performance across different sentence lengths. Following
previous settings Koehn and Knowles (2017), we categorize the test set into
groups based on sentence length, ranging from 1-9 words to 80-89 words.
Additionally, we include longer groups to evaluate the models’ capabilities in
handling document-level translation tasks, with groups containing 100-199
words to 400-499 words. We also merge the test set with the original
documents, with the last group comprising 500-599 words. The longest document
in our test set consists of 582 words. Figure 4 illustrates the number of
sentences in each group using bars. Our findings are as follows:

  * •

The LLM translation system exhibits proficiency in translating long sentences.
Both LLM-SFT and Enc2Dec models demonstrate stable performance in translating
sentences within $80$ words, as evidenced by the two smooth BLEU curves, with
LLM-SFT outperforming the latter.

  * •

LLMs possess an enhanced ability to translate documents. As depicted in Figure
4, for sentences with more than $100$ words, LLMs maintain high performance
consistently, whereas the Enc2Dec model’s curve abruptly drops to $0.4$ BLEU
score. For documents containing more than 500 words, LLM-SFT achieves a $38.4$
BLEU score, indicating a great improvement.

Our empirical results demonstrate that LLMs excel in translating extended
sentences and entire documents, underscoring their effectiveness as a
promising solution for addressing challenges associated with long-sentence and
document-level translation tasks. Moreover, the evaluation by Wang et al.
(2023b) emphasizes the document-level translation capabilities of LLMs,
providing valuable insights into further opportunities and challenges. Their
work serves as a complementary resource to our analyses in this section.

Figure 5: Average attention weight of target-to-source sentences of the
Llama2-7b-based SFT model. The left column is a target English sentence
“Today, I’m going to the park.”, and the bottom row is the source German
sentence “Heute gehe ich in den Park.”, tokenized by the Llama2 tokenizer. In
general, target tokens attend to the same source tokens within layers. The
detailed alignment information for each layer is listed in Table 6.

###  3.5 Word Alignment

Word alignment involves identifying word pairs with similar semantic
information in a given translation pair and is closely related to machine
translation. Previous research has shown that word alignment can be extracted
from the attention matrix within encoder-to-decoder translation models Garg et
al. (2019); Chen et al. (2020). Moreover, word alignment has been utilized for
interpreting translation models Yang et al. (2020, 2021), revealing that
Enc2Dec models cover more source content in lower layers and attend to the
most relevant source tokens in upper layers.

In this section, we investigate two research questions: 1) _Is it feasible to
extract word alignment from LLM attention weights?_ and 2) _Can word alignment
elucidate the LLM translation process?_ To address these questions, we conduct
a case study using the LLM-SFT model to process the instruction input and
target sentence. Through this analysis, we extract attention weights that
indicate the relationship between source and target words, as depicted in
Figure 5. Our findings are as follows:

  * •

Extracting alignments from LLM attention weights is not a viable option.
Figure 5 displays the average attention weight across $32$ layers in the LLM-
SFT translation model. Results reveal that each target sub-token tends to
attend the same source token, “$\\_$in” in this case, suggesting that the
attention weights do not explicitly provide word alignment information.

  * •

Aggregated attention weights serve as clues for interpreting LLMs. The
observed phenomenon, where target tokens attend the same source tokens,
resonates with the findings of Wang et al. (2023a). They discover that LLMs
tend to aggregate information into one token, and the predicted tokens pay the
most attention to this token during inference, referred to as the anchor. We
hypothesize that “$\\_$in” may act as a translation anchor for target sentence
generation. This insight emphasizes the importance of deepening our
understanding and interpretation of LLM models.

Additionally, to derive word alignment, methods such as prediction difference
Zintgraf et al. (2017); Li et al. (2019) and prompt design present promising
avenues and warrant further investigation. However, the most daunting
challenge resides in interpreting LLMs, for which the insights obtained from
this section provide valuable direction for future research.

BS0.10.51.0$30$$35$$40$TemperatureBLEULLM-SFT-BLEUEnc2Dec-BLEULLM-SFT-
COMETEnc2Dec-
COMETBS0.10.51.0$30$$35$$40$39.228.037.341.635.4TemperatureBLEUBS0.10.51.0$75$$80$$85$83.080.583.583.981.6COMET-
DA Figure 6: BLEU and COMET-DA scores of the German-to-English systems. “BS”
indicates the beam search method with a beam size of $5$.

###  3.6 Inference

In the context of inference, two major issues are inference strategies and
inference efficiency. On the one hand, beam search and sampling are two
commonly used strategies. The beam search strategy involves generating a
sentence by predicting the most promising word within a predefined beam size
Freitag and Al-Onaizan (2017). Sampling is a method that randomly picks the
next word based on the word probability distribution. On the other hand,
inference efficiency becomes a more challenging research problem due to the
abnormal size of LLMs, where many recent works propose to accelerate the
inference process Li et al. (2023); Alizadeh et al. (2023). Previous studies
have investigated the impact of beam size on beam search performance Koehn and
Knowles (2017), and typically, practitioners in the machine translation field
tend to use beam sizes of 4 or 5 Vaswani et al. (2017). In this section, we
first study the performance difference of these two inference strategies, then
we discuss the inference efficiency. In contrast to sampling, beam search is a
heuristic search algorithm that maintains a set of the most promising partial
sequences (or “beams”), which is a natural advantage in terms of BLEU
evaluation. Therefore, apart from BLEU, we attach the COMET-DA evaluation Rei
et al. (2022) for a semantic-level comparison. In our setting, we set the beam
size to $5$ for beam search, then vary the temperature with $0.1$, $0.5$, and
$1.0$ for sampling. We have:

  * •

Sampling exhibits marginally inferior BLEU performance compared to beam
search, while they are less different in COMET-DA. As depicted in Figure 6,
LLM-SFT achieves $41.6$ BLEU and $83.9$ COMET-DA scores when employing beam
search inference. Conversely, using sampling with a temperature of $0.1$, LLM-
SFT attains $39.2$ BLEU and $83.5$ COMET-DA scores. To delve further into
this, we explore the rare word challenge between these two methods. Our
analysis indicates that LLM-SFT, when utilizing sampling with a temperature of
0.1, achieves an accuracy of $55.79\%$ in predicting unknown words, while
exhibiting lower accuracies for frequent words in comparison to beam search.
This suggests that the sampling method’s capacity to investigate a more
diverse set of candidate sequences contributes to its enhanced performance in
addressing rare words.

  * •

Comapred to Enc2Dec, inference latency poses a significant obstacle in the
utilization of LLMs for MT. Regarding the aspect of inference, we find that
LLMs necessitate an average of 30 seconds, whereas Enc2Dec models require an
average of 0.3 seconds, indicating a nearly 100-fold discrepancy. This
substantial difference in inference latency between LLMs and Enc2Dec models
presents a significant challenge in the practical application of LLMs for
machine translation. The longer inference time of LLMs may impede their real-
time deployment in scenarios where fast translation is required.

Nowadays, some research studies begin to explore approaches to reducing the
inference latency of LLMs, such as model compression or hardware acceleration
Dettmers et al. (2022); Agrawal et al. (2023); Frantar and Alistarh (2023);
Dettmers et al. (2023); Lin et al. (2023); Kim et al. (2023). Additionally,
further investigation can be conducted to optimize the performance of sampling
methods in handling rare words, which can potentially enhance the overall
translation quality of LLMs.

010k50k100k500k1M5M10M$5$$10$$15$$20$$25$$30$$35$$40$$45$38.240.241.141.641.440.940.038.913.423.434.424.625.326.127.928,3Sizes
of Parallel DataBLEUEnglish$\leftrightarrow$GermanGerman-to-EnglishEnglish-to-
German

010k50k100k500k1M5M10M$5$$10$$15$$20$$25$$30$$35$$40$$45$20.221.521.720.620.619.119.418.79.520.620.120.321.122.320.322.5Sizes
of Parallel DataBLEUEnglish$\leftrightarrow$ChineseChinese-to-EnglishEnglish-
to-Chinese

010k50k100k500k1M5M10M$5$$10$$15$$20$$25$$30$$35$$40$$45$36.636.336.135.635.035.935.837.218.114.916.416.025.224.024.323.9Sizes
of Parallel DataBLEUEnglish$\leftrightarrow$UkrainianUkrainian-to-
EnglishEnglish-to-Ukrainian

010k50k100k500k1M5M10M$5$$10$$15$$20$$25$$30$$35$$40$17.219.723.825.830.031.824.135.01.54.68.39.013.615.619.88.4Sizes
of Parallel DataBLEUEnglish$\leftrightarrow$HebrewHebrew-to-EnglishEnglish-to-
Hebrew

Figure 7: BLEU scores of bi-directional translation results using Llama2-7b
based translation models across multiple language pairs, where English,
German, Chinese, Ukrainian, and Hebrew are sorted based on the level of
available resources, with English having the highest resources and Hebrew the
least in the pertaining stage.

##  4 New Challenges

Within the realm of LLMs, two compelling challenges emerge. One challenge
pertains to the translation quality for language pairs that are inadequately
represented during the pretraining stage, with a significantly smaller data
volume compared to major languages. In our case, approximately $90\%$ of the
data in Llama2-7b is English Touvron et al. (2023b), resulting in a highly
skewed data distribution. The other challenge lies in the evaluation of
translation quality. The auto-evaluation metrics, such as BLEU or COMET, may
not fully align with human evaluation standards Liu et al. (2023); Mao et al.
(2023), making it difficult to accurately assess the performance of LLMs in
translation tasks.

###  4.1 Pretraining Resource Imbalance

We conduct an experiment using Llama-7b-based translation models on four
translation pairs from the WMT23 tasks: German-to-English, Chinese-to-English,
Ukrainian-to-English, and Hebrew-to-English translations. According to the
technical report of Llama2 Touvron et al. (2023b), the distribution of the
five languages in the dataset of Llama2 is as follows: $89.70\%$ for English,
$0.17\%$ for German, $0.13\%$ for Chinese, $0.07\%$ for Ukrainian, and less
than $0.005\%$ for Hebrew. The results are presented in Figure 7, we have:

  * •

The translation performance is significantly influenced by the amount of
available resources for each language. The X-to-English tasks, with the
highest resource of English, consistently achieve stable translation
performance compared to other directions indicated by a smooth performance
curve (except for Hebrew-to-English) with different amounts of parallel data.
On the other hand, Hebrew, with the least resource in Llama2 pretraining data,
exhibits better translation performance with the increase of parallel data.
This observation highlights the importance of having a diverse and balanced
dataset for pretraining LLMs to ensure equitable performance across languages.

  * •

In translation of low-resource pretraining data languages, a substantial
change in slope is observed as the amount of parallel data increases. The
English-to-$X$ tasks, except English-to-Hebrew, exhibit fluctuating
performance curves as the volume of parallel data increases. Specifically, the
curve for English-to-German experiences a sharp rise from $0$k to $10$k
parallel data, a pattern also observed in English-to-Chinese. Both German and
Chinese account for more than $0.1\%$ of the pretraining data distribution.
The English-to-Ukrainian curve displays a notable inflection point between
$100$k and $500$k, with Ukrainian constituting $0.07\%$ of the data
distribution. In contrast, the English-to-Hebrew curve remains smooth, as
Hebrew does not fall within the languages that comprise more than $0.005\%$ of
the data distribution. These points of significant increase may serve as
indicators of the emergent capabilities of LLMs.

In summary, the above findings demonstrates the crucial role of diverse and
balanced datasets in ensuring equitable translation performance across
languages. As we move forward, addressing the challenges of low-resource
languages and identifying inflection points where LLMs’ capabilities emerge
will be essential to further enhance the effectiveness of translation models
in a wide range of language pairs. Future research should also explore methods
to optimize the utilization of available resources and develop strategies for
improving translation quality in underrepresented languages.

###  4.2 Evaluation Issues

We examine three representative systems on the WMT2023 Discourse-Level
Literary Translation Chinese-English Testset. Commercial Translation System:
Google Translate is utilized,888https://translate.google.com. as it typically
exhibits state-of-the-art translation performance. Commercial LLM Systems:
GPT-4 (8K) API999https://platform.openai.com. is employed for document
translation, renowned for its extensive context modeling capabilities Ouyang
et al. (2022); Wang et al. (2023b). Open-sourced LLM Models: Llama2-7b (4K)
Touvron et al. (2023b) is enhanced for document-level translation using the
200K general-domain document-level training set Zhu et al. (2023b).101010All
experiments were conducted between August 1st and 30th, 2023.

Both automatic and human evaluation methods are used to assess the
aforementioned systems. For automatic evaluation, document-level sacreBLEU
(d-BLEU) is employed as the metric, calculated by matching n-grams across the
entire document Liu et al. (2020); Post (2018). Regarding human evaluation, a
well-defined criterion is proposed by adapting multidimensional quality
metrics (MQM) Lommel et al. (2014); Wang et al. (2023c) to suit the literary
translation context.111111All evaluations are case-sensitive. The results in
Table 3 reveal:

  * •

A moderate negative correlation exists between human and automatic
evaluations. The computed Pearson correlation coefficient of -0.53 suggests a
divergence between the two evaluation methods. This discrepancy implies that
human and automatic evaluations can provide complementary insights into
document-level translation quality. The findings emphasize the importance of
combining both evaluation methods and highlight the potential limitations of
current automatic evaluation approaches in assessing machine translation
systems.

  * •

The need for human-like evaluation methods in LLMs. The use of LLMs for
translation tasks underscores the importance of developing evaluation methods
that can accurately reflect human-like or human-preferred translation quality.
The observed divergence between human and automatic evaluations in our study
indicates that current automatic metrics may not fully capture the nuances
appreciated by human evaluators. This calls for more research to develop and
refine evaluation methods that can better align with human preferences and
expectations in translation, particularly as we continue to scale up the
complexity and capabilities of language models. This human-centered approach
to evaluation will be crucial in ensuring that our translation models are not
only technically proficient but also practically useful and acceptable to end
users.

In summary, our findings underscore the significance of advancing evaluation
methodologies that align with human preferences in the context of LLMs. Future
research should prioritize the development of novel evaluation techniques that
capture the intricacies of human perception and appreciation, ensuring the
practical utility and acceptability of translation systems.

System | MQM | d-BLEU
---|---|---
GPT-4 | 54.81 | 43.7
Llama2-MT | 28.40 | 43.1
Google | 22.66 | 47.3
Table 3: The comparison between human (MQM) vs. automatic (d-BLEU) evaluation
methods over three representative systems, with a color scale to denote the
evaluation ranking. The Pearson correlation coefficient between MQM and d-BLEU
is $-0.53$.

##  5 Conclusions

In this study, we provide an exhaustive analysis of LLMs in addressing the six
classical translation challenges. On the positive side, our findings
demonstrate that LLMs eliminate dependence on parallel data and excel in
translating long sentences, marking a noteworthy advancement over traditional
translation models. However, several challenges persist, including domain
mismatch and the prediction of rare words, which remain formidable tasks. In
addition to these, the efficiency of inference, resource imbalance during the
pre-training phase, and human-like evaluation issues have emerged as new
challenges within the scope of LLMs. Furthermore, enhancing model
interpretability also represents a key area for future exploration.

## Limitations

This study presents several limitations that warrant acknowledgment.
Primarily, our experiments were conducted solely utilizing the Llama2-7b as
the foundational model. This constraint may curtail the generalizability of
our findings to other LLMs. Secondly, our investigation was not directly
tested on more sophisticated models such as GPT-4, which may impede the
applicability of our results in more intricate or advanced model contexts.

Nevertheless, the GPT-4 model is not openly accessible, and the distribution
of training data for various languages remains undisclosed. As a result, this
presents obstacles in conducting diverse experiments within our study. This
lack of transparency concerning the GPT-4 model and its associated training
data could have potentially impacted the comprehensiveness of our experimental
outcomes.

To mitigate these limitations, future research should endeavor to encompass a
broader array of base models and attempt to test approaches on more advanced
models, where feasible. Moreover, efforts should be directed towards obtaining
more information on the training data distribution of advanced models to
augment the robustness of experimental designs.

## References

  * Agrawal et al. (2023) Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee. 2023.  Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills.  _arXiv preprint arXiv:2308.16369_.
  * Aharoni and Goldberg (2020) Roee Aharoni and Yoav Goldberg. 2020.  Unsupervised domain clusters in pretrained language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7747–7763, Online. Association for Computational Linguistics.
  * Alizadeh et al. (2023) Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2023.  Llm in a flash: Efficient large language model inference with limited memory.  _arXiv preprint arXiv:2312.11514_.
  * Alves et al. (2023) Duarte Alves, Nuno Guerreiro, João Alves, José Pombal, Ricardo Rei, José de Souza, Pierre Colombo, and Andre Martins. 2023.  Steering large language models for machine translation with finetuning and in-context learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 11127–11148, Singapore. Association for Computational Linguistics.
  * Briakou et al. (2023) Eleftheria Briakou, Colin Cherry, and George Foster. 2023.  Searching for needles in a haystack: On the role of incidental bilingualism in PaLM’s translation capability.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 9432–9452, Toronto, Canada. Association for Computational Linguistics.
  * Chen et al. (2020) Yun Chen, Yang Liu, Guanhua Chen, Xin Jiang, and Qun Liu. 2020.  Accurate word alignment induction from neural machine translation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 566–576, Online. Association for Computational Linguistics.
  * Chu and Wang (2018) Chenhui Chu and Rui Wang. 2018.  A survey of domain adaptation for neural machine translation.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1304–1319, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022.  Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.  _Advances in Neural Information Processing Systems_ , 35:30318–30332.
  * Dettmers et al. (2023) Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. 2023.  Spqr: A sparse-quantized representation for near-lossless llm weight compression.  _arXiv preprint arXiv:2306.03078_.
  * Frantar and Alistarh (2023) Elias Frantar and Dan Alistarh. 2023.  Sparsegpt: Massive language models can be accurately pruned in one-shot.  _arXiv preprint arXiv:2301.00774_.
  * Freitag and Al-Onaizan (2017) Markus Freitag and Yaser Al-Onaizan. 2017.  Beam search strategies for neural machine translation.  In _Proceedings of the First Workshop on Neural Machine Translation_ , pages 56–60, Vancouver. Association for Computational Linguistics.
  * Garg et al. (2019) Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik. 2019.  Jointly learning to align and translate with transformer models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4453–4462, Hong Kong, China. Association for Computational Linguistics.
  * Jiao et al. (2023) Wenxiang Jiao, Jen-tse Huang, Wenxuan Wang, Zhiwei He, Tian Liang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023.  ParroT: Translating during chat using large language models tuned with human translation and feedback.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 15009–15020, Singapore. Association for Computational Linguistics.
  * Kim et al. (2023) Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. 2023.  Squeezellm: Dense-and-sparse quantization.  _arXiv preprint arXiv:2306.07629_.
  * Koehn (2009) Philipp Koehn. 2009.  _Statistical machine translation_.  Cambridge University Press.
  * Koehn and Haddow (2012) Philipp Koehn and Barry Haddow. 2012.  Interpolated backoff for factored translation models.  In _Proceedings of the 10th Conference of the Association for Machine Translation in the Americas: Research Papers_ , San Diego, California, USA. Association for Machine Translation in the Americas.
  * Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017.  Six challenges for neural machine translation.  In _Proceedings of the First Workshop on Neural Machine Translation_ , pages 28–39, Vancouver. Association for Computational Linguistics.
  * Li et al. (2019) Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019.  On the word alignment from neural machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1293–1303, Florence, Italy. Association for Computational Linguistics.
  * Li et al. (2023) Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023.  Compressing context to enhance inference efficiency of large language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6342–6353, Singapore. Association for Computational Linguistics.
  * Lin et al. (2023) Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. 2023.  Awq: Activation-aware weight quantization for llm compression and acceleration.  _arXiv preprint arXiv:2306.00978_.
  * Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023.  G-eval: NLG evaluation using gpt-4 with better human alignment.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 2511–2522, Singapore. Association for Computational Linguistics.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Lommel et al. (2014) Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014.  Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics.  _Tradumàtica_ , (12):0455–463.
  * Luong et al. (2015) Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015.  Addressing the rare word problem in neural machine translation.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 11–19, Beijing, China. Association for Computational Linguistics.
  * Lyu et al. (2023) Chenyang Lyu, Jitao Xu, and Longyue Wang. 2023.  New trends in machine translation using large language models: Case examples with chatgpt.  _arXiv preprint arXiv:2305.01181_.
  * Mao et al. (2023) Rui Mao, Guanyi Chen, Xulang Zhang, Frank Guerin, and Erik Cambria. 2023.  Gpteval: A survey on assessments of chatgpt and gpt-4.  _arXiv preprint arXiv:2308.12488_.
  * Neishi and Yoshinaga (2019) Masato Neishi and Naoki Yoshinaga. 2019.  On the relation between position information and sentence length in neural machine translation.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 328–338, Hong Kong, China. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Pang et al. (2023) Jianhui Pang, Derek Fai Wong, Dayiheng Liu, Jun Xie, Baosong Yang, Yu Wan, and Lidia Sam Chao. 2023.  Rethinking the exploitation of monolingual data for low-resource neural machine translation.  _Computational Linguistics_ , pages 1–22.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Poibeau (2017) Thierry Poibeau. 2017.  _Machine translation_.  MIT Press.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Rei et al. (2022) Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022.  COMET-22: Unbabel-IST 2022 submission for the metrics shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a.  Edinburgh neural machine translation systems for WMT 16.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 371–376, Berlin, Germany. Association for Computational Linguistics.
  * Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2023a) Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023a.  Label words are anchors: An information flow perspective for understanding in-context learning.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 9840–9855, Singapore. Association for Computational Linguistics.
  * Wang et al. (2023b) Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023b.  Document-level machine translation with large language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 16646–16661, Singapore. Association for Computational Linguistics.
  * Wang et al. (2023c) Longyue Wang, Zhaopeng Tu, Yan Gu, Siyou Liu, Dian Yu, Qingsong Ma, Chenyang Lyu, Liting Zhou, Chao-Hong Liu, Yufeng Ma, Weiyu Chen, Yvette Graham, Bonnie Webber, Philipp Koehn, Andy Way, Yulin Yuan, and Shuming Shi. 2023c.  Findings of the WMT 2023 shared task on discourse-level literary translation: A fresh orb in the cosmos of LLMs.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 55–67, Singapore. Association for Computational Linguistics.
  * Wang et al. (2017) Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017.  Exploiting cross-sentence context for neural machine translation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2826–2831, Copenhagen, Denmark. Association for Computational Linguistics.
  * Wang et al. (2020) Yong Wang, Longyue Wang, Shuming Shi, Victor OK Li, and Zhaopeng Tu. 2020.  Go from the general to the particular: Multi-domain translation with domain transformation networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 9233–9241.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, and et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.
  * Yang et al. (2021) Baosong Yang, Longyue Wang, Derek F Wong, Shuming Shi, and Zhaopeng Tu. 2021.  Context-aware self-attention networks for natural language processing.  _Neurocomputing_ , 458:157–169.
  * Yang et al. (2020) Yilin Yang, Longyue Wang, Shuming Shi, Prasad Tadepalli, Stefan Lee, and Zhaopeng Tu. 2020.  On the sub-layer functionalities of transformer decoder.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4799–4811, Online. Association for Computational Linguistics.
  * Yao et al. (2007) Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. 2007.  On early stopping in gradient descent learning.  _Constructive Approximation_ , 26:289–315.
  * Zhang et al. (2023a) Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a.  Prompting large language model for machine translation: A case study.  _arXiv preprint arXiv:2301.07069_.
  * Zhang et al. (2023b) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023b.  Siren’s song in the ai ocean: A survey on hallucination in large language models.  _arXiv preprint arXiv:2309.01219_.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023.  Lima: Less is more for alignment.  _arXiv preprint arXiv:2305.11206_.
  * Zhu et al. (2023a) Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang. 2023a.  Multilingual machine translation with large language models: Empirical results and analysis.  _arXiv preprint arXiv:2304.04675_.
  * Zhu et al. (2023b) Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023b.  Extrapolating large language models to non-english by aligning languages.  _arXiv preprint arXiv:2308.04948_.
  * Zintgraf et al. (2017) Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. 2017.  Visualizing deep neural network decisions: Prediction difference analysis.  _arXiv preprint arXiv:1702.04595_.

##  Appendix A Detailed Experimental Settings

###  A.1 Instruction Formats

Type | Data Format
---|---
SFT |  Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
2-2 |  ### Instruction: Translate the following sentences from German to English.
2-2 |  ### Input: Haben Sie einen Blick auf andere Restaurants in der Nähe mit ähnlicher Küche geworfen?
2-2 |  ### Response: Have you looked at other nearby restaurants with similar cuisine?
CPT |  [German]: Haben Sie einen Blick auf andere Restaurants in der Nähe mit ähnlicher Küche geworfen? [English]: Have you looked at other nearby restaurants with similar cuisine?

Table 4: An illustration of two data formats utilized for training LLMs, where
SFT represents supervised finetuning, and CPT denotes continual pertaining.

Table A.1 showcases two unique data formats that are employed for the purpose
of constructing bilingual pairs, which are essential for training Language-to-
Language Models (LLMs). The first format adheres to the design principles of
the Alpaca dataset, as described in the literature Taori et al. (2023). This
format has been widely used and accepted in various research studies and
applications. On the other hand, the second format involves concatenating two
pairs of text, each representing a different language. To distinguish between
the languages in this concatenated format, language tags are incorporated Zhu
et al. (2023a).

###  A.2 Rare Word Precision

Algorithm 1 Word Precision and Delete Rates

1:The source-to-target alignments of reference pairs $\mathcal{A}_{ref}$, the
source-to-target alignments of hypothesis pairs $\mathcal{A}_{hyp}$, with
identical sorted index;

2:Source word precision $\mathcal{P}$ and deletion rates $\mathcal{D}$;

3:$\mathcal{P},\mathcal{D}\leftarrow$ Initialized as an empty list;

4:for each a, b in ($\mathcal{A}_{ref}$, $\mathcal{A}_{hyp}$) do

5: for each source word $w_{s}$ in $\mathcal{A}_{ref}$ do

6: $\mathcal{P}_{w}=0$, $\mathcal{D}_{w}=0$;

7: List of target words $W^{t}_{ref}=$ $\mathcal{A}_{ref}$($w_{s}$);

8: List of target words $W^{t}_{hyp}=$ $\mathcal{A}_{hyp}$($w_{s}$);

9: for each $w$ in $W^{t}_{ref}$ do

10: $c^{t}_{ref}\leftarrow$ Count($w$) in $W^{t}_{ref}$;

11: if $w$ in $W^{t}_{hyp}$ then

12: $c^{t}_{hyp}\leftarrow$ Count($w$) in $W^{t}_{hyp}$;

13: $minc=$ Min($c^{t}_{ref}$,$c^{t}_{hyp}$);

14: $maxc=$ Max($c^{t}_{ref}$,$c^{t}_{hyp}$)

15: $\mathcal{P}_{w}+=$ Min($\frac{minc}{maxc}$,$1$);

16: else

17: $\mathcal{D}_{w}+=$ 1;

18: end if

19: end for

20: end for

21: $\mathcal{P}_{w}=$ Mean($\mathcal{P}_{w}$);

22: $\mathcal{D}_{w}=$ Mean($\mathcal{D}_{w}$);

23:end for

24:return $P_{w}$,$D_{w}$.

First, we calculate the source word type frequency using the entire WMT23
bilingual corpus, which comprises $295,805,439$ translation pairs.
Subsequently, we concatenate the translation results with the corpus and
employ _FastAlign_ 121212https://github.com/clab/fast_align to determine the
alignment information. In this manner, we adhere to the methodologies employed
in previous studies Koehn and Haddow (2012); Koehn and Knowles (2017) to
compute the word precision and deletion ratio for each source word, as
demonstrated in Algorithm 1.

##  Appendix B Further Experimental Resutls

###  B.1 Rare Word

Table B.1 displays the detailed word precision and deletion rates in relation
to source word type frequencies. Generally, within the scope of automatic
evaluation, a robust translation system consistently exhibits higher word
precision and lower word deletion rates for frequent words. Nonetheless, for
infrequent words, its performance in generating rare words might be
suboptimal. For instance, in Figure 6, LLM-SFT-S0.1 attains lower BLEU and
COMET-DA scores compared to LLM-SFT, yet LLM-SFT-S0.1 demonstrates better
precision for words appearing more than $2$ times, and Enc2Dec exhibits
improved precision for words within the frequency bin of [4,8].

Freq. Bin | Counts | LLM-SFT | LLM-SFT-S0.1 | Enc2Dec
---|---|---|---|---
ACC(%) | Delete(%) | ACC(%) | Delete(%) | ACC(%) | Delete(%)
0 | 80 | 50.65 | 6.57 | 55.79 | 4.12 | 50.82 | 8.25
1 | 22 | 52.05 | 3.85 | 53.82 | 11.54 | 50.86 | 14.00
2 | 11 | 20.51 | 0.00 | 21.37 | 0.00 | 25.00 | 0.00
4 | 28 | 34.55 | 13.46 | 37.50 | 13.46 | 38.55 | 8.70
8 | 22 | 44.47 | 8.70 | 42.78 | 4.35 | 46.89 | 9.09
16 | 46 | 52.06 | 6.25 | 54.70 | 4.17 | 54.34 | 4.44
32 | 62 | 50.61 | 6.76 | 50.62 | 6.76 | 47.55 | 16.86
64 | 82 | 58.07 | 3.98 | 55.63 | 10.23 | 51.38 | 18.28
128 | 139 | 60.38 | 6.00 | 58.82 | 6.83 | 57.73 | 10.81
256 | 165 | 60.20 | 8.20 | 62.26 | 5.46 | 54.88 | 14.66
512 | 183 | 69.25 | 6.22 | 67.25 | 6.78 | 60.46 | 14.15
999 | 244 | 68.51 | 5.62 | 67.52 | 6.78 | 63.71 | 12.48
1999 | 295 | 69.36 | 6.40 | 68.61 | 7.07 | 64.88 | 10.38
3999 | 380 | 74.44 | 4.83 | 74.46 | 5.92 | 70.78 | 10.13
7999 | 425 | 80.34 | 3.56 | 79.01 | 4.30 | 75.14 | 8.80
15999 | 554 | 80.65 | 4.91 | 79.33 | 5.34 | 73.27 | 9.80
31999 | 609 | 80.17 | 4.88 | 79.73 | 5.18 | 75.35 | 11.98
63999 | 680 | 81.73 | 3.97 | 80.41 | 5.12 | 75.13 | 10.66
64000+ | 2658 | 81.93 | 4.59 | 80.13 | 6.04 | 73.95 | 12.09
1-8 Average | 352 | 61.58 | 5.72 | 61.57 | 6.29 | 58.46 | 10.82

Table 5: Precision of translation and delete rates concerning source word
types with varying frequencies. “Freq. Bin” and “Counts” indicate the
frequency upper bounds and the count of each source type word. “ACC” and
“Delete” indicate the precision and deletion rates of word prediction. LLM-
SFT-S0.1 refers to conducting inference at a temperature setting of $0.1$.

###  B.2 Alignment

Layer | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
T |  $\\_$in | . | ute | . | . |  $\\_$in |  $\\_$den |  $\\_$ich |  $\\_$in |  $\\_$in | . | . |  $\\_$den |  $\\_$den |  $\\_$den |  $\\_$den
od | he |  $\\_$Park | ute |  $\\_$den | ute |  $\\_$in |  $\\_$den |  $\\_$den |  $\\_$in |  $\\_$in | . | . | he |  $\\_$den |  $\\_$den |  $\\_$den
ay | he | . | ute |  $\\_$den | ute |  $\\_$in |  $\\_$den |  $\\_$Park |  $\\_$in |  $\\_$in | . | ute | he |  $\\_$den |  $\\_$den |  $\\_$den
, | . | . | ute |  $\\_$den | ute |  $\\_$den |  $\\_$den |  $\\_$Park | He |  $\\_$in | . | ute |  $\\_$in |  $\\_$den |  $\\_$den |  $\\_$den
$\\_$I |  $\\_$in |  $\\_$den | . |  $\\_$den | ute |  $\\_$in |  $\\_$den | He |  $\\_$in |  $\\_$in | . | ute |  $\\_$den | . |  $\\_$den |  $\\_$den
’ |  $\\_$in | He | ute |  $\\_$den | ute | He |  $\\_$den | He | He |  $\\_$in | He | ute | . | . |  $\\_$den |  $\\_$den
m |  $\\_$in |  $\\_$Park | ute |  $\\_$den | he |  $\\_$in |  $\\_$den |  $\\_$ge |  $\\_$in |  $\\_$in |  $\\_$in | ute | he | . |  $\\_$den |  $\\_$den
$\\_$going |  $\\_$in | . | . |  $\\_$den | he |  $\\_$in |  $\\_$den |  $\\_$ge |  $\\_$in |  $\\_$in |  $\\_$in | ute |  $\\_$Park | . | . |  $\\_$den
$\\_$to |  $\\_$in |  $\\_$den | ute |  $\\_$den | he |  $\\_$in |  $\\_$den |  $\\_$ich |  $\\_$in |  $\\_$in | . | ute |  $\\_$den | . | . |  $\\_$den
$\\_$the |  $\\_$in |  $\\_$in | ute |  $\\_$den | . |  $\\_$den |  $\\_$den |  $\\_$den |  $\\_$in |  $\\_$in | . | ute |  $\\_$den | . | . |  $\\_$den
$\\_$park |  $\\_$in | . | . |  $\\_$den | . |  $\\_$in |  $\\_$den |  $\\_$Park |  $\\_$in |  $\\_$in |  $\\_$in | ute |  $\\_$Park | . | . |  $\\_$den
. |  $\\_$in |  $\\_$in | . | He | . | . |  $\\_$den |  $\\_$Park | He | . | He | ute | . | . | . |  $\\_$den
Layer | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32
T |  $\\_$in |  $\\_$in |  $\\_$Park | . |  $\\_$in |  $\\_$den |  $\\_$in | . |  $\\_$in | . |  $\\_$in | . |  $\\_$in |  $\\_$ich |  $\\_$den |  $\\_$in
od |  $\\_$den |  $\\_$in |  $\\_$in | . |  $\\_$in |  $\\_$den |  $\\_$in | . |  $\\_$in | . |  $\\_$in | . |  $\\_$in |  $\\_$ich | . |  $\\_$in
ay |  $\\_$den |  $\\_$in |  $\\_$in |  $\\_$den |  $\\_$in |  $\\_$den |  $\\_$in | ute | . | . |  $\\_$in | . |  $\\_$in |  $\\_$ich | . |  $\\_$in
, |  $\\_$in |  $\\_$in |  $\\_$in |  $\\_$den |  $\\_$in |  $\\_$den |  $\\_$in | . | . | . |  $\\_$in | . |  $\\_$in | . | He |  $\\_$in
$\\_$I |  $\\_$ich |  $\\_$in |  $\\_$in |  $\\_$den |  $\\_$in |  $\\_$den |  $\\_$in | . | . | . |  $\\_$in | . |  $\\_$in |  $\\_$ich |  $\\_$in |  $\\_$in
’ |  $\\_$ich |  $\\_$in |  $\\_$ich | . |  $\\_$in |  $\\_$den |  $\\_$in | He | ute |  $\\_$Park |  $\\_$in | . |  $\\_$in |  $\\_$ich |  $\\_$den |  $\\_$in
m |  $\\_$den |  $\\_$in |  $\\_$den | . |  $\\_$in |  $\\_$den |  $\\_$in | . | he | . |  $\\_$in | . |  $\\_$in |  $\\_$ich |  $\\_$den |  $\\_$in
$\\_$going |  $\\_$ich |  $\\_$in |  $\\_$den | . |  $\\_$in |  $\\_$den |  $\\_$in | . | he | . |  $\\_$in | . |  $\\_$in |  $\\_$ich |  $\\_$den |  $\\_$in
$\\_$to |  $\\_$in |  $\\_$in |  $\\_$in | . |  $\\_$in |  $\\_$den |  $\\_$in | . | he | . |  $\\_$in | . |  $\\_$in |  $\\_$ich |  $\\_$den |  $\\_$in
$\\_$the |  $\\_$in |  $\\_$in |  $\\_$in | . |  $\\_$in |  $\\_$den |  $\\_$in | . |  $\\_$in | . |  $\\_$in | . |  $\\_$in | . | He |  $\\_$in
$\\_$park |  $\\_$ich |  $\\_$in |  $\\_$den |  $\\_$den |  $\\_$in |  $\\_$den |  $\\_$in | . |  $\\_$Park | . |  $\\_$in |  $\\_$in |  $\\_$in |  $\\_$ich | He |  $\\_$in
. |  $\\_$in |  $\\_$Park | . |  $\\_$den |  $\\_$in |  $\\_$den | . | . | . | . |  $\\_$Park |  $\\_$in | . |  $\\_$den | He |  $\\_$in

Table 6: Word alignment induced from target-to-source attention weights for
each layer in LLM-SFT German-to-English translation model. The translation
model is supervised-finetuned on $100$k parallel data. The left row is a
target English sentence “Today, I’m going to the park.”, and its source German
sentence is “Heute gehe ich in den Park.”. Both sentences are tokenized by the
Llama2 tokenizer. We observe that target tokens tend to attend the same source
tokens within each layer.

Table 6 showcases the word alignment outcomes derived from each layer of the
LLM-SFT translation model, which was trained on $100$k German-to-English
datasets. The findings indicate a tendency for target tokens to align with the
same source tokens throughout all layers.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

CHR 2023: Computational Humanities Research Conference, December 6 – 8, 2023,
Paris, France

[<EMAIL_ADDRESS>] [1] [<EMAIL_ADDRESS>orcid=0000-0001-8177-4582, url=http://www.melvinwevers.nl, ] [1] [1] [
<EMAIL_ADDRESS>orcid=0000-0002-5145-3603,
url=https://nanne.github.io/, ] [1]

[1]Corresponding author. [1]These authors contributed equally.

# Blind Dates: Examining the Expression of Temporality in Historical
Photographs

Alexandra Barancová  Melvin Wevers  Nanne van Noord  University of Amsterdam

(2023)

###### Abstract

This paper explores the capacity of computer vision models to discern temporal
information in visual content, focusing specifically on historical
photographs. We investigate the dating of images using OpenCLIP, an open-
source implementation of CLIP, a multi-modal language and vision model. Our
experiment consists of three steps: zero-shot classification, fine-tuning, and
analysis of visual content. We use the De Boer Scene Detection dataset,
containing 39,866 gray-scale historical press photographs from 1950 to 1999.
The results show that zero-shot classification is relatively ineffective for
image dating, with a bias towards predicting dates in the past. Fine-tuning
OpenCLIP with a logistic classifier improves performance and eliminates the
bias. Additionally, our analysis reveals that images featuring buses, cars,
cats, dogs, and people are more accurately dated, suggesting the presence of
temporal markers. The study highlights the potential of machine learning
models like OpenCLIP in dating images and emphasizes the importance of fine-
tuning for accurate temporal analysis. Future research should explore the
application of these findings to color photographs and diverse datasets.

###### keywords:

Image dating Computer vision Temporal analysis Historical photographs OpenCLIP

##  1 Introduction

Time plays a crucial role in shaping our understanding and interpretation of
the world around us. Our perception of duration, the sequence of our memories,
and the authority with which historical records organize the past all
contribute to our lived experiences and memory. This perception extends to our
interpretation of visual content, where an image’s materiality, content, and
style can convey critical temporal information. Despite its significance, this
aspect of image understanding remains underexplored in artificial intelligence
research [1, 2].

AI models, typically trained on limited data periods, possess a narrow
understanding of temporality due to their lack of ‘awareness’ of historical
variations. Although efforts have been made to integrate historical data into
language models [3] and even encode time explicitly [4], these methods
primarily focus on text, leaving visual content interpretation largely
uncharted territory. In this paper, we experiment with the task of ‘dating’
images, predicting when an image was taken based on its visual content.111All
code used for this study is available on GitHub: https://github.com/CANAL-
amsterdam/dating-images/ We examine how different image aspects influence a
multimodal AI model’s predictive accuracy, uncover structural biases in pre-
trained computer vision models, and explore their effects on predictions. Our
research aims to extend our understanding of the visual representation of time
and its influence on image interpretation. This experiment is situated within
a broader goal of developing more temporally-aware computer vision and
multimodal models. For this, cross-pollination between AI and humanities
scholarship on cultural heritage, archiving, and temporality will be needed;
as [5] show, interdisciplinary work in this area has been limited, yet it has
the potential to be mutually beneficial.

##  2 Background

The challenge of automating dating has been addressed across a variety of
historical objects, spanning from photographs [1] and artworks [6, 7] to
archaeological sites [8, 9]. With the increasing digitization of historical
documents, many of which lack publication dates, computational methods have
been employed to estimate their creation dates, primarily analyzing writing
styles [10], focusing on both the text and the visual content of the writing.
The automatic dating of historical photographs offers substantial value to
archives and museums, but also domains like temporal forensic analysis, where
dating can serve as evidence. Forensic applications typically concentrate on
an image’s material aspects, using techniques that identify specific camera
models or devices [11, 12]. While such methods may be overly meticulous for
large-scale dating of historical images, they underscore the importance of
material information in establishing an image’s capture date.

Beyond material aspects, others show that low-level image features like RGB
color derivatives and color angles carry temporal information. Models trained
on these features often surpass human accuracy in dating photographs [1, 13].
Research in this domain has seen the adoption of neural networks for dating
photographs, treating it as an ordinal [14], regression [15], classification
[16, 17, 18], or retrieval task [19]. Studies have also started to pay
attention to image content, emphasizing the connection between time and visual
elements or semantic cues. Research indicates that temporal cues can be
derived from human appearance features, such as clothing, hairstyles, and
glasses [16, 17], or even from architectural elements like windows to estimate
the age of buildings [18]. A recent study on a family photo album dataset
found that the accuracy of the model used for the dating task improved as the
number of faces and/or people in a photograph increased [20] – this suggests
that certain high-level image features and visual elements may carry more
temporal information than others. Finally, [21] recently turned to generative
approaches to synthesize portrait images for specific decades between 1880 and
the present day to distinguish visual markers for these periods.

As we transition our focus from photograph materiality to content, an
essential challenge arises: deciphering how models interpret higher-level
input features to predict dates. This exploration aims to yield deeper
insights into the ways in which temporality is encoded in visual content and
how we can enhance computer vision systems’ ability to interpret cultural
artifacts.

##  3 Image Dating

We use OpenCLIP [22], the open source implementation of CLIP [23], to predict
when a photograph was taken. CLIP is a multi-modal language and vision model
that has been shown to have a strong zero-shot capability on diverse vision
tasks [23] and to outperform a number of domain-specific models on various
vision and language tasks following task-specific fine-tuning [24]. Our
interest lies in understanding how visual features, particularly objects, are
leveraged for dating purposes, while also evaluating the model’s aptitude for
the dating task. Among the various models that have been used for dating
images, we have not yet seen experiments with large models that have zero-shot
capabilities. Experimenting with the potential of such models is interesting
due to the lesser need for training data, their broader generalizability and
the possibility to examine a multimodal, based on textual and visual data,
perspective on tasks like dating.

### Data

Collections of press photographs are available with relatively reliable dates,
making them well-suited for examining the visual representation of time. For
this experiment, we have chosen to use the De Boer Scene Detection dataset,
which contains 52,160 digitized historical press photographs from the De Boer
newspaper agency spanning from 1945 to 2005.222The dataset is available at
https://zenodo.org/record/7137452 The images are scanned and cropped photo
negatives, the vast majority of which are gray-scale. [25] The dataset
contains relatively mundane photographs, rather than iconic ones, as well as a
wide variety of different scenes ranging from sporting events to landscapes
[26]; this makes it an interesting case for exploring the visual elements that
carry temporal information. Besides the exact year, each image has a label
describing the depicted scene. We excluded images lacking date information and
those taken before 1950 and after 1999, resulting in 39,866 photographs. These
cut-off points were chosen to include data spanning complete decades in our
analysis. Figure 1 illustrates the dataset distribution per year. We split the
dataset in a train and test set using stratified sampling based on the year,
with the aim of reducing uneven distributions across the splits to prevent
biases, in 80% and 20% respectively.

We structure our experiment in three steps: examining zero-shot classification
capabilities, fine-tuning the model, and assessing the impact of visual
content on the model’s dating ability.

Figure 1: Distribution graph of the De Boer Scene Detection dataset by year.
We utilized the data marked in blue in our experiments.

### Zero-shot Classification.

To investigate to what extent OpenCLIP can be used for dating we apply zero-
shot classification to the test set to predict the photograph’s date. This
process uses the prompt ‘a photograph from the year $x$’ where $x$ ranges from
1950 up to 2000. We employ Mean Absolute Error (MAE) for performance
evaluation, following [13, 15]. We find an MAE of 15.8, which indicates
relatively poor performance considering both the 50 year range of our dataset
and the comparison to results others have demonstrated on the dating task —
[15] and [19], for example, attained MAE values of up to 7.12 and 7.48
respectively in their experiments with the Date Estimation in the Wild (DEW)
dataset that covers the period 1930-1999. Additionally, [15] showed that human
participants had an MAE of 10.9 on the DEW dataset and as such were on average
about 11 years off in their predictions. Comparatively, the almost 16 years
that the zero-shot model achieves is quite poor.

The error rate distribution reveals a preference in the zero-shot model for
predicting dates earlier than the actual date. Suspecting a correlation with
the images’ gray-scale nature, we tested the zero-shot classification with a
colorized version of our dataset,333We colorized all the photographs using
DeOldify https://github.com/jantic/DeOldify resulting in the error
distribution leaning slightly more toward the future (see Figure 2(a)).444A
KS-test (KS-statistic: 0.49, p-value: 0.0) supports the difference between the
distributions. Figure 3 shows two sample images for which colorization had a
large effect on decreasing date prediction error. One depicts an outdoor view
of a church, the other a group of people in formal wear. Both images show a
large prediction error in the gray-scale variant, albeit in different
directions, i.e. overestimate or underestimating the actual date. We find that
colorizing the images improves the overall zero-shot capabilities of OpenCLIP,
however, with an MAE of 13.2 it is still relatively ineffective for the dating
task.

(a) Zero-shot classification

(b) Fine-tuned classification

Figure 2: Histograms showing the error distribution for the zero-shot and
fine-tuned classification.

Figure 3: Examples of original (left) and colorized (right) photographs from
the De Boer Scene Detection dataset for which colorization had the largest
impact on decreasing error for zero-shot classification. Colorization
decreased the error from -37 to 0 for the top image (actual year: 1999,
prediction original: 1952, prediction colorized: 1999), and from 37 to 2 for
the bottom image (actual year: 1952, prediction original: 1999, prediction
colorized: 1954).

### Fine-tuned Classifier.

To overcome the zero-shot limitations we explore whether fine-tuning OpenCLIP
improves performance on the dating task and eliminates the bias found in the
initial experiment. To this end, we train a logistic classifier using the
OpenCLIP image embeddings. Training a logistic classifier also allows us to
focus solely on the temporal information in the visual content, removing
possible confounding temporal bias in the model from text, through prompting.
When using textual prompts for zero-shot classification, the words used might
be better suited for specific historical periods, thereby introducing a bias
in the images corresponding to this prompt. Fine-tuning reduced the error and
the bias, with the MAE being 6.65 for the classifier trained and evaluated on
the original gray-scale images and 6.79 for the colorized images. The bias
between gray-scale and colorized images found in the zero-shot approach
disappears after fine-tuning (Figure 2), displaying a more normal error
distribution.555Also supported by KS-test (K-S statistic: 0.00, p-value:
0.92).

### Content Analysis.

Upon training a model to predict dates, we investigated the content of the
images to determine whether specific visual features improved or hindered the
predictions. An initial analysis using the available scene labels proved to be
inconclusive as there were large differences in the within-scene error rates,
as well as a large variety of visual features represented in individual
scenes. We opted to examine the visual features at the object level, using
Detectron2 outputs [27]. For the detection task, we used the 80 default
objects, as defined in COCO, default ROI threshold (0.5). Next, we only
selected detection with a confidence threshold above 0.8 and types that
appeared more than 200 times in the entire data set, in order to shorten the
long tail. The confidence threshold was output by Detectron2 per image.
Finally, we picked 12 classes representing modes of transport and living
beings to focus this experiment on.666‘bicycle’, ‘boat’, ‘bus’, ‘car’,
‘motorcycle’, ‘train’, ‘truck’, ‘bird’, ‘cat’, ‘dog’, ‘horse’, ‘person’. Our
motivation in picking these classes was to reduce the granularity of the
available categories so as to identify larger trends; classes like ‘tie‘ for
example, might be closely related to ‘person’, essentially functioning as a
sub-class thereof. An additional motivation for excluding some of the MS COCO
object classes, is that they did not all suit the context and/or time span of
our dataset, especially technology like ‘laptop’, ‘cell phone’ or ‘microwave’.

A Bayesian regression analysis was conducted to measure the effect of object
presence and absence on the error rate.777The analysis was performed using the
Python library Bambi and the NumPyro nuts sampler. The regression model was
defined as follows:

$prediction\\_error=1+object\\_presence$

Where $prediction\\_error$ is the outcome variable, $object\\_presence$ is the
binary prediction variable indicating each object’s presence. To model the
distribution, we assumed a negative binomial. We model the errors as counts,
where the event is the counts of predictions with a specific error.888Since
the variance and mean are not equal a zero-inflated Poisson was not warranted.
See the GitHub for more information on the models.

Figure 4(a) shows that for modes of transport, the presence of ‘bicycle’,
‘boat’, ‘motorcycle’, and ‘train’ increase the absolute error, whereas ‘bus’
and ‘car’ decrease the absolute error. We hypothesize that these vehicles are
more prone to exterior changes in this period. Of the animals, we see that
‘bird’ and ‘horse’ increase the error and ‘cat’ decreases the error. Our
hypothesis here is that cats might be depicted more often together with humans
and in interior environments, which may include more temporal markers.
Finally, we see that having a ‘person’ in an image has the strongest effect,
decreasing the MAE from approximately 7.2 to 5.5 (Figure 4(b)), indicating
that depictions of people convey visual cues about time. These results and
hypotheses need further examination, which we intend to undertake in future
work.

(a) Estimated effects of objects on absolute error. HDI .95, meaning that
there is a 95% chance that the true value lies within this range.

(b) Posterior predictions for the class ‘person’. A person in the image
reduces the absolute error from 7.2 to 5.5.

Figure 4: Output of the regression model. The MAE values are based on the
entire dataset, without taking into account random effects, which results in
higher reported MAE values. However, for this analysis the change in MAE is of
primary interest.

##  4 Conclusion and Discussion

Our exploration with OpenCLIP to date historical press photographs yielded
several findings.

### Ineffectiveness of Zero-shot Classification.

Our first finding is that the zero-shot classification capability of OpenCLIP
does not perform well in dating images. The model demonstrated a distinct bias
towards predicting earlier dates, which we attribute to the gray-scale nature
of our images. This suggests that OpenCLIP may have learned to associate gray-
scale with older photographs, as [28] also concluded in exploring the concept
of history in foundation models including CLIP. We attempted to counteract
this bias by colorizing the images, which mildly improved the model’s accuracy
and shifted the bias towards predicting more recent dates. However, despite
these adjustments, the efficacy of zero-shot classification for this task
remained limited.

### Improvement through Fine-tuning.

Our second finding is that fine-tuning OpenCLIP using a logistic classifier
significantly enhances the model’s performance. The fine-tuned model
effectively eliminates the bias towards past dates seen in the zero-shot
approach and offers comparable accuracy levels for both gray-scale and
colorized images. This indicates that the presence of color in images becomes
less significant for dating them when the model is trained to focus on visual
content. Future research could look into generating captions rather than
labels or scenes to provide a more enriched context for each image.

### Objects as Temporal Markers.

Our third finding, coming from the post-hoc regression analysis, is that the
presence of people in images generally leads to more accurate date
predictions. This echoes the findings of [20]. We posit that this could be
attributed to the time-dependent markers humans tend to carry, like fashion
and hairstyles, as has previously been shown in studies on yearbook portraits
by [17] and [16]. Moreover, we see that the presence of animals often kept as
house pets also reduces the error, we hypothesize that this might be due them
being photographed indoors or in proximity to humans, which might carry more
temporal markers than animals, such as horses or birds that are captured in
nature. Finally, certain modes of transportation increase the error rate while
others decrease it. We need to explore to what extent this is related to
innovations that lead to visual changes over time. All in, further
investigation is necessary to validate these hypotheses; considering our
findings on the influential role of human figures in the images, it would be
worthwhile to explore datasets containing fewer human figures – in our dataset
only 8,674 of the 39,866 photographs contained no people.999This is based on
outputs from Detectron2 at an object confidence threshold of 0.8. This could
shed light on whether the presence of humans is generally advantageous for
image dating, or if this is a specific characteristic of our dataset or a
manifestation of model bias.

In conclusion, this study deepens our understanding of how computer vision
models interpret and extract temporal information from historical visual
material. It highlights the potential of OpenCLIP for image dating tasks. It
also underscores the importance of model fine-tuning to counter biases. Future
work could test our findings’ generalizability to color images and datasets
from various periods and geographical regions. Such work can be a means to
identifying and using temporal information in visual material better, with the
aim of creating more temporally-aware computer vision and multimodal models.
To this end, we see case studies engaging with specific computer vision/
temporal tasks like image dating, as important steps in testing what works in
terms of both models and data.

## References

  * Palermo et al. [2012] F. Palermo, J. Hays, A. A. Efros,  Dating Historical Color Images,  in: Computer Vision – ECCV 2012. ECCV 2012. Lecture Notes in Computer Science, volume vol 7577, Springer, Florence, Italy, 2012, pp. 499–512. URL: https://doi.org/10.1007/978-3-642-33783-3_36. doi:10.1007/978-3-642-33783-3_36.
  * van Noord et al. [2022] N. van Noord, M. Wevers, T. Blanke, J. Noordegraaf, M. Worring, An Analytics of Culture: Modeling Subjectivity, Scalability, Contextuality, and Temporality, 2022. URL: https://arxiv.org/abs/2211.07460. doi:10.48550/ARXIV.2211.07460.
  * Manjavacas and Fonteyn [2021] E. Manjavacas, L. Fonteyn,  Macberth: Development and evaluation of a historically pre-trained language model for english (1450-1950),  in: Proceedings of the Workshop on Natural Language Processing for Digital Humanities, 2021, pp. 23–36.
  * Rosin et al. [2022] G. D. Rosin, I. Guy, K. Radinsky,  Time masking for temporal language models,  in: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, 2022, pp. 833–841.
  * Fiorucci et al. [2020] M. Fiorucci, M. Khoroshiltseva, M. Pontil, A. Traviglia, A. Del Bue, S. James,  Machine Learning for Cultural Heritage: A Survey,  Pattern Recognition Letters 133 (2020) 102–108. URL: https://www.sciencedirect.com/science/article/pii/S0167865520300532. doi:10.1016/j.patrec.2020.02.017.
  * Mensink and Van Gemert [2014] T. Mensink, J. Van Gemert,  The rijksmuseum challenge: Museum-centered visual recognition,  in: Proceedings of International Conference on Multimedia Retrieval, 2014, p. 451–454.
  * Khan and van Noord [2021] S. Khan, N. van Noord,  Stylistic multi-task analysis of ukiyo-e woodblock prints,  in: British Machine Vision Conference, 2021, p. 14.
  * Klassen et al. [2018] S. Klassen, J. Weed, D. Evans,  Semi-supervised machine learning approaches for predicting the chronology of archaeological sites: A case study of temples from medieval Angkor, Cambodia,  PLOS ONE 13 (2018) e0205649. URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0205649. doi:10.1371/journal.pone.0205649.
  * Toner and Han [2019] G. Toner, X. Han, Language and chronology: text dating by machine learning, Brill, 2019.
  * Hamid et al. [2019] A. Hamid, M. Bibi, M. Moetesum, I. Siddiqi,  Deep Learning Based Approach for Historical Manuscript Dating,  in: 2019 International Conference on Document Analysis and Recognition (ICDAR), IEEE, Sydney, Australia, 2019, pp. 967–972. URL: https://ieeexplore.ieee.org/document/8978080/. doi:10.1109/ICDAR.2019.00159.
  * Mao et al. [2009] J. Mao, O. Bulan, G. Sharma, S. Datta,  Device temporal forensics: An information theoretic approach,  in: 2009 16th IEEE International Conference on Image Processing (ICIP), 2009, pp. 1501–1504. doi:10.1109/ICIP.2009.5414612, iSSN: 2381-8549.
  * Ahmed et al. [2020] F. Ahmed, F. Khelifi, A. Lawgaly, A. Bouridane,  Temporal Image Forensic Analysis for Picture Dating with Deep Learning,  in: 2020 International Conference on Computing, Electronics & Communications Engineering (iCCECE), IEEE, Southend, United Kingdom, 2020, pp. 109–114. URL: https://ieeexplore.ieee.org/document/9231160/. doi:10.1109/iCCECE49321.2020.9231160.
  * Fernando et al. [2014] B. Fernando, D. Muselet, R. Khan, T. Tuytelaars,  Color features for dating historical color images,  in: 2014 IEEE International Conference on Image Processing (ICIP), IEEE, Paris, France, 2014, pp. 2589–2593. URL: http://ieeexplore.ieee.org/document/7025524/. doi:10.1109/ICIP.2014.7025524.
  * Liu et al. [2017] Y. Liu, A. W.-K. Kong, C. K. Goh,  Deep Ordinal Regression based on Data Relationship for Small Datasets,  IJCAI (2017) 2372–2378. URL: https://personal.ntu.edu.sg/AdamsKong/publication/ijcai17_final.pdf.
  * Müller et al. [2017] E. Müller, M. Springstein, R. Ewerth,  “When Was This Picture Taken?” – Image Date Estimation in the Wild,  in: J. M. Jose, C. Hauff, I. S. Altıngovde, D. Song, D. Albakour, S. Watt, J. Tait (Eds.), Advances in Information Retrieval, volume 10193, Springer International Publishing, Cham, 2017, pp. 619–625. URL: http://link.springer.com/10.1007/978-3-319-56608-5_57. doi:10.1007/978-3-319-56608-5_57.
  * Salem et al. [2016] T. Salem, S. Workman, M. Zhai, N. Jacobs,  Analyzing human appearance as a cue for dating images,  in: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, Lake Placid, NY, USA, 2016, pp. 1–8. URL: http://ieeexplore.ieee.org/document/7477678/. doi:10.1109/WACV.2016.7477678.
  * Ginosar et al. [2017] S. Ginosar, K. Rakelly, S. M. Sachs, B. Yin, C. Lee, P. Krähenbühl, A. A. Efros,  A Century of Portraits: A Visual Historical Record of American High School Yearbooks,  IEEE Transactions on Computational Imaging 3 (2017) 421 – 431. doi:10.1109/TCI.2017.2699865.
  * Sun et al. [2022] M. Sun, F. Zhang, F. Duarte, C. Ratti,  Understanding architecture age and style through deep learning,  Cities 128 (2022) 103787. URL: https://linkinghub.elsevier.com/retrieve/pii/S0264275122002268. doi:10.1016/j.cities.2022.103787.
  * Molina et al. [2021] A. Molina, P. Riba, L. Gomez, O. Ramos-Terrades, J. Lladós,  Date Estimation in the Wild of Scanned Historical Photos: An Image Retrieval Approach,  in: J. Lladós, D. Lopresti, S. Uchida (Eds.), Document Analysis and Recognition – ICDAR 2021, volume 12822, Springer International Publishing, Cham, 2021, pp. 306–320. URL: https://link.springer.com/10.1007/978-3-030-86331-9_20. doi:10.1007/978-3-030-86331-9_20.
  * Stacchio et al. [2022] L. Stacchio, A. Angeli, G. Lisanti, D. Calanca, G. Marfia,  IMAGO: A family photo album dataset for a socio-historical analysis of the twentieth century,  ACM Transactions on Multimedia Computing, Communications, and Applications 18 (2022) 1–23. URL: http://arxiv.org/abs/2012.01955. doi:10.1145/3507918, arXiv:2012.01955 [cs].
  * Chen et al. [2023] E. M. Chen, J. Sun, A. Khandelwal, D. Lischinski, N. Snavely, H. Averbuch-Elor,  What’s in a Decade? Transforming Faces Through Time,  Computer Graphics Forum 42 (2023) 281–291. URL: https://onlinelibrary.wiley.com/doi/10.1111/cgf.14761. doi:10.1111/cgf.14761.
  * Ilharco et al. [2021] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, L. Schmidt, OpenCLIP, 2021. URL: https://github.com/mlfoundations/open_clip. doi:10.5281/zenodo.5143773.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever,  Learning Transferable Visual Models From Natural Language Supervision,  in: Proceedings of the 38th International Conference on Machine Learning, PMLR, 2021, pp. 8748–8763. URL: https://proceedings.mlr.press/v139/radford21a.html.
  * Shen et al. [2021] S. Shen, L. H. Li, H. Tan, M. Bansal, A. Rohrbach, K.-W. Chang, Z. Yao, K. Keutzer, How Much Can CLIP Benefit Vision-and-Language Tasks?, 2021. URL: https://arxiv.org/abs/2107.06383. doi:10.48550/ARXIV.2107.06383.
  * Wevers et al. [2022] M. Wevers, N. Vriend, A. de Bruin,  What to do with 2.000.000 Historical Press Photos? The Challenges and Opportunities of Applying a Scene Detection Algorithm to a Digitised Press Photo Collection,  TMG Journal for Media History 25 (2022) 1–24. URL: http://www.tmgonline.nl/articles/10.18146/tmg.815/.
  * Wevers [2021] M. Wevers,  Scene Detection in De Boer Historical Photo Collection:,  in: Proceedings of the 13th International Conference on Agents and Artificial Intelligence, SCITEPRESS - Science and Technology Publications, Vienna, Austria, 2021, pp. 601–610. URL: https://www.scitepress.org/DigitalLibrary/Link.aspx?doi=10.5220/0010288206010610. doi:10.5220/0010288206010610.
  * Wu et al. [2019] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, R. Girshick, Detectron2, https://github.com/facebookresearch/detectron2, 2019.
  * Offert [2023] F. Offert,  On the Concept of History (in Foundation Models),  IMAGE 37 (2023) 121–134. URL: https://doi.org/10.1453/1614-0885-1-2023-15462. doi:10.1453/1614-0885-1-2023-15462.