#  DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents

Fuxiao Liu
University of Maryland, College Park
<EMAIL_ADDRESS>Hao Tan
Adobe Research
<EMAIL_ADDRESS>Chris Tensmeyer
Adobe Research
<EMAIL_ADDRESS>
###### Abstract

Vision-language pretraining models have achieved great success in supporting
multimedia applications by understanding the alignments between images and
text. While existing vision-language pretraining models primarily focus on
understanding single image associated with a single piece of text, they often
ignore the alignment at the intra-document level, consisting of multiple
sentences with multiple images. In this work, we propose DocumentCLIP, a
salience-aware contrastive learning framework to enforce vision-language
pretraining models to comprehend the interaction between images and longer
text within documents. Our model is beneficial for the real-world multimodal
document understanding like news article, magazines, product descriptions,
which contain linguistically and visually richer content. To the best of our
knowledge, we are the first to explore multimodal intra-document links by
contrastive learning. In addition, we collect a large Wikipedia dataset for
pretraining, which provides various topics and structures. Experiments show
DocumentCLIP not only outperforms the state-of-the-art baselines in the
supervised setting, but also achieves the best zero-shot performance in the
wild after human evaluation. Our code is available at
https://github.com/FuxiaoLiu/DocumentCLIP.

##  1 Introduction

Images and text act as natural complements on the modern web. A number of
works [28, 13, 16, 10, 1, 9, 23, 44] have been proposed to interpreting an
image with the corresponding short text such as a caption or question.
However, real-world media such as news articles, Wikipedia pages, magazines,
product descriptions consist of multiple sentences with multiple images.
Algorithms that identify document-internal connections between specific images
and specific unit of text could have the long-term promise. For example, alt-
text for vision-impaired users could be produced automatically via intra-
document retrieval. Additionally, by designing a user interface hinting the
readers that there is a figure associated with a part of the text upon users’s
action, it can not only help users to understand the whole document, but also
make it easy and comfortable to read on smart phones, where the font and
button is small.

Figure 1: Illustration of our task. Given an article, our model extracts the
most relevant text to the image/caption pair. (Left): The document is from our
pretraining Wikipedia dataset, which has the section to image/caption pair
links as the groundtruth label. We leverage this weak supervision to explore
multimodal intra-document interaction by contrastive learning. (Right): An
example is from Open Textbook dataset in the wild. The highlights are the most
relevant sentences extracted by our model.

This intra-document setting introduces several challenges compared to
conventional cross-model retrieval task with a single sentence and caption
[26, 17, 27, 46]. More specifically, since the images are to be incorporated
in the same article, their contents should be consistent according to the
theme of the article, thus making disambiguation more difficult than in the
usual one-image/one-sentence case (Figure 1). Additionally, a sentence in the
longer document may correspond to multiple images or no related images.
Another challenge is that it requires considering longer texts with
linguistically richer content and the relations between them, like the
structure and layout information. The previous relevant works [28, 13, 38, 11,
14, 40] trained to align short literal text descriptions of images with the
image failed to encode the structure features. Document structure analysis
models [5, 4, 37] process pages one by one, but the figures may relate to text
located on other pages.

To address the aforementioned challenges, we propose DocumentCLIP, a simple
yet effective pretraining learning method of intra-document vision and
language understanding tasks. Inspired by the BERT model [24], where input
textual information is mainly represented by text embeddings and position
embeddings, DocumentCLIP further adds two types of input embeddings: (1) an
entity embedding that indicates which section sharing more common entities.
This is because the images/caption pairs can be the literal visualization of
entities mentioned in document sentences. (2) a section position embedding
that denotes the relative position within a document to encode the layout and
structure information. In addition, we introduce the salient sentence
extraction strategy to extract the important information from the long
section. To effectively transfer event knowledge across modalities, we employ
early-fusion method to aggregate images and caption by [12] before the layout
transformer in Figure 2. Unlike the state-of-the-art vision-language pre-
training model CLIP [28], we optimize a salience-aware contrastive learning
objective between intra-document components: images, captions, section text.
In order to train robust representations capable of discriminating subtle
differences between different sentences in the same document, we not only have
normal negatives by replacing groundtruth section text with others from the
same document, but also propose to generate hard negatives by manipulating the
image and caption pairs by switching one of them.

In this work, we pretrain and evaluate DocumentCLIP on the Wikipedia articles,
which have natural correspondences between the text sections and image/caption
pairs in its HTML sources and describe various topics. Also, our evaluations
focus on zero-shot settings, since it is crucial to understand real-world
documents with various topics and sources. Examples are shown in Figure 1.
Experiment results show that DocumentCLIP significantly outperforms baselines
and meet users’ requirement in the wild111https://open.umn.edu/opentextbooks/.
The contributions of this paper are summarized as follows:

  * •

We introduce a novel contrastive learning framework with layout information,
multimodal interaction, novel section encoding strategy, salience-aware
contrastive loss and hard negative samples.

  * •

We collect a large visual article dataset from Wikipedia with various topics,
containing 66k articles, 320k images/caption pairs.

  * •

Our model outperforms baselines on the intra-document understanding task in
both the supervised and zero-shot settings after human evaluation.

Figure 2: Overview of DocumentCLIP in the training phrase. In this example,
$t_{00},t_{01},t_{02}$ are the Top 3 sentences selected by the Salient
Sentence Extraction from the image/caption pair in yellow. $x^{ic}_{12}$ is
the negative query from the image/caption pair in green to generated hard
negative pairs. $x^{ic}_{11}$ is the positive query to construct positive
pairs and normal negative pairs.

##  2 DocumentCLIP

###  2.1 Problem Statement

We assume that the structure of the document has been parsed, so that we can
apriori identify figure images and their captions (if present), and segment
the main body text into a sequence of sections, paragraphs, and sentences. The
granularity of the associated text (section vs sentence) can be set based on
the use case and/or training data available. In our prototype, we primarily
used Wikipedia articles, whose source markup associates figure-captions with
sections. Additionally, by leveraging this weak supervision with the section
ground truth label, our model is often able to retrieve the most relevant
sentence to the associated image and caption.

We define relevance based on the following criteria. (1) The images are the
literal visualization of the entities mentioned in the document. (2) Text
explicitly refers to the images. (3) The images provide evidences for the text
claims.

###  2.2 Model Architecture

We build a multi-modal Transformer architecture as the backbone of
DocumentCLIP, which takes text, images, and layout information as input to
establish deep cross-modal interactions. We describe the strategies to encode
sections of the document in 2.2.1. In 2.2.2, we present how DocumentCLIP leans
layout information and feature extraction. Finally, we discuss the objective
functions and model inference in 2.2.3. Detailed descriptions of the model are
illustrated in Figure 2.

####  2.2.1 Section Encoding Strategy

Although BERT-like models become the state-of-the-art techniques on several
challenging NLP tasks, they usually leverage only text information without
image features. CLIP [28] models achieves great success in image-text
retrieval task. However, it’s only able to handle short text (less than 77),
which is much shorter than the average length (195.5) of the sections within
Wikipedia documents. Therefore, we experiment different sentence sampling
strategies and propose the novel Salient Sentences Extraction method to learn
the section representation.

First Sentence. It is evident that the first sentence plays an important role
and always presents the central meaning of the sections. Therefore, we can
embed the first sentence to represent the section.

Weighted Average. Apart from the first sentence, the associated image can also
provides evidences for other sentences in a section. Thus, we train a one-
layer transformer to aggregate all the important sentence embeddings in a
section.

All Text Concat. Compared with the Weighted Average method, we regard the text
in a section as one sentence and encode them together by a text transformer.

Salient Sentences. Motivated by the success of transferring image-text pre-
training knowledge, we first adopt CLIP [28] to compute a scaled dot product
similarity score $Sim_{si}$ between each sentence in a section and images.
Then we also do the operation $Sim_{sc}$ for sentences and captions. After
that, we utilize the average value of the $Sim_{si}$ and $Sim_{sc}$ to
represent the correlation between the sentence and image/caption pairs.
Finally, we pick the top $K$ salient sentences in each section with the
highest scores as the candidates. In Figure 2, $K$ is 3 for simplicity and
candidates are generated from image/caption in yellow.

####  2.2.2 Document Layout Learning

In this section, we extend to jointly model text, image and layout information
in the DocumentCLIP framework.

Text Embedding. Following the common practice, we use lower-cased byte pair
encoding (BPE) [32] to tokenize the sentences in documents. Then, we add [CLS]
at the beginning of the sequence and [SEP] at the end. Extra [PAD] tokens are
appended to the end so that the final sequence’s length is exactly the maximum
sequence length L. We directly adopt CLIP for initialization to extend its
ability. The final token embedding is the sum of token embeddings and
positional embeddings. Formally, the sentence embedding is computed by the
text transformer [29] and output of [CLS] token which $T^{sec}_{j}$is utilized
as sentence representations in documents. $T^{sec}_{j}$ $\in$
$[t_{j0},...,t_{jK}]$, where $j$ $\in$ (0 $\leq$ j < N) means the section
index in a document. $[t_{j0},...,t_{jK}]$ are the top $K$ candidates selected
by salient sentence extraction in the j-th section. Similarly, the caption
representation is $T^{cap}_{i}$, where $i$ $\in$ (0 $\leq$ i < M) and M means
the caption number in a document.

| KVQA | TQA | Visual Genome | GoodNews | Visual News | Wikipedia (Ours)
---|---|---|---|---|---|---
Avg Unit Length | $84.8$ | $920.8$ | $263.9$ | $451.0$ | $773.0$ | $195.5$
Avg Doc Length | $11.4$ | $106.0$ | $5.3$ | $18.0$ | $18.8$ | $3346.6$
Avg Sent Length | $11.4$ | $12.2$ | $5.3$ | $18.0$ | $18.8$ | $22.3$
Avg Sent Num per Unit | $1.0$ | $7.7$ | $1.0$ | $1.0$ | $1.0$ | $8.2$
Avg Img Num per Doc | $-$ | $3.0$ | $-$ | $1.8$ | $1.7$ | $4.8$
Table 1: Statistics in terms of the document(Doc), unit and images(Img). Doc
donates document and Sent donates sentence. Unit means the section in our
dataset.

Visual Embedding. In order to obtain image embedding, vision transformer is
adopted to extract N non-overlapping image patches and perform linear
projection to map every patch into 1D token. With injection of positional
embedding and extra [CLS] token, the sequence of tokens are fed into image
transformer layers, to model the correlation of each patch, where each layer
ls comprises of Multi-Head Self-Attention, layer normalization, and Multi-
layer Perception. Then, output [CLS] token embedding is used to represent the
image feature $V^{img}_{k}$, where $k$ $\in$ (0 $\leq$ k < M) and M means the
image number in a document.

Layout Embedding. The layout embedding layer is to understand the global
context of the document. Thus, The final text embedding of captions
$l^{cap}_{i}$ is the sum of three embeddings. The sentence embedding
$T^{cap}_{i}$ represents the caption itself. PosE(i) is the positional
embedding, where $i$ represents the caption index in the document. The segment
embedding SegE($T^{cap}_{i}$)=[C] is used to distinguish different segments.
For instance in Figure 2, the position index of the image is 1 since it’s the
second image in the document. Similarly, the final image embedding
$l^{img}_{k}$ is aggregated by $V^{img}_{k}$, PosE(k) and
SegE($V^{img}_{k}$)=[V].

$l^{cap}_{i}$
$\displaystyle=T^{cap}_{i}+\texttt{PosE}(i)+\texttt{SegE}(T^{cap}_{i})$ (1)
$l^{img}_{k}$
$\displaystyle=V^{img}_{k}+\texttt{PosE}(k)+\texttt{SegE}(V^{img}_{k})$ (2)
$l^{sec}_{j}$
$\displaystyle=T^{sec}_{j}+\texttt{PosE}(j)+\texttt{SegE}(T^{sec}_{j})+\texttt{EntE}(s_{j})$
(3)

Inspired by the fact that the complementary images/caption pairs can be the
literal visualization of entities mentioned in document sentences, we
calculate the common entity numbers between sections with captios and sort
sections in the descending order. More formally, we introduce the entity
embedding EntE$(s_{j})$ for $l^{sec}_{j}$, where $s_{j}$ $\in$ (0 $\leq$ i <
N). If a section share most entities with a images/caption pair, $s_{j}$=0,
while $s_{j}$=N-1 if it shares the least entities. In Figure 2, given the
positive query (image/caption in yellow), $s_{0}$=0 and $s_{4}$=2.

Image and Caption Fusion. The intuitive way to select the related section is
to compute the similarity score between section text with the image and
caption separately. Then train a learnable coefficient to combine them.
However, it ignores the potential correlation between the image and caption
effectively. Therefore, we apply the Cross-modal Transformer [12] to fuse the
$l^{cap}_{i}$ and $l^{img}_{k}$ before the layout transformer. We also add the
[cls] token in the first place of the input sequence. The outputs from Cross-
modal Transformer is a sequence of contextualized embeddings. We use the
output from the [cls] token as the unified representation of a image and
caption pair.

$\displaystyle l^{ic}_{ki}$ $\displaystyle=\textit{Cross-
Transformer}(\texttt{$l^{img}_{k}$},\texttt{$l^{cap}_{i}$})$ (4)

Layout Transformer. We concatenates section sentence embeddings
{$l^{sec}_{0}$, $l^{sec}_{1}$, $...$, $l^{sec}_{N-1}$} and the image/caption
pair {$l^{ic}_{00}$, $l^{ic}_{11}$, $...$, $l^{ic}_{(M-1)(M-1)}$} to a unified
sequence $L$. In order to make the features more compact, we employ an FC
layer with a ReLU activation before the layout Transformer [19]. Formally, we
obtain the contextual multimodal embeddings $X$ after transformer layers with
layout learning. More formally, $X=$ {$x^{sec}_{0}$, $...$ $x^{sec}_{N-1}$,
$x^{ic}_{00}$, $...$, $x^{ic}_{(M-1)(M-1)}$}. $x^{sec}_{j}$ $\in$
$[x_{j0},...,x_{jK}]$ are the contextual embeddings of the Top $K$ sentences
each section, where $j$ $\in$ (0 $\leq$ j < N) means the section index.

$\displaystyle X$ $\displaystyle=\textit{Layout-Transformer}(ReLU(FC(L)))$ (5)

####  2.2.3 Salience-Aware Contrastive Learning.

In this section, we introduce the definition of positive/negative pairs for
similarity learning and our salient-aware contrastive loss..

Positive Pairs. We define that a section $x^{sec}_{j}$ and image/caption pair
$x^{ic}_{k,i}$ is positive if the section contains certain content that is
relevant to both the image and caption in $x^{ic}_{k,i}$. For example in
Figure 2, $\beta^{p}=$ {$(x_{40}$,$x^{ic}_{11})$,$(x_{41}$,
$x^{ic}_{11})$,$(x_{42}$,$x^{ic}_{11})$,...} are the positive pairs, where
$x^{ic}_{11}$ is the positive query. $[x_{40},x_{41},...,x_{4K}]$ are the Top
$K$ sentences embeddings in the forth section.

Negative Pairs. Our negative pairs set $\beta^{n}$ contains two parts: normal
negative pairs and hard negative pairs. In normal negative pairs, the
groundtruth section is replaced by other irrelevant sections in the same
document. In Figure 2, {$(x_{00}$, $x^{ic}_{11})$,$(x_{01}$,
$x^{ic}_{11})$,$(x_{02}$,$x^{ic}_{11})$,
$(x_{10}$,$x^{ic}_{11})$,$(x_{12}$,$x^{ic}_{11})$,$(x_{12}$,$x^{ic}_{11})$...}
are the positive pairs, where $x^{ic}_{11}$ is the positive query.
$[x_{40},x_{41},...,x_{4K}]$ are normal negative pairs. As for the hard
negative pairs, either the caption items or the image items might be changed
but not both of them. For example in Figure 2, $x^{ic}_{12}$ is the hard
negative query and {$(x_{40}$, $x^{ic}_{12})$,$(x_{40}$,
$x^{ic}_{21})$,$(x_{41}$,$x^{ic}_{12})$, $(x_{41}$,$x^{ic}_{21})$,...} are
hard negative pairs. This is inspired by the fact that both the caption and
image play important role to determine the relevant text from the document.

Salience-Aware Contrastive Loss. Motivated by the fact that the section label
is used as the weak supervision, we propose to equip the model with the
ability to extract most salient sentence from each section given the query
$x^{ic}_{ki}$ and document context. Additionally, unrelated sentences in
groundtruth section can introduce additional noise in the contrastive learning
process. Formally, we measure the cosine similarity between query
$x^{ic}_{ki}$ and $[x_{j0},...,x_{jK}]$. Then we select the maximum score to
represent the correlation between the section and image/caption pair:

$\displaystyle q=x^{ic}_{k,i},s=x^{sec}_{j}\in\texttt{$[x_{j0},...,x_{jK}]$}$
(6) $\displaystyle S(q,s)=max\\{cos(q,s)\\},(q,s)\in\beta^{p}$ (7)
$\displaystyle S^{*}(q,s)=max\\{cos(q,s)\\},(q,s)\in\beta^{n}$ (8)
$\displaystyle\textit{$L$}=-\sum_{(q,s)\in
D}[\textit{log}(\frac{S(q,s)}{S(q,s)+\sum_{S^{*}\in\beta^{n}}S^{*}(q,s)}]$ (9)

where $S(q,s)$ collects the maximum similarity scores of positive pairs and
$S^{*}(q,s)$ includes the negative pairs. $D$ means the document. Finally, our
model is trained by minimizing the infoNCE [28] loss.

Model Inference. Given a image/caption pair as the query $x^{ic}_{k,i}$, the
cosine similarity between the query and all sentences in the document is
computed. Then we sort the sentences in descending order according to the
similarity and the first candidate will be picked as the most relevant
sentence prediction. We also regard the section including this sentence as the
most relevant section prediction.

##  3 Pretraining Dataset

In this section, we describe the data creation procedure of our pretraining
dataset: Wikipedia dataset and present dataset statistic. First, we used the
latest English Wikipedia dump222https://github.com/muraoka7/tool4ipp to
extract Wikipedia articles [26]. In Wikipedia articles, images are nested in
two HTML tags, which delineate the sections. We regard all the sentences
within the same section as the ground truth since obtaining sentence-level
associations would require additional manual labeling. [34]’s dataset has the
groundtruth linking between image/caption with sentences in the news articles.
However, we don’t have the right to use them according to the official
website333https://help.nytimes.com/hc/en-us/articles/115014893428-Terms-of-
Service#2.

Consequently, we obtained 150,656 documents, not including ones failed to be
parsed. Then we only kept documents with 2 to 30 images and no more than 32
sections. We also converted all the extracted images to an RGB format in order
to guarantee quality. Eventually, this resulted in 66k articles and 320k
images. We summarize the statistics of our dataset in Table 1 with previous
datasets [31, 7, 19, 3]. Each section in our dataset has 8.14 sentences on
average, the average sentence length is 22.3 words and the average section
length is 195.5 words in our dataset, which is the largest among the compared
datasets. In addition, over 50% documents in our dataset have more than 10
sections and many sections don’t have corresponding images (Figure 3).
Compared to other datasets used in Image-Text alignment methods (e.g. COCO
captioning), the amount of text associated with an image is much larger. This
confirms that understanding of longer texts is the required ability.

Figure 3: Distributions of the image number per section and section number per
document in our Wikipedia dataset.

.

##  4 Experiments

In this section, we first introduce details of our implementation and present
comprehensive experiment results.

Model | R@1 | R@3 | A-R@1
---|---|---|---
CLIP(ViT32) [28] | 0.40 | 0.69 | 0.39
CLIP(RN50) [28] | 0.39 | 0.67 | 0.38
SBert [30] | 0.38 | 0.67 | 0.38
Pythia [26] | 0.33 | 0.55 | 0.34
Newsroom [17] | 0.31 | 0.51 | 0.31
SANDI [27] | 0.34 | 0.57 | 0.33
LAIS [46] | 0.45 | 0.74 | 0.45
DocumentCLIP-Early Fusion | 0.57 | 0.87 | 0.56
DocumentCLIP-Late Fusion | 0.56 | 0.85 | 0.54
DocumentCLIP-Early Fusion | 0.57 | 0.87 | 0.56
-w/o salience-aware loss | 0.54 | 0.84 | 0.53
-w/o entity check | 0.46 | 0.76 | 0.46
-w/o layout information | 0.39 | 0.67 | 0.39
Table 2: Comparative Experiments with baselines and ablation study on
Wikipedia datasets.

###  4.1 Experimental Details

Evaluation Setting. Supervised settings: We first evaluate the most relevant
section prediction on the Wikipedia dataset in both the image level and
document level. At the image level, we use R@N(N=1,3) to calculate the
percentage of images where the groundtruth section is among the TopN predicted
sections. At the document level, we measure the percentage of documents where
all images in the document have a correct Top1 predicted section: A-R@1. Zero-
shot setting: We ask human annotators to evaluate the most relevant sentence
prediction on the Open Textbook dataset. DocumentCLIP is trained on the
Wikipedia dataset and directly tested in the zero-shot setting. More detailed
more described in 4.3.

Implementation Details. As for the experiment setting, we randomly split our
dataset into 80% for training, 10% for validation and 10% for testing. We
trained our model with mini-batches of size 32 by using stochastic gradient
descent. We used AdamW [39] as an optimization algorithm and a warm-up
strategy with the initial learning rate of 0.001.

Baseline Models. We run the public release of CLIP [28] and SBert [30] and
finetune them on the Wikipedia dataset in the supervised task. As for the
zero-shot setting, they aren’t finedtuned. CLIP takes both images and captions
as the input while SBert only takes the captions. Additionally, only the first
sentence in each section is fed into CLIP since its maximum length of text
encoder is 77, which is much below the average section length. Pythia [26]
encodes different modalities separately and computes an attention layer to
attend specific parts in the image. Newsroom [26] adopts a hierarchical self-
attention mechanism to attend to both key words within a piece of captions and
informative components of a document. LAIS [46] introduce a two-stage
architecture to select the insertion position for the images.

Training | Inference | R@1 | R@3
---|---|---|---
Image Caption |  Image Caption |  |
✓ Original |  ✓ Original | 0.57 | 0.87
✓ Generated |  ✓ Generated | 0.51 | 0.82
✗ Original |  ✗ Original | 0.54 | 0.84
✗ Generated |  ✗ Generated | 0.49 | 0.79
✓ ✗ | ✓ ✗ | 0.50 | 0.81
✓ Original |  ✓ ✗ | 0.52 | 0.82
✓ Original |  ✗ Original | 0.55 | 0.85
Table 3: Performance of DocumentCLIP with or without images and captions. Generated captions are from BLIP [10]. Section Encoding | Initialization | R@1 | R@3
---|---|---|---
First Sentence | CLIP(ViT32) | 0.54 | 0.84
Weighted Average | CLIP(ViT32) | 0.55 | 0.85
All Sentences Concat | CLIP(ViT32) | 0.52 | 0.82
Salient Sentence | CLIP(ViT32) | 0.57 | 0.87
Salient Sentence | CLIP(RN50) | 0.55 | 0.86
Salient Sentence | ROBERTA* | 0.56 | 0.86
Salient Sentence | SCRATCH | 0.49 | 0.79
Table 4: Performance of DocumentCLIP to investigate different section encoding strategies and initialization methods. ROBERTA* means using the vision transformer CLIP(ViT32) to encode images and Roberta [24] to encode text.  Section Encoding |  Top$K$ | R@1 | R@3
---|---|---|---
Salient Sentence | $K=1$ | 0.54 | 0.84
Salient Sentence | $K=3$ | 0.56 | 0.86
Salient Sentence | $K=5$ | 0.57 | 0.87
Salient Sentence | $K=7$ | 0.57 | 0.86
Salient Sentence | $K=9$ | 0.55 | 0.84
Table 5: Performance of different number of candidates (Top $K$) in the
salient sentences extraction.

###  4.2 Supervised Experiment Results

####  4.2.1 Comparison with Baselines

We report the overall results of most relevant section prediction in the
supervised setting on the Wikipedia dataset (Table 2). We make the following
observations: our model DocumentCLIP achieves more competitive performance in
all scores by a large margin up to 10%. This remarkable improvement indicates
that DocumentCLIP successfully possesses the intra-document alignment between
image and text. The major reason is that previous methods aren’t able to
distinguish between the sections in the same document since these sections
have similar topics. Instead, we use salience-aware contrastive learning with
layout features to learn the document context and compare common entities with
captions to differentiate between sections.

Figure 4: Two examples of Wikipedia articles. The red section is the
prediction from baseline method while the blue section is the prediction by
DocumentCLIP. The sentence in blue is the most relevant candidate.

s

####  4.2.2 Ablation Studies

Contributions of Different Components. The below sections of Table 2
summarizes the ablative performance of our model with different components.
Without layout information, the performance shows 7% degradation in terms of
R@1 and 9% degradation in R@3, which demonstrates that the structural
information like the position features and segment features are essential for
our model to learn the correlation and difference between each sections and
how to connect to the semantic of image/caption pairs. We notice that
DocumentCLIP benefits from entity check and salient-aware loss since they can
not only filter out noisy sentences in sections but also check keyword
alignment between captions and sections. Additionally, the improvement from
early fusion strategy indicates the its efficiency to aggregate multimodal
features between the image and caption.

Model | Avg Rank | Rank #1 | Rank #2 | Rank #3
---|---|---|---|---
SBert | 1.29 | 22% | 31% | 47%
CLIP | 1.13 | 29% | 36% | 35%
DocumentCLIP | 0.75 | 49% | 33% | 18%
Table 6: Human evaluation of different models on Open Textbook dataset in the
zero-shot setting. Annotators are asked to put the sentences in order of
relevance to the Image/Caption pair. ”1.29” means the average rank of SBert is
1.29. ”22%” means only 22% of the predictions from SBert are put in the 1st
place.

Importance of Different Modalities. Besides, we investigate the contribution
of each modality (image or caption) in Table 3. Given only one modality in
training and inference, the performance of captions is superior to that of
images (third and fifth row). This can be reasonable because captions
sometimes share similar or same words with textual sections. To further
analyze the effect of captions, we replace the original captions with the ones
generated by BLIP [10], an state-of-art and pretrained vision-language
generation model (second and forth row). The performance degradation is
expected since generated captions always miss the key entities in the original
ones, which play the evidence role.

Figure 5: Selected image/caption samples and most relevant sentence
predictions from Open Textbook dataset. We compare our predictions with the
ones from CLIP, SBert and demonstrate the advance of DocumentCLIP.

Section Encoding Strategies. Table 4 studies the effect of different section
encoding strategies. Compared to other strategies, the improvement is evident
when using salient sentence extraction. The main reason for this might because
other strategies fail to remove the influence of the noisy sentences, which
will mislead the model to pick the wrong the section. To further investigate
its effect, we gradually increase the Top$K$ value and implement the salient
sentence extraction in Table 5. We notice that a larger $K$ will yield better
accuracy when $K$ is smaller than 5. The phenomenon indicates that single
sentence isn’t informational enough to represent the section while too many
ones will introduce additional noise.

Initialization. We also analyze different initialization methods in Table 4.
As the ROBERTA* (six row) method suggests, we use the vision transformer
CLIP(ViT32) to encode images and Roberta [24] to encode text. We found it
achieves best results if we use CLIP to encode both images and captions. This
is because CLIP is trained on a large vision and language dataset, alleviating
the semantic gap between images and text. Besides, SCRATCH performs worst due
to missing the prior knowledge.

####  4.2.3 Qualitative Analysis.

In order to evaluate the quality of the complementary text retrieved by our
model in the sentence level, examples are shown in Figure 4. This left
document talks about a Japanese manga series called Ranma ½. The image/caption
pair is in the 2nd place and talks about the TV drama version of Ranma ½.
Previous methods regard the first section as the prediction since it share
more common entities with the caption. However, they fail to encode image/pair
position information and understand commonsense knowledge. For example, ”live
action special” from the fifth section actually refers to ”TV drama” in the
caption. As for the right document, current methods only consider the keywords
within the captions without the image position information.

###  4.3 Zero-Shot Experiment Results

In this section, we gain further insights into the most relevant sentence
prediction on the Open Textbook dataset via human evaluation. With the human
evaluation, we access whether our model could pick more relevant sentences
than CLIP [28] and SBert [30] in the zero-shot setting.

Here, we randomly select a set of 40 image/captions pairs from 10 articles in
Open Textbook dataset. We conduct our evaluations on 75 workers from Amazon
Mechanical Turk444https://www.mturk.com. The 40 questions in the questionnaire
are shuffled for each worker and it takes 40 minutes to complete on average.
We also add 3 attention questions to guarantee the quality of the response. As
for each question, they are given a caption and image pair and asked to put
the three sentences in order of relevance to the Image/Caption pair and the
most relevant should be in the 1st place. Summary of the evaluation are in Tab
6. The average rank of the predictions from DocumentCLIP is 0.75, which is
much lower than SBert: 1.29 and CLIP: 1.13. In addition, 49% of the samples
have the DocumentCLIP in the 1st place and only 18% of the samples have the
DocumentCLIP in the final place. Figure 5 demonstrate the advance of our model
to understand the intra-document multimodal interaction.

##  5 Related Work

Cross-modal research [10, 1, 9, 23, 44, 38, 36, 23, 20, 19, 21, 22, 18, 15]
has recently received increasing attention as a result of the advances in both
computer vision and natural language processing. [6, 8, 45, 35] encode
multimodal features into a common embedding space in which instance similarity
is measured by conventional cosine or Euclidean distance. Recent state-of-the-
art works [28, 13, 25, 33] employ pretraining methods, cross transformers and
contrastive learning algorithms to learn the correlation between different
modalities. Various pre-training objectives have also been proposed over the
years, and have progressively converged to a few time-tested ones [38, 11, 14,
40, 42, 41, 43]. However, the limitation is that they are trained to align
short literal text descriptions with images but fail to handle real-world
media such as news articles [19], Wikipedia pages [26], magazines, product
descriptions consisting of multiple sentences with multiple images and layout
information. Document structure analysis models [5, 4, 37, 2] process pages
one by one, but the figures may relate to text located on other pages.

Different from existing methods, DocumentCLIP is not only able to effectively
process long text, but also understand text-image interaction at the sentence-
level in the document.

##  6 Conclusions

In this paper, we are the first to explore the sentence-level multimodal
intra-document links in vision-language pretraining. We also introduce a novel
contrastive learning framework with layout information, multimodal
interaction, novel section encoding strategy, salience-aware contrastive loss
and hard negative samples. We also collect a large visual document dataset
from Wikipedia with 66k articles, 320k images/caption pairs and various
topics. Furthermore, our proposed model DocumentCLIP achieves state-of-the-art
performance in both the supervised and zero-shot settings. We hope this work
paves the way for future studies in document understanding and multi-modal
alignment.

## References

  * [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  arXiv preprint arXiv:2204.14198, 2022.
  * [2] Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha.  Docformer: End-to-end transformer for document understanding.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 993–1003, 2021.
  * [3] Ali Furkan Biten, Lluis Gomez, Marçal Rusiñol, and Dimosthenis Karatzas.  Good news, everyone! context driven entity-aware captioning for news images.  arXiv e-prints, pages arXiv–1904, 2019.
  * [4] Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun.  Unidoc: Unified pretraining framework for document understanding.  Advances in Neural Information Processing Systems, 34:39–50, 2021\.
  * [5] Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei.  Layoutlmv3: Pre-training for document ai with unified text and image masking.  arXiv preprint arXiv:2204.08387, 2022.
  * [6] Yan Huang, Wei Wang, and Liang Wang.  Instance-aware image and sentence matching with selective multimodal lstm.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2310–2318, 2017.
  * [7] Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi.  Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension.  In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pages 4999–5007, 2017.
  * [8] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He.  Stacked cross attention for image-text matching.  In Proceedings of the European conference on computer vision (ECCV), pages 201–216, 2018.
  * [9] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [10] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  arXiv preprint arXiv:2201.12086, 2022.
  * [11] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  Advances in neural information processing systems, 34:9694–9705, 2021.
  * [12] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu.  Hero: Hierarchical encoder for video+ language omni-representation pre-training.  arXiv preprint arXiv:2005.00200, 2020.
  * [13] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [14] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16, pages 121–137. Springer, 2020.
  * [15] Zongxi Li, Paiheng Xu, Fuxiao Liu, and Hyemi Song.  Towards understanding in-context learning with contrastive demonstrations and saliency maps.  ArXiv, abs/2307.05052, 2023.
  * [16] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [17] Fangyu Liu, Rémi Lebret, Didier Orel, Philippe Sordet, and Karl Aberer.  Upgrading the newsroom: an automated image selection system for news articles.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 16(3):1–28, 2020.
  * [18] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.  Aligning large multi-modal model with robust instruction tuning.  ArXiv, abs/2306.14565, 2023.
  * [19] Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez.  Visual news: Benchmark and challenges in news image captioning.  arXiv preprint arXiv:2010.03743, 2020.
  * [20] Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez.  Visual news: Benchmark and challenges in news image captioning.  In Conference on Empirical Methods in Natural Language Processing, 2020.
  * [21] Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez.  Visualnews : A large multi-source news image dataset.  arXiv: Computer Vision and Pattern Recognition, 2020.
  * [22] Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez.  Visualnews : Benchmark and challenges in entity-aware image captioning.  ArXiv, abs/2010.03743, 2020.
  * [23] Fuxiao Liu, Yaser Yacoob, and Abhinav Shrivastava.  Covid-vts: Fact extraction and verification on short video platforms.  arXiv preprint arXiv:2302.07919, 2023.
  * [24] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [25] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  Advances in neural information processing systems, 32, 2019.
  * [26] Masayasu Muraoka, Ryosuke Kohita, and Etsuko Ishii.  Image position prediction in multimodal documents.  In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4265–4274, 2020.
  * [27] Sreyasi Nag Chowdhury, Simon Razniewski, and Gerhard Weikum.  Sandi: Story-and-images alignment.  In 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 989–999. ACL, 2021.
  * [28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [29] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [30] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084, 2019.
  * [31] Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar.  Kvqa: Knowledge-aware visual question answering.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8876–8884, 2019.
  * [32] Yusuxke Shibata, Takuya Kida, Shuichi Fukamachi, Masayuki Takeda, Ayumi Shinohara, Takeshi Shinohara, and Setsuo Arikawa.  Byte pair encoding: A text compression scheme that accelerates pattern matching.  1999\.
  * [33] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  arXiv preprint arXiv:1908.08530, 2019.
  * [34] Alasdair Tran, Alexander Mathews, and Lexing Xie.  Transform and tell: Entity-aware news image captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13035–13045, 2020.
  * [35] Tan Wang, Xing Xu, Yang Yang, Alan Hanjalic, Heng Tao Shen, and Jingkuan Song.  Matching images and text with multi-modal tensor fusion and re-ranking.  In Proceedings of the 27th ACM international conference on multimedia, pages 12–20, 2019.
  * [36] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.  Video question answering via gradually refined attention over appearance and motion.  In Proceedings of the 25th ACM international conference on Multimedia, pages 1645–1653, 2017.
  * [37] Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, et al.  Layoutlmv2: Multi-modal pre-training for visually-rich document understanding.  arXiv preprint arXiv:2012.14740, 2020.
  * [38] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: fine-grained interactive language-image pre-training.  arXiv preprint arXiv:2111.07783, 2021.
  * [39] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh.  Large batch optimization for deep learning: Training bert in 76 minutes.  arXiv preprint arXiv:1904.00962, 2019.
  * [40] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  arXiv preprint arXiv:2205.01917, 2022.
  * [41] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi.  Merlot reserve: Neural script knowledge through vision and language and sound.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16375–16387, 2022.
  * [42] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi.  Merlot: Multimodal neural script knowledge models.  Advances in Neural Information Processing Systems, 34:23634–23651, 2021.
  * [43] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123–18133, 2022.
  * [44] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Making visual representations matter in vision-language models.  arXiv preprint arXiv:2101.00529, 1(6):8, 2021.
  * [45] Ying Zhang and Huchuan Lu.  Deep cross-modal projection learning for image-text matching.  In Proceedings of the European conference on computer vision (ECCV), pages 686–701, 2018.
  * [46] Zhengkun Zhang, Jun Wang, Adam Jatowt, Zhe Sun, Shao-Ping Lu, and Zhenglu Yang.  News content completion with location-aware image selection.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14498–14505, 2021.


# Mitigating Translationese in Low-resource Languages: The Storyboard Approach

###### Abstract

Low-resource languages often face challenges in acquiring high-quality
language data due to the reliance on translation-based methods, which can
introduce the translationese effect. This phenomenon results in translated
sentences that lack fluency and naturalness in the target language. In this
paper, we propose a novel approach for data collection by leveraging
storyboards to elicit more fluent and natural sentences. Our method involves
presenting native speakers with visual stimuli in the form of storyboards and
collecting their descriptions without direct exposure to the source text. We
conducted a comprehensive evaluation comparing our storyboard-based approach
with traditional text translation-based methods in terms of accuracy and
fluency. Human annotators and quantitative metrics were used to assess
translation quality. The results indicate a preference for text translation in
terms of accuracy, while our method demonstrates worse accuracy but better
fluency in the language focused.

Keywords: Low-resource languages, Translationese, Translation Data

Mitigating Translationese in Low-resource Languages: The Storyboard Approach

Garry Kuwanto1, Eno-Abasi Urua2, Priscilla Amuok†,
---
Shamsuddeen Hassan Muhammad3†, Anuoluwapo Aremu†, Verrah Otiende†,
Loice Nanyanga†, Teresiah Nyoike†, Aniefon Akpan2, Nsima Udouboh2,
Idongesit Archibong2, Idara Moses2, Ifeoluwatayo Ige4†, Benjamin Ajibade†,
Olumide Awokoya†, Idris Abdulmumin5†, Saminu Mohammad Aliyu6†,
Ruqayya Iro†, Ibrahim Said Ahmad7†, Deontae Smith8, Praise-EL Michaels8,
David Ifeoluwa Adelani9†, Derry Tanti Wijaya1,10, Anietie Andy8
1Department of Computer Science, Boston University
2University of Uyo, Nigeria, 3Imperial College London, 4Rochester Institute of
Technology
5Data Science for Social Impact Research Group, University of Pretoria,
6Bayero University, Kano
7Institute For Experiential AI, Northeastern University
8Department of Electrical Engineering and Computer Science, Howard University
9University College London, 10Monash University Indonesia, †Masakhane
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Low-resource languages pose significant challenges when it comes to acquiring
high-quality language data for various applications, including language
documentation, linguistic research, and machine translation Kuwanto et al.
(2023). Traditionally, data collection in these languages involves obtaining
translations from higher-resource languages, such as English. However, this
approach often leads to the introduction of a translationese effect, where the
resulting sentences may be less fluent and natural including those produced by
professional translators.

Translationese refers to the linguistic phenomenon that occurs when
translations exhibit characteristics that are not typical of the target
language Gellerstam (1986), as well the use of more explicit and standardised
constructions Baker et al. (1993) compared to original text. These
characteristics can manifest as unnatural word choices, sentence structures,
or even the adoption of foreign syntactic patterns. Translators, while highly
skilled in bridging the language gap, often face challenges in recreating the
nuanced meaning and linguistic nuances of the original text. As a result, the
translated sentences may sound unnatural or stilted to native speakers,
detracting from the authenticity and quality of the collected data.

Translationese has been widely recognized for its detrimental impact not only
on machine translation tasks but also on other tasks involving cross-lingual
transfer learning Amponsah-Kaakyire et al. (2022); Ni et al. (2022); Artetxe
et al. (2020). The presence of translationese introduces biases, diminishes
fluency and naturalness, and ultimately affects the overall quality of the
output. Previous research efforts have primarily focused on mitigating the
translationese effect during the downstream phase, such as treating
translationese as a different language Riley et al. (2020), applying embedding
space projection Yu et al. (2022); Chowdhury et al. (2022), and utilizing
paraphrasing techniques Artetxe et al. (2020); Wein and Schneider (2023).
However, these approaches have limitations, requiring additional annotation or
modification of training data. Moreover, these methods primarily address
translationese after its occurrence, rather than preventing or minimizing its
presence during the data collection phase. To the best of our knowledge, there
has been no prior work specifically addressing the reduction of translationese
artifacts during the data collection phase itself

In this paper, we address this challenge by introducing the use of storyboards
Burton and Matthewson (2015), a common field linguist tool, in the data
collection process. Our method leverages the power of visual stimuli to elicit
more fluent and natural sentences from native speakers without the explicit
influence of the source language text. Instead of providing sentences, we
present native speakers with a storyboard consisting of images of scenes
accompanied by their English sentences an hour before the annotation process,
the process where annotators . During the annotation phase, participants are
asked to describe the scene in the image, focusing solely on the visual
content without access to the English sentences. The primary objective of our
research is to examine whether storyboards can be an alternative method to
data collection when improving fluency and reducing translationese bias are of
interest. By removing direct exposure to source language text during
annotation, we hypothesize that the resulting sentences will still hold the
meaning of the original sentence while exhibiting improved fluency and
naturalness. This approach has the potential to provide a more accurate
representation of the target language, facilitating the development of higher
quality language resources.

##### Contributions:

We make three major contributions: (1) the collection of data in four
typologically-diverse low-resource African languages (Hausa, Ibibio, Swahili,
and Yorùbá) in such a way that less translationese artifact arises, (2) the
evaluation of the effectiveness of the storyboard approach in generating
fluent and more natural sentences, and (3) to our knowledge, the first-ever
parallel resource created data for Ibibio in non-religious domain.

In the following sections, we will delve into the details of our data
collection method (Section 3), describe the experimental design, evaluation
process, and present the comprehensive results of our analysis (Section 4). By
combining qualitative evaluation by human annotators and quantitative metrics,
we aim to provide an assessment of the effectiveness of our proposed method
and its implications for data collection and the mitigation of the
translationese effect.

## 2\. Related Work

Data collection for low-resource languages has been a subject of ongoing
research and development. Several initiatives have contributed to this field,
such as the LORELEI project Strassel and Tracey (2016) and the REFLEX-LCTL
project Simpson et al. (2008). These projects, conducted by the Linguistic
Data Consortium (LDC), have released annotated corpora for multiple languages,
addressing the need for linguistic resources in low-resource settings.

However, the predominant approach to data collection in low-resource languages
still involves leveraging monolingual data from higher-resource languages and
manually translating it. This is primarily because of the scarcity of
monolingual and digital data available in the target language. However, this
reliance on translation-based methods introduces challenges, such as the
introduction of the translationese effect, where translated sentences may be
less fluent and natural in the target language Chowdhury et al. (2022).

Translationese refers to the phenomenon in which translations exhibit
linguistic characteristics that deviate from the typical patterns of the
target language Gellerstam (1986). The impact of translationese is
particularly noticeable in the areas of syntax and grammar Santos (1995).
Translations may exhibit unnatural sentence structures, lexical and word order
choices that are influenced by the source language Gellerstam (1996), adopt
foreign syntactic patterns, as well as use more explicit and simpler
constructions Baker et al. (1993). This can result in sentences that sound
unnatural or stilted to native speakers.

In the study by Aranberri (2020), an analysis of translationese was performed
on the Spanish-Basque language pair. The researchers measured various
linguistic features, including lexical variety, lexical density, length ratio,
and perplexity. These measurements provided insights into the extent of
translationese and its impact on the linguistic characteristics of the
translated text. Similarly, Kunilovskaya and Lapshinova-Koltunski (2019)
conducted a similar analysis focusing on English to Russian translation. In a
different domain, Bizzoni et al. (2020) conducted a study that compared
translationese across human and machine translations from text and speech.

Multimodal translation tasks, such as the WMT18 multimodal task Barrault et
al. (2018), involve the use of both image and text during the translation
process whether manually Elliott et al. (2016, 2017); Barrault et al. (2018)
or automatically Wijaya et al. (2017); Hewitt et al. (2018); Rasooli et al.
(2021); Khani et al. (2021). During the manual annotation for these tasks,
annotators are provided with both the source image and the corresponding
source text, allowing for direct reference and alignment between the two
modalities. However, this direct exposure to the source segment during
translation may introduce the translationese effect, as observed in previous
studies Elliott et al. (2016). For example, it has been observed that the
lengths of translations in German are more similar to the lengths of the
source English sentences than to the lengths of German image descriptions.
This suggests that the presence of the source text can influence the resulting
translations and potentially impact the fluency and naturalness of the target
language output. In contrast, our storyboard-based data collection approach
presents annotators with only the image, without the direct exposure to the
source text.

## 3\. Data and Data Collection

In this work, we propose a new method for collecting translations for low-
resource languages, with the aim of reducing the influence of source (English)
sentences during translation. To achieve this, we utilize a dataset comprising
images depicting various scenes and their corresponding English descriptions.
For each image and its associated English sentence, we engage two groups of
native speakers for each target language. One group translates the English
sentences, while the other group writes descriptions (in their respective
languages) based on the visual content of the images. In this section, we
provide a detailed description of the data and the data collection method
employed in our study.

### 3.1. English Sentence and Image Pair

We obtain our dataset from the Totem Field
Storyboards111https://totemfieldstoryboards.org/, which provides a collection
of storyboards consisting of sequential visual representations of stories in
specific contexts, accompanied by corresponding English sentences describing
the depicted scenes. An example of a sequence of English sentence and image
pairs is shown in Figure 1. For our research, we select 26 storyboards from
the Totem Field Storyboards, each containing an average of 19 English sentence
and image pairs.

Figure 1: Example of English sentence and Image pair

### 3.2. Translators and Focus Languages

Our study focuses on four low-resource African languages: Swahili, Yorùbá,
Hausa, and Ibibio. For each target language, we engage a total of four native
speakers. Among them, two native speakers are assigned to translate the
English sentences, and the other two are shown the images and asked to write
descriptions in their respective languages based on the visual content.

### 3.3. Data Collection

Considering our objective of determining and reducing the translationese
effect in low-resource language data collection, we design the data collection
process as follows: for each pair of English sentence and image in the
storyboards, one group of translators focuses on translating the English
sentence, while another group concentrates on translating the image. This
separation allows us to examine the impact of English sentences on the
resulting translations and evaluate the fluency and naturalness of the
sentences generated solely based on visual content. By collecting data through
this dual approach, we aim to obtain a more authentic representation of the
target languages and mitigate the translationese effect. For each language,
the translators were paid a total of 600 US dollars; which they shared equally
among themselves.

#### 3.3.1. Control Group: Text Translation

In our study, we include a control group that utilizes the traditional
approach of text translation. This control group serves as a baseline for
comparison and allows us to evaluate the effectiveness of our storyboard-based
method in reducing the translationese effect.

For the control group, native speakers are provided with the English sentences
from the storyboards and are instructed to translate them directly into the
target languages. These native speakers possess proficiency in both the source
and target languages.

The translations generated by the control group represent the typical output
obtained through traditional translation-based approaches. These translations
are expected to exhibit characteristics of translationese, such as potential
deviations from natural language usage, as translators may prioritize fidelity
to the source text over fluency in the target language.

#### 3.3.2. Treatment Group: Storyboard-Based Translation

In our storyboard-based translation method, we introduce a preparatory phase
before the actual annotation process. Before annotating each storyboard, the
annotators are grouped together in a meeting where they are given the
opportunity to familiarize themselves with the storyboard and the
corresponding English sentences. During this meeting, they can read through
the storyboard and comprehend the context and content of each image.

After this reading session, we introduce a time gap of $\sim$1 hour before the
annotation process begins. This time gap serves two purposes: (1) it allows
the annotators to internalize the visual information from the storyboard, and
(2) it minimizes the direct influence of the English sentences on their
subsequent annotations.

During the annotation phase, the annotators are provided with the storyboards
containing only the images, without any accompanying English sentences. They
are instructed to focus solely on the visual content and describe in their
respective target languages the scene that is being depicted in each image.
This approach ensures that the annotations are driven primarily by the visual
stimuli, encouraging the annotators to provide translations that capture the
essence of the scenes portrayed in the images.

By removing the explicit exposure to the English sentences during the
annotation process, our storyboard-based method aims to reduce the potential
influence of the source language and the translationese effect. The annotators
are encouraged to rely on their linguistic knowledge and cultural
understanding to generate fluent and natural translations that are more
aligned with the target language’s usage patterns and stylistic conventions.

### 3.4. Annotation Dataset

The resulting annotation dataset consists of the source English sentences and
their corresponding translations in four low-resource African languages:
Swahili, Yorùbá, Hausa, and Ibibio. The dataset includes both text
translations and translations obtained through our storyboard-based method.

In total, we collected translations for 486 unique English sentences. The
distribution of translations is shown in Table 1, indicating the number of
translations for each language and data collection method (text or
storyboard). The number between text and storyboard translations differs
slightly because we gave translators the option to provide alternative
translations for each English sentences.

Language | Text Translation | Storyboard
---|---|---
Hausa | 1154 | 968
Ibibio | 887 | 883
Swahili | 1334 | 1211
Yorùbá | 1448 | 1033
Table 1: Number of translations in the dataset

The dataset also includes additional information such as the title of the
storyboard and the scene number associated with each translation. We will make
the final dataset, along with the source English sentences, publicly
available. An anonymized version of the dataset is also attached in the
supplementary material for reference.

## 4\. Experimental Design

All computational experiments were conducted on a machine equipped with an
Intel Xeon Gold 6226R CPU running at a clock speed of 2.90GHz and an NVIDIA
RTX A6000 GPU.

### 4.1. Human Evaluation

Fluency, for our human evaluation setup, refers to the smoothness and
naturalness of the translated sentence in the target language. A fluent
sentence should not sound "foreign" or awkward, and should read as if a native
speaker had originally written it in that language. Fluency captures the
syntactic and grammatical correctness as well as the idiomatic usage of the
language.

Accuracy, for our case, refers to the extent to which the translated sentence
captures the meaning of the source sentence. An accurate or adequate
translation should convey all essential information from the source text
without adding, omitting, or distorting any content.

To assess the accuracy and fluency of the translated sentences obtained
through the traditional text translation and our proposed storyboard-based
data collection processes, we conducted human evaluation with native speakers
who are also proficient in English as annotators. Annotators were assigned two
tasks: one for accuracy evaluation and another for fluency evaluation. Each
task involved comparing a pair of sentences, one from the text translation and
the other from the storyboard-based collection process. It is important to
note that the sentence pairs provided to the annotators for different
languages were taken from the same storyboard scene to ensure consistency and
fair comparison across languages. For each task (accuracy and fluency), we
randomly select 100 samples from the storyboard scenes and obtain sentence
pairs.

To minimize bias and ensure reliable results, three human annotators were
assigned to each task. Each annotator independently evaluated the sentence
pairs and provided their preference based on accuracy or fluency. The
preferences of the annotators were then tallied up to determine the overall
preference for each translation approach.

After the evaluation, because we have 3 annotators for each language, the
inter-annotator agreement was calculated using the Fleiss’ Kappa statistic to
measure the consistency of the annotators’ preferences. A Fleiss’ Kappa value
of 1 indicates perfect agreement between the annotators, while a value of 0
suggests agreement equivalent to random chance. For our evaluation, the
Fleiss’ Kappa value was found to be 0.27 and 0.16 for accuracy and fluency
respectively, indicating fair and slight level of agreement among the
annotators respectively Landis and Koch (1977).

#### 4.1.1. Accuracy Evaluation

For the accuracy evaluation task, annotators were presented with three
sentences: the source English sentence, the sentence translated through text
translation, and the sentence translated through the storyboard-based method.
Their task was to choose which sentence they deemed more accurate for
translating the English sentence; they can pick one sentence over the other,
or both. The guideline was to "select which sentence is more adequate (i.e.,
more accurate) for translating the English sentence." An important criterion
emphasized in the guideline was that a better translation should include as
much content from the English sentence as possible, without adding information
not present in the original sentence. The annotators were also asked to
disregard the translations of named entities in their accuracy judgment. This
was particularly relevant as, during the storyboard data collection process,
the translators might not remember the exact names mentioned in the English
sentence, while the translators for the text translation could reference the
English sentence for accuracy.

To reduce bias, annotators were only provided with the sentences without any
indication of which sentence was from the text translation and which was from
the storyboard-based method. An example of the accuracy evaluation task for
Hausa can be seen in Table 2 where annotators would compare Sentence 1 and
Sentence 2, selecting the one they considered more accurate, or both if they
deemed the two sentences to be equally accurate.

English Sentence | I bought shoes, a hat, a shirt, and pants
---|---
Sentence 1 | Na siyo takalma, malafa, riga da kuma wando
English Translation | I bought shoes, mattress, shirt and pants
Sentence 2 | Na sayo takalma, hula, riga da kuma yan kamfai
English Translation | I bought shoes, a hat, a shirt and a few shoes
Table 2: An Example of Accuracy Human Evaluation for Hausa. Annotators were
asked to choose the sentence they deemed more accurate for translating the
English sentence. They can answer with Sentence 1, Sentence 2, or both if they
deemed both sentences to be equally accurate.

#### 4.1.2. Fluency Evaluation

For the fluency evaluation task, annotators were not provided with the source
English sentence. Instead, they were presented with two sentences from the
same storyboard scene: one sentence obtained through text translation and
another obtained through the storyboard-based method. Their task was to select
the sentence that they deemed to be more fluent and natural. The guideline was
to "select which sentence is more fluent (i.e., more natural). A better
sentence should be the one that is more natural and grammatical". The
annotators were asked to focus on factors such as natural language usage,
grammaticality, and overall fluency in making their decision.

To ensure unbiased evaluation, the annotators were only provided with the
sentences without any indication of which sentence was from the text
translation and which was from the storyboard-based method. An example of the
fluency evaluation task for Hausa can be seen in Table 3 where annotators
would compare Sentence 1 and Sentence 2, selecting the one they considered
more fluent, or both if they deemed the two sentences to be equally fluent.

Sentence 1 | Mary ta fita zuwa shago
---|---
English Translation | Mary goes out to the shop
Sentence 2 | Ta fita siyayya watarana
English Translation | She went out of the shopping
Table 3: An Example of Fluency Human Evaluation for Hausa. Annotators were
asked to choose the sentence they deemed more fluent. They can answer with
Sentence 1, Sentence 2, or both if they deemed both sentences to be equally
fluent.

### 4.2. Metric-Based Evaluation of Accuracy and Fluency

In addition to human evaluation, we employ several metrics to support and
complement the results obtained from the human annotators. These metrics
provide quantitative measures of accuracy and fluency, enabling a more
comprehensive analysis of the translation quality.

#### 4.2.1. Accuracy

To evaluate the accuracy of translated sentences, we utilize the LASER model
Heffernan et al. (2022), which excels in capturing semantic meaning and
supports a wide range of languages except for Ibibio, which is not in the list
of 147 languages that the LASER encoder supports. We employ this model to
compute the embeddings of the translated sentences and their corresponding
English sentences. To compare the embeddings, we compute the cosine distance,
which serves as a measure of semantic similarity between the translations and
the source English sentences. A higher cosine similarity indicates a stronger
alignment in semantic meaning, suggesting higher accuracy in the translation
process.

In addition to evaluating the accuracy between the translated sentences and
their corresponding English sentences, we also examine the similarity between
the translations generated through our storyboard-based method and the text
translation method. To measure whether the two methods result in translations
with comparable accuracy, we calculate the cosine similarity between
embeddings of the translated sentences. The cosine similarity provides an
indication of how similar the translations are in terms of their semantic
meaning. While we do not expect a perfect similarity between the two methods’
translations, we aim for a comparable level of meaning that allows for fluency
trade-off.

#### 4.2.2. Fluency

Fluency in translation involves the successful use of vocabulary and sentence
structure to convey meaning effectively in the target language. When
translationese phenomena occur, patterns emerge that can impact the fluency of
translated sentences. Previous studies Vanmassenhove et al. (2021); Bizzoni et
al. (2020) have highlighted two key factors in fluency: lexical diversity and
syntactic complexity.

Lexical Diversity To assess the lexical diversity or vocabulary richness of
the translated sentences, we employ the Measure of Textual Lexical Diversity
(MTLD) metric McCarthy (2005). Conceptually, MTLD reflects the average number
of words in a row for which a certain TTR (Type-Token Ratio) is maintained,
specifically in this analysis, following McCarthy and Jarvis (2010) we use TTR
of $0.72$. To generate a score, MTLD calculates the TTR for increasingly
longer parts of the sample. Every time the TTR drops below the predetermined
value, a count (called the factor count) increases by 1, and the TTR
evaluations are reset. The algorithm resumes from where it had stopped, and
the same process is repeated until the last token of the language sample has
been added and the TTR has been estimated. Then, the total number of words in
the text is divided by the total factor count. Subsequently, the whole text in
the language sample is reversed and another score of MTLD is estimated. The
forward and the reversed MTLD scores are averaged to provide the final MTLD
estimate.

Intuitively the MTLD value can be seen as the average number of words required
for the text to reach a point of stabilization.

Syntactic Complexity: To evaluate syntactic complexity, we leverage an off-
the-shelf language model trained on the languages used in the study,
specifically AfroXLMR-base Alabi et al. (2022)—an adaptation of XLM-R to 17
African languages, including Hausa, Swahili and Yorùbá. To estimate the POS
perplexity Bizzoni et al. (2020), we trained a Part-Of-Speech (POS) model by
fine-tuning the AfroXLMR language model on MasakhaPOS dataset (Dione et al.,
2023)— a large-scale POS dataset for 20 African languages, including all focus
languages except for Ibibio. POS perplexity measures the difficulty in
predicting the sequence of POS tags in a sentence. Higher perplexity values
indicate greater syntactic complexity. Perplexity is defined as the
exponentiation of the entropy:

$2^{H(p)}=2^{-\sum_{x}p(x)\log_{2}p(x)}$ (1)

Where $p(x)$ is the probability of the predicted POS.

### 4.3. Results

#### 4.3.1. Accuracy Evaluation

Language | Storyboard | Text | Both
---|---|---|---
Hausa | 21.67% | 78.33% | 0%
Swahili | 14% | 64% | 22%
Yorùbá | 7.67% | 68.67% | 23.67%
Ibibio | 18.33% | 79.67% | 2%
Table 4: Human Evaluation Results for Accuracy. More annotators choose text translations as more accurate across languages Language | Storyboard | Text
---|---|---
Hausa | $0.64\pm 0.14$ | $0.74\pm 0.13$
Swahili | $0.58\pm 0.11$ | $0.69\pm 0.10$
Yorùbá | $0.64\pm 0.13$ | $0.74\pm 0.12$
Table 5: The Average Cosine Similarity Scores between English and Translated
Sentences’ LASER Embeddings. Across languages, text translations have higher
semantic similarity to source English sentences, although the scores still
fall within each method’s standard deviation

The accuracy evaluation results in Table 4 show a clear preference among human
annotators for text translations in terms of accuracy across all languages.
This is consistent with the expectation that text translations, typically done
by professional translators, would be more semantically aligned with the
source sentences.

The cosine similarity values between LASER embeddings of the translated
sentences and their corresponding English sentences, as shown in Table 5,
further support this observation. Text translations generally exhibit higher
semantic similarity to the source English sentences. However, it’s important
to note that while the cosine similarity values for storyboard translations
are lower, they are still within the standard deviation range of the text
translations, suggesting that the semantic content is not drastically
different.

The cosine similarity between translations from the two methods, presented in
Table 6, indicates that the semantic content of translations from both methods
is relatively comparable, even if they are not identical.

Language | Cosine Similarity
---|---
Hausa | $0.63\pm 0.19$
Swahili | $0.62\pm 0.17$
Yorùbá | $0.59\pm 0.18$
Table 6: The Average Cosine Similarity Scores between text translations and
storyboard-based translations’ LASER embeddings.

#### 4.3.2. Fluency Evaluation

Language | Storyboard | Text | Both | p-value
---|---|---|---|---
Hausa | 60% | 39.67% | 0.33% | 0.0002
Swahili | 47.67% | 41.33% | 11% | 0.11
Yorùbá | 34% | 18.6% | 47.33% | 0.0008
Ibibio | 36% | 26% | 38% | 0.01
Table 7: Human Evaluation Results for Fluency. More annotators choose storyboard translations as more fluent across languages. P-Value is for the null hypothesis of the annotators choosing by random. Language | Storyboard | Text
---|---|---
Ibibio | $8.08\pm 17.06$ | $7.27\pm 18.35$
Hausa | $7.8\pm 16.77$ | $11.41\pm 23.48$
Yorùbá | $16.12\pm 30.05$ | $14.16\pm 29.84$
Swahili | $6.83\pm 19.64$ | $5.03\pm 16.61$
Table 8: MTLD of Translated Sentences. Across languages, except for Hausa,
storyboard translations have higher average MTLD scores

The fluency evaluation in Table 7 suggests a preference among human annotators
for storyboard translations in terms of fluency across all languages. This
indicates that while storyboard translations might not always capture the
exact semantic content of the source, they tend to produce more naturally
flowing sentences in the target languages.

The MTLD scores in Table 8, despite having a high standard deviation, provide
further evidence for this observation. Except for Hausa, storyboard
translations generally exhibit greater lexical diversity, which is often
associated with more natural and fluent sentences.

The POS perplexity values in Table 9 offer insights into the syntactic
complexity of the translations. Lower POS perplexity values typically indicate
more natural sentence structures in the target language. The results suggest
that, except for Swahili, storyboard translations tend to produce more
naturally structured sentences than text translations.

In summary, while text translations are generally more accurate and
semantically aligned with the source, storyboard translations offer advantages
in terms of fluency and naturalness of the produced sentences.

Language | Storyboard | Text
---|---|---
Hausa | $6.68$ | $49.42$
Swahili | $55.32$ | $14.6$
Yorùbá | $6.49\times 10^{2}$ | $7.93\times 10^{7}$
Table 9: POS Perplexity of Translated Sentences. Across languages, except for
Swahili, the storyboard translations have lower POS perplexity, indicating
more natural translations in terms of sentence structure than the text
translations

## 5\. Discussion

### 5.1. Comparison of Accuracy Evaluation

The human accuracy evaluation results (Table 4) reveal a clear preference for
text translation over our proposed method. Human evaluators consistently rated
text translation as more accurate compared to our method. This preference
aligns with the linguistic intuition that text translations, performed by
professional translators, tend to produce more accurate translations.
Similarly, looking at the cosine similarities between the source English
sentences and the translations (Table 5), we can see that text translations
have higher cosine similarities to the English sentences, indicating higher
semantic similarities. This observation is consistent with human evaluators’
preference for accuracy.

However, when comparing with the cosine similarities between our method’s
(storyboard) translations and the text translations (Table 6), we found that
the cosine similarity scores were not significantly lower. This suggests that
the information conveyed through our storyboard translations remains
comparable to that of text translations, albeit with a slightly lower degree
of semantic similarity. Despite the storyboard translations not matching the
accuracy of the text translations perfectly, our storyboard-based data
collection method still provides translations that are reasonably accurate and
comparable.

### 5.2. Comparison of Fluency Evaluation

The human fluency evaluation presents a contrast to the accuracy evaluations.
Human evaluation results reveal a clear preference for our storyboard
translations over text translations in terms of fluency (Table 7).

In analyzing the fluency metrics, looking at the Measure of Textual Lexical
Diversity (MTLD), our storyboard translations consistently yielded higher MTLD
scores compared to text translations for Ibibio, Yorùbá, and Swahili (Table
8). This suggests that our storyboard-based data collection method can capture
a wider range of vocabulary and exhibit greater lexical diversity, enhancing
the fluency of the resulting translations in these languages.

We also evaluated the syntactic complexity of the translations using the Part-
Of-Speech (POS) perplexity metric, which provide insights into the syntactic
intricacy and sentence structure of the translations. Looking at POS
perplexity, our storyboard translations exhibited lower POS perplexity values
for Hausa and Yorùbá compared to text translations (Table 9). This suggests
that our storyboard-based data collection method achieves a more natural
sentence structure in these languages.

### 5.3. Reflections on the Storyboard Approach

The storyboard approach, while pioneering, calls for an in-depth examination
of its intrinsic strengths and limitations, especially when contrasted against
conventional text translation techniques.

Enhanced Fluency As underscored by our findings, the storyboard technique
frequently yields translations perceived as more fluent by human evaluators.
This can be ascribed to the more spontaneous elicitation process, where native
speakers articulate visual stimuli, culminating in more innate sentence
constructs.

Lexical Diversity: The method appears to encapsulate a broader lexicon,
potentially enriching the translations with diverse linguistic expressions.

Accuracy Concerns The storyboard approach, while fluent, occasionally
compromises on accuracy. This is seen in both human evaluations and cosine
similarity metrics. The omission or modification of certain nuances,
particularly named entities, can influence the perceived precision of the
translations. We have a few suggestions on how to improve this. Firstly,
refining the storyboards to provide clearer context, especially regarding
named entities, can help translators capture essential details. Secondly,
implementing post-processing techniques can further align the translations
with the source content. By incorporating correct named entities from the
source and removing extraneous information, we can achieve a better balance
between fluency and accuracy. Finally, introducing temporal separation between
viewing the source and translating can help translators produce more natural
translations, less influenced by the source language structure. By integrating
these strategies, we aim to enhance the accuracy of the storyboard-based
approach without compromising its inherent fluency.

Feasibility for Complex Texts The viability of the method for translating
intricate and detailed texts remains a point of contention. Storyboards, by
their visual essence, might fall short in capturing the depth and nuances of
elaborate narratives or technical documents.

## 6\. Future Work

Figure 2: Comparison between the DALLE-3 generated storyboard (left) and the
manually designed storyboard (right). The similarities highlight the potential
of generative AI in automating the storyboard creation process

The storyboard approach, while innovative, faces challenges in terms of the
resource-intensive nature of creating detailed storyboards. A promising
solution lies in the integration of generative AI models, such as DALLE-3. Our
initial exploration with DALLE-3 to recreate a manually designed storyboard
showed potential in automating the storyboard creation process, as evidenced
by the comparison in Figure 2. However, inconsistencies in DALLE-3’s generated
characters and the challenge of conveying complex messages visually remain
areas for improvement. As generative AI models evolve, there’s potential for
more sophisticated storyboard generation. Collaborative efforts between
linguists, artists, and AI researchers could lead to hybrid methodologies that
merge manual design with AI-driven automation. Other future work could
consider expanding the complexity of messages captured in storyboard, and
improving the accuracy of the storyboard.

## 7\. Conclusion

We introduced an alternative method for gathering translation data in low-
resource settings using storyboards and visual-based translation techniques.
Our findings, derived from both human assessments and automated metrics for
accuracy and fluency, revealed that while traditional text translations excel
in accuracy, our method offers a balance of accuracy and heightened fluency
across multiple languages. Notably, the use of visual-based translation
enriched lexical variety and fostered more organic sentence formations,
leading to translations that resonate more naturally. these findings have
implications for the development of data collection systems that can mitigate
the translationese phenomena. The potential applications and impact of our
approach include improvement in performance of machine translation task and
other tasks that will benefit from better cross-lingual training data and
transfer.

## Limitations

While our study focuses on and observes the effectiveness of the storyboard-
based data collection method and its potential to mitigate the translationese
effect, it is important to acknowledge certain limitations.

Firstly, our study focused on four low-resource African languages, namely
Swahili, Yorùbá, Hausa, and Ibibio. While we also contribute to the
development of parallel resources for these languages (including the first-
ever parallel resource for Ibibio in non-religious domain), our analysis and
findings on the translationese phenomena in the resulting dataset may not
necessarily generalize to other languages or language families. Therefore,
caution should be exercised when applying the results to different linguistic
contexts. In addition, our study specifically focused on the use of
storyboards to collect data in these low-resource languages. The effectiveness
of this method may vary in different data collection scenarios, such as with
different types of visual stimuli or in languages with different linguistic
characteristics.

Secondly, the number of annotators and the size of the dataset used in our
study were limited. Although efforts were made to mitigate bias and ensure
reliability through multiple annotators, a larger sample size could provide
more robust and representative results. In addition, the human evaluation
process involves subjective judgments made by human annotators. Individual
preferences and biases may influence the evaluation results. While we
attempted to minimize bias by using multiple annotators and consistent
sentence pairings, the subjective nature of the evaluation should be
considered.

Thirdly, in translating, the fluency and naturalness of the translated
sentences can be influenced by various external factors, such as the
annotators’ language proficiency, cultural background, and familiarity with
the subject matter. While efforts were made to select skilled annotators,
these factors may still have an impact on the results.

Furthermore, the storyboard-based data collection method, while innovative, is
inherently more challenging compared to traditional text translation. This
method involves a more complex setup for data collection, as participants need
to be effectively oriented to provide translations based on visual stimuli
rather than textual source material. These logistical and financial
considerations make the method potentially less scalable and more expensive
than traditional approaches, especially for larger datasets or multiple
languages.

Lastly, our evaluation mainly focused on accuracy and fluency, and utilized
metrics such as cosine similarity, lexical diversity, and syntactic
complexity. While these metrics provide some measures of translationese, they
may not capture all aspects of translation quality. Additional metrics or
qualitative assessments could further enhance the evaluation.

## Ethics Statement

The data collection process adhered to standard guidelines. Informed consent
was obtained from all participants involved in the study, and they were fully
informed about the purpose of the research, their rights, and how their data
would be used. Participants were assured of the anonymity and confidentiality
of their information throughout the data collection process.

## Acknowledgements

This work is supported in part by the DARPA HR001118S0044 (the LwLL program)
and the Indonesia-US Research Collaboration in Open Digital Technology grant
funded by the Indonesian Ministry of Education, Culture, Research, and
Technology. The U.S. Government is authorized to reproduce and distribute
reprints for Governmental purposes. The views and conclusions contained in
this publication are those of the authors and should not be interpreted as
representing official policies or endorsements of DARPA or the U.S.
Government.

## 8\. Bibliographical References

##

  * Alabi et al. (2022) Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022.  Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 4336–4349, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Amponsah-Kaakyire et al. (2022) Kwabena Amponsah-Kaakyire, Daria Pylypenko, Josef van Genabith, and Cristina España-Bonet. 2022.  Explaining translationese: why are neural classifiers better and what do they learn?  In _BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_.
  * Aranberri (2020) Nora Aranberri. 2020.  Can translationese features help users select an mt system for post-editing?  _Proces. del Leng. Natural_ , 64:93–100.
  * Artetxe et al. (2020) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020.  Translation artifacts in cross-lingual transfer learning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7674–7684, Online. Association for Computational Linguistics.
  * Baker et al. (1993) Mona Baker, Gill Francis, and Elena Tognini-Bonelli. 1993.  ’corpus linguistics and translation studies: Implications and applications’.  In _Text and Technology: In Honour of John Sinclair_. John Benjamins Publishing Company.
  * Barrault et al. (2018) Loïc Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018.  Findings of the third shared task on multimodal machine translation.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 304–323, Belgium, Brussels. Association for Computational Linguistics.
  * Bizzoni et al. (2020) Yuri Bizzoni, Tom S Juzek, Cristina España-Bonet, Koel Dutta Chowdhury, Josef van Genabith, and Elke Teich. 2020.  How human is machine translationese? comparing human and machine translations of text and speech.  In _Proceedings of the 17th International Conference on Spoken Language Translation_ , pages 280–290, Online. Association for Computational Linguistics.
  * Burton and Matthewson (2015) Strang Burton and Lisa Matthewson. 2015.  Targeted construction storyboards in semantic fieldwork.  _Methodologies in semantic fieldwork_ , pages 135–156.
  * Chowdhury et al. (2022) Koel Dutta Chowdhury, Rricha Jalota, Cristina España-Bonet, and Josef van Genabith. 2022.  Towards debiasing translation artifacts.
  * Dione et al. (2023) Cheikh M. Bamba Dione, David Ifeoluwa Adelani, Peter Nabende, Jesujoba Alabi, Thapelo Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue, Perez Ogayo, Anuoluwapo Aremu, Catherine Gitau, Derguene Mbaye, Jonathan Mukiibi, Blessing Sibanda, Bonaventure F. P. Dossou, Andiswa Bukula, Rooweither Mabuya, Allahsera Auguste Tapo, Edwin Munkoh-Buabeng, victoire Memdjokam Koagne, Fatoumata Ouoba Kabore, Amelia Taylor, Godson KALIPE, Tebogo Macucwa, Vukosi Marivate, Tajuddeen Gwadabe, Mboning Tchiaze Elvis, Ikechukwu Onyenwe, Gratien Atindogbe, Tolulope Adelani, Idris Akinade, Olanrewaju Samuel, Marien Nahimana, Théogène Musabeyezu, Emile Niyomutabazi, Ester Chimhenga, Kudzai Gotosa, Patrick Mizha, Apelete Agbolo, Seydou Traore, Chinedu Uchechukwu, Aliyu Yusuf, Muhammad Abdullahi, and Dietrich Klakow. 2023.  MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African languages .  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics_.
  * Elliott et al. (2016) D. Elliott, S. Frank, K. Sima’an, and L. Specia. 2016.  Multi30k: Multilingual english-german image descriptions.  In _Proceedings of the 5th Workshop on Vision and Language_ , pages 70–74.
  * Elliott et al. (2017) Desmond Elliott, Stella Frank, Loïc Barrault, Fethi Bougares, and Lucia Specia. 2017.  Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description.  In _Proceedings of the Second Conference on Machine Translation_ , Copenhagen, Denmark.
  * Elliott et al. (2016) Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016.  Multi30K: Multilingual English-German image descriptions.  In _Proceedings of the 5th Workshop on Vision and Language_ , pages 70–74, Berlin, Germany. Association for Computational Linguistics.
  * Gellerstam (1986) Martin Gellerstam. 1986.  Translationese in swedish novels translated from english.  _Translation studies in Scandinavia_ , 1:88–95.
  * Gellerstam (1996) Martin Gellerstam. 1996.  Translations as a source for cross-linguistic studies.  _Lund studies in English_ , 88:53–62.
  * Heffernan et al. (2022) Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022.  Bitext mining using distilled sentence representations for low-resource languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 2101–2112, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Hewitt et al. (2018) John Hewitt, Daphne Ippolito, Brendan Callahan, Reno Kriz, Derry Tanti Wijaya, and Chris Callison-Burch. 2018.  Learning translations via images with a massively multilingual image dataset.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2566–2576, Melbourne, Australia. Association for Computational Linguistics.
  * Khani et al. (2021) Nikzad Khani, Isidora Tourni, Mohammad Sadegh Rasooli, Chris Callison-Burch, and Derry Tanti Wijaya. 2021.  Cultural and geographical influences on image translatability of words across languages.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 198–209, Online. Association for Computational Linguistics.
  * Kunilovskaya and Lapshinova-Koltunski (2019) Maria Kunilovskaya and Ekaterina Lapshinova-Koltunski. 2019.  Translationese Features as Indicators of Quality in English-Russian Human Translation.  In _Proceedings of the Second Workshop Human-Informed Translation and Interpreting Technology associated with RANLP 2019_ , pages 47–56. Incoma Ltd., Shoumen, Bulgaria.
  * Kuwanto et al. (2023) Garry Kuwanto, Afra Feyza Akyürek, Isidora Chara Tourni, Siyang Li, Alex Jones, and Derry Wijaya. 2023.  Low-resource machine translation training curriculum fit for low-resource languages.  In _Pacific Rim International Conference on Artificial Intelligence_ , pages 453–458. Springer.
  * Landis and Koch (1977) J Richard Landis and Gary G Koch. 1977.  The measurement of observer agreement for categorical data.  _biometrics_ , pages 159–174.
  * McCarthy (2005) Philip M McCarthy. 2005.  _An assessment of the range and usefulness of lexical diversity measures and the potential of the measure of textual, lexical diversity (MTLD)_.  Ph.D. thesis, The University of Memphis.
  * McCarthy and Jarvis (2010) Philip M. McCarthy and Scott Jarvis. 2010.  MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment.  _Behavior Research Methods_ , 42(2):381–392.
  * Ni et al. (2022) Jingwei Ni, Zhijing Jin, Markus Freitag, Mrinmaya Sachan, and Bernhard Schölkopf. 2022.  Original or translated? a causal analysis of the impact of translationese on machine translation performance.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5303–5320, Seattle, United States. Association for Computational Linguistics.
  * Rasooli et al. (2021) Mohammad Sadegh Rasooli, Chris Callison-Burch, and Derry Tanti Wijaya. 2021.  “wikily” supervised neural translation tailored to cross-lingual tasks.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1655–1670, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Riley et al. (2020) Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. 2020.  Translationese as a language in “multilingual” NMT.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7737–7746, Online. Association for Computational Linguistics.
  * Santos (1995) Diana Santos. 1995.  On grammatical translationese.  In _Short papers presented at the Tenth Scandinavian Conference on Computational Linguistics_ , pages 59–66.
  * Simpson et al. (2008) Heather Simpson, Christopher Cieri, Kazuaki Maeda, Kathryn Baker, and Boyan Onyshkevych. 2008.  Human language technology resources for less commonly taught languages: Lessons learned toward creation of basic language resources.  _Collaboration: interoperability between people in the creation of language resources for less-resourced languages_ , 7:7–11.
  * Strassel and Tracey (2016) Stephanie Strassel and Jennifer Tracey. 2016.  LORELEI language packs: Data, tools, and resources for technology development in low resource languages.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 3273–3280, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Vanmassenhove et al. (2021) Eva Vanmassenhove, Dimitar Shterionov, and Matthew Gwilliam. 2021.  Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2203–2213, Online. Association for Computational Linguistics.
  * Wein and Schneider (2023) Shira Wein and Nathan Schneider. 2023.  Translationese reduction using abstract meaning representation.
  * Wijaya et al. (2017) Derry Tanti Wijaya, Brendan Callahan, John Hewitt, Jie Gao, Xiao Ling, Marianna Apidianaki, and Chris Callison-Burch. 2017.  Learning translations via matrix completion.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1452–1463, Copenhagen, Denmark. Association for Computational Linguistics.
  * Yu et al. (2022) Sicheng Yu, Qianru Sun, Hao Zhang, and Jing Jiang. 2022.  Translate-train embracing translationese artifacts.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 362–370, Dublin, Ireland. Association for Computational Linguistics.