# Tackling Ambiguity with Images:
Improved Multimodal Machine Translation and Contrastive Evaluation

Matthieu Futeral1,2 Cordelia Schmid1,2 Ivan Laptev1,2
Benoît Sagot1 Rachel Bawden1
1Inria Paris
2Département d’informatique de l’ENS, CNRS, PSL Research University
<EMAIL_ADDRESS>
###### Abstract

One of the major challenges of machine translation (MT) is ambiguity, which
can in some cases be resolved by accompanying context such as images. However,
recent work in multimodal MT (MMT) has shown that obtaining improvements from
images is challenging, limited not only by the difficulty of building
effective cross-modal representations, but also by the lack of specific
evaluation and training data. We present a new MMT approach based on a strong
text-only MT model, which uses neural adapters, a novel guided self-attention
mechanism and which is jointly trained on both visually-conditioned masking
and MMT. We also introduce CoMMuTE, a Contrastive Multilingual Multimodal
Translation Evaluation set of ambiguous sentences and their possible
translations, accompanied by disambiguating images corresponding to each
translation. Our approach obtains competitive results compared to strong text-
only models on standard English$\rightarrow$French, English$\rightarrow$German
and English$\rightarrow$Czech benchmarks and outperforms baselines and state-
of-the-art MMT systems by a large margin on our contrastive test set. Our
code111https://github.com/MatthieuFP/VGAMT and
CoMMuTE222https://github.com/MatthieuFP/CoMMuTE are freely available.

##  1 Introduction

Multimodal machine translation (MMT) typically refers to the use of additional
non-textual data in text-based machine translation (MT). Here, we focus on the
case where source texts are accompanied by images, the idea being to exploit
visual data to improve the translation of ambiguous sentences. For example, in
Figure 1, the English word glasses can either be translated as French verres
‘drinking vessels’ or lunettes ‘spectacles’, an ambiguity which is resolved
using the image.

Figure 1: Visual context resolving the ambiguity of English word glasses for
English-to-French translation.

A main research direction of MMT has been how to best exploit image
representations and combine the image and text modalities (Yin et al., 2020;
Caglayan et al., 2021; Calixto et al., 2017; Li et al., 2022). It has
typically been difficult to surpass strong text-only baselines, the image
modality often being ignored (Wu et al., 2021). A major issue holding back
progress is that most current state-of-the-art MMT models (Yin et al., 2020;
Elliott and Kádár, 2017; Wu et al., 2021; Li et al., 2022) are trained solely
on the $\sim$30k examples of the Multi30k dataset (Elliott et al., 2016),
comprising image captions and their translations. This causes two issues: (i)
the models do not exploit the large amount of text-only data available and
therefore perform poorly in comparison to state-of-the-art text-only MT
systems, and (ii) we show that very few examples require images to be
correctly translated, which means that the datasets are ill-adapted to
evaluating the use of the image modality.

In this article, we aim to overcome these problems by proposing (i) a new MMT
approach that is able to exploit (text-only) monolingual and parallel data as
well as (multimodal) captioning data, and that reaches a good balance between
maintaining high MT quality and effectively exploiting images, and (ii) a test
set, CoMMuTE, containing contrastive evaluation pairs, where images provide
the necessary context to disambiguate between multiple meanings of the same
source sentence.

Our suggested model is inspired by work on adapting frozen language models
(LMs) to multimodal inputs (Sung et al., 2022; Yang et al., 2022; Eichenberg
et al., 2021; Pfeiffer et al., 2022); we propose to adapt a strong MT model to
multimodal inputs with lightweight modules (Houlsby et al., 2019) to exploit
the large amount of textual data it was trained on. We also propose to better
exploit the image by introducing guided self-attention and by combining the
standard MMT objective with a visually-conditioned masked language modelling
(VMLM) objective (Li et al., 2019; Lu et al., 2019; Su et al., 2020). Our
model obtains competitive results compared to strong text-only baselines on
standard En$\rightarrow${Fr,De,Cs} MMT benchmarks (Elliott et al., 2016, 2017;
Barrault et al., 2018) and outperforms them and state-of-the-art MMT models on
our lexically ambiguous contrastive test set.333CoMMuTE initially contained 50
lexically ambiguous sentences in a previous version of the paper. Results are
now computed on the updated version of CoMMuTE comprising 155 ambiguous
sentences. Conclusions remain the same.

##  2 Related Work

#### Multimodal MT data.

The reference dataset to train and evaluate MMT models is Multi30k (Elliott et
al., 2016). However, recent work has shown that most MMT systems trained and
evaluated on it do not effectively exploit the image information; Elliott
(2018) showed that replacing the ground truth image with a random one does not
lead to the drop in performance that would be expected, while Wu et al. (2021)
argued that the observed gain in performance was due to a regularisation
effect. It is also notoriously difficult to beat text-only baselines on this
benchmark (Barrault et al., 2018). This may be due to (i) some subsets of
Multi30k having been translated independently from the images (Elliott et al.,
2016) and (ii) most of the time, the source text being sufficient in theory to
produce a perfect translation (i.e. the image is not necessary; see Section
5.2 for our own analysis).

Based on this, alternative test sets and evaluation methods have been
proposed. Caglayan et al. (2019) proposed to probe the use of images in MMT
models, while Li et al. (2021) proposed another training corpus and evaluation
benchmark to evaluate MMT systems, but their work is only based on gender
ambiguity and requires specific training data to train MMT models. Lala and
Specia (2018) released a lexically ambiguous MMT evaluation dataset to
evaluate models ability to disambiguate source sentences, but we found that
text context is generally sufficient to translate the evaluation dataset
correctly.

#### Contrastive MT datasets.

Another means of evaluating (and the one we adopt here) is to target specific
phenomena through the use of contrastive test sets. They involve evaluating
models based on their ability to rank pairs of translations, where one is
correct and the other incorrect. They have been used for the evaluation of
different linguistic phenomena, including grammaticality (Sennrich, 2017),
multi-sense word disambiguation (Rios Gonzales et al., 2017; Raganato et al.,
2019), pronoun translation (Müller et al., 2018; Bawden et al., 2018; Voita et
al., 2019) and lexical coherence/consistency (Bawden et al., 2018; Voita et
al., 2019). Bawden et al. (2018) introduced the idea of conditioning which of
the translations is correct depending on linguistic context, and we adopt the
same strategy here with our CoMMuTE dataset, composed of lexically ambiguous
sentences whose translations are determined by the visual context.

Figure 2: Overview of our approach, multimodal MT (MMT) (left) and visually-
conditioned masked language modeling (VMLM) (right) objectives. We train VGAMT
on both objectives jointly.

#### Adapting pretrained LMs to multimodal inputs.

A lot of progress has been made through the use of pretrained LMs (Devlin et
al., 2019; Conneau and Lample, 2019; Liu et al., 2020), often trained on raw
text for text-only models or image captioning data for multimodal ones
(Radford et al., 2021; Alayrac et al., 2022; Chen et al., 2022). One of the
most efficient ways to learn multimodal LMs is the visually-conditioned masked
language modelling (VMLM) objective (Chen et al., 2020; Lu et al., 2019; Su et
al., 2020; Li et al., 2020; Zhou et al., 2021; Huang et al., 2021a; Li et al.,
2019). Inspired by the masked language modelling (MLM) objective (Devlin et
al., 2019), it consists in randomly masking input text tokens and predicting
them conditionally based on the visual features. A lot of interest has also
been shown in lightweight modules such as adapters (Houlsby et al., 2019) to
adapt large frozen LMs to multimodal tasks (Eichenberg et al., 2021; Yang et
al., 2022; Pfeiffer et al., 2022; Tsimpoukelli et al., 2021; Sung et al.,
2022) in order to avoid catastrophic forgetting (De Lange et al., 2021). Based
on these approaches, we propose to adapt a strong text-only MT model with
lightweight modules in order to exploit the large amount of data it previously
learned.

#### Which type of visual features in MMT systems?

In terms of how images are represented in multimodal models, different
strategies exist. Many works first proposed to incorporate global visual
features from object recognition models pretrained on ImageNet (Deng et al.,
2009), such as ResNet50 (He et al., 2016), either in the form of a single
vector or a set of features (Calixto et al., 2017; Elliott and Kádár, 2017;
Calixto and Liu, 2017; Yao and Wan, 2020; Helcl et al., 2018). More recent
global features extractor such as CLIP (Radford et al., 2021) exist, but to
our knowledge have not been used in MMT models. Extending this idea, other
works focused on entities in the image and extracted bounding boxes using a
pretrained Faster R-CNN (Ren et al., 2015) in order to introduce more semantic
visual information into MT (Grönroos et al., 2018; Ive et al., 2019; Caglayan
et al., 2021). Recent efforts have been made to only select parts of the image
that are relevant to the translation of the sentence. Some proposed to use a
more selective attention mechanism between modalities (Liu et al., 2021; Ye et
al., 2022), while others suggested extracting other types of visual features
(Huang et al., 2021b; Fang and Feng, 2022). Based on this, Yin et al. (2020)
decided to exploit local image-text correspondences in their model Graph-MMT.
Similar to their approach, we use a simpler method to extract relevant visual
features, using the output queries from a state-of-the-art free-form text
object detector MDETR (Kamath et al., 2021) as our local visual features (in
addition to global features from CLIP).

##  3 Our approach: VGAMT

The two main aims of our approach are to (i) exploit a maximum available data
(not just multimodal parallel text data) and to (ii) provide an effective way
to combine image and text modalities. Our approach, shown in Figure 2,
consists in taking a strong text-only MT model444In practice, our starting
point is mBART, which we fine-tune on a large parallel corpus (see Section 5
for more details). and adapting it to multimodal MT. To adapt this strong
text-only model to multimodal inputs, we add several lightweight
modules—bottleneck adapters (Houlsby et al., 2019) and linear visual
projection layers—to the otherwise frozen initial model. The bottleneck
adapters are lightweight linear layers introduced after each attention block
and each feed-forward layer to project embeddings down before projecting them
up.

Figure 3: Example of an MDETR alignment matrix.

In terms of representing visual information, we choose to use two types of
representation. We concatenate local (MDETR) features and global (CLIP)
features to the text inputs. We choose to use global features too, since the
source sentence can describe more general aspects of the image than mere
objects (such as scenes). We jointly train the non-frozen parts of our model
on two distinct objectives: multimodal MT (MMT) and visually-conditioned
masked language modelling (VMLM), as described in Section 3.1. We also
introduce a guided self-attention to exploit image information in a
straightforward manner (see Section 3.2) in the encoder (while the decoder
uses regular self- and cross-attentions and can only attend to embeddings
related to text positions). We call our approach Visually Guided and Adapted
Machine Translation (VGAMT).

###  3.1 Combining training objectives

As shown in Figure 2, we jointly train VGAMT on two objectives: visual masked
language modelling (VMLM) and multimodal MT (MMT). VMLM (resp. MMT) consists
in predicting masked tokens (resp. translating the sentence) conditioned on
the image.555During training, we randomly draw batches from a parallel
multimodal dataset (for MMT) and a monolingual multimodal one (for VMLM) with
equal probability. The use of the VMLM objective in addition to MMT ensures
that the model does not learn to ignore the visual inputs when translating
(since Multi30k is mainly composed of very standard and unambiguous parallel
sentences). We make sure to mask a high percentage (25%) of the text inputs so
that the model is forced to attend to the image when producing translations.

###  3.2 Guided self-attention

The backbone of VGAMT is an encoder-decoder MT model, in which image features
are concatenated to textual input embeddings and shared self-attention is used
over the two input modalities (see Figure 2). Instead of using full self-
attention (Caglayan et al., 2021) (connections between all image parts and all
text tokens), we introduce guided self-attention. Guided self-attention
consists in masking irrelevant connections between text and image
representations; each text (resp. image) embedding can attend to itself and
all other text (resp. image) positions, but can only attend to image (resp.
text) positions conditioned on pre-extracted text-image alignments. We obtain
these alignments (in the form of a cross-modal correspondence matrix) using
MDETR (Kamath et al., 2021), which detects image regions and corresponding
text spans based on a free-form text (see Figure 3 and Appendix B for more
details).

Concretely, let Q, K and V denote the learnable query, key and value
parameters of a standard self-attention mechanism. Attention can be defined as
$\text{Attention}(Q,K,V)=A\cdot V$, where the attention matrix $A=(a_{ij})$ is
defined as $A=\text{softmax}\left(QK^{T}/\sqrt{d_{k}}\right)$, where $d_{k}$
is the dimension of the key vector, i.e.:

$\displaystyle
a_{ij}=\frac{e^{Q_{i}K_{j}^{T}/\sqrt{d_{k}}}}{\sum_{l}e^{Q_{i}K_{l}^{T}/\sqrt{d_{k}}}}$
(1)

The idea behind our guided self-attention mechanism is that we want to allow
subwords to attend to all subwords, all bounding boxes to attend to all
bounding boxes, but to only allow cross-modal attention between a subword and
bounding boxes that are linked by MDETR (see Figure 3). We therefore define a
binary masking matrix $C=(c_{ij})$ where (i) $c_{ij}=1$ if indices $i$ and $j$
correspond to embeddings coming from the same modality, and (ii) $c_{ij}$ is
provided by the MDETR matrix otherwise: it is $1$ if MDETR has created a link
between subword (resp. bounding box) $i$ and bounding box (resp. subword) $j$.
Once this guiding matrix $C$ is defined, we can replace the standard attention
(1) with our guided attention:

$\displaystyle
a_{ij}=\frac{c_{ij}e^{Q_{i}K_{j}^{T}/\sqrt{d_{k}}}}{\sum_{l}c_{il}e^{Q_{i}K_{l}^{T}/\sqrt{d_{k}}}}.$
(2)

The main advantage of guided self-attention over full self-attention is that
the model does not have to learn to ignore irrelevant text-image
correspondences since alignments are introduced as a prior.

##  4 Contrastive Multilingual Multimodal Translation Evaluation (CoMMuTE)

To overcome the flaws of existing benchmarks (see Section 5.2), we introduce
CoMMuTE, a Contrastive Multilingual Multimodal Translation Evaluation
dataset666CoMMuTE is distributed under Creative Commons Attribution Share
Alike 4.0 International license.. It is composed of 155 lexically ambiguous
sentences in English, each associated with two translations corresponding to
two of the possible meanings of each sentence and two images that determine
which of the translations is correct. It covers English$\rightarrow$French,
English$\rightarrow$German and English$\rightarrow$Czech. An example is given
in Figure 4.

#### Data collection.

The test set contains 155 ambiguous sentences constructed around 155 lexically
ambiguous words: 29 of the examples are from Bawden et al. (2018), and we
created the remaining ones.777We could not take the entirety of the examples
in (Bawden et al., 2018) as some examples were not adapted to disambiguation
using visual (as opposed to linguistic) context. We collected two images for
each sentence under Creative Commons license (either Google Images or our own
photos), so that the image illustrates without ambiguity one of the two
meanings of the sentence. We do not restrict the image-text relation to be
strictly descriptive (as for image captions) in order to have a more general
evaluation dataset. Each sentence was translated into two possible
translations (each corresponding to one of the images) by a native speaker of
the target language. Appendix A provides some basic statistics.

The idea of CoMMuTE is to use MMT models to rank each of the two translations
based on image information. The perplexity of a sentence for a given model is
defined as: $PPL_{q}(y)=\prod_{i=1}^{N}q(y_{i})^{-\frac{1}{N}}$, where $q$ is
the probability distribution output by the model, N is the sequence length and
$y_{1},\dots,y_{N}$ is the sequence of tokens. Now, let
$y_{1},\dots,y_{N_{1}}$ be the sequence of tokens of the correct translation
and $y^{\prime}_{1},\dots,y^{\prime}_{N_{2}}$ the sequence of tokens of the
incorrect translation, a model makes a correct prediction if $PPL_{q}(y)\leq
PPL_{q}(y^{\prime})$. i.e. the model considers the correct translation more
likely than the incorrect one. For each example, we rank each of the
translations based on each of the images (2 comparisons per example), and
report the accuracy over all the examples. As CoMMuTE is perfectly balanced, a
text-only model will get exactly 50% accuracy on this task.

| En$\rightarrow$Fr | En$\rightarrow$De | En$\rightarrow$Cs
---|---|---|---
| size | #sents. | size | #sents. | size | #sents.
OpenSubtitles | 2.2GB | 24.2M | 1.2GB | 13.1M | 2.2GB | 24.7M
Ted Talks | 108MB | 535K | 83MB | 414K | 30MB | 158K
Books | 29MB | 119K | 12MB | 47K | -
Wikipedia | 187MB | 769K | 493MB | 2.2M | 3.2MB | 19K
Total | 2.5GB | 25.6M | 1.8GB | 15.8M | 2.2GB | 24.9M

Table 1: Parallel corpus sizes. Figure 4: Example from CoMMuTE. The English
word ‘mole’ can refer either to ‘a small dark, raised lump on a the skin’ (1)
or ‘a small burrowing mammal’ (2).

##  5 Experiments

###  5.1 Text-only data

All our experiments are based on the strong MT model mBART888mBART is
pretrained on CC25 (Wenzek et al., 2020). (Liu et al., 2020), which we fine-
tune on parallel text (see Table 1). We use
OpenSubtitles2018999http://www.opensubtitles.org (Lison et al., 2018),
Wikipedia (Wołk and Marasek, 2014), Ted Talks (Reimers and Gurevych, 2020) and
the Books datasets (Tiedemann, 2012). We preprocess the data using Moses
scripts (Koehn et al., 2007).101010remove-non-printing-char.pl, normalization-
punctuation.pl, and clean-corpus-n.pl (4-100 tokens).

###  5.2 Multimodal data

| Test2016 | Test2017 | MSCOCO
---|---|---|---
Ambiguous (%) | 21 (2.1%) | 20 (2%) | 6 (1.3%)

Table 2: Number (and percentage) of ambiguous examples in the
En$\rightarrow$Fr test sets.

#### Multi30k.

We train our frozen MT model on the Multi30k dataset (Specia et al., 2016;
Elliott et al., 2016) composed of English sentences, each accompanied by an
image and French, German and Czech translations. It contains 29k train, 1014
dev and 1000 test examples (Test2016). Elliott et al. (2017) and Barrault et
al. (2018) released two additional related test sets (Test2017 and Ambiguous
Coco). However, on analysis of these sets and as shown in Table 2, we found
that very few examples are image-dependent (i.e. the source sentence is
ambiguous and the image is required to solve the ambiguity in the target
language),111111We queried WordNet (Fellbaum, 1998) for all nouns and verbs in
the English sentences. An example was considered image-dependent if, on manual
assessment, there were multiple meanings, which were (i) compatible with the
text context and (ii) could be disambiguated using the image. meaning that an
MMT system is unlikely to perform better than a text-only system. Moreover,
most of these ambiguities are semantically similar and they only cover a few
multi-sense words. Although Ambiguous Coco (Elliott et al., 2017) is designed
to be an ambiguous test set as it is built around multi-sense verbs, it was
automatically created from sentences from MSCOCO (Lin et al., 2014) for which
the textual context is often sufficient for disambiguation. These benchmarks
remain useful to make sure MMT systems do not perform worse than text-only MT
models on examples where images are not necessary to translate correctly.
However, we consider them insufficient to assess how well MMT systems exploit
images to improve translation.

#### Monolingual multimodal data.

For the VMLM objective, we train our model on the Conceptual Captions (CC)
dataset (Sharma et al., 2018) composed of 3.3M121212At the time of writing, we
were able to collect $\sim$2M images and trained models on this subset. images
aligned with English text.

###  5.3 Implementation details

For all our experiments, we use the mBART implementation from Hugging Face
(Wolf et al., 2020). Experiments with adapters used bottleneck adapters
(Houlsby et al., 2019) with a reduction factor of 8 and ReLU activation
(Agarap, 2018). We use the implementation provided by adapter-transformers
(Pfeiffer et al., 2020). We use a batch size of 512, the Adam optimiser
(Kingma and Ba, 2014) with $\beta_{1}=0.9$, $\beta_{2}=0.99$ and a learning
rate of $10^{-4}$ for En$\rightarrow$Fr and $10^{-5}$ for
En$\rightarrow${De,Cs}. We also applied 0.1 label smoothing (Szegedy et al.,
2016) during training. We selected our final model according to the best BLEU
score (Papineni et al., 2002) on the Multi30k dev set after at least one full
pass over the Multi30k and Conceptual Captions training sets. We ran each
experiment 3 times with different seeds and report the average
BLEU131313computed with the default parameters of sacreBLEU v2.0.0 from
https://github.com/mjpost/sacrebleu. (Papineni et al., 2002) and COMET (Rei et
al., 2020) scores141414Using the wmt20-comet-da model. and the standard
errors. We also report METEOR scores (Banerjee and Lavie, 2005) in Appendix E.
All experiments were carried out on 8 NVIDIA V100 GPUs for $\sim$15h.

###  5.4 Baselines

We consider several text-only and multimodal baselines. All baselines except
the MT models fine-tuned from mBART were trained from scratch with the
original codebases and features released by the papers’ authors. Models
trained on the (multimodal) MT objective only where trained on Multi30k, while
models jointly trained on the (multimodal) MT and (V)MLM objectives were
trained on Multi30k and Conceptual Captions.

#### Text-only.

We trained a text-only Seq2seq Transformer (Vaswani et al., 2017) from scratch
and a text-only Seq2Seq Transformer initialised from TLM weights (Conneau and
Lample, 2019). We refer to these models as Vanilla MT and TLM + MT
respectively. We also trained several MT models initialised from pretrained
mBART (Liu et al., 2020) and which we fine-tuned on parallel data (Lison et
al., 2018; Wołk and Marasek, 2014). We refer to these models as mBART + MT.
‘w/ adapters’ specifies that the model’s weights are frozen except bottleneck
adapters (Houlsby et al., 2019).

#### Multimodal.

We trained several state-of-the-art multimodal MT models: Graph-MMT (Yin et
al., 2020), Gated Fusion (Wu et al., 2021) and a Seq2Seq Transformer trained
from VTLM weights (Caglayan et al., 2021) (hereafter VTLM + MMT).

###  5.5 Results and Analysis

Figure 5: Example from Test2017 with English biker, translated as French
‘cycliste’ (cyclist) or ‘motard’ (motorcyclist). VGAMT succeeds where the
baseline fails. Figure 6: VGAMT Perplexity scores on a CoMMuTE example,
illustrating that it is able to correctly rank each of the French translations
of ambiguous English bucks ‘male deer or dollars’ when conditioning on the
image.

En$\rightarrow$Fr
---
|  | # trainable | Test2016 | Test2017 | MSCOCO | CoMMuTE
Model | Objectives | params | BLEU | COMET | BLEU | COMET | BLEU | COMET | Accuracy
Text-only Machine Translation
Vanilla MT* | NMT | 4.0M | 59.4 ±0.2 | 0.711 ±0.004 | 51.6 ±0.2 | 0.568 ±0.009 | 41.2 ±0.4 | 0.403 ±0.005 |  50.0 ±0.0
TLM + MT* | NMT | 42M | 62.0 ±0.1 | 0.795 ±0.002 | 54.2 ±0.2 | 0.681 ±0.002 | 43.6 ±0.2 | 0.542 ±0.009 |  50.0 ±0.0
mBART + MT* | - | - | 49.0 ±0.0 | 0.819 ±0.000 | 48.1 ±0.0 | 0.779±0.000 | 47.0 ±0.0 | 0.733 ±0.000 |  50.0 ±0.0
mBART + MT* w/ adapters | NMT + MLM | 12.6M |  67.2 ±0.3 |  0.971 ±0.005 | 61.5 ±0.3 | 0.918 ±0.004 |  51.5 ±0.7 |  0.832 ±0.006 |  50.0 ±0.0
Multimodal Machine Translation
Graph-MMT* | MMT | 4.0M | 58.9 ±0.5 | 0.705 ±0.004 | 51.5 ±0.2 | 0.589 ±0.005 | 41.0 ±0.6 | 0.387 ±0.013 | 50.2 ±3.5
Gated Fusion* | MMT | 2.8M | 58.7 ±0.3 | 0.707 ±0.002 | 50.8 ±0.7 | 0.580 ±0.011 | 40.4 ±0.4 | 0.394 ±0.013 | 50.0 ±0.8
VTLM + MMT* | MMT | 44M | 61.4 ±0.2 | 0.783 ±0.005 | 53.6 ±0.1 | 0.672 ±0.005 | 43.4 ±0.3 | 0.500 ±0.006 | 50.1 ±0.3
VGAMT (ours) | MMT + VMLM | 13.2M |  67.2 ±0.1 | 0.968 ±0.002 |  61.6 ±0.1 |  0.921 ±0.002 | 51.1 ±0.6 | 0.811 ±0.003 |  67.1 ±0.7
En$\rightarrow$De
Text-only Machine Translation
Vanilla MT* | NMT | 4.1M | 38.5 ±0.3 | 0.394 ±0.005 | 30.3 ±0.5 | 0.259 ±0.012 | 27.8 ±0.4 | 0.092 ±0.018 |  50.0 ±0.0
TLM + MT* | NMT | 42M | 40.0 ±0.2 | 0.457 ±0.006 | 31.5 ±0.1 | 0.341 ±0.002 | 29.4 ±0.3 | 0.152 ±0.015 |  50.0 ±0.0
mBART + MT* | - | - | 36.2 ±0.0 | 0.595 ±0.000 | 32.3 ±0.0 | 0.506 ±0.000 | 27.6 ±0.0 | 0.383 ±0.000 |  50.0 ±0.0
mBART + MT* w/ adapters | NMT + MLM | 12.6M |  43.6 ±0.2 |  0.697 ±0.003 |  38.9 ±0.5 |  0.664 ±0.002 |  36.2 ±0.2 |  0.574 ±0.004 |  50.0 ±0.0
Multimodal Machine Translation
Graph-MMT* | MMT | 4.1M | 38.6 ±0.3 | 0.368 ±0.011 | 29.0 ±0.5 | 0.226 ±0.010 | 25.9 ±0.8 | 0.060 ±0.027 | 49.1 ±1.5
Gated Fusion* | MMT | 2.8M | 38.7 ±0.2 | 0.378 ±0.007 | 29.5 ±0.2 | 0.236 ±0.018 | 26.6 ±0.3 | 0.055 ±0.016 | 49.7 ±0.6
VTLM + MMT* | MMT | 44M | 39.4 ±0.2 | 0.439 ±0.004 | 30.7 ±0.2 | 0.322 ±0.005 | 28.2 ±0.2 | 0.168 ±0.014 | 50.0 ±0.2
VGAMT (ours) | MMT + VMLM | 13.2M | 43.3 ±0.2 | 0.694 ±0.003 | 38.3 ±0.2 | 0.653 ±0.005 | 35.7 ±0.3 | 0.544 ±0.006 |  59.0 ±0.5

Table 3: Results for En$\rightarrow${Fr,De} (average of three runs). The best
result is indicated in bold. * means the results were retrained by using the
original codebase provided by the authors of the paper.

En$\rightarrow$Cs
---
| Test2016 | Test2018 | CoMMuTE
Model | BLEU | COMET | BLEU | COMET | Accuracy
Text-only Machine Translation
Vanilla MT* | 31.3 ±0.0 | 0.593 ±0.008 | 26.0 ±0.2 | 0.379 ±0.008 |  50.0 ±0.0
TLM + MT* | 32.6 ±0.1 | 0.642 ±0.002 | 26.8 ±0.2 | 0.432 ±0.006 |  50.0 ±0.0
mBART + MT* | 32.1 ±0.0 | 0.865 ±0.000 | 29.6 ±0.0 | 0.747 ±0.000 |  50.0 ±0.0
w/ adapters | 37.3 ±0.1 |  0.940 ±0.005 |  35.2 ±0.4 |  0.876 ±0.002 |  50.0 ±0.0
Multimodal Machine Translation
Graph-MMT* | 30.8 ±0.4 | 0.562 ±0.011 | 24.9 ±0.5 | 0.344 ±0.011 | 49.2 ±1.8
Gated Fusion* | 30.8 ±0.4 | 0.560 ±0.014 | 25.8 ±0.1 | 0.342 ±0.008 | 51.0 ±1.9
VTLM + MMT* | 32.0 ±0.3 | 0.621 ±0.010 | 26.7 ±0.2 | 0.419 ±0.015 | 50.0 ±0.3
VGAMT (ours) |  37.6 ±0.2 | 0.934 ±0.004 | 34.2 ±0.1 | 0.833 ±0.003 |  55.6 ±0.8

Table 4: Results for En$\rightarrow$Cs (average of three runs). Same
formatting as for Table 3.

| Test2016 | Test2017 | MSCOCO | CoMMuTE
---|---|---|---|---
Model | BLEU | COMET | BLEU | COMET | BLEU | COMET | Accuracy
Text-only Machine Translation
mBART + MT* w/ adapters | 67.2 ±0.3 | 0.971 ±0.005 | 61.5 ±0.3 | 0.918 ±0.004 |  51.5 ±0.7 |  0.832 ±0.006 |  50.0 ±0.0
w/o MLM objective |  67.7 ±0.3 | 0.970 ±0.004 | 61.5 ±0.1 |  0.926 ±0.004 | 50.3 ±0.4 | 0.821 ±0.002 |  50.0 ±0.0
Multimodal Machine Translation
VGAMT (ours) | 67.2 ±0.1 | 0.968 ±0.002 |  61.6 ±0.1 | 0.921 ±0.002 | 51.1 ±0.6 | 0.811 ±0.003 |  67.1 ±0.7
unfrozen w/o adapters | 66.9 ±0.7 | 0.965 ±0.003 | 61.4 ±0.6 | 0.912 ±0.009 | 50.3 ±0.7 | 0.814 ±0.011 | 60.5 ±3.8
w/o VMLM objective |  67.7 ±0.2 |  0.976 ±0.001 | 61.4 ±0.2 | 0.920 ±0.003 | 50.5 ±0.0 | 0.809 ±0.004 | 52.0 ±1.2
w/o guided self-attention | 67.0 ±0.2 | 0.963 ±0.004 | 60.8 ±0.3 | 0.910 ±0.006 | 50.3 ±0.5 | 0.792 ±0.004 | 64.6 ±1.6
w/ pretraining (w/o co-training) | 66.2 ±0.1 | 0.950 ±0.001 | 59.3 ±0.1 | 0.875 ±0.003 | 49.2 ±0.2 | 0.777 ±0.001 | 63.3 ±0.5
w/o MDETR features | 66.7 ±0.5 | 0.967 ±0.004 | 61.1 ±0.1 | 0.912 ±0.002 | 51.0 ±0.6 | 0.810 ±0.003 | 63.0 ±1.2
w/o CLIP features | 66.4 ±0.8 | 0.959 ±0.008 | 60.4 ±0.7 | 0.909 ±0.002 | 51.0 ±0.6 | 0.810 ±0.008 | 50.3 ±0.0

Table 5: Results of the ablation studies described in Section 6
(En$\rightarrow$Fr test set). The best result is indicated in bold.

Tables 3 and 4 show BLEU, COMET and accuracy scores for all models compared on
several En$\rightarrow${Fr,De,Cs} test sets including CoMMuTE. An initial
observation is that the text-only model is a strong baseline on the three
standard benchmarks (Test2016, Test2017 and MSCOCO). As mentioned in Section
5.2, most of these evaluation datasets do not need visual context to be
correctly translated. Our model VGAMT is on average on par with its
counterpart text-only mBART+MT w/ adapters baseline for all Multi30k
En$\rightarrow$Fr test sets, while being on average just below this baseline
on En$\rightarrow${De,Cs} Multi30k benchmarks. It outperforms other MMT models
with a large margin due to both the effective use of textual knowledge from
the frozen MT model but also guided self-attention. Note that the scores
reported for the baselines are lower than the ones reported in the original
papers of the models for several reasons. First, we computed the scores on
fully detokenised data to have a uniform evaluation between all models. We
also report the average score from three different runs using different seeds
and not the best score obtained over a single run.

More importantly, our VGAMT obtains strong improvements over both text-only
baselines and state-of-the-art MMT systems on CoMMuTE; our model can use
visual context to disambiguate sentences. This can be seen in Figure 5 (one of
the ambiguous examples from Multi30k), where in contrast to the baseline VGAMT
produces the correct translation and Figure 6 (from CoMMuTE), where VGAMT
correctly ranks the two translations. More examples are provided in Appendix
D. We also propose to translate CoMMuTE source sentences and compare against
the reference translations; the results are shown in Appendix F.

##  6 Ablation Study

To better understand the role of VGAMT’s components, we carry out several
ablations for En$\rightarrow$Fr and report all results in Table 5.

#### Adapters versus Fine-tuning.

We compare the results of fine-tuning an unfrozen VGAMT model (w/o adapters)
in comparison to our frozen model with adapters (VGAMT), all other things
remaining equal. The unfrozen version faces a drop in scores on all test sets
except Test2017. Notably, the unfrozen model’s accuracy score of 60.5 on
CoMMuTE is 6.6 points lower than our final VGAMT model. As well as providing a
more lightweight solution that does not involve fine-tuning all parameters,
using neural adapters and freezing other weights is useful in terms of
performance.

#### Impact of the VMLM objective.

To evaluate the impact of jointly training with MMT and VMLM objectives, we
train a model on the MMT without VMLM (and therefore without monolingual
multimodal data). The MMT model trained on MMT alone obtains 52.0 on CoMMuTE,
compared to 67.1 for joint training, showing that VMLM helps our model to
better exploit disambiguating images.

#### Guided self-attention.

We study the impact of guided self-attention between modalities by comparing
against classic full self-attention. Guided self-attention obtains better
results than full self-attention, particularly on Test2017 and MSCOCO (+0.8
BLEU, +0.015 COMET on average). It also gets better results on CoMMuTE (+2.5
points). See Appendix C for analysis of guided attention scores.

#### VMLM and MMT joint training.

We compare our VMLM and MMT joint training with disjoint training where VGAMT
is first pretrained on VMLM then fine-tuned on MMT instead of co-training on
both VMLM and MMT. Table 5 shows that it results in a large drop of
performance on all scores in average including 3.8 points on CoMMuTE.

#### MDETR.

We examine the impact of MDETR features by training a model without
them.151515More details are available in Appendix B. The results without MDETR
features are slightly lower than the full model on standard MMT benchmarks.
However, the results are significantly lower on CoMMuTE (63.0±1.2 without
MDETR features and 67.1±0.7 with MDETR features). This means that VGAMT
benefits from MDETR features when disambiguating and translating sentences.

#### CLIP.

We also study the impact of CLIP features by training a model without them.15
Including CLIP features gives slightly higher results on standard MMT
benchmarks (+0.69 BLEU and +0.007 COMET scores on average on all benchmarks).
VGAMT without CLIP features faces an extreme drop on CoMMuTE (50.3±0.00 w/o
CLIP features vs. 67.1±0.7 w/ CLIP features), which shows that CLIP features
are required for disambiguation.

#### VMLM sampling probability and degree of masking.

We ran experiments to vary the VMLM sampling probability (see Section 3.1) and
the percentage of masked text inputs (see Figure 7 for results on CoMMuTE).
For the sampling between VMLM and MMT objectives, the maximum value is reached
for $p=$50%, i.e. equal sampling between VMLM and MMT objectives (Figure
7(a)). Similar results are obtained for $p=75\%$, i.e. 3 VMLM batches for 1
MMT batch, but the translation quality is lower. For the percentage of
masking, there is a peak at 25% masked text inputs and a constant decrease for
higher values (Figure 7(b)).

(a) Variation of VMLM sampling probability with fixed 25% masked text inputs.
(b) Variation of masked text inputs with fixed 50% VMLM sampling probability.

Figure 7: CoMMuTE Results comparing multiple VMLM sampling probabilities and
percentage of masked text inputs. 95% confidence interval in grey.

##  7 Conclusion

We propose a new MMT approach (VGAMT) based on (i) adapting a strong text-only
MT model with lightweight adapters and (ii) introducing better use of the text
and image modalities through a novel guided self-attention mechanism and joint
MMT and VMLM training. We also introduce CoMMuTE, a contrastive test set
designed to test the use of visual disambiguating context. Results for
En$\rightarrow${Fr,De,Cs} show that VGAMT obtains competitive results compared
with strong text-only baselines on standard benchmarks and widely outperforms
these baselines and state-of-the-art MMT systems on CoMMuTE.

## Limitations

In this work, we focused on En$\rightarrow${Fr,De,Cs} multimodal MT. At the
time of writing, our method can only be applied for En$\rightarrow$X MMT. It
is indeed necessary to have access to a modulated object detector in the
source language to extract the features and the image-text relationship
exploited by our model. This type of modulated object detector is only
available in English for the moment. We leave the extension of our method to
non-English source languages to future work. Moreover, our method requires
large amount of captioning data to perform well. It is therefore
computationally expensive.

## Acknowledgements

This work was granted access to the HPC resources of IDRIS under the
allocation 2022-AD011013908 and 2022-AD011012254 made by GENCI. It was also
partly funded by the last four authors’ chairs in the PRAIRIE institute funded
by the French national agency ANR as part of the “Investissements d’avenir”
programme under the reference ANR-19-P3IA-0001.

## References

  * Agarap (2018) Abien Fred Agarap. 2018.  Deep learning using rectified linear units (ReLU).  _arXiv preprint arXiv:1803.08375_.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. 2022.  Flamingo: a visual language model for few-shot learning.  _arXiv preprint arXiv:2204.14198_.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Barrault et al. (2018) Loïc Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018.  Findings of the third shared task on multimodal machine translation.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 304–323, Belgium, Brussels. Association for Computational Linguistics.
  * Bawden et al. (2018) Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018.  Evaluating discourse phenomena in neural machine translation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1304–1313, New Orleans, Louisiana. Association for Computational Linguistics.
  * Caglayan et al. (2021) Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, and Lucia Specia. 2021.  Cross-lingual visual pre-training for multimodal machine translation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1317–1324, Online. Association for Computational Linguistics.
  * Caglayan et al. (2019) Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Loïc Barrault. 2019.  Probing the need for visual context in multimodal machine translation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4159–4170, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Calixto and Liu (2017) Iacer Calixto and Qun Liu. 2017.  Incorporating global visual features into attention-based neural machine translation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 992–1003, Copenhagen, Denmark. Association for Computational Linguistics.
  * Calixto et al. (2017) Iacer Calixto, Qun Liu, and Nick Campbell. 2017.  Doubly-attentive decoder for multi-modal neural machine translation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1913–1924, Vancouver, Canada. Association for Computational Linguistics.
  * Chen et al. (2022) Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. 2022.  Pali: A jointly-scaled multilingual language-image model.  _arXiv preprint arXiv:2209.06794_.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Uniter: Universal image-text representation learning.  In _European conference on computer vision_ , pages 104–120. Springer.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * De Lange et al. (2021) Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(7):3366–3385.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 248–255.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _Proceedings of the 9th International Conference on Learning Representations_. OpenReview.net.
  * Eichenberg et al. (2021) Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. 2021.  Magma–multimodal augmentation of generative models through adapter-based finetuning.  _arXiv preprint arXiv:2112.05253_.
  * Elliott (2018) Desmond Elliott. 2018.  Adversarial evaluation of multimodal machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2974–2978, Brussels, Belgium. Association for Computational Linguistics.
  * Elliott et al. (2017) Desmond Elliott, Stella Frank, Loïc Barrault, Fethi Bougares, and Lucia Specia. 2017.  Findings of the second shared task on multimodal machine translation and multilingual image description.  In _Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers_ , pages 215–233, Copenhagen, Denmark. Association for Computational Linguistics.
  * Elliott et al. (2016) Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016.  Multi30K: Multilingual English-German image descriptions.  In _Proceedings of the 5th Workshop on Vision and Language_ , pages 70–74, Berlin, Germany. Association for Computational Linguistics.
  * Elliott and Kádár (2017) Desmond Elliott and Ákos Kádár. 2017.  Imagination improves multimodal translation.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 130–141, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Fang and Feng (2022) Qingkai Fang and Yang Feng. 2022.  Neural machine translation with phrase-level universal visual representations.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5687–5698, Dublin, Ireland. Association for Computational Linguistics.
  * Fellbaum (1998) Christiane Fellbaum. 1998.  _WordNet 1.6: An Electronic Lexical Database_.  Bradford Books. MIT Press.
  * Grönroos et al. (2018) Stig-Arne Grönroos, Benoit Huet, Mikko Kurimo, Jorma Laaksonen, Bernard Merialdo, Phu Pham, Mats Sjöberg, Umut Sulubacak, Jörg Tiedemann, Raphael Troncy, and Raúl Vázquez. 2018.  The MeMAD submission to the WMT18 multimodal translation task.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 603–611, Belgium, Brussels. Association for Computational Linguistics.
  * He et al. (2016) Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  _In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778.
  * Helcl et al. (2018) Jindřich Helcl, Jindřich Libovický, and Dušan Variš. 2018.  CUNI system for the WMT18 multimodal translation task.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 616–623, Belgium, Brussels. Association for Computational Linguistics.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _International Conference on Machine Learning_ , pages 2790–2799. PMLR.
  * Huang et al. (2021a) Haoyang Huang, Lin Su, Di Qi, Nan Duan, Edward Cui, Taroon Bharti, Lei Zhang, Lijuan Wang, Jianfeng Gao, Bei Liu, Jianlong Fu, Dongdong Zhang, Xin Liu, and Ming Zhou. 2021a.  M3P: Learning universal representations via multitask multilingual multimodal pre-training.  _In 2021 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3976–3985.
  * Huang et al. (2021b) Xin Huang, Jiajun Zhang, and Chengqing Zong. 2021b.  Entity-level cross-modal learning improves multi-modal machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1067–1080, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ive et al. (2019) Julia Ive, Pranava Madhyastha, and Lucia Specia. 2019.  Distilling translations with visual awareness.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6525–6538, Florence, Italy. Association for Computational Linguistics.
  * Kamath et al. (2021) Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. 2021.  MDETR - modulated detection for end-to-end multi-modal understanding.  _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 1760–1770.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kobayashi et al. (2020) Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020.  Attention is not only a weight: Analyzing transformers with vector norms.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7057–7075, Online. Association for Computational Linguistics.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007\.  Moses: Open source toolkit for statistical machine translation.  In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions_ , pages 177–180, Prague, Czech Republic. Association for Computational Linguistics.
  * Lala and Specia (2018) Chiraag Lala and Lucia Specia. 2018.  Multimodal lexical translation.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Li et al. (2021) Jiaoda Li, Duygu Ataman, and Rico Sennrich. 2021.  Vision matters when it should: Sanity checking multimodal machine translation models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8556–8562, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019\.  VisualBERT: A simple and performant baseline for vision and language.  _arXiv preprint arXiv:1908.03557_.
  * Li et al. (2020) Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _European Conference on Computer Vision_ , pages 121–137. Springer.
  * Li et al. (2022) Yi Li, Rameswar Panda, Yoon Kim, Chun-Fu (Richard) Chen, Rogerio Feris, David Cox, and Nuno Vasconcelos. 2022.  VALHALLA: Visual Hallucination for Machine Translation.  In _2022 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5216–5226.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft COCO: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer.
  * Lison et al. (2018) Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018.  OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Liu et al. (2021) Pengbo Liu, Hailong Cao, and Tiejun Zhao. 2021.  Gumbel-attention for multi-modal machine translation.  _arXiv preprint arXiv:2103.08862_.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Müller et al. (2018) Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018.  A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 61–72, Brussels, Belgium. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Pfeiffer et al. (2022) Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. 2022.  xGQA: Cross-lingual visual question answering.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2497–2511, Dublin, Ireland. Association for Computational Linguistics.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020.  AdapterHub: A framework for adapting transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 46–54, Online. Association for Computational Linguistics.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR.
  * Raganato et al. (2019) Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. 2019.  The MuCoW test suite at WMT 2019: Automatically harvested multilingual contrastive word sense disambiguation test sets for machine translation.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 470–480, Florence, Italy. Association for Computational Linguistics.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4512–4525, Online. Association for Computational Linguistics.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster R-CNN: Towards real-time object detection with region proposal networks.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.
  * Rios Gonzales et al. (2017) Annette Rios Gonzales, Laura Mascarell, and Rico Sennrich. 2017.  Improving word sense disambiguation in neural machine translation with sense embeddings.  In _Proceedings of the Second Conference on Machine Translation_ , pages 11–19, Copenhagen, Denmark. Association for Computational Linguistics.
  * Sennrich (2017) Rico Sennrich. 2017.  How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 376–382, Valencia, Spain. Association for Computational Linguistics.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565, Melbourne, Australia. Association for Computational Linguistics.
  * Specia et al. (2016) Lucia Specia, Stella Frank, Khalil Sima’an, and Desmond Elliott. 2016.  A shared task on multimodal machine translation and crosslingual image description.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 543–553, Berlin, Germany. Association for Computational Linguistics.
  * Su et al. (2020) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020\.  Vl-bert: Pre-training of generic visual-linguistic representations.  In _International Conference on Learning Representations_.
  * Sung et al. (2022) Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022.  VL-ADAPTER: Parameter-efficient transfer learning for vision-and-language tasks.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5217–5227.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2818–2826.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Tsimpoukelli et al. (2021) Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021.  Multimodal few-shot learning with frozen language models.  _Advances in Neural Information Processing Systems_ , 34:200–212.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Voita et al. (2019) Elena Voita, Rico Sennrich, and Ivan Titov. 2019.  When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1198–1212, Florence, Italy. Association for Computational Linguistics.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020.  CCNet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4003–4012, Marseille, France. European Language Resources Association.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wołk and Marasek (2014) Krzysztof Wołk and Krzysztof Marasek. 2014.  Building subject-aligned comparable corpora and mining it for truly parallel sentence pairs.  _Procedia Technology_ , 18:126–132.  International workshop on Innovations in Information and Communication Science and Technology, IICST 2014, 3-5 September 2014, Warsaw, Poland.
  * Wu et al. (2021) Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li, and Ben Kao. 2021.  Good for misconceived reasons: An empirical revisiting on the need for visual context in multimodal machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6153–6166, Online. Association for Computational Linguistics.
  * Yang et al. (2022) Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2022\.  Zero-shot video question answering via frozen bidirectional language models.  In _Advances in Neural Information Processing Systems_.
  * Yao and Wan (2020) Shaowei Yao and Xiaojun Wan. 2020.  Multimodal transformer for multimodal machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4346–4350, Online. Association for Computational Linguistics.
  * Ye et al. (2022) Junjie Ye, Junjun Guo, Yan Xiang, Kaiwen Tan, and Zhengtao Yu. 2022.  Noise-robust cross-modal interactive learning with Text2Image mask for multi-modal neural machine translation.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 5098–5108, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Yin et al. (2020) Yongjing Yin, Fandong Meng, Jinsong Su, Chulun Zhou, Zhengyuan Yang, Jie Zhou, and Jiebo Luo. 2020.  A novel graph-based multi-modal fusion encoder for neural machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3025–3035, Online. Association for Computational Linguistics.
  * Zhou et al. (2021) Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, and Jingjing Liu. 2021.  UC2: Universal cross-lingual cross-modal vision-and-language pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4155–4165, Nashville, TN, USA.

##  Appendix A CoMMuTE statistics

Some basic statistics of the CoMMuTE dataset can be found in Table 6. The
source side of the dataset is always English and two translations of each of
the 155 English ambiguous sentences are provided in French, German and Czech.

| En | Fr | De | Cs
---|---|---|---|---
#unique sents. | 155 | 308 | 300 | 308
Avg. sent. length | 6.54 | 6.90 | 6.48 | 5.07
#unique toks | 462 | 679 | 638 | 718
Table 6: CoMMuTE statistics.

##  Appendix B Visual features

We use MDETR (Kamath et al., 2021) features as our local visual features.
Concretely, we extract the set of output queries features of size 64 from the
MDETR decoder and introduce them as input.

In addition, we use CLIP (Radford et al., 2021) features as our global visual
features. More specifically, we extract the output [CLS] features of size 512
from the ViT (Dosovitskiy et al., 2021) image encoder used by CLIP and
introduced it as input.

##  Appendix C Guided self-attention analysis

We studied the values of the cross-modal part of our guided self-attention. To
do so, we followed the method proposed by Kobayashi et al. (2020) who showed
that raw attention scores $\alpha$ are meaningless and instead proposed to
conduct analysis on the normalised attention scores $\lVert\alpha f\rVert$,
where $\alpha$ are the raw attention scores and $f$ is the value vector in the
attention mechanism.

| En $\rightarrow$ Fr | En $\rightarrow$ De | En $\rightarrow$ Cs
---|---|---|---
| Test2016 | Test2017 | MSCOCO | Test2016 | Test2017 | MSCOCO | Test2016 | Test2018
Text-only Machine Translation
Vanilla MT* | 74.4 ±0.1 | 68.3 ±0.2 | 61.5 ±0.4 | 55.0 ±0.2 | 46.9 ±0.4 | 45.3 ±0.3 | 30.5 ±0.1 | 26.5 ±0.1
TLM + MT* | 76.3 ±0.1 | 70.3 ±0.2 | 63.4 ±0.3 | 56.0 ±0.2 | 48.1 ±0.1 | 46.1 ±0.2 | 31.0 ±0.0 | 26.6 ±0.1
mBART + MT* | 68.3 ±0.0 | 66.8 ±0.0 | 66.4 ±0.0 | 52.6 ±0.0 | 48.3 ±0.0 | 44.2 ±0.0 | 30.7 ±0.0 | 28.1 ±0.0
mBART + MT* w/ adapters |  79.9 ±0.3 |  76.0 ±0.2 |  69.5 ±0.6 |  58.5 ±0.1 |  53.9 ±0.3 |  51.7 ±0.2 |  33.8 ±0.2 |  31.4 ±0.2
Multimodal Machine Translation
Graph-MMT* | 74.1 ±0.4 | 68.7 ±0.5 | 61.6 ±0.6 | 54.4 ±0.4 | 45.7 ±0.4 | 43.2 ±0.7 | 30.1 ±0.1 | 26.0 ±0.2
Gated Fusion* | 73.1 ±0.3 | 67.1 ±0.5 | 60.1 ±0.4 | 54.9 ±0.4 | 46.2 ±0.3 | 44.2 ±0.4 | 28.8 ±0.2 | 25.1 ±0.1
VTLM + MMT* | 75.9 ±0.1 | 69.8 ±0.1 | 63.3 ±0.2 | 55.4 ±0.1 | 47.7 ±0.1 | 45.6 ±0.3 | 30.6 ±0.1 | 26.4 ±0.1
VGAMT (ours) | 79.7 ±0.0 | 75.9 ±0.1 | 68.9 ±0.4 | 58.1 ±0.2 | 53.6 ±0.2 |  51.7 ±0.2 | 33.7 ±0.1 | 30.5 ±0.0

Table 7: METEOR scores for standard En$\rightarrow${Fr,De,Cs} benchmarks
(average of three runs). The best result is indicated in Bold. * means the
results were retrained by using the original codebase provided by the authors
of the paper.

Figure 9(b) shows the cross-modal part of the guided self-attention map from
the example displayed in Figure 9(a) where all the values have been averaged
over all heads and all layers. In this example, the English word fans ‘cooling
device or ardent admirer’ is ambiguous and the two meanings have different
translations in French, German and Czech. Given the region-text couples
extracted by MDETR (Figure 9(a)), only the token fans can attend to the MDETR
region features. The normalised attention scores of the embedding of the token
fans on these regions are low in comparison to the scores on the text part and
on the CLIP embedding. On the contrary, all embeddings can attend to CLIP
embedding and the embedding of the token fans is the one with the highest
normalised attention score with CLIP embedding.

##  Appendix D Additional examples

Figure 8: Example from Test2017, illustrating how VGAMT is able to exploit
visual information to distinguish between the two types of football (soccer
(1) and American football (2) depending on whether British or American English
is used), whereas the text-only baseline produces a wrong translation.

(a) English sentence, its associated image and the region-text couples
extracted by MDETR.

(b) Normalised attention scores from the cross-modal part of our guided self-
attention mechanism. Only the embeddings of the token ‘fans’ can attend to
MDETR features. All the embeddings can attend to CLIP features.

Figure 9: Guided self-attention map for the English sentence The fans were
still out of order and its associated image. The sentence is ambiguous as
English fan refers to ‘a cooling device’ or ‘an ardent admirer’. Values are
averaged over all heads and all layers. <eos> refers to the end of sentence
token.

Figure 12 shows examples from CoMMuTE and the perplexity scores obtained by
VGAMT. It is able to choose the correct translations from English sentences
with the ambiguous words chips, bugs, red light. However, it fails to choose
the correct translation in the first case of Figure 12(d); the picture shows a
beam ‘ray of light’ and the perplexity of the correct (top) translation with
the French translation rayon is higher than the incorrect (bottom) one with
the French translation poutre. Nevertheless, the model gives a lower
perplexity to the sentence with the correct image (1.847) in comparison to the
same sentence with the incorrect image (2.616). So, even if VGAMT is not able
to choose the correct translation in the first case of this example, it shows
some evidence of being able to discriminate between the French translation
with the correct image and the same French translation with the incorrect
image. Figures 12(e) and 12(f) show two other similar examples in
En$\rightarrow$De MT.

In terms of translation (rather than reranking), Figure 8 shows an example
from Multi30k where our model correctly translates the ambiguous word while
the text-only baseline fails to do so.

Model | Test2016 | Test2017 | MSCOCO
---|---|---|---
Text-only Machine Translation
mBART + MT* w/ adapters | 79.9 ±0.3 | 76.0 ±0.2 |  69.5 ±0.6
w/o MLM objective |  80.3 ±0.2 |  76.3 ±0.2 | 68.7 ±0.3
Multimodal Machine Translation
VGAMT (ours) | 79.7 ±0.0 | 75.9 ±0.1 | 68.9 ±0.4
unfrozen w/o adapters | 79.8 ±0.5 | 75.8 ±0.2 | 68.7 ±0.6
w/o VMLM objective | 80.3 ±0.1 | 76.0 ±0.1 | 68.7 ±0.1
w/o guided self-attention | 79.6 ±0.1 | 75.4 ±0.2 | 68.4 ±0.3
w/ pretraining (w/o co-training) | 79.2 ±0.0 | 74.3 ±0.1 | 67.9 ±0.2
w/o MDETR features | 79.5 ±0.3 | 75.6 ±0.1 | 68.9 ±0.6
w/o CLIP features | 79.2 ±0.5 | 75.2 ±0.3 | 69.0 ±0.5

Table 8: METEOR scores for the ablations described in Section 6
(En$\rightarrow$Fr). The best result is indicated in bold.

##  Appendix E METEOR scores

In order to compare to previous work, we also provide METEOR scores in Table 7
for En$\rightarrow${Fr,De,Cs} standard benchmarks. It confirms that VGAMT
obtains competitive results over a strong text-only baseline on benchmarks
where images are not necessary for translation. METEOR scores for the
En$\rightarrow$Fr ablations conducted in Section 6 are shown in Table 8.

##  Appendix F Translating CoMMuTE

|  | VGAMT (ours) | mBART + MT* w/ adapters
---|---|---|---
En$\rightarrow$Fr | BLEU | 32.2 ±1.7 |  34.5 ±1.4
COMET |  0.362 ±0.048 | 0.306 ±0.014
METEOR | 48.5 ±2.1 |  52.3 ±1.4
En$\rightarrow$De | BLEU |  29.3 ±0.6 | 25.9 ±0.7
COMET |  0.184 ±0.024 | 0.182 ±0.007
METEOR |  43.0 ±0.8 | 41.3 ±1.3
En$\rightarrow$Cs | BLEU |  20.8 ±0.9 | 18.3 ±1.3
COMET |  0.525 ±0.024 | 0.491 ±0.022
METEOR |  23.4 ±0.8 | 22.4 ±0.7

Table 9: MT Generation results for CoMMuTE. Best results are indicated in
bold. Figure 10: Machine Translation example from CoMMuTE. VGAMT is able to
exploit visual information to translate ‘bow’ correctly in the two cases.

CoMMuTE is designed as a contrastive test set to be used for reranking.
However, it is possible to translate the source sentences too and compare
against the reference translations.

Table 9 shows the MT results on CoMMuTE comparing VGAMT and the strong text-
only baseline. They may indicate that traditional metrics for MT task are ill-
adapted to evaluating the use of visual information by MMT models. For
instance, BLEU and METEOR scores for the text-only baseline are significantly
higher than the scores for our model VGAMT on the En$\rightarrow$Fr split
whereas our VGAMT obtains 67.10 accuracy on the contrastive evaluation (Table
3). It might be due to the fact that such metrics are less reliable on small
datasets or that BLEU and METEOR are words matching metrics and therefore
output low scores for synonyms or similar content described differently. On
the other hand, COMET is an embedding-based metric, which outputs higher
scores for synonyms which may be why VGAMT outperforms the text-only baseline
with this metric; as illustrated by Figure 10 where VGAMT outputs noeud ‘bow’
which is a synonym of the reference translation ruban ‘bow’ in that case. That
being said, the use of our contrastive dataset CoMMuTE therefore seems
necessary to evaluate how well a MMT model exploits visual information in
order to produce correct translations instead of relying only on standard
metrics for MT.

Figure 10 illustrates how VGAMT can translate ambiguous words correctly by
using images, while mBART + MT (our strong text-only baseline) cannot. In both
cases, the baseline outputs French noeud papillon ‘bow tie’, while VGAMT
produces the correct translations of bow. Figures 11(a), 11(b), 11(c), 11(d),
11(e) and 11(f) show the same effect for En$\rightarrow${Cs,De} translations.
Even if VGAMT does not literally translate the ambiguous word as exemplified
by Figure 11(b), it produces a translation with the expected meaning based on
the image; the text-only models were not able to do so.

(a) English word mole correctly translated in both cases (znaménko ‘skin
blemish’ and krtka ‘burrowing mammal’).

(b) English word brush correctly translated in both cases (Bürste ‘cleaning
tool’ and Pinsel ‘object used for painting’).

(c) English word seal correctly translated in both cases (pečeť ‘official
stamp’ and tuleně ‘sea mammal’).

(d) English word boot correctly translated in both cases (Kofferraum ‘car
trunk’ and Stiefel ‘footwear’).

(e) English word arms correctly translated in both cases (zbraně ‘weapon’ and
ruce ‘parts of the human body’).

(f) English word palm correctly translated in both cases (Palme ‘tree’ and
Handfläche ‘anterior aspect of the hand’).

Figure 11: MT examples for different English$\rightarrow$Czech and
English$\rightarrow$German examples from CoMMuTE. For each one, VGAMT is able
to exploit visual information to translate English ambiguous words (underlined
and in bold) correctly in all cases.

(a) The English word chips refers to ‘french fries’ or ‘poker chips’.

(b) The English word bugs refers to ‘a problem in a computer program’ or ‘a
small insect’.

(c) The English phrase red light refers to ‘a traffic signal that instructs
moving vehicles to stop’ or ‘light that is red’.

(d) The English word beam refers to ‘a ray of light’ or ‘a piece of timber or
metal used to support the roof’.

(e) The English word pen refers to ‘an instrument for writing or drawing with
ink’ or ‘an area of land surrounded by a fence’.

(f) The English word tube refers to ‘a long cylinder for holding liquids or
gases.’ or ‘a railway system in some cities’.

Figure 12: Perplexity scores from VGAMT on different examples from CoMMuTE. It
is possible to produce at least two different French translations from each
source sentence in English, the correct translation therefore depends on the
input image. For each sub-example, the correct (resp. incorrect) translation
is the top (resp. bottom) one. The ambiguous parts of the sentences are
highlighted in bold.


††thanks<EMAIL_ADDRESS>Corresponding author:
<EMAIL_ADDRESS>
# Mass separation in an asymmetric channel

Narender Khatri  Department of Physics, Indian Institute of Technology
Kharagpur, Kharagpur - 721302, India  P. S. Burada  Department of Physics,
Indian Institute of Technology Kharagpur, Kharagpur - 721302, India

###### Abstract

We present a mechanism to sort out particles of different masses in an
asymmetric channel, where the entropic barriers arise naturally and control
the diffusion of these particles. When particles are subjected to an
oscillatory force, with the scaled amplitude $a$ and frequency $\omega$, the
mean particle velocity exhibits a bell-shaped behavior as a function of the
particle mass, indicating that particles with an optimal mass $m_{op}$ drift
faster than other particles. By tuning $a$ and $\omega$, we get an empirical
relation to estimate $m_{op}\sim(a\,\omega^{2})^{-0.4}$. An additional static
bias would push the particles of lighter mass to move in its direction while
the others drift opposite to it. This study is useful to design lab-on-a-chip
devices for separating particles of different masses.

##  I Introduction

In many natural systems and industrial productions, matter consists of
mixtures of various small parts on a range from macroscale to nanoscale, e.g.,
mixed polar and nonpolar macroparticles Brunet_JCP , mixed bacteria Petrides ,
mixed nanoparticles Ploschner_NL , various DNA fragments Dorfman , different
kinds of cells Franke_BMF , or size-dispersed particles in the liquid
Marchesoni , to name but a few. The separation and sorting of small-sized
particles have a wide range of industrial, biomedical and clinical
applications, e.g., wastewater purification, blood sample preparation, and
disease diagnostic Jin . Various particle separation mechanisms or tools are
in use to separate the desired particles from the bulk; for instance,
templates consist of arrays of obstacles Reimann_prl , centrifuge machines
Lee_prl , sieve or membrane Duke_prl , and by applying some external fields
Dholakia_Nature . These separation techniques are based on the drift or
diffusive behavior of the particles whose properties depend on their mass,
size, shape, and charge. In particular, compared to the macro-sized particles,
the controlled separation of the mesoscopic particles from the mixture, based
on their physical properties, is in high demand for laboratory research and
industrial applications Petrides ; Dorfman ; Marchesoni ; Austin ; Reguera_prl
; Reimann_prl ; Rubi_prl ; Mukhopadhyay_prl ; Sens_prl ; Slapik_prl . Note
that the ubiquitous Brownian dynamics is relevant particularly for the
particles in the mesoscopic size regime. The separation of mesoscopic
constituents presents major challenges in various disciplines, from biomedical
problems such as separating malignant circulating tumor cells from leukocytes
in the bloodstream Jin  to technological problems on colloidal scales such as
separating nanoparticles based on their physical properties Reguera_prl ;
Slapik_prl . Remarkably, the effective isolation and separating techniques
carry a strong potential to achieve selective transport of biological
particles such as cells, organelles, or DNA complexes Bhagat_MBEC ;
Aksimentiev_BJ .

Quite often, the separation of particles happens in microstructures such as
porous media Schwartz_Nano , microfluidic channels Muller , and living tissues
Newby_R . These systems have attracted the attention of physicists, chemists,
mathematicians, biologists, and engineers because the irregular shapes of the
walls control the volume of the phase space accessible to the particles, due
to which the entropic barriers arise and influence the diffusion of the
particles in such systems Reguera_prl ; Burada_CPC ; Burada_prl . So far, much
effort has been devoted to developing the size-based separation of particles
Reguera_prl ; Rubi_prl ; Sens_prl ; Slapik_pra ; Karimi_pre . However,
needless to say, the effective isolation of selected particles from the
mixture, based on their mass, is much in demand in chemistry, biology,
nanotechnology, and industry Mukhopadhyay_prl ; Slapik_prl ; Nagaoka_pre ;
Borromeo_prl . It is because anomalies in bioparticles mass might be an
important factor associated with disease initiation and propagation. For
instance, it has been observed that the mass of a cancer cell is higher than a
healthy one Suresh . Thus, one needs to develop new innovative mass-based
isolation techniques for the diagnosis and detection of diseases. Note that
this is a very challenging task because, typically, identically sized
particles may have different masses Slapik_prl .

In the present work, we introduce a mechanism to separate particles based on
their mass when they diffuse in an asymmetric triangular channel under the
influence of an oscillatory force and a static bias. The working principle
relies on the entropic rectification of the noisy motion of particles in the
presence of an oscillatory force Marchesoni ; Reguera_prl ; Muller ;
Reguera_pre ; Khatri_JSTAT  and a static bias that further controls the
movement of the particles. In particular, the strength of the entropic
rectification depends strongly on the mass of the particles. Therefore, in the
presence of a static bias, in the opposite direction to the channel direction,
it is possible to separate particles of different masses by pushing them in
opposite directions. The lighter particles follow the static bias, whereas
heavier particles drift in the opposite direction resulting in an efficient
and faster particle separation.

The rest of this article is organized in the following way. In Sec. II, we
introduce our model for the inertial Brownian particles of different masses
subjected to external forces in a two-dimensional asymmetric channel. The
results in the absence and presence of a static bias are discussed in Sec. III
and Sec. IV, respectively. Finally, we present the main conclusions in Sec. V.

##  II Model

Figure 1: Schematic illustration of a two-dimensional triangular channel with
a periodicity $L$ confining the motion of a Brownian particle of mass $M$. The
structure is defined by Eq. (2). The particle is driven by a static bias
$\vec{F}$ and an oscillatory force $\vec{A}(t)$ along the $x-$direction.

In order to illustrate the mass-based separation mechanism, we have chosen a
triangular channel depicted in Fig. 1. A Brownian particle of mass $M$ is
diffusing in this channel, filled with a fluid of viscosity $\mu$, subjected
to an unbiased oscillatory force $\vec{A}(t)$ and a static bias $\vec{F}$;
both are acting along the principal axis of the channel. The Brownian dynamics
of such a particle is represented by the Langevin equation

$M\frac{d^{2}\vec{r}}{dt^{2}}=-\eta\frac{d\vec{r}}{dt}-[F+A(t)]\hat{x}+\sqrt{\eta
k_{B}T}\vec{\xi}(t),$ (1)

where $\vec{r}$ is the position of the particle, $\eta$ is the friction
coefficient, $k_{B}$ is the Boltzmann constant, and $T$ is the temperature.
The explicit form of the oscillatory force is given by $A(t)=A\sin(\Omega t)$,
where $A$ is the amplitude and $\Omega$ is the frequency of the sinusoidal
driving. The Gaussian random force $\vec{\xi}(t)$ with zero mean is
uncorrelated in time and therefore obeys the fluctuation-dissipation relation
$\langle\xi_{i}(t)\xi_{j}(t^{\prime})\rangle=2\delta_{ij}\delta(t-t^{\prime})$
for $i,j=x,y$. The shape of the upper wall is defined by

$w_{u}(x)=\begin{cases}w_{\mathrm{min}},&x=0,\\\
w_{\mathrm{max}}-(w_{\mathrm{max}}-w_{\mathrm{min}})\frac{x}{L},&0<x\leq
L,\end{cases}$ (2)

where $w_{\mathrm{max}}$ and $w_{\mathrm{min}}$ correspond to the maximum
half-width and minimum half-width of the channel, respectively, and $L$
corresponds to the periodicity of the channel. The ratio of these two widths
defines the dimensionless aspect ratio given by
$\epsilon=w_{\mathrm{min}}/w_{\mathrm{max}}$. Due to the symmetry about the
principal axis of the channel, the lower wall is given by
$w_{l}(x)=-w_{u}(x)$. Consequently, $2w(x)=w_{u}(x)-w_{l}(x)$ corresponds to
the local width of the channel.

In order to achieve a dimensionless description, we henceforth scale all
lengths by the periodicity of the channel $L$ and time by $\tau=\eta
L^{2}/(k_{B}T)$, which is the characteristic diffusion time Burada_prl ;
Khatri_pre . In dimensionless form, the two-dimensional Langevin equation (1)
reads

$m\frac{d^{2}\vec{r}}{dt^{2}}=-\frac{d\vec{r}}{dt}-[f+a\sin(\omega
t)]\hat{x}+\vec{\xi}(t).$ (3)

Here, the dimensionless mass is given by
$m=\tau_{0}/\tau=Mk_{B}T/(\eta^{2}L^{2})$, where $\tau_{0}=M/\eta$ defines the
characteristic time of velocity relaxation for the free Brownian particle.
Therefore, the dimensionless mass $m$ depends not only on the actual physical
mass of the particle $M$ but also on the friction coefficient $\eta$, thermal
energy $k_{B}T$, and the periodicity of the channel $L$. The dimensionless
static force reads $f=FL/(k_{B}T)$, which is the ratio of work done to the
particle due to the external static force and the available thermal energy.
Other parameters read $a=AL/(k_{B}T)$ and $\omega=\Omega\tau$. In the rest of
the article, we use dimensionless variables only.

Since the particles along the $y$-direction are confined, the observable of
foremost interest for mass separation is the stationary average velocity
$\langle v\rangle$ of the particles along the $x$-direction. Unfortunately,
for the considered system, the Fokker-Planck equation corresponding to the
Langevin equation (3) cannot be solved analytically for $\langle v\rangle$ by
any known analytical method Burada_prl . Thus, $\langle v\rangle$ is
calculated using Brownian dynamics simulations performed by solving the
Langevin equation (3) using the standard stochastic Euler algorithm over
$10^{4}$ trajectories with reflecting boundary conditions at the channel walls
to ensure the confinement within the channel. Figure 1 shows the reflection of
a particle at the channel boundary. Let us say initially the particle was at
$P(x_{1},y_{1})$, and the position of the particle in the next instant of time
is $R(x_{2},y_{2})$, which is outside of the channel boundary. The line
joining points $P$ and $R$ intersects the channel boundary at a point
$Q(\alpha,\beta)$, i.e., the reflection point which can be calculated
numerically using the bisection method. The desired position $S(x_{3},y_{3})$
of the particle after reflection at point $Q$ is given by Khatri_JCP

$\displaystyle x_{3}$ $\displaystyle=\alpha\pm\,\frac{l}{\sqrt{1+m_{3}^{2}}},$
(4a) $\displaystyle y_{3}$ $\displaystyle=\beta+m_{3}(x_{3}-\alpha).$ (4b)

Here, $m_{3}$ is the slope of line $QS$ and
$l=\sqrt{(y_{2}-\beta)^{2}+(x_{2}-\alpha)^{2}}$. Note that depending on the
driving force acting on the particle, the particle may reflect multiple times
at the channel boundaries to reach the final position. The average velocity of
the particles, having mass $m$, along the $x$-direction is calculated, in the
long time limit, as Reguera_prl ; Khatri_JSTAT

$\langle v\rangle=\lim_{t\to\infty}\frac{\langle x(t)\rangle}{t}.$ (5)

##  III In the absence of static bias

Figure 2: The average velocity $\langle v\rangle$ versus the mass $m$ of
particles for various values of the driving frequency $\omega$ (a) and driving
amplitude $a$ (b) in the absence of static force. The other parameters of the
channel are $L=1$, $w_{\mathrm{max}}=1$, and $\epsilon=0.1$.

The average velocity as a function of the mass of particles is depicted in
Fig. 2 for various values of the driving frequency $\omega$ and driving
amplitude $a$ in the absence of static force. We observe that particles of
various masses exhibit rectification in the positive $x-$axis. It is due to
the fact that the oscillatory force can break the thermodynamic equilibrium
and induces the directed transport because of the asymmetry of the channel.
Here, the positive velocity is due to the chosen shape of the channel
structure. For example, if the channel shape is inverted with respect to the
$y-$axis, the rectification will be in the negative $x-$axis. Interestingly,
the average velocity exhibits a peak at an optimal mass for which the
particles show higher rectification compared to other particles. The peak is
more pronounced as $\omega$ decreases or $a$ increases. Note that the optimal
mass at which the maximum occurs can be effectively controlled by suitably
tuning $\omega$ and $a$. When the particles are very heavy, the inertial
forces start to dominate over the strength of oscillatory force; therefore, as
one would expect, the velocity tends to zero. In the other limit, i.e., $m\to
0$, $\langle v\rangle$ increases linearly with $m$, which is independent for
different values of $\omega$. In this limit, as $m$ increases, due to the rise
in the inertial forces, the effective drag from the surrounding fluid medium
on the particles reduces. This leads to an increase in $\langle v\rangle$ (see
Fig. 2(a)). However, in the limit $\omega\rightarrow\infty$, $\langle
v\rangle$ will tend to zero Khatri_JSTAT . It is because, in this limit,
particles experience a zero time-averaged constant force
$a=\int_{0}^{\frac{2\pi}{\omega}}a(t)dt=0$. Also, in $a\to\infty$ limit,
$\langle v\rangle$ will tend to zero because the effect of asymmetry of the
channel will disappear in this limit Reguera_prl ; Khatri_JSTAT . In
particular, this asymptotic regime of very large $\omega$ and $a$ is not yet
reached in Fig. 2.

###  III.1 Influence of channel aspect ratio $\epsilon$

Figure 3: The average velocity $\langle v\rangle$ versus the mass $m$ of
particles for different values of $\epsilon$. The set parameters are
$a=2.5,\omega=0.01,L=1$, and $w_{\mathrm{max}}=1$.

In order to obtain the influence of channel aspect ratio $\epsilon$, in Fig.
3, we study the average velocity as a function of the mass of particles for
different values of $\epsilon$. It is evident that the bottleneck width of the
channel does not influence the qualitative behavior of $\langle v\rangle$ as
long as the condition $w_{\mathrm{min}}\ll w_{\mathrm{max}}$, i.e.,
$\epsilon\ll 1$, is satisfied. For $\epsilon=1$, i.e., a flat channel, as one
would expect, there would be no rectification of particles resulting in
$\langle v\rangle=0$. Note that the main reason for the observed bell-shaped
behavior of $\langle v\rangle$ is the asymmetry of the channel geometry. In
the $m\to 0$ limit, for $\epsilon\ll 1$, $\langle v\rangle$ increases with
$\epsilon$ Khatri_JCP . In the other limit, i.e., when the particles are very
heavy, as mentioned earlier, $\langle v\rangle$ tends to zero irrespective of
$\epsilon$.

Figure 4: Dependence of the optimal mass $m_{op}$ on the driving frequency $\omega$ (a) and driving amplitude $a$ (b). The lines correspond to the empirical relation $m_{op}\sim(a\,\omega^{2})^{-0.4}$. The other parameters of the channel are $L=1$, $w_{\mathrm{max}}=1$, and $\epsilon=0.1$. $m_{op}=k_{1}\,\omega^{-0.8}$ | $m_{op}=k_{2}\,a^{-0.4}$
---|---
$a$ |  $k_{1}$ |  $\omega$ |  $k_{2}$
1 |  0.9 |  0.001 |  210
2.5 |  0.65 |  0.01 |  35
5 |  0.5 |  0.1 |  5.3
Table 1: Prefactors $k_{1}$ and $k_{2}$ correspond to the empirical relations
for different values of $a$ and $\omega$, respectively.

To get more insights into the optimal mass $m_{op}$, in Fig. 4, we extract
$m_{op}$ for various values of the driving amplitude and frequency. The
optimal mass $m_{op}$ decreases with $\omega$, for a fixed $a$ value, as
$m_{op}=k_{1}\,\omega^{-0.8}$. Also, $m_{op}$ decreases with $a$, for a fixed
$\omega$ value, as $m_{op}=k_{2}\,a^{-0.4}$. The Prefactors $k_{1}$ and
$k_{2}$ are listed in table 1 for a few values of $a$ and $\omega$,
respectively. Therefore, one can estimate that the value of the optimal mass
varies as $m_{op}=k\,(a\,\omega^{2})^{-0.4}$, where the prefactor $k\approx
0.91$. Here, $k$ is calculated by substituting the known values of $m_{op}$
corresponding to different $a$ and $\omega$ in the previous empirical
relation. Thus, by tuning $a$ and $\omega$, the particles with an optimal mass
$m_{op}$ can be rectified with higher speed compared to other particles. Note
that in real units, the $k$ value (or $k_{1},k_{2}$) depends on various
parameters like $L$, $k_{\mathrm{B}}\,T$, and $\eta$. However, we have
observed that the minimum half-width of the channel $w_{\mathrm{min}}$ (see
Fig. 1) does not influence the empirical relation for $m_{op}$ as long as the
condition $\epsilon\ll 1$ is satisfied.

##  IV Influence of the static bias

Figure 5: The average velocity $\langle v\rangle$ versus the mass $m$ of
particles for different values of the static force $f$. The inset depicts
$\langle v\rangle$ as a function of $f$ for the particles of different masses.
The set parameters are $a=2.5$, $\omega=0.01$, $L=1$, $w_{\mathrm{max}}=1$,
and $\epsilon=0.1$.

In order to achieve the separation of particles of different masses in
opposite directions, a static force $f$ can be applied to control the
transport direction. Figure 5 depicts the average velocity dependence on the
mass of particles for different strengths of the static bias for fixed values
of $a$ and $\omega$ of the oscillatory force. In the absence of a static bias,
due to the chosen channel structure, particles exhibit a net positive
velocity. However, if we switch on a small static bias in the negative
$x-$axis (see Fig. 1), particles of the lighter mass move to the left side of
the channel, whereas heavier particles drift to the right side (see Fig. 5).
This novel mechanism provides a way to separate the lighter particles from the
heavier ones. A similar behavior has been reported for the overdamped Brownian
particles of different radii Reguera_prl , i.e., large particles move towards
the right side of the channel whereas small particles follow the static force.
Note that the optimal mass $(m_{op})$ is independent of the static bias. The
inset of Fig. 5 depicts the average velocity dependence on the static force
for the particles of different masses. In particular, by tuning the static
bias $f$, one can control the separation of particles of different masses. For
example, by choosing $f=1$, the particles of lighter mass $m=0.01$ move
towards the left side of the channel with speed $0.23$, whereas the particles
of heavier mass $m=30$ drift to the right side with a speed of $0.24$.
Interestingly, as depicted in the inset of Fig. 5, the velocities are almost
parallel and vary linearly with the strength of the static bias, thus
facilitating an efficient control of the mass separation effect.

Figure 6: The average velocity $\langle v\rangle$ versus the driving amplitude
$a$ of the oscillatory force for the particles of different masses in the
presence of static bias. The inset depicts the same in the absence of static
bias. The set parameters are $\omega=0.01$, $L=1$, $w_{\mathrm{max}}=1$, and
$\epsilon=0.1$.

Figure 6 shows the dependence of the average velocity $\langle v\rangle$ on
the amplitude $a$ of the oscillatory force for the particles of different
masses. As mentioned earlier, in the absence of the static force, particles of
different masses move towards the positive $x-$axis, exhibiting a positive
velocity (see inset of Fig. 6). However, in the presence of static bias, by
suitably tuning $a$, one can control the separation of particles of different
masses. For example, by keeping $a=3$, the particles of mass $m<0.1$ move to
the left side of the channel, whereas particles of mass $m>0.1$ drift to the
right side. Moreover, the crossover of average velocity from negative to
positive can also be controlled by $a$.

The observed mass-based separation mechanism can be studied experimentally for
Brownian particles of various masses diffusing in an asymmetric channel, which
can be prepared by microprinting on a substrate Maier_nlett ; Mahmud_nature .
The fabrication of identical particles of different masses is already within
reach of today’s nanotechnology Marchesoni ; Muller . These particles can be
prepared from hydrophobic materials such as luminescent polystyrene,
polycrystalline silver, gold, copper, etc. In order to have an estimate in
real units, which is very useful for the experimentalists, the characteristic
values of the friction coefficient and characteristic diffusion time for the
particles in water moving in a triangular channel with aspect ratio
$\epsilon=0.1$ and period length $L\sim 10~{}\mu\mathrm{m}$ at room
temperature ($T\sim 300~{}\mathrm{K}$) are given by $\eta\sim
2~{}\mathrm{n\/kg/s}$ Cussler  and $\tau\sim 50~{}\mathrm{s}$, respectively
Marchesoni ; Muller . In the absence of static force, for the parameters
$A\sim 0.25~{}\mathrm{f\/N}$ and $\Omega\sim 0.0002~{}\mathrm{s^{-1}}$, the
particles of mass $M\sim 0.25~{}\mathrm{nkg}$ move with a higher average
velocity $\langle V\rangle\sim 0.13~{}\mu\mathrm{m/s}$ than the other
particles. Whereas, for the parameters $A\sim 0.25~{}\mathrm{f\/N}$ and
$\Omega\sim 0.002~{}\mathrm{s^{-1}}$, the optimal mass is $M\sim
0.04~{}\mathrm{n\/kg}$. This suggests that by suitably tuning the driving
frequency and amplitude, the particles of an optimal mass move faster than
other particles in the same direction. On the other hand, in the presence of a
static bias, for the parameters $F\sim 0.1~{}\mathrm{f\/N}$, $A\sim
0.25~{}\mathrm{f\/N}$, and $\Omega\sim 0.0002~{}\mathrm{s^{-1}}$, the
particles of mass $M\sim 0.1~{}\mathrm{n\/kg}$ move in the negative direction
with an average velocity $|\langle V\rangle|\sim 0.06~{}\mu\mathrm{m/s}$, and
particles of mass $M\sim 0.3~{}\mathrm{n\/kg}$ drift in the positive direction
with the average velocity $\langle V\rangle\sim 0.05~{}\mu\mathrm{m/s}$.
Therefore, in the presence of static force, the particles of different masses
can be separated in opposite directions. It is expected that these results
will motivate the experimentalists to design lab-on-a-chip devices for
separating nano- and microparticles, proteins, organelles, and cells based on
their mass.

##  V Conclusions

In this work, we have presented a mechanism for separating particles in an
asymmetric channel based on their mass. This mechanism is based on the
entropic rectification of the noisy motion of particles in the presence of an
unbiased oscillatory force and a static force that controls the drift
direction. It is found that entropic rectification depends strongly on mass of
the particles. We could demonstrate that the average velocity exhibits a
maximum at an optimal mass for various parameters, and this optimal mass
depends on the driving frequency and amplitude of the oscillatory force as
$m_{op}\sim(a\,\omega^{2})^{-0.4}$. In the presence of a small static force in
the negative $x$-axis, we have found that lighter particles can be separated
from the heavier ones by pushing them in opposite directions. It is
conceivable that this separation mechanism could be implemented in an
asymmetric structure where the entropic effects are prominent to isolate nano-
and microparticles, proteins, organelles, and cells.

##  VI Acknowledge

This work was supported by the Indian Institute of Technology Kharagpur under
the Grant No. IIT/SRIC/PHY/TAB/2015-16/114.

## References

  * (1) C. Brunet, J. G. Malherbe, and S. Amokrane, J. Chem. Phys. 130, 134908 (2009).
  * (2) R. G. Harrison, P. W. Todd, S. R. Rudge, and D. P. Petrides, Bioseparations Science and Engineering (Oxford University Press, Oxford, 2003).
  * (3) M. Ploschner, T. Cizmar, M. Mazilu, A. D. Falco, and K. Dholakia, Nano Lett. 12, 1923 (2012).
  * (4) K. D. Dorfman, Rev. Mod. Phys. 82, 2903 (2010).
  * (5) T. M. Geislinger and T. Franke, Biomicrofluidics 7, 044120 (2013).
  * (6) P. Hänggi and F. Marchesoni, Rev. Mod. Phys. 81, 387 (2009).
  * (7) C. Jin, S. M. McFaul, S. P. Duffy, X. Deng, P. Tavassoli, P. C. Black, and H. Ma, Lab Chip 14, 32 (2014).
  * (8) L. Bogunovic, M. Fliedner, R. Eichhorn, S. Wegener, J. Regtmeier, D. Anselmetti, and P. Reimann, Phys. Rev. Lett. 109, 100603 (2012).
  * (9) J. Lee and A. J. C. Ladd, Phys. Rev. Lett. 89, 104301 (2002).
  * (10) T. A. J. Duke and R. H. Austin, Phys. Rev. Lett. 80, 1552 (1998).
  * (11) M. P. MacDonald, G. C. Spalding, and K. Dholakia, Nature 426, 421 (2003).
  * (12) W. D. Volkmuth and R. H. Austin, Nature (London) 358, 600 (1992).
  * (13) D. Reguera, A. Luque, P. S. Burada, G. Schmid, J. M. Rubi, and P. Hänggi, Phys. Rev. Lett. 108, 020604 (2012).
  * (14) P. Malgaretti, I. Pagonabarraga, and J. M. Rubi, Phys. Rev. Lett. 113, 128301 (2014).
  * (15) A. K. Mukhopadhyay, B. Liebchen, and P. Schmelcher, Phys. Rev. Lett. 120, 218002 (2018).
  * (16) Q. Vagne and P. Sens, Phys. Rev. Lett. 120, 058102 (2018).
  * (17) A. Slapik, J. Luczka, P. Hänggi, and J. Spiechowicz, Phys. Rev. Lett. 122, 070602 (2019).
  * (18) A. A. S. Bhagat, H. Bow, H. W. Hou, S. J. Tan, J. Han, and C. T. Lim, Med. Biol. Eng. Comput. 48, 999 (2010).
  * (19) A. Aksimentiev, J. B. Heng, G. Timp, and K. Schulten, Biophys. J. 87, 2086 (2004).
  * (20) M. J. Skaug, L. Wang, Y. F. Ding, and D. K. Schwartz, ACS Nano 9, 2148 (2015).
  * (21) S. Matthias and F. Müller, Nature (London) 424, 53 (2003).
  * (22) P. C. Bressloff and J. M. Newby, Rev. Mod. Phys. 85, 135 (2013).
  * (23) P. S. Burada, P. Hänggi, F. Marchesoni, G. Schmid, and P. Talkner, ChemPhysChem 10, 45 (2009).
  * (24) D. Reguera, G. Schmid, P. S. Burada, J. M. Rubi, P. Reimann, and P. Hänggi, Phys. Rev. Lett. 96, 130603 (2006).
  * (25) A. Slapik, J. Luczka, and J. Spiechowicz, Phys. Rev. Applied 12, 054002 (2019).
  * (26) H. Karimi, M. R. Setare, and A. Moradian, Phys. Rev. E 102, 012610 (2020).
  * (27) B. Lindner, L. Schimansky-Geier, P. Reimann, P. Hänggi, and M. Nagaoka, Phys. Rev. E 59, 1417 (1999).
  * (28) M. Borromeo and F. Marchesoni, Phys. Rev. Lett. 99, 150605 (2007).
  * (29) S. Suresh, Acta Mater. 55, 3989 (2007).
  * (30) J. M. Rubi and D. Reguera, Chem. Phys. 375, 518 (2010).
  * (31) N. Khatri and P. S. Burada, J. Stat. Mech., 073202 (2021).
  * (32) N. Khatri and P. S. Burada, Phys. Rev. E 102, 012137 (2020).
  * (33) N. Khatri and P. S. Burada, J. Chem. Phys. 151, 094103 (2019).
  * (34) G. Mahmud, C. J. Campbell, K. J. Bishop, Y. A. Komarova, O. Chaga, S. Soh, S. Huda, K. Kandere-Grzybowska, and B. A. Grzybowski, Nat. Phys. 5, 606 (2009).
  * (35) C. Holz, D. Opitz, J. Mehlich, B. J. Ravoo, and B. Maier, Nano Lett. 9, 4553 (2009).
  * (36) E. L. Cussler, Diffusion: Mass Transfer in Fluid Systems (Cambridge University Press, 1997).