# Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques,
Challenges and Opportunities

Khyathi Raghavi Chandu<EMAIL_ADDRESS>
Meta
Alborz Geramifard<EMAIL_ADDRESS>
Meta

###### Abstract

Contextualizing language technologies beyond a single language kindled
embracing multiple modalities and languages. Individually, each of these
directions undoubtedly proliferated into several NLP tasks. Despite this
momentum, most of the multimodal research is primarily centered around English
and multilingual research is primarily centered around contexts from text
modality. Challenging this conventional setup, researchers studied the
unification of multilingual and multimodal (MultiX) streams. The main goal of
this work is to catalogue and characterize these works by charting out the
categories of tasks, datasets and methods to address MultiX scenarios. To this
end, we review the languages studied, gold or silver data with parallel
annotations, and understand how these modalities and languages interact in
modeling. We present an account of the modeling approaches along with their
strengths and weaknesses to better understand what scenarios they can be used
reliably. Following this, we present the high-level trends in the overall
paradigm of the field. Finally, we conclude by presenting a road map of
challenges and promising research directions.

Keywords: Multilingual, Cross-lingual, Multimodal

##  1 Introduction

> Democratic reach of ubiquitous contexts from vision and language(s)
> mitigates digital divide and alleviates cultural inclusion.

Our world contextualizes information in various modalities, the expression of
which varies based on the medium i.e., languages. The recent revolutions of
(i) aggrandizing NLP technologies to multiple languages Dabre et al. (2020) to
broaden the stakeholders to speakers of over 7000 living languages, and (ii)
deriving context from multiple modalities Baltrusaitis et al. (2019) have
mostly been independent of each other. However, to curtail the inequality of
information access and cultural biases in building resources, discern
multilingual and multimodal (MultiX) technologies is crucial. The status quo
is still that English is the most prevalent language in grounded language
agents. This creates a bottleneck for sharing information across models from
other languages.

Despite the conventional trend of independent approaches to multilinguality
and multimodality, researchers have sporadically yet consistently studied a
confluence of these. This paper presents a comprehensive survey of tasks,
datasets and methods tackling MultiX scenarios. We aggregated papers from the
last 60 years based on keyword search for variants of each of multiple
modalities and languages in the titles and abstracts. Upon this, we filtered
the papers manually that conduct research on both multilinguality and
multimodality, and annotated them for MultiX specifics. Distilling these
learnings, we characterize an ontology of tasks and methods as depicted in
Figure 1. The confluence of both ‘X’s brings about distinct challenges in
dataset collection and methodologies which inspires this setup. Note that the
names indicate how the modalities interact with several languages (as opposed
to a inter-modal interactions with a single language). The rest of the survey
is organized and presented according to the categories in Figure 1.

Figure 1: Organization of the survey

The main goal of this paper is to present insights into what distinguishes
MultiX in terms of resources and modeling. Section 2 discusses the various
tasks and datasets currently available. First, most MultiX datasets are built
on top of existing visual or multimodal source datasets. To review this, we
audit the source of the visuals used and the languages the corresponding tasks
are explored in. Second, with the unique aspect of textual data present in
multiple languages, we survey whether this data is parallel (each instance is
present across languages), and third, if they are gold (human annotated),
silver (weakly annotated with off-the-shelf tools and models), or direct
(scraped as is from sources). §3 categorizes the modeling approaches for
MultiX into 9 classes based on interactions between languages and modalities.
A distinctive feature of MultiX in modeling is leveraging annotations from one
language while predicting in another language, which is referred to as cross-
lingual. In contrast, some methods are developed and scaled to several
languages independent of one another. Finally §4 presents existing challenges
in the bleeding edge of research along with guided promising directions to
improve MultiX.

##  2 Tasks and Datasets

This section presents various tasks and the corresponding datasets as shown in
Table 1.

Task | Src | Citation | Languages | Gold/Silver | Parallel?
---|---|---|---|---|---
Image Captioning, | IAPR | Grubinger et al. (2006) | de, es | Gold | Yes
Retreival | Pascal Sentences | Funaki and Nakayama (2015) | ja | Gold | Yes
| Flickr8k | Li et al. (2016) | zh | Gold Test | Yes
| Flickr30k | Lan et al. (2017) | zh | Gold Test | Yes
|  | Nakayama et al. (2020a) | ja | Gold | Yes
|  | Elliott et al. (2016) | de, fr, cs | Gold | Yes
| Multi30k | Lala and Specia (2018) | de, fr | Silver + Gold | No
| MSCOCO | Li et al. (2019b) | zh | Gold test | Yes
|  | Yoshikawa et al. (2017) | ja | Silver | Yes
|  | Rajendran et al. (2016) | de, fr | Silver | Yes
|  | Hitschler et al. (2016) | de | Gold | Yes
|  | Elliott et al. (2017) | de, fr | Gold test | Yes
| Visual Genome | Parida et al. (2021) | hi | Gold | Yes
| CC | Caglayan et al. (2021) | de | Silver | Yes
| Flickr30k, MSCOCO, CC | Surís et al. (2022) | 52 langs | Silver | Yes
Transcription | Video Interviews | Jones and Muftic (2020) | kqz, naq | Direct | No
| TED Talks | Karakanta et al. (2020) | de, es, fr, it, |  |
|  |  | nl, pt, ro | Direct | No
| Wilderness | Black (2019) | 700 languages | Direct | Yes (Silver)
| Twitch.tv | Fu et al. (2017) | zh-tw, en | Direct | No
| YouTube | Sanabria et al. (2018) | pt, en | Direct | No
| Kinetics-600 | Wang et al. (2019b) | zh, en | Gold | Yes (Subset)
Description | IKEA website | Zhou et al. (2018) | en, de, fr | Direct | Close
| Euronews | Afli et al. (2017) | en, fr, ar, de, es, |  |
|  |  | it, pt, tr, ua | Direct | Yes (Silver)
VQA | ImageNet+MSCOCO | Ramnath et al. (2021) | hi, te | Silver | Yes
|  | Gao et al. (2015) | zh | Gold | No
|  | Koeva (2021) | bg, hr, da, nl, en, |  |
|  |  | fi, fr, el, it, lt, |  |
|  |  | pl, pt, ro, sk, sl, |  |
|  |  | es, sv, sq, is, he, sr | Silver | Yes
| VQA 2.0 | Raj Khan et al. (2021) | hi, bn, es, de, fr, |  |
|  |  | en-hi, en-bn, en-es, |  |
|  |  | en-de, en-fr | Silver | Yes
|  | Gupta et al. (2020) | hi, en-hi | Silver | Yes
| Visual Genome | Shimizu et al. (2018) | ja | Gold | Yes (Subset)
|  | Pfeiffer et al. (2022) | en, de, bn, pt | Gold | Yes
|  |  | ru, zh, ko, id |  |
Dialog | Matterport3D | Ku et al. (2020) | hi, te | Gold | No
| Pentomino puzzle | Zarrieß et al. (2016) | en, de | Gold | Yes
Comparison |  | Liu et al. (2021) | id, zh, sw, ta, tr | Gold | No
Misc |  | Alberts et al. (2021) | ar, zh, nl, en, |  |
|  |  | fa, fr, de, it, |  |
|  |  | ko, pl, pt, ru, |  |
|  |  | es, sv | Silver | Yes
|  | Moneglia et al. (2014b) | it, zh, es | Silver + Gold | No
|  | Gella et al. (2019) | de, es | Silver + Gold | Yes
Sentiment Recogn. | Youtube monologues | Bagher Zadeh et al. (2020) | es, pt, de, fr | Gold | No

Table 1: Datasets along with languages (ISO codes) spanned, gold/silver nature
of annotations and parallism across languages.

###  2.1 Image Captions:

#### IPAR-based:

Grubinger et al. (2006) collect a dataset of images of various locations and
actions accompanied with captions in three languages including English,
German, and Spanish.

#### Flickr-based:

Instead of semi-synthetic captions, Flickr30k Entities Plummer et al. (2015)
is extended to Japanese (F30kEnt-JP) Nakayama et al. (2020b) with phrase-to-
region linking so the cross-lingual phrase-to-phrase relations can be
exploited meaningfully. They are also extended to Chinese (Flickr8k-CN Li et
al. (2016), Flickr30k-CN Burger et al. (2003)) with a semi-automatic and human
post-editing step, and into Multi30k in German, French, Czech Elliott et al.
(2016) with human translations. A similar adaptation from images and English
captions of Flick8k dataset is extended to automatically create Chinese
captions Li et al. (2016) and a semi-human created test set.

#### MS-COCO-based:

The MS-COCO captions Lin et al. (2014) are extended to Chinese Li et al.
(2019b), Japanese Yoshikawa et al. (2017), German Rajendran et al. (2016);
Hitschler et al. (2016) and French Rajendran et al. (2016). Excepting
Hitschler et al. (2016), the rest are silver datasets, i.e., translated
automatically from English captions.

#### Visual Genome-based:

The annotations for Visual Genome Krishna et al. (2017) are partly reused by
automatically translating them to Hindi using segment NMT model Parida and
Bojar (2018).They are automatically translated to Hindi using segment NMT
model Parida and Bojar (2018) and then humans edit the translation in based on
the image. The challenge test set includes ambiguous English words (ambiguity
determined by embedding similarity), which can be resolved by visual context
alone. The same is replicated for Malayalam Visual Genome
111https://ufal.mff.cuni.cz/malayalam-visual-genome/wat2021-english-malayalam-
multi.

#### Retrieval:

The OpenCLIR Challenge 222https://www.nist.gov/itl/iad/mig/openclir-challenge
is aimed to locate snippets of text and speech in documents of low-resourced
languages such as Swahili. Several of the aforementioned captioning datasets
are also studied for caption-retrieval in tandem to generation.

###  2.2 Transcriptions:

Inspired from speech translation dataset MuST-C Gangi et al. (2019) with
sentence-level transcriptions, Karakanta et al. (2020) released MuST-Cinema
for subtitle generation of audio-visual content. It includes audio,
transcription, translation triplets of TED talks in 7 languages including
German, Spanish, French, Italian, Dutch, Portugese, Romanian. Scaling this up
with videos from the wild from YouTube, Sanabria et al. (2018) presented a
large-sclae dataset of How2 with of instructional videos on topics with $\sim$
8k clips and $\sim$ 2k hours. In addition to being multimodal, translations
into Portugese are also collected making the dataset multilingual. These
audio-visual recordings are valuable for (a) scaling multimodal multilingual
resources and (b) maximizing the utility of available data in low resourced
languages. (a) For scaling, Jones and Muftic (2020) collected interview
recordings from audio visual legacy media in N—uu, Kora and Khoekhoe
languages, emphasizing on the ethical steps to gather and publish such a
dataset. (b) For utilizing available data, Black (2019) massively scales to
700 languages with around 20 hours of speech for each language based on
religious recordings from YouTube annotated with a multi-pass alignment
technique. In addition to the video transcriptions, the chats based on videos
signal important events or highlights. Fu et al. (2017) present a dataset with
300 videos of 30-50 minute games from Twitch.tv channels based on League of
Legends championships and their corresponding $\sim$ 7k chats to study video
highlight prediction. Wang et al. (2019b) present VATEX based on the
Kinetics-600 actions dataset Kay et al. (2017) to perform the tasks of video
captioning and video-guided machine translation.

#### Online descriptions:

Internationally available online shopping platforms are a rich source of
product descriptions in multiple languages ranging beyond simple image
captions describing product semantics and their usage. Zhou et al. (2018)
present a parallel dataset of product descriptions from IKEA’s and UNIQLO’s
websites in 3 languages including English, French, and German. In contrast to
image captioning, the descriptions are not exactly parallel and are longer
than captions. Similarly, news outlets provide internationally viable
descriptions across multiple languages. Afli et al. (2017) introduced
MultiNews dataset comprising of images and their descriptions spanning 9
European languages sourced from Euronews website. The articles are not
automatic translations but are aligned finely over sentences from human
written texts in different languages.

###  2.3 Visual Interaction:

#### Visual Question Answering:

Raj Khan et al. (2021) extended VQA 2.0 Goyal et al. (2017) to a silver
dataset with 6 languages using Google translation and 5 pairs of code-switched
languages using matrix language frame theory Myers-Scotton (1997). Prior to
this, Gupta et al. (2020) translated these questions to Hinglish using
question generation method by Gupta et al. (2018). In addition to text,
Ramnath et al. (2021) created a synthetic dataset for spoken-VQA in 3
languages including English, Hindi, and Turkish. To increase question
diversity, Gao et al. (2015) crowdsourced FM-IQA dataset based on COCO images
with unconstrained human annotations for questions and answers. Later, Shimizu
et al. (2018) crowdsourced a Japanese visual question answering dataset with
$\sim$700k QA pairs based on Visual Genome and Japanese question types, which
are matched to the English versions to model cross-lingual transfer.

#### Vision-Language Navigation:

Extending Room2Room Anderson et al. (2018), inspired from localized
narratives. Ku et al. (2020) introduce a new dataset RoomXRoom in Hindi and
Telugu. Zarrieß et al. (2016) also propose PentoRef with around 20k utterances
in English and German transcribed and annotated with referring expressions for
reference resolution and generation.

###  2.4 Sentiment/Emotion Recognition:

Bagher Zadeh et al. (2020) introduced CMU-MOSEAS based on Youtube monologues
with $\sim$40k sentences in 4 languages including annotations for around 20
labels for sentiment, emotion, attributes etc.,

###  2.5 Comparison:

There is a stream of research on pivoting over the visual representations to
map multiple languages (discussed in §3). While this hypothesis is justifiable
for a majority of concepts, it does not cover culturally relevant and
distinguishable concepts. Comparing similar yet distinct words based on
cultural relevance and ambiguity is a unique challenge to MultiX. To address
this, Liu et al. (2021) introduce MARVL, focusing on culturally relevant
concepts in the corresponding nativities for 6 languages of Indonesian,
Mandarin Chinese, Swahili, Tamil, and Turkish. Elliott et al. (2017) introduce
Ambiguous COCO with a test set of ambiguous words in German, French and
English. and Gella et al. (2019) present the MultiSense dataset for
disambiguating verbs in different languages in German and Spanish.

#### Videos:

So far we have seen several datasets for static visuals, which can be extended
to videos Huang et al. (2021). This Multi-HowTo100M videos dataset includes
1.2 M videos, their corresponding speech and transcriptions from 9 languages
which can be used for massive pretraining.

###  2.6 Miscellaneous Datasets

This section covers more of the datasets in MultiX that are not covered
categorically in §2. IMAGACT Moneglia et al. (2014b) is based on an ontology
of actions from spontaneous speech over visuals. It is also intended to be
used as a multilingual dictionary of images. Gupta et al. (2021) worked
towards building a language-ontology based sets of images for object detection
and segmentation. Dominant classes and thematic domains are extracted from the
ontology and they are used to retrieve similar images from the web. The effort
targeted 20 European languages. Alberts et al. (2021) created a knowledge
graph over 900k unique images with 1.3M multilingual gloss for 14 languages.

##  3 Techniques

Modeling |  |  |  |  |
---|---|---|---|---|---
Category | Sub-category | Work | Cross-lingual? | AV Features | Text Features
Modular | Translation-first | Arora et al. (2020) | No | MFCC (speech) | Phrase
|  | Route et al. (2019) | No | MFCC (speech) | LSTM
|  | Ramnath et al. (2021) | Yes | MFCC (speech) | LSTM
|  | de Melo and Weikum (2010) | Yes | - | Surface forms
| Additional data | Gupta et al. (2021) | Yes | Faster R-CNN | mBART
|  | Lala et al. (2018) | Yes | ResNet-50 | MUSE
|  | Calixto and Liu (2017a) | Yes | VGG-19 | RNN
Embedding | Fusion | Zhang et al. (2020) | No | ResNet-50 | Transformer
Projection |  | Parida et al. (2021) | Yes | CNN | LSTM
|  | Caglayan et al. (2021) | Yes | Faster R-CNN | Transformer
|  | Parida et al. (2020) | Yes | InceptionResNetv2 | Transformer
|  | Gella et al. (2019) | Yes | ResNet-34 | Word2Vec
|  | Fu et al. (2017) | No | ResNet34-LSTM | Char-LSTM
| Multiplication | Susanto et al. (2021) | Yes | UVR | mBART
|  | Fei et al. (2021) | No | CLIP image encoder | CLIP text encoder(English)+SBERT
|  |  |  |  | paraphrase-multilingualmpnet-base-v2
|  |  |  |  | (for other languages)
|  | Ku et al. (2020) | Yes | EfficientNetB4 CNN | mBERT
|  | Shi et al. (2019) | No | ResNet-101 | FastText
|  | Mohammadshahi et al. (2019) | Yes | ResNet-152 | FastText
| CCA | Nakayama et al. (2020a) | Yes | Faster-RCNN, VGG16, | MeCab tokenizer,
|  |  |  | ResNet50 | word vectors
|  | Rotman et al. (2018) | Yes | VGG-19 | Skip-gram
| Distillation | Raj Khan et al. (2021) | Yes | LXMERT | mBERT
Mapping | Pivot on image | Huang et al. (2021) | Yes | R(2+1) spatio-temporal | mBERT
|  |  |  | CNN |
|  | Huang et al. (2020) | Yes | Faster-RCNN | XLM
|  | Kádár et al. (2018) | Yes | ResNet50 | GRU
|  | Gella et al. (2017) | Yes | VGG-19 | GRU
|  | Calixto and Liu (2017b) | Yes | VGG-19 | GRU
| Alignment | Fei et al. (2021) | Yes | VL-BERT | VL-BERT
|  | Surís et al. (2022) | Yes | ImageNet | BPE
|  | Nishihara et al. (2020) | Yes | ResNet-50 | Kytea for segmentation
|  | Huang et al. (2019b) | Yes | Faster-RCNN | mBERT
| Attention | Ramnath et al. (2021) | No | Faster R-CNN | IaK
|  | Singh et al. (2021) | Yes | CNN | RNN
|  | Mitzalis et al. (2021) | Yes | VL-BERT | mBERT
|  | Imankulova et al. (2020) | Yes | ResNet-50 | RNN
|  | Bagher Zadeh et al. (2020) | No | Conv1D + Transformer | Conv1D + Transformer
|  | Gupta et al. (2020) | Yes | R-CNN | fastText, CNN
|  | Zhou et al. (2018) | Yes | ResNet | LSTM
|  | Shimizu et al. (2018) | Yes | VGG-19 | LSTM
Other |  | Karakanta et al. (2021) | No | CNN | CNN
|  | Liu et al. (2021) | No | baselines | baselines
|  | Karakanta et al. (2020) | Yes | No images just NMT | Transformer
|  | Vilares et al. (2020) | No | X, Y coordinates | (linguistic ftrs)

Table 2: Categories of modeling approaches serving cross-lingually with
feature representations for modalities and languages.

This section presents a hierarchical categorization of modeling approaches as
depicted in Table 2.

###  3.1 Modular:

#### Translation-first:

Figure 2: Translation-first approaches

A broad idea of the modular approach to the translation-first approaches is
presented in Figure 2. As we can see the first step here is to translate the
source language input (often times unimodally in textual modality) into the
processing language which is equipped with the multimodal model to predict the
output. This chain of models can either be independently trained or jointly
trained end to end. Often times, the strategy is to translate into the
language that has a stronger performing multimodal model in a high-resource
language, which resolves to be English most of the times.

Most methods for spoken-VQA Zhang et al. (2017) includes a two-step process of
performing ASR and then answer prediction. In contrast, Ramnath et al. (2021)
proposes to use speech embeddings directly to predict the answer. This is
particularly useful when there are limited resources to perform ASR and
translation from low-resourced languages. Similarly, Huang et al. (2020) use
Google translate to translate the COCO captions to French and German for
pretraining followed by visual pivoting for training. In the same spirit of
modularity, Arora et al. (2020) approach this with individual components for
speech processing, translating the query and then performing retrieval in the
target language. Dictionaries (from the Kamusi project https://kamusi.org/))
and phrase based SMT Koehn et al. (2003) models with the MOSES toolkit Koehn
et al. (2007) is used for query translation followed by keyword search Trmal
et al. (2017) for retrieval. de Melo and Weikum (2010) also translate
Wikipedia pages first using Google Translate to respond to a user’s lexical
queries by responding with multilingual and multimodal information using token
similarity (for the first sense) and etymological approximation. Route et al.
(2019) use OpenNMT seq2seq framework to multitask decoder with input as text
and MFCC features for the task of TTS and use the hidden state of the first
output to predict IPA. Bugliarello et al. (2022) introduce the IGLUE benchmark
by aggregating pre-existing datasets and also collecting new ones across
several tasks such as —visual question answering, cross-modal retrieval,
grounded reasoning, multimodal entailment etc.,. They observe that “
translate-test transfer is superior to zero-shot transfer and that few-shot
learning is hard to harness for many tasks” making this approach both a very
strong baseline and state of the art.

#### Additional data (weak labels):

Figure 3: Weak supervision based approaches

Annotating the data with additional labels that can assist in modeling is
another way to approach multiple modalities and languages. This approach on a
broad level is presented in Figure 3. In the case of vision and language
tasks, either the images are annotated with labels in a processing language or
the source language input is annotated with additional labels in a processing
language. These annotations are often obtained using off-the-shelf tools as
weakly supervised labels. Upon the completion of this module, these additional
annotations obtained are used along with the initial inputs are given to a
multimodal model for the final predictions.

Gupta et al. (2021) use image recognition models to extract object tags in
English in the first stage to subsequently use them as weak labels in the
encoder to decode in a different language. This cross-lingual learning from
English tags helps map textual co-occurrences for different images in text
modality in addition to the image information. Calixto and Liu (2017a) use
global image features extracted from VGG19 as additional data to initialize
the encoder and the decoder hidden states in the RNNs. Barrault et al. (2018)
assimilate findings from various systems for multimodal machine translation
and observe that data augmentation significantly improves the task
performance. Along the same lines, Lala et al. (2018) augment n-best lists of
translated data from French, German and Czech to English to train an NMT
model. Despite being noisy, this augmentation of silver data performs better
than the corresponding baseline.

###  3.2 Embedding Projections:

#### Fusion and Gating:

Figure 4: Fusion based approaches

Fusion based approaches combine the information from different modalities, and
the text from different languages in a defined design. They can be categorized
into early fusion in which the inputs are combined at the feature embedding
level or late fusion where the projections are learnt by a deep task dependent
network, which are combined in at a later position. While early fusion is
considered an integration at the feature level, the late fusion is considered
an integration at the semantic level. An example of the approaches based on
fusion is presented in Figure 4 in the recent era of pretraining. The signals
from both the languages are integrated using a translation based objective
described in detail in the following paragraph. The objective across
modalities is optimized with image region based objectives.

Zhang et al. (2020) introduce universal visual representations (UVR) to
perform multimodal machine translation by leveraging a group of images that
have similar topics contained in the source sentence. This similarity is
determined by tf-idf scores followed by the fusion of the visual information
with gating to predict the target translations. Similarly, Parida et al.
(2021) performs region specific captioning by fusing image features with
region coordinates features. Caglayan et al. (2021) extend the TLM
(translation language model) objective Conneau and Lample (2019) to include
regional image features via VTLM model that concatenates the translations of
both languages as input to the same encoder optimized for masked textual and
visual tokens. The model is optimized for TLM and MRC (masked region
classification) where the visual and textual tokens in the input are masked.
In addition, a simple concatenation of image features and source language
features also demonstrate decent performance and are also used as baselines
for several tasks. For instance, Parida et al. (2020) represent images using
InceptionResNetv2 and text using transformers and combine their
representations to perform multimodal machine translation. Similarly, Gella et
al. (2019) concatenate the visual and textual features for predicting the word
sense in another language. Instead of direct concatenation, Fu et al. (2017)
add an MLP layer on the modality representations to predict highlights in a
video. The recent era of pretraining also relies on these heavily silver
standard data by using automatic translation tools to first translate the data
Zhou et al. (2021) and use them for MultiX modeling.

#### Matching:

Figure 5: Matching based approaches

The matching based techniques are also specific to the task at hand. The
embedding projections for both the languages are first combined. This combine
multilingual embedding is matched with the visual input to predict the output.
An overview of these approaches is presented in Figure 5.

Susanto et al. (2021) use the universal visual representations described
earlier and compute similarities with the language encodings obtained from
mBART. Similarly, Fei et al. (2021) perform entity, sentence and image
retrieval on VisualSem knowledge graph by computing the similarity between
image embeddings and text embeddings in multiple languages. The visual
representation is derived using the CLIP image encoder Radford et al. (2021)
and the textual representation is derived using the CLIP text encoder for
English and SBERT model paraphrase-multilingualmpnet-base-v2 Reimers and
Gurevych (2019) for other languages. An agent similar to Reinforced Cross-
Modal Matching Wang et al. (2019a) is adapted by replacing LSTMs with
successive 1D convolutions to encode longer utterances for multilingual
navigation in RXR Ku et al. (2020). They note that despite 3 times the data,
training a single multilingual agent on several languages performs worse
compared to their monolingual counterparts, while the multilingual agent
outperforms with a multitask setting combining annotations from Room2Room. For
matching textual representations to visual inputs and combining constituents,
Shi et al. (2019) optimize for the hinge triplet loss jointly by building
constituency structures recursively Kitaev and Klein (2018) with a bottom-up
score-sample-combine approach. The visual semantic embedding space Kiros et
al. (2014) is created with cosine similarity based matching score in the joint
space. Likewise, Mohammadshahi et al. (2019) computes this similarity where
the text representation is a combination of multiple languages with internal
alignment using kNN based algorithm. Afli et al. (2017) use word based and
named entity based scoring strategies to align news corpora from Euronews
spread across 9 languages.

#### Canonical Correlations:

Figure 6: CCA based approaches

Canonical Correlation Analysis (CCA) is a method to infer information from
cross-covariance matrices. With multiple vectors of random variables along
with correlations among these variables, then this method is used to find a
linear combination of these variables with maximum correlation among
themselves. This is depicted in Figure 6.

Nakayama et al. (2020a) use a generalized CCA (GCCA) Gong et al. (2014) to
include English translation as an additional modality to perform phrase
localization in images using Japanese text. This technique is earlier
developed to use higher level semantics as the third modality to perform text-
to-image alignment. Similarly, Japanese, English and images are the three
modalities where the nearest embedding in the canonical subspace is retrieved.
GCCA can thus perform cross-lingual retrieval. Instead of including an
additional modality, Rotman et al. (2018) present partial CCA (PCCA) to
maximize the canonical correlation of the multilingual descriptions in two
languages conditioned on the shared variable of the image representation. The
difference between PCCA and GCCA is that the former attempts to maximize the
canonical correlations of all pairs of views whereas the latter condition two
variables on the third shared one. Leviant and Reichart (2015) extended
SimLex-999 and wordsim353 annotations to Italian, German, and Russian, which
are later experimented for CCA performance by Rotman et al. (2018).

#### Distillation:

Figure 7: Distillation based approaches

Under the continued presupposition that the performance of the multimodal
models is better in a high resource language. As a teacher model is learnt in
this high resource language (usually English), the student model imitates this
teacher model to learn task specific parameters to achieve a generalized
performance. This is presented in Figure 7.

Building on top of the monolingual multimodal models, Raj Khan et al. (2021)
use distillation methods to transfer the learning to multilingual and code-
mixed scenarios for VQA. The teacher network is trained based on English
LXMERT model, the parameters of which are used to train the student network to
optimize for 2 mean squared errors and 2 binary cross entropy losses. The MSE
are computed for the CLS token, object attention loss and the BCE losses are
computed between the answer probability scores of teacher and student
networks, and between the gold and the predicted answer from the student
network.

###  3.3 Pivoting

#### Pivoting on the image:

Multilingual multimodal datasets are often translated to different languages
for the same visual context. Huang et al. (2021) use a noise contrastive
objective to the visually pivoted translation pairs between languages in
inter-modal (i.e one language and visual), and intra-modal (i.e 2 languages
and visual) ways. The goal of optimizing these objectives is to align the
visual to the transcriptions in different languages. Prior to this, they
perform pseudo visual pivoting Huang et al. (2020) motivated by back
translation to align multilingual spaces for the same image. Synthetic
multilingual captions from the source image are used to reconstruct the
synthetic captions from their corresponding translations and for translation
of the paired captions. Kádár et al. (2018) train a multilingual model to
minimize the ranking loss updated for the prediction of image and caption of
one language from caption in another language. A very similar approach with
two pairwise ranking objectives scoring sentences and images and another
scoring sentences in two different languages is also used by Calixto and Liu
(2017b). Gella et al. (2017) optimize for a pivoted loss function on the image
to bring gold description and image closer compared to other irrelevant
captions using monolingual corpora from multiple languages.

#### Alignment:

Figure 8: Alignment based approaches

Alignment is the method of forming a grouped plan to relate the languages and
the modalities. Typically, this is achieved using an attention matrix. The
attention based mehods are described in detailed after this method.
Specifically, in Figure 8 the positioning matrix between the image (on the
y-axis) and one of the languages (on the x-axis) to make predictions in the
second language.

Fei et al. (2021) perform cross-lingual cross-modal pretraining with a unified
framework using pretrained objectives adopted from prior studies including MLM
(Masked Language Modeling) Devlin et al. (2019), MRC (Masked region
classification) Li et al. (2020); Su et al. (2020), TLM, CLTR (Cross-lingual
text recovery), and CMTR (Cross-modal text recovery) Huang et al. (2019a).
Specifically, CLTR handles alignment of the parallel sentences in different
languages by using bi-linear attention mechanism to compute an attended
representation of the input sentence in the source language and its parallel
sentence in a different language. Similarly, CMTR computes the alignment
between word features and bounding box features by computing their bi-linear
attention. Similar alignment techniques are also used to compute cross-lingual
attention Garg et al. (2019). Nishihara et al. (2020) use a transformer based
multimodal neural machine translation model with an image and source language
sentence encoder and a target language decoder. The main addition to leverage
cross-lingual information is derived by minimizing the cross entropy between
one attention head of the multi-head cross-lingual attention and the word
alignments obtained using MGIZA Garg et al. (2019). This cross-lingual
alignment techniques such as XeroAlign Gritta and Iacobacci (2021) and
CrossAligner Gritta et al. (2022) is very prevalent in unimodal multilingual
scenarios. Introducing a diversity objective to explicitly capture different
forms in the joint embedding space is an effective way to align terms in
multiple langauges pivoted on an image Huang et al. (2019b). They perform
multi-head attention to attend to different visual objects and the textual
semantics in the caption with a margin-based diversity loss. Recently, Surís
et al. (2022) present a modeling technique to learn aligned embedding space
through a vision-based transitive relation across languages that learns an
alignment model across languages if and only if the visuals associated with
them are similar.

#### Attention:

Bagher Zadeh et al. (2020) use multimodal transformer Tsai et al. (2019)
composed of Conv1D and transformer to encode each modality to benchmark CMU-
MOSEAS. Each langauge is independently modeled with the visual attributes
without relying on cross-lingual information. Ramnath et al. (2021) use a co-
attention mechanism to fuse selective information from the image and question
for spoken visual question answering. Self-attention is performed on the
speech signal and the question embeddings are used to query the attention on
the image to answer a question from knowledge graphs represented by IaK
Ramnath and Hasegawa-Johnson (2020). Instead of fusing the attended
representation, Shimizu et al. (2018) replace the visual attention maps learnt
from English to perform VQA in Japanese. The motivation for this idea is
despite varying attention maps, their foci corresponding to the answer or the
subject of the question have reasonable overlap across languages. This is
followed by parallel coattention between visual and textual features. Similar
cross attention on a single input language also shows improvements in
multimodal machine translation in addition to multitasking with the auxiliary
objective to construct a vision and language joint semantic embedding Zhou et
al. (2018). Attention-based NMT frameworks with attention on spatial features
in the images are also used by combining multiple languages Singh et al.
(2021). They define three types of mapping based on the source and target
languages - many to one, one to many and many to many (where first and second
indicate the number of source and target languages respectively). The many to
one and many to many paradigms are cross-lingual where text from other
languages is used during training. A similar approach of attending to the
source sentence and the image is performed by Imankulova et al. (2020) for the
task of simultaneous multimodal machine translation using wait-k approach.
Instead of attention across languages, batching the data from different
languages during training enables learn a multlingual representation Gupta et
al. (2020). Sharing this representation is also extended to attention based
soft layer-sharing by attending over the encoder for each language and each
layer fusing the modalities with bilinear attention.

#### Other:

Karakanta et al. (2021) use speech translation based on an audio encoder and a
text decoder using listen-and-translate Berard et al. (2016) and direct
foreign speech translation Weiss et al. (2017). These techniques are combined
with the efficiency of the wait-k strategy Ma et al. (2019). Karakanta et al.
(2020) present an NMT based model for subtitle generation of audio visual
content using a transformer based seq2seq architecture for text only (visual
features are not used in the model). Liu et al. (2021) set up baselines for
MARVL on monolingual multimodal models including VL-BERT Su et al. (2020),
VisualBERT Li et al. (2019a), ViLBERT Lu et al. (2019), LXMERT, and by
extending English based multimodal models to multilingual scenarios. UNITER
Chen et al. (2020) is extended by initializing the text encoder with mBERT and
XLM-R as mUNITER and xUNITER respectively. A similar approach is adopted by
Mitzalis et al. (2021) to propose BERTGEN by fusing VL-BERT with M-BERT
initialization. Specifically, it is demonstrated successfully for the task of
MMT where unrolling is used as masking to create the next example and self
attention is performed at every time step.

#### Systems and Analysis

Akhlaghi et al. (2020) built LARA (Learning and Reading Assistant) which is an
open source platform that converts plain texts into multimodal online
versions. It involves semi-automatically tagging text, adding annotations,
recording audio to highlight relevant information for suitable for language
learners. Along similar lines, Willemsen et al. (2018) also develop an L2
acquisition (L2TOR ITS) with a curriculum, state tracking and a template based
NLG module for interaction. Vilares et al. (2020) developed a description
generator module for visually-impaired users to play 3 rogue-like games (The
Inner Eye, The Accessible Dungeon, Hamsun’s Amulet). They follow a modular
approach to content planning, micro-planning and surface realization of the
NLG system proposed by Reiter and Dale (1997) using textual features at
lexical, syntactic (POS tags), semantic (Multilingual Central Repository
Agirre et al. (2012)), discourse and pragmatic levels. Xu et al. (2020a) built
Xiaomingbot which is a software news reading robot with the capabilities of
news generation, news translation into other languages, news reading along
with avatar animation. The text for summarization is represented using BERT
and a neural network with sliding window is used to generate smooth animations
for the reader. Poignant et al. (2016) built a framework CAMOMILE client-
server platform which is a collaborative annotation framework for collecting
multimodal, multimedia, multilingual (3M) data. It is used to collect data for
for MediaEval task with 20 annotators and upto 73k annotations. Rinsche (2005)
presents LTC Communicator offering software web based multilingual support for
vendors of international markets to support customers across countries
interacting in multiple languages, which can be extended to information
exchange via visuals of the products.

Analysis Moneglia and Varvara (2020) perform analysis on IMAGACT Moneglia et
al. (2014a) to understand the relation between thematic structure and semantic
and lexical variation of action words.

##  4 Roadmap

Assimilating the takeaways from tasks and modeling for MultiX, we investigate
the forefront of current challenges and promising directions.

###  4.1 General Trends

Generally speaking, the studies on MultiX has seen a paradigm shift with
pretraining that demands a vast amount of data. Unlike monolingual processing
that just needs raw data for self-supervised learning, the constraints of
modalities and languages requires a degree of parallelness in the data.
Catering to this need, the field also obliged to using silver standard
translated data to perform large scale modeling of MultiX compared to prior
approaches of annotating gold standard multilingual data.

The base architectures or backbone models catering to the language aspect are
mostly multilingual models such as mBART Liu et al. (2020), etc., Prior to the
advent of pretrained multilingual models, most of the backbone architectures
are CNN and LSTM based for visual and textual information respectively.
Similarly, MFCC features are extracted to represent the speech modality. Some
multilingual models also train on multiple languages together with an
identifier token to uniquely identify to predict for that specific language
Mitzalis et al. (2021).

Overall, the field of multimodality seems to be extending a welcoming hand by
monolingual unimodal processing towards MultiX. However, the gap still remains
owing to the inter-disciplinary topics of translation and multimodality. This
creates an opportunity to progress in the field to build equitable
technologies for all languages.

###  4.2 Challenges and Directions

#### Comparison to Unimodal approaches:

From the findings of the shared task in Multimodal Machine Translation,
Barrault et al. (2018) note that text only models are as competitive as
multimodal models. While this is a common problem in monolingual multimodal
cases, this observation is also an emerging trend in MultiX owing to the lack
of strong underlying backbone architectures. We encourage the community to
scrupulously sub-select instances in the dataset containing concepts/words
with ambiguity that enforces understanding of both modalities to predict the
correct output. Maintaining the training data distributions with a wide
coverage of high quality instances with this unimodal or monolingual ambiguity
is promising to better compare unimodal and multimodal models.

> Direction: Designing tasks with ambiguous relations where a single modality
> curtails quality predictions are critical.

#### Few shot Cross-lingual Transfer:

Blasi et al. (2022) presented disparities in NLP technologies due to the
societal and academic factors. In a similar spirit, we present the demographic
utility of MultiX datasets in Figure 9, which proliferate to modeling
resources. We can observe that the digital presence in terms of the number of
Wikipedia articles is correlated with datasets built in these languages
irrespective of the number of speakers. In this sub-optimal reality,
incentivizing research in underrepresented and endangered languages Bird
(2020) is practical with zero or few-shot transfer. Lauscher et al. (2020)
study the relationship between the success of transfer with varying levels of
tasks, language proximity, amount of target language data. concluding that few
shot finetuning has significant boost over zero-shot transfer. A common
approach for a multilingual multimodal model, finetune the model for the task
on a high-resourced language, perform 0-shot or few-shot finetuning on the
low-resourced languages.

> Direction: Leveraging shared linguistic units grounded visually at syntactic
> & lexical levels to maximize supervision, assists transfer from high
> resourced languages.

#### Multi vs Bilingual:

Multilingually aware end-to-end systems are better for error propagation Zhu
et al. (2019); Xu et al. (2020b) in unimodal scenarios. Various studies
demonstrate that multilingual training improves performance over bilingual
training, or by extrapolation, monolingual training Kádár et al. (2018).
Similarly, the asymmetric loss Vendrov et al. (2016) in pivoted models by
Gella et al. (2017) suggest that multilingual information sharing is useful.
This observation motivates that when collecting multimodal data for a new
language, it is beneficial to collect for the same images with existing data
in another language Chandu et al. (2021) to exploit caption-to-caption
prediction objectives along with image prediction objectives. However, we need
to be cognizant of the abberations where Ku et al. (2020) were not able to
directly improve performance by multilingual models, so they take the task
diversity as an additional dimension for multitasking to improve multilingual
navigation.

Figure 9: Distribution of MultiX datasets for languages with the number of
speakers and Wikipedia articles

> Direction: Multilingual learning has the potential to enhance MultiX, which
> benefits from parallel/pseudo-parallel data across multiple languages.

#### Curse of Multilinguality:

Monolingual performance with fixed model capacity trained on multiple
languages starts weakening with the addition of new languages, known as the
curse of multilinguality Conneau et al. (2020). This problem of locked
capacity can be fixed with augmentation of model parameters and data. First,
augmenting data can be noisy supervision with silver annotations or
unsupervised data. Second, augmenting models involves adding language specific
adapters Pfeiffer et al. (2020) or multilingual tokenizers Rust et al. (2021).
The language and task specific adapters are tuned separately and the adapter
of the source language is replaced with that of the target language in
inference for the same multimodal model. However, a very recent work by Bapna
et al. (2022) showed that the curse of multilinguality does not hold when
there are several more languages of the order of thousands based on
experiments conducted where the authors do not observe this interference
effect on a single multilingual model.

> Direction: Adapter based tuning is a parameter-efficient way to ensure
> continual cross-lingual transfer for pretrained multimodal models.

#### Translationese artifacts:

While translating existing monolingual multimodal resources offers developing
parallel data effective for cross-lingual learning, they often fall prey to
translationese without explicit caution. Translationese Koppel and Ordan
(2011) is characterized as the language style resulting from a translator
attempting to closely replicate the properties of source text thereby heavily
biasing it. Translations are often error-proned with low-resourced languages
and colloquial usage of terms adversely impacting evaluation Graham et al.
(2020). Partial translations can help mitigate this. Studies show that this
language style also adversely impacts the evaluation Graham et al. (2020).
First, translating continuous monolingual chunks instead of full sentences
inherently models cross-lingual contextual information. A multimodal code-
switched stream is explored in pretraining M3P Ni et al. (2021) and in
unimodal tasks Qin et al. (2020); Krishnan et al. (2021) with random
replacements. Second, a hybrid approach is translating approximated or
delexicalized templates and then filling in with regionally relevant tokens in
the translated template Ding et al. (2022), which can also help reduce human
effort and cost for annotations. Therefore, translating partial utterances
grounded in entities, context, etc., mitigates translating entire source
annotation, thereby reducing bias.

> Direction: Minimizing source language artifacts to ground cross-lingual
> annotations curbs translationese.

#### Language Pivoting:

As we studied in §3, pivoting is primarily done on the image. Realistically,
availability of data for the axes, (i) paired images and text, compared to
other languages, and (ii) paired translations for English is relatively more
common. This imminently rises the opportunity to pivot on a resource rich
languages common between both axes. Gu et al. (2018) studied this way of
unpaired captioning with language pivoting performing machine translation and
autoencoding with paired data to in turn perform captioning without paired
data. However, selecting English as the pivot language is not the best choice
Anastasopoulos and Neubig (2020) and this selection based on typological
distance is still unsolved.

> Direction: Pivoting on a resource rich common language is promising for
> unpaired learning with a preference of common language proximal to target
> language.

#### Expansive X:

Societal grounding influences regional language usage, making the dialects,
vernaculars, accents, idiolects and other colloquial variants different from
the language itself. In addition, catering to domain shifts Ramponi and Plank
(2020) in languages encourages out-of-distribution generalization. Expanding
to these ever-evolving variants of Xs demands reliable evaluations in an ever-
evolving area that we need to be cognizant of.

> Direction: Sustainable research demands developing comparable evaluation
> framework leaning into the cultural differences and morphological richness.

With the rapid development of multimodal and multilingual axes independently,
it is now more critical than ever to build multimodal models accommodative of
several languages. With this broad goal, this paper presents categories of
tasks, datasets, and methods along with sketching out existing challenges and
a roadmap ahead.

## Limitations

This paper studies unifying two of the important dimensions in contexts.
However, the dimensions are not limited to the scope of multiple languages or
modalities. First, on top of the variants discussed in §4 in “Expansive X”,
domains are an important category of language usage that warrant specialized
care to ensure competitive performance. This survey does not broadly include
the domains within X, however, it is imperative to understand the effects of
domains and distributional shifts as we progress on the dimensions of
languages and modalities. Note that with the existing two dimensions, the
distributional shift can occur within a language or the visual sources,
thereby coupling the complexity of the space. Second, the majority of the work
covered in this paper does not address underrepresented languages due to lack
of existing literature. However, we need to take inspiration from the growing
research on low resourced languages Hedderich et al. (2021) to ensure we learn
from what works best in such unimodal cases to coalesce with the multimodal
techniques. To deliver an equitable societal impact of language technologies,
it is important to address this long-tail of low-resourced languages as well.

## Broader Impact

The fields of multilingual NLP and multimodality are racing with the vibrant
and fierce contributions from our community. However, these resources and
methods are not directly trivially extensible to languages other than the one
they are developed in, which is English in most cases. Hence, it is important
essential to retrospect the resources and techniques that researchers proposed
in MultiX to assimilate the takeaways to consequently build informed and
equitable models. Extending existing vision-and-language resources to multiple
languages ensures not only the re-usability of the work done so far but also
builds an ecosystem to compare the scalability and robustness of the models
for continued research. We hope this work can be used as an index into the
wealth of scattered resources unifying multimodality and multilinguality for
researchers in both these fields to be cognizant of the challenges and
promising directions to invest upon while making modeling choices.

## Ethical Considerations

We do not presage any immediate ethical concerns arising directly from our
work on surveying the landscape of MultiX. The taxonomy of datasets and
techniques focuses on multilingual multimodality i.e., MultiX, which does not
address undesired and harmful biases inherently present in the datasets.
Moreover, the cultural norms dictate the linguistic acceptability of
translated resources. Therefore the proposed translation based weak
supervision methods need to carefully take into account cultural
qualifications specific to each language and region.

##  A Acronyms

Here are some of the acronyms discussed in the paper along with their
expansions.

  * •

ASR: Automatic Speech Recognition

  * •

BCE: Binary Cross Entropy

  * •

COCO: Common Objects in Context

  * •

CCA: Canonical Correlation Analysis

  * •

CLIP: Contrastive Language–Image Pre-training

  * •

CMU-MOSEAS: CMU Multimodal Opinion Sentiment, Emotions and Attributes

  * •

FM-IQA: Freestyle Multilingual Image Question Answering

  * •

IaK: Image as Knowledge

  * •

IPA: International Phonetic Alphabet

  * •

MARVL: Multicultural Reasoning over Vision and Language

  * •

MFCC: Mel Frequency Cepstral Coefficient

  * •

MMT: Multimodal Machine Translation

  * •

MSE: Mean squared error

  * •

MuST-C: Multilingual Speech Translation Corpus

  * •

MuST-Cinema: a Multilingual Speech Translation Cinema

  * •

NMT: Neural Machine Translation

  * •

OpenCLIR: Open Cross Language Information Retrieval

  * •

OpenNMT: Open Neural Machine Traslation

  * •

QA: Question Answering

  * •

RNN: Recurrent Neural Network

  * •

seq2seq: sequence to sequence

  * •

TED: Technology, Entertainment, and Design

  * •

TTS: Text-to-Speech

  * •

VQA: Visual Question Answering

## References

  * Afli et al. (2017) Haithem Afli, Pintu Lohar, and Andy Way.  MultiNews: A web collection of an aligned multimodal and multilingual corpus.  In _Proceedings of the First Workshop on Curation and Applications of Parallel and Comparable Corpora_ , pages 11–15, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing.  URL https://aclanthology.org/W17-5602.
  * Agirre et al. (2012) Aitor González Agirre, Egoitz Laparra, German Rigau, and Basque Country Donostia.  Multilingual central repository version 3.0: upgrading a very large lexical knowledge base.  In _GWC 2012 6th International Global Wordnet Conference_ , page 118, 2012.
  * Akhlaghi et al. (2020) Elham Akhlaghi, Branislav Bedi, Fatih Bektaş, Harald Berthelsen, Matthias Butterweck, Cathy Chua, Catia Cucchiarin, Gulsen Eryigit, Johanna Gerlach, Hanieh Habibi, Neasa Ni Chiarain, Manny Rayner, Steinor Steingrimsson, and Helmer Strik.  Constructing multimodal language learner texts using LARA: Experiences with nine languages.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 323–331, Marseille, France, May 2020. European Language Resources Association.  ISBN 979-10-95546-34-4.  URL https://aclanthology.org/2020.lrec-1.40.
  * Alberts et al. (2021) Houda Alberts, Ningyuan Huang, Yash Deshpande, Yibo Liu, Kyunghyun Cho, Clara Vania, and Iacer Calixto.  VisualSem: a high-quality knowledge graph for vision and language.  In _Proceedings of the 1st Workshop on Multilingual Representation Learning_ , pages 138–152, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.mrl-1.13.  URL https://aclanthology.org/2021.mrl-1.13.
  * Anastasopoulos and Neubig (2020) Antonios Anastasopoulos and Graham Neubig.  Should all cross-lingual embeddings speak english?  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 8658–8679. Association for Computational Linguistics, 2020.  doi: 10.18653/v1/2020.acl-main.766.  URL https://doi.org/10.18653/v1/2020.acl-main.766.
  * Anderson et al. (2018) Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel.  Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.  In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_ , pages 3674–3683. Computer Vision Foundation / IEEE Computer Society, 2018.  doi: 10.1109/CVPR.2018.00387.  URL http://openaccess.thecvf.com/content_cvpr_2018/html/Anderson_Vision-and-Language_Navigation_Interpreting_CVPR_2018_paper.html.
  * Arora et al. (2020) Piyush Arora, Dimitar Shterionov, Yasufumi Moriya, Abhishek Kaushik, Daria Dzendzik, and Gareth Jones.  An investigative study of multi-modal cross-lingual retrieval.  In _Proceedings of the workshop on Cross-Language Search and Summarization of Text and Speech (CLSSTS2020)_ , pages 58–67, Marseille, France, May 2020. European Language Resources Association.  ISBN 979-10-95546-55-9.  URL https://aclanthology.org/2020.clssts-1.10.
  * Bagher Zadeh et al. (2020) AmirAli Bagher Zadeh, Yansheng Cao, Simon Hessner, Paul Pu Liang, Soujanya Poria, and Louis-Philippe Morency.  CMU-MOSEAS: A multimodal language dataset for Spanish, Portuguese, German and French.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1801–1812, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.141.  URL https://aclanthology.org/2020.emnlp-main.141.
  * Baltrusaitis et al. (2019) Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency.  Multimodal machine learning: A survey and taxonomy.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 41(2):423–443, 2019.  doi: 10.1109/TPAMI.2018.2798607.  URL https://doi.org/10.1109/TPAMI.2018.2798607.
  * Bapna et al. (2022) Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Axelrod, Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, Yanping Huang, Zhifeng Chen, Yonghui Wu, and Macduff Hughes.  Building machine translation systems for the next thousand languages.  _CoRR_ , abs/2205.03983, 2022.  doi: 10.48550/arXiv.2205.03983.  URL https://doi.org/10.48550/arXiv.2205.03983.
  * Barrault et al. (2018) Loïc Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank.  Findings of the third shared task on multimodal machine translation.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 304–323, Belgium, Brussels, October 2018. Association for Computational Linguistics.  doi: 10.18653/v1/W18-6402.  URL https://aclanthology.org/W18-6402.
  * Berard et al. (2016) Alexandre Berard, Olivier Pietquin, Christophe Servan, and Laurent Besacier.  Listen and translate: A proof of concept for end-to-end speech-to-text translation.  _CoRR_ , abs/1612.01744, 2016.  URL http://arxiv.org/abs/1612.01744.
  * Bird (2020) Steven Bird.  Decolonising speech and language technology.  In Donia Scott, Núria Bel, and Chengqing Zong, editors, _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_ , pages 3504–3519. International Committee on Computational Linguistics, 2020.  doi: 10.18653/v1/2020.coling-main.313.  URL https://doi.org/10.18653/v1/2020.coling-main.313.
  * Black (2019) Alan W. Black.  CMU wilderness multilingual speech dataset.  In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019_ , pages 5971–5975. IEEE, 2019.  doi: 10.1109/ICASSP.2019.8683536.  URL https://doi.org/10.1109/ICASSP.2019.8683536.
  * Blasi et al. (2022) Damián E. Blasi, Antonios Anastasopoulos, and Graham Neubig.  Systematic inequalities in language technology performance across the world’s languages.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 5486–5505. Association for Computational Linguistics, 2022.  URL https://aclanthology.org/2022.acl-long.376.
  * Bugliarello et al. (2022) Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulic.  IGLUE: A benchmark for transfer learning across modalities, tasks, and languages.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 2370–2392. PMLR, 2022.  URL https://proceedings.mlr.press/v162/bugliarello22a.html.
  * Burger et al. (2003) Susanne Burger, Erica Costantini, and Fabio Pianesi.  Communicative strategies and patterns of multimodal integration in a speech-to-speech translation system.  In _Proceedings of Machine Translation Summit IX: Papers_ , New Orleans, USA, September 23-27 2003.  URL https://aclanthology.org/2003.mtsummit-papers.5.
  * Caglayan et al. (2021) Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, and Lucia Specia.  Cross-lingual visual pre-training for multimodal machine translation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1317–1324, Online, April 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.eacl-main.112.  URL https://aclanthology.org/2021.eacl-main.112.
  * Calixto and Liu (2017a) Iacer Calixto and Qun Liu.  Incorporating global visual features into attention-based neural machine translation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 992–1003, Copenhagen, Denmark, September 2017a. Association for Computational Linguistics.  doi: 10.18653/v1/D17-1105.  URL https://aclanthology.org/D17-1105.
  * Calixto and Liu (2017b) Iacer Calixto and Qun Liu.  Sentence-level multilingual multi-modal embedding for natural language processing.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017_ , pages 139–148, Varna, Bulgaria, September 2017b. INCOMA Ltd.  doi: 10.26615/978-954-452-049-6˙020.  URL https://doi.org/10.26615/978-954-452-049-6_020.
  * Chandu et al. (2021) Khyathi Raghavi Chandu, Yonatan Bisk, and Alan W Black.  Grounding ‘grounding’in nlp.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4283–4305, 2021.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  UNITER: universal image-text representation learning.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX_ , volume 12375 of _Lecture Notes in Computer Science_ , pages 104–120. Springer, 2020\.  doi: 10.1007/978-3-030-58577-8“˙7.  URL https://doi.org/10.1007/978-3-030-58577-8_7.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample.  Cross-lingual language model pretraining.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 7057–7067, 2019.  URL https://proceedings.neurips.cc/paper/2019/hash/c04c19c2c2474dbf5f7ac4372c5b9af1-Abstract.html.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 8440–8451. Association for Computational Linguistics, 2020.  doi: 10.18653/v1/2020.acl-main.747.  URL https://doi.org/10.18653/v1/2020.acl-main.747.
  * Dabre et al. (2020) Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan.  A survey of multilingual neural machine translation.  _ACM Comput. Surv._ , 53(5):99:1–99:38, 2020\.  doi: 10.1145/3406095.  URL https://doi.org/10.1145/3406095.
  * de Melo and Weikum (2010) Gerard de Melo and Gerhard Weikum.  Providing multilingual, multimodal answers to lexical database queries.  In _Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10)_ , Valletta, Malta, May 2010. European Language Resources Association (ELRA).  URL http://www.lrec-conf.org/proceedings/lrec2010/pdf/312_Paper.pdf.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics, 2019\.  doi: 10.18653/v1/n19-1423.  URL https://doi.org/10.18653/v1/n19-1423.
  * Ding et al. (2022) Bosheng Ding, Junjie Hu, Lidong Bing, Mahani Aljunied Mahani, Shafiq R. Joty, Luo Si, and Chunyan Miao.  Globalwoz: Globalizing multiwoz to develop multilingual task-oriented dialogue systems.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 1639–1657. Association for Computational Linguistics, 2022.  URL https://aclanthology.org/2022.acl-long.115.
  * Elliott et al. (2016) Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia.  Multi30k: Multilingual english-german image descriptions.  In _Proceedings of the 5th Workshop on Vision and Language, hosted by the 54th Annual Meeting of the Association for Computational Linguistics, VL@ACL 2016, August 12, Berlin, Germany_. The Association for Computer Linguistics, 2016.  doi: 10.18653/v1/w16-3210.  URL https://doi.org/10.18653/v1/w16-3210.
  * Elliott et al. (2017) Desmond Elliott, Stella Frank, Loïc Barrault, Fethi Bougares, and Lucia Specia.  Findings of the second shared task on multimodal machine translation and multilingual image description.  In Ondrej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Philipp Koehn, and Julia Kreutzer, editors, _Proceedings of the Second Conference on Machine Translation, WMT 2017, Copenhagen, Denmark, September 7-8, 2017_ , pages 215–233. Association for Computational Linguistics, 2017.  doi: 10.18653/v1/w17-4718.  URL https://doi.org/10.18653/v1/w17-4718.
  * Fei et al. (2021) Hongliang Fei, Tan Yu, and Ping Li.  Cross-lingual cross-modal pretraining for multimodal retrieval.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3644–3650, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.285.  URL https://aclanthology.org/2021.naacl-main.285.
  * Fu et al. (2017) Cheng-Yang Fu, Joon Lee, Mohit Bansal, and Alexander Berg.  Video highlight prediction using audience chat reactions.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 972–978, Copenhagen, Denmark, September 2017\. Association for Computational Linguistics.  doi: 10.18653/v1/D17-1102.  URL https://aclanthology.org/D17-1102.
  * Funaki and Nakayama (2015) Ruka Funaki and Hideki Nakayama.  Image-mediated learning for zero-shot cross-lingual document retrieval.  In Lluís Màrquez, Chris Callison-Burch, Jian Su, Daniele Pighin, and Yuval Marton, editors, _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_ , pages 585–590. The Association for Computational Linguistics, 2015.  doi: 10.18653/v1/d15-1070.  URL https://doi.org/10.18653/v1/d15-1070.
  * Gangi et al. (2019) Mattia Antonino Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi.  Must-c: a multilingual speech translation corpus.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 2012–2017. Association for Computational Linguistics, 2019\.  doi: 10.18653/v1/n19-1202.  URL https://doi.org/10.18653/v1/n19-1202.
  * Gao et al. (2015) Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu.  Are you talking to a machine? dataset and methods for multilingual image question.  In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 2296–2304, 2015.  URL https://proceedings.neurips.cc/paper/2015/hash/fb508ef074ee78a0e58c68be06d8a2eb-Abstract.html.
  * Garg et al. (2019) Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik.  Jointly learning to align and translate with transformer models.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 4452–4461. Association for Computational Linguistics, 2019.  doi: 10.18653/v1/D19-1453.  URL https://doi.org/10.18653/v1/D19-1453.
  * Gella et al. (2017) Spandana Gella, Rico Sennrich, Frank Keller, and Mirella Lapata.  Image pivoting for learning multilingual multimodal representations.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2839–2845, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.  doi: 10.18653/v1/D17-1303.  URL https://aclanthology.org/D17-1303.
  * Gella et al. (2019) Spandana Gella, Desmond Elliott, and Frank Keller.  Cross-lingual visual verb sense disambiguation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1998–2004, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1200.  URL https://aclanthology.org/N19-1200.
  * Gong et al. (2014) Yunchao Gong, Qifa Ke, Michael Isard, and Svetlana Lazebnik.  A multi-view embedding space for modeling internet images, tags, and their semantics.  _Int. J. Comput. Vis._ , 106(2):210–233, 2014\.  doi: 10.1007/s11263-013-0658-4.  URL https://doi.org/10.1007/s11263-013-0658-4.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the V in VQA matter: Elevating the role of image understanding in visual question answering.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pages 6325–6334. IEEE Computer Society, 2017.  doi: 10.1109/CVPR.2017.670.  URL https://doi.org/10.1109/CVPR.2017.670.
  * Graham et al. (2020) Yvette Graham, Barry Haddow, and Philipp Koehn.  Statistical power and translationese in machine translation evaluation.  In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 72–81. Association for Computational Linguistics, 2020.  doi: 10.18653/v1/2020.emnlp-main.6.  URL https://doi.org/10.18653/v1/2020.emnlp-main.6.
  * Gritta and Iacobacci (2021) Milan Gritta and Ignacio Iacobacci.  Xeroalign: Zero-shot cross-lingual transformer alignment.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, _Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021_ , volume ACL/IJCNLP 2021 of _Findings of ACL_ , pages 371–381. Association for Computational Linguistics, 2021.  doi: 10.18653/v1/2021.findings-acl.32.  URL https://doi.org/10.18653/v1/2021.findings-acl.32.
  * Gritta et al. (2022) Milan Gritta, Ruoyu Hu, and Ignacio Iacobacci.  Crossaligner & co: Zero-shot transfer methods for task-oriented cross-lingual natural language understanding.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 4048–4061. Association for Computational Linguistics, 2022.  URL https://aclanthology.org/2022.findings-acl.319.
  * Grubinger et al. (2006) Michael Grubinger, Paul Clough, Henning Müller, and Thomas Deselaers.  The iapr tc-12 benchmark: A new evaluation resource for visual information systems.  In _International workshop ontoImage_ , volume 2, 2006.
  * Gu et al. (2018) Jiuxiang Gu, Shafiq R. Joty, Jianfei Cai, and Gang Wang.  Unpaired image captioning by language pivoting.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part I_ , volume 11205 of _Lecture Notes in Computer Science_ , pages 519–535. Springer, 2018.  doi: 10.1007/978-3-030-01246-5“˙31.  URL https://doi.org/10.1007/978-3-030-01246-5_31.
  * Gupta et al. (2018) Deepak Gupta, Pabitra Lenka, Asif Ekbal, and Pushpak Bhattacharyya.  Uncovering code-mixed challenges: A framework for linguistically driven question generation and neural based question answering.  In Anna Korhonen and Ivan Titov, editors, _Proceedings of the 22nd Conference on Computational Natural Language Learning, CoNLL 2018, Brussels, Belgium, October 31 - November 1, 2018_ , pages 119–130. Association for Computational Linguistics, 2018.  doi: 10.18653/v1/k18-1012.  URL https://doi.org/10.18653/v1/k18-1012.
  * Gupta et al. (2020) Deepak Gupta, Pabitra Lenka, Asif Ekbal, and Pushpak Bhattacharyya.  A unified framework for multilingual and code-mixed visual question answering.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 900–913, Suzhou, China, December 2020. Association for Computational Linguistics.  URL https://aclanthology.org/2020.aacl-main.90.
  * Gupta et al. (2021) Kshitij Gupta, Devansh Gautam, and Radhika Mamidi.  ViTA: Visual-linguistic translation by aligning object tags.  In _Proceedings of the 8th Workshop on Asian Translation (WAT2021)_ , pages 166–173, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.wat-1.19.  URL https://aclanthology.org/2021.wat-1.19.
  * Hedderich et al. (2021) Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow.  A survey on recent approaches for natural language processing in low-resource scenarios.  In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 2545–2568. Association for Computational Linguistics, 2021\.  doi: 10.18653/v1/2021.naacl-main.201.  URL https://doi.org/10.18653/v1/2021.naacl-main.201.
  * Hitschler et al. (2016) Julian Hitschler, Shigehiko Schamoni, and Stefan Riezler.  Multimodal pivots for image caption translation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics, 2016\.  doi: 10.18653/v1/p16-1227.  URL https://doi.org/10.18653/v1/p16-1227.
  * Huang et al. (2019a) Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou.  Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 2485–2494. Association for Computational Linguistics, 2019a.  doi: 10.18653/v1/D19-1252.  URL https://doi.org/10.18653/v1/D19-1252.
  * Huang et al. (2019b) Po-Yao Huang, Xiaojun Chang, and Alexander Hauptmann.  Multi-head attention with diversity for learning grounded multilingual multimodal representations.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1461–1467, Hong Kong, China, November 2019b. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1154.  URL https://aclanthology.org/D19-1154.
  * Huang et al. (2020) Po-Yao Huang, Junjie Hu, Xiaojun Chang, and Alexander Hauptmann.  Unsupervised multimodal neural machine translation with pseudo visual pivoting.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8226–8237, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.731.  URL https://aclanthology.org/2020.acl-main.731.
  * Huang et al. (2021) Po-Yao Huang, Mandela Patrick, Junjie Hu, Graham Neubig, Florian Metze, and Alexander Hauptmann.  Multilingual multimodal pre-training for zero-shot cross-lingual transfer of vision-language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2443–2459, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.195.  URL https://aclanthology.org/2021.naacl-main.195.
  * Imankulova et al. (2020) Aizhan Imankulova, Masahiro Kaneko, Tosho Hirasawa, and Mamoru Komachi.  Towards multimodal simultaneous neural machine translation.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 594–603, Online, November 2020. Association for Computational Linguistics.  URL https://aclanthology.org/2020.wmt-1.70.
  * Jones and Muftic (2020) Kerry Jones and Sanjin Muftic.  Endangered African languages featured in a digital collection: The case of the Khomani San, Hugh Brody Collection.  In _Proceedings of the first workshop on Resources for African Indigenous Languages_ , pages 1–8, Marseille, France, May 2020. European Language Resources Association (ELRA).  ISBN 979-10-95546-60-3.  URL https://aclanthology.org/2020.rail-1.1.
  * Kádár et al. (2018) Ákos Kádár, Desmond Elliott, Marc-Alexandre Côté, Grzegorz Chrupała, and Afra Alishahi.  Lessons learned in multilingual grounded language learning.  In _Proceedings of the 22nd Conference on Computational Natural Language Learning_ , pages 402–412, Brussels, Belgium, October 2018. Association for Computational Linguistics.  doi: 10.18653/v1/K18-1039.  URL https://aclanthology.org/K18-1039.
  * Karakanta et al. (2020) Alina Karakanta, Matteo Negri, and Marco Turchi.  MuST-cinema: a speech-to-subtitles corpus.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 3727–3734, Marseille, France, May 2020. European Language Resources Association.  ISBN 979-10-95546-34-4.  URL https://aclanthology.org/2020.lrec-1.460.
  * Karakanta et al. (2021) Alina Karakanta, Sara Papi, Matteo Negri, and Marco Turchi.  Simultaneous speech translation for live subtitling: from delay to display.  In _Proceedings of the 1st Workshop on Automatic Spoken Language Translation in Real-World Settings (ASLTRW)_ , pages 35–48, Virtual, August 2021\. Association for Machine Translation in the Americas.  URL https://aclanthology.org/2021.mtsummit-asltrw.4.
  * Kay et al. (2017) Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman.  The kinetics human action video dataset.  _CoRR_ , abs/1705.06950, 2017.  URL http://arxiv.org/abs/1705.06950.
  * Kiros et al. (2014) Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel.  Unifying visual-semantic embeddings with multimodal neural language models.  _CoRR_ , abs/1411.2539, 2014.  URL http://arxiv.org/abs/1411.2539.
  * Kitaev and Klein (2018) Nikita Kitaev and Dan Klein.  Constituency parsing with a self-attentive encoder.  In Iryna Gurevych and Yusuke Miyao, editors, _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , pages 2676–2686. Association for Computational Linguistics, 2018.  doi: 10.18653/v1/P18-1249.  URL https://aclanthology.org/P18-1249/.
  * Koehn et al. (2003) Philipp Koehn, Franz J Och, and Daniel Marcu.  Statistical phrase-based translation.  Technical report, University of Southern California Marina Del Rey Information Sciences Inst, 2003.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst.  Moses: Open source toolkit for statistical machine translation.  In John A. Carroll, Antal van den Bosch, and Annie Zaenen, editors, _ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic_. The Association for Computational Linguistics, 2007.  URL https://aclanthology.org/P07-2045/.
  * Koeva (2021) Svetla Koeva.  Multilingual image corpus: Annotation protocol.  In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)_ , pages 701–707, Held Online, September 2021. INCOMA Ltd.  URL https://aclanthology.org/2021.ranlp-1.80.
  * Koppel and Ordan (2011) Moshe Koppel and Noam Ordan.  Translationese and its dialects.  In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors, _The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA_ , pages 1318–1326. The Association for Computer Linguistics, 2011.  URL https://aclanthology.org/P11-1132/.
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _Int. J. Comput. Vis._ , 123(1):32–73, 2017.  doi: 10.1007/s11263-016-0981-7.  URL https://doi.org/10.1007/s11263-016-0981-7.
  * Krishnan et al. (2021) Jitin Krishnan, Antonios Anastasopoulos, Hemant Purohit, and Huzefa Rangwala.  Multilingual code-switching for zero-shot cross-lingual intent prediction and slot filling.  _CoRR_ , abs/2103.07792, 2021.  URL https://arxiv.org/abs/2103.07792.
  * Ku et al. (2020) Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge.  Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4392–4412, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.356.  URL https://aclanthology.org/2020.emnlp-main.356.
  * Lala and Specia (2018) Chiraag Lala and Lucia Specia.  Multimodal lexical translation.  In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Kôiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga, editors, _Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018_. European Language Resources Association (ELRA), 2018.  URL http://www.lrec-conf.org/proceedings/lrec2018/summaries/629.html.
  * Lala et al. (2018) Chiraag Lala, Pranava Swaroop Madhyastha, Carolina Scarton, and Lucia Specia.  Sheffield submissions for WMT18 multimodal translation shared task.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 624–631, Belgium, Brussels, October 2018. Association for Computational Linguistics.  doi: 10.18653/v1/W18-6442.  URL https://aclanthology.org/W18-6442.
  * Lan et al. (2017) Weiyu Lan, Xirong Li, and Jianfeng Dong.  Fluency-guided cross-lingual image captioning.  In Qiong Liu, Rainer Lienhart, Haohong Wang, Sheng-Wei ”Kuan-Ta” Chen, Susanne Boll, Yi-Ping Phoebe Chen, Gerald Friedland, Jia Li, and Shuicheng Yan, editors, _Proceedings of the 2017 ACM on Multimedia Conference, MM 2017, Mountain View, CA, USA, October 23-27, 2017_ , pages 1549–1557. ACM, 2017.  doi: 10.1145/3123266.3123366.  URL https://doi.org/10.1145/3123266.3123366.
  * Lauscher et al. (2020) Anne Lauscher, Vinit Ravishankar, Ivan Vulic, and Goran Glavas.  From zero to hero: On the limitations of zero-shot language transfer with multilingual transformers.  In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 4483–4499. Association for Computational Linguistics, 2020.  doi: 10.18653/v1/2020.emnlp-main.363.  URL https://doi.org/10.18653/v1/2020.emnlp-main.363.
  * Leviant and Reichart (2015) Ira Leviant and Roi Reichart.  Separated by an un-common language: Towards judgment language informed vector space modeling.  _arXiv preprint arXiv:1508.00106_ , 2015.
  * Li et al. (2020) Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 11336–11344. AAAI Press, 2020.  URL https://ojs.aaai.org/index.php/AAAI/article/view/6795.
  * Li et al. (2019a) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  _CoRR_ , abs/1908.03557, 2019a.  URL http://arxiv.org/abs/1908.03557.
  * Li et al. (2016) Xirong Li, Weiyu Lan, Jianfeng Dong, and Hailong Liu.  Adding chinese captions to images.  In John R. Kender, John R. Smith, Jiebo Luo, Susanne Boll, and Winston H. Hsu, editors, _Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval, ICMR 2016, New York, New York, USA, June 6-9, 2016_ , pages 271–275. ACM, 2016.  doi: 10.1145/2911996.2912049.  URL https://doi.org/10.1145/2911996.2912049.
  * Li et al. (2019b) Xirong Li, Chaoxi Xu, Xiaoxu Wang, Weiyu Lan, Zhengxiong Jia, Gang Yang, and Jieping Xu.  COCO-CN for cross-lingual image tagging, captioning, and retrieval.  _IEEE Trans. Multim._ , 21(9):2347–2360, 2019b.  doi: 10.1109/TMM.2019.2896494.  URL https://doi.org/10.1109/TMM.2019.2896494.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, _Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V_ , volume 8693 of _Lecture Notes in Computer Science_ , pages 740–755. Springer, 2014.  doi: 10.1007/978-3-319-10602-1“˙48.  URL https://doi.org/10.1007/978-3-319-10602-1_48.
  * Liu et al. (2021) Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott.  Visually grounded reasoning across languages and cultures.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10467–10485, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.818.  URL https://aclanthology.org/2021.emnlp-main.818.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer.  Multilingual denoising pre-training for neural machine translation.  _Trans. Assoc. Comput. Linguistics_ , 8:726–742, 2020.  URL https://transacl.org/ojs/index.php/tacl/article/view/2107.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 13–23, 2019.  URL https://proceedings.neurips.cc/paper/2019/hash/c74d97b01eae257e44aa9d5bade97baf-Abstract.html.
  * Ma et al. (2019) Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang.  STACL: simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework.  In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 3025–3036. Association for Computational Linguistics, 2019.  doi: 10.18653/v1/p19-1289.  URL https://doi.org/10.18653/v1/p19-1289.
  * Mitzalis et al. (2021) Faidon Mitzalis, Ozan Caglayan, Pranava Madhyastha, and Lucia Specia.  BERTGen: Multi-task generation through BERT.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6440–6455, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.503.  URL https://aclanthology.org/2021.acl-long.503.
  * Mohammadshahi et al. (2019) Alireza Mohammadshahi, Rémi Lebret, and Karl Aberer.  Aligning multilingual word embeddings for cross-modal retrieval task.  In _Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER)_ , pages 27–33, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-6605.  URL https://aclanthology.org/D19-6605.
  * Moneglia and Varvara (2020) Massimo Moneglia and Rossella Varvara.  The annotation of thematic structure and alternations face to the semantic variation of action verbs. current trends in the IMAGACT ontology.  In _16th Joint ACL - ISO Workshop on Interoperable Semantic Annotation PROCEEDINGS_ , pages 67–74, Marseille, May 2020. European Language Resources Association.  ISBN 979-10-95546-48-1.  URL https://aclanthology.org/2020.isa-1.8.
  * Moneglia et al. (2014a) Massimo Moneglia, Susan Brown, Francesca Frontini, Gloria Gagliardi, Anas Fahad Khan, Monica Monachini, and Alessandro Panunzi.  The IMAGACT visual ontology. an extendable multilingual infrastructure for the representation of lexical encoding of action.  In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asunción Moreno, Jan Odijk, and Stelios Piperidis, editors, _Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 2014_ , pages 3425–3432. European Language Resources Association (ELRA), 2014a.  URL http://www.lrec-conf.org/proceedings/lrec2014/summaries/318.html.
  * Moneglia et al. (2014b) Massimo Moneglia, Susan Brown, Francesca Frontini, Gloria Gagliardi, Fahad Khan, Monica Monachini, and Alessandro Panunzi.  The IMAGACT visual ontology. an extendable multilingual infrastructure for the representation of lexical encoding of action.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 3425–3432, Reykjavik, Iceland, May 2014b. European Language Resources Association (ELRA).  URL http://www.lrec-conf.org/proceedings/lrec2014/pdf/318_Paper.pdf.
  * Myers-Scotton (1997) Carol Myers-Scotton.  _Duelling languages: Grammatical structure in codeswitching_.  Oxford University Press, 1997.
  * Nakayama et al. (2020a) Hideki Nakayama, Akihiro Tamura, and Takashi Ninomiya.  A visually-grounded parallel corpus with phrase-to-region linking.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 4204–4210, Marseille, France, May 2020a. European Language Resources Association.  ISBN 979-10-95546-34-4.  URL https://aclanthology.org/2020.lrec-1.518.
  * Nakayama et al. (2020b) Hideki Nakayama, Akihiro Tamura, and Takashi Ninomiya.  A visually-grounded parallel corpus with phrase-to-region linking.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 4204–4210, 2020b.
  * Ni et al. (2021) Minheng Ni, Haoyang Huang, Lin Su, Edward Cui, Taroon Bharti, Lijuan Wang, Dongdong Zhang, and Nan Duan.  M3P: learning universal representations via multitask multilingual multimodal pre-training.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_ , pages 3977–3986. Computer Vision Foundation / IEEE, 2021.  URL https://openaccess.thecvf.com/content/CVPR2021/html/Ni_M3P_Learning_Universal_Representations_via_Multitask_Multilingual_Multimodal_Pre-Training_CVPR_2021_paper.html.
  * Nishihara et al. (2020) Tetsuro Nishihara, Akihiro Tamura, Takashi Ninomiya, Yutaro Omote, and Hideki Nakayama.  Supervised visual attention for multimodal neural machine translation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4304–4314, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.  doi: 10.18653/v1/2020.coling-main.380.  URL https://aclanthology.org/2020.coling-main.380.
  * Parida and Bojar (2018) Shantipriya Parida and Ondřej Bojar.  Translating short segments with nmt: A case study in english-to-hindi.  2018\.
  * Parida et al. (2020) Shantipriya Parida, Petr Motlicek, Amulya Ratna Dash, Satya Ranjan Dash, Debasish Kumar Mallick, Satya Prakash Biswal, Priyanka Pattnaik, Biranchi Narayan Nayak, and Ondřej Bojar.  ODIANLP’s participation in WAT2020.  In _Proceedings of the 7th Workshop on Asian Translation_ , pages 103–108, Suzhou, China, December 2020. Association for Computational Linguistics.  URL https://aclanthology.org/2020.wat-1.10.
  * Parida et al. (2021) Shantipriya Parida, Subhadarshi Panda, Ketan Kotwal, Amulya Ratna Dash, Satya Ranjan Dash, Yashvardhan Sharma, Petr Motlicek, and Ondřej Bojar.  NLPHut’s participation at WAT2021.  In _Proceedings of the 8th Workshop on Asian Translation (WAT2021)_ , pages 146–154, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.wat-1.16.  URL https://aclanthology.org/2021.wat-1.16.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder.  MAD-X: an adapter-based framework for multi-task cross-lingual transfer.  In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 7654–7673. Association for Computational Linguistics, 2020.  doi: 10.18653/v1/2020.emnlp-main.617.  URL https://doi.org/10.18653/v1/2020.emnlp-main.617.
  * Pfeiffer et al. (2022) Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin O. Steitz, Stefan Roth, Ivan Vulic, and Iryna Gurevych.  xgqa: Cross-lingual visual question answering.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 2497–2511. Association for Computational Linguistics, 2022.  doi: 10.18653/v1/2022.findings-acl.196.  URL https://doi.org/10.18653/v1/2022.findings-acl.196.
  * Plummer et al. (2015) Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2641–2649, 2015.
  * Poignant et al. (2016) Johann Poignant, Mateusz Budnik, Hervé Bredin, Claude Barras, Mickael Stefas, Pierrick Bruneau, Gilles Adda, Laurent Besacier, Hazim Ekenel, Gil Francopoulo, Javier Hernando, Joseph Mariani, Ramon Morros, Georges Quénot, Sophie Rosset, and Thomas Tamisier.  The CAMOMILE collaborative annotation platform for multi-modal, multi-lingual and multi-media documents.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 1421–1425, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA).  URL https://aclanthology.org/L16-1226.
  * Qin et al. (2020) Libo Qin, Minheng Ni, Yue Zhang, and Wanxiang Che.  Cosda-ml: Multi-lingual code-switching data augmentation for zero-shot cross-lingual NLP.  In Christian Bessiere, editor, _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020_ , pages 3853–3860. ijcai.org, 2020.  doi: 10.24963/ijcai.2020/533.  URL https://doi.org/10.24963/ijcai.2020/533.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR, 2021.  URL http://proceedings.mlr.press/v139/radford21a.html.
  * Raj Khan et al. (2021) Humair Raj Khan, Deepak Gupta, and Asif Ekbal.  Towards developing a multilingual and code-mixed visual question answering system by knowledge distillation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1753–1767, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.findings-emnlp.151.  URL https://aclanthology.org/2021.findings-emnlp.151.
  * Rajendran et al. (2016) Janarthanan Rajendran, Mitesh M. Khapra, Sarath Chandar, and Balaraman Ravindran.  Bridge correlational neural networks for multilingual multimodal representation learning.  In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, _NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016_ , pages 171–181. The Association for Computational Linguistics, 2016.  doi: 10.18653/v1/n16-1021.  URL https://doi.org/10.18653/v1/n16-1021.
  * Ramnath and Hasegawa-Johnson (2020) Kiran Ramnath and Mark Hasegawa-Johnson.  Seeing is knowing! fact-based visual question answering using knowledge graph embeddings.  _CoRR_ , abs/2012.15484, 2020.  URL https://arxiv.org/abs/2012.15484.
  * Ramnath et al. (2021) Kiran Ramnath, Leda Sari, Mark Hasegawa-Johnson, and Chang Yoo.  Worldly wise (WoW) - cross-lingual knowledge fusion for fact-based visual spoken-question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1908–1919, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.153.  URL https://aclanthology.org/2021.naacl-main.153.
  * Ramponi and Plank (2020) Alan Ramponi and Barbara Plank.  Neural unsupervised domain adaptation in NLP - A survey.  In Donia Scott, Núria Bel, and Chengqing Zong, editors, _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_ , pages 6838–6855. International Committee on Computational Linguistics, 2020.  doi: 10.18653/v1/2020.coling-main.603.  URL https://doi.org/10.18653/v1/2020.coling-main.603.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 3980–3990. Association for Computational Linguistics, 2019.  doi: 10.18653/v1/D19-1410.  URL https://doi.org/10.18653/v1/D19-1410.
  * Reiter and Dale (1997) Ehud Reiter and Robert Dale.  Building applied natural language generation systems.  _Nat. Lang. Eng._ , 3(1):57–87, 1997.  doi: 10.1017/S1351324997001502.  URL https://doi.org/10.1017/S1351324997001502.
  * Rinsche (2005) Adriane Rinsche.  Computer-assisted multingual E-communication in a variety of application areas.  In _Proceedings of Machine Translation Summit X: Posters_ , pages 458–464, Phuket, Thailand, September 13-15 2005.  URL https://aclanthology.org/2005.mtsummit-posters.20.
  * Rotman et al. (2018) Guy Rotman, Ivan Vulić, and Roi Reichart.  Bridging languages through images with deep partial canonical correlation analysis.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 910–921, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-1084.  URL https://aclanthology.org/P18-1084.
  * Route et al. (2019) James Route, Steven Hillis, Isak Czeresnia Etinger, Han Zhang, and Alan W Black.  Multimodal, multilingual grapheme-to-phoneme conversion for low-resource languages.  In _Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)_ , pages 192–201, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-6121.  URL https://aclanthology.org/D19-6121.
  * Rust et al. (2021) Phillip Rust, Jonas Pfeiffer, Ivan Vulic, Sebastian Ruder, and Iryna Gurevych.  How good is your tokenizer? on the monolingual performance of multilingual language models.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 3118–3135. Association for Computational Linguistics, 2021.  doi: 10.18653/v1/2021.acl-long.243.  URL https://doi.org/10.18653/v1/2021.acl-long.243.
  * Sanabria et al. (2018) Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loïc Barrault, Lucia Specia, and Florian Metze.  How2: A large-scale dataset for multimodal language understanding.  _CoRR_ , abs/1811.00347, 2018.  URL http://arxiv.org/abs/1811.00347.
  * Shi et al. (2019) Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu.  Visually grounded neural syntax acquisition.  In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 1842–1861. Association for Computational Linguistics, 2019.  doi: 10.18653/v1/p19-1180.  URL https://doi.org/10.18653/v1/p19-1180.
  * Shimizu et al. (2018) Nobuyuki Shimizu, Na Rong, and Takashi Miyazaki.  Visual question answering dataset for bilingual image understanding: A study of cross-lingual transfer using attention maps.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1918–1928, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics.  URL https://aclanthology.org/C18-1163.
  * Singh et al. (2021) Salam Michael Singh, Loitongbam Sanayai Meetei, Thoudam Doren Singh, and Sivaji Bandyopadhyay.  Multiple captions embellished multilingual multi-modal neural machine translation.  In _Proceedings of the First Workshop on Multimodal Machine Translation for Low Resource Languages (MMTLRL 2021)_ , pages 2–11, Online (Virtual Mode), September 2021. INCOMA Ltd.  URL https://aclanthology.org/2021.mmtlrl-1.2.
  * Su et al. (2020) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  VL-BERT: pre-training of generic visual-linguistic representations.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.  URL https://openreview.net/forum?id=SygXPaEYvH.
  * Surís et al. (2022) Dídac Surís, Dave Epstein, and Carl Vondrick.  Globetrotter: Connecting languages by connecting images.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_ , pages 16453–16463. IEEE, 2022.  doi: 10.1109/CVPR52688.2022.01598.  URL https://doi.org/10.1109/CVPR52688.2022.01598.
  * Susanto et al. (2021) Raymond Hendy Susanto, Dongzhe Wang, Sunil Yadav, Mausam Jain, and Ohnmar Htun.  Rakuten’s participation in WAT 2021: Examining the effectiveness of pre-trained models for multilingual and multimodal machine translation.  In _Proceedings of the 8th Workshop on Asian Translation (WAT2021)_ , pages 96–105, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.wat-1.9.  URL https://aclanthology.org/2021.wat-1.9.
  * Trmal et al. (2017) Jan Trmal, Matthew Wiesner, Vijayaditya Peddinti, Xiaohui Zhang, Pegah Ghahremani, Yiming Wang, Vimal Manohar, Hainan Xu, Daniel Povey, and Sanjeev Khudanpur.  The kaldi openkws system: Improving low resource keyword search.  In Francisco Lacerda, editor, _Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017_ , pages 3597–3601. ISCA, 2017.  URL http://www.isca-speech.org/archive/Interspeech_2017/abstracts/0601.html.
  * Tsai et al. (2019) Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov.  Multimodal transformer for unaligned multimodal language sequences.  In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 6558–6569. Association for Computational Linguistics, 2019.  doi: 10.18653/v1/p19-1656.  URL https://doi.org/10.18653/v1/p19-1656.
  * Vendrov et al. (2016) Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun.  Order-embeddings of images and language.  In Yoshua Bengio and Yann LeCun, editors, _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ , 2016.  URL http://arxiv.org/abs/1511.06361.
  * Vilares et al. (2020) Jesús Vilares, Carlos Gómez-Rodríguez, Luís Fernández-Núñez, Darío Penas, and Jorge Viteri.  Bringing roguelikes to visually-impaired players by using NLP.  In _Workshop on Games and Natural Language Processing_ , pages 59–67, Marseille, France, May 2020. European Language Resources Association.  ISBN 979-10-95546-40-5.  URL https://aclanthology.org/2020.gamnlp-1.9.
  * Wang et al. (2019a) Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang.  Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 6629–6638. Computer Vision Foundation / IEEE, 2019a.  doi: 10.1109/CVPR.2019.00679.  URL http://openaccess.thecvf.com/content_CVPR_2019/html/Wang_Reinforced_Cross-Modal_Matching_and_Self-Supervised_Imitation_Learning_for_Vision-Language_Navigation_CVPR_2019_paper.html.
  * Wang et al. (2019b) Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang.  Vatex: A large-scale, high-quality multilingual dataset for video-and-language research.  In _2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019_ , pages 4580–4590. IEEE, 2019b.  doi: 10.1109/ICCV.2019.00468.  URL https://doi.org/10.1109/ICCV.2019.00468.
  * Weiss et al. (2017) Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen.  Sequence-to-sequence models can directly translate foreign speech.  In Francisco Lacerda, editor, _Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017_ , pages 2625–2629. ISCA, 2017.  URL http://www.isca-speech.org/archive/Interspeech_2017/abstracts/0503.html.
  * Willemsen et al. (2018) Bram Willemsen, Jan de Wit, Emiel Krahmer, Mirjam de Haas, and Paul Vogt.  Context-sensitive natural language generation for robot-assisted second language tutoring.  In _Proceedings of the Workshop on NLG for Human–Robot Interaction_ , pages 1–7, Tilburg, The Netherlands, November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/W18-6901.  URL https://aclanthology.org/W18-6901.
  * Xu et al. (2020a) Runxin Xu, Jun Cao, Mingxuan Wang, Jiaze Chen, Hao Zhou, Ying Zeng, Yuping Wang, Li Chen, Xiang Yin, Xijin Zhang, Songcheng Jiang, Yuxuan Wang, and Lei Li.  Xiaomingbot: A Multilingual Robot News Reporter.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 1–8, Online, July 2020a. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-demos.1.  URL https://aclanthology.org/2020.acl-demos.1.
  * Xu et al. (2020b) Ruochen Xu, Chenguang Zhu, Yu Shi, Michael Zeng, and Xuedong Huang.  Mixed-lingual pre-training for cross-lingual summarization.  In Kam-Fai Wong, Kevin Knight, and Hua Wu, editors, _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, AACL/IJCNLP 2020, Suzhou, China, December 4-7, 2020_ , pages 536–541. Association for Computational Linguistics, 2020b.  URL https://aclanthology.org/2020.aacl-main.53/.
  * Yoshikawa et al. (2017) Yuya Yoshikawa, Yutaro Shigeto, and Akikazu Takeuchi.  STAIR captions: Constructing a large-scale japanese image caption dataset.  In Regina Barzilay and Min-Yen Kan, editors, _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 2: Short Papers_ , pages 417–421. Association for Computational Linguistics, 2017.  doi: 10.18653/v1/P17-2066.  URL https://doi.org/10.18653/v1/P17-2066.
  * Zarrieß et al. (2016) Sina Zarrieß, Julian Hough, Casey Kennington, Ramesh Manuvinakurike, David DeVault, Raquel Fernández, and David Schlangen.  PentoRef: A corpus of spoken references in task-oriented dialogues.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 125–131, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA).  URL https://aclanthology.org/L16-1019.
  * Zhang et al. (2017) Ted Zhang, Dengxin Dai, Tinne Tuytelaars, Marie-Francine Moens, and Luc Van Gool.  Speech-based visual question answering.  _CoRR_ , abs/1705.00464, 2017.  URL http://arxiv.org/abs/1705.00464.
  * Zhang et al. (2020) Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, Zuchao Li, and Hai Zhao.  Neural machine translation with universal visual representation.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.  URL https://openreview.net/forum?id=Byl8hhNYPS.
  * Zhou et al. (2018) Mingyang Zhou, Runxiang Cheng, Yong Jae Lee, and Zhou Yu.  A visual attention grounding neural model for multimodal machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3643–3653, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1400.  URL https://aclanthology.org/D18-1400.
  * Zhou et al. (2021) Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, and Jingjing Liu.  UC2: universal cross-lingual cross-modal vision-and-language pre-training.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_ , pages 4155–4165. Computer Vision Foundation / IEEE, 2021.  doi: 10.1109/CVPR46437.2021.00414.  URL https://openaccess.thecvf.com/content/CVPR2021/html/Zhou_UC2_Universal_Cross-Lingual_Cross-Modal_Vision-and-Language_Pre-Training_CVPR_2021_paper.html.
  * Zhu et al. (2019) Junnan Zhu, Qian Wang, Yining Wang, Yu Zhou, Jiajun Zhang, Shaonan Wang, and Chengqing Zong.  NCLS: neural cross-lingual summarization.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 3052–3062. Association for Computational Linguistics, 2019.  doi: 10.18653/v1/D19-1302.  URL https://doi.org/10.18653/v1/D19-1302.