# Meta-Learning For Vision-and-Language Cross-lingual Transfer

Hanxu Hu
University of Edinburgh
School of Informatics
<EMAIL_ADDRESS>
Frank Keller
University of Edinburgh
School of Informatics
<EMAIL_ADDRESS>

###### Abstract

Current pre-trained vison-language models (PVLMs) achieve excellent
performance on a range of multi-modal datasets. Recent work has aimed at
building multilingual models, and a range of novel multilingual multi-modal
datasets have been proposed. Current PVLMs typically perform poorly on these
datasets when used for multi-modal zero-shot or few-shot cross-lingual
transfer, especially for low-resource languages. To alleviate this problem, we
propose a novel meta-learning fine-tuning framework. Our framework makes
current PVLMs rapidly adaptive to new languages in vision-language scenarios
by designing MAML in a cross-lingual multi-modal manner. Experiments show that
our method boosts the performance of current state-of-the-art PVLMs in both
zero-shot and few-shot cross-lingual transfer on a range of vision-language
understanding tasks and datasets (XVNLI, xGQA, MaRVL, xFlicker&Co).

##  1 Introduction

Multi-modal models focus on jointly learning representations from multiple
modalities, such as vision and language. Many task require the integration
information of vision and language, including image captioning Vinyals et al.
(2015), natural language visual reasoning Zhou et al. (2017); Suhr et al.
(2019), and cross-modal retrieval Zhen et al. (2019). Multi-modal learning
models the interaction between different modalities, allowing the resulting
representations to be used in various multimedia applications to enhance
human-computer interaction.

Figure 1: Examples in IGLUEBugliarello et al. (2022) benchmark. The left
example comes from MaRVL Liu et al. (2021) dataset, and the right example
comes from XVNLI dataset proposed in IGLUE.

Recently, pre-trained vision-language models (PVLMs) Chen et al. (2020); Lu et
al. (2019); Tan and Bansal (2019) have achieved significant advances in multi-
modal tasks. However, the data which PVLMs learn from is mostly for high-
resource languages such as English. The resulting models reply on large
amounts of training data for good performance and often the models acquire
biases that mean they perform poorly for low-resource languages such as
Indonesian and Swahili. To address this, several multilingual PVLMs Zhou et
al. (2021); Ni et al. (2021) have been proposed. A number of studies have used
multilingual multi-modal datasets Bugliarello et al. (2022); Liu et al.
(2021), and Figure 1 shows two examples of these datasets. They have evaluated
PVLMs and have demonstrated that these models do not perform well in low-
resource cross-lingual transfer settings.

Meta-learning can mitigate this issue. It is a learning approach that enables
machine learning models to adapt quickly to new tasks by learning the learning
algorithm itself. Model-Agnostic Meta-Learning (MAML; Finn et al. 2017) is one
of the most widely used meta-learning frameworks. It is based on gradient-
descent optimization, does not require multiple models or complex settings,
and can be used for a range of models. In previous work Verma et al. (2020);
Finn et al. (2017); Nooralahzadeh et al. (2020), MAML-based methods have been
shown to be useful in low-resource and cross-lingual transfer scenarios,
including both few-shot and zero-shot cross-lingual tasks. However, prior work
has only attempted to use MAML for text-only tasks for cross-lingual
transferNooralahzadeh et al. (2020). Inspired by previous works about using
MAML for natural language tasks, this paper is focused on using MAML to
address the limitations of previous PVLMs in Vision-Language tasks for low-
resource cross-lingual transfer. We propose a meta-learning framework which is
specialize for vision-language cross-lingual transfer tasks. In this
framework, we proposed a novel algorithm which combines a traditional
supervised loss function for learning down-stream tasks and a contrastive loss
for learning alignments between modalities together in a cross-lingual MAML
optimization procedure and name it as XVL-MAML. We show that this can lead to
significant improvement in PVLMs performance for low resource target
languages. We also find that using Contrastive Learning in MAML framework on
its own can bring improvements in PVLM performance in unsupervised settings.

Specifically, we uses a contrastive learning loss as the objective function in
the MAML algorithm, based on the assumption that it allows the model to
generalize the matching of text and images in new languages. Because multi-
modal datasets consist of image-text pairs describing the same or similar
objects, labels for downstream tasks are not needed. Exploiting this insight,
we take an image-text pair from the original dataset as a positive sample and
a randomly paired text and image as a negative sample.

In sum, our contributions are as follows. 1) We propose a novel MAML framework
called XVL-MAML which is specialized for vision-and-language cross-lingual
transfers, as it combine contrastive learning and standard supervised learning
in the MAML algorithm. 2) We shows that using contrastive learning solely in
MAML framework as an un-supervsied setting can also be useful. 3) We
demonstrated our proposed framework can boost the performance of current PVLMs
across 14 languages and 4 tasks in both zero-shot learning and few-shot
learning. 4) We conducted ablation study to verify the effect of contrastive
learning in both supervised and un-supervised settings and made further
analysis across langauges and tasks.

##  2 Related Work

###  2.1 Multilingual Vision-and-Language Methods and Tasks

Recent work has investigated Vision-and-Language cross-lingual transfer tasks.
Elliott et al. (2016) proposed Multi30K, an image description dataset which
contains descriptions in multiple languagess. Previous methods Gella et al.
(2017); Rotman et al. (2018) focus on bridging languages through images, but
they mainly focus on image-text retrieval and only consider high resource
languages such as English and German. Pfeiffer et al. (2022) built a
multilingual visual question answering dataset called xGQA. Liu et al. (2021)
proposed a multilingual version of the grounded visual reasoning dataset
called MaRVL, which follow the same setting as natural language visual
reasoning dataset NLVR2 Su et al. (2019), but considers both cross-lingual
transfer and domain shift between languages. Qiu et al. (2022) uses machine
translation to help multilingual multimodal learning.

Several pre-trained models are recently proposed for vision-and-language
cross-lingual transfer. Ni et al. (2021) proposed M3P, a transformer-based
pre-trained model that maps the same concepts in different modalities and
languages into a common semantic space. Similar to M3P, Liu et al. (2021)
extended UNITER Chen et al. (2020), proposing mUNITER based on M-BERT Devlin
et al. (2019), and xUNITER based on XLM-R Conneau et al. (2020). Zhou et al.
(2021) proposed UC2, a model using a data augmentation method based on machine
translation for cross-lingual cross-modal pre-training. Although pre-training
methods have proven powerful across multiple tasks, they require large amounts
of training data and show clear performance gap between English and other low-
resource languages on the IGLUE benchmark Bugliarello et al. (2022).

###  2.2 Meta-Learning

Meta-learning has been increasingly popular in the machine learning community.
Whereas conventional machine learning methods learn by data points, meta-
learning learns by tasks. Previous meta-learning work Vinyals et al. (2016);
Finn et al. (2017) focused on adapting to new tasks quickly. But meta-learning
can be applied to other scenarios as well, including semi-supervised learning
Ren et al. (2018), multi-task learning Yu et al. (2020), and domain
generalization Li et al. (2018).

Prior work has also explored the effectiveness of meta-learning in NLP: Wang
et al. (2021) applied meta-learning in semantic parsing for domain
generalization based on MAML Finn et al. (2017); Li et al. (2018). Obamuyide
and Vlachos (2019) leveraged meta-learning under limited supervision in a
relation classification task. Recently, there have been some applications
using MAML in cross-lingual transfer: Gu et al. (2018) and Nooralahzadeh et
al. (2020) regard languages as tasks in their meta-learning framework. In
contrast to these existing approaches, which only explore text-only scenarios,
we are the first to utilize meta-learning for cross-lingual transfer in multi-
modal tasks.

##  3 Meta-learning for Vision-and-Language Cross-lingual Transfer

We first formally define the problem of Vision-and-Language Cross-lingual
Transfer in the context of zero-shot and few-shot scenarios in Section 3.1.
Then, we introduce our overall fine-tuning framework in Section 3.2. And we
introduce the contrastive learning used in vision-and-language tasks in
Section 3.3. Finally, we introduce our proposed XVL-MAML algorithm in Section
3.4.

###  3.1 Problem Definition

Following the multilingual vision-language IGLUE benchmark Bugliarello et al.
(2022), we formulate the problem of cross-lingual transfer learning in Vision-
and-Language scenarios. For understanding tasks, the input is a pair of an
image $V$ and text $U$ and the output $Y$ is the result inferred by the multi-
modal model. We can thus formulate this problem as computing
$P_{\theta}(Y|V,U)$, where $\theta$ are the parameters of the PVLMs. During
training, the image-text pairs comes from datasets $D_{s}$ in a set of known
languages, and our aim is to perform well on the datasets $D_{t}$ with the
same task in unseen low-resource languages. For zero-shot learning, the model
fine-tuned or pre-trained on $D_{s}$ is directly used in inference on $D_{t}$
in unseen low-resource languages. For few-shot learning, after trained on
$D_{s}$, the model is continually fine-tuned on several shots of $D_{t}$ in
target languages.

###  3.2 Overall Fine-tuning Framework For Cross-lingual Transfer

Our pipeline of the proposed meta-learning fine-tuning framework can be
divided into three parts:

  1. 1.

Fine-tune the pre-trained vision-language models on data of down-stream task
in English

  2. 2.

Fine-tune the models on data in the auxiliary language (one language other
than English) using our proposed XVL-MAML algorithm.

  3. 3.

Evaluate the fine-tuned models on data in the target languages (languages
other than English and the auxiliary language)

In traditional cross-lingual transfer learning procedure described in
Bugliarello et al. (2022), only part 1 and part 3 should be conducted. In part
3, if the setting is zero-shot, the model should be evaluated on data in
target language directly, but if the setting is few-shot, the model should
continue fine-tuning on few-shots of data in target languages then conduct
evaluation. The difference between our framework and traditional procedure is
that one additional fine-tuning step (part 2) will be conducted. We will
describe it specifically in Section 3.4, but before that, we will firstly
introduce the contrastive learning in Vision-and-Language tasks.

Figure 2: The proposed architecture of the model. The architecture consists of
a contrastive learning module and a downstream tasks module, and both modules
share the same parameters of a pre-trained vision-language model.

###  3.3 Contrastive Learning in Vision-and-Language tasks

Vision-and-Language Contrastive Learning loss proposed by Zhang et al. (2020)
has been proven effective in medical image scenarios and is used as the pre-
training objective function of CLIP Radford et al. (2021). It can be regarded
as an auxiliary task for representation learning, aiming to enable models gain
better aligned multi-modal representation for downstream tasks. In the
contrastive learning scheme, a batch of embeddings of images encoded by the
model can be written as $I=\\{I_{1},...,I_{N}\\}$, and a batch of embeddings
of texts encoded by the model can be written as $T=\\{T_{1},...,T_{N}\\}$,
where $N$ is the size of batch, and $(I_{i},T_{i})$ an image-text pair, and
the paired image-text data describe the same or similar concepts, so we can
assume they are positive examples, and non-paired data are negative examples.
Then, the embeddings of images and texts are fed into two different linear
transformation layers separately, which are noted as $W_{1}$ and $W_{2}$:

$U=I\cdot W_{1}^{\top}$ (1) $V=T\cdot W_{2}^{\top}$ (2)

Where $U$ and $V$ represent the batch of image-text pairs. Then the cosine
similarity of each pairs can be computed as as $\langle
U_{i},V_{j}\rangle=\frac{U^{\top}_{i}V_{j}}{\left\|U_{i}\right\|\left\|V_{j}\right\|}$.
The objective is to maximize the similarity of matched image-text pairs and
minimize others. So the image-text contrastive loss can be formulated as :

$\mathcal{L}^{1}_{i}=-\log\frac{\exp(\langle
U_{i},V_{i}\rangle)}{\sum^{N}_{K=1}\exp(\langle U_{i},V_{k}\rangle)}$ (3)

Following Zhang et al. (2020), the contrastive loss should be symmetric for
each modality, and the text-image contrastive loss as:

$\mathcal{L}^{2}_{i}=-\log\frac{\exp(\langle
V_{i},U_{i}\rangle)}{\sum^{N}_{K=1}\exp(\langle V_{i},U_{k}\rangle)}$ (4)

Finally, the final contrastive loss of this batch of paired data is:

$\mathcal{L}_{CL}=\sum_{i=1}^{N}(\mathcal{L}^{1}_{i}+\mathcal{L}^{2}_{i})$ (5)

Where $\mathcal{L}_{CL}$ is the overall contrastive loss. When we minimize
$\mathcal{L}_{CL}$, we actually maximize the similarity of image-text pairs
which are positive examples.

###  3.4 XVL-MAML

Inspired by the effectiveness of MAML for quickly adapting to new tasks, we
propose a novel MAML algorithm specialized for cross-lingual transfer in
vision and language tasks, called XVL-MAML. Specifically, we first integrate
traditional contrastive learning into the MAML algorithm, making it
specialized for the visual-language task of cross-lingual transfer learning.
Our intuition is that we can use MAML with a contrastive loss as its learning
objective for quickly adapting vision-language alignment to new languages. In
this framework, the alignment between image and text in a specific language
can be regard as a task. Inspired by Nooralahzadeh et al. (2020), we use data
of one auxiliary language for fine-tuning, but with contrastive loss as
objective function in the MAML algorithm.

Specifically, we sample a batch of support data $\mathcal{B}_{s}$ and a batch
of query data $\mathcal{B}_{q}$ in the data in auxiliary language $A$ for each
virtual task $\mathcal{T}$. Assuming the parameters of the model are $\theta$
and the contrastive loss on the support data is
$L_{CL}(\theta)_{\mathcal{B}_{s}}$, then the parameters of the model can be
updated by one step of gradient descent:

$\theta^{{}^{\prime}}=\theta-\alpha\nabla_{\theta}\mathcal{L}_{CL}(\theta)_{\mathcal{B}_{s}}$
(6)

Following the MAML algorithm, our final objective for this task is to minimize
$\mathcal{L}_{CL}(\theta^{{}^{\prime}})_{\mathcal{B}_{q}}$ on the query data
$\mathcal{B}_{q}$ using gradient descent:

$\theta\leftarrow\theta-\beta\nabla_{\theta}\mathcal{L}_{CL}(\theta^{{}^{\prime}})_{\mathcal{B}_{q}}$
(7)
$\theta\leftarrow\theta-\beta\nabla_{\theta}\mathcal{L}_{CL}(\theta-\alpha\nabla_{\theta}\mathcal{L}_{CL}(\theta)_{\mathcal{B}_{s}})_{\mathcal{B}_{q}}$
(8)

Optimized using this method, pre-trained vision-language models can quickly
adapt to new tasks in other languages without using any annotation in
auxiliary for downstream tasks, so we called it as un-supervised scenarios.

In supervised scenarios, where the downstream tasks labels in auxiliary
language is available, we combine the loss of downstream task $\mathcal{L}$
with vision-langauge contrastive loss $\mathcal{L}_{CL}$ by adding them
together. So during fine-tuning, Equation (8) is modified as:

$\theta\leftarrow\theta-\beta(\nabla_{\theta}\mathcal{L}(\theta^{{}^{\prime\prime}})_{\mathcal{B}_{q}}+\lambda\nabla_{\theta}\mathcal{L}_{CL}(\theta^{{}^{\prime}})_{\mathcal{B}_{q}})$
(9)

Where the temporary parameters optimized for one step by downstream task loss
$\mathcal{L}$ on the support set $\mathcal{B}_{s}$ is
$\theta^{{}^{\prime\prime}}$, $\beta$ is the meta-learning rate, and $\lambda$
the scale factor of contrastive learning. By simply adding the gradients of
downstream task and contrastive learning in the meta-update, the model learns
downstream tasks and vision-language alignment simultaneously for cross-
lingual transfer.

|  |  |  |  | xFlickr&Co
---|---|---|---|---|---
Method | Model | XNVLI | xGQA | MaRVL | IR | TR
| mUNITER | 53.7 | 10.0 | 53.7 | 8.1 | 8.9
| xUNITER | 59.0 | 20.8 | 56.0 | 13.8 | 12.5
| UC2 | 62.5 | 29.0 | 56.4 | 19.7 | 17.0
Baseline | M3P | 58.2 | 28.2 | 56.0 | 12.9 | 11.9
| xUNITER | 63.0 (+4.0) | 22.5 (+1.7) | 59.4 (+4.4) | 16.3 (+2.5) | 14.2 (+1.7)
Ours | UC2 | 64.4 (+1.9) | 29.9 (+0.9) | 57.0 (+0.6) | 21.3 (+1.6) | 18.7 (+1.7)
Table 1: Zero-shot performance (accuracy) of four baseline models only fine-
tuned by English data (Baseline) and two models fine-tuned by our meta-
learning method (Ours) on four IGLUE datasets Bugliarello et al. (2022).

##  4 Experiments

In this section, we introduce the base PVLMs we use for out multi-modal cross-
lingual transfer tasks, and we also introduce the datasets and metrics we use
to evaluate our proposed method. Then we describe how the experiments were
conducted and discuss the results.

###  4.1 Base models

In this paper, we choose xUNITER Liu et al. (2021) and UC2 Zhou et al. (2021)
as our baseline models, as these two models use different pre-training
methods. Then we applied our method to both models to show that our proposed
method is model-independent.

#### xUNITER

is a multilingual version of the UNITER model Chen et al. (2020). It has a
similar architecture to UNITER and uses Faster-RCNN Ren et al. (2015) as a
feature extractor for images. For a single images, 36 objects will be
extracted and represented as 36 vectors
$\textbf{v}=\\{\textbf{v}_{1},...,\textbf{v}_{M}\\}$, where v represents the
whole image, and $M=36$. It is worth noting that the parameters of Faster-RCNN
are frozen. Both features and positional embeddings pass through a fully
connected layer, and then merge together. The image features are pooled and
reshaped as vectors with the same dimension as text embeddings. UNITER has
four pre-training methods: Masked Language Modelling (MLM), Masked Region
Modelling (MRM), Image-Text Matching (ITM), and Word Region Alignment (WRA).
xUNITER, except for the pre-training method mentioned above, it uses Masked
Language Modelling for multilingual data and uses the same text embedder as
XLM-R Conneau et al. (2020).

#### UC2

uses a similar model architecture as UNITER, but different pre-training
methods. The pre-training method of UC2 augments pre-training on English data
by constructing multilingual corpus via machine translation and then uses this
augmented data for pre-training. It also proposes the Visual Translation
Language Modeling (VTLM) pre-training method, which uses the image as a pivot
to learn the relationship between parallel texts in two languages and their
corresponding images.

###  4.2 Datasets and Metrics

We use datasets corresponding to four tasks from the IGLUE benchmark
Bugliarello et al. (2022), which includes xGQA Pfeiffer et al. (2022), MaRVL
Liu et al. (2021), XVNLI, and xFlickr&Co Plummer et al. (2015); Lin et al.
(2014). We visualize examples of MaRVL and XVNLI in 1. Following the setting
in IGLUE, the evaluation metric is accuracy for all tasks except cross-modal
retrieval, which uses Recall@1 .

###  4.3 Implementation and Hyperparameters

We conduct all experiments based on the Visiolinguistic Transformer
Architectures framework VOLTA111https://github.com/e-bug/volta on 4 2080Ti
GPUs. We implement the MAML algorithm based on
Higher222https://github.com/facebookresearch/higher library. We use the AdamW
Loshchilov and Hutter (2018) optimizer to fine-tune all models in PyTorch.

#### Fine-tuning on English Data

Before evaluating models on the data in low-resource languages, we firstly
fine-tune the pre-trained models on the English datasets NLVR2 Suhr et al.
(2019), SNLI-VE Xie et al. (2019), Flickr30k Plummer et al. (2015), and GQA
Hudson and Manning (2019) for MaRVL, XVNLI, xFlickr&Co, and xGQA,
respecitively, following the procedure of Bugliarello et al. (2022) and Liu et
al. (2021). We follow the setting in IGLUE Bugliarello et al. (2022) and also
used the IGLUE hyper-parameters for each task when fine-tuning. We save the
parameters of models in each epoch, then picking the best performing model for
each task as the initialized parameters $\theta$ for meta-learning fine-tuning
stage.

#### Fine-tuning with Meta-learning

For the X-MAML and Contrastive-MAML algorithms, both the size of the support
set and the query set are 64. We explore learning rates $5\times 10^{-5}$,
$1\times 10^{-5}$, $5\times 10^{-6}$ , $1\times 10^{-6}$ for both UC2 and
xUNITER, and find the best learning rate is $5\times 10^{-6}$ for both the
normal fine-tuning stage and the meta-update of MAML. For the inner learning
rate of X-MAML and Contrastive-MAML, we explore learning rates $5\times
10^{-6}$, $5\times 10^{-5}$, $5\times 10^{-4}$ and $5\times 10^{-3}$, and find
that $5\times 10^{-4}$ is the best inner learning rate.

For the proposed meta-learning framework, we set the number of iterations to
25, 50, 100, 150, 200, 300, 400, 500, 1000 (for each iterations, we sample a
batch of data as support set and a batch as query set). We find that models
overfit after 300 iterations in most situations, so we set the number of
iterations as 400 for all our experiments, and evaluate the performance of
models for each 25 iterations to guarantee that we can pick the model with
best performance of each setting for evaluation.

##  5 Results and Discussion

###  5.1 Zero-shot

We report the results of the baseline models, and the results for fine-tuning
them using our meta-learning framework, in Table 1. In our setting, baseline
model means the PVLMs are only fine-tuned on the English datasets. For
simplicity, we only report the averaged results of all combinations of target
languages and auxiliary languages for each model and task. Our proposed meta-
learning framework is the combination of Contrastive-MAML and X-MAML. We set
the value of $\lambda$ in Equation (8) to $1\times 10^{-3}$ for xUNITER and
$0.2$ for UC2.

The results in the Table 1 indicate the effectiveness of our meta-learning
framework, and show that our method can boost the zero-shot performance of UC2
and XUNITER on all four datasets in the IGLUE benchmark. Note that to Table 1
shows average performance across all languages – performance for individual
languages can vary, and is shown in detail in Appendix A, Table 5. We also
shows the differences of improvements by using different auxiliary languages
for different target languages in Figure 4.

METHOD | ZH | TA | SW | TR | ID | avg
---|---|---|---|---|---|---
xUNITER
Base | 54.34/4.74 | 55.40/6.55 | 56.41/7.61 | 57.53/10.99 | 56.44/7.79 | 56.02/7.54
Ours ($zh\rightarrow X$) | - | 59.82/14.10 | 58.85/9.78 | 60.93/13.22 | 61.17/13.48 | -
Ours (AVG) | 58.34/9.88 | 58.49/10.25 | 59.59/10.33 | 60.06/12.03 | 60.35/12.41 | 59.37/10.98
Ours (MAX) | 59.75/10.28 | 59.82/14.10 | 60.83/10.14 | 62.20/15.25 | 61.17/13.48 | 60.75/12.65
UC2
Base | 57.81/12.25 | 60.06/11.15 | 51.81/1.09 | 55.76/7.46 | 56.56/8.51 | 56.40/8.09
Ours ($zh\rightarrow X$) | - | 58.94/12.13 | 53.61/7.57 | 55.34/7.99 | 56.74/8.03 | -
Ours (AVG) | 58.35/13.44 | 58.35/12.71 | 53.99/7.93 | 56.80/9.61 | 56.54/9.41 | 56.81/10.62
Ours (MAX) | 59.59/13.04 | 58.94/12.13 | 54.60/9.11 | 58.13/13.48 | 56.74/12.60 | 57.60/12.07
Table 2: Zero-shot performance (accuracy/consistency) of two baseline models
fine-tuned only by English data (Base) and them finetuned by our meta-learning
method (Ours) on the MARVL Liu et al. (2021) dataset. Columns indicates low-
resource target languages. The avg column indicates the average performance
across all low-resource languages in this row. We evaluate pre-trained models
on this dataset direclty. We also report results using our proposed method:
finetuning on one auxiliary language with X-MAML and Contrastive-MAML.
$zh\rightarrow X$ means the auxiliary language is Chinese and the target
languages is other low-resource languages $X$. We also show the average (AVG)
and maximum (MAX) performance across all auxiliary languages for each target
languages.

###  5.2 Few-shot

Figure 3: Average few-shot performance (accuracy) across all languages of two
baseline models on the XVNLI and MARVL datasets. The horizontal axis
represents the number of shots in training data. Figure 4:  Improvements of
zero-shot performance by fine-tuning xUNITER on different auxiliary languages
then evaluating on different target languages using our proposed framework
compared with baseline. The left heatmap is on MaRVL, and the right is on
XVNLI . Rows correspond to auxiliary and columns correspond to target
languages Un-supervised Setting
---
Method/Models | UC2 | xUNITER
Baseline | 62.5 | 59.1
XVL-MAML(un-sup) | 63.1 (+0.7) | 60.8 (+1.7)
Table 3: Ablation study in un-supervised setting, where the labels of down-
stream task data in auxiliary language is not given.

We also conduct few-shot experiments following the setting in IGLUE
Bugliarello et al. (2022) for both xUNITER and UC2 on XVNLI and MaRVL. The
results are shown in Figure 3, where the horizontal axis represents the number
of shots, and the vertical axis represents the accuracy score. The leftmost
point in the horizontal axis is zero, which represents the performance in
zero-shot setups. The blue points and lines show the performance of models
fine-tuned by our method. The yellow lines and points represent the
performance of the baseline. It is clear that in all these four figures, our
method achieves better performance in all shots. And it is worth noting that
although there is a slight increase from the performance of zero-shot to one-
shot, our proposed method without seeing any data in the target languages
outperforms the baselines in the few-shot setting, except for UC2 on MaRVL. In
other words, only a few instances of training data in target languages is not
enough to eliminate the advantage of our method. It further demonstrates that
while our method requires training data in one auxiliary language, there is no
need for few-shot data in target languages.

###  5.3 Ablation Study and Further Analysis

In this section, we conduct a series of ablation studies, which investigate
the effect of each part of our proposed meta-learning framework. We have
performed five runs for each settings and reported the average and standard
error to show the significance.

#### The Effect of Contrastive Learning

We investigate the effect of contrastive learning in our meta-learning fine-
tuning framework. Specifically, we fine-tune the model using contrastive
learning loss in the MAML algorithm, just as the un-supervised setting
described in the Section 3.2, where the labels of auxiliary langauges is not
available. We evaluate the performance of UC2 and xUNITER on XVNLI dataset in
the un-supervised setting in Table 3. It indicates using contrastive learning
solely in the MAML algorithm can also gain improvements of performance. It
provides evidence for the idea of contrastive learning can enable models learn
alignment of modalities in cross-lingual transfer, and hence gain better
representations.

we also compare the performance of the model in the supervised setting where
labels of data in auxiliary language is available, so in XVL-MAML algorithm,
both contrastive loss and down-stream task loss are used. Then, we remove the
contrastive learning loss in the XVMAML algorithm, only keeping the down-
stream task loss. We compare the performance of these two setting in Table 4
to show the effectiveness of contrastive learning loss in XVL-MAML algorithm
in the supervised setting. In Table 4, the first row is XVL-MAML algorithm
without using contrastive learning loss, which means only using down-stream
task loss when fine-tuning, and the second row it the normal XVL-MAML which
uses both contrastive loss and down-stream task loss.

Supervised Setting
---
Method/Models | UC2 | xUNITER
XVL-MAML (w/o contras) | 63.8 | 61.6
XVL-MAML(sup) | 64.4 (+0.6) | 62.9 (+1.3)
Table 4: Ablation study in supervised scenarios, where the labels of down-
stream task data in auxiliary language is provided.

#### Diverse down-stream tasks

We conduct experiments in four diverse tasks of Multilingual Vision-and-
Language Understanding in Table 1. In all tasks, our method can bring clear
improvements for both UC2 and xUNITER models. indicates that our method is
robust for the type of task. For these four tasks proposed by Bugliarello et
al. (2022), the distribution of language families are different, and the
domain of data are also different, which indicates our method can be useful in
multiple domains and language families. Moreover, surprisingly, our method can
significantly boost the performance of xUNITER even in challenging MaRVL
dataset which across 5 diverse language families and cultures, for improving
4.4 point in accuracy.

#### Diverse auxiliary and target languages

We also investigate how different auxiliary languages can effect performance
in different target languages. Specifically, we take the MaRVL dataset as an
example and report the results in Table 2. Then we visualize the improvements
of xUNITER by using different auxiliary languages for different target
languages on MaRVL and XVNLI datasets in Figure 4. The difference of
improvements in MaRVL (which range from 0.44 to 5.4)is larger than in XVNLI
(which range from 2.8 to 6.4), and one possible reason is that the language
families of MaRVL are more diverse than of XVNLI.

###  5.4 Example Predictions

We show some examples of inputs and predictions for baseline models and the
models fine-tuned by our method in Appendix A. In Figures 5 and 6, we use
xUNITER to predict the Chinese part of the MaRVL dataset. We have selected
three examples where baseline predicted wrongly but our method predicted
correctly, and two examples where both our method and baseline method
predicted correctly. In the first three examples, the label was True but the
baseline predicted False. We find that the same concepts have different visual
features in the left and the right image for each example, which makes it more
difficult for models to identify. For instance, in the first example, the
dining rooms in the left and right images look different. In the last two
examples, however, the concepts described in the text do not have diverse or
obscure visual features when they appear in the images. Therefore, based on
these cases, we can surmise that the meta-learning framework makes the model
more adaptive for diverse information, and have better generalization
capabilities of diverse mapping between texts and images.

##  6 Conclusions

In this paper, we focus on mitigating the problem of poor performance of
current PVLMs in vision-language cross-lingual transfer. We proposed a novel
MAML framework to make pre-trained models quickly adaptive for new languages
in vision-and-language scenarios. Our meta-learning framework combine
contrastive learning and downstream task supervised learning together. We
implement and verify the effectiveness of the our algorithm in both supervised
and un-supervised settings. The key strength of our approach is that we
leverage contrastive learning in the MAML procedure so that the models can
quickly learn aligning representations from different modalities and adapt it
for unseen languages.

Experimental results demonstrate that our proposed meta-learning framework
significantly improves the performance of models in vision-and-language cross-
lingual transfer both in zero-shot and few-shot setups. We applied our method
to two state-of-the-art PVLMs, UC2 and xUNITER, and verified its effectiveness
on four datasets in the IGLUE benchmark in 17 lanugages. We also conducted
ablation study to explore the effect of contrastive learning in supervsied and
un-supervised settings, and made further analysis for different languages and
tasks.

## Limitations

Our proposed method applies contrastive learning to samples of image-text
pairs. The alignments induced in this fashion work best if there is a concept
or an object that is both depicted in the image and referred to in the
sentence. If this is not the case, then the method may end up learning
alignments not so much accurately; this includes cases where the image or the
sentence contain multiple objects or concepts, not all of which can be
aligned. To address this limitation, future work should explore how to
construct better positive and negative samples and how to enable learning at a
more fine-grained level.

## Ethics Statement

The use of IGLUE benchmark in our paper is consistent with their intended use.
We have checked the datasets we use don’t contain any offensive content and
identifiers by sample and visualize examples in datasets. There are 14
languages in the datasets we used, we list them in Table 5, it can also be
found in IGLUE benchmark Bugliarello et al. (2022). The detail information of
datasets is described in Bugliarello et al. (2022).

## References

  * Bugliarello et al. (2022) Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulić. 2022.  Iglue: A benchmark for transfer learning across modalities, tasks, and languages.  _arXiv preprint arXiv:2201.11732_.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Uniter: Universal image-text representation learning.  In _European conference on computer vision_ , pages 104–120. Springer.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Elliott et al. (2016) Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016.  Multi30k: Multilingual english-german image descriptions.  _arXiv preprint arXiv:1605.00459_.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International conference on machine learning_ , pages 1126–1135. PMLR.
  * Gella et al. (2017) Spandana Gella, Rico Sennrich, Frank Keller, and Mirella Lapata. 2017.  Image pivoting for learning multilingual multimodal representations.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2839–2845, Copenhagen, Denmark. Association for Computational Linguistics.
  * Gu et al. (2018) Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018.  Meta-learning for low-resource neural machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3622–3631, Brussels, Belgium. Association for Computational Linguistics.
  * Hudson and Manning (2019) Drew A Hudson and Christopher D Manning. 2019.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6700–6709.
  * Li et al. (2018) Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. 2018.  Learning to generalize: Meta-learning for domain generalization.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer.
  * Liu et al. (2021) Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021.  Visually grounded reasoning across languages and cultures.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10467–10485.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _Advances in neural information processing systems_ , 32.
  * Ni et al. (2021) Minheng Ni, Haoyang Huang, Lin Su, Edward Cui, Taroon Bharti, Lijuan Wang, Dongdong Zhang, and Nan Duan. 2021.  M3p: Learning universal representations via multitask multilingual multimodal pre-training.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3977–3986.
  * Nooralahzadeh et al. (2020) Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. 2020.  Zero-shot cross-lingual transfer with meta learning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4547–4562.
  * Obamuyide and Vlachos (2019) Abiola Obamuyide and Andreas Vlachos. 2019.  Model-agnostic meta-learning for relation classification with limited supervision.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5873–5879.
  * Pfeiffer et al. (2022) Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. 2022.  xgqa: Cross-lingual visual question answering.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2497–2511.
  * Plummer et al. (2015) Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2641–2649.
  * Qiu et al. (2022) Chen Qiu, Dan Oneata, Emanuele Bugliarello, Stella Frank, and Desmond Elliott. 2022\.  Multilingual multimodal learning with machine translated text.  _arXiv preprint arXiv:2210.13134_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR.
  * Ren et al. (2018) Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. 2018.  Meta-learning for semi-supervised few-shot classification.  _arXiv preprint arXiv:1803.00676_.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28.
  * Rotman et al. (2018) Guy Rotman, Ivan Vulić, and Roi Reichart. 2018.  Bridging languages through images with deep partial canonical correlation analysis.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 910–921, Melbourne, Australia. Association for Computational Linguistics.
  * Su et al. (2019) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019\.  Vl-bert: Pre-training of generic visual-linguistic representations.  In _International Conference on Learning Representations_.
  * Suhr et al. (2019) Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019\.  A corpus for reasoning about natural language grounded in photographs.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6418–6428.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  Lxmert: Learning cross-modality encoder representations from transformers.  _arXiv preprint arXiv:1908.07490_.
  * Verma et al. (2020) Vinay Kumar Verma, Dhanajit Brahma, and Piyush Rai. 2020.  Meta-learning for generalized zero-shot learning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 6062–6069.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016.  Matching networks for one shot learning.  _Advances in neural information processing systems_ , 29.
  * Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and tell: A neural image caption generator.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3156–3164.
  * Wang et al. (2021) Bailin Wang, Mirella Lapata, and Ivan Titov. 2021.  Meta-learning for domain generalization in semantic parsing.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 366–379.
  * Xie et al. (2019) Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2019.  Visual entailment: A novel task for fine-grained image understanding.  _arXiv preprint arXiv:1901.06706_.
  * Yu et al. (2020) Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020.  Gradient surgery for multi-task learning.  _Advances in Neural Information Processing Systems_ , 33:5824–5836.
  * Zhang et al. (2020) Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. 2020.  Contrastive learning of medical visual representations from paired images and text.  _arXiv preprint arXiv:2010.00747_.
  * Zhen et al. (2019) Liangli Zhen, Peng Hu, Xu Wang, and Dezhong Peng. 2019.  Deep supervised cross-modal retrieval.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10394–10403.
  * Zhou et al. (2021) Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, and Jingjing Liu. 2021.  Uc2: Universal cross-lingual cross-modal vision-and-language pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4155–4165.
  * Zhou et al. (2017) Stephanie Zhou, Alane Suhr, and Yoav Artzi. 2017.  Visual reasoning with natural language.  _arXiv preprint arXiv:1710.00453_.

##  Appendix A Appendix

Figure 5: Examples from the MaRVL dataset which the baseline model predicted
wrongly but the model fine-tuned by our method predicted correctly. In these
three examples, the label was True but the baseline predicted False. We find
that the same concepts have different visual features in the left and the
right image for each example, which makes it more difficult for models to
identify. Figure 6: Examples from the MaRVL dataset which both the baseline
model and the model fine-tuned by our method predicted correctly. The concepts
described in the text do not have diverse or obscure visual features when they
appear in the images.

| Ar | Bn | De | Es | Id | Fr | Ja | Ko | Pt | Ru | Sw | Ta | Tr | Zh
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
MaRVL
xUNITER (Baseline) | - | - | - | - | 56.44 | - | - | - | - | - | 56.41 | 55.40 | 57.53 | 54.34
UC2 (Baseline) | - | - | - | - | 56.56 | - | - | - | - | - | 51.81 | 60.06 | 55.76 | 57.81
xUNITER (Ours) | - | - | - | - | 60.35 | - | - | - | - | - | 59.59 | 58.49 | 60.06 | 59.75
UC2 (Ours) | - | - | - | - | 56.74 | - | - | - | - | - | 54.60 | 58.94 | 58.13 | 59.59
XVNLI
xUNITER (Baseline) | 53.52 | - | - | 60.05 | - | 61.60 | - | - | - | 61.25 | - | - | - | -
UC2 (Baseline) | 58.36 | - | - | 63.86 | - | 65.01 | - | - | - | 64.72 | - | - | - | -
xUNITER (Ours) | 56.70 | - | - | 60.91 | - | 68.64 | - | - | - | 63.91 | - | - | - | -
UC2 (Ours) | 59.94 | - | - | 62.97 | - | 69.41 | - | - | - | 65,18 | - | - | - | -
xGQA
xUNITER (Baseline) | - | 11.41 | 33.21 | - | 32.38 | - | - | 13.28 | 20.51 | 17.84 | - | - | - | 17.20
UC2 (Baseline) | - | 19.49 | 33.52 | - | 29.83 | - | - | 23.29 | 31.23 | 32.61 | - | - | - | 33.25
xUNITER (Ours) | - | 12.46 | 34.10 | - | 33.63 | - | - | 15.05 | 22.71 | 20.27 | - | - | - | 19.27
UC2 (Ours) | - | 19.63 | 34.50 | - | 29.58 | - | - | 24.93 | 32.47 | 33.24 | - | - | - | 35.00
Xflickr&Co (IR)
xUNITER (Baseline) | - | - | 14.70 | 16.40 | 15.15 | - | 9.55 | - | - | 14.75 | - | - | 8.85 | 17.20
UC2 (Baseline) | - | - | 28.10 | 14.65 | 13.55 | - | 23.70 | - | - | 18.20 | - | - | 8.15 | 31.70
xUNITER (Ours) | - | - | 16.20 | 18.85 | 18.50 | - | 12.10 | - | - | 17.75 | - | - | 11.10 | 19.40
UC2 (Ours) | - | - | 29.35 | 16.90 | 14.25 | - | 25.15 | - | - | 20.50 | - | - | 10.50 | 32.10
Xflickr&Co (TR)
xUNITER (Baseline) | - | - | 14.2 | 15.45 | 13.95 | - | 8.30 | - | - | 13.15 | - | - | 7.75 | 14.4
UC2 (Baseline) | - | - | 23.55 | 11.90 | 10.35 | - | 22.75 | - | - | 17.50 | - | - | 6.15 | 26.85
xUNITER (Ours) | - | - | 15.50 | 16.15 | 16.70 | - | 9.90 | - | - | 15.70 | - | - | 9.50 | 15.75
UC2 (Ours) | - | - | 25.30 | 13.95 | 12.45 | - | 23.50 | - | - | 19.80 | - | - | 8.30 | 27.45

Table 5: Accuracy scores for each target language individually averaged over
auxiliary languages.

This is a section in the appendix.