# FAME-ViL: Multi-Tasking Vision-Language Model
for Heterogeneous Fashion Tasks

Xiao Han1,2 Xiatian Zhu1,3 Licheng Yu Li Zhang4 Yi-Zhe Song1,2 Tao Xiang1,2
1 CVSSP, University of Surrey 2 iFlyTek-Surrey Joint Research Centre on
Artificial Intelligence
3 Surrey Institute for People-Centred Artificial Intelligence 4 Fudan
University
{xiao.han, xiatian.zhu, y.song<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In the fashion domain, there exists a variety of vision-and-language (V+L)
tasks, including cross-modal retrieval, text-guided image retrieval, multi-
modal classification, and image captioning. They differ drastically in each
individual input/output format and dataset size. It has been common to design
a task-specific model and fine-tune it independently from a pre-trained V+L
model (_e.g_., CLIP). This results in parameter inefficiency and inability to
exploit inter-task relatedness. To address such issues, we propose a novel
FAshion-focused Multi-task Efficient learning method for Vision-and-Language
tasks (FAME-ViL) in this work. Compared with existing approaches, FAME-ViL
applies a single model for multiple heterogeneous fashion tasks, therefore
being much more parameter-efficient. It is enabled by two novel components:
(1) a task-versatile architecture with cross-attention adapters and task-
specific adapters integrated into a unified V+L model, and (2) a stable and
effective multi-task training strategy that supports learning from
heterogeneous data and prevents negative transfer. Extensive experiments on
four fashion tasks show that our FAME-ViL can save 61.5% of parameters over
alternatives, while significantly outperforming the conventional independently
trained single-task models. Code is available at
https://github.com/BrandonHanx/FAME-ViL.

##  1 Introduction

A variety of real-world multi-modal, particularly Vision-and-Language (V+L)
tasks exist in the fashion domain, including multi-modal recognition [44, 53,
61], multi-modal retrieval [21, 83] and image captioning [85]. The models
developed for these tasks have been applied in diverse e-commerce
applications, improving product discoverability, seller-buyer engagement, and
customer conversion rate after catalogue browsing. Intrinsically, those V+L
tasks are heterogeneous in terms of (1) different input and output formats
(_e.g_., text-guided garment retrieval [83] and image captioning [85] have
completely different inputs and outputs); (2) different dataset sizes as the
annotation difficulty of each task differ (_e.g_., the labeling effort for
text-guided image retrieval is much harder than that for text-to-image
retrieval [83, 48]).

Figure 1: By multi-task learning a single model for heterogeneous fashion
tasks, our FAME-ViL can significantly improve parameter efficiency, while
boosting the model performance per task over existing independently fine-tuned
single-task models. Note, each axis is normalized according to the respective
maximum value for easier visualization.

Due to the heterogeneous nature of the V+L fashion tasks, existing methods
[21, 94, 24, 87, 33] typically take a pre-trained generic V+L model [42, 72,
49, 67, 7, 43, 60, 38, 41, 79] and fine-tune it on every single task
independently. Such an approach suffers from two limitations. (1) Low
parameter efficiency: Each real-world application requires the deployment of
its dedicated fine-tuned model, where there is no parameter or inference
computation sharing. This leads to a linearly increasing storage and inference
compute redundancy in the long run. (2) Lack of inter-task relatedness: Though
the fashion tasks are heterogeneous in nature, the fundamental components of
the models are closely related in that all tasks require a deep content
(image/sentence) understanding. Exploiting the shared information across tasks
thus has the potential to improve model generalization capability leading to a
performance boost.

Perhaps a natural solution would be applying Multi-Task Learning (MTL) [13].
However, most existing multi-task training methods [8, 63, 36, 46, 56] are
designed for homogeneous tasks (_i.e_., one dataset with multi-task labels)
and thus cannot be directly applied to the heterogeneous fashion tasks. In our
case, we are facing two challenges in building the fashion-domain MTL model:
(1) Architecturally, it is non-trivial to model the diverse tasks in one
unified architecture. Taking the popular CLIP [60] as an example, its two-
stream architecture is designed for image-text alignment [52] and thus lacks
the modality fusion mechanism as required by many V+L fashion tasks (_e.g_.,
text-guided image retrieval [83, 2] and image captioning [85]). (2) In terms
of optimization, a fashion-domain MTL model is prone to the notorious negative
transfer problem [8, 63, 36, 46, 56, 13] due to both task input/output format
differences and imbalanced dataset sizes. To the best of our knowledge, there
has been no attempt at V+L MTL for the fashion domain.

In this work, we introduce a novel FAshion-focused Multi-task Efficient
learning method for various Vision-and-Language based fashion tasks, dubbed as
FAME-ViL. It achieves superior performance across a set of diverse fashion
tasks with much fewer parameters as in Fig. 1. Specifically, we design a task-
versatile architecture on top of a pre-trained generic V+L model (_i.e_., CLIP
[60]). To adapt the simple two-stream architecture of CLIP to various fashion
tasks, we introduce a lightweight Cross-Attention Adapter (XAA) to enable the
cross-modality interaction between the two streams. It makes the model
flexible to support multiple task modes (_e.g_., contrastive mode for
retrieval, fusion mode for understanding, and generative mode for generation).
To address the negative transfer challenge, we introduce a Task-Specific
Adapter (TSA) to absorb inter-task input/output format incompatibilities by
introducing lightweight additional per-task parameters. For further handling
the dataset imbalance problem, a multi-teacher distillation scheme [12] is
formulated for our heterogeneous MTL problem. It leverages the pre-trained
per-task teachers to guide the optimization of our multi-task model,
mitigating the overfitting risks of those tasks with smaller training dataset
sizes.

Our contributions are summarized as follows: (I) For the first time, we
investigate the problem of multi-task learning on heterogeneous fashion tasks,
eliminating the parameter redundancy and exploiting the inter-task
relatedness. (II) We propose FAME-ViL with two novel adapters, adapting a pre-
trained CLIP model to all tasks. (III) We introduce an efficient and effective
multi-task training strategy supporting heterogeneous task modes in one
unified model. (IV) Comprehensive experiments on four diverse fashion tasks
(_i.e_., cross-modal retrieval [61, 52], text-guided image retrieval [75, 83],
multi-modal classification [61, 94], and image captioning [85]) show that our
method significantly outperforms the previous single-task state-of-the-art
with 61.5% parameter saving (see Fig. 1).

Figure 2: An illustration of four diverse fashion V+L Tasks studied in this
work: cross-modal retrieval, text-guided image retrieval, sub-category
recognition, and fashion image captioning. Note, all predictions shown in this
figure are made by our FAME-ViL. Green box indicates the ground truth matches
of retrieval tasks.

##  2 Related work

Vision-Language Pre-training (VLP). With the advent of Transformers [73, 15,
17], many pioneer studies [42, 49, 67, 7, 43, 38, 41, 31, 88] have
demonstrated that VLP is effective in boosting various downstream V+L tasks in
the generic domain. Since then, we have witnessed further developments of VLP
methods, being bigger [60, 34, 19], more unified [79, 86, 66, 78, 77, 50] and
more flexible [76, 18, 84].

Fashion V+L tasks. There exist a variety of heterogeneous tasks in the fashion
domain. As depicted in Fig. 2, we consider four popular fashion tasks in this
work: (1) Cross-Modal Retrieval (XMR) requests to efficiently retrieve the
most matched image/sentence from a large candidate pool given a text/image
query [61, 52]. (2) Text-Guided Image Retrieval (TGIR) is a special type of
image retrieval with a multi-modal query (a combination of a reference image
and a modifying text) matched against a set of images [6, 39, 37, 64, 23, 14].
It not only requires a strong fusion of the reference image and modifying
text, but also an efficient matching between the fused representation and all
candidate images [83, 24]. (3) Sub-Category Recognition (SCR) requires an
accurate class prediction made upon the fusion of an image-text pair [61, 94].
(4) Fashion Image Captioning (FIC) generates a caption to describe the given
image with semantically meaningful, fine-grained, and accurate words [85].
Many recent works have been trying to address these fashion tasks through VLP
[21, 94, 22, 24, 87, 33, 55]. Most of them focus on the pre-training, then
simply fine-tune the pre-trained model on each downstream task independently.
In contrast, we integrate all these tasks into a unified architecture and thus
no separate fine-tuning is needed. Since our fashion data is also abundant,
most early works pre-train on the fashion domain directly. However, a number
of recent works [2, 3, 16, 10, 52] suggest that a generic-domain pre-trained
CLIP [60] generalizes even better on the fashion tasks. In this work, we also
exploit a pre-trained CLIP model. Different from the existing methods, we use
a single multi-task learned model for all mentioned fashion tasks during fine-
tuning.

Parameter-efficient tuning. Due to the increase in the size of V+L models,
there is a growing interest in developing parameter-efficient methods to
quickly adapt a large pre-trained model to specific tasks by using as few
extra parameters as possible. The most representative methods are adapters
[28, 5, 70, 9], prompt tuning [47, 92, 91, 35], low-rank adaptation [29] and
their unified variants [25, 54, 89]. Interestingly, whilst MTL can save much
larger parameters, it is under-studied in V+L modeling. In this work, we
propose two kinds of adapters (XAA and TSA in Sec. 3.1) to adapt CLIP
specifically designed for MTL in the fashion domain. Besides parameter-
efficiently adapting CLIP, our proposed adapters also serve as the key
component for task-versatile architecture design and enabling stable MTL.

Multi-task learning. Although MTL offers many advantages like improved data
efficiency and reduced over-fitting, how to avoid negative transfer and cope
with severely imbalanced dataset sizes is still an open question. One common
solution is to weight per-task losses or combine per-task gradients into a
joint update direction using various heuristics [8, 63, 36, 46, 56]. These
works require the MTL model to have at least one forward propagation on each
task so that they can manipulate the overall losses or gradients. However,
since V+L tasks are typically heterogeneous (especially in the fashion
domain), this requirement cannot be easily satisfied, making these methods not
directly applicable. In contrast, Task Sampling-based MTL (TS-MTL) is without
such a requirement and thus being widely adopted in V+L models [57, 51, 30, 7,
66]. In TS-MTL, only one task along with its data point is sampled per
iteration. Since then, the heuristic task-sampling strategies [51, 30, 32]
have been proposed, aiming to balance different tasks, avoiding the over-
fitting on easier (or data-poor) tasks or catastrophic forgetting [20] on
harder (or data-rich) tasks. However, it is found that TS-MTL on its own often
underperforms single-task trained models; it is thus typically followed by an
additional per-task fine-tuning step [51, 30]. In this work, we formulate an
effective knowledge distillation with multiple single-task teachers [12] to
alleviate the negative transfer without further fine-tuning on each task.

Figure 3: An illustration of a task-versatile Transformer layer equipped with
two newly-introduced adapters: cross-attention adapter (XAA) and task-specific
adapter (TSA).

##  3 Methodology

We aim to address the most popular fashion tasks (shown in Fig. 2) using one
single unified model. We introduce the details of our proposed FAME-ViL as
follows.

###  3.1 Model architecture

FAME-ViL consists of a language encoder and a vision encoder initialized from
the pre-trained CLIP (ViT-B/16 version) [60], as well as a set of newly
introduced adapters. Transformer layers. The key component in CLIP is the
transformer backbone [73, 17]. A vanilla Transformer encoder consists of an
input embedding layer (word embedding for language input and patch embedding
for vision input) and several alternating layers made of Multi-Head Self-
Attention (MHSA) and MLP blocks. Layer Normalization (LN) [1] is applied
before every block, and residual connections after every block [17, 60]. As
shown in the middle of Fig. 3, this process can be formulated as follows:

$\displaystyle\mathbf{z}_{0}$
$\displaystyle=\operatorname{Embedding}(\mathbf{x}),$
$\displaystyle\mathbf{z}_{0}$ $\displaystyle\in\mathbb{R}^{N\times D},$ (1)
$\displaystyle\mathbf{z}_{\ell}^{\prime}$
$\displaystyle=\operatorname{MHSA}(\operatorname{LN}(\mathbf{z}_{\ell-1}))+\mathbf{z}_{\ell-1},$
$\displaystyle\ell$ $\displaystyle=1\ldots L,$ (2)
$\displaystyle\mathbf{z}_{\ell}$
$\displaystyle=\operatorname{MLP}(\operatorname{LN}(\mathbf{z}_{\ell}^{\prime}))+\mathbf{z}_{\ell}^{\prime},$
$\displaystyle\ell$ $\displaystyle=1\ldots L,$ (3)

where $N,D,L$ denotes the number of input tokens, transformer dimension, and
the number of layers, respectively.

Proposed adapters. To adapt the original Transformer layers in CLIP to
different fashion downstream tasks, we design two kinds of adapters in
architecture design: (1) Task-Specific Adapter (TSA) for accommodating the
non-shareable characteristics of each individual task (Fig. 3 left); (2)
Cross-Attention Adapter (XAA) for enabling the interaction between different
modalities (Fig. 3 right).

For TSA we adopt the scaled parallel adapter [25, 5] that adds another
bottleneck MLP (AdaptMLP) in parallel with the original MLP of each
transformer layer. Given an immediate input $\mathbf{z}_{\ell}^{\prime}$, it
produces the adapted features, $\mathbf{z}_{\ell}^{tsa}$, via:

$\mathbf{z}_{\ell}^{tsa}=s\cdot\operatorname{AdaptMLP}(\operatorname{LN}(\mathbf{z}_{\ell}^{\prime})),$
(4)

where $s$ represents a learnable scale.

We construct an XAA module by further adding another Multi-Head Cross
Attention (MHXA) layer [49, 72, 18] at the bottom of a TSA. Specifically, this
MHXA uses the hidden state of the current stream $\mathbf{z}_{\ell}^{\prime}$
as the queries and the output $\mathbf{y}_{\ell}$ (_e.g_., hidden state after
MHSA or MLP) of another stream as the keys and values. The attended cross-
modality features $\mathbf{z}_{\ell}^{xaa}$ are computed via:

$\mathbf{z}_{\ell}^{xaa}=s\cdot\operatorname{AdaptMLP}(\operatorname{LN}(\operatorname{MHXA}(\mathbf{z}_{\ell}^{\prime},\mathbf{y}_{\ell}))).$
(5)

Within this mechanism, our XAA can exchange the information between different
modalities.

Finally, both $\mathbf{z}_{\ell}^{tsa}$ and $\mathbf{z}_{\ell}^{xaa}$ are
added up to the original output via residual connections. We thus rewrite Eq.
(3) as:

$\mathbf{z}_{\ell}=\operatorname{MLP}(\operatorname{LN}(\mathbf{z}_{\ell}^{\prime}))+\mathbf{z}_{\ell}^{\prime}+\mathbf{z}_{\ell}^{tsa}+\epsilon\cdot\mathbf{z}_{\ell}^{xaa},\epsilon\in\left\\{0,1\right\\},$
(6)

where $\epsilon$ represents a gate that can turn on/off XAA according to the
task need.

Figure 4:  Schematic overview of three operational modes with our FAME-ViL.
XAA: Cross-Attention Adapter; TSA: Task-Specific Adapter. Layer normalization
and original residual connections are not shown here for simplicity.

Operational modes and fashion tasks. Our FAME-ViL can switch among three
operational modes to flexibly support various fashion tasks (see Fig. 4).

$\diamond$ Contrastive mode: This mode supports Cross-Modal Retrieval (XMR)
tasks, including both text-to-image and image-to-text retrieval [52, 24]. All
XAA modules are turned off as no need for inter-modal interaction, whereas
only TSA modules are applied as in Fig. 4(a). During training, given a batch
of $B$ image-text pairs $(\mathbf{I}$, $\mathbf{T})$ =
$\\{(I_{1},T_{1}),\dots,(I_{B},T_{B})\\}$, we first compute their unimodal
representations by the TSA-equipped vision and language encoders
independently. Then, we maximize their similarities via symmetrical
contrastive loss:

$\mathcal{L}_{\mathrm{XMR}}=\frac{1}{2}\left[\mathcal{L}_{\mathrm{InfoNCE}}(\mathbf{T},\mathbf{I})+\mathcal{L}_{\mathrm{InfoNCE}}(\mathbf{I},\mathbf{T})\right],$
(7)
$\footnotesize\mathcal{L}_{\mathrm{InfoNCE}}(\mathbf{X},\mathbf{Y})=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(s(X_{i},Y_{i})/\tau)}{\sum_{j=1}^{B}\exp(s(X_{i},Y_{j})/\tau)},$
(8)

where $\tau$ is a learnable temperature. The similarity is measured by the dot
product of their pooled then normalized features:
$s(I_{i},T_{j})=f_{\theta}^{[c]}(I_{i})^{T}\cdot f_{\theta}^{[c]}(T_{j}).$

$\diamond$ Fusion mode: As in Fig. 4(b), both XAA and TSA modules are enabled
in this mode. Given an input image-text pair $(I,T)$, the model serves as a
single-stream fusion encoder producing two cross-modal attended
representations: $f_{\theta}^{[f]}([I;T])$ and $f_{\theta}^{[f]}([T;I])$. The
final fused representation is a simple addition:
$f_{\theta}^{[f]}(I,T)=f_{\theta}^{[f]}([I;T])+f_{\theta}^{[f]}([T;I])$
111More complex fusion methods (_e.g_., [2]) may yield better results. We
leave this for future study.. This mode is useful for the Sub-Category
Recognition (SCR) [61, 94] and Text-Guided Image Retrieval (TGIR) [75, 83].

SCR aims to predict the subcategory of fashion products based on both input
image and text. We thus append a classifier on top of the fused representation
and minimize its cross-entropy loss:

$\mathcal{L}_{\mathrm{SCR}}=-\mathbb{E}_{(I,T)\sim D}\log
P\left(f_{\theta}^{[f]}(I,T)\right).$ (9)

Considering the unique challenges of TGIR (_i.e_., requiring not only strong
fusion but also efficient matching), FAME-ViL operates in a hybrid mode for
it. Formally, given a batch of {reference images $\mathbf{I}^{r}$, modifying
text $\mathbf{T}$, target images $\mathbf{I}^{t}$}, we first calculate the
fused representation $f_{\theta}^{[f]}(I^{r},T)$ in the fusion mode; Then, we
obtain the target image representation $f_{\theta}^{[c]}(I^{t})$ in the
contrastive mode. During training, we pull them closer in a contrastive way:

$\mathcal{L}_{\mathrm{TGIR}}=\mathcal{L}_{\mathrm{InfoNCE}}\left((\mathbf{I}^{r},\mathbf{T}),\mathbf{I}^{t}\right),$
(10)

where the similarity is still measured by dot product:
$s((I_{i}^{r},T_{i}),I_{j}^{t})=f_{\theta}^{[f]}(I_{i}^{r},T_{i})^{T}\cdot
f_{\theta}^{[c]}(I_{j}^{t})$.

$\diamond$ Generative mode: This works as a seq2seq model performing the
generative tasks auto-regressively [71, 11], _e.g_., Fashion Image Captioning
(FIC) [85]. As in Fig. 4(c), we use a TSA-equipped vision encoder as the
encoder, a TSA-equipped language encoder as the decoder, and only image-to-
text XAA modules for the conditional caption synthesis. Following a standard
encoder-decoder architecture, our image encoder provides layer-wise latent
memory and the text decoder learns to maximize the conditional likelihood of
the paired text under the forward auto-regressive factorization [86, 40]:

$\mathcal{L}_{\mathrm{FIC}}=-\mathbb{E}_{(I,T)\sim D}\sum_{a=1}^{A}\log
P\left(T_{a}\middle|f_{\theta}^{[g]}(I;T_{<a})\right),$ (11)

where $A$ denotes the length of each sentence, and $f_{\theta}^{[g]}([I;T])$
denotes the representation. During training, we enforce the teacher forcing
[81] to achieve parallel computation and thus maximize the learning
efficiency.

###  3.2 Heterogeneous multi-task learning

For training $\mathcal{T}$ fashion tasks, we need to optimize both task-
agnostic parameters $\theta_{s}$ (_i.e_., CLIP backbone & XAA modules) and a
set of task-specific components
$\left\\{\theta_{t}\right\\}_{t=1}^{\mathcal{T}}$ (_i.e_., TSA modules & task
heads). Our objective is to maximize the overall performance across all tasks.
The heterogeneity nature of fashion tasks causes the discrepancy in mini-batch
construction and training dynamics (_e.g_., converging speed, overfitting) as
well as data imbalance, making our multi-task learning particularly
challenging. To address all these issues, we exploit the idea of Multi-Teacher
Distillation (MTD) [27, 12].

Specifically, MTD consists of two stages. In the first stage, we train a
teacher model with the identical architecture as our multi-task model on every
task. Then in the second stage, we apply these teachers to guide the training
of the multi-task model (_i.e_., the student) with designed per-task
distillation objectives.

For XMR, we first compute the image-text similarity using the features of the
single-task teacher $g_{\mathrm{xmr}}$:
$\tilde{s}(I_{i},T_{j})=g_{\mathrm{xmr}}(I_{i})^{T}\cdot
g_{\mathrm{xmr}}(T_{j}).$ Using this similarity as a pseudo-target, we
maximize its mutual information with the student’s counterpart [41]:

$\small\mathcal{L}_{\mathrm{XMR}}^{\mathrm{D}}=\frac{1}{2B}\sum_{b}^{B}\left(\operatorname{KL}\left(\mathbf{s}_{b,\cdot}\parallel\tilde{\mathbf{s}}_{b,\cdot}\right)+\operatorname{KL}\left(\mathbf{s}_{\cdot,b}\parallel\tilde{\mathbf{s}}_{\cdot,b}\right)\right),$
(12)

where $\operatorname{KL}(\cdot\parallel\cdot)$ denotes the Kullback–Leibler
divergence loss on the softmax of the inputs.

For TGIR, we use a similar method as XMR to distill the knowledge from single-
task teacher $g_{\mathrm{tgir}}$:

$\mathcal{L}_{\mathrm{TGIR}}^{\mathrm{D}}=\frac{1}{B}\sum_{b}^{B}\operatorname{KL}\left(\mathbf{s}_{(b,b),\cdot}\parallel\tilde{\mathbf{s}}_{(b,b),\cdot}\right),$
(13)

where the soft target is calculated via:
$\tilde{s}((I_{i}^{r},T_{i}),I_{j}^{t})=g_{\mathrm{tgir}}(I_{i}^{r},T_{i})^{T}\cdot
g_{\mathrm{tgir}}(I_{j}^{t})$.

For SCR and FIC, we directly use the classification probabilities predicted by
the teachers as pseudo-targets:

$\mathcal{L}_{\mathrm{SCR}}^{\mathrm{D}}=\operatorname{KL}\left(f_{\theta}^{[f]}(I,T)\parallel
g_{\mathrm{scr}}(I,T)\right),$ (14)
$\mathcal{L}_{\mathrm{FIC}}^{\mathrm{D}}=\sum_{a=1}^{A}\operatorname{KL}\left(f_{\theta}^{[g]}(I;T_{<a})_{a}\parallel
g_{\mathrm{fic}}(I;T_{<a})_{a}\right).$ (15)

Task scheduling. For training simplicity, we randomly sample one task per
iteration. We optimize the summation of the original loss and distillation
loss as:

$\mathcal{L}=\mathcal{L}_{\mathrm{[task]}}+\mathcal{L}_{\mathrm{[task]}}^{\mathrm{D}},\quad\scriptstyle\mathrm{[task]}\overset{P}{\sim}\\{\mathrm{XMR,TGIR,SCR,FIC}\\},$
(16)

where $P$ denotes the sampling probability. To tackle data imbalance, unless
stated otherwise we set the sampling probability for a particular task $\tau$
linearly proportional to the size of its dataset $|D_{\tau}|$ [62, 30]. We
name this strategy as size-proportional sampling.

##  4 Experiments

Datasets. We evaluate our model on the datasets commonly used in the previous
methods. Specifically, we test FashionGen [61] for XMR, SCR, and FIC, and
FashionIQ [83] for TGIR. FashionGen [61] contains $68k$ fashion products
accompanied by text descriptions. Each product includes $1\sim 6$ images from
different angles, resulting in $260.5k$ image-text pairs for training, and
$35.5k$ for testing. FashionIQ contains $18k$ training triplets (_i.e_.,
reference image, modifying text, target image) and $6k$ validation triplets
over three categories: Dress, Shirt, and Toptee. Each pair (reference image,
target image) is manually annotated with two modifying texts, which are
concatenated [83].

Implementation details. We use MMF [65] and PyTorch [59] to implement our
FAME-ViL. We use the off-the-shelf CLIP from HuggingFace [82] as our pre-
trained model. We use 4 RTX 3090 GPUs for the multi-task training. The default
bottleneck dimension of AdaptMLP is $64$. More implementation details are
listed in the supplementary file.

Performance metrics. Following [94, 24], we report R@K for retrieval, Accuracy
& Macro$F_{1}$ for classification and BLEU-4 [58] & METEOR [4] & ROUGE-L [45]
& CIDEr [74] for captioning. (1) For each task, we first report the average
absolute performance:
$\mu_{\mathcal{T}_{i}}=\frac{1}{|M|}\sum_{j=0}^{|M|}{M_{\mathcal{T}_{i},j}}$.
(2) Since there is no unified objective among multiple tasks and the scale of
per-task metrics often varies largely, we then report the average per-task
relative performance $\Delta_{\mathcal{T}_{i}}$ w.r.t. the single-task
baseline:
$\Delta_{\mathcal{T}_{i}}=\left(\mu_{\mathcal{T}_{i}}-\mu_{\mathrm{STL}}\right)/\mu_{\mathrm{STL}}.$
This can clearly indicate the positive/negative transfer effect. (3) We also
report the relative parameters saving of FAME-ViL and its variants w.r.t. the
vanilla CLIP baseline. Note, inference speed comparison is infeasible as this
depends on the application of different tasks (no fixed rules).

Methods | Image to Text | Text to Image | Mean
---|---|---|---
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10
FashionBERT [21] | 23.96 | 46.31 | 52.12 | 26.75 | 46.48 | 55.74 | 41.89
OSCAR [43] | 23.39 | 44.67 | 52.55 | 25.10 | 49.14 | 56.68 | 41.92
KaledioBERT [94] | 27.99 | 60.09 | 68.37 | 33.88 | 60.60 | 68.59 | 53.25
EI-CLIP [52] | 38.70 | 72.20 | 84.25 | 40.06 | 71.99 | 82.90 | 65.02
MVLT [33] | 33.10 | 77.20 | 91.10 | 34.60 | 78.00 | 89.50 | 67.25
FashionViL [24] | 65.54 | 91.34 | 96.30 | 61.88 | 87.32 | 93.22 | 82.60
FashionViL(vit)  | 63.74 | 90.02 | 95.98 | 60.76 | 86.18 | 92.96 | 81.61
FAME-ViL(ST)  | 65.02 | 90.96 | 96.20 | 63.56 | 86.84 | 93.06 | 82.61
FAME-ViL | 65.94 | 91.92 | 97.22 | 62.86 | 87.38 | 93.52 | 83.14

Table 1: Cross-Modal Retrieval (XMR) results on FashionGen [61]. Test
protocol: random 100 [21, 94, 24].

###  4.1 Comparisons with prior art methods

We compare our models with the previous state-of-the-art methods on each task.
For extensive and fair comparisons, all the prior competitors are based on
large-scale pre-trained models. We even implement an enhanced variant of the
latest art model FashionViL [24] by replacing ResNet50 [26] with the ViT-B/16
[17] backbone (same as our FAME-ViL), denoted as FashionViL(vit). In design,
all the previous methods adopt Single-Task Learning (STL). We compare them
with two variants of our model: (1) single unified MTL model; (2) STL variant
of our FAME-ViL, denoted as FAME-ViL(ST), which is trained on each task
independently using the same TSA and XAA design as FAME-ViL.

XMR evaluation. We consider both image-to-text retrieval and text-to-image
retrieval with two kinds of protocols used by previous methods: (1) random
100: For each query, 100 candidates are randomly sampled from the same
category to construct a retrieval database; The goal is to locate the positive
match depicting the same garment instance from these 100 same-category
negative matches[21, 94, 33]. (2) full database: We also adopt a more
challenging and practical protocol that conducts retrieval on the entire
product set [52, 24], which is in line with actual product retrieval
scenarios. We use random 100 to compare with prior art methods while using
full database to do ablation studies. The results of XMR on FashionGen [61]
are reported in Tab. 1. We draw several observations: (1) Our FAME-ViL
outperforms all prior art fashion models often by a large margin, validating
the performance advantages of our method over alternatives in addition to
better parameter efficiency. (2) FAME-ViL is superior over its single-task
variant FAME-ViL(ST) in most cases and on the average accuracy, suggesting
that our multi-task learning strategy is effective in exploiting the inter-
task relatedness. (3) Our FAME-ViL(ST) can surpass all prior models pre-
trained on large fashion domain data [21, 43, 94, 24], suggesting that using
fashion data in pre-training is not necessarily most important, and model
design (_e.g_., our TSA and XAA) could play a more significant role.
Similarly, its large margin over the previous pre-trained CLIP-based model
[52] further validates the significance of model architecture design.

Methods | Dress | Shirt | Toptee | Mean
---|---|---|---|---
| R@10 | R@50 | R@10 | R@50 | R@10 | R@50
CIRPLANT [48] | 17.45 | 40.41 | 17.53 | 38.81 | 21.64 | 45.38 | 30.20
TIRG(bert) [75]$\dagger$ | 27.17 | 53.25 | 22.28 | 45.58 | 27.84 | 57.11 | 38.87
FashionVLP [22] | 26.77 | 53.20 | 22.67 | 46.22 | 28.51 | 57.47 | 39.14
FashionViL [24] | 33.47 | 59.94 | 25.17 | 50.39 | 34.98 | 60.79 | 44.12
FashionViL(vit)  | 31.53 | 57.91 | 26.74 | 50.69 | 36.77 | 61.81 | 44.24
Baldrati _et al_. [2] | 33.81 | 59.40 | 39.99 | 60.45 | 41.41 | 65.37 | 50.07
Zhao _et al_. [90] | 33.60 | 58.90 | 39.45 | 61.78 | 43.96 | 68.33 | 51.00
FAME-ViL(ST)  | 37.78 | 63.86 | 45.63 | 66.78 | 47.22 | 70.88 | 55.36
FAME-ViL | 42.19 | 67.38 | 47.64 | 68.79 | 50.69 | 73.07 | 58.29

Table 2: Text-Guided Image Retrieval (TGIR) results on FashionIQ [83].
$\dagger$: The results taken from [24].

TGIR evaluation. We compare our FAME-ViL with TGIR-specialist methods [48, 75,
22, 2, 90] and the art fashion-focused V+L model FashionViL [24] under the
original protocol used by FashionIQ [83]. The results are given in Tab. 2. We
have similar observations as on XMR. In particular, we note that our single-
task variant already achieve a new art performance. With a simple addition-
based fusion mechanism, FAME-ViL can even outperform significantly [2] with
the same CLIP pre-training and a complex fusion module. We attribute this
mostly to the contribution of XAA-backed inter-modal interaction (See Tab. 4).

Methods |  SCR | FIC
---|---|---
| Acc | $\textbf{F}_{1}$ |  Mean | B | M | R | C | Mean
FashionBERT [21]$\dagger$ | 85.27 | 62.00 | 73.64  | 3.30 | 9.80 | 29.70 | 30.10 | 18.23
OSCAR [43]$\dagger$ | 84.23 | 59.10 | 71.67  | 4.50 | 10.90 | 30.10 | 30.70 | 19.05
KaleidoBERT [94] | 88.07 | 63.60 | 75.84  | 5.70 | 12.80 | 32.90 | 32.60 | 21.00
FashionViL [24] | 92.23 | 83.02 | 87.63  | 16.71 | 25.97 | 37.82 | 39.08 | 29.90
MVLT [33] | 93.57 | 82.90 | 88.24  | - | - | - | - | -
FashionViL(vit)  | 94.01 | 85.77 | 89.89  | 16.18 | 25.60 | 37.23 | 39.30 | 29.58
FAME-ViL(ST)  | 94.33 | 86.21 | 90.27  | 29.97 | 24.83 | 54.79 | 145.1 | 63.67
FAME-ViL | 94.67 | 88.21 |  91.44 | 30.73 | 25.04 | 55.83 | 150.4 | 65.50

Table 3: Results of Subcategory Recognition (SCR) and Fashion Image Captioning
(FIC) on FashionGen [61]. $\dagger$: copied from [94].

|  |  |  $\mathcal{T}_{1}$: XMR |  $\mathcal{T}_{2}$: TGIR |  $\mathcal{T}_{3}$: SCR |  $\mathcal{T}_{4}$: FIC |  |
---|---|---|---|---|---|---|---|---
Groups |  Methods | #Params (%) | $\mu$ | $\Delta$ | $\mu$ | $\Delta$ | $\mu$ | $\Delta$ | $\mu$ | $\Delta$ | $\bar{\mu}$ | $\bar{\Delta}$
| (1) |  STL | 0.0 | 66.30 | 0.0 | 51.87 | 0.0 | 90.34 | 0.0 | - | - | 52.13 | 0.0
| (2) |  STL + TSA | +1.35 | 69.99 | +5.56 | 52.59 | +1.39 | 90.10 | -0.27 | - | - | 53.25 | +1.67
| (3) |  STL + XAA | +14.70 | 66.30 | 0.0 | 53.83 | +3.78 | 89.89 | -0.50 | 63.70 | 0.0 | 68.43 | +0.82
I (Sec. 4.2) | (4) |  STL + TSA + XAA (FAME-ViL(ST))  | +15.96 | 69.99 | +5.56 | 55.47 | +6.94 | 90.27 | -0.07 | 63.67 | -0.05 | 69.85 | +3.10
| (5) |  MTL | -70.43 | 57.65 | -13.05 | 49.57 | -4.43 | 85.95 | -4.86 | - | - | 48.29 | -5.59
| (6) |  MTL + TSA | -70.11 | 67.97 | +2.52 | 52.04 | +0.33 | 90.32 | -0.02 | - | - | 52.58 | +0.71
| (7) |  MTL + XAA | -67.65 | 65.87 | -0.65 | 52.59 | +1.39 | 90.93 | +0.65 | 60.99 | -4.25 | 67.60 | -0.72
II (Sec. 4.2) | (8) |  MTL + TSA + XAA (base MTL) | -67.33 | 69.31 | +4.54 | 55.41 | +6.82 | 90.84 | +0.55 | 65.17 | +2.31 | 70.18 | +3.56
| (9) |  base MTL + MTD (FAME-ViL) | -67.33 | 70.00 | +5.56 | 58.29 | +12.38 | 91.44 | +1.22 | 65.50 | +2.83 | 71.31 | +5.50
| (10) |  base MTL + MTD + Uniform | -67.33 | 67.70 | +2.11 | 57.31 | +10.49 | 91.36 | +1.13 | 65.12 | +2.23 | 70.37 | +3.99
| (11) |  base MTL + MTD + Round-robin | -67.33 | 67.79 | +2.25 | 57.47 | +10.80 | 91.35 | +1.12 | 64.87 | +1.84 | 70.37 | +4.00
| (12) |  base MTL + IAS [32] | -67.33 | 69.13 | +4.27 | 55.26 | +6.54 | 90.51 | +0.19 | 63.67 | -0.05 | 69.64 | +2.74
| (13) |  base MTL + MTD + IAS [32] | -67.33 | 70.11 | +5.75 | 57.97 | +11.76 | 90.88 | +0.60 | 65.66 | +3.08 | 71.16 | +5.30
| (14) |  base MTL + IMTLG [46] | -67.33 | 64.11 | -3.30 | 47.12 | -9.16 | 90.21 | -0.14 | 55.61 | -12.70 | 64.26 | -6.33
III (Sec. 4.3) | (15) |  base MTL + MTD + IMTLG [46] | -67.33 | 67.14 | +1.27 | 57.22 | +10.31 | 90.09 | -0.28 | 58.14 | -9.56 | 68.15 | +0.44
| (16) |  FAME-ViL (bottleneck dim. = 128) | -65.14 | 70.73 | +6.68 | 58.03 | +11.88 | 91.54 | +1.33 | 66.20 | +3.92 | 71.63 | +5.95
| (17) |  FAME-ViL (bottleneck dim. = 256) | -62.67 | 71.77 | +8.25 | 58.45 | +12.69 | 91.10 | +0.84 | 66.81 | +4.88 | 72.03 | +6.67
IV (Sec. 4.4) | (18) |  FAME-ViL (bottleneck dim. = 512) | -57.73 | 72.32 | +9.08 | 58.51 | +12.80 | 90.96 | +0.69 | 66.92 | +5.05 | 72.18 | +6.91

Table 4:  Ablation study and further analysis of our method. Groups (I) and
(II): Ablation experiments of the proposed XAA and TSA under the single-task
learning (STL) and multi-task learning (MTL) scenarios. Group (III): The
comparison among our multi-teacher distillation (MTD) and other alternatives
designed for task-sampling based MTL (TS-MTL). Group (IV): The effect of the
bottleneck dimension of XAA and TSA. Yellow background: The baseline
performance used per column; Red background: negative transfer; Green
background: positive transfer. Bold number: The best result in each group.

SCR evaluation. We report the performance of SCR in the left part of Tab. 3,
following the common protocol [21, 94, 24]. Similar to TGIR, our FAME-ViL
surpasses clearly all previous works [21, 94, 24, 43, 33] with heavier fusion
mechanisms (_e.g_., modality-agnostic self-attention implemented by
concatenating text tokens and image patches at the very beginning). This
validates the efficacy of our proposed XAA, suggesting the superiority of
modality interaction over the conventional fusion at the input point.

FIC evaluation. The original FashionViL [24] has no decoder and cannot support
generation tasks. For comparison, we equip it with masked language modelling
(MLM) auto-regressively [93, 43, 94] enabling the image captioning. The
results of FIC are shown in the right part of Tab. 3, following the common
protocol [94]. Our FAME-ViL again achieves state-of-the-art performance with a
clear margin.

All the above comparisons show the superior generalization capability of our
method in both generative and discriminative tasks.

###  4.2 Ablation study on architecture

Given the strong performance of our method as evaluated in Sec. 4.1, we ablate
the proposed model architecture with a focus on two newly introduced adapters
(TSA and XAA) in both STL and MTL settings.

Single-task learning setting. For comparison on XMR, TGIR and SCR, we design
the baseline as directly fine-tuning the vanilla CLIP without any new modules
(L1). With the two-stream design, CLIP cannot tackle FIC, and we hence further
equip it with our XAA as the baseline for FIC (L3). From the results shown in
group (I) of Tab. 4, we find that TSA and XAA can bring in 3%-6% relative
improvements for XMR and TGIR. In particular, XAA gives TGIR a significant
improvement, demonstrating the superiority of our layer-wise modality
interaction mechanism. However, these adapters have only a marginal impact on
the performance of SCR and FIC, with a performance drop of less than 0.5% when
the model is independently trained on a single task.

Multi-task learning setting. Similarly, we construct the baselines for the MTL
setting using the vanilla CLIP and XAA-equipped CLIP (L5 and L7). As shown in
L5 in the group (II) of Tab. 4, a severe negative transfer occurs with an
overall 5.59% performance drop. Likewise, there is also a negative transfer
for the XAA-equipped CLIP model (L7), albeit with a slight increase in
performance. This suggests the challenges of heterogeneous multi-task learning
in the fashion domain. This problem can be well solved using our TSA, with an
overall 4%$\sim$6% improvement (L5 _vs_. L7 and L6 _vs_. L8), even though only
a few extra task-specific parameters are introduced (1.35% of the original
CLIP size). Interestingly, we also found that XAA and TSA are reciprocal: (1)
When TSA and XAA work together, the model can achieve better relative
performance than the sum of their gains (L4 _vs_. L2+L3 and L8 _vs_. L6+L7)
(2) When TSA or XAA is applied in isolation, the multi-task model always
underperforms its single-task counterpart (L6 _vs_. L2 and L7 _vs_. L3). But
the multi-task model with both TSA and XAA exceeds the single-task counterpart
(L8 vs L4), indicating that TSA and XAA play complementary roles better in the
multi-task setting, as expected and designed so.

###  4.3 Ablation study on multi-task training strategy

Following the above architecture analysis, we further ablate the proposed
multi-teacher distillation (MTD) based training strategy. We compare
extensively with previous sampling strategies and gradient manipulation
algorithms.

Task sampling. We start by comparing two common sampling strategies (uniform
and round-robin) with our size-proportional strategy. Round-robin sampling is
a special case of uniform sampling – each task is sampled one by one. As shown
in L9-L11 in the group (III) of Tab. 4, both uniform and round-robin sampling
underperform our size-proportional sampling by a gap of 1.5%. This is due to
imbalanced dataset sizes across different tasks, which is ignored in uniform
sampling and round-robin sampling.

Gradient manipulation. To compare with our MTD scheme, we consider two kinds
of gradient manipulation algorithms: Implicit Adaptive Scheduling (IAS) [32]
and IMTLG [46]. In particular, IAS is a representative strong method that
adaptively changes the task sampling ratio, learning rate, or gradient scale
for each task [80, 51]. Specifically, it scales the gradients of each task
according to the performance on the validation set. Instead, IMTLG is a
representative of those methods manipulating all the gradients together [8,
63, 36, 56]. It is featured by a closed-form solution to optimize the scaling
factors of each task, such that the aggregated gradients (sum of raw gradients
weighted by the scaling factors) have equal projections onto individual tasks.
Since IMTLG cannot be directly applied to task-sampling based MTL, we further
adapt it by maintaining a gradient buffer to store the gradients of each task
and update the parameters every four iterations (each corresponding one of the
four fashion tasks). As shown in the group (III) of Tab. 4, the performance of
IAS (L12) and IMTLG (L14) are significantly lower than that of our MTD (L9).
In particular, IMTLG suffers from a severe negative transfer (-6.33%). There
are two plausible reasons: (1) Relying on a heuristic strategy, IAS struggles
in finding the optimal status over all tasks, despite the access to the
validation performance. (2) IMTLG may experience over-fitting for the tasks
with smaller training data (_e.g_., TGIR), which cannot be addressed by the
idea of ensuring the final gradient direction to have the same impact on each
task. On the contrary, our MTD can implicitly regularize the gradients via
knowledge distillation, without a costly need of monitoring the validation
performance. Guided by the soft ground truth of each teacher, overfitting can
be well avoided in an elegant manner. Considering the methodical
orthogonality, we further apply our MTD on top of IAS (L13) and IMTLG (L15).
It is shown that this can improve both by a large margin (L13 _vs_. L12 and
L15 _vs_. L14), demonstrating the generic usability of our training method.

Figure 5: Training dynamics of our multi-teacher distillation (MTD) and
alternative multi-task learning methods (IAS [32] and IMTLG [46]). Metric: The
validation performance curves.

###  4.4 Further analysis

Regularizing effect of MTD. To shed more light on the regularization effect of
MTD, we plot the validation performance curves in Fig. 5. Without MTD, the
baseline MTL model is prone to overfit on TGIR after about $20k$ iterations
due to much less training data than other tasks. Interestingly, this
overfitting is even amplified by IMTLG. This is because IMTLG needs to pay
more attention to TIRG in order to achieve impartial learning. Overall,
neither IAS nor IMTLG can improve over the baseline MTL, regardless of
overfitting or not. Encouragingly, our MTD yields consistent and significant
performance boost per task, rendering it a more stable and effective learning
strategy.

Scaling up bottleneck dimension. We evaluate the effect of the bottleneck
dimension of the AdaptMLP in XAA and TSA (the only hyper-parameter of our
architecture). We vary this dimension from $64$ to $512$. As shown in the
group (IV) of Tab. 4, it is evident that the overall relative performance is
positively correlated with this bottleneck dimension. This indicates that
FAME-ViL could be potentially more performing at the cost of more parameters.
Also, we observe a trade-off between model size increase and performance gain.
For example, 10% more parameters are needed for exchanging a relative
performance gain of 1.4% (L18 _vs_. L9). We also notice that further
improvement is not consistent over tasks. For instance, the performance of SCR
will gradually deteriorate with the increase of bottleneck dimension. An
interesting direction for future works could be exploiting adaptive algorithms
[69, 68] to optimize the best bottleneck dimension per task.

Qualitative examples. To reflect the output of FAME-ViL more intuitively, in
addition to Fig. 2, we show more illustrative outputs from FAME-ViL in the
supplementary file.

##  5 Conclusions

We have introduced FAME-ViL for heterogeneous fashion tasks, grounded upon a
generic off-the-shelf V+L model. It addresses cross-modal retrieval, text-
guided image retrieval, multi-modal classification, and image captioning in a
unified architecture. This is made possible by the proposed task-versatile
architecture with cross-attention adapters and task-specific adapters, and a
scalable multi-task training pipeline with multi-teacher distillation.
Extensive experiments showed that our FAME-ViL achieves new state-of-the-art
performance on all tasks with significantly fewer parameters.

##  Appendix A Implementation details

Model architecture | Vision encoder (VE) | CLIP (ViT-B/16) [60]
---|---|---
Language encoder (LE) | CLIP (ViT-B/16) [60]
Bottleneck dim. | 64
Data augmentation | Resize | (256, 256)
RandomCrop | (224, 224)
RandomHorizontalFlip | ✓
Training setting | Number of iterations | $90k$
Batch size | 64
Initial LR of VE/LE | 1e-6
Initial LR of Adapters | 1e-4
LR schedule | Multi-step
LR steps |  $50k$ and $80k$
LR decrease ratio | 0.1
Warmup iterations | $10k$
Warmup factor | 0.25
Optimizer | AdamW (0.9, 0.999)
Weight decay | 1e-5
Hardware | GPU | 4 $\times$ RTX 3090
Training duration | 31.5h

Table 5: Details for multi-task training FAME-ViL.

This section describes our implementation and multi-task training details for
FAME-ViL.

Architecture details. As mentioned in the main paper, we build our FAME-ViL
upon off-the-shelf CLIP model [60]. We utilize the ViT-B/16 version and get
the pre-trained weights from HuggingFace Transformers [82]. Specifically, as
described in the original paper [60], the language encoder is a 12-layer
512-wide Transformer [73] with 8 attention heads, while the vision encoder is
a base-size Vision Transformer (ViT) [17] with patch size as 16. Masked self-
attention was used in the language encoder. For computational efficiency, the
max text sequence length was capped at 76. The text sequence is bracketed with
[SOS] and [EOS] tokens and the activations of the highest layer of the
language encoder at the [EOS] token are treated as the text feature
representation. Please find more details about CLIP and its pre-training in
the original paper [60].

(a)

(b)

(c)

(d)

(e)

Figure 6: XMR examples. Green box indicates the ground truth.

Training details. We list all hyper-parameters used for multi-task training in
Tab. 5, including data augmentation methods, optimizer setting, scheduler
setting, and _etc_. This results in about 31.5 hours of training time on four
RTX 3090 GPUs (24GB memory for each). For single-task training (used by
single-task teachers training and ablation study), we adopted the same hyper-
parameters except for shorter training iterations ($30k$ for tasks on
FashionGen [61], $6k$ for tasks on FashionIQ [83]).

XMR* (Image to Text) | XMR* (Text to Image)
---|---
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
$45.99\pm 0.25$ | $73.25\pm 0.15$ | $81.84\pm 0.12$ | $53.12\pm 0.07$ | $77.55\pm 0.35$ | $86.02\pm 0.22$
TGIR (Dress) | TGIR (Shirt) | TGIR (Toptee)
R@10 | R@50 | R@10 | R@50 | R@10 | R@50
$42.16\pm 0.32$ | $67.10\pm 0.22$ | $47.19\pm 0.33$ | $67.91\pm 0.66$ | $50.79\pm 0.08$ | $73.48\pm 0.38$
SCR (Acc) | SCR (F1) | FIC (B) | FIC (M) | FIC (R) | FIC (C)
$94.62\pm 0.08$ | $87.80\pm 0.25$ | $30.59\pm 0.15$ | $24.22\pm 1.99$ | $55.69\pm 0.13$ | $148.9\pm 1.19$

Table 6: Statistical significance quantification of our results. * XMR
evaluation is under full database protocol

##  Appendix B Additional quantitative results

We followed the same protocol used by previous works [24] and used the same
random seed for training, to ensure a direct comparison to these main
competitors. We also trained our model two more times with different random
seeds to measure our method’s stability. The statistical performance (w/ mean
and std) over three trials in Tab. 6 shows that our model is stable.

##  Appendix C Additional qualitative results

(a)

(b)

(c)

(d)

(e)

Figure 7: TGIR examples. Blue box indicates the reference image. Green box
indicates the ground truth.

| Images |  Ground Truth Captions |  Generated Captions
---|---|---|---
(a) |  |  White logo tank top. Relaxed-fit tank top in white. Ribbed scoopneck collar and armscyes. Logo print at black. Tonal logo embroidered at back hem. Tonal stitching. |  White logo tank top. Racer-back tank top in white. Scoopneck collar. Logo printed at front in black. Curved hem. Tonal stitching.
(b) |  |  Black python print shirt. Short sleeve shirt in tones of grey and black. Detailed python scale print throughout with ombre effect at bottom portions. Spread collar. Button closure at front. Tonal stitching. Single-button barrel cuffs with buttoned sleeve placket. |  Black paint splatter shirt. Long sleeve shirt in black. Graphic print throughout in white. Spread collar. Button closure at front. Tonal stitching. Single-button barrel cuffs with buttoned sleeve placket.
(c) |  |  Black jersey leather trim lounge pants. Leather-trimmed stretch jersey lounge pants in black. Partially elasticized waistband with leather  drawstring closure. Zip fly. Leather pocket trim. Elasticized grosgrain cuffs. |  Black lounge pants. Lounge pants in black. Elasticised waistband with drawstring closure. Four-pocket styling. Elasticised ankle cuffs. Tonal stitching. Zip fly.
(d) |  |  Navy pixel print atari edition polo. Short sleeve oversized polo in navy. Atari pixel print at front. Spread collar with two-button placket. Slits at side seams. Tonal stitching. |  Navy embroidered patch polo. Short sleeve cotton piqu & eacute polo in navy. Ribbed spread collar and trim at sleeve opening. Five-button placket at front. Signature tri-color tab at back collar. Tennis tail hem. Tonal stitching.
(e) |  |  Green wrap pencil skirt. High-waisted wrap pencil skirt in green. Gathered knot detail at waist. Vent at front hem. Zip closure at back. Tonal stitching. |  Green silk draping skirt. Silk skirt in green. Elasticized waistband with drawstring at interior. Vented at back waist. Seam pockets at sides. Tonal stitching.

Table 7: FIC examples. Green text indicates the matched phrases.

(a)

(b)

Figure 8: Failure cases. Blue box indicates the reference image. Green box
indicates the ground truth.

We provide more visualization results in this section to better understand the
performance of our FAME-ViL in a qualitative way. Specifically, we show cross-
modal retrieval (XMR) results in Fig. 6, text-guided image retrieval (TGIR)
results in Fig. 7 and fashion image captioning (FIC) results in Tab. 7. We
didn’t show subcategory recognition (SCR) results here because of the lack of
intuition when visualizing this classification task, but the visualized
attention maps are given in Fig. 9.

For retrieval tasks (XMR and TGIR), we observe ambiguities (_i.e_., the ground
truth is not the only one matching the query) in the fashion datasets [83,
61]. Especially in FashionIQ, there are many false negatives that are
neglected during the data-annotation stage. Even so, our FAME-ViL can offer us
a reliable and human-understandable ranking list, demonstrating its
superiority in fine-grained discrimination. Fig. 8 shows example failure cases
from (a) XMR and (b) TGIR. In the text query example, we can see that even the
human-annotated ground truth (indicated by green boxes) images do not fit the
text query perfectly. In both failure cases, the top retrieved results, though
wrong according to the “ground truth”, are still largely aligned with the
query/modify text.

Because of the fine-grained nature of the fashion domain, the ground truth
captions in fashion contain much more fine-grained phrases than those in the
generic domain [24]. Despite this challenge, our FAME-ViL can produce concrete
and accurate phrases in the generated captions. Even if some of the generated
phrases do not exist in the ground truth, they still conform to the content of
the image and human intuition. This point proves the effectiveness of FAME-ViL
in fine-grained generation.

Figure 9: Visualized attention maps of SCR.

To gain a more intuitive understanding of how attention is learned in our
model, we visualize the text-to-image attention maps (the average over all
heads) in the last XAA of the STL baseline (first row) and our MTL model
(second row), as shown in Fig. 9. It is observed that the attention maps from
our model are more accurate and meaningful.

## References

  * [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [2] Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo.  Conditioned and composed image retrieval combining and partially fine-tuning clip-based features.  In CVPR workshops, 2022.
  * [3] Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo.  Effective conditioned and composed image retrieval combining clip-based features.  In CVPR, 2022.
  * [4] Satanjeev Banerjee and Alon Lavie.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In ACL workshops, 2005.
  * [5] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo.  Adaptformer: Adapting vision transformers for scalable visual recognition.  In NeurIPS, 2022.
  * [6] Yanbei Chen, Shaogang Gong, and Loris Bazzani.  Image search with text feedback by visiolinguistic attention learning.  In CVPR, 2020.
  * [7] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In ECCV, 2020.
  * [8] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich.  Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.  In ICML, 2018.
  * [9] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao.  Vision transformer adapter for dense predictions.  arXiv preprint arXiv:2205.08534, 2022.
  * [10] Patrick John Chia, Giuseppe Attanasio, Federico Bianchi, Silvia Terragni, Ana Rita Magalhães, Diogo Goncalves, Ciro Greco, and Jacopo Tagliabue.  Fashionclip: Connecting language and images for product representations.  arXiv preprint arXiv:2204.03972, 2022.
  * [11] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  In EMNLP, 2014.
  * [12] Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoc Le.  Bam! born-again multi-task networks for natural language understanding.  In ACL, 2019.
  * [13] Michael Crawshaw.  Multi-task learning with deep neural networks: A survey.  arXiv preprint arXiv:2009.09796, 2020.
  * [14] Ginger Delmas, Rafael S Rezende, Gabriela Csurka, and Diane Larlus.  Artemis: Attention-based retrieval with text-explicit matching and implicit similarity.  In ICLR, 2022.
  * [15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In NAACL-HLT, 2019.
  * [16] Eric Dodds, Jack Culpepper, and Gaurav Srivastava.  Training and challenging models for text-guided fashion image retrieval.  arXiv preprint arXiv:2204.11004, 2022.
  * [17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2020.
  * [18] Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, et al.  Coarse-to-fine vision-language pre-training with fusion in the backbone.  In NeurIPS, 2022.
  * [19] Nanyi Fei, Zhiwu Lu, Yizhao Gao, Guoxing Yang, Yuqi Huo, Jingyuan Wen, Haoyu Lu, Ruihua Song, Xin Gao, Tao Xiang, et al.  Towards artificial general intelligence via a multimodal foundation model.  Nature Communications, 2022.
  * [20] Robert M French.  Catastrophic forgetting in connectionist networks.  Trends in cognitive sciences, 1999.
  * [21] Dehong Gao, Linbo Jin, Ben Chen, Minghui Qiu, Peng Li, Yi Wei, Yi Hu, and Hao Wang.  Fashionbert: Text and image matching with adaptive loss for cross-modal retrieval.  In SIGIR, 2020.
  * [22] Sonam Goenka, Zhaoheng Zheng, Ayush Jaiswal, Rakesh Chada, Yue Wu, Varsha Hedau, and Pradeep Natarajan.  Fashionvlp: Vision language transformer for fashion retrieval with feedback.  In CVPR, 2022.
  * [23] Xiao Han, Sen He, Li Zhang, Yi-Zhe Song, and Tao Xiang.  Uigr: Unified interactive garment retrieval.  In CVPR workshops, 2022.
  * [24] Xiao Han, Licheng Yu, Xiatian Zhu, Li Zhang, Yi-Zhe Song, and Tao Xiang.  Fashionvil: Fashion-focused vision-and-language representation learning.  In ECCV, 2022.
  * [25] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  In ICLR, 2022.
  * [26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [27] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [28] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In ICML, 2019.
  * [29] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [30] Ronghang Hu and Amanpreet Singh.  Unit: Multimodal multitask learning with a unified transformer.  In ICCV, 2021.
  * [31] Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu.  Seeing out of the box: End-to-end pre-training for vision-language representation learning.  In CVPR, 2021.
  * [32] Sébastien Jean, Orhan Firat, and Melvin Johnson.  Adaptive scheduling for multi-task learning.  arXiv preprint arXiv:1909.06434, 2019.
  * [33] Ge-Peng Ji, Mingchen Zhuge, Dehong Gao, Deng-Ping Fan, Christos Sakaridis, and Luc Van Gool.  Masked vision-language transformer in fashion.  Machine Intelligence Research, 2022.
  * [34] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In ICML, 2021.
  * [35] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In ECCV, 2022.
  * [36] Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In CVPR, 2018.
  * [37] Jongseok Kim, Youngjae Yu, Hoeseong Kim, and Gunhee Kim.  Dual compositional learning in interactive image retrieval.  In AAAI, 2021.
  * [38] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In ICML, 2021.
  * [39] Seungmin Lee, Dongwan Kim, and Bohyung Han.  Cosmo: Content-style modulation for image retrieval with text feedback.  In CVPR, 2021.
  * [40] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In ICML, 2022.
  * [41] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  In NeurIPS, 2021.
  * [42] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [43] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In ECCV, 2020.
  * [44] Lizi Liao, Xiangnan He, Bo Zhao, Chong-Wah Ngo, and Tat-Seng Chua.  Interpretable multimodal retrieval for fashion products.  In ACM MM, 2018.
  * [45] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In ACL workshops, 2004.
  * [46] Liyang Liu, Yi Li, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang.  Towards impartial multi-task learning.  In ICLR, 2020.
  * [47] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  arXiv preprint arXiv:2107.13586, 2021.
  * [48] Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould.  Image retrieval on real-life images with pre-trained vision-and-language models.  In ICCV, 2021.
  * [49] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In NeurIPS, 2019.
  * [50] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi.  Unified-io: A unified model for vision, language, and multi-modal tasks.  arXiv preprint arXiv:2206.08916, 2022.
  * [51] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee.  12-in-1: Multi-task vision and language representation learning.  In CVPR, 2020.
  * [52] Haoyu Ma, Handong Zhao, Zhe Lin, Ajinkya Kale, Zhangyang Wang, Tong Yu, Jiuxiang Gu, Sunav Choudhary, and Xiaohui Xie.  Ei-clip: Entity-aware interventional contrastive learning for e-commerce cross-modal retrieval.  In CVPR, 2022.
  * [53] Yihui Ma, Jia Jia, Suping Zhou, Jingtian Fu, Yejun Liu, and Zijian Tong.  Towards better understanding the clothing fashion styles: A multimodal deep learning approach.  In AAAI, 2017.
  * [54] Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa.  Unipelt: A unified framework for parameter-efficient language model tuning.  In ACL, 2022.
  * [55] Suvir Mirchandani, Licheng Yu, Mengjiao Wang, Animesh Sinha, Wenwen Jiang, Tao Xiang, and Ning Zhang.  Fad-vlp: Fashion vision-and-language pre-training towards unified retrieval and captioning.  In EMNLP, 2022.
  * [56] Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya.  Multi-task learning as a bargaining game.  In ICML, 2022.
  * [57] Duy-Kien Nguyen and Takayuki Okatani.  Multi-task learning of hierarchical vision-language representation.  In CVPR, 2019.
  * [58] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In ACL, 2002.
  * [59] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In NeurIPS, 2019.
  * [60] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [61] Negar Rostamzadeh, Seyedarian Hosseini, Thomas Boquet, Wojciech Stokowiec, Ying Zhang, Christian Jauvin, and Chris Pal.  Fashion-gen: The generative fashion dataset and challenge.  arXiv preprint arXiv:1806.08317, 2018.
  * [62] Victor Sanh, Thomas Wolf, and Sebastian Ruder.  A hierarchical multi-task approach for learning embeddings from semantic tasks.  In AAAI, 2019.
  * [63] Ozan Sener and Vladlen Koltun.  Multi-task learning as multi-objective optimization.  In NeurIPS, 2018.
  * [64] Minchul Shin, Yoonjae Cho, Byungsoo Ko, and Geonmo Gu.  Rtic: Residual learning for text and image composition using graph convolutional network.  arXiv preprint arXiv:2104.03015, 2021.
  * [65] Amanpreet Singh, Vedanuj Goswami, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh.  Mmf: A multimodal framework for vision and language research.  https://github.com/facebookresearch/mmf, 2020.
  * [66] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.  Flava: A foundational language and vision alignment model.  In CVPR, 2022.
  * [67] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  VL-BERT: pre-training of generic visual-linguistic representations.  In ICLR, 2020.
  * [68] Benyuan Sun, Jin Dai, Zihao Liang, Congying Liu, Yi Yang, and Bo Bai.  Gppf: A general perception pre-training framework via sparsely activated multi-task learning.  arXiv preprint arXiv:2208.02148, 2022.
  * [69] Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko.  Adashare: Learning what to share for efficient deep multi-task learning.  In NeurIPS, 2020.
  * [70] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal.  Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks.  In CVPR, 2022.
  * [71] Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  In NeurIPS, 2014.
  * [72] Hao Tan and Mohit Bansal.  LXMERT: Learning cross-modality encoder representations from transformers.  In EMNLP-IJCNLP, 2019.
  * [73] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [74] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In CVPR, 2015.
  * [75] Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays.  Composing Text and Image for Image Retrieval - an Empirical Odyssey.  In CVPR, 2019.
  * [76] Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang.  Ufo: A unified transformer for vision-language representation learning.  arXiv preprint arXiv:2111.10023, 2021.
  * [77] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  In ICML, 2022.
  * [78] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  arXiv preprint arXiv:2208.10442, 2022.
  * [79] Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei.  Vlmo: Unified vision-language pre-training with mixture-of-modality-experts.  arXiv preprint arXiv:2111.02358, 2021.
  * [80] Weiyao Wang, Du Tran, and Matt Feiszli.  What makes training multi-modal classification networks hard?  In CVPR, 2020.
  * [81] Ronald J Williams and David Zipser.  A learning algorithm for continually running fully recurrent neural networks.  Neural computation, 1989.
  * [82] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  Transformers: State-of-the-art natural language processing.  In EMNLP, 2020.
  * [83] Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris.  Fashion iq: A new dataset towards retrieving images by natural language feedback.  In CVPR, 2021.
  * [84] Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, and Nan Duan.  Bridge-tower: Building bridges between encoders in vision-language representation learning.  arXiv preprint arXiv:2206.08657, 2022.
  * [85] Xuewen Yang, Heming Zhang, Di Jin, Yingru Liu, Chi-Hao Wu, Jianchao Tan, Dongliang Xie, Jue Wang, and Xin Wang.  Fashion captioning: Towards generating accurate descriptions with semantic rewards.  In ECCV, 2020.
  * [86] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  arXiv preprint arXiv:2205.01917, 2022.
  * [87] Licheng Yu, Jun Chen, Animesh Sinha, Mengjiao MJ Wang, Hugo Chen, Tamara L Berg, and Ning Zhang.  Commercemm: Large-scale commerce multimodal representation learning with omni retrieval.  In KDD, 2022.
  * [88] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Revisiting visual representations in vision-language models.  In CVPR, 2021.
  * [89] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu.  Neural prompt search.  arXiv preprint arXiv:2206.04673, 2022.
  * [90] Yida Zhao, Yuqing Song, and Qin Jin.  Progressive learning for image retrieval with hybrid-modality queries.  In SIGIR, 2022.
  * [91] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In CVPR, 2022.
  * [92] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  IJCV, 2022.
  * [93] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao.  Unified vision-language pre-training for image captioning and vqa.  In AAAI, 2020.
  * [94] Mingchen Zhuge, Dehong Gao, Deng-Ping Fan, Linbo Jin, Ben Chen, Haoming Zhou, Minghui Qiu, and Ling Shao.  Kaleido-bert: Vision-language pre-training on fashion domain.  In CVPR, 2021.