# Compressing Visual-linguistic Model via Knowledge Distillation

Zhiyuan Fang† , Jianfeng Wang‡, Xiaowei Hu‡, Lijuan Wang‡, Yezhou Yang†,
Zicheng Liu‡
†Arizona State University, ‡Microsoft Corporation
{zy.fang<EMAIL_ADDRESS>{jianfw, lijuanw, leizhang<EMAIL_ADDRESS>

###### Abstract

Despite exciting progress in pre-training for visual-linguistic (VL)
representations, very few aspire to a small VL model. In this paper, we study
knowledge distillation (KD) to effectively compress a transformer based large
VL model into a small VL model. The major challenge arises from the
inconsistent regional visual tokens extracted from different detectors of
Teacher and Student, resulting in the misalignment of hidden representations
and attention distributions. To address the problem, we retrain and adapt the
Teacher by using the same region proposals from Student’s detector while the
features are from Teacher’s own object detector. With aligned network inputs,
the adapted Teacher is capable of transferring the knowledge through the
intermediate representations. Specifically, we use the mean square error loss
to mimic the attention distribution inside the transformer block, and present
a token-wise noise contrastive loss to align the hidden state by contrasting
with negative representations stored in a sample queue. To this end, we show
that our proposed distillation significantly improves the performance of small
VL models on image captioning and visual question answering tasks. It reaches
$120.8$ in CIDEr score on COCO captioning, an improvement of $5.1$ over its
non-distilled counterpart; and an accuracy of $69.8$ on VQA 2.0, a $0.8$ gain
from the baseline. Our extensive experiments and ablations confirm the
effectiveness of VL distillation in both pre-training and fine-tuning stages.

##  1 Introduction

There have been exciting progress in visual linguistic (VL) pre-training to
learn omni-representation models [39, 55, 8, 59, 74, 37] which could benefit a
number of downstream tasks (i.e. , image captioning, VQA, image retrieval,
etc.). The success can largely be attributed to the self-attention-based [63]
transformer architecture, e.g. , BERT [13], which is effective in learning
from image-text pairs at scale. So far, much of the work has focused on large
models that suffer from high latency and large memory footprints at the time
of inference, which limits their deployment to resource constrained edge
devices for real-world applications.

Figure 1: Overview of our proposed VL distillation schema. The VL model
typically contains a region feature extraction module and a multi-modal
transformer module. To have an aligned input, we adapt the Teacher VL model
based on the region proposals from Student’s region feature extractor. The VL
distillation is then performed in both the pre-training stage and the fine-
tuning stage.

As one of the effective techniques to compress large models, knowledge
distillation (KD) [19, 5] was proposed by injecting the knowledge from a
strong Teacher model into a smaller Student model without losing too much
generalization power. Typically, the knowledge is transferred though mimicking
the output logit [19, 52, 14], reducing the divergence of feature maps [72,
21, 71], or learning the intermediate layer representations [30, 1], _etc_.

In recent years, KD has been proven effective in compressing language models.
For instance, Kim et al. [29] adopt KD for sequential model compression. In
the transformer based language model, DistillBERT [52] reduces the size of the
BERT-base model by 40% using a cosine embedding loss on the basis of hidden
embedding in the transformer block, and a soft-target probability loss.
TinyBERT [28], MobileBERT [58] and MiniLM [66] further highlight the
importance of minimizing the self-attention distributions across Teacher and
Student networks. In particular, [9] visually shows that attention maps in
BERT capture substantial linguistic knowledge and syntactic relations that
provide critical information during the distillation [28].

Heretofore, these advances have not been carried over to VL model compression.
We identify the major challenges that prevent us from applying these
techniques directly to VL distillation: Most existing VLP works [74, 37] use
pre-trained object detector (e.g. , Faster-RCNN [50]) to extract regional
features as visual tokens then feed them into the multi-modal transformer
network for VL pre-training. A smaller VL model usually uses a lightweight
detector for faster inference (e.g. , EfficientNet [60] based detector is
adopted in [65] as visual feature extractor) that may be different from
Teacher’s detector. The object proposals from the two different detectors are
usually very different, and there is no easy way to obtain the semantic
correspondence between the two sets of object proposals. It is therefore
unable to align the attention distributions or hidden embeddings between
Student and Teacher.

To address the aforementioned challenges, we propose a set of strategies to
enable distillation of VL models. First, instead of using object proposals
from two different detectors, we use the same set of object proposals,
obtained from Student’s lightweight detector for the visual token extraction
of both Teacher and Student (as shown in Figure 2). This ensures the semantic
correspondence between the Teacher and Student’s visual tokens. Second, we use
a loss term to have the Student to mimic the Teacher’s self-attention
distribution at the last transformer layer. Third, We further distill the
knowledge from the outputs of the transformer layers (i.e. , the hidden
embeddings). We find that simply learning from the layer-wise Teacher
embedding does not provide adequate supervision for the distillation. Hence,
we use a noise contrastive loss to align the token embeddings by contrasting
them with randomly sampled negative embeddings that are held in a sample
queue. Figure 1 gives an overview of our proposed VL distillation schema,
where VL distillation is applied for both the pre-training and fine-tuning
stages. In order to examine the effectiveness of our VL distillation, we
choose the same compact transformer architecture used in [66, 65], and the
lightweight object detector as in [65], but leverages knowledge distillation
techniques to facilitate the training of the small VL model (dubbed as
DistillVLM ). We show that our DistillVLM achieves a comparable performance to
a large VL model, and clearly outperforms its non-distilled counterpart [65].

To summarize our contributions:

  * •

For the first time, we propose VL distillation, a technique that leverages
knowledge distillation to facilitate training of smaller VL models.

  * •

Compared to non-distilled VL model pre-training, VL distillation offers a
significant boosting in performance for VL tasks such as image captioning and
visual question answering: DistillVLM achieves 120.8 in CIDEr score on COCO
captioning [38] and 69.8 in accuracy on VQA [16] tasks, which are $5.1$ more
or $0.8$ higher than the VL pre-training baselines.

  * •

We provide extensive ablations of DistillVLM , and systematically analyze the
effect of various KD strategies. This provides insights for future research on
VL model distillation.

Figure 2:  Illustration of our proposed DistillVLM architecture. The
lightweight detector extracts the region features, and the region proposals
are injected into the strong detector so that the region features are aligned
between Teacher and Student. The Teacher transformer network is adapted with
the new input before distillation. The Student VLM is distilled based on the
hidden embedding matching and attention distribution alignment.

##  2 Related Work

Visual-linguistic Pre-training. Following the prominent progress in the
transformer-based [63] pre-training in natural language [13, 47, 32, 4, 10,
48], visual-linguistic pre-training models, either for image+text [39, 59, 8,
37, 20, 73, 36, 15, 35, 40] or for video+text [56, 35, 41, 75, 33], have
achieved great success on a number of downstream V+L tasks. Most existing VL
models are designed in a two-step fashion: a pre-trained object detector is
used to encode the image as set of regional features (as offline visual
tokens) followed by pre-training on a large scale visual-linguistic corpus
using tasks like masked language modeling, image-text matching or masked
region modeling losses. In particular, Zhang et al. [73] demonstrate the
significant role of visual features in VL pre-training and looks for more
effective visual representations from a larger object detector. Li et al. [37]
shows that a larger transformer VL model can learn better from larger VL
corpus. However, the marginal costs are greater than the marginal benefits.
Recently, Wang et al. [65] propose a small VL model called MiniVLM that uses a
lightweight visual feature extractor and smaller transformer to reduce the
model size by 73% and maintain good accuracy on VL tasks. Nevertheless, the
cost of pre-training on MiniVLM is associated with sub-optimal efficiency: it
requires a large amount of training data (14M) to learn a good representation.
Thus, it is worth exploring a more efficient way to train small VL models.
There are other lines of VL pre-training works in which grid features [23, 26]
are extracted from the convolutional layers without the proposal computation.
[49, 46, 12] learn visual representation from scratch using Convolutional
Neural Network as image encoder with a transformer for VL pre-training on a
large amount of image-text pairs. The notion of VL distillation is not limited
to just the two-stage VL models, it can potentially benefit other types of
transformer based VL models as well.

Knowledge Distillation has been applied to model compression task across
different domains with its main goal being to transfer the “knowledge”
$f(x_{i})$ of sample ($x_{i},y_{i}$) from a strong Teacher network ($T$) to
the Student network ($S$) by minimizing the divergence between them:

$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\bigg{(}\mathcal{L}_{\text{S}}(x_{i},y_{i})+\mathcal{L}_{\text{KD}}\Big{(}f^{S}(x_{i}),f^{T}(x_{i})\Big{)}\bigg{)},$
(1)

where $\mathcal{L}_{\text{S}}(\cdot)$ refers to the original supervision
signal(s) on the Student. In practice, this term can possibly be replaced by
the exclusive use of $L_{\text{KD}}$. Depending on the type of knowledge
transferred, $\mathcal{L}_{\text{KD}}$ can derive from soft cross-entropy,
mean squared error (MSE) function or KL-divergence. For example, [19, 5]
transfer the learned knowledge by mimicking the mass function of the output
probability across classes, or by minimizing the divergence of intermediate
features [71, 30, 22, 70, 67]. [62, 62, 14] propose contrastive distillation
for visual representation learning. In addition, remarkable advances have been
made in knowledge distillation for language model compression (i.e. , BERT
[13]), and these works show that mimicking the distribution of self-attention
and intermediate representations of transformer blocks increases performances
[52, 27, 58, 69] for downstream tasks. In particular, in the transformer-based
language model distillation, DistillBERT [52] proposes to train the small BERT
by mimicking the Teacher’s output probability of masked language prediction
and the embedding features. TinyBERT [27] and MobileBERT [58] leverage the
layer-wise attention distributions for distillation with MSE function. [66]
suggests distilling on the last transformer layer and bringing extra
flexibility for training. [57, 7] also use the contrastive distillation in
transformer based language model compression. [14, 57] propose using a sample
queue to store history embeddings and show that contrasting with more negative
samples is beneficial for knowledge distillation.

##  3 Visual-linguistic Knowledge Distillation

Compared to knowledge distillation in language models, VL knowledge
distillation requires knowledge transferring from Teacher to Student in both
modalities. We present DistillVLM for the task of visual-linguistic
distillation (the overall architecture is illustrated in Figure 2), together
with the detailed strategies for our model training.

###  3.1 Visual Token Alignment

VL pre-training methods such as OSCAR [37] take as input an image-text pair in
the format of Word-Tag-Image triple ($\boldsymbol{w}$, $\boldsymbol{q}$,
$\boldsymbol{v}$), where $\boldsymbol{w}$ and $\boldsymbol{q}$ denote the
sequence of caption embedding and the word embedding of detected object tags
(in texts). To obtain the visual tokens $\boldsymbol{v}$ and object tags, a
set of image regional vectors are extracted from an object detector. A Faster
R-CNN [51] detector pre-trained on Visual Genome [31] is used to extract the
visual feature vector of each region, which is concatenated with its regional
position coordinates to form a positional-sensitive region feature vector.
This vector is then fed into a linear projection to ensure that the final
vector $\boldsymbol{v}$ has the same dimension as the caption/tag embedding.
The VL pre-training can be seen as a semantic alignment process between the
image regions and the textual units. It is worth mentioning that, the top
regions to be extracted from the image is dependent on their associated
confidence score output by the detector [65], which leads to some over-sampled
and noisy visual tokens. Typically, the order of visual tokens is specified in
descending order using the confidence score. As an alternative to Faster-RCNN,
MiniVLM [65] uses a lightweight detector (i.e. , TEE) in which the backbone is
replaced with EfficientNet [60] and a BiFPN [61] module is added to generate
multi-scale features. These strategies obviously accelerate inference process,
but inevitably also lead to different visual tokens between the Teacher and
Student networks during distillation. For this reason, the direct application
of the distillation loss to attention matrices or hidden representations leads
to an invalid transfer of knowledge. Hence, we extract and align the
Teacher/Student’s visual tokens by using the same set of detected bounding
boxes recognized by the lightweight detector, and keep the same token orders
based on their confidence scores (as in Figure 2). Both the Teacher and
Student VLM use the same object tags from the lightweight detector during the
distillation. Having the Teacher use the visual tokens extracted by proposals
from the lightweight detector may result in small performance drop. In
practice, we address this issue by fine-tuning/re-training the Teacher VLM
using the new visual tokens (Teacher adaptation).

###  3.2 Attention Distribution Distillation

One critical component of the transformer block is the multi-head self-
attention module [63]. which enables contextualized information to be captured
from an input sequence. A multi-head attention module outputs a set of
attended values:

$\texttt{Attention(}\mathbf{Q},\mathbf{K},\mathbf{V}\text{)}=\texttt{softmax(}\frac{\mathbf{Q}\cdot\mathbf{K}}{\sqrt{d_{k}}}\text{)}\cdot\mathbf{V},$
(2)

where $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ denote query, key and value
that are retrieved after three independent linear transformations on the
hidden embedding $\mathbf{H}_{i}$ from $i$-th transformer block, and $d_{k}$
is the dimension of key as a scaling factor. The dot-product between key and
query after the softmax normalization is the attention matrix:

$\mathbf{A}=\texttt{softmax(}{\mathbf{Q}\cdot\mathbf{K}}/{\sqrt{d_{k}}}\text{)}.$
(3)

Each transformer block consists of a set of consecutive linear
transformations, which include one multi-head attention module, a two-layer
feed forward network, followed by a normalization layer, and finally a
residual connection.

Previous attempts in language model distillation [27, 58] have demonstrated
the importance of transferring self-attention matrices, that are believed to
contain latent linguistic information, e.g. , syntactic and co-reference
relation of input tokens [9, 25]. [66] shows that using just the last
transformer block’s attention map yields equivalent results, allowing the
Teacher and Student to have a different number of layers. In the case of the
VL pre-training task, Cao et al. [6] show that certain attention matrices of
the pre-trained VL models contain extensive intra-modal and cross-modal co-
reference relations. These visual-linguistic knowledge is implicitly encoded,
but shows a very promising potential for VL distillation. We formulate the
distillation loss of the attention distribution by minimizing the divergence
between the self-attention matrices of the last layer of the Teacher and the
Student:

$\mathcal{L}_{\text{ATT}}=\frac{1}{T\\!\cdot\\!H}\sum_{i=1}^{T}\sum_{j=1}^{H}\texttt{MSE}(\mathbf{A}^{S}_{i,j},\mathbf{A}^{T}_{i,j}),$
(4)

where $T$, $H$ denote the number of tokens and attention heads in a
transformer. $\mathbf{A}_{i,j}$ is the normalized attention for $i$-th token
at $j$-th head. We further study the effects of the distillation over the
attention distribution in ablations.

###  3.3 Hidden Representation Distillation

Similar to previous works [27, 58], we also use the hidden representations for
the Teacher and Student alignment during distillation. In particular, previous
efforts formulate the task as minimizing the divergence of the hidden
embedding ($\mathbf{H}\in\mathbb{R}^{T\times d}$) of every Transformer block,
whose objective is as follows:

$\mathcal{L}_{\text{HID-
MSE}}=\frac{1}{T\\!\cdot\\!L}\sum_{i=1}^{T}\sum_{j=1}^{L}\texttt{MSE}(\mathbf{H}^{S}_{i,j}\mathbf{W}_{h},\mathbf{H}^{T}_{i,j}),$
(5)

and $L$ stands for the number of transformer blocks. $\mathbf{W}_{h}$ is a
learnable linear transformation that maps the Student hidden embedding into
the identical dimension of Teacher embedding. However, there are limitations
for such layer-to-layer alignment method. For example, TinyBERT must employ a
uniform-function mapping to selectively choose a subset of the layers for
learning, and MobileBERT requires the Teacher and Student to have identical
number of layers. Since visual tokens are noisy during the VL distillation,
this also leads to an increased difficulty in alignment. Sun et al. [57]
propose CoDIR, which takes advantage of the noise contrastive estimation (NCE)
loss to align the Teacher & Student’s hidden representations by contrasting
the target instance ($\mathbf{h}^{S}$) with more random instances as negative
samples and aligning with its positive sample ($\mathbf{h}^{T}$),
$\mathbf{h}\in\mathbb{R}^{d_{T}}$. Following [17, 14, 57], we employ a pre-
defined instance queue
$[\mathbf{h}_{0}^{T},\mathbf{h}_{1}^{T}\cdots\mathbf{h}_{K}^{T}]$ to store $K$
random sampled embeddings and one positive embedding from the Teacher network.
And the objective of NCE is as:

$\mathcal{L}_{\text{HID}}=-\text{log}\frac{\text{exp}(\mathbf{h}^{S}_{i}\cdot\mathbf{h}^{T}_{i}/\tau)}{\sum_{j=0}^{K}\text{exp}(\mathbf{h}^{S}_{i}\cdot\mathbf{h}_{j}/\tau)},$
(6)

where $\tau$ denotes the temperature hyper-parameter, $\langle\cdot\rangle$ is
the cosine similarity function. There are different ways to retrieve hidden
representations $\mathbf{h}$, e.g. , [57] uses mean-pooled token
representations as layer-wise summarized embedding. We find that applying the
NCE loss to token-wise embedding leads to better distillation results, as
discussed in Section 3. A linear mapping is introduced for the identical
dimension transformation:
$\phi:\mathbb{R}^{d_{S}}\rightarrow\mathbb{R}^{d_{T}}$ ($d_{S}$, $d_{T}$
denote the hidden embedding dimension for Student and Teacher networks). To
update the instance queue, we en-queue the Teacher-derived representation of
the current batch ($\mathbf{h}^{T}$) and de-queue the earliest stored samples
after the iteration. The introduction of the queue design enables batch-size
independent distillation and allows the comparison with more contrastive
samples with limited computational resources. In ablations, we discuss the
effect of enlarging queue size and other distillation methods. In contrast to
[17, 57], we store representations from the pre-trained and frozen Teacher
network in the sample queue, which remain constant during training. This frees
us from the use of momentum encoder like in [14].

###  3.4 Classification Distillation

The losses mentioned above allow the task-agnostic distillation during the
pre-training stage. In addition, in the fine-tuning stage, we carry out
knowledge distillation that benefits certain VL downstream tasks.
Specifically, most VL downstream tasks are classification based tasks with
labels, e.g. , image captioning or VQA tasks. Continuing the distillation at
the downstream alleviates the domain gap brought by different pre-training VL
corpus. As in [19], we minimize the softmax prediction of Student and Teacher
networks and the loss is measured by the cross-entropy:

$\mathcal{L}_{\text{CLS}}=\texttt{CE}(\mathbf{z}^{S}/\tau_{d},\mathbf{z}^{T}/\tau_{d}),$
(7)

where $\tau_{d}$ refers to the temperature parameter, and we simply maintain
it as a constant $\mathbbm{1}$. $\mathbf{z}^{S}/\mathbf{z}^{T}$ are the soft
label outputs from Student/Teacher network.

###  3.5 Training

For the training, we keep the original VL pre-training objective losses
($\mathcal{L}_{\text{VLP}}$) [39] which consist of: masked language modeling
loss ($\mathcal{L}_{\text{MLM}}$), where 15% of the textual tokens are masked
and replaced with a special token [MASK] and the VL model is expected to
classify these tokens; Image-text (contrastive) matching (ITM) loss
($\mathcal{L}_{\text{ITM}}$) where the model is expected to predict whether
the image-text pair matches. Our final total loss on distillation at the pre-
training stage is the combination of the above:

$\mathcal{L}=\mathcal{L}_{\text{VLP}}+\alpha\mathcal{L}_{\text{ATT}}+\beta\mathcal{L}_{\text{HID}},$
(8)

where $\alpha$ are $\beta$ are the weights of the loss terms. We find that
$\mathcal{L}_{\text{CLS}}$ does not obviously contribute to the pre-training
stage so we simply apply it at the fine-tuning distillation stage as:

$\mathcal{L}=\mathcal{L}_{\text{CE}}+\mathcal{L}_{\text{CLS}}+\alpha\mathcal{L}_{\text{ATT}}+\beta\mathcal{L}_{\text{HID}},$
(9)

where $\mathcal{L}_{\text{CE}}$ is the original classification task in the
specfic downstream-task. We study the effects of different learning losses in
our ablations.

Method | # Param | # I-T Pairs | Visual Feat. | P. D. | F. D. | COCO Captioning | VQA
---|---|---|---|---|---|---|---
B@4 | M | C | S | test-std | test-dev
UVLP [74] |  $111.7$M |  $3$M | ResNeXt101 | ✗ | ✗ | $36.5$ | $28.4$ | $116.9$ | $21.2$ | $70.7$ | $-$
OSCAR${}_{\text{B}}$ [37] |  $111.7$M |  $7$M | R101-F | ✗ | ✗ | $36.5$ | $30.3$ | $123.7$ | $23.1$ | $73.4$ | $73.2$
MiniVLM [65] |  $34.5$M |  $7$M | TEE | ✗ | ✗ | $34.3$ | $28.1$ | $116.7$ | $21.3$ | - | -
MiniVLM [65] |  $34.5$M |  $14$M | TEE | ✗ | ✗ | $35.6$ | $28.6$ | $119.8$ | $21.6$ | $69.4$ | $69.1$
DistillVLM | $34.5$M | $7$M | TEE | ✗ | ✗ | $34.0$ | $28.0$ | $115.7$ | $21.1$ | 69.0 | 68.8
✗ | ✓ |  gray!15$34.5$ | $28.2$ |  gray!20$117.1$ | $21.5$ |  gray!10$69.2$ |  gray!10$69.0$
✓ | ✗ |  gray!25$35.2$ | $28.6$ |  gray!25$120.1$ | $21.9$ |  gray!25$69.7$ |  gray!25$69.6$
✓ | ✓ |  gray!35$35.6$ | $28.7$ |  gray!35$120.8$ | $22.1$ |  gray!25$69.8$ |  gray!25$69.6$
Table 1: DistillVLM distills from stronger VL model (as Teacher), and retains
high accuracy on COCO captioning task under different evaluating metrics,
regardless of the effect brought by the lightweight visual feature extractor
(TEE v.s. R101-F). Our model shows competitive results comparing to MiniVLM
[65], even only half of the image-text pairs (# I-T Pairs) are available for
pre-training. The VL distillation strategy brings consistent improvement in
both the pre-training stage (P.D.) and fine-tuning stage (F.D.). All
captioning methods are shown with cross-entropy optimization.

##  4 Experiments

In this section, we conduct extensive experiments on VL distillation both in
pre-training and fine-tuning stages. To evaluate the effectiveness of our
proposed distillation schema, we provide results and ablations for the image
captioning and VQA tasks.

###  4.1 Datasets

Following [37], we construct our VL pre-training dataset by combining multiple
existing VL datasets. Specifically, we use Conceptual Captions (CC) [54], SBU
captions [42], training splits of Flicker30k [45], GQA [24], COCO Captions
[38], and VQA-2.0 [16], yielding 4 million unique images, and 7 million image-
text pairs (VL-7M). Both our Teacher model and DistillVLM are pre-trained on
VL-7M and are then transferred to downstream VL tasks: image captioning on
COCO Captions and visual question answering on VQA-2.0. We follow Karpathy’s
split111https://github.com/karpathy/neuraltalk2 and have $\sim$11k images for
training, and $5$k/$5$k images for validation/testing. For the VQA task, we
conduct downstream fine-tuning and testing on VQA-2.0 dataset, which consists
of $83$k images/$444$k questions for training, $41$k images/$214$k questions
for validation. For a fair comparison with previous works, we report results
on test-std and test-dev splits via the online evaluation
server222https://visualqa.org/challenge.html, and compare ablation results
using test-dev split.

###  4.2 Implementation Details

Visual Representation. Earlier VL pre-training (VLP) works mostly use Faster
R-CNN [3, 50] or even advanced architecture [68, 74] for visual region
representation extraction. To obtain visual tokens with more semantics, the
object detector for VLP is usually pre-trained on Visual Genome Dataset [31],
which contains $1,600$ object and 500 attribute categories. Following MiniVLM
[65], we also adopt the EfficientNet [61] based lightweight object detector
(TEE) for visual feature extraction. TEE reduces $90\%$ of total inference
time and has $91\%$ fewer parameters ($86.9$M for R101-F _vs_. $7.5$M for
TEE). Same as MiniVLM, we also pre-train the TEE detector on Object365 [53]
and Visual Genome [31] datasets before the visual representation extraction.
We use R101 [18] based Faster-RCNN and TEE detected proposals for Teacher’s
regional visual representation extraction. This guarantees the semantic
correspondence of the input tokens between Teacher and Student. Prevailing VL
pre-training method like [37] shows that applying object tags in VL pre-
training contributes to the performances. During distillation, we use
consistent object tags detected by TEE for both the Teacher and Student
networks. The lengths for object tags and visual tokens are $15$ and $50$,
respectively.

VL Pre-training&Distillation. We use a compact transformer architecture for
the VLP and VL distillation. In particularly, we follow [66, 65] and adopt a
$12$-layer transformer with $12$ attention heads and $384$ hidden dimension.
For the Teacher model, we use Oscarb [37], a $12$-layer transformer with $12$
attention heads and $768$ hidden size, pre-trained on the VL-7M corpus for 1M
steps ($100$ epochs), with learning rate $5e^{-5}$ and batch size $768$, using
AdamW optimizer.333https://github.com/microsoft/Oscar Overall, our compact
transformer uses the same architecture as MiniVLM [65], and it has $34.5$M
learnable parameters and is 70% less than Oscar${}_{\text{b}}$. For VL
distillation, we first adapt the Teacher VLM by re-training it using the new
visual tokens. Then, we keep the Teacher model frozen without further updating
throughout the VL distillation. In contrast to [74, 37], weights in DistillVLM
are randomly initialized without inheriting weights from BERT [13]. We adopt a
learning rate at $2e^{-4}$ with batch size 768 for pre-training/distillation.
We report and compare the effect of VL distillation with previous VLP
baselines in Table 1. We set $\tau=\tau_{d}=1$ and $\alpha=10$, $\beta=10$.
Similar results are observed when using different values. We set the queue
size to $4,096$ and further study the effect of different hyper-parameters in
ablations.

Transferring to Downstream Tasks. In order to validate the efficacy of our
proposed VL distillation schema, we transfer the pre-trained model to VL
downstream tasks. Image captioning and VQA task can be formulated as a typical
classification task, which enables direct task-specific distillation and
comparisons in the downstream. We mainly examine them in this work, while the
VL distillation is not task-specific and can be extended to other VL tasks as
well. We conduct downstream distillation by using the output logit from
downstream fine-tuned Teacher as soft-labels ($\mathcal{L}_{\text{TASK}}$).
More details on distillations and ablations for the downstream tasks can be
found at Appendix.

Image Captioning. We evaluate our model by transferring it to the image
captioning task. We fine-tune our model by randomly masking out 15% of the
caption tokens and impose a classification task to predict the masked token id
using cross-entropy loss. Similar to [13], we trim and pad textual sentences
to the length of $20$. At inference, we recursively feed in [MASK] tokens and
predict out captions one after the other with the beam search size at 1. The
performance of captioning models is evaluated via BLEU@4 [43], METEOR [11],
CIDEr [64] and SPICE [2] metrics. We perform the parameter search in a limited
range: learning rate {$2e^{-5}$, $5e^{-6}$} and epochs {$20$, $30$, $40$}.

VQA. For the VQA task, the model must select the correct answer from the
multi-options list given an image and textual question. We conduct fine-tuning
on the VQA-2.0 dataset [16] and report the accuracy on test-std and test-dev
splits. Following [3], we train the VQA model as a $3,129$-way classification
task. We perform a light combinatorial parameter search on VQA task within a
limited range: learning rate {$1e^{-5}$, $5e^{-5}$} and epochs {$20$, $40$}.

###  4.3 Results and Analysis

Table 1 summarizes the results of DistillVLM using Oscar${}_{\text{b}}$ as the
Teacher model. We list VLP baselines with larger transformer architectures and
stronger visual representations in the top lines. In particularly, DistillVLM
without VL distillation achieves $34.0$ BLEU@4 and $115.7$ CIDEr scores with
TEE visual representations using VLP [37] (masked language prediction and
image-text matching losses). This is slightly lower than the performance
reported by MiniVLM [65] pre-trained on VL-7M: $116.7$ CIDEr score _vs_. our
reproduced $115.7$, which might be caused by the sub-optimal hyper-parameters.
The apparent performance gaps between larger and smaller VLP models indicate
the importance of visual representations so that the VL distillation is
desired on small VL architectures. Notably, when equipped with downstream
distillation, it performs better on COCO captioning dataset, $1.4$ more on
CIDEr, and $0.5$ more on BLEU@4 scores. Downstream distillation on VQA task
show marginal improvement: $69.2$ _vs_. $69.0$. We conjecture that this is
mainly because the classification distillation on YES/NO or counting type of
question does not provide better guidance, that the answers in the VQA task
are mostly irrelevant/mutually exclusive. However, VL distillation in the pre-
training stage increases the performances of DistillVLM on both captioning and
VQA tasks consistently across all metrics: $\Delta=1.2\%$ at B@4, $4.4$ at
CIDEr and $0.7$ higher on VQA test-std split. Compared to its non-distilled
counterpart MiniVLM [65], DistillVLM shows better results with only half the
size of VL-corpus. To this end, the combination of the VL distillation in both
pre-training and fine-tuning stage achieves the best results of DistillVLM ,
which shows comparable performances with Oscar${}_{\text{b}}$: $120.8$ _vs_.
$123.7$ with $70$% fewer parameters. To learn more about DistillVLM , we
conduct ablations on different designing options and examine the advantages of
distillation at different epochs and data usage at Section 4.

$\mathcal{L}_{\text{VLP}}$ | $\mathcal{L}_{\text{ATT}}$ | $\mathcal{L}_{\text{HID}}$ | COCO Captioning | VQA
---|---|---|---|---
B@4 | M | C | S | test-dev
✓ | ✗ | ✗ | $33.0$ | $27.3$ | $110.6$ | $20.4$ | $68.5$
✓ | ✓ | ✗ |  gray!0$32.9$ | $27.5$ |  gray!5$111.8$ | $20.6$ |  gray!5$68.9$
✓ | ✗ | ✓ |  gray!15$34.0$ | $27.8$ |  gray!15$114.4$ | $21.1$ |  gray!15$69.2$
✗ | ✓ | ✓ |  gray!10$33.9$ | $27.8$ |  gray!25$114.7$ | $21.1$ |  gray!15$69.2$
✓ | ✓ | ✓ |  gray!35$34.6$ | $27.9$ |  gray!35$115.6$ | $21.3$ |  gray!25$69.4$
Table 2: Detailed distillation effects based on attention matrices
($\mathcal{L}_{\text{ATT}}$), hidden hidden embedding
($\mathcal{L}_{\text{HID}}$), compared with VL pre-training losses
($\mathcal{L}_{\text{VLP}}$) at pre-training stage. Results are reported after
20 epochs of pre-training/distillation on 7M Image-Text pairs, then fine-tuned
at the downstream (with cross-entropy optimization only).

Distillation over Different Losses. Table 2 presents the individual
contribution of each distillation loss (attention matrices, hidden embedding)
on the basis of the VL pre-training. The experiments for VL Pre-
training/Distillation are trained for $20$ epochs using identical hyper-
parameters as before. From the table, we have the following observations:
First, the non-distilled baseline alone reaches $110.6$ CIDEr score for image
captioning and $67.2$ accuracy on VQA benchmark (shown in the first line of
Table 2). By mimicking the distribution of attention, minor improvements are
made, that is $1.2$ for CIDEr and $0.4$ for VQA scores respectively.
Similarly, we observe the same trend when combining VLP with hidden embedding
distillation. Compared with the VLP baseline, hidden embedding distillation
significantly improves the performance under all criteria, demonstrating the
efficacy of the alignment schema. In the end, the combination of all the loss
terms gives the best performance, confirming that our proposed attention and
hidden embedding distillation losses are complementary to each other. We find
that using distillation objective alone also produces satisfactory
performance, showing that knowledge transfer from distillation is to some
extent equivalent to VL pre-training loss.

Methods | COCO Captioning | VQA
---|---|---
B@4 | M | C | S | test-dev
VL Pre-training [37] | $33.0$ | $27.3$ | $110.6$ | $20.4$ | $68.5$
Textual Distill |  gray!10$34.1$ | $27.7$ |  gray!10$114.3$ | $20.9$ |  gray!10$69.0$
MSE + Layerwise |  gray!10$34.2$ | $27.8$ |  gray!10$114.8$ | $21.1$ |  gray!10$69.2$
MSE + Last-layer ∗ |  gray!5$33.3$ | $27.6$ |  gray!5$112.4$ | $20.7$ |  gray!5$68.5$
MSE + Last-layer |  gray!15$34.3$ | $27.8$ |  gray!15$115.3$ | $21.2$ |  gray!15$69.4$
NCE + Last-layer ∗ |  gray!15$34.3$ | $27.9$ |  gray!15$115.4$ | $21.2$ |  gray!12$69.3$
NCE + Last-layer |  gray!20$34.6$ | $27.9$ |  gray!25$115.6$ | $21.3$ |  gray!20$69.4$
Table 3: Ablation of DistillVLM using different distillation strategies, i.e.
, layer-to-layer distillation or last-layer distillation, using mean-square-
error distance (MSE) or noise-contrastive (NCE) loss. Textual Distill
represents applying the distillation only to the textual tokens without using
visual tokens. Captioning results are reported after $20$ epochs of
training/distillation on VL-7M with cross-entropy optimization.  ∗ is the
result using mean-pooled token embedding for distillation.

Different Distillation Strategies. Table 3 shows the results of distillation
using different strategies, i.e. , layer-to-layer distillation _vs_. last-
layer distillation, and MSE loss _vs_. NCE loss. We first study the effect of
our proposed visual token alignment by applying attention distribution and
hidden embedding distillation loss only to the textual token part: e.g. ,
using the “textual-to-textual” attention sub-matrices and their corresponded
textual token embedding. The second line in Table 3 is the result of textual
distillation, which shows a slight improvement over the VLP baseline.
Following previous language distillation works [27], we also conduct the
layer-to-layer attention and hidden distillation between Teacher and Student,
and observe inferior performances than the last-layer strategy. Beyond that,
the layer-to-layer method can also be severely limited by their architectural
structures [66] (e.g. , different number of layers and attention heads). “NCE
+ Last-layer” represents the results of DistillVLM using our proposed
contrastive objective function that uses negative samples for the alignment
learning. We find that contrastive learning leads to slightly better results
than MSE loss. To this end, we study the differences in using token-wise
embedding and the mean-pooled layer-wise embedding for contrastive learning
and observe that learning with token-wise embedding gives much better results,
which is a different observation from [57]. However, applying the mean-pooled
embedding with NCE loss mitigates this issue and gives on par results with
token-wise NCE method (see last two lines in Table 3). We further provide the
ablations of VL distillation for the downstream tasks in the appendix.

In Table 4, we study the effect of using more negative samples in NCE loss. We
observe that increasing the size of the sample queue can steadily contribute
to the performances of VL models. Especially, when we only use one negative
sample, the model reaches $112.5$ CIDEr score, which aligns with the MSE
results ($112.4$ CIDEr score) at Table 3. When increased to $4,096$, the model
performs best across all metrics. While continuing to use more negative
samples may produce better results, we just set the size of the sample queue
as $4,096$ in our experiments. Note that our queue stores the random sample
representations from Teacher VLM, which remain consistent throughout the
distillation process. This also implies the feasibility of leveraging in-batch
samples for contrastive learning, while the queue design relieves the model
from batch-size requirements and allows the use of more negative samples.

# Neg. | COCO Captioning | VQA
---|---|---
B@4 | M | C | S | test-dev
$1$ | $33.3$ | $27.6$ | $112.5$ | $20.7$ | $68.5$
$64$ |  gray!5$33.6$ | $27.7$ |  gray!5$112.7$ | $20.9$ |  gray!5$68.9$
$128$ |  gray!10$33.6$ | $27.7$ |  gray!10$112.7$ | $20.9$ |  gray!5$68.9$
$512$ |  gray!15$33.7$ | $27.8$ |  gray!15$113.3$ | $21.0$ |  gray!5$68.8$
$1,024$ |  gray!20$34.1$ | $27.9$ |  gray!20$114.7$ | $21.2$ |  gray!15$69.1$
$4,096$ |  gray!25$34.3$ | $27.9$ |  gray!25$115.4$ | $21.2$ |  gray!20$69.3$
Table 4: Effect of the number of negative samples for noise contrastive
estimation loss. A larger queue size incrementally contributes to the
distillation performance. When queue size approaches 1, the NCE loss is
approximately the MSE loss with an only positive anchor from the Teacher. All
the experiments are trained for 20 epochs using sample queues in different
sizes on VL-7M, and then transferred to downstream.

Data-efficient VL Distillation. One critical aspect of VL distillation for
real-world application is its ability to efficiently train smaller VL model
with limited cost, i.e. , with a smaller VL corpus (data scarcity) and less
converging epochs (training efficiency). To further assess whether VL
distillation can cope with these challenges, we perform VL distillation at the
pre-training stage when trained with $1$, $5$, $10$, $20$, $50$, $100$ epochs
and compare their results with VLP. In addition, as pointed by [40] that
specific partial VL data might contribute more to performances, we propose to
conduct VL distillation/pre-training using evenly sampled partial data ($1\%$,
$5\%$, $10\%$, $20\%$, $50\%$ and $100\%$ of VL-7M). These also help to verify
whether DistillVLM benefits from more converging epochs and more VL data.
Several conclusions can be drawn from the above results. First, VL
distillation brings a consistent CIDEr gain across different training epochs.
Non-distilled VL pre-training method achieves only $99.8$ CIDEr score with $1$
epoch of training, while DistillVLM reaches $103.1$ (see Figure 3). Notably,
CIDEr score of DistillVLM increases steadily with more training epochs,
revealing that VL distillation is more effective than VL pre-training. When it
comes to using different percentages of VL data, we also see a similar trend.
In the most extreme case, with only $1\%$ of VL-7M corpus available, VL pre-
training produces $89.1$ CIDEr score, $4.1$ lower than the VL distillation.
With more image-text pairs, VL distillation obviously gives even better
results: $8.6$ higher for $10\%$, $6.9$ higher for $20\%$ and $5.0\%$ higher
for $100\%$. This shows that regardless of the amount of data available, VL
distillation provides much more effective and informative supervision than the
normal pre-training strategy, which leads to better performances.

Figure 3: Top: Captioning CIDEr score gain from VL distillation under
different epochs ($1$, $10$, $20$, $50$, $100$) in pre-training/distillation
on VL-7M; Bottom: Using $1\%$, $10\%$, $20\%$, $50\%$ and $100\%$ of VL-7M
image-text pairs with $20$ epochs of pre-training/distillation.

##  5 Conclusion

We have proposed the first VL distillation, which leverages the knowledge
distillation technique to compress large visual-linguistic models. Our
experiments confirmed the validity of VL distillation from several aspects:
Compared to the non-distilled VL pre-training method, VL distillation not only
brings better performances, it is also more data efficient. Our extensive
ablations also verified that our VL distillation strategies are simple yet
effective.

## References

  * [1] S. Ahn, S. X. Hu, A. Damianou, N. D. Lawrence, and Z. Dai.  Variational information distillation for knowledge transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9163–9171, 2019.
  * [2] P. Anderson, B. Fernando, M. Johnson, and S. Gould.  Spice: Semantic propositional image caption evaluation.  In European conference on computer vision, pages 382–398. Springer, 2016.
  * [3] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077–6086, 2018.
  * [4] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  arXiv preprint arXiv:2005.14165, 2020.
  * [5] C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil.  Model compression.  In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535–541, 2006.
  * [6] J. Cao, Z. Gan, Y. Cheng, L. Yu, Y.-C. Chen, and J. Liu.  Behind the scene: Revealing the secrets of pre-trained vision-and-language models.  In European Conference on Computer Vision, pages 565–580. Springer, 2020.
  * [7] L. Chen, Z. Gan, D. Wang, J. Liu, R. Henao, and L. Carin.  Wasserstein contrastive representation distillation.  arXiv preprint arXiv:2012.08674, 2020.
  * [8] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu.  Uniter: Learning universal image-text representations.  European Conference on Computer Vision, 2020.
  * [9] K. Clark, U. Khandelwal, O. Levy, and C. D. Manning.  What does bert look at? an analysis of bert’s attention.  In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2019.
  * [10] K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning.  Electra: Pre-training text encoders as discriminators rather than generators.  arXiv preprint arXiv:2003.10555, 2020.
  * [11] M. Denkowski and A. Lavie.  Meteor universal: Language specific translation evaluation for any target language.  In Proceedings of the ninth workshop on statistical machine translation, pages 376–380, 2014.
  * [12] K. Desai and J. Johnson.  Virtex: Learning visual representations from textual annotations.  arXiv preprint arXiv:2006.06666, 2020.
  * [13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  North American Chapter of the Association for Computational Linguistics, 2019.
  * [14] Z. Fang, J. Wang, L. Wang, L. Zhang, Y. Yang, and Z. Liu.  Seed: Self-supervised distillation for visual representation.  The International Conference on Learning Representations, 2021.
  * [15] Z. Gan, Y.-C. Chen, L. Li, C. Zhu, Y. Cheng, and J. Liu.  Large-scale adversarial training for vision-and-language representation learning.  arXiv preprint arXiv:2006.06195, 2020.
  * [16] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh.  Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [17] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [18] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [19] G. Hinton, O. Vinyals, and J. Dean.  Distilling the knowledge in a neural network.  NIPS Deep Learning and Representation Learning Workshop (2015), 2015\.
  * [20] X. Hu, X. Yin, K. Lin, L. Wang, L. Zhang, J. Gao, and Z. Liu.  Vivo: Surpassing human performance in novel object captioning with visual vocabulary pre-training.  Proceedings of the AAAI Conference on Artificial Intelligence, 2021\.
  * [21] Z. Huang and N. Wang.  Like what you like: Knowledge distill via neuron selectivity transfer.  arXiv preprint arXiv:1707.01219, 2017.
  * [22] Z. Huang and N. Wang.  Like what you like: Knowledge distill via neuron selectivity transfer.  arXiv preprint arXiv:1707.01219, 2017.
  * [23] Z. Huang, Z. Zeng, B. Liu, D. Fu, and J. Fu.  Pixel-bert: Aligning image pixels with text by deep multi-modal transformers.  arXiv preprint arXiv:2004.00849, 2020.
  * [24] D. A. Hudson and C. D. Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6700–6709, 2019.
  * [25] G. Jawahar, B. Sagot, and D. Seddah.  What does bert learn about the structure of language?  In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics, 2019.
  * [26] H. Jiang, I. Misra, M. Rohrbach, E. Learned-Miller, and X. Chen.  In defense of grid features for visual question answering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10267–10276, 2020.
  * [27] X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu.  Tinybert: Distilling bert for natural language understanding.  arXiv preprint arXiv:1909.10351, 2019.
  * [28] X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu.  TinyBERT: Distilling BERT for natural language understanding.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174, Online, Nov. 2020. Association for Computational Linguistics.
  * [29] Y. Kim and A. M. Rush.  Sequence-level knowledge distillation.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas, Nov. 2016. Association for Computational Linguistics.
  * [30] A. Koratana, D. Kang, P. Bailis, and M. Zaharia.  Lit: Learned intermediate representation training for model compression.  In International Conference on Machine Learning, pages 3509–3518. PMLR, 2019.
  * [31] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123(1):32–73, 2017.
  * [32] K. Lagler, M. Schindelegger, J. Böhm, H. Krásná, and T. Nilsson.  Gpt2: Empirical slant delay model for radio space geodetic techniques.  Geophysical research letters, 40(6):1069–1073, 2013.
  * [33] J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu.  Less is more: Clipbert for video-and-language learning via sparse sampling.  arXiv preprint arXiv:2102.06183, 2021.
  * [34] G. Li, N. Duan, Y. Fang, M. Gong, and D. Jiang.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34-07, pages 11336–11344, 2020.
  * [35] L. Li, Y.-C. Chen, Y. Cheng, Z. Gan, L. Yu, and J. Liu.  Hero: Hierarchical encoder for video+ language omni-representation pre-training.  arXiv preprint arXiv:2005.00200, 2020.
  * [36] L. Li, Z. Gan, and J. Liu.  A closer look at the robustness of vision-and-language pre-trained models.  arXiv preprint arXiv:2012.08673, 2020.
  * [37] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In European Conference on Computer Vision, pages 121–137. Springer, 2020.
  * [38] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [39] J. Lu, D. Batra, D. Parikh, and S. Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  Advances in Neural Information Processing Systems (NeurIPS), 2019\.
  * [40] J. Lu, V. Goswami, M. Rohrbach, D. Parikh, and S. Lee.  12-in-1: Multi-task vision and language representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10437–10446, 2020.
  * [41] A. Miech, J.-B. Alayrac, L. Smaira, I. Laptev, J. Sivic, and A. Zisserman.  End-to-end learning of visual representations from uncurated instructional videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9879–9889, 2020.
  * [42] V. Ordonez, G. Kulkarni, and T. Berg.  Im2text: Describing images using 1 million captioned photographs.  Advances in neural information processing systems, 24:1143–1151, 2011.
  * [43] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  * [44] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.  Automatic differentiation in pytorch.  -, 2017.
  * [45] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015.
  * [46] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020, 2021.
  * [47] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever.  Improving language understanding by generative pre-training.  -, 2018.
  * [48] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  arXiv preprint arXiv:1910.10683, 2019.
  * [49] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever.  Zero-shot text-to-image generation.  arXiv preprint arXiv:2102.12092, 2021.
  * [50] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  arXiv preprint arXiv:1506.01497, 2015.
  * [51] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  * [52] V. Sanh, L. Debut, J. Chaumond, and T. Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  arXiv preprint arXiv:1910.01108, 2019.
  * [53] S. Shao, Z. Li, T. Zhang, C. Peng, G. Yu, X. Zhang, J. Li, and J. Sun.  Objects365: A large-scale, high-quality dataset for object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8430–8439, 2019.
  * [54] P. Sharma, N. Ding, S. Goodman, and R. Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [55] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  The International Conference on Learning Representations, 2020.
  * [56] C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid.  Videobert: A joint model for video and language representation learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464–7473, 2019.
  * [57] S. Sun, Z. Gan, Y. Cheng, Y. Fang, S. Wang, and J. Liu.  Contrastive distillation on intermediate representations for language model compression.  arXiv preprint arXiv:2009.14167, 2020.
  * [58] Z. Sun, H. Yu, X. Song, R. Liu, Y. Yang, and D. Zhou.  MobileBERT: a compact task-agnostic BERT for resource-limited devices.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2158–2170, Online, July 2020. Association for Computational Linguistics.
  * [59] H. Tan and M. Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  Conference on Empirical Methods in Natural Language Processing, 2019\.
  * [60] M. Tan and Q. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International Conference on Machine Learning, pages 6105–6114. PMLR, 2019.
  * [61] M. Tan, R. Pang, and Q. V. Le.  Efficientdet: Scalable and efficient object detection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10781–10790, 2020.
  * [62] Y. Tian, D. Krishnan, and P. Isola.  Contrastive multiview coding.  arXiv preprint arXiv:1906.05849, 2019.
  * [63] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need.  Advances in Neural Information Processing Systems (Neurips), 2017\.
  * [64] R. Vedantam, C. Lawrence Zitnick, and D. Parikh.  Cider: Consensus-based image description evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  * [65] J. Wang, X. Hu, P. Zhang, X. Li, L. Wang, L. Zhang, J. Gao, and Z. Liu.  Minivlm: A smaller and faster vision-language model.  arXiv preprint arXiv:2012.06946, 2020.
  * [66] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [67] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le.  Self-training with noisy student improves imagenet classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10687–10698, 2020.
  * [68] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He.  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017.
  * [69] C. Xu, W. Zhou, T. Ge, F. Wei, and M. Zhou.  Bert-of-theseus: Compressing bert by progressive module replacing.  arXiv preprint arXiv:2002.02925, 2020.
  * [70] I. Z. Yalniz, H. Jégou, K. Chen, M. Paluri, and D. Mahajan.  Billion-scale semi-supervised learning for image classification.  arXiv preprint arXiv:1905.00546, 2019.
  * [71] J. Yim, D. Joo, J. Bae, and J. Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4133–4141, 2017.
  * [72] S. Zagoruyko and N. Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  The International Conference on Learning Representations, 2017.
  * [73] P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, and J. Gao.  Vinvl: Making visual representations matter in vision-language models.  arXiv preprint arXiv:2101.00529, 2021.
  * [74] L. Zhou, H. Palangi, L. Zhang, H. Hu, J. Corso, and J. Gao.  Unified vision-language pre-training for image captioning and vqa.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34-07, pages 13041–13049, 2020.
  * [75] L. Zhu and Y. Yang.  Actbert: Learning global-local video-text representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8746–8755, 2020.

## Appendix

In this supplementary material, we provide additional details of the VL
distillation, which includes: a pseudo-implementation in PyTorch [44] style;
more results/ablations in the downstream; DistillVLM results transferred to
the image retrieval task. To the end, we present qualitative analysis of
attention distribution distillation and provide an analysis for VL
Distillation and its broader impact for future studies.

## Pseudo-implementation for VL Distillation

The following section presents the pseudo implementation (see Page 4) of our
proposed VL distillation schema in PyTorch style.

## VL Distillation in Image Captioning Task

We also conduct ablations of VL distillation in the task-specific fine-tuning
stage. Table 5 summarizes the fine-tuning results on image captioning task
using different distillation losses, whereby weight loss parameters within the
range of $[0,1,10]$ are searched for. All experiments are pre-
trained/distilled for 20 epochs on the VL-7M dataset. In contrast to the
results in pre-training stage VL distillation, we find that the
$\mathcal{L}_{\text{HID}}$ and $\mathcal{L}_{\text{att}}$ do not contribute to
the performances for downstream VL distillation. Instead, using just the
$\mathcal{L}_{\text{CLS}}$ obviously improves the results: the CIDEr score is
improved from $115.6$ to $116.3$.

## Transferring to Image-text Retrieval Task

In addition to the previously mentioned image captioning task and VQA task,
DistillVLM is not VL task specific and can be extended to other tasks as well.
In this section, we study whether the advantages from pre-training stage VL
distillation extend to the image-text retrieval task on COCO dataset. In
particular, the image-text retrieval task aims to retrieve the target image or
text given a query text or image. We set up the image-text retrieval task on
COCO dataset, and evaluate the performance of model using top-K retrieval
accuracy on both 1k and 5k test splits. To transfer and train the VL model for
the image-text retrieval task, we use the first [CLS] token from transformer
in a binary classification task, where the classifier predicts whether the
image and text match or not. To construct the negative tuple, irrelevant texts
are sampled as non-matching cases. We fine-tune the model with learning rate
at $2e-5$ and train it for $30$ epochs. The results are shown in Table 6.

$\mathcal{L}_{\text{CLS}}$ | $\mathcal{L}_{\text{ATT}}$ | $\mathcal{L}_{\text{HID}}$ | COCO Captioning
---|---|---|---
B@4 | M | C | S
$0$ | $0$ | $0$ | $34.6$ | $27.9$ | $115.6$ | $21.3$
$1$ | $0$ | $0$ | $34.6$ | $28.1$ | $116.3$ | $21.3$
$1$ | $1$ | $0$ | $34.6$ | $28.1$ | $116.3$ | $21.3$
$1$ | $10$ | $0$ | $34.5$ | $28.0$ | $116.1$ | $21.3$
$1$ | $0$ | $1$ | $34.5$ | $28.0$ | $116.0$ | $21.3$
$1$ | $0$ | $10$ | $34.3$ | $28.0$ | $115.2$ | $21.2$
$10$ | $10$ | $10$ | $34.6$ | $28.1$ | $116.4$ | $21.3$
Table 5: Detailed distillation effects based on classification logit ($\mathcal{L}_{\text{CLS}}$), attention matrices ($\mathcal{L}_{\text{ATT}}$), hidden embedding ($\mathcal{L}_{\text{HID}}$) with various weights ($\alpha$, $\beta$), compared with non-distilled strategy at fine-tuning stage. The first line shows the fine-tuning result without any distillation.  Method |  | 1K Test Set | 5K Test Set
---|---|---|---
| Text Retrieval | Image Retrieval | Text Retrieval | Image Retrieval
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
PixelBERT [23] | ✗ | $77.8$ | $95.4$ | $98.2$ | $64.1$ | $91.0$ | $96.2$ | $53.4$ | $80.4$ | $88.5$ | $41.1$ | $69.7$ | $80.5$
Unicoder-VL [34] | ✗ | $84.3$ | $97.3$ | $99.3$ | $69.7$ | $93.5$ | $97.2$ | $62.3$ | $87.1$ | $92.8$ | $46.7$ | $76.0$ | $85.3$
OSCAR$-\text{base}$ [37] | ✗ | $88.4$ | $99.1$ | $99.8$ | $75.7$ | $95.2$ | $98.3$ | $70.0$ | $91.1$ | $95.5$ | $54.0$ | $80.8$ | $88.5$
MiniVLM [65] | ✗ | $81.1$ | $96.1$ | $99.2$ | $68.5$ | $93.0$ | $97.1$ | $58.8$ | $85.1$ | $91.7$ | $45.0$ | $74.1$ | $84.0$
Baseline | ✗ | $77.3$ | $95.0$ | $98.5$ | $64.6$ | $92.0$ | $96.7$ | $54.4$ | $81.4$ | $89.3$ | $41.3$ | $71.2$ | $81.9$
DistillVLM | ✓ | $80.0$ | $95.5$ | $98.5$ | $68.3$ | $92.3$ | $96.9$ | $58.3$ | $84.1$ | $91.3$ | $43.9$ | $73.7$ | $83.3$
$\Delta$ |  | +2.7 | +0.5 | +0.0 | +3.7 | +0.3 | +0.2 | +3.9 | +2.7 | +2.0 | +2.6 | +2.5 | +1.4
Table 6: Results of DistillVLM with and without VL distillation transferring to Image-Text Retrieval task on COCO dataset. Method | COCO Captioning
---|---
B@4 | M | C | S
VL Pre-training | $33.0$ | $27.3$ | $110.6$ | $20.4$
VL Distill. w/o. Adaptation | $33.3$ | $27.5$ | $112.3$ | $20.6$
VL Distill. with Adaptation.∗ | $34.4$ | $27.9$ | $115.0$ | $21.0$
VL Distill. with Adaptation | $34.6$ | $27.9$ | $115.6$ | $21.3$
Table 7: Effect of VL distillation with and without Teacher VLM adaptation in pre-training stage. First row in the table shows the captioning performances using VL pre-training without VL distillation. ∗ denotes the VL distillation results without re-training the Teacher VLM.  Method | COCO Captioning
---|---
B@4 | M | C | S
VL Pre-training | $33.0$ | $27.3$ | $110.6$ | $20.4$
VL Distill. w/o. Adaptation | $33.2$ | $27.5$ | $110.7$ | $20.3$
VL Distill. with Adaptation | $33.5$ | $27.5$ | $111.6$ | $20.6$
Table 8: Effect of VL distillation with and without Teacher VLM adaptation in
fine-tuning stage. First row in the table shows the COCO captioning results
without any downstream distillation.

We just examine the effect of VL distillation in the pre-training stage.
DistillVLM without VL distillation is notably worse than MiniVLM: $81.1$ _vs_.
$77.3$ accuracy of R@$1$ on 1$K$ Test Set, since less training data is used.
Equipped with pre-training stage VL distillation, it is significantly improved
to $80.0$ and $58.3$ accuracy of R@$1$ on $1$K/$5$K Test Set respectively,
which are $2.7$ and $4.0$ higher. Compared to its counterpart MiniVLM [65] and
other state-of-the-art methods, DistillVLM achieves on par performances, while
MiniVLM is trained on 14M image-text pairs, which is twice the size of our
pre-training VL corpus. Other methods all use a larger transformer model or
stronger visual representations.

## Discussion on Teacher VLM Adaptation

# Epochs | COCO Captioning | % Data | COCO Captioning
---|---|---|---
B@4 | M | C | S | B@4 | M | C | S
gray!15 1 | $31.3$ | $25.6$ | $99.8$ | $19.0$ | 1% | $27.9$ | $24.2$ | $89.1$ | $17.2$
$31.3$ | $26.2$ | $103.3$ | $19.4$ | $29.2$ | $24.6$ | $93.2$ | $17.8$
gray!15  5 | $31.3$ | $26.4$ | $104.5$ | $19.6$ | 5% | $29.5$ | $25.1$ | $96.6$ | $18.4$
$32.8$ | $27.3$ | $109.5$ | $20.5$ | $31.6$ | $26.3$ | $103.9$ | $19.3$
gray!15  10 | $32.2$ | $26.8$ | $107.8$ | $20.2$ | 10% | $30.2$ | $25.7$ | $99.8$ | $18.9$
$33.9$ | $27.5$ | $112.8$ | $20.6$ | $32.8$ | $27.0$ | $108.4$ | $20.0$
gray!15  20 | $33.0$ | $27.3$ | $110.6$ | $20.4$ | 20% | $31.2$ | $27.3$ | $104.1$ | $19.6$
$34.6$ | $27.9$ | $115.6$ | $21.3$ | $33.4$ | $27.3$ | $111.1$ | $20.5$
gray!15  50 | $34.3$ | $27.8$ | $115.2$ | $21.1$ | 50% | $32.2$ | $26.9$ | $108.5$ | $20.1$
$35.1$ | $28.2$ | $118.2$ | $21.5$ | $34.0$ | $27.7$ | $114.1$ | $21.2$
gray!15  100 | $34.0$ | $28.0$ | $115.7$ | $21.1$ | 100% | $33.0$ | $27.3$ | $110.6$ | $20.4$
$35.2$ | $28.6$ | $120.1$ | $21.9$ | $34.6$ | $27.9$ | $115.6$ | $21.3$
Table 9: Detailed image captioning results for data efficient VL distillation.
The rows in gray show results using non-distilled VL pre-training method, and
the following rows show results with VL distillation.

Unlike the previous knowledge distillation where the Teacher model is usually
not re-trained, Teacher VLM in our work is adapted using the new visual
tokens. This is to ensure that the use of new Teacher visual tokens does not
have a negative impact to the Teacher. Nevertheless, it is also viable to use
Teacher VLM directly without adaptation. We compare the performances using
Teacher VLM with and without adaptation in Table 7 and 8 in both pre-training
and fine-tuning stages.

In particular, we first conduct VL distillation in pre-training stage without
using the adapted Teacher VLM, and use the old visual tokens as in [37]
without re-extracting them using object proposals from the lightweight
detector. As there is no semantic correspondence across Teacher and Student’s
tokens, we just adopt the classification distillation on the masked tokens’
logit. The second line of Table 7 gives the results (VL Distill. w/o.
Adaptation), where we observe a slight improvement on captioning performances:
$112.3$ _vs_. $110.6$. Furthermore, we apply the visual tokens extracted
according to the Student detector’s object proposals, which guarantees the
semantic correspondence across Teacher and Student tokens. With that, we
conduct the VL distillation using the our proposed losses
($\mathcal{L}_{\text{ATT}}$ and $\mathcal{L}_{\text{HID}}$), and find
significant improvement: CIDEr score increases from $110.6$ to $115.0$ (as
shown in third line of Table 7). Finally, by re-training the Teacher VLM using
the new extracted visual tokens will lead to more improvement: $115.6$ _vs_.
$115.0$. This confirms that, even without re-training the Teacher VLM, VL
distillation still gives satisfactory results.

We conduct similar investigations in the downstream and show results in Table
8. In particular, downstream VL Distillation without adaptation gives minor
improvement ($0.2$ and $0.1$ for BLEU@$4$ and CIDEr scores), while using the
adapted Teacher VLM leads to obvious better results ($0.5$ and $1.0$ higher on
BLEU@$4$ and CIDEr score). We only apply the classification distillation loss
in the downstream experiments.

## Detailed Results for Data Efficient VL Distillation

Table 9 summarizes more detailed results of data efficient VL distillation on
image captioning tasks. In particular, we compare the effectiveness of VL
distillation in pre-training stage using different converging epochs
$[1,5,10,20,50,100]$ and fraction data $[1\%,5\%,10\%,20\%,50\%,100\%]$. VL
distillation comprehensively improves the results on all metrics, regardless
of the amount of data available or different training epochs.

## Effect of Attention Distribution Distillation

In Figure 5, we show qualitative effect of attention distribution distillation
where we select random samples from VL corpus and retrieve their last
transformer block attention maps from the Teacher VLM, DistillVLM and the non-
distilled counterpart of DistillVLM . To illustrate this, the attention maps
are averaged over 12 self-attention heads. Based on the examples, we observe
that the attention maps after VL distillation exhibit higher similarities to
the Teacher VLM’s attention maps. This confirms that our attention
distribution distillation successfully mimic the Teacher VLM’s attention
distribution to some extent.

⬇

1 Q: Maintaining queue of random history representations: (-K- X -DT-)

2 teacher_model, student_model: Large/small Teacher/Student VL model.

3 FC: Linear layer for dimension transformation of student hidden
representation.

4 temp: Temperatures of the NCE loss.

5 lamda, beta: Weights for loss terms.

6

7

8 # activate evaluation mode for Teacher to freeze BN and updation.

9 with torch.no_grad():

10 # t_atts: 1 X N X N, t_hids: 1 X N X DT

11 t_atts, t_hids = teacher_model(**teacher_inputs)

12

13 # s_atts: 1 X N X N, s_hids: 1 X N X DS

14 mlm_loss, itm_loss, s_atts, s_hids = student_model(**student_inputs)

15

16 # dimension transformation on student hidden, s_hids: 1 X N X DT

17 s_hids = FC(s_hids)

18

19 # attention distillation loss

20 att_loss = mse_loss(s_atts, t_atts)

21

22 # NCE hidden states distillation loss

23

24 # l2-norm the hidden representation

25 t_hids, s_hids = L2_norm(s_hids), L2_norm(t_hids)

26

27 # positive logit: Nx1

28 l_pos = torch.einsum(’nd,nd->n’, [s_hids.squeeze(0), t_hids.squeeze(0)])

29

30 # negative logit: NxK

31 l_neg = torch.einsum(’nd,dk->nk’, [s_hids.squeeze(0), Q.detach().T()])

32

33 # logit: Nx(1+K)

34 logits = torch.cat([l_pos, l_neg], dim=1)

35

36 # apply the temperature for NCE loss

37 logits /= temp

38

39 # NCE label: N

40 labels = torch.zeros(logits.shape[0], dtype=torch.long).cuda()

41

42 nce_loss = ce_loss(logits, labels)

43

44 # update the queue with current teacher hidden states.

45 dequeue_and_enqueue(t_hids, Q)

46

47 # aggregated losses

48 loss = mlm_loss + itm_loss + lamda * att_loss + beta * nce_loss

Figure 4: Our VL distillation consists of the attention distribution and
hidden embedding mimicking when batch size $=1$ with $N$ textual/visual
tokens. We assume [PAD] tokens are excluded from input tokens before the
distillation.

Figure 5: Qualitative effect of attention distribution distillation. The 3
neighbouring attention maps are the averaged attention map from the last
transformer block of: Teacher VLM, DistillVLM , and non-distilled VL model.
The number at bottom is the $l2$ distance of the attention map with its
corresponded Teacher attention map.


| Link $(i,j)$
Parameter | (1,2) | (1,3) | (2,1) | (2,2) | (3,1) | (3,2)
link construction cost $c_{ij}$ | $1$ | $10$ | $1$ | $3$ | $10$ | $3$
link travel time $t_{ij}$ | $1$ | $10$ | $1$ | $2$ | $10$ | $2$
Table 4: Value of the parameters for scenario 1 (3-node network). The value of
$\omega_{t}$ is set to $0.1$.

In this section, we address the problem defined in equation (66) for a network
comprising 3 nodes (stations). We assume that the graph of potential links
$\mathcal{A}$ is complete, leading to a total of 6 link capacities that need
to be determined in the upper level. Additionally, as in the rest of the
manuscript, we assume that there is demand for all markets, meaning that the
set $\mathcal{K}$ is complete and includes all 6 possible origin-destination
pairs. The specific values of the key parameters can be found in Table 4, with
further details about the simulated scenario available in the online code
repository.

For BLOCC, we set the stepsize to $\eta=1.6\times 10^{-4}$ and analyze the
algorithm’s convergence for three different values of $\gamma$: $\gamma=2$,
$\gamma=3$, and $\gamma=4$. The upper-level objective values are computed
using $-f(x_{t},y_{g,t}^{T_{g}})$ and $-f(x_{t},y_{F,t}^{T_{F}})$. The results
are presented in Figures 8 and 8.

Figure 8 illustrates the performance of our BLOCC algorithm over time. The
orange line represents the evolution of $-f(x_{t},y_{t})$ for the LV-HBA
algorithm, while the other six lines represent different implementations of
BLOCC. Each color represents a different value of $\gamma$; solid lines
represent $-f(x_{t},y_{g,t}^{T_{g}})$ and dashed lines represent
$-f(x_{t},y_{F,t}^{T_{F}})$. Each simulation is conducted 10 times with 10
different random initializations of the upper-level variables, and both the
mean and the standard deviation values are displayed. The results indicate
that all versions of our BLOCC algorithm converge in less than 10 seconds,
while LV-HBA shows slight fluctuations even after running for more than 50
seconds. This may be attributed to the fact that the LV-HBA algorithm requires
a joint projection into $\\{\mathcal{X}\times\mathcal{Y}:g^{c}(x,y)\leq 0\\}$
at each iteration, involving 42 variables and 6 CCs.

Figure 8 shows the lower-level optimality gap, namely
$g(x_{t},y_{t})-g(x_{t},y^{*}(x_{t}))$ for LV-HBA and BLOCC with 3 different
values of $\gamma$. In the case of LV-HBA, lower-level optimality is not
attained within 70 seconds of running time. Conversely, for BLOCC, lower-level
optimality is achieved by construction when $y_{t}$ is set to
$y_{g,T}^{T_{g}}$, resulting in a zero gap. Additionally, when $y_{t}$ is set
to $y_{F,T}^{T_{F}}$, we observe that: i) lower-level optimality is
accomplished for $\gamma\geq 3$, and ii) when $\gamma=2$, an optimality gap
exists, but it is one order of magnitude smaller than that for LV-HBA.

Figure 9:  Negative upper-level objective $-f(x_{t},y_{t})$ for a 9-node
network design problem; Solid lines show the mean value of
$-f(x_{t},y_{g,t}^{T_{g}})$ with the shaded region as a standard deviation;
Dashed lines show the mean value of $-f(x_{t},y_{F,t}^{T_{F}})$ with the
shaded region as a standard deviation; Three different $\gamma$ values (red,
purple, blue) and fixed stepsize $\eta=1.6e-4$; The orange color represents
the result of the LV-HBA algorithm.

Figure 10: Negative upper-level objective $-f(x_{t},y_{t})$ for a metro
network design problem in Seville, Spain, for 2 random initializations of the
upper-level variables. Solid lines represent the mean of
$-f(x_{t},y_{g,t}^{T_{g}})$, and the shaded region is the standard deviation.
The dashed lines represent the mean of $-f(x_{t},y_{F,t}^{T_{F}})$, and the
shaded region is the standard deviation. Three different $\gamma$ values are
tested with a fixed stepsize $\eta=1.6e-4$.

####  F.3 Numerical results for the 9-node network

In this case, we consider the network in [21], see also Figure 6, which has
$|\mathcal{S}|=9$ nodes and $|\mathcal{A}|=30$ potential links. As before, we
consider that all markets exist, so that $|\mathcal{K}|=9\cdot 8=72$. The
remaining parameters are described in Figure 6 and the code repository.

Figure 10 is the counterpart of Figure 8 for the 9-node scenario, showing the
behavior of our BLOCC algorithm for $\eta=1.6e-4$ and $\gamma\in\\{2,3,4\\}$.
Each simulation is repeated 10 times (using 10 different random
initializations of the upper-level variables), and both the mean and the
standard deviation values are shown. Since the number of variables and
constraints is almost one order of magnitude larger, the algorithm requires
more time to converge. However, convergence takes place in a reasonable amount
of time (20-40 times longer than in the previous 3-node test case). Regarding
the optimal value, we observe that: i) the sensitivity of the (steady-state)
optimal value with respect to $\gamma$ is not too large; ii) the solutions
based on $y_{F,T}^{T_{F}}$ yield better upper-level values than those based on
$y_{g,T}^{T_{g}}$; and iii) the gap between $f(x_{T},y_{g,T}^{T_{g}})$ and
$f(x_{T},y_{F,T}^{T_{F}})$ decreases as $\gamma$ increases. Observation ii) is
due to the fact that $y_{F,T}^{T_{F}}$ is not feasible (meaning that it
violates the optimality of the lower level); hence, it is able to achieve a
better upper-level objective. In addition, the behavior observed in iii) is
consistent with the discussion in section 2.2, with the value of $\gamma$
having an impact on the suboptimality of $y_{F,t}^{T_{F}}$ at the lower level.
Specifically, higher values of $\gamma$ push $y_{F,t}^{T_{F}}$ closer to
$y_{g,t}^{T_{g}}$ and, hence, decrease the lower-level optimality gap.
Finally, we must note that for the solid lines (associated with
$y_{g,t}^{T_{g}}$), it holds that
$f(x_{t},y_{g,t}^{T_{g}})=f(x_{t},y^{*}(x_{t}))$. This implies that if we need
a solution that is feasible at the lower level, then better objective values
are associated with higher values of $\gamma$.

####  F.4 Numerical results for the Seville network

Figure 11: Topology of the Seville network.

In this section, we demonstrate the practical use of BLOCC in a real
transportation network design problem. Specifically, we address the design of
a metro network in the city of Seville, which has approximately one million
inhabitants and is located in the south of Spain, with the bus system as its
competitor. The data for the demand, number of stations, and locations have
been taken from [21]. Information about the construction costs, capacity, and
travel time has been obtained from Spanish rapid transit operators (see
references in [10, 55] for full details). The city authorities considered
$|\mathcal{S}|=24$ potential station locations. Regarding the links, the
following assumptions are made:

  1. A1)

The link between nodes $(i,j)\in\mathcal{S}\times\mathcal{S}$ only exists if
node $j$ is one of the three closest neighbors to $i$, or vice versa. The
distance here is measured in terms of travel time. This assumption limits the
number of lines in any station to be at most 3, which is a very mild
assumption for a network with 24 stations.

  2. A2)

The link between nodes $(i,j)\in\mathcal{S}\times\mathcal{S}$ only exists if
the travel time $t_{ij}$ is less than 7 minutes. This is also a mild
assumption, since it enables all the stations to be connected, including those
that are further away from the city center (the airport and the university
campus [21]).

Under these two conditions, the set $\mathcal{A}$ of potential links contains
$|\mathcal{A}|=88$ links. The sets of links and stations, along with their
actual locations, are shown in Figure 11. Finally, we consider all possible
markets between nodes, so $|\mathcal{K}|=24\times 23=552$.

Following the narrative in Sections F.2 and F.3, Figure 10 presents the
evolution of the upper-level objective function with time for three values of
the parameter $\gamma\in\\{2,3,4\\}$. The stepsize value has been set to
$\eta=1.6\times 10^{-4}$, and two different initializations have been
considered. Regarding the behavior of the algorithm with respect to the value
of $\gamma$ and the particular output chosen ($y_{T,g}^{T_{g}}$ vs.
$y_{T,F}^{T_{F}}$), the findings are very similar to those in Figure 10.
Namely, smaller gaps are found for larger values of $\gamma$, and if
feasibility at the lower level (i.e., consistency with the user preference
level) must be preserved, better objective values are achieved for larger
values of $\gamma$.

The most important observation, however, is related to the running time.
Specifically, Figure 10 reveals that, for this real-world scenario, our BLOCC
algorithm converges in 10-20 hours. While this is more than 1,000 times larger
than the convergence interval for the 3-node network, the number of variables
and constraints here is 100 times larger. More importantly, in the context of
network transportation design, optimization times of 100 hours are widely
accepted even for single-level formulations. Overall, we believe that the
numerical results demonstrate that the BLOCC algorithm proposed in this work
can solve problems with a large number of variables and CCs, which can have
practical value in real-world applications, such as the one studied in this
section.