11institutetext: 1Tsinghua University 2Microsoft Research Asia

# iCAR: Bridging Image Classification and Image-text Alignment for Visual
Recognition

Yixuan Wei 11 Yue Cao Contact person. Yixuan Wei, Zhuliang Yao and Zhenda Xie
are long-term interns at MSRA.22 Zheng Zhang 22 Zhuliang Yao 11
Zhenda Xie 11 Han Hu 22 and Baining Guo 22

###### Abstract

Image classification, which classifies images by pre-defined categories, has
been the dominant approach to visual representation learning over the last
decade. Visual learning through image-text alignment, however, has emerged to
show promising performance, especially for zero-shot recognition. We believe
that these two learning tasks are complementary, and suggest combining them
for better visual learning. We propose a deep fusion method with three
adaptations that effectively bridge two learning tasks, rather than shallow
fusion through naïve multi-task learning. First, we modify the previous common
practice in image classification, a linear classifier, with a cosine
classifier which shows comparable performance. Second, we convert the image
classification problem from learning parametric category classifier weights to
learning a text encoder as a meta network to generate category classifier
weights. The learnt text encoder is shared between image classification and
image-text alignment. Third, we enrich each class name with a description to
avoid confusion between classes and make the classification method closer to
the image-text alignment. We prove that this deep fusion approach performs
better on a variety of visual recognition tasks and setups than the individual
learning or shallow fusion approach, from zero-shot/few-shot image
classification, such as the Kornblith 12-dataset benchmark, to downstream
tasks of action recognition, semantic segmentation, and object detection in
fine-tuning and open-vocabulary settings. The code will be available at
https://github.com/weiyx16/iCAR.

##  1 Introduction

Image classification, a long-standing vision problem, has played as an
important driving force for the remarkable success of deep learning in
computer vision. The deep representations learnt through this task, such as
using the ImageNet-1K dataset [8] which involves 1,000 object categories, have
been transferred to various vision tasks such as object detection, semantic
segmentation, video classification, etc., to significantly advance relevant
fields [14, 35, 50].

Recently, visual learning through image-text alignment has received more and
more attention. The image-text alignment task treats an image and its
associated alt-text as a positive pair and the image with all other alt-texts
as negative ones. By contrasting positive and negative pairs, it learns visual
representations and associates images with arbitrary semantics. This approach
has been shown to seize the strong zero-shot classification capability [40]
and learn good visual representations [21].

Figure 1: An illustration of our proposed approach to bring image
classification task (see a) to image-text alignment task (see b). From the
perspective of training loss, classifier type, and label/input granularity, we
modify the linear classifier to a cosine classifier (see a.2), introduce a
text encoder as a meta classifier (see a.3) and enrich each ambiguous category
name with a meaningful description (see a.4). After the re-formulation, we are
able to deeply unify two tasks and benefit from both.

In our view, these two learning methods have their own strengths and are
essentially complementary. For example, annotations in image classification
tasks are often precise, compact and consistent, and embody good properties
for visual representation learning. However, there are also drawbacks, such as
relatively small public available datasets and insufficient concepts coverage
that categories are often limited to a pre-defined set, inaccurate or
ambiguous class names (14.6% of classes in ImageNet-22K have repetitions in
class names, but each class represents a completely different visual concept,
shown in Figure 2). These shortcomings of image classification in visual
learning can be well complemented by the image-text alignment task, which can
have good concepts coverage, semantically-rich sentences, and large-scale
datasets with lower collection costs such as the Laion dataset including 400
million image-text pairs [44]. Conversely, the disadvantage of the image-text
alignment task is that each image is described by a noisy and non-compact
sentence. And it can be supplemented by a clean and compact image
classification task.

While we note a straightforward solution is to combine both tasks within a
naïve multi-task learning framework, we investigate a deep fusion approach
that effectively unifies the two tasks from the perspective of training loss,
classifier type, and label/input granularity. First, we modify the previous
common practice of using a linear classifier in image classification by a
cosine classifier that shows competitive performance. Second, we convert the
image classification problem from learning parametric classifier weights to
learning a text encoder that acts as a meta network to generate category
classifier weights. The image classification task and the image-text alignment
task now perform in the same embedding space, with cosine distance as the
metric, and the features are extracted from shared visual/text encoders.
Third, we enrich each class name with a description to avoid misconceptions
between classes and bring the classification method closer to the image-text
alignment approach regarding the input granularity.

For example, after enrichment, “a photo of a night bird, any bird associated
with night: owl, nightingale, nighthawk” has a similar granularity and
expression, compared to a caption/sentence, and also provides more detailed
information than the given class name “night bird”. Figure 1 is an
illustration on the adaptations of the image classification task to the image-
text alignment task, so that they can be deeply fused.

This deep fusion framework that combines image classification and image-text
alignment allows for a wide range of applications, from straight-forward tasks
such as image classification and image-text retrieval that align with the
learning objectives in zero-shot/few-shot settings, to transferring to various
vision tasks such as video action recognition, semantic segmentation, and
object detection through fine-tuning. In particular, the fine-tuning is
conducted not only on the vision network as before, but also on the text
encoder, which empirically performs better, especially for down-stream tasks
that require adjusting the text space to distinguish fine-grained categories.

For evaluation of the straight-forward tasks of image classification and
image-text retrieval, we first experiment with Conceptual Captions [46] and
ImageNet-1K [8].

Our deep fusion approach performs better than the individual learning or
shallow fusion approach, on both the zero-shot and few-shot settings of the
Kornblith 12-dataset benchmark [26]. This indicates that the deep fusion
approach can better benefit from the strengths of both tasks. When using the
training datasets of Laion-400M [44] and ImageNet-22K [8], iCAR can surpass
the previous state-of-the-art approaches in both zero-shot and few-shot
settings for the standard 12-dataset benchmark, including CLIP [40] and ALIGN
[21].

With fine-tuning, we prove that the proposed approach has strong
representation learning and open-vocabulary recognition capabilities when
transferred to other vision tasks. The approach achieves 52.5 mIoU on ADE-20K
validation set, by using a MaskFormer [5] framework and the Swin-B backbone,
0.4/0.6 higher than previous methods based solely on image-classification or
image-text. What’s more, there are huge improvements in open-vocabulary
semantic segmentation performance compared to using the pre-trained models
directly, such as 47.7 mIoU v.s. 21.3 mIoU on PASCAL-VOC [11], 53.8 mIoU v.s.
10.0 mIoU on Cityscapes [6] and 14.7 mIoU v.s. 2.1 mIoU on COCO Stuff [31].

##  2 Related Work

#### Visual recognition with multi-way discrete classifier

Visual recognition is widely used in numerous vision tasks with different
recognition granularity. Previous dominated methods all adopt the multi-way
parametric classifier with the softmax cross-entropy loss, such as image-level
classification [28, 49, 47, 18, 54, 10, 33], object-level classification in
object detection [14, 42, 17, 1], pixel-level classification in
semantic/instance segmentation [35, 4, 56], video-level action classification
[50, 2, 39, 12, 34] and etc. Some recent approaches [53, 24] try to explore
the non-parametric classifier for supervised classification, which performs
the contrastive learning to attract the images within the same class and repel
those belonging to different classes inside each batch. However, all these
approaches aren’t aware of the semantic meaning of each class, thus they can
not classify images belonging to a newly-giving set of classes, if these
classes do not have a perfect match to the training classes.

#### Visual-linguistic alignment

Leveraging natural language as supervision from image-text pairs for joint
visual-linguistic representation learning is a long studied research topic
[13, 38, 22, 15, 9, 43, 58, 40, 21]. DeViSE [13] firstly introduces label
semantic embedding to refactor image embedding after context-free multi-way
classification. ICMLM [43] and VirTex [9] drive the representation learning by
predicting masked words in a sentence from corresponding image embedding in an
autoregressive way. And ConVIRT [58] conducts visual-linguistic contrastive
learning in medical-related datasets for task-specific visual representations.
Due to the increasing of computation during these years, the used datasets are
expanded from the small-scale datasets (e.g., Flickr [57], COCO Captions [31],
Conceptual Captions 3M [3]) to web-scale ones (e.g., CLIP [40], ALIGN [21],
Laion [44]). There are two pioneer works, CLIP [40] and ALIGN [21], which
leverage the web-scale datasets with noisy image-alt-text pairs from the
Internet and thus cover concepts with unlimited number during the training of
image-text matching. However, the crawled image-alt-text dataset is noisy,
somewhat downgrading the visual recognition performance.

To the best of our knowledge, our paper is the first trial to deeply bridge
the supervised classification and image-text alignment tasks, trying to design
a new unified learning framework to benefit both. And we demonstrate that,
with careful designs, image classification and image-text alignment could
complement each other, and the proposed iCAR could significantly outperform
the single-task baselines and the shallow fusion approach with separate task
heads.

In addition, there are still few study on how to leverage these pre-trained
models for open-vocabulary recognition in downstream tasks. In this paper, we
present an extremely simple but effective baseline, to both load pre-trained
visual encoder as backbone and the text encoder as visual classifier during
fine-tuning on downstream tasks. This technique incurs competitive results on
fine-tuned datasets, and remarkably good performance on open-vocabulary
recognition for unseen datasets of the fine-tuned task.

##  3 Approach

###  3.1 Preliminaries

Image classification Given a set of $<$image, category label index$>$ pairs,
$\mathcal{D}=<I_{i},C_{i}>_{i=1}^{|\mathcal{D}|}$, image classification task
targets to predict the category label of a given image, usually through a
visual encoder $f_{v}$, and a parametric category classifier $h_{c}$. The
visual encoder transforms each raw image $I_{i}$ to an embedding $v_{i}$ in
the feature space, and the classifier predicts the logit distribution $p_{i}$
over all pre-defined $N$ categories in $\mathcal{D}$, i.e. 1000 categories in
ImageNet-1K dataset, from the embedding $v_{i}$. In most cases, the parametric
category classifier $h_{c}$ is a weight matrix $W\in\mathcal{R}^{N\times H}$,
where $H$ is the dimension of $v_{i}$ (for simplicity, bias term of $h_{c}$ is
omitted). The logits $p_{i}$ of all categories are the inner product between
$W$ and $v_{i}$, i.e. $p_{i}=W\cdot v_{i}$. Consider a given image $I_{i}$, a
cross-entropy loss is applied between $p_{i}$ and $C_{i}$, and a complete
formulation can be defined as:

$\mathcal{L}_{i}=-\log\frac{{\exp\left({W_{C_{i}}\cdot
f_{v}\left(I_{i}\right)}\right)}}{{\sum_{j=1}^{N}{\exp\left({W_{j}\cdot
f_{v}\left(I_{i}\right)}\right)}}},$ (1)

where $W_{j}$ is the parametric weight of $j$-th category.

Image-text alignment Given a set of $<$image, caption$>$ pairs,
$\mathcal{D}=<I_{i},T_{i}>_{i=1}^{|\mathcal{D}|}$, image-text alignment task
targets to close the distance of paired image-text but enlarge that of
unpaired ones, through a visual encoder $f_{v}$ and a text encoder $f_{t}$.
The visual encoder and text encoder transforms the image $I_{i}$ and the
caption $T_{i}$ into embeddings $v_{i}$ and $s_{i}$, respectively. InfoNCE
[51], a contrastive loss function is often applied to shrink the cosine
distance of $v_{i}$ and $s_{i}$. Consider an image embedding $v_{i}$, a
formulation of contrastive learning loss is defined as:

$\mathcal{L}_{i}=-\log\frac{{\exp\left(\cos\left(f_{t}\left(T_{i}\right),v_{i}\right)/\tau\right)}}{{\sum\limits_{{T_{j}}\in
T}{\exp\left(\cos\left(f_{t}\left(T_{j}\right),v_{i}\right)/\tau\right)}}},$
(2)

where $\cos(\cdot,\cdot)$ denotes the cosine similarity between two vectors,
$T$ is all the captions in batch, including one positive paired caption and
$|T|-1$ negative ones, and $\tau$ is the temperature hyper-parameter to scale
the logits.

###  3.2 Bridging Image classification and Image-text alignment

To bridge image classification task and image-text alignment task for better
unification, we propose three adaptations to align the training loss, unify
the classifier and minimize the label granularity gap.

Cosine classifier As the formulation in Eqn. 1, the original image
classification loss is a cross-entropy loss on top of the inner product
similarity between the embedding $v_{i}$ and the parametric classifier
$h_{c}$. This formulation isn’t in line with the InfoNCE loss, shown in Eqn.
2, which is prevalent in image-text alignment task. We review the image
classification task from the perspective of metric learning, and formulate it
using a cosine classifier. To be more specific, we apply L2 normalization both
on the parametric category classifier $h_{c}$ and the embedding $v_{i}$, and
the optimization target is switched to maximize the cosine similarity of image
features and their corresponding class features. We also scale the logits
using a temperature $\tau$ to be consistent with InfoNCE and the cosine
classifier based image classification loss is like this:

$\mathcal{L}_{i}=-\log\frac{{\exp\left(\cos\left(W_{C_{i}},v_{i}\right)/\tau\right)}}{{\sum_{j=1}^{N}{\exp\left(\cos\left(W_{j},v_{i}\right)/\tau\right)}}},$
(3)

Figure 2: Statistical analysis on label names of 21843 categories in
ImageNet-22K dataset. Only 18657 categories use unique label names, and
thousands of categories have the same label names as at least one another
category.

We prove that a cosine classifier based approach can reach an on par
performance with the traditional classification method (see Table 1).

Text encoder as a meta network for image classification Re-formulating the
classification loss using a cosine classifier aligns the loss formats.
However, the label information from two tasks, categories and captions
respectively, are not shared between the parametric category classifier
$h_{c}$ and the text encoder $f_{t}$. As shown in Section 4.1, combining two
tasks shallowly leads to sub-optimal results, without benefiting much from the
accurate annotation in image classification and rich concepts and open-
vocabulary ability in image-text alignment.

To this end, we take label semantics into consideration and propose to utilize
the text encoder $f_{t}$ as a meta classifier for image classification.
Formally, given a pair of $<$image $I_{i}$, category label index $C_{i}>$, we
replace the $C_{i}$ with its corresponding class name $M_{i}$, for example,
tench for the 1st category in ImageNet-1K dataset. Furthermore, we adopt a
text encoder $f_{t}$ on $M_{i}$ to generate the classifier weight on-the-fly,
instead of optimizing a parametric category classifier $h_{c}$. The new
formulation is shown as:

$\mathcal{L}_{i}=-\log\frac{{\exp\left(\cos\left(f_{t}\left(M_{i}\right),v_{i}\right)/\tau\right)}}{{\sum_{j=1}^{N}{\exp\left(\cos\left(f_{t}\left(M_{j}\right),v_{i}\right)/\tau\right)}}}.$
(4)

Note that, integrating the text encoder into image classification enables
open-vocabulary ability and shrinks the gap between two tasks.

Enriched class name with description Replacing label index $C_{i}$ with label
name $M_{i}$ has largely bridged the image classification and image-text
alignment. To further minimize the label granularity gap between label names
(one or two words) and image captions (a complete sentence), we propose to
integrate the detailed description $D_{i}$ of each category. The description
can be found from corresponding synset in wordnet [37] for ImageNet dataset or
the beginning sentence of the corresponding Wikipedia page. We also add a
prompt to make the sentence more fluent. So, as shown in Figure 3, the label
for each category is formed through the following template: prompt
sentence+category name+description, and due to simplicity and similarity, we
annotate it as $T_{i}$. The formulation of our proposed image classification
framework is shown as

$\mathcal{L}_{i}=-\log\frac{{\exp\left(\cos\left(f_{t}\left(T_{i}\right),v_{i}\right)/\tau\right)}}{{\sum_{j=1}^{N}{\exp\left(\cos\left(f_{t}\left(T_{j}\right),v_{i}\right)/\tau\right)}}}.$
(5)

Figure 3: The detailed illustration of our unified training method. iCAR
performs both the image classification (in green) and image-text alignment (in
blue) in a unified contrastive learning scheme. _Best viewed in color_.

The detailed description enables a deeper understanding of each category and
reduces the misconception error, especially when only the class name is known
without further details. For example, in ImageNet-22K dataset, at least 6
categories are labeled as _jack_ but representing 6 various visual concepts,
e.g., one means any of several fast-swimming predacious fishes of tropical to
warm temperate seas, like Almaco jack, and another one means a tool for
exerting pressure or lifting. As shown in Figure 2, over 1,000 class names in
ImageNet-22K dataset repeat twice and hundreds repeat three times or more.
Also, the corresponding description with its class name would have similar
granularity to captions, and thus bring the classification method closer to
the image-text alignment.

###  3.3 A unified framework

We propose three methods to bridge image classification task and image-text
alignment task, from the perspective of training loss, classifier type and
label/input granularity. The image classification is re-formulated in Eqn. 5,
and it is finally in line with the InfoNCE loss (see Eqn. 2) in image-text
alignment.

Here, we present the unified contrastive learning loss, which is added on top
of the visual and text encoders, to simultaneously perform the tasks of image
classification and image-text alignment. A detailed illustration of our
unified contrastive learning loss is in Figure 3. A general formulation is
defined as:

$\small
L\left(\mathcal{D}\right)=-\frac{1}{|\mathcal{D}|}\sum\limits_{({I_{i}},{T_{i}})\in\mathcal{D}}{\log\frac{{\exp\left({\cos\left({f_{t}}\left({{T_{i}}}\right),{f_{v}}\left({{I_{i}}}\right)\right)/\tau}\right)}}{{\sum\limits_{{T_{j}}\in
T}{\exp\left({\cos\left({f_{t}}\left({{T_{j}}}\right),{f_{v}}\left({{I_{i}}}\right)\right)/\tau}\right)}}}},$
(6)

where $\mathcal{D}$ is the combination of image-text alignment and image
classification datasets. Notice that, as mentioned in section 3.2, we discard
the original category label index $C_{i}$ in image classification dataset and
adopt a complete sentence $T_{i}$, including prompt sentence+category
name+description, as the annotation for each category. $T$ in Eqn. 6 is the
whole text set for contrastive learning (e.g., all captions in batch or all
class labels with descriptions), $f_{v}(\cdot)$ and $f_{t}(\cdot)$ denotes the
visual and text encoders and $\cos(\cdot,\cdot)$ denotes cosine similarity
metric. $\tau$ is the temperature hyper-parameter to scale the logits as in
InfoNCE.

###  3.4 Applications

Figure 4: In our evaluation, we also integrate the pre-trained text encoder as
a meta visual classifier in every scenarios. We find it serves as a good
initialization and helps few-shot learning, and enables open-vocabulary
recognition in downstream tasks.

After the unified training, we evaluate our model on numerous applications,
such as _zero-shot evaluation_ , _few-shot learning_ , and _fine-tuning on
downstream tasks_ (e.g., semantic segmentation, action recognition, and object
detection). As shown in Figure 4, for applications, our major philosophy is to
load not only the visual encoder as backbone (as blue) but also the text
encoder as the meta visual classifier (as red) from the pre-trained model.

For few-shot learning, as the number of training samples for each class is
very small (e.g., 1 or 4 samples per class), the text encoder could serve as a
good initial classifier. This would alleviate the situation in [40] that the
performance of the few-shot setting is even worse than that of the zero-shot
setting.

For other downstream tasks, previous approaches adopt the task specific
framework with only the pre-trained visual encoder as initialization, and use
a discrete multi-way classifier on top for recognition. Here we replace the
original classifier with the pre-trained text encoder to generate the weight
of each class with class names as input. After fine-tuning, we find that the
models have surprisingly good open-vocabulary recognition capability on unseen
datasets of the same fine-tuning task (see Section 4.3). The reason may be
that the text encoder still maintains part of open-vocabulary capability
learned in the pre-training.

##  4 Experiments

###  4.1 Ablation Study

Setup We use Conceptual Cations 3M (CC) [46] as image-text alignment dataset
and ImageNet-1K (IN-1K) [8] as image classification dataset. In all our
ablation studies, we adopt a Swin-tiny [33] as our visual encoder and a
RoBERTa-base [32] as our text encoder. We train the models for 100 epochs on
IN-1K (or equivalently on CC) with a batch size of 128 per GPU on 8 V100 GPUs
in total. For approaches utilizing both datasets, we sample half number of
images from both in each mini-batch. An AdamW optimizer [25] with a base
learning rate of 2e-4, weight decay of 0.01, RandAugment [7] and stochastic
depth [20] of 0.1 are used. The learning rate schedule is composed of a linear
warm-up for 5 epochs and a cosine learning rate decay. For direct/zero-shot
classification, we adopt several variants of IN-1K validation sets as the
benchmark for comprehensive understanding, such as IN, IN-V2 [41], IN-
Rendition (IN-R) [19], IN-Sketch (IN-S) [52], and also the widely-used
Kornblith 12-dataset benchmark. We follow CLIP [40] on the test set splits and
evaluation metrics, and test the zero-shot classification ability of the pre-
trained model. We borrow the same prompt list as CLIP and ensemble the results
for fair comparison. For few-shot classification, we follow [40] to adopt
Kornblith 12-dataset benchmark. For zero-shot cross-modal retrieval, we
evaluate the models on Flickr and MSCOCO with the standard settings in [40].

Ablation: cosine classifier Firstly, we compare the cosine classifier in Eqn.
3 (with temperature $\tau$ as 0.05) to original linear classifier in Eqn. 1 on
IN-1K image classification task.

In Table 1, we observe that the cosine classifier performs competitive to an
inner product based classifier on all validation sets, and thus it supports us
to bridge image classification task to image-text alignment task using a
cosine classifier by default.

| ImageNet-Related
---|---
Method | IN | IN-V2 | IN-R | IN-S
Linear Classifier | 80.9 | 69.5 | 42.9 | 29.4
Cosine Classifier | 81.5 | 69.9 | 43.5 | 31.1
Table 1: Ablation study with the cosine classifier in image classification
task on ImageNet related benchmarks (top-1 accuracy).

Ablation: text encoder as a meta classifier Here we ablate whether to use text
encoder as a meta classifier for image classification, in a joint learning
setting which both performs image classification on IN-1K and image-text
alignment on CC. Our results are in Table 2. Both _Split head_ and _Text
encoder_ approaches adopt Eqn. (2) for image-text alignment. But _Split head_
adopts Eqn. (1) for image classification, which means that it performs two
tasks in separate heads. _Text encoder_ adopts Eqn. (4) for image
classification, which utilizes the text encoder as a meta classifier and
shares the text encoder among two tasks.

As shown in Table 2, we observe that adopting text encoder could benefit from
both tasks to achieve better performance on zero-shot classification of IN-R
and IN-S, and zero-/few-shot classification on the 12-dataset benchmarks.

| ImageNet-Related | 12-dataset Avg.
---|---|---
Method | IN | IN-V2 | IN-R | IN-S | 0 Shot | 1 Shot | 4 Shot | 16 Shot
Split head-Sup | 80.6 | 69.1 | 49.2 | 38.3 | - | - | - | -
Split head-Text | 45.0 | 38.5 | 41.8 | 24.7 | 35.1 | 42.7 | 54.9 | 65.4
Text encoder (w/o Desc.) | 80.5 | 69.1 | 49.8 | 38.6 | 37.7 | 44.2 | 55.8 | 66.1
Text encoder (w. Desc.) | 80.4 | 69.1 | 49.7 | 38.7 | 39.1 | 46.3 | 57.0 | 66.9
Table 2: Ablation study on whether adopting text encoder and integrating
description for image classification on ImageNet series and zero-/few-shot
evaluation on 12-dataset benchmark (top-1 accuracy).

Ablation: enriched category name with description Here we ablate whether to
enrich class names with descriptions in a joint learning setting, using Eqn.
(4) (w/o Desc.) and Eqn. (5) (w. Desc.) for image classification task,
respectively. Besides evaluating on zero-/few-shot classification, we also
perform this ablation study on the cross-modal retrieval tasks of Flickr and
MSCOCO.

We observe that enriching each class name with its description could reduce
the misalignment in the class names to benefit the classification capability
(1.4+ on averaged accuracy of 12-dataset benchmark in Table 2), and bridge the
input gap of two tasks to benefit the cross-modal retrieval or image-text
alignment capability (see Table 3).

| Text Retrieval | Image Retrieval
---|---|---
| Flickr | MSCOCO | Flickr | MSCOCO
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
w/o Desc. | 48.7 | 79.9 | 88.5 | 25.3 | 50.5 | 62.9 | 40.5 | 70.3 | 80.7 | 17.9 | 39.8 | 51.7
w Desc. | 51.4 | 81.7 | 89.0 | 28.2 | 53.0 | 65.7 | 41.0 | 71.2 | 81.1 | 20.4 | 44.2 | 56.4
Table 3: Ablation study on whether enriching category names with descriptions of zero-shot cross-modal retrieval performance on MSCOCO and Flickr. | ImageNet-Related | 12-dataset Avg.
---|---|---
Method | IN | IN-V2 | IN-R | IN-S | 0 Shot | 1 Shot | 4 Shot | 16 Shot
Sup-only | 80.9 | 69.5 | 42.9 | 29.4 | - | 34.4 | 53.5 | 65.1
VL-only | 32.4 | 27.7 | 34.4 | 18.3 | 31.4 | 35.7 | 47.5 | 58.3
iCAR | 80.5 | 69.1 | 49.8 | 38.6 | 39.1 | 46.3 | 57.0 | 66.9
Table 4: Ablation study on sing-task baselines of direct evaluation results on
ImageNet series and zero-/few-shot evaluation on 12-dataset benchmark.

Ablation: single-task baseline Here we compare the proposed approach with two
single task baselines, supervised-only (Eqn. (1)) and VL-only (Eqn. (2)) on
zero-/few-shot classification on IN-related and 12-dataset benchmarks. From
Table 4, our approach could perform competitively or significantly better than
two single-task baselines, which indicates that our deep fusion approach could
well absorb the strengths of these two tasks and outperform both.

| Visual Encoder | ImageNet-Related | K. 12-dataset
---|---|---|---
Method | input | #param. | FLOPs | IN | IN-V2 | IN-R | IN-S | Average
CLIP-ViT-B/16 [40] | $224^{2}$ | 86M | 18.9G | 68.6 | 61.9 | 76.4 | 46.6 | 68.8
CLIP-ViT-L/14 [40] | $336^{2}$ | 307M | 190.7G | 76.2 | 70.1 | 88.9 | 60.2 | 75.5
ALIGN-EffNet-L2 [21] | $360^{2}$ | 480M | 92.2G | 76.4 | 70.1 | 92.2 | - | -
Sup-only | $224^{2}$ | 88M | 15.4G | 82.6 | 73.1 | 56.5 | 42.0 | -
VL-only | $224^{2}$ | 88M | 15.4G | 60.1 | 55.1 | 73.6 | 51.1 | 65.6
iCAR | $224^{2}$ | 88M | 15.4G | 82.9 | 73.7 | 76.7 | 59.8 | 70.6
Table 5: System-level comparison with the state-of-the-art approaches (CLIP
and ALIGN) of direct evaluation results (top-1 accuracy) on ImageNet series
(IN, IN-V2, IN-Rendition, IN-Sketch) and zero-shot evaluation results (top-1
accuracy) on the Kornblith 12-dataset benchmark.

###  4.2 System-level Comparison

Setup We adopt Laion-400M [44] as the image-caption dataset and ImageNet-22K
[8] as the image-label dataset. Both these datasets are open-sourced, thus all
the experiments in our paper is reproducible for the whole community. In all
experiments, we adopt a base version of Swin Transformer with input size
224$\times$224 and window size 7 as the visual encoder, and a base version of
RoBERTa [32] model as the text encoder. More implementation details are shown
in Appendix 0.A.

Direct/Zero-shot Classification We evaluate our model on ImageNet-1K [8] with
several variants of validation sets [41, 19, 52] and Kornblith 12-dataset
benchmark [26]. Table 5 presents the comparisons to the state-of-the-art
models, CLIP [40] and ALIGN [21]. Compared to CLIP ViT-B/16 model which has
similar number of parameters and FLOPs, our iCAR gets over 12% improvements on
IN, IN-V2 and IN-S and obtains slightly better performance on IN-R. Achieving
better performance on IN and IN-V2 is reasonable because IN-22K contains
plenty of images in IN-1K. The results on ImageNet variants are higher or at
least on-par with the largest model of CLIP and ALIGN, which has either ten
times of FLOPs or five times of parameters. Besides, we find that a unified
combination of image classification and image-text alignment empowers the
model to learn knowledge from both tasks and datasets, better than strong
supervised-22K and VL-only baseline on all the ImageNet related validation
sets.

When evaluating the models on Kornblith 12-dataset classification benchmark
[26] which covers a wide range of visual recognition domains, our model
outperforms CLIP ViT-B/16 model by 1.8% on averaged accuracy of 12 datasets.
On some fine-grained datasets, like DTD (Texture) and Flowers102 (Flowers),
our model is better than the biggest CLIP ViT-L/14 (336$\uparrow$) model. And
on more general datasets, like CIFAR100 and Caltech101, our models can still
achieve on-par performance with a way larger CLIP model. Compared to VL-only
baseline, our model can achieve a huge gain of 5.0% on average, due to a
unified contrastive learning during pre-training. The detailed results are
shown in Appendix 0.B.1.

Figure 5: Major comparison with the state-of-the-art approach CLIP of few-shot
classification (top-1 accuracy) on the Kornblith 12-dataset benchmark. $\star$
denotes the zero-shot performance of different approaches. Results of CLIP-
ViT-B/16 on few-shot classification are reproduced using released model.

Few-shot Classification We compare our model with CLIP under the few-shot
classification scenario with the visual encoder frozen in Figure 5. We follow
[40] to train a linear probe CLIP with a careful grid search on hyper-
parameters. We notice that in CLIP, the performance of few-shot classification
cannot catch up with that of zero-shot classification, unless more than 4
examples per class are given. We assume that it is because the number of
training samples is not enough for training a randomly initialized classifier.
This situation can be alleviated by fine-tuning with the pre-trained text
encoder which serves as a better initialization, and this method closes the
gap between pre-training and fine-tuning. We evaluate our method on Kornblith
12-dataset benchmark. We run every experiments three times and the averaged
results are reported.

When only one example per class is given, by utilizing text encoder as the
classifier, our iCAR achieve 73.9% on 12-dataset average, surpassing the CLIP
model by 29.5%. And our model earns 3.3% better than our zero-shot baseline
which already has a strong performance. Even trained with 16 examples per
class, our model can still surpass CLIP method by 4.1%. Compared to
supervised-only model and visual-linguistic only model, our unified
contrastive learning pre-trained model is 24.6% and 6.1% better respectively
under one-shot setting, and the advantage keeps to 16 shots learning with
still 2.7% and 5.0% better.

We also evaluate our model on zero-shot cross-modal retrieval benchmarks of
Flickr-30K (1K test split) and MSCOCO (5K test split). Our approach achieves
highly competitive results with CLIP. We list our results in Appendix 0.B.2.

###  4.3 Fine-tuning on Downstream Tasks

| Evaluation Dataset (val mIoU)
---|---
Method (zero-shot) | ADE-20K | Cityscapes | VOC | COCO Stuff
CLIP | 1.7 | 0.8 | 12.6 | 1.0
VL-Laion | 8.9 | 6.7 | 22.2 | 2.1
iCAR | 9.7 | 10.0 | 21.3 | 2.1
|  Ft. Dataset | Open-vocabulary Dataset
Method | ADE-20K | Cityscapes | VOC | COCO Stuff
Sup-only-Ft. | 52.1 | - | - | -
VL-only-Ft. | 51.9 | - | - | -
iCAR-Ft. (visual enc. only) | 52.6 | - | - | -
iCAR-Ft. | 52.5 | 53.8 | 47.7 | 14.7
Table 6: Results of zero-shot evaluation (val mIoU) on four semantic
segmentation benchmarks, and open-vocabulary performance (val mIoU) on three
of four datasets (Cityscapes, VOC, COCO Stuff) when performing the text
encoder included fine-tuning on ADE20K with MaskFormer as default framework.
iCAR-Ft. (visual enc. only) denotes fine-tuning only the visual encoder of the
pre-trained model. iCAR-Ft. denotes further integrating text encoder as a
classifier during fine-tuning.

Semantic Segmentation We conduct the experiment mainly on the widely-used
ADE-20K [59] dataset, including 150 categories and 25K images. We utilize
MaskFormer [5] as our base framework and adopt its default training recipe
except for setting window size to 7. MaskFormer is a two-stage framework for
segmentation and therefore suitable for our situation. We replace the
classifier with the pre-trained text encoder, i.e. we generate the classifier
weight on-the-fly by feeding each class name and a prompt into the text
encoder. For generated masks which should be matched to the empty set
$\varnothing$, we add a special category - ”background”. We fine-tune our pre-
trained model on ADE-20K dataset and conduct open-vocabulary semantic
segmentation on Pascal VOC [11], Cityscapes [6] and COCO Stuff [31]. We
compare our approach with zero-shot baselines and the baseline fine-tuning
visual backbone only, which preserves original multi-way classifier and has no
ability for open-vocabulary segmentation. For zero-shot baselines, we extract
the feature map after the last stage and conduct a point-wise classification
following an interpolation to the original image size.

Table 6 shows single scale testing results of validation mIoU in different
datasets. We find that fine-tuning with the text encoder can achieve an on-par
result compared to the baseline which directly uses 151-way classification on
ADE-20K (52.5 v.s. 52.6). Furthermore, our method shows surprisingly good
transferability on other open-vocabulary segmentation datasets, achieving 47.7
mIoU on Pascal VOC, 53.8 mIoU on Cityscapes, 14.7 mIoU on COCO Stuff. Compared
to the best zero-shot baseline, our method incurs huge improvements, with
+26.4, +43.8, +12.5 on these three datasets, respectively, showing the
surprisingly good open vocabulary capability of our model.

| Ft. Dataset | Open-vocabulary Dataset
---|---|---
Method |  LVIS (Rare) | COCO | Objects365 | VOC
Sup-only-Ft. | 35.9 (25.2) | - | - | -
VL-only-Ft. | 36.3 (25.9) | - | - | -
iCAR-Ft. (visual enc. only) | 37.9 (29.0) | - | - | -
iCAR-Ft. | 38.5 (30.8) | 41.2 | 19.7 | 76.4
Table 7: Results of open-vocabulary performance (box AP on validation set) on
three object detection datasets (COCO, Objects365, Pascal VOC) when performing
the text encoder included fine-tuning on LVIS with Faster R-CNN as the default
framework.

Object Detection We conduct the object detection experiment on LVIS v1 [16]
with Faster R-CNN [42] and FPN [30] as framework. LVIS includes over 1200
object categories with an unbalanced distribution. Based on the frequency, all
the categories are splitted into three sets: f(requency),c(ommon),r(are).
Similar to the setting in semantic segmentation, we replace bounding box
classification head with the pre-trained text encoder. But we do not add in a
special “background” category, instead we keep it as a pure trainable
embedding. We fine-tune our model for 2x schedule (24 epochs) on LVIS dataset,
with multi-scale training (shorter size between 480 and 800) and a cosine
learning rate schedule mostly following the recipe from [33].

After fine-tuning, we conduct open-vocabulary detection on COCO [31],
Objects365 [45] and Pascal VOC [11]. We test with a single scale input (1333,
800) and report box mAP on validation set, except that for VOC, we report box
AP50 and use a default (1000, 600) input.

Table 7 shows the comparison results of our approach with baselines on four
datasets. When fine-tuning on LVIS with the text encoder as visual classifier,
our approach obtains 41.2 box mAP on COCO without any annotations, which is
only 7.4 points lower than fully supervised fine-tuning baseline with IN-22K
pre-trained model as initialization. Besides, we also achieve 19.7 mAP on O365
and 76.4 AP50 on VOC. In addition, our approach gains +0.6 mAP better on LVIS
(38.5 mAP v.s. 37.9 mAP) than baseline, which has a 1204-way classifier.

| Evaluation Dataset (top-1 accuracy)
---|---
Method (zero-shot) | K400 | K600 | UCF101 | HMDB51
CLIP | 51.3 | 47.7 | 67.4 | 46.1
VL-Laion | 40.5 | 37.5 | 56.5 | 36.0
iCAR | 46.4 | 43.7 | 61.4 | 41.2
|  Ft. Dataset | Open-vocabulary Dataset
Method | K400 | K600 | UCF101 | HMDB51
Sup-only-Ft. | 82.7 | - | - | -
VL-only-Ft. | 82.1 | - | - | -
iCAR-Ft. (visual enc. only) | 83.1 | - | - | -
iCAR-Ft. | 83.2 | 59.5 | 73.0 | 45.4
Table 8: Results of zero-shot evaluation (top-1 accuracy) on four video action
recognition benchmarks, and open-vocabulary evaluation (top-1 accuracy) on
three of four datasets (Kinetics-600, UCF101, HMDB51) when performing the text
encoder included fine-tuning on Kinetics-400 with Video-Swin as default
framework. The zero-shot result of CLIP on UCF101 is reproduced with released
model.

Video Action Recognition We also evaluate our approach on the video action
recognition task, following the same recipe in Video Swin Transformer [34]
except that we integrate the pre-trained text encoder as our classifier. We
fine-tune our model on Kinetics-400 (K400) [23] dataset for 30 epochs and
conduct open-vocabulary recognition on three other datasets, Kinetics-600
(K600) [23], UCF101 [48] and HMDB51 [29]. K400 consists of 400 human action
categories and is widely used in video action recognition. The overlap videos
between K600 validation set and K400 training set are carefully removed. We
also compare with several zero-shot baselines following the instruction
mentioned in CLIP [40].

Table 8 presents top-1 accuracy of each method on four datasets. Our model
fine-tuned on K400 earns 59.5% top-1 accuracy on K600, surpassing our zero-
shot baseline for +15.8%. To deeply understand this result, we disentangle the
good performance on K600, and create a new split named Kinetics-232 from the
validation set of K600, containing the classes which are not overlapped with
the ones in K400. We discover that our model after fine-tuning could still
perform relatively well on this split full of unseen classes in fine-tuning,
with only 11.4% accuracy dropped compared to zero-shot baseline. With benefits
from the other 368 classes, our model after fine-tuning reaches higher on
K600. On the other two datasets, our model reaches 73.0% and 45.4% on UCF101
and HMDB51 respectively, with +11.6% and +4.2% better than zero-shot baseline,
which reveals the open-vocabulary generalization ability of our approach. And
our approach performs equally with standard fine-tuning approach on K400
(83.2% v.s. 83.1%), which loads the visual encoder only and adds a multi-way
parametric classifier for classification.

##  5 Conclusion

In this paper, we present a new approach for bridging the image classification
task with the image-text alignment task, from the perspective of training
loss, classifier type and label granularity. The deep unification could help
the model benefit from both tasks and achieve significantly better performance
than the single-task baselines or the simple joint training baseline with
separate task heads. The effectiveness of the proposed approach is verified on
a wide range of tasks, such as zero-/few-shot classification on ImageNet
related benchmarks and Kornblith 12-dataset benchmarks, and fine-tuning on
three representative downstream tasks of both close-set and open-vocabulary
scenarios.

## References

  * [1] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European Conference on Computer Vision. pp. 213–229. Springer (2020)
  * [2] Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6299–6308 (2017)
  * [3] Changpinyo, S., Sharma, P., Ding, N., Soricut, R.: Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In: CVPR (2021)
  * [4] Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)
  * [5] Cheng, B., Schwing, A.G., Kirillov, A.: Per-pixel classification is not all you need for semantic segmentation. arXiv (2021)
  * [6] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3213–3223 (2016)
  * [7] Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.V.: Randaugment: Practical automated data augmentation with a reduced search space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 702–703 (2020)
  * [8] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [9] Desai, K., Johnson, J.: Virtex: Learning visual representations from textual annotations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11162–11173 (2021)
  * [10] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2021), https://openreview.net/forum?id=YicbFdNTTy
  * [11] Everingham, M., Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. Int. J. Comput. Vision 88(2), 303–338 (Jun 2010). https://doi.org/10.1007/s11263-009-0275-4, https://doi.org/10.1007/s11263-009-0275-4
  * [12] Feichtenhofer, C., Fan, H., Malik, J., He, K.: Slowfast networks for video recognition. In: Proceedings of the IEEE international conference on computer vision. pp. 6202–6211 (2019)
  * [13] Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Ranzato, M.A., Mikolov, T.: Devise: A deep visual-semantic embedding model. In: Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems. vol. 26. Curran Associates, Inc. (2013), https://proceedings.neurips.cc/paper/2013/file/7cce53cf90577442771720a370c3c723-Paper.pdf
  * [14] Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation (2014)
  * [15] Gomez, L., Patel, Y., Rusiñol, M., Karatzas, D., Jawahar, C.: Self-supervised learning of visual features through embedding images into text topic spaces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4230–4239 (2017)
  * [16] Gupta, A., Dollar, P., Girshick, R.: Lvis: A dataset for large vocabulary instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5356–5364 (2019)
  * [17] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 2961–2969 (2017)
  * [18] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [19] Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., Gilmer, J.: The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV (2021)
  * [20] Huang, G., Sun, Y., Liu, Z., Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. In: European conference on computer vision. pp. 646–661. Springer (2016)
  * [21] Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q.V., Sung, Y., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:2102.05918 (2021)
  * [22] Joulin, A., Van Der Maaten, L., Jabri, A., Vasilache, N.: Learning visual features from large weakly supervised data. In: European Conference on Computer Vision. pp. 67–84. Springer (2016)
  * [23] Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al.: The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 (2017)
  * [24] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. arXiv preprint arXiv:2004.11362 (2020)
  * [25] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [26] Kornblith, S., Shlens, J., Le, Q.V.: Do better imagenet models transfer better? In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2656–2666 (2019). https://doi.org/10.1109/CVPR.2019.00277
  * [27] Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for fine-grained categorization. In: 2013 IEEE International Conference on Computer Vision Workshops. pp. 554–561 (2013). https://doi.org/10.1109/ICCVW.2013.77
  * [28] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems. pp. 1097–1105 (2012)
  * [29] Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: HMDB: a large video database for human motion recognition. In: Proceedings of the International Conference on Computer Vision (ICCV) (2011)
  * [30] Lin, T.Y., Dollar, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017)
  * [31] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
  * [32] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
  * [33] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. CoRR abs/2103.14030 (2021), https://arxiv.org/abs/2103.14030
  * [34] Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., Hu, H.: Video swin transformer. arXiv preprint arXiv:2106.13230 (2021)
  * [35] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation (2015)
  * [36] Maji, S., Kannala, J., Rahtu, E., Blaschko, M., Vedaldi, A.: Fine-grained visual classification of aircraft. Tech. rep. (2013)
  * [37] Miller, G.A.: WordNet: A lexical database for English. In: Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994 (1994), https://aclanthology.org/H94-1111
  * [38] Norouzi, M., Mikolov, T., Bengio, S., Singer, Y., Shlens, J., Frome, A., Corrado, G.S., Dean, J.: Zero-shot learning by convex combination of semantic embeddings. arXiv preprint arXiv:1312.5650 (2013)
  * [39] Qiu, Z., Yao, T., Mei, T.: Learning spatio-temporal representation with pseudo-3d residual networks. In: proceedings of the IEEE International Conference on Computer Vision. pp. 5533–5541 (2017)
  * [40] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision (2021)
  * [41] Recht, B., Roelofs, R., Schmidt, L., Shankar, V.: Do ImageNet classifiers generalize to ImageNet? In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 5389–5400. PMLR (09–15 Jun 2019), http://proceedings.mlr.press/v97/recht19a.html
  * [42] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems. pp. 91–99 (2015)
  * [43] Sariyildiz, M.B., Perez, J., Larlus, D.: Learning visual representations with caption annotations. In: European Conference on Computer Vision (ECCV) (2020)
  * [44] Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., Komatsuzaki, A.: Laion-400m: Open dataset of clip-filtered 400 million image-text pairs (2021)
  * [45] Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., Sun, J.: Objects365: A large-scale, high-quality dataset for object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (October 2019)
  * [46] Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Proceedings of ACL (2018)
  * [47] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (May 2015)
  * [48] Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes from videos in the wild. CoRR abs/1212.0402 (2012)
  * [49] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1–9 (2015)
  * [50] Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE international conference on computer vision. pp. 4489–4497 (2015)
  * [51] van den Oord, A., Li, Y., Vinyals, O.: Representation Learning with Contrastive Predictive Coding. arXiv e-prints arXiv:1807.03748 (Jul 2018)
  * [52] Wang, H., Ge, S., Lipton, Z., Xing, E.P.: Learning robust global representations by penalizing local predictive power. In: Advances in Neural Information Processing Systems. pp. 10506–10518 (2019)
  * [53] Wu, Z., Efros, A.A., Yu, S.X.: Improving generalization via scalable neighborhood component analysis. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 685–701 (2018)
  * [54] Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1492–1500 (2017)
  * [55] Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., Xu, C.: Filip: Fine-grained interactive language-image pre-training (2021)
  * [56] Yin, M., Yao, Z., Cao, Y., Li, X., Zhang, Z., Lin, S., Hu, H.: Disentangled non-local neural networks. In: Proceedings of the European conference on computer vision (ECCV) (2020)
  * [57] Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2, 67–78 (2014)
  * [58] Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747 (2020)
  * [59] Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., Torralba, A.: Semantic understanding of scenes through the ade20k dataset. International Journal on Computer Vision (2018)

##  Appendix 0.A Implementation Details

###  0.A.1 System-level Comparison

Datasets We adopt Laion-400M [44] as the image-caption dataset and
ImageNet-22K [8] as the image-label dataset. Laion-400M is the largest openly
available visual-linguistic dataset with 400 million pairs. It extracts image-
caption pairs in random web pages crawled between 2014 and 2021 from the
Common Crawl web data, and de-noises the dataset by scoring and filtering
numerous raw pairs with a threshold using a CLIP ViT-B/32 [40] model.
ImageNet-22K is the full set of ImageNet, containing 14.2 million images and
long-tailed 22K classes. Each class is a synset in wordnet [37] with a lemma
and a brief description. Both these datasets are open-sourced, thus all the
experiments in our paper is reproducible for the whole community. For the
Laion-400M, we follow [40] to train a near-duplicate detector via contrastive
learning, and remove the duplicate pairs between validation sets and the
training set of Laion-400M.

Implementation Details We train iCAR for 100K iterations, with a batch size of
192 per GPU on 64 V100. In each mini batch, we sample 64 images from
ImageNet-22K and 128 images from Laion-400M, so the model is trained on
classification dataset for 30 epochs and on image-caption dataset for 2 epochs
equivalently. We employ an AdamW [25] optimizer with the learning rate set to
1e-3 and the weight decay set to 0.05, and a cosine learning rate schedule is
adopted with a linear warmup for 16.7K iterations. We also add in other
regularization, including RandAugment [7], stochastic depth [20] of 0.2 and a
gradient clipping with a max norm of 5. For supervised training baseline (Sup-
only / Supervised-22K), we use the released version of Swin Transformer [33],
which is trained on ImageNet-22K dataset for 90 epochs. For image-text
alignment baseline (VL-only / VL-Laion), we train the same architecture as
iCAR for 2 epochs on Laion-400M with the same hyper-parameters.

Methods |  Food101 |  CIFAR10 |  CIFAR100 |  Birdsnap |  SUN397 |  Stanford Cars |  FGVC Aircraft |  VOC2007 |  DTD |  Oxford Pets |  Caltech101 |  Flowers102 |  Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---
CLIP-ViT-B/16 | 89.2 | 91.6 | 68.7 | 39.1 | 65.2 | 65.6 | 27.1 | 83.9 | 46.0 | 88.9 | 89.3 | 70.4 | 68.8
CLIP-ViT-L/14 | 92.9 | 96.2 | 77.9 | 48.3 | 67.7 | 77.3 | 36.1 | 84.1 | 55.3 | 93.5 | 92.6 | 78.7 | 75.1
CLIP-ViT-L/14 (336$\uparrow$) | 93.8 | 95.7 | 77.5 | 49.5 | 68.4 | 78.8 | 37.2 | 84.3 | 55.7 | 93.5 | 92.8 | 78.3 | 75.5
VL-Laion | 77.8 | 92.4 | 68.1 | 29.1 | 58.0 | 65.4 | 7.1 | 83.9 | 62.1 | 81.0 | 90.0 | 71.7 | 65.6
iCAR | 82.7 | 94.8 | 78.4 | 48.5 | 62.9 | 63.1 | 8.4 | 84.5 | 62.9 | 87.9 | 92.1 | 81.3 | 70.6
Table 9: Detailed comparisons of zero-shot classification with the state-of-
the-art approach CLIP on Kornblith 12-dataset classification benchmark [26].

##  Appendix 0.B Detailed Results

###  0.B.1 Zero-shot classification

We compare iCAR with our image-text alignment baseline (VL-Laion) and the
state-of-the-art approach CLIP [40] on Kornblith 12-dataset benchmark [26].
Table 9 presents the detailed results on each dataset. Compared to image-text
alignment baseline, iCAR reaches higher score in 11 out of 12 datasets except
Stanford Cars dataset [27] and the average improvement is 5.0%. And with the
help of a unified contrastive learning scheme combining supervised dataset and
visual-linguistic dataset, our model could generally perform better than the
state-of-the-art CLIP ViT-B/16 approach. The main performance gap between iCAR
and CLIP methods is on FGVC Aircraft dataset [36] and we guess that some
variants of the aircraft are a little old and not included in Laion-400M
dataset [44] which is collected from the websites later than 2014.

###  0.B.2 Zero-shot retrieval

We evaluate iCAR on zero-shot cross-modal retrieval benchmarks of Flickr-30K
[57] (1K test split) and MSCOCO [31] (5K test split). Here we follow CLIP [55]
to use the similar prompt engineering in this task. Table 10 shows our results
and our iCAR achieves comparable results with CLIP on image-to-text retrieval
task, surpasses CLIP on text-to-image retrieval task of MSCOCO, and remains a
performance gap on Flickr-30K.

| Text Retrieval | Image Retrieval
---|---|---
| Flickr | MSCOCO | Flickr | MSCOCO
Methods | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
CLIP-ViT-B/16 [40] | 82.1 | 96.6 | 99.0 | 52.4 | 76.7 | 84.6 | 62.2 | 85.7 | 91.9 | 33.1 | 58.4 | 69.0
VL-Laion | 78.5 | 95.3 | 97.8 | 51.7 | 76.8 | 85.1 | 61.3 | 85.4 | 91.5 | 35.6 | 61.0 | 71.9
iCAR | 81.5 | 95.8 | 98.5 | 52.1 | 76.4 | 85.3 | 59.9 | 85.5 | 91.2 | 35.6 | 61.1 | 71.6
Table 10: Comparisons of our approach and state-of-the-art methods on zero-
shot cross-modal retrieval performance on Flickr-30K and MSCOCO.

###  0.B.3 Few-shot classification

Figure 6 shows few-shot classification comparison on each dataset of Kornblith
12-dataset benchmark. With the pre-trained text encoder as an initialized meta
classifier, iCAR achieves higher performance under any-shot setting than zero-
shot classification. It proves that the gap between zero-shot and few-shot
learning in CLIP methods can be alleviated with the guidance of a pre-trained
text encoder. In most datasets and few-shot settings, iCAR surpasses
supervised-only, visual-linguistic only and CLIP baselines. We also notice a
performance gap on FGVC Aircraft [36] dataset. The reason may be that the text
encoder do not have a strong ability to distinguish the names of these
aircraft variants due to lacking of related data during pre-training, which
could also be observed in the zero-shot classification experiments.

Figure 6: Detailed comparisons with the state-of-the-art approach CLIP of few-
shot classification (top-1 accuracy) on each dataset of Kornblith 12-dataset
benchmark. $\star$ denotes the zero-shot performance of different approaches.
Results of CLIP ViT-B/16 on few-shot classification are reproduced using the
released model.


# Estimation and numerical validation of inf-sup constant for bilinear form
$(p,\mathrm{div}\ \bm{u})$

V. Jain<EMAIL_ADDRESS>M. Gerritsma  Delft University of Technology,
Faculty of Aerospace Engineering, P.O. Box 5058, 2600 GB Delft, The
Netherlands

###### Abstract

We give a derivation for the value of inf-sup constant for the bilinear form
$\left(p\;,\mathrm{div}\ \bm{u}\right)$. We prove that the value of inf-sup
constant is equal to 1.0 in all cases and is independent of the size and shape
of the domain. Numerical tests for validation of inf-sup constants is
performed using finite dimensional spaces defined in [1] on two test domains
i) a square of size $\Omega=[0,1]^{2}$, ii) a square of size
$\Omega=[0,2]^{2}$, for varying mesh sizes and polynomial degrees. The numeric
values are in agreement with the theoretical value of inf-sup term.

###### keywords:

stability, inf-sup constant

##  1 Introduction

Numerical schemes for mixed finite element methods should result in stable
bounded solutions. The stability of finite element discretization is governed
by the inf-sup criterion for bilinear forms. The objective of this paper is to
derive and validate the inf-sup stability constant for the bilinear form of
the divergence term. This is of importance because it often appears as
pressure constraint term in discretization of fluid mechanics equations. For
given $\bm{u}\in H\left(\mathrm{div};\Omega\right)$, $p\in
L^{2}\left(\Omega\right)$, the bilinear form is give by

$b\left(p,\bm{u}\right)=\left(p\;,\mathrm{div}\ \bm{u}\right)\;.$ (1)

The inf-sup constant $\beta$ for (1) is given by

$\beta=\inf_{p\in L^{2}\left(\Omega\right)}\sup_{\bm{u}\in
H\left(\mathrm{div};\Omega\right)}\frac{\left(p\;,\mathrm{div}\
\bm{u}\right)}{\|p\|_{L^{2}\left(\Omega\right)}\|\bm{u}\|_{H\left(\mathrm{div};\Omega\right)}}\;.$
(2)

Quite some work has been done to approximate the value of inf-sup constant see
for eg. [2, 3, 4]. In this work we measure the norm of the velocity field in
the space orthogonal to kernel of the divergence operator and prove that the
value of the inf-sup constant $\beta=1.0$. For numerical tests we use finite
dimensional spaces defined in [1]. The validation study is performed on three
different domains: i) unit square $\Omega=[0,1]^{2}$; ii) square
$\Omega=[0,2]^{2}$, for varying mesh sizes, $h$, and polynomial degrees
$N=1,2,3$. The numerical values are in agreement with the derived values of
$\beta$, with maximum errors shown of the order of $10^{-6}$.

##  2 Derivation of inf-sup constant $\beta$

Let $\Omega\subset\mathbb{R}^{2}$ be an open, bounded domain. We will use the
finite dimensional spaces and the divergence operator defined in [1, §3]:
$D\left(\Omega\right)\subset H\left(\mathrm{div};\Omega\right)$,
$\widetilde{S}\left(\Omega\right)\subset L^{2}\left(\Omega\right)$,
$S\left(\Omega\right)\subset L^{2}\left(\Omega\right)$, and $\mathbb{E}^{2,1}$
the discrete representation of the divergence operator. Let $K=\mathrm{Ker}\
\mathbb{E}^{2,1}$,
$H=\mathrm{Ker}\left({\mathbb{E}^{2,1}}^{\intercal}\right)$. The discrete inf-
sup condition is then given by

$\beta=\inf_{p\in H^{\perp}}\sup_{\bm{u}\in
K^{\perp}}\frac{\mathcal{N}^{1}\left(\bm{u}\right)^{\intercal}{\mathbb{E}^{2,1}}^{\top}\widetilde{\mathcal{N}}^{0}\left(p\right)}{\|\bm{u}\|_{H(\mathrm{div};\Omega)}\|p\|_{L^{2}\left(\Omega\right)}}\;,$
(3)

where $\mathcal{N}^{x}$ denotes the degrees of freedom in our finite element,
i.e. the vector of expansion coefficients. Here
$p\in\widetilde{S}(\Omega)\subset L^{2}(\Omega)$ and $\bm{u}\in
D(\Omega)\subset H(\mathrm{div};\Omega)$ and the vectors
$\widetilde{\mathcal{N}}^{0}\left(p\right)$ and
$\mathcal{N}^{1}\left(\bm{u}\right)$ are the expansion coefficients. The norm
of $p\in\widetilde{S}(\Omega)$ is

$\|p\|_{L^{2}\left(\Omega\right)}^{2}={\widetilde{\mathcal{N}}^{0}\left(p\right)}^{\intercal}{\mathbb{M}^{(2)}}^{-1}\widetilde{\mathcal{N}}^{0}\left(p\right)\;,$
(4)

the norm of $\bm{u}\in H\left(\mathrm{div};\Omega\right)$ is

$\|\bm{u}\|^{2}_{H\left(\mathrm{div};\Omega\right)}={\mathcal{N}^{1}\left(\bm{u}\right)}^{\intercal}\left(\mathbb{M}^{(1)}+{\mathbb{E}^{2,1}}^{\intercal}\mathbb{M}^{(2)}\mathbb{E}^{2,1}\right)\mathcal{N}^{1}\left(\bm{u}\right)\;,$

and the norm of $\bm{u}\in K^{\perp}\subset D(\Omega)$ is

$\|\bm{u}\|_{K^{\perp}}^{2}=\mathcal{N}^{1}\left(\bm{u}\right)^{\intercal}{\mathbb{E}^{2,1}}^{\intercal}\mathbb{M}^{(2)}{\mathbb{E}^{2,1}}\mathcal{N}^{1}\left(\bm{u}\right)=\|\mathrm{div}\
\bm{u}\|_{L^{2}\left(\Omega\right)}\;.$ (5)

In the continuous case, using Cauchy Schwartz inequality, we have, for all
$p\in L^{2}\left(\Omega\right)$ and $\bm{u}\in K^{\perp}$

${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\frac{\left(p,\mathrm{div}\
\bm{u}\right)}{\|\bm{u}\|_{K^{\perp}}\|p\|_{L^{2}\left(\Omega\right)}}\leq\frac{\|\mathrm{div}\
\bm{u})\|_{L^{2}\left(\Omega\right)}\|p\|_{L^{2}\left(\Omega\right)}}{\|\bm{u}\|_{K^{\perp}}\|p\|_{L^{2}\left(\Omega\right)}}=1\;.}$
(6)

If this inequality holds for all $p$ and $\bm{u}$ it should also hold when we
take the infimum over $\widetilde{S}(\Omega)$ and the supremum over
$D(\Omega)$, from which we conclude that $\beta\leq 1$.

Now, for an arbitrary vector field $\bm{u}^{*}$, we have

$\sup_{u\in
K^{\perp}}\frac{\mathcal{N}^{1}\left(\bm{u}\right)^{\intercal}{\mathbb{E}^{2,1}}^{\intercal}\widetilde{\mathcal{N}}^{0}\left(p\right)}{\|\bm{u}\|_{H(\mathrm{div};\Omega)}\|p\|_{L^{2}\left(\Omega\right)}}\geq\frac{\mathcal{N}^{1}\left(\bm{u}^{*}\right)^{\intercal}{\mathbb{E}^{2,1}}^{\intercal}\widetilde{\mathcal{N}}^{0}\left(p\right)}{\|\bm{u}^{*}\|_{H(\mathrm{div};\Omega)}\|p\|_{L^{2}\left(\Omega\right)}}\;,$
(7)

If we now take for $\bm{u}^{*}$, the vector field with expansion coefficients
which satisfy
$\mathbb{M}^{(2)}\mathbb{E}^{2,1}\mathcal{N}^{1}\left(\bm{u}^{*}\right)=\widetilde{\mathcal{N}}^{0}\left(p\right)$,
then
$\|p\|_{L^{2}\left(\Omega\right)}=\|\bm{u}^{*}\|_{H(\mathrm{div};\Omega)}$ and
the numerator
$\mathcal{N}^{1}\left(\bm{u}\right)^{\intercal}\mathbb{E}^{\intercal}\widetilde{\mathcal{N}}^{0}\left(p\right)=\|p\|_{L^{2}\left(\Omega\right)}^{2}=\|\bm{u}^{*}\|_{H(\mathrm{div};\Omega)}\|p\|_{L^{2}\left(\Omega\right)}$.
If we insert these estimates in (7) we have

$\sup_{\bm{u}\in
K^{\perp}}\frac{\mathcal{N}^{1}\left(\bm{u}\right)^{\intercal}{\mathbb{E}^{2,1}}^{\intercal}\widetilde{\mathcal{N}}^{0}\left(p\right)}{\|\bm{u}\|_{H(\mathrm{div};\Omega)}\|p\|_{L^{2}\left(\Omega\right)}}\geq\frac{\mathcal{N}^{1}\left(\bm{u}\right)^{*\intercal}{\mathbb{E}^{2,1}}^{\intercal}\widetilde{\mathcal{N}}^{0}\left(p\right)}{\|\bm{u}^{*}\|_{H(\mathrm{div};\Omega)}\|p\|_{L^{2}\left(\Omega\right)}}=\frac{\|\bm{u}^{*}\|_{H(\mathrm{div};\Omega)}\|p\|_{L^{2}\left(\Omega\right)}}{\|\bm{u}^{*}\|_{H(\mathrm{div};\Omega)}\|p\|_{L^{2}\left(\Omega\right)}}=1\;,$
(8)

which shows that $\beta\geq 1$. From (6) and (8) we conclude that $\beta=1$ in
the discrete setting. This value is independent of the mesh size or polynomial
degree, so this value is also the inf-sup constant for $h\rightarrow 0$. This
value is also independent of the domain $\Omega$.

##  3 Evaluation of numeric inf-sup constant ${\beta}_{h}$

In this section we will follow [5, §3.4.3] to evaluate the inf-sup constant.
Let $S_{x}$ and $S_{y}$ be the two symmetric and (semi-)positive definite
matrices, such that

$\displaystyle S_{x}^{\intercal}S_{x}$ $\displaystyle=$
$\displaystyle{\mathbb{E}^{2,1}}^{\intercal}\mathbb{M}^{(2)}\mathbb{E}^{2,1}\;,$
(9) $\displaystyle S_{y}^{\intercal}S_{y}$ $\displaystyle=$
$\displaystyle{\mathbb{M}^{(2)}}^{-1}\;.$ (10)

We can write the norms in (4) and (5) as

$\displaystyle||\bm{u}||_{K^{\perp}}$ $\displaystyle=$
$\displaystyle{\mathcal{N}^{1}\left(\bm{u}\right)}^{\intercal}{\mathbb{E}^{2,1}}^{\intercal}\mathbb{M}^{(2)}\mathbb{E}^{2,1}\mathcal{N}^{1}\left(\bm{u}\right)={\mathcal{N}^{1}\left(\bm{u}\right)}^{\intercal}S_{x}^{\intercal}S_{x}\mathcal{N}^{1}\left(\bm{u}\right)=||S_{x}\mathcal{N}^{1}\left(\bm{u}\right)||_{E}\;,$
(11) $\displaystyle||p||_{L^{2}\left(\Omega\right)}$ $\displaystyle=$
$\displaystyle{\widetilde{\mathcal{N}}^{0}\left(p\right)}^{\intercal}{\mathbb{M}^{(2)}}^{-1}\widetilde{\mathcal{N}}^{0}\left(p\right)={\widetilde{\mathcal{N}}^{0}\left(p\right)}^{\intercal}S_{y}^{\intercal}S_{y}\widetilde{\mathcal{N}}^{0}\left(p\right)=||S_{y}\widetilde{\mathcal{N}}^{0}\left(p\right)||_{E}\;,$
(12)

where $||\cdot||_{E}$ is the Eucledian vector norm.

Let $M=S_{y}^{-1}\ \mathbb{E}^{2,1}\ S_{x}^{-1}$ and its singular value
decomposition be given by

$M=S_{y}^{-1}\ \mathbb{E}^{2,1}\ S_{x}^{-1}=V\Sigma U\;.$ (13)

Now we can write $\mathbb{E}^{2,1}$ as

$\mathbb{E}^{2,1}=S_{y}S_{y}^{-1}\ \mathbb{E}^{2,1}\
S_{x}^{-1}S_{x}=S_{y}MS_{x}=S_{y}V\Sigma US_{x}\;.$ (14)

Also, let

$\mathcal{N}^{1}\left(\bm{u}\right)=S_{x}^{-1}U^{\intercal}\bm{x}\;,\qquad\widetilde{\mathcal{N}}^{0}\left(p\right)=S_{y}^{-1}Vy\;.$
(15)

Now, if we substitute $\bm{u}$, $p$ from (15) and $\mathbb{E}^{2,1}$ from (14)
in the RHS term of (3), we get

$\displaystyle\inf_{p\in H^{\perp}}\sup_{\bm{u}\in
K^{\perp}}\frac{{\widetilde{\mathcal{N}}^{0}\left(p\right)}^{\intercal}\mathbb{E}^{2,1}\mathcal{N}^{1}\left(\bm{u}\right)}{||\bm{u}||_{K^{\perp}}||p||_{L^{2}\left(\Omega\right)}}$
$\displaystyle=$ $\displaystyle\inf_{y\in\left(\mathrm{Ker}\
\Sigma^{\intercal}\right)^{\perp}}\sup_{\bm{x}\in\left(\mathrm{Ker}\
\Sigma\right)^{\perp}}\frac{y^{\intercal}V^{\intercal}S_{y}^{-1}S_{y}V\Sigma
US_{x}S_{x}^{-1}U^{\intercal}\bm{x}}{||S_{x}{S_{x}}^{-1}U^{\intercal}\bm{x}||_{E}||S_{y}{S_{y}}^{-1}Vy||_{E}}$
(16) $\displaystyle=$ $\displaystyle\inf_{y\in\left(\mathrm{Ker}\
\Sigma^{\intercal}\right)^{\perp}}\sup_{\bm{x}\in\left(\mathrm{Ker}\
\Sigma\right)^{\perp}}\frac{y^{\intercal}\Sigma\bm{x}}{||\bm{x}||_{E}||y||_{E}}:=\beta_{h}\;.$

where in the last step we used [5, Prop 3.4.3] which states that there exists
a positive constant ${\beta}_{h}$ that is equivalent to the smallest positive
singular value of the matrix, $M=S_{y}^{-1}E^{2,1}S_{x}^{-1}$.

Figure 1: Inf-sup term ${\beta}_{h}$ for $\Omega=[0,1]^{2}$ and polynomial
degree $N=1,2,3$, with varying mesh refinement. Both the plots have same
values. The plot on the right side is a zoom in of data.

Figure 2: Inf-sup term ${\beta}_{h}$ for $\Omega=[0,2]^{2}$ and polynomial
degree $N=1,2,3$, with varying mesh refinement. Both the plots have same
values. The plot on the right side is a zoom in of data.

##  4 Numerical tests

In Figure 1, and Figure 2, we plot the value of the inf-sup term obtained
using (16) for domain $\Omega=[0,1]^{2}$, and $\Omega=[0,2]^{2}$,
respectively. On the y-axis we have $\beta_{h}$, and on the x-axis we have the
length of the element, $h=1/K$ for $K=1\;,2\;,4\;,8\;,16\;,32\;,64$. The plots
on the right side in the figure are a zoom in view of the plots on the left
side. The numeric values for plots in Figure 1 and Figure 2 are given in Table
1 and Table 2 respectively. In both the figures and the plots we observe that
the numeric values of inf-sup term are all very close to $1.0$ and exact upto
at least six decimal places, which is in agreement with theoretical
derivation.

Table 1: Numerical data for inf-sup term for $\Omega=[0,1]^{2}$. h | $N=1$ | $N=2$ | $N=3$
---|---|---|---
$1/2$ | 0.999999994172141 | 0.999999989168835 | 0.999999978662899
$1/4$ | 0.999999987740597 | 0.999999978618959 | 0.999999947316136
$1/8$ | 0.999999986310051 | 0.999999962318488 | 0.999999878165505
$1/16$ | 0.999999980277492 | 0.999999916432749 | 0.999999891563428
$1/32$ | 0.999999972413976 | 0.999999896705947 |
$1/64$ | 0.999999980522211 |  |
Table 2: Numerical data for inf-sup term for $\Omega=[0,2]^{2}$. h | $N=1$ | $N=2$ | $N=3$
---|---|---|---
$1/2$ | 0.999999994172141 | 0.999999983449168 | 0.999999976327674
$1/4$ | 0.999999985661708 | 0.999999971854628 | 0.999999961774105
$1/8$ | 0.999999985276638 | 0.999999932115646 | 0.999999862356188
$1/16$ | 0.999999982313628 | 0.999999842706989 | 0.999999877166154
$1/32$ | 0.999999972765685 | 0.999999906834163 |
$1/64$ | 0.999999980528361 |  |

##  5 Conclusions

In this paper, we derive a theoretical estimate for the discrete inf-sup
formulation and validate the value of the constant using finite dimensional
spaces defined in [1]. The theoretical proof of the inf-sup term becomes
straight forward when we use the appropriate norm on $K^{\perp}$ space, see
(5). We evaluate the constant for two different test cases, i) a unit square
domain $\Omega=[0,1]^{2}$, ii) a square domain $\Omega=[0,2]^{2}$. It is shown
that for all the cases the numerical value is in agreement with the
theoretical value.

## References

  * [1] V. Jain, Y. Zhang, A. Palha, M. Gerritsma, Construction and application of algebraic dual polynomial representations for finite element methods on quadrilateral and hexahedral meshes, Computers & Mathematics with Applications (2020).  doi:10.1016/j.camwa.2020.09.022.
  * [2] D. Chapelle, K.-J. Bathe, The inf-sup test, Computers & structures 47 (4-5) (1993) 537–545.
  * [3] K.-J. Bathe, The inf–sup condition and its evaluation for mixed finite element methods, Computers & structures 79 (2) (2001) 243–252.
  * [4] C. Bernardi, M. Costabel, M. Dauge, V. Girault, Continuity properties of the inf-sup constant for the divergence, SIAM Journal on Mathematical Analysis 48 (2) (2016) 1250–1271.
  * [5] D. Boffi, F. Brezzi, M. Fortin, Mixed finite elements methods and applications, Springer Series in Computational Mechanics (2010).