# Transferring Pre-trained Multimodal Representations with Cross-modal
Similarity Matching

Byoungjip Kim1, Sungik Choi1, Dasol Hwang1, Moontae Lee1,2, Honglak Lee1
LG AI Research1, University of Illinois Chicago2
{bjkim, sungik.choi, dasol.hwang, moontae.lee<EMAIL_ADDRESS>

###### Abstract

Despite surprising performance on zero-shot transfer, pre-training a large-
scale multimodal model is often prohibitive as it requires a huge amount of
data and computing resources. In this paper, we propose a method (BeamCLIP)
that can effectively transfer the representations of a large pre-trained
multimodal model (CLIP-ViT) into a small target model (e.g., ResNet-18). For
unsupervised transfer, we introduce cross-modal similarity matching (CSM) that
enables a student model to learn the representations of a teacher model by
matching the relative similarity distribution across text prompt embeddings.
To better encode the text prompts, we design context-based prompt augmentation
(CPA) that can alleviate the lexical ambiguity of input text prompts. Our
experiments show that unsupervised representation transfer of a pre-trained
vision-language model enables a small ResNet-18 to achieve a better
ImageNet-1K top-1 linear probe accuracy (66.2%) than vision-only self-
supervised learning (SSL) methods (e.g., SimCLR: 51.8%, SwAV: 63.7%), while
closing the gap with supervised learning (69.8%).

##  1 Introduction

Figure 1: ImageNet-1K top-1 linear probe accuracy on ResNet-18
representations. By transferring CLIP-ViT [32] vision-language representations
to ResNet-18, the BeamCLIP can learn better visual representations than
vision-only self-supervised learning (SSL) methods in terms of the linear
probe accuracy.

Learning transferable representations is crucial for successful downstream
tasks. Contrastive learning such as SimCLR [4] and MoCo-v2 [6] have shown
notable success by forcing features of individual classes to be clustered and
sufficiently scattered [41]. But their linear probe performances are still far
behind the supervised learning as shown in Figure 1. Recently, large-scale
vision and language pre-trained (VLP) models provide highly transferable
visual representations via language supervision. However, learning VLP models
from scratch is prohibitive as it requires large amounts of training data and
computing resources. For example, training CLIP [32] requires 400M paired
image-text data and several hundreds of GPUs. ALIGN [22] further scales up to
leverage alternative texts specified for descriptions of web images. While
these models are often based on large Transformers [40], small ConvNets such
as ResNet-50 [17] and MobileNet [20] are still widely used in practice [1] and
even more crucial for low-resource environments. We reformulate representation
learning in terms of knowledge transfer from a large pre-trained model to a
small practical model.

Large-scale vision-language pre-trained models exhibit strong alignments
between different modalities. CLIP [32] learns visual concepts from natural
language supervision, mapping image and text into the same vector space. As
their training data is not only huge but inaccessible, however, conventional
knowledge distillation [19] based on the source training data is no longer a
viable option. Instead, we propose cross-modal similarity matching (CSM).
Imagine your goal is to learn a high quality representation for an input dog
image in CIFAR-10 [23] as in Figure 2. Since CLIP was trained on numerous
image-caption pairs, angular distances from the dog image embedding to the
caption embeddings of other anchor prompt texts such as "A photo of cat" or "A
photo of horse" must comprehensively preserve their visual differences. By
training the student to preserve angular relations witnessed from the teacher,
our model achieves near benchmark performance without accessing the original
data for training CLIP.

To better encode the text prompts, we design context-based prompt augmentation
(CPA) that can alleviate the lexical ambiguity of the input text prompts. We
find that lexical ambiguity in prompt texts can lead to semantically incorrect
text embeddings. This may result in unexpected discrepancies of image-text
alignment in the teacher’s embedding space. Also, it is known that the zero-
shot performance of CLIP can be improved by designing task-specific prompt
texts. Inspired by this, we design CPA that extends the basic prompt of CLIP
to better encode prototypical anchor representations.

Our experimental results show that the BeamCLIP ("beam" means to transmit)
achieves the strongest and near benchmark performance on ImageNet-1K [10]
top-1 linear probe accuracy when using most popular ResNet-18 and ResNet-50 as
the student network. We also compare the effectiveness of the BeamCLIP against
zero-shot transfer learning. Further, we provide ablation study results to
show how much each component contributes to the performance.

Contributions of this paper can be summarized as follows:

  * •

We propose a method (BeamCLIP) that can effectively transfer the
representations of a large pre-trained multimodal model (e.g., CLIP-ViT) into
a small target model (e.g., ResNet-18 or ResNet-50). To achieve this, we
introduce cross-modal similarity matching (CSM) and context-based prompt
augmentation (CPA). (Figure 2).

  * •

We empirically show that BeamCLIP enables a small target model (e.g.,
ResNet-18) to achieve a better ImageNet-1K linear probe accuracy than vision-
only self-supervised learning (SSL) methods, by effectively transferring CLIP-
ViT representations. (Figure 1, Table 2, and Table 3.

  * •

We also explore the zero-shot capability of the BeamCLIP (Table 5) and analyze
the effectiveness of the BeamCLIP on various target datasets (Table 6 and
Table 7).

##  2 Related Work

#### Vision and language pre-trainig.

Vision and language pre-training (VLP) aims to jointly learn vision and
language representations that can be transferred to the downstream tasks such
as visual question answering (VQA), image captioning, and vision and language
navigation (VLN). There are BERT-based vision and language models such as
VLBERT [35], ViLBERT [26], and UNITER [7]. Also, there are contrastive
learning-based models such as CLIP [32] and ALIGN [22]. These models use
contrastive loss [30] to learn aligned vision and language representations by
performing a task of matching a large-scale image and text pairs. The BeamCLIP
aims to transfer the rich representations of large-scale vision and language
pre-trained models such as CLIP and ALIGN to a small target model.

#### Self-supervised learning.

Self-supervised learning (SSL) aims to learn highly transferable
representations by using unlabeled data. In computer vision, at the early
stage, task-specific self-supervised methods were introduced. These include
Context Prediction [11], Rotation Prediction [14], and Colorization [43]. More
recently, contrastive learning-based methods were introduced as a task-
agnostic approach. These include SimCLR [4] and MoCo-v2 [6]. However, since
contrastive self-supervised methods require a large batch size, non-
contrastive methods have been introduced. These include SwAV [3], BYOL [16],
and SimSiam [5]. In this paper, we empirically show that the BeamCLIP can
provide better visual representations than the state-of-the-art SSL methods by
leveraging a large-scale pre-trained multimodal model.

#### Knowledge distillation.

Knowledge distillation (KD) [19] aims to transfer rich knowledge from a strong
teacher model to a target student model. In a conventional setting, it
encourages the student model to mimic the task-specific prediction of the
teacher model. As the student model is trained to predict the same probability
distribution over pre-defined classes as the teacher model’s, using Kullback-
Leibler (KL) divergence is a natural metric to measure the error between the
two models. For a classification task, the loss function can be formulated as
follows:

$\mathcal{L}_{\text{KD}}=\sum_{i}{H(p_{i},q^{S}_{i})}+\sum_{i}KL(p^{T}_{i}||p^{S}_{i}).$
(1)

The first term indicates the supervised loss, where $p_{i}$ denotes the one-
hot labels and $H(p,q)$ denotes cross-entropy. The second term is the
distillation loss, where $p^{T}_{i}$ and $p^{S}_{i}$ are the softmax
predictions of the teacher and student models, respectively.

#### Similarity-based knowledge distillation.

Recently, similarity-based knowledge distillation such as SEED [13], OSS [8],
and ISD [37] was introduced in the context of self-supervised learning (SSL).
SEED [13] showed that the linear probe accuracy of a small student (ResNet-18)
can be improved by transferring the representations of a larger teacher
(ResNet-50) pre-trained by SSL methods such as MoCo-v2 [6]. Unlike this, OSS
[8] aims to transfer representations of an evolving teacher (ResNet-50) into a
smaller student (ResNet-18) on the fly. Unlike SEED and OSS, ISD [37]
considered the same size student and teacher network (ResNet-18), and showed a
student can learn visual representations by iterativly distilling the
similarity of teacher’s representations. These works are closely related to
our work. Unlike these works, the BeamCLIP aims to transfer rich vision and
language representations of large-scale pre-trained models such as CLIP-ViT/16
[32] into a smaller network such as ResNet-18.

#### Prompt engineering.

Recently, researchers showed that prompt engineering [2] is surprisingly
effective at improving the performance of large-scale language models (LLMs)
on downstream tasks without fine-tuning. Prompts are input texts of language
models that usually consist of a task description or several examples. To
further simplify prompt engineering, prompt tuning [24] proposed to add $k$
learnable tokens to the input texts, while having language models frozen.
Similar to GPT-3, it is known that the zero-shot performance of CLIP [32] can
be improved by designing the prompt texts to each task. For example, on
satellite image classification datasets, "A satellite photo of a {label}"
provides better performance than the default "A photo of a {label}". Inspired
by this, we propose context-based prompt augmentation that extends the basic
prompt of CLIP to better encode prototypical text anchor representations by
alleviating the lexical ambiguity of class label texts.

##  3 Method

#### Problem formulation.

Formally, our problem is to transfer aligned cross-modal representations of a
strong teacher model $f^{T}_{\phi}(\cdot)$ into a target student model
$f^{S}_{\theta}(\cdot)$ with unlabeled data
$\mathcal{D}_{u}=\\{x_{i}\\}^{N}_{i=1}$. Given each unlabeled image $x_{i}$,
we formulate representation transfer as a regression task that matches teacher
representations $f^{T}_{\phi}(x_{i})$ to a student’s $f^{S}_{\theta}(x_{i})$.
As the student network is parameterized by $\theta$, the learning objective is

$\arg\min_{\theta}\sum_{i}^{N}\|f^{S}_{\theta}(x_{i})-f^{T}_{\phi}(x_{i})\|_{2}^{2}.$
(2)

Normalizing the representations via $l_{2}$-normalization (i.e.,
$q_{i}=f^{S}_{\theta}(x_{i})/\|f^{S}_{\theta}(x_{i})\|_{2}$ and
$k_{i}=f^{T}_{\phi}(x_{i})/\|f^{T}_{\phi}(x_{i})\|_{2}$) leads to the
following simplification:

$\arg\min_{\theta}\sum_{i}^{N}\|q_{i}-k_{i}\|_{2}^{2}=\arg\min_{\theta}\sum_{i}^{N}(2-2q_{i}\cdot
k_{i}).$ (3)

The problem now involves maximizing the cosine similarity between
$l_{2}$-normalized representations from teacher and student models.

#### Method overview.

Figure 2: Overview of the BeamCLIP. Representation transfer can be viewed as a
task in which, given a query input, a student model learns to regress a vector
representation of a teacher model. The BeamCLIP first measures the normalized
cross-modal similarity of the query image compared to anchor text
representations in the teacher’s embedding space. Then, it encourages the
student to mimic the same cross-modal similarity in the student’s embedding
space. To better align image representations, our method uses self-supervised
pre-training of the student model. Finally, to avoid text ambiguity, we uses
context-based prompt augmentation.

The overview of BeamCLIP is shown in Figure 2. The teacher model of the
BeamCLIP consists of an image encoder $f^{T}_{\phi}(\cdot)$ and a text encoder
$g^{T}_{\psi}(\cdot)$. These encoders are pre-trained under a simple task of
matching images to texts with large-scale corpora. Image representations
$f^{T}_{\phi}(x_{i})$ and text representations $g^{T}_{\psi}(t_{i})$ are thus
well-aligned within a cross-modal embedding space. We provide the details of
the BeamCLIP in the following sections. More specifically, we describe how to
extend the basic problem setting by leveraging the unique features of CLIP
where vision and language representations are precisely aligned. Throughout
the paper, we use the notation CLIP-ViT/16 to denote the CLIP [32] model that
uses Vision Transformer (ViT) [12] with the patch size of 16x16 as the image
encoder. Similar to this, CLIP-RN50 denotes the CLIP model with ResNet-50 [17]
as the image encoder.

###  3.1 Similarity-based cross-modal representation transfer

To effectively distill cross-modal representations, we use similarity-based
matching as described above. Our similarity-based representation transfer
utilizes two carefully designed loss functions: (1) instance similarity
matching (ISM) loss and (2) cross-modal similarity matching (CSM) loss.

#### Instance similarity matching.

This objective is directly derived from Eq. 3. Given a query image $x_{i}$, it
encourages the student image encoder $f^{S}_{\theta}(\cdot)$ to regress the
representation of the teacher image encoder $f^{T}_{\phi}(\cdot)$. We apply
conventional image augmentations (see Appendix B.1) on a query image $x_{i}$,
and the same augmented image $\hat{x}_{i}$ is fed to both the teacher and
student image encoders. Given unlabeled query images
$\mathcal{D}_{u}=\\{x_{i}\\}^{N}_{i=1}$, it is formulated as follows:

$\mathcal{L}_{\text{ISM}}=-\sum_{i=1}^{N}(\frac{f^{S}_{\theta}(\hat{x}_{i})}{\|f^{S}_{\theta}(\hat{x}_{i})\|_{2}}\cdot\frac{f^{T}_{\phi}(\hat{x}_{i})}{\|f^{T}_{\phi}(\hat{x}_{i})\|_{2}})=-\sum_{i=1}^{N}(q_{i}\cdot
k_{i}).$ (4)

However, the similarity signal from a single instance is not enough to
constraint the student representations. For example, the topological ambiguity
may occur in image encoding, since two symmetric representations have the same
cosine similarity compared to a single teacher representation (see Appendix
B.2). We conjecture that this can be mitigated by incorporating multiple
anchor points to the query points. Based on this idea, we introduce cross-
modal similarity matching loss.

#### Cross-modal similarity matching.

To better align a student representation $q_{i}$ with the teacher
representation $k_{i}$, we introduce cross-modal similarity matching (CSM)
loss. We use multiple anchor points to cope with the ambiguity problem
mentioned above. Further, we use text representations as anchor points, since
we can easily generate prototypical anchor points by using text prompts and
class labels. Since image and text representations are precisely aligned in
CLIP, we can effectively apply this approach. More specifically, the BeamCLIP
first measures the normalized image-text similarity of the query image
compared to prototypical text points in the teacher’s embedding space. Then,
it encourages the student to mimic the same image-text similarity in the
student’s embedding space.

More formally, we generate multiple anchor representations
$A=\\{a_{j}\\}_{j=1}^{M}$ by encoding class texts $C=\\{c_{j}\\}_{j=1}^{M}$
with the teacher text encoder $g^{T}_{\psi}(\cdot)$ (in other words,
$a_{j}=g^{T}_{\psi}(c_{j})$). To measure the similarity regarding to multiple
anchor representations $A$, we define the normalized cross-modal similarity as
follows:

$s_{j}(k_{i},A)=\frac{\exp{((k_{i}\cdot
a_{j})/\tau)}}{\sum_{m=1}^{M}\exp{((k_{i}\cdot a_{m})/\tau)}}$ (5)

where $\tau$ is a temperature hyperparameter that is set to 0.01 in our
experiments.

Then, we evaluate the cross-modal similarity distribution by using a set of
normalized cross-modal similarities:

$P(k_{i}|A)=[s_{1}(k_{i},A),...,s_{M}(k_{i},A)].$ (6)

Then, the student model is optimized to mimic the normalized cross-modal
similarity of the teacher’s embedding space by minimizing the cross entropy,
i.e., $H(P(k_{i}|A),P(q_{i}|A))$.

We further minimize the entropy of normalized cross-modal similarities in the
student embedding space i.e., $H(P(q_{i}|A))$. This minimization helps the
student provide query representations $q_{i}$ that are more attracted to
anchor representations $A=\\{a_{j}\\}_{j=1}^{M}$. This entropy minimization is
also known to be effective in other domains such as semi-supervised learning
[15, 29].

Altogether, the CSM loss is formulated as follows:

$\mathcal{L}_{\text{CSM}}=\sum_{i=1}^{N}H(P(k_{i}|A),P(q_{i}|A))+\sum_{i=1}^{N}H(P(q_{i}|A)).$
(7)

Final Loss. The final loss of the BeamCLIP is formulated as follows:

$\mathcal{L}_{\text{BeamCLIP
}}=\mathcal{L}_{\text{CSM}}+\lambda_{\text{ISM}}\mathcal{L}_{\text{ISM}}$ (8)

where $\lambda_{\text{ISM}}$ is the scale hyperparameter that is set to 10 in
our experiments.

###  3.2 Context-based prompt augmentation

Table 1: Examples of context-based prompt augmentation for ambiguous class labels on Flowers102. Label Index | Label Name |  Text Prompt
---|---|---
7 | bird of paradise |  "A photo of {bird of paradise}. {Strelitzia is a genus of five species of perennial plants, native to South Africa. It belongs to the plant family Strelitziaceae}."
10 | snapdragon |  "A photo of {snapdragon}. {Antirrhinum is a genus of plants commonly known as dragon flowers or snapdragons because of the flowers’ fancied resemblance to the face of a dragon that opens and closes its mouth when laterally squeezed}."

We found that lexical ambiguity in prompt texts can lead to semantically
incorrect text embeddings. This may result in an unexpected discrepancy of
image-text alignment in the teacher’s embedding space. For example, Flowers102
[28] dataset has some classes with unusual and ambiguous flower names, such as
“snapdragon”, “bird of paradise”, and “colt’s foot”.111Examples can be found
at: 102 Category Flower Dataset. Therefore, incorrect prototypical anchor
points might be compared with a query image. To address this issue of semantic
ambiguities in the text, we introduce context-based prompt augmentation (CPA),
a data-driven approach that augments basic prompts with contextual text such
as Wikipedia descriptions or hierarchical labels.

For prompt tuning with Wikipedia descriptions, we use the template "A photo of
a {label}. {Wikipedia description}". We use this template for Flowers102 and
Pets37 in our experiments. We provide some examples from the Flowers102
dataset in Table 1. For prompt tuning with hierarchical labels, we use the
template "A photo of a {fine label}, categorized as {coarse label}". We use
this template for CIFAR100 and ImageNet in our experiments. Analogous examples
from CIFAR100 can be found in Table 11 in Appendix B.3.

###  3.3 Other details

#### Self-supervised pre-training of student.

To help the student mimic the teacher’s cross-modal embedding space better, we
pre-train the student image encoder with a self-supervised method. Since self-
supervised pre-training such as SimCLR [4], MoCo-v2 [6], and SwAV [3] provides
a weakly clustered embedding space based on similarities, it can be used as a
better initial state for the student to mimic the teacher’s embedding space.
The details can be found in B.4. We show the effect of SSL pre-training of the
student in the experiment section (see Table 4 and 7).

#### Optimization.

For optimization we use SGD with cosine annealing schedule (SGDR) [25]. To
stabilize training, we use a momentum encoder that updates its weights via
exponential moving average (EMA) [18, 16]. The momentum encoder of a student
$\theta_{\hat{S}}$ is updated using the following rule:

$\theta_{\hat{S}}\xleftarrow{}m\theta_{\hat{S}}+(1-m)\theta_{S}$ (9)

where $\theta_{S}$ is the image encoder of a student model and $m$ is a
momentum hyperparameter that is set 0.99 in our experiments. The model
hyperparameters are summarized in Table 12 in Appendix B.5.

##  4 Experiments

#### Downstream datasets.

We evaluate the BeamCLIP on six standard benchmark datasets: CIFAR10 [23],
CIFAR100 [23], STL10 [9], Flowers102 [28], Pets37 [31], and ImageNet-1K [10].
Following convention, we split the datasets into train, validation, and test
sets. Then, we use train set for transfer, and test set for evaluation. For
ImageNet, we use the validation set as a test set, since its test set does not
provide labels. More details on the datasets are summarized in Table 8 in
Appendix A.

###  4.1 Representation transfer with unlabeled target data

Table 2: ImageNet-1K top-1 linear probe accuracy on ResNet-50. We compare the BeamCLIP with vision-only self-supervised methods in terms of linear probe accuracy on ImageNet-1K. The BeamCLIP representations provide higher linear probe accuracy than self-supervised methods. This means better transferability. The values are quoted from the original paper, and n/a means "not available" from the paper. |  |  |  |  | Epochs |
---|---|---|---|---|---|---
Method | Teacher | Student | Batch | 200 | 400 | 800
Supervised | ✗ | RN50 | 256 |  | 76.2 |
SimCLR [4] | ✗ | RN50 | 512 | 65.6 | 66.7 | 67.4
MoCo-v2 [6] | ✗ | RN50 | 256 | 67.5 | 70.1 | 71.1
BYOL-GA [16] | ✗ | RN50 | 4096 | 70.6 | n/a | n/a
SwAV [3] | ✗ | RN50 | 256 | 72.0 | 74.3 | n/a
BeamCLIP  (ours) | CLIP ViT-B/16 | RN50 | 512 | 74.8 | 75.1 | 75.0
Table 3: ImageNet-1K top-1 linear probe accuracy on ResNet-18. |  |  |  |  | Epochs |
---|---|---|---|---|---|---
Method | Teacher | Student | Batch | 100 | 200 | 400
Supervised | ✗ | RN18 | 256 |  | 69.8 |
SimCLR [4] | ✗ | RN18 | 256 | 47.1 | 49.9 | 51.8
MoCo-V2 [6] | ✗ | RN18 | 256 | 48.6 | 49.9 | 51.9
BYOL [16] | ✗ | RN18 | 256 | 44.2 | 47.5 | 46.8
BYOL-GA [16] | ✗ | RN18 | 256 | 54.2 | 56.9 | 61.4
SwAV [3] | ✗ | RN18 | 256 | 57.7 | 61.2 | 63.7
OSS [8] | SSL RN50 | RN18 | 256 | 60.0 | 64.1 | 65.8
BeamCLIP  (ours) | CLIP ViT-B/16 | RN18 | 256 | 63.8 | 64.8 | 66.2

Setting. We compare the BeamCLIP with various self-supervised methods in terms
of linear probe accuracy on ImageNet-1K. Following the conventional protocol,
we use ResNet-18 and ResNet-50 [17] as the base encoder and evaluate the
learned representations by using logistic regression. We use LBFGS algorithm
[44] for logistic regression. Its hyperparameter $C$ is determined through
coarse-grained hyperparameter search on the validation split. And, the
accuracy is evaluated in the test split. We found that it provides the best
linear probe accuracy when $C$ is set to 30. We perform our experiments on 8
NVIDA A100 GPUs and it takes about 30 hours for 200 epoch training.

Transfer to ResNet-50. Table 2 shows the comparison of the BeamCLIP with
vision-only self-supervised methods such as SimCLR [4], MoCo-v2 [6], SwAV [3],
BYOL [16], and SimSiam [5]. The BeamCLIP provides better visual representation
by achieving 74.8% top-1 linear probe accuracy on ImageNet-1K [10]. While
self-supervised methods take long training epochs to achieve comparable
accuracy, BeamCLIP-RN50 achieves better accuracy with less training epochs.
Also, note that BeamCLIP-RN50’s representations provide better accuracy than
CLIP-RN50’s representations.

Transfer to ResNet-18. To check if the BeamCLIP can transfer CLIP
representations into smaller models than ResNet-50 (24M), we also measure
ImageNet-1K top-1 linear probe accuracy on ResNet-18 (11M). ResNet-18 is
trained from scratch (not self-supervised pre-trained with SimCLR), while
transferring CLIP ViT-B/16 representations. As shown in Table 3, BeamCLIP
learns better representations than SSL methods such as SimCLR [4], MoCo-v2
[6], BYOL [16], and SwAV [3]. More importantly, the BeamCLIP provides better
performance than OSS [8] that simultaneously learns and transfers
representations from ResNet-50. The learning curve is presented in Figure 8 in
Appendix C.1.

Effect of self-supervised pre-training. Table 4 shows ImageNet-1K top-1 linear
probe accuracy on BeamCLIP-RN50 representations by using different SSL pre-
training. With the better SSL method (SwAV [3] > SimCLR [4]), the BeamCLIP can
learn better representations with an increased linear probe accuracy.

Table 4: Effect of self-supervised pre-training of student. Method | Teacher | Student | Pre-training of Student | Batch | Epoch | Linear Probe
---|---|---|---|---|---|---
Supervised | ✗ | RN50 | ✗ | 256 | - | 76.2
BeamCLIP  (ours) | CLIP ViT-B/16 | RN50 |  SimCLR [4] | 512 | 200 | 74.8
BeamCLIP  (ours) | CLIP ViT-B/16 | RN50 |  SwAV [3] | 512 | 200 | 75.8

###  4.2 Representation transfer with unlabeled non-target data

To check if the BeamCLIP also inherits the powerful zero-shot capability of
CLIP, we compare zero-shot accuracy of CLIP variants on ImageNet-1K. For zero-
shot measure, we use CC-3M [34] and ImageNet-21K (12M samples) [33] that do
not have overlap with ImageNet-1K. Table 5 shows the comparison of zero-shot
accuracy. The BeamCLIP-RN50 achieves about 57.5% zero-shot accuracy that is
highly comparable with CLIP RN-50 (59.6%). The learning curve is presented in
Figure 9 of Appendix C.1.

Table 5: Comparison of zero-shot accuracy on ImageNet-1K. On ImageNet-1K, the BeamCLIP RN50 achieves about 57.5% zero-shot accuracy that is higly comparable with CLIP RN-50 (59.6%). To achieve such a high zero-shot accuracy, CLIP uses very large image-text pair data (WIT-400M). Instead, the BeamCLIP can achieve the comparable zero-shot accuracy by effectively transferring the teacher’s representations, while using only 3% data (ImageNet-21K (12M)). Note that OpenCLIP provides only about 36.5% zero-shot accuracy with the similar amount of data (CC-12M). Method | Image | Training | Teacher | Text | Batch | Epochs | ImageNet
---|---|---|---|---|---|---|---
| Encoder | Data | Model | Prompts | Size |  | Zero-shot
CLIP [32] | ViT-B/16 | WIT-400M | ✗ | ✗ | 32,768 | 32 | 68.6
CLIP [32] | RN50 | WIT-400M | ✗ | ✗ | 32,768 | 32 | 59.6
OpenCLIP [21] | RN50 | YFCC-15M | ✗ | ✗ | $256*8$ | 32 | 32.7
OpenCLIP [21] | RN50 | CC-12M | ✗ | ✗ | $256*8$ | 32 | 36.5
BeamCLIP  (ours) | RN50 | CC-3M | CLIP ViT-B/16 | IN-1K | $64*8$ | 100 | 49.5
BeamCLIP  (ours) | RN50 | IN-21K (12M) | CLIP ViT-B/16 | IN-1K | $64*8$ | 50 | 53.6
BeamCLIP  (ours) | RN50 | IN-21K (12M) | CLIP ViT-B/16 | IN-1K | $64*8$ | 200 | 57.5

###  4.3 Transfer learning accuracy on various target datasets

Table 6: Comparison of teacher and student accuracy on various datasets.
Conventional knowledge distillation with the KL divergence is not effective in
transferring CLIP-ViT representations. In contrast, the BeamCLIP effectively
transfers CLIP ViT-B/16 representations into ResNet-50 by using unlabeled
query data. We denoted with $*$ in cases our student model surpasses the
accuracy of the teacher model.
Method |  Model Type |  Img. Enc. |  Param. Size |  CIFAR10 |  CIFAR100 |  STL10 |  Flowers102 |  Pets37 |  ImageNet-1K
---|---|---|---|---|---|---|---|---|---
CLIP [32] (zero-shot) | T | ViT-B/16 | 76M | 91.6 | 68.7 | 98.2 | 70.4 | 88.9 | 68.6
CRD [38] | S | RN50 | 24M | 76.78 | 38.83 | 81.41 | 26.15 | 62.22 | 30.90
KD [19] | S | RN50 | 24M | 90.89 | 58.02 | 93.28 | 49.01 | 77.51 | 56.38
BeamCLIP  (ours) | S | RN50 | 24M | 92.10∗ | 67.35 | 97.45 | 75.86∗ | 86.94 | 66.17

Setting. We evaluate how effectively the BeamCLIP transfers CLIP-ViT
representations into a student model by evaluating classification accuracy on
various datasets (see Table 6). We choose ResNet-50 [17] as the student model
to compare the distilled target model with CLIP-RN50. Also, ResNet-50 is
conventionally used in evaluating linear probe accuracy on ImageNet-1K.
However, the BeamCLIP can adopt any architecture, not just ResNet-50.

As baselines, we choose two representative distillation methods among many
methods: (1) conventional knowledge distillation (KD) [19] and (2) contrastive
representation distillation (CRD) [38]. Since conventional KD aims to mimic
the task-specific predictions of the teacher model unlike the BeamCLIP , we
apply the KL divergence on minimizing the cross-modal similarity distribution
(i.e., $P(q_{i}|A)$ and $P(k_{i}|A)$), instead of the Cross-entropy (CE). CRD
proposes a variant of InfoNCE loss for representation distillation, which we
apply on normalized representations (i.e., $q_{i}$ and $k_{i}$). The details
of each method can be found in the related work section 2.

Results. Table 6 shows a comparison of teacher and student accuracy on various
datasets. We empirically demonstrate that the BeamCLIP can effectively
transfer vision and language representations of a large teacher model (CLIP
ViT-B/16) into a small student model (ResNet-50). We find that the KL
divergence used in conventional knowledge distillation (KD) is not effective
in transferring CLIP-ViT representations. Also, the contrastive learning-based
approach is not effective. Unlike this, the BeamCLIP can effectively transfer
CLIP ViT-B/16 representations into ResNet-50, achieving very high accuracy
that is comparable or better than the teacher accuracy. KD simply minimizes
the error between single instances. We conjecture that the cross-modal
similarity to multiple anchor points introduced in the BeamCLIP helps the
student preserve the topology of the teacher’s embedding space.

Also, note that context-based prompt augmentation helps achieve better
accuracy after representation transfer. Since Flowers102 has many ambiguous
labels, our experiment shows that text prompt augmentation significantly
increases the student’s accuracy compared to the teacher’s accuracy.

Ablation study. Table 7 shows the ablation study results of the BeamCLIP . Our
empirical findings are as follows: (1) Instance similarity matching (ISM) is
not enough by itself to preserve the topology of the teacher’s embedding
space. (2) Cross-modal similarity matching (SCM) compared to multiple anchor
points helps the student mimic the teacher’s embedding space. (3) Self-
supervised pre-training of the student (SSL PT) helps the student mimic the
teacher’s embedding space. (4) Entropy minimization (EntMin) helps to improve
the accuracy. (5) Context-based prompt augmentation (CPA) helps measure the
similarity more precisely. As shown in the table, Flowers102 dataset is
sensitive to self-supervised pre-training of student. We conjecture that since
Flowers102 dataset has only 1020 training samples for the 102 classes, it is
not enough to probe the teacher’s representation space.

Table 7: Ablation study results. The acronym denotes the sub-methods introduces in the BeamCLIP : (1-1) ISM means instance similarity matching loss, (1-2) CSM means cross-modal similarity matching loss, (2) CPA means context-based prompt augmentation, and (3) SSL PT means self-supervised pre-training of student, . All the technical components contribute to the improvements of transfer learning accuracy at the student. Note that we only perform CPA on datasets with more than 100 of class labels. Method |  Type |  Img. Enc. |  (1-1) ISM |  (1-2) CSM |  (2) CPA |  (3) SSL PT |  CIFAR10 |  CIFAR100 |  Flowers102 |  ImageNet-1K
---|---|---|---|---|---|---|---|---|---|---
CLIP [32] (zero-shot) | T | ViT-B/16 | - | - | - | - | 91.6 | 68.7 | 70.4 | 68.6
Unsupervised Representation Transfer (BeamCLIP) | S | RN50 | Cosine | ✗ | ✗ | ✗ | 39.07 | 6.04 | 1.43 | 21.83
S | RN50 | Cosine | ✗ | ✗ | SimCLR | 87.00 | 48.26 | 3.59 | 51.83
S | RN50 | ✗ | CE | ✗ | ✗ | 91.28 | 58.90 | 12.94 | 63.30
S | RN50 | ✗ | CE | ✗ | SimCLR | 91.53 | 65.64 | 62.18 | 65.45
S | RN50 | ✗ | CE+EntMin | ✗ | SimCLR | 91.71 | 66.14 | 63.96 | 66.23
S | RN50 | Cosine | CE+EntMin | ✗ | SimCLR | 92.10∗ | 66.18 | 64.14 | 65.76
S | RN50 | Cosine | CE+EntMin | ✓ | SimCLR | - | 67.35 | 75.86∗ | 66.17
Figure 3: Comparison of CLIP-RN50 and BeamCLIP-RN50. This figure shows the
top-5 text-image retrieval results. A red rectangle denotes an incorrect
result. CLIP-RN50 provides many incorrect results, since its zero-shot
accuracy is relatively low. In contrast, BeamCLIP-RN50 provides much improved
results, since it is transferred from CLIP-ViT/16 with higher zero-shot
accuracy.

Qualitative result. To see the quality of the transferred representations, we
analysed text-image retrieval results on the Flowers102 dataset. Figure 3
compares the top-5 text-image retrieval results between CLIP-RN50 and
BeamCLIP-RN50. A red rectangle denotes an incorrect result. Compared to CLIP-
RN50, BeamCLIP-RN50 provides much improved results, since its representations
are transferred from CLIP-ViT/16 with higher zero-shot accuracy. More
interestingly, BeamCLIP-RN50 provides surprisingly good text-image retrieval
results, even though unseen text prompts such as "a photo of {pink rose}" or
"a photo of {yellow rose}" are given.

###  4.4 Effect of random text prompts

(a) Subset text prompts from CIFAR100 classes

(b) Random text prompts from ImageNet classes

Figure 4: The effect of random text prompts on CIFAR100. (a) The text prompts
are randomly sampled from the set of 100 class names of CIFAR100. The red
dotted line denotes the teacher’s accuracy as an upper bound. It is more
efficient as it is closer to this line. As shown in the blue line, the
BeamCLIP (CE+EntMin) can effectively transfer the CLIP representations, even
when the class names of the target dataset are partially given. (b) The text
prompts are randomly sampled from the 1000 class names of ImageNet. The
BeamCLIP (CE+EntMin) is still effective, even though the class names are
randomly sampled from a non-target dataset (ImageNet-1K).

We measured how effective the BeamCLIP is in cases where the class names of
the target dataset are not perfectly given. Figure 4 shows the effect of the
randomly sampled text prompts on CIFAR100. We can see that the BeamCLIP is
still effective, even when (a) the subset of the 100 class names of CIFAR100
are given as the text prompts, or (b) the text prompts are randomly sampled
from a non-target dataset (ImageNet-1K). The exact values in Figure 4 are
presented in Table 15 and Table 16 in Appendix C.2. Also, the additional
results on CIFAR10 are provided in Appendix C.2.

##  5 Limitations and Conclusion

Limitations. With the help of rich representations of pre-trained CLIP, the
BeamCLIP can learn better representations than SSL methods. However, since SSL
methods can increase the performance at longer training epochs, the
performance margin may be decreased in such a setting. Another shortcoming is
that context-based prompt augmentations may require additional engineering
efforts.

Conclusion. In this paper, we provide the BeamCLIP that can effectively
transfer large pre-trained vision-language model (e.g., CLIP-ViT) into a small
target model (e.g., ResNet-18) with cross-modal similarity matching (CSM) and
context-based prompt augmentation (CPA). We empirically show that the BeamCLIP
can learn better visual representations than vision-only self-supervised
learning (SSL) methods, by leveraging a pre-trained vision-language model
(CLIP). The BeamCLIP is not intended to be another CLIP, but an effective CLIP
student.

## Broader impact

This research aims to provide a simple and effective way to leverage CLIP for
representation learning. With the help of CLIP, the BeamCLIP can learn better
representations than self-supervised learning (SSL) methods. Since training
CLIP requires very large data and hundreds of GPUs, it is important to provide
a way to effectively reuse the pre-trained CLIP rather than training from
scratch on a target model. We believe that the BeamCLIP can help to save cost
and time.

## Acknowledgements

We thank anonymous reviewers for their valuable comments. This work was fully
supported by LG AI Research.

## References

  * Beyer et al. [2021] L. Beyer, X. Zhai, A. Royer, L. Markeeva, R. Anil, and A. Kolesnikov.  Knowledge distillation: A good teacher is patient and consistent.  _arXiv preprint arXiv:2106.05237_ , 2021.
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020\.
  * Caron et al. [2020] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  In _Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Chen et al. [2020a] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton.  A simple framework for contrastive learning of visual representations.  In _International Conference on Machine Learning (ICML)_ , 2020a.
  * Chen and He [2021] X. Chen and K. He.  Exploring simple siamese representation learning.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 15750–15758, 2021.
  * Chen et al. [2020b] X. Chen, H. Fan, R. Girshick, and K. He.  Improved baselines with momentum contrastive learning.  _arXiv preprint arXiv:2003.04297_ , 2020b.
  * Chen et al. [2020c] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu.  Uniter: Universal image-text representation learning.  In _European Conference on Computer Vision (ECCV)_ , 2020c.
  * Choi et al. [2021] H. M. Choi, H. Kang, and D. Oh.  Unsupervised representation transfer for small networks: I believe i can distill on-the-fly.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Coates et al. [2011] A. Coates, A. Ng, and H. Lee.  An analysis of single-layer networks in unsupervised feature learning.  In _International Conference on Artificial Intelligence and Statistics_ , 2011.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009.
  * Doersch et al. [2015] C. Doersch, A. Gupta, and A. A. Efros.  Unsupervised visual representation learning by context prediction.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2015.
  * Dosovitskiy et al. [2021] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Fang et al. [2021] Z. Fang, J. Wang, L. Wang, L. Zhang, Y. Yang, and Z. Liu.  Seed: Self-supervised distillation for visual representation.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Gidaris et al. [2018] S. Gidaris, P. Singh, and N. Komodakis.  Unsupervised representation learning by predicting image rotations.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Grandvalet and Bengio [2005] Y. Grandvalet and Y. Bengio.  Semi-supervised learning by entropy minimization.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2005.
  * Grill et al. [2020] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * He et al. [2020] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick.  Momentum contrast for unsupervised visual representation learning.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Hinton et al. [2015] G. Hinton, O. Vinyals, and J. Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Howard et al. [2017] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_ , 2017.
  * Ilharco et al. [2021] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt.  Openclip, Jul 2021.  URL https://github.com/mlfoundations/open_clip.
  * Jia et al. [2021] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning (ICML)_ , 2021.
  * Krizhevsky et al. [2009] A. Krizhevsky, G. Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Lester et al. [2021] B. Lester, R. Al-Rfou, and N. Constant.  The power of scale for parameter-efficient prompt tuning.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2021\.
  * Loshchilov and Hutter [2017] I. Loshchilov and F. Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  In _International Conference on Learning Representations (ICLR)_ , 2017.
  * Lu et al. [2019] J. Lu, D. Batra, D. Parikh, and S. Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Müller et al. [2019] R. Müller, S. Kornblith, and G. Hinton.  When does label smoothing help?  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Nilsback and Zisserman [2008] M.-E. Nilsback and A. Zisserman.  Automated flower classification over a large number of classes.  In _Indian Conference on Computer Vision, Graphics & Image Processing_, 2008.
  * Oliver et al. [2018] A. Oliver, A. Odena, C. A. Raffel, E. D. Cubuk, and I. Goodfellow.  Realistic evaluation of deep semi-supervised learning algorithms.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018.
  * Oord et al. [2018] A. v. d. Oord, Y. Li, and O. Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Parkhi et al. [2012] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar.  Cats and dogs.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  _arXiv preprint arXiv:2103.00020_ , 2021.
  * Ridnik et al. [2021] T. Ridnik, E. Ben-Baruch, A. Noy, and L. Zelnik-Manor.  Imagenet-21k pretraining for the masses.  2021\.
  * Sharma et al. [2018] P. Sharma, N. Ding, S. Goodman, and R. Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018.
  * Su et al. [2019] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  In _International Conference on Learning Representations (ICLR)_ , 2019.
  * Szegedy et al. [2016] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.  Rethinking the inception architecture for computer vision.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Tejankar et al. [2021] A. Tejankar, S. A. Koohpayegani, V. Pillai, P. Favaro, and H. Pirsiavash.  Isd: Self-supervised learning by iterative similarity distillation.  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Tian et al. [2020] Y. Tian, D. Krishnan, and P. Isola.  Contrastive representation distillation.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * Tschannen et al. [2020] M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic.  On mutual information maximization for representation learning.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems (NeurIPS_ , 2017\.
  * Wang and Isola [2020] T. Wang and P. Isola.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  In _International Conference on Machine Learning_ , pages 9929–9939. PMLR, 2020.
  * Yuan et al. [2020] L. Yuan, F. E. Tay, G. Li, T. Wang, and J. Feng.  Revisiting knowledge distillation via label smoothing regularization.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Zhang et al. [2016] R. Zhang, P. Isola, and A. A. Efros.  Colorful image colorization.  In _European Conference on Computer Vision (ECCV)_ , 2016.
  * Zhu et al. [1997] C. Zhu, R. H. Byrd, P. Lu, and J. Nocedal.  Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization.  _ACM Transactions on mathematical software (TOMS)_ , 23(4):550–560, 1997.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes] See Section 1.

    2. (b)

Did you describe the limitations of your work? [Yes] See Section 4 and Section
5.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]
See Section Broader impact.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [No] Our
work does not include theoretical results.

    2. (b)

Did you include complete proofs of all theoretical results? [No] Our work does
not include theoretical results.

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
We provide the code, data, and instructions in the supplemental material.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] See Section 4, Appendix A, and Appendix B.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes] See Section 4.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section
4.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] See
Section 2, Section 3, and Section 4.

    2. (b)

Did you mention the license of the assets? [No] We only used public benchmark
datasets and open-sourced software. See Section 4.

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [No] We did not use new assets.

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [Yes] We only used public benchmark datasets. See
Section 4.

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [No] We did not use any data
containing personally identifiable information or offensive content.

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [No] We did not use crowdsourcing or conduct
research with human subjects.

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [No] We did not use crowdsourcing
or conduct research with human subjects.

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [No] We did not use crowdsourcing or
conduct research with human subjects.

##  Appendix A Datasets Details

We demonstrate the effectiveness of the BeamCLIP by using six downstream
datasets. Table 8 shows the details of the downstream datasets.

Table 8: Details of datasets used for the BeamCLIP evaluation.
Dataset | Image Size | Classes | Train Size | Val Size | Test Size
---|---|---|---|---|---
CIFAR10 [23] | 32x32 | 10 | 40,000 | 10,000 | 10,000
CIFAR100 [23] | 32x32 | 100 | 40,000 | 10,000 | 10,000
STL10 [9] | 128x128 | 10 | 4,000 | 1,000 | 8,000
Flowers102 [28] | 224x224 | 102 | 1,020 | 1,020 | 6,149
Pets37 [31] | 224x224 | 37 | 2,944 | 736 | 3,669
ImageNet [10] | 224x224 | 1,000 | 1,231,167 | 50,000 | 50,000

##  Appendix B Method Details

In this section, we provide some details of the BeamCLIP . More specifically,
we provide the details of two main contributions that are (1) cross-modal
similarity matching (CSM) and (2) context-based prompt augmentation (CPA).
Also, we provide the other implementation details such as image augmentation,
similarity smoothing, model hyperparameters, etc.

###  B.1 Image augmentation details

We use conventional image augmentation when performing representation transfer
by using unlabeled images in downstream datasets. Table 9 provides a list of
image augmentation used for unsupervised representation transfer on downstream
datasets.

Table 9: A list of image augmentations used in the BeamCLIP .
Mode | Augmentation | Parameters
---|---|---
Train | RandomResizedCrop | -
| RandomHorizontalFlip | p=0.5
| RandomColorJitter | p=0.8
| GaussianBlur | p=0.5, min=0.1, miax=2.0
| Normalize | -
Val | Resize | input_size + 0.1 * input_size
| CenterCrop | input_size
| Normalize | -

###  B.2 Cross-modal similarity matching details

Cross-modal similarity matching (CSM) is the main method of the BeamCLIP . To
make the concept of CSM clearer, we provide an illustration of CSM in Figure
5.

Figure 5: Illustration of cross-modal similarity matching. Topological
ambiguity may occur in image encoding, since query image embedding $q_{i1}$
and $q_{i2}$ can have the same cosine similarity compared to a single teacher
image embedding $k$, while heading towards different directions. To mitigate
this problem, we introduce cross-modal similarity matching that encourage the
student to mimic the same cross-modal similarity distribution (measured
against multiple anchor text points) in teacher’s embedding space.

###  B.3 Context-based prompt augmentation details

To prepare for better text anchor embeddings for unsupervised representation
transfer, we introduce context-based prompt augmentation (CPA). To make the
concept of CPA clearer, we provide an illustration of CPA in Figure 6.

Also, we provide an example of the hierarchical class labels in Table 10 and
an example context-based prompt augmentation for CIFAR100 in Table 11.

Figure 6: Illustration of context-based prompt augmentation. The lexical
ambiguity may occur in text encoding, since the same text may have multiple
different meanings. To mitigate this problem, we introduce context-based
prompt augmentation that helps resolve the ambiguity with contextual texts
such as Wikipedia descriptions. Table 10: Coarse and fine labels for CIFAR100.
Coarse Label | Fine Label
---|---
aquatic mammals | beaver, dolphin, otter, seal, whale
fish | aquarium fish, flatfish, ray, shark, trout
flowers | orchid, poppy, rose, sunflower, tulip
food containers | bottle, bowl, can, cup, plate
household electrical devices | clock, keyboard, lamp, telephone, television
household furniture | bed, chair, couch, table, wardrobe
insects | bee, beetle, butterfly, caterpillar, cockroach
large carnivores | bear, leopard, lion, tiger, wolf
large man-made outdoor things | bridge, castle, house, road, skyscraper
large natural outdoor scenes | cloud, forest, mountain, plain, sea
large omnivores and herbivores | camel, cattle, chimpanzee, elephant, kangaroo
medium mammals | fox, porcupine, possum, raccoon, skunk
non-insect invertebrates | crab, lobster, snail, spider, worm
people | baby, boy, girl, man, woman
reptiles | crocodile, dinosaur, lizard, snake, turtle
small mammals | hamster, mouse, rabbit, shrew, squirrel
trees | maple tree, oak tree, palm tree, pine tree, willow tree
vehicles 1 | bicycle, bus, motocycle, pickup truck, train
vehicles 2 | lawn mower, rocket, streetcar, tank, tractor
Table 11: Examples of prompt augmentation with hierarchical labels for
CIFAR100.
Label Name | Text Prompt
---|---
baby | "A photo of a {baby}, categorized as {people}."
beaver | "A photo of a {beaver}, categorized as {aquatic mammals}."
bee | "A photo of a {bee}, categorized as {insect}."

###  B.4 Other implementation details

#### Self-supervised pre-training of student.

For self-supervised pre-training, we adopt SimCLR, since it is simple and
effective. SimCLR learns transferable visual representations by using InfoNCE
loss [30, 39] which encourages agreement between multiple views of the same
image. More specifically, InfoNCE maximizes the similarity between multiple
views of the same image (i.e., positive samples) and minimizes the similarity
to multiple views of all other images in a training batch (i.e., negative
samples). InfoNCE loss of SimCLR can be formulated as follows:

$\mathcal{L}_{\text{InfoNCE}}=-\log\frac{\exp{((h^{S}_{i}\cdot
h^{S}_{i^{\prime}})/\tau)}}{\sum_{k=1}^{2B}\mathbbm{1}_{[k\neq
i]}\exp{((h^{S}_{i}\cdot h^{S}_{k})/\tau)}}$ (10)

where $h^{S}_{i}\in\mathbbm{R}^{128}$ is a projection of a student
representation $q_{i}\in\mathbbm{R}^{512}$, $\tau$ is a temperature
hyperparameter that is set to 0.1, $\mathbbm{1}_{[k\neq i]}$ is an indicator
function whose value is 1 if $k\neq i$, and $B$ is a batch size. Here, $h_{i}$
and $h_{i^{\prime}}$ are projections of multiple views of the same input
images $x_{i}$.

#### Similarity Smoothing.

To improve the effectiveness of distillation, we apply Label Smoothing (LS)
[36] to the cross-modal similarity distillation loss. Recent works [27, 42]
show that Label Smoothing helps knowledge distillation. To apply Label
Smoothing, we determine the most similar anchor representation as follows:

$j^{*}=\arg\max_{j}s_{j}(k_{i},A).$ (11)

Then, we generate a modified cross-modal similarity distribution:

$s_{j}(k_{i},A)^{LS}=\mathbbm{1}_{[j=j^{*}]}(1-\alpha)+\alpha/M$ (12)

where $\mathbbm{1}_{[j=j^{*}]}$ is the indicator function whose value is 1 if
$j=j^{*}$, $M$ is the number of anchors, and $\alpha$ is the smoothing
hyperparameter that is set to 0.2 in our experiments.

###  B.5 Model hyperparameters

Table 12 provides the summary of model hyperparamters. We use the same
hyperparameters on all downstream datasets if not explicitly declared.

Table 12: BeamCLIP hyperparameters.
Hyperparameter | Value
---|---
CSM loss temperature | 0.01
ISM loss scale | {0.1, 1.0, 10.0}
similarity smoothing (LS) sacle | 0.2
optimizer |  SGDR [25]
initial learning rate | 0.5
weight decay | 1e-6
EMA momentum | 0.99
batch size | {256, 512}
epochs | 200

##  Appendix C Additional Experiment Results

In this section, we provide additional experiment results. First, we provide
the learning curves that are generated while training the BeamCLIP . Second,
we provide some experiment results on the effects of random text prompts.
Third, we provide an example qualitative result that shows the advantage of
the BeamCLIP .

###  C.1 Learning curves of the BeamCLIP

We provide the learning curve of the BeamCLIP for the experiment section.
Figure 7 shows the learning curve for ImageNet-1K validation accuracy of
BeamCLIP-RN50 representations trained with unlabeled ImageNet-1K. Figure 8
shows the learning curve for ImageNet-1K validation accuracy of BeamCLIP-RN18
representations trained with unlabeled ImageNet-1K. Figure 9 shows the
learning curve for ImageNet-1K zero-shot accuracy of BeamCLIP-RN50
representations trained with unlabeled non-target data (ImageNet-21K).

Figure 7: ImageNet-1K top-1 validation accuracy of BeamCLIP-RN50
representations learned with unlabeled target data (ImageNet-1K). Figure 8:
ImageNet-1K top-1 validation accuracy of BeamCLIP-RN18 representations learned
with unlabeled target data (ImageNet-1K). Figure 9: Zero-shot ImageNet-1K
top-1 accuracy of BeamCLIP-RN50 representations learned with unlabeled non-
target data (ImageNet-21K).

###  C.2 The effect of random text prompts

In this section, we further analyze the effect of text prompts from the
perspective of unsupervised learning. Before that, we briefly review the
proposed method. In this paper, we propose the BeamCLIP , an unsupervised
representation transfer method of a large pre-trained multimodal model such as
CLIP. The BeamCLIP can transfer the visual representations of CLIP by using
unlabeled images on a downstream dataset. To achieve this, we propose cross-
modal similarity matching (CSM). In CSM, at first, given an unlabeled image,
cross-modal similarity distribution is measured from multiple text prompt
embeddings in the teacher’s embedding space. Then, a student model is
encouraged to mimic the cross-modal similarity distribution of the teacher
model by matching these similarity distributions. To achieve effective
transfer, we use anchor text embeddings by encoding text prompts. For example,
on CIFAR10, we use ten text prompts in the form of "a photo of {class name}".
Note that the text prompts are not paired with each image.

#### CIFAR10.

We measured how effective the BeamCLIP is in cases where the class names of
the target dataset are not perfectly given. Table 10 shows the effect of the
randomly sampled text prompts on CIFAR10 [23]. The values in Figure 10 are
also presented in Table 13 and Table 14.

#### CIFAR100.

Table 4 shows the effect of the randomly sampled text prompts on CIFAR100
[23]. The values in Figure 4 are also presented in Table 15 and Table 16.

(a) Subset prompts from CIFAR10

(b) Random prompts from ImageNet

Figure 10: Effect of the random text prompts on CIFAR10. (a) The text prompts are randomly sampled from the 10 class names of CIFAR10. The red dotted line denotes the teacher’s accuracy as an upper bound. It is more efficient as it is closer to this line. As shown in the blue line, the BeamCLIP is still effective, even when the class names of the target dataset are partially given. The BeamCLIP (KL) means to use the KL-divergence for matching cross-modal similarity distribution. The BeamCLIP (CE+EntMin) is more effective, as more text prompts are given. (b) The text prompts are randomly selected from the 1000 class names of ImageNet. The BeamCLIP (CE+EntMin) is still effective, even though the class names are randomly sampled from a non-target dataset (ImageNet-1K). Table 13: Effect of the partial text prompts on CIFAR10. |  |  |  |  | Prompts |  |  |
---|---|---|---|---|---|---|---|---
Method | Type | Img. Enc. | 3 | 5 | 7 | 9 | 10 | -
CLIP [32] (zero-shot) | T | ViT-B/16 | - | - | - | - | - | 91.6
CLIP [32] (zero-shot) | - | RN50 | - | - | - | - | - | 75.6
BeamCLIP (CE+EntMin) | S | RN50 | 83.47 | 83.25 | 88.26 | 91.84 | 92.10∗ | -
BeamCLIP (KL) | S | RN50 | 89.36 | 90.43 | 90.15 | 90.54 | 90.85 | -
Table 14: Effect of the random text prompts on CIFAR10. |  |  |  |  | Prompts |  |
---|---|---|---|---|---|---|---
Method | Type | Img. Enc. | 10 | 20 | 30 | 40 | -
CLIP [32] (zero-shot) | T | ViT-B/16 | - | - | - | - | 91.6
CLIP [32] (zero-shot) | - | RN50 | - | - | - | - | 75.6
BeamCLIP (CE+EntMin) | S | RN50 | 81.49 | 82.05 | 84.09 | 84.51 | -
BeamCLIP (KL) | S | RN50 | 87.76 | 87.67 | 87.83 | 87.88 | -

Table 15: Effect of the partial text prompts on CIFAR100. |  |  |  |  | Prompts |  |  |
---|---|---|---|---|---|---|---|---
Method | Type | Img. Enc. | 20 | 40 | 60 | 80 | 100 | -
CLIP [32] (zero-shot) | T | ViT-B/16 | - | - | - | - | - | 68.7
CLIP [32] (zero-shot) | - | RN50 | - | - | - | - | - | 41.6
BeamCLIP (CE+EntMin) | S | RN50 | 21.72 | 32.19 | 45.38 | 56.93 | 67.35 | -
BeamCLIP (KL) | S | RN50 | 52.10 | 52.93 | 53.88 | 55.15 | 56.12 | -
Table 16: Effect of the random text prompts on CIFAR100. |  |  |  | Prompts |  |
---|---|---|---|---|---|---
Method | Type | Img. Enc. | 100 | 500 | 1000 | -
CLIP [32] (zero-shot) | T | ViT-B/16 | - | - | - | 68.7
CLIP [32] (zero-shot) | - | RN50 | - | - | - | 41.6
BeamCLIP (CE+EntMin) | S | RN50 | 45.36 | 52.44 | 58.93 | -
BeamCLIP (KL) | S | RN50 | 48.68 | 46.82 | 46.15 | -


# Engineering multiple GHz mechanical modes in optomechanical crystal cavities

Laura Mercadé  Nanophotonics Technology Center, Universitat Politècnica de
València, Camino de Vera s/n, 46022 Valencia, Spain  MIND-IN2UB, Departament
d’Enginyeria Electrònica i Biomèdica, Facultat de Física, Universitat de
Barcelona, Martí i Franquès 1, Barcelona 08028, Spain  Raúl Ortiz
Nanophotonics Technology Center, Universitat Politècnica de València, Camino
de Vera s/n, 46022 Valencia, Spain  Alberto Grau  Nanophotonics Technology
Center, Universitat Politècnica de València, Camino de Vera s/n, 46022
Valencia, Spain  Amadeu Griol  Nanophotonics Technology Center, Universitat
Politècnica de València, Camino de Vera s/n, 46022 Valencia, Spain  Daniel
Navarro-Urrios  MIND-IN2UB, Departament d’Enginyeria Electrònica i Biomèdica,
Facultat de Física, Universitat de Barcelona, Martí i Franquès 1, Barcelona
08028, Spain  Alejandro Martínez  Nanophotonics Technology Center, Universitat
Politècnica de València, Camino de Vera s/n, 46022 Valencia, Spain
<EMAIL_ADDRESS>
###### Abstract

Optomechanical crystal cavities (OMCCs) are fundamental nanostructures for a
wide range of phenomena and applications. Usually, optomechanical interaction
in such OMCCs is limited to a single optical mode and a unique mechanical
mode. In this sense, eliminating the single mode constraint - for instance, by
adding more mechanical modes - should enable more complex physical phenomena,
giving rise to a context of multimode optomechanical interaction. However, a
general method to produce in a controlled way multiple mechanical modes with
large coupling rates in OMCCs is still missing. In this work, we present a
route to confine multiple GHz mechanical modes coupled to the same optical
field with similar optomechanical coupling rates - up to 600 kHz - by OMCC
engineering. In essence, we increase the number of unit cells (consisting of a
silicon nanobrick perforated by a circular holes with corrugations at its both
sides) in the adiabatic transition between the cavity center and the mirror
region. Remarkably, the mechanical modes in our cavities are located within a
full phononic bandgap, which is a key requirement to achieve ultra high
mechanical Q factors at cryogenic temperatures. The multimode bevavior in a
full phononic bandgap and the easiness of realization using standard silicon
nanotechnology make our OMCCs highly appealing for applications in the
classical and quantum realms.

††preprint: AIP/123-QED

##  I Introduction

Cavity optomechanics, the scientific field that studies the interaction
between light and mechanics in solid cavities Aspelmeyer, Kippenberg, and
Marquardt (2014); Pennec  _et al._ (2014), usually considers the coupling of a
single mechanical mode with a single optical field Chan  _et al._ (2011);
Gröblacher  _et al._ (2009); Verhagen  _et al._ (2012). But more complex
physics and novel phenomena may arise when considering multiple mechanical
modes coupled to a single optical mode Nielsen  _et al._ (2017); del Pino,
Slim, and Verhagen (2022); Mercadé  _et al._ (2021). Among others, the
interest in multimode self-oscillating systems has led to emergent phenomena
including dynamical topological phases Walter and Marquardt (2016), analog
simulators Mahboob, Okamoto, and Yamaguchi (2016), synchronization Heinrich
_et al._ (2011); Zhang  _et al._ (2012); Holmes, Meaney, and Milburn (2012);
Lörch  _et al._ (2017); Colombano  _et al._ (2019); Pelka, Peano, and Xuereb
(2020); Madiot  _et al._ (2020) and stability enhancement Zhang  _et al._
(2015), with the last two applications employing multiple confined mechanical
modes coupled through an optical field.

In many of these multimode systems, the involved mechanical modes are not
confined into the same physical structure. This is the case, for example, of
confined mechanical modes in different optomechanical crystal cavities (OMCCs)
that are coupled via mechanical interaction Colombano  _et al._ (2019) or
coupled micromechanical oscillators that interact through an optical radiation
field but are also physically localized at different resonators Gil-Santos
_et al._ (2017); Zhang  _et al._ (2012, 2015). Conversely, multiple mechanical
modes confined in the same structure and interacting with one common
intracavity optical field have also been studied in the literature for a given
number of mechanical oscillators Sheng  _et al._ (2020); Mathew, Pino, and
Verhagen (2020); Leijssen  _et al._ (2017). Remarkably, most of these systems
involve oscillators up to MHz frequencies, so a general route towards multiple
GHz mechanical resonances in a single cavity is still missing. Recently, a
bullseye optomechanical (OM) resonator enabling multiple mechanical and
optical modes has been fabricated and tested Santos  _et al._ (2017), but the
vacuum OM coupling rates are about one of magnitude smaller than in silicon
OMCCs Chan  _et al._ (2011); Mercadé  _et al._ (2020).

For some applications, it would also be interesting to have all these
mechanical modes placed in a full phononic bandgap that prevents phonons to
escape from the cavity. This could be particularly helpful in quantum
applications at cryogenic temperatures, because of the enhancement of the
mechanical Q factor in these conditions MacCabe  _et al._ (2019). A reliable
route to multiple phonon modes with large coupling rates would also open the
door to versatile all-optical OM-based microwave signal synthesis Mercadé  _et
al._ (2020) and processing Mercadé  _et al._ (2021), which is especially
valuable for application in wireless systems, in particular those requiring
extreme compactness and lightweight (satellite communications) Mercadé;  _et
al._ (2022).

Here, we propose a general method to get multiple mechanical modes with GHz
frequencies and placed in a full phononic bandgap of a silicon OMCC. Starting
with the design in Oudich  _et al._ (2014) and demonstrated experimentally in
Mercadé  _et al._ (2020), we go a step further and demonstrate that
engineering the central region of the cavity enables to systematically
increase the number of mechanical modes whilst ensuring relatively large
values of the OM coupling rate ($g_{0}$/2$\pi$ $\approx 400kHz$). Although we
use silicon as underlying material, this approach should also be of
application when using other high refractive index materials to build the
cavities. Such large values of $g_{0}$ allows us to easily transduce all the
confined modes into the driving optical field. We demonstrate experimentally
the existence of up to six modes in a single cavity, though our method could
be used to get even more mechanical modes. All these features, together with
the fabrication on a silicon chip using standard nanofabrication tools,
suggest that our cavities could play a role in the development of multimode
cavity optomechanics for different classical and quantum applications.

##  II Photonic and phononic band diagrams of the OMCC

Prior to the design of the defect cells and the cavity modes, we perform a
broad analysis of the mirror region of the OMCC to optimize both the photonic
and phononic bandgaps. Figure 1(a) sketches the unit cell used to build up our
cavity. It consists of a 220 nm thick silicon nanobrick drilled with a
circular hole and surrounded by lateral stabs or corrugations. The resulting
photonic band diagram of this unit cell is depicted in Fig. 1(b), showing the
TE-like (in blue) and the TM-like (in orange) bands, respectively. The gray
shadowed area corresponds to the light cone and the blue shadowed area,
between the third and fourth TE-like bands, denotes the quasi-TE bandgap,
which will be used to confine the driving optical mode. It must be noted that
this unit cell provides a large tunability in wavelength, as can be seen in
Fig. 1(c), which depicts the evolution of the first TE bandgap as a function
of the aspect ratio of the unit cell for three different period lattices. A
broad tunability for wavelengths between 1200 nm and 2200 nm can be obtained,
which provides a great versatility in the design. As our objective is to
confine an optical mode around 1550 nm (depicted as a dashed line in Fig.
1(b,c)), from now on we will focus our studies in a period lattice for the
mirror part of the cavity with $a=$500 nm.

Figure 1: Photonic band calculations of the mirror unit cell. (a) Array of the
unit cells under study depicting the main parameters. (b) Photonic band
diagram of the mirror unit cell for a lattice period $a$=500 nm, $t$=220 nm,
$l/a=$0.9, $d/a=$0.4 and $r/a=0.3$. (c) Evolution of the total photonic
bandgap of different aspect ratios for three different lattice periods (shown
in the inset, values in nm). From left to right, the results shown in the
panels are obtained for the ratios $l/a=$0.9 and $d/a=$0.4 (first panel),
$r/a=$0.3 and $d/a=$0.4 (second panel) and $r/a=$0.3 and $l/a=$0.9 (third
panel), where $d$, $r$ and $l$ are geometrical parameters of the cavity cell
showed in Fig. 1(a). Figure 2: Phononic band calculations of the mirror unit
cell. (a) Phononic band diagram of the mirror unit cell for a lattice period
$a$=500 nm, $t$=220 nm, $l/a=$0.9, $d/a=$0.4 and $r/a=0.3$. (b) Transmission
acoustic spectra for an array of multiple mirror cells ranging from 5 to 15
mirrors. (c) Evolution of the total phononic bandgap for the same aspect
ratios shown in Fig. 1(c).

Once we have the appropriate lattice period to confine the optical mode at the
target wavelength, we study the mechanical properties of the mirror cell. One
of the main advantages of this unit cell profile is that it holds a complete
phononic bandgap Oudich  _et al._ (2014); Mercadé  _et al._ (2020), as can be
seen in Fig. 2(a). Additionally, an analysis in the phononic transmission
spectra for an array of multiple mirror cells is shown in Fig. 2(b). Here, a
higher number of mirror cells provides deeper transmission gaps, as expected
from the existence of a phononic bandgap. Finally, the evolution of the full
phononic bandgap for different aspect ratios is presented in Fig. 2(c). Again,
the main advantage can be seen in the tunability of the system ranging almost
all the microwave S-band (between 2 and 4 GHz). More details about the
mechanical calculations are provided in the Supporting Information (Fig. S1).

##  III Design of the multimode OMCC

After obtaining the mirror unit cell through the calculations of the photonic
and phononic band diagrams, we have tailored the defect and transition unit
cells to be able to confine multiple mechanical modes inside the phononic
bandgap. Previous designs just focused on a cavity where the mechanical mode
was mostly confined into the lateral corrugations of the defect cell Oudich
_et al._ (2014); Mercadé  _et al._ (2020). However, other mechanical modes,
located in other lateral corrugations, could also lie in the bandgap. Figure
3(a) shows a top view of the silicon OMCC that we consider in this work,
including the labeled number of transition cells that will be under study. As
shown in Fig. 2, this cavity presents a full phononic bandgap for mechanical
modes in at frequencies around 3-4 GHz (chosen by design). In order to
understand the existence of multiple GHz mechanical modes in a single OMCC and
with the final aim of setting a route towards a general method, we obtain the
phononic band diagrams of the different unit cells in the transition region
between the cavity center and the mirrors (blue region in Fig. 3(a)).
Depending on the total number of transition cells (NT), there will be a
certain number of mechanical modes confined in the OMCC with frequencies in
the phononic bandgap.

Figure 3: Multimode mechanical design. (a) Sketch of the silicon OMCC used in
this work. (b) Evolution of the confined mechanical band at the $\Gamma$ point
for different cavity lengths as a function of the analyzed unit cell. (c)
Estimation of the mechanical modes lying into the total bandgap for OMCC with
different transition cells. The predicted frequency corresponds to the
mechanical frequency of the confined band for each defect and transition cells
lying into the total bandgap.

Figure 3(b) shows the evolution of the frequency of the mechanical mode inside
the bangap at the $\Gamma$ point for different unit cells that represent the
transition from the central defect (UD) to the side mirror (UM), at each
constitutive cavity cell. The panels correspond to OMCCs with 6, 9 and 12
transition cells, respectively. This suggests that, when forming the OMCC,
mechanical modes will appear at the calculated frequencies (depicted with
dots) inside the total bandgap (drawn with a shaded area) and localized in the
unit cells that form the transition between the center of the cavity and the
mirrors.

Remarkably, as the total number of transition cells (NT) increases, more
mechanical modes appear inside the phononic bandgap, though the increase is
not linear. Consequently, an OMCC created with more transition cells should
result in more confined mechanical modes. This can be appreciated in Fig.
3(c), which shows the evolution of the $\Gamma$ point frequencies of the
mechanical modes inside the full phononic bandgap as a function of NT.
Furthermore, the mechanical frequencies get closer as more transition cells
are added, since the physical dimensions between successive cells change more
smoothly.

The next question to address is if all mechanical modes are well coupled with
the optical field. To this end we calculate the OM coupling rates $g_{0}$ of
the different mechanical modes of an OMCC with $N_{T}=12$. Figure 4(a) shows
the optical mode of this OMCC. The electric field pattern shows a strong
localization in the center of the cavity whilst the intensity exponentially
decays as we move toward the mirror regions (see Fig. 4(b)). Once the optical
mode is obtained, we separately simulate the fundamental mechanical modes of
the OMCC. As it was explained before, not all the mechanical modes display
large values of $g_{0}$. Hence, for all of them we calculate the OM coupling
rate $g_{0}$ with the optical field obtained in 4(a) to find the ones that
have the largest $g_{0}$. Figure 4(c) depicts the six mechanical modes that
have the highest OM coupling rates. We consider both the photo-elastic (PE) as
well as the moving interface (MI) effects Pennec  _et al._ (2014) that
contribute to the total $g_{0}$. It is worth noting that the six mechanical
modes correspond to the ones predicted in Fig. 3(c). Finally, Fig. 4(d)
gathers the mechanical profiles of those modes, which are localized at the
lateral corrugation of the defect and transition cells. Indeed, the first mode
shows localization in the center of the cavity whilst the mechanical
localization is displaced toward the mirror regions for higher-order modes.
Noticeably, we get relatively large $g_{0}$ values even when the mechanical
modes are localized close to the mirror regions and far from the cavity
center.

Figure 4: Optical and mechanical modes and OM coupling rates in the OMCC. (a)
Electric field profile for an OMCC with 12 transition unit cells at a
wavelength of 1570 nm. (b) Normalized total electric field variation along the
total length of the cavity. (c) Calculated OM coupling rate contributions for
the six fundamental mechanical modes within the phononic bandgap. (d)
Displacement field profile for the six mechanical modes giving the largest
$g_{0}$. Figure 5: Experimental setup for the optical and mechanical
characterization of the OMCCs. VOA: variable optical attenuator, PC:
polarization controller, EDFA: erbium doped fiber amplifier, SCOPE:
oscilloscope, RSA: radiofrequency spectrum analyzer.

##  IV Experimental results

The optical and mechanical experimental characterization of a set of
fabricated OMCCs with different transition cells has been performed with the
setup sketched in Fig. 5. Here, a tunable laser is fed into the system and a
variable optical attenuator (VOA) and a controller polarizer are used to set
the required input power and polarization in the experimental measurements.
Then, the optical signal is sent through an optical circulator. From port 2,
the optical signal arrives to a tapered fiber loop which couples to the OMCC
via evanescent field coupling. The transmitted signal arrives to a low
frequency photodetector connected to the oscilloscope to measure the optical
response of the system. On the other side, the reflected signal returns to the
optical circulator (port 3) and its optical power is regulated by means of a
VOA and an erbium doper fiber amplifier (EDFA). Finally, the output signal
(with the mechanical modes transduced on the optical drive) is photodetected
with a 12 GHz band photoreceiver and processed with a radiofrequency spectrum
analyzer (RSA).

Figure 6: Optical characterization of different OMCCs. (a) Scanning electron
microscopy (SEM) images of a fabricated optomechanical crystal cavity with 6
and 11 transition cells. (b) Evolution of the measured wavelength of the two
detected optical modes as a function of the number of transition cells. (d)
Optical quality factor as a function of the number of transition cells of the
cavity. (d) Optical field profiles (E) for the first and second measured
optical modes calculated by retrieving the OMCC pattern from an SEM image.

Regarding the optical response, the measurements were performed under a low
laser input power in order to prevent the appearance of thermo-optic effect,
which may result in a bistability asymmetric ”saw-tooth” shaped transmission
that gives rise to a shift in the optical resonance Navarro-Urrios  _et al._
(2014). For a given design, a set of 12 OMCCs for each number of transition
cells was fabricated and characterized. The fabrication process is described
elsewhere Mercadé  _et al._ (2020). Figure 6(a) shows two scanning electron
microscope (SEM) images of fabricated OMCCs having 6 and 11 transition cells
OMCCs. In comparison with the designed cavity, a fabrication-induced rounding
of the lateral corrugations is clearly appreciated. The optical response was
studied for each fabricated cavity as shown in Fig. 6. The signal obtained
from characterizing each optical mode was a symmetric resonance where a
Lorentzian fit was performed to retrieve the optical frequency and quality
factor. As shown in Fig. 6(b), the cavity supported two optical modes, whose
wavelength decreases as more transition cells are added to the cavity. On the
other hand, the quality factor increases with $N_{T}$ for both modes as seen
in Fig. 6(c), with some exceptions that can be attributed to fabrication
irregularities. Figure 6(d) represents the electric field profile of both
optical modes, which are localized at the defect and transition cells. These
profiles were obtained by retrieving the real fabricated pattern from the SEM
images.

Figure 7: Mechanical response of the OMCCs. Experimental mechanical mode
response (represented as a normalized RF power) for different OMCCs as a
function of the number of transition cells.

Concerning the mechanical response, a typical evolution of the measured
normalized radiofrequency (RF) spectra showing the mechanical response as a
function of the number of transition cells is depicted in Fig. 10(a). The
increase in the number of mechanical modes with $N_{T}$ predicted in the
simulations is also observed in the experiments. Although we did not find all
predicted modes, probably because fabrication imperfections (as shown in Fig.
6(a)), the measured frequencies are around 4 GHz, which is close to the
simulated values and ensures us that the modes come from the lateral
corrugations of the defect and transition cells, since those are the only ones
that can vibrate at those frequencies in the OMCC. The experimental mechanical
modes obtained for other cavities are included in the Supporting Information.

Figure 8: Study of a fabricated OMCC with 10 transition cells. (a) Phononic
band diagram of the retrieved SEM unit cells for one of the measured cavities
under study. (b) PSD spectrum measurements for a cavity with 10 transition
cells. (c) Retrieved SEM mechanical mode profiles from the fabricated
structure.

A detailed analysis of the characterization of an OMCC with 10 transition
cells is presented in Fig. 8. First, an analysis of the phononic band diagram
for the fabricated structure was performed. As one of the main objectives here
is to confine the mechanical modes into a total bandgap, we retrieved from the
SEM images the real profile for the different unit cells of the mirror cavity
and calculated the expected band diagram. The result can be seen in Fig. 8(a),
where an inset of a mirror unit cell of the measured OMCC is presented. As
expected, the cavity shows a full phononic bandgap around 4 GHz and the
measured mechanical modes lies into it, as shown in Fig. 8(b). Here, it can be
seen that, even after fabrication imperfections, it is quite in accordance
with the mechanical modes predicted with the band diagram shown in Fig. 3(c).
However, it must be noted that double peaks as the one corresponding to peak 2
can appear as a result in difference between the fabricated corrugations in
cells of the same nominal dimensions. To ascertain the mechanical mode
profiles of each measured peak, we retrieved and simulated the actual
fabricated OMCC profile from its SEM image. Figure 8(c) shows the mechanical
mode profiles for the retrieved profile showing that, despite the mechanical
motion is not totally confined in a single corrugation, as in the nominal
cavity in Fig. 4, the position of mechanical displacement moves toward the
extremes of the cavity as the frequency decreased, as expected from the
numerical modelling.

##  V Conclusions

In this work, we have proposed and demonstrated a method to engineer multiple
mechanical modes with GHz frequencies within a full phononic bandgap in a
silicon OMCC. The OMCC is formed by drilling circular holes and adding lateral
corrugations to a released silicon nanobeam. By increasing the number of cells
in the adiabatic transition between the cavity center and the lateral mirrors,
more and more mechanical modes appear in the cavity. All the mechanical modes
show reasonably large values of $g_{0}/2\pi$ up to 600 kHz which enable
efficient transduction into a driving optical signal. Multiple applications
can be envisaged, including multimode phonon lasers Mercadé  _et al._ (2021),
frequency up- and down-conversion of multiple wireless signals Mercadé  _et
al._ (2021) or building chiral nano-optomechanical networks del Pino, Slim,
and Verhagen (2022).

##  Appendix A Phononic band diagram and transmission simulations

Simulations of phononic bands were performed with COMSOL Multiphysics com , as
in previous works Mercadé  _et al._ (2020). The evolution of the frequency at
at the $\Gamma$ symmetry point for the studied unit cell changing the
parameters from the central defect to the mirror region is presented in Fig.
S9(a). Figure S9(b) shows in more detail all the involved bands $-$ including
different symmetries $-$ for the mirror unit cell. The most important feature
to emphasize is the existence of a complete phononic bandgap where, by proper
design, the frequencies of the engineered confined mechanical modes have to be
placed. As noted in the main text, this should reduce the phonon leakage of
the final structure as it prevents that the confined mechanical mode couples
with modes of different symmetries existing in the mirror regions. In these
simulations, we set Floquet Periodic Conditions (FPC) at the lateral
boundaries of the structure, as shown in Fig. S9(c), and the remaining
boundaries were kept as free. The dimension values of the mirror and the unit
cell are the same as the ones used in the main text. Regarding the estimation
of the number of expected mechanical modes to be confined in cavities with
different transition cells, the analysis was similar to the one performed in
S9(a). Here, the point is that, as the number of transition cells used to
build up the cavity increases, there will be more unit cells with dimensions
close to those of the center region with frequencies within the bandgap.
Because of that, as presented in Fig. 3(b) in the main text, we can estimate
the total number of mechanical modes just by analyzing the eigenfrequencies
and eigenvalues of each unit cell. It must be noted, however, that the cavity
may also support less localized mechanical modes, as presented in Fig. 3(d),
but, as shown in the same figure, the estimation of the number of confined
mechanical modes is still correct.

Figure 9: (a) Phononic band diagram evolution of the confined mechanical mode
band from the defect (UM) to the mirror (UD) unit cell. (b) Phononic bands for
mechanical modes with (odd, odd) in purple, (even, even) in blue, (even, odd)
in yellow and (odd, even) in green symmetries respect to the Y and Z axis
planes. The gray-shaded area denotes the total phononic bandgap and the
confined mechanical mode in the defect zone has been depicted with a dashed
line. (c) Mechanical unit cell showing the boundary Floquet Periodic Condition
(FPC) of the simulation. (d) Mechanical field profile of (UM) in A and (UD) in
B. (e) Tranmission phononic simulations scheme. The PMLs are shadowed in pink
and the source and receiver areas are depicted in blue.

Regarding the phononic transmission simulations, an scheme of the boundary
conditions of the system is presented in Fig. S9(e). Here, we simulated the
transmission coefficient of a signal generated at the load source (set as a
boundary source in this case) and received at the receiver area. The
transmission coefficient was calculated as the ratio total displacement (u)
integrated in the source and the receiver as
$\int_{Receiver}(\textbf{u})/\int_{Source}(\textbf{u})$. At the extreme
boundaries of the system we set perfectly matched layers and the rest of the
boundaries were kept as free. This simulation was also performed with COMSOL
Multiphysics.

##  Appendix B Measurement of the transduced mechanical modes

In the experiments, we were able to test OMCCs with different values of
$N_{T}$, all with the same nominal parameters according to our simulations.
Figure S10 shows the measured RF spectra for different values of $N_{T}$ of
cavities with the same nominal parameters as those reported in the main text
but fabricated with a different e-beam dose. The appearance of multiple
mechanical modes within the phononic bandgap is also evident here.

We also measured the mechanical Q factor of the transduced mechanical modes.
The results are shown in Fig. S11, which represents the mechanical Q factor of
the different mechanical modes for each value of $N_{T}$. The mechanical
quality factor was evaluated from the ratio between the center frequency of
the peak and mechanical linewidth. The presented values were obtained through
a lorentzian fit of each peak, as can be seen in the green fits in Fig. S10
for different cavity lengths. In all these panels we can also presented in red
the total fit envelope of the system. It can be seen that the average value is
around 1000, as expected in this kind of cavity when operated at room
temperature.

Figure 10: Experimental mechanical mode response for different OMCCs as a
function of the number of transition cells. In green are presented the
lorentizian peaks of each individual mechanical mode and in red the resulting
envelope fit. Figure 11: Retrieved experimental mechanical quality factors
from the lorentzian fits of the transduced peaks shown in Fig. S10.

We also compared the response of OMCCs with the same $N_{T}$ and identical
nominal values, fabricated with the same e-beam does. The results for
different fabricated OMCCs having $N_{T}=6$ are depicted in Fig. S12(a) for a
set of 6 measurements, for the sake of clarity. It can be observed that the
spectral dispersion is low, as seen in Fig. S12(b) that shows the mean value
and the standard deviation for a set of 12 cavities.

Figure 12: (a) Normalized RF power spectra of multiple measurements of
cavities with 6 transition cells nominally equal. (b) Mean value and standard
deviation of the two main peaks for a cavity with 6 transition cells for a
statistic sample of 12 cavities.

###### Acknowledgements.

The authors acknowledge funding from the Spanish State Research Agency
(PID2021-124618NB-C21 and PID2021-124618NB-C22); Generalitat Valenciana
(BEST/2020/178, PROMETEO/2019/123, IDIFEDER/2020/041 and IDIFEDER/2021/061).
L.M. thanks financial support from the Next generation EU program, Ministerio
de Universidades (Gobierno de España).

## Data Availability Statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * Aspelmeyer, Kippenberg, and Marquardt (2014) M. Aspelmeyer, T. J. Kippenberg, and F. Marquardt, “Cavity optomechanics,” Rev. Mod. Phys. 86, 1391–1452 (2014).
  * Pennec  _et al._ (2014) Y. Pennec, V. Laude, N. Papanikolaou, B. Djafari-Rouhani, M. Oudich, S. E. Jallal, J. C. Beugnot, J. M. Escalante, and A. Martínez, “Modeling light-sound interaction in nanoscale cavities and waveguides,” Nanophotonics 3, 413 – 440 (2014).
  * Chan  _et al._ (2011) J. Chan, T. P. M. Alegre, A. H. Safavi-Naeini, J. T. Hill, A. Krause, S. Gröblacher, M. Aspelmeyer, and O. Painter, “Laser cooling of a nanomechanical oscillator into its quantum ground state,” Nature 478, 89–92 (2011).
  * Gröblacher  _et al._ (2009) S. Gröblacher, K. Hammerer, M. R. Vanner, and M. Aspelmeyer, “Observation of strong coupling between a micromechanical resonator and an optical cavity field,” Nature 460, 724–727 (2009).
  * Verhagen  _et al._ (2012) E. Verhagen, S. Deléglise, S. Weis, A. Schliesser, and T. J. Kippenberg, “Quantum-coherent coupling of a mechanical oscillator to an optical cavity mode,” Nature 482, 63–67 (2012).
  * Nielsen  _et al._ (2017) W. H. P. Nielsen, Y. Tsaturyan, C. B. Møller, E. S. Polzik, and A. Schliesser, “Multimode optomechanical system in the quantum regime,” Proceedings of the National Academy of Sciences 114, 62–66 (2017), https://www.pnas.org/doi/pdf/10.1073/pnas.1608412114 .
  * del Pino, Slim, and Verhagen (2022) J. del Pino, J. J. Slim, and E. Verhagen, “Non-hermitian chiral phononics through optomechanically induced squeezing,” Nature 606, 82–87 (2022).
  * Mercadé  _et al._ (2021) L. Mercadé, K. Pelka, R. Burgwal, A. Xuereb, A. Martínez, and E. Verhagen, “Floquet phonon lasing in multimode optomechanical systems,” Phys. Rev. Lett. 127, 073601 (2021).
  * Walter and Marquardt (2016) S. Walter and F. Marquardt, “Classical dynamical gauge fields in optomechanics,” New Journal of Physics 18, 113029 (2016).
  * Mahboob, Okamoto, and Yamaguchi (2016) I. Mahboob, H. Okamoto, and H. Yamaguchi, “An electromechanical ising hamiltonian,” Sci Adv 2, e1600236 (2016).
  * Heinrich  _et al._ (2011) G. Heinrich, M. Ludwig, J. Qian, B. Kubala, and F. Marquardt, “Collective dynamics in optomechanical arrays,” Phys. Rev. Lett. 107, 043603 (2011).
  * Zhang  _et al._ (2012) M. Zhang, G. S. Wiederhecker, S. Manipatruni, A. Barnard, P. McEuen, and M. Lipson, “Synchronization of micromechanical oscillators using light,” Phys. Rev. Lett. 109, 233906 (2012).
  * Holmes, Meaney, and Milburn (2012) C. A. Holmes, C. P. Meaney, and G. J. Milburn, “Synchronization of many nanomechanical resonators coupled via a common cavity field,” Phys. Rev. E 85, 066203 (2012).
  * Lörch  _et al._ (2017) N. Lörch, S. E. Nigg, A. Nunnenkamp, R. P. Tiwari, and C. Bruder, “Quantum synchronization blockade: Energy quantization hinders synchronization of identical oscillators,” Phys. Rev. Lett. 118, 243602 (2017).
  * Colombano  _et al._ (2019) M. F. Colombano, G. Arregui, N. E. Capuj, A. Pitanti, J. Maire, A. Griol, B. Garrido, A. Martinez, C. M. Sotomayor-Torres, and D. Navarro-Urrios, “Synchronization of optomechanical nanobeams by mechanical interaction,” Phys. Rev. Lett. 123, 017402 (2019).
  * Pelka, Peano, and Xuereb (2020) K. Pelka, V. Peano, and A. Xuereb, “Chimera states in small optomechanical arrays,” Phys. Rev. Research 2, 013201 (2020).
  * Madiot  _et al._ (2020) G. Madiot, F. Correia, S. Barbay, and R. Braive, “Bichromatic synchronized chaos in coupled optomechanical nanoresonators,” (2020), arXiv:2005.08896 [nlin.CD] .
  * Zhang  _et al._ (2015) M. Zhang, S. Shah, J. Cardenas, and M. Lipson, “Synchronization and phase noise reduction in micromechanical oscillator arrays coupled through light,” Phys. Rev. Lett. 115, 163902 (2015).
  * Gil-Santos  _et al._ (2017) E. Gil-Santos, M. Labousse, C. Baker, A. Goetschy, W. Hease, C. Gomez, A. Lemaître, G. Leo, C. Ciuti, and I. Favero, “Light-mediated cascaded locking of multiple nano-optomechanical oscillators,” Phys. Rev. Lett. 118, 063605 (2017).
  * Sheng  _et al._ (2020) J. Sheng, X. Wei, C. Yang, and H. Wu, “Self-organized synchronization of phonon lasers,” Phys. Rev. Lett. 124, 053604 (2020).
  * Mathew, Pino, and Verhagen (2020) J. P. Mathew, J. d. Pino, and E. Verhagen, “Synthetic gauge fields for phonon transport in a nano-optomechanical system,” Nature Nanotechnology 15, 198–202 (2020).
  * Leijssen  _et al._ (2017) R. Leijssen, G. R. La Gala, L. Freisem, J. T. Muhonen, and E. Verhagen, “Nonlinear cavity optomechanics with nanomechanical thermal fluctuations,” Nature Communications 8, ncomms16024 (2017).
  * Santos  _et al._ (2017) F. G. S. Santos, Y. A. V. Espinel, G. O. Luiz, R. S. Benevides, G. S. Wiederhecker, and T. P. M. Alegre, “Hybrid confinement of optical and mechanical modes in a bullseye optomechanical resonator,” Opt. Express 25, 508–529 (2017).
  * Mercadé  _et al._ (2020) L. Mercadé, L. L. Martín, A. Griol, D. Navarro-Urrios, and A. Martínez, “Microwave oscillator and frequency comb in a silicon optomechanical cavity with a full phononic bandgap,” Nanophotonics 9, 3535–3544 (2020).
  * MacCabe  _et al._ (2019) G. S. MacCabe, H. Ren, J. Luo, J. D. Cohen, H. Zhou, A. Sipahigil, M. Mirhosseini, and O. Painter, “Phononic bandgap nano-acoustic cavity with ultralong phonon lifetime,” (2019), arXiv:1901.04129 [cond-mat.mes-hall] .
  * Mercadé  _et al._ (2021) L. Mercadé, M. Morant, A. Griol, R. Llorente, and A. Martínez, “Photonic frequency conversion of ofdm microwave signals in a wavelength-scale optomechanical cavity,” Laser & Photonics Reviews 15, 2100175 (2021), https://onlinelibrary.wiley.com/doi/pdf/10.1002/lpor.202100175 .
  * Mercadé;  _et al._ (2022) L. Mercadé;, E. Rico, J. R. Garnica, J. C. Gómez, A. Griol, M. A. Piqueras, A. Martínez, and V. C. Duarte, “Testing optomechanical microwave oscillators for satcom application,” Journal of Lightwave Technology 40, 4539–4547 (2022).
  * Oudich  _et al._ (2014) M. Oudich, S. El-Jallal, Y. Pennec, B. Djafari-Rouhani, J. Gomis-Bresco, D. Navarro-Urrios, C. M. Sotomayor Torres, A. Martínez, and A. Makhoute, “Optomechanic interaction in a corrugated phoxonic nanobeam cavity,” Phys. Rev. B 89, 245122 (2014).
  * Navarro-Urrios  _et al._ (2014) D. Navarro-Urrios, J. Gomis-Bresco, S. El-Jallal, M. Oudich, A. Pitanti, N. Capuj, A. Tredicucci, F. Alzina, A. Griol, Y. Pennec, B. Djafari-Rouhani, A. Martínez, and C. M. Sotomayor Torres, “Dynamical back-action at 5.5 ghz in a corrugated optomechanical beam,” AIP Advances 4, 124601 (2014).
  * (30) _COMSOL Multiphysics 5.3a_.