# RWKV-CLIP: A Robust Vision-Language Representation Learner

Tiancheng Gu${}^{\text{\char 170}}$*, Kaicheng Yang${}^{\text{\char 171}}$*,
Xiang An${}^{\text{\char 171}}$, Ziyong Feng${}^{\text{\char 171}}$
Dongnan Liu${}^{\text{\char 170}}$, Weidong Cai${}^{\text{\char
170}}$$\dagger$, Jiankang Deng${}^{\text{\char 105}}$$\dagger$
${}^{\text{\char 170}}$University of Sydney ${}^{\text{\char 171}}$DeepGlint
${}^{\text{\char 105}}$Huawei UKRD
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Contrastive Language-Image Pre-training (CLIP) has significantly improved
performance in various vision-language tasks by expanding the dataset with
image-text pairs obtained from websites. This paper further explores CLIP from
the perspectives of data and model architecture. To address the prevalence of
noisy data and enhance the quality of large-scale image-text data crawled from
the internet, we introduce a diverse description generation framework that can
leverage Large Language Models (LLMs) to synthesize and refine content from
web-based texts, synthetic captions, and detection tags. Furthermore, we
propose RWKV-CLIP, the first RWKV-driven vision-language representation
learning model that combines the effective parallel training of transformers
with the efficient inference of RNNs. Comprehensive experiments across various
model scales and pre-training datasets demonstrate that RWKV-CLIP is a robust
and efficient vision-language representation learner, it achieves state-of-
the-art performance in several downstream tasks, including linear probe, zero-
shot classification, and zero-shot image-text retrieval. To facilitate future
research, the code and pre-trained models are released at
https://github.com/deepglint/RWKV-CLIP ††* Equal contribution. ††$\dagger$
Corresponding author.

##  1 Introduction

The proliferation of mobile networks and social platforms has greatly
accelerated the large-scale production of image-text pairs. This unprecedented
abundance of data has established the foundation for vision-language pre-
training. Contrastive Language-Image Pre-training (CLIP) employs two distinct
unimodal encoders for images and text, utilizing a contrastive loss, a highly
effective mechanism for representation learning. Having been pre-trained on
extensive image-text pairs collected from the internet, CLIP demonstrates
strong transferability and has been widely applied across various domains
zhou2022zegclip; yao2023detclipv2.

Figure 1: The proposed RWKV-CLIP combines the effective parallel training of
transformers with the efficient inference of RNNs, achieving better efficiency
and accuracy than the baseline methods (e.g., CLIP and ALIP).

In recent years, many large-scale image-text datasets collected from the
internet have been released. LAION400M schuhmann2021laion is created for
research purposes and it contains 400 million image-text pairs curated using
the CLIP model. Building on this, LAION5B schuhmann2022laion, which consists
of 5.85 billion CLIP-filtered image-text pairs, successfully replicates and
fine-tunes basic models such as CLIP. However, using the CLIP model to filter
web-based image-text pairs still retains a considerable presence of noisy
data. To improve data quality, DataComp gadre2024datacomp employs various
strategies such as basic filtering, CLIP score filtering, and text&image-based
filtering. However, inherent characteristics of internet data, such as
abstract text representations and semantic discrepancies between text and
images, remain significant obstacles.

In recent years, the Transformer vaswani2017attention model has been
extensively applied in large-scale representation learning, yielding
significant performance improvements across multiple downstream tasks
acosta2022multimodal; kirillov2023segment; wang2023internimage, including
image classification dosovitskiy2020image; Wang_2023_CVPR, text generation
brown2020language, and speech recognition radford2023robust. Despite these
achievements, the quadratic computational complexity inherent in Transformer
limits its capacity to effectively process high-resolution images and long
sequences, posing a substantial challenge to its broader applicability across
varied domains.

In this paper, we design a framework for generating diverse descriptions.
Following ALIP alip_Yang_2023_ICCV, we first use the OFA wang2022ofa model to
generate synthetic descriptions consistent with image content. However,
constrained by the training data, OFA can only partially identify coarse-
grained object categories. Therefore, we introduce an open-set image tagging
model RAM++ huang2023open to capture more detailed and precise semantic
information from images. By leveraging LLMs, we synthesize and refine
information from web-based texts, synthetic captions, and detection tags.
Additionally, inspired by RWKV peng2024eagle and Vision-RWKV duan2024vision,
we propose RWKV-CLIP, the first RWKV-driven vision-language representation
learning model that combines the effective parallel training of Transformers
with the efficient inference of RNNs. Extensive experiments across various
model scales and pre-training datasets demonstrate that RWKV-CLIP is a robust
and efficient vision-language representation learner. The main contributions
of this paper are summarized as follows:

  * •

We introduce a diverse description generation framework, which can leverage
LLMs to synthesize and refine information from web-based texts, synthetic
captions, and detection tags to produce more accurate and semantically
enriched descriptions.

  * •

We propose the RWKV-CLIP, the first RWKV-driven vision-language representation
learning model, which combines the parallel training effectiveness of
Transformers with the inference efficiency of RNNs.

  * •

We demonstrate the robustness and effectiveness of RWKV-CLIP as a vision-
language representation learner through extensive experiments across various
model scales and pre-training datasets.

##  2 Related Work

Figure 2: The architecture of our proposed diverse description generation
framework.

###  2.1 Vision-Language Representation Learning

As the milestone in vision-language representation learning, CLIP
radford2021learning has garnered unparalleled interest due to its remarkable
zero-shot recognition capability and outstanding transfer performance.
Subsequently, a significant amount of enhancement works based on CLIP have
been proposed. SLIP mu2022slip combines self-supervised learning with CLIP
pre-training to achieve significant performance improvements. DeCLIP
li2021supervision employs multi-view supervision across modalities and
nearest-neighbor supervision from similar pairs to enhance representation
learning efficiency. FILIP yao2021filip refines contrastive loss to learn
fine-grained representations for image patches and sentence words. UniCLIP
lee2022uniclip boosts data efficiency by integrating contrastive loss across
multiple domains into a single universal space. HiCLIP geng2023hiclip enhances
cross-modal alignment by incorporating hierarchy-aware attention into both
visual and language branches of CLIP. ALIP alip_Yang_2023_ICCV introduces a
gating mechanism to reduce the influence of noisy pairs using synthetic data.
Different from the above methods, this paper conducts further exploration of
both the data and model architecture, proposing a diverse description
generation framework and introducing RWKV-CLIP, the first RWKV-driven vision-
language representation model.

###  2.2 Text Augumentation

With the success of LLMs in Natural Language Processing (NLP), there is
growing interest in leveraging LLMs to enhance text descriptions in large-
scale image-text pairs. LaCLIP fan2023improving explores different strategies
to generate rewrite examples and uses the in-context learning ability of LLMs
to rewrite text within image-text datasets. However, the hallucination issue
of LLMs and reliance on limited samples to guide the rewriting process can
still introduce significant noise. To address this, CapsFusion
yu2023capsfusion generates synthetic captions for each image and utilizes
ChatGPT to merge raw texts and synthetic captions, creating a dataset with one
million instructions for LLaMA fine-tuning. Despite this, caption generation
models such as OFA wang2022ofa and BLIP li2022blip are limited by their
training data and can only identify a restricted set of coarse-grained object
categories. In this paper, we introduce the open-set image tagging model RAM++
huang2023open to assign semantic detection tags to each image. Beneficial from
detection tags, more semantic information can be introduced from images, which
in turn further constrains LLMs and mitigates hallucinations.

###  2.3 Receptance Weighted Key Value

RWKV peng2023rwkv is first proposed in NLP, it addresses memory bottleneck and
quadratic scaling in Transformers through efficient linear scaling while
retaining expressive characteristics like parallelized training and robust
scalability. Recently, Vision-RWKV duan2024vision successfully transferred the
RWKV from NLP to vision tasks, outperforming ViT in image classification with
faster processing and reduced memory consumption for high-resolution inputs.
PointRWKV he2024pointrwkv demonstrates leading performance across various
downstream tasks, surpassing Transformer- and Mamba-based counterparts in
efficiency and computational complexity. Furthermore, Diffusion-RWKV
fei2024diffusion adapts RWKV for diffusion models in image generation tasks,
achieving competitive or superior performance compared to existing CNN or
Transformer-based diffusion models. However, these methods have only validated
RWKV in specific downstream tasks, and the potential of RWKVs to replace ViTs
in vision-language representation learning remains unverified.

##  3 Method

Figure 3: The architecture of RWKV-CLIP, which consists of M$\times$ and
N$\times$ RKWV-driven blocks followed by an average pooling layer.

In this section, we first introduce a diverse description generation framework
that leverages the capabilities of large language models to integrate
information from web-based texts, synthetic captions, and detection tags.
Subsequently, we provide a detailed exposition of RWKV-CLIP.

###  3.1 Diverse Description Generation

The architecture of our proposed diverse description generation framework is
illustrated in Fig. 2. To mitigate the effects of mismatched image-text pairs,
following ALIP alip_Yang_2023_ICCV, we first adopt the OFAbase model to
generate a synthetic caption for each image. The synthetic captions exhibit a
high degree of semantic alignment with the image, facilitating alignment
across different modal feature spaces. However, constrained by the training
data, OFAbase can recognize a limited number of object categories and tends to
produce captions with a simplistic sentence structure. To capture finer-
grained semantic information within images, we incorporate the open-set image
tagging models RAM++ huang2023open to extract object detection tags for each
image.

To assess the viability of our approach, following CapsFusion
yu2023capsfusion, we initially leverage ChatGPT to combine information from
raw texts, synthetic captions, and detection tags. However, the time and
computational effort involved is prohibitive. Therefore, we construct an
instruction dataset based on ChatGPT interactions and fine-tuned the open-
source LLaMA3 with this dataset. After that, we leverage the fine-tuned LLaMA3
model for large-scale inference. Specifically, we select 70K image-text pairs
from YFCC15M that have more than 10 detection tags. Then, we input the raw
texts, synthetic captions, and detection tags of these data into ChatGPT to
get instruction responses. The details of the instruction prompt are provided
in the supplementary material.

After obtaining the instruction dataset, we utilize the LLaMA Factory
zheng2024llamafactory to finetune the LLaMA3-8B and leverage vLLM
kwon2023efficient to accelerate large-scale inference.

###  3.2 RWKV-CLIP

In this section, we propose RWKV-CLIP, a robust and efficient RWKV-driven
vision-language representation learner. Inspired by CLIP radford2021learning
and Vision-RWKV duan2024vision, RWKV-CLIP adopts a dual-tower architecture
with a block-stacked encoder design like the Transformer vaswani2017attention,
where each block consists of a spatial mixing and a channel mixing module. The
overview architecture of our proposed RWKV-CLIP is shown in Fig. 3.

Input Augmentation. Based on our proposed diverse description generation
framework, we can obtain three types of text: raw text $T_{r}$, synthetic
caption $T_{s}$, and generated description $T_{g}$. To improve the robustness
of the model, we randomly select a text from $[T_{r},T_{s},T_{g}]$ as the
augmentation for text inputs:

$\mathrm{aug}(T)=\mathrm{Sample}([T_{r},T_{s},T_{g}])$ (1)

Meanwhile, the input image $I\in\mathbb{R}^{H\times W\times 3}$ is transformed
into $HW/p^{2}$ patches, where $p$ is the patch size.

Spatial Mixing. The input text $\text{aug}(T)$ and image $I$ are passed
through the spatial mixing module, which acts as an attention mechanism and
performs global attention computation of linear complexity. Specifically, the
input data is shifted and entered into four parallel linear layers to obtain
multi-head vectors $G_{x}^{s},R_{x}^{s},K_{x}^{s},V_{x}^{s}$:

$\psi_{x}^{s}=\mathrm{Lerp}_{\psi}(x)\cdot
w_{\psi}^{s},\quad\psi\in\\{G,R,K,V\\},$ (2)

where $\rm Lerp$ is the linear interpolation peng2024eagle. In this paper, we
adopt Q-Lerp and B-Lerp for image and text encoders respectively. The Q-Lerp
can be formulated as:

$\displaystyle\text{Q-Lerp}_{\Psi}(I)=I+(1-\eta_{\Psi})\cdot I^{\star},$ (3)
$\displaystyle I^{\star}=\text{Concat}(I_{1},I_{2},I_{3},I_{4}).$

The B-Lerp can be presented as:

$\displaystyle\text{B-Lerp}_{\Psi}(T)=T+(1-\eta_{\Psi})\cdot T^{\star},$ (4)
$\displaystyle T^{\star}=\text{Concat}(T_{1},T_{2}),$

where $\Psi\in\\{G,R,K,V,w\\}$, $\eta_{\Psi}$ denotes learnable vectors,
$I^{\star}$ is the quad-directional shift vector in the image, _i.e._ ,
$I_{1}=x[h-1,w,0:C/4],I_{2}=x[h+1,w,C/4:C/2],I_{3}=x[h,w-1,C/2:3C/4],I_{4}=x[h,w+1,3C/4:C]$,
$T^{\star}$ is the bi-directional shift in the text _i.e._ ,
$T_{1}=[w-1,0:C/2]$, $T_{2}=[w+1,C/2:C]$, where $h,w,C$ present the number of
height, width, and channel. These shift functions enhance feature interaction
at the channel level, enabling a focus on neighboring tokens. Specifically,
the bi-directional shift ensures forward and backward interaction of text
tokens without increasing additional FLOPs. To avoid a fixed learned vector, a
new time-varying decay $w_{x}$ is calculated as follows:

$\displaystyle\phi(x)=\lambda+\tanh(x\cdot M_{i})\cdot M_{j},$ (5)
$\displaystyle\hat{w}_{x}^{s}=x+(1-\phi(\text{Lerp}_{w}(x)))\cdot x^{\star},$
$\displaystyle\tilde{w}_{x}^{s}=\phi(\hat{w}_{x}^{s}),w_{x}^{s}=\exp\left(-\exp(\tilde{w}_{x}^{s})\right),$

where $x\in\\{I,T\\}$, $\lambda$ is a learnable vector, $M_{i},M_{j}$ are
learnable weight matrices. The function $\phi$ is used to obtain learned
vectors by inexpensively augmenting inputs with additional offsets.
$\hat{w}_{x}^{s}$ and $\tilde{w}_{x}^{s}$ are middle values of $w_{x}^{s}$
during the calculation process. This process allows each channel of $w_{x}$ to
vary based on a mix of the current and prior tokens $x^{\star}$.

Subsequently, $w_{x}^{s},R_{x}^{s},K_{x}^{s},V_{x}^{s}$ are used to compute
the global attention result $wkv_{t}$ via a linear complexity bidirectional
attention mechanism. This result is then multiplied by $\sigma(G_{x}^{s})$,
functioning as a gate mechanism to control the output $O_{x}^{s}$:

$\displaystyle wkv_{t}=\text{Bi-
WKV}_{t}(w_{x}^{s},R_{x}^{s},K_{x}^{s},V_{x}^{s}),$ (6) $\displaystyle
O_{x}^{s}=\text{Concat}\left(\sigma(G_{x}^{s})\odot\text{LN}(wkv_{t})\right)\cdot
w_{o}^{s},$

where $\sigma(\cdot)$ denotes the SiLU function elfwing2018sigmoid, and
$\odot$ means element-wise multiplication, $\rm LN$ is the layer norm and the
Bi-WKV duan2024vision; peng2024eagle can be formulated as:

$\displaystyle\text{Bi-WKV}_{t}$
$\displaystyle=R_{s,t}\cdot(\text{diag}(u)\cdot K_{s,t}^{\mathsf{T}}\cdot
V_{s,t}$ (7)
$\displaystyle\quad+\sum_{i=0}^{t-1}\text{diag}(\epsilon_{i,j})\cdot
K_{s,i}^{\mathsf{T}}\cdot V_{s,i}$
$\displaystyle\quad+\sum_{i=t+1}^{T-1}\text{diag}(\epsilon_{i,j})\cdot
K_{s,i}^{\mathsf{T}}\cdot V_{s,i})$

where $u$ is a per-channel learned boost and
$\epsilon_{i,j}=\odot^{i-1}_{j=1}w_{j}$ is a dynamic decay.

Channel Mixing. The spatial mixing module is followed by the channel-mixing
module. Similarly, the $R_{x}^{c},K_{x}^{c}$ are obtained by Lerp:

$\psi_{x}^{c}=\text{Lerp}_{\psi}(x)\cdot w_{\psi}^{c},\quad\psi\in\\{R,K\\}$
(8)

After that, a linear projection and a gate mechanism are performed
respectively and the final output $O_{x}^{c}$ is formulated as:

$O_{x}^{c}=(\sigma(R_{x}^{c})\odot\rho(K_{x}^{c}))\cdot w_{o}^{c}$ (9)

where $\rho$ is the squaredReLU agarap2018deep. After passing through the
stack RWKV-based image and text encoders $E_{I}$ and $E_{T}$, we can get the
image embeddings $\hat{I}=E_{I}(I)$ and text embeddings
$\hat{T}=E_{T}(aug(T))$, the loss function $L$ is defined as:

$\begin{split}L=-\sum_{i=1}^{N}\left[\log\frac{e^{\hat{I}_{i}^{\top}\hat{T}_{i}/\tau}}{\sum_{j}e^{\hat{I}_{i}^{\top}\hat{T}_{j}/\tau}}+\log\frac{e^{\hat{I}_{i}^{\top}\hat{T}_{i}/\tau}}{\sum_{j}e^{\hat{I}_{j}^{\top}\hat{T}_{i}/\tau}}\right]\end{split}$
(10)

##  4 Experiments

###  4.1 Experimental Settings

Pre-training Datasets. We train our model on the YFCC15M dataset, which is a
subset of YFCC100M thomee2016yfcc100m filtered by DeCLIP li2021supervision. To
further verify the effectiveness and generalizability of RWKV-CLIP, following
ALIP alip_Yang_2023_ICCV, we randomly select subsets of 10M and 30M from the
LAION400M schuhmann2021laion. We then conduct a series of experiments with
different model scales and pre-training datasets.

Implementation Details. Consistent with ALIP alip_Yang_2023_ICCV, we employ
OFAbase to generate synthetic captions. The instruction dataset is constructed
using ChatGPT-35-turbo, and we fine-tune LLaMA3-8B to enhance the generation
of diverse descriptions. We employ AdamW loshchilov2017decoupled as the
optimizer, initialized with a learning rate of $1e-3$ and a weight decay of
$0.2$. The parameters $\beta{1}$ and $\beta_{2}$ are set to $0.9$ and $0.98$,
respectively. The input image size is $224\times 224$, and the input text
sequence length is truncated or padded to $77$. The temperature parameter
$\tau$ is initialized to $0.07$. We train RWKV-CLIP for 32 epochs with a batch
size of $4096$ on $8$ NVIDIA A100(80G) GPUs. We meticulously regulate the
parameters and FLOPs of RWKV-CLIP to ensure the fairness of the experimental
comparison. Please refer to the supplementary material for more detailed
parameters, FLOPs, and settings of RWKV-CLIP.

###  4.2 Experimental Results

Linear Probe. Building upon previous works alip_Yang_2023_ICCV;
li2021supervision; geng2023hiclip, we use RWKV-CLIP as a feature extractor and
train only a logistic regression classifier. Tab. LABEL:tab:linear details the
linear probe performance across 10 downstream datasets, as referenced in ALIP
alip_Yang_2023_ICCV. RWKV-CLIP achieves a significant performance improvement
ranging from 1.9% $\sim$ 11.1% over the baseline models, outperforming ALIP in
8 of the 10 datasets. The observed performance improvements are primarily due
to two main factors: (1) Our proposed description generation framework
effectively synthesizes and refines information from web-based texts,
synthetic captions, and detection tags, producing more accurate and semanti-
cally enriched descriptions. (2) RWKV-CLIP exhibits superior representation
learning capabilities compared to ViT-based models.

Zero-shot Image-text Retrieval. In Tab. LABEL:tab:retrieval, we compare our
method with state-of-the-art approaches in zero-shot image-text retrieval on
Flickr30k and MSCOCO. RWKV-CLIP achieves new state-of-the-art results on all
evaluation metrics. Specifically, RWKV-CLIP achieves 76.0%/57.6% I2T/T2I
retrieval Recall@1 on Flickr30K, surpassing ALIP by 5.5%/8.7%. Similarly,
significant improvements of 3.5%/4.7% in I2T/T2I retrieval Recall@1 are
observed for RWKV-CLIP on MSCOCO. This exceptional image-text retrieval
capability indicates that the representations learned by RWKV-CLIP are robust
and exhibit enhanced cross-modal alignment.

Zero-shot Classification. We present the zero-shot classification performance
across 11 datasets. To ensure fair comparisons, we use the same prompt
templates and class names as established in ALIP alip_Yang_2023_ICCV and SLIP
mu2022slip. As shown in Tab. LABEL:tab:transfer, RWKV-CLIP achieves an average
performance improvement of 2.6% $\sim$ 14.4% over baseline models. Notably,
our model outperforms ALIP in 10 out of the 11 datasets, with significant
enhancements on instance discrimination datasets such as Food101, and
ImageNet. This improvement is mainly due to the diverse descriptions generated
by our framework, providing more fine-grained semantic information.

Zero-Shot Robustness Evaluation. In Tab. 1, we present a robustness evaluation
comparing ALIP and RWKV-CLIP. Our results show that RWKV-CLIP consistently
outperforms ALIP in terms of robustness across all datasets with an average
improvement of 2.0%. These experimental results establish the RWKV-driven
model as a robust representation learner.

Method | IN-V2 | IN-A | IN-R | IN-Sketch | Average
---|---|---|---|---|---
ALIP-ViT-B/32 | 34.1 | 16.1 | 35.2 | 12.1 | 24.4
RWKV-CLIP-B/32 | 37.5 | 16.7 | 37.0 | 14.5 | 26.4

Table 1: Zero-shot robustness comparison of ALIP and RWKV-CLIP pretrained on
YFCC15M.

###  4.3 Ablation Study

Effectiveness of Model and Data Scaling. To evaluate the effectiveness of
RWKV-CLIP on model and data scaling, we conduct experiments on randomly
selected subsets of 10M and 30M from LAION400M. For a more comprehensive
comparison, we report the linear probe performance on 26 downstream datasets.
As shown in Fig. 5, RWKV-CLIP significantly improves performance across
different model scales and pre-training datasets. These results demonstrate
the robustness and extensibility of RWKV-CLIP. Detailed experimental results
can be found in the supplementary material.

Comparision Analysis with CapsFusion. To further demonstrate the performance
differences between our proposed diverse description generation framework and
CapsFusion, we used CapsFusion-LLaMA to rewrite the YFCC15M dataset based on
raw texts and synthetic captions. We then trained RWKV-CLIP using texts
generated by our framework and CapsFusion. As shown in Tab. 2, our framework
achieves a 0.9% and 2.1% improvement in the average linear probe and zero-shot
classification performance, respectively. This improvement is primarily due to
the detection tags introducing more semantic information from images, which
further constrains LLMs and reduces hallucinations (as shown in Fig. 4).

Method |  Text Generation Model |  Linear probe Avg |  Zero-shot Avg
---|---|---|---
RWKV-CLIP-B/32 | CapsFusion | 72.6 | 33.1
RWKV-CLIP-B/32 | Ours | 73.5 | 35.2

Table 2: Performance comparison using text generated by our proposed diverse
description generation framework vs. CapsFusion. Figure 4: Comparison of our
proposed diverse description generation framework vs. CapsFusion.
Hallucinations are highlighted in red, and additional semantic information is
highlighted in green.

Figure 5: Linear probe performance comparison between RWKV-CLIP and ALIP on 26
downstream datasets. The comparisons include RWKV-CLIP-B/32 vs. ALIP-ViT-B/32
on LAION10M, RWKV-CLIP-B/16 vs. ALIP-ViT-B/16 on LAION10M, and RWKV-CLIP-B/32
vs. ALIP-ViT-B/32 on LAION30M, presented from left to right.

$T_{r}$ | $T_{s}$ | $T_{g}$ | Dataset |  Linear probe Avg |  Zero-shot Avg
---|---|---|---|---|---
✓ | ✗ | ✗ | YFCC15M | 71.3 | 38.7
✗ | ✓ | ✗ | YFCC15M | 72.4 | 23.1
✗ | ✗ | ✓ | YFCC15M | 73.5 | 35.2
✓ | ✓ | ✗ | YFCC15M | 73.0 | 43.0
✓ | ✗ | ✓ | YFCC15M | 73.8 | 43.4
✓ | ✓ | ✓ | YFCC15M | 74.1 | 44.4

Table 3: Ablation experiment results using different types of text. $T_{r}$:
raw text. $T_{s}$: synthetic caption. $T_{g}$: generated diverse description
using our proposed framework.

Ablation on Different Types of Text. We conduct ablation experiments on
different categories of text, the average linear probe results on 10 datasets
and the average zero-shot classification accuracy on 11 datasets are shown in
Tab. 3. Synthetic captions and generated diverse descriptions yielded superior
linear probe performance compared to raw texts. This improvement is attributed
to the high incidence of mismatched image-text pairs in raw texts, which can
adversely affect representation learning. As shown in Fig. 6, our analysis of
cosine similarity (computed by CLIP-L14) and token counts across different
text types reveals that synthetic captions and generated diverse descriptions
have higher average similarity and token counts than raw texts. Furthermore,
despite these advantages, raw texts achieve superior zero-shot classification
results, mainly due to the constraints imposed by the prompt template.

Figure 6: Statistical analysis of raw texts, synthetic captions, and generated
diverse descriptions on the YFCC15M.

###  4.4 Ablation on Model Architecture

In Tab. 4, base on text augmentation, we perform an ablation study combining
RWKV and Transformer architectures. Compared with TransformerI and
TransformerT, the integration of RWKVI and TransformerT achieves a 2.7%
improvement on linear probe but the zero-shot classification performance
declines by 10.8%. This reduction is primarily due to the poor compatibility
between the RWKV and Transformer architectures. Conversely, the combination of
RWKVI and RWKVT yields improvements of 3.2% and 2.7% in linear probe and zero-
shot classification, respectively, indicating that RWKV outperforms
Transformer in vision-language representation learning.

Image | Text | Linear Probe | Zero-shot
---|---|---|---
RWKVI | TransformerI | RWKVT | TransformerT | Avg | Avg
✗ | ✓ | ✗ | ✓ | 70.9 | 41.7
✓ | ✗ | ✗ | ✓ | 73.6 | 30.9
✗ | ✓ | ✓ | ✗ | 71.0 | 41.1
✓ | ✗ | ✓ | ✗ | 74.1 | 44.4

Table 4: Ablation on model architecture.

Analysis of Feature Embedding. To understand what makes RWKV-CLIP effective,
we randomly select 250 image-text pairs from YFCC15M and visualize the
modality gaps of ALIP and RWKV-CLIP. Specifically, each image and its
corresponding text are encoded into embedding space and reduced to two
dimensions using UMAP mcinnes2018umap. As shown in Fig. 7, we found that the
representations learned by RWKV-CLIP exhibit clearer discriminability within
the same modality. Additionally, compared to ALIP, RWKV-CLIP demonstrates
closer distances in the image-text modality space, indicating superior cross-
modal alignment performance.

Figure 7: Visualization of modality gaps.

##  5 Conclusion

In this paper, we further explore CLIP from the perspectives of data and model
architecture. We introduce a diverse description generation framework that can
leverage Large Language Models (LLMs) to combine and refine information from
web-based image-text pairs, synthetic captions, and detection tags. Besides,
we propose RWKV-CLIP, the first RWKV-driven vision-language representation
learning model that combines the effective parallel training of transformers
with the efficient inference of RNNs. Our method demonstrates superior
performance across various model scales and pre-training datasets on different
downstream tasks. We hope that our work provides insights into vision-language
representation learning models.