# On the Language Encoder of Contrastive Cross-modal Models

Mengjie Zhao† Junya Ono† Zhi Zhong† Chieh-Hsin Lai‡ Yuhta Takida‡ Naoki
Murata‡
Wei-Hsiang Liao‡ Takashi Shibuya‡ Hiromi Wakaki† Yuki Mitsufuji‡†
†Sony Group Corporation ‡Sony AI

###### Abstract

Contrastive cross-modal models such as CLIP and CLAP aid various vision-
language (VL) and audio-language (AL) tasks. However, there has been limited
investigation of and improvement in their language encoder, which is the
central component of encoding natural language descriptions of image/audio
into vector representations. We extensively evaluate how unsupervised and
supervised sentence embedding training affect language encoder quality and
cross-modal task performance. In VL pretraining, we found that sentence
embedding training language encoder quality and aids in cross-modal tasks,
improving contrastive VL models such as CyCLIP. In contrast, AL pretraining
benefits less from sentence embedding training, which may result from the
limited amount of pretraining data. We analyze the representation spaces to
understand the strengths of sentence embedding training, and find that it
improves text-space uniformity, at the cost of decreased cross-modal
alignment.

##  1 Introduction

Significant progress have been made in pretraining large-scale cross-modal
models, such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), for
various vision-language (VL) applications such as retrieval and zero-shot
image classification. These models are often pretrained with large amounts of
data, e.g., OpenAI leverages $\approx$400M caption-image pairs to train CLIP.
The amount of data has been scaled up to 5B with LAION-5B (Schuhmann et al.,
2022; Cherti et al., 2023). Such a large amount of multimodal pretraining data
contains text captions at the same scale as the pretraining corpora of large
language models (LLMs) such as BERT, which is pretrained on 3.3B words (Devlin
et al., 2019).

The success of VL pretraining encourages research on contrastive learning
models for other modalities like audio. Pretrained audio-language (AL) models
such as AudioCLIP (Guzhov et al., 2022) and CLAP (Wu et al., 2023; Elizalde et
al., 2023) show promising results on AL retrieval and zero-shot audio
classification tasks.

It is clear that the language encoder in cross-modal contrastive models plays
a central role when scaling-up pretraining of a specific modality and/or the
amount of modalities. Therefore, analyzing and improving the language encoder
become increasingly crucial. CLIP’s language encoder (CLIP LM) – a decoder-
only language model similar to GPT-2 (Radford et al., 2019) – has been
investigated. Yan et al. (2022) showed that the CLIP LM outperforms BERT
(Devlin et al., 2019) in clustering entities with prompting. Wolfe and
Caliskan (2022) probed the CLIP LM, showing that word representations from it
are less anisotropic (Ethayarajh, 2019), i.e., more uniformly distributed with
respect to direction, than the similar-sized GPT-2. Complementary to research
on CLIP LM, we focus on pretraining. CLIP-like models are often pretrained
with cross-modal contrastive learning. _We measure – during pretraining – the
effectiveness of systematically modeling the image captions with sentence
embedding training_ (Reimers and Gurevych, 2019; Gao et al., 2021), which is a
natural fit to the captions.

We pretrain CLIP and one of its new variants CyCLIP (Goel et al., 2022) with
sentence embedding training, as well as the conventional cross-modal
contrastive learning. In addition to CLIP’s cross-modal contrastive learning
objective, CyCLIP explicitly optimizes for geometry consistency between the
text and image representation spaces, making it a suitable model for
validating the effectiveness of NLP methods. We evaluate pretrained models on
an array of tasks involving one or two modalities such as SentEval, zero-shot
VL retrieval, and image classification. We find that unsupervised sentence
embedding training improves the language encoder quality and VL tasks.
Supervised sentence embedding training improves language encoder quality, but
the benefit does not necessarily transfer to VL tasks. We analyze the learned
representation spaces and find that sentence embedding training improves text-
space uniformity (Wang and Isola, 2020) and reduces anisotropy (Ethayarajh,
2019; Wolfe and Caliskan, 2022).

We also investigate AL contrastive models such as CLAP (Wu et al., 2023). In
contrast to VL pretraining, AL pretraining suffers from data scarcity and
often leverages pretrained LLMs and audio encoders. We determine the
effectiveness of sentence embedding training in both scenarios: continued
pretraining with LLMs and audio encoders, and pretraining from scratch. We
find that the benefits of sentence embedding training are less noticeable and
noisy. To the best of our knowledge, this is the first study on investigating
and trying to improve the language encoder of AL contrastive learning, and we
expect our results will encourage more research in this direction.

In summary, our contributions are as follows: (i) We extensively evaluate how
unsupervised and supervised sentence embedding trainings affect VL and AL
contrastive pretraining. Experimental results indicate improved VL
performance, however, the results on AL tasks are noisy and the improvements
are less noticeable. (ii) We show that unsupervised sentence embedding
training improves the language encoder of CyCLIP (Goel et al., 2022), hence
improves performance of cross-modal tasks. (iii) We conduct a comprehensive
analysis on the alignment and uniformity of learned representation spaces
following Wang and Isola (2020), and show that sentence embedding training
improves uniformity of the text representation space, but at the cost of
decreased cross-modal alignment.

##  2 Related work

CLIP LM. Research has focused on the language encoder of OpenAI CLIP (Radford
et al., 2021). The model consists of a language encoder (CLIP LM) and an image
encoder that are jointly trained on Web-scale caption-image pairs. Yan et al.
(2022) stressed the importance of CLIP LM, showing that it outperforms BERT
(Devlin et al., 2019) in tasks, such as entity clustering, through prompting.
Bielawski et al. (2022) showed that CLIP LM outperforms BERT in “human-
centric” tasks such as genre classification on books or movies. Santurkar et
al. (2023) highlighted the importance of text captions for representation
learning of CLIP by comparing it with SimCLR (Chen et al., 2020) in which no
language supervisions is present. Training signals from language are shown to
be detrimental, worthing any number of images in a sufficiently large dataset.
Our work follows this direction, with a focus on determining how supervised or
unsupervised sentence embedding trainings affect CLIP LM and VL contrastive
learning.

Goel et al. (2022) introduced CyCLIP, incorporating extra training objectives
than cross-modal contrastive learning such that the geometry consistency
between the text and image spaces is improved. One of CyCLIP’s training
objectives is computing similarities between captions; this motivates us to
determine how systematically modeling the captions through supervised or
unsupervised sentence embedding training affects CyCLIP/CLIP.

Contrastive audio-language pretraining models have also been proposed. Guzhov
et al. (2022) extended CLIP to audio tasks by adding an extra module and
continued training on audio datasets. Similar distillation methods such as
Wav2CLIP (Wu et al., 2022), have also been proposed. Elizalde et al. (2023)
and Wu et al. (2023) independently proposed CLAP, in which a language encoder
and an audio encoder are jointly trained on AL datasets, which resembles CLIP.
We focused on the language encoder in pretraining AL models, and demonstrated
the impact of sentence embedding training. To the best of our knowledge, this
is the first step in this direction.

Sentence embedding is an extensively investigated NLP topic. Methods ranging
from bag-of-word averaging non-contextualized embeddings (Mikolov et al.,
2013; Pennington et al., 2014) to training LSTMs (Hochreiter and Schmidhuber,
1997), e.g., SkipThought (Kiros et al., 2015) and InferSent (Conneau et al.,
2017), have been proposed to effectively compose individual tokens to
meaningful sentence representations. Methods that leverage post-hoc
transforming (Li et al., 2020; Su et al., 2021) or finetuning the pretrained
BERT in supervised (Reimers and Gurevych, 2019) or unsupervised scenarios (Gao
et al., 2021) are also introduced. Zhang et al. (2022) show that grounding
sentence embedding learning to images improves semantic textual similarity
tasks. We present a focused investigation on the LM in CLIP/CyCLIP. We
pretrained from scratch an LM and ResNet-50 (He et al., 2016) with cross-modal
contrastive learning, as well as unsupervised or supervised sentence embedding
training with image captions. Verifying the effectiveness of sentence
embeddings – a critical component for retrieval and clustering (Reimers and
Gurevych, 2019; Gao et al., 2021; Wang et al., 2021; Thakur et al., 2021;
Geigle et al., 2022) – is of great importance because retrieval has been one
of the main applications of CLIP-like models.

##  3 Method

Cross-modal contrastive learning plays a key role in training models such as
CLIP/CLAP. We take image modality as an example for introducing this method.
Consider a caption-image dataset $\\{(I_{i},T_{i})\\}_{i=1}^{N}$ that includes
$N$ caption-image pairs, and denote $I^{e}$ and $T^{e}$ as the output
representations from an image and langauge encoder, respectively.

The cross-modal contrastive loss (Radford et al., 2021) is defined as

$\displaystyle\mathcal{L_{\textnormal{contra.}}(\tau)}=-\sum_{j=1}^{N}\log$
$\displaystyle\frac{\exp\left({\langle}I^{e}_{j},T^{e}_{j}{\rangle}/\tau\right)}{\sum_{k}{\exp\left({\langle}I^{e}_{j},T^{e}_{k}{\rangle}/\tau\right)}}$
$\displaystyle\quad-\sum_{k=1}^{N}\log\frac{\exp\left({\langle}I^{e}_{k},T^{e}_{k}{\rangle}/\tau\right)}{\sum_{j}{\exp\left({\langle}I^{e}_{j},T^{e}_{k}{\rangle}/\tau\right)}},$

where $\tau$ is a trainable temperature parameter initialized to 0.07 and
$\langle\cdot,\cdot\rangle$ computes cosine similarity.

CLIP’s training objective solely stresses the alignment between the two
modalities. CyCLIP (Goel et al., 2022) has improved CLIP (Radford et al.,
2021) by additionally optimizing for improved representation space geometry,
such that the image and text spaces are more consistent with each other.
Concretely, CyCLIP explicitly optimizes two additional objectives for cross-
modal and in-modal consistency as well as
$\mathcal{L_{\textnormal{contra.}}}$:

$\displaystyle\mathcal{L_{\textnormal{C-cyclic}}}=\sum_{j}\sum_{k}\left({\langle}I^{e}_{j},T^{e}_{k}{\rangle}-{\langle}I^{e}_{k},T^{e}_{j}{\rangle}\right)^{2},$
$\displaystyle\mathcal{L_{\textnormal{I-cyclic}}}=\sum_{j}\sum_{k}\left({\langle}I^{e}_{j},I^{e}_{k}{\rangle}-{\langle}T^{e}_{k},T^{e}_{j}{\rangle}\right)^{2}.$

Intuitively, decreasing the cross-modal consistency loss
$\mathcal{L}_{\textnormal{C-cyclic}}$ makes the cross-modal similarity matrix
more symmetric, as shown in Figure 1. Note that solely optimizing
$\mathcal{L_{\textnormal{contra.}}}$ is expected to symmetrize the cross-modal
similarity matrix because the similarity of non-diagonal pairs are trained to
be zero. Goel et al. (2022) showed that this scenario does not occur in
practice and explicitly optimizing $L_{\textnormal{C-cyclic}}$ is beneficial.

Figure 1:  Cross-modal similarity matrix. Diagonal elements refer to cosine
similarity between aligned caption-image pairs while non-diagonal elements
refer to mismatched caption-image pairs.

Optimizing $\mathcal{L}_{\textnormal{I-cyclic}}$, however, reduces the
inconsistency between the overall geometry of the text and image spaces.
Computing $\mathcal{L}_{\textnormal{I-cyclic}}$ requires calculating the
similarity between two text captions, i.e.,
${\langle}T^{e}_{k},T^{e}_{j}{\rangle}$. It is thus reasonable to hypothesize
that _accurately computing caption similarities is beneficial for optimizing_
$\mathcal{L}_{\textnormal{I-cyclic}}$. We use unsupervised and supervised
sentence embedding training methods and test the hypothesis (§4).

We use the widely used SimCSE (Gao et al., 2021) method for learning caption
representations. SimCSE also uses contrastive learning: a caption is input to
a language encoder twice to obtain two vectors $T^{e}$ and $T^{e}_{+}$. With
dropout enabled, $T^{e}$ and $T^{e}_{+}$ are generally different. These paired
vectors serve as the positive training pairs for contrastive learning, while
mismatched captions form negative pairs. We denote the unsupervised SimCSE
loss as

$\displaystyle\mathcal{L_{\textnormal{s}}(\tau)}=-\sum_{j}\log$
$\displaystyle\frac{\exp\left({\langle}T^{e}_{j},T^{e}_{j,+}{\rangle}/\tau\right)}{\sum_{k}{\exp\left({\langle}T^{e}_{j},T^{e}_{k,+}{\rangle}/\tau\right)}},$

where $\tau$ is fixed to 0.05 following (Gao et al., 2021).

Another direction of sentence embedding training is supervised training
(Reimers and Gurevych, 2019) on natural language inference (NLI) datasets,
e.g., SNLI and MNLI (Bowman et al., 2015; Williams et al., 2018). We denote
the objective as $\mathcal{L_{\textnormal{n}}}$ and follow Gao et al. (2021)
in using entailment pairs as $T^{e}$ and $T^{e}_{+}$ and the contradiction
sentence as a hard negative.

Table 1 lists various models we use in our experiments as well as their
training objectives. During the experiments, we sum up the objectives but
weight them with different hyperparameters ($\lambda$) depending on the
combinations, which are shown in §4. We add a suffix “s” to the name of models
trained with $\mathcal{L_{\textnormal{s}}}$ and “n” to models trained with
$\mathcal{L_{\textnormal{n}}}$.

| $\mathcal{L_{\textnormal{contra.}}}$ | $\mathcal{L_{\textnormal{C-cyclic}}}$ | $\mathcal{L_{\textnormal{I-cyclic}}}$ | $\mathcal{L_{\textnormal{s}}}$ | $\mathcal{L_{\textnormal{n}}}$
---|---|---|---|---|---
CLI (A) P | ✓ | - | - | - | -
CLI (A) Ps | ✓ | - | - | ✓ | -
CLI (A) Pn | ✓ | - | - | - | ✓
CyCLI (A) P | ✓ | ✓ | ✓ | - | -
CyCLI (A) Ps | ✓ | ✓ | ✓ | ✓ | -
CyCLI (A) Pn | ✓ | ✓ | ✓ | - | (✓)
Table 1: List of training objectives. We follow Radford et al. (2021) for
CLIP, Wu et al. (2023) for CLAP, and Goel et al. (2022) for CyCLIP.

##  4 Experiments

###  4.1 Datasets

To _pretrain VL models_ such as CLIP and CyCLIP, we follow Bugliarello et al.
(2021); Goel et al. (2022) and use the Conceptual Captions dataset, which
consists of approximately111 CC3M images need to be downloaded by users. Due
to broken URLs, the exact amount of data varies from time to time; Table 2
shows the exact number of images. 3M caption-image pairs (CC3M; Sharma et al.
(2018)). CC3M has a reasonable size for pretraining and contains a broad
coverage of Web content, making it a good option for learning generic VL
representations (Bugliarello et al., 2021).

To _evaluate the trained VL models_ , we follow (Radford et al., 2021) and
conduct evaluations with zero-shot image-text retrieval on the Karpathy
(Karpathy and Fei-Fei, 2015) test splits of Flickr30K (Plummer et al., 2015)
and MSCOCO (Chen et al., 2015). We skip the evaluation on Flickr30K when
supervised sentence embedding training is used, i.e., when
$\mathcal{L_{\textnormal{n}}}$ is considered in training. This is because
Flickr30K captions are the premises in the SNLI dataset (Bowman et al., 2015),
overlapping with the supervised sentence embedding training data. For zero-
shot image classification, we use the standard benchmarks CIFAR10, CIFAR100
(Krizhevsky et al., 2009), and ImageNet1K (Russakovsky et al., 2015). Zero-
shot image classification with domain shift, out-of-domain, and adversarial
examples are also considered: ImageNetV2 (Recht et al., 2019), ImageNet-Sketch
(Wang et al., 2019), ImageNet-O, ImageNet-A, and ImageNet-R (Hendrycks et al.,
2021b, a).

To _pretrain AL models_ , e.g., CLAP and CyCLAP, we conduct experiments with
Clotho (Drossos et al., 2020) consisting of $\approx$6K caption-audio pairs
and AudioCaps consisting of $\approx$50K222 Similar to CC3M, AudioCaps only
provides audio captions while users need to download corresponding YouTube
videos and convert their audio to waveforms; Table 2 shows the exact amount of
waveforms we used.  caption-audio pairs (Kim et al., 2019), similar to LAION-
CLAP (Wu et al., 2023). In contrast to the VL scenario, AL pretraining is
known to be challenging due to data scarcity (Wu et al., 2023). We use Clotho
and AudioCaps because they contain natural language descriptions of the audio
instead of simple tags; thus, there is no need to generate pseudo-natural
language captions from tags using services such as ChatGPT/GPT3 (Mei et al.,
2023).

To _evaluate pretrained AL models_ , we conduct cross-modal retrieval and
zero-shot audio classification tasks. For Clotho, we train the models on the
training split and report retrieval results on the validation split. For
AudioCaps, we select the best-performing checkpoint on the validation split
and report test split results. We conduct zero-shot classification on the
Environmental Sound Classification dataset (ESC50; Piczak (2015)) and
UrbanSound8K (US8K; Salamon et al. (2014)), which have been widely used (Wu et
al., 2023; Elizalde et al., 2023). ESC50 contains short audio clips containing
the sound of different common events such as cats meowing and dogs barking;
the clips are categorized into 50 classes, and US8K contains audio clips of
urban event sounds such as drilling and street music; the clips are
categorized into ten classes.

| Dataset | Pretraining | Retrieval | ZS | Size
---|---|---|---|---|---
| CC3M | ✓ | - | - | 2,806,641
| MSCOCO | - | ✓ | - | 5,000
| Flickr30K | - | ✓ | - | 1,000
| CIFAR10 | - | - | ✓ | 10,000
| CIFAR100 | - | - | ✓ | 10,000
VL | ImageNet1K | - | - | ✓ | 50,000
| ImageNetV2 | - | - | ✓ | 10,000
| ImageNetSketch | - | - | ✓ | 5,0889
| ImageNet-O | - | - | ✓ | 2,000
| ImageNet-A | - | - | ✓ | 7,500
| ImageNet-R | - | - | ✓ | 30,000
AL | Clotho | ✓ | ✓ | - | 5,929
AudioCaps | ✓ | ✓ | - | 50,725
ESC50 | - | - | ✓ | 400
US8K | - | - | ✓ | 8,732
Table 2: Datasets and their amount of examples. We report amount of images for
VL datasets and of waveform files for AL datasets. “ZS”: zero-shot
classification.

Table 2 lists all the cross-modal datasets and their usage. We follow Goel et
al. (2022) in processing the VL datasets and Wu et al. (2023) in processing
the AL datasets; the detailed steps of these processes are shown in Appendix
§A.1.

###  4.2 Experiment settings

To _pretrain the VL models_ , we use the same model architecture as CyCLIP
(Goel et al., 2022), i.e., a ResNet-50 as the image encoder and Transformer
(Vaswani et al., 2017) as the language encoder. We pretrain the model from
scratch and largely reuse CyCLIP’s hyperparameters. To weight different
training objectives (Table 1) in CyCLIP, we set $\lambda$I-cyclic and
$\lambda$C-cyclic to 0.25, $\lambda$contra. is set to 1.0, and we empirically
set $\lambda$s and $\lambda$n to 0.1. We use a batch size of 80, and each
pretraining trial is run for 64 epochs, taking four days with four A100 GPUs.
We enable dropout in the language encoder and use dropout rate of 0.1.
Appendix §A.1 lists the details of the hyperparameters.

The VL pretraining dataset CC3M has a validation split consisting of
$\approx$15K text-image pairs. We select the best-performing checkpoint on
this validation split for downstream task evaluations.

Due to data scarcity, _AL pretraining_ often leverages pretrained language and
audio encoders. We use the same model architecture as LAION-CLAP (Wu et al.,
2023): pretrained RoBERTa-base (Liu et al., 2020) as the language encoder and
pretrained Hierarchical Token-Semantic Audio Transformer (HTSAT; Chen et al.
(2022)) as the audio encoder. HTSAT has shown to outperform CNNs in various
audio tasks (Chen et al., 2022). We use the HTSAT-tiny variant with 31M
parameters. Due to small dataset size, each pretraining experiment takes less
than one day on a single A100 GPU. We use the default dropout rate of 0.1 of
RoBERTa-base in our experiments. In pilot experiments on Clotho, we found that
the default hyperparameters333 https://github.com/LAION-
AI/CLAP/blob/main/experiment_scripts/train-only-clotho.sh lead to
underfitting444This is reflected by the increasing validation performance in
learning curves publicly shared by the LAION-CLAP project on WnB. . We find
that longer training and a smaller learning rate improve performance, as shown
in Table 4. Table 10 lists our hyperparameters, which are kept the same when
we pretrain the AL models on Clotho or AudioCaps.

For weighting the training objectives, due to the small model size and dataset
size, we grid search the optimal $\lambda$I-cyclic, $\lambda$C-cyclic, and
$\lambda_{\textnormal{s}}$ from [0.1, 0.25, 0.5]; $\lambda$contra. is set to
1.0. Supervised sentence embedding training ($\lambda$n) is not considered due
to the small size of the AL datasets.

Clotho and AudioCaps both contain validation splits; thus, we select the best-
performing checkpoint on the validation splits then conduct evaluation on
downstream tasks.

##  5 Results and analyses

###  5.1 Results

| Text Retrieval |  | Image Retrieval
---|---|---|---
| R@1 | R@5 | R@10 |  | R@1 | R@5 | R@10
CLIP | 15.70 | 37.22 | 49.06 |  | 12.48 | 31.10 | 42.23
CLIPs | 17.78 | 38.92 | 50.10 |  | 13.46 | 32.93 | 44.09
CLIPn | 15.74 | 35.66 | 47.38 |  | 13.12 | 31.46 | 42.55
CyCLIP | 18.92 | 41.46 | 54.00 |  | 15.40 | 35.61 | 46.95
CyCLIPs | 21.30 | 44.34 | 56.54 |  | 16.69 | 37.75 | 49.24
CyCLIPn | 16.32 | 36.76 | 48.16 |  | 14.53 | 34.07 | 45.52

| Text Retrieval |  | Image Retrieval
---|---|---|---
| R@1 | R@5 | R@10 |  | R@1 | R@5 | R@10
CLIP | 31.80 | 62.10 | 72.90 |  | 25.50 | 52.28 | 64.34
CLIPs | 35.20 | 63.20 | 75.30 |  | 26.70 | 52.34 | 64.32
CyCLIP | 37.30 | 66.10 | 76.40 |  | 30.22 | 56.70 | 67.40
CyCLIPs | 40.00 | 69.30 | 79.70 |  | 31.74 | 58.02 | 69.46

Table 3:  Zero-shot VL retrieval results (%) on MSCOCO (top) and Flickr30K (bottom). Integrating unsupervised sentence embedding learning (CLIPs and CyCLIPs) noticeably improves zero-shot retrieval; supervised embedding training (CLIPn and CyCLIPn) has neutral to negative impacts.  mAP@10 | Text Retrieval | Audio Retrieval
---|---|---
CLAP | 12.24 | 21.21
CLAP (Wu et al., 2023) | 13.80 | 20.40
CLAP | 51.34 | 55.60
CLAP (Wu et al., 2023) | 45.70 | 51.30
Table 4:  AL training validation performance (%). We report mean average
precision at ten (mAP@10) on validation split of Clotho (top) and AudioCaps
(bottom).

Table 3 lists the zero-shot VL retrieval results on MSCOCO and Flickr30K,
following the settings in Radford et al. (2021). We first utilize the
pretrained image and language encoders in a VL model to encode images and
captions into vectors. In text retrieval, we then input the image vector to
retrieve the aligned captions and vice-versa for image retrieval. We report
Recall@N for N in [1, 5, 10].

Comparing CyCLIP and CLIP variants. We see that CyCLIP clearly outperforms
CLIP across the board for both datasets, highlighting the significant value of
incorporating $\mathcal{L_{\textnormal{C-cyclic}}}$ and
$\mathcal{L_{\textnormal{I-cyclic}}}$ in optimizing for consistent geometry of
the text and image representation spaces (Goel et al., 2022).

When comparing CyCLIPs/CLIPs to CyCLIP/CLIP, we observe the effectiveness of
improving the language encoder with unsupervised sentence embedding training
$\mathcal{L_{\textnormal{s}}}$. CyCLIPs/CLIPs clearly surpass CyCLIP/CLIP in
all configurations except Flickr30K-CLIPs-ImageRetrieval-R@10. This suggests
that, _we improve CyCLIP on zero-shot vision-language retrieval tasks through
learning better representations of the captions_. We also observe more gains
in text retrieval than in image retrieval. For example, on Flickr30K@1,
CyCLIPs outperforms CyCLIP by 2.70% (absolute) in text retrieval and by 1.52%
(absolute) in image retrieval. This observation reflects the effectiveness of
improved caption representations.

In contrast to the unsupervised embedding training scenario
($\mathcal{L_{\textnormal{s}}}$ and CyCLIPs/CLIPs), supervised sentence
embedding training ($\mathcal{L_{\textnormal{n}}}$ and CyCLIPn/CLIPn) results
in a neutral to negative impact on the overall retrieval results.
Investigating the text and image representation spaces, we find that
$\mathcal{L_{\textnormal{n}}}$ extensively enforces a uniform text
representation space such that the alignment between the text and image spaces
is negatively affected; we provide more in-depth analyses in §5.2.

| Text Retrieval |  | Audio Retrieval
---|---|---|---
| R@1 | R@5 | R@10 |  | R@1 | R@5 | R@10
CLAP | 13.88 | 34.16 | 48.90 |  | 11.67 | 33.80 | 47.10
CLAPs | 13.49 | 35.60 | 49.00 |  | 11.92 | 32.54 | 45.47
CyCLAP | 14.74 | 35.50 | 48.52 |  | 11.90 | 34.95 | 48.61
CyCLAPs | 14.93 | 36.84 | 51.00 |  | 12.08 | 34.09 | 46.76

| Text Retrieval |  | Audio Retrieval
---|---|---|---
| R@1 | R@5 | R@10 |  | R@1 | R@5 | R@10
CLAP | 44.10 | 76.80 | 87.67 |  | 34.82 | 70.62 | 82.93
CLAPs | 42.73 | 75.44 | 87.57 |  | 34.69 | 69.80 | 82.99
CyCLAP | 40.65 | 74.19 | 86.52 |  | 34.13 | 69.24 | 82.30
CyCLAPs | 39.81 | 74.40 | 85.79 |  | 34.23 | 70.24 | 82.74

Table 5:  Text-audio retrieval results (%) on Clotho (top) and AudioCaps
(bottom). Adding unsupervised sentence embedding training improves performance
of CLAP/CyCLAP in general, but improvements are noisy and less noticeable than
in VL scenario.

AL retrieval results on Clotho and AudioCaps are listed in Table 5. Supervised
sentence embedding training objective $\mathcal{L_{\textnormal{n}}}$ is not
considered because NLI datasets are much larger than AL datasets (MNLI: 433K;
Clotho: 6K; AudioCaps: 53K); subsampling introduces extra random factors that
are difficult to control.

On Clotho, we observe overall improvements when comparing CyCLAP to CLAP. We
see one exception on text-retrieval-R@10 ( i.e., 48.90 for CLAP and 48.52 for
CyCLAP); however, the overall improvement outweighs performance drops,
demonstrating that explicitly optimizing for the consistency between the audio
and text spaces (Goel et al., 2022) is also promising for improving AL
retrieval on Clotho. When comparing CyCLAPs/CLAPs with CyCLAP/CLAP, we see
clear improvements on text retrieval. However, this comes with a decreased
performance on audio retrieval results.

On AudioCaps, CyCLAP falls behind CLAP. This is somewhat surprising because
AudioCaps is considered an easier dataset than Clotho (Koepke et al., 2022).
The HTSAT audio encoder has already been pretrained with audio classification
tasks on AudioSet (Gemmeke et al., 2017), from which AudioCaps is derived.
This may contribute to the noisy results. For example, LAION-CLAP (Wu et al.,
2023) reported that adding additional 630K AL pairs largely boosts AL
retrieval performance on Clotho, but hurts on AudioCaps. We observe similar
results when comparing CyCLAPs/CLAPs with CyCLAP/CLAP.

Comparing VL and AL retrieval results in Table 3 and Table 5, we observe that
(1) CyCLIP noticeably improves over CLIP than CyCLAP over CLAP, and CLAP even
outperforms CyCLAP on AudioCaps; and (2) improving the language encoder with
sentence embedding training is more beneficial to VL than AL. We hypothesize
that this is because AL pretraining starts with pretrained encoders, which
have geometry that is difficult to alter due to the small AL dataset size. We
further conduct AL pretraining from scratch (Appendix §A.4), where the
language and audio encoders are re-initialized. Sentence embedding training
brings more noticeable results, however, the the overall results are still
noisy due to the particularly small dataset sizes. We believe resolving the
data scarcity issue is a critical step towards clearer regularities in AL
pretraining. Researchers have started to exploit directions like using ChatGPT
(Mei et al., 2023).

| CLIP | CLIPn | CLIPs | CyCLIP | CyCLIPn | CyCLIPs
---|---|---|---|---|---|---
CIFAR10 | 28.31 | 44.06 | 36.80 | 38.67 | 41.16 | 44.97
CIFAR100 | 13.23 | 17.93 | 10.72 | 17.44 | 19.82 | 22.05
ImageNet1K | 14.94 | 15.97 | 16.01 | 20.99 | 18.13 | 22.13
ImageNetV2 | 12.85 | 13.41 | 14.09 | 17.77 | 15.65 | 18.68
ImageNet-Sk. | 7.72 | 7.75 | 8.14 | 11.67 | 9.93 | 12.85
ImageNet-O | 20.75 | 21.95 | 21.30 | 27.05 | 24.45 | 29.55
ImageNet-A | 3.59 | 3.41 | 3.95 | 5.03 | 4.45 | 5.19
ImageNet-R | 18.39 | 18.51 | 18.24 | 24.37 | 23.07 | 26.72
Table 6:  Zero-shot image classification (R@1 in %) on standard datasets (top)
and datasets with distribution shift or adversarial examples (bottom).

For zero-shot image classification, Table 6 lists the Top1 accuracy on
standard image classification datasets (top) and datasets with distribution
shifts or adversarial examples (bottom). We follow Radford et al. (2021) and
use their prompts for zero-shot classification. For an image to be classified,
we compute the cosine similarity between its vector and the encoded vector of
all classes. Each of the classes is reformulated with various prompts. E.g.,
the ImageNet class “plane” is reformulated with 80 templates 555OpenAI
templates are public on GitHub link. such as “a photo of a ” and “a blurry
photo of a ”, resulting in prompts “a photo of a plane” and “a blurry photo of
a plane” (Radford et al., 2021). The vectors of encoded prompts of a class are
averaged; we select the class with the maximum cosine similarity with the
image vector.

Similar trends as in the retrieval tasks are observed. CyCLIP variants
outperform their CLIP counterparts; unsupervised sentence embedding training
benefits both CyCLIP/CLIP while supervised sentence embedding training does
not result in consistent improvement or deterioration.

For zero-shot audio classification, Table 7 list the Top1 accuracy on ESC50
and US8K of models pretrained on Clotho and AudioCaps respectively. We follow
LAION-CLAP (Wu et al., 2023) and use their template “This is a sound of ”. In
contrast to the image scenario, only one template is provided. CyCLAP
generally outperforms CLAP, except for AudioCaps-US8K. We observe no
improvements of sentence embedding training when comparing CyCLAPs/CLAPs with
CyCLAP/CLAP. We leave the extensive “prompt engineering” of designing more
prompts for future work666 Following the VL scenario, we write prompts and
conduct more experiments on zero-shot audio classification. Appendix §A.3
shows results and discussions. We see improved performance, however, the
generalization ability of this prompt engineering needs to be tested, and we
leave it for future work. .

|  | CLAP | CLAPs | CyCLAP | CyCLAPs
---|---|---|---|---|---
Clotho | ESC50 | 72.25 | 71.75 | 78.00 | 75.75
US8K | 67.76 | 67.42 | 70.46 | 65.98
AudioCaps | ESC50 | 74.50 | 72.75 | 77.50 | 75.50
US8K | 69.78 | 64.14 | 67.90 | 66.94
Table 7:  Zero-shot audio classification (R@1 in %) on ESC50 and US8K of
models pretrained on Clotho or AudioCaps.

###  5.2 Analyses

Alignment and uniformity of representation spaces. In this section, we take a
closer look at the learned representation spaces. Following Wang and Isola
(2020), we inspect the alignment and uniformity on the hypersphere of learned
spaces. Considering a caption-image dataset $\\{(I_{i},T_{i})\\}_{i=1}^{N}$,
we can compute the alignment and uniformity scores defined as:

$\begin{gathered}\mathcal{L}_{\text{align}}\triangleq\mathop{\mathbb{E}}_{{{(I,T)}}~{}\sim~{}p_{\text{pos}}}\left\|I^{e}-T^{e}\right\|^{2}_{2}\\\
\mathcal{L}_{\text{T,uniform}}\triangleq\log\mathop{\mathbb{E}}_{{{T_{i},T_{j}}}\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}p_{\text{data}}}e^{-2\left\|T_{i}^{e}-T_{j}^{e}\right\|^{2}_{2}}\\\
\mathcal{L}_{\text{I,uniform}}\triangleq\log\mathop{\mathbb{E}}_{{{I_{i},I_{j}}}\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}p_{\text{data}}}e^{-2\left\|I_{i}^{e}-I_{j}^{e}\right\|^{2}_{2}}\end{gathered}$

where $(I,T)\sim p_{\text{pos}}$ refers to aligned text-images pairs,
$(T_{i},T_{j})\sim p_{\text{data}}$ refers to independent and identically
distributed (IID) sampled text pairs, $(I_{i},I_{j})\sim p_{\text{data}}$
refers to IID sampled image pairs, and $I^{e}$, $T^{e}$ respectively refer to
the encoded image and text vectors.

Recall that the trained models output L2 normalized vectors residing on the
unit ball. Intuitively, we want vectors of aligned pairs of two modalities to
be well aligned in the representation space, such that
$\mathcal{L}_{\text{align}}$ is close to zero. However, we want the space of a
single modality to be more uniform than anisotropic (Ethayarajh, 2019; Wolfe
and Caliskan, 2022), such that the overall representation space capacity is
well used. This results in a near minus infinite
$\mathcal{L}_{\text{uniform}}$.

(a)

(b)

(c)

(d)

Figure 2:  Visualizing cross-modal alignment w.r.t. text space uniformity of
trained VL and AL models. To visualize AL results, we use models pretrained on
AudioCaps due to it being larger than Clotho. We observe that sentence
embedding training trades cross-modal alignment for text space uniformity.

Figure 2 illustrates the results. We only show $\mathcal{L}_{\text{align}}$
w.r.t. $\mathcal{L}_{\text{T,uniform}}$ since our main focus is the language
encoder. We see that unsupervised sentence embedding training trades cross-
modal alignment for improving the text space uniformity. For VL pretraining,
supervised sentence embedding training overly focuses on text space uniformity
while the VL space alignment deteriorates. This observation is further
evidenced when visualizing with CyCLIP and CyCLIPn using the Flickr30K
($\varstar$) dataset in which the captions largely overlap with the dataset
for supervised sentence embedding training (§4.1).

Another interesting observation is that the text encoder of CyCLIP/CyCLAP
outputs representation space less uniform than that of CLIP/CLAP, as shown in
Figure 2 (x-axis). Liang et al. (2022) show that randomly initialized encoders
output vectors residing in different cones. The in-modal cyclic loss
$\mathcal{L_{\textnormal{I-cyclic}}}$ (§3) stresses consistency between cones;
it is thus expected to be challenging to learn a uniform space while
simultaneously preserving consistency between two spaces777 We find that
improving text space uniformity also benefits the image space for CyCLIP. More
discussions are presented in Appendix §A.2. . Sentence embedding training
provides extra training signals.

LM quality. We evaluate the language encoder quality of pretrained VL models.
Our motivation is two-fold. First, as a sanity check, we want to verify that
incorporating sentence embedding training in VL contrastive learning still
improves the language encoder’s ability of representing general sentences.
Second, the evaluation results help us measure the compatibility and possible
interferences among the various training objectives (Pfeiffer et al., 2023).

To this goal, we leverage the sentence embedding benchmark SentEval (Conneau
and Kiela, 2018). Default SentEval configurations are used in all experiments.
Our intrinsic evaluation tasks are the semantic textual similarity tasks:
STS12-STS16, STS-B, SICKR (Marelli et al., 2014; Cer et al., 2017; Agirre et
al., 2012, 2013, 2014, 2015, 2016). Extrinsic evaluation tasks are movie
review (MR; Pang and Lee (2005)) product review (CR; Hu and Liu (2004))
subjectivity status (SUBJ; Pang and Lee (2004)), opinion polarity (MPQA; Wiebe
et al. (2005)), sentiment analysis on SST2 (Socher et al., 2013), question-
type classification (TREC; Voorhees and Tice (2000)), and paraphrase detection
(MRPC; Dolan et al. (2004)). Table 8 lists the results. We observe that
unsupervised sentence embedding training is beneficial generally for
CyCLIP/CLIP on both intrinsic and extrinsic tasks. Supervised sentence
embedding training results in more significant improvements888 Improvements of
supervised sentence embedding training may due to the observation that NLI
datasets have similar domains and language use as SentEval tasks. In Appendix
§A.5, we show that the improvements are indeed from supervised training,
rather than domain similarity. , however, it negatively affects CyCLIPn on the
sensitive intrinsic tasks.

| CLIP | CLIPn | CLIPs | CyCLIP | CyCLIPn | CyCLIPs
---|---|---|---|---|---|---
STS12 | 46.14 | 54.25 | 50.31 | 37.84 | 45.60 | 40.42
STS13 | 50.24 | 59.67 | 48.44 | 52.35 | 37.82 | 54.90
STS14 | 48.70 | 59.26 | 51.73 | 46.58 | 40.55 | 49.46
STS15 | 64.90 | 73.81 | 66.09 | 63.25 | 59.62 | 67.01
STS16 | 51.94 | 63.08 | 55.62 | 50.96 | 46.80 | 52.87
STS-B | 61.54 | 68.36 | 65.04 | 60.30 | 54.88 | 60.72
SICKR | 64.70 | 73.09 | 65.82 | 64.78 | 62.62 | 64.34
_Avg_ | 55.45 | 64.50 | 57.58 | 53.72 | 49.70 | 55.67
MR | 61.07 | 63.66 | 61.11 | 59.51 | 62.78 | 60.65
CR | 67.63 | 71.07 | 68.03 | 67.02 | 73.67 | 66.12
SUBJ | 76.39 | 78.09 | 77.52 | 74.24 | 78.90 | 77.36
MPQA | 74.60 | 77.25 | 74.80 | 74.69 | 80.13 | 76.16
SST2 | 61.67 | 66.89 | 63.65 | 61.50 | 68.42 | 64.25
TREC | 60.80 | 56.00 | 60.60 | 55.80 | 53.80 | 62.80
MRPC | 67.07 | 67.77 | 67.77 | 68.00 | 68.75 | 68.29
_Avg_ | 67.03 | 68.68 | 67.64 | 65.82 | 69.49 | 67.95
Table 8:  Intrinsic (top) and extrinsic (bottom) SentEval task performance of
the language encoder in VL models.

In this section, we have shown that (1) Sentence embedding training improves
text-space uniformity with decreased cross-modal alignment. Better balancing
the two goals helps improve retrieval performance. (2) Sentence embedding
training improves language encoder quality on SentEval, resonating the
improved text-space uniformity.

##  6 Conclusion

We extensively evaluate the effectiveness of sentence embedding training for
pretraining contrastive vision-language and audio-language models. We show
that it improves vision-language pretraining, resulting in a better CyCLIP.
However, the benefits to audio-language pretraining are noisy and less
noticeable. We conduct comprehensive analyses and show that sentence embedding
training increases text space uniformity, but with a cost of reduced cross-
modal alignment.

##  7 Limitations

We acknowledge a few limitations that should be considered.

We restrict our scope to cross-modal contrastive models involving three
specific modalities: language, image, and audio. While contrastive learning
has been successfully extended to other modalities such as music,
incorporating music poses additional challenges, particularly regarding
licensing and the heterogeneity of music sources. Downloading music from the
internet and mining reliable music-language pairs are time-consuming tasks,
which we did not consider in detail in this study. Nevertheless, we conducted
initial experiments on the music modality using MusicCaps (Agostinelli et al.,
2023). The results are shown in §A.6.

In our audio-language pretraining experiments, we explored both pretraining
from scratch and pretraining from publicly available language and audio
encoders. However, we believe that a more promising direction would involve
adapting the pretrained language encoder to the audio domain by performing
additional pretraining on audio descriptions before engaging in cross-modal
contrastive learning. Nevertheless, we chose to follow the current methods in
the literature to ensure consistent evaluations and facilitate meaningful
comparisons.

## References

  * Agirre et al. (2015) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, et al. 2015.  Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability.  In _Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015)_ , pages 252–263.
  * Agirre et al. (2014) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M Cer, Mona T Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014\.  Semeval-2014 task 10: Multilingual semantic textual similarity.  In _SemEval@ COLING_ , pages 81–91.
  * Agirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. 2016.  Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation.  In _SemEval-2016. 10th International Workshop on Semantic Evaluation; 2016 Jun 16-17; San Diego, CA. Stroudsburg (PA): ACL; 2016. p. 497-511._ ACL (Association for Computational Linguistics).
  * Agirre et al. (2012) Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012.  Semeval-2012 task 6: A pilot on semantic textual similarity.  In _* SEM 2012: The First Joint Conference on Lexical and Computational Semantics–Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_ , pages 385–393.
  * Agirre et al. (2013) Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013\.  *SEM 2013 shared task: Semantic textual similarity.  In _Second joint conference on lexical and computational semantics (* SEM), volume 1: proceedings of the Main conference and the shared task: semantic textual similarity_ , pages 32–43.
  * Agostinelli et al. (2023) Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. 2023.  MusicLM: Generating music from text.  _arXiv preprint arXiv:2301.11325_.
  * Bielawski et al. (2022) Romain Bielawski, Benjamin Devillers, Tim Van De Cruys, and Rufin Vanrullen. 2022\.  When does CLIP generalize better than unimodal models? when judging human-centric concepts.  In _Proceedings of the 7th Workshop on Representation Learning for NLP_ , pages 29–38, Dublin, Ireland. Association for Computational Linguistics.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Bugliarello et al. (2021) Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott. 2021\.  Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs.  _Transactions of the Association for Computational Linguistics_ , 9:978–994.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Chen et al. (2022) Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2022.  HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 646–650. IEEE.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_.
  * Cherti et al. (2023) Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. 2023\.  Reproducible scaling laws for contrastive language-image learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2818–2829.
  * Conneau and Kiela (2018) Alexis Conneau and Douwe Kiela. 2018.  SentEval: An evaluation toolkit for universal sentence representations.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017.  Supervised learning of universal sentence representations from natural language inference data.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dolan et al. (2004) Bill Dolan, Chris Quirk, and Chris Brockett. 2004.  Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources.  In _COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics_ , pages 350–356, Geneva, Switzerland. COLING.
  * Drossos et al. (2020) Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. 2020.  Clotho: an audio captioning dataset.  In _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 736–740.
  * Elizalde et al. (2023) Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023.  Clap learning audio concepts from natural language supervision.  In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5.
  * Ethayarajh (2019) Kawin Ethayarajh. 2019.  How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 55–65. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Geigle et al. (2022) Gregor Geigle, Jonas Pfeiffer, Nils Reimers, Ivan Vulić, and Iryna Gurevych. 2022\.  Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval.  _Transactions of the Association for Computational Linguistics_ , 10:503–521.
  * Gemmeke et al. (2017) Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017.  Audio set: An ontology and human-labeled dataset for audio events.  In _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 776–780. IEEE.
  * Goel et al. (2022) Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan A. Rossi, Vishwa Vinay, and Aditya Grover. 2022.  CyCLIP: Cyclic contrastive language-image pretraining.  In _Advances in Neural Information Processing Systems_.
  * Guzhov et al. (2022) Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. 2022.  Audioclip: Extending clip to image, text and audio.  In _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 976–980.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778.
  * Hendrycks et al. (2021a) Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. 2021a.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8340–8349.
  * Hendrycks et al. (2021b) Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. 2021b.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15262–15271.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Hu and Liu (2004) Minqing Hu and Bing Liu. 2004.  Mining and summarizing customer reviews.  In _Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’04, page 168–177, New York, NY, USA. Association for Computing Machinery.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning_ , pages 4904–4916. PMLR.
  * Karpathy and Fei-Fei (2015) Andrej Karpathy and Li Fei-Fei. 2015.  Deep visual-semantic alignments for generating image descriptions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3128–3137.
  * Kim et al. (2019) Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019.  AudioCaps: Generating captions for audios in the wild.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 119–132, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Skip-thought vectors.  _Advances in neural information processing systems_ , 28.
  * Koepke et al. (2022) A. Sophia Koepke, Andreea-Maria Oncescu, Joao Henriques, Zeynep Akata, and Samuel Albanie. 2022.  Audio retrieval with natural language queries: A benchmark study.  _IEEE Transactions on Multimedia_ , pages 1–1.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. 2009.  Learning multiple layers of features from tiny images.
  * Li et al. (2020) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020.  On the sentence embeddings from pre-trained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9119–9130, Online. Association for Computational Linguistics.
  * Liang et al. (2022) Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. 2022.  Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.  In _Advances in Neural Information Processing Systems_.
  * Liu et al. (2020) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Ro{bert}a: A robustly optimized {bert} pretraining approach.
  * Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014.  A SICK cure for the evaluation of compositional distributional semantic models.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 216–223, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Mei et al. (2023) Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, and Wenwu Wang. 2023.  WavCaps: A ChatGPT-assisted weakly-labelled audio captioning dataset for audio-language multimodal research.  _arXiv preprint arXiv:2303.17395_.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _Advances in neural information processing systems_ , 26.
  * Pang and Lee (2004) Bo Pang and Lillian Lee. 2004.  A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts.  In _Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)_ , pages 271–278, Barcelona, Spain.
  * Pang and Lee (2005) Bo Pang and Lillian Lee. 2005.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 115–124, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Pfeiffer et al. (2023) Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Maria Ponti. 2023.  Modular deep learning.
  * Piczak (2015) Karol J. Piczak. 2015.  ESC: Dataset for Environmental Sound Classification.  In _Proceedings of the 23rd Annual ACM Conference on Multimedia_ , pages 1015–1018. ACM Press.
  * Plummer et al. (2015) Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In _2015 IEEE International Conference on Computer Vision (ICCV)_ , pages 2641–2649.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Recht et al. (2019) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019.  Do imagenet classifiers generalize to imagenet?  In _International conference on machine learning_ , pages 5389–5400. PMLR.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992. Association for Computational Linguistics.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015\.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115:211–252.
  * Salamon et al. (2014) Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. 2014.  A dataset and taxonomy for urban sound research.  In _Proceedings of the 22nd ACM International Conference on Multimedia_ , MM ’14, page 1041–1044, New York, NY, USA. Association for Computing Machinery.
  * Santurkar et al. (2023) Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. 2023.  Is a caption worth a thousand images? a study on representation learning.  In _International Conference on Learning Representations_.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022.  LAION-5B: An open large-scale dataset for training next generation image-text models.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565, Melbourne, Australia. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642.
  * Su et al. (2021) Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021.  Whitening sentence representations for better semantics and faster retrieval.  _arXiv preprint arXiv:2103.15316_.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Voorhees and Tice (2000) Ellen M. Voorhees and Dawn M. Tice. 2000.  Building a question answering test collection.  In _Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’00, page 200–207, New York, NY, USA. Association for Computing Machinery.
  * Wang et al. (2019) Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. 2019.  Learning robust global representations by penalizing local predictive power.  In _Advances in Neural Information Processing Systems_ , pages 10506–10518.
  * Wang et al. (2021) Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021.  TSDAE: Using transformer-based sequential denoising auto-encoderfor unsupervised sentence embedding learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 671–688, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wang and Isola (2020) Tongzhou Wang and Phillip Isola. 2020.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  In _International Conference on Machine Learning_ , pages 9929–9939. PMLR.
  * Wiebe et al. (2005) Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005.  Annotating expressions of opinions and emotions in language.  _Language resources and evaluation_ , 39:165–210.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Wolfe and Caliskan (2022) Robert Wolfe and Aylin Caliskan. 2022.  Contrastive visual semantic pretraining magnifies the semantics of natural language representations.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3050–3061, Dublin, Ireland. Association for Computational Linguistics.
  * Wu et al. (2022) Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. 2022.  Wav2clip: Learning robust audio representations from clip.  In _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 4563–4567.
  * Wu et al. (2023) Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023.  Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.  In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP_.
  * Yan et al. (2022) An Yan, Jiacheng Li, Wanrong Zhu, Yujie Lu, William Yang Wang, and Julian McAuley. 2022.  CLIP also understands text: Prompting CLIP for phrase understanding.
  * Zhang et al. (2022) Miaoran Zhang, Marius Mosbach, David Adelani, Michael Hedderich, and Dietrich Klakow. 2022.  MCSE: Multimodal contrastive learning of sentence embeddings.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5959–5969, Seattle, United States. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Datasets and hyperparameters

For VL pretraining, our experiments largely follow those for CyCLIP (Goel et
al., 2022) and for CLIP (Radford et al., 2021). We also directly reuse CyCLIP
training hyperparameters but with a smaller batch size, as listed in Table 9.

For AL pretraining, our experiments largely follow that of for LAION-CLAP (Wu
et al., 2023). To process the audio data, we sample the wavefiles at a rate of
48kHz and then convert them to FLAC format using FFmpeg999https://ffmpeg.org/.
We then use a hop size of 480, window size of 1024, and 64 mel-bins for
computing Short-time Fourier transform (STFT) and mel-spectrograms. The audio
encoder input thus has a dimension of 1024 for time steps and 64 for frequency
bins. We list the hyperparameters in Table 10.

Hyperparameter | Value
---|---
Logit scale range | 0 to 4.6052
Epochs | 64
Batch size | 80
Learning rate | 0.0005
Optimizer | Adam
Scheduler | Cosine
Learning rate warmup steps | 10000
Language encoder dropout | 0.1
Table 9: Hyperparameters used for training VL models Hyperparameter | Value
---|---
Logit scale range | 0 to 4.6052
Epochs | 90
Batch size | 80
Learning rate | 0.00009
Optimizer | AdamW
Scheduler | Cosine
Learning rate warmup steps | 9600
Language encoder dropout | 0.1
Table 10: Hyperparameters used for training AL models

###  A.2 Text and image space consistency

CyCLIP variants ensure cross-modal consistency, such that improving the
uniformity of the text space with sentence embedding training also benefits
image space uniformity as shown in Figure 3. As expected, this observation
does not hold for CLIP.

(a)

(b)

(c)

(d)

Figure 3:  Comparing the text and image space consistency between CLIP and
CyCLIP variants. Improving uniformity of the text space also benefits image
space in CyCLIP.

###  A.3 Zero-shot audio classification with more prompts

To resemble zero-shot image classification in VL experiments, we write more
prompts for zero-shot audio classification, as listed in Table 12. Table 11
shows zero-shot audio classification with these prompts.

It can be observed that (1) results in Table 11 clearly outperform Table 7
results, illustrating the importance of using many prompts; (2) sentence
embedding training improves the models trained on Clotho.

|  | CLAP | CLAPs | CyCLAP | CyCLAPs
---|---|---|---|---|---
Clotho | ESC50 | 73.00 | 73.25 | 77.00 | 77.75
US8K | 69.64 | 70.27 | 71.82 | 68.75
AudioCaps | ESC50 | 79.25 | 75.00 | 80.50 | 78.50
US8K | 72.00 | 66.49 | 69.46 | 69.38
Table 11:  Zero-shot audio classification (R@1 in %) on ESC50 and US8K.
Prompts in Table 12 are used.  A sound of label
---
a sound of label.
A constant sound of label
a constant sound of label
A constant sound of label.
a constant sound of label.
A big sound of label
a big sound of label
A big sound of label.
A small sound of label
a small sound of label
A small sound of label.
A label is making a sound.
a label is making a sound
An label is making a sound.
an label is making a sound
A sound of label followed by a sound of label
A sound of label followed by label
A label@
A label
An label
an label
label@
label
label and label
A label is running.
A label is running
A label is happening.
A label is happening
Table 12:  We prepare additional prompts for zero-shot audio classification
resembling the VL prompts. “label” refers to the audio class label. When a
prompt contains whitespace, we display it with “@” for better visualization.

| Text Retrieval |  | Audio Retrieval
---|---|---|---
| R@1 | R@5 | R@10 |  | R@1 | R@5 | R@10
CLAP | 2.30 | 7.85 | 13.88 |  | 2.28 | 7.94 | 13.47
CLAPs | 2.11 | 8.32 | 15.98 |  | 2.81 | 8.84 | 14.91
CyCLAP | 2.97 | 9.09 | 14.07 |  | 2.07 | 7.67 | 13.11
CyCLAPs | 3.54 | 10.05 | 15.41 |  | 2.64 | 8.88 | 14.87

| Text Retrieval |  | Audio Retrieval
---|---|---|---
| R@1 | R@5 | R@10 |  | R@1 | R@5 | R@10
CLAP | 16.30 | 43.78 | 58.41 |  | 14.19 | 40.02 | 53.98
CLAPs | 17.76 | 44.10 | 57.99 |  | 13.81 | 38.60 | 52.02
CyCLAP | 18.81 | 44.83 | 61.23 |  | 14.96 | 39.94 | 54.67
CyCLAPs | 18.18 | 47.23 | 61.02 |  | 13.96 | 40.56 | 54.84

Table 13:  Text and audio retrieval results (%) on Clotho (top) and AudioCaps
(bottom) when pretraining AL models from scratch.

###  A.4 Training audio-language models from scratch

In §5.1 we show that sentence embedding training brings more noticeable
impacts in learning VL models than in AL models; we conjecture that this is
resulted from the fact that AL pretraining often leverages pretrained language
and audio encoders (Elizalde et al., 2023; Wu et al., 2023). As a result, we
conduct the experiments of pretraining the AL model from scratch, i.e., the
language and audio encoders re-initialized. Table 13 lists the retrieval
results on Clotho and AudioCaps.

Compared with Table 5, we observe a significant performance drop since the
encoders are pretrained from scratch. The results are still noisy. We consider
that larger scale AL datasets are necessary to highlight the effectiveness of
learning consistent representation spaces and sentence embedding training.

###  A.5 Unsupervised sentence embedding training with NLI datasets

When evaluting the language encoder on SentEval tasks (§5.2), it is possible
that the improvements brought by supervised sentence embedding training is due
to the fact that NLI datasets have similar domain and language use as the
SentEval tasks. We thus condcut a new type of training, where we use sentences
in the NLI datasets for unsupervised sentence embedding training with SimCSE,
in addition to VL contrastive learning. We name this new training scheme as
CLIPe and CyCLIPe.

| Text Retrieval |  | Image Retrieval
---|---|---|---
| R@1 | R@5 | R@10 |  | R@1 | R@5 | R@10
CLIP | 15.70 | 37.22 | 49.06 |  | 12.48 | 31.10 | 42.23
CLIPs | 17.78 | 38.92 | 50.10 |  | 13.46 | 32.93 | 44.09
CLIPn | 15.74 | 35.66 | 47.38 |  | 13.12 | 31.46 | 42.55
CLIPe | 16.90 | 38.40 | 49.92 |  | 13.21 | 31.21 | 42.26
CyCLIP | 18.92 | 41.46 | 54.00 |  | 15.40 | 35.61 | 46.95
CyCLIPs | 21.30 | 44.34 | 56.54 |  | 16.69 | 37.75 | 49.24
CyCLIPn | 16.32 | 36.76 | 48.16 |  | 14.53 | 34.07 | 45.52
CLIPe | 16.22 | 37.52 | 49.22 |  | 14.05 | 32.56 | 43.15
Table 14:  Zero-shot VL retrieval results (%) on MSCOCO (top) and Flickr30K
(bottom).

Table 16 shows that the new training schemes, CLIPe and CyCLIPe fall behind
the supervised sentence embedding training counterparts CLIPn and CyCLIPn on
SentEval. This confirms that the gains of supervised sentence embedding
trainings is from the NLI task supervision, e.g., premise and hypothesis
relations, instead of other factors such as domain. For completeness, we also
report CLIPe/CyCLIPe performance on VL retrieval tasks in Table 14 and zero-
shot image classification in Table 17.

###  A.6 Preliminary experiments on the music modality

We further conducted a new experiment with the music modality: music-text
retrieval on the MusicCaps dataset introduced by MusicLM (Agostinelli et al.,
2023). MusicCaps consists of 5521 music-caption pairs, of which 2858 pairs are
for training and 2663 are for validation. Each music clip is associated with
hand-curated English descriptions (including genre, mood, tempo, singer voices
etc.) from expert musicians. We use MusicCaps because it is open-sourced and
publicly available. Following tables show the retrieval results (the same
experiment configurations as the audio modality are used; cf. §A.1).

It can be observed from Table 15 that improving the text encoder with
unsupervised sentence embedding training also helps music-text retrieval in
the music modality, especially in the text retrieval scenario (CLAPs generally
outperforms CLAP; CyCLAPs generally outperforms CyCLAP). These music modality
results are consistent with our previous findings on the image and audio
modalities, and plan to explore more in this direction in future work.

| Text Retrieval |  | Music Retrieval
---|---|---|---
| R@1 | R@5 | R@10 |  | R@1 | R@5 | R@10
CLAP | 6.05 | 18.42 | 28.33 |  | 5.46 | 18.45 | 28.50
CLAPs | 5.99 | 18.66 | 28.85 |  | 6.05 | 19.01 | 28.43
CyCLAP | 6.37 | 18.98 | 29.45 |  | 6.69 | 18.87 | 29.45
CyCLAPs | 6.34 | 20.06 | 30.08 |  | 6.62 | 19.36 | 29.13

Table 15: Text and music retrieval results (%) on MusicCaps. | CLIP | CLIPn | CLIPe | CLIPs | CyCLIP | CyCLIPn | CyCLIPe | CyCLIPs
---|---|---|---|---|---|---|---|---
STS12 | 46.14 | 54.25 | 46.54 | 50.31 | 37.84 | 45.60 | 42.03 | 40.42
STS13 | 50.24 | 59.67 | 46.29 | 48.44 | 52.35 | 37.82 | 35.56 | 54.90
STS14 | 48.70 | 59.26 | 47.66 | 51.73 | 46.58 | 40.55 | 27.57 | 49.46
STS15 | 64.90 | 73.81 | 65.48 | 66.09 | 63.25 | 59.62 | 46.65 | 67.01
STS16 | 51.94 | 63.08 | 52.39 | 55.62 | 50.96 | 46.80 | 33.83 | 52.87
STS-B | 61.54 | 68.36 | 60.99 | 65.04 | 60.30 | 54.88 | 44.63 | 60.72
SICKR | 64.70 | 73.09 | 62.86 | 65.82 | 64.78 | 62.62 | 47.53 | 64.34
_Avg_ | 55.45 | 64.50 | 54.60 | 57.58 | 53.72 | 49.70 | 39.69 | 55.67
MR | 61.07 | 63.66 | 59.91 | 61.11 | 59.51 | 62.78 | 59.60 | 60.65
CR | 67.63 | 71.07 | 68.74 | 68.03 | 67.02 | 73.67 | 64.61 | 66.12
SUBJ | 76.39 | 78.09 | 75.86 | 77.52 | 74.24 | 78.90 | 74.16 | 77.36
MPQA | 74.60 | 77.25 | 73.54 | 74.80 | 74.69 | 80.13 | 73.95 | 76.16
SST2 | 61.67 | 66.89 | 60.19 | 63.65 | 61.50 | 68.42 | 60.46 | 64.25
TREC | 60.80 | 56.00 | 56.60 | 60.60 | 55.80 | 53.80 | 57.60 | 62.80
MRPC | 67.07 | 67.77 | 67.83 | 67.77 | 68.00 | 68.75 | 67.48 | 68.29
_Avg_ | 67.03 | 68.68 | 66.10 | 67.64 | 65.82 | 69.49 | 65.41 | 67.95
Table 16: Evaluating the language encoder of different VL models with intrinsic (top) and extrinsic (bottom) SentEval tasks.  | CLIP | CLIPn | CLIPe | CLIPs | CyCLIP | CyCLIPn | CLIPe | CyCLIPs
---|---|---|---|---|---|---|---|---
CIFAR10 | 28.31 | 44.06 | 33.97 | 36.80 | 38.67 | 41.16 | 50.48 | 44.97
CIFAR100 | 13.23 | 17.93 | 12.30 | 10.72 | 17.44 | 19.82 | 21.76 | 22.05
ImageNet1K | 14.94 | 15.97 | 15.74 | 16.01 | 20.99 | 18.13 | 20.07 | 22.13
ImageNetV2 | 12.85 | 13.41 | 13.51 | 14.09 | 17.77 | 15.65 | 17.34 | 18.68
ImageNet-Sk. | 7.72 | 7.75 | 6.36 | 8.14 | 11.67 | 9.93 | 11.54 | 12.85
ImageNet-O | 20.75 | 21.95 | 20.45 | 21.30 | 27.05 | 24.45 | 27.20 | 29.55
ImageNet-A | 3.59 | 3.41 | 3.59 | 3.95 | 5.03 | 4.45 | 4.93 | 5.19
ImageNet-R | 18.39 | 18.51 | 18.25 | 18.24 | 24.37 | 23.07 | 24.36 | 26.72
Table 17:  Zero-shot image classification (R@1 in %) on standard datasets
(top) and datasets with distribution shift or adversarial examples (bottom).


# Inequalities for Lorentz Polynomials

Tamás Erdélyi  Department of Mathematics, Texas A&M University, College
Station, Texas 77843 (T. Erdélyi)  terdelyimath.tamu.edu (T. Erdélyi)
terdelyimath.tamu.edu

(June 9, 2014 )

We prove a few interesting inequalities for Lorentz polynomials. A highlight
of this paper states that the Markov-type inequality

$\max_{x\in[-1,1]}{|f^{\prime}(x)|}\leq n\max_{x\in[-1,1]}{|f(x)|}$

holds for all polynomials of degree at most $n$ with real coefficients for
which $f^{\prime}$ has all its zeros outside the open unit disk. Equality
holds only for $f(x):=c((x\pm 1)^{n}-2^{n-1})$ with a constant $0\neq
c\in{\mathbb{R}}$. This should be compared with Erdős’s classical result
stating that

$\max_{x\in[-1,1]}{|f^{\prime}(x)|}\leq\frac{n}{2}\left(\frac{n}{n-1}\right)^{n-1}\max_{x\in[-1,1]}{|f(x)|}$

for all polynomials of degree at most $n$ having all their zeros in
${\mathbb{R}}\setminus(-1,1)$.

Lorentz representation of polynomials, constrained polynomials, monotone
polynomials, Nikolskii-type inequalities, Markov-type inequalities

††support: 2000 Mathematics Subject Classifications. 11C08, 41A17

## 1\. Introduction

Let ${\Cal{P}}_{n}$ denote the collection of all polynomials of degree at most
$n$ with real coefficients. Let ${\Cal{P}}_{n}^{c}$ denote the collection of
all polynomials of degree at most $n$ with complex coefficients. Let

$\|f\|_{A}:=\sup_{x\in A}{|f(x)|}$

denote the supremum norm of a complex-valed function $f$ defined on a set $A$.
The Markov inequality asserts that

$\|f^{\prime}\|_{[-1,1]}\leq n^{2}\|f\|_{[-1,1]}$

holds for all $f\in{\Cal{P}}_{n}^{c}$. The inequality

$|f^{\prime}(x)|\leq\frac{n}{\sqrt{1-x^{2}}}\,\|f\|_{[-1,1]}$

holds for all $f\in{\Cal{P}}_{n}^{c}$ and for all $x\in(-1,1)$, and is known
as Bernstein inequality. For proofs of these see [2] or [5], for instance.
Various analogues of the above two inequalities are known in which the
underlying intervals, the maximum norms, and the family of functions are
replaced by more general sets, norms, and families of functions, respectively.
These inequalities are called Markov-type and Bernstein-type inequalities. If
the norms are the same in both sides, the inequality is called Markov-type,
otherwise it is called Bernstein-type (this distinction is not completely
standard). Markov- and Bernstein-type inequalities are known on various
regions of the complex plane and the $n$-dimensional Euclidean space, for
various norms such as weighted $L_{p}$ norms, and for many classes of
functions such as polynomials with various constraints, exponential sums of
$n$ terms, just to mention a few. Markov- and Bernstein-type inequalities have
their own intrinsic interest. In addition, they play a fundamental role in
approximation theory.

It had been observed by Bernstein that Markov’s inequality for monotone
polynomials is not essentially better than for arbitrary polynomials.
Bernstein proved that

$\sup_{f}{\frac{\|f^{\prime}\|_{[-1,1]}}{\|f\|_{[-1,1]}}}=\left\\{\begin{aligned}
\textstyle{\frac{1}{4}}(n+1)^{2}\,,\qquad&\text{if}\enskip n\enskip\text{is
odd}\\\ \textstyle{\frac{1}{4}}n(n+2)\,,\qquad&\text{if}\enskip
n\enskip\text{is even}\,,\end{aligned}\right.$

where the supremum is taken over all $f\in{\Cal{P}}_{n}$ which are monotone on
$[-1,1]$. See [22], for instance. This is surprising, since one would expect
that if a polynomial is this far away from the “equioscillating” property of
the Chebyshev polynomial $T_{n}$, then there should be a more significant
improvement in the Markov inequality. In [16] Erdős gave a class of restricted
polynomials for which the Markov factor $n^{2}$ improves to $cn$. He proved
that there is an absolute constant $c$ such that

$|f^{\prime}(x)|\leq\min\left\\{\frac{c\,\sqrt{n}}{\left(1-x^{2}\right)^{2}}\,,\enskip\frac{en}{2}\right\\}\,\|f\|_{[-1,1]}\,,\qquad
x\in(-1,1)\,,$

for all $f\in{\Cal{P}}_{n}$ having all their zeros in
${\mathbb{R}}\setminus(-1,1)$. This result motivated several people to study
Markov- and Bernstein-type inequalities for polynomials with restricted zeros
and under some other constraints. Generalizations of the above Markov- and
Bernstein-type inequality of Erdős have been extended in various directions by
several people including Lorentz [20], Scheick [23], Szabados [24], Máté [21],
P. Borwein [1], Erdélyi [6,7,9,12,13], Rahman and Schmeisser [22], Kroó and
Szabados [18,19], Halász [17], and the list can be even longer. A special
attention is paid to the classes ${\Cal{P}}_{n,k}$ and ${\Cal{P}}_{n,k}^{c}$,
where ${\Cal{P}}_{n,k}$ denotes the set of all polynomials of degree at most
$n$ with real coefficients and with at most $k$ ($0\leq k\leq n$) zeros in the
open unit disk, and ${\Cal{P}}_{n,k}^{c}$ denotes the set of all polynomials
of degree at most $n$ with complex coefficients and with at most $k$ ($0\leq
k\leq n$) zeros in the open unit disk. Associated with $0\leq k\leq n$ and
$x\in(-1,1)$, let

$B_{n,k,x}:=\sqrt{\frac{n(k+1)}{1-x^{2}}}\,,\qquad\quad
B_{n,k,x}^{*}:=\max\left\\{\sqrt{\frac{n(k+1)}{1-x^{2}}}\,,\enskip
n\log\left(\frac{e}{1-x^{2}}\right)\right\\}\,,$

and

$M_{n,k}:=n(k+1)\,,\qquad\quad M_{n,k}^{*}:=\max\\{n(k+1),\enskip n\log
n\\}\,.$

In [10] and [11] it is shown that

$c_{1}\min\\{B_{n,k,x}^{*},M_{n,k}^{*}\\}\leq\sup_{f\in{\Cal{P}}_{n,k}^{c}}{\frac{|f^{\prime}(x)|}{\|f\|_{[-1,1]}}}\leq
c_{2}\min\\{B_{n,k,x}^{*},M_{n,k}^{*}\\}$

for all $x\in(-1,1)$, where $c_{1}>0$ and $c_{2}>0$ are absolute constants.
This result should be compared with the inequalities

$c_{1}\min\\{B_{n,k,x},M_{n,k}\\}\leq\sup_{f\in{\Cal{P}}_{n,k}}{\frac{|f^{\prime}(x)|}{\|f\|_{[-1,1]}}}\leq
c_{2}\min\\{B_{n,k,x},M_{n,k}\\}$

for all $x\in(-1,1)$, where $c_{1}>0$ and $c_{2}>0$ are absolute constants.
See [4] and [11]. It may be surprising that there is a significant difference
between the real and complex cases as far as Markov- and Bernstein-type
inequalities are concerned. In [3] essentially sharp Markov- and Bernstein-
type inequalities for the classes ${\Cal{P}}_{n,k}$ are proved even in $L_{p}$
norms on $[-1,1]$ for all $p>0$.

In this paper we revisit Erdős’s paper [16] and make several remarks to his
Markovi-type inequality in it. Erdős claimed in [16] that his method gave a
Markov factor slightly better than $en/2$, namely,

$\|f^{\prime}\|_{[-1,1]}\leq\frac{n}{2}\left(\frac{n}{n-1}\right)^{n-1}\|f\|_{[-1,1]}$

for all $f\in{\Cal{P}}_{n}$ having all their zeros in
${\mathbb{R}}\setminus(-1,1)$. Indeed, at some points of his arguments, by
replacing applications of the inequality $1+x\leq e^{x}$ with an application
of the inequality between the geometric and arithmetic means of nonnegative
numbers, we can easily see this slight improvement.

In 1963 Lorentz [20] proved that there is an absolute constant $c>0$ such that

$|f^{\prime}(x)|\leq
c\,\min\left\\{\sqrt{\frac{n}{1-x^{2}}}\,,n\right\\}|f|_{[-1,1]}\,,\qquad
x\in(-1,1)\,,$

for all $f\in{\Cal{B}}_{n}(-1,1)$, where

${\Cal{B}}_{d}(a,b):=\left\\{f:f(x)=\sum_{j=0}^{d}{a_{j}(b-x)^{j}(x-a)^{d-j}}\,,\quad
a_{j}\geq 0,\quad j=0,1,\ldots,d\right\\}\,.$

for real numbers $a\leq b$ and nonnegative integers $d$. He also made the
observation that if $f\in{\Cal{P}}_{n,0}$ then either
$f\in{\Cal{B}}_{n}(-1,1)$ or $-f\in{\Cal{B}}_{n}(-1,1)$, where
${\Cal{P}}_{n,0}$ denotes the collection of all $f\in{\Cal{P}}_{n}$ having all
their zeros outside the open unit disk. Scheick [23] has found the best
possible constant $c$ in Lorentz’s Markov-type inequality for
$f\in{\Cal{B}}_{n}(-1,1)$. He showed that

$\|f^{\prime}\|_{[-1,1]}\leq\frac{en}{2}\|f\|_{[-1,1]}$

for all $f\in{\Cal{B}}_{n}(-1,1)$.

An elementary, but very useful tool for proving inequalities for polynomials
with restricted zeros is the Bernstein or Lorentz representation of
polynomials. Namely, as Lorentz observed it, if $p\in\Cal{P}_{n,0}$ is
positive on $(-1,1)$ then it is of the form

$f(x)=\sum_{j=0}^{d}{a_{j}(1-x)^{j}(x+1)^{d-j}}\,,\qquad a_{j}\geq 0\,,\quad
j=0,1,\,\ldots,d\,,$ $None$

with $d=n$. This is formulated as Lemma 3.1 in this paper and its simple proof
is reproduced. Moreover, if a polynomial $p\in\Cal{P}_{n}$ is positive on
$(-1,1)$ and has no zeros in the ellipse $L_{\varepsilon}$ with large axis
$[-1,1]$ and small axis $[-\varepsilon i,\varepsilon i]$
($\varepsilon\in[-1,1]$) then it has a Lorentz representation (1.1) with
$d\leq 3n\varepsilon^{-2}$. See [14]. Combining this with Lorentz’s Markov-
and Bernstein-type inequality gives that there is an absolute constant $c>0$
such that

$|p^{\prime}(x)|\leq
c\,\min\left\\{\frac{\sqrt{n}}{\varepsilon\sqrt{1-x^{2}}}\,,\enskip\frac{n}{\varepsilon^{2}}\right\\}\,\|p\|_{[-1,1]}\,,\qquad
x\in(-1,1)\,,$

for all $p\in{\Cal{P}_{n}}$ having no zeros in $L_{\varepsilon}$.

The minimal value of $d\in{\mathbb{N}}$ for which a polynomial $f$ has a
representation (1.1) is called the Lorentz degree of the polynomial and it is
denoted by $d(f)$. It follows from the already mentioned result in [14] that
$d(p)<\infty$ if and only if $p$ has no zeros in $(-1,1)$. This is a theorem
ascribed to Hausdorff. In addition, it has been proved in [8] that if

$p(x)=((x-a)^{2}+\varepsilon^{2}(1-a^{2}))^{n},\qquad 0<\varepsilon\leq
1,\quad-1<a<1\,,$

then

$c_{1}n\varepsilon^{-2}\leq d(p)\leq c_{2}n\varepsilon^{-2}$

with absolute constants $c_{1}>0$ and $c_{2}>0$. Lorentz degree of
trigonometric polynomials on an interval $(-\omega,\omega)$ shorter than the
period is studied in [15].

## 2\. New Results

For $p>0$ let

$\|f\|_{p}:=\bigg{(}\int_{-1}^{1}{|f(x|\,dx}\bigg{)}^{1/p}\,,\qquad\|f\|_{\infty}:=\max_{x\in[-1,1]}{|f(x)|}\,.$

As in Section 1 we will use the following notation. Let ${\Cal{P}}_{n}$ denote
the collection of all polynomials of degree at most $n$ with real
coefficients. For real numbers $a\leq b$ and $d\in{\mathbb{N}}$ let

${\Cal{B}}_{d}(a,b):=\left\\{f:f(x)=\sum_{j=0}^{d}{a_{j}(b-x)^{j}(x-a)^{d-j}}\,,\quad
a_{j}\geq 0,\quad j=0,1,\ldots,d\right\\}\,.$

Let ${\Cal{P}}_{n,0}$ denote the collection of all $f\in{\Cal{P}}_{n}$ having
all their zeros outside the open unit disk. Our first two results are the
right Nikolskii-type inequalities for the classes ${\Cal{B}}_{d}(-1,1)$ and
${\Cal{P}}_{n,0}$.

###### Theorem 2.1

We have

$\|f\|_{p}\leq\left(\frac{qd+1}{2}\right)^{1/q-1/p}\|f\|_{q}$

for all $f\in{\Cal{B}}_{d}(-1,1)$ and for all $0<q<p\leq\infty$. Equality
holds only for $f(x):=c(x\pm 1)^{n}$ with a constant $c\geq 0$.

Combining Theorem 2.1 with Lemma 3.2 gives the following.

###### Theorem 2.2

We have

$\|f\|_{p}\leq\left(\frac{qn+1}{2}\right)^{1/q-1/p}\|f\|_{q}$

for all $f\in{\Cal{P}}_{n,0}$ and for all $0<q<p\leq\infty$. Equality holds
only for $f(x):=c(x\pm 1)^{n}$ with a constant $0\neq c\in{\mathbb{R}}$.

An application of Theorem 2.1 with $q=1$ and $p=\infty$ allows us to prove the
following a sharp Markov-type inequality for all $f\in{\Cal{P}}_{n}$ such that
$f^{\prime}\in{\Cal{B}}_{d-1}(-1,1)$.

###### Theorem 2.3

We have

$\|f^{\prime}\|_{\infty}\leq d\,\|f\|_{\infty}$

for all $f\in{\Cal{P}}_{d}$ for which $f^{\prime}\in{\Cal{B}}_{d-1}(-1,1)$.
Equality holds only for $f(x):=c(x\pm 1)^{n}$ with a constant $c\geq 0$.

Combining Theorem 2.3 with Lemma 3.2 gives the following.

###### Theorem 2.4

We have

$\|f^{\prime}\|_{\infty}\leq n\,\|f\|_{\infty}$

for all $f\in{\Cal{P}}_{n}$ for which $f^{\prime}$ has all its zeros outside
the open unit disk. Equality holds only for $f(x):=c((x\pm 1)^{n}-2^{n-1})$
with a constant $0\neq c\in{\mathbb{R}}$.

Our final result is a sharp Markov-type inequality for all $f\in{\Cal{P}}_{n}$
which are monotone on $[-1,1]$ and have all their zeros in
${\mathbb{R}}\setminus(-1,1)$. Erdős claimed this in [16] but he did not give
a hint how to prove this. Experts seem to be puzzled by this observation of
Erdős even today.

###### Theorem 2.5

We have

$\|f^{\prime}\|_{\infty}\leq\frac{n}{2}\,\|f\|_{\infty}$

for all $f\in{\Cal{P}}_{n}$ which is monotone on $[-1,1]$ and has all its
zeros in ${\mathbb{R}}\setminus(-1,1)$. Equality holds only for $f(x):=c(x\pm
1)^{n}$ with a constant $0\neq c\in{\mathbb{R}}$.

We note that there is a incorrect hint to Part c] if Exercise 10 on page 482
of the book [2] suggesting that Theorem 2.5 holds. However, it was discovered
by M. Boedihardjo that the hint to part c] of E.10 on page 482 of the book [2]
does not work out. Here we claim a proof of Theorem 2.5 as a consequence of
Theorem 2.4. A direct elementary proof of Theorem 2.5 by using undergraduate
calculus would be desirable.

## 3\. Lemmas

###### Lemma 3.1

Let $a\leq c\leq d\leq b$ be real numbers, and let $d$ be a nonnegative
integer. Then ${\Cal{B}}_{d}(a,b)\subset{\Cal{B}}_{d}(c,d)$.

###### Demonstration Proof of Lemma 3.1

This follows from the identities

$x-a=\frac{c-a}{d-c}\,(x-c)+\frac{d-a}{d-c}\,(d-x)$

and

$b-x=\frac{b-c}{d-c}\,(x-c)+\frac{b-d}{d-c}\,(d-x)$

valid for all $x\in{\mathbb{C}}$. ∎

###### Lemma 3.2

Suppose $f\in{\Cal{P}}_{n}$ has all its zeros outside the open unit disk. Then
either $f\in{\Cal{B}}_{n}(-1,1)$ or $-f\in{\Cal{B}}_{n}(-1,1)$.

###### Demonstration Proof of Lemma 3.2

This follows from the identities

$x-\alpha=\frac{1-\alpha}{2}\,(x+1)-\frac{\alpha+1}{2}\,(1-x)$

and

$(x-\alpha)(x-\overline{\alpha})=\frac{1}{2}|1+\alpha|^{2}(1-x)^{2}+\frac{1}{2}(|\alpha|^{2}-1)(1-x^{2})+\frac{1}{2}|1-\alpha|^{2}(x+1)^{2}$

valid for all $x\in{\mathbb{C}}$ and $\alpha\in{\mathbb{C}}$. ∎

###### Lemma 3.3

We have

$(\max\\{f(a),f(b)\\})^{q}\leq\frac{qd+1}{b-a}\int_{a}^{b}{f(x)^{q}\,dx}$

for all $f\in{\Cal{B}}_{d}(a,b)$.

###### Demonstration Proof of Lemma 3.3

Let $f\in{\Cal{B}}_{d}(a,b)$ be of the form

$f(x)=\sum_{j=0}^{d}{a_{j}(b-x)^{j}(x-a)^{d-j}}\,,\qquad a_{j}\geq 0,\quad
j=0,1,\ldots,d\,.$

Then

$\begin{split}f(b)^{q}=&\,(a_{0}(b-a))^{dq}=\frac{qd+1}{b-a}\,\int_{a}^{b}{(a_{0}(b-x))^{dq}\,dx}\cr\leq&\,\frac{qd+1}{b-a}\,\int_{a}^{b}{\Bigg{(}\sum_{j=0}^{d}{a_{j}(b-x)^{j}(x-a)^{d-j}}\Bigg{)}^{q}\,dx}\cr\leq&\,\frac{qd+1}{b-a}\int_{a}^{b}{f(x)^{q}\,dx}\,.\cr\end{split}$

Similarly,

$\begin{split}f(a)^{q}=&\,(a_{d}(b-a))^{dq}=\frac{qd+1}{b-a}\,\int_{a}^{b}{(a_{d}(x-a))^{dq}\,dx}\cr\leq&\,\frac{qd+1}{b-a}\,\int_{a}^{b}{\Bigg{(}\sum_{j=0}^{d}{a_{j}(b-x)^{j}(x-a)^{d-j}}\Bigg{)}^{q}\,dx}\cr\leq&\,\frac{qd+1}{b-a}\int_{a}^{b}{f(x)^{q}\,dx}\,.\cr\end{split}$

∎

###### Lemma 3.4

We have

$\|f\|_{\infty}^{q}\leq\frac{qd+1}{2}\,\|f\|_{q}^{q}\,.$

for all $f\in{\Cal{B}}_{d}(-1,1)$.

###### Demonstration Proof of Lemma 3.4

Let $y\in[-1,1]$ be such that $f(y)=\|f\|_{\infty}$. By Lemma 3.1 we have

${\Cal{B}}_{d}(-1,1)\subset{\Cal{B}}_{d}(-1,y)\cap{\Cal{B}}_{d}(y,1)\,.$

Hence Lemma 3.2 yields

$(y+1)f(y)^{q}\leq(qd+1)\int_{-1}^{y}{f(x)^{q}\,dx}$

and

$(1-y)f(y)^{q}\leq(qd+1)\int_{y}^{1}{f(x)^{q}\,dx}$

Adding the above two inequalities, we conclude

$\|f\|_{\infty}^{q}=f(y)^{q}\leq\frac{qd+1}{2}\,\int_{-1}^{1}{f(x)^{q}\,dx}=\frac{qd+1}{2}\,\|f\|_{q}^{q}\,.$

∎

## Proof of the Theorems

###### Demonstration Proof of Theorem 2.1

When $p=\infty$ the Theorem follows from Lemma 3.3. Now let Let
$f\in{\Cal{B}}_{d}(-1,1)$ and $0<q\leq p\leq\infty$. Using Lemma 3.3 we obtain

$\begin{split}\|f\|_{p}^{p}=&\,\int_{-1}^{1}{f(x)^{p}\,dx}\leq\Bigg{(}\int_{-1}^{1}{f(x)^{q}\,dx}\Bigg{)}\|f\|_{\infty}^{p-q}\leq\,\|f\|_{q}^{q}\left(\frac{qd+1}{2}\right)^{(p-q)/q}\|f\|_{q}^{p-q}\cr=&\,\left(\frac{qd+1}{2}\right)^{(p-q)/q}\|f\|_{q}^{p}\,,\cr\end{split}$

hence

$\|f\|_{p}\leq\left(\frac{qd+1}{2}\right)^{1/q-1/p}\|f\|_{q}\,.$

∎

###### Demonstration Proof of Theorem 2.2

. Combining Theorem 2.1 and Lemma 3.2 gives the result. ∎

###### Demonstration Proof of Theorem 2.3

Applying Theorem 2.1 with $f$ replaced by $f^{\prime}\in{\Cal{B}}_{d-1}$,
$p:=\infty$ and $q:=1$, we obtain

$\|f^{\prime}\|_{\infty}\leq\frac{d}{2}\,\int_{-1}^{1}{f^{\prime}(x)\,dx}=\frac{d}{2}\,(f(1)-f(-1))\leq
d\,\|f\|_{\infty}\,.$

∎

###### Demonstration Proof of Theorem 2.4

Assume that $f^{\prime}\in{\Cal{P}}_{n-1}$ has no zeros in the open unit disk.
Then, by Lemma 3.2 either $f^{\prime}\in{\Cal{B}}_{n-1}(-1,1)$ or
$-f^{\prime}\in{\Cal{B}}_{n-1}(-1,1)$. Without loss of generality we may
assume that $f^{\prime}\in{\Cal{B}}_{n}(-1,1)$, and Theorem 2.3 gives the
result. ∎

###### Demonstration Proof of Theorem 2.5

Assume that $f\in{\Cal{P}}_{n}$ is monotone on $[-1,1]$ and has all its zeros
in ${\mathbb{R}}\setminus(-1,1)$. Then, by Rolle’s Theorem $f^{\prime}$ has
all its zeros in ${\mathbb{R}}\setminus(-1,1)$, and hence Lemma 3.2 implies
that either $f^{\prime}\in{\Cal{B}}_{n-1}(-1,1)$ or
$-f^{\prime}\in{\Cal{B}}_{n-1}(-1,1)$. Without loss of generality we may
assume that $f^{\prime}\in{\Cal{B}}_{n}(-1,1)$. Applying Theorem 2.3 we
conclude that

$\|f^{\prime}\|_{\infty}\leq\frac{n}{2}\,\int_{-1}^{1}{f^{\prime}(x)\,dx}=\frac{n}{2}\,(f(1)-f(-1))\leq\frac{n}{2}\,\|f\|_{[-1,1]}\,,$

where in the last step we used that $f(1)f(-1)\geq 0$ since
$f\in{\Cal{P}}_{n}$ has all its zeros in ${\mathbb{R}}\setminus(-1,1)$. ∎

## References

  * 1 Borwein, P.B., Markov’s inequality for polynomials with real zeros, Proc. Amer. Math. Soc. 93 (1985), 43–48.
  * 2 Borwein, P.B., & T. Erdélyi, Polynomials and Polynomial Inequalities, Springer-Verlag, Graduate Texts in Mathematics, 1995a.
  * 3 Borwein, P.B., & T. Erdélyi, Markov and Bernstein type inequalities in $L_{p}$ for classes of polynomials with constraints, J. London Math. Soc. 51 (1995b), 573–588.
  * 4 Borwein, P.B., & T. Erdélyi, Sharp Markov-Bernstein type inequalities for classes of polynomials with restricted zeros, Constr. Approx. 10 (1994), 411–425.
  * 5 DeVore, R.A., & G.G. Lorentz, Constructive Approximation, Springer-Verlag, 1993.
  * 6 Erdélyi, T., Pointwise estimates for derivatives of polynomials with restricted zeros, in: Haar Memorial Conference, J. Szabados & K. Tandori, Eds., North-Holland, Amsterdam, 1987, pp. 329–343.
  * 7 Erdélyi, T., Bernstein-type inequalities for the derivative of constrained polynomials, Proc. Amer. Math. Soc. 112 (1991), 829–838.
  * 8 Erdélyi, T., Estimates for the Lorentz degree of polynomials, J. Approx. Theory 67 (1991), 187–198.
  * 9 Erdélyi, T., Markov-Bernstein type inequalities for polynomials under Erdős-type constraints, in Paul Erdős and his Mathematics I, Bolyai Society Mathematical Studies, 11, Gábor Halász, László Lovász, Dezső Miklós, and Vera T. Sós (Eds.) (2002), Springer Verlag, 219–239.
  * 10 Erdélyi, T., Markov-type inequalities for constrained polynomials with complex coefficients, Illinois J. Math. 42 (1998a), 544–563.
  * 11 Erdélyi, T., Markov-Bernstein type inequalities for constrained polynomials with real versus complex coefficients, Journal d’Analyse Mathematique 74 (1998b), 165–181.
  * 12 Erdélyi, T., Extremal properties of polynomials, in “A Panorama of Hungarian Mathematics in the XXth Century” János Horváth (Ed.), Springer, New York, 2005, pp. 119–156.
  * 13 Erdélyi, T., Markov-Nikolskii type inequality for absolutely monotone polynomials of order $k$, Journal d’Analyse Math. 112 (2010), 369-381.
  * 14 Erdélyi, T., & J. Szabados, On polynomials with positive coefficients, J. Approx. Theory 54 (1988), 107–122.
  * 15 Erdélyi, T., & J. Szabados, On trigonometric polynomials with positive coefficients, Studia Sci. Math. Hungar. 24 (1989a), 71–91.
  * 16 Erdős, P., On extremal properties of the derivatives of polynomials, Ann. of Math. 2 (1940), 310–313.
  * 17 Halász, G., Markov-type inequalities for polynomials with restricted zeros, J. Approx. Theory 101 (1999), 148–155.
  * 18 Kroó, A., & J. Szabados, Constructive properties of self-reciprocal polynomials, Analysis 14 (1994), 319–339.
  * 19 A. Kroó , & J. Szabados, On the exact Markov inequality for $k$-monotone polynomials in the uniform and $L_{1}$ norms, Acta Math. Hungar. 125 (2009), 99–112.
  * 20 Lorentz, G.G., The degree of approximation by polynomials with positive coefficients, Math. Ann. 151 (1963), 239–251.
  * 21 Máté, A., Inequalities for derivatives of polynomials with restricted zeros, Proc. Amer. Math. Soc. 82 (1981), 221–224.
  * 22 Rahman, Q.I., & G. Schmeisser, Analytic Theory of Polynomials, Oxford University Press, 2002.
  * 23 Scheick, J.T., Inequalities for derivatives of polynomials of special type, J. Approx. Theory 6 (1972), 354–358.
  * 24 Szabados, J., Bernstein and Markov type estimates for the derivative of a polynomial with real zeros, in “Functional Analysis and Approximation” (1981), Birkhäuser Verlag, 177-188.