# Improving Medical Multi-modal Contrastive Learning
with Expert Annotations

Yogesh Kumar<EMAIL_ADDRESS>
Department of Computer Science
Aalto University, Finland

Pekka Marttinen<EMAIL_ADDRESS>
Department of Computer Science
Aalto University, Finland

###### Abstract

We introduce eCLIP, an enhanced version of the CLIP model that integrates
expert annotations in the form of radiologist eye-gaze heatmaps. It tackles
key challenges in contrastive multi-modal medical imaging analysis, notably
data scarcity and the “modality gap” – a significant disparity between image
and text embeddings that diminishes the quality of representations and hampers
cross-modal interoperability. eCLIP integrates a heatmap processor and
leverages mixup augmentation to efficiently utilize the scarce expert
annotations, thus boosting the model’s learning effectiveness. eCLIP is
designed to be generally applicable to any variant of CLIP without requiring
any modifications of the core architecture. Through detailed evaluations
across several tasks, including zero-shot inference, linear probing, cross-
modal retrieval, and Retrieval Augmented Generation (RAG) of radiology reports
using a frozen Large Language Model, eCLIP showcases consistent improvements
in embedding quality. The outcomes reveal enhanced alignment and uniformity,
affirming eCLIP’s capability to harness high-quality annotations for enriched
multi-modal analysis in the medical imaging domain.

##  1 Introduction

Pretraining foundation models on multi-modal data – particularly leveraging
the relationships between text and images – has proven to be a robust strategy
for generating versatile embeddings (Radford et al., 2021; Jia et al., 2021).
These embeddings enhance the efficacy in several downstream tasks, from image
generation to advanced vision-language integration (Liu et al., 2023; Ramesh
et al., 2021; Singh et al., 2022). Central to this approach is the employment
of a contrastive learning (CL) loss objective (Oord et al., 2018; Chen et al.,
2020; Zbontar et al., 2021), where models are trained to align positive pairs
(e.g., an image and its corresponding caption) while diversifying negative
ones. A significant hurdle in this approach is the necessity of vast datasets,
often comprising several millions of data points, for competitive results.
Models such as CLIP (Radford et al., 2021) have been trained on internet-scale
datasets, estimated to encompass hundreds of millions of image-text pairs
(Cherti et al., 2023; Xu et al., 2023a). Acquiring datasets of this magnitude
poses substantial challenges in specialized fields that require expert
knowledge for data collection, processing and annotation. The medical imaging
domain exemplifies these difficulties, where acquiring even a single data
point, such as a chest X-ray, involves complex processes requiring expertise
and significant resources. Moreover, the procurement of such data for machine
learning research is further complicated by ethical considerations, patient
privacy concerns and the need for extensive de-identification procedures.

This has led to the prevalent use of foundation models, initialized with
weights from models trained on extensive internet-scale datasets, for tasks in
the medical domain (Zhang et al., 2022; Wang et al., 2022; Huang et al., 2021;
Krishnan et al., 2022). However, the areas of interest within medical images
are often nuanced and require expert knowledge to interpret, rendering them
indistinguishable to a general-purpose model. In 1(a), we investigate the
embeddings generated by a CLIP model – initially pretrained on internet data –
using samples from the Open-I dataset(Demner-Fushman et al., 2016), which
includes X-rays and corresponding radiology reports. We categorize the samples
into subgroups based on the primary abnormality identified in each report,
such as ‘normal’, cardiomegaly, atelectasis and opacity. A histogram of the
cosine similarities between embeddings from different groups indicates a high
degree of similarity, with values approaching 1. This could lead to potential
challenges in downstream zero-shot inference tasks, which rely on the spatial
segregation of embeddings from different groups (Radford et al., 2021).
Typically, continual pretraining on medically relevant data is employed to
enhance the model’s ability to differentiate between various abnormalities.

((a)) Histogram of Cosine Similarities Among Subgroups. The model exhibits
high cosine similarity among embeddings within the text (_red_) and image
(_blue_) modalities, regardless of the differences among subgroups. This
underscores the model’s challenge in capturing the subtleties inherent in
medical data.

((b)) Modality Gap. Despite the CLIP contrastive loss aiming at closely
aligning image and text embeddings within a shared space, the modalities
remain segregated into distinct regions.

Figure 1: Analysis of CLIP Embeddings in Medical Imaging The figure presents
embeddings generated by a CLIP model, pretrained on an internet-scale dataset,
applied to the Open-I dataset pairing X-rays with corresponding radiology
reports.

Recent studies (Liang et al., 2022; Goel et al., 2022; Oh et al., 2023; Zhang
et al., 2024; Tschannen et al., 2023) have identified a “modality gap” in
multi-modal contrastive representation learning, where the embeddings from
different modalities (e.g., images and text) fall in distinct regions in the
shared embedding space. This separation, which arises from factors such as
initial model weights and the objectives of contrastive learning (Liang et
al., 2022; Zhang et al., 2024), leads to the “cone effect” where embeddings of
each modality are restricted to a narrow region of the embedding hypersphere.
In 1(b), we illustrate this within the medical domain with a 2D UMAP(McInnes
et al., 2018) projection of image and text embeddings. This example highlights
how embeddings from the same modality but different semantic groups, such as
X-ray images of varying abnormalities, cluster closely together. This makes it
difficult for a model to distinguish between semantically different images,
undermining its performance in medical image analysis.

Figure 2: eCLIP Pretraining with Expert Annotations. eCLIP adds a Heatmap
Processor (right), featuring a multi-headed attention layer, to the standard
Image and Text encoders in CLIP. This processor, along with vision and text
encoders, maps inputs into a shared hypersphere. Here, the original image
($I_{i}$), its text ($T_{i}$) and the heatmap-processed image ($I^{E}_{i}$)
are positioned within a tripartite area (shown here after 2D UMAP projection,
please refer to the Supplement for a scaled version). We employ mixup between
$I_{i}$ and $I^{E}_{i}$ to generate the embedding $v^{\lambda}_{i}$, which
gives us additional positive pairs to enhance the CLIP InfoNCE loss
optimization. An auxillary loss, $\mathcal{L}_{\text{priming}}$, is used
during the initial training steps to “prime” the heatmap processor to imitate
an identity function when the heatmap is composed of all ones.

We investigate the potential of integrating expert annotations, specifically
radiologist eye-gaze heatmaps, to alleviate these issues. Processing the eye-
gaze data from radiologists(Karargyris et al., 2021) provides us with heatmaps
indicative of the radiologist’s attention across different regions of the
X-ray images. This heatmap reflects areas of clinical interest aligned with
details present in radiology reports. We posit that this could help capture
nuanced visual cues in the X-rays and therefore pairing it with reports can
enrich the CLIP training data with high-quality positive pairs. Due to the
scarcity of such expert annotated data, we employ the mixup strategy, a data
augmentation technique which has been effective in both supervised (Zhang et
al., 2017; Verma et al., 2019; Han et al., 2022) and contrastive
learning(Verma et al., 2021; Oh et al., 2023), to create additional synthetic
samples.

We present eCLIP (expert-annotated CLIP), an adaptation of the CLIP model that
incorporates expert eye-gaze heatmaps, without modifying the CLIP model’s core
architecture. The operational workflow of eCLIP is depicted in Fig. 2. Our
contributions are as follows:

  * •

Utilization of Expert Annotations. We harness radiologist eye-gaze heatmaps to
create additional embeddings, effectively introducing valuable positive and
negative pairs for enhancing the contrastive learning process.

  * •

eCLIP Architecture. Our implementation features a heatmap processing mechanism
utilizing multi-headed attention (MHA), optimized for handling both heatmaps
and original images. This is complemented by a mixup strategy to addresses the
challenge of data scarcity, and curriculum learning to ensure a gradual
introduction of expert annotations.

  * •

Comprehensive Evaluation. We assess eCLIP’s zero-shot classification accuracy,
sample efficiency and cross-modal retrieval performance and embedding quality
across multiple chest X-ray datasets. We also evaluate the cross-modal
embeddings to generate radiology reports using a frozen Large Language Model
(LLM) without explicitly fine-tuning on medical data.

##  2 Related Work

Modality Gap:  Liang et al. (2022) pinpoint the origins of the modality gap to
the nuances of model initialization and the objectives of contrastive
learning, underscoring its impact on downstream tasks and fairness. Oh et al.
(2023) highlight poor uniformity and alignment in CLIP’s embeddings and
propose a finetuning method for robust representations. Zhang et al. (2024)
explore the geometry of this embedding space, and provide both theoretical and
empirical insights on the nature of this geometry. Subsequent research has
produced methods to mitigate the modality gap through diverse and creative
approaches (Goel et al., 2022; Tschannen et al., 2023; Zhang et al., 2023b; Gu
et al., 2023; Nukrai et al., 2022).

Improving Contrastive Learning:  Several methods have improved upon the CLIP
objective by introducing auxilliary losses, e.g., SLIP (Mu et al., 2022) uses
SimCLR (Chen et al., 2020) loss, M3MAE (Geng et al., ; Weers et al., 2023)
augment the Masked Autoencoder (He et al., 2022) reconstruction loss, FLIP (Li
et al., 2023) randomly masks out input images to improve scaling, DACL (Verma
et al., 2021) proposes a domain agnostic mixup strategy, SILC (Naeem et al.,
2023) uses self-distillation, Mo et al. (2024) utilized specialist captions to
generate pseudo labels for unpaired images, Zhang et al. (2023a) propose
Multi-task Paired Masking with Alignment to improve cross-modal interaction.
Similarly there have been works that have identified the need to make the CLIP
model focus on sub-regions in order to enhance its utility and downstream
performance, GLoRIA (Huang et al., 2021) considers the loss from local regions
from within the image and reports, Alpha-CLIP (Sun et al., 2023) uses the
alpha channel to guide the CLIP model to focus on different regions of the
image and generate the masks for all the images in the corpus using an image
segmentation pipeline and TIER (Palepu & Beam, 2023) uses a regularization
term to improve the local focus of the model.

Multi-modal Contrastive Learning in Medical Imaging: Zhang et al. (2022)
demonstrated enhanced downstream performance by jointly using chest X-ray and
report pairing for training a contrastive learning model. This was further
improved by Huang et al. (2021), by exploiting local and global features from
both modalities; Wang et al. (2022) and You et al. (2023) achieved impressive
results by using a Swin Tiny model as the image encoder and by adding
modifications to the contrastive loss. Several other works have developed
similar contrastive learning foundation models while utilizing the biomedical
image texts to achieve impressive results (Huang et al., 2023; Xu et al.,
2023b; Moor et al., 2023; Wu et al., 2023; Sowrirajan et al., 2010; Tu et al.,
2024). Karargyris et al. (2021) and Bigolin Lanfredi et al. (2022) augment a
subset of the MIMIC-CXR (Johnson et al., 2019b) samples with high quality eye-
tracking and verbal transcripts from several radiologists. van Sonsbeek et al.
(2023) and Wang et al. (2024) utilize the heatmaps from eye-gaze to improve
image classification.

##  3 Method

Notations. For a given chest X-ray image $I_{i}$ and its radiology report
$T_{i}$, indexed by $i$ in our dataset, we denote their L2-normalized
embeddings as $v_{i}$ and $t_{i}$ respectively, residing in a d-dimensional
space ($v_{i},t_{i}\in\mathcal{R}^{d}$). Image embeddings are obtained through
an encoder $v_{i}=f(I_{i})$ and text embeddings via $t_{i}=g(T_{i})$. Applying
an expert heatmap $E_{i}$ to an image results in the corresponding image
embedding $v_{i}^{E}$. We denote the loss value for the $i$-th sample as
$\mathcal{L}_{i}$. For the case of contrastive loss, this is computed in terms
of some similarity measure between the embeddings, $sim(v_{i},t_{i})$,
typically cosine similarity defined as $v_{i}\cdot t_{i}$.

###  3.1 Background

Central to CLIP’s effectiveness is the InfoNCE loss (Oord et al., 2018), a
mechanism engineered to optimize the similarity measures between corresponding
(positive) pairs and to minimize those among non-corresponding (negative)
pairs. The formulation of the CLIP loss objective is as follows:

$\displaystyle\mathcal{L}_{\text{text}}$
$\displaystyle=\mathop{\mathbb{E}}_{(t_{i},v_{i})\sim
pos}\left[-\log\frac{\exp(sim(t_{i},v_{i})/\tau)}{\exp(sim(t_{i},v_{i})/\tau)+\sum_{j\neq
i}\exp(sim(t_{i},v_{j})/\tau)}\right]$ (1)

Total loss is then defined as,
$\mathcal{L}_{\text{total}}=\frac{1}{2}\left(\mathcal{L}_{\text{text}}+\mathcal{L}_{\text{image}}\right)$,
where $\mathcal{L}_{\text{image}}$ denotes the corresponding loss for the
image to text mapping. Here, $\tau$ represents the temperature parameter that
controls the scale of the similarity scores, typically framed as a learnable
parameter during training. The loss expectation is taken over all the positive
pairings in the dataset.

Theoretical results on CL indicate the concepts of _alignment_ and
_uniformity_ as critical for the quality of embeddings (Wang & Isola, 2020;
Wang & Liu, 2021). Alignment focuses on reducing the distance between positive
pairs while uniformity seeks to evenly distribute the embeddings across the
unit hypersphere, preventing extreme clustering that could impair the model’s
generalizability and discriminative capabilities. The alignment and uniformity
can be defined formally as follows (Oh et al., 2023):

Alignment $\displaystyle=-\mathbb{E}_{(v_{i},t_{i})\sim
pos}\left[\|v_{i}-t_{i}\|_{2}^{2}-\min_{j\neq
i}\|v_{i}-t_{j}\|_{2}^{2}\right]$ (2) Uniformity
$\displaystyle=-\log\mathbb{E}_{(v_{i},t_{j})\sim\mathcal{D}}\left[\exp(-2\|v_{i}-t_{j}\|_{2}^{2})\right]$
(3)

Figure 3: Comparing eCLIP with $m^{2}$-mixup(Oh et al., 2023). (left) Standard
CLIP showing image-text positive pairs $(v_{i},t_{i})$ (solid line), while the
other image embeddings serve as negative pairs (dashed line). (center) the
$m^{2}$-mixup creates negative pairs $(v^{\lambda}_{j},t_{i})$ via
interpolation between embeddings along the geodesic. (right) eCLIP adds expert
image embedding, $v^{E}_{i}$, in addition to $v_{i}$ for text $t_{i}$, forming
additional positive and negative pairs

High intra-modal similarity, e.g. between two images as seen in Fig. 1, can
inadvertently enhance similarity among negative pairs, inflating the
denominator of the loss function in Equation (1), and, consequently, hurting
the model’s ability to differentiate between positive and negative pairs
during training. A conventional method to counter this involves incorporating
hard negative pairs, a strategy (Oh et al., 2023) employ by mixing embeddings
from different modalities. While effective, this cross-modal mixup may obscure
the semantic clarity of embeddings. As an alternative we propose increasing
the dataset with additional positive pairs that exhibit minimal semantic
overlap by integrating expert annotations. Fig. 3 compares the positive and
negative pair creation in eCLIP with traditional CLIP and $m^{2}$-mixup.

###  3.2 Introducing Expert Annotations to CLIP

Algorithm 1 eCLIP Algorithm

0: Image Encoder $f(.)$

0: Text Encoder $g(.)$

1: $\mathcal{L}_{\text{priming}}\leftarrow 0$; $n_{\text{p}}\leftarrow 0$

2: for minibatch $\\{x_{i}\\}_{i=1}^{N}$ do

3: Unpack $x_{i}$ to $(T_{i},I_{i})$ and optionally $E_{i}$

4: $t_{i}\leftarrow g(T_{i})$

5: $v_{i}\leftarrow f(I_{i})$

6: $p_{\text{uni}}\sim\text{Uniform}(0,1)$

7: if $p_{\text{uni}}<p_{\text{curr}}$ and $E_{i}$ is provided then

8: $\lambda\sim\text{Beta}(\alpha,\alpha)$

9: $I_{i}^{E}\leftarrow\text{HeatmapProcessor}(I_{i},E_{i})$

10: $I_{i}^{\lambda}\leftarrow I_{i}\lambda+I_{i}^{E}(1-\lambda)$

11: $v_{i}^{\lambda}\leftarrow f(I_{i}^{\lambda})$

12: end if

13: if $E_{i}$ is entirely ones then

14: $I_{i}^{\text{R}}\leftarrow\text{HeatmapProcessor}(I_{i},E_{i})$

15:
$\mathcal{L}_{\text{priming}}\leftarrow\mathcal{L}_{\text{priming}}+(I_{i}-I_{i}^{\text{R}})^{2}$

16: $n_{\text{p}}\leftarrow n_{\text{p}}+1$

17: end if

18: end for

Our objective with eCLIP is to enhance the CLIP framework by integrating
expert annotations – radiologist eye-gaze heatmaps – to diversify the pool of
positive samples. The eCLIP model is designed to be compatible across all CLIP
variants without modifying its core architecture. The heatmap processor (Fig.
2, right) first converts the images and heatmaps into a sequence of patches
and applies multi-headed attention (MHA) over the sequences. The patchified
heatmap overlaid images serve as queries, while the original image’s patches
act as keys and values. The processed output is then reconstructed back to its
original image format, enabling the standard CLIP image encoder to obtain
expert image embeddings. These new embeddings and their text embedding pair
introduce additional positive samples for the contrastive loss objective (Fig.
3, right).

However, the size of the expert annotated data is orders of magnitude smaller
than the data available for CLIP training. To effectively leverage the scarce
expert-annotated data, we implement mixup augmentation (Zhang et al., 2017).
As illustrated in Fig. 2 (left), this involves blending an original image
$I_{i}$ with its expert version $I_{i}^{E}$ to create $I_{i}^{\lambda}=\lambda
I_{i}+(1-\lambda)I_{i}^{E}$, where $\lambda\sim\text{Beta}(\alpha,\alpha)$.
(We set $\alpha=0.3$ in all our experiments.) The eCLIP image encoder then
processes $I_{i}^{\lambda}$ to produce the image embedding
$v_{i}^{\lambda}=f(I_{i}^{\lambda})$. These expert embeddings form new
positive pairs $(v_{i}^{\lambda},t_{i})$ as well as corresponding negative
pairs, which are added with existing pairs $(v_{i},t_{i})$ during the
computation of the CLIP InfoNCE Loss, $\mathcal{L}_{i}$.

To seamlessly integrate expert annotations without disrupting the foundational
training of the eCLIP model, we employ a phased curriculum learning strategy
(Bengio et al., 2009). This approach comprises a cold start phase where the
model is initially trained without the expert annotations to establish a
robust baseline. This phase accounts for about 10% of the total training
iterations. It then transitions into a warmup phase, gradually increasing the
inclusion of expert examples from 0.05 to 0.5 probability over the next 30% of
iterations. Finally, a cooldown phase reduces expert example probability to
0.1 for the subsequent 40% of iterations, fine-tuning the model’s performance
by balancing foundational and expert-driven insights.

Additionally, we regularize the heatmap processor to behave as an identity
function in scenarios where the heatmap is entirely composed of ones. We
achieve this through a priming phase that coincides with the curriculum
learning’s cold start phase. We setup an auxillary mean-squared error loss to
force the heatmap processor to reconstruct the original image $I_{i}$ when the
heatmap $E_{i}=1$. This priming ensures the heatmap processor’s adaptability,
allowing it to process expert annotations effectively when available, while
falling back to the model’s original performance in their absence. The total
loss during this phase is,
$\mathcal{L}_{\text{total}}=w_{\text{p}}\cdot\mathcal{L}_{\text{priming}}+(1-w_{\text{p}})\cdot\mathcal{L}_{\text{clip}}$,
where $w_{\text{p}}$ is a hyperparameter which we set to 0.1. The pseudocode
for eCLIP is shown in Algorithm 1.

##  4 Experiments

Our experiments are designed to evaluate the influence of expert heatmap
annotations on the quality of the learned representations. Unless stated
otherwise, we assume that a large set of image-text pairs, of which a small
fraction is annotated with eye-gaze heatmaps, is used for training, but for
test samples no annotations are used. We utilize both quantitative measures
and qualitative assessments to study the contributions of these annotations
towards enhancing model performance.

###  4.1 Setup

####  4.1.1 Baselines

To validate our approach, we compare eCLIP against a model trained using
traditional CLIP (referred to as the base model) and a “naive” baseline, where
the expert annotated samples are directly added to the training set without
using mixup or curriculum learning. We also examine the impact of two mixup
methods: Domain Agnostic Contrastive Learning (DACL) (Verma et al., 2021) and
$m^{3}$-mixup (Oh et al., 2023) which blends image and text embeddings to
improve alignment and uniformity across modalities. While DACL is integrated
during pretraining, $m^{3}$-mixup is applied post-pretraining, in a manner
akin to fine-tuning. eCLIP can be applied to any variant of CLIP, which we
demonstrate also with GLoRIA (Huang et al., 2021) which has a Resnet50 image
encoder. We introduce two variants of our technique: eCLIP, which integrates
expert annotations during the initial CLIP pretraining phase, and eCLIPP,
which instead continually finetunes a trained CLIP model with expert
annotations, similar to $m^{3}$-mixup.

####  4.1.2 Datasets

For pretraining phase we utilize the MIMIC-CXR dataset (Johnson et al.,
2019b), which pairs roughly 200K chest X-rays with free-text radiology
reports. The images were processed into JPEG format as described in (Johnson
et al., 2019a), and the accompanying reports were stripped of unnecessary
punctuation and tokenized using the Wordpiece scheme (Devlin et al., 2018). We
obtain the eye-gaze heatmap from the EGD-CXR dataset (Demner-Fushman et al.,
2016) and process the eye-tracking data to obtain the normalized eye-gaze
heatmap which are available for 1080 datapoints.

Our evaluation setup includes multiple publicly available chest X-ray
datasets, specifically CheXpert (Irvin et al., 2019), RSNA Pneumonia (Shih et
al., 2019), NIH CXR (Wang et al., 2017) and Open-I (Demner-Fushman et al.,
2016), each offering a distinct set of imaging and reporting characteristics.
Following previous works (Wang et al., 2022; Huang et al., 2021; You et al.,
2023), we prepare the test sets from MIMIC and CheXpert, selecting 200 random
samples for five specific pathologies from the CheXpert competition, resulting
in 1000 samples for each dataset (MIMIC 5x200 and CheXpert 5x200,
respectively). For the NIH-CXR dataset, we assembled a subset of 100 samples
for each of 14 abnormalities, thereby creating CXR 14x100 test set. The Open-I
dataset is utilized for text retrieval and radiology report generation tasks.
For linear probe evaluations, we use CXR-8 (Wang et al., 2017), RSNA dataset
and construct an OpenI-5 dataset by extracting labels from the ‘Problems’
field within the reports that match the CheXpert competition labels.

####  4.1.3 Training

We employed CLIP pretraining on the MIMIC-CXR dataset, utilizing a subset with
roughly 1000 images with expert eye-gaze heatmap annotations, while validation
and all other downstream evaluations proceed without these annotations. Our
model architecture includes Swin Tiny, following recent studies(Wang et al.,
2022; You et al., 2023), alongside Vision Transformer (ViT) Small and Base,
with image encoders pretrained on ImageNet and Clinical BERT (Alsentzer et
al., 2019) with max length of sequences set to 256 as the text encoder. We
cropped images to $(224,224)$ using random resized crop augmentation and
turned off all other image augmentations. Pretraining utilized 8 AMD MI250X
GPUs, maintaining an effective batch size of 512 for 10,000 steps. The
learning rate was $1e^{-4}$ for standard CLIP, increased to $2e^{-4}$ for
eCLIP variants, with cosine annealing plus a linear warmup for the first 10%
of iterations, weight decay of $1e^{-3}$, and learnable temperature parameter
in the contrastive loss initialized at 0.07. Models for $m^{3}$-mixup and
eCLIPP are initialized with weights from CLIP pretraining and further
finetuned for 1,000 iterations with a learning rate of $1e^{-5}$. Detailed
setup information is available in the Supplement.

###  4.2 Zero-shot Image Classification

Table 1: Zero-shot classification performance on 4 X-ray datasets and model configurations, reported as macro-averaged F1 scores from three independent random seeds. The highest score per dataset and model configuration is underlined. The overall best-performing model for each dataset is highlighted in bold. Model | Dataset
---|---
| Chexpert 5x200 | MIMIC 5x200 | RSNA | CXR 14x100
GLoRIA${}_{\text{Resnet50}}$ | $0.478_{\pm.023}$ | $0.457_{\pm.016}$ | $0.736_{\pm.024}$ | $0.155_{\pm.001}$
${}_{+\text{naive}}$ | $0.391_{\pm.069}$ | $0.334_{\pm.057}$ | $0.731_{\pm.023}$ | $0.113_{\pm.024}$
${}_{+\text{DACL}}$ | $0.506_{\pm.029}$ | $0.430_{\pm.011}$ | $0.736_{\pm.007}$ | $0.158_{\pm.004}$
${}_{+\text{$m^{3}$-mix}}$ | $0.512_{\pm.005}$ | $0.467_{\pm.004}$ | $0.760_{\pm.006}$ | $0.160_{\pm.003}$
${}_{+\text{expert (ours)}}$ | $0.507_{\pm.004}$ | $0.430_{\pm.009}$ | $0.761_{\pm.017}$ | $0.156_{\pm.023}$
${}_{+\text{expert}^{\mathcal{P}}\text{(ours)}}$ | $0.507_{\pm.005}$ | $0.475_{\pm.008}$ | $0.775_{\pm.001}$ | $0.159_{\pm.001}$
CLIP${}_{\text{Swin Tiny}}$ | $0.517_{\pm.025}$ | $0.452_{\pm.002}$ | $0.808_{\pm.000}$ | $0.169_{\pm.003}$
${}_{+\text{naive}}$ | $0.532_{\pm.010}$ | $0.452_{\pm.022}$ | $0.807_{\pm.007}$ | $0.167_{\pm.007}$
${}_{+\text{DACL}}$ | $0.465_{\pm.008}$ | $0.389_{\pm.015}$ | $0.768_{\pm.018}$ | $0.101_{\pm.013}$
${}_{+\text{$m^{3}$-mix}}$ | $0.554_{\pm.006}$ | $0.469_{\pm.008}$ | $0.802_{\pm.004}$ | $0.179_{\pm.008}$
${}_{+\text{expert (ours)}}$ | $0.549_{\pm.016}$ | $0.445_{\pm.021}$ | $0.818_{\pm.004}$ | $0.172_{\pm.006}$
${}_{+\text{expert}^{\mathcal{P}}\text{(ours)}}$ | $0.558_{\pm.004}$ | $0.463_{\pm.007}$ | $0.819_{\pm.000}$ | $0.192_{\pm.003}$
CLIP${}_{\text{ViT Small}}$ | $0.525_{\pm.024}$ | $0.441_{\pm.006}$ | $0.807_{\pm.006}$ | $0.159_{\pm.007}$
${}_{+\text{naive}}$ | $0.534_{\pm.016}$ | $0.440_{\pm.019}$ | $0.805_{\pm.004}$ | $0.156_{\pm.017}$
${}_{+\text{DACL}}$ | $0.475_{\pm.025}$ | $0.398_{\pm 015}$ | $0.761_{\pm.009}$ | $0.133_{\pm.007}$
${}_{+\text{$m^{3}$-mix}}$ | $0.557_{\pm.002}$ | $0.454_{\pm.003}$ | $0.809_{\pm.002}$ | $0.164_{\pm.002}$
${}_{+\text{expert (ours)}}$ | $0.545_{\pm.016}$ | $0.452_{\pm.013}$ | $0.803_{\pm.003}$ | $0.165_{\pm.007}$
${}_{+\text{expert}^{\mathcal{P}}\text{(ours)}}$ | $0.559_{\pm.001}$ | $0.439_{\pm.004}$ | $0.817_{\pm.001}$ | $0.165_{\pm.004}$
CLIP${}_{\text{ViT Base}}$ | $0.540_{\pm.017}$ | $0.465_{\pm.004}$ | $0.805_{\pm.001}$ | $0.183_{\pm.011}$
${}_{+\text{naive}}$ | $0.506_{\pm.011}$ | $0.426_{\pm.006}$ | $0.805_{\pm.004}$ | $0.151_{\pm.009}$
${}_{+\text{DACL}}$ | $0.474_{\pm.007}$ | $0.400_{\pm.002}$ | $0.759_{\pm.001}$ | $0.106_{\pm.003}$
${}_{+\text{$m^{3}$-mix}}$ | $0.542_{\pm.021}$ | $0.465_{\pm.013}$ | $0.798_{\pm.004}$ | $0.183_{\pm.020}$
${}_{+\text{expert (ours)}}$ | $0.563_{\pm.021}$ | $0.477_{\pm.004}$ | $0.814_{\pm.003}$ | $0.193_{\pm.017}$
${}_{+\text{expert}^{\mathcal{P}}\text{(ours)}}$ | $0.549_{\pm.003}$ | $0.452_{\pm.007}$ | $0.810_{\pm.001}$ | $0.185_{\pm.002}$

Following CLIP(Radford et al., 2021), our zero-shot classification method
categorizes images into predefined classes without direct finetuning, thus
relying on the quality of embeddings generated during pretraining. To
formulate the embedding of each class label, we first generate descriptive
prompts to obtain a list of text embeddings corresponding to the label using
the text encoder (Huang et al., 2021; Wang et al., 2022; You et al., 2023).
The mean of these embeddings is taken as the representation of the label. For
each image, classification is then performed by matching the image embedding
with its closest label embedding through cosine similarity. More details of
the prompts used are provided in the Supplement.

Table 1 illustrates the zero-shot classification performance on CheXpert
5x200, MIMIC 5x200, RSNA, and CXR 14x100 datasets. The results, based on
macro-averaged F1 scores from three random initializations, highlight eCLIP
variants’ superior performance over the base models across all datasets. While
the $m^{3}$-mixup excels in MIMIC for certain architectures, eCLIP variants
show broader generalization. Notably, eCLIP’s advantages are more pronounced
in multi-class scenarios (CheXpert, MIMIC and CXR14) compared to binary
classification on RSNA.

###  4.3 Sample Efficiency

Figure 4: Sample Efficiency. (top row) Zero-shot performance on three multi-
label classification test sets for CLIP and eCLIP Swin Tiny models, trained
with varying amounts of training batches. (bottom row) Linear probe scores
with varying amounts of training data.

Sample efficiency measures how well a model learns from limited amount of
training data. eCLIP improves this efficiency by using expert annotated images
to form new positive and negative pairs, aiming to improve the quality of the
learned embeddings. To test this, we first looked at zero-shot classification
performance, adjusting the number of training batches available for
pretraining. Results, shown in Fig. 4 (top row), reveal that eCLIP is more
sample efficient across MIMIC 5x200, CheXpert 5x200 and CXR 14x100 datasets
compared to the base model. Additionally, by applying supervised fine-tuning
(SFT) with a linear probe on class-imbalanced datasets – CXR-8, RSNA and
OpenI-5 (Section 4.1.2) – eCLIP demonstrates stronger performance in multi-
label classification tasks, CXR-8 and OpenI-5 and remains competitive in
binary classification for RSNA. This is shown Fig. 4 (bottom row) where we
plot the ROC AUC scores against different training sample sizes for linear
probing. These findings highlight eCLIP’s ability to effectively learn from
fewer samples.

###  4.4 Text retrieval and retrieval augmented generation (RAG)

Table 2: Performance on Image to Report retrieval with Open-I dataset. We
report the Recall@{1, 5, 10}

Model | R@1 | R@5 | R@10
---|---|---|---
CLIP${}_{\text{Swin Tiny}}$ | 3.1 | 7.6 | 11.3
${}_{+\text{$m^{3}$-mix}}$ | 2.4 | 6.5 | 9.8
${}_{+\text{expert (ours)}}$ | 3.7 | 9.4 | 13.4
${}_{+\text{expert}^{\mathcal{P}}\text{(ours)}}$ | 3.1 | 8.2 | 11.7
CLIP${}_{\text{ViT Base}}$ | 3.7 | 9.2 | 13.2
${}_{+\text{$m^{3}$-mix}}$ | 4.1 | 8.8 | 13.0
${}_{+\text{expert (ours)}}$ | 4.4 | 10.3 | 13.5
${}_{+\text{expert}^{\mathcal{P}}\text{(ours)}}$ | 2.9 | 7.5 | 10.3

Table 3: Performance on report generation with Open-I dataset. We report the
BLEU-2 score (BL-2), BERT recall score(Zhang et al., 2019) (B-R), Cosine
similarity between the sentence embeddings of the generated and ground-truth
report for MPNet (Song et al., 2020) Sentence Transformer model (Reimers &
Gurevych, 2019) (Semb), and for the CheXBERT model(Smit et al., 2020) (CBemb)

Model | BL-2 | B-R | Semb | CBemb
---|---|---|---|---
CLIP | 0.172 | 0.713 | 0.791 | 0.492
eCLIP | 0.177 | 0.712 | 0.795 | 0.506

Table 4: Ablation Study with Swin Tiny. Zero-shot performance on CheXpert (CXP) and CXR14 (C14) datasets for the base CLIP and models with expert annotation integration (+E) is presented. Methods include mask multiplication ($\odot$), CNN, and Multi-headed Attention (MHA) encoders. Key augmentations: Mixup (+M), Curriculum Learning (+C), and Encoder Priming (+P) demonstrate performance gains. A control with a random mask (rand) confirms the significance of expert annotations. We report macro-averaged F1 scores from three random initialization Method | CXP | C14
---|---|---
Base | $0.517_{\pm.024}$ | $0.169_{\pm.003}$
$\odot$ Mask (+ E) | $0.540_{\pm.019}$ | $0.165_{\pm.006}$
CNN Encoder (+ E) | $0.534_{\pm.012}$ | $0.163_{\pm.008}$
MHA Encoder (+ E) | $0.534_{\pm.013}$ | $0.153_{\pm.002}$
| MHA Encoder
---
(+ E, M)
$0.532_{\pm.018}$ | $0.160_{\pm.010}$
| MHA Encoder
---
(+ E, M, C)
$0.545_{\pm.008}$ | $0.173_{\pm.018}$
| MHA Encoder
---
(+ rand, M, C, P)
$0.537_{\pm.020}$ | $0.166_{\pm.003}$
| MHA Encoder
---
(+ E, M, C, P)
$0.549_{\pm.016}$ | $0.172_{\pm.006}$

Table 5: Random samples of generated report. For each image in the Open-I dataset, the five closest text snippets based on embedding cosine similarity is used as prompts for Mistral 7B LLM. Utilizing in-context learning, we prompt the LLM with two such snippet-report pairs. The conditions that the generated report identified correctly are highlighted in green while those it missed are shown in red. Problems |  Ground Truth |  Generated
---|---|---
normal |  The lungs and pleural spaces show no acute abnormality. Heart size and pulmonary vascularity within normal limits. |  The lungs are clear with no evidence of consolidation, pneumothorax, or pleural effusion. Heart size and mediastinal contour are normal. No acute abnormalities noted in the visualized osseous structures.
Pulmonary Atelectasis. Pleural Effusion. Cardiomegaly. Pulmonary Congestion |  AP and lateral views were obtained. Bibasilar atelectasis and small left-sided pleural effusion. Stable cardiomegaly. No pneumothorax. Mild pulmonary vascular congestion. |  The lungs show bibasilar airspace opacities, possibly due to infection or atelectasis. Heart size is mildly enlarged. No pneumothorax or pleural effusion.
Lung. Density. Emphysema |  There is hyperexpansion. The heart size is normal. There is no pleural effusion or pneumothorax. Two circular densities overlying the right ribs which were not present in the XXXX CT. No focal infiltrates |  The lungs are hyperexpanded with clear fields and no acute consolidations or pleural effusions. Heart size and pulmonary vascularity are within normal limits. Degenerative changes are present in the thoracic spine. A calcified granuloma is noted in the left perihilar region. No pneumothorax or large pleural effusion is observed.

To compare eCLIP’s cross-modal functionality with that of CLIP we focused on
text retrieval task using the Open-I dataset, which consists of pairs of
X-rays and radiology reports. We used the FAISS vector database (Douze et al.,
2024) to index the text embeddings generated by the text encoder. For a given
X-ray image $I_{i}$, we then retrieve the closest text reports from the
database based on the cosine similarity in the embedding space,
$min_{j}(v_{i}\cdot t_{j})$. Results in Table 4 compare the performance of
eCLIP against CLIP in text retrieval measured in Recall@1, 5, and 10. The
performance of eCLIP indicates a notable improvement in its embedding quality.
Note that our evaluation followed a strict criterion for recall computation,
where a retrieval was counted as successful only if the exact correct report
was identified. While more nuanced measures based on semantic similarity could
be employed (You et al., 2023), we opted this approach to maintain a clear and
simple evaluation framework.

Next we extend our analysis from retrieval to report generation using a frozen
Large Language Model (LLM), Mistral 7B Instruct v2 (Jiang et al., 2023),
aiming to generate radiology reports through Retrieval Augmented Generation
(RAG). This setup tests the CLIP model’s capacity to retrieve texts which can
be used to prompt an LLM to generate a report without finetuning on medical
data. First we randomly selected 389 samples from the Open-I dataset for
testing and utilized the FAISS database to index the reports from the
remaining samples (i.e., training set). Given a test image we retrieve five
closest reports from the training set and use them in the prompt for the LLM
to generate a report for the test image. The eCLIP variant showed a small but
consistent improvement over the base model in generating reports, as indicated
in Table 4. A comparative analysis of generated report versus ground truth
shown in Table 5, with discrepancies marked, further validates the
effectiveness of eCLIP’s embeddings in supporting complex cross-modal tasks.
Additional details, including LLM prompts and generated report samples are
available in the Supplement.

###  4.5 Embedding Quality

Figure 5: Qualitative Analysis of CLIP Pretraining. top row illustrates the
cosine similarity distributions for CLIP and eCLIP image embeddings. bottom
left and center sections display uniformity, alignment, and modality gap
comparisons among the internet pretrained model (ZS), CLIP pretrained on
MIMIC, and eCLIP. bottom right details K-means clustering metrics for image
embeddings with k=5 for both CLIP and eCLIP models.

For qualitative evaluations, we first examine the histogram of the cosine
similarities of the embeddings from different abnormality subgroups obtained
from the CLIP image encoder. In Fig. 5 (top row), we can see that the
similarities for the CLIP model has considerably dropped below 1 after
continual pretraining on MIMIC-CXR compared to Fig. 1. This indicates that the
model’s ability to distinguish between different conditions has improved. The
introduction of expert annotations in the eCLIP variant further improves this
with mean cosine similarities for ‘normal’ versus cardiomegaly, atelectasis
and opacity dropping to 0.36, 0.4 and 0.4 respectively.

Our evaluation of uniformity and alignment reveals that eCLIP surpasses both
the MIMIC-pretrained and internet-pretrained models in these key metrics,
indicating a marked improvement in the quality of embeddings (Fig. 5, bottom
left). We also note a modest decrease in the modality gap with eCLIP (Fig. 5,
bottom center). Clustering analysis via K-means (with k=5 for 5 abnormalities
in data) highlights eCLIP’s superior performance in grouping abnormalities, as
seen from improved scored in Normalized Mutual Information (NMI), Silhouette
score, and Calinski-Harabasz (CH) index (Fig. 5, bottom right).

###  4.6 Ablation Study

Our ablation study with the Swin Tiny encoder shows the impact of key
components in our eCLIP model: multi-headed attention (MHA) layer for heatmap
procesing, curriculum learning for phased introduction of expert annotations,
mixup augmentation to compensate for limited number of export annotated data
and priming of heatmap processor during initial training phase. Results shown
in Table 4 reveal that the MHA-based heatmap processor improves zero-shot
classification performance on CheXpert 5x200 and CXR 14x100 datasets compared
to basic methods like direct application of heatmaps as mask ($\odot Mask)$ or
using a CNN encoder. We note a significant performance drop with randomly
generated heatmaps versus expert eye-gaze heatmaps. This highlights that while
our methodological improvements contribute to the performance gains, the
integration of meaningful, expert-derived signals is essential for achieving
optimal results.

##  5 Conclusion

We introduce eCLIP, an adaptation of CLIP, demonstrating the integration of
radiologist eye-tracking heatmap to overcome challenges faced in multi-modal
contrastive learning. This study highlights the impact of integrating these
high-quality expert annotations on improving the quality of learned embeddings
and assess its influence on sample efficiency and cross-modal retrieval tasks.
An important future research direction would be extending this approach to
include expert annotations from the text modality (e.g., by adapting SimCSE
(Gao et al., 2021)) and to leverage the temporal dynamics of eye-tracking data
by aligning the sequential frames with the corresponding report snippets.

####  5.0.1 Limitations

Our study is limited by the small size of expert annotated data and thus does
not comprehensively analyze the impact of size or distribution of expert
annotations across different abnormalities. eCLIP also incurs extra
computational costs during training due to the additional forward pass
required for processing expert images in the warmup and cool-down phases.
Additionally, the clinical relevance of generated radiology reports has not
been validated by medical experts, relying instead on standard metrics known
for potential biases and inaccuracies in reflecting clinical accuracy(Yu et
al., 2023).

## References

  * Alsentzer et al. (2019) Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott.  Publicly available clinical BERT embeddings.  _arXiv preprint arXiv:1904.03323_ , 2019.
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston.  Curriculum learning.  In _Proceedings of the 26th annual international conference on machine learning_ , pp. 41–48, 2009.
  * Bigolin Lanfredi et al. (2022) Ricardo Bigolin Lanfredi, Mingyuan Zhang, William F Auffermann, Jessica Chan, Phuong-Anh T Duong, Vivek Srikumar, Trafton Drew, Joyce D Schroeder, and Tolga Tasdizen.  Reflacx, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays.  _Scientific data_ , 9(1):350, 2022.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pp. 1597–1607. PMLR, 2020.
  * Cherti et al. (2023) Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.  Reproducible scaling laws for contrastive language-image learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2818–2829, 2023.
  * Demner-Fushman et al. (2016) Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald.  Preparing a collection of radiology examinations for distribution and retrieval.  _Journal of the American Medical Informatics Association_ , 23(2):304–310, 2016.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Douze et al. (2024) Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou.  The faiss library.  _arXiv preprint arXiv:2401.08281_ , 2024.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen.  SimCSE: Simple contrastive learning of sentence embeddings.  _arXiv preprint arXiv:2104.08821_ , 2021.
  * (10) Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel.  M3ae: Multimodal masked autoencoders learn transferable representations.  Technical report, Technical Report.
  * Goel et al. (2022) Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover.  CyCLIP: Cyclic contrastive language-image pretraining.  _Advances in Neural Information Processing Systems_ , 35:6704–6719, 2022.
  * Gu et al. (2023) Sophia Gu, Christopher Clark, and Aniruddha Kembhavi.  I can’t believe there’s no images! learning visual tasks using only language supervision.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 2672–2683, 2023.
  * Han et al. (2022) Zongbo Han, Zhipeng Liang, Fan Yang, Liu Liu, Lanqing Li, Yatao Bian, Peilin Zhao, Bingzhe Wu, Changqing Zhang, and Jianhua Yao.  Umix: Improving importance weighting for subpopulation shift via uncertainty-aware mixup.  _Advances in Neural Information Processing Systems_ , 35:37704–37718, 2022.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 16000–16009, 2022.
  * Huang et al. (2021) Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung.  Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 3942–3951, 2021.
  * Huang et al. (2023) Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou.  A visual–language foundation model for pathology image analysis using medical twitter.  _Nature medicine_ , 29(9):2307–2316, 2023.
  * Irvin et al. (2019) Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al.  CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pp. 590–597, 2019.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , pp. 4904–4916. PMLR, 2021.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7B.  _arXiv preprint arXiv:2310.06825_ , 2023.
  * Johnson et al. (2019a) Alistair Johnson, Matt Lungren, Yifan Peng, Zhiyong Lu, Roger Mark, Seth Berkowitz, and Steven Horng.  MIMIC-CXR-JPG-Chest radiographs with structured labels.  _PhysioNet_ , 2019a.
  * Johnson et al. (2019b) Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng.  MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports.  _Scientific data_ , 6(1):317, 2019b.
  * Karargyris et al. (2021) Alexandros Karargyris, Satyananda Kashyap, Ismini Lourentzou, Joy T Wu, Arjun Sharma, Matthew Tong, Shafiq Abedin, David Beymer, Vandana Mukherjee, Elizabeth A Krupinski, et al.  Creation and validation of a chest X-ray dataset with eye-tracking and report dictation for AI development.  _Scientific Data_ , 8(1):92, 2021.
  * Krishnan et al. (2022) Rayan Krishnan, Pranav Rajpurkar, and Eric J Topol.  Self-supervised learning in medicine and healthcare.  _Nature Biomedical Engineering_ , 6(12):1346–1352, 2022.
  * Li et al. (2023) Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He.  Scaling language-image pre-training via masking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 23390–23400, 2023.
  * Liang et al. (2022) Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou.  Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.  _Advances in Neural Information Processing Systems_ , 35:17612–17625, 2022.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023.
  * McInnes et al. (2018) Leland McInnes, John Healy, and James Melville.  UMAP: Uniform manifold approximation and projection for dimension reduction.  _arXiv preprint arXiv:1802.03426_ , 2018.
  * Mo et al. (2024) Sangwoo Mo, Minkyu Kim, Kyungmin Lee, and Jinwoo Shin.  S-clip: Semi-supervised vision-language learning using few specialist captions.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Moor et al. (2023) Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar.  Med-flamingo: a multimodal medical few-shot learner.  In _Machine Learning for Health (ML4H)_ , pp. 353–367. PMLR, 2023.
  * Mu et al. (2022) Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  Slip: Self-supervision meets language-image pre-training.  In _European Conference on Computer Vision_ , pp. 529–544. Springer, 2022.
  * Naeem et al. (2023) Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Luc Van Gool, and Federico Tombari.  SILC: Improving vision language pretraining with self-distillation.  _arXiv preprint arXiv:2310.13355_ , 2023.
  * Nukrai et al. (2022) David Nukrai, Ron Mokady, and Amir Globerson.  Text-only training for image captioning using noise-injected clip.  _arXiv preprint arXiv:2211.00575_ , 2022.
  * Oh et al. (2023) Changdae Oh, Junhyuk So, Hoyoon Byun, YongTaek Lim, Minchul Shin, Jong-June Jeon, and Kyungwoo Song.  Geodesic multi-modal mixup for robust fine-tuning.  _Advances in Neural Information Processing Systems_ , 36, 2023.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Palepu & Beam (2023) Anil Palepu and Andrew Beam.  Tier: Text-image entropy regularization for medical clip-style models.  In _Machine Learning for Healthcare Conference_ , pp. 548–564. PMLR, 2023.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pp. 8821–8831. PMLR, 2021.
  * Reimers & Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-BERT: Sentence embeddings using siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 3982–3992, 2019.
  * Shih et al. (2019) George Shih, Carol C Wu, Safwan S Halabi, Marc D Kohli, Luciano M Prevedello, Tessa S Cook, Arjun Sharma, Judith K Amorosa, Veronica Arteaga, Maya Galperin-Aizenberg, et al.  Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia.  _Radiology: Artificial Intelligence_ , 1(1):e180041, 2019.
  * Singh et al. (2022) Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.  Flava: A foundational language and vision alignment model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15638–15650, 2022.
  * Smit et al. (2020) Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew Lungren.  Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 1500–1519, 2020.
  * Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu.  MPNet: Masked and permuted pre-training for language understanding.  _Advances in Neural Information Processing Systems_ , 33:16857–16867, 2020.
  * Sowrirajan et al. (2010) Hari Sowrirajan, Jingbo Yang, Andrew Y Ng, and Pranav Rajpurkar.  MoCo-CXR: MoCo pretraining improves representation and transferability of chest X-ray models, 2021.  _URL https://arxiv. org/abs_ , 2010.
  * Sun et al. (2023) Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang.  Alpha-clip: A clip model focusing on wherever you want.  _arXiv preprint arXiv:2312.03818_ , 2023.
  * Tschannen et al. (2023) Michael Tschannen, Basil Mustafa, and Neil Houlsby.  CLIPPO: Image-and-language understanding from pixels only.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 11006–11017, 2023.
  * Tu et al. (2024) Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al.  Towards generalist biomedical ai.  _NEJM AI_ , 1(3):AIoa2300138, 2024.
  * van Sonsbeek et al. (2023) Tom van Sonsbeek, Xiantong Zhen, Dwarikanath Mahapatra, and Marcel Worring.  Probabilistic integration of object level annotations in chest x-ray classification.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 3630–3640, 2023.
  * Verma et al. (2019) Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio.  Manifold mixup: Better representations by interpolating hidden states.  In _International conference on machine learning_ , pp. 6438–6447. PMLR, 2019.
  * Verma et al. (2021) Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc Le.  Towards domain-agnostic contrastive learning.  In _International Conference on Machine Learning_ , pp. 10530–10541. PMLR, 2021.
  * Wang et al. (2024) Bin Wang, Hongyi Pan, Armstrong Aboah, Zheyuan Zhang, Elif Keles, Drew Torigian, Baris Turkbey, Elizabeth Krupinski, Jayaram Udupa, and Ulas Bagci.  Gazegnn: A gaze-guided graph neural network for chest x-ray classification.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 2194–2203, 2024.
  * Wang & Liu (2021) Feng Wang and Huaping Liu.  Understanding the behaviour of contrastive loss.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 2495–2504, 2021.
  * Wang & Isola (2020) Tongzhou Wang and Phillip Isola.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  In _International Conference on Machine Learning_ , pp. 9929–9939. PMLR, 2020.
  * Wang et al. (2017) Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers.  ChestX-Ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2097–2106, 2017.
  * Wang et al. (2022) Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun.  MedCLIP: Contrastive learning from unpaired medical images and text.  _arXiv preprint arXiv:2210.10163_ , 2022.
  * Weers et al. (2023) Floris Weers, Vaishaal Shankar, Angelos Katharopoulos, Yinfei Yang, and Tom Gunter.  Masked autoencoding does not help natural language supervision at scale.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 23432–23444, 2023.
  * Wu et al. (2023) Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie.  Medklip: Medical knowledge enhanced language-image pre-training.  _medRxiv_ , pp. 2023–01, 2023.
  * Xu et al. (2023a) Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer.  Demystifying clip data.  _arXiv preprint arXiv:2309.16671_ , 2023a.
  * Xu et al. (2023b) Shawn Xu, Lin Yang, Christopher Kelly, Marcin Sieniek, Timo Kohlberger, Martin Ma, Wei-Hung Weng, Attila Kiraly, Sahar Kazemzadeh, Zakkai Melamed, et al.  Elixr: Towards a general purpose x-ray artificial intelligence system through alignment of large language models and radiology vision encoders.  _arXiv preprint arXiv:2308.01317_ , 2023b.
  * You et al. (2023) Kihyun You, Jawook Gu, Jiyeon Ham, Beomhee Park, Jiho Kim, Eun K Hong, Woonhyuk Baek, and Byungseok Roh.  CXR-CLIP: Toward large scale chest X-ray language-image pre-training.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pp. 101–111. Springer, 2023.
  * Yu et al. (2023) Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururahy Nunes Fonseca, Henrique Min Ho Lee, Zahra Shakeri Hossein Abad, Andrew Y Ng, et al.  Evaluating progress in automatic chest x-ray radiology report generation.  _Patterns_ , 4(9), 2023.
  * Zbontar et al. (2021) Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In _International Conference on Machine Learning_ , pp. 12310–12320. PMLR, 2021.
  * Zhang et al. (2017) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  _arXiv preprint arXiv:1710.09412_ , 2017.
  * Zhang et al. (2023a) Ke Zhang, Yan Yang, Jun Yu, Hanliang Jiang, Jianping Fan, Qingming Huang, and Weidong Han.  Multi-task paired masking with alignment modeling for medical vision-language pre-training.  _IEEE Transactions on Multimedia_ , 2023a.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  BERTScore: Evaluating text generation with BERT.  In _International Conference on Learning Representations_ , 2019.
  * Zhang et al. (2022) Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz.  Contrastive learning of medical visual representations from paired images and text.  In _Machine Learning for Healthcare Conference_ , pp. 2–25. PMLR, 2022.
  * Zhang et al. (2023b) Yuhui Zhang, Jeff Z HaoChen, Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung.  Diagnosing and rectifying vision models using language.  _arXiv preprint arXiv:2302.04269_ , 2023b.
  * Zhang et al. (2024) Yuhui Zhang, Elaine Sui, and Serena Yeung.  Connect, Collapse, Corrupt: Learning cross-modal tasks with uni-modal data.  In _The Twelfth International Conference on Learning Representations_ , 2024.