# On the infinitary van der Waerden Theorem

Shahram Mohsenipour  School of Mathematics, Institute for Research in
Fundamental Sciences (IPM) P. O. Box 19395-5746, Tehran, Iran
<EMAIL_ADDRESS>
###### Abstract.

We give a purely combinatorial proof for the infinitary van der Waerden
theorem.

###### Key words and phrases:

Infinitary van der Waerden’s theorem

###### 2010 Mathematics Subject Classification:

05D10

##  1\. Introduction

In [1], we introduced the following infinitary generalization of van der
Waerden’s theorem which has the same relation with van der Waerden’s theorem
as the Carlson-Simpson theorem [3] has with the Halse-Jewett theorem [5].

Let $k\geq 2$ be a positive integer, then for any finite coloring of
$\mathbb{N}$, there exist positive integers $d_{1},d_{2},\dots,d_{n},\dots$
and a color $\gamma$ such that for every $n\in\mathbb{N}$, there exists a
positive integer $a_{n}$ such that the set

$\big{\\{}a_{n}+j_{1}d_{1}+\cdots+j_{n}d_{n}|0\leq j_{1},\dots,j_{n}\leq
k-1\big{\\}}$

is monochromatic with the color $\gamma$.

It is possible to deduce the above theorem from the Carlson-Simpson theorem by
generalizing the usual proof of deducing van der Waerden’s theorem from the
Hales-Jewett theorem. The proof given in [1] uses a more elementary fact on
uniformly recurrent infinite words from [6] which has its root in symbolic
dynamics. It is worth noting that the above theorem can also be easily deduced
from the Furstenberg-Weiss Theorem ([4], Theorem 1.5) by a successive
iteration. In this note we present a direct combinatorial proof.

Now we explain some notation and conventions that we will use in our proof
presented in the next section. By interval, we mean an interval of positive
integers which is a set of the form $I=\\{a,a+1,a+2,\dots,b\\}$ for some
positive integers $a\leq b$. We also denote $I$ by $[a,b]$, and for $1\leq
i\leq b-a+1$, by $I[i]$ we mean the $i$th element of the interval $[a,b]$,
namely, $a+i-1$. The number of elements of $I$ is denoted by $|I|$. If $b$ is
a non-negative integer and $A$ is a set of integers, then $\mathbb{T}^{b}A$
will denote the set $\\{a+b|a\in A\\}$. Let $c$ be a positive integer, by a
$c$-coloring, we mean a coloring with $c$ many colors and furthermore the set
of colors is $[c]=\\{1,\dots,c\\}$. For positive integers $k\geq 2,c\geq 1$,
$W(k,c)$ will denote the van der Waerden number which is the least integer $n$
such that for any $c$-coloring of $[n]$, there exists a monochromatic
arithmetic progression of length $k$.

##  2\. Main Result

Fix positive integers $k\geq 2,c\geq 1$. We inductively define sequences of
positive integers $\langle W_{n}:n\in\mathbb{N}\rangle$ and $\langle
c_{n}:n\in\mathbb{N}\rangle$ (depending on $k,c$) as follows. Put
$W_{1}=W(k,c),W_{n+1}=W(k,c_{n})$ and $c_{n}=c^{W_{n}\cdots W_{1}}$. Now let
$I$ be an interval with $|I|=W_{1}$. Now from $I$ we inductively define a
sequence of intervals $I_{n},n\in\mathbb{N}$ as follows. Let $I_{1}=I$ and

$I_{n+1}=I_{n}\cup\mathbb{T}^{W_{n}\cdots
W_{1}}I_{n}\cup\mathbb{T}^{2W_{n}\cdots
W_{1}}I_{n}\cup\cdots\cup\mathbb{T}^{(W_{n+1}-1)W_{n}\cdots W_{1}}I_{n}.$

For instance for $n=2$, we have

$I_{2}=I_{1}\cup\mathbb{T}^{W_{1}}I_{1}\cup\mathbb{T}^{2W_{1}}I_{1}\cup\cdots\cup\mathbb{T}^{(W_{2}-1)W_{1}}I_{1},$

which according to $|I_{1}|=W_{1}$, it is easily seen that $I_{2}$ is an
interval with $|I_{2}|=W_{2}W_{1}$. Similarly for each $n\in\mathbb{N}$, we
have that $I_{n}$ is an interval with $|I_{n}|=W_{n}\cdots W_{1}$. Obviously
in the above definition, $I_{n}$ depends on $k,c,n,I$, so we denote it by
$I_{n}=\mathcal{I}(I,n,k,c)$. If $k,c$ were fixed and were clear from the
context, then we may denote it by $I_{n}=\mathcal{I}(I,n)$. We will also use
the easy fact that for any positive integer $b$, the interval
$\mathbb{T}^{b}\mathcal{I}(I,n,k,c)$ can be written in the form of
$\mathcal{I}(J,n,k,c)$ for some interval $J$ with $|J|=W_{1}$. In fact due to
the additive nature of the definition, we have
$\mathbb{T}^{b}\mathcal{I}(I,n,k,c)=\mathcal{I}(\mathbb{T}^{b}I,n,k,c)$.

######  Proposition 2.1.

Let $k\geq 2,c\geq 1,n\geq 1$ be positive integers and let $I$ be an interval
with $|I|=W_{1}$. Then for any $c$-coloring of $\mathcal{I}(I,n)$, there exist
$a,d_{1},\dots,d_{n}$ such that $d_{k}\leq W_{k}$ for $k=1,\dots,n$ and the
following set

$\big{\\{}a+j_{1}d_{1}+\cdots+j_{n}d_{n}|0\leq j_{1},\dots,j_{n}\leq
k-1\big{\\}}$

is included in $\mathcal{I}(I,n)$ and is monochromatic.

###### Proof.

The proof is by induction on $n$. The case $n=1$ is just van der Waerden’s
theorem and $d_{1}\leq W_{1}$ is obvious. Now assume that the assertion is
true for $n$, we prove it for $n+1$. Let
$\chi\colon\mathcal{I}(I,n+1)\rightarrow[c]$ is a $c$-coloring. By definition
we have

$\mathcal{I}(I,n+1)=\bigcup_{i=0}^{W_{n+1}-1}\mathbb{T}^{iW_{n}\cdots
W_{1}}\mathcal{I}(I,n).$

For simplicity we denote the interval $\mathbb{T}^{iW_{n}\cdots
W_{1}}\mathcal{I}(I,n)$ by $J_{i}$ (Recall that $|J_{i}|=W_{n}\cdots W_{1}$).
Now we define a coloring $\chi^{*}$ on the set $\\{0,1,\dots,W_{n+1}-1\\}$ as
follows. For $0\leq i<j<W_{n+1}$ we put $\chi^{*}(i)=\chi^{*}(j)$ if for every
$1\leq t\leq W_{n}\cdots W_{1}$ we have $\chi(J_{i}[t])=\chi(J_{j}[t])$. Since
$\chi$ is a $c$-coloring, the number of colors for $\chi^{*}$ is
$c^{W_{n}\cdots W_{1}}=c_{n}$. Thus from $W_{n+1}=W(k,c_{n})$ it follows that
there exist $0\leq b_{1}<b_{2}<\cdots<b_{k}\leq W_{n+1}-1$ with

$b_{2}-b_{1}=b_{3}-b_{2}=\cdots=b_{k}-b_{k-1}=:d^{*}\leq W_{n+1}$

such that $\chi^{*}(b_{1})=\chi^{*}(b_{2})=\cdots=\chi^{*}(b_{k})$. Having

$J_{b_{1}}=\mathbb{T}^{b_{1}W_{n}\cdots
W_{1}}\mathcal{I}(I,n)=\mathcal{I}(\mathbb{T}^{b_{1}W_{n}\cdots W_{1}}I,n),$

we can use the induction hypothesis for the interval
$\mathbb{T}^{b_{1}W_{n}\cdots W_{1}}I$ and the restriction of the $c$-coloring
$\chi$ to this interval and thus obtain positive integers
$a,d_{1},\dots,d_{n}$ and a color $\gamma\in[c]$ such that the set

$\big{\\{}a+j_{1}d_{1}+\cdots+j_{n}d_{n}|0\leq j_{1},\dots,j_{n}\leq
k-1\big{\\}}$

is included in $J_{b_{1}}$ and is monochromatic with the color $\gamma$.
Moreover we have $d_{1}\leq W_{1},\dots,d_{n}\leq W_{n}$. Now according to
what we have obtained it is easily seen that for $j=0,\dots,k-1$ we have
$\mathbb{T}^{jd^{*}}J_{b_{1}}=J_{b_{1+j}}$. So for $1\leq i\leq W_{n}\cdots
W_{1}$ and $j=0,\dots,k-1$ we get the relation
$J_{b_{1+j}}[i]=J_{b_{1}}[i]+jd^{*}$. Now let $a^{*}\in J_{b_{1}}$, then there
is $1\leq i^{*}\leq W_{n}\cdots W_{1}$ such that $a^{*}=J_{b_{1}}[i^{*}]$.
Therefore for $j=0,\dots,k-1$, we have

$\chi(a^{*})=\chi(J_{b_{1}}[i^{*}])=\chi(J_{b_{1+j}}[i^{*}])=\chi(J_{b_{1}}[i^{*}]+jd^{*})=\chi(a^{*}+jd^{*}).$

Putting this together with the induction hypothesis, we infer that the set

$\big{\\{}a+j_{1}d_{1}+\cdots+j_{n}d_{n}+j_{n+1}d^{*}|0\leq
j_{1},\dots,j_{n},j_{n+1}\leq k-1\big{\\}}$

is monochromatic with the color $\gamma$ too. Now set $d_{n+1}=d^{*}$. Also
recall that $d^{*}\leq W_{n+1}$ and observe that the above set is included in
$J_{b_{1}}\cup\cdots\cup J_{b_{k}}$ which is included in $\mathcal{I}(I,n+1)$.
So the assertion is proved for $n+1$ and the proof is complete. ∎

######  Theorem 2.2.

Let $k\geq 2,c\geq 1$ be positive integers, then for any $c$-coloring of
$\mathbb{N}$, there exist positive integers $d_{1},d_{2},\dots,d_{n},\dots$
and the color $\gamma\in[c]$ such that for every $n\in\mathbb{N}$, there
exists a positive integer $a_{n}$ such that the set

$\big{\\{}a_{n}+j_{1}d_{1}+\cdots+j_{n}d_{n}|0\leq j_{1},\dots,j_{n}\leq
k-1\big{\\}}$

is monochromatic with the color $\gamma$.

###### Proof.

We fix $k\geq 2,c\geq 1$ and a coloring $\chi\colon\mathbb{N}\rightarrow[c]$.
We inductively define sequences of intervals $\langle
J_{m};m\in\mathbb{N}\rangle$ and $\langle J_{m}^{*};m\in\mathbb{N}\rangle$ as
follows. Let $J_{1}=[1,W_{1}]$ and put $J_{m}^{*}=\big{[}1+\max
J_{m},W_{1}+\max J_{m}\big{]}$, $J_{m+1}=\mathcal{I}(J_{m}^{*},m)$. According
to Proposition 2.1, for every $m\in\mathbb{N}$, there exist positive integers
$e_{m},l_{m1},\dots,l_{mm}$ such that the set

$A_{m}:=\big{\\{}e_{m}+j_{1}l_{m1}+\cdots+j_{m}l_{mm}|0\leq
j_{1},\dots,j_{m}\leq k-1\big{\\}}\subset J_{m}$

is monochromatic. Moreover for every $m\in\mathbb{N}$ we have

$l_{m1}\leq W_{1},\dots,l_{mm}\leq W_{m}.$

Since the set of colors is finite, there exist $\gamma\in[c]$ and an infinite
subset $S_{0}\subset\mathbb{N}$ such that for each $m\in S_{0}$, the set
$A_{m}$ is monochromatic with the color $\gamma$. Let
$S_{0}=\\{m_{i}|i\in\mathbb{N}\\}$ with $m_{i}<m_{i+1}$, then from the
pigeonhole principle and

$l_{m_{1}1},l_{m_{2}1},l_{m_{3}1},\ldots\leq W_{1}$

it follows that there exists an infinite subset $S_{1}\subset S_{0}$ such that
for every $t_{1},t_{2}\in S_{1}$, we have $l_{t_{1}1}=l_{t_{2}1}$. Similarly,
by iterative use of the pigeonhole principle we obtain an infinite sequence of
subsets

$S_{0}\supset S_{1}\supset\cdots\supset S_{n}\supset\cdots$

such that each $S_{n}$ is infinite and we have that if $t_{1},t_{2}\in S_{n}$,
then $l_{t_{1}n}=l_{t_{2}n}$. For $n\in\mathbb{N}$, let $s_{n}=\min S_{n}$ and
set $a_{n}=e_{s_{n}},d_{n}=l_{s_{n}n}$. Now we verify that the set

$\big{\\{}a_{n}+j_{1}d_{1}+\cdots+j_{n}d_{n}|0\leq j_{1},\dots,j_{n}\leq
k-1\big{\\}}$

is monochromatic with the color $\gamma$. To see this, first observe that

$a_{n}+j_{1}d_{1}+\cdots+j_{n}d_{n}=e_{s_{n}}+j_{1}l_{s_{1}1}+\cdots+j_{n}l_{s_{n}n}.$

As $s_{n}\in S_{0}$, we have
$e_{s_{n}}+j_{1}l_{s_{n}1}+\cdots+j_{n}l_{s_{n}n}\in A_{s_{n}}$ and therefore

$\chi(e_{s_{n}}+j_{1}l_{s_{n}1}+\cdots+j_{n}l_{s_{n}n})=\gamma.$

Also from $s_{n}\in S_{1},\dots,s_{n}\in S_{n-1}$, we get

$l_{s_{n}1}=l_{s_{1}1},\dots,l_{s_{n}(n-1)}=l_{s_{n-1}(n-1)},$

which implies that

$\displaystyle\chi(a_{n}+j_{1}d_{1}+\cdots+j_{n}d_{n})$ $\displaystyle=$
$\displaystyle\chi(e_{s_{n}}+j_{1}l_{s_{1}1}+\cdots+j_{n}l_{s_{n}n})$
$\displaystyle=$
$\displaystyle\chi(e_{s_{n}}+j_{1}l_{s_{n}1}+\cdots+j_{n}l_{s_{n}n})=\gamma.$

This finishes the proof. ∎

By an easy modification of the arguments, we actually can prove a stronger
theorem which is correspondent to Carlson’s generalization [2] of the Carlson-
Simpson theorem. See also Theorem 3 of [1].

######  Theorem 2.3.

Let $c\geq 1$ be a positive integer. Let $k_{1}\leq k_{2}\leq\cdots\leq
k_{n}\leq\cdots$ be a sequence of positive integers $\geq 2$, then for any
$c$-coloring of $\mathbb{N}$, there exist positive integers
$d_{1},d_{2},\dots,d_{n},\dots$ and the color $\gamma\in[c]$ such that for
every $n\in\mathbb{N}$, there exists a positive integer $a_{n}$ such that the
set

$\big{\\{}a_{n}+j_{1}d_{1}+\cdots+j_{n}d_{n}|0\leq j_{1}\leq
k_{1}-1,\dots,0\leq j_{n}\leq k_{n}-1\big{\\}}$

is monochromatic with the color $\gamma$.

### Acknowledgment

The research of the author was in part supported by a grant from IPM (No.
00030403).

## References

  * [1] Thomas Brown and Shahram Mohsenipour, _Two extensions of hilbert’s cube lemma_ , To appear in Integers.
  * [2] Timothy J. Carlson, _Some unifying principles in ramsey theory_ , Discrte Math. 68 (1988), 117–169.
  * [3] Timothy J. Carlson and Stephen G. Simpson, _A dual form of ramsey’s theorem_ , Adv. in Math. 53 (1984), 265–290.
  * [4] Hillel Furstenberg and Benjamin Weiss, _Topological dynamics and combinatorial number theory_ , J. Analyse Math. 49 (1979), no. 4, 61–85.
  * [5] A. W. Hales and R. I. Jewett, _Regularity and positional games_ , Trans. Amer. Math. Soc. 106 (1963), 222–229.
  * [6] J. Justin and G. Pirillo, _Shirshov’s theorem and $\omega$-permutability of semigroups_, Adv. Math. 87 (1991), 151–159.


# PRIOR: Prototype Representation Joint Learning
from Medical Images and Reports

Pujin Cheng 1,2, Li Lin 1,3, Junyan Lyu1,4, Yijin Huang 1,5,
Wenhan Luo 6, Xiaoying Tang 1,2
1Department of Electronic and Electrical Engineering, Southern University of
Science and Technology
2Jiaxing Research Institute, Southern University of Science and Technology
3Department of Electrical and Electronic Engineering, The University of Hong
Kong
4Queensland Brain Institute, The Univeristy of Queensland
5School of Biomedical Engineering, University of British Columbia
6Shenzhen Campus of Sun Yat-sen University

###### Abstract

Contrastive learning based vision-language joint pre-training has emerged as a
successful representation learning strategy. In this paper, we present a
prototype representation learning framework incorporating both global and
local alignment between medical images and reports. In contrast to standard
global multi-modality alignment methods, we employ a local alignment module
for fine-grained representation. Furthermore, a cross-modality conditional
reconstruction module is designed to interchange information across modalities
in the training phase by reconstructing masked images and reports. For
reconstructing long reports, a sentence-wise prototype memory bank is
constructed, enabling the network to focus on low-level localized visual and
high-level clinical linguistic features. Additionally, a non-auto-regressive
generation paradigm is proposed for reconstructing non-sequential reports.
Experimental results on five downstream tasks, including supervised
classification, zero-shot classification, image-to-text retrieval, semantic
segmentation, and object detection, show the proposed method outperforms other
state-of-the-art methods across multiple datasets and under different dataset
size settings. The code is available at https://github.com/QtacierP/PRIOR.

Figure 1: Illustration of the proposed sentence-wise prototype memory bank.
The prototype embedding can group sentences sharing similar information. Each
sentence representation is updated to the nearest prototype after querying.

##  1 Introduction

Powered by large-scale labeled natural image datasets, deep learning has
achieved great success in computer vision [16, 10, 42, 43, 53]. However,
annotating medical images is extremely expensive and labor-intensive [3, 46].
A practical approach is to first pre-train a model on a large-scale labeled
natural image dataset like ImageNet [9], and then fine-tune it on the
downstream medical image dataset with limited annotation [40, 38, 60]. This
approach may nevertheless fail to achieve generalized performance due to the
domain gap between natural images and medical images [45, 34]. To effectively
inherit representation from images of the same domain, self-supervised
learning (SSL) methods [4, 15, 5, 7] have been proposed through pre-training
on unlabeled datasets. However, it has been suggested that the performance
gain of pre-training on unlabeled medical images is relatively limited
compared to ImageNet initialization [59]. There are potentially two reasons:
1) The sample size of medical images, even unlabeled, is still quite limited
compared to that of the ImageNet dataset; 2) Medical images often exhibit high
inter-class similarity.

Recently, vision-language pre-training (VLP) has shown natural language
supervision can effectively transfer linguistic information to visual
representation via well-designed proxy tasks such as contrastive learning [39,
27, 57, 32, 26] and generative reconstruction [30, 58]. VLP may particularly
work for medical image analysis since medical reports are highly likely to be
accessible in most situations. However, jointly pre-training medical images
and reports is still challenging. First, there are typically multiple
sentences in a medical report, and the textual information is highly complex
[56, 28, 25]. Second, most descriptions in a medical report are exclusively
related to specific sub-regions in the corresponding medical image [33]. Most
existing VLP methods tend to ignore fine-grained representation and may fail
to transfer to locality-aware downstream tasks such as semantic segmentation
and object detection. Several methods have employed local alignment losses
through contrastive tasks [17, 33, 52]. For example, Huang _et al_. design a
localized feature representation framework, but it still falls into the global
alignment category [17]. Wang _et al_. propose a local contrastive loss to
align locality-aware information [52], but a vanilla contrastive loss may
easily ignore the similarity among nearby sub-regions due to their spatial
locations and the overlapped sliding windows in convolution. To address this
issue, Muller _et al_. employ positiveness probability sampling to avoid
selecting a nearby sub-region as the negative sample [33]. However, this
strategy is computationally heavy and time-consuming. Moreover, since
contrastive learning is a discriminative SSL paradigm mainly focusing on high-
level features, all those aforementioned methods tend to overlook low-level
features, such as lesion boundaries in images and symptom descriptions in
reports, which are nevertheless highly crucial for downstream medical image
analysis tasks.

In this paper, we present a Prototype Representation framework via joint
global and local alignment between medical Images and repORts (PRIOR), wherein
we effectively combine contrastive learning and cross-modality conditional
reconstruction. We consider sentence in reports and sub-region in images as
the elementary local representation units, and the global representation is
obtained via attention pooling over localized features. We propose a cross-
modality alignment module to align representation between images and reports
from both global and local views. To further learn locality-aware and fine-
grained information, we utilize an encoder-decoder architecture to maximize
the conditional mutual information between paired images and reports. The
reconstruction decoder aims to reconstruct the masked image given the report
and generate the report’s representation given the image. We make use of the
prior information that descriptions of most medical reports essentially can be
summarized by multiple structured labels [19]. That is, sentence-level feature
representation can be approximated by prototype categorization without any
need to accurately retain redundant information such as syntax. As shown in
Figure 1, each sentence in a report is discretely embedded as prototype
representation. In this way, the sentence-level representation learning
process can be treated as a classification-like task. Different from the image
caption task which predicts each word auto-regressively [54, 29], sentences in
a medical report are usually non-sequential. Inspired by [2], we use a
parallel decoder to reconstruct sentence prototype embedding via bipartite
matching. We successfully demonstrate the effectiveness of PRIOR on three
challenging datasets for five downstream tasks, including supervised
classification, zero-shot classification, image-to-text retrieval, semantic
segmentation, and object detection.

Our main contributions are four-fold:

  * •

We propose a novel cross-modality alignment module utilizing both global and
local information to capture fine-grained features. Compared with previous
works, our proposed alignment is more effectively operated on local
representation.

  * •

Considering the writing paradigm of a medical report, we present a prototype
memory bank for report’s sentence embeddings. Such discrete representation
guides the linguistic features to further focus on high-level representation
that links tightly to medical images.

  * •

We leverage a conditional reconstruction task for both vision and language
representation, which further facilitates cross-modality feature interaction
and explores more structural and causal representation.

  * •

PRIOR outperforms existing state-of-the-art (SOTA) methods on five tasks,
including supervised classification, zero-shot classification, image-to-text
retrieval, semantic segmentation, and object detection.

Figure 2: The overall framework of the proposed PRIOR. Given a pair of medical
image and report, two independent encoders first encode each modality into a
common embedding space. Then, the cross-modality alignment module aligns both
global and local information between the two modalities. Finally, the cross-
modality conditional reconstruction module reconstructs the masked image given
the report and generates the sentence prototypes given the image.

##  2 Related Work

Self-Supervised Learning. Self-Supervised Learning (SSL) addresses the dilemma
of limited annotated data. There are two main categories: (1) Contrastive SSL
methods make use of the invariability of the same image under different
augmented views. SimCLR [4] and MoCo [15, 5] attempt to maintain the
invariability by spreading the representation of different images (negative
samples) apart and clustering the representation of the same image (positive
sample) together. Recent works have shown specific metric learning frameworks
without any negative sample can outperform contrastive learning. Chen _et al_.
propose a simple siamese representation learning framework with low memory and
computation cost [6]. Several other works also identify that SSL with no image
discrimination could obtain comparable performance to contrastive SSL [13].
(2) Generative SSL methods aim to learn data distribution with latent space
embedding. Many of them rely on well-designed proxy tasks, such as
colorization [24], image reconstruction [36, 47], inpainting [37], and so on.
Recently, He _et al_. present a simple masked image auto-encoder with SOTA
performance by using vision transformers [14].

Vision-Language Pre-training. Vision-Language Pre-training (VLP) is a specific
type of SSL, wherein the self-supervised task usually relies on cross-modality
interactions between visual and linguistic information. One of the most
representative methods is CLIP [39]. It shows that contrastive learning can
transfer rich semantic information from language supervision to visual
representation [12, 8]. However, CLIP only learns coarse-grained
representation and ignores localized information. Recently, many works have
tried to build fine-grained relationships between natural images and text
captions. For example, GLIP converts object detection into a phrase grounding
task [27]. Yao _et al_. propose a VLP framework named FLIP involving fine-
grained interactions, leveraging localized representation between visual and
textual tokens [57]. However, none of them are specifically designed for pre-
training medical images and reports. Furthermore, GLIP needs annotated
bounding boxes for local alignment. And FLIP only focuses on the local
alignment between coarse-grained sub-regions and words, which tends to ignore
low-level information like textual structure in images and sentence-wise
relation in reports. To capture low-level representation, CoCa adopts an
encoder-decoder architecture for both contrastive learning and conditional
generation [58], but it still cannot deal with the complex structures in
medical reports and the low-level information in medical images.

Medical Image and Report Pre-training. In recent years, models that jointly
pre-train vision and language for medical utility have been explored. ConVIRT
performs global alignment through a contrastive loss [59]. On top of ConVIRT,
GLoRIA [17], LoVT [33] and MGCA [52] all propose their own local alignment
mechanisms. GLoRIA and MGCA consider each keyword in a report as the
elementary local representation unit, but they do not consider any context
across sentences. LoVT aligns sub-regions at the sentence level. However, it
still abandons low-level features such as the shape of lesions and detailed
symptom descriptions. Inspired by CoCa, we address the issues of existing
medical image and report pre-training methods by introducing a conditional
reconstruction task combined with prototype representation learning.

Prototype Representation Learning. The goal of prototype representation
learning is to cluster similar units into a single embedding. Van _et al_.
show that images could be represented discretely [48]. And Ramesh _et al_.
prove that prototype learning works well on cross-modality interaction tasks
[41]. Here, we apply prototype learning to medical reports’ sentence
representation, converting a continuous sentence embedding space into a
categorical distribution.

##  3 Methodology

VLP aims to learn a joint distribution $P(X_{I},X_{R})$ over a group of
medical image $X_{I}=\\{x_{I}^{1},...,x_{I}^{N}\\}$ and report
$X_{R}=\\{x_{R}^{1},...,x_{R}^{N}\\}$ pairs. Each sample $x^{i}$ consists of
an image $x_{I}^{i}$ and the corresponding report $x_{R}^{i}$.

###  3.1 Framework Overview

The overall framework is shown in Figure 2. Given a pair of an image
$x_{I}^{i}$ and a report $x_{R}^{i}$, we first input them into two separate
encoders. Then, each sentence embedding is updated by a learnable sentence-
wise prototype memory bank (SPB) for the final linguistic representation. In
addition to standard global contrastive learning, a local alignment module
(LAM) is introduced to align local representation between sub-regions and
sentences, aided by a cross-modality attention mechanism. Unlike the common
Softmax based attention module, our proposed Sigmoid based LAM makes use of
the common sense that not all sentences nor sub-regions are meaningful for
cross-modality interaction. A cross-modality conditional reconstruction (CCR)
module is designed to further leverage fine-grained representation. We
reconstruct the masked image and the sentence prototypes as guided by the
cross-modality representation.

###  3.2 Image and Report Representation

The image encoder $E_{I}$ encodes each image ${x_{I}^{i}}$ into $M_{I}$ sub-
regional representations, formulated as
$f_{LI}^{i,v}\in\mathbb{R}^{M_{I}\times C_{I}}$, where $v$ is the spatial
location index of each sub-region. The global image representation
$f_{GI}^{i}$ is obtained by self-attention pooling over localized features
[39].

Similarly, the text encoder $E_{R}$ first encodes each report ${x_{R}^{i}}$
into token-wise representations. After that, self-attention pooling is
employed to derive $M_{R}^{i}$ sentence-wise representations $q_{LR}^{i,u}$
over all tokens in the same sentence, where $u$ indexes sentences and
$M_{R}^{i}$ is the total number of sentences in the report. Note that all
these localized features will be updated by a sentence-wise memory bank that
will be described in the next subsection. Different from previous works that
consider the [CLS] token as the global representation, we gather sentence-wise
features via an additional self-attention pooling operation to serve as the
global representation $f_{GR}^{i}$.

###  3.3 Sentence Prototype Memory Bank

Normally, each sentence in a medical report identifies an observation,
including symptoms, locations of lesions, etc. As such, a medical report’s
sentences can be treated as different representations of prototype features.
Therefore, before going through the cross-modality LAM, we update the report’s
representation via a learnable memory bank. The proposed SPB
$S\in\mathbb{R}^{K\times D}$ consists of $K$ representation prototypes, where
$D$ is the dimension of the common embedding space. The operation of querying
the sentence-wise prototype representation is formulated as

$f_{LR}^{i,u}=s_{k},\ \ \
k=\mathop{\arg\max}\limits_{j}\frac{q_{LR}^{i,u}\cdot
s_{j}^{T}}{\|q_{LR}^{i,u}\|\|s_{j}\|},$ (1)

where $q_{LR}^{i,u}$ is the $u$-th sentence-wise representation of the $i$-th
report, and $s_{j}$ denotes the $j$-th prototype in the memory bank. We update
the sentence-wise representation through querying SPB. Since Eq. (1) is
discrete and not differentiable, we employ the Gumbel-softmax
reparameterization trick [20] based on the querying distribution $p_{K}$ over
$K$ cells. To push the localized representation towards the central prototype,
an L1 loss is applied to explicitly update the memory bank, namely

$\mathcal{L}_{proto}=\frac{1}{\sum_{i=1}^{N}M_{R}^{i}}\sum_{i=1}^{N}\sum_{u=1}^{M_{R}^{i}}\|f_{LR}^{i,u}-q_{LR}^{i,u}\|_{1}.$
(2)

###  3.4 Cross-modality Representation Alignment

Paired image and report usually describe similar semantic information. For
further cross-modality alignment, both global and local representation is
projected into a common embedding space with dimension $D$ via four
independent MLPs. To explicitly align the global representation between the
paired image and report, we maximize their mutual information. Specifically,
we employ the InfoNCE loss [35] to estimate the lower bound of the mutual
information. The global report-to-image alignment loss is defined as

$\mathcal{L}_{g}^{I\leftarrow
R}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(f_{GI}^{i}\cdot{f_{GR}^{i}}^{T}/\tau_{1})}{\sum_{j=1}^{B}\exp(f_{GI}^{i}\cdot{f_{GR}^{j}}^{T}/\tau_{1})},$
(3)

where $B$ is the batch size and $\tau_{1}$ is the temperature parameter.
Similarly, the global image-to-report alignment loss is formulated as

$\mathcal{L}_{g}^{R\leftarrow
I}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(f_{GI}^{i}\cdot{f_{GR}^{i}}^{T}/\tau_{1})}{\sum_{j=1}^{B}\exp(f_{GI}^{j}\cdot{f_{GR}^{i}}^{T}/\tau_{1})}.$
(4)

Global alignment focuses on discriminative high-level features. However,
medical image analysis tasks are generally highly sensitive to fine-grained
low-level information like lesion boundaries. As such, we design an LAM.

The first step of LAM is to generate cross-modality attention representation.
Specifically, sub-region/sentence representation is respectively learned by
sub-region-wise/sentence-wise weighted sum. Given that not all sub-
regions/sentences contain meaningful semantic information, we do not use the
Softmax function which regards each localized unit as contributing constantly
to the cross-modality representation (i.e., sum of weights is 1). The local
report-to-image attention-based representation is formulated as

$c_{LI}^{i,v}=\sum_{k=1}^{M_{R}^{i}}\sigma\left(\frac{Q^{I}f_{LI}^{i,v}\cdot{K^{I}f_{LR}^{i,k}}^{T}}{\sqrt{D}}\right)\cdot
V^{I}f_{LR}^{i,k},$ (5)

where $Q^{I}$, $K^{I}$ and $V^{I}$ are learnable matrices, and $\sigma$
denotes the Sigmoid function. Similarly, the local image-to-report attention-
based representation is

$c_{LR}^{i,u}=\sum_{k=1}^{M_{I}}\sigma\left(\frac{Q^{R}f_{LR}^{i,u}\cdot{K^{R}f_{LI}^{i,k}}^{T}}{\sqrt{D}}\right)\cdot
V^{R}f_{LI}^{i,k}.$ (6)

After obtaining the cross-modality attention representation, we perform local
alignment for localized cross-modality information interaction. For local
image-to-report alignment, the cross-modality representation and the localized
linguistic representation of the same sentence are regarded as a positive
pair, while those from different sentences are negative samples. We observe
that different reports may share similar sentence representation. Therefore,
for local alignment we use only sentences from the same report as the negative
samples. As such, the image-to-report local alignment loss is defined as

$\mathcal{L}_{l}^{R\leftarrow
I}=-\frac{1}{\sum\limits_{i=1}^{B}M_{R}^{i}}\sum_{i=1}^{B}\sum\limits_{u=1}^{M_{R}^{i}}\log\frac{\exp(f_{LR}^{i,u}\cdot{c_{LR}^{i,u}}^{T}/\tau_{2})}{\sum_{k=1}^{M_{R}^{i}}\exp(f_{LR}^{i,u}\cdot{c_{LR}^{i,k}}^{T}/\tau_{2})},$
(7)

where $\tau_{2}$ is the temperature parameter. For local report-to-image
alignment, we conjecture that nearby sub-regions contain similar information
induced by spatial structure and overlapping convolutions. It may cause
feature collapse if we consider a positive-like sample as a negative one.
Therefore, we apply the alignment methods proposed in [6], utilizing neither
negative pairs in contrastive loss nor prior sampling strategies in [33]. The
report-to-image local alignment loss is based on the cosine similarity and
gets combined with asymmetrical projection, namely

$\begin{split}\mathcal{L}_{l}^{I\leftarrow
R}&=-\frac{1}{BM_{I}}\sum_{i=1}^{B}\sum_{v=1}^{M_{I}}\frac{1}{2}\text{\bf{sim}}\left(h(f_{LI}^{i,v}),S(c_{LI}^{i,v})\right)\\\
&+\frac{1}{2}\text{\bf{sim}}\left(h(c_{LI}^{i,v}),S(f_{LI}^{i,v})\right),\end{split}$
(8)

where $h$ is an independent MLP head, $S$ is the stop-gradient operation, and
sim denotes the cosine similarity. Finally, the total cross-modality
representation alignment loss is

$\mathcal{L}_{align}=\mathcal{L}_{g}^{I\leftarrow
R}+\mathcal{L}_{g}^{R\leftarrow I}+\mathcal{L}_{l}^{I\leftarrow
R}+\mathcal{L}_{l}^{R\leftarrow I}.$ (9)

###  3.5 Cross-modality Conditional Reconstruction

Both global and local alignments concentrate on high-level features but are
likely to ignore low-level information. To address this issue, we introduce a
cross-modality conditional reconstruction task aiming to recover low-level
information based on cross-modality interaction.

We consider two reconstruction tasks: masked image reconstruction and sentence
prototype generation. We randomly zero out 50% image features via dropout.
Then we concatenate the masked image features $\hat{f}_{LI}^{i}$ and the
unmasked cross-modality features $c_{LI}^{i}$. To reconstruct the image, a
lightweight decoder $D_{I}$ is applied. The reconstruction loss of the masked
image is presented as

$\mathcal{L}_{ir}=\frac{1}{NM_{I}}\sum_{i=1}^{N}\|D_{I}\left(\text{\bf{cat}}(\hat{f}_{LI}^{i},c_{LI}^{i})\right)-x_{I}^{i}\|_{1},$
(10)

where cat is the concatenation operation, $\hat{f}_{LI}^{i}$ represents the
masked feature map, and $c_{LI}^{i}$ denotes the cross-modality feature map.

For reconstructing the report information, we follow an encoder-decoder
architecture and apply a cross-attention mechanism [50] to interact with
visual features. Inspired by [2], parallel decoding is used to generate the
sentence-wise prototypes. We employ $U$ learnable embeddings (we call
prototype queries) to generate different predicted prototypes in the decoder.
If the number of sentences $M_{R}^{i}$ is smaller than $U$, we pad the
sentence embeddings with zero prototype $\varnothing$. Since the sentences in
a report are not necessarily sequential, which means the decoding process
should be permutation invariant, we apply the Hungarian algorithm [23] to
identify the optimal prediction-label pair $\hat{\sigma}$ in the matching
space $\mathfrak{S}_{U}$. We consider the L1 distance as the matching cost to
perform bipartite matching

$\hat{\sigma}=\mathop{\arg\min}\limits_{\sigma\in\mathfrak{S}_{U}}\sum_{i=0}^{N}\sum_{u=0}^{M_{R}^{i}}\|s^{i,u}-\hat{q}_{\sigma(i,u)}\|_{1},$
(11)

where $s^{i,u}$ is the ground truth prototype and $\hat{q}_{\sigma(i,u)}$ is
the predicted querying embedding that matches index $\sigma(i,u)$. After
deriving the optimal matching pairs, we apply Eq. (1) to obtain the predicted
prototype $\hat{s}_{\sigma(i)}$ based on the predicted distribution $q_{K}$.

The loss function for sentence prototype prediction has three components. The
first term is the L1 distance between the predicted querying embedding and the
target prototype

$\mathcal{L}_{q}=\frac{1}{\sum_{i=1}^{N}M_{R}^{i}}\sum_{i=0}^{N}\sum_{u=0}^{M_{R}^{i}}\|s^{i,u}-\hat{q}_{\sigma(i,u)}\|_{1}.$
(12)

The second term focuses on distribution consistency, namely the KL divergence
between the querying distribution $p_{K}$ and the predicted distribution
$q_{K}$

$\mathcal{L}_{kl}=D_{KL}(q_{k}\|p_{k}).$ (13)

The last term is similar to that in the global alignment. We first obtain the
predicted global report representation $p_{GR}^{i}$ via self-attention pooling
over predicted prototypes $p_{LR}$. Then a global prediction alignment loss
aiming to align the predicted global representation and the realistic global
report representation is formulated as follows

$\mathcal{L}_{gpa}=-\sum_{i=1}^{B}\log\frac{\exp(f_{GR}^{i}\cdot{p_{GR}^{i}}^{T}/\tau_{3})}{\sum_{j=1}^{B}\exp(f_{GR}^{j}\cdot{p_{GR}^{i}}^{T}/\tau_{3})}.$
(14)

Finally, the overall loss for the cross-modality conditional reconstruction is

$\mathcal{L}_{recon}=\mathcal{L}_{ir}+\mathcal{L}_{q}+\mathcal{L}_{kl}+\mathcal{L}_{gpa}.$
(15)

###  3.6 Loss

The overall loss function of the proposed PRIOR contains the following terms:
the cross-modality representation alignment loss $\mathcal{L}_{align}$, the
SPB updating loss $\mathcal{L}_{proto}$ and the cross-modality conditional
reconstruction loss $\mathcal{L}_{recon}$. We optimize these terms jointly via
a weighted sum

$\mathcal{L}=\mathcal{L}_{align}+\lambda_{proto}\mathcal{L}_{proto}+\lambda_{recon}\mathcal{L}_{recon},$
(16)

where $\lambda_{proto}$ and $\lambda_{recon}$ are hyper-parameters.

Table 1: Supervised classification results by fine-tuning on three downstream
datasets. All methods are trained on different portions of the training set
from 1% to 100% and evaluated by AUC-ROC. Each value is the average of five
runs. The best results are highlighted in bold and red, and the second-best
results are highlighted in blue.

Init. Methods | RSNA |  | SIIM |  | CheXpert
---|---|---|---|---|---
1% | 10% | 100% |  | 1% | 10% | 100% |  | 1% | 10% | 100%
Random Init. | 56.42 $\pm$ 2.79 | 57.62 $\pm$ 3.27 | 83.63 $\pm$ 0.27 |  | 55.41 $\pm$ 1.72 | 57.83 $\pm$ 0.60 | 80.63 $\pm$ 0.55 |  | 52.96 $\pm$ 3.52 | 73.37 $\pm$ 0.68 | 75.12 $\pm$ 0.35
ImageNet Init. | 80.27 $\pm$ 0.71 | 81.85 $\pm$ 0.46 | 88.01 $\pm$ 0.13 |  | 78.33 $\pm$ 1.79 | 82.88 $\pm$ 0.37 | 88.41 $\pm$ 0.41 |  | 77.57 $\pm$ 1.13 | 82.94 $\pm$ 0.91 | 87.08 $\pm$ 0.37
MoCo [15] | 82.33 $\pm$ 0.47 | 85.22 $\pm$ 0.11 | 87.90 $\pm$ 0.09 |  | 75.49 $\pm$ 0.29 | 81.01 $\pm$ 0.67 | 88.43 $\pm$ 0.24 |  | 78.00 $\pm$ 0.62 | 86.27 $\pm$ 0.30 | 87.24 $\pm$ 0.05
MoCoV2 [5] | 83.07 $\pm$ 0.49 | 85.88 $\pm$ 0.28 | 88.60 $\pm$ 0.07 |  | 77.10 $\pm$ 0.49 | 81.12 $\pm$ 0.64 | 90.72 $\pm$ 0.21 |  | 79.64 $\pm$ 0.53 | 86.04 $\pm$ 0.23 | 87.44 $\pm$ 0.27
SimCLR [4] | 80.18 $\pm$ 2.78 | 84.60 $\pm$ 0.12 | 88.07 $\pm$ 0.11 |  | 74.97 $\pm$ 2.17 | 83.21 $\pm$ 0.49 | 88.72 $\pm$ 0.28 |  | 67.41 $\pm$ 2.74 | 86.74 $\pm$ 0.36 | 87.97 $\pm$ 0.22
ConVIRT [59] | 84.17 $\pm$ 0.77 | 86.92 $\pm$ 0.13 | 88.74 $\pm$ 0.36 |  | 84.17 $\pm$ 0.77 | 85.66 $\pm$ 0.45 | 91.50 $\pm$ 0.08 |  | 85.02 $\pm$ 0.28 | 87.58 $\pm$ 0.53 | 88.21 $\pm$ 0.46
GLoRIA [17] | 84.12 $\pm$ 0.47 | 86.83 $\pm$ 0.53 | 89.13 $\pm$ 0.12 |  | 85.05 $\pm$ 1.62 | 88.51 $\pm$ 0.78 | 92.11 $\pm$ 0.18 |  | 83.61 $\pm$ 0.52 | 87.40 $\pm$ 0.39 | 88.34 $\pm$ 0.12
LoVT [33] | 85.51 $\pm$ 0.36 | 86.53 $\pm$ 0.54 | 89.27 $\pm$ 0.13 |  | 85.47 $\pm$ 0.85 | 88.50 $\pm$ 0.76 | 92.16 $\pm$ 0.36 |  | 85.13 $\pm$ 0.48 | 88.05 $\pm$ 0.15 | 88.27 $\pm$ 0.07
MGCA [52] | 85.80 $\pm$ 0.68 | 87.66 $\pm$ 0.21 | 89.30 $\pm$ 0.16 |  | 86.10 $\pm$ 0.31 | 89.68 $\pm$ 0.09 | 92.04 $\pm$ 0.09 |  | 85.63 $\pm$ 0.33 | 87.65 $\pm$ 0.33 | 88.30 $\pm$ 1.48
PRIOR (ours) | 85.74 $\pm$ 0.36 | 87.08 $\pm$ 0.19 | 89.22 $\pm$ 0.17 |  | 87.27 $\pm$ 0.39 | 89.13 $\pm$ 0.11 | 92.39 $\pm$ 0.23 |  | 86.16 $\pm$ 0.64 | 88.31 $\pm$ 0.20 | 88.61 $\pm$ 0.29

##  4 Experiments

###  4.1 Datasets

MIMIC-CXR [21]. We pre-train all SSL and VLP methods on version 2 of MIMIC-
CXR, which contains $377110$ X-ray chest images and $227827$ corresponding
reports. Because downstream tasks all employ frontal-view images, we remove
all lateral-view images. Additionally, we abandon short reports containing
fewer than four sentences, ending up with $182475$ image-report pairs.

RSNA Pneumonia Detection [44]. The RSNA Pneumonia Detection dataset contains
more than $30000$ X-ray chest images. All data have image-level pneumonia
classification and bounding-box pneumonia detection labels. Therefore, we
utilize the RSNA dataset for two downstream tasks: supervised classification
and object detection. For the classification task, we split the dataset by
60%/20%/20% for training/validation/testing. For the object detection task, we
only use the pneumonia samples (a total of 6012) through five-fold cross-
validation.

SIIM-ACR Pneumothorax. SIIM-ACR Pneumothorax contains $12954$ X-ray chest
images, together with image-level pneumothorax annotation and pixel-level
segmentation mask if pneumothorax exists. We use them for downstream
supervised classification and semantic segmentation. For the classification
task, we split the dataset by 60%/20%/20% for training/validation/testing. For
the segmentation task, we only use the pneumothorax samples (a total of 2669)
through five-fold cross-validation.

CheXpert. [19] CheXpert contains more than $220000$ X-ray chest images. Each
image is labeled for five independent diseases. And thus the downstream task
for this dataset is multi-label classification. We split the training set by
80%/20% for training/validation and use the official validation set for
testing. Similar to that in GLoRIA [17], we create a sub-set (named CheXpert
5x200) from the training set of CheXpert for zero-shot classification and
image-to-text retrieval, including $200$ exclusively positive samples from
each disease.

###  4.2 Implementation Details

Following previous works, we use the BERT model pre-trained by ClinicalBERT
[1]. For image feature encoding, we apply ResNet50 [16] as the backbone. We
resize all images into $224\times 224$, and all reports are padded or
truncated to have a fixed length of 256 tokens. We choose 768 to be the common
latent dimension $D$. The SPB size $K$ is set as $512$. The temperature
coefficients $\tau_{1}$, $\tau_{2}$, and $\tau_{3}$ are all set as $0.01$. The
loss weight coefficients are set as $\lambda_{recon}=1$ and
$\lambda_{proto}=10$. The overall framework is trained with a batch size of
$128$ for $100$ epochs. The learning rate is initialized as $1e-5$ and decays
following a cosine policy.

###  4.3 Evaluation Experiments

We compare our method with several widely-used initialization methods,
including three image-level SSL methods (MoCo, MoCoV2, SimCLR), four medical
VLP methods (ConVIRT, GLoRIA, LoVT, MGCA), and ImageNet pre-training. All SSL
and VLP methods are re-trained on the MIMIC-CXR dataset with their official
implementations and default hyper-parameters. ConVIRT applies only a global
alignment loss, and may not perform well on locality-aware downstream tasks.
GLoRIA, LoVT and MGCA are three SOTA VLP methods targeting medical images and
reports with well-designed local alignment losses.

All compared methods are evaluated on five downstream tasks, including
supervised classification, zero-shot classification, image-to-text retrieval,
semantic segmentation, and object detection. For zero-shot classification and
image-to-text retrieval, we compare with GLoRIA and ConVIRT which present
prompt engineering in their original papers. Although most of the existing
methods perform linear evaluation, similarly to [14], we conduct a partial
fine-tuning evaluation on CheXpert with 1% training data, updating only the
last several residual blocks in ResNet50. We observe that the linear
evaluation performance is not strictly keeping in line with the fine-tuning
performance, as clearly shown in Figure 3. Additionally, if we unfreeze only
the last block, the performance significantly increases, which agrees with
previously-reported results [14]. Linear evaluation cannot well deal with non-
linear features which are nevertheless critical for downstream tasks.
Therefore, we only report the fine-tuning results.

Figure 3: Partial fine-tuning results on 1% CheXpert. The number of blocks for
fine-tuning increases from left to right. Fine-tuning with 0 block is
equivalent to linear evaluation, while fine-tuning with 4 blocks is equivalent
to full fine-tuning.

###  4.4 Results on Classification

We conduct supervised classification experiments on RSNA, SIIM, and CheXpert.
AUC-ROC is employed as the evaluation metric. Table 1 shows that all VLP
methods outperform image-level SSL methods and ImageNet initialization.
Collectively, PRIOR achieves the best results, especially on SIIM and CheXpert
with different training data sizes. Even though ConVIRT, GLoRIA and LoVT have
leveraged semantic representation from large-scale report supervision, they
may have ignored the clinical paradigm relation between images and reports,
the relation of which is successfully modeled by our proposed SPB. MGCA
conducts disease-level alignment and performs well on the RSNA dataset, but it
neglects sentence-level clinical information which is nevertheless crucial for
complex downstream tasks like multi-label classification on CheXpert.

To analyze the quality of the cross-modality representation, we measure the
zero-shot classification performance on CheXpert 5x200. We use models pre-
trained on MIMIC-CXR to predict the labels of CheXpert. Since LoVT and MGCA do
not propose any ad-hoc zero-shot learning strategy in their original papers,
we compare PRIOR with the other two VLP methods on the zero-shot
classification task. For generalization, we employ the simplest prompt
engineering strategy without any heavy ad-hoc design. The prediction is given
by measuring the similarly between the predicted image and prompts. We take
the sum of the global alignment loss and the cross-modality local attention as
the similarity. As shown in Table 2, neither ConVIRT nor GLoRIA can capture
correlative semantic information with simple prompt engineering. PRIOR attains
the best zero-shot classification results, suggesting it can effectively
transfer knowledge from linguistic supervision to vision domain.

###  4.5 Image-to-text Retrieval

We conduct image-to-text retrieval experiments on the CheXpert 5x200 dataset.
Since the reports in CheXpert are inaccessible, we randomly select 1000
reports from MIMIC-CXR with exclusive 200 samples for each of 5 diseases. We
measure the performance using Precision@K to check if the retrieved report
matches the query image label. Our results in Table 3 demonstrate that PRIOR
significantly outperforms both GLoRIA and ConVIRT.

###  4.6 Results on Segmentation and Detection

We examine the contribution of the local representation through the semantic
segmentation task. We use U-Net [43] as the segmentation architecture. Table 4
presents the results of all methods on the SIIM dataset. Since GLoRIA, LoVT,
MGCA and the proposed PRIOR consider localized feature representation during
the pre-training phase, all of them outperform other methods. Among all the
four medical image and report VLP methods, PRIOR achieves the most superior
performance, especially with limited training data. A plausible reason is that
PRIOR learns detailed locality-aware information via conditional
reconstruction.

Table 2: Zero-shot classification results on CheXpert 5x200.

VLP Methods | CheXpert Zero-shot Classification
---|---
Accuracy | F1-Score | Precision
ConVIRT [59] | 21.30 | 19.03 | 17.65
GLoRIA [17] | 23.20 | 15.79 | 44.01
PRIOR (ours) | 34.90 | 30.56 | 35.88

Table 3: Image-to-text retrieval results on CheXpert 5x200.

VLP Methods | CheXpert Image-to-text Retrieval
---|---
Prec $@$ 1 | Prec $@$ 2 | Prec $@$ 5 | Prec $@$ 10
ConVIRT [59] | 20.3 | 19.8 | 19.7 | 19.9
GLoRIA [17] | 29.3 | 29.0 | 27.8 | 26.8
PRIOR(ours) | 40.2 | 39.6 | 39.3 | 38.0

To further evaluate the effectiveness of our proposed method, we conduct
object detection on the RSNA dataset. We use Faster R-CNN [42] as the
detection architecture. Table 5 presents the performance comparison results on
RSNA. We observe that some SSL methods outperform VLP methods, and we
conjecture it is because medical reports may be incapable of precisely
providing pneumonia information, which is consistent with the original
motivation of building the RSNA dataset [44]. With that being said, our
proposed PRIOR still outperforms all other methods, successfully establishing
its effectiveness.

Table 4: Semantic segmentation results by fine-tuning on SIIM. All methods are
trained on different portions of the training set from 1% to 100% and
evaluated by Dice.

Init. Methods | SIIM Segmentation
---|---
1% | 10% | 100%
Random Init. | 6.59 $\pm$ 0.40 | 16.61 $\pm$ 1.44 | 38.25 $\pm$ 0.90
ImageNet Init. | 14.92 $\pm$ 3.38 | 25.26 $\pm$ 3.04 | 45.07 $\pm$ 0.87
MoCo [15] | 17.66 $\pm$ 2.14 | 27.81 $\pm$ 2.19 | 41.18 $\pm$ 1.53
MoCoV2 [5] | 18.19 $\pm$ 3.35 | 28.57 $\pm$ 2.92 | 44.01 $\pm$ 1.36
SimCLR [4] | 16.55 $\pm$ 2.27 | 23.36 $\pm$ 0.66 | 40.62 $\pm$ 0.69
ConVIRT [59] | 18.48 $\pm$ 2.43 | 27.32 $\pm$ 2.60 | 41.72 $\pm$ 0.69
GLoRIA [17] | 18.78 $\pm$ 2.85 | 33.55 $\pm$ 2.14 | 45.46 $\pm$ 1.69
LoVT [33] | 18.81 $\pm$ 1.04 | 32.68 $\pm$ 0.95 | 44.65 $\pm$ 1.36
MGCA [52] | 18.84 $\pm$ 2.37 | 33.54 $\pm$ 1.89 | 45.39 $\pm$ 1.35
PRIOR (ours) | 20.43 $\pm$ 2.20 | 34.81 $\pm$ 1.60 | 46.01 $\pm$ 1.03

###  4.7 Ablation Study

This section presents the ablation study results of PRIOR under the 1% SIIM
dataset setting for supervised classification and semantic segmentation. We
compare the performance of the following variants: (1) Baseline, which only
uses global alignment; (2) Baseline+LAM, which uses both global alignment and
local alignment; (3) Baseline+LAM+CCR, which uses global alignment, local
alignment and cross-modality conditional reconstruction; (4)
Baseline+LAM+CCR+SPB (proposed PRIOR), which uses global alignment, local
alignment, cross-modality conditional reconstruction, and sentence prototype
memory bank. Note that variant (3) reconstructs only the masked image because
the sentence-level representation is not categorized and unavailable without
SPB. Table 6 tabulates the ablation study results, demonstrating all
components are essential for extracting both global and local features.

Figure 4: Representative cross-modality attention maps. (a) The related
sentence is “increased bibasilar opacities are combination of increased
bilateral pleural effusions and bibasilar atelectasis”. (b) The related
sentence is “unchanged normal size of the cardiac silhouette ”.

###  4.8 Visualization of Cross-modality Attention

We visualize the cross-modality attention map to show the effectiveness and
interpretability of the proposed PRIOR. Given an image and a corresponding
sentence, we first obtain the cross-modality attention map by Eq. (5). Then,
we visualize the attention map by resizing it to the same size as the image
and superimposing it on top of the image. Figure 4 presents two examples to
illustrate that the proposed attention module can detect correlative regions
given a sentence. For example, the heatmap in Figure 4a highlights the
abnormal regions precisely, aligning well with the report. Figure 4b also
correctly detects the heart location.

###  4.9 Visualization of Feature Representation

To assess the quality of the feature representation from the proposed method,
we utilize t-SNE [49] to visualize the high-level embeddings from the last
layer of the image encoder on CheXpert 5x200. Apparently, results from the
proposed method exhibit a clustering pattern more evidently, while results
from other methods are more isotropic and homogeneous and fail to distinguish
different categories.

Table 5: Object detection results by fine-tuning on RSNA. All methods are
trained on different portions of the training set from 1% to 100% and
evaluated by mAP(0.5:0.95). - means the method’s mAP is lower than 0.1.

Init. Methods | RSNA Object Detection
---|---
1% | 10% | 100%
Random Init. | - | 2.66 $\pm$ 1.34 | 18.99 $\pm$ 1.36
ImageNet Init. | - | 7.85 $\pm$ 2.68 | 21.16 $\pm$ 1.94
MoCo [15] | - | 16.25 $\pm$ 1.96 | 21.00 $\pm$ 2.00
MoCoV2 [5] | 0.19 $\pm$ 0.22 | 18.45 $\pm$ 1.75 | 21.65 $\pm$ 2.10
SimCLR [4] | - | 14.00 $\pm$ 1.96 | 21.76 $\pm$ 1.93
ConVIRT [59] | - | 18.80 $\pm$ 2.79 | 21.28 $\pm$ 1.65
GLoRIA [17] | - | 17.95 $\pm$ 1.37 | 21.37 $\pm$ 1.07
LoVT [33] | - | 17.46 $\pm$ 2.21 | 21.80 $\pm$ 2.75
MGCA [52] | - | 19.10 $\pm$ 1.83 | 21.33 $\pm$ 1.63
PRIOR (ours) | 0.20 $\pm$ 0.27 | 19.61 $\pm$ 1.93 | 22.20 $\pm$ 1.58

Table 6: Ablation results of the proposed method on 1% SIIM. LAM denotes the
local alignment module, CCR denotes the cross-modality conditional
reconstruction and SPB denotes the sentence prototype memory bank.

Components | 1% SIIM Classification | 1% SIIM Segmentation
---|---|---
LAM | CCR | SPB
|  |  | 82.01 $\pm$ 2.21 | 17.88 $\pm$ 1.24
✓ |  |  | 84.86 $\pm$ 0.65 | 18.70 $\pm$ 2.33
✓ | ✓ |  | 85.80 $\pm$ 0.86 | 19.33 $\pm$ 2.22
✓ |  | ✓ | 86.81 $\pm$ 0.47 | 19.87 $\pm$ 2.03
✓ | ✓ | ✓ | 87.27 $\pm$ 0.39 | 20.43 $\pm$ 2.20

Figure 5: t-SNE visualization of the high-level embeddings from the last layer
of the image encoder on CheXpert 5x200.

##  5 Conclusion

In this paper, we have explored fine-grained representation learning from
paired medical images and reports. We present a cross-modality alignment
strategy by learning from both global and local information. A sentence-wise
prototype memory bank is proposed to translate clinical descriptions into
categorical representations and cluster similar clinical conditions. Based on
the prototype memory bank, a cross-modality conditional reconstruction module
is designed to reconstruct masked images and missing sentence prototypes
through cross-modality interaction, facilitating low-level representation
learning which is crucial for locality-aware downstream tasks. Extensive
experiments on five downstream tasks demonstrate the effectiveness of the
proposed method.

## References

  * [1] Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott.  Publicly available clinical bert embeddings.  arXiv preprint arXiv:1904.03323, 2019.
  * [2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [3] Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Michitaka Fujiwara, and Daniel Rueckert.  Self-supervised learning for medical image analysis using image context restoration.  Medical image analysis, 58:101539, 2019.
  * [4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [5] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [6] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
  * [7] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640–9649, 2021.
  * [8] Zhenfang Chen, Lin Ma, Wenhan Luo, and Kwan-Yee Kenneth Wong.  Weakly-supervised spatio-temporally grounding natural sentence in video.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1884–1894, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [11] William Falcon et al.  Pytorch lightning.  GitHub. Note: https://github.com/PyTorchLightning/pytorch-lightning, 3, 2019.
  * [12] Han Fang, Pengfei Xiong, Luhui Xu, and Wenhan Luo.  Transferring image-clip to video-text retrieval via temporal relations.  IEEE Transactions on Multimedia, 2022.
  * [13] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in neural information processing systems, 33:21271–21284, 2020.
  * [14] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [17] Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung.  Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942–3951, 2021.
  * [18] Pavel Iakubovskii.  Segmentation models pytorch.  https://github.com/qubvel/segmentation_models.pytorch, 2019.
  * [19] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590–597, 2019.
  * [20] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  arXiv preprint arXiv:1611.01144, 2016.
  * [21] Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng.  Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs.  arXiv preprint arXiv:1901.07042, 2019.
  * [22] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [23] Harold W Kuhn.  The hungarian method for the assignment problem.  Naval research logistics quarterly, 2(1-2):83–97, 1955.
  * [24] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich.  Colorization as a proxy task for visual understanding.  In CVPR, 2017.
  * [25] Christy Y Li, Xiaodan Liang, Zhiting Hu, and Eric P Xing.  Knowledge-driven encode, retrieve, paraphrase for medical image report generation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6666–6673, 2019.
  * [26] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  arXiv preprint arXiv:2201.12086, 2022.
  * [27] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.  Grounded language-image pre-training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965–10975, 2022.
  * [28] Yuan Li, Xiaodan Liang, Zhiting Hu, and Eric P Xing.  Hybrid retrieval-generation reinforced agent for medical image report generation.  Advances in neural information processing systems, 31, 2018.
  * [29] Wei Liu, Sihan Chen, Longteng Guo, Xinxin Zhu, and Jing Liu.  Cptr: Full transformer network for image captioning.  arXiv preprint arXiv:2101.10804, 2021.
  * [30] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  Advances in neural information processing systems, 32, 2019.
  * [31] Sébastien Marcel and Yann Rodriguez.  Torchvision the machine-vision package of torch.  In Proceedings of the 18th ACM international conference on Multimedia, pages 1485–1488, 2010.
  * [32] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  Slip: Self-supervision meets language-image pre-training.  In European Conference on Computer Vision, pages 529–544. Springer, 2022.
  * [33] Philip Müller, Georgios Kaissis, Congyu Zou, and Daniel Rückert.  Joint learning of localized representations from medical images and reports.  arXiv preprint arXiv:2112.02889, 2021.
  * [34] Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc V Le, and Ruoming Pang.  Domain adaptive transfer learning with specialist models.  arXiv preprint arXiv:1811.07056, 2018.
  * [35] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [36] Federico Paredes-Vallés and Guido CHE de Croon.  Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3446–3455, 2021.
  * [37] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [38] Yadunath Pathak, Prashant Kumar Shukla, Akhilesh Tiwari, Shalini Stalin, and Saurabh Singh.  Deep transfer learning based classification model for covid-19 disease.  Irbm, 2020.
  * [39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [40] Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio.  Transfusion: Understanding transfer learning for medical imaging.  Advances in neural information processing systems, 32, 2019.
  * [41] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [42] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [43] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [44] George Shih, Carol C Wu, Safwan S Halabi, Marc D Kohli, Luciano M Prevedello, Tessa S Cook, Arjun Sharma, Judith K Amorosa, Veronica Arteaga, Maya Galperin-Aizenberg, et al.  Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia.  Radiology. Artificial intelligence, 1(1), 2019.
  * [45] Hari Sowrirajan, Jingbo Yang, Andrew Y Ng, and Pranav Rajpurkar.  Moco pretraining improves representation and transferability of chest x-ray models.  In Medical Imaging with Deep Learning, pages 728–744. PMLR, 2021\.
  * [46] Aiham Taleb, Winfried Loetzsch, Noel Danz, Julius Severin, Thomas Gaertner, Benjamin Bergner, and Christoph Lippert.  3d self-supervised methods for medical imaging.  Advances in Neural Information Processing Systems, 33:18158–18172, 2020.
  * [47] Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Pérez, and Christian Theobalt.  Self-supervised multi-level face model learning for monocular reconstruction at over 250 hz.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2549–2559, 2018.
  * [48] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  Advances in neural information processing systems, 30, 2017.
  * [49] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [51] Pauli Virtanen and Ralf etal Gommers.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  Nature Methods, 17:261–272, 2020.
  * [52] Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanabhuti, and Lequan Yu.  Multi-granularity cross-modal alignment for generalized medical visual representation learning.  Advances in Neural Information Processing Systems, 2022.
  * [53] Yizhi Wang, Guo Pu, Wenhan Luo, Yexin Wang, Pengfei Xiong, Hongwen Kang, and Zhouhui Lian.  Aesthetic text logo synthesis via content-aware layout inferring.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2436–2445, 2022.
  * [54] Yiyu Wang, Jungang Xu, and Yingfei Sun.  End-to-end transformer based model for image captioning.  arXiv preprint arXiv:2203.15350, 2022.
  * [55] Thomas Wolf and Lysandre Debut et al.  Transformers: State-of-the-art natural language processing.  pages 38–45, Online, Oct. 2020. Association for Computational Linguistics.
  * [56] Xingyi Yang, Muchao Ye, Quanzeng You, and Fenglong Ma.  Writing by memorizing: Hierarchical retrieval-based medical report generation.  arXiv preprint arXiv:2106.06471, 2021.
  * [57] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: Fine-grained interactive language-image pre-training.  arXiv preprint arXiv:2111.07783, 2021.
  * [58] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  arXiv preprint arXiv:2205.01917, 2022.
  * [59] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz.  Contrastive learning of medical visual representations from paired images and text.  arXiv preprint arXiv:2010.00747, 2020.
  * [60] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He.  A comprehensive survey on transfer learning.  Proceedings of the IEEE, 109(1):43–76, 2020.

Supplementary Material

##  6 Introduction

In this document, we provide supplementary material for our paper “PRIOR:
Prototype Representation Joint Learning from Medical Images and Reports”. We
first provide pre-training details. Then we present details of fine-tuning on
different downstream tasks. Finally, we perform detailed analysis of each key
component in PRIOR, including sentence-wise prototype memory bank (SPB), local
alignment module (LAM), and cross-modality conditional reconstruction (CCR).

##  7 Pre-training Details

###  7.1 Data Preprocessing

We pre-train the proposed PRIOR on the MIMIC-CXR-JPG dataset. We remove all
lateral-view images and images whose corresponding reports have fewer than
four sentences. Finally, we end up with $182475$ image-report pairs.

For image preprocessing, we first normalize the intensities of all images into
$[0,1]$ and then calculate their mean ($0.4755$) and standard deviation
($0.3011$). Z-score normalization is employed. All images are resized to
$224\times 224$ before being fed into a model of interest.

For report preprocessing, we utilize all sentences from the Findings and
Impression sections. We employ BioClinicalBERT’s tokenizer [1] implemented in
the Transformer library [55] to tokenize each report into a sequence of
tokens. We then pad all reports to have the same length of $256$ tokens.

###  7.2 Single-modality Encoder Architecture

We employ ResNet50 as the image encoder, which is implemented in the
TorchVision library [31] and gets pre-trained on ImageNet. For global
representation, we apply an attention pooling layer to obtain a
$2048$-dimensional vector. For local representation, we take the feature map
$f\in\mathbb{R}^{7\times 7\times 2048}$ of the last layer of ResNet50.

We adopt BioClinicalBERT [1] as the report encoder, which is implemented in
the Transformer library [1] and gets pre-trained on the MIMIC-CXR dataset. We
take the hidden state from the last layer as the token-level representation.
After that, we gather all token representations in the same sentence via self-
attention pooling to serve as the sentence-level representation
$q_{i}\in\mathbb{R}^{M_{R}^{i}\times 768}$, where $M_{R}^{i}$ is the number of
sentences in the $i$-th report. Finally, we use an additional self-attention
pooling layer to obtain the global representation over all sentence-level
representations.

To embed linguist and visual representation into the same dimension, we attach
four independent MLPs to embed both local and global report/image
representation into an embedding space of dimension $768$.

###  7.3 Cross-modality Interaction Architecture

The cross-modality interaction architecture mainly contains three components,
namely sentence-wise prototype memory bank, cross-modality alignment, and
cross-modality conditional reconstruction.

Sentence-wise prototype memory bank. Each prototype in SPB is initialized with
the Standard Gaussian distribution and L1 normalization. For stable
convergence, the temperature coefficient in Gumbel-softmax reparameterization
decays from $0.9$ to $0.01$.

Cross-modality alignment. We employ the contrastive loss implemented in
PyTorch-Lightning [11] as the global alignment loss and the local image-to-
report alignment loss. We adopt the loss proposed elsewhere [6] as the local
report-to-image alignment loss.

Cross-modality conditional reconstruction. We adopt five up-sampling blocks
for image reconstruction. Each block consists of two convolution operations
with a size of $3\times 3$ and a stride of $1\times 1$, followed by a ReLU
activation function and a batch normalization layer. For report
reconstruction, we adopt a decoder with 6 BERT layers in the same setting as
that of BioClinicalBERT [1]. We employ the bipartite matching algorithm to
match the predicted prototypes with the ground-truth ones, which is
implemented in the SciPy library [51].

###  7.4 Training Details

We train PRIOR for 100 epochs on NVIDIA A100 GPUs with a batch size of 128.
The total training process consists of three stages: (1) In stage 1, we train
the cross-modality alignment module for 20 epochs, which is directly trained
on sentence-wise representation without SPB. (2) In stage 2, we jointly train
SPB and the cross-modality alignment module for 30 epochs. (3) In stage 3, we
jointly train SPB, the cross-modality alignment module, and the cross-modality
conditional reconstruction module for 50 epochs. We use the Adam optimizer
[22] with a learning rate of $1e-5$ and a weight decay of $1e-6$. A cosine
annealing scheduler is employed to adjust the learning rate. We employ grid
search to identify the optimal hyper-parameter combination that performs the
best on downstream tasks.

##  8 Fine-tuning Details

We evaluate the proposed PRIOR on five downstream tasks, namely supervised
classification, zero-shot classification, image-to-text retrieval, semantic
segmentation, and object detection. For each task, we employ grid search to
identify the best-performing hyper-parameters. We here only provide the
specific hyper-parameters for VLP methods since they are our essential objects
of interest; all VLP methods share the same set of hyper-parameters. For
hyper-parameters employed in non-VLP methods, please refer to our source code
for details. The intensity of each image is normalized into $[0,1]$ via
Z-score normalization using the mean and standard deviation of the training
set. We resize all images into $224\times 224$. A cosine decay scheduler is
used to adjust the learning rate.

###  8.1 Supervised Classification

For the downstream task of supervised classification, we fine-tune the image
encoder with an additional fully-connected layer on the RSNA dataset, the SIIM
Pneumothorax dataset, and the CheXpert dataset. The hyper-parameters of all
VLP methods are listed in Table 7. Since each dataset involves either binary
classification or multi-label classification, we use the binary cross-entropy
loss as the loss function. The model with the highest AUC-ROC on the
validation set is selected for testing.

Table 7: Hyper-parameter details for supervised classification.

Dataset |  | Training Data
---
Ratio
Learning Rate | Epochs | Batch Size
RSNA | 100% | $1e-4$ | 5 | 64
10% | $1e-4$ | 5 | 64
1% | $1e-4$ | 10 | 64
SIIM | 100% | $1e-5$ | 5 | 64
10% | $1e-5$ | 20 | 64
1% | $1e-4$ | 20 | 64
CheXpert | 100% | $1e-6$ | 5 | 256
10% | $1e-5$ | 5 | 256
1% | $1e-4$ | 5 | 256

###  8.2 Zero-shot Classification

We conduct zero-shot classification on the CheXpert 5x200 dataset, which is
based on the similarity between manually designed prompt sentences and images.
We generate the prompt simply through using “X is observed”, where X
represents the name of a specific disease. For generalization, we do not use
any ensemble method. We calculate the similarity between the prompt and the
image via the sum of the global alignment loss and the local cross attention
weight. Note that before summing the two scores, we employ the softmax
function to normalize each of them into $[0,1]$.

###  8.3 Image-to-text Retrieval

Similar to zero-shot classification, we conduct image-to-text retrieval on the
CheXpert 5x200 dataset, which is based on the similarity between queried
reports and images. Note that because CheXpert does not have public reports,
we sample 1000 exclusive reports from MIMIC-CXR dataset for each of 5
diseases.

###  8.4 Semantic Segmentation

We adopt U-Net [43] as the segmentation network. Different from the original
U-Net, we employ U-Net equipped with a ResNet50 backbone, which is implemented
in the Segmentation Models Pytorch library [18]. We present the hyper-
parameter details in Table 8. The Dice loss is used for training, which
performs better than either the cross-entropy loss or a combination of the two
losses according to our experimental results. The model with the highest Dice
score on the validation set is selected for testing.

###  8.5 Object Detection

We adopt Faster-RCNN [42] as the object detection network. The training and
evaluation scripts are based on Pytorch-Lightning Flash [11]. The hyper-
parameter details are also tabulated in Table 8. We follow the original
Faster-RCNN setting in terms of the loss function [42]. The model with the
highest mAP (0.5:0.95) on the validation set is selected for testing.

Table 8: Hyper-parameter details for semantic segmentation and object
detection.

Dataset |  | Training Data
---
Ratio
Learning Rate | Epochs | Batch Size
SIIM Segmentation | 100% | $1e-3$ | 50 | 128
10% | $1e-4$ | 50 | 128
1% | $1e-4$ | 100 | 16
RSNA Detection | 100% | $5e-5$ | 5 | 16
10% | $1e-4$ | 20 | 16
1% | $1e-4$ | 5 | 16

##  9 Component Analysis

In this section, we perform further analysis on each component of PRIOR,
including SPB, LAM, and CCR.

###  9.1 Analysis on SPB

To demonstrate the effectiveness of SPB, we utilize several synonymous
sentences to demonstrate the querying distribution over the memory bank, as
shown in Figure 7. It is worth noting that SPB highlights the most relevant
prototype in a sharp distribution over SPB, which means that SPB can
effectively capture the high-level semantic information of the sentence of
interest.

Figure 6: The t-SNE visualization comparison between the sentence-level
embeddings without and with SPB pre-training.

Furthermore, we visualize the sentence-wise embeddings via t-SNE [49] on 1000
randomly selected sentences from MIMIC-CXR in Figure 6. Apparently, the
sentence-wise embeddings with SPB pre-training are more compact and well-
separated. In other words, SPB significantly improves the quality of the
sentence embeddings, which benefits cross-modality interaction.

Figure 7: Representative examples demonstrating that SPB can cluster sentences
with similar information. The horizontal axis represents the sentence index in
the memory bank, and the color bar quantifies the querying score.

###  9.2 Analysis on LAM

In Figure 8, we visualize the attention weights of LAM on CheXpert 5x200. We
first generate a sentence in the form of “X is observed”, where X represents
the name of a specific disease. Then the cross-modality attention map is
obtained by the proposed LAM over the generated sentence and the corresponding
image. Clearly, LAM can effectively localize the affected region of the given
disease.

Unlike natural image captioning, some sentences in medical reports are
irrelevant to medical images, such as “AP single view of the chest has been
obtained”. Softmax assigns a constant probability summing up to 1 when
aligning sentence with image, resulting in an average representation of image
and failing to represent local features precisely. Instead, Sigmoid assigns
probability to each pixel separately. Our experiments demonstrate the
effectiveness of the Sigmoid function in cross-modality attention, as shown in
Table 9.

Table 9: The influence of the activation functions in LAM.

Activation function | CheXpert Classification | SIIM Segmentation
---|---|---
Softmax | 86.01 $\pm$ 0.68 | 45.65 $\pm$ 1.31
Sigmoid | 86.16 $\pm$ 0.64 | 46.01 $\pm$ 1.03

###  9.3 Analysis on CCR

CCR is a key component in our PRIOR, which can effectively capture fine-
grained cross-modality information. We visualize representative reconstructed
images from the CCR module in Figure 9. We find that the reconstructed images
can well maintain the low-level information that is related to the report
descriptions. Meanwhile, the reconstructed images successfully reveal the
severity of the corresponding disease as well as the lesion locations.

For sentence prototype reconstruction, we compare the query distribution of
the original report and the predicted distribution from CCR in Figure 10. The
distribution is obtained from all sentences in the report of interest. We
observe that the predicted distribution is similar to the original
distribution, which demonstrates that the CCR module can effectively capture
cross-modality information and reconstruct sentence prototypes.

Figure 8: Representative examples of cross-modality attention maps related to
different diseases. Figure 9: Representative examples of reconstructed images
from CCR. Figure 10: Representative examples of conditional reconstruction for
sentence-wise prototypes. From left to right, the first column shows the
original reports, the second column shows the original images, the top panel
of the third column shows report distributions over the memory bank, and the
bottom panel of the third column shows the predicted sentence representation
distributions. Note that the top and bottom panels of the third column are
very similar to each for all provided examples.