* [94] Weinberg, S. The Quantum Theory of Fields; Cambridge University: Cambridge, 2005.
  * [95] Abhignan, V.; Sankaranarayanan, R. Continued functions and perturbation series: Simple tools for convergence of diverging series in $O(n)$ \- symmetric $\varphi^{4}$ field theory at weak coupling limit. J. Stat. Phys. 2021, 183, 4.
  * [96] Yukalov, V.I. Self-similar approach to market analysis. Eur. Phys. J. B 2001, 20, 609–617.
  * [97] Yukalov, V.I.; Yukalova, E.P. Bose-Einstein condensation temperature of weakly interacting atoms. Laser Phys. Lett. 2017, 14, 073001.
  * [98] He, H.X.; Hamer, C.J.; Oitmaa, J. High-temperature series expansions for the $(2+1)$-dimensional Ising model. J. Phys. A 1990, 23, 1775–1788.
  * [99] Gluzman, S.; Yukalov, V.I. Critical indices from self-similar root approximants. Eur. Phys. J. Plus 2017, 132, 535.
  * [100] Zeide, B. Analysis of growth equations. Forest Sci. 1993, 39, 594–616.
  * [101] Day, T.; Taylor, P.D. Von Bertalanffy’s growth equation should not be used to model age and size at maturity. Am. Naturalist 1997, 149, 381–393.
  * [102] Yukalov, V.I.; Yukalova, E.P.; Sornette, D. Extreme events in population dynamics with functional carrying capacity. Eur. Phys. J. Spec. Top. 2012, 205, 313–354.
  * [103] Yukalov, V.I.; Yukalova, E.P.; Sornette, D. Population dynamics with nonlinear delayed carrying capacity. Int. J. Bifur. Chaos 2014, 24, 1450021.
  * [104] Yukalov, V.I.; Yukalova, E.P.; Sornette, D. Dynamical system theory of periodically collapsing bubbles. Eur. Phys. J. B 2015, 88, 179.
  * [105] Gluzman, S. Nonlinear approximations to critical and relaxation processes. Axioms 2020, 9, 126.
  * [106] Yukalov, V.I.; Yukalova, E.P. Self-similar extrapolation in quantum field theory. Phys. Rev. D 2021, 103, 076019.
  * [107] Baym, G.; Blaizot, J.P.; Holzmann, M.; Laloö, F.; Vautherin, D. The transition temperature of the dilute interacting Bose gas. Phys. Rev. Lett. 1999, 83, 1703–1706.
  * [108] Baym, G.; Blaizot, J.P.; Zinn-Justin, J. The transition temperature of the dilute interacting Bose gas for $N$ internal states. Eur. Phys. Lett. 2000, 49, 150–155.
  * [109] Kastening, B. Shift of BEC temperature of homogeneous weakly interacting Bose gas. Laser Phys. 2004, 14, 586–590.
  * [110] Kastening, B. Bose-Einstein condensation temperature of a homogeneous weakly interacting Bose gas in variational perturbation theory through seven loops. Phys. Rev. A 2004, 69, 043613.
  * [111] Kastening, B. Nonuniversal critical quantities from variational perturbation theory and their application to the Bose-Einstein condensation temperature shift. Phys. Rev. A 2004, 70, 043621.
  * [112] Yukalov, V.I.; Yukalova, E.P. Critical temperature in weakly interacting multicomponent field theory. Eur. Phys. J. Web Conf. 2017, 138, 03011.
  * [113] Kashurnikov, A.A.; Prokof’ev, N.; Svistunov, B. Critical temperature shift in weakly interacting Bose gas. Phys. Rev. Lett. 2001, 87, 120402.
  * [114] Arnold, P.; Moore, G. BEC transition temperature of a dilute homogeneous imperfect Bose gas. Phys. Rev. Lett. 2001, 87, 120401.
  * [115] Arnold, P.; Moore, G. Monte Carlo simulation of $O(2)$ $\varphi^{4}$ field theory in three dimensions. Phys. Rev. E 2001, 64, 066113.
  * [116] Sun, X. Monte Carlo studies of three-dimensional $O(1)$ and $O(4)$ $\varphi^{4}$ theory related to BEC phase transition temperatures. Phys. Rev. E 2003, 67, 066702.
  * [117] Pelissetto, A.; Vicari, E. Critical phenomena and renormalization-group theory. Phys. Rep. 2002, 368, 549–727.
  * [118] Deng, Y.; Blöte, H.W. Simultaneous analysis of several models in the three-dimensional Ising universality class. Phys. Rev. E 2003, 68, 036125.
  * [119] Campostrini, M.; Hasenbusch, M.; Pelissetto, A.; Vicari, E. Theoretical estimates of the critical exponents of the superfluid transition in 4He by lattice methods. Phys. Rev. B 2006, 74, 144506.
  * [120] Hasenbusch, M. Finite size scaling study of lattice models in the three-dimensional Ising universality class. Phys. Rev. B 2010, 82, 174433.
  * [121] Ferrenberg, A.; Xu, J.; Landau, D.P. Pushing the limits of Monte Carlo simulations for the three-dimensional Ising model. Phys. Rev. E 2018, 97, 043301.
  * [122] Kleinert, H.; Schulte-Frohlinde, V. Critical Properties of $\varphi^{4}$–Theories; World Scientific: Singapore, 2001.
  * [123] Yukalov, V.I.; Yukalova, E.P. Calculation of critical exponents by self-similar factor approximants. Eur. Phys. J. B 2007, 55, 93–99.
  * [124] Yukalov, V.I.; Yukalova, E.P. Describing phase transitions in field theory by self-similar approximants. Eur. Phys. J. Web Conf. 2019, 204, 02003.


Euler and the Legendre Polynomials

Alexander Aycock, Johannes-Gutenberg University Mainz

Staudinger Weg 9, 55128 Mainz

<EMAIL_ADDRESS>

###### Abstract

In this note we will present how Euler’s investigations on various different
subjects lead to certain properties of the Legendre polynomials. More
precisely, we will show that the generating function and the difference
equation for the Legendre polynomials was already written down by Euler in at
least two different papers. Furthermore, we will demonstrate that some
familiar expressions for the Legendre polynomials are corollaries of the
before-mentioned works. Finally, we will show that Euler’s ideas on continued
fractions lead to an integral representation for the Legendre polynomials that
seems to be less generally known.

##  1 Introduction

The Legendre polynomials occur frequently at various places in physics. They
were introduced by Legendre (1752-1833) in his paper “Recherches sur
l’attraction des sphéroïdes homogènes” [11] (“Researches on the attraction of
homogeneous spheroids”) in his study of the gravitational potential
expressible as:

$\dfrac{1}{|\bf{x}-\bf{x}^{\prime}|}=\dfrac{1}{\sqrt{r^{2}-2rr^{\prime}\cos(\beta)+(r^{\prime})^{2}}}=\sum_{n=0}^{\infty}\dfrac{{r^{\prime}}^{n}}{r^{n+1}}P_{n}(\cos(\beta)),$

where $r$ and $r^{\prime}$ are the lengths of the vectors $\bf{x}$ and
$\bf{x}^{\prime}$, respectively, $\beta$ is the angle between those two
vectors, and $P_{n}(t)$ denotes a polynomial of degree $n$ in $t$. They are
also interesting from a mathematical viewpoint, because they exhibit a system
of complete and orthogonal polynomials.

Nowadays, the $n$-th Legendre polynomial, $P_{n}(t)$, is often introduced as
the coefficient a Taylor series. We have:

$\dfrac{1}{\sqrt{1-2xt+x^{2}}}:=\sum_{n=0}^{\infty}P_{n}(t)x^{n}.$ (1)

Aside from Legendre’s original definition and the more modern (1), there are
many ways to define the Legendre polynomials and hence as many different
explicit formulas for them. In this note we will first use their definition as
solutions of the following difference equationaaaLater, in 4, we will also use
their definition as coefficients of a generating function.:

$(n+1)P_{n+1}(t)=(2n+1)tP_{n}(t)-nP_{n-1}(t),\quad\text{for}\quad
n\in\mathbb{N}_{0}.$ (2)

Requiring that $P_{0}(t)=1$ and $P_{1}(t)=t$ this equation defines all
Legendre polynomials uniquely. The first Legendre polynomials are seen in
Table 1 below.

$\begin{array}[]{|l|l|}\hline\cr\quad n&\quad P_{n}(t)\\\ \hline\cr\quad
0&\quad 1\\\ \quad 1&\quad t\\\ \quad 2&\quad\frac{1}{2}(3t^{2}-1)\\\ \quad
3&\quad\frac{1}{2}(5t^{3}-3t)\\\ \quad
4&\quad\frac{1}{8}(35t^{4}-30t^{2}+3)\\\ \quad
5&\quad\frac{1}{8}(63t^{5}-70t^{3}+15t)\\\ \quad
6&\quad\frac{1}{16}(231t^{6}-315t^{4}+105t^{4}-5)\\\ \quad
7&\quad\frac{1}{16}(429t^{7}-693t^{5}+315t^{3}-35t)\\\ \hline\cr\end{array}$
---
Table 1: The first Legendre polynomials for small values of $n$.

We will show that (2) occurs as a special case of more general formulas in
some papers written by Euler (1707-1783). Those papers include “Speculationes
super formula integrali $\int\frac{x^{n}dx}{\sqrt{aa-2bx+cxx}}$, ubi simul
egregiae observationes circa fractiones continuas occurrunt” [5] (E606:
“Speculations about the integral formula
$\int\frac{x^{n}dx}{\sqrt{aa-2bx+cxx}}$, where at the same time extraordinary
observations on continued fractions occur”), “Theorema maxime memorabile circa
formulam integralem
$\int\frac{\partial\phi\cos\lambda\phi}{(1+aa-2a\cos\phi)^{n+1}}$” [6] (E672:
“A most memorable theorem on the integral formula
$\int\frac{\partial\phi\cos\lambda\phi}{(1+aa-2a\cos\phi)^{n+1}}$”),
“Disquitio coniecturalis super formula integrali $\int\frac{\partial\phi\cos
i\phi}{(\alpha+\beta\cos\phi)^{n}}$” [7] (E673, “A conjectural investiagtion
on the integral formula $\int\frac{\partial\phi\cos
i\phi}{(\alpha+\beta\cos\phi)^{n}}$”), “Demonstratio theorematis insignis per
coniecturam eruti circa intagrationem formulae $\int\frac{\partial\phi\cos
i\phi}{(1+aa-2a\cos\phi)^{n+1}}$” [8]bbbThe papers [6], [7], [8] were
essentially part of a single work within the book “Institutionum calculi
integralis”, Volume 4. (E674: “Proof of the extraordinary theorem that had
been found by conjecture on the integral formula $\int\frac{\partial\phi\cos
i\phi}{(1+aa-2a\cos\phi)^{n+1}}$”) and “Specimen transformationis singularis
serierum” [9] (E710: “A specimen of a singular transformation of series”).
Whereas the representation in the first paper seems to be less familiarcccIt
is not listed in the modern table of series and integrals [14] for example.,
the one in the remaining papers can be reduced to the integral:

$P_{n}(t)=\dfrac{1}{\pi}\int\limits_{0}^{\pi}\left(t+\sqrt{t^{2}-1}\cos\varphi\right)^{n}d\varphi,$
(3)

which was given by Laplace (1749-1827) in his work “Traité de mécanique
céleste” [13] (“Treatise on celestial mechanics”) and is also attributed to
him in the preface to Volume 16,2 of Series 1 of Euler’s Opera Omnia ([10]) .
Hence we refer to (3) as the Laplace representation.

We will consider this representation first (see Section 2), before moving on
to the less familiar one (see Section 3).

##  2 Euler and the Laplace Representation for the Legendre Polynomials

In the papers [6], [7], [8] and [9] Euler considered the family of integrals

$A_{n}(a,i):=\int\limits_{0}^{\pi}\dfrac{d\varphi\cos(i\varphi)}{(1+a^{2}-2a\cos(\varphi))^{n}},$
(4)

about which he proved three main resultsdddIn this formula, $i$ denotes an
integer and is not to be confused with $\sqrt{-1}$.. First, he found an
explicit formula for the integrals in (4). Secondly, he discovered a
functional equation satisfied by the family of integrals. And finally, he
stated a difference equation (in $n$) for them. We will discuss each result
separately, stating it and showing how it can be applied to the Legendre
polynomials.

The difference equation of interest for us was given by Euler in § 74 of [7]
as follows:

$n(n-1)(1-aa)^{2}\int\dfrac{d\varphi\cos(i\varphi)}{\Delta^{n+1}}$
$=(n-1)(2n-1)(1+aa)\int\dfrac{d\varphi\cos(i\varphi)}{\Delta^{n}}+(ii-(n-1)^{2})\int\dfrac{d\varphi\cos(i\varphi)}{\Delta^{n-1}},$

where Euler understood the integrals to be taken from $0$ to $\pi$ and used
the abbreviation $\Delta=1-2a\cos(\varphi)+a^{2}$.

Using the notation we introduced in (4), Euler’s difference equation reads as
follows:

$n(n-1)(1-a^{2})^{2}\cdot A_{n+1}(a,i)$ $=(n-1)(2n-1)(1+a^{2})\cdot
A_{n}(a,i)+(i^{2}-(n-1)^{2})\cdot A_{n-1}(a,i).$

For $i=0$, defining $A_{n}(a,0):=A_{n}(a)$ and simplifying the equation, we
arrive at:

$n(1-a^{2})^{2}\cdot A_{n+1}(a)=(2n-1)(1+a^{2})\cdot A_{n}(a)-(n-1)\cdot
A_{n-1}(a).$

After an index shift $n\mapsto n+1$, this equation reads:

$(n+1)(1-a^{2})^{2}\cdot A_{n+2}(a)=(2n+1)(1+a^{2})\cdot A_{n+1}(a)-n\cdot
A_{n}(a).$

To simplify this equation even further, we need to reverse Euler’s
introduction of the letter $a$ that he made in [7]. There, he started from the
integral:

$\int\dfrac{d\varphi\cos(i\varphi)}{(\alpha+\beta\cos(\varphi))^{n}}.$

The integrals we denoted by $A_{n}(a,i)$ in (4) arise from this by writing
$\alpha=1+a^{2}$ and $\beta=-2a$, as Euler also did in § 42 of [7]. Therefore,
the previous difference equation can also be expressed this way:

$(n+1)(\alpha^{2}-\beta^{2})\cdot A_{n+2}(a)=(2n+1)\alpha\cdot
A_{n+1}(a)-n\cdot A_{n}(a).$

Thus, setting $\alpha=x$ and $\beta=\sqrt{x^{2}-1}$ and, for the sake of
brevity, writing simply $A_{n}(x)$ instead of $A_{n}(a)$, we have:

$(n+1)\cdot A_{n+2}(x)=(2n+1)x\cdot A_{n+1}(x)-n\cdot A_{n}(x),$

which is the difference equation for the Legendre polynomials (2). Moreover,
we have $A_{n+1}(x)=\pi P_{n}(x)$, since by direct calculation:

$A_{1}(x)=\int\limits_{0}^{\pi}\dfrac{d\varphi}{x+\sqrt{x^{2}-1}\cos\varphi}=\pi=\pi\cdot
P_{0}(x),$

provided $x\in[-1,1]$. Further, with the same restrictions on $x$,

$A_{2}(x)=\int\limits_{0}^{\pi}\dfrac{d\varphi}{(x+\sqrt{x^{2}-1}\cos\varphi)^{2}}=\pi\cdot
x=\pi\cdot P_{1}(x).$

Therefore, we conclude that

$P_{n}(x)=\dfrac{1}{\pi}\int\limits_{0}^{\pi}\dfrac{d\varphi}{(x+\sqrt{x^{2}-1}\cos(\varphi))^{n}},$

which already looks similar to Laplace’s representation (3).

To derive the Laplace integral representation for the Legendre polynomials
from the last equation, we need a functional equation found by Euler for the
integrals $A_{n}(a,i)$. In § 21 of [9] Euler proved the formulaeeeEuler stated
the same formula in his papers ([6], [7], [8]) without a proof in the first
two papers and also remarked himself that the formula was conjectural at that
point. The first proof was given by Euler in [8].:

$\binom{n+i}{i}(1-aa)^{-n}\int\Delta^{n}d\varphi\cos(i\varphi)$
$=\binom{-n-1+i}{i}(1-aa)^{n+1}\int\Delta^{-n-1}d\varphi\cos(i\varphi),$ (5)

where $\Delta=1-2a\cos(\varphi)+a^{2}$ and the integrals are understood to be
taken from $0$ to $\pi$.

We are again interested in the case $i=0$, in which the binomial coefficients
in (5) become $=1$. After the same substitutions we made earlier to introduce
the letter $x$ instead of $a$, in our notation Euler’s equation reads:

$A_{-n}(a)=A_{n+1}(a),$

where we used that by the previous substitutions
$1-a^{2}=\alpha^{2}-\beta^{2}=x^{2}-(x^{2}-1)=1.$ Combining this with the fact
that $A_{n+1}(a)=\pi\cdot P_{n}(x)$, we also have $A_{-n}=\pi\cdot P_{n}(x)$.
Hence by the definition of $A_{n}(a)$, we also have:

$P_{n}(x)=\dfrac{1}{\pi}\int\limits_{0}^{\pi}(x+\sqrt{x^{2}-1}\cos(\varphi))^{n}d\varphi,$

which is (3).
Therefore, using Euler’s formulas, we were also able to prove the formula

$\int\limits_{0}^{\pi}(x+\sqrt{x^{2}-1}\cos(\varphi))^{n}d\varphi=\int\limits_{0}^{\pi}(x+\sqrt{x^{2}-1}\cos(\varphi))^{-n-1}d\varphi,$
(6)

a formula attributed to Jacobi (1854 - 1851) in [10], who actually also proved
(5) in his paper “Über die Entwickelung des Ausdrucks
$(aa-2aa^{\prime}[\cos\omega\cos\varphi+\sin\omega\sin\varphi\cos(\vartheta-\vartheta^{\prime})]+a^{\prime}a^{\prime})^{-\frac{1}{2}}$”
[12] (“On the expansion of the expression
$(aa-2aa^{\prime}[\cos\omega\cos\varphi+\sin\omega\sin\varphi\cos(\vartheta-\vartheta^{\prime})]+a^{\prime}a^{\prime})^{-\frac{1}{2}}$”).
But the argument can be made that Euler could also have proven it, if he had
the intention. For, his functional equation (5) is much more general than (6).
Furthermore, (5) is a special case of the Euler transformation for the
hypergeometric series, i.e., the formula

${}_{2}F_{1}(a,b,c;x)=(1-x)^{c-a-b}{}_{2}F_{1}(c-a,c-b,c;x),$

which Euler proved in § 9 of [9]. Here, ${}_{2}F_{1}$ is the Gaussian
hypergeometric series. Therefore, (6) can be understood as a special case of a
certain property of the Gaussian hypergeometric function.

For the sake of completeness, we also state an explicit formula that Euler
gave for the integrals $A_{n}(a,i)$. The formula reads as follows (see, e.g.,
§ 43 in [7]):

$\int\limits_{0}^{\pi}\dfrac{d\varphi\cos(i\varphi)}{(1+aa-2a\cos(\varphi))^{n+1}}=\dfrac{\pi
a^{i}}{(1-a^{2})^{2n+1}}\cdot V,$

where

$V=\binom{n+i}{i}+\binom{n+i}{i+1}\binom{n-i}{1}a^{2}+\binom{n+i}{i+2}\binom{n-i}{2}a^{4}+\cdots$

While Euler himself noted in [6] and [7] that the formula for $V$ was
conjectural, he gave a proof in [8].

##  3 Euler’s other Integral Representation

As mentioned in the introduction, the other integral representation for the
Legendre polynomials originates in Euler’s paper [5], a paper originally
devoted to continued fractions. Having shown in his papers “De fractionibus
continuis observationes” [1] (E123: “Observations on continued fractions”) and
“Methodus inveniendi formulas integrales, quae certis casibus datam inter se
teneant rationem, ubi simul methodus traditur fractiones continuas summandi”
[4] (E594: “A method to find integral formulas which in certain case are in a
given ratio to each other, where at the same a method is given to sum
continued fractions”) how homogeneous difference equations with linear
coefficients can be solved by means of integrals, in [5], Euler applied his
theory to the family of integrals

$G(n):=\int\limits_{\frac{b-\sqrt{b^{2}-a^{2}c}}{c}}^{\frac{b+\sqrt{b^{2}-a^{2}c}}{c}}\dfrac{x^{n}dx}{\sqrt{a^{2}-2bx+cx^{2}}}$

and showed that those integrals satisfy the equation:

$na^{2}\cdot G(n-1)=(2n+1)b\cdot G(n)-(n+1)c\cdot G(n+1).$

Specializing to the case $a=c=1$ and writing $t$ for $b$, this equation
reduces to the difference equation for the Legendre polynomials (2) such that
we can conclude

$C(t)\cdot
P_{n}(t)=\int\limits_{t-\sqrt{t^{2}-1}}^{t+\sqrt{t^{2}-1}}\dfrac{x^{n}dx}{\sqrt{1-2xt+x^{2}}}.$

$C(t)$ is a function independent of $n$ that we have to determine.

To do this, we evaluate the integrals for $n=0$ and $n=1$ explicitly. By a
formal calculation, we have:

$\int\limits_{t-\sqrt{t^{2}-1}}^{t+\sqrt{t^{2}-1}}\dfrac{x^{0}dx}{\sqrt{1-2xt+x^{2}}}=\log(-1)$

and

$\int\limits_{t-\sqrt{t^{2}-1}}^{t+\sqrt{t^{2}-1}}\dfrac{x^{1}dx}{\sqrt{1-2xt+x^{2}}}=\log(-1)\cdot
t.$

Hence, it turns out that the function $C(t)$ is actually a constant.
Unfortunately, $\log(-1)$ is not uniquely defined; we have to choose a
specific branch of the complex logarithm. Thus, if we write

$P_{n}(t)=\dfrac{1}{\log(-1)}\cdot\int\limits_{t-\sqrt{t^{2}-1}}^{t+\sqrt{t^{2}-1}}\dfrac{x^{n}dx}{\sqrt{1-2xt+x^{2}}},$
(7)

this equation gives us the correct formula, as long as we do not care about
the ambiguous meaning of the symbol $\log(-1)$. If we want to avoid this
ambiguity, we can use the indefinite integral

$\int\dfrac{x^{n}dx}{\sqrt{1-2xt+x^{2}}}$
$=P_{n}(t)\operatorname{artanh}\left(\dfrac{x-t}{\sqrt{1-2xt+x^{2}}}\right)+Q_{n}(x,t)\sqrt{1-2xt+x^{2}}+C$

where $Q_{n}$ is a polynomial in $x$ and $t$ and $C$ is a constant of
integration. This formula can be proved by induction and the difference
equation for the Legendre polynomials (2). Note that we do not need to know
the polynomial $Q_{n}$ explicitly, since from the boundaries of integration in
the previous equation we have $Q_{n}(x,t)\sqrt{1-2xt+x^{2}}=0$.

##  4 Another Approach leading to similar Insights

As seen in the introduction, the Legendre polynomials can also defined as
coefficients of a generating function (see (1)). In this section we intend to
show that Euler was led to the aforementioned generating function and (2) in
his investigations in the papers “Observationes analyticae” [2] (E326:
“Analytical Observations”) and “Varia artificia in serierum indolem
inquirendi” [3] (E551: “Various Artifices to investigate the Nature of
Series”). In both these papers, Euler considered the general trinomials
$(a+bx+cx^{2})^{n}$ for $n\in\mathbb{N}$ and $a,b,c\in\mathbb{C}$ and was
interested in the sequence of the coefficients of the power $x^{n}$ in the
expansion of those trinomials. Therefore, let us assume

$(a+bx+cx^{2})^{n}=a^{n}+\cdots+B_{n}\cdot x^{n}+\cdots+c^{n}x^{2n},$

where the coefficient $B_{n}$ depends on the numbers $a$, $b$ and $c$. From
these coefficients Euler then defined a Taylor series:

$F(x):=\sum_{n=0}^{\infty}B_{n}x^{n}.$ (8)

The problem he solved in [2] and [3] isfffThe case $a=b=c=1$ gives
coefficients comprising sequence A002426 in the Online Encyclopedia of Integer
Sequences (https://oeis.org/).: What is a closed expression for $F(x)$? His
answer to the problem (see, e.g., § 21 of [3]) reads:

$F(x):=\dfrac{1}{\sqrt{1-2bx+(b^{2}-4ac)x^{2}}}.$

This expression bears close resemblance to the generating function for the
Legendre polynomials in (1). Indeed, writing $t$ for $b$ and setting
$b^{2}-4ac=1$ in Euler’s solution, we arrive at the generating functions for
the Legendre polynomials.

Another of Euler’s important contributions in [3] was his discovery of the
difference equation between three consecutive Legendre polynomials, i.e, (2).
Euler gave the following relation among the consecutive expansion coefficients
of the Taylor series for $F$ in (8) in § 22 of [3]:

$r=bq+\dfrac{n-1}{n}(bq-p).$

Here, Euler suppressed the dependence of $r,p,q$ on $n$ and also on the letter
$b$, which is the same $b$ as in explicit expression for the function $F$. But
Euler told us that, $r$ being the $n$-th term in the sequence of coefficients,
we obtain $q$ from $r$ by writing $n-1$ instead of $n$ in $r$ and likewise,
$p$ from $r$ by writing $n-2$ instead of $n$ in $r$. Thus, writing $t$ instead
of $b$ and presenting everything in modern fashion with indices, i.e., setting
$r=B_{n}(t)$, $q=B_{n-1}(t)$ and $p=B_{n-2}(t)$, Euler’s difference equation
becomes:

$B_{n}(t)=t\cdot B_{n-1}(t)+\dfrac{n-1}{n}(t\cdot B_{n-1}(t)-B_{n-2}(t)).$

A simple calculation shows that this equation is equivalent to:

$nB_{n}(t)=(2n-1)t\cdot B_{n-1}(t)-(n-1)B_{n-2}(t).$

After a shift $n\mapsto n+1$, this equation is to be seen to be identical to
the difference equation for the Legendre polynomials (2).

As a side note we mention that, we can use Euler’s approach and define the
Legendre polynomials as follows:

######  Definition 1.

The Legendre polynomial $P_{n}(t)$ is given as the coefficient of the power
$x^{n}$ the expansion of the the following trinomial:

$T(t,x)=\left(\dfrac{t-1}{2}+xt+\dfrac{t+1}{2}x^{2}\right)^{n}.$

This definition allows to calculate the Legendre polynomials, since, as we saw
above, for this particular choice of the letters $a,b,c$ in the assumed form
of the expanded trinomial the function $F$ in (8) reduces to the generating
function in (1).

##  5 Conclusion

In this note we showed that Euler’s discoveries lead to two integral
representations for the Legendre polynomials. The first equation (3) that we
derived from Euler’s many investigations ([6], [7], [8] and [9]) is familiar
and is attributed to Laplace ([10]), whereas the other equation (7), which
follows from Euler’s formulas in ([5]), seems to be less known, since it does
not appear in a standard integral reference tables ([14]). Furthermore, we
showed that Euler’s investigations in [2] and [3] contain the generating
function for the Legendre polynomials (1) as a special case and the difference
equation (2) was stated explicitly in [3]. Additionally, the papers [5], [7]
include it as a special case, the general equation being (5). Unfortunately,
it seems that Euler himself never made the connection between his findings,
although they all led him to (2).

Euler himself did not know the Legendre polynomials and their definition; they
were introduced by Legendre in [11] in his studies of the gravitational
potential after Euler’s death. Therefore, it is even more interesting that
they also appeared in Euler’s investigations, which were not motivated by a
problem in physics. Indeed, Euler’s papers rather seem like curious
intellectual challenges without any clear origin. Hence this should provide us
with a lot of motivation to consider more of Euler’s papers from a modern
perspective. They might even lead to new interpretations of familiar results
and contain more insights attributed to other mathematicians like (5), a
formula attributed to Jacobi in [10] who actually proved it later than Euler
in [12].

## References

  * [1] Euler, L. (1739). “De fractionibus continuis observationes” (E123). Commentarii academiae scientiarum Petropolitanae, Volume 11, 1750: pp. 32–81. Reprinted in Opera Omnia: Series 1, Vol. 14, pp. 291–349. Original text available online at https://scholarlycommons.pacific.edu/euler/.
  * [2] Euler, L. (1763). “Observationes analyticae” (E326). Novi Commentarii academiae scientiarum Petropolitanae, Volume 11, 1767: pp. 124–143. Reprinted in Opera Omnia: Series 1, Vol. 15, pp. 50–69. Original text available online at https://scholarlycommons.pacific.edu/euler/.
  * [3] Euler, L. (1772). “Varia artificia in serierum indolem inquirendi” (E551). Opuscula Analytica, Volume 1, 1783: pp. 48–63. Reprinted in Opera Omnia: Series 1, Vol. 15, pp. 383–399. Original text available online at https://scholarlycommons.pacific.edu/euler/. English translation by Cynthia Huffman, PH.D. (2023) available online at: https://scholarlycommons.pacific.edu/euleriana/.
  * [4] Euler, L. (1775). “Methodus inveniendi formulas integrales, quae certis casibus datam inter se teneant rationem, ubi simul methodus traditur fractiones continuas summandi” (E594). Opuscula Analytica, Volume 2, 1785: pp. 178–216. Reprinted in Opera Omnia: Series 1, Volume 18, pp. 209 – 243. Original text available online at https://scholarlycommons.pacific.edu/euler/.
  * [5] Euler, L. (1775). “Speculationes super formula integrali $\int\frac{x^{n}dx}{\sqrt{aa-2bx+cxx}}$, ubi simul egregiae observationes circa fractiones continuas occurrunt” (E606). Acta Academiae Scientarum Imperialis Petropolitinae, Volume 1782: II, 1786: pp. 62–84. Reprinted in Opera Omnia: Series 1, Volume 18, pp. 244 – 264. Original text available online at https://scholarlycommons.pacific.edu/euler/.
  * [6] Euler, L. (1778). “Theorema maxime memoragile circa formulam integralem $\int\frac{\partial\phi\cos\lambda\phi}{(1+aa-2a\cos\phi)^{n+1}}$” (E672). Institutionum calculi integralis, Volume 4, 1794: pp. 194–217. Reprinted in Opera Omnia: Series 1, Volume 19, pp. 141–167. Original text available online at https://scholarlycommons.pacific.edu/euler/.
  * [7] Euler, L. (1778). “Disquitio coniecturalis super formula integrali $\int\frac{\partial\phi\cos i\phi}{(\alpha+\beta\cos\phi)^{n}}$” (E673). Institutionum calculi integralis, Volume 4, 1794: pp. 217–242. Reprinted in Opera Omnia: Series 1, Volume 19, pp. 168–196. Original text available online at https://scholarlycommons.pacific.edu/euler/.
  * [8] Euler, L. (1778). “Demonstratio theorematis insignis per coniecturam eruti circa intagrationem formulae $\int\frac{\partial\phi\cos i\phi}{(1+aa-2a\cos\phi)^{n+1}}$” (E674). Institutionum calculi integralis, Volume 4, 1794: pp. 242–259. Reprinted in Opera Omnia: Series 1, Volume 19, pp. 197–216. Original text available online at https://scholarlycommons.pacific.edu/euler/.
  * [9] Euler, L. (1778). “Specimen transformationis singularis serierum” (E710). Nova Acta Academiae Scientiarum Imperialis Petropolitanae, Volume 12, 1801: pp. 242–259. Reprinted in Opera Omnia Series 1, Volume 16,2, pp. 41–55. Original text available online at https://scholarlycommons.pacific.edu/euler/.
  * [10] Faber, G. (1935). Preface to Leonhardi Euleri Commentationes Analyticae Ad Theoriam Serierum Infinitarum Pertinentes – Volumen Tertium Sectio Altera Edidit Carl Boehm., Leonhard Euler’s Opera Omnia, Series 1, Volume 16,2, printed in Basel 1935.
  * [11] A. Legendre, A. (1782). “Recherches sur l’attraction des sphéroïdes homogènes”, Mémoires de Mathématiques et de Physique, présentés à l’Académie Royale des Sciences, par divers savans, et lus dans ses Assemblées Vol. X., 1785: pp 411–435.
  * [12] Jacobi, C. (1843). Über die Entwickelung des Ausdrucks $(aa-2aa^{\prime}[\cos\omega\cos\varphi+\sin\omega\sin\varphi\cos(\vartheta-\vartheta^{\prime})]+a^{\prime}a^{\prime})^{-\frac{1}{2}}$. Crelle Journal für die reine und die angewandte Mathematik, Volume 26, 1843: pp. 81-87.
  * [13] Laplace, P.-S. (1825). Traité de mécanique céleste, Tome 5, Livre IX, Chap. II., De L’Imprimerie de Crapelet. Original text available online at: https://catalog.hathitrust.org/Record/100242426.
  * [14] D. Zwillinger (Editor) (2014). Table of Integrals, Series, and Products, Academic Press, 8th edition.


# Generalized Radiograph Representation Learning via Cross-supervision between
Images and Free-text Radiology Reports

Hong-Yu Zhou1,†, Xiaoyu Chen2,†, Yinghao Zhang2,†, Ruibang Luo1,
Liansheng Wang2,∗, Yizhou Yu1,∗
1 Department of Computer Science, The University of Hong Kong,
Pokfulam, Hong Kong
2 Department of Computer Science, Xiamen University,
Xiamen, China
† These authors contributed equally
∗ Corresponding authors: L.Wang<EMAIL_ADDRESS>and
Y.Yu<EMAIL_ADDRESS>
###### Abstract

Pre-training lays the foundation for recent successes in radiograph analysis
supported by deep learning. It learns transferable image representations by
conducting large-scale fully-supervised or self-supervised learning on a
source domain. However, supervised pre-training requires a complex and labor
intensive two-stage human-assisted annotation process while self-supervised
learning cannot compete with the supervised paradigm. To tackle these issues,
we propose a cross-supervised methodology named REviewing FreE-text Reports
for Supervision (REFERS), which acquires free supervision signals from
original radiology reports accompanying the radiographs. The proposed approach
employs a vision transformer and is designed to learn joint representations
from multiple views within every patient study. REFERS outperforms its
transfer learning and self-supervised learning counterparts on 4 well-known
X-ray datasets under extremely limited supervision. Moreover, REFERS even
surpasses methods based on a source domain of radiographs with human-assisted
structured labels. Thus REFERS has the potential to replace canonical pre-
training methodologies.

##  1 Introduction

Medical image analysis has achieved tremendous progress in recent years,
thanks to the development of deep convolutional neural networks (DCNNs) [1, 2,
3, 4, 5]. At the core of DCNNs is visual representation learning [6], where
pre-training has been widely adopted and become the most dominant approach to
obtain transferable representations. Typically, a large-scale dataset, also
called the source domain, is first used for model pre-training. Transferable
representations from the pre-trained model are further fine-tuned on other
smaller downstream datasets, called target domains.

As one of the most general forms of medical images, radiographs have a great
potential to be used in widespread applications [7, 8, 9]. In order to achieve
(or at least approximate) radiologist-level diagnosis performance in these
applications, it is common to transfer learned representations from natural
images to radiographs [10, 11], and ImageNet [12] based pre-training is most
widely adopted in this context. On the other hand, self-supervised learning
[13, 14, 15, 16] has attracted much attention in the community because it is
capable of learning transferable radiograph representations without any human
annotations. Both methodologies have been proven to be effective in solving
medical image analysis tasks, especially when the amount of labeled data in
the target domain is quite limited. However, in the first approach, there is
an inevitable problem, which is the existence of domain shifts between medical
and natural images. For instance, it is possible to introduce harmful noises
from natural images as radiographs have a different pixel intensity
distribution. As for self-supervised learning, to the best of our knowledge,
there still exist clear performance gaps between radiograph representations
learned through self-supervised and label-supervised pre-training. To avoid
these problems, building large-scale annotated radiograph datasets for label-
supervised pre-training becomes an essential and urgent issue in radiograph
analysis.

Recently, radiologists and computer scientists have managed to build medical
datasets for label-supervised pre-training at the size of hundreds of
thousands of images, such as ChestX-ray [11], MIMIC [17] and CheXpert [18]. To
acquire accurate labels for radiographs, these datasets often rely on a two-
stage human intervention process. A radiology report is first prepared by
radiologists for every patient study as part of the clinical routine. In the
second stage, human annotators extract and confirm structured labels from
these reports using artificial rules and existing natural language processing
(NLP) tools. However, there are two major limitations of this label extraction
workflow. First, it is still complex and labor intensive. For example, human
annotators have to define a list of alternate spellings, synonyms, and
abbreviations for every target label. Consequently, the final accuracy of
extracted labels heavily depends on the quality of human assistance and
various NLP tools. A small mistake in a single step or a single tool may give
rise to disastrous annotation results. Second, those human-defined rules are
often severely restricted to application-oriented tasks instead of general-
purpose tasks. It is difficult for DCNNs to learn universal representations
from such application-oriented tasks.

In this paper, we propose REviewing FreE-text Reports for Supervision (REFERS)
to directly learn radiograph representations from accompanying free-text
radiology reports. We believe abstract and complex logic reasoning sentences
in radiology reports provide sufficient information for learning well-
transferable visual features. As shown in Figure 1a, REFERS is realized using
a set of transformers, where the most important part is a radiograph
transformer serving as the backbone. The main reason why we choose the
transformer as the backbone in REFERS is that it not only exhibits the
advantages of DCNNs, but also has been shown to be more effective [19] because
of the self-attention mechanism [20]. Moreover, we have found that, in
comparison to features generated from DCNNs, features from transformers are
more compatible with textual tasks.

Different from aforementioned representation learning methodologies, REFERS
performs cross-supervised learning and does not need structured labels during
the pre-training stage. Instead, supervision signals are defined by
automatically cross-checking the two different data modalities, radiographs
and free-text reports. Considering in daily clinical routine, there is
typically a free-text report associated with every patient study, which
usually involves more than one radiographs. To fully utilize the study-level
information in each report, we design a view fusion module based on an
attention mechanism to process all radiographs in a patient study
simultaneously, and fuse the resulting multiple features. In this way, the
learned representations are able to preserve both study-level and image-level
information. In contrast, only image-level information is addressed in
traditional representation learning paradigms [11, 13, 14, 15, 16] that use a
single image as input. On top of the view fusion module, we conduct two tasks,
i.e., report generation and study-report representation consistency
reinforcement, to extract study-level supervision signals from free-text
reports. To carry out the first task, we apply a decoder, called report
transformer, to the fused feature with the goal to reproduce the radiology
report associated with the study. For the second task, we apply our radiograph
transformer and an NLP transformer to a study-report pair. These transformers
produce a pair of feature representations for the patient study and radiology
report in the pair, respectively. The consistency between such a pair of
feature representations within every study-report pair is reinforced via a
contrastive loss function. Some previous works [21, 22] tried to learn joint
text-image representations for single-domain medical image analysis tasks.
Compared to them, REFERS focuses on learning well-transferable image features
from study-level free-text reports on a large-scale source domain and fine-
tuning them on one or more target domains.

On four well-known X-ray datasets, REFERS outperforms self-supervised learning
and transfer learning on natural source images in producing more transferable
representations, often bringing impressive improvements (more than 5%) under
limited supervision from target domains. This capability can be extremely
important in real-world applications as medical data is scarce and their
annotations are usually hard to acquire. More surprisingly, we found that
REFERS clearly surpasses those methods that employ a source domain with a
large collection of medical images with structured labels. In terms of
specific abnormalities and diseases, REFERS is quite effective under extremely
limited supervision ($<$ 1k annotated radiographs during fine-tuning). For
instance, REFERS brings about 9-percent improvements on pneumothorax.
Meanwhile, over 7-percent improvements are achieved on two common lung
diseases (atelectasis and emphysema).

##  2 Results

All self-supervised learning (SSL) and label-supervised pre-training (LSP)
baselines as well as our REFERS are first pre-trained on a source domain of
medical images (i.e., MIMIC-CXR-JPG [23]). Then, pre-trained models are fine-
tuned on each of four well-established datasets (target domains with labels),
including NIH ChestX-ray [11], VinBigData Chest X-ray Abnormalities Detection
[24], Shenzhen Tuberculosis [25] and COVID-19 Image Data Collection [26].
During the fine-tuning stage, we always perform fully-supervised learning on
the target domain, which only consists of radiographs with structured labels.
Furthermore, we verify model performance by varying the percentage of actually
used training images (sampled from the predefined whole training set) in the
target domain, and this percentage is called label ratio. When the label ratio
is 100%, we use the whole training set in the target domain for fine-tuning.

NIH ChestX-ray. Table 1, Extended Data Figures 4 and 6 present experimental
results from our REFERS and other approaches under different label ratios. As
shown in Table 1 and Extended Data Figure 4, our approach significantly
outperforms self-supervised baselines and transfer learning on natural source
images. To be specific, REFERS achieves the highest AUC on all 14 classes
using different amounts of training data during the fine-tuning stage.
Moreover, REFERS shows the largest performance improvements with respect to
these baselines when only 0.8k training images (1% label ratio) in the target
domain are utilized. For example, REFERS surpasses the widely adopted
ImageNet-based pre-training [11] by about 7 percents on average. Even when
compared to LSP, our REFERS still gives quite competitive results. In Table 2,
it is easy to find out that the average performance of REFERS actually
surpasses LSP, and consistently maintains an advantage of at least 2 percents.
Compared to self-supervised baselines [13, 14, 15, 16] and ImageNet-based pre-
training [11], REFERS achieves the largest improvements on emphysema (7
percents) and cardiomegaly ($>$ 10 percents), especially under limited
supervision. When compared to LSP, our method achieves consistent improvements
on mass ($>$ 4 percents).

VinBigData Chest X-ray Abnormalities Detection. Our REFERS exhibits more
advantage on this target domain dataset than it does on NIH ChestX-ray as
VinBigData comprises a much smaller number of annotated radiographs (about
$\frac{1}{8}$ of the NIH dataset). This phenomenon again demonstrates the
ability of REFERS in dealing with limited supervision. REFERS consistently
maintains large advantages over other methods under different conditions (see
Tables 1, 2, Extended Figures 5 and 7). For instance, when we only have 105
annotated radiographs (1% label ratio) as fine-tuning data, REFERS surpasses
C2L [16], the best performing self-supervised method, by over 7 percents in
AUC. The performance of REFERS once again surpasses LSP with human-assisted
structured labels even when all annotated training data (100% label ratio) in
the target domain is used. When we check specific abnormalities and diseases,
we found REFERS consistently improves the diagnosis of atelectasis, lung
opacity and pneumothorax in comparison to LSP.

COVID-19 and Shenzhen Tuberculosis Image Collections Both datasets serve as
target domains and comprise a small number of labeled images (fewer than 1k
X-rays), which are employed to test the transferability of the representation
learned on the source domain. This is because few training images in such
small target domains are not capable of training powerful models themselves.
Thus, the performance of the trained models is more dependent on the quality
of the learned representation. In Table 1, although separating tuberculosis
from normal cases is not a hard task, our method still achieves 2.5%
improvements over C2L [16] in AUC. When looking at COVID-19 Image Data
Collection which includes two harder tasks, we can find that the relative
performance improvements over self-supervised baselines [13, 14, 15, 16] and
transfer learning on natural source images [11] become quite clear. For
instance, on the “Viral vs. Bacterial” task, REFERS outperforms C2L [16] by 7
percents in AUC, demonstrating the effectiveness of REFERS in helping achieve
better performance over small-scale target datasets. Even if we compare REFERS
against LSP, the performance advantage is still maintained at more than 1
percent.

##  3 Discussion

REFERS outperforms self-supervised learning and transfer learning on natural
source images by substantial and significant margins. This is the most
prominent observation obtained from our experimental results, which holds on
different datasets and with different amounts of annotated training data
during fine-tuning. Among self-supervised baselines [13, 14, 15, 16], C2L [16]
and TransVW [15] are the two best performing methods. Our REFERS outperforms
C2L and TransVW by at least 4 percents when very limited annotated training
data (at most 10% label ratio) from NIH ChestX-ray and VinBigData datasets is
used. Somewhat interestingly, as the label ratio increases, ImageNet-based
pre-training [11] gradually narrows its gap with self-supervised learning.
Nonetheless, our REFERS still surpasses it by a large margin (4 percents at
least). Similar results can also be observed on Shenzhen Tuberculosis and
COVID Image Collection. Since our REFERS employs a cross-supervised learning
manner, it does not require structured labels as conventional fully-supervised
learning approaches. As radiographs and radiology reports are readily
available medical data, we believe our approach is as practical as self-
supervised learning methodologies in real-world scenarios.

REFERS consistently surpasses label-supervised pre-training with human-
assisted structured labels. This is another clear observation obtained from
our experimental results. Even though our approach does not use any structured
labels in the source domain, over all four target domain datasets, our pre-
trained model exhibits clear advantages. Specifically, REFERS outperforms the
most competitive LSP method, LSP (Transformer), which is based on Transformer
and human-assisted structured labels in the source domain. In particular, our
method shows more advantages at small label ratios. For instance, when NIH
ChestX-ray and VinBigData are used as target domain datasets, REFERS achieves
about 2.5% improvements when the number of training images is smaller than
10k. Similarly, on Shenzhen Tuberculosis and COVID-19 Data Collection, REFERS
consistently surpasses LSP by significant margins. It is worth mentioning that
when a classification problem is difficult to solve and has limited
supervision, REFERS becomes more advantageous and achieves impressive
improvements. For example, on the “Viral vs. Bacterial” task (Table 2), REFERS
surpasses label-supervised pre-training methods based on two-stage human
intervention by approximately 4 percents. These improvements demonstrate that
raw radiology reports contain more useful information than human-assisted
structured labels. In other words, the advantages exhibited by our approach on
small-scale target domain training data can be attributed to the rich
information carried by radiology reports in the source domain. Such
information provides additional supervision to help learn transferable
representations for radiographs while the supervision signals from structured
labels have less information. We believe this is an important step towards
directly using natural language descriptions as supervision signals for image
representation learning. As an example, our REFERS can be used to learn
natural image representations from text descriptions at corresponding
websites.

REFERS significantly reduces the need of annotated data in target domains.
Figures 2a and 2b present the performance of our approach under various label
ratios. On NIH ChestX-ray, REFERS needs 90% fewer annotated target domain data
(10% label ratio) to deliver a performance comparable to those of Model
Genesis [14] and ImageNet-based pre-training [11]. Similarly, on VinBigData,
our method only needs 10% annotated training data to achieve much better
results than those of Model Genesis and ImageNet-based pre-training under 100%
label ratio. This phenomenon shows the potential of REFERS in providing high-
quality pre-trained representations for downstream fine-tuning tasks with
limited annotations. Due to the difficulty to acquire reliable annotations for
medical image analysis, the ability to achieve good performance with limited
annotations means much to the community.

Improvements on specific abnormalities and diseases. In Extended Data Figures
6, REFERS brings 5-percent performance gains on emphysema and mass even when
compared to LSP with limited supervision in the target domain ($<$ 10k
training images). Since both abnormalities have a dispersed spatial
distribution in the lung area, the considerable improvements demonstrate that
REFERS is able to handle elusive chest abnormalities in radiographs well. When
the amount of supervision in the target domain becomes extremely limited, such
as using 105 training images from VinBigData, REFERS becomes more
advantageous. For instance, REFERS outperforms LSP on atelectasis and
pneumothorax by over 7 and 9 percents in Extended Data Figures 7,
respectively. Different from emphysema, mass and atelectasis, pneumothorax
maintains a concentrated spatial distribution and is often located around the
pleura. These successes imply that REFERS can deal with the diagnosis of both
elusive and regular abnormalities and diseases well using a small number of
training radiographs in the target domain. A similar phenomenon can be
observed when REFERS is used for distinguishing viral pneumonia cases from
bacterial ones in Tables 1 and 2.

Transformer is more effective under limited supervision. In Tables 1 and 2, we
observe a trend of CNNs (i.e., ResNet series [4]): LSP (ConvNet) shows
mediocre performance when a relatively small number of training images in the
target domain are used. However, when all training data (100% label ratio) is
used, ConvNet shows competitive results. It seems that LSP (ConvNet) cannot
well handle little amount of supervision. In contrast, LSP (Transformer)
exhibits much better performance at small label ratios. This comparison
demonstrates that pre-trained transformers generate more transferable
representations than pre-trained CNNs. The underlying reason might be that the
self-attention mechanism in transformers makes the learned representations
more transferable due to captured long-distance dependencies.

REFERS provides reliable evidences for clinical decisions. Figure 3 presents
randomly chosen radiographs and their corresponding class activation maps
(CAMs) [27]. We can find that REFERS generates reliable attention regions, on
top of which we can apply a fixed confidence threshold to further identify the
location of different types of lesions (green boxes in Figure 3). The overall
IoUs (Intersection over Unions) between green and red boxes (drawn by
radiologists) are mostly higher than 0.5, indicating that the generated
attention regions can well match radiologists’ diagnoses. When lesions have a
large size (such as the fifth image from NIH ChestX-ray), our method captures
well-aligned lesion areas. Even when lesions are quite small and thus hard to
detect (such as the last image from NIH ChestX-ray and the first image from
VinBigData), REFERS can still identify the right locations.

Replication of experimental results and their statistical significance. There
are a number of factors that influence pre-training results exhibit a certain
level of randomness. These factors include, but are not limited to network
initialization, training strategy (e.g., how to randomly crop images and
perform mini-batch gradient descent) and even non-deterministic
characteristics in computational tools (e.g., cuDNN [28] would choose
different algorithms in different runs due to benchmarking noise and hardware
configuration). A good pre-training methodology should be able to produce
relatively stable pre-trained representations when randomness in these factors
is controlled within an acceptable limit. To take into account the influence
of such randomness on experimental results, when REFERS and baseline pre-
trained models are fine-tuned, we independently repeat each experiment three
times and report their average results in Tables 1 and 2. Then, we calculate
p-values between mean class AUCs of our REFERS and the best performing
baseline model according to their fine-tuned performance using independent
two-sample t-test. According to Tables 1 and 2, nearly all p-values are much
smaller than 0.01, indicating that our REFERS is significantly better than its
counterparts when various amounts of labeled training data in the target
domain is used. In contrast, making the number of times (repeating each
experiment) smaller than three would give rise to less stable mean AUCs while
simply repeating more times would produce meaninglessly smaller p-values.

Last but not the least, we provide a thorough ablation study of REFERS in
Table 3. More details can be found in the Methods section.

##  4 Methods

Dataset for pre-training (source domain). MIMIC-CXR-JPG [23] contains over
370k radiographs organized into patient studies, each of which may have one or
more radiographs taken from different views or at different times for the same
patient. Each patient study has one free-text radiology report, and each
radiograph is associated with a set of abnormality/disease labels obtained
from two-stage human-assisted intervention as mentioned above. There are two
major sections in each report: Findings and Impressions. The Findings section
includes detailed descriptions of important aspects in the radiographs while
the Impressions section summarizes most immediately relevant findings.

To acquire human-assisted structured labels for radiographs (i.e., two-stage
human intervention), annotators need to first define a list of labels for
abnormalities and diseases, including alternate spellings, synonyms, and
abbreviations. On the basis of local contexts and existing NLP tools, mentions
of labels in reports are classified as positive, uncertain, or negative. An
aggregation procedure is further applied to aggregate multiple mentions of a
single label. Uncertain labels need to be double-checked by radiologists.

As radiology reports were originally prepared by radiologists as part of the
daily clinical routine, they can be regarded as freely available information
that does not require extra human efforts in contrast to structured labels. In
practice, we only keep the Findings and Impressions sections in the reports.
Also, we remove all study-report pairs, where the text section has less than 3
tokens (words and phrases), from the dataset. This screening procedure
produces 217k patient studies.

Datasets for fine-tuning (target domains). We do not require these datasets
adopted for fine-tuning to have radiology reports. Instead, only human-
assisted annotations are used during the fine-tuning stage. We follow the
official split of NIH ChestX-ray, where the percentages of training,
validation and testing sets are 70%, 10% and 20%, respectively. The same set
of ratios are also employed for VinBigData Chest X-ray, Shenzhen Tuberculosis
and COVID-19 Image Data Collection to build randomly split training,
validation and testing sets.

  * •

_NIH ChestX-ray_ is a dataset for multi-label classification of 14 chest
abnormalities (i.e., Atelectasis, Cardiomegaly, Consolidation, Edema,
Effusion, Emphysema, Fibrosis, Hernia, Infiltration, Mass, Nodule, Pleural
Thickening, Pneumonia and Pneumothorax). There are over 100k frontal-view
X-ray images of about 32k patients in NIH ChestX-ray, where labels of
radiographs were extracted from associated reports following a similar
procedure as that for MIMIC-CXR-JPG.

  * •

_VinBigData Chest X-ray_ provides labels of 14 chest diseases (i.e., Aortic
enlargement, Atelectasis, Pneumothorax, Lung Opacity, Pleural thickening, ILD,
Pulmonary fibrosis, Calcification, Pleural effusion, Consolidation,
Cardiomegaly, Other lesion, Nodule-Mass and Infiltration), and consists of 15k
postero-anterior chest X-ray images. Here we did not use the test set in
Kaggle, which does not provide any annotations. All images were labeled by a
panel of experienced radiologists.

  * •

_Shenzhen Tuberculosis_ is a small dataset containing 662 frontal chest X-ray
images primarily from hospital clinical routine. 336 abnormal X-rays show
various manifestations of tuberculosis, and the remaining 326 images are
normal. We simply perform binary classification on this dataset.

  * •

_COVID-19 Image Data Collection_ is a dataset involving more than 900
pneumonia cases with chest X-rays, which was built to improve the
identification of COVID-19. We conduct experiments on two tasks, which are a)
distinguishing COVID-19 from the rest and b) separating viral pneumonia cases
from bacterial ones.

Baselines and label-supervised pre-training. Since our method does not need
structured labels required by traditional fully-supervised learning, we
compare it against four recent self-supervised learning methods [13, 14, 15,
16] and ImageNet-based pre-training [11]:

  * •

_Context Restoration_ [13] repeats the operation of swapping two randomly
chosen small X-ray patches for a fixed number of times, and the neural network
is asked to restore each altered image back to its original version.

  * •

_Model Genesis_ [14] applies multiple types of distortions to the input X-ray,
including local shuffling, non-linear transformation, in- and out-painting.
Similar to Context Restoration, Model Genesis asks the model to reconstruct
the original image from the distorted one.

  * •

_TransVW_ [15] contrasts local X-ray patches to exploit the semantics of
anatomical patterns while restoring distorted image contents.

  * •

_C2L_ [16] proposes to construct homogeneous and heterogeneous data pairs by
mixing both images and features on top of MoCo [29]. C2L outperforms MoCo by
observable margins on multiple X-ray benchmarks.

  * •

_ImageNet-based pre-training_ [11] is taken as a representative method that
sets a large-scale dataset of annotated natural images as the source domain.

Note that all above baselines are implemented using the same transformer-based
network architecture as our REFERS (i.e, a ViT architecture plus the proposed
recurrent concatenation module). Such an implementation arrangement is meant
to rule out the influence of network architectures on final performance and
maintain fairness in experimental comparisons.

Finally, our approach is compared against label-supervised pre-training (LSP)
that directly sets a large collection of X-ray images with human-assisted
structured labels as the source domain. For better comparison, we implement
LSP on top of both CNN and Transformer based backbone networks. Specifically,
LSP (Transformer) adopts the same Transformer based network architecture as
REFERS and the aforementioned self-supervised and ImageNet-based pre-training
baselines. LSP (ConvNet) stands for the best performing residual network among
ResNet-18, ResNet-50 and ResNet-101 [4].

Data augmentation and image resizing. During the _pre-training_ stage, we
resize each radiograph in the source domain to 256$\times$256 pixels, and then
apply random cropping to produce 224$\times$224 images. Random horizontal
flip, random rotation (-10 to 10 degrees) and random grayscale (brightness and
contrast) are also applied to generate augmented images. When using random
horizontal flip, we change the words ‘left’ and ‘right’ in the accompanying
radiology report accordingly. During the _fine-tuning_ stage, we apply the
same set of data augmentation strategies, which are random cropping, random
rotation, random grayscale and random horizontal flip, to all four target
domain datasets. As in the pre-training stage, we resize each radiograph in a
target domain to 256$\times$256, and then generate 224$\times$224 cropped and
augmented radiographs as input images.

Algorithm Overview. REFERS performs cross-supervised learning on top of a
transformer based backbone, called radiograph transformer. Given a patient
study, we first forward its views to the radiograph transformer for extracting
view-dependent feature representations. Next, we perform cross-supervised
learning that acquires study-level supervision signals from free-text
radiology reports. To this aim, it is necessary and essential to use view
fusion to obtain a unified visual representation for an entire patient study
because each radiology report is associated with a patient study but not
individual radiographs within the patient study. Such fused representations
are then used in two tasks during the pre-training stage: report generation
and study-report representation consistency reinforcement. The first task
takes the free texts in original radiology reports to supervise the training
process of the radiograph transformer. The second task reinforces the
consistency between the visual representations of patient studies and the
textual representations of their corresponding reports.

###  4.1 Radiograph Transformer

The radiograph transformer accepts image patches as inputs. We divide each
image into a grid of 14$\times$14 cells, each of which has 16$\times$16
pixels. We then flatten each image patch to form a 1D vector of pixels, and
feed it to the transformer. At the beginning of the transformer, a patch
embedding layer linearly transforms each 1D pixel vector into a feature
vector. This vector is concatenated with a position feature produced from a
learnable position embedding to help clarify the relative location of each
patch in the whole input patch sequence. The concatenated feature is then
passed through another linear transformation layer to make its dimensionality
the same as that of the final radiograph feature. At the core part of the
radiograph transformer, we stack twelve self-attention blocks, which have the
same architecture but independent parameters (Figure 1b). We first follow the
practice in [20] to build a single self-attention block and then repeat its
operations multiple times. In each block, we apply layer normalization [30]
before the multi-head attention and perceptron layers, after which residual
connections are added to stabilize the training process. In the perceptron
layer, we employ a two-layer perceptron with the Rectified Linear Unit (ReLU)
[31] as the activation function. Moreover, we add an aggregation embedding,
which is responsible for gathering the information from different input
features. As shown in Figure 1b, in the last layer, recurrent concatenation is
performed to repeatedly concatenates the learned aggregation embedding with
the learned representation of every patch. This is different from the
operation in vision transformer (ViT) [19], which only concatenates the
aggregation embedding with patch features once.

###  4.2 Cross-supervised Learning

There are two major components in cross-supervised learning: the view fusion
module for producing study-level representations and two report-related tasks
exploiting study-level information from associated free-text reports.

As aforementioned, we forward all radiographs in a patient study through the
radiograph transformer simultaneously to obtain their individual
representations. We further employ an attention mechanism to fuse these
individual representations to obtain an overall representation of the given
study. Supposing a study has three radiographs (i.e., views), as shown in
Figure 1c. We first concatenate the features of all views, and then feed the
concatenated features to a multi-layer perceptron to compute an attention
value for each view. Next, we apply the softmax function to normalize these
attention values, which are used as weights to produce a weighted version of
the individual representations. Finally, these weighted representations are
concatenated to form a unified visual feature for describing the whole study.
Note that for studies that contain few than three radiographs, we randomly
select one of the radiographs, and then repeat it once or twice to have a
total of three views. For studies that contain more than three radiographs, we
randomly select three of them from each study as input views.

We design two report-related tasks that acquire cross-supervision signals from
free-text reports: report generation and study-report representation
consistency reinforcement. In practice, these two tasks exploit study-level
free-text information for better training study-level visual representations
produced from the view fusion module. The first task applies a decoder, called
report transformer, to the unified visual feature $\mathbf{v}^{k}$ of the
$k$-th patient study to reproduce its associated radiology report denoted as
$c_{1:T}^{k}$. Here, $c_{1}^{k}$ represents the start-of-sequence token and
$c_{T}^{k}$ the end-of-sequence token. As a result, the report transformer
generates a sequence of token-level predictions, $\hat{c}_{1:T}^{k}$, for the
$k$-th patient study. The prediction of the $t$-th token in this sequence
depends on the predicted subsequence $\hat{c}^{k}_{1:t-1}$ and the visual
feature $\mathbf{v}^{k}$. The network architecture of the report transformer
follows the architecture of the decoder in [20]. We wish the predicted token
sequence ($\hat{c}_{1:T}^{k}$) resembles the sequence ($c_{1:T}^{k}$)
representing the original report of the $k$-th patient study. Therefore, as
shown in Figure 1d, we apply a language modeling loss to both
$\hat{c}_{1:T}^{k}$ and $c_{1:T}^{k}$ to maximize the following log-likelihood
of the tokens in the original report.

$\mathcal{L}^{k}_{\text{language}}=\sum_{t=2}^{T}\log
P\left(c_{t}^{k}\mid\hat{c}_{1:t-1}^{k},\mathbf{v}^{k};\phi_{v},\phi_{t}\right),$
(1)

where $\hat{c}_{1}^{k}$ is a special symbol indicating the start of the
predicted sequence, $\phi_{v}$ and $\phi_{t}$ stand for the parameters of the
radiograph transformer and report transformer, respectively.

For the second task on study-report representation consistency reinforcement,
we employ a contrastive loss [32] to align cross-modal representations. Here,
we use $\mathbf{t}^{k}$ to stand for the textual feature vector of the k-$th$
radiology report. In practice, we obtain $\mathbf{t}_{k}$ by forwarding the
sequence of tokens in the k-$th$ report (i.e., $c^{k}_{1:T}$) to a BERT (i.e.,
Bidirectional Encoder Representations from Transformer) model [33]. BERT is
built on top of the encoder in [20] using large-scale pre-training on a great
number of corpus resources. Thus, BERT can help produce a generalized textual
representation for the input report. Suppose we have $B$ patient studies in
each training mini-batch, as shown in Figure 1d. The contrastive loss for the
k-$th$ study can be formulated as

$\mathcal{L}_{\text{contrast}}^{k}=-\log\frac{e^{\cos(\mathbf{v}^{k},\mathbf{t}^{k})/\tau}}{\sum_{i=1}^{B}e^{\cos(\mathbf{v}^{k},\mathbf{t}^{i})/\tau}},$
(2)

where $\cos(\cdot,\cdot)$ means the cosine similarity,
$\cos(\mathbf{v}^{k},\mathbf{t}^{k})=\frac{(\mathbf{v}^{k})^{\top}\mathbf{t}^{k}}{\|\mathbf{v}^{k}\|\|\mathbf{t}^{k}\|}$,
$\top$ denotes the transpose operation, $\|\cdot\|$ stands for L2
normalization, and $\tau$ is the temperature factor. Finally, for each patient
study, we simply sum up $\mathcal{L}_{\text{contrast}}^{k}$ and
$\mathcal{L}^{k}_{\text{language}}$ as the overall loss. During the _fine-
tuning_ stage, we typically use the cross entropy loss for model tuning.

Training and testing methodologies. We first pre-train the radiograph
transformer on the source domain and then fine-tune it on downstream target
domain datasets to verify the quality of pre-training. During the _pre-
training_ stage, we sample 4.6k studies to form a held-out validation set
according to the official division of the MIMIC-CXR-JPG dataset [23]. We train
the entire network using stochastic gradient descent (SGD) while setting the
momentum value to 0.9 [34] and the weight decay to 1e-4. Following [33], we do
not apply weight decay to layer normalization and the bias terms in all
layers. We use a fixed batch size of 32 for 300k iterations (about 45 epochs).
We calculate the validation loss after each epoch and save the checkpoint that
achieves the lowest validation loss. We adopt the linear learning rate warm-up
strategy [35] for the first 10k iterations, and then switch to cosine decay
[36] until the end. Empirically, we found that training the radiograph
transformer requires a large learning rate for fast convergence. Thus, its
learning rate is set to 3e-3 while the learning rate for the report
transformer and BERT is set to 3e-4. We initialize the aggregation embedding
to all zeros while randomly initializing all position embeddings. We use
PyTorch [37] and NVIDIA Apex for mixed-precision training [38]. The complete
pre-training process on the MIMIC-CXR dataset takes about 2 days on a single
RTX 3090 GPU.

During the _fine-tuning_ stage, we fine-tune all transformer based models
(including transformer based baselines) using SGD with the momentum set to 0.9
and the initial learning rate set to 3e-3 for all datasets. We fine-tune
ResNet models using Adam [39] instead of SGD, and set the initial learning
rate to 1e-4. All downstream models use the same learning rate decay strategy
as that used in the pre-training stage, and are trained with a batch size of
128.

###  4.3 Ablation Study

We conduct a thorough ablation study of REFERS by removing or replacing
individual modules, and the results are shown in Table 3.

First, we investigate the impact of replacing the radiograph transformer (rows
1-2 in Table 3). If we replace the radiograph transformer with ResNet-101 [4]
(row 1), the overall performance of REFERS on COVID-19 Image Data Collection
would drop by about 7 percents (compared to row 0). This comparison
demonstrates that the radiograph transformer is more effective in dealing with
limited annotations, which is also verified with results in Tables 1 and 2.
Next, when we replace the radiograph transformer with the original ViT
architecture (row 2), which does not have the recurrent concatenation
operator, the overall performance would drop by 3.3 percents. This result
verifies the helpfulness of recurrently concatenating the learned aggregation
embedding with patch representations. We also note that there exists a
3.8-percent performance difference between ResNet and ViT based architectures
(rows 1&2), showing the advantage of a transformer-like architecture.

In addition to the radiograph transformer, we also investigate the impact of
cross-supervised learning. First of all, we remove the view fusion module so
that different radiographs within a patient study become associated with the
same study-level radiology report (row 3). Such an operation is counter-
intuitive as each individual radiograph alone cannot provide enough
information to produce a study-level report. By comparing row 3 with row 0, we
found that dropping the view fusion module would reduce the performance by
nearly 2 percents on COVID-19 Image Data Collection. This result implies that
learning study-level pre-trained representation is better than image-level
pre-training as the former includes more patient-level information. Next, we
completely replace cross-supervised learning with label-supervised learning
(row 4), and REFERS deteriorates into LSP (Transformer) in Table 2. We found
that dropping the two report-related tasks would adversely affect the
performance by 2 percents. Last but not the least, we study the two report-
related learning tasks individually. By comparing row 0 with row 5 and row 6,
respectively, we observed that dropping either of them would not affect the
overall performance too much (about 1 percent). This result implies that the
effects of both tasks may partially overlap to some extent. Nonetheless,
either of them along with the view fusion module can still outperform LSP
(Transformer) (row 4). In addition, we found that although both of them
improve the overall performance, reinforcing the consistency between
representations of each patient study and its associated report (i.e., the
second task) is more crucial than report generation (i.e., the first task). We
believe the reason behind is that the representation learned in the second
task can be regarded as a summary of each report, and thus provides more
global information than token-level predictions in the first task. Such
advantages make it more beneficial for the second task to include more study-
level information for learning better study-level radiograph features.

## Code Availability

All codes are available at https://github.com/funnyzhou/REFERS [40].

## Data Availability

MIMIC-CXR-JPG: https://physionet.org/content/mimic-cxr-jpg/2.0.0/.

NIH ChestX-ray: https://nihcc.app.box.com/v/ChestXray-
NIHCC/folder/36938765345.

VinBigData Chest X-ray Abnormalities Detection:
https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection.

Shenzhen Tuberculosis: https://www.kaggle.com/raddar/tuberculosis-chest-xrays-
shenzhen.

COVID-19 Image Data Collection: https://github.com/ieee8023/covid-chestxray-
dataset.

## Acknowledgements

This work was supported in part by the Fundamental Research Funds for the
Central Universities (Grant No. 20720190012, 20720210121).

## Author Contributions Statement

H.Z. and Y.Y. conceived the idea and designed the experiments. H.Z., X.C. and
Y.Z. implemented and performed the experiments. H.Z. and Y.Y. wrote the
manuscript. All authors analyzed the data and experimental results, commented
on the manuscript.

## Competing Interests Statement

The authors declare no competing interests.

## Tables

| NIH | NIH | NIH | VBD | VBD | VBD | SZ | C-T1 | C-T2
---|---|---|---|---|---|---|---|---|---
Method | 0.8k (1%) | 8k (10%) | 80k (100%) | 0.1k (1%) | 1k (10%) | 10k (100%) | All | All | All
Our REFERS | 76.7 | 80.9 | 84.7 | 83.0 | 88.2 | 90.1 | 98.0 | 82.1 | 80.4
Model Genesis | 70.3 | 75.7 | 81.0 | 70.7 | 82.7 | 85.8 | 94.9 | 76.0 | 71.8
C2L | 71.0 | 76.6 | 82.2 | 75.3 | 83.3 | 85.9 | 95.5 | 77.8 | 73.0
Context Restoration | 67.8 | 73.9 | 78.7 | 67.9 | 82.4 | 83.8 | 92.7 | 74.6 | 69.8
TransVW | 71.2 | 74.3 | 81.7 | 73.6 | 83.8 | 86.2 | 94.2 | 76.1 | 71.5
ImageNet Pre-training | 69.8 | 74.4 | 80.0 | 69.7 | 82.9 | 84.5 | 94.5 | 74.1 | 70.3
p-value | 8.35e-4 | 8.72e-4 | 1.94e-3 | 8.72e-5 | 4.34e-4 | 9.33e-4 | 1.73e-3 | 5.88e-4 | 3.59e-4

Table 1: Comparison with self-supervised learning and transfer learning
baselines. NIH, VBD and SZ stand for NIH ChestX-ray, VinBigData Chest X-ray
Abnormalities Detection and Shenzhen Tuberculosis datasets, respectively. C-T1
and C-T2 denote the two tasks in COVID-19 Image Data Collection, where one
task is to distinguish COVID-19 from the rest (C-T1) and the other task is to
separate viral pneumonia cases from bacterial ones (C-T2). Note that for the
sake of fairness, all baselines use the same transformer-based backbone as the
radiograph transformer of REFERS (i.e., a ViT-like architecture plus the
recurrent concatenation operator). Each p-value is calculated between our
REFERS and the best performing baseline. The evaluation metric is Area under
the ROC Curve (AUC). Best results are bolded.

| NIH | NIH | NIH | VBD | VBD | VBD | SZ | C-T1 | C-T2
---|---|---|---|---|---|---|---|---|---
Method | 0.8k (1%) | 8k (10%) | 80k (100%) | 0.1k (1%) | 1k (10%) | 10k (100%) | All | All | All
Our REFERS | 76.7 | 80.9 | 84.7 | 83.0 | 88.2 | 90.1 | 98.0 | 82.1 | 80.4
LSP (Transformer) | 74.2 | 78.2 | 82.1 | 78.5 | 85.8 | 87.6 | 96.4 | 80.2 | 76.6
LSP (ConvNet) | 65.8 | 74.5 | 81.9 | 76.0 | 85.2 | 87.2 | 96.7 | 80.1 | 76.2
p-value | 3.25e-3 | 2.89e-3 | 5.23e-3 | 3.56e-4 | 8.69e-4 | 1.05e-3 | 9.65e-3 | 7.61e-3 | 1.47e-3

Table 2: Comparison with methods using human-assisted structured labels. NIH,
VBD and SZ stand for NIH ChestX-ray, VinBigData Chest X-ray Abnormalities
Detection and Shenzhen Tuberculosis datasets, respectively. C-T1 and C-T2
denote the two tasks in COVID-19 Image Data Collection, where one task is to
distinguish COVID-19 from the rest (C-T1) and the other task is to separate
viral pneumonia cases from bacterial ones (C-T2). Note that for fairness, both
LSP (Transformer) and REFERS share the same transformer-based backbone (i.e.,
the ViT architecture plus the recurrent concatenation operator). Each p-value
is calculated between the results from our REFERS and LSP (Transformer). The
evaluation metric is Area under the ROC Curve (AUC). Best results are bolded.

Row | ViT | RecConcate | View Fusion | Task1 | Task2 | Viral vs. Bacterial
---|---|---|---|---|---|---
0 | ✓ | ✓ | ✓ | ✓ | ✓ | 80.4
1 |  |  | ✓ | ✓ | ✓ | 73.3
2 | ✓ |  | ✓ | ✓ | ✓ | 77.1
3 | ✓ | ✓ |  | ✓ | ✓ | 78.6
4 | ✓ | ✓ |  |  |  | 76.6
5 | ✓ | ✓ | ✓ | ✓ |  | 79.1
6 | ✓ | ✓ | ✓ |  | ✓ | 79.3

Table 3: An ablation study of REFERS by removing or replacing individual
modules. RecConcate stands for the recurrent concatenation operation in the
radiograph transformer. Task1 and Task2 refer to the two tasks in cross-
supervised learning. Row 1 corresponds to the result of a convolutional neural
network while row 4 corresponds to LSP (Transformer).

## Figures

Figure 1: Workflow of REFERS: forwarding radiographs of the $k$-th patient
study through the radiograph transformer, fusing representations of different
views using an attention mechanism, and utilizing report generation and study-
report representation consistency reinforcement to exploit the information in
radiology reports. Part a provides an overview of the whole pipeline. Part b
shows the architecture of the radiograph transformer. Attention for view
fusion is elaborated in Part c. Part d presents two supervision tasks, report
generation and study-report representation consistency reinforcement. In Part
d, $\mathbf{v}^{k}$ and $\mathbf{t}^{k}$ denote the visual and textual
features of the $k$-th patient study, respectively. $\hat{c}_{1:T}^{k}$ and
$c_{1:T}^{k}$ stand for the token-level prediction and ground truth of the
$k$-th radiology report whose length is $T$. Figure 2: Performance obtained
with different amounts of annotated training data in the target domain (a. NIH
ChestX-ray and b. VinBigData Chest X-ray Abnormalities Detection). We also
denote the percentage of annotated training data in the target domain that our
REFERS requires to achieve comparable results with those of Model Genesis and
ImageNet pre-training. Note that all three methods share the same transformer-
based backbone. Figure 3: Visualization of twelve randomly chosen samples from
NIH ChestX-ray (a-f) and VinBigData (i-vi) (fine-tuned with all annotated
training data). For each sample, we present both the original image (left) and
an attention map generated from REFERS. In each original image, red boxes
denote lesion areas annotated by radiologists. In attention maps, fuchsia
color stands for attention values generated from REFERS. The darker the
fuchsia color, the higher the confidence of a specific disease. Green boxes in
original images are our predicted lesion areas generated by applying a fixed
confidence threshold to attention maps. Figure 4: Comparison with self-
supervised learning and transfer learning baselines on NIH ChestX-ray dataset.
Note that for the sake of fairness, all baselines use the same transformer-
based backbone as the radiograph transformer of REFERS (i.e., a ViT-like
architecture plus the recurrent concatenation operator). Each p-value is
calculated between our REFERS and the best performing baseline. The evaluation
metric is Area under the ROC Curve (AUC). Best results are bolded. Figure 5:
Comparison with label-supervised pre-training (LSP) on NIH ChestX-ray dataset.
For fairness, both LSP (Transformer) and REFERS share the same transformer-
based backbone (i.e., the ViT architecture plus the recurrent concatenation
operator). Each p-value is calculated between the results from our REFERS and
LSP (Transformer). The evaluation metric is Area under the ROC Curve (AUC).
Best results are bolded. Figure 6: Comparison with self-supervised learning
and transfer learning baselines on VinBigData Chest X-ray Abnormalities
Detection. Note that for the sake of fairness, all baselines use the same
transformer-based backbone as the radiograph transformer of REFERS (i.e., a
ViT-like architecture plus the recurrent concatenation operator). Each p-value
is calculated between our REFERS and the best performing baseline. The
evaluation metric is Area under the ROC Curve (AUC). Best results are bolded.
Figure 7: Comparison with label-supervised pre-training (LSP) on VinBigData
Chest X-ray Abnormalities Detection. For fairness, both LSP (Transformer) and
REFERS share the same transformer-based backbone (i.e., the ViT architecture
plus the recurrent concatenation operator). Each p-value is calculated between
the results from our REFERS and LSP (Transformer). The evaluation metric is
Area under the ROC Curve (AUC). Best results are bolded.

## References

  * [1] Krizhevsky, A., Sutskever, I. & Hinton, G.E. ImageNet classification with deep convolutional neural networks. In _Proc. Advances in Neural Information Processing Systems_ , 1097-1105 (2012).
  * [2] Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. In _International Conference on Learning Representations_ (2014).
  * [3] Szegedy, C. et al. Going deeper with convolutions. In _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , 1-9 (IEEE, 2015).
  * [4] He, K.M., Zhang, X.Y., Ren, S.Q. & Sun, J. Deep residual learning for image recognition. In _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , 770-778 (IEEE, 2016).
  * [5] Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K.Q. Densely connected convolutional networks. In _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , 4700-4708 (IEEE, 2017).
  * [6] Bengio, Y., Courville, A. & Vincent, P. Representation learning: A review and new perspectives. _IEEE Trans. Pattern Anal. Mach. Intell._ 35, 1798-1828 (IEEE, 2013).
  * [7] Phillips, N.A. et al. CheXphoto: 10,000+ photos and transformations of chest x-rays for benchmarking deep learning robustness. In _Proc. Machine Learning for Health_ , 318-327 (PMLR, 2020).
  * [8] Taylor, A.G., Mielke, C. & Mongan, J. Automated detection of moderate and large pneumothorax on frontal chest x-rays using deep convolutional neural networks: a retrospective study. _PLoS medicine_ 15, e1002697 (Public Library of Science San Francisco, 2018).
  * [9] Carlile, M. et al. Deployment of artificial intelligence for radiographic diagnosis of COVID-19 pneumonia in the emergency department. _Jour. of the Amer. Coll. of Emer. Phys. Open_ 1, 1459-1464 (Wiley Online Library, 2018).
  * [10] Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? In _Advances in Neural Information Processing Systems_ , 3320-3328 (2014).
  * [11] Wang, X.S. et al. ChestX-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , 2097-2106 (IEEE, 2017).
  * [12] Deng, J. et al. ImageNet: A large-scale hierarchical image database. In _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , 248-255 (IEEE, 2009).
  * [13] Chen, L. et al. Self-supervised learning for medical image analysis using image context restoration. _Med. Image Anal._ 58, 101539 (Elsevier, 2019).
  * [14] Zhou, Z.W., Sodha, V., Pang, J.X., Gotway, M.B. & Liang, J.M. Model genesis. _Med. Image Anal._ 67, 101840 (Elsevier, 2021).
  * [15] Haghighi, F., Taher, M.R.H., Zhou, Z.W., Gotway, M.B. & Liang, J.M. Transferable visual words: Exploiting the semantics of anatomical patterns for self-supervised learning. _IEEE Trans. Med. Imag._ , early access (IEEE, 2021).
  * [16] Zhou, H.-Y. et al. Comparing to learn: Surpassing ImageNet pretraining on radiographs by comparing image representations. In _Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 398-407 (Springer, 2020).
  * [17] Johnson, A.E.W. et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. _Sci. Data_ 6, 1-8 (NPG, 2019).
  * [18] Irvin, J. et al. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In _Proc. the AAAI Conference on Artificial Intelligence_ , 590-597 (AAAI, 2019).
  * [19] Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. In _International Conference on Learning Representations_ (2021).
  * [20] Vaswani, A. et al. Attention is all you need. In _Proc. Advances in Neural Information Processing Systems_ , 5998-6008 (2017).
  * [21] Shin, H.-C. et al. Interleaved text/image deep mining on a very large-scale radiology database. In _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , 1090-1099 (IEEE, 2015).
  * [22] Wang, X.S., Peng, Y.F., Lu, L., Lu, Z.Y & Summers, R.M. Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , 9049-9058 (IEEE, 2018).
  * [23] Johnson, A.E.W. et al. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. Preprint at https://arxiv.org/abs/1901.07042 (2019).
  * [24] Nguyen, H.Q. et al. VinDr-CXR: An open dataset of chest x-rays with radiologist’s annotations. Preprint at https://arxiv.org/abs/2012.15029 (2021).
  * [25] Jaeger, S. et al. Two public chest x-ray datasets for computer-aided screening of pulmonary diseases. _Quantitative Imaging in Medicine and Surgery_ 4, 475 (AME Publications, 2014).
  * [26] Joseph, P.C. et al. COVID-19 image data collection: prospective predictions are the future. _Journal of Machine Learning for Biomedical Imaging_ , early access (2020).
  * [27] Zhou, B.L., Khosla, A., Lapedriza, A., Oliva, A. & Torralba, A. Learning deep features for discriminative localization. In _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , 2921-2929 (IEEE, 2016).
  * [28] Chetlur, S. et al. cuDNN: Efficient primitives for deep learning. Preprint at https://arxiv.org/abs/1410.0759 (2014).
  * [29] He, K.M., Fan, H.Q., Wu, Y.X., Xie, S.N., & Girshick, R. Momentum contrast for unsupervised visual representation learning. In _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , 9729–9738 (IEEE, 2020).
  * [30] Ba, J.L., Kiros, J.R. & Hinton, G.E. Layer normalization. In _International Conference on Learning Representations_ (2016).
  * [31] Dahl, G.E., Sainath, T.N. & Hinton, G.E. Improving deep neural networks for LVCSR using rectified linear units and dropout. In _Proc. International Conference on Acoustics, Speech and Signal Processing_ , 8609-8613 (IEEE, 2013).
  * [32] Gutmann, M. & Hyvärinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In _Proc. the Thirteenth International Conference on Artificial Intelligence and Statistics_ , 297-304 (JMLR, 2010).
  * [33] Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proc. the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 4171-4186 (ACL, 2019).
  * [34] Sutskever, I., Martens, J., Dahl, G. & Hinton, G.E. On the importance of initialization and momentum in deep learning. In _Proc. International Conference on Machine Learning_ , 1139-1147 (PMLR, 2013).
  * [35] Goyal, P., Mahajan, D., Gupta, A. & Misra, I. Scaling and benchmarking self-supervised visual representation learning. In _Proc. International Conference on Computer Vision_ , 6391-6400 (IEEE, 2019).
  * [36] Loshchilov, I. & Hutter, F. SGDR: Stochastic gradient descent with warm restarts. In _International Conference on Learning Representations_ (2017).
  * [37] Paszke, A. et al. PyTorch: An imperative style, high-performance deep learning library. In _Proc. Advances in Neural Information Processing Systems_ , 8024-8035 (2019).
  * [38] Micikevicius, P. et al. Mixed precision training. In _International Conference on Learning Representations_ (2018).
  * [39] Kingma, D.P. & Ba, J.L. Adam: A method for stochastic optimization. In _International Conference on Learning Representations_ (2014).
  * [40] Zhou, H.Y., Chen, X.Y., Zhang, Y.H., Luo, R.B., Wang, L.S., & Yu, Y. Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reports. _Zenodo_ https://doi.org/10.5281/zenodo.5624117 (2021).