11institutetext: Institute for Information Transmission Problems, B. Karetnuj
per., 19, Moscow, 127051, Russia 22institutetext: University of Hradec
Králové, Faculty of Science, Rokitanského 62, 500 03 Hradec Králové, Czech
Republic
22email<EMAIL_ADDRESS>
33institutetext: St. Petersburg Department of Steklov Mathematical Institute
Fontanka, 27, 191023, St. Petersburg, Russia
33email<EMAIL_ADDRESS>
# Geometry of saccades and saccadic cycles ††thanks: D. V. Alekseevsky was
supported by the Grant "Basis-foundation (Leader)" 22-7-1-34-1. I.M. Shirokov
was supported by the Grant "Basis-foundation (Leader)" 22-7-1-34-1 and
Ministry of Science and Higher Education of the Russian Federation, agreement
no. 075-15-2022-289.

D. V. Alekseevsky 1122 0000-0002-6622-7975 I.M. Shirokov 33
0009-0005-1538-2712

###### Abstract

The paper is devoted to the development of the differential geometry of
saccades and saccadic cycles. We recall an interpretation of Donder’s and
Listing’s law in terms of the Hopf fibration of the $3$-sphere over the
$2$-sphere. In particular, the configuration space of the eye ball (when the
head is fixed) is the 2-dimensional hemisphere $S^{+}_{L}$, which is called
Listing’s hemisphere. We give three characterizations of saccades: as geodesic
segment $ab$ in the Listing’s hemisphere, as the gaze curve and as a piecewise
geodesic curve of the orthogonal group. We study the geometry of saccadic
cycle, which is represented by a geodesic polygon in the Listing hemisphere,
and give necessary and sufficient conditions, when a system of lines through
the center of eye ball is the system of axes of rotation for saccades of the
saccadic cycle, described in terms of world coordinates and retinotopic
coordinates. This gives an approach to the study the visual stability problem.

###### Keywords:

Donder’s and Listing’s law quaternions Hopf bundle fixation eyes movements
drift micrsasccades remapping shift of receptive fields neurogeometry

##  1 Introduction

Our eyes continually move. Even while we fix our gaze on an object, they
participate in fixational eye movements – tremor, drift and microsaccades (see
Fig. 1.). The classical experiments by A. Yarbus show that compensation of eye
movements leads to loss of vision in 2-3 sec. The information about saccades
is coded into control commands of the oculomotor system which governs muscles
contraction. A copy of these command (efference copy or corollary discharge)
is send to the frontal eye field of the frontal cortex [W], where it meets the
statistical information about retina images, described in coordinates
associated to the end gaze direction of the previous saccade.

Figure 1: Fixational eye movements (tremor is not shown). From Wikipedia, The
Free Encyclopedia, 2022. Available at: Visual fixation

The vision is the result of interaction of these two types of information. The
time synchronization of oculomotor information about eye rotations and retinal
information is very important for the correct decoding.

In [A] a new interpretation of the Donder’s and Listing’s laws in terms of the
Hopf bundle had been given and used for geometrical descriptions of saccades
and drift. This paper, which is a continuation of [A], is devoted to
development of the differential geometry of saccades and saccadic cycles. We
give three characterizations of saccades – as geodesic segment $ab$ in the
Listing’s hemisphere (which is the configuration space of the eye), as the
gaze curve $Sac(AB)=\chi(a)\chi(b)\subset
S^{2}_{E}\cap\mathrm{s}pan\;(A,B,-i)$, where $\chi:S^{3}\to S^{2}$ is the Hopf
projection, and as a piecewise geodesic curve of the orthogonal group. Then we
study the geometry of saccadic cycle which is represented by a geodesic
polygon in the Listing hemisphere. We give necessary and sufficient condition,
when a system of lines $([\Omega_{1}],\cdots,[\Omega_{n}])$ , where
$\Omega_{\ell}\in S^{2}_{E}$, is the system of axes for saccades of the
saccadic cycle, described in terms of world coordinates and retinotopic
coordinates.

This clarifies the relation between oculomotor information about saccadic
rotation the gaze direction and hence the eye position and shows that all
visual information may be transformed into world-centered coordinates, which
gives an approach to the study of the visual stability problem.

##  2 The Hopf bundle and the Listing’s section

Let

${\mathbb{H}}={\mathbb{R}}^{4}=\mathrm{s}pan\;(1,i,j,k)={\mathbb{R}}1+E^{3},\,E^{3}=\mathrm{Im}{\mathbb{H}}=\mathrm{s}pan\;(i,j,k)$

be the algebra of quaternions with the standard Euclidean metric
$|q|^{2}=<q,q>=q\bar{q}$, where $\bar{q}=q_{R}-q_{E}$ is the conjugation.

The sphere ${\mathbb{H}}_{1}=S^{3}$ of unit quaternions is the Lie group and
the adjoint action

$\mathrm{Ad}:{\mathbb{H}}_{1}\to
SO(E^{3}),\,a\mapsto\mathrm{Ad}_{a},\,\mathrm{Ad}_{a}x=axa^{-1}=ax\bar{a}$

is the universal ${\mathbb{Z}}_{2}$-covering.

The orbit
$\mathrm{Ad}_{{\mathbb{H}}_{1}}i=S^{2}_{E}\simeq{\mathbb{H}}_{1}/SO(2)$ is the
unit sphere and $\chi:S^{3}\to S^{2}_{E}$ is the principal bundle.

###### Definition 1

The Hopf bundle is the natural ${\mathbb{H}}_{1}$-equivariant map

$\chi:S^{3}={\mathbb{H}}_{1}\to S^{2}_{E},\,a\mapsto
A=\chi(a):=\mathrm{Ad}_{a}i=aia^{-1}.$

Denote by $S^{2}_{L}=S^{3}\cap\mathrm{s}pan\;(1,j,k)=S^{3}\cap i^{\perp}$ the
equatorial 2 -sphere of $S^{3}$ w.r.t. the poles $\pm i$ and by
$S^{+}_{L}=\\{a=a_{0}1+a_{2}j+a_{3}k\in S^{2}_{L},\,a_{0}>0\\}$ the north
hemisphere. It is called the Listing hemisphere [A] and its boundary
$S^{1}_{L}$ is called the Listing’s circle (see Fig. 2.).

###### Proposition 1

The (restricted to $S^{+}_{L}$) Hopf map

$\chi:S^{+}_{L}\to S^{2}_{E},\,a\mapsto A=\chi(a)=\mathrm{Ad}_{a}i=aia^{-1}$

is a diffeomorphism of $S^{+}_{L}$ onto the punctured sphere
$\tilde{S}^{2}_{E}=S^{2}_{E}\setminus\\{-i\\}$ and $\chi(S^{1}_{L})=\\{-i\\}.$

###### Definition 2

i) The map $\sigma=\chi^{-1}:\tilde{S}^{2}_{E}\to S^{+}_{L}$ is called the
Listing’s section.
ii) The frame $f^{1}=(i,j,k)$ is called the standard and the frame
$f^{a}:=\mathrm{Ad}_{a}f^{1}=(aia^{-1},aja^{-1},aka^{-1})$ is called
admissible.

###### Proposition 2

[A] i)Any two points $a,b\in S^{+}_{L}$ determine a unique (oriented) geodesic
of the Listing’s sphere with canonical parametrization

$\gamma_{a,b}=\gamma_{p,m}(t)=\cos t\,p+\sin t\,m=e^{tv}p$

where $p={\cos\theta}\,j+\sin{\theta}\,k$ is the intersection of
$\gamma_{a,b}$ with $S^{1}_{L}$, $m=e^{\psi q}=\cos\psi+\sin\psi q,\,q=ip$ is
the top point of the geodesic and $v=m\bar{p}=\sin\psi\,i-\cos\psi p\in
S^{2}_{E}$.
ii) If $m=1$, then $\gamma_{a,b}=\gamma_{p,1}=e^{tp},\,p\in S^{1}_{L}$ is
1-parameter subgroup.
iii) The group $\mathrm{Ad}_{e^{tp}}=R^{2t}_{p}$ is the group of rotation
w.r.t. the axis $[p]:={\mathbb{R}}p$ and the Hopf image
$\chi(\gamma_{p,m})=\mathrm{Ad}_{e^{tv}}(-i)=R^{2t}_{v}(-i)$ is the gaze
circle $S^{1}_{a,b}=S^{1}_{p,m}=S^{2}_{E}\cap\Pi(A,B,-i)$, where
$A=\chi(a),\,B=\chi(b)$ and $\Pi(A,B,-i)=-i+\mathrm{s}pan\;(i+A,i+B)$.

Figure 2: The Listing’s sphere. Figure 3: The eye sphere.

##  3 Donder’s and Lising’s laws

We consider the sphere $S^{2}_{E}\subset E^{3}$ in the Euclidean space $E^{3}$
as a model of the eye sphere (see Fig. 3.), i.e. the boundary of the eye ball
in the primary position, defined by the primary frame $f^{1}=(i,j,k)$. Here
$i$ indicates the primary position of the gaze (the frontal direction), $j$ is
the lateral direction from right to left and $k$ is the vertical direction up.

_Donder’s law_. If the head is fixed, the configuration space of the eye is
2-dimensional and the position $A\in S^{2}_{E}$ of the gaze determines the
position of the eye.
The Listing’s law describes the configuration space.

_Listing’s law_. The configuration space of the eye is the Listing’s
hemisphere. The eye position associated to a gaze $A\in\tilde{S}^{2}_{E}$ is
given by

$f(A)=f^{\sigma(A)}:=\mathrm{Ad}_{\sigma(A)}f^{1}.$

##  4 Saccades, three descriptions

The saccade $\mathrm{Sac\;}(A,B)$ from a gaze direction
$A\in\tilde{S}^{2}_{E}$ to the gaze direction $B$ is described by the geodesic
segment $ab=\sigma(A)\sigma(B)$ and the corresponding gaze curve
$AB=\chi(ab)$, which is the segment of the gaze circle $S^{1}_{a,b}$ with the
endpoints $A$ and $B$.

Let $a,b\in S^{+}_{L}$ be two eye positions and $A=\chi(a),B=\chi(b)\in
S^{2}_{E}$ the associated gaze directions. It defines a unique saccade
$\mathrm{Sac\;}(A,B)$ from the eye position $a$ to the position $b$.

We have three descriptions of the saccade $\mathrm{Sac\;}(A,B)$:
i) As the evolution of the eye position in the configuration space $S^{+}_{L}$
that is represented by a segment $ab$ of a geodesic $\gamma_{a,b}\subset
S^{+}_{L}$ of the Listing hemisphere.
ii) As the evolution of the gaze in the gaze sphere $\tilde{S}^{2}_{E}$,
described by the gaze curve $AB=\chi(ab)$. It is the segment of the gaze
circle $S^{1}(A,B)=S^{2}_{E}\cap\Pi(A,B,-i)$ which is the section of the
sphere $S^{2}_{E}$ by the plane $\Pi(A,B,-i)=-i+\mathrm{s}pan\;(A+i,B+i)$.
iii) As the curve $R^{t\varphi}_{\Omega},\,t\in[0,1]$ in the orthogonal group,
which is a segment of the one-parameter group of rotations around the axis
$[\Omega]={\mathbb{R}}\Omega$.

To relate these three characterisations of saccades, we introduce the
following definition.

###### Definition 3

Let $A\in\tilde{S}^{2}_{E}$ be a unit vector. Then the oriented 2-dimensional
subspace $L_{A}=N_{A}^{\perp}$ with the positive unit normal
$N_{A}=\frac{i+A}{|i+A|}$ is called $A$-Listing plane and $N_{A}$ is called
$A$-Listing’s normal.

###### Proposition 3

i) The axis $[\Omega]={\mathbb{R}}\Omega$ of rotation of any saccade with
initial gaze direction $A\in\tilde{S}^{2}_{E}$ is orthogonal to the
$A$-Listing’s normal $N_{A}$ and belongs to the $A$-Listing’s plane
$L_{A}=N_{A}^{\perp}$.
ii) Law of half angle: the ( unique) saccade $\mathrm{Sac\;}(A,B)$ from the
gaze direction $A$ to the gaze direction $B$ is obtained by the rotation
$R_{\Omega}^{\varphi}$ around the axis $[\Omega]={\mathbb{R}}(N_{A}\times
N_{B})=L_{A}\cap L_{B}$ by the angle $\varphi=2\psi$ where
$\psi=\angle(N_{A},N_{B})$ is the angle between the normal vectors
$N_{A},\,N_{B}$ of the A-Listing’s plane $L_{A}$ and the $B$-Listing’s plane
$L_{B}$.
iii) Let $Sac(A_{0},A_{1}),\,Sac(A_{1},A_{2})$ be two consecutive saccades
with distinct axes of rotation $\Omega_{1},\,\Omega_{2}$. Then the
$A_{1}$-Listing’s plane is given by

$L_{A_{1}}=\mathrm{s}pan\;(\Omega_{1},\Omega_{2})$

and the $A_{1}$-Listing’s normal is given by:

$N_{A_{1}}=\frac{i^{*}}{|i^{*}|},$

where $i^{*}$ stands for the first dual vector in the dual basis to the basis
$(i,\Omega_{1},\Omega_{2})$.

Proof. i) Let $B\in\tilde{S}^{2}_{E}\setminus\\{A\\}$. Then during the saccade
$Sac(A,B)$ the gaze belongs to the circle $\Pi(A,B,-i)\cap S^{2}_{E}$. Hence
the axis of rotation of $Sac(A,B)$ is orthogonal to the plane $\Pi(A,B,-i)$
and in particular orthogonal to the vector $N_{A}$.

ii) Indeed, denote by $O^{\prime}$ the center of the saccadic circle
$S^{1}_{A,B}=\Pi(A,B,-i)\cap S^{2}_{E}$. Then the rotation angle
$\varphi=\angle(O^{\prime}A,O^{\prime}B)$ is the central angle associated to
the inscribed angle

$\psi=\angle((-i)A,(-i)B)=\angle(N_{A},N_{B}).$

Hence $\psi=\frac{1}{2}\varphi.$

iii) First note that $\Omega_{1},\Omega_{2}$ and $i$ are linear independent.
We have

$i^{*}=\frac{[\Omega_{1},\Omega_{2}]}{(i,[\Omega_{1},\Omega_{2}])}$

Thus $\mathbb{R}i^{*}=\mathbb{R}N_{A_{1}}$ and $(i^{*},i)=1>0$. So
$N_{A_{1}}=\frac{i^{*}}{|i^{*}|}$.

$\square$

##  5 Saccadic $n$-cycles

We define a _saccadic $n$-cycle_ as a geodesic polygon $Sac(a_{1}a_{2}\cdots
a_{n+1})\subset S^{+}_{L},a_{1}=a_{n+1}$ and $a_{k}\neq a_{k+1}$ with the
natural parametrization, which is identified with the gaze curve

$\chi(Sac(a_{1}a_{2}\cdots a_{n+1})=Sac(A_{1},...,A_{n},A_{n+1}).$

In terms of the rotations, the saccadic curve is described as the orbit
$R^{t}A_{1}$ of a piecewise geodesic curve $R^{t},\,t\in[0,T]$ in the
orthogonal group $SO(3)$ such that

$R^{T_{j}}A_{1}=A_{j},\,R^{T_{j}+t}A_{1}=R^{t}_{\Omega_{j}}A_{j},\,0\leq t\leq
T_{j+1}-T_{j},\,$ $0<T_{1}<\cdots<T_{n+1}=T.$

We call the vectors $\Omega_{j}$ which determine the axes of rotation
$[\Omega_{j}]={\mathbb{R}}\Omega_{j}$ of the saccades $Sac(A_{j},A_{j+1})$ the
axes of the $n$-cycle.

###  5.1 Saccadic $n$-cycles via systems of axes expressed at the primary
frame

Note that the gaze directions $A_{j},A_{j+1}$ of the saccade
$S_{j}=Sac(A_{j},A_{j+1})$ determine the saccade rotation
$R^{\varphi}_{\Omega_{j}}$ as follows

$\Omega_{j}\equiv N_{A_{j}}\times
N_{A_{j+1}},\,\,\varphi_{j}=2\angle(N_{A_{j}},N_{A_{j+1}})$

(Where "$\equiv$" means equality up to sign). Now we consider the inverse
problem: when a system $(\Omega_{1},\cdots,\Omega_{n})$ of unit vectors
determines a saccadic $n$-cycle and how to describe the gaze directions
$(A_{1},\cdots,A_{n+1}=A_{1})$ and the angles of rotations of the $n$-cycle.

We show that it can be done under a mild necessary and sufficient condition
and give an explicit formula for the system of gaze directions
$(A_{1},\cdots,A_{n+1})$.

###### Theorem 1

Let $(\Omega_{1},\cdots,\Omega_{n})$ be a system of $n\geq 3$ unit vectors in
$E^{3}$. Then there exists a (unique) saccadic $n$-cycle

$Sac(A_{1},A_{2},\cdots,A_{n+1}),\,A_{n+1}=A_{1}$

with axes of rotation

$[\Omega_{1}],\cdots,[\Omega_{n}]$

if and only if

  1. 1.

All triples $(\Omega_{1},\Omega_{2},i),\cdots,(\Omega_{n},\Omega_{1},i)$ are
linear independent.

  2. 2.

All triples
$(\Omega_{1},\Omega_{2},\Omega_{3}),\cdots,(\Omega_{n-1},\Omega_{n},\Omega_{1}),(\Omega_{n},\Omega_{1},\Omega_{2})$
are linear independent.

If $span(\Omega_{n},\Omega_{1})=span(j,k)$ then $A_{1}=A_{n+1}=i$.

Proof. Necessity follows from Proposition 1 and definition of saccadic cycle.
The scheme of proof of sufficiency is as follows. Planes

$\Pi_{1}:=span(\Omega_{n},\Omega_{1}),\Pi_{2}:=span(\Omega_{1},\Omega_{2}),\cdots,\Pi_{n}:=span(\Omega_{n-1},\Omega_{n})$

define points $a_{1},\cdots,a_{n}$ of configuration space $S^{+}_{L}$ such
that $\Pi_{k}$ is $\chi(a_{k})$-Listing plane. It remains to check that
$Sac(a_{1},...,a_{n},a_{n+1}=a_{1})$ is a desired cycle.

$\square$

###  5.2 Transition functions between primary frame and the moving frame

In this subsection we show how coordinates of Listing’s normal $N_{A}$ w.r.t
the primary frame $f^{1}$ and "moving" frame $f^{\sigma(A)}$ are related.

###### Lemma 1

Let $a\in S^{+}_{L}$ and $A=\chi(a)$ be the gaze direction. Then coordinates
of $A$-Listing’s normal $N_{A}$ w.r.t. the primary frame $f^{1}$ and "moving"
frame $f^{a}$ related as follows:

$N_{A}=N_{1}i+N_{2}j+N_{3}k=N_{1}i^{a}-N_{2}j^{a}-N_{3}k^{a}$

Proof. We consider the saccade $Sac(1,a)=Sac(i,A)$. By the half-angle law
(Proposition 3) we have $N_{A}=a^{1/2}ia^{-1/2}$. We may write $a=e^{\varphi
p}$ for some $p=\cos\theta j+\sin\theta k$ and $\varphi\in[0,\pi/2)$, then
w.r.t. the primary frame $f^{1}$, the vector $N_{A}$ is obtained from $i$ by
rotation around $[p]$ on $\varphi/2$: $N_{A}=R^{\varphi/2}_{[p]}i$, whereas
w.r.t. the "moving" frame $f^{a}$, the vector the vector $N_{A}$ is obtained
from $i^{a}$ by rotation around $[p^{a}]=[\cos\theta j^{a}+\sin\theta k^{a}]$
on $-\varphi/2$: $N_{A}=R^{-\varphi/2}_{[p^{a}]}i^{a}$. $\square$

###  5.3 Saccadic $n$-cycles via systems of axes fixed at the moving frame

###### Definition 4

We associate with a vector $\Omega\in S^{2}_{E}$ the map

$\sigma_{\Omega}:S^{+}_{L}\to
S^{2}_{E},,a\mapsto\Omega^{a}=\mathrm{Ad}_{a}\Omega=a\Omega a^{-1}$

We call parameterized by $a\in S^{+}_{L}$ vector $\Omega^{a}$ the moving
vector.

The coordinates $\Omega^{a}_{f^{a}}$ of the vector $\Omega^{a}$ in the frame
$f^{a}$ does not depend on $a$ and coincide with the coordinates of vector
$\Omega$ w.r.t. the primary frame. Note that according to the Listing’s law
the eye can rotate around axis $[\Omega^{a}]$ if $\Omega^{a}$ belongs to the
$\chi(a)$-Listing’s plane. For example, $[\Omega^{1}]$ is the axis of rotation
only if it belongs to $\chi(1)=i$-Listing’s plane, which is
$\mathrm{s}pan\;(j,k)$.

Given a system $(\Omega_{1},\cdots,\Omega_{n})$ of vectors from $S^{2}_{E}$,
our aim is to describe when there exists a saccadic $n$-cycle
$Sac(a_{1},...,a_{n},a_{n+1})$ with axes of rotation

$[\Omega_{1}^{a_{1}}],\cdots,[\Omega_{n}^{a_{n}}].$

We need the following

###### Proposition 4

Assume the vectors $\Omega_{1},\Omega_{2}\in S^{2}_{E}$ span the plane
$\Pi=\mathrm{s}pan\;(\Omega_{1},\Omega_{2})$ which does not contains $i$. Then
there exists unique $a\in S^{+}_{L}$, hence gaze direction $A=\chi(a)$, such
that
$\Pi^{a}:=\mathrm{Ad}_{a}\Pi=\mathrm{s}pan\;(\Omega_{1}^{a},\Omega_{2}^{a})$
coincides with $A$-Listing plane $L_{A}$.

Proof. Proposition 3 implies that there is unique $b\in S^{+}_{L}$ such that
$\Pi=\mathrm{Ad}_{b^{1/2}}L_{i}=L_{b}.$ We have to find $a\in S^{+}_{L}$ such
that

$\Pi^{a}=\mathrm{Ad}_{ab^{1/2}}L_{i}=L_{a}=\mathrm{Ad}_{a^{1/2}}L_{i}$

The solution is $a=b^{-1}$. $\square$

###### Theorem 2

The system $(\Omega_{1},\cdots,\Omega_{n})$ of vectors from $S^{2}_{E}$
defines the system of axes of rotation

$([\Omega_{1}^{a_{1}}],\cdots,[\Omega_{n}^{a_{n}}])$

for a (unique) saccadic $n$-cycle

$Sac(a_{1},\cdots,a_{n+1}),\,a_{1}=a_{n+1}$

if and only if it satisfies the conditions of Theorem 1. If
$span(\Omega_{n},\Omega_{1})=span(j,k)$ then $a_{1}=a_{n+1}=1$.

Proof.

We prove only sufficiency. Let $\Omega_{n+1}^{a}:=\Omega_{1}^{a}$. Proposition
4 implies that for each pair $\Omega_{k},\Omega_{k+1},k\in\\{1,...,n\\}$ there
exists $a_{k}\in S^{+}_{L}$, such that

$\Pi_{k}^{a_{k}}:=span(\Omega_{k}^{a_{k}},\Omega_{k+1}^{a_{k}})=L_{a_{k}}$
$None$

Let $a_{n+1}:=a_{1}$. Proof of Proposition 4 implies that
$\Pi_{k}^{a_{k}}=S_{L_{i}}(\Pi_{k}^{1})$, where

$S_{L_{i}}:\xi_{1}i+\xi_{2}j+\xi_{3}k\mapsto-\xi_{1}i+\xi_{2}j+\xi_{3}k$

is the symmetry w.r.t.the Listing’s plane $L_{i}$. Hence we have

$\Omega_{1}^{a_{n}}=\Omega_{1}^{a_{1}},\>\>\Omega_{k}^{a_{k-1}}=\Omega_{k}^{a_{k}},k\in\\{2,...,n\\}$
$None$

We claim that $Sac(a_{1},...,a_{n},a_{n+1})$ is a desired cycle. Indeed
$a_{k}\neq a_{k+1}$ for $k\in\\{1,...,n\\}$ due to the assumptions on
$\Omega_{1},...,\Omega_{n}$, the geodesic $a_{k}a_{k+1}$ corresponds to
rotation around $[\Omega_{k}^{a_{k}}]$ due to (2) for each
$k\in\\{1,...,n\\}$; and $[\Omega_{k}^{a_{k}}]\subset L_{a_{k}}$ for each
$k\in\\{1,...,n\\}$ due to (1). $\square$

###### Corollary 1

Let $Sac(a_{1},...,a_{n+1})$ be a saccadic cycle described in terms of Theorem
4 and $[\Omega_{m}],\,m=1,\cdots,n$ be axes of rotation. Then the coordinates
of any vector $\Omega:=\Omega^{a_{m}}_{m}$ w.r.t. the “moving“ frame
$f^{a_{m}}$ and the primary frame $f^{1}$ are:

$\Omega_{f^{a_{m}}}=(\Omega_{1},\Omega_{2},\Omega_{3})$
$\Omega_{f^{1}}=(-\Omega_{1},\Omega_{2},\Omega_{3})$

Corollary 1 means that during the saccadic cycle the oculomotor system can
easily recalculate coordinates of axis vector w.r.t the primary frame from
coordinates of it w.r.t the "moving" frame and vice versa at each switching
point. In other words, the way of setting a saccadic cycle in terms of fixed
axis vectors in Theorem 1 can be reformulated in terms of moving vectors in
Theorem 2 and vice versa.

## References

  * [A] D.V.Alekseevsky, Microsaccades, Drifts, Hopf Bundle and Neurogeometry, J. Imaging, 2022, 8, 76.
  * [W] Wikipedia contributors. Corollary discharge theory. Wikipedia, The Free Encyclopedia, 2022. Available at: Corollary discharge theory.
  * [Y] A. Yarbus, Eye Movements and vision (B. Haigh, Trans.). New York: Plenum Press, 1967.
  * [P] M. Poletti , M. Rucci, A compact field guide to the study of microsaccades: Challenges and functions., Vis. Res. 2016, 118, 83–97.32.
  * [M] R.H. Wurtz, W.M. Joiner, R.A. Berman. Neuronal mechanisms for visual stability: Progress and problems. Philos. Trans. R. Soc. B
  * [R] R. G. Alexander and S. Martinez-Conde. In book: Eye Movement Research, 2019, Chapter 3 Fixational Eye Movements, 74-108.
  * [K] K.Hepp, On Listing’s law, Commun. Math. Phys. 132, 285-292


# Bidirectional Captioning for Clinically Accurate and Interpretable Models

Keegan Quigley 1, Miriam Cha1, Josh Barua1,2, Geeticka Chauhan3,
Seth Berkowitz4,5, Steven Horng4,5, Polina Golland3

###### Abstract

> Vision-language pretraining has been shown to produce high-quality visual
> encoders which transfer efficiently to downstream computer vision tasks.
> While generative language models have gained widespread attention, image
> captioning has thus far been mostly overlooked as a form of cross-modal
> pretraining in favor of contrastive learning, especially in medical image
> analysis. In this paper, we experiment with bidirectional captioning of
> radiology reports as a form of pretraining and compare the quality and
> utility of learned embeddings with those from contrastive pretraining
> methods. We optimize a CNN encoder, transformer decoder architecture named
> RadTex for the radiology domain. Results show that not only does captioning
> pretraining yield visual encoders that are competitive with contrastive
> pretraining (CheXpert competition multi-label AUC of 89.4%), but also that
> our transformer decoder is capable of generating clinically relevant reports
> (captioning macro-F1 score of 0.349 using CheXpert labeler) and responding
> to prompts with targeted, interactive outputs.

## Introduction

Automated medical image analysis has the potential to revolutionize healthcare
diagnostics, but its efficacy is often constrained by the limited availability
of annotated data. Acquiring high-quality annotations is time-consuming and
requires expert knowledge, making fully supervised modeling approaches
impractical.

The vast repository of radiology reports—unstructured textual descriptions
paired with radiographs—offers a valuable resource. By leveraging these paired
reports, it is feasible to improve the quality of visual representations and
mitigate the challenges posed by limited labeled data. Recently, contrastive
learning has gained attention with impressive results shown in various
domains. Notably, ConVIRT (Zhang et al. 2022) first demonstrated dual-encoder
contrastive learning with radiology reports to improve image representations,
and CheXzero (Tiu et al. 2022) highlighted the impressive zero-shot
capabilities of a dual-encoder architecture. However, a limitation of methods
based on contrastive learning is their inability to provide interpretable
outputs akin to comprehensive radiology reports. By probing the visual
embedding with textual embeddings, an indirect interpretation can be achieved
but this requires careful formulation of textual inputs, and is not fully
studied beyond classification tasks. Thus, while contrastive learning based
methods improve visual representations, further steps are necessary to ensure
these models deliver clinically meaningful and understandable results.

Figure 1: Overview of RadTex outputs for interpretability Figure 2: RadTex
architecture overview, showing available training data, bidirectional
captioning for pretraining, downstream classification training, and prompted
captioning. In prompted captioning, the outputs from the classification module
are used to construct the starting sequence for forward captioning. Note that
the arrows in the Data Source Key pane denote which input is applied during
training and inference in the other panes.

In this work, our objective is twofold: improve medical image representations
and simultaneously generate interpretable radiology reports. We present
RadTex, a pretraining approach using bidirectional captioning to learn medical
image representations. Figure 1 provides a visual explanation of the RadTex
capabilities. Given an image, RadTex is able to utilize visual embeddings to
output classification labels and a detailed radiology report (i.e. unprompted
report). In an automated diagnostics advancement, RadTex can also process
user-specified textual prompts. For example, given an image and a prompt
pleural effusion, RadTex generates Possible small right pleural effusion
providing image-aligned explanations. Furthermore, RadTex can ingest discrete
classification labels, translating them into detailed diagnostic narratives
elucidating each classification output. To the best of our knowledge, this is
the first framework to prompt a transformer decoder with textual inputs to
output interpretable pathological findings.

While providing a flexible prompting mechanism that outputs interpretation of
diagnostics, RadTex achieves competitive performance compared to state-of-the-
art contrastive learning methods. On the CheXpert competition dataset, we
achieve 89.4% AUC with frozen linear classification, which is a 0.4%
improvement over the best contrastive learning method. The gain in performance
is accompanied by increased interpretability. Our main contributions are
summarized below:

  * •

We present optimal architecture and hyperparameters for RadTex, extending the
contributions of VirTex (Desai and Johnson 2021) for data-efficient
pretraining to the radiology domain.

  * •

We show that the bicaptioning pretraining task is competitive with contrastive
image-text pretraining for learning strong visual representations that
transfer efficiently to downstream medical image classification tasks.

  * •

We demonstrate RadTex’s capability to generate clinically accurate radiology
reports using flexible user-driven prompts, indicating a promising avenue for
enhanced radiologist-AI interaction.

## Related Work

Our work is most relevant to vision-language pretraining in the medical domain
and automated radiology report generation.

#### Vision-Language Pretraining

While great strides have been made in supervised deep learning using
convolutional neural networks and transformers (He et al. 2015; Vaswani et al.
2017; Dosovitskiy et al. 2021), limited labeled examples and time-consuming
annotation in the medical domain limit progress of such approaches. Two
approaches have been effective at countering this labeled data barrier: the
release of large-scale weakly annotated datasets (Irvin et al. 2019; Johnson
et al. 2019b) and the development of self-supervised vision-language modeling.

ConVIRT (Zhang et al. 2022) first proposed adapting contrastive learning (Chen
et al. 2020) to vision-language tasks for learning visual representations.
Their work inspired Radford et al. (2021), whose CLIP model quickly became
state-of-the-art for natural image-text representation learning. CLIP’s dual-
encoder architecture has been adapted back to radiology as CheXzero (Tiu et
al. 2022), and improved with more careful selection of image-text pairs (Wang
et al. 2022) and in-domain pretraining (Mishra et al. 2023).

Additionally, self-supervised learning approaches in the general domain have
utilized encoder-decoder architectures for image-text pretraining (Lu et al.
2019; Li et al. 2020; Zhou et al. 2020). Recently, Yu et al. (2022b)
demonstrated the impressive potential of such encoder-decoder architectures as
foundation models that can be transferred to downstream tasks by using
contrastive and captioning losses. (Zhang et al. 2022) experimented with a
Meshed Memory transformer for learning visual representations, but found
performance worse than contrastive learning. Our architecture, inspired by
VirTex (Desai and Johnson 2021), employs an encoder-decoder architecture,
learning visual representations with a bidirectional image captioning task.
This approach not only produces interpretable radiology reports but also
offers competitive performance compared to contrastive learning techniques in
downstream vision tasks.

#### Radiology Report Generation

Many authors have proposed methods for radiology report generation from chest
X-rays, which have the potential to improve clinical interpretability of deep
learning methods by describing clinical observations in plain text. The image-
captioning task has been studied extensively in the natural image domain
(Vinyals et al. 2015; Stefanini et al. 2023), but captioning in the radiology
domain has distinct challenges which require unique approaches. To address
challenges of linguistic and semantic consistency of generated reports, which
are typically longer than natural image captions, many authors have proposed
using hybrid retrieval-generation models or templates (Li et al. 2018; Pino et
al. 2021). Endo et al. (2021) proposed a report generation system that uses
vision-language pretraining to improve retrieval.

Others have explored neural text generation for radiology, which has the
potential to generate observations and relations purely based on the image.
(Jing, Xie, and Xing 2018) and (Wang et al. 2018) use LSTM-based generation,
while (Miura et al. 2021) and (Chen et al. 2022) utilize transformers (Vaswani
et al. 2017) to generate relevant clinical information. Many of these encoder-
decoder architectures utilize a semantic feature extractor to focus generation
on clinically relevant observations, and we build on their approaches with
Prompted Captioning.

## Method

We introduce RadTex, which employs bidirectional captioning (bicaptioning) as
a pretraining approach for interpretable medical image analysis. The overall
framework is presented in Figure 2. We adopt the bicaptioning framework by
Desai and Johnson (2021) to implement the RadTex model.

### Pretraining

Pretraining consists of jointly training an image encoder and a text decoder
via bidirectional image captioning. Training begins with a ResNet50 (He et al.
2015) as our visual encoder, extracting visual features from an X-ray image
$I$. We take the output of the last convolutional layer as spatial image
features, and use a linear projection to create a sequence of image features
for “translation,” which we call $x_{vis}$. A paired radiology report $R$ is
processed by tokenizer $\phi_{TOK}$, and resultant tokens
$w=\\{w_{0},...,w_{T+1}\\}$ make up the input sequence of length $T+2$ ,
including the special [SOS] and [SEP] tokens at either end. Each token
$w_{i}\in D$, the vocabulary of the tokenizer. Tokens $w$ are encoded by a
learned word and positional embedding into a textual sequence $x_{text}$.

Textual decoding during pretraining is carried out by a transformer decoder
(Vaswani et al. 2017) with self-attention and multihead attention modules. The
$t$-th token $w_{t}$ in the report is predicted twice by the transformer
decoder during pretraining: once forward and once backward. In each layer of
the forward transformer decoder, masked multiheaded self-attention is applied
to the input token embeddings $x_{text}$ (token embeddings after $t$ are
masked), followed by multiheaded attention applied to both transformed token
embeddings and the visual embedding sequence $x_{vis}$. Between these
attention modules, element-wise sum, layer normalization, and dropout are
applied and residual connections are made around attention layers. This
architecture is duplicated for the backward captioning transformer, and
textual sequence is reversed before input.

At the output of the last transformer layer, a single linear layer (weights
shared between forward and backward decoders) is applied to predict the
element of the vocabulary. We compute cross entropy loss to measure the
difference between predicted token and the ground truth, minimizing negative
log-likelihood as follows:

$L=-\sum^{T+1}_{t=1}\log p(w_{t}|I,w_{i<t};\theta,\phi_{f})-\sum^{T}_{t=0}\log
p(w_{t}|I,w_{i>t};\theta,\phi_{b})$ (1)

Here, $w_{i<t}$ and $w_{i>t}$ represent the tokens appearing before and after
the $t$-th token respectively. The parameter $\theta$ corresponds to the
parameters of the image encoder and $\phi_{f}$ and $\phi_{b}$ represent the
parameters for the forward and backward decoders. In this way, prediction of
$w_{t}$ is conditioned on {$w_{i<t}$, $x_{vis}$} for forward captioning and
{$w_{i>t}$, $x_{vis}$} for backward captioning.

### Explainable Captioning

Given the pretrained ability to predict the probability of a token $w_{t}$
with either {$w_{i<t},x_{vis};\phi_{f}$} or {$w_{i>t},x_{vis};\phi_{b}$}, we
can utilize our learned model from pretraining for autoregressive captioning.
Autoregressive captioning uses an iterative approach to build up a caption
from an initial sequence, appending a token at each step. During each step,
token probabilities are computed using the frozen visual encoder and textual
decoder, conditional on the input image $I$ and the pre-existing tokens in the
sequence. An algorithm (denoted $\eta$) such as beam search or nucleus
sampling (Holtzman et al. 2020) can then be used to sample a token from the
computed probabilities. We formalize our autoregressive captioning in
Algorithm 1.

Algorithm 1 Autoregressive Prompted Captioning

Input: Prompt $P$; Sampling Strategy $\eta$; Image $I$
Output: Caption tokens $w$

1:$x_{vis}\leftarrow\theta(I)$ $\triangleright$ Visual Embedding

2:$w\leftarrow[\phi_{TOK}(\texttt{[SOS]}),\phi_{TOK}(P)]$ $\triangleright$
Tokenize

3:$w^{*}\leftarrow[\phi_{TOK}(\texttt{"."}),\phi_{TOK}(\texttt{[SEP]})]$
$\triangleright$ Stop Tokens

4:$t\leftarrow$ length($w$) + 1

5:while $t<\text{max caption length}$ do

6: $x_{text}\leftarrow\phi_{EMB}(w)$ $\triangleright$ Textual Embedding

7: $l\leftarrow\phi_{f}(x_{vis},x_{text})$ $\triangleright$ Token Logits

8: $w_{t}\leftarrow\eta(l)\in D$ $\triangleright$ New token

9: $w[t]\leftarrow w_{t}$

10: $t\leftarrow t+1$

11: if $w_{t}\in w^{*}$ then break

In Algorithm 1, an arbitrary sequence $P$ can be introduced to the model to
form the starting sequence for captioning. Since token probability prediction
is conditional not only on the image features, but also on the preexisting
tokens, adding tokens to the starting sequence serves to encourage the model
to align its token prediction with this sequence, and to generate relevant
textual outputs. We refer to this as prompting the textual decoder, and
conduct experiments around this capability to explore its application in
radiology.

With the captioning formulation in Algorithm 1, we treat Unprompted captioning
as a special case of prompted captioning where the Prompt $P$ is empty,
leaving $w=[\phi_{TOK}(\texttt{[SOS]})]$ as our initial prompt token. Only
forward captioning is presented, but backward captioning requires just a
change to the $w^{*}$ special token embeddings, the initialization of $w$ with
". [SEP]", and $\phi_{f}\rightarrow\phi_{b}$. The $w^{*}$ token embeddings
serve to break the prompted captioning process into sentence-length outputs,
allowing us to structure a radiology report with multiple prompts, but the "."
can be omitted for full context-length unprompted captioning.

We propose two different methods for prompting. In the first, which we refer
to as Prompted captioning, we add a word or phrase (typically a pathology of
interest) to $P$, and allow the model to run forward captioning given this
starting sequence and generate output tokens $w$. In the second, which we
refer to as Iterative Prompted captioning, we follow the Prompted procedure,
but we prompt the model once more with outputs $w_{i>0}$ in the backward
captioning setting. This allows the model to generate additional tokens before
and after the initial prompt $P$. Given the typically rigid linguistics of
radiology reports, allowing for forward and backward captioning on the same
prompt may allow the generated text to fully describe a finding with similar
semantics to those seen during pretraining.

## Datasets

We use MS-COCO (Lin et al. 2014) and MIMIC-CXR (Johnson et al. 2019a, b) for
pretraining, and transfer our visual encoder to a variety of downstream
datasets. Dataset details are described here:

MS-COCO (Lin et al. 2014) comprises natural images each paired with captions.
For pretraining the RadTex model, we use the official 2017 split with 118K
images.

MIMIC-CXR (Johnson et al. 2019a, b) provides 377,110 chest X-rays (CXRs) with
paired reports and CheXpert labels. For pre-training, we use the Findings and
Impression sections and both frontal and lateral MIMIC-CXR-JPG images. Reports
are preprocessed to remove references to prior studies in the training and
validation sets (train/val/test: 368,960/2991/5159). For downstream
classification, we use Pathology9 (Liao et al. 2021), containing 9 of the 14
original CheXpert labels with over 100 test examples.

CheXpert (Irvin et al. 2019) includes 224,316 chest radiographs from Stanford
Hospital. Following the official split, our experiments focus on five
competition pathologies: Atelectasis, Cardiomegaly, Consolidation, Edema, and
Pleural Effusion.

Edema Severity (Horng et al. 2021) is derived from MIMIC-CXR, grading 7,390
radiographs for pulmonary edema severity on a 0-3 scale (0: none, 1: vascular
congestion, 2: interstitial edema, 3: alveolar edema). While most labels
derive from regex, the test set ($n=141$) uses consensus labels from
radiologists. This test set is unseen during MIMIC-CXR-JPG pretraining.

RSNA Pneumonia (Shih et al. 2019) comprises approximately 30,000 frontal
radiographs from NIH CXR-8 (Wang et al. 2017), labeled for pneumonia presence.

## Experiments

We perform an extensive set of experiments to evaluate our model. We run an
ablation study on a bicaptioning pretraining architecture to determine the
final RadTex model. To assess the efficacy of the vision encoder, we freeze
RadTex encoder weights and train a linear classifier, comparing our vision
encoder with those from contrastive pretraining on various downstream tasks.
Furthermore, we evaluate the interpretability of captioning by prompting the
transformer decoder of pretrained RadTex models.

### Implementation Details

Images are resized to $224\times 224$ for processing by the vision encoder.
The last convolutional layer of the ResNet50 returns spatial image features of
size $7\times 7\times 2048$.

All models are pretrained with an SGD optimizer with Lookahead ($k=5$,
$\alpha=0.5$, as in (Zhang et al. 2019)). A linear warmup of the learning rate
(5% of total steps) is followed by cosine decay, with maximum LR of 0.4 and
0.002 for visual encoder and transformer decoder, respectively. Models are
pretrained on 8 Nvidia V100 GPUs with an effective batch size of 512 samples.
We monitor validation loss on the pretraining task and use the best model
checkpoint for downstream tasks.

For all downstream tasks, we extract the visual encoder from the pretrained
model and freeze model weights. We train a classification head for the visual
encoder on a single V100 GPU with a learning rate of 0.02 and batch size of
16, reducing LR by half if validation loss has not decreased after 5 epochs.
All reported downstream results are the mean and standard deviation of three
trials, unless otherwise stated. We provide our code for reproducibility.

| AUC (%)
---|---
50K | $79.3_{\pm 0.1}$
100K | $\mathbf{80.1}_{\pm 0.1}$
200K | $79.8_{\pm 0.1}$
(a) Pretraining Steps

| AUC (%)
---|---
L = 1 | $79.3_{\pm 0.1}$
L = 2 | $\mathbf{80.1}_{\pm 0.1}$
L = 4 | $79.7_{\pm 0.1}$
(b) Transformer Layers

| AUC (%)
---|---
SciBERT | $\mathbf{79.3_{\pm 0.1}}$
COCO SP | $\mathbf{79.4_{\pm 0.0}}$
MIMIC SP | $79.0_{\pm 0.1}$
(c) Tokenizer Vocabulary

| AUC (%)
---|---
Removed | $\mathbf{79.3}_{\pm 0.1}$
Present | $78.7_{\pm 0.1}$
(d) Prior Report Comparisons

Table 1: Ablation study of RadTex architecture on Pathology9. Average of three
trials is reported with $\text{Mean}_{\pm\text{SD}}$.

### Pretraining Ablations

Prior to the ablation study, we make adjustments to key settings to better
suit the general-domain bicaptioning model (i.e. VirTex by Desai and Johnson
2021) for the radiology domain. Specifically:

  * •

Context length: Expanded from 30 to 170 tokens, to encompass 99% of MIMIC-CXR
Findings and Impression.

  * •

Tokenizer vocabulary: scibert_scivocab_uncased (Beltagy, Lo, and Cohan 2019)
used instead of SentencePiece vocabulary (Kudo and Richardson 2018), following
Chauhan et al. (2020) for MIMIC-CXR tokenization.

  * •

Normalization: Adjusted to match the mean and standard deviation of the MIMIC-
CXR-JPG training corpus.

  * •

Prior reference removal: Filtered out sentences that reference previous scans
using regex.

With these settings as our baseline, we evaluate ablations on Pathology9
classification task as summarized in Table 1.

#### Pretraining Steps

We increase the number of pretraining steps of our baseline model from 50K to
100K and 200K, a change which amounts to an adjustment to the cosine decay
scheduler. As shown in Table 1(a), we find that both improve downstream
classification performance, but the model with 100K steps performs the best.

#### Transformer Layers

Following the ablation study in VirTex (Desai and Johnson 2021), we experiment
with additional layers in the transformer decoder. Notably, we observe
improvement in the Pathology9 AUC with 2 layers. Results are shown in Table
1(b).

#### Tokenizer Vocabulary

We test the new tokenizer against COCO SentencePiece and MIMIC SentencePiece
(computed from the MIMIC training corpus). Surprisingly, despite an ability to
represent words in radiology reports with fewer tokens given the in-domain
vocabulary, the MIMIC SentencePiece model performs worse when it comes to
downstream classification (Table 1(c)). The COCO model performs slightly
better than our baseline despite the out-of-domain vocabulary, but by a
negligible margin ($p<0.05$ in a one-sided t-test).

#### Prior Reference Removal

Another key setting for our baseline model is the removal of sentences in the
training corpus containing the phrases “prior” and “compar.” We hypothesize
that the presence of these phrases trains the model to hallucinate prior
studies, of which it could have no knowledge. Removing the phrases is shown to
both improve downstream classification (Table 1(d)) and reduce the number of
references to prior reports, but may slightly harm clinical efficacy during
captioning (Table 2).

#### Final RadTex Model

Our final RadTex model incorporates baseline settings and findings from the
ablation study (100K steps, 2 layers, scibert_scivocab vocabulary, priors
removed). This model achieves a Pathology9 AUC of $80.5$%, which exceeds all
other ablations.

For visual encoder quality experiments, we pretrain RadTex on MS-COCO before
MIMIC-CXR for better weight initialization. We report these results as
RadTex/COCO+MIMIC, or simply RadTex, while the model trained soley on MIMIC-
CXR is denoted as RadTex/MIMIC.

(a) CheXpert Competition AUC

(b) Edema Severity F1

(c) RSNA Pneumonia AUC

Figure 3: RadTex is competitive with CheXzero and other methods on a variety of downstream classification tasks. RadTex results are for RadTex/COCO+MIMIC pretraining. Downstream tasks are trained in three trials, with $\text{Mean}_{\pm\text{SD}}$ across 3 runs displayed. Pretraining | Priors (%) | macro-F1
---|---|---
Ground Truth | 52.5 | —
Unprompted (Priors Present) | 43.0 | 0.254
Unprompted (Priors Removed) | 0.1 | 0.242
Table 2: Removing prior report comparisons reduces hallucinated references.
Priors indicate the percentage of reports with prior keywords, and macro-F1 is
evaluated against ground truth by CheXpert labeler.

### Visual Downstream Tasks

To evaluate RadTex’s pretrained visual encoder, we compare to contrastive
pretraining, which has previously been shown to perform better than captioning
pretraining on natural image (Radford et al. 2021) and radiology datasets
(Zhang et al. 2022). We compare performance on three downstream classification
tasks: CheXpert (multilabel classification on five competition pathologies),
RSNA pneumonia detection (binary classification), and Pulmonary Edema Severity
classification (multiclass grading).

For a direct comparison of RadTex with contrastive learning, we obtain the
final visual encoder weights from CheXzero (ViT-B/32), BiomedCLIP (ViT-B/16),
and OpenAI’s CLIP model (ResNet50). CheXzero is initialized from CLIP and
pretrained on the Impressions section from MIMIC-CXR; BiomedCLIP, with an
initialized text encoder, is pretrained on PMC-15M, with 15 million image-text
pairs from PubMed; OpenAI’s CLIP is pretrained on 400M internet-sourced image-
text pairs. We use our COCO/MIMIC-pretrained RadTex model for experiments.

To compare RadTex with other models, we freeze visual encoder weights and
train a randomly initialized linear classification head on varying amounts of
data for each classification task. We use a fully supervised ResNet50 model as
a baseline. Frozen linear classification performance is considered to be a
good measure of visual encoder performance, as it is a reflection of visual
embedding quality.

Figure 3 shows RadTex visual encoder performance against other contrastive
methods on downstream tasks. RadTex is competitive, surpassing CheXzero on
CheXpert Competition and Edema Severity tasks with most amounts of labeled
data, despite CheXzero’s vision transformer architecture and larger size
(87.9M vs. 23.5M parameters). RSNA Pneumonia detection yields similar results
between CheXzero and RadTex. OpenAI’s CLIP, pretrained on general web data
performs poorly on the radiology-domain tasks with a frozen visual encoder.

| CheXpert Competition AUC (%)
---|---
| 1% | 10% | 100%
Fully Supervised | $71.9_{\pm 2.6}$ | $82.3_{\pm 3.5}$ | $85.0_{\pm 4.7}$
OpenAI CLIP | $59.8_{\pm 0.4}$ | $72.4_{\pm 0.6}$ | $73.3_{\pm 0.2}$
ConVIRT | $85.9$ | $86.8$ | $87.3$
BiomedCLIP | $77.2_{\pm 0.9}$ | $82.1_{\pm 2.6}$ | $83.1_{\pm 0.5}$
CheXzero | $88.9_{\pm 0.6}$ | $89.1_{\pm 0.2}$ | $89.0_{\pm 0.4}$
VirTex | $86.6_{\pm 0.8}$ | $86.7_{\pm 0.7}$ | $87.3_{\pm 0.2}$
RadTex/MIMIC | $88.4_{\pm 0.2}$ | $89.2_{\pm 0.5}$ | $89.0_{\pm 0.4}$
RadTex | $\mathbf{89.2_{\pm 0.4}}$ | $\mathbf{89.6_{\pm 0.1}}$ | $\mathbf{89.4_{\pm 0.1}}$
Table 3: CheXpert linear classification results with variable amounts of
downstream training data. $\text{Mean}_{\pm\text{SD}}$ AUC across 3 trials
presented. ConVIRT results are from (Zhang et al. 2022). RadTex (bottom)
denotes RadTex/COCO+MIMIC, with VirTex pretrained using identical data.

For RadTex, initialization with pretraining on a natural image dataset does
improve performance slightly. In Table 3, we find improvement from
RadTex/MIMIC to RadTex/COCO+MIMIC at all labeled data levels. We also compare
to a VirTex model, pretrained with the same settings as (Desai and Johnson
2021) on COCO followed by MIMIC-CXR. Improvement from this VirTex/COCO+MIMIC
model shows that by optimizing for bicaptioning settings on the radiology
domain, we achieve better performance.

### Interpretable Report Generation

| Textual Similarity | Clinical Efficacy
---|---|---
Model | BLEU-2 | BERTScore† | CheXpert macro-F1 | CheXbert Sim.† | RadGraph F1†
Random Report Retrieval | 0.089 | 0.213 | 0.177 | 0.166 | 0.048
M2 Trans (Miura et al. 2021) | – | 0.227 | 0.304 | 0.268 | 0.110
R2Gen (Chen et al. 2022) | 0.218 | 0.186 | 0.276 | 0.204 | 0.057
CXR-RePaiR (Endo et al. 2021) | 0.069 | 0.191 | 0.256 | 0.379 | 0.091
RadTex Unprompted | 0.100 | 0.261 | 0.289 | 0.259 | 0.096
RadTex Prompted | 0.069 | 0.262 | 0.349 | 0.336 | 0.098
RadTex Iterative Prompted | 0.082 | 0.271 | 0.349 | 0.333 | 0.112
Table 4: Comparison of radiology report captioning techniques on a range of
metrics. BLEU and BERTScore represent measures of textual similarity, without
clinical awareness. CheXpert F1, CheXbert, and RadGraph F1 scores represent
measures of clinical efficacy. Other model scores are drawn from existing
literature, and we follow their setups as described when comparing RadTex
results. †Following (Yu et al. 2022a) and using only Impression section for
ground truth. For BLEU-2 and CheXpert F1 scores, R2Gen and CXR-RePaiR compare
to ground truths with both Findings and Impression, while M2 Trans uses just
Findings.

Bicaptioning pretraining enables projection of visual embeddings to
interpretable language outputs. Using autoregressive captioning (see Alg. 1),
RadTex can produce a radiology report based on visual embedding, with an
optional prompt. We explore Unprompted, Prompted, and Iterative Prompted
captioning settings. Nucleus sampling, found superior in early tests, is our
default sampling strategy over beam search.

When employing Prompted Captioning, any word or sequence of tokens can be used
as a prompt, potentially eliciting insights into rare or abnormal pathologies.
For optimal clinical efficacy in radiology reports, we use the outputs from a
Pathology9 classifier. We determine prompts based on thresholds set by
maximizing individual pathology $\textit{TPR}*(1-\textit{FPR})$ on the
validation set, where TPR is the true positive rate and FPR the false positive
rate. The model is prompted with only the positive pathologies, and their
outputs are concatenated to form the final report.

#### Measuring Caption Quality

We generate captions for 5,159 MIMIC-CXR test set radiographs. Table 4
compares RadTex to some of the best-performing radiology report generation
models: $M^{2}$ Trans (Cornia et al. 2020; Miura et al. 2021), R2Gen (Chen et
al. 2022), and CXR-RePaiR (Endo et al. 2021). Evaluation metrics include
BLEU-2 and BERTScore (Zhang et al. 2020) for textual similarity, macro-F1 from
the CheXpert labeler, CheXbert similarity, and RadGraph F1 for clinical
efficacy. Boag et al. (2020) noted that randomly retrieved reports score
surprisingly well on these metrics, and these results are included in our
baseline.

RadTex scores well on the five reported metrics across the board. The
Unprompted model achieves higher BERTScore and trails only M2 Trans in
CheXpert F1 among baseline methods. RadTex Prompted achieves similar RadGraph
F1 score to Unprompted, yet Iterative Prompted improves this score, possibly
benefiting from its ability to modify textual output with bidirectional
captioning.

Image and Attention |  Ground Truth |  Unprompted Report |  Prompted Report
---|---|---|---
|  right port catheter tip ends in the mid svc. sternal wires are intact and midline. there are small bilateral pleural effusions, slightly larger on the left than on the right. the cardiac silhouette is moderately enlarged. there is mild engorgement of the pulmonary vasculature. there has been improvement in the previously noted pulmonary edema with minimal residual edema. there is plate-like atelectasis seen in the left base. there is no consolidation or pneumothorax. number. small bilateral pleural effusions. number. improvement in pulmonary edema. |  left-sided central venous catheter terminates in the mid svc. overall cardiac size is top normal. aortic knob is noted. patient is status post median sternotomy and intact median sternotomy with intact median sternotomy wires. mediastinal contours are stable in appearance. residual right pleural effusion with airspace opacity seen on the left is present, likely atelectasis. atelectasis is seen at the left lung base. no pneumothorax. if there is a small left pleural effusion. |  atelectasis at the left base and no evidence of pneumonia, vascular congestion, or pleural effusion. cardiomegaly is unchanged. pleural effusions have decreased in size and mild cardiomegaly and the previous right lower lobe. pneumothorax in the right lower lobe remain in constant position.
Table 5: RadTex captioning example with CXR image (top) and averaged first
layer cross-attention overlaid (bottom) for a sequence of words in the
Unprompted Report. Light refers to higher attention.

#### RadTex Explainability

Figure 4: Classification (CheXpert AUC) and Unprompted Captioning (RadGraph
F1) performance during RadTex/MIMIC training. Model is trained a total of 100K
steps or $\sim 140$ epochs.

Observations of RadTex reports by authors and an attending radiologist suggest
that prompting may improve explainability of the reports. By prompting with a
pathology, we aim to probe its visual embedding and encourage the model to
discuss related observations as shown in Table 5. Yet, the radiologist pointed
out instances where RadTex generated reports that connected unrelated
concepts, such as associating pathologies with no causal link. This
underscores the need for better metrics on report explainability, potentially
more suited for assistive diagnostics systems than just textual similarity and
clinical efficacy metrics.

In Figure 4, we evaluate both linear classification (CheXpert competition AUC)
and captioning performance (RadGraph F1) at every 10th epoch during training
of RadTex/MIMIC. We find that the capability to differentiate pathologies
develops in the model before the ability to mention correct entities and
relations in the generated reports. We believe that by re-optimizing RadTex
hyperparameters and settings for captioning (including by training the model
for longer), we may be able to achieve even stronger captioning performance.

## Conclusion and Future Work

In this paper, we show the efficacy of bidirectional captioning as a
pretraining strategy for interpretable medical image analysis. Our model,
RadTex, not only yields competitive performance against contrastive methods in
visual downstream tasks but also surpasses CheXzero on CheXpert and Edema
Severity linear classification. A distinct advantage of RadTex is its
capability to generate radiology reports, providing levels of interpretability
not achieved by other pretraining methods. Additionally, we introduce a
flexible prompting mechanism in RadTex for pathologies of interest,
demonstrating its potential in improving clinical efficacy.

A future iteration of RadTex could study methods to further improve
interpretability. Adopting the strategy proposed by Ramesh, Chi, and Rajpurkar
(2022), which involves removing prior references using a token classifier,
could further increase temporal consistency and remove hallucinations from
report generation. Another promising area of future work is alignment of the
visual embeddings with textual encoders for zero-shot classification, or with
LLM token embeddings to refine report generation and visual question
answering. Recent studies have begun to explore this area, as seen in the work
by (Xu et al. 2023).

##  Acknowledgments

This work was supported in part by MIT Lincoln Laboratory, US Air Force, NIH
NIBIB NAC P41EB015902, Wistron, IBM Watson, MIT Deshpande Center, and MIT
J-Clinic.

DISTRIBUTION STATEMENT A. Approved for public release. Distribution is
unlimited. This material is based upon work supported by the Department of the
Air Force under Air Force Contract No. FA8702-15-D-0001. Any opinions,
findings, conclusions or recommendations expressed in this material are those
of the author(s) and do not necessarily reflect the views of the Department of
the Air Force. © 2023 Massachusetts Institute of Technology. Delivered to the
U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013
or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government
rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as
detailed above. Use of this work other than as specifically authorized by the
U.S. Government may violate any copyrights that exist in this work.

## References

  * Beltagy, Lo, and Cohan (2019) Beltagy, I.; Lo, K.; and Cohan, A. 2019.  SciBERT: A Pretrained Language Model for Scientific Text.  In _EMNLP_.
  * Boag et al. (2020) Boag, W.; Hsu, T.-M. H.; Mcdermott, M.; Berner, G.; Alesentzer, E.; and Szolovits, P. 2020.  Baselines for Chest X-Ray Report Generation.  In _Proceedings of the Machine Learning for Health NeurIPS Workshop_.
  * Chauhan et al. (2020) Chauhan, G.; Liao, R.; Wells, W.; Andreas, J.; Wang, X.; Berkowitz, S.; Horng, S.; Szolovits, P.; and Golland, P. 2020.  Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment.  In _MICCAI_.
  * Chen et al. (2020) Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020.  A Simple Framework for Contrastive Learning of Visual Representations.  In _Proceedings of the 37th International Conference on Machine Learning_.
  * Chen et al. (2022) Chen, Z.; Song, Y.; Chang, T.-H.; and Wan, X. 2022.  Generating Radiology Reports via Memory-driven Transformer.  arXiv:2010.16056.
  * Cornia et al. (2020) Cornia, M.; Stefanini, M.; Baraldi, L.; and Cucchiara, R. 2020.  Meshed-Memory Transformer for Image Captioning.  arXiv:1912.08226.
  * Desai and Johnson (2021) Desai, K.; and Johnson, J. 2021.  VirTex: Learning Visual Representations from Textual Annotations.  In _CVPR_.
  * Dosovitskiy et al. (2021) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _ICLR_.
  * Endo et al. (2021) Endo, M.; Krishnan, R.; Krishna, V.; Ng, A. Y.; and Rajpurkar, P. 2021.  Retrieval-Based Chest X-Ray Report Generation Using a Pre-trained Contrastive Language-Image Model.  In _Proceedings of Machine Learning for Health_.
  * He et al. (2015) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015.  Deep Residual Learning for Image Recognition.  ArXiv:1512.03385 [cs].
  * Holtzman et al. (2020) Holtzman, A.; Buys, J.; Du, L.; Forbes, M.; and Choi, Y. 2020.  The Curious Case of Neural Text Degeneration.  In _ICLR_.
  * Horng et al. (2021) Horng, S.; Liao, R.; Wang, X.; Dalal, S.; Golland, P.; and Berkowitz, S. J. 2021.  Deep Learning to Quantify Pulmonary Edema in Chest Radiographs.  _Radiology: Artificial Intelligence_.
  * Irvin et al. (2019) Irvin, J.; Rajpurkar, P.; Ko, M.; Yu, Y.; Ciurea-Ilcus, S.; Chute, C.; Marklund, H.; Haghgoo, B.; Ball, R.; Shpanskaya, K.; Seekins, J.; Mong, D. A.; Halabi, S. S.; Sandberg, J. K.; Jones, R.; Larson, D. B.; Langlotz, C. P.; Patel, B. N.; Lungren, M. P.; and Ng, A. Y. 2019.  CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison.  _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Jing, Xie, and Xing (2018) Jing, B.; Xie, P.; and Xing, E. 2018.  On the Automatic Generation of Medical Imaging Reports.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_.
  * Johnson et al. (2019a) Johnson, A. E. W.; Pollard, T. J.; Berkowitz, S. J.; Greenbaum, N. R.; Lungren, M. P.; Deng, C.-y.; Mark, R. G.; and Horng, S. 2019a.  MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports.  _Scientific Data_.
  * Johnson et al. (2019b) Johnson, A. E. W.; Pollard, T. J.; Greenbaum, N. R.; Lungren, M. P.; ying Deng, C.; Peng, Y.; Lu, Z.; Mark, R. G.; Berkowitz, S. J.; and Horng, S. 2019b.  MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs.  arXiv:1901.07042.
  * Kudo and Richardson (2018) Kudo, T.; and Richardson, J. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing.  In _EMNLP: System Demonstrations_.
  * Li et al. (2020) Li, G.; Duan, N.; Fang, Y.; Gong, M.; and Jiang, D. 2020.  Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training.  _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Li et al. (2018) Li, Y.; Liang, X.; Hu, Z.; and Xing, E. P. 2018.  Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation.  In _NeurIPS_.
  * Liao et al. (2021) Liao, R.; Moyer, D.; Cha, M.; Quigley, K.; Berkowitz, S.; Horng, S.; Golland, P.; and Wells, W. M. 2021.  Multimodal Representation Learning via Maximization of Local Mutual Information.  In _MICCAI_.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft COCO: Common Objects in Context.  In _ECCV_.
  * Lu et al. (2019) Lu, J.; Batra, D.; Parikh, D.; and Lee, S. 2019.  ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks.  In _NeurIPS_.
  * Mishra et al. (2023) Mishra, A.; Mittal, R.; Jestin, C.; Tingos, K.; and Rajpurkar, P. 2023.  Improving Zero-Shot Detection of Low Prevalence Chest Pathologies using Domain Pre-trained Language Models.  arXiv:2306.08000.
  * Miura et al. (2021) Miura, Y.; Zhang, Y.; Tsai, E.; Langlotz, C.; and Jurafsky, D. 2021.  Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Pino et al. (2021) Pino, P.; Parra, D.; Besa, C.; and Lagos, C. 2021.  Clinically Correct Report Generation from Chest X-Rays Using Templates.  In _Machine Learning in Medical Imaging_. Springer International Publishing.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021.  Learning Transferable Visual Models From Natural Language Supervision.  arXiv:2103.00020.
  * Ramesh, Chi, and Rajpurkar (2022) Ramesh, V.; Chi, N. A.; and Rajpurkar, P. 2022.  Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors.  In _Proceedings of the 2nd Machine Learning for Health symposium_.
  * Shih et al. (2019) Shih, G.; Wu, C. C.; Halabi, S. S.; Kohli, M. D.; Prevedello, L. M.; Cook, T. S.; Sharma, A.; Amorosa, J. K.; Arteaga, V.; Galperin-Aizenberg, M.; Gill, R. R.; Godoy, M. C.; Hobbs, S.; Jeudy, J.; Laroia, A.; Shah, P. N.; Vummidi, D.; Yaddanapudi, K.; and Stein, A. 2019.  Augmenting the National Institutes of Health Chest Radiograph Dataset with Expert Annotations of Possible Pneumonia.  _Radiology: Artificial Intelligence_.
  * Stefanini et al. (2023) Stefanini, M.; Cornia, M.; Baraldi, L.; Cascianelli, S.; Fiameni, G.; and Cucchiara, R. 2023.  From Show to Tell: A Survey on Deep Learning-Based Image Captioning.  _IEEE Transactions on Pattern Analysis & Machine Intelligence_.
  * Tiu et al. (2022) Tiu, E.; Talius, E.; Patel, P.; Langlotz, C. P.; Ng, A. Y.; and Rajpurkar, P. 2022.  Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning.  _Nature Biomedical Engineering_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention is All you Need.  In _NeurIPS_.
  * Vinyals et al. (2015) Vinyals, O.; Toshev, A.; Bengio, S.; and Erhan, D. 2015.  Show and Tell: A Neural Image Caption Generator.  arXiv:1411.4555.
  * Wang et al. (2017) Wang, X.; Peng, Y.; Lu, L.; Lu, Z.; Bagheri, M.; and Summers, R. M. 2017.  ChestX-ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases.  In _CVPR_.
  * Wang et al. (2018) Wang, X.; Peng, Y.; Lu, L.; Lu, Z.; and Summers, R. M. 2018.  TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays.  In _CVPR_.
  * Wang et al. (2022) Wang, Z.; Wu, Z.; Agarwal, D.; and Sun, J. 2022.  MedCLIP: Contrastive Learning from Unpaired Medical Images and Text.  In _EMNLP_.
  * Xu et al. (2023) Xu, S.; Yang, L.; Kelly, C.; Sieniek, M.; Kohlberger, T.; Ma, M.; Weng, W.-H.; Kiraly, A.; Kazemzadeh, S.; Melamed, Z.; Park, J.; Strachan, P.; Liu, Y.; Lau, C.; Singh, P.; Chen, C.; Etemadi, M.; Kalidindi, S. R.; Matias, Y.; Chou, K.; Corrado, G. S.; Shetty, S.; Tse, D.; Prabhakara, S.; Golden, D.; Pilgrim, R.; Eswaran, K.; and Sellergren, A. 2023.  ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders.  arXiv:2308.01317.
  * Yu et al. (2022a) Yu, F.; Endo, M.; Krishnan, R.; Pan, I.; Tsai, A.; Reis, E. P.; Fonseca, E. K. U. N.; Lee, H. M. H.; Abad, Z. S. H.; Ng, A. Y.; Langlotz, C. P.; Venugopal, V. K.; and Rajpurkar, P. 2022a.  Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation.  _medRxiv_.
  * Yu et al. (2022b) Yu, J.; Wang, Z.; Vasudevan, V.; Yeung, L.; Seyedhosseini, M.; and Wu, Y. 2022b.  CoCa: Contrastive Captioners are Image-Text Foundation Models.  arXiv:2205.01917.
  * Zhang et al. (2019) Zhang, M.; Lucas, J.; Ba, J.; and Hinton, G. E. 2019.  Lookahead Optimizer: k steps forward, 1 step back.  In _NeurIPS_.
  * Zhang et al. (2020) Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K. Q.; and Artzi, Y. 2020.  BERTScore: Evaluating Text Generation with BERT.  In _ICLR_.
  * Zhang et al. (2022) Zhang, Y.; Jiang, H.; Miura, Y.; Manning, C. D.; and Langlotz, C. P. 2022.  Contrastive Learning of Medical Visual Representations from Paired Images and Text.  arXiv:2010.00747.
  * Zhou et al. (2020) Zhou, L.; Palangi, H.; Zhang, L.; Hu, H.; Corso, J.; and Gao, J. 2020.  Unified Vision-Language Pre-Training for Image Captioning and VQA.  _Proceedings of the AAAI Conference on Artificial Intelligence_.

##  Appendix A Additional Implementation Details

The transformer decoder in RadTex is implemented following VirTex (Desai and
Johnson 2021), with a hidden dimension size of 2048, 32 attention heads, and a
feedforward size of 8192. Each layer of the transformer decoder includes
dropout ($p=0.1$) subsequent to layer normalization.

### VirTex Comparison

When comparing RadTex results to other models on the CheXpert competition
classification task in Table 3, we report performance for VirTex. This VirTex
model is identical to the architecture presented in Desai and Johnson (2021),
but we re-pretrain on MS-COCO, followed by MIMIC-CXR, rather than using VirTex
weights. Crucially, we keep the context length at 30 tokens for each of these
stages of pretraining. However, because the MIMIC-CXR reports typically span
more than 30 tokens, we configure the dataloader to randomly select a sentence
from the report when loading a batch. This randomization allows the entire
report to be seen during pretraining with many epochs, giving models with
shorter context-length a better chance to perform competitively with longer
context-length models.

##  Appendix B Radiology Report Generation

In this section, we provide additional information on metrics and experiments
used to test generated report quality.

### Literature Comparisons

In Table 4, we repeat results reported in existing literature. Here, we detail
the listed values for R2Gen, M2 Trans, and CXR-RePaiR.

For metrics BERTscore, CheXbert Sim., and RadGraph F1, we take numbers
directly from (Yu et al. 2022a) (Supplementary Table 4). These listed scores
were based on comparisons to Impression section as ground truth, so we only
compare to Impression when computing our scores using their code. Their scores
also represent the average of test set studies, rather than images. To select
the X-ray used to generate a report for a given study, a PA or AP view image
is randomly selected from associated examples. If no PA or AP view is
available, a random example from the study is selected. We follow the same
selection process when computing these three metrics and comparing to their
results.

For metrics BLEU-2 and CheXpert macro-F1, we compute performance on the full
number of test set image-report pairs (5,159), and compute BLEU-2 by comparing
to the Findings and Impression sections of the report. We draw numbers for
R2Gen from Chen et al. (2022) (Table 2, Base model + relational memory +
memory-driven conditional layer normalization), and numbers for CXR-RePaiR
from Endo et al. (2021) (Table 1, CXR-RePaiR-2). For M2 Trans, we compute the
macro-F1 score based on the per-pathology F1 scores in Miura et al. (2021),
Tables 6 and 7 (NLL loss with BERTscore and Entailing Entity Match Reward).
BLEU-2 score was not reported for M2 Trans.

### Per-Pathology Report Quality

| Random Retrieval | Unprompted | Prompted | Iterative Prompted |  Train Freq.
---|---|---|---|---|---
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Atelectasis | 0.234 | 0.271 | 0.251 | 0.347 | 0.423 | 0.381 | 0.394 | 0.729 | 0.512 | 0.390 | 0.749 | 0.513 |  0.253
Cardiomegaly | 0.276 | 0.334 | 0.302 | 0.369 | 0.501 | 0.425 | 0.440 | 0.788 | 0.565 | 0.430 | 0.786 | 0.556 |  0.292
Consolidation | 0.089 | 0.082 | 0.086 | 0.151 | 0.152 | 0.152 | 0.188 | 0.514 | 0.275 | 0.185 | 0.524 | 0.274 |  0.101
Edema | 0.265 | 0.192 | 0.223 | 0.468 | 0.422 | 0.444 | 0.475 | 0.789 | 0.593 | 0.472 | 0.774 | 0.586 |  0.287
Enlarged C. | 0.088 | 0.179 | 0.118 | 0.086 | 0.220 | 0.124 | 0.099 | 0.070 | 0.082 | 0.087 | 0.070 | 0.078 |  0.095
Fracture | 0.019 | 0.023 | 0.021 | 0.085 | 0.069 | 0.076 | 0.031 | 0.006 | 0.010 | 0.023 | 0.006 | 0.009 |  0.026
Lung Lesion | 0.042 | 0.038 | 0.040 | 0.086 | 0.051 | 0.064 | 0.057 | 0.025 | 0.035 | 0.074 | 0.046 | 0.057 |  0.036
Lung Opacity | 0.324 | 0.319 | 0.322 | 0.398 | 0.391 | 0.394 | 0.450 | 0.638 | 0.528 | 0.443 | 0.669 | 0.533 |  0.255
Pleural Effusion | 0.346 | 0.314 | 0.329 | 0.543 | 0.579 | 0.560 | 0.636 | 0.761 | 0.693 | 0.613 | 0.770 | 0.683 |  0.382
Pleural Other | 0.057 | 0.034 | 0.043 | 0.049 | 0.028 | 0.035 | 0.079 | 0.034 | 0.048 | 0.055 | 0.021 | 0.030 |  0.013
Pneumonia | 0.212 | 0.175 | 0.191 | 0.299 | 0.236 | 0.264 | 0.339 | 0.633 | 0.442 | 0.341 | 0.583 | 0.430 |  0.259
Pneumothorax | 0.022 | 0.041 | 0.028 | 0.154 | 0.275 | 0.197 | 0.065 | 0.901 | 0.122 | 0.085 | 0.632 | 0.149 |  0.236
Support Devices | 0.284 | 0.392 | 0.329 | 0.523 | 0.618 | 0.566 | 0.675 | 0.636 | 0.655 | 0.663 | 0.636 | 0.649 |  0.308
No Finding | 0.186 | 0.197 | 0.192 | 0.396 | 0.338 | 0.365 | 0.562 | 0.227 | 0.323 | 0.535 | 0.247 | 0.338 |  0.334
Macro-Average | 0.174 | 0.185 | 0.177 | 0.282 | 0.307 | 0.289 | 0.321 | 0.482 | 0.349 | 0.314 | 0.465 | 0.349 |  —
Table 6: Per-pathology captioning CheXpert label scores for four different
methods of captioning on the MIMIC-CXR test set. Training set frequency, as
fraction of reports containing Chexpert-labeled pathology, is shown at the
right. Enlarged C. stands for Enlarged Cardiomediastinum. Best F1 Scores for
each pathology are bolded.

In Table 6, we report the per-pathology breakdown of CheXpert labeling for
Unprompted, Prompted, Iterative Prompted, and Random Retrieval captioning
methods. For Random Retrieval, we randomly select a report from the training
set to use as a baseline for report generation. Beyond CheXpert F1, Precision,
and Recall scores, which represent measures of clinical efficacy, we also
report the frequency of each pathology in the training dataset. This is
computed as the fraction of total 222,750 non-empty ground truth reports that
are labeled for a pathology by the Chexpert labeler.

In Figure 5, we plot these results, showing a clear positive trend between the
pathology F1 score and training frequency. When pathologies are mentioned more
frequently in the training data, they have a higher captioning F1 score in the
test data (Unprompted Captioning Pearson $r=0.92$). This trend holds for all
four captioning methods, including Random Retrieval.

Figure 5: Unprompted Captioning F1 Score vs pathology frequency in training
data. Linear regression with 95% confidence interval shown, Pearson $r=0.92$.
Similar trends hold for Prompted ($r=0.86$), Iterative Prompted ($r=0.88$),
and Random Retrieval ($r=0.82$)

### Text Generation Examples

Additional examples of radiology report captioning are shown in Table 7, along
with decoder attention during Unprompted Captioning. We average visual
sequence attention across the attention heads from the first layer of the two-
layer transformer decoder, and resize the sequence back to the 7x7 visual
features output from the ResNet50, normalizing attention between 0 and 1. We
can see in the examples that the attention is focused on clinically relevant
regions of the chest X-rays: typically over the heart and center of the chest.
We find that while attention focuses on relevant portions of the image, it
does not move around within the image very much as different tokens are
predicted.

When comparing Iterative Prompted and Prompted captioning results, we observe
that iterative prompting does effectively allow for more detailed descriptions
of pathologies. By backward captioning with a forward-prompted output,
Iterative Prompting yields reports that use modifiers to better describe
findings. For example, in Table 7, Iterative Prompting allows for
“interstitial edema…” (row 1), and “minimal atelectasis…” (row 3). We
hypothesize that the ability to add modifiers in front of pathology
descriptions may contribute to the higher RadGraph F1 score with Iterative
Prompting (Table 4).

Image and Attention |  Ground Truth |  Unprompted Report |  Prompted Report |  Iterative Prompted
---|---|---|---|---
|  No acute cardiopulmonary process. |  there is mild cardiomegaly. the aorta is mildly tortuous. no focal consolidation, pleural effusion, or pneumothorax is seen. the cardiac and mediastinal silhouettes are stable. old healed right rib fractures are present. degenerative changes are seen along the thoracic spine. no acute cardiopulmonary process. |  edema is now moderate. |  interstitial edema is again seen in the right lung.
|  As compared to the previous radiograph, the endotracheal tube, nasogastric tube and right internal jugular vein catheter are unchanged. The pre-existing pulmonary edema might have mildly improved, there is increased retrocardiac and right basal atelectasis. No pleural effusions. No major atelectasis. |  mild pulmonary edema has worsened and has not decreased. right lower lobe opacity has probably a combination of pleural effusion or alternatively lymphadenopathy. the catheter has been removed. the heart size is mildly enlarged but is normal. the pulmonary vasculature is not engorged. there is no pneumothorax or pleural effusion. no pneumothorax. sternotomy wires are noted. interval placement of a nasogastric tube whose tip number - up for its tip over number cm. number. et tube ends within number cm above the carina. new right atrium. number. |  edema has changed in the patient has worsened since. support devices are unchanged in position. |  enteric tube suggests mild pulmonary edema. monitoring and support devices remain in unchanged position, right central venous catheter in place.
|  The patient has had a prior left lower lobectomy. Since the prior exam, nodular pleural thickening encasing the left lung has increased at the expense of aeration of the left lung with stable elevation of the left hemidiaphragm. Central adenopathy in the left hilus and adjacent mediastinum has also progressed. The right lung is clear. Cardiomediastinal silhouette is unchanged. Progression of left pleural and nodal metastases. |  pa and lateral chest views were provided. there is extensive areas of consolidation in the right lung with a left lung likely representing metastatic lesion. there is a moderate right pleural effusion. there is a large left pleural effusion. there is persistent left effusion. degenerative changes noted in the thoracic spine, better visualized on the right. cardiomediastinal silhouette appears similar. bony structures are intact. the possibility of pneumonia. |  atelectasis at the left lung base. consolidation in the left upper lobe is stable. lung opacity is prominent. pleural effusion is volume loss in the right lung. pneumonia or is known between the left upper and has changed in diameter. pneumothorax. |  minimal atelectasis at the right lung base. airspace consolidation at the right upper lung is unchanged. left lower lung opacity is concerning for pneumonia, but stable. minimal right basilar loculated pleural effusion and possibly moderate on the left. unchanged right basilar opacity concerning for pneumonia is noted within the left upper lobe. there is no pneumothorax have been stable.
Table 7: RadTex captioning examples with CXR image (top) and overlaid averaged
cross-attention from the first layer (bottom), corresponding to a word
sequence in the Unprompted Report.


# Color Deconvolution applied to Domain Adaptation in HER2 histopathological
images

###### Abstract

Breast cancer early detection is crucial for improving patient outcomes. The
Institut Català de la Salut (ICS) has launched the DigiPatICS project to
develop and implement artificial intelligence algorithms to assist with the
diagnosis of cancer. In this paper, we propose a new approach for facing the
color normalization problem in HER2-stained histopathological images of breast
cancer tissue, posed as an style transfer problem. We combine the Color
Deconvolution technique with the Pix2Pix GAN network to present a novel
approach to correct the color variations between different HER2 stain brands.
Our approach focuses on maintaining the HER2 score of the cells in the
transformed images, which is crucial for the HER2 analysis. Results
demonstrate that our final model outperforms the state-of-the-art image style
transfer methods in maintaining the cell classes in the transformed images and
is as effective as them in generating realistic images.

Index Terms—  Domain Adaptation, Style Transfer, GAN, Deep Learning, Breast
Cancer, Digital Pathology, Histopathological Images.

##  1 Introduction

Cancer is a major public health concern worldwide, with more than 10 million
deaths each year. Early detection of cancer is crucial for improving patient
outcomes, and biopsy is the primary method for diagnosis. However, the process
of diagnosing cancer can be time-consuming and subject to variability among
hospitals and pathologists.

The HER2 stain identifies the overproduction of the HER2 protein, a receptor
in breast cells that controls cell growth and division. The HER2 stain
classifies cells into four different classes based on the intensity and
completeness of the stained membrane, and estimates the percentage of cells in
each class to assign a score to the patient. However, HER2 stained images can
vary significantly among different hospitals because of the use of different
stain brands (Figure 1), leading to inconsistencies in diagnosis and treatment
decisions. To address this challenge, we propose a novel approach to normalize
color variations between HER2-stained histopathological images from different
domains. Our approach combines the Color Deconvolution technique and the
Pix2Pix GAN network to perform a style transfer that maintains the membrane
intensity of each cell, a key feature used in cell classification for HER2
stained images.

The ability to compute the translation between different HER2 stain brands is
a crucial step in multicenter projects, as it enables the use of a single,
common model for all hospitals regardless the use of different stain brands.
This eliminates the need for acquiring separate datasets to train individual
models for each strain brand, which can be a time-consuming and resource-
intensive process. Moreover, this translation helps to ensure consistency in
diagnosis and treatment decisions, ultimately improving patient outcomes. Our
method outperforms in maintaining the cell classes, which is crucial in the
context of HER2 stained images. Furthermore, our study demonstrates that the
proposed approach is effective in generating images that are as realistic as
those generated by state-of-the-art models.

Fig. 1: Example of the color variability between HER2 stain brands. HER2 tile
from brand A (right) and brand B (left).

##  2 Related work

###  2.1 Color Deconvolution

The final result of the staining and scanning process are digital images
called Whole Slide Images (WSIs). Colour deconvolution is an image processing
technique that has been widely used in the analysis of WSIs. This technique,
first defined by Ruifrok A. et al. [1], enables the decomposition of an RGB
image into channels representing the optical absorbance and transmittance of
dyes. This is particularly useful in the case of tissue staining, where
different dyes are used to highlight specific structures or features within
the tissue. In the case of HER2 staining, Hematoxylin is used to stain nuclei
blue, and the component called DAB stains the membrane brown.

Colour deconvolution [2] separates the different components of each stain,
creating an image for each individual component. The original RGB image is
orthonormal transformed, which enables the extraction of independent
information about each stain contribution and the correct balancing of the
absorption factor for each separate stain.

The algorithm converts the RGB image into the optical-density (OD) space. Each
pure stain is characterized by a specific optical density for the light in
each of the three RGB channels, which is represented by a $3\times 1$ OD
vector, describing the stain in the OD-converted RGB color space. Being $Y$
the OD space image, $n_{s}$ the number of stains and $n\times m$ the
dimensions of the image, by defining the colour vector matrix
$M\in\mathbf{R}^{3\times n_{s}}$ (composed by the normalized OD vectors) and
the stain concentrations matrix $C\in\mathbf{R}^{n\times m\times n_{s}}$, we
can decompose $Y$ in the following way: $Y^{T}=MC^{T}+N^{T}$. Here, $N$ just
represents random noise. Having a reference matrix $M_{ref}$ for a specific
stain brand, we can compute the Moore-Penrose pseudo-inverse of $M_{ref}$,
$\hat{M}^{-1}$ [3]. Then, matrix $C$ can be estimated from $C=\hat{M}^{-1}Y$
and the color deconvolution of image $Y$ can be performed, decomposing it in
terms of $C$ and $M$.

As explained in Pérez-Bueno F. et al. [3], this image processing technique can
directly be used to perform a domain adaptation between two different images.
Being $M_{ref}$ and $C_{ref}$ the matrices from the reference domain image and
$M$ and $C$ the matrices from the input image, the domain adaptation can be
performed multiplying the reference color vector matrix $M_{ref}$ times the
input image stain concentration matrix $C$. Using directly the Color
Deconvolution technique to perform the domain adaptation is a linear
operation. However, in the case of HER2 stain variability, the transformation
from the stain brand A domain to the stain brand B domain is not a linear
operation. Hence, in this work, we present a non-linear approach based on the
color deconvolution operation that will solve the style transfer problem
between the stain brands A and B in HER2 histopathological images.

###  2.2 Generative Adversarial Networks (GANs)

Generative Adversarial Networks (GANs), first presented by Goodfellow I. et
al. [4], have been widely used for various tasks in computer vision, including
image generation, super-resolution, and style transfer. Pix2Pix GAN, first
presented by Isola P. et al. [5], is a type of GAN architecture that is used
to generate images from a source domain to a target domain. Pix2Pix GANs are
trained on paired images, where the source image is mapped to the target
image, allowing the GAN to learn the mapping between the two domains.

As stated by Jose L. et al. [6], GANs have achieved state-of-the-art results
in many generation tasks, but also in various medical imaging applications
such as medical image style tranfer, which is the aim of this work. In Shaban
M. et al. [7], they present StainGAN, a network inspired by CycleGAN that
corrects the color variations in the stain process of Hematoxylin and Eosin
(HE). Same problem has been adressed with GAN inspired achitectures by Zhou N.
et al. [8] and Cai S. et. al. [9]. In Wollmann T. et al. [10], the authors
used a CycleGAN based architecture to transform the stained images in HE from
different hospitals into data from one center to reduce variability and
improve the classification step. In Ren J. et al. [11], they face the same
problem but with a different strategy: with two different datasets (source and
target domains), they train the target network to extract the domain invariant
features from the source input samples.

Overall, these works demonstrate the effectiveness of Pix2Pix GANs in
correcting color variability among different WSI, by learning a mapping
between the source and target domains. However, there is still a need for
improving the performance and robustness of GAN-based color correction methods
for histopathological images. In particular, as explained in Section 1, in the
case of HER2 stain, the analysis of the stained image depends on the intensity
of the stained membranes of the cells. Thus, it is crucial that the style
transfer method does not alter this characteristic of each cell, in order to
maintain their classes. As shown in Section 5, GAN-based methods generate
realistic images in the desired style but do not maintain the intensity of the
membranes, altering the HER2 classes of the cells.

Hence, our contributions in this work are the following: (i) We present an
approach that brings non-linearity to the Color Deconvolution technique, to
deal with more complex color normalization problems, as the HER2 one. (ii) We
present a GAN-based style transfer approach that, combined with the Color
Deconvolution technique, ensures the maintenance of the membrane intensity of
each cell, in order not to alter the HER2 score of the translated image.

##  3 Dataset

Training deep learning models on WSIs presents a significant challenge due to
their large size. Algorithms developed for WSIs typically use tiles, which are
cropped images of relevant zones with a size of $1024\times 1024$ pixels.

It is also crucial to consider the various classes of cells present in
HER2-stained WSIs. In order to ensure a balanced representation of these
classes, 3-4 WSIs from patients in each domain were selected and classified
according to defined HER2 scores, and tiles were extracted using the QuPath
[12] tool, a software platform for bioimage analysis.

Fig. 2: Graphic representation of the training (red arrows) and inference
(green arrows) procedure.

As a result of the selection and tiling process, two datasets were generated,
one for each domain (stain brand A and stain brand B), each containing 3,000
tiles. These datasets will be used for training and testing of our final
model.

##  4 Our contribution

Our method combines the Color Deconvolution technique and the Pix2Pix GAN
network to perform style transfer in HER2 images. Our main hypothesis is that
by computing a domain-independent representation for the stained images, we
can recover them in the desired style maintaining the intensity of the
membranes to be able to recover the cell classes.

The proposed architecture is an Encoder-Decoder, where the Encoder receives a
HER2 stained tile from a specific hospital (stain brand A) and computes the
domain-independent representation of the image, whereas the Decoder recovers
the original image from this representation, but in the stain brand B style.
Thus, a key challenge is to find an effective method for computing the domain-
independent representation.

For the Encoder, we use the color deconvolution technique to decompose the
images into the color vector matrix $M$ and the concentrations matrix $C$
described in Section 2.1.

In the case of HER2 stain, we have 2 different stain channels, one for the
brown color (DAB) and one for the blue color (Hematoxylin), and a third
channel for residual noise. As domain-independent representation we use the
concentrations matrix $C$. To compute this concentrations matrix, a reference
color matrix $M_{ref}$ is needed. Using the QuPath application, we extracted
the color vectors for each one of our domains. Then, for each image, we
compute the concentrations matrix for each one of the stains and the noise
channel.

In Pérez-Bueno F. et al. [3], they present an approach that optimizes the
reference color vector matrix over the given image, in order to improve the
results of the color deconvolution algorithm. We have applied this
optimization process over the reference color vector matrices $M_{B}$ and
$M_{A}$.

For the Decoder, we use the Pix2Pix GAN network, presented by Isola P. et al.
[5]. Using this conditional GAN architecture, the Decoder is responsible for
recovering the images in the stain brand B style from the representation,
which is going to be the concentration matrices of the image.

To perform the translation, the Pix2Pix GAN network is trained only using
stain brand B images and their concentration matrices. We feed the Encoder
with stain brand B images and, with the computed domain-independent
representations, we train the whole conditional GAN architecture. During this
procedure, the Decoder learns how to recover images in the stain brand B style
just using the information stored in the concentration matrices, which is
nearly composed only by domain-independent features. As mentioned in [5], this
conditional GAN is trained with two different loss functions. First, we train
it using the classical adversarial loss of the GANs architectures but, in this
case, it also includes a $\mathcal{L}_{2}$ loss between the original and the
generated images (Figure 2).

Once the whole Decoder is trained, to actually perform the style transfer, we
just freeze the Generator of the Pix2Pix GAN architecture, and feed the whole
model (Encoder and Decoder) using stain brand A images (Figure 2). Hence,
using the Color Deconvolution technique, we compute the domain-independent
concentration matrices and feed them to the Generator, which recovers the
image in the stain brand B style.

We developed two models, combining the optimization process in the computation
of the reference color vector matrices in the Color Deconvolution technique
[3] (suffix WO / O for no optimization / optimization in the model name).

Fig. 3: Visualization of original tiles (left) and the obtained translated
images for the different models.

##  5 Results

We present the results of our approach to perform the style transfer task
between stain brands A and B in HER2 images. As mentioned before, this
approach aims to generate realistic images while preserving the HER2 score of
the input image. For that reason, we evaluate our results using the Fréchet
Inception Distance (FID) metric [13], to measure the realism of the generated
images, and the Weighted F1-Score as a task metric to ensure that the
translation maintains the HER2 classes of the cells. We assess our model with
these metrics and compare its performance against three baseline models:
CycleGAN, FastCUT, and Color Deconvolution (see Table 1).

FID. We have used the implementation presented by M. Seitzer [14]. In our
experiments, we found that our proposed combination of Color Deconvolution and
Pix2Pix GAN network model had a FID score of 166, which is a good indication
of the high quality of the generated images.

Weighted F1-Score. To evaluate the maintenance of cell classes on the
generated images, we have computed the class of each cell with a U-Net
classification model [15] trained with images of the brand B style. We have
compared the transformed images classification results with the ground truth
classification of the images from stain brand A style.

In terms of the FID metric, our approach outperforms the Color Deconvolution
and achieves nearly as good scores as the state-of-the-art style transfer
method FastCUT and CycleGAN. Regarding the Weighted F1-Score, our both models
outperform the Color Deconvolution baseline as well as the FastCUT and
CycleGAN based techniques.

In summary, the experimental results demonstrate that our proposed GAN-based
model achieves similar performance than the state-of-the-art models for style
transfer while better preserving the structure of cell classes.

|  | $\downarrow$ FID |  $\uparrow$ W - F1
---|---|---|---
Pix2Pix + CD | Pix2Pix WO | 165.08 | 0.6395
| Pix2Pix O | 166.61 | 0.6641
CD | Only CD | 195.79 | 0.3986
| CycleGAN | 160.52 | 0.5616
| FastCUT | 159.83 | 0.4514
Table 1: Results table: All metrics have been computed over a test set of 144
images of stain brand A style

##  6 Conclusions

In this study, we have proposed a novel approach to address the challenge of
color variations between HER2-stained histopathological images from different
domains. Our method combines the Color Deconvolution technique and the Pix2Pix
GAN network to perform a style transfer.

We have shown that this approach is able to generate images as realistic as
those generated by state-of-the-art style transfer models. Moreover, it
outperforms the linear approach of using only Color Deconvolution. This
indicates that the non-linearity introduced by the Pix2Pix network is
effective in solving the style transfer problem in histopathological images.
We have used the Weighted F1-score as a task metric to evaluate the
translation of cell classes during the style transfer application. Results
indicate that our approach is better at maintaining the cell classes than the
state-of-the-art models.

Using the proposed translation between stain brands, multicenter projects can
perform all the inference with a single, common model for hospitals that may
use different HER2 stain brands. Thus, less effort is required in acquiring
the data, making our approach a practical and effective solution for improving
cancer diagnosis and treatment outcomes.

To the best of our knowledge, this is the first approach that addresses the
color normalization problem in HER2-stained histopathological images, where
maintaining the intensity of the membranes and the structure of the image is
key to maintain the HER2 score of the image. Overall, this work contributes to
the development of more accurate and efficient pathological image analysis
tools, which could ultimately improve clinical decision-making and patient
outcomes.

##  7 Acknowledgments

This reseach has been partially funded by the project DigiPatICS, from ICS,
and the project PID2020-116907RB-100 AIMING, funded by MCIN/ AEI /10.13039/
501100011033.

## References

  * [1] Arnout Ruifrok and Dennis Johnston,  “Quantification of histochemical staining by color deconvolution,”  Anal Quant Cytol Histol, vol. 23, 01 2001.
  * [2] Gabriel Landini, Martinelli Giovanni, and Filippo Piccinini,  “Colour deconvolution - stain unmixing in histological imaging,”  Bioinformatics (Oxford, England), vol. 37, 09 2020.
  * [3] Fernando Pérez-Bueno, Miguel Vega, María A. Sales, José Aneiros-Fernández, Valery Naranjo, Rafael Molina, and Aggelos K. Katsaggelos,  “Blind color deconvolution, normalization, and classification of histological images using general super gaussian priors and bayesian inference,”  Computer Methods and Programs in Biomedicine, vol. 211, pp. 106453, 2021.
  * [4] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio,  “Generative adversarial networks,”  In Advances in neural information processing systems, 2014.
  * [5] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei Efros,  “Image-to-image translation with conditional adversarial networks,”  in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 07 2017, pp. 5967–5976.
  * [6] Laya Jose, Sidong Liu, Carlo Russo, Annemarie Nadort, and Antonio Di Ieva,  “Generative adversarial networks in digital pathology and histopathological image processing: A review,”  Journal of Pathology Informatics, vol. 12, pp. 43, 11 2021.
  * [7] M. Shaban, Christoph Baur, Nassir Navab, and Shadi Albarqouni,  “Staingan: Stain style transfer for digital histological images,”  in IEEE 16th International Symposium on Biomedical Imaging (ISBI), 04 2019, pp. 953–956.
  * [8] Niyun Zhou, De Cai, Xiao Han, and Jianhua Yao,  “Enhanced cycle-consistent generative adversarial network for color normalization of h&e stained images,”  in Medical Image Computing and Computer Assisted Intervention – MICCAI 2019: 22nd International Conference, Berlin, Heidelberg, 2019, Springer-Verlag.
  * [9] Shaojin Cai, Yuyang Xue, Qinquan Gao, Min Du, Gang Chen, Hejun Zhang, and Tong Tong,  “Stain style transfer using transitive adversarial networks,”  in Conference: International Workshop on Machine Learning for Medical Image Reconstruction, 10 2019, pp. 163–172.
  * [10] T. Wollmann, C. S. Eijkman, and K. Rohr,  “Adversarial domain adaptation to improve automatic breast cancer grading in lymph nodes,”  in 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 2018, pp. 582–585.
  * [11] Jian Ren, Ilker Hacihaliloglu, Eric A. Singer, David J. Foran, and Xin Qi,  “Unsupervised domain adaptation for classification of histopathology whole-slide images,”  Frontiers in Bioengineering and Biotechnology, vol. 7, 2019.
  * [12] Peter Bankhead, Maurice Loughrey, Jose Fernandez, Yvonne Dombrowski, Darragh Mcart, Philip Dunne, Stephen Mcquaid, Ronan Gray, Liam Murray, Helen Coleman, Jacqueline James, Manuel Salto-Tellez, and Peter Hamilton,  “Qupath: Open source software for digital pathology image analysis,”  Scientific Reports, vol. 7, 12 2017.
  * [13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter,  “Gans trained by a two time-scale update rule converge to a local nash equilibrium,”  in Neural Information Processing Systems (NIPS)At: Long Beach, CaliforniaVolume: 30, 12 2017.
  * [14] Maximilian Seitzer,  “pytorch-fid: FID Score for PyTorch,” https://github.com/mseitzer/pytorch-fid, August 2020,  Version 0.3.0.
  * [15] Mireia Boneta Camí,  “Analysis of her2 receptor proteins in breast cancer histology images using semantic segmentation,” Final Degree Thesis, UPC, Facultat d’Informàtica de Barcelona, Departament de Teoria del Senyal i Comunicacions, Jun 2021,  http://hdl.handle.net/2117/354402.