[1]Young-June Kim

1]Department of Physics, University of Toronto, Toronto, M5S 1A7, Ontario,
Canada

2]National Synchrotron Light Source II, Brookhaven National Laboratory, Upton,
11973, NY, USA

3]Cornell High Energy Synchrotron Source, Cornell University, Ithaca, 14853,
NY, USA

# Re-investigation of Moment Direction in a Kitaev Material $\alpha$-RuCl3

Subin Kim<EMAIL_ADDRESS>Ezekiel J. Horsley<EMAIL_ADDRESS>Christie S. Nelson<EMAIL_ADDRESS>Jacob P. C. Ruff<EMAIL_ADDRESS><EMAIL_ADDRESS>[  [  [

###### Abstract

We report X-ray diffraction and resonant elastic X-ray scattering (REXS)
studies on two $\alpha$-RuCl3 crystals with distinct magnetic transition
temperatures: TN=7.3K and 6.5K. We find that the sample with TN=6.5K exhibits
a high degree of structural twinning at low temperature, whereas the TN=7.3K
sample primarily comprises a single domain of R$\bar{3}$. Notwithstanding,
both samples exhibit an identical zigzag magnetic structure, with magnetic
moments pointing away from the honeycomb plane by $\alpha=31(2)^{\circ}$. We
argue that the identical ordered moment directions in these samples suggest
that the intralayer magnetic Hamiltonian remains mostly unchanged regardless
of TN.

##  1 Introduction

In recent years, $\alpha$-RuCl3 has been extensively investigated as a
potential candidate for a Kitaev quantum spin liquid [1, 2, 3, 4, 5, 6, 7, 8,
9, 10, 11, 12]. The intriguing interplay of frustrating, bond-dependent
interactions arising from spin-orbit coupling offers a promising platform for
having a quantum spin liquid [13]. While the system exhibits magnetic ordering
below T${}_{N}\approx$7K, it can be effectively suppressed by applying a
moderate magnetic field parallel to the honeycomb plane [14, 15, 7, 10, 16].
This field-induced quantum phase has been the subject of many studies [17, 8,
18, 19, 20, 21, 22, 23, 24, 25, 26], including the observation of the half-
quantized thermal Hall effect, which is regarded as a compelling piece of
evidence of a potential quantum spin liquid phase [18]. However, this
experimental observation remains contentious due to limited reproducibility
[27, 28]. The thermal Hall effect displays a significant dependence on the
sample quality and composition, and the quantization was only observed in
select samples [18, 27, 28, 29, 30, 31, 32].

Figure 1: Lattice and Magnetic Structure of $\alpha$-RuCl3. a High-temperature
C2/m structure. Adjacent layers are stacked with a relative shift along
$\vec{a}$. We use the pseudo-orthorhombic coordinate system as described in
the text. b-c Low-temperature R$\bar{3}$ structure of two twin domains.
Adjacent layers are relatively shifted along $\vec{b}$ but in opposite
directions. d-f Zigzag antiferromagnetic structure of $\alpha$-RuCl3 projected
onto different planes. The magnetic moment vector lies within the a-c plane,
canted away from the honeycomb plane by angle $\alpha$ as shown in e. The
magnetic structure is three-layer-periodic perpendicular to the honeycomb
plane which can be explained by nearest neighbour antiferromagnetic coupling
shown in f.

The quality of the crystals varies among samples based on the amount of
defects present in the crystal structure. $\alpha$-RuCl3 features a honeycomb
structure with weak coupling between its layers through van der Waals forces,
making it susceptible to structural defects known as stacking faults (see Fig.
1). This inherent predisposition to stacking sequence disorder results in
diverse structural arrangements among crystals, leading to the proposition of
multiple crystal structures for this material [33, 7, 34, 31, 35]. Careful
studies using high-quality $\alpha$-RuCl3 crystals indicate the crystal
structure symmetry is C2/m at room temperature. However, this structure is
disrupted below the structural phase transition around 150K. The exact
identification of the low-temperature structure has been controversial, but
recent studies converge on $R\bar{3}$ symmetry as the low-temperature
structure (see Fig. 1a) [36, 37, 38, 39].

The magnetic structure of $\alpha$-RuCl3 is sample-dependent as well. Early
neutron diffraction studies on a single crystal by Sears et al. observed
magnetic Bragg peaks with three-layer periodicity [6]. In contrast, neutron
diffraction studies on a powder sample by Johnson et al. observed magnetic
Bragg peaks with two-layer periodicity instead [33]. Subsequently, Banerjee et
al. showed that the three-layer and two-layer periodic magnetic structures
have distinct magnetic transition temperatures, TN=7 K and TN=14 K,
respectively [7]. The magnetic structure characterized by a two-layer
periodicity is associated with samples with a substantial number of stacking
faults. This correlation is emphasized by the presence of prominent diffuse
scattering, notably observed in samples with TN=14 K [4]. This correlation is
reinforced by the difficulty in reconciling a two-layer magnetic periodicity
with the low-temperature R$\bar{3}$ structure [34]. Johnson et al. explained
this two-layer periodicity by invoking the high-temperature C2/m structure
instead [15], suggesting that the C2/m structure persists down to low
temperature in these samples with TN=14 K. On the other hand, the three-layer
magnetic structure with TN=7 K is typically observed for samples without
strong diffuse scattering [4], and it can be explained exclusively using the
R$\bar{3}$ structure (see Fig. 1) [10]. Therefore, it is generally accepted
that crystals with a single magnetic transition around TN=7K are of high
quality.

However, in addition to this large variation in TN, recent reports showed that
a smaller variation in TN is found even among high-quality crystals, with
values ranging from 6.5K to 7.5K [16, 9, 30, 40, 38]. While these studies
suggest that samples with higher TN tend to be of better quality, the precise
origin of this sample-dependent variation in TN remains unresolved [38]. It is
this small variation that is the main subject of this paper. Specifically, we
examine whether the magnetic structure and the ordered moment direction are
different between these samples. We carried out a resonant elastic X-ray
scattering (REXS) study of two samples with different magnetic transition
temperatures of TN=6.5K and TN=7.3K. We confirm the zigzag antiferromagnetic
arrangement with three-layer periodicity in both samples (See Fig. 1d-f). In
particular, a study of the azimuthal angle dependence shows that the out-of-
plane canting angle $\alpha$ remains unchanged between the two samples (see
Fig. 1e). In addition, the observed canting angle is consistent with that
previously observed for a sample with TN=12K and two-layer-periodicity [41].
The fact that no discernible differences in the moment direction were observed
across these samples suggests that the magnetic Hamiltonian remains unchanged
regardless of the sample quality. The only distinction we find between the two
samples we studied pertains to their structural domain populations. The sample
exhibiting TN=7.3K is predominantly comprised of a single domain of the
R$\bar{3}$ structure, while the TN=6.5K sample showed highly twinned
R$\bar{3}$ structures.

##  2 Results

###  2.1 Overview of the Crystals

Minor variations in the transition temperature was confirmed between 5
$\alpha$-RuCl3 crystals, for which the TN values varied from 6.5K to 7.3K (see
Supplementary Information). The specific heat and the magnetic susceptibility
measurements suggest a distinction in the crystal quality between samples with
TN=6.5(1) K and samples with TN=7.3(1) K. Here, we choose one sample from each
group: S1 with TN=7.3K and S3 with TN=6.5K, and investigated their crystal and
magnetic structures.

###  2.2 Structural Characterization

Figure 2: X-ray diffraction results Maps of structural peaks at T=20K in the
reciprocal 0KL plane of a S1 and b S3. In the case of S1, the Bragg peaks are
discernible at (0,K,L-K/3). For S3, additional Bragg peaks are evident at
(0,K,L+K/3) together with (0,K,L+K/3). Each of these Bragg peak types:
(0,K,L-K/3) and (0,K,L+K/3) corresponds to the two possible twin domain
structures shown in Fig. 1b and c, respectively. c-d (0,2,L) line scan of S1
and S3, respectively. Predominant peaks are observable at (0,2,L-2/3) in the
case of S1, while additional peaks manifest at (0,2,L+2/3) for S3. The minor
peak centered at (0,2,L), which displays a broader width compared to other
Bragg peaks, is derived from the high-temperature C2/m structure depicted in
Fig. 1a.

Both samples undergo a first-order structural transition from a monoclinic
C2/m structure above the structural transition temperature,
T${}_{s}\approx$150K, to a rhombohedral R$\bar{3}$ structure below it, as
depicted in Fig. 1. To describe the two structures using the same coordinate
system, we adopt a pseudo-orthorhombic notation as shown in Fig. 1. The
$\vec{a}$ and $\vec{b}$ vectors are identical to the monoclinic structure, and
describe two unique high-symmetry directions within the honeycomb structure.
The $\vec{c}$ vector points perpendicular to the honeycomb plane and describes
the separation vector between adjacent layers. Note that this $\vec{c}$ is not
a lattice vector for either the C2/m or the R$\bar{3}$ structure. In the C2/m
structure, each honeycomb layer is stacked on top of another with a shift by
$-\vec{a}$/3. On the other hand, the layers are shifted by $\vec{b}$/3 in the
R$\bar{3}$ structure. The primary distinction between these two structures
lies in their stacking direction, and this can be differentiated using single
crystal X-ray diffraction.

Fig. 2a-b show X-ray diffraction maps, comparing the reciprocal (0,K,L) plane
diffraction patterns of S1 and S3 at 20K, well below the structural
transition. In the case of S1, Bragg peaks are found at (0,K,L-K/3), where K
is an even integer and L is an integer. However, for S3, additional Bragg
peaks are also visible at (0,K,L+K/3), together with (0,K,L-K/3). Both
(0,K,L-K/3) and (0,K,L+K/3) types of Bragg peaks arise from the R$\bar{3}$
structure but they correspond to different structural twin domains, as shown
in Fig. 1b-c. The translation vectors $\pm$$\vec{b}$/3+$\vec{c}$ for each twin
domain give rise to Bragg peaks at (0,K,L$\mp$K/3), respectively. Therefore,
we can conclude that S1 mainly comprises a single domain, while S3 has a
significant mixture of twin domains. This distinction becomes clearer when
examining L scans along (0,2,L), illustrated in Fig. 2c-d. In S1, intense
peaks are found at (0,2,L-2/3), with much smaller peaks present at
(0,2,L+2/3). However, in the case of S3, significant intensity is present at
both (0,2,L$\pm$2/3), indicative of substantial twinning at a ratio of about
2:1. Additionally, weak peaks at (0,2,L) are present (more pronounced when
compared to S1). These peaks indicate that a small fraction of the sample
remains in the high-temperature C2/m structure. The origin of this is unclear,
but it may be related to an incomplete transition from the high-temperature
phase. Alternatively, the C2/m phase may act as a domain wall between two twin
domains. The widths of these Bragg peaks are much broader than that of the
R$\bar{3}$ counterparts in which the domain size is roughly 20 layers. The
extensive twinning combined with the presence of the high-temperature
structure in S3 results in a prominent diffuse rod of intensity along
$\vec{c}$ (see Fig. 2b) in comparison to S1. This observation agrees with
Zhang et al. who reported that samples with lower TN exhibit larger diffuse
scattering [38].

###  2.3 Magnetic Structure

Figure 3: Resonant elastic X-ray scattering - temperature dependence a
Schematic diagram of the scattering geometry of the REXS experiment. $\psi$
represents azimuthal angle when the sample is rotated with respect to
scattering vector ${\bf Q}={\bf k}_{f}-{\bf k}_{i}$. b Photon energy
dependence of the structural (0,0,1) and magnetic (0,1,1.67) Bragg peaks
intensities. The intensity of the magnetic Bragg peak resonates at E=2.837keV
while the intensity of the structural Bragg peak is suppressed due to
absorption. c-d Rocking curves of magnetic Bragg peak ${\bf
Q}_{m1}=(-0.5,-0.5,1.33)$ and ${\bf Q}_{m1}^{\prime}=(-0.5,-0.5,1.67)$
respectively in S1, at various temperatures. Both magnetic Bragg peaks are
suppressed upon heating, vanishing above the magnetic transition temperature
of TN=7.3 K. e Sample rocking curve of magnetic Bragg peak ${\bf
Q}_{m2}^{\prime}=(0,1,1.67)$ in S3 at varying temperatures. The magnetic Bragg
peak vanished above the magnetic transition temperature at TN=6.5 K. f
Integrated intensity of the rocking curves c-e as a function of temperature.
The lines are fitted to $\sim(1-T/T_{N})^{0.4}$ to extract the transition
temperatures. A clear difference in TN can be seen between the two samples.

The magnetic structures of S1 and S3 were studied using REXS. In both samples,
the magnetic Bragg peaks were observed using the $\sigma$-$\pi^{\prime}$
polarization channel for which the incident beam is polarized perpendicular to
the scattering plane ($\sigma$) and the scattered beam is polarized parallel
to the scattering plane ($\pi$’), as illustrated in Fig. 3a. The magnetic
Bragg peak intensities show resonant enhancement at the L3 edge of Ruthenium
(2.837keV), as shown in Fig. 3b. In the case of S1, magnetic Bragg peaks were
observed at ${\bf Q}_{m1}=(-0.5,-0.5,1.33)$ and ${\bf
Q}_{m1}^{\prime}=(-0.5,-0.5,1.67)$. Here, the primed notation indicates an
equivalent Q-position in the other rhombohedral twin domain (see Fig. 1b and
1c), as discussed below. For S3, magnetic Bragg peaks were also observed at
${\bf Q}_{m2}=(0,1,1.33)$ and ${\bf Q}_{m2}^{\prime}=(0,1,1.67)$, in addition
to ${\bf Q}_{m1}$ and ${\bf Q}_{m1}^{\prime}$. Note that ${\bf Q}_{m1}$ and
${\bf Q}_{m2}$ as well as ${\bf Q}_{m1}^{\prime}$ and ${\bf Q}_{m2}^{\prime}$
are equivalent under the three-fold rotational symmetry of the rhombohedral
R$\bar{3}$ structure (see Supplementary Information).

Fig. 3c-f presents the temperature-dependence of magnetic Bragg peaks.
Specifically, Fig. 3c and d demonstrate the temperature dependence of the
sample rocking curve scans for ${\bf Q}_{m1}$ and ${\bf Q}_{m1}^{\prime}$ in
S1, respectively. The temperature dependence of the rocking curve scans at
${\bf Q}_{m2}^{\prime}$ for S3 are shown in Fig. 3e. In each case, the
magnetic Bragg peak intensity decreases with increasing temperature,
confirming their magnetic origin. The temperature dependence of the integrated
intensity is plotted in Fig. 3f, clearly illustrating the difference in the
magnetic transition temperatures between the two samples. For S1, the magnetic
Bragg peaks vanish around TN=7.3(1) K for both twin domains while in S3, the
magnetic Bragg peak vanishes around TN=6.5(1) K. These transition temperatures
agree well with the specific heat data. (see Supplementary Information) The
magnetic Bragg peak intensity is proportional to the magnetic order parameter
squared. The intensity data in Fig. 3f could be fitted to
$\sim(1-T/T_{N})^{2\beta}$. The critical exponent $\beta$ could not be
determined with high accuracy due to large error bars. However, $\beta\approx
0.2$ describes the data fairly well, which indicates the two-dimensional
nature of the phase transition.

We first focus on S1 to examine its lattice and magnetic structure more
carefully. Fig. 4a compares the scans of structural Bragg peaks ${\bf
Q}_{s1}^{\prime}=(-1,-1,3.33)$ and ${\bf Q}_{s1}=(-1,-1,3.67)$, which are
equivalent to (0,2,3.33) and (0,2,3.67), respectively. As discussed in the
previous section, these peaks arise from different twin domains (corresponding
to Fig. 1c and Fig. 1b, respectively), explaining large discrepancies in shape
and intensity between them. In particular, the intensity differs by two orders
of magnitudes, confirming that S1 comprises a predominantly single domain of
R$\bar{3}$.

In Fig. 4b, we present L scans for the magnetic Bragg peaks ${\bf Q}_{m1}$ and
${\bf Q}^{\prime}_{m1}$, compared with the structural Bragg peak (0,0,1). A
noticeable width difference is observed between ${\bf Q}_{m1}$ and ${\bf
Q}^{\prime}_{m1}$. While the width of the ${\bf Q}^{\prime}_{m1}$ peak is
resolution-limited and equals that of the structural Bragg peak (0,0,1), the
width of the ${\bf Q}_{m1}$ peak is not resolution-limited, indicating a
correlation length of about 250 layers. This difference in correlation lengths
strongly suggests that they originate from different magnetic domains.

Furthermore, we can identify that these two magnetic peaks originate from the
two structural twin domains. Fig. 4c and d show sample rocking curves along
$\theta$ and $\chi$, respectively, for the magnetic ${\bf Q}^{\prime}_{m1}$
peak, compared with the structural ${\bf Q}^{\prime}_{s1}$ peak. Similarly,
Fig. 4e and f display rocking curves along $\theta$ and $\chi$, respectively,
for the magnetic ${\bf Q}_{m1}$ peak, compared with the structural ${\bf
Q}_{s1}$ peak. Each pair of the magnetic and structural peaks agree well in
their shape profiles which strongly suggest that they originate from the same
domains. Since ${\bf Q}_{s1}$ and ${\bf Q}^{\prime}_{s1}$ correspond to
different structural twin domains of $R\bar{3}$, the observed ${\bf Q}_{m1}$
and ${\bf Q}^{\prime}_{m1}$ peaks correspond to magnetic domains originating
from different structural domains. See Supplementary Information for these
peak positions. Note that each pair follows the same selection rule
(0,K,L$\mp$K/3), respectively. The magnetic structure that best accounts for
this observation is the three-layer periodic zigzag structure shown in Fig. 1.
For S3, unravelling the magnetic structure proves challenging due to the
presence of extensive twinning in both the structure and magnetic structure.
Nonetheless, the intensity can still be well-explained using the magnetic
structure provided in Fig. 1.

It is interesting to note that during the initial cool-down of S1, only the
magnetic Bragg peak at ${\bf Q}^{\prime}_{m1}$ was observed. However,
subsequent cool-downs introduced an additional strong Bragg peak at ${\bf
Q}_{m1}$ (see Supplementary Information). This suggests the possibility of a
change in the magnetic domain configuration after thermal cycling. However,
the situation seems to be much more complicated than a simple history
dependence if one considers the relatively weak intensity of the corresponding
structural peak at ${\bf Q}_{s1}$. Note that the structural peaks are observed
using photons with the 3rd harmonic of the primary X-ray beam energy
(E=8.511keV), while the magnetic peaks are probed using photons with the
primary energy (E=2.837keV). The penetration depths for these beams are 30 µm
and 1 µm, respectively. We speculate that the thermal cycling might affect the
near-surface region of the sample disproportionally, giving rise to a
proliferation of twin domain walls, and the corresponding change in the
magnetic Bragg peak intensity.

Figure 4: Sample rocking curve scans comparison between structural and
magnetic Bragg peaks a Rocking curve comparison between two structural Bragg
peak ${\bf Q}_{s1}$=(-1,-1,3.67) and ${\bf Q}^{\prime}_{s1}$=(-1,-1,3.33) for
S1. The intensity is multiplied 100 times for ${\bf Q}_{s1}$. b L scan
comparison between magnetic Bragg peaks ${\bf Q}_{m1}$ and ${\bf
Q}^{\prime}_{m1}$ with structural Bragg peak (0,0,1). To facilitate this
comparison, the scan is plotted against relative change in L, $\Delta$L. The
width of magnetic Bragg peak ${\bf Q}^{\prime}_{m1}$ is consistent with
structural (0,0,1) while the width of ${\bf Q}_{m1}$ is observed to be much
broader. c-d rocking curve scans along $\theta$ and $\chi$ respectively,
comparing magnetic ${\bf Q}^{\prime}_{m1}$ with structural ${\bf
Q}^{\prime}_{s1}$. e-f Rocking curves along $\theta$ and $\chi$ respectively,
comparing magnetic ${\bf Q}_{m1}$ with structural ${\bf Q}_{s1}$. Figure 5:
Azimuthal dependence of magnetic intensitya-b Azimuthal dependence of
integrated intensity of magnetic Bragg peaks of S1 and S3 respectively. In
case of S1, azimuthal dependence of ${\bf Q}_{m1}$ is shown while in S3,
azimuthal dependence of ${\bf Q}^{\prime}_{m2}$ is shown instead. The lines
are simulated azimuthal dependence for various canted moment angles, $\alpha$.
The observed azimuthal dependence agrees with $\alpha$=31∘ c-d Azimuthal
dependence close to zero intensity for S1 and S3. The vanishing angle
$\psi_{c}$ agrees well with $\alpha$=31∘ for both S1 and S3.

###  2.4 Magnetic Moment Direction

In the $\sigma$-$\pi$’ polarization configuration, the magnetic scattering
intensity is proportional to the square of the projection of the magnetic
moment, ${\bf M}$, onto the scattered photon wavevector, ${\bf k}_{f}$: $|{\bf
M}\cdot{\bf k}_{f}|^{2}$. By rotating the sample by the azimuthal angle $\psi$
(See Fig. 3a) with the scattering vector ${\bf Q}={\bf k}_{f}-{\bf k}_{i}$
fixed, the projection $|{\bf M}\cdot{\bf k}_{f}|^{2}$ undergoes modulation as
a function of $\psi$. By fitting the $\psi$-dependence of the REXS intensity,
the out-of-plane canting angle $\alpha$ (see the inset in Fig. 5a) can be
precisely determined.

To obtain the magnetic scattering intensity at each azimuthal angle, we
integrated both rocking-curve scans of $\theta$ and $\chi$. One of the
difficulties we encountered during our azimuthal dependence study was the
mismatch in the beam footprint and the sample shape as $\psi$ varied. To
eliminate the magnetic scattering intensity modulation attributed to this beam
footprint variation, the magnetic scattering intensity was further normalized
with respect to the (0,0,1) structural Bragg peak intensity at each $\psi$.
The details of the normalization process are provided in the Supplementary
Information.

Fig. 5a-b show the azimuthal dependence of the magnetic scattering intensities
at ${\bf Q}_{m1}$ for S1 and ${\bf Q}^{\prime}_{m2}$ for S3, respectively.
Both intensities show local maxima around $\psi$=0∘ and $\psi$=180∘, which
suggests that magnetic moments are confined within the ac plane, perpendicular
to the zigzag propagation direction, consistent with the previously proposed
structure shown in Fig. 1 [10]. The global maximum at $\psi$=180∘ implies that
the moment is canted in the positive $\alpha$ direction [41].

The value of canting angle $\alpha$ can be determined by fitting the intensity
data. Our best fit result yield $\alpha$=31(2)∘ for S1. However, for S3,
determining the azimuthal angle was more challenging due to the weaker signal
and the difficulty in normalization as described in Supplementary Information.
In order to determine the canted angle in this case, another method was used
to support the fitting result. Fig. 5c and d provide a zoomed-in view of
$\psi$-range where the magnetic intensity vanishes. The azimuthal angle for
zero intensity, $\psi_{c}$, corresponds to the condition that ${\bf k}_{f}$ is
perpendicular to ${\bf M}$ and is very sensitive to the canted angle $\alpha$.
For instance, if $\alpha$ changes from 25∘ to 35∘, $\psi_{c}$ varies from 40∘
to 60∘ for (0,1,1.67), as illustrated in Fig. 5d. The $\psi_{c}$ could be
determined with high precision without the need to normalize intensity between
measurements. Both S1 and S3 data give $\alpha=31(2)^{\circ}$ which is
consistent with the values obtained using the direct fitting method. Note that
S1 and S3 data were obtained for different ${\bf Q}$-vectors, which explains
the difference in $\psi_{c}$. For S1, the reference angle 0∘ is along the
(1,-3,0) direction. For S3, the reference angle 0∘ is along the (-1,0,0)
direction (see Supplementary Information).

##  3 Discussion and Conclusions

We want to point out that the observed canted moment angle of
$\alpha=31(2)^{\circ}$ in the current study closely mirrors the result found
by Sears et. al.[41]. This is important since the crystal studied in Ref. [41]
has a magnetic transition temperature TN=12 K, substantially different from
the samples studied here. In addition, the magnetic stacking structure of that
sample is also different. In Ref. [41], the magnetic Bragg peak was found at
(0,1,3/2), indicative of a magnetic structure characterized by a two-layer
periodicity, which is a clear departure from the three-layer periodicity found
in our study and several other recent neutron scattering studies [6, 7, 8,
10]. The observed robustness of the moment direction $\alpha$, independent of
the sample details, is the main result of our study. However, this raises a
question as to the origin of the difference in TN.

Variations in interlayer couplings could presumably explain the observed
sample dependence of TN. The variations can naturally arise from stacking
sequence differences. However, the interlayer interactions have been mostly
ignored because the magnetic ordering can be well-explained using only the
intralayer interactions, and the interlayer interactions are orders of
magnitude weaker than their intralayer counterparts [42, 43, 44, 45]. It is
interesting to note that stacking structure changes seem to have a dramatic
effect on the magnetism of another honeycomb lattice material CrI3 [46, 47,
48, 49]. This material also goes through a structural transition from high-
temperature C2/m structure to low-temperature R$\bar{3}$ [50]. For this
material, each layer orders ferromagnetically with the moment direction
pointing perpendicular to the honeycomb planes. However, depending on how the
layers are stacked, the system can order antiferromagnetically or
ferromagnetically between layers [47, 48, 49]. It was found that
antiferromagnetic order is preferred in the C2/m structure while ferromagnetic
order is preferred in the R$\bar{3}$ structure [47]. A recent muon-spin-
rotation investigation reported that the magnetic ordering temperature is
sensitive to the volume fraction of monoclinic and rhombohedral phases in
crystals with coexisting C2/m and R$\bar{3}$ phases [51]. The coexistence of
C2/m and R$\bar{3}$ is also frequently observed in $\alpha$-RuCl3, especially
in thin crystals [4], and a similar investigation could be useful to elucidate
the origin of the TN variation.

However, the $T_{N}$ variation in $\alpha$-RuCl3 may have a complicated
origin, due to the complexity of the in-plane physics in this material. The
nearest neighbour J-K-$\Gamma$ model is widely used to describe the magnetism
in $\alpha$-RuCl3:

$H=\sum_{<i,j>_{\gamma}}JS_{i}S_{j}+KS_{i}^{\gamma}S_{j}^{\gamma}+\Gamma(S_{i}^{\alpha}S_{j}^{\beta}+S_{i}^{\alpha}S_{j}^{\beta}),$
(1)

where $\gamma$ describes the three distinct bonds between nearest neighbour
spins $S_{i}$ and $S_{j}$, with mutually exclusive
$\alpha,\beta,\gamma\in\\{x,y,z\\}$. The model consists of an isotropic
Heisenberg term (J), a bond-dependent anisotropic Kitaev term (K), and a
symmetric off-diagonal ($\Gamma$) term. The presence of relatively large K and
$\Gamma$ was found in many works [42, 43, 44, 52, 53, 54, 41, 55, 56, 57, 58,
59, 60, 61, 62, 63, 59, 64, 65]. The current result again corroborates these
earlier studies and confirms the presence of a large $\Gamma$ term. One
consequence of large $\Gamma$ is the large magnetic anisotropy between in-
plane and out-of-plane susceptibilities [54, 1, 41]. However, this anisotropy
ratio $M_{ab}/M_{c}$ varies from 5 to 8 depending on the sample, strongly
suggesting that sample dependence might be present for $\Gamma$ [66, 14, 67,
68, 54, 69]. While the moment canting angle $\alpha$ is sensitive to the ratio
$|\Gamma/K|$, this sensitivity is diminished in the limit of a large $\Gamma$
interaction. According to the classical model [44], a wide range of
$|\Gamma/K|$ between 1.0 and 1.6 agree within the observed range of
$\alpha=31(2)^{\circ}$. Therefore, the variation of $|\Gamma/K|$ in this range
would be still compatible with our observation of robustness of $\alpha$.

To summarize, we investigated the sample dependence of the magnetic structure
of two $\alpha$-RuCl3 samples S1 and S3, characterized by TN=7.3 K and TN=6.5
K, respectively. Two samples are distinguished clearly in their structures by
the domain population. While S1 consists primarily of a single domain of the
R$\bar{3}$ structure, S3 has a significant mixing between two twin domains of
R$\bar{3}$. However, the magnetic structure of these two samples is consistent
with previously determined magnetic structure with three-layer-periodicity
[10]. Despite the clear difference in $T_{N}$ and domain distribution, we find
the canting angle of the ordered moment, $\alpha$, remains unchanged in both
samples.

##  4 Methods

All batches of $\alpha$-RuCl3 crystals were grown using the chemical vapor
transport method described in previous studies [4]. S1, S2, and S3 were from
the same batch while S4 and S5 were grown in separate batches. Two samples,
designated as sample 1 (S1) and sample 3 (S3), were selected for single
crystal X-ray diffraction measurements performed at the QM2 beamline at
Cornell High Energy Synchrotron Source (CHESS). A photon energy of 20keV was
employed for studying the samples in transmission geometry. During the data
collection, the samples underwent a 360∘ rotation in 0.1∘ steps, and a Pilatus
6M area detector was used to capture the intensity. The temperature was
controlled by a helium cryostream onto the $\alpha$-RuCl3 crystals, enabling
reciprocal maps to be obtained at T=200K and 20K, above and below the
structural transition, respectively.

The same two samples (S1 and S3) were also subjected to resonant elastic X-ray
scattering study at the 4-ID beamline at National Synchrotron Light Source II
(NSLS II) at Brookhaven National Lab. The incident photon energy was tuned to
the ruthenium L3 edge (2.837keV). A graphite analyzer (002) was utilized for
analyzing the scattered photons, allowing those with polarization parallel to
the scattering plane ($\pi$ polarization) to be selected. The samples were
cooled using a closed-cycle cryostat, enabling cooling down to T=5K, below the
magnetic transition temperature.

Data Availability The data that support the findings of this study are
available from the corresponding author upon reasonable request.

Code Availability The custom codes for analyzing the data and implementing the
calculation in this study are available from the corresponding author upon
reasonable request.

Acknowledgments We thank Jiefu Cen and Hae Young Kee for insightful
discussions. Work at the University of Toronto was supported by the Natural
Science and Engineering Research Council (NSERC) of Canada, Canadian
Foundation for Innovation, and Ontario Research Fund. Resonant elastic x-ray
scattering experiment was conducted at 4-ID beamline of the National
Synchrotron Light Source II, a U.S. Department of Energy (DOE) Office of
Science User Facility operated for the DOE Office of Science by Brookhaven
National Laboratory under Contract No. DE-SC0012704. Single crystal X-ray
diffraction work is based on research conducted at the Center for High-Energy
X-ray Sciences (CHEXS), which is supported by the National Science Foundation
(BIO, ENG and MPS Directorates) under award DMR-1829070.

Supplementary Information The supplementary material is available at …

Author Contributions The samples were grown by S.K. and prepared together with
E.H. for the experiment. The resonant elastic experiments were carried out by
S.K. and E.H. with the help of C.N. and the X-ray diffraction experiments were
carried out by S.K. and E.H. with the help of J.R. The data was analyzed by
S.K.. The idea was conceived by Y.K. and the work was supervised by Y.K. All
authors contributed to the writing and preparation of the manuscript.

Competing Interests The authors declare no competing interests.

## References

  *   * [1] Winter, S. M. _et al._ Breakdown of magnons in a strongly spin-orbital coupled magnet.  _Nature Communications_ 8, 1152 (2017).
  * [2] Takagi, H., Takayama, T., Jackeli, G., Khaliullin, G. & Nagler, S. E.  Concept and realization of Kitaev quantum spin liquids.  _Nat. Rev. Phys._ 1, 264–280 (2019).
  * [3] Motome, Y., Sano, R., Jang, S., Sugita, Y. & Kato, Y.  Materials design of Kitaev spin liquids beyond the Jackeli–Khaliullin mechanism.  _Journal of Physics: Condensed Matter_ 32, 404001 (2020).
  * [4] Kim, S., Yuan, B. & Kim, Y.-J.  $\alpha$-RuCl3 and other Kitaev materials.  _APL Materials_ 10 (2022).  080903\.
  * [5] Plumb, K. W. _et al._ $\alpha$-RuCl3: A spin-orbit assisted Mott insulator on a honeycomb lattice.  _Phys. Rev. B_ 90, 041112 (2014).
  * [6] Sears, J. A. _et al._ Magnetic order in $\alpha$-RuCl3: A honeycomb-lattice quantum magnet with strong spin-orbit coupling.  _Phys. Rev. B_ 91, 144420 (2015).
  * [7] Banerjee, A. _et al._ Proximate Kitaev quantum spin liquid behaviour in a honeycomb magnet.  _Nat. Mater._ 15, 733–740 (2016).  Article.
  * [8] Banerjee, A. _et al._ Excitations in the field-induced quantum spin liquid state of ${\alpha}$-RuCl3.  _npj Quantum Materials_ 3, 8 (2018).
  * [9] Do, S.-H. _et al._ Majorana fermions in the Kitaev quantum spin system ${\alpha}$-RuCl3.  _Nature Physics_ 13, 1079–1084 (2017).
  * [10] Balz, C. _et al._ Field-induced intermediate ordered phase and anisotropic interlayer interactions in ${\alpha}$-RuCl3.  _Phys. Rev. B_ 103, 174417 (2021).
  * [11] Sandilands, L. J., Tian, Y., Plumb, K. W., Kim, Y.-J. & Burch, K. S.  Scattering continuum and possible fractionalized excitations in $\alpha$-RuCl3.  _Phys. Rev. Lett._ 114, 147201 (2015).
  * [12] Zheng, J. _et al._ Gapless spin excitations in the field-induced quantum spin liquid phase of ${\alpha}$-RuCl3.  _Phys. Rev. Lett._ 119, 227208 (2017).
  * [13] Jackeli, G. & Khaliullin, G.  Mott insulators in the strong spin-orbit coupling limit: From Heisenberg to a quantum compass and Kitaev models.  _Phys. Rev. Lett._ 102, 017205 (2009).
  * [14] Sears, J. A. _et al._ Magnetic order in ${\alpha}$-RuCl3: A honeycomb-lattice quantum magnet with strong spin-orbit coupling.  _Phys. Rev. B_ 91, 144420 (2015).
  * [15] Johnson, R. D. _et al._ Monoclinic crystal structure of ${\alpha}$-RuCl3 and the zigzag antiferromagnetic ground state.  _Phys. Rev. B_ 92, 235119 (2015).
  * [16] Sears, J. A., Zhao, Y., Xu, Z., Lynn, J. W. & Kim, Y.-J.  Phase diagram of ${\alpha}$-RuCl3 in an in-plane magnetic field.  _Phys. Rev. B_ 95, 180411 (2017).
  * [17] Baek, S.-H. _et al._ Evidence for a field-induced quantum spin liquid in ${\alpha}$-RuCl3.  _Phys. Rev. Lett._ 119, 037201 (2017).
  * [18] Kasahara, Y. _et al._ Majorana quantization and half-integer thermal quantum Hall effect in a Kitaev spin liquid.  _Nature_ 559, 227–231 (2018).
  * [19] Ponomaryov, A. N. _et al._ Nature of magnetic excitations in the high-field phase of ${\alpha}$-RuCl3.  _Phys. Rev. Lett._ 125, 037202 (2020).
  * [20] Leahy, I. A. _et al._ Anomalous thermal conductivity and magnetic torque response in the honeycomb magnet $\alpha$-RuCl3.  _Phys. Rev. Lett._ 118, 187203 (2017).
  * [21] Modic, K. A. _et al._ Scale-invariant magnetic anisotropy in ${\alpha}$-RuCl3 at high magnetic fields.  _Nature Physics_ 17, 240–244 (2021).
  * [22] Kasahara, Y. _et al._ Unusual thermal Hall effect in a Kitaev spin liquid candidate ${\alpha}$-RuCl3.  _Phys. Rev. Lett._ 120, 217205 (2018).
  * [23] Yokoi, T. _et al._ Half-integer quantized anomalous thermal Hall effect in the Kitaev material candidate ${\alpha}$-RuCl3.  _Science_ 373, 568–572 (2021).
  * [24] Yamashita, M., Gouchi, J., Uwatoko, Y., Kurita, N. & Tanaka, H.  Sample dependence of half-integer quantized thermal Hall effect in the Kitaev spin-liquid candidate $\alpha$-RuCl3.  _Phys. Rev. B_ 102, 220404 (2020).
  * [25] Li, H., Li, W. & Su, G.  High-field quantum spin liquid transitions and angle-field phase diagram of the Kitaev magnet ${\alpha}$-RuCl3.  _Phys. Rev. B_ 107, 115124 (2023).
  * [26] Gass, S. _et al._ Field-induced transitions in the Kitaev material ${\alpha}$-RuCl3 probed by thermal expansion and magnetostriction.  _Phys. Rev. B_ 101, 245158 (2020).
  * [27] Czajka, P. _et al._ Oscillations of the thermal conductivity in the spin-liquid state of $\alpha$-RuCl3.  _Nat. Phys._ 17, 915–919 (2021).
  * [28] Lefrançois, E. _et al._ Evidence of a phonon Hall effect in the Kitaev spin liquid candidate ${\alpha}$-RuCl3.  _Phys. Rev. X_ 12, 021025 (2022).
  * [29] Tanaka, O. _et al._ Thermodynamic evidence for a field-angle-dependent Majorana gap in a Kitaev spin liquid.  _Nat. Phys._ 18, 429–435 (2022).
  * [30] Kasahara, Y. _et al._ Quantized and unquantized thermal Hall conductance of Kitaev spin-liquid candidate $\alpha$-RuCl3 (2022).
  * [31] Bruin, J. A. N. _et al._ Robustness of the thermal Hall effect close to half-quantization in $\alpha$-RuCl3.  _Nat. Phys._ 18, 401–405 (2022).
  * [32] Yamashita, M., Gouchi, J., Uwatoko, Y., Kurita, N. & Tanaka, H.  Sample dependence of half-integer quantized thermal Hall effect in the Kitaev spin-liquid candidate ${\alpha}$-RuCl3.  _Phys. Rev. B_ 102, 220404 (2020).
  * [33] Johnson, R. D. _et al._ Monoclinic crystal structure of $\alpha$-RuCl3 and the zigzag antiferromagnetic ground state.  _Phys. Rev. B_ 92, 235119 (2015).
  * [34] Cao, H. B. _et al._ Low-temperature crystal and magnetic structure of $\alpha$-RuCl3.  _Phys. Rev. B_ 93, 134423 (2016).
  * [35] Stahl, Q. _et al._ Pressure-tuning of ${\alpha}$-RuCl3 towards the ideal Kitaev-limit (2022).
  * [36] Park, S.-Y. _et al._ Emergence of the Isotropic Kitaev Honeycomb Lattice with Two-dimensional Ising Universality in $\alpha$-RuCl3.  _ArXiv e-prints_ (2016).
  * [37] Mu, S. _et al._ Role of the third dimension in searching for majorana fermions in ${\alpha}$-RuCl3 via phonons.  _Phys. Rev. Res._ 4, 013067 (2022).
  * [38] Zhang, H. _et al._ Stacking disorder and thermal transport properties of $\alpha$-RuCl3.  _ArXiv e-prints_ (2023).
  * [39] Kim, S., Horsley, E., Ruff, J. P. C., Moreno, B. D. & Kim, Y.-J.  Structural transition and magnetic anisotropy in $\alpha$-RuCl3 (2023).
  * [40] Bruin, J. A. N. _et al._ Origin of oscillatory structures in the magnetothermal conductivity of the putative Kitaev magnet ${\alpha}$-RuCl3.  _APL Materials_ 10, 090703 (2022).
  * [41] Sears, J. A. _et al._ Ferromagnetic Kitaev interaction and the origin of large magnetic anisotropy in ${\alpha}$-RuCl3.  _Nature Physics_ 16, 837–840 (2020).
  * [42] Rau, J. G., Lee, E. K.-H. & Kee, H.-Y.  Generic spin model for the honeycomb iridates beyond the Kitaev limit.  _Phys. Rev. Lett._ 112, 077204 (2014).
  * [43] Winter, S. M., Li, Y., Jeschke, H. O. & Valentí, R.  Challenges in design of Kitaev materials: Magnetic interactions from competing energy scales.  _Phys. Rev. B_ 93, 214431 (2016).
  * [44] Chaloupka, J. & Khaliullin, G.  Magnetic anisotropy in the Kitaev model systems ${\mathrm{na}}_{2}{\mathrm{iro}}_{3}$ and ${\mathrm{rucl}}_{3}$.  _Phys. Rev. B_ 94, 064435 (2016).
  * [45] Maksimov, P. A. & Chernyshev, A. L.  Rethinking ${\alpha}$-RuCl3.  _Phys. Rev. Res._ 2, 033011 (2020).
  * [46] Huang, B. _et al._ Layer-dependent ferromagnetism in a van der Waals crystal down to the monolayer limit.  _Nature_ 546, 270–273 (2017).
  * [47] Sivadas, N., Okamoto, S., Xu, X., Fennie, C. J. & Xiao, D.  Stacking-dependent magnetism in bilayer CrI3.  _Nano Letters_ 18, 7658–7664 (2018).
  * [48] Jiang, P. _et al._ Stacking tunable interlayer magnetism in bilayer ${\mathrm{cri}}_{3}$.  _Phys. Rev. B_ 99, 144401 (2019).
  * [49] Kong, X., Yoon, H., Han, M. J. & Liang, L.  Switching interlayer magnetic order in bilayer CrI3 by stacking reversal.  _Nanoscale_ 13, 16172–16181 (2021).
  * [50] McGuire, M. A., Dixit, H., Cooper, V. R. & Sales, B. C.  Coupling of crystal structure and magnetism in the layered, ferromagnetic insulator CrI3.  _Chemistry of Materials_ 27, 612–620 (2015).
  * [51] Meseguer-Sánchez, J. _et al._ Coexistence of structural and magnetic phases in van der waals magnet CrI3.  _Nature Communications_ 12, 6265 (2021).
  * [52] Chaloupka, J. c. v. & Khaliullin, G.  Hidden symmetries of the extended Kitaev-Heisenberg model: Implications for the honeycomb-lattice iridates ${A}_{2}{\mathrm{iro}}_{3}$.  _Phys. Rev. B_ 92, 024413 (2015).
  * [53] Rusnačko, J., Gotfryd, D. & Chaloupka, J. c. v.  Kitaev-like honeycomb magnets: Global phase behavior and emergent effective models.  _Phys. Rev. B_ 99, 064425 (2019).
  * [54] Lampen-Kelley, P. _et al._ Anisotropic susceptibilities in the honeycomb Kitaev system ${\alpha}$-RuCl3.  _Phys. Rev. B_ 98, 100403 (2018).
  * [55] Suzuki, H. _et al._ Proximate ferromagnetic state in the Kitaev model material ${\alpha}$-RuCl3.  _Nature Communications_ 12, 4512 (2021).
  * [56] Little, A. _et al._ Antiferromagnetic resonance and terahertz continuum in ${\alpha}$-RuCl3.  _Phys. Rev. Lett._ 119, 227201 (2017).
  * [57] Ozel, I. O. _et al._ Magnetic field-dependent low-energy magnon dynamics in ${\alpha}$-RuCl3.  _Phys. Rev. B_ 100, 085108 (2019).
  * [58] Wu, L. _et al._ Field evolution of magnons in ${\alpha}$-RuCl3 by high-resolution polarized terahertz spectroscopy.  _Phys. Rev. B_ 98, 094425 (2018).
  * [59] Eichstaedt, C. _et al._ Deriving models for the Kitaev spin-liquid candidate material ${\alpha}$-RuCl3 from first principles.  _Phys. Rev. B_ 100, 075110 (2019).
  * [60] Kim, H.-S. & Kee, H.-Y.  Crystal structure and magnetism in ${\alpha}$-RuCl3: An ab initio study.  _Phys. Rev. B_ 93, 155143 (2016).
  * [61] Yadav, R. _et al._ Kitaev exchange and field-induced quantum spin-liquid states in honeycomb ${\alpha}$-RuCl3.  _Scientific Reports_ 6, 37925 (2016).
  * [62] Hou, Y. S., Xiang, H. J. & Gong, X. G.  Unveiling magnetic interactions of ruthenium trichloride via constraining direction of orbital moments: Potential routes to realize a quantum spin liquid.  _Phys. Rev. B_ 96, 054410 (2017).
  * [63] Wang, W., Dong, Z.-Y., Yu, S.-L. & Li, J.-X.  Theoretical investigation of magnetic dynamics in ${\alpha}$-RuCl3.  _Phys. Rev. B_ 96, 115103 (2017).
  * [64] Suzuki, T. & Suga, S.-i.  Effective model with strong Kitaev interactions for ${\alpha}$-RuCl3.  _Phys. Rev. B_ 97, 134424 (2018).
  * [65] Cookmeyer, T. & Moore, J. E.  Spin-wave analysis of the low-temperature thermal Hall effect in the candidate Kitaev spin liquid ${\alpha}$-RuCl3.  _Phys. Rev. B_ 98, 060412 (2018).
  * [66] Kubota, Y., Tanaka, H., Ono, T., Narumi, Y. & Kindo, K.  Successive magnetic phase transitions in ${\alpha}$-RuCl3: Xy-like frustrated magnet on the honeycomb lattice.  _Phys. Rev. B_ 91, 094422 (2015).
  * [67] Weber, D. _et al._ Magnetic Properties of Restacked 2D Spin 1/2 honeycomb ${\alpha}$-RuCl3 Nanosheets.  _Nano Lett._ 16, 3578–3584 (2016).
  * [68] Banerjee, A. _et al._ Neutron scattering in the proximate quantum spin liquid $\alpha$-rucl3.  _Science_ 356, 1055–1059 (2017).
  * [69] Breitner, F. A., Jesche, A., Tsurkan, V. & Gegenwart, P.  Thermal decomposition of the Kitaev material ${\alpha}$-RuCl3 and its influence on low-temperature behavior.  _Phys. Rev. B_ 108, 045103 (2023).


# Preservational Learning Improves Self-supervised Medical Image Models by
Reconstructing Diverse Contexts

Hong-Yu Zhou1 Chixiang Lu1* Sibei Yang2 Xiaoguang Han4 Yizhou Yu*1 First two
authors contributed equally.  3]
1The University of Hong Kong 2ShanghaiTech University 3Deepwise AI Lab
4Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong
(Shenzhen)
{whuzhouhongyu<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Corresponding author.

###### Abstract

Preserving maximal information is one of principles of designing self-
supervised learning methodologies. To reach this goal, contrastive learning
adopts an implicit way which is contrasting image pairs. However, we believe
it is not fully optimal to simply use the contrastive estimation for
preservation. Moreover, it is necessary and complemental to introduce an
explicit solution to preserve more information. From this perspective, we
introduce Preservational Learning to reconstruct diverse image contexts in
order to preserve more information in learned representations. Together with
the contrastive loss, we present Preservational Contrastive Representation
Learning (PCRL) for learning self-supervised medical representations. PCRL
provides very competitive results under the pretraining-finetuning protocol,
outperforming both self-supervised and supervised counterparts in 5
classification/segmentation tasks substantially. Codes are available at
https://bit.ly/3rJydb1.

##  1 Introduction

It is common practice that training deep neural networks often requires a
large amount of manually labeled data. This requirement is easy to satisfy in
natural images as both the cost of labor and the difficulty of labeling can be
acceptable. However, in medical image analysis, reliable medical annotations
usually come from domain experts’ diagnoses which are hard to access
considering the scarcity of target disease, the protection of patient’s
privacy and the limited medical resources. To address these problems, self-
supervised learning has been widely adopted as a practical way to learn
medical image representations without manual annotations.

Figure 1: Conceptual illustration of proposed method. GO stands for global
operations which convert feature maps to feature vectors. The blue feature
vector comes from the momentum encoder. $vec(\mathcal{T})$ represents the
indicator vector of $\mathcal{T}$ which contains a set of transformation
functions. Each component in $vec(\mathcal{T})$ is 1 or 0 denoting whether the
corresponding transformation is applied or not. $\odot$ stands for a channel-
wise multiplication operation.

Nowadays, contrastive representation learning has been widely applied and
outstandingly successful in medical image analysis [51, 33, 6]. The goal of
contrastive learning is to learn invariant representations via contrasting
medical image pairs, which can be regarded as an implicit way to preserve
maximal information. Nonetheless, we think it is still beneficial and
complemental to explicitly preserve more information in addition to the
contrastive loss. To achieve this goal, an intuitive solution is to
reconstruct the original inputs using learned representations so that these
representations can preserve the information closely related to the inputs.
However, we discover that directly adding a plain reconstruction branch for
restoring the original inputs would not significantly improve the learned
representations. To address this problem, we introduce Preservational
Contrastive Representation Learning to reconstruct diverse contexts using
representations learned from the contrastive loss.

As shown in Fig.1, we attempt to incorporate the diverse image reconstruction,
as a pretext task, into contrastive learning. The main motivation is to encode
more information into the learned representations. Specifically, we introduce
Transformation-conditioned Attention and Cross-model Mixup to enrich the
information carried by representations. The first module embeds a
transformation indicator vector ($vec(\mathcal{T})$ in Figure 1) to high-level
feature maps following an attentional mechanism. Based on the embedded vector,
the network is required to dynamically reconstruct different image targets
while the input is fixed. Cross-model Mixup is developed to generate a hybrid
encoder by mixing the feature maps of the ordinary and the momentum encoders,
where the hybrid encoder is asked to reconstruct mixed image targets. We show
that both modules can help to encode more information and produce stronger
representations compared to using contrastive learning only.

Besides the learning algorithm, this paper also addresses another issue when
using unlabeled medical images for pretraining, that is lacking a fair and
thorough comparison of different self-supervised learning methodologies. In
this paper, we design extensive experiments to analyze the performance of
different algorithms across different datasets and data modalities. Generally
speaking, the contributions of this paper can be summarized into three
aspects:

  * •

Preservational Contrastive Representation Learning is introduced to encode
more information into the representations learned from the contrastive loss by
reconstructing diverse contexts.

  * •

In order to restore diverse images, we propose two modules: Transformation-
conditioned Attention and Cross-model Mixup to build a triple encoder, single
decoder architecture for self-supervised learning.

  * •

Extensive experiments and analyses show that the proposed PCRL has observable
advantages in 5 classification/segmentation tasks, outperforming both self-
supervised and supervised counterparts by substantial and significant margins.

##  2 Related Work

In this section, we mainly review deep model based self-supervised learning
approaches and mixup strategies. Note that for self-supervised learning, we
only list the most related ones based on pretext tasks, ignoring clustering
based approaches [4, 47] and video based representation learning [38, 39, 28,
40].

Pretext-based self-supervised learning in natural images. Pretext-based
methods rely on predicting input images’ properties that are covariant to the
transformations, such as recognizing image patches’ content [29], relative
position [14, 24], rotation degree [17, 14], object color [23, 46], the number
of objects [25] and the applied transformation function [30]. Contrastive-
estimation based approaches also utilize pretext tasks to learn invariant
representations by contrasting image pairs [43, 26, 10, 5, 20, 48]. Recently,
there are some works trying to remove the negative pairs in contrastive
learning [18, 12]. By comparison, our method follows a different principle
which is making representations able to fully describe their sources (_i.e_.,
corresponding input images).

Figure 2: Overview of proposed framework. PCRL employs a U-Net like
architecture to learn representations. For both encoder and decoder, we plot
their feature maps for better demonstration. The hybrid encoder takes no input
images as it consists of mixed feature maps from both the ordinary encoder and
the momentum encoder. {C., F., R., I., O., B.} are abbreviations for random
crop, random flip, random rotation, inpainting, outpainting and gaussian blur,
respectively. NCE is short for noise-contrastive estimation. GO represents
global operations which include global average pooling and fully-connected
layers. $vec(\cdot)$ represents the indicator vector.
$\mathcal{T}_{\\{o,m,h\\}}(\cdot)$ denote a set of transformation functions
for different encoders. $\odot$ represents channel-wise multiplication. For
simplicity, we do not plot the skip connections.

Self-supervised learning in medical image analysis. Before contrastive
learning, solving the jigsaw problem [54, 53, 35] and reconstructing corrupted
images [9, 52] are two major topics for pretext-based approaches in medical
images. Besides them, Xie _et al_. [44] introduced a triplet loss for self-
supervised learning in nuclei images. Haghighi _et al_. [19] improved [52] by
appending a classification branch to classify the high-level features into
different anatomical patterns. For contrastive learning, Zhou _et al_. [51]
applied contrastive loss to 2D radiographs. Similar ideas have also appeared
in few-shot [49] and semi-supervised learning [50]. Taleb _et al_. [34]
proposed 3D Contrastive Predictive Coding from utilizing 3D medical images.
There are two works [16, 8] most related to ours. Feng _et al_. [16] showed
that the process of reconstructing part images displays similar effects with
those of employing a contrastive loss. Chakraborty _et al_. [8] introduced a
denoising autoencoder to capture a latent space representation. However, both
methods failed to improve contrastive learning with context reconstruction
while our methodology succeeds in this aspect.

Mixup in medical imaging. Mixup [45], as an augmentation strategy, has been
widely adopted in medical imaging [27, 7, 22, 15, 2, 37]. The proposed Cross-
model Mixup is most related to Manifold mixup [36, 22, 2]. However, as far as
we know, there is no previous method applying manifold mixup to cross-model
representations, which is exactly the core contribution of our Cross-model
Mixup.

##  3 Methodology

An overview of Preservational Contrastive Representation Learning (PCRL) is
provided in Figure 2. Generally, PCRL contains three different encoders and
one shared decoder. The encoder and the decoder are connected via a U-Net like
architecture. We first apply exponential moving average to the parameters of
the _ordinary encoder_ to produce the _momentum encoder_. Then, for each
input, we apply Cross-model mixup to both encoders’ representations (feature
maps) to build a _hybrid encoder_. Given a batch of images $\mathcal{X}$, we
first apply random crop, random flip and random rotation to generate three
batches of images $\mathcal{X}_{o}^{1}$, $\mathcal{X}_{m}^{1}$ and
$\mathcal{X}_{h}^{1}$ for three different encoders, respectively. Then, we
apply low-level processing operations, including inpainting, outpainting and
gaussian blur, to each batch in order to generate the final inputs
$\mathcal{X}_{\\{o,m,h\\}}^{2}$ for different encoders. In each training step,
we randomly generate three sets of transformations (including flip and
rotation): $\mathcal{T}_{o}$, $\mathcal{T}_{m}$ and $\mathcal{T}_{h}$ (please
refer to Sec.3.1 for more details), and encode them into the last
convolutional layer of each encoder. The ground truth targets of the MSE (mean
square error) loss in image reconstruction are
$\mathcal{T}_{o}(\mathcal{X}_{o}^{1})$, $\mathcal{T}_{m}(\mathcal{X}_{m}^{1})$
and $\mathcal{T}_{h}(\mathcal{X}_{h}^{1})$, corresponding to different
encoders. For contrastive learning in PCRL, we introduce noise-contrastive
estimation which stores past representations in a queue [20] and then apply
contrastive loss to both positive and negative image pairs.

###  3.1 Transformation-conditioned Attention

Figure 3: Our Transformation-conditioned Attention module. F. and R. stand for
flip and rotation, respectively. {$x,y,z$} denote the axes.
{0,$None$,$None$,$None$} denote the rotation degree. $vec(\mathcal{T})$
denotes the indicator vector of $\mathcal{T}$ whose subscript is omitted for
simplicity. $\otimes$ means the outer product. $\odot$ represents the channel-
wise multiplication. Note that the above figure demonstrates the
implementation when each input is 3D. For 2D inputs, there is no F.($z$) in
the indicator vector. For both 2D and 3D inputs, the rotation is only applied
to the xy-plane.

In this section, we propose Transformation-conditioned Attention (TransAtt) to
enable the reconstruction of diverse contexts. This module encodes the
transformation vector into the high-level representations following an
attentional mechanism. Such process can force the encoder to preserve more
information in learned representations.

As shown in Figure 3, for each input, the indicator vector contains a
combination of different transformations. Specifically, given 3D inputs (CT
and MRI scans), the indicator vector has 7 components denoting different
transformation strategies (cf. Figure 3). For 2D inputs (such as X-rays), the
number of transformations decreases to 6 where F.($z$) does not exist. Each
component contains an indicator function (1 or 0) representing whether the
specific transformation is applied or not. To encode the indicator vector into
high-level feature maps, we propose an attentional mechanism where we suppose
that different channels of feature maps may have different impacts on the
reconstructed results. Note that TransAtt is only applied to the last
convolutional layer (before FC layers) of each encoder.

To imitate such process, we first forward the indicator vector
$vec(\mathcal{T})$ to two fully-connected (FC) layers which produces a vector
$f^{p}\in\mathbb{R}^{C\times 1}$. Meanwhile, we apply global average pooling
to each encoder’s high-level feature maps
$\mathcal{F}^{l}\in\mathbb{R}^{C\times D\times H\times W}$111Here we omit the
subscript $\\{o,m,h\\}$ which means $\mathcal{F}^{l}$ can represent feature
maps from different encoders. resulting in a vector
$f^{l}\in\mathbb{R}^{C\times 1}$, where $l$ denotes the layer index. Then, we
compute the outer product of $f^{p}$ and $f^{l}$:

$\displaystyle\begin{split}M=f^{p}\otimes f^{l},\end{split}$ (1)

$M\in\mathbb{R}^{C\times C}$. Next, we flatten $M$ and forward it to another
fully-connected layer:

$\displaystyle\begin{split}f^{q}=\text{ReLU}\ (W_{\theta}\
\text{flat.}(M)),\end{split}$ (2)

where $W_{\theta}\in\mathbb{R}^{C\times C^{2}}$ stands for the weight
parameters of the FC layer. To perform rescaling, we further append a sigmoid
function to $f^{q}$:

$\displaystyle\begin{split}f^{w}=\text{sigmoid}(f^{q}),\end{split}$ (3)

where $f^{w}\in\mathbb{R}^{C\times 1\times 1\times 1}$. Finally, we apply
channel-wise multiplication between $\mathcal{F}^{l}$ and $f^{w}$ and append a
convolutional layer whose kernel size is 3:

$\displaystyle\begin{split}\mathcal{F}^{l+1}=\text{conv}(\mathcal{F}^{l}\odot
f^{w}),\end{split}$ (4)

where $\mathcal{F}^{l+1}\in\mathbb{R}^{C\times D\times H\times W}$.

###  3.2 Cross-model Mixup

Apart from TransAtt, we introduce Cross-model Mixup (CrossMix) for shuffling
these feature representations in order to enable more diverse restoration.
Different from traditional mixup [45] which applies to network inputs, we
propose to mix the feature maps from two different models to build a new
hybrid encoder.

Accordingly, the reconstruction target of the hybrid encoder is a mixed input
$\mathcal{X}^{1}_{h}$. In practice, for each training iteration,

$\displaystyle\begin{split}\mathcal{X}^{1}_{h}=\lambda\mathcal{X}^{1}_{o}+(1-\lambda)\mathcal{X}^{1}_{m},\end{split}$
(5)

where $\lambda\sim\text{Beta}(\alpha,\alpha)$, $\alpha$ is a
hyperparameter222Beta distribution is employed in the original mixup paper..
For network feature maps, we use $\mathcal{F}^{i}_{o}$ to denote the feature
maps at layer $i$ of the ordinary encoder, $i\in\\{1,...,l\\}$. Similarly,
$\mathcal{F}^{i}_{m}$ and $\mathcal{F}^{i}_{h}$ stand for the features maps at
the same location in the momentum encoder and the hybrid encoder,
respectively. Thus, the process of cross-model representation mixup can be
formulated as:

$\displaystyle\begin{split}\mathcal{F}^{i}_{h}=\lambda\mathcal{F}^{i}_{o}+(1-\lambda)\mathcal{F}^{i}_{m}.\end{split}$
(6)

Together with the one shared decoder, we can directly use
$\mathcal{F}_{h}^{\\{1,...,l\\}}$ to reconstruct
$\mathcal{T}_{h}(\mathcal{X}_{h}^{1})$.

###  3.3 Loss Functions and Model Update

To store past features for contrasting, we employ a queue
$K=\\{k_{1},...,k_{N}\\}$ to store them following [20]. The length of $K$ is
$N$. In contrastive learning, we treat all features in queue $K$ as negative
samples. Here we use $g_{o}(\cdot)$ and $g_{m}(\cdot)$ to denote the
projectors of the ordinary encoder and the momentum encoder, separately. The
contrastive loss $\mathcal{L}_{c}$ can be formulated as:

$\displaystyle\begin{split}\mathcal{L}_{c}=-\text{log}\frac{\exp([g_{o}(\mathcal{F}^{l+1}_{o})]^{T}g_{m}(\mathcal{F}^{l+1}_{m})/\tau)}{\sum_{j=1}^{N}\exp([g_{o}(\mathcal{F}^{l+1}_{o})]^{T}k_{j}/\tau)},\end{split}$
(7)

where $\tau$ is a temperature hyperparameter. $g_{o}(\cdot)$ and
$g_{m}(\cdot)$ contains global average pooling and two FC layers,
independently. After each training iteration, we push
$g_{o}(\mathcal{F}^{l+1}_{o})$, $g_{m}(\mathcal{F}^{l+1}_{m})$, and
$g_{h}(\mathcal{F}^{l+1}_{h})$ to $K$ as negative samples for further
contrasting.

For reconstructing diverse contexts, we use mean square error (MSE) as the
default reconstruction loss. Formally, if we denote the _shared_ decoder
network as $D_{\theta}$, considering the whole network has a U-Net like
architecture, the decoder’s inputs should be multi-layer feature maps
$\mathcal{F}_{\\{o,m,h\\}}$333We omit the superscript which is {1,…,$l$+1}..
The computation of the reconstruction loss can be summarized as follows:

$\displaystyle\small\begin{split}\mathcal{L}_{p}&=\text{MSE}(D_{\theta}(\mathcal{F}_{o}),\
\mathcal{T}_{o}(\mathcal{X}_{o}^{1}))+\text{MSE}(D_{\theta}(\mathcal{F}_{m}),\
\mathcal{T}_{m}(\mathcal{X}_{m}^{1}))\\\ &\ \ \ \ \
+\text{MSE}(D_{\theta}(\mathcal{F}_{h}),\
\mathcal{T}_{h}(\mathcal{X}_{h}^{1})).\end{split}$ (8)

We finally sum up $\mathcal{L}_{c}$ and $\mathcal{L}_{p}$ as the complete loss
function with equal weight (0.5 to 0.5). For network parameters, we denote the
parameters of the ordinary encoder and the momentum encoder as $\theta_{o}$
and $\theta_{m}$, respectively. We update $\theta_{m}$ by using an exponential
moving average (EMA) factor $\beta$:

$\displaystyle\begin{split}\theta_{m}=\beta\theta_{m}+(1-\beta)\theta_{o}.\end{split}$
(9)

Note that the hybrid encoder has no encoder parameters as it directly takes a
combination of the feature maps from the ordinary and the momentum encoders.
The mixed feature maps are then treated as the inputs to the shared decoder as
shown in Equation 8.

##  4 Experiments

In this section, we first make ablation studies to demonstrate the advantages
of TransAtt and CrossMix. Then, we introduce a thorough analysis of different
self-supervised algorithms from different aspects. For all tasks, we employ
the notation of _source dataset_ $\rightarrow$_target dataset_. The source
dataset is used for self-supervised pretraining while the target dataset is
used for supervised finetuning.

###  4.1 Baselines

For medical pretraining approaches, we divide them into two categories: 2D and
3D, simply based on their input dimension (e.g., X-ray is 2D while CT scan is
3D). For 2D image pretraining, our baselines include train from scratch (TS),
ImageNet pretraining (IN), Model Genesis (MG) [52], Semantic Genesis (SG) [19]
and Comparing to Learn (C2L) [51]. Here we ignore the method proposed in [44]
which made prior assumptions on the number of nuclei and is not be suitable
for other datasets. For 3D volume pretraining, we also include train from
scratch (TS), Model Genesis (MG), Semantic Genesis (SG) and 3D-CPC [34].
Additionally, Cube++ [35] is included as it is an improved version of Rubik’s
Cube [54] and Rubik’s Cube+ [53].

###  4.2 Datasets

In 2D tasks, we make experiments on two X-ray datasets: Chest14 [41] and
CheXpert [21]. We use Chest14 for both 2D pretraining and 2D finetuning while
CheXpert is only used for pretraining considering CheXpert contains a number
of uncertain labels. The evaluation metric in 2D tasks is AUC. In order to
evaluate the performance of algorithms on 3D volumes, we make experiments on
CT and MRI datasets, including LUNA [32], BraTS [1] and LiTS [3]. We use LUNA
for both 3D pretraining and 3D finetuning. The evaluation metric of finetuning
on LUNA is AUC. BraTS is only used for supervised finetuning to test the
cross-modal transferability following [52]. LiTS is mainly used for 3D
finetuning on liver segmentation. The evaluation metric for segmentation is
mean dice score. In practice, we divide each dataset into the training set,
the validation set and the test set. The pretraining data always come from the
training set (without labels). Please refer to the supplementary material for
more details.

###  4.3 Implementation Details

We use 2D U-Net [31] and 3D U-Net [13] as the backbone networks for 2D and 3D
tasks, where we replace the encoder in 2D U-Net with ResNet-18. The EMA factor
$\beta$ of updating momentum encoder is set to 0.99. For self-supervised
pretraining, we employ momentum SGD as the default optimizer whose initial
learning rate is set to 1e-3 while the momentum value is set to 0.9. We employ
the cosine annealing strategy for decreasing learning rate and stop the
training when the validation loss does not change for 30 epochs. The
checkpoints with lowest validation loss values are saved for finetuning. For
supervised finetuning, we use Adam as the optimizer with 1e-4 as the initial
learning rate. Similar to pretraining, we rely on validation loss to determine
when to end the training stage, and we save the checkpoints with lowest
validation loss values for testing. Dice loss is used for segmentation tasks
while cross entropy is employed for classification tasks. For other
hyperparameters in baselines, we simply follow the choices in their official
papers. $\alpha$ is set to 1 (for $\lambda$) in both Equation 5 and 6. We set
the temperature factor $\tau$ of softmax function in Equation 7 to 0.2 in
practice. For each experiment, we repeat it for three times and report their
average results. More details can be found in attached supplementary material.

###  4.4 Ablation Study

Method | Pretext Task
---|---
|  Rotation [17] |  Position [14]
ContraLoss | 85.5 | 82.3
ContraLoss + Self-Recons. | 87.9 | 84.5
ContraLoss + TransAtt (Flip) | 89.2 | 86.4
ContraLoss + TransAtt | 91.0 | 89.3
ContraLoss + CrossMix | 90.5 | 88.3
PCRL (All modules) | 93.2 | 91.6
Table 1: Investigation of whether our method contains more information. ContraLoss stands for the contrastive loss. Self-Recons. represents self-reconstruction which is reconstructing the input images without any variations. Acc. stands for the classification accuracy. TransAtt (Flip) means that the indicator function in TransAtt only contains the flip operation. Method | Chest14$\rightarrow$Chest14
---|---
9:1 | 8:2
ContraLoss | 71.7 | 74.8
ContraLoss + Self-Recons. | 72.5 | 75.4
ContraLoss + RotNet | 72.3 | 75.2
ContraLoss + TransAtt (Flip) | 73.5 | 76.6
ContraLoss + TransAtt | 74.4 | 77.4
ContraLoss + CrossMix ($\alpha$=0.5) | 73.3 | 76.1
ContraLoss + CrossMix ($\alpha$=1) | 73.7 | 76.6
PCRL (All modules) | 76.2 | 78.8
Table 2: Investigation of different module combinations. In
Chest14$\rightarrow$Chest14, 9:1 demonstrates that we use 90% data for self-
supervised pretraining while the rest 10% are used for finetuning. RotNet
represents that we replace Self-Recons. with the task of rotation prediction.

Method |  Chest14$\rightarrow$Chest14 |  CheXpert$\rightarrow$Chest14
---|---|---
9.5:0.5 | 9:1 | 8:2 | 7:3 | 6:4 | 10% | 20% | 30% | 40% | 50% | 60% | 100%
TS | 61.8 | 68.1 | 71.5 | 73.4 | 75.4 | 68.1 | 71.5 | 73.4 | 75.4 | 77.5 | 79.1 | 80.9
IN | 70.5 | 73.6 | 75.3 | 76.9 | 78.0 | 73.5 | 76.3 | 78.4 | 79.0 | 79.5 | 79.7 | 81.0
MG | 66.4 | 70.0 | 73.9 | 76.1 | 77.3 | 70.1 | 73.9 | 75.5 | 76.5 | 77.6 | 79.3 | 80.8
SG | 66.5 | 70.2 | 74.3 | 76.7 | 77.6 | 69.7 | 73.8 | 75.6 | 77.3 | 77.3 | 79.6 | 81.3
C2L | 71.7 | 74.1 | 76.4 | 77.5 | 79.0 | 73.1 | 77.0 | 78.5 | 79.1 | 79.8 | 80.2 | 81.5
PCRL | 74.1 | 76.2 | 78.8 | 79.0 | 79.9 | 75.8 | 77.6 | 79.8 | 80.8 | 81.2 | 81.7 | 83.1
p-value | 5.2e-4 | 9.6e-4 | 2e-3 | 1.8e-3 | 2.3e-3 | 2.4e-3 | 8.1e-4 | 2.4e-3 | 3.5e-4 | 5.6e-4 | 3.6e-3 | 2.7e-3
(a) 2D tasks: pretraining using Chest14 or CheXpert

Method |  LUNA$\rightarrow$LUNA |  LUNA$\rightarrow$LiTS |  LUNA$\rightarrow$BraTS
---|---|---|---
9:1 | 8:2 | 7:3 | 6:4 | 10% | 20% | 30% | 40% | 100% | 10% | 20% | 30% | 40% | 100%
TS | 78.4 | 83.0 | 85.7 | 87.5 | 71.1 | 77.2 | 84.1 | 87.3 | 90.7 | 66.6 | 72.7 | 76.7 | 77.1 | 81.5
MG | 80.2 | 85.0 | 87.5 | 90.3 | 73.3 | 79.5 | 84.3 | 87.9 | 91.3 | 69.6 | 75.5 | 79.6 | 80.4 | 82.4
Cube++ | 81.4 | 85.2 | 87.9 | 90.0 | 74.2 | 79.3 | 84.5 | 88.2 | 91.8 | 69.0 | 74.9 | 79.3 | 79.7 | 82.2
SG | 79.3 | 84.5 | 87.9 | 90.5 | 73.8 | 79.3 | 85.5 | 88.2 | 91.4 | 70.3 | 75.6 | 79.1 | 80.8 | 82.3
3D-CPC | 80.2 | 85.2 | 88.3 | 90.6 | 74.8 | 80.2 | 85.6 | 88.9 | 91.9 | 70.1 | 75.9 | 79.4 | 81.2 | 82.9
PCRL | 84.4 | 87.5 | 89.8 | 92.2 | 77.3 | 83.5 | 87.8 | 90.1 | 93.7 | 71.6 | 77.6 | 81.1 | 83.3 | 85.0
p-value | 7.5e-4 | 1.5e-3 | 2.1e-3 | 1.9e-3 | 2e-3 | 1.7e-3 | 9e-4 | 2.5e-3 | 2.4e-4 | 8.4e-4 | 3.5e-3 | 5.7e-3 | 2.5e-3 | 2.4e-3
(b) 3D tasks: pretraining using LUNA

Figure 4: Comparison of different methods. In (a), we report the results of 2D
tasks. In (b), the results of 3D tasks are displayed. The ratios in
Chest14$\rightarrow$Chest14 and LUNA$\rightarrow$LUNA stand for the amount of
unlabeled data (for pretraining) with respect to the amount of labeled data
(for finetuning). In other tasks, the ratios represent the amount of data of
the source dataset used for pretraining. For experiments of LiTS, we report
the dice score of liver segmentation. For BraTS, we compute the mean dice of
whole tumor, tumor core and enhancing tumor. We also report the p-values
between the best and the second best results for each ratio to demonstrate the
significance of PCRL. Figure 5: Visual analysis of segmentation results when
finetuning on LiTS and BraTS. For each dataset, we provide 2 cases where we
report the dice scores using different self-supervised pretraining
methodologies. Specifically, in LiTS, the goal is to segment liver. In BraTS,
we only display the results of WT. We ignore MG because SG is built on top of
MG.

In this section, we mainly investigate two problems: 1) whether the proposed
method preserves more information than contrastive learning (Table 1) and 2)
if the preserved information lead to the improved performance (Table 2). In
Table 2, we make experiments on Chest14 to investigate the effectiveness of
different module combinations, where we treat different ratios of the dataset
as labeled data for supervised finetuning while the rest are used as unlabeled
data for self-supervised pretraining.

Preservational learning brings more information to representations. In Table
1, we show that reconstructing diverse contexts does bring more information in
learned representations. We introduce two pretext tasks: predicting the
rotation degree [17] and the relative position between images patches [14] to
evaluate the amount of information in representations. In practice, we fixed
the pretrained models as feature extractors and finetune the last fully
connected layer for pretext tasks. Note that we directly utilize the
pretrained models in CheXpert and use Chest14 to conduct pretext tasks.
Specifically, when predicting the relative position between two image patches,
we first divide the original input image to 14$\times$14 patches. Then, we
extract adjacent image patches and formalize the position prediction problem
as a 8-class classification problem (top, top left, top right, left, right,
bottom left, bottom, bottom right). Similarly, when predicting the rotation
degree, we manually rotate each input image by a specific degree and train the
network to predict this degree, which can also be converted to a
classification problem following [17]. We display the classification results
in Table 1. It is obvious that ContraLoss + Self-Recons. can already perform
better than using ContraLoss only by preserving more information obtained from
simply reconstructing the original input images. More importantly, the
proposed TransAtt module outperforms Self-Recons. by only employing the flip
operation. Together with the rotation transformations, TransAtt greatly
surpasses Self-Recons. on both pretext tasks, which demonstrates that TransAtt
is able to preserve much more information than reconstructing the original
input images. Similar phenomena can also be observed when applying CrossMix.
Finally, PCRL achieves much higher accuracy than the others, again verifying
reconstructing diverse contexts do help preserve more information in learned
features.

Preserved information lead to better performance. We report the performance of
different module combinations in Table 2, where we can observe similar trends
as those in Table 1. It is obvious that the results on pretext tasks are
closely correlated with the performance on Chest14. In other words, given a
method, we can rely on its performance on two pretext tasks to roughly predict
its performance in Chest14. Considering the performance on two pretext tasks
can reflect the amount of information in learned representations, we can
easily draw a conclusion: _reconstructing diverse contexts introduce more
information which help improve the overall performance of algorithms._

From Table 2, we can easily find that adding a self-reconstruction branch only
brings marginal improvements over the baseline model. Similar phenomena can
also be observed when we replace Self-Recons. with RotNet [17]. These results
show that the contrastive loss already captures the information about simple
pretext tasks without directly implementing these tasks. The fact that Self-
Recons. performs better than RotNet shows that image reconstruction can
preserve more information than RotNet. As for TransAtt, by comparing TransAtt
(Flip) with TransAtt, we find that adding the rotation transformations can
obviously improve the overall performance. This is consistent with the results
in Table 1, where TransAtt also performs better than TransAtt (Flip) on two
pretext tasks. We also investigate the influence of the hyperparameter
$\alpha$ in CrossMix. The observation is that by decreasing its value by half,
the overall performance slightly drops. Equipped with TransAtt and CrossMix,
PCRL can surpass the baseline model ContraLoss by approximate 4 points in
different labeled ratios. Moreover, we find that the improvement is most
significant at 10%. This phenomenon implies that the reconstructing diverse
contexts is more useful when the amount of labeled data is small.

###  4.5 Comparison with State-of-the-Arts in 2D Tasks

In this part, we evaluate the performance of various self-supervised
pretraining approaches on 2 different 2D tasks: Chest14$\rightarrow$Chest14
and CheXpert$\rightarrow$Chest14. All results are displayed in Table 4a.

If we look at the results in Chest14$\rightarrow$Chest14, it is obvious that
all pretraining methods (including IN) can boost the performance apparently
when compared to TS. We can see that MG and SG achieve similar performance in
different ratios. Such comparison is easy to explain as SG is built upon MG.
However, both MG and SG still cannot surpass IN especially when the amount of
labeled data is limited, which demonstrates that being pretrained on a large-
scale natural image dataset can benefit medical image analysis a lot. As for
C2L, we find that C2L is the only baseline method which is able to surpass IN
in different ratios. When we compare PCRL with other baseline algorithms, it
is easy to find that PCRL has the ability to outperform different baselines in
various ratios significantly. Particularly, PCRL seems to have more advantages
in small labeled ratios. The underlying reason may be that TransAtt and
CrossMix may help to learn more diversified representations and alleviate the
overfitting problem of training deep neural networks with limited supervision.

In CheXpert$\rightarrow$Chest14, we can see that MG and SG achieve comparable
results with TS when the labeled ratio is equal or greater than 50%,
demonstrating purely pretext-based approaches may have unstable performance
under varying labeled ratios. If we look at C2L, we can find that C2L
consistently outperforms IN and other pretraining methods in almost all
ratios. Somewhat surprisingly, we find that PCRL can still outperform C2L and
IN by a significant margin even if the labeled ratio is 100%. Such comparison
further demonstrates the robustness of PCRL.

Method | #. Epoch | Cityscapes | COCO
---|---|---|---
SimCLR [9] | 1000 | 75.6 | 39.6
SwAV [4] | 400 | 76.0 | -
MoCov2[11] | 800 | 76.3 | 40.5
PCRL | 800 | 77.3 | 41.3
Table 3: Results in natural images. We transfer the self-supervised pretrained
models on ImageNet-1k to downstream tasks, including segmentation (Cityscapes)
and detection (COCO). On Cityscapes, we use ResNet-50 as backbone to build a
FCN segmentation model where the evaluation metric is mIoU. On COCO, we use
the ResNet-50-FPN model from Detectron2 [42] and the evaluation metric is mAP
(0.5:0.05:0.95).

###  4.6 Comparison with State-of-the-Arts in 3D Tasks

Besides 2D tasks, we also analyze the results of 3D self-supervised learning
approaches in 3 different 3D tasks: LUNA$\rightarrow$LUNA,
LUNA$\rightarrow$LiTS and LUNA$\rightarrow$BraTS, where all experimental
results are shown in Table 4b.

In LUNA$\rightarrow$LUNA, it is interesting to find that the performance gaps
between TS and self-supervised pretraining are smaller than those in Chest14.
One explanation is that the nodule classification task is less sensitive to
the amount of labeled data. Among MG, SG, Cube++ and 3D-CPC, 3D-CPC gives the
best results in large labeled ratios while Cube++ performs better in small
ones. Interestingly, as the labeled ratio increases, SG quickly catches up
with MG and Cube++, showing its ability to utilize a large number of labeled
images. Again, we can see that PCRL is able to outperform other baselines
significantly in different ratios. Particularly, when the baseline approaches
show similar results as the labeled ratio becomes larger, PCRL can still
display impressive improvements over previous self-supervised pretraining
approaches and outperforms TS substantially. In LUNA$\rightarrow$LiTS, Cube++
performs slightly better than MG and SG while 3D-CPC outperforms Cube++ in
allmost all ratios. By comparison, PCRL has apparent advantages over other
baselines especially when the labeled ratio is smaller or equal to 50%.

When we transfer knowledge from LUNA to BraTS, MG, SG and Cube++ display
similar performance, all surpassing TS significantly in different labeled
ratios. Due to advantages of contrastive learning, 3D-CPC again outperforms
other baselines. Meanwhile, PCRL once again surpasses previous baselines
consistently and remarkably. We think that such significant improvements can
be attributed to the incorporation of the reconstruction of diverse contexts.

###  4.7 Visual Analysis

In Figure 5, we provide comparative visual analysis results of segmentation
tasks in LiTS and BraTS, where the samples are _randomly_ selected. We can
obviously observe that PCRL handles the details much better than those of
other baselines. For instance, in the first example of LiTS, PCRL delineates
the corners accurately. In the second example of BraTS, PCRL can detect the
isolated tumor regions while other methods cannot well handle these difficult
cases.

###  4.8 Comparison with State-of-the-Arts in Natural Image Segmentation and
Detection Tasks

To investigate the performance of PCRL in natural images, we conduct
pretraining tasks on ImageNet-1k and transfer the pretrained models to
downstream segmentation and detection tasks. The results are displayed in
Table 3. We can see that PCRL is able to outperform MoCov2 and other popular
self-supervised learning methods substantially in both Cityscapes and COCO,
which are two widely adopted datasets in segmentation and detection. The
superior performance on Cityscapes and COCO again verify the advantages of
incorporating diverse context reconstruction.

##  5 Discussion and Conclusion

We show that by reconstructing diverse contexts, the learned representations
using the contrastive loss can be greatly improved in medical image analysis.
Our approach has shown positive results of self-supervised learning in a
variety of medical tasks and datasets. There are some questions worth further
discussing and verifying. For example, is preserving more information the only
reason leading to the improvements over the contrastive loss? We hope the
proposed PCRL can lay the foundations for real-world medical imaging tasks.

##  6 Acknowledgements

This work and the related project were funded in part by National Key Research
and Development Program of China (No.2019YFC0118101), National Natural Science
Foundation of China (No.81971616), Zhejiang Province Key Research &
Development Program (No.2020C03073), National Natural Science Foundation of
China (No.61931024) and the Guangdong Provincial Key Laboratory of Big Data
Computing, The Chinese University of Hong Kong, Shenzhen.

##  7 Thorough Implementation Details (Supplementary Material)

In each dataset, we build the training set, the validation set and the test
set using 70%, 10% and 20% of the whole dataset, respectively. Particularly,
if a dataset is used for self-supervised pretraining and supervised finetuning
in a semi-supervised manner, we construct the training set of pretraining by
extracting a specific amount of data from the complete training set while the
rest of the training set are used for finetuning. More details will be
introduced in the following.

  * •

Chest14 [41] is comprised of 112,120 X-ray images with 14 diseases from 30,805
unique patients. Each input image is resized to 224$\times$224 after random
crop. In our experiments, Chest14 is used for both 2D pretraining and 2D
finetuning. The batch size of pretraining is 256 and the batch size of
finetuning is 512, where we use batch normalization for all experiments. The
evaluation metric of finetuning is AUC.

  * •

CheXpert [21] is a large public dataset for chest radiograph interpretation,
consisting of 224,316 chest radiographs of 65,240 patients. Each input image
is resized to 224$\times$224 after random crop. Considering CheXpert contains
uncertain labels, we mainly use CheXpert for 2D pretraining. The batch size of
pretraining is 256 and we use batch normalization for all experiments.

  * •

LUNA [32] contains 888 CT scans with lung nodule annotations. Note that LUNA
has different preprocessing strategies for different approaches which will be
clarified in the following. We use LUNA for both 3D pretraining and 3D
finetuning. The evaluation metric of finetuning is AUC. The range of the
Hounsfield Units (HU) is [-1000,1000]. The batch size of pretraining and
finetuning is 32 and we use batch normalization in all experiments.

  * •

BraTS [1] includes 351 MRI scans of brain tumor (in 2018 challenge). BraTS
shares similar preprocessing strategies with those of LUNA. We use BraTS for
3D finetuning. The evaluation metric of finetuning is dice score. Since the
batch size is 4 for all finetuning experiments, we use group normalization to
replace batch normalization in the pretrained models.

  * •

LiTS [3] contains 131 available CT scans with their annotations for liver and
tumor. LiTS is mainly used for 3D finetuning where we first localize the liver
and expand the target volume by 30 slices in each axis. The segmentation
target is liver only. The input size is 256$\times$256$\times$64 after random
crop. Dice coefficient is employed as the evaluation metric while the range of
HU is [-200,200]. The batch size is 4. Similar to BraTS, we also use group
normalization to replace batch normalization in the pretrained models.

For network architectures, in Chest14$\rightarrow$Chest14 and
CheXpert$\rightarrow$Chest14, we simply follow [51, 33] to conduct experiments
based on ResNet-18. In 3D tasks, we directly use the ordinary 3D U-Net as
backbone. We will demonstrate the implementation details of each approach in
the following.

  * •

MG [52]. In LUNA and BraTS datasets, we apply random crop to produce each
input tensor whose size is 64$\times$64$\times$32 (corresponding to x, y and z
axes). The augmentation strategies on all datasets include patch shuffling,
non-linear transformation, inpainting, outpainting and random flip.

  * •

SG [19]. Similar to MG, we apply random crop to generate
64$\times$64$\times$32 patches in LUNA and BraTS datasets. For the number of
nearest neighbors, we set it to 200/1000 in 3D and 2D tasks, respectively.
Note that SG follows similar augmentation strategies used in MG.

  * •

Cube++ [35]. Similar to MG and SG, the input size is 64$\times$64$\times$32 in
LUNA and BraTS datasets. Then, we divide each volume into 32 subvolumes. The
size of each subvolume is 16$\times$16$\times$16\. We then apply random
rotation to each subvolume and employs a generative neural network to
reconstruct the original input volume.

  * •

C2L [51]. Give a 2D input image, we apply random crop, flip, rotation (10
degree) and grayscale to generate random patches. For each patch, we resize it
to 224$\times$224 and apply inpainting to it.

  * •

3D-CPC [13]. We follow the instructions in [13] to implement a 3D-CPC model.

  * •

PCRL. The augmentation strategies include random crop, random flip, random
rotation, inpainting, outpainting and gaussian blur. For 2D images, we simply
apply these operations to generate two input patches. Specially, for each 3D
input, we randomly crop 16 pairs of volumetric inputs. The two volumes in each
pair have the same size which is randomly sampled from
{64$\times$64$\times$32, 96$\times$96$\times$48, 96$\times$96$\times$96,
32$\times$32$\times$16, 112$\times$112$\times$56}. Note that we require the
Interaction over Union (IoU) within each pair to be larger than 0.2. Finally,
we resample each volume to 64$\times$64$\times$32444We only use LUNA for self-
supervised pretraining in 3D volumes.. Note that random rotation is only
applied to xy-plane in 3D volume where the maximum degree is $None$. The
length of queue which stores past representations is 16384 in PCRL. Note that
the reason why we do not use online random crop in 3D images is that the
computational cost of such operation (including crop and resize) is somewhat
unacceptable in our GPU device. For both 2D and 3D tasks, we take 4 NVIDIA
TITAN V GPUs to train PCRL models. The pretraining tasks on Chest14 and LUNA
take 1 day and a half day, respectively.

## References

  * [1] Spyridon Bakas, Mauricio Reyes, Andras Jakab, Stefan Bauer, Markus Rempfler, Alessandro Crimi, Russell Takeshi Shinohara, Christoph Berger, Sung Min Ha, Martin Rozycki, et al.  Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge.  arXiv preprint arXiv:1811.02629, 2018.
  * [2] Tariq Bdair, Nassir Navab, and Shadi Albarqouni.  ROAM: Random Layer Mixup for Semi-Supervised Learning in Medical Imaging.  arXiv preprint arXiv:2003.09439, 2020.
  * [3] Patrick Bilic, Patrick Ferdinand Christ, Eugene Vorontsov, Grzegorz Chlebus, Hao Chen, Qi Dou, Chi-Wing Fu, Xiao Han, Pheng-Ann Heng, Jürgen Hesser, et al.  The Liver Tumor Segmentation Benchmark (LiTS).  arXiv preprint arXiv:1901.04056, 2019.
  * [4] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep Clustering for Unsupervised Learning of Visual Features.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 132–149, 2018.
  * [5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised Learning of Visual Features by Contrasting Cluster Assignments.  arXiv preprint arXiv:2006.09882, 2020.
  * [6] Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu.  Contrastive Learning of Global and Local Features for Medical Image Segmentation with Limited Annotations.  arXiv preprint arXiv:2006.10511, 2020.
  * [7] Krishna Chaitanya, Neerav Karani, Christian F Baumgartner, Anton Becker, Olivio Donati, and Ender Konukoglu.  Semi-supervised and Task-driven Data Augmentation.  In International Conference on Information Processing in Medical Imaging, pages 29–41. Springer, 2019.
  * [8] Souradip Chakraborty, Aritra Roy Gosthipaty, and Sayak Paul.  G-SimCLR: Self-Supervised Contrastive Learning with Guided Projection via Pseudo Labelling.  arXiv preprint arXiv:2009.12007, 2020.
  * [9] Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Michitaka Fujiwara, and Daniel Rueckert.  Self-supervised Learning for Medical Image Analysis using Image Context Restoration.  Medical Image Analysis, 58:101539, 2019.
  * [10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A Simple Framework for Contrastive Learning of Visual Representations.  arXiv preprint arXiv:2002.05709, 2020.
  * [11] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved Baselines with Momentum Contrastive Learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [12] Xinlei Chen and Kaiming He.  Exploring Simple Siamese Representation Learning.  arXiv preprint arXiv:2011.10566, 2020.
  * [13] Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 424–432. Springer, 2016.
  * [14] Carl Doersch, Abhinav Gupta, and Alexei A Efros.  Unsupervised Visual Representation Learning by Context Prediction.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1422–1430, 2015.
  * [15] Zach Eaton-Rosen, Felix Bragman, Sebastien Ourselin, and M Jorge Cardoso.  Improving Data Augmentation for Medical Image Segmentation.  Medical Imaging with Deep Learning, 2018.
  * [16] Zeyu Feng, Chang Xu, and Dacheng Tao.  Self-supervised Representation Learning by Rotation Feature Decoupling.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10364–10374, 2019.
  * [17] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised Representation Learning by Predicting Image Rotations.  arXiv preprint arXiv:1803.07728, 2018.
  * [18] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap Your Own Latent: A New Approach to Self-supervised Learning.  arXiv preprint arXiv:2006.07733, 2020.
  * [19] Fatemeh Haghighi, Mohammad Reza Hosseinzadeh Taher, Zongwei Zhou, Michael B Gotway, and Jianming Liang.  Learning Semantics-enriched Representation via Self-discovery, Self-classification, and Self-restoration.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 137–147. Springer, 2020.
  * [20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum Contrast for Unsupervised Visual Representation Learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [21] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al.  CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 590–597, 2019.
  * [22] Wonmo Jung, Sejin Park, Kyu-Hwan Jung, and Sung Il Hwang.  Prostate Cancer Segmentation using Manifold Mixup U-Net.  In International Conference on Medical Imaging with Deep Learning–Extended Abstract Track, 2019.
  * [23] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich.  Learning Representations for Automatic Colorization.  In European conference on computer vision, pages 577–593. Springer, 2016.
  * [24] Mehdi Noroozi and Paolo Favaro.  Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles.  In European Conference on Computer Vision, pages 69–84. Springer, 2016.
  * [25] Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro.  Representation Learning by Learning to Count.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5898–5906, 2017.
  * [26] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation Learning with Contrastive Predictive Coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [27] Egor Panfilov, Aleksei Tiulpin, Stefan Klein, Miika T Nieminen, and Simo Saarakkala.  Improving Robustness of Deep Learning based Knee MRI Segmentation: Mixup and Adversarial Domain Adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [28] Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan.  Learning Features by Watching Objects Move.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2701–2710, 2017.
  * [29] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [30] Guo-Jun Qi, Liheng Zhang, Feng Lin, and Xiao Wang.  Learning Generalized Transformation Equivariant Representations via Autoencoding Transformations.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [31] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234–241. Springer, 2015.
  * [32] Arnaud Arindra Adiyoso Setio, Alberto Traverso, Thomas De Bel, Moira SN Berens, Cas van den Bogaard, Piergiorgio Cerello, Hao Chen, Qi Dou, Maria Evelina Fantacci, Bram Geurts, et al.  Validation, Comparison, and Combination of Algorithms for Automatic Detection of Pulmonary Nodules in Computed Tomography Images: the LUNA16 Challenge.  Medical image analysis, 42:1–13, 2017.
  * [33] Hari Sowrirajan, Jingbo Yang, Andrew Y Ng, and Pranav Rajpurkar.  MoCo Pretraining Improves Representation and Transferability of Chest X-ray Models.  arXiv preprint arXiv:2010.05352, 2020.
  * [34] Aiham Taleb, Winfried Loetzsch, Noel Danz, Julius Severin, Thomas Gaertner, Benjamin Bergner, and Christoph Lippert.  3D Self-supervised Methods for Medical Imaging.  arXiv preprint arXiv:2006.03829, 2020.
  * [35] Xing Tao, Yuexiang Li, Wenhui Zhou, Kai Ma, and Yefeng Zheng.  Revisiting Rubik’s Cube: Self-supervised Learning with Volume-wise Transformation for 3D medical Image Segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 238–248. Springer, 2020.
  * [36] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio.  Manifold Mixup: Better Representations by Interpolating Hidden States.  In International Conference on Machine Learning, pages 6438–6447. PMLR, 2019.
  * [37] Dong Wang, Yuan Zhang, Kexin Zhang, and Liwei Wang.  FocalMix: Semi-supervised Learning for 3D Medical Image Detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3951–3960, 2020.
  * [38] Xiaolong Wang and Abhinav Gupta.  Unsupervised Learning of Visual Representations using Videos.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2794–2802, 2015.
  * [39] Xiaolong Wang, Kaiming He, and Abhinav Gupta.  Transitive invariance for self-supervised visual representation learning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1329–1338, 2017.
  * [40] Xiaolong Wang, Allan Jabri, and Alexei A Efros.  Learning Correspondence from the Cycle-consistency of Time.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2566–2576, 2019.
  * [41] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers.  ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-supervised Classification and Localization of Common Thorax Diseases.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2097–2106, 2017.
  * [42] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [43] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised Feature Learning via Non-parametric Instance Discrimination.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733–3742, 2018.
  * [44] Xinpeng Xie, Jiawei Chen, Yuexiang Li, Linlin Shen, Kai Ma, and Yefeng Zheng.  Instance-Aware Self-supervised Learning for Nuclei Segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 341–350. Springer, 2020.
  * [45] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  Mixup: Beyond Empirical Risk Minimization.  arXiv preprint arXiv:1710.09412, 2017.
  * [46] Richard Zhang, Phillip Isola, and Alexei A Efros.  Split-brain Autoencoders: Unsupervised Learning by Cross-channel Prediction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1058–1067, 2017.
  * [47] Junjie Zhao, Donghuan Lu, Kai Ma, Yu Zhang, and Yefeng Zheng.  Deep Image Clustering with Category-Style Representation.  arXiv preprint arXiv:2007.10004, 2020.
  * [48] Hong-Yu Zhou, Bin-Bin Gao, and Jianxin Wu.  Sunrise or Sunset: Selective Comparison Learning for Subtle Attribute Recognition.  arXiv preprint arXiv:1707.06335, 2017.
  * [49] Hong-Yu Zhou, Hualuo Liu, Shilei Cao, Dong Wei, Chixiang Lu, Yizhou Yu, Kai Ma, and Yefeng Zheng.  Generalized Organ Segmentation by Imitating One-shot Reasoning using Anatomical Correlation.  In International Conference on Information Processing in Medical Imaging, pages 452–464. Springer, 2021.
  * [50] Hong-Yu Zhou, Chengdi Wang, Haofeng Li, Gang Wang, Shu Zhang, Weimin Li, and Yizhou Yu.  SSMD: Semi-Supervised Medical Image Detection with Adaptive Consistency and Heterogeneous Perturbation.  Medical Image Analysis, page 102117, 2021.
  * [51] Hong-Yu Zhou, Shuang Yu, Cheng Bian, Yifan Hu, Kai Ma, and Yefeng Zheng.  Comparing to Learn: Surpassing ImageNet Pretraining on Radiographs By Comparing Image Representations.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 398–407. Springer, 2020.
  * [52] Zongwei Zhou, Vatsal Sodha, Jiaxuan Pang, Michael B Gotway, and Jianming Liang.  Models genesis.  arXiv preprint arXiv:2004.07882, 2020.
  * [53] Jiuwen Zhu, Yuexiang Li, Yifan Hu, Kai Ma, S Kevin Zhou, and Yefeng Zheng.  Rubik’s Cube+: A Self-supervised Feature Learning Framework for 3D Medical Image Analysis.  Medical Image Analysis, page 101746, 2020.
  * [54] Xinrui Zhuang, Yuexiang Li, Yifan Hu, Kai Ma, Yujiu Yang, and Yefeng Zheng.  Self-supervised Feature Learning for 3D Medical Images by Playing a Rubik’s Cube.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 420–428. Springer, 2019.