2021

# Measurement of High Density Electrons Above a Helium Film on an Amorphous
Metal Substrate

K. E. Castoria<EMAIL_ADDRESS>S. A. Lyon<EMAIL_ADDRESS>Department of Electrical Engineering, Princeton University, Princeton, 08544,
New Jersey

###### Abstract

We have measured two-dimensional electron systems bound to a thin helium film
supported by a metallic substrate. We report on our measurement of electron
density obtained via a Kelvin probe technique. The underlying metallic
substrate is an amorphous metallic alloy (TaWSi), which can support large
uniform densities due to its low surface roughness and homogeneous work
function. We find that this substrate is able to support high enough densities
that the electrons are expected to be Fermi-degenerate.

###### keywords:

Electrons on Helium, Fermi Degenerate, Amorphous Metal, Thin Film

##  1 Introduction

Electrons bound to the surface of superfluid helium form a very clean 2D
electron gas owing to the smooth helium surface and the electrons’ relative
isolation inside the vacuumMonarkha_2004 . The system offers a unique platform
from which to probe new regimes in two-dimensional physics. One area of
particular interest is understanding Wigner crystallization with Fermi
degenerate electrons. For over forty years, quantum melting of a Wigner
crystal has been an elusive experimental goal driven by the predictions of
exotic phasesSpivak_2010  and quantum magnetismOrtiz_1999 . The phase boundary
separating the solid electron phase from the fluid phase goes up to
2.4$\times$ 1012 cm-2 for electrons on bulk helium at T = 0 K Peeters_1983 , a
density for which the electron system is degenerate. For bulk helium however,
this is experimentally infeasible, as a hydrodynamic instability prevents the
accumulation of electron densities higher than 2.2$\times$109
cm-2Williams_1971 . On thin films of helium however, the maximum density that
can be supported increases as the film gets thinnerPeeters_1984 . Films on the
order of ten nanometers and less can support densities in the range of a 1011
– 1012 cm-2Lytvynenko_2017 . In this paper, we report measurements up to an
electron density $n=4.74\times 10^{11}$ cm-2 on a helium film calculated to be
2 nm thick.

If the substrate below the helium film is metallic, the electron-electron
interaction is significantly altered. As the inter-electron spacing becomes
larger than the distance between an electron and its image, the interaction is
largely screened by the substrate. This effect depends on a quantity referred
to in Fig. 1 as Equivalent Helium Thickness (EHT), which is defined as

$EHT=d+r_{e}\epsilon_{He}$ (1)

Where $d$ is the helium film thickness, $r_{e}$ is the distance between the
electron layer and helium film surface, and $\epsilon_{He}$ is the dielectric
constant of helium.

Most notably the screening opens a gap in the phase diagram such that even as
the temperature approaches zero, the system remains a liquid for finite
densities. The blue curve in Fig. 1 shows the EHT as a function of density for
a helium film supported by our metal plate 2 mm above the bulk helium surface.
The details of this calculation are discussed in Sec. 2. The classical Wigner
crystal melting point as calculated by SaitohSaitoh_1989  is marked with a
purple dashed line. For densities lower than this value, the system is a
classical electron gas, and for densities larger than this value, the system
is a Wigner solid. As the density is increased, the helium film is compressed
and the electron is bound closer to the helium surface, increasing the
screening. It is expected that on helium films of ten nanometers and less, the
Wigner crystal is no longer stable at electron densities on the order of 1011
cm-2. At these densities, the Fermi energy is well above our experimental
temperature of 1.75 K (the black dashed line). While classical Wigner
transitions have been well studied both experimentally and
theoreticallygrimes_1979 ; Kosterlitz_1973 , the processes governing melting
in the Fermi degenerate regime are not yet fully known. Both the possibility
of polaron formation and quantum effects can significantly change the phase
diagram in the quantum regime.

Observing quantum melting has been difficult in part due to substrate
disorder, which leads to both electron loss and reduced mobility, precluding
transport measurements. We believe these problems can be mitigated with the
use of amorphous metals. Our helium film in this experiment is supported by a
very smooth amorphous metal (TaWSi). While this substrate is expected to allow
transport measurements(Asfaw_2019, ), none are reported here. Further
experiments will be required to fully understand the phase of the electron
system present. In this work, a description of the experimental set up and the
measurement techniques are given, followed by details about the amorphous
metal substrate and concluding with a report of our measured electron density.

Figure 1: Calculated Equivalent Helium Thickness (as defined in the text)
versus electron density (blue). As the density increases and the thickness
decreases, the electron system transitions from a classical gas (yellow
region) to a classical Wigner solid (purple region). The classical melting
point of the Wigner crystal as calculated by SaitohSaitoh_1989  is shown as
the purple dashed line. The density for which the system’s Fermi energy is
equal to our operating temperature of 1.75 K is denoted by the black dashed
line. For densities above this point, the system is Fermi degenerate (green
region). Also plotted are our measured densities discussed later in the text
(yellow and red points).

##  2 Description of Experiment

The results presented in this paper are measured with a Kelvin probe
techniqueLeners_1972  that can determine electron density irrespective of the
electron mobility. This is done so that measurements of electron density can
be made even if the electrons are localized due to crystallization, disorder,
or polaron formation. The basic experimental system is illustrated in Fig. 2a.
A small controllable amount of helium is let into our leak tight cell until a
desired quantity of helium is condensed at the bottom. Below 2.2 K, the helium
becomes a superfluid and a thin van der Waals film coats all the surfaces in
the cell above the bulk levelAtkins_1950 ; Sabisky_1973 . At a height, $H$,
above this inner liquid level, a metal plate is mounted that supports the thin
film and 2D electrons. Together with a second metal plate mounted a height,
$D_{0}$, above the first, a simple parallel plate capacitor is formed. This
capacitor is the Kelvin probe that will be used to measure electron density.
The upper plate has a small hole cut out, through which a tungsten filament
can emit electrons onto the lower helium surface. The filament and upper plate
are mounted on a stainless-steel tube that runs to room temperature and
terminates on a small piezo stack to drive vertical motion.

The top and bottom capacitor plates are made of an amorphous metallic alloy.
We are using Ta40W40Si20 (referred to as TaWSi), which has been used
previously in experiments showing transport of electrons on helium in channel
devices with helium films down to 200 nm Asfaw_2019 . These transport results
suggest that TaWSi exhibits the low surface potential disorder required to
support a uniform electron density across the whole Kelvin probe area. The
bottom plate consists of 100 nm of TaWSi sputtered onto a polished sapphire
substrate 2.25 cm2 in area. The top plate consists of 100 nm of TaWSi
sputtered onto the copper pad of a printed circuit board. Having both the top
and bottom plate made of TaWSi simplifies Eq. 3 by setting $\Delta_{\Phi}=0$.
Fig. 2b shows the surface topography for a 5 $\mu$m $\times$ 5 $\mu$m square
of a typical TaWSi film sputtered on polished sapphire as measured by an
atomic force microscope, and the profile along the center line of this area is
shown in Fig. 2c. The RMS surface roughness is measured to be 1.5 Å.

Figure 2: a Schematic of the experimental setup for measuring the electron
density. The electrons (orange dots) are bound to the helium film (blue). The
film is supported by the lower plate (red) which is a height, $H$, above the
bulk helium level and connected to a lock-in amplifier to measure the Kelvin
probe current ($I_{KP}$). The upper plate (red) is a distance, $D_{0}$, above
the lower plate and connected to a DC voltage source, $V_{Bias}$. This plate
moves vertically by an amount, $\Delta$, as determined by the AC drive
voltage, $V_{Piezo}$, applied to the the piezo stack (green). b The surface
profile of a 5 $\mu$m$\times$5 $\mu$m patch of TaWSi sputtered on sapphire
measured via atomic force microscopy. c The height profile along the black
line pictured in b.

The piezo stack can move the top plate by a maximum distance, $\Delta$, and is
controlled with an AC voltage, $V_{Piezo}$. By driving the stack at a
frequency, $\omega$, the top plate displacement creates a time-varying
capacitance. An AC current, $I_{KP}$, at the drive frequency, $\omega$, will
then flow given a DC voltage across the capacitor. In our experiment the DC
voltage across the capacitor depends on an externally applied voltage
difference, $V_{Bias}$, the work function difference between the top and
bottom metals, $\Delta_{\Phi}$, and the electron potential of the 2D electrons
on the helium film. By differentiating the equation for the charge on a
capacitor with respect to time, we find that the current through the capacitor
is

$I_{KP}=\frac{\partial}{\partial
t}\left(\frac{\epsilon_{0}A}{D(t)}\right)\left(V_{Bias}+V_{e}+\Delta_{\Phi}\right)$
(2)

where $\epsilon_{0}$ is the permitivity of free space, $A$ is the plate area,
$D(t)$ is the time dependent plate separation distance, and $V_{e}$ is the
electron potential.

We drive the piezo such that $D(t)=D_{0}+\Delta sin\left(\omega t\right)$. In
the limit $\left(\frac{\Delta}{D_{0}}\ll 1\right)$, it can be shown that the
current through the capacitor at the drive frequency is given by

$I_{KP}=C_{0}\omega\frac{\Delta_{RMS}}{D_{0}}\left(V_{Bias}+\Delta_{\Phi}+V_{e}\right)$
(3)

where $C_{0}$ is the mean parallel plate capacitance and $\Delta_{RMS}$ is the
RMS travel of the top plate.

From Eq. 3, it can be seen that by sweeping $V_{Bias}$ and using a lock-in
amplifier to measure $I_{KP}$ at $\omega$, we expect to measure a linear I-V
curve. Further, by charging the helium film and sweeping $V_{Bias}$ again, we
expect the line to be shifted by a voltage offset proportional to $V_{e}$.

We then need to to determine the electron density from the measured electron
potential. The helium film thickness, $d$, is much smaller than the mean plate
separation, $D_{0}$, so we can ignore the electrons’ capacitance to the top
plate and just consider their potential from their capacitance to the bottom
plate. The film thickness for densities above 10$\times$1011 cm-2 is on the
order of single nanometers, comparable to the size of the electrons’
wavefunction. For this reason, it is necessary to numerically solve the
Schrodinger equation to account for the wavefunction extent of the electron in
computing its capacitance to the bottom plateJoseph_2021 .

For a given electron density and substrate height above the bulk helium, the
film thickness on a plate is well understoodSabisky_1973 ; Leiderer_1984 .
Using this thickness, we compute the electron wavefunction normal to the
surface. We use this wavefunction to find $V_{e}$ by integrating the potential
from the bottom plate and find

$V_{e}=\int_{0}^{\infty}\psi^{\dagger}(z)\psi(z)\frac{4\pi
ne}{\epsilon_{0}\epsilon_{He}}\left(d+z\epsilon_{He}\right)dz$ (4)

where $e$ is the electronic charge and $z$ is the coordinate normal to the
helium surface. This calculation implicitly assumes that the TaWSi is a
perfect metal. From spectroscopic ellipsometry of the TaWSi we estimate a
plasma energy of about 10 eV which would be consistent with this assumption.

Using this, we get a one-to-one mapping between electron density and electron
potential. We therefore can use the measured voltage shifts of the Kelvin
probe I-V curve to determine the electron density and film thickness.

##  3 TaWSi Plate Results

Results from this Kelvin probe apparatus show that we were able to charge our
helium films to densities above 1011 cm-2. Measurements from two such
experiments are shown in Fig. 3 with the insets showing the Kelvin probe
current shifting during charging. For the results presented in Fig. 3a, the
piezo is driven its full travel range of $\Delta_{RMS}$ = 17 $\mu$m at
$\omega$ = 14.5 Hz. The bottom plate is mounted about 2 mm above the bulk
helium and the plates are separated by D0 = 404 $\mu$m. Prior to emission,
$V_{Bias}$ is swept from -2 V to 0 V and the current through the uncharged
probe is recorded (blue circles). With the top plate held at $V_{Bias}=-2V$
and no electrons emitted, the uncharged current through the probe is measured
to be 11.68 pA. This is slightly less than the expected current of 13.5 pA,
possibly due to the imperfect AC response of the piezo. The tungsten filament
is pulsed until no more change in the Kelvin probe current is observed with
further flashes. A typical charging plot is shown in the inset. Once charged,
$V_{Bias}$ is again swept, this time to +2 V (red circles). For $V_{Bias}<0$,
the measured current follows a shifted I-V curve (red line). Once the top
plate voltage becomes more positive than 0.5 V, the electrons desorb from the
helium film, and the measured current closely follows the linear fit to the
uncharged I-V curve (blue line). The voltage offset between these fits is 300
mV, which corresponds to a density of 4.74$\times$1011 cm-2, film thickness of
2.0 nm, and an EHT of 3.4 nm. As long as $V_{Bias}$ is kept at the initial -2
V, the electrons are stably held on the surface. No change in density was
observed while held for a period of thirty minutes.

In Fig. 3b, a similar experiment is shown. In this experiment, the drive
frequency was 17.5 Hz and the plates were measured to be 483 $\mu$m apart.
Here, $V_{Bias}$ is swept from -2 V to 4 V and back, all after emitting
electrons (as shown in the inset). The electrons desorbed once $V_{Bias}$
became more positive than 1.4 V. Until this point, the I-V curve closely
followed the charged fit (red line). For the remainder of the sweep to 4 V and
the entire sweep back to -2 V, the electrons followed the uncharged fit (blue
line). A voltage shift of 247 mV is measured, which corresponds to a density,
film thickness, and EHT of 3.06$\times$1011 cm-2, 2.7 nm, and 4.3 nm
respectively.

Figure 3:  Kelvin probe measurements of the 2D electron system. a Measurement
of the Kelvin probe current before (blue circles) and after (red circles)
while scanning the top plate voltage, $V_{bias}$. Linear fits to both sweeps
over the region $V_{Bias}<0$ are shown as solid lines. b Measurement of the
Kelvin probe current while sweeping $V_{Bias}$ from -2 V to 4 V and back to
-2V. Overlaid in red (blue) is a linear fit to the I-V curve while the film is
charged (uncharged). Both insets show the charging of the helium surface as a
function of time with discrete filament pulses about once per minute, while
$V_{Bias}$ is held at -2 V.

##  4 Channel Device Results

The Kelvin probe shifts are susceptible to stray capacitance with the
environmentBaikie_1991 , so it is conceivable that charging insulating
surfaces inside the cell or helium films outside of the capacitor region could
lead to misleading voltage shifts. To confirm that the electrons on the bottom
plate’s film cause the observed shifts, a similar Kelvin probe experiment is
performed as described in Sec. 3. In these experiments, the bottom TaWSi plate
is replaced with a two layer device consisting of helium filled channels of
the same area, and $V_{Bias}$ is connected to two large area gates on this
device, as shown in Fig. 4. The top plate is biased to a negative DC voltage
(-2.5 V in the experiments shown) and capacitively coupled to the lock-in
amplifier to measure the $I_{KP}$.

A top down diagram of the two layer device is shown in Fig 4a. The bottom
layer (grey) is made of aluminum and is divided into three gates. The large
outer two reservoir gates are 7 $\times$ 18 mm2 and the center barrier gate is
0.01 $\times$ 18 mm2. The top layer of TaWSi (red) divides these gates into
1210 channels, each 7 $\mu$m wide. A cross section of one such channel filled
with helium is shown in Fig 4b. An 800 nm layer of silicon nitride (yellow)
below the 40 nm thick TaWSi creates channels 840 nm deep. With the device
above the bulk helium, the channels are filled with superfluid helium by
capillary actionMarty_1986 . Electrons are confined to the channels by biasing
the bottom aluminum gates positive relative to the TaWSi layer at the top of
the channels.

Figure 4: Two layer channel device diagrams and measurements a Top down view
of the channel device. The 1210 channels are 14 mm long, 7 $\mu$m wide, and
separated by 7 $\mu$m wide walls. b Side view of the channel device. The
channels are comprised of 800 nm of silicon nitride (yellow) and 40 nm of
TaWSi (red) deposited onto the bottom aluminum layer (grey). Superfluid helium
(blue) fills the channels due to capillary action. c Circuit diagram of the
transport measurements. In all of the experiments discussed, the upper TaWSi
layer is tied to ground. The values $R$ and $C$ are chosen to have no effect
in the frequency range of operation. d Measurements of the Kelvin probe
current (as measured from the top capacitor plate) while sweeping the bias on
the aluminum plates. The scans are taken when the channels are empty (blue),
partially filled (green), and full (orange). The inset shows corresponding
transport current along the channels while sweeping the barrier gate. The
dashed vertical lines show the voltage offsets measured by the Kelvin probe.

To determine electron potential in the channels, we measure electron transport
across the device as a function of the barrier gate voltage. A circuit diagram
of this scheme is shown in Fig 4c. Both reservoirs are connected to a positive
DC bias. One of the reservoirs is capacitively coupled to an AC voltage,
$V_{Res}$, with a frequency $\omega_{Res}>>\frac{1}{RC}$, and the other is
coupled to a lock-in amplifier to measure the induced current, $I_{Res}$. Most
of the current is carried by the surface electrons, so it can be controlled by
the difference between the barrier gate voltage and the reservoirs’ DC bias
$\Delta V_{Barrier}$. By sweeping $\Delta V_{barrier}$ negative, the current
will remain constant until the barrier potential is equal to the electron
potential in the channels. As it becomes more negative than the electron
potential, the electrons in the channels can no longer travel between
reservoirs, and a sharp decrease in $I_{Res}$ is observed. The voltage where
this transition occurs is equal to the electron potential in the channels.

To show that the voltage shift measured by the Kelvin probe is only due to
electrons above the bottom capacitor surface, the electron potential is
measured both via channel transport and the Kelvin probe. The channels are
charged to two different densities as shown in Fig. 4d. Measurements are shown
with the channels empty (blue), partially filled with electrons (green), and
charged to saturation (orange). The channel transport measurements are shown
in the inset with dashed lines indicating the corresponding Kelvin probe
voltage shift (285 mV for the low density and 431 mV for the high density). In
both cases, the shift voltage is in good agreement with the $\Delta
V_{barrier}$ where transport begins to turn off.

##  5 Conclusion

In these measurements, we have shown that high densities of electrons can be
stably held on a thin helium surface over a metallic substrate. Using
amorphous metals and fabricating smoother substrates, we are able to hold
higher densities than previously reported in this system. These densities were
determined with a Kelvin probe measurement technique. With densities above
4$\times$1011 cm-2, the Fermi energy is an order of magnitude larger than the
ambient temperature of 1.75K. Using computed electron wavefunctions, electron
densities and helium film thicknesses were determined from the measured
voltage shifts. The results of this measurement technique were compared to an
independent measurement of electron density via transport in channels. We find
that the two different methods are in good agreement with each other. In order
to get information regarding the nature of the many-body electron system,
further measurements will be necessary. The calculations of Peeters and
PlatzmanPeeters_1983  suggest that the Wigner crystal will not form at these
densities and film thicknesses. Unless the electron mass is increased, due to
polaron formation for example, the electrons are Fermi degenerate. However, it
is thought that the the self-trapping energy for polarons is on the
millikelvin scaleJackson_1986 , well below the temperatures used here and far
below the Fermi temperature. Disorder may also impede the formation of a Fermi
liquid. In some situations it is thought that isolated ”puddles” of a Fermi
liquid are formed, but disorder does not allow them to mergeDeng_2016 . The
strength of the disorder seen by these electrons is not yet known.

##  6 Data Availability

The datasets generated during and/or analysed during the current study are
available from the corresponding author on reasonable request.

##  7 Acknowledgments

This material is based upon work supported by the U.S. Department of Energy,
Office of Science, Office of Basic Energy Sciences, under Award number DE-
SC0020136. Support was provided, in part, through the Program in Plasma
Science and Technology under DOE Contract Number DE-AC02-09CH11466.

##  8 Compliance with Ethical Standards

S.A Lyon is an officer of Eeroq Corp.

## References

  *   * (1) Monarkha, Y., Kono, K.: Two-Dimensional Coulomb Liquids and Solids. Springer (2004). https://doi.org/10.1007/978-3-662-10639-6
  * (2) Spivak, B., Kravchenko, S.V., Kivelson, S.A., Gao, X.P.A.: Colloquium: Transport in strongly correlated two dimensional electron fluids. Reviews of Modern Physics 82(2), 1743–1766 (2010) arXiv:0905.0414 [cond-mat.dis-nn]. https://doi.org/10.1103/RevModPhys.82.1743
  * (3) Ortiz, G., Harris, M., Ballone, P.: Zero Temperature Phases of the Electron Gas. prl 82(26), 5317–5320 (1999) arXiv:cond-mat/9810126 [cond-mat]. https://doi.org/10.1103/PhysRevLett.82.5317
  * (4) Peeters, F.M., Platzman, P.M.: Electrons on films of helium: A quantum mechanical two-dimensional fermion system. Phys. Rev. Lett. 50, 2021–2023 (1983). https://doi.org/10.1103/PhysRevLett.50.2021
  * (5) Williams, R., Crandall, R.S.: Deformation of the surface of liquid helium by electrons. Physics Letters A 36(1), 35–36 (1971). https://doi.org/10.1016/0375-9601(71)90051-X
  * (6) Peeters, F.M.: Two-dimensional wigner crystal of electrons on a helium film: Static and dynamical properties. Phys. Rev. B 30, 159–165 (1984). https://doi.org/10.1103/PhysRevB.30.159
  * (7) Lytvynenko, D.M., Slyusarenko, Y.V.: Analyzing the equilibrium states of a quasi-neutral spatially inhomogeneous system of charges above a liquid dielectric film based on the first principles of quantum statistics. Journal of Physics A: Mathematical and Theoretical 50(31), 315202 (2017). https://doi.org/10.1088/1751-8121/aa76ab
  * (8) Saitoh, M.: Melting temperature of two-dimensional electron crystals trapped on thin-film liquid he. Phys. Rev. B 40, 810–812 (1989). https://doi.org/10.1103/PhysRevB.40.810
  * (9) Grimes, C.C., Adams, G.: Evidence for a liquid-to-crystal phase transition in a classical, two-dimensional sheet of electrons. Phys. Rev. Lett. 42, 795–798 (1979). https://doi.org/10.1103/PhysRevLett.42.795
  * (10) Kosterlitz, J.M., Thouless, D.J.: Ordering, metastability and phase transitions in two-dimensional systems. Journal of Physics C: Solid State Physics 6(7), 1181–1203 (1973). https://doi.org/10.1088/0022-3719/6/7/010
  * (11) Asfaw, A.T., Kleinbaum, E.I., Henry, M.D., Shaner, E.A., Lyon, S.A.: Transport measurements of surface electrons in 200-nm-deep helium-filled microchannels above amorphous metallic electrodes. Journal of Low Temperature Physics 195(3-4), 300–306 (2019). https://doi.org/10.1007/s10909-018-02139-6
  * (12) Leners, K.H., Kearney, R.J., Dresser, M.J.: Stress-dependent contact potential in copper. Phys. Rev. B 6, 2943–2950 (1972). https://doi.org/10.1103/PhysRevB.6.2943
  * (13) Atkins, K.R., Bragg, W.L.: Liquid helium films. i. the thickness of the film. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences 203(1072), 119–132 (1950) https://royalsocietypublishing.org/doi/pdf/10.1098/rspa.1950.0129. https://doi.org/10.1098/rspa.1950.0129
  * (14) Sabisky, E.S., Anderson, C.H.: Verification of the lifshitz theory of the van der waals potential using liquid-helium films. Physical Review A 7, 790–806 (1973)
  * (15) Joseph, E. Private communication
  * (16) Etz, H., Gombert, W., Idstein, W., Leiderer, P.: Stability of charged ${}^{4}\mathrm{He}$ films. Phys. Rev. Lett. 53, 2567–2570 (1984). https://doi.org/10.1103/PhysRevLett.53.2567
  * (17) Baikie, I.D., Venderbosch, E., Meyer, J.A., Estrup, P.J.Z.: Analysis of stray capacitance in the kelvin method. Review of Scientific Instruments 62(3), 725–735 (1991) https://doi.org/10.1063/1.1142075. https://doi.org/10.1063/1.1142075
  * (18) Marty, D.: Stability of two-dimensional electrons on a fractionated helium surface. Journal of Physics C: Solid State Physics 19(30), 6097–6104 (1986). https://doi.org/10.1088/0022-3719/19/30/019
  * (19) Peeters, F.M., Jackson, S.A.: Temperature dependence of the dynamical response of an electron on a thin liquid-helium film. Phys. Rev. B 34, 1539–1549 (1986). https://doi.org/10.1103/PhysRevB.34.1539
  * (20) Deng, H., Liu, Y., Jo, I., Pfeiffer, L.N., West, K.W., Baldwin, K.W., Shayegan, M.: Commensurability oscillations of composite fermions induced by the periodic potential of a wigner crystal. Phys. Rev. Lett. 117, 096601 (2016). https://doi.org/10.1103/PhysRevLett.117.096601


* [8] R.M. Chen, J. Hu and D. Wang, Linear stability of compressible vortex sheets in 2D elastodynamics: variable coefficients, Math. Ann. (2019), 1-50.
  * [9] R.M. Chen, J. Hu, D. Wang, T. Wang and D. Yuan, Nonlinear stability and existence of compressible vortex sheets in 2D elastodynamics, J. Differential Equations. 269 (2020), 6899-6940.
  * [10] R.M. Chen, D. Wang, F.M. Huang and D. Yuan, Stabilization effect of elasticity on three-dimensional compressible vortex sheets, preprint, 2019.
  * [11] S. Chen, Initial boundary value problems for quasilinear symmetric hyperbolic systems with characteristic boundary. Translate from Chinese Ann. Math.3(2) (1982), 222-232. Front. Math. China 2(1) (2007), 87-102.
  * [12] J.-F. Coulombel, A. Morando, Stability of contact discontinuities for the nonisentropic Euler equations, Ann. Univ. Ferrara Sez. VII (N.S.) 50 (2004), 79–90.
  * [13] J.-F. Coulombel and P. Secchi, The stability of compressible vortex sheets in two space dimensions, Indiana Univ. Math. J. 53 (2004), 941-1012.
  * [14] J.-F. Coulombel and P. Secchi, Nonlinear compressible vortex sheets in two space dimensions, Ann. Sci. Ec. Norm. Super. 41 (2008), 85-139.
  * [15] C.M. Dafermos, Hyperbolic Conservation Laws in Continuum Physics. Third Edition, Grundlehren der Mathematischen Wissenschaften (Fundatmental Principles of Mathematical Sciences), vol. 235, Springer-Verlag, Berlin, 2010.
  * [16] J.A. Fejer and W. Miles, On the stability of a plane vortex sheet with respect to three dimensional disturbances, J. Fluid Mech. 15 (1963), 335-336.
  * [17] H. Freistühler, Y. Trakhinin, Symmetrizations of RMHD equations and stability of relativistic current-vortex sheets, Classical and Quantum Gravity, 30(8) (2012), 85012-85028(17).
  * [18] J.P. Goedbloed, S. Poedts, Principles of magnetohydrodynamics with applications to laboratory and astrophysical plasmas. Cambridge University Press, Cambridge, 2004.
  * [19] L.D. Landau, E.M. Lifshiz and L.P. Pitaevskii, Electrodynamics of continuous media. Pergamon Press, Oxford, 1984.
  * [20] G. Métivier, Stability of multidimensional shocks. In: Freistühler, H., Szepessy, A. (eds.) Advances in the Theory of Shock waves, 25-103. Birkhäuser, Boston (2001)
  * [21] D.H. Michael.  The stability of a combined current and vortex sheet in a perfectly conducting fluid.  Proc. Cambridge Philos. Soc., 51:528–532, 1955.
  * [22] J.W. Miles, On the reflection of sound at an interface of relative motion, J. Acoust. Soc. Am. 29 (1957), 226-228.
  * [23] J.W. Miles, On the disturbed motion of a plane vortex sheet, J. Fluid. Mech. 4 (1958), 538-552.
  * [24] A. Morando, P. Secchi and P. Trebeschi, Regularity of solutions to characteristic initial-boundary value problems for symmetrizable systems, J. Hyperbolic Differ. Equ, 6(4) (2009), 753-808.
  * [25] A. Morando and P. Trebeschi, Two-dimensional vortex sheets for the nonisentropic Euler equations: linear stability, J. Hyperbolic Differ. Equ. 5(3) (2008), 487-518.
  * [26] A. Morando, P. Trebeschi and T. Wang, Two-dimensional vortex sheets for the nonisentropic Euler equations: nonlinear stability, J. Differential Equations, 266(9) (2019), 5397-5430.
  * [27] M. Ohno, Y. Shizuta and T. Yanagisawa, The trace theorem on anisotropic Sobolev spaces, Tôhoku Math. J. 46 (1994),393-401.
  * [28] M. Ohno, Y. Shizuta and T. Yanagisawa, The initial boundary value problem for linear symmetric hyperbolic problems with boundary characteristic of constant multiplicity, J. Math. Kyoto Univ. 35 (1995), 143-210.
  * [29] J. Rauch, Symmetric positive systems with boundary characteristic of constant multiplicity. Trans. Amer. Math. Soc. 291 (1985), 167-187.
  * [30] P. Secchi, The initial-boundary value problem for linear symmetric hyperbolic systems with characteristic boundary of constant multiplicity. Differential Integral Equations, 9(4): 671–700, 1996.
  * [31] P. Secchi, Well-posedness of characteristic symmetric hyperbolic systems. Arch. Ration. Mech. Anal. 134 (1996), 155-197.
  * [32] P. Secchi, A higher-order Hardy-type inequality in anisotropic Sobolev spaces. Int. J. Differ. Equ. 2012, Art. ID 129691, 7 pp.
  * [33] P. Secchi, On the Nash-Moser iteration technique. In: Amann, H., Giga, Y. Kozono, H., Okamoto, H., Yamazaki, M., (eds.) Recent Developments of Mathematical Fluid Mechanics, pp. 443-457, Birkhäuser, Basel (2016).
  * [34] P. Secchi, Y. Trakhinin, Well-posedness of plasma-vacuum interface problem. Nonlinearity 27 (2014), 105-169.
  * [35] D. Serre, Systems of Conservation Laws.2. Geometric Structure, Oscillations, and Initial-Boundary Value Problems, Cambridge University Press, Cambridge, 2000.
  * [36] B. Stevens, Short-time structural stability of compressible vortex sheets with surface tension. Arch. Ration. Mech. Anal. 222(2) (2016), 603–730.
  * [37] S.I. Syrovatskij,  The stability of tangential discontinuities in a magnetohydrodynamic medium.  Zhurnal Eksper. Teor. Fiz., 24 (1953), 622–629.
  * [38] Y. Trakhinin, Existence of compressible current-vortex sheets: variable coefficients linear analysis, Arch. Ration. Mech. Anal. 177(3) (2005), 331-366,
  * [39] Y. Trakhinin, The existence of current-vortex sheets in ideal compressible magnetohydrodynamics, Arch. Ration. Mech. Anal. 191 (2009), 245-310.
  * [40] Y.-G. Wang and F. Yu, Stabilization effect of magnetic fields on two-dimensional compressible current-vortex sheets, Arch. Ration. Mech. Anal. 208 (2013), 341-389.
  * [41] Y.-G. Wang and F. Yu, Stability of contact discontinuities in three-dimensional compressible steady flows, J. Differential Equations. 255 (2013), 1278-1356.
  * [42] Y.-G. Wang and F. Yu, Structural stability of supersonic contact discontinuities in three-dimensonal compressible steady flows, SIAM J. Math. Anal. 47 (2015), 1291-1329.


# DeepSet SimCLR: Self-supervised deep sets for improved pathology
representation learning

David Torpey
School of Computer Science and Applied Mathematics
University of the Witwatersrand, Johannesburg
South Africa
<EMAIL_ADDRESS>
& Richard Klein
School of Computer Science and Applied Mathematics
University of the Witwatersrand, Johannesburg
South Africa
<EMAIL_ADDRESS>
Alternative email address<EMAIL_ADDRESS>
###### Abstract

Often, applications of self-supervised learning to 3D medical data opt to use
3D variants of successful 2D network architectures. Although promising
approaches, they are significantly more computationally demanding to train,
and thus reduce the widespread applicability of these methods away from those
with modest computational resources. Thus, in this paper, we aim to improve
standard 2D SSL algorithms by modelling the inherent 3D nature of these
datasets implicitly. We propose two variants that build upon a strong baseline
model and show that both of these variants often outperform the baseline in a
variety of downstream tasks. Importantly, in contrast to previous works in
both 2D and 3D approaches for 3D medical data, both of our proposals introduce
negligible additional overhead over the baseline, improving the
democratisation of these approaches for medical applications.

_Keywords_ Deep learning $\cdot$ Self-supervised learning $\cdot$
Representation learning $\cdot$ Affine transformation $\cdot$ Medical image
analysis

##  1 Introduction

Self-supervised learning (SSL) has pushed forward the capabilities of large-
scale visual representation learning due to its ability to model large
datasets without manual or human annotation (Chen et al., 2020a, b; Caron et
al., 2021; Chen et al., 2020c). Instead, these methods derive a supervision
signal from the data itself and often are architected to solve a proxy task of
some type. Through recent improvements, self-supervised pretraining has been
shown to outperform supervised pretraining on many downstream tasks (Ericsson
et al., 2020; Chen et al., 2020b). This is particularly important for medical
domains, where annotation is extremely expensive, laborious, and time-
consuming to obtain. However, these methods, even in their standard 2D
setting, are often notably computationally intensive to train, often requiring
tens of GPUs to train (Caron et al., 2021; Zbontar et al., 2021; Grill et al.,
2020; Caron et al., 2020) and sometimes requiring hundreds (Chen et al.,
2020b). 3D extensions of these techniques are even more computationally
intensive. It should be noted that even tens of GPUs severely reduce the
potential widespread applicability of these techniques.

Motivated by the success of the DeepSet (Zaheer et al., 2017) architecture for
the application of whole slide image (WSI) classification (Hemati et al.,
2021), we propose two variants that build upon a strong modern contrastive SSL
baseline. The variants use an intra-scan sampling strategy, with one variant
employing the DeepSet architecture, to implicitly model the 3D nature of
medical data using an inherently 2D architecture. Whereas previous approaches
focus on modelling the 3D volumes using either 3D native architectures (Taleb
et al., 2020; Tang et al., 2022) or computationally-intensive 2D architectures
(Nguyen et al., 2023), we instead focus on making this process more
computationally efficient using an architecture proven to work well for the
modelling of pathology data (Hemati et al., 2021) while inducing limited
computational overhead.

We propose two novel variants of a multi-view contrastive baseline: SimCLR
(Chen et al., 2020a). First, we devise a simple and effective slice sampling
strategy for the generation of random views. Next, we propose a new
architecture where each ‘view’ is a set of images rather than a single image,
which is operated upon using a Deep Set network. Our adaptations introduce
negligible additional computational cost to the architecture (at most a small
multi-layer perceptron). In this way, we inherit the plethora of benefits from
the base SimCLR architecture, while improving performance in the medical
domain.

Through extensive experiments on various downstream datasets and task types,
we show that the two variants consistently outperform the baseline, sometimes
by a large margin. Further, we perform analyses of core architectural
decisions, including the size of the set fed into the Deep Set, the sampling
strategy, and how the Deep Set is embedded within SimCLR.

We note that our goal is to improve upon a standard SSL baseline model, and
_not to achieve a new state-of-the-art_. This is because the ubiquity of SSL
methods means that any improvements upon them can greatly benefit many
domains, particularly those tailored for the medical domain due to its
exceptionally prohibitive annotation bottleneck. Our focus is on changes to an
SSL baseline that are easy to integrate and deploy, while simultaneously
robust and performant with negligible performance overhead. This is in stark
contrast to existing works for self-supervision on medical data which are
typically computationally intensive compared to standard SSL models (Nguyen et
al., 2023; Taleb et al., 2020; Tang et al., 2022). As such, we note our
contributions:

  * •

Two proposed models that implicitly model the 3D nature of pathology data
while remaining similarly computationally efficient as, and significantly more
performant than, the strong baseline.

  * •

A detailed study of the effect of the key hyperparameters introduced by our
proposed models.

  * •

A comprehensive evaluation on a large set of downstream datasets that cover a
variety of medical tasks and pathology modalities.

The rest of this paper is structured as follows. In Section 2, we describe the
relevant literature in both SSL and its applications to the medical domain.
Section 3, we describe the baseline model and our two proposed variants in
detail. Section 4 contains the experiments and analyses for our variants on
multiple downstream datasets and tasks. Lastly, Section 5 contains concluding
remarks and proposals for future work.

##  2 Related Work

###  2.1 Self-Supervised Learning

Modern approaches to SSL typically fall into one of 3 categories: 1)
contrastive, 2) non-contrastive, or 3) clustering. Techniques such as SimCLR
(Chen et al., 2020a, b), NNCLR (Dwibedi et al., 2021), and MoCo (He et al.,
2020; Chen et al., 2020c) are contrastive approaches. These approaches aim to
pull representations of similar ‘views’ (i.e. those sourced from the same
image) together while pulling the representations of dissimilar ‘views’ apart.
Non-contrastive approaches such as Barlow Twins (Zbontar et al., 2021) and
BYOL (Grill et al., 2020) still discriminate between instances but instead
optimise for different objectives that are reliant on negative samples such as
contrastive losses. For example, Barlow Twins aims to make the cross-
correlation matrix between two batches of representations of views to be as
similar to the identity matrix as possible. BYOL is a self-distillation
approach that aims to train a teacher-student architecture where the goal of
the first network is to predict the representation produced by the other.

Clustering approaches such as SwAV (Caron et al., 2020) try to circumvent a
strong assumption made by contrastive approaches where views from the same
image, whatever they may depict, are trained to be represented by the same
vector, and views from different images are always trained to be as dissimilar
as possible. SwAV therefore attempts to discriminate between clusters of
representations instead of individual instance representations, where clusters
contain representations of different views of the same image.

In general, these types of SSL approaches rely on generating the views through
data augmentation using a stochastic augmentation function which is typically
comprised of transformations such as random cropping, horizontal flipping,
Gaussian blurring, and colour jittering. This augmentation function must
produce a variety of views while maintaining the semantics between views
generated from the same image. There is also a trend towards attempting to
better understand the properties of these SSL methods, as they as not yet well
understood (Zhang et al., 2022; Torpey and Klein, 2022; Jing et al., 2022).

Another class of SSL techniques are pretext tasks where the network is tasked
with solving manual proxy tasks usually based on either image restoration or
spatial content. These include techniques such as rotation prediction (Gidaris
et al., 2018), colourisation (Zhang et al., 2016), inpainting (Pathak et al.,
2016), and image jigsaw puzzles (Noroozi and Favaro, 2016).

###  2.2 Applications of SSL to Medical Data

Since SSL is a promising and scalable approach to modelling large datasets
while being annotation-free, attempts have been made to apply it to the
medical domain where obtaining annotations is particularly prohibitive to do
at scale. Most SSL approaches in the medical domain can be classed as either
2D or 3D - whether they model 2D medical image data, or 3D volumes from
sources such as computed tomography (CT) or medical resonance imaging (MRI)
scans.

Kang et al. (2023) show that domain-aligned pretraining (i.e. pretraining on a
medical dataset) consistently outperforms the standard paradigm of ImageNet
pretraining for multiple common SSL methods. Further, they propose an altered
set of transformations for the data augmentation pipeline to generate views
more aligned with the medical domain. These changes include weaker colour
jittering, random vertical flips, and a novel stain augmentation (tailored to
WSI images). This new set of transformations improves performance over the
standard set of augmentations included in most modern SSL techniques (Chen et
al., 2020a; Grill et al., 2020; Zbontar et al., 2021).

Taleb et al. (2020) proposes 3D variants of multiple successful 2D SSL
algorithms, including CPC (van den Oord et al., 2018), RotNet (Gidaris et al.,
2018), image jigsaw puzzles (Noroozi and Favaro, 2016), relative patch
prediction (Doersch et al., 2015), and Exemplar networks (Dosovitskiy et al.,
2014). These SSL models are naturally extended into 3D due to their reliance
on spatial context. They demonstrate their success for multiple downstream
medical tasks.

Tang et al. (2022) proposed a novel 3D vision transformer (ViT) (Kolesnikov et
al., 2021) architecture to facilitate SSL on 3D medical volumes. They propose
random cutout and rotation as the transformations for the generation of 3D
views as they posit that these are more aligned with human anatomy. This SS
ViT is tasked with solving a multi-task objective consisting of inpainting,
rotation prediction, and contrastive learning.

Nguyen et al. (2023) attempts to jointly model 2D and 3D medical data to
capture both the spatial semantics of the individual 2D images, as well as
long-range dependencies between the slices from a 3D volume. The framework is
built upon clustering-based SSL algorithms SwAV (Caron et al., 2020) and Deep-
ClusterV2 (Caron et al., 2019). A transformer (Vaswani et al., 2017) with a
novel deformable self-attention mechanism is added into the base architecture
to capture the aforementioned long-range dependencies of 3D volume data. Our
proposed method differs from this approach in that we model the 3D volume as a
set or implicitly through sampling, instead of as a sequence with a
transformer. Further, our proposed models introduce significantly less
computational overhead than the additional transformer and its masked
embedding prediction task.

##  3 Methodology

Consider a dataset of 3D volumes, such as CT or MRI scans. Each scan $S_{i}$
is comprised of an _ordered_ collection of slices:
$S_{i}=\\{x_{j}\in\mathbb{R}^{H\times W\times C}\\}_{j=1}^{L_{i}}$, where
$H\times W\times C$ are the spatial dimensions of slice $x_{j}$ and $L_{i}$ is
the number of slices in scan $S_{i}$. We define a dataset of $N$ scans
$\mathcal{S}\coloneqq\\{S_{i}\\}_{i=1}^{N}$ and a flattened version consisting
of all slices from all scans $\mathcal{S}_{\texttt{flat}}\coloneqq\cup S_{i}$.
Note that $|\mathcal{S}_{\texttt{flat}}|=\sum_{i=1}^{N}L_{i}$.

Define an encoder network $f$ and projection head $g$, which are the
equivalent functions as defined in SimCLR (Chen et al., 2020a). Finally, we
define a (possibly infinite) set of random augmentation functions
$\mathcal{A}$, where each $\alpha\in\mathcal{A}$ is a function that applies a
pipeline of random transformations to generate views. We use the same base set
of transformations as Chen et al. (2020a) in order to construct $\mathcal{A}$.

###  3.1 SimCLR Baseline

Our baseline model is the standard SimCLR approach. A slice is randomly
sampled across all the slices from all scans:
$x\in\mathcal{S}_{\texttt{flat}}$. We generate views using two random
augmentation functions: $x_{1}=\alpha_{1}(x)$ and $x_{2}=\alpha_{2}(x)$ where
$\alpha_{1},\alpha_{2}\in\mathcal{A}$. These views are then sent through the
encoder and projection head to compute latent representations:
$z_{1}=g(f(x_{1}))$ and $z_{2}=g(f(x_{2}))$. The contrastive NT-Xent loss is
then computed using these representations. In this way, a positive pair of
views is always sourced from the same input slice. This precludes the ability
to model the 3D nature of the scans as the model never compares regions from
different slices within the same scan.

Instead, it attempts to pull the representations of different views of the
same slice together in latent space and push the representations of views from
different slices (_even if from different slices potentially within the same
scan_) apart. It is unclear apriori whether this assumption is detrimental to
transfer learning performance, however, it is the standard SSL contrastive
setup, and we make novel adaptations to it to build our two proposed variants:
_Per-scan SimCLR_ and _DeepSet SimCLR_.

###  3.2 Per-scan SimCLR

The intuition for our Per-scan SimCLR (PS-SimCLR) approach is that the
sampling strategy of the baseline precludes the model from accounting for the
3D nature of the scans. We address this by proposing a simple and effective
novel sampling strategy that attempts to implicitly encode the 3D nature of
the scans during the modelling process. In this way, PS-SimCLR changes how
random views are generated but keeps the network architecture consistent with
the baseline.

Our sampling strategy is controlled by a sampling width $\omega\in(0,1]$. This
parameter controls the width of a contiguous region (i.e. window) of the scan
$s_{i}\in X$ from which slices can be sampled, as well as the amount of shared
semantic and mutual information between random views. We randomly choose the
start location $j$ of this window from the range $\\{0,\cdots,\omega
L_{i}\\}$. The end location of the window is then given by $j+\omega L_{i}$.
The positive pair of random views is then generated by sampling two slices
from this window of the scan: $x,x^{\prime}\in\\{x_{j},\cdots,x_{j+\omega
L_{i}}\\}$. The positive view is pair is then defined as $x_{1}=\alpha_{1}(x)$
and $x_{2}=\alpha_{2}(x^{\prime})$. The rest of the PS-SimCLR pipeline is
identical to the baseline: we compute the NT-Xent loss from representations
$z_{1}=g(f(x_{1}))$ and $z_{2}=g(f(x_{2}))$. Note that we default to the
baseline strategy of generating views if $L_{i}$ is sufficiently small, i.e.
$L_{i}<T$ for some threshold $T$ (this happens infrequently).

In contrast to the baseline model, PS-SimCLR pulls representations of
different views of _any_ slice within the same scan together and contrasts
them against representations of slices from any other scan. In this way, we
implicitly model the 3D nature of the volume by imposing that views of the
same volume should be close in latent space, instead of only views of a single
slice of a scan needing to be close in latent space.

###  3.3 DeepSet SimCLR

Our DeepSet SimCLR (DS-SimCLR) variant adds a more explicit mechanism to model
the 3D nature of the scans in addition to the sampling strategy used in PS-
SimCLR. We achieve this by modelling a random view as a set of slices instead
of as an individual slice (as in PS-SimCLR and the baseline). Motivated by the
success of the Deep Set architecture for supervised WSI classification (Hemati
et al., 2021), we propose a novel combination of the Deep Set architecture and
SimCLR. See Figure 1 for a depiction of this architecture.

Similarly to PS-SimCLR, we use $\omega$ to construct the window of a scan
$S_{i}$ with start and end locations $j$ and $j+\omega L_{i}$. We sample $K$
equidistant images from this window: $\\{x^{(i)}_{l}\\}_{l=1}^{K}$. The
augmentation functions within $\mathcal{A}$ are constructed using the same
base transformations as in PS-SimCLR and the baseline, however, _the random
crop occurs at the same location for all $K$ sampled slices in the set_. This
is done to ensure semantic consistency of the slices in the set. We then
compute a random view $X_{1}$ of the scan:
$X_{1}\coloneqq\\{\alpha_{1}(x^{(i)}_{l})\\}_{l=1}^{K}$ for
$\alpha_{1}\in\mathcal{A}$. We construct the other random view $X_{2}$ using a
different random window of the scan. Note that for DS-SimCLR, random views are
defined as a set of transformed slices, unlike PS-SimCLR or the baseline where
views are individual transformed slices.

To compute latent representations for these set-based random views, we employ
the Deep Set architecture. We project each transformed slice of $X_{1}$ and
$X_{2}$ using $f$: $V_{1}=\\{f(x)|x\in X_{1}\\}_{l=1}^{K}$ (similarly for
$X_{2}$). Next, we sum the representations of the set’s slices and send them
through an MLP, formally (as per Zaheer et al. (2017)):

$\tilde{h}_{1}=g_{\texttt{DS}}\left(\sum_{j=1}^{K}V_{1,j}\right)$ (1)

where $g_{\texttt{DS}}$ is a function from the Deep Set architecture used to
compute the representation of a set $\tilde{h}_{1}$. We do the same for
$V_{2}$ to compute $\tilde{h}_{2}$. Finally, we compute the final latent
representations from which we compute the NT-Xent loss:
$z_{1}=g(\tilde{h}_{1})$ and $z_{2}=g(\tilde{h}_{2})$. Note that we experiment
with different parameterisations of $g_{\texttt{DS}}$, including the identity
function and an MLP.

The use of the DeepSet architecture means that slices of a scan are modelled
in a permutation invariant way since we are not modelling the scan’s 3D volume
as a sequence, but instead as a set. This means we trade off the ability to
model a scan as a sequence in favour of significantly lower computational
demand. Further, Deep Sets have previously been shown as a promising way of
modelling medical data (Hemati et al., 2021). Note that we still do indeed
incorporate the inherent 3D nature of the volume into our modelling process by
treating it as a set. We ensure that subvolumes (along the axial plane) of the
same scan are pulled together, and subvolumes from different scans are pushed
apart. Note that if $K=1$ and $g_{\texttt{DS}}=\texttt{id}$, our DS-SimCLR
model is equivalent to the baseline model.

Figure 1: DS-SimCLR architecture.

###  3.4 Implementation Details

####  3.4.1 Datasets

We pretrain on the NLST dataset, which consists of 191133 CT scans of human
lungs. We use a random subset of 2000 scans to align with the size of
pretraining datasets used in previous works (Nguyen et al., 2023) and for
computational efficiency reasons. For downstream evaluation, we use the PCAM,
CRC, CovidX-CT, and CheXpert (henceforth denoted as CX) datasets for the
linear evaluation task. We use the Kvasir Instruments and TBX11 datasets for
downstream fine-tuning for the semantic segmentation (SemSeg) and object
detection (OD) tasks, respectively. These datasets provide a variety of
pathology-related tasks to ensure a comprehensive evaluation of our pretrained
models.

####  3.4.2 Architecture

We parameterise $f$ as a ResNet50 (He et al., 2016) and $g$ as the standard
two-layer non-linear MLP in all cases. $g_{\texttt{DS}}$ is parameterised as a
one-layer non-linear MLP. Note that we experiment with an alternate
parameterisation of $g_{\texttt{DS}}$ as the identity function, denoted id.

####  3.4.3 Optimisation

In all cases, we train with an initial learning rate of $0.07$ (following the
learning rate formula from (Chen et al., 2020a)) and implement cosine learning
rate decay throughout training. We set the weight decay to $1e-10$, batch size
to $32$, and train for $100$ epochs using the Adam (Kingma and Ba, 2015)
optimiser.

####  3.4.4 Key Hyperparameters

We train with an image size of $224\times 224$ for the baseline and PS-SimCLR,
and $128\times 128$ for DS-SimCLR by default (to reduce computational
complexity). For PS-SimCLR and DS-SimCLR, we set $\omega=0.1$ and
$\omega=0.5$, respectively (unless otherwise specified). We set $T=5$ in PS-
SimCLR, and $K=3$ by default in DS-SimCLR. Note that we experiment with other
values for these hyperparameters in our analyses.

##  4 Experiments

###  4.1 Linear Evaluation on Downstream Datasets

Table 1: Linear evaluation results of the baseline and two proposed variants, PS-SimCLR and DS-SimCLR, for various classification datasets. | CRC | CovidX-CT | PCAM | CX (Atelectasis) | CX (Cardiomegaly) | CX (Edema)
---|---|---|---|---|---|---
Standard | 82.27 | 79.82 | 76.55 | 35.38 | 59.74 | 57.95
PS-SimCLR | 83.69 | 81.96 | 79.22 | 34.79 | 62.91 | 61.71
DS-SimCLR | 89.13 | 84.72 | 81.14 | 35.98 | 56.75 | 59.06

From Table 1 we can see that our two proposed variants - PS-SimCLR and DS-
SimCLR - consistently outperform the baseline for all linear evaluation
datasets, with DS-SimCLR outperforming both by a large margin on the CRC,
CovidX-CT, and PCAM datasets. For example, DS-SimCLR shows an 8.3% and 6.1%
increase in performance for the CRC and CovidX-CT datasets compared to the
baseline. Overall, DS-SimCLR shows a notable average relative increase in
performance of 3.1% across these datasets, and PS-SimCLR shows 3.0% for the
same metric. This suggests that implicitly encoding the 3D nature of medical
datasets during pretraining assists with downstream linear evaluation
performance, even for such tasks, which are not inherently 3D. This is also
particularly notable because DS-SimCLR is trained on lower resolution random
views than the baseline and PS-SimCLR in these experiments.

Our proposed methods are also fairly robust to the modality of the pathology
in the downstream task. Pretraining was performed using the NLST datasets,
which consists of 3D CT scans, and both variants perform well on downstream
linear evaluation tasks consisting of X-rays (CheXpert), RGB histopathologic
scans (CRC, PCAM), and CT scans (CovidX-CT). Interestingly, the results
suggest that PS-SimCLR performs better than DS-SimCLR on 2 of the 3 studied
CheXpert X-ray tasks. We posit that the stronger bias imposed upon DS-SimCLR
regarding modelling the 3D nature of the pretraining data compared to PS-
SimCLR’s means that performance is better on this particular inherently 2D
modality. It should be noted, however, that this does not reflect similarly in
the inherently 2D tasks of CRC and PCAM. Finally, we note our improved
performance within the linear evaluation paradigm suggests that the
representations learned from our proposed methods better encode the domain-
related information to make them more effective downstream as compared to the
baseline. We posit this is mainly due to the implicit 3D modelling of PS-
SimCLR’s sampling strategy and DS-SimCLR’s DeepSet component.

###  4.2 Fine-Tuning on Downstream Datasets

Table 2: Fine-tuning results of the baseline and two proposed variants, PS-SimCLR and DS-SimCLR, for the TBX11 OD task and Kvasir Instruments SemSeg task. | TBX11 | Kvasir Instruments
---|---|---
Standard | 23.04 | 78.99
PS-SimCLR | 23.1 | 80.12
DS-SimCLR | 23.37 | 81.03

Table 2 shows the results of fine-tuning for two important medical tasks: OD
and SemSeg. Note that in this setting, the encoder network $f$ changes its
weights during training. Similar to the linear evaluation results shown in
Table 1, both of our PS-SimCLR and DS-SimCLR proposed variants outperform the
standard SSL baseline. The performance difference for these tasks is less
pronounced as compared to linear evaluation. Apart from simply attributing
this to a smaller performance difference, another possible reason is the
sensitivity of the metric. The mAP and mIOU metrics are more sensitive to
slight perturbations in predictions than accuracy. To investigate this
further, we randomly sample 3 testing images and visualise their predicted
segmentation masks.

(a) Standard.

(b) PS-SimCLR.

(c) DS-SimCLR.

Figure 2: Visualisation of SemSeg fine-tuning results on the Kvasir
Instruments dataset for 3 randomly chosen test images. The three columns are
L-R: input image, ground-truth mask, and predicted mask.

Figure 2 visualises the SemSeg results for the 3 models. The predicted masks
for DS-SimCLR are notably more coherent and accurate than both the baseline
and PS-SimCLR. The standard model completely fails to detect the instrument in
two of these random images. Further, PS-SimCLR is consistently able to detect
that an instrument is present in the image, however, the localisations are not
as accurate as DS-SimCLR. These results suggest that both proposed variants,
and DS-SimCLR in particular, outperform the baseline in both sparse prediction
tasks (linear evaluation) and these dense prediction tasks.

###  4.3 Impact of Set Size and Sampling Width

Next, we perform experiments to analyse how the most important hyperparameters
introduced by two proposed methods affect performance. For PS-SimCLR, we focus
on the sampling width $\omega$. We note that for PS-SimCLR, $\omega$ has a
bigger impact on performance than $T$, and thus focus on it here. See Appendix
6 for more experiments around hyperparameter $T$. For DS-SimCLR, we also
analyse on sampling width $\omega$. Further, we include analyses of the set
size $K$, as this is a core hyperparameter for the DeepSet architecture within
our proposed framework.

####  4.3.1 Set Size

First, we analyse the impact of the set size $K$ on the performance of our DS-
SimCLR model. We analyse three equally-spaced values for $K$ while keeping
other hyperparameters constant. We use the values $K\in\\{1,3,5\\}$ to keep
the computational complexity comparable with the baseline. Results can be seen
in Table 3. When $K=5$, linear evaluation (i.e. the classification datasets)
performance is highest overall. However, the fine-tuning, dense prediction
tasks of TBX11 and Kvasir Instruments result in the best performance for
$K=3$. Interestingly, $K=1$ is consistently the worst-performing setting.

We hypothesise that the improved performance of the $K=3$ setting versus the
$K=5$ setting for the fine-tuning, dense prediction tasks is because dense
features may be harder to encode during learning when $K$ is larger since the
variance in the set grows accordingly. When the variance is larger, two very
distinct sets of images are encouraged to be similar through the use of the
contrastive loss. Thus, detailed semantic information is lost through the
aggregation of the distinct set of images. Similarly, when $K$ is too small
(i.e. $K=1$), we completely lose the benefit of the DeepSet and associated
image set comparison, and therefore performance degrades.

Table 3: Results of DS-SimCLR for various set sizes $K$. We set $\omega=0.5$, $D=64$, and $g_{\texttt{DS}}=\texttt{id}$ for these experiments. We sample three equidistant values for $K$. | CRC | CovidX-CT | PCAM | CX (Atelectasis) | CX (Cardiomegaly) | CX (Edema) | TBX11 | Kvasir
---|---|---|---|---|---|---|---|---
$K=1$ | 77.53 | 77.34 | 79.06 | 34.27 | 54.87 | 53.93 | 20.53 | 75.28
$K=3$ | 88.15 | 84.6 | 80.83 | 34.7 | 60.09 | 57.61 | 22.19 | 78.56
$K=5$ | 89.26 | 83.78 | 78.83 | 36.32 | 58.72 | 59.32 | 20.39 | 75.52

####  4.3.2 Sampling Width

The sampling width $\omega$ essentially controls the width of the contiguous
window of the scan from which we can sample slices for PS-SimCLR and DS-
SimCLR. Similar to the set size analysis, we vary $\omega$ while keeping all
other hyperparameters constant. We test a small, medium, and large value for
$\omega$ in both cases. Results for both variants can be seen in Table 4. We
see that for PS-SimCLR, $\omega=0.1$ results in the best performance overall,
with that setting having the best performance in $6/8$ of the downstream
tasks. In these two other tasks, the performance difference is small. We posit
that this is because as $\omega\rightarrow 1$, the mutual information between
the two sampled slices decreases. As the mutual information decreases, the
semantic consistency between the two images decreases, which makes the
contrastive task harder to solve.

For DS-SimCLR, we note that overall, $\omega=0.5$ performs best (i.e. for
$4/8$ tasks), however, there are 2 tasks each where $\omega=0.8$ and
$\omega=0.2$ perform best. The reason there is no clear winner is because
$\omega$ has less of an effect on the representations learned by the DeepSet
architecture compared with the set size $K$ due to the equidistant sampling
utilised to create the set. One can still achieve a balance of image variety
and mutual information by sampling more images within the same sampling width
region. It is clear from these results that $\omega=0.5$ provides a good
balance of this mutual information within the set such that useful
representations can be learned during pretraining for a variety of downstream
tasks.

Table 4: Results of PS-SimCLR and DS-SimCLR for various sampling widths. For
PS-SimCLR, we set $T=5$. For DS-SimCLR, we set $K=3$, $D=64$, and
$g_{\texttt{DS}}=\texttt{id}$. We sample three equidistant values for $\omega$
for each model.

|  | CRC | CovidX-CT | PCAM | CX (Atelectasis) | CX (Cardiomegaly) | CX (Edema) | TBX11 | Kvasir
---|---|---|---|---|---|---|---|---|---
PS-SimCLR | $\omega=0.1$ | 83.69 | 81.96 | 79.22 | 34.79 | 62.91 | 61.71 | 23.1 | 80.12
$\omega=0.4$ | 79.72 | 78.26 | 76.79 | 34.96 | 61.03 | 54.7 | 22.32 | 79.64
$\omega=0.7$ | 79.55 | 78.43 | 77.63 | 35.3 | 63.59 | 58.72 | 22.82 | 78.24
DS-SimCLR | $\omega=0.8$ | 88.79 | 84.59 | 79.93 | 35.38 | 56.15 | 57.35 | 21.9 | 76.23
$\omega=0.5$ | 89.26 | 83.78 | 78.83 | 36.32 | 58.72 | 59.32 | 20.39 | 75.52
$\omega=0.2$ | 88.75 | 85.13 | 79.61 | 34.62 | 56.15 | 58.03 | 21.12 | 78.32

###  4.4 Analysis of $g_{\texttt{DS}}$

Next, we analyse the effect of different parameterisations of
$g_{\texttt{DS}}$ in Equation 1. If we were to make $g_{\texttt{DS}}$ the
identity function (i.e. $g_{\texttt{DS}}(x)=x$ $\forall x$), then we would
leave it to the projection head $g$ to serve as the DeepSet’s proxy MLP. This
slightly reduces computational overhead by negating the need for a forward
pass through a small MLP. Results for both the identity and MLP
parameterisations of $g_{\texttt{DS}}$ can be seen in Table 5.

The results suggest that for certain parameterisations of DS-SimCLR, there is
a clear performance increase, on average, when an MLP is used (such as in
$\texttt{DS-SimCLR}(3,128,0.5)$ and $\texttt{DS-SimCLR}(2,224,0.7)$). However,
for different parameterisations, the identity function results in superior
performance, such as in $\texttt{DS-SimCLR}(5,64,0.5)$ and $\texttt{DS-
SimCLR}(5,64,0.8)$. These results suggest that the choice of $g_{\texttt{DS}}$
is heavily influenced by the choice of the remaining parameters $K$ $D$, and
$\omega$. Interestingly, for the SemSeg downstream task (Kvasir), the best
performance is always achieved when an MLP is used.

Moreover, if we aggregate the results by dataset based on $g_{\texttt{DS}}$’s
parameterisation (see Table 7 in the Appendix), we see that for the majority
of downstream tasks, an MLP performs best on average. This suggests that
decoupling the DeepSet’s MLP from the projection head, and thus from the
SimCLR architecture, results in better performance. However, we reiterate that
this is largely dependent on the hyperparameter choices.

Table 5: Results for the inclusion or exclusion of an MLP in the DeepSet
formulation (see Equation 1). Note that in $\texttt{DS-SimCLR}(K,D,\omega)$,
$K$ is the set size, $D$ is the image size (height and width), and $\omega$ is
the sampling width.

| $g_{\texttt{DS}}$ | CRC | CovidX-CT | PCAM | CX (Atelectasis) | CX (Cardiomegaly) | CX (Edema) | TBX11 | Kvasir
---|---|---|---|---|---|---|---|---|---
DS(5, 64, 0.5) | id | 89.26 | 83.78 | 78.83 | 36.32 | 58.72 | 59.32 | 20.39 | 75.52
DS(5, 64, 0.5) | MLP | 88.82 | 84.5 | 78.79 | 34.79 | 52.48 | 57.09 | 20.32 | 79.07
DS(5, 64, 0.8) | id | 88.79 | 84.59 | 79.93 | 35.38 | 56.15 | 57.35 | 21.9 | 76.23
DS(5, 64, 0.8) | MLP | 88.92 | 85.56 | 80.55 | 34.7 | 60.68 | 55.81 | 21.09 | 77.28
DS(3, 128, 0.5) | id | 88.53 | 84.07 | 81.23 | 35.38 | 58.55 | 57.09 | 22.47 | 79.27
DS(3, 128, 0.5) | MLP | 89.13 | 84.72 | 81.14 | 35.98 | 56.75 | 59.06 | 23.37 | 81.03
DS(2, 224, 0.7) | id | 83.08 | 82.91 | 78.58 | 35.13 | 62.05 | 56.5 | 20.4 | 75.73
DS(2, 224, 0.7) | MLP | 82.82 | 81.61 | 79.35 | 35.04 | 64.62 | 60.0 | 22.65 | 78.17

##  5 Conclusion

We choose to model the 3D volumes as a set using the DeepSet since this
architecture has proven to work well in supervised medical settings. Our
integration of the DeepSet into an SS architecture (DS-SimCLR) has multiple
benefits, including implicitly modelling the 3D nature of the data while
notably improving performance for little additional overhead. In this way, we
contrast against existing approaches which are either inherently 3D
architectures or use complicated transformer-based mechanisms, both of which
are significantly more computationally expensive to train than our proposed
methods. Further, our simple sampling strategy proposed for PS-SimCLR improves
upon the baseline for no additional training overhead.

## References

  * Chen et al. [2020a] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _Proceedings of the 37th International Conference on Machine Learning_ , pages 1597–1607, 2020a.
  * Chen et al. [2020b] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton.  Big self-supervised models are strong semi-supervised learners.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , 2020b.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the International Conference on Computer Vision_ , 2021.
  * Chen et al. [2020c] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  _arXiv preprint arXiv:2003.04297_ , 2020c.
  * Ericsson et al. [2020] Linus Ericsson, Henry G. R. Gouk, and Timothy M. Hospedales.  How well do self-supervised models transfer?  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5410–5419, 2020.
  * Zbontar et al. [2021] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In _Proceedings of the 38th International Conference on Machine Learning_ , pages 12310–12320, 2021.
  * Grill et al. [2020] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 33:21271–21284, 2020.
  * Caron et al. [2020] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in neural information processing systems_ , 33:9912–9924, 2020.
  * Zaheer et al. [2017] Manzil Zaheer, Satwik Kottur, Siamak Ravanbhakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J Smola.  Deep sets.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , 2017.
  * Hemati et al. [2021] Sobhan Hemati, Shivam Kalra, Cameron Meaney, Morteza Babaie, Ali Ghodsi, and Hamid Tizhoosh.  CNN and deep sets for end-to-end whole slide image representation learning.  In _Proceedings of the Fourth Conference on Medical Imaging with Deep Learning_ , 2021.
  * Taleb et al. [2020] Aiham Taleb, Winfried Loetzsch, Noel Danz, Julius Severin, Thomas Gaertner, Benjamin Bergner, and Christoph Lippert.  3d self-supervised methods for medical imaging.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Tang et al. [2022] Yucheng Tang, Dong Yang, Wenqi Li, Holger R. Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh.  Self-supervised pre-training of swin transformers for 3d medical image analysis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Nguyen et al. [2023] Duy M. H. Nguyen, Hoang Nguyen, Truong T. N. Mai, Tri Cao, Binh T. Nguyen, Nhat Ho, Paul Swoboda, Shadi Albarqouni, Pengtao Xie, and Daniel Sonntag.  Joint self-supervised image-volume representation learning with intra-inter contrastive clustering.  In _Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence_ , 2023.
  * Dwibedi et al. [2021] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman.  With a little help from my friends: Nearest-neighbor contrastive learning of visual representations.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Zhang et al. [2022] Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X. Pham, Chang D. Yoo, and In So Kweon.  How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=bwq6O4Cwdl.
  * Torpey and Klein [2022] David Torpey and Richard Klein.  On the robustness of self-supervised representations for multi-view object classification.  _Pattern Recognition Letters_ , 161:82–89, 2022.  ISSN 0167-8655.  doi:https://doi.org/10.1016/j.patrec.2022.07.016.
  * Jing et al. [2022] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian.  Understanding dimensional collapse in contrastive self-supervised learning.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=YevsQ05DEN7.
  * Gidaris et al. [2018] Spyros Gidaris, Praveer Singh, and N. Komodakis.  Unsupervised representation learning by predicting image rotations.  2018\.
  * Zhang et al. [2016] Richard Zhang, Phillip Isola, and Alexei A Efros.  Colorful image colorization.  In _ECCV_ , 2016.
  * Pathak et al. [2016] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2536–2544, 2016.
  * Noroozi and Favaro [2016] M. Noroozi and P. Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In _ECCV_ , 2016.
  * Kang et al. [2023] Mingu Kang, Heon Song, Seonwook Park, Donggeun Yoo, and Sérgio Pereira.  Benchmarking self-supervised learning on diverse pathology datasets.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * van den Oord et al. [2018] Aäron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _ArXiv_ , abs/1807.03748, 2018.
  * Doersch et al. [2015] Carl Doersch, Abhinav Gupta, and Alexei A. Efros.  Unsupervised visual representation learning by context prediction.  In _IEEE International Conference on Computer Vision_ , pages 1422–1430, 2015.
  * Dosovitskiy et al. [2014] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox.  Discriminative unsupervised feature learning with convolutional neural networks.  In _Advances in Neural Information Processing Systems_ , 2014.
  * Kolesnikov et al. [2021] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.
  * Caron et al. [2019] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin.  Unsupervised pre-training of image features on non-curated data.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2019.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , 2017.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2016.
  * Kingma and Ba [2015] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_ , 2015.

##  Appendix A Additional PS-SimCLR Results

Table 6: Additional results for different settings for $T$ for PS-SimCLR. It
is clear that in both the $\omega=0.1$ and $\omega=0.7$ cases, there is no
clear trend as to which value of $T$ is best. This suggests that the method,
PS-SimCLR, is fairly robust to the choice of this parameter. However, we
recommend the value be kept fairly small relative to the average number of
slices in a scan for a particular dataset. This ensures that we do not revert
to the baseline sampling during training often.

|  | CRC | CovidX-CT | PCAM | CX (Atelectasis) | CX (Cardiomegaly) | CX (Edema) | TBX11 | Kvasir
---|---|---|---|---|---|---|---|---|---
$\omega=0.1$ | $T=5$ | 83.69 | 81.96 | 79.22 | 34.79 | 62.91 | 61.71 | 23.1 | 80.12
$T=50$ | 82.88 | 80.45 | 78.69 | 36.58 | 58.46 | 61.54 | 22.79 | 79.07
$\omega=0.7$ | $T=5$ | 79.55 | 78.43 | 77.63 | 35.3 | 63.59 | 58.72 | 22.82 | 78.24
$T=50$ | 79.96 | 76.58 | 78.04 | 35.13 | 66.07 | 57.95 | 21.88 | 78.38

##  Appendix B Aggregated $g_{\texttt{DS}}$ Results

Table 7: Aggregated results of different parameterisations of
$g_{\texttt{DS}}$.

| CRC | CovidX-CT | PCAM | CX (Atelectasis) | CX (Cardiomegaly) | CX (Edema) | TBX11 | Kvasir
---|---|---|---|---|---|---|---|---
id | 87.42 | 83.84 | 79.64 | 35.55 | 58.87 | 57.56 | 21.29 | 76.69
MLP | 87.42 | 84.1 | 79.96 | 35.13 | 58.63 | 57.99 | 21.86 | 78.89