# Anomalously small superconducting gap in the strong spin-orbit coupled
superconductor: $\beta-$Tungsten

Prashant Chauhan  Department of Physics and Astronomy, The Johns Hopkins
University, 3701 San Martin Dr., Baltimore, Maryland 21218, USA  Ramesh
Budhani  Department of Physics, Morgan State University, 1700E Cold Spring
Lane, Baltimore, Maryland 21251, USA  N. P. Armitage  Department of Physics
and Astronomy, The Johns Hopkins University, 3701 San Martin Dr., Baltimore,
Maryland 21218, USA

###### Abstract

Thin films of $\beta-$tungsten host superconductivity in the presence of
strong spin-orbit coupling. This non-equilibrium crystalline phase of tungsten
has attracted considerable attention in recent years due to its giant spin
Hall effect and the potential promise of exotic superconductivity. However,
more than 60 years after its discovery, superconductivity in this material is
still not well understood. Using time-domain THz spectroscopy, we measure the
frequency response of the complex optical conductivity of $\beta-$tungsten
thin film with a Tc of 3.7 K in its superconducting state. At temperatures
down to 1.6 K, we find that both the superconducting gap and the superfluid
spectral weight are much smaller than that expected for a weakly coupled
superconductor given the Tc. The conclusion of a small gap holds up even when
accounting for possible inhomogeneities in the system, which could come from
other crystalline forms of tungsten (that are not superconducting at these
temperatures) or surface states on $\beta-$tungsten grains. Using detailed
X-ray diffraction measurements, we preclude the possibility of significant
amount of other tungsten allotropes, strongly suggesting the topological
surface states of $\beta-$tungsten play the role of inhomogeneity in these
films. Our observations pose a challenge and opportunity for a theory of
strongly anisotropic normal metals with strong spin-orbit coupling to
describe.

Bulk crystalline tungsten (W) in the most stable $\alpha-$W bcc form has a
very low superconducting Tc of 11 mK Gibson and Hein (1964). However, thin
films of W can have Tc’s as large as 5 K e.g. two orders of magnitude higher
than the bulk. This has been attributed to the presence of a metastable A15
$\beta$ phase structure that can be stabilized in thin films Basavaiah and
Pollack (1968); Hofer and Haberkorn (2019). Such $\beta-$W has very
distinctive mechanical, electrical and optical properties. Its room
temperature resistivity ($\sim$$200\text{\,}\mu\Omega\mathrm{c}\mathrm{m}$) is
much higher than that of $\alpha-$W
($\sim$$20\text{\,}\mu\Omega\mathrm{c}\mathrm{m}$) Hao  _et al._ (2015). Due
to strong spin orbit coupling (SOC), $\beta-$W exhibits a giant spin Hall
effect with large spin Hall angle ($\theta_{SH}\sim-0.45$), making it
potentially useful in spintronic applicationsMondal  _et al._ (2017); Demasius
_et al._ (2016). DFT calculations show that $\beta-$W may be a Dirac system
that hosts massive Dirac fermions and may host surface arc states with novel
spin textures reminiscent of those in topological insulators Li  _et al._
(2019); Xie  _et al._ (2018). The helical spin-polarized electrons in
topological insulators can host exotic excitations like Majorana fermions if
their topological surface states become superconducting by proximity effect
Mellnik  _et al._ (2014). These facts prompt us to investigate the possibility
of exotic bulk and surface superconductivity in $\beta-$W.

The study of superconductivity in $\beta-$W thin films has proven to be
challenging. There have been only a few experimental and theoretical
investigations Gibson and Hein (1964); Bond  _et al._ (1965); Basavaiah and
Pollack (1968) that have given understanding of its superconducting state and
order parameter. This is mainly because of the difficulty in growing clean
$\beta-$W thin films without impurity phases like $\alpha-$W and the
instability of this metastable phase at room temperature where it can
spontaneously transform into the $\alpha-$W phase Xiao  _et al._ (2013).
However, there has been a long-term (and now renewed) interest in both the
metallic state and the superconductivity of $\beta-$W. The superconductivity
exhibits a number of unusual aspects. Using tunneling, Basavaiah et al showed
that the temperature dependence of the superconducting energy gap $\Delta(T)$
follows the Bardeen-Cooper-Schrieffer (BCS) dependence Basavaiah and Pollack
(1968), but with a reduced ratio $\alpha=\Delta/k_{\mathrm{B}}$T${}_{c}\sim
1.1-1.8$, (where $\Delta$ is the superconducting energy gap) as opposed to the
universal weak coupling BCS value of $\alpha_{\mathrm{BCS}}\sim 1.76$
Basavaiah and Pollack (1968); Bardeen  _et al._ (1957). It was suggested that
the reduced gap could be related to the presence of $\alpha-$W phase Basavaiah
and Pollack (1968). We shall revisit this possibillity below.

In general, the $\alpha=\Delta/k_{\mathrm{B}}$Tc ratio provides important
insight into the physics. In a clean system, the coupling strength of
superconductivity can be defined relative to $\alpha_{\mathrm{BCS}}$, with
$\alpha\simeq\alpha_{\mathrm{BCS}}$ for a weakly coupled superconductor and
$\alpha$ somewhat greater than $\alpha_{\mathrm{BCS}}$ for a strongly coupled
superconductor. In disordered systems, pair breaking tends to decrease Tc
faster than it does the $\Delta$, which increases the ratio above
$\alpha_{\mathrm{BCS}}$ Khasanov and Mazin (2021). Thus, in general we expect
$\alpha\geq\alpha_{\mathrm{BCS}}$ Khasanov and Mazin (2021), which is in
contrast to observations in $\beta-$W Basavaiah and Pollack (1968). However,
in such considerations, there is an implicit assumption that the order
parameter is uniform both in space and in momentum. The order parameter can be
anisotropic in momentum space and the gap on the Fermi surface
$\langle\alpha\rangle$ can be at points smaller than $\alpha_{\mathrm{BCS}}$
Khasanov and Mazin (2021). Due to a complicated and non-uniform Fermi surface
Li  _et al._ (2019), we expect $\beta-$W to possesses an anisotropic
superconducting gap Carbotte (1977). However, the question is to what degree?
Real space inhomogeneity can of course create regions where the gap is
suppressed and this may also be an effect.

Figure 1: (Top) Zero-field magnitude of transmission, (Bottom) Real
$G_{1}(\nu)$ and imaginary $G_{2}(\nu)$ parts of the zero-field complex
conductance of $\beta$-W 70 nm thin film grown on Si substrate as a function
of frequency from room temperature (T$\gg$Tc) to
($$1.6\text{\,}\mathrm{K}$\ll$Tc). Inset: Temperature dependence of the
4-probe dc resistance. Figure 2:  Real $G_{1}(\nu)$ and imaginary $G_{2}(\nu)$
parts of the zero-field complex conductance of $\beta$-W 70 nm thin film,
normalized to the normal-state conductance $G_{n}$. The solid lines are
experimental data normalized w.r.t the conductance at 5 K. The dashed lines
are fit to the data using Mattis-Bardeen theory for BCS superconductor. Inset:
Black squares show the temperature dependence of the superfluid spectral
weight $S_{\delta}(T)$ determined from lim${}_{\nu\rightarrow 0}\nu
G_{2}/G_{n}$. The blue line shows the expected spectral weight for a weakly
coupled BCS superconductor ($\alpha_{BCS}=1.76$). The purple line shows the
expected spectral weight for a BCS-like superconductor but with the
experimentally determined $\Delta(0)=0.32$ meV that gives $\alpha=1$.

Here we use high precision time-domain THz spectroscopy (TDTS) to measure the
low-energy complex optical conductivity of $\beta-$tungsten films as a
function of temperature. We systematically study and track the superconducting
gap, $\Delta(T)$, as a function of temperature. We find that the
superconducting energy gap and the superfluid density can be described
phenomenologically in terms of BCS theory, however, with a small energy gap
parameter. This confirms a value of $\alpha$ much smaller than the 1.76 than
expected for a weakly coupled BCS superconductor. The conclusion of a small
gap holds up even when using extant effective medium models that account for
inhomogeneity in the form of normal metal inclusions. Such inclusions could
come from other crystalline forms of tungsten (not superconducting at these
temperatures) or surface states on $\beta-$Tungsten grains. Our observations
pose a challenge and opportunity for a theory of strongly anisotropic normal
metals with strong spin-orbit coupling to describe.

Thin films of the $\beta-$Tungsten (A15 structure) were grown on 0.5 mm thin
Si(001) with $\sim 200$ nm thick thermal oxide and 0.5 mm MgO(100) substrates
using sputtering in presence of N2:Ar gas mixture at room temperature. The Ar
sputtering pressure was kept at $5$ mTorr during the deposition. The fact that
$\beta-$W thin films require N2:Ar or O2:Ar mixture assisted growth has made
it difficult to tune the quality and Tc of the superconducting film. TDTS
measurements were performed in transmission geometry on thin films of varying
thicknesses ranging from 70 - 130 nm. The 70 nm $\beta-$W thin-films had the
highest Tc’s of $\sim 3.7$ K. The samples on both Si and MgO substrates gave
similar results (See Supplementary MaterialSup ; Garnett (1904); Xi  _et al._
(2013); Carr  _et al._ (1985); Granqvist and Hunderi (1977); Sihvola (2007)
(SM)). Both real and imaginary parts of the complex conductance,
$\tilde{G}(\nu)$, were extracted from the c omplex transmission measured in
TDTS measurements, performed down to 1.6 K Chauhan  _et al._ (2019).

Fig. 1 shows the frequency dependent transmission and complex conductivity
$\tilde{G}(\nu)$ of a 70 nm $\beta-$W film, between $0.2-2$ THz in zero
magnetic field at a few different temperatures above and below Tc. For T$>$Tc,
the real part of the conductance, $G_{1}(\nu)$ is constant in both this
frequency and temperature range with RRR $\sim 1$. In the normal state,
$G_{2}(\nu)$ is zero in correspondence with the large scattering rate ($\gg$10
THz). The THz conductance in the normal state $0.03$ $\Omega^{-1}$ matches
well with DC conductance measurement, $G_{dc}=0.028$ $\Omega^{-1}$. Below Tc
both $G_{1}(\nu)$ and $G_{2}(\nu)$ shows features indicative of the opening of
a superconducting energy gap. As the temperature falls below Tc, a depletion
develops in $G_{1}(\nu)$ at low $\nu$, corresponding to shift of
superconducting carrier spectral weight to the zero frequency delta function
Palmer and Tinkham (1968). However, the depletion is not large and even at the
lowest temperatures of 1.6 K, the conductance remains $\sim 66\%$ of the
normal state at $0.2$ THz. For a weak coupling BCS superconductor with
T${}_{c}\sim 3.7$ K, one expects an optical $2\Delta$ gap of about 0.27 THz at
this temperature. $G_{1}(\nu)$ shows a small upturn below 0.2 THz. As
$\nu\rightarrow 0$, $G_{2}(\nu)$ shows $1/\nu$-like dependence at the lowest
temperatures, characteristic of the superconducting state.

To understand the superconducting state better and determine the energy gap
$\Delta(T)$, we simultaneously fit the normalized $G_{1}(\nu)$ for all
temperatures to the Mattis-Bardeen (MB) theory Mattis and Bardeen (1958);
Chauhan  _et al._ (2019); Tinkham (2004). Normalizing the low temperature
conductance with the normal state conductance, $G_{1}(5\text{K})$ eliminates a
number of the systematic errors in the transmission data and reduces the
number of fitting parameters. For the fitting procedure, the only free
parameter is the zero temperature superconducting gap, $\Delta(0)$. The result
of the fits as well as the normalized conductance data are shown as dashed and
solid lines in Fig. 2. The upturn in $G_{1}(\nu)$ at 0.2 THz becomes more
apparent after normalizing the data. The global fit to the real part of the
normalized conductance for all temperatures gives $\Delta(0)=0.32$ meV, which
is similar to values obtained from tunneling spectroscopy (0.31 - 0.52 meV for
films with Tc ranging from 3.1 K to 3.3 K) Basavaiah and Pollack (1968). There
is close agreement between the MB fits and $G_{1}(\nu)$. In contrast, the
correspondence with the imaginary part using the same parameters as the real
part gives poor agreement at low frequencies [Fig. 2]. Nevertheless, the
temperature evolution of the superconducting gap $\Delta(T)$, follows standard
BCS $\Delta(T)=\Delta(0)$tanh$[1.74\sqrt{\mathrm{T}_{c}/\mathrm{T}-1}]$, given
for a weakly coupled BCS superconductor. However the extracted gap
$2\Delta(0)=0.15$ THz (0.64 meV) or
$\alpha=\Delta(0)/k_{\mathrm{B}}$T${}_{c}\sim 1$ is much less than the weak
coupling limit of 1.76 for a fully gapped BCS superconductor.

To get further insight into the superconducting gap and confirm the Mattis-
Bardeen fits, we study the temperature dependence of the superfluid spectral
weight, $S_{\delta}(T)$, as a measure of superfluid density $n_{s}$. We
calculate $S_{\delta,G_{2}}=\text{lim}_{\nu\rightarrow 0}\nu G_{2}/G_{n}$
which is a measure of superfluid spectral weight determined directly from the
TDTS experimental data. We can compare it to the value calculated
$S_{\delta,\text{MB}}$ for $\alpha=1$ and $\alpha_{\text{BCS}}=1.76$ using the
approximation for a fully gapped BCS superconductor given by
$S_{\delta}(T)=\textstyle\frac{S_{\delta}(0)\Delta(T)}{\Delta(0)}\tanh[\Delta(T)/2k_{\mathrm{B}}T]$
Tinkham (2004). The normalization constant $S_{\delta}(0)$ is given by the
Ferrell-Glover-Tinkham (FGT) sum rule, $S_{\delta}(0)=S_{n}-S_{qp}(0)$, where
$S_{n}$ is the total spectral weight in normal state and $S_{qp}(0)$ is the
above gap spectral weight at $T\sim 0$ as given by the MB theory Chauhan  _et
al._ (2019). In Fig. 2(inset) we compare the temperature evolution of
$S_{\delta,G_{2}}$ with both $S_{\delta,\text{MB}}$ for $\alpha=1$ and
$\alpha_{\text{BCS}}=1.76$. Near Tc, the curve for $\alpha=1$ is close to the
experimental value, but over estimates the spectral weight found in the delta
function. Thus we conclude that the Mattis-Bardeen fits below 0.2 THz do not
match the actual conductance $G_{1}(\nu)$ and $S_{\delta,\text{MB}}(T)$ at low
temperatures. It is likely that there is a subgap conductance coming from a
contribution other than the ones considered in MB theory. Hence, as compared
to a weakly coupled BCS superconductor, $\beta-$W has a much lower superfluid
density.

In order to qualitatively understand the origin of the low superfluid spectral
weight in comparison to the MB theory prediction, we compare the experimental
data phenomenologically with the individual components of the MB response
function Mattis and Bardeen (1958); Zimmermann  _et al._ (1991); Dressel
(2013). There are both thermally and photon excited contributions to the total
optical response in superconductors as shown in Fig. 3. Within the BCS
framework, the low energy response of $G_{1}(\nu<2\Delta$) is only from the
thermal excitations, whereas the higher energy response above $2\Delta$ is
dominated by the photoexcitations (breaking of Cooper pairs due to photon
absorption). For $G_{2}(\nu)$, most of the response in our spectral range is
from the superfluid with a small negative contribution coming from the thermal
and photon excitations, which decrease with increasing energy. On comparing
the measured conductance of $\beta-$W in Fig. 2 with the individual components
contributing to the optical response shown in Fig. 3, we find that the
negative thermal contribution to $G_{2}(\nu)$ cannot account for the smaller
than expected $G_{2}(\nu)$. It is both too small in magnitude and of course
disappears in the limit of low temperature. It appears that there must be some
residual low frequency metal-like conduction that has the effect of giving a
smaller contribution to the lowest frequency $G_{2}(\nu)$ than the same
spectral weight in the superconducting delta function would.

Figure 3:  Frequency dependent real $G_{1}(\nu)$ and imaginary $G_{2}(\nu)$
conductance of a superconductor at T/T${}_{c}=0.6$. The red shaded region
indicate the thermally excited contribution. In $G_{1}(\nu)$, the Green line
indicates the photon excited response and blue shaded region corresponds to
the total response. In addition to these contributions there is a zero
frequency delta function in $G_{1}$ and its very large $1/\nu$ contribution to
$G_{2}$.

One explanation for this low $n_{s}$, a larger than expected response in
$G_{1}(\nu)$, smaller fitted gap, and anomalous sub gap absorption could be
that the superconductor is inhomogeneous and that there are parts of the film
which are not superconducting even at the lowest measured temperature. This
would have the effect of making the apparent $\Delta/k_{\mathrm{B}}$Tc ratio
from the MB fits smaller than 1.76. Such inhomogeneity could stem from the
presence of $\alpha-$W, which only becomes superconducting at much lower T.
$\alpha-$W can either form directly during deposition or transform from
$\beta-$W due to its unstable nature Basavaiah and Pollack (1968); Xiao  _et
al._ (2013). Another and surely more exciting possibility is the presence of
topological surface states on the exterior of $\beta-W$ grains, which could
act as sources of dissipation even when the bulk of the grains becomes
superconducting Li  _et al._ (2019). In order to confirm the presence of
inhomogeneity, we analyze the system in terms of effective medium models. We
compare the conductivity with calculations based on two different effective
medium models, namely the Bruggeman-effective-medium-approximation (BEMA)
Bruggeman (1935) and Maxwell-Garnett theory (MGT) Garnett (1904), for the
collective response of a mixture of two materials Chauhan  _et al._ (2019); Xi
_et al._ (2013); Stroud (1975). See SMSup  for details of these models. The
BEMA treats all constituents equivalently, and is thus appropriate for
mixtures with connected grains. In contrast, the MGT treats one constituent as
the host and others as embedded media making it more suitable for mixtures
with isolated inclusions Xi  _et al._ (2013).

For the BEMA model, we consider an inhomogeneous two component medium of
normal Drude-metal and superconductor that have volume fractions $f$ and
$1-f$. Similar to the analysis above, we fit the normalized $\tilde{G}(\nu)$
using the MB theory for the superconducting fraction and the normalized
$G_{n}(\nu)=1$ for the normal fraction. Here the free parameters are $f$, and
$\Delta(0)$. We obtain reasonable fits for $G_{1}(\nu)$ with a slightly larger
energy gap than in the MB homogeneous case of $\Delta(0)=0.42(2)$ meV (0.10
THz) and normal volume fraction $f=0.30(3)$ as shown in Fig. 4(a-b). The
spectral gap $2\Delta(0)$ fit in this fashion increases only by 33$\%$ giving
$\alpha_{\text{BEMA}}=1.32$ which is somewhat closer to, but still less than
the weakly coupled BCS superconductor value of 1.76. The BEMA fits for
$G_{2}(\nu)$ match even better to the data as compared to MB theory fits in
Fig. 2(b). Although there are still discrepancies, they appear to be
converging towards the data at frequencies below 0.2 THz. In order to confirm
the fits we determine the superfluid spectral weights of $G_{1}(\nu)$ from
BEMA $S_{\delta,\text{BEMA}}$ using FGT sum rule and compare it to
$S_{\delta,G_{2}}(T)$ in Fig. 5. Unlike the overestimated superfluid spectral
weights obtained from MB theory, $S_{\delta,\text{BEMA}}$ falls only slightly
below the BCS prediction and $S_{\delta,G_{2}}$.

Figure 4:  Zero-field normalized real $G_{1}(\nu)$ and imaginary $G_{2}(\nu)$
conductance of 70 nm $\beta-$W thin film shown with solid lines. (a)-(b)
Effective optical conductance calculated from BEMA (dashed lines) with normal
volume fraction $f=0.3$. (c)-(d) Effective optical conductance calculated from
MGT (dashed lines) with normal volume fraction $f=0.3$.

For the MGT model the superconducting component is taken as the host medium
and the normal volume fraction $f$ is taken as the embedded media. We fit
$\tilde{G}(\nu)$ using MB theory for the superconducting medium, taking the
energy gap $\Delta(0)$ and normal volume fraction $f$ as the only free
parameters. Fig. 4(c-d) shows that the MGT fits are consistent with
$G_{1}(\nu)$ for $f=0.3$ and energy gap $\Delta(0)=0.42$ meV (0.1 THz), or
$\alpha_{\text{MGT}}=1.32$ ($<\alpha_{\text{BCS}}$) to within errors giving
the same parameters as BEMA fits. As shown in Fig. 5, the superfluid spectral
weight $S_{\delta,\text{MGT}}$ extracted from MGT $G_{1}(\nu)$ is now just
slightly below the experimental $S_{\delta,G_{2}}$. This indicates that the
MGT $G_{2}$ also converges to near the experimental data at frequencies lower
than 0.2 THz.

Figure 5:  Temperature dependent superfluid spectral weight, $S_{\delta}(T)$.
The black squares show lim${}_{\nu\rightarrow 0}\nu G_{2}/G_{n}$. Solid lines
are superfluid spectral weight calculated for MB theory for $\alpha=1$ and
1.76. Dashed lines are the superfluid spectral weights calculated for MGT and
BEMA for a normal volume fraction of $f=0.3$.

These fits with effective medium models show that although there can be a low
frequency absorption coming from inhomogeneity, this does not completely
explain the extracted small gaps in these systems. Therefore one can take this
as an intrinsic feature of the superconducting state of $\beta-$W. One
possibility is that the gap is strongly anisotropic in momentum space.
Although gap anisotropy in s-wave superconductors is believed to largely
depends on phonon spectrum anistoropy and not on Fermi surface anisotropies
Bennett (1965); Tomlinson and Carbotte (1976); Crabtree  _et al._ (1987), this
issue has not been investigated for the strongly anisotropic Fermi surface of
tungsten. Moreover, calculations taking into account strong-spin orbit
coupling in a system like tungsten with its strongly anisotropic Fermi surface
Mattheiss (1965) have not been done. This is an area for future investigation.

What is the source of these metallic regions in the film? First, we would like
to discuss the possibility that the other tungsten allotrope $\alpha-$W could
be acting as normal metal inclusions embedded in the superconducting
$\beta-$W. It is generally known that depending on the growth conditions and
thickness of the thin films, tungsten grows in bcc phase, A15 phase, mixed
phases or in amorphous phase Shen  _et al._ (2000); Basavaiah and Pollack
(1968); Hofer and Haberkorn (2019). $\theta-2\theta$ X-ray diffraction
measurements on our W thin films (see Fig. 6) indicates a pure $\beta-$W phase
with no $\alpha-$W peaks observed within our instrumental uncertainty. We
believe that, $\alpha-W$ even if present must be much less than $10\%$
implying that the normal volume fraction of $f=0.3$ obtained from the
effective medium models cannot be explained by normal metal inclusions of bcc
W phase. Further, all the W peaks are sharp
($\beta(200)_{\text{FWHM}}=0.38^{\circ}$) indicating absence or negligiible
amount of the amorphous phase Adelfar  _et al._ (2019); Abrosimova  _et al._
(2020). To confirm this, we performed XRD measurements on W thin films grown
on MgO substrate and obtained results similar to that on Si substrate (see
SMSup ).

All the above, suggests that the nontrivial surface states of topological
Dirac metal of $\beta-$W Li  _et al._ (2019) are a likely source for the
inhomogeneity found in the effective medium models. We estimate the grain size
of $\beta-$W crystals to be $23(3)$ nm using Scherrer equationPatterson
(1939); Hao  _et al._ (2015) (see SMSup ). This implies that the grains of
$\beta-$W are nanocrystalline and the surface to volume ratio of the entire
thin film is large. Thus, in the superconducting state of nano-crystalline
$\beta-$W thin films, the topological surface states on the exterior of the
grains would act as sources of dissipation explaining the anomalous subgap
absorption observed in our films.

Figure 6:  $\theta-2\theta$ x-ray diffraction pattern of tungsten thin film
grown on Si(001)+SiO2 substrate after subtracting the substrate intensity
measured under exactly the same condition as the thin film. The $\beta-$W
peaks are marked in black and position of $\alpha-$W peaks as shown with blue
marker. The inset show zoomed-in view of diffraction intensity from
$76^{\circ}-116^{\circ}$.

###### Acknowledgements.

PC and NPA at JHU was supported through NSF grant DMR-1905519. RB at Morgan
State University was supported through Air Force Office of Scientific
Research, Grant No. FA9550-19-1-0082. We would like to thank T. McQueen for
access to his x-ray diffractometers, and E. A. Pogue for help in performing
these experiments.

## References

  * Gibson and Hein (1964) J. W. Gibson and R. A. Hein, Phys. Rev. Lett. 12, 688 (1964).
  * Basavaiah and Pollack (1968) S. Basavaiah and S. R. Pollack, Journal of Applied Physics 39, 5548 (1968), https://doi.org/10.1063/1.1656012 .
  * Hofer and Haberkorn (2019) J. Hofer and N. Haberkorn, Thin Solid Films 685, 117 (2019).
  * Hao  _et al._ (2015) Q. Hao, W. Chen, and G. Xiao, Applied Physics Letters 106, 182403 (2015), https://doi.org/10.1063/1.4919867 .
  * Mondal  _et al._ (2017) S. Mondal, S. Choudhury, N. Jha, A. Ganguly, J. Sinha, and A. Barman, Phys. Rev. B 96, 054414 (2017).
  * Demasius  _et al._ (2016) K.-U. Demasius, T. Phung, W. Zhang, B. P. Hughes, S.-H. Yang, A. Kellock, W. Han, A. Pushp, and S. S. P. Parkin, Nature Communications 7, 10644 (2016).
  * Li  _et al._ (2019) J. Li, S. Ullah, R. Li, M. Liu, H. Cao, D. Li, Y. Li, and X.-Q. Chen, Phys. Rev. B 99, 165110 (2019).
  * Xie  _et al._ (2018) T. Xie, M. Dreyer, D. Bowen, D. Hinkel, R. E. Butera, C. Krafft, and I. Mayergoyz, IEEE Transactions on Nanotechnology 17, 914 (2018).
  * Mellnik  _et al._ (2014) A. R. Mellnik, J. S. Lee, A. Richardella, J. L. Grab, P. J. Mintun, M. H. Fischer, A. Vaezi, A. Manchon, E.-A. Kim, N. Samarth, and D. C. Ralph, Nature 511, 449 (2014).
  * Bond  _et al._ (1965) W. L. Bond, A. S. Cooper, K. Andres, G. W. Hull, T. H. Geballe, and B. T. Matthias, Phys. Rev. Lett. 15, 260 (1965).
  * Xiao  _et al._ (2013) J. Xiao, P. Liu, Y. Liang, H. B. Li, and G. W. Yang, Nanoscale 5, 899 (2013).
  * Bardeen  _et al._ (1957) J. Bardeen, L. N. Cooper, and J. R. Schrieffer, Phys. Rev. 106, 162 (1957).
  * Khasanov and Mazin (2021) R. Khasanov and I. I. Mazin, Phys. Rev. B 103, L060502 (2021).
  * Carbotte (1977) J. Carbotte, in  _Anisotropy Effects in Superconductors_ (Springer, 1977) pp. 183–212.
  * (15) See Supplementary materials [url] which include Refs. [16-20], for details about the THz measurement, Effective medium approximation and XRD characterization.
  * Garnett (1904) J. C. M. Garnett, Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 203, 385 (1904).
  * Xi  _et al._ (2013) X. Xi, J.-H. Park, D. Graf, G. L. Carr, and D. B. Tanner, Phys. Rev. B 87, 184503 (2013).
  * Carr  _et al._ (1985) G. L. Carr, S. Perkowitz, and D. B. Tanner,  _Infrared and Millimeter Waves_ (Academic Press, Inc. New York, 1985) pp. 171–263.
  * Granqvist and Hunderi (1977) C. G. Granqvist and O. Hunderi, Phys. Rev. B 16, 3513 (1977).
  * Sihvola (2007) A. Sihvola, Journal of Nanomaterials 2007, 1 (2007).
  * Chauhan  _et al._ (2019) P. Chauhan, F. Mahmood, D. Yue, P.-C. Xu, X. Jin, and N. P. Armitage, Phys. Rev. Lett. 122, 017002 (2019).
  * Palmer and Tinkham (1968) L. H. Palmer and M. Tinkham, Phys. Rev. 165, 588 (1968).
  * Mattis and Bardeen (1958) D. C. Mattis and J. Bardeen, Phys. Rev. 111, 412 (1958).
  * Tinkham (2004) M. Tinkham,  _Introduction to Superconductivity: Second Edition_ (Dover Publications, 2004) pp. 99,103.
  * Zimmermann  _et al._ (1991) W. Zimmermann, E. Brandt, M. Bauer, E. Seider, and L. Genzel, Physica C: Superconductivity 183, 99 (1991).
  * Dressel (2013) M. Dressel, Advances in Condensed Matter Physics 2013, 104379 (2013).
  * Bruggeman (1935) D. A. G. Bruggeman, Annalen der Physik 416, 636 (1935).
  * Stroud (1975) D. Stroud, Phys. Rev. B 12, 3368 (1975).
  * Bennett (1965) A. J. Bennett, Physical Review 140, A1902 (1965).
  * Tomlinson and Carbotte (1976) P. Tomlinson and J. Carbotte, Physical Review B 13, 4738 (1976).
  * Crabtree  _et al._ (1987) G. Crabtree, D. Dye, D. Karim, S. Campbell, and J. Ketterson, Physical Review B 35, 1728 (1987).
  * Mattheiss (1965) L. Mattheiss, Physical review 139, A1893 (1965).
  * Shen  _et al._ (2000) Y. G. Shen, Y. W. Mai, Q. C. Zhang, D. R. McKenzie, W. D. McFall, and W. E. McBride, Journal of Applied Physics 87, 177 (2000), https://doi.org/10.1063/1.371841 .
  * Adelfar  _et al._ (2019) R. Adelfar, H. Mirzadeh, A. Ataie, and M. Malekan, Journal of Non-Crystalline Solids 520, 119466 (2019).
  * Abrosimova  _et al._ (2020) G. Abrosimova, B. Gnesin, D. Gunderov, A. Drozdenko, D. Matveev, B. Mironchuk, E. Pershina, I. Sholin, and A. Aronin, Metals 10 (2020), 10.3390/met10101329.
  * Patterson (1939) A. L. Patterson, Phys. Rev. 56, 978 (1939).

Prashant Chauhan1, Ramesh Budhani2, and N. P. Armitage1

1 The Institute for Quantum Matter, Department of Physics and Astronomy
The Johns Hopkins University, Baltimore, Maryland 21218, USA
2 Department of Physics, Morgan State University, 1700E Cold Spring Lane,
Baltimore, Maryland 21251, USA

##  I Time-domain THz measurement

The complex optical conductance was obtained using time-domain THz
spectroscopy. A femtosecond IR laser pulse is split along two paths to excite
a pair of photoconductive ‘Auston’-switch antennae grown on LT-GaAs wafers. A
broadband THz range pulse is emitted by one antenna and measured at the other
antenna. By varying the length-difference of the two paths, we map out
electric field of the pulse as a function of time, both through the $\beta-$W
sample on a Si substrate and through a bare reference Si substrate. The
electric fields are converted to the frequency domain by taking a Fast Fourier
Transform (FFT). By dividing the complex FFTs of the sample and reference
scans, we obtain the complex transmission of the sample. We then invert the
transmission to obtain the complex conductance via the standard formula for
thin films on a substrate: $\tilde{\rm
T}(\nu)=\frac{1+n}{1+n+Z_{0}\tilde{\sigma}(\nu)d}e^{i\Phi_{s}}$ where
$\Phi_{s}$ is the phase accumulated from the small difference in thickness
between the sample and reference substrates and $n$ is the substrate index of
refraction. By measuring both the magnitude and phase of the transmission,
both the real and imaginary conductance are obtained directly and no Kramers-
Kronig transformation is required. The complex conductance, $\tilde{G}$, is
then obtained from the complex transmission in the thin-film limit as
$\tilde{G}\left(\nu\right)=\frac{(n+1)}{Z_{0}}(\frac{e^{i\omega(n-1)\frac{\Delta
L}{c}}}{T(\nu)}-1)$, where $n$ is the refractive index of the substrate and
$\Delta L$ is the thickness difference between sample and reference
substrates.

##  II Maxwell-Garnett Theory

Maxwell-Garnett theory (MGT) can be used to describe the effective optical
conductance, $\tilde{G}(\nu)$, of an inhomogeneous superconductor. The thin-
film can be treated as an inhomogeneous medium with two components, a with
volume fraction $f$ embedded in a surrounding medium $b$ with volume fraction
$1-f$ Garnett (1904); Xi  _et al._ (2013). By assuming that the separation
between the grains is large enough for an individual grain to scatter light
and that the medium $b$ remains unaffected by grains, MGT gives an effective
dielectric function for oriented ellipsoidal grains as Xi  _et al._ (2013);
Carr  _et al._ (1985); Granqvist and Hunderi (1977)

$\tilde{\epsilon}_{MGT}=\tilde{\epsilon}_{b}+\tilde{\epsilon}_{b}\frac{f(\tilde{\epsilon}_{a}-\tilde{\epsilon}_{b})}{g(1-f)(\tilde{\epsilon}_{a}-\tilde{\epsilon}_{b})+\tilde{\epsilon}_{b}},$
(1)

where $g$ is the depolarization factor that corresponds to the the shape of
the ellipsoid inclusions. Taking the inclusions as cylindrical tubes with a
normal core, we set $g=1/2$ Sihvola (2007). Using,
$\tilde{\epsilon}=1+2i\tilde{G}/\nu$, we obtain the effective optical
conductance as

$\tilde{G}(\nu)=\tilde{G_{s}}+\frac{f(\tilde{G_{N}}-\tilde{G_{s}})(1+\dfrac{2i\tilde{G_{s}}}{\nu})}{0.5(1-f)(\dfrac{2i}{\nu})(\tilde{G_{N}}-\tilde{G_{s}})+(1+\dfrac{2i\tilde{G_{s}}}{\nu})},$
(2)

where $f$ is the volume fraction of the normal metal cores, $\tilde{G_{N}}$
and $\tilde{G_{s}}$ are the conductances of the normal and superconducting
fractions respectively.

Fig. S1: Simulated frequency dependent (a) real and (b) imaginary parts of the
complex optical conductance of a superconductor with $T_{c}=3.7$ K and energy
gap $\Delta(0)=0.42$ meV based on Maxwell-Garnett theory for varying normal
fraction $f$. The superconducting component is accounted using Mattis-Bardeen
theory while the normal component is accounted for by a purely real and
frequency independent conductance.

##  III  Bruggeman effective medium approximation

In MGT, due to the presence of the grains with properties different from the
host medium, the electric field in the region surrounding the grain gets
modified causing deviation of electric flux in the host. Bruggeman suggested
that for an adequate choice of self-consistent local field the average flux
deviation should be zero. Thus we can consider an effective medium in which
all inclusions are treated equally e.g. there is no host media, giving average
flux deviation as zero. According to Bruggeman effective medium approximation
an inhomogeneous medium of two components $a$ with fraction $f$ and $b$ with
fraction $1-f$, the effective dielectric function for oriented ellipsoid
grains as the solution of the equation

Fig. S2: Simulated frequency dependent (a) Real and (b) imaginary part of
complex optical conductance of a superconductor with $T_{c}=3.7$ K and energy
gap $\Delta(0)=0.42$ meV based on Bruggeman effective medium approximation for
varying normal fraction $f$. The superconducting component is accounted using
Mattis-Bardeen theory while again the normal component is accounted for by a
purely real and frequency independent conductance.

$f\frac{\tilde{\epsilon_{a}}-\tilde{\epsilon}_{EMA}}{g\tilde{\epsilon}_{a}+(1-g)\tilde{\epsilon}_{EMA}}+(1-f)\frac{\tilde{\epsilon_{b}}-\tilde{\epsilon}_{EMA}}{g\tilde{\epsilon}_{b}+(1-g)\tilde{\epsilon}_{EMA}}=0.$
(3)

Taking inclusions as cylindrical particles, we set $g=1/2$. The solution for
the above equation is

$\tilde{\epsilon}_{EMA}=\frac{1}{2}[(2f-1)(\tilde{\epsilon}_{a}-\tilde{\epsilon}_{b})+\sqrt{(2f-1)^{2}(\tilde{\epsilon}_{a}-\tilde{\epsilon}_{b})^{2}+4\tilde{\epsilon}_{a}\tilde{\epsilon}_{b}}].$
(4)

We obtain the conductance by substituting,
$\tilde{\epsilon}=1+2i\tilde{G}/\nu$. Figure S2 shows the real and imaginary
part of conductance at different fractions of non-superconducting parts of the
film. With increasing fraction of the normal part both the components of the
conductance approach $G_{n}$. From the deviations in $G_{2}(\nu)$ one can
notice that, the EMA starts to fail after $f=0.6$, which was also noted in a
similar analysis by Xi et. al. Xi  _et al._ (2013).

##  IV Mattis-Bardeen theory: Photon and Thermal excitation

Mattis-Bardeen theory describes the complex electromagnetic response of cooper
pairs. It accounts for both the thermally and photon excited quasiparticles
that accounts for the total optical response in a superconductor. Fig. S3
shows the individual thermal and photon excited contributions for varying
temperature. The low energy response below $h\nu/2\Delta$ is mainly occupied
by the thermally excited quasiparticles and it increases with temperature.
Similarly, the response above $h\nu/2\Delta$ is mainly occupied by photon
excitations across the gap.

Fig. S3: Simulated temperature variation of frequency dependent normalized
conductance $G_{1}(\nu)/G_{n}$ of a superconductor using Mattis-Bardeen
theory. (a) Contribution from thermal excitations, (b) Contribution from
photon excitations across the energy gap (c) combined thermal and photon
spectra.

##  V Optical conductance $\beta-$W on MgO substrate

We measure the optical conductance of $\beta-$tungsten on MgO grown along with
the samples on Si substarate. Both the real ($G_{1}(\nu)$) and imaginary
($G_{2}(\nu)$) parts of complex conductance show similar results as the films
grown on Si substrate shown in the main text.

Fig. S4: Real $G_{1}(\nu)$ and imaginary $G_{2}(\nu)$ parts of the normalized
zero-field complex conductance of $\beta$-W 70 nm thin film grown on MgO
substrate as a function of frequency from normal state (T$>$Tc) to
($$1.6\text{\,}\mathrm{K}$\ll$Tc).

##  VI X-ray Diffraction characterization

Fig. S5:  $\theta-2\theta$ x-ray diffraction pattern of (a)tungsten thin film
grown on Si(001)+SiO2 substrate and the Si substrate. (b)W XRD after
subtracting the substrate intensity measured under exactly the same condition
as the thin film. The $\beta-$W peaks are marked in black and position of
$\alpha-$W peaks as shown with blue marker. The inset show zoomed-in view of
diffraction intensity from $76^{\circ}-116^{\circ}$

The samples were characterized using Bruker D8 advance X-ray diffractometer in
$\theta-2\theta$ mode between $30^{\circ}-120^{\circ}$. Fig. S5 shows the XRD
pattern of both the W thin film grown on Si(001)+SiO2 substrate measured in
exact same conditions. The Si peaks match exactly in both the sample and the
Si substrate. After subtracting the substrate XRD pattern from the sample we
identify only $\beta-$W peaks. No $\alpha-$W peaks are observed. XRD data can
also provide us with an estimate on the grain size. The broadening of the
diffraction peak can be related to the size of the grain, using Scherrer
equation given by $d=K\lambda/\Gamma cos(\theta)$ where $K$ is the shape
factor whose value is approximately 0.9, $\lambda$ is the X-ray wavelength,
$\Gamma$ is the peak width or FWHM and $\theta$ is the Bragg angle Patterson
(1939); Hao  _et al._ (2015). For most of our thin films grown on both Si and
MgO substrates the Bragg angle for the first $\beta-$W peak is 35.6(3)∘ with
width of $0.43(6)^{\circ}$. Thus the grain size for our thin films range
between $20-26$ nm.

Fig. S5 shows the XRD pattern of both W/MgO and MgO measured in exact same
conditions. We again only identify $\beta-$W peaks and no $\alpha-$W peaks are
observed. At the peak positions we do not see any large broadening which is
indicative of amorphous W. Thus concluding that our W thin films are pure A15
phase within our instrumental uncertainty. In order to get an approximate
estimate of A15:bcc phases we take ratio of $\beta-$W at $40^{\circ}$ to
background after the peak (from Fig. S5(b)) which might have hidden $\alpha-$W
peak. we find that the amount of $\alpha-$W phase is $<10\%$.

Fig. S6:  $\theta-2\theta$ x-ray diffraction pattern of (a)tungsten thin film
grown on MgO substrate and the MgO. (b)W XRD after subtracting the MgO
intensity measured under exactly the same condition as the thin film. The
$\beta-$W peaks are marked in black and position of $\alpha-$W peaks as shown
with blue marker. The inset show zoomed-in view of diffraction intensity from
$109^{\circ}-120^{\circ}$

## References

  * Garnett (1904) J. C. M. Garnett, Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 203, 385 (1904).
  * Xi  _et al._ (2013) X. Xi, J.-H. Park, D. Graf, G. L. Carr, and D. B. Tanner, Phys. Rev. B 87, 184503 (2013).
  * Carr  _et al._ (1985) G. L. Carr, S. Perkowitz, and D. B. Tanner,  _Infrared and Millimeter Waves_ (Academic Press, Inc. New York, 1985) pp. 171–263.
  * Granqvist and Hunderi (1977) C. G. Granqvist and O. Hunderi, Phys. Rev. B 16, 3513 (1977).
  * Sihvola (2007) A. Sihvola, Journal of Nanomaterials 2007, 1 (2007).
  * Patterson (1939) A. L. Patterson, Phys. Rev. 56, 978 (1939).
  * Hao  _et al._ (2015) Q. Hao, W. Chen, and G. Xiao, Applied Physics Letters 106, 182403 (2015), https://doi.org/10.1063/1.4919867 .


– Under Review 2022 Full Paper – MIDL 2022 Mark S. Graham1
<EMAIL_ADDRESS>
Petru-Daniel<EMAIL_ADDRESS>
Paul<EMAIL_ADDRESS>
Walter Hugo Lopez<EMAIL_ADDRESS>
Jean-Marie<EMAIL_ADDRESS>
Yee H<EMAIL_ADDRESS>
James T<EMAIL_ADDRESS>
Rolf Jager4<EMAIL_ADDRESS>
David Werring5<EMAIL_ADDRESS>
Parashkev Nachev4<EMAIL_ADDRESS>
Sebastien Ourselin1<EMAIL_ADDRESS>
M. Jorge Cardoso1<EMAIL_ADDRESS>
1 Department of Biomedical Engineering, School of Biomedical Engineering &
Imaging Sciences, King’s College London, London, UK
2 King’s College Hospital NHS Foundation Trust, Denmark Hill, London, UK
3 Institute of Psychiatry, Psychology & Neuroscience, King’s College London,
London, UK
4 Institute of Neurology, University College London, London, UK
5 Stroke Research Centre, UCL Queen Square Institute of Neurology, London, UK

# Transformer-based out-of-distribution detection for clinically safe
segmentation

###### Abstract

In a clinical setting it is essential that deployed image processing systems
are robust to the full range of inputs they might encounter and, in
particular, do not make confidently wrong predictions. The most popular
approach to safe processing is to train networks that can provide a measure of
their uncertainty, but these tend to fail for inputs that are far outside the
training data distribution. Recently, generative modelling approaches have
been proposed as an alternative; these can quantify the likelihood of a data
sample explicitly, filtering out any out-of-distribution (OOD) samples before
further processing is performed. In this work, we focus on image segmentation
and evaluate several approaches to network uncertainty in the far-OOD and
near-OOD cases for the task of segmenting haemorrhages in head CTs. We find
all of these approaches are unsuitable for safe segmentation as they provide
confidently wrong predictions when operating OOD. We propose performing full
3D OOD detection using a VQ-GAN to provide a compressed latent representation
of the image and a transformer to estimate the data likelihood. Our approach
successfully identifies images in both the far- and near-OOD cases. We find a
strong relationship between image likelihood and the quality of a model’s
segmentation, making this approach viable for filtering images unsuitable for
segmentation. To our knowledge, this is the first time transformers have been
applied to perform OOD detection on 3D image data. Code is available at
github.com/marksgraham/transformer-ood.

###### keywords:

Transformers, out-of-distribution detection, segmentation, uncertainty.

##  1 Introduction

An important aim of the medical deep learning field is to develop image
processing algorithms that can be deployed in clinical settings. These tools
need to be robust to the full range of potential inputs they might receive in
a clinical context. We can expect this data to be much more diverse than the
typically clean, sanitised datasets on which these algorithms are developed
and tested. Even if attempts are made to train on messier datasets with images
exhibiting artefacts and other issues, we would expect that eventually the
tool will be presented with data it has not seen during training. Typically,
deep learning networks perform well when operating in-distribution, but
performance can degrade unpredictably and substantially when operating on data
out-of-distribution (OOD).

One approach to this problem is to develop networks that provide both
predictions and a measure of their uncertainty, enabling decisions to be
referred to humans when they are presented with difficult or OOD data samples.
Bayesian Neural Networks (BNN) that learn a distribution of weights are
capable of this; one popular aproach is to appromixate a BNN using dropout-
based variational inference [Gal and Ghahramani(2016)]. Another common
approach is the deployment of an ensemble of neural networks [Lakshminarayanan
et al.(2017)Lakshminarayanan, Pritzel, and Blundell]. A comprehensive
evaluation of uncertainty methods for classification found that the quality of
uncertainty measures degraded as the size of the distributional shift
increased [Ovadia et al.(2019)Ovadia, Fertig, Ren, Nado, Sculley, Nowozin,
Dillon, Lakshminarayanan, and Snoek]. Some work has evaluated these methods in
the context of image segmentation but is typically confined to evaluating the
utility of the uncertainty methods in-distribution [Jungo et al.(2018)Jungo,
Meier, Ermis, Herrmann, and Reyes, Nair et al.(2020)Nair, Precup, Arnold, and
Arbel]; work that does evaluate the uncertainty in an OOD setting typically
investigates only small dataset shifts such as increased noise [Haas and
Rabus(2021)] or lower quality scans [McClure et al.(2019)McClure, Rho, Lee,
Kaczmarzyk, Zheng, Ghosh, Nielson, Thomas, Bandettini, and Pereira].

A second approach is to filter our anomalous data before it is fed to the
task-specific network by using a generative model that can quantify the
probability that a data sample is drawn from the distribution that the task-
specific model was trained on. Of the generative approaches, autoregressive
methods are attractive for two reasons: firstly, they allow for the
computation of exact likelihoods, and secondly, a class of architectures that
can be used for autoregressive prediction termed transformers [Vaswani et
al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and
Polosukhin] are proving highly effective general-purpose architectures,
achieving state-of-the-art performance across a range of tasks in language
[Devlin et al.(2018)Devlin, Chang, Lee, and Toutanova, Brown et
al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam,
Sastry, Askell, et al.] and, increasingly, vision [Dosovitskiy et
al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner,
Dehghani, Minderer, Heigold, Gelly, et al.]. The high dimensionality of
medical images makes it computationally infeasible to use transformers to
model the sequence of raw pixel values, and a recent body of work has instead
used transformers to model the compressed discrete latent space of an image
obtained from a vector-quantised network such as a VQ-VAE or VQ-GAN [Oord et
al.(2017)Oord, Vinyals, and Kavukcuoglu, Razavi et al.(2019)Razavi, van den
Oord, and Vinyals, Esser et al.(2021)Esser, Rombach, and Ommer]. This approach
has provided state-of-the-art unsupervised pathology segmentation for 2D
medical images [Pinaya et al.(2021)Pinaya, Tudosiu, Gray, Rees, Nachev,
Ourselin, and Cardoso]. Prior work has also shown that this vector-quantised
class of auto-encoding methods can be used to substantially compress 3D
medical images [Tudosiu et al.(2020)Tudosiu, Varsavsky, Shaw, Graham, Nachev,
Ourselin, Sudre, and Cardoso] indicating the VQ-GAN + transformer approach
might be applied to fully 3D anomaly detection but, to our knowledge, there is
no published work attempting this.

In this work, we make two principal contributions, focusing on the problem of
segmentation of haemorrhagic lesions in head CT data. Firstly, we evaluate
segmentation uncertainty methods on a range of OOD inputs and demonstrate they
can catastrophically fail, producing confidently wrong predictions. Secondly,
we use transformers to perform image-wide OOD detection on 3D images. We find
this can effectively flag OOD data that segmentation networks fail to perform
well on, demonstrating their viability as a filter in clinical settings where
robust and fully-automated segmentation pipelines are needed.

##  2 Methods

In this work, we focus on the challenge of segmenting Intracerebral
Haemorrhages (ICH) in head CT data. The following sections summarise the
datasets used, the trained segmentation networks, and the approach to training
the VQ-GANs and transformers.

###  2.1 Datasets

We use three main datasets in this work; two head CT datasets (one used for
training, and an independent one for model evaluation), and a non-head CT
dataset.

The CROMIS dataset is a set of 687 head CT scans used for training all the
networks in this paper. The data consists of CTs containing ICH, acquired
across multiple sites as part of a trial [Werring(2017), Wilson et
al.(2018)Wilson, Ambler, Shakeshaft, Brown, Charidimou, Salman, Lip, Cohen,
Banerjee, Houlden, et al.]. Ground-truth haemorrhage segmentation masks are
available for 221 scans in the dataset.

The KCH dataset was used for algorithm validation. It consists of 47 clinical
scans selected for the presence of ICH, all with ground-truth masks provided
by an experienced neuroradiologist. This dataset was used to represent in-
distribution test data; it was further used to produce a set of corrupted
scans to test our algorithms in the near-OOD setting. A range of corruptions
were applied to each scan in the dataset, designed to emulate a number of
scenarios such as imaging artefact, image header errors, and errors in the
preprocessing pipeline that is typically applied before data is input into a
network. The corruptions included: addition of Gaussian noise, inversion
through each of the three image planes, skull-stripping, setting the image
background to values not equal to 0, global scaling of all image intensities
by a fixed factor, and the deletion of a set of slices (or chunks) of the
image. Any spatial manipulations of the image were also applied to the labels.
This totalled 15 corruptions applied to each image, creating a corrupted
dataset of 705 images. Examples of the corruptions applied are included in
Appendix A.

The Medical Decathlon dataset was used to test our algorithms in the far-OOD
setting. It comprises of a number of medical images covering a variety of
organs and imaging modalities, none of which are head CT. We selected 22
images from the test set of each of the ten classes (or as many as were
available in the test set if less than 22). A more detailed description of
this dataset can be found in [Simpson et al.(2019)Simpson, Antonelli, Bakas,
Bilello, Farahani, Van Ginneken, Kopp-Schneider, Landman, Litjens, Menze, et
al.].

Data processing was harmonised between all datasets as much as possible. All
CT head images were registered to MNI using an affine transformation,
resampled to 1$\mathrm{mm}$ isotropic, tightly cropped to a $176\times
208\times 176$ grid, intensities clamped between $[-15,100]$ and then rescaled
to the range $[0,1]$. For the images in the Decathlon dataset, all were
resampled to be 1$\mathrm{mm}$ isotropic and either cropped or zero-padded
depending on size to produce a $176\times 208\times 176$ grid. All CT images
had their intensities clamped between $[-15,100]$ and then rescaled to lie in
the range $[0,1]$, all non-CT images were rescaled based on their minimum and
maximum values to lie in the range $[0,1]$.

###  2.2 Segmentation networks

We tested three uncertainty methods commonly employed in the literature. The
first is intended as a simple baseline and uses the softmax of the network’s
output as a per-pixel probability map. The second is an ensemble of $N$ neural
networks, identical in architecture but each trained on a different subset of
the data [Lakshminarayanan et al.(2017)Lakshminarayanan, Pritzel, and
Blundell]. Based on recommendations from [Ovadia et al.(2019)Ovadia, Fertig,
Ren, Nado, Sculley, Nowozin, Dillon, Lakshminarayanan, and Snoek] we chose
$N$=5. Finally, we use an approximation of a BNN obtained through dropout-
based variational inference, training each dropout layer with a dropout
probability of $p=0.5$ and using 5 passes during inference to approximate the
posterior [Gal and Ghahramani(2016)].

All networks used the same UNet backbone based on [Falk et al.(2019)Falk, Mai,
Bensch, Çiçek, Abdulkadir, Marrakchi, Böhm, Deubner, Jäckel, Seiwald, et al.]
and implemented in Project MONAI111https://monai.io/ as the ‘BasicUnet’ class,
with (32, 32, 64, 128, 256) features in the 5 encoding layers, instance
normalisation, and LeakyReLU activations. We trained the networks using a
batch size of 3, augmenting with affine and elastic transformations and
sampling patches of size $128^{3}$. The Dice loss was used except for the
baseline network, which was trained using cross-entropy loss as Dice is known
to provide poorly calibrated, overly-confident predictions [Mehrtash et
al.(2020)Mehrtash, Wells, Tempany, Abolmaesumi, and Kapur]. All networks were
optimised using the AMSGrad variant of Adam [Reddi et al.(2019)Reddi, Kale,
and Kumar] with a learning rate of $1e^{-3}$.

For each network we sought to assign a single uncertainty value to each
predicted lesion. Firstly, we produced a per-voxel uncertainty map for each
method. For the baseline method this was the per-voxel softmax of the network
output. For the remaining networks we used the entropy between the $N$
predictions at each voxel, $\left(1-\sum_{i}^{N}p_{i}\ln p_{i}\right)$ as
described in [Nair et al.(2020)Nair, Precup, Arnold, and Arbel], subtracting
the entropy from 1 to produce a measure where larger values reflect higher
certainty. We produce per-lesion certainty by taking the average of the per-
voxel measures across each lesion, where each separate lesion is taken as each
fully connected component from the majority vote prediction of each network.

###  2.3 VQ-GAN + Transformer networks

Our approach to outlier detection uses a VQ-GAN to compress the information
content of each 3D volume into a discrete latent representation, and a
transformer to learn the probability density of these representations.

The VQ-GAN [Esser et al.(2021)Esser, Rombach, and Ommer] contains an encoder
$E$ which takes input $x\in\mathbb{R}^{H\times W\times D}$ and produces a
latent representation $z\in\mathbb{R}^{h\times w\times d\times n}$ where $n$
is the dimension of the latent embedding vector. The representation is
quantised by finding its nearest neighbour, as measured by an $L_{2}$ norm, in
a codebook of $K$ $n$-dimensional vectors and replacing the representation
with the nearest neighbour’s codebook index, $k$. A decoder uses the quantised
latent space to reconstruct the input, $\hat{x}\in\mathbb{R}^{H\times W\times
D}$. To encourage the network to learn a rich codebook, a discriminator $D$ is
used to try to differentiate between real and reconstructed images. Our
implementation’s encoder contains four levels, each consisting of a
convolution with stride=2 and a residual layer, each followed by ReLU layers.
This produces a latent space $16\times$ smaller along each dimension, so an
input with size $176\times 208\times 176$ is compressed to a latent size of
$11\times 13\times 11=1573$ elements. The codebook has $K=256$, each with
dimension $n=256$. The decoder also contained four levels, each consisting of
a residual layer followed by a transposed convolution with stride=2. The
codebook was updated using the exponential moving average as described in
[Oord et al.(2017)Oord, Vinyals, and Kavukcuoglu]. The VQ-GAN paper combined a
mean-squared error loss and a perceptual loss [Zhang et al.(2018)Zhang, Isola,
Efros, Shechtman, and Wang] for the reconstruction loss - we used both these
and an additional spectral loss [Dhariwal et al.(2020)Dhariwal, Jun, Payne,
Kim, Radford, and Sutskever]. Given state-of-the-art anomaly detection results
have been reported in 2D using a simpler VQ-VAE with MSE loss [Pinaya et
al.(2021)Pinaya, Tudosiu, Gray, Rees, Nachev, Ourselin, and Cardoso], we also
performed an ablation study to understand how the additional components of the
VQ-GAN contributed to performance. Models were trained using Adam with a
learning rate=1.65$e^{-4}$ and a batch-size of 96 on a Nvidia DGX A100.

After training the VQ-GAN, we can estimate the probability density of the
training data using a transformer. Each 3D discrete representation obtained
from the trained VQ-GAN is flattened into a 1D sequence, and the data-
likelihood is represented as the product of conditional probabilities,
$p(x)=\prod_{i}^{n}p(x_{i}|\mathbf{x}_{<i})$, with the transformer learning
the distribution of $p(x_{i}|\mathbf{x}_{<i})$ by being trained to maximise
the log-likelihood of the training data. In addition to estimating the whole
image likelihood $p(x)$, we produced spatial likelihood maps by reshaping each
$p(x_{i}|\mathbf{x}_{<i})$ from the 1D sequence back into the 3D shape of the
latent representation and upsampling to produce a spatial likelihood map of
the same dimension as the input image. The transformer’s attention mechanism
has a quadratic memory dependence on sequence length that makes it difficult
to train on large sequences, so we made use of the more efficient Performer
architecture [Choromanski et al.(2020)Choromanski, Likhosherstov, Dohan, Song,
Gane, Sarlos, Hawkins, Davis, Mohiuddin, Kaiser, et al.] which uses a
linearised approximation of the attention matrices to allow for training on
longer sequences. We used a 22 layer Performer with 8 attention heads and a
latent representation of size 256. The model was trained using the cross-
entropy loss using a learning rate of $5e^{-4}$ and a batch size of 240 on a
Nvidia DGX A100.

Both models were trained on the full CROMIS dataset. It should be noted that
this dataset contains pathological images containing haemorrhages; the
definition of in-distribution here is not healthy scans but rather scans that
are similar to the segmentation network’s training set, and the aim is to
estimate whether a new input is similar enough to the segmentation network’s
training set, so that it will be segmented accurately.

##  3 Experiments and Results

###  3.1 Segmentation uncertainty

We firstly examine the performance of segmentation algorithms in the far-OOD
case where images are of a different organ and/or modality than the intended
target for segmentation. In this case, any detection can be considered a
false-positive (FP). We calculated per-lesion confidence scores for each
detection and compared them to the per-lesion scores of every true-positive
(TP) detection on the normal head CT dataset. fig:seg_uncertainty_decathlon
shows the distribution of lesion confidence scores for these two datasets
overlap regardless of the segmentation uncertainty method used, meaning it is
not possible to separate FP detections made on far-OOD data from TP detections
on in-distribution data using any of these lesion confidence scores alone.
This motivates the need for explicit OOD detection models.

fig:seg_uncertainty_decathlon

Figure 1: The distribution of per-lesion confidence scores for far-OOD non-
head CT data and all true positive detections in the head CT dataset. Lines
also show confidence thresholds for operating in three regimes: high-
sensitivity ($>90\%$), high-specificity ($>90\%$) and balanced, as determined
using each method on the head CT dataset.

Secondly, we look at the segmentation network’s uncertainty performance for
near-OOD data: corrupted head CT scans. As these scans contain haemorrhages
both TPs and FPs exist. We defined a TP as a predicted mask with at least 50%
overlap with a ground truth mask, and computed the AUC obtained when using the
per-lesion certainty scores to classify detections, see
tab:seg_uncertainty_yee. The networks are able to classify lesions relatively
well for certain types of corruption, including noise, image flipping, and the
removal of ‘chunks’ from the data. However, they perform poorly for images
with modified background values or those that have been skull-stripped. For
scaled images all three methods had an AUC $\leq 0.36$, showing they often
assigned higher confidence values to FP detections than TP detections.

###  3.2 Transformers for OOD detection

We examined the ability of transformers to filter out OOD inputs based on the
whole-image log-likelihood. fig:likelihood_coarse_classes shows the
distribution of log-likelihood values for far-OOD, near-OOD and in-
distribution classes (plots showing each sub-class can be found in
fig:likelihood_fine_classes). tab:likelihood_auc show the ability of the log-
likelihood to distinguish OOD classes from normal head CT data. Performance is
perfect for the far-OOD case. In the case of near-OOD data, classes on which
the segmentation uncertainty performed poorly are distinguished well: images
with adapted backgrounds, skull-stripping, and global intensity scaling are
all distinguished with an AUC=1. Subtler corruptions, namely noise with
$\sigma\leq 0.1$ and L-R flips, are not distinguished well. These are classes
for which the segmentation network uncertainty measures perform well,
suggesting these corruptions are more in-distribution and explaining why they
have been assigned likelihoods more similar to the normal head CT data. This
result also suggests transformers and segmentation networks with uncertainty
may be used in tandem, with highly OOD images being filtered out by the
transformer and the segmentation network providing meaningful uncertainty
estimates on images that are only slightly OOD. fig:cromis_outliers shows some
qualitative results on real data: the CROMIS volumes with the lowest and
highest log-likelihood values as assigned by the transformer. fig:mse_plots
shows that reconstruction MSE alone is unable to separate out in-distribution
and OOD data, indicating the transformer component is essential for OOD
detection.

fig:likelihood_coarse_classes

Figure 2: Data log-likelihoods for far-OOD, near-OOD and in-distribution
volumes using transformers. Log-likelihoods for each sub-class are included in
fig:likelihood_fine_classes.

tab:ablation_study reports an ablation study for these results. The results
demonstrate the VQ-GAN outperforms the VQ-VAE; but that both the perceptual
loss and adversarial loss are needed together to improve performance. Our
inclusion of an additional spectral loss provides a modest improvement in
performance over the standard VQ-GAN losses. Examination of spatial likelihood
maps in fig:spatial_likelihood_demo show good localisation of image
corruptions. These maps might be used to explain what part of the image has
caused the system to flag it as low-likelihood. A common failure mode reported
for generative approaches to anomaly detection is the assignment of higher
log-likelihoods to very OOD samples [Nalisnick et al.(2018)Nalisnick,
Matsukawa, Teh, Gorur, and Lakshminarayanan]; this did not occur in our
experiments, which suggests that transformer-based anomaly detection methods
could be more robust to this type of failure.

tab:likelihood_auc

Table 1: AUC for distinguishing between normal head CT and far- and near-OOD
classes.

| Dataset | Likelihood | AUC
---|---|---|---
Non Head CT | Head MR | -7288 (134) | 1.00
Colon CT | -10809 (789) | 1.00
Hepatic CT | -10712 (763) | 1.00
Hippocampal MR | -7465 (20) | 1.00
Liver CT | -11116 (658) | 1.00
Lung CT | -9957 (289) | 1.00
Pancreas CT | -10798 (791) | 1.00
Prostate MR | -9140 (134) | 1.00
Spleen CT | -10895 (382) | 1.00
Cardiac MR | -9661 (318) | 1.00
Corrupted Head CT |  Noise $\sigma=0.01$ | -5796 (253) | 0.49
Noise $\sigma=0.1$ | -5793 (237) | 0.49
Noise $\sigma=0.2$ | -6637 (324) | 0.98
BG value=0.3 | -9022 (89) | 1.00
BG value=0.6 | -8803 (100) | 1.00
BG value=1.0 | -9979 (127) | 1.00
Flip L-R | -5850 (253) | 0.55
Flip A-P | -7435 (205) | 1.00
Flip I-S | -9036 (165) | 1.00
Chunk top | -6382 (214) | 0.96
Chunk middle | -7784 (179) | 1.00
Skull stripped | -7226 (125) | 1.00
Scaling 10% | -7436 (119) | 1.00
Scaling 1% | -7205 (25) | 1.00
Normal Head CT |  | -5803 (256) | -

We further sought to characterise the relationship between an image’s
likelihood and the performance of the segmentation network; we show results
for the best-performing dropout network in fig:likelihood_vs_seg_fps and
include results for other networks, which are similar, in
fig:likelihood_vs_seg_fps_extra. While we would not necessarily expect a
relationship between examples the transformer assigns a low likelihood to and
examples the networks perform poorly on, our results indicate there is a
strong one and that poor segmentations could be effectively filtered out using
a log-likelihood threshold of -7000. Furthermore, the results show that
corrupted CT-scans that were assigned similar likelihoods to the in-
distribution data tend to be handled better by the segmentation network,
supporting our claim that these indeed are more subtle corruptions.

fig:likelihood_vs_seg_fps

Figure 3: Image log-likelihood and the number of FP detections made by the
dropout network.

##  4 Conclusion

In a clinical setting, deployed systems must be robust to the full range of
data they may be presented with. Our results show that commonly used methods
for obtaining uncertainty from segmentation networks fail when OOD, making
confidently wrong predictions in far-OOD and near-OOD settings. We demonstrate
that with a VQ-GAN for image compression and a transformer for density
estimation we can successfully detect both far- and near-OOD data, and we find
the images flagged as OOD by this approach are precisely the ones that
segmentation networks struggle with. These results suggest our approach can be
used to filter out OOD inputs that a segmentation network is likely to fail
on, ensuring it is only run on in-distribution data they are likely to perform
well on. Future work will focus on further validating these findings on real-
world clinical datasets.

MG, PW, WS, SO, PN and MJC are supported by a grant from the Wellcome Trust
(WT213038/Z/18/Z). MJC and SO are also supported by the Wellcome/EPSRC Centre
for Medical Engineering (WT203148/Z/16/Z), and the InnovateUK-funded London AI
centre for Value-based Healthcare. PN is also supported by NIHR UCLH
Biomedical Research Centre. YM is supported by a grant from the Medical
Research Council (MR/T005351/1). The models in this work were trained on
NVIDIA Cambridge-1, the UK’s largest supercomputer, aimed at accelerating
digital biology.

## References

  * [Brown et al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, et al.] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ , 2020.
  * [Choromanski et al.(2020)Choromanski, Likhosherstov, Dohan, Song, Gane, Sarlos, Hawkins, Davis, Mohiuddin, Kaiser, et al.] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al.  Rethinking attention with performers.  _arXiv preprint arXiv:2009.14794_ , 2020.
  * [Devlin et al.(2018)Devlin, Chang, Lee, and Toutanova] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * [Dhariwal et al.(2020)Dhariwal, Jun, Payne, Kim, Radford, and Sutskever] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever.  Jukebox: A generative model for music.  _arXiv preprint arXiv:2005.00341_ , 2020.
  * [Dosovitskiy et al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * [Esser et al.(2021)Esser, Rombach, and Ommer] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12873–12883, 2021.
  * [Falk et al.(2019)Falk, Mai, Bensch, Çiçek, Abdulkadir, Marrakchi, Böhm, Deubner, Jäckel, Seiwald, et al.] Thorsten Falk, Dominic Mai, Robert Bensch, Özgün Çiçek, Ahmed Abdulkadir, Yassine Marrakchi, Anton Böhm, Jan Deubner, Zoe Jäckel, Katharina Seiwald, et al.  U-net: deep learning for cell counting, detection, and morphometry.  _Nature methods_ , 16(1):67–70, 2019.
  * [Gal and Ghahramani(2016)] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , pages 1050–1059. PMLR, 2016.
  * [Haas and Rabus(2021)] Jarrod Haas and Bernhard Rabus.  Uncertainty estimation for deep learning-based segmentation of roads in synthetic aperture radar imagery.  _Remote Sensing_ , 13(8):1472, 2021.
  * [Jungo et al.(2018)Jungo, Meier, Ermis, Herrmann, and Reyes] Alain Jungo, Raphael Meier, Ekin Ermis, Evelyn Herrmann, and Mauricio Reyes.  Uncertainty-driven sanity check: Application to postoperative brain tumor cavity segmentation.  _arXiv preprint arXiv:1806.03106_ , 2018.
  * [Lakshminarayanan et al.(2017)Lakshminarayanan, Pritzel, and Blundell] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _Advances in neural information processing systems_ , 30, 2017.
  * [McClure et al.(2019)McClure, Rho, Lee, Kaczmarzyk, Zheng, Ghosh, Nielson, Thomas, Bandettini, and Pereira] Patrick McClure, Nao Rho, John A Lee, Jakub R Kaczmarzyk, Charles Y Zheng, Satrajit S Ghosh, Dylan M Nielson, Adam G Thomas, Peter Bandettini, and Francisco Pereira.  Knowing what you know in brain segmentation using bayesian deep neural networks.  _Frontiers in neuroinformatics_ , 13:67, 2019.
  * [Mehrtash et al.(2020)Mehrtash, Wells, Tempany, Abolmaesumi, and Kapur] Alireza Mehrtash, William M Wells, Clare M Tempany, Purang Abolmaesumi, and Tina Kapur.  Confidence calibration and predictive uncertainty estimation for deep medical image segmentation.  _IEEE transactions on medical imaging_ , 39(12):3868–3878, 2020.
  * [Muschelli et al.(2015)Muschelli, Ullman, Mould, Vespa, Hanley, and Crainiceanu] John Muschelli, Natalie L Ullman, W Andrew Mould, Paul Vespa, Daniel F Hanley, and Ciprian M Crainiceanu.  Validated automatic brain extraction of head ct images.  _Neuroimage_ , 114:379–385, 2015.
  * [Nair et al.(2020)Nair, Precup, Arnold, and Arbel] Tanya Nair, Doina Precup, Douglas L Arnold, and Tal Arbel.  Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation.  _Medical image analysis_ , 59:101557, 2020.
  * [Nalisnick et al.(2018)Nalisnick, Matsukawa, Teh, Gorur, and Lakshminarayanan] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan.  Do deep generative models know what they don’t know?  _arXiv preprint arXiv:1810.09136_ , 2018.
  * [Oord et al.(2017)Oord, Vinyals, and Kavukcuoglu] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning.  _arXiv preprint arXiv:1711.00937_ , 2017.
  * [Ovadia et al.(2019)Ovadia, Fertig, Ren, Nado, Sculley, Nowozin, Dillon, Lakshminarayanan, and Snoek] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  _arXiv preprint arXiv:1906.02530_ , 2019.
  * [Pinaya et al.(2021)Pinaya, Tudosiu, Gray, Rees, Nachev, Ourselin, and Cardoso] Walter Hugo Lopez Pinaya, Petru-Daniel Tudosiu, Robert Gray, Geraint Rees, Parashkev Nachev, Sébastien Ourselin, and M Jorge Cardoso.  Unsupervised brain anomaly detection and segmentation with transformers.  _arXiv preprint arXiv:2102.11650_ , 2021.
  * [Razavi et al.(2019)Razavi, van den Oord, and Vinyals] Ali Razavi, Aaron van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  In _Advances in neural information processing systems_ , pages 14866–14876, 2019.
  * [Reddi et al.(2019)Reddi, Kale, and Kumar] Sashank J Reddi, Satyen Kale, and Sanjiv Kumar.  On the convergence of adam and beyond.  _arXiv preprint arXiv:1904.09237_ , 2019.
  * [Simpson et al.(2019)Simpson, Antonelli, Bakas, Bilello, Farahani, Van Ginneken, Kopp-Schneider, Landman, Litjens, Menze, et al.] Amber L Simpson, Michela Antonelli, Spyridon Bakas, Michel Bilello, Keyvan Farahani, Bram Van Ginneken, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, et al.  A large annotated medical image dataset for the development and evaluation of segmentation algorithms.  _arXiv preprint arXiv:1902.09063_ , 2019.
  * [Tudosiu et al.(2020)Tudosiu, Varsavsky, Shaw, Graham, Nachev, Ourselin, Sudre, and Cardoso] Petru-Daniel Tudosiu, Thomas Varsavsky, Richard Shaw, Mark Graham, Parashkev Nachev, Sebastien Ourselin, Carole H Sudre, and M Jorge Cardoso.  Neuromorphologicaly-preserving volumetric data encoding using vq-vae.  _arXiv preprint arXiv:2002.05692_ , 2020.
  * [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008, 2017.
  * [Werring(2017)] David Werring.  Clinical trial: Clinical relevance of microbleeds in stroke (cromis-2).  Technical Report NCT02513316, University College London, November 2017\.
  * [Wilson et al.(2018)Wilson, Ambler, Shakeshaft, Brown, Charidimou, Salman, Lip, Cohen, Banerjee, Houlden, et al.] Duncan Wilson, Gareth Ambler, Clare Shakeshaft, Martin M Brown, Andreas Charidimou, Rustam Al-Shahi Salman, Gregory YH Lip, Hannah Cohen, Gargi Banerjee, Henry Houlden, et al.  Cerebral microbleeds and intracranial haemorrhage risk in patients anticoagulated for atrial fibrillation after acute ischaemic stroke or transient ischaemic attack (cromis-2): a multicentre observational cohort study.  _The Lancet Neurology_ , 17(6):539–547, 2018\.
  * [Zhang et al.(2018)Zhang, Isola, Efros, Shechtman, and Wang] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018.

##  Appendix A Demonstration of image corruptions

fig:corruptions_demo

Figure A.1: Example of all the corruptions applied to one subject from the KCH
head CT dataset. All images are shown with the same intensity range.
Corruptions are: Noise: Adding Gaussian noise $\sim\mathcal{N}(0,\sigma^{2})$,
BG value: replacing background value of 0 with a new constant value, Flip:
invert image through described plane; Chunk: set a number of parallel
slices=0, Skull strip: remove skull using method described by [Muschelli et
al.(2015)Muschelli, Ullman, Mould, Vespa, Hanley, and Crainiceanu], Scaling:
reduce global image intensity by multiplying by a fixed factor.

##  Appendix B Additional results

tab:seg_uncertainty_yee  Network Perturbation None Noise Background Flipping
Chunks Skullstrip Scaling Baseline 0.84 0.82 0.38 0.86 0.87 0.45 0.33 Deep
Ensemble 0.75 0.85 0.42 0.77 0.82 0.54 0.36 Dropout 0.83 0.87 0.52 0.86 0.86
0.58 0.35

Table B.1: AUC for distinguishing TP and FP lesion detections using per-lesion
confidence scores from several segmentation uncertainty methods, for near-OOD
corrupted head CT data. It is noteable that the Baseline network outperforms
the Deep Ensemble for data with no corruptions applied. This occurs because
the baseline network makes a large number of wrong, low-confidence predictions
that inflate the AUC score. This illustrates that AUC scores alone are not
sufficient for assessing the effectiveness of segmentation networks. However,
they suffice for making the key point here: none of the networks assessed are
able to provide accurate measures of uncertainty when operating OOD.

fig:likelihood_fine_classes

Figure B.1: Image log-likelihoods for each sub class in the far-OOD dataset
(top) and near-OOD dataset (bottom).

fig:cromis_outliers

Figure B.2: For the CROMIS dataset, this shows the three volumes assigned the
lowest log-likelihood values (top three rows) and the highest values (bottom
three rows), with three planes shown for each volume.

fig:mse_plots

Figure B.3: VQ-GAN reconstruction mean-squared error for the far-OOD case
(left column) and the near-OOD case (right column), for coarse class labels
(top row) and fine-grained labels (bottom row). These results show that
reconstruction MSE alone is unable to identify OOD data.

tab:ablation_study  Dataset Model 3-layer MSE no-GAN 4-layer MSE no-GAN
4-layer MSE GAN 4-layer Perceptual no-GAN 4-layer Perceptual GAN 4-layer
Spectral GAN (ours) Non Head CT Head MR 0.00 1.00∗ 0.85∗ 0.84∗ 1.00∗ 1.00∗
Colon CT 1.00 1.00∗ 1.00∗ 1.00∗ 1.00∗ 1.00∗ Hepatic CT 1.00 1.00∗ 1.00∗ 1.00∗
1.00∗ 1.00∗ Hippocampal MR 0.00 1.00∗ 0.00∗ 0.01∗ 1.00∗ 1.00∗ Liver CT 1.00
1.00∗ 1.00∗ 1.00∗ 1.00∗ 1.00∗ Lung CT 1.00 1.00∗ 1.00∗ 1.00∗ 1.00∗ 1.00∗
Pancreas CT 1.00 1.00∗ 1.00∗ 1.00∗ 1.00∗ 1.00∗ Prostate MR 0.11 1.00∗ 1.00∗
1.00∗ 1.00∗ 1.00∗ Spleen CT 1.00 1.00∗ 1.00∗ 1.00∗ 1.00∗ 1.00∗ Cardiac MR 1.00
1.00∗ 1.00∗ 1.00∗ 1.00∗ 1.00∗ Corrupted Head CT Noise $\sigma=0.01$ 0.48 0.49∗
0.48∗ 0.49∗ 0.49∗ 0.49∗ Noise $\sigma=0.1$ 0.61 0.47∗ 0.44∗ 0.48∗ 0.49∗ 0.49∗
Noise $\sigma=0.2$ 0.82 0.70∗ 0.71∗ 0.58∗ 0.80∗ 0.98∗ BG value=0.3 1.00 1.00∗
1.00∗ 1.00∗ 1.00∗ 1.00∗ BG value=0.6 1.00 1.00∗ 1.00∗ 1.00∗ 1.00∗ 1.00∗ BG
value=1.0 1.00 1.00∗ 1.00∗ 1.00∗ 1.00∗ 1.00∗ Flip L-R 0.53 0.55∗ 0.58∗ 0.55∗
0.57∗ 0.55∗ Flip A-P 1.00 1.00∗ 1.00∗ 1.00∗ 1.00∗ 1.00∗ Flip I-S 1.00 1.00∗
1.00∗ 1.00∗ 1.00∗ 1.00∗ Chunk top 0.62 0.95∗ 0.92∗ 0.94∗ 0.95∗ 0.96∗ Chunk
middle 0.99 1.00∗ 1.00∗ 1.00∗ 1.00∗ 1.00∗ Skull stripped 0.00 1.00∗ 1.00∗
0.98∗ 1.00∗ 1.00∗ Scaling 10% 0.44 1.00∗ 0.62∗ 0.85∗ 1.00∗ 1.00∗ Scaling 1%
0.00 0.96∗ 0.00∗ 0.00∗ 1.00∗ 1.00∗

Table B.2: Study of changes to the VQ architecture and their influence on
model performance. We report AUC scores for differentiating between each sub-
class and normal CT data using the image log-likelihood values provided by
each model (1.0 is perfect performance). Model elements changed are: layers: 3
or 4 layers in the encoder and decoder of the auto-encoder, losses: MSE - mean
squared error, Perceptual - perceptual loss + MSE, Spectral -
spectral+perceptual loss+MSE, GAN: whether or not there was an adversarial
component to the training. Highlighted in red are any AUC scores $<1.0$ for
the far-OOD non head CT data, or $\leq 0.95$ for the near-OOD corrupted head
CT data. An asterisk indicates the AUC score is significantly greater than the
that of the baseline model (3-layer MSE, no-GAN) at a level of $p<0.05$ as
determined by bootstrapping with 1000 repetitions.

fig:spatial_likelihood_demo

Figure B.4: Spatial likelihood maps obtained from the transformer. These are
calculated by taking the likelihood values for each code in the sequence,
$p(x_{i}|\mathbf{x}_{<i})$, reshaping from 1D back to 3D, and then upsampling
by a factor of 16 along each axis using nearest-neighbour interpolation to
match the size of the latent representation to the size of the input image.
The network accurately assigns low likelihood values to regions of
corruptions, such as the brain in the noise $\sigma=0.2$ case, missing chunks,
and the absent skull. The same images without likelihood values overlaid are
shown in the appendix, fig:corruptions_demo.

fig:likelihood_vs_seg_fps_extra

Figure B.5: Image log-likelihood and the number of FP detections.