# Relativistic Gas: Lorentz-invariant Distribution for the velocities

a,bEvaldo M. F. Curado  aCentro Brasileiro de Pesquisas Fisicas, Rua Xavier
Sigaud 150, Rio de Janeiro 22290-180, Brazil  aCarlos E. Cedeño  aCentro
Brasileiro de Pesquisas Fisicas, Rua Xavier Sigaud 150, Rio de Janeiro
22290-180, Brazil  aIvano Damião-Soares  aCentro Brasileiro de Pesquisas
Fisicas, Rua Xavier Sigaud 150, Rio de Janeiro 22290-180, Brazil
a,b,c,dConstantino Tsallis  aCentro Brasileiro de Pesquisas Fisicas, Rua
Xavier Sigaud 150, Rio de Janeiro 22290-180, Brazil

###### Abstract

In 1911, Jüttner proposed the generalization, for a relativistic gas, of the
Maxwell-Boltzmann distribution of velocities. Here we want to discuss, among
others, Jüttner probability density function (PDF). Both the velocity space
and, consequently, the momentum space are not flat in special relativity. The
velocity space corresponds to the Lobachevsky one, which has a negative
curvature. This curvature induces a specific power for the Lorentz factor in
the PDF, affecting Jüttner normalization constant in one, two, and three
dimensions. Furthermore, Jüttner distribution, written in terms of a more
convenient variable, the rapidity, presents a curvature change at the origin
at sufficiently high energy, which does not agree with our computational
dynamics simulations of a relativistic gas. However, in one dimension, the
rapidity satisfies a simple additivity law. This allows us to obtain, through
the Central Limit Theorem, a new, Lorentz-invariant, PDF whose curvature at
the origin does not change for any energy value and which agrees with our
computational dynamics simulations data. Also, we perform extensive first-
principle simulations of a one-dimensional relativistic gas constituted by
light and heavy particles.

<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
> The proper relativistic generalization of the Maxwell distribution of
> velocities of an ideal gas is still an open question. The main proposal,
> Jüttner distribution, was introduced in 1911 and many attempts to improve it
> do exist. We show here that neither the Jüttner nor the so-called Modified
> Jüttner distributions are Lorentz-invariant. The Jüttner distribution, in
> particular, was constructed on the basis of a flat velocity space whereas
> this space is curved, namely the Einstein-Lobachevsky space. In the one-
> dimensional case, by using the relativistic addition of velocities and the
> central limit theorem, we were able to build a Lorentz-invariant
> distribution (LID). We performed molecular dynamical simulations of a one-
> dimensional relativistic gas and verified that, at high enough energy, the
> Jüttner and the Modified Jüttner distributions do not match the numerical
> simulations, in contrast with the LID which does agree. Additionally, we
> exhibit how the corresponding temperatures of these three distributions
> behave as functions of the gas mean energy $\langle E\rangle$. An important
> corollary of our framework is that the temperature of the relativistic gas
> is a Lorentz-invariant quantity.

##  1 Introduction

Probability density functions (PDF) for the velocity of the particles of a
relativistic gas are, actually, controversial themes in both astrophysics [1,
2, 3] and particle physics of heavy ions[4, 5, 6]. The first proposal and one
of the most well-known probability distribution of velocities for a
relativistic gas was originally derived by Jüttner [7] in 1911. Until now, it
remains as an interesting attempt to construct the basis of relativistic
thermodynamics.

Jüttner constructed his distribution maximizing the entropy $S$ of a
relativistic gas, conserving constant the relativistic energy $E$ (a quantity
which is not Lorentz-invariant) and the number $N$ of particles. The
distribution derived by him in the limit of velocities much smaller than the
velocity of the light, $v\ll c$, reproduces the classical Maxwell-Boltzmann
distribution of velocities [8, 9, 10, 11]. However, the Jüttner distribution
is not a Lorentz-invariant one as we will point out soon. Moreover, this
distribution has been, mainly in the last twenty years, a matter of intense
debate in the literature [12, 13, 14, 15, 16, 17, 18].

In a previous paper Curado et al. [17] worked on the construction of Lorentz-
invariant distributions, where they explored histograms with constant bins in
both velocity and rapidity, using the velocity as the abscissa of their
histograms. The distributions obtained there are Lorentz-invariant but in
terms of the velocity instead of the rapidity.

Here, we extended the work of Curado et al. [17], showing the rapidity as a
natural variable to construct Lorentz-invariant distributions, with the aim to
describe the statistical behaviour of this kind of systems. We compare the
different temperatures predicted for Jüttner, the modified one, and the
Lorentz-invariant distribution (LID). We also derive from the maximum entropy
principle the proper Lorentz-invariant distribution in terms of the rapidity.
Here, we exhibit the non Lorentz-invariant character of the Jüttner
distribution and also an attempt to adapt the Jüttner distribution to the
requirements of the Lorentz transformation, the so-called Modified Jüttner
distribution. We expose the missteps within the Jüttner derivation and
present, for the one-dimensional case, a new Lorentz-invariant distribution
(LID) based on the variable rapidity. The latter is defined in a flat space,
contrary to the velocity, which is curved. Because of this, the rapidity is an
ideal variable for constructing a new Lorentz-invariant distribution, as
presented later on. This novel probability density distribution is derived
from first principles. We show that this new distribution fits with high
accuracy the numerical data of molecular dynamics simulation for the rapidity
distribution of one-dimensional gases.

To present our discussion, we divide this paper as follows. Section I is the
introduction. In Section 2 we discuss the original work by Jüttner [7] and
show why his distribution - and also the so-called modified Jüttner
distribution - are not Lorentz invariant PDF. In Section III we essentially
follow the derivation of the new LID presented in [17] with a slightly
different approach. In Section IV we present new molecular dynamics
simulations we have performed, clearly showing that this novel Lorentz-
invariant distribution (LID) matches well the numerical simulations for
relativistic gases. We also verify that both the Jüttner and the Modified
Jüttner distributions fail in this respect. The analytical expressions of the
average energy of each particle were calculated and compared, showing the
possible discrepancies in the temperature estimates depending on which
distribution is used. In Section V we show how the LID could be derived from a
MaxEnt principle. Finally, we discuss our findings in the conclusions.

##  2 The Jüttner and the modified Jüttner distributions

In his original work[7], Jüttner expressed the number of particles $dN$ of a
given gas in a volume element in the phase-space $(x,y,z,p_{x},p_{y},p_{z})$
as
$dN=\tilde{P}_{J}(x,y,z,p_{x},p_{y},p_{z})\,dx\,dy\,dz\,dp_{x}\,dp_{y}\,dp_{z}$,
where $p_{i}$ are the reduced relativistic linear momentum components, or the
relativistic linear momentum per unit mass and $(x,y,z)$ denote the position.

Thus the total number of particles is given by

$N=\int\tilde{P}_{J}\,dx\,dy\,dz\,dp_{x}\,dp_{y}\,dp_{z}\,,$ (1)

where $p_{i}=\gamma(v)\dot{x}_{i}$, $v_{i}=\dot{x}_{i}\,,\,i=x,y,z$,
$v=\sqrt{\dot{x}+\dot{y}+\dot{z}}$, being $v$ the velocity of a particle of
the system with respect to an inertial frame, and
$\gamma(v)\equiv(1-v^{2}/c^{2})^{-1/2}$ is the Lorentz factor. The mean energy
of the particles is

$E=m_{0}c^{2}\int\gamma(v)\tilde{P}_{J}dx\,dy\,dz\,dp_{x}\,dp_{y}\,dp_{z}\,,$
(2)

where $m_{0}$ is the rest mass, and $c$ is the speed of light. The entropy $S$
of the system associated with such a state is

$S=-k\int\tilde{P}_{J}\ln\tilde{P}_{J}dx\,dy\,dz\,dp_{x}\,dp_{y}\,dp_{z}.$ (3)

Maximizing the entropy $S$, and conserving as constants both $N$ and the
relativistic energy $E$, Jüttner found a probability distribution of
velocities for a relativistic gas in three-dimensions,
$\tilde{P}_{J}(\gamma\,v)$. Transforming the phase space
$(x,y,z,p_{x},p_{y},p_{z})\rightarrow(x,y,z,v_{x},v_{y},v_{z})$, Jüttner
obtained his three-dimensional relativistic distribution of velocities,
$P_{J}(v_{x},v_{y},v_{z})$. This distribution yields, as expected, zero
probability for a particle to have a speed larger than that of light, $c$.

This three-dimensional distribution can be generalized to a $d$-dimensional
distribution of velocities as follows:

$\displaystyle p_{J}(v)\,d^{d}v$
$\displaystyle=\dfrac{m_{0}^{d}\gamma^{d+2}(v)e^{-\beta_{J}\gamma(v)}}{Z_{J}}d^{d}v,$
$\displaystyle Z_{J}$
$\displaystyle=\begin{cases}2m_{0}cK_{1}(\beta_{J})\quad&\text{for}\quad
d=1\\\ \dfrac{4\pi
m_{0}^{2}c^{2}(\beta_{J}+1)e^{-\beta_{J}}}{\beta_{J}^{2}}\quad&\text{for}\quad
d=2\\\ \dfrac{8\pi
m_{0}^{3}c^{3}K_{2}(\beta_{J})}{\beta_{J}}\quad&\text{for}\quad
d=3\end{cases}\,,$ (4a)

where $d$ is the dimension,
$d^{d}v=\\{dv,dv_{x}\,dv_{y},dv_{x}\,dv_{y}\,dv_{z}\\}=\\{dv,vdv\,d\theta,v^{2}\sin\theta
dv\,d\theta\,d\phi\\}$, for $d=1,2,3$ dimensions respectively,
$\beta_{J}=m_{0}c^{2}/(kT_{J})$, $k$ is the Boltzmann constant, $T_{J}$ is the
temperature associated with the Jüttner distribution, $Z_{J}$ is the
normalization constant, and, $K_{1}(x)$ and $K_{2}(x)$ are the modified Bessel
functions of the second kind.

It is worthwhile to note that Jüttner (wrongly) considered a flat space for
the velocities (it should be remembered that this distribution was proposed as
early as in 1911), as shown above. In fact, the relativistic space of
velocities is not flat, it is a curved space, with negative curvature, more
precisely a Lobachevsky space. This ensures that the pre-factor before the
exponential in the distribution is not $\gamma^{d+2}$, as in the Eq. (4), but
rather the pre-factor $\gamma^{d+1}$ which, with $d^{d}v$, is a Lorentz
invariant quantity. This is one flaw in the Jüttner distribution, since
$\gamma^{d+2}d^{d}v$ is not a Lorentz-invariant quantity. A second problem is
the argument of the exponential in Eq. (4), the relativistic energy. It is
well-known that the energy is not a Lorentz-invariant quantity. These two
aspects make the Jüttner distribution a non Lorentz-invariant distribution.

In an attempt to correct Jüttner’s distribution, at least at the pre-factor
level, Lehmann [19], and a year later, Dunkel and Hänggi [20], modified the
Jüttner’s distribution as follows:

$\displaystyle p_{MJ}(v)\,d^{d}v$
$\displaystyle=\dfrac{m_{0}^{d-1}\gamma^{d+1}(v)e^{-\beta_{MJ}\gamma(v)}}{Z_{MJ}}\,d^{d}v,$
$\displaystyle Z_{MJ}$
$\displaystyle=\begin{cases}2cK_{0}(\beta_{MJ})\quad&\text{for}\quad d=1\\\
\dfrac{4\pi m_{0}c^{2}e^{-\beta_{MJ}}}{\beta_{MJ}}\quad&\text{for}\quad d=2\\\
\dfrac{8\pi m_{0}^{2}c^{3}K_{1}(\beta_{MJ})}{\beta_{MJ}}\quad&\text{for}\quad
d=3\\\ \end{cases}\,,$ (5a)

replacing the factor $\gamma^{d+2}$ by the factor $\gamma^{d+1}$;
$\beta_{MJ}\equiv m_{0}c^{2}/(kT_{MJ})$, where $T_{MJ}$ is the temperature
associated to this distribution and $Z_{MJ}$ is its normalization constant.

It is worth emphasizing that neither the momentum nor the velocity spaces are
flat in special relativity and that the relativistic energy is not a Lorentz-
invariant quantity. Either for both of these reasons, or for only one of them,
neither Jüttner nor Modified Jüttner are Lorentz-invariant distributions for a
relativistic gas.

We note that Jüttner distribution $P_{J}(v)$ and the Modified Jüttner
distribution $P_{MJ}(v)$ change their behavior from mono-modal to bimodal
functions for $T_{J}=m_{0}c^{2}/((2+d)k)$ and $T_{MJ}=m_{0}c^{2}/((1+d)k)$,
respectively. For comparison, we plot in the left and the middle panels of
Figs. (1) the one dimensional Jüttner $P_{J}(v)$ and the Modified Jüttner
$P_{MJ}(v)$ distributions as functions of the velocity, respectively. Note
that the concavity change in both distributions at the origin occurs at the
temperatures $T_{J}=m_{0}c^{2}/(3k)$ and ${T_{MJ}=m_{0}c^{2}/(2k)}$
respectively. In the low-temperature regime both distributions reproduce the
classical Maxwell-Boltzmann one, when the particles move in the low-velocity
limit $v\ll c$. In the figures we use normalised units $c=k=m_{0}=1$.

Now, we exhibit that the Jüttner and the Modified Jüttner distributions for
the velocities, in one dimension, are not Lorentz-invariant. It implies that
both Jüttner and the Modified Jüttner distributions seen by one observer with
velocity $v_{0}$ with respect to the laboratory frame must be distinct from
those seen by the laboratory frame. Using the relative velocity
$v^{\prime}=(v-v_{0})/(1-vv_{0}))$ into $P_{J}(v)$ and $P_{JM}(v)$ we obtain

$\displaystyle P_{J}(v^{\prime})dv^{\prime}$
$\displaystyle=\dfrac{c^{2}\exp\left(-\beta_{J}\left(1-\frac{c^{2}(v-v_{0})^{2}}{\left(c^{2}-vv_{0}\right)^{2}}\right)^{-1/2}\right)}{2\left(c^{2}-v^{2}\right)K_{1}(\beta_{J})\sqrt{1-\frac{c^{2}(v-v_{0})^{2}}{\left(c^{2}-vv_{0}\right)^{2}}}}dv$
$\displaystyle\neq p_{J}(v)dv\,,$ (6) $\displaystyle
P_{MJ}(v^{\prime})dv^{\prime}$
$\displaystyle=\frac{c\exp\left(-\beta_{J}\left(\frac{\left(c^{2}-v^{2}\right)\left(c^{2}-v_{0}^{2}\right)}{\left(c^{2}-vv_{0}\right)^{2}}\right)^{-1/2}\right)}{2\left(c^{2}-v^{2}\right)K_{0}(\beta_{J})}dv$
$\displaystyle\neq p_{MJ}(v)dv\,.$ (7)

We plot Jüttner’s distribution (2) under a Lorentz transformation, for
$T=1.0$, in the right panel of Fig. 1, taking normalized units. We verify that
for different velocity values of the external observer $v_{0}$, the
distribution form is not maintained invariant under Lorentz transformations.

|  |
---|---|---
Figure 1: One-dimensional distributions. Left panel: $P_{J}(v)$ as a function
of the velocity for different values of $T_{J}$. We plot here the
computational results of our simulations, which are explained in details
below, for comparison with the analytical result. Middle panel: $P_{MJ}(v)$ as
a function of the velocity. We use the same parameters as in the left panel,
for comparison. Both functions recover the Maxwell-Boltzmann distribution at
the limit $T\rightarrow 0$ and become bi-modal for high values of $T$. Right
panel: $P_{J}(v)$, for $T_{J}=1.0$, as a function of the velocity as measured
by different external observers, see Eq. (2), for
$v_{0}/c=0.0,\,,0.3,\,0.6,\,0.9$. We consider in all these plots
$m_{0}=c=k=1$.

##  3 On a Lorentz invariant distribution

The relativistic one-dimensional addition of velocities of two particles,
viewed by an observer (at rest with the box) as having velocities $v_{1}$ and
$v_{2}$, satisfies the well-known formula,

$v=\dfrac{v_{1}+v_{2}}{1+v_{1}v_{2}/c^{2}},$ (8)

which can be rewritten in a more symmetric way as

$\frac{1+v}{1-v}=\frac{1+v_{1}}{1-v_{1}}\frac{1+v_{2}}{1-v_{2}}.$ (9)

Notice that we are considering here the velocities $v_{1}$ and $v_{2}$ as the
the relativistic relative velocities between particles $1$ or $2$ and the
velocity of the reference frame (box), which is taking here equal to zero; $v$
is a relative velocity with respect to the velocity of the box. As well-known,
relative velocities are Lorentz-invariant quantities.

If we are adding $N$ particles, Eq. (9) can be easily generalized as follows:

$\frac{1+v}{1-v}=\prod_{i=1}^{N}\frac{1+v_{i}}{1-v_{i}}.$ (10)

Taking the logarithm on both sides and dividing the resulting expression by 2,
we have

$\frac{1}{2}\ln\left(\frac{1+v}{1-v}\right)=\frac{1}{2}\sum_{i=1}^{N}\ln\left(\frac{1+v_{i}}{1-v_{i}}\right).$
(11)

The rapidity $\sigma$ is defined as

$\sigma\equiv\dfrac{1}{2}\ln\left(\dfrac{1+v/c}{1-v/c}\right)=\tanh^{-1}\left(\frac{v}{c}\right)\,,\,\,\,\,\,\,\,\quad\sigma\in(-\infty,\infty)\,.$
(12)

Then Eq. (11) is reduced to $\sigma=\sum_{i=1}^{N}\sigma_{i}$. This, means
that the rapidity $\sigma$, which is a function of the relative velocity $v$,
satisfies the usual addition law of velocities and that the rapidity space is
flat. As the rapidity is a function of relative velocities, it is clearly a
Lorentz-invariant quantity. On the other hand, in order to discuss the line
element in the velocity space, and following Fock [21], the relativistic
relative velocity $v^{\prime}$ between two particles, which is a Lorentz-
invariant quantity, is, in one dimension,

$v^{\prime}=\dfrac{v_{1}-v_{2}}{1-v_{1}v_{2}/c^{2}},$ (13)

where $v_{1}$ and $v_{2}$ are the velocities of the particles 1 and 2 as
measured from the same inertial observer.

From (13), and considering that both particles move with infinitesimally
differing velocities, let us say, $v_{1}=v$ and $v_{2}=v+dv$, the square of
the line element associated to the velocity space is

$ds^{2}\equiv\frac{d(v^{\prime})^{2}}{c^{2}}=\frac{c^{2}dv^{2}}{(c^{2}-v^{2})^{2}}=\frac{\gamma^{4}}{c^{2}}dv^{2}\,,$
(14)

where, clearly, $ds=d\sigma$.

Considering now a relativistic gas of non-interacting particles whose
rapidities (functions of the relative velocities of the particles with respect
to the box) are independent variables, the Central Limit Theorem tell us that
the sum $\sigma=\sum_{i=1}^{N}\sigma_{i}$ gives us, for a sufficiently large
enough number of particles the following distribution of the rapidities, or of
the velocities, if we use Eq. (12),

$\displaystyle P_{LID}(\sigma)d\sigma$
$\displaystyle=\sqrt{\dfrac{\beta}{N\pi}}\,e^{-\frac{\beta\sigma^{2}}{N}}d\sigma$
$\displaystyle=\sqrt{\dfrac{\beta}{N\pi
c^{2}}}\,e^{-\frac{\beta\tanh^{-1}(\frac{v}{c})^{2}}{N}}\gamma^{2}\left(v/c\right)dv$
(15a) $\displaystyle\beta\equiv\dfrac{m_{0}c^{2}}{2kT}\,.$ (15b)

As the rapidities are functions of the relative velocities of the particles
with regard to the reference frame (box), and as the correct one-dimensional
pre-factor $\gamma^{2}$, required by the – Lorentz-invariant – Lobachevsky
velocity space, appears naturally here, this distribution is a Lorentz-
invariant distribution one (LID). Eq. (15) recovers the Maxwell distribution
of velocities for $v/c\ll 1$, and it does not admit velocities higher than the
velocity of the light. Therefore, this Lorentz-invariant distribution is
obtained by theoretical arguments, and its validity will be checked with
numerical data obtained from dynamical molecular simulations.

Then we can represent this Lorentz-invariant distribution as a function of the
velocity but also as function of the rapidity, which is a more convenient
variable to characterize a one-dimension relativistic gas, and whose domain is
the whole set of real numbers, $\mathbb{R}$. In fact, by representing the
Jüttner distribution in terms of the $\tanh^{-1}(v/c)$ it is possible to see
that this distribution changes concavity at the origin for sufficiently high
energy per particle, a behavior that is not observed if we use the Lorentz-
invariant distribution (LID). As we can see in the next section, numerical
simulations support the LID, showing no change of concavity at the origin.

A key point about the invariance under a Lorentz transformation of a given PDF
is that the temperature itself must be a Lorentz-invariant quantity, i.e.,
that the temperature measured by different inertial observers must be
invariant. A direct consequence of that is the validity of second law of
thermodynamics in special relativity. This fundamental point was raised by
Landsberg in two seminal papers at the end of the 1960s [12, 13] and it is
confirmed here.

##  4 Mean Kinetic Energy

We evaluate the mean kinetic energy in one dimension

$E_{\text{dist}}=\int_{-\infty}^{\infty}P_{\text{dist}}(v)m_{0}c^{2}\gamma(v)dv\,,$
(16)

where $E_{\text{dist}}$ and $P_{\text{dist}}$ are respectively the mean energy
and the PDF of the distribution which is being used ($P_{J}(v)$, $P_{MJ}(v)$
or $P_{LID}(v)$). Thus, for $P_{\text{dist}}(v)=P_{J}(v)$,

$E_{J}=m_{0}c^{2}\left(\dfrac{1}{\beta_{J}}+\dfrac{K_{0}(\beta_{J})}{K_{1}(\beta_{J})}\right)\,,$
(17)

for $P_{\text{dist}}(v)=P_{MJ}(v)$,

$E_{MJ}=\dfrac{m_{0}c^{2}K_{1}(\beta_{MJ})}{K_{0}(\beta_{MJ})}\,,$ (18)

for the LID $P_{\text{dist}}(v)=P(v)$,

$E=m_{0}c^{2}\exp\left(\dfrac{1}{4\beta}\right)\,,$ (19)

and, for the Maxwell-Boltzmann distribution $P_{\text{dist}}(v)=P_{MB}(v)$,

$E_{MB}=\frac{kT_{MB}}{2}\,.$ (20)

The present numerical results computed here were averaged over 200
relativistic molecular dynamical independent simulations. We considered a one-
dimensional gas with $N=5000$ particles, half of them having a light mass,
$m=1$, the other half having a heavy mass, $m=2$, with no interaction
excepting perfect elastic binary collisions. Thus, we constructed histograms
monitoring both the velocity $v$ and the rapidity variable $\sigma$.

Our simulations correspond to a microcanonical ensemble, having a constant
total energy $E$ in the box frame. We consider that, in each binary collision,
the relativistic energy is conserved. To perform our simulations we solve the
equations for the momentum and the energy conservation. Thus, before collision
$(p_{i},e_{i})$ and after collision $(P_{i},E_{i})$, $i=1,2$, we have

$\displaystyle P_{1}+P_{2}$ $\displaystyle=p_{1}+p_{2},$ (21) $\displaystyle
E_{1}+E_{2}$ $\displaystyle=e_{1}+e_{2}.$ (22)

As a result, the momenta after the collision become

$\displaystyle P_{i}$
$\displaystyle=\left[\gamma(v_{zm})\right]^{2}\left[\dfrac{2v_{zm}e_{i}}{c^{2}}-\left(1+\dfrac{v_{zm}}{c}\right)^{2}p_{i}\right],$
(23)

where

$v_{zm}=\dfrac{(p_{1}+p_{2})c^{2}}{e_{1}+e_{2}}$ (24)

is the velocity of the zero momentum frame, or the generalization of the
center of mass in special relativity. With this, the time between two
successive collisions is given by

$t=\dfrac{-v_{12}r_{12}-\sqrt{(v_{12}r_{12})^{2}-v_{12}^{2}r_{12}^{2}}}{v_{12}^{2}}$
(25)

where the relative position $r_{12}$ and velocity $v_{12}$ are respectively
given by $r_{12}=r_{1}-r_{2}$ and $v_{12}=(v_{1}-v_{2})/(1-v_{1}v_{2}/c^{2})$.
Time $t$ is used to evolve the system. We consider the gas inside a one-
dimensional box of length $L$ with periodic boundary conditions.

Our results are in complete agreement with those obtained by Cubero et al [14]
for distribution at low temperatures $T\leq 0.4$. We have numerically checked
this by constructing histograms for the velocities of the particles with
respect to the box, and then observing this system from external inertial
frames moving at different velocities with respect to the box. To compute the
probability density function (PDF) for the rapidity, we arbitrarily choose,
once for ever, a particle and measure the rapidities of all the others with
respect to this one. The rapidity was computed directly from the velocities of
the particles with respect to the laboratory frame, see Eqs. (8) and (12). The
rapidity histograms are constructed with constant rapidity bins[17]. To
minimize statistical fluctuations, we average $200$ simulations for a given
total energy $E$.

Just for completeness, we have also performed these numerical simulations by
using as inertial frame either the box or an arbitrarily chosen particle, and
we have obtained the same statistical distribution. The important point here
is that we are using the rapidity of the relativistic difference of
velocities, which is a Lorentz-invariant quantity. The fact that these various
ways to calculate the numerical distribution yield the same histogram confirms
the correctness of our simulations. Also, our numerical results for the
velocities recover those of Cubero et al [14] for not very high temperatures.

In the left panel of figure 1, we show the numerical result of our
simulations. The blue and the purple points correspond to the numerical data.
We plot the Jüttner distribution, in particular for $T_{J}=0.18$ and for
$T_{J}=1.01$, and compare with the blue and purple point distributions,
respectively. We found that for lower values of $T_{J}\leq 0.4$ Jüttner’s
distribution fits very well the distribution of points, however for values of
$T_{J}$ greater that $0.4$, the differences between the point distribution and
the Jüttner fitting grows in particular near to $v=\pm 1$. On the other hand,
in the left panel of Fig. 2 we show our numerical results in rapidity. The
black and red points correspond to the numerical data, whereas the red and
black lines are LID adjustments for the heavy and light particles
respectively. Also, for comparison, we plot the Jüttner and the Modified
Jüttner distributions, exhibiting that the latter two can not adjust the
numerical data. In variance, the numerical simulations support very well the
LID. In the right panel of Fig. 2. we show in dashed blue, dashed red and
continuous black the temperature, $T$, as a function of $E_{J}$, $E_{MJ}$ and
$E$ respectively. Black points correspond to numerical simulations for
different temperature values. $T_{MJ}>T_{J}$ and $T_{MJ}>T$ for all values of
$\langle E\rangle$ . For $\langle m_{0}c^{2}\gamma(v)\rangle<2.127$, $T>T_{J}$
where $\langle\rangle$ denotes average over velocities. Note that for $\langle
m_{0}c^{2}\gamma(v)\rangle>2.127$, $T<T_{J}$. If $\langle
m_{0}c^{2}\gamma(v)\rangle=2.127$, $T_{J}=T=1.509$.

|
---|---
Figure 2: Left panel: Numerical results of the simulations. Black and red
points are the numerical results for heavy, $m_{H}$, and light, $m_{L}$,
particles, respectively. Dashed red and black lines are the adjustments using
$P_{LID}(\sigma)$, blue and purple dashed lines represent the Jüttner and the
Modified Jüttner distributions respectively. Note that all three distributions
asymptotically coincide only in the $v/c\rightarrow 0$ limit. Right panel:
Temperature $T$ as a function of the mean energy $\langle
m_{0}\gamma(v)c^{2}\rangle\equiv\langle E\rangle$ for $P_{J}(v)$, $P_{MJ}(v)$
and $P_{LID}(v)$. Dashed blue, dashed red and continuous black represent
$E_{J}$, $E_{MJ}$ and the present $E$ respectively. Black points correspond to
numerical simulations for typical temperature values. All three analytical
curves recover, at low temperatures, the Maxwell-Botzmann behavior: $\langle
E\rangle-m_{0}c^{2}\sim kT/2\;\;(T\to 0)$.

##  5 LID derivation from a MaxEnt principle

Given that the rapidity is a Lorentz-invariant quantity, we could redefine
$\tilde{\sigma}$ by convenience as $\tilde{\sigma}=\sigma/N$, then the entropy

$S=-k\int_{\infty}^{\infty}\,P(\tilde{\sigma})\ln
P(\tilde{\sigma})d\tilde{\sigma},$ (26)

can be maximized with the constraints

$\int_{\infty}^{\infty}\,P(\tilde{\sigma})d\tilde{\sigma}=1\,,\qquad\int_{\infty}^{\infty}\,\tilde{\sigma}^{2}P(\tilde{\sigma})d\tilde{\sigma}=\text{constant}\,.$
(27)

This yields

$\displaystyle P(\tilde{\sigma})d\tilde{\sigma}\equiv
P_{LID}(\sigma)d\sigma=\sqrt{\dfrac{\beta}{N\pi}}\,e^{-\frac{\beta\sigma^{2}}{N}}d\sigma\,,$
(28a) $\displaystyle\beta\equiv\dfrac{m_{0}c^{2}}{2kT}\,.$ (28b)

This distribution is a Lorentz-invariant one (LID).

##  6 Conclusions

We have discussed the original paper by Jüttner (1911) and we have shown that
his distribution, as well as a particular modification of the Jüttner
distribution, are not Lorentz-invariant distributions and, therefore, are not
the correct relativistic generalizations of the Maxwell distribution of
velocities of a gas. Essentially, the space of velocities considered by
Jüttner is a flat space while the correct relativistic velocity space is a
Lobachevsky space. Based on the relativistic addition of velocities and on the
Central Limit Theorem, we have derived a truly Lorentz-invariant distribution
(LID) that recovers the Maxwell distribution for $|v|/c\ll 1$ and has zero
probability to have particles with speed higher than the speed of light. We
have performed relativistic molecular dynamics simulations and our numerical
results confirm the LID for all values of $E$, whereas they are in odd with
both the Jüttner and the so-called Modified Jüttner distribution at
sufficiently high energies per particle of the gas. It is worth to notice that
all distributions $P_{J}(v)$, $P_{MJ}(v)$, $P_{LID}(v)$ and $P_{MB}(v)$ yield
relations between the mean energies and the associated temperatures, some of
which are shown in Fig. 2. Finally, an important consequence of the LID is
that the temperature is, as argued by Landsberg, a Lorentz-invariant quantity,
a crucial requirement to preserve the second law of thermodynamics in special
relativity.

We are working in extensions of these one-dimensional results for two and
three-dimensions. In these higher dimensions we have to consider the finite
size of the particles during collisions, in molecular dynamics simulations.
Also, the addition rules for the velocities are much more complicated and
there is no simple rule like the addition of rapidities in one dimension. But,
certainly, mainly in three-dimensions, finding a correct Lorentz-invariant
distribution is very important, including for the correct determination of the
temperature of stellar objects.

## acknowledgments

We acknowledge partial financial support by Conselho Nacional de
Desenvolvimento Científico e Tecnológico (CNPq), Fundação Carlos Chagas Filho
de Amparo à Pesquisa do Estado do Rio de Janeiro (FAPERJ), and Coordenação de
Aperfeiçoamento de Pessoal de Nível Superior (CAPES) (Brazilian agencies).

## References

  * [1] S. Sadegzadeh and A. Mousavi, Physics of Plasmas 25, 112107 (2018).
  * [2] S. M. Molnar and J. Godfrey, Astrophysical Journal 902, 143 (2020).
  * [3] D. B. Melrose, M. Z. Rafat, and A. Mastrano, MNRAS 500, 4530 (2021).
  * [4] K. Adcox and et al. (PHENIX Collaboration), Phys. Rev. Lett. 88, 022301 (2001).
  * [5] C. Adler and et al. (STAR Collaboration), Phys. Rev. Lett. 89, 202301 (2002).
  * [6] L. Marques, J. Cleymans, and A. Deppman, Phys. Rev. D 91, 054025 (2015).
  * [7] F. Jüttner, Annalen der Physik 339, 856 (1911).
  * [8] L. Boltzmann, Wiener Berichte 66, 275 (1872).
  * [9] L. Boltzmann, Wiener Berichte 76, 373 (1877).
  * [10] J. C. Maxwell, The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 19, 19 (1860a).
  * [11] J. C. Maxwell, The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 20, 21 (1860b).
  * [12] P. T. Landsberg, Nature 212, 571 (1966).
  * [13] P. T. Landsberg, Nature 214, 903 (1967).
  * [14] D. Cubero, J. Casado-Pascual, J. Dunkel, P. Talkner, and P. Hänggi, Phys. Rev. Lett. 99, 170601 (2007).
  * [15] A. Montakhab, M. Ghodrat, and M. Barati, Phys. Rev. E 79, 031124 (2009).
  * [16] M. Ghodrat and A. Montakhab, Computer Physics Communications 182, 1909 (2011), computer Physics Communications Special Edition for Conference on Computational Physics Trondheim, Norway, June 23-26, 2010\.
  * [17] E. M. F. Curado, F. T. L. Germani and I. D. Soares, Physica A: Statistical Mechanics and its Applications 444, 963-969 (2016).
  * [18] N. Kubli and H. J. Herrmann, Physica A Statistical Mechanics and its Applications 561, 125261 (2021).
  * [19] E. Lehmann, Journal of Mathematical Physics 47, 023303 (2006).
  * [20] J. Dunkel and P. Hänggi, Physica A: Statistical Mechanics and its Applications 374, 559 (2007).
  * [21] V. Fock,  _The Theory of Space, Time and Gravitation_, 2nd ed. (Pergamon Press, 1964) Chap. I - the theory of relativity, pp. 9–53.


# Out-of-Distribution Detection in Dermatology using Input Perturbation and
Subset Scanning

Hannah Kim
Duke University
Durham, NC, USA
&Girmaw Abebe Tadesse
IBM Research
Nairobi, Kenya
&Celia Cintas
IBM Research
Nairobi, Kenya
&Skyler Speakman
IBM Research
Nairobi, Kenya
Kush Varshney
IBM Research
Yorktown Heights, NY, USA

###### Abstract

Recent advances in deep learning have led to breakthroughs in the development
of automated skin disease classification. As we observe an increasing interest
in these models in the dermatology space, it is crucial to address aspects
such as the robustness towards input data distribution shifts. Current skin
disease models could make incorrect inferences for test samples from different
hardware devices and clinical settings or unknown disease samples, which are
out-of-distribution (OOD) from the training samples.To this end, we propose a
simple yet effective approach that detect these OOD samples prior to making
any decision. The detection is performed via scanning in the latent space
representation (e.g., activations of the inner layers of any pre-trained skin
disease classifier). The input samples could also perturbed to maximise
divergence of OOD samples. We validate our ODD detection approach in two use
cases: 1) identify samples collected from different protocols, and 2) detect
samples from unknown disease classes. Additionally, we evaluate the
performance of the proposed approach and compare it with other state-of-the-
art methods. Furthermore, data-driven dermatology applications may deepen the
disparity in clinical care across racial and ethnic groups since most datasets
are reported to suffer from bias in skin tone distribution. Therefore, we also
evaluate the fairness of these OOD detection methods across different skin
tones. Our experiments resulted in competitive performance across multiple
datasets in detecting OOD samples, which could be used (in the future) to
design more effective transfer learning techniques prior to inferring on these
samples.

_Keywords_ Skin disease classification $\cdot$ Out-of-distribution sample
detection $\cdot$ Algorithmic Fairness

##  1 Introduction

Skin disease remains a global health challenge, with skin cancer being the
most common cancer worldwide (Codella et al., 2017). Following the recent
success of deep learning (DL) in various computer vision problems (partly due
to its automated feature encoding capability), convolutional neural networks
(CNNs) (Huang et al., 2016) have been employed for skin disease classification
tasks. As we observe increasing interest in DL in applying dermatology (Esteva
et al., 2017; Gomolin et al., 2020), it is imperative to address transparency,
robustness, and fairness of these solutions (Adamson and Smith, 2018; Qayyum
et al., 2020). While many existing deep learning techniques (Mahbod et al.,
2020; Gessert et al., 2018; Ahmed et al., 2019) achieve high performance on
publicly available datasets (Codella et al., 2017; Tschandl et al., 2018;
Combalia et al., 2019; Sun et al., 2016), they utilize ensembles of multiple
models aimed at maximising performance with limited consideration to shifts in
the input data (Ahmed et al., 2019; Gessert et al., 2019; Zhang et al., 2019),
which might result in incorrectly classifying new samples as one of the
training classes (with high confidence) though these samples might be from
previously unknown or new classes.

Thus, it is necessary to detect out-of-distribution (OOD) samples prior to
making decisions in order to achieve principled transfer of knowledge from in-
distribution training samples to OOD test samples, thereby extending the
usability of the models to previously unseen scenarios. Furthermore, OOD
detectors and other DL solutions need to guarantee equivalent detection
capability across sub-populations. Particularly in dermatology, bias in
representations of skin tones in academic materials (Mcf, ) and clinical care
(rab, ) is becoming a primary concern. For instance, New York Times reports
major disparities in dermatology when treating skin of color (rab, ) as common
conditions often manifest differently on dark skin, and physicians are trained
mostly to diagnose them on light skin. STAT (Mcf, ) also reported that lack of
darker skin tones in dermatology academic materials adversely affects the
quality of care for patients of color. Alarmingly, the growing practice of
using artificial intelligence to aid the diagnosis of skin diseases will
further deepen the divide in patient care because of the machine learning
algorithms, which are trained with such imbalanced datasets (Codella et al.,
2019, 2017; Tschandl et al., 2018; Combalia et al., 2019; Sun et al., 2016)
(with overwhelming majority of samples with light skin tones). This is
supported by the work of Kinyanjui et al. (Kinyanjui et al., 2019), which use
Individual Typology Angle (ITA) to approximate skin tones in various publicly
available skin disease datasets (Codella et al., 2017; Tschandl et al., 2018;
Combalia et al., 2019; Sun et al., 2016) and show that these datasets heavily
under-represent darker skin tones. As a result, we also validated the
performance of

To this end, we propose a simple yet effective approach that scans over the
activations of the inner layers of any pre-trained skin disease classifier to
detect OOD samples. We additionally perturb the input data beforehand with our
proposed ODINlow, a modification of ODIN (Liang et al., 2017), which improve
OOD detection performance in earlier layers of the network. In our framework,
we define two different OOD use cases: protocol variations (e.g., different
hardware devices, lighting settings and not compliant with clinical protocol);
and unknown disease types (e.g., samples from new disease type that was not
observed during training). Without requiring any prior knowledge of the OOD
samples, the proposed approach out-performed existing OOD detectors, softmax
score (Hendrycks and Gimpel, 2016) and ODIN (Liang et al., 2017), for OOD
samples with different validation protocols, and competitive performance is
achieved in detecting samples with unknown disease types. We further explore
how our proposed and existing OOD detectors perform across skin tones to
evaluate fairness. We show that the current OOD detectors show higher
performance in detecting darker skin tones as OOD samples than those of
lighter skin tones, which is likely impacted by the imbalanced training skin
datasets that heavily lack samples of dark skin tones.

Generally, our main contributions are highlighted as follows: 1) We propose a
weakly-supervised approach based on subset scanning over the activations of
the inner layers of a pre-trained skin disease classifier to detect OOD
samples across two use cases: detection of OOD samples from different
collection protocol and those from unknown disease classes; 2)We propose
perturb input images with ODINlow noise, for improved OOD detection
performance;3) We evaluate our methods against existing OOD detectors: Softmax
Score (Hendrycks and Gimpel, 2016) and ODIN (Liang et al., 2017); Furthermore,
we evaluate the fairness of the proposed approach and existing methods in
their detection performance across skin tones.

| Ensemble |  | Test Data
---
Augmentation
| OOD Detection
---
Post-Training
| New Protocol
---
Detection
| New Disease
---
Detection
| Algorithmic
---
Fairness
(Ahmed et al., 2019) | ✓ | ✓ | ✗ | ✗ | ✓ | ✗
(Zhang et al., 2019) | ✓ | ✗ | ✗ | ✗ | ✓ | ✗
(Gessert et al., 2019) | ✓ | ✓ | ✗ | ✗ | ✓ | ✗
(Bagchi et al., 2020) | ✗ | ✗ | ✗ | ✗ | ✓ | ✗
(Pacheco et al., 2019) | ✓ | ✗ | ✓ | ✗ | ✓ | ✗
(Combalia et al., In Press) | ✗ | ✓ | ✓ | ✗ | ✓ | ✗
(Pacheco et al., 2020) | ✗ | ✗ | ✓ | ✓ | ✓ | ✗
Ours | ✗ | ✗ | ✓ | ✓ | ✓ | ✓

Table 1: Summary of the state-of-the-art OOD sample detection in skin disease
classification task, and the differentiation of our proposed approach.

##  2 Related Work

Our review of existing OOD detection methods is grouped into pre-training
(Ahmed et al., 2019; Bagchi et al., 2020; Gessert et al., 2019; Zhang et al.,
2019) and post-training (Combalia et al., In Press; Pacheco et al., 2019,
2020), based on where the detection step is applied.

Pre-training OOD detection approaches have prior knowledge of the OOD samples
and incorporate it during their training phases. Many of these approaches
utilize ensembles of existing CNNs (and their variants) to detect OOD samples
(Ahmed et al., 2019; Gessert et al., 2019; Zhang et al., 2019). Ahmed et al.
(Ahmed et al., 2019) applied one-class learning using deep neural network
features where one-class samples were iteratively discarded as OOD samples in
a one-vs-all cross-validation strategy, and the OOD samples were detected by
taking the prediction average of all the models. Gessert et al. (Gessert et
al., 2019) utilized an additional dataset of skin lesions as OOD samples to
train their ensemble of CNNs to detect OODs. Zhang et al. (Zhang et al., 2019)
employed an ensemble DenseNet-based CNNs consisting of both multi-class and
binary classifiers to detect OOD samples. Bagchi et al. (Bagchi et al., 2020)
proposed Class Specific - Known vs. Simulated Unknown to detect OOD samples.

Post-training OOD detection approaches do not require any prior knowledge of
the OOD samples during training (Combalia et al., In Press; Pacheco et al.,
2019, 2020). Pacheco et al. (Pacheco et al., 2019) detected OOD samples using
Shannon entropy (Shannon, 1948) and cosine similarity metrics on their CNN’s
probability outputs. Instead, Combalia et al. (Combalia et al., In Press)
detected OOD samples using Monte-Carlo Dropout (Gal and Ghahramani, 2016) and
test data augmentation to estimate uncertainty such as entropy and variance in
their network predictions. Pacheco et al. (Pacheco et al., 2020) extended
Gram-OOD (Sastry and Oore, 2019) with layer-specific normalization of Gram
Matrix values to detect OOD samples.

Table 1 summarizes notable OOD detection studies in dermatology. The majority
of these studies employ pre-training approaches using ensembles of CNNs, which
result in model complexity and impracticality due to their need of prior
knowledge of OOD samples. Test data augmentation is also less plausible to
domain experts as it might partially re-synthesize the samples. In this work,
we propose a simple, post-training OOD detector that can be applied to any
single pre-trained network without any test data augmentation nor prior
knowledge of the OOD samples.

Figure 1: Block diagram of the proposed approach. $C$: a trained model for
skin disease classification over mentioned datasets ($\mathcal{D}_{1}$,
$\mathcal{D}_{2}$); $T$: a skin tone extractor.

##  3 Proposed Framework

We propose a weakly-supervised OOD detection method to identify skin images
collected in different validation protocols and derived from unknown skin
disease types, based on subset scanning (Cintas et al., 2020) and ODIN (Liang
et al., 2017). Subset scanning treats the OOD detection problem as a search
for the most anomalous subset of observations in the activation space of any
pre-trained classifier. This exponentially large search space is efficiently
explored by exploiting mathematical properties of our measure of anomalousness
(Neill, 2012). Our solution can be applied to any off-the-shelf skin disease
classifier. Additionally, we evaluate algorithmic fairness of the proposed and
existing OOD detectors across skin tones. The overview of the proposed
approach is shown in Fig. 1. Given a set of skin datasets $D$ and a pre-
trained skin disease classifier $C$ as an input; first, we stratify each
dataset through a skin tone distribution extractor $T$ for evaluation
purposes. Then, we apply subset scanning across each layer of the classifier
$C$ and compute the subset score for the unknown disease use case. To detect
protocol variations, we first perturb the input data for the best performing
results. In the following sections, we describe the details of the proposed
approach.

###  3.1 Subset scanning for out-of-distribution sample detection

Given a pre-trained network $C$ for skin disease classification, we apply
subset scanning (Cintas et al., 2020) on the activations in the intermediate
layers of the network $C$ to detect a subset ($S$) of OOD samples (see
Algorithm 1). Subset scanning searches for the most anomalous subset
$S^{*}=\arg\max_{S}F(S)$ in each layer, where the anomalousness is quantified
by a scoring function $F(\cdot)$, such as a log-likelihood ratio statistic.
When searching for this subset, an exhaustive search across all possible
subsets is computationally infeasible as the number of subsets ($2^{N}$)
increases exponentially with the number of nodes ($N$) in a layer. Instead, we
utilize a scoring function that satisfies the Linear Time Subset Scanning
(LTSS) (Neill, 2012) property, which enables efficient maximization over all
subsets of data. This LTSS property guarantees that the highest-scoring subset
of nodes in a layer are identified within $N$ searches instead of $2^{N}$
searches. Following the literature on pattern detection (McFowland et al.,
2013), we utilize non-parametric scan statistics (NPSS) (McFowland et al.,
2013) as our scoring function as it satisfies LTSS property and makes minimal
assumptions on the underlying distribution of node activations.

We apply subset scanning on set of layers $C_{Y}$ of our pre-trained network
$C$. For each layer $C_{y}\in C_{Y}$, we form a distribution of expected
activations at each node using the known in-distribution (ID) samples $X_{z}$,
which were used during training and can also be referred as background images.
Comparing this expected distribution to the node activations of each test or
evaluation sample $X_{i}$, we can obtain p-values $p_{ij}$ for each $i^{th}$
test sample and $j^{th}$ node of layer $C_{y}$. We can then quantify the
anomalousness of the p-values by finding the subset of nodes that maximize
divergence of the test sample activations from the expected. This yields
$|C_{Y}|$ anomalous scores $S^{\ast}_{(C_{y})}$ for each test sample. We
expect OOD samples to yield higher anomalous scores $S$ than ID samples, and
we detect OOD samples with simple thresholding. Note that the OOD detection is
performed in an unsupervised fashion without any prior knowledge of the OOD
samples.

1

input : Background Image: $X_{z}\in D^{H_{0}}$, Evaluation Image: $X_{i}$,
training dataset: $D_{train}$, $\alpha_{\text{max}}$.

output : $AUROC$, $F_{1}$, $AUROC^{t}$, and $F_{1}^{t}$ for $X_{i}$

2 $C\leftarrow$ TrainSkinDiseaseClassifier ($D_{train}$);

3 $C_{Y}\leftarrow$ Set of layers in $C$;

4 $X_{i}^{t}\leftarrow$ PredictITASkinTone ($X_{i}$);

5

6$\hat{X}_{z}\leftarrow$ AddODINNoise ($X_{z}$);

7 $\hat{X}_{i}\leftarrow$ AddODINNoise ($X_{i}$) ;

8 for  _$C_{y}$ in $C_{Y}$_ do

9 for  _$j\leftarrow 0$ to $|C_{y}|$_ do

10 $A^{H_{0}}_{zj}\leftarrow$ ExtractActivation ($C_{y}$, $\hat{X}_{z}$);

11

12 $A_{ij}\leftarrow$ ExtractActivation ($C_{y}$,$\hat{X}_{i}$);

13

14 $p_{ij}=\frac{\sum_{X_{z}\in D^{H_{0}}}I(A_{zj}>=A_{ij})+1}{M+1}$;

15 $p^{\ast}_{ij}=\\{y<\alpha_{\text{max}}\>\forall\>y\subseteq p_{ij}\\}$;

16 $p^{s}_{ij}\leftarrow$ SortAscending ($p^{\ast}_{ij}$);

17 for  _$k\leftarrow 1$ to $|C_{y}|$_ do

18 $S_{(k)}=\\{p_{y}\subseteq p^{s}_{ij}\forall y\in\\{1,\ldots,k\\}\\}$;

19 $\alpha_{k}=max(S_{(k)})$;

20 $F(S_{(k)})\leftarrow$ NPSS ($\alpha_{k}$, k, k);

21

22 $k^{\ast}_{(C_{y})}\leftarrow\arg\max F(S_{(k)})$;

23 $\alpha^{\ast}_{(C_{y})}=\alpha_{k^{\ast}_{(C_{y})}}$;

24 $S^{\ast}_{(C_{y})}=S_{(k^{\ast}_{(C_{y})})}$;

25

26$AUROC$, $F_{1}$ = ComputeDetectionPerformance
($\sum_{C_{y}}{S^{\ast}_{(C_{y})}}$);

27 $AUROC^{t}$, $F_{1}^{t}$ = StratifyPerSkinTone(_$X_{i}^{t}$ , $AUROC$,
$F_{1}$_);

return $AUROC$, $F_{1}$, $AUROC^{t}$, and $F_{1}^{t}$

Algorithm 1 Pseudo-code for the proposed new protocol (OOD) detection.

###  3.2 ODIN and ODINlow Perturbations

We have also evaluated the impact of adding small perturbations, prior to
subset scanning, to each test sample following ODIN (Liang et al., 2017) for
enhanced OOD. ODIN involves two steps, input pre-processing and temperature
scaling. In the first step, $X_{i}$ is perturbed by adding a small
perturbation computed by back-propagating the gradient of the training loss
with respect to $X_{i}$ and weighted by parameter $\epsilon$. This pre-
processed $X_{i}$ is then fed into the neural network and temperature scaling
with parameter $\tau$ is applied in the final softmax layer $C_{s}$. The two
hyperparemters, $\epsilon$ and $\tau$, are chosen so that the OOD detection
performance of softmax score (Hendrycks and Gimpel, 2016), the maximum value
of the softmax layer output, is optimized. We further modified ODIN and
propose ODINlow with parameters $\tau_{low}$ and $\epsilon_{low}$ that leads
to the lowest softmax score performance. As subset scanning is applied not
only on the softmax layer but also on the the inner layers of the network, we
show that ODINlow helps improve OOD detection in the earlier layers of the
network.

###  3.3 Algorithmic Fairness of OOD detectors across skin tone

We further evaluate algorithmic fairness of our proposed OOD dectector across
skin tones, estimated by adopting an existing framework (Kinyanjui et al.,
2019). To this end, the non-diseased regions of a given skin image are
segmented using Mask R-CNN (He et al., 2017), and individual typology angle
(ITA) values are computed as
$ITA=\arctan\left(\frac{L_{\mu}-50}{b_{\mu}}\right)\times\frac{180^{\circ}}{\pi}$,
where $L_{\mu}$ and $b_{\mu}$ are the average of luminance and yellow values
of non-diseased pixels in CIELab-space. ITA values are used to stratify the
samples into three Fitzpatrick skin tone categories, Light, Intermediate, and
Dark, as shown in Table 2.

ITA Range |  Skin Tone Category
---|---
$ITA>41^{\circ}$ |  Light
$28^{\circ}<ITA\leq 41^{\circ}$ |  Intermediate
$ITA\leq 28^{\circ}$ |  Dark
Table 2: Summary of Fitzpatrick skin tone categorization of computed $ITA$
values.

##  4 Datasets

We validate the proposed frame work using two datasets: ISIC 2019 (Codella et
al., 2017; Tschandl et al., 2018; Combalia et al., 2019) for samples of
unknown diseases; and SD-198 (Sun et al., 2016) for samples from unknown
collection protocols. We stratify the samples from both datasets based on
skin-tones to observe the impact of various OOD methods across the population
spectrum (see Figure 2).

###  4.1 ISIC 2019

ISIC 2019 (Codella et al., 2017; Tschandl et al., 2018; Combalia et al., 2019)
dataset is an extension of ISIC 2018 and merges HAM10000 (Tschandl et al.,
2018), BCN20000 (Combalia et al., 2019), and MSK (Codella et al., 2017)
datasets. It consists of $25,331$ dermoscopic images among eight diagnostic
categories: Melanoma, Melanocytic nevus, Basal cell carcinoma, Actinic
keratosis, Benign keratosis, Dermatofibroma, Vascular lesion, and Squamous
cell carcinoma. As its test set is not available publicly, we set aside
Dermatofibroma (DF) and Vascular lesion (VASC) samples during training, and
utilize them during the test time as OOD samples of unknown diseases. These
two classes are chosen as they contain the least number of samples in the
dataset. First row of Figure 2 show example images of this dataset for each of
the three skin tone categories we consider in this work.

###  4.2 SD-198

SD-198 (Sun et al., 2016) dataset contains $198$ different diseases from
different types of eczema, acne and various cancerous conditions, totalling
$6,584$ images. The images are collected via various devices, mostly digital
cameras and mobile phones with higher levels of noise and varying
illumination. We use this dataset for OOD samples that are collected from
unknown protocols. We show some example images of the dataset in the second
row of Figure 2 that are stratified into three skin-tone categories, Light,
Intermediate, and Dark.

|  |
---|---|---
|  |
Light | Intermediate | Dark
Figure 2: Example images from ISIC 2019 (Codella et al., 2017) (top) and
SD-198 (Sun et al., 2016) (bottom) stratified into three skin tone categories:
Light, Intermediate, and Dark.

##  5 Experimental setup

###  5.1 Skin disease model setup

We adopt DenseNet-121 (Huang et al., 2016) pre-trained on ImageNet (Deng et
al., 2009) for the skin disease classification task and fine-tune it on ISIC
2019 (Codella et al., 2017). To accommodate for the change in number of
classes for the skin disease classification task, we resize the last four
fully connected layers of DenseNet to $512$, $256$, $128$, and $7$ nodes
followed by a SoftMax with $7$ nodes for the seven skin disease classes. We
use Adam (Kingma and Ba, 2015) optimization with a learning rate of $1e^{-4}$
and a batch size of $40$. To address the class imbalance problem, we employ
weighted cross-entropy loss. The implementation is done with the Python 3.6
(Harris et al., 2020) and TensorFlow 1.14 (Abadi et al., 2016). To validate
detection of unknown disease samples, we use DF and VASC classes from
ISIC-2019, consisting of $253$ and $225$ samples, respectively. Similarly, for
samples with different collection protocols, we extract $10$ sets of $260$
samples from SD-198 and report their aggregate performance.

###  5.2 Subset scanning setup

We apply subset scanning across eight layers $C_{Y}$ consisting of six
convolutional layers $(C_{conv_{1}},...,C_{conv_{6}})$, global pooling layer
$(C_{gp})$, and softmax layer $(C_{s})$. For ODIN (Liang et al., 2017), we use
temperature scaling parameter $\tau=10$ and perturbation magnitude
$\epsilon=0$ (optimized on ISIC-2019) for SD-198 samples and $\tau=5$ and
$\epsilon=0.0002$ (optimized on SD-198) for ISIC-2019 samples. For ODINlow, we
use $\tau_{low}=2$ and $\epsilon_{low}=0.2$, which leads to AUROC equal to 0.5
for Softmax Score for both OOD use cases. We employ Area Under Receiver
Operating Characteristic Curve (AUROC) and maximum $F_{1}$-score ($F_{1}$) as
our metrics to evaluate the OOD detection performance.

##  6 Results

In this section, we show the result of proposed OOD detector with subset
scanning and ODIN as detailed in Section 3. We first compare our result of OOD
detection to Softmax Score (Hendrycks and Gimpel, 2016) and ODIN (Liang et
al., 2017) in Tables 3 for OOD samples with different collection protocol and
in 4 for OOD samples with unknown disease types. We further stratify OOD
samples based on skin tone for these approaches and report their performance
in Table 5. We show in Figure 3 the detection performance of our proposed
method on individual layers across our network and further stratify these
performances across skin tone in Figure 4.

Methods | AUROC | $F_{1}$
---|---|---
Softmax Score (Hendrycks and Gimpel, 2016) | $74.4\pm 1.7$ | ${71.0\pm 1.1}$
ODIN (Liang et al., 2017) | ${74.5\pm 1.6}$ | $70.8\pm 1.1$
SS ($C_{s}$) | $68.2\pm 1.4$ | $71.3\pm 0.5$
SS ($C_{gp}$) | $62.7\pm 1.2$ | ${72.5\pm 0.6}$
SS ($C_{conv_{1}}$) | $41.6\pm 1.8$ | $68.1\pm 0.2$
SS ($C_{s}$)+ODIN | $51.2\pm 1.9$ | $67.9\pm 0.3$
SS ($C_{conv_{1}}$)+ODINlow | $85.4\pm 0.6$ | $81.9\pm 0.6$
SS (Sum All Layers)+ODINlow | $\mathbf{{91.0\pm 0.8}}$ | $\mathbf{{86.9\pm 1.1}}$
Table 3: Detection performance for OOD samples of unknown collection protocols
validated with SD-198 (Sun et al., 2016). Bold values are the best performers
in each column.

###  6.1 OOD samples from a different protocol or equipment

We first show the result of detecting OOD samples that are collected with
different protocols or equipment. Table 3 summarizes the results of the
proposed approach - subset scanning (SS) with and without noise, and compared
with the existing baselines (Hendrycks and Gimpel, 2016; Liang et al., 2017).
In the top panel, we see that ODIN (Liang et al., 2017) increases the AUROC
performance of Softmax Score by around 0.1 on average. For samples with ODIN
noise, we show the performance of subset scanning on the softmax layer
$C_{s}$, as ODIN is optimized on Softmax Score, and for samples with ODINlow
noise, we show the result of subset scanning on the first convolutional layer
($C_{conv_{1}}$). We achieve the best performance with AUROC of $91.0\pm 0.8$
and maximum $F_{1}$-score of $86.9\pm 1.1$ using the sum of subset scores
$S^{*}_{(C_{y})}$ across all eight layers with ODINlow (bottom row in Table
3).

Methods | AUROC | $F_{1}$
---|---|---
DF | VASC | DF | VASC
Softmax Score (Hendrycks and Gimpel, 2016) | 80.9 | 73.2 | 76.5 | 70.5
ODIN (Liang et al., 2017) | 72.3 | 65.3 | 70.3 | 67.4
SS ($C_{s}$) | 80.8 | 70.8 | 75.7 | 72.3
SS ($C_{gp}$) | 37.4 | 57.9 | 65.9 | 69.2
SS ($C_{conv_{1}}$) | 50.9 | 62.5 | 65.8 | 68.7
SS ($C_{s}$)+ODIN | 71.8 | 63.3 | 70.4 | 67.4
SS ($C_{conv_{1}}$)+ODINlow | 47.6 | 39.8 | 65.9 | 67.1
SS (Sum All Layers)+ODINlow | 47.6 | 40.4 | 65.9 | 67.2
Table 4: Performances of detecting OOD samples of unknown disease types, DF
and VASC. Bold values are the best performers in each column.

###  6.2 OOD samples of unknown diseases

Table 4 shows the performance of detecting OOD samples of unknown diseases (DF
and VASC) that are unseen during training. While Softmax Score (Hendrycks and
Gimpel, 2016) yields the best performance, subset scanning on the softmax
layer $C_{s}$ shows comparable performance. We see worse performances with
ODIN as these OOD samples are from the same dataset as ID samples and adding
noise likely blurs the unique features present in each skin disease class.

Methods | Skin Tone | Unknown diseases | Collection protocol
---|---|---|---
DF | VASC | SD-198
R | AUROC | R | AUROC | R | AUROC
Softmax Score (Hendrycks and Gimpel, 2016) | Light | 171 | 81.0 | 185 | 72.1 | 986 | 75.8
Intermediate | 52 | 80.7 | 58 | 75.8 | 1278 | 73.7
Dark | 10 | 74.9 | 9 | 77.0 | 326 | 73.2
ODIN (Liang et al., 2017) | Light | 171 | 71.6 | 185 | 64.0 | 986 | 76.2
Intermediate | 52 | 69.9 | 58 | 64.9 | 1278 | 73.8
Dark | 10 | 86.3 | 9 | 89.4 | 326 | 72.1
SS ($C_{s}$) | Light | 171 | 78.6 | 185 | 70.7 | 986 | 68.3
Intermediate | 52 | 87.0 | 58 | 71.3 | 1278 | 68.0
Dark | 10 | 87.6 | 9 | 69.5 | 326 | 68.6
SS ($C_{s}$)+ODIN | Light | 171 | 69.7 | 185 | 62.7 | 986 | 52.1
Intermediate | 52 | 73.8 | 58 | 63.1 | 1278 | 50.6
Dark | 10 | 88.2 | 9 | 74.5 | 326 | 50.9
SS ($C_{conv_{1}}$) \+ ODINlow | Light | 171 | 45.1 | 185 | 38.8 | 986 | 83.1
Intermediate | 52 | 49.9 | 58 | 37.8 | 1278 | 86.7
Dark | 10 | 63.6 | 9 | 68.4 | 326 | 87.2
SS (Sum All Layers) \+ ODINlow | Light | 171 | 45.1 | 185 | 38.4 | 986 | 89.3
Intermediate | 52 | 51.8 | 58 | 40.0 | 1278 | 92.0
Dark | 10 | 56.2 | 9 | 78.7 | 326 | 92.3
Table 5: Performance of methods in Tables 3 and 4 stratified into three
different skin tone categories. $R$ represents the number of OOD samples in
each category.

###  6.3 Performance stratified by skin-tone

We further stratify the OOD samples into three skin tone categories and show
the results in Table 5. In each set of columns, we include the number of test
samples $R$ for each skin tone category and its corresponding AUROC
performance. Samples of Dark skin tones constitute only around $3.9\%$ of DF
and VASC samples and around $13\%$ of SD-198 samples. Majority of the listed
methods (13 out of 18), show higher detection performance of Dark OOD samples.
This could be partially because network is trained on the ISIC 2019 dataset
that heavily lacks samples of dark skin tones, and thus easily detects OOD
samples of dark skin tone to be out of distribution. Overall, it requires
further investigation to clearly understand whether such performance reveals
the lack of Dark samples in these datasets or variant manifestations of skin
diseases in Dark skin.

SS | SS+ODIN | SS+ODINlow
---|---|---
|  |
Figure 3: AUROC performance of subset scanning (SS) across various layer of
DenseNet-121 that we consider. First column shows the results without any
ODIN, the other two columns show the result with ODIN and ODINlow,
respectively for OOD samples of DF (yellow), VASC (green), and SD-198 (red).

###  6.4 OOD detection across individual layers

Figure 3 shows the OOD detection performance in terms of AUROC of our proposed
work on the eight layers of our pre-trained CNN in $C_{Y}$ that we consider.
The first column shows the result of subset scanning without any added noise,
and the other columns show the result of applying ODIN (Liang et al., 2017)
and ODINlow perturbations, respectively, to our test images before applying
subset scanning. In each sub-plot, we show results of both use cases, i.e.,
detection of samples of unknown diseases (DF (yellow), VASC (green)) and
samples from different protocols (SD-198 (red)). Overall, DF and VASC samples
from ISIC 2019 dataset have similar performance across the eight layers we
consider while samples from SD-198 dataset leads to varying performances
depending on the layer and ODIN parameters. This is partly because DF and VASC
samples are from the same distribution as the training set as they are both
from the same ISIC 2019 dataset, while SD-198 has different distribution than
the training set of ISIC 2019 with different collection protocol. Comparing
the last two plots, we see that standard ODIN leads to better performance near
the end of the network while ODINlow leads to better performance in earlier
layers of the network. This is as expected as ODIN parameters ($\tau$ and
$\epsilon$) are optimized on the Softmax Scores while ODINlow parameters,
$\tau_{low}$ and $\epsilon_{low}$, are not.

SS | SS+ODIN | SS+ODINlow
---|---|---
|  |
|  |
|  |
Figure 4: Change in performance ($\Delta$ AUROC) of OOD detection in Figure 3
for SD-198 (top), DF (middle) and VASC (bottom) stratified into three
different skin-tone categories, Light (blue), Intermediate (magenta), and Dark
(cyan). First column shows the results without any ODIN, the other two columns
show the result with ODIN and ODINlow, respectively.

We further stratify the performance of individual layers based on skin tone
represented in the samples and show the change in AUROC with the
stratification in Figure 4. While the samples of Light (blue) and Intermediate
(magenta) skin tones show consistent performances throughout the layers, we
see varying performances for samples of Dark (cyan) skin tones. This
instability of performance for samples of Dark skin tones may be partially
because network is trained on the ISIC 2019 dataset that heavily lacks samples
of Dark skin tones.

##  7 Conclusion

We propose a weakly-supervised method to detect out-of-distribution (OOD) skin
images (collected in different protocols or from unknown disease types) using
input perturbation and scanning of the activations in the intermediate layers
of pre-trained on-the-shelf classifier. The scanning of activations is
optimised as a search problem to identify nodes in a layer that results in
maximum divergence of the activations from subset of test samples compared to
the expected activations derived from the training (in-distribution) samples.
We exploited Linear Time Subset Scanning (LTSS) (Neill, 2012) property of
subset scanning to achieve efficient search that scales linearly with the
number of nodes in the a layer. Our proposed method improves on the state-of-
the-art detection for OOD samples that are collected from a different protocol
or equipment than those in-distribution samples used to train the classifier,
and it achieves competitive performance with the state-of-the-art in detecting
samples of unknown diseases. We further stratify these OOD samples based on
three skin tone categories, Light, Intermediate, and Dark. From our results we
observe imbalanced detection performance across skin tones, where the Dark
samples are detected as OOD with higher performance. Thus, future work aims to
understand the reasons for such detection disparity across skin tones, e.g.,
lack of training representation or different manifestation of skin diseases.

## References

  * Codella et al. [2017] Noel C. F. Codella, David Gutman, M. Emre Celebi, Brian Helba, Michael A. Marchetti, Stephen W. Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin K. Mishra, Harald Kittler, and Allan Halpern.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (ISIC).  _CoRR_ , abs/1710.05006, 2017.  URL http://arxiv.org/abs/1710.05006.
  * Huang et al. [2016] Gao Huang, Zhuang Liu, and Kilian Q. Weinberger.  Densely connected convolutional networks.  _CoRR_ , abs/1608.06993, 2016.  URL http://arxiv.org/abs/1608.06993.
  * Esteva et al. [2017] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun.  Dermatologist-level classification of skin cancer with deep neural networks.  _Nature_ , 542(7639):115–118, 2017.
  * Gomolin et al. [2020] Arieh Gomolin, Elena Netchiporouk, Robert Gniadecki, and Ivan V Litvinov.  Artificial intelligence applications in dermatology: where do we stand?  _Frontiers in medicine_ , 7, 2020.
  * Adamson and Smith [2018] Adewole S Adamson and Avery Smith.  Machine learning and health care disparities in dermatology.  _JAMA dermatology_ , 154(11):1247–1248, 2018\.
  * Qayyum et al. [2020] Adnan Qayyum, Junaid Qadir, Muhammad Bilal, and Ala Al-Fuqaha.  Secure and robust machine learning for healthcare: A survey.  _arXiv preprint arXiv:2001.08103_ , 2020.
  * Mahbod et al. [2020] Amirreza Mahbod, Gerald Schaefer, Chunliang Wang, Georg Dorffner, Rupert Ecker, and Isabella Ellinger.  Transfer learning using a multi-scale and multi-network ensemble for skin lesion classification.  _Computer Methods and Programs in Biomedicine_ , 193:105475, 03 2020.
  * Gessert et al. [2018] Nils Gessert, Thilo Sentker, Frederic Madesta, Rüdiger Schmitz, Helge Kniep, Ivo M. Baltruschat, René Werner, and Alexander Schlaefer.  Skin lesion diagnosis using ensembles, unscaled multi-crop evaluation and loss weighting.  _CoRR_ , abs/1808.01694, 2018.  URL http://arxiv.org/abs/1808.01694.
  * Ahmed et al. [2019] Sara Atito Ali Ahmed, Berrin Yanikoglu, Erchan Aptoula, and Ozgu Goksu.  Skin lesion classification with deep learning ensembles in isic 2019.  2019\.
  * Tschandl et al. [2018] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler.  The HAM10000 dataset: A large collection of multi-source dermatoscopic images of common pigmented skin lesions.  _CoRR_ , abs/1803.10417, 2018.  URL http://arxiv.org/abs/1803.10417.
  * Combalia et al. [2019] Marc Combalia, Noel C. F. Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C. Halpern, Susana Puig, and Josep Malvehy.  Bcn20000: Dermoscopic lesions in the wild.  2019\.
  * Sun et al. [2016] Xiaoxiao Sun, Jufeng Yang, Ming Sun, and Kai Wang.  A benchmark for automatic visual classification of clinical skin disease images.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, _Computer Vision – ECCV 2016_ , pages 206–222, Cham, 2016. Springer International Publishing.
  * Gessert et al. [2019] Nils Gessert, Maximilian Nielsen, Mohsin Shaikh, René Werner, and Alexander Schlaefer.  Skin lesion classification using loss balancing and ensembles of multi-resolution efficientnets.  2019\.
  * Zhang et al. [2019] Pengyi Zhang, Yunxin Zhong, and Xiaoqiong Li.  Melanet: A deep dense attention network for melanoma detection in dermoscopy images.  2019\.
  * [15] Dermatology faces a reckoning: Lack of darker skin in textbooks and journals harms care for patients of color.
  * [16] Dermatology has a problem with skin color.
  * Codella et al. [2019] Noel C. F. Codella, Veronica Rotemberg, Philipp Tschandl, M. Emre Celebi, Stephen W. Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael A. Marchetti, Harald Kittler, and Allan Halpern.  Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (ISIC).  _CoRR_ , abs/1902.03368, 2019.  URL http://arxiv.org/abs/1902.03368.
  * Kinyanjui et al. [2019] Newton M. Kinyanjui, Timothy Odonga, Celia Cintas, Noel C. F. Codella, Rameswar Panda, Prasanna Sattigeri, and Kush R. Varshney.  Estimating skin tone and effects on classification performance in dermatology datasets.  2019\.
  * Liang et al. [2017] Shiyu Liang, Yixuan Li, and R. Srikant.  Principled detection of out-of-distribution examples in neural networks.  _CoRR_ , abs/1706.02690, 2017.  URL http://arxiv.org/abs/1706.02690.
  * Hendrycks and Gimpel [2016] Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _CoRR_ , abs/1610.02136, 2016.  URL http://arxiv.org/abs/1610.02136.
  * Bagchi et al. [2020] Subhranil Bagchi, Anurag Banerjee, and Deepti R. Bathula.  Learning a meta-ensemble technique for skin lesion classification and novel class detection.  In _CVPR Workshops_ , June 2020.
  * Pacheco et al. [2019] Andre G. C. Pacheco, Abder-Rahman Ali, and Thomas Trappenberg.  Skin cancer detection based on deep learning and entropy to detect outlier samples, 2019.
  * Combalia et al. [In Press] Marc Combalia, Ferran Hueto, Susana Puig, Josep Malvehy, and Verónica Vilaplana.  Uncertainty estimation in deep neural networks for dermoscopic image classification.  In _CVPR 2020, ISIC Skin Image Analysis Workshop_ , 2020 In Press.
  * Pacheco et al. [2020] Andre G. C. Pacheco, Chandramouli S. Sastry, Thomas Trappenberg, Sageev Oore, and Renato A. Krohling.  On out-of-distribution detection algorithms with deep neural skin cancer classifiers.  In _CVPR Workshops_ , June 2020.
  * Shannon [1948] Claude E. Shannon.  A mathematical theory of communication.  _Bell Syst. Tech. J._ , 27(3):379–423, 1948.  URL http://dblp.uni-trier.de/db/journals/bstj/bstj27.html#Shannon48.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ICML’16, page 1050–1059. JMLR.org, 2016.
  * Sastry and Oore [2019] Chandramouli Shama Sastry and Sageev Oore.  Detecting out-of-distribution examples with in-distribution examples and gram matrices.  2019\.
  * Cintas et al. [2020] Celia Cintas, Skyler Speakman, Victor Akinwande, William Ogallo, Komminist Weldemariam, Srihari Sridharan, and Edward McFowland.  Detecting adversarial attacks via subset scanning of autoencoder activations and reconstruction error.  In _IJCAI 2020_ , 2020.
  * Neill [2012] Daniel B. Neill.  Fast subset scan for spatial pattern detection, 2012.
  * McFowland et al. [2013] Edward McFowland, Skyler Speakman, and Daniel B. Neill.  Fast generalized subset scan for anomalous pattern detection.  _J. Mach. Learn. Res._ , 14(1):1533–1561, January 2013.  ISSN 1532-4435.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick.  Mask R-CNN.  _CoRR_ , abs/1703.06870, 2017.  URL http://arxiv.org/abs/1703.06870.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In _CVPR09_ , 2009.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6980.
  * Harris et al. [2020] Charles R Harris, K Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al.  Array programming with numpy.  _Nature_ , 585(7825):357–362, 2020.
  * Abadi et al. [2016] Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  Tensorflow: A system for large-scale machine learning.  In _12th USENIX Symposium on Operating Systems Design and Implementation_ , pages 265–283, 2016.  URL https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf.