# Quantum Simulation of non-Born-Oppenheimer dynamics in molecular systems by
path integrals

Sumita Datta 1,2
1 Alliance School of Applied Mathematics, Alliance University,
Bengaluru 562 106, India
2 Department of Physics, University of Texas at Arlington,
Texas 76019, USA

###### Abstract

A numerical algorithm based on the probabilistic path integral approach for
solving Schrödinger equation has been devised to treat molecular systems
without Born-Oppenheimer approximation in the nonrelativistic limit at zero
temperature as an alternative to conventioanl variational and perturbation
methods. Using high quality variational trial functions and path integral
method based on Generalized Feynman-Kac method, we have been able to calculate
the non-Born-Oppenheimer energy for hydrogen molecule for the
$X^{1}{{\Sigma}^{1}}_{g}$ state and hydrogen molecular ion. Combining these
values and the value for ionization potential for atomic hydrogen, the
dissociation energy and ionization potential for hydrogen molecules have been
determined to be 36 113.672(3) $cm^{-1}$ and 124 446.066(10) $cm^{-1}$
respectively. Our results favorably compare with other theoretical and
experimental results and thus show the promise of being a nonperturbative
alternative for testing fundamental physical theories.

##  1 Introducton

Sovling eigenvalue problems for molecular systems are in general complicated
and people quite often make several simplications to handle the difficulties
associted with it, namely Born-Oppenheimer(BO) approximation or adiabatic
Born-Oppenheimer(ABO)[1,2,3] and rovibrational approach[4]. In the Born-
Oppenheimer(BO) approximation in a chemical system, the coupling between the
nuclear and electronic movements is neglected. The large mass disparity in the
nuclei and electrons justifies the decoupling of their different time scale
motions and provides a very practical way to model a lot of chemical system
adequetely. In quantum mechanics, a solution of Schrödinger equation in the
adiabatic approximation is defined to have a time scale separation in the fast
and slow degrees of motion. Sometimes the BO approximation is also referred to
as ’Adiabatic Born-Oppenheimer’ approximation as the lighter electrons follow
the motion of the heavier nulei adiabatically. The ABO approximation breaks
down when two or more potential energy surfaces approach each other or cross
and one must take resort to the coupled equations. It is justifiable when
energy gap between the ground and excited states is larger than the energy
scale of nucleus. In metals, the applicability of ABO is quaestionble as this
energy gap turns out to be zero. For example, ABO fails in the case of
Graphene[5].

With the typical BO approximation in a diatomic molecule the non-relativistic
ground state energy can be evaluated by solving the relevant Schrödinger
equation neglecting the nuclear kinetic energy. One can choose some fixed
value for the nuclear confugurations and solve for the electronic wavefunction
which depends parametrically on this fixed value of nuclear configuration
which we will describe in the next section. Since the nuclear configuration is
considered as a parameter and not a quantum mechanical variable the nuclear
motion is modelled only classically. To get the full quantum dynamics of the
diatomic molecule one needs to treat molecular systems as a whole, including
electrons and atom nuclei on the same footing. Or in ther words one needs to
consider the motion of all the constituents of the molecule simultaneously
assuming nuclei have finite masses and they move in the configuration space as
freely as the electrons do. In view of the above reasons in this paper we have
taken a relook at the sigma state of hydrogen molecule and hydrogen ion
molecule as a testbed so explore the fully non-Born-Oppenheimer(nBO) or non-
adiabatic efects in diatomic molecular systems in general. As a matter of fact
we have adopted a quantum Monte Carlo method based on Generalized Feynman-
Kac(GFK) method[6-11] to calculate the energies for the sigma state of
hydrogen molecule and molecular hydrogen. Since GFK is a non-perterbative
approach, it is easier to study motion of all the particles in the molecular
system even in a fully quantum mechanical scenario. In the framework of GFK
the non-BO study of hydrogen molecule now boils down to solving a four
particle Schrödinger equation. In the Stochastic scenario, the all four
particles execute twelve dimensional random walk scaled differently due to
mass disparity of electrons and nuclei. In the case of a hydrogen molecular
ion it turns out to be a nine dimensional walk of three particles(one electron
and two nuclei). Adopting this idea we prescribe the nBO model for any
aribitrary many body system with more than one electron and nucleus. Now to
test the power of our theory we calculate the total energy of hydrogen
molecular system(hydrogen molecule and hydrogen molecular ion) using path
integral Monte Carlo technique use those to determine the ionization potential
$E_{p}$ and dissociation energy $E_{d}$ of hydrogen molecule. At this point we
need to review the theoretical and experimental endeavors and their outcomes
so far. Hydrogen molcule and hydrogen molecular ion are well explored topics
in quantum mechanics. Their long history dates back to the first theoretical
work of Heitler and London[12] followed by the work of James and
Coolidge[13,14] and extension by Kolos and Wolniewicz[15-17]. The controversy
of [16,17,18] was apparently resolved by the experimental results of
Hertzberg[19] and Stwalley[20]. Subsequent theoretical[21,22,23] and
experimental endeavors[24,25,26] seemed to reduce the discrepency between the
theory and experiment. Up to 2017 the best accepted theoretical value of the
dissociation energy of hydrogen molecule in the non-BO basis is
36,118.0695(10)$cm^{-1}$[Piszczatowski et al [27],Pachuki et al [28] and
Puchalski[29] and the best experimental value was 36,118.0696(4)$cm^{-1}$[Liu
et al[30] and Altmann et al[31]. One of the most recent experimental works[32]
has reported the values for ionization potentials($E_{p}$) and dissociation
energies($E_{d}$)for hydrogen molecule which are significantly lower than the
accpted values so far. Most of the theoretical approaches[33-39] were based on
variational principle and provided the sophistcated upper bound to the total
energies, dissociation energy and ionization energy etc. In this work using
Non-BO basis functions as the trial functions[40] our path integral approach
yielded the nonrelativistic non-BO energies for both hydrogen molecule and
molecular ion which are lower than the previously accepted values. These
ground state energies for the hydrogen molecular system were used in
calculating the ionization potentials($E_{p}$) and dissociation
energies($E_{d}$) for hydrogen molecule and we get new benchmarks for those.

The orgnization of the paper is as follows: In Section 1, we introduce the
problem and describe the contents of the different sections of the paper. In
Section 2.1, we discuss the general aspects of BO and nBO approaches to
Quantum Mechanical problems. In Section 2.2 we discuss our path integral
approach for calculating the eigen energies of hydrogen molecular system. In
Section 3, we show how it can be generalized to any N particle molecular
systems. In Section 4 we discuss our results and we conclude in Section 5. Fig
1 and 2 are schematic diagrams for the overall motion of the molecule in the
nBO scenario. Fig 3 show how our numerical code works in the BO limit.

##  2 Theory

###  2.1 General theoretical considerations in BO and nBO approach in
connection with hydrogen molecular system

In the BO approximation the nuclei are clamped at a fixed position and only
electrons are moving in the configuration space. In the following Figs 1 and 2
the arrows beside the electrons as well as the nuclei signify that in the non-
BO approach electrons and nuclei all are moving in the configuration space and
they are being treated in the same footing.

Figure 1: A plot for the nBO dynamics of the hydrogen molecular ion Figure 2:
A plot for the nBO dynamics of the hydrogen molecule

The exact time dependent Schrödinger equation for hydrogen molecule can be
represented by $H\Psi(r_{1},r_{2},R_{1},R_{2})=E\Psi(r_{1},r_{2},R_{1},R_{2})$
with

$H=\sum_{{\alpha}=A,B}-\frac{{{\nabla}_{\alpha}^{2}}}{2\frac{M}{m}}-\sum_{{\beta}=1,2}-\frac{{{\nabla}_{\beta}^{2}}}{2}-\frac{1}{r_{1A}}-\frac{1}{r_{1B}}-\frac{1}{r_{2A}}-\frac{1}{r_{2B}}+\frac{1}{r_{12}}+\frac{1}{R}\\\
$

where $r_{\beta},{\beta}=1,2,R_{\alpha},{\alpha}=A,B$ are the positions of the
electrons and the nuclei respectively. As a matter of fact
$r_{1A}=r_{1}-R_{A},r_{1B}=r_{1}-R_{B}$ and so on
$r=r_{1}-r_{2},R=R_{A}-R_{B}$. In defining the above Hamiltonian we have used
atomic units and m and M denote the mass of electrons and nuclei respectively.
Also $\frac{M}{m}=1836.152701$. The above Schrödinger equation describes the
quantum motion of the four constituents of hydrogen molecule(two electrons and
2 nuclei). For the hydrogem molecular ion the Schrödinger equation can read as

$H\Psi(r_{A},r_{B},R)=E\psi(r_{A},r_{B},R)$ (1)
$H=\sum_{\alpha=A,B}-\frac{{{\nabla}_{\alpha}^{2}}}{2\frac{M}{m}}-\frac{{{\nabla}^{2}}}{2}-\frac{1}{r_{A}}-\frac{1}{r_{B}}+\frac{1}{R}\\\
$

In the case of Born-Oppenheimer approximation, corresponding to the general
Hamiltonian for the electron-nuclei system of hydrogen molecule
$H(R,r)=T_{R}+H_{e}(R,r)$ there will be only one potential surface defined by
$V(R)=<{\psi}_{0}|H_{e}(R,r)|{\psi}_{0}>$(as evident from Fig 3), whereas in
the case of a non-BO calculation there will be multiple potential surfaces[41]
corresponding to $V_{ii}(R)=<{\psi}_{i}|H_{e}(R,r)|{\psi}_{i}>$ and
$V_{ij}(R)=<{\psi}_{i}|H_{e}(R,r)|{\psi}_{j}>$.

Figure 3: A plot for the BO energy vs Internuclear distances

###  2.2  General Path Integral Theory for the Energy and other Properties for
many body systems

Let us consider the time-dependent Schroedinger equation for a system of N
particles with Hamiltonian $H=-{{\nabla}^{2}}/2+V(x)$ as follows:

$\displaystyle i\frac{\partial{\psi}(x,t)}{\partial
t}=(-\frac{{\nabla}^{2}}{2}+V){\psi}(x,t)$ (2) $\displaystyle\psi(x,0)=f(x)$

where $x\in R^{d}$. The above initial value problem in imaginary time can be
represented as

$\displaystyle\frac{\partial{\psi}(x,t)}{\partial
t}=(\frac{{\nabla}^{2}}{2}-V){\psi}(x,t)$ (3) $\displaystyle{\psi}(x,0)=f(x)$

The solution of Eq(5) for $f(x)=1$ can be written in the following Feynaman-
Kac representation[42-43] and it provides a rigorous justification unlike
ordinary Feynman path integration[44-47].

$\psi(x,t)=E_{x}exp\\{-\int_{0}^{t}V(X(s))ds\\}$ (4)

for $V\in K_{\nu}$, the Kato class of potential[48] where $x(t)$ is a Brownian
motion trajectory and E is the average value of the exponetial term with
respect to these trajectories. The lowest eigenvalue for a given symmetry can
be obtained by large deviation principle of Donsker and Varadhan[49],

${\lambda}_{1}=-\lim_{t\rightarrow\infty}\frac{1}{t}lnE_{x}[{e^{-\int_{0}^{t}V(X(s))ds})}]$
(5)

The above representation(Eq[6]) suffers from poor convergence rate as the
underlying diffusion process-Brownian motion(Wiener Process[50] is non-
recurrent. So it is necessary to use a representation which employs a
diffusion which unlike Brownian motion, has a stationary distributions. To
speed up the calculation we use generalized Feynman-Kac method in which the
Hamiltonian is rewritten as $H=H_{0}+V_{p}$ where
$H_{0}=-{{\nabla}^{2}}/2+{\lambda}_{T}+{\nabla}{\psi}_{T}/2{\psi}_{T}$. Here
${\psi}_{T}$ is a twice differentiable nonnegative reference function and
$H{\psi}_{T}={\lambda}_{T}{\psi}_{T}$. The expression for the energy can now
be rewritten as

${\lambda}_{1}={\lambda}_{T}-\lim_{t\rightarrow\infty}\frac{1}{t}lnE_{x}[{e^{-\int_{0}^{t}V(Y(s))ds})}]$
(6)

where $Y(t)$ is the diffusion process which solves the stochastic differential
equation and is known as Ornstein-Uhlenbeck process.

$dY(t)=\frac{\nabla{\Psi}_{T}(Y(t))}{{\Phi}_{T}(Y(t))}+dX(t)\\\ $ (7)

$V(Y(s))$ is summed over all the time steps and $e^{-V(Y(s))}$ is summed over
all the trajectories. The presence of both drift and diffusion terms in the
above expression enables the trajectory to be highly localized. As a result,
the important regions of the potential are frequently sampled and Eq(4)
converges rapidly. The expectation value for the other properties can be
evaluated as follows[6,51]:

${\langle Y|A|Y\rangle}=\frac{\lim_{t\to\infty}\int
dY(t)A(Y(t))e^{-\int[{V}_{p}(Y(s)]ds}}{\int
dY(t)e^{-\int[{V}_{p}(Y(s)]ds}}\,.$ (8)

##  3 Calculation BO and nBO energies and properties

The nonrelativistic Hamiltonian for hydrogen molecule can be written as a sum
of five terms:

$=-\frac{{\hbar}^{2}}{2M}\sum_{\alpha}{\nabla}_{\alpha}^{2}-\frac{{\hbar}^{2}}{2m}\sum_{\beta}{\nabla}_{\beta}^{2}-\sum_{{\alpha},{\beta}}\frac{e^{2}}{4\pi{\epsilon}_{0}r_{{\alpha}{\beta}}}+\sum_{A>B}\frac{e^{2}}{4\pi{\epsilon}_{0}R_{AB}}+\sum_{{\beta}>{\gamma}}\frac{e^{2}}{4\pi{\epsilon}_{0}r_{{\beta}{\gamma}}}$
(9)

In atomic units the above expression is just

$H=-\frac{1}{2M}\sum_{\alpha}{\nabla}_{\alpha}^{2}-\frac{1}{2}\sum_{\beta}{\nabla}_{\beta}^{2}-\sum_{{\alpha},{\beta}}\frac{1}{r_{{\alpha}{\beta}}}+\sum_{A>B}\frac{1}{R_{AB}}+\sum_{{\beta}>{\gamma}}\frac{1}{r_{{\beta}{\gamma}}}$
(10)

For Born-Oppenheimer calculations the kinetic energy of nuclei is neglected
due to its small contribution towards the total energy as it gets divided by
’M’, the mass of nuclei. So within BO approximation,the above equation reads
as follows:

$H=-\frac{1}{2}\sum_{\beta}{\nabla}_{\beta}^{2}-\sum_{{\alpha},{\beta}}\frac{1}{r_{{\alpha}{\beta}}}+\sum_{A>B}\frac{1}{R_{AB}}+\sum_{{\beta}>{\gamma}}\frac{1}{r_{{\beta}{\gamma}}}$
(11)

Here ${R_{A}}$ and ${R_{B}}$ are not variables hence $R_{AB}$ is treated as a
parameter.
In general the Hamiltonian for all the moving electrons and the nuclei has the
form

$\displaystyle
H=-\frac{1}{2m}{\nabla}^{2}(r_{1},r_{2})-\frac{1}{2M}{\nabla}^{2}(R_{A},R_{B})$
$\displaystyle+\frac{1}{|r_{1}-r_{2}|}+\frac{1}{|R_{A}-R_{B}|}$
$\displaystyle-\frac{1}{|r_{1}-R_{A}|}-\frac{1}{|r_{1}-R_{B}|}-\frac{1}{|r_{2}-R_{A}|}-\frac{1}{|r_{2}-R_{B}|}$
$\displaystyle=-\frac{1}{2}{\nabla}^{2}+V(r_{1},r_{2},R_{A},R_{B})$ (12)

where $r_{\beta},{\beta}=1,2,R_{\alpha},{\alpha}=A,B$ are the positions of the
electrons and the ’quantum nuclei’(moving nuclei) respectively(Fig 2). $|.|$
is the 3 dimensional Euclidean distance and $V$ is the Coulombic interaction.
For non-Born-Oppenheimer calculations the hydrogen molecules can be treated as
general physical systems with inequivalent masses(electrons and nuclei) and
can be represented with the above Hamiltonian. For any physical system with N
ineqivalent masses the above can be generalized as follows: The Schrödinger
equation for the above system can be written as

$\displaystyle[-\sum_{i}\frac{{\hbar}^{2}}{2{m_{i}}}{\nabla}^{2}(\vec{r}_{i})-\sum_{j}\frac{{\hbar}^{2}}{2{M_{j}}}{\nabla}^{2}(\vec{R}_{j})$
$\displaystyle+\sum_{ik}\frac{1}{\sqrt{(r_{i}-r_{k})^{2}}}+\sum_{jl}\frac{1}{\sqrt{(R_{j}-R_{l})^{2}}}-\sum_{ij}\frac{1}{\sqrt{(R_{j}-r_{i})^{2}}}]{\psi}({\vec{r}}_{i},{\vec{R}}_{j})=\mu{\psi}({\vec{r}}_{i},{\vec{R}}_{j})$

i,j refer to number of elctrons and number of nuclei respectively. Now using
$\vec{r_{i}}=s_{i}\vec{r_{i}}^{\prime}$
$\vec{R_{j}}=s_{j}\vec{R_{j}}^{\prime}$,
${\nabla}^{2}(\vec{r}_{i})={\nabla}^{2}(s_{i}\vec{r_{i}}^{\prime})=\frac{1}{{s_{i}}^{2}}{\nabla}^{2}(\vec{r_{i}}^{\prime})$
${\nabla}^{2}(\vec{R}_{j})={\nabla}^{2}(s_{j}\vec{R_{j}}^{\prime})=\frac{1}{{s_{j}}^{2}}{\nabla}^{2}(\vec{R_{j}}^{\prime})$
and putting $m_{i}{s_{i}}^{2}=M_{j}{s_{j}}^{2}$, multiplying throughout by
$\frac{m_{i}{s_{i}}^{2}}{\hbar^{2}}$ the Schrödinger equation in the
dimensionless form reads as (in $m_{i}=\hbar=1$ units)

$\displaystyle[-\sum_{i}\frac{{\nabla}^{2}(\vec{r_{i}}^{\prime})}{2}-\sum_{j}\frac{{\nabla}^{2}(\vec{R_{j}}^{\prime})}{2}$
$\displaystyle+\sum_{ik}\frac{1}{\sqrt{(\vec{r_{i}}^{\prime}-\vec{r_{k}}^{\prime})^{2}}}+\sum_{jl}\frac{1}{\sqrt{(\vec{R_{j}}^{\prime}-\vec{R_{l}}^{\prime})^{2}}}-\sum_{ij}\frac{1}{\sqrt{(\frac{m_{i}}{M_{j}}\vec{R_{j}}^{\prime}-\vec{r_{i}}^{\prime})^{2}}}]{\psi}({\vec{r}^{\prime}}_{i},{\vec{R}^{\prime}}_{j})=\mu{\psi}({\vec{r}^{\prime}}_{i},{\vec{R}^{\prime}}_{j})$

In general let $m_{i}{s_{i}}^{2}=M_{j}{s_{j}}^{2}$ and set
$s_{i}=\sqrt{\frac{m_{i}}{M_{j}}}s_{j}$ No matter the number of distinct
masses, the scale for the corresponding random walker will always be the
square root of the ratio of their masses. Using this result, the Generalized
Feynman-Kac path integral can be used for non adiabatic treatment of the
molecules. Now for hydrogen molecule we need to simulate the random ealk
associated with the following Hamiltonian

$\displaystyle
H=-\frac{1}{2}{\nabla}^{2}(r_{1},r_{2})-\frac{1}{2}{\nabla}^{2}(R_{A},R_{B})$
$\displaystyle-\frac{1}{\sqrt{(\sqrt{\frac{m}{M}}X_{A}-x_{1})^{2}+(\sqrt{\frac{m}{M}}Y_{A}-y_{1})^{2}+(\sqrt{\frac{m}{M}}Z_{A}-z_{1})^{2}}}$
(15)
$\displaystyle-\frac{1}{\sqrt{(\sqrt{\frac{m}{M}}X_{B}-x_{1})^{2}+(\sqrt{\frac{m}{M}}Y_{B}-y_{1})^{2}+(\sqrt{\frac{m}{M}}Z_{B}-z_{1})^{2}}}$
$\displaystyle-\frac{1}{\sqrt{(\sqrt{\frac{m}{M}}X_{A}-x_{2})^{2}+(\sqrt{\frac{m}{M}}Y_{A}-y_{2})^{2}+(\sqrt{\frac{m}{M}}Z_{A}-z_{2})^{2}}}$
$\displaystyle-\frac{1}{\sqrt{(\sqrt{\frac{m}{M}}X_{B}-x_{2})^{2}+(\sqrt{\frac{m}{M}}Y_{A}-y_{2})^{2}+(\sqrt{\frac{m}{M}}Z_{A}-z_{2})^{2}}}$
$\displaystyle+\frac{1}{\sqrt{(X_{A}-X_{B})^{2}+(Y_{A}-Y_{B})^{2}+(Z_{A}-Z_{B})^{2}}}$
$\displaystyle+\frac{1}{\sqrt{(x_{1}-x_{2})^{2}+(y_{1}-y_{2})^{2}+(z_{1}-z_{2})^{2}}}$

##  4 Results and discusssions

Now by setting $x=\\{x_{k}\\},k=1,2,....,12,{x_{k}}{\bf{{\in}R^{12}}}$ we can
write $V(x)=V(r_{1},r_{2},R_{A},R_{B})$ To calculate energies, we use Eq(8) of
Sec 2.2 whereas the other properties are calculated using Eq(10) with $V$ as
defined in Eq(16). In our program the stepsize is fixed and the direction of
the path is chosen randomly. In Eq(9),the first and second terms represent the
drift and diffusion respectively. For each system a number of paths are
generated with a specific path length. These are then summed to produce an
average value and a statistical error. In order to examine the behavior of our
energies and other properties as a function of path length, we compute several
different path lengths - from 8 to 48 units of time. To implement Eq(8)
numerically we replace 12 dimensional Brownian motion with 12 dimensional
Ornstein-Uhlenbeck process and simulate them by 12 independent, properly
scaled one dimensional drifted random walk. For details plesae see Appendix A
. The Hamiltonian in Eq(17) is represented as twelve dimensional random walk
where six dimensions involve stepsizes $\frac{m}{M}$ of the size of other six
as shown above. We calculate $E(H_{2})$ and $E(H_{2}^{+})$ using the formula
given Eqn(8). Using these values and the value of ionization potential of
hydrogen one can calculate the Ionization Potential and Dissociation Energy
according to the following expressions[52].
Ionization Potential:

$E_{P}=E(H_{2}^{+})-E(H_{2})\\\ $ (16)

Dissociation Energy:

$E_{d}=-\lim_{R\rightarrow\infty}E(H_{2})-E(H_{2})\\\ =-2E(H)-E(H_{2})$

For the BO and NBO calculations for the hydrogen molecule we use the trial
function of the following form:

${{\psi}_{T}}=(1+P_{12})(1+P_{AB})exp({\sum}_{k=0}a_{k}{q_{1A}}^{u}{q_{1B}}^{v}{q_{2A}}^{w}{q_{2B}}^{n}{q_{12}}^{g}{q_{AB}}^{h}-\chi
r_{1A}-\delta r_{2B})$ (17)

Here $P_{12}$ is the operator that interchanges the two electrons, $P_{AB}$ is
the operator that interchages the two nuclei, and $q_{x}={r_{x}}/(1+c{r_{x}})$
is a coordinate transformation that allows terms in the exponent to go
smoothly to the separated atom limit. The exponents u, v, w, n, g,and h are
integers(0,1,……..) and all possible terms adding up to $N=u+v+w+n+g+h$ are
selected. In Table 2, we show the variation of BO energies with internuclear
distances and its agreement with the best nonrealtivistic estimates for this
system. We show this agreement for BO case just to establish that the same
code works when the nuclei are frozen as well. In Table 3, we show
extrapolated values for the BO energies for hydrogen molecule and hydrogen
molecular ion. In Table 4, we run simulations for ${H_{2}}^{+}$ for different
time 8-48. In Table 5, we show the comparison of extrapolated energy from the
data in Table 4 with the best non relativistic variational calculatios. Table
6 contains the simlations for nbo dynamics of $H_{2}$ molecule at different
time (8-48). In Table 7 we show the extrapolated value of the nbo energy of
hydrogen moecule and its agreement with other theoretical values. As can be
seen from Eq(8) the most accurate estimate of the energy is obtained when we
extrapolate our results to infinite time. We do this by performing a least
square fit. We have verified that the 5000 path lengths selected with runtime
selected from 8 to 48 are more than enough to perform an accurate fit. Oher
tests have confirmed that a stepsize of 1/30 has little influence on the value
of extrapolated energy. The final value obtained for the nBO energy for the
hydrogen molecule,-1.164 546(3), is in excellent agreement with the best non
relativistic value for this system[36,58]. It is, better than the value
obtained from the Variational Monte Carlo calculation. This agreement can,
however, be attributed to the quality of the original trial wavefunction.
Other tests have confirmed that the stepsize used has little influence on
these values. Unlike the energy there is no need to extrapolate any of the
properties to infinite time In Table 8, we have compared our Ionization
potential $E_{p}$ and dissociation energy $E_{d}$ with other theoretical and
experimental results.
For the BO and NBO calculations for the hydrogen molecular ion the following
trial function function is used:

${\phi}_{T}=exp(-\sigma r^{2})\\\ $ (18)

where $\sigma$ is a variational parameter. Using the calculated values for
$E(H_{2}^{+})$(The extrapolated nBO energy for hydrogen molecular ion from
Table 5), $E(H_{2})$ (The extrapolated nBO energy for hydrogen molecule Table
7)and Eq(18), we calculate the value for the ionization potential
$E_{p}(H_{2})=0.567018a.u.=124446.0663cm^{-1}$. Using ionization energy for
atomic hydrogen $E_{p}(H)$, and $E_{(}H_{2})$ and Eq(19), we calculate the
dissociation energy $E_{d}(H_{2})=0.164546a.u.=36113.672cm^{-1}$. Now adding
the value of previously calculated relativistic corrections[53] for sigma
state to our nBO energy we determine the $E_{d}(H_{2})$ to be equal to
36,116.672(10).
Acknowledgements: The author would like to thank Alliance University for
providing partial support for carrying the research work and The University of
Texas at Arlington,USA where the idea behind the research work was conceived.

##  5 Conclusions and Outlook

In this paper using a probabilistic approach a solution to time dependent
Schrödinger equation has been constructed in the form of a path integral. By
simulating an approximation to Ornstein-Uhenlenbech process through trial
function(drift term) and a toss of a unbiased coin(diffusion term), we have
determined the ionization potential and dissociation energy of hydrogen
molecule with a high accuracy. It looks very promising to observe that if we
guide our random walk using a non-BO variational trial function and perform
the numerical simulation for the path integarl solution for the molecular
system, we already improve the variational energy for the hydrogen molecule.
For implementing the simulation of our path integral solution we make only
onefold approximation as the probalistic representation to Schrödinger
equation can be written in a closed form. To be precise to calculate energy we
approximate an exact solution (i. e. the GFK representation of it) to the
Schrödinger equation, whereas most of the other numerical procedures
approximate a solution to an approximate Schrödinger equation. Also since the
path integral solution is based on fully quantum mechanical approach it can
improve the variational energy to a graet extent provided our trial function
has the right symmetry of the Hamiltonian. At this point our dissociation
energy a little less than established theoretical and experimental values. We
believe we need to add a non-BO relativistic correction to our present nBO
non-relativistic energy to have a better agreement with the experimental
values. We see a better agreement if we add our BO relativistic
corrections[53] as a rough estimate of non-BO relativistic corrections. Also
we need to improve the quality of the trial funtions particularly in the case
of hydrogen molecular ion. The procedre can be applied to more complex systems
for which energies are known up to a few significat figures from variational
calculations and accuracy can be increased to include more significant
figures. Our benchmarks for the dissociation energy and ionization potential
can be a useful input for the other work for evaluating non-BO relativistic
corrections for the molecular systems. We observe that adopting this Monte
Carlo method and taking advantage of modern computer technology solving
eigenvalue calculations in Quantum mechanics can be simplified to a great deal
and hope it will inspire other people to carry out research along the same
line.

Table 1: Notation Table Notation/Phrase | Meaning
---|---
$r_{\beta}$ | position vector of electrons inside the atom
$R_{\alpha}$ | position vector of nuclei
$R$ | distance between two nuclei
$r$ | distance between two electrons
$X(t)$ | Brownian motion with a non-ergodic probabilistic measure or
| Wiener Measure
$Y(t)$ | A stochastic process with an ergodic or stationary measure
$\psi_{T}$ | The trial function corresponding to mathematical ground state
$\phi_{T}$ | The trial function for the molecular ion
$\hat{T}$ | Kinetic energy operator
$\hat{V}$ | Potential Energy operator
$2<\hat{T}>=n<\hat{V}$ | Virial theorem
$E_{p}(H)$ | ionization potential of atomic hydrogen $=0.5a.u.$
$E(H_{2})$ | Total energy for the hydrogen molecule
$E({H_{2}}^{+})$ | Total energy for the hydrogen molecular ion
$E_{d}(H_{2})$ | dissociation energy for the hydrogen molecule
$E_{p}(H_{2})$ | ionization potential for the hydrogen molecule

Table 2: Born-Oppenheimer Energy of Hydrgen molecule for the ground state for different internuclear distances. The number in the parentheses is the statiscal error. R | E | Refs.
---|---|---
0.4 | -0.122 348(5) | This work
| -0.120 230 | Sims et al[54]
| -0.120 228 2(9) | Alexander et al[55]
0.6 | -0.771 535(1) | This Work
| -0.769 635 | Sims et al[54]
| -0.769 635 1 (4) | Alexander et al[55]
0.8 | -1.021 424(6) | This Work
| -1.020 056 | Sims et al[54]
| -1.020 056 1(4) | Alexander et al[55]
1.0 | -1.125 406 | This work
| -1.124 539 | Sims et al[54]
| -1.124 539 2(4) | Aleaxnder et al[55]
1.2 | -1.165 377(1) | This work
| -1.164 935 | Sims et al[54]
| -1.164 934 8(5) | Aleaxnder at al[55]
1.4 | -1.174 564(2) | This work
| -1.174 475 | Sims at al[54]
| -1.174 475 4(6) | Aleaxnder et al[55]
| -1.174 475 1(5) | Datta et al[56]
| -1.174 447 477 | Kolos et al[17]
| -1.174 475 686 | Kolos[57]
1.6 | -1.168 371(1) | This work
| -1.168 583 | Sims et al[54]
| -1.168 583 4(5) | Alexander et al[55]
2.0 | -1.137 488(2) | This work
| -1.138 132 | Sims et al[54]
| -1.138 132 0(4) | Aleaxnder et al[55]
3.0 | -1.057 351(1) | This work
| -1.057 326 | Sims et al[54]
| -1.057 324 9(3) | Alexander et al[55]
4.0 | -1.019 750(4) | This work
| -1.016 390 | Sims et al[54]
| -1.016 389 2(2) | Alexander et al[55]
Table 3: Total nonrelativistic Born-Oppenheimer Energy of Hydrgen molecule ion and hydrogen molecule for the ground state at equilibrium distances. The number in the parentheses is the statiscal error. molecule | R | E |
---|---|---|---
$H_{2}^{+}$ | 2.0 | -0.609 148(1) | This work
| 2.01 | -0.602 1 | Swarwono et al[59]
$H_{2}$ | 1.4 | -1.174 564(2) | This work
Table 4: Total non-relativistic non Born-Oppenheimer energy in (a.u.) of hydrogen molecular ion for the ground state with Generalized Feynman-Kac(GFK) path integral method at different time with a stepsize of $1/30$ and 5000 paths. Time | $E_{mol}(GFK)$ | $<T>$ | $<V>$ | Virial ratio
---|---|---|---|---
8 | -0.554700(1) | 0.59921 | -0.92368 | 1.541
16 | -0.553210(4) | 0.78576 | -1.4218 | 1.809
24 | -0.458169(2) | 0.3784 | -0.6724 | 1.776
32 | -0.431555(6) | 0.7140 | -1.2891 | 1.805
40 | -0.420000(4) | 0.616 | -1.036 | 1.681
48 | -0.376000(6) | 0.417 | -0.793 | 1.9

Table 5: Total Nonrelativistic non-Born-Oppenheimer enegies $E_{ion}$ of Hydrgen molecule ion in the ground state. The number in the parentheses is the statistical error. Work | Method |  $E_{ion}$ (a.u.)
---|---|---
Yuh et al | free iterative complement method(variational) | -0.597 139
Jeziorski et al | rel variational | -0.597 144
present work | GFK | -0.597 528(2)

Table 6: Total non-relativistic non Born-Oppenheimer energy in (a.u.) of hydrogen molecule for the ground state with Generalized Feynman-Kac(GFK) path integral method at different time with a stepsize of $1/30$ and 10000 paths. Time | $E_{mol}(GFK)$ | $<T>$ | $<V>$ | Virial ratio
---|---|---|---|---
8 | -1.164376(5) | 1.17594 | -2.159756 | 1.836
16 | -1.159098(4) | 1.136411 | -2.23381 | 1.97
24 | -1.163043(1) | 1.266482 | -2.275504 | 1.8
32 | -1.165153(2) | 0.9991487 | -1.90263 | 1.904
40 | -1.164310(5) | 1.235915 | -2.584469 | 2.09
48 | -1.15559(2) | 1.120355 | -2.27630 | 2.03
Table 7: Total Nonrelativistic non-Born-Oppenheimer enegies$E_{mol}$ of Hydrgen molecule in the ground state . The number in the parentheses is the statiscal error. Work | Method |  $E_{mol}$ (a.u.)
---|---|---
Tubman et al[58] | FN DMC-full | -1.164 01(5)
Bubin et al[36] | non-BO-Var | -1.164 0250
Alexander et al[40] | non-BO-Var | -1.164 02491(8)
present work | GFK | -1.164 546(3)

Table 8: ionization energy $E_{p}$,dissociation energy $E_{d}$ of hydrogen molecule in ${cm}^{-1})$ for the ground state. The number in the parentheses is the statiscal error. Work | method | $E_{p}$ | $E_{d}$
---|---|---|---
Herzberg et al[18] | Expt |  | 36 113.6$\pm 0.3$
Herzberg[19] | Expt |  | 36 116.3$<D_{0}<36.118.3$
Wolneiwicz[57] | Theo(BO) | 124 417.491 |
Zhang et al[26] | Expt |  | 36 118.062(10)
Liu et al[30] | Hybrid |  |
| Expt |  |
| -Theo | 124 417.491 13(37) | 36 118.069 62(37)
Altmann et al[31] | Expt |  | 36 118 06945(31)
Piszczatowski |  |  |
et al[27] | Var |  |
| (Theo) |  | 36 118.0695(10)
Stwalley[20] | Expt |  | 36 118.6
Pachuki et al[28] | Var |  |
| (Theo) |  | 36 118.797 746 3(2)
Puchalskii et al[29] | (Theo) |  | 36 118.067 8(6)
Cheng et al[32] | Expt | 124 357.238062(25) | 35 999.582 894 (25)
Wang et al[38] | Var(Theo) |  | 36 118.069 71(33)
Present work | GFK(Theo)(non rel) | 124 446.066 (10) | 36 113.672(3)
Present work | GFK with rel correction |  |
| (energy nBO+rel corr BO) |  | 36 116.072(10)

##  Appendix A Details of Numerical Calculations

The formalism described in section 2 can include any generalized potential
[60] and valid for any arbitrary dimension d (d=3N). To implement Eq(3)
numerically, the 3N dimensional Brownian motion can be replaced by properly
scaled one dimensional random walks as follows [9, 43, 61]:

$\displaystyle W(l)\equiv W(t,n,l)$ $\displaystyle=$
$\displaystyle{w_{1}}^{1}(t,n,l),{w_{2}}^{1}(t,n,l),{w_{3}}^{1}(t,n,l)....$
$\displaystyle.......{w_{1}}^{N}(t,n,l){w_{2}}^{N}(t,n,l){w_{3}}^{N}(t,n,l)$

where

${w_{j}}^{i}(t,n,l)=\sum^{l}_{k=1}\frac{{\epsilon}^{i}_{jk}}{\sqrt{n}}$ (A2)

with ${w_{j}}^{i}(0,n,l)=0$ for $i=1,2,....,N$;$j=1,2,3$ and $l=1,2,.....,nt$.
Here $\epsilon$ denotes the binomially distributed random variables which are
chosen independently and randomly with probability P for all i,j,k such that
$P({\epsilon}^{i}_{jk}=1)$=$P({\epsilon}^{i}_{jk}=-1)$=$\frac{1}{2}$. It is
known by an invariance principle [62] that for every $\nu$ and W(l) defined in
Eq.(A1) and Eq(A2)

$\displaystyle\lim_{n\to\infty}P(\frac{1}{n}\sum^{nt}_{l=1}V(W(l)))\leq\nu$
(A3) $\displaystyle=P(\int\limits^{t}_{0}V(X(s))ds)\leq\nu\,\,.$

Consequently for large n,

$\displaystyle P[\exp(-\int\limits^{t}_{0}V(X(s))ds)\leq\nu]$ (A4)
$\displaystyle\approx P[\exp(-\frac{1}{n}\sum^{nt}_{l=1}V(W(l)))\leq\nu]$

Finally, by generating $N_{rep}$ independent realization
$Z_{1}$,$Z_{2}$,….$Z_{N_{rep}}$ of

$Z_{m}=\exp(-(-\frac{1}{n}\sum^{nt}_{l=1}V(W(l)))$ (A5)

and using the law of large numbers,with regard to Eq(A3), we conclude that

$(Z_{1}+Z_{2}+...Z_{N_{rep}})/N_{rep}=Z(t)$ (A6)

is an approximation to Eq.(6) Here $W^{m}(l),m=1,2,N_{rep}$ denotes the
$m^{th}$ realization of W(l) out of $N_{rep}$ independently run simulations.
In the limit of large t and $N_{rep}$ this approximation approaches an
equality, and forms the basis of a computational scheme for the lowest energy
of a many particle system with a prescribed symmetry.

## References

  * [1] M. Born and R. Oppenheimer, Ann. Phys.,84,457(1927)
  * [2] M. Born, Festschrift Gott. Nachr. Math. Phys.,KI,1(1951)
  * [3] M. Born and K. Huang, The dynamical Theory of Crystal Lattices, Oxford University Press, London(1954)
  * [4] S. A. Alexander and R. L. Coldwell, J Chem Phys,129,345(2007)
  * [5] S. Pisana,M. Lazzeri, C. Casiraghi, K. S. Novoselov, A. K. Geim, A. C. Ferrrari and F. Mauri, Nature Material6, 198–201 (2007). https://doi.org/10.1038/nmat1846
  * [6] M.Cafferel and P. Claverie, J. Chem Phys. 88 , 1088 (1988);88, 1100 (1988).
  * [7] F. Soto-Eguibar and P Claverie,in Stochastic Processes Applied to Physics and other A Rueda(World Scientific, Singapore,1983).
  * [8] A. Korzeniowski, J Comp and App Math 66, 333 (1996)
  * [9] S. Datta, J. L Fry, N. G. Fazleev, S. A. Alexander and R. L. Coldwell, Phys Rev A 61 R030502 (2000); S. Datta, Ph. D dissertation, The University of Texas at Arlington (1996).
  * [10] S. Datta, Int. J. Mod. Phys.B,37,2350024(2023)
  * [11] S. Datta, V. Dunjko, M. Olshanii, Physics, 4, 12, 2022
  * [12] W.Heitler, F. London, Z. Phys.,44455,1927
  * [13] H. M. James and A. S. Coolidge,1,825,1933
  * [14] H. M. James and A. S.Coolidge,3,129,1935
  * [15] W.Kolos and L. Wolniewicz, J. Chem. Phys.,41,3663(1964)
  * [16] W.Kolos and L. Wolniewicz, Phys. Rev. Lett.,20,243(1968)
  * [17] W.Kolos and L. Wolniewicz, J. Chem. Phys.,49,404(1968)
  * [18] G Herzberg and A. Monfils, J Mol. Spectr.5,482(1960)
  * [19] G. Herzberg, J. Mol. Spectr,33,147(1970)
  * [20] W. C. Stwalley, Chem. Phys. Lett.,6,241(1970)
  * [21] W. Kolos, K. Szalewicz and H. J. Monkhorst, J. Chem.Phys., 84,3278(1986)
  * [22] W. Kolos and J. Rychlewski,J. Chem. Phys.,98,3960(1993)
  * [23] L.J. Wolniewicz, J. Chem. Phys., 103,1792(1995)
  * [24] A. Balakrishnan, V. Smith and B. P. Stoicheff, Phys. Rev. Lett.,68,2149(1992)
  * [25] E. E. Eyler and N. Melikechi,Phys.Rev. A,48,R18(1993)
  * [26] Y. P. Zhang, C. H Zhang, J. T. Kim, J Stanojevic and the E. E. Eyler, Phys. Rev. Lett. 92, 203003(2004)
  * [27] K. Piszczatowski, G.Lach, B.Jeziorski, M. Przybytek, J Komasa and K. Pachuki, J. Chem. Theo. and computation,5,3039(2009)
  * [28] K. Pachuki and J. Komasa, J. Chem. Phys.,144, 164306(2016)
  * [29] M. Puchalski and J. Komasa, Phys. Rev. A,95, 052506(2017)
  * [30] J. Liu, E. J. Salumbides, U. Hollenstein, J. C. J. Koelemeij, K. S. E. Eikema, W. Ubachs and F. Merkt, J. Chem Phys.,130174306,(2009)
  * [31] R. K. Altmann, L.S. Dressen, E. J. Salumbides, W. Ubachs, K. S. E. Eikema, Phys. Rev. Lett,120, 043204(2018)204(2018)
  * [32] C. Cheng, J. Hussels, M. Niu, H. L. Bethem, K. S. E. Eikema, E. J. Salumbides, W. Ubachs, M. Beyer, N. J. Holsch, J. A. Agner, F. Merkt, L.-G Tao,S.-M Hu and Ch. Jungen, Phys. Rev. Lett.121, 013001(2018)
  * [33] 33)B. Jeziorski and W Kolos, Chem. Phys. Lett.,3,678(1969)
  * [34] Y. Hijikata,H. Nakashima and H. Nakatsuji, J. Chem Phys., 130,024102(2009)
  * [35] M. Stanke, D.Kedziera, S. Bubin, M. Molski and L. Adamiwicz, J Chem Phys., 128, 114313(2008)
  * [36] S. Bubin, F. Leonarski, M Stanke and L Adamowicz, Chem. Phys. Lett.,477,12(2009)
  * [37] B. Chen and J. B. Anderson, J Chem Phys.,102,2802(1995)
  * [38] L. M. Wang and Z. C. Yan,Phys. Rev. A,97, 060501(2018)
  * [39] R. Pino and V. Mujica, J. Phys. B,314537(1998)
  * [40] S. A. Alexander and R. L. Coldwell, J. Chem. Phys.,129,114306(2008)
  * [41] A. W. Jasper, C. Zhu, S. Nangia and D. G. Truhlar, Faraday Discuss, 127, 1-22(2004)
  * [42] M. D. Donsker and M. Kac, J. Res. Natl. Bur. Stand 44, 581 (1950); see also, M.Kac, in Proceedings of the Second Berkeley Symposium (Berkeley Press, California (1951)).
  * [43] A. Korzeniowski, J.L. Fry, D. E. Orr and N. G. Fazleev, Phys Lett 69, 893,1992
  * [44] R. P. Feynman, Rev. Mod. Phys.20,367(1948)
  * [45] R. P. Feynman and A. R. Hibbs, Quantum Mechanics and Path Integrals(McGraw-Hill,NY(1965))
  * [46] H. S. Schulman, Techniques and Applications of Path Integrations,(Wiley, NY, 1993)
  * [47] P. Exner, Open Quantum Systems and Feynman Integrals(Reidal Pub. Co., Boston, MA, 1985)
  * [48] T Kato, Commun. Pure Appl Math,10,151(1957)
  * [49] M. D. Donsker and S. R. Varadhan, in Proc. of the International Conference on Function space Integration ( Oxford Univ. Press 1975)pp. 15-33.
  * [50] N. Wiener, J. Math and Phys., 2,132,(1923).
  * [51] G. Roepstorff, Path integral approach to quantum Physics(Springer, 1994)
  * [52] S. B. Doma, M. Abu-Shady, F. N. EI-Gammal and A. A. Amer, Molecular Physics114,1787(2016)
  * [53] Sumita Datta, S. A. Aleaxnder and R. L. Coldwell, Int. J. Q. Chem.,112,731(2012)
  * [54] J. S. Sims and S. A. Hagstrom, J. Chem Phys.,124,094101(2006)
  * [55] S.A. Alexander and R. L. Coldwell, J. Chem. Phys.,121,11557(2004)
  * [56] S. Datta, S. A. Alexander and R. L. Coldwell, Int. J. Q. Chem, 111, 4106,(2011)
  * [57] 57) W. Kolos, J. Chem. Phys., 101(1994)
  * [58] N. M. Tubman, I.Kylanpaa, S. H. Hammes-Schiffer, D. M. Ceperley, Phys. Rev A 90, 042507(2014)
  * [59] Y. P. Sarwono, F. U. Rahman and R Zhang, New J. Phys.,22,093059(2020)
  * [60] B. Simon, Functional Integrals and Quantum Mechanics(Academic Press, NY, 1979).
  * [61] R. Griego and R. Hersh, Theory of random evolutions with applications to partial differential equations, Transact. Am. Math. Soc 156, 405(1971)
  * [62] P. Billingsley,Convergence of Probability Measures; Wiley:New York, USA,1968


# Stylist: Style-Driven Feature Ranking for Robust Novelty Detection

Ștefan Smeu 1,2, Elena Burceanu 111footnotemark: 1 , Emanuella Haller
111footnotemark: 1 & Andrei Liviu Nicolicioiu 3,4
1Bitdefender, Bucharest, Romania
2University of Bucharest, Romania
3Mila, Montreal, Canada
4University of Montreal, Canada
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Equal contribution.

###### Abstract

Novelty detection aims at finding samples that differ in some form from the
distribution of seen samples. But not all changes are created equal. Data can
suffer a multitude of distribution shifts, and we might want to detect only
some types of relevant changes. Similar to works in out-of-distribution
generalization, we propose to use the formalization of separating into
semantic or content changes, that are relevant to our task, and style changes,
that are irrelevant. Within this formalization, we define the robust novelty
detection as the task of finding semantic changes while being robust to style
distributional shifts. Leveraging pretrained, large-scale model
representations, we introduce Stylist, a novel method that focuses on dropping
environment-biased features. First, we compute a per-feature score based on
the feature distribution distances between environments. Next, we show that
our selection manages to remove features responsible for spurious correlations
and improve novelty detection performance. For evaluation, we adapt domain
generalization datasets to our task and analyze the methods’ behaviors. We
additionally built a large synthetic dataset where we have control over the
spurious correlations degree. We prove that our selection mechanism improves
novelty detection algorithms across multiple datasets, containing both
stylistic and content shifts.

##  1 Introduction

In the wider body of literature, Novelty Detection (ND) (Li et al., 2023a;
Pimentel et al., 2014; Salehi et al., 2022; Tack et al., 2020; Ruff et al.,
2021; Yang et al., 2021) has conventionally revolved around the identification
of notable and meaningful deviations from established data distributions. The
ND task is often used interchangeably with the broader anomaly detection task,
but there is a noteworthy difference between the two. Anomalies are
fundamentally distinct from typical samples and can manifest as deviations in
various forms. Novelties, or semantic anomalies, represent a subset of
anomalies, specifically targeting semantic deviations, aiming to identify any
test sample that does not conform to the established training categories. For
instance, in practical scenarios such as medical diagnosis (Chauhan & Vig,
2015), financial fraud detection (Bhattacharyya et al., 2011) or network
intrusion detection (Dragoi et al., 2022), the primary objective is to detect
novelties, such as unique aspects of a cell’s biological structure, while
disregarding irrelevant divergent characteristics, such as artifacts stemming
from equipment.

Our main point is that not all changes are created equal. When we move across
a continent using a self-driving car, we might be amazed by the style of
different houses that we haven’t seen before, but the self-driving car should
still behave the same. On the other hand, when encountering a new structure
that it hasn’t seen before, such as a new type of intersection or bridge, the
car should detect that this is a novel situation and cease the control to the
driver.

Thus, we define semantic or content shifts as the changes in data distribution
that involve some factors that are relevant to our task (such as driving), and
style shifts as the changes that involve some factors that are irrelevant to
our task. Many times, the style factors are correlated with content factors,
so when learning the semantics of a problem, we might learn some spurious
correlations involving irrelevant style factors. These spurious correlations
might not always hold, thus we should not rely on them.

In this context, we focus on robust novelty detection, which aims to identify
distribution changes in content, while ignoring style changes.

Figure 1: Multi-environment setup for the Robust Novelty Detection task.

To be able to distinguish between the two, we consider the multi-environment
setup from the distribution shift studies (Koh et al., 2021; Zhou et al.,
2022), where besides the usual content label, we also have access to a style
label. An environment is composed of samples with a particular style category,
but having any content categories. In this scenario, a style category is
basically a set of factors or relations that hold only in one environment
(_e.g_. for the self-driving car example, driving in the forest or near a
beach, or even in some fictional, Disney like scenario can be seen as
different styles). On the other hand, a content category refers to a set of
factors or relations that hold across all environments (_e.g_. the roads,
cars, bikes, humans categories). The style component characterizes the data in
an uncertain, maybe even spurious way, toward the content classification task.
During training, the content may be correlated with other factors from the
training environments, which are irrelevant to this new task and might become
spurious. This is a challenging problem for content classification tasks and
even more challenging in the novelty detection setup, where during training
you only observe a set of known classes.

With this in mind, our work centers on detecting novel content, while removing
environment-biased features. Specifically, we propose a method to rank
features based on their distribution changes across training environments.
This ranking mechanism, followed by dropping environment-biased features, aims
to enhance the performance of novelty detection methods, enabling them to
generalize more effectively in the presence of spurious correlations and to
give us a glimpse of features’ interpretability.

Summarized, our main contributions are the following:

  * •

We propose Stylist, a simple, yet very effective algorithm that scores
pretrained features, based on their distribution changes between training
environments. We prove that this approach ranks features based on how much
they are focused on the environmental details and gives a glimpse of
interpretability to the ”black-box” embeddings.

  * •

We show that, by gradually removing the environment-biased features proposed
by our algorithm, we significantly improve the ND models’ generalization
capabilities, both in the covariate and sub-population shift setups, by up to
8%, with the benefits being more pronounced at a high degree of spuriousness
correlations.

  * •

We introduce COCOShift, a comprehensive, synthetic benchmark, with 4 levels of
spuriousness, which enables a detailed analysis for the Robust Novelty
Detection task. We also adapt the DomainNet and fMoW multi-environment real
datasets to novelty detection and validate our main results in this setting.

Figure 2: Stylist. We improve the ND performance by identifying (Step 1) and
gradually removing (Step 2) environment-biased features. From this point of
view, higher distribution distances between environments proved to be a good
indicator for ranking features.

##  2 Problem formulation

Real-world data suffers from a multitude of changes that we usually refer to
as distributional shifts. As described by Smeu et al. (2022) these kinds of
shifts are involved in different lines of work, with different goals: domain
generalization wants to be robust to style shifts while most anomaly detection
methods want to detect either style or semantic shifts. We denote robust
novelty detection as the task of detecting semantic novelties while being
robust to style distributional shifts. More exactly, detect samples that
differ by some semantic shifts from some seen training samples, while ignoring
samples that are only affected by style shifts.

We work in a multi-environment setup, where each training environment changes
the style of the samples, while all the environments contain a set of seen
content classes. The goal of training environments is to define what is
content and what is style. Consequently, we are not restricted to a certain
style definition but rather expect the training environments to define what
might be correlated with our task, but is not actually relevant. Then, we
define an evaluation environment, containing both seen and novel classes with
an associated new style. The goal of robust novelty detection is to separate
between seen and novel content classes, without being affected by the new
style.

We focus on multi-class novelty detection, where we have training environments
with multiple content classes. However, we treat them as a single group of
normal samples and ignore their content labels. By the definition of novelty
detection task, there is a zero level of corruption among the normal samples,
as opposed to the more common setup of anomaly detection.

In Fig. 1 we present two scenarios to exemplify our setup. In the first
example, the normal samples encompass representations of objects in various
formats (real images or paintings). In this context, style is defined as the
manner of depiction. During testing, our objective is to correctly categorize
the laptop as a novel class. Furthermore, we must discern that the sketch of
the banana, despite the shift in style (from real images and paintings to
sketches), is not a novel class. In the second example, we observe a different
definition of style, namely the background of the images, which should also be
irrelevant for classifying the content.

##  3 Our approach

Some dimensions of a given pretrained representation could be more
representative of the semantic aspects while others might be more
representative of style elements. To minimize the impact of style factors on
our novelty detection task, we aim to reduce our reliance on them. Thus, it
might be that we are better off ignoring the dimensions that mostly contain
style information, which we denote as environment-biased features. We focus on
discovering which features from a given, pretrained representation, are more
environment-biased, thus prone to contain spurious correlations, and should be
better ignored. Finding the robust part of a representation is closely linked
to invariance between environments, thus we want to have a measure of variance
for each dimension in our representation. We first quantify the degree of
change in each feature distribution, and then we drop the ones that vary more,
as depicted in Fig. 2.

Our algorithm works over the representations extracted from a frozen model. We
assume that for each sample of our training set, we start with a vector of $N$
features. We proceed in two steps:

##### Step 1. Feature ranking in training environments

First, we compute a score that says how much a feature changes across
environments. For each feature $i$, we consider $f_{i}(env)$ to be the
distribution of this feature in environment $env$. We employ the Wasserstein
distance to compute the distance between the distributions of each feature
$i$, across pairs of environments $(a,b)$.

$dist_{i}(env_{a},env_{b})=W(f_{i}(env_{a}),f_{i}(env_{b})),\quad\forall
i\in[1..N]$ (1)

The per-feature score is obtained next as the expected value of the
Wasserstein distance across all pairs of environments $(env_{a},env_{b})$,
where $a\neq b$.

$score_{i}=\mathbb{E}_{a,b}[dist_{i}(env_{a},env_{b})],\quad\forall
i\in[1..N]$ (2)

##### Step 2. Features selection for Robust Novelty Detection

Next, since our purpose is to be robust and to be able to ignore environment
changes, we remove features with top scores. The intuition here is that
environment-biased features facilitate spuriousness, providing a training
setup prone to such correlations.

The exact used distance might not be that important, but what matters is the
process of looking at differences between environments and searching for what
consistently changes between them (_e.g_. in terms of distribution). For this,
in our approach, we rely on the following assumptions:

  1. 1.

The pretrained feature extractor is able to represent both seen and novel
content categories, as well as known and new styles.

  2. 2.

The change in style dominates over the change in content when we compare
between different environments.

First, our approach leverages pretrained embeddings with extensive coverage
across various content and style categories. This eliminates the need for
employing domain adaptation techniques on the pretrained representation,
enabling us to keep the model frozen and exclusively utilize its features. To
illustrate this concept, consider the initial scenario presented in Fig. 1,
where we presume that the pretrained model effectively captures pertinent
features for objects like bananas, pizza, and laptops across diverse
depictions such as real images, paintings, and sketches. This assumption is
easily met in practice nowadays when we have access to powerful pretrained
models that have been trained on large and comprehensive datasets. So the
difficulty does not lie in getting good representation, but at the next level,
where, given a set of features, you need to select the ones that are relevant
for identifying novel content while dropping style-related features that can
cause spurious correlations.

As for the second assumption, in our setup, both style and content can vary
across environments. Since random changes would create an impossible problem,
we assume that style-induced changes in the data distribution are greater than
content ones, when we look at two different environments. This implies that
our environments have different associated styles, and the considered
representation can capture these relevant differences, which is covered by our
first assumption. In order to mitigate these challenges, it is imperative that
we operate in a well-constructed multi-environment setup to be able to gain
relevant information about the style shift. In the context of the Novelty
Detection task, where both style and content distributions can shift, the
quality and quantity of these environments play a pivotal role. They serve as
reference points to delineate and clarify what holds significance and what
does not within our task.

##  4 Experimental analysis

Our experimental analysis is conducted using two real datasets and a synthetic
one. For the first two, we employ adaptations of well-established domain
generalization datasets: fMoW (Christie et al., 2018) and DomainNet (Peng et
al., 2019). All are multi-environment datasets and for each, we divide the
environments into two sets denoted as follows: in-distribution (ID)
environments (associated with styles that we observe during training) and out-
of-distribution (OOD) environments (associated with styles that we only
observe during testing). Each dataset contains a set of annotated semantic
categories, and we divide them into two sets: normal classes (content
categories observed during training) and novel classes (content categories
that should be distinguished from the normal ones during testing). For each
sample, we have a style label and a novelty label (normal vs. novel).

fMoW comprises satellite images of various functional buildings. The style is
defined by the location of the image, while the content is defined by the
class of the observed building. To generate a greater shift between ID and OOD
styles, we chose the environments to be given by the continent in which the
images were taken. As such, we considered photos taken in Europe, America,
Asia, and Africa to compose the ID styles, while those taken in Australia as
OOD style. The content separation into normal and novel categories was
randomly generated (see Appendix A.5.1).

DomainNet contains images of common objects in six different domains. The
style is defined by the domain, while the content is defined by the object
class. We separated the environments into ID: clipart, infograph, painting,
and real and OOD: quickdraw and sketch. We randomly split the classes into
normals and novelties (see Appendix A.5.2).

COCOShift is a synthetic dataset generated to allow an in-depth analysis of
our approach. We combined segmented objects from COCO (Lin et al., 2014) with
natural landscape imagery from Places365 (Zhou et al., 2017). The landscape
images define the style of the data, while the object categories depict the
content. We have grouped the landscape images into 9 categories (_e.g_.
forest, mountain), each of an equal number of samples, and further split them
into 5 ID and 4 OOD styles. The object categories were split into normal and
novel classes by following a proper balancing between them. (see Appendix
A.5.3). Spuriousness: For COCOShift, we deliberately introduced and varied the
level of spurious correlations between style and content, similar to
Kirichenko et al. (2023); Sagawa* et al. (2020). The spuriousness level ranges
from 50% (balanced dataset) to 95% (where the normal class is strongly
correlated with some environments, while we observe few samples in the rest of
the environments). This results in the COCOShift benchmark, with 4 versions of
spuriousness present in the training sets: COCOShift_balanced, COCOShift75,
COCOShift90, and COCOShift95.

Metrics: For our ND experiments we report ROC-AUC metric, as the average
performance over test environments. Unless otherwise specified, we report
performance over OOD environments.

Feature selection algorithms: Besides our Stylist method, we consider InfoGain
(Kraskov et al., 2004) and FisherScore (Gu et al., 2011), which we adapted to
discard the environment-biased features. We denote those three methods as
’Env-Aware’ methods. As ’Not Env-Aware’ methods, we evaluate MAD (mean
absolute difference), Dispersion (as the ratio between arithmetic and
geometric mean), Variance, and PCA Loadings. We use all those methods to
compute an individual score per feature (see Appendix A.1 for details).

Novelty detection algorithms: We observe the impact of our method on several
ND solutions: OCSVM (Schölkopf et al., 1999), LOF (Breunig et al., 2000), and
kNN (Angiulli & Pizzuti, 2002) with different variations (normalized or not at
sample level, with 10 or 30 neighbors to measure variations). We also tested
the impact in the state-of-the-art solution for OOD detection, kNN+ (Sun et
al., 2022a), which trains a kNN on top of normalized samples, but on top of
ResNet-18 features, fine-tuned using a supervised contrastive loss like in
Khosla et al. (2020).

Pretrained features: We validate over multiple feature extractors, from
different tasks, architectures, and datasets (supervised, multi-modal,
contrastive, from basic ResNet to Visual Transformers, trained on ImageNet
(Deng et al., 2009) and other larger datasets): ResNet-18, ResNet-34 (He et
al., 2016), CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), BLIP-2 (Li
et al., 2023b). Unless otherwise specified the experiments are conducted using
ResNet-18 pretrained on ImageNet.

Table 1: Stylist feature ranking on ND Methods. Notice how, for almost all ND algorithms and dataset combinations, dropping top environment-biased features, as identified by Stylist, increases the ROC-AUC performance (see the improvement in green). ND Method | fMoW | DomainNet | COCOShift95
---|---|---|---
ROC-AUC $\uparrow$ |  % selected feat. | ROC-AUC $\uparrow$ |  % selected feat. | ROC-AUC $\uparrow$ |  % selected feat.
all feat. |  Stylist feat. |  all feat. |  Stylist feat. |  all feat. |  Stylist feat.
OCSVM | 46.9 | 54.3 (+7.4) | 85 | 50.4 | 51.4 (+1.0) | 95 | 52.6 | 58.4 (+5.8) | 90
LOF | 58.0 | 60.8 (+2.8) | 15 | 51.1 | 52.0 (+0.9) | 90 | 83.4 | 86.5 (+3.1) | 30
kNN | 59.0 | 60.3 (+1.3) | 20 | 50.6 | 50.8 (+0.2) | 40 | 79.8 | 85.1 (+5.3) | 30
kNN norm | 41.9 | 49.9 (+8.0) | 5 | 52.5 | 52.8 (+0.3) | 70 | 86.2 | 86.2 (+0.0) | 100
kNN+ | 58.0 | 60.8 (+2.8) | 15 | 51.1 | 52.0 (+0.9) | 90 | 82.3 | 82.3 (+0.0) | 100

###  4.1 Robust Novelty Detection

##### Stylist for Novelty Detection

We evaluate how our selection affects the robustness of various Novelty
Detection algorithms. Tab. 1 presents the initial performance using all
features and the best result achieved by dropping environment-biased features
as identified by Stylist. Notice how for almost all cases, using only a
percentage of features improves the performance by up to 8%.

Comparison with other feature selection algorithms We compare in Fig. 3
between different methods of feature selection. For all algorithms, we drop
the features ranked as the most irrelevant. We see that while we vary the
spuriousness level in the training dataset, the relative order of the
algorithms changes, showing that some perform better when working on a
balanced dataset (like PCA based ones), while our Stylist works the best in
difficult scenarios with an increased level of spurious correlations. See in
Appendix A.1 the individual performances and notice in Appendix A.2 how those
covariate shift results are consistent also for sub-population OOD shifts.

Figure 3: Feature selection algorithms. From left to right on the horizontal
axis, we remove features according to the ranking of each feature selection
algorithm. As the spuriousness level of the train set increases
($a)\rightarrow b)\rightarrow c)$), the performance of Stylist (in black)
increases, while the performance of other methods decreases. This proves that
our approach is better at identifying environment-biased features responsible
for the spurious correlations. The reported ROC-AUC performance is for the
same OOD sets in all three plots.

Stylist robustness to dataset spuriousness level To better understand the real
cases, we further analyze the impact of spurious correlations in each step of
our approach. We use datasets with various levels of spuriousness between
style and content, in three setups (Fig. 4): a) use the same dataset in both
algorithm steps; b) keep the spuriousity level fixed for Step2 while varying
the spuriousity level for Step1 c) keep the spuriousity level fixed for Step1
while varying the spuriousity level for Step2. The dataset kept constant in b)
and c) is COCOShift_balanced. We observe that having a higher degree of
spuriousness in feature selection (Step 1), leads to better performance for
our Stylist method. Nevertheless, in all cases, even in the most degenerated
ones (with very high correlations to none), we see an increase after removing
the top-ranked environment-biased features.

Figure 4: Dataset spuriousness impact. We vary the train set spuriousness
level between style and content for the two steps of our algorithm. a) same
dataset for both steps; b) fixed dataset (COCOShift_balanced) for ND training
in Step 2; c) fixed dataset (COCOShift_balanced) for Feature ranking in Step
1. Our method always manages to improve the ND performance (w.r.t. all
features baseline), even in degenerated cases like 95% (or no) spurious
correlation, in only one or in both steps (see the positive slopes in all
curves).

Feature Selection vs. Dimensionality Reduction Classical dimensionality
reduction approaches (like PCA) address the idea of reducing space
dimensionality while preserving or maximizing the most important information.
In PCA, we can assume that a projection into the space of the principal
components will produce a robust representation. Although this projection
method is different from feature selection methods, since the first reprojects
the features into a new space, instead of keeping certain existing features,
we compare it with Stylist, in Fig. 5, for the robust novelty detection task.
Consistently, for all datasets, Stylist selection performs better. We also
combine Stylist with PCA, by applying an additional dimensionality reduction
over the best percentage of features from Stylist. We observe an improvement
in the curves, highlighting the potential of combining the two approaches.

Figure 5: Features Selection vs Dimensionality Reduction. When comparing
Stylist (black) with PCA (orange), we see that selection works better in all
cases. Moreover, when combining the best selection percentage of Stylist with
further dimensionality reduction using PCA (green), we observe an improvement
(note that the green curve corresponds to different absolute numbers of
features).

###  4.2 A glimpse of interpretability

Our approach ranks features based on their correlation with the style. To
validate the quality of the ranking, we perform two experiments.

a) We build a synthetic scenario with disentangled features for style and
content. We split each image from COCOShift_balanced train set into two
images. One only contains the object (content), and the other only contains
the background (style). We extract and combine features for the two images.
The first half of the features are unrelated to style, while the second half
is unrelated to content (for evaluation purposes we have considered that the
first 50% of features are content features and the rest are style features).
Further, we apply Stylist over this combined representation, on
COCOShift_balanced dataset. In Fig. 6a) we present the results of our
experiment. For the first 40% top-ranked environment-biased features, Stylist
has a perfect accuracy score, with other env-aware methods (InfoGain and
FisherScore) having also impressive scores of 99.1% and 99.7%, while the non
env-aware methods performing significantly lower. In this scenario with
disentangled features, env-aware methods consistently select as top features,
those associated with the style.

b) In a real case scenario, where we know nothing about the meaning of each
feature, we analyze the capacity of our top-scored environment-biased features
to predict the style of an image. Starting from our COCOShift dataset, we
build a balanced dataset, with no spurious correlations, for the task of
classifying the style category of an image (1 out of 9). In Fig. 6b) we
present the results of our experiment, where we have trained a classifier for
each percent of features. We observe that with only a small fraction of the
features, we achieve almost the maximum score for predicting the environment,
showing that the top-ranked features are indeed predictive for the style. In
contrast, when randomly selecting features, the same performance is achieved
using significantly more features.

FisherScore selection method works very good in a), which covers the perfect
feature disentanglement case. But in the real scenario from b), it fails below
the random baseline. FisherScore looks for features that can effectively
distinguish between different environments and also provide unique (or non-
redundant) information compared to other features. It relies on computing full
feature space distances when finding neighbours, but those distances are
largely affected by the imperfect scenario, where features are intertwined and
the feature extractor can contain an unbalanced ratio of style vs. content
features. In contrast, Stylist analyzes distances between individual feature
distributions, implicitly balancing the impact of content vs. style if let’s
say part of the spectrum looks similar because it represents the content part.
This way, Stylist manages to be more robust in the real-case scenario like in
b).

Figure 6: Features’ focus analysis. a) In a controlled setup, with 50% of
features being content-related and 50% being style-related, we evaluate how
accurate is our Stylist ranking. We observe that the top-ranked 40% of
environment-biased features are correctly identified with a perfect accuracy
score. In fact, all env-aware methods have impressive results, overcoming non
env-aware methods by a large margin. b) In a balanced setup, we have also
evaluated the ability of our top-ranked environment-biased features to
classify the style category of an image. Note that our approach reaches a high
accuracy with only 5% of the top-ranked environment-biased features. This
indicates that the identified features are indeed strongly correlated with the
style.

###  4.3 Ablations

Feature extractors We show in Tab. 2 that our feature selection method is
model-agnostic, improving over 100% feature usage baseline, over a wide
variety of pretrained models, coming from basic supervised classification,
multi-modal and contrastive approaches.

Table 2: Feature extractors. Stylist improves the performance for all types of pretrained features considered, over all three datasets. For simplicity, we use only ResNet-18 in other experiments. Features | fMoW | DomainNet | COCOShift95
---|---|---|---
ROC-AUC $\uparrow$ |  % selected feat. | ROC-AUC $\uparrow$ |  % selected feat. | ROC-AUC $\uparrow$ |  % selected feat.
all feat. |  Stylist feat. |  all feat. |  Stylist feat. |  all feat. |  Stylist feat.
ResNet-18 | 59.0 | 60.3 (+1.3) | 20 | 50.6 | 50.8 (+0.2) | 40 | 79.8 | 85.1 (+5.3) | 30
ResNet-34 | 61.9 | 65.6 (+3.7) | 30 | 51.1 | 51.1 (+0.1) | 40 | 78.9 | 82.6 (+3.7) | 20
CLIP | 54.3 | 55.5 (+1.3) | 25 | 60.8 | 61.5 (+0.8) | 30 | 94.5 | 94.9 (+0.4) | 95
ALIGN | 54.6 | 56.2 (+1.6) | 40 | 60.6 | 60.8 (+0.3) | 75 | 89.6 | 89.7 (+0.1) | 80
BLIP-2 | 58.6 | 59.1 (+0.4) | 15 | 65.1 | 65.8 (+0.7) | 20 | 96.7 | 96.8 (+0.1) | 95

Stylist distance We validate the algorithmic decisions of our proposed Stylist
approach. To compute the per feature scores, we measure the per-feature
distance in distribution, between any two training environments (Eq. 1), and
combine those per-pair distances to obtain a more informative ranking, based
on all training environments (Eq. 2). See in Fig. 7 how the per-pair ranking
combinations do not influence the overall performance, while the used distance
seems to be dataset-specific (symmetric KL is better on fMoW, while
Wasserstein is better on DomainNet and the synthetic COCOShift95). For
simplicity, we have used Wasserstein distance with mean over the features per-
pair scores in all our experiments. See Appendix A.3 for detailed scores.

Figure 7: Stylist ablations. We vary both the distance metrics (based on
Wasserstein or symmetric KL) and the ranking combination approaches for pairs
of environments (mean, median ranking, weighted ranking). Notice how usually
Wasserstein distances perform better (except for a) fMoW). Also, see how the
ranking combination does not have a high influence on the result.

##  5 Related work

OOD generalization: Machine learning methods proved to have remarkable
capabilities, but still being subject to mistakes when dealing with out-of-
distribution data (Geirhos et al., 2020; Beery et al., 2018; Hendrycks et al.,
2021; Lechner et al., 2022).

Invariant learning: To tackle the changing distribution, one possible solution
involves learning some invariant mechanisms of the data (Muandet et al., 2013;
Peters et al., 2016; Arjovsky et al., 2019). IRM (Arjovsky et al., 2019)
constraints the model such to obtain the same classifier in different
environments, while vREx (Krueger et al., 2021) constrains the loss to have
low variance across domains. The work of Ye et al. (2021) proves that features
with small variations between training environments are important for out-of-
distribution generalization. This also gives a formal motivation to our work.
In Wang et al. (2022) a subspace of invariant features is determined through
PCA of class-embeddings. A formalization of invariant learning is proposed in
Wang & Veitch (2022) and suggests that depending on the structure of the data,
different constraints should be used.

Novelty detection: Semantic anomaly detection (Ahmed & Courville, 2020) aims
to detect only changes in some high-level semantic factors (_e.g_. object
classes) as opposed to low-level cues (such as image artifacts). Methods like
the ones in Tack et al. (2020); Sehwag et al. (2021); Winkens et al. (2020);
Sun et al. (2022b) use a self-supervised method for anomaly or out-of-
distribution detection while the methods in Li et al. (2021); Zhou et al.
(2021); Reiss & Hoshen (2023) also adapt pre-trained extractors using
contrastive methods. RedPanda (Cohen et al., 2023) method learns to ignore
some irrelevant factors but achieves this using labels of such factors. Still,
most works in this space only focus on settings containing only one type of
factor, semantic or non-semantic, but not both.

Robust novelty detection: We propose this term for the setting that contains
both content and style factors, where the goal is to detect changes in content
while being robust to style. This setting is introduced in Smeu et al. (2022)
where they show that robustness methods based on multi-environment learning
can help anomaly detection. Our work shows that a simple, but efficient method
of ranking the feature invariance, improves the performance in the context of
robust novelty detection.

##  6 Conclusions

In this work, we first propose Stylist, a feature selection method that
searches for features focused more on the environment, which are irrelevant
for a pursued task, by emphasizing the distribution distances between
environments, at the feature level. Next, we prove that by dropping features
for which our algorithm gives a high probability to be environment-biased, we
improve the generalization performance of novelty detection, in the setup
where both style and content distribution shifts. To validate our approach, we
introduce COCOShift, a synthetic benchmark, on which we tested our solution on
splits with various levels of spuriousness, alongside other two datasets,
DomainNet and fMoW, composed of real sampled data.

## References

  * Ahmed & Courville (2020) Faruk Ahmed and Aaron Courville.  Detecting semantic anomalies.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2020.
  * Angiulli & Pizzuti (2002) Fabrizio Angiulli and Clara Pizzuti.  Fast outlier detection in high dimensional spaces.  In _Principles of Data Mining and Knowledge Discovery, PKDD_ , 2002\.
  * Arjovsky et al. (2019) Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization.  _arXiv preprint arXiv:1907.02893_ , 2019.
  * Beery et al. (2018) Sara Beery, Grant Van Horn, and Pietro Perona.  Recognition in terra incognita.  In _Proceedings of the European conference on computer vision (ECCV)_ , 2018.
  * Bhattacharyya et al. (2011) Siddhartha Bhattacharyya, Sanjeev Jha, Kurian K. Tharakunnel, and J. Christopher Westland.  Data mining for credit card fraud: A comparative study.  _Decis. Support Syst._ , 2011.
  * Breunig et al. (2000) Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander.  LOF: identifying density-based local outliers.  In _SIGMOD International Conference on Management of Data_ , 2000\.
  * Chauhan & Vig (2015) Sucheta Chauhan and Lovekesh Vig.  Anomaly detection in ecg time signals via deep long short-term memory networks.  _2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA)_ , 2015.  URL https://api.semanticscholar.org/CorpusID:5917674.
  * Christie et al. (2018) Gordon A. Christie, Neil Fendley, James Wilson, and Ryan Mukherjee.  Functional map of the world.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , 2018.
  * Cohen et al. (2023) Niv Cohen, Jonathan Kahana, and Yedid Hoshen.  Red PANDA: Disambiguating image anomaly detection by removing nuisance factors.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=z37tDDHHgi.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009)_ , 2009.
  * Dragoi et al. (2022) Marius Dragoi, Elena Burceanu, Emanuela Haller, Andrei Manolache, and Florin Brad.  Anoshift: A distribution shift benchmark for unsupervised anomaly detection.  _Advances in Neural Information Processing Systems_ , 35:32854–32867, 2022.
  * Geirhos et al. (2020) Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann.  Shortcut learning in deep neural networks.  _Nature Machine Intelligence_ , 2(11), 2020.
  * Gu et al. (2011) Quanquan Gu, Zhenhui Li, and Jiawei Han.  Generalized fisher score for feature selection.  In _UAI Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence_ , 2011.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition CVPR_ , 2016.
  * Hendrycks et al. (2021) Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15262–15271, 2021.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _Proceedings of the 38th International Conference on Machine Learning, ICML_ , 2021.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  In _Advances in Neural Information Processing Systems, NeurIPS_ , 2020\.
  * Kirichenko et al. (2023) Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson.  Last layer re-training is sufficient for robustness to spurious correlations.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=Zb6c8A-Fghk.
  * Koh et al. (2021) Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al.  Wilds: A benchmark of in-the-wild distribution shifts.  In _International Conference on Machine Learning_. PMLR, 2021.
  * Kraskov et al. (2004) Alexander Kraskov, Harald Stoegbauer, and Peter Grassberger.  Estimating mutual information.  In _Phys. Rev_ , 2004.
  * Krueger et al. (2021) David Krueger, Ethan Caballero, Jörn-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Rémi Le Priol, and Aaron C. Courville.  Out-of-distribution generalization via risk extrapolation (rex).  In _Proceedings of the 38th International Conference on Machine Learning, ICML_ , 2021.
  * Lechner et al. (2022) Mathias Lechner, Ramin Hasani, Alexander Amini, Tsun-Hsuan Wang, Thomas A Henzinger, and Daniela Rus.  Are all vision models created equal? a study of the open-loop to closed-loop causality gap.  _arXiv preprint arXiv:2210.04303_ , 2022.
  * Li et al. (2023a) Aodong Li, Chen Qiu, Marius Kloft, Padhraic Smyth, Maja Rudolph, and Stephan Mandt.  Zero-shot anomaly detection without foundation models.  _arXiv preprint arXiv:2302.07849_ , 2023a.
  * Li et al. (2021) Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister.  Cutpaste: Self-supervised learning for anomaly detection and localization.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021.
  * Li et al. (2023b) Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi.  BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.  In _International Conference on Machine Learning, ICML_ , 2023b.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  _CoRR_ , abs/1405.0312, 2014.  URL http://arxiv.org/abs/1405.0312.
  * Muandet et al. (2013) Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf.  Domain generalization via invariant feature representation.  In _International Conference on Machine Learning_. PMLR, 2013.
  * Peng et al. (2019) Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * Peters et al. (2016) Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen.  Causal inference by using invariant prediction: identification and confidence intervals.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 2016.
  * Pimentel et al. (2014) Marco AF Pimentel, David A Clifton, Lei Clifton, and Lionel Tarassenko.  A review of novelty detection.  _Signal processing_ , 2014.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning, ICML_ , 2021.
  * Reiss & Hoshen (2023) Tal Reiss and Yedid Hoshen.  Mean-shifted contrastive loss for anomaly detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2023.
  * Ruff et al. (2021) Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller.  A unifying review of deep and shallow anomaly detection.  _Proceedings of the IEEE_ , 2021.
  * Sagawa* et al. (2020) Shiori Sagawa*, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang.  Distributionally robust neural networks.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=ryxGuJrFvS.
  * Salehi et al. (2022) Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou.  A unified survey on anomaly, novelty, open-set, and out of-distribution detection: Solutions and future challenges.  _Transactions on Machine Learning Research_ , 2022.  URL https://openreview.net/forum?id=aRtjVZvbpK.
  * Schölkopf et al. (1999) Bernhard Schölkopf, Robert C. Williamson, Alexander J. Smola, John Shawe-Taylor, and John C. Platt.  Support vector method for novelty detection.  In _Advances in Neural Information Processing Systems, NIPS_ , 1999\.
  * Sehwag et al. (2021) Vikash Sehwag, Mung Chiang, and Prateek Mittal.  Ssd: A unified framework for self-supervised outlier detection.  _arXiv preprint arXiv:2103.12051_ , 2021.
  * Smeu et al. (2022) Stefan Smeu, Elena Burceanu, Andrei Liviu Nicolicioiu, and Emanuela Haller.  Env-aware anomaly detection: Ignore style changes, stay true to content!  _NeurIPSW on Distribution Shifts_ , 2022.
  * Sun et al. (2022a) Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li.  Out-of-distribution detection with deep nearest neighbors.  In _International Conference on Machine Learning, ICML_ , 2022a.
  * Sun et al. (2022b) Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li.  Out-of-distribution detection with deep nearest neighbors.  In _International Conference on Machine Learning_. PMLR, 2022b.
  * Tack et al. (2020) Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin.  Csi: Novelty detection via contrastive learning on distributionally shifted instances.  _Advances in neural information processing systems_ , 2020.
  * Wang et al. (2022) Haoxiang Wang, Haozhe Si, Bo Li, and Han Zhao.  Provable domain generalization via invariant-feature subspace recovery.  In _International Conference on Machine Learning_. PMLR, 2022.
  * Wang & Veitch (2022) Zihao Wang and Victor Veitch.  A unified causal view of domain invariant representation learning.  In _ICML 2022: Workshop on Spurious Correlations, Invariance and Stability_ , 2022.  URL https://openreview.net/forum?id=-l9cpeEYwJJ.
  * Winkens et al. (2020) Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al.  Contrastive training for improved out-of-distribution detection.  _arXiv preprint arXiv:2007.05566_ , 2020.
  * Yang et al. (2021) Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu.  Generalized out-of-distribution detection: A survey.  _arXiv preprint arXiv:2110.11334_ , 2021.
  * Ye et al. (2021) Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang.  Towards a theoretical framework of out-of-distribution generalization.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=kFJoj7zuDVi.
  * Zhou et al. (2017) Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2017.
  * Zhou et al. (2022) Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy.  Domain generalization: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Zhou et al. (2021) Wenxuan Zhou, Fangyu Liu, and Muhao Chen.  Contrastive out-of-distribution detection for pretrained transformers.  _arXiv preprint arXiv:2104.08812_ , 2021.

##  Appendix A Appendix

###  A.1 Feature selection methods - detailed

We show in Tab. 3 individual results for multiple feature selection
algorithms, grouped into env-aware ones and algorithms that are not env-aware.
Please note that we adapt basic algorithms for feature selection to make them
env-aware.

Considered feature selection methods:

  * •

env-aware

    * –

InfoGain (Kraskov et al., 2004): We adapt the method to the env-ware setup. We
compute the mutual information between each feature and the style labels. High
scores indicate a higher dependency between feature and style labels
$\rightarrow$ environment-biased feature.

    * –

FisherScore (Gu et al., 2011): We adapt the method to the env-aware setup. We
rank the features based on their relevance for the classification of style
categories.

  * •

non env-aware

    * –

MAD: For one feature, compute the average of absolute differences between each
sample value and the mean value. High MAD values indicate a high
discriminatory power.

    * –

Dispersion: This is computed as the ratio of arithmetic and geometric means.
High dispersion implies a higher discriminatory power.

    * –

Variance: Generally you can use variance to discard zero variance features as
being completely uninformative. We have ranked the features based on their
variance, considering high-variance features as being more informative.

    * –

PCA Loadings: We compute the contribution of each feature to the set of
principal components identified by PCA.

###  A.2 Different shift robustness

We analyzed in Fig. 8 what is the impact of Stylist when we consider Sub-
Population shifts, in extension to Covariate shifts, presented in the main
paper. The testing dataset in this case is balanced, ID with the first plot
(with no correlation between style and content). Our method manages to improve
its performance when compared with other feature selectors, as the training
and testing become more OOD. The observations are similar in both kinds of
shifts.

Table 3: Feature selection methods. Selection Method | COCOShift_balanced | COCOShift75 | COCOShift95
---|---|---|---
ROC-AUC $\uparrow$ |  % selected feat. | ROC-AUC $\uparrow$ |  % selected feat. | ROC-AUC $\uparrow$ |  % selected feat.
all feat. |  kept feat. |  all feat. |  kept feat. |  all feat. |  kept feat.
Stylist (ours) | 80.9 | 83.5 (+2.6) | 40 | 80.4 | 84.7 (+4.3) | 10 | 79.8 | 85.1 (+5.3) | 30
InfoGain | 80.9 | 81.0 (+0.0) | 95 | 80.4 | 80.7 (+0.3) | 90 | 79.8 | 79.9 (+0.1) | 90
FisherScore | 80.9 | 80.9 (+0.0) | 100 | 80.4 | 80.4 (+0.0) | 100 | 79.8 | 79.8 (+0.0) | 100
MAD | 80.9 | 80.9 (+0.0) | 100 | 80.4 | 80.4 (+0.0) | 100 | 79.8 | 79.8 (+0.0) | 100
Dispersion | 80.9 | 84.1 (+3.2) | 35 | 80.4 | 83.0 (+2.6) | 45 | 79.8 | 82.2 (+2.4) | 50
Variance | 80.9 | 80.9 (+0.0) | 100 | 80.4 | 80.4 (+0.0) | 100 | 79.8 | 79.8 (+0.0) | 100
PCA Loadings | 80.9 | 84.6 (+3.7) | 35 | 80.4 | 84.7 (+4.3) | 30 | 79.8 | 83.7 (+3.9) | 35

Figure 8: Feature selection algorithms. The reported ROC-AUC performance is
for a testing dataset coming from the same (ID) distribution with a), showing
that similar observations remains for sub-population OOD shifts.

###  A.3 Stylist ablation distances

In Tab. 4 we show individual scores when using symmetric KL or Wasserstein to
measure per feature distribution distances between any two training
environments. We combine the score or ranking obtained, over all pairs of
training envs, using mean, median, median ranking or weighted mean ranking.

Table 4: Ablation distance and ranking combining. We notice here that all variants of distances and feature ranking combinations manage to improve. Depending on the dataset, there are different chooses to make as hyper-parameters. We use for Stylist the mean of Wasserstein distances over all training pairs. Distance |  Method | fMoW | DomainNet | COCOShift95
---|---|---|---|---
ROC-AUC $\uparrow$ |  % selected feat. | ROC-AUC $\uparrow$ |  % selected feat. | ROC-AUC $\uparrow$ |  % selected feat.
all feat. |  Stylist feat. |  all feat. |  Stylist feat. |  all feat. |  Stylist feat.
Wasserstein |  |  |  |  |  |  |  |  |
mean | 59.0 | 60.3 (+1.3) | 20 | 50.6 | 50.8 (+0.2) | 40 | 79.8 | 85.1 (+5.3) | 30
median | 59.0 | 60.6 (+1.5) | 15 | 50.6 | 50.8 (+0.2) | 75 | 79.8 | 84.2 (+4.4) | 20
median ranking | 59.0 | 60.9 (+1.9) | 15 | 50.6 | 50.8 (+0.2) | 55 | 79.8 | 85.5 (+5.7) | 30
weighted mean ranking | 59.0 | 60.7 (+1.7) | 15 | 50.6 | 50.8 (+0.2) | 45 | 79.8 | 85.2 (+5.4) | 25
KL symmetric |  |  |  |  |  |  |  |  |  |
mean | 59.0 | 61.3 (+2.2) | 30 | 50.6 | 50.8 (+0.2) | 50 | 79.8 | 83.0 (+3.1) | 15
median | 59.0 | 60.7 (+1.6) | 50 | 50.6 | 50.8 (+0.2) | 45 | 79.8 | 83.7 (+3.8) | 25
median ranking | 59.0 | 61.6 (+2.6) | 30 | 50.6 | 50.7 (+0.1) | 45 | 79.8 | 83.3 (+3.5) | 10
|  weighted mean ranking | 59.0 | 60.9 (+1.9) | 25 | 50.6 | 50.8 (+0.2) | 35 | 79.8 | 83.4 (+3.6) | 30

###  A.4 Future Work

We leave here several unexplored directions, that we consider valuable for
further investigations:

  1. 1.

Analyze the impact of the pretrained feature extractor, looking after
different axes of variation: supervised/unsupervised pretraining, high/low
disentanglement. And going even further, find an unsupervised way to choose
the best feature extractor, given a dataset.

  2. 2.

Explore more complex algorithms for ranking, based on the same principle of
emphasizing the intra and inter environment distances. More related to the
algorithm, explore an unsupervised manner to choose the best percent of
features to keep.

  3. 3.

Take the approach beyond novelty detection, analyzing the performance
improvement of feature ranking and selection w.r.t. other supervised
approaches for OOD robustness.

###  A.5 Benchmarks

####  A.5.1 fMoW

  * •

Content (functional purpose of buildings):

    * –

normal: airport, airport terminal, barn, burial site, car dealership, dam,
debris or rubble, educational institution, electric substation, fountain, gas
station, golf course, hospital, interchange, multi-unit residential, parking
lot or garage, police station, port, railway bridge, recreational facility,
road bridge, runway, shipyard, shopping mall, solar farm, space facility,
surface mine, swimming pool, waste disposal, water treatment facility, zoo

    * –

novel: airport hangar, amusement park, aquaculture, archaeological site,
border checkpoint, construction site, crop field, factory or powerplant, fire
station, flooded road, ground transportation station, helipad, impoverished
settlement, lake or pond, lighthouse, military facility, nuclear powerplant,
office building, oil or gas facility, park, place of worship, prison, race
track, single-unit residential, smokestack, stadium, storage tank, toll booth,
tower, tunnel opening, wind farm

  * •

Style (geographical area):

    * –

ID: Europe, America, Asia, Africa

    * –

OOD: Australia

  * •

Number of samples:

    * –

OOD test set: 3469

    * –

ID train set: 55288

    * –

ID test set: 13817

    * –

ID val set: 6911

####  A.5.2 DomainNet

  * •

Content (object classes):

    * –

normal: aircraft carrier, angel, animal migration, apple, arm, backpack, barn,
basketball, bed, belt, birthday cake, blackberry, blueberry, book, boomerang,
bowtie, brain, bread, bucket, butterfly, cactus, cake, camouflage, cannon,
carrot, cat, cello, chandelier, circle, cloud, coffee cup, computer, cookie,
couch, crab, crayon, crocodile, cruise ship, diamond, diving board, dog,
donut, door, dresser, drill, drums, duck, ear, elbow, envelope, eraser, fan,
fence, flower, flying saucer, fork, frog, garden, guitar, hand, headphones,
helicopter, helmet, hexagon, hockey stick, horse, hospital, hot air balloon,
hot dog, hourglass, house, hurricane, jacket, jail, kangaroo, knife, laptop,
leg, light bulb, lighter, lightning, lipstick, lobster, map, microphone,
microwave, mountain, moustache, mug, mushroom, necklace, nose, owl, paint can,
paintbrush, palm tree, parachute, parrot, peanut, pear, peas, piano, pig,
pillow, pineapple, pizza, police car, pond, postcard, power outlet, radio,
rain, rake, remote control, roller coaster, sailboat, saw, saxophone,
screwdriver, sea turtle, see saw, shark, shorts, skull, sleeping bag, snail,
snowman, soccer ball, spider, spoon, square, stairs, star, stethoscope,
stitches, stop sign, strawberry, streetlight, string bean, submarine,
suitcase, sun, swan, sweater, swing set, syringe, table, teapot, teddy-bear,
telephone, television, tennis racquet, tent, toaster, toe, tooth, traffic
light, train, tree, trombone, truck, trumpet, umbrella, underwear, van, vase,
violin, whale, wheel, wine bottle, wristwatch, zebra

    * –

novel: airplane, alarm clock, ambulance, ant, anvil, asparagus, axe, banana,
bandage, baseball, baseball bat, basket, bat, bathtub, beach, bear, beard,
bee, bench, bicycle, binoculars, bird, bottlecap, bracelet, bridge, broccoli,
broom, bulldozer, bus, bush, calculator, calendar, camel, camera, campfire,
candle, canoe, car, castle, ceiling fan, cell phone, chair, church, clarinet,
clock, compass, cooler, cow, crown, cup, dishwasher, dolphin, dragon,
dumbbell, elephant, eye, eyeglasses, face, feather, finger, fire hydrant,
fireplace, firetruck, fish, flamingo, flashlight, flip flops, floor lamp,
foot, frying pan, garden hose, giraffe, goatee, golf club, grapes, grass,
hamburger, hammer, harp, hat, hedgehog, hockey puck, hot tub, house plant, ice
cream, key, keyboard, knee, ladder, lantern, leaf, lighthouse, line, lion,
lollipop, mailbox, marker, matches, megaphone, mermaid, monkey, moon,
mosquito, motorbike, mouse, mouth, nail, ocean, octagon, octopus, onion, oven,
panda, pants, paper clip, passport, pencil, penguin, pickup truck, picture
frame, pliers, pool, popsicle, potato, purse, rabbit, raccoon, rainbow,
rhinoceros, rifle, river, rollerskates, sandwich, school bus, scissors,
scorpion, sheep, shoe, shovel, sink, skateboard, skyscraper, smiley face,
snake, snorkel, snowflake, sock, speedboat, spreadsheet, squiggle, squirrel,
steak, stereo, stove, sword, t-shirt, The Eiffel Tower, The Great Wall of
China, The Mona Lisa, tiger, toilet, toothbrush, toothpaste, tornado, tractor,
triangle, washing machine, watermelon, waterslide, windmill, wine glass, yoga,
zigzag

  * •

Style (manner of depiction):

    * –

ID: real, painting, clipart, infograph

    * –

OOD: sketch, quickdraw

  * •

Number of samples:

    * –

OOD test set: 242886

    * –

ID train set: 142026

    * –

ID test set: 35313

    * –

ID val set: 17753

####  A.5.3 COCOShift

Each COCOShift environment is composed of 5 closely related categories of
Places365 as follows:

  * •

forest: forest, rainforest, bamboo_forest, forest_path, forest_road,

  * •

mountain: mountain, mountain_snowy, glacier, mountain_path, crevasse

  * •

seaside: beach, coast, ocean, boathouse, beach_house”

  * •

garden: botanical_garden, formal_garden, japanese_garden, vegetable_garden,
greenhouse

  * •

field: field_cultivated, field_wild, wheat_field, corn_field, field_road

  * •

rock: badlands, butte, canyon, cliff, grotto

  * •

lake: lake, lagoon, swamp, marsh, hot_spring

  * •

farm: orchard, vineyard, farm, rice_paddy, pasture

  * •

sport_field: soccer_field, football_field, golf_course, baseball_field,
athletic_field

On the other hand, we worked with superclasses from COCO (Lin et al., 2014)
such that there would be a significant shift between classes.

To make sure that the content is identifiable from each image (as many COCO
segmentations provide little content without it’s context), we tested each
generated image against CLIP (Radford et al., 2021). Specifically, we took a
given merged picture into COCOShift dataset if CLIP could correctly identify
the content between the selected COCO classes (not superclasses) listed below.
The same test is effectuated on images of segmentations over white
backgrounds.

  * •

Content (object category):

    * –

normal: food (composed of classes: hot dog, cake, donut, carrot, sandwich,
broccoli, banana, apple, pizza, orange)

    * –

novel: electronic (composed of classes: remote, laptop, tv, cell phone,
keyboard), kitchen (composed of classes: bottle, cup, wine glass, knife, fork,
bowl, spoon)

  * •

Style (background area surrounding the object):

    * –

ID: forest, mountain, field, rock, farm

    * –

OOD: lake, seaside, garden, sport field

  * •

Number of samples:

    * –

OOD test set: 13013

    * –

ID test set: 1623

    * –

COCOShift_balanced

      * *

ID train set: 7033

      * *

ID val set: 880

    * –

COCOShift75

      * *

ID train set: 4221

      * *

ID val set: 529

    * –

COCOShift90

      * *

ID train set: 3284

      * *

ID val set: 412

    * –

COCOShift95

      * *

ID train set: 3037

      * *

ID val set: 379

  * •

Spurious correlation sets are variants of train sets used for the synthetic
COCOShift dataset, where we eliminate samples to create spuriousity.

    * –

COCOShift_balanced: normal samples are uniformly distributed among the ID
environments ($\approx 1.4k$ samples per environment)

    * –

COCOShift75: environments [farm, mountain] have $\approx 1.4k$ normal samples,
while environments [rock, forest, field] have $\approx 400$ normal samples

    * –

COCOShift90: environments [farm, mountain] have $\approx 1.4k$ normal samples,
while environments [rock, forest, field] have $\approx 150$ normal samples

    * –

COCOShift95: environments [farm, mountain] have $\approx 1.4k$ normal samples,
while environments [rock, forest, field] have $\approx 70$ normal samples