# Recalibration of Predictive Models as
Approximate Probabilistic Updates

Evan T. R. Rosenman
_Data Science Initiative_
_Harvard University_ Santiago Olivella
_Department of Political Science_
_University of North Carolina, Chapel Hill_

###### Abstract

The output of predictive models is routinely recalibrated by reconciling low-
level predictions with known derived quantities defined at higher levels of
aggregation. For example, models predicting turnout probabilities at the
individual level in U.S. elections can be adjusted so that their aggregation
matches the observed vote totals in each state, thus producing better
calibrated predictions. In this research note, we provide theoretical
grounding for one of the most commonly used recalibration strategies, known
colloquially as the “logit shift”. Typically cast as a heuristic optimization
problem (whereby an adjustment is found such that it minimizes the difference
between aggregated predictions and the target totals), we show that the “logit
shift” in fact offers a fast and accurate approximation to a principled, but
often computationally impractical adjustment strategy: computing the posterior
prediction probabilities, conditional on the target totals. After deriving
analytical bounds on the quality of the approximation, we illustrate the
accuracy of the approach using Monte Carlo simulations. The simulations also
confirm analytical results regarding scenarios in which users of the simple
logit shift can expect it to perform best — namely, when the aggregated
targets are comprised of many individual predictions, and when the
distribution of true probabilities is symmetric and tight around 0.5.

##  1 Problem Description

A common problem in predictive modeling is that of calibrating probabilities
to observed totals. For example, an analyst may obtain individual-level scores
$p_{i}\in(0,1),i=1,\dots,N,$ to estimate the probability that each of the $N$
registered voters in a particular voting precinct will support the Democratic
candidate in an upcoming election. After the election occurs, the analyst can
observe the total number of Democratic votes, $D$, cast among the subset
$\mathcal{V}\subset\\{1,\dots,N\\}$ of registered voters who cast a ballot.
But she cannot observe individual-level outcomes due to the secret ballot. In
the absence of perfect prediction, the analyst will find that
$\sum_{i\in\mathcal{V}}p_{i}\neq D$. She must then decide how to compute
recalibrated scores, $\tilde{p}_{i}$, to better reflect the realized electoral
outcome.

This practical problem has direct implications for public opinion research.
For example, Ghitza and Gelman, (2020) recalibrate their MRP estimates of
voter support levels after an election to match county-level totals, while
Schwenzfeier, (2019) proposes using the magnitude of the calibration to
estimate non-response bias in public opinion polls. The problem is also of
great importance in campaign work. Campaigns frequently seek to target voters
who are most likely to have supported their party in the prior presidential
election. Estimates of prior party support may also serve as predictor
variables in models estimating support in successive elections. Recalibrating
the scores to match observed outcomes is thus a crucial step to improve the
scores’ accuracy and bolster future electioneering.

A common heuristic solution to the recalibration problem is the use of the
“uniform swing” (Butler,, 1951) on the logit scale. This approach is simple:
first, one defines the function

$h(\alpha)=\sum_{i\in\mathcal{V}}\frac{1}{1+\frac{1-p_{i}}{p_{i}}\alpha}\,,$

and, having observed a total $D$, one finds the $\alpha$ that satisfies the
equation

$h(\alpha)=D\,.$ (1)

The function $h(\cdot)$ is monotonic in $\alpha$, so Equation 1 can be solved
in logarithmic time using binary search. The updated scores are then computed
as

$\tilde{p}_{i}=\frac{1}{1+\frac{1-p_{i}}{p_{i}}\alpha}\,.$

Solving Eq. 1 is equivalent to finding the set of probabilities
$\tilde{p}_{i}$ which sum to $D$ and minimize the Kullback–Leibler divergence
(Kullback and Leibler,, 1951) with the distribution induced by the original
scores $p_{i}$. Moreover, if the $p_{i}$ are defined based on a logistic
regression, then this update is equivalent to shifting the intercept in the
model by $\log(\alpha)$. For more details on these characterizations, see the
Appendix, Section A.

Examples of this simple recalibration strategy are given by Ghitza and Gelman,
(2013), Hanretty et al., (2016), and Ghitza and Gelman, (2020). This procedure
is also familiarly referred to by campaign workers as the “logit shift”.111The
term “logit swing” is also commonly used.

In this research note, we provide analytical justification for the logit
shift. First, we introduce an alternative procedure for score updating, which
simply computes the updated scores as posterior probabilities, conditional on
the target totals. In this procedure, we assume the original scores $p_{i}$
capture a kind of prior Democratic support probability, while the updated
scores $\tilde{p}_{i}$ reflect the conditional Democratic voting probability
given observed outcomes. Next, we show that this second, more principled
approach is well approximated by the logit shift in large samples. We
demonstrate this result analytically and illustrate it in a small simulation
study. Finally, we discuss potential extensions to cases where a uniform swing
is insufficient to capture observed electoral dynamics.

##  2 Recalibration as a posterior update

To motivate the posterior update approach, we introduce some additional
notation. We define each voter’s choice as a binary variable
$W_{i}\in\\{0,1\\}$, where $W_{i}=1$ signifies a Democratic vote and $W_{i}=0$
signifies a Republican vote (we suppose a two-candidate election for
simplicity). The $W_{i}$ are modeled as independent Bernoulli random
variables, where $W_{i}\sim\text{Bern}(p_{i})$. In other words, the
$p_{i}=P(W_{i}=1)$ can be thought of as the prior, unconditional probability
of casting a Democratic vote. In this model, it is straightforward to approach
score recalibration by defining a new set of updated scores,
$\\{p_{i}^{\star}\\}$, using the following conditional probability (which
automatically sum to $D$ over voters $i$):

$\displaystyle\begin{split}p_{i}^{\star}&=\mathbb{P}\left(W_{i}=1\,\middle|\,\sum_{j\in\mathcal{V}}W_{j}=D\right)\\\
&=\frac{\mathbb{P}\left(W_{i}=1,\sum_{j\in\mathcal{V}}W_{j}=D\right)}{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D\right)}\\\
&=p_{i}\times\xi_{i}\end{split}$ (2)

where $\xi_{i}=\frac{\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-1\right)}{\mathbb{P}\left(\sum_{j}W_{j\in\mathcal{V}}=D\right)}$ is
a ratio of two Poisson-Binomial probabilities — that is, probabilities over
the sum of independent _but not identically distributed_ Bernoulli random
variables (Chen and Liu,, 1997). Explicit computation of the $p_{i}^{\star}$
is quite challenging, as efficient computation of Poisson-Binomial
probabilities is extremely computationally demanding at even moderate sample
sizes, despite substantial recent advances in the literature (Olivella and
Shiraito,, 2017; Junge,, 2020). To compute the $p_{i}^{\star}$, we would need
to compute one unique Poisson-Binomial probability per unit in the population.
Hence, if the number of actual voters $|\mathcal{V}|$ were even modestly
large, it would be computationally infeasible to obtain these exact posterior
probabilities.

##  3 Logit shift approximates the correct posterior

###  3.1 Preliminaries

In this section, we show analytically why the logit shift is a good
approximation to the general posterior update in Eq. 2. To do so, we begin by
defining two terms, the ratio $\phi_{i}=\frac{\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D\right)}{\mathbb{P}\left(\sum_{j\neq i}W_{j}=D-1\right)}$, and the
function $f(x,s)=\frac{1}{1+\frac{1-x}{x}(s)}\,.$

Simple substitution, along with a useful recursive property of the Poisson-
Binomial distribution,222Namely,
$\mathbb{P}\left(\sum_{j}W_{j}=D\right)=p_{i}\times\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-1\right)+(1-p_{i})\times\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D\right)\,.$ makes it clear that

$\displaystyle\begin{split}\sum_{i}f(p_{i},\phi_{i})&=\sum_{i}\frac{1}{1+\frac{1-p_{i}}{p_{i}}\phi_{i}}\\\
&=\sum_{i}\frac{1}{1+\frac{1-p_{i}}{p_{i}}\frac{\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D\right)}{\mathbb{P}\left(\sum_{j\neq i}W_{j}=D-1\right)}}\\\
&=\sum_{i}\frac{p_{i}\times\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-1\right)}{p_{i}\times\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-1\right)+(1-p_{i})\times\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D\right)}\\\
&=\sum_{i}\frac{\mathbb{P}\left(W_{i}=1,\sum_{i}W_{i}=D\right)}{\mathbb{P}\left(\sum_{i}W_{i}=D\right)}\\\
&=\sum_{i}p_{i}^{\star}\\\ &=D\end{split}$ (3)

In words, Eq. 3 shows that $\phi_{i}$ is precisely the “shift” that turns each
$p_{i}$ into the desired, recalibrated posterior probability $p_{i}^{\star}$.
The logit shift, however, uses a constant $\alpha$ to approximate the vector
of recalibrating shifts $\\{\phi_{i}\\}_{i\in\mathcal{V}}$. What remains,
therefore, is to show that the value of $\alpha$ that solves Eq. 1 is a very
good approximation of $\phi_{i}$ for all values of $i$.

To do so, we establish a couple of facts: that the value of $\alpha$ is
bounded by the range of $\\{\phi_{i}\\}_{i\in\mathcal{V}}$, and that each
$\phi_{i}$ in turn has well-defined bounds:

######  Theorem 1.

The value of $\alpha$ which solves Equation 1 satisfies:

$\min_{i}\frac{\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D\right)}{\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-1\right)}\leq\alpha\leq\max_{i}\frac{\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D\right)}{\mathbb{P}\left(\sum_{j\neq i}W_{j}=D-1\right)}\,.$

###### Proof.

The proof can be found in the Appendix, Section B. ∎

######  Theorem 2.

For any choice of $i\in\mathcal{V}$, we have

$\frac{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D+1\right)}{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D\right)}\leq\frac{\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D\right)}{\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-1\right)}\leq\frac{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D\right)}{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D-1\right)}$

###### Proof.

The proof can be found in the Appendix, Section C.

∎

###  3.2 Main Results

The bounds from Theorem 2 apply regardless of the choice of $i$, so we can
combine the two theorems to find that

$\displaystyle\frac{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D+1\right)}{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D\right)}$
$\displaystyle\leq\min_{i}\frac{\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D\right)}{\mathbb{P}\left(\sum_{j\neq i}W_{j}=D-1\right)}\leq\alpha$
(4) $\displaystyle\leq\max_{i}\frac{\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D\right)}{\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-1\right)}\leq\frac{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D\right)}{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D-1\right)}.$

This is useful, because we can now use the outer bounds in Eq. 4 to obtain a
bound on the approximation error when estimating recalibrated scores
$p_{i}^{\star}$ (obtained from the posterior update approach) via
$\tilde{p}_{i}$ (obtained from the logit shift):

######  Theorem 3.

For large sample sizes, we obtain

$\tilde{p}_{i}=p_{i}^{\star}+\mathcal{O}\left(\frac{1}{\sum_{j\in\mathcal{V}}p_{j}(1-p_{j})}\right)\,.$

###### Proof.

The proof can be found in the Appendix, Section D.

∎

Theorem 3 states that the error in using the logit shift approach to
approximate the posterior recalibration update depends on the precision of the
Poisson-Binomial distribution over sums of binary outcomes being aggregated
(votes for the Democratic candidate, in our running example). As the variance
of Poisson-Binomial deviates is maximal when all underlying probabilities are
equal to 0.5, it follows that, in our running example, the approximation will
perform best when voters in $\mathcal{V}$ are equally likely to vote for
either party. As voters become more heterogenous, or as their support becomes
more lopsided (or both, as would be the case in heavily polarized
electorates), the quality of the approximation is expected to suffer.
Fortunately, the binding bounds in Eq. 4 are extremely tight for large enough
samples, so that even in the worst case-scenarios, the approximation can be
expected to perform well. We now briefly illustrate our analytical results
with a small Monte-Carlo simulation.

##  4 Simulations

We conduct a brief simulation study to empirically demonstrate the efficacy of
this approach. We simulate using $1,000$ units, a sample size at which
computations of the true $p_{i}^{\star}$ are possible. We draw the initial
probabilities $p_{i}$ according to the six distributions discussed in Biscarri
et al., (2018). We then consider the cases in which the observed $D$ is either
20% above or 20% below the expectation, $\sum_{i}p_{i}$.

We compute the true probabilities using Biscarri’s algorithm as implemented in
the PoissonBinomial package (Junge,, 2020), and compare it against the
estimates obtained using our heuristic method. We report the RMSE as well as
the proportion of variance in the true $p_{i}^{\star}$ that is _not_ explained
by our method. Results are given in Table 1. Across all settings, our
approximations perform extremely well.

$\boldsymbol{p_{i}}$ Setting | Sampling Distribution | Observed D | RMSE | 1 $-$ R2
---|---|---|---|---
Uniform | Uniform(0, 1) |  $-$20% | 0.00022 | 5.58$\times 10^{-7}$
Uniform | Uniform(0, 1) | 20% | 0.00021 | 5.46$\times 10^{-7}$
Close to Zero | Beta(0.1, 3) |  $-$20% | 0.00047 | 3.41$\times 10^{-5}$
Close to Zero | Beta(0.1, 3) | 20% | 0.00039 | 1.85$\times 10^{-5}$
Close to One | Beta(3, 0.1) |  $-$20% | 0.00036 | 1.22$\times 10^{-6}$
Close to One | Beta(3, 0.1) | 20% | – | –
Extremal | 0.5*Beta(0.1, 3) + 0.5*Beta(3, 0.1) |  $-$20% | 0.00048 | 1.14$\times 10^{-6}$
Extremal | 0.5*Beta(0.1, 3) + 0.5*Beta(3, 0.1) | 20% | 0.00048 | 1.12$\times 10^{-6}$
Central | Beta(3, 3) |  $-$20% | 0.00015 | 6.86$\times 10^{-7}$
Central | Beta(3, 3) | 20% | 0.00015 | 6.86$\times 10^{-7}$
Bimodal | 0.5*Beta(3, 10) + 0.5*Beta(10, 3) |  $-$20% | 0.00023 | 6.92$\times 10^{-7}$
Bimodal | 0.5*Beta(3, 10) + 0.5*Beta(10, 3) | 20% | 0.00023 | 6.86$\times 10^{-7}$
Table 1: Approximation error, as measured by RMSE and $1-R^{2}$, using our
heuristic method vs. the true Poisson-Binomial probabilities, under various
settings. No results are reported in the row in which $1.2\times\sum_{i}p_{i}$
would exceed the sample size of $1,000$.

##  5 Discussion

In this paper, we have considered the problem of updating voter scores to
match observed vote totals from an election. We have shown that the relatively
simple “logit shift” algorithm is a very good approximation to computing the
true conditional probability. This is an especially useful insight for
campaign analysts and researchers alike, because the logit shift is
significantly more efficient computationally than the calculation of the exact
posterior recalibration update.

It is worth being explicit about the limitations of this approach. Under the
posterior update model, we treat the original scores $p_{i}$ as a prior over
Democratic vote probability. In turn, the updated scores $p_{i}^{\star}$
deviate from the initial scores only by assuming the observed vote tallies
deviate from the expectation of $\sum_{i}p_{i}$ due to random error.
Crucially, the updated probabilities retain the same ordering as the prior
probabilities, which implies the original scoring model must discriminate
positive and negative (but unobservable, in the case of voting) individual
cases well. It is also important to note that the realization of $\mathcal{V}$
over which values of $D$ are defined can have an impact on the quality of the
approximation: the approximation will be better when the number of Democratic
votes $D$ tallies the choices of voters in very competitive districts than
when it tallies votes in landslide ones, and choosing a level of aggregation
with too few voters in it could render the error bounds too loose. In most
practical instances, however, the logit shift can be expected to perform very
well.

Hence, this approach represents a useful – albeit crude – method of updating
individual-level scores to incorporate information from a completed election.
More complex insights about the electorate, such as the marked
underperformance of Democrats among Hispanics voters in the 2020 election,
cannot be directly incorporated by computing the posterior probabilities (or
their approximation via the logit shift). Methods based on ecological
inference (e.g. King et al.,, 2004) would be necessary to capture this
structure. Such methods represent a promising potential extension of the
insights provided in this manuscript.

## References

  * Biscarri et al., (2018) Biscarri, W., Zhao, S. D., and Brunner, R. J. (2018).  A simple and fast method for computing the poisson binomial distribution function.  Computational Statistics & Data Analysis, 122:92–100.
  * Boyd et al., (2004) Boyd, S., Boyd, S. P., and Vandenberghe, L. (2004).  Convex optimization.  Cambridge University Press.
  * Butler, (1951) Butler, D. E. (1951).  Appendix to the british general election of 1950. ed. h. g. nichols.
  * Chen and Liu, (1997) Chen, S. X. and Liu, J. S. (1997).  Statistical applications of the poisson-binomial and conditional bernoulli distributions.  Statistica Sinica, pages 875–892.
  * Ghitza and Gelman, (2013) Ghitza, Y. and Gelman, A. (2013).  Deep interactions with mrp: Election turnout and voting patterns among small electoral subgroups.  American Journal of Political Science, 57(3):762–776.
  * Ghitza and Gelman, (2020) Ghitza, Y. and Gelman, A. (2020).  Voter registration databases and mrp: Toward the use of large-scale databases in public opinion research.  Political Analysis, 28(4):507–531.
  * Hanretty et al., (2016) Hanretty, C., Lauderdale, B., and Vivyan, N. (2016).  Combining national and constituency polling for forecasting.  Electoral Studies, 41:239–243.
  * Junge, (2020) Junge, F. (2020).  Package ‘poissonbinomial’.  Computational Statistics & Data Analysis, 59:41–51.
  * King et al., (2004) King, G., Tanner, M. A., and Rosen, O. (2004).  Ecological inference: New methodological strategies.  Cambridge University Press.
  * Kullback and Leibler, (1951) Kullback, S. and Leibler, R. A. (1951).  On information and sufficiency.  The annals of mathematical statistics, 22(1):79–86.
  * Olivella and Shiraito, (2017) Olivella, S. and Shiraito, Y. (2017).  poisbinom: A faster implementation of the poisson–binomial distribution. r package version 1.0. 1.
  * Schwenzfeier, (2019) Schwenzfeier, M. (2019).  Which non-responders drive non-response bias?  In PolMeth XXXVI, Cambridge, MA.
  * Siripraparat and Neammanee, (2021) Siripraparat, T. and Neammanee, K. (2021).  A local limit theorem for poisson binomial random variable.  Sci. Asia. https://doi. org/10.2306/scienceasia1513-1874.2021, 6\.
  * Wang, (1993) Wang, Y. H. (1993).  On the number of successes in independent trials.  Statistica Sinica, pages 295–312.

##  Appendix A Characterizations of the Logit Shift

First, we show that the logit shift minimizes the summed KL divergence between
$\tilde{p}_{i}$ and $p_{i}$, subject to $\sum\tilde{p}_{i}=D$ constraint.

Define the minimum-KL-divergence optimization problem as

minimize $\displaystyle\hskip
14.22636pt\sum_{i\in\mathcal{V}}-\tilde{x}_{i}\log\left(\frac{p_{i}}{\tilde{x}_{i}}\right)-(1-\tilde{x}_{i})\log\left(\frac{1-p_{i}}{1-\tilde{x}_{i}}\right)$
(5) subject to $\displaystyle\hskip
14.22636pt\sum_{i\in\mathcal{V}}\tilde{x}_{i}=D,\hskip
8.53581pt0\leq\tilde{x}_{i}\leq 1\text{ for }i\in\mathcal{V}.$

The Lagrangian for Optimization Problem 5 is

$\displaystyle
L\left(\\{\tilde{x}_{i}\\},\\{\lambda_{i}\\},\\{\nu_{i}\\},\gamma\right)$
$\displaystyle=\sum_{i\in\mathcal{V}}-\tilde{x}_{i}\log\left(\frac{p_{i}}{\tilde{x}_{i}}\right)-(1-\tilde{x}_{i})\log\left(\frac{1-p_{i}}{1-\tilde{x}_{i}}\right)+$
$\displaystyle\sum_{i\in\mathcal{V}}\lambda_{i}(\tilde{x}_{i}-1)-\nu_{i}\tilde{x}_{i}+\gamma\left(\sum_{i\in\mathcal{V}}\tilde{x}_{i}-D\right)\,.$

We make the standard assumptions that $0<p_{i}<1$ for all $i\in\mathcal{V}$
and $0<D<|\mathcal{V}|$. Define the point
$\left(\\{\tilde{x}_{i}\\},\\{\lambda_{i}\\},\\{\nu_{i}\\},\gamma\right)=\left(\\{\tilde{p}_{i}\\},\\{0\\},\\{0\\},\log(\alpha)\right)$.
We consider the Karush-Kuhn-Tucker (KKT) conditions at this point. For the
Lagrangian gradient condition, observe:

$\displaystyle\nabla
L\left(\\{\tilde{p}_{i}\\},\\{0\\},\\{0\\},\log(\alpha)\right)$
$\displaystyle=-\log\left(\frac{p_{i}/(1-p_{i})}{\tilde{p}_{i}/(1-\tilde{p}_{i})}\right)+\log(\alpha)$
$\displaystyle=-\log\left(\frac{p_{i}/(1-p_{i})}{p_{i}/(\alpha(1-p_{i}))}\right)+\log(\alpha)$
$\displaystyle=0\,,$

while the other four KKT conditions are automatically satisfied at this point.
It follows that this point is dual optimal. Lastly, because the objective
function is convex and there exist choices of $x_{i}$ satisfying $0<x_{i}<1$
for $i\in\mathcal{V}$ and $\sum_{i\in\mathcal{V}}x_{i}=D$, strong duality is
attained. Hence, our point is optimal and the $\tilde{p}_{i}$ are a solution
to Optimization Problem 5. For background technical details, see Boyd et al.,
(2004).

Next, we show that this procedure is equivalent to an intercept shift if the
$p_{i}$ are defined based on a logistic regression, i.e.

$p_{i}=\frac{\exp(\beta_{0}+X_{i}\beta)}{1+\exp(\beta_{0}+X_{i}\beta)\,.}$

where $\beta_{0}\in\mathbb{R}$ is the intercept, $X_{i}\in\mathbb{R}^{p}$ is
the covariate vector for unit $i$, and $\beta\in\mathbb{R}^{p}$ is the
coefficient vector. Plugging this expression into the definition of
$\tilde{p}_{i}$, we obtain

$\displaystyle\tilde{p}_{i}$
$\displaystyle=\frac{1}{1+\exp(-\beta_{0}-X_{i}\beta)\alpha}$
$\displaystyle=\frac{1}{1+\exp(-(\beta_{0}-\log(\alpha))-X_{i}\beta)}$
$\displaystyle=\frac{\exp(\beta_{0}-\log(\alpha)+X_{i}\beta)}{1+\exp(\beta_{0}-\log(\alpha)+X_{i}\beta)}\,,$

where on the last line we see that $\tilde{p}_{i}$ is formulated as a logistic
regression with an identical coefficient vector and its intercept shifted by
$-\log(\alpha)$.

##  Appendix B Proof of Theorem 1

For a fixed choice of $\boldsymbol{x}$, observe that
$g(\boldsymbol{x},\boldsymbol{s})$ is monotonically decreasing in every
component of $s$. Denote $\boldsymbol{\alpha}=\\{\alpha\\}_{i\in\mathcal{V}}$,
the vector repeating $\alpha$ a total of $|\mathcal{V}|$ times. Because

$g(\boldsymbol{p},\boldsymbol{\phi})=D\hskip 14.22636pt\text{ and }\hskip
14.22636ptg(\boldsymbol{p},\boldsymbol{\alpha})=D,$

it follows immediately that $\alpha$ must lie between the largest and smallest
value of $\phi_{i}$ across all choices of $i$.

##  Appendix C Proof of Theorem 2

The log-concavity of the Poisson Binomial distribution is a well-established
result (see e.g. Wang,, 1993). Hence, for any choice of $i$, we have

$\mathbb{P}\left(\sum_{j\neq i}W_{j}=D-2\right)\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D\right)\leq\mathbb{P}\left(\sum_{j\neq i}W_{j}=D-1\right)^{2}$ (6)

Multiplying both sides of the equality by $p_{i}$, and adding the same
quantity to both sides, we obtain an updated inequality

$\displaystyle p_{i}\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-2\right)\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D\right)+(1-p_{i})\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-1\right)\mathbb{P}\left(\sum_{j\neq i}W_{j}=D\right)\leq$
$\displaystyle p_{i}\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-1\right)^{2}+(1-p_{i})\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-1\right)\mathbb{P}\left(\sum_{j\neq i}W_{j}=D\right).$

Collecting terms, we get

$\displaystyle\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D\right)\left(p_{i}\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-2\right)+(1-p_{i})\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-1\right)\right)\leq$ (7) $\displaystyle\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-1\right)\left(p_{i}\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-1\right)+(1-p_{i})p\left(\sum_{j\neq i}W_{j}=D\right)\right)$

The terms in parentheses can be collapsed into a single Poisson Binomial
probability, making use of the recursion defined in Footnote 1. Subbing these
expressions into Inequality 7, we obtain

$\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D\right)\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D-1\right)\leq\mathbb{P}\left(\sum_{j\neq
i}W_{j}=D-1\right)\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D\right)\,,$

which yields the upper bound in Theorem 2.

The proof of the lower bound proceeds by incrementing $D$ by 1 in Inequality 6
and following the same set of steps.

##  Appendix D Proof of Theorem 3

Under the bounds defined in Inequality 4 in the main text, we obtain the
following approximation bounds for $\alpha_{i}$:

$\frac{|\alpha-\phi_{i}|}{\phi_{i}}\leq\frac{\frac{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D\right)}{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D-1\right)}-\frac{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D+1\right)}{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D\right)}}{\frac{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D+1\right)}{\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=D\right)}}\,.$

It is a well-established result that, for large $N$, the Poisson-Binomial
behaves approximately as a Normal random variable with the same mean and
variance (see e.g. Siripraparat and Neammanee,, 2021), namely

$\mu=\sum_{j}p_{j}\hskip 14.22636pt\text{ and }\hskip
14.22636pt\sigma^{2}=\sum_{j}p_{j}(1-p_{j}).$

Denote as $\psi(d)$ the density of a Normal distribution
$\mathcal{N}(\mu,\sigma^{2})$ with this mean and variance, evaluated at $d$.
Siripraparat and Neammanee, (2021) show that over all possible choices of $d$,
the largest deviation between $\psi(d)$ and
$\mathbb{P}\left(\sum_{j\in\mathcal{V}}W_{j}=d\right)$ is bounded above by
$C_{1}/\sigma^{2}$ for a constant $C_{1}>0$. Hence

$\displaystyle\frac{|\alpha-\phi_{i}|}{\phi_{i}}$
$\displaystyle\leq\frac{\psi(D)^{2}}{\psi(D-1)\psi(D+1)}-1+\mathcal{O}\left(\frac{1}{\sigma^{2}}\right)$
(8)
$\displaystyle=\exp\left(\frac{1}{\sigma^{2}}\right)-1+\mathcal{O}\left(\frac{1}{\sigma^{2}}\right)=\mathcal{O}\left(\frac{1}{\sigma^{2}}\right)$

Lastly, we observe

$\displaystyle\tilde{p}_{i}$
$\displaystyle=\frac{1}{1+\frac{1-p_{i}}{p_{i}}\alpha}$
$\displaystyle=\frac{1}{1+\frac{1-p_{i}}{p_{i}}\phi_{i}\left(1+\frac{\alpha-\phi_{i}}{\phi_{i}}\right)}$
$\displaystyle=\frac{1}{1+\frac{1-p_{i}}{p_{i}}\phi_{i}}-\frac{(1-p_{i})p_{i}\phi_{i}}{(p_{i}+\phi_{i}-p_{i}\phi_{i})^{2}}\left(\frac{\alpha-\phi_{i}}{\phi_{i}}\right)+\mathcal{O}\left(\left(\frac{\alpha-\phi_{i}}{\phi_{i}}\right)^{2}\right)$
$\displaystyle=p_{i}^{\star}+\mathcal{O}\left(\frac{1}{\sigma^{2}}\right)\,,$

where the last line follows by plugging in the bound on
$(\alpha-\phi_{i})/\phi_{i}$ from Inequality 8 and observing

$\left|\frac{(1-p_{i})p_{i}\phi_{i}}{(p_{i}+\phi_{i}-p_{i}\phi_{i})^{2}}\right|\leq\frac{1}{4}\hskip
14.22636pt\text{for $0<p_{i}<1,0<\phi_{i}$.}$


[1,2]Kimberly T. Mai

[1]Department of Computer Science, University College London, Gower Street,
London, WC1E 6BT, United Kingdom

2]Department of Security and Crime Science, University College London, Gower
Street, London, WC1E 6BT, United Kingdom

3]School of Law, University of Leeds, Woodhouse, Leeds, LS2 9JT, United
Kingdom

# Understanding the limitations of self-supervised learning for tabular
anomaly detection

<EMAIL_ADDRESS>Toby Davies<EMAIL_ADDRESS>Lewis D. Griffin
<EMAIL_ADDRESS>*  [  [

###### Abstract

While self-supervised learning has improved anomaly detection in computer
vision and natural language processing, it is unclear whether tabular data can
benefit from it. This paper explores the limitations of self-supervision for
tabular anomaly detection. We conduct several experiments spanning various
pretext tasks on 26 benchmark datasets to understand why this is the case. Our
results confirm representations derived from self-supervision do not improve
tabular anomaly detection performance compared to using the raw
representations of the data. We show this is due to neural networks
introducing irrelevant features, which reduces the effectiveness of anomaly
detectors. However, we demonstrate that using a subspace of the neural
network’s representation can recover performance.

###### keywords:

anomaly detection, deep learning, self-supervised learning, tabular data

Statements and Declarations: This work was supported by funding from EPSRC
under grant EP/R513143/1.

##  1 Introduction

Anomaly detection is the task of identifying unusual instances. Two issues
hinder performance: how to obtain a “good” representation of the normal data
and a lack of knowledge about the nature of anomalies. The emergence of self-
supervised learning techniques has primarily addressed these issues in complex
domains such as computer vision and natural language processing [1, 2].
However, these techniques have not yielded the same benefits for tabular data
[3].

Self-supervised learning typically uses a pretext task to learn the intrinsic
structure of the training data [4]. Examples of pretext tasks include
colourising greyscale images [5] or predicting the next word in a sentence [6,
7]. Understanding the typical characteristics of a domain allows one to choose
an effective pretext task. For instance, colourisation requires knowledge of
object boundaries and semantics. These aspects are useful for image
classification [8, 9]. However, unlike images or text where spatial or
sequential biases are natural starting points for self-supervision, the
starting points for tabular data are unclear.

A recent study indicated that self-supervised learning does not help tabular
anomaly detection [3]. Reiss et al. compared two self-supervised methods with
$k$-nearest neighbours ($k$-NN) on the original features. Even though the
methods were designed for tabular data, they found that $k$-NN on the original
features worked the best.

We seek to understand why this is the case. We extend the experiments to
include a more comprehensive suite of pretext tasks. We also incorporate
synthetic test cases and analyse the underlying learnt representations. Our
results reinforce that self-supervision does not improve tabular anomaly
detection performance and indicate deep neural networks introduce redundant
features, which reduces the effectiveness of anomaly detectors. Conversely, we
can recover performance using a subspace of the neural network’s
representation. We also show that self-supervised learning can outperform the
original representation in the case of purely localised anomalies and those
with different dependency structures.

In addition to the above investigations, we ran a series of experiments to
benchmark anomaly detection performance in a setting where we do not have
access to anomalies during training. We include our findings as a complement
to the self-supervision results and to provide practical insight into
scenarios where specific detectors work better than others.

Our contributions are as follows:

  1. 1.

We reconfirm the ineffectiveness of self-supervision for tabular anomaly
detection.

  2. 2.

We empirically investigate why self-supervision does not benefit tabular
anomaly detection.

  3. 3.

We introduce a comprehensive one-class anomaly detection benchmark using
several self-supervised methods.

  4. 4.

We provide practical insights and identify instances where particular anomaly
detectors and pretext tasks may be beneficial.

In Section 2, we cover the anomaly detection setup. We proceed to outline our
experimental approach in Section 3. We evaluate our findings in Section 4.
Finally, we summarise our work and conclude in Section 5.

##  2 Background

###  2.1 Anomaly detection

Anomaly detection can be characterised as follows:

Let $\mathcal{X}\in\mathbb{R}^{d}$ represent the data space. We assume the
normal data is drawn from a distribution $\mathcal{P}$ on $\mathcal{X}$.
Anomalies are data points $\mathbf{x}\in\mathcal{X}$ that lie in a low
probability region in $\mathcal{P}$. Therefore, the set of anomalies can be
defined as follows [10]:

$\mathcal{A}=\\{\mathbf{x}\in\mathcal{X}|p(\mathbf{x})\leq\tau\\},\quad\tau\geq
0$ (1)

Where $\tau$ is a threshold. Often, the original input space is not used as
anomaly detection performance can be improved by using a different
representation space. In the context of deep learning, a neural network
parameterised by $\theta:\mathcal{X}\mapsto\mathcal{Y}$ (where
$\mathcal{Y}\in\mathbb{R}^{m}$) is used to transform the input data. The
anomalies are assumed to lie in a low-probability region in the new space.
Namely, $\mathcal{P}$ on $\mathcal{X}$ transforms to $\mathcal{P}^{\prime}$ on
$\mathcal{Y}$ according to
$\mathcal{P}^{\prime}(\theta(\mathbf{x}))=|\mathbf{J}_{\theta}|$, where
$\mathbf{J}$ is the Jacobian of $\theta$. If $\theta$ is an effective mapping,
then $\theta(\mathcal{A})$ will still be a low probability of
$\mathcal{P}^{\prime}$ and $\theta(\mathcal{A})$ will have a simpler boundary
in $\mathcal{Y}$ than $\mathcal{A}$ in $\mathcal{X}$.

There are deep anomaly detectors (which aim to simultaneously transform the
data to a new subspace and classify it) and shallow anomaly detectors (which
do not transform the data but solely rely on an existing representation). This
paper focuses on shallow anomaly detectors to isolate the differences in
representations derived from different self-supervision tasks. Evaluating the
transformative properties of deep anomaly detectors is out of scope. In this
setup, we also assume only normal samples are present in the training set.
This is commonly referred to as a “one-class” setting in anomaly detection
literature. We describe the anomaly detectors used in our analyses below. For
a more detailed overview of anomaly detection techniques, we refer the reader
to Ruff et al. [10].

$k$-NN assumes normal data closely surround other similar samples in the
feature space, while anomalies have relatively fewer nearby neighbours.
Despite being a simple approach, $k$-NN remains competitive in big data
instances [3, 11, 12, 13, 14]. $k$-NN typically uses features extracted from
pre-trained classification neural networks [12, 13, 14] for image-based
anomaly detection. However, equivalent neural networks for tabular data do not
exist.

Local outlier factor (LOF) is a density-based outlier detection method [15].
It compares the local density of a data point against its $k$-nearest
neighbours. If the point’s density is significantly lower, it is deemed
anomalous.

Isolation forest (iForest) is an ensemble-based algorithm [16]. It uses a set
of isolation trees. Each tree aims to isolate the training data into leaves.
The tree construction algorithm randomly selects an attribute and a random
split inside the attribute’s range until each data point lies in a leaf. Each
observation is assigned a score by calculating the length of the root node to
the leaf and averaging across the trees. Points with shorter path lengths are
considered more unusual, as the algorithm assumes anomalies are easier to
isolate.

One-class support vector machine (OCSVM) assumes normal data lies in a high-
density region [17]. Taking the origin as an anchor in the absence of
anomalous data during training, it learns a maximum margin hyperplane that
separates most training data from the origin. The algorithm considers a test
datum’s distance to the learnt hyperplane to classify anomalies. The method
classifies a point as an anomaly if it lies on the side of the hyperplane
closer to the origin.

Residual norms belong to the category of dictionary-based approaches.
Dictionary-based approaches assume the building blocks of a feature space can
reconstruct normal data but cannot construct anomalies. Methods using
dictionaries use either linear or non-linear manifold learning techniques
(e.g., principal components analysis or autoencoders) to determine the
building blocks [18, 19, 20]. We use the linear principal space approach from
Wang et al. [20] for our experiments. This technique achieves state-of-the-art
results for out-of-distribution detection on images.

Given $\mathbf{X}$ as the in-distribution data matrix of training samples, we
find the principal subspace $\mathbf{W}$ from the matrix
$\mathbf{X}^{T}\mathbf{X}$. This subspace spans the eigenvectors of the $D$
largest eigenvalues of $\mathbf{X}^{T}\mathbf{X}$. We assume anomalies have
more variance on the components with smaller explained variance [21].
Therefore, we project $\mathbf{X}$ to the subspace spanned by the smallest
eigenvalues of $D$ (represented by $\mathbf{W}^{\perp}$) to encapsulate the
residual space and take its norm as the anomaly score:

$||\mathbf{x}^{W^{\perp}}||$ (2)

###  2.2 An overview of self-supervised learning

Self-supervision approaches devise tasks based on the intrinsic properties of
the training data. By exploiting these properties, neural networks hopefully
learn about the regularities of the data. Examples include:

Classifying perturbations: Each training datum is subject to a perturbation
randomly selected from a fixed set, such as rotating the input data [22] or
reordering patches in an image [23]. A classification model then learns to
predict which perturbation was applied.

Conditional prediction. A neural network sees pieces of the input data and
learns to complete the remaining parts. Examples include predicting the next
word given a portion of a sentence [6] or filling in masked areas of an image
[24, 25].

Clustering. Under this category, models learn to group semantically similar
instances and place them far away from observations representing other
semantic categories. $k$-means clustering is a classic example that measures
similarity in Euclidean space.

More modern techniques learn a similarity metric using neural mappings. One
popular loss function that enables this is InfoNCE [26, 27]. InfoNCE takes
augmented views of the same data point as positives and learns to group them
while pushing away other data points. Augmentations are usually in the form of
transformations. In the case of images, these can involve adding noise, colour
jittering, or horizontal flips. However, InfoNCE relies on large batch sizes
to enable sufficiently challenging comparisons. Augmentation choices are also
vital, as aggressive transformations could remove relevant semantic features.

VICReg [28] attempts to overcome some of the issues of InfoNCE by enforcing
specific statistical properties. It encourages augmented views to have a high
variance to ensure the neural mapping learns diverse aspects of the data. It
also regularises the covariance matrix of the representations. This
regularisation ensures the neural mapping covers complementary information
across the representation space.

Additional pretext tasks are covered in more detail in Balestriero et al. [4].

###  2.3 Self-supervised learning and anomaly detection for non-tabular data

Anomaly detection for non-tabular data has benefited from self-supervision.
Golan and El-Yaniv [29] show that compared to OCSVMs trained on pixel space,
outputs from a convolutional neural network trained to predict image rotations
were more reliable for anomaly detection. Mai et al. [2] demonstrate similar
findings on text. They show that good anomaly detection performance is
achievable by fine-tuning a transformer with a self-supervised objective and
using the loss as an anomaly score.

Other successful approaches do not use a self-supervised model in an end-to-
end manner for anomaly detection. The works of Sehwag et al. [30] and Tack et
al. [31] both extract features from neural networks trained with an InfoNCE
objective to perform anomaly detection on images. Sehwag et al. classify
anomalies using the Mahalanobis distance on the extracted space, while Tack et
al. use a product of cosine similarities and norms.

###  2.4 Self-supervised learning and anomaly detection for tabular data

Literature covering self-supervision for anomaly detection in tabular data is
more limited. GOAD [32] extends the work of Golan and El-Yaniv [29] to a more
generalised setting. They apply random affine transformations to the data and
train a neural network to predict these transformations. At inference, they
apply all possible transformations to the test data, obtain the prediction of
each transformation from the network and aggregate the predictions to produce
the anomaly score. The network should be able to predict the correct
modification with higher confidence for the normal data versus the anomalies.

ICL [33] adapts the InfoNCE objective. It considers one sample at a time.
Taking a sample $\mathbf{x}_{i}$ of dimensionality $d$, ICL splits
$\mathbf{x}_{i}$ into two parts. The dimensionality of the two parts depends
on a given window size, $k$ $(k<d)$. The first part $\mathbf{a}_{i}$ is a
continuous section of size $k$, while the second $\mathbf{b}_{i}$ is its
complement of size $d-k$. A Siamese neural network containing two heads with
dimensionalities $k$ and $d-k$ aims to push the representations together. The
negatives are other contiguous segments of $\mathbf{x}_{i}$ of size $k$. As
the neural network should be capable of aligning the normal data and not
anomalies, the loss is the anomaly score.

Although both methods claim to be state-of-the-art for tabular anomaly
detection, Reiss et al. [3] did not find this to be the case. They replicated
the pipelines of GOAD and ICL. In addition, they used the trained neural
networks of GOAD and ICL as feature extractors. After extracting the features,
they ran $k$-NN on the new representations. They compared both setups to
$k$-NN on the original data. Although GOAD and ICL are specifically designed
to process tabular data, Reiss et al. found that $k$-NN on the original data
was the best-performing approach. However, they did not run a hyperparameter
search to optimise the choice of $k$ (leaving it as $k=5$). They also used the
original architectures designed for GOAD and ICL, which differ from each
other. This choice could be another confounding factor affecting results.

##  3 Method

###  3.1 Datasets

We use 26 multi-dimensional point datasets from Outlier Detection Datasets
(ODDS) [34]. Each datum comprises one record, which contains multiple
attributes. Table 1 summarises the properties of the datasets.

We follow the data split protocols described in previous tabular anomaly
detection literature [32, 33]. We randomly select 50% of the normal data for
training, with the remainder used for testing. The test split includes all
anomalies. The training split did not use any anomalies as we adopt a one-
class setup. We partition the training set further by leaving 20% for
validation.

Table 1: Summary of ODDS datasets. Dataset | Total size | Number of anomalies (%) | Dimensionality
---|---|---|---
Annthyroid | 7,200 | 534 (7.4%) | 6
Arrhythmia | 452 | 66 (14.6%) | 274
BreastW | 683 | 239 (35.0%) | 9
Cardio | 1,831 | 176 (9.6%) | 9
Glass | 214 | 9 (4.2%) | 9
Heart | 224 | 10 (4.4%) | 44
HTTP | 567,469 | 2,211 (0.4%) | 3
Ionosphere | 351 | 126 (35.8%) | 33
Letter | 1,600 | 100 (6.3%) | 32
Lympho | 148 | 6 (4.1%) | 18
Mammography | 11,183 | 260 (2.3%) | 6
MNIST | 7,603 | 700 (9.2%) | 100
Musk | 3,062 | 97 (3.2%) | 166
Optdigits | 5,216 | 150 (2.9%) | 64
Pendigits | 6,870 | 156 (2.3%) | 16
Pima | 768 | 268 (34.9%) | 8
Satellite | 6,435 | 2,036 (31.6%) | 36
Satimage-2 | 5,803 | 71 (1.2%) | 36
Seismic | 2,584 | 170 (6.5%) | 11
Shuttle | 49,097 | 3,511 (6.6%) | 9
SMTP | 95,156 | 30 (0.03%) | 3
Speech | 3,686 | 61 (1.7%) | 400
Thyroid | 3,772 | 93 (2.4%) | 6
Vertebral | 240 | 30 (12.5%) | 6
Vowels | 1,456 | 50 (3.4%) | 12
WBC | 278 | 21 (5.6%) | 30
Wine | 129 | 10 (7.7%) | 13

###  3.2 Baseline approach

We run $k$-NN, iForest, LOF, OCSVM, and residual norms on the original
training data. Even though Reiss et al. [3] only use $k$-NN in their
experiments, we use multiple detectors to establish whether $k$-NN is the best
detector or if there are other more appropriate detectors depending on the
type of anomalies present. We analyse our findings in Section 4.7. Another
anomaly detection study, ADBench [35], follows a similar protocol. However,
their setup assumes anomalies are present in the training data. Through our
experiments, we establish whether a purely one-class setup affects overall
detector ranking. We use scikit-learn [36] to implement all detectors except
for $k$-NN, which uses the faiss library [37].

We also investigate the detectors’ sensitivity to different configurations by
varying the hyperparameters. For $k$-NN and LOF, we report results for
$k=\\{1,2,5,10,20,50\\}$. For the residual norms, we look at how results
change with a proportion of features, with percentages ranging from 10% to 90%
in 10% increments $[10\%,20\%,...,90\%]$. We record our findings in Section
4.7. For the self-supervised tasks, we report the results based on the best
hyperparameter configuration derived from these ablations. We retain the
default scikit-learn parameters for iForest and OCSVM, which uses a radial
basis function kernel.

The detectors run directly on the data and on a standardised version. We
standardise each dimension independently by removing the mean and scaling to
unit variance. We also experimented with fully whitening the data but found
attribute-wise standardisation rendered similar results.

###  3.3 Self-supervision

####  3.3.1 Pretext tasks

Although tabular data lacks overt intrinsic properties like those in images or
text, we choose self-supervised tasks that we hypothesise can take advantage
of its structure.

Firstly, we adapt ICL [33] and GOAD [32] to use them as pretext tasks. We do
not directly implement ICL and GOAD as they score anomalies in an end-to-end
manner. In contrast, our experiments focus on how representations from
different pretext tasks affect shallow detection performance. Therefore, we
refer to the ICL-inspired task as “EICL” (embedding-ICL) for the remainder of
the paper. As GOAD uses random affine transformations, we can consider this a
combination of predicting rotation and stretches. This configuration conflates
two different tasks and could be trivial to solve. Therefore, we attempt to
align it closer to the RotNet [1, 22] experiments for image-based anomaly
detection by training a model to classify orthonormal rotations. This pretext
task should profit from the rotationally invariant property of tabular data
[38]. Hence we refer to the GOAD-inspired task as “Rotation”.

The additional objectives used in the experiments are as follows:

Predefined shuffling prediction (Shuffle): We pick a permutation of the
dimensions of the data from a fixed set of permutations and shuffle the order
of the attributes based on the selection. The model learns to predict that
permutation.

Predefined mask prediction (Mask classification): Given a mask rate
$r\quad(r<d)$, we initialise predefined classes that indicate which attributes
to mask. We perform masking by randomly selecting another sample
$\mathbf{x_{j}}$ from the training set and replacing the chosen attributes in
$\mathbf{x_{i}}$ with those from $\mathbf{x_{j}}$. We follow the protocol
outlined in other tabular data literature [39], where they found this approach
generated better representations compared to alternative masking strategies
like imputation. The model learns to classify which predefined class was
applied.

Masked columns prediction (Mask columns): The model picks which attributes
were masked given a mask rate $r$. For example, if only the first attribute
was masked, a correct classification should identify the first attribute and
should not pick the other attributes. This is different from the mask
classification task, where the predefined mask class is given a label from a
fixed set of combinations rather than from the particular attribute that has
been masked (for example, if there are only two classes, the labels for mask
classification are 0 or 1).

Denoising autoencoding (Autoencoder): Given a mask rate $r$, we perturb
$\mathbf{x_{i}}$ by randomly selecting another sample $\mathbf{x_{j}}$ and
replacing a subset of $\mathbf{x_{i}}$’s attributes with those of
$\mathbf{x_{j}}$. The perturbed $\mathbf{x_{i}}$ is the input. Given this
input, the model learns to reconstruct the unperturbed $\mathbf{x_{i}}$.

Contrastive learning: We create positive views of $\mathbf{x_{i}}$ by rotating
the data using an orthonormal matrix (Contrastive rotation), permuting the
attributes per the shuffle task (Contrastive shuffle), or masking the
attributes per the mask classification task (Contrastive mask). We treat other
data points in a minibatch as negatives. We only apply one augmentation at a
time to isolate their effects.

####  3.3.2 Network architectures and loss functions

We use the same neural network architectures to control for any potential
effects on performance. Per the findings of Gorishniy et al. [40], we use
ResNets [41] and FT-Transformers which performed well on their tabular data
experiments (classification and regression). FT-Transformer is a transformer
specially adapted for tabular inputs where each transformer layer operates on
the feature level of one datum.

We train both architectures on all objectives except for EICL, where we only
use ResNets. As EICL requires specific partitioning of the features, the FT-
Transformer architecture would need to be modified. This modification is out
of the scope of our experiments. We retain the same architecture (e.g., the
number of blocks) for each pretext task and only vary the dimensionality of
the output layer. The dimensionality corresponds to the number of preset
classes for the rotation, shuffle, and mask classification tasks. The output
dimensionality of the autoencoder task mirrors the input dimensionality. For
the contrastive objectives (including EICL), we set the output as one of
$\\{128,256,512\\}$ depending on validation performance.

As previous literature has claimed specialised loss functions can improve out-
of-distribution detection on other modalities [42, 43], we examine these to
confirm whether they also improve tabular anomaly detection.

For the rotation, shuffle, and mask classification tasks, we use cross-
entropy, adversarial reciprocal points learning (ARPL) [42], and additive
angular margin (AAM) [44]. ARPL is a specialised loss function for out-of-
distribution detection. The probability of a datum belonging to a class is
proportional to its distance to a reciprocal point. The point represents
“otherness” in the learnt feature space. AAM is a loss function typically used
for facial recognition. AAM specifically enforces interclass similarity and
ensures interclass separation using a specified margin. This results in more
spherical features for each class. We include AAM as some literature claims
spherical per-class features make out-of-distribution detection easier [45].
Finally, we incorporate the cross-entropy loss as studies have shown models
trained with this loss function can meet or outperform specialised losses like
ARPL with careful hyperparameter selection [43]. We experiment with mean
squared error and mean absolute error for the autoencoders. We use the binary
cross-entropy loss for masked column prediction, as multiple masked columns
correspond to more than one label for each datum. For the contrastive
objectives, we experiment with both InfoNCE and VICReg.

We summarise all the possible model configurations in Table 2.

Table 2: Summary of the model configurations. Anomaly detectors | Architectures | Self-supervised tasks | Loss functions
---|---|---|---
|  ResNet FT-Transformer | Rotation |  Cross-entropy ARPL AAM
| Shuffle
| Mask classification
| Mask columns | Binary cross-entropy
| Autoencoder |  | MSE
---
MAE
| EICL |  InfoNCE VICReg
| Contrastive - rotation
| Contrastive - shuffle
$k$-nearest neighbours Isolation forest Local outlier factor One-class support vector machine Residual norms | Contrastive - mask

####  3.3.3 Model selection

Due to the number of potential hyperparameter combinations, we perform random
searches to determine the most appropriate models for anomaly detection. We
pick hyperparameters randomly and train on the training split for each self-
supervised task and dataset. As we cannot evaluate using anomalies, we select
models that achieve the lowest loss on the normal validation data. As we want
to analyse the effect of different loss functions and architecture, the
hyperparameter sweep stage results in a maximum of twelve configurations for
each dataset and task. For example, the models trained on the rotation task
would include ResNets and FT-Transformers, each architecture also includes the
cross-entropy, ARPL, and AAM losses. There are also different configurations
for standardised and non-standardised input data.

####  3.3.4 Feature extraction

After training, we obtain the learnt features by passing input data through
the self-supervised models. We extract the features from the penultimate
layer. As we fix the architecture for the different tasks, we obtain
128-dimensional embeddings for ResNets and 192-dimensional embeddings for FT-
Transformer. We train the anomaly detectors using the new training features
and test them using the transformed test features. We do not apply any
augmentations during inference to ensure a fair comparison between the self-
supervised tasks. Figure 1 shows the workflow.

Figure 1: Self-supervised anomaly detection workflow. The data are only
augmented and fed through the projector during training.

###  3.4 Evaluation

We evaluate all anomaly detectors using the area under the receiving operator
curve (AUROC) score. We can consider AUROC as the probability that a randomly
selected anomaly will be ranked as more abnormal than a normal sample. Scores
fall between 0% and 100%. A score of 50% indicates the detector cannot
distinguish between anomalies and normal data points, while a score of 100%
signals perfect anomaly discrimination. We choose AUROC as it does not require
a threshold to control for false positives, for example.

###  3.5 Additional ablations

In addition to evaluations with the ODDS dataset, we run more experiments to
understand detector performance and scenarios where specific self-supervised
objectives may perform better than others.

####  3.5.1 Synthesised anomalies

Although ODDS contains several datasets, the datasets may mix different types
of anomalies. These mixes can make it difficult to diagnose why one
representation performs better than another. Therefore, we evaluate how the
pretext tasks and their learnt representations fare with synthesised
anomalies. We keep the normal data in the train and test splits and only
generate anomalies by perturbing the properties of the normal training data.
We use the four synthetic anomaly categories as defined in ADBench [35, 46].
We use ADBench’s code to create all types.

  * •

Local anomalies deviate from their local cluster. We use Gaussian mixture
models (GMM) to learn the underlying normal distribution. The covariance
matrix undergoes scaling by a factor $\alpha$ to generate the anomalies. We
use $\alpha=2$ in our experiments.

  * •

Cluster anomalies use GMMs to learn the normal distribution. A factor $\beta$
scales the mean feature vector to create the cluster anomalies. We use
$\beta=2$ in our experiments.

  * •

Global anomalies originate from a uniform distribution
$U[\delta\cdot\textrm{min}(\mathbf{X}_{i}^{k}),\delta\cdot\textrm{max}(\mathbf{X}_{i}^{k})]$.
$\delta$ is a scaling factor, and the minimum and maximum values of an
attribute $\mathbf{X}_{i}^{k}$ define the boundaries. We use $\delta=0.01$.

  * •

Dependency anomalies do not follow the regular dependency structure seen in
normal data. We use vine copulas to learn the normal distribution and Gaussian
kernel density estimators to generate anomalies.

####  3.5.2 Corrupted input data

Previous work hypothesises neural networks underperform on tabular
classification and regression because of their rotational invariance and lack
of robustness to uninformative features [38]. We investigate if this occurs
for anomaly detection. Simultaneously, we explore the shallow anomaly
detectors’ sensitivity to corrupted attributes. Understanding these results
can give a practical insight into what self-supervision objectives and anomaly
detectors work best when the data is noisy or incomplete. For our ablations,
we follow Grinsztajn et al. [38] and apply the following corruptions to the
raw data:

  1. 1.

Adding uninformative features: We add extra attributes to $\mathbf{X}$. We
select a subset of attributes to imitate. We then generate features by
sampling from a multivariate Gaussian based on the mean and interquartile
range of the subset’s values. We experiment with different proportions of
additional features and limit the maximum number of extra attributes to be no
greater than the existing number of features in the dataset.

  2. 2.

Missing values: We randomly remove a proportion of the entries and replace the
missing values using the mean of the attribute the value belongs to. We apply
this transformation to both the train and test sets.

  3. 3.

Removing important features: We train a random forest classifier to classify
between normal samples and anomalies. We then drop a proportion of attributes
based on the feature importance values output by the random forest, starting
from the least important. This corruption violates the one-class assumption
within our anomaly detection setup. However, we use this to analyse the
robustness of the detectors and self-supervised models.

  4. 4.

Selecting a subset of features: Similar to (3), we train a random forest
classifier. We choose a proportion of attributes based on the feature
importance values output from the random forest, starting from the most
important.

After corrupting the data, we follow the same process of training the self-
supervised models and feature extraction for the neural network experiments.

##  4 Results

###  4.1 Self-supervision results

Figure 2: Box plot comparing nearest neighbour AUROCs for each of the
embeddings, ordered by median performance. For each self-supervised task, we
filter the results by architecture and loss function to include the embedding
with the best-performing results. Figure 3: Critical difference diagram
comparing the embeddings in a pairwise manner. The horizontal scale denotes
the average rank of each embedding. The dark lines between different detectors
indicate a statistical difference ($p<0.05$) in results when running pairwise
comparison tests. The baseline scores greatly outrank the pretext tasks. In
contrast, the scores among the pretext tasks are more closely aligned.

No self-supervision task outperforms the baseline. Figure 2 summarises the
nearest neighbour performance derived from the embeddings of each self-
supervised approach. No self-supervision task exceeds $k$-NN on the raw
tabular data. When comparing results at a pairwise level, Figure 3 shows that
the baseline scores greatly outrank the self-supervised objectives. Similarly,
performance using the self-supervised embeddings drops in the presence of
corrupted data (Appendix, Figure 18). These results extend the findings in
[38] that neural networks are also more sensitive to corrupted attributes in
the anomaly detection task. When excluding the baseline, the classification-
based tasks (shuffle, mask classification, and rotation) outperform their
contrastive and reconstructive counterparts.

Figure 4: Box plot comparing detector performance on the self-supervised
embeddings.

We observe similar results when we use different shallow detectors to perform
anomaly detection (Figure 4), with one exception. Using residual norms on the
embedding space is a better choice than $k$-NN. However, they still lag behind
$k$-NN scores on the original embeddings. We also observe that OCSVM performs
consistently worse across all tasks.

###  4.2 A case study on HTTP

To understand why self-supervision does not help, we will explore one ODDS
dataset in detail. We proceed to test our reasoning on toy datasets and then
analyse the remaining ODDS datasets.

We use HTTP for our analyses. HTTP is a modified subset of the KDD Cup 1999
competition data [47]. The competition task involved building a detector to
distinguish between intrusive (attack) and typical network connections. The
dataset initially contained 41 attributes from different sources, including
HTTP, SMTP, and FTP. The ODDS version only uses the “service” attribute from
the HTTP information as it is considered one of the most basic features. The
resulting subset is three-dimensional and comprises over 500,000 observations.
Out of these samples, 2,211 (0.4%) are attacks.

It is easy to find attacks when running detectors directly on the raw ODDS
variant of HTTP. In our experiments, all shallow methods achieve AUROCs
between 87.9% and 100% on non-standardised data, with the median score being
99.7%. Further investigations show the attacks are separate from typical
connections. A supervised logistic regression model trained to classify the
two classes achieves 99.6% AUROC, even with only 200 sample anomalies for
training.

Figure 5: Bar chart comparing baseline and self-supervised embedding results
on HTTP.

However, we observe peculiar results when using representations devised from
the pretext tasks for HTTP. $k$-NN performance drops drastically across the
majority of tasks (Figure 5), sometimes yielding scores worse than random.
Conversely, the other detectors maintain their performance. For example, when
extracting features from the rotation task111Using the best-performing
rotation model, which is an FT-Transformer trained with ARPL loss., $k$-NN
obtains 71.8% AUROC, while iForest, OCSVM, and residual norms preserve AUROCs
around 99%. In addition, logistic regression continues to classify anomalies
with 99% AUROC in the supervised setting using the rotation task
representations. As $k$-NN is susceptible to the curse of dimensionality,
these initial results suggest the neural network representation introduces
directions that obscure informative distances between the typical and
intrusive samples. Moreover, as iForest uses a splitting strategy for
detection, its consistent results indicate some direction signalling
anomalousness exists.

###  4.3 Toy data analysis

It can be challenging to draw conclusions based on existing datasets, as they
are large and often contain uninterpretable features. Therefore, we pivot to
toy examples to understand these behaviours. We devise nine two-dimensional
toy datasets of varying difficulty (Appendix, Figure 19). Like the experiments
on the ODDS, we first evaluate performance directly on the two-dimensional
representations. We then train ResNets on a two-class rotation prediction
task, extract features from the penultimate embedding and re-run the detectors
on the new space. We use this setting as rotations can be performed on two-
dimensional data, and ResNets require less compute than the FT-Transformers.
We apply the same architecture as the ODDS experiments, making the extracted
features 128-dimensional.

(a) Original 2D data

(b) t-SNE visualisation of the self-supervised features

Figure 6: Visualisations of the multiple Gaussian toy dataset. Light blue are
the normal data and orange are the anomalies. The features extracted from the
neural network appear to be more narrow (b) and stretched compared to their
original 2D representation (a).

Regardless of whether the network can or cannot identify the rotation applied
to the data, we observe behaviours consistent with ODDS in most toy instances.
Compared to the original two-dimensional results, detection performance drops
for almost all detectors after extracting representations from the ResNets. As
two dimensions are sufficient to capture the characteristics of the datasets,
projecting the data to a 128-dimensional space only results in a stretched and
narrow representation without extra information. The t-SNE plots highlight
this activity. We show an example of the multiple Gaussian dataset in Figure
6.

Figure 7: Nearest neighbour performance on the toy datasets. The raw embedding
(blue) is the best in almost all instances. However, the self-supervision
embeddings (orange) improve when projecting to a lower dimensional space
(green).

We project the embeddings extracted from the ResNets to a lower dimensional
space using the residual eigenvectors from the training data to verify whether
the curse of dimensionality affects performance. We conduct this projection
because the residual norm method outperforms $k$-NN in the self-supervised
experiments. Therefore, we hypothesise that projecting to a smaller space
should reduce the distracting influence of the primary principal components.
Consequently, running shallow detectors in this new space should garner
improvements. We discard half of the directions for the toy experiments to
form 64-dimensional embeddings. The anomaly detectors perform better in this
new space (Figure 7), corroborating the view that the neural network
embeddings introduce irrelevant directions.

We can also use the toy scenarios to attempt to understand the behaviour of
the detectors such as OCSVM. Our experiments suggest OCSVM fails when
anomalies lie in the centre of the normal data. For example, the AUROC for
OCSVM trained on the raw ring data signalled random performance at 50%,
whereas $k$-NN could detect the anomalies perfectly.

###  4.4 Analysing ODDS embeddings

(a) Classification results on the raw embeddings.

(b) Differences between linear classification performance on the raw
embeddings compared to the self-supervised embeddings, aggregated across the
ODDS datasets. Changes greater than 0 mean the self-supervision embedding
reduced separability.

Figure 8: Supervised linear classification results (normal versus anomaly) on
raw data (a) and supervised classification comparisons against the self-
supervised embeddings (b).

We now proceed to run ablations on ODDS. Previous studies have shown that
supervised classification performance correlates highly with out-of-
distribution detection performance [43]. Therefore, we train linear
classifiers on the self-supervised and original representation and compare
classification performance. If there is a drop in performance on the self-
supervised embeddings, the results would suggest the neural networks transform
the data in a way that mixes anomalies with the normal samples. We could
consequently attribute the poor self-supervised performance to this mixing
rather than the presence of irrelevant directions. Figure 8(a) illustrates
classification scores on the raw data while Figure 8(b) depicts the mean
difference between the raw and self-supervised classification performances.
Except for EICL, the differences are negligible. We can rule out the mixing
effect and conclude that self-supervision generally does not affect the
separability between the two classes.

Figure 9: Ablation study showing how shallow detector results vary with
subspace dimensionality.

We now investigate the residual space of the embeddings by extending the toy
dataset analyses to ODDS. We take the smallest eigenvalues (from 1% to 90% in
10% increments) to project the neural network embeddings to their residual
representations. We proceed to re-run the shallow anomaly detectors in the new
space. Figure 9 shows the results, aggregating both ResNet and FT-Transformer
scores. We observed similar behaviour across the two architectures. Reducing
the dimensionality indeed boosts performance. Throwing away the top 10% of
principal components helps, although throwing away the top 90% also performs
similarly. This observation aligns with previous findings that show residual
directions capture information important for out-of-distribution detection
[21]. The magnitude of normal data is minute in this space which is not
necessarily the case for anomalies. Based on these results, we do not need
complete neural network representations to perform anomaly detection. A subset
suffices.

###  4.5 Synthetic anomalies

Anomaly detection depends on two factors: the nature of the normal data and
the nature of anomalies. Both classes can originate from complex, irregular
distributions. These aspects make it difficult to pinpoint the causes of
results on ODDS and other curated datasets. We attempt to disentangle these
factors by analysing performance on synthetic anomalies. The anomalies curated
in ODDS are a composite of these types. We calculated the correlation between
the ODDS and the synthetic anomaly scores and found that the datasets
exhibited correlations between multiple synthetic categories, highlighting the
complex qualities of the anomalies. For example, when analysing the raw data
representations, $k$-NN on the curated Letter anomalies correlates strongly
with local ($\rho=0.84$), global ($\rho=0.49$), and dependency ($\rho=0.94$)
anomaly scores.

(a) Local anomalies ($\alpha=2$).

(b) Cluster anomalies ($\beta$ = 2).

(c) Global anomalies ($\delta=0.01$).

(d) Dependency anomalies.

Figure 10: Bar plots comparing synthetic anomaly results across the
representations.

Figures 10(a) to 10(d) show the results across the four synthetic types. We
show comparisons using $k$-NN as we found similar behaviours across the
detectors. The contrastive objectives outperform the baseline in the local
(Figure 10(a)) and cluster anomaly (Figure 10(b)) scenarios. This result
suggests contrastive tasks are better at discerning differences at a local
neighbourhood level.

No self-supervised approach beats the baseline when faced with global
anomalies (Figure 10(c)). This result contributes to the idea that self-
supervised representations introduce irrelevant directions. Since the global
anomalies scatter across the representation space, these additional directions
mask the meaningful distances between the anomalies and normal points. As a
result, methods like $k$-NN become less effective. In addition, the ranking of
the self-supervised tasks aligns most closely with their rankings on ODDS
(Figure 13), which potentially highlights the overall properties of the ODDS
datasets.

For the dependency anomalies, rotation and mask classification surpass the
baseline (Figure 10(d)). Conversely, contrastive tasks perform the worst.
Using a rotation or mask classification pretext task could help promote the
intrinsic property that tabular data are non-invariant, which may help
identify this type of anomaly.

###  4.6 Architectural choices for self-supervision

(a) Architectural results.

(b) Standardisation results.

(c) Classification losses.

(d) Contrastive losses.

Figure 11: Comparisons of how architecture and losses affect performance on
the self-supervised embeddings.

We analyse the effects of architectures and loss functions on performance to
provide starting points for improving deep learning methods for tabular
anomaly detection. We illustrate the results using $k$-NN as we observe
similar behaviours across detectors.

ResNets outperform transformers. Our experiments indicate ResNets are a better
choice than FT-Transformer (Figure 11(a)). This result may be due to
transformers needing more training data during the learning phase [48] \- the
ODDS datasets are relatively small.

Standardisation is not necessary. Standardising data before training neural
networks does not offer much benefit (Figure 11(b)).

ARPL is a better choice for classification-type losses. ARPL significantly
outperforms cross-entropy and AAM when training classification-type tasks
(Figure 11(c)). Specialised losses like ARPL might represent “other” spaces
better in the context of smaller datasets.

InfoNCE is better than VICReg for contrastive-type losses. This result (Figure
11(d)) may be due to the intricacies of VICReg, which requires balancing three
components (pair similarity, variance and covariance).

###  4.7 Benchmarking unsupervised anomaly detection

Finally, we compare the performance of each of the detectors overall to see
how well they perform in one-class settings. We aggregate results across the
baseline and self-supervised embeddings to provide a more generalised
understanding of detector behaviour.

Figure 12: Box plot comparing detector performance on the raw and standardised
data. The results include all hyperparameter variations where available.
Figure 13: Critical difference diagram ranking the different detectors.

Figures 12 and 13 summarise the overall performances of each anomaly detector
on ODDS. Even with the inclusion of self-supervised representations, $k$-NN
performs best.

####  4.7.1 Hyperparameter ablations

We now examine the sensitivity of the detectors to changes in hyperparameters.
These experiments were conducted directly on the raw ODDS data only to
understand detector performance in an optimal representation space. By doing
so, these results enable a better understanding of the detectors’ inductive
biases and why they may deteriorate in suboptimal self-supervised
representations.

Figure 14: Line plot showing how $k$-NN varies with the change in the number
of nearest neighbours, aggregated across the ODDS datasets.

$k$-NN: Figure 14 shows performance remains relatively stable to changes in
$k$, suggesting the choice of this hyperparameter is trivial. As $k$-NN
considers global relationships, this result indicates that anomalies already
lie in distinct regions separate from the normal raw data.

Figure 15: Line plot showing how LOF varies with the change in the number of
nearest neighbours, aggregated across the ODDS dataset.

LOF: Figure 15 illustrates how LOF performance changes with $k$. Although LOF
and $k$-NN consider points in a neighbourhood, LOF is more sensitive to the
number of neighbours (as evidenced by the increase in performance when $k$ = 1
and $k$ = 5 for LOF). However, it is unclear how to choose a value of $k$ so
that LOF is competitive with the other detectors in the one-class setting.

Figure 16: Line plot showing how residual norm varies with the change in
residual dimensionality, aggregated across the ODDS dataset.

Residual norms: Figure 16 shows how performance varies with the percentage of
attributes used. There are no notable trends, although performance remains
better than random, even with a small subset (10%) of features. The number of
relevant attributes in the original representation space is dataset-dependent
as ODDS contains datasets from differing tasks. It is unclear how to choose
the number of features to maximise the performance of residual norms in the
original dataset space.

####  4.7.2 Corrupted input data

(a) Additional features.

(b) Removing features.

(c) Selecting features.

(d) Missing values.

Figure 17: Ablations showing how detector performance varies with changing
levels of corrupt data.

Adding uninformative features: All detectors are sensitive to irrelevant
features (Figure 17(a)). Although residual norms do not achieve the highest
performance, it is more stable under increasing noise levels. This result may
be due to the residuals capturing the most meaningful directions of the data.
In contrast, $k$-NN performance declines the most.

Removing and selecting important features: Overall, performance plateaus at
around 50% of attributes, suggesting half of the raw features are irrelevant
for anomaly detection. iForest and OCSVM are the most stable under varying
subsets of features (Figures 17(b) and 17(c)).

Missing values: Most detectors exhibit a slight decline in AUROC with
increasing proportions of missing values (Figure 17(d)). LOF is the exception,
as performance drops significantly.

Overall, the results indicate $k$-NN is the best-performing detector when
faced with clean and relevant features. However, the relative ranking of
detectors changes in the presence of corrupted input data. As observed in our
self-supervised results (Section 4.4), residual norms might be better at
filtering out noisy directions. Furthermore, when there are fewer relevant
features, iForest may be a better choice.

##  5 Conclusion

###  5.1 Limitations and future work

We limited our experiments to the ODDS, which is not necessarily
representative of all tabular anomaly datasets. Several datasets underwent
preprocessing during the curation of ODDS, which could affect results. For
example, the values in HTTP were log-transformed. In addition, the datasets
are relatively small. As neural networks (particularly transformers) benefit
from large amounts of data [48], it is unclear if self-supervision would be
more advantageous in the big data case.

Furthermore, we isolated our analyses by extracting embeddings at the
penultimate layer and running shallow anomaly detection algorithms. Although
feature extraction at this stage combined with simple detectors is a popular
strategy [11, 13, 14, 30], different parts of the neural network could provide
more informative features [19]. Moreover, we chose to use shallow detectors to
prioritise studying the effect of representations rather than studying the
detection approach. Furthermore, the original implementations of ICL and GOAD
evaluate anomalies using an entire neural network pipeline and use specific
architectures for the tasks. Adapting these implementations for a pretext task
with different architectures deviates from the original setup and could affect
performance. Future work could look at extending the experiments to examine
how varying pretext tasks with deep anomaly detection can yield better results
[49]. In addition, studies focusing on improving deep tabular anomaly
detectors could also start examining regularisation strategies. Our
experiments suggest neural networks add irrelevant features, hence
regularisation during the training process could help to control this
behaviour.

###  5.2 Summary

We trained multiple neural networks on various self-supervised pretext tasks
to learn new representations for ODDS, a series of tabular anomaly detection
datasets. We ran a suite of shallow anomaly detectors on the new embeddings
and compared the results to the performance of the original data. None of the
self-supervised representations outperformed the raw baseline.

We conducted ablations to try to understand this behaviour. Our empirical
findings suggested that neural networks introduce irrelevant features, which
degrade detector capability. As normal and anomalous data were easily
distinguishable in the original tabular representations, neural networks
merely stretched the data. They did not introduce any additional informative
information. However, we demonstrated performance was recoverable by
projecting the embeddings to a residual subspace.

As the anomalies from ODDS derive from complex distributions, we repeated the
experiments on synthetic data to understand the pretext tasks’ influence on
detecting particular anomaly types. We showed in specific scenarios that self-
supervision can be beneficial. Contrastive tasks were better at picking up
localised anomalies, while classification tasks were better at identifying
differences in dependency structures.

Finally, we studied different shallow detectors by aggregating performances
across the baseline and self-supervised representations. We showed that
localised methods like $k$-NN and LOF worked best on ODDS but were susceptible
to performance degradation with corrupted data. In contrast, iForest was more
robust. Our findings provided practical insights into when one detector might
be preferable to another.

Overall, our findings suggest current deep learning approaches do not add much
benefit when the original feature space succinctly represents the normal data.
This situation is often the case for tabular data, and we demonstrated this by
showing performance degrades when removing features in the original space. If
the feature space did not succinctly represent the normal data, we would not
observe such large degradations. This setup differs from other domains. For
example, pixels in images contain lots of semantically irrelevant information.
Therefore, neural networks can distil information from pixels to extract
useful semantic features and self-supervision is beneficial.

##  6 Data availability

Publicly available datasets were analysed in this study. The ODDS datasets are
accessible from https://odds.cs.stonybrook.edu/.

##  Appendix A Additional figures

(a) 100% additional features

(b) 90% removed entries

(c) 90% removed features

(d) 10% selected features

Figure 18: Box plot comparing nearest neighbour AUROCs for each of the self-
supervised pretext tasks on corrupted input data.

(a) Curve

(b) Flower

(c) Gaussians

(d) Multiple Gaussians

(e) T-SNE projection of Letter from ODDS

(f) Moons

(g) Ring

(h) Pinched ring

(i) Spiral

Figure 19: Illustrations of the toy test data. Blue points are normal whereas
orange points are anomalous.

## References

  *   * Hendrycks et al. [2019] Hendrycks, D., Mazeika, M., Kadavath, S., Song, D.: Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty. Curran Associates Inc., Red Hook, NY, USA (2019)
  * Mai et al. [2022] Mai, K.T., Davies, T., Griffin, L.D.: Self-supervised losses for one-class textual anomaly detection. CoRR abs/2204.05695 (2022) https://doi.org/10.48550/arXiv.2204.05695 2204.05695
  * Reiss et al. [2022] Reiss, T., Cohen, N., Horwitz, E., Abutbul, R., Hoshen, Y.: Anomaly detection requires better representations. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds.) Computer Vision - ECCV 2022 Workshops, October 23-27, 2022, Proceedings, Part IV. Lecture Notes in Computer Science, vol. 13804, pp. 56–68. Springer, Tel Aviv, Israel (2022). https://doi.org/10.1007/978-3-031-25069-9_4 . https://doi.org/10.1007/978-3-031-25069-9_4
  * Balestriero et al. [2023] Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., Schwarzschild, A., Wilson, A.G., Geiping, J., Garrido, Q., Fernandez, P., Bar, A., Pirsiavash, H., LeCun, Y., Goldblum, M.: A cookbook of self-supervised learning. CoRR abs/2304.12210 (2023) https://doi.org/%****␣main.bbl␣Line␣125␣****10.48550/arXiv.2304.12210 2304.12210
  * Zhang et al. [2016] Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision - ECCV 2016 - 14th European Conference, October 11-14, 2016, Proceedings, Part III. Lecture Notes in Computer Science, vol. 9907, pp. 649–666. Springer, Amsterdam, The Netherlands (2016). https://doi.org/10.1007/978-3-319-46487-9_40 . https://doi.org/10.1007/978-3-319-46487-9_40
  * Peters et al. [2018] Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. In: Walker, M.A., Ji, H., Stent, A. (eds.) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, June 1-6, 2018, Volume 1 (Long Papers), pp. 2227–2237. Association for Computational Linguistics, New Orleans, Louisiana, USA (2018). https://doi.org/10.18653/v1/n18-1202 . https://doi.org/10.18653/v1/n18-1202
  * Radford et al. [2018] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training (2018)
  * Geirhos et al. [2019] Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., Brendel, W.: Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In: 7th International Conference on Learning Representations, ICLR 2019, May 6-9, 2019. OpenReview.net, New Orleans, Louisiana, USA (2019). https://openreview.net/forum?id=Bygh9j09KX
  * Hermann et al. [2020] Hermann, K.L., Chen, T., Kornblith, S.: The origins and prevalence of texture bias in convolutional neural networks. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual (2020). https://proceedings.neurips.cc/paper/2020/hash/db5f9f42a7157abe65bb145000b5871a-Abstract.html
  * Ruff et al. [2021] Ruff, L., Kauffmann, J.R., Vandermeulen, R.A., Montavon, G., Samek, W., Kloft, M., Dietterich, T.G., Müller, K.: A unifying review of deep and shallow anomaly detection. Proc. IEEE 109(5), 756–795 (2021) https://doi.org/10.1109/JPROC.2021.3052449
  * Gu et al. [2019] Gu, X., Akoglu, L., Rinaldo, A.: Statistical analysis of nearest neighbor methods for anomaly detection. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 10921–10931 (2019). https://proceedings.neurips.cc/paper/2019/hash/805163a0f0f128e473726ccda5f91bac-Abstract.html
  * Bergman et al. [2020] Bergman, L., Cohen, N., Hoshen, Y.: Deep nearest neighbor anomaly detection. CoRR abs/2002.10445 (2020) 2002.10445
  * Reiss et al. [2021] Reiss, T., Cohen, N., Bergman, L., Hoshen, Y.: PANDA: adapting pretrained features for anomaly detection and segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, Virtual, June 19-25, 2021, pp. 2806–2814. Computer Vision Foundation / IEEE, Virtual (2021). https://doi.org/10.1109/CVPR46437.2021.00283 . https://openaccess.thecvf.com/content/CVPR2021/html/Reiss_PANDA_Adapting_Pretrained_Features_for_Anomaly_Detection_and_Segmentation_CVPR_2021_paper.html
  * Sun et al. [2022] Sun, Y., Ming, Y., Zhu, X., Li, Y.: Out-of-distribution detection with deep nearest neighbors. In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., Sabato, S. (eds.) International Conference on Machine Learning, ICML 2022, 17-23 July 2022. Proceedings of Machine Learning Research, vol. 162, pp. 20827–20840. PMLR, Baltimore, Maryland, USA (2022). https://proceedings.mlr.press/v162/sun22d.html
  * Breunig et al. [2000] Breunig, M.M., Kriegel, H., Ng, R.T., Sander, J.: LOF: identifying density-based local outliers. In: Chen, W., Naughton, J.F., Bernstein, P.A. (eds.) Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, May 16-18, 2000, Dallas, Texas, USA, pp. 93–104. ACM, Dallas, Texas, USA (2000). https://doi.org/10.1145/342009.335388 . https://doi.org/10.1145/342009.335388
  * Liu et al. [2008] Liu, F.T., Ting, K.M., Zhou, Z.: Isolation forest. In: Proceedings of the 8th IEEE International Conference on Data Mining (ICDM 2008), December 15-19, 2008, pp. 413–422. IEEE Computer Society, Pisa, Italy (2008). https://doi.org/10.1109/ICDM.2008.17 . https://doi.org/10.1109/ICDM.2008.17
  * Schölkopf et al. [1999] Schölkopf, B., Williamson, R.C., Smola, A.J., Shawe-Taylor, J., Platt, J.C.: Support vector method for novelty detection. In: Solla, S.A., Leen, T.K., Müller, K. (eds.) Advances in Neural Information Processing Systems 12, [NIPS Conference, November 29 - December 4, 1999], pp. 582–588. The MIT Press, Denver, Colorado, USA (1999). http://papers.nips.cc/paper/1723-support-vector-method-for-novelty-detection
  * Huang et al. [2006] Huang, L., Nguyen, X., Garofalakis, M.N., Jordan, M.I., Joseph, A.D., Taft, N.: In-network PCA and anomaly detection. In: Schölkopf, B., Platt, J.C., Hofmann, T. (eds.) Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, December 4-7, 2006, pp. 617–624. MIT Press, Vancouver, British Columbia, Canada (2006). https://proceedings.neurips.cc/paper/2006/hash/2227d753dc18505031869d44673728e2-Abstract.html
  * Kim et al. [2020] Kim, K.H., Shim, S., Lim, Y., Jeon, J., Choi, J., Kim, B., Yoon, A.S.: Rapp: Novelty detection with reconstruction along projection pathway. In: 8th International Conference on Learning Representations, ICLR 2020, April 26-30, 2020. OpenReview.net, Addis Ababa, Ethiopia (2020). https://openreview.net/forum?id=HkgeGeBYDB
  * Wang et al. [2022] Wang, H., Li, Z., Feng, L., Zhang, W.: Vim: Out-of-distribution with virtual-logit matching. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, June 18-24, 2022, pp. 4911–4920. IEEE, New Orleans, Louisiana, USA (2022). https://doi.org/10.1109/CVPR52688.2022.00487 . https://doi.org/10.1109/CVPR52688.2022.00487
  * Kamoi and Kobayashi [2020] Kamoi, R., Kobayashi, K.: Why is the mahalanobis distance effective for anomaly detection? CoRR abs/2003.00402 (2020) 2003.00402
  * Gidaris et al. [2018] Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: 6th International Conference on Learning Representations, ICLR 2018, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, Vancouver, British Columbia, Canada (2018). https://openreview.net/forum?id=S1v4N2l0-
  * Noroozi and Favaro [2016] Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision - ECCV 2016 - 14th European Conference, October 11-14, 2016, Proceedings, Part VI. Lecture Notes in Computer Science, vol. 9910, pp. 69–84. Springer, Amsterdam, The Netherlands (2016). https://doi.org/10.1007/978-3-319-46466-4_5 . https://doi.org/10.1007/978-3-319-46466-4_5
  * Pathak et al. [2016] Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, June 27-30, 2016, pp. 2536–2544. IEEE Computer Society, Las Vegas, Nevada, USA (2016). https://doi.org/10.1109/CVPR.2016.278 . https://doi.org/10.1109/CVPR.2016.278
  * He et al. [2022] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.B.: Masked autoencoders are scalable vision learners. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, June 18-24, 2022, pp. 15979–15988. IEEE, ??? (2022). https://doi.org/10.1109/CVPR52688.2022.01553 . https://doi.org/10.1109/CVPR52688.2022.01553
  * van den Oord et al. [2018] Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. CoRR abs/1807.03748 (2018) 1807.03748
  * Chen et al. [2020] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.E.: A simple framework for contrastive learning of visual representations. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020. Proceedings of Machine Learning Research, vol. 119, pp. 1597–1607. PMLR, Virtual (2020). http://proceedings.mlr.press/v119/chen20j.html
  * Bardes et al. [2022] Bardes, A., Ponce, J., LeCun, Y.: Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In: The Tenth International Conference on Learning Representations, ICLR 2022, April 25-29, 2022. OpenReview.net, Virtual (2022). https://openreview.net/forum?id=xm6YD62D1Ub
  * Golan and El-Yaniv [2018] Golan, I., El-Yaniv, R.: Deep anomaly detection using geometric transformations. In: Bengio, S., Wallach, H.M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 9781–9791 (2018). https://proceedings.neurips.cc/paper/2018/hash/5e62d03aec0d17facfc5355dd90d441c-Abstract.html
  * Sehwag et al. [2021] Sehwag, V., Chiang, M., Mittal, P.: SSD: A unified framework for self-supervised outlier detection. In: 9th International Conference on Learning Representations, ICLR 2021, May 3-7, 2021. OpenReview.net, Virtual (2021). https://openreview.net/forum?id=v5gjXpmR8J
  * Tack et al. [2020] Tack, J., Mo, S., Jeong, J., Shin, J.: CSI: novelty detection via contrastive learning on distributionally shifted instances. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual (2020). https://proceedings.neurips.cc/paper/2020/hash/8965f76632d7672e7d3cf29c87ecaa0c-Abstract.html
  * Bergman and Hoshen [2020] Bergman, L., Hoshen, Y.: Classification-based anomaly detection for general data. In: 8th International Conference on Learning Representations, ICLR 2020, April 26-30, 2020. OpenReview.net, Addis Ababa, Ethiopia (2020). https://openreview.net/forum?id=H1lK_lBtvS
  * Shenkar and Wolf [2022] Shenkar, T., Wolf, L.: Anomaly detection for tabular data with internal contrastive learning. In: The Tenth International Conference on Learning Representations, ICLR 2022 April 25-29, 2022. OpenReview.net, Virtual (2022). https://openreview.net/forum?id=_hszZbt46bT
  * Rayana [2016] Rayana, S.: ODDS Library (2016). https://odds.cs.stonybrook.edu
  * Han et al. [2022] Han, S., Hu, X., Huang, H., Jiang, M., Zhao, Y.: Adbench: Anomaly detection benchmark. In: NeurIPS (2022). http://papers.nips.cc/paper_files/paper/2022/hash/cf93972b116ca5268827d575f2cc226b-Abstract-Datasets_and_Benchmarks.html
  * Pedregosa et al. [2011] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., VanderPlas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011) https://doi.org/10.5555/1953048.2078195
  * Johnson et al. [2021] Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with gpus. IEEE Trans. Big Data 7(3), 535–547 (2021) https://doi.org/10.1109/TBDATA.2019.2921572
  * Grinsztajn et al. [2022] Grinsztajn, L., Oyallon, E., Varoquaux, G.: Why do tree-based models still outperform deep learning on typical tabular data? In: NeurIPS (2022). http://papers.nips.cc/paper_files/paper/2022/hash/0378c7692da36807bdec87ab043cdadc-Abstract-Datasets_and_Benchmarks.html
  * Yoon et al. [2020] Yoon, J., Zhang, Y., Jordon, J., Schaar, M.: VIME: extending the success of self- and semi-supervised learning to tabular domain. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual (2020). https://proceedings.neurips.cc/paper/2020/hash/7d97667a3e056acab9aaf653807b4a03-Abstract.html
  * Gorishniy et al. [2021] Gorishniy, Y., Rubachev, I., Khrulkov, V., Babenko, A.: Revisiting deep learning models for tabular data. In: Ranzato, M., Beygelzimer, A., Dauphin, Y.N., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, Virtual, pp. 18932–18943 (2021). https://proceedings.neurips.cc/paper/2021/hash/9d86d83f925f2149e9edb0ac3b49229c-Abstract.html
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, June 27-30, 2016, pp. 770–778. IEEE Computer Society, Las Vegas, Nevada, USA (2016). https://doi.org/10.1109/CVPR.2016.90 . https://doi.org/10.1109/CVPR.2016.90
  * Chen et al. [2022] Chen, G., Peng, P., Wang, X., Tian, Y.: Adversarial reciprocal points learning for open set recognition. IEEE Trans. Pattern Anal. Mach. Intell. 44(11), 8065–8081 (2022) https://doi.org/10.1109/TPAMI.2021.3106743
  * Vaze et al. [2022] Vaze, S., Han, K., Vedaldi, A., Zisserman, A.: Open-set recognition: A good closed-set classifier is all you need. In: The Tenth International Conference on Learning Representations, ICLR 2022, April 25-29, 2022. OpenReview.net, Virtual (2022). https://openreview.net/forum?id=5hLP5JY9S2d
  * Deng et al. [2022] Deng, J., Guo, J., Yang, J., Xue, N., Kotsia, I., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 5962–5979 (2022) https://doi.org/10.1109/TPAMI.2021.3087709
  * Ming et al. [2023] Ming, Y., Sun, Y., Dia, O., Li, Y.: How to exploit hyperspherical embeddings for out-of-distribution detection? In: The Eleventh International Conference on Learning Representations (2023). https://openreview.net/forum?id=aEFaE0W5pAd
  * Steinbuss and Böhm [2021] Steinbuss, G., Böhm, K.: Benchmarking unsupervised outlier detection with realistic synthetic data. ACM Trans. Knowl. Discov. Data 15(4), 65–16520 (2021) https://doi.org/10.1145/3441453
  * Stolfo et al. [1999] Stolfo, S., Fan, W., Lee, W., Prodromidis, A., Chan, P.: KDD Cup 1999 Data. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C51C7N (1999)
  * Dosovitskiy et al. [2021] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: 9th International Conference on Learning Representations, ICLR 2021, May 3-7, 2021. OpenReview.net, Virtual (2021). https://openreview.net/forum?id=YicbFdNTTy
  * Ruff et al. [2018] Ruff, L., Görnitz, N., Deecke, L., Siddiqui, S.A., Vandermeulen, R.A., Binder, A., Müller, E., Kloft, M.: Deep one-class classification. In: Dy, J.G., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, ICML 2018, July 10-15, 2018. Proceedings of Machine Learning Research, vol. 80, pp. 4390–4399. PMLR, Stockholmsmässan, Stockholm, Sweden (2018). http://proceedings.mlr.press/v80/ruff18a.html