:

# Unmasking the Chameleons: A Benchmark for Out-of-Distribution Detection in
Medical Tabular Data

Mohammad Azizmalayeri 1<EMAIL_ADDRESS>
Ameen Abu-Hanna 1<EMAIL_ADDRESS>
Giovanni Ciná 1,2,3<EMAIL_ADDRESS>
1Department of Medical Informatics  Amsterdam Public Health Research Institute
Amsterdam UMC  University of Amsterdam  Netherlands.
2Institute of Logic  Language and Computation  University of Amsterdam
Netherlands.
3Pacmed  Amsterdam  Netherlands

###### Abstract

Despite their success, Machine Learning (ML) models do not generalize
effectively to data not originating from the training distribution. To
reliably employ ML models in real-world healthcare systems and avoid
inaccurate predictions on out-of-distribution (OOD) data, it is crucial to
detect OOD samples. Numerous OOD detection approaches have been suggested in
other fields - especially in computer vision - but it remains unclear whether
the challenge is resolved when dealing with medical tabular data. To answer
this pressing need, we propose an extensive reproducible benchmark to compare
different methods across a suite of tests including both near and far OODs.
Our benchmark leverages the latest versions of eICU and MIMIC-IV, two public
datasets encompassing tens of thousands of ICU patients in several hospitals.
We consider a wide array of density-based methods and SOTA post-hoc detectors
across diverse predictive architectures, including MLP, ResNet, and
Transformer. Our findings show that i) the problem appears to be solved for
far-OODs, but remains open for near-OODs; ii) post-hoc methods alone perform
poorly, but improve substantially when coupled with distance-based mechanisms;
iii) the transformer architecture is far less overconfident compared to MLP
and ResNet.

###### keywords:

Out-of-Distribution Detection, Medical Tabular Data

##  1 Introduction

Figure 1:  The practical dilemma of OOD data: there are no guarantees on how a
model will perform on OOD data, hence real-time OOD detection becomes
imperative.

The utilization of ML models in health-related applications is rapidly
increasing (Greener et al., 2022; Varoquaux and Cheplygina, 2022). However, a
significant limitation lies in their performance evaluation, which is
primarily based on optimizing the algorithms for data from the training
distribution. This means that they may fail under distribution shift: a model
trained on the data from a hospital may not generalize to other hospitals
(Rios and Abu-Hanna, 2021; de Hond et al., 2023). Since such ML models are
meant to be deployed in real-world healthcare scenarios, ensuring their
reliability becomes of utmost importance. One way to prevent models from
providing unreliable suggestions is to detect OOD samples in real time, prior
to generating predictions. This is known as OOD detection, where a model
trained on an in-distribution (ID) set is employed for distinguishing OOD
samples from ID data.

In this paper, we investigate this problem for tabular medical data. The
problem has already been investigated mainly in the field of computer vision
(Yang et al., 2021; Zimmerer et al., 2022), but the results may not extend to
tabular medical data. For example, while computer vision has focused on
improving post-hoc OOD detectors (Yang et al., 2022), it is demonstrated that
these kinds of methods perform even worse than a random classifier in medical
tabular data due to the phenomenon of over-confidence (Ulmer et al., 2020;
Ulmer and Cinà, 2021; Zadorozhny et al., 2022).

Existing OOD detection methods can be categorized into three main groups: i)
post-hoc methods, which are detectors that can be applied on top of any
trained classifier, such as Maximum Softmax Probability (MSP) (Hendrycks and
Gimpel, 2017), ii) density-based methods, which are trained to estimate the
marginal distribution of the training set in order to detect samples that fall
out of the training distribution, such as auto-encoders (AEs) (Kingma and
Welling, 2014), and finally iii) methods that require retraining of the
prediction model which are mainly designed specifically to be applied on
images, such as OpenGAN (Kong and Ramanan, 2021).

The focus of this work is medical tabular data, thus we only consider the
first two OOD categories since they can be applied to any medical tabular
dataset without limitations. Furthermore, we examine three important
architectures for the classifier on which post-hoc methods are implemented,
namely MLP, ResNet, and FT-Transformer (Gorishniy et al., 2021)—to assess
their impact on OOD detection.

A crucial aspect of comparing these methods involves evaluating their
performance across diverse scenarios. Therefore, we incorporate the latest
publicly available datasets including eICU and MIMIC-IV (Pollard et al., 2018;
Johnson et al., 2023) in our experiments. We also employ different approaches
for classifying the data into ID and OOD sets. This lets us consider OOD sets
near to and far from the IDs in the experiments, which are referred to as
near-OOD and far-OOD (Fort et al., 2021). For example, to consider near-OOD
instances, the ID and OOD sets can be constructed by dividing a dataset based
on a distinguishing feature such as gender (male vs. female), and for a far-
OOD experiment, we can use a dataset from a different data generating process,
such as a distinct care protocol (de Hond et al., 2023), or generate
artificial OOD samples.

We use this setup to conduct broad experiments that can illustrate the
differences between various types of OOD detection methods in order to provide
the community with an extensive reproducible benchmark in the medical tabular
data domain111The codes are available at
https://github.com/mazizmalayeri/TabMedOOD.. Our results are provided in
section 5, and the main findings are as follows: i) the OOD detection problem
appears to be almost resolved on far-OOD samples, but it is still an open
issue for near-OODs, ii) the distance-based post-hoc OOD detectors are better
than other post-hoc methods, iii) unlike the claims in previous work, post-hoc
methods can be competitive with the density-based ones, iv) the transformer
architecture mitigates the problem of over-confidence that other models suffer
from.

##  2 Related Work

OOD detection has received a lot of attention, with several kinds of
approaches being proposed for this purpose. This has resulted in interest in
comparing these methods fairly within the same setting. Some benchmarks
compare methods using standard image datasets (Han et al., 2022; Yang et al.,
2022; Zhang et al., 2023a). However, it is still necessary to have benchmarks
that are specific to other domains and data modalities. For example, the MOOD
challenge (Zimmerer et al., 2022) investigated OOD detection within the
context of medical imaging, with various new methods showing outstanding
results (Marimont and Tarroni, 2021; Tan et al., 2022).

In the realm of medical tabular datasets, notable endeavors have been
undertaken. A pipeline is presented in Nicora et al. (2022) to train a model
on ICU data and detect test samples for which the model might exhibit poor
performance. BEDS-Bench (Avati et al., 2021) has provided a benchmark on the
generalization ability of ML models over electronic health records,
highlighting performance drops under distribution shift. Moreover, it has been
found that access to OOD data does not improve the test performance on ICU
data (Spathis and Hyland, 2022). The work by Ulmer et al. (2020), proposed one
of the first comparisons between basic OOD detectors in the space of medical
tabular data, pointing to the fact that the problem of OOD detection was wide
open. Also, some guidelines have been provided on how to evaluate an OOD
detector in practice within the context of medical data (Zadorozhny et al.,
2022) such as methods for selecting the OOD set during evaluation. Compared to
this related work, we benchmark the most recent SOTA OOD detection approaches
and SOTA architectures, leading to novel insight e.g. on the over-confidence
of transformers and the combination of post-hoc and distance-based methods.

##  3 Problem Definition and Evaluation Protocol

In the following, we describe the problem definition, the metrics used to
measure performance, and how we select the ID and OOD sets.

###  3.1 OOD Detection

In training any ML model, we require a training set
$\mathcal{D}_{in}=\\{(x_{i},y_{i})\\}_{i=1}^{n}$, where each instance $x_{i}$
has a label $y_{i}$. Our goal is to have a model
$f:\mathcal{X}\rightarrow\mathcal{R}$ and a binary classifier
$G_{\lambda}:\mathcal{X}\rightarrow\\{0,1\\}$ such that for a test input
$x\sim\mathcal{X}$:

$G_{\lambda}(x;f)=\begin{cases}\text{OOD}\quad\quad f(x)\geq\lambda\\\
\text{ID}\quad\quad f(x)<\lambda\end{cases}.$ (1)

The score according to $f$ is sometimes called the ‘novelty score’. Samples
whose novelty score is higher than the threshold $\lambda$ are classified as
OOD. Hence, the final goal would be to train model $f$ such that it assigns
lower scores to $x\sim\mathcal{D}_{in}$ and higher scores to
$x\sim\mathcal{D}_{out}$, where $\mathcal{D}_{out}$ is the dataset that
includes the OOD samples.

###  3.2 Metrics

To assess the separability of the ID and OOD scores given by $f$, we measure
the area under the receiver operating characteristic (AUROC) as a well-known
threshold-independent classification criterion, and FPR@95, which measures the
FPR at a $\lambda$ corresponding to the TPR being equal to 95%.

###  3.3 Near, Far, and Synthesized OOD

The distance between $\mathcal{D}_{in}$ and $\mathcal{D}_{out}$ plays an
important role in the performance of OOD detection methods. It is relatively
simpler to detect samples far from the training distribution than samples
close to it; we refer to the former samples as far-OOD and to the latter as
near-OOD. To consider this in the comparisons, we define different sets of ID
and OOD as follows.

Near and far OODs: Assuming that we have a dataset $\mathcal{D}$, we can
define the ID and OOD sets in two ways: First, we can separate $\mathcal{D}$
based on a specific feature such as gender (male vs. female) or age (e.g.
elderly vs. young). This reflects what may happen when employing an ML model
in practice. As an example, if one develops a model on a population of mostly
young people, it might not be fully reliable when applied to the elderly. The
second way would be to use $\mathcal{D}$ as ID and a totally different dataset
as OOD. Since identifying OODs in the second scenario appears to be easier, we
will refer to the first scenario as near-OOD and the second as far-OOD
following the convention in computer vision (Winkens et al., 2020; Fort et
al., 2021).

Synthesized-OOD: Following the data corruption suggested in Ulmer et al.
(2020), we can simulate the OOD samples by scaling a single feature from ID
set by a factor of 10, 100, or 1000. For each factor, we will repeat the
experiments 100 times with different features, and average the results to
isolate the influence of scaling in the results, minimizing the impact of the
chosen feature. By increasing the scaling factor, it looks like we are
gradually transforming near-OOD samples into far-OOD ones.

##  4 Experiment Setup

###  4.1 Supported OOD Detectors

Two main OOD detection method categories, described in Table 1, are included
in our experiments. The first category is density estimators, which learn the
marginal distribution of the ID data and label OOD samples by comparison to
such distribution. We include 7 density-based models covering different types
of density estimators. These methods are used in prior works and have reached
outstanding results (Ulmer et al., 2020; Zadorozhny et al., 2022).

The second group is the post-hoc detectors, which can be integrated into any
pre-trained classifier without requiring any additional fine-tuning or
training. They mostly generate a novelty score for the input based on the
classifier output or intermediate representations. We evaluate 17 different
post-hoc detectors, including both commonly used detectors and top-performing
ones in their respective fields.

Method | Short Description
---|---
AE, VAE (Kingma and Welling, 2014) | Encodes input in a latent representation and reconstructs it.
HI-VAE (Nazabal et al., 2020) | Modifies VAE to consider heterogeneous data.
Flow (Papamakarios et al., 2017) | Estimates ID data by transformations into a normal distribution.
PPCA (Tipping and Bishop, 1999) | Reduces data dimensionality based on singular value decomposition.
LOF (De Vries et al., 2010) | Compares local density of input to the density of its closest neighbors.
DUE (van Amersfoort et al., 2021) | Uses a deep neural Gaussian process for modeling the ID data.
MDS (Lee et al., 2018) | Uses Mahalanobis distances to the class-conditional normal distributions.
RMDS (Ren et al., 2021) | Modifies MDS for detecting near-OODs.
KNN (Sun et al., 2022) |  Measures distance of input to the $k_{\text{th}}$ nearest neighbor in the ID data.
VIM (Wang et al., 2022) | Uses logits and features norm simultaneously.
SHE (Zhang et al., 2023b) | Measures the distance of input to the class-conditional representations.
KLM (Hendrycks et al., 2022) | Uses KL distance of softmax output from its range over the ID data.
OpenMax (Bendale and Boult, 2016) | Fits a Weibull distribution on the logits instead of softmax.
MSP (Hendrycks and Gimpel, 2017) | Uses maximum softmax probability as a simple but effective baseline.
MLS (Hendrycks et al., 2022) | Uses the maximum logit score instead of MSP.
TempScale (Guo et al., 2017) | Calibrates the temperature parameter in the softmax.
ODIN (Liang et al., 2018) | Perturbs the input adversarially before using TempScaling.
EBO (Liu et al., 2020) | Uses an energy function instead of softmax.
GRAM (Sastry and Oore, 2020) | Measures deviation of the Gram matrix from its range over the ID data.
GradNorm (Huang et al., 2021) | Uses norm of the backpropagated gradients.
ReAct (Sun et al., 2021) | Rectifies the model activations at an upper limit.
DICE (Sun and Li, 2022) | Suggests sparsification of the last linear layer.
ASH (Djurisic et al., 2023) | Extends the DICE idea to the intermediate feature layers.
Table 1:  Density-based models and post-hoc detectors evaluated in this work,
with a short description.

###  4.2 Datasets

We use eICU (Pollard et al., 2018) and MIMIC-IV (Johnson et al., 2023) in our
experiments as two public and popular datasets encompassing tens of thousands
of ICU patients in several hospitals. Following the descriptions in section
3.3, each of these datasets is considered as far-OOD set for the other one.
For the near-OOD case, we divide the datasets based on the time-independent
variables, as this choice better emulates the potential distribution shifts in
ID data that can be encountered in practice than the time-dependent ones.
Among the time-independent variables available for each dataset, we have
selected age (older than 70 as ID), gender (females as ID), and ethnicity
(Caucasian or African American as ID) in eICU, and Age (older than 70 as ID),
gender (females as ID), admission type (surgical in the same day of admission
as ID), and first care unit (CVICU as ID) in MIMIC-IV dataset. In each case,
the remaining part of the dataset would be OOD. The results for the age (older
than 70), ethnicity (Caucasian), and first care unit (CVICU) are reported in
the main text, and others in Appendix C.

###  4.3 Pre-processing

We pre-processed the eICU data using the pipeline provided in Sheikhalishahi
et al. (2020). Subsequently, the data is filtered to keep only patients with a
length of stay of at least 48 hours and an age greater than 18 years old.
Additionally, data with unknown discharge status were removed. Furthermore,
patients with NaN values in the features used in our models are also removed.
This pre-process resulted in a total $54826$ unique patients for this dataset.

For the MIMIC-IV, we used the pipeline provided in Gupta et al. (2022) with
slight modifications e.g., we added a mapping from the feature IDs to feature
names to have each feature name in the final pre-processed data. The data is
then filtered similarly to the eICU dataset. This resulted in $18180$ unique
patients for this dataset.

These datasets contain different types of clinical variables, but they are not
recorded for all the patients. To avoid NaN values as much as possible, we are
interested in using only the more important variables that are recorded for
more patients. Based on these criteria and considering the important clinical
variables suggested in Ulmer et al. (2020), we have selected a combination of
time-dependent and time-independent variables for each dataset, which are
reported in Appendix A.

It should be noted that when datasets are evaluated against each other, only
the variables found in both datasets are taken into account. Moreover, for the
time-dependent variables, we aggregated the time series by means of 6
different statistics including mean, standard deviation, minimum, maximum,
skewness, and number of observations calculated over windows consisting of the
full time-series and its first and last 10%, 25%, and 50%.

###  4.4 Task and Prediction Models

To perform post-hoc OOD detection, we would need a prediction model performing
a supervised classification task. The main task that is used in prior work is
mortality prediction (Sheikhalishahi et al., 2020; Ulmer et al., 2020;
Meijerink et al., 2020; Zadorozhny et al., 2022). In mortality prediction, we
only use the first 48 hours of data from the intensive care unit collected
from patients to predict the in-hospital mortality. It is noteworthy that the
mortality rate in the pre-processed data is $12.57\%$ in the MIMIC-IV and
$6.77\%$ in the eICU dataset.

To perform this task, we consider three widely used architectures: MLP,
ResNet, and FT-Transformer (Gorishniy et al., 2021). MLP passes data through
the fully-connected layers and non-linear activation functions with dropout to
improve the generalization. ResNet adds batchnorm and residual connections to
MLP. We also consider FT-Transformer, constituted of transformer blocks that
utilize the attention mechanism (Vaswani et al., 2017).

##  5 Results

In this section, we describe the results for each of the OOD settings based on
the AUROC criterion. Results for FPR@95 are presented in Appendix D. Moreover,
each experiment is repeated 5 times, and the results are averaged to reduce
the impact of randomness in selecting train/test data and training itself.
Additionally, we have measured the mortality prediction performance of the
prediction models in Appendix B, indicating that they are trained well.

###  5.1 Far-OOD

Results for the far-OOD setting are displayed in Table 2. According to this
table, there are methods that can effectively detect OOD data on each dataset.

Among the density-based methods, Flow attains the best result on eICU, while
others exhibit superior performance on MIMIC-IV. Additionally, DUE can detect
OODs on MIMIC-IV, but it falls short of being competitive on the eICU dataset.
Except for these two approaches and HI-VAE, other density-based methods
including AE, VAE, PPCA, and LOF demonstrate strong performance on both
datasets.

Within the post-hoc methodologies, MDS exhibits better results compared to
others regardless of the choice of the prediction model. Moreover, MDS applied
on ResNet is competitive with the density-based approaches, even marginally
outperforming them based on the average results across both datasets. After
MDS, there is not a single winner. However, approaches like KNN, VIM, and SHE
which somehow compute the distance to the training set as the novelty score,
outperform the rest.

Regarding the prediction model, the top-performing methods on ResNet
demonstrate superior results compared to both MLP and FT-Transformer. However,
a problem with ResNet and MLP is that they have over-confidence issues with
some approaches on MIMIC-IV. This means that they have more confidence in the
OODs, resulting in a performance even worse than a random detector. This is
mainly being observed with the detectors that do not rely on distance-based
novelty scores such as MSP and GradNorm. Note that the eICU dataset contains
data from a more extensive array of hospitals, which can increase diversity in
the data and reduce the over-confidence on this dataset.

FT-Transformer seems to solve the over-confidence problem observed in MLP and
ResNet, as all detectors perform better than random on both datasets. The
attention mechanism in the transformer blocks enables this model to consider
relationships between input elements, leading to a better understanding of ID
data.

###  5.2 Near-OOD

Results for the near-OOD setting are presented in Table 2. In the eICU
dataset, the diversity among the ID data and the proximity of OODs to ID have
collectively yielded an almost random performance for all the approaches.
Still, it indicates that methods like MDS and Flow are marginally better than
others.

In MIMIC-IV, which contains less diversity, the age variable still results in
an almost random performance across all approaches, but FCU reflects some
differences between detectors. Among the density-based approaches, AE and VAE
are the best choices, followed by PPAC, LOF, and HI-VAE. The post-hoc methods
mostly demonstrate similar performance within the same architecture category.
Moreover, they can be competitive with density-based methods when applied on
the FT-Transformer.

Model | Method | eICU | MIMIC-IV
---|---|---|---
Far-OOD | Near-OOD(Eth) | Near-OOD(Age) | Far-OOD | Near-OOD(FCU) | Near-OOD(Age)
| AE | 96.5$\pm$0.2 | 55.1$\pm$0.8 | 50.0$\pm$0.4 | 99.8$\pm$0.0 | 79.4$\pm$0.6 | 56.8$\pm$0.4
| VAE | 95.8$\pm$0.2 | 55.4$\pm$0.7 | 50.1$\pm$0.4 | 99.8$\pm$0.0 | 79.7$\pm$0.6 | 57.0$\pm$0.4
| HI-VAE | 56.7$\pm$1.6 | 44.3$\pm$1.2 | 44.4$\pm$2.1 | 68.8$\pm$8.8 | 79.1$\pm$13.4 | 56.8$\pm$10.0
Density | Flow | 100.0$\pm$0.0 | 61.1$\pm$0.9 | 49.7$\pm$0.4 | 87.4$\pm$7.5 | 31.8$\pm$3.9 | 51.6$\pm$0.5
Based | PPCA | 96.7$\pm$0.2 | 59.1$\pm$0.6 | 51.3$\pm$0.5 | 99.8$\pm$0.0 | 75.1$\pm$0.6 | 56.8$\pm$0.7
| LOF | 96.5$\pm$0.1 | 56.0$\pm$0.8 | 49.5$\pm$0.5 | 99.2$\pm$0.2 | 73.5$\pm$0.5 | 55.3$\pm$0.8
| DUE | 73.4$\pm$0.5 | 53.7$\pm$0.9 | 49.5$\pm$0.4 | 98.2$\pm$0.2 | 59.8$\pm$1.7 | 51.6$\pm$0.6
MLP | MDS | 84.3$\pm$1.4 | 56.8$\pm$0.9 | 51.7$\pm$0.7 | 98.9$\pm$0.6 | 68.0$\pm$3.9 | 53.8$\pm$0.9
RMDS | 59.2$\pm$2.1 | 50.0$\pm$1.7 | 49.1$\pm$0.5 | 79.9$\pm$12.1 | 60.9$\pm$1.1 | 50.5$\pm$0.4
KNN | 79.4$\pm$1.2 | 53.4$\pm$2.4 | 48.5$\pm$0.8 | 65.4$\pm$8.3 | 73.3$\pm$0.9 | 54.0$\pm$0.5
VIM | 69.9$\pm$1.4 | 51.2$\pm$1.7 | 46.7$\pm$0.8 | 97.8$\pm$2.4 | 71.1$\pm$1.2 | 54.0$\pm$0.9
SHE | 61.5$\pm$0.8 | 50.8$\pm$1.9 | 50.1$\pm$0.1 | 93.9$\pm$5.7 | 56.7$\pm$0.7 | 49.6$\pm$0.2
KLM | 56.8$\pm$1.2 | 52.7$\pm$1.8 | 51.7$\pm$0.2 | 79.5$\pm$5.8 | 46.1$\pm$2.0 | 49.3$\pm$0.6
OpenMax | 52.1$\pm$1.3 | 47.4$\pm$1.5 | 46.3$\pm$1.3 | 62.7$\pm$10.3 | 66.9$\pm$0.9 | 52.6$\pm$0.5
MSP | 51.2$\pm$1.2 | 47.1$\pm$1.8 | 46.3$\pm$1.3 | 13.4$\pm$7.7 | 66.0$\pm$0.6 | 52.4$\pm$0.4
MLP | MLS | 51.3$\pm$1.4 | 46.8$\pm$1.7 | 46.2$\pm$1.3 | 14.2$\pm$7.6 | 65.9$\pm$0.8 | 52.3$\pm$0.4
| TempScale | 51.2$\pm$1.2 | 46.8$\pm$1.8 | 46.3$\pm$1.3 | 13.3$\pm$7.7 | 66.0$\pm$0.8 | 52.4$\pm$0.3
| ODIN | 51.3$\pm$1.2 | 46.8$\pm$1.8 | 46.3$\pm$1.3 | 13.5$\pm$7.8 | 65.9$\pm$0.8 | 52.5$\pm$0.3
| EBO | 51.4$\pm$1.4 | 46.8$\pm$1.7 | 46.2$\pm$1.3 | 14.4$\pm$7.7 | 65.9$\pm$0.8 | 52.3$\pm$0.4
| GRAM | 46.7$\pm$1.4 | 46.6$\pm$1.5 | 48.4$\pm$0.6 | 16.7$\pm$11.7 | 53.6$\pm$4.1 | 49.9$\pm$0.1
| GradNorm | 50.2$\pm$1.3 | 46.8$\pm$1.8 | 46.4$\pm$1.3 | 12.7$\pm$7.9 | 65.8$\pm$0.8 | 52.5$\pm$0.4
| ReAct | 52.5$\pm$1.2 | 46.7$\pm$1.5 | 46.6$\pm$1.4 | 74.3$\pm$19.6 | 66.1$\pm$0.9 | 52.5$\pm$0.4
| DICE | 50.6$\pm$1.3 | 46.9$\pm$1.8 | 46.6$\pm$1.3 | 14.7$\pm$8.4 | 65.8$\pm$0.9 | 52.7$\pm$0.8
| ASH | 50.6$\pm$1.3 | 46.8$\pm$1.5 | 46.6$\pm$1.7 | 14.0$\pm$7.3 | 65.7$\pm$0.9 | 52.1$\pm$0.4
ResNet | MDS | 96.9$\pm$0.3 | 58.4$\pm$0.6 | 51.6$\pm$0.7 | 99.7$\pm$0.1 | 74.0$\pm$2.4 | 55.4$\pm$0.3
RMDS | 45.8$\pm$2.9 | 50.1$\pm$2.5 | 49.9$\pm$0.3 | 79.5$\pm$13.1 | 62.6$\pm$1.4 | 50.8$\pm$0.4
KNN | 91.2$\pm$0.9 | 59.2$\pm$1.6 | 50.1$\pm$0.1 | 93.0$\pm$1.9 | 57.7$\pm$2.3 | 54.3$\pm$0.5
VIM | 93.5$\pm$1.0 | 56.9$\pm$1.5 | 48.2$\pm$1.1 | 99.5$\pm$0.2 | 75.0$\pm$1.8 | 56.2$\pm$0.4
SHE | 86.7$\pm$1.6 | 50.5$\pm$1.3 | 51.7$\pm$0.8 | 99.7$\pm$0.1 | 73.0$\pm$1.6 | 52.8$\pm$0.6
KLM | 55.5$\pm$2.6 | 47.5$\pm$1.5 | 51.5$\pm$0.4 | 78.1$\pm$6.7 | 55.2$\pm$1.4 | 47.7$\pm$0.2
OpenMax | 64.8$\pm$5.4 | 50.7$\pm$1.3 | 47.0$\pm$1.1 | 65.2$\pm$24.9 | 67.2$\pm$4.1 | 54.3$\pm$0.4
MSP | 63.6$\pm$5.0 | 51.3$\pm$2.4 | 47.1$\pm$0.8 | 19.5$\pm$10.3 | 65.0$\pm$4.3 | 53.9$\pm$0.7
ResNet | MLS | 64.4$\pm$6.0 | 51.9$\pm$2.2 | 46.8$\pm$1.4 | 38.0$\pm$26.5 | 66.5$\pm$4.6 | 54.1$\pm$0.3
| TempScale | 63.6$\pm$5.0 | 51.3$\pm$2.4 | 47.1$\pm$0.8 | 19.5$\pm$10.3 | 65.0$\pm$4.3 | 53.9$\pm$0.7
| ODIN | 63.7$\pm$5.0 | 51.3$\pm$2.4 | 47.1$\pm$0.8 | 19.6$\pm$10.4 | 65.0$\pm$4.4 | 53.9$\pm$0.7
| EBO | 64.4$\pm$6.1 | 52.0$\pm$2.2 | 46.8$\pm$1.4 | 38.5$\pm$27.0 | 66.5$\pm$4.6 | 54.2$\pm$0.3
| GRAM | 39.3$\pm$2.6 | 48.6$\pm$1.2 | 49.0$\pm$0.9 | 11.3$\pm$5.0 | 44.2$\pm$1.7 | 49.6$\pm$0.1
| GradNorm | 48.1$\pm$5.6 | 49.8$\pm$1.6 | 47.5$\pm$1.1 | 7.5$\pm$4.8 | 48.2$\pm$6.9 | 52.7$\pm$0.6
| ReAct | 71.8$\pm$3.9 | 52.7$\pm$1.8 | 47.4$\pm$1.5 | 81.4$\pm$9.1 | 73.9$\pm$2.6 | 55.3$\pm$0.4
| DICE | 52.9$\pm$7.9 | 50.5$\pm$1.2 | 47.3$\pm$0.6 | 17.0$\pm$7.6 | 57.7$\pm$7.4 | 55.5$\pm$0.5
| ASH | 68.1$\pm$5.6 | 51.5$\pm$2.4 | 47.2$\pm$0.8 | 40.5$\pm$27.9 | 63.9$\pm$4.3 | 53.9$\pm$0.5
FT-T | MDS | 84.1$\pm$4.2 | 58.5$\pm$2.2 | 50.8$\pm$1.1 | 94.2$\pm$0.7 | 77.8$\pm$2.0 | 50.1$\pm$0.2
RMDS | 57.3$\pm$0.9 | 51.6$\pm$1.5 | 48.3$\pm$0.7 | 75.8$\pm$13.5 | 60.6$\pm$4.4 | 50.9$\pm$1.5
KNN | 85.5$\pm$3.7 | 55.8$\pm$1.9 | 49.6$\pm$0.2 | 92.0$\pm$0.6 | 78.6$\pm$3.2 | 51.7$\pm$1.5
VIM | 85.5$\pm$2.8 | 57.3$\pm$2.3 | 48.8$\pm$0.1 | 96.8$\pm$0.7 | 79.4$\pm$1.7 | 53.8$\pm$0.3
SHE | 62.9$\pm$1.3 | 50.5$\pm$1.7 | 50.4$\pm$0.7 | 85.2$\pm$9.1 | 67.5$\pm$0.5 | 47.1$\pm$0.5
KLM | 52.0$\pm$2.1 | 51.6$\pm$2.1 | 51.0$\pm$0.7 | 74.8$\pm$6.0 | 49.3$\pm$2.6 | 47.3$\pm$1.1
OpenMax | 74.3$\pm$2.4 | 48.7$\pm$0.8 | 48.1$\pm$0.5 | 66.5$\pm$8.3 | 77.7$\pm$3.5 | 53.5$\pm$0.2
MSP | 74.3$\pm$2.5 | 48.4$\pm$1.0 | 48.1$\pm$0.5 | 66.1$\pm$7.6 | 77.6$\pm$3.7 | 53.4$\pm$0.2
FT-T | MLS | 74.1$\pm$2.5 | 49.1$\pm$1.1 | 48.0$\pm$0.5 | 65.2$\pm$6.4 | 77.8$\pm$3.2 | 53.3$\pm$0.2
| TempScale | 74.3$\pm$2.5 | 48.4$\pm$1.0 | 48.1$\pm$0.5 | 66.1$\pm$7.6 | 77.6$\pm$3.7 | 53.4$\pm$0.2
| ODIN | 74.4$\pm$2.4 | 48.4$\pm$1.0 | 48.1$\pm$0.5 | 66.3$\pm$7.5 | 77.6$\pm$3.7 | 53.4$\pm$0.2
| EBO | 74.1$\pm$2.4 | 49.1$\pm$1.1 | 48.0$\pm$0.5 | 64.4$\pm$5.9 | 77.8$\pm$3.1 | 53.3$\pm$0.3
| GRAM | 73.5$\pm$2.3 | 50.8$\pm$2.7 | 48.6$\pm$0.6 | 70.0$\pm$6.8 | 75.2$\pm$5.4 | 51.7$\pm$0.4
| GradNorm | 69.5$\pm$5.9 | 50.4$\pm$3.4 | 48.7$\pm$0.8 | 66.4$\pm$6.8 | 62.5$\pm$8.9 | 53.1$\pm$0.5
| ReAct | 73.9$\pm$2.6 | 48.1$\pm$1.1 | 48.2$\pm$0.7 | 64.0$\pm$4.3 | 77.8$\pm$2.6 | 53.4$\pm$0.3
| DICE | 74.3$\pm$2.2 | 49.9$\pm$2.8 | 48.1$\pm$0.6 | 77.7$\pm$7.1 | 77.3$\pm$4.1 | 53.2$\pm$0.6
| ASH | 73.0$\pm$3.2 | 52.6$\pm$2.7 | 48.6$\pm$1.3 | 58.0$\pm$8.5 | 75.4$\pm$3.4 | 52.5$\pm$1.1
Table 2:  Comparing OOD detection methods in the far-OOD and near-OOD settings using eICU and MIMIC-IV datasets, with performance evaluated through AUROC. The name in the top row shows the dataset considered as ID in the far-OOD experiment and divided for the near-OOD test. FCU and Eth stand for first care unit and ethnicity as the variables utilized for constructing the ID/OOD sets. Results are averaged over 5 runs. For each model, the best value in each column is shown in bold, and distance-based post-hoc approaches are distinguished with a lighter color.  Model | Method | eICU | MIMIC-IV
---|---|---|---
$\mathcal{F}$=10 |  $\mathcal{F}$=100 |  $\mathcal{F}$=1000 |  $\mathcal{F}$=10 |  $\mathcal{F}$=100 |  $\mathcal{F}$=1000
| AE | 80.5$\pm$1.3 | 88.4$\pm$0.9 | 90.0$\pm$0.7 | 76.4$\pm$1.6 | 83.9$\pm$2.1 | 86.6$\pm$2.1
| VAE | 80.0$\pm$1.3 | 88.3$\pm$0.9 | 89.9$\pm$0.7 | 76.4$\pm$1.6 | 83.8$\pm$2.1 | 86.6$\pm$2.1
| HI-VAE | 50.0$\pm$0.1 | 50.0$\pm$0.1 | 50.1$\pm$0.2 | 50.59$\pm$0.8 | 51.3$\pm$1.4 | 52.0$\pm$2.0
Density | Flow | 70.2$\pm$2.5 | 82.1$\pm$2.5 | 87.7$\pm$1.4 | 53.8$\pm$1.8 | 65.0$\pm$3.0 | 75.7$\pm$2.5
Based | PPCA | 80.7$\pm$1.3 | 88.3$\pm$0.9 | 89.7$\pm$0.8 | 76.9$\pm$1.5 | 84.0$\pm$2.1 | 86.6$\pm$2.0
| LOF | 84.4$\pm$1.3 | 89.4$\pm$0.8 | 90.5$\pm$0.7 | 78.4$\pm$1.5 | 84.7$\pm$2.1 | 86.9$\pm$1.9
| DUE | 63.9$\pm$1.6 | 80.5$\pm$1.3 | 88.7$\pm$0.8 | 60.3$\pm$2.2 | 76.0$\pm$1.9 | 83.0$\pm$2.0
MLP | MDS | 68.5$\pm$2.4 | 82.8$\pm$2.0 | 89.2$\pm$2.0 | 69.3$\pm$2.4 | 80.8$\pm$1.6 | 84.7$\pm$1.8
RMDS | 60.8$\pm$1.0 | 75.2$\pm$2.0 | 85.8$\pm$2.0 | 52.7$\pm$4.1 | 64.5$\pm$6.9 | 76.4$\pm$2.8
KNN | 60.3$\pm$1.5 | 68.0$\pm$2.0 | 70.9$\pm$2.3 | 59.1$\pm$4.9 | 67.2$\pm$8.9 | 70.7$\pm$10.0
VIM | 48.5$\pm$2.0 | 47.2$\pm$3.1 | 46.0$\pm$4.9 | 56.8$\pm$7.9 | 63.5$\pm$13.7 | 66.6$\pm$15.1
SHE | 62.5$\pm$2.8 | 77.8$\pm$1.6 | 89.8$\pm$0.1 | 62.3$\pm$3.0 | 77.8$\pm$2.2 | 83.6$\pm$1.5
KLM | 60.2$\pm$1.4 | 69.6$\pm$1.5 | 78.1$\pm$1.4 | 55.4$\pm$2.0 | 66.9$\pm$1.6 | 74.0$\pm$1.5
OpenMax | 52.6$\pm$2.0 | 65.2$\pm$2.5 | 77.4$\pm$2.5 | 48.4$\pm$5.3 | 54.0$\pm$5.1 | 69.4$\pm$5.6
MSP | 40.5$\pm$2.2 | 27.1$\pm$3.0 | 14.2$\pm$2.4 | 45.7$\pm$4.8 | 31.7$\pm$5.5 | 21.8$\pm$1.3
MLP | MLS | 40.4$\pm$2.4 | 27.0$\pm$3.4 | 13.8$\pm$2.8 | 45.6$\pm$4.1 | 32.1$\pm$4.0 | 24.9$\pm$3.7
| TempScale | 40.4$\pm$2.2 | 27.0$\pm$3.1 | 14.0$\pm$2.4 | 45.7$\pm$4.8 | 31.6$\pm$5.5 | 21.7$\pm$1.3
| ODIN | 40.5$\pm$2.2 | 27.0$\pm$3.1 | 14.0$\pm$2.4 | 45.7$\pm$4.8 | 31.7$\pm$5.5 | 21.8$\pm$1.3
| EBO | 40.4$\pm$2.4 | 27.0$\pm$3.4 | 13.8$\pm$2.8 | 45.5$\pm$3.9 | 32.0$\pm$4.0 | 24.9$\pm$3.8
| GRAM | 38.7$\pm$2.0 | 25.2$\pm$2.6 | 12.7$\pm$2.1 | 47.0$\pm$0.7 | 32.8$\pm$3.0 | 20.4$\pm$2.2
| GradNorm | 40.2$\pm$2.1 | 26.4$\pm$2.9 | 13.2$\pm$2.4 | 42.6$\pm$2.4 | 26.1$\pm$1.1 | 18.8$\pm$1.4
| ReAct | 45.0$\pm$1.6 | 37.3$\pm$2.7 | 31.3$\pm$2.5 | 49.0$\pm$5.7 | 46.6$\pm$12.4 | 46.7$\pm$14.5
| DICE | 40.9$\pm$2.7 | 28.9$\pm$3.5 | 16.7$\pm$2.9 | 44.8$\pm$3.6 | 31.4$\pm$2.7 | 22.0$\pm$1.1
| ASH | 40.4$\pm$2.4 | 27.3$\pm$3.1 | 14.3$\pm$2.6 | 45.3$\pm$4.1 | 32.1$\pm$4.3 | 25.1$\pm$3.7
ResNet | MDS | 74.4$\pm$2.0 | 88.9$\pm$1.7 | 91.3$\pm$1.4 | 72.4$\pm$1.1 | 81.8$\pm$0.8 | 85.4$\pm$1.0
RMDS | 51.7$\pm$0.6 | 59.8$\pm$2.5 | 78.7$\pm$2.4 | 46.2$\pm$1.5 | 57.6$\pm$2.7 | 73.9$\pm$1.2
KNN | 69.5$\pm$2.1 | 84.8$\pm$2.2 | 87.9$\pm$2.2 | 67.5$\pm$1.5 | 79.1$\pm$1.2 | 83.7$\pm$0.8
VIM | 72.2$\pm$2.5 | 88.7$\pm$1.8 | 91.5$\pm$1.4 | 68.8$\pm$1.3 | 80.2$\pm$1.2 | 84.5$\pm$1.1
SHE | 68.2$\pm$0.7 | 86.6$\pm$0.5 | 91.5$\pm$0.2 | 67.2$\pm$1.4 | 80.0$\pm$0.9 | 84.5$\pm$0.7
KLM | 56.6$\pm$0.8 | 69.4$\pm$1.7 | 80.5$\pm$1.9 | 53.2$\pm$0.9 | 65.4$\pm$0.5 | 75.3$\pm$0.8
OpenMax | 57.5$\pm$0.7 | 66.4$\pm$4.3 | 77.4$\pm$2.9 | 55.2$\pm$1.0 | 61.6$\pm$1.4 | 61.7$\pm$11.4
MSP | 49.4$\pm$2.2 | 35.1$\pm$2.1 | 16.8$\pm$1.0 | 52.2$\pm$1.1 | 38.0$\pm$0.7 | 23.1$\pm$0.9
ResNet | MLS | 49.0$\pm$1.7 | 34.7$\pm$2.0 | 18.5$\pm$2.8 | 52.5$\pm$1.9 | 40.5$\pm$7.0 | 31.0$\pm$8.8
| TempScale | 49.4$\pm$2.2 | 35.1$\pm$2.1 | 16.8$\pm$1.0 | 52.2$\pm$1.1 | 38.0$\pm$0.7 | 23.1$\pm$0.9
| ODIN | 49.4$\pm$2.2 | 35.2$\pm$2.1 | 16.8$\pm$1.0 | 52.3$\pm$1.1 | 38.1$\pm$0.7 | 23.1$\pm$0.9
| EBO | 49.0$\pm$1.6 | 34.5$\pm$2.1 | 18.5$\pm$2.8 | 52.4$\pm$2.1 | 40.6$\pm$7.4 | 31.0$\pm$9.0
| GRAM | 37.0$\pm$3.1 | 17.7$\pm$3.0 | 9.1$\pm$1.6 | 46.6$\pm$1.2 | 29.0$\pm$0.7 | 18.9$\pm$0.9
| GradNorm | 40.1$\pm$2.4 | 20.4$\pm$1.0 | 9.7$\pm$1.6 | 44.9$\pm$1.0 | 25.7$\pm$0.9 | 17.2$\pm$0.3
| ReAct | 59.4$\pm$1.5 | 70.4$\pm$3.5 | 74.8$\pm$3.5 | 58.7$\pm$2.0 | 67.2$\pm$4.1 | 70.2$\pm$4.4
| DICE | 42.6$\pm$2.0 | 24.6$\pm$1.2 | 10.8$\pm$0.9 | 50.6$\pm$1.6 | 36.8$\pm$1.6 | 23.1$\pm$1.5
| ASH | 49.8$\pm$0.3 | 36.7$\pm$2.4 | 20.1$\pm$2.9 | 52.3$\pm$1.7 | 40.9$\pm$5.2 | 31.1$\pm$8.5
FT-T | MDS | 59.9$\pm$1.4 | 79.5$\pm$1.4 | 87.5$\pm$0.9 | 59.7$\pm$1.1 | 76.0$\pm$1.2 | 81.8$\pm$1.4
RMDS | 51.5$\pm$1.3 | 57.8$\pm$7.4 | 64.0$\pm$13.0 | 49.0$\pm$0.9 | 55.0$\pm$2.0 | 67.0$\pm$5.1
KNN | 57.3$\pm$1.4 | 75.4$\pm$2.2 | 86.5$\pm$1.3 | 57.6$\pm$1.2 | 74.6$\pm$1.0 | 81.4$\pm$1.3
VIM | 57.9$\pm$1.6 | 77.6$\pm$1.3 | 88.3$\pm$0.7 | 57.4$\pm$2.5 | 74.3$\pm$1.2 | 80.9$\pm$1.2
SHE | 55.7$\pm$1.3 | 71.2$\pm$2.9 | 80.4$\pm$1.6 | 53.2$\pm$0.7 | 67.6$\pm$1.6 | 80.8$\pm$2.4
KLM | 54.1$\pm$0.8 | 63.1$\pm$1.1 | 72.1$\pm$4.2 | 52.8$\pm$0.4 | 60.8$\pm$1.1 | 61.8$\pm$6.9
OpenMax | 51.0$\pm$0.7 | 56.1$\pm$2.7 | 71.4$\pm$3.2 | 48.8$\pm$0.9 | 54.1$\pm$2.6 | 66.4$\pm$3.8
MSP | 50.9$\pm$0.6 | 55.8$\pm$2.7 | 71.3$\pm$3.1 | 48.4$\pm$0.8 | 49.6$\pm$1.5 | 62.9$\pm$2.9
FT-T | MLS | 50.9$\pm$0.7 | 55.8$\pm$3.1 | 70.8$\pm$3.1 | 48.4$\pm$0.9 | 49.5$\pm$2.7 | 60.5$\pm$5.6
| TempScale | 50.9$\pm$0.6 | 55.8$\pm$2.7 | 71.3$\pm$3.1 | 48.4$\pm$0.8 | 49.6$\pm$1.5 | 62.9$\pm$2.9
| ODIN | 50.9$\pm$0.6 | 55.9$\pm$2.7 | 71.4$\pm$3.1 | 48.4$\pm$0.8 | 49.7$\pm$1.5 | 63.1$\pm$2.9
| EBO | 50.9$\pm$0.7 | 55.5$\pm$3.1 | 70.2$\pm$3.2 | 48.5$\pm$1.0 | 49.0$\pm$3.4 | 57.3$\pm$7.4
| GRAM | 48.9$\pm$0.1 | 50.1$\pm$0.8 | 62.5$\pm$3.0 | 47.8$\pm$1.3 | 44.0$\pm$2.7 | 49.6$\pm$7.6
| GradNorm | 52.0$\pm$0.8 | 59.5$\pm$2.9 | 73.6$\pm$2.9 | 48.8$\pm$0.9 | 51.4$\pm$1.7 | 64.2$\pm$2.7
| ReAct | 50.9$\pm$0.8 | 55.7$\pm$3.4 | 69.9$\pm$3.2 | 48.6$\pm$1.1 | 49.7$\pm$3.4 | 57.4$\pm$7.1
| DICE | 52.2$\pm$1.5 | 62.8$\pm$2.5 | 75.6$\pm$1.4 | 50.2$\pm$0.9 | 64.5$\pm$1.4 | 76.5$\pm$1.6
| ASH | 50.1$\pm$1.4 | 54.6$\pm$2.1 | 69.7$\pm$1.3 | 48.7$\pm$0.9 | 49.9$\pm$2.8 | 59.2$\pm$5.6
Table 3:  Comparing OOD detection methods in the synthesized-OOD setting using
eICU and MIMIC-IV datasets, with performance evaluated through AUROC. In this
setting, one feature is multiplied by a specific factor $\mathcal{F}$ to
construct an OOD sample. For each model and factor we display the average
performance, calculated over 100 resampling of the feature and 5 different
initializations of the model. For each model, the best value in each column is
shown in bold, and distance-based post-hoc approaches are distinguished with a
lighter color.  Figure 2: Depiction of confidence scores for different
architectures. High confidence (low entropy) is represented in light orange.
MLP and ResNet exhibit regions of confidence extending far away from the ID
data. FT-Transformer mitigates this over-confidence but does not solve it.

###  5.3 Synthesized-OOD

Results for the synthesized-OOD setting are displayed in Table 3. It is
expected that increasing the scaling factor ($\mathcal{F}$) in this setting
facilitates the detection of OODs. However, over-confidence hampers OOD
detection, causing certain methods to exhibit a totally opposite behavior with
MLP and ResNet architectures, on both datasets. In this case, even the
diversity in eICU does not prevent over-confidence. Similar to the far-OOD
scenario, FT-Transformer seems to solve this issue, as increasing
$\mathcal{F}$ results in an improved performance for this architecture.

To compare different approaches, density-based methods like AE, VAE, PPCA, and
LOF demonstrate better performance than post-hoc ones when $\mathcal{F}=10$.
However, this gap in performance is reduced with increasing $\mathcal{F}$ to
the extent that methods like MDS, VIM, and SHE applied on ResNet are
outperforming density-based ones on the eICU dataset with $\mathcal{F}=1000$.

###  5.4 Over-confidence

The results above showed over-confidence for the MLP and ResNet architectures,
whereas the FT-Transformer appeared as a solution to this problem. For a more
visual exploration of this issue, we employ a classification toy example. In
this example, each of the predictive architectures is trained on a multi-class
classification task with 2D samples. Next, the entropy of the model’s softmax
output is plotted for a wide array of inputs. Plots are shown in Fig. 2, with
a lighter color indicating more confidence (i.e. low entropy in the softmax
output). As depicted in this figure, MLP and ResNet confidence is increased by
getting further away from the ID data. This observation validates the presence
of over-confidence in both the MLP and ResNet models. Conversely, in the FT-
Transformer, confidence increases along certain directions and decreases in
others. This suggests that the transformer can mitigate the over-confidence in
some directions, but does not solve it entirely.

##  6 Discussion

According to our results, when OODs are far from ID data, there exist methods
that can effectively perform OOD detection. However, OODs near to ID data such
as multiplication by $\mathcal{F}=10$ or in a near-OOD setting are still
challenging to spot. While the poor results in the near-OOD scenarios might be
due to a large overlap with the ID data, when multiplying features with a
scaling factor one can ensure that there is a clear difference between such
OOD and ID data. This points to the fact that there is still ample room for
improvement in detecting near OODs.

Delving into OOD detectors, density-based methods, particularly AE, VAE, PPCA,
and Flow, exhibit consistent good performance across different settings and
datasets. Within the post-hoc methods, they perform poorly in some cases, but
they improve substantially and become competitive with the density-based ones
when used in conjunction with the distance-based mechanisms. For example, MDS
applied on ResNet can even marginally outperform density-based methods in some
cases, such as the experiment with a scaling factor of $\mathcal{F}=1000$ on
the eICU dataset. This nuances previous claims that post-hoc methods generally
exhibit poor performance (Ulmer et al., 2020; Zadorozhny et al., 2022).

To compare different prediction models, ResNet combined with distance-based
detectors demonstrates better results than MLP and FT-Transformer. On the
other hand, MLP and ResNet suffer from over-confidence, causing certain
detectors to perform even worse than a random classifier. This aligns with
what was highlighted in prior studies (Hein et al., 2019; Ulmer and Cinà,
2021). Our numerical results suggest that FT-Transformer could be a solution
to this problem, however, our simple example with toy data showcases that
transformers do not completely eliminate over-confidence.

This benchmark is built on two intensive care datasets, which are highly
granular. Hence, caution should be exercised when transporting these findings
to alternative healthcare tabular datasets with different characteristics. To
facilitate the extension of this benchmark, we provided a modular
implementation allowing for the addition of new datasets and methods to the
experiments.

## References

  * Avati et al. (2021) Anand Avati, Martin Seneviratne, Yuan Xue, Zhen Xu, Balaji Lakshminarayanan, and Andrew M. Dai.  BEDS-bench: Behavior of EHR-models under distributional shift - a benchmark.  In _NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications_ , 2021.
  * Bendale and Boult (2016) Abhijit Bendale and Terrance E Boult.  Towards open set deep networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016.
  * de Hond et al. (2023) Anne AH de Hond, Ilse MJ Kant, Mattia Fornasa, Giovanni Cinà, Paul WG Elbers, Patrick J Thoral, M Sesmu Arbous, and Ewout W Steyerberg.  Predicting readmission or death after discharge from the icu: external validation and retraining of a machine learning model.  _Critical Care Medicine_ , 51(2):291, 2023.
  * De Vries et al. (2010) Timothy De Vries, Sanjay Chawla, and Michael E Houle.  Finding local anomalies in very high dimensional space.  In _2010 IEEE International Conference on Data Mining_ , pages 128–137. IEEE, 2010.
  * Djurisic et al. (2023) Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu.  Extremely simple activation shaping for out-of-distribution detection.  In _International Conference on Learning Representations_ , 2023.
  * Durkan et al. (2020) Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios.  nflows: normalizing flows in PyTorch, November 2020.  URL https://doi.org/10.5281/zenodo.4296287.
  * Fort et al. (2021) Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan.  Exploring the limits of out-of-distribution detection.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Gorishniy et al. (2021) Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko.  Revisiting deep learning models for tabular data.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , 2021.
  * Greener et al. (2022) Joe G Greener, Shaun M Kandathil, Lewis Moffat, and David T Jones.  A guide to machine learning for biologists.  _Nature Reviews Molecular Cell Biology_ , 23(1):40–55, 2022.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In _International conference on machine learning_ , pages 1321–1330. PMLR, 2017.
  * Gupta et al. (2022) Mehak Gupta, Brennan Gallamoza, Nicolas Cutrona, Pranjal Dhakal, Raphael Poulain, and Rahmatollah Beheshti.  An Extensive Data Processing Pipeline for MIMIC-IV.  In _Proceedings of the 2nd Machine Learning for Health symposium_ , volume 193 of _Proceedings of Machine Learning Research_ , pages 311–325. PMLR, 28 Nov 2022.
  * Han et al. (2022) Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao.  Adbench: Anomaly detection benchmark.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Hein et al. (2019) Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf.  Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 41–50, 2019.
  * Hendrycks and Gimpel (2017) Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In _International Conference on Learning Representations_ , 2017.
  * Hendrycks et al. (2022) Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song.  Scaling out-of-distribution detection for real-world settings.  In _International Conference on Machine Learning_ , 2022.
  * Huang et al. (2021) Rui Huang, Andrew Geng, and Yixuan Li.  On the importance of gradients for detecting distributional shifts in the wild.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Johnson et al. (2023) Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al.  Mimic-iv, a freely accessible electronic health record dataset.  _Scientific data_ , 10(1):1, 2023.
  * Kingma and Welling (2014) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  In _International Conference on Learning Representations_ , 2014.
  * Kong and Ramanan (2021) Shu Kong and Deva Ramanan.  Opengan: Open-set recognition via open data generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 813–822, 2021.
  * Lee et al. (2018) Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  2018\.
  * Liang et al. (2018) Shiyu Liang, Yixuan Li, and R. Srikant.  Enhancing the reliability of out-of-distribution image detection in neural networks.  In _International Conference on Learning Representations_ , 2018.
  * Liu et al. (2020) Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li.  Energy-based out-of-distribution detection.  In _Advances in neural information processing systems_ , 2020.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2019.
  * Marimont and Tarroni (2021) Sergio Naval Marimont and Giacomo Tarroni.  Anomaly detection through latent space restoration using vector quantized variational autoencoders.  In _2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)_ , 2021.
  * Meijerink et al. (2020) Lotta Meijerink, Giovanni Cinà, and Michele Tonutti.  Uncertainty estimation for classification and risk prediction on medical tabular data.  _arXiv preprint arXiv:2004.05824_ , 2020.
  * Nazabal et al. (2020) Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera.  Handling incomplete heterogeneous data using vaes.  _Pattern Recognition_ , 107, 2020.
  * Nicora et al. (2022) Giovanna Nicora, Miguel Rios, Ameen Abu-Hanna, and Riccardo Bellazzi.  Evaluating pointwise reliability of machine learning prediction.  _Journal of Biomedical Informatics_ , 127:103996, 2022.
  * Papamakarios et al. (2017) George Papamakarios, Theo Pavlakou, and Iain Murray.  Masked autoregressive flow for density estimation.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2017.
  * Pollard et al. (2018) Tom J Pollard, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi.  The eicu collaborative research database, a freely available multi-center database for critical care research.  _Scientific data_ , 5(1):1–13, 2018.
  * Ren et al. (2021) Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan.  A simple fix to mahalanobis distance for improving near-ood detection.  _arXiv preprint arXiv:2106.09022_ , 2021.
  * Rios and Abu-Hanna (2021) Miguel Rios and Ameen Abu-Hanna.  Deep kernel learning for mortality prediction in the face of temporal shift.  In _Artificial Intelligence in Medicine_ , pages 199–208, Cham, 2021\. Springer International Publishing.
  * Sastry and Oore (2020) Chandramouli Shama Sastry and Sageev Oore.  Detecting out-of-distribution examples with gram matrices.  In _International Conference on Machine Learning_ , pages 8491–8501. PMLR, 2020.
  * Sheikhalishahi et al. (2020) Seyedmostafa Sheikhalishahi, Vevake Balaraman, and Venet Osmani.  Benchmarking machine learning models on multi-centre eicu critical care dataset.  _Plos one_ , 15(7):e0235424, 2020.
  * Spathis and Hyland (2022) Dimitris Spathis and Stephanie L Hyland.  Looking for out-of-distribution environments in critical care: A case study with the eicu database.  _arXiv preprint arXiv:2205.13398_ , 2022.
  * Sun and Li (2022) Yiyou Sun and Yixuan Li.  Dice: Leveraging sparsification for out-of-distribution detection.  In _European Conference on Computer Vision_ , 2022.
  * Sun et al. (2021) Yiyou Sun, Chuan Guo, and Yixuan Li.  React: Out-of-distribution detection with rectified activations.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Sun et al. (2022) Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li.  Out-of-distribution detection with deep nearest neighbors.  In _International Conference on Machine Learning_ , pages 20827–20840. PMLR, 2022.
  * Tan et al. (2022) Jeremy Tan, Benjamin Hou, James Batten, Huaqi Qiu, Bernhard Kainz, et al.  Detecting outliers with foreign patch interpolation.  _Machine Learning for Biomedical Imaging_ , 1(April 2022 issue):1–27, 2022.
  * Tipping and Bishop (1999) Michael E Tipping and Christopher M Bishop.  Probabilistic principal component analysis.  _Journal of the Royal Statistical Society Series B: Statistical Methodology_ , 61(3):611–622, 1999.
  * Ulmer and Cinà (2021) Dennis Ulmer and Giovanni Cinà.  Know your limits: Uncertainty estimation with relu classifiers fails at reliable ood detection.  In _Uncertainty in Artificial Intelligence_ , pages 1766–1776. PMLR, 2021.
  * Ulmer et al. (2020) Dennis Ulmer, Lotta Meijerink, and Giovanni Cinà.  Trust issues: Uncertainty estimation does not enable reliable ood detection on medical tabular data.  In _Proceedings of the Machine Learning for Health NeurIPS Workshop_ , volume 136 of _Proceedings of Machine Learning Research_ , pages 341–354. PMLR, 11 Dec 2020.
  * van Amersfoort et al. (2021) Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal.  Improving deterministic uncertainty estimation in deep learning for classification and regression.  _arXiv preprint arXiv:2102.11409_ , 2021.
  * Varoquaux and Cheplygina (2022) Gaël Varoquaux and Veronika Cheplygina.  Machine learning for medical imaging: methodological failures and recommendations for the future.  _NPJ digital medicine_ , 5(1):48, 2022.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , 2017.
  * Vaze et al. (2021) Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman.  Open-set recognition: A good closed-set classifier is all you need.  In _International Conference on Learning Representations_ , 2021.
  * Wang et al. (2022) Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang.  Vim: Out-of-distribution with virtual-logit matching.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022.
  * Winkens et al. (2020) Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al.  Contrastive training for improved out-of-distribution detection.  _arXiv preprint arXiv:2007.05566_ , 2020.
  * Yang et al. (2021) Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu.  Generalized out-of-distribution detection: A survey.  _arXiv preprint arXiv:2110.11334_ , 2021.
  * Yang et al. (2022) Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, WENXUAN PENG, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu.  OpenOOD: Benchmarking generalized out-of-distribution detection.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.
  * Zadorozhny et al. (2022) Karina Zadorozhny, Patrick Thoral, Paul Elbers, and Giovanni Cinà.  Out-of-distribution detection for medical applications: Guidelines for practical evaluation.  In _Multimodal AI in healthcare: A paradigm shift in health intelligence_ , pages 137–153. Springer, 2022.
  * Zhang et al. (2023a) Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li.  Openood v1.5: Enhanced benchmark for out-of-distribution detection.  _arXiv preprint arXiv:2306.09301_ , 2023a.
  * Zhang et al. (2023b) Jinsong Zhang, Qiang Fu, Xu Chen, Lun Du, Zelin Li, Gang Wang, xiaoguang Liu, Shi Han, and Dongmei Zhang.  Out-of-distribution detection based on in-distribution data patterns memorization with modern hopfield energy.  In _International Conference on Learning Representations_ , 2023b.
  * Zimmerer et al. (2022) David Zimmerer, Peter M Full, Fabian Isensee, Paul Jäger, Tim Adler, Jens Petersen, Gregor Köhler, Tobias Ross, Annika Reinke, Antanas Kascenas, et al.  Mood 2020: A public benchmark for out-of-distribution detection and localization on medical images.  _IEEE Transactions on Medical Imaging_ , 41(10):2728–2738, 2022.

##  Appendix A Clinical Variables

Clinical variables selected for each of the eICU and MIMIC-IV datasets are
provided in Table 4. These variables are mostly selected as suggested in prior
works (Ulmer et al., 2020; Spathis and Hyland, 2022; Rios and Abu-Hanna,
2021). Additionally, we have excluded some time-dependent variables in MIMIC-
IV since they are not recorded for most patients. Adding these variables would
result in a lot of NaN values. Note that when a dataset is considered as an
OOD set for the other one, only available variables for both datasets are
included in that experiment.

eICU | MIMIC-IV
---|---
• Time-dependent
pH | pH
Temperature (c) | Temperature
Respiratory Rate | Respiratory rate
O2 Saturation | Oxygen saturation
MAP (mmHg) | Mean blood pressure
Heart Rate | Heart Rate
glucose | Glucose
GCS Total | -
Motor | -
Eyes | -
Verbal | -
FiO2 | -
Invasive BP Diastolic | Diastolic blood pressure
Invasive BP Systolic | Systolic blood pressure
• Time-independent
gender | gender
age | age
ethnicity | -
admissionheight | -
admissionweight | -
- | admission_type
- | first_careunit
Table 4: Clinical time-dependent and time-independent variables used for each
of eICU and MIMIC-IV datasets. Corresponding variables are put in the same
row, with a dash indicating that the variable is not included in a dataset.

##  Appendix B Mortality Prediction

Post-hoc OOD detectors are utilized alongside a prediction model. It is
essential that this prediction model is trained well on the ID data (Vaze et
al., 2021). Otherwise, the detector would not be able to distinguish IDs and
OODs.

Following the guidelines in Gorishniy et al. (2021), we have trained the
prediction models for $10$ epochs with a batch size equal to 64 and AdamW
(Loshchilov and Hutter, 2019) optimizer with a learning rate equal to
$1e\text{-}3$.

To ensure that our prediction models are trained well, we have measured the
AUROC of the mortality prediction task on the test set. Results are presented
in Table 5, indicating that models are able to solve the task well. Also, it
demonstrates that transformer architecture achieves better performance on this
task than MLP and ResNet across both datasets.

Additionally, according to the results in Table 2 and 3, MLP and ResNet in
conjunction with distance-based post-hoc detectors can outperform transformers
in many cases. This suggests that superior performance as a closed-set
classifier on ID data does not necessarily translate to better OOD detection,
challenging the claim in Vaze et al. (2021) that better closed-set classifiers
lead to improved OOD detection performance.

| eICU | MIMIC-IV
---|---|---
MLP | 85.7$\pm$1.6 | 75.6$\pm$1.0
ResNet | 87.9$\pm$0.5 | 77.3$\pm$1.2
FT-Transformer | 89.1$\pm$0.4 | 80.1$\pm$0.3
Table 5: AUROC of mortality prediction task for MLP, ResNet, and Ft-
Transformer architectures on eICU and MIMIC-IV. Results are averaged over 5
runs.

##  Appendix C Other Near-OOD Results

Near-OOD results based on age (older than 70 as ID), ethnicity (Caucasian as
ID), and first care unit (CVICU as ID) are provided in Table 2. To explore
this with other different variables, results for alternate variables are
presented in table 6. The variables used in this table are gender (females as
ID) and ethnicity (African Americans as ID) for eICU, and gender (females as
ID) and admission type (surgical on the same day of admission as ID) for
MIMIC-IV.

In eICU, results for both variables indicate an almost random AUC (50%) for
all the methods. In MIMIC-IV, gender shows a similar behavior, while for
admission type, certain approaches like MDS and AE exhibit marginal
superiority over the rest. These results align well with the observations from
previous variables in the table 2. First of all, the wide-ranging diversity
present in the eICU dataset poses challenges in identifying nearly out-of-
distribution sets. Additionally, in scenarios where the difficulty level is
not excessively high to the point of random performance, methods like AE and
MDS emerge as competitive options. Lastly, the issue of detecting near-OOD
samples is still challenging.

Model | Method | eICU | MIMIC-IV
---|---|---|---
Near-OOD(Gender) | Near-OOD(Ethnicity) | Near-OOD(Gender) | Near-OOD(Admission)
AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95
| AE | 51.0$\pm$0.1 | 95.0$\pm$0.2 | 47.4$\pm$0.4 | 95.9$\pm$0.7 | 50.2$\pm$0.0 | 94.4$\pm$0.1 | 63.1$\pm$1.3 | 94.0$\pm$0.1
| VAE | 50.7$\pm$0.2 | 95.0$\pm$0.2 | 47.3$\pm$0.4 | 95.9$\pm$0.7 | 50.2$\pm$0.0 | 94.4$\pm$0.1 | 63.2$\pm$1.2 | 93.9$\pm$0.1
| HI-VAE | 42.9$\pm$ 0.4 | 100.0$\pm$0.0 | 46.9$\pm$1.6 | 97.2$\pm$1.5 | 50.2$\pm$1.1 | 94.8$\pm$0.4 | 59.0$\pm$18.0 | 82.1$\pm$12.8
Density | Flow | 48.6$\pm$0.3 | 96.3$\pm$0.6 | 48.3$\pm$0.4 | 95.9$\pm$0.1 | 48.2$\pm$0.4 | 95.1$\pm$0.5 | 44.0$\pm$4.2 | 93.9$\pm$0.2
Based | PPCA | 51.2$\pm$0.2 | 95.1$\pm$0.1 | 48.1$\pm$0.4 | 95.9$\pm$0.2 | 50.4$\pm$0.2 | 94.4$\pm$0.0 | 61.4$\pm$0.5 | 92.1$\pm$1.2
| LOF | 53.1$\pm$0.3 | 94.2$\pm$0.1 | 50.2$\pm$0.6 | 94.4$\pm$0.1 | 50.6$\pm$0.0 | 94.1$\pm$0.3 | 60.5$\pm$0.5 | 92.1$\pm$0.8
| DUE | 50.9$\pm$0.6 | 94.5$\pm$0.3 | 48.7$\pm$0.3 | 95.0$\pm$0.2 | 49.9$\pm$0.1 | 95.4$\pm$0.5 | 57.1$\pm$1.0 | 93.7$\pm$1.5
MLP | MDS | 48.8$\pm$0.1 | 94.9$\pm$0.0 | 45.4$\pm$0.2 | 95.4$\pm$0.4 | 51.5$\pm$0.7 | 94.4$\pm$0.2 | 60.8$\pm$0.2 | 91.5$\pm$1.0
RMDS | 50.9$\pm$0.3 | 94.6$\pm$0.3 | 50.5$\pm$0.8 | 94.8$\pm$0.3 | 48.8$\pm$0.8 | 95.3$\pm$0.6 | 56.1$\pm$1.4 | 91.1$\pm$1.8
KNN | 52.1$\pm$0.4 | 94.4$\pm$0.3 | 54.5$\pm$0.6 | 94.5$\pm$0.3 | 48.3$\pm$1.0 | 95.0$\pm$0.1 | 61.6$\pm$0.6 | 93.5$\pm$1.2
VIM | 52.2$\pm$0.2 | 94.5$\pm$0.1 | 54.9$\pm$0.7 | 95.2$\pm$0.4 | 48.0$\pm$0.7 | 94.9$\pm$0.2 | 60.5$\pm$0.4 | 91.3$\pm$1.1
SHE | 49.0$\pm$0.1 | 95.9$\pm$0.3 | 49.8$\pm$0.8 | 97.3$\pm$0.1 | 50.8$\pm$0.2 | 93.5$\pm$0.8 | 54.0$\pm$1.0 | 92.9$\pm$0.9
KLM | 48.8$\pm$0.1 | 94.7$\pm$0.2 | 45.1$\pm$0.6 | 94.1$\pm$0.2 | 51.2$\pm$0.2 | 95.4$\pm$0.1 | 48.5$\pm$0.5 | 91.9$\pm$1.3
OpenMax | 52.3$\pm$0.2 | 95.0$\pm$0.1 | 56.5$\pm$0.7 | 94.3$\pm$0.1 | 47.1$\pm$0.7 | 95.5$\pm$0.1 | 57.7$\pm$1.3 | 91.2$\pm$0.6
MSP | 52.2$\pm$0.2 | 94.7$\pm$0.2 | 56.8$\pm$0.7 | 94.0$\pm$0.0 | 47.1$\pm$0.8 | 95.5$\pm$0.3 | 57.2$\pm$0.9 | 92.3$\pm$1.9
MLP | MLS | 52.3$\pm$0.2 | 94.6$\pm$0.1 | 56.9$\pm$0.7 | 94.0$\pm$0.1 | 47.0$\pm$0.8 | 95.2$\pm$0.1 | 57.0$\pm$1.0 | 91.9$\pm$2.1
| TempScale | 52.2$\pm$0.2 | 94.7$\pm$0.2 | 56.9$\pm$0.7 | 94.0$\pm$0.0 | 47.1$\pm$0.8 | 95.5$\pm$0.3 | 57.0$\pm$0.9 | 92.3$\pm$1.9
| ODIN | 52.2$\pm$0.2 | 94.7$\pm$0.1 | 56.8$\pm$0.7 | 94.0$\pm$0.1 | 47.1$\pm$0.8 | 95.3$\pm$0.2 | 57.0$\pm$1.0 | 92.3$\pm$1.8
| EBO | 52.3$\pm$0.3 | 94.9$\pm$0.1 | 56.9$\pm$0.7 | 94.1$\pm$0.1 | 47.0$\pm$0.8 | 95.3$\pm$0.0 | 57.0$\pm$1.0 | 91.8$\pm$2.1
| GRAM | 51.6$\pm$0.6 | 100.0$\pm$0.0 | 55.0$\pm$1.8 | 100.0$\pm$0.0 | 50.0$\pm$0.1 | 100.0$\pm$0.0 | 49.5$\pm$0.8 | 100.0$\pm$0.0
| GradNorm | 52.2$\pm$0.2 | 94.5$\pm$0.2 | 56.8$\pm$0.7 | 93.8$\pm$0.2 | 47.1$\pm$0.7 | 95.4$\pm$0.3 | 57.0$\pm$0.9 | 90.8$\pm$1.1
| ReAct | 51.9$\pm$0.3 | 94.9$\pm$0.1 | 56.8$\pm$0.7 | 94.1$\pm$0.1 | 47.2$\pm$0.8 | 95.3$\pm$0.0 | 57.0$\pm$1.2 | 91.8$\pm$2.1
| DICE | 52.3$\pm$0.2 | 94.2$\pm$0.1 | 56.7$\pm$0.8 | 94.6$\pm$0.5 | 47.1$\pm$0.5 | 95.5$\pm$0.3 | 57.0$\pm$1.0 | 90.3$\pm$1.0
| ASH | 52.1$\pm$0.3 | 94.8$\pm$0.1 | 56.9$\pm$0.4 | 94.0$\pm$0.2 | 47.3$\pm$1.0 | 95.2$\pm$0.1 | 56.7$\pm$2.2 | 90.6$\pm$1.0
ResNet | MDS | 50.4$\pm$0.4 | 95.2$\pm$0.2 | 47.8$\pm$0.2 | 96.2$\pm$0.1 | 50.6$\pm$0.1 | 94.2$\pm$0.4 | 62.2$\pm$0.3 | 92.2$\pm$1.0
RMDS | 50.5$\pm$0.5 | 94.8$\pm$0.2 | 50.7$\pm$0.6 | 94.9$\pm$0.3 | 50.2$\pm$0.1 | 94.2$\pm$0.1 | 55.1$\pm$0.6 | 91.9$\pm$0.9
KNN | 53.1$\pm$0.2 | 94.4$\pm$0.1 | 51.3$\pm$0.6 | 95.2$\pm$0.3 | 47.9$\pm$0.8 | 94.6$\pm$0.3 | 55.7$\pm$1.2 | 93.9$\pm$0.9
VIM | 53.5$\pm$0.2 | 94.7$\pm$0.2 | 51.8$\pm$0.8 | 95.6$\pm$0.2 | 47.4$\pm$0.7 | 95.0$\pm$0.1 | 59.9$\pm$0.2 | 92.0$\pm$1.2
SHE | 47.6$\pm$0.5 | 95.7$\pm$0.2 | 45.6$\pm$0.6 | 96.5$\pm$0.5 | 51.7$\pm$0.1 | 94.2$\pm$0.1 | 61.6$\pm$1.0 | 92.6$\pm$1.4
KLM | 48.2$\pm$0.0 | 94.1$\pm$0.1 | 47.3$\pm$1.0 | 95.5$\pm$0.5 | 50.6$\pm$0.6 | 95.6$\pm$0.5 | 53.7$\pm$0.1 | 91.4$\pm$1.1
OpenMax | 53.7$\pm$0.5 | 94.3$\pm$0.1 | 53.4$\pm$1.6 | 95.0$\pm$0.5 | 46.9$\pm$0.6 | 95.8$\pm$0.3 | 55.2$\pm$0.3 | 92.4$\pm$0.9
MSP | 53.7$\pm$0.5 | 93.8$\pm$0.2 | 53.4$\pm$1.6 | 95.0$\pm$0.5 | 46.9$\pm$0.7 | 95.9$\pm$0.2 | 55.2$\pm$0.7 | 91.9$\pm$1.1
ResNet | MLS | 53.7$\pm$0.4 | 94.1$\pm$0.3 | 53.6$\pm$1.3 | 95.0$\pm$0.6 | 46.8$\pm$0.6 | 96.0$\pm$0.3 | 55.2$\pm$0.4 | 91.3$\pm$1.1
| TempScale | 53.7$\pm$0.5 | 93.8$\pm$0.2 | 53.4$\pm$1.6 | 95.0$\pm$0.5 | 46.9$\pm$0.7 | 95.9$\pm$0.2 | 55.2$\pm$0.7 | 91.9$\pm$1.1
| ODIN | 53.7$\pm$0.5 | 93.8$\pm$0.2 | 53.4$\pm$1.5 | 95.0$\pm$0.4 | 46.9$\pm$0.7 | 95.8$\pm$0.2 | 55.2$\pm$0.7 | 91.9$\pm$1.1
| EBO | 53.7$\pm$0.4 | 94.3$\pm$0.1 | 53.6$\pm$1.3 | 94.9$\pm$0.7 | 46.8$\pm$0.5 | 96.1$\pm$0.3 | 55.1$\pm$0.4 | 91.5$\pm$0.9
| GRAM | 52.5$\pm$0.2 | 100.0$\pm$0.0 | 55.6$\pm$1.1 | 96.3$\pm$2.4 | 50.0$\pm$0.0 | 100.0$\pm$0.0 | 42.6$\pm$2.7 | 100.0$\pm$0.0
| GradNorm | 53.8$\pm$0.7 | 93.8$\pm$0.2 | 55.0$\pm$1.2 | 94.5$\pm$0.5 | 47.0$\pm$0.7 | 95.8$\pm$0.2 | 46.6$\pm$1.7 | 91.4$\pm$0.9
| ReAct | 53.3$\pm$0.3 | 94.3$\pm$0.1 | 52.6$\pm$1.4 | 95.4$\pm$0.6 | 47.0$\pm$0.8 | 95.9$\pm$0.2 | 58.0$\pm$0.3 | 92.0$\pm$1.1
| DICE | 52.9$\pm$0.2 | 94.2$\pm$0.4 | 52.6$\pm$1.9 | 94.8$\pm$0.3 | 47.3$\pm$0.6 | 95.2$\pm$0.4 | 51.8$\pm$1.4 | 92.9$\pm$0.2
| ASH | 53.5$\pm$0.6 | 94.3$\pm$0.4 | 52.8$\pm$1.2 | 95.1$\pm$0.6 | 46.8$\pm$0.5 | 96.1$\pm$0.4 | 53.9$\pm$0.5 | 91.8$\pm$1.5
FT-T | MDS | 51.2$\pm$0.2 | 95.2$\pm$0.3 | 52.0$\pm$0.7 | 95.7$\pm$0.1 | 51.6$\pm$0.4 | 94.2$\pm$0.1 | 61.9$\pm$0.7 | 90.1$\pm$1.1
RMDS | 51.6$\pm$0.8 | 94.0$\pm$0.1 | 50.8$\pm$0.5 | 94.5$\pm$1.0 | 49.2$\pm$0.2 | 95.6$\pm$0.2 | 53.8$\pm$1.0 | 90.3$\pm$1.5
KNN | 52.7$\pm$0.3 | 95.4$\pm$0.2 | 53.1$\pm$0.3 | 95.3$\pm$0.4 | 48.8$\pm$0.2 | 95.0$\pm$0.5 | 63.3$\pm$1.1 | 92.4$\pm$1.6
VIM | 53.7$\pm$0.3 | 95.1$\pm$0.3 | 53.7$\pm$0.1 | 95.1$\pm$0.2 | 47.3$\pm$0.2 | 95.4$\pm$0.2 | 62.5$\pm$0.3 | 89.6$\pm$1.1
SHE | 49.5$\pm$0.2 | 94.4$\pm$0.4 | 50.5$\pm$0.8 | 94.6$\pm$0.2 | 51.3$\pm$0.2 | 95.2$\pm$0.4 | 55.4$\pm$0.6 | 90.5$\pm$1.4
KLM | 47.6$\pm$0.6 | 93.9$\pm$0.3 | 46.0$\pm$0.9 | 94.1$\pm$0.3 | 52.1$\pm$0.5 | 95.5$\pm$0.5 | 46.1$\pm$1.0 | 91.2$\pm$1.0
OpenMax | 55.1$\pm$0.5 | 93.8$\pm$0.1 | 55.4$\pm$0.1 | 94.3$\pm$0.3 | 46.3$\pm$0.4 | 95.5$\pm$0.1 | 62.5$\pm$0.1 | 90.2$\pm$1.0
MSP | 55.1$\pm$0.5 | 93.8$\pm$0.1 | 55.4$\pm$0.1 | 94.3$\pm$0.3 | 46.4$\pm$0.4 | 95.4$\pm$0.3 | 62.6$\pm$0.1 | 90.2$\pm$1.0
FT-T | MLS | 55.0$\pm$0.5 | 94.0$\pm$0.1 | 55.4$\pm$0.1 | 94.2$\pm$0.4 | 46.4$\pm$0.3 | 95.6$\pm$0.1 | 62.2$\pm$0.1 | 90.4$\pm$0.9
| TempScale | 55.1$\pm$0.5 | 93.8$\pm$0.1 | 55.4$\pm$0.1 | 94.3$\pm$0.3 | 46.4$\pm$0.4 | 95.4$\pm$0.3 | 62.6$\pm$0.1 | 90.2$\pm$1.0
| ODIN | 55.1$\pm$0.5 | 93.8$\pm$0.1 | 55.4$\pm$0.1 | 94.3$\pm$0.3 | 46.4$\pm$0.4 | 95.4$\pm$0.3 | 62.6$\pm$0.1 | 90.3$\pm$1.0
| EBO | 55.0$\pm$0.5 | 94.0$\pm$0.1 | 55.4$\pm$0.1 | 94.6$\pm$0.3 | 46.4$\pm$0.3 | 95.6$\pm$0.3 | 62.2$\pm$0.2 | 90.9$\pm$1.2
| GRAM | 54.3$\pm$1.2 | 94.4$\pm$0.2 | 54.7$\pm$0.8 | 94.6$\pm$0.2 | 46.6$\pm$0.9 | 95.7$\pm$0.3 | 55.8$\pm$5.4 | 89.9$\pm$1.5
| GradNorm | 53.5$\pm$0.1 | 93.8$\pm$0.1 | 51.1$\pm$4.3 | 94.6$\pm$0.4 | 47.2$\pm$1.3 | 95.4$\pm$0.3 | 49.3$\pm$5.8 | 90.5$\pm$1.1
| ReAct | 54.9$\pm$0.4 | 94.0$\pm$0.1 | 55.6$\pm$0.1 | 94.3$\pm$0.3 | 46.4$\pm$0.4 | 95.4$\pm$0.4 | 61.7$\pm$0.8 | 90.9$\pm$1.4
| DICE | 54.9$\pm$0.5 | 94.3$\pm$0.5 | 55.3$\pm$0.1 | 94.4$\pm$0.2 | 46.6$\pm$0.5 | 95.5$\pm$0.2 | 61.6$\pm$0.2 | 90.4$\pm$0.9
| ASH | 54.7$\pm$0.6 | 94.3$\pm$0.2 | 51.4$\pm$1.6 | 95.1$\pm$0.5 | 46.7$\pm$0.2 | 95.4$\pm$0.3 | 54.8$\pm$2.2 | 90.5$\pm$1.4
Table 6:  Comparing OOD detection methods in the near-OOD setting, with
performance evaluated through AUROC and FPR@95. In eICU, distinguishing
variables are gender (females as ID) and ethnicity (African Americans as ID).
In MIMIC-IV, distinguishing variables are gender (females as ID) and admission
type (surgical on the same day of admission as ID). Results are averaged over
3 runs. For each model, the best value in each column is shown in bold, and
distance-based post-hoc approaches are distinguished with a lighter color.

##  Appendix D FPR@95 Results

In Tables 2 and 3, we presented the results of our analyses, highlighting the
performance based on the AUROC. While AUROC serves as a measure of the
discriminative capability between novelty scores assigned to ID and OOD data,
devoid of any reliance on a predetermined threshold, it would be beneficial to
complement this assessment by evaluating the performance at a specific
threshold. For this purpose, we have measured the false positive rate at the
$\lambda$ corresponding to the TPR equal to 95% as an acceptable rate for TPR,
which is known as FPR@95.

By replacing AUROC with FPR@95 in Tables 2 and 3, we have replicated the
corresponding outcomes within Tables 7 and 8. These new tables are in
agreement with our conclusions, and if one method has a relatively high AUROC,
it would mostly have a relatively low FPR@95. However, it is helpful for
comparing methods at a specific threshold. For example, RMDS and KLM applied
to an MLP model are achieving similar AUROCs, but RMDS is definitely better at
the measured threshold.

Model | Method | eICU | MIMIC-IV
---|---|---|---
Far-OOD | Near-OOD (Eth) | Near-OOD (Age) | Far-OOD | Near-OOD (FCU) | Near-OOD (Age)
| AE | 17.5$\pm$5.3 | 93.3$\pm$1.6 | 94.3$\pm$0.2 | 0.2$\pm$0.3 | 72.7$\pm$11.5 | 90.7$\pm$0.7
| VAE | 22.5$\pm$7.4 | 93.3$\pm$1.5 | 94.6$\pm$0.2 | 0.2$\pm$0.3 | 71.9$\pm$11.8 | 90.7$\pm$0.8
| HI-VAE | 83.4$\pm$1.5 | 100.0$\pm$0.0 | 100.0$\pm$0.0 | 99.9$\pm$0.1 | 44.5$\pm$21.9 | 90.9$\pm$6.9
Density | Flow | 0.0$\pm$0.0 | 89.6$\pm$4.9 | 94.7$\pm$0.4 | 35.9$\pm$39.2 | 96.8$\pm$1.1 | 93.5$\pm$1.0
Based | PPCA | 15.5$\pm$4.3 | 91.5$\pm$3.5 | 93.8$\pm$0.3 | 0.3$\pm$0.3 | 76.7$\pm$9.8 | 92.0$\pm$0.6
| LOF | 17.4$\pm$2.3 | 92.8$\pm$2.4 | 95.5$\pm$0.2 | 3.5$\pm$0.6 | 78.4$\pm$5.4 | 91.9$\pm$0.4
| DUE | 80.9$\pm$6.1 | 94.3$\pm$1.6 | 94.5$\pm$0.1 | 7.6$\pm$2.5 | 88.6$\pm$3.2 | 91.1$\pm$0.4
MLP | MDS | 62.7$\pm$6.6 | 91.1$\pm$2.6 | 94.6$\pm$0.3 | 4.3$\pm$3.2 | 85.2$\pm$2.2 | 92.5$\pm$0.6
RMDS | 90.3$\pm$5.0 | 94.7$\pm$1.5 | 95.7$\pm$0.2 | 60.0$\pm$29.2 | 89.8$\pm$5.3 | 95.0$\pm$0.1
KNN | 73.3$\pm$13.1 | 93.5$\pm$3.7 | 95.9$\pm$0.2 | 81.9$\pm$18.8 | 78.8$\pm$3.8 | 93.4$\pm$0.6
VIM | 84.9$\pm$7.3 | 94.2$\pm$1.8 | 95.4$\pm$0.5 | 9.5$\pm$6.4 | 82.3$\pm$1.1 | 93.0$\pm$0.8
SHE | 87.8$\pm$1.1 | 94.4$\pm$1.4 | 94.0$\pm$0.8 | 26.6$\pm$15.7 | 93.3$\pm$1.5 | 96.1$\pm$0.4
KLM | 93.3$\pm$1.1 | 94.7$\pm$1.5 | 95.8$\pm$0.2 | 82.2$\pm$16.5 | 90.8$\pm$5.1 | 94.2$\pm$0.5
OpenMax | 93.8$\pm$3.9 | 95.9$\pm$1.4 | 95.9$\pm$0.1 | 70.4$\pm$24.4 | 87.0$\pm$2.1 | 94.7$\pm$0.9
MSP | 94.7$\pm$3.4 | 97.2$\pm$2.0 | 95.9$\pm$0.2 | 99.1$\pm$1.0 | 91.6$\pm$6.5 | 94.7$\pm$0.7
MLP | MLS | 94.6$\pm$3.6 | 95.4$\pm$1.3 | 96.0$\pm$0.2 | 98.3$\pm$1.6 | 87.8$\pm$3.9 | 94.5$\pm$0.6
| TempScale | 94.7$\pm$3.4 | 95.9$\pm$1.1 | 95.9$\pm$0.2 | 99.1$\pm$1.0 | 89.1$\pm$6.4 | 94.7$\pm$0.7
| ODIN | 94.7$\pm$3.4 | 95.4$\pm$1.3 | 95.9$\pm$0.3 | 98.9$\pm$1.1 | 88.0$\pm$4.0 | 94.6$\pm$0.8
| EBO | 94.5$\pm$3.7 | 95.4$\pm$1.3 | 96.0$\pm$0.2 | 96.8$\pm$3.1 | 87.8$\pm$3.9 | 94.8$\pm$0.6
| GRAM | 99.4$\pm$0.3 | 98.1$\pm$3.0 | 100.0$\pm$0.0 | 100.0$\pm$0.0 | 98.7$\pm$2.1 | 100.0$\pm$0.0
| GradNorm | 95.3$\pm$2.7 | 95.5$\pm$1.2 | 95.8$\pm$0.2 | 99.1$\pm$0.8 | 88.4$\pm$3.9 | 94.6$\pm$0.6
| ReAct | 93.6$\pm$2.4 | 95.4$\pm$1.0 | 96.0$\pm$0.2 | 72.5$\pm$17.4 | 87.7$\pm$3.8 | 94.8$\pm$0.6
| DICE | 95.8$\pm$2.5 | 95.3$\pm$0.9 | 95.0$\pm$0.2 | 97.8$\pm$3.2 | 88.2$\pm$3.8 | 94.2$\pm$1.1
| ASH | 94.3$\pm$3.7 | 95.9$\pm$0.9 | 95.8$\pm$0.4 | 97.2$\pm$2.7 | 88.7$\pm$2.7 | 94.7$\pm$0.3
ResNet | MDS | 14.0$\pm$1.9 | 90.5$\pm$3.3 | 94.0$\pm$0.2 | 0.4$\pm$0.4 | 77.4$\pm$9.2 | 92.1$\pm$0.9
RMDS | 93.0$\pm$4.1 | 95.0$\pm$1.8 | 95.1$\pm$0.1 | 54.9$\pm$32.1 | 88.2$\pm$8.1 | 94.7$\pm$0.4
KNN | 39.4$\pm$7.0 | 88.3$\pm$3.3 | 95.3$\pm$0.1 | 32.9$\pm$17.0 | 93.0$\pm$2.2 | 93.4$\pm$0.6
VIM | 28.7$\pm$2.2 | 89.8$\pm$2.0 | 94.5$\pm$0.2 | 1.1$\pm$1.1 | 81.3$\pm$2.2 | 91.8$\pm$0.8
SHE | 54.2$\pm$4.7 | 94.8$\pm$0.6 | 94.2$\pm$0.2 | 0.5$\pm$0.6 | 79.2$\pm$7.2 | 93.3$\pm$0.4
KLM | 90.9$\pm$3.2 | 95.0$\pm$0.7 | 95.1$\pm$0.3 | 88.5$\pm$8.6 | 89.0$\pm$5.7 | 93.9$\pm$0.1
OpenMax | 85.2$\pm$4.6 | 95.8$\pm$1.5 | 95.7$\pm$0.6 | 62.6$\pm$31.8 | 88.8$\pm$6.4 | 93.6$\pm$0.2
MSP | 89.3$\pm$2.9 | 94.5$\pm$0.9 | 95.5$\pm$0.2 | 97.1$\pm$3.5 | 89.6$\pm$7.1 | 94.7$\pm$0.3
ResNet | MLS | 86.2$\pm$4.4 | 94.3$\pm$1.2 | 95.7$\pm$0.5 | 88.3$\pm$8.6 | 88.5$\pm$7.4 | 94.2$\pm$0.1
| TempScale | 89.3$\pm$2.9 | 94.5$\pm$0.9 | 95.5$\pm$0.2 | 97.1$\pm$3.5 | 89.6$\pm$7.1 | 94.7$\pm$0.3
| ODIN | 89.3$\pm$2.9 | 94.5$\pm$0.9 | 95.5$\pm$0.2 | 97.0$\pm$3.5 | 89.6$\pm$7.0 | 94.8$\pm$0.3
| EBO | 84.8$\pm$5.5 | 94.3$\pm$1.2 | 95.7$\pm$0.8 | 86.9$\pm$10.1 | 87.7$\pm$8.9 | 93.3$\pm$0.6
| GRAM | 99.8$\pm$0.3 | 95.2$\pm$1.2 | 100.0$\pm$0.0 | 100.0$\pm$0.0 | 99.5$\pm$0.4 | 100.0$\pm$0.0
| GradNorm | 94.7$\pm$2.4 | 94.7$\pm$1.7 | 95.6$\pm$0.2 | 99.1$\pm$1.1 | 91.5$\pm$6.1 | 94.8$\pm$0.2
| ReAct | 79.7$\pm$6.8 | 94.0$\pm$1.4 | 95.5$\pm$0.6 | 60.1$\pm$32.3 | 81.2$\pm$4.8 | 93.0$\pm$0.5
| DICE | 95.2$\pm$1.4 | 95.9$\pm$0.9 | 95.4$\pm$0.2 | 98.6$\pm$2.6 | 90.9$\pm$6.2 | 93.3$\pm$0.6
| ASH | 81.2$\pm$5.9 | 93.3$\pm$2.8 | 95.7$\pm$0.6 | 86.5$\pm$17.5 | 88.9$\pm$7.0 | 93.4$\pm$0.7
FT-T | MDS | 61.1$\pm$14.6 | 89.7$\pm$4.3 | 94.3$\pm$0.5 | 29.5$\pm$11.5 | 73.8$\pm$7.9 | 94.2$\pm$0.9
RMDS | 91.1$\pm$4.9 | 93.1$\pm$1.6 | 95.6$\pm$0.1 | 64.4$\pm$25.9 | 88.5$\pm$6.8 | 94.4$\pm$0.3
KNN | 53.2$\pm$13.4 | 91.8$\pm$2.0 | 94.7$\pm$0.2 | 38.4$\pm$22.3 | 70.7$\pm$10.5 | 93.9$\pm$0.1
VIM | 57.6$\pm$11.5 | 90.7$\pm$4.3 | 94.6$\pm$0.0 | 18.1$\pm$2.1 | 76.1$\pm$1.6 | 92.5$\pm$0.6
SHE | 84.7$\pm$2.6 | 93.2$\pm$1.8 | 95.1$\pm$0.3 | 52.5$\pm$17.1 | 83.8$\pm$5.9 | 96.7$\pm$0.2
KLM | 89.5$\pm$5.1 | 92.8$\pm$1.4 | 95.2$\pm$0.3 | 70.4$\pm$8.8 | 86.8$\pm$9.0 | 95.7$\pm$0.5
OpenMax | 80.4$\pm$1.9 | 93.8$\pm$1.9 | 95.2$\pm$0.1 | 78.6$\pm$17.3 | 78.3$\pm$3.7 | 94.3$\pm$0.3
MSP | 80.4$\pm$2.0 | 93.5$\pm$1.6 | 95.3$\pm$0.2 | 80.2$\pm$15.2 | 77.8$\pm$4.4 | 95.0$\pm$0.2
FT-T | MLS | 80.7$\pm$2.0 | 93.4$\pm$1.6 | 95.3$\pm$0.1 | 82.5$\pm$10.9 | 77.3$\pm$3.9 | 95.1$\pm$0.2
| TempScale | 80.4$\pm$2.0 | 93.5$\pm$1.6 | 95.3$\pm$0.2 | 80.2$\pm$15.2 | 77.8$\pm$4.4 | 95.0$\pm$0.2
| ODIN | 80.3$\pm$2.0 | 93.5$\pm$1.6 | 95.3$\pm$0.2 | 80.1$\pm$15.2 | 77.8$\pm$4.2 | 95.0$\pm$0.2
| EBO | 80.9$\pm$2.1 | 93.3$\pm$1.6 | 95.2$\pm$0.1 | 83.1$\pm$9.5 | 78.1$\pm$3.2 | 94.6$\pm$0.3
| GRAM | 87.7$\pm$2.7 | 93.6$\pm$2.3 | 95.0$\pm$0.3 | 67.6$\pm$18.8 | 77.6$\pm$7.8 | 94.5$\pm$0.3
| GradNorm | 87.2$\pm$2.1 | 92.0$\pm$3.2 | 95.3$\pm$0.2 | 78.1$\pm$14.4 | 86.0$\pm$6.5 | 95.0$\pm$0.2
| ReAct | 81.0$\pm$2.0 | 94.5$\pm$2.3 | 95.2$\pm$0.2 | 82.4$\pm$9.2 | 80.1$\pm$1.3 | 94.2$\pm$0.3
| DICE | 80.1$\pm$2.1 | 93.0$\pm$2.2 | 94.9$\pm$0.1 | 57.8$\pm$7.8 | 80.2$\pm$4.4 | 92.4$\pm$0.8
| ASH | 81.2$\pm$2.6 | 92.6$\pm$2.1 | 95.3$\pm$0.1 | 92.1$\pm$8.0 | 80.5$\pm$5.5 | 95.3$\pm$0.6
Table 7:  Comparing OOD detection methods in the far-OOD and near-OOD settings using eICU and MIMIC-IV datasets, with performance evaluated through FPR@95. This Table is similar to Table 2, but assesses FPR@95 instead of utilizing AUROC. FCU and Eth stand for first care unit and ethnicity as the variables utilized for constructing the ID/OOD sets. Results are averaged over 5 runs. For each model, the best value in each column is shown in bold, and distance-based post-hoc approaches are distinguished with a lighter color.  Model | Method | eICU | MIMIC-IV
---|---|---|---
$\mathcal{F}$=10 |  $\mathcal{F}$=100 |  $\mathcal{F}$=1000 |  $\mathcal{F}$=10 |  $\mathcal{F}$=100 |  $\mathcal{F}$=1000
| AE | 47.1$\pm$3.9 | 25.4$\pm$3.7 | 22.0$\pm$4.1 | 51.6$\pm$3.8 | 35.3$\pm$3.2 | 28.5$\pm$4.2
| VAE | 47.9$\pm$4.0 | 25.6$\pm$3.7 | 22.1$\pm$4.2 | 51.5$\pm$3.7 | 35.1$\pm$3.1 | 28.4$\pm$4.1
| HI-VAE | 94.9$\pm$0.2 | 94.7$\pm$0.2 | 94.7$\pm$0.5 | 94.9$\pm$0.4 | 94.5$\pm$0.6 | 94.1$\pm$0.9
Density | Flow | 60.2$\pm$7.2 | 36.8$\pm$7.4 | 26.5$\pm$5.2 | 82.1$\pm$3.1 | 62.6$\pm$3.3 | 44.6$\pm$2.1
Based | PPCA | 46.6$\pm$2.7 | 25.8$\pm$4.4 | 22.8$\pm$5.0 | 50.7$\pm$3.3 | 35.0$\pm$3.1 | 28.6$\pm$4.2
| LOF | 35.4$\pm$3.6 | 23.9$\pm$4.3 | 21.5$\pm$4.4 | 46.4$\pm$3.2 | 32.6$\pm$3.7 | 27.5$\pm$3.5
| DUE | 78.2$\pm$4.6 | 47.9$\pm$2.5 | 25.8$\pm$3.9 | 83.1$\pm$2.5 | 53.0$\pm$4.7 | 37.2$\pm$2.7
MLP | MDS | 67.9$\pm$5.4 | 42.3$\pm$3.2 | 24.5$\pm$4.0 | 67.1$\pm$4.7 | 43.0$\pm$5.1 | 32.6$\pm$3.6
RMDS | 77.5$\pm$0.9 | 50.6$\pm$2.6 | 30.7$\pm$4.0 | 87.9$\pm$2.0 | 66.2$\pm$2.4 | 45.4$\pm$2.1
KNN | 82.8$\pm$8.4 | 73.1$\pm$12.8 | 67.9$\pm$15.8 | 79.9$\pm$6.1 | 61.7$\pm$14.6 | 53.0$\pm$18.5
VIM | 86.4$\pm$2.2 | 77.1$\pm$2.6 | 69.0$\pm$5.1 | 79.0$\pm$9.1 | 59.8$\pm$16.8 | 51.0$\pm$17.6
SHE | 77.1$\pm$4.7 | 48.0$\pm$2.8 | 24.5$\pm$4.0 | 77.5$\pm$1.7 | 48.4$\pm$6.2 | 35.0$\pm$4.1
KLM | 86.3$\pm$8.5 | 77.5$\pm$17.9 | 71.2$\pm$22.7 | 88.6$\pm$4.8 | 74.5$\pm$11.6 | 65.5$\pm$15.9
OpenMax | 83.9$\pm$0.8 | 60.6$\pm$2.4 | 42.1$\pm$4.1 | 91.3$\pm$1.1 | 74.5$\pm$3.1 | 55.1$\pm$6.7
MSP | 94.0$\pm$0.4 | 95.6$\pm$0.6 | 98.4$\pm$1.0 | 93.7$\pm$0.6 | 96.0$\pm$0.6 | 97.1$\pm$0.7
MLP | MLS | 94.1$\pm$0.4 | 95.8$\pm$0.9 | 98.5$\pm$0.9 | 93.4$\pm$0.8 | 93.8$\pm$2.1 | 93.2$\pm$3.1
| TempScale | 94.0$\pm$0.4 | 95.6$\pm$0.6 | 98.4$\pm$0.9 | 93.8$\pm$0.6 | 96.1$\pm$0.6 | 97.2$\pm$0.6
| ODIN | 94.0$\pm$0.4 | 95.6$\pm$0.6 | 98.4$\pm$1.0 | 93.7$\pm$0.6 | 96.0$\pm$0.5 | 97.1$\pm$0.6
| EBO | 94.2$\pm$0.4 | 95.7$\pm$1.1 | 98.5$\pm$0.9 | 93.3$\pm$1.1 | 93.4$\pm$2.5 | 92.9$\pm$3.3
| GRAM | 98.0$\pm$1.7 | 98.7$\pm$0.8 | 99.9$\pm$0.0 | 99.9$\pm$0.1 | 99.9$\pm$0.1 | 99.9$\pm$0.1
| GradNorm | 94.2$\pm$0.5 | 96.0$\pm$0.4 | 98.9$\pm$0.7 | 94.3$\pm$0.5 | 96.9$\pm$0.4 | 97.7$\pm$0.6
| ReAct | 92.4$\pm$2.3 | 91.8$\pm$4.0 | 91.4$\pm$5.3 | 92.5$\pm$1.2 | 89.7$\pm$3.0 | 86.8$\pm$4.6
| DICE | 93.1$\pm$0.6 | 93.7$\pm$0.9 | 95.5$\pm$2.0 | 93.2$\pm$0.4 | 94.6$\pm$0.9 | 96.0$\pm$2.0
| ASH | 94.2$\pm$0.5 | 95.6$\pm$0.8 | 98.1$\pm$1.0 | 93.6$\pm$0.8 | 93.4$\pm$2.3 | 92.6$\pm$3.0
ResNet | MDS | 57.7$\pm$4.5 | 26.6$\pm$4.5 | 20.6$\pm$4.9 | 60.9$\pm$2.3 | 39.7$\pm$2.7 | 30.7$\pm$3.0
RMDS | 87.2$\pm$2.7 | 64.5$\pm$6.3 | 39.1$\pm$6.0 | 91.6$\pm$1.7 | 70.7$\pm$2.9 | 47.9$\pm$4.0
KNN | 69.4$\pm$4.9 | 41.4$\pm$6.1 | 33.0$\pm$6.5 | 70.3$\pm$3.3 | 46.7$\pm$0.7 | 35.5$\pm$1.8
VIM | 60.9$\pm$4.8 | 26.9$\pm$4.1 | 19.7$\pm$4.3 | 65.8$\pm$0.8 | 42.4$\pm$3.5 | 31.9$\pm$3.7
SHE | 68.3$\pm$3.2 | 34.4$\pm$2.8 | 20.4$\pm$3.0 | 71.4$\pm$4.6 | 44.2$\pm$2.9 | 32.3$\pm$2.7
KLM | 88.4$\pm$1.3 | 72.7$\pm$8.0 | 58.0$\pm$14.1 | 90.9$\pm$1.3 | 74.8$\pm$4.9 | 63.7$\pm$8.5
OpenMax | 83.0$\pm$1.0 | 64.0$\pm$6.0 | 42.9$\pm$9.6 | 87.2$\pm$3.7 | 69.7$\pm$2.1 | 61.0$\pm$13.3
MSP | 91.4$\pm$2.5 | 93.6$\pm$1.6 | 97.7$\pm$0.4 | 90.6$\pm$3.1 | 94.1$\pm$1.7 | 97.5$\pm$0.2
ResNet | MLS | 90.8$\pm$1.6 | 92.0$\pm$1.3 | 94.6$\pm$2.2 | 89.9$\pm$3.6 | 89.8$\pm$5.4 | 89.1$\pm$7.6
| TempScale | 91.4$\pm$2.5 | 93.6$\pm$1.6 | 97.8$\pm$0.4 | 90.7$\pm$3.1 | 94.1$\pm$1.7 | 97.5$\pm$0.2
| ODIN | 91.4$\pm$2.5 | 93.6$\pm$1.6 | 97.7$\pm$0.4 | 90.5$\pm$3.0 | 94.1$\pm$1.6 | 97.5$\pm$0.3
| EBO | 90.6$\pm$1.5 | 91.8$\pm$1.5 | 94.4$\pm$2.3 | 89.7$\pm$3.8 | 89.0$\pm$6.1 | 88.8$\pm$8.2
| GRAM | 98.6$\pm$1.6 | 99.9$\pm$0.1 | 100.0$\pm$0.0 | 99.8$\pm$0.2 | 99.9$\pm$0.1 | 99.9$\pm$0.1
| GradNorm | 95.0$\pm$0.4 | 98.2$\pm$1.0 | 99.4$\pm$0.4 | 94.2$\pm$0.4 | 97.5$\pm$0.7 | 98.4$\pm$0.2
| ReAct | 83.7$\pm$5.9 | 71.9$\pm$14.7 | 64.8$\pm$19.4 | 86.1$\pm$7.3 | 75.7$\pm$15.0 | 70.2$\pm$19.0
| DICE | 94.3$\pm$1.4 | 97.4$\pm$0.5 | 99.2$\pm$0.6 | 90.4$\pm$3.7 | 92.6$\pm$2.6 | 95.6$\pm$1.2
| ASH | 91.1$\pm$1.0 | 91.5$\pm$1.9 | 93.7$\pm$3.1 | 90.3$\pm$3.4 | 89.7$\pm$5.2 | 89.1$\pm$7.6
FT-T | MDS | 81.6$\pm$1.3 | 48.1$\pm$4.1 | 28.0$\pm$4.9 | 82.3$\pm$3.6 | 52.5$\pm$2.4 | 38.6$\pm$4.7
RMDS | 91.9$\pm$2.3 | 78.7$\pm$10.4 | 64.2$\pm$17.7 | 94.1$\pm$0.3 | 79.0$\pm$8.2 | 58.2$\pm$11.6
KNN | 83.6$\pm$1.8 | 53.4$\pm$5.3 | 29.8$\pm$5.6 | 85.5$\pm$2.3 | 55.0$\pm$1.7 | 39.7$\pm$4.6
VIM | 81.9$\pm$2.8 | 48.7$\pm$3.5 | 26.6$\pm$3.0 | 85.5$\pm$4.3 | 54.1$\pm$2.1 | 40.5$\pm$4.5
SHE | 88.2$\pm$2.5 | 63.0$\pm$6.6 | 41.6$\pm$4.4 | 91.2$\pm$1.3 | 68.4$\pm$2.9 | 43.3$\pm$5.3
KLM | 91.0$\pm$1.8 | 79.5$\pm$4.4 | 60.9$\pm$9.7 | 93.2$\pm$0.9 | 81.8$\pm$3.6 | 76.5$\pm$7.2
OpenMax | 91.2$\pm$1.9 | 80.7$\pm$4.8 | 60.6$\pm$9.2 | 94.2$\pm$0.5 | 83.9$\pm$4.8 | 75.0$\pm$13.1
MSP | 91.0$\pm$1.4 | 79.2$\pm$2.5 | 58.8$\pm$5.3 | 94.7$\pm$0.4 | 88.1$\pm$3.0 | 74.8$\pm$7.9
FT-T | MLS | 91.2$\pm$1.0 | 80.4$\pm$3.7 | 61.0$\pm$7.7 | 94.5$\pm$0.5 | 87.9$\pm$5.8 | 79.2$\pm$13.8
| TempScale | 91.0$\pm$1.4 | 79.2$\pm$2.5 | 58.8$\pm$5.3 | 94.7$\pm$0.4 | 88.0$\pm$3.0 | 74.8$\pm$7.9
| ODIN | 90.9$\pm$1.4 | 79.0$\pm$2.4 | 58.4$\pm$5.1 | 94.8$\pm$0.4 | 88.1$\pm$3.0 | 74.6$\pm$7.7
| EBO | 91.6$\pm$0.8 | 82.0$\pm$5.1 | 63.7$\pm$9.0 | 94.4$\pm$0.7 | 88.1$\pm$6.0 | 80.3$\pm$14.3
| GRAM | 94.1$\pm$0.6 | 89.8$\pm$4.7 | 79.9$\pm$8.3 | 95.1$\pm$0.3 | 88.9$\pm$2.8 | 77.9$\pm$7.0
| GradNorm | 91.3$\pm$1.4 | 78.6$\pm$4.5 | 58.3$\pm$6.7 | 94.5$\pm$0.5 | 87.2$\pm$3.7 | 74.1$\pm$8.1
| ReAct | 91.8$\pm$0.7 | 82.9$\pm$3.9 | 65.5$\pm$6.7 | 94.1$\pm$0.8 | 87.2$\pm$5.3 | 78.3$\pm$11.8
| DICE | 88.3$\pm$1.1 | 66.9$\pm$2.3 | 45.1$\pm$2.7 | 92.6$\pm$1.6 | 69.5$\pm$2.4 | 48.3$\pm$3.8
| ASH | 91.6$\pm$0.7 | 81.0$\pm$4.5 | 61.0$\pm$6.9 | 94.6$\pm$0.6 | 88.2$\pm$5.0 | 80.4$\pm$13.6
Table 8:  Comparing OOD detection methods in the synthesized-OOD setting using
eICU and MIMIC-IV datasets, with performance evaluated through FPR@95. This
Table is similar to Table 3, but assesses FPR@95 instead of utilizing AUROC.
In this setting, one feature is multiplied by a specific factor $\mathcal{F}$
to construct an OOD sample. For each model and factor we display the average
performance, calculated over 100 resampling of the feature and 5 different
initializations of the model. For each model, the best value in each column is
shown in bold, and distance-based post-hoc approaches are distinguished with a
lighter color.