ifaamas [AAMAS ’24]Proc. of the 23rd International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2024)May 6 – 10, 2024 Auckland, New
ZealandN. Alechina, V. Dignum, M. Dastani, J.S. Sichman (eds.) 2024 2024 477
Harvard University Cambridge, MA United States University of Oxford Oxford
United Kingdom University of Oxford Oxford United Kingdom University of Oxford
Oxford United Kingdom University of Oxford Oxford United Kingdom

# Rethinking Out-of-Distribution Detection for Reinforcement Learning:
Advancing Methods for Evaluation and Detection

Linas Nasvytis<EMAIL_ADDRESS>,  Kai Sandbrink
<EMAIL_ADDRESS>,  Jakob Foerster<EMAIL_ADDRESS>,
Tim Franzmeyer†<EMAIL_ADDRESS>and  Christian Schroeder de Witt†
<EMAIL_ADDRESS>
###### Abstract.

While reinforcement learning (RL) algorithms have been successfully applied
across numerous sequential decision-making problems, their generalization to
unforeseen testing environments remains a significant concern. In this paper,
we study the problem of out-of-distribution (OOD) detection in RL, which
focuses on identifying situations at test time that RL agents have not
encountered in their training environments. We first propose a clarification
of terminology for OOD detection in RL, which aligns it with the literature
from other machine learning domains. We then present new benchmark scenarios
for OOD detection, which introduce anomalies with temporal autocorrelation
into different components of the agent-environment loop. We argue that such
scenarios have been understudied in the current literature, despite their
relevance to real-world situations. Confirming our theoretical predictions,
our experimental results suggest that state-of-the-art OOD detectors are not
able to identify such anomalies. To address this problem, we propose a novel
method for OOD detection, which we call DEXTER (Detection via Extraction of
Time Series Representations). By treating environment observations as time
series data, DEXTER extracts salient time series features, and then leverages
an ensemble of isolation forest algorithms to detect anomalies. We find that
DEXTER can reliably identify anomalies across benchmark scenarios, exhibiting
superior performance compared to both state-of-the-art OOD detectors and high-
dimensional changepoint detectors adopted from statistics.

###### Key words and phrases:

Reinforcement Learning; Out-of-Distribution Detection; Anomaly Detection;
Robust RL; AI Safety

##  1\. Introduction

Deep reinforcement learning (RL) algorithms (Mnih et al., 2015; Schulman et
al., 2017a; Haarnoja et al., 2018) have been applied to numerous sequential
decision-making problems, ranging from robotics (Todorov et al., 2012;
Andrychowicz et al., 2020) and nuclear fusion (Degrave et al., 2022) to solar
geoengineering (de Witt and Hornigold, 2019). However, their low reliability
in situations that are not well represented in the training environment
hinders the deployment of RL agents in safety-critical scenarios Chan et al.
(2019); Packer et al. (2018). Such discrepant deployment situations are often
referred to as out-of-distribution scenarios. The task of identifying whether
a given environment differs from the train-time environment is hence referred
to as out-of-distribution (OOD) detection Haider et al. (2023). OOD detection
constitutes an important desiderata for the deployment of RL agents in the
real world, as reliable OOD detection would increase the safety of deployed AI
agents by allowing contingency actions to be taken in unfamiliar or dangerous
situations. Examples of such contingency options include the automatic parking
of an autonomous car on the side of the road, or raising security escalations
in cyber systems.††† Equal supervision

The OOD detection framework assumes that the agent has access to data from the
training process, which must be used to develop a mechanism for detecting OOD
scenarios in the unknown deployment environment within a minimal amount of
interactions. Such OOD deployment environments are often simulated by adding
anomalies (e.g. sensor or process noise) to the train-time environments
(Mohammed and Valdenegro-Toro, 2021; Haider et al., 2023; Danesh and Fern,
2021). In prior work, benchmark environments consider either injecting
independent and identically distributed (i.i.d.) anomalies, such as Gaussian
noise, or time-independent anomalies, such as changing gravity (Mohammed and
Valdenegro-Toro, 2021; Haider et al., 2023; Danesh and Fern, 2021). We argue
that these scenarios fail to capture the _temporally dependent_ nature of many
real-world anomalies. For example, in the case of an autonomous robot, any
dust or smudge appearing on its camera lens would lead to a series of readings
that contain systematic errors, instead of causing random misreadings of
environmental data. Further, previous works (Haider et al., 2023; Danesh and
Fern, 2021) employ decision rules based on per-step anomaly scores. While such
methods may prove empirically effective in some cases, we propose an
information-theoretically grounded approach based on sequential hypothesis
testing (Wald, 1945), which results in a decision rule based on accumulated
anomaly scores over multiple timesteps.

We start by introducing novel benchmark environments, which include temporally
dependent anomalies, and find that state-of-the-art OOD detection methods fail
in such scenarios. To detect such anomalies, we introduce a novel detection
mechanism called Detection via Extraction of Time Series Representations
(DEXTER). This mechanism first extracts time-series features, which are then
used with a random-forest algorithm to compute anomaly scores. We leverage
information-theoretically optimal sequential hypothesis testing techniques to
derive a cumulative sum (CUSUM) detector using the full history of DEXTER’s
anomaly scores, which we refer to as DEXTER+C. Lastly, we evaluate DEXTER and
DEXTER+C on a range of novel and standard benchmark environments and compare
their performance to relevant baselines. We find that DEXTER significantly
outperforms state-of-the-art detectors across various metrics, including Area
under the Receiver Operator Characteristic (AUROC) scores. Importantly,
DEXTER+C significantly decreases the number of timesteps needed to detect OOD
scenarios.

Our work makes the following contributions:

  * •

We propose a clarification of the terminology of OOD detection in
reinforcement learning.

  * •

We introduce new testing scenarios for OOD detection in reinforcement
learning, which consider a broad class of disturbances focused on temporally-
correlated noise.

  * •

We propose a new detector, which we refer to as DEXTER (Detection via
Extraction of Time Series Representations), as well as a new decision rule,
DEXTER+C, and show that these outperform state-of-the-art methods across
relevant scenarios.

##  2\. Related Work

####  2.0.1. Algorithms for OOD Detection

The research on out-of-distri-bution (OOD) detection in reinforcement learning
(Müller et al., 2022; Haider et al., 2023) is more limited compared to
supervised and unsupervised learning. However, interest in the field has grown
more recently. To our knowledge, Sedlmeier et al. (2019) outline the first
practical method for OOD detection in reinforcement learning. The authors use
epistemic uncertainty of the agent’s actions to quantify the anomaly scores of
different states, reasoning that epistemic uncertainty tends to be higher in
areas of low data density. Mohammed and Valdenegro-Toro (2021) introduce a
benchmark to evaluate generalized OOD detection methods in reinforcement
learning on three environments. In Cartpole and Pendulum, gravity is varied,
while in Pong, noise is added to state observations. Danesh and Fern (2021)
propose a more extensive benchmark to test OOD detection, implementing
different types of observational noise across seven classic RL environments.
Additionally, they introduce a new OOD detector called the Recurrent Implicit
Quantile Network (RIQN). At each time step, RIQN uses the current and prior
states in the environment, $s_{1:t}$, to generate auto-regressive predictions
for the next $\delta$ states, $s_{t+1:t+\delta}$, then computes the difference
between its predictions and the realized environment states, and uses this
difference as the anomaly score for a given transition $(s_{t},s_{t+1})$. The
authors demonstrate that RIQN outperforms several baseline detectors across
several of the proposed anomalous environments.

The state-of-the-art OOD detection method, Probabilistic Ensemble Dynamics
Model (PEDM), was proposed by Haider et al. (2023), and consists of two
components. First, a 1-step forward dynamics model $f_{\theta}$ is learned,
modeling the transition dynamics of the training environment, implemented as a
Probabilistic Deep Neural Network Ensemble model (Lakshminarayanan et al.,
2017). For a given state and action pair $(s_{t},a_{t})$, this model predicts
the next state in the environment as
$s^{\prime}_{t+1}=f_{\theta}(s_{t},a_{t})$. Second, an anomaly score generator
is applied, which compares the predictions of the world dynamics model to
outcomes in the test-time deployment environment, and generates an anomaly
score for each transition.

To measure performance in detecting anomalies applied to the observation
space, Haider et al. (2023) use the environments from Danesh and Fern (2021).
For the detection of anomalies applied to the transition dynamics of the RL
environment, the authors propose four additional benchmark environments. They
modify the classical Cartpole, HalfCheetah, Pusher, and Reacher environments
by adding semantic anomalies, such as changing the gravity or multiplying the
velocity applied to all body parts of the agent by a constant factor. The
performance of PEDM is measured against the RIQN algorithm (Danesh and Fern,
2021), as well as several other benchmark models. For each algorithm, the
performance is measured according to AUROC. Based on the results, PEDM
outperforms the other detection algorithms across almost all newly-proposed
environments with anomalous transition dynamics. Moreover, within the
environments from Danesh and Fern (2021), PEDM is tied with the RIQN model for
the best performance (with RIQN exhibiting significantly worse performance on
the new benchmarks).

####  2.0.2. Limitations of current OOD detection approaches.

We argue that current approaches to OOD detection suffer from three
weaknesses. First, most of the literature of OOD detection in RL makes the
simplifying assumption of injecting independent and identically distributed
(i.i.d.) noise across different timesteps in the environment (Danesh and Fern,
2021; Haider et al., 2023; Mohammed and Valdenegro-Toro, 2021), which can
hence be reliably detected with one-step detection approaches. However,
relevant real-world scenarios will likely feature more complex disturbances,
which have temporally correlated anomalies. Second, these approaches rely on
anomaly detection via prediction error, computing anomaly scores based on the
prediction error from a forward-dynamics model trained on training samples.
Such prediction models are unlikely to detect temporally-correlated anomalies
which may only have a minor effect on the transition dynamics. Third, the
decision rules for online out-of-distribution detection rely only on the
individual anomaly scores observed at each time step, instead of taking into
account the full history of anomaly scores. While such approaches may prove
empirically effective, they neglect information that would be used in
information-theoretically grounded sequential hypothesis testing (Wald, 1945).

Figure 1. Illustration of temporally autocorrelated anomalies. Left: at
injection time (t = 48), noise applied to the observation changes from no
correlation to 1-step autocorrelation. Right: at injection time (t = 56),
noise changes from no correlation to 2-step autocorrelation.

####  2.0.3. Sequential Hypothesis Testing.

Beyond machine learning, the task of online detection of whether a sample of
time series data differs from a predefined distribution has also been analyzed
in statistics, primarily in the field of changepoint detection (CPD).
Sequential hypothesis testing, as pioneered by Wald in his sequential
probability ratio test (SPRT) (Wald, 1945), aims to discern between two
hypotheses in an online manner, using as few samples as possible. CUSUM
methods, a popular technique in CPD, are designed to detect shifts in the mean
or variance of a process (Page, 1954). CUSUM methods focus on capturing
cumulative information over time, providing a mechanism to identify time-
dependent anomalies.

##  3\. Background and Notation

A Markov Decision Process (MDP) is defined by a tuple $(S,A,P,R)$. Here, $S$
is a finite set of states, $A$ is a finite set of actions, $P:S\times A\times
S\rightarrow[0,1]$ defines the state transition probabilities, and $R:S\times
A\rightarrow\mathbb{R}$ is the reward function. The goal of the agent at each
time step is to maximize the cumulative sum of discounted future rewards
$\sum_{t}^{t\rightarrow\infty}\gamma^{t}R(s_{t},a_{t})$, where
$\gamma\in[0,1]$ is the discount rate.

In the general machine learning literature – which includes supervised
learning, unsupervised learning, and reinforcement learning – there are
several terms used to describe the task of detecting if a new sample of data
differs from a specified training set. The terms can include out-of-
distribution (OOD) detection, anomaly detection, novelty detection, and
outlier detection. To clarify the terminology, Yang et al. (2021) provide an
in-depth literature review of this topic, focusing primarily on supervised
learning, where each data point contains an input-label pair ($x$,$y$) $\in
X\times Y$, where $X$ is the input (sensory) space, and $Y$ the label
(semantic) space. A data distribution is defined as a joint distribution
$P(X,Y)$ over $X\times Y$. A distribution shift can occur in either the
marginal distribution $P(X)$, or both $P(Y)$ and $P(X)$. As noted by Yang et
al. (2021), a shift in $P(Y)$ naturally triggers shift in $P(X)$. Denote the
distributions of normal and anomalous data points by $P$ and $P^{\prime}$,
respectively. Covariate shift (sensory) anomalies occur when the inputs of
normal data points are drawn from an in-distribution $P(X)$, whereas inputs of
anomalies are drawn from out-of-distribution $P^{\prime}(X)$, such that
$P(X)\neq P^{\prime}(X)$. However, no distribution shift occurs in the label
space: $P(Y)=P^{\prime}(Y)$. Semantic shift anomalies occur when a
distributional shift occurs in the label space, such that $P(Y)\neq
P^{\prime}(Y)$. Following this distinction, the authors define:

  1. (1)

Sensory anomaly detection as the task of detecting covariate shift anomalies,
i.e. samples from $P^{\prime}(X)$.

  2. (2)

Semantic anomaly detection as the task of detecting semantic shift anomalies,
i.e. samples from $P^{\prime}(Y)$.

  3. (3)

Out-of-distribution (OOD) detection as the sub-domain within semantic anomaly
detection, where in-distribution samples are drawn from multiple classes (i.e.
label space $Y$ is not binary).

  4. (4)

Generalized out-of-distribution detection as the general task of detecting all
anomalies, i.e. both sensory and semantic anomalies.

The main issue is that the training and testing data in reinforcement learning
does not contain a label space $Y$. As a result, the terms out-of-distribution
detection and anomaly detection are often used interchangeably (Haider et al.,
2023), and some of their definitions seem to conflict with each other. A
detailed overview of such terminological discussions on OOD detection in
reinforcement learning can be found in the Appendix.

##  4\. Terminology of OOD Detection in Reinforcement Learning

Given terminological discussions on how to label different types of anomalies
for OOD detection in reinforcement learning, we propose terminology adapted
from the framework introduced by Yang et al. (2021), while incorporating
insights from existing literature from Danesh and Fern (2021) and Haider et
al. (2023). We first differentiate between two kinds of anomalies based on
their effects on the MDP: First, sensory anomalies change the observation that
the agent receives (e.g. adding observational noise), while leaving the
underlying environment dynamics unchanged. Hence, these are closely related to
covariate shift anomalies defined in Yang et al. (2021). In contrast, semantic
anomalies change the underlying environment dynamics (e.g. changing the
gravity in the environment). Hence, these are very similar to semantic shift
anomalies defined in Yang et al. (2021). We further define Generalized Out-of-
Distribution Detection as the task of detecting either of the two.

######  Definition 0.

Sensory anomaly detection in reinforcement learning refers to the task of
identifying sensory anomalies. A sensory anomaly is a perturbation to the
reinforcement learning environment, which changes the observations $O$ that
the agent receives, but leaves the underlying environment dynamics unchanged.
If the change in observation leads the environment to become partially
observable, such an anomaly changes the underlying Markov Decision Process
(MDP) to a Partially Observable Markov Decision Process (POMDP).

######  Definition 0.

Semantic anomaly detection in reinforcement learning refers to the task of
identifying semantic anomalies. Semantic anomalies are perturbations to the
reinforcement learning environment, which change the transition function of a
Markov Decision Process $P(S^{\prime}|S,A)$ by changing the environment
dynamics.

######  Definition 0.

Generalized out-of-distribution detection in reinforcement learning refers to
the task of identifying any type of anomaly in the environment, hence
including sensory anomaly detection and semantic anomaly detection.

The proposed terminology unites the task of detecting all types of anomalies
in reinforcement learning environments under the single term of generalized
out-of-distribution detection, while drawing a meaningful distinction between
the two major types of anomalies that could exist in an environment. This is
especially relevant as some recent works have been inaccurate about the
changes to the MDP that the anomalies introduce (see Appendix for a
discussion).

##  5\. Novel Testing Scenarios for OOD Detection

We now propose new benchmark environments for generalized OOD detection that
contain both sensory and semantic anomalies. In contrast to the benchmark
environments used in previous works, these environments contain temporally
correlated anomalies, which is achieved by generating and adding noise using
the autoregressive (AR) model (Box et al., 2015b).

####  5.0.1. Autocorrelated noise patterns

To create custom benchmark environments with different types of noise
correlations, we use an AR model of order $p$, denoted by AR(p), where the
noise value $Y_{t}$ of a series at any point in time is linearly dependent on
its own $p$ past values:

$Y_{t}=\mu+\phi_{1}Y_{t-1}+\phi_{2}Y_{t-2}+\cdots+\phi_{p}Y_{t-p}+\varepsilon_{t}$

Using this AR model, we implement noise with three different types of
autocorrelations:

  1. (1)

No correlation, in which case the noise is not correlated across time, i.e.,
containing white noise, given as $Y_{t}=\epsilon_{t}$.

  2. (2)

1-step correlation, in which case the noise is autocorrelated across each
step, hence given as $Y_{t}=\phi_{1}Y_{t-1}+\epsilon_{t}$.

  3. (3)

2-step correlation, where the noise is autocorrelated only at every second
step (i.e., step 2, step 4), in which case,
$Y_{t}=\phi_{2}Y_{t-2}+\epsilon_{t}$.

Examples of the three types of noise for correlation coefficients are
displayed for $\phi=0.95$ in Figure 2.

Figure 2. Illustrations of Autoregressive (AR) model with parameters for no
correlation (top), 1-step autocorrelation (bottom-left), and 2-step
autocorrelation (bottom-right), which is used to create three different types
of noise in new testing scenarios.

Based on the autocorrelated noise patterns introduced above, we introduce
three new benchmark environments.

####  5.0.2. ARTS: Autoregressive Time Series environments

Autoregressive Time Series (ARTS) serves as our baseline environment. At the
beginning of each episode, we generate a 1-dimensional vector using the AR
process, with the number of elements equal to the maximum number of episode
steps. At each time step $t$, the agent receives a 1-dimensional observation
$o_{t}$ that contains the noise generated by the AR process, while the
environment state is treated as constant.

####  5.0.3. ARNO: Autoregressive Noised Observation environments

In the Autoregressive Noised Observation (ARNO) setting, we introduce sensory
anomalies. At the beginning of the episode, we generate a noise matrix, where
each row of the matrix is a time series, independently drawn from the AR
Process. The number of rows in the matrix corresponds to the dimension of a
single observation in the environment.

For each step $t$, the agent takes an action $a_{t}$, which is passed to the
environment’s state transition function to generate the next state:
$s_{t+1}\sim P(s_{t+1}|s_{t},a_{t})$. After each state transition, the noise
vector sliced from the matrix is added to the state $s_{t+1}$, transforming it
into an observation $o_{t+1}$. This simulates a scenario where the
environment’s underlying state undergoes a distortion before being presented
to the agent, similar to the effects of a camera glitch or a sensory failure.

####  5.0.4. ARNS: Autoregressive Noised State environments

In Autoregressive Noised State (ARNS) environments, we introduce semantic
anomalies. Similarly to ARNO, we generate the AR noise matrix at the beginning
of each episode, with the same shape. At each step $t$, the agent takes an
action $a_{t}$, which is passed to the environment’s state transition
function. However, before the transition, a noise vector sliced from the
matrix is applied to each dimension of the state vector, effectively changing
the transition function. This simulates a scenario where, for example, the
underlying physics or rules of the environment are unpredictably changed due
to systematic issues.

##  6\. DEXTER: Detection via Extraction of Time Series Representations

We now move on to discuss our proposed algorithm for out-of-distribution
detection in reinforcement learning, which we refer to as DEXTER (Detection
via Extraction of Time-series Representations). The model is composed of two
components: first, a feature extractor $f(s_{1},...s_{n})$ that extracts
relevant time series features from a given time series data; and second, an
anomaly detector $h(f(s_{1},...s_{n})$, which we implement as an ensemble of
isolation forest models that takes as an input the extracted features, and
outputs an anomaly score for the time series.

####  6.0.1. Feature extractor $f$

Given a sample of states $s_{0},s_{1},\ldots s_{t}$, DEXTER first extracts the
relevant time series features from each state dimension. The goal behind the
proposed detector is to be anomaly agnostic: Instead of choosing features that
could detect a specific anomaly (e.g. autocorrelations), we aim to extract a
diversity of features to maximize the number of anomalies DEXTER could detect.
For this purpose, we use the tsfresh feature extractor (Christ et al., 2018),
which captures features of a diverse set of statistics of the time series,
including fundamental descriptive statistics (e.g. number of peaks, minimum,
maximum, and median values), autocorrelation statistics (e.g. autocorrelation
coefficients for $k$ lags; partial autocorrelation function (Box et al.,
2015a) at the given lag $k$), advanced features (e.g. descriptive statistics
of the absolute Fourier transform spectrum (Marple Jr and Carey, 1989) and
approximate entropy (Delgado-Bonal and Marshak, 2019)).

####  6.0.2. Isolation Forest Algorithm $h$

Next, we fit an ensemble of isolation forest algorithms, which predicts the
probability of an anomaly based on the extracted features. The isolation
forest algorithm is an ensemble-based unsupervised machine learning method
designed for anomaly detection, introduced by Liu et al. (2008). Given a
training set, the algorithm first constructs multiple random trees to
partition data points. For each tree, the algorithm selects a sample of
training data, and recursively partitions the sample of data points by
randomly selecting a split value that falls between the minimum and maximum
values of a given attribute (i.e. dimension). Once an ensemble of such
isolation trees is constructed, a test data point is passed down each of these
trees. The path length it takes to isolate the data point is averaged over the
trees to produce an anomaly score for the data point. The algorithm relies on
the assumption that anomalous points will be more easily separable from the
rest of the sample (Liu et al., 2012).

Isolation forest has the following beneficial properties: First, it is an
unsupervised method, which is essential as we are not provided anomalous data
and hence must train an anomaly-agnostic detector. Second, it has linear time
complexity (Liu et al., 2008), allowing to scale to high-dimensional data.
Third, it allows for anomaly scoring, enabling a nuanced interpretation of how
”anomalous” a data point is, instead of providing a mere binary
classification.

####  6.0.3. Training and testing.

An overview of DEXTER Anomaly Score computation is displayed in Algorithm 1.
We will denote the training data as a sample
$D=\\{s_{n},a_{n},s_{n+1}\\}^{N}_{n=1}$.

##### Training.

First, DEXTER partitions $D$ into windows of size $W$ states. For each window
$w_{i}$, the model extracts time-series features along each dimension $d$,
$1\leq d\leq m$, to obtain $f_{i,d}$. Once feature extraction for all windows
is complete: for each dimension $d$, all extracted features are concatenated
to make $f_{d}$. This aggregate feature set is used to fit the ensemble of
isolation forest models, where each model is assigned to a different state
dimension, resulting in $m$ total models. For each model in the ensemble
$IF_{d}$, the aim is to instill an understanding of the predominant patterns
present in the given dimension of in-distribution observations.

##### Dexter Anomaly Score Computation.

For a given timestep T, DEXTER collates the last $W$ states to constitute a
window, and extracts features $f_{d}^{\prime}$ along each dimension in $m$.
The ensemble of trained isolation forest models is then utilized to compute
anomaly scores for these features. The concluding anomaly score for a timestep
is calculated as the arithmetic mean of the scores spanning all the dimensions
in $m$.

###  6.1. Sequential Hypothesis Testing with DEXTER+C

The previously described DEXTER algorithm outputs an anomaly score after each
transition in the test-time deployment environment. We now introduce DEXTER+C
(short for DEXTER+CUSUM), which uses the information-theoretic hypothesis
testing CUSUM algorithm (Page, 1954) to derive a decision rule for when to
classify a test-time deployment as OOD. DEXTER+C is detailed in Algorithm 2.
First, the average DEXTER anomaly score $\bar{A}$ for a held-out set of
training transitions is computed. Then, the CUSUM detection threshold $\tau$
is computed by evaluating CUSUM on held-out training transitions and choosing
$\tau$ such that a targeted False Positive Rate ($FPR$) is achieved on the set
of held-out transitions. This tunable targeted $FPR$ indicates the ratio of
anomaly-free episodes that are falsely classified as OOD. Note that both steps
only require access to training transitions and that the only hyperparameter
is the targeted $FPR$. At test time, an online CUSUM detector with detection
threshold $\tau$ is employed. This detector updates the CUSUM score $S_{t}$
based on the previous timestep’s score and the current DEXTER anomaly score
$A_{t}$. If the CUSUM score $S_{t}$ exceeds $\tau$, an OOD scenario is
detected and the execution is halted.

Algorithm 1 DEXTER

State dimensions $m$, window size $W$, policy $\pi$, dataset $D$ of training
transitions

Training

Initialize ensemble of Isolation Forests
$\text{IF}=\\{\text{IF}_{1},...,\text{IF}_{m}\\}$

Partition $D$ into windows ${w_{1},...,w_{N}}$ of size $W$

for each $w_{i}$ in windows do

for dimension $d$ from 1 to $m$ do

Extract time series features $f_{i,d}$

end for

end for

for dimension $d$ from 1 to $m$ do

Form $f_{d}$ by concatenating $f_{i,d}$

Train $IF_{d}$ using $f_{d}$

end for

DEXTER Anomaly Score Computation

for Time $t$ from 0 to $T$ do

Action $a_{t}\leftarrow\pi(s_{t})$, observe $s_{t+1}$

Update window with $s_{t+1}$

for dimension $d$ from 1 to $m$ do

Extract features $f_{d}$

Compute score $a_{d}$ with $\text{IF}_{d}$

end for

Set $A_{T}$ as average over all $a_{d}$

end for

Algorithm 2 DEXTER+C

targeted $FPR$, state dimensions $m$, window size $W$, policy $\pi$,
validation dataset $D$ of training transitions

Compute Mean Anomaly Score $\bar{A}$ using Validation Set

Compute DEXTER anomaly scores for $\frac{1}{2}D$, storing in { $A_{t}$ }

$\bar{A}\leftarrow$ mean of { $A_{t}$ }

Compute CUSUM Threshold $\tau$ using Validation Set

$S_{max\\_list}\leftarrow[]$

for each episode $ep$ in $\frac{1}{2}D$ do

$S_{0}\leftarrow 0,S_{max}\leftarrow 0$

for each score $A_{t}$ in $ep$ do

$S\leftarrow S+A_{t}-\bar{A}$

$S_{max}\leftarrow\max(S,S_{max})$

end for

Append $S_{max}$ to $S_{max\\_list}$

end for

$\tau\leftarrow$ 1-$FPR$ percentile of $S_{max\\_list}$

DEXTER+C Out-of-Distribution Detection

$S_{0}\leftarrow 0$

for Time $t$ from 0 to $T$ do

Compute $A_{t}$ using DEXTER Anomaly Score Computation

$S_{t}\leftarrow\max(0,S_{t-1}+A_{t}-\bar{A})$

if $S_{t}$ ¿ $\tau$ then

Raise out-of-distribution alert

Break

end if

end for

##  7\. Empirical Evaluation

We first focus on the newly introduced benchmark scenarios ARNO, ARNS, and
ARTS, which contain temporally correlated anomalies. We implement ARNO
scenarios with three levels of noise (described in Section 7.1) in three
different environments – Cartpole, Reacher, and Acrobot (Brockman et al.,
2016). We implement ARNS scenarios with analogous levels of noise on Cartpole
and Reacher, as the implementation on Acrobot leads to inconsistent agent
policies, which are described in more detail in the Appendix. Lastly, as the
ARTS environment does not contain a reward signal, we only implement a single
noise level. The codebase and supplementary materials are publicly available
at: https://github.com/LinasNas/DEXTER.

We afterward consider the common benchmark scenarios introduced by Haider et
al. (2023), which contain either i.i.d anomalies or time-independent anomalies
(five scenarios in total), two noise levels for each anomaly, and two
different environments (Cartpole and Reacher), resulting in 20 different
evaluation frameworks.

Previous works (Haider et al., 2023; Danesh and Fern, 2021) focus on AUROC
computed for per-transition anomaly scores as the main metric for detector
performance. However, AUROC scores do not yield a decision rule for
classifying observation as OOD. We hence also consider the required timesteps
to classify the anomaly-containing test-time environment as OOD as a metric
for detector performance, we refer to this metric as _Detection Time_.

###  7.1. Evaluations for ARTS, ARNS and ARNO Scenarios

##### Noise levels in each environment

In ARNO and ARNS environments, we apply three different levels of noise to
generate the anomalies. The levels are classified as Light, Medium, and
Strong, based on their effect on the agent’s reward in the environment. The
procedure to choose these levels is as follows. First, we normalize the noise
magnitude by the standard deviation of that dimension’s observation. Then, we
apply uncorrelated noise of different magnitudes to the environment and train
a reinforcement learning agent using Proximal Policy Optimization (PPO)
(Schulman et al., 2017b) in discrete action-space environments, and Twin
Delayed Deep Deterministic (TD3)Fujimoto et al. (2018) algorithm for
continuous action-space environments, until it converges to a stable
cumulative episodic reward. We then measure how much, averaged over 50
episodes, the reward differs from the reward achieved when an agent’s policy
is optimized in an undisturbed environment. Lastly, in each environment, we
identify three magnitudes of noise, which we classify as:

  * •

Light noise: reduces $\sum_{t=0}^{T}r_{t}$ by $\sim 1\%$

  * •

Medium noise: reduces $\sum_{t=0}^{T}r_{t}$ by $\sim 25\%$

  * •

Strong noise: reduces $\sum_{t=0}^{T}r_{t}$ by $\sim 50\%$

####  7.1.1. Evaluation Procedure

Our experimental evaluation follows recent frameworks (Haider et al., 2023;
Danesh and Fern, 2021). We first apply uncorrelated noise of each of the three
levels. Then for each noise level, we train a policy $\pi_{A}(a|s)$ until it
is optimized for the episodic task with episode length $H\in\mathbb{N}$.

In each episode, we introduce an anomaly at a random time
$t_{a}\in(t_{0}+5,t_{H}-5)$, and apply this anomaly until the end of the
episode. That anomaly changes the correlation structure of noise from no
correlation to either 1-step or 2-step correlation, as it can be observed in
Figure 1. We test each of the two cases separately. All transitions before the
anomaly is applied, $[(s_{t_{0}},s_{t_{1}}),\ldots,(s_{t_{a}-1},s_{t_{a}})]$
are labeled as in-distribution, and all transitions after,
$[(s_{t_{a}},s_{t_{a+1}})$ $\ldots,(s_{t_{H-1}},s_{t_{H}})]$ are labelled as
anomalous.

To account for the effects of initialization and varying injection times, we
repeat this procedure for multiple random point time points and different
initial environment states. Therefore, in expectation, we obtain a balanced
dataset.

Importantly, the noise level throughout the episode is constant, since after
the injection time, we simply change the correlation structure of the noise,
but not its magnitude. The experiments are implemented in such a way since we
are specifically interested in whether the detectors can identify structural
changes to the noise, rather than detect the sudden emergence of noise itself,
unlike most prior benchmark environments (Haider et al., 2023; Danesh and
Fern, 2021).

Table 1. ARTS scenarios: Detector performance (AUC above, Detection time
below).

|  | 1-step | 2-step
---|---|---|---
AUROC $\uparrow$ | CPD: OCD | 0.79 | 0.77
CPD: Chan | 0.64 | 0.64
PEDM | 0.51 | 0.5
DEXTER | 0.89 | 0.83
Det. Time $\downarrow$ | PEDM+C | 200.0 | 200.0
DEXTER+C | 19.8 | 28.0

Table 2. ARNO scenarios: Detector performance (AUC above, Detection time
below).

|  | Cartpole | Acrobot | Reacher
---|---|---|---|---
|  | Light Noise | Medium Noise | Strong Noise | Light Noise | Medium Noise | Strong Noise | Light Noise | Medium Noise | Strong Noise
| Detector | 1-step | 2-step | 1-step | 2-step | 1-step | 2-step | 1-step | 2-step | 1-step | 2-step | 1-step | 2-step | 1-step | 2-step | 1-step | 2-step | 1-step | 2-step
AUROC $\uparrow$ | CPD: OCD | 0.67 | 0.69 | 0.76 | 0.72 | 0.78 | 0.73 | 0.64 | 0.65 | 0.75 | 0.76 | 0.8 | 0.79 | 0.51 | 0.51 | 0.51 | 0.51 | 0.52 | 0.52
CPD: Chan | 0.69 | 0.68 | 0.72 | 0.75 | 0.75 | 0.73 | 0.62 | 0.59 | 0.76 | 0.71 | 0.86 | 0.77 | 0.51 | 0.51 | 0.52 | 0.52 | 0.53 | 0.53
PEDM | 0.55 | 0.62 | 0.6 | 0.51 | 0.6 | 0.55 | 0.57 | 0.54 | 0.52 | 0.54 | 0.5 | 0.53 | 0.81 | 0.5 | 0.84 | 0.51 | 0.87 | 0.5
DEXTER | 0.81 | 0.85 | 0.89 | 0.9 | 0.93 | 0.9 | 0.74 | 0.71 | 0.96 | 0.91 | 0.99 | 0.95 | 0.67 | 0.6 | 0.91 | 0.63 | 0.97 | 0.61
Det. Time $\downarrow$ | PEDM+C | 138.8 | 143.2 | 133.0 | 79.3 | 199.1 | 183.1 | 163.9 | 199.2 | 79.7 | 197.2 | 177.2 | 169.9 | 21.3 | 200.7 | 19.3 | 193.6 | 18.1 | 200.8
DEXTER+C | 23.2 | 15.2 | 13.1 | 19.9 | 14.85 | 14.7 | 32.1 | 42.5 | 12.7 | 13.9 | 8.0 | 11.5 | 60.9 | 72.1 | 20.0 | 197.1 | 12.4 | 110.6

Table 3. ARNS scenarios: Detector performance (AUC above, Detection time
below).

|  | Cartpole | Reacher
---|---|---|---
|  | Light Noise | Medium Noise | Strong Noise | Light Noise | Medium Noise | Strong Noise
| Detector | 1-step | 2-step | 1-step | 2-step | 1-step | 2-step | 1-step | 2-step | 1-step | 2-step | 1-step | 2-step
AUROC $\uparrow$ | CPD: OCD | 0.66 | 0.66 | 0.68 | 0.68 | 0.67 | 0.68 | 0.51 | 0.51 | 0.51 | 0.51 | 0.51 | 0.51
CPD: Chan | 0.67 | 0.68 | 0.68 | 0.69 | 0.68 | 0.7 | 0.51 | 0.51 | 0.51 | 0.51 | 0.51 | 0.51
PEDM | 0.66 | 0.64 | 0.63 | 0.61 | 0.59 | 0.56 | 0.52 | 0.51 | 0.55 | 0.55 | 0.51 | 0.5
DEXTER | 0.73 | 0.73 | 0.88 | 0.8 | 0.84 | 0.77 | 0.56 | 0.62 | 0.51 | 0.7 | 0.55 | 0.67
Det. Time $\downarrow$ | PEDM+C | 33.0 | 44.5 | 53.1 | 50 | 44.5 | 125.3 | 189.4 | 197.7 | 201.0 | 201.0 | 195.8 | 197.2
DEXTER+C | 32.2 | 16.9 | 17.8 | 37.8 | 24.0 | 27.4 | 199.0 | 95.1 | 193.6 | 61.1 | 198.7 | 66.7

####  7.1.2. DEXTER and DEXTER+C

We implement DEXTER (described in Algorithm 1) with a window size of $W=10$
timesteps, since it strikes a balance between a sample that is long enough to
allow for meaningful time series feature extraction, while being short enough
to allow for quick anomaly detection. We implement DEXTER+C (described in
Algorithm 2) with a target $FPR$ of 1%.

Table 4. Benchmark scenarios: Detector performance (AUC above, Detection time
below).

|  | Cartpole | Reacher
---|---|---|---
|  | Action Fact. | Action Noise | Action Offset | Body M. Fact. | Force Vector | Action Fact. | Action Noise | Action Offset | Body M. Fact. | Force Vector
| Detector | Minor | Severe | Minor | Severe | Minor | Severe | Minor | Severe | Minor | Severe | Minor | Severe | Minor | Severe | Minor | Severe | Minor | Severe | Minor | Severe
AUROC $\uparrow$ | PEDM | 0.59 | 0.94 | 0.66 | 0.98 | 0.82 | 1.0 | 0.6 | 0.9 | 0.59 | 1.0 | 0.62 | 0.95 | 0.59 | 0.99 | 0.61 | 0.98 | 0.64 | 0.56 | 0.74 | 0.98
DEXTER | 0.76 | 0.64 | 0.75 | 0.55 | 0.76 | 0.71 | 0.75 | 0.66 | 0.72 | 0.56 | 0.73 | 0.69 | 0.72 | 0.62 | 0.74 | 0.66 | 0.71 | 0.73 | 0.76 | 0.69
Det. Time $\downarrow$ | PEDM+C | 153.1 | 1.7 | 76.3 | 1.7 | 200.0 | 1.1 | 151.3 | 2.9 | 127.1 | 1.1 | 199.6 | 196.6 | 200.8 | 52.7 | 200.5 | 192.3 | 199.3 | 201.0 | 200.1 | 200.8
DEXTER+C | 44.8 | 50.4 | 42.1 | 69.4 | 44.0 | 52.8 | 46.2 | 50.9 | 48.2 | 200.0 | 53.6 | 53.3 | 51.9 | 129.9 | 46.8 | 65.6 | 53.7 | 52.1 | 44.0 | 55.2

####  7.1.3. Probabilistic Ensemble Dynamics Model (PEDM)

We implement the PEDM from Haider et al. (2023) as the state-of-the-art
benchmark for generalized out-of-distribution detection. For a fair
comparison, we further also implement a CUSUM-based PEDM detector analogously
to DEXTER+C, which we refer to as PEDM-C.

####  7.1.4. Changepoint detectors

In the newly introduced scenarios, we also implement change point detectors
from Chen et al. (2022), which the authors introduce under the name OCD, as
well as another changepoint detector proposed by Chan (2017). OCD conducts
likelihood ratio tests against simple alternatives of different scales in each
coordinate, and then aggregates these test statistics across scales and
coordinates. Unlike OCD, which is tailored for detecting changes in the mean
of a multi-dimensional Gaussian data stream, a detector from Chan (2017) tests
the null hypothesis of data following a multivariate normal distribution
against an alternative where one of the coordinates has a mixture
distribution. Such methods constitute state-of-the-art models used to identify
changes in high-dimensional data streams, with a focus on changes to the mean,
from online changepoint detection literature in statistics. More information
on their implementation is provided in the Appendix.

###  7.2. Detection Performance Metrics

To evaluate the performance of the different detectors, we consider both AUROC
as well as average detection time, i.e. the average number of steps before an
anomalous episode is classified as out-of-distribution. If the anomaly is not
classified as OOD before the end of the episode, we set the detection time to
the maximum episode length.

###  7.3. Detection Performance Results in novel scenarios

The results for both performance metrics and for all detectors across all
scenarios and environments are given in Tables 1, 2, 3.

##### AUROC results

We find that DEXTER outperforms all other detectors in the vast majority of
scenarios and noise levels, achieving the best performance in 2 out of 2 ARTS
settings, 17 out of 18 ARNO settings, and 11 out of 12 ARNS settings. We
observe that PEDM detector performs no better than random guessing in ARTS
setting (average AUROC of 0.51), as well as marginally better than random
guessing in ARNO scenarios (average AUROC of 0.59). The main exception is the
1-step correlated noise setting in ARNO Reacher environment, where PEDM
performs with an average of 0.84 AUROC across the three noise levels, and
outperforms DEXTER on the Light Noise level. The performance of PEDM on ARNS
is similar, with an average AUROC score of 0.57 across all noise levels and
environments. Such results confirm our hypothesis about the model’s
limitations outlined in Section 2.0.2. CPD detectors perform only marginally
better than PEDM across the three types of noise in Cartpole and Acrobot,
though their AUROC scores across Reacher environments in ARNO and ARNS
settings are below PEDM, with an average of around 0.51. The scores of
changepoint detectors in these settings are also quite repetitive, since in
the vast majority of cases, they consistently detect an anomaly at one of the
earliest steps in the episode. Since both algorithms are designed to identify
changes to the mean of time series data, this may be especially difficult in
high-dimension noisy environments, such as Reacher.

##### Detection time results

We find that DEXTER+C outperforms PEDM+C by a large margin in the vast
majority of settings, in accordance with the higher AUROC scores.

###  7.4. Detection Performance Results in Benchmark Scenarios

We observe in Table 2 that DEXTER (DEXTER+C) generally outperforms PEDM
(PEDM+C) for minor anomaly scenarios, while the opposite holds true for severe
anomalies. This finding is as expected, as DEXTER requires a larger time-
series window to extract features. These results suggest that a combination of
both DEXTER (DEXTER+C) and PEDM (PEDM+C) approaches might result in optimal
outcomes.

##  8\. Conclusion and future work

This paper focused on the problem of generalized OOD detection in
reinforcement learning. We started by outlining the terminological discussions
that are currently present in the literature, and offered a framework that
formalizes the problem and aligns its terminology with the broader field of
generalized OOD detection in machine learning. We then outlined potential
shortcomings in the architectures of current state-of-the-art detectors, and
introduced a new set of testing scenarios for generalized OOD detection in RL.

By introducing noise with different types of autocorrelations, these scenarios
focus on anomalies that, to the best of our knowledge, have not been
explicitly studied in the literature, but are highly relevant to many real-
world scenarios. We also proposed a new generalized OOD detection model called
DEXTER, which first extracts relevant time series features from observations,
and then applies an ensemble of isolation forest algorithms to detect
potential anomalies. Its extension DEXTER+C further establishes a decision
rule by use of CUSUM.

Lastly, we also adopted several existing changepoint detection methods from
statistics for generalized OOD detection. After a series of experiments, we
found that DEXTER outperforms the existing state-of-the-art models for OOD
detection on the novel test scenarios. However, in some benchmark scenarios, a
combination of previously used approaches together with DEXTER appears to
yield improved results. In future work, we plan to explore such combinations
of different detection mechanisms.

Despite the successes of our method, it still faces several important
limitations, which point in the directions to take in future work. First, our
evaluations are limited to simulated environments. Future work should test on
sim-to-real settings. Second, we do not test on scenarios with noise that is
correlated across dimensions. Future work should address cross-dimensional
feature extraction. Third, DEXTER uses a fixed window length to detect
anomalies, which is fixed before interacting with the test-time environment.
Future work should study how this can be replaced by a sliding scale and/or
hierarchical pyramid of window sizes. Lastly, drawing on existing work (Teh et
al., 2021), we foresee the process of effectively selecting relevant features
at test-time as a natural step to improve DEXTER’s efficiency in high-
dimensional environments.

Generalized OOD detection is critical to ensure the safety of RL algorithms
when they are deployed in the real world. Unlike the kinds of out-of-
distribution perturbations that have been considered in previous work, the
robotic systems such as drones that have already been used in the real world
Kaufmann et al. (2023) risk facing noise that is correlated across timepoints.
This noise could come from physical disturbances such as a malfunctioning
robot joint or a partially broken camera lens. If the disturbance at each
individual time point is slight, it risks not to be caught by systems relying
on one-step transitions like PEDM.

However, over a long time period, these disturbances can cause significant
damage to the system, decreasing performance and putting potentially dangerous
strain on the rest of the robot. Further, not addressing temporally correlated
noise risks leaving the control system open to adversarial attacks, which
induce perturbations that are correlated across time, such as illusory attacks
Franzmeyer et al. (2022). This suggests that inclusion of a DEXTER-like system
for detection of temporally-correlated noise – possibly in conjunction with a
PEDM-like system optimized at detecting one-step perturbations – is necessary
to ensure safe deployment of RL systems.

More broadly, we hope that our work helps shift future OOD dynamics detection
towards more general approaches, improving both AI safety and security.
Importantly, this work helps facilitate the move from individual time-step
decision rules to information-theoretic optimal CUSUM detection methods.

We’d like to thank James Duffy for an in-depth overview of out-of-distribution
detection methods in econometrics and statistics. We further thank Philip Torr
for helpful comments. LN was supported by the Center for AI Safety (CAIS) and
Future of Life Institute for this project. KS was supported by a Cusanuswerk
Doctoral Fellowship. CSDW received generous support from the Cooperative AI
Foundation. This project received funding by Armasuisse Science+Technology.

## References

  * (1)
  * Andrychowicz et al. (2020) OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. 2020\.  Learning dexterous in-hand manipulation.  _The International Journal of Robotics Research_ (2020).
  * Box et al. (2015a) GE Box, Gwilym M Jenkins, and Gregory C Reinsel. 2015a.  i GM Ljung, Time series analysis: forecasting and control.
  * Box et al. (2015b) George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. 2015b.  _Time series analysis: forecasting and control_.  John Wiley & Sons.
  * Brockman et al. (2016) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016.  OpenAI Gym.  arXiv:arXiv:1606.01540
  * Chan (2017) Hock Peng Chan. 2017.  Optimal sequential detection in multi-stream data.  (2017).
  * Chan et al. (2019) Stephanie CY Chan, Samuel Fishman, John Canny, Anoop Korattikara, and Sergio Guadarrama. 2019.  Measuring the reliability of reinforcement learning algorithms.  _arXiv preprint arXiv:1912.05663_ (2019).
  * Chen et al. (2022) Yudong Chen, Tengyao Wang, and Richard J Samworth. 2022.  High-dimensional, multiscale online changepoint detection.  _Journal of the Royal Statistical Society Series B: Statistical Methodology_ 84, 1 (2022), 234–266.
  * Christ et al. (2018) Maximilian Christ, Nils Braun, Julius Neuffer, and Andreas W Kempa-Liehr. 2018.  Time series feature extraction on basis of scalable hypothesis tests (tsfresh–a python package).  _Neurocomputing_ 307 (2018), 72–77.
  * Danesh and Fern (2021) Mohamad H Danesh and Alan Fern. 2021.  Out-of-Distribution Dynamics Detection: RL-Relevant Benchmarks and Results.  _arXiv preprint arXiv:2107.04982_ (2021).
  * de Witt and Hornigold (2019) Christian Schroeder de Witt and Thomas Hornigold. 2019.  Stratospheric Aerosol Injection as a Deep Reinforcement Learning Problem.  https://doi.org/10.48550/arXiv.1905.07366 arXiv:1905.07366 [physics, stat].
  * Degrave et al. (2022) Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. 2022\.  Magnetic control of tokamak plasmas through deep reinforcement learning.  _Nature_ 602, 7897 (2022), 414–419.
  * Delgado-Bonal and Marshak (2019) Alfonso Delgado-Bonal and Alexander Marshak. 2019.  Approximate entropy and sample entropy: A comprehensive tutorial.  _Entropy_ 21, 6 (2019), 541.
  * Franzmeyer et al. (2022) Tim Franzmeyer, João F Henriques, Jakob N Foerster, Philip HS Torr, Adel Bibi, and Christian Schroeder de Witt. 2022.  Illusionary Attacks on Sequential Decision Makers and Countermeasures.  _arXiv preprint arXiv:2207.10170_ (2022).
  * Fujimoto et al. (2018) Scott Fujimoto, Herke Hoof, and David Meger. 2018.  Addressing function approximation error in actor-critic methods. In _International conference on machine learning_. PMLR, 1587–1596.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In _International conference on machine learning_. PMLR.
  * Haider et al. (2023) Tom Haider, Karsten Roscher, Felippe Schmoeller da Roza, and Stephan Günnemann. 2023.  Out-of-Distribution Detection for Reinforcement Learning Agents with Probabilistic Dynamics Models. In _Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems_. 851–859.
  * Kaufmann et al. (2023) Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. 2023.  Champion-Level Drone Racing Using Deep Reinforcement Learning.  _Nature_ 620, 7976 (Aug. 2023), 982–987.  https://doi.org/10.1038/s41586-023-06419-4
  * Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _Advances in neural information processing systems_ 30 (2017).
  * Liu et al. (2008) Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2008.  Isolation forest. In _2008 eighth ieee international conference on data mining_. IEEE, 413–422.
  * Liu et al. (2012) Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2012.  Isolation-based anomaly detection.  _ACM Transactions on Knowledge Discovery from Data (TKDD)_ 6, 1 (2012), 1–39.
  * Marple Jr and Carey (1989) S Lawrence Marple Jr and William M Carey. 1989.  Digital spectral analysis with applications.
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015\.  Human-level control through deep reinforcement learning.  _nature_ (2015).
  * Mohammed and Valdenegro-Toro (2021) Aaqib Parvez Mohammed and Matias Valdenegro-Toro. 2021.  Benchmark for out-of-distribution detection in deep reinforcement learning.  _arXiv preprint arXiv:2112.02694_ (2021).
  * Müller et al. (2022) Robert Müller, Steffen Illium, Thomy Phan, Tom Haider, and Claudia Linnhoff-Popien. 2022.  Towards Anomaly Detection in Reinforcement Learning. In _Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems_. 1799–1803.
  * Packer et al. (2018) Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. 2018.  Assessing generalization in deep reinforcement learning.  _arXiv preprint arXiv:1810.12282_ (2018).
  * Page (1954) E. S. Page. 1954.  Continuous Inspection Schemes.  _Biometrika_ 41, 1/2 (1954), 100–115.  https://doi.org/10.2307/2333009 Publisher: [Oxford University Press, Biometrika Trust].
  * Schulman et al. (2017a) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017a.  _Proximal Policy Optimization Algorithms_.  Technical Report. arXiv.
  * Schulman et al. (2017b) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017b.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ (2017).
  * Sedlmeier et al. (2019) Andreas Sedlmeier, Thomas Gabor, Thomy Phan, Lenz Belzner, and Claudia Linnhoff-Popien. 2019.  Uncertainty-based out-of-distribution classification in deep reinforcement learning.  _arXiv preprint arXiv:2001.00496_ (2019).
  * Teh et al. (2021) Hui Yie Teh, I Kevin, Kai Wang, and Andreas W Kempa-Liehr. 2021.  Expect the unexpected: unsupervised feature selection for automated sensor anomaly detection.  _IEEE Sensors Journal_ 21, 16 (2021), 18033–18046.
  * Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012.  MuJoCo: A physics engine for model-based control. In _2012 International Conference on Intelligent Robots and Systems_.
  * Wald (1945) A. Wald. 1945.  Sequential Tests of Statistical Hypotheses.  _The Annals of Mathematical Statistics_ 16, 2 (1945), 117–186.  https://www.jstor.org/stable/2235829 Publisher: Institute of Mathematical Statistics.
  * Yang et al. (2021) Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. 2021.  Generalized out-of-distribution detection: A survey.  _arXiv preprint arXiv:2110.11334_ (2021).

##  Appendix A Appendix

###  A.1. Novel environment definition

Algorithms 3, 4, and 5 display pseudocodes for generating ARTS, ARNO, and ARNS
environments, respectively.

Algorithm 3 Pseudocode for ARTS environment

1:Initialization at $t=0$:

2:generate AR_noise of size $(num\\_dimensions=1,max\\_steps)$

3:Agent step

4:sample action $a_{t}\sim\pi_{A}(a_{t}|s_{t})$

5:Environment step

6:function Step($a_{t}$)

7: $o_{t+1}$ $\leftarrow$ AR_noise(t)

8: return $o_{t+1}$, $r_{t}$, $terminated$

9:end function

Algorithm 4 Pseudocode for ARNO environment

1:Initialization at $t=0$:

2:generate AR_noise of size $(num\\_dimensions,max\\_steps)$

3:Agent step

4:sample action $a_{t}\sim\pi_{A}(a_{t}|s_{t})$

5:Environment step

6:function step($a_{t}$)

7: $s_{t+1}$, $r_{t}$, $terminated$ $\leftarrow$ environment.step($a_{t}$,
$s_{t}$)

8: $o_{t+1}$ $\leftarrow$ $s_{t+1}$ \+ AR_noise(t)

9: return $o_{t+1}$, $r_{t}$, $terminated$

10:end function

Algorithm 5 Pseudocode for ARNS environment

1:Initialization at $t=0$:

2:generate AR_noise of size $(num\\_dimensions,max\\_steps)$

3:Agent step

4:sample action $a_{t}\sim\pi_{A}(a_{t}|s_{t})$

5:Environment step

6:function step($a_{t}$)

7: $s^{\prime}_{t}$ $\leftarrow$ $s_{t}$ \+ AR_noise(t)

8: $s_{t+1}$, $r_{t}$, $terminated$ $\leftarrow$ environment.step($a_{t}$,
$s^{\prime}_{t}$)

9: return $s_{t+1}$, $r_{t}$, $terminated$

10:end function

###  A.2. Out-of-Distribution Detection Terminology

Müller et al. (2022) refer to anomaly detection as the general task of
identifying datapoints that substantially differ from normality in RL
environments. The authors do not use the term out-of-distribution (OOD)
detection at all. Danesh and Fern (2021) coin the term out-of-distribution
dynamics (OODD) detection to define the general task of ”detecting when the
dynamics of a temporal process change compared to the training-distribution
dynamics”. The authors differentiate between sensor-injected anomalies, which
”corrupt the environment observations received by the agent”, and dynamics-
injected anomalies, which ”directly change the dynamics of the environment by
modifying key physical parameters of the environment simulator”.

Haider et al. (2023) provide a more comprehensive review of terminology issues
in the field of OOD detection in reinforcement learning. To our knowledge,
they also propose the most recent re-definition of out-of-distribution in
reinforcement learning as ”severe perturbations of the Markov Decision
Process, which effectively change the semantics of the system”, instead of
simply changing the observations that the agent receives. The authors draw on
the aforementioned review by Yang et al. (2021) to argue that a semantic
anomaly should effectively shift the transition function of an MDP, i.e.
introduce new semantic concepts or changing the environment dynamics. In
effect, this corresponds to renaming dynamics-injected anomalies from Danesh
and Fern (2021) to semantic anomalies.

While we try to follow the existing terminologies, we believe that the most
recent definitions of OOD detection offered by Danesh and Fern (2021) and
Haider et al. (2023) could be improved for several reasons. First, Haider et
al. (2023) define OOD detection as the identification of semantic anomalies,
but do not provide a meaningful term for detecting anomalies that target the
observations that the agent receives, which has been a focus of many previous
works in the literature on OOD detection in RL (Sedlmeier et al., 2019)
(Mohammed and Valdenegro-Toro, 2021) (Danesh and Fern, 2021).

Second, in many real-world environments, reinforcement learning agents could
face both sensory and semantic anomalies simultaneously. We can take the
example of a self-driving car that is trained to drive in normal conditions,
and suddenly exposed to a heavy hailstorm. On the one hand, the ice will
change the observations that the car receives from its cameras, introducing
observational noise. On the other, the ice will also make the road more
slippery, effectively changing the environment dynamics. However, the
terminology offered by (Haider et al., 2023) does not allow to effectively
distinguish between these anomalies. We think it is reasonable to suggest
there should be a common terminology to differentiate between these two types
of anomalies in the environment, and that ideally, a well-performing detector
should be able to detect both of them.

Lastly, while the terminology of Danesh and Fern (2021) offers a distinction
between environmental anomalies, the suggested terms do not correspond to the
terminology used by OOD detection outside of reinforcement learning. As we
discuss in Section 4, it is relatively simple to align the labels for
anomalies in reinforcement learning with the standardized terminology from
Yang et al. (2021).

For this reason, Section 4 proposes a clarification of terminology for OOD
detection in reinforcement learning, which aligns it with literature from
other machine learning domains.

###  A.3. Implementation details

####  A.3.1. ARTS, ARNO, ARNS scenarios

To generate noise with varied orders of correlation, we use the implementation
of Autoregressive Process from statsmodels111More information about the
library can be found here: https://www.statsmodels.org/stable/index.html
library in Python.

####  A.3.2. ARNS Acrobot environment

We implement ARNS scenarios with Light, Medium, and Strong levels of noise on
Cartpole and Reacher, as the implementation on Acrobot leads to inconsistent
agent policies. In the implementation of Acrobot environment used in this
project (Brockman et al., 2016), adding additional noise to the underlying
states tends to increase the agent reward, which is inconsistent with our
definition of Light, Medium, and Strong noise in terms of the reduction in the
average cumulative average reward over an episode.

####  A.3.3. Changepoint detectors

To implement the changepoint detectors from Chen et al. (2022) and Chan
(2017), we use the ocd package222More information about the ocd package can be
found at: https://cran.r-project.org/web/packages/ocd/index.html for R,
published by Chen et al. (2022). The package contains the implementation for
both detectors. Both algorithms rely on the patience parameter, which is
defined as the average run length under the null hypothesis (Chen et al.,
2022). Following the definition, we set patience to the average length of the
episode under uncorrelated noise in a given setting. For detection threshold,
we use the option of automatically calculating the threshold using Monte Carlo
simulations, implemented in the ocd package.

Since this project is built on Python codebase, we also had to adapt these
detectors from R to Python. Therefore, a small contribution that we provide
is, to our knowledge, the first implementation of the algorithms of Chen et
al. (2022) and Chan (2017) in Python.

###  A.4. AUROC results

When reporting the performance of detectors using the Area under the Receiver
Operator Characteristic (AUROC), we consider a two-sided test. In other words,
the victim agent tests whether the predicted anomaly scores are either higher
or lower than the anomaly scores predicted for the unperturbed observations.
Therefore, we report the AUC score as $max(AUROC,1-AUROC)$.