# Multiple Instance Learning for Detecting Anomalies over Sequential Real-
World Datasets

Parastoo Kamranfar  George Mason University4400 University
DriveFairfaxVirginiaUSA22030<EMAIL_ADDRESS>,  David Lattanzi  George Mason
University4400 University DriveFairfaxVirginiaUSA22030<EMAIL_ADDRESS>,
Amarda Shehu  George Mason University4400 University
DriveFairfaxVirginiaUSA22030<EMAIL_ADDRESS>and  Daniel Barbará  George Mason
University4400 University DriveFairfaxVirginiaUSA22030<EMAIL_ADDRESS>
(2022)

###### Abstract.

Detecting anomalies over real-world datasets remains a challenging task. Data
annotation is an intensive human labor problem, particularly in sequential
datasets, where the start and end time of anomalies are not known. As a
result, data collected from sequential real-world processes can be largely
unlabeled or contain inaccurate labels. These characteristics challenge the
application of anomaly detection techniques based on supervised learning. In
contrast, Multiple Instance Learning (MIL) has been shown effective on
problems with incomplete knowledge of labels in the training dataset, mainly
due to the notion of bags. While largely under-leveraged for anomaly
detection, MIL provides an appealing formulation for anomaly detection over
real-world datasets, and it is the primary contribution of this paper. In this
paper, we propose an MIL-based formulation and various algorithmic
instantiations of this framework based on different design decisions for key
components of the framework. We evaluate the resulting algorithms over four
datasets that capture different physical processes along different modalities.
The experimental evaluation draws out several observations. The MIL-based
formulation performs no worse than single instance learning on easy to
moderate datasets and outperforms single-instance learning on more challenging
datasets. Altogether, the results show that the framework generalizes well
over diverse datasets resulting from different real-world application domains.

Anomaly Detection, Multiple Instance Learning, Strangeness, Outlier Detection.

††copyright: acmcopyright††journalyear: 2022††doi:
XXXXXXX.XXXXXXX††conference: ACM SIGKDD Workshops; August 14–18, 2022;
Washington, DC††booktitle: ACM SIGKDD ’22: ACM SIGKDD Workshops, August 14–18,
2022, Washington, DC††price: 15.00††isbn: 978-1-4503-XXXX-X/22/08††ccs:
Computing methodologies Machine learning algorithms††ccs: Computing
methodologies Multiple Instance-based learning

##  1\. Introduction

Anomaly detection (AD) is a well-studied problem in machine learning and seeks
to detect data points/instances that deviate from an expected behavior; the
deviating instances are known as outliers or anomalies (Chandola et al.,
2009a). Applications of AD have been explored extensively in literature and
over a variety of datasets, including data generated from sequential real-
world processes (Blázquez-García et al., 2021; Chandola et al., 2008).

Despite being well-studied, AD over real-world datasets remains challenging.
In many real-world settings, annotation of sequential data requires intensive
human labor; annotation is also made difficult by unknown or noisy start and
end times of anomalies. As a result, data collected from sequential real-world
processes are often largely unlabeled, or labels are often inaccurate. This
setting is particularly challenging for AD methods formulated under the
umbrella of supervised learning.

In contrast, Multiple Instance Learning (MIL) has been shown effective in
dealing with incomplete knowledge of labels in the training dataset. MIL was
introduced in (Dietterich et al., 1997) in the context of binary
classification as a weakly-supervised approach that reduces annotation
efforts. MIL assumes that the data points/instances are organized in sets,
also known as bags. Essentially, MIL deals with bag labels instead of
individual instance labels (Guan et al., 2016). The label of a bag
(negative/positive) is assigned according to the instance labels the bag
contains. The existence of at least one positive instance is enough for
labelling the bag as positive, while negative bags need to include all
negative instances (Feng et al., 2021b; Carbonneau et al., 2018). This
provides a way of building a more robust classifier than what can be built by
relying exclusively on single-instance labels.

MIL has received much attention in the machine learning community in the past
two decades, and various techniques have been developed to exploit the
structure of the data to enhance performance in a variety of applications
(Quellec et al., 2016; Zhang and Viola, 2007; Andrews et al., 2002; Alam and
Shehu, 2020; Zhang et al., 2021b; Alam and Shehu, 2021). In particular,
computer vision, where videos are naturally structured into bags and require
significant single-instance labeling efforts, has benefited tremendously from
MIL-based methods. Application of MIL for AD remains largely under-leveraged
(Amores, 2013; Foulds and Frank, 2010; Yang et al., 2013). Yet, MIL provides
an appealing formulation for semi-supervised AD over real-world datasets, and
it is the primary contribution of this paper.

In this paper, we propose an MIL-based formulation and propose various
algorithmic instantiations of this framework based on different design
decisions for key components of the framework. In particular, we leverage the
combination of MIL and the Strangeness based OUtlier Detection (StrOUD)
algorithm (Barbará et al., 2006). StrOUD computes a strangeness/anomaly value
for each data point and detects outliers by means of statistical testing and
calculation of p-value. Thus, the framework is dependent on two primary design
decisions: definition of the strangeness factor and the aggregation function.
The degree of outlying/strangeness is needed for recognizing anomalous data
points, while the aggregate function is needed to aggregate the measures of
strangeness into a single anomaly score of a bag. In this paper, we utilize
two alternative scores, the ’Local Outlier Factor’ and the Autoencoder (AE)
reconstruction error. We utilize six aggregate functions (minimum, maximum,
average, median, spread, dspread) for the strangeness measure of a bag. We
note that we are not limited to these design decisions, and they can be easily
generalized to any other choices. To the best of our knowledge, the effort we
describe here is the first to truly leverage the power of MIL in AD by
hybridizing the concept of bags naturally with AD methods that are routinely
used in the field.

We evaluate various algorithmic instantiations of the MIL framework over four
datasets that capture different physical processes along different modalities
(video and vibration signals). The results show that the MIL-based formulation
performs no worse than single instance learning on easy to moderate datasets
and outperforms single-instance learning on more challenging datasets.
Altogether, the results show that the framework generalizes well over diverse
benchmark datasets resulting from different real-world application domains.

The rest of the paper is organized as follows. Section 2 relates prior work in
AD. The proposed methodology is described in Section 3, and the experimental
evaluation is related in Section 4. Section 5 concludes the paper.

##  2\. Prior Work

Extensive studies have been performed on AD in many application domains, from
fraud detection in credit cards, to structural health monitoring in
engineering, to bioinformatics in molecular biology (Kou et al., 2004; Zhang
et al., 2021a; Alam and Shehu, 2021). Different types of data have been
considered, from sequential time series data (Blázquez-García et al., 2021;
Chandola et al., 2009b), to image data, to molecular structure data (da Costa
et al., 2020; Zenati et al., 2018; Alam and Shehu, 2021).

Many methods have been developed for AD, varying from traditional density-
based methods (Idé et al., 2007; Tang et al., 2002) to more recent AE-based
ones (Chen et al., 2018; Babaei et al., 2021). Density-based methods assign an
anomaly score to a single data point/instance by comparing the local
neighborhood of a point to the local neighborhoods of its $k$ nearest
neighbours. Higher scores are indicative of anomalous instances.

The Local Outlier Factor (LOF) and its variants are density-based anomaly
scores that are utilized extensively in AD literature (Hautamaki et al., 2004;
Ahmed et al., 2017). For instance, work in (Tang et al., 2002) introduced an
LOF variant score called ’Connectivity based Outlier Factor’ (COF) which
differs from LOF in the way that the neighborhood of an instance is computed.
’Outlier Detection using In-degree Number’ (ODIN) score has been presented in
(Hautamaki et al., 2004). ODIN measures the number $k$ of nearest neighbors of
a data point which also have that data point in their neighborhood. The
inverse of ODIN is defined as the anomaly score.

There are other methods, known as deviation-based methods, that also utilize
anomaly scores. These methods attempt to find a lower-dimensional space of
normal data by capturing the correlation among the features. The data are
projected onto a latent, lower-dimensional subspace, and unseen test data
points with large reconstruction errors are determined to be anomalies. As
these methods only encounter normal data in the training phase that seeks to
learn the latent space, they are known as semi-supervised methods.

PCA- and AE-based methods are in the category of AD methods that seek to
capture linear and non-linear feature correlations, respectively (Shyu et al.,
2003; Chen et al., 2018). Due to the capability of AEs in finding more
complex, non-linear correlations, AE-based AD methods tends to perform better
than PCA-based ones, generating fewer false anomalies.

In (Chen et al., 2018), conventional and convolutional AE-based (CAE) methods
are presented and compared with PCA-based methods. Work in (An and Cho, 2015)
proposes a variational AE-based (VAE) method which takes advantage of the
probabilistic nature of VAE. The method leverages the reconstruction
probability instead of the reconstruction error as the anomaly score. AE-based
methods, however, require setting a threshold for how large the reconstruction
error or reconstruction probability has to be for an instance to be predicted
as anomalous.

Generally, most AD methods require careful setting of many parameters,
including the anomaly threshold, which is an ad-hoc process. This
hyperparameter regulates sensitivity to anomalies and the false alarm rate
(rate of normal instances detected as outliers/anomalies), and is the
indicator of AD performance. To maintain a low false alarm rate, conformal AD
(CAD) methods build on the conformal prediction (CFD) concept (Laxhammar and
Falkman, 2011).

The underlying idea in CFD methods is to predict potential labels for each
test data point by means of the p-value (one p-value per possible label). The
non-conformity measure is utilized as an anomaly score, and p-values are
calculated To this end, the significance level needs to be determined in order
to retain or reject the null hypothesis (Laxhammar and Falkman, 2015).

In their utilization of significance testing to avoid overfitting, CFD methods
are highly similar to the classic Strangeness based OUtlier Detection (StrOUD)
method (Barbará et al., 2006). StrOUD was proposed as an AD method that
combines the ideas of transduction and hypothesis testing. It eliminates the
need for anomaly ad-hoc thresholds and can additionally be used for dataset
cleaning.

Apart from determination of the anomaly score or the AD category, there are
many fundamental challenges in AD, not the least of which is finding
appropriate training instances. It is generally easier to obtain instances
from the normal behaviour of a system rather than anomalies, especially in
real-world settings (i.e. large engineering infrastructures or industrial
systems) (Villa-Pérez et al., 2021), where anomalous physical processes that
generate anomalous data are rare events.

As a result, there is strong potential for weakly-supervised techniques such
as MIL for AD. MIL has been thoroughly explored for classification problems in
a variety of applications, such as face detection (Zhang and Viola, 2007),
text categorization (Andrews et al., 2002), and speech context classification
(Zhang et al., 2021b), but it has been largely under-explored for AD. A local
AD method, TRASMIL, has been proposed in (Yang et al., 2013) for abnormal
event detection in video clips. The method was constructed on two main
concepts: trajectory segmentation and MIL. Trajectories are defined as bags,
and they are partitioned into sub-trajectories to form the instances. Another
connection between MIL and AD in machine learning literature is work in
(Quellec et al., 2016), which employs MIL on medical images for the purpose of
beast cancer diagnosis in mammography screening. In this work, each image is
considered as a group of regions, and features are extracted from each region.
A bag is composed of regions from four images, and labels are assigned to
mammography examinations instead of individual images. Recent works in video
surveillance (Sultani et al., 2018; Feng et al., 2021a) employ MIL to perform
anomaly detection. However, they do so in the context of weakly supervised
anomaly detection, assuming some anomalous labels are available. Our approach
is completely unsupervised, since we wish to apply it to scenarios where such
labels are not available.

In this paper we leverage MIL due to its appealing formulation for semi-
supervised AD over real-world datasets. Furthermore, to avoid the
determination of an anomaly threshold, our MIL framework builds on the StrOUD
method. The LOF and AE reconstruction error are considered as alternative
anomaly scores. Various aggregation functions are considered, and in turn
various algorithmic instantiations are obtained, effectively resulting in an
ablation study. Methodological details now follow.

##  3\. Methodology

As we pointed out in Section 1, the proposed framework adapts the ideas of MIL
and StrOUD to detect anomalous test bags. We first summarize key ingredients
of MIL and StrOUD to set the stage for how we hybridize them in a novel MIL-
StrOUD framework.

StrOUD was initially presented in (Barbará et al., 2006) for the purpose of
finding outliers in a pre-defined clustering setting. It utilizes the concept
of transduction to directly make estimates for each data point within the
training/reference set and so reason from these cases to the test set. In
essence, the StrOUD algorithm places an unknown data point in a known data
distribution (the baseline), and determines the fitness of the point by
carrying out statistical testing.

In evidential studies, a null hypothesis is formulated first, and then a
statistical testing is conducted to retain or reject the null hypothesis. The
p-value as the outcome of such statistical testing is a probability value used
as a measurement of evidence against the null hypothesis. The smaller the
p-value, the stronger the evidence to reject the null hypothesis (Du Prel et
al., 2009). In statistics, by making the assumption that the null hypothesis
is correct, the p-value is defined as the probability of observing results
that can be considered at least as extreme as the observed results from
statistical testing. To this end, StrOUD takes advantage of a strangeness
value for every point of interest and utilizes the p-value significance
testing formulation to support or reject a null hypothesis (i.e. a point $i$
belongs to baseline/normal distribution).

MIL on the other hand assumes that data points are organized in sets called
bags (Carbonneau et al., 2018). MIL can be categorized as weakly-supervised
binary classification; instead of dealing with labels of individual instances
within a bag, it works with negative or positive labels that are assigned to
each bag (Feng et al., 2021b; Carbonneau et al., 2018). This provides a more
robust classifier than the one which is built by relying exclusively on
instance labels and relaxes the dependency on individual instance diagnosis. A
classic MIL assumption is that a negative bag only contains negative instances
(normal), while a positive bag contains at least one positive instance
(anomalous) known as a witness. The bag classifier, $f(X)$, is defined as:

(1) $f(X)=\begin{cases}1,&if\exists{b\in B:g(x)=1}\\\
0,&\text{otherwise}.\end{cases}$

where $b$ and $B$ denotes an instance and a bag, respectively.
$g:X\rightarrow\\{0,1\\}$ is a mapping process that corresponds negative and
positive instances to the $0$ and $1$ classes (Carbonneau et al., 2018). Due
to the goal of prediction at the bag-level, there is no need to identify all
witnesses in a bag, and existence of one suffices to label a bag as positive
(label of ’1’).

The MIL-StrOUD approach we propose in this paper is shown in pseudocode in
Algorithms 1 to 3. It uses the same strategy as MIL classification or AE-based
anomaly detection, where data needs to be split into training and testing
sets. We note that in the MIL setting training includes only normal data,
while the testing set contains both normal data and anomalies.

There is no constraint on the number of bags or the number of instances inside
a bag, but we know that bag composition (the number of instances within a bag)
has an impact on the performance of MIL-based approaches (Carbonneau et al.,
2018). There is no standard/general solution on how to organize bags in MIL
literature. Bag size is typically determined empirically for a problem/data at
hand (Guan et al., 2016). In our problem, by reasoning on data and attempting
to achieve a balanced number of positive and negative bags, we define the
number of instances in bags for each dataset. This process is explained in
section 4.

1:function CreateBaseline(training bags ${rb_{1},rb_{2},...,rb_{k}}$)

2: for $\forall rb_{i}\in X_{train}$ do

3: $AnomalyScore_{rb_{i}}\leftarrow FindAnomalyScore({rb_{i}})$

4: $AnomalyScore_{X_{train}}\leftarrow Append(AnomalyScore_{rb_{i}})$

5: end for

6: $OrderedAnomalyList_{X_{train}}\leftarrow
Sort(AnomalyScore_{X_{train}},ascending)$ return
$OrderedAnomalyList_{X_{train}}$

7:end function

Algorithm 1 Create Baseline Anomaly Distribution

1:function FindQueryBagAnomaly(query bag ${qb}$, baseline anomaly
${OrderedAnomalyList_{X_{train}}}$)

2: $prediction_{qb}\leftarrow 0$

3: $AnomalyScore_{qb}\leftarrow FindAnomalyScore({qb})$

4: $index_{qb}\leftarrow$index of $AnomalyScore_{qb}$ in
$OrderedAnomalyList_{X_{train}}$

5: $p-value_{qb}\leftarrow\frac{1+n_{x}-index_{qb}}{|n_{x}|+1}$

6: for $\forall c_{i}\in ConfidenceRange$ do

7: $\tau\leftarrow 1-c_{i}$

8: if $p_{qb}<=\tau$ then

9: $prediction_{qb}\leftarrow 1$

10: end if

11: $Compare(GroundTruth_{qb},Prediction_{qb})$

12: end for

13:return

14:end function

Algorithm 2 Find Anomaly of a Query Bag

1:function FindAnomalyScore(bag ${b}$)

2: for $\forall x_{i}\in b$ do

3: $AnomalyScore_{x_{i}}\leftarrow LOF(b,x_{i})$ or $MSE(b,x_{i})$

4: $AnomalyScoreList_{b}\leftarrow Append(AnomalyScore_{x_{i}})$

5: end for

6: $AnomalyScore_{b}\leftarrow Aggregate(AnomalyScoreList_{b},Function)$
return $AnomalyScore_{b}$

7:end function

Algorithm 3 Find Bag Anomaly Score

Algorithm 1 creates a baseline anomaly distribution from training bags which
all are normal. For this purpose, the anomaly score of each bag needs to be
computed first. This is what algorithm 3 carries out. Once the values are
computed, all anomaly scores of the reference bags
(${rb_{1},rb_{2},...,rb_{k}}$) are merged, and a list containing strangeness
values of training set (${AnomalyScore_{X_{train}}}$) is then created (line
$2$–$5$). The final step is to order these anomaly measurements and return the
baseline list which is needed for the p-value computation of a query/test bag.

Algorithm 2 predicts the label of a query bag ($qb$) by means of the baseline
strangeness distribution calculated in Algorithm 1. The function starts with
the assumption that $qb$ is ’normal’ (line $2$). Like the reference bags, the
anomaly score of $qb$ should be computed by calling the function in 3. Once
the strangeness of the bag is computed through the aggregate function, it can
be compared to a sample distribution of normal bags (baseline distribution)
whose strangeness is computed using the same aggregate function.

To predict the label of $qb$, we utilize the hypothesis testing part of the
StrOUD algorithm. Thus, we need to find the appropriate index of $qb$, where
it should be inserted to maintain the order of the baseline (line $4$). The
next step is to determine the $p-value$ of the query bag by considering a
fraction of reference bags whose anomaly scores are equal or greater than that
of $qb$ (line $5$). This probability value is used for statistical analysis,
where the null hypothesis is that the $qb$ belongs to normal distribution.

Therefore, the next block (lines $6$–$12$) is designed to support or reject
the null hypothesis by defining a confidence level $c_{i}$. If the $p-value$
is less than the confidence threshold, we reject the null hypothesis and the
bag is predicted as ’anomalous.’ The prediction evaluation is done by
comparing with the ground truth (line $11$).

Algorithm 2 is given for a single test bag but needs to be repeated for all
testing bags. To better evaluate the results, we report the area under the
receiver-operating-characteristic (ROC) curve (AUC) instead of accuracy (Guan
et al., 2016). A wide range of confidence levels (from $0$ to $1$ with $0.001$
step) are examined, and AUC values are calculated by means of
$FalsePositiveRate$ and $TruePositiveRate$.

Many MIL-based algorithms used for classification of bags solve the problem by
predicting the class of each participating instance in a bag as a side
feature. We follow a similar procedure by first computing the degree of
outlying/strangeness for each individual instance inside a bag. Algorithm 3 is
designed to do so. As related earlier, we consider two different approaches to
compute the strangeness factor for an instance, LOF or MSE (lines $2$–$3$).
LOF or MSE constitute the first important design decision of our proposed MIL-
StrouD framework.

The strangeness factor determination is the first design decision of the
framework we propose. From all these strangeness scores, a single measure of a
bag should be obtained (line $6$). We propose the use of an aggregate function
$f_{a}(\bar{s})$, where the $\bar{s}$ is the vector of strangeness of
instances, namely ($s_{1},s_{2},...,s_{k}$) and $k$ is the number of instances
in a bag. There are many choices for $f_{a}(\bar{s})$, including, but not
limited to, the maximum, minimum, average, median, spread, and dspread
functions. The aggregation function is the second design decision of our
proposed MIL-StrOUD framework.

Details of the design choices are elaborated as follows.

###  3.1. I. Strangeness Factor

####  3.1.1. Local Outlier Factor (LOF)

LOF is known as a density-based anomaly detection factor that captures
relative degree of isolation for each data point. In summary, LOF detects
local outliers with respect to their neighborhood and with the assumption that
outliers are far away from inliners located in a dense region of the
neighborhood. This value is computed by determination of a _MinPts_
hyperparameter identifying the expected number of neighbors around each data
point/instance (Breunig et al., 2000). A low LOF score is an indication of an
inliner, while a high value shows an anomalous point. After computing the
proximity matrix from all training/reference and test points to all reference
points, as explained in (Babaei et al., 2021), the LOF calculation process is
performed in four steps.

First, the k-nearest distances (_k-distance_) for all reference and test
points to every reference point are computed.

Second, the reachability distance (RD) is then calculated as:

(2) $RD(a,b)=max\\{k-distance(b),Euclidean-dist(a,b)\\}$

which depends on the query point location in the neighborhood, the output is
either _k-distance_ or the actual Euclidean distance between two points.

In the third step, the Local Reachability Distance (LRD) needs to be
calculated. LRD can be interpreted as an estimated distance at which a point
can be found by its neighbors. This value is computed by inverting the average
reachability distance of a point from its neighbors

(3) $LRD(a)=\frac{N_{MinPts(a)}}{\sum_{b\in{N_{MinPtns(a)}}}RD(a,b)}$

where $N_{MinPts}$ is the LOF hyperparameter which is the number of neighbors
in data point neighborhood.

In the fourth and last step, the LRD ratio of the point and its neighbors are
measured to determine if the point is close to its neighbors to be labelled as
an inlier or is far away to be labelled as anomaly.

(4) $LOF(a)=\frac{\sum_{b\in
N_{MinPts(a)}}\frac{LRD(b)}{LRD(a)}}{N_{MinPts(a)}}$

####  3.1.2. Autoencoder (AE) Mean Reconstruction Error (MSE)

An AE is a neural network composed of two parts, an encoder and decoder. The
encoder maps an input ($x$) to the latent space ($h$), while the decoder
reconstructs the original input from the hidden representation by the same
transformation used in encoder ($z$) (Tschannen et al., 2018). The
reconstruction error defines as the difference between the original $x$ and
reconstructed $z$, and an AE learns to minimize this error in training phase.
An encoder and decoder equations of a single hidden layer AE are given in 5
(An and Cho, 2015).

(5) $h={g(W_{e}x+b_{e})}\qquad z={g(W_{d}x+b_{d})}$

where $g$ is a nonlinear transformation function, $W_{e}$ and $W_{d}$ are
encoder and decoder weights, while $b_{e}$ and $b_{d}$ are network biases.

We employ the same idea as AE-based anomaly detection which utilizes the
reconstruction error as strangeness factor. The underlying assumption is that,
as the AE is trained only on normal data, normal points in the test set ought
to be reconstructed well but not anomalous data. The higher the reconstruction
error, the higher an anomaly score of a point.

We empirically select the AE architectures for each dataset. In all four
architectures, three dense hidden layers along with two dropouts are utilized.
An $80-20$ split is followed to split the training dataset into training and
validation sets. We use the _tanh_ activation function in the first hidden
layer and two _ReLu_ s in the other two hidden layers. In general, the number
of neural units are gradually dropped in each layer until the desired latent
dimensions are reached. Different loss functions are also examined for each
dataset. Furthermore, the number of training epochs is determined by plotting
training versus validation loss over the number of epochs. We have explored
many variations of this fundamental architecture, varying the number of hidden
layer, the layer width (number of neurons), the activation functions, and the
number of latent dimensions (data not shown). We adopt the notation
$x\rightarrow L1\rightarrow L2\rightarrow y$ to denote the encoder
architecture, with the decoder being a mirror image. We have experimented with
$|L_{1}|\in[100,512]$, $|L_{2}|\in[25,128]$, and $|y|\in[5,32]$.

###  3.2. II. Aggregate Function

We define below the six different aggregate functions we utilize for different
instantiations of our framework:

  * •

The maximum strangeness, that is $f_{a}(\bar{s})=max(s_{1},s_{2},...,s_{k})$

  * •

The minimum strangeness, that is $f_{a}(\bar{s})=min(s_{1},s_{2},...,s_{k})$

  * •

The average strangeness, that is $f_{a}(\bar{s})=mean(s_{1},s_{2},...,s_{k})$

  * •

The median strangeness, that is $f_{a}(\bar{s})=median(s_{1},s_{2},...,s_{k})$

  * •

The dspread strangeness, that is
$f_{a}(\bar{s})=mean(s_{1},s_{2},...,s_{k})+2*stdev(s_{1},s_{2},...,s_{k})$

  * •

The spread strangeness, that is
$f_{a}(\bar{s})=mean(s_{1},s_{2},...,s_{k})*stdev(s_{1},s_{2},...,s_{k})$

In all the above, $s_{1},s_{2},\ldots,s_{k}$ are samples/instances in a bag
$\bar{s}$. We allow for these various aggregation functions in our
experimental evaluation, but we do expect that the maximum strangeness will
confer better performance. The reason for this is tied to the definition of an
anomalous bag. As described above, we adopt the definition of a bag as
anomalous as long as there is at least one anomalous sample in it. It is worth
noting, however, that this definition is conventionally employed in
applications of MIL for classification (even though this restriction is
relaxed in cases where positive bags cannot be identified by a single instance
but by the accumulation of several of them (Carbonneau et al., 2018)), and we
adopt it here for MIL for AD, as a first step. Future research can consider
alternative definitions.

##  4\. Results

We first provide a brief summary of each of the four datasets before relating
the experimental evaluation.

###  4.1. Datasets

_I. Case Western Reserve University (CWRU)_ : The CWRU bearing dataset is a
standardized dataset for pattern classification with mechanical systems. It
has known ground-truth labels associated with five motor-bearing condition
states (one ’normal’ versus four ’damaged’). More details about this dataset
can be found in (Yang et al., 2018). All four damaged states’ measurements are
considered as anomalies. The data contains $1853$ instances; $300$ normal
instances are utilized for the training dataset, while the rest constitute the
testing dataset. We divide the data into $30$ normal bags and $35$ anomalous
bags in order to have a balanced number of normal and anomalous test bags.

_II. Smartphone-Based Recognition of Human Activities and Postural Transitions
Data Set (HAPT):_ HAPT is a publicly-labelled dataset obtained from basic
static, dynamic activities, and postural transitions performed by $30$
volunteers, captured by a smartphone, and pre-processed by applying filters
and using fixed-length sliding window with overlap. Each data instance is a
$561$-featured vector containing time and frequency domain variables. More
details can be found in the UCI repository (Dua and Graff, 2017). Here, we
define postural transition instances ($518/10,929$ of the data instances) as
anomalous activities. To balance the number of bags, $104$ training bags are
extracted from $5,200$ normal instances (static and dynamic activities,) while
$104$ test bags are formed from the remaining $5,211$ data points for test
set; $518$ outliers are randomly assigned to half of the test bags to generate
$52$ normal and anomalous bags. Each bag contains at least $50$ instances.

_III. Virat Video Dataset:_ Virat is a well-known, public video surveillance
dataset. It contains data collected from multiple outdoor scenes showing
people doing normal activities. Typical uses of the dataset are for activity
recognition; more details can be found in (Oh et al., 2011). We consider one
out of $16$ available scenes. There are multiple video clips from a scene, and
each clip can contain from zero to multiple activities. Normal instances are
defined as those with normal human activities. Video clips that show humans
with objects, such as scooters or bicycles are defined as anomalies. In this
setting, the video clips are natural bags; we note that in this dataset, there
are no single-instance labels but only clip/bag labels. Our training and test
datasets contain $2,000$ images each, organized into $24$ and $34$ bags,
respectively. We utilize the pre-trained InceptionV3 model on ImageNet
(Szegedy et al., 2016) for feature extraction. By adding an extra layer of
_GlobalMaxPooling_ , we obtain $2048$ features to encode each image/instance.

_IV. Bridge Ambient Vibration Dataset_ This dataset was collected from the Old
ADA bridge in Japan over four different bridge states (ground-truth labels) of
the bridge: 1) INT: no damage; 2) DMG1: half cut of one vertical member truss
at the bridge midspan; 3) DMG2: fully cut of the same member; 4) DMG3: fully
cut of one vertical member at the bridge 5/8th span. The experiment was
conducted at 200 Hz sampling rate. Depending on a damage state, the test was
repeated for a varying number of times. More details about the experiment can
be found in (Kim et al., 2021). Signals are segmented into $1$ second of
measurements to generate the instances. We have included all three INT test
repetitions in our analysis; the first and third are considered for training,
and the second is reserved for testing. $30$ seconds of measurements from the
normal and $60$ seconds of measurements from the three damaged states are
considered in our bag composition, leading to $622$ ($21$ training bags) and
$1,433$ data instances in the training and test sets ($14$ normal vs. $17$
anomalous testing bags), respectively.

###  4.2. Experimental Results

We relate our experiments on each of the four datasets in order. In each, we
compare the AUC obtained by MIL versus StrOUD; as described in Section 3, the
latter is a natural choice for single-instance learning, given our leverage of
the LOF in one instantiation of MIL. We consolidate the comparisons of the
MIL-derived algorithms to compare the incorporation of LOF versus MSE (by
autoencoder). In the interest of space, we suppress the varying instantiations
based on the aggregation function, and only note when presented results are
representative or in contrast indicative of superior performance due to any
aggregation function in particular. We note that the number of nearest
neighbors needs to be specified when using LOF, but the aggregate function is
only needed for MIL due to its bag-level prediction. For all datasets, the
same aggregate functions are employed, but the number of neighbors depends on
the number of instances in a bag. So, this value is related for each dataset
separately. When MSE is considered, the best autoencoder architecture and
model are identified empirically for each dataset.

####  4.2.1. Comparative Evaluation on the CWRU Dataset

CWRU is considered a clean dataset. We employ it here as a baseline or control
setting. Due to the existence of at least $10$ instances in each bag, a range
of $2$ to $10$ number of nearest neighbors is examined for the LOF computation
in the MIL-StrOUD setting; in StrOUD, where there is no such a limitation, a
larger range of values from $2$ to $50$ is considered. In the encoder
architecture employed for MIL-StrOUD with MSE, $|x|=400$, $|L_{1}|=200$,
$|L_{2}|=50$, and $|y|=10$.

The AUC comparison is related in Table 1. It is evident that no matter what
anomaly score is employed, both algorithms achieve the highest AUC; we do not
repeat the results but instead relate LOF/MSE in Table 1. The results confirm
that anomalies behave differently enough from normal data in the CWRU dataset
that they can be distinguished even with single-instance learning. For LOF,
the results achieved with the _maximum_ and _dspread_ aggregates are the same,
while for MSE, _maximum_ and _spread_ aggregates achieved the top performance.

Table 1. Results on the CWRU Dataset. Method | Strangeness Factor | Best AUC
---|---|---
MIL-StrOUD | LOF/MSE | $1.0$
StrOUD | LOF/MSE | $1.0$

####  4.2.2. Comparative Evaluation on the HAPT Dataset

We consider a similar range of number of neighbors for the MIL setting on the
HAPT dataset ($2$ to $10$), but a larger range of neighbours for StrOUD ($2$
to $70$), as the HAPT dataset contains more instances than the CWRU dataset.
We note that in the encoder architecture employed for MIL-StrOUD with MSE,
$|x|=561$, $|L_{2}|=280$, $|L_{3}|=35$, and $|y|=5$.

Table 2 relates the comparative evaluation. As HAPT is a clean benchmark
dataset, the AUCs of the MIL-based algorithms and StrOUD are close. This
indicates that the extracted features are meaningful enough to distinguish
postural transitions from other activities, and that labels are accurately
assigned to the instances. We do observe MIL being able to produce slightly
better AUCs than StrOUD using either LOF or MSE score. Fig. 1, which shows the
ROC curves for MIL-StrOUD with LOF (top panel) or MSE (bottom panel)
additionally relates the high performance of MIL for this dataset.

Table 2. Results on the HAPT Dataset. Method | Strangeness Factor | Best AUC
---|---|---
MIL-StrOUD | LOF | $0.996$
MSE | $0.994$
StrOUD | LOF | $0.990$
MSE | $0.980$
MIL-StrOUD with LOF
---
MIL-StrOUD with MSE
Figure 1. ROC curves are related for MIl-StrOUD with LOF (top panel) and MSE
(bottom panel) for the HAPT dataset. These correspond to the setting that
confers the highest performance, related in Table 2.

In general, we observe that the results related above are not overly sensitive
to the number of nearest neighbors; that is, the AUC varies in a narrow range
from $0.964$ to $0.996$, achieving the best value for $NN=2$. The _maximum_
aggregate function is the one conferring the highest performance to MIL,
demonstrating that the assumption of a single anomaly in a bag is sufficient
for determining the bag to be anomalous holds for this dataset.

####  4.2.3. Comparative Evaluation on the Virat Dataset

We assign bag labels manually based on the objects of the scene in the VIRAT
dataset. However, instance-level labeling cannot be carried out for this
dataset, as such information is not available, and the process is too
laborious. The performance of the MIL algorithms on the Virat dataset is
related in Table 3. The comparison shows higher performance by LOF over MSE.
Fig. 2, which shows the ROC curves for MIL-StrOUD with LOF (top panel) or MSE
(bottom panel) supports these results.

We note that in the encoder architecture employed for MIL-StrOUD with MSE,
$|x|=2048$, $|L_{2}|=512$, $|L_{3}|=128$, and $|y|=32$. We observe that the
results are mildly sensitive to the number of nearest neighbors; that is,
there is a wide range of the number of nearest neighbors where the performance
ranges from $0.897$ to $0.937$, reaching the latter at $NN=3$. We again
observe the _maximum_ aggregate function as the one conferring the highest
performance to MIL, as in the previous dataset.

Table 3. Results on the Virat Dataset. Method | Strangeness Factor | Best AUC
---|---|---
MIL-StrOUD | LOF | $0.937$
MSE | $0.911$
MIL-StrOUD with LOF
---
MIL-StrOUD with MSE
Figure 2. ROC curves are related for MIl-StrOUD with LOF (top panel) and MSE
(bottom panel) for the VIRAT dataset. These correspond to the setting that
confers the highest performance, related in Table 3.

####  4.2.4. Comparative Evaluation on the Bridge Dataset

According to the minimum number of instances in each bag ($30$), a range of
$2$ to $20$ neighbours is examined for MIL. This value ranges from $2$ to $50$
for StrOUD. The encoder architecture employed for MIL-StrOUD with MSE,
$|x|=200$, $|L_{2}|=100$, $|L_{3}|=25$, and $|y|=5$.

Table 4 relates the comparison of MIL-based algorithms to StrOUD. It is
clearly evident that using MIL improves the results significantly over StrOUD.
Fig. 2 relates the ROC curves for MIL-StrOUD with LOF (top panel) or MSE
(bottom panel). We observe that the results on the Bridge dataset are
moderately sensitive to the number of nearest neighbors; that is, for a wide
range of values, the AUC ranges from $0.895$ to $0.931$ at $NN=2$. Again, the
_maximum_ aggregate function is the best choice.

Table 4. Results on the Bridge Dataset. Method | Strangeness Factor | Best AUC
---|---|---
MIL-StrOUD | LOF | $0.931$
MSE | $0.905$
StrOUD | LOF | $0.720$
MSE | $0.729$
MIL-StrOUD with LOF
---
MIL-StrOUD with MSE
Figure 3. ROC curves are related for MIl-StrOUD with LOF (top panel) and MSE
(bottom panel) for the Bridge dataset. These correspond to the setting that
confers the highest performance, related in Table 4.

In this dataset, there is a significant difference in performance between
StrOUD and MIL. The lower AUC values obtained by StrOUD indicate that,
although the measurements are labeled according to the bridge health status,
they are likely not accurately assigned. In fact, our detailed analysis
reveals that even though the bridge has been damaged, most of the anomalous
vibrations resemble normal segments. We relate here one dimension of this
analysis that supports this characterization. We train an AE on the normal
instances of the bridge data, and then relate reconstruction errors of normal
versus damaged samples. The top panel of Fig. 4 shows the histogram of
reconstruction errors over the training dataset. The bottom panel shows the
histogram of reconstruction errors over the testing dataset. For the latter,
the reconstruction errors of samples labeled as damaged are superimposed over
those of samples labeled as normal. Fig. 5 relates a similar analysis but on
the LOFs.

---
Figure 4. The top panel shows the histogram of reconstruction errors over the
training dataset for an AE trained over this dataset. The bottom panel shows
the histogram of AE reconstruction errors over the testing dataset. The
reconstruction errors of damaged samples are superimposed over those of normal
samples in the testing dataset.
---
Figure 5. The top panel shows the histogram of LOFs over the training dataset
for an AE trained over this dataset. The bottom panel shows the histogram of
LOFs over the testing dataset. The LOFs of damaged samples are superimposed
over those of normal samples in the testing dataset.

Fig. 4-5 clearly show that it is hard to distinguish normal from
damaged/anomalous samples. In fact, this analysis additionally makes the case
why threshold-based AD methods, whether they utilize LOF or AE reconstruction
error are not effective for real-world data. This particular dataset
demonstrates that the location, spatial orientation, and proximity of a sensor
to the defect will lead to very different signal response. Additionally, a
sensor will not continuously respond the same way to a defect.

It is very clear from the Bridge dataset that, when we are dealing with raw
real-world or manually-labeled datasets, evaluation becomes a challenging task
due to very complex physical responses, and sometimes the uncertainty about
label accuracy. However, such an issue is alleviated under the MIL
formulation, as MIL depends on group rather than single-instance labels.

##  5\. Conclusion

Though largely under-leveraged for AD, we present in this paper an MIL
formulation for AD and pursue an empirical evaluation of various algorithmic
instantiations under this formulation. We evaluate the resulting algorithms
over four different datasets that capture different physical processes along
different modalities. The results show the superiority of the MIL-based
formulation over single instance learning, as well as the ability of the MIL
framework to generalize well over diverse benchmark datasets resulting from
different real-world application domains. While in this paper we have focused
the comparison against StrOUD, effectively carrying out an ablation study,
future work will consider expanding comparisons to other AD algorithms to
obtain a better understanding of their relationship with real-world datasets
with varying label accuracy confidence. Future work will also leverage the
versatility of the MIL framework and consider different strangeness score
formulations.

## References

  * (1)
  * Ahmed et al. (2017) Mohiuddin Ahmed, Nazim Choudhury, and Shahadat Uddin. 2017\.  Anomaly detection on big data in financial markets. In _2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)_. IEEE, 998–1001.
  * Alam and Shehu (2020) F. F. Alam and A. Shehu. 2020.  From Unsupervised Multi-Instance Learning to Identification of Near-Native Protein Structures. In _Intl Conf on Bioinf and Comput Biol_ , Vol. 70. 59–68.
  * Alam and Shehu (2021) F. F. Alam and A. Shehu. 2021.  Unsupervised Multi-Instance Learning for Protein Structure Determination.  _J Bioinf & Comput Biol_ 19, 1 (2021), 2140002\.
  * Amores (2013) Jaume Amores. 2013\.  Multiple instance classification: Review, taxonomy and comparative study.  _Artificial intelligence_ 201 (2013), 81–105.
  * An and Cho (2015) Jinwon An and Sungzoon Cho. 2015.  Variational autoencoder based anomaly detection using reconstruction probability.  _Special Lecture on IE_ 2, 1 (2015), 1–18.
  * Andrews et al. (2002) Stuart Andrews, Ioannis Tsochantaridis, and Thomas Hofmann. 2002\.  Support vector machines for multiple-instance learning.  _Advances in neural information processing systems_ 15 (2002).
  * Babaei et al. (2021) Kasra Babaei, Zhi Yuan Chen, and Tomas Maul. 2021.  AEGR: A simple approach to gradient reversal in autoencoders for network anomaly detection.  _Soft Computing_ 25, 24 (2021), 15269–15280.
  * Barbará et al. (2006) Daniel Barbará, Carlotta Domeniconi, and James P Rogers. 2006.  Detecting outliers using transduction and statistical testing. In _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_. 55–64.
  * Blázquez-García et al. (2021) Ane Blázquez-García, Angel Conde, Usue Mori, and Jose A Lozano. 2021.  A review on outlier/anomaly detection in time series data.  _ACM Computing Surveys (CSUR)_ 54, 3 (2021), 1–33.
  * Breunig et al. (2000) Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. 2000.  LOF: identifying density-based local outliers. In _Proceedings of the 2000 ACM SIGMOD international conference on Management of data_. 93–104.
  * Carbonneau et al. (2018) Marc-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. 2018.  Multiple instance learning: A survey of problem characteristics and applications.  _Pattern Recognition_ 77 (2018), 329–353.
  * Chandola et al. (2009a) Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009a.  Anomaly detection: A survey.  _ACM computing surveys (CSUR)_ 41, 3 (2009), 1–58.
  * Chandola et al. (2009b) Varun Chandola, Deepthi Cheboli, and Vipin Kumar. 2009b.  Detecting anomalies in a time series database.  (2009).
  * Chandola et al. (2008) Varun Chandola, Varun Mithal, and Vipin Kumar. 2008\.  Comparative evaluation of anomaly detection techniques for sequence data. In _2008 Eighth IEEE international conference on data mining_. IEEE, 743–748.
  * Chen et al. (2018) Zhaomin Chen, Chai Kiat Yeo, Bu Sung Lee, and Chiew Tong Lau. 2018\.  Autoencoder-based network anomaly detection. In _2018 Wireless Telecommunications Symposium (WTS)_. IEEE, 1–5.
  * da Costa et al. (2020) Kelton AP da Costa, João P Papa, Leandro A Passos, Danilo Colombo, Javier Del Ser, Khan Muhammad, and Victor Hugo C de Albuquerque. 2020.  A critical literature survey and prospects on tampering and anomaly detection in image data.  _Applied Soft Computing_ 97 (2020), 106727.
  * Dietterich et al. (1997) Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez. 1997.  Solving the multiple instance problem with axis-parallel rectangles.  _Artificial intelligence_ 89, 1-2 (1997), 31–71.
  * Du Prel et al. (2009) Jean-Baptist Du Prel, Gerhard Hommel, Bernd Röhrig, and Maria Blettner. 2009.  Confidence interval or p-value?: part 4 of a series on evaluation of scientific publications.  _Deutsches Ärzteblatt International_ 106, 19 (2009), 335\.
  * Dua and Graff (2017) Dheeru Dua and Casey Graff. 2017.  UCI Machine Learning Repository.  http://archive.ics.uci.edu/ml
  * Feng et al. (2021a) Jia-Chang Feng, Fa-Ting Hong, and Wei-Shi Zheng. 2021a.  Mist: Multiple instance self-training framework for video anomaly detection. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 14009–14018.
  * Feng et al. (2021b) Lei Feng, Senlin Shu, Yuzhou Cao, Lue Tao, Hongxin Wei, Tao Xiang, Bo An, and Gang Niu. 2021b.  Multiple-Instance Learning from Similar and Dissimilar Bags. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_. 374–382.
  * Foulds and Frank (2010) James Foulds and Eibe Frank. 2010.  A review of multi-instance learning assumptions.  _The knowledge engineering review_ 25, 1 (2010), 1–25.
  * Guan et al. (2016) Xinze Guan, Raviv Raich, and Weng-Keen Wong. 2016.  Efficient multi-instance learning for activity recognition from time series data using an auto-regressive hidden markov model. In _International Conference on Machine Learning_. PMLR, 2330–2339.
  * Hautamaki et al. (2004) Ville Hautamaki, Ismo Karkkainen, and Pasi Franti. 2004\.  Outlier detection using k-nearest neighbour graph. In _Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004._ , Vol. 3. IEEE, 430–433.
  * Idé et al. (2007) Tsuyoshi Idé, Spiros Papadimitriou, and Michail Vlachos. 2007\.  Computing correlation anomaly scores using stochastic nearest neighbors. In _Seventh IEEE international conference on data mining (ICDM 2007)_. IEEE, 523–528.
  * Kim et al. (2021) Chul-Woo Kim, Feng-Liang Zhang, Kai-Chun Chang, Patrick John McGetrick, and Yoshinao Goi. 2021\.  Ambient and vehicle-induced vibration data of a steel truss bridge subject to artificial damage.  _Journal of Bridge Engineering_ 26, 7 (2021), 04721002\.
  * Kou et al. (2004) Yufeng Kou, Chang-Tien Lu, Sirirat Sirwongwattana, and Yo-Ping Huang. 2004.  Survey of fraud detection techniques. In _IEEE international conference on networking, sensing and control, 2004_ , Vol. 2. IEEE, 749–754.
  * Laxhammar and Falkman (2011) Rikard Laxhammar and Göran Falkman. 2011.  Sequential conformal anomaly detection in trajectories based on hausdorff distance. In _14th International Conference on Information Fusion_. IEEE, 1–8.
  * Laxhammar and Falkman (2015) Rikard Laxhammar and Göran Falkman. 2015.  Inductive conformal anomaly detection for sequential detection of anomalous sub-trajectories.  _Annals of Mathematics and Artificial Intelligence_ 74, 1 (2015), 67–94.
  * Oh et al. (2011) Sangmin Oh, Anthony Hoogs, Amitha Perera, Naresh Cuntoor, Chia-Chih Chen, Jong Taek Lee, Saurajit Mukherjee, JK Aggarwal, Hyungtae Lee, Larry Davis, et al. 2011\.  A large-scale benchmark dataset for event recognition in surveillance video. In _CVPR 2011_. IEEE, 3153–3160.
  * Quellec et al. (2016) Gwenolé Quellec, Mathieu Lamard, Michel Cozic, Gouenou Coatrieux, and Guy Cazuguel. 2016\.  Multiple-instance learning for anomaly detection in digital mammography.  _Ieee transactions on medical imaging_ 35, 7 (2016), 1604–1614.
  * Shyu et al. (2003) Mei-Ling Shyu, Shu-Ching Chen, Kanoksri Sarinnapakorn, and LiWu Chang. 2003.  _A novel anomaly detection scheme based on principal component classifier_.  Technical Report. Miami Univ Coral Gables Fl Dept of Electrical and Computer Engineering.
  * Sultani et al. (2018) Waqas Sultani, Chen Chen, and Mubarak Shah. 2018.  Real-world anomaly detection in surveillance videos. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6479–6488.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016.  Rethinking the inception architecture for computer vision. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2818–2826.
  * Tang et al. (2002) Jian Tang, Zhixiang Chen, Ada Wai-Chee Fu, and David W Cheung. 2002\.  Enhancing effectiveness of outlier detections for low density patterns. In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_. Springer, 535–548.
  * Tschannen et al. (2018) Michael Tschannen, Olivier Bachem, and Mario Lucic. 2018\.  Recent advances in autoencoder-based representation learning.  _arXiv preprint arXiv:1812.05069_ (2018).
  * Villa-Pérez et al. (2021) Miryam Elizabeth Villa-Pérez, Miguel Á Álvarez-Carmona, Octavio Loyola-González, Miguel Angel Medina-Pérez, Juan Carlos Velazco-Rossell, and Kim-Kwang Raymond Choo. 2021.  Semi-supervised anomaly detection algorithms: A comparative summary and future research directions.  _Knowledge-Based Systems_ 218 (2021), 106878.
  * Yang et al. (2013) Wanki Yang, Yang Gao, and Longbing Cao. 2013.  TRASMIL: A local anomaly detection framework based on trajectory segmentation and multi-instance learning.  _Computer Vision and Image Understanding_ 117, 10 (2013), 1273–1286.
  * Yang et al. (2018) Yanli Yang, Peiying Fu, and Yichuan He. 2018.  Bearing fault automatic classification based on deep learning.  _IEEE Access_ 6 (2018), 71540–71554.
  * Zenati et al. (2018) Houssam Zenati, Manon Romain, Chuan-Sheng Foo, Bruno Lecouat, and Vijay Chandrasekhar. 2018\.  Adversarially learned anomaly detection. In _2018 IEEE International conference on data mining (ICDM)_. IEEE, 727–736.
  * Zhang and Viola (2007) Cha Zhang and Paul Viola. 2007.  Multiple-instance pruning for learning efficient cascade detectors.  _Advances in neural information processing systems_ 20 (2007).
  * Zhang et al. (2021b) Jianyi Zhang, Jiaqi Yao, Yanjie Chu, and Jikun Yan. 2021b.  A Multiple Instance Learning Algorithm Using Graph Convolutional Network for Speech Content Classification. In _2021 IEEE 5th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC)_ , Vol. 5. IEEE, 1480–1484.
  * Zhang et al. (2021a) Yi-Ming Zhang, Hao Wang, Hua-Ping Wan, Jian-Xiao Mao, and Yi-Chao Xu. 2021a.  Anomaly detection of structural health monitoring data using the maximum likelihood estimation-based Bayesian dynamic linear model.  _Structural Health Monitoring_ 20, 6 (2021), 2936–2952.


11institutetext: School of Instrument Science and Engineering, Southeast
University, Nanjing 211189, China
11email<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
# CILF:Causality Inspired Learning Framework for Out-of-Distribution Vehicle
Trajectory Prediction

Shengyi Li  Qifan Xue  Yezhuo Zhang  Xuanpeng Li Corresponding Author

###### Abstract

Trajectory prediction is critical for autonomous driving vehicles. Most
existing methods tend to model the correlation between history trajectory
(input) and future trajectory (output). Since correlation is just a
superficial description of reality, these methods rely heavily on the i.i.d.
assumption and evince a heightened susceptibility to out-of-distribution data.
To address this problem, we propose an Out-of-Distribution Causal Graph (OOD-
CG), which explicitly defines the underlying causal structure of the data with
three entangled latent features: 1) domain-invariant causal feature (IC), 2)
domain-variant causal feature (VC), and 3) domain-variant non-causal feature
(VN). While these features are confounded by confounder (C) and domain
selector (D). To leverage causal features for prediction, we propose a Causal
Inspired Learning Framework (CILF), which includes three steps: 1) extracting
domain-invariant causal feature by means of an invariance loss, 2) extracting
domain variant feature by domain contrastive learning, and 3) separating
domain-variant causal and non-causal feature by encouraging causal
sufficiency. We evaluate the performance of CILF in different vehicle
trajectory prediction models on the mainstream datasets NGSIM and INTERACTION.
Experiments show promising improvements in CILF on domain generalization.

###### Keywords:

Causal Representation Learning Out-of-Distribution Domain Generalization
Vehicle trajectory Prediction

##  1 Introduction

Trajectory prediction is essential for both the perception and planning
modules of autonomous vehicles [19, 12] in order to reduce the risk of
collisions [13]. Recent trajectory prediction methods are primarily built with
deep neural networks, which are trained to model the correlation between
history trajectory and future trajectory. The robustness of such correlation
is guaranteed by the independent and identically distributed (i.i.d.)
assumption. As a result, the model trained on i.i.d. samples often fails to be
generalized to out-of-distribution (OOD) samples.

Recently, there has been a growing interest in utilizing causal representation
learning [25] to tackle the challenge of out-of-domain generalization [32].
Causal representation learning is based on the Structure Causal Model (SCM)
[7, 20], a mathematical tool for modeling human metaphysical concepts about
causation. Causal representation learning enables the model to discern the
underlying causal structure of data by incorporating causal-related prior
knowledge into the model.

This paper proposes an Out-of-Distribution Causal Graph (OOD-CG) based on SCM,
as shown in Fig.1. OOD-DG divides the latent features into three categories:
1) Domain-invariant Causal Feature (IC) such as physical laws, driving habits,
etc. 2) Domain-variant Causal Feature (VC) such as road traffic flow, traffic
scenes, etc. 3) Domain-variant Non-causal Feature (VN) like sensor noise, etc.
These features are entangled due to the confounding effects of backdoor
confounder (C) and domain selector (D).

Figure 1:  The step-by-step introduction of OOD-CG. White nodes represent
latent features; gray nodes represent observable variables; bidirectional
arrows represent correlations; unidirectional arrows represent causal
mechanisms; red arrows represent causal mechanisms critical to prediction; and
green arrows represent domain effects.

To leverage causal features for trajectory prediction, we introduce a Causal-
Inspired Learning Framework (CILF) based on causal representation learning.
CILF includes three parts to block the backdoor paths associated with IC, VC,
and VN. First, to block the backdoor path between IC and domain-variant
feature V, CILF utilizes invariant risk minimization (IRM) [2] to extract IC
and domain contrastive learning [10] to extract V. Then, to block the backdoor
path between VC and VN, CILF introduces domain adversarial learning [9] to
separate VC and VN.

Our contributions can be summarized as follows:

(1) We propose a theoretical model named OOD-CG, which explicitly elucidates
the causal mechanisms and causal structure of the out-of-distribution
generalization problem.

(2) Based on causal representation learning, we propose a learning framework
called CILF for out-of-distribution vehicle trajectory prediction. CILF
contains three steps to block the backdoor connection associated with IC, VC,
and VN, allowing the model to employ causal features for prediction.

##  2 Related Work

####  2.0.1 Vehicle trajectory Prediction

Recent works widely employ the sequence-to-sequence (Seq2seq) framework [26]
to predict a vehicle’s future trajectory based on its history trajectory [1,
5, 11, 27]. Alahi et al. introduce S-LSTM [1], which incorporates a social
pooling mechanism to aggregate and encode the social behaviors of surrounding
vehicles. Deo et al. propose CS-LSTM [5], which utilizes convolutional
operations to enhance the model’s performance. Lee et al. introduce a
trajectory prediction model called DESIRE [11], which combines conditional
variational auto-encoder (CVAE) and GRU to generate multimodal predictions of
future trajectories. Tang et al. incorporate an attention mechanism into the
Seq2seq framework named MFP [27], a model that can effectively learn motion
representations of multiple vehicles across multiple time steps. However,
current vehicle trajectory prediction approaches still face the challenge of
OOD generalization, posing a serious threat to the safety of autonomous
vehicles.

####  2.0.2 Out-of-Distribution generalization

Previous methods handle OOD generalization in two paradigms: domain adaptation
(DA) and domain generalization (DG) [32]. DA allows the model to access a
small portion of unlabeled target domain data during training, thereby
reducing the difficulty of the OOD problem to some extent. DA aims to learn an
embedding space where source domain samples and target domain samples follow
similar distributions via minimizing divergence [8, 15], domain adversarial
learning [6, 28], etc. DG prohibits models from accessing any form of target
domain data during training. DG aims to learn knowledge that can be directly
transferred to unknown target domains. Relevant literature has proposed a
range of solutions, including contrastive learning [17, 29], domain
adversarial learning [28, 31], etc. As for vehicle trajectory prediction, it
is difficult to acquire even unlabeled target domain samples. As a result, we
follow the paradigm of domain generalization.

####  2.0.3 Causality inspired approaches for domain generalization

There has been a growing trend toward utilizing causality inspired methods to
address the OOD problem. Some of them suggest using mathematical formulas from
causal inference [20], e.g.,front-door adjustment [18] and back-door
adjustment [3] formulas, to directly derive the loss function. However, these
methods often make strong assumptions (e.g.,restricting the model to be a
variational auto-encoder [3]), which decrease the diversity of the model’s
hypothesis space and limit the applicability of the model. Some approaches
instead only make assumptions about the general causal structure of the OOD
problem in order to guide the design of network structures [14, 16]. We follow
this approach in order to enhance the versatility of the proposed model.

##  3 Theoretical Analysis

###  3.1 Problem Formulation

We formulated vehicle trajectory prediction as estimating future trajectories
$Y_{i}=\\{(x_{i,t},y_{i,t})\in\mathbb{R}^{2}|t=t_{obs}+1,...,t_{pred}\\}$
based on the observed history trajectories
$X_{i}=\\{(x_{i,t},y_{i,t})\in\mathbb{R}^{2}|t=1,2,...,t_{obs}\\}$ of $N$
visible vehicles in the current scene. In the context of domain
generalization, the training dataset is collected from $K$ source domains,
denoted as $S\in\\{S_{1},S_{2},...,S_{K}\\}$, while the test dataset is
collected from $M$ target domains, denoted as
$T\in\\{T_{1},T_{2},...,T_{M}\\}$.

###  3.2 OOD-CG

Fig.1. Traditional deep learning is designed to capture statistical
correlations between inputs $X$ and outputs $Y$. Correlations are obviously
inadequate to solve the OOD problem. Fortunately, Reichenbach provides us with
a strong tool called the common causal principle [23] to decompose these
correlations into a set of backdoor features.

###### Theorem 3.1

Common causal principle: if two random variables $X$ and $Y$ are correlated,
there must exist another random variable $S$ that has causal relationships
with both $X$ and $Y$. Furthermore, $S$ can completely substitute for the
correlations between $X$ and $Y$, i.e., $X\perp\\!\\!\\!\perp Y|S$.

Fig.1. We divide these backdoor features into three classes: (1) Domain-
Invariant Causal Feature (IC): Driving habits, physical laws, etc. (2) Domain-
Variant Causal Feature (VC): Traffic density, traffic scenario, etc. (3)
Domain-Variant Non-causal Feature (VN): Sensor measurement noise, etc.
Bidirectional arrows are used in this step to represent the entanglement
between these backdoor features. Previous studies often ignore domain-variant
causal feature VC, and focus solely on utilizing domain-invariant causal
feature IC for prediction [3, 4, 18]. These approaches utilize insufficient
causal information for prediction and thus fail to simultaneously improve
prediction accuracy in both the source and target domains.

Fig.1. According to Theorem.3.1, we introduce a Confounder (C) to summarize
all the correlations among IC, VC, and VN. We also introduce a Domain Selector
(D) to represent domain effects. Since domain label is unavailable during
testing, we treat D as an unobservable latent variable consistent with [3,
16]. Now we can concretize distribution shifts as differences in feature prior
distributions $P(C),P(VC),P(VN)$ and causal mechanism conditional
distributions $P(VC|C),P(VN|C)$ between source and target domains.

Fig.1. In order to extract domain-invariant causal mechanisms $P(Y|VC)$,
$P(Y|IC)$ for prediction, it is necessary to block the backdoor paths
connecting these entangled features. To do so, we only need to block the
backdoor paths of IC and VN, which are represented by dashed lines in the
figure. Once done, VC actually serves as a mediator [7, 20] in causal effects
from D and C to Y, which can be completely substituted by VC. In the next
chapter, we introduce a causal-inspired learning framework (CILF) for OOD
vehicle trajectory prediction, encouraging models to block the backdoor paths
of IC and VN.

##  4 CILF

Compared to traditional machine learning, deep learning requires searching
within an extremely complex hypothesis space. As a result, researchers often
introduce inductive bias into deep learning models to reduce search
difficulty, allowing models to find some relatively acceptable local optimums
within limited training iterations. OOD-CG is proposed to impose such
inductive bias theoretically. Guided by OOD-CG, we propose CILF (see Fig.2) to
extract causal features for prediction. First, to block the backdoor
connection between IC and V, CILF introduces two kinds of losses: IRM loss to
encourage domain-invariant causal feature IC; Domain contrastive loss to
encourage domain-variant feature V. Then, to block the backdoor path between
VC and VN, CILF introduces domain adversarial learning to train a mask
generator able to detect the causal dimensions VC and non-causal dimensions VN
of domain-variant feature V.

Figure 2:  CILF consists of three parts: (i)IC encoder $\Phi_{ic}(.)$ takes
history trajectories of the current vehicle as input. The IRM loss
$\mathcal{L}_{irm}$ is utilized to encourage the extracted feature to exhibit
domain invariance; (ii)Domain-variant feature encoder $\Phi_{v}(.)$ takes full
observations from each source domain as input, such as history trajectories of
neighboring vehicles, map information, etc. The domain contrastive learning
loss $\mathcal{L}_{contrast}$ is utilized to encourage the extracted feature
to exhibit domain variance; (iii)Mask generator $M(.)$ generates a causally
sufficient mask to select the causal dimensions VC and non-causal dimensions
VN from V.

###  4.1 Extract Domain-invariant Causal Feature

Domain-invariant features are the intersection of feature sets from different
domains. The non-causal information within these features exhibits a
significant reduction after taking the intersection. As a result, extracting
IC can be approximated as extracting the domain-invariant feature. The
definition of a domain-invariant encoder is given as follows [2]:

###### Definition 1

Domain-invariant feature encoder: An encoder
$\Phi(.):\mathcal{X}\to\mathcal{H}$ is said to be a domain-invariant encoder
across all domains $s\in S$ if and only if there exists a decoder
$\Theta(.):\mathcal{H}\to\mathcal{Y}$ that achieves optimum across all
domains. This condition can be further formulated as a conditional
optimization problem:

$\begin{split}&\underset{\Phi,\Theta}{\mathrm{min}}\Sigma_{s\in
S}\mathcal{L}_{s}(\Theta\circ\Phi),\\\ \text{subject to
}\Theta\in&\underset{\bar{\Theta}}{\mathrm{argmin}}\mathcal{L}_{s}(\bar{\Theta}\circ\Phi),\text{for
all }s\in S.\end{split}$ (1)

However, this is a bi-level optimization problem, which is computationally
difficult to solve. In current research, the problem is often relaxed to a
gradient regularization penalty based on empirical risk minimization [2]:

$\underset{\Phi_{IC},\Theta}{\mathrm{min}}\frac{1}{\lvert S\rvert}\Sigma_{s\in
S}[\mathcal{L}_{s}(\Theta\circ\Phi_{IC})+\lambda\lVert\nabla_{\Theta}\mathcal{L}_{s}(\Theta\circ\Phi_{IC})\rVert^{2}],$
(2) $\mathcal{L}_{irm}=\frac{1}{\lvert
S\rvert}\lVert\nabla_{\Theta}\mathcal{L}_{s}(\Theta\circ\Phi_{IC})\rVert^{2},$
(3)

where $\mathcal{L}_{irm}$ denotes the IRM loss, $\lambda$ denotes the balance
parameter.

We separate $\Phi_{IC}$ from the backbone to further block the backdoor path
of IC, as shown in Fig.2. $\Phi_{IC}$ only accepts the history trajectories of
the current vehicle as input, which contains relatively limited domain
knowledge.

###  4.2 Extract Domain-variant Feature

To extract domain-variant feature, we design $\Phi_{v}(.)$, which takes full
observations from different domains (distinguished by the colors in Fig.2) as
input. Full observations include all the information that an autonomous
driving vehicle can observe in a traffic scene (e.g., history trajectories of
neighboring vehicles, map information, etc.), which are highly domain-related.
$\Phi_{v}(.)$ is trained using domain contrastive loss [10, 14]. In order to
calculate this loss, CILF employs dimensional reduction to V through a
projection layer $p_{i}=\beta(V_{i})$, where $i$ denotes the domain label. The
contrastive loss between a pair of samples $p_{i}$ and $p_{j}$ is defined as
follows:

$\mathcal{L}_{contrast}=-\mathrm{log}\frac{\mathrm{exp}(p_{i},p_{j})/\tau}{\Sigma_{k}\mathbb{I}_{[k=j\lor
S_{k}\neq S_{i}]}\mathrm{exp}(\mathrm{sim}(p_{i},p_{k})/\tau)}$ (4)

where $\mathbb{I}_{[k=j\lor S_{k}\neq S_{i}]}$ denotes the indicator function.
When sample $p_{i}$ and $p_{k}$ are from the same domain, this function equals
to 1. $\tau$ is the temperature parameter of contrastive learning,
$\mathrm{sim}(a,b)=\frac{a^{\mathrm{T}}b}{\lVert a\rVert\lVert b\rVert}$ is
the cosine similarity.

###  4.3 Separate Domain-variant Causal and Non-causal Feature

In Section 4.1, we argue that domain-invariant features naturally possess
causality as they are the intersection of feature sets from different domains.
However, domain-variant features do not naturally carry this causal property.
It is necessary to filter out the non-causal dimensions within these features
according to causal sufficiency [21, 23, 24]:

###### Definition 2

Causally sufficient feature set: for the prediction task from $X$ to $Y$, a
feature set is considered causally sufficient if and only if it captures all
causal effects from $X$ to $Y$.

Supervised learning can’t guarantee the learned features to be causally
sufficient according to Definition.2. Some dimensions may contain more causal
information and have a decisive impact on prediction, while other dimensions
may have little influence on prediction. CILF introduces a neural network
based mask generator $M(.)$ that produces a causally sufficient mask [16]
using the Gumbel-SoftMax technique [9]:

$m=\mathrm{Gumbel-SoftMax}(M(V),kN),$ (5)

The causally sufficient mask can identify the contribution $k\in(0,1)$ of each
dimension in V to prediction. The top $kN$ dimensions with the highest
contributions are regarded as domain-variant causal features VC, while the
remaining dimensions are considered domain-variant non-causal features VN:

$\textit{VC}=V\cdot m,\textit{VN}=V\cdot(1-m).$ (6)

VC and VN are fused with IC by the fusion module $F(.)$ to form causal
features CF and spurious features SF, which are then decoded separately by
$\Theta(.)$ to generate future predictions. Prediction losses
$\mathcal{L}_{causal}$ and $\mathcal{L}_{spurious}$ can be calculated by RMSE:

$\begin{split}CF=F(VC,IC)&=f_{2}(f_{1}(VC,IC)+IC),\\\
SF=F(VN,IC)&=f_{2}(f_{1}(VN,IC)+IC),\end{split}$ (7)
$\begin{split}\mathcal{L}_{causal}&=\mathrm{RMSE}(\Theta(VC),Y),\\\
\mathcal{L}_{spurious}&=\mathrm{RMSE}(\Theta(VN),Y),\end{split}$ (8)

where $f_{1}$ and $f_{2}$ are FC layers in fusion module.

Decoder $\Theta$ can be trained by minimizing $\mathcal{L}_{causal}$ and
$\mathcal{L}_{spurious}$. Mask generator $M$ can be trained by minimizing
$\mathcal{L}_{causal}$ while adversarially maximizing
$\mathcal{L}_{spurious}$. Overall, the optimization objective can be
formulated as follows:

$\underset{\Phi_{IC},F,\Theta}{\mathrm{min}}\mathcal{L}_{causal}+\mathcal{L}_{spurious}+\lambda\mathcal{L}_{irm}$
(9)
$\underset{\Phi_{v},F,\Theta}{\mathrm{min}}\mathcal{L}_{causal}+\mathcal{L}_{spurious}+\alpha\mathcal{L}_{irm}$
(10) $\underset{M}{\mathrm{min}}\mathcal{L}_{causal}-\mathcal{L}_{spurious}$
(11)

where $\lambda$ and $\alpha$ are balance parameters.

##  5 Experiments

This chapter presents quantitative and qualitative domain generalization
experiments of the proposed CILF framework on the public vehicle trajectory
prediction datasets NGSIM [22] and INTERACTION [30].

###  5.1 Experiment Design

INTERACTION is a large-scale real-world dataset that includes top-down vehicle
trajectory data in three scenarios: intersections, highway ramps (referred to
as merging), and roundabouts, collected from multiple locations in America,
Asia, and Europe. INTERACTION consists of 11 subsets, categorized into three
types of scenarios: roundabouts, highway ramps, and intersections (see Table
1). NGSIM is a collection of vehicle trajectory data derived from video
recordings. It includes vehicle trajectory data from three different
locations: the US-101 highway, Lankershim Blvd. in Los Angeles, and the I-80
highway in Emeryville. The sampling rate of both datasets is 10Hz. For each
trajectory that lasts 8 seconds, the model takes the first 3 seconds as input
and predicts the trajectory for the next 5 seconds.

As shown in Table 1, data volume among different subsets of INTERACTION varies
significantly. Subsets 0, 1, and 7 have notably higher data volumes compared
to other subsets. To mitigate the potential influence of data volume
disparities, we design three contrastive experiments to demonstrate the
effectiveness of CILF in addressing the DG problem:

Table 1: INTERACTION Dataset. Data Ratio represents the proportion of data volume in each subset relative to the total data volume. Subset ID | Sample Location | Scenario | Data Ratio
---|---|---|---
0 | USA | Roundabout | 17.9%
1 | CHN | Merging | 35.2%
2 | USA | Intersection | 7.9%
3 | USA | Intersection | 2.6%
4 | GER | Roundabout | 1.8%
5 | USA | Roundabout | 5.3%
6 | GER | Merging | 1.2%
7 | USA | Intersection | 21.6%
8 | USA | Roundabout | 3.2%
9 | USA | Intersection | 2.6%
10 | CHN | Roundabout | 0.6%

(1)Single-scenario domain generalization: Both training and test datasets come
from the same scenario within the INTERACTION. Specifically, the subset with
the largest data volume in each scenario is designated as the test set, while
the remaining subsets serve as the training set.

(2)Cross-scenario domain generalization: The training and test datasets come
from different scenarios within the INTERACTION. Specifically, the three
subsets with the largest data volumes (roundabout-0, merging-1, and
intersection-7) are selected as training sets, while the remaining subsets are
chosen as test sets.

(3)Cross-dataset domain generalization: The INTERACTION is selected as the
training set, while the NGSIM is selected as the test set.

We choose three classic vehicle trajectory prediction models as baselines:

S-LSTM [1]:An influential method based on a social pooling model;

CS-LSTM [5]:A model using a convolutional social pooling structure to learn
vehicle interactions;

MFP [27]:An advanced model that learns semantic latent variables for
trajectory prediction. These baselines are trained under CILF to compare their
domain generalization performance.

We employ two commonly used metrics in vehicle trajectory prediction to
measure the model’s domain generalization performance:

Average Displacement Error (ADE): Average L2 distance between predicted points
and groundtruth points across all timesteps.

$\mathrm{ADE}=\frac{\Sigma_{i=1}^{N}\Sigma_{t=t_{obs}+1}^{t_{pred}}\lVert\hat{Y}_{i,t}-Y_{i,t}\rVert}{N},$
(12)

Final Displacement Error (FDE): L2 distance between predicted points and
ground truth points at the final timestep.

$\mathrm{FDE}=\frac{\Sigma_{i=1}^{N}\lVert\hat{Y}_{i,t}-Y_{i,t}\rVert}{N},t=t_{obs}+1,...,t_{pred}.$
(13)

###  5.2 Quantitative Experiment and Analysis

####  5.2.1 Single-scenario domain generalization

Table 2 and Table 3 present the comparison between different models trained
under CILF and vanilla conditions in terms of single-scenario domain
generalization on the INTERACTION.

Table 2: ADE and FDE (in meters) results of domain generalization experiment in the intersection scenario. Subset 7 is left for testing, while other subsets of the intersection scenario are used for training.  | S-LSTM | CS-LSTM | MFP
---|---|---|---
|  ADE |  FDE |  ADE |  FDE |  ADE |  FDE
Intersection-2 |  1.52/1.59 |  4.65/4.85 |  1.53/1.64 |  4.62/4.86 |  1.62/1.72 |  5.05/5.29
Intersection-3 |  1.21/1.29 |  3.48/3.60 |  1.18/1.28 |  3.43/3.59 |  1.25/1.32 |  3.67/3.83
Intersection-9 |  1.20/1.29 |  3.37/3.53 |  1.20/1.30 |  3.36/3.56 |  1.25/1.34 |  3.57/3.79
Intersection-7 |  2.22/2.25 |  6.20/6.32 |  2.34/2.37 |  6.30/6.47 |  2.12/2.21 |  6.02/6.04
Table 3: ADE and FDE (in meters) results of domain generalization experiment in the roundabout scenario. Subset 0 is left for testing, while other subsets of the roundabout scenario are used for training.  | S-LSTM | CS-LSTM | MFP
---|---|---|---
|  ADE |  FDE |  ADE |  FDE |  ADE |  FDE
Roundabout-4 |  1.68/1.75 |  4.41/4.64 |  1.62/1.83 |  4.41/4.81 |  1.59/1.73 |  2.86/3.14
Roundabout-6 |  1.03/1.12 |  2.97/3.09 |  1.01/1.10 |  2.86/3.02 |  1.01/1.14 |  2.86/3.14
Roundabout-10 |  1.42/1.57 |  3.60/3.94 |  1.38/1.51 |  3.50/3.94 |  1.30/1.60 |  3.42/4.05
Roundabout-0 |  3.31/3.34 |  9.04/9.38 |  3.24/3.35 |  9.01/9.09 |  3.15/3.25 |  8.80/8.83

CILF achieves both ADE and FDE improvements for all models in both the source
and target domains.

In the intersection scenario, as shown in Table 2, for the source domains, CS-
LSTM achieves the largest improvement under CILF, with average increments of
7.40% and 5.00% (ADE and FDE). Conversely, S-LSTM shows the lowest
improvement, with average increments of 5.86% and 4.00%. As for the target
domain, S-LSTM and MFP demonstrate similar improvements, with increments of
1.33%, 1.90% for S-LSTM and 1.40%, 1.33% for MFP.

In the roundabout scenario, as shown in Table 3, for the source domains, MFP
achieves the largest improvement under CILF, with average increments of 12.75%
and 9.31% (ADE and FDE). Conversely, S-LSTM shows the lowest improvement, with
average increments of 7.20% and 5.82%. As for the target domain, S-LSTM
demonstrates the largest improvements, with increments of 3.90% and 3.62%. MFP
achieves the lowest improvement, with increments of 3.08% and 2.34%.
Obviously, in the roundabout scenario, CILF exhibits larger improvements in
both the source and target domains compared to intersection.

####  5.2.2 Cross-scenario domain generalization

Table 4 presents the comparison between different models trained under CILF
and vanilla conditions in terms of cross-scenario domain generalization on the
INTERACTION. CILF achieves both ADE and FDE improvements for all models in
both the source and target domains. For the source domains, CS-LSTM achieves
the largest improvement under CILF, with average increments of 7.54% and 5.95%
(ADE and FDE). Conversely, MFP shows the lowest improvement, with average
increments of 3.58% and 2.95%. As for the target domain, both three models
achieve similar improvements, with increments of 4.73%, 3.53% for S-LSTM;
5.47%, 3.17% for CS-LSTM; and 3.65%, 2.53% for MFP.

Table 4: ADE and FDE (in meters) results of cross-scenario domain generalization experiment. Subset 0, 1, and 7 are left for training, while other subsets of INTERACTION are used for testing.  | S-LSTM | CS-LSTM | MFP
---|---|---|---
|  ADE |  FDE |  ADE |  FDE |  ADE |  FDE
Roundabout-0 |  1.40/1.51 |  4.33/4.58 |  1.36/1.46 |  4.24/4.46 |  1.45/1.50 |  4.47/4.56
Merging-1 |  0.83/0.85 |  2.28/2.33 |  0.80/0.86 |  2.24/2.36 |  0.86/0.89 |  2.48/2.55
Intersection-7 |  1.18/1.26 |  3.60/3.78 |  1.14/1.25 |  3.53/3.83 |  1.19/1.24 |  3.70/3.86
Intersection-2 |  2.42/2.52 |  7.59/7.82 |  2.45/2.55 |  7.72/7.90 |  2.42/2.48 |  7.60/7.72
Intersection-3 |  1.63/1.73 |  4.92/5.23 |  1.68/1.77 |  5.13/5.42 |  1.66/1.71 |  5.02/5.18
Roundabout-4 |  3.32/3.67 |  10.1311.01 |  3.42/3.65 |  10.60/10.76 |  3.59/3.78 |  10.71/11.26
Roundabout-5 |  1.75/1.83 |  5.16/5.38 |  1.72/1.84 |  5.20/5.36 |  1.71/1.82 |  4.97/5.25
Merging-6 |  1.19/1.21 |  3.59/3.63 |  1.15/1.24 |  3.54/3.69 |  1.07/1.14 |  3.34/3.43
Roundabout-8 |  2.07/2.14 |  6.21/6.34 |  2.07/2.22 |  6.21/6.48 |  2.08/2.13 |  6.17/6.28
Intersection-9 |  1.65/1.70 |  4.90/5.04 |  1.66/1,75 |  4.99/5.19 |  1.62/1.64 |  4.86/4.86
Roundabout-10 |  2.80/2.99 |  8.64/8.76 |  2.84/2.92 |  8.85/8.75 |  2.86/2.95 |  8.50/8.59

####  5.2.3 Cross-dataset domain generalization

Table 5 presents the comparison between different models trained under CILF
and vanilla conditions in terms of cross-dataset domain generalization. CILF
achieves both ADE and FDE improvements for all models in both the source and
target domains. For the source domains, CS-LSTM achieves the largest
improvement under CILF, with average increments of 7.54% and 5.95% (ADE and
FDE). Conversely, MFP shows the lowest improvement, with average increments of
3.58% and 2.95%. For the target domains, S-LSTM achieves the largest
improvement under CILF, with average increments of 4.39% and 3.53%.
Conversely, MFP shows the lowest improvement, with average increments of 1.12%
and 0.64%.

Table 5: ADE and FDE (in meters) results of cross-dataset domain generalization experiment. Subset 0, 1, and 7 of the INTERACTION are left for training, while the NGSIM is used for testing. | S-LSTM | CS-LSTM | MFP
---|---|---|---
|  ADE |  FDE |  ADE |  FDE |  ADE |  FDE
Roundabout-0 |  1.40/1.51 |  4.33/4.58 |  1.36/1.46 |  4.24/4.46 |  1.45/1.50 |  4.47/4.56
Merging-1 |  0.83/0.85 |  2.28/2.33 |  0.80/0.86 |  2.24/2.36 |  0.86/0.89 |  2.48/2.55
Intersection-7 |  1.18/1.26 |  3.60/3.78 |  1.14/1.25 |  3.53/3.83 |  1.19/1.24 |  3.70/3.86
NGSIM |  3.27/3.42 |  8.47/8.78 |  3.50/3.60 |  9.17/9.47 |  3.54/3.58 |  9.28/9.38

###  5.3 Qualitative Experiment and Analysis

This section presents the comparison of predicted trajectories generated by
CILF-MFP and Vanilla-MFP in the cross-scenario domain generalization
experiment.

Fig.3 illustrates the comparison of predicted trajectories between CILF-MFP
and MFP in target domain subset-3 (intersection) and subset-4 (roundabout).
The black bold solid lines represent the curb, and the white and gray bold
solid lines represent guide lines. The red square denotes the current vehicle,
and the blue square represents neighboring vehicles perceived by the current
vehicle. The yellow dots represent history trajectories of the current
vehicle, while the blue dots represent future trajectories. The red dots
represent the predicted future trajectory. The green dots represent the
historical and future trajectories of neighboring vehicles.

In both scenarios, CILF-MFP demonstrates remarkable improvement in prediction
quality compared to MFP, particularly at the end of the predicted
trajectories.

(a) CILF-MFP

(b) Vanilla-MFP

(c) CILF-MFP

(d) Vanilla-MFP

Figure 3:  Contrastive experiments between CILF-MFP and Vanilla-MFP on two
target domains. (a) and (b) demonstrate the contrastive experiment on subset-3
(intersection) of INTERACTION, (c) and (d) demonstrate the contrastive
experiment on subset-4 (roundabout) of INTERACTION.

##  6 Conclusion

To improve the generalization ability of vehicle trajectory prediction models,
we first analyze the causal structure of OOD generalization and propose OOD-
CG, which highlights the limitations of conventional correlation-based
learning framework. Then we propose CILF to employ only causal features for
prediction by three steps: (a) extracting IC by invariant risk minimization,
(b) extracting V by domain contrastive learning, and (c) separating VC and VN
by domain adversarial learning. Quantitative and qualitative experiments on
several mainstream datasets prove the effectiveness of our model.

## References

  * [1] Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., Savarese, S.: Social lstm: Human trajectory prediction in crowded spaces. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 961–971 (2016)
  * [2] Arjovsky, M., Bottou, L., Gulrajani, I., Lopez-Paz, D.: Invariant risk minimization. arXiv preprint arXiv:1907.02893 (2019)
  * [3] Bagi, S.S.G., Gharaee, Z., Schulte, O., Crowley, M.: Generative causal representation learning for out-of-distribution motion forecasting. arXiv preprint arXiv:2302.08635 (2023)
  * [4] Chen, G., Li, J., Lu, J., Zhou, J.: Human trajectory prediction via counterfactual analysis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9824–9833 (2021)
  * [5] Deo, N., Trivedi, M.M.: Convolutional social pooling for vehicle trajectory prediction. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 1468–1476 (2018)
  * [6] Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V.: Domain-adversarial training of neural networks. The journal of machine learning research 17(1), 2096–2030 (2016)
  * [7] Glymour, M., Pearl, J., Jewell, N.P.: Causal inference in statistics: A primer. John Wiley & Sons (2016)
  * [8] Gretton, A., Borgwardt, K.M., Rasch, M.J., Schölkopf, B., Smola, A.: A kernel two-sample test. The Journal of Machine Learning Research 13(1), 723–773 (2012)
  * [9] Jang, E., Gu, S., Poole, B.: Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 (2016)
  * [10] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. Advances in neural information processing systems 33, 18661–18673 (2020)
  * [11] Lee, N., Choi, W., Vernaza, P., Choy, C.B., Torr, P.H., Chandraker, M.: Desire: Distant future prediction in dynamic scenes with interacting agents. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 336–345 (2017)
  * [12] Lefèvre, S., Vasquez, D., Laugier, C.: A survey on motion prediction and risk assessment for intelligent vehicles. ROBOMECH journal 1(1), 1–14 (2014)
  * [13] Li, S., Xue, Q., Shi, D., Li, X., Zhang, W.: Recursive least squares based refinement network for vehicle trajectory prediction. Electronics 11(12), 1859 (2022)
  * [14] Liu, Y., Cadei, R., Schweizer, J., Bahmani, S., Alahi, A.: Towards robust and adaptive motion forecasting: A causal representation perspective. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17081–17092 (2022)
  * [15] Long, M., Zhu, H., Wang, J., Jordan, M.I.: Deep transfer learning with joint adaptation networks. In: International conference on machine learning. pp. 2208–2217. PMLR (2017)
  * [16] Lv, F., Liang, J., Li, S., Zang, B., Liu, C.H., Wang, Z., Liu, D.: Causality inspired representation learning for domain generalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8046–8056 (2022)
  * [17] Motiian, S., Piccirilli, M., Adjeroh, D.A., Doretto, G.: Unified deep supervised domain adaptation and generalization. In: Proceedings of the IEEE international conference on computer vision. pp. 5715–5725 (2017)
  * [18] Nguyen, T., Do, K., Nguyen, D.T., Duong, B., Nguyen, T.: Front-door adjustment via style transfer for out-of-distribution generalisation. arXiv preprint arXiv:2212.03063 (2022)
  * [19] Paden, B., Čáp, M., Yong, S.Z., Yershov, D., Frazzoli, E.: A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on intelligent vehicles 1(1), 33–55 (2016)
  * [20] Pearl, J.: Causality. Cambridge university press (2009)
  * [21] Peters, J., Janzing, D., Schölkopf, B.: Elements of causal inference: foundations and learning algorithms. The MIT Press (2017)
  * [22] Punzo, V., Borzacchiello, M.T., Ciuffo, B.: On the assessment of vehicle trajectory data accuracy and application to the next generation simulation (ngsim) program data. Transportation Research Part C: Emerging Technologies 19(6), 1243–1262 (2011)
  * [23] Reichenbach, H.: The Direction of Time. Mineola, N.Y.: Dover Publications (1956)
  * [24] Schölkopf, B., Janzing, D., Peters, J., Sgouritsa, E., Zhang, K., Mooij, J.: On causal and anticausal learning. arXiv preprint arXiv:1206.6471 (2012)
  * [25] Schölkopf, B., Locatello, F., Bauer, S., Ke, N.R., Kalchbrenner, N., Goyal, A., Bengio, Y.: Toward causal representation learning. Proceedings of the IEEE 109(5), 612–634 (2021)
  * [26] Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. Advances in neural information processing systems 27 (2014)
  * [27] Tang, C., Salakhutdinov, R.R.: Multiple futures prediction. Advances in neural information processing systems 32 (2019)
  * [28] Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7167–7176 (2017)
  * [29] Yoon, C., Hamarneh, G., Garbi, R.: Generalizable feature learning in the presence of data bias and domain class imbalance with application to skin lesion classification. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part IV 22. pp. 365–373. Springer (2019)
  * [30] Zhan, W., Sun, L., Wang, D., Shi, H., Clausse, A., Naumann, M., Kummerle, J., Konigshof, H., Stiller, C., de La Fortelle, A., et al.: Interaction dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps. arXiv preprint arXiv:1910.03088 (2019)
  * [31] Zhang, W., Ouyang, W., Li, W., Xu, D.: Collaborative and adversarial network for unsupervised domain adaptation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3801–3809 (2018)
  * [32] Zhou, K., Liu, Z., Qiao, Y., Xiang, T., Loy, C.C.: Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)