# Action-slot: Visual Action-centric Representations for Multi-label Atomic
Activity Recognition in Traffic Scenes

Chi-Hsi Kung1 Shu-Wei Lu1 Yi-Hsuan Tsai2 Yi-Ting Chen1 National Yang Ming
Chiao Tung University1
<EMAIL_ADDRESS>Google2
<EMAIL_ADDRESS>
###### Abstract

In this paper, we study multi-label atomic activity recognition. Despite the
notable progress in action recognition, it is still challenging to recognize
atomic activities due to a deficiency in holistic understanding of both
multiple road users’ motions and their contextual information. In this paper,
we introduce Action-slot, a slot attention-based approach that learns visual
action-centric representations, capturing both motion and contextual
information. Our key idea is to design action slots that are capable of paying
attention to regions where atomic activities occur, without the need for
explicit perception guidance. To further enhance slot attention, we introduce
a background slot that competes with action slots, aiding the training process
in avoiding unnecessary focus on background regions devoid of activities. Yet,
the imbalanced class distribution in the existing dataset hampers the
assessment of rare activities. To address the limitation, we collect a
synthetic dataset called TACO, which is four times larger than OATS and
features a balanced distribution of atomic activities. To validate the
effectiveness of our method, we conduct comprehensive experiments and ablation
studies against various action recognition baselines. We also show that the
performance of multi-label atomic activity recognition on real-world datasets
can be improved by pretraining representations on TACO. We will release our
source code and dataset. See the videos of visualization on the project page:
https://hcis-lab.github.io/Action-slot/

Figure 1:  Illustration of the concept of multi-label atomic activity
recognition and our proposed Action-slot. In the scene, three atomic
activities are presented and depicted by colored arrows. For example, the red
arrow represents the Z1-Z4: C+ atomic activity, indicating a group of vehicles
turning left. Atomic activities are defined based on road user’s type and
their motion patterns grounded in the underlying road structure. We introduce
Action-slot to learn visual action-centric representations that enable
decomposing multiple atomic activities in videos. We demonstrate that our
framework can effectively recognize multiple atomic activities via learned
representations.  Figure 2:  The distribution of atomic activity classes in
our TACO dataset compared to the OATS dataset. Please note that, for space
considerations, we omit the road topology notations of C and Z on the x-axis
of the figure.

##  1 Introduction

There has been a noteworthy rise of community interest in activity recognition
within traffic scenes [13, 57, 58, 52, 47, 66, 3]. Recognizing activities of
road users is essential for advancing the development of intelligent driving
systems, as it enables various applications, such as intent prediction [38,
47, 77], scenario retrieval [46, 69, 63, 55], scenario-based assessment [1,
78], and safety-critical scenario generation [78, 59, 10, 60].

In this paper, we focus on multi-label atomic activity recognition, a new task
proposed by the OATS dataset for interactive traffic scenario understanding
[3]. An atomic activity is a higher-level semantic motion pattern rooted in
the underlying road topology. For instance, a group of vehicles turning left
is described as Z1-Z4: C+, as shown in Figure Action-slot: Visual Action-
centric Representations for Multi-label Atomic Activity Recognition in Traffic
Scenes. The notions Z1 and Z4 are roadways within an intersection. A left-turn
motion pattern is represented as Z1-Z4. We use C+ to denote a group of
vehicles. This task presents distinctive challenges as it demands a
comprehensive understanding of contextual information and motions of road
users. Moreover, it requires the ability to break down activities within
crowded and intricate traffic scenes from videos.

We identify the limitations of current action recognition models, showing less
desirable performance in the OATS benchmark [3]. We argue that video-level
representations, employing 3D ConvNet [12, 72, 29, 28] or transformers [27,
71] models, pose a challenge for classifiers in distinguishing individual
atomic activities within traffic scenes due to the complicated structure of
traffic scenes. On the other hand, object-aware representations [6, 75, 3]
using object features [35] struggle to associate the relationship between the
road structures and motions of objects. Then the question we like to answer in
this paper: Can we learn visual representations that decompose multiple atomic
activities from videos, without using object proposals?

To this end, we propose Action-slot, a slot attention-based framework,
inspired by the recent success of slot attention for unsupervised object
discovery [48]. Action-slot incorporates four crucial design choices that
allow the model to decompose multiple atomic activities from videos and
represent them without relying on object proposals. First, we allocate a fixed
number of action slots, assigning each action slot to focus on regions where a
specific atomic activity occurs, such as C2-C1: P. Second, we introduce an
additional background slot and attention guidance to enforce the slot paying
attention to regions without active atomic activities. Third, we discourage
action slots allocated with negative classes (i.e., those activities not
present in a video), from attending to any regions. Fourth, we modify the slot
updating strategy used in [42, 25, 5] to a parallel updating approach that
integrates spatial-temporal information at the video level into a slot.

In our experiments, we demonstrate the effectiveness of Action-slot in
accurately identifying a variety of atomic activities annotated in the OATS
dataset [3]. Furthermore, we conduct thorough experiments, comparing various
action recognition baselines and presenting ablation studies that validate the
design of Action-slot. Moreover, we provide qualitative evidence demonstrating
that the learned action-centric representations can reliably identify distinct
atomic activities in the visual domain, even without the need for supervision
signals such as object locations [25]. However, due to imbalanced class
distribution within the dataset, OATS only uses 35 out of 64 atomic activity
classes for training and evaluation. Specifically, there are no activities
associated with group two-wheelers (K+). This observation extends to nearly
half of the activity classes, impeding the evaluation of rare classes.

To address this limitation, we introduce the Traffic Activity Recognition
(TACO) dataset, an extensive dataset for atomic activity recognition. We
utilize the CARLA simulator [23] to gather instances of all conceivable
activity classes, ensuring a well-balanced distribution, as illustrated in
Figure 2. We again benchmark all methods on the TACO dataset and show that
Action-slot outperforms all baselines by a large margin.

Beyond comprehensive performance analysis, we demonstrate that TACO can
enhance the efficacy of atomic activity recognition on real-world datasets
through pretraining feature representations on TACO. Then we finetune the
pretrained model on both the OATS dataset and a newly annotated nuScenes
dataset [9]. We present significant improvements in all experiments through
TACO pre-training, underscoring the robust real-world transferability of TACO.
Our main contributions are as follows:

  1. 1.

We introduce Action-slot: an action-centric slot attention-based framework
that decomposes multiple atomic activities from videos.

  2. 2.

We present the TACO dataset, an extensive and balanced dataset for multi-label
atomic activity recognition, facilitating comprehensive performance analysis.

  3. 3.

We conduct extensive evaluations and ablation studies across multiple datasets
to justify the effectiveness and generalization of Action-slot.

  4. 4.

We provide qualitative evidence demonstrating that the learned action-centric
representations can reliably identify distinct atomic activities in the visual
domain, without explicit perception guidance.

##  2 Related Work

#### Traffic Scenario Understanding Datasets.

The development of intelligent driving systems, such as scene analysis [57,
47, 77], scenario retrieval [46, 69, 63, 55], and safety-critical scenario
generation [78, 59, 10, 60], heavily relies on traffic activity understanding.
While previous efforts have designed datasets [13, 57, 45, 52] by labeling
high-level actions (e.g., left turn) for traffic pattern recognition, they
fall short in supporting fine-grained scenario analysis. For instance, high-
level actions cannot differentiate “left turns” that start from different
locations, such as the ego lane versus the oncoming lane. To address this
limitation, both Inner-City [13] and ROAD [66] datasets augment high-level
action labels (e.g., turning left) with location labels (e.g., on the oncoming
lane or on our side of the road). Recently, OATS [3] propose a topology-aware
traffic activity description language to unify the action and location labels
via the topology-aware pattern description. Specifically, the language
decomposes an interactive scenario into a set of atomic activities, that are
defined based on the types of road users and the corresponding motion patterns
grounded in road structures. Nevertheless, some real-world instances of
traffic atomic activities are scarce, presenting a challenge in forming a
well-balanced category distribution and impeding the evaluation of rare
classes. In response, we construct the TACO dataset utilizing the CARLA
simulator [23] to gather instances of all conceivable activity classes,
ensuring a well-balanced distribution.

#### Video Action Recognition.

Substantial improvements in spatial-temporal modeling via 3D ConvNet [12, 28,
72, 29] and transformer [7, 4, 27, 71] have been observed because of the
Kinetics human action dataset [41] that facilitates model pre-training. The
pre-trained models can further be finetuned on other action recognition
datasets such as the HMDB [44], THUMOS [39], and Charades [65] datasets.
Meanwhile, the community also explores multi-label action recognition and
establishes benchmarking datasets, such as MultiTHUMOS [80], Charades [65],
and AVA [31]. The most relevant task to multi-label atomic activity
recognition is group activity recognition [18, 37], where multiple players
engage in distinct actions. In contrast, atomic activity recognition in
traffic scenes demands a comprehensive understanding of objects and their
motion patterns within the underlying topology. In addition, a noteworthy
distinction is that the majority of objects in traffic scenes do not exhibit
any activity, constituting a negative class, a concept absent in group
activity recognition datasets.

#### Slot Attention for Representation Learning.

Locatello et al., [48] propose slot attention as a new perspective for
learning object-centric representations in an unsupervised manner. They have
successfully stimulated the community to explore strategies to bootstrap slot
attention from synthetic datasets [40] to real-world dynamic visual scenes
with moving cameras such as the Waymo [67] and KITTI [30] datasets.
Particularly, they utilize auxiliary information such as optical flow [42],
motion segmentation [5], object initial locations [25], and depth [25] to
facilitate object-centric representation learning in challenging real-world
applications.

Multi-label atomic activity recognition poses three challenges for existing
slot attention methods. First, the slots are permutation invariant. This
property is designed to discover arbitrary objects [48, 42, 25, 5, 82, 62, 8]
but is not suitable for a classification task with fixed-length classes.
Second, current slot attention architectures used in videos, such as SAVi [42]
and Slot-VPS [82], are designed for object-centric tasks, such as tracking.
Specifically, they update a slot for each frame in a recurrent manner, which
is not suitable for action recognition that requires a holistic temporal
understanding. Third, to enable slot attention in complex scenes, they
leverage additional object signals to guide the attention during training,
such as flow [42], depth [25], or motion segmentation [5]. Object signals may
lead to misinterpretations in atomic activity recognition since not all
objects participate in activities (e.g., pedestrians strolling on the sidewalk
and vehicles waiting at traffic lights). In this work, we propose a novel
action-centric slot attention-based framework and demonstrate the
effectiveness of the framework on multiple datasets.

Figure 3: The top of the figure illustrates the proposed framework. Action-
slot takes video as input and uses a CNN encoder to extract feature patches.
All patches are then processed with individual slots simultaneously to find
the most relevant spatial-temporal patches corresponding to each action slot.
The updated action slots are fed into a fully connected layer to predict the
probability of the corresponding action class, excluding the background slot.
The bottom of the figure depicts the attention maps of action and background
slots. We propose to incorporate a background mask $M_{\mathtt{bg}}$ to
supervise the background slot. The design facilitates other action slots to
capture action signals. Furthermore, we design a regularization for slots
allocated to negative classes (e.g., Z3-Z4: C) using an all-zero mask
$M_{\mathtt{neg}}$.

##  3 Method

In this section, we first introduce the background of slot attention. Then, we
provide an overview of the problem definition in multi-label atomic activity
recognition and the proposed method. Finally, we explain the specific
modifications we make to the slot attention for our task.

###  3.1 Preliminary: Slot Attention

The Slot Attention module [48] can be viewed as a clustering algorithm that
maps image patches input to a set of $K$ output slots $S\in\mathbb{R}^{K\times
D_{\mathtt{slot}}}$, where $D_{\mathtt{slot}}$ is the dimension of each slot.
Specifically, the slots $S$ are first initialized by randomly sampling $K$
vectors from a Gaussian distribution, where the parameter $K$ is usually
defined as the maximum number of objects in an image. An input frame, i.e.,
image features $F\in$ $\mathbb{R}^{H\times W\times D_{\mathtt{in}}}$ with size
$H\times W$ and dimension $D_{\mathtt{in}}$, is first flattened to patch
tokens $F^{{}^{\prime}}\in$ $N\times D_{\mathtt{in}}$, where $N=H\times W$.
Then the tokens are mapped to the slots using the dot product attention
module. That is, the attention weight can be calculated as
$A=\frac{1}{\sqrt{D}}k(F^{{}^{\prime}})\cdot q(S)\in\mathbb{R}^{N\times K}$,
and $q$ and $k$ are linear transformations that map the input $F$ and slots
$S$ to a common dimension $D$.

We obtain the updated values for slots through $U=\bar{A}^{T}\cdot
v(F^{{}^{\prime}})\in\mathbb{R}^{K\times D}$, where $\bar{A}$ is the
normalized attention weight calculated via the softmax operation and $v$ is a
linear transformation. Finally, the slots are updated via a GRU:
$S^{{}^{\prime}}=\mathtt{GRU}(S,U)$ [17]. To refine the slots, the updating
process repeats $M$ iterations [48, 42, 25]. Note that, the key difference
between classical attention [73] and slot attention is that the attention
weights $A$ are normalized with the softmax operation slot-wise instead of
token-wise. This difference enables slots to compete with each other so that
each slot attends to different relevant regions of input. To extend slot
attention to video tasks, the previous works [42, 25, 5] propagate slots
recurrently, i.e., $S^{t}=\mathtt{GRU}(S^{t-1},U)$.

###  3.2 Overview of Proposed Method

#### Problem Formulation.

Given a video clip $V_{i}$ with $T$ image frames $\\{I_{t}^{i}\\}_{t=1}^{T}$,
our goal is to recognize whether there are any atomic activities $Y$
presenting in the video. The ground truth of $Y^{i}$ for video $V_{i}$ is a
binary multi-label vector, i.e., $Y^{i}=\\{y_{c}\\}_{c=1}^{N_{\mathtt{cl}}}$,
where $y_{c}$ is 1 if the corresponding activity appears and vice versa, and
$N_{\mathtt{cl}}$ denotes the total number of possible activities. Note that,
we do not constrain when a certain activity appears, e.g., $y_{c}$ can happen
in any image frames of $\\{I_{t}^{i}\\}_{t=1}^{T}$, or may appear multiple
times in one video.

#### Overview of Action-slot.

We propose Action-slot, an action-centric slot-attention-based framework to
decompose multiple atomic activities from videos. Our idea is to assign each
slot to learn specific action-centric representations for the corresponding
atomic activity. Figure 3 presents the overview of Action-slot. Different from
existing slot attention algorithms designed for unsupervised object discovery
and tracking [42, 25, 5], we make the following three modifications. First, we
allocate a fixed number of slots $K$, which is equal to the number of atomic
activities $N_{\mathtt{cl}}$. We supervise the prediction of each action slot
with the corresponding ground truth label $y_{c}$, i.e., whether the atomic
activity $c$ appears. Second, we introduce a background slot to focus on the
regions that are not possible to present an activity, so that the other action
slots can compete with this background slot. In other words, the design forces
other action slots to pay attention to the important regions. Third, we
discourage action slots associated with negative classes (i.e., those
activities not present in a video), from attending to any regions. Lastly, we
update all action slots in a parallel fashion by considering all the image
frames together, instead of recurrently updating slots along the temporal
domain utilized in [42, 25, 5].

###  3.3 Action-slot

In this section, we describe the details of our design for learning action-
centric representations. We follow [5] to use learnable parameters for slot
initializations. We extract image features $F\in\mathbb{R}^{T\times H\times
W}$ from the convolution-based encoder [34, 12, 29, 28] for each frame in a
video clip and then flatten them to $F^{{}^{\prime}}\in\mathbb{R}^{N\times
D_{\mathtt{in}}}$, where $N=T\times H\times W$. We add learnable 3D spatial-
temporal positional embeddings $E\in\mathbb{R}^{T\times H\times W}$ to the
tokens $F^{{}^{\prime}}$ [7, 4, 27, 5, 71].

#### Allocated slot.

We then define a set of slots $\\{S_{k}\\}_{k=1}^{K}$, where
$K=N_{\mathtt{cl}}$, representing the number of action slots (i.e., number of
activity classes). In contrast to the previous slot attention works [48, 42,
25, 82] where they do not specify a task to each slot and use a Hungarian
matcher for bipartite matching with objects, we instead allocate each slot a
class of activity, namely, action slots. Each action slot is processed with an
independent binary classifier to output a prediction $\hat{y}_{c}$ for each
class $c$, where a binary cross entropy (BCE) function is used as the
objective for each data sample:
$L_{\mathtt{act}}=\sum_{c=1}^{N_{\mathtt{cl}}}\textnormal{BCE}(\hat{y}_{c},y_{c})$.

Figure 4:  (a) Our parallel scheme considers updating slots based on the
spatial-temporal features for all the frames. (b) The slots are updated
recurrently along the temporal dimension, where each time the slot only
considers a frame-wise feature.

#### Parallel updating.

We update slots across image frames in a parallel manner, as shown in Figure 4
(a). Specifically, we calculate normalized attention weight
$\bar{A}\in\mathbb{R}^{N\times K}$ while considering temporal dimension at the
same time, i.e., $N=T\times H\times W$. We then update the $K$ slots with
attention weights $\bar{A}$ in a single pass. This is different from the
previous works [42, 25, 5] where they update slots across frames recurrently,
namely, using the previous state of slots and current feature maps to update
the current state of slots for $T$ iterations, as shown in Figure 4 (b).

#### Background slot and attention guidance.

It is non-trivial to provide guidance for positive classes as the objects are
not necessarily involved in activities all the time, and not all objects are
involved in activities. Therefore, we propose to use a background slot that is
not supervised by any activity classes. By the softmax operation in slot
attention, this background slot would attend regions that are not relevant to
any activities and force other action slots to focus on other regions, as
shown in the bottom of Figure 3. To further enhance the effectiveness of the
background slot, we provide an attention mask $M_{\mathtt{bg}}$ as the
supervision, directly guiding the background slot where to pay attention. The
mask is generated by excluding pixels with the classes of vehicles,
pedestrians, drivable areas, crosswalks, and sidewalks. We guide the
background attention via the loss function:
$L_{\mathtt{bg}}=\textnormal{BCE}(\bar{A}_{\mathtt{bg}},M_{\mathtt{bg}})$,
where $\bar{A}_{\mathtt{bg}}$ is the normalized attention from the background
slot.

#### Regularization for action slots.

Instead of providing attention guidance for positive classes, which is non-
trivial as explained previously, we introduce a regularization term to
discourage action slots associated with negative classes (i.e., those
activities not present in a video), from attending to any regions. We
construct an attention mask $M_{\mathtt{neg}}$ wherein all element values are
set to 0. We design a loss function:
$L_{\mathtt{neg}}=\sum_{\\{c|y_{c}=0\\}}\textnormal{BCE}(\bar{A}_{c},M_{\mathtt{neg}})$,
where $\bar{A}_{c}$ is the normalized attention output of slot $S_{c}$ for
negative class $c$. By regularizing the action slots with negative classes,
the competition mechanism makes slots with positive classes more competitive
in the important regions. Note that the regularization does not require
additional annotations. Finally, the full objective function of Action-slot
is:
$L_{\mathtt{all}}=L_{\mathtt{act}}+w_{\mathtt{bg}}L_{\mathtt{bg}}+w_{\mathtt{neg}}L_{\mathtt{neg}},$
where $w_{\mathtt{bg}}$ and $w_{\mathtt{neg}}$ are the weights to balance two
terms (set as 0.5 and 1, respectively in this paper).

##  4 Experiments

###  4.1 Datasets

OATS [3]. The dataset was collected in San Francisco with an instrumented
vehicle and comprises 1026 labeled clips and is divided into 3 splits for
cross-validation. OATS labels 59 traffic atomic activity classes. We follow
the same experimental protocol to train and evaluate the 35 classes. The
dimensions of an image are 1200 $\times$ 1920 pixels. We downsample the image
size to 224 $\times$ 224, as in [3].

TACO. We construct the TACO dataset in the CARLA simulator [23] to attain a
balanced distribution encompassing 64 classes of traffic atomic activities.
The TACO dataset comprises 5178 clips. Among them, 1148 are used for testing.
We collect image and instance segmentation data with a size of 512 $\times$
1536 pixels, and subsequently downsample each frame to 256 $\times$ 768 as
input to the models.

nuScenes [9]. We annotate the $train\\_val$ set of nuScenes dataset with
atomic activities. The $train\\_val$ set comprises 850 videos. Our annotation
protocol yields a total of 426 short clips, each comprising 16 frames. The
annotated nuScenes encompasses 42 classes of atomic activity. Please refer to
the supplementary material for additional details on the dataset construction
and annotation.

###  4.2 Baselines

Video-level models. We implement our baselines using the PyTorchVideo [26]
library to construct different video-level architectures, including I3D [12],
X3D [28], CSN [72], SlowFast [29], MViT [27], and VideoMAE [71]. Note that we
modify the models to perform multi-label prediction with one linear layer that
outputs $N_{\mathtt{cl}}$ class channels.

Object-aware models. Object-aware models [6, 75, 3] first extract object
features via the RoI-Align operation [35]. To learn the relations between
objects, ORN [6], ARG [75], and OATS [3] uses MLP, Graph Convolutional Network
(GCN) [43], and spatical-temporal GCN [53] to enhance object features. OATS
also uses an additional graph branch to encode low-level object trajectories.
To extract trajectories of objects, we use Deep OC-SORT [51], the state-of-
the-art model on the MOT20 benchmark [20] in terms of HOTA [50]. We follow
OATS by setting the number of tracklets as 20 for the input of the models. To
encourage the object-aware models [6, 75] to learn the context information, we
concatenate object features with the global features, which are obtained by
convolving the $5^{\mathtt{th}}$ ResBlock of X3D [28] with a 3D kernel size of
1. These models then aggregate all frames’ object features by summing them
channel-wise, followed by a linear layer to output multi-class predictions for
each proposal, with a channel number of $N_{cl}$ plus one negative class. Note
that we do not implement the model proposed in OATS [3] due to the absence of
source code.

Slot-based models. Slot-based models use the last ConvBlock of the encoders
[34, 28] as backbone features. Note that we remove the last projection layer
of X3D, which makes the size of some slot-based models slightly smaller than
X3D. We re-implement existing slot-attention-based methods for videos [42, 25,
5, 82] to use the proposed “allocated slots” for multi-label atomic activity
recognition. Specifically, SAVi [42] randomly initialize slots for each
forward pass from a learned distribution, while MO [5] and Slot-VPS [82] treat
slots as learnable queries [11]. For the slot update mechanism, SAVi and MO
recurrently update slots in time, while Slot-VPS uses an extra self-attention
[73] to update slots across frames. We then use their slots in the last frame
as the input to the classifier.

Table 1: Quantitative results on the OATS dataset. The symbol ${\ddagger}$
denotes the re-implementation of slot-based methods, where each slot is
allocated to a specific atomic activity class. S1, S2, and S3 represents the
three defined splits in the OATS dataset.

Method | Backbone | Pretrain | S1 | S2 | S3 | mAP
---|---|---|---|---|---|---
CSN [72, 3] | ResNet152 | IG65M | 12.1 | 12.6 | 12.9 | 12.5
TPN [79, 3] | ResNet50 | ImageNet | 11.6 | 13.3 | 12.9 | 12.6
SlowOnly [29, 3] | ResNet50 | ImageNet | 11.2 | 14.7 | 12.9 | 13.0
SlowFast [29, 3] | ResNet50 | None | 10.8 | 15.1 | 14.5 | 13.5
I3D (NL) [74, 3] | ResNet50 | ImageNet | 11.9 | 15.5 | 14.0 | 13.8
I3D [12, 3] | ResNet50 | ImageNet | 11.8 | 14.3 | 16.8 | 14.3
ORN [6, 3] | ResNet50 | ImageNet | 16.8 | 13.4 | 18.1 | 16.1
ARG [75, 3] | Inceptionv3 | ImageNet | 20.2 | 21.3 | 19.3 | 20.3
OATS [3] | Inceptionv3 | ImageNet | 24.3 | 28.6 | 27.2 | 26.7
SAVi${\ddagger}$ [42, 25] | ResNet50 | ImageNet | 21.1 | 22.4 | 22.5 | 22.0
MO${\ddagger}$ [5] | ResNet50 | ImageNet | 14.3 | 15.6 | 18.2 | 16.0
Slot-VPS${\ddagger}$ [82] | ResNet50 | ImageNet | 15.7 | 17.8 | 17.2 | 16.9
Action-Slot (Ours) | ResNet50 | ImageNet | 26.6 | 28.6 | 30.8 | 28.6
I3D [12] | ResNet50 | Kinetics-400 | 21.7 | 24.6 | 24.4 | 23.6
X3D [28] | N/A | Kinetics-400 | 30.4 | 33.2 | 30.6 | 31.4
CSN [72] | ResNet101 | Kinetics-400 | 43.1 | 47.1 | 44.3 | 44.8
SlowFast [29] | ResNet50 | Kinetics-400 | 36.1 | 36.6 | 34.2 | 35.6
ORN [6] | X3D | Kinetics-400 | 19.3 | 22.5 | 23.6 | 21.8
ARG [75] | X3D | Kinetics-400 | 24.8 | 25.9 | 29.3 | 26.7
SAVi${\ddagger}$ [42, 25] | X3D | Kinetics-400 | 19.0 | 22.1 | 21.6 | 20.9
MO${\ddagger}$ [5] | X3D | Kinetics-400 | 25.3 | 25.0 | 24.2 | 24.8
Slot-VPS${\ddagger}$ [82] | X3D | Kinetics-400 | 24.7 | 24.6 | 25.5 | 24.9
Action-Slot (Ours) | X3D | Kinetics-400 | 48.1 | 47.7 | 48.8 | 48.2

Table 2: Quantitative results on the TACO dataset. “Seq” denotes the input
sequence length. C, K, P, C+, K+, and P+ denote activities involved with
different types of road users. The symbol ${\ddagger}$ denotes the re-
implementation of slot-based methods, where each slot is allocated to a
specific atomic activity class.

Method | Para. (M) | Seq | C | K | P | C+ | K+ | P+ | mAP
---|---|---|---|---|---|---|---|---|---
I3D [12] | 27.3 | 8 | 27.3 | 19.4 | 30.5 | 34.6 | 33.6 | 34.8 | 29.7
X3D [28] | 3.0 | 16 | 37.5 | 20.3 | 34.6 | 56.3 | 51.5 | 38.8 | 38.3
CSN [72] | 21.4 | 32 | 43.5 | 35.5 | 43.0 | 52.5 | 46.1 | 43.4 | 44.0
SlowFast [29] | 33.7 | 16 | 33.9 | 19.7 | 36.7 | 39.9 | 42.6 | 41.0 | 35.2
MViT [27] | 36.6 | 16 | 21.4 | 13.8 | 26.3 | 43.7 | 30.0 | 33.8 | 27.9
VideoMAE [27] | 57.9 | 16 | 30.6 | 18.6 | 27.1 | 51.6 | 33.0 | 37.4 | 33.1
ORN [6] | 4.8 | 16 | 25.5 | 15.8 | 24.6 | 31.6 | 19.8 | 13.9 | 22.2
ARG [75] | 12.2 | 16 | 27.8 | 15.0 | 27.2 | 35.6 | 13.7 | 20.0 | 23.2
SAVi${\ddagger}$ [42, 25] | 2.3 | 16 | 19.2 | 13.6 | 27.5 | 23.3 | 25.9 | 35.9 | 23.3
MO${\ddagger}$ [5] | 2.3 | 16 | 34.2 | 24.9 | 39.2 | 38.3 | 39.4 | 37.7 | 35.3
Slot-VPS${\ddagger}$ [82] | 3.5 | 16 | 31.9 | 21.3 | 32.0 | 51.9 | 44.7 | 31.3 | 36.0
Action-Slot (Ours) | 2.3 | 16 | 48.1 | 41.2 | 49.2 | 70.1 | 62.6 | 52.8 | 54.4

Pretraining and backbone choice. We first follow the setting of OATS and
implement Action-slot and slot-based baselines with ResNet50 [34] pretrained
on ImageNet [21]. To further study the effect of pre-trained models, we train
models with state-of-the-art 3D bakcbones [12, 29, 28], pretrained on
Kinetics-400 [41], for the experiments on both OATS and TACO datasets.
Specifically, we use X3D [28] as the backbone by default because of the
compact model size. Please refer to the supplementary material for the
additional implementation details.

###  4.3 Implementation Details

Background masks. We extract background masks from the instance segmentation
collected in TACO. For OATS and nuScenes, we use an off-the-shelf DeepLabV3+
[15].

Metrics. We follow the common practice in multi-label action recognition [80,
65, 31, 3] to report mean average precision (mAP).

Visualization. We visualize attention maps of each atomic activity according
to the allocated action slots to validate if Action-slot learns action-centric
representations. We keep pixels with an attention score greater than 0.5 on
OATS and 0.2 on TACO.

Table 3: Ablation study of Action-slot. The reported results are the average of the 3 splits in the OATS dataset. ID | Allocated | Update | BG Slot | $L_{\mathtt{bg}}$ | $L_{\mathtt{neg}}$ | mAP
---|---|---|---|---|---|---
1 |  | parallel |  |  |  | 10.8
2 | ✓ | recurrent |  |  |  | 24.8
3 | ✓ | parallel |  |  |  | 42.7
4 | ✓ | parallel | .✓ |  |  | 40.8
5 | ✓ | parallel | ✓ | ✓ |  | 43.6
6 | ✓ | parallel | ✓ |  | ✓ | 43.0
7 | ✓ | parallel | ✓ | ✓ | ✓ | 48.2

Table 4: Comparisons of Action-slot and object-level guidance on TACO across different numbers of road users (denoted as $N$) present in a video. BG and Neg denote background slot and regularization in our method. |  $N$ $\leq$ 5 |  5 $<$ $N$ $\leq$ 15 |  $N$ $>$ 15 |
---|---|---|---|---
object | 49.4 | 47.7 | 43.5 |
BG+Neg | 55.2 | 50.9 | 46.3 |

Table 5: Results of various backbones for Action-slot on TACO. | mAP
---|---
I3D | 29.7
Action-slot | 37.6 (+7.9)
X3D [28] | 37.8
Action-slot |  54.4 (+16.6)
SlowFast [29] | 35.2
Action-slot | 46.7 (+11.5)

###  4.4 Main Results

OATS. In Table 1, we show experimental comparisons with video-level-based,
object-aware, and slot-attention-based methods. In the top group, we use the
ImageNet pre-trained ResNet50, following the evaluation protocol designed in
OATS [3] We further compare all methods with advanced visual backbones
pretrained from Kinetics-400 [41] and report the corresponding results in the
bottom group. In both ImageNet and Kinetics pretrained settings, Action-slot
achieves state-of-the-art performance on the OATS dataset. Specifically,
Action-slot improves the performance of atomic activity recognition by
identifying the region of interest via the proposed slot attention mechanism,
without explicit perception guidance.

TACO. We establish a new benchmark on the proposed TACO dataset to validate
the effectiveness of Action-slot for a wide range of atomic activities, which
is challenging for the OATS dataset [3] due to its imbalanced distribution. In
the top group of Table 2, we evaluate two recent transformer-based models,
i.e., MViT [27] and VideoMAE [71]. The models show worse performance compared
to other video-level methods, which could be attributed to their data-hungry
nature [24].

Object-aware models [6, 75, 3] show suboptimal performance, particularly for
atomic activities related to grouped road users, as reported in the middle
group. The findings indicate that the relation modeling of objects learned
from the MLP and GCN [43] is not effective for the task because it requires a
holistic understanding of contextual information and motions of objects.

In the bottom group of Table 2, we re-implement the object-centric slot-based
models with allocated slots (denoted by ${\ddagger}$). Action-slot outperforms
all models by a large margin on activities with all types of road users
because of our parallel architecture design and the proposed attention
guidance. Particularly, Action-slot excels in activities involved with grouped
road users. It is worth noting that the Action-slot is powerful yet efficient
in terms of model size. The results validate the effectiveness of action-
centric representations over the existing action recognition models and
object-centric representations.

###  4.5 Ablation Study

We conduct ablation studies in Table 5 to understand the effect of our
designed components.

Non-Allocated vs. Allocated. We show that adding allocated slots significantly
improves the performance of Non-Allocated by 31.9% (ID 1 vs. 3). The results
suggest the need to revisit the design of slot attention for our task.

Recurrent vs. Parallel. We demonstrate that updating action slots in a
parallel fashion obtains a 17.9% performance gain (ID 2 vs. 3), compared to
conventional methods using a recurrent manner.

Background slot and attention guidance $L_{\mathtt{bg}}$. We find that simply
adding a non-allocated background slot without mask supervision slightly harms
the performance (ID 3 vs. 4), which is different from the observation in
previous slot attention literature [5, 82]. Our insight is that the non-
allocated slot may distract the action slots’ attention. With the background
mask supervision $L_{\mathtt{bg}}$, the model shows additional performance
gain compared to the one only using the allocated slot (ID 3 vs. 5).

Action slots regularization $L_{\mathtt{neg}}$. We regularize action slots
with the $L_{\mathtt{neg}}$ term to discourage action slots allocated to
negative classes from attending any regions in spatial-temporal features. This
design enhances the likelihood of other action slots allocated to positive
classes identifying the region of interest more effectively. We demonstrate
that the inclusion of the regularization term $L_{\mathtt{neg}}$ further
improves the one only using the allocated slot (ID 3 vs. 6).

The final model (ID 7 in Table 5) that incorporates allocated slots, parallel
updating, background slot with attention guidance, and action slot
regularization, performs the best on OATS, which validates the efficacy of
Action-slot.

Action-slot |  |  |  |  |
---|---|---|---|---|---
MO${\ddagger}$ [5] |  |  |  |  |
Figure 5:  Visualization of attention maps learned from OATS. Colored masks represent the attention of the activity slots Z4-Z3:C and C2-C1:P+. Note that, while MO successfully predicts the activity C2-C1:P+, the corresponding attention scores are very low. Action-slot |  |  |  |  |
---|---|---|---|---|---
MO${\ddagger}$ [5] |  |  |  |  |
Figure 6:  Visualization of attention maps learned from TACO. Colored masks
are from the allocated slots of activity Z4-Z1:B and Z1-Z2:K+. Note that
Z4-Z1:K appears twice in the 1st and 5th frames independently.  Table 6:
Effectiveness of using our TACO dataset for representation pretraining and
then finetuning on real-world OATS and nuScenes datasets. Kinetics denotes
finetuning from the Kinetics-400 pretrained weights.

| OATS | nuScenes
---|---|---
| Kinetics
---
| +TACO
---
| Kinetics
---
| +TACO
---
X3D [28] | 31.4 | 34.9 (+3.5) | 19.8 | 27.8 (+8.0)
ARG [75] | 26.7 | 31.3 (+4.6) | 12.2 | 17.0 (+4.8)
Action-slot | 48.2 |  59.5 (+11.3) | 23.6 |  32.3 (+8.7)

##  5 More Analysis and Discussions

Action-slot vs. Object-level guidance. We conduct experiments on the TACO
dataset to evaluate the impact of different numbers of road users in videos
with respect to different attention guidance mechanisms in our Actio-slot
framework. Specifically, we compare the two attention guidance mechanisms
discussed in Section 3.3 and the object-level guidance. For object-level
guidance, we use instance segmentation collected in TACO to obtain masks for
vehicles and pedestrians. Hungarian matcher is used to assign object masks to
slots, following MO [5]. Table 5 shows that Action-slot with strong ground
truth object-level supervisions performs worse, compared to the proposed
attention guidance. The observation indicates that only considering object
cues may not effectively tackle our task because not every object is always
involved in the same atomic activity.

Action-slot with different backbones. Table 5 reports the performance of
Action-slot using different backbone. We choose I3D [12], X3D [28], and
SlowFast [29] for experiments. Action-slot significantly improves all 3D CNN
methods, showing great generalization ability. Among them, X3D achieves the
most improvement. We hypothesize that this is because X3D’s features retain
the original temporal dimension, affording Action-slot greater spatial-
temporal information to analyze atomic activities. In contrast, other backbone
encoders downsample the temporal dimension; for instance, SlowFast takes a
video sequence of length 16 as input and reduces the temporal dimension to 4.

Representation pretraining on TACO. We demonstrate that the proposed TACO
dataset can enhance the efficacy of atomic activity recognition on real-world
datasets through representation pretraining. Specifically, we first pretrain
the model on TACO and then fine-tune it on real-world datasets, including OATS
[3] and a newly annotated nuScenes dataset [9]. We report the average mAP of
the three splits in the OATS dataset in Table 6. Due to the large-scale and
balanced distribution of TACO, all models show a significant improvement on
both real-world datasets when pretrained on TACO. Notably, Action-slot
outperforms all baselines, showcasing its robustness and generalizability.

Qualitative Results. We visualize the attention maps learned using the re-
implemented MO [5] and Action-slot on the OATS and TACO datasets in Figure 5
and Figure 6, respectively. Action-slot demonstrates the capability to
decompose multiple atomic activities from intricate scenes in both datasets.
In Figure 5, Action-slot localizes the group activity C2-C1:P+, while MO [5]
fails to attend any relevant regions. Moreover, Action-slot recognizes two
independent Z4-Z1:K activities, instead of predicting them as a group
activity, i.e., Z4-Z1:K+ in Figure 6. The evidence justifies the capability of
Action-slot that learns action-centric representations. More interestingly,
Action-slot attends to relevant regions when road users initiate an action and
cease attention once the action is completed, as seen in the case of the
bicyclists in the fourth frame of Figure 6.

##  6 Conclusions

We present Action-slot, a slot attention-based approach that learns visual
action-centric representations, capturing both motion and contextual
information. We introduce several crucial designs in slot attentions that
enable decomposing multiple atomic activities in videos, without the need for
explicit perception guidance. We conduct comprehensive experiments to validate
the proposed method on both OATS and our TACO dataset, comparing it against
various action recognition baseline methods. In addition, we demonstrate that
Action-slot learns meaningful attention maps, identifying objects in actions
without explicit object-level guidance. Lastly, we show that the TACO dataset
enhances the efficacy of multi-label atomic activity recognition on real-world
datasets through representation pretraining.

Limitation. Although Action-slot demonstrates favorable quantitative and
qualitative performance, we observe that it is still challenging to handle
cases when two activities occlude with each other. In these cases, the
corresponding action slots may confuse where they should attend. This
observation is closely relevant to the tracking task under occlusion, where
frequent ID switches are often observed. We hope our findings can encourage
the community to explore more advanced designs for action-centric
representations.

## References

  * car [2022] CARLA Autonomous Driving Challenge.  https://carlachallenge.org/, 2022.
  * sce [2023] ScenarioRunner for CARLA.  https://github.com/carla-simulator/scenario_runner, 2023.
  * Agarwal and Chen [2023] Nakul Agarwal and Yi-Ting Chen.  Ordered atomic activity for fine-grained interactive traffic scenario understanding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 8624–8636, 2023.
  * Arnab et al. [2021] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  ViViT: A Video Vision Transformer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 6836–6846, 2021.
  * Bao et al. [2022] Zhipeng Bao, Pavel Tokmakov, Allan Jabri, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert.  Discovering Objects that Can Move.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11789–11798, 2022.
  * Baradel et al. [2018] Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori.  Object Level Visual Reasoning in Videos.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018.
  * Bertasius et al. [2021] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In _International Conference on Machine Learning (ICML)_ , 2021.
  * Biza et al. [2023] Ondrej Biza, Sjoerd van Steenkiste, Mehdi SM Sajjadi, Gamaleldin F Elsayed, Aravindh Mahendran, and Thomas Kipf.  Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames.  _International Conference on Machine Learning (ICML)_ , 2023.
  * Caesar et al. [2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A Multimodal Dataset for Autonomous Driving.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Cao et al. [2022] Yulong Cao, Chaowei Xiao, Anima Anandkumar, Danfei Xu, and Marco Pavone.  AdvDO: Realistic Adversarial Attacks for Trajectory Prediction.  In _European Conference on Computer Vision (ECCV)_. Springer, 2022.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end Object Detection with Transformers.  In _European Conference on Computer Vision (ECCV)_ , pages 213–229. Springer, 2020.
  * Carreira and Zisserman [2017] Joao Carreira and Andrew Zisserman.  Quo vadis, Action Recognition? A New Model and the Kinetics Dataset.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6299–6308, 2017.
  * Chen et al. [2016] Chao-Yeh Chen, Wongun Choi, and Manmohan Chandraker.  Atomic Scenes for Scalable Traffic Scene Recognition in Monocular Videos.  In _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 1–9, 2016.
  * Chen and Krähenbühl [2022] Dian Chen and Philipp Krähenbühl.  Learning from all vehicles.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 17222–17231, 2022.
  * Chen et al. [2018] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 801–818, 2018.
  * Chitta et al. [2021] Kashyap Chitta, Aditya Prakash, and Andreas Geiger.  Neat: Neural attention fields for end-to-end autonomous driving.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15793–15803, 2021.
  * Cho et al. [2014] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2014.
  * Choi et al. [2009] Wongun Choi, Khuram Shahid, and Silvio Savarese.  What are they doing?: Collective activity classification using spatio-temporal relationship among people.  In _IEEE International Conference on Computer Vision Workshops (ICCV-W)_ , pages 1282–1289, 2009.
  * Cui et al. [2019] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie.  Class-balanced loss based on effective number of samples.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9268–9277, 2019.
  * Dendorfer et al. [2020] Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taixé.  MOT20: A Benchmark for Multi Object Tracking in Crowded Scenes.  _arXiv preprint arXiv:2003.09003_ , 2020.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A Large-scale Hierarchical Image Database.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 248–255, 2009.
  * Ding et al. [2023] Wenhao Ding, Haohong Lin, Bo Li, and Ding Zhao.  Causalaf: causal autoregressive flow for safety-critical driving scenario generation.  In _Conference on Robot Learning_ , pages 812–823. PMLR, 2023.
  * Dosovitskiy et al. [2017] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.  CARLA: An Open Urban Driving Simulator.  In _Proceedings of the 1st Annual Conference on Robot Learning_ , pages 1–16, 2017.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Elsayed et al. [2022] Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf.  SAVi++: Towards End-to-End Object-centric Learning from Real-world Videos.  _Conference on Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Fan et al. [2021a] Haoqi Fan, Tullie Murrell, Heng Wang, Kalyan Vasudev Alwala, Yanghao Li, Yilei Li, Bo Xiong, Nikhila Ravi, Meng Li, Haichuan Yang, Jitendra Malik, Ross Girshick, Matt Feiszli, Aaron Adcock, Wan-Yen Lo, and Christoph Feichtenhofer.  PyTorchVideo: A Deep Learning Library for Video Understanding.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021a.  https://pytorchvideo.org/.
  * Fan et al. [2021b] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer.  Multiscale vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 6824–6835, 2021b.
  * Feichtenhofer [2020] Christoph Feichtenhofer.  X3D: Expanding Architectures for Efficient Video Recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 203–213, 2020.
  * Feichtenhofer et al. [2019] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  SlowFast Networks for Video Recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 6202–6211, 2019.
  * Geiger et al. [2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012.
  * Gu et al. [2018] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al.  AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6047–6056, 2018.
  * Gu et al. [2023] Junru Gu, Chenxu Hu, Tianyuan Zhang, Xuanyao Chen, Yilun Wang, Yue Wang, and Hang Zhao.  Vip3d: End-to-end visual trajectory prediction via 3d agent queries.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5496–5506, 2023.
  * Hanselmann et al. [2022] Niklas Hanselmann, Katrin Renz, Kashyap Chitta, Apratim Bhattacharyya, and Andreas Geiger.  King: Generating safety-critical driving scenarios for robust imitation via kinematics gradients.  In _European Conference on Computer Vision_ , pages 335–352. Springer, 2022.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, 2016.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask R-CNN.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 2961–2969, 2017.
  * Hu et al. [2023] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li.  Planning-oriented autonomous driving.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Ibrahim et al. [2016] Mostafa S Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, and Greg Mori.  A hierarchical deep temporal model for group activity recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1971–1980, 2016.
  * Jain et al. [2015] Ashesh Jain, Hema S. Koppula, Bharad Raghavan, Shane Soh, and Ashutosh Saxena.  Car that Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2015.
  * Jiang et al. [2014] Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar.  THUMOS challenge: Action recognition with a large number of classes.  http://crcv.ucf.edu/THUMOS14/, 2014.
  * Johnson et al. [2017] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2901–2910, 2017.
  * Kay et al. [2017] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  _arXiv preprint arXiv:1705.06950_ , 2017.
  * Kipf et al. [2022] Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff.  Conditional Object-Centric Learning from Video.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Kipf and Welling [2016] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ , 2016.
  * Kuehne et al. [2011] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre.  HMDB: A Large Video Database for Human Motion Recognition.  In _2011 International Conference on Computer Vision (ICCV)_ , pages 2556–2563, 2011.
  * Li et al. [2019] Max Guangyu Li, Bo Jiang, Zhengping Che, Xuefeng Shi, Mengyao Liu, Yiping Meng, Jieping Ye, and Yan Liu.  DBUS: Human Driving Behavior Understanding System.  In _ICCV Workshops_ , pages 2436–2444, 2019.
  * Lin et al. [2014] Dahua Lin, Sanja Fidler, Chen Kong, and Raquel Urtasun.  Visual Semantic Search: Retrieving Videos via Complex Textual Queries.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2014.
  * Liu et al. [2020] Bingbin Liu, Ehsan Adeli, Zhangjie Cao, Kuan-Hui Lee, Abhijeet Shenoi, Adrien Gaidon, and Juan Carlos Niebles.  Spatiotemporal relationship reasoning for pedestrian intent prediction.  pages 3485–3492, 2020.
  * Locatello et al. [2020] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf.  Object-centric Learning with Slot Attention.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 33:11525–11538, 2020.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled Weight Decay Regularization.  In _International Conference on Learning Representations_ , 2019.
  * Luiten et al. [2021] Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe.  HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking.  _International Journal of Computer Vision_ , 129:548–578, 2021.
  * Maggiolino et al. [2023] Gerard Maggiolino, Adnan Ahmad, Jinkun Cao, and Kris Kitani.  Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification.  _arXiv preprint arXiv:2302.11813_ , 2023.
  * Malla et al. [2020] Srikanth Malla, Behzad Dariush, and Chiho Choi.  TITAN: Future Forecast using Action Priors.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11186–11196, 2020.
  * Mohamed et al. [2020] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel.  Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 14424–14432, 2020.
  * Nagarajan et al. [2022] Tushar Nagarajan, Santhosh Kumar Ramakrishnan, Ruta Desai, James Hillis, and Kristen Grauman.  Egoenv: Human-centric environment representations from egocentric video.  _arXiv preprint arXiv:2207.11365_ , 2022.
  * Naphade et al. [2023] Milind Naphade, Shuo Wang, David C. Anastasiu, Zheng Tang, Ming-Ching Chang, Yue Yao, Liang Zheng, Mohammed Shaiqur Rahman, Meenakshi S. Arya, Anuj Sharma, Qi Feng, Vitaly Ablavsky, Stan Sclaroff, Pranamesh Chakraborty, Sanjita Prajapati, Alice Li, Shangru Li, Krishna Kunadharaju, Shenxin Jiang, and Rama Chellappa.  The 7th AI City Challenge.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , 2023.
  * Ohn-Bar et al. [2020] Eshed Ohn-Bar, Aditya Prakash, Aseem Behl, Kashyap Chitta, and Andreas Geiger.  Learning situational driving.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Ramanishka et al. [2018] Vasili Ramanishka, Yi-Ting Chen, Teruhisa Misu, and Kate Saenko.  Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7699–7707, 2018.
  * Rasouli et al. [2019] Amir Rasouli, Iuliia Kotseruba, Toni Kunic, and John K. Tsotsos.  PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction.  In _The IEEE International Conference on Computer Vision (ICCV)_ , 2019.
  * Rempe et al. [2022] Davis Rempe, Jonah Philion, Leonidas J. Guibas, Sanja Fidler, and Or Litany.  Generating useful accident-prone driving scenarios via a learned traffic prior.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Rempe et al. [2023] Davis Rempe, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, and Or Litany.  Trace and pace: Controllable pedestrian animation via guided trajectory diffusion.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Renz et al. [2022] Katrin Renz, Kashyap Chitta, Otniel-Bogdan Mercea, A Koepke, Zeynep Akata, and Andreas Geiger.  Plant: Explainable planning transformers via object-level representations.  _arXiv preprint arXiv:2210.14222_ , 2022.
  * Sajjadi et al. [2022] Mehdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf.  Object scene representation transformer.  In _Advances in Neural Information Processing Systems_ , pages 9512–9524. Curran Associates, Inc., 2022.
  * Segal et al. [2020] Sean Segal, Eric Kee, Wenjie Luo, Abbas Sadat, Ersin Yumer, and Raquel Urtasun.  Universal Embeddings for Spatio-Temporal Tagging of Self-Driving Logs.  In _Conference on Robot Learning (CoRL)_ , 2020.
  * Shao et al. [2023] Hao Shao, Letian Wang, Ruobing Chen, Steven L Waslander, Hongsheng Li, and Yu Liu.  Reasonnet: End-to-end driving with temporal and global reasoning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13723–13733, 2023.
  * Sigurdsson et al. [2016] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.  Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 510–526. Springer, 2016.
  * Singh et al. [2022] Gurkirt Singh, Stephen Akrigg, Manuele Di Maio, Valentina Fontana, Reza Javanmard Alitappeh, Salman Khan, Suman Saha, Kossar Jeddisaravi, Farzad Yousefi, Jacob Culley, et al.  ROAD: The ROad event Awareness Dataset for Autonomous Driving.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(1):1036–1054, 2022.
  * Sun et al. [2020] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in Perception for Autonomous Driving: Waymo Open Dataset.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * Sun et al. [2022] Tao Sun, Mattia Segu, Janis Postels, Yuxuan Wang, Luc Van Gool, Bernt Schiele, Federico Tombari, and Fisher Yu.  Shift: a synthetic driving dataset for continuous multi-task domain adaptation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 21371–21382, 2022.
  * Taha et al. [2020] Ahmed Taha, Yi-Ting Chen, Teruhisa Misu, Abhinav Shrivastava, and Larry Davis.  Boosting Standard Classification Architectures Through a Ranking Regularizer.  In _IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , 2020.
  * Tan et al. [2023] Shuhan Tan, Tushar Nagarajan, and Kristen Grauman.  Egodistill: Egocentric head motion distillation for efficient video understanding.  _arXiv preprint arXiv:2301.02217_ , 2023.
  * Tong et al. [2022] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Tran et al. [2019] Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli.  Video Classification with Channel-separated Convolutional Networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 5552–5561, 2019.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Conference on Neural Information Processing Systems_ , 30, 2017.
  * Wang et al. [2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7794–7803, 2018.
  * Wu et al. [2019] Jianchao Wu, Limin Wang, Li Wang, Jie Guo, and Gangshan Wu.  Learning Actor Relation Graphs for Group Activity Recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * Wu et al. [2020] Tong Wu, Qingqiu Huang, Ziwei Liu, Yu Wang, and Dahua Lin.  Distribution-balanced loss for multi-label classification in long-tailed datasets.  In _European Conference on Computer Vision (ECCV)_ , 2020.
  * Xiao et al. [2022] Zihao Xiao, Allan Yuille, and Yi-Ting Chen.  Learning Road Scene-level Representations via Semantic Region Prediction.  In _Conference on Robot Learning (CoRL)_ , 2022.
  * Xu et al. [2022] Chejian Xu, Wenhao Ding, Weijie Lyu, Zuxin Liu, Shuai Wang, Yihan He, Hanjiang Hu, Ding Zhao, and Bo Li.  Safebench: A benchmarking platform for safety evaluation of autonomous vehicles.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.
  * Yang et al. [2020] Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou.  Temporal pyramid network for action recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 591–600, 2020.
  * Yeung et al. [2017] Serena Yeung, Olga Russakovsky, Ning Jin, Mykhaylo Andriluka, Greg Mori, and Li Fei-Fei.  Every moment counts: Dense detailed labeling of actions in complex videos.  _International Journal of Computer Vision_ , 2017.
  * Zhang et al. [2023] Jimuyang Zhang, Zanming Huang, and Eshed Ohn-Bar.  Coaching a teachable student.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7805–7815, 2023.
  * Zhou et al. [2022] Yi Zhou, Hui Zhang, Hana Lee, Shuyang Sun, Pingjun Li, Yangguang Zhu, ByungIn Yoo, Xiaojuan Qi, and Jae-Joon Han.  Slot-VPS: Object-centric Representation Learning for Video Panoptic Segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3093–3103, 2022.
  * Zhou et al. [2023] Zikang Zhou, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang.  Query-centric trajectory prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17863–17873, 2023.

## Appendix

In this appendix, we cover,

  1. A)

Construction of the TACO dataset

  2. B)

nuScenes annotation

  3. C)

More ablation study of Action-slot

  4. D)

Analysis in challenging scenarios

  5. E)

Analysis of the TACO Dataset

  6. F)

Limitations

  7. G)

Implementation details

##  Appendix A Construction of The TACO Dataset

We introduce the construction of the proposed Traffic Activity Recognition
(TACO) dataset. We leverage the CARLA simulator [23] to collect arbitrary
traffic scenarios for achieving a balanced activity class distribution in a
large scale.

#### Scenario Collection.

Certain atomic activities are rare and difficult to collect in the real world,
as shown in the class distribution comparison of TACO and OATS in Figure 2 of
the main paper We propose to leverage the CARLA simulator [24] to construct
the synthetic dataset. We choose CARLA simulator because it is widely accepted
and popular in the computer vision community, where it provides various sensor
suites and high-fidelity simulations to facilitate autonomous driving
development and test [56, 16, 1, 78, 14, 61, 81, 64], safety-critical scenario
generation [59, 33, 22], and domain adaption [68].

Figure 7: Illustration of the topology in atomic activity for three types of
T-intersections.  Figure 8: Ego-vehicle’s action distribution in TACO and
OATS.  Figure 9: The distribution of atomic activities in the nuScenes dataset
[9]. Note that we neglect the notation of topology (i.e., roadway Z and corner
C) in the x-axis due to limited space.

Scenarios are collected in the built-in maps (i.e., Town01, Town02, Town03,
Town04, Town05, Town06, Town07, and Town10HD) defined in CARLA 0.9.14. We use
Town10HD as the testing set. The rest are for the training set. We pinpoint
all intersections on each map and subsequently gather specific scenarios
related to these intersections. Note that, we also collect T-intersections,
following the topology definition discussed in OATS [3], as shown in Figure 7.

We collect scenarios with the following two approaches:

  1. (a)

Auto-pilot We use the built-in auto-pilot to control ego-vehicle and road
users, including vehicles and pedestrians. We do not set destinations for all
road users. We program an automatic scenario collection process when (1) ego-
vehicle approaches an intersection and (2) ego-vehicle is surrounded by at
least one road user.

The length of the recordings varies from 51 to 242 frames. We set the duration
for capturing various ego motions. We randomly set the numbers of road users
and spawn them on a map. To collect diverse atomic activities, we set all road
users (except for the ego-vehicle) to ignore any traffic rules, including
traffic lights and stop signs.

  2. (b)

Scenario Runner [2]. Scenario runner, a scenario collection tool developed by
CARLA [23], can explicitly generate a scripted scenario by defining a set of
routes for road users. Although auto-pilot can help collect diverse traffic
activities, it is difficult to generate some atomic activities frequently. For
example, Z3-Z2:K+, a group of bicyclists turns left from the opposite roadway
of ego-vehicle. Therefore, we leverage scenario runner to explicitly collect
the scenarios that are difficult for the auto-pilot method. A scenario starts
to collect when ego vehicle reaches a trigger point (i.e., location)
predefined in the script. A scenario ends when all scripted actions are
accomplished. To further enhance the diversity of scenarios, we randomly spawn
road users surrounding the ego vehicle. Note that road users are not
guaranteed to be involved in an atomic activity.

For both collection methods, we randomly set the weather and light conditions.
Note that we exclude night scenes because of the poor visibility.

#### Sensor Suites.

We deploy a wide field-of-view camera (120 degrees) to record events taking
place on the extreme left and right sides of the ego-vehicle. For example,
pedestrians crossing the street on the left (C3-C4:P and C4-C3:P), and a
vehicle turns right from the left roadway (Z4-Z1:C). We collect the
corresponding images and instance segmentation. In addition, we collect
instance segmentation from Bird’s eye view.

#### Annotation Criterion.

We follow the same annotation criterion defined in OATS [3]. The annotation of
atomic activities can be subjective, e.g., whether a turning right car that
stops for a crossing pedestrian should be annotated. Thus, the whole
annotation work is done by one person to ensure annotation consistency. In
addition, we list a set of annotation criteria to enhance the quality of data.
If any of the criteria is valid in a video, the annotator is asked to discard
the whole scenario.

  1. 1.

Annotator cannot determine the starting roadway of a road user.

  2. 2.

Annotator cannot determine a road user’s destination.

  3. 3.

Annotator cannot determine any road user’s action in a video.

  4. 4.

One of the road users has completed an atomic activity at the beginning of the
video, e.g., driving away from Z4 and almost arriving at Z1.

We additionally filter out scenarios with zero atomic activity, meaning there
are no other road users engaged in any actions. This helps address the
positive-negative imbalance issue in multi-label recognition [65, 19, 76]. We
also ensure that each atomic activity remains observable for a minimum
specified duration. This is due to the fact that the video will be subsampled
into a fixed-length $K$ short clip before being inputted into a model. We set
$K$ as 16 because the majority of models we benchmarked utilize 16-frame
clips. Consequently, for an N-frame video, we ensure that each atomic activity
is present for a minimum of $N/K$ frames. Otherwise, annotators will discard
the entire scenario.

Object-guided |  |  |  |
---|---|---|---|---
Action-slot (ours) |  |  |  |
Object-guided |  |  |  |
Action-slot (ours) |  |  |  |
Figure 10:  Visualization of attention maps of Object-guided and Action-slot
from slots that predict false positives on additional scenarios. The scenario
presents zero atomic activity but many static road users. False positive
predicted by the object-guided model: C1-C4:P, C4-C1:P (upper) and C1-C2:P,
C2-C1:P, C1-C2:P+ (bottom). The attention scores within the Action-slot do not
surpass the threshold in any region.

#### Annotation for Ego-vehicle’s Action.

We provide annotations of the ego-vehicle’s actions for every scenario. We
annotate Z1-Z1:E if the annotator is unable to determine the ego vehicle’s
action. For instance, when the ego-vehicle remains stationary at a traffic
light throughout the entire scenario or moves slowly towards an intersection
without exhibiting a specific action. This approach differs from OATS which
excludes scenarios lacking a specific ego-vehicle action.

#### Detailed Dataset Statistics.

We have annotated a total of 16,521 instances of atomic activity. The maximum
number of road users observed in a single frame is 37. In a video, it reaches
up to 63. On average, each video encompasses 2.43 labeled traffic pattern
descriptions. The duration of the captured videos varies, ranging from 51
frames to 242 frames. The average length of the videos is 109.341 frames, with
a frame rate of 20Hz.

##  Appendix B nuScenes Annotations

We annotate nuScenes [9], one of the most popular real-world traffic scene
datasets, for additional experiments. The dataset was collected in Boston and
Singapore. We follow the same annotation criterion in TACO for nuScenes.
Specifically, we scan through the train_val set of nuScenes and annotate any
scenarios in the 4-way and T-intersections. However, the data in nuScenes is
collected with a very low frame per second (FPS), we thus select 16
consecutive frames as a short clip. Note that we discard the night scenes
because of poor visibility. To this end, we obtain 426 clips and 933 atomic
activity labels for the new nuScenes dataset. The atomic activity class
distribution is presented in Figure 9. We randomly divide the set with 340
clips for the training set and 86 clips for the testing set. We downsample the
image size to 256 $\times$ 768, which is the same as TACO. For transfer
learning, we downsample the image size to the same size with pre-trained
datasets, i.e., 224 $\times$ 224 for OATS pre-trained and 256 $\times$ 768 for
TACO pre-trained. We neglect the atomic activities involved with grouped two-
wheelers (K+) when calculating mAP because there are only 2 labels in the
whole dataset.

##  Appendix C More Ablation Study of Action-slot

Table 7: Comparison of using cross-attention and slot-attention in Action-slot on OATS. S1, S2, and S3 denote the three test splits in OATS. | S1 | S2 | S3 | mAP
---|---|---|---|---
Cross attention | 29.0 | 32.8 | 35.4 | 32.4
Slot attention | 48.1 | 47.7 | 48.8 | 48.2
|  |  |
---|---|---|---
Black vehicles stop at Z3 in this clip. Atomic activities: C2-C3:P+ and
C3-C2:P+.
|  |  |
Many parked cars on both sides of the street. Atomic activities: Z3-Z1:C and
C1-C2:P.
|  |  |
Parked car and pedestrians on the left and car stopping temperately at the Z3.
Atomic activity: Z3-Z2:C.
Figure 11:  Action-slot’s attention visualization in scenarios where atomic activities and static road users are present. We show scenarios from the TACO, OATS [3], and nuScenes [9] datasets in the first, second, and third row, respectively. Table 8: Results of Boston and Singapore splits in nuScenes. | nuScenes
---|---
| Boston
---
| Singapore
---
X3D [28] | 33.7 | 11.6
ARG [75] | 17.2 | 6.6
Action-slot | 34.7 | 18.3

#### Object-guided v.s. Action-slot.

We further provide detailed insights into the failure cases of object
guidance. We hypothesize that the object guidance may mislead the model
because not all objects are involved in an activity.

We create scenarios where many pedestrians are static on sidewalks and not
involved in any activities, as shown in Figure 10, to better demonstrate the
misleading signal caused by the object guidance. We visualize the attention
from any slot that predicts false positive with red masks. The object-guided
method pays attention to the static road users and produces false positive
predictions. Moreover, the attention to the false positives is accumulated
temporally. We hypothesize this is because the method is not robust to the
spatial-temporal features extracted from the backbone [28]. On the other hand,
our Action-slot demonstrates the robustness in the scenarios with many static
road users.

#### Cross-Attention v.s. Slot Attention.

We study the difference between cross-attention and slot-attention for multi-
label atomic activity recognition on OATS. Cross-attention can be seen as a
query-based method that recently has achieved remarkable success in many tasks
[83, 36, 32]. The key difference between cross-attention and slot attention is
the dimension to which the softmax operation is applied. The classic cross-
attention [73] applies softmax on the tokens, i.e., tokens compete over
queries. On the other hand, the softmax in slot attention is applied to the
slot dimension, which makes slots compete with each other over the tokens. In
Table 7, we conduct experiments by replacing the slot attention in Action-slot
with the classic cross-attention. The results show inferior performance of
cross-attention compared to Action-slot using slot attention. Moreover, we
observe that the background guidance loss $L_{bg}$ in the query-based method
can not converge well. The results indicate that the cross-attention method
may need a stronger supervision signal.

##  Appendix D Analysis in Challenging Scenarios

|  |  |  |  |
---|---|---|---|---|---
| Scenario presenting traffic cones in the intersection. Atomic activities:
Z4-Z3:C+ and Z1-Z3:K.
|  |  |  |  |
| Scenario presenting the construction cover the entire corner C4. Atomic
activities: C4-C1:P.
Figure 12:  Action-slot’s attention visualization in nuScenes [9]. In the two
scenarios, road structures are partially occluded by the traffic cones and
construction. Colored masks represented the action slots’ attention on
distinct activities.

#### Atomic Activities and Static Road Users in Scenarios.

We show qualitative results in scenarios where both atomic activities and
multiple static road users are present. In Figure 11, the attention learned by
Action-slot focuses on the regions where activities occur instead of being
distracted by static road users, e.g., vehicles waiting at traffic lights
(first and third row), many parked cars (second row), and pedestrians walking
on the sidewalk (third row). The results demonstrate the proposed action-
centric representations are robust in crowded traffic scenes and can decompose
the atomic activities and non-relevant regions from videos.

#### Boston v.s. Singapore.

To verify the generalization of Action-slot, we evaluate models on the Boston
split and Singapore split in nuScenes [9] in Table 8. Note that the scenarios
collected in TACO and OATS [3] are right-hand-traffic. The left-hand-traffic
scenarios collected in Singapore thus pose a challenging domain discrepancy
for atomic activities. We use the model pretrained on TACO for better
performance. Experimental results show that the video-level and object-aware
representations both perform inferior in the Singapore split. On the other
hand, Action-slot shows the generalization in the scenes with significant
domain discrepancy.

Table 9: Comparisons of pretrained representations from OATS and TACO. We perform transfer learning on the nuScenes dataset.  | nuScenes
---|---
| Kinetics
---
| \+ OATS
---
| +TACO
---
X3D [28] | 19.8 | 18.9 | 27.8
ARG [75] | 12.2 | 12.7 | 17.0
Action-slot | 23.6 | 23.6 | 32.3

#### Occluded Road Topology.

We present the qualitative results to demonstrate that Action-slot can handle
challenging scenarios where the road topology is significantly occluded in
Figure. 12. We observe Action-slot can accurately predict and localize the
activities despite the intersection (upper) and corner (lower) occupied by the
traffic cones and construction, respectively. This demonstrates the strong
generalization of Action-slot on recognizing road topology and the reasoning
ability of road topology.

|  |  |  |
---|---|---|---|---
Figure 13:  Action-slot’s attention visualization in the TACO scenario where a
crossing pedestrian first performs action C4-C3:P then returns with C3-C4:P.

#### Road User with Multiple Actions.

We find that Action-slot can handle the scenarios where a road user performs
multiple actions consecutively. In Figure 13, Action-slot accurately predicts
the two atomic activities involved with the crossing pedestrian and spatial-
temporally localizes the transition of two actions. The result again
demonstrates the effectiveness of the proposed action-centric representations
for the action-aware task.

##  Appendix E Analysis of The TACO Dataset

#### OATS Pretrain v.s. TACO Pretrain.

We compare the pertained representations learned from OATS and TACO by fine-
tuning them on nuScenes in Table 9. The results show that models pretrained on
TACO outperform the ones pretrained on OATS, verifying the real-world value
and transferability of the proposed TACO dataset.

|  |  |  |
---|---|---|---|---
Figure 14:  Action-slot’s attention visualization in the TACO scenario where an atomic activity is occluded. The yellow bus with red masks partially occludes the white car on the right side (Z2). Action-slot successfully predicts Z3-Z1:C and Z2-Z1:C but misses the occluded white car Z2-Z3:C (black arrow in illustration). Table 10: Results of all 64 classes of atomic activity. Each grouped row is divided by the type of road users involved. | Z1-Z2:C | Z1-Z3:C | Z1-Z4:C | Z2-Z1:C | Z2-Z3:C | Z2-Z4:C | Z3-Z1:C | Z3-Z2:C | Z3-Z4:C | Z4-Z1:C | Z4-Z2:C | Z4-Z3:C
---|---|---|---|---|---|---|---|---|---|---|---|---
X3D [28] | 21.1 | 27.1 | 31.5 | 28.2 | 33.4 | 21.8 | 28.4 | 34.5 | 18.9 | 22.1 | 28.8 | 32.1
ARG [75] | 36.2 | 17.9 | 23.6 | 25.9 | 26.8 | 14.5 | 15.3 | 15.7 | 16.2 | 41.4 | 30.0 | 19.1
Action-slot | 48.5 | 47.9 | 53.1 | 57.7 | 54.1 | 45.9 | 41.5 | 43.5 | 47.0 | 48.1 | 44.8 | 44.7
| Z1-Z2:C+ | Z1-Z3:C+ | Z1-Z4:C+ | Z2-Z1:C+ | Z2-Z3:C+ | Z2-Z4:C+ | Z3-Z1:C+ | Z3-Z2:C+ | Z3-Z4:C+ | Z4-Z1:C+ | Z4-Z2:C+ | Z4-Z3:C+
X3D [28] | 76.6 | 42.7 | 3.1 | 61.7 | 58.7 | 48.6 | 41.1 | 60.9 | 64.5 | 76.4 | 73.4 | 70.8
ARG [75] | 32.8 | 11.0 | 0.1 | 38.3 | 3.8 | 17.8 | 9.8 | 13.2 | 9.4 | 4.1 | 15.6 | 1.2
Action-slot | 86.9 | 63.7 | 28.7 | 74.9 | 59.5 | 75.2 | 64.6 | 79.0 | 79.1 | 82.1 | 78.0 | 69.7
| Z1-Z2:K | Z1-Z3:K | Z1-Z4:K | Z2-Z1:K | Z2-Z3:K | Z2-Z4:K | Z3-Z1:K | Z3-Z2:K | Z3-Z4:K | Z4-Z1:K | Z4-Z2:K | Z4-Z3:K
X3D [28] | 17.3 | 27.9 | 24.9 | 36.0 | 10.6 | 14.2 | 30.5 | 21.5 | 9.1 | 13.7 | 21.7 | 13.0
ARG [75] | 55.0 | 17.3 | 0.7 | 50.3 | 19.2 | 20.7 | 22.7 | 57.9 | 10.3 | 46.9 | 25.6 | 23.2
Action-slot | 39.9 | 54.2 | 45.3 | 51.1 | 30.3 | 41.9 | 46.1 | 36.9 | 36.9 | 35.5 | 38.8 | 35.4
| Z1-Z2:K+ | Z1-Z3:K+ | Z1-Z4:K+ | Z2-Z1:K+ | Z2-Z3:K+ | Z2-Z4:K+ | Z3-Z1:K+ | Z3-Z2:K+ | Z3-Z4:K+ | Z4-Z1:K+ | Z4-Z2:K+ | Z4-Z3:K+
X3D [28] | 76.7 | 31.0 | 1.0 | 83.5 | 26.3 | 53.2 | 54.1 | 75.0 | 36.6 | 40.4 | 79.8 | 55.2
ARG [75] | 22.1 | 22.1 | 20.6 | 5.0 | 11.3 | 3.9 | 24.2 | 7.0 | 2.5 | 5.3 | 20.4 | 9.1
Action-slot | 93.7 | 56.7 | 7.7 | 67.3 | 51.9 | 65.7 | 74.3 | 74.0 | 68.3 | 60.6 | 76.8 | 48.4
| C1-C2:P | C1-C4:P | C2-C1:P | C2-C3:P | C3-C2:P | C3-C4:P | C4-C1:P | C4-C3:P |  |  |  |
X3D [28] | 34.4 | 43.5 | 38.9 | 35.0 | 25.1 | 29.6 | 45.5 | 27.6 |  |  |  |
ARG [75] | 31.1 | 38.7 | 23.7 | 12.8 | 10.6 | 22.5 | 39.3 | 15.3 |  |  |  |
Action-slot | 52.4 | 60.6 | 55.9 | 42.5 | 44.3 | 38.2 | 65.2 | 34.2 |  |  |  |
| C1-C2:P+ | C1-C4:P+ | C2-C1:P+ | C2-C3:P+ | C3-C2:P+ | C3-C4:P+ | C4-C1:P+ | C4-C3:P+ |  |  |  |
X3D [28] | 39.2 | 53.3 | 47.3 | 24.2 | 23.2 | 32.4 | 62.6 | 29.1 |  |  |  |
ARG [75] | 19.6 | 24.8 | 18.0 | 6.1 | 5.7 | 9.4 | 24.6 | 8.4 |  |  |  |
Action-slot | 61.0 | 74.5 | 65.0 | 34.8 | 33.0 | 37.4 | 80.8 | 35.3 |  |  |  |

#### Activity Classes Analysis.

We report the performance of models for all 64 classes of atomic activities,
which can not be achieved in OATS [3] and nuScenes [9]. We find two
interesting observations. First, most results of activities involved with
grouped road users (e.g., C1-C2:C+) are better than the ones involved with a
single road user (e.g., C1-C2:C), in which one possible reason is that the
larger regions of interest are easier to predict. Second, the smaller or more
distant activities are more challenging, which can be attributed to the
insufficient representations learned from the backbone.

##  Appendix F Limitation

#### Action-slot.

We find Action-slot performs less effectively in the occluded scenarios where
the activities are visually overlapped. In Figure 14, a bus with Z2-Z1 action
occludes a white car on the right side with action Z2-Z3. Action-slot
successfully predicts the bus’s action Z2-Z1 but fails to predict Z2-Z3 and
can not localize it via attention. We hypothesize that this is because the
occlusion may confuse the competition mechanism in slot attention, i.e., two
slots compete over the overlapped regions. We hope our findings can inspire
the community to discover more advanced action-centric representations that
can handle occlusion issues.

#### The TACO Dataset.

We observe the 64 classes of atomic activities in TACO can not fully cover the
diverse events in traffic scenes. For example, vehicles can only move between
roadways and pedestrians can only move between two near corners. However, two-
wheelers can move between corners, e.g., C1-C2:K+, and pedestrians can also
move diagonally, e.g., C1-C3:P. These atomic activities are important to many
applications, such as safety-critical scenario generation [78, 33, 59].
However, the existing auto-pilot mechanism in the CARLA simulator does not
support the collection of such atomic activities. We aspire for our research
to inspire collaborative efforts within the community to improve existing
atomic activity datasets. This involves gathering larger, more diverse
datasets from real-world scenarios and advancing the sophistication of
simulator-based auto-pilots.

##  Appendix G Implementation Details

#### Data Preprocessing.

We downsample a video into a fixed-length short clip as models’ input, which
is common practice in video recognition [65, 3, 70, 54]. Specifically, we
randomly sample subsequent with uniform intervals between frames for the
training set and fix the subsequent for the testing set. Note that we neglect
this process for nuScenes because we annotate each sample as a fixed 16-frame
clip.

#### Ego-vehicle’s action.

To enhance the awareness of ego motion, we include a module for all models to
predict ego-vehicle’s action via global features. The global features
$F_{ego}\in\mathbb{R}^{256}$ are generated by applying a Conv3D with kernel
size 1. Since each video must have a label for the corresponding ego-vehicle’s
action, the models output a multi-class prediction with a fully connected
layer and softmax operation. Note that we neglect the ego vehicle’s action
prediction in OATS [3] since the absence of the annotation in the released
dataset. In this paper, we do not report the accuracy of predicting ego-
vehicle’s pattern because the performance of all models saturates with nearly
100%.

#### Architecture of Action-slot.

In this work, we eliminate the GRU from the slot updating process [48, 42, 25,
5, 82] for Action-Slot. This is because of its negligible impact on enhancing
performance in our experiments.

#### Training for transformer-based methods.

We freeze the pretrained transformer blocks except for the first three and
last three blocks during training. We downsample the input image to $224\times
224$ for both MViT [27] and VideoMAE [71] on all datasets. We attempt to adapt
the pre-trained positional embedding of MViT as suggested in ViT [24] by
interpolating spatial positional embedding. For example, we adapt token
numbers from $7\times 7$ to $8\times 24$ in our TACO dataset. However, we find
the performance degraded with interpolation. We thus simply use image size
$224\times 224$ to match the original positional embedding size.

#### Training for object-aware methods.

We follow OATS [3] to set the number of object proposals to 20. Specifically,
we select the 20 largest bounding boxes in each frame. To match the ground-
truth atomic activity labels with the length of proposals, we pad the ground
truth with the negative class and use a Hungarian matcher to associate them
during training. It is worth noting that because object-aware models output
multi-class results for each proposal, we rearrange the outputs to a set for
calculating the metrics.

#### Training for slot-based methods.

In order to adapt slot-based baselines to atomic activity recognition, we use
the last state of slots as the input to the classifier for the recurrent
fashion, i.e., SAVi [42, 25] and MO [5]. As for Slot-VPS [82], we sum up the
slots across temporal dimensions. For all slot-based baselines, including our
Action-Slot, we set both the dimensions of slots and image features to 256.

#### Backbone modification for Action-Slot.

We use the features of the last convolution block as the input to action slots
for all backbone encoders except for SlowFast. SlowFast processes two paths:
path _Fast_ takes an original input sequence (16 frames) as input and the path
_Slow_ takes the sub-sequence with $1/4$ length (4 frames). We apply a pooling
operation to the output of path _Fast_ for aligning the length of path _Slow_
and combine them with channel-wise concatenation. We freeze the entire
backbone except for the last ConvBlock.

#### Hyperparameters.

All the models including our Action-Slot are trained for 50, 100, and 100
epochs on OATS [3], TACO, and nuscenes [9], respectively. We use AdamW
optimizer [49] with a batch size of 8. The learning rate varies from 1e-4 to
5e-5 and weight decay varies from 0.1 to 0.0001. We apply a 50% dropout to all
models’ last features layer, e.g., X3D’s last ConvBlock. We conduct all
experiments with a single NVIDIA 3090 GPU with 24GB.