# Semantically Guided Representation Learning For Action Anticipation

Anxhelo Diko
Computer Science Department
Sapienza University of Rome
<EMAIL_ADDRESS>
&Danilo Avola
Computer Science Department
Sapienza University of Rome
<EMAIL_ADDRESS>
&Bardh Prenkaj11footnotemark: 1
Chair of Responsible Data Science
Technical University of Munich
<EMAIL_ADDRESS>
&Federico Fontana
Computer Science Department
Sapienza University of Rome
<EMAIL_ADDRESS>
&Luigi Cinque
Computer Science Department
Sapienza University of Rome
<EMAIL_ADDRESS>Equal second author contribution.

###### Abstract

Action anticipation is the task of forecasting future activity from a
partially observed sequence of events. However, this task is exposed to
intrinsic future uncertainty and the difficulty of reasoning upon
interconnected actions. Unlike previous works that focus on extrapolating
better visual and temporal information, we concentrate on learning action
representations that are aware of their semantic interconnectivity based on
prototypical action patterns and contextual co-occurrences. To this end, we
propose the novel Semantically Guided Representation Learning (S-GEAR)
framework. S-GEAR learns visual action prototypes and leverages language
models to structure their relationship, inducing semanticity. To gather
insights on S-GEAR’s effectiveness, we test it on four action anticipation
benchmarks, obtaining improved results compared to previous works: +3.5, +2.7,
and +3.5 absolute points on Top-1 Accuracy on Epic-Kitchen 55, EGTEA Gaze+ and
50 Salads, respectively, and +0.8 on Top-5 Recall on Epic-Kitchens 100. We
further observe that S-GEAR effectively transfers the geometric associations
between actions from language to visual prototypes. Finally, S-GEAR opens new
research frontiers in anticipation tasks by demonstrating the intricate impact
of action semantic interconnectivity. https://github.com/ADiko1997/S-GEAR

_K_ eywords Action Anticipation $\cdot$ Semantic Interconnection $\cdot$
Prototype Learning $\cdot$ Geometric Associations

##  1 Introduction

Anticipating future actions is a key attribute of human intelligence for
navigating the world. This remarkable skill translates directly to advanced
computer vision applications such as self-driving cars [1, 2] or wearable
assistants [3, 4], enabling safer navigation and better user experience [5].

Figure 1: We propose learning action prototypes that encode typical action
representations and meaningful semantic interconnections. The model leverages
these prototypes to enhance the network encodings of observed actions and to
forecast upcoming ones.

Recent developments in deep learning techniques have boosted the research on
video understanding, reaching remarkable milestones on tasks like action
recognition [6, 7, 8, 9, 10, 11]. Models related to action recognition can
extrapolate essential spatiotemporal information from videos of isolated
actions and correctly classify them. However, real-world applications operate
in dynamic environments where actions are interconnected. For instance,
imagine a self-driving car observing pedestrians. Predicting their intent to
cross the street requires analyzing how observed dynamics relate to likely
future events. This temporal misalignment between observation and future
target introduces a challenge for recognition models to capture actionable
insights, proving them insufficient and shifting the attention towards action
anticipation [1, 3, 12, 13, 5, 4]. This emerging research area focuses on
enabling vision systems to predict future activity by observing ongoing
events. A temporal gap between the observed events and the target future
further amplifies its difficulty.

In trying to deal with the implications of action anticipation, previous
methods extended recognition models with sequence units like LSTMs [1, 5, 4,
14] and causal transformers [12, 4, 3]. The success of these approaches relies
on the network’s ability to effectively represent significant visual
information from videos and the ability of the sequence units to propagate and
preserve information in time. However, these methods have limitations. They
cannot explicitly model the semantic connectivity between actions beyond the
immediate video context, which is critical when dealing with co-occurring
action sequences. According to cognitive sciences, semantic interconnectivity
is fundamental for anticipating the future [15]. It helps structure our
knowledge by associating actions with objects, intentions, and likely
outcomes. This enables us to draw on past experiences to form reliable
predictions even in unseen situations. Inspired by such observations, we raise
the question: Is it possible to encode meaningful semanticity between action
representations in a vision model?

In pursuit of answering our question, we propose the Semantically Guided
REpresentation LeARning (S-GEAR) framework (see Fig. 1). S-GEAR tackles action
anticipation with a novel representation learning approach oriented by two
fundamentals of actions semantic connectivity: (1) understanding the typical
patterns of individual actions, and (2) modeling relationships between actions
based on contextual co-occurrences [16, 17, 18]. For (1), S-GEAR learns a set
of visual action prototypes. Each prototype encodes specific action patterns,
capturing typical movements or gestures that define and distinguish action
categories, reducing reliance on the specific appearance details of individual
videos. Conversely, for (2), building semantic relationships between actions
solely from videos is challenging. First, it requires processing long action
sequences to include enough context and defining co-occurrence relationships.
Second, actions are usually not represented equally in videos [19, 20],
hardening the modeling of under-represented action relationships. S-GEAR
circumnavigates these issues by exploiting language models known to extract
inter-concept semantic relationships [21, 22] – i.e., language models
effectively tackle (2). Specifically, S-GEAR creates language prototypes based
on action labels and transfers their inherent semantic connectivity to visual
prototypes without aligning them directly. To achieve this, S-GEAR uses a new
loss function that enables visual prototypes to maintain visual cues, such as
object and movement patterns, while encoding semanticity by mimicking the
geometric associations between actions from language.

S-GEAR uses an encoder-decoder transformer architecture to learn prototypes
and encode semantic relationships between actions. The encoder consists of a
standard Vision Transformer (ViT) [23, 24] for visual context, while the
decoder is a Causal Transformer (CT) [25, 24] which models temporal causality.
These structures are interconnected through two novel computational blocks,
namely Temporal Context Aggregator (TCA) and Prototype Attention (PA) for,
respectively, causality enhancement and semanticity promotion. Lastly, S-GEAR
appends a classification head that produces future class probabilities based
on the decoder’s output and geometric association with the visual prototypes.

To assess S-GEAR’s performance, we conduct extensive experiments on two
egocentric video datasets, Epic-Kitchens [19, 20] (both Epic-Kitchens 55 and
Epic-Kitchens 100 versions) and EGTEA Gaze+ [26]. Moreover, we evaluate S-GEAR
on an exocentric dataset, namely 50 Salads [27], to demonstrate its
versatility in tackling long-term dense anticipation. We show that S-GEAR
improves over the current state-of-the-art in most scenarios. We also conduct
ablation studies highlighting the usefulness of the semantic connectivity
between the actions that S-GEAR incorporates.

This paper’s contributions are fourfold. (1) We present S-GEAR, a novel
prototype learning framework for action anticipation leveraging action
interconnectivity. (2) We introduce a novel approach to map semanticity from
language to vision without direct alignment between modalities. (3) We conduct
extensive experiments on two egocentric datasets and an exocentric one to
highlight S-GEAR’s versatility in different action anticipation scenarios
(i.e., egocentric vs. exocentric and short-term vs. long-term). (4) We
showcase the benefits of S-GEAR w.r.t. its counterparts that do not rely on
semantic relationships.

##  2 Related Work

Action anticipation predicts future actions before they occur in video clips
and is well explored both in third-person (exocentric) videos [28, 13, 29, 30,
31], and first-person (egocentric) videos [1, 32, 5, 26, 33, 4, 34, 12, 35,
36, 37], due to its applicability on autonomous agents and wearable assistants
[38, 2, 12]. Funari et al. [1] introduce RU-LSTM, a model with two LSTMs and a
modality attention component. Osman et al. [39] integrate RU-LSTM into
SlowFast. Qi et al. [5] enhance LSTMs with Self-Regulated Learning (SRL).
Dessalene et al. [33] use hand-object contact representations for action
anticipation. Xu et al. [4] employ curriculum learning. Roy et al. [34]
predict final goals for near-future anticipation. Liu et al. [36] store long-
term action prototypes for richer short-term representations. Girdhar et al.
[12] propose AVT, combining ViT and causal transformer, paving the way for [3,
35, 7]. Unlike previous works, S-GEAR considers the semantic relationship
between action representations [15] by using vision and language prototypes to
guide the model’s training process semantically.

Vision-Language alignment relies on effectively aligning concepts between
vision and language in a unified representation space. Typically achieved
through contrastive training of modality encoders [40, 41, 42], these methods
use vision-language pairs for encouraging proximity between corresponding
visual and text embeddings. Zhai et al. [40] utilize contrastive learning to
align text encoder representations with a frozen pre-trained vision model.
Radford et al. [42] introduce CLIP, training separate encoders for text and
images and aligning representations through contrastive loss. Ma et al. [43]
extend CLIP to videos, employing multi-grained contrastive learning.
Advancements include cross-modal fusion architectures using a cross-modality
encoder for text and visual inputs [44, 45]. Unlike previous works, S-GEAR
only translates the geometric association between action prototypes from
language to vision without shifting spaces.

Prototype Learning involves creating characteristic “prototypes” of labeled
data samples. Initially dominant in few-shot learning for novel class
prediction [46, 47], this strategy now successfully encodes spatial and
temporal patterns in domains such as video semantic segmentation [48] and
action recognition [49].

##  3 Method

We propose S-GEAR for action anticipation. S-GEAR discerns essential
spatiotemporal signals and understands the semantic relationships between
actions. It contains a neural network architecture tailored for understanding
spatiotemporal video sequences and a learning policy that guides the network
semantically to map out the interconnections between actions.

Task Formulation. Action anticipation involves predicting an action category
for an event starting at time $\tau_{s}$, observing a video segment $V_{o}$
within the interval $[\tau_{s}-(\tau_{o}+\tau_{a});\tau_{s}-\tau_{a}]$ [20].
Here, $\tau_{o}$ and $\tau_{a}$ denote the observation and anticipation
periods set specifically to the dataset.

Figure 2:  S-GEAR processes frame sequence patches and creates input token
sequences $S_{t}$. ViT $\phi$ encodes $S_{t}$ into intermediate features
$I_{t}$. PA $\gamma$ and TCA $\varphi$ process $I_{t}$, merging outputs into
semantically enhanced causal features $\hat{I}_{t}$. Class tokens pass through
the CT decoder $\Omega$, predicting future features $z_{t}$. The features
$z_{t}$ and the proposed prototypes are trained for action anticipation
($\mathcal{L}_{Cls}$) and semantic relation encodings ($\mathcal{L}_{Sem}$).
The network is also regularized for accurate future representations
($\mathcal{L}_{Feat}$) and correct past action classification
($\mathcal{L}_{Past}$). Finally, a distance loss ($\mathcal{L}_{reg}$) is
applied to $z_{t}$.

###  3.1 Proposed Architecture

S-GEAR processes a sequence of video frames and produces a set of features
that can accurately describe the subsequent action. To achieve this, as shown
in Fig. 2, S-GEAR employs an architecture composed of (1) a visual encoder for
extracting feature vectors from the input frames; (2) the Temporal Context
Aggregator (TCA) module designed to incorporate detailed temporal context from
past to current observations; (3) the Prototype Attention (PA) block, which
combines visual features with learned prototypes and (4) the Causal
Transformer (CT) decoder responsible for predicting future representations.

Visual Encoder. Upon receiving a video segment $V_{o}=\\{f_{0},...,f_{T-1}\\}$
of length $T$, S-GEAR relies on ViT [23] as the visual encoder $\phi$ to
obtain spatial features from each frame. ViT splits each frame into $P$ non-
overlapping patches of equal size, which are then flattened and transformed
into a series of feature tokens $S_{t}\in\mathbb{R}^{P\times d}$ corresponding
to frame $f_{t}\in V_{o}$. Here, $d$ represents the token dimensionality.
Then, to preserve the spatial context, learnable positional encodings are
added to $S_{t}$. Additionally, the so-called “class token” $\text{CLS}_{t}$,
which captures the global context of frame $f_{t}$, is prepended to $S_{t}$.
The transformer blocks then act on $S_{t}$, generating visual features
$I_{t}=\phi(S_{t})$ with the same dimension as $S_{t}$.

Temporal Context Aggregator (TCA) and Prototype Attention (PA). In this stage,
$I_{t}$ passes through two specialized units to enhance temporal causality and
semantic interconnections between actions. Inspired by the left-to-right
causal transformer [24], we craft TCA $\varphi$, the first unit, to
effectively transfer comprehensive context from the past to the current frame
representation. In TCA, unlike standard causal blocks that mainly rely on the
global representations $I^{0}_{t}$, we consider all feature patches in each
frame. Thus, given all global and local representations $I$ of the frames, we
obtain causal intermediate features
$\overline{I}\in\mathbb{R}^{T\times(P+1)\times d}$ where each
$\overline{I}_{t}\in\overline{I}$ is enhanced with detailed contextual
information from past frames. Contrarily, the second unit PA, denoted as
$\gamma$, operates parallel to the TCA on $I_{t}^{0}$ and the visual
prototypes. Specifically, PA aggregates information from selected visual
prototypes upon feature similarity to $I_{t}^{0}$, promoting semantic relation
encoding between actions as inferred from the different prototypes. We rely on
the attention mechanism using $I^{0}_{t}$ as queries and the visual prototypes
as keys and values to produce semantically enhanced feature sets
$\tilde{I}\in\mathbb{R}^{T\times d}$. We then combine $\tilde{I}$ and
$\overline{I}^{0}$ as a weighted sum
$\hat{I}=\lambda\overline{I}^{0}+(1-\lambda)\tilde{I}$ ($\lambda$ is
learnable). We point the reader to Appendix A.1-2 for details on TCA and PA.

Temporal Decoder. We rely on an autoregressive Causal Transformer (CT) decoder
$\Omega$, as presented in [4, 12, 3] to analyze $\hat{I}$ from $t=0$ to
$t=T-1$ and generate a set of features that describes the likely future.
Similar to the visual encoder, we add learnable positional encodings to
$\hat{I}$ to preserve the temporal context. Afterward, we feed the embedded
features with positional encodings to the decoder blocks, built upon the
masked multi-head self-attention [12]. Thus, $\Omega$ generates a new sequence
$\zeta=\Omega(\hat{I})$ s.t. $\forall t$, $z_{t}\in\zeta$ represents the
future features of $\hat{I}_{t}$ after observing all the past ones including
itself. For $t=T-1$, $z_{t}$ represents the future action happening $\tau_{a}$
seconds after the observed sequences.

###  3.2 Semantic Guiding Policy

We exploit vision/language prototypes and a common communication space between
them to facilitate a semantic-based guiding policy for action anticipation.

Prototypes. We aim to translate semantic relationships from language-based
action concepts to the visual domain. Thus, we define two sets of prototypes.
The first, defined as the language prototypes
$\rho_{\ell}\in\mathbb{R}^{K\times d}$ (where $K$ is the number of action
classes), is extracted by encoding action labels composed of verb and noun
combination using the “Sentence Transformer” proposed in [22]. These
prototypes serve as the reference space for learning actions “semantic
connectivity” [21]. The second, defined as the visual prototypes
$\rho_{\upsilon}\in\mathbb{R}^{K\times d}$, ensures that S-GEAR remains in the
visual domain and effectively preserves characteristic visual patterns. Such
prototypes are learnable and initialized from typical action samples encoded
from the proposed architecture trained for action recognition. We exploit
$\rho_{\upsilon}$ to encode visual action representations and inherit the
semanticity from $\rho_{\ell}$. Refer to Appendix A.3 for initialization
details.

Common Communications Space. To translate action relationships from language
to vision without shifting domains, we define a common space where vision and
language representations co-exist and are compared via their relative
associations w.r.t. the prototypes. In more detail, given an action visual
encoding $z_{t}\in\zeta$, we compute its relative representation by comparing
it against all elements in $\rho_{\upsilon}$ using a similarity function:
i.e., $\mathbf{r}^{z_{t}}=\\{r_{1}^{z_{t}},\dots,r_{K}^{z_{t}}\\}$ s.t.
$r_{k}^{z_{t}}=cos(z_{t},\rho_{\upsilon}[k])$ for each action class
$k\in\\{1,\dots,K\\}$. Similarly, we compute the relative representation of a
language encoding $\text{enc}(y)_{t}$ – i.e., the language encoding of the
action label at time $t$ – against the prototypes in $\rho_{\ell}$ as
$\mathbf{r}^{\text{enc}(y)_{t}}=\\{r_{1}^{\text{enc}(y)_{t}},\dots,r_{K}^{\text{enc}(y)_{t}}\\}$
s.t.
$\mathbf{r}_{k}^{\text{enc}(y)_{t}}=cos(\text{enc}(y)_{t},\rho_{\ell}[k])$ for
all action classes $k\in\\{1,\dots,K\\}$. Now, we ensure that each $k$-th
entry in $\mathbf{r}^{z_{t}}$ and $\mathbf{r}_{k}^{\text{enc}(y)_{t}}$
represents the geometric association with the $k$-th class prototype in the
language/vision domain. Hence, we can directly compare these two
representations based on their relative position in their original vector
spaces.

###  3.3 Training

To train the model, for each labeled action segment, we sample a clip
preceding it and ending exactly $\tau_{a}$ seconds before the start of the
action. We pass the clip through S-GEAR to obtain $z_{t}$ and then optimize it
to learn semantically and visually meaningful prototypes for action
anticipation.

Prototype Learning. Learning prototypes aim to establish a visual latent space
where predefined semantic connections describe actions by “aligning” the
latent space topology defined by $\rho_{\upsilon}$ with $\rho_{\ell}$. To do
so, we calculate the relative positions111$z_{t}$ is detached from the
gradient when calculating $\mathbb{\textbf{r}}^{z_{t}}$ to avoid collapsing
issues. $\mathbb{\textbf{r}}^{z_{t}}$ and $\mathbf{r}^{\text{enc}(y)_{t}}$,
which we use to define the semantic loss in Eq. 1.

$\mathcal{L}_{Sem}=\big{|}\mathbb{\textbf{r}}^{z_{t}}-\mathbf{r}^{\text{enc}(y)_{t}}\big{|}.$
(1)

During optimization, the prototypes in $\rho_{\upsilon}$ will be refined to
represent relative relationships between actions akin to those inferred from
the language space. Additionally, to guide S-GEAR push the action $z_{t}$
towards the prototype of the same class $k$ (i.e., $\rho_{\upsilon}[k]$) and
avoid divergences, we add a lasso regularization222$\rho_{\upsilon}[k]$ is
detached from the gradient to avoid collapsing issues during regularization.
to $\mathcal{L}_{Sem}$ as in Eq. 2.

$\begin{gathered}\mathcal{L}_{reg}=\big{|}\big{|}z_{t}-\rho_{\upsilon}[k]\big{|}\big{|}_{2}^{2}\\\
\mathcal{L}_{Sem}=\mathcal{L}_{Sem}+\mathcal{L}_{reg}.\end{gathered}$ (2)

Thus, while shaping the visual latent space geometry defined by
$\rho_{\upsilon}$ (Eq. 1), we enforce action representations to fall close to
their visual prototype (Eq. 2).

Anticipation Training. Besides prototype learning, we train S-GEAR for action
anticipation by optimizing the cross-entropy loss between the predicted class
label $\hat{y}_{T}$ and the ground truth $y_{T}$. $\hat{y}_{T}$ is obtained
from the encoded action representation and its relative position w.r.t. the
visual prototypes. More specifically, for the action representation $z_{T-1}$,
we calculate $\mathbf{r}^{z_{T-1}}$. Since $\mathbf{r}^{z_{T-1}}$ contains
values in $[-1,+1]$, we transform them into probabilistic weights using
softmax. Now, we aggregate all the prototype vectors into a single
representation, $\overline{z}_{T-1}\in\mathbb{R}^{d}$, according to the
obtained weights (see Eq. 3).

$\overline{z}_{T-1}=\text{softmax}(\mathbb{\textbf{r}}^{z_{T-1}})\cdot\rho_{\upsilon}.$
(3)

To jointly learn the action representation and its exact collocation in the
visual space w.r.t. the prototypes, we perform a weighted sum as in Eq. 4:

$\hat{z}_{T-1}=\sigma(\alpha)z_{T-1}+(1-\sigma(\alpha))\overline{z}_{T-1},$
(4)

where $\sigma$ is a sigmoid function, and $\alpha$ is a learnable scalar. Such
operations are represented as Cosine Attention in Fig. 2. Lastly, we feed
$\hat{z}_{T-1}$ through a linear layer and softmax its output to obtain
$\hat{y}_{T}$. We calculate the cross-entropy loss (Eq. 5) between the ground
truth and the predicted action class.

$\mathcal{L}_{Cls}=-\sum_{i}^{K}y_{T}^{i}\log(\hat{y}_{T}^{i}).$ (5)

Additionally, inspired by [12, 13], we leverage the causality of the decoder
$\Omega$. Here, we use any true class label for the past frames and minimize
the cross-entropy on past label predictions (Eq. 6). Notice that the predicted
label $\hat{y}_{t}$ is produced following the same reasoning described above
for $\hat{y}_{T}$ (see Eq. 3, 4).

$\mathcal{L}_{Past}=-\sum_{t=0}^{T-2}\sum_{i}^{K}y_{t+1}^{i}\log(\hat{y}_{t+1}^{i}).$
(6)

To produce faithful future features, we minimize the distance between the
predicted future frame features and the actual ones:

$\mathcal{L}_{Feat}=\sum_{t=0}^{T-2}\big{|}\big{|}\hat{I}_{t+1}-z_{t}\big{|}\big{|}.$
(7)

The overall loss function used to train S-GEAR is a weighted sum of all the
individual losses:
$\mathcal{L}_{tot}=\lambda_{1}\mathcal{L}_{Sem}+\lambda_{2}\mathcal{L}_{Cls}+\lambda_{3}\mathcal{L}_{Past}+\lambda_{4}\mathcal{L}_{Feat}$.

##  4 Experiments

###  4.1 Datasets and Metrics

The EPIC-Kitchens 55 (EK55) dataset [19] is a medium-scale first-person
cooking dataset comprising 432 videos from 32 different individuals and
approximately 40,000 segments. It encompasses 92 verbs and 272 object classes,
resulting in 2,747 action classes. Additionally, we use the train and
validation splits provided in [1]. Our model’s performance on EK55 is
evaluated using Top-1/5 Accuracy at $\tau_{a}=1$s, following prior works [1,
5, 4, 12].

The EPIC-Kitchens 100 (EK100) dataset [20] is a substantial extension of EK55,
encompassing 700 videos from 37 individuals in 45 diverse kitchens. It
comprises $\sim$90,000 activity segments spanning 495 training, 138
validation, and 67 test videos. EK100 offers a richer representation of
cooking activities through its broader range of verbs (97), objects/nouns
(300), and action classes (4,053). To assess model performance on EK100, we
employ the class aware mean Top-5 Recall [20, 12, 4] metric at $\tau_{a}=1$s.

The EGTEA Gaze+ dataset (EG) [26] includes 28 hours of first-person cooking
videos from 32 subjects across 86 sessions, covering 7 tasks. The dataset
contains 10,325 activity instances, categorized into 19 verbs, 51 objects, and
106 activity classes. To evaluate our model, we employed Top-1 Accuracy on
split 1 for $\tau_{a}=0.5$s [50, 3, 12] and Top-5 Accuracy averaged across all
three splits to evaluate overall performance for $\tau_{a}=1$s [1, 5, 36].

The 50 Salads dataset (50S) [27] comprises 50 exocentric videos featuring
salad preparation activities performed by 25 different actors and categorized
into 17 activity classes. We assess our model using mean Top-1 Accuracy across
the 5 official splits following previous works [5, 1]. Unlike other
benchmarks, the 50S offers a dense action anticipation challenge with variable
observation and anticipation times. Specifically, for a given video segment in
input, $\tau_{a}$ goes from $10\%$ to $50\%$ of the video’s duration while
$\tau_{o}$ is set to $20\%$ or $30\%$.

###  4.2 Implementation Settings

Visual Encoder. S-GEAR employs the ViT Base (ViT-B) architecture as its visual
encoder with a patch size of 16$\times$16\. It comprises 12 transformer
blocks, feature dimension 768, and operates with 12 attention heads. We set
each frame size for input dimensions to 384$\times$384 for the EK55/100
datasets and 224$\times$224 for the EG and 50S datasets. Besides the default
encoder, following prior works [4, 5, 12], we show that S-GEAR can also be
used with other backbones like TSN and irCSN using pre-extracted features as
in [1] and [12], respectively.

Intermediate Stage. Our intermediate processing stage, crucial for linking the
visual encoder’s output to the causal transformer decoder, consists of 2 TCA
blocks and 1 PA block. Note that when replacing ViT with other backbones, we
omit TCA blocks. This is because, without ViT’s detailed local patches, the
architecture essentially becomes a standard causal transformer.

Causal Transformer Decoder. For EK55/100 datasets, we employ a 6-layer causal
transformer decoder with 4 heads and a dimensionality of 2048 to process the
observed context and predict future events. For the EG dataset, we reduce the
number of layers to 2. Meanwhile, for the 50S dataset, an 8-layer decoder with
eight heads and the same dimensionality is used.

Observation. For EK100, we set the observation time, $\tau_{o}$, to 15s,
processing video segments at 1fps. For EK55 and EG, we maintain the same
processing rate but reduce $\tau_{o}$ to 10s. In contrast, for the 50S, we
align with [5, 51, 14] and adopt observation rates of 20% and 30% for each
input sequence, with 0.25fps.

Training Settings. We employ different training strategies for each dataset.
For EK100, EK55, and EG, we use an SGD optimizer with a momentum of 0.9 and
weight decay of 1e-5, processing mini-batches of 3. The learning rates are
1e-4 for EK55/100 and 4.75e-4 for EG, all with cosine scheduling and warmup of
10, 20, and 5 epochs, respectively. The total training durations are 50 epochs
for EK100, 35 for EK55, and 10 for EG. In contrast, for 50S, we opt for AdamW
optimizer with parameters $\beta_{1}$, $\beta_{2}$ set to 0.9, 0.999, a weight
decay of 1e-4, and a learning rate 5e-6. This setup also includes cosine
scheduling and 20 warmup epochs, with the model training for 100 epochs on
mini-batches of 2. Finally, we run our experiments on an RTX4090 and
2$\times$V100 GPUs.

###  4.3 Baselines

We compare against RU-LSTM [1], SRL [5], AVT [12], DCR [4], MeMViT [7],
RAFTformer [35], HRO [36], AFFT [3], TempAgg. [52], Imagination [53] and more
to ensure a fair comparison. Bold and underlined values in the tables
illustrate the best and second-best results, respectively.

Table 1: Experiments on Epic-Kitchens 55/100 for $\tau_{a}$=1s.

(a) Unimodal results on EK55 validation set.

RGB | Model | Encoder | Initialization | Top-1 Acc. | Top-5 Acc.
---|---|---|---|---|---
RU-LSTM [1] | TSN | IN1K | 13.1 | 30.8
SRL [5] | TSN | IN1K | / | 31.7
AVT [12] | TSN | IN1K | 13.1 | 28.1
AVT [12] | ViT-B | IN21K | 12.5 | 30.1
AVT [12] | irCSN | IG65M | 14.4 | 31.7
DCR [4] | TSN | IN1K | 13.6 | 30.8
DCR [4] | irCSN | IG65M | 14.4 | 34.0
DCR [4] | TSM | K400 | 16.1 | 33.1
S-GEAR (ours) | TSN | IN1K | 15.6 | 32.8
S-GEAR (ours) | irCSN | IG65M | 16.2 | 33.1
S-GEAR (ours) | ViT-B | IN21K | 15.8 | 34.5
Obj | RULSTM | FRCNN | IN1K | 10.0 | 29.8
DCR | FRCNN | IN1K | 11.5 | 30.5
S-GEAR (ours) | FRCNN | IN1K | 12.45 | 30.4
Flow | RULSTM | TSN | IN1K | 8.7 | 21.4
DCR | TSN | IN1K | 8.9 | 22.7
S-GEAR (ours) | TSN | IN1K | 10.8 | 25.8

(b) Unimodal results on EK100 validation set.

RGB | Model | Encoder | Initialization | Verb | Noun | Action
---|---|---|---|---|---|---
RU-LSTM [1] | TSN | IN1K | / | / | 13.3
AVT [12] | ViT | IN21K | 30.2 | 31.7 | 14.9
DCR [4] | TSM | K400 | 32.6 | 32.7 | 16.1
MeMViT [7] | MViTv2-16 | K400 | 32.8 | 33.2 | 15.1
MeMViT [7] | MViTv2-24 | K700 | 32.2 | 37.0 | 17.7
RAFTformer [35] | MViTv2-16 | K400 | 33.3 | 35.5 | 17.6
RAFTformer [35] | MViTv2-24 | K700 | 33.7 | 37.1 | 18.0
RAFTformer-2B [35] | MViTv2-16&24 | K400&700 | 33.8 | 37.9 | 19.1
S-GEAR (ours) | ViT-B | IN21K | 31.1 | 37.3 | 18.3
S-GEAR (ours) | TSN | IN1K | 25.8 | 29.8 | 14.9
S-GEAR (ours) | irCSN | IG65M | 26.8 | 28.8 | 13.3
S-GEAR-2B (ours) | ViT-B$\times$2 | IN21K | 32.7 | 37.9 | 19.6
Obj | AVT | FRCNN | IN1K | 18.0 | 24.3 | 8.7
DCR | FRCNN | IN1K | 22.2 | 24.2 | 9.7
S-GEAR (ours) | FRCNN | IN1K | 20.8 | 28.6 | 11.4
Flow | AVT | TSN | IN1K | 20.9 | 16.9 | 6.6
DCR | TSN | IN1K | 25.9 | 17.6 | 8.4
S-GEAR (ours) | TSN | IN1K | 21.5 | 18.2 | 7.9

(c) Multimodal results on EK55 validation set.

Model | Modalities |  | Top-1 Acc. | Top-5 Acc.
---|---|---|---|---
RU-LSTM | RGB+Obj+Flow |  | 15.3 | 35.3
TempAgg. | RGB+Obj+Flow |  | 15.1 | 35.6
Imagination | RGB+Obj+Flow |  | 15.2 | 35.4
SRL | RGB+Obj+Flow |  | / | 35.5
Ego-OMG | RGB+HOI+NAO |  | 19.2 | /
AVT+ [12] | RGB+Obj |  | 16.6 | 37.6
HRO | RGB+Obj+Flow |  | / | 37.4
DCR | RGB+Obj+Flow |  | 19.2 | 41.2
S-GEAR (ours) | RGB+Obj+Flow |  | 22.7 | 43.2

(d) Multimodal results on EK100 validation and test sets. HOI refers to Hand-
Object-Interaction.

Model | Modalities |  | Validation |  | Test
---|---|---|---|---|---
| Verb | Noun | Action |  | Verb | Noun | Action
RU-LSTM | RGB+Obj+Flow |  | 27.8 | 30.8 | 14.0 |  | 25.3 | 26.7 | 11.2
TempAgg. | RGB+Obj+Flow+HOI+Audio |  | 23.2 | 31.4 | 14.7 |  | 21.8 | 30.8 | 12.6
AVT+ | RGB+Obj |  | 28.2 | 32.0 | 15.9 |  | 25.6 | 28.8 | 12.6
AFFT | RGB+Obj+Flow+HOI+Audio |  | 22.8 | 34.6 | 18.5 |  | 20.7 | 31.8 | 14.9
S-GEAR (ours) | RGB+Obj |  | 29.5 | 37.8 | 18.9 |  | 25.9 | 32.0 | 14.7
S-GEAR-2B (ours) | RGB+Obj |  | 30.5 | 38.4 | 19.6 |  | 25.5 | 31.7 | 15.3
S-GEAR-4B (ours) | RGB+Obj |  | 30.2 | 37.0 | 19.9 |  | 26.6 | 32.6 | 15.5

###  4.4 Unimodal Comparison

Table 1(d) (a), (b) provide unimodal results on EK55 and EK100 datasets,
ensuring a fair comparison of S-GEAR against baselines. In EK55 (Table 1(d)
(a)), in RGB, S-GEAR demonstrates a point improvement of 1.1 on Top-5 Acc.
(vs. the second-best SRL) and 2.0 on Top-1 Acc. (vs. the second-best DCR) for
the TSN features. Regarding the irCSN features, S-GEAR surpasses DCR by 1.8
points in Top-1 Acc. while trailing it on Top-5 Acc. by 0.9. Using the ViT-B
backbone, S-GEAR surpasses AVT by 3.3 (Top-1) and 6.4 (Top-5) although S-GEAR
uses bigger frame sizes. For the object modality, we use Faster R-CNN features
(as in [1, 5, 4, 12]) for a fair comparison, obtaining 0.9 Top-1 Acc.
improvement, yet failing behind on Top-5 by 0.1. Finally, S-GEAR yields 1.9
(Top-1) and 3.1 (Top-5) point gains for the flow modality over prior works.

Table 1(d) (b) details our results on the more complex EK100 benchmark. Here,
S-GEAR competes with MeMViT [7] and RAFTformer [35] (with the MViTv2-16
backbone) on the RGB modality. S-GEAR demonstrates improvements in Top-5
Recall for actions (3.2 over MeMViT, 0.7 over RAFTformer) and nouns (4.1 over
MeMViT, 1.8 over RAFTformer). While trailing slightly on verbs, unlike its
competitors (Kinetics-400), S-GEAR performs well without spatiotemporal
initialization. Surprisingly, even when MeMViT and RAFTformer employ larger
backbones with Kinetics-700 initialization, S-GEAR exceeds them on actions and
nouns. Additionally, we formed S-GEAR-2B by late-fusing two S-GEAR versions
with ViT-B backbones (input $224\times 224$ and $384\times 384$). Despite
being a late fusion (compared to RAFTformer-2B’s joint architecture),
S-GEAR-2B achieves a 0.5 improvement in action –- all without spatiotemporal
initialization. Furthermore, S-GEAR demonstrates strong performance compared
to AVT and DCR across modalities, achieving gains of 3.4 and 2.2 for action
Top-5 Recall in RGB. On the other hand, on object modality, S-GEAR shows gains
of 1.7 and 4.4 (actions, nouns) and slightly trails DCR on verbs. Finally,
S-GEAR remains competitive even in the flow modality. Certainly, this detailed
comparison verifies the contribution of S-GEAR in training effective
anticipation models aware of action semantic interconnections, paving the way
for further research.

Table 2: EG results regarding Top-1 Acc. for $\tau_{o}=0.5$s and Top-5 Acc.
for $\tau_{a}=1.0$s.

Model | Modalities |  | Top-1 Acc.
---
($\tau_{o}=0.5$s)
| Top-5 Acc.
---
($\tau_{a}=1$s)
RU-LSTM [1] | RGB+Flow | / | 66.40
DCR [4] | RGB+Flow | / | 67.9
SRL [5] | RGB+Flow | / | 70.7
HRO [36] | RGB+Flow+Obj | / | 71.5
AVT [12] | RGB | 43.0 | /
AFFT [3] | RGB+Flow | 42.5 | /
S-GEAR (ours) | RGB | 45.7 | 71.9

Table 3: 50S results on dense action anticipation. (Percentages are w.r.t. the
video duration).

$\tau_{o}\rightarrow$ | 20% | 30%
---|---|---
$\tau_{a}\rightarrow$ | 10% | 20% | 30% | 50% | 10% | 20% | 30% | 50%
RU-LSTM [1] | 22.2 | 17.8 | 12.7 | 08.3 | 22.3 | 15.5 | 10.8 | 05.2
CNN model [50] | 21.2 | 19.0 | 16.0 | 09.9 | 29.1 | 20.1 | 17.5 | 10.9
Grammar-based [54] | 24.7 | 22.3 | 19.8 | 12.7 | 29.7 | 19.2 | 15.2 | 13.1
Uncertainty [14] | 28.9 | 22.4 | 19.9 | 12.8 | 29.1 | 20.5 | 15.3 | 12.3
RNN [50] | 30.1 | 25.4 | 18.7 | 13.5 | 30.8 | 17.2 | 14.8 | 09.8
Time-Cond. [51] | 32.5 | 27.6 | 21.3 | 16.0 | 35.1 | 27.1 | 22.1 | 15.5
SRL [5] | 37.9 | 28.8 | 21.3 | 11.1 | 37.5 | 24.1 | 17.1 | 09.1
S-GEAR (ours) [5] | 41.0 | 28.5 | 21.5 | 15.3 | 41.0 | 27.8 | 21.4 | 16.7

###  4.5 Comparison with the SOTA

Epic-Kitchens. Previous approaches often utilize cross-modality ensembling
[20, 3] or joint training [1, 3] for multimodal evaluation on these
benchmarks. Ensembling S-GEAR across modalities, we observe significant gains.
On EK55 (Table 1(d) (c)), late-fusing our models (RGB+Obj+Flow) yields a boost
of 3.5 (Top-1 Acc.) and 2.0 (Top-5 Acc.) absolute points, outperforming prior
work. Similarly, on EK100 (Table 1(d) (d)), late-fusing RGB modalities with
object features leads to a 1.4 (0.8 against RAFTformer) point improvement in
action Top-5 Recall. Finally, though we report EK100 test set results (Table
1(d) (d)) and obtain competitive performances, it is crucial to note that
leaderboard rankings often rely on large-scale external data or fusion across
diverse models (i.e., the de-emphasized models on Table 1(d) (d)). This makes
the test set less effective for comparing the core strengths of models [4]. We
point the reader to Appendix B.2 for details on our specific ensembling
weights.

EGTEA Gaze+. We evaluate S-GEAR on two task on EG (Table 2). The first
includes Top-1 Acc. on split-1 for $\tau_{a}=0.5$ where we achieve 2.5 point
improvement compared to previous work. The second includes the average Top-5
Acc. across the three splits at $\tau_{a}=1s$ where we surprisingly improve on
HRO with 0.4 points despite using only the RGB modality with our ViT-B
backbone.

50 Salads. Our dense anticipation experiments on the 50S (Table 3) show
S-GEAR’s potential for long-term and exocentric tasks. It outperforms
competitors in 5/8 scenarios, with Top-1 Accuracy gains of up to 3.5, despite
not being tailored for long-term anticipation like Time-Cond. [51].

###  4.6 Ablation Study

We analyze the importance of S-GEAR’s components to justify our design
choices. Specifically, we investigate (a) the impact of architectural and
training elements, (b) the significance of encoding semantic action
relationships, (c) the number of prototypes for defining relative action
positions, and (d) S-GEAR’s performance for different anticipation time
$\tau_{a}$.

Table 4: Ablation study (Top-5 Recall) on EK100 validation set.

Settings | TCA | Sem | PA | Verb | Noun | Action
---|---|---|---|---|---|---
(1) Baseline | - | - | - | 30.5 | 32.6 | 15.2
(2) Sem | - | ✓ | - | 30.7 | 35.7 | 17.8
(3) TCA | ✓ | - | - | 31.0 | 33.9 | 16.7
(4) PA + Sem | - | ✓ | ✓ | 32.0 | 36.2 | 18.0
(5) TCA + PA ($\rho_{\ell}$) | ✓ | - | ✓ | 30.6 | 33.3 | 17.4
S-GEAR | ✓ | ✓ | ✓ | 31.1 | 37.3 | 18.3

[]

Figure 3: Ablation on language encoders.

[]

Figure 4: Performance according to used prototype ratio.

(a) We use EK100 (RGB) to evaluate the impact of architectural components and
our prototype learning strategy (see Table 4). We use a baseline (1)
comprising a ViT-B encoder, a casual decoder, and a linear classification head
similar to AVT [12]. On top of this baseline, we switch on/off each component
that comprises S-GEAR: i.e., (2) the prototype learning with semantic
guidance, including the cosine attention block on the classification head, (3)
the TCA block, and (4) the PA block. Note that the PA block needs prototypes;
thus, in the table, we toggle the semantic column as well. While all
strategies improve over (1), (2) has the most impact, adding up to 2.6 points
on Top-5 Recall for action classes. Such improvements are caused by the
ability of the prototype learning strategy to cluster actions that co-occur
frequently. The network then uses this proximity to encode action
representations aware of their exact collocation through the cosine attention
block, taking hints that the next probable action can be found in its
proximity in the latent space. Finally, to motivate our choice of learning
visual prototypes $\rho_{\upsilon}$ rather than directly using language
prototypes $\rho_{\ell}$, we rely on (5), which includes all the architecture
components except the prototype learning strategy. Instead, action
representations are directly aligned with fixed $\rho_{\ell}$. This resulted
in decreased performance compared to S-GEAR. While $\rho_{\ell}$ captures
semantic structure, we believe it lacks the scene information crucial for
accurate anticipation, such as motion and visual context. S-GEAR overcomes
this limitation by learning its visual prototypes, allowing them to adapt to
the specific visual cues relevant to the task.

(b) S-GEAR builds on the principle that semantically similar actions often co-
occur, making semantic relationship encoding crucial. To ablate on the
importance of such relationships, we leverage two Sentence Transformer
variations from HuggingFace: “bert-large-nli-max-token” (BERT) and “stsb-
mpnet-base-v2” (STSB). These models share a similar architecture but differ in
training data size, with STSB being better at semantic relation extraction.
Fig. 4 shows that S-GEAR performs better with STSB-generated prototypes,
highlighting that modeling accurate semantic interconnections gives better
results.

(c) While prototypes are valuable, they introduce a computational cost due to
their large matrix size (e.g., in EK100 with 4053 actions). In this regard, we
investigate the possibility of approximating an action’s relative position by
comparing it to only a subset of prototypes. Experiments on EK100, using
varying portions of visual prototypes (see Fig. 4) show that we can achieve
good results using only a fraction (i.e., 17.8 Top-5 Recall at 10% vs. 18.3
Top-5 Recall at 100%) of the prototypes while significantly reducing the
number of computations (i.e., $\sim$405 instead of 4053 comparisons, entailing
a 90% drop when computing the similarity between the action representations
and prototypes).

[]

Figure 5: EK55 (top) and EG (bottom) Top-5 Acc. for variable $\tau_{a}$.

[]

Figure 6: Qualitative example of observed actions (Observation), the target
activity (Target), S-GEAR’s Top-5 predictions, and the Top-5 semantically
similar actions with the observed sequence based on language encoding
(Semantic).

(d) Finally, we evaluate the performance of S-GEAR for variable $\tau_{a}$. We
expect the performance to drop as $\tau_{a}$ increases. Hence, we experiment
on EK55 and EG training S-GEAR with $\tau_{a}=0.25$ and test its
autoregressive capabilities by increasing $\tau_{a}$ up to 2s at inference
time. We report the results in Fig. 6. While the performance drop is
highlighted as $\tau_{a}\rightarrow\infty$, we notice that S-GEAR performs
better than previous works on EK55. On the other hand, on EG, S-GEAR remains
highly competitive, slightly trailing HRO and SRL with $\tau_{a}>1$s.

###  4.7 Qualitative Results

Fig. 6 demonstrates S-GEAR’s ability to anticipate future actions on the EG
dataset, using $\tau_{a}=1$s and $\tau_{o}=32$s. Alongside S-GEAR’s Top-5
predictions, we include the Top-5 semantically similar language prototypes
given the observed action sequence. These examples reveal the connection
between anticipation and semantics, suggesting that an alignment between the
two exists. On the other hand, the last row example also highlights
divergences emphasizing S-GEAR’s room for semantic improvement. To further
investigate the semantic alignment between S-GEAR and language prototypes, in
Fig. 7, we illustrate the geometric association learned by S-GEAR prototypes
(middle) on EG, comparing it with its initial values (left) and the language
prototypes (right) both in terms of absolute and relative positions. The
latter is determined using cosine similarity to compare each prototype against
all others. S-GEAR’s prototypes demonstrate a latent space topology closer to
the language prototypes than its counterpart w/o semantic alignment in terms
of absolute and relative position. Such phenomenon indicates that S-GEAR can
reason upon the semantic connectivity between actions, projecting contextually
similar ones closer in latent space. However, S-GEAR’s topology is slightly
different since visual cues influence inter-prototype distances. We point the
reader to Appendix B for more experimental details.

Figure 7:  Illustration (via UMAP [55]) of the absolute and relative position
of visual action prototypes w/a semantic alignment (left), after semantic
alignment (middle), and language prototypes (right) for EG.

##  5 Conclusion

We presented S-GEAR, a novel framework for action anticipation that leverages
semantic interconnectivity between actions. S-GEAR learns visual and language
prototypes that encode typical action patterns and their relationships based
on contextual co-occurrences. S-GEAR transfers the geometric associations
between actions from language to vision without direct alignment, creating a
common communication space. S-GEAR employs a transformer-based architecture
incorporating temporal context aggregation and prototype attention to enhance
the action representations and predict future events. We evaluate S-GEAR on
four action anticipation benchmarks, showing improved results compared to
previous works. We also demonstrate that we can effectively encode semantic
relationships between actions, opening new research frontiers in anticipation
tasks. While S-GEAR shows promising results, its limitations include the lack
of an in-built multimodal mechanism and semantic interconnections that
explicitly account for occurrence order. Accounting for co-occurrence orders
can reduce future prediction uncertainty, narrowing the scope of future action
to those likely to follow the observed sequence. We will address these
limitations in future work.

## Acknowledgments

We would like to express our sincere gratitude to Professor Giovanni Maria
Farinella and Professor Antonino Furnari from the University of Catania for
their invaluable guidance, support, and insightful feedback. Their expertise
and encouragement were instrumental in the successful completion of this work.

## References

  * [1] Antonino Furnari and Giovanni Maria Farinella.  Rolling-unrolling lstms for action anticipation from first-person video.  IEEE Trans. on Pattern Anal. and Mach. Intell., 43(11):4021–4036, 2021.
  * [2] Francesco Marchetti, Federico Becattini, Lorenzo Seidenari, and Alberto Del Bimbo.  Multiple trajectory prediction of moving agents with memory augmented networks.  IEEE Trans. on Pattern Anal. and Mach. Intell., 45(6):6688–6702, 2023.
  * [3] Zeyun Zhong, David Schneider, Michael Voit, Rainer Stiefelhagen, and Jurgen Beyerer.  Anticipative feature fusion transformer for multi-modal action anticipation.  In Proc. of the IEEE/CVF Winter Conf. on Appl. of Comput. Vis., pages 6068–6077, 2023.
  * [4] Xinyu Xu, Yong-Lu Li, and Cewu Lu.  Learning to anticipate future with dynamic context removal.  In Proc. of the IEEE/CVF Conf. on Comput. Vis. and Pattern Recognit., pages 12724–12734, 2022.
  * [5] Zhaobo Qi, Shuhui Wang, Chi Su, Li Su, Qingming Huang, and Qi Tian.  Self-regulated learning for egocentric video activity anticipation.  IEEE Trans. on Pattern Anal. and Mach. Intell., 45(6):6715–6730, 2023.
  * [6] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer.  Mvitv2: Improved multiscale vision transformers for classification and detection.  In Proc. of the IEEE/CVF Conf. on Comput. Vis. and Pattern Recognit., pages 4804–4814, 2022.
  * [7] Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer.  Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition.  In Proc. of the IEEE/CVF Conf. on Comput. Vis. and Pattern Recognit., pages 13587–13597, 2022.
  * [8] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In Proc. of the IEEE/CVF Int. Conf. on Comput. Vis., pages 6836–6846, 2021.
  * [9] Rohit Girdhar and Deva Ramanan.  Attentional pooling for action recognit.  Adv. Neural Inf. Process. Syst., 30:1–10, 2017.
  * [10] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al.  Swin transformer v2: Scaling up capacity and resolution.  In Proc. of the IEEE/CVF Conf. on Comput. Vis. and pattern Recognit., pages 12009–12019, 2022.
  * [11] Antoine Miech, Ivan Laptev, and Josef Sivic.  Learnable pooling with context gating for video classification.  In Proc. of the IEEE/CVF Conf. on Comput. Vis. and pattern Recognit. Workshop, pages 1–8, 2017.
  * [12] Rohit Girdhar and Kristen Grauman.  Anticipative video transformer.  In Proc. of the IEEE/CVF Int. Conf. on Comput. Vis., pages 13505–13515, 2021.
  * [13] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba.  Anticipating visual representations from unlabeled video.  In Proc. of the IEEE/CVF Conf. on Comput. Vis. and Pattern Recognit., pages 98–106, 2016.
  * [14] Yazan Abu Farha and Juergen Gall.  Uncertainty-aware anticipation of activities.  In Proc. of the IEEE/CVF Int. Conf. on Comput. Vis. Workshops, pages 0–8, 2019.
  * [15] C.H. Frederiksen.  Propositional representations in psychology.  In Int. Encyclopedia of the Social & Behavioral Sci.s, pages 12219–12224. Springer, 2001.
  * [16] Vignesh Ramanathan, Congcong Li, Jia Deng, Wei Han, Zhen Li, Kunlong Gu, Yang Song, Samy Bengio, Charles Rosenberg, and Li Fei-Fei.  Learning semantic relationships for better action retrieval in images.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 1100–1109, 2015.
  * [17] Steven Wilson and Rada Mihalcea.  Measuring semantic relations between human activities.  In Int. Joint Conf. on Nat. Lang. Process., pages 664–673, 2017.
  * [18] John A Bullinaria and Joseph P Levy.  Extracting semantic representations from word co-occurrence statistics: A computational study.  Behavior Research Method., 39:510–526, 2007.
  * [19] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al.  Scaling egocentric vision: The epic-kitchens dataset.  In Proc. of the Eur. Conf. on Comput. Vis., pages 720–736, 2018.
  * [20] Damen Dima, Hazel Doughty, Giovanni Maria Farinella, Furnari Antonino, Kazakos Evangelos, Jian Ma, Moltisanti Davide, Jonathan Munro, Perrett Toby, Will Price, et al.  Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.  Int. J. of Comput. Vis., 130(1):33–55, 2022.
  * [21] Wadhwa Somin, Amir Silvio, and C. Wallace Byron.  Revisiting relation extraction in the era of large language models.  In Proc. of the 61st Conf. Assoc. Comput. Linguist. Meet., pages 15566–15589, 2023.
  * [22] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In Proc. of the 2019 Conf. on Empir. Methods Nat. Lang. Process. and the 9th Int. Joint Conf. on Nat. Lang. Process., pages 3980–3990, 2019.
  * [23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In Int. Conf. on Learning Representations, pages 1–21, 2021.
  * [24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Adv. Neural. Inf. Process. Syst., volume 30, pages 1–11, 2017.
  * [25] Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao.  Learning deep transformer models for machine translation.  In Proc. of the 57th Conf. Assoc. Comput. Linguist. Meet., pages 1–13, 2019.
  * [26] Mengmi Zhang, Keng Teck Ma, Joo Hwee Lim, Qi Zhao, and Jiashi Feng.  Deep future gaze: Gaze anticipation on egocentric videos using adversarial networks.  In IEEE Conf. on Comput. Vis. and Pattern Recognit., pages 3539–3548, 2017.
  * [27] Sebastian Stein and Stephen J McKenna.  Combining embedded accelerometers with computer visio for recognizing food preparation activities.  In Proc. of the 2013 ACM Int. Conf. Ubiquitous Comput., pages 729–738, 2013.
  * [28] Hema S Koppula and Ashutosh Saxena.  Anticipating human activities using object affordances for reactive robotic response.  IEEE Trans. on Pattern Anal. and Mach. Intell., 38(1):14–29, 2015.
  * [29] Krishna Kumar Singh, Kayvon Fatahalian, and Alexei A Efros.  Krishnacam: Using a longitudinal, single-person, egocentric dataset for scene understanding tasks.  In Proc. of the IEEE/CVF Winter Conf. on Appl. of Comput. Vis., pages 1–9, 2016.
  * [30] Bilge Soran, Ali Farhadi, and Linda Shapiro.  Generating notifications for missing actions: Don’t forget to turn the lights off!  In Proc. of the IEEE/CVF Int. Conf. on Comput. Vis., pages 4669–4677, 2015.
  * [31] Syed Zahir Bokhari and Kris M Kitani.  Long-term activity forecasting using first-person vision.  In 13th Asian Conf. on Comput. Vis., pages 346–360, 2017.
  * [32] Antonino Furnari, Sebastiano Battiato, and Giovanni Maria Farinella.  Leveraging uncertainty to rethink loss functions and evaluation measures for egocentric action anticipation.  In Proc. of the Eur. Conf. on Comput. Vis. Workshops, pages 0–10, 2018.
  * [33] Eadom Dessalene, Chinmaya Devaraj, Michael Maynord, Cornelia Fermüller, and Yiannis Aloimonos.  Forecasting action through contact representations from first person video.  IEEE Trans. on Pattern Anal. and Mach. Intell., 45(6):6703–6714, 2023.
  * [34] Debaditya Roy and Basura Fernando.  Action anticipation using latent goal learning.  In Proc. of the IEEE/CVF Winter Conf. on Appl. of Comput. Vis., pages 2745–2753, 2022.
  * [35] Harshayu Girase, Nakul Agarwal, Chiho Choi, and Karttikeya Mangalam.  Latency matters: Real-time action forecasting transformer.  In IEEE/CVF Conf. on Comput. Vis. and Pattern Recognit., pages 18759–18769, 2023.
  * [36] Tianshan Liu and Kin-Man Lam.  A hybrid egocentric activity anticipation framework via memory-augmented recurrent and one-shot representation forecasting.  In Proc. of the IEEE/CVF Conf. on Comput. Vis. and Pattern Recognit., pages 13904–13913, 2022.
  * [37] Debaditya Roy, Ramanathan Rajendiran, and Basura Fernando.  Interaction region visual transformer for egocentric action anticipation.  In Winter Conf. on Appl. of Comp. Vis., pages 6740–6750, 2024.
  * [38] Dafeng Wang, Hongbo Liu, Naiyao Wang, Yiyang Wang, Hua Wang, and Seán McLoone.  Seem: A sequence entropy energy-based model for pedestrian trajectory all-then-one prediction.  IEEE Trans. on Pattern Anal. and Mach. Intell., 45(1):1070–1086, 2023.
  * [39] Nada Osman, Guglielmo Camporese, Pasquale Coscia, and Lamberto Ballan.  Slowfast rolling-unrolling lstms for action anticipation in egocentric videos.  In IEEE/CVF Int. Conf. on Comput. Vis. Workshops, pages 3430–3438, 2021.
  * [40] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In Proc. of the IEEE/CVF Conf. on Comput. Vis. and Pattern Recognit., pages 18123–18133, 2022.
  * [41] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In Int. Conf. on Mach. Learning, pages 4904–4916, 2021.
  * [42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In Int. Conf. on Mach. Learning, pages 8748–8763, 2021.
  * [43] Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji.  X-clip: End-to-end multi-grained contrastive learning for video-text retrieval.  In Proc. of the 30th ACM Int. Conf. on Multimed., pages 638–647, 2022.
  * [44] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong.  Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.  Adv. Neural Inf. Process. Syst., 34:24206–24221, 2021.
  * [45] Jingjia Huang, Yinan Li, Jiashi Feng, Xinglong Wu, Xiaoshuai Sun, and Rongrong Ji.  Clover: Towards a unified video-language alignment and fusion model.  In Proc. of the IEEE/CVF Conf. on Comput. Vis. and Pattern Recognit., pages 14856–14866, 2023.
  * [46] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 1199–1208, 2018.
  * [47] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  Adv. Neural Inf. Process. Syst., 30, 2017.
  * [48] Fanchao Lin, Zhaofan Qiu, Chuanbin Liu, Ting Yao, Hongtao Xie, and Yongdong Zhang.  Prototypical matching networks for video object segmentation.  IEEE Trans. on Imag. Process., 32:5623–5636, 2023.
  * [49] Brais Martinez, Davide Modolo, Yuanjun Xiong, and Joseph Tighe.  Action recognition with spatial-temporal discriminative filter banks.  In Int. Conf. Comp. Vis., pages 5482–5491, 2019.
  * [50] Yazan Abu Farha, Alexander Richard, and Juergen Gall.  When will you do what?-anticipating temporal occurrences of activities.  In Proc. of the IEEE/CVF Conf. on Comput. Vis. and Pattern Recognit., pages 5343–5352, 2018.
  * [51] Qiuhong Ke, Mario Fritz, and Bernt Schiele.  Time-conditioned action anticipation in one shot.  In Proc. of the IEEE/CVF Conf. on Comput. Vis. and Pattern Recognit., pages 9925–9934, 2019.
  * [52] Fadime Sener, Dipika Singhania, and Angela Yao.  Temporal aggregate representations for long-range video understanding.  In Proc. of the Eur. Conf. on Comput. Vis. (ECCV), pages 154–171. Springer, 2020.
  * [53] Yu Wu, Linchao Zhu, Xiaohan Wang, Yi Yang, and Fei Wu.  Learning to anticipate egocentric actions by imagination.  IEEE Trans. on Imag. Process., 30:1143–1152, 2020.
  * [54] Alexander Richard, Hilde Kuehne, and Juergen Gall.  Weakly supervised action learning with rnn based fine-to-coarse modeling.  In Proc. of the IEEE/CVF Conf. on Comput. Vis. and Pattern Recognit., pages 754–763, 2017.
  * [55] Leland McInnes, John Healy, and James Melville.  Umap: Uniform manifold approximation and projection for dimension reduction.  arXiv preprint arXiv:1802.03426, 2018.
  * [56] Feiqing Huang, Kexin Lu, CAI Yuxi, Zhen Qin, Yanwen Fang, Guangjian Tian, and Guodong Li.  Encoding recurrence into transformers.  In The Eleventh Int. Conf. on Learning Representations, pages 1–13, 2022.
  * [57] Roeland De Geest and Tinne Tuytelaars.  Modeling temporal structure with lstm for online action detection.  In Proc. of the IEEE/CVF Winter Conf. on Appl. of Comput. Vis., pages 1549–1557, 2018.
  * [58] Shugao Ma, Leonid Sigal, and Stan Sclaroff.  Learning activity progression in lstms for activity detection and early detection.  In Proc. of the IEEE/CVF Conf. on Comput. Vis. and pattern Recognit., pages 1942–1950, 2016.

##  Appendix A S-GEAR’s Details

The main paper gives a general overview of S-GEAR architecture, which
comprises a visual encoder, the temporal content aggregation module, the
prototype attention module, and the causal transformer decoder. While the
visual encoder [23, 24] – a standard ViT – and the temporal decoder [24, 4,
12]– causal transformer based on masked-attention – are well-known
architectures in the literature, we specifically tailor the two intermediate
modules to serve our purposes. We provide the details of such blocks in Sec.
A.1 and A.2. In Sec. A.3, give details regarding the visual prototype
initialization. Finally, in Sec. A.4, we give details regarding the training
strategy used to train S-GEAR with pre-extracted features.

Figure 8: Standard causal transformer flow vs. TCA flow.

###  A.1 Temporal Context Aggregation

Sequential models like LSTMs and causal transformers excel at handling
temporal frame sequences. However, relying on class tokens, they prioritize
global information [4, 12] and neglect spatial cues. To illustrate the
difference between the standard causal transformer and our proposed Temporal
Context Aggregator (TCA), we provide the reader with Fig. 8. Here, the left-
hand side shows the workflow of a standard causal transformer applied on a
sequence of ViT frame features composed of local feature tokens and a global
class token. In this scenario, the causal transformer omits the local
information and only propagates the global information in time to create
causal representations [12, 4]. Because local tokens encode specific scene
details within different regions, not propagating their information hinders
the model from understanding scene dynamics (e.g., how an object’s location
changes as a particular action progresses). Therefore, we design the TCA
$\varphi$ block. TCA extends the information flow by propagating global and
local tokens across time (see Fig. 8 right), building causal representations
considering scene dynamics at a finer spatial scale.

TCA builds on the attention mechanism and processes intermediate features
$I_{t}$. Thus, $I_{t}$ undergoes linear processing to generate the query
($Q_{t}$), key ($K_{t}$), and value ($V_{t}$) vector representations.
Afterward, as shown in Fig. 9 (left), the TCA uniquely aggregates keys and
values from past frames to subsequent ones before computing the attention
matrix. This approach enables the queries of each frame to access a rich set
of keys and values infused with comprehensive spatiotemporal information about
past contexts, enabling better temporal dependency [7]. Specifically, the
$K_{t}$ and $V_{t}$ vectors are augmented as in Eqns. 8 and 9, respectively:

$\hat{K}_{t}=\begin{cases}K_{t}&\text{if}\ t=0\\\
\delta(K_{t},\alpha_{t-1}\cdot\hat{K}_{t-1})&\text{otherwise}\end{cases},$ (8)
$\hat{V}_{t}=\begin{cases}V_{t}&\text{if}\ t=0\\\
\delta(V_{t},\alpha_{t-1}\cdot\hat{V}_{t-1})&\text{otherwise}\end{cases},$ (9)

where $\hat{K}_{t}$ and $\hat{V}_{t}$ are the augmented keys and queries of
frame $f_{t}$, $\alpha_{t-1}$ is a learnable weight parameter that balances
the quantity of the information transmitted from past observations, and
$\delta$ is a permutation invariant aggregation function. Note that we tried
different functions for $\delta$ (e.g., cumulative-max), but through empirical
analyses, we chose summation.

Figure 9:  The TCA block (left) generates detailed and sequential
representations by aggregating past observation values $V$ and keys $K$ into
current ones. It follows a standard attention mechanism to capture temporal
dependencies. The PA block (right) performs cross-modality attention to
aggregate selected prototypes from visual queries and uses a TOE weight matrix
to encode temporal awareness. Note that the red gradient in TOE represents the
magnitude of learnable weights.

Once these vectors are obtained, the computation proceeds with the standard
self-attention operations. Precisely, $Q_{t}$ and $\hat{K}_{t}$ compute the
attention scores through scaled matrix multiplication, then normalized into
$[0,1]$ weights through a softmax function. Afterwards, the weights aggregate
information between the augmented feature tokens of $\hat{V}_{t}$ and create
causal representations $\overline{I}_{t}$. Formally, the procedure can be
defined as follows:

$\overline{I}_{t}=\text{softmax}\bigg{(}\frac{Q_{t}\hat{K}_{t}^{\top}}{\sqrt{d}}\bigg{)}\hat{V}_{t}.$
(10)

Now, the class token representations $\overline{I}_{t}^{0}$ of
$\overline{I}_{t}$ encodes the global representation of frame $t$ enhanced by
detailed contextual information of the past frames.

###  A.2 Prototype Attention

Recall that S-GEAR encodes semantic relationships between actions. To help the
encoding of such relationships, we integrate a Prototype Attention (PA) block
$\gamma$ detailed in Fig. 9 (right). The PA module helps the network learn
meaningful representations by incorporating semantic information from the
visual prototypes. PA has two stages: (1) selecting the prototypes and (2)
modeling the relationship between features and prototypes.

Similar to TCA, we build PA upon the attention mechanism, giving in input both
the class tokens from the intermediate encodings generated by ViT – i.e.,
$I^{0}=\\{I_{0}^{0},I_{1}^{0},\dots,I_{T-1}^{0}\\}$ and the visual prototypes
$\rho_{\upsilon}$. We rely on the relative similarities between actions to
address (1). Specifically, we begin by calculating the cosine similarity
between each $I_{t}^{0}$ and the visual prototypes, obtaining the relative
representation vector $\mathbf{r}^{I^{0}_{t}}$ of frame $t$ as in Eq. 11:

$\mathbf{r}^{I^{0}_{t}}=cos(I_{t}^{0},\rho_{\upsilon}).$ (11)

Then, we select the top $k$ most similar prototypes for each feature vector
for the remaining calculations. However, for simplicity, let us assume we
select the most similar prototype for each feature vector. After acquiring the
estimated prototypes, PA addresses (2) by modeling their relationship with the
feature encodings using the attention mechanism. In this case, the set of
prototypes represents both the key ($K$) and value ($V$) vectors. Conversely,
the query ($Q$) vector is derived from $I^{0}$. The first step of the
relationship modeling is the computation of the attention scores $W_{a}$
through a scaled matrix multiplication between $Q$ and $K$ as in Eq. 12:

$W_{a}=\frac{QK^{\top}}{d}.$ (12)

Following the standard attention procedure, the next step in the attention
process should be normalizing and applying $W_{a}$ to $V$ and having the
output features. However, the selected prototypes do not have temporal
continuity like the sequential frames and contradict the temporal causality
built from TCA when the fusion occurs (see Sec. 3.1 in MP). Inspired by [56],
we introduce a Temporal Order Encoding (TOE) weight vector shaped as a
Toeplitz matrix to model the temporal order between elements of $V$. We
provide the reader with an example to illustrate Toeplitz matrices and their
unique structure. Here, we show a 5-element TOE as a $3\times 3$ Toeplitz
matrix $\Delta$ as in Eq. 13:

$\Delta=\left(\begin{array}[]{ccc}w_{0}&w_{1}&w_{2}\\\ w_{3}&w_{0}&w_{1}\\\
w_{4}&w_{3}&w_{0}\\\ \end{array}\right),$ (13)

where $w_{i}$ represents the $i^{\text{th}}$ weight from the TOE for
$i\in\\{0,1,...,4\\}$. Notice that a single weight represents each diagonal.
Additionally, with a $3\times 3$ matrix, we can model the temporal
relationships of a sequence of three elements. Hence, for a sequence of $T$
elements like $V$, we need a $T\times T$ Toeplitz matrix built from a
($2T-1$)-element TOE. Generalizing, the $T\times T$ $\Delta$ functionality
allows PA to model the relative temporal position or order between elements of
$V$ when aggregating features. To apply $\Delta$ to $V$, we first sum $\Delta$
with the normalized $W_{a}$ and then perform a matrix multiplication between
the resulting matrix and $V$ as in Eq. 14:

$\tilde{I}=(\beta(\text{softmax}(W_{a}))+(1-\beta)\Delta)V,$ (14)

where $\beta$ is a learnable scaling factor that balances the sum between
$W_{a}$ and $\Delta$. $\tilde{I}\in\mathbb{R}^{T\times d}$ now represents the
selected visual prototypes fused with the current context and with encoded
relative temporal awareness.

###  A.3 Prototype Initialization

We initialize our visual prototypes $\rho_{\upsilon}$ using action samples
generated by the proposed architecture (detailed in Sec. 3.1 in MP). First, we
train the network (PA omitted) for action recognition on EK100 for the
egocentric tasks and 50S for the exocentric tasks. Then, without fine-tuning,
we extract multiple representations for each action class present in a
dataset. The average of these representations becomes the “typical action
sample” used to initialize the corresponding class prototype. We also
experimented with other initialization methods like random values and using
the language prototypes $\rho_{\ell}$ as initial values. However, such methods
did not provide significant improvements. Thus, the results we report in MP
rely on the first initialization approach.

Figure 10: Training S-GEAR with pre-extracted features.

###  A.4 Training S-GEAR With Pre-extracted Features

S-GEAR’s modular design allows it to work with different visual backbones,
although it’s primarily designed for end-to-end training with a ViT
architecture [23]. To ensure a fair comparison with the SOTA [1, 4, 12], we
also train our network using pre-extracted features from TSN, Faster R-CNN
(FRCNN) and irCSN backbones provided by Furnari et al. [1] and Girdhar et al.
[12], respectively.

Our training process (see Fig. 10) is built upon aligning the pre-extracted
feature distribution with $\rho_{\upsilon}$ learned from S-GEAR in its end-to-
end training with ViT. These prototypes already capture the desired structure
of the latent space. By aligning with them, we simplify training when we lack
the variability introduced by typical video preprocessing techniques (i.e.,
random cropping or flipping). To this end, given pre-extracted visual features
$\chi_{t}$ $\forall t\in[0,T-1]$, we first apply a linear transformation and
then normalize them using the mean $\mu_{\rho_{\upsilon}}$ and standard
deviation $\sigma_{\rho_{\upsilon}}$ from the learned prototypes as in Eq. 15:

$I_{t}=\frac{\text{lin}(\chi_{t})-\mu_{\rho_{\upsilon}}}{\sigma_{\rho_{\upsilon}}}.$
(15)

Hence, leveraging the pre-trained weights of PA, causal decoder, and
classification head from the end-to-end training, we fine-tune S-GEAR to adapt
to the new visual features $I_{t}$. During this process, $\rho_{\upsilon}$
remains unchanged.

##  Appendix B Experiments Extension

Here, we provide additional details regarding our experiments. In Sec. B.1, we
provide details regarding the weights used for each loss function to train
S-GEAR for each dataset. In Sec. B.2, we provide the results supporting the
ablation study on the anticipation time $\tau_{a}$ from Sec. 4.6 in the MP. In
Sec. B.3, we give details regarding the ensemble setup of our models used to
obtain the multimodal results on EK55/100 from the MP. In Sec. B.4, we try
different combinations of backbones and modalities on EK100’s validation set.
Finally, in Sec. B.5, we extend Sec. 4.7 from the MP and compare the
semanticity learned from $\rho_{\upsilon}$ and that encoded from
$\rho_{\ell}$.

###  B.1 Composed Loss Weights

Here, we provide the specific weights for each loss term introduced in Sec.
3.3 of the MP. Table 5 details these weights. Notice that we combine
$\mathcal{L}_{Sem}$ and $\mathcal{L}_{Reg}$ due to their similar purpose in
MP. However, they are weighted individually during optimization to account for
their different magnitudes. Importantly, we use the same weights for both
EK100 and EK55 datasets, regardless of whether training end-to-end with ViT
features or using pre-extracted features. The sole exception is
$\mathcal{L}_{Sem}$ because visual prototypes remain frozen in the pre-
extracted feature training scenario (see Sec. A.4).

Table 5: Weights associated with individual loss terms to train S-GEAR for each task. Dataset |  | $\mathcal{L}_{Sem}$ | $\mathcal{L}_{Reg}$ | $\mathcal{L}_{Cls}$ | $\mathcal{L}_{Past}$ | $\mathcal{L}_{Feat}$
---|---|---|---|---|---|---
EK100 |  | 4.0 | 1.0 | 1.0 | 1.0 | 1.0
EK55 |  | 2.0 | 1.0 | 1.0 | 1.0 | 1.0
EG |  | 2.0 | 1.0 | 1.0 | 0.1 | 1.0
50S |  | 1.0 | 0.1 | 1.0 | 0.1 | 1.0

###  B.2 Ablation on anticipation time

Here, we report the results of experiments in Sec. 4.6 of MP. Specifically,
the results correspond to Fig. 5 of MP exploring Top-5 Acc. on EK55 and EG for
$\tau_{a}$ spanning from 0.25s to 2.0s.

Table 6: Results on the EK55 validation set regarding Top-5 Accuracy at different Anticipation Time-Steps averaged across the three official splits. All results in this table are obtained using only RGB modality. Model | Top-5 Accuracy % at different $\tau_{a}$ (s)
---|---
$2.0$ | $1.75$ | $1.5$ | $1.25$ | $1.0$ | $0.75$ | $0.5$ | $0.25$
FN [57] | 23.47 | 24.07 | 24.68 | 25.66 | 26.27 | 26.87 | 27.88 | 28.96
RL [58] | 25.95 | 26.49 | 27.15 | 28.48 | 29.61 | 30.81 | 31.86 | 32.84
RU-LSTM [1] | 25.44 | 26.89 | 28.32 | 29.42 | 30.83 | 32.00 | 33.31 | 34.47
SRL [5] | 25.82 | 27.21 | 28.52 | 29.81 | 31.68 | 33.11 | 34.75 | 36.89
S-GEAR (ours) | 28.57 | 29.95 | 31.34 | 32.87 | 34.48 | 34.92 | 36.49 | 37.49
Table 7: Results on the EGTEA Gaze+ validation set regarding Top-5 Accuracy at different Anticipation Time-Steps averaged across the three official splits. Model | Top-5 Accuracy % at different $\tau_{a}$ (s)
---|---
$2.0$ | $1.75$ | $1.5$ | $1.25$ | $1.0$ | $0.75$ | $0.5$ | $0.25$
FN [57] | 54.06 | 54.94 | 56.75 | 58.34 | 60.12 | 62.03 | 63.96 | 66.45
RL [58] | 55.18 | 56.31 | 58.22 | 60.35 | 62.56 | 64.65 | 67.35 | 70.42
RU-LSTM [1] | 56.82 | 59.13 | 61.42 | 63.53 | 66.40 | 68.41 | 71.84 | 74.25
SRL [5] | 59.69 | 61.79 | 64.93 | 66.45 | 70.67 | 73.49 | 78.02 | 82.61
HRO [36] | 60.12 | 62.32 | 65.53 | 67.18 | 71.46 | 74.05 | 79.24 | 83.92
S-GEAR (ours) | 58.85 | 61.52 | 64.99 | 70.56 | 71.93 | 73.71 | 79.26 | 85.41

###  B.3 Multimodal Ensemble

For EK100’s validation set, we evaluate three S-GEAR versions with varying
backbone combinations. Firstly, S-GEAR uses late fusion333Weighted Combination
of predictions from different models and modalities. of ViT-based RGB S-GEAR
(weight: 2.5) and FRCNN object features (weight: 0.5). S-GEAR-2B adds
ViT$\downarrow$-based RGB S-GEAR (weight: 1.5) to the previous fusion.
Finally, S-GEAR-4B combines all RGB S-GEAR variants (ViT, ViT↓, TSN, irCSN
with weights 2.5:1.5:1:1) and FRCNN object features (weight: 0.5). The same
weight combinations apply for the EK100 test set. On the EK55 validation set,
we late fuse ViT, irCSN, and TSN-based RGB S-GEAR (all weighted 1.5), along
with TSN flow features (weight: 1) and FRCNN object features (weight: 1). Note
that except for the ViT backbones, all other RGB and modality features are
pre-extracted as provided in [1, 12].

###  B.4 Fine-grained Multimodal Ablation

Table 8: Ablation of different RGB backbones and modalities on EK100 validation set. ViT$\downarrow$ represents ViT with an input size of 224$\times$224\. All backbone and modalities are combined through late fusion. RGB Backbones |  | Modalities |  | Verb | Noun | Action
---|---|---|---|---|---|---
ViT |  | RGB + Obj |  | 29.5 | 37.8 | 18.9
ViT |  | RGB + Flow |  | 30.2 | 35.4 | 18.4
ViT |  | RGB + Obj + Flow |  | 29.6 | 37.4 | 18.9
ViT + ViT$\downarrow$ |  | RGB + Obj |  | 30.5 | 38.4 | 19.6
ViT + ViT$\downarrow$ |  | RGB + Flow |  | 30.8 | 36.2 | 19.5
ViT + ViT$\downarrow$ |  | RGB + Obj + Flow |  | 30.2 | 36.9 | 19.6
ViT + TSN |  | RGB + Obj |  | 30.4 | 38.1 | 19.5
ViT + TSN |  | RGB + Flow |  | 30.8 | 36.6 | 19.2
ViT + TSN |  | RGB + Obj + Flow |  | 29.7 | 37.8 | 19.4
ViT + irCSN |  | RGB + Obj |  | 30.5 | 38.2 | 19.6
ViT + irCSN |  | RGB + Flow |  | 29.5 | 37.0 | 19.1
ViT + irCSN |  | RGB + Obj + Flow |  | 29.4 | 37.6 | 19.6
ViT + ViT$\downarrow$ \+ irCSN + TSN |  | RGB + Obj |  | 30.2 | 37.0 | 19.9
ViT + ViT$\downarrow$ \+ irCSN + TSN |  | RGB + Flow |  | 30.4 | 36.6 | 19.5
ViT + ViT$\downarrow$ \+ irCSN + TSN |  | RGB + Obj + Flow |  | 29.9 | 37.3 | 19.6

We present results from combining different backbones and modalities on the
EK100 validation set (see Table 8). We use late fusion with the following
backbone weights: ViT (weight: 2.5), ViT↓ (weight: 1.5), TSN (weight: 1.0),
irCSN (weight: 1.0), and equal weights (weight: 0.5) for Object and Flow
features. We report results for various combinations, ranging from models
using a single backbone to those combining up to four. We can notice that
combining RGB and object modalities provides the best overall results
regarding Top-5 Recall, which motivates our choice to exclude flow from the
final ensemble model.

###  B.5 Closer Look at Semanticity

In Sec. 4.7 of MP, we graphically show that the latent space topology defined
from S-GEAR’s learned prototypes is similar to the one defined from language
prototypes but not perfectly aligned due to the influence of visual cues. Such
resemblance in latent space geometry suggests that S-GEAR can semantically
reason regarding action associations and consider scene information
simultaneously. To dive deeper into this aspect, here we consider a group of
randomly selected reference actions – i.e., Pour Oil, Put Pan, Take Sponge,
Compress Sandwich, Cut Tomato, and Move Around Bacon – and analyze the
prototypes found in their proximity. Graphically, we show these comparisons in
Fig. 11. The grey boxes report the top-5 most similar actions for each
reference action (top bold string). Green actions (text on the left in each
grey box) represent the most similar actions in the language space. Black
actions (right side of the grey boxes) highlight alignment between vision and
language, whereas red actions state that there is a mismatch between the two.
Each action contains the cosine similarity between it and the reference one
(see values inside the brackets). Notice that actions are associated with the
same activities in both spaces regarding action classes and the magnitude of
their cosine similarity. Nevertheless, in 5/6 reported cases, we have at least
one divergence between the actions and the reference one. This phenomenon re-
emphasizes the divergence of visual prototypes influenced by visual cues and
the co-occurrences of actions in videos. Interestingly, even divergent actions
mostly have reasonable connections (i.e., they are likely to co-occur) with
the reference action. This underscores S-GEAR’s prototypes’ ability to keep
their semantic composure and account for the co-occurrence of actions
influenced by the observed action segments444Notice that the observed segments
come from the EG dataset. Notice that in the case of Take Sponge as a
reference action, the divergence between S-GEAR and language prototypes is the
action Squeeze Washing Liquid, which is probably the most likely action to co-
occur with Take Sponge in a kitchen scenario. Additionally, it shows that with
the proposed method, a perfect alignment between two spaces cannot be obtained
as long as the task is bounded to a given dataset. However, in such cases, as
we showed in Sec. 4.6 of MP (5th setup from Table 3), relying completely on
global semantics (i.e., language prototypes) is less beneficial for action
anticipation than merging it with the visual cues (S-GEAR) due to the
importance of motion and scene composition in suggesting possible future
actions.

Figure 11: (best viewed in color) Fine-grained semanticity comparison.