20

# A Hierarchical Graph-based Approach for Recognition and Description
Generation of Bimanual Actions in Videos

Fatemeh Ziaeetabar1,2, , Reza Safabakhsh2, Saeedeh Momtazi2,
Minija Tamosiunaite1,3, and Florentin Wörgötter1
1 Georg-Agust-Universität Göttingen, Göttingen, Germany
2 Amirkabir University of Technology, Tehran, Iran
3 Vytautas Magnus University, Kaunas, Lithuania  Corresponding author:
<EMAIL_ADDRESS>
###### Abstract

Nuanced understanding and the generation of detailed descriptive content for
(bimanual) manipulation actions in videos is important for disciplines such as
robotics, human-computer interaction, and video content analysis. This study
describes a novel method, integrating graph-based modeling with layered
hierarchical attention mechanisms, resulting in higher precision and better
comprehensiveness of video descriptions.

To achieve this, we encode, first, the spatio-temporal interdependencies
between objects and actions with scene graphs and we combine this, in a second
step, with a novel 3-level architecture creating a hierarchical attention
mechanism using Graph Attention Networks (GATs). The 3-level GAT architecture
allows recognizing local, but also global contextual elements. This way
several descriptions with different semantic complexity can be generated in
parallel for the same video clip, enhancing the discriminative accuracy of
action recognition and action description.

The performance of our approach is empirically tested using several 2D and 3D
datasets. By comparing our method to the state of the art we consistently
obtain better performance concerning accuracy, precision, and contextual
relevance when evaluating action recognition as well as description
generation. In a large set of ablation experiments we also assess the role of
the different components of our model. With our multi-level approach the
system obtains different semantic description depths, often observed in
descriptions made by different people, too. Furthermore, better insight into
bimanual hand-object interactions as achieved by our model may portend
advancements in the field of robotics, enabling the emulation of intricate
human actions with heightened precision.

Keywords: Bimanual Action Recognition, Manipulation Actions, Graph-based
Modeling, Hierarchical Attention Mechanisms, Video Action Analysis and Hand-
Object Interactions.

Statements and Declarations: The authors have no competing interests.

##  1 Introduction

Video understanding and description generation of manipulation and bimanual
actions play a crucial role in various applications, including robotics,
human-computer interaction, and video content analysis. The ability to
accurately recognize, interpret, and describe human actions in videos is
essential for enabling machines to understand and interact with the
surrounding environment. Additionally, generating informative and contextually
relevant video descriptions enhances user comprehension and facilitates
efficient retrieval and analysis of video content.

Manipulation actions, involving the interaction between humans and objects,
are fundamental to many tasks in robotics. They encompass a wide range of
actions such as picking, grasping, manipulating, and releasing objects.
Accurately recognizing and understanding manipulation actions is crucial for
developing intelligent robotic systems capable of performing complex tasks in
unstructured environments. By effectively analyzing manipulation actions,
robots can adapt their behavior and manipulate objects with precision and
dexterity, enhancing their utility in various domains, including
manufacturing, healthcare, and home assistance.

Bimanual actions, which involve coordinated movements of both hands, are
particularly important in robotics and human-robot collaboration scenarios.
Bimanual actions enable humans and robots to perform tasks that require
coordination, synchronization, and fine motor control. Examples include tasks
such as assembling objects, manipulating tools, and operating complex
machinery. By understanding and replicating bimanual actions, robots can
better collaborate with humans, improving productivity, safety, and efficiency
in shared workspaces.

Despite the significance of manipulation and bimanual actions, existing
approaches to video understanding and description generation often face
challenges in capturing the intricate relationships between objects and
actions, modeling fine-grained spatial and temporal dependencies, and
generating accurate and meaningful descriptions. These limitations hinder the
development of robust and precise systems for manipulation and bimanual action
recognition and description generation.

To address these challenges, we propose a novel methodology that combines
graph-based modeling and hierarchical attention mechanisms. Our approach
explicitly represents hand-object interactions as scene graphs, allowing for a
comprehensive understanding of manipulation and bimanual actions by capturing
the complex relationships between objects and actions. The utilization of
graph-based modeling enables the modeling of fine-grained spatial and temporal
dependencies, facilitating accurate recognition and interpretation of actions.

Furthermore, our methodology incorporates hierarchical attention mechanisms,
specifically Graph Attention Networks (GATs), which capture local and global
contextual information and enhance the precision and discriminative power of
action recognition. By attending to relevant objects and temporal features,
our model effectively aligns video content with corresponding textual
descriptions, resulting in accurate and contextually relevant video
descriptions.

To evaluate the effectiveness of our methodology, we conduct extensive
experiments on diverse 2D and 3D datasets, encompassing a wide range of
manipulation and bimanual actions. The experimental results demonstrate the
superiority of our approach compared to state-of-the-art methods in terms of
accuracy, precision, and contextual relevancy. Additionally, we introduce an
enriched bimanual action taxonomy that improves the understanding and
classification of actions, further enhancing the generation of accurate and
nuanced video descriptions.

Figure 1: Hierarchical framework for generating video descriptions at
different levels of detail. The figures include three video frames and
examples of their generated descriptions at three levels of detail: object-
level (Level 1), single hand action-level (Level 2), and bimanual action-level
(Level 3).

The contributions of this paper can be summarized as follows:

  1. 1.

We propose a novel methodology that integrates graph-based modeling and
hierarchical attention mechanisms to achieve highly accurate video
understanding and description generation of manipulation and bimanual actions.
To achieve this we use a novel “layered Graph Attentional Network (GAT)”
architecture.

  2. 2.

The layered architecture makes it possible to design a system that describes
actions with different levels of detail, allowing us to flexibly generate
concise summaries for quick overviews or elaborate and informative, paragraph-
like captions that provide in-depth insights into the video content. This
adaptability in capturing action complexity makes our model compatible with a
wide range of applications, from video browsing and retrieval to educational
content creation and video analysis in various domains. The concept of
descriptions at different levels is depicted in Fig. 1, which had been created
by our methods.

  3. 3.

We conduct extensive experiments on diverse 2D and 3D datasets to evaluate the
performance of our methodology. The evaluation includes metrics such as
accuracy, precision, and contextual relevance. The results demonstrate the
superiority of our approach compared to state-of-the-art methods, highlighting
its effectiveness in accurately understanding and describing manipulation and
bimanual actions.

The remainder of this paper is organized as follows: Section 2 provides a
review of related work in video understanding, description generation, and
manipulation and bimanual action recognition.

Section 3 presents the details of our proposed methodology. We describe the
key components of our approach, including graph-based modeling, hierarchical
attention mechanisms, and an enriched bimanual action taxonomy. We explain how
these components work together to achieve accurate video understanding and
description generation of manipulation and bimanual actions. Note that we have
moved many implementation aspects, notably all learning processes, into the
Supplementary Material, where algorithmic procedures are described in great
detail.

Section 4 is dedicated to the experimental results and analysis. We provide
detailed explanations of the datasets used in our evaluation, and present the
performance comparisons of our approach against state-of-the-art methods. In
addition, we show results from a set of ablation experiments to demonstrate
the contributions of the different algorithmic components. In this section, we
also discuss the results, highlight the strengths of our approach, and provide
insights into the potential applications of our methodology.

Finally, Section 5 concludes the paper by summarizing the contributions of our
work and discussing future research directions.

##  2 RELATED WORKS

The domain of video understanding, encompassing tasks such as action
recognition and video description generation, has experienced substantial
progress in recent years. However, understanding and describing complex
bimanual actions in videos remains a challenge due to the intricacies involved
in capturing the relationships between objects and actions. This section
provides a review of the literature in key areas of relevance to our study.

###  2.1 Action Recognition

The field of action recognition as a pivotal task in video understanding, has
witnessed significant advancements in recent years, driven by the rise of deep
learning techniques and the availability of large-scale video datasets. While
early approaches heavily relied on handcrafted features and traditional
machine learning algorithms (e.g. [1][2][3] and [4]), the emergence of
Convolutional Neural Networks (CNNs) revolutionized the field.

CNN-based architectures, such as Two-Stream Networks [5][6] and 3D
Convolutional Networks [7][8], have demonstrated remarkable performance in
recognizing various human actions, including manipulation actions [9]. These
models leverage spatiotemporal information in videos to capture the dynamics
and visual patterns associated with different actions. Notably, recent trends
have focused on incorporating attention mechanisms to attend to salient
spatiotemporal regions, enabling more accurate recognition of fine-grained
actions [10][11].

Moreover, the recognition of manipulation actions has gained specific
attention due to its relevance in robotics and human-robot interaction
[12][13]. Researchers have explored specialized architectures that explicitly
model hand-object interactions, using both, appearance and motion, cues. These
approaches often combine deep learning techniques with hand-crafted features,
such as hand pose estimation or object tracking, to improve the recognition
accuracy of manipulation actions [14][15].

However, a notable gap in the literature is the limited focus on bimanual
actions, which involve coordinated movements and interactions between both
hands. Bimanual actions are prevalent in various domains, including
manufacturing, cooking, and assembly tasks [16]. Understanding and recognizing
bimanual actions require capturing the complex dependencies and coordination
between the hands, which present unique challenges compared to unimanual
actions.

While there have been studies addressing bimanual actions in robotics (e.g.
[17][18] and [19]), the application of deep learning-based action recognition
techniques to bimanual actions is relatively unexplored. This provides an
exciting opportunity for future research to develop specialized architectures
and datasets that specifically target bimanual action recognition. Such
advancements would greatly benefit applications in robotics, where
understanding bimanual interactions is crucial for human-robot collaboration
and task execution [20][21].

###  2.2 Scene Graphs and Graph-based Modeling

In the realm of video understanding, scene graphs and graph-based modeling
have emerged as powerful techniques for capturing the intricate relationships
between objects and actions (e.g. [22][23] and [24]).

One leading trend in scene graph generation is the integration of deep
learning methodologies. Traditional approaches heavily relied on handcrafted
features and rule-based heuristics, which limited their scalability and
adaptability to diverse video datasets. With the advent of deep learning,
researchers have used convolutional neural networks (CNNs) and recurrent
neural networks (RNNs) to automatically extract rich visual features and learn
complex representations from raw video data [25][26]. These deep learning-
based techniques have demonstrated remarkable performance improvements in
scene graph generation by effectively capturing the semantic relationships
between objects [27][28].

A growing trend in scene graph generation involves incorporating contextual
information to enhance the quality and relevance of generated graphs. Context
plays a pivotal role in understanding visual scenes, as the relationships
between objects are influenced by their surrounding environment. Recent
methods have explored the integration of contextual cues, such as global scene
context [29], spatial relationships [30], and temporal dependencies [31], to
improve the accuracy and contextual understanding of scene graphs. Attention
mechanisms and graph propagation techniques have been employed to selectively
attend to relevant regions or objects in the scene and propagate information
across the graph, capturing higher-order relationships and contextual
dependencies [32][33].

Graph-based modeling techniques have also witnessed significant advancements
in various aspects of video understanding. Notably, the adoption of graph
neural networks (GNNs) has revolutionized the field by extending convolutional
operations to graph structures. GNNs enable information propagation and
aggregation across nodes in the graph, allowing for the modeling of complex
relationships and capturing both spatial and temporal dependencies. By
leveraging GNNs in scene graphs, researchers have achieved state-of-the-art
results in critical tasks such as action recognition, video captioning, and
object detection [34][35] and [36]. The application of GNNs facilitates end-
to-end learning and enables comprehensive modeling of visual scenes,
contributing to improved video understanding capabilities.

Another noteworthy trend in graph-based modeling is the integration of multi-
modal information. Combining visual data with complementary modalities such as
textual descriptions or sensor data enhances the understanding and
interpretation of video content. For instance, textual information can provide
valuable semantic cues that complement the visual scene graph, leading to
improved performance in tasks such as video captioning and activity
recognition [37][38]. By using the synergistic fusion of multiple modalities,
researchers aim to achieve a more holistic and comprehensive understanding of
video content [39].

In conclusion, the current trends and methods in scene graph generation and
graph-based modeling for video understanding underscore the integration of
deep learning methodologies, the incorporation of contextual information, the
utilization of graph neural networks, the mitigation of scalability
challenges, and the integration of multi-modal data. These advancements have
propelled the field forward, enabling more accurate and contextually aware
video understanding systems. However, there remain challenges, including
handling large-scale scene graphs, improving interpretability, and advancing
graph reasoning capabilities, which warrant further research and exploration
to unlock the full potential of graph-based modeling in video understanding.

###  2.3 Video Description Generation

Video description generation is a task within the domain of video
understanding that aims to automatically generate textual descriptions or
captions for video content. This subsection provides a concise overview of the
historical developments and the current state of the art in video description
generation.

####  2.3.1 Historical Developments

The task of generating textual descriptions for videos has evolved
significantly over the years. Early approaches focused on rule-based systems,
where predefined templates and handcrafted rules were used to generate
descriptions based on visual cues and temporal analysis of the video content
(e.g. [40][41][42] and [43]). These methods, although limited in their
descriptive capabilities, laid the foundation for subsequent research in the
field.

With the advent of deep learning and the availability of large-scale video
datasets, the field witnessed a shift towards data-driven approaches.
Researchers started to employ convolutional neural networks (CNNs) for visual
feature extraction and recurrent neural networks (RNNs) for generating
sequential descriptions. This approach enabled the modeling of temporal
dependencies and improved the descriptive quality of the generated captions.

####  2.3.2 Current State of the Art

In recent years, significant advancements have been made in video description
generation, driven by the combination of deep learning techniques [44],
multimodal fusion [45], and attention mechanisms [46]. State-of-the-art models
employ deep neural networks to extract visual features from video frames and
textual features from accompanying transcripts or subtitles. These features
are then fused and processed through sophisticated architectures to generate
coherent and contextually relevant descriptions (e.g. [47][48] and [49]).

One notable approach is the Transformer model, originally proposed for natural
language processing tasks. The Transformer’s self-attention mechanism allows
it to capture long-range dependencies and effectively model the relationships
between video frames and their corresponding textual descriptions. This has
led to remarkable improvements in generating accurate and contextually rich
captions for videos ([50][51][52] and [53]).

Another area of active research is the incorporation of external knowledge
sources, such as pre-trained language models and commonsense reasoning, to
enhance the descriptive capabilities of video captioning systems. These
knowledge-driven approaches aim to generate captions that are not only
faithful to the visual content but also demonstrate a deeper understanding of
the context and semantics [54][55].

Furthermore, researchers are exploring techniques to address challenges in
video description generation, such as handling complex scenes, temporal
dynamics, and generating captions for videos without accompanying textual
annotations. This involves developing novel architectures, relying on
unsupervised learning, and integrating multimodal cues, including audio and
motion, to provide more comprehensive and informative descriptions
[56][57][58].

###  2.4 Gap Analysis and Our Proposed Work

In this section, we would like to point out key gaps in the state of the art
that our research aims to address.

A significant gap within action recognition literature lies in the limited
exploration of bimanual actions, which involve detailed interactions between
both hands. Despite the complexity and common occurrence of such actions in
various sectors, they have been largely disregarded in deep learning-based
action recognition. Our research proposes an innovative methodology for
recognizing and understanding bimanual actions.

In terms of Scene Graphs and Graph-based Modeling, the handling of large-scale
scene graphs and the furthering of graph based reasoning capabilities are
identified as challenges that demand additional research. Despite the progress
in the field, scalability and interpretability of scene graphs still remain as
hurdles. Our research contributes to this area, introducing methods to manage
large-scale scene graphs effectively, and providing enhanced reasoning
techniques to improve the interpretability of these graphs.

For video description generation, a comprehensive approach to describe complex
video content is another hurdle not yet surmounted by the existing body of
work. Challenges lie in handling complex scenes, temporal dynamics, and in
generating descriptions for videos that lack textual annotations. Our research
seeks to make a contribution in this area, adopting a comprehensive approach
that utilises contextual cues and innovative architectural designs to generate
relevant descriptions for complex video content.

Figure 2: The flow diagram of BiGNN framework

##  3 METHODS

In this section, we present the Bimanual Graph Neural Network (BiGNN), a
framework for recognizing bimanual actions and generating natural language
descriptions of these actions at multiple levels of detail using graph neural
networks. BiGNN comprises several interconnected components, each serving a
unique role in the overall action recognition and description process. Note
the most of the finer algorithmic details are found in the Supplementary
Material including all learning aspects.

As shown in Figure 2, the first step of BiGNN involves recognizing objects and
spatial relations to create a scene graph representation of the bimanual
action. This scene graph is then processed through an embedding layer, and the
resulting features are utilized by a modified Graph Attention Network (GAT)
architecture, designed to extract important features while taking spatial
information into account. The extracted features are then passed into a
temporal convolutional network (TCN) to identify the temporal dependencies
between the video frames. The roles of the different hands are also extracted
from the video frames to further enhance the recognition of bimanual actions.
Finally, the hierarchical processing of the BiGNN framework comes into play,
where the action is recognized at different levels of detail. As will be shown
below, this way BiGNN generates human-like sentences describing the action at
different levels of detail, from high-level overviews to more detailed
descriptions of the individual objects and their interactions, which enables
understanding of the bimanual action at different level of description
granularity. In the following the different components will be described.

###  3.1 Pre-processing and Scene Graph Creation

The different pre-processing steps are crucial components of the Bimanual
Graph Neural Network (BiGNN) framework, responsible for detecting and
localizing objects in the video frames, and recognizing their spatial
relations.

Pre-processing begins with object detection, utilizing a pre-trained deep
learning model to detect and localize objects within each video frame.
Following object detection, the spatial relations between objects are
recognized and represented as a graph. Each object is represented as a node in
the graph, and the edges between nodes represent the spatial relations between
objects, such as proximity or relative position.

In the following sections, we will discuss the details of pre-processing,
including object detection, spatial relation recognition, and scene graph
creation.

####  3.1.1 Object Detection

The initial stage of our pre-processing pipeline involves the 2D analysis of
RGB images to detect objects using the You Only Look Once (YOLO) algorithm
[59]. YOLO has been trained on the labeled objects within our dataset,
enabling it to accurately identify and localize objects of interest.
Additionally, we utilize OpenPose [60] to detect human hands, which provides
key points representing hand positions. From these key points, we calculate 2D
bounding boxes for each hand. This stage produces a list of 2D bounding boxes,
encompassing both the objects detected by YOLO and the hands detected by
OpenPose. It is worth noting that the hands are treated as any other object in
the subsequent stages of our pipeline. Note that BiGNN is not bound to using
these modules (YOLO, OpenPose) but they fully suffice for our purpose.

####  3.1.2 Spatial Relation Recognition

The second stage of this step involves 3D pre-processing of the data, which is
specifically performed on 3D datasets (for 2D datasets see end of this
subsection). During this stage, the information obtained from the first step
is used alongside the point clouds generated from the depth images of the 3D
dataset to acquire precise 3D bounding boxes for the objects. If the pre-
segmented information of the objects is already available in the dataset, we
generate their corresponding 3D models by creating the 3D surrounded convex
hulls. In cases where the dataset lacks pre-segmented objects, their
corresponding Axis Aligned Bounding Box (AABB) is utilized to model the
objects in 3D space. When depth images are unavailable in the dataset, the
approach proposed in [61] is employed to add a third dimension to the 2D
images, allowing for the generation of 3D bounding boxes. Finally, the spatial
relations between each pair of objects are computed using the well-established
methods from [62][63][64]. The methodology of computing these spatial
relations has been explained in [64].

The list of our defined spatial relations SR includes static spatial relations
($SSR$) pertaining to object positions, comprising the thirteen items. In the
original framework [63] also dynamic spatial relations pertaining to object
movement ($DSR$) had been defined but are not used here. SSRs are given as:

$\begin{multlined}\text{SSR}=\\{\text{contact, above, below, around, left,
right, front, behind,}\\\ \text{contain, partial contain, within, partial
within, cross\\}}\\\ \end{multlined}\text{SSR}=\\{\text{contact, above, below,
around, left, right, front, behind,}\\\ \text{contain, partial contain,
within, partial within, cross\\}}\\\ $ (1)

####  3.1.3 Spatio-Temporal Scene Graph Construction

In our approach, each frame of the video contains a scene graph, which is
represented as an undirected graph
$\mathcal{G}=(\mathcal{V},[\mathcal{E}_{1},\mathcal{E}_{2},\mathcal{E}_{3}])$.
The set $\mathcal{V}$ consists of node attributes $\tilde{h}$, each of which
represents an object in the scene, $\mathcal{E}$ are edges between those nodes
(see below).

Node attributes $\tilde{h}$ for node $i$ are given as a mixed tuple
$\tilde{h}_{i}=\\{n_{i},\eta(p_{i}),\eta(s_{i})\\}$, where $n$ is a string
(label of the object), and $p$ as well as $s$ are position and speed, encoded
by numerical values. The operator $\eta$ refers to a normalization function
used to keep values bounded. We used, as also commonly done:
$\eta(y)=(y-\text{mean}(y))/\text{std}(y)$. Below we will describe how to
transform the mixed tuple into a fully numerical feature vector.

The set $[\mathcal{E}_{1},\mathcal{E}_{2},\mathcal{E}_{3}]$ consists of
undirected edges, where $\mathcal{E}_{1}$ represents spatial relation
attributes $\tilde{e}$ between two objects in one given video frame,
$\mathcal{E}_{2}$ refers to temporal-relation attributes $\tilde{v}$ between
objects from the last ($t-1$) to the current ($t$) movie frame, and
$\mathcal{E}_{3}$ represents actions $\tilde{a}$ that can be performed given
an object-pair in the scene. Note that temporal relations will eventually be
defined by object-object distance changes and only be used much later (see
Section on TCNs, below). Until then we will only deal with spatial- and
action-attributes.

Hence, for the spatial attributes $\tilde{e}$ of an edge of type
$\mathcal{E}_{1}$ between nodes $i$ and $j$, we define accordingly the tuple:
$\tilde{e}_{i,j}=\\{y_{i,j},\eta(p_{i}),\eta(p_{j})\\}$. Here the string
variable $y$ refers to the spatial relation type (see Eq. 1) encoded by the
edge, and, as before, $p$ refers to the position of an object (=node). String
entries $y$ are computed for the spatial edges from the static spatial
relations (SSR, see Eq. 1 and [63]).

The action-edge attributes $\tilde{a}$ of an edge of type $\mathcal{E}_{3}$
between nodes $i$ and $j$ are defined for each possible action $k$. Hence,
each action index $k$ corresponds to a specific action category in our
dataset(s). These edges, thus, represent the actions that can be performed
between pairs of objects. For example, if one object is “knife” and the other
is “fruit,” the object-action edge could correspond to the action of
“cutting.” Each action attribute $\tilde{a}_{i,j,k}$ is, thus, represented by
a tuple $\\{s_{i,j,k},\eta(p_{i}),\eta(p_{j})\\}$, where: $s_{i,j,k}$
represents the specific action’s importance score between nodes $i$ and $j$
for the given action index $k$. This importance score quantifies the relevance
of the action between the two nodes and is learned using the GAT (see below).
It captures how significant a particular action is in the context of the
interaction between nodes $i$ and $j$. The variables $\eta(p_{i})$ and
$\eta(p_{j})$ represent, as before, the spatial positions of nodes $i$ and
$j$, respectively. These attributes combine the action-related information
(importance scores), with the spatial context of the nodes, enabling the graph
attention mechanism to capture both action-specific relationships and spatial
dependencies during the graph attention computation. The model can now handle
multiple possible actions between objects, capturing the diversity and
complexity of bimanual actions in the scene graph representation.

A sample of three consecutive video frames in a bimanual action manipulation
and their corresponding scene graphs is shown in Figures 3 and 4. In this
figure blue edges represent spatial relations (e.g. above, left…) between each
pair of objects in a frame that their distance is less than a threshold and
red edges demonstrate temporal relations between those through continuous
frames. For example, in the second frame, the left hand lifts the bottle from
the table and holds it above the bowl which means the bottle is moving apart
from the table (their corresponding SSR changes from Top/Bottom to
Above/Below) and is getting close to the bowl (their corresponding SSR changes
from Left/Right to Above/Below). To avoid cluttering the figure, only a subset
of edges are shown.

Figure 3: An example of a scene in the KIT bimanual action dataset [65] and
its related scene graphs at three consecutive frames. In this figure blue and
red lines indicate spatial and temporal edges respectively.  Figure 4: Object-
action edges related to the scene graph in Figure 3. The purple lines show
object-action edges. In the depicted dataset, the available actions are
labeled as: approach 1, retreat 2, lift 3, place 4, hold 5, stir 6, pour 7,
cut 8, drink 9, wipe 10, hammer 11, saw 12, and screw 13. The numbers written
above the purple edges represent the possible actions that can occur between
those two node objects.

###  3.2 Node and Edge Feature Encoding

After the creation of the scene graph in Section 3.1, we proceed to transform
the node as well as spatial- and action-edge attributes into a numerical
encoding. For this, we utilize the Word2Vec algorithm [66]. In doing so we
obtain now fully numerical feature vectors $\vec{h}_{i}$ for nodes and
$\vec{e}_{i,j}$ as well as $\vec{a}_{i,j,k}$ for the edges, resulting in
$\mathbf{\hat{H},~{}\mathbf{\hat{E}}}$ and $\mathbf{\hat{A}}$. Note that, for
simplicity, we will use the word “matrix” for all such entities even if their
dimension is larger than 2 (where “tensor” should be used instead).

The preliminary feature matrices are then augmented using linear layers
represented by weight matrices $\mathbf{L}_{H}$, $\mathbf{L}_{E}$, and
$\mathbf{L}_{A}$, respectively. Please see Supplementary Material for the
procedures for computing $\mathbf{L}$. Such linear layers allow capturing more
complex patterns and relationships in the data. They operate on the
preliminary feature matrices $\mathbf{\hat{H}}$, $\mathbf{\hat{E}}$, and
$\mathbf{\hat{A}}$, transforming them into the final feature matrices
$\mathbf{H}$, $\mathbf{E}$, and $\mathbf{A}$, respectively. The linear
transformation is mathematically defined as follows:

$\displaystyle\mathbf{H}$ $\displaystyle=$
$\displaystyle\mathbf{L}_{H}\otimes\mathbf{\hat{H}}$ (2)
$\displaystyle\mathbf{E}$ $\displaystyle=$
$\displaystyle\mathbf{L}_{E}\otimes\mathbf{\hat{E}}$ (3)
$\displaystyle\mathbf{A}$ $\displaystyle=$
$\displaystyle\mathbf{L}_{A}\otimes\mathbf{\hat{A}}$ (4)

In this process, $\otimes$ represents the Hadamard product. These final
feature matrices $\mathbf{H}$, $\mathbf{E}$, and $\mathbf{A}$ now contain
encoded information about the nodes, spatial and temporal edges, and object-
action edges, which are then used in subsequent stages of our method for
action description.

Figure 5 illustrates an example scene with three objects: an apple, a knife,
and a cutting board. The apple and the knife are placed on the cutting board,
and the knife is positioned around the apple. The corresponding scene graph
and the embedded node and edge feature matrices.

Figure 5: Illustration of the node and edge feature encoding process in a
scene graph, where object attributes and spatial relations are represented as
continuous vectors using an embedding layer. $x$, $y$, $z$ are the Cartesian
coordinates of an object, and $v$ represents its velocity.

###  3.3 Graph Attention Network (GAT): general definition

We use GATs in two sub-processes, below. Hence, we will define a GAT first in
general terms and eventually augment this definition in the respective sub-
sections as needed.

Let $\vec{h}_{i}^{(l)}$ be a feature vector of node $i$ at the $l$-th layer of
a GAT. The GAT model computes new features for each node by aggregating
features from its neighbors weighted by attention scores $\alpha$:

$\displaystyle\vec{z}_{i}^{(l)}$ $\displaystyle=$
$\displaystyle\mathbf{W}^{(l)}\vec{h}_{i}^{(l)},$ (5) $\displaystyle
f_{ij}^{(l)}$ $\displaystyle=$
$\displaystyle\text{LeakyReLU}\left(\vec{d}^{(l)}\left[\vec{z}_{i}^{(l)},\vec{z}_{j}^{(l)}\right]^{T}\right),$
(6) $\displaystyle\alpha_{ij}^{(l)}$ $\displaystyle=$
$\displaystyle\frac{\exp(f_{ij}^{(l)})}{\sum_{k\in N_{i}}\exp(f_{ik}^{(l)})},$
(7) $\displaystyle\vec{h}_{i}^{(l+1)}$ $\displaystyle=$
$\displaystyle\sigma\left(\sum_{j\in
N_{i}}\alpha_{ij}^{(l)}\vec{z}_{j}^{(l)}\right),$ (8)

where $\mathbf{W}^{(l)}$ and $\vec{d}^{(l)}$ are learnable (see Supplementary
Materials), LeakyReLU is the leaky rectified linear unit activation function,
$\sigma$ is the activation function (e.g., sigmoid or softmax), and $N_{i}$ is
the set of neighbors of node $i$. The notation $[\dots,\dots]$ means
concatenation.

Each attention score $\alpha_{i,j}^{(l)}$ measures the importance of node $j$
to node $i$ at layer $l$.

In total, we have implemented three layers. Hence, our GAT-process ends at
$\vec{h}^{(3)}$. We call this feature vector “final” and abbreviate it with
$\vec{h}^{f}$.

Note that in section 3.3.1 we augment this general GAT process by replacing
node features $h$ with action featuers $a$ in Eq. 5. In section 3.3.2 we
modify Eq. 6 in several ways defining three more GAT processes.

####  3.3.1 Preprocessing action features $a$ using GAT

Action features were defined above as $\vec{a}_{i,j,k}$, but — for simplicity
— we are dropping index $k$ at the action-edges and emphasize that all further
processing is performed for each action $k$ in the same way.

We use a first GAT process to perform aggregation of $\vec{a}^{(l)}_{i,j}$
across the GAT layers. By aggregating information from each layer, the final
feature vector $\vec{a}_{i,j}$ encodes more reliably how the edge between
nodes $i$ and $j$ represents a specific action performed by those objects.

Initialization of $\vec{a}_{i,j}^{(0)}$:
As a common approach, the initial feature vector $\vec{a}_{i,j}^{(0)}$ is
randomly initialized using a Gaussian distribution with mean 0 and a small
standard deviation. This initialization ensures that the initial features are
centered around zero and have small magnitudes, which aids in the initial
stages of learning. The initialization process can be expressed as:
$\vec{a}_{i,j}^{(0)}\sim\mathcal{N}(0,\sigma^{2})$, where $\sigma$ is a small
positive value representing the standard deviation of the Gaussian
distribution.

By using this approach, the initial edge features are set to random values,
which allows the model to adapt and update these features based on the
patterns and information present in the data during training. Although the
exact values of the initial edge features are not critical, this
initialization provides a reasonable starting point for the learning process
and for this we replace $\vec{h}_{i}^{(0)}$ (Eq. 5) with
$\vec{a}_{i,j}^{(0)}$.

Aggregation across layers:
After the GAT-learning is completed we combine the information from all layers
and obtain a single representation for each action edge $\vec{a}_{i,j}$ by an
aggregation step. The aggregation of $\vec{a}_{i,j}$ across layers is
typically performed using a pooling operation, such as summation or mean
pooling. We use summation:

$\vec{a}_{i,j}=\frac{1}{L}\sum_{l=1}^{L}\vec{a}_{i,j}^{(l)}$ (9)

where $L$ is the total number of GAT layers, where we use $L=3$.

####  3.3.2 Hierarchical GAT Levels

The above definitions and the preprocessing of the actions allows now to
introduce a key feature of our approach. Different from conventional GATs we
are implementing a new aspect by creating a 3-level hierarchical GAT process,
which allows us to recognize actions at multiple levels of detail.
Specifically, we define three levels of GAT: 1) object-level, 2) single hand
action-level, and 3) bimanual action-level annotated with superscripts
$.^{(1)},.^{(2)},.^{(3)}$ in the following.

At the object-level, our GAT learns relationships between individual objects
in the scene graph, such as which objects are near each other or which objects
are frequently used together. For example, the GAT might learn that the knife
is frequently used to cut fruit, or that the bowl is frequently used to hold
chopped fruit.

At the single hand action-level, we use the output of the object-level GAT to
identify which objects are being manipulated by each hand. We can then use
this information to recognize the actions of each individual hand. For
example, the GAT might recognize that the left hand is holding a knife and the
right hand is holding a piece of fruit, and that the action being performed is
“slice”.

At the bimanual action-level, we use the output of the action-level GAT to
identify which actions involve multiple hands working together. We can then
use the output of the object-level GAT to identify which objects are being
manipulated by each hand in the bimanual action. For example, the GAT might
recognize that both hands are being used to mix the chopped fruit together.

To formalize this we create three levels, where a higher level uses the output
of the previous level to identify and analyze more complex relationships
between nodes in the scene graph. Remember that, in the following,
superscripts refer to GAT levels and not (as above in Eqs. 5-8) to the layers
of the GAT.

We define $\mathbf{H}^{(0)}$ and $\mathbf{E}^{(0)}$ as embedded original node
and edge feature matrices, then:

$\mathbf{G}^{(1)}=\text{GAT}(\mathbf{H}^{(0)},\mathbf{E}^{(0)})$ (10)

This equation represents the first GAT level (object level), which is applied
to the initial node feature matrix $\mathbf{H}^{(0)}$ and the spatial edge
feature matrix $\mathbf{E}^{(0)}$. To get this we expand the right side of Eq.
6 with spatial edge and action-edge features such that it reads now:

$f^{(1)}_{i,j}=\text{LeakyReLU}\left(\vec{d}\left[\vec{z}_{i},\vec{z}_{j},\mathbf{W}_{e}\vec{e}_{i,j},\mathbf{W}_{a}\vec{a}_{i,j}\right]^{T}\right),$
(11)

where indices $e$ and $a$ depict that the (learnable) matrices $\mathbf{W}$
correspond to edges $e$ and $a$. The notation $[\dots,\dots]$ represents
concatenation.

In this step, the GAT uses the original node and edge features to compute the
attention scores $f_{i,j}$ (as described in Eq. 6) between each pair of nodes
$i$ and $j$. The output of this step is the updated node feature matrix
$\mathbf{G}^{(1)}$, which is obtained according to the node and edge
information at Level 1 (object level).

As a result all feature matrices above level 0 now contain mixed node and edge
information. Next we define:

$\mathbf{G}^{(2)}=\text{GAT}(\mathbf{G}^{(1)},\mathbf{B}^{(2)},\mathbf{F}^{(1)})$
(12)

We explain the new terms $\mathbf{G}^{(1)},\textbf{B}^{(2)}$, and
$\mathbf{F}^{(1)}$ in order of appearance. They are used to define
$f_{ij}^{(2)}$ (compare to Eq. 6) as:

$f_{ij}^{(2)}=\text{LeakyReLU}\left(\vec{d}\left[\mathbf{G}^{(1)}\vec{h}_{i}^{(0)},\mathbf{G}^{(1)}\vec{h}_{j}^{(0)},\mathbf{W}_{e}\vec{e}_{i,j},\mathbf{W}_{a}\tilde{\vec{a}}_{i,j},\mathbf{F}^{(1)}\vec{\gamma}\right]^{T}\right)$
(13)

On $\mathbf{G}^{(1)}$: In the equation for $f_{ij}^{(2)}$, $\mathbf{G}^{(1)}$
refines the initial node embeddings. It transforms the initial features using
learned weights and captures the collaborative relationships between nodes at
the first GAT layer. Therefore, the refined embeddings,
$\mathbf{G}^{(1)}\vec{h}_{i}^{(0)}$ and $\mathbf{G}^{(1)}\vec{h}_{j}^{(0)}$,
encode complex node interactions based on both intrinsic node properties and
their relationships with neighboring nodes.

On $\mathbf{B}^{(2)}$: The attention matrix $\mathbf{B}^{(2)}$ is used to
weigh $\vec{a}_{i,j}$ to obtain $\tilde{\vec{a}}_{i,j}$ in Eq. 13. For this we
define the elements of the matrix $\mathbf{B}^{(2)}$ as $\beta_{i,j}$ given by
the following softmax operation:

$\beta_{i,j}=\frac{\exp((\mathbf{W}^{(2)}\cdot\mathbf{G}^{(1)}_{j})\cdot(\mathbf{U}^{(2)}\cdot\mathbf{G}^{(1)}_{i})^{T})}{\sum_{k\in\mathcal{N}(i)}\exp((\mathbf{W}^{(2)}\cdot\mathbf{G}^{(1)}_{k})\cdot(\mathbf{U}^{(2)}\cdot\mathbf{G}^{(1)}_{i})^{T})}$
(14)

In Equation 14, $\mathbf{U}^{(2)}$ is a learnable weight matrix specific to
the second GAT layer. It is used to transform the node features of neighboring
nodes $\mathbf{G}^{(1)}_{j}$ before calculating the attention scores.
$\mathbf{W}^{(2)}$ is a transformation matrix for node features at Layer 2,
refining the representations based on the outputs of Layer 1. See
Supplementary Material for details.

In this equation, $\mathcal{N}(i)$ denotes the set of neighbors of node $i$.
The outcome, $\beta_{i,j}$, provides another normalized attention score that
captures the significance of the information from node $j$ for node $i$ but
here concerning action edges which is computed by element-wise multiplication
given as:

$\tilde{\vec{a}}_{i,j}=\beta_{i,j}\vec{a}_{i,j}$ (15)

$\mathbf{B}^{(2)}$ is called attention matrix at the single-hand action-level.
It determines the importance of each action edge for recognizing single-hand
actions.

On $\mathbf{F}^{(1)}$: The feature matrix $\mathbf{F}^{(1)}$ originates from
the object-level-1 GAT layer, which operates on the embedded node feature
matrix $\mathbf{H}^{(0)}$. It is column-wise defined as:

$\mathbf{F}_{c}^{(1)}=\left[\alpha_{ij}^{(1)}\vec{h}_{i}^{(0)}\right]_{c}$
(16)

where index $c$ stands for column $c$ of the matrix. Thus, $\mathbf{F}^{(1)}$
captures the spatial relationships between objects in the scene graph, and it
is obtained by weighting the embedded node feature matrix $\mathbf{H}^{(0)}$
with attention weights $\alpha$ from $\mathbf{G}^{(1)}$ calculated based on
the relationships between nodes at the object-level GAT layer.

To correctly define Eq. 13, $\mathbf{F}^{(1)}$ is multiplied with
$\vec{\gamma}$. The vector $\vec{\gamma}$ is a learnable parameter and serves
as a weighting mechanism that allows us to control the influence of the matrix
$F^{(1)}$ on the attention score calculation. It determines the relative
importance/contribution of each column of $F^{(1)}$ to the attention
calculation. This introduces a fine-grained level of control over how the
matrix $F^{(1)}$ impacts the attention mechanism.

Accordingly the next layer is defined by:

$\mathbf{G}^{(3)}=GAT(\mathbf{G}^{(2)},\mathbf{B}^{(3)},\mathbf{F}^{(2)}),$
(17)

with the same definitions as above. GAT layer 3 then captures bi-manual
manipulations.

Hence, the 3-level GAT process produces as final outputs
$h^{(1),f},h^{(2),f},h^{(3),f}$.

This hierarchical GAT framework enables us, thus, to recognize actions at
different levels of detail. By recognizing relationships between objects,
individual hands, and both hands, we can, thus, describe actions with greater
specificity and accuracy.

Figure 6 depicts two instances of this scene along with their corresponding
scene graphs and the respective output of each GAT layer involved.

Figure 6: Illustration of the use of GAT layers with edge features on two
scenes of a scenario (a) chopping a cucumber and (b) mixing the chopped pieces
of cucumber in a bowl

###  3.4 TCN-Based Spatio-Temporal Processing

In this section, we extend the framework established in Section 3.3, by
introducing Temporal Convolutional Networks (TCNs) along with a sliding window
procedure to allow addressing “time”, too. This integration empowers our model
to capture both short-term and long-term dependencies in the temporal
evolution of embedded feature vectors, enabling spatio-temporal reasoning.

To achieve this, we had above introduced three levels of GAT layers: object-
level, single hand action-level, and bimanual action-level. Each level is now
paired with a corresponding TCN layer to enable temporal reasoning.
Specifically, the output of the object-level GAT layer, denoted as
$\mathbf{G}^{(1)}$, serves as the input to the TCN layer at the object-level.
This TCN layer processes the temporal evolution of embedded node features,
capturing short-term and long-term dependencies within the individual objects’
relationships. Similarly, the output of the single hand action-level GAT
layer, denoted as $\mathbf{G}^{(2)}$, is fed into the TCN layer at the single
hand action-level, which focuses on recognizing actions performed by each hand
independently. Finally, the output of the bimanual action-level GAT layer,
denoted as $\mathbf{G}^{(3)}$, is processed by the TCN layer at the bimanual
action-level, allowing the model to recognize actions that involve both hands
working together. This hierarchical combination of GAT and TCN layers enables
our framework to perform comprehensive spatio-temporal reasoning and produce
detailed action descriptions.

Temporal Convolutional Networks (TCNs) are designed for processing sequential
data. To achieve this, TCNs use convolutional filters that are shared across
the input sequence, enabling information to propagate through the network
without degradation. This aspect is combined with a sliding window approach
where the TCN applies a fixed-duration kernel, with temporal kernel size
denoted by $K$, to the input sequence. At each time step, the convolutional
filter slides across the input sequence, capturing information within a local
temporal context of size $K$ video frames. For our different data sets, we use
different kernel sizes adapted to the characteristics of the respective data
(for values of $K$ see Supplementary Material).

To enhance performance, furthermore we are stacking multiple TCN layers on top
of each other. This way, the model can capture increasingly complex temporal
dependencies. Each TCN layer extracts higher-level features from the output of
the previous layer, allowing the network to recognize longer-term patterns in
the data.

####  3.4.1 Definition of the TCN Inputs

The TCN receives input $X=\\{\vec{x}_{1},\vec{x}_{2},\dots,\vec{x}_{T}\\}$
defined in the following.

Pooling spatial information: For each node $i$, the enhanced final feature
vector $h^{f}$, resulting from the GAT layers, is combined with the embeddings
of its associated spatial and action edges. This can be represented as:

$\text{Agg}(\vec{h}^{f}_{i})=\max\left(\vec{h}_{i}^{f},\max_{j}(\mathbf{E}_{i,j}),\max_{j}(\mathbf{A}_{i,j})\right)$
(18)

We use maxpooling here because it captures dominant features. This is
particularly useful in action recognition because it brings the most dominant
and “actionable” information to the forefront. In dynamic scenes, where
multiple activities might be taking place, it’s the dominant or significant
(inter-)actions that often define the overall action.

After node-level aggregation, the next step is to aggregate across all nodes
to obtain a comprehensive representation for the entire frame at time $t$.

$\vec{x}^{*}_{t}=\max_{i\in\\{1,2,...,n\\}}\text{Agg}(\vec{h}^{f}_{i})$ (19)

This hierarchical method using max-pooling ensures that the final
representation $\vec{x}^{*}_{t}$ for each frame captures the most dominant
features from both nodes and edges, but temporal information is still missing
and will be added by the following steps.

Incorporating temporal information: Temporal information between two objects
$i$ and $j$ over consecutive frames is defined as:

$\Delta_{i,j}^{t,t+1}=r_{i,j}^{t+1}-r_{i,j}^{t}$ (20)

where $r_{i,j}^{t}$ represents the relative position of objects $i$ and $j$ at
frame $t$. For a given frame $t$, the enriched feature vector that aggregates
both node information and temporal information is:

$\vec{x}_{t}=\vec{x}^{*}_{t}\oplus\sum_{j}w_{i,j}\cdot\Delta_{i,j}^{t,t+1}$
(21)

where $\oplus$ denotes element-wise summation and $w_{i,j}$ signifies the
influence of the positional change between objects $i$ and $j$, which we
define by an inverse distance relation given as: $w_{i,j}=1/(1+\delta_{i,j})$,
where $\delta$ is the distance between both objects.

Through this method, the frame-level feature vectors encapsulate the dynamic
interplay of object positions and their relationships across the video and
this way we get as one input to the TCN:
$X=\\{\vec{x}_{1},\vec{x}_{2},\dots,\vec{x}_{T}\\}$. Note that we segment
these vectors into smaller clusters by a windowing process where we use a
window size of $30$ frames and a windowing overlap of $10$ frames to divide
the video into shorter chunks. Thus, each frame is part of multiple windows
which improves the method’s capability to capture short-term and long-term
contextual dependencies. To handle boundaries, we use padding of the temporal
sequence with zero vectors to extend the sequence of both sides.

Thus, with this sliding window procedure, we transform the original temporal
sequence $X$ into a collection of overlapping input sequences, which are
subsequently fed into the TCNs for action recognition.

Output of the TCN We define the resulting output vectors and their matrix as
$\vec{g}^{(k)}$ and $\mathbf{G}^{(k)}$, for further processing as described
next, where $k$ refers to the GAT level.

Figure 7: A schematic of the temporal convolution step which incorporates
dynamic embedded edges of the scene graph into our framework.

###  3.5 Rule-based Computation of Hand Groups

In order to accurately compute bimanual action types, it is essential to
determine which hands are involved in each bimanual interaction. Inspired by
the work in [67] to achieve this, we create a contact graph which is a binary
adjacency matrix representation of the hand-object contacts in the scene
graph. The contact graph is computed for each frame by examining the spatial
relationships between objects and hands in the scene graph, and determining
whether any of the hands is in contact with any of the objects [62]. Each
entry in the contact graph indicates whether or not there is a contact between
the corresponding nodes, where the rows and columns correspond to the nodes in
the scene graph (i.e., hands and objects). A sample of a video frame and its
corresponding contact graph is shown Figure 8. In the left scene the right
hand is touching a box while the left hand is touching a bowl, therefore the
box and the bowl are in the right hand group $(H_{right})$ and the left hand
group $(H_{left})$, respectively. In this frame, the cutting board, knife,
banana and bottle do not touch a hand (directly or indirectly) and therefore
have no hand group (please notice the table as the main support surface is not
considered as a contact).

Figure 8: An example of two scenes in the KIT bimanual action dataset [65] and
their corresponding contact graphs

The role of the contact graph in our framework is to provide a way to identify
which hands are involved in each bimanual interaction. To achieve this, we
compute hand groups based on the connectivity between hands in the contact
graph. If two hands are connected in the contact graph (i.e., there is a path
of one more more connecting edges between them), they are considered to belong
to the same hand group. Hand groups are identified separately for each frame,
so the hand group information is a sequence of hand group labels, one label
for each frame. Let $\mathcal{G}_{c}=(\mathcal{V}_{c},\mathcal{E}_{c})$
represent the contact graph for a given frame, where $\mathcal{V}_{c}$ is the
set of nodes representing hands and objects, and $\mathcal{E}_{c}$ is the set
of edges indicating contacts between nodes. Suppose $\mathcal{H}$ is defined
as the set of nodes representing hands, and $h_{i}$ and $h_{j}$ are two hand
nodes. Then, $h_{i}$ and $h_{j}$ belong to the same hand group if and only if
there exists a path between them in this graph. In the right scene of Figure
8, the right hand is touching a screw driver and the left hand is touching a
hard disk, while the screwdriver and the hard disk have contact with each
other, therefore there is a path between two hands and the screwdriver and the
hard disk belong to both hand groups. In this scene the saw has no direct or
indirect contact with the hands and therefore it belongs to no hand group.
More details about these rule-based process-computation steps were discussed
in [67].

The hand group information is then used in section 3.6 to compute bimanual
action types based on the position and velocity information of the hands.
Without the hand group information, we would not be able to distinguish
between bimanual interactions where both hands are involved and those where
only one hand is involved. This information is important in our framework
since it helps us to accurately classify the different types of bimanual
actions that can occur.

###  3.6 Bimanual Action Type Computation

####  3.6.1 Concept

Bimanual type computation is an essential step for producing detailed and
informative descriptions of bimanual actions. It also allows us to better
understand the nature of the actions and how they are performed which can be
especially useful in domains such as cooking and sports.

Krebs et al. [67] proposed a taxonomy that characterizes bimanual actions
based on the symmetry or asymmetry, the coordination or independence of hand
movements, and the dominance of one hand over the other. This taxonomy
considers the position and velocity information of each hand in order to
categorize bimanual actions into one of four categories: symmetric
coordinated, asymmetric coordinated, symmetric uncoordinated, and asymmetric
uncoordinated. In symmetric coordinated actions, both hands move together in a
coordinated manner, such as clapping or playing piano. In asymmetric
coordinated actions, one hand is dominant and the other is subordinate, such
as when one hand leads while the other follows during writing or playing a
musical instrument. In symmetric uncoordinated actions, both hands move
independently but in a similar manner, such as when waving both hands. In
asymmetric uncoordinated actions, both hands move independently and in
different ways, such as when using a knife and fork to eat. The details of
this taxonomy and its measurements were discussed in [67].

However, in order to better suit our needs, we have extended the above
taxonomy to include a subcategory for hand spatial relationships, which
includes subcategories for close-hand, crossed-hand, and stacked-hand actions,
under the existing category of symmetry and asymmetry. We have also introduced
a new category for level of precision, which describes the degree of precision
required in performing the bimanual actions. This new category provides a
finer-grained description of the actions and can help to differentiate between
actions that require high or low levels of precision. These additions to the
taxonomy allow for a more detailed and comprehensive description of bimanual
actions, which can lead to more accurate and informative video descriptions.
Figure 13 represents our three newly defined hand spatial relations in a
bimanual action plus the output of involving different levels of precision in
chopping a carrot which shows the importance of considering precision details
in our recognition and description framework.

The modified bimanual taxonomy we apply is depicted in Figure 9. In this
taxonomy the blue cells were based on the Krebs et al. taxonomy [67] while the
pink cells depict the items in our modified taxonomy.

Figure 9: Our modified bimanual action taxonomy inspired by the proposed
taxonomy in [67]

Details how to determine had groups are found in the Supplementary Material
and below (Results) we show a comparison of using different levels in the
bimanual action taxonomy.

###  3.7 Feature Aggregation

For GAT levels 1 and 2 (see e.g. Fig. 7) we can directly use the feature
vectors $\vec{g}$ from the TCN. For the third level, we concatenate the output
of the TCN at level 3 (bimanual action) with a one-hot encoded vector
representing the bimanual type. For simplicity we do not change notation and
continue to use $\vec{g}$ and $\mathbf{G}$ for the feature vectors and its
matrix – non-aggregated or aggregated – used in the next steps.

###  3.8 Hierarchical Action Classification

The feature vectors $\vec{g}$ obtained above are used as inputs to a
hierarchical action classification module. The goal of this module is to
classify the action performed in the input video at multiple levels of detail
based on the outputs of the GAT layers.

For each GAT layer $t$ where $1\leq t\leq L$ and each action category $j$
among $N$ broad action categories, the feature matrix $\mathbf{G}^{(t,j)}$ is
generated. These feature matrices encode different levels of information and
granularity for action recognition in the $t$-th GAT layer (see the
supplementary material for a better understanding).

In the context of $N$ action categories, each containing $M$ sublevels and
each sublevel including $O$ items, the hierarchical action classification
process is as follows. For each action category $j$, sublevel $k$, and item
$o$, the feature matrix $\mathbf{G}^{(t,j,k,o)}$ is obtained from the GAT
output at layer $t$.

Each feature matrix $\mathbf{G}^{(t,j,k,o)}$ undergoes a series of fully
connected layers followed by a softmax function. The outcome is the action
probability distribution $P^{(t,j,k,o)}$ corresponding to the $t$-th GAT
layer, action category $j$, sublevel $k$, and item $o$. See Supplementary
Material for details of the learning process.

The predicted action label for the $t$-th GAT layer, action category $j$,
sublevel $k$, and item $o$ is represented as
$\hat{y}^{(t,j,k,o)}=\arg\max_{j}(P^{(t,j,k,o)})$. The classification process
is illustrated in Figure 10.

Figure 10: Overview of the hierarchical action classification module. The
feature vectors obtained from the feature aggregation step (Section 3.7) are
used as inputs to a series of fully connected layers followed by a softmax
function to obtain the action probability distribution at each level of the
hierarchy. The predicted action labels at each level can be combined to form a
complete action label for the entire video. In this figure, $y_{i,j}$ and
$p_{i,j}$ represent the possible broad action categories and their
corresponding probabilities at each level. Here for the simplicity we have
only shown the broad action categories.

###  3.9 Post-processing: Decision Making

After obtaining the probability vectors for hierarchical action classification
from (section 3.8), the final decision for each level is made by choosing the
action class with the highest predicted probability. However, due to the noisy
nature of the prediction results, the output may contain sudden spikes or
drops in the predicted probabilities, which can lead to inaccurate
classification results. To address this issue, we apply a moving average
filter, with a window size of $W$ frames, to smooth the predicted probability
curves and improve the overall performance of the system. This filter is
applied separately to each probability curve for each level, and the resulting
smoothed curves are used to make the final decision.

Let $P^{(k)}_{j}(t)$ be the probability of action class $j$ at time $t$ for
level $k$ and let $\hat{P}^{(k)}_{j}(t)$ be the corresponding smoothed
probability. The moving average filter is defined as:

$\hat{P}^{(k)}_{j}(t)=\frac{1}{W}\sum_{t-\lfloor W/2\rfloor}^{t+\lfloor
W/2\rfloor}P^{(k)}_{j}(t)$ (22)

where time $t$ is defined as frame-steps with $\lfloor W/2\rfloor$ denoting
the largest integer frame-number less than or equal to $W/2$, were we use
$W=5$.

The smoothed probabilities $\hat{P}^{(k)}_{j}(t)$ are then used to make the
final decision for each level, as follows:

$\hat{y}^{(k)}(t)=\arg\max_{j}\hat{P}^{(k)}_{j}(t)$ (23)

where $\hat{y}^{(k)}(t)$ is the predicted action label at time $t$ and level
$k$.

To demonstrate the importance of post-processing and the use of smoothing
techniques, we created a figure showing a man cutting a banana and pouring
something into a bowl sequentially (Figure 11). During the cutting process,
the action of his left hand was wrongly detected as “place” instead of “cut”
in one frame, and in another frame, the action of his right hand was wrongly
detected as “untouch” instead of “hold”. However, by applying a moving average
filter with a window size of 5, these mistakes were corrected.

Figure 11: The incorrect detection of the left-hand action in frame 3 and the
incorrect detection of the right-hand action in frame 9 were corrected by
applying a moving average filter with a window size of 5. The frames were
selected from the KIT bimanual action dataset [65].

###  3.10 Description Generation

In recent years, pre-trained language models, such as GPT-2, [68][69][70][71]
have shown promising results in various natural language processing tasks. In
this section, we discuss the use of a pre-trained GPT-2 model for generating
bimanual action descriptions.

The GPT-2 model has a relatively simple architecture compared to other models
such as GPT-3 [72], BERT [73], and T5 [74], which makes it easier to fine-tune
and adapt to our task. We use the GPT-2 medium model, which has 345 million
parameters and has been trained on a diverse range of text data, including web
pages, books, and articles. Figure 12 shows a schematic of how GPT-2 enters
our system.

However, to adapt the pre-trained GPT-2 model to the task of generating
bimanual action descriptions, we fine-tune the model on our annotated dataset.
The fine-tuning process involves updating the weights of the pre-trained model
to better capture the patterns and relationships in our dataset. The fine-
tuning process consists of the following steps (See Supplementary Material for
implementation details):

  * •

Tokenization of the input data.

  * •

Vectorization of these tokens to create numerical representations.

  * •

Using a sliding windows to handle longer input sentences.

  * •

Designing a specific model architecture adding generation layers responsible
for generating the action descriptions.

  * •

Performing weight adaptation of GPT-2 with a cross-entropy loss function and
by using the Adam optimizer.

The fine-tuning process is complete when the model has achieved satisfactory
performance on our annotated dataset for bimanual action description
generation and can generate accurate and coherent descriptions for new input
examples in the context of our specific task.

Figure 12: GPT-2 Based architecture for bimanual action description generation
Figure 13: Top row: Crossed hands, close hands and stacked hands in wiping,
cutting and hamming actions from the KIT bimanual action dataset [65].
Bottom row: Carrot pieces after chopping with low, medium and high levels of
precision

##  4 Results and Discussion

###  4.1 Qualitative Results

In Figure 13 we first demonstrate the effectiveness of our modified taxonomy.
For this we do a comparison.

  * •

Without involving bimanual type: “A person is performing a chopping action on
a cutting board.”

  * •

With involving Krebs taxonomy [67]: “A person is performing an asymmetric and
coordinated chopping action with a knife in his/her right hand and a cutting
board in his/her left hand.”

  * •

With involving our modified taxonomy: “A person is performing an asymmetric
and coordinated chopping action with a knife in his/her right hand and a
cutting board in his/her left hand, while maintaining a stacked-hand spatial
relationship and a high level of precision in cutting the vegetables into
small pieces.”

As you see with the modified taxonomy we can better capture the nuances and
complexities of the action, which can be important in certain contexts. For
example, in a surgical setting where delicate and precise bimanual actions are
required, having information about the hand spatial relationship and level of
precision can be important for describing the success of the procedure.

While computing this added information may require some additional
computational load, the benefits of having more detailed and precise
descriptions can be significant in certain contexts, and may be worth the
additional effort.

The general qualitative performance is shown in the samples of Figure 1. They
show a representative video frame for context and depicts the three levels of
our hierarchical framework for generating video descriptions, along with the
amount of detail included at each level.

###  4.2 Quantifications

Our evaluation is structured into several subsections, each focusing on
specific aspects of our research. We begin by assessing the recognition of
object-action relations, followed by an evaluation of manipulation action
recognition, bimanual action recognition, and description generation.
Additionally, we provide a cross-domain evaluation to gauge the generalization
capabilities of our model across different datasets. Furthermore, we compare
our results against state-of-the-art methods to underscore the benefits of our
approach.

Note that in the Supplement we provide details about the different data sets,
describing why they are useful for our tasks, in the main text we only name
them.

##### Object-Action Relation Evaluation

In this subsection, we evaluate the performance of our proposed method in
recognizing object-action relations.

Datasets: MS-COCO [75] and SomethingSomething [76]. For our experiments, we
select the subset of both datasets that specifically focuses on manipulation
actions. This subset includes relevant object categories and action labels
associated with manipulation actions. By using this subset, we can concentrate
our evaluation on the specific domain of object-action relations and obtain
more targeted insights into the performance of our algorithm.

Performance Comparison: To assess the performance of our method in learning
object-action relations, we compare it with the following selected state-of-
the-art (SOA) approaches: [24], [77], [78], and [65]. These approaches have
made significant contributions to the field of human-object interaction and
action recognition.

The performance comparison is summarized in Table 1, which presents the
results in terms of F1-score, recall, and precision for each approach on the
two datasets. The F1-score provides a measure of the overall accuracy, while
recall and precision offer insights into the completeness and correctness of
object-action relation predictions.

Table 1: ‘Performance Comparison of Object-Action Relation Learning between Our Method and State-of-the-Art Approaches on MS-COCO and Something-Something Datasets Approach | MS-COCO | Something-Something
---|---|---
F1-Score | Recall | Precision | F1-Score | Recall | Precision
Our Method | 0.83 | 0.86 | 0.81 | 0.78 | 0.80 | 0.76
[24] | 0.70 | 0.77 | 0.64 | 0.67 | 0.75 | 0.60
[77] | 0.63 | 0.72 | 0.56 | 0.61 | 0.70 | 0.54
[78] | 0.66 | 0.61 | 0.72 | 0.64 | 0.59 | 0.70
[65] | 0.69 | 0.64 | 0.75 | 0.66 | 0.61 | 0.72

The results of our method demonstrate its strong performance in both precision
and recall, and also the resulting high F1-Score, indicating that it achieves
a low false positive rate and effectively captures a large proportion of the
actual positive instances. In contrast, some of the compared methods show a
clear distinction between precision and recall, with one metric performing
better than the other (their corresponding recall or precision is high but
their F1-Score is not). This suggests that these methods may excel in either
accurately identifying positive instances or capturing a comprehensive set of
relevant instances, but do not achieve both simultaneously.

Discussion: In the field of object-action relations, several notable
approaches have been proposed, including those by Ji et al. [24], Wang et al.
[77], Kim et al. [78], and Dreher et al. [65].

Ji et al. [24] present a framework that represents actions as compositions of
spatio-temporal scene graphs. This approach focuses on capturing the
structural relationships between objects and actions within a scene.
Similarly, Wang et al. [77] propose an approach that models spatial
relationships between interaction points to detect human-object interactions.
Both methods contribute to the understanding of object-action relations but do
not explicitly emphasize the detailed learning and representation of such
relations.

Kim et al. [78] introduce an end-to-end framework that utilizes transformers
for human-object interaction detection. This approach benefits from the
ability of transformers to capture global context and long-range dependencies.
However, it does not explicitly focus on the fine-grained learning of object-
action relations.

By contrast, our method stands out by placing a specific emphasis on learning
and representing detailed object-action relations through the utilization of
graph networks and advanced relational modeling techniques. By capturing the
complex interplay between objects and actions more effectively, our method
achieves superior results in object-action relation learning. It excels at
capturing fine-grained dependencies and interactions between objects and
actions, resulting in a more comprehensive understanding of object-action
dynamics.

Dreher et al. [65] on the other hand directly focus on understanding of
object-action relations but their work has limitations related to dataset
dependence, scalability, sensitivity to variations, handling ambiguous
relationships, and capturing temporal dynamics. Their approach does not
explicitly address the temporal aspects of object-action relations. While they
focus on learning relationships from demonstrations, the temporal dynamics and
sequential patterns of object-action interactions may not be fully captured or
explicitly modeled in their method but our approach considers the temporal
dynamics of object-action relations. By explicitly modeling and incorporating
sequential patterns and temporal dependencies, we capture the logical
sequences of actions and their relationships over time. This temporal
awareness contributes to a more comprehensive understanding of object-action
dynamics and improves the overall accuracy and robustness of our approach.

Moreover, our method exhibits scalability and efficiency in handling larger
datasets and complex scenarios. We have designed our approach to address
computational complexity and memory requirements, ensuring its feasibility and
effectiveness even with increasing problem scales.

####  4.2.1 Manipulation Action Recognition Evaluation

In this section, we evaluate the performance of our proposed method for
manipulation action recognition on both, 2D and 3D, datasets. The evaluation
includes a comprehensive analysis of performance metrics and comparisons with
state-of-the-art methods.

2D and 3D Datasets: To assess the effectiveness of our proposed method for
manipulation action recognition in 2D, we selected several widely used
datasets known for their diverse collection of manipulation actions. We used
the following 2D datasets: Youcook2 [79], Charades-STA [80], ActivityNet
Captions [81], TACoS, [82]. To evaluate the performance of our proposed method
in a 3D context, we selected datasets that provide realistic and immersive
environments for manipulation action recognition which are: KIT Bimanual
Action Dataset [65], YCB-Video Dataset [83], G3D Dataset [84].

By utilizing these 3D datasets, we can evaluate the performance of our
proposed method in realistic 3D environments, where precise object
segmentations and depth information enhance the accuracy and robustness of
manipulation action recognition.

The inclusion of both 2D and 3D datasets in our evaluation ensures a
comprehensive assessment of our proposed method’s capabilities in recognizing
and understanding manipulation actions across different modalities and
contexts.

Performance Comparison: In this section, we compare the performance of our
proposed approach for manipulation action recognition with three state-of-the-
art papers: [24], [77], and [85]. These papers represent the state-of-the-art
in manipulation action recognition.

The selected papers are comparable to our work as they address similar
research objectives and propose novel methodologies for manipulation action
recognition. They provide solutions for capturing spatial and temporal
dependencies, recognizing human-object interactions, and using egocentric RGB
videos for accurate action recognition.

To compare our approach with these papers, we present a table summarizing the
key aspects and performance metrics on the selected datasets (Table 2).

Table 2: Performance Comparison of Manipulation Action Recognition Approaches.

Approach | Dataset | Accuracy | Precision
---|---|---|---
Our Method | Youcook2 | 0.85 | 0.79
Charades-STA | 0.83 | 0.76
ActivityNet Captions | 0.89 | 0.92
TACoS | 0.80 | 0.89
Kit Bimanual Action Dataset | 0.91 | 0.88
YCB-Video Dataset | 0.84 | 0.79
G3D Dataset | 0.89 | 0.84
[24] | Youcook2 | 0.82 | 0.77
Charades-STA | 0.75 | 0.82
ActivityNet Captions | 0.82 | 0.78
TACoS | 0.76 | 0.72
Kit Bimanual Action Dataset | 0.73 | 0.69
YCB-Video Dataset | 0.64 | 0.68
G3D Dataset | 0.80 | 0.76
[77] | Youcook2 | 0.66 | 0.75
Charades-STA | 0.72 | 0.68
ActivityNet Captions | 0.68 | 0.65
TACoS | 0.73 | 0.69
Kit Bimanual Action Dataset | 0.78 | 0.74
YCB-Video Dataset | 0.69 | 0.66
G3D Dataset | 0.70 | 0.63
[85] | Kit Bimanual Action Dataset | 0.81 | 0.78
YCB-Video Dataset | 0.84 | 0.85
G3D Dataset | 0.86 | 0.82

The comparison table provides a quantitative evaluation of the performance of
each approach on the specified datasets in recognizing manipulation actions.

In the extensive evaluation conducted across a variety of datasets, our method
consistently showcased exemplary performance, outperforming the competing
state-of-the-art approaches in most instances. Nonetheless, there were
specific scenarios where our method did not clinch the top spot. Particularly,
in the YCB-Video Dataset, our approach, despite demonstrating strong results,
was slightly edged out by the method introduced by Wen et al. [85], which
secured the highest precision scores $0.85$, while having the same accuracy
with our method $0.84$. This variance can be attributed to the nuanced
complexities and unique challenges presented by the dataset, which may have
been more effectively addressed by the hierarchical attention mechanisms
employed by Wen et al. In addition, while our method exhibited leading
accuracy on the Charades-STA dataset, it was narrowly outperformed in
precision, with the method by Ji et al. [24] recording the highest precision
of $0.82$. This divergence in performance might be due to the adaptability of
Ji et al.’s approach to diverse and dynamic environments inherent in the
dataset, allowing for more accurate identification and tracking of multiple
objects and their interactions over time. These instances not only underscore
the multifaceted challenges posed by different datasets but also serve as
invaluable learning points, guiding future refinements and advancements in our
methodology.

Discussion: Wen et al. [85] introduced a hierarchical temporal transformer for
3D hand pose estimation and action recognition from egocentric RGB videos.
Their method utilizes hierarchical attention mechanisms to capture fine-
grained temporal dynamics in hand actions. However, when compared to our
approach, their method does not demonstrate comparable performance in
manipulation action recognition tasks. This can be attributed to several
factors, including limitations in the modeling of spatial and temporal
relationships, potential challenges in handling complex object interactions,
and potential difficulties in capturing nuanced hand-object dynamics.

Ji et al. [24] and Wang et al. [77] as the other comparison candidates hd
partially been discussed in Section 4.2 already. Ji et al. involve
understanding both, the objects present in a scene (spatial aspect) and how
these objects interact or change over time (temporal aspect). This creates
challenges in terms of the complexity of data representation, scalability, and
computational efficiency. The approach struggles with accurately identifying
and tracking multiple objects and their interactions over time, especially in
cluttered or dynamically changing environments. Moreover, a common challenge
in Human-Object Interaction (HOI) detection in Wang et al.’s method is the
diversity and complexity of possible interactions between humans and objects.
Using interaction points for HOI detection does not capture the complete
semantics of interactions, particularly for complex or multi-stage actions.

In comparison, our approach overcomes these limitations by combining the power
of graph-based modeling and attention mechanisms. By explicitly representing
hand-object relationships as a scene graph and incorporating hierarchical
attention mechanisms, our method captures the intricate spatial and temporal
dependencies involved in manipulation actions. This enables us to achieve more
accurate and robust recognition of manipulation actions, even in complex and
challenging scenarios. Furthermore, our approach takes into account the
semantic and contextual information of hand-object interactions, leading to a
more comprehensive understanding of manipulation actions and improved
recognition performance.

####  4.2.2 Bimanual Action Recognition Evaluation

Datasets: In order to evaluate the performance of our proposed bimanual action
recognition approach, we selected the following datasets: KIT Bimanual Action
Dataset [65], YCB-Video Dataset [83],G3D Dataset [84], CMU Fine Manipulation
Dataset [86]. These datasets provide a diverse range of bimanual manipulation
actions and enable us to evaluate the performance of our approach in
recognizing and understanding bimanual action types, object interactions, and
precision levels.

Performance Comparison: In this subsection, we present a detailed performance
comparison of our proposed bimanual action recognition approach in two key
aspects: bimanual action recognition and bimanual taxonomy identification. We
compare the performance of our approach with two state-of-the-art papers:
Dreher et al. [65] and Andrade et al. [87].

Table 3 summarizes the performance comparison for bimanual action recognition
across the evaluated datasets.

Table 3: Performance Comparison for Action Recognition Approach | Dataset | Accuracy | Precision
---|---|---|---
Our Method | KIT Bimanual Action | 0.87 | 0.85
YCB-Video | 0.80 | 0.82
G3D | 0.86 | 0.82
CMU Fine Manipulation | 0.77 | 0.83
[65] | KIT Bimanual Action | 0.83 | 0.80
YCB-Video | 0.81 | 0.78
G3D | 0.83 | 0.76
CMU Fine Manipulation | 0.64 | 0.73
[87] | KIT Bimanual Action | 0.80 | 0.76
YCB-Video | 0.78 | 0.66
G3D | 0.75 | 0.70
CMU Fine Manipulation | 0.71 | 0.67

The performance comparison table elucidates the efficacy of our method across
various datasets. Our method predominantly outperforms the other approaches,
achieving the highest accuracy and precision in three out of the four
datasets. Notably, for the KIT Bimanual Action, YCB-Video, and CMU Fine
Manipulation datasets, our method exhibits superior performance, underscoring
its robustness and versatility in action recognition tasks.

However, it is important to acknowledge that our method does not universally
dominate in all scenarios. Specifically, in the G3D dataset, the approach
proposed by Dreher et al. [65] slightly surpasses ours, achieving an accuracy
and precision of 0.86 and 0.82, respectively, compared to our 0.83 and 0.76.
This divergence in performance could be attributed to the inherent
characteristics and challenges posed by the G3D dataset, potentially favoring
the techniques employed by Dreher et al.

Nevertheless, the overall superior performance of our method across diverse
datasets underscores its adaptability and effectiveness in manipulation action
recognition, demonstrating its potential as a valuable tool for advancing
research in this domain.

Discussion: Dreher et al. [65] specifically concentrated on bimanual action
recognition using graph neural networks. Their method demonstrated promising
results in capturing object-action relations and temporal dependencies in
bimanual scenarios. On the other hand, Andrade et al. [87] focused on general
action recognition without explicitly targeting bimanual actions. While their
approach did not directly address bimanual actions, their method incorporated
spatial and temporal dependencies between objects and actions. This aspect is
particularly relevant in understanding bimanual action scenarios where
interactions between objects and hands play a crucial role.

Our approach was initially inspired by Dreher et al.’s method, as both
approaches leverage scene graphs and graph neural networks (GNNs) for bimanual
action recognition. However, there are key differences that contribute to the
superior performance of our approach.

Firstly, the architecture of Dreher et al.’s method is simpler, consisting of
an encoder, a core, and a decoder including a simple two-layer perceptron as
update function and a sum function for node and edge aggregation. In contrast,
our approach incorporates a more complex architecture with a hierarchical
Graph Attention Network (GAT). The GAT’s multi-head attention mechanism allows
our model to capture more accurate and informative features, as it attends to
different aspects of the scene graph and captures both local and global
contextual information.

Furthermore, the treatment of spatial relations and temporal edges differs
between the two approaches. In Dreher et al.’s method, spatial relations and
temporal edges are treated as mutually exclusive. This means that they do not
consider the interaction between spatial and temporal aspects of the bimanual
actions. In contrast, our approach explicitly incorporates both spatial and
temporal dependencies by considering temporal concatenation of scene graphs.
This enables our model to capture the temporal evolution of object
relationships and actions, providing a more comprehensive understanding of
bimanual actions.

Additionally, Dreher et al. utilize one-hot vectors for encoding object and
action labels, representing each category as a binary feature. In contrast, we
leverage from learned embeddings for object and action labels, which capture
more nuanced and continuous semantic representations. This allows our model to
focus on objects that have a more significant contribution to the action
progression, rather than assigning equal weights to all objects. As a result,
our approach achieves greater efficiency and discriminative power in capturing
relevant object-action relationships.

Furthermore, the comparison between our work and the method proposed by
Andrade et al. reveals that our approach excels in terms of accuracy and
precision, despite both methods incorporating the temporal dimension in
activity recognition. One of the key strengths of our approach lies in the
utilization of graph neural networks (GNNs), which allows us to capture
complex spatial and temporal relationships. By representing interactions
between objects as a graph, our model gains a deeper understanding of human
activities. In contrast, Andrade et al.’s method relies on a temporal
convolutional neural network (CNN) architecture, primarily focusing on
capturing temporal patterns. While their approach demonstrates competence in
activity recognition, our use of graph-based modeling enables more detailed
analysis of object relationships and captures fine-grained temporal dynamics.
Additionally, our approach incorporates a hierarchical attention mechanism,
specifically Graph Attention Networks (GATs), which enhances precision and
discriminative power in activity recognition. The hierarchical attention
mechanism allows our model to selectively attend to relevant objects and
temporal features, further improving the accuracy of activity recognition. In
contrast, Andrade et al.’s method does not include such an attention
mechanism, limiting their ability to capture fine-grained spatial and temporal
dependencies.

Bimanual Action Types and Hand Spatial Relations:  In this section, we
evaluate our enriched version of the bimanual action taxonomy compared to the
taxonomy proposed by Krebs et al. [67], as illustrated in Fig.9. While our
method aligns with their approach in recognizing the blue cells in the
taxonomy, we have introduced a novel method to accurately identify hand
spatial relations and precision levels, denoted by the pink cell (Fig.9). As
no other taxonomy method exists in the literature for comparison, we focus on
reporting the accuracy of our system across the datasets.

Table 4: Accuracy of Bimanual Action Type Determination (Blue Cell Items in
Fig.9)

Dataset | Uncoordinated | Symmetric | Asymmetric | Dominant Hand
---|---|---|---|---
KIT Bimanual Action | 0.87 | 0.92 | 0.89 | 0.88
YCB-Video | 0.84 | 0.91 | 0.86 | 0.85
G3D | 0.89 | 0.93 | 0.90 | 0.89
CMU Fine Manipulation | 0.86 | 0.90 | 0.87 | 0.86

Table 5: Accuracy of Bimanual Action Type Determination (Pink Cell Items in
Fig.9)

Dataset | Hand Spatial Relation | Precision Level
---|---|---
Close Hands | Crossed Hands | Stacked Hands
KIT Bimanual Action | 0.91 | 0.88 | 0.93 | 0.92
YCB-Video | 0.88 | 0.87 | 0.90 | 0.89
G3D | 0.90 | 0.89 | 0.92 | 0.91
CMU Fine Manipulation | 0.89 | 0.86 | 0.91 | 0.88

As shown in Table 4, our system achieves notable accuracy in determining
bimanual action types. The accuracy scores range from 0.84 to 0.93, indicating
the robustness and effectiveness of the method. Specifically, our system
consistently achieves accuracy levels above 0.85 for symmetric and asymmetric
actions, as well as accuracy levels above 0.86 for uncoordinated actions.
Moreover, the accuracy in determining the dominant hand ranges from 0.85 to
0.89, demonstrating the reliability of our approach.

Additionally, our method exhibits impressive precision in identifying hand
spatial relations, as depicted in the second table of Table 4. The precision
scores also range from 0.88 to 0.92 for finding hand spatial relations
regarding each other.

While a direct comparison with existing taxonomy methods is not possible due
to the absence of alternative approaches in the literature, our enriched
taxonomy reports convincing accuracies across multiple datasets. These notable
accuracy scores highlight the reliability of our approach, which holds
significant potential for various applications, including video description
generation. Moreover this way, robots can better understand and replicate
human actions, leading to more efficient human-robot collaboration.

####  4.2.3 Description Generation Evaluation

Datasets:  For evaluating the performance of our description generation
approach, we have selected a set of datasets that provide diverse and
annotated video descriptions. These datasets allow us to assess the
effectiveness of our method in generating accurate and informative
descriptions of both 2D and 3D manipulation and bimanual actions. For 2D
Datasets we used: Youcook2 Dataset [79], TACoS Multi-Level Dataset [82],
ActivityNet Captions Dataset [81], Charades Dataset [88].

For 3D Datasets we used the KIT Bimanual Action Dataset [65].

These datasets were chosen for their strong relevance to manipulation and
bimanual actions, as well as their availability of natural language
descriptions or annotations. Please note that although datasets like MSR-VTT
[89] and MSVD [90] are also commonly used in state-of-the-art research on
video description but could not be used here because of their limitations
concerning the level of annotation detail (e.g. no bimanual annotations).

Performance Comparison: In this subsection, we compare the performance of our
description generation approach with three state-of-the-art papers: Wang et
al. (2018) [91], Seo et al. (2022) [54], and Lin et al. (2022) [53]. These
papers represent significant contributions to the field of video captioning
and provide valuable insights into different approaches and techniques.

To evaluate the performance, we compare our approach with the aforementioned
papers using several commonly used evaluation metrics, including BLEU-4,
METEOR, ROUGE-L, and CIDEr. These metrics provide a quantitative assessment of
the quality and similarity of the generated descriptions compared to the
ground truth annotations.

Table 6 summarizes the performance of our approach and the three selected
papers on the various datasets used in our evaluation.

Table 6: Performance Comparison of Video Captioning Approaches Approach | Dataset | B-4 | M | R | C
---|---|---|---|---|---
Our Method | Youcook2 | 51 | 39 | 59 | 68
| TACoS Multi-Level | 53 | 41 | 63 | 74
| Charades | 49 | 38 | 54 | 62
| KIT Bimanual Action | 63 | 46 | 69 | 77
Wang et al. [91] | Youcook2 | 48 | 37 | 58 | 61
| TACoS Multi-Level | 46 | 33 | 56 | 66
| Charades | 41 | 30 | 49 | 55
| KIT Bimanual Action | 52 | 42 | 64 | 68
Seo et al. [54] | Youcook2 | 45 | 33 | 51 | 59
| TACoS Multi-Level | 49 | 37 | 57 | 66
| Charades | 38 | 29 | 49 | 53
| KIT Bimanual Action | 55 | 43 | 62 | 68
Lin et al. [53] | Youcook2 | 47 | 37 | 50 | 52
| TACoS Multi-Level | 51 | 42 | 61 | 70
| Charades | 40 | 29 | 43 | 49
| KIT Bimanual Action | 47 | 33 | 52 | 55

The comparison table provides a quantitative evaluation of the performance of
each approach on the specified datasets. It showcases the performance of our
method in terms of BLEU-4, METEOR, ROUGE-L, and CIDEr scores, providing
insights into the effectiveness of our description generation approach in
comparison to the state-of-the-art methods.

Discussion:  Lin et al. [53] introduce S-WIN-BERT, a comprehensive end-to-end
Transformer model, underpinned by a sparse attention mask. Despite its
noteworthy performance on multiple benchmark datasets, the model contains a
caveat: its predilection towards specific frames can instigate information
loss, culminating in potential inaccuracies in caption generation.

By contrast, our approach, rooted in graph-based modeling, better represents
the intricate relationships between objects and actions. This significant
feature raises the precision and relevancy of video descriptions generated by
out model. In addition, our hierarchically structured Graph Attention Network
(GATs) mechanism promotes accuracy in action recognition, further
supplementing the generation of appropriate video descriptions. A pivotal part
of our methodology is the detailed representation of hand-object interactions
through scene graphs, elevating the contextual relevancy of video
descriptions.

A salient feature of our methodology is the enriched bimanual action taxonomy
that amplifies our model’s comprehensive understanding of actions. This, in
turn, aids in generating contextually appropriate and comprehensive video
descriptions.

When juxtaposed with Seo et al.’s [54] innovative generative pretraining
framework, our method harnesses a robust attention mechanism, facilitating
better alignment between video content and corresponding text. This feature
yields video descriptions that mirror the video content accurately, thereby
enhancing user comprehension. The diversity and scale of our dataset prime our
model to tackle a wide array of video domains, thus elevating its resilience
and performance in video understanding tasks.

Our end-to-end training approach amalgamates the pretraining and fine-tuning
stages effectively. This integration not only optimizes learning outcomes but
also yields coherent, meaningful, and contextually pertinent video
descriptions. Therefore, our methodology outperforms Seo et al.’s approach by
generating more accurate and contextually relevant video descriptions.

Finally, the RecNet model by Wang et al. [91] showcases potential in producing
accurate video captions, but faces challenges when dealing with complex
content-description relationships or videos with substantial visual
variations. Our methodology offers a here a better alternative. Owing to its
resilient adaptability to diverse video content, our model produces accurate
video descriptions even in the face of complex scenarios. Furthermore, our
emphasis on hand-object interactions, captured through detailed scene graphs,
enables the generation of nuanced video descriptions.

Our refined bimanual action taxonomy contributes to a broader and more
comprehensive understanding of actions, where the hierarchically structured
attention mechanism ensures the generation of coherent, meaningful, and
contextually relevant video descriptions.

###  4.3 Ablation Study

In this section, we conduct ablation studies to investigate the individual
contributions of various model components in our action recognition framework.
We systematically assess the impact of different ablations on the overall
action recognition accuracy, shedding light on the importance of features,
processing steps, and model configurations.

####  4.3.1 Ablation on the Embedding Layer

One of the foundational steps in our action recognition framework is the
conversion of categorical or symbolic node and edge information into
continuous vector representations through the embedding layer. This conversion
enables neural networks to process and learn from the data more effectively.
In this subsection, we explore the significance of this embedding process by
removing it from our model and analyzing the subsequent impact on performance.

##### Methodology:

To investigate the role of the embedding layer, we modified our architecture
to bypass this layer. Instead of feeding dense embedded vectors to subsequent
layers, raw categorical inputs for nodes and edges were directly used. To make
this categorical data suitable for neural network processing, we employed one-
hot encoding. This method represents each unique category as a binary vector,
where the position corresponding to the category is marked as ‘1’, and all
other positions are set to ‘0’.

While this encoding ensures that the data remains numerical, it increases the
dimensionality, leading to larger input sizes for the subsequent layers.
Necessary adjustments and dimension compatibility checks were made to ensure
smooth data flow in the network.

##### Training and Evaluation:

The modified model, devoid of the embedding layers, was trained using the same
datasets, parameters, and settings as our original model. This consistency
guarantees that any observed performance variation is attributable solely to
the removal of the embedding process. Post-training, we evaluated the model
using our standard metrics and evaluation datasets.

##### Results

Tables 7-9 show the results of these ablation experiments for (7): object-
action relation evaluation, (8): manipulation action recognition and (9):
bimanual action recognition.

Table 7: Object-Action relation performance for manipulation actions in a subset of the MS-COCO dataset: having the embedding layer vs. removing it. Metric | F1-Score | Recall | Precision
---|---|---|---
With Embedding Layer | 0.83 | 0.86 | 0.81
Without Embedding Layer | 0.74 | 0.76 | 0.68
Table 8: Performance of manipulation action recognition for Youcook2 (2D) and KIT (3D) with and without embedding. Dataset | Metric | Embedding | No Embedding
---|---|---|---
Youcook2 | Accuracy | 0.85 | 0.79
Precision | 0.79 | 0.73
KIT | Accuracy | 0.91 | 0.88
Precision | 0.88 | 0.73
Table 9: Bimanual action recognition performance on KIT and YCB datasets with and without the embedding layer. Dataset | Metric | Embedding | No Embedding
---|---|---|---
KIT | Accuracy | 0.91 | 0.88
Precision | 0.85 | 0.77
YCB-Video | Accuracy | 0.84 | 0.79
Precision | 0.82 | 0.74

In summary, the embedding layer’s removal led to a decline in performance
(approximately 10% to 15%), underscoring its pivotal role in our framework.
While performance degradation was present, it was not as severe as one might
anticipate. This can be attributed to the inherent robustness of our model
architecture and its ability to adapt. The graph neural networks have, it
seems, partially compensated the absence of embeddings. Additionally, while
one-hot encoding lacks semantic depth, it still offers categorical clarity,
which can provide cues for certain recognition tasks.

####  4.3.2 Ablation on the Edge Features

Edges in our graph-based action recognition framework embody the relational
and contextual information between nodes, pivotal for deciphering complex
actions. In this segment, we show the results of an extensive ablation study,
zeroing in on the significance of disparate edge features: spatial relations
(SR) and object-action associations. We address three distinct configurations:

  1. 1.

Eliminating solely Spatial Relations (SR), yet preserving object-action edges.

  2. 2.

Discarding only object-action edges, maintaining SRs.

  3. 3.

Thoroughly stripping all edge features, leaving solely the node features.

Every configuration is assessed across three primary benchmarks: object-action
relation evaluation, manipulation action recognition, and bimanual action
recognition.

##### Methodology:

To scrutinize the role of edge features in our action recognition framework,
we crafted several modified versions of our model architecture. In the first
setting, we bypassed the spatial relations (SR) but retained the object-action
associations. In the second configuration, we retained SRs and eliminated the
object-action edges. Finally, in the third scenario, we completely stripped
off all edge features, leaving only the node features for processing.

While the nodes remained consistent across all versions, the nature and type
of relationships between these nodes varied. These modifications, especially
the complete removal of edge information, led to significant changes in the
graph’s structure, affecting the subsequent processing layers. Adjustments
were made to ensure the modified graph’s compatibility with later stages of
the framework.

##### Training and Evaluation:

Each modified model, adjusted based on the type of edge feature ablation, was
trained using the same datasets, parameters, and settings as our baseline
model. Maintaining consistency ensured that any deviation in performance could
be attributed solely to the edge features’ modifications. After training, we
subjected the models to evaluations using our standard metrics and evaluation
datasets.

##### Results

Tables 10-12 show the results of these ablation experiments for (10): object-
action relation evaluation, (11): manipulation action recognition and (12):
bimanual action recognition.

Table 10: Performance metrics post edge-feature ablation for object-action relations on MS-COCO dataset. Metric | Baseline | No SR | No Object-Action | No Edges
---|---|---|---|---
F1-Score | 0.85 | 0.64 | 0.52 | 0.41
Recall | 0.87 | 0.65 | 0.54 | 0.44
Precision | 0.83 | 0.60 | 0.49 | 0.42
Table 11: Performance metrics post edge-feature ablation for manipulation action recognition. Dataset & Metric | Baseline | No SR | No Object-Action | No Edges
---|---|---|---|---
Youcook2 Accuracy | 0.85 | 0.58 | 0.52 | 0.38
Youcook2 Precision | 0.87 | 0.61 | 0.50 | 0.41
KIT Accuracy | 0.83 | 0.54 | 0.48 | 0.35
KIT Precision | 0.86 | 0.59 | 0.52 | 0.31
Table 12: Performance metrics post edge-feature ablation for bimanual action recognition. Dataset & Metric | Baseline | No SR | No Object-Action | No Edges
---|---|---|---|---
KIT Accuracy | 0.91 | 0.68 | 0.59 | 0.46
KIT Precision | 0.69 | 0.52 | 0.45 | 0.35
YCB-Video Accuracy | 0.89 | 0.67 | 0.58 | 0.45
YCB-Video Precision | 0.85 | 0.64 | 0.55 | 0.43

In summary, while each edge feature plays its part, the object-action
associations emerge as the most indispensable. Their removal leads to the most
significant performance dip, reiterating their foundational role in the
framework. The simultaneous presence of both spatial relations and object-
action associations ensures optimal performance, highlighting the symbiotic
nature of these features in our action recognition framework.

####  4.3.3 Ablation on GAT Layers

Graph Attention Networks (GAT) form the backbone of our graph-based action
recognition framework, allowing for attention-driven, dynamic aggregation of
neighboring node features. The hierarchical structure of our GAT consists of
three layers, with each layer’s output serving as an input for the subsequent
one. To understand the contribution of each layer, we performed an ablation
study, focusing on the removal of the third layer and both the second and
third layers.

##### Methodology:

We modified our GAT architecture in two distinct ways for this ablation study:

  1. 1.

Removing only the third GAT layer.

  2. 2.

Removing both the second and third GAT layers.

In both scenarios, the subsequent architecture adjustments were made to ensure
that the data flow remained uninterrupted.

##### Training and Evaluation:

Post modifications, the altered models were trained with the same parameters
and settings as our baseline model. This approach ensured that any performance
variations were attributed solely to the ablation of the GAT layers. After
training, we evaluated the models using our standard evaluation metrics.

##### Results:

Tables 13-15 show the results of these ablation experiments for (13): object-
action relation evaluation, (14): manipulation action recognition and (15):
bimanual action recognition.

Table 13: Object-Action relation performance on the MS-COCO dataset after GAT layer ablation. Metric | Baseline | No 3rd Layer | No 2nd & 3rd Layer
---|---|---|---
F1-Score | 0.85 | 0.80 | 0.71
Recall | 0.87 | 0.81 | 0.66
Precision | 0.83 | 0.75 | 0.61
Table 14: Performance after GAT layer ablation on manipulation actions. Dataset | Metric | All GAT Layers | No 3rd Layer | No 2nd & 3rd Layer
---|---|---|---|---
Youcook2 | Accuracy | 0.85 | 0.68 | 0.59
Precision | 0.79 | 0.65 | 0.63
KIT | Accuracy | 0.91 | 0.57 | 0.55
Precision | 0.88 | 0.74 | 0.60
Table 15: Performance after GAT layer ablation on bimanual actions. Dataset | Metric | All GAT Layers | No 3rd Layer | No 2nd & 3rd Layer
---|---|---|---|---
KIT | Accuracy | 0.91 | 0.69 | 0.58
Precision | 0.88 | 0.72 | 0.51
YCB-Video | Accuracy | 0.84 | 0.57 | 0.65
Precision | 0.79 | 0.42 | 0.47

In summary, the results from our ablation study show the critical importance
of the GAT layers in our action recognition framework. Across various
contexts, from object-action relations on the COCO dataset to manipulation
action recognition, and even bimanual action recognition on the KIT and YCB
datasets, the presence of the GAT layers consistently bolsters performance.
The third GAT layer, in particular, acts as a fine-tuner, refining the model’s
ability to discern nuanced relationships and interactions. When this layer is
removed, we observe a tangible decline in recognition accuracy across all
tasks. However, the consequences are even more pronounced when both the second
and third layers are omitted. This signifies that while the third layer is
crucial for refining the model’s insights, the combined depth and hierarchical
processing provided by the second and third layers are fundamental for the
model’s efficacy. In a nutshell, the GAT layers, especially in a stacked
configuration, play a paramount role in ensuring the model’s robust
performance across a spectrum of action recognition tasks.

####  4.3.4 Ablation on the Temporal Aspect

In this section, we explore the implications of omitting the temporal aspect
from our model and assess the potential compensatory role of Graph Attention
Networks (GAT).

##### Methodology:

To examine the repercussions of excluding the temporal component, we adjusted
our model architecture to bypass the TCN layers. While the spatial and
relational aspects remained intact, the model lacked the capability to
understand sequences or the chronological order of frames. The presence of GAT
in our architecture offered a potential buffer. GAT’s attention mechanisms can
capture certain temporal relationships, albeit not as intricately as dedicated
temporal layers like TCN.

##### Training and Evaluation:

The modified model, devoid of the TCN layers but still incorporating GAT, was
trained on our standard datasets using the same parameters and settings as the
baseline model. This consistency in training ensured that observed performance
variations were solely due to the omission of the temporal aspect. The model
was then subjected to evaluations using our typical metrics and datasets.

##### Results

Manipulation Action Recognition: Manipulation actions, by their inherent
nature, often involve a sequence of intricate movements and interactions with
objects. These sequences unfold over time, making the temporal aspect
especially critical for accurate recognition. In datasets like Youcook2 (2D)
and KIT bimanual action dataset (3D), where manipulation actions are
prevalent, understanding the temporal progression can be the key to
distinguishing between similar actions or nuances of the same action executed
differently.

Table 16: Performance metrics post temporal ablation for manipulation action recognition. Dataset & Metric | Baseline | No Temporal Aspect
---|---|---
Youcook2 Accuracy | 0.85 | 0.43
Youcook2 Precision | 0.79 | 0.45
KIT Accuracy | 0.91 | 0.41
KIT Precision | 0.88 | 0.50

Upon removal of the temporal aspect, even with the presence of GAT, a
noticeable decline in performance metrics was observed (Tab. 16). While the
GAT could somewhat mitigate the loss by capturing certain temporal
relationships through its attention mechanisms, it could not entirely
compensate for the absence of dedicated temporal modeling. This experiment
accentuates the role of time in understanding and recognizing manipulation
actions, emphasizing the need for both spatial and temporal modeling in action
recognition frameworks.

##  5 Conclusion and Future Works

This research has laid the groundwork for a significant improvement in the
domain of video description generation, emphasizing manipulation and bimanual
actions. We have introduced a robust methodology that blends graph-based
modeling, an enriched bimanual action taxonomy, and hierarchically structured
Graph Attention Networks (GATs) to obtain a high level of precision and
contextual relevance in video description. Our model outperforms existing
state-of-the-art methods on numerous 2D and 3D datasets, demonstrating its
effectiveness and adaptability.

By recognizing actions at different levels of detail, we can generate more
comprehensive and informative action descriptions. For example, instead of
simply describing the action as “cutting fruit”, we can describe it as “using
a knife to slice a piece of fruit with the right hand”. Similarly, instead of
simply describing the bimanual action as “mixing fruit”, we can describe it as
“using both hands to hold and mix the chopped fruit together in a bowl”.

This hierarchical structure provides several advantages over a flat GAT model.
Firstly, it allows us to capture relationships between objects and actions at
different levels of granularity, leading to improved performance on action
recognition tasks. Secondly, it allows for more interpretable results, as we
can easily trace how the model arrived at a particular action recognition
decision. Finally, it allows for better generalization to new scenes, as the
model can learn more abstract representations of objects and actions that are
applicable across a wide range of scenarios.

However, there are a few potential limitations. The model’s reliance on the
granularity of the scene graphs could limit the precision of the video
descriptions under certain circumstances. This limitation arises from the
inherent complexity of visual data and the potential variability in the
representation of hand-object interactions.

Despite this, the potential avenues for future exploration are broad and
promising. An immediate avenue for research is to refine our scene graph
generation methodology to capture more nuanced object-action relationships.
This refinement could enhance our model’s ability to generate even more
accurate and contextually relevant video descriptions.

In addition, the advancement of our model to handle 3D video content presents
an exciting direction for future work. As 3D data becomes increasingly
prevalent, the ability to effectively generate descriptions for such content
will become even more crucial.

Furthermore, our research has implications that extend well beyond the domain
of video description generation, offering potential advancements in the realm
of robotics. The understanding of human manipulation actions our model offers
can directly inform and enhance the functionality of robotic manipulators. The
scene graphs and detailed hand-object interactions, as revealed by our model,
could serve as a knowledge base for robots, enabling them to learn human-like
manipulations. This understanding, grounded in real-world actions, could
enrich the control strategies of robotic systems, making them more precise and
adaptable.

In addition, our emphasis on bimanual actions adds another contribution. Most
current robotic systems primarily rely on single-hand manipulations. By
incorporating our model’s understanding of dual-handed human actions, the
design of robotic systems could be significantly improved, broadening the
range of tasks they can perform, and enhancing their capacity to execute
complex activities.

As a consequence, our approach has potential applications in the field of
collaborative robotics, where robots need to work in tandem with humans. Our
model, with its fine-grained understanding of human actions, can help in
creating robots that can better comprehend and replicate human actions,
fostering improved human-robot collaborations. To contribute to the design of
more intelligent, adaptable, and human-like robots is indeed a compelling
prospect and serves as a valuable direction for future research.

##  6 Acknowledgements

FW acknowledges funding by the German Science Foundation (DFG), collaborative
research center SFB 1528, sub-project B01.

SUPPLEMENTARY MATERIAL

##  1 Datasets

Here we first describe the datasets used and their characteristics. We also
explain how our algorithms had been adapted to them. Then we also describe the
required adjustments of temporal kernel sizes in the TCN, the optimization of
window sizes, stride lengths, and input sequence lengths for the GPT-2 model
across these different datasets.

###  1.1 Datasets: Scope, Characteristics, and Dataset-specific Configurations

  * •

The Youcook2 Dataset [79] comprises a large collection of cooking videos,
covering a wide range of manipulation actions involved in food preparation. It
provides detailed temporal annotations, making it suitable for evaluating the
recognition and understanding of fine-grained manipulation actions in a
cooking context. We follow the prevalent split in the literature, partitioning
the dataset into 70% for training, 15% for validation, and 15% for testing.

  * •

The Charades-STA Dataset [80] consists of daily activity videos recorded in a
home environment, including a diverse set of manipulation actions. With
extensive annotations and natural language descriptions, this dataset allows
us to evaluate our method’s capability to recognize and describe manipulation
actions in various real-life scenarios. We allocate 80% of the data for
training, 10% for validation, and 10% for testing, consistent with standard
practices.

  * •

The ActivityNet Captions Dataset [81] offers a broad range of videos capturing
different activities, including numerous manipulation actions. It provides
temporal annotations and natural language descriptions for video segments,
enabling us to evaluate the performance of our method in generating accurate
and informative descriptions of manipulation actions. The majority of the
literature prescribes a split of 50% for training, 25% for validation, and 25%
for testing. We adhere to this configuration in our experiments.

  * •

The TACoS Dataset [82] focuses specifically on cooking videos and provides
detailed temporal annotations of fine-grained manipulation actions involved in
cooking tasks. This dataset allows us to evaluate our method’s ability to
accurately recognize and understand manipulation actions in the context of
cooking. For our experiments, we adhere to a common split used in the
literature, allocating 75% of the data for training, 12.5% for validation, and
12.5% for testing.

  * •

The KIT Bimanual Action Dataset [65] focuses on bimanual manipulation actions
and provides synchronized depth and color videos along with accurate 3D object
segmentations. This dataset offers a rich variety of manipulation actions
performed with both hands, allowing us to evaluate our method’s ability to
recognize and understand complex bimanual interactions. Here, we adopt a
typical split of 70% for training, 15% for validation, and 15% for testing.
This partitioning aligns with many works in the domain.

  * •

The YCB-Video Dataset [83] consists of RGB-D videos capturing various object
manipulation tasks. It includes precise object models and segmentation masks,
facilitating accurate analysis and recognition of manipulation actions. This
dataset enables us to evaluate our method’s performance in handling object-
centric manipulation actions. We partition the dataset into 80% for training,
10% for validation, and 10% for testing.

  * •

The G3D Dataset [84] captures hand-object interactions in diverse daily
activities, providing RGB-D videos, 3D hand and object segmentations, and
annotations related to hand-object interactions. This dataset offers a
comprehensive set of manipulation actions performed in naturalistic settings,
enabling us to evaluate the robustness and accuracy of our method in
recognizing and understanding complex hand-object interactions. We use,
similar to others, 75% for training, 12.5% for validation, and 12.5% for
testing.

  * •

The COCO Dataset [75] contains a large number of images with bounding box
annotations for various object categories. It provides annotations for 80
common object categories, including many objects relevant to manipulation
actions such as “tools,” “utensils,” and “vehicles”. We follow established
practices and partition the data into approximately 60% for training, 20% for
validation, and 20% for testing.

  * •

The Something-Something Dataset [76] provides diverse and rich annotations of
object and action information, making them suitable for studying manipulation
actions and their relations with objects. We adopt the widely used split of
60% for training, 20% for validation, and 20% for testing.

  * •

The Charades Dataset [88] includes 9848 videos of 267 individuals performing
scripted activities, annotated with 157 action classes and 27847 textual
descriptions. It is used for tasks such as action recognition and video
captioning. The Charades-STA [80], a derivative of Charades, is specifically
designed for temporal localization of described events within videos.
Following literature, the split becomes approximately 70% for training, 11%
for validation, and 19% for testing.

  * •

The CMU Fine Manipulation Dataset [86] offers a controlled environment for
performing fine-grained manipulation tasks. It includes RGB-D videos capturing
various object manipulation actions with high precision requirements. The
dataset provides detailed annotations, such as object poses and hand poses,
allowing for the analysis of precision levels in executing manipulation
actions. By incorporating this dataset into our evaluation, we can assess the
fine-grained details of precision in bimanual actions and evaluate the
effectiveness of our approach in capturing and quantifying precision levels.
While this dataset is newer, most works, as of our last literature survey,
utilize a split around the 70-15-15 or sometimes 80-10-10 mark. We adopt the
former, dividing our data into 70% for training, 15% for validation, and 15%
for testing.

###  1.2 Variation of TCN Kernel Sizes across Datasets

For the Kitchen Actions dataset, which involves short-term temporal
dependencies and quick action transitions, we discovered that an optimal $K=4$
allowed the TCN to capture the relevant temporal context efficiently.
For the Daily Life Actions dataset, which comprises a mix of short and long-
term temporal patterns, we determined that an optimal $K=6$ struck a balance
in capturing both types of dependencies accurately.
In the case of datasets of simulated environments, where measures are
independent and without significant temporal relationships, a smaller $K=2$
proved sufficient for TCN to detect those minimal temporal associations.

###  1.3 Variation of Window Size, Stride Length, and Input Sequence Length
for Fine-tuning the GPT-2 Model across Datasets

Given the datasets we’re working with, the lengths of sequences can vary
significantly. Here’s a breakdown:

  1. 1.

Length of the Input Sequence ($L$):

     * •

YouCook2 Dataset and ActivityNet Captions Dataset: These datasets are
characterized by longer, more descriptive sequences. Typical sequences can
range from 300 to 700 tokens, especially when considering the entire context
of a video.

     * •

Charades-STA Dataset and YCB-Video Dataset: Sequences from these datasets are
moderately long, often ranging between 150 to 400 tokens.

     * •

KIT Bimanual Action Dataset and G3D Dataset: These datasets have relatively
short annotations, often within 50 to 150 tokens.

  2. 2.

Window Size ($W$):

     * •

For datasets with longer sequences like YouCook2 and ActivityNet Captions,
we’ve used a window size of 512 tokens to capture adequate context.

     * •

For datasets with moderate sequences like Charades-STA and YCB-Video, a window
size of 384 tokens has been optimal.

     * •

For datasets with shorter sequences, such as KIT Bimanual Action and G3D, a
window size of 256 tokens has been found to be sufficient.

  3. 3.

Stride Length ($S$): To ensure overlap and maintain context continuity, we
have adopted a stride length of half the window size, i.e., $S=W/2$. This
provides a balance between computational efficiency and context preservation
across all datasets.

These sequence lengths and window sizes have been determined through detailed
experimentation on each dataset, ensuring that the chosen parameters offer the
best trade-off between capturing meaningful context and computational
efficiency.

##  2 Hierarchical Learning Strategy

Given the complexity of our approach and the diverse demands of different
computational stages, we employed a carefully designed “staged training”
strategy. Initially, each individual stage or sub-task is trained in
isolation, allowing us to refine the performance of each component. Once
satisfactory performance levels are achieved for each individual stage, we
move on to an end-to-end training paradigm. During this phase, all the model
components are harmoniously fine-tuned together. This strategic transition not
only guarantees stability of the learning process but also facilitates
improved convergence. Importantly, each module enters the end-to-end training
with a foundation of pre-trained knowledge, which further enhances the model’s
ability to capture intricate relationships and nuances in video content.

###  2.1 Decomposing the Learning Process

Our approach follows a step-by-step process:

  1. 1.

Separate Training of Learnable Steps: We initiate the training process by
independently training the different steps that have learnable parameters,
which are Steps 2 (embedding), 3 (GAT), 4 (TCN), and 10 (video description).
Numbering here follows the section numbers in the main text.

     * •

Step 2: Discussed in subsection 3.2.

     * •

Step 3: Discussed in subsection 3.3.

     * •

Step 4: Discussed in subsection 3.4.

     * •

Step 10: Discussed in subsection 3.10.

This initial phase enables each component to acquire specialized knowledge and
learn task-specific patterns.

  2. 2.

Joint Training of Steps 3 and 4 Building upon the insights gained from their
separate training, we enter a joint training phase for Steps 3 (GAT) and 4
(TCN). By merging the capabilities of these stages, we enhance the model’s
capacity to capture both spatial and temporal dynamics effectively.

  3. 3.

Integration of Steps 2, 3 and 4 With the enriched features from the joint
training of Steps 3 and 4, we proceed to integrate them with the output of
Step 2 (node and edge embedding). This comprehensive feature fusion uses both
spatial and temporal information to create a refined representation.

  4. 4.

End-to-End Training with Step 10: The final phase involves an end-to-end
training process, where the integrated features from Steps 2, 3, and 4 are
further merged with Step 10 (description generation). This all-encompassing
training approach hones the model’s ability to generate coherent video
descriptions that are both contextually accurate and narratively coherent.

Each stage of this staged training approach builds upon the knowledge gained
from previous phases, allowing the model to iteratively refine its
understanding and representation of video content.

During these training stages, we also employ regularization techniques to
avoid over-fitting. Regularization penalizes excessive complexity in the
model, promoting a balance between fitting the data well while maintaining
generalization capabilities. Furthermore, to ensure that our model achieves
optimal performance without over-adapting to the training data, we use “early
stopping”. This approach continuously monitors the validation loss, halting
training if no substantial improvement is detected over a set number of
epochs, ensuring both stability and convergence in our learning process.

####  2.1.1 Learning Process for the Node and Edge Feature Encoding Parameters

To learn the transformation matrices $\mathbf{L}_{H}$, $\mathbf{L}_{E}$, and
$\mathbf{L}_{A}$, we employ Siamese Networks [92] in tandem with contrastive
learning [93].
Contrastive learning offers a compelling solution for our current task,
because this learning type is specifically useful for situations where the
objective is to discern the similarity between pairs of instances. In the
context of our embeddings, similarity is defined based on the action sequences
they represent. Let $f(x)$ be the embedding of an instance $x$ after
transformation through matrices $\mathbf{L}_{H}$, $\mathbf{L}_{E}$, and
$\mathbf{L}_{A}$. Given two instances $x_{1}$ and $x_{2}$, their similarity
$S$ can be defined as:

$S(x_{1},x_{2})=\cos(f(x_{1}),f(x_{2}))=\frac{f(x_{1})\cdot
f(x_{2})}{\|f(x_{1})\|_{2}\|f(x_{2})\|_{2}}$ (24)

Initialization of the Weight Matrices

Here we have employed Glorot (Xavier) initialization:

$l_{ij}\sim\text{Uniform}\left(-\sqrt{\frac{6}{n_{in}+n_{out}}},\sqrt{\frac{6}{n_{in}+n_{out}}}\right)$
(25)

Where $n_{in}$ is the number of features of the input instance and $n_{out}$
is the number of features of the output instance.

Any bias terms are initialized as:

$b_{i}=0$ (26)

Contrastive Loss Mechanism

The contrastive loss, $L_{C}$, can be formulated as:

$L_{C}(x_{1},x_{2})=y\cdot d^{2}+(1-y)\cdot\max(0,m-d)^{2}$ (27)

Where:

\- $d$ is the Euclidean distance between embeddings $f(x_{1})$ and $f(x_{2})$.

\- $y$ is 1 if $x_{1}$ and $x_{2}$ are similar (i.e., from the same action
class) and 0 otherwise.

\- $m$ is a margin that ensures dissimilar pairs are separated by at least
this value.

Training Procedure

The learning process can be described by the following stages in summary:

  1. 1.

Pair Formulation: Extract pairs of instances from the dataset. Associate each
pair with a binary similarity label, where $1$ indicates a similar action
sequence and $0$ otherwise. This label aids in supervising the embedding
learning process.

  2. 2.

Embedding Extraction via Siamese Network: For each instance in the formulated
pair, propagate it through the Siamese Network’s architecture. The output of
this process is a dense embedding vector that captures the underlying action
semantics of the instance.

  3. 3.

Contrastive Loss Computation: Utilize the embeddings from the previous step to
compute the contrastive loss for each pair. This loss quantifies the degree to
which the embeddings accurately represent the similarity or dissimilarity
between instances in the pair, as indicated by the associated label.

  4. 4.

Gradient-Based Optimization of Transformation Matrices: Utilizing the computed
contrastive loss, perform backpropagation to compute the gradients with
respect to the transformation matrices $\mathbf{L}_{H}$, $\mathbf{L}_{E}$, and
$\mathbf{L}_{A}$. Subsequently, adopt an optimization algorithm, the Adam
optimizer, to update these matrices in a manner that minimizes the contrastive
loss. This iterative optimization process refines the matrices to yield more
expressive and accurate embeddings for action sequences.

Optimization and Regularization

Our optimization objective, $O_{\mathbf{L}}$, is:

$O_{\mathbf{L}}=\text{MSE}_{\mathbf{H}}+\text{MSE}_{\mathbf{E}}+\text{MSE}_{\mathbf{A}}+\lambda\cdot
R$ (28)

Where:

\- $R$ is the regularization term.

\- $\lambda$ is the regularization coefficient, set at 0.01 to prevent
overfitting.

We employ the Adam optimizer with a learning rate of 0.001 for minimizing
$O_{\mathbf{L}}$. Training is executed over 100 epochs with batches of 32.

Cross-Validation for Robust Learning