To gauge the robustness of the learning mechanism, a k-fold cross-validation
strategy is used. The dataset is sectioned into five segments. Each subset
alternately serves as the validation set, while the rest contribute to
training. Final performance is ascertained as an average over all iterations.
We assume $k=3$ in large datasets and $k=5$ and $k=10$ for moderate and small
datasets, respectively.

####  2.1.2 Learning Process for the Graph Attention Network (GAT) Parameters

The hierarchical framework of our system necessitates a methodical approach to
the learning process, ensuring optimal convergence and model efficiency.
Within this multi-layered architecture, each stage introduces specific
learnable parameters. These are crucial for discerning intricate relationships
within video frames and, consequently, for achieving precise action
recognition.

Structured Learning Paradigm for GAT:

Here we also use staged training:

  1. 1.

Object-Level GAT Training: Initiates by forming a base, ensuring recognition
of fundamental object relationships.

  2. 2.

Single Hand Action-Level GAT Training: Builds on the object-level GAT’s
weights, refining recognition of single-hand actions.

  3. 3.

Bimanual Action-Level GAT Training: Progresses with insights from the single-
hand GAT, concentrating on dual-hand coordinated actions.

A core challenge within our framework is the lack of direct ground truth
labels for the GAT layers. This stems from the model’s aim of unraveling
complex spatio-temporal relationships, leading to overarching action
categorizations.

Supervised Learning with Parameter Freezing:

To counter the label deficit, we employ supervised learning, using final
action labels as reference points. This entails freezing subsequent layer
parameters, concentrating solely on the active GAT layer, thereby maintaining
the holistic model context without unnecessary simultaneous adjustments.

##### Optimization Strategy for GAT Layers

To ensure efficient and effective learning of the GAT parameters, we adopt the
following strategies:

  * •

Loss Function: We employ the cross-entropy loss, given the classification
nature of action recognition, given by (with $C$ classes):

$L_{\text{CE}}=-\sum_{i=1}^{N}\sum_{c=1}^{C}y_{i,c}\log(p_{i,c})$ (29)

where $N$ is the number of samples, $y_{i,c}$ is a binary indicator for the
correct classification of sample $i$ to class $c$, and $p_{i,c}$ is the
predicted probability of sample $i$ belonging to class $c$.

  * •

Learning Rate: We use an initial learning rate of $0.001$ with the Adam
optimizer. If the validation loss remains stagnant for 10 epochs, the rate is
halved.

  * •

Batch Training: Due to dataset intricacies, mini-batch training is employed
with a batch size of $32$.

  * •

Regularization: To prevent overfitting and enhance model generalization, we
apply L2 regularization with a $0.0001$ coefficient to the weights and a
dropout rate of $0.5$ to attention scores.

  * •

Early Stopping: If the validation loss does not improve for $20$ consecutive
epochs, training is halted, retaining the parameters from the epoch with the
least loss.

  * •

Cross-Validation: For hyperparameter tuning, k-fold cross-validation
techniques are incorporated. We assume $k=5$ in large datasets and $k=10$ for
moderate and small datasets.

####  2.1.3 Deriving the learnable parameters in the GAT layers

Next we describe details how to arrive at the different learnable parameters
in the three GAT layers.

Level 1: Object-Level GAT

  * •

$\mathbf{W}^{(1)}$: Transformation matrix that linearly maps node features
from their original space to a new representation in Layer 1.
Initialization: By the Xavier method.

  * •

$\mathbf{d}^{(1)}$: Learnable parameter that determines the importance of node
and its neighbors in the attention mechanism at Layer 1.
Initialization: Small random values.

  * •

$\mathbf{W}_{e}$: Weight matrix that transforms the edge features. It
encapsulates spatial and relational dependencies between nodes. It is shared
across all three layers.

  * •

$\mathbf{W}_{a}$: Weight matrix responsible for transforming action-edge
features. It captures the essence of the actions performed between objects. It
is shared across all three layers.

Initialization of $\mathbf{W}_{e}$ and $\mathbf{W}_{a}$: Given the non-linear
activations in GATs, particularly the LeakyReLU activations, an initialization
method tailored to this type of activation function is desired. We adopt the
He Initialization (also known as Kaiming Initialization) method for
initializing $W_{a}$ and $W_{e}$. This method is specifically designed for
ReLU-based activation functions, including LeakyReLU. The key idea behind He
Initialization is to draw the initial weights from a distribution with a mean
of 0 and a variance of $\frac{2}{n_{\text{in}}}$, where $n_{\text{in}}$
represents the number of input units to the layer. Mathematically, the
initialization can be represented as:

$W_{a},W_{e}\sim\mathcal{N}\left(0,\sqrt{\frac{2}{n_{\text{in}}}}\right).$

This initialization approach ensures that the model does not start with
activations and gradients that are excessively small or large, thus promoting
efficient gradient flow and convergence during training.

Level 2: Single Hand Action-Level GAT

  * •

$\mathbf{W}^{(2)}$: Transformation matrix for node features at Layer 2,
refining the representations based on the outputs of Layer 1.
Initialization: Gaussian distribution with mean $0$ and standard deviation
$0.01$.

  * •

$\mathbf{U}^{(2)}$: Learnable weight matrix specific to the second GAT layer,
capturing complex relationships between nodes.
Initialization: Gaussian distribution with mean $0$ and standard deviation
$0.01$.

  * •

$\mathbf{d}^{(2)}$: Learnable parameter that refines the attention mechanism,
focusing on single-hand actions between nodes.
Initialization: Small random values from a uniform distribution.

Level 3: Bimanual Action-Level GAT

  * •

$\mathbf{W}^{(3)}$: Transformation matrix for node features at Layer 3, which
focuses on refining node representations considering bimanual actions.
Given that this layer is even deeper, the Xavier method is employed, suitable
for layers with tanh or sigmoid activations.

  * •

$\mathbf{U}^{(3)}$: Weight matrix at Layer 3 that captures the intricacies of
bimanual interactions in the graph.
Given that this layer is even deeper, the Xavier method is employed, suitable
for layers with tanh or sigmoid activations.

  * •

$\mathbf{d}^{(3)}$: Determines the attention scores for Layer 3, emphasizing
bimanual interactions.
Initialization: Using a Gaussian distribution with mean $0$ and standard
deviation $0.01$.

Justification for Sharing $\mathbf{W}_{e}$ and $\mathbf{W}_{a}$ Across Levels

Choosing to share $\mathbf{W}_{e}$ and $\mathbf{W}_{a}$ across the layers is
guided by the following considerations:

  * •

Parameter Efficiency: Sharing the weights reduces the total number of model
parameters. This not only makes the model computationally more efficient but
also reduces the risk of overfitting, especially when there’s limited training
data.

  * •

Consistency: Using shared transformation weights for edge and action-edge
features ensures a consistent representation across layers. This can be
particularly useful if the fundamental nature of these relationships doesn’t
change across layers, even though their context or interpretation might.

  * •

Regularization: Sharing parameters acts as a form of implicit regularization.
Instead of letting each layer learn its own transformation, which can lead to
overfitting, sharing forces the model to find a general transformation that
works well across all layers.

  * •

Simplification: A model with fewer parameters is simpler and can be more
interpretable. It is easier to understand and diagnose the transformations
applied by the model when the same transformation matrices $\mathbf{W}_{e}$
and $\mathbf{W}_{a}$ are used across layers.

####  2.1.4 Learning Process for the TCN-Based Spatio-Temporal Parameters

In our system, the TCN learning process ensures the capture of intricate time-
dependent characteristics embedded within video frames.

TCN Learnable Parameters:

  * •

$\mathbf{K}^{(l)}$: Convolutional kernel at layer $l$.

  * •

$\vec{b}^{(l)}$: Bias term for the convolution at layer $l$.

  * •

$\mathbf{V}^{(l)}$: Dilation rate for the convolutional kernel at layer $l$.

Structured Learning Paradigm:

Here we use the principle of progressive dilation, which ensures that temporal
patterns across various scales are captured accurately:

  1. 1.

Short-Term Temporal Dependencies: By concentrating on immediate temporal
relationships, this level discerns swift actions or alterations.

  2. 2.

Mid-Term Temporal Dependencies: This stage augments the previous one by
broadening the temporal horizon, allowing for an extended field of view.

  3. 3.

Long-Term Temporal Dependencies: With a larger receptive field, this level
identifies prolonged actions or evolving sequences in the scene.

Backpropagation Through Time (BPTT): Given the sequential nature of the TCN,
BPTT is pivotal for model weight updates, ensuring that the learning process
acknowledges dependencies spanning across different time instances.

Optimization Strategy for TCN Layers

  * •

Loss Function: The Mean Squared Error (MSE) loss, suited for the regression
character of temporal sequences, is utilized:

$L_{\text{MSE}}=\frac{1}{N}\sum_{i=1}^{N}(y_{i}-\hat{y}_{i})^{2}$ (30)

where $N$ signifies the sample count, $y_{i}$ represents the true value, and
$\hat{y}_{i}$ is the predicted counterpart.

  * •

Learning Rate: An Adam optimizer is used, with an initial learning rate set to
$0.001$. If the validation loss stagnates for 5 consecutive epochs, the
learning rate is reduced by half.

  * •

Batch Training: Mini-batch training, with batches of $64$, addresses the
sequential intricacy and offers computational efficiency.

  * •

Regularization: Dropout, with a rate of $0.2$, is applied after convolution
layer, mitigating the risk of overfitting.

  * •

Early Stopping: The training halts if there’s no validation loss improvement
over $10$ epochs, ensuring the model’s state with the least loss is retained.

Deriving Learnable Parameters in the Initial TCN Layer:

The preliminary layer in the TCN captures immediate temporal nuances, laying a
solid foundation for subsequent layers.

Convolutional Kernel $\mathbf{K}^{(1)}$:

  * •

Initialization: Xavier method.

  * •

Forward Propagation: Convolution operation on the GAT output.

Bias Term $\vec{b}^{(1)}$:

  * •

Initialization: Zeroes.

  * •

Forward Propagation: Incorporated post-convolution, offering an affine shift.

Dilation Rate $\mathbf{V}^{(1)}$:

  * •

Initialization: Set to $1$, ensuring that proximate relationships are
recognized.

  * •

Forward Propagation: Modulates the convolution kernel’s spacing.

Learning Strategy for Deeper TCN Layers:

The ensuing layers in the TCN, building upon the initial layer, increment
their receptive scope to detect longer-lasting temporal dependencies.

Layer 2 and Beyond:

  * •

Exponential enhancement of the dilation rate $\mathbf{V}^{(l)}$, accommodating
expansive temporal durations.

  * •

$\mathbf{K}^{(l)}$ and $\vec{b}^{(l)}$ mirror the learning and initialization
patterns of the first layer but are adapted per their dilation rates.

Conclusively, by employing this stratified approach, our TCN captures spatio-
temporal associations across different temporal magnitudes, thus offering a
holistic video analysis. Furthermore, given the ground truth at the pipeline’s
end and the already optimized GAT parameters, our learning approach ensures
harmonious integration between spatial attention and temporal convolution
mechanisms.

####  2.1.5 Learning Process for the Hierarchical Action Classification Step

Learning Process for Fully Connected Layers

In the step of the framework involving fully connected layers, we aim to
efficiently map the abstract features obtained from previous layers to
actionable labels. While traditionally these layers might involve learning
parameters, our approach focuses on using well-established fully connected
architectures from the literature, without fine-tuning the parameters. The
rationale behind this decision lies in the comprehensive learnable parameters
already present in our framework, ensuring that introducing additional
learning parameters for these layers is not necessary.

To determine the most suitable fully connected architecture for our action
recognition task, we conducted thorough experimentation using various well-
known architectures available in the literature. Specifically, we explored the
performance of two-layer fully connected architectures, each with varying
numbers of neurons. The choice of these architectures was inspired by their
wide usage in related tasks and their simplicity, which aligns with our
framework’s hierarchical structure.

We evaluated the performance of architectures such as LeNet-5, AlexNet,
VGG-16, VGG-19, and ResNet-50, which are renowned for their effectiveness in
various image-related tasks. These architectures come with different
configurations of fully connected layers, including varying numbers of neurons
and layers. Through rigorous experimentation, we found that the architecture
that yielded the best results for our action recognition problem consisted of
two dense layers, featuring 128 neurons in the first layer and 64 neurons in
the second layer.

####  2.1.6 Learning Process for the Description Generation Parameters

The learning process for adapting the pre-trained GPT-2 model to generate
bimanual action descriptions has only been touched in the main text in Section
3.10. Here we provide details of these steps

  1. 1.

Tokenization: We first tokenize the input data, including the object names,
SRs, and action types, into a sequence of tokens that the GPT-2 model can
understand.

  2. 2.

Vectorization: Next, we use vectorization to convert the tokens into fixed-
size numerical vectors.

  3. 3.

Sliding Windows: To handle longer input sentences, we employ a sliding window
approach that divides the input sequence into overlapping segments, where each
segment is of fixed size. The window size is chosen based on the maximum
length of the input sequence that the model can process. If the input sentence
is longer than the fixed-size window, we divide the sentence into overlapping
segments and each segment is used as an input to the model.

Let the length of the input sequence be denoted by $L$, and the window size be
denoted by $W$. Then, we can define the number of windows $N$ as
$N=\lfloor(L-W)/S\rfloor+1$ where $S$ is the stride length, which determines
the degree of overlap between adjacent windows. In our implementation, we set
$S=W/2$ to ensure significant overlap between adjacent windows. For each
window $i$, we extract the corresponding sub-sequence of length $W$ starting
at position $(i-1)\times S$ and use it as input to the model. This way, we
ensure that all the tokens in the input sequence are considered by the model.
The output of the model for each window can be concatenated together to form
the final output for the entire sequence. Values for the differnt varibables
are different for the datasets that we have used and are found in the
Supplementary Material.

  4. 4.

Model Architecture: A generation layer is added on top of the pre-trained
GPT-2 model. The generation layer is responsible for generating the action
descriptions based on the input of object names, SRs, and action types. For
generating descriptions with different levels of detail, we add three separate
layers on top of the pre-trained GPT-2 model, one for each level of detail.

The generation layer consists of three sub-layers, each responsible for
generating descriptions at a different level of detail. The input to each sub-
layer is the output of the previous sub-layer, which allows for the generation
of increasingly detailed descriptions. Let ${r_{1},r_{2},...,r_{n}}$ be the
output of GPT-2, then $Z_{1}=Q_{1}(r_{1},r_{2},...,r_{n})$ is the output of
the first generation layer and $Z_{i+1}=Q_{i+1}(Z_{i})$ defines the output of
the next 2 generation layers accordingly.

  5. 5.

Loss Function: We use the cross-entropy loss function to measure the
difference between the predicted output and the ground truth.

  6. 6.

Optimizer: We use the Adam optimizer to update the model weights based on the
gradients of the loss function with respect to the model parameters.

  7. 7.

Training Data: The annotated dataset containing the input and corresponding
output (i.e., bimanual action descriptions) is used to train the model. The
data is preprocessed and transformed into a format that can be fed into the
model.

###  2.2 Joint Learning and End-to-End Training

In the pursuit of enhancing the synergy between different components of our
video description generation model, we employ a joint learning approach. This
approach aims to capitalize on the interdependencies between specific stages,
allowing them to collaborate more effectively and contribute collectively to
the model’s understanding of video content. By sharing information and
refining features through joint training, we create a comprehensive framework
that can produce more accurate and coherent descriptions.

In this section, we describe our joint learning strategies, which encompass
collaborations between various stages of the video description generation
pipeline. We focus on three distinct joint learning scenarios, each tailored
to optimize the interaction between specific sets of components:

  1. 1.

Joint Training of GAT (Step 3) and TCN (Step 4): Our initial joint learning
phase involves the concurrent training of Graph Attention Networks (GAT) in
Step 3 and Temporal Convolutional Networks (TCN) in Step 4. This combination
capitalizes on both the spatial relationships captured by GAT and the temporal
dynamics captured by TCN. By jointly learning these stages, we promote the
fusion of spatial and temporal features, enhancing the overall representation
of video content.

  2. 2.

Joint Training of Node and Edge Embedding (Step 2), combined with GAT and TCN
steps (steps 3 and 4): We further extend our joint learning to incorporate
node and edge embedding from Step 2 into the procedure. This enables the
fusion of enriched spatial embeddings, graph attention mechanisms, and
temporal features. By simultaneously refining these representations, we pave
the way for more robust and nuanced feature aggregation in subsequent stages.

  3. 3.

End-to-End Training of Node and Edge Embedding (Step 2), GAT (Step 3), TCN
(Step 4), and Description Generation (Step 10): Our final joint learning
scenario encapsulates the essence of the entire video description generation
pipeline. By integrating the node and edge embedding, GAT, and TCN stages with
the description generation step, we enable an end-to-end training approach.
This training strategy allows the model to holistically optimize its feature
extraction, understanding of actions, and narrative generation capabilities.
The interactions cultivated through joint learning enrich the information flow
between different stages, culminating in more coherent and contextually
aligned descriptions.

Every phase of joint learning aims to foster collaboration among specific
components, enhancing their collective performance and, in turn, improving the
overall effectiveness of our video description generation model. In the
following sections, we describe each joint learning process, providing
detailed explanations of how these collaborative efforts are coordinated.

###  2.3 Gradual Joint Learning of GAT (Step 3) and TCN (Step 4)

Starting Phase - Training TCN with Fixed GAT:

  1. 1.

We begin by keeping the GAT component fixed, setting the GAT parameters to
their optimal values obtained during their training process.

  2. 2.

During this time, only the TCN part learns and adjusts. But it benefits from
the information coming from GAT.

Step by Step Unfreezing:

  * •

When the TCN’s learning starts to slow down, we start adjusting the GAT part.

  * •

We begin with the last part of GAT (the one closest to TCN) and let it learn
and adjust.

  * •

As we go on, we allow earlier parts of GAT to adjust too, going backwards
until every part learns.

Learning Together:

  * •

When both GAT and TCN parts can learn, we train them together.

  * •

We use a combined way to check their performance, which considers both the
GAT’s and TCN’s outputs.

$L_{\text{combined}}=\alpha L_{\text{GAT}}+\beta L_{\text{TCN}}$ (31)

where $\alpha$ and $\beta$ are weights we choose.

  * •

We also use techniques like dropout across both parts to make sure they don’t
over-adjust.

Fine-tuning:

  * •

After they have learned together, we do a final round of fine-tuning. This
means we make small adjustments to get even better results.

  * •

We check the model’s performance on a test set regularly and decide when to
stop based on its results.

Tuning Hyperparameters $\alpha$ and $\beta$:

The selection of hyperparameters $\alpha$ and $\beta$ is a crucial aspect of
achieving an effective balance between spatial and temporal learning.

  * •

These hyperparameters were fine-tuned through a methodical grid search
process, systematically exploring various combinations of values.

  * •

We assessed the impact of different $\alpha$ and $\beta$ values on the
validation performance, aiming to optimize the convergence and effectiveness
of joint learning.

  * •

The final chosen values for $\alpha$ and $\beta$ were $\alpha=0.6$ and
$\beta=0.4$, respectively, reflecting a balanced emphasis on both spatial and
temporal learning.

By following these steps, we make sure that the knowledge in GAT is respected
and blended with the new learning from TCN. This way, our model can understand
both space (from GAT) and time (from TCN) in a reasonable way. Here we applied
an initial learning rate of 0.001 and a batch size of 32.

###  2.4 Joint Training of Node and Edge Embedding (Step 2), GAT (Step 3), and
TCN (Step 4)

Initialization and Incorporation of Mixture Knowledge:

  * •

For this phase, we initiate the model parameters by drawing upon the
understanding acquired from the previous joint training of GAT and TCN (Steps
3 and 4). This ensures that the components commence their collaboration with a
foundation enriched by spatial and temporal insights.

  * •

The node and edge embeddings, which encapsulate spatial relationships, are
further augmented by the combined comprehension of dynamic relationships (TCN)
and graph attention (GAT).

Loss Function and Training Objective:

  * •

The core of this joint learning phase lies in an encompassing loss function
that takes into consideration the goals of all three stages: embedding (Step
2), GAT (Step 3), and TCN (Step 4).

  * •

The overarching loss function is defined as:

$L_{\text{joint}}=\alpha L_{\text{Embedding}}+\beta L_{\text{GAT}}+\gamma
L_{\text{TCN}}$ (32)

Here, the hyperparameters $\alpha$, $\beta$, and $\gamma$ play a pivotal role
in dictating the relative significance of each stage’s contribution within the
joint learning process.

Hyperparameter Computation and Optimization:

  * •

The selection of hyperparameters $\alpha$, $\beta$, and $\gamma$ is guided by
the outcomes of the previous joint learning phases, with specific numerical
values.

  * •

Inspired by the favorable results from the joint learning of GAT and TCN, we
chose $\beta=0.4$ and $\gamma=0.3$ as the initial values for $\alpha$ and
$\beta$, respectively.

  * •

To account for the embedding step insights we set the initial value of
$\alpha=0.3$.

  * •

Following these initial values, we explored a grid of hyperparameter
combinations to determine the optimal configuration that maximizes the
collaborative potential of node and edge embedding, graph attention, and
temporal convolution.

  * •

We converged upon the optimal hyperparameters: $\alpha=0.25$, $\beta=0.45$,
and $\gamma=0.3$.

Regularization and Optimization:

  * •

To maintain a balanced learning process and mitigate overfitting, dropout
regularization is uniformly applied across all three stages during the joint
training.

  * •

The optimization strategy involves employing gradient-based methods such as
stochastic gradient descent (SGD) or Adam. The initial learning rates are
informed by the previous joint training phase’s mixture.

  * •

We proactively monitor loss convergence and validation performance to fine-
tune hyperparameters, attaining an optimal equilibrium that harmonizes the
diverse contributions of different stages.

Enriching Feature Fusion:

  * •

The integration of node and edge embedding, GAT, and TCN results in a unified
feature representation that holistically captures spatial, temporal, and
relational intricacies inherent in video data.

  * •

The insights previously garnered from the GAT and TCN collaboration (Steps 3
and 4) continue to guide the learning paths of all three stages. This
synergistic effect amplifies the quality of feature fusion and deepens the
model’s comprehension of video content.

By co-training the node and edge embedding, GAT, and TCN components while
incorporating insights from their previous mixture, we construct a more
interwoven model that capitalizes on spatial, temporal, and relational cues.
This multi-dimensional approach lays the groundwork for subsequent joint
learning phases, further refining the model’s descriptive prowess.

###  2.5 End-to-End Training

In this final phase of joint learning, we integrate the insights distilled
from Steps 2, 3, and 4, with the description generation component (Step 10),
through an end-to-end training approach. This ensures that the entire video
description generation pipeline collaborates cohesively, yielding descriptions
that are coherent, contextually relevant, and accurate.

Initialization and Knowledge Incorporation:

  * •

Parameters initialization: We initialize the model parameters using the
representations learned from the integrated joint learning of Steps 2, 3, and
4. The enriched representations from these steps serve as a solid foundation
for the end-to-end training.

Loss Function and Training Objective:

  * •

Loss Function: The overarching loss function for end-to-end training comprises
the objectives of Steps 2, 3, 4, and 10:

$L_{\text{end-to-end}}=\alpha L_{\text{Embedding}}+\beta L_{\text{GAT}}+\gamma
L_{\text{TCN}}+\delta L_{\text{Description}}$ (33)

Here, $\alpha$, $\beta$, $\gamma$, and $\delta$ are hyperparameters that
control the relative weight of each objective in the training process.

Hyperparameter Selection and Optimization:

  * •

Initial values: The initial values for the hyperparameters $\alpha$, $\beta$,
$\gamma$, and $\delta$ were chosen based on insights from previous joint
learning phases. We set $\alpha=0.1$, $\beta=0.3$, $\gamma=0.2$, and
$\delta=0.4$, prioritizing a slightly stronger influence from the GAT.

  * •

Influence of Step 10: Given that Step 10 represents the final stage of our
end-to-end approach, we assign a higher weight to $\delta$ to prioritize the
description generation process.

  * •

Optimal hyperparameters: Through grid search, the final optimal
hyperparameters were determined as $\alpha=0.05$, $\beta=0.35$, $\gamma=0.25$,
and $\delta=0.35$. These values reflect a balance between the contributions of
embedding, GAT, TCN, and description generation.

Regularization and Optimization:

  * •

Dropout regularization: Dropout with a rae of $0.2$ is applied to all model
components to prevent overfitting.

  * •

Optimization algorithm: We utilize gradient-based optimization algorithms
(Adam). The initial learning rates are informed by the joint learning phases
and start with $0.001$.

  * •

Learning rate adjustments: Monitor the training progress and validation loss.
If the validation loss stagnates for a certain number of epochs, reduce the
learning rate by half to prevent overshooting.

Validation and Convergence:

  * •

Validation set: Regularly assess the model’s performance on a dedicated
validation dataset during training.

  * •

Early stopping: Implement an early stopping mechanism. If the validation loss
does not improve over 10 epochs, halt the training to prevent overfitting and
retain the best model state.

###  2.6 Computing Hand Groups

Hand Spatial Relations: To compute the hand spatial relationship category for
a bimanual action, we start by extracting the spatial coordinates of the hands
for each frame in the video using a hand detection and tracking algorithm
[94]. Let $p_{1}$ and $p_{2}$ be the 3D coordinates of the left and right hand
centers, respectively, and let $d$ be the Euclidean distance between $p_{1}$
and $p_{2}$.

We define the hand spatial relationship category based on the following
thresholds:

  * •

Close-hand: $d<d_{c}$

  * •

Crossed-hand: $d_{c}\leq d<d_{s}$

  * •

Stacked-hand: $d\geq d_{s}$

Here, $d_{c}$ and $d_{s}$ are the thresholds for close-hand and stacked-hand,
respectively. These thresholds can be computed based on the characteristics of
the dataset, such as the average hand span or the maximum distance between the
hands in the dataset.

To determine the appropriate threshold values, we analyze the distribution of
hand distances in the dataset and choose values that best distinguish between
the different hand spatial relationships. For example, if the average hand
span is 20 cm, we may set $d_{c}$ to 5 cm and $d_{s}$ to 15 cm.

Level of precision category: To compute the level of precision category for a
bimanual action, we first extract the types of objects and actions involved in
the action using object recognition and action recognition algorithms. We then
define a precision score $s_{p}$ for each action based on the level of
precision required to perform it. Specifically, the precision score is
computed as follows (using an example to explain this):

Let $d_{min}$ and $d_{max}$ be the minimum and maximum distance between the
knife and the vegetables during the chopping action, respectively. We define
the following thresholds to determine the precision score:

  * •

Low precision: $d_{max}-d_{min}<d_{lp}$

  * •

Medium precision: $d_{lp}\leq d_{max}-d_{min}<d_{mp}$

  * •

High precision: $d_{max}-d_{min}\geq d_{mp}$

Here, $d_{lp}$ and $d_{mp}$ are the thresholds for low precision and medium
precision, respectively. One method to derive these thresholds is to analyze
the dataset and determine the minimum and maximum distances between the
objects involved in the bimanual actions. The difference between those can
then be utilized to define the range of distances that correspond to low,
medium and high precision actions, respectively. This data-driven approach
provides a quantitative way to determine the thresholds based on the level of
precision required for the actions in the dataset.

The hand spatial relationship and level of precision categories can then be
combined with the symmetric/asymmetric and coordinated/uncoordinated
categories from [67] to form the complete bimanual action type.

##  3 Hierarchical Action Breakdown

One of the prominent challenges is the hierarchical nature of actions, where a
broad action category might be decomposed into multiple sub-levels, each
offering finer granularity. While the depth of action categorization can span
numerous nested levels, for the purposes of this breakdown, we have used a
maximum of five levels. It is pertinent to understand that many of our
datasets can be dissected into even finer categorizations, extending beyond
the five levels highlighted here. However, to create a balance between
comprehensive understanding and readability, we have prioritized certain
actions over others, focusing on those that best exemplify the dataset’s
essence.

###  3.1 Learning Processes

Each feature matrix $\mathbf{G}^{(t,j,k,o)}$ undergoes a series of fully
connected layers followed by a softmax function. The predicted action label
for the $t$-th GAT layer, action category $j$, sublevel $k$, and item $o$ is
represented as $\hat{y}^{(t,j,k,o)}=\arg\max_{j}(P^{(t,j,k,o)})$. The
classification process is trained using the cross-entropy loss between the
predicted action probabilities and the ground truth action labels is
minimized. This loss calculation involves all GAT layers, action categories,
sublevels, and items:

$L(y,\hat{y})=-\sum_{t=1}^{L}\sum_{j=1}^{N}\sum_{k=1}^{M}\sum_{o=1}^{O}y^{(t,j,k,o)}\log{\hat{y}^{(t,j,k,o)}}$
(34)

Where $y^{(t,j,k,o)}$ is the ground truth probability for item $o$ of sublevel
$k$ of action category $j$ in the $t$-th GAT layer, and $\hat{y}^{(t,j,k,o)}$
is the predicted probability.

If an item or level does not exist within a certain sublevel, it can be
denoted with a placeholder such as ’…’ to indicate the absence of that item or
level, and its related probability(s) will be considered as $0$.

The fully connected layers are responsible for mapping the abstract features
from GAT outputs to actionable labels. These layers encompass two dense
layers, with 128 and 64 neurons respectively. ReLU activations follow the
linear transformations, leading to the final layer that corresponds to the
number of items. The softmax activation ensures a probability distribution
over items.

The outcome of the classification process offers action predictions across GAT
layers, action categories, sublevels, and items. These predictions can be
harnessed to generate descriptive sentences at varying levels of detail,
providing a comprehensive depiction of actions in the video.

It is noteworthy that the probability distribution of each action category at
a GAT layer serves as input to the fully connected layers of the subsequent
action category. This hierarchical arrangement enables action recognition
across multiple levels of detail and GAT layers.

###  3.2 Action Categories

Note that the complete list of action categories is quite extensive. Thus, we
have chosen to present here only a few illustrative instances, offering an
insight into the inherent intricate hierarchy of the datasets.

I. Meal Preparation (Level 1)

  * •

Setting the Scene (Level 2)

    * –

Organizing Workspace (Level 3)

      * @itemiii

Retrieving Tools (Level 4)

        * +

Selecting appropriate utensils (Level 5)

        * +

Placing tools on countertop (Level 5)

      * @itemiii

Gathering Ingredients (Level 4)

        * +

Sorting by type (Level 5)

        * +

Organizing in order of use (Level 5)

  * •

Ingredient Manipulation (Level 2)

    * –

Texture Alteration (Level 3)

      * @itemiii

Cutting (Level 4)

        * +

Selecting knife type (Level 5)

        * +

Chopping motion (Level 5)

      * @itemiii

Peeling (Level 4)

        * +

Holding the peeler (Level 5)

        * +

Removing skin without waste (Level 5)

    * –

Flavor Infusion (Level 3)

      * @itemiii

Marinating (Level 4)

        * +

Mixing marinade components (Level 5)

        * +

Ensuring even coating on ingredient (Level 5)

      * @itemiii

Seasoning (Level 4)

        * +

Selecting spices (Level 5)

        * +

Applying evenly (Level 5)

    * –

Mixing Ingredients (Level 4)

      * @itemiii

Using a hand whisk (Level 5)

      * @itemiii

Using an electric mixer (Level 5)

  * •

Cooking Process (Level 2)

    * –

Heat Application (Level 3)

      * @itemiii

Baking (Level 4)

        * +

Preheating oven (Level 5)

        * +

Monitoring cooking time (Level 5)

      * @itemiii

Frying (Level 4)

        * +

Selecting oil type (Level 5)

        * +

Regulating heat level (Level 5)

      * @itemiii

Boiling (Level 4)

        * +

Filling pot with water (Level 5)

        * +

Adjusting stove temperature (Level 5)

    * –

Dough Manipulation (Level 3)

      * @itemiii

Kneading (Level 4)

        * +

Using hands for manual kneading (Level 5)

        * +

Using a kneading machine (Level 5)

      * @itemiii

Rolling (Level 4)

        * +

Choosing a rolling pin (Level 5)

        * +

Applying even pressure (Level 5)

  * •

Plating & Serving (Level 2)

    * –

Presentation (Level 3)

      * @itemiii

Garnishing (Level 4)

        * +

Selecting garnish type (Level 5)

        * +

Placing attractively on dish (Level 5)

      * @itemiii

Portioning (Level 4)

        * +

Using serving tools (Level 5)

        * +

Allocating even servings (Level 5)

      * @itemiii

Arrangement (Level 4)

        * +

Designing plate layout (Level 5)

        * +

Adjusting for visual appeal (Level 5)

  * •

Cleanup & Storage (Level 2)

    * –

Storage (Level 3)

      * @itemiii

Refrigerating (Level 4)

        * +

Setting correct temperature (Level 5)

        * +

Allocating space for dishes (Level 5)

      * @itemiii

Freezing (Level 4)

        * +

Sealing food in containers (Level 5)

        * +

Labeling with dates and names (Level 5)

    * –

Cleaning (Level 3)

      * @itemiii

Dishwashing (Level 4)

        * +

Pre-rinsing dishes (Level 5)

        * +

Using appropriate soap quantity (Level 5)

      * @itemiii

Wiping Countertops (Level 4)

        * +

Selecting cleaning agent (Level 5)

        * +

Ensuring no residue remains (Level 5)

II: Assembly (Level 1)

  * •

Assembling Wooden Pieces (Level 2)

    * –

Placing wooden pieces (Level 3)

    * –

Joining pieces with nails and hammers (Level 3)

      * @itemiii

Hammering nails into wood (Level 4)

        * +

Striking nail with hammer to penetrate wood (Level 5)

      * @itemiii

Attaching second piece of wood (Level 4)

        * +

Placing second piece on top of the first (Level 5)

III: Painting a Wall (Level 1)

  * •

Applying Paint (Level 2)

    * –

Preparing Paint and Supplies (Level 3)

      * @itemiii

Opening paint can (Level 4)

      * @itemiii

Mixing paint thoroughly (Level 4)

      * @itemiii

Getting paintbrush and tray (Level 4)

    * –

Applying Paint to Wall (Level 3)

      * @itemiii

Dipping brush in paint (Level 4)

      * @itemiii

Spreading paint on wall surface (Level 4)

      * @itemiii

Using roller for larger areas (Level 4)

    * –

Achieving Desired Finish (Level 3)

      * @itemiii

Applying additional coats (Level 4)

      * @itemiii

Checking for uniform coverage (Level 4)

  * •

Cleanup and Finishing (Level 2)

    * –

Cleaning Tools (Level 3)

      * @itemiii

Cleaning paintbrush (Level 4)

      * @itemiii

Cleaning paint tray and roller (Level 4)

      * @itemiii

Sealing paint can (Level 4)

IV: Juicing an Orange (Level 1)

  * •

Extracting Juice (Level 2)

    * –

Preparing Orange (Level 3)

      * @itemiii

Selecting a ripe orange (Level 4)

        * +

Rubbing the orange for texture checking (Level 5)

      * @itemiii

Washing the orange (Level 4)

        * +

Rinsing under water (Level 5)

        * +

Drying with a cloth (Level 5)

    * –

Cutting and Preparing (Level 3)

      * @itemiii

Cutting the orange in half (Level 4)

        * +

Using a sharp knife (Level 5)

        * +

Placing cut side up (Level 5)

      * @itemiii

Removing seeds (Level 4)

        * +

Scooping out seeds with a spoon (Level 5)

    * –

Using a juicer (Level 3)

      * @itemiii

Using a manual juicer (Level 4)

        * +

Placing orange half on juicer (Level 5)

        * +

Twisting (Level 5)

      * @itemiii

Squeezing the orange by hand (Level 4)

        * +

Using both hands to squeeze (Level 5)

        * +

Pouring juice into a container (Level 5)

  * •

Serving (Level 2)

    * –

Straining the Juice (Level 3)

      * @itemiii

Using a fine mesh strainer (Level 4)

        * +

Holding strainer over a glass (Level 5)

        * +

Pouring juice through strainer (Level 5)

    * –

Presentation (Level 3)

      * @itemiii

Pouring the fresh juice into a glass (Level 4)

      * @itemiii

Garnishing with orange slices (Level 4)

        * +

Cutting thin slices from an orange (Level 5)

        * +

Placing slices on the rim of the glass (Level 5)

  * •

Cleaning Up (Level 2)

    * –

Cleaning Equipment (Level 3)

      * @itemiii

Washing the juicer (Level 4)

        * +

Disassembling juicer parts (Level 5)

        * +

Scrubbing with soap and water (Level 5)

      * @itemiii

Rinsing the strainer (Level 4)

        * +

Towel drying (Level 5)

    * –

Wiping Surfaces (Level 3)

      * @itemiii

Cleaning the countertop (Level 4)

        * +

Drying the surface with a clean cloth (Level 5)

LARGEReferences

## References

  * [1] J. R. Kwapisz, G. M. Weiss, S. A. Moore, Activity recognition using cell phone accelerometers, ACM SigKDD Explorations Newsletter 12 (2) (2011) 74–82.
  * [2] F. J. Ordonez, G. Englebienne, P. De Toledo, T. Van Kasteren, A. Sanchis, B. Kröse, In-home activity recognition: Bayesian inference for hidden markov models, IEEE Pervasive Computing 13 (3) (2014) 67–75.
  * [3] R. Poppe, A survey on vision-based human action recognition, Image and Vision Computing 28 (6) (2010) 976–990.
  * [4] V. Aggarwal, C. K. Reddy, Human activity recognition: A survey, Computer Vision and Image Understanding 131 (2015) 3–33.
  * [5] C. Liu, J. Ying, H. Yang, X. Hu, J. Liu, Improved human action recognition approach based on two-stream convolutional neural network model, The visual computer 37 (2021) 1327–1341.
  * [6] Q. Xiong, J. Zhang, P. Wang, D. Liu, R. X. Gao, Transferable two-stream convolutional neural network for human action recognition, Journal of Manufacturing Systems 56 (2020) 605–614.
  * [7] H. Yang, C. Yuan, B. Li, Y. Du, J. Xing, W. Hu, S. J. Maybank, Asymmetric 3d convolutional neural networks for action recognition, Pattern recognition 85 (2019) 1–12.
  * [8] J. Arunnehru, G. Chamundeeswari, S. P. Bharathi, Human action recognition using 3d convolutional neural networks with 3d motion cuboids in surveillance videos, Procedia computer science 133 (2018) 471–477.
  * [9] G. Chéron, I. Laptev, C. Schmid, P-cnn: Pose-based cnn features for action recognition, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 3218–3226.
  * [10] H. Ma, W. Li, X. Zhang, S. Gao, S. Lu, Attnsense: Multi-level attention mechanism for multimodal human activity recognition., in: IJCAI, 2019, pp. 3109–3115.
  * [11] X. Yin, Z. Liu, D. Liu, X. Ren, A novel cnn-based bi-lstm parallel model with attention mechanism for human activity recognition with noisy data, Scientific Reports 12 (1) (2022) 7878.
  * [12] A. Billard, D. Kragic, Trends and challenges in robot manipulation, Science 364 (6446) (2019) eaat8414.
  * [13] A. Thomaz, G. Hoffman, M. Cakmak, Computational human-robot interaction, Foundations and TrendsÂ® in Robotics 4 (2-3) (2016) 105–223.
  * [14] M. Oberweger, P. Wohlhart, V. Lepetit, Hands deep in deep learning for hand pose estimation, arXiv preprint arXiv:1502.06807.
  * [15] T. Chatzis, X. Yuan, A. A. Argyros, A comprehensive study on deep learning-based 3d hand pose estimation methods, Applied Sciences 10 (19) (2020) 6850.
  * [16] E. A. Franz, Bimanual action representation: A window to human evolution, Taking action: Cognitive neuroscience perspectives on intentional acts (2003) 259–88.
  * [17] D. Rakita, A. Grinshpun, O. Brock, Shared control–based bimanual robot manipulation, Science Robotics 4 (30) (2019) eaaw0955.
  * [18] F. Xie, J. Hwangbo, H. Lee, J. Oh, J. Hwang, I. Lee, H.-W. Park, S. Lee, Deep imitation learning for bimanual robotic manipulation, in: Advances in Neural Information Processing Systems, Vol. 33, 2020, pp. 2327–2337.
  * [19] L. P. Ureche, A. Billard, Constraints extraction from asymmetrical bimanual tasks and their use in coordinated behavior, Robotics and Autonomous Systems 103 (2018) 222–235.
  * [20] Z. J. Hu, R. Xu, C. Lu, H. Ding, H. Qiao, H. Ding, Towards human-robot collaborative surgery: Trajectory and strategy learning in bimanual peg transfer, IEEE Robotics and Automation Letters.
  * [21] E. Galofaro, E. D’Antonio, N. Lotti, F. Patané, M. Casadio, L. Masia, Bimanual motor strategies and handedness role in human-robot haptic interaction, IEEE Transactions on Haptics.
  * [22] E. E. Aksoy, A. Abramov, F. Wörgötter, B. Dellen, Categorizing object-action relations from semantic scene graphs, in: 2010 IEEE International Conference on Robotics and Automation, IEEE, 2010, pp. 398–405.
  * [23] J. Wu, L. Wang, L. Wang, J. Guo, G. Wu, Learning actor relation graphs for group activity recognition, in: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 2019, pp. 9964–9974.
  * [24] J. Ji, R. Krishna, L. Fei-Fei, J. C. Niebles, Action genome: Actions as compositions of spatio-temporal scene graphs, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10236–10247.
  * [25] D. Xu, Y. Zhu, C. Choy, L. Fei-Fei, Scene graph generation by iterative message passing, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3097–3106.
  * [26] J. Yang, J. Lu, S. Lee, D. Batra, D. Parikh, J. Chen, Graph r-cnn for scene graph generation, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 774–790.
  * [27] P. Xu, X. Liang, L. Li, L. Zhang, H. Huang, A survey of scene graph: Generation and application, IEEE Transactions on Neural Networks and Learning Systems 31 (1) (2020) 1–19.
  * [28] Y. Teng, L. Wang, Structured sparse r-cnn for direct scene graph generation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [29] W. Wang, D. Li, J. Zhu, L. Tian, Y. Shan, Exploring context and visual pattern of relationship for scene graph generation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  * [30] Y. Cong, W. Liao, H. Ackermann, B. Rosenhahn, M. Y. Yang, Spatial-temporal transformer for dynamic scene graph generation, in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 16372–16382.
  * [31] S. Feng, H. Mostafa, M. Nassar, S. Majumdar, S. Tripathi, Exploiting long-term dependencies for generating dynamic scene graphs, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 5130–5139.
  * [32] A. Airin, R. U. Dawla, A. S. Noor, M. Al Hasan, A. R. Hasan, A. Zaman, D. M. Farid, Attention-based scene graph generation: A review, in: 2022 14th International Conference on Software, Knowledge, Information Management and Applications (SKIMA), IEEE, 2022, pp. 210–215.
  * [33] E. Nguyen, T. Bui, V. Swaminathan, J. Collomosse, Oscar-net: Object-centric scene graph attention for image attribution, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14499–14508.
  * [34] M. Khademi, O. Schulte, Deep generative probabilistic graph neural networks for scene graph generation, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, 2020, pp. 11237–11245.
  * [35] R. Li, S. Zhang, B. Wan, X. He, Bipartite graph network with adaptive message passing for unbiased scene graph generation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 11109–11119.
  * [36] Z. Ravichandran, L. Peng, N. Hughes, J. D. Griffith, L. Carlone, Hierarchical representations and explicit memory: Learning effective navigation policies on 3d scene graphs using graph neural networks, in: 2022 International Conference on Robotics and Automation (ICRA), IEEE, 2022, pp. 9272–9279.
  * [37] N. Xu, A.-A. Liu, J. Liu, W. Nie, Y. Su, Scene graph captioner: Image captioning based on structural visual representation, Journal of Visual Communication and Image Representation 58 (2019) 477–485.
  * [38] K. Ye, A. Kovashka, Linguistic structures as weak supervision for visual scene graph generation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 8289–8299.
  * [39] C. Chen, Q. Dou, H. Chen, J. Qin, P.-A. Heng, Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation, in: Proceedings of the AAAI conference on artificial intelligence, Vol. 33, 2019, pp. 865–872.
  * [40] A. Kojima, T. Tamura, K. Fukunaga, Natural language description of human activities from video images based on concept hierarchy of actions, International Journal of Computer Vision 50 (2) (2002) 171–184.
  * [41] Nishida, S. Takamatsu, Japanese-english translation through internal expressions, in: Proceedings of the 9th conference on Computational linguistics-Volume 1, Academia Praha, 1982, pp. 271–276.
  * [42] F. Nishida, S. Takamatsu, T. Tani, T. Doi, Feedback of correcting information in post editing to a machine translation system, in: Proceedings of the 12th conference on Computational linguistics-Volume 2, ACL, 1988, pp. 476–481.
  * [43] M. W. Lee, A. Hakeem, N. Haering, S. Zhu, Save: A framework for semantic annotation of visual events, in: IEEE Computer Society Conference on CVPR Workshops, 2008, pp. 1–8.
  * [44] Q. Abbas, M. E. Ibrahim, M. A. Jaffar, Video scene analysis: An overview and challenges on deep learning algorithms, Multimedia Tools and Applications 77 (16) (2018) 20415–20453.
  * [45] C. Hori, R. Yamamoto, J. Kim, T. Hori, B. Harsham, J. R. Hershey, Attention-based multimodal fusion for video description, in: Proceedings of the IEEE International Conference on Computer Vision, 2017.
  * [46] C. Wu, L. Wang, W. Guo, W. Liu, Hierarchical attention-based multimodal fusion for video captioning, Neurocomputing 315 (2018) 362–370.
  * [47] S. Chen, T. Yao, Y.-G. Jiang, Deep learning for video captioning: A review, in: IJCAI, Vol. 1, 2019.
  * [48] S. Islam, N. D. Roy, S. Majumder, G. Saha, Exploring video captioning techniques: A comprehensive survey on deep learning methods, SN Computer Science 2 (2) (2021) 1–28.
  * [49] J. Xu, T. Mei, T. Yao, Y. Rui, M. Dai, Learning multimodal attention lstm networks for video captioning, in: Proceedings of the 25th ACM International Conference on Multimedia, ACM, 2017, pp. 1008–1016.
  * [50] L. Zhou, T. Yao, A. Torabi, M. Paluri, J. Li, L. Fei-Fei, End-to-end dense video captioning with masked transformer, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8739–8748.
  * [51] M. Chen, X. Xu, Q. Liu, X. Jin, H. Sun, Tvt: Two-view transformer network for video captioning, in: Asian Conference on Machine Learning, PMLR, 2018, pp. 785–800.
  * [52] J. Lei, Z. Qi, H. Zhang, M.-M. Cheng, Mart: Memory-augmented recurrent transformer for coherent video paragraph captioning, arXiv preprint arXiv:2005.05402.
  * [53] K. Lin, L. Li, F. Yu, J. Chen, M.-H. Yang, B. Zhou, Swinbert: End-to-end transformers with sparse attention for video captioning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [54] P. H. Seo, B. Zhou, Y. Cui, M. Cho, S. Kim, S. J. Oh, I. S. Kweon, End-to-end generative pretraining for multimodal video captioning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [55] A. Yang, A. Zhang, J. Gu, X. Li, Y. Chen, X. Dong, L. Ma, M.-H. Yang, Vid2seq: Large-scale pretraining of a visual language model for dense video captioning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  * [56] V. Iashin, E. Rahtu, A better use of audio-visual cues: Dense video captioning with bi-modal transformer, arXiv preprint arXiv:2005.08271.
  * [57] V. Iashin, E. Rahtu, Multi-modal dense video captioning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020\.
  * [58] A. Nagrani, H.-Y. Lee, A. Zisserman, G. Trigeorgis, Learning audio-video modalities from image captions, in: European Conference on Computer Vision, Springer, 2022.
  * [59] J. Redmon, A. Farhadi, Yolov3: An incremental improvement, arXiv preprint arXiv:1804.02767.
  * [60] Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, Y. Sheikh, Openpose: realtime multi-person 2d pose estimation using part affinity fields, IEEE transactions on pattern analysis and machine intelligence 43 (1) (2021) 172–186.
  * [61] B. Birmingham, A. Muscat, A. Belz, Adding the third dimension to spatial relation detection in 2d images, in: Proceedings of the 11th International Conference on Natural Language Generation, 2018, pp. 146–151.
  * [62] E. E. Aksoy, A. Abramov, J. Dörr, K. Ning, B. Dellen, F. Wörgötter, Learning the semantics of object–action relations by observation, The International Journal of Robotics Research 30 (10) (2011) 1229–1249.
  * [63] F. Ziaeetabar, E. E. Aksoy, F. Wörgötter, M. Tamosiunaite, Semantic analysis of manipulation actions using spatial relations, in: 2017 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2017, pp. 4612–4619.
  * [64] F. Ziaeetabar, T. Kulvicius, M. Tamosiunaite, F. Wörgötter, Recognition and prediction of manipulation actions using enriched semantic event chains, Robotics and Autonomous Systems 110 (2018) 173–188.
  * [65] C. R. G. Dreher, M. Wächter, T. Asfour, Learning object-action relations from bimanual human demonstration using graph networks, IEEE Robotics and Automation Letters (RA-L) 5 (1) (2020) 187–194.
  * [66] T. Mikolov, K. Chen, G. Corrado, J. Dean, Efficient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781.
  * [67] F. Krebs, T. Asfour, A bimanual manipulation taxonomy, IEEE Robotics and Automation Letters 7 (4) (2022) 11031–11038.
  * [68] P. Schramowski, C. Turan, N. Andersen, C. A. Rothkopf, K. Kersting, Large pre-trained language models contain human-like biases of what is right and wrong to do, Nature Machine Intelligence 4 (3) (2022) 258–268.
  * [69] M. Gira, R. Zhang, K. Lee, Debiasing pre-trained language models via efficient fine-tuning, in: Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, 2022, pp. 59–69.
  * [70] X. Zhou, Y. Zhang, L. Cui, D. Huang, Evaluating commonsense in pre-trained language models, in: Proceedings of the AAAI conference on artificial intelligence, Vol. 34, 2020, pp. 9733–9740.
  * [71] Y. Yu, J. Chung, H. Yun, J. Hessel, J. S. Park, X. Lu, R. Zellers, P. Ammanabrolu, R. Le Bras, G. Kim, et al., Fusing pre-trained language models with multimodal prompts through reinforcement learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10845–10856.
  * [72] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, arXiv preprint arXiv:2005.14165.
  * [73] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805.
  * [74] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, arXiv preprint arXiv:1910.10683.
  * [75] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick, Microsoft coco: Common objects in context, in: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, Springer, 2014, pp. 740–755.
  * [76] R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al., The ”something something” video database for learning and evaluating visual common sense, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 5842–5850.
  * [77] T. Wang, T. Yang, M. Danelljan, F. S. Khan, X. Zhang, J. Sun, Learning human-object interaction detection using interaction points, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 4116–4125.
  * [78] B. Kim, J. Lee, J. Kang, E.-S. Kim, H. J. Kim, Hotr: End-to-end human-object interaction detection with transformers, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 74–83.
  * [79] L. Zhou, C. Xu, J. J. Corso, D. Wei, Towards automatic learning of procedures from web instructional videos, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 3692–3701.
  * [80] J. Gao, C. Sun, Z. Yang, R. Nevatia, Tall: Temporal activity localization via language query, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 5267–5276.
  * [81] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, J. Carlos Niebles, Dense-captioning events in videos, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 706–715.
  * [82] M. Regneri, M. Rohrbach, D. Wetzel, S. Thater, B. Schiele, Grounding action descriptions in videos, Transactions of the Association for Computational Linguistics (TACL) 1 (2013) 25–36.
  * [83] Y. Xiang, T. Schmidt, V. Narayanan, D. Fox, Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes, arXiv preprint arXiv:1711.00199.
  * [84] V. Bloom, D. Makris, V. Argyriou, G3d: A gaming action dataset and real time action recognition evaluation framework, in: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, IEEE, 2012.
  * [85] Y. Wen, H. Pan, L. Yang, J. Pan, T. Komura, W. Wang, Hierarchical temporal transformer for 3d hand pose estimation and action recognition from egocentric rgb videos, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 21243–21253.
  * [86] B. Calli, A. Singh, J. Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, A. M. Dollar, Yale-cmu-berkeley dataset for robotic manipulation research, The International Journal of Robotics Research 36 (3) (2017) 261–268.
  * [87] Y. A. Andrade-Ambriz, S. Ledesma, M. A. Ibarra-Manzano, M. I. Oros-Flores, D. L. Almanza-Ojeda, Human activity recognition using temporal convolutional neural network architecture, Expert Systems with Applications 191 (2022) 116287\.
  * [88] G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, A. Gupta, Hollywood in homes: Crowdsourcing data collection for activity understanding, in: European Conference on Computer Vision (ECCV), 2016.
  * [89] H. Xu, A. Das, K. Saenko, R. Doell, J. J. Corso, Msr-vtt: A large video description dataset for bridging video and language, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 5288–5296.
  * [90] D. L. Chen, W. B. Dolan, Collecting highly parallel data for paraphrase evaluation, in: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 2011, pp. 190–200.
  * [91] B. Wang, L. Ma, W. Zhang, X. Chang, M.-H. Yang, Reconstruction network for video captioning, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
  * [92] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, P. H. Torr, Fully-convolutional siamese networks for object tracking, in: Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II 14, Springer, 2016, pp. 850–865.
  * [93] A. v. d. Oord, Y. Li, O. Vinyals, Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03748.
  * [94] Z. Cao, T. Simon, S.-E. Wei, Y. Sheikh, Realtime multi-person 2d pose estimation using part affinity fields, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.