# Hand Guided High Resolution Feature Enhancement for Fine-Grained Atomic
Action Segmentation within Complex Human Assemblies

Matthew Kent Myers
School of Engineering
Newcastle University
<EMAIL_ADDRESS>Nick Wright
School of Engineering
Newcastle University
A. Stephen McGough
School of Computing
Newcastle University
Nicholas Martin
Tharsus Ltd.

###### Abstract

Due to the rapid temporal and fine-grained nature of complex human assembly
atomic actions, traditional action segmentation approaches requiring the
spatial (and often temporal) down sampling of video frames often loose vital
fine-grained spatial and temporal information required for accurate
classification within the manufacturing domain. In order to fully utilise
higher resolution video data (often collected within the manufacturing domain)
and facilitate real time accurate action segmentation – required for human
robot collaboration – we present a novel hand location guided high resolution
feature enhanced model. We also propose a simple yet effective method of
deploying offline trained action recognition models for real time action
segmentation on temporally short fine-grained actions, through the use of
surround sampling while training and temporally aware label cleaning at
inference. We evaluate our model on a novel action segmentation dataset
containing 24 (+background) atomic actions from video data of a real world
robotics assembly production line. Showing both high resolution hand features
as well as traditional frame wide features improve fine-grained atomic action
classification, and that though temporally aware label clearing our model is
capable of surpassing similar encoder/decoder methods, while allowing for real
time classification.

##  1 Introduction

Figure 1: Overview of our action segmentation approach to classifying real
world assembly data. We operate temporally aware label cleaning in an
overlapping sliding window fashion, utilising a temporally aware high
resolution hand feature enhanced frame wide backbone model.

While advances in robotics have made significant progress in automating many
simple and repetitive labour heavy tasks within manufacturing, many complex
manufacturing assemblies requiring assembly adaptability or robustness still
require human operators. Human-Robot Collaboration (HRC) has been identified
as a key component in moving towards a smart assembly procedure in the next
few years [42]. Where humans and robots can work in tandem on assemblies,
leading to increased productivity, reliability, more attractive working
conditions [39], as well create a plethora of quality assurance and guidance
tools (_e.g_. [11, 12]) reducing human error and speeding up assembly. One of
the core challenges of implementing successful HRC systems is the ability for
machines to understand the environment around them in a quick and robust
manner; with a successful HRC system requiring the ability to understand an
assembler’s actions and intentions in real time in order to intelligently
assist them and provide an efficient hybrid work system [27].

Motivated by recent atomic-action datasets (_e.g_. [4, 5, 21, 30]) and the
significant advances in fine-grained action recognition (AR) the utilisation
of video data captured within the shared work environment is now an
attractive, unintrusive, and information rich approach to classifying
fundamental atomic actions between a worker, product, part or tool. With the
accurate classification of these actions allowing assemblies to be understood
as a nested set of fundamental atomic actions. Whilst many works have focused
on classifying actions within manufacturing environments, most of these works
have been limited in their approach; often operating within a controlled
environment, classifying coarse actions with single frames, or performing
action segmentation in an offline manner. In contrast, many HRC systems
require real time classification of spatially and temporally similar fine-
grained actions, which is often very challenging due to the spatial
limitations of AR inputs (224x224) where often significant downsampling is
used – removing vital information collected by the camera.

With this motivation in mind we develop a novel AR model facilitating the use
of enhanced spatial features via hand guided high resolution feature
enhancement, capable of operating in real time in a sliding window fashion
with temporally aware label cleaning for action segmentation, as shown in Fig.
1. Our novel model architecture is evaluated on a domain specific novel
assembly dataset collected from a real world robotics sub assembly
manufacturing production line. We perform extensive experimentation on
extended sequences of unseen video data and provide a class specific
performance analysis, revealing the importance of enhanced hand features for
improving fine-grained atomic action classification. Our main contributions
are as follows:

  * •

We Introduce a novel AR model utilising hand location guided high resolution
feature extraction to allow fine grained details to aid a backbone AR model
and show both high resolution and frame wide features are key to accurate
classification.

  * •

We show that running a temporally aware model trained on short video clips
from our domain dataset can be successfully applied in a simple sliding window
via the use of surround sampling during training and temporally aware label
cleaning, surpassing the performance of similar encoder/decoder methods,
whilst operating in real time.

  * •

We show that the performance of encoder/decoder methods can be improved
significantly, and training time reduced, via the use of in domain feature
extraction via model training under our framework.

##  2 Related Work

###  2.1 Action Segmentation

Current action segmentation approaches can be broadly broken down into two
stages: An initial feature extraction process (often on a frame or short
segment level) followed by a frame-wise sequence prediction stage, often
facilitating the ability to learn longer term temporal dependencies to refine
action boundaries and reduce over segmentation.

####  2.1.1 Feature Extraction

Feature extraction generally falls into two main categories: 2D or 3D
Convolutional Neural Network (CNN) based methods. 2D CNN methods utilise a 2D
CNN [14, 20, 35] backbone to extract frame level predictions with a temporal
aggregation step, such as a simple averaging [40] or a more complex operation
such as an LSTM block [6] to produce features for a set of input frames. In
contrast, 3D CNN based methods (_e.g_. [13, 26, 34, 37, 38]) expand the 2D
convolutional operation into the temporal dimension, performing 3D
convolutions across space and time, extracting joint spatial and temporal
semantics from voxels of raw video data. Other works utilise the inherent
computational efficiency of 2D CNNs via the incorporation of temporally aware
modules inserted within 2D CNN architectures (_e.g_. [23, 33]) facilitating
the mixing of features across frames, allowing for a computationally efficient
and parameter light methods of facilitating temporal learning.

Our work is similar in motivation to many works focusing on improving feature
extraction via lateral connections between different layers of a model [24,
44], different models operating on different frame rates [9] or modalities
[10]. In contrast to these approaches which all utilise a single fixed input
resolution, we utilise a secondary stream with access to higher resolution
image information which is used to enhance the backbone frame wide features.

####  2.1.2 Sequence Predictions

In order to produce frame level predictions across a long untrimmed video many
temporal modelling methods have been used. Most simply, a sliding window [28],
often operating with a recurrent network [31] or a feature buffer [19, 23] to
allow for arbitrary length temporal modelling in an online manner, can be
utilised to make sequence predictions. Many approaches replace the relatively
short term temporal modelling capabilities of recurrent models with temporal
convolutions [2], improving action boundaries and minimising over segmentation
errors via encoder/decoder methods [7, 22] typically operating on precomputed
features extracted by a pretrained 3D CNN model. In contrast, we show show
through the use of surround sampling while training on in domain data, a
simple temporally aware label cleaning method at inference can outperform more
computationally expensive encoder/decoder methods operating on kinetics [17]
pretrained features, while performing real time action segmentation.

###  2.2 Assembly Understanding

Figure 2: An overview of our proposed method of utilising high resolution hand
features to improve frame level features. Our method can be broken into three
main sections: a) A backbone model extracting frame wide features from a set
of spatially downsampled video frames. b) Feature extraction from higher
resolution images focused on locations extracted via a hand regression model.
c) Backbone model feature enhancement via the spatially aware combination of
extracted features.

Since the breakthrough of the CNN during the 2010s, there has been fairly
limited work applying AR models to the manufacturing domain in order to
understand and facilitate HRC. Wang _et al_. [41] fine-tune two AlexNet[20]
models to classify three parts being used as well as three current “motions”
(grasping, holding and assembling) during an engine block assembly within a
controlled environment. Al-Amin et al. [1] utilise a multi-modal sensor
fusion, including image data, to classify seven actions within a controlled
assembly process. Zhang et al. [45] utilise a bi-stream CNN to extract human
and object information from video frames, with a variable-length Markov model
making future action predictions within a controlled assembly scenario. Xiong
_et al_. [43] incorporates motion information via the use of a modest two
stream network classifying actions occurring within an engine block assembly
video scraped from YouTube. Jones _et al_. [16] develop a framework for
understanding fine-grained assembly actions as a structure. In contrast to
these works, which are all limited in approach, and are deployed within
controlled environments, our work utilises fully spatio-temporal feature
extraction and is applied to our novel real world assembly dataset.

As human assemblies naturally utilise hands, there are several works with
similar motivations to ours. Lui _et al_. [25] utilise a modest 3D CNN
operating on fixed length segments composed of cropped hand images to classify
real world assembly data into seven classes. Kobayashi _et al_. [18] implement
a similar method however note that such inputs remove much of the contextual
surrounding information which is often required for accurate classification.
As such in the same work Kobayashi _et al_. introduce an attention based model
where pose features from single video frames are used to draw attention to
hands on real world assembly data. Images are classified into eleven actions
(such as take product, put on component, grab driver) using an encoder/decoder
for action segmentation. In contrast, we show that using higher resolution
pixel data from around the hands to enhance full frame features provides more
information than drawing attention to hands during assemblies, whilst allowing
for full frame features to provide context when classifying, unlike hand
cropping methods.

##  3 Proposed Method

The following section introduces our proposed method for action segmentation
within the manufacturing domain. Our method aims to specifically tackle the
challenge of distinguishing between spatially fine-grained actions, where two
temporally similar actions _e.g_. tighten screw with torque wrench vs. tighten
nut with torque wrench require fine-grained spatial detail to separate. As the
vast majority of AR approaches are based on image classification CNNs
requiring a fixed size input of 224x224 pixels, high resolution video frames
(typically upwards of 720p) must be downsampled before being input into the
model, removing vital high resolution information required to distinguish
between small or similar looking components/parts. In order to address this
issue our method outlined in Fig. 2 utilises high resolution data extracted
around the assembler’s hands (where most relevant fine grained information:
tool, component, fixings _etc_. is typically concentrated) to enhance the
features extracted by a backbone frame wide model. Our model can be separated
into 3 main section: a) a backbone frame wide feature extractor, b) high
resolution hand feature extraction and c) back bone feature enhancement via a
spatially aware feature enhancement module, discussed in section 3.1, 3.2 and
3.3 respectively.

###  3.1 Backbone Model

As shown in section (a) (highlighted red) of Fig. 2 we build upon a generic
backbone action recognition model (see section 3.4.1 for specific model
selection) to extract features from the frame level across the input clip and
produce a final action classification. This backbone operates on spatially
downsized video voxels of size $\mathbb{R}^{T,3,224,224}$ where T is the
number of 224x224 RGB image frames, producing frame level feature maps,
$\mathcal{F}\in\mathbb{R}^{T,C,H,W}$, which we seek to enhance via high
resolution hand features.

###  3.2 High Resolution Hand Feature Extraction

Shown in section (b) (highlighted green) of Fig. 2 the second section of our
model is designed to extract features from around the hands in order to aid
the classification of actions with fine-grained spatial detail and is a two
stage process.

First, to detect the presence and location of an assemblers hand within an
image we train a custom mobilenetv2 [29] model to predict the probability and
location of hands within an image. As our model relies on the accurate
localisation of an assemblers hands within video frames we report the high
performance of our hand model in section A of the supplementary material along
with model architecture, implementation and training details.

Second, a crop of size $(h_{c},w_{c})$ is taken centred on hand locations
retaining the resolution of the full input image111If no hand is predicted by
the model we input a central crop from the full image, which we found to
perform better than a zeroed input.. We truncate the hand models to extract
features from an intermediate layer, producing two hand feature maps,
$\mathcal{F}_{h}\in\mathbb{R}^{T,C’,H’,W’}$ in order to retain spatial
features and limit parameters used within these network streams (see section
3.4 where we discuss our specific model and layer choices).

###  3.3 Feature Enhancement

At the core to our model is the feature enhancement stage whereby extracted
hand features maps ($\mathcal{F}_{h}$) are used to enhance the backbone model
features ($\mathcal{F}$). We introduce the feature enhancement strategy
highlighted by (c) (highlighted yellow) in Fig. 2 providing the models and
dimensions of feature maps we select in section 3.4. Our feature enhancement
strategy allows for the spatial alignment and combination of features of any
size from the hand stream to the features at any point in the backbone
architecture.

####  3.3.1 Spatial Alignment

Due to the higher relative resolution of the hand input, compared to the frame
wide input image, the hand input image and all hand feature maps
($\mathcal{F}_{h}$) extracted by the hand model correspond to a fixed region
within the larger input frame and all subsequent feature maps $\mathcal{F}$ of
the backbone model. In order to account for the large discrepancy in spatial
input space between the hand and backbone stream, and to combine features in a
spatially consistent manner we propose a spatial alignment process whereby
features extracted from the high resolution hand data are mapped to their
corresponding size and locations with respect to the input image via the use
of up/down sampling zero padding. These padded feature maps represent the high
resolution hand features with respect to the backbone architecture, which can
be used to enhance the spatial features of the backbone feature maps. In order
to achieve this spatial alignment between hand cutouts of size
$(w_{c}$,$h_{c})$ at location $(x_{hand},y_{hand})$, and the backbone randomly
cropped input of size $C_{s}$ at an offset $(x_{c}$,$y_{c})$, we calculate the
normalised size $(\overline{w}_{hand},\overline{h}_{hand})$, and normalised
offsets $(\overline{x}_{hand},\overline{y}_{hand})$ with in equation 1 and 2
respectively. Where H is the height of full resolution frame and S is the
height of the scaled input image before random cropping – a more detailed
description and diagram are provided in section 2 of the supplementary
material.

$\overline{w}_{hand}=\biggl{(}\frac{w_{c}}{H}\biggr{)}\left(\frac{S}{C_{s}}\right),\quad\overline{h}_{hand}=\biggl{(}\frac{h_{c}}{H}\biggr{)}\left(\frac{S}{C_{s}}\right)$
(1)

$\overline{x}_{hand}=\frac{x_{hand}-x_{c}}{C_{s}},\quad\overline{y}_{hand}=\frac{y_{hand}-y_{c}}{C_{s}}$
(2)

Once the normalised offsets and size have been calculated, $\mathcal{F}_{h}$
is up/downsampled (using nearest neighbour when upsampling) to spatially match
the size of the hand with respect to back bone feature map $\mathcal{F}$, to
produce an intermediate feature map
$\mathcal{F}_{h}’\in\mathbb{R}^{T,C,H*\overline{h},W*\overline{w}}$. We then
zero pad around $\mathcal{F}_{h}^{\prime}$ to match the size of $\mathcal{F}$
using the offsets $(\overline{x}_{hand},\overline{y}_{hand})$ to place the
features at the corresponding location within $\mathcal{F}$ producing a set of
feature maps $\mathcal{F}’\in\mathbb{R}^{T,C,H,W}$ matching the backbone
spatial dimensions whilst containing high resolution hand features spatially
located at the position of hands within the backbone feature map
$\mathcal{F}$.

####  3.3.2 Feature Combination

Once hand features have been spatially aligned using zero padding to match the
spatial dimension of the full frame features, they are concatenated along the
channel dimension, producing a set of separate but spatially matched backbone
and high resolution hand features $\in\mathbb{R}^{T,3C,H,W}$.

To combine these separate features into one feature set of enhanced full frame
feature maps a 1x1 convolution, of kernel size C is utilised to reduce the
channel dimension from 3C back to that of the original extracted feature map
$\mathcal{F}$. This convolution allows for spatially preserved channel mixing
between the feature maps and produce a set of enhanced hand feature maps. The
enhanced feature maps are then added to the original extracted frame features,
$\mathcal{F}$, via a residual connection, followed by a linear activation
function and batch normalisation.

###  3.4 Model & Feature Combination Stage Selection

Whilst our outlined method has so far has been invariant to model choice hand
crop size and the intermediate feature maps selection, we now outline the
choices made when implementing our model for the experiments described in
section 4.

####  3.4.1 Model Selection

Following convention we opt for a ResNet-50[14] backbone and ResNet-18 models
for the two hand streams. However note other more efficient backbones _e.g_.
[15] [36] could be utilised for real time operation on edge devices. We also
select a hand crop size of $w_{c}=h_{c}=224$ and the random spatial crop
$C_{s}=224$ as this is the standard input size for a ResNet model, and
simplifies equation 1 to $\overline{w}=\overline{h}=\frac{S}{W}$.

We operate models across multiple frames utilising the Temporal Segment
Network[40] approach of averaging frame levels predictions over an input clip.
To facilitate the efficient learning of temporal features between frames, we
utilise the Temporal Shift Module[23] which is inserted in every residual
block for all models.

####  3.4.2 Combination Stage

When selecting the feature maps to utilise for feature combination, we note
that minimal downsampling/upsampling is preferred when matching the two
feature sets. As such we utilise the consecutive downsampling of the hand
streams to the point where the downsampled hand features match the size of the
hand within one of the early feature map within the backbone model (_i.e_.
$H\overline{h}\approx H’$). We therefore opt to enhance the features from the
first block of a ResNet-50 backbone $(\mathcal{F}\in\mathbb{R}^{T,256,56,56})$
with the features from a truncated hand models at the penultimate residual
block $(\mathcal{F}_{h}\in\mathbb{R}^{T,256,14,14})$, as the backbone hands
have a size of $H\frac{S}{H}=56*(\frac{256}{720})\approx 20$ and
$H^{\prime}=14$.

###  3.5 Real Time Action Segmentation

We extend our offline action recognition approach to action segmentation via a
simple overlapping sliding window methodology. During inference we utilise a
dense frame sampling strategy to artificially create input segments
corresponding to the current video frame $x_{0}$. T frames are sampled at a
temporal stride of $\tau$ going backwards from the most recently received
video frame to create an input clip of fixed length
$\\{x_{-T\tau},x_{-(T-1)\tau},...,x_{-\tau},x_{0}\\}$ which is used to create
a classification for the middle frame of the clip, $\emph{i.e}.\hbox{}$
$x_{-T\tau/2}$.

####  3.5.1 Training Clip Creation - Surround Sampling

To match the dense sampling strategy utilised at inference, during training
clips of length $T\tau$ must be sampled from examples within our dataset
containing N frames. To ensure our model can accurately determine the start
and end of actions in sequences within a video feed we implement a surround
sampling procedure where the starting frame of the training sample
($\emph{i.e}.\hbox{}$ $x_{-T\tau}$) is chosen via equation 3, where $N_{s}$
and $N_{e}$ are the segment level labelled start and end frame of the segment
within the dataset, T is the number of frames selected and $\tau$ is the
temporal stride between frames.

$x_{-T\tau}=U\\{N_{s}-\frac{T\tau}{2},N_{e}-\frac{T\tau}{2}\\},s.t.\ \
x_{-T\tau}\in\mathbb{N}$ (3)

Importantly, this sampling procedure allows frames from outside the labelled
segment to be seen by the model during training. As the model produces a
prediction for the central frame of the segment this sampling strategy assists
the model at accurately predicting the boundaries of actions much better than
sampling strategy where training clips are created from data fixed within the
bounds of an action.

During offline testing clips are sampled from the centre of the N frame
examples, with the initial frame selected as
$x_{-T\tau}=\frac{N_{f}-N_{s}}{2}-\frac{T\tau}{2}$.

####  3.5.2 Temporally Aware Label Cleaning

Under the outlined framework, our model is used to produce per frame
classification when operating in an overlapping sliding window fashion on
sequences of video data. We implement an additional temporally aware label
cleaning stage to improve the model performance on sequences of data as a
method of reducing over segmentation while retaining real time implementation
rather than using an offline method such as encoder/decoder. While the model
is running across a sequence of data any predictions which are statistically
too short, that is have a length shorter than $C_{mean}-\kappa C_{std}$, where
$C_{mean}$ and $C_{std}$ are the mean and standard deviation of the lengths of
class C within the training data, and $\kappa$ is a constant which we find by
sweeping between 1 $\rightarrow$ 2 on the training data to determine the best
cutoff for performance, are ignored by the model with the previous action
prediction being taken as true.

##  4 Experimental Results

The following section presents the experimental results of our enhanced hand
feature model on our newly collected atomic assembly dataset, comparing to
baseline and similar works. We further discuss the deployment of our model
(and others) for real time action segmentation through the use of temporally
aware label cleaning, comparing the model to more computationally heavy
encoder/decoder methods, highlighting the importance of short term temporal
feature learning for fine-grained atomic action segmentation.

###  4.1 Atomic Assembly Dataset

i.d |  Interaction |  Object |  Secondary Object |  No of Examples |  Mean Length (sec) |  Total Length (min)
---|---|---|---|---|---|---
0 |  Unbox |  Component |  - |  82 |  8.63 |  11.80
1 |  Pick Up |  Washer and/or Screw |  - |  610 |  2.07 |  21.08
2 |  Spanner |  - |  50 |  1.44 |  1.20
3 |  Screwdriver |  - |  101 |  1.02 |  1.73
4 |  Marker Pen |  - |  149 |  1.12 |  2.78
5 |  Torque Wrench |  - |  85 |  1.14 |  1.62
6 |  Put Down |  Spanner |  - |  49 |  0.96 |  0.78
7 |  Screwdriver |  - |  102 |  1.13 |  1.92
8 |  Marker Pen |  - |  140 |  1.27 |  2.96
9 |  Torque Wrench |  - |  81 |  1.07 |  1.49
10 |  Place |  Washer and Screw |  on Screwdriver |  428 |  1.86 |  13.29
11 |  Washer and Screw |  on Product |  159 |  1.78 |  4.72
12 |  Component |  on Product |  123 |  3.43 |  7.03
13 |  Washer |  on Bolt |  74 |  2.76 |  3.40
14 |  Metal Bar |  on Product |  38 |  3.59 |  2.27
15 |  Tighten |  Screw |  with Hand |  163 |  3.80 |  10.32
16 |  Screw |  with Screwdriver |  359 |  5.32 |  31.85
17 |  Screw |  with Torque Wrench |  168 |  3.39 |  9.49
18 |  Nut |  with Hand |  77 |  8.43 |  10.82
19 |  Nut |  with Spanner |  220 |  1.99 |  7.28
20 |  Nut |  with Torque Wrench |  95 |  7.74 |  12.26
21 |  Mark |  Bolt |  with Marker Pen |  134 |  4.46 |  9.97
22 |  Screw |  with Marker Pen |  105 |  12.89 |  22.55
23 |  Remove |  Washer and Nut |  from Product |  76 |  7.95 |  10.07
24 |  No Action |  - |  - |  1220 |  3.75 |  76.26

Table 1: Summary of assembly dataset statistics.

In order to evaluate the effectiveness of our model for HRC tasks we create a
novel manufacturing action segmentation dataset. A D415 Intel RealSense depth
camera (15 fps @ 920x720) was used over 3 days to record 3 different workers
complete a real world robotics assembly procedure. The camera was positioned
above a workbench with the camera angled to capture the bench surface, workers
hands and lower torso, avoiding discernible features. Assemblers were informed
to perform assemblies as normal, making no alterations for the camera. In
total 38 full assemblies were recorded, corresponding to $\sim$6 hours of
footage.

Labelling. The data is labelled using a fine-grained noun/verb labelling
procedure with class selection guided by the standard operating procedure for
the given assembly. We select fundamental interactions between the worker and
either the product being assembled, a part/component or a tool - while
ignoring unuseful or ambiguous actions for understanding the assembly process.
In total seven primary interactions (verbs) were found: Unbox, Pick Up, Put
Down, Place, Tighten, Mark with Pen, Remove. These interactions either
occurred with a single object or with a primary and secondary object. In total
24 relevant actions verb/noun pairs were found between the operator and all
objects within the assembly and are summarised in Table 1. To label the data
we utilise a custom action segmentation labelling GUI, which we make
public222https://github.com/Matthewkm/Action-Segmentation-Labeller-GUI. In
total 186 632 from a total of $\sim$324 000 frames ($\sim$57.6$\%$) were found
to contain an atomic action from our label list.

Properties. Our dataset follows a long tail distribution with the number of
collected examples ranging from 38 (Place Metal Bar \- only occurring once per
assembly) to 610 (Pick up Washer and/or Screw) with average clip lengths
ranging from as short as 0.96 seconds (Put Down Spanner) to 12.89 seconds
(Mark Screw(s) with Pen). Total clip lengths of classes range from as low as
46 seconds (Put Down Spanner) to over 30 minutes (Tightening Screw with
Screwdriver). While relatively modest in size our dataset is emblematic of
many dataset collected within real world environments where the collection and
labelling of large datasets is challenging due to quick turnaround times and
domain novelty. Compared to other action segmentation datasets ours is much
more fine-grained than large datasets such as Breakfast [21] (77 hours - 10
actions) and larger in size than comparative fine-grained datasets such as
GTEA [8] (28 minutes - 20 actions) and 50 salads [32] (4 hours - 17 actions).
Our dataset is noticeably fine-grained as the background remains constant
throughout all classes, with only subtle changes in spatial and temporal
information between many classes. In addition, due to the noun/verb pairing
process there are overlaps between some classes where the same noun (e.g tool
or component) appears within multiple verb pairings (e.g screw or place) and
vice versa. As such the ability to model both spatial and temporal information
will be key to accurate classification.

Model | Segment | Sequence Segmentation
---|---|---
Level F1 | F1@{0.5,0.25,0.1}
Single Frame Baseline | 68.6 | 30.7 | 44.9 | 51.7
TSM [23] ($\emph{i.e}.\hbox{}$ backbone only) | 86.2 | 67.3 | 79.8 | 81.3
TSMHA [18] | 79.7 | 57.7 | 70.2 | 72.3
TSMEHF (ours) | 89.2 | 69.2 | 80.5 | 82.5
Reduced Resolution TSMEHF | 86.1 | 67.2 | 78.2 | 79.9
High Res Hands Only | 87.6 | 67.1 | 78.6 | 80.4

Table 2: Comparison of model performance on both a segment level and on unseen
extended assembly sequences utilising temporally aware label cleaning.

###  4.2 Implementation Details

All models are trained within the Pytorch deep learning framework on segments
extracted from 25 of the full assembly sequences, reserving the remaining 13
assembly sequences for unseen testing on both a segment and sequence level.
See the supplementary information for a detailed account of parameter settings
when training the models.

###  4.3 Evaluation Metrics

In order to test our models spatial and temporal reasoning we report results
on both pre-cropped segments as well as temporal segmentation on unseen
sequences. We report $F_{1}$ score on a segment level treating each pre-
cropped segment as an single example regardless of length. When analysing
segmentation results on an unseen sequence we report frame wise accuracy (Acc)
as well as segmental edit distance, and segmental F1 score (as proposed by
[22]) at various IoU thresholds (10%, 25% and 50%) to accurately account for
over-segmentation errors.

###  4.4 Evaluation of High Resolution Hand Model

Figure 3: Improvement in F1 score across classes on a segment level when
comparing an 8 frame temporally aware TSM model to an equivalent model with
our high resolution hand enhanced features.

Table 2 compares the performance of our enhanced hand feature model (TSMEHF)
to an equivalent unenhanced TSM [23] model (both T=8 frames, with a temporal
stride, $\tau$=8) on a segment and extended sequence level. We further compare
our model to an implementation of the similar hand attention (TSMHA) work by
Kobayashi $\emph{et al}.\hbox{}$333We note that Kobayashi $\emph{et
al}.\hbox{}$ operate in an encoder/decoder structure, however, we re-implement
their work following our method outlined in section 3.5 as well as utilise a
temporally aware backbone structure to match our work. [18] and a baseline
single frame ResNet-50 model. We find our model outperforms the backbone TSM
model and the hand attention model by Kobayashi.

Fig. 3 highlights how TSMEHF improves the per class performance for nearly all
classes compared to an equivalent backbone model, with the largest improvement
seen in many classes with fine grained spatial differences, such as
distinguishing between place washer on bolt and place metal bar. Our model
does not harm the temporal reasoning as we see no decrease in temporally
salient classes, and even see large improvements in pick up/put down
screwdriver and tighten nut with hand vs remove washer and nut by hand, which
require both fine-grained spatial and temporal reasoning. Only a small
improvement is seen for No Action, suggesting our model is not improving the
ability to distinguish between No Action and an action (as we see going from
temporally unaware to temporally aware models - see section 4.5.1) but is
extracting complementary enhanced spatial features, helping to distinguish
between fine-grained spatial classes, without harming a models ability to
temporally model frame wide features.

Figure 4: Improvement in<EMAIL_ADDRESS>score across classes on a sequence level
when comparing a single frame to an 8 frame temporally aware TSM model
operating in a sliding window fashion.

To further verify the performance improvements of TSMEHF are due to high
resolution hand data otherwise not available to the baseline TSM model, we
implement two modified models also shown in Table 2. Firstly, just the high
resolution hand section of our model (implementing the full ResNet-18 models
with averaged softmax outputs) finding that the while the model performs
better than the backbone TSM model on a temporally cropped segment level
(despite having many more parameters) it performs worse on a sequence level,
suggesting the larger frame wide features of the backbone model are useful for
accurate segmentation under our framework ($\emph{i.e}.\hbox{}$ distinguishing
boundaries of actions). Secondly, a low resolution hand version of our model,
where hand input images are downsampled to match the resolution of the input
image to the backbone model, essentially creating a model with the same number
of parameters, but without increased resolution. As TSMEHF outperforms both
these models and the TSM baseline on a segment level we can see that high
resolution hand features provide separate yet complementary features useful
for classification, and that it is the access to higher resolution features
that improves classification performance.

###  4.5 Real Time Action Segmentation Results

Segmentation Approach | Model | Finetuned From | Real time | Acc | F1@{0.5,0.25,0.1} | Edit
---|---|---|---|---|---|---
Simple Sliding Window | TSN [40] | Kinetics | ✓ | 74.1 | 24.9 | 34.6 | 39.5 | 31.2
TSM [23] | Kinetics | ✓ | 81.2 | 47.9 | 56.1 | 57.8 | 44.5
TSMEHF (ours) | Kinetics | ✓ | 81.8 | 49.1 | 56.5 | 58.9 | 45.4
Temporally Aware Label Cleaning (ours) | TSN [40] | Kinetics | ✓ | 73.6 | 49.1 | 67.8 | 72.9 | 65.4
TSM [23] | Kinetics | ✓ | 79.7 | 74.0 | 84.6 | 85.6 | 77.7
TSMEHF (ours) | Kinetics | ✓ | 80.9 | 75.8 | 85.6 | 87.2 | 79.9
Encoder/Decoder [7] | ms-TCN | TSM Kinetics | ✗ | 70.7 | 51.2 | 68.5 | 73.8 | 71.4
ms-TCN | TSM Assembly Dataset | ✗ | 84.1 | 77.6 | 85.3 | 86.3 | 79.6
ms-TCN | TSMEHF Assembly Dataset | ✗ | 83.7 | 78.4 | 87.0 | 88.4 | 83.6

Table 3: Sequence Segmentation results for temporally aware label cleaning of
sliding window predictions and ms-TCN using background emitted performance
metrics as utilised in [7]. All models operating on T=8 frames with temporal
stride $\tau$=8

In the following section we compare the performance of temporally aware label
cleaning as a method of deploying AR models to unseen extended sequences of
assembly data. We show fine-grained temporally aware backbones are key to
successful classification of short atomic actions and that temporally aware
label cleaning is capable of matching the performance of encoder/decoder
methods on temporally short atomic actions while maintaining the ability to
operate in a near real time manner.

####  4.5.1 Importance of Short Range Temporal Learning

One key design choice within our model framework is that of the backbone model
for classification and specifically how important is the ability for the
backbone model to learn temporal features across multiple frames. Under our
dense sampling strategy outlined in 3.5 there are two variables facilitating
how much temporal information the backbone model can access from an input
clip: the number of input frames, T, and the stride between them, $\tau$. In
order to investigate the importance of temporal reasoning we train a set of
identical Temporal Segment Networks (TSN) with and without inserted TSM with
varying numbers of input frames and temporal stride between input frames, with
detailed results shown in section D of the supplementary material. Through our
evaluation we find, for atomically short actions, that a temporally aware
backbone operating on multiple frames is key to accurate classification on a
sequence level. We find that performance saturates at T=8,$\tau$=8 (real time
FPS of 1.875 with the model only having access to 4.27 seconds of video) when
temporally aware label cleaning is utilised.

Fig. 4 shows the improvement in individual class performance when changing
from a single to an 8 frame temporally aware backbone model when performing
action segmentationn in sliding window fashion. When utilising a single frame
model many classes achieve very poor performances especially the more
temporally salient classes, such as the pick up/put down tool and place
classes, as a single frame model can’t distinguish between picking up a tool
and holding a tool (which falls under no action). As no action is a more
prominent class, in the absence of the ability to distinguish between these
actions and no action the model ends up classifying all these actions as no
action. Once temporal reasoning has been enabled all classes are substantially
improved, with some such as put down spanner seeing an improvement from 0 to
78<EMAIL_ADDRESS>We also observe that the improvement in classification of
temporally salient classes and no action is also much larger on a sequence
level compared to a segment level (see supplementary information) suggesting a
temporally aware backbone is key for accurate action segmentation when
operating in a sliding window fashion, allowing the model to distinguish
between actions and no action much more successfully.

####  4.5.2 Importance of Temporally Aware Label Cleaning

The first two sections of Table 3 show the performance before and after
temporally aware label cleaning (section 3.5.2) of various models operating in
a sliding window fashion on unseen assembly sequences from our fine-grained
assembly dataset. Again we see that enhanced hand features continue to improve
the model performance. Whilst we only see a modest change in frame level
accuracy we observe that temporally aware label cleaning significantly
improves the performance of our models on a sequence level, while allowing the
models to maintain their near real time implementation444There is a small
delays in model inference due to the need for T$\tau$/2 future frames for
prediction and $<$T$\tau$/2 frames for cleaning - however this delay is $<$2
second for a 15 fps camera). This intuitively suggests temporally aware label
cleaning is combating over-segmentation errors, which is further supported by
the fact that as the short term temporal modelling ability of a model increase
($\emph{i.e}.\hbox{}$ TSN $\rightarrow$ TSM) the effects of the temporally
aware label cleaning also increase, however much less of a change is observed
for the enhanced spatial feature models ($\emph{i.e}.\hbox{}$ TSM
$\rightarrow$ TSMEHF).

####  4.5.3 Comparison to Encoder/Decoder Methods

Finally we compare our temporally aware label cleaning method to the commonly
used encoder/decoder multi-sage Temporal Convolution Network (ms-TCN) [7].
Table 3 shows temporally aware label cleaning out performs a kinetics
pretrained ms-TCN (utilising the same feature extraction model), while being
able to operate in a near real time manner. We posit this is due to the fact
that actions within our dataset are typically much shorter in nature than many
other action segmentation datasets and do not necessarily occur in a fixed
order, thus removing the advantage of extended temporal reasoning provided by
an encoder/decoder method. We further show that features extracted from our
assembly dataset trained models can be used to as base features to train ms-
TCN models, providing significant improvements over the kinetics feature
extraction<EMAIL_ADDRESS>51.2 $\rightarrow$ 78.4) while reducing training time by
$\sim$65% ( 200$\rightarrow$70 epochs) showing that significant improvements
can be obtained for other encoder/decoder methods via the use of in domain
pretraining via the methods outlined in section 3.5.

##  5 Conclusion

In this paper we present a novel approach to utilising high resolution image
features around an assemblers hand to improve action segmentation within real
world assembly video. We show that through the use of short range temporally
aware backbones, surround sampling and temporally aware label cleaning our
model can be applied to extended unseen video sequences in a sliding window
fashion, proving real time action segmentation required for many assembly HRC
tasks. Further work will focus on applying models to other similar domain
publicly available datasets as well as a more generalised network capable of
high resolution features extraction from self determined areas of interest for
use outside hand orientated datasets.

## References

  * [1] Md. Al-Amin, Wenjin Tao, David Doell, Ravon Lingard, Zhaozheng Yin, Ming C. Leu, and Ruwen Qin.  Action recognition in manufacturing assembly using multimodal sensor fusion.  Procedia Manufacturing, 39:158–167, 2019.  25th International Conference on Production Research Manufacturing Innovation: Cyber Physical Manufacturing August 9-14, 2019 — Chicago, Illinois (USA).
  * [2] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun.  An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, 2018.
  * [3] Chun-Fu Chen, Rameswar Panda, Kandan Ramakrishnan, Rogerio Feris, John Cohn, Aude Oliva, and Quanfu Fan.  Deep analysis of cnn-based spatio-temporal representations for action recognition, 2021.
  * [4] Jihoon Chung, Cheng hsin Wuu, Hsuan ru Yang, Yu-Wing Tai, and Chi-Keung Tang.  Haa500: Human-centric atomic action dataset with curated videos, 2020\.
  * [5] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Rescaling egocentric vision.  CoRR, abs/2006.13256, 2020.
  * [6] Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, and Trevor Darrell.  Long-term recurrent convolutional networks for visual recognition and description, 2014.
  * [7] Yazan Abu Farha and Juergen Gall.  Ms-tcn: Multi-stage temporal convolutional network for action segmentation, 2019.
  * [8] Alireza Fathi, Ali Farhadi, and James M. Rehg.  Understanding egocentric activities.  In 2011 International Conference on Computer Vision, pages 407–414, 2011.
  * [9] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition, 2019.
  * [10] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.  Convolutional two-stream network fusion for video action recognition, 2016\.
  * [11] Markus Funk, Thomas Kosch, and Albrecht Schmidt.  Interactive worker assistance: Comparing the effects of in-situ projection, head-mounted displays, tablet, and paper instructions.  In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp ’16, page 934–939, New York, NY, USA, 2016. Association for Computing Machinery.
  * [12] Ankit Gupta, Dieter Fox, Brian Curless, and Michael Cohen.  DuploTrack: A Real-Time System for Authoring and Guiding Duplo Block Assembly, page 389–402.  Association for Computing Machinery, New York, NY, USA, 2012.
  * [13] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh.  Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?, 2018.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [15] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam.  Searching for mobilenetv3, 2019.
  * [16] Jonathan D. Jones, Cathryn Cortesa, Amy Shelton, Barbara Landau, Sanjeev Khudanpur, and Gregory D. Hager.  Fine-grained activity recognition for assembly videos, 2020.
  * [17] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman.  The kinetics human action video dataset, 2017.
  * [18] Takuya Kobayashi, Yoshimitsu Aoki, Shogo Shimizu, Katsuhiro Kusano, and Seiji Okumura.  Fine-grained action recognition in assembly work scenes by drawing attention to the hands.  In 2019 15th International Conference on Signal-Image Technology Internet-Based Systems (SITIS), pages 440–446, 2019.
  * [19] Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong.  Movinets: Mobile video networks for efficient video recognition, 2021\.
  * [20] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
  * [21] Hilde Kuehne, Ali Arslan, and Thomas Serre.  The language of actions: Recovering the syntax and semantics of goal-directed human activities.  In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 780–787, 2014.
  * [22] Colin Lea, Michael D. Flynn, Rene Vidal, Austin Reiter, and Gregory D. Hager.  Temporal convolutional networks for action segmentation and detection, 2016.
  * [23] Ji Lin, Chuang Gan, and Song Han.  Tsm: Temporal shift module for efficient video understanding.  In Proceedings of the IEEE International Conference on Computer Vision, 2019.
  * [24] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection, 2016.
  * [25] Liang Liu, Yong Liu, and Jiangning Zhang.  Learning-based hand motion capture and understanding in assembly process.  IEEE Transactions on Industrial Electronics, 66(12):9703–9712, 2019\.
  * [26] Zhaofan Qiu, Ting Yao, and Tao Mei.  Learning spatio-temporal representation with pseudo-3d residual networks, 2017.
  * [27] Fabian Ranz, Titanilla Komenda, Gerhard Reisinger, Philipp Hold, Vera Hummel, and Wilfried Sihn.  A morphology of human robot collaboration systems for industrial assembly.  Procedia CIRP, 72:99–104, 2018.  51st CIRP Conference on Manufacturing Systems.
  * [28] Marcus Rohrbach, Anna Rohrbach, Michaela Regneri, Sikandar Amin, Mykhaylo Andriluka, Manfred Pinkal, and Bernt Schiele.  Recognizing fine-grained and composite activities using hand-centric features and script data.  International Journal of Computer Vision, 119(3):346–373, aug 2015\.
  * [29] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  2018\.
  * [30] Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.  Hollywood in homes: Crowdsourcing data collection for activity understanding, 2016.
  * [31] Bharat Singh, Tim K. Marks, Michael Jones, Oncel Tuzel, and Ming Shao.  A multi-stream bi-directional recurrent neural network for fine-grained action detection.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1961–1970, 2016.
  * [32] Sebastian Stein and Stephen J. McKenna.  Combining embedded accelerometers with computer vision for recognizing food preparation activities.  In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp ’13, page 729–738, New York, NY, USA, 2013. Association for Computing Machinery.
  * [33] Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz.  Gate-shift networks for video action recognition, 2019.
  * [34] Lin Sun, Kui Jia, Dit-Yan Yeung, and Bertram E. Shi.  Human action recognition using factorized spatio-temporal convolutional networks, 2015.
  * [35] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions, 2014.
  * [36] Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  2019\.
  * [37] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In 2015 IEEE International Conference on Computer Vision (ICCV), pages 4489–4497, 2015.
  * [38] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition, 2017\.
  * [39] L. Wang, R. Gao, J. Váncza, J. Krüger, X.V. Wang, S. Makris, and G. Chryssolouris.  Symbiotic human-robot collaborative assembly.  CIRP Annals, 68(2):701–726, 2019.
  * [40] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 20–36, Cham, 2016. Springer International Publishing.
  * [41] Peng Wang, Hongyi Liu, Lihui Wang, and Robert X. Gao.  Deep learning-based human motion recognition for predictive context-aware human-robot collaboration.  CIRP Annals, 67(1):17–20, 2018.
  * [42] Xi Vincent Wang, Zsolt Kemény, József Váncza, and Lihui Wang.  Human–robot collaborative assembly in cyber-physical production: Classification framework and implementation.  CIRP Annals, 66(1):5–8, 2017.
  * [43] Qianqian Xiong, Jianjing Zhang, Peng Wang, Dongdong Liu, and Robert X. Gao.  Transferable two-stream convolutional neural network for human action recognition.  Journal of Manufacturing Systems, 56:605–614, 2020.
  * [44] Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou.  Temporal pyramid network for action recognition, 2020.
  * [45] Jianjing Zhang, Peng Wang, and Robert X. Gao.  Hybrid machine learning for human action recognition and prediction in assembly.  Robotics and Computer-Integrated Manufacturing, 72:102184, 2021\.

##  Appendix A Hand Localisation

###  A.1 Model details

In order to detect the presence and location of an assemblers hand within an
image we train a mobilentv2 [29] model to output a fixed Sigmoid normalised
vector of length 6 containing the elements
$[\hat{P}_{1},\hat{x}_{1},\hat{y}_{1},\hat{P}_{2},\hat{x}_{2},\hat{y}_{2}]$
where $\hat{P}_{1}$ and $\hat{P}_{2}$ are the probability of the left and
right hand existing in the image (1 if hand present 0 otherwise) and
$\hat{x}_{i}$, $\hat{y}_{i}$ are the normalised position of hand i in a given
input image. We train with a modified mean squared error loss, shown in
equation A.1, that doesn’t penalise the model for incorrectly predicting a
hand position when no hand is present, where $P_{i}$ is the (1,0) label of a
present hand and $x_{i}$ and $y_{i}$ are the ground truth normalised location
of hand i within a given image. $\lambda$ is an equalising constant set to 0.1
to get similar learning rates across the loss function.

$\displaystyle\mathcal{L}$
$\displaystyle=\lambda\sum_{i=1}^{2}(P_{i}-\hat{P}_{i})^{2}+P_{1}[(x_{1}-\hat{x}_{1})^{2}+(y_{1}-\hat{y}_{1})^{2}]$
$\displaystyle\hskip
50.00008pt+P_{2}[(x_{2}-\hat{x}_{2})^{2}+(y_{2}-\hat{y}_{2})^{2}]$ (4)

###  A.2 Implementation Details

Our hand localisation algorithms were trained with a learning rate of 0.04 for
200 epochs with a learning rate decrease at 100 and 150 epochs. With gradient
clipping set to 20, parameter weight decay set to 0.0005 and momentum set to
0.9.

We train our hand model on 4531 randomly selected frames within our training
data and test on a further 1919 frames. Roughly 15% of selected frames contain
either just one or no hands. We labelled our hand data with a custom GUI which
we also plan to make publicly available.

###  A.3 Hand Localisation Performance

In order to evaluate the performance of of hand localisation model we
implement the following metric: $F1@T_{L}$$>$$X$. Under our model output we
get a prediction per hand in the form $(\hat{P},\hat{x},\hat{y})$ where
$\hat{P}$ is the probability of a hand being present within the frame, and
$\hat{x},\hat{y}$ are the normalised position of the hand within the frame. We
allocate our predictions a true positive label if $\hat{P}$ is grater than 0.5
($\emph{i.e}.\hbox{}$ there is more than 50% probability the hand is present)
and the there is a ground truth hand present, and the spatial prediction of
the hand location is within a certain threshold of the ground truth location,
given by $\sqrt{(\hat{x}-x)^{2}+(\hat{y}-y)^{2}}<T_{L}$, where $T_{L}$ is a
changeable threshold and $x$ and $y$ are the ground truth hand position. If a
hand is predicted to exist but has the wrong location or there is no hand
present then the prediction will be assigned false positive. A prediction is
false negative when a hand not predicted but a hand is present within the
image. We report the hand performance for various location thresholds $T_{L}$
in table 4.

$T_{L}$ | $F1@T_{L}$
---|---
0.05 | 74.5
0.1 | 83.0
0.2 | 92.5
0.3 | 96.8
Table 4: Hand localisation performance on our novel assembly dataset.

##  Appendix B Hand Feature Spatial Alignment

Assuming an input image of size (W,H) and a down sample operation to size S
for input into the back bone model, the relative resolution between the two
inputs is $(H/S)$. In order to achieve this spatial alignment between hand
cutouts of size $(w_{c}$,$h_{c})$, and the backbone cutout of size $C_{s}$, we
calculate the normalised size $(\overline{w}_{hand},\overline{h}_{hand})$ via
equation 1 and offsets $(\overline{x}_{hand},\overline{y}_{hand})$ via
equation 2 with respect to the input image to the backbone stream. The
variables in these equations are defined in Fig. 5 which shows the down
scaling of the image from the shortest side from $H$ to $S$ and random crop of
size $C_{s}$ at offset $(x_{c},y_{c})$ (for regularisation during training) of
the input frame.

Figure 5: The location and size of the extracted hand image with respect to
the input image to the backbone model after pre-processing.

##  Appendix C Model Implementation Details

All models were implemented within the Pytorch deep learning framework with a
Resnet50 model used for the backbone architecture and a ResNet18 used for the
hand model. All models utilised Kinetics-400[17] pretraining, with batch
normalisation statistics frozen from pretrained weights to reduce overfitting.
The initial learning rate was set to 0.002 for parameters within the backbone
model and 0.0002 for parameters within the hand feature extraction model, with
gradient clipping set to 20. Parameter weight decay was set to 0.0005 and
momentum set to 0.9 for all parameters. All models were trained for 150 epochs
with a batch size of 64 (except for the 16 frame model which utilised a batch
size of 32), with learning rate decreased by a factor of 10 at 100 and 125
epochs. A dropout of 0.8 was implemented in the final fully connected layer of
all model to reduce overfitting.

Models were trained on the a GPU node of the Hartree Centre Jade-2 HPC
555https://www.jade.ac.uk/, utilising a single Tesla V-100 GPU with a wall
clock training time of $\sim$ 6 hours per model.

##  Appendix D Importance of Temporal learning

|
---|---
(a) | (b)
|
(c) | (d)
Figure 6: Importance of temporal reasoning in the backbone model. All graphs
show the F1 class distributions for a temporal segment network [40] with and
without inserted temporal shift modules[23] varying either the temporal
stride, $\tau$, with fixed T=8 (a and c), or varying number of input frames,
T, with fixed $\tau$=8 (b and d) on either a segment (a and b) or on an
extended sequence (c and d) level after temporally aware label cleaning.

In order to investigate the importance of temporal reasoning we train a set of
identical Temporal Segment Networks (TSN) with and without inserted TSM with
varying numbers of input frames and temporal stride between input frames, with
results shown in Fig. 6.

We first fix T to the commonly used 8 frames [3, 23] and vary the temporal
stride $\tau$$\in${1,2,4,6,8,10} with results shown in Fig. 6 (a and c on a
segment and sequence level respectively). It is clear that temporally aware
TSM models outperform standard TSN models across all temporal strides, and
maintain a high F1 score beyond $\tau=2$, while the performance of a regular
TSN model, incapable of temporal learning, drops as $\tau$ increases,
suggesting sparser frames can introduce more useful long term information for
classification, but only when temporal learning is possible. This drop in
performance is more significant on a sequence level with the performance of
some classes dropping to an F1 score of 0 at $\tau=10$, suggesting sparser
frames when temporal learning is not applicable leads to significant confusion
when operating in a sliding window fashion.

Secondly, we vary the number of input frames T$\in${1,2,4,8,16} while keeping
$\tau=8$. Fig. 6 (b and d on a segment and sequence level respectively) show
that for both a TSN and TSM model increasing the number of input frames helps
the models initially, however, beyond $T=4$ the TSN model performance drops
significantly, while the TSM model continues to perform well, suggesting again
that providing more temporal information by using multiple frames at input is
only useful if a temporally aware backbone model is used.

##  Appendix E Single vs. 8 Frame - Segment Level

Figure 7 shows the class F1 improvements comparing a single frame and a
temporally aware 8 frame model on a temporally cropped segment level. In
keeping with analysis on a sequence level, temporally salient classes see the
largest improvement, with place metal bar seeing the largest improvement from
an F1 score of 17 to 79. It is noted that the improvements in class
performance on a segment level is smaller than that on a sequence level as
mentioned in section 1.4.4, suggesting temporal modelling is more imperative
when operating on a sequence level, with no action seeing a much smaller
improvement, suggesting when segments are neatly temporally cropped they are
easier to distinguish on a spatial level.

Figure 7: Improvement in $F1$ score across classes on a segment level when
comparing a single frame to an 8 frame temporally aware TSM model.

##  Appendix F High Resolution Hand model - Sequence Level

For completeness we also include the difference in class performance between a
baseline TSM model and our high resolution hand model on sequence level.
Figure 8 shows that there is virtually no change in no action performance on a
sequence level, again suggesting high resolution hand features are improving
the models ability to distinguish between classes rather than distinguish
between an action and no action.

Figure 8: Improvement in<EMAIL_ADDRESS>score across classes on a sequence level
when comparing an 8 frame temporally aware TSM model to an equivalent model
with our high resolution hand enhanced features.

##  Appendix G Model Prediction Visualisations

In the following section we produce two model prediction visualisations
extracted from two extended unseen assembly sequences using an 8 frame TSM
model with high resolution hand feature enhancement. Figure 9 and 10 show
class colour coded predictions against ground truth predictions in grey, with
black predictions representing incorrectly predicted frames.

As can be seen, despite the model operating in a sliding window fashion it is
capable of accurately classify nearly all actions, predicting the start and
end of actions successfully despite not being explicitly trained to do so. One
notable challenging scenario encountered by the model is rapid changes from
one action to another which are often separated by a few frames of “no
action”. This problem is highlighted in Figure 10 when the operator repeatedly
places a screw on the product and then tightens the screw with their hand. In
this situation the model is liable to miss many short no action segments -
combining them into one of the rapidly changing actions, however can still
accurately segment the respective actions with very good precision.

Figure 9: Model predictions on an extended video sequence.

Figure 10: Model predictions on an extended video sequence.