# Pixel-wise Graph Attention Networks for Person Re-identification

Wenyu Zhang, Qing Ding*, Jian Hu, Yi Ma, Mingzhe Lu  University of Science and
Technology of ChinaHefeiAnHuiChina wenyuz, hujian1, my1996mh,
<EMAIL_ADDRESS><EMAIL_ADDRESS>
(2021)

###### Abstract.

Graph convolutional networks (GCN) is widely used to handle irregular data
since it updates node features by using the structure information of graph.
With the help of iterated GCN, high-order information can be obtained to
further enhance the representation of nodes. However, how to apply GCN to
structured data (such as pictures) has not been deeply studied. In this paper,
we explore the application of graph attention networks (GAT) in image feature
extraction. First of all, we propose a novel graph generation algorithm to
convert images into graphs through matrix transformation. It is one magnitude
faster than the algorithm based on K Nearest Neighbors (KNN). Then, GAT is
used on the generated graph to update the node features. Thus, a more robust
representation is obtained. These two steps are combined into a module called
pixel-wise graph attention module (PGA). Since the graph obtained by our graph
generation algorithm can still be transformed into a picture after processing,
PGA can be well combined with CNN. Based on these two modules, we consulted
the ResNet and design a pixel-wise graph attention network (PGANet). The
PGANet is applied to the task of person re-identification in the datasets
Market1501, DukeMTMC-reID and Occluded-DukeMTMC (outperforms state-of-the-art
by 0.8%, 1.1% and 11% respectively, in mAP scores). Experiment results show
that it achieves the state-of-the-art performance. The code is available here.

graph convolutional networks, person re-identification

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
29th ACM International Conference on Multimedia; October 20–24, 2021; Virtual
Event, China††booktitle: Proceedings of the 29th ACM International Conference
on Multimedia (MM ’21), October 20–24, 2021, Virtual Event, China††price:
15.00††doi: 10.1145/3474085.3475640††isbn: 978-1-4503-8651-7/21/10††ccs:
Computing methodologies Image presentation††ccs: Computing methodologies
Visual content-based indexing and retrieval

##  1\. Introduction

Person re-identification attracts the attention of scholars. These studies
mainly use the image of a specific pedestrian as the query to find all the
images of the pedestrian in the datasets (gallery). The main application
fields of it include video surveillance, intelligent security and so on. There
are many challenges in person re-identification tasks: limited training data,
occlusion, lighting, scale, pose, and so on. How to obtain robust features of
the image is the key to solving these challenges.

Convolutional neural networks (CNN) has been successfully applied in the field
of person re-identification. Existing methods mainly adopt ResNet (He et al.,
2016) as the backbone to extract the features of images. However, due to the
limitation of CNN’s receptive field, the CNN based methods are difficult to
identify blurred pictures or small targets. To solve above problems, it is
necessary to give higher weight to human body information in the feature
extraction stage and extend the receptive field to obtain global information.

Since the attention mechanism (Vaswani et al., 2017) can get the correlation
between objects, and generate different weights. CNN with attention mechanism
(Wang et al., 2018; Hu et al., 2020; Woo et al., 2018) is a good way to solve
the above problems. In the field of person re-recognition, Some typical models
(Chen et al., 2019; Zhang et al., 2020) apply attention mechanism to the
feature maps output by CNN to calculate the importance of each feature in
feature maps and assign different weights to highlight important features and
suppress unimportant features. However, the existing methods only get the
relationship between nodes as one-order relationship (Chen et al., 2019) or
global relationship (Zhang et al., 2020), which can’t obtain the implicit
high-order relationship.

To expand the receptive field, dilated convolutions can be applied in the
feature extraction stage (Yu and Koltun, 2016), but the continuity of features
will be lost due to the discontinuity of convolution kernel. As a result, it
is difficult to obtain complete features of small objects.

Human beings get the overall cognition through the continuous expansion of a
certain point to the surroundings. Inspired by this ecology, graph attention
networks (GAT) (Velikovi et al., 2017) can be used to solve the above problems
by simulating the characteristics of human vision. The features of nodes are
propagated on the edges given different weights, and each node obtains a new
representation by aggregating the neighbor features. In this way, besides
important features will be assigned a higher weight, the range of receptive
field will be scalable.

Figure 1. We treat the picture as a regular graph and optimize the features of
the picture from the perspective of graph convolution.

picture transf to graph.

The premise of using GAT is to convert an image into a graph and obtain its
adjacency matrix. As shown in Fig. 1, a single pixel is regarded as a node,
and its adjacent pixels are neighbors. Based on this setting, we propose a
novel graph generation algorithm, which is not only faster than the
traditional algorithm but also obtains a sparse and strictly local graph.
Subsequently, using GAT on the obtained graph, not only can obtain high-order
information of neighbor nodes but also get a large receptive field. These two
steps are combined into a module called pixel-wise graph attention module
(PGA). Combined with ResNet, we build a new model pixel-wise graph attention
network (PGANet). And we apply the model to the commonly used data sets
Market1501(Zheng et al., 2015), DukeMTMC-reID(Ristani et al., 2016) and
Occluded-DukeMTMC(Miao et al., 2019) for person re-identification.

In summary, the contribution of this paper is threefold:

  * •

`We propose a novel graph generation algorithm`: The algorithm can transform
the image into a sparse and strictly local graph, which can make use of the
structure information. It is one order of magnitude faster than the
traditional KNN based algorithm.

  * •

`We propose a PGA module`: Multi-layer PGA can simulate human vision. With the
continuous iteration of PGA, the receptive field is continuously expanded and
high-level information is obtained.

  * •

`We propose a new model PGANet`: It is composed of CNN and PGA. We apply it to
person re-identification to demonstrate the effectiveness of our model.

##  2\. Related Work

In this section, we introduce the existing methods of image feature extraction
and the application of graph convolution networks in the field of computer
vision.

###  2.1. Image Feature Extraction Based On CNN

CNN based methods for person re-identification often use ResNet (Sun et al.,
2017; Hermans et al., 2017; Ristani et al., 2016) as the backbone to extract
image features. However, due to the limited receptive field, CNN can not
obtain global information. As a result, it is difficult to distinguish blurred
images and small targets. To solve this problem, some methods have been
proposed. The part-based models (Wen et al., 2016; Sun et al., 2018; Liu et
al., 2017; Zhao et al., 2017; Tian et al., 2018; Fu et al., 2019; Wang et al.,
2018), divide the picture into sub-pictures to get the part-level features. So
they can provide fine-grained information for better representation. But the
processed images are required to be aligned. Attention-based models (Wan et
al., 2020; Chen et al., 2019; Zhang et al., 2020) assign different importance
to different features in the feature extraction process. Therefore, it can
obtain the relationship between each pixel in the picture. But, the existing
methods only obtain one-order relationships or global information(Zhang et
al., 2020) and the implicit high-order information cannot be obtained. Dilated
convolutions can be applied in the feature extraction to overcome the limited
receptive field problem (Yu and Koltun, 2016), but the continuity of features
will be lost due to the discontinuity of the convolution kernel.

###  2.2. Image Feature Extraction Based On GCN

GCN aggregates neighbor node data along the direction of the edge to update
the new representation of the source node. It is mainly divided into two
categories: spectral domain approaches (Bruna et al., 2014; Defferrard et al.,
2016; Kipf and Welling, 2017) and spatial domain approaches (Velikovi et al.,
2017; Monti et al., 2017; Hamilton et al., 2017; Li et al., 2019). Because
node neighbors can be flexibly aggregated and it has lower computational
complexity, spatial GCN is the mainstream method. Hamilton et al.(Hamilton et
al., 2017) propose the GraphSAGE framework, which efficiently generates node
embeddings for previously invisible data using node feature information. In
GAT model (Velikovi et al., 2017), attention mechanism is adopted in the
propagation stage. Li et al. (Li et al., 2019) proposed that GCN can go deeper
as CNN by applying residual structure to alleviate the problem of vanishing
gradient. Recently, GCN has been widely used in the field of computer vision.
In the task of multi-label image recognition, Chen et al. (Chen et al., 2019)
propose a novel reweighting scheme to create an effective label correlation
matrix to guide the information dissemination between nodes in GCN. Context-
aware graph convolutional network (CAGCN) (Ji et al., 2021), in which the
relationship of the probe-gallery is encoded into the graph, processes the
hard samples through graph reasoning by using the context information flow and
other simple samples. However, GCN is rarely used in image feature extraction.
Defferrard et al. (Defferrard et al., 2016) proposed to apply a clustering
algorithm to convert the image into the graph, and then use GCN in the
generated graph. The computation complexity of the method to build a graph is
as high as $\mathbf{O(N^{2})}$. In the Dual Attention Network (DANet) model(Fu
et al., 2019), attention is applied between each pixel to establish a fully
connected graph, which was used to aggregate the features of nodes and applied
it in image segmentation. But the graph obtained in this way connects each
node to all nodes, ignoring the structural information.

##  3\. Methodology

PGANet is the backbone of image processing in our method. Its structure is
shown in Fig. 2 and 3. Firstly, we design a graph generation module to
transform the output of CNN into a graph. Then the graph is processed by GAT.
These two modules are built into a multi-layer PGA. Finally, multi-layer PGA
is inserted into Conv_x of ResNet. As shown in Fig. 2, a new framework PGANet
is constructed by stacking multi-layer PGA and CNN.

Figure 2. In PGANet, the features extracted by CNN are continuously optimized
in the PGA module. Ultimately get a more robust picture representation

architecture

###  3.1. Graph Generation Algorithm

The graph can be formulated as $G=<V,E,A>$, in which $V$ is the set of nodes
in the graph, $E$ is the edges between nodes in the graph, and the adjacency
matrix $A$ is usually used to represent the connection relationship of each
node. The feature maps output by CNN is expressed as $F\in\mathbb{R}^{H\times
W\times C}$, where H, W, and C represent the height, width, and channel
respectively. We regard the feature maps as regular graph. The pixel in the
feature map is regarded as the node in the graph, and its value in every
channel is the feature of the node, which is a C-dimensional feature vector.

(1) $A=\left\\{\begin{array}[]{ll}1,&v_{j}\text{ is the neighbor of }v_{i}\\\
0,&v_{j}\text{ isn't the neighbor of }v_{i}\end{array}\right.$

As shown in Eq. 1, a graph generation algorithm is needed to generate a sparse
and strictly local adjacency matrix $A$. The traditional method uses a
clustering algorithm to obtain the adjacency matrix of the graph, the time
complexity is as high as $\mathbf{O(N^{2})}$. We propose a novel graph
generation algorithm, which can efficiently find 4-neighbor, 8-neighbor, or
2-neighbor of nodes. In particular, the 2-neighbor’s graph is constructed in
the channel of the picture, thus each feature map is flattened as the feature
of the node. Our algorithm finds neighbors of nodes in parallel and does not
need to calculate the distance between nodes. The time complexity of algorithm
1 is $\mathbf{O(kHW)}$, since the $\mathbf{for}$ loop has $\mathbf{O(H)}$
complexity, the operation of $\mathbf{addition}$, $\mathbf{subtraction}$,
$\mathbf{slicing}$, and $\mathbf{assignment}$ in each loop has
$\mathbf{O(5kW)}$ complexity. We know that $\mathbf{N=H*W}$ and $\mathbf{k}$
is 2, 4, or 8. When $\mathbf{N}$ is very large, the time complexity of
algorithm 1 is smaller than the traditional algorithm,
$\mathbf{O(kHW)<<O(N^{2})}$. As shown in Algorithm 1, the algorithm which
generates a 4-neighbor graph is described.

Algorithm 1 Graph Generation Algorithm

1:$\mathtt{h}$ is the height of feature maps, $\mathtt{w}$ is the width of
feature maps,$\mathtt{n}=\mathtt{h}\times\mathtt{w}$ is the number of pixels
in the feature maps, AdjacencyGeneration($\mathbf{node},\mathbf{neighbor}$)is
a function to generate adjacency matrix based on node pairs.

2:$\mathbf{A}$ is Adjacency matrix of feature maps.

3:function GraphGeneration($\mathtt{h},\mathtt{w},\mathtt{n}$)

4: $\mathbf{M}\leftarrow$ range($\mathtt{n}$).reshape($\mathtt{h},\mathtt{w}$)

5: $\mathbf{r}\leftarrow\mathbf{M}$.rows ;

6: $\mathbf{node}\leftarrow[]$ ; $\mathbf{neighbor}\leftarrow[]$;

7: for $i=0\to\mathtt{h-1}$ do

8: $\mathbf{node}\leftarrow\mathbf{r_{i}}[1:]$

9: $\mathbf{neighbor}\leftarrow\mathbf{(r_{i}}-1)[1:]$

10: $\mathbf{node}\leftarrow\mathbf{r_{i}}[:-1]$

11: $\mathbf{neighbor}\leftarrow\mathbf{(r_{i}}+1)[:-1]$

12: if i != h-1 then

13: $\mathbf{node}\leftarrow\mathbf{r_{i}}$

14: $\mathbf{neighbor}\leftarrow\mathbf{(r_{i}}+w)$

15: end if

16: if i != 0 then

17: $\mathbf{node}\leftarrow\mathbf{r_{i}}$

18: $\mathbf{neighbor}\leftarrow\mathbf{(r_{i}}-w)$

19: end if

20: end for

21: $\mathbf{A}\leftarrow$
AdjacencyGeneration($\mathbf{node},\mathbf{neighbor}$)

22: return $\mathbf{A}$

23:end function

###  3.2. Pixel-wise Graph Attention Networks

PGANet is composed of multiple PGA modules connected in series. The operation
process of PGA is shown in Fig. 3. The self-attention mechanism utilized by
PGA closely follows the work of non-local operations (Wang et al., 2018). In
PGA, the propagation mechanism of node features is modified by GAT (Velikovi
et al., 2017).

Figure 3. Diagram of our proposed pixel-wise graph attention module (PGA). PGA
obtains the attention between each node through non-local operations. We use
GAT to update the features of each node.

Detail of PGA.

We first describe a single PGA. As shown in Fig. 3, the feature maps
$F\in\mathbb{R}^{C\times W\times H}$ output by Conv_x in ResNet are used as
the input of PGA. The corresponding adjacency matrix $A$ is generated based on
the graph generation algorithm proposed in Section 3.1. Subsequently, the
correlation matrix $R$ is obtained by calculating the correlation of each node
in the feature maps, as shown in Eq 2.

(2) $R=\theta\left(F\right)\phi\left(F\right)^{T}$

where $(\textperiodcentered)^{T}$ stands for transpose.
$\phi(\textperiodcentered)$ and $\theta(\textperiodcentered)$ are the same
transfer functions with different parameters, which including 1×1
convolutional layers, batch normalization, ReLU activation, and matrix
transformation. The transfer function changes the feature maps to new feature
space and transforms the new feature maps into a set of nodes through matrix
transformation, e.g., $V^{{}^{\prime}}=\phi(F)$, it can be defined as
$V^{\prime}=\left\\{v_{i}^{\prime}\in\mathbb{R}^{C^{\prime}},i=1,\ldots,N\right\\}$,
where $v^{{}^{\prime}}_{i}$ represents the $i^{th}$ node, $C^{{}^{\prime}}$ is
the transformed feature dimension, $N=H\times W$ is the number of all nodes in
the feature maps. As shown in the Fig. 3, we obtain a matrix of node
correlation $R\in\mathbb{R}^{N\times N}$ through the help of matrix change and
matrix multiplication.

The previous method (Fu et al., 2019) regards this correlation matrix $R$ as a
graph of feature maps. But $R$ is a fully connected graph, each node is
connected to all nodes. As a result, all structural information is dropped
(Velikovi et al., 2017). Therefore, we map the correlation matrix $R$ to the
adjacency matrix $A$. We obtain the adjacency matrix with correlation and use
the softmax function to normalize it. As shown in Eq. 3:

(3) $\tilde{A}=\operatorname{softmax}(A\odot R)$

As shown in Eq.3, different weights are assigned to all the edges on the
graph. We get a sparse and strictly local adjacency matrix
$\tilde{A}\in\mathbb{R}^{N\times N}$. The next step is to propagate the node
features on the graph. As shown in Eq. 4:

(4) $\tilde{V}=\operatorname{ReLU}(\tilde{A}V)$

The new feature is $\tilde{V}\in\mathbb{R}^{N\times C}$. By using matrix
transformation, it can be converted into feature maps again. Essentially, PGA
is a variant of GAT. The single-layer PGA can be expressed as:

(5)
$PGA\left(F\right)=\operatorname{transf}\left(\operatorname{softmax}\left(A\odot
R\right)\operatorname{transf}(F)\right).$

where the operation of function $transf(\textperiodcentered)$ is related to
the form of the input. If the form of the input data is $\mathbb{R}^{C\times
H\times W}$, we first transpose it and then execute function $reshape(-1,C)$.
If the form is $\mathbb{R}^{N\times C}$, we transpose it and then execute
function $reshape(H,W,C)$. As same as GCN, PGA has the problems of vanishing
gradient and over-smooth. We add the residual structure to the PGA with
regarding to DeepGCNs (Li et al., 2019). As shown in Eq. 6:

(6) $\tilde{F}=\alpha F+(1-\alpha)PGA\left(F\right)$

where $\alpha$ is a learnable parameter. Therefore, multi-layer PGA is shown
in Eq. 7:

(7) $\tilde{F}^{l+1}=\alpha F^{l}+(1-\alpha)PGA\left(F^{l}\right)$

###  3.3. Loss Function

(8) $L=L_{ID}+L_{\text{Triplet }}+\beta L_{C}$

AS shown in Eq. 8, the loss function of the model mainly includes ID loss,
triplet loss(Hermans et al., 2017), and Center loss(Wen et al., 2016).
Following strong baseline(Luo et al., 2019), ID Loss use label smoothing
strategy.

##  4\. Experiment

We introduced the implementation details of our model in Section 4.1. The
datasets and evaluation metrics adopted in the experiment are introduced in
section 4.2. In section 4.3, we compare our model with several state-of-the-
art methods in the task of person re-identification to verify the superiority
of it. In section 4.4, the effectiveness of the graph generation algorithm and
multi-layer PGA module is proved through ablation study.

###  4.1. Implementation Details

We use ResNet (He et al., 2016) to build the model. Refer to (Zhang et al.,
2020), we insert multi-layer PGA module after each Conv_x module (including
Conv_2, Conv_3, Conv_4 and Conv_5). In our model, the number of layers of PGA
is set to 3. The input image size is 256X128, and the padding value is set to
10. We use all the tricks summarized in Strong Baseline (Luo et al., 2019). To
expand the size of the feature maps output by the model, the last spatial
down-sampling operation is set to 1. The batch Standardization (BN) layer is
added before the classifier. For data augmentation, we apply random cropping,
horizontal flipping, and random erasing. The random probability for image
horizontal flip and random erasing is set to $0.5$ respectively. In our
experiments includes three losses as following: Identification loss(Szegedy et
al., 2016), Triplet loss(Schroff et al., 2015) and Center loss(Wen et al.,
2016). We adopt Adam optimizer with the weight decay of $5\times 10^{-4}$ and
the learning rate as $3e^{-4}$. The warm-up strategy is used for the learning
rate, and iterations of warm-up are set to 500. Finally, at the re-rank stage,
we use the k-reciprocal method(Zhong et al., 2017).

###  4.2. Datasets and Evaluation Metrics

We verify PGANet on the common datasets including Market1501(Zheng et al.,
2015), DukeMTMC-reID(Ristani et al., 2016) and Occluded-DukeMTMC(Miao et al.,
2019).

Market1501 It contains 1501 pedestrians captured by 6 cameras ( 5 high-
definition cameras and 1 low-definition camera) and 32668 detected pedestrian
rectangular boxes. Each pedestrian is captured by at least 2 cameras and may
have multiple images in one camera. The training set contains 751 people, with
12936 images, and each person has an average of 17.2 training data. In the
test set, there are 750 people with 19732 images, and each person has an
average of 26.3 test data.

DukeMTMC-reID It is a new type large high definition (HD) video data set
recorded by 8 synchronous cameras. The video is sampled every 120 frames to
form an image, and total 36, 411 images are obtained. A total of 1404 people
appeared under more than two cameras, and 408 people appeared under only one
camera. 702 people are randomly sampled into the training set and 1110 people
in the test set. In the test set, one photo under each camera of each ID is
sampled as a query image. The remaining images are added to the search gallery
of the test, and the previous 408 people are also added to the gallery as
interference items.

Occluded-DukeMTMC It derived from DukeMTMC-reID. In this dataset, all query
images are occluded by large variety of objects (e.g., trees, cars, other
persons). The training, query, and gallery set contain 14%/15%/10% occluded
images, respectively. The training set of Occluded DukeMTMC contains 15,618
images covering 702 identities in total. The testing set contains 1,110
identities, with 17, 661 gallery images and 2, 210 query images.

Evaluation Metrics Since cumulative matching characteristics (CMC) and mean
average precision (mAP) are commonly used metrics for person re-identification
tasks, they are also considered as evaluation indicators in this paper.

Table 1. Performance (%) comparisons with the state-of-the-arts methods on Market1501 and DukeMTMC-reID. Type | Method | Market1501 | DukeMTMC
---|---|---|---
mAP | Rank-1 | mAP | Rank-1
Pose-guided | PGFA (Miao et al., 2019) | 76.8 | 91.2 | 65.5 | 82.6
HOReI (Guan’an et al., 2020) | 84.9 | 94.2 | 75.6 | 86.9
Part-feature | BDB (Dai et al., 2019) | 84.3 | 94.2 | 72.1 | 86.8
Pyramid(Zheng et al., 2019) | 88.2 | 95.7 | 79 | 89
MGN(Wang et al., 2018) | 86.9 | 95.7 | 78.4 | 88.7
AANet-152 (Tay et al., 2019) | 83.41 | 93.93 | 74.29 | 87.65
| ISP (Zhu et al., 2020) | 88.6 | 95.3 | 80 | 89.6
Attention | ABD-Net(Chen et al., 2019) | 88.2 | 95.6 | 78.5 | 89
RGA-SC (Zhang et al., 2020) | 88.1 | 95.8 | 74.9 | 86.1
Mancs (Wang et al., 2018) | 82.3 | 93.1 | 71.8 | 84.9
Other | Baseline (Luo et al., 2019) | 85.9 | 94.5 | 76.4 | 86.4
CAGCN (Ji et al., 2021) | 91.7 | 95.9 | 85.9 | 91.3
Ours | PGANet-50 | 88.5 | 95.4 | 78.8 | 89.4
PGANet-152 | 89.3 | 95.4 | 79.6 | 89.1

###  4.3. Comparison With State-of-the-Art Methods

PGANet versus Other Model. To prove the effectiveness of PGANet. We compare it
with several state-of-the-art models in person re-identification task. As
shown in Table 1, our model achieves competitive performance in rank stage. We
tested the effects of PGANet using ResNet50 and ResNet152 as backbones
respectively. Compared with the attention-based model, PGANet mixes graph
structure and attention to optimize features, thus can obtain hidden high-
order information and highlight important features well. Compared with part-
based models, in PGANet, global features can be obtained to achieve
competitive performance. With the help of the part-based method, we can
achieve better performance. Without any additional models, PGA module can more
easily obtain important features than Pose-guided models. CAGCN can obtain
better performance by using probe-gallery relations with the help of picture-
wise GCN. So we can draw the conclusion that GCN has great potential in the
field of computer vision.

Table 2. Performance (%) comparisons with the state-of-the-arts methods on Occluded-Duke. Method | Occluded-Duke
---|---
mAP | Rank-1
Ad-Occluded (Huang et al., 2018) (CVPR2018) | 32.2 | 44.5
PGFA (Miao et al., 2019) (ICCV2019) | 37.3 | 51.4
HOReID (Guan’an et al., 2020) (CVPR2020) | 43.8 | 55.1
PGANet-50 | 50.5 | 59.7
PGANet-152 | 54.8 | 65.2

Experiment on Occluded-Duke As shown in Table2, PGANet achieves mAP 54.8% and
Rank-1 65.2% on Occluded-Duke. Compared with HOReID, it has an increase of 11
points. The results show that PGANet can give higher attention to important
features, thus obtain more robust representation even in complex scenes

Table 3. Performance (%) comparisons with the state-of-the-arts in rerank Method | Market1501 | DukeMTMC
---|---|---
mAP | Rank-1 | mAP | Rank-1
Strong Baseline+RR (Luo et al., 2019) (CVPR2019) | 94.2 | 95.2 | 89.1 | 90.3
AANet-152+RR (Tay et al., 2019) (CVPR2020) | 92.38 | 95.1 | 86.87 | 90.36
CAGCN+RR (Ji et al., 2021) (AAAI2021) | 94.0 | 96.2 | 88.9 | 92.3
PGANet-50+RR | 94.6 | 96.0 | 89.9 | 91.5
PGANet-152+RR | 94.8 | 96.2 | 90.0 | 92.0

Re-rank experiment As shown in Table 3, our model also achieves excellent
performance (mAP 94.6% and Rank-1 96.0%). In the re-rank stage, the
k-reciprocal method (Zhong et al., 2017) is adopted as post-processing in our
model, which optimizes the results by making effective use of the relationship
between the ranking results and the query images. Why does our model achieve
state-of-the-art performance after using the re-rank strategy? We compare our
model with the best model CAGCN in Table 1. The core idea of CAGCN is to take
advantage of the probe-gallery relations, which is consistent with the idea of
k-reciprocal method. It essentially uses information between pictures. Our
model focuses on optimizing the features of images and ignores the
relationship between images. Therefore, Once the relationship between pictures
is also used, the model effect has been greatly improved. The experimental
results also prove that our model can obtain more robust features.

###  4.4. Ablation Study

Advantages of Graph Generation algorithm To test the efficiency of our
algorithm, we compare the algorithm with the traditional KNN based algorithm.
The size of the feature map used for testing is [16,8,512]. We evaluate the
time required to create three graphs in which nodes have different numbers of
neighbors, ie. 2-neighbor, 4-neighbor, and 8-neighbor. In 2-neighbor, the
graph is built on the channel of the picture, and in 4-neighbor and
8-neighbor, is built on the pixels of the picture. The values are obtained as
the average of running time of the algorithm, and drawn as a line graph. As
shown in Fig. 4, the algorithm proposed in this paper increases the speed of
graph generation by an order of magnitude, at least 12 times faster than a
traditional KNN based algorithm.

Figure 4. The speed of algorithm.

Figure 5. The effect of different layers of multi-layer PGA on the results.

Advantage of multi-layer PGA In this section, we test the influence of the
number of layers of PGA on the model. Fig. 5 shows continuous enhancement of
the effect with the increase of the number of PGA layers. Compared with strong
baseline, the mAP of the 3-layer model has increased by 2.6%, and the accuracy
of Rnk-1 has increased by 0.9% (based on ResNet50). The experimental results
show that the larger receptive field effectively improves the model’s
representation ability and can obtain hidden high-level information as the
number of PGA layers increases. As shown in Fig. 5 and 6, even if the model
has been trained for many rounds, we can not find the phenomenon of vanishing
gradient and over smooth. It is noticeable that PGA with residual structure
can effectively alleviate the vanishing gradients problem.

Figure 6. The impact of using different graphs on the model.

Influence of neighbors In PGA, graph convolution to optimize feature maps for
better representation. In fully connected graphs, the non local method is used
for nodes to obtain the one-to-all relationship. As shown in Fig. 6, better
performance is gotten since structural information can be obtained by the
model, with the help of the better sparse and strict local graph produced by
our algorithm.

##  5\. Conclusion

In this paper, we explored the application of GCN in the image feature
extraction stage. A new image feature extraction framework PGANet is proposed,
which can obtain a larger receptive field and make good use of structural
information to obtain hidden high-order information. PGANet uses an alternate
structure of CNN and multi-layer PGA.CNN is used to extract image features,
multi-layer PGA is used for the obtained feature maps to optimize features and
expand the receptive field. Therefore, PGANet can get a more robust
representation of the picture. To apply GCN in the feature extraction stage of
the picture, a brand-new graph generation algorithm is proposed which can
efficiently generate a sparse and strictly local graph corresponding to the
picture. Combine our algorithm with the modified GAT, we get the PGA module.
With iteration of PGA, high-ord information can be obtained. Extensive
experiments on person re-identification demonstrate the effectiveness of our
proposed framework.

###### Acknowledgements.

This work was supported by the National Key Research and Development Program
of China under Grant 2020YFB2103803.

## References

  * (1)
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ 2016-December, 770 – 778.  http://dx.doi.org/10.1109/CVPR.2016.90
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is all you need.  _Advances in Neural Information Processing Systems_ 2017-December, 5999 – 6009.
  * Wang et al. (2018) Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018.  Non-local Neural Networks.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ (2018), 7794 – 7803.
  * Hu et al. (2020) Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. 2020.  Squeeze-and-Excitation Networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 42, 8 (2020), 2011 – 2023.
  * Woo et al. (2018) Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. 2018.  Cbam: Convolutional block attention module. In _Proceedings of the European conference on computer vision (ECCV)_. 3–19.
  * Chen et al. (2019) Guangyi Chen, Chunze Lin, Liangliang Ren, Jiwen Lu, and Jie Zhou. 2019.  Self-critical attention learning for person re-identification.  _Proceedings of the IEEE International Conference on Computer Vision_ 2019-October, 9636 – 9645.  http://dx.doi.org/10.1109/ICCV.2019.00973
  * Zhang et al. (2020) Zhizheng Zhang, Cuiling Lan, Wenjun Zeng, Xin Jin, and Zhibo Chen. 2020.  Relation-Aware Global Attention for Person Re-Identification.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 3183 – 3192.  http://dx.doi.org/10.1109/CVPR42600.2020.00325
  * Yu and Koltun (2016) Fisher Yu and Vladlen Koltun. 2016.  Multi-scale context aggregation by dilated convolutions.  _4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings_.
  * Velikovi et al. (2017) Petar Velikovi, Cucurul Guillem, Casanova Arantxa, Romero Adriana, and Bengio Pietro, Lio‘and Yoshua. 2017\.  Graph Attention Networks.  _ICLR_ (2017).
  * Zheng et al. (2015) Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. 2015\.  Scalable person re-identification: A benchmark.  _Proceedings of the IEEE International Conference on Computer Vision_ 2015 International Conference on Computer Vision, ICCV 2015, 1116 – 1124.  http://dx.doi.org/10.1109/ICCV.2015.133
  * Ristani et al. (2016) Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. 2016.  Performance measures and a data set for multi-target, multi-camera tracking.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ 9914 LNCS, 17 – 35.  http://dx.doi.org/10.1007/978-3-319-48881-3_2
  * Miao et al. (2019) Jiaxu Miao, Yu Wu, Ping Liu, Yuhang DIng, and Yi Yang. 2019.  Pose-guided feature alignment for occluded person re-identification.  _Proceedings of the IEEE International Conference on Computer Vision_ 2019-October, 542 – 551.  http://dx.doi.org/10.1109/ICCV.2019.00063
  * Sun et al. (2017) Yifan Sun, Liang Zheng, Weijian Deng, and Shengjin Wang. 2017\.  SVDNet for Pedestrian Retrieval.  _Proceedings of the IEEE International Conference on Computer Vision_ 2017-October (2017), 3820 – 3828.  http://dx.doi.org/10.1109/ICCV.2017.410
  * Hermans et al. (2017) Alexander Hermans, Lucas Beyer, and Bastian Leibe. 2017\.  In Defense of the Triplet Loss for Person Re-Identification.  arXiv:1703.07737 [cs.CV]
  * Wen et al. (2016) Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. 2016\.  A Discriminative Feature Learning Approach for Deep Face Recognition. In _European Conference on Computer Vision_.
  * Sun et al. (2018) Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang. 2018.  Beyond Part Models: Person Retrieval with Refined Part Pooling (and A Strong Convolutional Baseline).  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ 11208 LNCS (2018), 501 – 518.  http://dx.doi.org/10.1007/978-3-030-01225-0_30
  * Liu et al. (2017) Xihui Liu, Haiyu Zhao, Maoqing Tian, Lu Sheng, Jing Shao, Shuai Yi, Junjie Yan, and Xiaogang Wang. 2017\.  HydraPlus-Net: Attentive Deep Features for Pedestrian Analysis.  _Proceedings of the IEEE International Conference on Computer Vision_ 2017-October (2017), 350 – 359.
  * Zhao et al. (2017) Liming Zhao, Xi Li, Yueting Zhuang, and Jingdong Wang. 2017\.  Deeply-Learned Part-Aligned Representations for Person Re-identification.  _Proceedings of the IEEE International Conference on Computer Vision_ 2017-October, 3239 – 3248.  http://dx.doi.org/10.1109/ICCV.2017.349
  * Tian et al. (2018) Yumin Tian, Qiang Li, Di Wang, and Bo Wan. 2018\.  Robust joint learning network: improved deep representation learning for person re-identification.  _Multimedia Tools and Applications_ 78, 17 (2018), 24187–24203.
  * Fu et al. (2019) Yang Fu, Yunchao Wei, Yuqian Zhou, Honghui Shi, Gao Huang, Xinchao Wang, Zhiqiang Yao, and Thomas Huang. 2019\.  Horizontal pyramid matching for person re-identification.  _33rd AAAI Conference on Artificial Intelligence, AAAI 2019_ (2019), 8295 – 8302\.
  * Wang et al. (2018) Guanshuo Wang, Yufeng Yuan, Xiong Chen, Jiwei Li, and Xi Zhou. 2018.  Learning discriminative features with multiple granularities for person re-identification.  _MM 2018 - Proceedings of the 2018 ACM Multimedia Conference_ , 274 – 282.  http://dx.doi.org/10.1145/3240508.3240552
  * Wan et al. (2020) Chaoqun Wan, Yue Wu, Xinmei Tian, Jianqiang Huang, and Xian-Sheng Hua. 2020.  Concentrated Local Part Discovery with Fine-Grained Part Representation for Person Re-Identification.  _IEEE Transactions on Multimedia_ 22, 6 (2020), 1605 – 1618.
  * Bruna et al. (2014) Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2014.  Spectral networks and deep locally connected networks on graphs.  _2nd International Conference on Learning Representations, ICLR 2014 - Conference Track Proceedings_ (2014).
  * Defferrard et al. (2016) Michael Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016\.  Convolutional neural networks on graphs with fast localized spectral filtering.  _Advances in Neural Information Processing Systems_ 0, 3844 – 3852.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  _5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings_ (2017).
  * Monti et al. (2017) Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M. Bronstein. 2017.  Geometric deep learning on graphs and manifolds using mixture model CNNs.  _Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017_ 2017-January, 5425 – 5434.  http://dx.doi.org/10.1109/CVPR.2017.576
  * Hamilton et al. (2017) William L. Hamilton, Rex Ying, and Jure Leskovec. 2017\.  Inductive representation learning on large graphs.  _Advances in Neural Information Processing Systems_ 2017-December, 1025 – 1035.
  * Li et al. (2019) Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. 2019.  DeepGCNs: Can GCNs go as deep as CNNs?  _Proceedings of the IEEE International Conference on Computer Vision_ 2019-October, 9266 – 9275.  http://dx.doi.org/10.1109/ICCV.2019.00936
  * Chen et al. (2019) Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. 2019\.  Multi-label image recognition with graph convolutional networks.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ 2019-June, 5172 – 5181.  http://dx.doi.org/10.1109/CVPR.2019.00532
  * Ji et al. (2021) Deyi Ji, Haoran Wang, Hanzhe Hu, Weihao Gan, Wei Wu, and Junjie Yan. 2021\.  Context-Aware Graph Convolution Network for Target Re-identification.  _AAAI 2021_ (2021).  arXiv:2012.04298 [cs.CV]
  * Fu et al. (2019) Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. 2019.  Dual attention network for scene segmentation.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ 2019-June, 3141 – 3149.  http://dx.doi.org/10.1109/CVPR.2019.00326
  * Luo et al. (2019) Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. 2019.  Bag of tricks and a strong baseline for deep person re-identification.  _IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops_ 2019-June, 1487 – 1495.  http://dx.doi.org/10.1109/CVPRW.2019.00190
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016.  Rethinking the Inception Architecture for Computer Vision.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ 2016-December (2016), 2818 – 2826.  http://dx.doi.org/10.1109/CVPR.2016.308
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015\.  FaceNet: A unified embedding for face recognition and clustering.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ 07-12-June-2015, 815 – 823.  http://dx.doi.org/10.1109/CVPR.2015.7298682
  * Zhong et al. (2017) Zhun Zhong, Liang Zheng, Donglin Cao, and Shaozi Li. 2017\.  Re-ranking person re-identification with k-reciprocal encoding.  _Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017_ 2017-January, 3652 – 3661.  http://dx.doi.org/10.1109/CVPR.2017.389
  * Guan’an et al. (2020) Wang Guan’an, Shuo Yang, Huanyu Liu, Zhicheng Wang, Yang Yang, Shuliang Wang, Gang Yu, Erjin Zhou, and Jian Sun. 2020.  High-order information matters: Learning relation and topology for occluded person re-identification.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 6448 – 6457.  http://dx.doi.org/10.1109/CVPR42600.2020.00648
  * Dai et al. (2019) Zuozhuo Dai, Mingqiang Chen, Xiaodong Gu, Siyu Zhu, and Ping Tan. 2019.  Batch dropblock network for person re-identification and beyond.  _Proceedings of the IEEE International Conference on Computer Vision_ 2019-October, 3690 – 3700.  http://dx.doi.org/10.1109/ICCV.2019.00379
  * Zheng et al. (2019) Feng Zheng, Cheng Deng, Xing Sun, Xinyang Jiang, Xiaowei Guo, Zongqiao Yu, Feiyue Huang, and Rongrong Ji. 2019\.  Pyramidal person re-identification via multi-loss dynamic training.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ 2019-June, 8506 – 8514.  http://dx.doi.org/10.1109/CVPR.2019.00871
  * Tay et al. (2019) Chiat-Pin Tay, Sharmili Roy, and Kim-Hui Yap. 2019.  Aanet: Attribute attention network for person re-identifications.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ 2019-June, 7127 – 7136.
  * Zhu et al. (2020) Kuan Zhu, Haiyun Guo, Zhiwei Liu, Ming Tang, and Jinqiao Wang. 2020.  Identity-Guided Human Semantic Parsing for Person Re-identification.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ 12348 LNCS, 346 – 363.  http://dx.doi.org/10.1007/978-3-030-58580-8_21
  * Chen et al. (2019) Tianlong Chen, Shaojin DIng, Jingyi Xie, Ye Yuan, Wuyang Chen, Yang Yang, Zhou Ren, and Zhangyang Wang. 2019\.  ABD-net: Attentive but diverse person re-identification.  _Proceedings of the IEEE International Conference on Computer Vision_ 2019-October, 8350 – 8360.  http://dx.doi.org/10.1109/ICCV.2019.00844
  * Wang et al. (2018) Cheng Wang, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. 2018.  _Mancs: A Multi-task Attentional Network with Curriculum Sampling for Person Re-Identification_. Vol. 11208 LNCS.  Munich, Germany. 384 – 400 pages.  http://dx.doi.org/10.1007/978-3-030-01225-0_23
  * Huang et al. (2018) Houjing Huang, Dangwei Li, Zhang Zhang, Xiaotang Chen, and Kaiqi Huang. 2018.  Adversarially Occluded Samples for Person Re-identification.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 5098 – 5107.  http://dx.doi.org/10.1109/CVPR.2018.00535


# Self-supervised Learning for Human Activity Recognition Using 700,000
Person-days of
Wearable Data

Hang Yuan1,2,4* Shing Chan1,2* Andrew P. Creagh2,3 Catherine Tong4
David A. Clifton3 Aiden Doherty1,2
1 Nuffield Department of Population Health, University of Oxford
2 Big Data Institute, University of Oxford
3 Department of Engineering Science, University of Oxford
4 Department of Computer Science, University of Oxford
Equal contributions. Correspondence to<EMAIL_ADDRESS>The code
and model can be downloaded from https://github.com/OxWearables/ssl-wearables.

###### Abstract

Advances in deep learning for human activity recognition have been relatively
limited due to the lack of large labelled datasets. In this study, we leverage
self-supervised learning techniques on the UK-Biobank activity tracker
dataset–the largest of its kind to date–containing more than 700,000 person-
days of unlabelled wearable sensor data. Our resulting activity recognition
model consistently outperformed strong baselines across seven benchmark
datasets, with an F1 relative improvement of 2.5%-100% (median 18.4%), the
largest improvements occurring in the smaller datasets. In contrast to
previous studies, our results generalise across external datasets, devices,
and environments. Our open-source model will help researchers and developers
to build customisable and generalisable activity classifiers with high
performance.

##  1 Introduction

Current human activity recognition (HAR) models typically rely on manual
feature engineering (Twomey et al.,, 2018; Haresamudram et al.,, 2019) partly
due to the very limited size of existing labelled datasets. This small data
issue caps the effectiveness of data-hungry deep learning methods. In general,
obtaining _labelled_ data is labour intensive, but it is especially so for HAR
data because one would need to annotate the corresponding video stream for the
ground truth. On the other hand, collecting large-scale _unlabelled_ HAR data
is highly feasible, as evidenced by projects such as the UK-Biobank (UKB)
(Doherty et al.,, 2017) and NHANES (Belcher et al.,, 2021). This prompts the
use of self-supervised learning (SSL) methods to leverage unlabelled data in a
similar manner as language models (Mikolov et al.,, 2013; Devlin et al.,,
2018; Radford et al.,, 2018, 2019) and vision models (Doersch et al.,, 2015;
Zhang et al.,, 2016; Noroozi and Favaro,, 2016; Wei et al.,, 2018; He et al.,,
2020; Chen et al.,, 2020). Recent studies explored the utility of SSL for HAR
(Saeed et al.,, 2019; Tang et al.,, 2020), but these still relied on small-
scale laboratory-style datasets, hence the full potential of SSL-HAR remains
unknown.

In this paper, we investigate how learning three simple self-supervised
learning tasks independently and jointly could facilitate HAR across a variety
of environments using the UKB dataset, which contains terabytes of wearable
sensor data collected in the real world. To fully appreciate the benefit of
SSL in HAR, we chose the tasks that would prioritise the temporal dependencies
of human motion, namely, _arrow of time_ (AoT), _permutation_ and _time-
warping_ (TW) (Um et al.,, 2017). Figure 1 illustrates an example of these
transformations for a sequence of tri-axial accelerometer data. We show for
the first time that multi-task SSL can train a HAR model that generalises well
across seven external datasets that differ in activity classes, devices,
populations and recording environments. In contrast to previous works, we
provide a more realistic evaluation of the utility of SSL-HAR by factoring in
common issues seen in the practical use cases of pre-trained models such as
domain shift and task shift (Quiñonero-Candela et al.,, 2008). Our publicly
available models will enable the research community to build high-performing
activity recognition models even in a resource-restricted environment.

Figure 1: A ten-second window of tri-axial accelerometer data when a person is
ironing in its original form and three corresponding transformations:
reversed, permuted, and time-warped.

##  2 Related work

Activity recognition from on-body inertial sensors is a fundamental problem in
wearable computing. HAR models are usually built following the Activity
Recognition Chain (Bulling et al.,, 2014), which describes the pre-processing
and segmentation of raw time series into fixed-length frames, from which
features are extracted for classification. Early work in HAR found some
success in tree-based approaches with hand-crafted features (Hammerla et al.,,
2013). More recent works focused on translating advances in deep learning into
HAR (Yang et al.,, 2015; Hammerla et al.,, 2016; Ronao and Cho,, 2016) but
were limited by small dataset sizes. Indeed, some studies found that earlier
approaches, such as random forests, tended to generalise better than deep
learning models in certain small data regimes (Willetts et al.,, 2018; Kwon et
al.,, 2020).

Recent studies explored the use of SSL for HAR, such as forecasting (Taghanaki
and Etemad,, 2020), masked reconstruction (Haresamudram et al., 2020a, ),
contrastive learning (Haresamudram et al., 2020b, ; Tang et al.,, 2020), and
multi-task SSL (Saeed et al.,, 2019; Tang et al.,, 2021). However, these
studies were still limited by small laboratory-style datasets
($n<10{\small,}000$), which confounded their findings. In comparison, we
leverage the largest ever collection of wrist-worn raw accelerometer data
($n>6\;\textrm{billion}$) collected in the real world. Aside from being
several orders of magnitude larger, this dataset is vastly more diverse as it
contains hundreds, if not thousands, of natural human activities – a crucial
aspect regarding the input distribution and generalisation. Therefore, we aim
to provide a more definitive investigation of the utility of SSL for HAR.

##  3 Methods

Table 1: Wrist-worn accelerometer datasets used to evaluate the utility of
self-supervised learning for human activity recognition tasks
Dataset |  #Subjects |  #Samples |  #Classes |  Environment | References
---|---|---|---|---|---
UK-Biobank |  $\sim$100K |  6 B |  Unlabelled |  Free-living |  Doherty et al., (2017)
Capture-24 |  152 |  573K |  4 |  Free-living |  Willetts et al., (2018)
Rowlands |  55 |  36K |  13 |  Lab |  Esliger et al., (2011)
WISDM |  46 |  28K |  18 |  Semi free-living |  Weiss et al., (2019)
REALWORLD |  14 |  12K |  8 |  Lab |  Sztyler and Stuckenschmidt, (2016)
Opportunity |  4 |  3.9K |  4 |  Semi free-living |  Roggen et al., (2010)
PAMAP2 |  8 |  2.9K |  8 |  Lab |  Reiss and Stricker, (2012)
ADL |  7 |  0.6K |  5 |  Lab |  Bruno et al., (2013)

We used tri-axial accelerometer data from wrist-worn activity trackers, which
record acceleration on three orthogonal axes at a high sampling rate (e.g.
$100\text{\,}\mathrm{Hz}$). The main benefit of wrist-worn activity trackers
is their high user compliance, resulting in days, if not weeks, of continuous
recordings. Following Bulling et al., (2014), we split the signals into
windows of equal duration, effectively treating them as independent inputs to
the HAR models. We can then label each window with an activity class.
Throughout this study, we linearly resampled all data to
$30\text{\,}\mathrm{Hz}$ resolution and used ten-second-long windows to
compare the downstream benchmarks fairly. The $30\text{\,}\mathrm{Hz}$
sampling rate was used because most human activities have a frequency less
than $10\text{\,}\mathrm{Hz}$. We used a sampling rate that is higher than the
presumed Nyquist rate ($20\text{\,}\mathrm{Hz}$) to ensure that we did not
lose any useful signal.

###  3.1 Datasets

Our multi-task SSL training relied on the _unlabelled_ UKB dataset, which
contains roughly 700,000 person-days of free-living activity data (>100,000
participants, seven days of wear). The free-living aspect is important because
the data can contain all sorts of activities, as opposed to lab data which are
constrained to scripted activities only. The UKB data (project ref Anonymous)
is covered by ethical approval from the NHS National Research Ethics.

For the subsequent activity recognition benchmarks, we considered seven
external _labelled_ datasets that vary in size (600 to 600,000 samples),
activity classes (4 to 18 classes), devices used (five different brands), and
collection protocol (free-living, scripted, and lab settings). See Table 1 for
additional details. Three datasets had license information, and four datasets
had explained informed consent information (Appx. Table 5). We removed the
classes that were not present in all the subjects in small datasets with less
than 10 individuals during data cleaning.

Even though we reused existing datasets, we made our best effort to enumerate
the license and consent information for all the included datasets, as our data
involved human subjects. We observed that many open benchmark datasets that we
used did not have suitable licencing or consent information, possibly due to
the lack of data governance awareness at the time of the study.

###  3.2 Multi-task self-supervised-learning

We considered three self-supervised tasks from Um et al., (2017) who used them
as data augmentation techniques for motor state classification in Parkinson’s
patients.

Arrow of time (AoT) flips the signal along the time axis, effectively playing
the signal in reverse.

Permutation breaks the signal into chunks and shuffles them. We set the number
of chunks to four and the minimum length of each chunk to at least 10
timestamps.

Time warping (TW) stretches and compresses arbitrary segments of the signal,
effectively slowing down and speeding up the signal randomly.

Following Saeed et al., (2019), we treated each of the tasks as a binary
problem predicting whether a transformation has been applied. In the multi-
task learning (MTL) setting, not all the tasks might benefit HAR when trained
jointly, so we assessed how different task combinations could influence the
downstream performance (Section 4.2). We computed the cross-entropy loss for
each task and weighed all the tasks equally in the loss calculation.

##### Weighted sampling

Motion data collected in the real world contains large portions of low
movement periods that are less informative (Appx. Figure 6), which is an issue
for our SSL tasks as static signals remain virtually unchanged after the
transformations. We found it crucial to perform weighted sampling for improved
training stability and convergence: During training, we sample the data
windows in proportion to their standard deviation so as to give more weight to
high-movement periods.

###  3.3 Network training

We adapted a ResNet-V2 with 18 layers and 1D convolutions (He et al.,, 2016)
for the main trunk (feature extractor), totalling 10M parameters. The learned
feature vector was of size 1024. All the tasks shared the same feature
extractor. Then, we attached a softmax layer for each of the self-supervised
tasks. In the downstream evaluation, we added a fully-connected (FC) layer of
size 512 in between the feature extractor and softmax readout. The network
structure was fixed for all the downstream evaluations.

For SSL, we load up to four subjects from the UKB at each iteration. For each
subject, we first sampled one day out of the week-long data, from which we
again sampled 1500 10-second windows to make up a training batch. Self-
supervised transformations were then applied to the batch of data. Since the
axis orientation differs between device manufacturers, we used random axis
swaps and rotations to augment the training data to embed this invariance into
our models. For optimisation, we used Adam (Kingma and Ba,, 2014) with a
learning rate of 1e-3. To account for large batch sizes, $1500\times 4=6000$,
we applied linear scaling for the learning rate with 5 epochs as burn-in
(Goyal et al.,, 2017). We distributed the network training over four Tesla
V100-SXM2 with 32GB of memory. It took about 420 GPU hours to train the MTL
model (about 20 epochs). We used an 8:2 ratio for the train/test split for all
the self-supervised experiments. For fine-tuning, we used the same training
setup as the pre-training where possible except for the batch size, which was
re-adjusted depending on the size of each dataset.

###  3.4 Evaluation – human activity recognition

To evaluate the downstream (HAR) performance, We used held-one-subject-out
cross-validation for the datasets that had $<10$ subjects. We additionally
removed activity classes not done by all the subjects in these small datasets.
For datasets with $\geq 10$ subjects, we used five-fold subject-wise cross-
validation instead. Each cross-validation had a 7:1:2 split ratio for
train/validation/test sets. We used early-stopping with a patience of five to
avoid over-fitting.

After the network was trained on the UKB using $\sim$100,000 participants, we
further fine-tuned the network on the seven labelled downstream datasets to
perform human activity detection using two approaches: (1) fine-tuning all the
layers (2) freezing the trunk (feature extractor) and fine-tuning only the FC
layers in the end. We also report the model performance for a network of the
same architecture but fully trained from scratch, and a strong random forest
model with tried-and-tested time series features, which has often been
neglected in baseline model comparisons (Zhang et al.,, 2012; Mannini et al.,,
2013; Ellis et al.,, 2016; Willetts et al.,, 2018). See Appx. A.3 for the list
of features used.

In addition, a shared implementation was introduced for our network training,
model evaluation and preprocessing. Differences in experiment setup such as
training rates, regularisation and data augmentation can lead to inconsistent
results (Oliver et al.,, 2018). A unified evaluation framework would ensure a
fair comparison between different baseline models. Our evaluation framework
contrasts with previous work, where there is no fixed evaluation protocol
across the benchmark datasets, making it hard to compare model performance
with the current state-of-the-art. The results produced in our paper would
serve as the baseline for future HAR research.

##### Transfer learning

Pre-training on a larger labelled-datset and fine-tuning on a smaller dataset
is a common technique in practical application that has been under-reported as
a baseline for SSL. The success of transfer learning, however, depends on how
similar the source and target datasets are. Hence, we included experiments
using the two largest labelled datasets, Capture-24 and Rowlands for pre-
training, which were then fine-tuned on other labelled datasets.

##### The benefits of data volume

In the ablation studies, we investigated how the downstream performance differ
on two axes, the amount of labelled data and the amount of unlabelled data.
Concretely, we gradually increase the number of labelled subjects in both
Capture-24 and Rowlands in the downstream evaluation to assess whether our
pre-trained model can still do well in a limited-data regime. In terms of
unlabelled data, we experimented with pre-training that had 100 to 100,0000
participants with one order of magnitude increment. We also varied the amount
of unlabelled data per subject from .25 to 1 using 10,000 participants. A data
ratio of .25 means that if one day of data per subject was used previously,
then only six hours of data per subject would now be used for training.
Investigating how unlabelled data influences downstream performance will guide
how much data one needs to have to obtain an effective SSL model for HAR.

###  3.5 Understanding network representation

##### Contextualising layer-wise relevance propagation

We applied layer-wise relevance propagation (LRP) to visually investigate the
signal characteristics relevant for detecting the pretext tasks (Montavon et
al.,, 2019; Creagh et al.,, 2021). It is inherently more difficult to visually
interpret attribution heatmaps generated through Explainable AI (XAI)
frameworks on time-series signals. To overcome this lack of visual ground
truth, we devised a set of simple contextual experiments to evaluate our LRP
attribution results. Using the same accelerometer as the UKB, we recorded a
participant performing two activities under video observation: (1) low
intensity scripted (hand-shaking) and (2) high intensity unscripted (playing
tennis). We acquired a ground truth (the context) for the accelerometer
activity through the time-synced video observations, enabling a better visual
interpretation of the sensor-based characteristics attributed as relevant for
detecting different pretexts. Holistic interpretations were formed based on
visualising the raw sensor signal, its analogues time-frequency representation
through continuous wavelet transform (CWT) scalograms (Addison et al.,, 2009),
as well as the time- and pretext task-localised LRP relevance scores, all with
respect to observing the concurrent video recordings. Details on the XAI
contextual LRP (cLRP) framework are described in Appx. A.3.1.

##  4 Results

###  4.1 Weighted single task training

Figure 2: Accuracy test curves for training four self-supervised tasks
individually using 1000 subjects from the UK-Biobank with and without weighted
sampling. The patience for early-stopping was five.

When training individual pretext tasks, we found that without weighted
sampling, all the tasks had worse convergence behaviour (Figure 2). The
performance degradation was most pronounced for the AoT and permutation. The
test performance for the AoT stayed at the random chance level, and the test
performance for permutation dropped roughly 10 percentage points without
weighted sampling.

###  4.2 Multi-task self-supervised learning

Table 2: Downstream human activity recognition performance (subject-wise F1
($\pm$SD)) for different self-supervised task combinations using 1,000 UK-
Biobank participants. $N$ is the number of samples.
AoT | Permutation | TW | Capture-24 | Rowlands | Opportunity
---|---|---|---|---|---
n=573K | n=36k | n=3.9K
Single task |  |  |  |  |
✓ | ✗ | ✗ |  .671 $\pm$ .094 |  .565 $\pm$ .120 |  .582 $\pm$ .054
✗ | ✓ | ✗ |  .721 $\pm$ .093 |  .783 $\pm$ .099 | .588 $\pm$ .076
✗ | ✗ | ✓ |  .715 $\pm$ .093 |  .776 $\pm$ .110 |  .584 $\pm$ .064
Multi-task |  |  |  |  |
✗ | ✓ | ✓ |  .714 $\pm$ .094 |  .755 $\pm$ .103 |  .587 $\pm$ .070
✓ | ✗ | ✓ | .719 $\pm$ .094 |  .762 $\pm$ .102 |  .530 $\pm$ .071
✓ | ✓ | ✗ |  .718 $\pm$ .092 | .781 $\pm$ .101 |  .502 $\pm$ .081
✓ | ✓ | ✓ |  .718 $\pm$ .095 |  .770 $\pm$ .102 |  .482 $\pm$ .078

To investigate how different SSL configurations perform in three downstream
datasets, we picked one large (Capture-24), medium (Rowlands) and small
(Opportunity) dataset for evaluation. We trained different tasks both
individually and jointly using 1000 subjects from the UKB, then we fine-tuned
the models on the subsequent HAR benchmarks (Table 2).

The differences between different SSL combinations on large datasets
(Capture-24 and Rowlands) was smaller than that of the smaller dataset
(Opportunity). There was no clear best performing configuration, and thus, for
ease of comparison, we chose to use all tasks in pre-training for the
remaining experiments. In addition, training more tasks together might yield
the most general representation for different downstream datasets.

###  4.3 Downstream performance - human activity recognition

Table 3: Subject-wise F1 and Kappa ($\kappa$) for downstream human activity
recognition tasks (mean $\pm$ SD) using 100,000 participants for pre-training.
The relative improvement compares the performance between the model that is
trained from scratch and fine-tuning using all the layers.
Data |  | Random forest | ResNet | Improvement
---|---|---|---|---
| Trained from scratch | Fine-tune self-supervised
| After ConV layers | All layers
Capture-24 | F1 |  .694 $\pm$ .099 |  .708 $\pm$ .094 |  .723 $\pm$ .097 | .726 $\pm$ .093 | 2.5 %
$\kappa$ |  .683 $\pm$ .101 |  .703 $\pm$ .092 |  .718 $\pm$ .090 | .737 $\pm$ .087 | 4.8%
Rowlands | F1 |  .700 $\pm$ .090 |  .696 $\pm$ .106 |  .724 $\pm$ .081 | .796 $\pm$ .093 | 14.4%
$\kappa$ |  .830 $\pm$ .086 |  .810 $\pm$ .098 |  .850 $\pm$ .062 | .874 $\pm$ .073 | 7.9%
WISDM | F1 |  .711 $\pm$ .149 |  .684 $\pm$ .123 |  .759 $\pm$ .121 | .810 $\pm$ .127 | 18.4%
$\kappa$ |  .715 $\pm$ .153 |  .685 $\pm$ .124 |  .758 $\pm$ .121 | .809 $\pm$ .126 | 18.1%
REALWORLD | F1 |  .731 $\pm$ .119 |  .705 $\pm$ .062 |  .764 $\pm$ .052 | .792 $\pm$ .075 | 12.3%
$\kappa$ |  .680 $\pm$ .142 |  .638 $\pm$ .079 |  .703 $\pm$ .063 | .739 $\pm$ .086 | 15.8%
Opportunity | F1 |  .416 $\pm$ .185 |  .383 $\pm$ .124 |  .570 $\pm$ .078 | .595 $\pm$ .085 | 55.4%
$\kappa$ |  .318 $\pm$ .206 |  .238 $\pm$ .154 |  .435 $\pm$ .092 | .471 $\pm$ .104 | 97.9%
PAMAP2 | F1 |  .753 $\pm$ .093 |  .605 $\pm$ .086 |  .725 $\pm$ .054 | .789 $\pm$ .054 | 30.4%
$\kappa$ |  .744 $\pm$ .101 |  .596 $\pm$ .086 |  .717 $\pm$ .057 | .769 $\pm$ .059 | 29.0 %
ADL | F1 |  .764 $\pm$ .180 |  .414 $\pm$ .179 |  .645 $\pm$ .107 | .829 $\pm$ .101 | 100.0 %
$\kappa$ |  .720 $\pm$ .199 |  .368 $\pm$ .198 |  .654 $\pm$ .123 | .849 $\pm$ .113 | 130.7%

Table 3 summarises the F1 and Kappa scores for seven HAR datasets. The random
forest models outperformed the deep learning models trained from scratch for
all except the Capture-24 dataset, which is the largest dataset in our
evaluations (Table 1). The performance gap between random forest and training
from scratch was the largest in smaller datasets. Meanwhile, pre-trained
models outperformed the models trained from scratch and random forest in all
seven datasets. Fine-tuning all layers was better than fine-tuning just the
fully connected (FC) layers after the ConV layers.

The most significant improvement using pre-training was seen on the small
datasets. Conversely, the SSL benefit was more modest for larger datasets. In
Capture-24, the F1 improvement was 2.5% when comparing the model with and
without SSL pre-training. Nonetheless, with SSL pre-training, the median
relative F1 improvement was 18.4% when compared against the same network
trained from scratch and 8.5% when compared against the random forest model.

###  4.4 Transfer learning using labelled pre-training

Even though supervised pre-training can boost the learning outcome
substantially than training from scratch (Table 3 vs Table 4), it was
surprising to see self-supervised pre-training could outperform supervised
pre-training when using Rowlands and Capture-24 as the source data. We suspect
the limited number of labels that the source datasets had did not contain
enough information as what would have been learnt using the SSL pretext tasks.

Table 4: Transfer learning (subject-wise F1) performance comparison between
supervised pre-training with self-supervised pre-training.
Target data | Source data
---|---
Rowlands | Capture-24 | UK-Biobank
| Supervised | Self-supervised | Supervised | Self-supervised | Self-supervised
Capture-24 |  .707 $\pm$ .094 |  .709 $\pm$ .094 | - |  .707 $\pm$ .094 | .726 $\pm$ .093
Rowlands | - |  .734 $\pm$ .082 |  .728 $\pm$ .094 |  .730 $\pm$.084 | .796 $\pm$ .093
WISDM |  .680 $\pm$ .109 |  .702 $\pm$ .123 |  .715 $\pm$ .119 |  .723 $\pm$ .121 | .810 $\pm$ .127
REALWORLD |  .712 $\pm$ .086 |  .737 $\pm$ .105 |  .759 $\pm$ .070 |  .771 $\pm$ .061 | .792 $\pm$ .075
Opportunity |  .536 $\pm$ .019 |  .539 $\pm$ .018 |  .547 $\pm$ .043 |  .547 $\pm$ .042 | .595 $\pm$ .085
PAMAP2 |  .677 $\pm$ .082 |  .689 $\pm$ .078 |  .678 $\pm$ .118 |  .725 $\pm$ .725 | .789 $\pm$ .054
ADL |  .634 $\pm$ .182 |  .701 $\pm$ .111 |  .768 $\pm$ .169 |  .754 $\pm$ .159 | .829 $\pm$ .101

###  4.5 Ablation studies

(a) How different models perform in the downstream tasks when we change the
number of labelled subjects: Capture-24 (left) and Rowlands (right).

(b) Left: More #unlabelled subjects in the SSL leads to better downstream
performance. Right: Unlabelled data per subject does not improve downstream
performance when using 10,000 subjects for pre-training.

Figure 3: The impact of varying amount of labelled or unlabelled data in the
self-supervised training and the fine-tuning stage on subsequent human
activity recognition performance. Mean F1$\pm$ SD are plotted.

##### Varying labelled data in the downstream

We observed that pre-trained models did well regardless of the number of
labelled subjects in two downstream datasets (Figure 3). However, fully-
supervised and random forest models were more susceptible to the number of
labelled subjects. The performance gain for having more labelled subjects was
roughly linear w.r.t. the number of people included with a greater increase
when we had fewer labelled subjects.

##### Varying unlabelled data in the pre-training

We also found that the downstream HAR performance appeared to increase
linearly w.r.t. the number of unlabelled subjects on a log scale ( Figure 3
left). This conforms with the previous finding in a semi-supervised setting
(Oliver et al.,, 2018). The SSL performance boost with more SSL unlabelled
subjects was most significant in the smallest dataset, Opportunity.
Furthermore, if the number of participants is fixed at 10,000 in pre-training,
the data ratio included per subject did not significantly influence the
downstream performance. To our surprise, the downstream performance did not
degrade too much even when we reduced the subject data ratio from 1.0 to .25
(Figure 3 right). We suspect that because we had a sufficiently large sample
size (10,000), having more data per subject would not help to learn a better
representation anymore. The trade-off between inter-subject and intra-subject
variability in self-supervision warrants future investigation. Knowing how to
prioritize the number of subjects and how much data per subject would benefit
the data curation process, especially in fields like medical sciences, where
it might be easier to get some data from lots of people than get lots of data
from some people.

###  4.6 Understanding the representation

(a) Rowlands

(b) CAPTURE-24

Figure 4:  Cluster analysis on raw inputs, untrained features and SSL-
pretrained features. We use color gradients to denote activity intensities.
Results suggest that SSL-derived features are better at clustering similar
activities (e.g. walking, stair climbing vs. sitting, writing, typing) as well
as their intensities (e.g. lying down, sitting, standing vs. jogging, sports).

(a) Shaking hands

(b) Playing tennis

Figure 5: Arrow of time + permutation + time-warped signals of (a) scripted,
repetitive low intensity activities, e.g. shaking hands and (b) unscripted,
repetitive high intensity activity, e.g. playing tennis. The first row shows
the concurrent video frames when a participant performs an activity. The
second row is the accelerometry trace. The third row is the continuous wavelet
transform scalograms. The last row is the task-specific LRP attribution.
Dashed patches over the acceleration trace correspond to the concurrent video
video frames. Probability (Pr.) of individual transform applied to: (a) raw:
AoT (0.), permutation (.68), TW (0.); augmented: AoT (1.), permutation (1.),
TW (1.); (b) AoT (.03), permutation (.18) , TW (0.) ; augmented: AoT (1.),
permutation (.98), TW (.36);

##### Cluster analysis

We used UMAP (McInnes et al.,, 2018) with default parameters for low-
dimensional projections for visualization. This was applied to the raw inputs,
untrained features, and SSL-derived features _without fine-tuning_. Results
for two of the downstream datasets are shown in Figure 4, and remaining
results can be found in Appx. Figure 7. Across all datasets, we observed that
the SSL-derived features were better at clustering similar activities (e.g.
walking, stair climbing vs sitting, writing, typing) as well as their
intensities (e.g. lying down, sitting, standing vs jogging, sports),
exhibiting better intra-class compactness and inter-class separability.

##### Feature interpretation

Next, we visualised two exemplary pretext SSL task predictions in the presence
of repetitive low- and high-intensity activities: shaking hands (Figure 5(a))
and playing tennis (Figure 5(b)). During tennis playing, a repetitive high,
intensity activity, relevance scores tended to highlight the moments around
the natural movements of swinging and hitting the tennis ball (Figure 5(b),
Appx. Figure 9). When performing a repetitive low-intensity activity
experiment, for example, shaking hands (Figure 5(a)), LRP appeared to also
identify the intensity and natural signal periodicity as indicative of the
original activity. In contrast, for augmented signals, our model attributed
more during periods of visually unrealistic motion dynamics, such as unnatural
fragmentation in activity frequency or synchronisation mismatches between
sensor axes. Interestingly, stationary movement periods were not relevant for
detecting the pretext tasks, further emphasising the importance of our
movement-based weighted sampling approach during training. We found that most
tasks tended to identify similar regions as relevant. However, this might be
expected due to the large shared feature extractor base across all tasks, with
smaller MTL heads to identify each pretext task.

Finally, we empirically compared the faithfulness of the XAI algorithms
investigated and the combination of various LRP parameters, using sample-
masking experiments for a random subset of 1000 (out-of-sample) subjects in
the UKB (Appx. Section A.4.2). Most XAI models consistently demonstrated the
ability to identify relevant patterns for discriminating transformed samples
from the original raw data when compared against a random model.

##  5 Conclusion and future work

Our main findings are: (1) Self-supervised pre-training consistently improved
downstream HAR, especially in the small data regime, reducing the need for
labelled data. (2) It is possible to learn representations that generalise
well across external datasets, tasks, devices and populations – this contrasts
with previous studies where SSL training and evaluation were performed on the
same data sources. (3) Multi-task SSL appears to be learning human motion
dynamics, intensity levels, and the synchronisation between different axes
(Figure 4 and Figure 5). Even before fine-tuning, the pre-trained models
already have the capacity to distinguish different human activities.

In the future, we could include other compatible data modalities such as
electrocardiogram data. Having a multi-modal representation would allow us to
build foundation models that could be used in critical fields like health
research (Bommasani et al.,, 2021; Spathis et al.,, 2022). Another potential
work is the analysis of inter-subject and intra-subject variability in sensor
data which could better inform the data curation procedures.

Due to a current lack of raw accelerometer datasets in different regions of
the world, a limitation of our work is that the pre-training data (UKB
dataset) consists mostly of Caucasians from the UK. A natural extension of
this work could include datasets from different demographic groups, as they
become available in the future, to improve model fairness.

## Acknowledgments and Disclosure of Funding

We would like to thank all the helpful discussions and feedback we recevied
from Aidan Acquah, Gert Mertes, Henrique Aguiar, Andres Tamm, and Korsuk
Sirinukunwattana.

This research has been conducted using the UK Biobank Resource under
Application Number 59070. This work is supported by: Novo Nordisk (HY, AD);
the Wellcome Trust [223100/Z/21/Z] (AD); GlaxoSmithKline (AC, DC); the British
Heart Foundation Centre of Research Excellence [RE/18/3/34214] (AD); the
National Institute for Health Research (NIHR) Oxford Biomedical Research
Centre (AD, DC); and Health Data Research UK, an initiative funded by UK
Research and Innovation, Department of Health and Social Care (England) and
the devolved administrations, and leading medical research charities. It is
also supported by the UK’s Engineering and Physical Sciences Research Council
(EPSRC) with grants EP/S001530/1 (the MOA project) and EP/R018677/1 (the OPERA
project); and the European Research Council (ERC) via the REDIAL project
(Grant Agreement ID: 805194), and industrial funding from Samsung AI.

We would also like to thank Alex Rowlands and Mike Catt, who kindly shared
their activity dataset with us. Their project was funded by a grant from
Unilever Discover to the School of Sports and Health Sciences, University of
Exeter.

No funding bodies had any role in the analysis, decision to publish, or
preparation of the manuscript.

## References

  * Addison et al., (2009) Addison, P. S., Walker, J., and Guido, R. C. (2009).  Time–frequency analysis of biosignals.  IEEE Engineering in Medicine and Biology Magazine, 28(5):14–29.
  * Belcher et al., (2021) Belcher, B. R., Wolff-Hughes, D. L., Dooley, E. E., Staudenmayer, J., Berrigan, D., Eberhardt, M. S., and Troiano, R. P. (2021).  Us population-referenced percentiles for wrist-worn accelerometer-derived activity.  Medicine and Science in Sports and Exercise.
  * Bommasani et al., (2021) Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021).  On the opportunities and risks of foundation models.  arXiv preprint arXiv:2108.07258.
  * Bruno et al., (2013) Bruno, B., Mastrogiovanni, F., Sgorbissa, A., Vernazza, T., and Zaccaria, R. (2013).  Analysis of human behavior recognition algorithms based on acceleration data.  In 2013 IEEE International Conference on Robotics and Automation, pages 1602–1607. IEEE.
  * Bulling et al., (2014) Bulling, A., Blanke, U., and Schiele, B. (2014).  A tutorial on human activity recognition using body-worn inertial sensors.  ACM Computing Surveys (CSUR), 46(3):1–33.
  * Chen et al., (2020) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020).  A simple framework for contrastive learning of visual representations.  In ICML, pages 1597–1607. PMLR.
  * Creagh et al., (2021) Creagh, A. P., Lipsmeier, F., Lindemann, M., and Vos, M. D. (2021).  Interpretable deep learning for the remote characterisation of ambulation in multiple sclerosis using smartphones.  Scientific Reports, 11(1):14301.
  * Creagh et al., (2020) Creagh, A. P., Simillion, C., Bourke, A., Scotland, A., Lipsmeier, F., Bernasconi, C., Beek, J. v., Baker, M., Gossens, C., Lindemann, M., and Vos, M. D. (2020).  Smartphone- and smartwatch-based remote characterisation of ambulation in multiple sclerosis during the two-minute walk test.  IEEE Journal of Biomedical and Health Informatics.
  * Devlin et al., (2018) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018).  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * Doersch et al., (2015) Doersch, C., Gupta, A., and Efros, A. A. (2015).  Unsupervised visual representation learning by context prediction.  In ICCV, pages 1422–1430.
  * Doherty et al., (2017) Doherty, A., Jackson, D., Hammerla, N., Plötz, T., Olivier, P., Granat, M. H., White, T., Van Hees, V. T., Trenell, M. I., Owen, C. G., et al. (2017).  Large scale population assessment of physical activity using wrist worn accelerometers: The uk biobank study.  PloS one, 12(2):e0169649.
  * Ellis et al., (2016) Ellis, K., Kerr, J., Godbole, S., Staudenmayer, J., and Lanckriet, G. (2016).  Hip and wrist accelerometer algorithms for free-living behavior classification.  Medicine and Science in Sports and Exercise, 48(5):933.
  * Esliger et al., (2011) Esliger, D., Rowlands, A. V., Hurst, T. L., Catt, M., Murray, P., and Eston, R. G. (2011).  Validation of the genea accelerometer.  Med Sci Sports Exerc.
  * Goyal et al., (2017) Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. (2017).  Accurate, large minibatch sgd: Training imagenet in 1 hour.  arXiv preprint arXiv:1706.02677.
  * Hammerla et al., (2016) Hammerla, N. Y., Halloran, S., and Plötz, T. (2016).  Deep, convolutional, and recurrent models for human activity recognition using wearables.  arXiv preprint arXiv:1604.08880.
  * Hammerla et al., (2013) Hammerla, N. Y., Kirkham, R., Andras, P., and Ploetz, T. (2013).  On preserving statistical characteristics of accelerometry data using their empirical cumulative distribution.  In Proceedings of the 2013 International Symposium on Wearable Computers, pages 65–68.
  * Haresamudram et al., (2019) Haresamudram, H., Anderson, D. V., and Plötz, T. (2019).  On the role of features in human activity recognition.  In Proceedings of the 23rd International Symposium on Wearable Computers, pages 78–88.
  * (18) Haresamudram, H., Beedu, A., Agrawal, V., Grady, P. L., Essa, I., Hoffman, J., and Plötz, T. (2020a).  Masked reconstruction based self-supervision for human activity recognition.  In Proceedings of the 2020 International Symposium on Wearable Computers, pages 45–49.
  * (19) Haresamudram, H., Essa, I., and Ploetz, T. (2020b).  Contrastive predictive coding for human activity recognition.  arXiv preprint arXiv:2012.05333.
  * He et al., (2020) He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020).  Momentum contrast for unsupervised visual representation learning.  In CVPR, pages 9729–9738.
  * He et al., (2016) He, K., Zhang, X., Ren, S., and Sun, J. (2016).  Identity mappings in deep residual networks.  In ECCV, pages 630–645. Springer.
  * Kingma and Ba, (2014) Kingma, D. P. and Ba, J. (2014).  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * Kohlbrenner et al., (2020) Kohlbrenner, M., Bauer, A., Nakajima, S., Binder, A., Samek, W., and Lapuschkin, S. (2020).  Towards best practice in explaining neural network decisions with lrp.  In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–7.
  * Kwon et al., (2020) Kwon, H., Tong, C., Haresamudram, H., Gao, Y., Abowd, G. D., Lane, N. D., and Ploetz, T. (2020).  Imutube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(3):1–29.
  * Mannini et al., (2013) Mannini, A., Intille, S. S., Rosenberger, M., Sabatini, A. M., and Haskell, W. (2013).  Activity recognition using a single accelerometer placed at the wrist or ankle.  Medicine and Science in Sports and Exercise, 45(11):2193.
  * McInnes et al., (2018) McInnes, L., Healy, J., and Melville, J. (2018).  Umap: Uniform manifold approximation and projection for dimension reduction.  arXiv preprint arXiv:1802.03426.
  * Mikolov et al., (2013) Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013).  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781.
  * Montavon et al., (2019) Montavon, G., Binder, A., Lapuschkin, S., Samek, W., and Müller, K.-R. (2019).  Layer-wise relevance propagation: an overview, pages 193–209.  Springer.
  * Noroozi and Favaro, (2016) Noroozi, M. and Favaro, P. (2016).  Unsupervised learning of visual representations by solving jigsaw puzzles.  In ECCV, pages 69–84. Springer.
  * Oliver et al., (2018) Oliver, A., Odena, A., Raffel, C. A., Cubuk, E. D., and Goodfellow, I. (2018).  Realistic evaluation of deep semi-supervised learning algorithms.  NeurIPS, 31.
  * Quiñonero-Candela et al., (2008) Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A., and Lawrence, N. D. (2008).  Dataset shift in machine learning.  Mit Press.
  * Radford et al., (2018) Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018).  Improving language understanding by generative pre-training.  OpenAI blog.
  * Radford et al., (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019).  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9.
  * Reiss and Stricker, (2012) Reiss, A. and Stricker, D. (2012).  Introducing a new benchmarked dataset for activity monitoring.  In 2012 16th International Symposium on Wearable Computers, pages 108–109. IEEE.
  * Roggen et al., (2010) Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., Förster, K., Tröster, G., Lukowicz, P., Bannach, D., Pirkl, G., Ferscha, A., et al. (2010).  Collecting complex activity datasets in highly rich networked sensor environments.  In 2010 Seventh International Conference on Networked Sensing Systems (INSS), pages 233–240. IEEE.
  * Ronao and Cho, (2016) Ronao, C. A. and Cho, S.-B. (2016).  Human activity recognition with smartphone sensors using deep learning neural networks.  Expert Systems with Applications, 59:235–244.
  * Saeed et al., (2019) Saeed, A., Ozcelebi, T., and Lukkien, J. (2019).  Multi-task self-supervised learning for human activity detection.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(2):1–30.
  * Samek et al., (2021) Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., and Müller, K.-R. (2021).  Explaining deep neural networks and beyond: A review of methods and applications.  Proceedings of the IEEE, 109(3):247–278.
  * Simonyan et al., (2013) Simonyan, K., Vedaldi, A., and Zisserman, A. (2013).  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034.
  * Spathis et al., (2022) Spathis, D., Perez-Pozuelo, I., Marques-Fernandez, L., and Mascolo, C. (2022).  Breaking away from labels: The promise of self-supervised machine learning in intelligent health.  Patterns, 3(2):100410.
  * Springenberg et al., (2014) Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. (2014).  Striving for simplicity: The all convolutional net.  arXiv preprint arXiv:1412.6806.
  * Sundararajan et al., (2017) Sundararajan, M., Taly, A., and Yan, Q. (2017).  Axiomatic attribution for deep networks.  In ICML, pages 3319–3328. PMLR.
  * Sztyler and Stuckenschmidt, (2016) Sztyler, T. and Stuckenschmidt, H. (2016).  On-body localization of wearable devices: An investigation of position-aware activity recognition.  In 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom), pages 1–9. IEEE.
  * Taghanaki and Etemad, (2020) Taghanaki, S. R. and Etemad, A. (2020).  Self-supervised wearable-based activity recognition by learning to forecast motion.  arXiv preprint arXiv:2010.13713.
  * Tang et al., (2021) Tang, C. I., Perez-Pozuelo, I., Spathis, D., Brage, S., Wareham, N., and Mascolo, C. (2021).  Selfhar: Improving human activity recognition through self-training with unlabeled data.  arXiv preprint arXiv:2102.06073.
  * Tang et al., (2020) Tang, C. I., Perez-Pozuelo, I., Spathis, D., and Mascolo, C. (2020).  Exploring contrastive learning in human activity recognition for healthcare.  arXiv preprint arXiv:2011.11542.
  * Twomey et al., (2018) Twomey, N., Diethe, T., Fafoutis, X., Elsts, A., McConville, R., Flach, P., and Craddock, I. (2018).  A comprehensive study of activity recognition using accelerometers.  In Informatics, volume 5, page 27. Multidisciplinary Digital Publishing Institute.
  * Um et al., (2017) Um, T. T., Pfister, F. M. J., Pichler, D., Endo, S., Lang, M., Hirche, S., Fietzek, U., and Kulić, D. (2017).  Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks.  In Proceedings of the 19th ACM International Conference on Multimodal Interaction, ICMI 2017, pages 216–220, New York, NY, USA. ACM.
  * Wei et al., (2018) Wei, D., Lim, J. J., Zisserman, A., and Freeman, W. T. (2018).  Learning and using the arrow of time.  In CVPR, pages 8052–8060.
  * Weiss et al., (2019) Weiss, G. M., Yoneda, K., and Hayajneh, T. (2019).  Smartphone and smartwatch-based biometrics using activities of daily living.  IEEE Access, 7:133190–133202.
  * Willetts et al., (2018) Willetts, M., Hollowell, S., Aslett, L., Holmes, C., and Doherty, A. (2018).  Statistical machine learning of sleep and physical activity phenotypes from sensor data in 96,220 uk biobank participants.  Scientific Reports, 8(1):1–10.
  * Yang et al., (2015) Yang, J., Nguyen, M. N., San, P. P., Li, X., and Krishnaswamy, S. (2015).  Deep convolutional neural networks on multichannel time series for human activity recognition.  In IJCAI, volume 15, pages 3995–4001. Buenos Aires, Argentina.
  * Zhang et al., (2016) Zhang, R., Isola, P., and Efros, A. A. (2016).  Colorful image colorization.  In ECCV, pages 649–666. Springer.
  * Zhang et al., (2012) Zhang, S., Rowlands, A. V., Murray, P., Hurst, T. L., et al. (2012).  Physical activity classification using the genea wrist-worn accelerometer.  Medicine and Science in Sports and Exercise.

##  Appendix A Appendix

###  A.1 Datasets

Table 5: The license, consent and device used for each dataset
Dataset | Personal Info |  Licence |  Consent |  Device
---|---|---|---|---
UK-Biobank | ✗ |  Non-exclusive licence (but not ownership rights) for permitted purpose only |  Informed consent for research |  Axitivity (AX3)
Capture-24 | ✗ |  CC Attribution |  Informed consent for research |  Axitivity (AX3)
Rowlands | ✗ |  Non-specified |  Informed consent for research |  Genea
WISDM | ✗ |  Non-specified |  Non-specified |  LG G Watch
REALWORLD | ✗ |  Non-specified |  Non-specified |  LG G Watch R
Opportunity | ✗ |  Non-specified |  Non-specified |  Custom inertial measurement units (IMUs)
PAMAP2 | ✗ |  Non-specified |  Informed consent for research |  Colibri inertial measurement units (IMUs)
ADL | ✗ |  Free for research |  Non-specified |  Non-specified

###  A.2 Methods

###  A.3 Feature engineering

The list of hand-crafted features that were extracted for random forest:

  * •

Mean, standard deviation, and range for each axis.

  * •

Correlation between each axis pair.

  * •

Euclidean norm, its mean, standard deviation, range, median absolute
deviation, kurtosis, and skew.

  * •

The top two dominant frequencies in the power spectrum.

####  A.3.1 Explainable AI Framework for Time-Series Prediction

Holistic visual interpretation of the AoT predictions were determined by (1)
visualising the raw data, (2) its time-frequency representation using the
(discretised) continuous wavelet transform (CWT) and (3) assessing the time-
localised LRP attribution.

##### The Continuous Wavelet Transform:

The CWT is a method used to measure the similarity between a signal and an
analysing function (in this case the Morlet wavelet) which can provide a
precise time-frequency representation of a signal (Creagh et al.,, 2020;
Addison et al.,, 2009).

##### Layer Wise Relevance Propagation:

The LRP algorithm back-propagates through a network to decompose the final
output decision, $f(\bm{x})$ (Samek et al.,, 2021; Montavon et al.,, 2019).
Briefly, a trained model’s activations, weights and biases are first obtained
in a forward pass through the network. Secondly, during a backwards pass
through the model, LRP attributes relevance to the individual input nodes,
layer by layer. For example $R_{k}$ denotes the relevance for neuron $k$ in
layer (l+1), and $R_{j\leftarrow k}$ defines the share of $R_{k}$ that is
redistributed to neuron $j$ in layer (l). The fundamental concept underpinning
LRP is that the conservation of relevance per layer, which can be denoted as:

$\sum_{j}R_{j\leftarrow k}^{(l)}=R_{k}^{(l+1)}$ (1)

The LRP algorithm initiates at the model output and iterates over all layers
in the model in a backwards pass until the relevance scores $R_{i}$ for all
inputs of $x_{i}$ are computed. Relevance values $R_{i}>0$ signify components
$x_{i}$ which represent the presence of the predicted class, while conversely
$R_{i}<0$ contradict the prediction of that class. $R_{i}\approx 0$ indicate
inputs $x_{i}$ which have little or no influence to the model’s decision.

It has been demonstrated that the combination of different rules throughout a
network yields the most faithful and understandable LRP explanations
(Kohlbrenner et al.,, 2020). As such, we applied a composite rule (LRP-CMP)
with LRP-$\gamma$ applied to the shallower convolutional layers,
LRP-$\epsilon$ rules ($\epsilon$=$\\{1e^{-9},1e^{-3},10\\}$) in the middle of
the network, and LRP-$0$ applied to the final linear classification layer.

##### Visually interpreting SSL attribution:

To visually interpret SSL attribution, we compared the raw time-series
accelerometery with the analogous SSL task transformed data. In in the panel
plot depicted in figures 5 and 9, the top rows represent the 3-axis
accelerometer trace for each channel:
$(\textit{{a}}_{x},\textit{{a}}_{y},\textit{{a}}_{z})$; the second rows depict
the top view of the continuous wavelet transform (CWT) scalogram of
$\|\textit{{a}}\|$, which is the absolute value of the CWT as a function of
time and frequency. The bottom rows denote the relevance values ($R_{i}$)
attributed using LRP. Red and hot colours identify input segments where
$R_{i}>0$ (contribution to a class prediction), whereas blue and cold hues
identify $R_{i}<0$ (contradicting a class prediction), while black represents
($R_{i}\approx 0$) inputs which have little or no influence to the model’s
decision. Square patches over the raw accelerometer trace correspond to the
video frames depicted above each panel plot.

##### Evaluating XAI algorithm faithfulness:

In order to test the faithfulness of an explanation provided by an XAI
framework, a sample-masking experiment was performed, comparing some popular
XAI models and LRP parameters options. Briefly, to conduct a permutation
analysis, the most relevant samples identified from each XAI algorithm were
cumulatively masked (imputation with random Gaussian noise), from most
relevant to least relevant (Samek et al.,, 2021). As such, the faster the
accuracy of the model decreases with the number of masked samples, the more
faithful the explanation method is with respect to the decision of the neural
network.

Permutation tests were performed in random batches containing correctly
identified duplicate original and augmented samples (AoT + permutation + TW)
from 1000 subjects in the out-of-sample test set sampled in the UKB. The mean
degradation in SSL prediction through our permutation experiment is shown in
figure 8. We compared various LRP parameter options, LRP-$0$, LRP-$\epsilon$,
LRP-CMP, as well as some popular off-the-shelf XAI attribution frameworks,
such as saliency mapping (Simonyan et al.,, 2013), Guided Backpropagation
(GBP) (Springenberg et al.,, 2014) and Integrated Gradients (IG) (Sundararajan
et al.,, 2017).

###  A.4 Results

Figure 6: Data collected from free-living environment tends to have more
stationary periods than the data collected in lab environment. Table 6:
Downstream human activity recognition performance (subject-wise F1 ($\pm$SD))
when using different self-supervised learning tasks after fine-tuning: Arrow
of the Time (AoT), Permutation (P), Time warping (TW). The tasks were trained
using the same 1,000 UK-Biobank participants.
Task | Capture-24 | Rowlands | WISDM | REALWORLD | Opportunity | PAMAP2 | ADL
---|---|---|---|---|---|---|---
#Subjects | 152 | 55 | 46 | 14 | 4 | 8 | 7
#Samples | 573K | 36K | 28k | 12k | 3882 | 2869 | 635
AoT |  .671 $\pm$ .094 |  .565 $\pm$ .120 |  .767 $\pm$ .124 |  .750 $\pm$ .084 |  .582 $\pm$ .054 |  .715 $\pm$ .036 |  .754 $\pm$ .157
Permutation | .721 $\pm$ .093 | .783 $\pm$ .099 | .778 $\pm$ .109 |  .766 $\pm$ .063 | .588 $\pm$ .076 | .750 $\pm$ .057 | .810 $\pm$ .097
TW |  .715 $\pm$ .093 |  .776 $\pm$ .110 |  .767 $\pm$ .120 | .772 $\pm$ .073 |  .584 $\pm$ .064 |  .737 $\pm$ .079 |  .765 $\pm$ .117
Table 7: Test accuracies on different tasks 1000 subjects.
AoT | Permutation | Time warped
---|---|---
0.93 | .90 | .97

####  A.4.1 Additional UMAP visualizations

(a) ADL

(b) Opportunity

(c) PAMAP2

(d) REALWORLD

(e) WISDM

Figure 7:  Cluster analysis on raw inputs, untrained features and SSL-
pretrained features. We use color gradients to denote activity intensities.
Results suggest that SSL-derived features are better at clustering similar
activities (e.g. walking, stair climbing vs. sitting, writing, typing) as well
as their intensities (e.g. lying down, sitting, standing vs. jogging, sports).

####  A.4.2 Explainable Framework Performance

We observed that, for most explanation methods, removing relevant features
quickly destroyed class evidence, degrading the out-of-sample test accuracy of
the SSL model, as shown in figure 8. This inferred that the XAI methods had
fatefully identified relevant patterns within the accelerometer signal for
determining the SSL pretext task compared to randomly mixing and masking
samples in the signal. Adding random noise to the accelerometer was
demonstrated to degrade model performance (which is to be expected) but at a
slower rate than masking the most relevant samples first.

Furthermore, replacing the accelerometer signal with noise, cumulatively from
the first time-step until the last (essentially disrupting the AoT), was found
to degrade AoT-specific model accuracy at a slower rate than masking samples
based on XAI relevance sorting or randomly. Note: AoT disruption was
calculated from the start of the signal to the end (forward) and from the end
of the signal to the start (reverse) and presented as the mean AoT disruption.
As such, the the model does not appear to learn a single sequential pattern
within a time-series (either forward or reverse), but instead suggests that
there may be certain signal morphologies that may indicate the SSL
augmentation, such as the characteristics of dynamic human motion.

Figure 8: Comparison of popular explainability algorithm performance on AoT +
permutation + TW for 1000 random subjects in the UK Biobank out-of-sample test
data, through consecutively masking samples from most relevant to least
relevant. We observed that, for most explanation methods, removing identified
relevant features quickly destroyed class evidence, thus degrading the out-of-
sample test accuracy of the model compared to randomly masking samples.

####  A.4.3 Further Contextual LRP Examples

Figures 9(a) and 9(b) depict further examples of contextual LRP collected
during the during unscripted, repetitive, high intensity activity of playing
tennis. For example, figure 9(a) depicts when the participant dropped the
ball; figure 9(b) illustrates an overhead tennis serve. It was observed that
LRP tended to also attribute relevance to the moments of natural human motion
captured during these shots.

(a) playing tennis (unstructured dropping the ball)

(b) playing tennis (serve)

Figure 9: Arrow of time + permutation + time-warped signals during unscripted,
repetitive high intensity intensity activity of playing tennis. The square
patches over the acceleration trace correspond to the video frame depicted
during when the participant (a) dropped the ball and (b) performed a tennis
serve. Probability (Pr.) of individual transform applied to: (a) raw: AoT
(.01), TW (0.), perm (1.); augmented: AoT (1.), TW (1.), perm (1.); (b) raw:
AoT (.01), TW (0.), perm (.04); augmented: AoT (1.), TW (1.), perm (.98);