$\Lambda_{j}\nsubseteq\text{reg}(S_{j_{1}},S_{j_{2}})$. Moreover, by
assumption of this lemma, $P^{\prime\prime}\notin\\{S_{1},S_{2},S_{3}\\}$. So
if $P^{\prime}\in\text{reg}^{\circ}(S_{j_{1}},S_{j_{2}})$, then
$P^{\prime\prime}\in\text{reg}^{\circ}(S_{j_{1}},S_{j_{3}})\cup\text{reg}^{\circ}(S_{j_{2}},S_{j_{3}})$.

For example, if $\Lambda_{j}=\\{S_{1},P\\}$, then
$P\in\text{reg}^{\circ}(S_{2},S_{3})$; If
$\Lambda_{j}=\\{P^{\prime},P^{\prime\prime}\\}\subseteq\Pi\backslash\\{S_{1},S_{2},S_{3}\\}$
and $P^{\prime}\in\text{reg}^{\circ}(S_{1},S_{2})$, then we have
$P^{\prime\prime}\in\text{reg}^{\circ}(S_{1},S_{3})$ or
$P^{\prime\prime}\in\text{reg}^{\circ}(S_{2},S_{3})$.

###### Lemma C.3

Suppose $P_{1}\in\text{reg}^{\circ}(S_{1},S_{2})$ and
$P_{2},P_{3}\in\text{reg}^{\circ}(S_{2},S_{3})$ such that
$\Lambda_{1}=\\{P_{1},P_{2}\\}$ and $\Lambda_{2}=\\{P_{1},P_{3}\\}$. Then
$\text{RG}(D^{**})$ is feasible.

###### proof C.4

If $\Lambda_{3}\cap\text{reg}^{\circ}(S_{1},S_{2})\neq\emptyset$ or
$\Lambda_{3}\cap\text{reg}^{\circ}(S_{2},S_{3})\neq\emptyset$, then by (3) of
Theorem VI.16, $\text{RG}(D^{**})$ is feasible. So we assume
$\Lambda_{3}\cap(\text{reg}^{\circ}(S_{1},S_{2})\cup\text{reg}^{\circ}(S_{2},S_{3}))=\emptyset$.
Then we have
$\Lambda_{3}\subseteq\Pi\setminus(\text{reg}^{\circ}(S_{1},S_{2})\cup\text{reg}^{\circ}(S_{2},S_{3}))=\\{S_{2}\\}\cup\text{reg}(S_{1},S_{3})$.
Moreover, since by 2) of Lemma IV.9,
$\Lambda_{3}\nsubseteq\text{reg}(S_{1},S_{3})$, then $S_{2}\in\Lambda_{3}$.
Thus, we have

$S_{2}\in\Lambda_{3}\subseteq\\{S_{2}\\}\cup\text{reg}(S_{1},S_{3}).$

Again, by 2) of Lemma IV.9, $\Lambda_{3}\nsubseteq\text{reg}(S_{1},S_{2})$ and
$\Lambda_{3}\nsubseteq\text{reg}(S_{2},S_{3})$. Then either
$\Lambda_{3}=\\{S_{1},S_{2},S_{3}\\}$ or $\\{S_{2},P\\}\subseteq\Lambda_{3}$
for some $P\in\text{reg}^{\circ}(S_{1},S_{3})$.

Let $\mathcal{I}=\\{[S_{1}],[S_{2}],[S_{3}],[P_{1}]\\}$, where
$[S_{1}]=\\{S_{1}\\}$,
$[S_{2}]=\\{S_{2}\\}\cup\text{reg}^{\circ}(S_{1},S_{3})$,
$[S_{3}]=\\{S_{3}\\}$ and
$[P_{1}]=\text{reg}^{\circ}(S_{1},S_{2})\cup\text{reg}^{\circ}(S_{2},S_{3})$.
By Definition VI.9, it is easy to check that $\mathcal{I}$ is a compatible
partition of $\Pi$. So by Lemma VI.13, $\text{RG}(D^{**})$ is feasible. An
illustration is given in Fig. 12.

Fig 12: An example of code construction: (a) is a region graph satisfies
conditions of Lemma C.3 with $\Lambda_{3}=\\{S_{2},P\\}$; (b) illustrates a
code, where the sets of coding vectors
$\mathcal{B}_{1}=\\{\beta^{(1)}_{1},\beta^{(1)}_{2,3}\\},\cdots,\mathcal{B}_{4}=\\{\beta^{(4)}_{1,2},\beta^{(4)}_{1,3},\beta^{(4)}_{2,3}\\}$
are as in Lemma V.3. We can check that it is still a feasible code if
$\Lambda_{3}=\\{S_{1},S_{2},S_{3}\\}$.

###### Lemma C.5

Suppose $\Lambda_{1}=\\{P_{1},P_{2}\\}$ and $\Lambda_{2}=\\{P_{1},P_{3}\\}$
for some $P_{1}\in\text{reg}^{\circ}(S_{1},S_{2})$,
$P_{2}\in\text{reg}^{\circ}(S_{2},S_{3})$ and
$P_{3}\in\text{reg}^{\circ}(S_{1},S_{3})$. If $\text{RG}(D^{**})$ is
infeasible, then the condition (C-IR) holds.

###### proof C.6

If $\Lambda_{3}\subseteq\Pi\setminus\\{S_{1},S_{2},S_{3}\\}$, then by (2) of
Theorem VI.16, $\text{RG}(D^{**})$ is feasible. So we assume that
$\Lambda_{3}\cap\\{S_{1},S_{2},S_{3}\\}\neq\emptyset$. Moreover, if
$|\Lambda_{3}|\geq 3$, then it is easy to see that
$\mathcal{I}=\\{[P_{1}]\\}\cup\\{[R];R\in\Pi\backslash[P_{1}]\\}$ is a
compatible partition of $\Pi$, where $[P_{1}]=\\{P_{1},P_{2},P_{3}\\}$ and
$[R]=\\{R\\}$ for all $R\in\Pi\backslash[P_{1}]$. By Lemma VI.13,
$\text{RG}(D^{**})$ is feasible. An illustration is given in Fig. 13 (a). So
we further assume that $|\Lambda_{3}|=2$. Thus, by 1) of Lemma C.1, we have
the following three cases:

Case 1: $\Lambda_{3}=\\{S_{3},P\\}$. By 1) of Lemma C.1,
$P\in\text{reg}^{\circ}(S_{1},S_{2})$. By (3) of Theorem VI.16,
$\text{RG}(D^{**})$ is feasible.

Case 2: $\Lambda_{3}=\\{S_{2},P\\}$. By 1) of Lemma C.1,
$P\in\text{reg}^{\circ}(S_{1},S_{3})$. We assert that $\text{RG}(D^{**})$ is
feasible if $P\neq P_{3}$. In fact, let
$\mathcal{I}=\\{[S_{2}],[P_{1}]\\}\cup\\{[R];R\in\Pi\backslash([S_{2}]\cup[P_{1}])\\}$,
where $[S_{2}]=\\{S_{2},P\\},[P_{1}]=\\{P_{1},P_{2},P_{3}\\}$ and
$[R]=\\{R\\}$ for all $R\in\Pi\backslash([S_{2}]\cup[P_{1}])$. By Definition
VI.9, we can check that $\mathcal{I}$ is a compatible partition of $\Pi$. By
Lemma VI.13, $\text{RG}(D^{**})$ is feasible. An illustration is given in Fig.
13 (b).

Case 3: $\Lambda_{3}=\\{S_{1},P\\}$ and $P\in\text{reg}^{\circ}(S_{2},S_{3})$.
Similar to Case 2, we can prove that $\text{RG}(D^{**})$ is feasible if $P\neq
P_{2}$.

By the above discussion, if $\text{RG}(D^{**})$ is infeasible, then either
$\Lambda_{3}=\\{S_{1},P_{2}\\}$ or $\Lambda_{3}=\\{S_{2},P_{3}\\}$. Suppose
$\Lambda_{3}=\\{S_{1},P_{2}\\}$. (See Fig. 14 (a).) Then by renaming regions,
condition (C-IR) holds. (See Fig. 14 (b).) Similarly, if
$\Lambda_{3}=\\{S_{2},P_{3}\\}$, then by renaming regions, condition (C-IR)
holds.

Thus, we proved that if $\text{RG}(D^{**})$ is infeasible, then the condition
(C-IR) holds.

Fig 13: Illustrations of code construction for Lemma C.5: (a) illustrates a
code for $|\Lambda_{3}|\geq 3$; (b) illustrates a code for
$\Lambda_{3}=\\{S_{2},P\\}$.

Fig 14: Illustration of region renaming: The region graph (b) is obtain from
(a) by renaming regions. We can check that (b) satisfies condition (C-IR).

###### Lemma C.7

Suppose $P_{2},P_{3}\in\text{reg}^{\circ}(S_{2},S_{3})$ such that
$\Lambda_{1}=\\{S_{1},P_{2}\\}$ and $\Lambda_{2}=\\{S_{1},P_{3}\\}$. Then
$\text{RG}(D^{**})$ is feasible.

###### proof C.8

If $\Lambda_{3}\cap\text{reg}^{\circ}(S_{2},S_{3})\neq\emptyset$, then by (3)
of Theorem VI.16, $\text{RG}(D^{**})$ is feasible. So we assume
$\Lambda_{3}\cap\text{reg}^{\circ}(S_{2},S_{{}_{3}})=\emptyset$. Then we have

$\displaystyle\Lambda_{3}\subseteq\Pi\setminus\text{reg}^{\circ}(S_{2},S_{{}_{3}})=\text{reg}(S_{1},S_{2})\cup\text{reg}(S_{1},S_{3}).$
(C.1)

By enumerating, we have the following two cases:

Case 1:
$\Lambda_{3}\cap(\text{reg}^{\circ}(S_{1},S_{2})\cup\text{reg}^{\circ}(S_{1},S_{3}))\neq\emptyset$.
Without loss of generality, assume
$Q_{1}\in\Lambda_{3}\cap\text{reg}^{\circ}(S_{1},S_{2})$. Since by 2) of Lemma
IV.9, $\Lambda_{3}\nsubseteq\text{reg}(S_{1},S_{2})$, then by (C.1), there is
a $Q_{2}\in\Lambda_{3}\cap\text{reg}(S_{1},S_{3})\backslash\\{S_{1}\\}$. Let
$\mathcal{I}=\\{[S_{1}],[S_{2}],[S_{3}]\\}$, where
$[S_{1}]=\\{S_{1}\\}\cup\text{reg}^{\circ}(S_{2},S_{3})$,
$[S_{2}]=\\{S_{2}\\}$ and
$[S_{3}]=\text{reg}^{\circ}(S_{1},S_{2})\cup\text{reg}^{\circ}(S_{1},S_{3})\cup\\{S_{3}\\}$.
Then by Definition VI.9, we can check that $\mathcal{I}$ is compatible. By
Lemma VI.13, $\text{RG}(D^{**})$ is feasible. An illustration is given in Fig.
15 (a).

Case 2:
$\Lambda_{3}\cap(\text{reg}^{\circ}(S_{1},S_{2})\cup\text{reg}^{\circ}(S_{1},S_{3}))=\emptyset$.
By (C.1), $\Lambda_{j_{3}}\subseteq\\{S_{1},S_{2},S_{3}\\}$. On the other
hand, by 2) of Lemma IV.9,
$\Lambda_{3}\nsubseteq\text{reg}(S_{i_{1}},S_{i_{2}})$ for all
$\\{i_{1},i_{2}\\}\subseteq\\{1,2,3\\}$. Then
$\Lambda_{3}=\\{S_{1},S_{2},S_{3}\\}$. Let
$\mathcal{I}=\\{[S_{1}]\\}\cup\\{[R];R\in\Pi\backslash[S_{1}]\\}$, where
$[S_{1}]=\\{S_{1}\\}\cup\text{reg}^{\circ}(S_{2},S_{3})$ and
$[R]=\\{R\\},\forall R\in\Pi\backslash[S_{1}]$. We can check that
$\mathcal{I}$ is compatible. By Lemma VI.13, $\text{RG}(D^{**})$ is feasible.
An illustration is given in Fig. 15 (b).

In both cases, $\text{RG}(D^{**})$ is feasible, which completes the proof.

Fig 15: Illustrations of code construction for the proof of Lemma C.7: (a)
illustrates a code for Case 1 and (b) illustrates a code for Case 2\.

###### Lemma C.9

Suppose $\Lambda_{1}=\\{S_{1},P_{1}\\}$ and $\Lambda_{2}=\\{P_{1},P_{2}\\}$
for some $P_{1}\in\text{reg}^{\circ}(S_{2},S_{3})$ and
$P_{2}\in\text{reg}^{\circ}(S_{1},S_{2})$. If $\text{RG}(D^{**})$ is
infeasible, then condition (C-IR) holds.

###### proof C.10

If $\Lambda_{3}\cap\text{reg}^{\circ}(S_{2},S_{3})\neq\emptyset$, then by (3)
of Theorem VI.16, $\text{RG}(D^{**})$ is feasible. So we assume
$\Lambda_{3}\cap\text{reg}^{\circ}(S_{2},S_{3})=\emptyset$. Then we have

$\displaystyle\Lambda_{3}\subseteq\Pi\setminus\text{reg}^{\circ}(S_{2},S_{3})=\text{reg}(S_{1},S_{2})\cup\text{reg}(S_{1},S_{3}).$
(C.2)

Now, suppose

$\displaystyle\Lambda_{3}\nsubseteq\text{reg}(S_{1},P_{2})\cup\text{reg}(S_{1},S_{3}).$
(C.3)

By enumerating, we have the following three cases:

Case 1: $\Lambda_{3}\cap\text{reg}(S_{1},P_{2})\neq\emptyset$. We can assume

$Q_{1}\in\Lambda_{3}\cap\text{reg}(S_{1},P_{2}).$

Since by 2) of Lemma IV.9, $\Lambda_{3}\nsubseteq\text{reg}(S_{1},S_{2})$,
then by (C.2),
$\Lambda_{3}\cap(\text{reg}S_{1},S_{3})\backslash\\{S_{1}\\}\neq\emptyset.$
Assume

$Q_{2}\in\Lambda_{3}\cap\text{reg}(S_{1},S_{3})\backslash\\{S_{1}\\}.$

Moreover, by (C.2) and (C.3),
$\Lambda_{3}\cap\text{reg}(S_{1},S_{2})\backslash\text{reg}(S_{1},P_{2})\neq\emptyset$.
Then we can assume

$Q_{3}\in\Lambda_{3}\cap\text{reg}(S_{1},S_{2})\backslash\text{reg}(S_{1},P_{2}).$

Let $\mathcal{I}=\\{[S_{1}]\\}\cup\\{[R];R\in\Pi\backslash[S_{1}]\\}$, where
$[S_{1}]=\text{reg}(S_{1},P_{2})\cup\\{P_{1}\\}$ and $[R]=\\{R\\}$ for all
$R\in\Pi\backslash[S_{1}]\\}$. We can check that $\mathcal{I}$ is compatible.
By Lemma VI.13, $\text{RG}(D^{**})$ is feasible. An illustration is given in
Fig. 16.

Case 2: $\Lambda_{3}\cap\text{reg}(S_{1},P_{2})=\emptyset$ and
$|\Lambda_{3}|\geq 3.$ Similar to Case 1, we can prove that
$\text{RG}(D^{**})$ is feasible.

Case 3: $\Lambda_{3}\cap\text{reg}(S_{1},P_{2})=\emptyset$ and
$|\Lambda_{3}|=2.$ Since by 2) of Lemma IV.9,
$\Lambda_{3}\nsubseteq\text{reg}(S_{1},S_{2})$ and
$\Lambda_{3}\nsubseteq\text{reg}(S_{1},S_{3})$, then by (C.2) and (C.3),
$\Lambda_{3}=\\{P_{3},P_{4}\\}$ for some
$P_{3}\in\text{reg}(S_{1},S_{2})\backslash\text{reg}(S_{1},P_{2})$ and
$P_{4}\in\text{reg}(S_{1},S_{3})\backslash\\{S_{1}\\}$. Also by 2) of Lemma
IV.9, $\Lambda_{3}\nsubseteq\text{reg}(S_{2},S_{3})$, then
$\Lambda_{3}=\\{P_{3},P_{4}\\}\neq\\{S_{2},S_{3}\\}$. Let
$\mathcal{I}=\\{[S_{1}],[P_{3}]\\}\cup\\{[R];R\in\Pi\backslash[S_{1}]\cup[P_{3}]\\}$,
where $[S_{1}]=\text{reg}(S_{1},P_{2})\cup\\{P_{1}\\}$,
$[P_{3}]=\\{P_{3},P_{4}\\}$ and $[R]=\\{R\\}$ for all
$R\in\Pi\backslash[S_{1}]\cup[P_{3}]\\}$. By Definition VI.9, we can check
that $\mathcal{I}$ is compatible. By Lemma VI.13, $\text{RG}(D^{**})$ is
feasible. An illustration is given in Fig. 17.

Thus, if $\text{RG}(D^{**})$ is infeasible, then (C.3) is violated. So
$\Lambda_{3}\subseteq\text{reg}(S_{1},P_{2})\cup\text{reg}(S_{1},S_{3})$.
Combining assumptions of this lemma, we derived condition (C-IR).

Fig 16: Illustration of code construction for case 1 in the proof of Lemma
C.9: (a) is the region graph and (b) is a code on the graph.

Fig 17: Illustration of code construction for case 3 in the proof of Lemma
C.9: (a) is the region graph and (b) is a code on the graph.

Now we can prove Theorem VI.21. We will prove the necessity and sufficiency
separately.

######  proof C.11 (Proof of Necessity)

Suppose $\text{RG}(D^{**})$ is infeasible. We need to prove the condition
(C-IR). We can prove this by enumerating.

If $\Lambda_{j_{1}}\cap\Lambda_{j_{2}}=\emptyset$ for all
$j_{1},j_{2}\subseteq\\{1,2,3\\}$ such that
$|\Lambda_{j_{1}}|=|\Lambda_{j_{2}}|=2$, then by (1) of Theorem VI.16,
$\text{RG}(D^{**})$ is feasible. So we assume that the collection
$\\{\Lambda_{1},\Lambda_{2},\Lambda_{3}\\}$ contains two subsets which have
size $2$ and have a non-empty intersection. By proper naming, we can assume

$\displaystyle\Lambda_{1}=\\{P_{1},P_{2}\\}~{}\text{and}~{}\Lambda_{2}=\\{P_{1},P_{3}\\}.$
(C.4)

Then by enumerating, we have the following two cases:

Case 1:
$\\{P_{1},P_{2},P_{3}\\}\subseteq\Pi\backslash\\{S_{1},S_{2},S_{3}\\}=\text{reg}^{\circ}(S_{1},S_{2})\cup\text{reg}^{\circ}(S_{1},S_{3})\cup\text{reg}^{\circ}(S_{2},S_{3}).$
Note that by (C.4), $\Lambda_{1}=\\{P_{1},P_{2}\\}$. Then by 2) of Lemma C.1
and by proper naming,

$\displaystyle
P_{1}\in\text{reg}^{\circ}(S_{1},S_{2})~{}\text{and}~{}P_{2}\in\text{reg}^{\circ}(S_{2},S_{3}).$

Again by (C.4), $\Lambda_{2}=\\{P_{1},P_{3}\\}$. Then by 2) of Lemma C.1,
$P_{3}\in\text{reg}^{\circ}(S_{2},S_{3})$ or
$P_{3}\in\text{reg}^{\circ}(S_{1},S_{3})$. If
$P_{3}\in\text{reg}^{\circ}(S_{2},S_{3})$, then by Lemma C.3,
$\text{RG}(D^{**})$ is feasible; Otherwise, by Lemma C.5, the condition (C-IR)
holds.

Case 2: $\\{P_{1},P_{2},P_{3}\\}\cap\\{S_{1},S_{2},S_{3}\\}\neq\emptyset$. By
proper naming, we can assume $S_{1}\in\\{P_{1},P_{2},P_{3}\\}$. Then we can
enumerating the following two subcases:

Case 2.1: $S_{1}=P_{1}$. Then by (C.4) and 1) of Lemma C.1,
$P_{2},P_{3}\in\text{reg}^{\circ}(S_{2},S_{3})$. By Lemma C.7,
$\text{RG}(D^{**})$ is feasible.

Case 2.2: $S_{1}=P_{2}$ or $S_{1}=P_{3}$. By proper naming, we can assume
$S_{1}=P_{3}$. Since by (C.4),
$\Lambda_{2}=\\{P_{1},P_{3}\\}=\\{P_{1},S_{1}\\}$, then by 1) of Lemma C.1,
$P_{1}\in\text{reg}^{\circ}(S_{2},S_{3})$. Moreover, since by (C.4),
$\Lambda_{1}=\\{P_{1},P_{2}\\}$, then by 2) of Lemma C.1 and by proper naming,
we have $P_{2}\in\text{reg}^{\circ}(S_{1},S_{2})$. Now, we interchange the
name of $T_{1}$ and $T_{2}$. Then by Lemma C.9, the condition (C-IR) holds.

Combining the above discussions, we can conclude that if $\text{RG}(D^{**})$
is infeasible, then the condition (C-IR) holds.

######  proof C.12 (Proof of Sufficiency)

Suppose the condition (C-IR) holds, we need to prove that $\text{RG}(D^{**})$
is infeasible. We can prove this by contradiction. Suppose
$\tilde{C}_{\Pi}=\\{d_{R}\in\mathbb{F}^{3};R\in\Pi\\}$ is an arbitrary
feasible code on $\Pi$. Since by the condition (C-IR),
$P_{2}\in\text{reg}^{\circ}(S_{1},S_{2})$, then by 1) of Lemma IV.15,

$\displaystyle d_{P_{2}}\in\langle\alpha_{1},\alpha_{2}\rangle.$

Moreover, since $\Lambda_{1}=\\{S_{1},P_{1}\\}$ and
$\Lambda_{2}=\\{P_{1},P_{2}\\}$, then by 2) of Lemma IV.15,
$\langle\bar{\alpha},d_{S_{1}}\rangle=\langle\bar{\alpha},d_{P_{1}}\rangle=\langle\bar{\alpha},d_{P_{2}}\rangle$.
So

$\displaystyle
d_{P_{2}}\in\langle\bar{\alpha},d_{S_{1}}\rangle\cap\langle\alpha_{1},\alpha_{2}\rangle=\langle\bar{\alpha},\alpha_{1}\rangle\cap\langle\alpha_{1},\alpha_{2}\rangle=\langle\alpha_{1}\rangle.$
(C.5)

By (C.5) and 1) of Lemma IV.15, we have $d_{R}\in\langle\alpha_{1}\rangle$ for
all $R\in\text{reg}(S_{1},P_{2})$ and
$d_{R}\in\langle\alpha_{1},\alpha_{3}\rangle$ for all
$R\in\text{reg}(S_{1},S_{3})$. Since by the condition (C-IR),
$\Lambda_{3}\subseteq\text{reg}(S_{1},P_{2})\cup\text{reg}(S_{1},S_{3})$, then
by condition (3) of Definition IV.11 and (C.5), we have

$\bar{\alpha}\in\langle
d_{R};R\in\Lambda_{3}\rangle\subseteq\langle\alpha_{1},d_{P_{2}},\alpha_{3}\rangle=\langle\alpha_{1},\alpha_{3}\rangle,$

which is a contradiction. Thus, there exists no feasible code on $\Pi$ and by
Theorem IV.12, $\text{RG}(D^{**})$ is infeasible.


#  SelfAct: Personalized Activity Recognition based on Self-Supervised and
Active Learning

Luca Arrotta, Gabriele Civitarese, Samuele Valente, Claudio Bettini
<EMAIL_ADDRESS>EveryWare Lab, Dept. of Computer Science
University of Milan
{luca.arrotta, gabriele.civitarese<EMAIL_ADDRESS>
###### Abstract

Supervised Deep Learning (DL) models are currently the leading approach for
sensor-based Human Activity Recognition (HAR) on wearable and mobile devices.
However, training them requires large amounts of labeled data whose collection
is often time-consuming, expensive, and error-prone. At the same time, due to
the intra- and inter-variability of activity execution, activity models should
be personalized for each user. In this work, we propose SelfAct: a novel
framework for HAR combining self-supervised and active learning to mitigate
these problems. SelfAct leverages a large pool of unlabeled data collected
from many users to pre-train through self-supervision a DL model, with the
goal of learning a meaningful and efficient latent representation of sensor
data. The resulting pre-trained model can be locally used by new users, which
will fine-tune it thanks to a novel unsupervised active learning strategy. Our
experiments on two publicly available HAR datasets demonstrate that SelfAct
achieves results that are close to or even better than the ones of fully
supervised approaches with a small number of active learning queries.

###### Index Terms:

sensor-based activity recognition, self-supervised learning, active learning

##  I Introduction

The recent research on sensor-based Human Activity Recognition (HAR) on
mobile/wearable devices is dominated by solutions based on Deep Learning (DL)
approaches [1]. Supervised learning is the most commonly proposed approach in
this area since is capable of reaching significantly high recognition rates
[2].

However, such approaches require a huge amount of labeled data to create
models capable of generalizing. Indeed, the high intra- and inter-variability
of activity execution is a significant challenge in this domain, and
personalized models (i.e., models trained only with data from the target
users) are the ones performing the best [3].

However, annotation is costly, time-consuming, intrusive, and thus often
prohibitive. For this reason, several research groups are investigating
solutions for HAR requiring limited labeled datasets [4].

Among these approaches, self-supervised learning methods are emerging [5, 6,
7]. The goal of self-supervised learning is to leverage large corpora of
unlabeled data to learn to extract effective features from sensor data. While
self-supervised approaches showed their effectiveness on NLP and computer
vision tasks [8], their application to sensor-based HAR in real-world
scenarios is still an open research question.

One of the major drawbacks of existing approaches is that they assume that a
small labeled dataset for fine-tuning is available. However, those works do
not investigate how this dataset can be obtained in practice.

In the literature, active learning has been widely investigated to obtain
annotated data samples in HAR data scarcity scenarios [9, 4, 10]. However,
those approaches assume that a pre-training labeled dataset is available to
initialize the recognition model. Indeed, a query is triggered based on the
confidence level of the classifier. Applying the concept of active learning to
a self-supervised setting is not trivial, since it requires deciding when to
trigger queries by analyzing an unlabeled data stream.

In this work, we propose SelfAct: a novel framework bridging the gap between
self-supervised learning and active learning. SelfAct relies on self-
supervised learning to learn, from a large number of users, an effective
feature representation of sensor data. Then, each user who wants to use
SelfAct downloads this pre-trained model on personal mobile/wearable devices.
After the first phase where the personal device accumulates unlabeled sensor
data, a clustering algorithm on the embeddings generated by the pre-trained
model is performed. After the accumulation phase, SelfAct analyzes in real-
time each data sample in the stream and matches it with the closest cluster. A
novel unsupervised active learning strategy, based on cluster density, is
adopted to decide whether to trigger a query or not. Labeled data samples
obtained through active learning are finally used to fine-tune the pre-trained
self-supervised model and personalize it on the specific user.

The contributions of this work are the following:

  * •

We propose SelfAct: a framework for personalized sensor-based HAR on
mobile/wearable devices based on self-supervised learning.

  * •

We design and integrate into SelfAct a novel unsupervised active learning
strategy to obtain labeled data samples. These samples are used to fine-tune a
self-supervised model.

  * •

Our experiments on two public datasets show that SelfAct achieves similar or
even better recognition rates than fully supervised approaches with a very
limited amount of active learning queries.

##  II Related work

Most of the existing works in the literature tackling sensor-based HAR on
mobile/wearable devices are based on supervised Deep Learning (DL) models [1].
However, during their learning process, such models require large amounts of
labeled data that are often prohibitive to acquire [11]. Moreover, activity
models should be personalized for each user to capture her peculiarities in
activity execution [11]. Several research groups are indeed working hard to
mitigate the data scarcity problem, by proposing several categories of
solutions.

Some research efforts proposed solutions based on automatic data annotation.
Some of these works assume the existence of previously annotated datasets to
pre-train an activity model, that is used for annotation [12]. Other
approaches require auxiliary data sources in the environment (e.g., acoustic
data from microphones) to automatically annotate activity data [13]. Finally,
other works proposed knowledge-based heuristic approaches to generate weak
labels (e.g., combining step count and GPS data) [14]. However, the
scalability of such an approach is questionable since the heuristic was
evaluated considering data collected by monitoring a single user. Moreover,
the proposed method can not be used for data annotation when multiple
activities share similar patterns in terms of step count and GPS data (e.g.,
standing and sitting).

Some research groups focused on solutions based on transfer learning [15, 16].
However, such methods rely on models that are pre-trained with significant
amounts of labeled data available in a source domain and then fine-tuned in
the target domain with a few annotated samples.

Another category of solutions to mitigate the data scarcity problem is semi-
supervised learning [17]. However, these approaches still assume the existence
of a small labeled dataset to initialize an activity classifier, that is then
incrementally updated with samples of the unlabeled data stream annotated
through techniques like self-learning, co-learning, active learning [18, 9,
19], and label propagation [4].

Unsupervised approaches have also been proposed to tackle data scarcity. For
instance, some research groups directly relied on clustering techniques to
distinguish the different user’s activities [20, 21, 22, 23]. However, they
did not consider how to match clusters with the actual activity classes the
user performs in realistic scenarios.

More recently, self-supervised learning has been studied to efficiently learn
(without supervision) meaningful features from sensor data for both mobile
[24, 25, 26] and smart home [27] activity recognition. The majority of
existing solutions assume the availability of a labeled dataset to fine-tune
the self-supervised model and obtain the final activity classifier. A closely
related work [28] uses a labeled validation dataset to match activities with
clusters identified by k-means on self-supervised embeddings. Different from
this work, SelfAct does not assume the knowledge of the number of activities.
Also, SelfAct includes an unsupervised active learning strategy to select the
samples representing each cluster to label only a few data samples through
active learning. Overall, the major drawback of these works based on self-
supervised learning is that they do not clarify how fine-tuning datasets can
be obtained in realistic settings.

Finally, the work in [29] considered BERT to extract from smart home sensor
data a latent representation of the action units performed by the resident
(e.g., a movement in the kitchen) that are then clustered. Hence, frequently
occurring patterns of action units are found through a motif discovery
process. The most frequent motifs are finally labeled by the resident thanks
to active learning. However, NLP-based techniques like BERT and motif
discovery work well for discrete environmental sensors, but they cannot be
directly applied to the raw inertial measurements collected by mobile and
wearable devices.

##  III The SelfAct framework

Figure 1: The overall architecture of SelfAct

In this section, we present SelfAct: our framework that generates a
personalized activity classifier by combining self-supervised learning and
unsupervised active learning.

As depicted in Figure 1, in the first phase (called self-supervised phase),
SelfAct requires huge amounts of unlabeled inertial sensor data automatically
and collaboratively collected from the mobile/wearable devices of multiple
users (e.g., volunteers). This unlabeled dataset is used to train, in a self-
supervised fashion, a deep neural network capable of extracting effective
latent representations (i.e., embeddings) from sensor data. This process is
described in detail in Section III-A.

In a second phase (called accumulation phase), new users start using SelfAct
by downloading and deploying the pre-trained self-supervised model on their
devices. Each user locally accumulates unlabeled inertial sensor data for a
certain period. At the end of the accumulation phase, such unlabeled data are
transformed into embeddings with the pre-trained model and then used to train
a personal dimensionality reduction model. Finally, the reduced embeddings are
clustered. Intuitively, each cluster would likely represent a personal sensor
data pattern. More details about this accumulation phase are presented in
Section III-B.

After the accumulation phase, SelfAct starts the active learning phase
(presented in detail in Section III-C). This phase analyzes, in real-time,
each new sensor data sample in the stream to decide whether to ask the user
for feedback about the corresponding label (i.e., the activity that the user
is actually performing). Intuitively, SelfAct aims at asking for a label only
whenever a sample is recognized as representative of a cluster.

Thanks to active learning, SelfAct automatically collects a small amount of
labeled data specific to the subject and this data is used during the fine-
tuning phase (presented in Section III-D) to obtain a personalized activity
classifier.

The SelfAct’s algorithm running locally on clients is summarized in Algorithm
1. In the following, we will describe each phase of SelfAct in detail.

Algorithm 1 Client algorithm of SelfAct

Input: the Feature Extraction Layers $fe$ of the pre-trained self-supervised
model, the accumulation threshold $ACC\\_TH$

1:$fth\leftarrow$ randomly initialized layers of the Fine-Tuning Head

2:$dr\leftarrow$ NIL

3:$samplesNumber\leftarrow 0$

4:$storage\leftarrow\emptyset$

5:$reducedStorage\leftarrow\emptyset$

6:$labeledSamples\leftarrow\emptyset$

7:$clusters\leftarrow\emptyset$

8:for $s$ data sample in the stream do

9: $samplesNumber\leftarrow samplesNumber+1$

10: $emb_{s}\leftarrow$ obtain embeddings using $fe$

11: $storage\leftarrow storage\bigcup\\{emb_{s}\\}$

12: if $samplesNumber==ACC\\_{TH}$ then

13: $dr\leftarrow$ data reduction model trained using $storage$

14: $reducedStorage\leftarrow$ embeddings in $storage$ reduced using $dr$

15: $clusters\leftarrow$ apply clustering algorithm on $reducedStorage$

16: else

17: if $samplesNumber>ACC\\_{TH}$ then

18: $emb_{s}^{\prime}\leftarrow$ dimensionally reduced $emb_{s}$ using $dr$
model

19: $matchingClust\leftarrow$ closest cluster to $emb_{s}^{\prime}$ in
$clusters$

20: if $ActiveLearningNeeded(emb_{s}^{\prime},matchingClust)$ then

21: $l\leftarrow$ feedback from user on the activity associated with $s$

22: $labeledSamples\leftarrow labeledSamples\bigcup\\{(s,l)\\}$

23: end if

24: $matchingClust\leftarrow matchingClust\bigcup\\{emb_{s}^{\prime}\\}$

25: if Fine-tuning conditions are verified then

26: adjust $fth$ to cover all the classes in $labeledSamples$

27: fine-tune $fe+fth$ using $labeledSamples$

28: $fe+fth$ can be used to classify unlabeled samples

29: end if

30: end if

31: end if

32:end for

###  III-A Self-supervised phase

The goal of the self-supervised phase is to train, in a self-supervised
fashion, a deep learning model capable of transforming raw sensor data samples
into an efficient latent representation (i.e., embeddings). Intuitively, data
samples representing the same activity pattern should be close in the latent
space, while data samples from different activity patterns should be distant.

In this phase, SelfAct relies on unlabeled raw sensor data that can be
collaboratively and automatically collected from the mobile/wearable devices
of several users. For the sake of this work, we do not focus on how such data
are actually collected. For instance, such users can be volunteers involved in
a large data collection campaign. Alternatively, unlabeled data could be
collected by combining existing publicly available datasets for HAR.

SelfAct pre-processes and segments the available unlabeled data into fixed-
length segmentation windows that are used to train the self-supervised model.
During the self-supervised phase, this model is composed of two parts: i)
Feature Extraction Layers that learns how to extract embeddings from sensor
data, and ii) a Self-Supervised Head, necessary to perform the self-supervised
training process. SelfAct does not impose a specific self-supervised approach.
As we will discuss in Section IV, in our experiments we implemented SelfAct
with a version of the SimCLR [30] approach specifically adapted for the HAR
domain [25].

The output of the self-supervised phase is a pre-trained deep-learning model
only composed of the Feature Extraction Layers, that is able to extract
embeddings from unlabeled sensor data. This model can be locally deployed on
the mobile devices of the new users that start using SelfAct.

###  III-B Accumulation phase

The accumulation phase starts when a new user downloads and locally deploys on
her device the model pre-trained in the self-supervised phase (i.e., Feature
Extraction Layers). The goal of the accumulation phase is to accumulate from
the user sufficient amounts of unlabeled data to reliably discover personal
activity patterns through clustering.

As depicted in Figure 2,

Figure 2: The accumulation phase of SelfAct

the user’s mobile device starts collecting the stream of unlabeled inertial
sensor data that is segmented into fixed-length windows until an arbitrary
number of $ACC\\_TH$ (accumulation threshold) samples is acquired. This
parameter should be tuned based on the specific application, and, in real-
world scenarios, its value should be high in order to collect unlabeled data
over a sufficiently long time period (e.g., a day, or a week).

Once unlabeled data accumulation is completed, each sample is provided to the
pre-trained self-supervised model to obtain embeddings. Then, to efficiently
perform clustering to find activity patterns, we leverage all the generated
embeddings to train a personalized dimensionality reduction model. Indeed,
recent works demonstrated the impact of dimensionality reduction in improving
the performance of clustering techniques [31]. Finally, the dimensionally
reduced accumulated embeddings are provided to a clustering algorithm that
does not make any prior assumption on the number of clusters (e.g., DBSCAN).
The goal is to group in the same cluster those embeddings that have been
generated while the user was performing the same activity pattern. We expect
our method to obtain more than one cluster for each activity, since users may
perform the same activity in different ways (e.g., walking indoors and walking
outdoors).

###  III-C Active learning phase

The goal of the active learning phase is to select, in real-time, data samples
that should be labeled by the user. These labeled samples are then used by
SelfAct to fine-tune a personalized activity classifier.

As shown in Figure 3,

Figure 3: The active learning phase of SelfAct

in the active learning phase, each new unlabeled sample collected by the
user’s mobile device is provided to the pre-trained model, and the result is
dimensionally reduced through the personalized dimensionality reduction model.
Each new reduced embedding is then mapped to the closest cluster among the
ones derived during the accumulation phase (e.g., the cluster with the closest
centroid). The active learning strategy of SelfAct asks for a label only for
those samples that can be considered as highly representative of the closest
cluster (i.e., of each activity pattern) in order to minimize the number of
queries that are prompted to the user.

More specifically, an active learning process is started by evaluating whether
a new reduced embeddings sample $s$ improves the density of its cluster or
not. SelfAct assigns to each cluster $c$ a threshold $T_{c}$, which reflects
the density level of the cluster. This threshold is automatically computed as
the average distance between each pair of points within the cluster.

To determine whether a new sample $s$ is highly representative of cluster $c$,
we evaluate the average distance between each pair of points in $c$ after
adding $s$ to the cluster. If this average distance is lower than $T_{c}$,
then $s$ is considered highly representative of $c$ and an active learning
query is requested to the user. Intuitively, this happens when $s$ is a sample
that increases the density of $c$.

Algorithm 2 shows the pseudo-code of the strategy adopted by SelfAct to decide
if an active learning query is needed.

Algorithm 2 ActiveLearningNeeded

Input: a new sample $emb^{\prime}_{s}$, and its closest cluster
$matchingClust$
Output: a boolean value

1:$avg\leftarrow$ average distance between points in $matchingClust$

2:$avg^{\prime}\leftarrow$ average distance between points in
$matchingClust\bigcup\\{emb_{s}^{\prime}\\}$

3:if $avg^{\prime}<avg$ then

4: return True

5:end if

6:return False

Independently if the active learning query is triggered or not, $s$ is added
to $c$, and $T_{c}$ is updated accordingly. The active learning strategy of
SelfAct assumes that the clusters found during the accumulation phase are
reliable. For this reason, as we will describe in Section IV, we empirically
determined $ACC\\_TH$, by balancing the trade-off between recognition rate and
a number of triggered questions.

###  III-D Fine-tuning phase

Finally, during the fine-tuning phase, SelfAct relies on the data labeled
through active learning at the previous phase to fine-tune the deep-learning
model pre-trained during the self-supervised phase. In this way, SelfAct
builds an activity classifier personalized for each new user of the framework.
Specifically, SelfAct dynamically adds to the Feature Extraction Layers of the
deep-learning model pre-trained during the self-supervised phase a Fine-Tuning
Head that includes classification layers. Since it is not possible to know in
advance how many activities (and which) the user actually performed, the Fine-
Tuning Head is created dynamically based on the number of activity classes in
the labeled data. During the fine-tuning process, the layers of both the
Feature Extraction Module and the Fine-Tuning Head are unfrozen.

The fine-tuning phase should be triggered by considering specific conditions.
For instance, it can be a periodic task (e.g., with a time periodicity or each
time a certain amount of labels are collected). However, defining when fine-
tuning should be started is out of the scope of this paper and it will be
investigated in future work.

##  IV Experimental evaluation

TABLE I: Main results of SelfAct. The accumulation phase was performed on 75% of the test set. The table reports the average number of active learning queries prompted to the user by SelfAct (i.e., number of samples for fine-tuning), and the average number of labeled samples in the training set used to train fully supervised Dataset |  | Average number of
---
active learning queries
(fine-tuning samples)
SelfAct F1 |  | Average supervised
---
training set size
| Fully supervised
---
F1
HHAR | 369 | 0.9734 | 25053 | 0.8792
MotionSense | 143 | 0.9291 | 5304 | 0.9483

This section describes the experiments we carried out to evaluate SelfAct. We
first introduce the two publicly available datasets we considered for our
experiments. Then, we discuss details about our experimental setup: the data
pre-processing, the specific models we used to implement SelfAct, and the
evaluation methodology we adopted. Finally, we present the results of our
evaluation.

###  IV-A Datasets

####  IV-A1 HHAR

The Heterogeneity Human Activity Recognition (HHAR) dataset [32] includes
accelerometer and gyroscope data from $9$ users wearing a smartphone (in a
pouch carried on the waist) and a smartwatch. The dataset was collected
considering $6$ different activities: biking, sitting, standing, walking,
stairs up, and stairs down. HHAR includes heterogeneous sampling rates, based
on the specific devices worn by the user. In particular, the dataset includes
$2$ different smartwatch devices: LG Watch and Samsung Galaxy Gears with
maximum sampling rates of $200Hz$ and $100Hz$, respectively. Moreover, HHAR
contains $4$ different smartphone models: Samsung Galaxy S3 mini (maximum
sampling rate of $100Hz$), Samsung Galaxy S3 ($150Hz$), LG Nexus 4 ($200Hz$),
and Samsung Galaxy S+ ($50Hz$).

####  IV-A2 MotionSense

MotionSense [33] is a publicly available HAR dataset collected from $24$
subjects carrying an iPhone 6s in their trousers’ front pocket, and performing
$6$ different activities: walking downstairs, walking upstairs, walking,
jogging, sitting, and standing. The dataset includes both accelerometer and
gyroscope data sampled at $50Hz$.

###  IV-B Experimental setup

In the following, we describe our experimental setup.

####  IV-B1 Machine learning models

As self-supervised learning model, in our experiments, we used a version of
the SimCLR technique [30] adapted for HAR in previous works [25, 7].
Consistently with the previous work, the Feature Extraction Layers consist of
three 1D convolutional layers with $32$, $64$, and $96$ filters and $24$,
$16$, and $8$ kernel sizes, respectively, followed by a global maximum pooling
layer. Each couple of consecutive layers is separated by a dropout layer with
a dropout rate of $0.1$. The Self-Supervised Head consists of three fully
connected layers with $256$, $128$, and $50$ neurons, respectively. The self-
supervised model is pre-trained for $50$ epochs and a batch size of $512$ with
the SGD optimizer, considering a cosine decay for the learning rate. Moreover,
we empirically determined the optimal transformations to apply to the
unlabeled data during SimCLR pre-training, finding that, consistently with
[25], the application of the rotation transformation only was the best option.

The Fine-Tuning Head consists of a fully connected layer containing $1024$
neurons and a softmax layer for the final classification. Fine-tuning is
performed by running $50$ training epochs with a batch size of $1$, using an
early stopping technique that stopped the training process when validation
loss did not improve for $5$ consecutive epochs, and adopting the Adam
optimizer.

As a dimensionality reduction model, we adopted the Uniform Manifold
Approximation and Projection (UMAP) technique [34] since previous work
demonstrated how its use is effective in considerably improving the
performance of clustering algorithms [31]. Moreover, since SelfAct considers a
real-time scenario for active learning, it is crucial to train a
dimensionality reduction model only at the end of the accumulation phase, and
then to map new data samples in the reduced space. These real-time constraints
can be addressed by using UMAP, while t-SNE [35], for example, would not be
appropriate. During our evaluation, we experimentally determined the best
hyperparameters for UMAP: $15$ as the number of neighbors, and an euclidean
minimum distance equal to $0.4$.

Realistic and real-time constraints mainly guided the choice of the clustering
algorithm of SelfAct. First of all, it is not realistic to assume knowing the
exact number of different activity patterns that the user will perform while
using SelfAct. Thus, it would not be appropriate to consider clustering
algorithms that require knowing the number of clusters in advance (e.g.,
K-means). Moreover, in real-time scenarios, we need to rely on an algorithm
that (i) clusters the reduced embeddings accumulated during the accumulation
phase, and then (ii) directly maps new data into the existing clusters during
the active learning phase. For these reasons, in our experiments, we relied on
the HDBSCAN clustering algorithm [36]. In particular, we empirically
determined its hyperparameters: a minimum cluster size of $6$, and a minimum
number of samples of $10$.

####  IV-B2 Pre-processing

Since the version of SimCLR we are using only considers data from a single
tri-axial sensor as input, we focus on the accelerometer data collected
through the users’ smartphones. Such data are segmented into fixed-length
windows with a $50\%$ of overlap. In particular, following the evaluation
protocol proposed in [25], we considered windows of $400$ samples for
MotionSense. On the other hand, since HHAR includes data collected with
different sampling rates, we experimentally determined the size of the
windows, obtaining a window size of $800$ samples. Due to unsatisfying
recognition rates during preliminary experiments, in MotionSense we grouped
the labels walking downstairs and walking upstairs into the label walking
stairs, while in HHAR we grouped stairs down and stairs up into stairs.
Indeed, these activities are particularly challenging to discriminate.

####  IV-B3 Baseline

In our evaluation, we compared SelfAct with a fully supervised approach: a
deep neural network with the same architecture of the network used by SelfAct
during the fine-tuning phase, trained in a supervised way considering full
availability of labeled training data.

####  IV-B4 Evaluation Methodology

The SelfAct framework is designed for personalizing the HAR model for each
user by leveraging a limited amount of labeled data. Hence, we performed our
experiments considering the leave-one-subject-out cross-validation technique.
At each iteration of the cross-validation process, data from one user is used
as the test set (i.e., accumulation, active learning, and fine-tuning phases),
while the remaining folds are used as the training set (i,e., self-supervised
phase). At the end of the process, we averaged the results obtained at each
iteration in terms of weighted F1 score.

Considering the fully supervised baseline, at each fold, $10\%$ of the
training set was used as the validation set. On the other hand, in order to
simulate the accumulation and active learning phases of SelfAct, a certain
percentage (i.e. accumulation threshold) of the test set was used to simulate
the samples that are accumulated by SelfAct during the accumulation phase. The
remaining samples of the test set were used to simulate the data stream
processed during the active learning phase111This setting resembles a scenario
in which practitioners must allocate both accumulation and the active learning
phases within a given timeframe (e.g., one month) that is constrained by
deployment or project requirements..

During fine-tuning, we used $10\%$ of labeled data as the validation set.
Finally, all the samples in the test set that were not labeled (i.e., the ones
collected in the accumulation phase and the samples not labeled during the
active learning phase) were used to assess the recognition rate of the fine-
tuned model. We also compute the active learning rate, i.e., the ratio between
the amount of triggered active learning questions and the total number of
samples available during the active learning phase.

While it was straightforward to apply the leave-one-out cross-validation on
HHAR, it was more challenging on MotionSense. Indeed, users in this dataset do
not have enough data samples to obtain meaningful results from the
accumulation and active learning phases. Hence, we created groups of $4$ or
$5$ users performing activities in a similar way (based on available meta-data
like age, height, weight, etcetera) and we considered each group as a single
user. This resulted in $5$ pseudo-users that we used to perform the leave-one-
out cross-validation.

(a) Recognition rates of SelfAct by varying the accumulation threshold

(b) Number of clusters derived by SelfAct at the end of the accumulation phase
by varying the accumulation threshold

(c) Active learning rate during the active learning phase of SelfAct by
varying accumulation threshold

Figure 4: The impact of accumulation threshold on recognition rate, number of
clusters, and active learning rate

###  IV-C Results

In the following, we describe the results we obtained during our experimental
evaluation.

####  IV-C1 Recognition rate

Table I compares SelfAct and the fully supervised baseline in terms of F1
score and the average number of necessary labeled data to deploy such
solutions on both datasets. Such results were obtained by accumulating during
the accumulation phase a number of samples that corresponds to $75\%$ of the
data available in the test set. Considering HHAR, SelfAct dramatically
outperforms fully supervised in terms of F1 score (i.e., $\approx+10\%$).
Moreover, this result is obtained considering only $369$ labeled samples per
user, while the fully supervised baseline, at each iteration of the cross-
validation, relied on average on $\approx 25k$ labeled samples.

On the other hand, considering MotionSense, our framework did not improve the
recognition rates of fully supervised. This happens since the active learning
phase of SelfAct is actually computed on the data of more users (see Section
IV-B4). Hence, with a few fine-tuning samples labeled with active learning, it
is hard to cover all the possible ways in which an activity can be performed
by different users. Nonetheless, the F1 score reached by SelfAct is only
$\approx 2\%$ behind fully supervised, but relying only on $143$ labeled
samples per group of users included in the test set, instead of considering on
average more than $5k$ labeled training samples. Hence, we believe that these
results still highlight the effectiveness of SelfAct, which reaches
recognition rates close to supervised approaches with a limited amount of
labeled samples.

####  IV-C2 Impact of accumulation threshold

Figure 4 depicts how the accumulation threshold affects the performance of
SelfAct. By setting the accumulation threshold at $50\%$ (i.e., $50\%$ of data
used for accumulation and $50\%$ of data in the active learning phase) we get
a high recognition rate, at the cost of having the highest active learning
rate. Indeed, with this threshold, SelfAct finds a lower number of sparse
clusters, thus requiring a higher number of samples during the active learning
phase to increase their densities. This also implies a higher number of active
learning queries.

On the other hand, when the accumulation threshold is $95\%$ (i.e., $95\%$ of
data used for accumulation and $5\%$ of data in the active learning phase) the
recognition rate is the lowest. This is due to the fact that this setting does
not provide a sufficient amount of data in the active learning phase for
properly fine-tuning the model.

Hence, we observed that setting the threshold to $75\%$ (i.e., $75\%$ of data
used for accumulation and $25\%$ of data in the active learning phase) was the
best trade-off between the recognition rate (that is similar to the one
obtained with the threshold set to $50\%$) with a significantly reduced active
learning rate. Indeed, this setting allows SelfAct to identify clusters that
better reflect activity patterns and that can be effectively used for the
active learning phase.

In general, our results show that increasing the accumulation threshold
decreases the active learning rate. This is due to the fact that our
unsupervised learning strategy is based on cluster density. Indeed, the higher
the amount of accumulated data and the denser the clusters.

##  V Conclusion and Future Work

In this paper, we presented SelfAct: a framework for personalized sensor-based
HAR. Thanks to a novel combination of self-supervised learning and active
learning, SelfAct requires only a limited amount of labeled data from target
users to fine-tune a model pre-trained with unlabeled data only. While the
results obtained so far are promising, we plan to investigate several aspects
in future work.

First, even though SelfAct does not assume knowing in advance the activities
performed by the target user, in this work the pre-trained feature extractor
and fine-tuning are trained considering the same activity classes. In future
work, we want to truly assess the capability of SelfAct for activity discovery
and domain adaptation. A challenge will be to distinguish outliers from new
activities.

A drawback of SelfAct is that it relies on a fixed threshold to decide when
switching from accumulation phase to active learning phase. In future work, we
will investigate automatic methods based on the silhouette score of the
clusters.

Another limitation of SelfAct is that, while it requires limited labeled data
samples to obtain high recognition rates, the active learning query rate is
relatively high. This aspect has a negative impact on usability. We will
investigate more sophisticated active learning strategies aiming at further
minimizing the number of required samples. For instance, an interesting
strategy is to establish a budget of queries for each cluster. Nonetheless,
these results are also probably due to the fact that the considered datasets
are small, and we could not evaluate SelfAct with large amounts of data for
the accumulation and active learning phases. We expect that, in real-world
scenarios where it is possible to collect a large amount of unlabeled data,
the active learning rate would be significantly lower.

SelfAct also assumes that it is possible to store and process locally
unlabeled and labeled datasets. However, in real-world scenarios, unlabeled
data may be collected for a significantly long time. For instance, the
accumulation phase may last several days. We will investigate solutions based
on trusted edge devices with storage and computational capabilities. Users’
devices should be only in charge of running the fine-tuned classifier.

## Acknowledgment

This work was partially supported by the project “MUSA - Multilayered Urban
Sustainability Action”, NextGeneration EU, MUR PNRR.

## References

  * [1] J. Wang, Y. Chen, S. Hao, X. Peng, and L. Hu, “Deep learning for sensor-based activity recognition: A survey,” Pattern recognition letters, vol. 119, pp. 3–11, 2019.
  * [2] M. Ronald, A. Poulose, and D. S. Han, “isplinception: An inception-resnet deep learning architecture for human activity recognition,” IEEE Access, vol. 9, pp. 68985–69001, 2021.
  * [3] G. M. Weiss and J. Lockhart, “The impact of personalization on smartphone-based activity recognition,” in Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence, 2012.
  * [4] R. Presotto, G. Civitarese, and C. Bettini, “Semi-supervised and personalized federated activity recognition based on active learning and label propagation,” Personal and Ubiquitous Computing, vol. 26, no. 5, pp. 1281–1298, 2022.
  * [5] B. Khaertdinov, E. Ghaleb, and S. Asteriadis, “Contrastive self-supervised learning for sensor-based human activity recognition,” in 2021 IEEE International Joint Conference on Biometrics (IJCB), pp. 1–8, IEEE, 2021.
  * [6] Y. Jain, C. I. Tang, C. Min, F. Kawsar, and A. Mathur, “Collossl: Collaborative self-supervised learning for human activity recognition,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 6, no. 1, pp. 1–28, 2022.
  * [7] H. Haresamudram, I. Essa, and T. Plötz, “Assessing the state of self-supervised human activity recognition using wearables,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 6, no. 3, pp. 1–47, 2022.
  * [8] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “Data2vec: A general framework for self-supervised learning in speech, vision and language,” in International Conference on Machine Learning, pp. 1298–1312, PMLR, 2022.
  * [9] H. S. Hossain and N. Roy, “Active deep learning for activity recognition with context aware annotator selection,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1862–1870, 2019.
  * [10] Z. S. Abdallah, M. M. Gaber, B. Srinivasan, and S. Krishnaswamy, “Streamar: incremental and active learning with evolving sensory data for activity recognition,” in 2012 IEEE 24th International Conference on Tools with Artificial Intelligence, vol. 1, pp. 1163–1170, IEEE, 2012.
  * [11] K. Chen, D. Zhang, L. Yao, B. Guo, Z. Yu, and Y. Liu, “Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities,” ACM Computing Surveys (CSUR), vol. 54, no. 4, pp. 1–40, 2021.
  * [12] I. Hassan, A. Mursalin, R. B. Salam, N. Sakib, and H. Z. Haque, “Autoact: An auto labeling approach based on activities of daily living in the wild domain,” in 2021 Joint 10th International Conference on Informatics, Electronics & Vision (ICIEV) and 2021 5th International Conference on Imaging, Vision & Pattern Recognition (icIVPR), pp. 1–8, IEEE, 2021.
  * [13] S. Chatterjee, A. Chakma, A. Gangopadhyay, N. Roy, B. Mitra, and S. Chakraborty, “Laso: Exploiting locomotive and acoustic signatures over the edge to annotate imu data for human activity recognition,” in Proceedings of the 2020 International Conference on Multimodal Interaction, pp. 333–342, 2020.
  * [14] F. Cruciani, I. Cleland, C. Nugent, P. McCullagh, K. Synnes, and J. Hallberg, “Automatic annotation for human activity recognition in free living using a smartphone,” Sensors, vol. 18, no. 7, p. 2203, 2018.
  * [15] A. R. Sanabria, F. Zambonelli, and J. Ye, “Unsupervised domain adaptation in activity recognition: A gan-based approach,” IEEE Access, vol. 9, pp. 19421–19438, 2021.
  * [16] E. Soleimani and E. Nazerfard, “Cross-subject transfer learning in human activity recognition systems using generative adversarial networks,” Neurocomputing, vol. 426, pp. 26–34, 2021.
  * [17] Z. S. Abdallah, M. M. Gaber, B. Srinivasan, and S. Krishnaswamy, “Activity recognition with evolving data streams: A review,” ACM Computing Surveys (CSUR), vol. 51, no. 4, pp. 1–36, 2018.
  * [18] R. Adaimi and E. Thomaz, “Leveraging active learning and conditional mutual information to minimize data annotation in human activity recognition,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 3, no. 3, pp. 1–23, 2019.
  * [19] Y. Cui, S. K. Hiremath, and T. Ploetz, “Reinforcement learning based online active learning for human activity recognition,” in Proceedings of the 2022 ACM International Symposium on Wearable Computers, pp. 23–27, 2022.
  * [20] D. Trabelsi, S. Mohammed, F. Chamroukhi, L. Oukhellou, and Y. Amirat, “An unsupervised approach for automatic activity recognition based on hidden markov model regression,” IEEE Transactions on automation science and engineering, vol. 10, no. 3, pp. 829–835, 2013.
  * [21] Y. Kwon, K. Kang, and C. Bae, “Unsupervised learning for human activity recognition using smartphone sensors,” Expert Systems with Applications, vol. 41, no. 14, pp. 6067–6074, 2014.
  * [22] A. Abedin, F. Motlagh, Q. Shi, H. Rezatofighi, and D. Ranasinghe, “Towards deep clustering of human activities from wearables,” in Proceedings of the 2020 ACM International Symposium on Wearable Computers, pp. 1–6, 2020.
  * [23] A. Ahmed, H. Haresamudram, and T. Ploetz, “Clustering of human activities from wearables by adopting nearest neighbors,” in Proceedings of the 2022 ACM International Symposium on Wearable Computers, pp. 1–5, 2022.
  * [24] A. Saeed, T. Ozcelebi, and J. Lukkien, “Multi-task self-supervised learning for human activity detection,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 3, no. 2, pp. 1–30, 2019\.
  * [25] C. I. Tang, I. Perez-Pozuelo, D. Spathis, and C. Mascolo, “Exploring contrastive learning in human activity recognition for healthcare,” arXiv preprint arXiv:2011.11542, 2020.
  * [26] B. Almaslukh, J. AlMuhtadi, and A. Artoli, “An effective deep autoencoder approach for online smartphone-based human activity recognition,” Int. J. Comput. Sci. Netw. Secur, vol. 17, no. 4, pp. 160–165, 2017.
  * [27] D. Bouchabou, S. M. Nguyen, C. Lohr, B. LeDuc, and I. Kanellos, “Using language model to bootstrap human activity recognition ambient sensors based in smart homes,” Electronics, vol. 10, no. 20, p. 2498, 2021.
  * [28] H. Ma, Z. Zhang, W. Li, and S. Lu, “Unsupervised human activity representation learning with multi-task deep clustering,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 5, no. 1, pp. 1–25, 2021.
  * [29] S. K. Hiremath, Y. Nishimura, S. Chernova, and T. Plötz, “Bootstrapping human activity recognition systems for smart homes from scratch,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 6, no. 3, pp. 1–27, 2022.
  * [30] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International conference on machine learning, pp. 1597–1607, PMLR, 2020.
  * [31] M. Allaoui, M. L. Kherfi, and A. Cheriet, “Considerably improving clustering algorithms using umap dimensionality reduction technique: a comparative study,” in Image and Signal Processing: 9th International Conference, ICISP 2020, Marrakesh, Morocco, June 4–6, 2020, Proceedings 9, pp. 317–325, Springer, 2020.
  * [32] A. Stisen, H. Blunck, S. Bhattacharya, T. S. Prentow, M. B. Kjærgaard, A. Dey, T. Sonne, and M. M. Jensen, “Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition,” in Proceedings of the 13th ACM conference on embedded networked sensor systems, pp. 127–140, 2015.
  * [33] M. Malekzadeh, R. G. Clegg, A. Cavallaro, and H. Haddadi, “Protecting sensory data against sensitive inferences,” in Proceedings of the 1st Workshop on Privacy by Design in Distributed Systems, pp. 1–6, 2018.
  * [34] L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” arXiv preprint arXiv:1802.03426, 2018.
  * [35] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.,” Journal of machine learning research, vol. 9, no. 11, 2008.
  * [36] L. McInnes, J. Healy, and S. Astels, “hdbscan: Hierarchical density based clustering.,” J. Open Source Softw., vol. 2, no. 11, p. 205, 2017.