# ContextGPT: Infusing LLMs Knowledge into Neuro-Symbolic Activity Recognition
Models

Luca Arrotta, Claudio Bettini, Gabriele Civitarese, Michele Fiori
luca.arrotta, claudio.bettini, gabriele.civitarese<EMAIL_ADDRESS>EveryWare Lab, Dept. of Computer Science, University of MilanMilanItaly

(2018; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Context-aware Human Activity Recognition (HAR) is a hot research area in
mobile computing, and the most effective solutions in the literature are based
on supervised deep learning models. However, the actual deployment of these
systems is limited by the scarcity of labeled data that is required for
training. Neuro-Symbolic AI (NeSy) provides an interesting research direction
to mitigate this issue, by infusing common-sense knowledge about human
activities and the contexts in which they can be performed into HAR deep
learning classifiers. Existing NeSy methods for context-aware HAR rely on
knowledge encoded in logic-based models (e.g., ontologies) whose design,
implementation, and maintenance to capture new activities and contexts require
significant human engineering efforts, technical knowledge, and domain
expertise. Recent works show that pre-trained Large Language Models (LLMs)
effectively encode common-sense knowledge about human activities. In this
work, we propose ContextGPT: a novel prompt engineering approach to retrieve
from LLMs common-sense knowledge about the relationship between human
activities and the context in which they are performed. Unlike ontologies,
ContextGPT requires limited human effort and expertise. An extensive
evaluation carried out on two public datasets shows how a NeSy model obtained
by infusing common-sense knowledge from ContextGPT is effective in data
scarcity scenarios, leading to similar (and sometimes better) recognition
rates than logic-based approaches with a fraction of the effort.

human activity recognition, context-awareness, large language models, neuro-
symbolic, knowledge infusion

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Machine
learning††ccs: Human-centered computing Ubiquitous and mobile devices††ccs:
Information systems Information extraction

##  1\. Introduction

The analysis of sensor data gathered from mobile and wearable devices for
Human Activity Recognition (HAR) has been extensively researched (Chen et al.,
2021; Gu et al., 2021). This is attributed to its high potential for
applications across various domains such as well-being, healthcare, and
sports, fueled by the widespread adoption of these devices.

Although the majority of the studies in this area focused on inertial sensors
only, context-aware HAR approaches also take into account the user’s
surrounding context as derived from the devices, user profiles or online
services (e.g., semantic location, temporal information, etc.) to further
improve the recognition rate and, at the same time, extend the set of
recognizable activities (Bettini et al., 2020).

In the last few years, a vast amount of solutions based on deep learning
classifiers have been proposed (Wang et al., 2019). However, most of those
approaches rely on supervised learning and require large labeled datasets to
be trained. The need to acquire reliable labels from a large number of users
and for an extended set of activities is currently a major obstacle to the
deployment of effective sensor-based HAR systems.

Among the several solutions proposed in the general machine learning community
to tackle the labeled data scarcity problem, Neuro-Symbolic (NeSy) approaches
represent a promising research direction (Sarker et al., 2021). NeSy methods
aim at combining data-driven and knowledge-based approaches to reduce the
amount of labeled data needed to effectively train the model and, at the same
time, to improve its interpretability. In NeSy approaches, well-known domain
constraints are infused into the model, thus avoiding learning them from data
and simplifying the overall learning process.

Existing NeSy approaches proposed for context-aware HAR retrieve common-sense
knowledge from logic-based models (e.g., ontologies) manually designed and
implemented by human experts (Arrotta et al., 2022). Such knowledge models
encode the relationship between an activity and the context in which it can be
performed. For instance, they encode that biking is not a plausible activity
if the user’s semantic location is highway or museum. However, manually
building a comprehensive knowledge model that captures all possible context
situations is challenging, and, even relying on a domain expert, it does not
guarantee that all the possible situations in which an activity can be
performed are captured. Moreover, logic-based models are not scalable since
including new activities, new context conditions, or new constraints requires
further manual work and expertise.

This paper explores the idea of infusing common-sense knowledge into NeSy HAR
models from Large Language Models (LLMs) instead of ontologies. Indeed, LLMs
implicitly encode knowledge spanning a wide range of domains and recent
results suggest that they can be efficiently exploited to retrieve common-
sense knowledge about human activities (Takeda et al., 2023; Graule and Isler,
2023; Zhou et al., 2023; Kaneko and Inoue, 2023; Xia et al., 2023; Leng et
al., 2023).

In this paper, we propose ContextGPT, a novel prompt engineering approach
that, based on the high-level user’s context, leverages a pre-trained LLM
model to determine the most plausible activities. Specifically, given a time
window of sensor data, ContextGPT converts high-level context information into
a natural language description. Thanks to a carefully designed system message,
ContextGPT generates a prompt by asking the LLM to determine the activities
that are consistent with the current context. The prompts generated by
ContextGPT also include a few examples (created by the prompt engineer)
depicting how the task should be carried out (i.e., few-shot prompting).

Our contributions are the following:

  * •

We propose ContextGPT: a novel prompt engineering approach to retrieve common-
sense knowledge about the relationships between activities and high-level
contexts from pre-trained Large Language Models.

  * •

We illustrate how this knowledge can be infused into a Neuro-Symbolic Context-
Aware model to mitigate labeled data scarcity.

  * •

Our extensive experimental evaluation using two public datasets shows that
infusing ContextGPT knowledge leads to recognition rates similar to methods
based on logic-based models, while significantly reducing the cost in terms of
expert human effort.

##  2\. Related Work

###  2.1. Data scarcity in HAR

Most of the solutions proposed in the literature for sensor-based HAR on
mobile/wearable devices rely on supervised Deep Learning (DL) methods (Wang et
al., 2019; Chen et al., 2021). Even though the majority of these works only
focused on inertial sensors, several studies highlight how also including
high-level context information can significantly improve the recognition
rate(Bettini et al., 2020; Saguna et al., 2013).

Due to the unrealistic amount of training data required to train supervised
models, several research groups are proposing solutions to leverage small
amounts of labeled samples. Proposed methods to mitigate labeled data scarcity
are based on transfer learning (Sanabria et al., 2021; Soleimani and
Nazerfard, 2021), self-supervised learning (Haresamudram et al., 2022; Jain et
al., 2022; Hiremath et al., 2022), and semi-supervised learning approaches
(Abdallah et al., 2018). We believe that Neuro-Symbolic AI (NeSy) could be
coupled with such approaches to further mitigate data scarcity when fine-
tuning pre-trained HAR models with limited labeled data.

###  2.2. Neuro-Symbolic HAR

While several NeSy methods have been proposed in the computer vision and NLP
domains (Dash et al., 2022), only a few approaches have been proposed for HAR.

Existing NeSy methods for context-aware HAR retrieve common-sense knowledge
from logic-based models (e.g., ontologies). To the best of our knowledge,
three main strategies have been proposed so far to combine extracted knowledge
with deep learning models: a) using knowledge to refine the deep model’s
output (Bettini et al., 2020), b) including retrieved knowledge as additional
features in the latent space (Arrotta et al., 2020), and c) using a loss
function that penalizes predictions violating domain constraints (Arrotta et
al., 2023a; Xing et al., 2020). However, designing and implementing knowledge
models require significant human effort, and those models may not capture all
the possible situations in which activities can be performed. While there are
information retrieval approaches to semi-automatically obtaining common-sense
knowledge from public external sources (e.g., images (Riboni and Murtas,
2019), web (Perkowitz et al., 2004), text (Yordanova, 2016)) such methods
still faces challenges in creating comprehensive knowledge models.

###  2.3. Using LLMs for Human Activity Recognition

The adoption of Large Language Models (LLMs) in sensor-based HAR is a recently
emerging trend, and we expect a surge of contributions in this area in the
near future. For instance, the work in (Takeda et al., 2023) takes advantage
of the GPT-2 model to predict sequences of sensor events in smart-home
settings. Specifically, since the GPT-2 model is pre-trained to predict the
next word in a sentence, it has been fine-tuned to predict the most likely
sequence of sensor events that are treated similarly to textual data. A
similar approach was also proposed in (Graule and Isler, 2023), leveraging the
LLaMA2 language model. The work in (Zhou et al., 2023) uses the CLIP pre-
trained model to align sensor data embeddings and text embeddings by means of
contrastive learning, with the goal of improving the overall recognition rate.

A few research groups recently proposed solutions based on the well-known
ChatGPT tool. In (Kaneko and Inoue, 2023), ChatGPT is prompted with the
description of a specific sensor-based HAR task and the current choice for
sensor positioning, with the objective of obtaining feedback about where to
install new sensors to further improve the recognition rate. The work in (Xia
et al., 2023) proposed an unsupervised approach for smart-home settings. The
approach consists of two steps: the former asks ChatGPT to provide a
description of the target activities, and the latter asks ChatGPT to infer the
performed activity given a temporal sequence of events describing interaction
with household items. Finally, based on the recent advances in generating
synthetic inertial sensor data from textual descriptions (Guo et al., 2022),
the work in (Leng et al., 2023) leverages ChatGPT to generate textual
descriptions of activities, that are then used to generate virtual inertial
sensors data.

To the best of our knowledge, we are the first to leverage LLMs to mitigate
data scarcity in context-aware HAR based on mobile/wearable devices. In
particular, we query LLMs to determine which activities are compatible with
the current user’s context, with the goal of infusing this knowledge into a
neuro-symbolic model.

##  3\. Methodology

###  3.1. A Neuro-Symbolic Framework for Context-Aware HAR

We consider a mobile and wearable computing scenario in which users perform
activities while carrying their devices (e.g., smartphone, smartwatch,
tracker).

Figure 1 provides an overview of the Neuro-Symbolic (NeSy) system taking as
input the continuous stream of sensor data generated by the user devices, and
providing as output the most likely performed activity.

Figure 1. A Neuro-Symbolic AI framework for context-aware HAR gathering
knowledge from ContextGPT

As a common practice in this field (Chen et al., 2021), data are partitioned
into temporal windows. For each window $w$ (representing $z$ seconds of
consecutive raw sensor data), we derive two subsets $w^{R}$ and $w^{C}$: the
former includes raw data that we consider appropriate for being directly
processed by a data-driven model (e.g., inertial sensors data), while the
latter involves raw sensor data that we consider useful for deriving high-
level contexts through reasoning and/or abstraction.

A high-level context provides information, at a certain point in time, about
the user, the environment surrounding them, and their interactions with
objects or other subjects. Let $C=\langle C_{1},\dots,C_{n}\rangle$ be a set
of possible contexts that are meaningful for the application domain (e.g.,
$C_{1}$ = it is raining, $C_{2}$ = location is a park, $C_{3}$ = current
user’s speed is high).

Note that $w^{R}$ and $w^{C}$ may have a non-empty intersection and
$w^{R}\bigcup w^{C}=w$. For instance, it may be appropriate to exclude raw GPS
data from $w^{R}$ since it may be difficult to find robust correlations
capable of generalizing between different users (especially in data scarcity
settings). On the other hand, raw GPS data can be included in $w^{C}$ to
obtain high-level contexts that are more easily correlated with activities
(e.g., semantic location: ”public park”).

The Context Aggregation module is in charge of deriving all the most likely
high-level contexts $C^{w}\subset C$ that occur in a window $w$ based on
$w^{C}$. Context Aggregation can be implemented using simple rules, available
services, and/or context-aware middlewares (Henricksen et al., 2005). For
example, raw GPS coordinates can be used to derive the semantic location by
querying a dedicated web service (e.g., by using Google Places APIs).

Context-aware HAR could be addressed by relying on machine learning models
taking $\langle w^{R},C^{w}\rangle$ only as input. However, a more effective
approach is to complement data-driven approaches with common-sense knowledge
about the relationships between activities and contexts (Riboni and Bettini,
2011). For example, based on common-sense knowledge, people typically run
outdoors in parks, on trails, or along sidewalks (preferably in dry weather)
and indoors on a treadmill. Moreover, running requires the user to move at a
positive speed. The relationships between activities and the typical
conditions in which they can be performed can be used in the HAR process to
reduce the amount of labeled data required to learn them.

The ContextGPT module (that is the main contribution of this work) is in
charge of reasoning on the contexts in $C^{w}$ to derive the most likely
activities that are consistent according to $C^{w}$. ContextGPT relies on a
Large Language Model (LLM) and it is described in detail in the rest of this
section. The information about context-consistent activities is infused into a
NeSy HAR model, which also receives as input raw data and high-level context
data (i.e. $\langle w^{R},C^{w}\rangle$). The output of the model is a
probability distribution $P=\langle p_{1},\dots,p_{k}\rangle$, where $p_{i}$
is the probability that the user performed the activity $a_{i}$.

###  3.2. ContextGPT architecture

Figure 2 illustrates the overall architecture of ContextGPT.

Figure 2. Overall architecture of ContextGPT

ContextGPT receives as input a temporal window of high-level context data
$C^{w}$. First, $C^{w}$ is provided to the Context2Text module to obtain a
natural language description of the user’s context. Since LLMs also benefit
from a few examples depicting how the required task should be carried out
(i.e., the so-called few-shot prompting (Brown et al., 2020)), the Example
selection module considers a pool of examples and includes in the prompt those
having their context similar to $C^{w}$. Each example depicts a context
situation and the activities that are consistent with that situation. Finally,
the Prompt Construction module generates the complete prompt that is composed
of:

  * •

System Message: general instructions to the LLM about the task that it has to
carry out.

  * •

Examples: the most similar examples to the input context, selected from a
pool.

  * •

Context Description: the description in natural language of $C^{w}$.

Figure 3 shows an example of a typical prompt generated by ContextGPT.

Figure 3. An example of ContextGPT prompt

The prompt is provided as input to a pre-trained LLM, and the output is post-
processed to obtain the list of activities that are consistent with the
current context.

###  3.3. Prompt Engineering

While existing Neuro-Symbolic methods for Context-Aware HAR demand knowledge
engineers to manually build logic-based models, ContextGPT requires a Prompt
Engineer in charge of designing: i) the system message, ii) the translation of
context data into natural language descriptions, and iii) the examples in the
pool. Due to the sensitivity of LLMs to the specific wording in the prompt,
the Prompt Engineer currently has a non-trivial role for the success (or
failure) in obtaining the desired goal (Meyer et al., 2023). However, since
these tasks are based on natural language, there is no need for designing
comprehensive and complex relationships between activities and contexts using
logic formalisms; hence, the required expertise and human effort is
significantly reduced. In the following, we describe each component of the
prompts of ContextGPT in detail.

####  3.3.1. System Message

The system message defines the general description of the task the LLM should
complete. In our case, the task is determining the activities that are
consistent with a given context. Hence, we divided the system message into two
parts. The former instructs the LLM about the overall task and the list of
possible activities. The latter provides a detailed list of steps the LLM
should undertake to complete the task (i.e., Chain-Of-Thought approach (Wei et
al., 2022)).

The first step directs the LLM to focus on the input context. The second step
requires following an open-world assumption since, in our preliminary
experiments, we noticed instances where the model mistakenly excluded
activities not explicitly supported by the context description. For instance,
consider the following context description: “In the last 4 seconds the user
Bob was in an outdoor environment, where he was moving/traveling at speed
between 1 and 4 km/h, experiencing rainy weather, not following/close to a
public transportation route, and not experiencing elevation changes.”. Without
the second step, in our preliminary experiments, the LLM often excluded the
activity Moving by Car with the following motivation: “Not consistent as there
is no mention of being in a car or any other vehicle besides walking speed
movement”. While it is true that being in a car is not explicitly provided in
the context description, the Moving by car activity should still be considered
as possible. Indeed, it is impractical to monitor all the possible contextual
conditions through sensors in mobile/wearable devices. Finally, the last step
forces the LLM to provide context-consistent activities in a specific format
to simplify the automatic extraction of this information during post-
processing.

Figure 4 shows the system message of ContextGPT.

Figure 4. The system message of ContextGPT. The possible activities, in this
case, are the ones of the DOMINO (Arrotta et al., 2023b) dataset.

####  3.3.2. Context2Text

In order to be ingested by the LLM, the Context2Text module transforms the
input context data $C^{w}$ into a natural language sentence describing it.
Each description starts with ‘‘In the last $z$ seconds, the user $u$” to
contextualize that the described context refers to a specific temporal window
of $z$ seconds and that it is associated with a specific user $u$. Then, the
sentence continues by describing in natural language each context variable.
Designing the specific mapping between context data and natural language
sentences is currently not trivial, since the prompt engineer has to
understand (through trial and error) how the LLM interprets specific words.
For instance, the context “the user is on a public transportation route” means
that the path that the user is following (e.g., as captured by a GPS trace) is
the one of a public transportation route. However, these words sometimes led
the model to mistakenly interpret it as “the user is on a public
transportation vehicle”, thus excluding activities like Moving by car and
Cycling. We observed that translating the same context information as “the
user is following/close to a public transportation route” significantly
reduced the instances where this issue occurs.

An example of application of Context2Text is depicted in Figure 5.

Figure 5. Context2Text applied to a specific context situation.

####  3.3.3. Example pool

As previously mentioned, LLMs benefit from including a few examples in the
prompt, showing the output that should be associated with specific inputs. In
our case, an example represents a context situation and the corresponding
consistent activities. We assume that the Prompt Engineer is sufficiently
knowledgeable in the activity domain and, given a context description, they
are capable of determining which activities are consistent. For each activity
$a$, the Prompt Engineer populates the pool of examples $P$ with the following
strategy:

  * •

Define a set of examples $E_{a}$ (i.e. a set of tuples ¡$context$,
$consistentActivities$¿) referring to context situations that are uncommon but
possible for the activity $a$ (the context may be common or uncommon for the
other consistent activities)

  * •

For each example $e\in E_{a}$:

    * –

Consider $e$ as an input context for the LLM (using the system message without
examples) and analyze the response about the set of consistent activities.

    * –

If the response of the LLM is different from the consistent activities in the
example, the Prompt Engineer decides whether to include $e$ in $P$ to fill the
knowledge gap.

Considering one activity at a time significantly simplifies the Prompt
Engineer’s task of generating examples. The number of examples created by the
prompt engineer for each activity is not fixed: it depends on the activity,
the available contexts, and their experience in the activity domain. An
example is added to the pool only if the Prompt Engineer feels that the LLM is
not providing a satisfactory outcome, revealing a gap in the LLM knowledge. We
consider “uncommon” context situations in which an activity is rarely
performed but it is still possible; these are the cases most likely not
covered by the LLM. For instance, consider the activity running. While this
activity is usually performed outdoors, it may be uncommon to think that it
may also performed at the gym (e.g., on a treadmill). In our preliminary
experiments, we observed that such uncommon context for running was often not
captured by the LLM, and including it as an example improved ContextGPT
outputs significantly.

####  3.3.4. Example Selection

Since the number of examples in the pool is not fixed, including all of them
in the prompt may result in increased costs, especially when using external
services whose price depends on the prompt’s length. Moreover, there are often
limits to the LLM’s input size. Hence, if the number of examples is high, they
may exceed the model’s capacity. ContextGPT employs a method to include in the
prompt only those examples that are similar to the input context $C^{w}$, and
its pseudo-code is outlined in Algorithm 1.

$S\leftarrow\emptyset$ ;

$C^{w}\leftarrow$ current user’s context;

$T^{C}\leftarrow Context2Text(C^{w})$ ;

$E^{C}\leftarrow$ compute embeddings of $T^{C}$ using a LLM ;

forall  _$e\in P$_ do

$T^{e}\leftarrow Context2Text(e.context)$;

$E^{e}\leftarrow$ compute embeddings of $T^{e}$ using a LLM ;

$s_{e}\leftarrow\frac{E^{C}\cdot E^{e}}{|E^{C}||E^{e}|}$ ;

if  _$s_{e} >k$_ then

$S\leftarrow S\bigcup\\{e\\}$;

end if

end forall

return $S$

Algorithm 1 Example Selection

Specifically, we use a pre-trained LLM to extract text embeddings from $C^{w}$
and all the examples in the pool using their description in natural language
obtained with Context2Text. Then, we use cosine similarity to compute the
similarity of each example with $C^{w}$, and we include in the prompt only the
examples with a similarity higher than a threshold $k$. This threshold is
determined empirically, and we will show its impact on the recognition rate in
Section 4.

###  3.4. Post-processing

Figure 6 shows an example of LLMs output in ContextGPT.

Figure 6. An example of LLM output

As required by the system message (Section 3.3.1), besides explaining the
reasoning process, the output includes the list of consistent activities in
square brackets. Hence, it is possible to easily extract this list $L$ using
regular expressions and transform it into a binary vector
$b=[b_{1},b_{2},\dots,b_{n}]$ where $b_{i}$ is $1$ if the activity $a_{i}\in
L$ (i.e., the activity $a_{i}$ is consistent with $C^{w}$ according to the
LLM), $0$ otherwise. In the following, we will refer to $b$ as the consistency
vector. Note that in the actual implementation of ContextGPT there is a cache
mechanism that avoids recomputing the consistency vector when $C^{w}$ has
already been processed. Consistency vectors are used to infuse knowledge
inside the NeSy model.

###  3.5. Infusing knowledge into Neuro-Symbolic model

While ContextGPT is agnostic with respect to the Neuro-Symbolic approach for
Context-Aware HAR, in this work we use a specific knowledge infusion approach
proposed in the general AI community (Sheth et al., 2019; Kursuncu et al.,
2019). This method relies on a hidden layer in the Deep Neural Network (DNN)
in charge of infusing knowledge in the latent space. An adaptation for
Context-Aware HAR named NIMBUS has been recently proposed in the literature,
exhibiting promising recognition rates (Arrotta et al., 2022). In NIMBUS,
knowledge is infused from a manually defined ontology of activity and
contexts. In this work, we adapted NIMBUS to fetch knowledge from ContextGPT.

The overall mechanism of NIMBUS is depicted in Figure 7. The consistency
vector obtained from ContextGPT is infused in the hidden layers of the Deep
Neural Network (DNN) through a dedicated layer named knowledge infusion layer.
This hidden layer makes it possible to learn correlations between the latent
representation of raw sensor data, high-level context data, and context-
consistent activities.

Figure 7. Infusing ContextGPT into the Symbolic Features approach

##  4\. Experimental Evaluation

###  4.1. Datasets

In the following, we describe the two publicly available datasets we used to
evaluate ContextGPT. Both datasets contain real data with the former obtained
in a realistic but scripted setting, and the latter collected in a in-the-wild
setting. They include two types of data collected from mobile devices: a) raw
inertial sensor data and b) pre-processed high-level context data.

####  4.1.1. DOMINO

The DOMINO (Arrotta et al., 2023b) dataset includes data from $25$ subjects
wearing a smartwatch on the wrist of their dominant hand and a smartphone in
their pants front pocket. Both devices gathered raw sensor data from inertial
sensors (accelerometer, gyroscope, and magnetometer) and a wide variety of
high-level context data. This dataset includes nearly $9$ hours of labeled
data (approximately $350$ activity instances) covering $14$ distinct
activities: walking, running, standing, lying, sitting, stairs up, stairs
down, elevator up, elevator down, cycling, moving by car, sitting on
transport, standing on transport, and brushing teeth. The DOMINO dataset was
collected in a scripted setting: the volunteers were instructed to perform
sequences of indoor/outdoor activities, even though without specific guidance
on their execution.

DOMINO included the following pre-processed high-level context information:

  * •

User’s height variations (discretized: negative, null, positive)

  * •

User’s speed variations (discretized: null, low, medium, and high)

  * •

Whether the user is indoors or outdoors

  * •

Semantic locations (Home, Office, University, Mall, Station, Museum, Gym,
Shop, Bar, Restaurant, Barbershop, Bank, Church)

  * •

Weather condition (Sunny, Rainy, Misty, Cloudy, Drizzly, Stormy)

  * •

Whether the user is following a public transportation route

Overall, the DOMINO dataset includes data collected in $412$ unique context
conditions.

####  4.1.2. ExtraSensory

ExtraSensory (Vaizman et al., 2018) was collected in an unscripted in-the-wild
setting from $60$ subjects. Similarly to DOMINO, users wore a smartwatch on
the wrist of their dominant hand and a smartphone in their pants front pocket.
ExtraSensory includes approximately $300$,$000$ minutes of labeled data,
including $51$ distinct labels self-reported by the users. These labels encode
both high-level context information (e.g., at home, with friends, phone in
bag, phone is charging) and performed activities (e.g., sitting, bicycling).
Considering inertial sensor data, each smartphone collected raw data from the
accelerometer, gyroscope, and magnetometer; on the other hand, the smartwatch
only collected raw data from the accelerometer.

Since it was collected in the wild, ExtraSensory is widely considered a
challenging benchmark. Existing works on this dataset report low recognition
rates although considering small subsets of activities (Arrotta et al., 2023a;
Tarafdar and Bose, 2021; Cruciani et al., 2020).

In this paper, we pre-process the dataset consistently with previous works in
context-aware HAR (Arrotta et al., 2023a). We consider the following $7$
activities: bicycling, lying down, moving by car, on transport, sitting,
standing, and walking. All the context information in the dataset that could
be easily collected by mobile devices is provided as input to the Neuro-
Symbolic model (e.g., audio level, screen brightness, ringer mode, etc.).

However, based on preliminary experiments, we provide to the LLM only the
context information where common-sense knowledge can be practically used to
reason about the target physical activities:

  * •

Whether the user is indoors or outdoors

  * •

Semantic location (Home, Workplace, School, Gym, Restaurant, Shopping, Bar,
Beach)

  * •

User’s speed (null, low, medium, and high)

  * •

User’s movements diameter (null, low, medium, and high)

  * •

Whether the user is following a public transportation route

Overall, the ExtraSensory data presents $144$ unique context conditions for
the LLM. This number is significantly lower compared to DOMINO, due to a
reduced number of context variables and target activities.

###  4.2. Experimental setup

We implemented a working prototype of ContextGPT as well as the Neuro-Symbolic
model explained in Section 3.5 using the Python language (version 3.12). We
run our experiments on a machine of our department running Ubuntu 20.04.4 LTS
and equipped with an AMD EPYC Processor x86-64, an NVIDIA A100 GPU (80 GB),
with $43.5$ GBs RAM allocated. In the following, we provide details about our
experimental setup.

####  4.2.1. Large Language Models

The pre-trained LLM we used in our experiments is gpt-3.5-turbo by OpenAI,
queried through its API through the Python OpenAI package (version 0.28.1). We
set the temperature of the model to $0$ to ensure a mostly deterministic
output since our task does not require leveraging the model’s creativity. The
pre-trained LLM model we adopted to compute example embeddings and thus
computing similarity (see Section 3.3.4) is all-
MiniLM-L6-v2111https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2.

####  4.2.2. Generating examples

A member of our research group, without experience with the specific datasets
used in this work, created the examples following the strategy described in
Section 3.3.4. To simplify the example creation task and to automatically
generate examples in natural language, we developed a simple examples creation
tool (see Figure 8).

Figure 8. A screenshot of our tool to create examples

Since each dataset has different sets of activities and context variables,
examples were created separately for DOMINO and ExtraSensory. The output of
this process is $21$ examples for DOMINO and $12$ for ExtraSensory.

####  4.2.3. Model and Hyper-parameters

For the NeSy model, we use the implementation of NIMBUS proposed in (Arrotta
et al., 2023a)222Note that in (Arrotta et al., 2023a) NIMBUS is named Symbolic
Features.. Smartphone inertial sensor data are provided to a sequence of three
convolutional layers, each consisting of $32$, $64$, and $96$ filters with
corresponding kernel sizes of $24$, $16$, and $8$. These layers are
interleaved with max-pooling layers with a pool size of $4$. Then, the flow
continues with a global max pooling layer and a fully connected layer with
$128$ neurons. In parallel on another channel, the smartwatch inertial sensor
data is processed by an identical sequence of layers with a small difference:
the kernel size of the first three convolutional layers is $16$, $8$, and $4$,
respectively. In parallel, high-level context data (represented with one-hot
encoding) is provided to a single fully connected layer with $8$ neurons. The
features extracted from these three parallel streams are merged with the
consistency vector from ContextGPT using a concatenation layer (i.e., the
knowledge-infusion layer presented in Section 3.5). Then, a dropout layer
(with a rate of $0.1$) is introduced for regularization, followed by a fully
connected layer of $256$ neurons to extract correlations among the features
from the different data sources and the consistency vector. Finally, a softmax
layer returns the probability distribution over the possible activities.

We used the same hyper-parameters proposed in (Arrotta et al., 2023a):

  * •

Segmentation window: $4$ seconds

  * •

Optimizer: Adam

  * •

Loss: Categorical cross-entropy

  * •

Learning rate: $0.001$

  * •

Batch size: $32$

  * •

Epochs: $200$

  * •

Early stopping patience on the validation loss: $5$

###  4.3. Evaluation methodology

####  4.3.1. Baselines

In this paper, we compare the infusion of ContextGPT knowledge into a Neuro-
Symbolic HAR model with two alternatives:

  * •

No knowledge: a purely data-driven baseline without knowledge infusion, where
high-level context data is only used as input of the deep learning model.

  * •

Ontology: the NIMBUS approach originally proposed in (Arrotta et al., 2022),
where knowledge is infused from an ontology encoding relationships between
activities and contexts. Hence, we use the ontology adopted in that work for
the DOMINO dataset and its adaptation to the ExtraSensory dataset recently
proposed in (Arrotta et al., 2023a).

####  4.3.2. Evaluation Strategy

We evaluated the recognition rate of the HAR models by using a
leave-$n$-users-out cross-validation technique. At each fold, $n$ users were
designated for the test set, while the remaining users were split between the
training ($90\%$) and validation ($10\%$) sets. Specifically, for the DOMINO
dataset we set $n=1$, while for the Extrasensory dataset, consistently with
other works in the literature (Cruciani et al., 2020), we set $n=5$.

We simulated several data scarcity scenarios by downsampling the available
training data at each fold. At each iteration, we used the test set to
evaluate the recognition rate in terms of macro F1 score. To ensure robust
results, we conducted each experiment $5$ times, calculating the average F1
score.

###  4.4. Results

####  4.4.1. Comparison with the baselines

Figures 9 and 10 show, for both datasets, the impact of infusing ContextGPT
knowledge (using the $k$ value leading to the best results) compared to the
baselines.

Figure 9. DOMINO: Knowledge infusion with ContextGPT compared to the
baselines. This plot shows the F1 score in different data scarcity scenarios.
Figure 10. ExtraSensory: ContextGPT compared to the baselines. This plot shows
the F1 score in different data scarcity scenarios.

As expected, infusing knowledge from ContextGPT significantly outperforms the
purely data-driven No knowledge baseline in data scarcity scenarios. At the
same time, ContextGPT reaches competitive results to the Ontology baseline,
with the great advantage of significantly reduced human effort. We also
observe that, when the training set includes a sufficient number of labeled
samples, the contribution of knowledge infusion is significantly reduced. In
these cases, it is likely that most domain constraints are learned from the
training data itself.

Consistently with existing works (Cruciani et al., 2020; Tarafdar and Bose,
2021; Arrotta et al., 2023a), the in-the-wild nature of ExtraSensory leads to
relatively low recognition rates. Moreover, on this dataset, increasing the
percentage of training data slightly degrades the recognition rate of both
knowledge infusion approaches. The reason may be due to the underlying NeSy
model, in which raw sensor data may overshadow the infused knowledge when the
training set is large. Since DOMINO is significantly smaller, we do not
observe the same phenomenon on this dataset.

For both datasets, knowledge infusion is particularly effective for those
activities that significantly depend on the context (e.g., using the elevator,
moving by car, moving on public transportation), while its benefits are
reduced for those activities where context influence is limited (e.g.,
sitting, walking).

####  4.4.2. Impact of examples selection

Figures 11 and 12 illustrate the distribution of the number of selected
examples by varying $k$ for DOMINO and ExtraSensory, respectively.

Figure 11. DOMINO: Distribution of the number of selected examples by varying
$k$. Figure 12. ExtraSensory: Distribution of the number of selected examples
by varying $k$.

We observe that in both cases $k=0.75$ is the lowest threshold value
significantly restricting the number of selected examples, while lower values
lead to a higher number of examples.

In Figure 9, the best results on DOMINO were achieved with low values of $k$
(i.e., ranging from $0$ to $0.25$) and thus a high number of examples. This is
due to the fact that, in this dataset, there is a high number of possible
context conditions. Thus, the LLM benefits from more examples describing the
required task.

On the other hand, in Figure 10, the best results on the ExtraSensory dataset
are associated with higher values of $k$ (i.e., ranging from $0.25$ to $0.95$)
and thus selecting a smaller number of examples. In this case, since the
number of possible context conditions is significantly lower compared to
DOMINO, a few examples (similar to the input context) provide the LLM model a
better guidance on the required task.

In the following, we show the impact of example selection on the recognition
rate. Figure 13 and 14 show how $k$ impacts the recognition rate on each
dataset when only the $10\%$ of the training data is available. We observe
that the DOMINO dataset in this case benefits from $k=0$ (i.e., using all the
examples), while higher values of $k$ lead to worse recognition rates.
Nonetheless, it is interesting to note that even $k=1$ (i.e., no examples)
leads to outperforming the No knowledge baseline by $~{}\approx+4\%$.

On the other hand, the best recognition rates reached on the ExtraSensory
dataset in the same data scarcity scenario are associated with $k=0.5$ and
$k=0.75$. Notably, $k=0.75$ leads to a significantly smaller number of
examples than $k=0.5$ and to a similar recognition rate.

Our results on these datasets suggest that the higher the number of context
variables and activities, the higher the number of examples needed in the
prompt to maximize the recognition rate. We believe that this reliance on
examples may be mitigated by future more comprehensive LLMs and with fine-
tuning.

Figure 13. DOMINO: Impact of k on macro F1 with 10% of the training set Figure
14. ExtraSensory: Impact of k on macro F1 with 10% of the training set

While examples play a major role in improving the recognition rate for data
scarcity scenarios, they do not lead to significant improvement when
sufficient amounts of training data are available. This phenomenon is depicted
in Figures 15 and 16, where we show the impact of $k$ when the $50\%$ of
training data is available.

Figure 15. DOMINO: Impact of k on macro F1 with 50% of the training set Figure
16. ExtraSensory: Impact of k on macro F1 with 50% of the training set

When enough training data is available, the knowledge gaps in the LLM models
addressed by examples are probably learned from training data. Nonetheless,
infusing knowledge from ContextGPT still outperforms the No knowledge
baseline.

##  5\. LLM vs. Ontology for Neuro-Symbolic HAR

Our results show that by using pre-trained LLMs instead of ontologies in NeSy
HAR systems, we can reach similar (and sometimes better) recognition results.

Then we may wonder why LLMs are a more promising approach for future real
deployment of these systems, since we already have some ontologies like the
ones used in our work, while for LLMs some prompt engineering work is
required.

Considering the extension of these systems to a large class of human
activities and different context situations, there are clear limitations for
ontologies. To the best of our knowledge, we are not aware of publicly
available ontologies offering a comprehensive coverage of all possible human
activities and context situations. Hence, significant human effort would be
required to extend and adapt the ontology to new datasets with different sets
of activities and context data.

Indeed, extending an ontology means defining complex knowledge relationships
between activities and contexts, it requires skills in the logic-based
formalism model (e.g., OWL2 in the case of ontologies) and significant
expertise in the HAR domain. This task is also usually assigned to a single
knowledge engineer or small team with high risks of incompleteness.

In our case, adapting ContextGPT to a new dataset only requires using natural
language to adjust the system message on the target activities, extending the
Context2Text module to map new contexts to natural language descriptions, and
generating a new pool of examples.

However, a significant disadvantage of LLMs compared to ontologies is the
absence of real semantic reasoning, since the output is based on data-driven
text generation. Hence, there may be contradictions and/or model
hallucinations that we would not experience by using rigorous logic systems.
For instance, in one instance where the user was moving slowly, the model
considered as possible standing with the following motivation:“Possible, as
the user is moving at a relatively slow pace”. While hallucinations may be
mitigated by more advanced LLMs models (e.g., GPT-4) we believe that knowledge
infusion needs to cope with possibly noisy information.

Finally, we investigate the discrepancies between the sets of consistent
activities obtained by ContextGPT with the ones generated by the ontology.
Specifically, given a context condition, we compute the set $L$ of activities
considered consistent by the LLM in ContextGPT and $O$ as the set of
activities considered consistent by the ontology. Hence, we define the
following metrics:

  * •

$L2O$ Inclusion: the proportion of activities considered as consistent by
ContextGPT that are also consistent for the ontology, computed as
$\frac{|L\bigcap O|}{|L|}$.

  * •

$O2L$ Inclusion: the proportion of activities considered as consistent by the
ontology that are also consistent for ContextGPT, computed as $\frac{|L\bigcap
O|}{|O|}$.

Note that the low values of $L2O$ Inclusion imply that ContextGPT includes
several activities that are not considered consistent by the ontology.
Conversely, low values of $O2L$ Inclusion imply that ContextGPT does not
consider as consistent several activities that are deemed consistent by the
ontology.

Figure 17. DOMINO: Average $L2O$ and $O2L$ inclusion metrics varying $k$.
Figure 18. ExtraSensory: Average $L2O$ and $O2L$ inclusion metrics varying
$k$.

Figure 17 shows, for the DOMINO dataset, the average scores of $L2O$ and $O2L$
inclusion metrics at different values of $k$. By increasing $k$, we observe
that the $O2L$ metric is the one associated with the more drastic decrease,
thus denoting that considering only a small number of examples leads
ContextGPT to be restrictive on the consistent activities compared to the
ontology. Moreover, high values of $k$ are also associated with lower values
of $L2O$. This decrease in both metrics indicates a significant discrepancy
between the activities considered consistent by ContextGPT and those
considered consistent by the ontology. We hypothesize this may be correlated
with the results depicted in Figure 13, where a substantial number of examples
was required by ContextGPT to achieve results comparable to those obtained by
Ontology on the DOMINO dataset.

Figure 18 shows the impact of the inclusion metrics on ExtraSensory. In this
case, increasing $k$ significantly impacts only the $O2L$ Inclusion metric,
with ContextGPT considering a restricted set of consistent activities compared
to the ontology. On the other hand, the $L2O$ Inclusion metric exhibits only a
slight decrease since the activities consistent for ContextGPT are often
consistent also for the ontology. We hypothesize that such high values of the
$L2O$ metric are because, on this dataset, reducing the number of examples
does not lead to significant LLM’s hallucinations. This seems to be consistent
with the results previously depicted in Figure 14.

##  6\. Conclusions and Future Work

In this paper, we introduced ContextGPT: a novel method based on Large
Language Models (LLMs) to retrieve common-sense knowledge about human
activities to be infused in Neuro-Symbolic (NeSy) context-aware HAR models. We
showed the effectiveness of ContextGPT in data scarcity scenarios with an
extensive experimental evaluation using a state-of-the-art NeSy model. Our
results show that LLMs may effectively replace logic-based models in NeSy
systems to reduce human effort.

We have several plans for future work. First, in this work we considered a
general LLM incorporating knowledge from many domains. We want to investigate
how to specialize the LLM in the HAR domain. This is challenging since it
requires identifying reliable sources to effectively fine-tune the model.

Besides fine-tuning, we will investigate personalization aspects. Different
individuals may have customized habits requiring ad-hoc models (e.g., “Bob
usually enjoys running at the park some late afternoons after work, even if
it’s raining”). Thus, we will investigate how to introduce into the prompt
personalized aspects that may help in capturing personalized relationships
between activities and contexts.

Finally, ContextGPT currently generates a list of activities consistent with
the current context without considering fuzziness. However, associating a
consistency score with each activity may lead to more effective knowledge
infusion (Arrotta et al., 2022). The main challenge in this direction is that
LLMs are generally not good at dealing with numbers and mathematical
operations (Frieder et al., 2023).

Finally, we will also investigate alternative LLM models (e.g., GPT-4, LLAMA2)
and NeSy approaches (e.g., the Semantic Loss method recently proposed in
(Arrotta et al., 2023a)) and their impact on the recognition rate.

## Acknowledgments

The authors want to thank Mohcen Laaroussi for his excellent work on software
implementation. This work was partially supported by Italian Ministry of
University and Research with project “MUSA-Multilayered Urban Sustainability
Action” (project ID: ECS_00000037).

## References

  * (1)
  * Abdallah et al. (2018) Zahraa S Abdallah, Mohamed Medhat Gaber, Bala Srinivasan, and Shonali Krishnaswamy. 2018.  Activity recognition with evolving data streams: A review.  _ACM Computing Surveys (CSUR)_ 51, 4 (2018), 1–36.
  * Arrotta et al. (2020) Luca Arrotta, Claudio Bettini, Gabriele Civitarese, and Riccardo Presotto. 2020.  Context-aware data association for multi-inhabitant sensor-based activity recognition. In _2020 21st IEEE International Conference on Mobile Data Management (MDM)_. IEEE, 125–130.
  * Arrotta et al. (2022) Luca Arrotta, Gabriele Civitarese, and Claudio Bettini. 2022.  Knowledge Infusion for Context-Aware Sensor-Based Human Activity Recognition. In _2022 IEEE International Conference on Smart Computing (SMARTCOMP)_. IEEE, 1–8.
  * Arrotta et al. (2023a) Luca Arrotta, Gabriele Civitarese, and Claudio Bettini. 2023a.  Neuro-Symbolic Approaches for Context-Aware Human Activity Recognition.  _arXiv preprint arXiv:2306.05058_ (2023).
  * Arrotta et al. (2023b) Luca Arrotta, Gabriele Civitarese, Riccardo Presotto, and Claudio Bettini. 2023b.  DOMINO: A Dataset for Context-Aware Human Activity Recognition using Mobile Devices. In _2023 24th IEEE International Conference on Mobile Data Management (MDM)_. IEEE, 346–351.
  * Bettini et al. (2020) Claudio Bettini, Gabriele Civitarese, and Riccardo Presotto. 2020.  Caviar: Context-driven active and incremental activity recognition.  _Knowledge-Based Systems_ 196 (2020), 105816.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chen et al. (2021) Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu. 2021.  Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities.  _ACM Computing Surveys (CSUR)_ 54, 4 (2021), 1–40.
  * Cruciani et al. (2020) Federico Cruciani, Anastasios Vafeiadis, Chris Nugent, Ian Cleland, Paul McCullagh, Konstantinos Votis, Dimitrios Giakoumis, Dimitrios Tzovaras, Liming Chen, and Raouf Hamzaoui. 2020.  Feature learning for human activity recognition using convolutional neural networks: A case study for inertial measurement unit and audio data.  _CCF Transactions on Pervasive Computing and Interaction_ 2, 1 (2020), 18–32.
  * Dash et al. (2022) Tirtharaj Dash, Sharad Chitlangia, Aditya Ahuja, and Ashwin Srinivasan. 2022.  A review of some techniques for inclusion of domain-knowledge into deep neural networks.  _Scientific Reports_ 12, 1 (2022), 1040.
  * Frieder et al. (2023) Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023.  Mathematical capabilities of chatgpt.  _arXiv preprint arXiv:2301.13867_ (2023).
  * Graule and Isler (2023) Moritz A Graule and Volkan Isler. 2023.  GG-LLM: Geometrically Grounding Large Language Models for Zero-shot Human Activity Forecasting in Human-Aware Task Planning.  _arXiv preprint arXiv:2310.20034_ (2023).
  * Gu et al. (2021) Fuqiang Gu, Mu-Huan Chung, Mark Chignell, Shahrokh Valaee, Baoding Zhou, and Xue Liu. 2021.  A survey on deep learning for human activity recognition.  _ACM Computing Surveys (CSUR)_ 54, 8 (2021), 1–34.
  * Guo et al. (2022) Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. 2022.  Generating diverse and natural 3d human motions from text. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5152–5161.
  * Haresamudram et al. (2022) Harish Haresamudram, Irfan Essa, and Thomas Plötz. 2022.  Assessing the state of self-supervised human activity recognition using wearables.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 6, 3 (2022), 1–47.
  * Henricksen et al. (2005) Karen Henricksen, Jadwiga Indulska, Ted McFadden, and Sasitharan Balasubramaniam. 2005.  Middleware for distributed context-aware systems. In _OTM Confederated International Conferences” On the Move to Meaningful Internet Systems”_. Springer, 846–863.
  * Hiremath et al. (2022) Shruthi K Hiremath, Yasutaka Nishimura, Sonia Chernova, and Thomas Plötz. 2022.  Bootstrapping Human Activity Recognition Systems for Smart Homes from Scratch.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 6, 3 (2022), 1–27.
  * Jain et al. (2022) Yash Jain, Chi Ian Tang, Chulhong Min, Fahim Kawsar, and Akhil Mathur. 2022.  ColloSSL: Collaborative Self-Supervised Learning for Human Activity Recognition.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 6, 1 (2022), 1–28.
  * Kaneko and Inoue (2023) Haru Kaneko and Sozo Inoue. 2023.  Toward Pioneering Sensors and Features Using Large Language Models in Human Activity Recognition.  _arXiv preprint arXiv:2306.16017_ (2023).
  * Kursuncu et al. (2019) Ugur Kursuncu, Manas Gaur, and Amit Sheth. 2019.  Knowledge infused learning (k-il): Towards deep incorporation of knowledge in deep learning.  _arXiv preprint arXiv:1912.00512_ (2019).
  * Leng et al. (2023) Zikang Leng, Hyeokhyen Kwon, and Thomas Plötz. 2023.  Generating Virtual On-body Accelerometer Data from Virtual Textual Descriptions for Human Activity Recognition.  _arXiv preprint arXiv:2305.03187_ (2023).
  * Meyer et al. (2023) Jesse G Meyer, Ryan J Urbanowicz, Patrick CN Martin, Karen O’Connor, Ruowang Li, Pei-Chen Peng, Tiffani J Bright, Nicholas Tatonetti, Kyoung Jae Won, Graciela Gonzalez-Hernandez, et al. 2023\.  ChatGPT and large language models in academia: opportunities and challenges.  _BioData Mining_ 16, 1 (2023), 20.
  * Perkowitz et al. (2004) Mike Perkowitz, Matthai Philipose, Kenneth Fishkin, and Donald J Patterson. 2004.  Mining models of human activities from the web. In _Proceedings of the 13th international conference on World Wide Web_. 573–582.
  * Riboni and Bettini (2011) Daniele Riboni and Claudio Bettini. 2011.  COSAR: hybrid reasoning for context-aware activity recognition.  _Personal and Ubiquitous Computing_ 15, 3 (2011), 271–289.
  * Riboni and Murtas (2019) Daniele Riboni and Marta Murtas. 2019.  Sensor-based activity recognition: One picture is worth a thousand words.  _Future Generation Computer Systems_ 101 (2019), 709–722.
  * Saguna et al. (2013) Saguna Saguna, Arkady Zaslavsky, and Dipanjan Chakraborty. 2013.  Complex activity recognition using context-driven activity theory and activity signatures.  _ACM Transactions on Computer-Human Interaction (TOCHI)_ 20, 6 (2013), 1–34.
  * Sanabria et al. (2021) Andrea Rosales Sanabria, Franco Zambonelli, and Juan Ye. 2021.  Unsupervised Domain Adaptation in Activity Recognition: A GAN-Based Approach.  _IEEE Access_ 9 (2021), 19421–19438.
  * Sarker et al. (2021) Md Kamruzzaman Sarker, Lu Zhou, Aaron Eberhart, and Pascal Hitzler. 2021.  Neuro-symbolic artificial intelligence: Current trends.  _arXiv preprint arXiv:2105.05330_ (2021).
  * Sheth et al. (2019) Amit Sheth, Manas Gaur, Ugur Kursuncu, and Ruwan Wickramarachchi. 2019.  Shades of knowledge-infused learning for enhancing deep learning.  _IEEE Internet Computing_ 23, 6 (2019), 54–63.
  * Soleimani and Nazerfard (2021) Elnaz Soleimani and Ehsan Nazerfard. 2021.  Cross-subject transfer learning in human activity recognition systems using generative adversarial networks.  _Neurocomputing_ 426 (2021), 26–34.
  * Takeda et al. (2023) Naoto Takeda, Roberto Legaspi, Yasutaka Nishimura, Kazushi Ikeda, Atsunori Minamikawa, Thomas Plötz, and Sonia Chernova. 2023.  Sensor Event Sequence Prediction for Proactive Smart Home Support Using Autoregressive Language Model. In _2023 19th International Conference on Intelligent Environments (IE)_. IEEE, 1–8.
  * Tarafdar and Bose (2021) Pratik Tarafdar and Indranil Bose. 2021.  Recognition of human activities for wellness management using a smartphone and a smartwatch: a boosting approach.  _Decision Support Systems_ 140 (2021), 113426.
  * Vaizman et al. (2018) Yonatan Vaizman, Katherine Ellis, Gert Lanckriet, and Nadir Weibel. 2018.  Extrasensory app: Data collection in-the-wild with rich user interface to self-report behavior. In _Proceedings of the 2018 CHI conference on human factors in computing systems_. 1–12.
  * Wang et al. (2019) Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu. 2019.  Deep learning for sensor-based activity recognition: A survey.  _Pattern Recognition Letters_ 119 (2019), 3–11.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022\.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ 35 (2022), 24824–24837.
  * Xia et al. (2023) Qingxin Xia, Takuya Maekawa, and Takahiro Hara. 2023.  Unsupervised Human Activity Recognition through Two-stage Prompting with ChatGPT.  _arXiv preprint arXiv:2306.02140_ (2023).
  * Xing et al. (2020) Tianwei Xing, Luis Garcia, Marc Roig Vilamala, Federico Cerutti, Lance Kaplan, Alun Preece, and Mani Srivastava. 2020.  Neuroplex: learning to detect complex events in sensor networks through knowledge injection. In _Proceedings of the 18th conference on embedded networked sensor systems_. 489–502.
  * Yordanova (2016) Kristina Yordanova. 2016.  From textual instructions to sensor-based recognition of user behaviour. In _Companion Publication of the 21st International Conference on Intelligent User Interfaces_. 67–73.
  * Zhou et al. (2023) Yunjiao Zhou, Jianfei Yang, Han Zou, and Lihua Xie. 2023.  TENT: Connect Language Models with IoT Sensors for Zero-Shot Activity Recognition.  _arXiv preprint arXiv:2311.08245_ (2023).