# Large Language Models are Zero-Shot Recognizers for Activities of Daily
Living

Gabriele Civitarese<EMAIL_ADDRESS>,  Michele Fiori
<EMAIL_ADDRESS>,  Priyankar Choudhary<EMAIL_ADDRESS>and  Claudio Bettini<EMAIL_ADDRESS>EveryWare Lab, Dept. of
Computer Science, University of MilanItaly

(2018; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

The sensor-based recognition of Activities of Daily Living (ADLs) in smart
home environments enables several applications in the areas of energy
management, safety, well-being, and healthcare. ADLs recognition is typically
based on deep learning methods requiring large datasets to be trained.
Recently, several studies proved that Large Language Models (LLMs) effectively
capture common-sense knowledge about human activities. However, the
effectiveness of LLMs for ADLs recognition in smart home environments still
deserves to be investigated. In this work, we propose ADL-LLM, a novel LLM-
based ADLs recognition system. ADL-LLM transforms raw sensor data into textual
representations, that are processed by an LLM to perform zero-shot ADLs
recognition. Moreover, in the scenario where a small labeled dataset is
available, ADL-LLM can also be empowered with few-shot prompting. We evaluated
ADL-LLM on two public datasets, showing its effectiveness in this domain.

Human Activity Recognition, Large Language Models, Smart Home, Activities of
Daily Living

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††journal:
TIST††journalvolume: 37††journalnumber: 4††article: 111††publicationmonth:
8††ccs: Human-centered computing Ubiquitous and mobile computing††ccs: Applied
computing Health care information systems††ccs: Computing methodologies
Artificial intelligence

##  1\. Introduction

Sensor-based Human Activity Recognition (HAR) is a well-known research topic
that has been extensively studied by several research groups (Chen et al.,
2021). The HAR task involves determining the most likely activity performed by
a subject through the intelligent analysis of sensor data collected from one
or multiple devices (e.g., wearables, environmental, radio-based). The adopted
sensing technologies influence the granularity at which activities can be
recognized. For instance, by using inertial sensors in mobile/wearable
devices, it is possible to detect physical activities like running, walking,
and standing still. On the other hand, to recognize high-level Activities of
Daily Living (ADLs) like cooking, taking medicines, or watering plants in
smart-home environments, environmental sensors are particularly useful, also
considering that subjects at home are less inclined to carry or wear devices
all the time.

Among the many applications of sensor-based ADL recognition in the areas of
energy management, safety, well-being, and healthcare, the long-term
monitoring of the ADLs performed by elderly subjects in their homes gained
significant importance (Rashidi and Mihailidis, 2012). The continuous advances
in the medical domain significantly extended the world population’s life
expectancy (Chen et al., 2023). While this is surely an outstanding
achievement, aging also brings several challenges to society. Among the aging-
related issues, cognitive decline is one of the greatest health threats of old
age, with nearly $50\%$ of adults over the age of $85$ afflicted with the
Alzheimer’s disease (Bishop et al., 2010). Detecting the early symptoms of
cognitive decline is crucial to implement primary or secondary preventive
therapeutic strategies (Hudon et al., 2020). One of the main markers used by
clinicians to assess the cognitive status of a subject is represented by ADLs,
whose impairment (i.e., difficulty or inability to perform them independently)
is correlated to a rapid functional decline and a higher likelihood of
progression to dementia (Jefferson et al., 2008). However, the subjects at
risk usually have only a few sporadic visits with medical experts (e.g.,
neurologists, neuropsychologists, and geriatricians). Such visits often fail
to accurately diagnose cognitive decline, especially when it may be revealed
by subtle behavioral changes. Indeed, besides a few instrumental tests, and
neuro-psychological evaluations at the time of visit, the clinicians mainly
rely on possibly inaccurate reports provided by the subjects themselves or
their caregivers. It is hence of great value to implement smart telemedicine
strategies to continuously and unobtrusively monitor the subjects at risk
while keeping them in the comfort of their homes.

In the last decades, several research groups proposed the use of unobtrusive
sensing infrastructures in smart homes to continuously recognize ADLs in smart
homes, to identify digital markers that may indicate the onset of cognitive
decline (Riboni et al., 2016; Khodabandehloo et al., 2021; Grammatikopoulou et
al., 2024). For the sake of privacy, most of the existing solutions do not
consider cameras or microphones (that may not be accepted in home
environments), but they are usually based on environmental sensors monitoring
the interaction of the subject with the home infrastructure (e.g., motion
sensors, door sensors, pressure sensors).

However, the majority of the proposed approaches to automatically detect ADLs
from sensor data require training deep learning models (Chen et al., 2021),
thus necessitating the collection of large datasets in the home environment
(possibly labeled with the ground truth reporting what activity is being
performed). Labeled data collection in these settings is privacy-intrusive,
costly, and prohibitive (Babangida et al., 2022). Another well-known line of
research in this domain considers the adoption of purely knowledge-based
approaches (Chen and Nugent, 2009). The rationale behind these methods is that
common-sense knowledge about human activities (e.g., a person usually prepares
their meals in the kitchen, turns on the stove, and uses cooking instruments)
can be modeled with a set of rules. Knowledge-based ADLs recognition is
performed by mapping sensor data windows to semantic concepts and performing
rule-based reasoning. The advantage of this strategy is that it eliminates the
problem of collecting labeled data in smart environments. However, the design
and implementation of knowledge models require the expertise of domain experts
and knowledge engineers, and their capability of handling uncertainty and
generalizing is typically limited (Civitarese et al., 2019).

The recent success of Large Language Models (LLMs) paved the way to LLM-based
human activity recognition. Preliminary results in the state-of-the-art
indicate that LLMs implicitly encode the knowledge about human activities, and
hence they can potentially be used to cope with labeled data scarcity (Liu et
al., 2023; Ji et al., 2024). Given these promising results, we hypothesize
that LLMs could be also leveraged to process data from environmental sensors
in smart-home environments to reveal the ADLs performed by the monitored
subjects, without the need to collect any labeled data. We expect that LLMs
would generalize across individuals and environments without requiring any
patient- or environment-specific training data, thus offering a scalable
approach. Moreover, an LLM-based approach would be more flexible in terms of
the set of recognizable ADLs (that is usually fixed in supervised approaches).

In this work, we propose ADL-LLM, a method leveraging pre-trained LLMs to
recognize ADLs in smart-home environments. At its core, ADL-LLM transforms raw
sensor data into natural language sentences describing the high-level events
observed in the home. ADL-LLM uses a specifically designed prompting strategy
to query the LLM about the most likely ADL performed by the subject. Despite
ADL-LLM does not require training data, we show that a small labeled dataset
can further improve the recognition rate thanks to a few-shot prompting
mechanism.

The main contributions of this paper are the following:

  * •

We propose ADL-LLM, a novel LLM-based method for zero-shot sensor-based ADLs
recognition in smart home environments.

  * •

Our results on two public datasets show that zero-shot ADL-LLM reaches
recognition rates comparable (and sometimes better) than standard supervised
approaches.

  * •

We extend ADL-LLM to perform few-shot prompting when a small labeled dataset
is available, by leveraging a semantic-based example selection strategy. Our
experiments show that few-shot ADL-LLM outperforms standard supervised
approaches in labeled data scarcity scenarios.

We provide the source code implementing our system in order to make our
results reproducible111A link to this material will be provided upon
acceptance of this paper..

##  2\. Related Work

###  2.1. Data Scarcity in Human Activity Recognition

The labeled data scarcity problem in Human Activity Recognition (HAR) is one
of the major challenges limiting large-scale real deployments, and several
research groups have been working on this problem for several years. Three
categories of approaches have been proposed to mitigate labeled data scarcity:
data-driven, knowledge-based, and neuro-symbolic.

####  2.1.1. Data-Driven Methods

The majority of the existing approaches to mitigate labeled data scarcity rely
on deep learning models. The most common strategies are semi-supervised
learning (Abdallah et al., 2018), transfer learning (Dhekane and Ploetz,
2024), and self-supervised learning (Haresamudram et al., 2022). Semi-
supervised approaches use small labeled datasets to initialize the model,
which is incrementally updated using pseudo-labels inferred on the unlabeled
data stream (Abdallah et al., 2018). Examples of semi-supervised methods
proposed for HAR are self-learning, co-learning, active learning, and label
propagation. On the other hand, transfer learning methods leverage models that
are pre-trained on a source domain with a significant amount of labeled data,
and then adapted in a target domain with labeled data scarcity (Sanabria et
al., 2021; Soleimani and Nazerfard, 2021). Self-supervised learning is a
specific transfer learning approach, where the model is pre-trained on massive
unlabeled datasets to obtain a reliable feature extractor (Haresamudram et
al., 2022; Jain et al., 2022; Hiremath et al., 2022). The pre-trained feature
extractor is then fine-tuned only using a limited amount of labeled data on
the target domain. Existing approaches for self-supervised learning focuses on
data from wearable/mobile devices, since this domain offers offers large
datasets (e.g., Biobank (Yuan et al., 2024), Capture-24 (Chan Chang et al.,
2021)) that can be used for model pre-training. To the best of our knowledge,
we are not aware of self-supervised learning approaches for smart-home ADLs
recognition, due to the fact that in this domain there are only a few small
datasets, and learning a comprehensive representation that accommodates the
diverse layouts of different homes is still an open challenge.

ADL-LLM can be considered as a sort of zero-shot transfer learning approach,
where a model pre-trained on language is used to recognize activities from a
textual representation of sensor data.

####  2.1.2. Knowledge-Based Methods

Several research groups investigated whether common-sense knowledge about
human activities could be leveraged to completely avoid labeled data
collection. These approaches proposed the use of logic formalisms (e.g.,
ontologies) to model the relationships between sensor events and activities
(Civitarese et al., 2019; Chen and Nugent, 2009). However, building such logic
models requires skills in the formalism model (e.g., OWL2 in the case of
ontologies) and significant domain expertise. This task is usually performed
by small groups of persons. Building a comprehensive knowledge model capturing
all the possible ways of performing an activity is challenging even for
experts, and we are not aware of public ontologies comprehensively covering
all the possible human activities and context situations.

ADL-LLM has the same goal of knowledge-based approaches: using common-sense
knowledge about human activities to recognize them. However, instead of
relying on manually designed logic models, ADL-LLM leverages the knowledge
implicitly encoded in pre-trained LLM to guarantee better generalization
capabilities.

####  2.1.3. Neuro-Symbolic Methods

Neuro-Symbolic AI (NeSy) methods combine the best from data-driven and
knowledge-based worlds. Indeed, the goal of NeSy models is to combine symbolic
reasoning and machine learning to mitigate data scarcity and improve the
model’s interpretability (Hitzler and Sarker, 2022). Considering the HAR
domain, logic formalisms have been used to refine the output of the data-
driven model (Bettini et al., 2020) or to infuse knowledge inside deep
learning models (Xing et al., 2020; Arrotta et al., 2024b). In these methods
knowledge constraints allow the model to capture high-level patterns without
learning them from data, thus mitigating labeled data scarcity. However, NeSy
methods have been only proposed for wearable-based HAR and their applicability
to smart home ADLs recognition has not been explored yet.

For certain aspects, the capabilities of LLMs can be considered as Neuro-
Symbolic since they can implicitly mimic reasoning (Fang et al., 2024). Hence,
ADL-LLM could be considered as a NeSy model whose data-driven component
attempts to mimic symbolic reasoning.

###  2.2. Using LLMs for Human Activity Recognition

In the following, we present relevant research works using LLMs in the HAR
domain, distinguishing supervised and zero-shot techniques. Supervised methods
combines LLMs and labeled HAR datasets for specific tasks (e.g., prediction,
classification), while zero-shot techniques directly utilize pre-trained LLMs
to identify activities on unlabeled sensor data.

####  2.2.1. LLM-based approaches for supervised activity recognition

Several research groups studied how to use LLMs for the sensor-based HAR
domain. The most common approach is to fine-tune LLMs with labeled sensor
datasets, in order to specialize them on a specific domain (Liu et al., 2023;
Okita et al., 2023). Other works adopted multi-modal models to align sensor
data embeddings and text embeddings leveraging contrastive learning to improve
the recognition rate (Zhou et al., 2023). In Neuro-Symbolic HAR, some works
used LLMs to infuse common-sense knowledge about human activities into a deep
learning model (Arrotta et al., 2024a). LLMs has also been leveraged to
transform sensor data windows first into sentences in natural language, and
then into embeddings to improve the recognition rate and generalization
capabilities of supervised models (Thukral et al., 2024). Finally, since LLMs
are pre-trained to predict the next word in a sentence, some research groups
fine-tuned them also to predict future sensor events (Takeda et al., 2023;
Graule and Isler, 2023).

####  2.2.2. Zero-shot approaches for human activity recognition using LLMs

Similarly to ADL-LLM, other works aim at recognizing human activities using
LLMs without any training phase. For instance, some works show that LLMs can
reason on raw inertial sensor data from mobile/wearable devies to distinguish
between a limited set of low-level physical activities (Ji et al., 2024; Hota
et al., 2024). Another line of research leverages pre-trained model generating
3D motion sequences from text (Zhang et al., 2023) to generate synthetic
labeled data from wearable/mobile devices (Leng et al., 2024). Considering
smart environments, ChatGPT has been used for zero-shot activity recognition
(Xia et al., 2023). However, this work assumes the availability of several
sensors capturing the interaction of the subject with many household items
(e.g., specific food ingredients), that is unrealistic in real-world
deployments. In this work, we consider a more realistic setting considering
the sensors that are usually deployed in smart-homes for ADLs recognition
(e.g., motion sensors, magnetic sensors, plug sensors). Finally, another work
uses LLMs to provide a label to clusters of raw sensor data time windows
(collected in an initial deployment phase) (Gao et al., 2024). A supervised
classifier is then trained using the pseudo-labels obtained through the LLM.
However, such approach is susceptible to the quality of clustering and, once
the supervised classifier is trained, it may fail in capturing behavioral
changes or activities not performed while collecting unlabeled data.
Differently from that work, ADL-LLM aims at leveraging the flexibility of LLMs
for each observed time window.

##  3\. The ADL-LLM framework

In this section, we present ADL-LLM: our LLM-based method for ADLs recognition
in smart home environments.

###  3.1. Problem Formulation

We consider a classic problem formulation for ADL recognition. A smart home is
equipped with a set $\mathbf{S}=\\{S_{1},S_{2},\dots,S_{m}\\}$ of
environmental sensors. Examples are magnetic sensors revealing open-close
events (e.g., doors, cabinets, etc.), motion/presence sensors revealing
movement/presence in specific areas, and smart plugs revealing the use of
appliances. There is a set of target activities of interest
$\mathbf{A}=\\{A_{1},A_{2},\dots,A_{n}\\}$, usually suggested by clinicians.
As in most works in this domain, we consider a smart home environment
inhabited by a single subject or with a sensing infrastructure able to
properly associate a sensor’s activation with the subject that activated it.
Based on the interaction of the subject with the environment, the smart home
generates a continuous stream of sensor activations and de-activations. This
stream is partitioned in fixed-time windows of $\tau$ seconds with a given
overlapping factor $o$. Given a window $w_{t}$, the goal of ADL-LLM is to
determine the activity $A\in\mathbf{A}$ that the subject was more likely
performing in the interval $[t,t+\tau]$.

###  3.2. Overall architecture of ADL-LLM

We assume that ADL-LLM runs on a trusted platform. In the case of the medical
application, it may run on a telemedicine platform as one of the components of
a complex system that continuously monitors subjects at risk of cognitive
decline. Figure 1 shows the overall architecture of the ADL-LLM framework.

Figure 1. Overall architecture of ADL-LLM. When the pool of examples is empty
ADL-LLM, acts as a zero-shot ADLs recognition method. Otherwise, it is a few-
shot approach.

The smart home generates a continuous stream of raw sensor data that is pre-
processed by the Sensor State Generation module to obtain a stream of sensor
states. Intuitively, a sensor state represents the timespan in which a sensor
is active (e.g., “the TV in the living room is ON from 12:34 to 13:02’’).
Then, the stream of sensor states is segmented in fixed-time windows by the
Segmentation module.

Since LLMs are specialized in natural language text, the main idea of ADL-LLM
is to generate a natural language representation for each window, so that an
LLM can “reason” on it leveraging its implicit common-sense knowledge. The
Window2Text module is in charge of transforming windows of sensor states into
natural language sentences describing what happened in the home during that
window. The LLM receives as input a System Prompt (i.e., instructions about
the task it has to perform) and a User Prompt (i.e., the textual
representation of the window enriched with additional instructions for the LLM
by the User Prompt Construction module). The output of the LLM is finally
processed by the Activity Label Extraction module to obtain the most likely
activity performed by the subject.

ADL-LLM works in two different modalities:

  * •

Zero-Shot, where no labeled data is available

  * •

Few-Shot, where a small labeled dataset is available

Indeed, in certain circumstances it may be possible to collect a small dataset
of sensor data windows labeled with the corresponding ADL. For instance, when
the sensing infrastructure is physically installed in the home, the
technicians may ask the subject to perform a limited number of activities (as
a sort of ”system’s calibration” phase). The data associated with these
activities is manually labeled and stored in the Pool of Examples. In few-shot
ADL-LLM, when processing a window, the System Prompt is enriched by the
Semantic-based Example Selection module, which picks from this pool the
samples that are the most “semantically” similar to the textual representation
of currently processed window. The selected samples have the role of further
guiding the LLM on the ADLs recognition task. In the LLM domain, this approach
is known as few-shot prompting, and it is widely used to improve the output of
LLMs (Brown et al., 2020).

###  3.3. The main components of ADL-LLM

In the following, we describe the main modules of ADL-LLM, i.e., those used
both in the zero-shot and the few-shot modalities.

####  3.3.1. Sensor States Generation

The smart home generates a continuous stream of binary sensor events
$Stream=\langle\dots,E_{t-1}^{S_{l}},E_{t}^{S_{m}},E_{t+1}^{S_{q}},\dots\rangle$.
Each $E_{t}^{S}$ represents the activation or the deactivation of a sensor $S$
at a specific time $t$. For instance, consider a magnetic sensor on the
fridge. This sensor generates an ON event when activated (when the subject
opens the fridge) and an OFF event when deactivated (when the subject closes
the fridge). Other sensors share a similar behavior, like motion and pressure
sensors. For sensors that generate continuous values, as the smart plug that
continuously detect the power consumption of specific appliances, we use a
threshold on the detected energy consumption to infer when the appliance is
turned ON and OFF. For sensors that provide almost continuous values but are
not directly associated with activation and deactivation events, like the
temperature sensor, we discretize the values in different relevant ranges
(e.g., for the temperature: $15$-$18$°C, $19$-$24$°C, $24$-$30$°C, $>$$30$°C).
Suppose that the thermostat in the living room detected $18$°C at $t_{i}$ and
then $19$°C at time $t_{i+1}$. In this case, the stream includes a
deactivation event at time $t_{i}$ for the temperature being in the range
$15$-$18$°C, and an activation event at time $t_{i+1}$ for the temperature
being in the range $19$-$24$°C.

We convert the stream of events into a stream of sensor states, where a sensor
state $st(\sigma_{i},t_{s},t_{e})$ denotes that the sensor state $\sigma_{i}$
(derived from $S_{i}$) is true during the time interval $[t_{s},t_{e}]$. For
instance, $st(\textit{FridgeDoorOpen},\textit{12:32pm},\textit{12:33pm})$
represents the fact that the fridge door was open in the interval
$[\textit{12:32pm},\textit{12:33pm}]$. As another example, the
$st(\textit{NearTheStove},\textit{11:45am},\textit{12:00pm})$ represent the
fact that the motion sensor near the stove detected the subject’s movements
from 11:45am to 12:00pm.

More formally, we generate a sensor state $st(\sigma,t_{s},t_{e})$ when the
stream contains two events $E_{t_{s}}^{S}$ and $E_{t_{e}}^{S}$ with
$t_{s}<t_{e}$ being the instants in which the events of activation and de-
activation of sensor $S$ occurred, $\sigma$ indicates a property revealed by
the activation state of $S$, and there is no $E_{t_{q}}^{S}$ such that
$t_{s}<t_{q}<t_{e}$.

####  3.3.2. Segmentation

ADL-LLM employs a fixed-time segmentation of the stream of sensor states,
considering windows of $\tau$ seconds and an overlap factor $o$. Intuitively,
each window is associated with all the sensor states whose interval intersects
with the window interval. More formally, a window $w_{t}$ over the interval
$[t,t+\tau]$ is associated with the sensor states $st(\sigma,t_{s},t_{e})$
such that $t_{s}\leq t+\tau$ and $t_{e}\geq t$.

Note that a window may be associated with multiple states even derived from
different sensors. Based on the intervals we distinguish three categories that
will be useful for text generation:

  * •

Inner states: this is the case of $t_{s}\geq t$ and $t_{e}\leq t+\tau$. This
represents the simple case of the state occurring completely within the
window.

  * •

Already active states: this is the case of $t_{s}<t$. These states are already
active before the start of the window.

  * •

Persistent states: this is the case of $t_{e}>t$. These states persist after
the end of the window.

Note that a state may be both Already active and Persistent when $t_{s}<t$ and
$t_{e}>t$.

Figure 2. An example of Window2Text in action on the UCI ADL dataset

####  3.3.3. Window2Text

The Window2Text module converts a sensor state window $w_{t}$ into its
corresponding textual representation $T(w_{t})$. Intuitively $T(w_{t})$
describes what happened in the home from the subject’s perspective.

In order to do this, the sensor states associated with a window are mapped to
a description in natural language of what the subject may have done to lead to
those states.

###### Example 3.0.

If the state $st(StoveON,t_{s},t_{e})$ is an inner state in the current
window, it is mapped to the the sentence “The subject turned on the stove and
turned it off after $t_{e}-t_{s}$ seconds”. When the same state is an Already
active state for the current window, the sentence “The stove is already turned
on” is produced, and if there is another already active state (e.g., about the
fridge) the corresponding sentence is combined with the one for the stove.
When the $StoveON$ state ends before the end of the window, the following
sentence is produced “After $t_{e}-t$ seconds the subject turned off the
stove”. As another example, the inner state $st(NearBathroomSink,t_{s},t_{e})$
generated by a PIR sensor leads to the sentence “The subject was near the sink
for $t_{e}-t_{s}$ seconds”.

The states associated with a window may also be generated by sensors in
different rooms. Since the location is semantically relevant to detect
possible activities, the description generated by Window2Text always reports
this information, inferring it from the associated states.

###### Example 3.0.

Consider a window $w_{t}$ that has associated the Inner states
$st(FridgeDoorOpen,t1,t2)$, $st(NearStove,t3,t4)$ and a Persistent state
$st(OnTheCouch,t5,t6)$, where $t2<t3$, $t4<t5$, and $t6>t+\tau$. ADL-LLM
automatically infers that $FridgeDoor$ and $NearStove$ occurred consecutively
in kitchen, and after the subject moves to the living room for the
$OnTheCouch$ state. This window leads to the following natural language
sentence: “The subject is in the kitchen. Here, they open the fridge door.
After $t2-t1$ seconds, they close the fridge door. Then, they are near the
stove for $t4-t3$ seconds. Then, they move to the living room. Here, they sit
on the couch.”

Overall, given a window $w_{t}$ the output of the Window2Text module is a
description $T(w_{t})$ with the following structure:

  1. (1)

A first sentence reports the length ($\tau$ seconds) of the window and the
approximate time of the day. The first was found useful to provide a temporal
context to the LLM since $\tau$ is usually shorter than the typical duration
of an ADL; The second was found useful for the LLM to exploit correlations
between common ADLs and the time of the day in which they are typically
executed (e.g., preparing dinner, preparing lunch).

  2. (2)

The following sentences are generated based on the associated states. The
first reports the location where the subject is at the beginning of the window
(if it can be inferred). Then, a sentence generated based on the Already
active states follows. Finally, for each room visited by the subject during
$w_{t}$ (in chronological order), the description includes the sentences
generated based on Inner and Persistent states.

Figure 2 shows an example of Window2Text in action on the UCI ADL dataset.

A relevant requirement for an effective text generation by Window2Text is the
availability for each sensor of meta-data describing its location in the home
and the mapping from the sensor $S$ to the state property $\sigma$ that is
used in sentence generation.

####  3.3.4. System prompt

The system prompt is the set of instructions defining the general task
assigned to the LLM and is independent of the specific window being
considered. Our prompt first asks the LLM to act as an activity recognition
system. Then, it provides information about the rooms in the smart home, the
main interactions with the environment that can be captured by the available
sensors, and the list of activities of interest. Since this information
depends on the specific smart environment and application, it must be provided
during the installation phase. An example of a system prompt is reported in
Figure 3.

Figure 3. An example of system prompt in ADL-LLM for the UCI ADL dataset (Home
B). The sets of locations, household elements captured by sensors, and
activities are variables depending on the specific home where ADL-LLM is
installed.

####  3.3.5. User Prompt Construction

The user prompt contains the information about the specific window $w_{t}$ for
which the LLM has to predict the most likely activity. The user prompt
contains:

  * •

The textual representation $T(w_{t})$ of the window obtained with Window2Text
(described in Section 3.3.3).

  * •

Instructions about selecting only one activity among the target ones.

  * •

The sentence “Reason step by step”, used to instruct the LLM to perform
intermediate “reasoning” steps before providing the predicted ADL. This
technique is well-known as Chain Of Thought (COT), it usually leads to
increasing the correctness of the LLM’s output (Wei et al., 2022), and proved
to be useful even in our system.

####  3.3.6. LLM-based ADLs Recognition and Activity Label Extraction

The system prompt and the user prompt are provided as input to the LLM for
each window $w_{t}$, and the LLM provides a textual description as output,
reporting some facts supporting the selection of the predicted activity.
Figure 4 shows an example of the output returned by the LLM.

Figure 4. ADL-LLM: An example of input and corresponding output.

Finally, the Activity Label Extraction module uses regular expressions to
extract only the activity label from the LLM’s output. In rare cases, despite
the instructions, the LLM provides an activity label that does not
syntactically match one the candidate activities in $A$. For instance, instead
of snacking, in a few cases the LLM returned as output preparing late night
snack or preparing midnight snack. In these cases, the Activity Label
Extraction module uses the cosine similarity to compare the text embeddings of
the labels. The label with the highest similarity to the predicted output is
selected as the final prediction.

###  3.4. Few-shot ADL-LLM components

When a small dataset of labeled sensor data is available, ADL-LLM can be
improved by implementing few-shot prompting. This is achieved by extending the
zero-shot architecture with two components described in the following.

####  3.4.1. Building a Pool of Examples

Suppose that a small labeled dataset $\mathbf{D}$ is available. Let
$\mathbf{D}$ be a set of pairs $(w_{t},A)$ where $w_{t}$ is a window of raw
sensor data, while $A$ is the corresponding activity label. To perform
semantic comparison, we create the set $\mathbf{E}$ that we call Pool of
Examples. For each pair $(w_{t},A)\in\mathbf{D}$, $\mathbf{E}$ includes a pair
$(T(w_{t}),A)$. Note that, since $\mathbf{E}$ is a set, there is only one
occurrence for each pair $(T(w_{t}),A)$. We remove from $\mathbf{E}$ all the
examples sharing the same textual description $T(w_{t})$ and associated with
different labels. Indeed, such examples represent patterns shared by multiple
activities, and including these as examples in the prompt may be misleading
for the LLM.

####  3.4.2. Semantic-based Example Selection

Given a textual representation $T(w_{t})$ of a window $w_{t}$ currently
processed by ADL-LLM, the corresponding embedding
$\phi_{w_{t}}=\Phi(T(w_{t}))$ is computed, where $\Phi$ is a function (e.g., a
pre-trained LLM) mapping sentences into a latent space representation (i.e.,
their embedding). Then, for each pair $(T(w),A)\in\mathbf{E}$, we compute the
pairwise cosine similarity between $\phi_{w_{t}}$ and $\phi_{w}=\Phi(T(w))$.
We enrich the system prompt with the $k$ pairs $(T(w),A)\in E$ that are
associated with the highest pairwise similarity between $\phi_{w_{t}}$ and
$\phi_{w}$. The pseudo-code for Semantic-Based Example Selection is summarized
in Algorithm 1.

Algorithm 1 Semantic-based Example Selection

0: The pool of examples $\mathbf{E}$, the textual representation of the
current window $T(w_{t})$, an embedding function $\Phi$, a parameter
$k\in\mathbb{N^{+}}$

0: $k$ examples from $\mathbf{E}$ semantically similar to $T(w_{t})$

$\phi_{w_{t}}\leftarrow\Phi(T(w_{t}))$

$pairwiseDict\leftarrow\\{\\}$ // Initialize an empty dictionary

for $(T(w),A)\in\mathbf{E}$ do

$\phi_{w}\leftarrow\Phi(T(w))$

$pairwiseDict[T(w)]\leftarrow\frac{\phi_{w_{t}}\cdot\phi_{w}}{|\phi_{w_{t}}||\phi_{w}|}$

end for

$selectedExamples\leftarrow\emptyset$

for $i=1$ to $k$ do

$e\leftarrow\underset{{{(T(w),A)\in\mathbf{E}}}}{\arg\max}pairwviseDict[T(w)]$

$selectedExamples\leftarrow selectedExamples\bigcup\\{e\\}$

remove $e$ from $pairwiseDict$

end for

return $selectedExamples$

##  4\. Experimental Evaluation

###  4.1. Datasets

We evaluated ADL-LLM using two public datasets: MARBLE (Arrotta et al., 2021)
and UCI ADL (Ordónez et al., 2013). We chose these datasets because they
include sensor data from a wide variety of sensors (e.g., magnetic, plug,
motion) that we are currently considering on a project involving data
collection in the homes of subjects with Mild Cognitive Impairment (MCI).
While the majority of works in the literature utilize the well-known CASAS
datasets (Cook et al., 2012), which primarily include data from motion
sensors, these datasets are limited in their ability to precisely characterize
the specific activities performed by subjects. Although data-driven models can
identify low-level patterns within motion sensor data, our goal is to leverage
the common-sense knowledge implicitly encoded in LLMs. Preliminary experiments
suggested that using LLMs with datasets mostly based on motion sensors can not
reliably recognize ADLs.

####  4.1.1. MARBLE

The MARBLE dataset (Arrotta et al., 2021) includes data collected from $12$
subjects in both single and multi-inhabitant scenarios. The dataset was
collected using the following environmental sensors: magnetic sensors to
capture the opening and closing of various drawers and cabinets, pressure
sensors to detect when a person was sitting at the dining table or on the sofa
in the living room, and smart plugs to monitor the usage of the stove and
television. Additionally, the dataset includes data from an Android
application on the user’s smartphone, which automatically detects incoming and
outgoing phone calls. Finally, the dataset incorporates data from a smartwatch
worn by the user, providing micro-localization information (based on an
infrastructure using Bluetooth Low Energy (BLE) beacons) at the room-level.

We performed our experiments only considering the single-inhabitant scenarios.
Moreover, while the dataset also included inertial sensor data from the
smartwatch (e.g., continuous acceleration on three axis), we ignore these type
of data as it can not be directly transformed into a prompt in natural
language.

The dataset includes the following $13$ activities: answering phone, clearing
table, preparing hot meal, preparing cold meal, eating, entering home, leaving
home, making phone call, setting up table, taking medicines, using PC, washing
dishes, and watching TV.

In our experiments we do not consider the washing dishes activity since it is
mainly characterized by inertial sensors in smartwatch data (not used by ADL-
LLM). Moreover, we merged the making phone call and answering phone call
activities into a single activity called phone call.

####  4.1.2. UCI ADL

The UCI ADL dataset (Ordónez et al., 2013) includes data about the ADLs
performed by two subjects in their own homes. The two homes are named Home A
and Home B in the dataset. The overall data collection duration is $35$ days:
$14$ days for Home A and $21$ days for Home B.

Different environmental sensors are considered: PIR sensors (installed on
doors, bathroom sink, stove, and shower areas) to detect the subject’s
movements, pressure sensors to detect when the subject was sitting on the
couch and the bed, magnetic sensors to detect when the subject was opening and
closing cupboards and cabinets, and plug sensors sensors to detect the usage
of microwave and toaster.

Home A and Home B sensing infrastructure have some differences. Compared to
Home A, Home B has a slightly more limited sensing infrastructure since it is
not equipped with the PIR sensor over the stove, the plug sensor to detect the
usage of the toaster, and the magnetic sensor to detect interactions with the
bathroom cabinet. Nonetheless, with respect to Home A, Home B includes
additional PIR sensors installed on the kitchen, bedroom and living room
doors.

The dataset covers the following activities: preparing breakfast, preparing
lunch, preparing dinner, snacking, personal care (labeled as grooming in the
dataset), showering, leaving,relaxing on couch (labeled as spare time/TV in
the dataset), sleeping, and toileting. Note that preparing dinner occurs only
in the dataset collected in Home B.

In our experiments, we ignored the toileting activity since the real semantic
of this activity and of the flush sensor was not clear in the dataset (and it
was not reported in the dataset’s documentation).

###  4.2. Supervised baselines

In order to show the effectiveness of ADL-LLM, we compare it with three state-
of-the-art supervised approaches for ADLs recognition.

  * •

BiLSTM (Liciotti et al., 2020): Also known as Bidirectional LSTM. It
represents a standard recurrent neural network based on LSTM, processing
windows in both forward and backward directions.

  * •

CascEnsLSTM (Liciotti et al., 2020): Also known as Cascade Ensemble LSTM. A
more complex recurrent network that is composed of two heads (an LSTM and a
Bi-LSTM) processing the same window in parallel. The output of the two heads
is concatenated and fed to a final LSTM layer.

  * •

DeepConvLSTM (Ordóñez and Roggen, 2016): Also known as Deep Convolutional and
LSTM Recurrent Neural Network. It combines convolutional and LSTM layers to
effectively capture both spatial and temporal features.

These baselines are representative because they reflect the most effective
architectures used in the literature for supervised ADLs recognition. We did
not include methods specifically designed for labeled data scarcity scenarios
because we are not aware of state-of-the-art approaches specifically designed
for this setting. As we already mention in Section 2, methods like self-
supervised learning or neuro-symbolic approaches have been proposed only for
mobile/wearable-based HAR. Smart-Home ADLs recognition introduces different
challenges, since there are no large dataset available for pre-training, and
different homes may have significantly different sensor layouts.

###  4.3. Experimental Setup

####  4.3.1. Implementation

We implemented a prototype of ADL-LLM in Python 3.10.12 language using the
langchain 0.1.17 library. As LLM for ADLs recognition, we used the OpenAI
‘gpt-3.5-turbo-0125’ model (setting temperature to $0$), while we used the
OpenAI ‘text-embedding-3-small’ model to compute embeddings from the textual
representation of windows for few-shot prompting. Based on what is suggested
in the literature, we used $16$-second windows with $80\%$ overlap for the
MARBLE dataset (Arrotta et al., 2021) and $60$ seconds windows with $80\%$
overlap for the UCI ADL dataset (Ordónez et al., 2013).

Considering the supervised baselines, we used the code available in public
repositories (DeepCASAS 222https://github.com/danielelic/deep-casas/ and
DeepConvLSTM 333https://github.com/STRCWearlab/DeepConvLSTM/). We slightly
modified the existing implementations to ensure compatibility with our
software environment and data structures, while faithfully reproducing the
architectures described in the literature. In all of our baselines, the LSTM
layers have $64$ units, while for the convolutional block of the DeepConvLSTM
we used a single convolutional layer with $8$ $2$x$2$ filters. Before feeding
it to the networks, environmental sensor data is converted into multivariate
binary time-series following a standard approach in the literature (Van
Kasteren et al., 2008).

####  4.3.2. Dataset splitting strategy

Since our major contribution is a zero-shot approach, in principle we could
have used the entire datasets as test set. However, we also need a training
set for two reasons: to train the supervised baselines and for few-shot ADL-
LLM. For this reason, we divided each dataset into $30\%$ for training set and
$70\%$ for testing set. This splitting reflects a labeled data scarcity
scenario, where only a reduced labeled dataset is available. The training set
is not used in any way by zero-shot ADL-LLM, while it is used to train the
supervised baselines and to create the pool of examples in few-shot ADL-LLM.

####  4.3.3. Simulating data scarcity scenarios

In order to evaluate the effectiveness of few-shot ADL-LLM we consider
different labeled data availability scenarios to populate the pool: $100\%$,
$50\%$, $25\%$, $10\%$, and $5\%$ of the training set. In these settings, the
supervised baselines use the available labeled data for training. We expect
that, in data scarcity scenarios, few-shot ADL-LLM outperforms the supervised
baselines.

###  4.4. Results

####  4.4.1. Zero-shot ADL-LLM

Tables 1 and 2 show, for both datasets, the recognition rate of zero-shot ADL-
LLM (not using any labeled data) with the supervised baselines trained using
the $100\%$ of the training set.

Table 1. UCI ADL: F1 score of Zero-Shot (using no training data) ADL-LLM vs. supervised baselines trained using $100\%$ of the training set (i.e., $30\%$ of the dataset) | Home A | Home B
---|---|---
Activity | ADL-LLM | BiLSTM | CascEnsLSTM | DeepConvLSTM | ADL-LLM | BiLSTM | CascEnsLSTM | DeepConvLSTM
Leaving home | 1.00 | 0.95 | 0.92 | 0.99 | 0.90 | 0.84 | 0.81 | 0.88
Personal care | 0.97 | 0.98 | 0.97 | 0.99 | 0.96 | 0.92 | 0.92 | 0.93
Preparing breakfast | 0.75 | 0.90 | 0.90 | 0.94 | 0.71 | 0.35 | 0.35 | 0.39
Preparing dinner | – | – | – | – | 0.33 | 0.06 | 0.13 | 0.13
Preparing lunch | 0.91 | 0.95 | 0.95 | 0.96 | 0.40 | 0.35 | 0.36 | 0.41
Relaxing on couch | 0.96 | 1.00 | 1.00 | 1.00 | 0.71 | 0.85 | 0.85 | 0.87
Showering | 0.99 | 0.99 | 0.99 | 1.00 | 0.95 | 0.93 | 0.93 | 0.96
Sleeping | 1.00 | 0.99 | 0.99 | 1.00 | 0.94 | 0.99 | 0.99 | 0.99
Snacking | 0.36 | 0.04 | 0.07 | 0.00 | 0.28 | 0.37 | 0.34 | 0.40
Weighted Avg. | 0.94 | 0.96 | 0.96 | 0.97 | 0.75 | 0.72 | 0.72 | 0.75
Table 2. MARBLE: F1 score of Zero-Shot ADL-LLM (using no training data) vs. supervised baselines using $100\%$ of the training set (i.e., $30\%$ of the dataset) Activity | ADL-LLM | BiLSTM | CascEnsLSTM | DeepConvLSTM
---|---|---|---|---
Clearing table | 0.25 | 0.44 | 0.42 | 0.55
Eating | 0.91 | 0.91 | 0.91 | 0.91
Entering home | 0.41 | 0.00 | 0.00 | 0.13
Leaving home | 0.71 | 0.00 | 0.00 | 0.00
Phone call | 0.97 | 0.76 | 0.76 | 0.81
Preparing cold meal | 0.51 | 0.63 | 0.62 | 0.73
Preparing hot meal | 0.82 | 0.82 | 0.82 | 0.90
Setting up table | 0.13 | 0.55 | 0.48 | 0.57
Taking medicines | 0.47 | 0.54 | 0.53 | 0.59
Using pc | 0.97 | 0.91 | 0.91 | 0.91
Watching TV | 0.97 | 0.94 | 0.94 | 0.96
Weighted F1-score | 0.80 | 0.78 | 0.77 | 0.82

We observe that, for all the datasets, zero-shot ADL-LLM reaches recognition
rates that are very close to the best supervised baseline, that is
DeepConvLSTM. Indeed, on the UCI ADL dataset, ADL-LLM is only $3\%$ behind
DeepConvLSTM in terms of F1 score, while only $2\%$ on the MARBLE dataset. We
observe that Home A of the UCI ADL dataset represents the easiest benchmark
while Home B, having a reduced sensing infrastructure (like we explained in
Section 4.1), makes ADLs recognition more challenging.

An interesting insight is that sometimes ADL-LLM is capable of recognizing
ADLs that are not captured by supervised baselines. For instance, this happens
in the Home A of the UCI ADL dataset, where ADL-LLM is capable of recognizing
snacking (even if with a low F1 score), while the supervised baselines fail.
Similar phenomena occur in Home B for preparing dinner and in MARBLE for
entering home and leaving home. This is due to the fact that these activities
are poorly represented in their respective datasets, and a supervised
classifier can not learn them properly during the training phase (considering
that the training set is $30\%$ of the whole dataset). On the other hand, ADL-
LLM leverages the common-sense knowledge implicitly encoded in the LLM, and it
is capable of recognizing these activities without any type of training.

Figures 5 and 6 depict the confusion matrices of zero-shot ADL-LLM, that can
be used to inspect the results more closely.

(a) Home A

(b) Home B

Figure 5. Zero-Shot ADL-LLM: Confusion matrices on UCI ADL dataset Figure 6.
Zero-Shot ADL-LLM: Confusion matrix on MARBLE dataset

Considering the UCI-ADL dataset, the most challenging activity in both homes
is snacking. This phenomena is evident in the confusion matrices depicted in
Figure 5. Indeed, snacking is often confused with the other food-related
activities like preparing breakfast, since they share similar sensor patterns.
In the Home B of the UCI ADL dataset, the relaxing on couch activity is
associated with a high number of false positives. Indeed, Figure 5 shows that
this activity is often misclassified by ADL-LLM with eating-related activities
and leaving home. We believe that this may be due to the fact that the subject
was often eating in the living room, that is also close to the main entrance
door. Hence, while supervised learning baselines were able to recognize these
activities since they have been trained with the subject’s data, the general
knowledge of ADL-LLM often inferred relaxing on couch when the subject spent
some time in the living room.

Considering the MARBLE dataset, we observe that ADL-LLM struggles in
distinguishing clearing table, setting up the table, and preparing a cold
meal. This is due to the lack of specific sensors characterizing these
activities that involve similar actions (e.g., using the pantry, cutlery
drawer, fridge). Considering taking medicines, it is correctly classified only
when the windows are associated with a state related to the interaction with
the medicine cabinet. However, these interactions are usually short and might
not capture the entire activity. Indeed, the subject may move to another
location to take medicines, such as a dining room or kitchen, where other
activities usually take place. Finally, the misclassifications of entering
home and leaving home are due to an insufficient sensing infrastructure to
recognize these activities (i.e., no magnetic sensor on the main door). For
instance, when a subject leaves home starting from the kitchen, ADL-LLM
sometimes mislabels it as meal preparation, while windows only including
movements in the hall lead to label leaving home as entering home due to LLM’s
assumptions about the home entry points.

Finally, Figure 7 shows the Empirical Cumulative Density Function (ECDF) of
the response times provided by gpt-3.5 on both datasets.

Figure 7. Zero-Shot ADL-LLM: Empirical Cumulative Density Function (ECDF) of
response times.

We observe that $\approx 80\%$ of the responses are generated within $2$
seconds, and almost all responses for both datasets are generated within four
seconds. Only in a couple of instances, the response time was longer than $10$
seconds, probably due to networking issues. We believe that these results show
that ADL-LLM could be applied for real-time ADLs recognition. UCI-ADL is
associated with slightly higher response times due to the fact that it
involves $60$ seconds windows, while MARBLE considered $16$ seconds windows.

####  4.4.2. Few-shot ADL-LLM

In the following, we show the results of few-shot ADL-LLM. We performed the
experiments by testing different values of $k$ (i.e., the number of examples),
specifically $3$, $5$, $7$. Figures 8 and 9 show the overall recognition rate
of few-shot prompting compared to DeepConvLSTM (i.e., the best performing
supervised baseline) at different data scarcity scenarios. For both datasets,
we empirically determined that the best value for $k$ is $7$.

(a) Home A

(b) Home B

Figure 8. UCI-ADL: Few-shot ADL-LLM ($k=7$) vs DeepConvLSTM at different data
scarcity scenarios. Figure 9. MARBLE: Few-shot ADL-LLM ($k=7$) vs DeepConvLSTM
at different data scarcity scenarios.

Our results show that, even when only the $5\%$ of training data is used to
populate the pool of examples, few-shot ADL-LLM outperforms zero-shot ADL-LLM,
thus showing the effectiveness of adding examples in the prompt. Except for
Home A of the UCI ADL dataset, few-shot ADL-LLM outperforms the supervised
baseline, whose recognition rates significantly decrease as data scarcity
increase. As we mentioned before, Home A (Figure 8(a)) represents a too simple
benchmark and even a small dataset leads to excellent recognition rates, and
ADL-LLM slightly outperforms the baseline only when considering the $5\%$ of
the training set. These results confirm that, even when some labeled data are
available, ADL-LLM is a better choice than standard data-driven approaches.
Tables 3 and 4 show the results at the activity level when only the $5\%$ of
the training set is available. It is possible to see that increasing $k$ leads
to a monotonic increase of the recognition rate. The detailed results for the
other data availability scenarios can be found in the appendix (Section A).

Table 3. UCI ADL: Few-Shot ADL-LLM vs the supervised baselines (5% of the training set) | Few-Shot ADL-LLM | Supervised baselines | Few-Shot ADL-LLM | Supervised baselines
---|---|---|---|---
Activity | K=3 | K=5 | K=7 | BiLSTM |  | Casc
---
Ens
BiLSTM
| DeepConv
---
LSTM
K=3 | K=5 | K=7 | BiLSTM |  | CascEns
---
BiLSTM
| Deep
---
Conv
LSTM
Leaving home | 1.00 | 1.00 | 1.00 | 0.02 | 0.00 | 0.97 | 0.85 | 0.85 | 0.85 | 0.01 | 0.06 | 0.85
Personal care | 0.99 | 0.99 | 0.99 | 0.04 | 0.02 | 0.94 | 0.96 | 0.96 | 0.96 | 0.88 | 0.85 | 0.95
Preparing breakfast | 0.68 | 0.85 | 0.89 | 0.00 | 0.00 | 0.85 | 0.76 | 0.75 | 0.78 | 0.27 | 0.29 | 0.30
Preparing dinner | - | - | - | - | - | - | 0.30 | 0.34 | 0.35 | 0.05 | 0.05 | 0.32
Preparing lunch | 0.90 | 0.95 | 0.92 | 0.43 | 0.43 | 0.95 | 0.31 | 0.41 | 0.45 | 0.22 | 0.09 | 0.31
Relaxing on couch | 0.97 | 0.97 | 0.97 | 0.00 | 0.00 | 0.99 | 0.63 | 0.66 | 0.69 | 0.84 | 0.79 | 0.86
Showering | 1.00 | 1.00 | 1.00 | 0.01 | 0.00 | 0.99 | 0.95 | 0.95 | 0.95 | 0.87 | 0.79 | 0.95
Sleeping | 0.98 | 0.98 | 0.98 | 0.00 | 0.00 | 0.99 | 0.87 | 0.90 | 0.93 | 0.82 | 0.83 | 0.99
Snacking | 0.33 | 0.43 | 0.34 | 0.00 | 0.00 | 1.00 | 0.37 | 0.34 | 0.32 | 0.20 | 0.17 | 0.34
Weighted F1-score | 0.93 | 0.96 | 0.96 | 0.13 | 0.12 | 0.95 | 0.74 | 0.75 | 0.77 | 0.59 | 0.56 | 0.73
Table 4. MARBLE: Few-Shot ADL-LLM vs the supervised baselines (5% of the training set) | Few-Shot ADL-LLM | Supervised baselines
---|---|---
Activity | K=3 | K=5 | K=7 | BiLSTM |  | Casc
---
Ens
BiLSTM
| Deep
---
Conv
LSTM
Clearing table | 0.20 | 0.21 | 0.26 | 0.00 | 0.00 | 0.01
Eating | 0.81 | 0.83 | 0.83 | 0.88 | 0.79 | 0.87
Entering home | 0.48 | 0.34 | 0.38 | 0.00 | 0.00 | 0.00
Leaving home | 0.75 | 0.48 | 0.44 | 0.00 | 0.00 | 0.00
Phone call | 0.99 | 0.99 | 0.99 | 0.54 | 0.49 | 0.78
Preparing cold meal | 0.62 | 0.53 | 0.52 | 0.40 | 0.04 | 0.57
Preparing hot meal | 0.75 | 0.78 | 0.82 | 0.65 | 0.36 | 0.84
Setting up table | 0.15 | 0.32 | 0.27 | 0.14 | 0.04 | 0.46
Taking medicines | 0.22 | 0.23 | 0.30 | 0.00 | 0.00 | 0.37
Using pc | 0.96 | 0.97 | 0.97 | 0.81 | 0.62 | 0.90
Watching tv | 0.98 | 0.97 | 0.98 | 0.94 | 0.62 | 0.96
Weighted F1-score | 0.78 | 0.79 | 0.80 | 0.60 | 0.43 | 0.75

##  5\. Discussion

###  5.1. Segmentation

In this work, we segmented sensor data using fixed-size windows, that is a
standard approach in the literature. However, while this approach makes sense
to train machine learning classifiers and find low-level patterns, this may be
less suitable for LLMs. Indeed, fixed-size segments capture only a small
portion of the activity, and they often include transitions between different
activities.

We will explore dynamic segmentation strategies based on change-point
detection (Aminikhanghahi and Cook, 2017a). Indeed, dynamic segmentation can
be leveraged to detect significant changes in sensor data distribution that
likely associated with changes in the performed activities (Aminikhanghahi and
Cook, 2017b). We believe that dynamic segmentation would allow LLMs to process
more meaningful information to reliably perform ADLs recognition.

###  5.2. Open-world HAR

For the sake of this work, we constrained the LLM in predicting the most
likely activity among a set of candidates. This is the most common approach in
the ADLs recognition literature. However, this is a limitation, since in real
life deployments we can not foresee in advance all the possible activities
carried out by the subeject. Moreover, the answers from the model may
“overfit” on the specific activity sets, especially when there is a one-to-one
mapping between an activity in the set and sensor events.

We will explore whether the flexibility of pre-trained LLMs can also be
effective in open-world HAR, where there is no fixed set of activities to
consider. One of the main challenges will be that the LLM may provide
different labels for the same activity (e.g., eating may also be recognized as
dining, snacking, having lunch, etcetera). Hence, we believe that in such
settings it will be necessary to semantically cluster the predicted labels.

###  5.3. Cloud vs Local LLM processing

In our experiments, we adopted third-parties LLM models due to their
effectiveness and to simplify the implementation. However, outsourcing
information about personal activities to such untrusted third-parties raises
several concerns about privacy. Indeed, personal data about human activities
may reveal private habits and health issues (Bettini and Riboni, 2015).
Moreoever, since we use the LLM for each sensor data time window, ADL-LLM may
be too costly when using third parties LLMs.

Our vision is that open source LLMs should be used in real-world
implementations of ADL-LLM. Since the deployment of such models may be
prohibitive in smart-home gateways, we believe that they should run on high-
performance computing machines in a trusted domain (e.g., in a telemedicine
infrastructure considering a medical application). When this is not possible,
compressed versions (e.g., quantized versions and/or with reduced number of
parameters) may be even deployed in the home domain (e.g., in a home gateway
powered with a GPU). However, since these models are currently less effective,
we expect a significant drop in the recognition rate.

We evaluated the impact of smaller open source LLMs on ADL-LLM performing
preliminary experiments by deploying a LLaMa2 model on our infrastructure.
Specifically, we used the llama-2-13b-chat-hf model through HuggingFace
444https://huggingface.co/meta-llama/Llama-2-13b-chat-hf. We ran these
experiments on a Ubuntu server with a partitioned NVIDIA A100 GPU, with $42$GB
of RAM, and a AMD EPYC 8-core CPU. Table 5 compares zero-shot ADL-LLM using
LLaMa2 and gpt3.5 on the MARBLE dataset.

Table 5. Zero-shot ADL-LLM (MARBLE): llama-2-13b-chat-hf vs. gpt-3.5-turbo-0125 Activity | llama-2-13b-chat-hf | gpt-3.5-turbo-0125
---|---|---
Clearing table | 0.23 | 0.25
Eating | 0.60 | 0.91
Entering home | 0.58 | 0.41
Leaving home | 0.86 | 0.71
Phone call | 0.99 | 0.97
Preparing cold meal | 0.00 | 0.51
Preparing hot meal | 0.61 | 0.82
Setting up table | 0.00 | 0.13
Taking medicines | 0.34 | 0.47
Using pc | 0.98 | 0.97
Watching tv | 0.96 | 0.97
Weighted F1-score | 0.68 | 0.80

As expected, gpt-3.5-turbo-0125 shows superior performance than
llama-2-13b-chat-hf. Specifically, LLaMa struggles even more in recognizing
the activities lacking distinct sensor inputs like preparing a cold meal,
setting up table, and clearing table. Additionally, llama-2-13b-chat-hf was
not accurate in providing the output activity in the required format. As we
described in Section 3.3.4, the system prompt specifies to enclose the most
likely activity within the curly braces. While gpt-3.5-turbo-0125 always
complied, llama-2-13b-chat-hf often did not adhere to this instruction. This
required us to find ad-hoc solutions to extract activity labels when the
output format was different.

Furthermore, llama-2-13b-chat-hf exhibited a tendency to output activity
labels not even semantic close to the predefined activity sets, such as
‘moving around the house’. Hence, we will investigate alternative prompting
strategies and personalization strategies (e.g., fine-tuning) for these
smaller models.

###  5.4. Impact of LLMs evolution

While we executed all of our experiments running gpt-3.5 due to its cost-
effectiveness and accuracy, one may argue that more advanced LLM models may
lead to even better results. In the following, we show preliminary experiments
of zero-shot ADL-LLM using the more advanced gpt-4o model on the MARBLE
dataset. Table 6 shows that gpt-4o improves the F1-score by $+4\%$ over
gpt-3.5-turbo-0125.

Table 6. Zero-Shot ADL-LLM (MARBLE): gpt-4o vs. gpt-3.5-turbo-0125 Activity | gpt-4o | gpt-3.5-turbo-0125
---|---|---
Clearing table | 0.23 | 0.25
Eating | 0.91 | 0.91
Entering home | 0.60 | 0.41
Leaving home | 0.74 | 0.71
Phone call | 0.99 | 0.97
Preparing cold meal | 0.55 | 0.51
Preparing hot meal | 0.85 | 0.82
Setting up table | 0.49 | 0.13
Taking medicines | 0.56 | 0.47
Using pc | 0.99 | 0.97
Watching tv | 0.99 | 0.97
Weighted F1-score | 0.84 | 0.80

These results indicate that, using more advanced LLMs, it is possible to even
surpass DeepConvLSTM trained on the whole training set (see Table 2). However,
this performance gain comes with a significant increase in costs, since at the
time of writing running experiments with gpt-4o is $10$ times more expensive
than with gpt-3.5-turbo-0125. Moreover, this advanced model was significantly
slower in generating the output. This experiment confirms that LLMs are zero-
shot ADLs recognition models, but doing it accurately by minimizing costs and
ensuring privacy is still an open challenge.

##  6\. Conclusion and Future Work

In this paper, we presented ADL-LLM: a zero-shot method for ADLs recognition
in smart-home environments. We showcased that LLMs can reach recognition rates
in ADLs recognition that are close to supervised approaches, with the benefit
of not requiring any labeled data.

While we evaluated ADL-LLM on public datasets, we intend to investigate its
effectiveness in the specific medical context of neurology, considering data
from real subjects at risk of cognitive decline. We are currently
collaborating with clinicians in the SERENADE
project555https://ecare.unimi.it/pilots/serenade/, collecting data from the
sensorized homes of a significantly large group of subjects, mostly living
alone and diagnosed with Mild Cognitive Impairment. Our long-term objective in
this project is a fine-grained recognition of the onset of cognitive decline.
The sensor data that we are collecting in these homes is unlabeled, and we
plan to adopt zero-shot ADL-LLM to capture the performed ADLs in this setting
and work from there on the detection of behavioral changes and anomalies under
the guidance of clinicians.

###### Acknowledgements.

This work was supported in part by projects MUSA, FAIR, and SERICS under the
NRRP MUR program funded by the EU-NGEU. Views and opinions expressed are
however those of the authors only and do not necessarily reflect those of the
European Union or the Italian MUR. Neither the European Union nor the Italian
MUR can be held responsible for them.

## References

  * (1)
  * Abdallah et al. (2018) Zahraa S Abdallah, Mohamed Medhat Gaber, Bala Srinivasan, and Shonali Krishnaswamy. 2018.  Activity recognition with evolving data streams: A review.  _ACM Computing Surveys (CSUR)_ 51, 4 (2018), 1–36.
  * Aminikhanghahi and Cook (2017a) Samaneh Aminikhanghahi and Diane J Cook. 2017a.  A survey of methods for time series change point detection.  _Knowledge and information systems_ 51, 2 (2017), 339–367.
  * Aminikhanghahi and Cook (2017b) Samaneh Aminikhanghahi and Diane J Cook. 2017b.  Using change point detection to automate daily activity segmentation. In _2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops)_. IEEE, 262–267.
  * Arrotta et al. (2021) Luca Arrotta, Claudio Bettini, and Gabriele Civitarese. 2021.  The marble dataset: Multi-inhabitant activities of daily living combining wearable and environmental sensors data. In _International Conference on Mobile and Ubiquitous Systems: Computing, Networking, and Services_. Springer, 451–468.
  * Arrotta et al. (2024a) Luca Arrotta, Claudio Bettini, Gabriele Civitarese, and Michele Fiori. 2024a.  ContextGPT: Infusing LLMs Knowledge into Neuro-Symbolic Activity Recognition Models.  _2024 IEEE SMARTCOMP_ (2024).
  * Arrotta et al. (2024b) Luca Arrotta, Gabriele Civitarese, and Claudio Bettini. 2024b.  Semantic Loss: A New Neuro-Symbolic Approach for Context-Aware Human Activity Recognition.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 7, 4 (2024), 1–29.
  * Babangida et al. (2022) Lawal Babangida, Thinagaran Perumal, Norwati Mustapha, and Razali Yaakob. 2022.  Internet of things (IoT) based activity recognition strategies in smart homes: A review.  _IEEE Sensors Journal_ 22, 9 (2022), 8327–8336.
  * Bettini et al. (2020) Claudio Bettini, Gabriele Civitarese, and Riccardo Presotto. 2020.  Caviar: Context-driven active and incremental activity recognition.  _Knowledge-Based Systems_ 196 (2020), 105816.
  * Bettini and Riboni (2015) Claudio Bettini and Daniele Riboni. 2015.  Privacy protection in pervasive systems: State of the art and technical challenges.  _Pervasive and Mobile Computing_ 17 (2015), 159–174.
  * Bishop et al. (2010) Nicholas A Bishop, Tao Lu, and Bruce A Yankner. 2010.  Neural mechanisms of ageing and cognitive decline.  _Nature_ 464, 7288 (2010), 529–535.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chan Chang et al. (2021) S Chan Chang, R Walmsley, J Gershuny, T Harms, E Thomas, K Milton, P Kelly, C Foster, A Wong, N Gray, et al. 2021\.  Capture-24: Activity tracker dataset for human activity recognition.  _University of Oxford_ (2021).
  * Chen et al. (2023) Chuanrui Chen, Shichao Ding, and Joseph Wang. 2023.  Digital health for aging populations.  _Nature medicine_ 29, 7 (2023), 1623–1630.
  * Chen et al. (2021) Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu. 2021.  Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities.  _ACM Computing Surveys (CSUR)_ 54, 4 (2021), 1–40.
  * Chen and Nugent (2009) Liming Chen and Chris Nugent. 2009.  Ontology-based activity recognition in intelligent pervasive environments.  _International Journal of Web Information Systems_ 5, 4 (2009), 410–430.
  * Civitarese et al. (2019) Gabriele Civitarese, Timo Sztyler, Daniele Riboni, Claudio Bettini, and Heiner Stuckenschmidt. 2019.  POLARIS: Probabilistic and ontological activity recognition in smart-homes.  _IEEE Transactions on Knowledge and Data Engineering_ 33, 1 (2019), 209–223.
  * Cook et al. (2012) Diane J Cook, Aaron S Crandall, Brian L Thomas, and Narayanan C Krishnan. 2012.  CASAS: A smart home in a box.  _Computer_ 46, 7 (2012), 62–69.
  * Dhekane and Ploetz (2024) Sourish Gunesh Dhekane and Thomas Ploetz. 2024.  Transfer Learning in Human Activity Recognition: A Survey.  _arXiv preprint arXiv:2401.10185_ (2024).
  * Fang et al. (2024) Meng Fang, Shilong Deng, Yudi Zhang, Zijing Shi, Ling Chen, Mykola Pechenizkiy, and Jun Wang. 2024.  Large language models are neurosymbolic reasoners.  _arXiv preprint arXiv:2401.09334_ (2024).
  * Gao et al. (2024) Jiayuan Gao, Yingwei Zhang, Yiqiang Chen, Tengxiang Zhang, Boshi Tang, and Xiaoyu Wang. 2024.  Unsupervised Human Activity Recognition Via Large Language Models and Iterative Evolution. In _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 91–95.
  * Grammatikopoulou et al. (2024) Margarita Grammatikopoulou, Ioulietta Lazarou, Vasilis Alepopoulos, Lampros Mpaltadoros, Vangelis P Oikonomou, Thanos G Stavropoulos, Spiros Nikolopoulos, Ioannis Kompatsiaris, and Magda Tsolaki. 2024.  Assessing the cognitive decline of people in the spectrum of AD by monitoring their activities of daily living in an IoT-enabled smart home environment: a cross-sectional pilot study.  _Frontiers in Aging Neuroscience_ 16 (2024), 1375131.
  * Graule and Isler (2023) Moritz A Graule and Volkan Isler. 2023.  GG-LLM: Geometrically Grounding Large Language Models for Zero-shot Human Activity Forecasting in Human-Aware Task Planning.  _arXiv preprint arXiv:2310.20034_ (2023).
  * Haresamudram et al. (2022) Harish Haresamudram, Irfan Essa, and Thomas Plötz. 2022.  Assessing the state of self-supervised human activity recognition using wearables.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 6, 3 (2022), 1–47.
  * Hiremath et al. (2022) Shruthi K Hiremath, Yasutaka Nishimura, Sonia Chernova, and Thomas Plötz. 2022.  Bootstrapping Human Activity Recognition Systems for Smart Homes from Scratch.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 6, 3 (2022), 1–27.
  * Hitzler and Sarker (2022) Pascal Hitzler and Md Kamruzzaman Sarker. 2022.  Neuro-symbolic artificial intelligence: The state of the art.  _IOS press_ (2022).
  * Hota et al. (2024) Aritra Hota, Soumyajit Chatterjee, and Sandip Chakraborty. 2024.  Evaluating Large Language Models as Virtual Annotators for Time-series Physical Sensing Data.  _arXiv preprint arXiv:2403.01133_ (2024).
  * Hudon et al. (2020) Carol Hudon, Frederique Escudier, Jessie De Roy, Jordie Croteau, Nathan Cross, Thien Thanh Dang-Vu, Herve Tchala Vignon Zomahoun, Sebastien Grenier, Jean-Francois Gagnon, Andreanne Parent, et al. 2020\.  Behavioral and psychological symptoms that predict cognitive decline or impairment in cognitively normal middle-aged or older adults: a meta-analysis.  _Neuropsychology Review_ 30 (2020), 558–579.
  * Jain et al. (2022) Yash Jain, Chi Ian Tang, Chulhong Min, Fahim Kawsar, and Akhil Mathur. 2022.  ColloSSL: Collaborative Self-Supervised Learning for Human Activity Recognition.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 6, 1 (2022), 1–28.
  * Jefferson et al. (2008) Angela L Jefferson, Laura K Byerly, Susan Vanderhill, Susan Lambe, Sarah Wong, Al Ozonoff, and Jason H Karlawish. 2008.  Characterization of activities of daily living in individuals with mild cognitive impairment.  _The American journal of geriatric psychiatry_ 16, 5 (2008), 375–383.
  * Ji et al. (2024) Sijie Ji, Xinzhe Zheng, and Chenshu Wu. 2024.  HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?  _arXiv preprint arXiv:2403.02727_ (2024).
  * Khodabandehloo et al. (2021) Elham Khodabandehloo, Daniele Riboni, and Abbas Alimohammadi. 2021.  HealthXAI: Collaborative and explainable AI for supporting early diagnosis of cognitive decline.  _Future Generation Computer Systems_ 116 (2021), 168–189.
  * Leng et al. (2024) Zikang Leng, Amitrajit Bhattacharjee, Hrudhai Rajasekhar, Lizhe Zhang, Elizabeth Bruda, Hyeokhyen Kwon, and Thomas Plötz. 2024.  IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition.  _arXiv preprint arXiv:2402.01049_ (2024).
  * Liciotti et al. (2020) Daniele Liciotti, Michele Bernardini, Luca Romeo, and Emanuele Frontoni. 2020.  A sequential deep learning application for recognising human activities in smart homes.  _Neurocomputing_ 396 (2020), 501–513.
  * Liu et al. (2023) Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer-Levy, Jacob Sunshine, Jiening Zhan, Ming-Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel. 2023.  Large language models are few-shot health learners.  _arXiv preprint arXiv:2305.15525_ (2023).
  * Okita et al. (2023) Tsuyoshi Okita, Kosuke Ukita, Koki Matsuishi, Masaharu Kagiyama, Kodai Hirata, and Asahi Miyazaki. 2023.  Towards LLMs for Sensor Data: Multi-Task Self-Supervised Learning. In _Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing_. 499–504.
  * Ordónez et al. (2013) Fco Javier Ordónez, Paula De Toledo, and Araceli Sanchis. 2013.  Activity recognition using hybrid generative/discriminative models on home environments using binary sensors.  _Sensors_ 13, 5 (2013), 5460–5477.
  * Ordóñez and Roggen (2016) Francisco Javier Ordóñez and Daniel Roggen. 2016.  Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition.  _Sensors_ 16, 1 (2016), 115.
  * Rashidi and Mihailidis (2012) Parisa Rashidi and Alex Mihailidis. 2012.  A survey on ambient-assisted living tools for older adults.  _IEEE journal of biomedical and health informatics_ 17, 3 (2012), 579–590.
  * Riboni et al. (2016) Daniele Riboni, Claudio Bettini, Gabriele Civitarese, Zaffar Haider Janjua, and Rim Helaoui. 2016.  SmartFABER: Recognizing fine-grained abnormal behaviors for early detection of mild cognitive impairment.  _Artificial intelligence in medicine_ 67 (2016), 57–74.
  * Sanabria et al. (2021) Andrea Rosales Sanabria, Franco Zambonelli, and Juan Ye. 2021.  Unsupervised Domain Adaptation in Activity Recognition: A GAN-Based Approach.  _IEEE Access_ 9 (2021), 19421–19438.
  * Soleimani and Nazerfard (2021) Elnaz Soleimani and Ehsan Nazerfard. 2021.  Cross-subject transfer learning in human activity recognition systems using generative adversarial networks.  _Neurocomputing_ 426 (2021), 26–34.
  * Takeda et al. (2023) Naoto Takeda, Roberto Legaspi, Yasutaka Nishimura, Kazushi Ikeda, Atsunori Minamikawa, Thomas Plötz, and Sonia Chernova. 2023.  Sensor Event Sequence Prediction for Proactive Smart Home Support Using Autoregressive Language Model. In _2023 19th International Conference on Intelligent Environments (IE)_. IEEE, 1–8.
  * Thukral et al. (2024) Megha Thukral, Sourish Gunesh Dhekane, Shruthi K Hiremath, Harish Haresamudram, and Thomas Ploetz. 2024.  Layout Agnostic Human Activity Recognition in Smart Homes through Textual Descriptions Of Sensor Triggers (TDOST).  _arXiv preprint arXiv:2405.12368_ (2024).
  * Van Kasteren et al. (2008) Tim Van Kasteren, Athanasios Noulas, Gwenn Englebienne, and Ben Kröse. 2008.  Accurate activity recognition in a home setting. In _Proceedings of the 10th international conference on Ubiquitous computing_. 1–9.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022\.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ 35 (2022), 24824–24837.
  * Xia et al. (2023) Qingxin Xia, Takuya Maekawa, and Takahiro Hara. 2023.  Unsupervised Human Activity Recognition through Two-stage Prompting with ChatGPT.  _arXiv preprint arXiv:2306.02140_ (2023).
  * Xing et al. (2020) Tianwei Xing, Luis Garcia, Marc Roig Vilamala, Federico Cerutti, Lance Kaplan, Alun Preece, and Mani Srivastava. 2020.  Neuroplex: learning to detect complex events in sensor networks through knowledge injection. In _Proceedings of the 18th conference on embedded networked sensor systems_. 489–502.
  * Yuan et al. (2024) Hang Yuan, Shing Chan, Andrew P Creagh, Catherine Tong, Aidan Acquah, David A Clifton, and Aiden Doherty. 2024.  Self-supervised learning for human activity recognition using 700,000 person-days of wearable data.  _NPJ digital medicine_ 7, 1 (2024), 91.
  * Zhang et al. (2023) Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. 2023.  T2m-gpt: Generating human motion from textual descriptions with discrete representations.  _arXiv preprint arXiv:2301.06052_ (2023).
  * Zhou et al. (2023) Yunjiao Zhou, Jianfei Yang, Han Zou, and Lihua Xie. 2023.  TENT: Connect Language Models with IoT Sensors for Zero-Shot Activity Recognition.  _arXiv preprint arXiv:2311.08245_ (2023).

##  Appendix A Detailed results at different data availability scenarios

In the following, we report the results of few-shot prompting (and the
baselines) at different data availability scenarios: $100\%$, $50\%$, $25\%$,
$10\%$ of the training set.

Table 7. UCI ADL: Few-Shot ADL-LLM vs the supervised baselines ($100\%$ of the training set) | Few-Shot ADL-LLM | Supervised baselines | Few-Shot-Prompting | Supervised baselines
---|---|---|---|---
Activity | K=3 | K=5 | K=7 | BiLSTM |  | CascEns
---
BiLSTM
| DeepConv
---
LSTM
K=3 | K=5 | K=7 | BiLSTM |  | CascEns
---
BiLSTM
| DeepConv
---
LSTM
Leaving home | 1.00 | 1.00 | 1.00 | 0.95 | 0.92 | 0.99 | 0.90 | 0.90 | 0.90 | 0.84 | 0.81 | 0.88
Personal care | 1.00 | 1.00 | 0.99 | 0.98 | 0.97 | 0.99 | 0.95 | 0.96 | 0.96 | 0.92 | 0.92 | 0.93
Preparing breakfast | 0.81 | 0.89 | 0.94 | 0.90 | 0.90 | 0.94 | 0.71 | 0.83 | 0.81 | 0.35 | 0.35 | 0.39
Preparing dinner | - | - | - | - | - | - | 0.29 | 0.39 | 0.40 | 0.06 | 0.13 | 0.13
Preparing lunch | 0.94 | 0.96 | 0.97 | 0.95 | 0.95 | 0.96 | 0.38 | 0.45 | 0.50 | 0.35 | 0.36 | 0.41
Relaxing on couch | 0.86 | 0.87 | 0.87 | 1.00 | 1.00 | 1.00 | 0.61 | 0.64 | 0.71 | 0.85 | 0.85 | 0.87
Showering | 1.00 | 1.00 | 1.00 | 0.99 | 0.99 | 1.00 | 0.92 | 0.94 | 0.95 | 0.93 | 0.93 | 0.96
Sleeping | 0.89 | 0.90 | 0.90 | 0.99 | 0.99 | 1.00 | 0.84 | 0.82 | 0.89 | 0.99 | 0.99 | 0.99
Snacking | 0.41 | 0.54 | 0.60 | 0.04 | 0.07 | 0.00 | 0.35 | 0.39 | 0.37 | 0.37 | 0.34 | 0.40
Weighted F1-score | 0.93 | 0.95 | 0.95 | 0.96 | 0.96 | 0.97 | 0.73 | 0.76 | 0.78 | 0.72 | 0.72 | 0.75
Table 8. MARBLE: Few-Shot ADL-LLM vs the supervised baselines ($100\%$ of the training set) | Few-Shot ADL-LLM | Supervised baselines
---|---|---
Activity | K=3 | K=5 | K=7 | BiLSTM |  | CascEns
---
BiLSTM
| DeepConv
---
LSTM
Clearing table | 0.51 | 0.62 | 0.56 | 0.44 | 0.42 | 0.55
Eating | 0.88 | 0.87 | 0.88 | 0.91 | 0.91 | 0.91
Entering home | 0.69 | 0.65 | 0.73 | 0.00 | 0.00 | 0.13
Leaving home | 0.77 | 0.77 | 0.77 | 0.00 | 0.00 | 0.00
Phone call | 0.99 | 0.99 | 0.99 | 0.76 | 0.76 | 0.81
Preparing cold meal | 0.68 | 0.69 | 0.71 | 0.63 | 0.62 | 0.73
Preparing hot meal | 0.81 | 0.86 | 0.86 | 0.82 | 0.82 | 0.90
Setting up table | 0.44 | 0.50 | 0.52 | 0.55 | 0.48 | 0.57
Taking medicines | 0.59 | 0.60 | 0.58 | 0.54 | 0.53 | 0.59
Using PC | 0.98 | 0.99 | 0.99 | 0.91 | 0.91 | 0.91
Watching TV | 0.99 | 0.99 | 0.99 | 0.94 | 0.94 | 0.96
Weighted F1-score | 0.85 | 0.87 | 0.87 | 0.78 | 0.77 | 0.82
Table 9. UCI ADL: Few-Shot ADL-LLM vs the supervised baselines ($50\%$ of the training set) | Few-Shot ADL-LLM | Supervised baselines | Few-Shot ADL-LLM | Supervised baselines
---|---|---|---|---
Activity | K=3 | K=5 | K=7 | BiLSTM |  | CascEns
---
BiLSTM
| DeepConv
---
LSTM
K=3 | K=5 | K=7 | BiLSTM |  | CascEns
---
BiLSTM
| DeepConv
---
LSTM
Leaving home | 1.00 | 1.00 | 1.00 | 0.93 | 0.83 | 0.99 | 0.89 | 0.88 | 0.89 | 0.73 | 0.67 | 0.86
Personal care | 0.99 | 1.00 | 1.00 | 0.96 | 0.95 | 0.99 | 0.93 | 0.95 | 0.96 | 0.92 | 0.91 | 0.93
Preparing breakfast | 0.76 | 0.86 | 0.91 | 0.88 | 0.87 | 0.92 | 0.80 | 0.82 | 0.83 | 0.36 | 0.35 | 0.38
Preparing dinner | - | - | - | - | - | - | 0.39 | 0.40 | 0.39 | 0.05 | 0.05 | 0.14
Preparing lunch | 0.91 | 0.95 | 0.97 | 0.95 | 0.94 | 0.96 | 0.39 | 0.43 | 0.44 | 0.32 | 0.29 | 0.39
Relaxing on couch | 0.83 | 0.93 | 0.98 | 1.00 | 0.99 | 0.99 | 0.64 | 0.64 | 0.65 | 0.84 | 0.85 | 0.87
Showering | 0.99 | 1.00 | 1.00 | 0.98 | 0.96 | 0.99 | 0.95 | 0.95 | 0.96 | 0.92 | 0.92 | 0.95
Sleeping | 0.90 | 0.94 | 0.98 | 0.99 | 0.99 | 1.00 | 0.81 | 0.85 | 0.85 | 0.99 | 0.99 | 0.99
Snacking | 0.36 | 0.56 | 0.69 | 0.04 | 0.15 | 0.04 | 0.38 | 0.34 | 0.35 | 0.25 | 0.28 | 0.38
Weighted F1-score | 0.91 | 0.95 | 0.97 | 0.95 | 0.94 | 0.97 | 0.75 | 0.76 | 0.76 | 0.70 | 0.69 | 0.74
Table 10. UCI ADL: Few-Shot ADL-LLM vs the supervised baselines ($50\%$ of the training set) | Few-Shot-Prompting | Supervised baselines
---|---|---
Activity | K=3 | K=5 | K=7 | BiLSTM |  | CascEns
---
BiLSTM
| DeepConv
---
LSTM
Clearing table | 0.44 | 0.49 | 0.57 | 0.29 | 0.26 | 0.46
Eating | 0.87 | 0.90 | 0.91 | 0.91 | 0.91 | 0.91
Entering home | 0.60 | 0.57 | 0.57 | 0.00 | 0.00 | 0.16
Leaving home | 0.80 | 0.75 | 0.86 | 0.00 | 0.00 | 0.39
Phone call | 0.99 | 0.99 | 0.99 | 0.76 | 0.76 | 0.81
Preparing cold meal | 0.67 | 0.64 | 0.69 | 0.61 | 0.62 | 0.71
Preparing hot meal | 0.84 | 0.83 | 0.85 | 0.83 | 0.82 | 0.89
Setting up table | 0.37 | 0.52 | 0.47 | 0.52 | 0.50 | 0.62
Taking medicines | 0.42 | 0.43 | 0.42 | 0.53 | 0.51 | 0.58
Using PC | 0.99 | 0.99 | 0.99 | 0.91 | 0.91 | 0.90
Watching TV | 0.99 | 0.99 | 0.99 | 0.95 | 0.95 | 0.97
Weighted F1-score | 0.84 | 0.85 | 0.86 | 0.77 | 0.77 | 0.82
Table 11. UCI ADL: Few-Shot ADL-LLM vs the supervised baselines ($25\%$ of the training set) | Few-Shot ADL-LLM | Supervised baselines | Few-Shot ADL-LLM | Supervised baselines
---|---|---|---|---
Activity | K=3 | K=5 | K=7 | BiLSTM |  | CascEns
---
BiLSTM
| DeepConv
---
LSTM
K=3 | K=5 | K=7 | BiLSTM |  | CascEns
---
BiLSTM
| DeepConv
---
LSTM
Leaving home | 1.00 | 1.00 | 1.00 | 0.79 | 0.78 | 0.98 | 0.86 | 0.88 | 0.88 | 0.64 | 0.53 | 0.84
Personal care | 0.99 | 0.99 | 1.00 | 0.96 | 0.94 | 0.99 | 0.95 | 0.95 | 0.96 | 0.90 | 0.90 | 0.94
Preparing breakfast | 0.77 | 0.82 | 0.86 | 0.85 | 0.84 | 0.91 | 0.73 | 0.78 | 0.78 | 0.36 | 0.33 | 0.38
Preparing dinner | - | - | - | - | - | - | 0.33 | 0.37 | 0.42 | 0.04 | 0.09 | 0.25
Preparing lunch | 0.92 | 0.95 | 0.95 | 0.94 | 0.94 | 0.95 | 0.39 | 0.43 | 0.44 | 0.16 | 0.19 | 0.18
Relaxing on couch | 0.85 | 0.95 | 0.98 | 1.00 | 0.99 | 0.99 | 0.67 | 0.67 | 0.71 | 0.84 | 0.84 | 0.86
Showering | 0.99 | 1.00 | 1.00 | 0.98 | 0.95 | 1.00 | 0.95 | 0.94 | 0.96 | 0.90 | 0.92 | 0.95
Sleeping | 0.89 | 0.96 | 0.99 | 0.99 | 0.99 | 1.00 | 0.88 | 0.86 | 0.88 | 0.98 | 0.98 | 0.99
Snacking | 0.37 | 0.58 | 0.61 | 0.02 | 0.00 | 0.00 | 0.36 | 0.35 | 0.34 | 0.29 | 0.27 | 0.44
Weighted F1-score | 0.91 | 0.95 | 0.97 | 0.94 | 0.93 | 0.97 | 0.75 | 0.76 | 0.77 | 0.67 | 0.66 | 0.74
Table 12. MARBLE: Few-Shot ADL-LLM vs the supervised baselines ($25\%$ of the training set) | Few-Shot ADL-LLM | Supervised baselines
---|---|---
Activity | K=3 | K=5 | K=7 | BiLSTM |  | CascEns
---
BiLSTM
| DeepConv
---
LSTM
Clearing table | 0.39 | 0.44 | 0.41 | 0.30 | 0.28 | 0.49
Eating | 0.89 | 0.89 | 0.89 | 0.91 | 0.91 | 0.90
Entering home | 0.32 | 0.59 | 0.48 | 0.00 | 0.00 | 0.00
Leaving home | 0.88 | 0.88 | 0.93 | 0.00 | 0.00 | 0.14
Phone call | 0.98 | 0.99 | 0.99 | 0.71 | 0.69 | 0.78
Preparing cold meal | 0.64 | 0.60 | 0.57 | 0.61 | 0.59 | 0.69
Preparing hot meal | 0.78 | 0.81 | 0.80 | 0.79 | 0.77 | 0.86
Setting up table | 0.46 | 0.47 | 0.40 | 0.36 | 0.35 | 0.58
Taking medicines | 0.49 | 0.50 | 0.47 | 0.45 | 0.30 | 0.54
Using pc | 0.98 | 0.98 | 0.99 | 0.91 | 0.90 | 0.91
Watching tv | 0.99 | 0.99 | 0.98 | 0.94 | 0.94 | 0.95
Weighted F1-score | 0.83 | 0.84 | 0.83 | 0.74 | 0.72 | 0.80
Table 13. UCI ADL: Few-Shot ADL-LLM vs the supervised baselines ($10\%$ of the training set) | Few-Shot ADL-LLM | Supervised baselines | Few-Shot ADL-LLM | Supervised baselines
---|---|---|---|---
Activity | K=3 | K=5 | K=7 | BiLSTM |  | CascEns
---
BiLSTM
| DeepConv
---
LSTM
K=3 | K=5 | K=7 | BiLSTM |  | CascEns
---
BiLSTM
| DeepConv
---
LSTM
Leaving home | 1.00 | 1.00 | 1.00 | 0.71 | 0.73 | 0.97 | 0.87 | 0.87 | 0.88 | 0.49 | 0.41 | 0.88
Personal care | 0.99 | 0.99 | 0.99 | 0.92 | 0.93 | 0.98 | 0.96 | 0.96 | 0.96 | 0.88 | 0.85 | 0.94
Preparing breakfast | 0.70 | 0.88 | 0.87 | 0.80 | 0.82 | 0.91 | 0.73 | 0.79 | 0.78 | 0.19 | 0.07 | 0.33
Preparing dinner | - | - | - | - | - | - | 0.29 | 0.33 | 0.36 | 0.04 | 0.04 | 0.09
Preparing lunch | 0.92 | 0.96 | 0.96 | 0.93 | 0.94 | 0.96 | 0.32 | 0.41 | 0.43 | 0.18 | 0.09 | 0.08
Relaxing on couch | 0.85 | 0.91 | 0.96 | 1.00 | 0.99 | 0.99 | 0.61 | 0.67 | 0.71 | 0.84 | 0.83 | 0.85
Showering | 0.99 | 1.00 | 1.00 | 0.93 | 0.94 | 0.98 | 0.95 | 0.95 | 0.96 | 0.87 | 0.88 | 0.95
Sleeping | 0.92 | 0.94 | 0.97 | 0.97 | 0.97 | 1.00 | 0.86 | 0.87 | 0.92 | 0.86 | 0.84 | 0.99
Snacking | 0.27 | 0.48 | 0.57 | 0.09 | 0.00 | 0.31 | 0.34 | 0.36 | 0.38 | 0.29 | 0.27 | 0.46
Weighted F1-score | 0.91 | 0.95 | 0.96 | 0.91 | 0.92 | 0.97 | 0.74 | 0.76 | 0.78 | 0.63 | 0.59 | 0.72
Table 14. MARBLE: Few-Shot ADL-LLM vs the supervised baselines ($10\%$ of the training set) | Few-Shot ADL-LLM | Supervised baselines
---|---|---
Activity | K=3 | K=5 | K=7 | BiLSTM |  | CascEns
---
BiLSTM
| DeepConv
---
LSTM
Clearing table | 0.47 | 0.28 | 0.33 | 0.00 | 0.00 | 0.46
Eating | 0.87 | 0.84 | 0.85 | 0.80 | 0.78 | 0.90
Entering home | 0.65 | 0.69 | 0.61 | 0.00 | 0.00 | 0.00
Leaving home | 0.71 | 0.86 | 0.80 | 0.00 | 0.00 | 0.00
Phone call | 0.99 | 0.99 | 0.99 | 0.63 | 0.61 | 0.79
Preparing cold meal | 0.64 | 0.63 | 0.65 | 0.43 | 0.38 | 0.58
Preparing hot meal | 0.80 | 0.83 | 0.87 | 0.66 | 0.64 | 0.86
Setting up table | 0.27 | 0.38 | 0.45 | 0.00 | 0.00 | 0.08
Taking medicines | 0.26 | 0.40 | 0.38 | 0.06 | 0.18 | 0.35
Using pc | 0.97 | 0.98 | 0.98 | 0.81 | 0.81 | 0.91
Watching tv | 0.98 | 0.99 | 0.99 | 0.73 | 0.74 | 0.96
Weighted F1-score | 0.82 | 0.82 | 0.84 | 0.58 | 0.57 | 0.76