# PAPER-HILT: Personalized and Adaptive Privacy-Aware Early-Exit for
Reinforcement Learning in Human-in-the-Loop Systems

Mojtaba Taherisadr<EMAIL_ADDRESS>and  Salma Elmalaki
<EMAIL_ADDRESS>University of California,
IrvineIrvineCaliforniaUSA92617

###### Abstract.

Reinforcement Learning (RL) has increasingly become a preferred method over
traditional rule-based systems in diverse human-in-the-loop (HITL)
applications due to its adaptability to the dynamic nature of human
interactions. However, integrating RL in such settings raises significant
privacy concerns, as it might inadvertently expose sensitive user information.
Addressing this, our paper focuses on developing PAPER-HILT, an innovative,
adaptive RL strategy through exploiting an early-exit approach designed
explicitly for privacy preservation in HITL environments. This approach
dynamically adjusts the tradeoff between privacy protection and system
utility, tailoring its operation to individual behavioral patterns and
preferences. We mainly highlight the challenge of dealing with the variable
and evolving nature of human behavior, which renders static privacy models
ineffective. PAPER-HILT’s effectiveness is evaluated through its application
in two distinct contexts: Smart Home environments and Virtual Reality (VR)
Smart Classrooms. The empirical results demonstrate PAPER-HILT’s capability to
provide a personalized equilibrium between user privacy and application
utility, adapting effectively to individual user needs and preferences. On
average for both experiments, utility (performance) drops by $24$%, and
privacy (state prediction) improves by $31$%.

Reiforcement learning, early-exit, privacy, human-in-the-loop

††copyright: none††ccs: Human-centered computing††ccs: Security and privacy
Privacy protections††ccs: Computing methodologies Reinforcement learning

##  1\. Introduction

The rapid evolution of sensor networks and mobile computing has given us
unprecedented capabilities to observe human behavior and their interaction
with their environment, paving the way for developing human-aware Internet of
Things (IoT) applications (Elmalaki et al., 2018a). This synergy between human
behavior and technology is revolutionizing the design of IoT applications,
leading to the emergence of Human-in-the-Loop (HITL) IoT systems (Cambeiro,
2019). By continually sensing and adapting to both the environment and human
inputs, these systems offer a novel paradigm in IoT, promising highly
personalized user experiences in these advanced technologies.

However, this intimate integration of human behavior with IoT systems
introduces significant privacy challenges. The diversity in human behavior
patterns, routines, and preferences, which greatly influence the data
generated, can lead to varied levels of privacy vulnerabilities. In
particular, this behavioral diversity leads to distinct privacy implications,
as certain individuals’ behavior may inadvertently leak sensitive information
in some cases more than in others (Taherisadr et al., 2023b; Caine, 2009).
Accordingly, when designing AI algorithms in HITL IoT systems, it is crucial
to recognize that privacy considerations are not static but can dynamically
evolve due to shifts in human behavior. Recognizing these nuances and
tailoring privacy mitigation algorithms accordingly becomes essential for
ensuring the responsible utilization of HITL IoT technologies. This approach
to privacy mitigation acknowledges that privacy is not a one-size-fits-all
concept (Bass, 2019; Io, 2011).

In addressing these challenges, this paper focuses on mitigating privacy leaks
in machine learning-based HITL IoT systems. We explore the use of privacy-
aware Deep Reinforcement Learning (DRL), an approach that has shown promise in
adapting to human intentions and responses (Sadigh et al., 2017; Hadfield-
Menell et al., 2016). Our focus on RL stems from its adoption in methods such
as Multisample RL and adaptive scaling RL (ADAS-RL) that can accommodate
variations in both inter- and intra-human variability in behaviors, adapting
to a wide range of automated actions (Elmalaki et al., 2018b; Ahadi-Sarkani
and Elmalaki, 2021). Furthermore, recent research has highlighted the
potential of RL models to address concerns related to equity, fairness, risk
sensitivity, safety, and human diversity (Elmalaki, 2021; Zhao et al., 2023;
Eslinger and Damasio, 1985; Hans et al., 2008; Elmalaki et al., 2018b).

DRL’s ability to personalize IoT systems for individual experiences is well-
documented, yet it also raises privacy concerns due to its potential to access
private human states (Omonkhoa, 2021). This issue is particularly pertinent in
domains like healthcare, finance, and autonomous vehicles, where the
continuous use of sensitive user data poses significant privacy concerns
(Hlávka, 2020). Therefore, balancing the need for personalized HITL IoT
experiences with protecting individual privacy is a critical challenge.

Previous research has explored privacy vulnerabilities in learning-based RL
systems, often focusing on the privacy-utility tradeoff. However, the unique
context of HITL IoT systems presents distinct challenges. Recognizing the
inherent variability among human subjects, we propose PAPER-HILT, an adaptive
personalized DRL approach that leverages an early-exit strategy. This approach
aims to tailor the privacy-utility tradeoff to individual variability in HITL
IoT.

##  2\. Related Work and Paper Contribution

Privacy concerns have persisted for many years (Petrescu and Krishen, 2018),
prompting extensive research to address privacy breaches and their mitigation
through various strategies. Hong et al. (Hong et al., 2022) addressed the
imperative of balancing privacy and model performance through differentially
private learning. Their work introduces dynamic policies, including
adjustments to model parameters. These dynamic techniques enhance the
performance of the models within privacy constraints. Game-theoretical methods
have been employed to create an objective function that maximizes utility
while minimizing privacy vulnerabilities (Jin et al., 2017). Additionally,
data encryption has been proposed as a defense against side-channel attacks on
communication links between edge and cloud services (Mishra et al., 2022).
Zhang et al. (Zhang et al., 2022) tackle training data privacy in machine
learning, shifting focus to attribute privacy and concealing sensitive dataset
properties during analysis. They propose definitions and efficient mechanisms
for safeguarding specific dataset properties and distribution parameters,
laying the foundation for addressing attribute privacy in preserving global
properties during analysis. Patel et al. (Patel et al., 2022) address
information leakage in model explanations for algorithmic transparency. They
introduce differentially private mechanisms, balancing transparency and data
privacy in machine learning.

Our emphasis on using reinforcement learning (RL) methods is driven by two
critical attributes:

  * •

Computational Complexity and Scalability: RL offers computational scalability
advantages compared to other techniques, especially game-theoretic approaches.
Many game-theoretic methods for sequential decision-making are recognized as
computationally infeasible (Fraenkel, 2004).

  * •

Generalizability: RL possesses a unique capacity to directly model the
consequences of decisions, leverage temporal feedback during the learning
process, and enhance decision-making policy performance across a broad
spectrum of systems. This attribute is particularly valuable for Human-in-the-
Loop (HITL) systems.

###  2.1. Related Work

####  2.1.1. Privacy-preserving RL

Several facets of privacy-preserving RL challenges have been explored and
addressed, including online learning with bandit feedback (Malekzadeh et al.,
2020), linear contextual bandits (Garcelon et al., 2022), and deep
reinforcement learning (DRL) (Pan et al., 2019). In the tabular setting,
Garcelon et al. (Garcelon et al., 2021) introduced an algorithm that ensures
both regret minimization and privacy preservation. In continuous state
scenarios, Wang et al. (Wang and Hegde, 2019) devised a Q-learning variant
capable of discovering a policy that adheres to differential privacy
constraints while optimizing the reward function. Erdemir et al. (Erdemir et
al., 2020) studied the trade-off between privacy and utility (PUT) in this
context, noting that existing approaches often overlook temporal correlations
in time-series data. They proposed sharing a distorted version of the user
data sequence to mitigate privacy risks. Their approach quantifies privacy
risk through mutual information assessment and employs asynchronous actor-
critic deep reinforcement learning (RL) to minimize history-dependent mutual
information. adaPARL (Taherisadr et al., 2023b) examined the privacy leaks
inherent in sequential decision-making setups for HITL IoT, where interactions
with humans introduce variability in behavior and preferences over time. It
introduces an adaptive privacy-aware algorithm for such systems, which adjusts
to varying human behaviors. adaPARL is designed based on Q-learning and is
typically used for discrete state and action spaces. The drawback of limited
state space encourages exploring more advanced RL algorithms.

####  2.1.2. Early Exit in HILT IoT

The concept of Early Exit (EE) was initially proposed by Teerapittayanon et
al. (Teerapittayanon et al., 2016) to eliminate the necessity of reducing the
size of deep neural network (DNN) models. This is achieved by introducing
early exits within the DNN architecture, which enables the termination of
execution as soon as a desired confidence level is reached for a given input
sample. In practical terms, certain samples in test datasets and real-world
scenarios can be readily processed by a DNN model. In contrast, others might
pose challenges, depending on the specific machine learning models employed.
Consequently, EE optimizes the inference process for straightforward samples
by reducing the number of transformation layers involved, thus reducing
overall inference time and computational costs (Matsubara et al., 2022). Some
work in the literature examined the effects of EE mechanism in IoT and
federated learning scenarios, aiming to enhance the robustness of training and
inference against privacy attacks, such as Membership Inference Attacks (MIAs)
(Wu et al., 2023). EE-equipped models in dynamic neural networks adjust their
computation path for each sample based on its complexity. The study employs
threshold and logistic regression attacks to measure privacy, revealing that
EE techniques improve privacy preservation. Through experiments, they
determined that models lacking early exits are more susceptible to MIAs. The
vulnerability of Convolutional Neural Network (CNN), AlexNet, and ResNet18
models to these attacks is assessed in both centralized and federated learning
settings. The findings highlight the potential of dynamic architectures in
enhancing privacy, underscoring the need for further exploration and design in
this model category (Wu et al., 2023).

While EE strategies have been extensively explored in the context of DNN,
their application in RL settings offers a novel perspective on improving the
efficiency of learning algorithms. The concept of terminating an episode
prematurely, based on intermediate observations or partial trajectories, holds
promise in accelerating convergence and optimizing resource utilization.
RAPID-RL (Kosta et al., 2022) introduces a novel approach to enhance deep RL.
RAPID-RL framework enhances a standard $DQN$ with multiple exit branches,
enabling the rapid inference of most input states and leading to quicker
processing and reduced compute operations. Additionally, they present a
sequential training method for these branches, ensuring optimal performance
for each while circumventing performance interdependence and creating a
unified joint $DQN$ adhering to power constraints. In this paper, we exploit
EE for $DQN$ and explore its efficiency as a privacy mitigation technique in
HITL IoT systems with human variability.

###  2.2. Paper Contribution

Our contributions can be summarized as follows.

  1. (1)

PAPER-HILT: We propose an algorithm for personalized privacy-aware HITL
systems with human behavior variability. This algorithm is based on DRL with
EE and is tailored for sequential decision-making systems.

  2. (2)

Personalization: We offer the capability to personalize the tradeoff between
privacy mitigation and the application’s utility. It is designed to adaptively
mitigate privacy leaks based on the variability in human behavior. 3.
Versatility and adaptability: We introduce general design parameters within
our proposed algorithm that can be adjusted to suit the requirements of
various application domains.

  3. (3)

Simulation and real-world applications: We implement our proposed algorithm in
two HITL systems: a simulated smart home environment and a real-world smart
classroom with VR technology.

The paper is organized as follows: Section 3 reviews the deep reinforcement
learning for the Human-in-the-Loop systems. Then, we structure the threat
model for the HITL IoT technologies in Section 4. The paper’s subsequent
sections focus on the PAPER-HILT algorithm and then evaluate the proposed
PAPER-HILT algorithm on two different applications.

##  3\. Deep Reinforcement Learning ($DQN$) for Human-in-the-Loop applications

Deep Q-Networks ($DQN$), a reinforcement learning paradigm, integrates deep
neural networks with Q-learning, operating within the framework of Markov
Decision Processes (MDP). The crux of $DQN$ lies in estimating the action-
value function $Q(s,a)$, which encapsulates the expected rewards for actions
within specific states. This function is typically represented by a neural
network with state inputs and estimated action values as outputs. At each
discrete time step $t$, the $DQN$ agent employs an $\varepsilon$-greedy
strategy for action selection. It updates the Q-function based on observed
states, rewards, and the next state using an update rule:

(1) $Q(s_{t},a_{t})\leftarrow
Q(s_{t},a_{t})+\alpha(r_{t+1}+\gamma\max_{a}Q(s_{t+1},a)-Q(s_{t},a_{t})),$

where $\alpha$ is the learning rate, $\gamma$ is the discount factor, and
$\max_{a}Q(s_{t+1},a)$ is the estimated maximum action-value in the next state
$s_{t+1}$.

$DQN$ aims to learn a policy $\pi(s,t)$ that maximizes the cumulative reward
over time in a given environment. It combines Q-learning with deep learning,
allowing the agent to handle high-dimensional observations and non-linear
function approximations.

$DQN$ can model the dynamic interplay between environments and human
interaction. Through interaction with the HITL environment, the $DQN$ agent
continually updates these Q-value estimates, guided by the reward signals
originating from humans. These rewards, whether stemming from explicit
annotations or implicit cues, enable the agent to learn between favorable and
undesirable actions, ultimately reinforcing the learning of optimal policy
$\pi(s,a)$ that shape the HITL adaptation (Chen et al., 2021).

In this ecosystem, humans are more than mere bystanders; they are active
contributors whose feedback shapes the reward landscape for the $DQN$ agent.
This feedback, manifested in various forms like explicit annotations or subtle
sensory cues, is categorized as positive (indicating “good” human experience)
or negative indicating “bad” human experience). The integration of these
human-originated rewards is pivotal in steering the RL agent towards learning
optimal policies. These policies, in turn, dictate the adaptive actions
executed within IoT systems, aligning technological innovation with human-
centric values.

##  4\. Threat model in HITL IoT technologies

Recent advancements in edge devices’ power and memory capabilities have
enabled the execution of relatively resource-intensive computations at the
edge. Consequently, it can be assumed that the edge layer can perform
demanding data processing tasks, including raw data processing and the
execution of RL algorithms. Moreover, synchronizing numerous applications,
particularly in pervasive environments, necessitates a central decision-making
server operating at a cloud-based level. As a result, the edge layer does not
directly transmit control actions to the environment. Still, it must first
share them with the cloud to address various constraints it may need to
enforce, such as synchronizing multiple applications. This edge-cloud
computing model represents a typical architecture for numerous pervasive and
ubiquitous applications (Bovornkeeratiroj et al., 2020; Papst et al., 2022;
Stirapongsasuti et al., 2019).

The proposed threat model can be summarized as follows:

  * •

An application operating under the HITL paradigm collects information from the
environment and human interactions through multiple sensors on edge devices.

  * •

A $DQN$ agent runs on the edge to infer the state of the human and the
environment and recommends adaptive actions based on human preferences and
behavior.

  * •

Only the recommended action generated by the $DQN$ agent is communicated to a
cloud-based server; in particular, the inferred states of the environment or
the human are not shared with the cloud. The Edge is considered a trusted
entity.

  * •

The cloud selects the appropriate control signals, which may be based on other
enforced constraints, and sends them to actuator nodes (edge devices) for
adaptation in the environment.

Therefore, based on this threat model, our attack vector is as follows:

  * •

An eavesdropper is positioned within the cloud and cannot access edge devices
(sensors or actuators) or the communication channel between the edge and the
cloud.

  * •

The eavesdropper has access to the time-series data of the desired recommended
actions within the cloud.

  * •

The eavesdropper can employ any machine learning model leveraging this time-
series data.

  * •

The eavesdropper possesses prior knowledge of the application domain.

In this attack vector, even if the communication channel between the edge and
the cloud is secured (e.g., through encryption), the cloud is still required
to decrypt the transmitted packet, which contains the desired action, to
select the control signals to be sent back to the edge actuators for pervasive
applications. Consequently, the eavesdropper can observe and record the time-
series data of the desired actions of this application. Figure 1 illustrates
the threat model and attack vector. Our objective in this study is to provide
a guarantee that limits the ability of an eavesdropper to infer the private
state, even with unlimited computational power and complete knowledge of the
application domain.

Figure 1. Threat Model for HITL IoT application. Figure 2. Structure of the
PAPER-HILT algorithm. Each layer is followed by an exit branch and two
confidence paths associated with the utility and privacy. Branches and
confidence paths are trained independently.

##  5\. PAPER-HILT algorithm

PAPER-HILT algorithm addresses the challenge of balancing the privacy-utility
trade-off in HITL systems using $DQN$ models. Our approach entails training
$DQN$ to decide the optimal layer within the network at which an exit should
be taken based on the two objectives of utility and privacy. Specifically, the
algorithm employs two criteria for identifying network exit points that
maximize utility while ensuring privacy protection. To achieve this, PAPER-
HILT evaluates each early exit branch’s ability to provide a reasonable trade-
off between utility and privacy. This distinctive methodology offers a
personalized and adaptive mechanism to determine exit points based on human
behavior variability, ultimately allowing for more personalized privacy-aware
decision-making during inference. A pictorial figure for PAPER-HILT is shown
in Figure 2. In particular, the $DQN$ is designed with early exit (EE) at
every layer, providing a particular utility and privacy budget. The training
process consists of two phases. In the first phase (PHASE 1), we train the
$DQN$ with all of the EE branches. Then, in the second phase (PHASE 2), we
fine-tune the branches based on the privacy and utility budgets. In the
following, we will provide a detailed explanation of the training phases.

###  5.1. Privacy Leak Metric

We draw upon the information theory literature and employ mutual information
(MI) to quantify the degree of correlation (or dependence) between two random
variables. In our specific context, we utilize MI to gauge how effectively an
eavesdropper can deduce a system state from observed actions. MI establishes a
theoretical limit on the inferential capability of any learning algorithm.
Generally, the lower the MI between the state and action, the less accurate
any inference algorithm can be. To illustrate the extreme case, no algorithm
can infer the state from the observed actions when MI is zero. Therefore, we
regard the MI measure $\mathcal{I}_{\pi}(a_{t};s_{t})$ for a given policy
$\pi(s,a)$ as a quantifiable constraint on the capacity to extract maximum
information about $s_{t}$ by observing $a_{t}$. Specifically, the information
disclosed to the eavesdropper is bounded by $\mathcal{I}_{\pi}(a_{t};s_{t})$.
We will exploit this metric to determine the privacy leak at each exit, as
shown in Figure 2.

###  5.2. PAPER-HILT training

Training PAPER-HILT entails the following design choices: (1) the placement of
the EE branches in $DQN$, (2) the architecture of these EE branches, (3) the
criteria that determine when a branch should be taken to exit, and (4) the
training strategy applied to these criteria for branches. We divide our
training for PAPER-HILT into two phases, as explained below.

####  5.2.1. PHASE 1: Train $DQN$ with EE

The process of training a Deep Q-Network ($DQN$) to learn the optimal policy,
denoted as $\pi(s,a)$, is a dynamic procedure that requires interaction with
the environment. This interaction is through taking actions generated by the
policy of an online Q-network. These actions and states are accumulated within
an experience replay buffer. This buffer serves as a reservoir of past
experiences. Periodically, at predefined intervals during the training phase,
a batch of transitions is randomly sampled from this experience replay. These
sampled transitions are subsequently evaluated using a separate entity known
as the target Q-network, which plays a crucial role in stabilizing the
learning process. The target Q-network helps approximate the expected future
rewards, guiding the $DQN$ toward the optimal policy. The parameter updates
made in the online Q-network are based on the calculated loss, which varies
depending on the specific application or task. This loss reflects the
disparity between the Q-values estimated by the online Q-network and the
target Q-network. Consequently, this entire training process is designed to
optimize the performance of $DQN$ within application environments, and the
success of the model relies on the iterative refinement of these interactions
and updates.

Our first training phase (PHASE 1) is to train $DQN$ with EE. We exploit the
recent work in the literature that proposes a “sequential training” approach
(Kosta et al., 2022). It starts with a single-layer Q-network and gradually
expands by introducing and training a new layer with a new exit one at a time
while keeping the parameters of previously trained layers and exit branches
fixed. As a result, the previously trained layers and branches remain
unaffected when a new layer and branch are introduced. Figure 3 illustrates
$DQN$ with EE using sequential training111We used a branch layer of two fully
connected layers and an output layer which generates the q values..

####  5.2.2. PHASE 2: Train for privacy-utility tradeoff

The resulting $DQN$ from PHASE 1 contains EE branches, each individually
optimized to deliver peak performance at this exit per a predefined
application utility. However, as motivated earlier, we want to train PAPER-
HILT to deliver a privacy-utility tradeoff. This tradeoff should account for
the dynamic nature of the environment and the varying behavior of humans.
Accordingly, as seen in Figure 2, each EE branch maintains a _confidence
labels_ on the utility and privacy at this branch (generated by the privacy
and utility paths). Hence, based on a particular privacy-utility tradeoff, a
particular EE branch will be chosen to exit.

The intuition is that we want to give each EE branch two labels: a utility
label and a privacy label. Each label can take a value of “1” or “0” to
indicate if a particular EE satisfies a particular utility or privacy budget
(given a label “1”). These labels will determine which EE branches are
available to exit based on a particular utility and privacy budget.

To achieve this, we further augment the trained $DQN$ with two other paths,
utility path, and privacy path, at each exit that will output these labels. We
train these two paths to learn these labels based on two parameters: ‘$u$’
representing the $utility\ budget$ and ‘$p$’ representing the $privacy\
budget$. Lower values of ‘$u$’ and higher values of ‘$p$’ prioritize privacy
while potentially sacrificing utility. Conversely, higher ‘$u$’ and lower
‘$p$’ values favor utility-centric learning at the potential expense of
privacy leaks. Thus, depending on the specific application requirements and
the variation of human behavior patterns, the tradeoff between ‘$u$’ and ‘$p$’
governs the exit selection in the $DQN$.

##### Utility Confidence Label (UCL) Training:

For the utility confidence label training, we follow the (Kosta et al., 2022).
During the interaction of the trained model from PHASE 1 with the environment,
we create a utility replay buffer containing the state, action, and utility
labels. We randomly choose one of the actions determined by the branches.
Monitoring the max of the q-values of each branch (${Q}_{imax}$), we create
labels for all branches for the replay buffer, where $Q_{max}$ is the maximum
of all of the branches.

(2) $UCL_{i}(s,a)=\begin{cases}1,&\text{if }Q_{imax}(s,a)>u\times
Q_{max}(s,a)\\\ 0,&\text{otherwise,}\end{cases}$

where parameter $u\in(0,1]$ is the utility budget that determines the amount
of the utility that we want to preserve.

##### Privacy Confidence Label ($PCL$) Training

Similarly, to train $PCL$, we let the trained $DQN$ from PHASE 1 interact with
the environment. During this interaction, the input to the $DQN$ is the
current environment state with the human-in-the-loop $s$, and then each EE
will output one action $a$. As explained in Section 5.1, we use MI as a metric
to measure the privacy leak. We use ${I}_{i}(s,a)$ to represent the MI of
branch $i$ 222As MI can only be determined through a time series, we update
the privacy replay buffer for every $n$ interaction.. At the end of every
$n_{th}$ interaction with the environment, the min of MI of all branches for
the past $n$ interactions is calculated as
${I}_{max}(s,a)=\max({I}_{i}(s,a))$. The $PCL$ is then determined as,

(3) ${PCL}_{i}(s,a)=\begin{cases}1,&\text{if
}{I}_{i}(s,a)<p\times{I}_{max}(s,a)\\\ 0,&\text{otherwise,}\end{cases}$

here, the $PCL$ of branch $i$ is set to $1$ if the ${I}_{i}(s)$ is smaller
than the ${I}_{max}$ multiplied by a constant $p\in(0,1]$; otherwise, it is
set to $0$. This parameter $p$ is the $privacy\ budgect$ as described earlier.
A higher $p$ prioritizes high privacy, while a lower $p$ results in more exits
at the cost of privacy.

Accordingly, to train this label, PAPER-HILT augments the trained Q-network
with a branch called a privacy path that is going to output this label at
every exit333We used a branch of a fully connected layer with a binary cross-
entropy (BCE) loss that creates a binary output as the confidence outcome..
Hence, an experience replay buffer is created that contains the state, action,
and all the $PCL$ from all the branches. Since we have different actions from
each EE branch, the action ($a$) chosen for a given input state ($s$) is
selected randomly from the actions predicted by all the EE branches ($a_{i}$)
to prevent bias towards a specific branch.

Figure 4 shows this process of maintaining privacy and utility reply buffers
for every $DQN$ interaction with the environment. By adopting this confidence-
label training framework for utility and privacy, we allow for EE that
considers privacy preservation and utility maximization based on a particular
privacy budget $p$ and a utility budget $u$. The two training phases are
described in the Algorithms 1 and depicted in Figure 4. In particular, at the
end of this training, each EE branch will give three outputs, $Q(s,a)$ from
PHASE 1 training, and the two labels ($UCL$ and $PCL$).

Figure 3. Phase 1 of the training flow of the PAPER-HILT algorithm. An exit
branch follows each layer. The layer and associated branch are trained in each
step, and the next layer and its branch are added in the following step.
Training of the new layer and its branch happens while the parameters of the
previous layers and branches are frozen. Figure 4. Phase 2 of the training:
Privacy and utility confidence path training for each exit branch of the
$DQN$. The network has two separate buffers. The privacy buffer contains the
action, state, and privacy labels for all branches. The utility buffer
includes the action, state, and utility labels of the branches.

Input:

Environment $\mathcal{E}$,

Number of layers $n$,

Utility budget $u$,

Privacy budget $p$.

Output:

Trained Q-network $\mathcal{Q}$ with utility and privacy confidence labels.

PHASE 1:

Initialize $\mathcal{Q}$ with random weights. for  _$\text{layer}\leftarrow 1$
to $n$_ do

Add branch for layer layer. while  _Training_ do

Sample a transition tuple $(s,a,r,s^{\prime})$ from $\mathcal{E}$;

Compute the Q-value target;

Compute the Q-value estimate; Compute the loss $\mathcal{L}=\text{mean squared
error}(Q_{\text{target}},Q_{\text{estimate}})$;

Update the parameters using gradient descent.

Freeze the parameters of layer layer and its branch. Add layer
$\text{layer}+1$ to $\mathcal{Q}$.

PHASE 2:

Input:

Trained $n$ layered Q-network $\mathcal{Q}$,

Utility budget $u$,

Privacy budget $p$.

Interact with the environment;

Create privacy and utility buffers (Choose $a_{i}$ randomly from the
branches);

Utility buffer $(s,a,UCL)$ for utility training;

Privacy buffer $(s,a,$PCL$)$ for privacy training;

Augment $\mathcal{Q}$ with branches for the privacy and utility confidence
paths.

while  _Training_ do

Sample $(s,a,UCL)$ from Utility buffer;

Sample $(s,a,$PCL$)$ from Privacy buffer;

Train privacy and utility confidence paths;

Update the parameters of the confidence paths.

Algorithm 1 Phase 1 & 2 of the $DQN$ training

Input:

State: $s$;

Trained $DQN$ with $n$ branches trained for a particular privacy $p$ and
utility $u$ budgets: $\mathcal{Q}$;

Output:

Branches satisfy privacy $p$ and utility $u$ budgets;

Take the current state $s$ as input

for  _Each branch $i$ in $\mathcal{Q}$_ do

Get the action $a_{i}$ for branch $i$ based on current $s$;

Retrieve the confidence scores for privacy and utility as PCLi and UCLi;

if  _PCL ${}_{i}=1$_ then

Branch $i$ is satisfactory for the privacy budget $p$;

if  _UCL ${}_{i}=1$_ then

Branch $i$ is satisfactory for the utility budget $u$;

Algorithm 2 Inference using a Trained $DQN$

Input:

State $s$;

Trained $DQN$ with $n$ branches with a particular privacy $p$ and utility $u$
budgets: $\mathcal{Q}$;

Replay buffer size: $\mathcal{B}$;

Human variability threshold: $I_{\text{threshold}}=v\times I_{max}$, where
$v\in[0,1]$;

Output:

Retrained $DQN$ with $n$ branches $\mathcal{Q^{\prime}}$.

Initialize a replay buffer $\mathcal{B}$;

while  _Inference with $\mathcal{Q}$_ do

Update the replay buffer $\mathcal{B}$,

Monitor $I_{current}$ of the selected exit branch;

if  _$I_{current} <I_{\text{threshold}}$_ then

Train $\mathcal{Q^{\prime}}$ using the data in $\mathcal{B}$;

$\mathcal{Q}$ = $\mathcal{Q^{\prime}}$

Reset $\mathcal{B}$;

Algorithm 3 Inference and human variability-aware $DQN$ retraining

###  5.3. PAPER-HILT Inferrence

Once the training is complete, the trained $DQN$ with the confidence paths is
deployed to guide the model’s decision-making process. The algorithm utilizes
confidence branches at each decision point to make informed choices regarding
the optimal exit. The EE is selected to align with the specified utility ($u$)
and privacy ($p$) budgets as shown in Algorithm 2.

###  5.4. Incooperating Human Variability

As mentioned earlier, human behavior is not fixed, and hence, the labels used
($PCL$ and $UCL$) to train the confidence path can change over time. Hence,
after the trained $DQN$ is deployed, PAPER-HILT accumulates a repository of
interactions with the environment, which contains selected actions, state
observations, and corresponding rewards associated with the chosen exit
branch. This repository will be used to monitor the variations in MI. If these
variations drop below a predetermined threshold (as an indication of the
change in the human behavior), PAPER-HILT initiates a training phase, wherein
the $DQN$ is retrained to adapt to the evolving environment, effectively
accommodating variations in user behavior or environmental conditions. We set
this threshold as a percentage of the max of the MI ($I_{max}$ at the point
the agent converges to a specific action that maximizes the reward that we
reach during Phase 1 of the training). $I_{\text{threshold}}=v\times I_{max}$,
where $v\in[0,1]$. This continuous training through updating the $DQN$
whenever the human behavior or environment changes is described in Algorithm
3.

##  6\. Application 1 - House Model

Recent research in the field of smart heating, ventilation, and air
conditioning systems (HVAC) with human-in-the-loop aims to enhance human
satisfaction (Jung and Jazizadeh, 2017; Elmalaki, 2021). Accordingly, it is
essential to incorporate human conditions and preferences into the decision-
making process when determining the HVAC setpoint. For instance, the human
body temperature naturally decreases during sleep (Barrett et al., 1993),
while it rises during physical exercise (Lim et al., 2008), and in response to
stress and anxiety (Olivier et al., 2003). To monitor human conditions such as
sleep patterns and physical activity, IoT edge devices can be used (Nguyen et
al., 2016; Shany et al., 2012). These devices can play a crucial role in
gathering relevant data. However, it’s worth noting that while the primary
objective of this section is to assess the trade-off between privacy and
utility offered by our model, we will first provide an overview of the IoT
environment design and the DRL agent used for this specific application.

###  6.1. Human-in-the-Loop IoT System Design

In our study, we simulated a thermodynamic model representing residential
houses. This model considers various factors, such as the house’s
architectural design, the number of windows, the roof’s pitch angle, and the
insulation material employed. The heating system in the house is facilitated
by a heater, which releases air at a temperature of $50^{\circ}C$, while
cooling is achieved through a cooler that emits air at $10^{\circ}C$. To
regulate the indoor temperature, a thermostat is employed, which permits a
deviation of up to $2^{\circ}C$ both above and below the desired set-point.
This setpoint temperature is managed by an external controller running PAPER-
HILT (MATLAB, 2022).

In our approach, we represent humans as thermal sources, with heat generation
influenced by both the average exhale breath temperature $(EBT)$ and the
respiratory minute volume ($RMV$) (Elmalaki, 2021). The $RMV$ is determined by
the breathing frequency $(f)$ and the volume of gas exchanged during each
breathing cycle, which varies significantly based on the level of human
activity(Carroll, 2006).

We model the behavior of three distinct human subjects in three homes by
manipulating their $RMV$ and metabolic rate. Our simulation encompasses six
activity categories, including five related to in-home activities and a state
labeled as “not at home.” These categories include sleeping, relaxing,
watching TV, cooking, and exercising. Each human is characterized by unique
and diverse lifestyle patterns. Human $H_{1}$ adheres to a structured routine
that repeats weekly, displaying limited randomness. In contrast, $H_{3}$
exhibits a highly irregular lifestyle with activities that lack a specific
daily or weekly pattern, featuring numerous unexpected changes. $H_{2}$ falls
in between, displaying a moderate level of randomness compared to $H_{1}$ and
$H_{3}$. Figure 5 provides a visual representation of the behavioral pattern
of $H_{1}$, $H_{2}$, and $H_{3}$ showcasing alternating engagement in six
primary activities, including sleeping, not at home, watching TV, cooking,
exercise, and relaxation.

Figure 5. Human activity profile for three humans.
Figure 6. Activity pattern of the house model. Top: ground truth of the
activity patterns of human $H_{1}$. Middle: clustering result as seen by the
eavesdropper. Bottom: clustering result after applying mitigation with PAPER-
HILT at $p=0.7$ and $u=0.75$.

To introduce variability into our simulation, we randomly assign different
activities within the same time slots to create three diverse daily human
behaviors. It is important to note that we do not account for factors such as
age, sex, or time of day in our model. We have expanded the thermal house
model provided by Mathworks to incorporate a cooling system and our human
simulation (MATLAB, 2022)444While more complex simulators like EnergyPlus
exist for comprehensive assessments of energy consumption and electrical loads
in smart houses and buildings (Gerber, 2014), we have opted for a simplified
thermal house model to evaluate PAPER-HILT.. Each simulation step corresponds
to one hour and every 24 steps is one day.

###  6.2. DRL Design

We have developed a DRL framework, which is elaborated upon below, to
personalize the HVAC setpoint according to human activity and the desired
level of thermal comfort.

####  6.2.1. State, Action, and Reward

State space $\mathcal{S}$: For every possible combination of six unique human
activities and the indoor temperature of the house, our state space
$\mathcal{S}$ is defined as $\mathcal{S}=\\{(act,temp):act\in[1,6]\text{ and
}temp\in[60,80]\\}$. Here, $act$ represents the current human activity, and
$temp$ represents the current temperature inside the house, a continuous
value. Action Space $\mathcal{A}$: we consider discrete values within the
temperature range of $[60,80]$, including heating and cooling options. Reward
$\mathcal{R}$: We used Prediction Mean Vote $(PMV)$ as a means to gauge human
thermal comfort (Fanger, 1970). The PMV scale spans $-3$ (indicating very
cold) to $3$ (indicating very hot). According to the ISO standard ASHRAE 55
(ASHRAE/ANSI Standard 55-2010 American Society of Heating, Refrigerating, and
Air-Conditioning Engineers, 2010), maintaining a PMV within $[-0.5,0.5]$ for
indoor spaces is recommended to ensure thermal comfort. Estimating the $PMV$
score relies on clothing insulation, metabolic rate, air vapor pressure,
temperature, and mean radiant temperature (Fanger, 1970). Hence, $PMV$ values
falling within $[-0.5,0.5]$ yield positive rewards, while PMV values
indicating discomfort result in negative rewards. In practical applications,
PMV can be estimated using edge devices, such as black globe thermometers
(campbellsci, 2022).

###  6.3. Evaluation

Our evaluation strategy starts by first showing PAPER-HILT’s ability to adapt
to different human behavior patterns by choosing different EE for each human
to achieve a utility budget $u$ (Section 6.3.1). Afterward, we asses the
privacy leak resulting from this personalization based on our threat model
(Section 6.3.2). Then, we show the effect of using PAPER-HILT with $PCL$ and
$UCL$ at a particular privacy $p$ and utility $u$ budgets on privacy leak
mitigation (Section 6.3.3). Finally, we analyze the tradeoff of choosing $p$
and $u$ in this application (Section 6.3.4). Our results show that using
PAPER-HILT, we were to provide _personalized_ privacy leak mitigation by
improving privacy protection.

####  6.3.1. PAPER-HILT Adaptation

PAPER-HILT’s architectural design involves considerations that can be viewed
as design hyperparameters. These include determining the location of exit
branches and defining the structure of the exit branch. We start by designing
a $DQN$ model without exit branches to analyze the performance relationships
with the base network architecture. First, we design $DQN$ models with varying
sizes from 1 to 10 and measure the $PMV$ percentage in the comfortable range
$([-0.5,+0.5])$ for different layer depths per human as the metric for the
utility performance of the network. As the agent interacts with the simulated
environment to maximize the PMV in the comfortable range, $DQN$s with varying
depths learn to converge to specific actions. Figure 8 demonstrates the
convergence of actions as the training progresses for a $DQN$ of $6$ layers,
revealing how the $DQN$ model converges to specific actions for $H_{1}$.

Our analysis demonstrates that for $H_{1}$, the network with a depth of $6$
layers outperformed network depths, while for $H_{2}$ and $H_{3}$, the optimal
number of layers were $8$ and $3$, respectively. Outperformed depths of layers
are in the range 1 to 10, which indicates that the base model of depths $10$
is a proper choice for PAPER-HILT architecture design for this application.

We designed a $DQN$ of $10$ fully connected layers with a branch composed of
$2$ fully connected layers for each base layer. The employed loss function is
the mean square error (MSE) of the online and target $Q$ networks, which
utilize a replay buffer containing action, state, and reward information for
updating the network’s parameters. A batch size of $16$ (1 sample per hour)
was used to train the model, which means it takes $16$ hours to have enough
samples in the replay to update the model parameters. We examined the impact
of using base $DQN$s (without branches) with varying layers on the system
performance in this setup. Subsequently, we started PHASE 1 in PAPER-HILT,
which has “sequential training” with up to $10$ layers. We trained three
independent $DQN$s for the three humans. We measured the $MI$ and $PMV$ for
all exit branches to assess the relationship between utility performance and
information leakage. Figure 9 depicts the $MI$ and $PMV$ for $H_{1}$ for all
exit branches. The first row of Figure 9 shows the $MI$ between the chosen
action for that exit branch and the input state $s$. The second row plots the
histogram of the $PMV$ values for each exit branch as they interact with the
environment, where the percentage value indicates the percentage of the $PMV$
values in the comfortable range of $[-0.5,+0.5]$. Figure 9 shows that both
$MI$ and $PMV$ percentages were highest at branch $6$, indicating that higher
utility performance reveals more private information about the user.

Figure 7. Action convergence of human $H_{1}$ using $DQN$. Figure 8. PMV
before and after applying PAPER-HILT mitigation using utility budget $u=0.75$
and privacy budget $p=0.7$ for $H_{1}$.

Figure 9. PMV and corresponding MI at every branch for human $H_{1}$. Top:
mutual information between the current state and the chosen action in the
associated exit layer. Bottom: PMV histogram for the associated exit with the
percentage of the PMV values in the comfortable range ($[-0.5,+0.5]$). Mutual
information and PMV percentage in the comfortable range occurs at the layer
$4$ exit branch.

####  6.3.2. Privacy Leak

We assess this application’s potential risk of private information leakage,
assuming that an eavesdropper can monitor the DRL agent’s actions. This
scenario is plausible when we consider a smart thermostat system utilizing a
cloud-based service, such as NEST (Times, 2022), along with a covert spyware
eavesdropper situated within the cloud infrastructure as discussed in our
threat model in Section 4.

Given that the eavesdropper possesses no prior knowledge about human behavior
within the household, they might use unsupervised learning techniques to
deduce concealed patterns. For instance, by implementing a simple clustering
algorithm, such as $K$-means, the eavesdropper can reveal sensitive
information such as occupancy and sleeping schedules. Since the eavesdropper
lacks prior knowledge regarding human activity, the optimal number of clusters
remains unknown. Consequently, a possible method for the eavesdropper is to
utilize the elbow point technique to find the most appropriate number of
clusters. Utilizing the clustering on the actions shared with the cloud engine
from the branch that maximizes the utility (in this case exit branch $6$ for
human $H_{1}$), an eavesdropper can plot the WCSS (Within-Cluster Sum of
Square) vs. the number of clusters as shown in Figure 10 to analyze the time
series data of HVAC setpoints (actions) collected from human $H_{1}$ with over
100 simulated days. Figure 10 reveals that six clusters emerge as the
predominant outcome for clustering numbers, aligning with the actual number of
human activities in the simulation model.

u u = 0.85 $L$ $L_{1}$ $L_{2}$ $L_{3}$ $L_{4}$ $L_{5}$ $L_{6}$ $L_{7}$ $L_{8}$
$L_{9}$ $L_{10}$ $H_{1}$ ✓ $H_{2}$ ✓ $H_{3}$ ✓ Table 1. Permitted exit
branches for the utility budget $u=0.85$ for all human subjects. Figure 10.
WCSS vs. the number of clusters for human $H_{1}$.

Consequently, we compare the actual behavior of an individual ($H_{1}$) in the
simulation model and the clustering results obtained by an eavesdropper (with
six clusters) for each day. This comparison is presented in the top and middle
subfigures of Figure 6. Figure 6 (middle) shows that the clustering is closely
related to human activity patterns. In this scenario, the observer achieves an
$87$% accuracy in clustering. This indicates that after approximately $15$
days (as seen in Figure 6), there are some discernible leaks of information
regarding the human’s behavioral patterns. It’s important to note that the
observer does not precisely infer the exact human activities. Nevertheless, we
demonstrate here that the observer can deduce specific behavioral patterns of
the human, leveraging their knowledge of the application domain. This allows
them to make educated inferences, such as when the individual is likely to go
to sleep, when they depart from their residence, or when there are deviations
from the typical daily behavior of the individual.

####  6.3.3. PAPER-HILT Personalized Privacy-aware Adaptation

We evaluate the effectiveness of PAPER-HILT in adaptively reducing privacy
vulnerabilities tied to human behavior while considering the privacy-utility
tradeoff. To this end, we first start PHASE 2 of the $DQN$ training for $PCL$
and $UCL$ paths as described in algorithm 1. For PHASE 2, we interact with the
environment to create utility and privacy buffers for specific values of
privacy budget $p$ and utility budget $u$ as explained in sections5.2.1 and
5.2.2. For instance, we assess the performance at $u=0.85$ as shown in Table 1
for humans $H_{1}$, $H_{2}$, and $H_{3}$. For $H_{1}$ with a utility budget of
$u=0.85$, exit branch $6$ ($L_{6}$) has label $1$ (satisfying the utility
requirements) and is permitted during the inference, meaning that the network
can choose this exit to meet the utility budget constraint ($u=0.85$). After
these interactions with the environment used to generate utility and privacy
buffers, we start the PHASE 2 training for $UCL$ and $PCL$ paths.

We implemented the PHASE 2 training based on different budgets for privacy $p$
and utility $u$ to investigate the impact of those budgets on the permitted
exit branches for different humans. Table 2 presents various values of $u$ and
$p$ and allowed exit branches for each human. For instance, at $u=0.75$ and
$p=0.7$, the allowed exit branches for $H_{1}$ are exits at layers $L_{1}$ and
$L_{6}$. Decreasing the values of the budgets is associated with a higher
number of permitted exit branches. For instance, at $u=0.75$ and $p=0.6$,
exits at layers $L_{1}$, $L_{6}$, and $L_{7}$ are permitted to take. We
validated these outcomes by examining the eavesdropper’s ability to cluster
the time series of shared actions with the cloud engine. We chose the first
available branch among the permitted exit branches, but as they are labeled as
“1, ” they provide the same performance on utility and privacy. Figure 6
depicts the ground truth of the activities for human $H_{1}$ (top), the
clustering result by the eavesdropper (middle) when there is no privacy
mitigation, and the bottom one is clustering results after taking allowed
exits (in this case exit layer $L_{6}$ with $u=0.75$ and $p=0.7$. As Figure 6
illustrates, the clustering capability of the eavesdropper drops to $61$%
accuracy after applying the mitigation, indicating an improvement in the
privacy protection by $26$% for human $H_{1}$. We observed similar results for
$H_{2}$ and $H_{3}$ with $23$% and $21$% improvement, respectively.

The effect of this using $u=0.75$ and $p=0.7$ on human $H_{1}$ is shown in
Figure 8. The histogram of the PMV before and after using the PAPER-HILT shows
that before mitigation (early exit at the optimal utility at Layer $L_{4}$),
the percentage of PMV samples within the $[-0.5-0.5]$ is $86\%$, compared to
$79\%$ when using mitigation indicating a reduction in the performance by
$9$%. We have observed similar results for $H_{x}$ and $H_{x}$ with reduction
performance of $12$% and $16$%, respectively.

####  6.3.4. PAPER-HILT Privacy-utility Tradeoff Analysis

We used the standard deviation ($STD$) of the PMV as a utility variation
metric. A higher $STD$ in PMV values signifies lower utility (indicating a
broader spread of PMV values). We assessed the clustering accuracy as an
indicator of privacy leakage against the $STD$ of the PMV for various values
of $u$. Figure 12 shows the privacy vs. utility plots using different values
of $u$ for all human subjects. Increasing the value of $u$ corresponds to a
decrease in the $STD$ of the PMV, indicating better performance at the expense
of lower privacy protection (higher clustering accuracy). On average, for all
humans, utility (performance) drops by $23$%, and privacy (state prediction)
improves by $34$%.

u | u = 0.95 | u = 0.85 | u = 0.75 | u = 0.65 | u = 0.55
---|---|---|---|---|---
human | $H_{1}$ | $H_{2}$ | $H_{3}$ | $H_{1}$ | $H_{2}$ | $H_{3}$ | $H_{1}$ | $H_{2}$ | $H_{3}$ | $H_{1}$ | $H_{2}$ | $H_{3}$ | $H_{1}$ | $H_{2}$ | $H_{3}$
p = 0.9 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $L_{3}$ | $L_{4,6}$ | $L_{8}$ | $L_{2,3}$
p = 0.8 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $L_{6}$ | $\times$ | $L_{3}$ | $L_{1,6}$ | $L_{5,8}$ | $L_{3,9}$ | $L_{1,3,6}$ | $L_{5,8}$ | $L_{3,9}$
p = 0.7 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $L_{1,6}$ | $L_{8}$ | $L_{3,5}$ | $L_{1,4,6}$ | $L_{3,6,9}$ | $L_{4,7}$ | $L_{1,6}$ | $L_{3,6,8}$ | $L_{3,5,7}$
p = 0.6 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $L_{1,6,7}$ | $L_{5,8}$ | $L_{3,4,5}$ | $L_{1,6,7}$ | $L_{5,8}$ | $L_{3,4,5}$ | $L_{1,6,7}$ | $L_{4,5,7}$ | $L_{4,7,8,9}$

Table 2. $PCL$ and $UCL$ confidence per layer for all humans at various values
of $u$ and $p$.

Figure 11. Privacy-utility trade-off for three humans with various values of
$u$.

Figure 12. Human behavior changes over time and the corresponding MI using
$u=0.75$ and $p=0.7$.

####  6.3.5. Adapting to Human Variability

During the inference, we monitor the current value of mutual information
$I_{current}$ of the exit branch. If the $I_{current}$ drops below a certain
percentage of the $I_{max}$ – where $I_{max}$ is the maximum of the $I$ during
the training PHASE 1, we retrain the Q network to adapt to this variability in
human behavior. Figure 12 illustrates the MI for $H_{3}$ where the model
learns the behavior $MI$ adapts to that specific behavior. However, the human
behavioral pattern eventually shifts to a new pattern, and $MI$ begins to
drop. After it drops below a certain threshold $I_{threshold}=0.8$, the $Q$
network is retrained and replaced by the new model. We observe the growth of
the $MI$ until it converges to a new maximum value to conform to the new
behavior. To simulate this transition in behavior, we switched between the
behaviors of $H_{3}$ and $H_{1}$ where the $H_{3}$ interacts with the
environment for 25 days, and then we switched to $H_{1}$ for another 25 days
of the simulation. It took 10 days to retrain the model and adapt to the new
behavior. As depicted in Figure 12, this results in a newly adjusted MI curve
that accommodates these alterations in human behavioral patterns.

##  7\. Application 2 - Smart Classroom with VR technology

In this section, we create a real-world virtual reality (VR) application to
assess the effectiveness of PAPER-HILT. This decision was inspired by the
recent transformation in the education sector, emphasizing personalized and
remote learning setups. In prolonged training or educational sessions,
especially in the online or remote context, human performance often
deteriorates due to distractions, drowsiness, and fatigue (Terai et al.,
2020). In our experiment, we employ an RL agent to monitor these changes in
the human state and offer personalized feedback to enhance learning
performance. This RL agent operates from the cloud and has access to the
actions executed by the RL model. As established in our previous experiment,
these adaptive actions are linked to the individual’s private state,
encompassing learning performance and mental well-being. Our approach
commences with the design of the application, followed by the exploration of
the relationship between the actions taken by the RL agent and the mental
state of the human participant. Finally, we introduce the use of PAPER-HILT,
to counteract potential breaches of private data.

###  7.1. System Design

We introduced two presentation modes, 2D and 3D, to deliver educational
content to our study participants. To implement the 3D presentation, we
harnessed cutting-edge Virtual Reality (VR) technologies, specifically
utilizing an Oculus device. The choice of VR was informed by recent research,
indicating its substantial potential for revolutionizing learning and
workforce training (Ibáñez et al., 2014). Meanwhile, for the 2D presentation
mode, we relied on traditional laptop screens. The content we used for these
presentations was sourced from Khan Academy, along with its accompanying
quizzes covering topics in biology (KhanAcademy, 2021a), chemistry
(KhanAcademy, 2021c), and physics (KhanAcademy, 2021b). We engaged $15$
participants, all within the age range of $20$ to $30$ years, to view these
stand-alone lectures. Importantly, these lectures were designed to be
comprehensible without any prerequisite knowledge on the participants’ part.

The primary task assigned to the participants was to attentively watch the
lectures and subsequently respond to questions related to the content at the
end. To facilitate the 3D experience, we transformed these 2D lectures into 3D
versions suitable for VR presentations. Each lecture lasted $55$ minutes and
followed a narrative style, devoid of quizzes or interruptions. To gauge the
mental states associated with learning, we equipped participants with EEG
wearable devices. While various wearable devices are capable of measuring
different physiological signals indicative of mental states (i.e., alertness
or drowsiness), we opted for the EEG signal. This choice was based on recent
research demonstrating that the activation of the frontal lobe in the brain
can be leveraged to gauge an individual’s learning ability and cognitive
performance (Eslinger and Damasio, 1985). We specifically employed the EMOTIV
EPOC_X 14-channel portable EEG device (EMOTIV, 2020). Before the experiment,
we administered a 10-minute 2D video presentation on a laptop screen to
establish a baseline for participants’ mental state signals. In monitoring the
participants’ alertness and readiness to learn, we analyzed the raw EEG data
in 10-minute intervals. Each lecture was divided into 10-minute segments,
referred to as ”stages,” with approximately five such stages in each
lecture555The average attention span of the human is $10$ to $15$ minutes
(McKeachie and Svinicki, 2006).. Figure 14 illustrates the setup worn by a
participant (VR and EEG). The laptop on the left casts the participant’s view
on the Oculus device, and the laptop on the right presents the live EEG signal
collected from the EMOTIV device. Figure 14 presents the screenshot of the
view of the participant while watching biology content on an Oculus device
with an office background.

Figure 13. EMOTIV $Epoc_{x}$ and Oculus device worn by a participant. Figure
14. Screenshot of the view of the Oculus device. AL 1 1 1 1 0 0 0 0 FL 1 1 0 0
1 1 0 0 VL 1 0 1 0 1 0 1 0 State S8 S7 S6 S5 S4 S3 S2 S1 Table 3. Human state
is one of 8 states depending on the alertness level (AL), fatigue level (FL),
and vertigo level (VL).

###  7.2. RL Design

Although VR technologies have shown their advantages in both education and
workplace training, people’s responses to VR environments can vary
significantly. Some individuals may experience symptoms of vertigo and
cybersickness while using VR, which can negatively impact the learning
experience. To address this issue, we have defined the human state as a
combination of three key features: alertness level (AL), fatigue level (FL),
and vertigo level (VL). We have categorized these three features into binary
classes to simplify our state space. Specifically, AL is classified as either
“Alert (1)” or “Not Alert (0),” FL is categorized as “Fatigue (1)” or “Vigor
(0),” and VL is grouped as “Not Vertigo (1)” or “Hypocalcemia (0).” We
represent the human state as a tuple of these three features:
$S=\\{(AL,FL,VL):AL\in\\{0,1\\},FL\in\\{0,1\\},VL\in\\{0,1\\}\\}$. Table 3
depicts an overview of the state space.

Each feature is determined based on a specific threshold ($\delta_{AL},\
\delta_{FL},\ \delta_{VL}$). If a feature’s measured value surpasses the
threshold, it is categorized as 1; otherwise, it is labeled as 0. The optimal
human state for effective learning is $s8$, where the individual is alert and
not experiencing cybersickness (neither fatigue nor vertigo). Conversely, the
least favorable state is $s1$, where the person is not alert and suffers from
cybersickness (both fatigue and vertigo). It’s important to note that
individuals can transition between these states, and these threshold values
can be adjusted to suit the specific application and participants. To
determine the human state $(AL,FL,VL)$ and compute the thresholds
($\delta_{AL},\ \delta_{FL},\ \delta_{VL}$) we follow the procedure studied in
(Taherisadr et al., 2023a).

The action space $\mathcal{A}$ in our VR application includes the following
actions: (1) Giving a break to the user, (2) Enabling VR mode by transitioning
from 2D to 3D, (3) Disabling VR mode by reverting from 3D to 2D, (4) Altering
the content of the presentation, and (5) Maintaining the state in the learning
environment (no change). Notably, activating VR mode heightens cognitive
engagement and improves learning performance. However, some users may
encounter cybersickness when exposed to VR, necessitating the reinforcement
learning agent to revert to the standard 2D mode to alleviate cybersickness
symptoms. Additionally, taking a break during a learning session may also be
necessary to combat drowsiness, cybersickness symptoms, or cognitive overload.
Therefore, $\mathcal{A}$ is a finite and bounded set of discrete actions
within the range of [1, 5]: $\mathcal{A}=\\{a:a\in[1,5],a\in\mathbb{N}\\}$.
The performance of individuals in a quiz determines the reward value assigned
after each learning module. The score in this quiz is expressed as a
percentage. The quantification of the quiz is based on ten multiple-choice
questions, with perfect and zero scores corresponding to $10/10$ (100%) and
$0/10$ (0%), and scores receive rewards 100 and 0, respectively.

###  7.3. PAPER-HILT Adaptation

We designed the same $DQN$ network described in section 6.3.1. Accordingly, we
employed the dataset of $15$ participants to classify their behaviors into
three distinct groups. The initial group comprises individuals highly tolerant
to VR, indicating they do not encounter simulator sickness even after
prolonged use of VR devices exceeding $20$ minutes. The second group includes
individuals who may exhibit cybersickness during VR exposure, while the third
group consists of individuals with the lowest VR tolerance. We represent the
behavior of these groups using Markov Decision Processes (MDP), as illustrated
in Figure 16 as described in (Zhao et al., 2023). Subsequently, we started
PHASE 1 in PAPER-HILT, which has “sequential training” with up to $10$ layers.
We trained three independent $DQN$s for the three profiles. Figure 16
illustrates the mutual information for all branches of the $DQN$ trained for
profile1. As Figure 16 depicts, exit branch $L_{4}$ of profile $1$ has the
highest mutual information growth.

Figure 15. MDP for three human profiles in VR learning environment. Figure 16.
Mutual information for all of the exit branches of profile $1$.

###  7.4. Privacy Leak

As in our outlined threat model, eavesdroppers within the cloud environment
can access RL actions and utilize machine learning algorithms to infer
confidential information related to participants. The calculation of the human
state occurs at the edge, and the privacy of the human mental state is of
utmost significance. Any unauthorized access to this information by potential
attackers is regarded as a breach of privacy. The prediction of the human
state is utilized, and we gauge the accuracy of human state prediction through
clustering, similar to application $1$, to measure privacy leaks. It is
important to highlight that even if the eavesdropper is unaware of the exact
human state, the alteration in the human state (learning pattern) resulting
from observed actions can be deduced, potentially revealing private
information such as the human attention span.

Figure 16 visually represents the mutual information growth between actions
taken by the human profile 1 and the states of the associated profile, where
the mutual information of exit branch $L_{4}$ is observed to reach
approximately $1.3$ bits. A strategy similar to the one devised for
eavesdropping in our initial application can also be applied to this scenario.
Eavesdroppers can employ unsupervised learning methods, such as clustering,
where our analysis shows that it can reach $82$% of accuracy for profile 1.

###  7.5. PAPER-HILT Personalized Privacy-aware Adaptation

We evaluate the PAPER-HILT’s effectiveness in adaptively reducing privacy
vulnerabilities, considering the privacy-utility tradeoff. This involves
initiating the second phase of $DQN$ training for $PCL$ and $UCL$ paths, as
specified in algorithm 1. For PHASE 2 training, we follow the same steps
described in section 6.3.3. Table 4 depicts the different budgets of $u$ and
$p$ and the permitted exit branches for each profile. For instance, at
$u=0.75$ and $p=0.7$, the allowed exit branches for $P_{1}$ are exits at
layers $L_{3}$, $L_{4}$ and $L_{6}$. Furthermore, we evaluate the trade-off
between utility (drop in performance) and the amount of the data leak (state
prediction) as illustrated in Figure 17. For instance, for $P_{1}$ at utility
budget $u=0.75$, the state prediction accuracy is $75$%. On average, for all
profiles, utility (performance) drops by $28$%, and privacy (state prediction)
improves by $28$%.

Figure 17. Privacy-utility trade-off for three profiles with various values of
$u$.

u | u = 0.85 | u = 0.75 | u = 0.65 | u = 0.55
---|---|---|---|---
human | $P_{1}$ | $P_{2}$ | $P_{3}$ | $P_{1}$ | $P_{2}$ | $P_{3}$ | $P_{1}$ | $P_{2}$ | $P_{3}$ | $P_{1}$ | $P_{2}$ | $P_{3}$
p = 0.9 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $L_{5}$ | $L_{7,8}$
p = 0.8 | $\times$ | $\times$ | $\times$ | $L_{4}$ | $L_{5}$ | $L_{7}$ | $L_{2,4}$ | $L_{5,7}$ | $L_{6,7}$ | $L_{2,4,8}$ | $L_{1,5,8,}$ | $L_{2,6,7}$
p = 0.7 | $\times$ | $\times$ | $\times$ | $L_{3,4,6}$ | $L_{5,9}$ | $L_{3,5,6}$ | $L_{4,7}$ | $L_{4,5,9}$ | $L_{4,7}$ | $L_{1,6}$ | $L_{3,6,8}$ | $L_{3,5,7}$
p = 0.6 | $\times$ | $\times$ | $L_{3}$ | $L_{1,6,7}$ | $L_{4,5}$ | $L_{4,5,9}$ | $L_{2,4,10}$ | $L_{5,8,9}$ | $L_{4,7}$ | $L_{1,4,9}$ | $L_{4,5,7}$ | $L_{7,8,10}$

Table 4. $PCL$ and $UCL$ confidence per layer for all profiles at various
values of utility budget $u$ and privacy budget $p$.

##  8\. Conclusion

In this paper, we introduced PAPER-HILT, an innovative approach that combines
the strengths of adaptive personalized DRL with an early-exit strategy. This
early-exit mechanism is a critical addition, allowing the system to make
prompt and informed decisions during the learning process. It enables the
system to efficiently navigate the complexities of privacy concerns, swiftly
intervening when privacy risks are detected. This approach is particularly
beneficial in scenarios where user behavior varies, catering to the
individualized nature of privacy requirements. By integrating this early-exit
strategy into the DRL framework, PAPER-HILT offers an effective solution for
managing the privacy-utility tradeoff, tailored to the specific needs and
variations in human behavior within HITL IoT environments. On average, across
two HITL applications, PAPER-HILT is able to mitigate privacy leaks by $31$%
while still providing acceptable utility.

Through PAPER-HILT, we aim to set a new standard in the development of
privacy-aware IoT systems, emphasizing the importance of responsive and
adaptable AI techniques in addressing the evolving landscape of privacy
concerns in technology, especially with human behavioral variability.

##  9\. Acknowledgments

This research was partially supported by the National Science Foundation (NSF)
awards $2105084$ and $2339266$. The authors would like to thank the Center for
Embedded & Cyber-physical Systems (CECS) at UCI for partially supporting
Mojtaba Taherisadr with the 2023-2024 ECPS Fellowship during the development
of this work.

## References

  * (1)
  * Ahadi-Sarkani and Elmalaki (2021) Armand Ahadi-Sarkani and Salma Elmalaki. 2021.  Adas-rl: Adaptive vector scaling reinforcement learning for human-in-the-loop lane departure warning. In _Proceedings of the First International Workshop on Cyber-Physical-Human System Design and Implementation_. 13–18.
  * ASHRAE/ANSI Standard 55-2010 American Society of Heating, Refrigerating, and Air-Conditioning Engineers (2010) ASHRAE/ANSI Standard 55-2010 American Society of Heating, Refrigerating, and Air-Conditioning Engineers. 2010.  Thermal environmental conditions for human occupancy.  _Inc.Atlanta, GA, USA_ (2010).
  * Barrett et al. (1993) Judith Barrett, Leon Lack, and Mary Morris. 1993.  The sleep-evoked decrease of body temperature.  _Sleep_ 16, 2 (1993), 93–99.
  * Bass (2019) Lauren Bass. 2019.  The Concealed Cost of Convenience: Protecting Personal Data Privacy in the Age of Alexa.  _Fordham Intell. Prop. Media & Ent. LJ_ 30 (2019), 261.
  * Bovornkeeratiroj et al. (2020) Phuthipong Bovornkeeratiroj, Srinivasan Iyengar, Stephen Lee, David Irwin, and Prashant Shenoy. 2020.  Repel: A utility-preserving privacy system for iot-based energy meters. In _2020 IEEE/ACM Fifth International Conference on Internet-of-Things Design and Implementation (IoTDI)_. IEEE, 79–91.
  * Caine (2009) Kelly Caine. 2009.  _Exploring everyday privacy behaviors and misclosures_.  Georgia Institute of Technology.
  * Cambeiro (2019) João Miguel Reis Araújo Proença Cambeiro. 2019.  _The Human in the loop in Cyber-Physical Systems: the case of Building Automation_.  Ph. D. Dissertation.
  * campbellsci (2022) campbellsci. 2022.  black globe thermometer.  Retrieved June 10, 2022 from https://www.campbellsci.com/blackglobe
  * Carroll (2006) Robert G Carroll. 2006.  _Elsevier’s Integrated Physiology E-Book_.  Elsevier Health Sciences.
  * Chen et al. (2021) Wuhui Chen, Xiaoyu Qiu, Ting Cai, Hong-Ning Dai, Zibin Zheng, and Yan Zhang. 2021.  Deep reinforcement learning for Internet of Things: A comprehensive survey.  _IEEE Communications Surveys & Tutorials_ 23, 3 (2021), 1659–1692.
  * Elmalaki (2021) Salma Elmalaki. 2021.  Fair-iot: Fairness-aware human-in-the-loop reinforcement learning for harnessing human variability in personalized iot. In _Proceedings of the International Conference on Internet-of-Things Design and Implementation_. 119–132.
  * Elmalaki et al. (2018a) Salma Elmalaki, Yasser Shoukry, and Mani Srivastava. 2018a.  Internet of Personalized and Autonomous Things (IoPAT): Smart Homes Case Study. In _Proceedings of the 1st ACM International Workshop on Smart Cities and Fog Computing_. ACM, 35–40.
  * Elmalaki et al. (2018b) Salma Elmalaki, Huey-Ru Tsai, and Mani Srivastava. 2018b.  Sentio: Driver-in-the-loop forward collision warning using multisample reinforcement learning. In _Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems_. 28–40.
  * EMOTIV (2020) EMOTIV. 2020.  EMOTIV EPOC+ 14 Channel Mobile Brainwear.  https://www.emotiv.com/product/emotiv-epoc-14-channel-mobile-eeg/.  Last accessed June 2023.
  * Erdemir et al. (2020) Ecenaz Erdemir, Pier Luigi Dragotti, and Deniz Gündüz. 2020.  Privacy-Aware Time-Series Data Sharing With Deep Reinforcement Learning.  _IEEE Transactions on Information Forensics and Security_ 16 (2020), 389–401.
  * Eslinger and Damasio (1985) Paul J Eslinger and Antonio R Damasio. 1985.  Severe disturbance of higher cognition after bilateral frontal lobe ablation: patient EVR.  _Neurology_ 35, 12 (1985), 1731–1731.
  * Fanger (1970) Poul O Fanger. 1970.  Thermal comfort. Analysis and applications in environmental engineering.  _Thermal comfort. Analysis and applications in environmental engineering._ (1970).
  * Fraenkel (2004) Aviezri S Fraenkel. 2004.  Complexity, appeal and challenges of combinatorial games.  _Theoretical Computer Science_ 313, 3 (2004), 393–415.
  * Garcelon et al. (2022) Evrard Garcelon, Kamalika Chaudhuri, Vianney Perchet, and Matteo Pirotta. 2022.  Privacy amplification via shuffling for linear contextual bandits. In _International Conference on Algorithmic Learning Theory_. PMLR, 381–407.
  * Garcelon et al. (2021) Evrard Garcelon, Vianney Perchet, Ciara Pike-Burke, and Matteo Pirotta. 2021.  Local differential privacy for regret minimization in reinforcement learning.  _Advances in Neural Information Processing Systems_ 34 (2021), 10561–10573.
  * Gerber (2014) Michael Gerber. 2014.  energyplus energy Simulation Software.  (2014).
  * Hadfield-Menell et al. (2016) Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. 2016.  Cooperative inverse reinforcement learning.  _Advances in neural information processing systems_ 29 (2016).
  * Hans et al. (2008) Alexander Hans, Daniel Schneegaß, Anton Maximilian Schäfer, and Steffen Udluft. 2008.  Safe exploration for reinforcement learning.. In _ESANN_. 143–148.
  * Hlávka (2020) Jakub P Hlávka. 2020.  Security, privacy, and information-sharing aspects of healthcare artificial intelligence.  In _Artificial Intelligence in Healthcare_. Elsevier, 235–270.
  * Hong et al. (2022) Junyuan Hong, Zhangyang Wang, and Jiayu Zhou. 2022.  Dynamic privacy budget allocation improves data efficiency of differentially private gradient descent. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_. 11–35.
  * Ibáñez et al. (2014) María Blanca Ibáñez, Ángela Di Serio, Diego Villarán, and Carlos Delgado Kloos. 2014.  Experimenting with electromagnetism using augmented reality: Impact on flow student experience and educational effectiveness.  _Computers & Education_ 71 (2014), 1–13.
  * Io (2011) T Io. 2011.  economic and Business Dimensions online advertising, Behavioral targeting, and Privacy.  _CommunICatIons of the aCm_ 54, 5 (2011).
  * Jin et al. (2017) Richeng Jin, Xiaofan He, and Huaiyu Dai. 2017.  On the tradeoff between privacy and utility in collaborative intrusion detection systems-a game theoretical approach. In _Proceedings of the Hot Topics in Science of Security: Symposium and Bootcamp_. 45–51.
  * Jung and Jazizadeh (2017) Wooyoung Jung and Farrokh Jazizadeh. 2017.  Towards integration of doppler radar sensors into personalized thermoregulation-based control of HVAC. In _Proceedings of the 4th ACM International Conference on Systems for Energy-Efficient Built Environments_. ACM, 21.
  * KhanAcademy (2021a) KhanAcademy. 2021a.  AP College Biology.  https://www.khanacademy.org/science/ap-biology.  Accessed: 2021-10-07.
  * KhanAcademy (2021b) KhanAcademy. 2021b.  AP College Physics.  https://www.khanacademy.org/science/ap-physics-2.  Accessed: 2021-10-07.
  * KhanAcademy (2021c) KhanAcademy. 2021c.  Organic Chemistry.  https://www.khanacademy.org/science/organic-chemistry.  Accessed: 2021-10-07.
  * Kosta et al. (2022) Adarsh Kumar Kosta, Malik Aqeel Anwar, Priyadarshini Panda, Arijit Raychowdhury, and Kaushik Roy. 2022.  RAPID-RL: A Reconfigurable Architecture with Preemptive-Exits for Efficient Deep-Reinforcement Learning. In _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 7492–7498.
  * Lim et al. (2008) Chin Leong Lim, Chris Byrne, and Jason KW Lee. 2008.  Human thermoregulation and measurement of body temperature in exercise and clinical settings.  _Annals Academy of Medicine Singapore_ 37, 4 (2008), 347.
  * Malekzadeh et al. (2020) Mohammad Malekzadeh, Dimitrios Athanasakis, Hamed Haddadi, and Ben Livshits. 2020.  Privacy-preserving bandits.  _Proceedings of Machine Learning and Systems_ 2 (2020), 350–362.
  * MATLAB (2022) MATLAB. 2022.  Thermal Model of a House.  Retrieved June 10, 2022 from https://www.mathworks.com/help/simulink/slref/thermal-model-of-a-house.html
  * Matsubara et al. (2022) Yoshitomo Matsubara, Marco Levorato, and Francesco Restuccia. 2022.  Split computing and early exiting for deep learning applications: Survey and research challenges.  _Comput. Surveys_ 55, 5 (2022), 1–30.
  * McKeachie and Svinicki (2006) W. J. McKeachie and M. Svinicki. 2006.  _McKeachie’s teaching tips: Strategies, research, and theory for college and university teachers (12th ed.)_.  Houghton-Mifflin.
  * Mishra et al. (2022) Shilpi Mishra, Divyapratap Singh, Divyansh Pant, and Akash Rawat. 2022.  Secure Data Communication Using Information Hiding and Encryption Algorithms. In _2022 Second International Conference on Artificial Intelligence and Smart Energy (ICAIS)_. IEEE, 1448–1452.
  * Nguyen et al. (2016) Anh Nguyen, Raghda Alqurashi, Zohreh Raghebi, Farnoush Banaei-Kashani, Ann C Halbower, and Tam Vu. 2016.  A lightweight and inexpensive in-ear sensing system for automatic whole-night sleep stage monitoring. In _Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems_. 230–244.
  * Olivier et al. (2003) Berend Olivier, Theo Zethof, Tommy Pattij, Meg van Boogaert, Ruud van Oorschot, Christina Leahy, Ronald Oosting, Arjan Bouwknecht, Jan Veening, Jan van der Gugten, et al. 2003\.  Stress-induced hyperthermia and anxiety: pharmacological validation.  _European journal of pharmacology_ 463, 1-3 (2003), 117–132.
  * Omonkhoa (2021) Liberty Adams Omonkhoa. 2021.  A review of the issues and challenges in IoT security using machine learning techniques.  (2021).
  * Pan et al. (2019) Xinlei Pan, Weiyao Wang, Xiaoshuai Zhang, Bo Li, Jinfeng Yi, and Dawn Song. 2019.  How You Act Tells a Lot: Privacy-Leaking Attack on Deep Reinforcement Learning.  _AAMAS_ (2019).
  * Papst et al. (2022) Franz Papst, Naomi Stricker, Rahim Entezari, and Olga Saukh. 2022.  To Share or Not to Share: On Location Privacy in IoT Sensor Data. In _2022 IEEE/ACM Seventh International Conference on Internet-of-Things Design and Implementation (IoTDI)_. IEEE, 128–140.
  * Patel et al. (2022) Neel Patel, Reza Shokri, and Yair Zick. 2022.  Model explanations with differential privacy. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_. 1895–1904.
  * Petrescu and Krishen (2018) Maria Petrescu and Anjala S Krishen. 2018.  Analyzing the analytics: data privacy concerns.  , 41–43 pages.
  * Sadigh et al. (2017) Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. 2017.  _Active preference-based learning of reward functions_.
  * Shany et al. (2012) Tal Shany, Stephen J Redmond, Michael R Narayanan, and Nigel H Lovell. 2012.  Sensors-based wearable systems for monitoring of human movement and falls.  _IEEE Sensors Journal_ 12, 3 (2012), 658–670.
  * Stirapongsasuti et al. (2019) Sopicha Stirapongsasuti, Wataru Sasaki, and Keiichi Yasumoto. 2019.  Decision making support for privacy data upload in smart home. In _Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers_. 214–217.
  * Taherisadr et al. (2023a) Mojtaba Taherisadr, Mohammad Abdullah Al Faruque, and Salma Elmalaki. 2023a.  Erudite: Human-in-the-loop iot for an adaptive personalized learning system.  _IEEE Internet of Things Journal_ (2023).
  * Taherisadr et al. (2023b) Mojtaba Taherisadr, Stelios Andrew Stavroulakis, and Salma Elmalaki. 2023b.  adaparl: Adaptive privacy-aware reinforcement learning for sequential-decision making human-in-the-loop systems.  _arXiv preprint arXiv:2303.04257_ (2023).
  * Teerapittayanon et al. (2016) Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. 2016.  Branchynet: Fast inference via early exiting from deep neural networks. In _2016 23rd international conference on pattern recognition (ICPR)_. IEEE, 2464–2469.
  * Terai et al. (2020) Shogo Terai, Shizuka Shirai, Mehrasa Alizadeh, Ryosuke Kawamura, Noriko Takemura, Yuki Uranishi, Haruo Takemura, and Hajime Nagahara. 2020.  Detecting Learner Drowsiness Based on Facial Expressions and Head Movements in Online Courses. In _Proceedings of the 25th International Conference on Intelligent User Interfaces Companion_. 124–125.
  * Times (2022) The New York Times. 2022.  The Best Smart Thermostat.  https://www.nytimes.com/wirecutter/reviews/the-best-thermostat/.  Accessed: 2022-06-10.
  * Wang and Hegde (2019) Baoxiang Wang and Nidhi Hegde. 2019.  Privacy-preserving q-learning with functional noise in continuous spaces.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Wu et al. (2023) Yashuo Wu, Carla Fabiana Chiasserini, Francesco Malandrino, and Marco Levorato. 2023.  Enhancing Privacy in Federated Learning via Early Exit. In _Proceedings of the 5th workshop on Advanced tools, programming languages, and PLatforms for Implementing and Evaluating algorithms for Distributed systems_. 1–5.
  * Zhang et al. (2022) Wanrong Zhang, Olga Ohrimenko, and Rachel Cummings. 2022.  Attribute privacy: Framework and mechanisms. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_. 757–766.
  * Zhao et al. (2023) Tianyu Zhao, Mojtaba Taherisadr, and Salma Elmalaki. 2023.  FAIRO: Fairness-aware Adaptation in Sequential-Decision Making for Human-in-the-Loop Systems.  _arXiv preprint arXiv:2307.05857_ (2023).