Towards Safe Load Balancing based on Control Barrier Functions and Deep Reinforcement Learning

Lam Dinh, Pham Tran Anh Quang, Jérémie Leguay

Huawei Technologies Ltd., Paris Research Center, France

{lam.dinh, phamt.quang<EMAIL_ADDRESS>
drl algorithms have recently made significant strides in improving network performance. Nonetheless, their practical use is still limited in the absence of safe exploration and safe decision-making.
In the context of commercial solutions, reliable and safe-to-operate systems are of paramount importance.
Taking this problem into account, we propose a safe learning-based load balancing algorithm for sdwan, which is empowered by drl combined with a cbf. It safely projects unsafe actions into feasible ones during both training and testing, and it guides learning towards safe policies. We successfully implemented the solution on gpu to accelerate training by approximately 110x times and achieve model updates for on-policy methods within a few seconds, making the solution practical. We show that our approach delivers near-optimal qos performance in terms of end-to-end delay while respecting safety requirements related to link capacity constraints. We also demonstrated that on-policy learning based on ppo performs better than off-policy learning with ddpg when both are combined with a cbf for safe load balancing.

sdwan, drl, cbf.

§ INTRODUCTION

Many enterprises are adopting sdwan <cit.>  technologies to trade-off between cost-effectiveness and qos satisfaction. Relying on a network overlay, this architecture allows businesses to interconnect multiple sites (enterprise branches, headquarter, data centers) without the need to deploy their own physical infrastructure, making it cost effective. A centralized controller maintains a set of policies deployed at access routers which send traffic
to their peers over several transport networks (e.g., private lines, broadband internet, 5G).
Typically, access routers are responsible for enforcing traffic engineering, for instance load balancing,
and queuing policies to meet sla requirements in terms of
end-to-end qos, security, etc. At a slow
pace, the controller maintains policies, while access devices
make real-time decisions for every flow. sdwan provides a cost efficient alternative to private lines and drastically improves qos compared to best effort solutions such as vpn.

Several solutions have been proposed for load balancing in sdwan networks to satisfy sla requirements. Centralized and distributed path selection mechanisms have been proposed to maximize an utility function <cit.> or sla satisfaction using a performance model <cit.>. To optimize latency and other qos parameters for a specific set of flows, closed-form performance models, using network calculus or queuing models, can be embedded into routing optimization algorithms. For instance, authors in <cit.> considered the Kleinrock function <cit.> to minimize latency. However, accurate analytical models for end-to-end qos performance metrics can be difficult to derive and integrate into routing optimization algorithms. Indeed, unknown scheduling parameters, behaviors inside the wan and transport-layer mechanisms are, in practice, too difficult to capture by tractable performance models. The challenge to integrate accurate analytical
models makes qos routing and lod balancing a great opportunity for
model-free solutions.

Instead of having a predefined performance model, rl agents
can interact with the environment and evaluate the outcomes of their actions thanks to a reward function.
drl <cit.> combines dl and rl principles (e.g., parameterize policies with neural networks). It has been first applied for routing to optimize network utility
under the umbrella of experience-driven networking <cit.>. Since, several single-agent and multi-agent drl solutions have been proposed to tune queues and load balancing policies to satisfy qos requirements or minimize congestion  <cit.>. However, even if drl has demonstrated a tremendous potential for improving sdwan performance,
most of the literature only focuses on off-policies and their performance once training has converged, without paying attention to safety during both learning and testing. In addition, as  network environments often drift, frequent retraining of the full model to adapt to changes can be cumbersome. In drl, the trial-and-error process is crucial for environment exploration and learning but, at the same time, unsafe actions cannot be deployed as they degrade network quality in production systems. For example, a poor load balancing policy might cause severe congestion and accidentally degrades the overall network performance.
Therefore, considering safety during both training phase, in particular for more practical on-policies, and testing is key for the wide adoption of rl algorithms in network systems.

Taking those issues into consideration, this work seeks to complement current drl-based load balancing solutions with an additional safety shield.
Based on the safe learning approach primarily presented in <cit.>, which is designed for critical robotic systems, we propose a safe load balancing solution for sdwan. Our solution achieves
near-optimal qos performance in terms of average end-to-end delay, while safety concerns related to the violation of link capacity constraints are fully considered.

The contributions of our work are the following. First, we describe the target sdwan system and formulate a load balancing problem to minimize the average tunnel latency that can efficiently be solved with rl. Then, we design a dedicated cbf based on local search to deliver safety on top of gradient-based drl algorithms (e.g., off/on policy learning).
Finally, we evaluate our drl-cbf solution in both training and testing phases. We compare our solution to traditional learning algorithms (e.g., ddpg, ppo) where safety is only handled in the reward function, without any strict guarantees. We show that our solution can minimize latency while providing full safety guarantees. In a controlled environment, we also demonstrate that the qos obtained is very close to the optimal solution derived from a non linear integer model solved with the scip <cit.> solver. In addition, in terms of execution time, we implemented drl-cbf algorithms on  gpu and managed to accelerate training by approximately 110x times and achieve model updates for on-policy methods within a few seconds, making the full solution practical.

The paper is organized as follows. Related work is discussed in Section <ref>. Section <ref> presents the system model and formulates the load balancing problem. Section <ref> describes the proposed solution, whereas Section <ref> provides numerical results, demonstrating our proposed algorithm performance. Finally, Section <ref> concludes the paper.

§ RELATED WORK

drl algorithms for sdwan controllers has been proven to improve overall network performance <cit.>. For instance, Troia et al. <cit.> have shown a target qos can be achieved through the proper design of the reward function. Similar results have been achieved using multiple agents <cit.>.
However, as mentioned before, most of the literature only focuses on 1) off-policies and 2) network performance  without paying attention to safety.

Indeed, RL-based load balancing systems may violate capacity constraints both in training and testing phases. When globally minimizing the end-to-end delay, unsafe actions creating congestion and violating  capacity may be taken to get rid of portions of the traffic and improve the reward. To deal with such abnormal behaviors, the LearnQueue <cit.> reward has been introduced to minimize the end-to-end delay while penalizing traffic rejections.
However, it requires to weight properly the two objectives, which can be tedious in practice since it heavily depends on the environment.
To address these limitations and avoid manual parameter tuning, the first work to systematically optimize qos under safety constraints for load balancing has been presented by Kamri et al. <cit.>.
This work applies the rcpo algorithm<cit.> where the reward integrates traffic rejection as a constraint using  Lagrangian relaxation. During training, the algorithm finds the optimal Lagrangian multiplier. Following this work, Zhang et al.
<cit.> investigate a path planning problem based on constrained policy iteration to improve the performance of multiple path selection. A safe load balancing strategy for ultra-dense network is also discussed by Huang et al. <cit.>. In this work, they proposed a proactive load balancing algorithm on top of cpo <cit.>, which has been proven to guarantee optimal policy under (safety) constraints.

Although several papers have presented drl algorithms for load balancing  <cit.> with some constraints<cit.>), they 1) all provide soft guarantees during exploration  and 2) mostly ensure safety during exploitation. To mitigate these issues, we propose to employ a cbf <cit.>  on top of current drl algorithms for (safe) load balancing optimization with hard guarantees.

§ SYSTEM MODEL

Figure <ref> presents a typical sdwan use case where the headquarter and 3 branches of an enterprise are interconnected either via an Internet connection and a mpls private line. Traffic is issued by applications at both headquarter and branches. $6$ OD (Origin-Destination) flows, also called tunnels, are considered, one per headquarter and branch pair in each direction. Each tunnel has two paths for Internet and mpls. A lb agent at each ar splits the traffic according to the policy received by the centralized controller.

SD-WAN network with an headquarter and 3 branches.

This overlay network can be modeled as a graph where core links represent overlay links and edge links correspond to wan ports at ar routers. As depicted in Figure <ref>, links $N'-N$ at ports $N$ are used to model the egress / ingress capacity provisioned at each transport network. In practice, wan ports can receive traffic from multiple sites, potentially of higher capacity, and they might be congested in high load conditions. Let formally consider a graph $G=(V,E)$ where $V$ is the set of nodes and $E$ is the set of edges.
Each tunnel $k$ in a set of tunnels $K$ can use a set of candidate paths denoted as  $\gls{pk}$ (e.g., Internet and mpls) to load balance traffic. Each edge $e$ carries an instantaneous load $l_e$ and has a capacity $\gls{ce}$. Let denote  $\gls{Dtk}$
the traffic demand
of tunnel k  at time t. Each lb agent applies at time $t$ a split ratio $\gls{xpk}$ for each tunnel $k$ over each path $p \in \gls{pk}$
($\gls{xpk} \in [0,1]$ and $\sum_{p \in \gls{pk}} \gls{xpk}=1\ \forall k \in K$).

The delay on each path $p$ for a tunnel $k$ is denoted $\gls{dpk}$ and the tunnel delay, denoted $\gls{dk}$ is computed as follows:

\begin{equation}
\gls{dk} = \underset{p \in \gls{pk} }{\max} \quad \gls{dpk}
\end{equation}

In practice, $\gls{dpk}$ is continuously measured by access routers.

Problem formulation. The main objective is to derive an optimal load balancing policy so that the sdwan overlay delivers the best qos. In the rest of the paper, we consider as primary target the  minimization of the average tunnel delay under the constraint that link
capacity constraints are not violated (safety constraint). Indeed, this safety measure prevents that 1) congestion is induced by miss configured split ratios and that 2) the average tunnel delay is not artificially minimized by rejecting traffic (i.e., by intentionally creating some congestion).
To prevent high link delays, or very heterogeneous ones even if the average delay is low, a common practice is to enforce a mlu $\mu \in [0,1]$ over all links.
lb agents at headquarter and branches are configured by a network controller.

In this case, load balancing policies, which are derived according to dynamic tunnel traffic Dtk, solve the following optimization problem:

\begin{align*}
\underset{\gls{xpk}}{\min} ~~~~~~~&   \frac{\sum_{k\in K}\gls{dk} }{K}\tag{$\mathcal{P}$} \label{eq:P0}\\
\text{s.t.}~~~~~& \sum_{k \in K}  \sum _{i=0, p \in \gls{pk}} ^{|p|} \gls{Dtk}.x^k_i \leq \mu . c_e ~~~~~ \forall e \in E \tag{$\mathcal{C}_0$} \label{eq:C0} \\
& \sum _{i=0} ^{|p|} x^k_i =1  ~~~~~~~~~~~~~~~~~~~~~~~~\forall x^k_i \in [0,1]\tag{$\mathcal{C}_1$} \label{eq:C1} \\
%     & 0 \leq \gls{mlue} \leq 1 & \forall e \in E \tag{$\mathcal{C}_2$} \label{eq:C2} \\
\end{align*}

where problem
(<ref>) minimizes the average tunnel delay. Constraints (<ref>) guarantee that the traffic over each edge $e$ in the network is kept under the mlu.
Constraints (<ref>) ensure that splits ratios sum to $1$.

§ LEARNING-BASED AND SAFE LOAD BALANCING

In this section, we propose a constrained policy optimization for load balancing based on  drl algorithms and a cbf <cit.>.

§.§ Learning-based optimization

Our optimization problem can be formulated as a mdp which is defined by the tuple  $\left<\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{T},\gamma  \right>$ where $\mathcal{S}$ represents the set of states, $\mathcal{A}$ is the set of available actions, $\mathcal{R}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow \mathbb{R}$ is the reward function which gives the reward for the transition from one state to another given an action, $\mathcal{T}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow [0,1]$ is the transition matrix, which gives the probabilities of transitioning from one state to another given an action
and $\gamma \in [0,1]$ is the discount factor. A policy $\pi: S \rightarrow  A$ refers to the probability of taking an action $a \in \mathcal{A}$ under state $s \in \mathcal{S}$. The agent iteratively interacts with the (networking) environment to learn an optimal policy $\pi^*$ that chooses actions with the best payoff. To solve the mdp associated with problem (<ref>), a centralized controller is employed in the network.  The controller can periodically collect at every time $t$ information from the different ar routers: the traffic demand $\gls{Dtk}$, the delay $\gls{dk}$ of each tunnel $k\in K$ and the maximum link utilization $\mu$. In this context, we consider the observation space or state $s_t$ as the set of traffic demands $\gls{Dtk}$ for all tunnels $k \in K$. The action space is determined as the set of split ratios $\gls{xpk}$ for all paths $p \in \gls{pk}$ of each tunnel $k \in K$.

By using rl algorithms, we learn the optimal stochastic control policy $\pi(a|s)$ that maximizes the performance measure (i.e., $J(\pi)$) which is expressed as follows:

\begin{equation}
J(\pi) = \mathbb{E}_{\tau \sim \pi} \left[ \sum_t ^\infty \gamma^t r_t(s_t,a_t) \right]
\end{equation}

where $\tau \sim \pi$ denotes a trajectory during which actions are sampled according to the policy $\pi(a|s)$.

Given the fact that policy gradient  rl methods have shown good performance in continuous control problem <cit.> <cit.>, in the rest of this paper, we consider off-policy learning (e.g. ddpg, td3) and on-policy learning (e.g., ppo) algorithms. In the former technique, actions are sampled to encourage the agent to explore the environment. Then, a target deterministic policy is derived from these actions, supported by the actor-critic architecture <cit.> with a replay buffer <cit.>. On the other hand, on-policy methods iteratively derive the target policy based on samples obtained from their own previous actions and the update is carried out by solving the following optimization problem:

\begin{align}
& \pi_{i+1} = \underset{\pi}{argmax}  \sum_s \rho_{\pi_i} (s) \sum_a \pi(a|s) A^{\pi_i}(s,a) \\
& \text{s.t.} ~~~~ KL\left[\pi_{i+1}(.|s_t),\pi_{i}(.|s_t) \right]  \leq \delta_{KL}
\end{align}

where $A^{\pi_i}(s,a)$ is the advantage of performing action $a$ (sampled by old policy $\pi_i$) at state $s$, $\rho_{\pi_i} (s)$ represents the frequency of visit of state $s$ under policy $\pi_i$  and $KL\left[\pi_{i+1}(.|s_t),\pi_{i}(.|s_t) \right]$ refers to Kullback-Leibler divergence between new policy and old policy <cit.> and this value is bounded by a target $\delta_{KL}$. Both on and off-policy gradient algorithms do not consider safety, therefore our goal is to complement these model-free learning algorithms with a cbf which guarantees safety during exploration and exploitation.

Safety-based actor-critic learning architecture.

Figure <ref> introduces the control plane architecture where each lb agent is centrally configured by the controller. The controller can periodically query the network state (i.e., tunnel delays, OD traffic, MLU) at each lb and its control system is composed by two main blocks that focus on policy optimization and safety, respectively. The former block, which is based on an actor-critic  architecture<cit.>, learns the (unconstrained) optimal policy.
Accordingly, the value function and the policy are approximated using two different neural networks, parameterized by $\phi$ and $\theta$, respectively. Each learning parameter is responsible for either maximizing the actor's objective function (i.e., $\mathcal{J}_b^A$) or the critic's loss function (i.e., $\mathcal{L}_b^C$). The specific expressions for $\mathcal{J}_b^A$ and $\mathcal{L}_b^C$ heavily depend on the off/on policy reinforcement learning algorithms being used. For instance, the explicit functions for ddpg (off-policy) and ppo (on-policy) algorithms  can be respectively found in <cit.> and <cit.>.
The details of the optimization procedure are shown in Algorithm <ref>.

In order to enforce safe exploration during learning and safe policy execution during testing, a safety block, based on a cbf function <cit.>, is implemented on top of drl algorithms.

RL-based optimization algorithm
Initialize  parameterized rl policy $\pi_{\theta_0}$

Initialize value function parameter $\phi_0$

Define episode length $L_{eps}$, batch size $B$, total episodes $N_{eps}$

Initialize experience array $\gls{exp}=\{\oslash\}$

episode $k < N_{eps}$
time step $t=1,...,L_{eps}$
Observe state $s_t$

Sample action $a_t \sim \pi_{\theta_k}(.|s_t)$

Perform local search algorithm

\begin{equation*}
{\colorbox{gray}{$\gls{acbf}_t = Local\_Search(s_t,a_t)$}}
\end{equation*}

Deploy action $\gls{acbf}_t$ and obtain reward $r_t$, next state $s_{t+1}$

Store experience tuple $<s_t,\gls{acbf}_t,s_{t+1},r_t>$ in $\gls{exp}$

batch sampling $B$ experiences in exp
Perform gradient ascent on actor network

\begin{equation*}
\theta_{k+1} \leftarrow \frac{1}{|B|}\underset{\theta_k}{\arg \max} \sum _{b=0}^{|B|} \mathcal{J}_b^A(\theta_k)
\end{equation*}

Perform gradient descent on critic network

\begin{equation*}
\phi_{k+1} \leftarrow \frac{1}{|B|}\underset{\phi_k}{\arg \min} \sum _{b=0}^{|B|} \mathcal{L}_b^C(\phi_k)
\end{equation*}

return parameterized policy $\pi_\theta$, parameterized value function $\phi$

§.§ Safe policy exploration and exploitation

The CBF function serves as a projector to convert the proto-policy $\pi_\theta$, which is the parameterized actor network from which unsafe actions  might cause congestion, into a safe policy $\pi^{CBF}$. Its main objective is to guarantee that the mlu $\mu$ remains below $1$ (i.e., $\mu(\gls{acbf}) \leq 1$, $\forall \gls{acbf} = \pi^{CBF}(.|s)$).
As illustrated by Figure <ref>, this  projection maintains the safe policy $\pi^{CBF}$ as close as possible to the proto policy and keeps load balancing system in safety condition. Safe policy projection is performed at the centralized controller in both learning and testing. Following the safe policy $\pi^{CBF}$,
each cbf action is determined according to the following optimization problem:

\begin{align} \label{eq:safe}
\begin{split}
&\gls{acbf}= \underset{\gls{acbf}_t}{argmin}\left\|\gls{acbf}_t - \gls{arl}\right\|_1 \\
\textit{s.t.} ~~~~~~~~~&  \gls{acbf} \in \mathcal{A} \\
& \mu(\gls{acbf}) \leq 1
\end{split}
\end{align}

From proto-policy to safe policy with CBF.

With respect to reward function design, we highlight that the objective function of problem (<ref>) is enough to minimize the average tunnel delay in the network.
However, as mentioned in Section <ref>, when only  minimizing the average delay, the system may induce large delays for a small number of tunnels due to a high link utilization or congestion, i.e. packet drops to further reduce the average delay. Therefore, in order to make the drl agent also aware of link utilization, we design our reward function to be the weighted sum of the average delay and the mlu (i.e., $\mu$) as given by Equation <ref>.

\begin{equation} \label{eq:rew}
r_t(s_t,a_t)=-\sigma \frac{\sum_{k \in K} d_{k,t}}{\left|K \right|} - (1-\sigma )\mu
\end{equation}

where $\sigma \in [0,1]$ emphasizes the importance of the average tunnel delay over a low mlu. While this reward cannot guarantee itself a hard safety, it guides rl agent in learning a policy which is both qos optimal and safe after convergence. To ensure anytime and hard safety, we explain in the following how we incorporate a cbf function on top of drl algorithms to constrain the exploration and exploitation so that the mlu does not exceed 1 (i.e., $\mu\leq 1$).

CBF function. The cbf function, applied on top of drl algorithms, brings two main benefits. Firstly, it guarantees that any action, which are stochastically sampled from the parameterized actor network, will not cause a during both exploration and exploitation. Secondly, it considerably contributes to obtaining a better reward-per-learning-step as the mlu is part of the reward. Indeed, by correcting "bad" proto-actions, which produce a high mlu and overloaded links, the output safe action  results in a better or at least equal mlu compared to the initial proto-action. It leads to a fast convergence towards an optimal safe policy. For the projection of proto-actions into safe actions, we propose to use as a cbf function the local search algorithm detailed in Algorithm <ref>.

In principle, given a proto-action, the cbf  stochastically attempts to generates $N$ safe actions within a neighborhood of radius $\delta_s$. The generation of actions is based on three different policies where the information with regards to the link utilization of each path $p$ in the tunnel $k$ is exploited:

cbf based on Local Search algorithm
Returned action from RL agent $a_{\theta}^{RL}$

Local search radius $\delta_s$

CBF solutions $N$

Local search policy $\pi^{CBF}$

Local search max iteration $M$

Feasible solution $feas\_sol=\{\}$

$\mu(a_{\theta}^{RL}) \leq 1$$\gls{acbf}=a_{\theta}^{RL}$

$m$ in $M$
$n$ in $N$
Randomly sample $\epsilon _n \sim Uniform(0,\delta_s)$

Stochastic action generation $a_n^{CBF} \sim \pi^{CBF}$

Perform action update: $a_n^{CBF} = a_{\theta}^{RL} \pm \epsilon _n $

$\mu(a_n^{CBF})  \leq 1$Append $a_n^{CBF}$ to $feas\_sol$

$feas\_sol \neq \oslash $$\gls{acbf}=\underset{a_i^{CBF} \in feas\_sol}{argmin}\left\|a_i^{CBF} - a_{\theta}^{RL} \right\|_1$
$\gls{acbf}= \underset{n}{argmin}\quad \mu\{a_n^{cbf}\}$

* Naive policy: This policy randomly picks any tunnel $k \in K$. For each selected tunnel, a random value $\epsilon \sim Uniform(0,\delta_s)$ is added/subtracted to current split ratios given by the proto-action on each path $p \in \gls{pk}$. In particular, the traffic load on the highest utilization path $p$ of the selected tunnel $k$ will be reduced by an amount of $\epsilon.T^k_{t,p}$. This traffic amount will be equally distributed to the remaining paths of the tunnel. In our scenario where there are only 2 paths per tunnel, the reduced traffic on the path with higher utilization will be directly transferred to the path with lower utilization.

* DeltaUtil policy: This policy selectively focuses on tunnels where the difference between their highest path utilization and lowest path utilization is greater than a certain threshold. In our case, a threshold on the difference of 50 $\%$ is chosen.  Once this criteria is met, a randomly generated value $\epsilon \sim Uniform(0,\delta_s)$ is used to fine-tune split ratios in the proto-action, similarly to the Naive policy.

* MaxUtil policy: This new policy inherits the principles of the DeltaUtil policy policy, but it uses a different criteria for selecting tunnels.
Specifically, any tunnel $k$ that has a path load utilization above a threshold of 100 $\%$ (e.g., $ \exists p \in \gls{pk} ~ | ~\mu_p \geq 1$, which is unsafe) will be the target for proto-action modification. Once the set of congested tunnels is determined, each random value $\epsilon_k \sim Uniform(0,\delta_s)$ will be used to adjust the initial split-ratio of each selected tunnel $k$, which is primarily decided by the proto-policy. As a consequence, an amount of traffic $\epsilon_k.T^k_{t,p}$ will be withdrawn from the path $p$ of tunnel $k$ with the highest path utilization, and added to remaining paths.

After generating a large number of actions around a proto action, a feasible action (i.e., mlu is below $100 \%$) is selected
in such a way that its distance to the original proto action is the smallest. The returned action is quasi-guaranteed to be safe and helps learning safe policies. As local search policies are heuristic, it may be that they cannot correct a proto-action for which a safe action exists. In this case, the cbf action, which returns the lowest mlu, will be selected.

Our local search algorithm is important in both training and exploitation phases. In the training phase, it helps to ensure that the network learns safe policies by eliminating the possibility of taking actions that could cause safety issues (i.e., violation of link capacity constraint). In the exploitation phase, it helps to ensure that the network continues to operate safely even under unseen network conditions.

§ RESULTS AND DISCUSSIONS

We now evaluate the performance of the DRL-CBF solution on the sdwan scenario presented in Section <ref>.

§.§ Network environment

Our simulations are carried out using Python's Gym toolkit <cit.>, which is a well-known api for interfacing between our (safe) learning algorithms and our Python-based sdwan environment.

In order to demonstrate the stochastic traffic behavior at each OD flows, we generate noisy sinusoidal traffic to model diurnal variations, as shown by Figure <ref>. The phase of each OD flow is shifted to make incoming traffic to each site somewhat variable and likely to cause congestion at  bottleneck links without appropriate load balancing.

Example of tunnels' traffic over a window of 1000s.

We adopt a simple M/M/1 queuing model to compute the delay on each link $e$ and considers a propagation delay $d_{prop}$. The link delay $\gls{de}$ is then derived as follows:

\begin{equation}
\gls{de} = d_{prop} + \frac{1}{\gls{ce}-l_e}
\end{equation}

In our scenario, each mpls and Internet link has a fixed capacity of $\gls{ce}=6Mbps$ and  $\gls{ce}=15Mbps$, respectively and  $d_{prop}$ represents the propagation delay.

The delay on each path $p$ of a tunnel $k$ (i.e. $\gls{dpk}$) is then calculated as the sum of the delay on the edge that is involved in that path as follows:

\begin{equation}
\gls{dpk} = \sum_{e \in p, p\in \gls{pk}} \gls{de}
\end{equation}

Furthermore, in order to emulate the behavior of tcp in the considered network, we adopted a min-max fairness rate allocation policy using a standard water-filling algorithm <cit.>.

§.§ Algorithms implementation

Relying on Stable-Baseline3 <cit.>, a well-known library of rl algorithms, we have exploited two different types of rl algorithms: off-policy with ddpg <cit.>) and on-policy with ppo <cit.>). Given that they do not guarantee  safety, we have applied the cbf function based on local search algorithms (Algorithm <ref>) on top of them.

System architecture with 1) network environment running on CPU and 2) safe RL algorithms on GPU.

We implemented the solution on a server composed by a cpu Intel® Xeon® Platinum 8164 (104 logical cores and 1024 GB of RAM memory) and a gpu NVIDIA®  Tesla V100 (5120 cuda cores and 32 GB of DRAM), as shown in Figure <ref>. As local search algorithms can be massively parallelized, we implemented them with cuda libraries so that they fully benefit from all gpu cores available. Therefore, model training and intensive local search algorithms are fully performed at gpu side. Once a safe policy is found, it is transferred to the cpu, where our sdwan environment is located, to perform a rollout step. The resulted tuple experiences are then moved back to gpu for further model training.

Simulation (hyper)parameters

(Hyper) parameters     1c|c]@c@Off-policy Algo
(DDPG)     c]@c@On-Policy Algo
1l|Gradient clipping maximum                                                          1l|[HTML]000000                                                      0.5
Clipping parameter   $\varepsilon$                                                                                1c|[HTML]000000                                                      0.2
Target KL divergence  $\delta_{KL}$                                                                                1c|[HTML]000000                                                      0.03
Delayed network update rate                                                                       1c|0.05                                                                          [HTML]000000
Learning rate                                                                                2c1e-5
Discounted factor  $\gamma$                                                                                2c0.7
Hidden layers                                                                                2c3
Dimension of hidden layer                                                                               2c512
Batch size  $|B|$                                                                                2c256
Frequency of model updates (steps)                                                                                2c256
Episode length $L_{eps}$ (steps)                                                                                2c128
Max local search iteration ($M$)                                                                           2c20
Local search solutions ($N$)                                                                           2c1.000
Local search radius ($\delta_s$)                                                                        2c0.3
Reward parameter $\sigma$                                                                               2c0.8
Training steps                                                                              2c1.000.000

Table <ref> enumerates the list of parameters that we applied. To stabilize the training process in ddpg, delayed network updates have been used so that the target actor/critic networks are updated less frequently than the main actor/critic by a factor of $0.05$. The clipped version of ppo  <cit.> has been used with a clipping parameter  $\varepsilon=0.2$ and a target $KL$ divergence  set at $\delta_{KL}=0.03$. To avoid destructively large weight updates, the gradient is also clipped at 0.5. Besides, a fully connected neural network with 3 layers of 512 neurons is used to approximate policy and value functions.  With regards to local search algorithms, we take the best solution out of  $N*M=20.000$ points generated around each unsafe action returned by the drl agent
in a radius of $\delta_s=0.3$. Finally, our models are updated after each rollout of 256 steps, which is equivalent to 2 episodes, and the learning process is studied in $t_s=1.000.000$ training steps.

Episodic training rewards

Traffic acceptance rate

[ Testing performance results ]
Episodic training reward and traffic acceptance rate at learning phase.

To obtain an optimal solution as benchmark, the following constraints (<ref>) are added to Problem <ref> so that the link delay is computed according to M/M/1 queuing model. The resulting Non Linear Problem (NLP) is solved by the scip <cit.> solver. Note that this constraint plays an important role for our benchmark but, in practice, it is not explicitly known because performance might not follow the M/M/1 model.

\begin{align*}
&\gls{de} \geq \frac{1}{c_e - \sum_{k\in K} \sum_{p \in \gls{pk}}\gls{lpk}} \quad \forall e \in E\tag{$\mathcal{C}_2$} \label{eq:C2} \\
\end{align*}

§.§ Obtained results

On baseline without safety. Figure <ref> compares the average training reward
during each episode  for ddpg and ppo, respectively. This figure highlights that ppo  typically achieves better rewards than ddpg. Besides, the ddpg agent tends to be trapped into local optimum policies, which can lead to sub-optimal performance.
In terms of safety, Figure <ref> illustrates the percentage of the total traffic  that is accepted due link capacity constraints.
Both algorithms encourage policy exploration at early stages in training, which accidentally causes link congestion and traffic rejection. A severe unsafe attempt of policy exploration for DDPG can be observed at episode 6000. However, even if it helps learning better policies,
unsafe exploratory actions should be avoided in production systems, especially for on-policy algorithms where the model is updated and applied after a smaller number of episodes.

Delay results of DDPG, PPO and NLP

Delay results of  DDPG-CBF and NLP

Delay results of PPO-CBF and NLP

MLU results of DDPG, PPO and NLP

MLU results of DDPG-CBF and NLP

MLU results of PPO-CBF and NLP

[ Local traffic and corresponding shaping rate, loss of each solution ]
Testing performance on average tunnel delay and MLU.

In order to compare performance during testing, our learning models using ddpg and ppo algorithms are selected after the last episode training. At this point, training models are well aware of delay and mlu minimization in the objective function. Therefore, traffic rejection due to link capacity violation is unlikely to happen. To define a testing scenario, we generated 100 traffic samples for each OD tunnel following the traffic pattern presented in Figure <ref>.
As illustrated in Figures <ref> and <ref>, delay and mlu of learning-without-safety models are compared to the optimal nlp solution obtained with scip. They reveal that near-optimal delays are obtained using conventional ddpg and ppo learning algorithms. Besides, the mlu during testing is safely kept below $1$, resulting in no traffic rejection. Furthermore, network delay is better for ppo compared to ddpg  since the local optimum trap is not observed during training.

With safety. When the safety cbf layers are applied on top of current off/on policy learning algorithms, testing performance results
are depicted in
Figures <ref>, <ref>, <ref> and <ref>, respectively. With respect to safety, off-policy learning using ddpg, Figures <ref> and <ref> demonstrate that the ddpg-cbf agent does not handle delay well, especially when the total traffic demand is high. In particular, many high delay peaks are observed during testing phase regardless cbf policies, although safety is always respected (mlu belows $1$).
ppo-cbf  significantly improves network delay and better controls the mlu in testing phase as illustrated by Figures <ref> and <ref>. Furthermore, no capacity violations happen during learning as indicated by Figure <ref>. These results suggest that, with safety, on-policy learning outperforms off-policy learning as near-optimal performance is obtained and hard safety requirements are met.

In addition, Figure <ref> also shows the training performance for the three cbf functions we implemented. As we can see, traffic acceptance during learning is significantly improved compared to the case without cbf, as illustrated in Figure <ref>. Moreover, the MaxUtil policy outperforms Naive and DeltaUtil  policies in achieving no traffic rejection during learning. Indeed, this policy directly targets tunnels that make the network congested. With respect to testing performance, Figure <ref>, <ref>, <ref> and <ref> show that all three cbf functions nearly get the same performance and that ppo-cbf performs much better than ddpg-cbf. Indeed, no delay spikes and smoother performance are observed for ppo-cbf.

To shed the light on the convergence of training rewards with and without cbf functions, Figure <ref> and <ref> compare the episodic training rewards of ddpg-cbf and ppo-cbf, respectively. These figures reveal that on-policy learning is faster and stable compared to off-policy learning.
Although ddpg-cbf achieves a better reward than its non-safe version, its training curve is slightly unstable compared to the training curves of ppo-cbf. As a result, combining safe learning with this off-policy approach is more challenging.

Reward of DDPG-CBF

Reward of PPO-CBF

Traffic acceptance rate

Calculation time

[ Training performance results ]
Training performance: (a,b) average episodic training reward, (c) traffic acceptance rate and (d) average computation time per step.

Figure <ref> demonstrates the benefits of our hardware architecture illustrated on Figure <ref> for training acceleration.
We compare the average, real execution time required to successfully perform one iteration of Algorithm <ref>, which includes local search algorithm and model update, with and without (i.e. cpu) using the gpu. It can be observed that the gpu implementation significantly speeds up calculation and training time. In particular, a feasible (safe) action is found in less than $0.1(s)$ when using gpu, compared to more than $11(s)$ using cpu. This significant time saving favors the usage of our methods in practice, since  the model can also be updated every 256 steps in roughly 25.6 seconds, rather than 2816 seconds using only cpu. As a result, our model can be fully trained in one day using 1.000.000 training steps instead of 134 days when using the cpu. This time acceleration has a significant impact on the possibility to deploy our solution in practice. Indeed, since network measurements (e.g., delay, traffic, loss, etc.) can be collected every $1$s, on-policy leaning models can be updated every $256$s in $25$s,
which is reasonable.

§ CONCLUSION

We presented a novel approach combining the drl and a cbf to guarantee safe exploration and exploitation in the context of sdwan. Tackling a typical load balancing problem where latency needs to be  optimized while meeting safety requirements in terms of capacity constraints, our drl-cbf approach is able to achieve near-optimal performance with safe exploration and exploitation. Furthermore, we show that on-policy optimization based on ppo achieves better performance than off-policy learning with ddpg. We implemented all the algorithms on GPU to accelerate training by approximately 110x times and achieve model updates for on-policy methods within a few seconds, making the full solution practical.

Future works along these lines include the integration with a network simulator and a testbed for a more realistic performance evaluation. We also plan to address more challenging environments where, for instance, QoE needs to be optimized and other constraints need to be handled.

§ ACKNOWLEDGMENT

This work was partially supported by the French Nation Research Agency (ANR) SAFE project  under grant ANR-21-CE25-0005.


# Leveraging LLMs to Predict Affective States via Smartphone Sensor Features

Tianyi Zhang, Songyan Teng, Hong Jia, Simon D’Alfonso  t.zhang59,
stteng@student.unimelb.edu.au,hong.jia<EMAIL_ADDRESS>University of
MelbourneAustralia

(2024; 18 June 2024)

###### Abstract.

As mental health issues for young adults present a pressing public health
concern, daily digital mood monitoring for early detection has become an
important prospect. An active research area, digital phenotyping, involves
collecting and analysing data from personal digital devices such as
smartphones (usage and sensors) and wearables to infer behaviours and mental
health. Whilst this data is standardly analysed using statistical and machine
learning approaches, the emergence of large language models (LLMs) offers a
new approach to make sense of smartphone sensing data. Despite their
effectiveness across various domains, LLMs remain relatively unexplored in
digital mental health, particularly in integrating mobile sensor data. Our
study aims to bridge this gap by employing LLMs to predict affect outcomes
based on smartphone sensing data from university students. We demonstrate the
efficacy of zero-shot and few-shot embedding LLMs in inferring general
wellbeing. Our findings reveal that LLMs can make promising predictions of
affect measures using solely smartphone sensing data. This research sheds
light on the potential of LLMs for affective state prediction, emphasizing the
intricate link between smartphone behavioral patterns and affective states. To
our knowledge, this is the first work to leverage LLMs for affective state
prediction and digital phenotyping tasks.

Large Language Models, Digital phenotyping, Mental wellbeing, Ubiquituous
computing, Mobile Sensing

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
2024 UbiComp Mental Health Sensing and Intervention workshop; October 6, 2024;
Melbourne, Australia††ccs: Human-centered computing Empirical studies in
ubiquitous and mobile computing

##  1\. Introduction

With the increasing prevalence of mental health issues among young adults,
particularly within the student population, there is significant rationale for
researching and developing systems to monitor mental health, including
affective states. Daily mood monitoring may aid early detection of potential
mental health issues in timely intervention and support (Beames et al., 2021).
Digital phenotyping, also called personal sensing, may serve as one of the
potential solutions. Utilizing digital devices to collect behavioral traces
seamlessly to analyse and predict mental health status, digital phenotyping is
a potentially effective tool to understand the relationship between human
activities and mental health or wellbeing.

Smartphones are a premium device for digital phenotyping due to their ubiquity
and diverse data collection capacities, including collection of multi-sensor
data (e.g. GPS and accelerator) and also captures of virtual social behaviors,
software usage, and screen interactions, thereby capturing the intricate
relationship between humans and technology. To deliver emotional wellbeing
predictions, powerful classic machine learning models such as deep neural
networks (LeCun et al., 2015), gradient boosting (Chen and Guestrin, 2016),
and transformers (Vaswani et al., 2017) often face overfitting issues and are
not suitable for small datasets typically found in mental healthcare.
Therefore, innovative and effective new approaches are required to assist and
improve digital phenotyping techniques and methods.

Recently, large language models (LLMs) have emerged as powerful tools for
natural language generation, natural language understanding, and contextual
comprehension. Their ability to leverage pre-trained knowledge makes them
applicable to various research problems across different domains,
demonstrating superior performance in tasks such as sentiment analysis
(Vaswani et al., 2017), activity recognition (Liu et al., 2023), and
information retrieval (Zhai and Lafferty, 2001). The inherent nature of
foundational LLMs, trained on vast online corpora that document human behavior
and psychological knowledge, underscores their potential to reveal more
nuanced healthcare information compared to traditional machine learning models
in personal sensing, as exemplified by recent studies (Le Glaz et al., 2021).
While these initial studies provide a direction for utilizing LLMs in
addressing general healthcare issues, the application of LLMs to digital
phenotyping for mental health (DPMH) and emotional wellbeing has been scarcely
studied, particularly regarding the integration of smartphone sensing data.

In contrast to broader data-driven tasks, the intricacies of DPMH and
emotional wellbeing involve smaller-scale personal and sensitive data and
subjective measures. The nature of this sensitivity and subjectivity renders
the DPMH tasks more challenging because of contextual dependence and
difficulty in identifying generalized patterns. Additionally, in the nascent
field of DPMH, the exploration of multimodal or multisensor activities in
daily behavioral data collected via smartphones remains limited, despite its
potential to indicate individuals’ psychological fluctuations. Given that LLMs
are well-known for capturing contextual information, they are a natural choice
for analyzing digital traces inferred from smartphones. Compared with
traditional ML models focused on analysing numeric features produced from
individual sensors, LLMs can analyse generated lifecycle descriptions from
data collected across multiple smartphone sensors at a higher level of
abstraction. Then, powered by prior knowledge, LLMs are capable of unveiling
the latent associations between behavioral patterns and individuals’ mental
health or affective status. As such, our research aims to explore whether LLMs
can effectively predict affective states with minimal amounts of data,
utilizing human behaviors and interactions with smartphones as input data.

In this study, we investigate the relationship between behavioral features
collected from smartphone sensors and the affects of university students. We
also demonstrate the capability of zero-shot and few-shot embedding LLMs to
infer affective states based on smartphone-captured human activities. Our
results suggest a discernible connection between smartphone-sensed activities
of university students and their affective states, which LLMs interpret
through their chains of thought. To the best of our knowledge, this is the
first paper to delve into grounding LLMs with mobile sensing features for
affect prediction tasks.

##  2\. Method and Experiments

In this section, we describe the study design (§ 2.1), data collection (§
2.1), data engineering (§ 2.2) and tasks delivered to LLMs (§ 2.3).
Specifically, we detail in §2.3 the prompts for LLMs on multiple emotional
state prediction tasks, including zero-shot prompting, few-shot prompting and
chain-of-thought reasoning.

###  2.1. Participants and Data Collection

Study Design.To explore the capability of LLMs in predicting individuals’
general feelings based on smartphone-collected passive data, we investigated a
subset of data obtained from a digital phenotyping study of Australian
university students conducted in 2023. In this study (D’Alfonso and Zhang,
2024), approximately 150 university students were observed over a full
semester (17 weeks). Data was gathered from multiple smartphone sensors and a
variety of psychometric assessments were conducted. Of the approximately 150
participants in this study, we chose to analyse the data of 10 students for
this investigation. The study utilised the AWARE-Light smartphone sensing app
(van Berkel et al., 2023) to collect passive sensing data. Weekly self-
reported assessments were collected using an emailed link to a Qualtrics
questionnaire. Our study continuously collected sensor data for 17 weeks from
the primary smartphones university students used. Informed by our own previous
work as well as existing literature, for example (Politou et al., 2017), the
following sensors were chosen: battery levels, screen unlocks, location
information, application usage, keyboard usage, and communication traces
(calls and messages). Data collection was in accordance with ethics approval
from the University of Melbourne.

Self-reported Measures. At the end of each week, participants were prompted to
complete the International version of the Positive and Negative Affect
Schedule (I-PANAS-SF) questionnaire (Thompson, 2007). This questionnaire
comprises 10 items rated on a Likert scale ranging from 1 (indicating ”Never”)
to 5 (indicating ”Always”). Five of the items indicate positive affects
(active, determined, attentive, inspired, alert) and the other five indicate
negative affects (upset, hostile, ashamed, nervous, afraid).

###  2.2. Data processing

We first cleaned the raw data from the self-reported questionnaire and
passively collected smartphone data to remove duplicates and corrupted entries
before carrying out data engineering. We utilized the RAPIDS tool (Vega et
al., 2021) to generate and select 77 behavioral features at the daily level of
granularity. These features include metrics such as time spent at home, the
number of received messages, and duration of screen unlocks per day. Missing
passive data was treated as such when constructing prompts, without any
imputation.

Previous studies focusing on machine learning models for predicting
individuals’ mental health scales have typically targeted the total score of a
questionnaire, such as the PHQ-9 for depressions or GAD-7 for anxiety. In
terms of I-PANAS-SF, a total score is not feasible, as the questionnaire is
not designed to provide an overall score. Whilst it is possible to sum up the
five positive items to get a total positive score, and likewise for negative
items, our approach was to construct a prediction for each individual scale
item. By doing so, we acknowledge that each item represents a distinct affect,
making our predictions more comprehensive. For instance, predicting solely the
total score of I-PANAS-SF negative affect provides little insight into the
extent to which a person felt upset or nervous. Therefore, we constructed the
task in a manner where LLMs are prompted to predict scores for each single
item of the I-PANAS-SF separately within one inference.

Given the nature of LLMs as language models, and their known limitations in
handling numerical raw data (Spathis and Kawsar, 2023), we opted to reference
participants’ daily features with concise English descriptions. Each feature
description is presented sequentially, preceded by the corresponding date and
time range in the format YYYY-MM-DD HH:MM:SS to YYYY-MM-DD HH:MM:SS (e.g.,
2023-08-02 00:00:00 to 2023-08-02 23:59:59). We intentionally kept the time
series information for training purposes to understand if the LLM has the
capacity to complete the tasks rather than build a robust and usable
predictive model.

###  2.3. Task description

To investigate whether LLMs can grasp the connection between smartphone-
derived behaviours/contexts and affective states, we conducted zero-shot and
few-shot tasks on I-PANAS-SF. Each week’s description of daily activities
consists of approximately 5,000 tokens, and we restricted our experimentation
to Gemini 1.5 Pro (Reid et al., 2024) due to resource limitations in other
LLMs. The LLM was tasked with providing Likert scores rather than textual
descriptions. To ensure deterministic responses from the LLMs, we adjusted the
temperature to zero. Although Gemini still yields slightly different answers
each time, the predictions remain relatively consistent across low (i.e., 1 or
2) and high (i.e., 4 or 5) scores.

In this exploratory study, we assigned individual tasks to 10 subjects and
randomly divided the 17-week data into a 10-week training dataset and a 7-week
test dataset. We performed a repeated random train-test split on the
10-participant dataset three times. In the few-shot prompting method, we
incrementally added one more example from the training dataset for each shot,
starting with one shot and continuing until we reached ten shots, to test the
individual instances in the test dataset.

Zero-Shot. We conducted zero-shot as the baseline of our study for comparison.
The prompt was constructed following Figure 1:

Below is a description of a university student’s activities over a week,
gathered from their smartphone sensors. Based on the descriptions provided,
select the option that best represents how the student felt for the provided
week’s description below for the following feelings: 1\. Active …Determined,
Attentive, Inspired, Alert, Upset, Hostile, Ashamedm, Nervous…
10\. Afraid For each feeling, choose a Likert score ranging from 1 to 5 that
best represents how the student generally felt during the week where 1
represents Never and 5 represents Always. Description of the student’s
activities for the future week: {feature description} Provide your choices in
the following form with no other reasoning: Active: [predicted number]
…Determined, Attentive, Inspired, Alert, Upset, Hostile, Ashamedm, Nervous…
Afraid: [predicted number]

Figure 1. Prompt for Zero-Shot Tasks

Few-Shots. For each few-shot task, we randomly selected one or more weeks of
data from the training set as labeled data. The prompts for the few-shot tasks
were constructed following Figure 2:

Given a series of descriptions detailing a university student’s weekly
activities collected from their smartphone sensors, along with their
corresponding feelings, your task is to identify patterns between the
student’s activities and feelings. Based on these patterns, make predictions
for the student’s feelings according to their future activities. According to
the following behaviors of the student during a week, how active they felt is
score, how determined they felt is score, how attentive they felt is score,
how inspired they felt is score, how alert they felt is score, how upset they
felt is score, how hostile they felt is score, how ashamed they felt is score,
how nervous they felt is score, how afraid they felt is score: weekly
description * ¡number-of-shot¿ weeks Based on the patterns you learnt from the
data provided, select the option that best represents how the student felt for
the future week’s description below for the following feelings: ¡10 listed
I-PANAS-SF items¿

For each feeling, choose an Likert score ranging from 1 to 5 that best
represents how the student generally felt during the week where 1 represents
Never and 5 represents Always. Description of the student’s activities for the
future week: {feature description} Provide your choices in the following form
with no other reasoning: ¡10 listed I-PANAS-SF items¿: [predict number]

Figure 2. Prompt for Few-Shot Tasks

Chain-of-Thought. We constructed chain-of-thought tasks to better understand
how the LLM produced its predictions and evaluate its logical inference
ability. To do this, we modified the prompt by changing the sentence “Provide
your choices in the following form with no other reasoning” to “Provide your
choices in the following form with reasoning for each item. The reasoning
should be based on the comparison of provided student’s weekly behaviors:
item: [predicted number and reasoning].”

##  3\. Results & Discussion

In this section, we first introduce the metrics used to evaluate the LLM’s
predictions. Next, we discuss the zero-shot and few-shot performance of these
models. Finally, we present a chain-of-thought analysis of the LLM with a
zero-shot case study.

Metrics. To assess the accuracy of our predictions, we employed a macro
calculation approach for both the Mean Absolute Error (MAE) and the relative
error ($\epsilon$). This approach involved two main steps: first calculating
the error metrics for each participant individually, and then averaging these
metrics across all participants.

The overall Mean Absolute Error (MAE) is calculated as follows:

$\text{MAE}_{\text{overall}}=\frac{1}{N}\sum_{i=1}^{N}\left(\frac{1}{n_{i}}\sum_{j=1}^{n_{i}}\left|P_{ij}-T_{ij}\right|\right)$

where $P_{ij}$ represents the predicted value, $T_{ij}$ represents the true
value for the $j$-th observation of the $i$-th participant, $n_{i}$ is the
number of observations for participant $i$, and $N$ is the total number of
participants.

The mean of the true values for each participant is calculated as:

$\bar{T}_{i}=\frac{1}{n_{i}}\sum_{j=1}^{n_{i}}T_{ij}$

Finally, the overall relative error is obtained by averaging the relative
errors of all participants:

$\epsilon_{\text{overall}}=\frac{1}{N}\sum_{i=1}^{N}\left(\frac{\text{MAE}_{i}}{\bar{T}_{i}}\times
100\%\right)$

Zero-shot performance. Table 1 illustrates that zero-shot Mean Absolute Errors
(MAEs) indicate relatively underperformed predictions, averaging 1.65 out of 5
($\epsilon=40.9\%$). The zero-shot MAE values across all ten participants
exhibit minimal variability or dispersion (stdtotal = 0.01, stdpos = 0.06,
$\text{std}_{\text{neg}}$ = 0.02), indicating a consistent predictive ability
of the LLMs across various subjects.

The LLM performs similarly for both positive ($MAE=1.62\%$, $\epsilon=38.1\%$)
and negative ($MAE=1.67\%$, $\epsilon=43.8\%$) affects but demonstrates
varying levels of performance across different items. Specifically, it excels
with the ”Alert” item ($MAE=0.87\%$, $\epsilon=32.9\%$) in zero-shot scenarios
while demonstrating poorest performance with ”Afraid” ($MAE=2.51\%$,
$\epsilon=48.7\%$) and ”Nervous” ($MAE=2.29\%$, $\epsilon=54.3\%$). These
differences may stem from varying levels of prior knowledge within the LLM or
distinct indications of daily behaviors collected via smartphones across
different affective states. In short, our results suggest that while LLMs
possess a relatively limited capability to infer personal affective states in
zero-shot, their performance remains consistent across different subjects
based on observations of daily behavioral features.

Few-shot learners. To explore if LLMs are effective few-shot learners for the
association between human activities and affects, few-shot tasks were designed
to showcase LLMs’ predictive power for affective states. A consistent trend is
revealed that as more shots are provided to the LLM, the MAE decreases for
each I-PANAS-SF item, as shown in Table 1. In particular, the one-shot
approach emerges as the most effective, with performance exhibiting slight
fluctuations thereafter. Specifically, the MAE of nine I-PANAS-SF items
decreases in one-shot learning, with a mitigation of an average 10.0% for
positive affects and 43.5% for negative affects. As more examples are
provided, the rate of performance improvement decreases.

Meanwhile, when averaging the MAE for positive and negative affects
respectively, the LLM performs better for negative affects compared to
positive affects in one-shot learning, while they perform similarly in ten-
shot learning ($MAE_{pos}=0.75$, $MAE_{neg}=0.74$). This suggests that the LLM
attains a similar level of knowledge more rapidly for I-PANAS-SF negative
affects.

When comparing individual items of the I-PANAS-SF, the LLM performs optimally
with ”Hostile” ($MAE_{average}=0.66$) and least effectively with
”Active”($MAE_{average}=1.18$) and ”Nervous” ($MAE_{average}=1.15$) overall.
Interestingly, it demonstrates the most significant improvement on ”Afraid”
with additional examples with a 69.7% decrease in MAE between zero-shot and
ten-shot. Note that although the performance across the ten items varies,
there is a converging trend among the results when providing more examples to
the LLMs, as depicted by the change in standard deviation of MAE from 0.55 in
zero shot to 0.09 in ten shots. This suggests that provided with more
examples, the performance gap across individual I-PANAS-SF items depending
solely on the LLM’s prior knowledge tend to diminish due to the LLM’s enhanced
information about the subject.

The performance improvement is also evident for each participant, with most
items experiencing a decrease in MAE, although certain items prove more
effective for some participants than others. For example, Figure 3 displays
two sets of notably diverse learning curves for the I-PANAS-SF items tested on
data from two participants (a and b for one participant, c and d for the
other).

Furthermore, the learning curves for positive and negative affects exhibit
similar trends for each individual. Figure 4 illustrates the correlational
relationship between positive and negative affects for four participants. The
performance of the LLM in predicting the affects based on smartphone
behavioral features improves at a similar rate for both positive and negative
affects, representing linear relationships. This consistent pattern between
positive and negative affects in the learning trend indicates that each
behavioral description and the corresponding affective states provided to the
LLM offer similar levels of predictive information. We observe that zero-shot
learning reveals larger errors (upper right corner of the graph), whereas as
the number of shots increases, the points tend to shift to areas with smaller
MAEs (lower left corner). This finding demonstrates that LLMs learn and
perform better in prediction tasks when provided with more examples. As a
result, the feasibility of using smartphone-inferred behavioral data to infer
affective states with LLMs suggests a meaningful relationship between the two.

Figure 3. Learning curve for data from two participants

Figure 4. Four participant examples of the linear relationship between MAEs
for predicting positive and negative affects.

Shot | Active | Determined | Attentive | Inspired | Alert | Upset | Hostile | Ashamed | Nervous | Afraid | Mean | Std | Positive affects | Negative affects
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
0 | 1.96 | 1.56 | 1.84 | 1.89 | 0.87 | 1.57 | 0.68 | 1.30 | 2.29 | 2.51 | 1.65 | 0.55 | 1.62 | 1.67
1 | 1.33 | 0.87 | 1.07 | 1.07 | 0.85 | 0.79 | 0.69 | 0.69 | 1.00 | 0.85 | 1.10 | 0.20 | 1.04 | 0.80
2 | 1.13 | 0.84 | 1.00 | 0.97 | 0.84 | 0.82 | 0.65 | 0.68 | 0.94 | 0.78 | 1.73 | 0.30 | 0.96 | 0.77
3 | 1.03 | 0.72 | 0.98 | 0.92 | 0.78 | 0.81 | 0.56 | 0.61 | 0.92 | 0.77 | 1.62 | 0.28 | 0.89 | 0.73
4 | 0.97 | 0.75 | 0.99 | 0.91 | 0.71 | 0.88 | 0.57 | 0.67 | 0.96 | 0.88 | 0.83 | 0.14 | 0.87 | 0.79
5 | 0.98 | 0.78 | 0.89 | 0.90 | 0.71 | 0.89 | 0.60 | 0.70 | 0.92 | 0.86 | 0.82 | 0.11 | 0.85 | 0.79
6 | 0.95 | 0.78 | 0.87 | 0.88 | 0.75 | 0.78 | 0.56 | 0.63 | 0.91 | 0.78 | 0.79 | 0.12 | 0.85 | 0.73
7 | 0.88 | 0.80 | 0.90 | 0.84 | 0.72 | 0.73 | 0.53 | 0.63 | 0.94 | 0.77 | 0.79 | 0.12 | 0.83 | 0.74
8 | 0.93 | 0.78 | 0.86 | 0.84 | 0.83 | 0.77 | 0.54 | 0.64 | 0.90 | 0.80 | 0.79 | 0.11 | 0.85 | 0.73
9 | 0.83 | 0.68 | 0.84 | 0.83 | 0.73 | 0.73 | 0.58 | 0.66 | 0.86 | 0.77 | 0.76 | 0.09 | 0.79 | 0.72
10 | 0.81 | 0.62 | 0.83 | 0.76 | 0.73 | 0.80 | 0.59 | 0.64 | 0.90 | 0.76 | 0.75 | 0.09 | 0.75 | 0.74

Table 1. The averaged Mean Absolute Error across the ten participants for
I-PANAS-SF items. The means and standard deviations are computed across the 10
I-PANAS-SF items. The positive and negative affects are derived as averages of
the MAEs for the corresponding items. The underlined results indicate the
best-performing shot for each item.

Chain-of-thought. By providing the LLM with a general understanding of the
task, the model can offer conceptual insights into the relationship between
individual daily activities and their affective states. Although the ultimate
validity of these insights perhaps remains an open matter, they are
nonetheless interesting points. Specifically, for positive affects, the model
emphasizes high usage episodes and frequent screen unlocks for being active,
increased app usage and long sessions with focused typing for feeling
inspired, long sessions with high typing rates and specific application focus
for feeling determined, and frequent typing events with short average time
between keystrokes for being attentive. For negative affects, it underscores
frequent screen unlocks throughout the day for alertness, an increased number
of missed calls, abrupt changes in typing patterns (frequent backspacing and
deletions), and increased phone calls of very short durations for feeling
upset and hostile. It also highlights decreased overall phone usage and lower
response rates to messages for feeling ashamed and increased numbers of very
short screen unlock episodes and frequent location changes with short
durations for feeling afraid.

The smartphone sensor features serve as potential indicators of individuals’
affects, drawing on the LLM’s prior knowledge for analysis. Leveraging these
indicators, the LLM can provide logical reasoning aligned with the predicted
results. The example in Figure 2 demonstrates the consistency between the
predicted outcome and the logical reasoning, offering concise yet precise
indications of its decisions. Among the provided sensor data (battery usage,
application usage, location, calls, messages, keyboard and screen usage), the
LLM’s explanations emphasized involving typing events, screen unlocks, calls,
and application usage. This sheds light on the perspectives guiding the LLM’s
analysis in predicting individuals’ affects. Future endeavors could explore
predictions based on more detailed or comprehensive information gathered from
these sensors.

During the tasks, the LLM exhibits less confidence that the provided
behavioral features could accurately uncover the negative affects compared to
the positive ones. We observe a pattern of the LLM outputting lower and less
varied scores (e.g., 1 or 2) for negative affects. This suggests that
predicting I-PANAS-SF negative affects based on the provided features poses a
challenge for the LLM.

We observed that for those providing 1 to be the predicted score, the
reasoning was often ”There is no clear evidence to suggest the person was
¡affect¿ during the week”. In this case, rather than assigning a random score
or middle score, the LLM seems to assign a minimum value of 1 as the default
value for affect items, and increases an affect item’s score to 2, 3, 4 or 5
as evidence is found in the behavioral traces. Further confirmation was sought
by introducing an additional option, ”Not able to decide (-1)” but the LLM did
not produce this answer. This suggests that the LLM possesses evidence-based
analytical capabilities, producing reliable predictions in which it has the
highest confidence.

Tables 2 and 3 present the results for the same participant data. As more
shots are provided to the LLM, it reasons with greater evidence by comparing
behavioral traces across different weeks, leading to improved performance.
This demonstrates the LLM’s capability to learn, summarize patterns, and
adjust predictions accordingly. These findings also reveal a potential
connection between smartphone behavioral features and affective states,
emphasizing the importance of studying the concrete indications and
associations between them.

Furthermore, in its output texts, the LLM accentuates the specifics of
activities, such as the content the individual is reading on their phone,
frequent application usage, and a high volume of phone unlocking episodes. It
highlights that the accuracy of predictions may benefit from additional
information sources, including sleep status, activity categories, specific
application usage times and names, and the content of typing and viewing,
rather than just speed. These features can be further explored in future
studies.

Item (True label) |  Predicted Score | Reasoning
---|---|---
Active (5) | 3 |  The person had a moderate number of unlocking episodes (average 8 minutes) and travelled a moderate distance (total travelled 4166 meters on 2023-08-08, 9261 meters on 2023-08-10, 17230 meters on 2023-08-12) during the week.
Determined (4) | 3 |  There is no clear evidence to suggest the person was determined during the week.
Attentive (5) | 2 |  The person did not use any social media applications or dating applications, and there were no emails or messages sent or received during this period. This suggests they were not attentive to these aspects of their phone.
Inspired (4) | 1 |  There is no clear evidence to suggest the person was inspired during the week.
Alert (4) | 3 |  The person had a moderate number of unlocking episodes (average 6 minutes across the week) and travelled a moderate distance (total travelled ranging from 4166 meters on 2023-08-08 to 17230 meters on 2023-08-12) which could indicate some alertness.
Upset (3) | 1 |  There is no clear evidence to suggest the person was upset during the week.
Hostile (2) | 1 |  There is no clear evidence to suggest the person was hostile during the week.
Ashamed (1) | 1 |  There is no clear evidence to suggest the person was ashamed during the week.
Nervous (5) | 2 |  There is no clear evidence to suggest the person was nervous during the week. They did not use social media or dating applications, and there were no emails or messages sent or received.
Afraid (4) | 1 |  There is no clear evidence to suggest the person was afraid during the week.

Table 2. Zero-Shot Chain-of-Thought reasoning for one participant.

Item (True label) |  Predicted Score | Reasoning
---|---|---
Active (5) | 4 |  The student’s average time spent moving between locations is consistently around 10 km/h, and they have a significant amount of total traveled meters (often over 10,000 meters). These indicate a generally active lifestyle. While their future week involves slightly less movement than usual, it is still above average, so I predict a score of 4.
Determined (4) | 4 |  The student frequently uses applications for extended periods. This suggests a consistent engagement with tasks, potentially indicating a sense of determination. The future week involves similar levels of application usage, making a score of 4 appropriate.
Attentive (5) | 4 |  The student’s average time spent at the most visited location is very high, often exceeding 1000 minutes. This could imply a focus on tasks or activities within this location. This pattern persists in the future week. Thus, a score of 4 is likely.
Inspired (4) | 4 |  While there isn’t direct evidence in the provided data, the student’s frequent use of applications, particularly for extended periods, suggests a potential engagement with learning or creative pursuits. The future week maintains this trend, so a score of 4 seems fitting.
Alert (4) | 3 |  The student has a relatively low number of unlock episodes per day and a low average duration of unlock episodes. This could indicate a less frequent need to access the phone, suggesting a possible lower level of alertness. The future week maintains this pattern, so I predict a score of 3.
Upset (3) | 3 |  The student often experiences missed calls, suggesting potential disruptions and stress. In the future week, there is a small increase in missed calls, so I predict a score of 3.
Hostile (2) | 3 |  There is no clear indication of hostile behaviors based on the provided data. The student has consistent low numbers of missed and outgoing calls, suggesting a general calmness. However, the future week sees a small uptick in missed calls, potentially signaling a slight increase in tension, making a score of 3 more likely.
Ashamed (1) | 2 |  No clear indicators of shame can be found in the provided data. The student’s low levels of social media and dating app usage suggest a possible preference for privacy or a lack of social engagement. I predict a score of 2 for the future week.
Nervous (5) | 4 |  The student often shows a high number of typing events per session. This could indicate a possible nervousness or anxiousness, particularly if there are many events where the length of the text changes in fewer characters, suggesting possible backspacing or edits. The future week reflects this pattern, so a score of 4 seems appropriate.
Afraid (4) | 3 |  The provided data lacks clear indicators of fear. The student’s consistent low numbers of missed and outgoing calls suggest a general feeling of safety and security. However, the future week features a slight increase in missed calls, which might indicate a potential for feeling slightly more vulnerable or anxious, leading to a score of 3.

Table 3. Ten-Shot Chain-of-Thought reasoning for one participant

##  4\. Future Work and Conclusions

In this study, we demonstrated the potential of LLMs in harnessing smartphone-
collected behavioral data to predict affective states of university students
by leveraging both zero-shot and few-shot learning approaches. Our results
indicate that the LLM’s ability to perform zero-shot tasks for mental
wellbeing based on weekly summarized behaviors suffers from a relatively high
error rate. However, when provided with labeled information, the LLMs achieve
significantly improved predictions. This positions LLMs as an invaluable
resource for future research and practical applications in wellbeing and other
fields, deepening our understanding of individuals’ mental health, emotions,
and affective states.

For future work, fine-tuning tasks could be conducted to develop a model
driven by daily activities, thereby incorporating more data and possibly
enabling a comparison between individual and general predictive models. Given
the subjective nature of self-reported datasets, models may exhibit biases
stemming from imbalanced class distributions, with some classes being
significantly underrepresented compared to others. Hence, increasing the
dataset size or employing resampling techniques becomes imperative to
construct a more robust model overall. Furthermore, leveraging LLMs to predict
individual items of additional psychometric measures for prediction tasks
would be advisable, facilitating the construction of more nuanced insights
into people’s affective states at a wellbeing level.

##  5\. Appendix

Item | Feature | Sensor
---|---|---
1 | the count of episodes using Email applications | Applications
2 | the count of episodes using all applications
3 | the count of episodes using social media applications
4 | the count of episodes using dating applications
5 | the count of episodes using social applications
6 | the count of episodes using entertainment applications
7 | the count of episodes using Facebook Moments
8 | the count of usage episodes for the application with top usage
9 | the count of episodes using YouTube
10 | the count of episodes using Twitter
11 | the duration of using Email
12 | the duration of using all applications
13 | the duration using social media applications
14 | the duration of using dating applications
15 | the duration of using social applications
16 | the duration of using entertainment applications
17 | the duration of using Facebook Moments
18 | the duration of using the application with top usage
19 | the duration of using YouTube
20 | the duration of using Twitter
21 | the count of discharging episodes | Battery
22 | the count of charging episodes
23 | the number of missed calls | Calls
24 | the number of distinct contacts associated with missed calls
25 | the number of missed calls from the most frequent contact
26 | the number of incoming calls
27 | the number of distinct contacts associated with incoming calls
28 | the mean of incoming call duration
29 | the sum of incoming call duration
30 | the mode of incoming call duration
31 | the count of incoming calls from the most frequent contact
32 | the number of outgoing calls
33 | the number of distinct contacts associated with outgoing calls
34 | the mean of outgoing call duration
35 | the sum of outgoing call duration
36 | the mode of outgoing call duration
37 | the count of outgoing calls from the most frequent contact
38 | the count of typing events | Keyboard
39 |  the count of keyboard typing or swiping event where the length changes exactly one more character
40 |  the count of keyboard typing or swiping event where the length changes less than one fewer character
41 |  the count of keyboard typing or swiping event where the length changes exactly one fewer character
42 | the number of characters in average in a session
43 | the number of typing sessions
44 | the average time between keystrokes

Table 4. Smartphone sensor features embedded in the prompts.

45 | time spent at the second most visited location | Locations
---|---|---
46 | maximum time spent at any location cluster
47 |  the ratio of time spent moving between locations to time spent stationary at a location
48 | total travelled distance
49 | standard deviation of the time spent at location clusters
50 | time spent at the most visited location
51 | average time spent at location clusters
52 | normalized entropy of location visits
53 | variance of speed during movement between locations
54 | time spent at home
55 | time spent at the third most visited location
56 | the number of transitions between distinct locations
57 | minimum time spent at any location cluster
58 | radius of Gyration (RoG) indicating the area covered
59 | average speed during movement between locations
60 | percent of time considered outliers in location data
61 | the variance of locations visited
62 | the logarithm of the variance of locations visited
63 | the number of significant places visited
64 | the entropy of location visits
65 | the number of received messages from the most frequent contact | Messages
66 | the number of received messages
67 | the number of distinct contacts associated with received messages
68 | the number of sent messages to the most frequent contact
69 | the number of sent messages
70 | the number of distinct contacts associated with sent messages
71 | the number of unlock episodes | Screen status
72 | total duration of unlock episodes
73 | the length of longest unlock episode
74 | average time of unlock episodes
75 | the length of shortest unlock episode
76 | standard deviation of unlock episodes
77 | time between the first unlock episode and midnight

Table 5. Smartphone sensor features embedded in the prompts Cont.

## References

  * (1)
  * Beames et al. (2021) Joanne R Beames, Katarina Kikas, and Aliza Werner-Seidler. 2021.  Prevention and early intervention of depression in young people: an integrated narrative review of affective awareness and Ecological Momentary Assessment.  _BMC psychology_ 9, 1 (2021), 113.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system. In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_. 785–794.
  * D’Alfonso and Zhang (2024) S. D’Alfonso and T. Zhang. 2024.  StudentSense.
  * Le Glaz et al. (2021) Aziliz Le Glaz, Yannis Haralambous, Deok-Hee Kim-Dufor, Philippe Lenca, Romain Billot, Taylor C Ryan, Jonathan Marsh, Jordan Devylder, Michel Walter, Sofian Berrouiguet, et al. 2021\.  Machine learning and natural language processing in mental health: systematic review.  _Journal of medical Internet research_ 23, 5 (2021), e15708.
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015.  Deep learning.  _nature_ 521, 7553 (2015), 436–444.
  * Liu et al. (2023) Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer-Levy, Jacob Sunshine, Jiening Zhan, Ming-Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel. 2023.  Large language models are few-shot health learners.  _arXiv preprint arXiv:2305.15525_ (2023).
  * Politou et al. (2017) Eugenia Politou, Efthimios Alepis, and Constantinos Patsakis. 2017.  A survey on mobile affective computing.  _Computer Science Review_ 25 (2017), 79–100.
  * Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024\.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  _arXiv preprint arXiv:2403.05530_ (2024).
  * Spathis and Kawsar (2023) Dimitris Spathis and Fahim Kawsar. 2023.  The first step is the hardest: Pitfalls of representing and tokenizing temporal data for large language models.  _arXiv preprint arXiv:2309.06236_ (2023).
  * Thompson (2007) ER Thompson. 2007.  Development and validation of an internationally reliable short-form of the positive and negative affect schedule (PANAS), 38 (2), 227–242.
  * van Berkel et al. (2023) Niels van Berkel, Simon D’Alfonso, Rio Kurnia Susanto, Denzil Ferreira, and Vassilis Kostakos. 2023.  AWARE-Light: A smartphone tool for experience sampling and digital phenotyping.  _Personal and Ubiquitous Computing_ 27, 2 (2023), 435–445.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Vega et al. (2021) Julio Vega, Meng Li, Kwesi Aguillera, Nikunj Goel, Echhit Joshi, Kirtiraj Khandekar, Krina C Durica, Abhineeth R Kunta, and Carissa A Low. 2021.  Reproducible analysis pipeline for data streams: Open-source software to process data collected with mobile devices.  _Frontiers in Digital Health_ 3 (2021), 769823.
  * Zhai and Lafferty (2001) Chengxiang Zhai and John Lafferty. 2001.  Model-based feedback in the language modeling approach to information retrieval. In _Proceedings of the tenth international conference on Information and knowledge management_. 403–410.