#  SensEmo: Enabling Affective Learning through Real-time Emotion Recognition
with Smartwatches

Kushan Choksi1, Hongkai Chen2, Karan Joshi1, Sukrutha Jade1, Shahriar Nirjon3,
and Shan Lin1  1Department of Electrical and Computer Engineering
Stony Brook University, Stony Brook, NY, USA
Email<EMAIL_ADDRESS>{karan3502<EMAIL_ADDRESS><EMAIL_ADDRESS>2Department of Information Engineering
The Chinese University of Hong Kong, Hong Kong SAR, China
Email<EMAIL_ADDRESS>3Department of Computer Science
University of North Carolina, Chapel Hill, NC, USA
Email<EMAIL_ADDRESS>
###### Abstract

Recent research has demonstrated the capability of physiological signals to
infer both user emotional and attention responses. This presents an
opportunity for leveraging widely available physiological sensors in
smartwatches, to detect real-time emotional cues in users, such as stress and
excitement. In this paper, we introduce SensEmo, a smartwatch-based system
designed for affective learning. SensEmo utilizes multiple physiological
sensor data, including heart rate and galvanic skin response, to recognize a
student’s motivation and concentration levels during class. This recognition
is facilitated by a personalized emotion recognition model that predicts
emotional states based on degrees of valence and arousal. With real-time
emotion and attention feedback from students, we design a Markov decision
process-based algorithm to enhance student learning effectiveness and
experience by offering suggestions to the teacher regarding teaching content
and pacing. We evaluate SensEmo with 22 participants in real-world classroom
environments. Evaluation results show that SensEmo recognizes student emotion
with an average of 88.9% accuracy. More importantly, SensEmo assists students
to achieve better online learning outcomes, e.g., an average of 40.0% higher
grades in quizzes, over the traditional learning without student emotional
feedback.

##  I Introduction

Emotion impacts learning, with curiosity and motivation being crucial for
effective learning and academic success [1]. However, students often struggle
to maintain motivation and concentration during class, especially when the
difficulty level and teaching pace do not match their needs. Experienced
teachers use strategies like interactive discussions to boost engagement, but
there is a lack of effective methods to monitor students’ emotions. This
problem is more significant in online classrooms, where interactions are
limited, making it difficult to get emotional feedback from students. Thus, an
affective learning system that accurately identifies a learner’s emotional
state is needed.

Affective learning involves emotional engagement in acquiring knowledge,
skills, and attitudes, recognizing emotions’ role in cognitive processes,
memory, and decision-making. Research has used methods like eye tracking [2],
facial recognition [3], speech recognition [4], text recognition [5], and
gesture movement [6] to detect emotions. However, these methods are not
suitable for classroom settings. For instance, camera-based methods raise
privacy concerns and struggle to capture all students at once. Additionally,
in online learning, factors like lighting, network conditions, and privacy
further limit these methods.

To address these challenges, we use off-the-shelf smartwatches with
physiological sensing capabilities for emotion recognition. Studies have
showed strong links between physiological signals and emotional states [7]. We
develop a personalized emotion model by analyzing physiological signals from
different users, identifying changes related to motivation and concentration.
Leveraging this model, we develop SensEmo, which helps teachers monitor
students’ emotions and adjust teaching methods. For example, SensEmo suggests
providing concrete examples to clarify key concepts when it detects confusion
among multiple students. Similarly, it advises increasing the pace when most
students appear bored. It is important to note that various factors can
influence a student’s motivation level in real-world scenarios, but this paper
focuses on classroom learning situations.

To assess our emotion recognition solution, we implemented SensEmo on
commercial smartwatches and tested it with 22 volunteers using International
Affective Picture System (IAPS) emotion image dataset [8]. Physiological
signal features collected by smartwatches were mapped to the valence-arousal
space and classified into four states: curious, bored, confused, and
satisfied. Additionally, we conducted simulated and real-world experiments to
evaluate the system’s impact on affective learning. Results indicate that
SensEmo helps maintain students’ emotional states and enhances learning
outcomes.

In summary, our main contributions are the following.

  * •

We developed SensEmo, the first affective learning system that utilizes real-
time emotion sensing and recognition with a smartwatch to provide students’
feedback.

  * •

We proposed a personalized emotion model that classifies physiological signals
into specific levels of motivation and concentration tailored to the learning
environment.

  * •

We introduced an affective learning model using a reinforcement learning-based
controller, which adapts teaching content and pace according to real-time
emotion and individual learning preferences of students. This model is
applicable to both online and classroom learning systems.

  * •

In real learning scenarios with 22 student volunteers, SensEmo achieved 88.9%
accuracy in emotion recognition. Our preliminary experiments in online and
classroom settings demonstrated SensEmo’s promise to boost learning outcomes
and enhance student emotions.

##  II Related Work

Emotion recognition has been explored in many domains. A KNN algorithm
classifies emotions using physiological and subjective components [9].
StressSense uses smartphone-detected voice for stress recognition [10]. Text
analysis studies emotions via semantic labels, attributes [11], and typing
rhythms [12]. Wearable devices detect emotions using respiratory frequency,
heart rate variability, skin conductance, and accelerometer data [13].
However, limited research focuses on emotion recognition in learning
environments. Studies explore wearable biosensors to enhance learning by
monitoring physiological signals [14], assessing cognitive load during
problem-solving [15], and predicting depression in students [16]. However, few
have examined emotion recognition’s role in improving learning performance.

Adaptive systems have been proposed to adjust game difficulty based on users’
emotions to maintain engagement [17], and e-learning systems have used neuro-
fuzzy networks to estimate user behavior and adapt content presentation [18].
However, they are usually limited by real-world implementation and validation.
In contrast, SensEmo utilizes physiological signals from smartwatches to guide
teaching content and pace in real-world settings.

##  III System Design

###  III-A Application Scenarios

SensEmo can be implemented in both in-person and online remote classroom
settings, where students wear a smartwatch, and a mobile app collects real-
time physiological data. This data is then sent to a central server for
emotion recognition, adapting the learning process for students. In online
remote learning, teaching content and pace can be automatically adjusted for
each individual student. In an in-person classroom setting, the instructor
focuses on maintaining positive emotional responses among students. This paper
explores SensEmo’s use in affective learning, with potential applications in
driving monitoring [19], health monitoring [20], and interviewing/consultation
assistance [21].

###  III-B System Overview

Figure 1:  SensEmo uses physiological signals to personalize the emotion model
and adapt the learning content and pace to the student’s emotional state.

SensEmo collects real-time emotional feedback from students’ physiological
signals during engagement with teaching content. This feedback is used to
adapt the course automatically. The components of SensEmo are shown in Figure
1.

Sensor system. SensEmo’s sensor system uses a smartwatch to capture real-time
physiological signals, including heart rate, skin resistance, and skin
temperature.

Physiological data collection. A smartphone application communicates with
smartwatches to retrieve sensor data, which is sent to the cloud for
processing and feature extraction.

Cloud computing. A cloud platform is used due to storage and computational
requirements, facilitating feature mapping, emotion recognition, and
controller computations.

Affective learning controller. The learning controller uses real-time feedback
on emotions and concentration to adjust teaching content and pace. It employs
a Markov Decision Process (MDP) trained via reinforcement learning, creating a
loop that informs the instructor of students’ emotions and preferences. The
design, shown in Figure 2, includes emotion recognition and reinforcement
learning-based control.

Figure 2: Overview of the affective learning feedback controller, including
emotion recognition and a reinforcement learning-based controller.

###  III-C Emotion Recognition

During the students’ engagement in learning, SensEmo uses smartwatches to
capture physiological data. To interpret this data, it is crucial to correlate
physiological signals with emotional states. In this section, we describe our
approach to emotion recognition.

Physiological feature selection. Emotional responses trigger autonomic nervous
system reactions, captured through physiological signals. SensEmo uses the
following features to deduce emotions. (1) Electrodermal activity: Continuous
changes in skin’s electrical characteristics, influenced by moisture. Skin
conductance response, increasing with stress and perspiration, and skin
conductance level can be obtained from electrodermal activity. (2) Blood
volume pulse: Captures changes in blood volume, indicating blood flow. The
inter-beat interval, which represents the time duration between two
heartbeats, is derived from the blood volume pulse signal. This interval is
utilized to determine features such as heart rate and heart rate variability.
(3) Skin temperature: Linked to the autonomic nervous system. Excitement can
raise skin temperature. We measure both skin temperature response and level.

Personalized emotion model calibration. Emotion recognition using
physiological signals varies among individuals [22]. For instance, one
person’s heart rate may rise to 100 bps when excited, another’s to 120 bps. To
address the user-specific differences, we developed a personalized calibration
technique to normalize emotion-reflecting physiological changes. Our local
min-max normalization maps physiological signals to a feature space from -1 to
1, calculated as $f^{\prime}=(f-{f_{min}})/({f_{max}}-{f_{min}})$, where $f$
is the feature, ${f_{max}}$ and ${f_{min}}$ are local maxima and minima. This
normalization has been used to analyze different physiological features
consistently, disregarding individual emotional responses [23].

Valence/arousal scales from physiological features. SensEmo predicts students’
valence and arousal scales from calibrated features. We use the IAPS image
dataset, which elicits emotional responses. Images are categorized as
pleasant, neutral, or unpleasant, and rated on valence (pleasant to
unpleasant) and arousal (calm to excited) scales. These scales are target
values for mapping physiological features into the valence-arousal space,
widely used in psychological research [24]. During our study, users’
physiological signals were collected while viewing these images. This
collected dataset is used to train a model to translate physiological signals
into valence and arousal scales.

Learning emotion recognition from valence-arousal space. The valence-arousal
space categorizes emotions in a two-dimensional space [25]. Existing theories
label emotions within this space [26]. SensEmo maps valence and arousal scales
to learning-related emotions like boredom, confusion, curiosity, and
satisfaction [27]. Figure 3 shows the valence-arousal space highlighting
learning-relevant emotions. This approach, allowing continuous emotion
modeling, has been validated by clinical data [28].

Figure 3: The 2-dimensional valence-arousal space and mapping of emotions of
bored, satisfied, curious, and confused.

###  III-D Reinforcement Learning-based Control

After processing student emotion data, the system identifies the current
emotional state, $S_{t}$, and preferences for content difficulty and pace.
These inputs are fed to a reinforcement learning-based controller with two
main components: a decision-making policy lookup and an MDP controller. The
lookup retrieves information about optimal and sub-optimal decisions made by
the MDP controller in the current and previous steps. Sub-optimal decisions
are fallback options if the optimal policy isn’t feasible. For instance,
lectures may not able to change content or adjust pace as suggested by the MDP
controller. The MDP controller optimizes decision sequences based on the
current emotional state $S_{t}$ and student preferences, aiming to keep the
user in a state of curiosity, characterized by relaxed alertness [29]. The MDP
structure is shown in Figure 4, comprising states (bored, satisfied, curious,
confused) and actions (increase/decrease pace, simplify/no change in content).
Transition probabilities determine the likelihood of moving from one state to
another, influenced by the current state and the student’s preferences.
However, determining the optimal decision that maintains the user in the
desired emotional state of curiosity relies on value iteration of the MDP
model.

Figure 4: The discrete MDP representation of SensEmo.

SensEmo faces uncertainties in both the students’ emotional states and
learning performance, which are influenced by the instructor’s decisions. The
system uses a MDP represented by a 4-tuple: states $S$, actions $A$,
transition probabilities $P$, and rewards $R$. The goal is to find an optimal
policy $\pi^{*}$ that maximizes cumulative rewards over time, guiding
decisions to induce curiosity. The MDP is defined as follows.

  * •

States: $S=\\{s_{0},s_{1},s_{2},s_{3}\\}$. Each represents emotion state
bored, satisfied, curious, and confused, respectively. Additionally,
$s^{\prime}$ denotes the state that follows after $s$.

  * •

Actions: $A=\\{a_{0},a_{1},a_{2},a_{3}\\}$. Each represents action increase
pace, decrease pace, simplify content, and no change in content. Note that $a$
is the action taken during each iteration, as indicated in equations (1) and
(2).

  * •

Transition Probabilities: Determined by a Markov chain asymptotic analysis
[30] with our experimental data.

  * •

Reward: $R_{a}(s|s^{\prime})$. It represents the incentives for state
transition. The initial rewards were defined based on the observed differences
in asymptotic state transition probabilities. These values were obtained
empirically, considering the variations in state transition probabilities.

The optimization problem for the MDP aims at maximizing the accumulated
rewards based on current emotional states of the students, in order to
determine the optimal policy $\pi^{*}$. The optimal policy $\pi^{*}(s)$ is
determined asymptotically using the value iteration optimization function,
which is defined by the following equations.

$\displaystyle\pi(s)$
$\displaystyle:=\operatorname*{arg\,max}_{a}\\{\sum_{s^{\prime}}P(s^{\prime}|s,a)(R(s^{\prime}|s,a)+\gamma
V(s^{\prime}))\\}$ (1) $\displaystyle V_{i+1}(s)$
$\displaystyle:=\max_{a}\\{\sum_{s^{\prime}}P_{a}(s^{\prime}|s)(R_{a}(s,s^{\prime})+\gamma
V_{i}(s^{\prime}))\\}$ (2)

The policy $\pi(s)$ maps states $s$ to actions $a$ and the value function
$V_{i+1}(s)$ at iteration $i+1$ represents the maximum expected return from
state $s$, subject to the optimal action and the initialization of the value
function. Initial values for states are assigned randomly. The discount factor
$\gamma\in[0,1]$ weighs the importance of future rewards, empirically set to
guide actions towards curious or satisfied states. In our system, the discount
factors for $s_{1},\ldots,s_{4}$ are set to $0.1$, $0.45$, $0.35$, and $0.1$,
respectively. Furthermore, the decision-making process takes into account the
student’s preferences, such as desirable learning pace or preference between
illustrations and descriptions. This preference data is collected through a
pre-lecture questionnaire that can be conveniently stored online.

##  IV System Implementation

In this section, we describe the implementation and realization of both the
hardware and algorithms of SensEmo.

###  IV-A Sensor System

Data Acquisition. We implemented SensEmo on Microsoft Band 2 smartwatches,
which can detect electrodermal activity, blood volume pulse, and skin
temperature. Each sensor has a default sampling rate of $1$ Hz. Proper sensor
placement is crucial to avoid motion artifacts that can affect measurement
accuracy. An Android app was installed on users’ smartphones to receive sensor
data from the smartwatches and facilitate data collection.

Feature extraction. SensEmo extracts six features from the sensor data: skin
conductance response (SCR), skin conductance level (SCL), heart rate (HR),
heart rate variability (HRV), skin temperature response (STR), and skin
temperature level (STL). It computes moving averages of HR, STR, and SCL, and
running deviations of HRV, SCR, and STR using a 50-sample moving window. HRV
is extracted from the RR-interval data stream using the Root-mean-square of
successive differences (RMSSD) method.

###  IV-B Emotion Recognition

We describe two major components of emotion recognition: personalized model
calibration and emotion recognition.

Personalized model calibration. First, SensEmo calculates the maximum and
minimum values of the six physiological sensor features (i.e., SCR, SCL, HR,
HRV, STR, and STL) within a 50-sample window for each user. Then, it computes
the normalized features for all users.

Emotion recognition. We use the IAPS image dataset to map normalized features
to valence and arousal scales. During data collection, users’ physiological
signals were recorded while viewing images, with the valence and arousal
scales of those images considered as the ground truth. A Fine Gaussian Support
Vector Machine (SVM) was chosen for predicting valence and arousal scales due
to its memory efficiency. The dataset was split into a 70:15:15 ratio for
training, validation, and testing. Finally, a fuzzy emotion modeling approach
[28] mapped the valence and arousal scales onto the valence-arousal space,
identifying specific emotions with nuanced representation by considering fuzzy
boundaries between emotional states.

###  IV-C MDP Control with Collective Emotion

The input to the MDP is the collective emotion state from individual student
emotions, which is defined as the weighted average of the individual emotions.
It is computed as
$s^{*}\approx(\sum_{i=1}^{N}W_{i}s_{i})/(\sum_{i=1}^{N}n_{i})$, where $n_{i}$
is the number of students in emotional state $s_{i}$, and $N$ is total number
of emotions.

Figure 5: Convergent asymptotic behavior of the MDP.

To ensure continuous operation, we analyzed the MDP structure’s stability,
irreducibility, aperiodicity, ergodicity, and control process consistency,
involving asymptotic behavior and eigenvalue analysis. Figure 5 shows a
convergence sample of the Markov chain’s asymptotic behavior. By analyzing
asymptotics with our data, we determined state transition values and computed
the stationary state distribution from a uniform distribution.

Table I shows an optimal and a sub-optimal policy for SensEmo, helping it
adjust decisions based on value iteration results. Note that Table I
represents one optimal policy, which can vary with data, iterations, reward
schemes, and student preferences. SensEmo stores sub-optimal solutions with
lower rewards than the optimal policy for alternative suggestions. These are
used when optimal suggestions cannot be implemented. For instance, if
confusion is the collective emotion, SensEmo may suggest content
simplification. If infeasible, it proposes sub-optimal adjustments like
reducing the teaching pace. We observe that the sub-optimal policy can change
depending on the prevailing conditions, while the optimal policy remains
stable.

###  IV-D Discussion of System Generalization

In SensEmo, personalized calibration enables our approach to be applied to
diverse groups of individuals with varying baselines, ensuring its
adaptability. In addition, the MDP-based decision-making framework can be
easily extended to accommodate different numbers of states and objectives,
allowing for flexible customization of the state transition probabilities in a
generalized system function. For example, SensEmo can be utilized for
monitoring mental states in medical systems [31]. The methodology employed for
emotion recognition in our approach is also capable of incorporating a broader
range of emotion states within the valence-arousal space. While we utilized a
simple four-quadrant classification of emotional space in our case, the number
of emotions can be expanded, allowing for further classification of emotional
subspaces.

TABLE I: Optimal and sub-optimal policies based on asymptotic value iterations for MDP | Policy Based on Value Iteration
---|---
Current State $s$ | Optimal Policy $\pi(s)$ | Sub-optimal Policy $\pi^{*}(s)$
bored | Enriching content | Simplifying content
satisfied | Making no change | Decreasing pace
confused | Simplifying content | Decreasing pace
curious | Decreasing pace | Enriching content

##  V System Evaluation

In this section, we evaluate the performance of SensEmo in emotion recognition
and its impact on improving learning outcomes. To conduct the evaluation, we
recruited 22 graduate students between the ages of 22 and 27. Additionally,
all experiments described in this paper received approval/waiver from the IRB.

###  V-A Emotion Recognition Evaluation

Experimental Setup. To collect data for different emotion states, it is
crucial to use stimuli that are consistent and reliable throughout the
experiment. In order to establish a neutral emotional state for volunteers, we
utilizes the IAPS image dataset. We present the user with a series of five
images that have medium valence and low arousal, lasting for 15 minutes.
Following this, the desired images are displayed for data collection purposes.
Between subsequent data collection intervals, we again show the user images
with medium valence and low arousal. It is important to note that images with
low valence and high arousal, which may contain negative content such as
mutilation, can have a long-lasting emotional impact and consequently
influence subsequent data collection. For this reason, data collection
involving such images is performed towards the end of the experiments.

Figure 6: Emotion recognition confusion matrices of SensEmo with (left) and
without (right) personalized calibration.

System Accuracy. To create the dataset, we carefully select IAPS images that
span a wide range of valence and arousal levels, encompassing both very low
and very high values. From the dataset, we gather data for a total of 200
images. Among these, 100 images are shared among the volunteers, while the
remaining 100 images are distinct, covering various ranges of valence and
arousal. This compilation forms our dataset, which includes recorded
physiological sensor data along with rated arousal and valence scales.
Following personalized calibration, we train a Fine Gaussian SVM with
supervised learning. To evaluate the classification accuracy, we conduct a
10-fold cross-validation. The resulting confusion matrix is shown in in Figure
6 (left). The experimental findings indicate that the Fine Gaussian SVM
achieves an overall accuracy of 88.9%. In comparison, the k-nearest neighbors
algorithm (KNN) yields an accuracy of 75.2%.

System Reliability. In this evaluation, we examine the reliability of SensEmo
when the users are presented with instantaneous stimuli, i.e., video and music
[32]. Specifically, we gather physiological data while users are exposed to
video stimuli in the form of music videos that incorporate elements of strong
emotional stimuli. It is important to note that these videos are compilations
of multiple graphics sourced from the IAPS dataset, which elicit various
emotions at different times. The same set of videos is utilized for all users
in order to evaluate if similar patterns of physiological signals emerge
across different individuals in response to the same stimuli. Our findings
reveal that the response patterns of GSR and HRV exhibit similarities among
different users when experiencing the same emotion. These results demonstrate
that not only can we rely on patterns of physiological signals to infer
emotions, but SensEmo also reliably captures and records such data.
Furthermore, through personalized calibration, we mitigate differences in the
physiological signal responses of different users and enhance the accuracy of
emotion recognition.

Ablation study on personalized calibration. Figures 6 shows the confusion
matrices obtained with and without personalized calibration, respectively.
Across all emotions, there is a significant and noticeable increase in the
true positive rate. Additionally, the overall recognition accuracy improves
from 27.4% to 88.9% with the implementation of personalized calibration.
Throughout our experiments, we observed instances of incorrect and unstable
classification of user emotional states when personalized calibration was not
applied, even when the user’s emotions remained consistent. These findings
suggest that personalized calibration improves robustness of SensEmo. It is
worth noting that similar outcomes were observed for all the selected
features. These results underscore the importance of personalized calibration
in standardizing feature extraction and enhancing emotional recognition.

Figure 7: Comparison of various aspects in online and in-person classroom
settings. (Left) Students using SensEmo exhibit a longer duration of curiosity
emotion in online remote learning setting. (Middle) Emotional responses from
all users in two sessions for the classroom evaluation. (Right) Quiz scores of
two sessions in in-person classroom setting.

###  V-B Reinforcement Learning-based Controller Evaluation

The role of the reinforcement learning-based controller is to adjust the
system according to the students’ current emotional state. It is essential for
the controller to respond promptly and efficiently, ensuring it can adapt to
the right moment. Consequently, the response time of the feedback control
system, and consequently, the entire system, should be minimized. In addition
to response time, another crucial factor is the acceptance of the feedback
system by the students. It is important that the students perceive the
feedback system positively and find it valuable in their learning experience.

Response Time. We evaluate the response time for the adaptation decision
inference using the emotion recognition results. In this analysis, we also
take into consideration the time required by the emotion recognition model. By
examining the response time, we gain insights into how long it takes for the
system to provide a useful suggestion or adaptation once the user’s data
becomes available. To measure the response time, we utilize software timers.
These experiments involve using real lecture slides obtained from the MIT
OpenCourseWare [33], where no adaptation is provided to users but only
recorded for the purpose of analysis. After analyzing the results, we find
that the average response time of the feedback system is approximately 3.1
seconds, which is significantly shorter than the duration of any individual
lecture slide.

System Acceptance. The purpose of the feedback is to enhance the student’s
learning experience with minimal effort on their part. To evaluate the
effectiveness of the feedback system, we introduce another parameter: the
number of manual interventions made by the student to adjust the pace or
content. This parameter serves as a measure of feedback acceptance, where a
lower number of interventions indicates better performance of the feedback
system. We track these interventions using a software counter. Our experiments
show that, on average, students only make approximately 2.2 interventions per
minute. This suggests that the feedback system effectively assists students in
maintaining a suitable teaching pace and content, minimizing the need for
frequent manual interventions.

###  V-C Resource Utilization Study

We evaluate the energy consumption and memory usage of SensEmo. In our
experiments, the estimated battery life for a Microsoft Band 2 with the
SensEmo app connected is 5 hours, which exceeds the duration of a typical
class session. Regarding memory usage, the Android application requires only
3.28 MB of internal memory space. Furthermore, the average runtime memory
space utilized by the application is a mere 215 KB. In addition, Table II
shows a detailed breakdown of power usage while the app runs on Android
devices.

TABLE II: Power usage breakdown of SensEmo application running on Android devices State | Screen | Bluetooth | Band | Power
---|---|---|---|---
Standby | Off | Off | Disconnected |  $\leq$ 22 mW
App | Off | On | Disconnected | 273 mW
App | Off | On | Connected | 340 mW

###  V-D Real-world Learning Scenarios

In this section, we conducted real-world experiments to evaluate SensEmo ’s
impact on learning outcomes in both online remote learning and in-person
classroom settings. For the online remote learning, 22 graduate students were
randomly divided into two groups of 11. One group used SensEmo while the other
did not. All students wore identical smartwatches and viewed lecture slides
from four different lectures (30-50 minutes each) on topics ranging from
biology to geography, without any prior knowledge. Their physiological data
was recorded, and comprehension was tested with a quiz of 10 concept questions
at the end. In the in-person classroom setting, 10 graduate students
participated in two sessions, wearing identical smartwatches. In the first
session, the instructor did not receive real-time feedback. In the second
session, the same instructor received real-time feedback from SensEmo and
adjusted the lecture content and pace based on students’ emotions. Both
sessions concluded with a quiz of 5 conceptual questions. Students also
completed a survey about their emotional states during the sessions.

Learning outcome. For the online setting, students using SensEmo scored 40.0%
higher on the quiz compared to those who did not. Emotion recognition analysis
showed longer durations of curiosity among SensEmo users, suggesting
significant potential learning impact, shown in Figure 7 (left). This suggests
that even a modest increase in the time spent experiencing the desired emotion
can significantly influence learning outcomes. Also, SensEmo’s adaptability in
teaching pace and content further contributed to improved learning
performance. In the classroom setting, students exhibited more curiosity and
less boredom in the session with real-time feedback, shown in Figure 7
(middle). Although confusion increased, it might indicate active learning.
Quiz scores, shown in Figure 7 (right), were higher in the second session,
hinting at SensEmo’s positive impact despite non-randomization and material
control issues. Students also reported that wearing the smartwatch was neither
distracting nor inconvenient.

##  VI Conclusion

In this paper, we present SensEmo, the first affective learning system that
uses real-time data from physiological sensors available in commercial
smartwatches. It achieves an average of 88.9% accuracy in inferring users’
emotions based on valence and arousal scales, within learning environments. By
utilizing sensed emotional states, SensEmo enables the adaptation of teaching
materials and pace. The effectiveness of this adaptability has been evaluated
in both online remote learning for individual users and in-person classroom
settings with multiple users. Our findings suggest that integrating wearable
sensing into affective learning systems has the potential to enhance learning
outcomes when compared to traditional approaches.

## Acknowledgements

We thank the anonymous reviewers for their valuable feedback and suggestions
for improving the quality of this paper. Research supported in part by NSF
CNS-1952096, CNS-1553273 (CAREER), Hong Kong RGC GRF 14201924, and CUHK Direct
Grant for Research 4055216.

## References

  * [1] C.-H. Su and C.-H. Cheng, “A mobile gamification learning system for improving the learning motivation and achievements,” _Journal of Computer Assisted Learning_ , 2015.
  * [2] V. M. G. Barrios, C. Gütl, A. M. Preis, K. Andrews, M. Pivec, F. Mödritscher, and C. Trummer, “AdELE: A framework for adaptive e-learning through eye tracking,” _Proceedings of IKNOW_ , 2004.
  * [3] A. Dhall, A. Asthana, R. Goecke, and T. Gedeon, “Emotion recognition using PHOG and LPQ features,” in _2011 IEEE International Conference on Automatic Face & Gesture Recognition_. IEEE, 2011, pp. 878–883.
  * [4] A. S. Utane and S. Nalbalwar, “Emotion recognition through speech,” _International Journal of Applied Information Systems (IJAIS)_ , 2013.
  * [5] K. Choksi, S. Nagaraj, R. Thielke, and S. Lin, “mDB: Monitoring dysfunctional behaviors for patients with bipolar disorder,” in _42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_. IEEE, 2020.
  * [6] S. Singh, V. Sharma, K. Jain, and R. Bhall, “EDBL-algorithm for detection and analysis of emotion using body language,” in _1st International Conference on Next Generation Computing Technologies (NGCT)_. IEEE, 2015, pp. 820–823.
  * [7] R. W. Levenson, “The autonomic nervous system and emotion,” _Emotion Review_.
  * [8] P. J. Lang, “International affective picture system (IAPS): Technical manual and affective ratings,” _The Center for Research in Psychophysiology, University of Florida_ , 1995.
  * [9] F. Nasoz, K. Alvarez, C. L. Lisetti, and N. Finkelstein, “Emotion recognition from physiological signals using wireless sensors for presence technologies,” _Cognition, Technology & Work_, 2004.
  * [10] H. Lu, D. Frauendorfer, M. Rabbi, M. S. Mast, G. T. Chittaranjan, A. T. Campbell, D. Gatica-Perez, and T. Choudhury, “StressSense: Detecting stress in unconstrained acoustic environments using smartphones,” in _Proceedings of the 2012 ACM Conference on Ubiquitous Computing_. ACM, 2012.
  * [11] C.-H. Wu, Z.-J. Chuang, and Y.-C. Lin, “Emotion recognition from text using semantic labels and separable mixture models,” _ACM transactions on Asian language information processing (TALIP)_ , 2006.
  * [12] C. Epp, M. Lippold, and R. L. Mandryk, “Identifying emotional states using keystroke dynamics,” in _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. ACM, 2011.
  * [13] A. Hernando, J. Lazaro, E. Gil, A. Arza, J. M. Garzón, R. Lopez-Anton, C. de la Camara, P. Laguna, J. Aguiló, and R. Bailón, “Inclusion of respiratory frequency information in heart rate variability analysis for stress assessment,” _IEEE journal of biomedical and health informatics_ , vol. 20, no. 4, pp. 1016–1025, 2016.
  * [14] M. A. Hernández-Mustieles, Y. E. Lima-Carmona, M. A. Pacheco-Ramírez, A. A. Mendoza-Armenta, J. E. Romero-Gómez, C. F. Cruz-Gómez, D. C. Rodríguez-Alvarado, A. Arceo, J. G. Cruz-Garza, M. A. Ramírez-Moreno _et al._ , “Wearable biosensor technology in education: A systematic review,” _Sensors_ , vol. 24, no. 8, p. 2437, 2024.
  * [15] W. L. Romine, N. L. Schroeder, J. Graft, F. Yang, R. Sadeghi, M. Zabihimayvan, D. Kadariya, and T. Banerjee, “Using machine learning to train a wearable device for measuring students’ cognitive load during problem-solving activities based on electrodermal activity, body temperature, and heart rate: Development of a cognitive load tracker for both personal and classroom use,” _Sensors_ , vol. 20, no. 17, p. 4833, 2020.
  * [16] R. Wang, W. Wang, A. DaSilva, J. F. Huckins, W. M. Kelley, T. F. Heatherton, and A. T. Campbell, “Tracking depression dynamics in college students using mobile phone and wearable sensing,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 2, no. 1, pp. 1–26, 2018.
  * [17] G. Chanel, C. Rebetez, M. Bétrancourt, and T. Pun, “Emotion assessment from physiological signals for adaptation of game difficulty,” _IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans_ , 2011.
  * [18] S. Asteriadis, P. Tzouveli, K. Karpouzis, and S. Kollias, “Estimation of behavioral user state based on eye gaze and head pose—application in an e-learning environment,” _Multimedia Tools and Applications_ , 2009.
  * [19] H. Huang, H. Chen, and S. Lin, “Magtrack: Enabling safe driving monitoring with wearable magnetics,” in _Proceedings of the 17th annual international conference on mobile systems, applications, and services_ , 2019, pp. 326–339.
  * [20] Y. Li, D. S. F. Yu, S. Chen, G. Xing, and H. Chen, “EmoMarker: A privacy-preserving, multi-modal sensing system for dyadic digital biomarkers of expressed emotions for patients with dementia,” in _Proceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services_ , 2024, pp. 614–615.
  * [21] B. Yang, S. Jiang, L. Xu, K. Liu, H. Li, G. Xing, H. Chen, X. Jiang, and Z. Yan, “DrHouse: An LLM-empowered diagnostic reasoning system through harnessing outcomes from sensor data and expert knowledge,” _arXiv preprint arXiv:2405.12541_ , 2024.
  * [22] J. M. Gottman, N. S. Jacobson, R. H. Rushe, and J. W. Shortt, “The relationship between heart rate reactivity, emotionally aggressive behavior, and general violence in batterers.” _Journal of family psychology_ , 1995.
  * [23] H. Henderi, T. Wahyuningsih, and E. Rahwanto, “Comparison of Min-Max normalization and Z-score normalization in the K-nearest neighbor (kNN) algorithm to test the accuracy of types of breast cancer,” _International Journal of Informatics and Information Systems_ , vol. 4, no. 1, pp. 13–20, 2021.
  * [24] K.-H. Choi, J. Kim, O. S. Kwon, M. J. Kim, Y. H. Ryu, and J.-E. Park, “Is heart rate variability (HRV) an adequate tool for evaluating human emotions?–a focus on the use of the International Affective Picture System (IAPS),” _Psychiatry research_ , 2017.
  * [25] P. J. Lang, “The emotion probe: studies of motivation and attention.” _American psychologist_ , vol. 50, no. 5, p. 372, 1995.
  * [26] J. Posner, J. A. Russell, and B. S. Peterson, “The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology,” _Development and psychopathology_ , 2005.
  * [27] B. Kort, R. Reilly, and R. W. Picard, “An affective model of interplay between emotions and learning: Reengineering educational pedagogy-building a learning companion.” in _icalt_ , 2001.
  * [28] R. L. Mandryk and M. S. Atkins, “A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies,” _International journal of human-computer studies_ , 2007.
  * [29] R. N. Caine, G. Caine, C. McClintic, and K. J. Klimek, _12 brain/mind learning principles in action: Teach for the development of higher-order thinking and executive function_. Corwin Press, 2015.
  * [30] E. Seneta, _Non-negative matrices and Markov chains_. Springer Science & Business Media, 2006.
  * [31] Z. Shao, R. Chandramouli, K. Subbalakshmi, and C. T. Boyadjiev, “An analytical system for user emotion extraction, mental state modeling, and rating,” _Expert Systems with Applications_ , 2019.
  * [32] F. Silveira, B. Eriksson, A. Sheth, and A. Sheppard, “Predicting audience responses to movie content from electro-dermal activity signals,” in _Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing_. ACM, 2013.
  * [33] “MIT OpenCourseWare,” https://ocw.mit.edu/.


$\tilde{\phi}_{0}(\eta):=\phi_{0}(2^{N_{0}}\varepsilon_{2}\eta).$ By
definition of para-differential operators (A.13) and (A.29):

$\mathrm{pdo}(a)-{\rm
op}(a)=\mathrm{pdo}((1-\tilde{\phi}_{0}(D_{x}))(a_{2}+a_{R}+a_{LF})).$

By Proposition 20 of [8], for all $s^{\prime}\leq s:$

(A.30)
$\|\mathrm{pdo}((1-\tilde{\phi}_{0}(D_{x}))a_{2})v\|_{H^{s^{\prime}}}\lesssim\sup_{\begin{smallmatrix}|\beta|\leq
2[d/2]+2\\\
\xi\in\mathbb{R}^{d}\end{smallmatrix}}\langle\xi\rangle^{|\beta|-m}\|\partial_{\xi}^{\beta}((1-\tilde{\phi}_{0}(D_{x}))a)\|_{H^{s^{\prime}}}\|v\|_{H^{m+d/2}},$

and also, for $d/2+m^{\prime}\leq s:$

(A.31)
$\|\mathrm{pdo}((1-\tilde{\phi}_{0}(D_{x}))a_{2})v\|_{L^{2}}\lesssim\sup_{\begin{smallmatrix}|\beta|\leq
2[d/2]+2\\\
\xi\in\mathbb{R}^{d}\end{smallmatrix}}\langle\xi\rangle^{|\beta|-m}\|\partial_{\xi}^{\beta}((1-\tilde{\phi}_{0}(D_{x}))a)\|_{H^{d/2+m^{\prime}}}\|v\|_{H^{m-m^{\prime}}}.$

By Proposition 23 of [8], the $a_{R}$ term in (A.29) satisfies the same
bounds, (A.30) and (A.31).

The low-frequency term $a_{LF},$ or $(1-\tilde{\phi}_{0}(D_{x}))a_{LF}$ in
decomposition (A.29), is a finite sum of terms of the form
$\mathrm{pdo}((1-\tilde{\phi}_{0}(D_{x}))\phi_{0}(2^{-q}D_{x})a(\cdot,\xi)\phi_{q}(\xi)).$
Given $v\in L^{2},$ the map
$\mathrm{pdo}((1-\tilde{\phi}_{0}(D_{x}))\phi_{0}(2^{-q}D_{x})a(\cdot,\xi)\phi_{q}(\xi))v$
has compact support in Fourier. In particular, its $H^{s^{\prime}}$ norm, for
$0\leq s^{\prime}\leq s,$ is controlled by its $L^{2}$ norm. We may use the
classical bound (A.17):

$\displaystyle\|\mathrm{pdo}((1-\tilde{\phi}_{0}(D_{x}))$
$\displaystyle\phi_{0}(2^{-q}D_{x})a(\cdot,\xi)\phi_{q}(\xi))v\|_{L^{2}}$
$\displaystyle\lesssim\|(1-\tilde{\phi}_{0}(D_{x}))\phi_{0}(2^{-q}D_{x})a(\cdot,\xi)\|_{m,1+[d/2],1+[d/2]}\|\phi_{q}(D_{x})v\|_{H^{m}}.$

For fixed $q$ and any $m,$ we have
$\|\phi_{q}(D_{x})v\|_{H^{m}}\lesssim\|v\|_{L^{2}},$ for all $v\in L^{2}.$
Thus

(A.32)
$\|\mathrm{pdo}((1-\tilde{\phi}_{0}(D_{x}))a_{LF})v\|_{H^{s^{\prime}}}\lesssim\|(1-\tilde{\phi}_{0}(D_{x}))\phi_{0}(2^{-N_{0}}D_{x})a(\cdot,\xi)\|_{m,1+[d/2],1+[d/2]}\|v\|_{L^{2}}.$

With the decomposition (A.29), estimates (A.30) and (A.32) bound the action of
the difference $\mathrm{pdo}(a)-{\rm op}(a)$ on $H^{m+d/2},$ and estimates
(A.31) and (A.32) bound the action of $\mathrm{pdo}(a)-{\rm op}(a)$ on
$H^{m-m^{\prime}},$ with $m^{\prime}\leq s-d/2,$ where $s$ measures the
spatial regularity of $a.$ We note that these estimates do not depend on the
form of $a$ assumed in (A.4), and are valid for any symbol $a$ of order $m$
with an adequate degree of spatial regularity.

For $a$ in the form (A.4), we now put in the semiclassical quantization, and
consider

(A.33) $\|(\mathrm{pdo}_{j}(a)-{\rm
op}_{j}(a))v\|_{L^{2}}=\|H_{j}^{-1}\big{(}\mathrm{pdo}(h_{j}a)-{\rm
op}(h_{j}a))H_{j}v\|_{L^{2}},$

for $v\in H^{m+d/2}(\mathbb{R}^{d})$ or $v\in H^{m}(\mathbb{R}^{d}).$ In view
of bounds (A.30) and (A.31), and the expression of $a$ in terms of $\sigma$ in
(A.4), we now want a control of

$\sup_{\begin{smallmatrix}|\beta|\leq 2[d/2]+2\\\
\xi\in\mathbb{R}^{d}\end{smallmatrix}}\langle\xi\rangle^{|\beta|-m}\|((1-\tilde{\phi}_{0}(D_{x}))\partial_{\xi}^{\beta}h_{j}\sigma(u,\xi))\|_{H^{s^{\prime}}}.$

We have, by definition of $h_{j}$ (3.1):

$\displaystyle\|((1-\tilde{\phi}_{0}(D_{x}))\partial_{\xi}^{\beta}h_{j}\sigma(u,\xi))\|^{2}_{H^{s^{\prime}}}\leq\int_{|\eta|\geq
c}2^{-2jd}\langle\eta\rangle^{2s^{\prime}}|\partial_{\xi}^{\beta}{\mathcal{F}}\big{(}\sigma(u(\cdot),\xi)\big{)}(2^{j}\eta)|^{2}\,d\eta.$

where $c=\varepsilon_{1}/(2^{N_{0}}\varepsilon_{2}).$ The above upper bound is
equal to

$2^{-jd}\int_{|\eta|\geq
2^{j}c}\langle\eta\rangle^{2s^{\prime}}|\partial_{\xi}^{\beta}{\mathcal{F}}\big{(}\sigma(u(\cdot),\xi)\big{)}(\eta)|^{2}\,d\eta,$

which we can bound by

$2^{2js^{\prime}-jd}\int_{|\eta|\geq
2^{j}c}|\eta|^{2s}(2^{j}c)^{-2s}|\partial_{\xi}^{\beta}{\mathcal{F}}\big{(}\sigma(u(\cdot),\xi)\big{)}(\eta)|^{2}\,d\eta\lesssim
2^{-j(2s-2s^{\prime}-d)}\|\partial_{\xi}^{\beta}\sigma(u,\xi)\|_{H^{s}}.$

We now use Lemma A.2 (applied to $\partial_{\xi}^{\beta}\sigma,$ which
satisfies assumption (A.3) with $m$ replaced by $m-|\beta|$), and find that
the contribution of $(h_{j}a)_{2}$ and $(h_{j}a)_{R}$ in the difference (A.33)
is controlled by

$2^{-j(s-d/2)}C(\|u\|_{L^{\infty}})\|u\|_{H^{s}}\|H_{j}v\|_{H^{m+d/2}},\quad\mbox{if
we use \eqref{a2},}$

and by

$2^{-j(s-d)}C(\|u\|_{L^{\infty}})\|u\|_{H^{s}}\|H_{j}v\|_{H^{m}},\quad\mbox{if
we use \eqref{a2bis}.}$

Finally, we deduce from (A.32) that the low-frequency term $(h_{j}a)_{LF}$
contributes

$\langle\xi\rangle^{m-|\beta|}\|\partial_{\xi}^{\beta}\partial_{x}^{\alpha}(1-\tilde{\phi}_{0}(D_{x}))\phi_{0}(2^{-N}D)a\|_{L^{\infty}}\lesssim\langle\xi\rangle^{m-|\beta|}\big{\|}{\mathcal{F}}\Big{(}\partial_{x}^{\alpha}(1-\tilde{\phi}_{0}(D_{x}))\phi_{0}(2^{-N_{0}}D)\partial_{\xi}^{\beta}h_{j}a\Big{)}\big{\|}_{L^{1}}.$

Since
$(1-\tilde{\phi}_{0}(D_{x}))\phi_{0}(2^{-N_{0}}D)\partial_{\xi}^{\beta}h_{j}a$
is compactly supported in Fourier (with respect to $x$), the
$\partial_{x}^{\alpha}$ contributes only a constant, and we focus on

$\displaystyle\big{\|}{\mathcal{F}}\Big{(}(1-\tilde{\phi}_{0}(D_{x}))\phi_{0}(2^{-N_{0}}D)\partial_{\xi}^{\beta}h_{j}a\Big{)}\big{\|}_{L^{1}}$
$\displaystyle\leq\int_{c_{1}\leq|\eta|\leq
c_{2}}|{\mathcal{F}}(\partial_{\xi}^{\beta}h_{j}a)(\eta)|\,d\eta$
$\displaystyle=\int_{2^{j}c_{1}\leq|\eta|\leq
2^{j}c_{2}}|{\mathcal{F}}(\partial_{\xi}^{\beta}a)(\eta)|\,d\eta,$

for some $0<c_{1}<c_{2}.$ In the integrand we can write $1\lesssim
2^{-js}|\eta|^{s},$ and then by Hölder’s inequality we bound the above by a
constant times

$2^{jd/2}2^{-js}\|\partial_{\xi}^{\beta}a\|_{H^{s}},$

and we may now apply Lemma A.2 to $\partial_{\xi}^{\beta}a.$ This completes
the proof of Proposition A.3.

###  A.5 Operator composition

Given $m_{1},m_{2}\in\mathbb{R},$ given $r>0$, given $a_{1}\in
C^{r}S^{m_{1}},$ $a_{2}\in C^{r}S^{m_{2}}$ if $r\in\mathbb{N},$ or given
$a_{1}\in C^{[r],r-[r]}S^{m_{1}},$ $a_{2}\in C^{[r],r-[r]}S^{m_{2}}$ if
$r\notin\mathbb{N},$ we have

(A.34) ${\rm op}_{j}(a_{1}){\rm op}_{j}(a_{2})=\sum_{0\leq
k<r}2^{-jk}\frac{(-i)^{|\alpha|}}{\alpha!}{\rm
op}_{j}\left(\partial_{\xi}^{\alpha}a_{1}\partial_{x}^{\alpha}a_{2}\right)+2^{-jr}R_{r}(a_{1},a_{2}),$

with a remainder $R_{r}(a_{1},a_{2})$ which maps $H^{m_{1}+m_{2}-r}$ to
$L^{2},$ with norm

(A.35)
$\big{\|}R_{r}(a_{1},a_{2})v\big{\|}_{L^{2}}\lesssim\Big{(}\|a_{1}\|_{m_{1},0,m(d,r)}\|a_{2}\|_{m_{2},r,d}+\|a_{1}\|_{m_{1},r,m(d,r)}\|a_{2}\|_{m_{2},0,d}\Big{)}\|v\|_{j,m_{1}+m_{2}-r},$

for some $m(d,r)$ depending on $d$ and $r.$ For a proof of (A.34)-(A.35), see
for instance Theorem 6.1.4 of [4].

The above fractional-order composition result has a simple proof in the
particular case of a Fourier multiplier and a Sobolev map:

######  Lemma A.5.

Given $s>1+d/2,$ given a Fourier multiplier $\chi(D_{x})$ with
$|\cdot|^{s-1-d/2}{\mathcal{F}}^{-1}\chi\in L^{1}(\mathbb{R}^{d}),$ given
$f\in H^{s-1}(\mathbb{R}^{d}),$ we have

$\displaystyle\Big{\|}[\chi(D_{x}),f]v-\sum_{1\leq\alpha\leq[s-1-d/2]}\frac{(-i)^{|\alpha|}}{|\alpha|!}$
$\displaystyle(\partial_{x}^{\alpha}f)(x)(\partial_{\xi}^{\alpha}\chi)(D_{x})v\Big{\|}_{L^{2}}$
$\displaystyle\lesssim\|f\|_{H^{s-1}}\big{\|}|\cdot|^{s-1-d/2}{\mathcal{F}}^{-1}\chi\big{\|}_{L^{1}}\|v\|_{L^{2}},$

In particular, under the above assumptions, we have

$\displaystyle\Big{\|}[\chi(2^{-j}D_{x},f]v$
$\displaystyle-\sum_{1\leq\alpha\leq[\theta]-1}2^{-j|\alpha|}\frac{(-i)^{|\alpha|}}{|\alpha|!}(\partial_{x}^{\alpha}f)(x)(\partial_{\xi}^{\alpha}\chi)(2^{-j}D_{x})v\Big{\|}_{L^{2}}$
$\displaystyle\leq
2^{-j\theta}\sup_{|\alpha|=[\theta]}\|\partial_{x}^{\alpha}f\|_{0,\theta-[\theta]}\big{\|}|\cdot|^{\theta}{\mathcal{F}}^{-1}\chi\big{\|}_{L^{1}}\|v\|_{L^{2}}.$

The Hölder norm in the upper bound is finite by the Sobolev embedding (A.38).

In the statement of Lemma A.5, we used the convention that the sum over an
empty set is equal to zero. This simple composition result is used in the
proof of Lemma 3.9.

###### Proof.

For any $v\in L^{2}:$

$[\chi(D_{x}),f]v=\int_{\mathbb{R}^{3}}\big{(}{\mathcal{F}}^{-1}\chi\big{)}(x-x^{\prime})(f(x^{\prime})-f(x))v(x^{\prime})\
dx^{\prime},$

and with a Taylor expansion of $f\in H^{s-1},$ with $s-1>d/2:$

(A.36)
$[\chi(D_{x}),f]=\sum_{1\leq\alpha\leq[s-1-d/2]-1}\frac{(-i)^{|\alpha|}}{|\alpha|!}(\partial_{x}^{\alpha}f)(x)(\partial_{\xi}^{\alpha}\chi)(D_{x})+R_{[s-1-d/2]}(\chi,f).$

The remainder is explicitly

(A.37) $\displaystyle R_{[\theta]}(\chi,f)v=\int_{\mathbb{R}^{3}}$
$\displaystyle\int_{0}^{1}\frac{(-i(1-t))^{[\theta]-1}}{([\theta]-1)!}$
$\displaystyle\quad\cdot\big{(}{\mathcal{F}}^{-1}\chi\big{)}(x-x^{\prime})\sum_{|\alpha|=[\theta]}\partial_{x}^{\alpha}f(x+t(x^{\prime}-x))\cdot(x^{\prime}-x)^{(\alpha)}v(x^{\prime})\,dx^{\prime}\,dt^{\prime}.$

In (A.37), given $\alpha=(\alpha_{1},\dots,\alpha_{d})\in\mathbb{N}^{d},$
given $y=(y_{1},\dots,y_{d})\in\mathbb{R}^{d},$ we used notation

$\partial_{x}^{\alpha}f(x+ty)\cdot y^{(\alpha)}=\sum_{1\leq k\leq
d}y_{1}^{\alpha_{1}}\cdots
y_{d}^{\alpha_{d}}\partial_{x_{1}}^{\alpha_{1}}\cdots\partial_{x_{d}}^{\alpha_{d}}f(x+ty).$

By the Sobolev embedding

(A.38) $H^{s-1}\hookrightarrow
C^{[\theta],\theta-[\theta]},\qquad\theta=s-1-d/2,$

given $f\in H^{s-1}$ and $|\alpha|=[\theta],$ the map $\partial_{x}^{\alpha}f$
belong to $C^{0,\theta-[\theta]}.$ Thus we may write

$\partial_{x}^{\alpha}f(x+t(x^{\prime}-x))=\partial_{x}^{\alpha}f(x)+f_{\alpha}(x,t(x^{\prime}-x))|x^{\prime}-x|^{\theta-[\theta]},$

where $f_{\alpha}$ is bounded in both its arguments. This implies

(A.39) $\displaystyle
R_{[\theta]}(\chi,f)v=\frac{(-i)^{[\theta]}}{[\theta]!}\sum_{|\alpha|=m}(\partial_{x}^{\alpha}f)(x)(\partial_{\xi}^{\alpha}\chi)(D_{x})v+\tilde{R}_{\theta}(\chi,f)v,$

with notation

$\displaystyle\tilde{R}_{\theta}(\chi,f)v=\int_{\mathbb{R}^{3}}\int_{0}^{1}\frac{(-i(1-t))^{m-1}}{(m-1)!}\big{(}{\mathcal{F}}^{-1}\chi\big{)}(x-x^{\prime})\tilde{f}(x,t(x^{\prime}-x))|x^{\prime}-x|^{\theta}v(x^{\prime})\,dx^{\prime}\,dt^{\prime},$

where $\tilde{f}$ is bounded in both its arguments, with

$\|\tilde{f}\|_{L^{\infty}}\leq\sup_{\begin{smallmatrix}|\alpha|=[\theta]\\\
x\neq
y\end{smallmatrix}}\frac{|\partial_{x}^{\alpha}f(x)-\partial_{x}^{\alpha}f(y)|}{|x-y|^{\theta-[\theta]}}=:\sup_{|\alpha|=[\theta]}\|\partial_{x}^{\alpha}f\|_{0,\theta-[\theta]}.$

In particular, given $\chi$ compactly supported and $f\in H^{s-1},$ we find

$\|\tilde{R}_{\theta}(\chi,f)v\|_{L^{2}}\lesssim\||\cdot|^{\theta}{\mathcal{F}}^{-1}\chi\|_{L^{1}}\sup_{|\alpha|=[\theta]}\|\partial_{x}^{\alpha}f\|_{0,\theta-[\theta]}\|v\|_{L^{2}},$

where the upper bound is finite by (A.38).

In semiclassical quantization, that is with $\chi(2^{-j}D_{x})$ in place of
$\chi(D_{x}),$ it suffices to observe that

$\big{\|}|\cdot|^{\theta}{\mathcal{F}}^{-1}(\chi(2^{-j}\cdot))\big{\|}_{L^{1}}=2^{-j\theta}\big{\|}|\cdot|^{\theta}{\mathcal{F}}^{-1}(\cdot))\big{\|}_{L^{1}}.$

∎

###  A.6 Gårding’s inequality

For $d_{\star}:=3+d,$ for all $Q\in C^{d_{\star}}S^{0},$ if $\Re e\,Q\geq 0,$
then

(A.40) $\Re
e\,(\mathrm{pdo}_{j}(Q)u,u)_{L^{2}}+2^{-j}\|Q\|_{0,d_{\star},d_{\star}}\|u\|_{L^{2}}^{2}\geq
0.$

Inequality (A.40) is Gårding’s classical inequality in a semiclassical
setting, as it can be found for instance in Theorem 4.32 in [19]222Theorem
4.32 in [19] deals with operators in Weyl quantization. We can for instance
use Lemma 4.1.5 in [10] in order to bound the difference between the operator
in Weyl quantization and the operator in classical quantization. The
difference involves spatial derivatives of the symbol, which in semiclassical
quantization bring out a factor $2^{-j}.$. The specific number of derivatives
$d_{\star}=3+d$ needed in (A.40) is found in the proof of Theorem 1.1.26 in
[10].

The Gårding inequality (A.40) extends to para-differential operators, via the
analysis of Section A.4.1. Indeed, we observe that
$(H_{j}^{-1})^{\star}=H_{j}$ (the adjoint is in the $L^{2}$ sense), so that

$\Re e\,({\rm op}_{j}(Q)u,u)_{L^{2}}=\Re e\,(({\rm
op}(h_{j}Q)-\mathrm{pdo}(h_{j}Q))H_{j}u,H_{j}u)_{L^{2}}+\Re
e\,(\mathrm{pdo}_{j}(Q)u,u)_{L^{2}},$

and, as stated in Remark A.4, we may use bounds (A.31) and (A.32) in order to
bound the action of ${\rm op}(h_{j}Q)-\mathrm{pdo}_{j}(Q)$ as an operator from
$L^{2}$ to $L^{2}.$

Consider the case that $Q$ is compactly supported, jointly in $(x,\xi),$ and
the estimates (A.31) and (A.32) bounding the action of ${\rm
op}_{j}(h_{j}Q)-\mathrm{pdo}_{j}(h_{j}Q).$ We can use the arguments from the
proof of Proposition A.3 to verify the bound

$\langle\xi\rangle^{|\beta|}\|(1-\tilde{\phi}_{0}(D_{x}))(h_{j}\partial_{\xi}^{\beta}Q)\|_{H^{d/2}}\lesssim
2^{-j(s-d)}\|\partial_{\xi}^{\beta}Q\|_{H^{s}},$

for $s$ as large as allowed by the regularity of $Q.$ Since $Q$ is compactly
supported in $x,$ the above is controlled by
$2^{-j(s-d)}\|\partial_{\xi}^{\beta}\partial_{x}^{\alpha}Q\|_{L^{\infty}},$
for $|\alpha|\leq[s]+1.$ We see in the last lines of the proof of Proposition
A.3 that the low-frequency term $Q_{LF}$ contributes a similar term. Thus, for
$Q$ compactly supported in $(x,\xi),$ we find

$\|({\rm op}_{j}(h_{j}Q)-\mathrm{pdo}_{j}(h_{j}Q))v\|_{L^{2}}\lesssim
2^{-j(s-d)}\|Q\|_{0,[s]+1,2([d/2]+1)}\|v\|_{L^{2}},$

hence a paradifferential version of Gårding’s inequality as follows:

(A.41) $\displaystyle\Re e\,($ $\displaystyle{\rm op}_{j}(Q)u,u)_{L^{2}}$
$\displaystyle+\Big{(}2^{-j}\|Q\|_{0,d_{\star},d_{\star}}+2^{-j(s-d)}\|Q\|_{0,[s]+1,2([d/2]+1)}\Big{)}\|u\|_{L^{2}}^{2}\geq
0,$ for all $Q\in C^{\max(d_{\star},[s]+1)}S^{0}$ compactly supported in
$(x,\xi),$ and all $u\in L^{2},$

with $s>d.$

##  Appendix B Rates of growth via Gårding for the flows of operators with
symbols having Hölder spatial regularity

Given $\theta>0$ and $T>0,$ consider a continuous map $Q:t\in[0,T]\to Q(t)\in
C^{[\theta],\theta-[\theta]}S^{0}$ (see the definition of classical classes of
symbols with limited spatial regularity in Appendix A) with support near
$(x^{0},\xi^{0}),$ for some
$(x^{0},\xi^{0})\in\mathbb{R}^{d}\times\mathbb{S}^{d-1},$ in the following
sense:

(B.1)
$\mbox{supp}\,Q(t)\subset\big{\\{}(x,\xi)\in\mathbb{R}^{d}\times\mathbb{R}^{d},\quad|x-x^{0}|+|\xi-\xi^{0}|\leq
R\\},\quad\mbox{for some $0<R<1,$}$

uniformly in $t.$ The family of symmetric matrices $\Re e\,Q(t,\cdot)$ is
continuous and compactly supported. We denote $\gamma_{+}$ an upper spectral
bound:

(B.2) $\Re e\,\,Q(t)\leq\gamma_{+}(t),\quad\mbox{for some
$t\to\gamma_{+}(t)\in\mathbb{R},$ for all $(x,\xi),$}$

and $\gamma_{-}(t)$ a spectral lower bound in restriction to a smaller ball
near $(x^{0},\xi^{0}):$

(B.3) $\displaystyle\gamma_{-}(t)\leq\Re e\,Q(t),$ for some
$\gamma_{-}\in\mathbb{R},$ for all $(x,\xi)$ such that
$|x-x^{0}|+|\xi-\xi^{0}|\leq r,$ for some $0<r<R.$

In (B.2) and (B.3), the matrix $\Re e\,Q$ is the symmetric matrix
$(Q+Q^{*})/2.$ We use notation $\psi,$ like in the main proof, to describe a
smooth space-frequency cut-off such that (B.3) holds on the support of $\psi.$

Consider the initial-value problem

(B.4) $\left\\{\begin{aligned} \partial_{t}u={\rm op}_{j}(Q)u,\\\
u(0)=u_{0}\in L^{2}.\end{aligned}\right.$

Since $Q(t)$ is order 0, the linear operator ${\rm op}_{j}(Q(t))$ maps $L^{2}$
to $L^{2},$ hence (B.4) has a unique global solution $u$ by the Cauchy-
Lipschitz theorem.

We can prove the following lower and upper rates of growth for the solution
$u$ to (B.4):

######  Proposition B.1.

For the solution $u$ to (B.4), for some $C>0$ which depends only on $Q$ and
the dimension $d:$

  * •

We have the upper bound

(B.5)
$\|u(t)\|_{L^{2}}\leq\exp\Big{(}\int_{0}^{t}\big{(}\gamma_{+}(t^{\prime})+C2^{-j\theta_{\star}}\big{)}\,dt^{\prime}\,\Big{)}\|u_{0}\|_{L^{2}},$

where $\theta_{\star}:=\theta/(\theta+d_{\star}),$ with $d_{\star}$ is a
number of derivatives of the symbols in Gårding’s inequality (A.40),
$\theta>0$ is the Hölder regularity of $Q,$ and $\gamma_{+}$ is introduced in
(B.2).

  * •

Let $\tilde{\psi}$ be a space-frequency cut-off such that the lower bound
(B.3) holds over the support of $\tilde{\psi}.$ Then, given $u_{0}$ such that
${\rm op}_{j}(\tilde{\psi})u_{0}\neq 0,$ for $t$ such that

(B.6) $0\leq t\leq t_{\star},\quad\mbox{with $t_{\star}$ defined by
$\displaystyle{\int_{0}^{t_{\star}}(\gamma^{+}-\gamma^{-})(t^{\prime})\,dt^{\prime}=j(\theta_{\star}\ln
2-\varepsilon)}$}$

for any $\varepsilon>0,$ where $\theta_{\star}$ is introduced below (B.5) and
the lower rate $\gamma_{-}$ is introduced in (B.3), we have the lower bound,

(B.7)
$\frac{9}{10}\exp\Big{(}\int_{0}^{t}\big{(}\gamma_{-}(t^{\prime})-C2^{-j\theta_{\star}})\,dt^{\prime}\Big{)}\,\Big{)}\|{\rm
op}_{j}(\tilde{\psi})u_{0}\|_{L^{2}}\leq\|u(t)\|_{L^{2}},$

for large enough $j,$ depending only on $Q,$ $d,$ and $\varepsilon.$

###### Proof.

We use notations and results from Section A.3, as we regularize $Q$ into
$Q^{\varepsilon}\in C^{\infty}S^{0},$ by spatial convolution with a smoothing
kernel. By (A.19)-(A.22), we have

(B.8) $\|{\rm op}_{j}(Q-Q^{\varepsilon})\|_{L^{2}\to
L^{2}}\lesssim\varepsilon^{\theta},\qquad\mbox{and}\qquad\|Q^{\varepsilon}\|_{0,k,k^{\prime}}\lesssim\varepsilon^{-k},$

using notation (A.9). Via (B.8), the lower and upper bounds (B.3) and (B.2)
for $Q$ translate into lower and upper bounds for $Q^{\varepsilon}.$ We let

$\gamma_{+}^{\varepsilon}=\gamma_{+}+C\varepsilon^{\theta},\qquad\gamma_{-}^{\varepsilon}:=\gamma_{-}-C\varepsilon^{\theta},$

for some $C>0,$ and

$v_{\pm}=\exp\Big{(}-\int_{0}^{1}\gamma^{\varepsilon}_{\pm}(t^{\prime})\,dt^{\prime}\Big{)}u.$

Then, $v_{\pm}$ solve

$\displaystyle\partial_{t}v_{-}={\rm
op}_{j}(Q^{\varepsilon}-\gamma_{-}^{\varepsilon})v_{-}+{\rm
op}_{j}(Q-Q^{\varepsilon})v_{-}-C\varepsilon^{\theta}v_{-},$
$\displaystyle\partial_{t}v_{+}+{\rm
op}_{j}(\gamma_{+}^{\varepsilon}-Q^{\varepsilon})v_{+}={\rm
op}_{j}(Q-Q^{\varepsilon})v_{+}+C\varepsilon^{\theta}v_{+}.$

Since $\Re e\,(\gamma_{+}^{\varepsilon}-Q^{\varepsilon})\geq 0$ in the whole
domain $\mathbb{R}^{d}_{x}\times\mathbb{R}^{d}_{\xi},$ we may apply Gårding’s
inequality (A.40), and obtain

$\frac{1}{2}\partial_{t}\|v_{+}\|_{L^{2}}^{2}\leq
C(2^{-j}\varepsilon^{-d_{\star}}+2^{-j(s-d)}\varepsilon^{-([s]+1)}+\varepsilon^{\theta})\|v_{+}\|_{L^{2}}^{2},$

for any $s$ and for some constant $C>0,$ which depends neither on $j$ nor on
$\varepsilon.$ We choose $\varepsilon$ and $s$ so that the errors above are
the same order of magnitude:

(B.9) $\varepsilon:=2^{-j/(\theta+d_{\star})},$

and $s\geq(1-1/(\theta+d_{\star}))^{-1}(d+(\theta+1)/(\theta+d_{\star})).$
Then all three errors above have size $2^{-j\theta/(\theta+d_{\star})},$ up to
a multiplicative constant which does not depend on $j.$ The upper bound (B.5)
follows.

The proof of the lower bound (B.7) is just a bit more delicate, since the
posited lower bound (B.3) on $\Re e\,Q$ is valid only locally, on the support
of $\tilde{\psi}.$ Let $\tilde{\psi}^{\flat}$ such that
$\tilde{\psi}^{\flat}\prec\tilde{\psi},$ and

$w:={\rm op}_{j}(\tilde{\psi}^{\flat})v_{-}.$

Then, $w$ solves

$\partial_{t}w={\rm op}_{j}(\tilde{\psi}^{\flat}){\rm
op}_{j}(Q^{\varepsilon}-\gamma_{-}^{\varepsilon})v_{-}+{\rm
op}_{j}(\tilde{\psi}^{\flat}){\rm op}_{j}(Q-Q^{\varepsilon})v_{-}.$

Using $\tilde{\psi}^{\flat}\prec\tilde{\psi}$ and (A.34), we find

${\rm op}_{j}(\tilde{\psi}^{\flat}){\rm op}_{j}(Q^{\varepsilon})={\rm
op}_{j}(\tilde{\psi}^{\flat}\tilde{\psi}){\rm op}_{j}(Q^{\varepsilon})={\rm
op}_{j}(\tilde{\psi}Q^{\varepsilon}){\rm
op}_{j}(\tilde{\psi}^{\flat})+2^{-j}{\rm op}_{j}(Q^{\varepsilon}_{1}),$

where $Q^{\varepsilon}_{1}\in S^{0}$ is bounded by (A.35), in particular
involves one spatial derivative of $Q^{\varepsilon},$ so that, with the second
bound in (B.8),

$\|{\rm op}_{j}(Q^{\varepsilon}_{1})\|_{L^{2}\to
L^{2}}\lesssim\varepsilon^{-1}.$

uniformly in $j.$ Thus the equation in $w$ is

$\partial_{t}w={\rm
op}_{j}(\tilde{\psi}(Q^{\varepsilon}-\gamma_{-}^{\varepsilon}))w+2^{-j}{\rm
op}_{j}(Q^{\varepsilon}_{1})v_{-}+{\rm op}_{j}(\tilde{\psi}^{\flat}){\rm
op}_{j}(Q-Q^{\varepsilon})v_{-}+C\varepsilon^{\theta}w.$

We now apply (A.41):

$\frac{1}{2}\partial_{t}\|w\|_{L^{2}}^{2}+2^{-j\theta/(\theta+d_{\star})}C\|w\|_{L^{2}}^{2}\geq\Re
e\,\Big{(}2^{-j}{\rm op}_{j}(Q^{\varepsilon}_{1})v_{-}+{\rm
op}_{j}(\tilde{\psi}^{\flat}){\rm
op}_{j}(Q-Q^{\varepsilon})v_{-},w\Big{)}_{L^{2}}.$

The $2^{-j\theta/(\theta+d_{\star})}$ prefactor in the Gårding error term is
as in the upper bound, above. The upper bound (B.5) gives an upper bound for
$v_{-}:$

$\|v_{-}(t)\|_{L^{2}}\leq\|u_{0}\|_{L^{2}}\exp\Big{(}\int_{0}^{t}\big{(}\,(\gamma^{\varepsilon}_{+}-\gamma^{\varepsilon}_{-})(t^{\prime})+C2^{-j\theta_{\star}}\big{)}\,dt^{\prime}\Big{)},$

hence

$\displaystyle\partial_{t}\|w\|_{L^{2}}$
$\displaystyle+C2^{-j\theta_{\star}}\|w\|_{L^{2}}$
$\displaystyle\geq-C\Big{(}2^{-j}\varepsilon^{-1}+\varepsilon^{\theta}\Big{)}\exp\Big{(}\int_{0}^{t}\big{(}\,\gamma^{\varepsilon}_{+}-\gamma^{\varepsilon}_{-}+C2^{-j\theta_{\star}}\big{)}\,dt^{\prime}\Big{)}\|u_{0}\|_{L^{2}}.$

The above lower bound implies

$\displaystyle\|w(t)\|_{L^{2}}$ $\displaystyle\geq
e^{-tC2^{-j\theta_{\star}}}\|w_{0}\|_{L^{2}}$
$\displaystyle-C\Big{(}2^{-j}\varepsilon^{-1}+\varepsilon^{\theta}\Big{)}\exp\Big{(}\int_{0}^{t}\big{(}(\gamma^{\varepsilon}_{+}-\gamma^{\varepsilon}_{-})(t^{\prime})+C2^{-j\theta_{\star}}\big{)}\,dt^{\prime}\Big{)}\|u_{0}\|_{L^{2}}.$

Recall the definition of $\varepsilon$ in terms of $j$ in (B.9). We have
$2^{-j}\varepsilon^{-1}\leq 2^{-j\theta/(\theta+d_{\star})}.$ Thus the above
lower bound takes the form

$\displaystyle\|w(t)\|_{L^{2}}$ $\displaystyle\geq
e^{-tC2^{-j\theta_{\star}}}\|w_{0}\|_{L^{2}}$
$\displaystyle-C2^{-j\theta_{\star}}\exp\Big{(}\int_{0}^{t}\big{(}(\gamma^{\varepsilon}_{+}-\gamma^{\varepsilon}_{-})(t^{\prime})+C2^{-j\theta_{\star}}\big{)}\,dt^{\prime}\Big{)}\|u_{0}\|_{L^{2}}.$

By assumption on $u_{0},$ the ratio
$c_{0}:=\|w(0)\|_{L^{2}}/\|u_{0}\|_{L^{2}}$ is positive, and we have

$\displaystyle\|w(t)\|_{L^{2}}$ $\displaystyle\geq
e^{-tC2^{-j\theta_{\star}}}\|w_{0}\|_{L^{2}}\Big{(}1-\frac{C}{c_{0}}2^{-j\theta_{\star}}\exp\Big{(}\int_{0}^{t}(\gamma^{\varepsilon}_{+}-\gamma^{\varepsilon}_{-}+C2^{-j\theta_{\star}}\,)\Big{)}.$

Thus the condition is

$\frac{C}{c_{0}}2^{-j\theta_{\star}}\exp\Big{(}\int_{0}^{t}(\,\gamma^{\varepsilon}_{+}-\gamma^{\varepsilon}_{-}+C2^{-j\theta_{\star}})\,\Big{)}\leq\frac{1}{10}.$

That is, for some $C>0$ which depends only on $Q$ and the dimension $d,$

$C2^{-j\theta_{\star}}\exp\Big{(}\int_{0}^{t}(\gamma_{+}-\gamma_{-}+C2^{-j\theta_{\star}})\,\Big{)}\|u_{0}\|_{L^{2}}\leq\frac{1}{10}\|{\rm
op}_{j}(\psi^{\flat})u_{0}\|_{L^{2}}.$

If $t$ is not too large, in the sense of (B.6), and $j$ is large enough, then
the above inequality holds, and this implies the lower bound (B.7). ∎

## References

  * [1] J. Bourgain, D. Li. Strong ill-posedness of the incompressible Euler equation in borderline Sobolev spaces. Invent. Math. 201 (2015), no. 1, 97-157.
  * [2] T. Kato, Perturbation theory for linear operators. Reprint of the 1980 edition. Classics in Mathematics. Springer-Verlag, Berlin, 1995.
  * [3] G.Métivier, _Remarks on the well posedness of the nonlinear Cauchy problem._ Contemp.Math., vol. 368, Amer. Math. Soc., Providence, RI, 2005.
  * [4] G.Métivier, _Para-differential Calculus and applications to the Cauchy problem for nonlinear systems_ , Centro di Ricerca Mathematica Ennio De Giorgi (CRM) Series, 5. Edizioni della Normale, Pisa, 2008.
  * [5] B. Morisse, On hyperbolicity and Gevrey well-posedness. Part one: the elliptic case. Annales Henri Lebesgue, Volume 3 (2020) p. 1195–1239.
  * [6] B. Morisse, On hyperbolicity and Gevrey well-posedness. Part two: Scalar or degenerate transitions. J. Differential Equations 264 (2018), no. 8, 5221-5262.
  * [7] B. Morisse, On hyperbolicity and Gevrey well-posedness. Part three: a model of weakly hyperbolic systems. https://arxiv.org/abs/1803.04724, 2018. To appear in Indiana University Mathematics Journal.
  * [8] D. Lannes, Sharp estimates for pseudo-differential operators with symbols of limited smoothness and commutators. J. Funct. Anal. 232 (2006), no. 2, 495–539.
  * [9] P. D. Lax, Asymptotic solutions of oscillatory initial value problems. Duke Math. J. 24 (1957) 627–646.
  * [10] N. Lerner, _Metrics on the Phase Space and Non-Selfadjoint Operators_. Pseudo-Differential Operators. Theory and Applications, 3. Birkhäuser 2010. xii+397 pp.
  * [11] N. Lerner, Y. Morimoto, C.-J. Xu, Instability of the Cauchy-Kovalevskaya solution for a class of non-linear systems, American J. Math., 132 (2010), 1, 99-123.
  * [12] N. Lerner, T. Nguyen, B. Texier, The onset of instability in first-order systems. J. Eur. Math. Soc. (JEMS) 20 (2018), no. 6, 1303-1373.
  * [13] Y. Lu, B. Texier, A stability criterion for high-frequency oscillations, Mém. Soc. Math. Fr. 142 (2015).
  * [14] S. Mizohata, Some remarks on the Cauchy problem. J. Math. Kyoto Univ. 1961-1962 109–127.
  * [15] K. Ndoumajoud, B. Texier, On Métivier’s Lax-Mizohata theorem and extensions to weak defects of hyperbolicity. Part two. Preprint, 2020.
  * [16] B. Texier, _Derivation of the Zakharov equations_ , Archive for Rational Mechanics and Analysis 184 (2007), 121–183.
  * [17] B. Texier, _Approximations of pseudo-differential flows,_ Indiana Univ. Math. J. 65 (2016), no.1, 243-272.
  * [18] B. Texier, _Basic matrix perturbation theory._ Enseign. Math. 64 (2018), no. 3-4, 249–263.
  * [19] M. Zworski, _Semi-classical analysis_ , Graduate studies in mathematics, AMS, 2012.