# A “Game of Like” : Online Social Network Sharing As Strategic Interaction

Emmanuel J. Genot I wish to thank Erik Mohlin, Jens Ulrik Hansen, Justine
Jacot, and Patricia Rich, for their invaluable help at the various stages of
this paper’s development; three anonymous referees, whose comments and
suggestions brought about some major changes and (hopefully) improvements; and
Rineke Verbrugge, who reviewed those changes, and suggested further
improvements. Any mistakes left are on me.Department of Philosophy
Lund University
Lund, Sweden<EMAIL_ADDRESS>
###### Abstract

We argue that behavioral science models of online content-sharing overlook the
role of strategic interactions between users. Borrowing from accuracy-nudges
studies decision-theoretic models, we propose a basic game model and explore
special cases with idealized parameter settings to identify refinements
necessary to capture real-world online social network behavior. Anticipating
those refinements, we sketch a strategic analysis of content amplification and
draw a connection between Keynes’ “beauty contest” analogy and recent social-
epistemological work on echo chambers. We conclude on the model’s prospects
from analytical and empirical perspectives.

##  1 Motivations

Online search engines garnered attention from social epistemologists in the
early days of the commercial Internet, when A. Goldman analyzed them as
retrieval systems in [6]. Later, T.A. Simpson extended Goldman’s analysis into
a model of surrogate expertise in [22] in direct response to Google Search
personalization algorithms. Recently, epistemologists have turned to online
social networks (hereafter osn), fulfilling a similar function of online
information sources, with even greater personalization. Notably, T.C. Nguyen
[13] provided a much-needed conceptual analysis of osn epistemic bubbles and
echo chambers, and C. O’Connor and J.O. Weatherall [15] proposed that applying
network epistemology to osn could address limitations of contagion models of
online information spread. At the same time, behavioral scientists have
independently addressed the limitations of contagion models by looking at osn-
sharing through a rational choice lens. Particularly, studies that shaped the
field and its public perception have manifested a Bayesian influence. Widely
publicized studies like [24] (a Science cover story: “How lies spread–On
social media, fake news beats the truth”) and [17] (a Nature cover story:
“Misinformation–A prompt to check accuracy cuts online sharing of fake news”)
appealed to Bayesian decision theory and expected utility theory to
rationalize osn content-sharing and interpret diffusion-model data
analyses.111“[U]ser characteristics and network structure could not explain
the differential diffusion of truth and falsity, we sought alternative
explanations for the differences in their diffusion dynamics. One alternative
explanation emerges from information theory and Bayesian decision theory.
Novelty attracts human attention, contributes to productive decision-making,
and encourages information sharing because novelty updates our understanding
of the world.” [24, p. 1149]. Similarly, “people do care more about accuracy
than other content dimensions but accuracy nonetheless often has little effect
on sharing, because (ii) the social media context focuses [users’] attention
on other factors such as the desire to attract and please followers/friends or
to signal one’s group membership. In the language of utility theory, an
‘attentional spotlight’ is shone upon certain terms in the decider’s utility
function, such that only those terms are weighed when making a decision” [17,
p. 591]. The framework of [17] is implicitly decision-theoretic, as utilities
take as argument proxies for individual choices (content shared) rather than
strategic profiles (cf. Section 2).

Decision theory best models decisions under uncertainty about the state of
nature, but osn-sharing outcomes depend on reactions from a community of
users. The preferred model for decisions under uncertainty about other agents’
decisions is game theory, and while the formalisms are inter-translatable,
decision theory is less expressive. As pointed out by J. Harsanyi, the game-
to-decision direction loses in translation the explicit expression of mutual
expectations of rationality (via solution concepts [7, 8]). Compounding this
issue, decision-theoretic models from behaviral science studies (such as [24,
17]) were not proposed as translations for games and thus did not explicitly
translate mutual expectations into constraints on decision-makers priors (as
per the games-to-decision direction, cf. [9, 7]), leaving their role almost
entirely unanalyzed. Unfortunately, social epistemology offers no ready-made
solution. Nguyen’s analysis is strategic but informal and cannot bear on the
data without a formal reconstruction, while network epistemology does not
address strategic expectations formally.

The absence of a strategic analysis of osn-sharing motivated the approach
presented in the remainder of this paper. Section 2 builds upon behavioral
science decision-theoretic models to propose a simplified game model for osn-
sharing, differentiating between content-based and engagement-based
preferences. Section 3 examines special cases that highlight the model’s
salient features and limitations and identifies extensions necessary to
reconstruct real-life osn users’ behavior. Section 4 extrapolates informally
and proposes that special cases of osn-sharing elicit strategy selection akin
to reasoning in guessing games and could illuminate content amplification
scenarios, including Nguyen’s epistemic bubbles and echo chambers. We conclude
with the analytic prospects of a strategic re-interpretation of extant data,
and a suggestion for the design of new studies.

##  2 A Game of Like

Behavioral science studies of osn-sharing often acknowledge the role of
strategic interactions between users but have so far fallen short of factoring
in their contribution. Pennycook et al. (2021) is a paradigmatic example: the
authors note that “the desire to attract and please followers/friends or to
signal one’s group membership” [17, p. 591] contributes to content-sharing
decisions, but propose a utility function limited to personal preferences for
content having such-and-such characteristics.222“Consider a piece of content
$x$ which is defined by $k$ different characteristic dimensions [including]
whether the content is false/misleading $F(x)$, and other $k-1$ dimensions
[that] are non-accuracy-related (e.g. partisan alignment, humorousness, etc)
defined as $C_{2}(x)\dots C_{k}(x)$. In our model, the utility […] from
sharing content $x$ is given by:
$U(x)=-\alpha_{1}\beta_{F}F(x)+\sum^{k}_{i=2}\alpha_{i}\beta_{i}C_{i}(x)$
where $\beta_{F}$ indicates how much they dislike sharing misleading content
and $\beta_{2}\dots\beta_{k}$ indicate how much they care about each of the
other dimensions (i.e. $\beta$s indicate preferences); while $\alpha_{1}$
indicates how much the person is paying attention to accuracy, and
$\alpha_{2}\dots\alpha_{k}$ indicate how much the person is paying attention
to each of the other dimensions.” [17, Supplementary Information: S9-10] A
natural first step toward a strategic model is thus to introduce the missing
terms, then specify a game based on this completed picture. For simplicity, we
can let $u_{p_{i}}(\cdot)$ denote $i$’s personal utility, expressing how some
content aligns with $i$’s personal preferences for content having such-and-
such characteristics, with the understanding that this alignment could be
further analyzed along multiple dimensions (as in [17], cf. n. 2). To that, we
add a term that we denote $u_{s_{i}}(\cdot)$, for the social utility,
expressing how reactions to the content shared—‘likes,’ re-shares, comments,
etc.—satisfy $i$’s preferences for social validation or, more generally,
engagement. Finally, we introduce a parameter, that we denote $\gamma$, to
represent the relative weights of $i$’s personal preferences for content and
social preferences for engagement. In the decision-theoretic model of [17],
the only action being ‘sharing,’ actions and content are indiscernible, and
the utility function can range over the content. In a game-theoretic model,
the utility function ranges over strategy profiles, and we must distinguish
content from actions.

As a basic model, we consider an $n$-player repeated game $G$ in strategic
form with a set $P$ of players, where any $i\in P$ can, at each round, ‘like’
or ‘share’ content. As simplifications, we assume that players only share new
content at round $r=0$, so any ‘share’ action at round $r\geq 1$ is a
‘reshare.’ Under this simplification, we can specify a set of (original)
content $C_{G}=\\{c_{1},\dots,c_{n}\\}$ for $G$, where $c_{i}$ is the content
introduced at $r=0$ by agent $i$. The action set for some player $i$ at some
round $r$ is $A_{i}=\\{\textsf{like}_{i}(x,y),\textsf{share}_{i}(x,y):x\in
C_{G},y\in P\\}$, where $y$ is a player who shared $x$ at some round
$r^{\prime}<r$, and from whom $i$ is re-sharing $x$. Note that, under our
simplifying assumption, at $r=0$, there is nothing to ‘like.’ If all actions
are visible to all players, no restriction is imposed on $x$ or $y$.
Explicitly: any content shared by some player at round $r$ can be reshared by
any other player at round $(r+1)$. This amounts to a game with perfect
information, adequate for demonstrating the strategic standpoint’s
fruitfulness but insufficient to model real-world osns (see below). Our
earlier discussion of personal and social preferences yields a utility
function, as below.

$\displaystyle U_{i}(\cdot)$ $\displaystyle=$
$\displaystyle\gamma_{i}u_{p_{i}}(\cdot)+(1-\gamma_{i})u_{s_{i}}(\cdot)$ (1)

Intuitively, in the decision-theoretic approach, $u_{p_{i}}(c)$ expresses
$i$’s preferences as a function of the distance between $c$ and $i$’s ideal
content located in a multi-dimensional space whose dimensions correspond to
$i$’s criteria of evaluation. In a round of $G$, the argument of (1) is a
strategy profile $\overline{\sigma}=(\sigma_{1},\dots\sigma_{n})$ where
$\sigma_{i}$ is player $i$’s strategy at that round. Following the same
intuition, $u_{p_{i}}(\overline{\sigma})$ can be thought of as a function of
the relative distances between $i$’s ideal content and the ‘community content’
$c_{1},\dots,c_{n}$, or some weighed sum thereof, representing how close
$C_{G}$ is to $i$’s ideal.333Note that $i$ may be indifferent to others’
strategies, in which case
$u_{p_{i}}(\overline{\sigma})=u_{p_{i}}(\overline{\sigma}^{\prime})$ whenever
$i$’s strategy $\sigma_{i}$ is the same in $\overline{\sigma}$ and
$\overline{\sigma}^{\prime}$. So construed, and under our simplification,
$u_{p_{i}}(\overline{\sigma})$ remains constant after $r=0$. Again, this
personal preference model is sufficient for our purposes. Still, in a real-
world osn, overall engagement could indirectly affect
$u_{p_{i}}(\overline{\sigma})$ ($i$ may care for overall visibility, and in a
model with incomplete information, visibility would depend on engagement, see
below).

In a decision-theoretic model (e.g., extending that of [17]) $u_{s_{i}}(c)$
would be a function of the (accumulated) engagement from users other than $i$,
with $c$ (when shared by $i$). In $G$, $u_{s_{i}}(\overline{\sigma})$ at round
$r$ is, in part, a function of how other players have engaged in $r$ with the
content $i$ shared at some $r^{\prime}>r$; and in part, of the accrued social
utility inherited from earlier rounds. The candidate functions for computing
either component are too numerous to review here, and which one applies to
particular cases may be empirically constrained by algorithms. Still, it
suffices for our purposes to note that, at some round $r$,
$u_{s_{i}}(\overline{\sigma})$ does not ‘reset’ $i$’s social utility; that the
contribution of ‘likes’ and (re)shares may vary; and that evaluations may
depend on players’ knowledge.444For a concrete example, Twitter’s ranking
algorithm weighs ‘like’ reactions more than re-tweets (reshares) when
determining which content should appear in users’ feeds. A knowledgeable user
may prioritize sharing content they believe would receive ‘likes’ to optimize
the chances that other users are exposed to their content later, whether they
value engagement as social validation or as a means to increase content
visibility. For definiteness, we can assume a function
$u_{s_{i}}(\overline{\sigma})$ that ranks higher strategy profiles where
content $i$ shared (or reshared) is both liked and reshared rather than liked
or shared (alone)—i.e., a function that takes some weighed sum of ‘likes’ and
(re)shares, rather than an average (or an argmax). This justifies the
shorthand “game of like”—as a nod to J. Conway’s “game of life” [5]—since the
preferred social outcome, over repetitions, is like-and-reshare, a
strengthened form of ‘like’ (“game of share” would be equally justified, but
the homage and homophony would be lost).

Let us conclude this section with a few words on our model’s (self-imposed)
limitations. In real-world osn, new content can be introduced at any time, and
players have only a partial picture of the content they can reshare. A more
realistic “game of like” would have imperfect information (e.g., as a model of
bounded attention): any content $c$ would be available to a player $i$ to
react to at $r$ with a certain probability, depending on overall engagement
with $c$ prior to $r$. In such a model, $i$ could be aware of some $c$, close
to $i$’s ideal content, and care for its visibility (the probability of $c$
being available to other players) and thus for other players’ engagement with
$c$. Conversely, $i$ might not worry much about some $c$, far removed from
their ideal, as long as $c$’s probability of being available to other players
would remain low. Still, a simplified model with perfect information already
acknowledges the relevance of overall interaction by virtue of the argument of
$\gamma_{i}u_{p_{i}}(\cdot)$ being a strategy profile, and thus furthers goal
of identifying strategic components of osn-sharing. Hence, our “game of like”
with perfect information is a proof-of-concept and a foundation for future
developments. The next section considers special cases, varying players’
$\gamma$ types, to determine which refinements would be necessary to turn the
proof-of-concept model into a model for real-world osns.

##  3 Strategy Selection

Let us begin with the limit case where, for all $i\in P$, $\gamma_{i}=1$,
denoted $G_{\gamma=1}$ for later reference. We could distinguish a priori
between a variety of subcases, depending on whether players have non-equivocal
prior beliefs about other players’ personal preferences; and/or whether they
have non-equivocal prior beliefs about other players’ $\gamma$. However, the
differences between those subcases are inconsequential. To see this, assume an
arbitrary player $i$ in $G_{\gamma=1}$ who does have non-equivocal prior
beliefs about other players’ personal preferences for content and
$\gamma$-type (say, following a round of cheap talk). Ex hypothesis, at any
round $r$ of $G_{\gamma=1}$, for any $i\in P$,
$U_{i}(\overline{\sigma})$=$u_{p_{i}}(\overline{\sigma})$. Hence, $i$’s best
strategy at round $r=0$ is to share whatever content $c_{i}$ available to them
that is closest to their ideal content (according to their dimensions of
evaluation). Beliefs about other players’ preferences and $\gamma$ type do not
affect that choice. Hence, $i$ would choose the same content without any
information about other players. Since the only assumption we made about $i$
is that $\gamma_{i}=1$, this generalizes to any $i\in P$ for $G_{\gamma=1}$
(and yields an equilibrium solution in the basic model for $r=0$ in
$G_{\gamma=1}$). Under the assumption that content is only introduced at round
$r=0$, the distance between the ‘community content’ and any player $i$’s ideal
content remains constant across repetitions, whatever their strategy at
$(r\geq 1)$. Relaxing this simplifying assumption is one way to model how
players can attempt to drive community content closer to their preferences by
sharing more content closer to their ideal at any new round $(r\geq 1)$. But
this would not bring the model closer to real-world osn, as “spamming” content
is only efficient if the content is visible, bringing us back to a version of
the “game of like” with imperfect information. Conversely, a “game of like”
without content introduced at round $r>0$, and with $(\gamma=1)$-players only,
would be susceptible to manipulations by coalitions of like-minded players,
who would want to see some content promoted. Therefore, relaxing the
assumption that no new content is introduced past $r=0$ would not be
especially illuminating without an explicit topological model of content
distances and auxiliary assumptions about how variable availability of content
correlates with engagement.

In a second limit case, denoted $G_{\gamma=0}$, all $i\in P$ are such that
$\gamma_{i}=0$. Unlike $G_{\gamma=1}$, player priors about others can
significantly impact the game. To see this, consider the limit subcase where
players’ $\gamma$ type is common knowledge. Then, $G_{\gamma=0}$ becomes a
game of reciprocation-or-retaliation or quid pro quo, where players either
trade reciprocal ‘likes’ and re-shares, or ignore one another, and where
content becomes inconsequential (so that it matters little whether new content
can be introduced after $r=0$ or not). To see this, consider a simple
$G_{\gamma=0}$ case with two players $i$ and $j$, content introduction
restricted to $r=0$, and (as a simplification) no marginal utility for
‘liking’ or re-sharing one’s content. Hence, the only utility $i$ and $j$ can
get is from the other player’s liking or re-sharing their content. At $r=0$,
they share (resp.) $c_{i}$ and $c_{j}$. At $r=1$, $i$ ($j$) can like or re-
share $c_{j}$ ($c_{i}$), or do nothing (for definiteness: repeating their move
from $r=0$). If either does nothing at $r=1$, the other can retaliate at $r=2$
by playing nothing; otherwise, they can reciprocate and play the remaining
action (like, or reshare) they did not play at $r=1$. With no introduction of
new content, they can repeat the cycle over $c_{i}$ and $c_{j}$. If new
content is allowed, they can repeat cycles of three rounds (introduction,
like, or re-share, then reciprocation or retaliation) to accrue utility. The
strategy just described turning the “game of like” into a game of
reciprocation-or-retaliation, and resembles the tit-for-tat strategy in the
repeated Prisonner’s Dilemma.

As extreme as it is, this case suggests that when $(\gamma=0)$-players have
non-equivocal beliefs about one another’s $\gamma$ type, the closer the
players are to having correct beliefs, the closer $G_{\gamma=0}$ resembles a
quid pro quo game. Assume now a subcase of $G_{\gamma=0}$ where players have
equivocal beliefs about $\gamma$ types—i.e., do not know that other players
are $(\gamma=0)$-players. If they also have equivocal beliefs about other
players’ personal preferences for content, the rational choice (for any $i$)
is a mixed strategy assigning equal weight to any content $i$ has access to at
$r=0$ and hope for the best. Lifting the restriction on content introduction
is more consequential than in the $G_{\gamma=1}$ case, as repeated
observations of others’ sharing behavior are necessary to infer their personal
preferences for content from their actions or their preferences for
engagement. Since, ex hypothesis, no player in $G_{\gamma=0}$ actually cares
for content (as long as they receive engagement), inferences from sharing
behavior to personal preferences could result in ‘false consensus’ situations
if players gradually amplify a salient type of content, leading to an echo
chamber (in the sense of [13]; cf. Section 4). However, even without lifting
the assumption, we can form a picture of a repeated game with new content by
assuming a round of cheap talk prior to $r=0$, during which players can form
priors (or update equivocal priors) about other players’ preferences based on
observed behavior. Suppose that some candidate content appears salient for
eliciting positive reactions—say, pictures of cats in precarious positions.
Then, upon engaging in $G_{\gamma=0}$, players could anticipate similar
pictures to elicit ‘like’ and ‘share’ reactions, skewing the content shared at
$r=0$ toward pictures of cats in precarious positions. Thus, it would appear
that a majority of players favor cat pictures. Even without the introduction
of new content, this could lead to cat pictures being increasingly reshared at
every $r\geq 1$ without (ex hypothesis) any player selecting their strategy
out of personal preference for that type of content, resulting in a ‘false
consensus.’ Again, as with $G_{\gamma=1}$, how engagement could impact
visibility appears more critical than whether or not content may be repeatedly
introduced. Subsequently, the need to accommodate $(\gamma<1)$-players does
not require further refinements beyond those suggested by $G_{\gamma=1}$:
imperfect information and an explicit content evaluation and comparison model.
The latter would, in particular, suffice for representing how
$(\gamma<1)$-players form (and revise) beliefs about the majority’s opinion,
instrumental in selecting strategies for eliciting engagement.

##  4 Mutual Expectations and Social Influence

Our remark about the majority’s opinion being of import to
$(\gamma<1)$-players may remind the reader of J.M. Keynes’ “Beauty Contest”
analogy for professional investment, quoted below.

> [P]rofessional investment may be likened to those newspaper competitions in
> which the competitors have to pick out the six prettiest faces from a
> hundred photographs, the prize being awarded to the competitor whose choice
> most nearly corresponds to the average preferences of the competitors as a
> whole. […] It is not a case of choosing those [faces] that, to the best of
> one’s judgment, are really the prettiest, nor even those that average
> opinion genuinely thinks the prettiest. We have reached the third degree
> where we devote our intelligences to anticipating what average opinion
> expects the average opinion to be. [10, p. 156]

The parallel is intentional: we propose that Keynes’ “third degree” describes
the reasoning of a $(\gamma<1)$-player selecting a strategy that could elicit
(re)share reactions from other $(\gamma<1)$-players who would want to elicit
‘like’ reactions. More generally, a “game of like” with some proportion of
$(\gamma<1)$-players relates to guessing games, proposed as a generalization
of J.M. Keynes’ beauty contest by R. Nagel (first, in [12]; see [2] for an
overview of empirical studies). A formal reconstruction of this suggestion
would require an explicit model of preference distances (already identified as
a necessary refinement for our basic model to capture real-world osns), but we
can offer an informal sketch.

Asssume the standpoint of a player of type $\gamma=0$, that we will denote
$\gamma_{0}$, reasoning about other players of a “game of like.”555We assume
that the agent is a $(\gamma=0)$-player rather than a weaker $(\gamma<1)$ to
avoid dealing with correlations between personal preferences for content and
preferences for engagement. Otherwise, we would have to factor in the cost of
sharing contrary-to-preference content, which could offset the benefit of
engagement. When choosing between multiple options for content to share, when
$\gamma_{0}$’s goal is accruing “like” reactions, $\gamma_{0}$ is equally
well-off: ($i$) choosing based on their own preferences for content, or:
($ii$) choosing based on the majority’s preference (e.g., as inferred
following a round of cheap talk) when preferences agree; and: ($iii$) possibly
worse off, when preferences disagree. In case ($iii$), $\gamma_{0}$ would be
better off switching to an option that agrees with the majority’s (displayed)
preferences. Thus, options based on $\gamma_{0}$ preferences are weakly
dominated by options based on the majority’s preferences (as inferred by
$\gamma_{0}$). Consider now how $\gamma_{0}$ would approach selecting a
strategy for eliciting “share” reactions; as simplification, assume that
$\gamma_{0}$ believes that most players are like him and care more for
engagement than for content. Then, $\gamma_{0}$ expects that most players
would (re-)share content to elicit (at least) ‘like’ reactions. If
$\gamma_{0}$ assumes that those players are rational, they expect those
players to reason to ($i$–$iii$) above. From there, $\gamma_{0}$ can conclude
that selecting an option based on their own preferences for content would
yield the same payoff as choosing based on the majority’s opinion of the
majority’s (displayed) preferences for content (if in agreement); and possibly
a worse payoff (if in disagreement). In the latter case, $\gamma_{0}$ would be
better off switching options. Hence, a selection based on the majority’s
opinion of the majority’s (displayed) preferences weakly dominates a selection
based on $\gamma_{0}$’s preferences for content alone.

The argument just sketched guesstimates too many important parameters to be
general—e.g., the respective distribution of $\gamma$ types among the players,
the cost of seeking social feedback with contrary-to-personal preferences for
other players that $\gamma_{0}$, how $\gamma_{0}$ would arrive at estimates
for those, etc. However, it suffices to motivate a comparison between a
subclass of “game of like,” Keynes’ beauty contest, and Nagel’s guessing
games. And empirical motivation for this comparison would be the
reconstruction of the real-world osn behavior colloquially called ‘signal
boosting,’ whereby users of an osn leverage the influence of public figures
(“influencers”) with a larger following base, tagging them in hope to be re-
shared. A well-reported example is a November 13, 2020 Twitter video featuring
actor R. Quaid reading aloud an earlier tweet from then-US president D.J.
Trump under a stroboscopic light, with an over-dramatic voice. Trump
(unsurprisingly) reshared Quaid’s video, which then accrued millions of views
from Trump’s followers, reaching beyond Quaid’s following. In fact, we have
already encountered in Section 3 a variant of (involuntary) signal-boosting
behavior, as a pathway to amplification (false consensus) when discussing
$G_{\gamma=0}$. This seems grounds enough to suggest that a “game of like”
model of osn with influencers could contribute to a formal theory of online
amplification, echoing Keynes’ motivations for the beauty-contest analogy
(speculative asset bubbles).

Another possible contribution that circles back to social epistemology is a
possible formal reconstruction of Nguyen’s conceptual analysis [13]. Nguyen
proposes that epistemic bubbles occur when individuals receive limited
exposure to information sources challenging their pre-existing beliefs, in
contrast to echo chambers, which emerge when individuals receive extensive
exposure to information sources that align with their pre-existing beliefs.
Epistemic bubbles result from combined personal choice and algorithmic
curation, particularly when online platforms tailor content to individual
preferences, thereby restricting the information individuals encounter. In an
echo chamber, people reinforce their views and are shielded from diverse
perspectives and alternative information, leading to the exclusion of
dissenting opinions. Nguyen notes that epistemic bubbles are easy to burst
with the presentation of contrary evidence, while echo chambers are self-
reinforcing, with social interaction actively fostering distrust of outside
sources. Nguyen’s analysis of echo chambers invites a formal reconstruction in
a “game of like” model with imperfect information, bringing it closer to the
methodological frameworks of behavioral science (modulo a game-to-decision
translation).

##  5 Concluding Remarks

We argued in Section 1 that, while osn-sharing is a strategic interaction,
behavioral science models overlook the contribution of strategic
anticipations. We extrapolated from behavioral science decision-theoretic
models a basic game model of osn-sharing (Section 2) and explored some limit
cases to determine refinements necessary to capture real-world osn-sharing
(Section 3). A connection with Keynes’ Beauty contest (and, more generally,
guessing games) allowed us to sketch a strategic analysis of content
amplification in the presence of influencers and users leveraging influence
and suggested a direction for the model’s development (Section 4). Still, a
“game of like” model may not contribute to conceptual analysis beyond a formal
reconstruction of Nguyen’s framework. And Nguyen’s informal analysis has
already done the heavy lifting of rigorously ordering concepts inherited from
unsystematic public discourse, such as “echo chambers” and “filter bubbles”
(introduced, resp., in [23] and [16]), whose previously heterogeneous use had
prevented consensus among researchers (see [21]). Rather, the litmus test for
a “game of like” model would be a contribution to the critical re-evaluation
of empirical data assessed from a decision-theoretic standpoint; and a
suggestion of empirical investigations that a decision-theoretic standpoint
would have neglected. To conclude, we want to suggest that, as incomplete as
it is, our “game of like” model already achieves that.

As for critical re-evaluation, consider the widely-publicized study by
Pennycook et al. [17], in which the intervention condition proceeds from the
auxiliary hypothesis that accuracy competes for attention with social
incentives.666“In the control condition of each experiment, participants were
shown 24 news headlines (balanced on veracity and partisanship) and asked how
likely they would be to share each headline on Facebook. In the treatment
condition, participants were asked to rate the accuracy of a single non-
partisan news headline at the outset of the study (ostensibly as part of a
pretest for stimuli for another study). They then went on to complete the same
sharing intentions task as in the control condition, but with the concept of
accuracy more likely to be salient in their minds.” [17, p. 591] From a
strategic standpoint, the authors’ other auxiliary hypothesis—that “people do
care more about accuracy than other content dimensions” (p. 591)—could
characterize common knowledge of one dimension of users’ preferences. If it
does, having “the concept of accuracy more […] salient in [one’s] mind” (ibid)
could prime engagement-based expectations, rather than shutting them down; in
a game-to-decision translation, a Bayesian decision-maker would then
anticipate a better prospect of eliciting other users’ reactions conditional
on being perceived as accurate (compared to conditional on being perceived as
inaccurate). Compare this with the intervention condition from the more recent
study by Ren et al. [19], which socially incentivized both accuracy and
engagement.777“In each incentive condition, we told participants that they
would be entered into a lottery for a $50 prize, and we manipulated how they
would earn tickets to increase their odds of winning the prize. In the
Accuracy condition, we told participants that they would earn a ticket if the
post they shared was validated to be true by a professional fact-checker. In
the Like condition, we told participants that they would earn a ticket for
each “like” they received from others. In the Comment condition, we told
participants that they would earn a ticket for each comment they received from
others. In the Control condition, we did not provide additional incentives.
[19, 104421:4] As for the design of new studies, consider the question of
whether differences in intervention conditions between [17] and [19] translate
into differences in reasoning about other users’ strategies is an interesting
question. A positive answer would partition “accuracy nudges” into two classes
(engagement-based and non-engagement-based). A negative answer would
invalidate the auxiliary hypothesis that accuracy competes with the social
dimension. The connection we drew with Nagel’s work on guessing games suggest
an empirical approach to answering this question, with following the
methodology of [4], which established neural correlates of lower- and higher-
order “Keynes degree” reasoning in guessing games.

## References

  * [1]
  * [2] Antoni Bosch-Domenech, Jose G. Montalvo, Rosemarie Nagel & Albert Satorra (2002): _One, two, (three), infinity,…: Newspaper and lab beauty-contest experiments_.  American Economic Review 92(5), pp. 1687–1701, 10.1257/000282802762024737.
  * [3] Nick Chater & Mike Oaksford (2008): _The probabilistic mind: Prospects for Bayesian cognitive science_.  Oxford University Press, USA, New York, NY, 10.1093/acprof:oso/9780199216093.001.0001.
  * [4] Giorgio Coricelli & Rosemarie Nagel (2009): _Neural correlates of depth of strategic reasoning in medial prefrontal cortex_.  Proceedings of the National Academy of Sciences 106(23), pp. 9163–9168, 10.1073/pnas.0807721106.
  * [5] Martin Gardner (1970): _The fantastic combinations of John Conway’s new solitaire game ”life”_.  Scientific American 223(4), pp. 120–123, 10.1038/scientificamerican1070-120.
  * [6] Alvin Goldman (1999): _Knowledge in a Social World_.  Oxford University Press, Oxford, 10.1093/0198238207.001.0001.
  * [7] John C. Harsanyi (1982): _Comment—Subjective probability and the theory of games: Comments on Kadane and Larkey’s paper_.  Management Science 28(2), pp. 120–124, 10.1287/mnsc.28.2.120.
  * [8] John C. Harsanyi (1982): _Rejoinder to professors Kadane and Larkey_.  Management Science 28(2), pp. 124–125, 10.1287/mnsc.28.2.124a.
  * [9] Joseph B. Kadane & Patrick D. Larkey (1982): _Subjective probability and the theory of games_.  Management Science 28(2), pp. 113–120, 10.1287/mnsc.28.2.113.
  * [10] John M. Keynes (1978): _The General Theory of Employment, Interest and Money (1936)_.  In Johnson E. & Moggridge D., editors: The Collected Writings of John Maynard Keynes, 7, Royal Economic Society, 10.1017/UPO9781139524278.
  * [11] Hause Lin, Gordon Pennycook & David G. Rand (2023): _Thinking more or thinking differently? Using drift-diffusion modeling to illuminate why accuracy prompts decrease misinformation sharing_.  Cognition 230, 105312, 10.1016/j.cognition.2022.105312.
  * [12] Rosemarie Nagel (1995): _Unraveling in guessing games: An experimental study_.  The American Economic Review 85(5), pp. 1313–1326.  Available at http://www.jstor.org/stable/2950991.
  * [13] C. Thi Nguyen (2020): _Echo chambers and epistemic bubbles_.  Episteme 17(2), pp. 141–161, 10.1017/epi.2018.32.
  * [14] Mike Oaksford & Nick Chater (1994): _A rational analysis of the selection task as optimal data selection_.  Psychological Review 101(4), pp. 608–631, 10.1037/0033-295X.101.4.608.
  * [15] Cailin O’Connor & James O. Weatherall (2019): _The Misinformation Age: How False Beliefs Spread_.  Yale University Press, New Haven, CT, 10.2307/j.ctv8jp0hk.
  * [16] Eli Pariser (2011): _The Filter Bubble: What the Internet is Hiding From You_.  Penguin UK, London.
  * [17] Gordon Pennycook, Ziv Epstein, Mohsen Mosleh, Antonio A. Arechar, Dean Eckles & David G. Rand (2021): _Shifting attention to accuracy can reduce misinformation online_.  Nature 592(7855), pp. 590–595, 10.1038/s41586-021-03344-2.
  * [18] Steve Rathje, Jon Roozenbeek, Jay J. Van Bavel & Sander van der Linden (2023): _Accuracy and social motivations shape judgments of (mis)information_.  Nature Human Behaviour, pp. 1–12, 10.1038/s41562-023-01540-w.
  * [19] Zhiying (Bella) Ren, Eugen Dimant & Maurice Schweitzer (2023): _Beyond belief: How social engagement motives influence the spread of conspiracy theories_.  Journal of Experimental Social Psychology 104, 104421, 10.1016/j.jesp.2022.104421.
  * [20] Jon Roozenbeek, Alexandra L. J. Freeman & Sander van der Linden (2021): _How accurate are accuracy-nudge interventions? A preregistered direct replication of Pennycook et al. (2020)_.  Psychological Science 32(7), pp. 1169–1178, 10.1177/09567976211024535.
  * [21] Amy Ross Arguedas, Craig T. Robertson, Richard Fletcher & Rasmus K. Nielsen (2022): _Echo chambers, filter bubbles, and polarisation: a literature review_.  University of Oxford, Reuters Institute for the Study of Journalism.  Available at https://reutersinstitute.politics.ox.ac.uk/echo-chambers-filter-bubbles-and-polarisation-literature-review.
  * [22] Thomas W. Simpson (2012): _Evaluating Google as an epistemic tool_.  Metaphilosophy 43(4), pp. 426–445, 10.1111/j.1467-9973.2012.01759.x.
  * [23] Cass R. Sunstein (2001): _Echo chambers: Bush v. Gore, Impeachment, and Beyond_.  Princeton University Press Princeton, NJ.
  * [24] Soroush Vosoughi, Deb Roy & Sinan Aral (2018): _The spread of true and false news online_.  Science 359(6380), pp. 1146–1151, 10.1126/science.aap9559.


# Personalized Prediction of Recurrent Stress Events Using Self-Supervised
Learning on Multimodal Time-Series Data

Tanvir Islam  Peter Washington

###### Abstract

Chronic stress can significantly affect physical and mental health. The advent
of wearable technology allows for the tracking of physiological signals,
potentially leading to innovative stress prediction and intervention methods.
However, challenges such as label scarcity and data heterogeneity render
stress prediction difficult in practice. To counter these issues, we have
developed a multimodal personalized stress prediction system using wearable
biosignal data. We employ self-supervised learning (SSL) to pre-train the
models on each subject’s data, allowing the models to learn the baseline
dynamics of the participant’s biosignals prior to fine-tuning the stress
prediction task. We test our model on the Wearable Stress and Affect Detection
(WESAD) dataset, demonstrating that our SSL models outperform non-SSL models
while utilizing less than 5% of the annotations. These results suggest that
our approach can personalize stress prediction to each user with minimal
annotations. This paradigm has the potential to enable personalized prediction
of a variety of recurring health events using complex multimodal data streams.

Machine Learning, ICML

##  1 Introduction

Chronic stress can have profound detrimental effects on health and well-being.
Research has shown that prolonged exposure to stress hormones can lead to
increased risk of mental health disorders such as anxiety and depression
(McEwen, 2008) and can contribute to the development of cardiovascular
diseases, including hypertension and heart disease (Chrousos & Gold, 1992).
Furthermore, chronic stress has been associated with dysregulation of the
immune system, leading to impaired immune function and increased
susceptibility to infections and autoimmune disorders (Dhabhar, 2014).

Biosignal-based stress detection methods are of increasing interest to the
digital health community due to their potential to recognize stress levels in
real-time. Commonly used biosignals for stress prediction include
electrocardiograms (ECG) (Rabbani & Khan, 2022), galvanic skin response (GSR)
(Lundberg et al., 1994), electromyograms (EMG) (Wijsman et al., 2010;
Crescentini et al., 2016; Taelman et al., 2009), skin temperature (ST) (Lorig,
2016), skin conductivity (Munla et al., 2015), and respiratory rate (Yu et
al., 2016).

A fundamental challenge for the prediction of stress from multimodal
biosignals is the engineering of appropriate features. Traditional machine
learning approaches face a significant challenge in need for manual feature
generation, and the feature representation approaches vary depending on the
physiological signal (Li & Liu, 2020; Washington et al., 2022). These manually
created features, although widely used (Munla et al., 2015; Gjoreski et al.,
2016), often do not lead to optimal performance outcomes for subjective
prediction targets such as stress due to the inherent subjectivity,
complexity, and heterogeneity of the data. Deep neural networks (DNNs) have
revolutionized machine learning (LeCun et al., 2015) by learning to extract
complex data patterns across various fields (Amodei et al., 2016; He et al.,
2015), including in the medical sciences (Gulshan et al., 2016), without the
need for manual feature extraction (Li & Liu, 2020; Hannun et al., 2019).

Another major challenge in machine learning with multimodal biosignal data is
obtaining datasets with high-quality labels (Washington & Wall, 2023;
Washington, 2023), a process which is notably costly and where the clinical
ground truth is often difficult to define (Kalantarian et al., 2019;
Washington et al., 2020). Annotation of stress events is laborious for the end
user, leading to relatively sparse labels in most, if not all, datasets
containing biosignals. The sparse labels, coupled with the user-dependent
nature of the physiological stress response, lead to immense difficulty in
training a generalizable stress prediction model.

To address these two challenges, which have traditionally led to a subpar
performance in the use of machine learning prediction of subjective
psychiatric outcomes such as stress, we suggest a personalized self-supervised
learning (SSL) approach to stress prediction. By training a separate model per
user (personalization) and using SSL to learn the baseline dynamics of each
user’s biosignals, we are able to learn using relatively few annotations from
each user, even for annotations which are inherently subjective such as
stress. We capitalize on multiple sensor modalities, performing stress
recognition through the integration of diverse information (Walambe et al.,
2021). We use electrodermal activity (EDA), electrocardiogram (ECG),
electromyography (EMG), respiration (RESP), core body temperature (TEMP), and
three-axis acceleration (ACC) to train our stress prediction models.

Our contributions to the machine learning for the healthcare field are as
follows:

  * •

We evaluate stress prediction models trained using multiple biosignal
modalities.

  * •

We explore the personalization of these multimodal stress prediction models,
enabled through SSL procedures which learn the baseline temporal dynamics of
each user’s biosignals.

  * •

We compare multimodal personalized models with and without SSL pre-training to
quantify the impact of personalized multimodal SSL.

##  2 Related Works

Stress prediction from biosignals is a rich and expanding field. Established
machine learning approaches have yielded preliminary successes while leaving
room for improvement. Fernández et al. introduced a non-invasive, radar-based
stress detection method that primarily uses respiratory patterns and applied
Recurrence Quantification Analysis (RQA) to achieve 94.4% accuracy, offering
broader applicability than heartbeat-based methods, particularly for
individuals with a higher body mass index (Fernández & Anishchenko, 2018).
Ghaderi et al. employed machine learning algorithms to process various
biological signals for accurate stress level classification, emphasizing the
role of respiration as a crucial sensor and suggesting its potential
applications for tailoring stress management treatments (Ghaderi et al.,
2015). Karthikeyan et al. studied the correlation between stress levels and
EMG measurements of muscle tension (Karthikeyan et al., 2012). Other lines of
research emphasize the utility of ECG data in stress measurement, including a
novel ECG-based mono-fuzzy measure to assess stress levels (Ahn et al., 2019)
as well as the application of person-specific and person-independent stress
prediction methods (Walocha et al., 2022).

SSL has only recently been applied to biosignals-based stress prediction.
Robert et al. proposed a stress prediction system based on contrastive
learning, utilizing modified EDA signals to classify stress and non-stress
scenarios (Matton et al., 2022). Contrastive learning techniques such as
SimCLR (Cheng et al., 2020) and BOYL (Grill et al., 2020) heavily rely on data
augmentation. However, this approach has limitations, as determining the most
beneficial augmentations is challenging (Zhang & Ma, 2022). By contrast, our
study explores a different approach by excluding data augmentation and instead
focusing on self-supervised pre-training using the raw data.

Multimodal machine learning techniques are required to handle the multiple
concurrent biosignals which are recorded by consumer and research wearables.
Bobade et al. detected stress levels in individuals by analyzing various bio-
signals such as acceleration, ECG, blood volume pulse, body temperature,
respiration, EMG, and EDA. Their experimental results included accuracy levels
of up to 84.32% for three-way classification (amusement vs. baseline vs.
stress) and up to 95.21% accuracy for binary classification (stress vs. non-
stress), outperforming previous work in the field (Bobade & Vani, 2020).
Aigrain et al. developed a methodology for analyzing multimodal stress
detection results by considering multiple assessments of stress (Aigrain et
al., 2016). Data from 25 subjects in a stressful situation were collected,
along with three different assessments: self-assessment, assessments from
external observers, and assessments from a physiology expert. The study found
that a combination of behavioral and physiological features, such as body
movement, blood volume pulse, and heart rate, provided valuable information
for classifying stress across the three assessments (Aigrain et al., 2016).
Bara et al. introduced a deep learning-based approach for multimodal stress
detection, utilizing the MuSE dataset (Jaiswal et al., 2019) to evaluate
various configurations and modular architectures. The results demonstrated the
potential of deep learning methods in capturing affective representations
related to stress, paving the way for further investigations and applications
in affective computing (Bara et al., 2020).

In contrast to these prior works, we combine the fields of multimodal machine
learning and SSL to achieve optimal performance with a minimal number of
training labels. Multimodal models traditionally require more labels to
accommodate the expanded feature space, but we hypothesize that SSL can
drastically reduce this need. Furthermore, we combine these ideas with the
idea of model personalization, where a separate model is trained per
individual.

##  3 Methodology

Figure 1: Personalized stress prediction framework involves segmenting the raw
signal into overlapping data points and utilizing a self-supervised pre-
training forecasting model (right model), which is then fine-tuned for stress
prediction. We compare this model to a supervised learning model (left model)
without self-supervised pre-training to assess its effectiveness.

We aim to develop a model that can accurately predict stress by learning
representations of six concurrent biosignals: EDA, ECG, EMG, Temp, Resp, and
ACC. Our model is developed in two stages: (1) pre-training to predict a
pretext task $T_{p}$ and (2) fine-tuning to the downstream task $T_{d}$, which
in this case is stress prediction. In the pretext task ($T_{p}$), the goal is
to learn robust and generalized features from unlabeled biosignals through a
self-supervised process which learns meaningful representations that can
capture relevant patterns and information from each user’s individual
biosignal dynamics. Once the representations are learned from the pretext
task, the second stage ($T_{d}$) focuses on predicting stress using the
original biosignal data along with human-annotated stress labels ($y_{i}$). We
compare the fine-tuned SSL-based model against a purely supervised model. For
fully supervised training, we use the same architecture as $T_{d}$. Figure 1
illustrates the proposed self-supervised solution, depicting the two-stage
process of learning representations and predicting stress.

###  3.1 Dataset

We use a publicly available dataset called WESAD (Schmidt et al., 2018),
consisting of data collected using the Empatica E4 wrist-worn device and
RespiBAN device worn on the chest. WESAD contains six biosignals per
participant: ECG, EDA, EMG, RESP, TEMP, and ACC. In addition to the
physiological data collected, participants periodically completed
questionnaires assessing their emotional state during the data collection
session. Our models are trained exclusively on the RespiBAN device data.

During the data collection process, information related to three distinct
affective states, namely stress, amusement, and relaxation, were gathered from
a group of 15 participants. In order to assess each participant’s anxiety
levels, they were presented with six questions derived from the State-Trait
Anxiety Inventory (STAI). These questions required the participants to
indicate their responses on a four-point Likert scale. The six STAI questions
covered six distinct affective states: feeling at ease, nervous, jittery,
relaxed, worried, and pleasant. By asking these STAI questions, the developers
of WESAD aimed to gauge each participant’s anxiety levels and to capture their
subjective emotional states.

Our models were trained using these six biosignals from the RespiBAN device
for baseline condition, thus facilitating a multimodal learning training
process.

###  3.2 Label Representation

The WESAD dataset includes participants’ responses to six questions from the
STAI, which were rated on a four-point Likert scale (ranging from 1 to 4). In
this paper, we address the task of stress detection by proposing a label
quantification approach. Rather than treating stress detection as a discrete
classification problem with predefined stress levels, we transform it into a
regression problem by assigning continuous values to the stress labels.
Specifically, we convert the stress labels of 1, 2, 3, and 4 into quantified
values of 0.25, 0.5, 0.75, and 1, respectively. To ensure a quantified and
uniformly distributed representation of labels, we perform a conversion based
on the evenly spaced four-point Likert scale. This label quantification
technique allows us to capture the underlying intensity or severity of stress,
enabling more fine-grained analysis and prediction. By leveraging DNN over raw
biosignals, we can now model and predict stress levels with greater precision
and sensitivity, contributing to more accurate and nuanced stress prediction
systems. Our experimental results demonstrate the effectiveness of label
quantification in improving the performance and interpretability of stress
prediction models, highlighting its potential for advancing the field of
stress analysis and management.

###  3.3 Self-supervised Learning of Biosignal Representations

Self-supervised pre-training involves gaining a comprehensive understanding of
data without relying on labeled information, known as the “pretext task”. By
training a model with a robust representation, it becomes more feasible to
transfer to a specific task. We trained separate models for each subject using
only their respective data, allowing the models to learn the baseline dynamics
of each individual.

Table 1: Architectural details of the pre-trained network (1D CNN). Module | Layer Details |  Activation |  Shape
---|---|---|---
Input | - | - |  $7000\times 1$
Layers |  Conv1D, four $1\times 40$ kernels |  Leaky-ReLU |  $7000\times 40$
Conv1D, two $1\times 30$ kernels |  Leaky ReLU |  $7000\times 30$
Conv1D, four $1\times 18$ kernels |  Leaky-ReLU |  $7000\times 18$
Conv1D, two $1\times 30$ kernels |  Leaky-ReLU |  $7000\times 30$
Dense |  Leaky-ReLU |  $1\times 70$
Dense |  Leaky-ReLU |  $1\times 30$
Output | - |  Linear |  40

The model is pre-trained by dividing each of the six signals into fixed
windows of $7000$ ($10$ seconds) dimensions, resulting in $7910$ data points
with a 100-point overlap using the forecasting method. Each training data
point consists of a $7000$-dimensional vector, with the target output being
the subsequent $40$ data points. We set the window size to $10$ seconds
because we aim to predict stress for $10$ seconds. A 1D CNN is utilized for
the pre-training process. The 1D CNN architecture of this model pre-training
is presented in Table 1.

Let the original signal sequence is $S={S_{1},S_{2},...,S_{n}}$. We then
create a series of training data points $X$ and corresponding targets $Y$ as
follows: For each $i$ from $1$ to $D$ (where $D$ is the total number of data
points), we have $X_{i}={S_{(i-1)*O+1},...,S_{(i-1)*O+W}}$ and the
corresponding target $Y_{i}={S_{(i-1)*O+W+1},...,S_{(i-1)*O+W+P}}$, where $S$
is the original signal, $D$ is the total number of data points, $W$ is the
window size (7000 in our case), $P$ is the number of points to predict (40 in
our case), $O$ is the overlap (100 in our case), $X_{i}$ is the $i$-th data
point (an array of $W$ elements from the original signal), $Y_{i}$ is the
corresponding target (an array of $P$ elements from the signal, immediately
following the elements in $X_{i}$).

Each $X_{i}$ is a window of the original signal, and each $Y_{i}$ is the
sequence of points immediately following the corresponding $X_{i}$ in the
signal.

With this method, we have learned baseline representations of all biosignal
modalities for each user: $R_{EDA}$, $R_{ECG}$, $R_{EMG}$, $R_{Temp}$,
$R_{ACC}$, $R_{Resp}$

###  3.4 Stress Prediction Task

Figure 2: RMSE between a model fine-tuned after SSL and a model trained purely
with supervised training for all subjects and all annotation types.

In the second phase of learning ($T_{d}$), the model uses the initial
biosignals and responses provided by participants ($y_{i}$). In this phase,
the framework utilizes the fixed convolution layers from the first network to
extract features that will help to predict stress. The fully-connected layers
at the end of the network are subsequently trained in a supervised manner to
predict stress using the extracted features.

Table 2: Architectural details of the fine-tuned network (1D CNN). Module |  Layer Details |  Activation |  Shape
---|---|---|---
Input | - | - |  $7000\times 1$
Frozen Layers |  Conv1D, four $1\times 40$ kernels |  Leaky ReLU |  $\small 7000\times 40$
|  Conv1D, two $1\times 30$ kernels |  Leaky ReLU |  $7000\times 30$
|  Conv1D, four $1\times 18$ kernels |  Leaky ReLU |  $7000\times 18$
|  Conv1D, two $1\times 30$ kernels |  Leaky ReLU |  $7000\times 30$
Task-Specific Layers | Dense |  Leaky ReLU |  $1\times 50$
| Dense |  Leaky ReLU |  $1\times 30$
| Dense |  Leaky ReLU |  $1\times 30$
Output | - |  Linear |  1

In order to perform the stress prediction task ($T_{d}$) using multiple
modalities, the representations of the biosignals, namely $R_{EDA}$,
$R_{ECG}$, $R_{EMG}$, $R_{Temp}$, $R_{ACC}$, and $R_{Resp}$, are fused
together into a single multimodal representation $R$. This late-stage fusion
process combines the information from each modality to create a comprehensive
representation that captures the relevant features for predicting stress. To
accomplish $T_{d}$ with representation $R$, a network: $\rho=w(R,\theta)$ is
trained where $\rho$ is the output prediction vector and $\theta$ is the set
of trainable parameters. Finally, we calculate the optimal value of $\theta$
by minimizing mean squared error loss:

$\text{L}=\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}$

where $n$ represents the number of data points, $y_{i}$ represents the
observed values, and $\hat{y}_{i}$ represents the predicted values.

We describe the details of the network that is used for the downstream stress
prediction task ($T_{d}$) in Table 2.

###  3.5 Experimental Procedures

We examine the performance of the models pre-trained with SSL in comparison to
models trained solely through supervised training without pre-training. To
conduct this comparison, we utilize a dataset consisting of $7,000$ data
points from six biosignals, with each data point representing a $10$-second
window and all six signal modalities. These data points are used to train both
types of models (SSL pre-training followed by supervised fine-tuning and
purely supervised training). To test the models, we hold out the last set of
$910$ data points with $10$-second windows. Both models are evaluated using
the same test set for each subject.

Figure 3: Comparison of the performance between SSL pre-training and solely
supervised training methods for a demonstrative user (Subject 2). Similar
trends were observed for all subjects.

##  4 Results and Evaluation

We evaluate each model using RMSE (Root Mean Squared Error). We show the RMSE
scores for all subjects across various questions in Figure 2. The scores were
obtained by training 10 sets of random data points three times, each time
using different random data samples, and calculating the average RMSE score.
We observe that self-supervised pre-training achieves significantly lower RMSE
compared to models trained solely through supervised learning. This indicates
that SSL, which involves learning representations from unlabeled data, results
in improved performance and reduced prediction errors compared to models
trained only with labeled data. The figure demonstrates the superiority of SSL
pre-training in terms of RMSE scores.

We also compare the models pre-trained with SSL and the purely supervised
model using a set of different labeled data points. Figure 3 shows the
comparison between these two models for Subject 2 for all questions. Figure 3
demonstrates that even when sampling only 5 random data points, the SSL pre-
trained model repeatedly outperforms purely supervised training for all users
and all outcome measures.

##  5 Discussion

We present an analysis of a personalized multimodal learning framework,
focusing on the advantages of self-supervised pre-training in comparison to a
fully-supervised training paradigm. The results of this study emphasize the
superiority of SSL over purely supervised training methods within the context
of personalized learning. We observe this benefit with only a few labeled
examples, demonstrating the personalized SSL can enable learning of complex
and subjective outcomes such as stress using relatively small datasets.

With the aid of SSL, the personalized models can leverage both labeled and
unlabeled data, making more efficient use of available information. This is
especially valuable in real-world situations where limited labeled data or
data of varying quality is present such as in the personalization of machine
learning-powered healthcare applications (Daniels et al., 2018a, b; Kline et
al., 2019; Voss et al., 2019; Washington et al., 2017), where ample data are
recorded per individual but each label is burdensome for the end-user to
provide. By contrast, purely supervised models display high variability in
their performance as indicated by a significant fluctuation in RMSE values
across different training runs or data samples. This variability can be
problematic for personalized applications, where consistent and dependable
outcomes are crucial.

There are some limitations of this study which should be addressed in follow-
up work. The WESAD dataset was created in a highly controlled environment.
Future research endeavors should involve comprehensive and unconstrained data
collection setups to test the system using real-time, “in-the-wild” data
streams. Such experimental procedures will provide valuable insights into
multimodal personalization in real-world scenarios.

##  6 Conclusion

We introduce a novel approach for enhancing deep learning models to predict
stress using multimodal biosignals. Our technique empowers deep learning
models to adapt to an individual’s unique baseline temporal dynamics, enabling
more precise and personalized predictions of stress with significantly fewer
annotations required. By leveraging multi-modal biosignals, our method opens
up new possibilities for understanding and addressing stress-related
challenges in various contexts, including healthcare, workplace environments,
and personalized mental health interventions. The potential impact of our
findings is substantial, as the personalized learning technique presented here
significantly reduces the need for extensive human annotation typically
associated with deep learning models for healthcare.

## Acknowledgements

The technical support and advanced computing resources from University of
Hawaii Information Technology Services – Cyberinfrastructure, funded in part
by the National Science Foundation CC* awards # 2201428 and # 2232862 are
gratefully acknowledged.

## References

  * Ahn et al. (2019) Ahn, J., Ku, Y., and Kim, H. C.  A novel wearable eeg and ecg recording system for stress assessment.  _Sensors_ , 19:1991, 04 2019.  doi: 10.3390/s19091991.
  * Aigrain et al. (2016) Aigrain, J., Spodenkiewicz, M., Dubuisson, S., Detyniecki, M., Cohen, D., and Chetouani, M.  Multimodal stress detection from multiple assessments.  _IEEE Transactions on Affective Computing_ , 9(4):491–506, 2016.
  * Amodei et al. (2016) Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Cheng, Q., Chen, G., Chen, J., Chen, J., Chen, Z., Chrzanowski, M., Coates, A., Diamos, G., Ding, K., Du, N., Elsen, E., Engel, J., Fang, W., Fan, L., Fougner, C., Gao, L., Gong, C., Hannun, A., Han, T., Johannes, L., Jiang, B., Ju, C., Jun, B., LeGresley, P., Lin, L., Liu, J., Liu, Y., Li, W., Li, X., Ma, D., Narang, S., Ng, A., Ozair, S., Peng, Y., Prenger, R., Qian, S., Quan, Z., Raiman, J., Rao, V., Satheesh, S., Seetapun, D., Sengupta, S., Srinet, K., Sriram, A., Tang, H., Tang, L., Wang, C., Wang, J., Wang, K., Wang, Y., Wang, Z., Wang, Z., Wu, S., Wei, L., Xiao, B., Xie, W., Xie, Y., Yogatama, D., Yuan, B., Zhan, J., and Zhu, Z.  Deep speech 2 : End-to-end speech recognition in english and mandarin.  In Balcan, M. F. and Weinberger, K. Q. (eds.), _Proceedings of The 33rd International Conference on Machine Learning_ , volume 48 of _Proceedings of Machine Learning Research_ , pp. 173–182, New York, New York, USA, 20–22 Jun 2016. PMLR.  URL https://proceedings.mlr.press/v48/amodei16.html.
  * Bara et al. (2020) Bara, C.-P., Papakostas, M., and Mihalcea, R.  A deep learning approach towards multimodal stress detection.  In _AffCon@ AAAI_ , pp. 67–81, 2020.
  * Bobade & Vani (2020) Bobade, P. and Vani, M.  Stress detection with machine learning and deep learning using multimodal physiological data.  In _2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA)_ , pp. 51–57. IEEE, 2020.
  * Cheng et al. (2020) Cheng, J. Y., Goh, H., Dogrusoz, K., Tuzel, O., and Azemi, E.  Subject-aware contrastive learning for biosignals.  _arXiv preprint arXiv:2007.04871_ , 2020.
  * Chrousos & Gold (1992) Chrousos, G. P. and Gold, P. W.  The concepts of stress and stress system disorders: overview of physical and behavioral homeostasis.  _Jama_ , 267(9):1244–1252, 1992.
  * Crescentini et al. (2016) Crescentini, C., Chittaro, L., Capurso, V., Sioni, R., and Fabbro, F.  Psychological and physiological responses to stressful situations in immersive virtual reality: Differences between users who practice mindfulness meditation and controls.  _Computers in Human Behavior_ , 59:304–316, 2016.  ISSN 0747-5632.  doi: https://doi.org/10.1016/j.chb.2016.02.031.  URL https://www.sciencedirect.com/science/article/pii/S0747563216300826.
  * Daniels et al. (2018a) Daniels, J., Haber, N., Voss, C., Schwartz, J., Tamura, S., Fazel, A., Kline, A., Washington, P., Phillips, J., Winograd, T., et al.  Feasibility testing of a wearable behavioral aid for social learning in children with autism.  _Applied clinical informatics_ , 9(01):129–140, 2018a.
  * Daniels et al. (2018b) Daniels, J., Schwartz, J. N., Voss, C., Haber, N., Fazel, A., Kline, A., Washington, P., Feinstein, C., Winograd, T., and Wall, D. P.  Exploratory study examining the at-home feasibility of a wearable tool for social-affective learning in children with autism.  _NPJ digital medicine_ , 1(1):32, 2018b.
  * Dhabhar (2014) Dhabhar, F. S.  Effects of stress on immune function: the good, the bad, and the beautiful.  _Immunologic research_ , 58:193–210, 2014.
  * Fernández & Anishchenko (2018) Fernández, J. R. M. and Anishchenko, L.  Mental stress detection using bioradar respiratory signals.  _Biomedical signal processing and control_ , 43:244–249, 2018.
  * Ghaderi et al. (2015) Ghaderi, A., Frounchi, J., and Farnam, A.  Machine learning-based signal processing using physiological signals for stress detection.  In _2015 22nd Iranian Conference on Biomedical Engineering (ICBME)_ , pp. 93–98. IEEE, 2015.
  * Gjoreski et al. (2016) Gjoreski, M., Gjoreski, H., Lustrek, M., and Gams, M.  Continuous stress detection using a wrist device: in laboratory and real life.  pp. 1185–1193, 09 2016.  doi: 10.1145/2968219.2968306.
  * Grill et al. (2020) Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 33:21271–21284, 2020.
  * Gulshan et al. (2016) Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., Kim, R., Raman, R., Nelson, P. C., Mega, J. L., and Webster, D. R.  Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs.  _JAMA_ , 316(22):2402–2410, 12 2016.  ISSN 0098-7484.  doi: 10.1001/jama.2016.17216.  URL https://doi.org/10.1001/jama.2016.17216.
  * Hannun et al. (2019) Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H., Bourn, C., Turakhia, M. P., and Ng, A. Y.  Publisher correction: Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network.  _Nature Medicine_ , 25(3):530–530, Mar 2019.  ISSN 1546-170X.  doi: 10.1038/s41591-019-0359-9.  URL https://doi.org/10.1038/s41591-019-0359-9.
  * He et al. (2015) He, K., Zhang, X., Ren, S., and Sun, J.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _2015 IEEE International Conference on Computer Vision (ICCV)_ , pp. 1026–1034, 2015.  doi: 10.1109/ICCV.2015.123.
  * Jaiswal et al. (2019) Jaiswal, M., Aldeneh, Z., Bara, C.-P., Luo, Y., Burzo, M., Mihalcea, R., and Mower Provost, E.  Muse-ing on the impact of utterance ordering on crowdsourced emotion annotations.  pp. 7415–7419, 05 2019.  doi: 10.1109/ICASSP.2019.8682793.
  * Kalantarian et al. (2019) Kalantarian, H., Jedoui, K., Washington, P., Tariq, Q., Dunlap, K., Schwartz, J., and Wall, D. P.  Labeling images with facial emotion and the potential for pediatric healthcare.  _Artificial intelligence in medicine_ , 98:77–86, 2019\.
  * Karthikeyan et al. (2012) Karthikeyan, P., Murugappan, M., and Yaacob, S.  Emg signal based human stress level classification using wavelet packet transform.  In Ponnambalam, S. G., Parkkinen, J., and Ramanathan, K. C. (eds.), _Trends in Intelligent Robotics, Automation, and Manufacturing_ , pp. 236–243, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg.  ISBN 978-3-642-35197-6.
  * Kline et al. (2019) Kline, A., Voss, C., Washington, P., Haber, N., Schwartz, H., Tariq, Q., Winograd, T., Feinstein, C., and Wall, D. P.  Superpower glass.  _GetMobile: Mobile Computing and Communications_ , 23(2):35–38, 2019.
  * LeCun et al. (2015) LeCun, Y., Bengio, Y., and Hinton, G.  Deep learning.  _nature_ , 521(7553):436–444, 2015.
  * Li & Liu (2020) Li, R. and Liu, Z.  Stress detection using deep neural networks.  _BMC Medical Informatics and Decision Making_ , 20, 12 2020.  doi: 10.1186/s12911-020-01299-4.
  * Lorig (2016) Lorig, T. S.  _The Respiratory System_ , pp. 244–257.  Cambridge Handbooks in Psychology. Cambridge University Press, 4 edition, 2016.  doi: 10.1017/9781107415782.011.
  * Lundberg et al. (1994) Lundberg, U., Kadefors, R., Melin, B., Palmerud, G., Hassmén, P., Engström, M., and Elfsberg Dohns, I.  Psychophysiological stress and emg activity of the trapezius muscle.  _International Journal of Behavioral Medicine_ , 1(4):354–370, Dec 1994.  ISSN 1532-7558.  doi: 10.1207/s15327558ijbm0104˙5.  URL https://doi.org/10.1207/s15327558ijbm0104_5.
  * Matton et al. (2022) Matton, K., Lewis, R. A., Guttag, J., and Picard, R.  Contrastive learning of electrodermal activity representations for stress detection.  In _NeurIPS 2022 Workshop on Learning from Time Series for Health_ , 2022.  URL https://openreview.net/forum?id=bSC_xo8VQ1b.
  * McEwen (2008) McEwen, B. S.  Central effects of stress hormones in health and disease: Understanding the protective and damaging effects of stress and stress mediators.  _European journal of pharmacology_ , 583(2-3):174–185, 2008.
  * Munla et al. (2015) Munla, N., Khalil, M., Shahin, A., and Mourad, A.  Driver stress level detection using hrv analysis.  In _2015 International Conference on Advances in Biomedical Engineering (ICABME)_ , pp. 61–64, 2015.  doi: 10.1109/ICABME.2015.7323251.
  * Rabbani & Khan (2022) Rabbani, S. and Khan, N.  Contrastive self-supervised learning for stress detection from ecg data.  _Bioengineering_ , 9(8):374, 2022.
  * Schmidt et al. (2018) Schmidt, P., Reiss, A., Duerichen, R., Marberger, C., and Van Laerhoven, K.  Introducing wesad, a multimodal dataset for wearable stress and affect detection.  In _Proceedings of the 20th ACM international conference on multimodal interaction_ , pp. 400–408, 2018.
  * Taelman et al. (2009) Taelman, J., Vandeput, S., Spaepen, A., and Huffel, S.  _Influence of Mental Stress on Heart Rate and Heart Rate Variability_ , volume 22, pp. 1366–1369.  01 2009.  ISBN 978-3-540-89207-6.  doi: 10.1007/978-3-540-89208-3˙324.
  * Voss et al. (2019) Voss, C., Schwartz, J., Daniels, J., Kline, A., Haber, N., Washington, P., Tariq, Q., Robinson, T. N., Desai, M., Phillips, J. M., et al.  Effect of wearable digital intervention for improving socialization in children with autism spectrum disorder: a randomized clinical trial.  _JAMA pediatrics_ , 173(5):446–454, 2019.
  * Walambe et al. (2021) Walambe, R., Nayak, P., Bhardwaj, A., and Kotecha, K.  Employing multimodal machine learning for stress detection.  _Journal of Healthcare Engineering_ , 2021:9356452, Oct 2021\.  ISSN 2040-2295.  doi: 10.1155/2021/9356452.  URL https://doi.org/10.1155/2021/9356452.
  * Walocha et al. (2022) Walocha, F., Drewitz, U., and Ihme, K.  Activity and stress estimation based on openpose and electrocardiogram for user-focused level-4-vehicles.  _IEEE Transactions on Human-Machine Systems_ , 52(4):538–546, 2022.  doi: 10.1109/THMS.2022.3155375.
  * Washington (2023) Washington, P.  Crowdsourcing in precision healthcare: Short review.  _arXiv preprint arXiv:2303.03578_ , 2023.
  * Washington & Wall (2023) Washington, P. and Wall, D. P.  A review of and roadmap for data science and machine learning for the neuropsychiatric phenotype of autism.  _Annual Review of Biomedical Data Science_ , 6, 2023.
  * Washington et al. (2017) Washington, P., Voss, C., Kline, A., Haber, N., Daniels, J., Fazel, A., De, T., Feinstein, C., Winograd, T., and Wall, D.  Superpowerglass: a wearable aid for the at-home therapy of children with autism.  _Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies_ , 1(3):1–22, 2017.
  * Washington et al. (2020) Washington, P., Park, N., Srivastava, P., Voss, C., Kline, A., Varma, M., Tariq, Q., Kalantarian, H., Schwartz, J., Patnaik, R., et al.  Data-driven diagnostics and the potential of mobile artificial intelligence for digital therapeutic phenotyping in computational psychiatry.  _Biological Psychiatry: Cognitive Neuroscience and Neuroimaging_ , 5(8):759–769, 2020.
  * Washington et al. (2022) Washington, P., Mutlu, C. O., Kline, A., Paskov, K., Stockham, N. T., Chrisman, B., Deveau, N., Surhabi, M., Haber, N., and Wall, D. P.  Challenges and opportunities for machine learning classification of behavior and mental state from images.  _arXiv preprint arXiv:2201.11197_ , 2022.
  * Wijsman et al. (2010) Wijsman, J., Grundlehner, B., Penders, J., and Hermens, H.  Trapezius muscle emg as predictor of mental stress.  volume 12, pp. 155–163, 10 2010.  doi: 10.1145/2485984.2485987.
  * Yu et al. (2016) Yu, Y. J., Yang, Z., Oh, B.-S., Yeo, Y. K., Liu, Q., Huang, G.-B., and Lin, Z.  Investigation on driver stress utilizing ecg signals with on-board navigation systems in use.  In _2016 14th International Conference on Control, Automation, Robotics and Vision (ICARCV)_ , pp. 1–6, 2016.  doi: 10.1109/ICARCV.2016.7838780.
  * Zhang & Ma (2022) Zhang, J. and Ma, K.  Rethinking the augmentation module in contrastive learning: Learning hierarchical augmentation invariance with expanded views.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 16650–16659, 2022.