# From Optimizing Engagement to Measuring Value

Smitha Milli111Work done while the author was an intern at Twitter.
UC Berkeley
<EMAIL_ADDRESS>Luca Belli
Twitter
<EMAIL_ADDRESS>Moritz Hardt222MH is a paid consultant at Twitter. Work
performed while consulting for Twitter.
UC Berkeley
<EMAIL_ADDRESS>
###### Abstract

Most recommendation engines today are based on predicting user engagement,
e.g. predicting whether a user will click on an item or not. However, there is
potentially a large gap between engagement signals and a desired notion of
_value_ that is worth optimizing for. We use the framework of measurement
theory to (a) confront the designer with a normative question about what the
designer values, (b) provide a general latent variable model approach that can
be used to operationalize the target construct and directly optimize for it,
and (c) guide the designer in evaluating and revising their
operationalization. We implement our approach on the Twitter platform on
millions of users. In line with established approaches to assessing the
_validity_ of measurements, we perform a qualitative evaluation of how well
our model captures a desired notion of “value”.

##  1 Introduction

Most recommendation engines today are based on predicting user engagement,
e.g. predicting whether a user will click an item or not. However, there is
potentially a large gap between engagement signals and a desired notion of
_value_ that is worth optimizing for (Ekstrand and Willemsen, 2016). Just
because a user engages with an item doesn’t mean they value it. A user might
reply to an item because they are angry about it, or click an item in order to
gain more information about it (Wen et al., 2019), or watch addictive videos
out of temptation.

It is clear that engagements provide some signal for “value”, but are not
equivalent to it. Further, different types of engagement may provide differing
levels of evidence for value. For example, if a user explicitly likes an item,
we are more likely to believe that they value it, compared to if they had
merely clicked on it. Ideally, we want the objective for our recommender
system to take engagement signals into account, but only insofar as they
relate to a desired notion of “value”. However, directly specifying such an
objective is a non-trivial problem. Exactly how much should we rely on likes
versus clicks versus shares and so on? How do we evaluate whether our designed
objective captures our intended notion of “value”?

###  1.1 Our contributions

We make three primary contributions.

1\. We propose measurement theory as a principled approach to aggregating
engagement signals into an objective function that captures a desired notion
of “value”. The resulting objective function can be optimized from data,
serving as a plug-in replacement for the ad-hoc objectives typically used in
engagement optimization frameworks.

2\. Our approach is based on the creation of a latent variable model that
relates value to various observed engagement signals. We devise a new
identification strategy for the latent variable model tailored to the intended
use case of online recommendation systems. Our identification strategy needs
only a single robust engagement signal for which we know the conditional
probability of value given the signal.

3\. We implemented our approach on the Twitter platform on millions of users.
In line with an established validity framework for measurement theory, we
conduct a qualitative analysis of how well our model captures “value”.

###  1.2 Measurement theory and latent variable models

The framework of _measurement theory_ (Hand, 2004; Jackman, 2009) is widely
used in the social sciences as a guide to measuring _unobservable theoretical
constructs_ like “quality of life”, “political ideology”, or “socio-economic
status”. Under the measurement approach, theoretical constructs are
operationalized as latent variables, which are related to observable data
through a latent variable model (LVM). ‘

Similarly, we treat the “value” of a recommendation as a theoretical
construct, which we operationalize as a (binary) latent variable $V$. We
represent the LVM as a a _Bayesian network_ (Pearl, 2009) that contains $V$ as
well as each of the possible types of user engagements (clicks, shares, etc).
The structure of the Bayesian network allows us to specify conditional
independences between variables, enabling us to capture dependencies like e.g.
needing to click an item before replying to it.

Under the measurement approach, the ideal objective becomes clear:
$\mathbb{P}(V=1\mid\mathtt{Behaviors})$ \- the probability the user values the
item given their engagements with it. Such an objective uses all engagement
signals, but only insofar provide evidence of Value $V$. If we can identify
$\mathbb{P}(V=1\mid\mathtt{Behaviors})$, then it can be used as a drop-in
replacement for any objective that scores items based on engagement signals.

Our key insight is that we can identify $\mathbb{P}(V\mid\mathtt{Behaviors})$
— the probability of Value given _all_ behaviors — through the use of a single
_anchor variable_ $A$ for which we know $\mathbb{P}(V=1\mid A=1)$. The anchor
variable, together with the structure of the Bayesian network, is what gives
“value” its meaning. Through the choice of the anchor variable and the
structure of the Bayesian network, the designer has the flexibility to give
“value” subtly different meanings.

Recommendation engines have natural candidates for anchor variables: strong,
explicit feedback from the user. For example, strong negative feedback could
include downvoting or reporting a content item, or blocking another user.
Strong positive feedback could be explicitly liking or upvoting an item. For
negative feedback, we make the assumption that $\mathbb{P}(V=1\mid
A=1)=\epsilon$ for $\epsilon\approx 0$, while for positive feedback we make
the assumption that $\mathbb{P}(V=1\mid A=1)=1-\epsilon$.

###  1.3 A case study on the Twitter platform

We implemented our approach on the Twitter platform on millions of users. On
Twitter, there are numerous user behaviors: clicks, favorites, retweets,
replies, and many more. It would be difficult to directly specify an objective
that properly trades-off all these behaviors. Instead, we identify a natural
anchor variable. On Twitter, users can give explicit feedback on tweets by
clicking “See less often” (SLO) on them. We use SLO as our anchor and assume
that the user does not value tweets they click “See less often” on. After
specifying the anchor variable and the Bayesian network, we are able to learn
$\mathbb{P}(V\mid\mathtt{Behaviors})$ from data.

The model automatically learns a natural ordering of which behaviors should
provide stronger evidence for Value $V$, e.g.
$\mathbb{P}(V=1\mid\mathtt{Retweet}=1)>\mathbb{P}(V=1\mid\mathtt{Reply}=1)>\mathbb{P}(V=1\mid\mathtt{Click}=1)$.
Furthermore, it learns complex inferences about the evidence provided by
_combinations_ of behavior. Such inferences would not be possible under the
standard approach, which uses a linear combination of behaviors as the
objective.

Unlike other work on recommender systems, we do not evaluate through
engagement metrics. If we believe that engagement is not the same as the
construct “value”, then we cannot evaluate our approach merely by reporting
engagement numbers. Instead, we must take a more holisitc approach. We discuss
established approaches to assessing the _validity_ (; ; ) of a measurement,
and explain how they translate to the recommender system setting by using
Twitter as an example.

##  2 Related work

In the social sciences, especially in psychology, education, and political
science, measurement theory (Hand, 2004) has long been used to operationalize
constructs like “personality”, “intelligence”, “political ideology”, etc.
Often the operationalization of such constructs is heavily contested, and many
types of evidence for validity and reliability are used to evaluate the match
between a construct and its operationalization (; ).

Recently, Jacobs and Wallach (2019) introduced the language of measurement in
the context of computer science. They argue that many harms effected by
computational systems are the direct result of a mis-match between a
theoretical construct and its operationalization. In the context of
recommender systems, many have argued that the engagement metrics used in
practice are a poor operationalization of “value” (Ekstrand and Willemsen,
2016).

We use measurement theory as a principled way to disentangle latent value from
observed engagement. We provide a general latent variable model approach in
which an _anchor variable_ provides the key link between the latent variable
and the observed behaviors. The term anchor variable has been used been used
in various ways in prior work on factor models (Arora et al., 2012, 2013;
Halpern et al., 2016a, b); our usage is most similar to (Halpern et al.,
2016b). Our use of the anchor variable is also similar to the use of a _proxy
variable_ to identify causal effects under unobserved confounding (Pearl,
2010; Kuroki and Pearl, 2014).

##  3 Identification of the LVM with anchor $A$

We now describe our general approach to operationalizing a target construct
through a latent variable model (LVM) with an _anchor variable_. We
operationalize the construct for value through a LVM in which the construct is
represented through an unobserved, binary latent variable $V$ that the other
binary, observed behaviors provide evidence for. We assume there is one
observed behavior, an _anchor variable_ $A$, which we know $\mathbb{P}(V=1\mid
A=1)$ for. We represent all other observed behaviors in the binary random
vector $\mathbf{B}=(B_{1},\dots,B_{n})$. We refer to $A$ as an anchor variable
because it will provide the crucial link to identifying $\mathbb{P}(V\mid
A,\mathbf{B})$. In other words, it will _anchor_ the other observed behaviors
$\mathbf{B}$ to Value $V$.

We represent the LVM as a Bayesian network. A Bayesian network is a directed
acyclic graph (DAG) that graphically encodes a factorization of the joint
distribution of the variables in the network. In particular, the DAG encodes
all conditional independences among the nodes through the $d$-separation rule
(Pearl, 2009). This is important because in most real-world settings, the
observed behaviors have complex dependencies among each other (e.g. one may
need to click on an item before replying to it). Through our choice of the DAG
we can model both the dependencies among the observed behaviors as well as the
dependence of the unobserved variable $V$ on the observed behaviors.

Our goal is to determine $\mathbb{P}(V\mid A,\mathbf{B})$ so that it can later
be used downstream as a target for optimization. We now discuss sufficient
conditions for identifying the conditional distribution $\mathbb{P}(V\mid
A,\mathbf{B})$. There are three assumptions on the anchor variable $A$ that we
will consider in turn.

_Notation._ We use $\operatorname{Pa}(X)$ to denote the parents of a node $X$
and use $\operatorname{Pa}_{-V}(X)=\operatorname{Pa}(X)\setminus V$ to denote
all parents of $X$ except for $V$.

######  Assumption 1 (Value-sensitive).

For every realization $b$ of the random vector $\mathbf{B}$, we have that
$\mathbb{P}(A=1\mid\mathbf{B}=b,V=1)\neq\mathbb{P}(A=1\mid\mathbf{B}=b,V=0)$.

Assumption 1 simply means that the anchor $A$ carries signal about Value $V$,
regardless of what the other variables $\mathbf{B}$ are.333When combined with
Assumption 2, Assumption 1 simplifies to the condition
$\mathbb{P}(A=1\mid\operatorname{Pa}_{-V}(A)=z,V=1)\neq\mathbb{P}(A=1\mid\operatorname{Pa}_{-V}(A)=z,V=0)$
for every realization $z$ of $\operatorname{Pa}_{-V}(A)$, the parents of $A$
excluding $V$.

######  Assumption 2 (No children).

The anchor variable $A$ has no children.

Since the anchor $A$ is chosen to be a strong type of explicit feedback, it is
usually the last type of behavior the user engages in on a content item (e.g.
a “report” button that removes the content from the user’s timeline), and
thus, it typically makes sense to model $A$ as having no children.

######  Assumption 3 (One-sided conditional independence).

Let $\operatorname{Pa}_{-V}(A)$ be all parents of $A$ excluding $V$. Value $V$
is independent from $\operatorname{Pa}_{-V}(A)$ given that $A=1$:

$\mathbb{P}(V=1\mid A=1,\operatorname{Pa}_{-V}(A))=\mathbb{P}(V=1\mid A=1)\,.$

Assumption 3 means that when the user has opted to give feedback ($A=1$), the
level of information that feedback contains about Value $V$ does not depend on
the other parents of $A$. The assumption rests on the fact that $A$ is a
strong type of feedback that the user only provides when they are confident of
their assessment.

###  3.1 Conditions for identification

The next theorem establishes that under A1, the distribution of observable
behaviors $\mathbb{P}(A,\mathbf{B})$ and the conditional distribution
$\mathbb{P}(A\mid V,\mathbf{B})$ are sufficient for identifying the
conditional distribution, $\mathbb{P}(V\mid A,\mathbf{B})$. The proof uses a
_matrix adjustment method_ (Rothman et al., 2008; pg. 360) and is very similar
to that in Pearl (2010); Kuroki and Pearl (2014).

######  Theorem 1.

Let $V$ and $A$ be binary random variables and let
$\mathbf{B}=(B_{1},\dots,B_{n})$ be a binary random vector. If A1 holds, then
the distributions $\mathbb{P}(A,\mathbf{B})$ and $\mathbb{P}(A\mid
V,\mathbf{B})$ uniquely identify the conditional distribution
$\mathbb{P}(V\mid A,\mathbf{B})$.

###### Proof.

Since the conditional distribution $\mathbb{P}(V\mid A,\mathbf{B})$ is equal
to
$\frac{\mathbb{P}(\mathbf{B},V)\cdot\mathbb{P}(A\mid\mathbf{B},V)}{\mathbb{P}(A,\mathbf{B})}$,
we can reduce the problem to determining the distribution
$\mathbb{P}(\mathbf{B},V)$. We can relate $\mathbb{P}(\mathbf{B},V)$ to the
given distributions, $\mathbb{P}(A,\mathbf{B})$ and
$\mathbb{P}(A\mid\mathbf{B},V)$, via the law of total probability:

$\displaystyle\mathbb{P}(A,\mathbf{B})=\sum_{v\in\\{0,1\\}}\mathbb{P}(\mathbf{B},V=v)\mathbb{P}(A\mid\mathbf{B},V=v)\,.$
(1)

For every realization $b$ of the random vector $\mathbf{B}$, we can write
Equation 1 as $z^{b}=\mathbf{P}^{b}\mu^{b}$ where the matrix
$\mathbf{P}^{b}\in[0,1]^{2\times 2}$ and the vectors
$\mu^{b},z^{b}\in[0,1]^{2}$ are defined as

$\displaystyle\mathbf{P}^{b}_{i,j}=\mathbb{P}(A=i\mid\mathbf{B}=b,V=j)~{}\text{
for }~{}i,j\in\\{0,1\\}\,,$
$\displaystyle\mu^{b}=[\mathbb{P}(\mathbf{B}=b,V=0),\mathbb{P}(\mathbf{B}=b,V=1)]^{T}\,,$
$\displaystyle
z^{b}=[\mathbb{P}(\mathbf{B}=b,A=0),\mathbb{P}(\mathbf{B}=b,A=1)]^{T}\,.$

Determining the distribution $\mathbb{P}(\mathbf{B},V)$ is equivalent to
determining $\mu^{b}$ for all $b$. By Assumption 1, for all $b$ we have
$\mathbb{P}(A=1\mid B=b,V=1)\neq\mathbb{P}(A=1\mid B=b,V=0)$, which implies
that the determinant of the matrix $\mathbf{P}^{b}$ is non-zero. Therefore,
for all $b$, the vector $\mu^{b}$ is equal to
$\mu^{b}=(\mathbf{P}^{b})^{-1}z^{b}$. Thus, $\mathbb{P}(\mathbf{B},V)$, and
therefore the conditional distribution $\mathbb{P}(V\mid A,\mathbf{B})$, is
identified by the given distributions. ∎

If we add Assumption 2, i.e. the anchor $A$ has no children, then the
distributions $\mathbb{P}(A,\mathbf{B})$ and
$\mathbb{P}(A\mid\operatorname{Pa}(A))$ are sufficient to identify
$\mathbb{P}(V\mid A,\mathbf{B})$.

######  Corollary 1.

If the joint distribution $\mathbb{P}(V,A,\mathbf{B})$ is Markov444A
distribution $\mathbb{P}(X_{1},\dots,X_{n})$ is said to be Markov with respect
to a DAG $G$ if it factorizes according to $G$, i.e.
$\mathbb{P}(X_{1},\dots,X_{n})=\prod_{i\in[n]}\mathbb{P}(X_{i}\mid\operatorname{Pa}(X_{i}))$.
with respect to a DAG $G$ in which A1 and A2 hold, then the distributions
$\mathbb{P}(A,\mathbf{B})$ and $\mathbb{P}(A\mid\operatorname{Pa}(A))$
uniquely identify the conditional distribution $\mathbb{P}(V\mid
A,\mathbf{B})$.

###### Proof.

In a Bayesian network, the _Markov blanket_ for a variable $X$ is the set of
variables $\operatorname{MB}(X)\subseteq\mathcal{Z}$ that shield $X$ from all
other variables $\mathcal{Z}$ in the DAG, i.e.
$\mathbb{P}(X\mid\mathcal{Z})=\mathbb{P}(X\mid\operatorname{MB}(X))$ (Pearl,
2009). The Markov blanket for a variable $X$ consists of its parents,
children, and parents of its children. Since the anchor $A$ has no children,
$\mathbb{P}(A\mid
V,\mathbf{B})=\mathbb{P}(A\mid\operatorname{MB}(A))=\mathbb{P}(A\mid\operatorname{Pa}(A))$.
Thus, by Theorem 1, $\mathbb{P}(A\mid\operatorname{Pa}(A))$, and
$\mathbb{P}(A,\mathbf{B})$ identify the conditional distribution
$\mathbb{P}(V\mid A,\mathbf{B})$ ∎

Finally, when we add Assumption 3, one-sided conditional independence, then
the distributions $\mathbb{P}(V)$, $\mathbb{P}(A,\mathbf{B})$,
$\mathbb{P}(V=1\mid A=1)$, and $\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid V)$
are sufficient. The proof follows from Corollary 1 because, under Assumption
3, the distributions $\mathbb{P}(V=1\mid A=1)$,
$\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid V)$, and $\mathbb{P}(V)$ identify
$\mathbb{P}(A\mid\operatorname{Pa}(A))$.

######  Corollary 2.

If the joint distribution $\mathbb{P}(V,A,\mathbf{B})$ is Markov with respect
to a DAG $G$ in which A1-3 hold, then $\mathbb{P}(V)$,
$\mathbb{P}(A,\mathbf{B})$, $\mathbb{P}(V=1\mid A=1)$, and
$\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid V)$ uniquely identify the
conditional distribution $\mathbb{P}(V\mid A,\mathbf{B})$.

###### Proof.

We will show that, under Assumption 3, the distributions $\mathbb{P}(V=1\mid
A=1)$, $\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid V)$, and $\mathbb{P}(V)$
identify $\mathbb{P}(A\mid\operatorname{Pa}(A))$. The proof then follows from
Corollary 1.

We show that we can identify $\mathbb{P}(A\mid\operatorname{Pa}(A))$ by
solving a set of linear equations. For short-hand let
$p_{w,a,v}=\mathbb{P}(\operatorname{Pa}_{-V}(A)=w,A=a,V=v)$. For any
realization $w$, by marginalizing over $A$ and $V$, we can derive the
following four equations for the four unknown probabilities
$p_{w,0,0},\,p_{w,0,1},\,p_{w,1,0},\,p_{w,1,1}$:

$\displaystyle\mathbb{P}(\operatorname{Pa}_{-V}(A)=w,A=0)=p_{w,0,0}+p_{w,0,1}$
(2)
$\displaystyle\mathbb{P}(\operatorname{Pa}_{-V}(A)=w,A=1)=p_{w,1,0}+p_{w,1,1}$
(3)
$\displaystyle\mathbb{P}(\operatorname{Pa}_{-V}(A)=w,V=0)=p_{w,0,0}+p_{w,1,0}$
(4)
$\displaystyle\mathbb{P}(\operatorname{Pa}_{-V}(A)=w,V=1)=p_{w,0,1}+p_{w,1,1}$
(5)

Note that the LHS of Equations 2 and 3 are given by $\mathbb{P}(A,\mathbf{B})$
and the LHS of Equations 4 and 5 are given by the prior $\mathbb{P}(V)$ and
$\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid V)$.

From Assumption 3, one-sided conditional independence, we know that
$\mathbb{P}(V=1\mid A=1,\operatorname{Pa}_{-V}(A))=\mathbb{P}(V=1\mid A=1)$.
Under one-sided conditional independence, the probability $p_{w,1,1}$ is
determined by the given distributions:

$\displaystyle p_{w,1,1}=~{}$
$\displaystyle\mathbb{P}(A=1)\cdot\mathbb{P}(\operatorname{Pa}_{-V}(A)=w\mid
A=1)$ $\displaystyle\cdot\mathbb{P}(V=1\mid A=1,\operatorname{Pa}_{-V}(A)=w)$
$\displaystyle=~{}$
$\displaystyle\mathbb{P}(A=1)\cdot\mathbb{P}(\operatorname{Pa}_{-V}(A)=w\mid
A=1)$ $\displaystyle\cdot\mathbb{P}(V=1\mid A=1)\,.$ (6)

Since $p_{w,1,1}$ is determined by the given distributions, so are
$p_{w,0,0}$, $p_{w,1,0}$, and $p_{w,0,1}$, which can be solved for through
Equations 2-5. Since this holds for any realization $w$, the distribution
$\mathbb{P}(A,V,\operatorname{Pa}_{-V}(A))=\mathbb{P}(A,\operatorname{Pa}(A))$
is determined, which by Collorary 1 means that the conditional distribution
$\mathbb{P}(V\mid A,\mathbf{B})$ is determined. ∎

###  3.2 Specifying the distributions for identification

Corollary 2 establishes that, under Assumptions 1-3, the distributions
$\mathbb{P}(V)$, $\mathbb{P}(A,\mathbf{B})$,
$\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid V)$, and $\mathbb{P}(A=1\mid V=1)$
are sufficient to determine the conditional distribution $\mathbb{P}(V\mid
A,\mathbf{B})$ for the LVM. Where do we get these distributions?

  1. 1.

The distribution of observable nodes $\mathbb{P}(A,\mathbf{B})$ is estimated
by the empirical distribution of observed data.

  2. 2.

The distribution $\mathbb{P}(V)$ over the latent variable for value $V$ is a
prior distribution that is specified by the modeler. Recall that our goal with
the LVM is to use $\mathbb{P}(V=1\mid\mathbf{B},A)$ as an objective to
optimize. Since the prior $\mathbb{P}(V)$ only has a scaling effect on
$\mathbb{P}(V=1\mid\mathbf{B},A)$, it does not matter greatly. We set
$\mathbb{P}(V)$ to be uniform, i.e. $\mathbb{P}(V=1)=0.5$.

  3. 3.

The conditional probability $\mathbb{P}(V=1\mid A=1)$ is specified by our
assumption on the the anchor variable. The probability $\mathbb{P}(V=1\mid
A=1)$ is set to $\epsilon$ where $\epsilon\approx 0$ if $A$ is explicit
negative feedback or to $1-\epsilon$ if $A$ is explicit positive feedback.

  4. 4.

That leaves the distribution $\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid V)$. We
estimate $\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid A)$ heuristically using two
sources of historical data that vary in their distribution of Value $V$.
Suppose we have access to a dataset of historical recommendations
$\mathcal{D}_{R}$ that were sent to users at random, as well as a dataset of
historical recommendations that were algorithmically chosen,
$\mathcal{D}_{C}$. The randomized and algorithmic datasets will have different
distributions of valuable content, $\mathbb{P}_{R}(V)$ and
$\mathbb{P}_{C}(V)$, and different distributions of observed behavior,
$\mathbb{P}_{R}(A,\mathbf{B})$ and $\mathbb{P}_{C}(A,\mathbf{B})$. However, we
assume that $\mathbb{P}(A,\mathbf{B}\mid V)$, the probability of the observed
behavior given Value $V$, is the same between the two datasets.555If our DAG
has Value $V$ as a root node and can be interpreted as a causal Bayesian
network (Pearl, 2009), then this is equivalent to assuming that the difference
between the datasets corresponds to an intervention on $V$. The following
equations then hold:

$\displaystyle\mathbb{P}_{R}(\operatorname{Pa}_{-V}(A))$
$\displaystyle=~{}\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid
V=1)\mathbb{P}_{R}(V=1)$
$\displaystyle+\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid
V=0)\mathbb{P}_{R}(V=0)\,,$ (7)
$\displaystyle\mathbb{P}_{C}(\operatorname{Pa}_{-V}(A))$
$\displaystyle=~{}\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid
V=1)\mathbb{P}_{C}(V=1)$
$\displaystyle+\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid
V=0)\mathbb{P}_{C}(V=0)\,.$ (8)

We specify $\mathbb{P}_{R}(V)$ and $\mathbb{P}_{C}(V)$ in an application-
dependent way, but, generally, we assume the randomized dataset is lower value
than the algorithmic one: $\mathbb{P}_{R}(V)<\mathbb{P}_{C}(V)$. Once we
specify $\mathbb{P}_{R}(V)$ and $\mathbb{P}_{C}(V)$ and estimate
$\mathbb{P}_{R}(A,\mathbf{B})$ and $\mathbb{P}_{C}(A,\mathbf{B})$ empirically,
then we can solve Equations 4 and 4 to estimate
$\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid V=1)$. This is a heuristic approach
that is appropriate for getting a rough estimate, but needs to be used with
care. In practice, not all the differences between the randomized and
algorithmic dataset can be explained by an intervention on Value $V$. For
example, if the recommendation algorithm has historically been optimized for
user clicks, then users in the algorithmic dataset may click on items more,
but for reasons other than increased value.

###  3.3 Algorithm for identification

We now give more details on how we calculate the joint distribution
$\mathbb{P}(V,A,\mathbf{B})$ given the distributions $\mathbb{P}(V)$,
$\mathbb{P}(A,\mathbf{B})$, $\mathbb{P}(V=1\mid A=1)$ and
$\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid V)$. We use the structure of the
Bayesian network to efficiently identify the joint distribution
$\mathbb{P}(V,A,\mathbf{B})$ by fitting each factor
$\mathbb{P}(X\mid\operatorname{Pa}(X))$ for every variable $X$.

  1. 1.

The factor for $V$ is given by the prior $\mathbb{P}(V)$.666Assuming that $V$
is a root node, which is the case in any network we are interested in.

  2. 2.

The factor for $A$, i.e. $\mathbb{P}(A\mid\operatorname{Pa}(A))$, can be
identified from $\mathbb{P}(V)$, $\mathbb{P}(V=1\mid A=1)$, and
$\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid V)$ by solving a set of linear
equations as in the proof of Corollary 2.

  3. 3.

The factor for any behavior that does not have $V$ as a parent is directly
identified by the distribution of observable behaviors
$\mathbb{P}(A,\mathbf{B})$.

  4. 4.

The factors for the remaining behaviors which have $V$ as a parent are fit
through a _matrix adjustment method_ (Rothman et al., 2008; pg. 360). In
particular, note that

$\displaystyle\mathbb{P}(X=1,\operatorname{Pa}_{-V}(X)=z,\operatorname{Pa}_{-V}(A)=w,A=a)=$
$\displaystyle(\sum_{v\in\\{0,1\\}}\mathbb{P}(A=a\mid\operatorname{Pa}_{-V}(A)=w,V=v)$
$\displaystyle\cdot\mathbb{P}(X=1,\operatorname{Pa}_{-V}(X)=z,\operatorname{Pa}_{-V}(A)=w,V=v))$

We can also write the above equation in matrix form. Let $z_{1},\dots,z_{m}$
be all realizations of $\operatorname{Pa}_{-V}(X)$, and define the matrices
$Q^{w}\in[0,1]^{2\times m}$, $R^{w}\in[0,1]^{2\times 2}$,
$S^{w}\in[0,1]^{2\times m}$ as777If
$\operatorname{Pa}_{-V}(X)\cap\operatorname{Pa}_{-V}(A)\neq\emptyset$ and
$\operatorname{Pa}_{-V}(X)=z_{i}$ and $\operatorname{Pa}_{-V}(A)=w$ conflict,
then simply set $Q^{w}_{0,i}=Q^{w}_{1,i}=0$.

$\displaystyle Q^{w}_{a,i}=\mathbb{P}(X=1,\operatorname{Pa}_{-V}(X)=z_{i},$
(9) $\displaystyle\qquad\qquad\operatorname{Pa}_{-V}(A)=w,A=a),$
$\displaystyle R^{w}_{av}=\mathbb{P}(A=a\mid\operatorname{Pa}_{-V}(A)=w,V=v),$
(10) $\displaystyle
S^{w}_{v,i}=\mathbb{P}(X=1,\operatorname{Pa}_{-V}(X)=z_{i},$ (11)
$\displaystyle\qquad\qquad\operatorname{Pa}_{-V}(A)=w,V=v).$

Then, $Q^{w}=R^{w}S^{w}$ and $S^{w}=(R^{w})^{-1}Q^{w}$.888$R^{w}$ is
invertible because of Assumption 1. Let $S$ be the marginalization over $w$:
$\sum_{w}S^{w}=(R^{w})^{-1}Q^{w}$. Then
$S_{v,i}=\mathbb{P}(X=1,\operatorname{Pa}_{-V}(X)=z_{i},V=v)$. Thus, the
factor for $X$ is equal to
$\mathbb{P}(X\mid\operatorname{Pa}_{-V}(X)=z_{i},V=v)=S_{v,i}/\mathbb{P}(\operatorname{Pa}_{-V}(X)=z_{i},V=v)$.
We fit nodes with $V$ as a parent in topological order, so that we can always
calculate the denominator from previously fit factors.

Figure 1: A workflow of how users can interact with ML-based notifications on
Twitter. To view the tweet, the user can either “open” the notification from
the home screen on their phone or “click” on it from the notifications tab
within the app. If the user sees the tweet from their notifications tab, they
can also click "See Less Often" on it. Once the user has opened or clicked on
the notification, they can engage with the tweet in many ways, e.g. replying,
retweeting, or favoriting. At any point, the user can opt-out of notifications
all-together. Figure 2: Bayesian network for Twitter notifications. An arrow
from a node $X$ to a box means that the node $X$ is a parent of all the nodes
in the box, e.g. $\mathtt{Click}$ and $\mathtt{Open}$ are parents of
$\mathtt{Fav}$, $\mathtt{RT}$, …, $\mathtt{Linger>6s}$. The latent variable
$\mathtt{Value}$ is a parent of everything except
$\mathtt{NTab}\mathtt{View}$. The measurement node $\mathtt{SLO}$ is
highlighted in pink.

##  4 Application to Twitter

We implemented our approach on the Twitter platform on millions of users. On
Twitter, there are many kinds of user behaviors: clicks, replies, favorites,
retweets, etc. The typical approach to recommendations would involve
optimizing an objective that trades-off these behaviors, usually with linear
weights. However, designing an objective is a non-trivial problem. How exactly
should we weigh favorites compared to clicks or replies or retweets or any of
the numerous other behaviors? It is difficult to assess whether the weights we
chose match the notion of “value” we intended.

Furthermore, even supposing that we could manually specify the “correct”
weights through laborious trial-and-error, the correct weights change over
time. For example, after videos shared on Twitter began to auto-play, the
signal of whether or not a user watched a video presumably became less
relevant. The reality is that the objective is never static - how users
interact with the platform is constantly changing, and the objective must
change accordingly.

Our approach provides a principled solution to objective specification. We
directly operationalize our intended construct “value” as a latent variable
$V$. The meaning of Value $V$ is defined by the Bayesian network and the
_anchor variable_ $A$, a behavior that we believe provides strong evidence for
value or the lack of it. On Twitter, the user can provide strong, explicit
feedback by clicking “See less often” (SLO) on a tweet. We use SLO as our
anchor $A$ and assume that if a user clicks "See less often" on a tweet, they
do not value it: $\mathbb{P}(V=1\mid\mathtt{SLO}=1)=0$.

Under this approach, there is no need to manually specify how all the
behaviors should factor into the objective. Having operationalized Value, the
ideal objective to use is clear: $\mathbb{P}(V=1\mid\mathbf{B},A)$ \- the
probability of Value $V$ given the observed behaviors. As discussed in Section
4, we can directly estimate $\mathbb{P}(V=1\mid\mathbf{B},A)$ from data.
Furthermore, presuming that the anchor and structure of Bayesian network
remain stable, we can regularly re-estimate the model with new data at any
point, allowing us to account for change in user behavior on the platform.

The Bayesian network. We applied our approach to ML-driven notifications on
Twitter. These notifications have various forms, e.g. "Users A, B, C just
liked User Z’s tweet", "User A just tweeted after a long time", or "Users A,
B, C followed User Z". Figure 1 shows an example notification and how a user
can interact with it. The Bayesian network in Figure 2 succinctly encodes the
dependencies between different types of interactions users can have with
notifications.999The network can be interpreted as a _causal_ Bayesian network
(Pearl, 2009), although for our purposes, we do not strictly need the causal
interpretation.

Notifications are sent both to the user’s home screen on their mobile phone,
as well as to the notifications tab within the Twitter app. The user can start
their interaction either by seeing the notification in their notification tab
($\mathtt{NTab}\mathtt{View}$), and then clicking on it ($\mathtt{Click}$), or
by seeing it as a the notification on their phone home screen and opening it
from there directly ($\mathtt{Open}$). After clicking or opening the
notification, the user can engage in many more interactions: they can favorite
($\mathtt{Fav}$), retweet ($\mathtt{RT}$), quote retweet ($\mathtt{Quote}$),
or reply ($\mathtt{Reply}$) to the tweet; if the tweet has a link, they can
click on it ($\mathtt{LinkClick}$); if it has a video, they can watch it
($\mathtt{VidWatch}$). In addition, other implicit signals are logged: whether
the amount the user lingered on the tweet exceeds certain thresholds
($\mathtt{Linger>6s}$, $\mathtt{Linger>12s}$, $\mathtt{Linger>20s}$) and
whether the number of user active minutes ($\mathtt{UAM}$) spent in the app
after clicking/opening the notification exceeds a threshold.

Furthermore, when the user is in the notification tab, the user can provide
explicit feedback on a particular notification by clicking "See Less Often"
($\mathtt{SLO}$) on it. Notably, unlike other types of behavior, the user does
not need to actually click or open the notification before clicking SLO.
However, we found empirically that users are more likely to click SLO after
clicking or opening the notification, probably because they need to gain more
information before making an assessment. Thus, in addition to
$\mathtt{NTab}\mathtt{View}$, we also model $\mathtt{Click}$ and
$\mathtt{Open}$ as parents of $\mathtt{SLO}$.

Finally, at any time the user can opt-out of notifications to their phone home
screen ($\mathtt{Opt}\mathtt{Out}$). When the user decides to opt-out, it is
attributed to any ML-based notification saw within a day of choosing to opt-
out. Since ML-based notifications are relatively rare on Twitter (users
usually get less than one a day), there are usually at most one or two
notifications attributed to an opt-out event.

We model the latent variable $V$ as being a parent of all behaviors except
$\mathtt{NTab}\mathtt{View}$ (whether or not the user saw the notification in
their notifications tab or not). Since users may check their notifications tab
for many other notifications, it is difficult to attribute
$\mathtt{NTab}\mathtt{View}$ to a particular notification, and so we consider
it to be an exogenous, random event.

Identifying the joint distribution We fit our model on three days of data that
contained of all user interactions with ML-based push notifications on
Twitter. In Section 3, we proved that the target objective - the conditional
distribution $\mathbb{P}(V=1\mid\mathbf{B},A)$ \- is uniquely identified from
$\mathbb{P}(V=1\mid A=1)$, $\mathbb{P}(V)$, $\mathbb{P}(\mathbf{B},A)$, and
$\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid A)$ (see Corollary 2). We set the
four distributions as follows. We used $\mathtt{SLO}$ as our anchor variable
$A$ and assumed that $\mathbb{P}(V=1\mid A=1)=0$, i.e. a user never says “See
less often” if they value the notification. The prior distribution of value
$\mathbb{P}(V)$ was set to be uniform. The distribution of observed behaviors
$\mathbb{P}(\mathbf{B},A)$ was set to the empirical distribution. The
distribution $\mathbb{P}(\operatorname{Pa}_{-V}(A)\mid V)$ was estimated as
described in Section 3.2 by using two sources of historical data, one in which
notifications were sent at random and the other in which notifications were
sent according to a recommendation algorithm.101010We assume that the dataset
of randomized notifications has a prior probability $\mathbb{P}_{R}(V=1)=0$
and the dataset of algorithmically chosen notifications has a prior
probability $\mathbb{P}_{C}(V=1)=0.5$.

$\mathbb{P}(V=1\mid\mathtt{Behavior}=1)$
---
$\mathtt{Behavior}$ | Naive Bayes | $\mathtt{Click},\mathtt{Open}\nrightarrow\mathtt{SLO}$ | Full Model
$\mathtt{Opt}\mathtt{Out}$ | 0 | 0 | 0
$\mathtt{Click}$ | 0 | 0.316 | 0.652
$\mathtt{Open}$ | 0 | 0.442 | 0.685
$\mathtt{UAM}$ | 0 | 0.157 | 0.719
$\mathtt{VidWatch}$ | 0 | 0.254 | 0.772
$\mathtt{Linger>6s}$ | 0 | 0.264 | 0.802
$\mathtt{LinkClick}$ | 0 | 0.320 | 0.836
$\mathtt{Reply}$ | 0.358 | 0.570 | 0.932
$\mathtt{Linger>12s}$ | 0 | 0.245 | 0.948
$\mathtt{Fav}$ | 0.579 | 0.672 | 0.949
$\mathtt{RT}$ | 0.680 | 0.720 | 0.956
$\mathtt{Linger>20s}$ | 0.019 | 0.296 | 0.991
$\mathtt{Quote}$ | 1.0 | 1.0 | 1.0
Table 1: The inferences made by LVMs with different DAGs. For each model and
for each behavior, we list $\mathbb{P}(V=1\mid\mathtt{Behavior}=1)$ – how much
evidence the model learns that a behavior provides for Value $V$ (when all
other behaviors are marginalized over).

Evaluation of internal structure. Assessing our measure of “value” for
validity will necessarily be an on-going and multi-faceted process. We do not,
as typical of papers on recommendation, report engagement metrics. The reason
is that if we expect our measure of “value” to differ from engagement, we
cannot evaluate it by simply reporting engagement metrics. The evaluation of a
measurement necessitates a more holistic approach. In Section 5, we describe
the five categories of evidence for validity described by the _Standards for
educational and psychological testing_ , the handbook considered the gold
standard on approaches to testing (AERA, APA, NCME, ).

Here, we focus on evaluating what is known as _evidence based on internal
structure_ , i.e whether expected theoretical relationships between the
variables in the model hold. To justify why the structure of our Bayesian
network is necessary, we compare our full model from Figure 2 to two other
models: a naive Bayes model and the full model but without arrows from
$\mathtt{Open}$ and $\mathtt{Click}$ to $\mathtt{SLO}$. In Table 1, we show
$\mathbb{P}(V=1\mid\mathtt{Behavior}=1)$ for all behaviors and models. As
noted by prior work (Pearl, 2009; Halpern et al., 2016b), matrix adjustment
methods can result in negative values when conditional independence
assumptions are not satisfied. To address this, we clamp all inferences to the
interval $[0,1]$. We include the table of non-clamped inferences in the
appendix (Table 2).

The first, simple theoretical relationship we expect to hold is that compared
to observing no user interaction, observing any user behavior besides opt-out
should increase the probability that the user values the tweet, i.e.
$\mathbb{P}(V=1\mid\mathtt{Behavior}=1)<\mathbb{P}(V=1)=0.5$ for all
$\mathtt{Behavior}\neq\mathtt{Opt}\mathtt{Out}$. Furthermore, we also expect
some behaviors to provide stronger signals of value than others, e.g. that
$\mathbb{P}(V=1\mid\mathtt{Fav}=1)>\mathbb{P}(V=1\mid\mathtt{Click}=1)$.

The first model is the naive Bayes model, which simply assumes that all
behaviors are conditionally independent given Value $V$. It does extremely
poorly - almost all inferences have negative values and are clamped to zero,
indicating that the conditional independence assumptions are unrealistic.

The second model is the full model except without arrows from $\mathtt{Click}$
and $\mathtt{Open}$ to $\mathtt{SLO}$. It models all pre-requisite
relationships between behaviors, i.e. if a behavior $X$ is required for
another behavior $Y$, then there is an arrow from $X$ to $Y$. Compared to the
naive Bayes model, the second model does not make mainly negative-valued
inferences, indicating that its conditional independence assumptions are more
realistic. However, relative to the prior, most behaviors actually reduce the
probability of $\mathtt{Value}$, rather than increase it!

After investigation, we realized that although users were not technically
_required_ to click or open the notification before clicking SLO, in practice,
they were more likely to do so, probably because they needed to gain
information before making an assessment. We found that explicitly modeling the
connection, i.e. adding arrows from $\mathtt{Click}$ and $\mathtt{Open}$ to
$\mathtt{SLO}$ was critical for making reasonable inferences. We believe this
takeaway will apply across recommender systems. The user never has perfect
information and may need to engage with an item before providing explicit
feedback (Wen et al., 2019). It is important to model the relationship between
information-gaining behavior and explicit feedback in the Bayesian network.

Our full model satisfies the theoretical relationships we expect. All the
behaviors that we expect to increase the probability of Value $V$ do indeed do
so. Furthermore, the relative strength of different types of behavior seems
reasonable as well, e.g. $\mathbb{P}(V=1\mid\mathtt{Fav}=1)$ and
$\mathbb{P}(V=1\mid\mathtt{RT}=1)$ are higher than
$\mathbb{P}(V=1\mid\mathtt{VidWatch}=1)$ and
$\mathbb{P}(V=1\mid\mathtt{LinkClick}=1)$.

The full model also makes more nuanced theoretical inferences. Recall that
$\mathtt{UAM}$ is whether or not the user had high user active minutes after
either clicking the notification from notifications tab or by opening the
notification from their phone home screen. The model learns that
$\mathtt{UAM}$ is a highly indicative signal after $\mathtt{Open}$, but not
after $\mathtt{Click}$:
$\mathbb{P}(V=1\mid\mathtt{Open}=1,\mathtt{UAM}=1)=0.906$ and
$\mathbb{P}(V=1\mid\mathtt{Click}=1,\mathtt{UAM}=1)=0.641$. This makes sense
because if the user clicks from notifications tab, it means they were already
in the app, and it is difficult to attribute their high UAM to the
notification in particular. On the other hand, if the user enters the app
because of the notification, it is much more direct of an attribution.

It is clear that manually specifying the inferences our model makes would be
very difficult. The advantage of our approach is that after specifying (a) the
anchor variable and (b) the Bayesian network, we can automatically learn these
inferences from data. Further, the model is able to learn complex inferences
(e.g. that $\mathtt{UAM}$ is more reliable after $\mathtt{Open}$ than
$\mathtt{Click}$) that would be impossible to specify under the typical linear
weighting of behaviors.

##  5 Assessing validity

Thus far, we have described our framework for designing a measure of “value”,
which can be used as a principled replacement for the ad-hoc objectives
ordinarily used in engagement optimization. How do we evaluate such a measure?
Notably, we do not advocate evaluating the measure purely through engagement
metrics. If we expect our measure of “value” to differ from engagement, then
we cannot evaluate it by simply reporting engagement metrics. Instead, the
assessment of any measure is necessarily an ongoing, multi-faceted, and
interdisciplinary process.

To complete the presentation of our framework, we now discuss approaches to
assess the _validity_ (; ; ) of a measurement. In the most recent (2014)
edition of the _Standards for educational and psychological testing_ , the
handbook considered the gold standard on approaches to testing, there are five
categories of evidence for validity (AERA, APA, NCME, ). We visit each in
turn, and describe how they translate to the recommender system setting, using
Twitter as an example.

Evidence based on content refers to whether the content of a measurement is
sufficient to fully capture the target construct. For example, we may question
whether a measure of “socio-economic status” that includes income, but does
not account for wealth, accurately captures the content of the construct
(Jacobs and Wallach, 2019). In the recommender engine setting, content-based
evidence asks us to reflect on whether the behaviors available on the platform
are sufficient to capture a worthy notion of the construct “value”. For
example, if the only behavior observed on the platform were clicks by the
user, then we may be skeptical of any measurement of “value” derived from user
behavior. What content-based evidence makes clear is that to measure any
worthy notion of “value”, it is essential to design platforms in which users
are empowered with richer channels of feedback. Otherwise, no measurement
derived from user behavior will accurately capture the construct.

Evidence based on cognitive processes. Measurements derived from human
behavior are often based on implicit assumptions about the cognitive processes
subjects engage in. Cognitive process evidence refers to evidence about such
assumptions, often derived from explicit studies with subjects. For example,
consider a reading comprehension test. We assume that high-scoring students
succeed by using critical reading skills, rather than a superficial heuristic
like picking the answers with the longest length. To gain evidence about
whether this assumption holds, we might, for instance, ask students to take
the test while verbalizing what they are thinking.

Similarly, in the recommender engine setting, we want to verify whether user
behaviors occur for the reasons we think they do. On Twitter, one might think
to use $\mathtt{Favorite}$ as an anchor for Value $V$, assuming that
$\mathbb{P}(V=1\mid\mathtt{Favorite}=1)\approx 1$. However, users actually
favorite items for reasons that may not reflect value – like to bookmark a
tweet or to stop a conversation. Cognitive process evidence highlights the
importance of user research in assessing the validity of any measure of
“value”.

Evidence based on internal structure refers to whether the observations the
measurement is derived from conform to expected, theoretical relationships.
For example, for a test with questions which we expect to be of increasing
difficulty, we would assess whether students actually perform worse on later
questions, compared to earlier ones. In the recommender system context, we may
have expectations on which types of user behaviors should provide stronger
signal for value. In Section 4, we evaluated internal structure by comparing
$\mathbb{P}(V=1\mid\mathtt{Behavior}=1)$ for all behaviors.

Evidence based on relations with other variables is concerned with the
relationships between the measurement and other variables that are external to
the measurement. The external variables could be variables which the
measurement is expected to be similar to or predict, as well as variables
which the measurement is expected to differ from. For example, a new measure
of depression should correlate with other, existing measures of depression,
but correlate less with measures of other disorders. In the recommender system
context, we might look at whether our derived measurement of “value” is
predictive of answers that users give in explicit surveys about content they
value. We could also verify that our measure of “value” does not differ based
on protected attributes, like the sex or race of the author of the content.

Evidence based on consequences. Finally, the consequences of a measurement
cannot be separated from its validity. Consider a test to measure student
mathematical ability. The test is used to sort students into beginner or
advanced classes with the hypothesis that all students will do better after
sorted into their appropriate class. If it turns out that students sorted by
the test do _not_ perform better, that may give us reason to reassess the
original test. In the recommender system context, if we find that after using
our measurement of value to optimize recommendations, more users complain or
quit the platform, then we would have reason to revise our measurement.

##  6 Summary

We have presented a framework for designing an objective function that
captures a desired notion of “value”. In line with the principles of
measurement theory, we treat “value” as a theoretical construct which must be
operationalized. Our framework allows the designer to operationalize “value”
in a principled manner by specifying only an _anchor variable_ and the
structure of the Bayesian network. Through these two choices, the designer has
the flexibility to give “value” subtly different meanings.

We applied our approach on the Twitter platform on millions of users. We do
not, as typical of papers on recommendation, report engagement metrics. The
reason is that if we expect our measure of “value” to differ from engagement,
we cannot evaluate it by simply reporting engagement metrics. Instead, we
discussed established ways to assess the validity of a measurement and how
they translate to the recommendation system setting. For the scope of this
work, we focused on assessing _evidence based on internal structure_ and found
that our measure of “value” satisfied many desired theoretical relationships.

## Acknowledgements

We thank Naz Erkan for giving us the opportunity and freedom to conduct this
work through her bold leadership and savvy managerial support. We thank
Prakhar Biyani for his extensive effort in helping us apply our approach at
scale at Twitter. We thank Tom Everitt for feedback on a draft of the paper.

## References

  * Ekstrand and Willemsen (2016) Michael D Ekstrand and Martijn C Willemsen.  Behaviorism is not enough: better recommendations through listening to users.  In _Proceedings of the 10th ACM Conference on Recommender Systems_ , pages 221–224, 2016.
  * Wen et al. (2019) Hongyi Wen, Longqi Yang, and Deborah Estrin.  Leveraging post-click feedback for content recommendations.  In _Proceedings of the 13th ACM Conference on Recommender Systems_ , pages 278–286, 2019.
  * Hand (2004) David J Hand.  _Measurement theory and practice: The world through quantification_.  Arnold London, 2004.
  * Jackman (2009) Simon Jackman.  Measurement.  In _The Oxford Handbook of Political Methodology_ , chapter 9. Oxford University Press, 09 2009.  ISBN 9780199286546.
  * Pearl (2009) Judea Pearl.  _Causality_.  Cambridge university press, 2009.
  * American Educational Research Association, American Psychological Association, National Council on Measurement in Education, Joint Committee on Standards for Educational and Psychological Testing (2014) American Educational Research Association, American Psychological Association, National Council on Measurement in Education, Joint Committee on Standards for Educational and Psychological Testing.  _Standards for educational and psychological testing_.  AERA, 2014.
  * Messick (1987) Samuel Messick.  Validity.  _ETS Research Report Series_ , 1987(2):i–208, 1987.
  * Reeves and Marbach-Ad (2016) Todd D Reeves and Gili Marbach-Ad.  Contemporary test validity in theory and practice: A primer for discipline-based education researchers.  _CBE—Life Sciences Education_ , 15(1):rm1, 2016\.
  * Jacobs and Wallach (2019) Abigail Z Jacobs and Hanna Wallach.  Measurement and fairness.  _arXiv preprint arXiv:1912.05511_ , 2019.
  * Arora et al. (2012) Sanjeev Arora, Rong Ge, and Ankur Moitra.  Learning topic models–going beyond svd.  In _2012 IEEE 53rd Annual Symposium on Foundations of Computer Science_ , pages 1–10. IEEE, 2012.
  * Arora et al. (2013) Sanjeev Arora, Rong Ge, Yonatan Halpern, David Mimno, Ankur Moitra, David Sontag, Yichen Wu, and Michael Zhu.  A practical algorithm for topic modeling with provable guarantees.  In _International Conference on Machine Learning_ , pages 280–288, 2013.
  * Halpern et al. (2016a) Yoni Halpern, Steven Horng, Youngduck Choi, and David Sontag.  Electronic medical record phenotyping using the anchor and learn framework.  _Journal of the American Medical Informatics Association_ , 23(4):731–740, 2016a.
  * Halpern et al. (2016b) Yoni Halpern, Steven Horng, and David Sontag.  Clinical tagging with joint probabilistic models.  In _Conference on Machine Learning for Health Care_ , 2016b.
  * Pearl (2010) Judea Pearl.  On measurement bias in causal inference.  In _Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence_ , pages 425–432. AUAI Press, 2010.
  * Kuroki and Pearl (2014) Manabu Kuroki and Judea Pearl.  Measurement bias and effect restoration in causal inference.  _Biometrika_ , 101(2):423–437, 2014.
  * Rothman et al. (2008) Kenneth J Rothman, Sander Greenland, and Timothy L Lash.  _Modern epidemiology_ , volume 3.  Wolters Kluwer Health/Lippincott Williams & Wilkins Philadelphia, 2008\.

$\mathbb{P}(V=1\mid\mathtt{Behavior}=1)$
---
$\mathtt{Behavior}$ | Naive Bayes | $\mathtt{Click},\mathtt{Open}\nrightarrow\mathtt{SLO}$ | Full Model
$\mathtt{Opt}\mathtt{Out}$ | -99.74 | -0.932 | -0.072
$\mathtt{Click}$ | -1.194 | 0.316 | 0.652
$\mathtt{Open}$ | -0.366 | 0.442 | 0.685
$\mathtt{UAM}$ | -1.092 | 0.157 | 0.719
$\mathtt{VidWatch}$ | -0.475 | 0.254 | 0.772
$\mathtt{Linger>6s}$ | -0.525 | 0.264 | 0.802
$\mathtt{LinkClick}$ | -0.302 | 0.320 | 0.836
$\mathtt{Reply}$ | 0.358 | 0.570 | 0.932
$\mathtt{Linger>12s}$ | -0.254 | 0.245 | 0.948
$\mathtt{Fav}$ | 0.579 | 0.672 | 0.949
$\mathtt{RT}$ | 0.680 | 0.720 | 0.956
$\mathtt{Linger>20s}$ | 0.019 | 0.296 | 0.991
$\mathtt{Quote}$ | 1.0 | 1.0 | 1.0
Table 2: The same inferences as in Table 1, except without clamping to
$[0,1]$.


# A nonlinear room mode determines the operating conditions of a large-cavity
synthetic jet actuator at low frequencies

L.F. Olivera-Reyes  Instituto de Ciencias Aplicadas y Tecnología, Universidad
Nacional Autónoma de México, Circuito Exterior S/N, Ciudad Universitaria,
04510, Mexico City.  E.S. Palacios de Paz  Instituto de Ciencias Aplicadas y
Tecnología, Universidad Nacional Autónoma de México, Circuito Exterior S/N,
Ciudad Universitaria, 04510, Mexico City.  S. Sánchez  Instituto de Ciencias
Aplicadas y Tecnología, Universidad Nacional Autónoma de México, Circuito
Exterior S/N, Ciudad Universitaria, 04510, Mexico City.  J.F. Hernández-
Sánchez  Instituto de Ciencias Aplicadas y Tecnología, Universidad Nacional
Autónoma de México, Circuito Exterior S/N, Ciudad Universitaria, 04510, Mexico
City<EMAIL_ADDRESS>
###### Abstract

Synthetic Jet (SJ) actuators are an intrinsically complex combination of
electronics, electric and mechanical systems. When studied theoretically,
these elements are often simplified to coupled damped harmonic oscillators
(DHO) that induce a pressure field within the cavity and drive momentum
exchange. Thus, the performance of an SJ actuator results from coupling these
DHOs, naturally leading to a few resonant modes. There is good evidence in the
specialized literature of two resonant modes developing in SJ actuators: the
membrane/piezoelectric mode and the Helmholtz resonance. In this work, we
report on the effect of a third resonant mode that develops at very low
frequencies due to a cavity much larger than the volume displaced by the
actuator. We present evidence that the large-cavity dynamics determine the SJ
performance in combination with the well-described formation criteria. We
compare the intensity of this resonant mode with the first room modes using
standard frequency analysis. Unlike typical room modes, the distribution of
this resonant mode is very biased to lower frequencies. We also show that the
resonant mode may be dimmed and focused by adding an obstacle in different
cavity positions for the lower sound intensities. This mode overcomes the
Helmholtz resonance, dominating the dynamics for higher sound intensities. We
show that jet and vortex velocities mimic the sound pressure curve for the
low-frequency range. Its effect mitigates for the higher range due to a delve
through smaller stroke lengths, characterized as a fixed relation between the
Reynolds and the Stokes numbers. We consider that the large-cavity dynamics is
an additional element that, if integrated as design criteria, could extend the
optimum frequency response of SJs.

##  I Introduction

Synthetic jet (SJ) actuators produce a periodic flow outwards the neck of a
Helmholtz resonator operating as intermittent pumps. Since the cavity of the
resonator refills during the negative oscillation cycle, the outward flow does
not require the addition of net massMohseni and Mittal (2014). Thus, SJs are
alternatively known in the literature as zero-net-mass-flux (ZNMF) jets.
Despite not adding net mass, SJs produce periodic bursts of momentum Krishnan
and Mohseni (2009). The SJ dynamics have been investigated extensively for
more than 20 years nowGlezer and Amitay (2002). One of the most important
findings was the vortex formation criteria in dimensionless parametersSmith
and Glezer (1998); Cater and Soria (2002); Broučková et al. (2016). The two
most relevant dimensionless parameters that summarize the vortex formation
dynamics are the Reynolds and the Stokes numbers. In this chart, curves of Re
$\sim$ S2 correspond to constant dimensionless stroke lengths. In this
relationship, a prefactor of 0.16 corresponds to a dimensionless stroke length
of 0.25 and the threshold of the vortex formation criteria. Even though the
dimensionless stroke length is critical for the vortex formation, the
transition through different prefactors influences the velocity and flow rate.
Thus, crossing through dimensionless stroke lengths does not represent a sharp
transition but changes to weaker flow rates.

A typical SJ actuator combines two components: (1) a transducer and (2) a
Helmholtz resonatorChiatto et al. (2017). The transducer may be an
electroacoustic (speaker) or a piezoelectric membrane. Typically, the function
of the transducer is to change the volume of the cavity periodically. In some
cases, the flow is considered incompressibleTrávníček et al. (2015), but in
others the neck of the resonator limits the outflow, leading to pressure
accumulation in the cavityAmitay and Cannelle (2006). Through the lens of
vibration theory, a speaker has equivalent spring and damping constants, and
the cone and the dust cap have inertia. Conversely, the air within the cavity
of the Helmholtz resonator acts as a spring, and the fluid on its neck acts as
a massRayleigh (1916). Here, the fluid friction with the neck transforms the
viscosity and the nozzle geometry to a damping constant. Both the transducer
and the resonator have elements of damped harmonic oscillators (DHOs). In
standard vibration theory the mass, the spring, and damping constants are
simplified to the resonant frequency and the damping ratio. Changes in the
resonant frequency have been studied along the operational range of SJ
actuators. For example, SJ actuators exhibit at least one resonant
frequencyChiatto et al. (2017); Ahmed and Bangash (2009); Alvi and Cattafesta
III (2010). But since there are at least two elements that can be simplified
as DHOs, some SJ actuators have exhibited two resonant frequenciesSharma
(2007); Gallas et al. (2003).

Table 1: Cavity maximum dimensions (diameter and length), and maximum
frequency of operation of different SJ actuators in the literature.

Reference $D_{\textrm{max}}$ [mm] $L_{\textrm{max}}$ [mm] $V_{\textrm{max}}$
[mm3] $f_{max}$ [Hz] Chiatto et al.Chiatto et al. (2018) (2018) 6 6
1.7$\times$102 3000 Krishnan & MohseniKrishnan and Mohseni (2009) (2009) 40
3.4 4.3$\times$103 3000 Chiatto et al.Chiatto et al. (2019) (2019) 42 7.5
1.0$\times$104 3000 Kordík & Trávníc̆ekKordík and Trávníček (2017) (2017) 60
10 2.8$\times$104 170 Kordík & Trávníc̆ekKordík and Trávníček (2018) (2018)
60.8 10 2.9$\times$104 150 Förner & PolifkeFörner and Polifke (2015) (2015)
50.8 25.4 5.1$\times$104 1000 Persoons & O’DonovanPersoons and O’Donovan
(2007) (2007) 44 75 1.1$\times$105 1000 Gil & StrzelczykGil and Strzelczyk
(2016) (2016) 50 60 1.2$\times$105 600 Ingard & IsingIngård and Ising (1967)
(1967) 95 70 5.0$\times$105 370 Smyk et al.Smyk et al. (2020) (2020) 164 30
6.3$\times$105 200 This study 127 200 8.3$\times$106 300

Lumped Element Models, a common strategy to study SJ actuators, collapse the
three dimensions of these devices to a single point, which is only possible if
the pressure and velocity fields are accurately portrayed by their averaged
valuesChiatto et al. (2018). For a speaker, this assumption signifies that the
cone displacement occurs in a single direction. Although this is not true in
some vibration modesChiatto et al. (2017); Chiang and Huang (2015), this
assumption is valid if the dominant length scales are much larger than the
device dimensions. For an SJ, the dominant length scale is the sound
wavelength. Typical driving frequencies for SJ actuators are on the order of
1000 Hz. If we assume the speed of sound is 340 m/s, then the wavelength is
about 300 mm. Most cavities are considerably smaller than this length, as seen
in Table 1. The dimensions of SJ actuators, reported in Table 1, are for
several applications, ranging from efficiency characterizationGil and
Strzelczyk (2016) to plasma ejectionChiatto et al. (2018). Other applications
include the study of the flow field driven by a piezoelectric actuatorKrishnan
and Mohseni (2009), and the coupling of formation criteria with the lumped
element modelsChiatto et al. (2019). Table 1 also includes a study of acoustic
dissipation via numeric simulations in a cylindrical cavityFörner and Polifke
(2015) and its corresponding experimental validationPersoons and O’Donovan
(2007). We also added the actuator dimensions of the classic work about
acoustic non-linearities of an orifice of Ingård and IsingIngård and Ising
(1967) (1967). Table 1 is organized from small to large cavity volumes and
shows that even the largest cavity is more than one order of magnitude smaller
than the device used in this study. The cavity length is much larger than that
of other devices as well. One reason for the size discrepancy is that SJ
actuators are commonly designed for the aerospace industry and the fabrication
of small devicesChiatto et al. (2017). The weight limitation of the industry
and the small size have fueled an interest in designing SJ actuators as small
as possible, which resulted in compact cavities as a rule of thumb. Thus, not
only the assumption needed to simplify the SJ actuators is typically met, but
the frequencies required to form room modes within the cavity of SJ actuators
are typically higher than the operational range. Thus, Lumped Element (LE)
models correctly assumed SJ actuators are small compared with acoustic
wavelengthsCrocker (1998).

In any room in a house or building, some sound frequencies are amplified more
than others due to their geometry. In a rectangular, closed, and empty room,
the amplified frequencies correspond to the formation of standing waves due to
reflection. The fundamental mode develops when the sound wavelength is at
least half the distance between parallel walls. The process repeats for higher
harmonics. These wavelengths are called room modes and are determined
correctly by linear room acousticsKuttruff (2016). Standard room acoustics is
used extensively to predict the frequencies amplified in rooms where human
activities occur, such as concert halls and recording studios. The larger the
room, the smaller the frequency amplified in it. In this context, a small room
measures at least two meters, and the frequencies are as low as a hundred
HertzKleiner and Tichy (2014). Smaller rooms, such as cavities of SJ
actuators, have not been characterized using standard room acoustics because
human activities do not occur in them. However, in micro- and nano-particle
manipulation, it has been reported that an acoustic field groups particles by
size in different cavity locations. Such is the case of Qiao et al. Qiao et
al. (2015) (2015), where aerosol drops are organized by size due to the
formation of standing waves within a cavity of similar size, 5.4$\times$106
mm3. In their investigation, the resonant frequencies corresponded to standard
room acoustics modesKuttruff (2016). While other studies might be
relevantReyes et al. (2018), the operating frequencies used in aerosol
manipulation are much larger than the hearing threshold.

In this work, we revisited the SJ actuator designed and built by Boullosa &
Orduña-BustamanteBoullosa and Orduña-Bustamante (2010) (2010) to float
polystyrene spheres using airflow. The cavity volume of this actuator is one
order of magnitude larger than the largest of other SJ devices, and a non-
linear room mode develops for a lower frequency than the first room mode.
Unlike linear room modes, this one is asymmetric, reaching its peak during the
tenths of Hertz and diminishing during hundreds of Hertz. We performed
frequency analysis on each element to compare this asymmetric room mode with
the effect of previously reported DHOs. We show that this asymmetric room mode
dominates the SJ dynamics above the two resonating frequencies that have been
reported and found conclusive evidence that the cavity dynamics introduce a
third DHO that determines the performance of the SJ actuator.

##  II Experimental setup

###  II.1 Materials and instruments

The SJ actuator, depicted in Figure 1, consists of an acrylic box with two
inner cavities. The walls have a thickness of 10 mm. The speaker is a Dayton
DC160-4 of 6.5” in diameter, and it is attached to the dividing inner wall.
The speaker can be driven at a maximum power of 120 W and has a nominal
impedance of 4 $\Omega$. The inner driver consists of an analog sine wave
generator and an audio amplifier that drives the speaker at frequencies up to
300 Hz and voltages up to 10 V. On the back of the speaker, there is a closed
cavity where the driver is enclosed. The back wall of the cavity is removable
and remains sealed in all the experiments presented here. However, we verified
the results are the same when the wall is removed. The front cavity has a
cubic volume of 20 cm in length. Considering an additional volume due to the
recess of the cone, we estimated the cavity volume as 8250 cm3. The front
cavity is open to the environment through a nylamid nozzle tightly fitted on
top of its top wall. However, in many of our tests, the neck of the SJ
actuator remained blocked, decoupling the Helmholtz resonator to study the
cavity dynamics independently. For the last set of experiments, we used two
nozzles with 8 and 10 mm of diameter, both with a length of 40 mm. Here, we
were interested in the effect of coupling two Helmholtz resonators. The inner
edges of the neck have a chamfer radius of about 1 mm.

Figure 1: (Color online)(a) Sketch of the SJ actuator used in this work. (a)
Side view. The elements of the Synthetic Jet device are highlighted using
broken squares. (b) Top View. The obstacle positions within the cavity.

The magnitude of the electrical impedance was measured for low voltages using
an Audio Test System (DATS V2 from Dayton Audio). This test system requires
the use of a computer and installing the software. We detached the speaker
from the SJ actuator to perform this test.

An accelerometer is attached to the speaker dust cap to measure the RMS
displacement as a function of the frequency and voltage. The accelerometer
attached is an Endevco 27A11 with its signal conditioner Endevco 4416B. The
mass of the accelerometer is 1 g. The combined mass of the coil, the cone, and
the dust cap is 23 g, as measured in a similar and disassembled speaker. We
captured the signal using an oscilloscope (Tektronix MDO3032). The
acceleration was calibrated using a vibration calibrator Brüel & Kjær type
4294. This calibrator produces a reference RMS acceleration amplitude of 10
m/s2 and a reference RMS displacement amplitude of 10 $\mu$m at 1000 rad/s
($\approx$159 Hz) of frequency. This calibration resulted in a sensitivity of
10 m/s2 every 102 mV.

###  II.2 Calibration methods and measurements

Two frequency characterizations were performed in anechoic conditions: (1) the
speaker and (2) the cavity responses. For the first test, we placed the
speaker in the anechoic chamber of the Laboratorio de Acústica y Vibraciones
of the Instituto de Ciencias Aplicadas y Tecnología. Then, we performed
measurements using a probe microphone at 30 cm above the speaker. As a second
test, we fixed the speaker to the SJ actuator and placed the device inside the
anechoic chamber. We inserted the microphone-probe tip within the cavity of
the SJ actuator through the small hole seen in Figure 1. We performed both
characterizations by driving the speaker with a sine sweep at constant
voltages ranging from 10 mV up to 10 V. This is called frequency analysis in
room acoustics. To perform the frequency analysis, we bypassed the inner
driver using the digital function generator and the Yamaha audio amplifier to
be able to test sine sweeps in a single run. We used a signal analyzer to
estimate the maximum sound pressure spectrum. We focused on the pressure
measurements when the cavity is blocked, but we also tested 8- and 10-mm
nozzles.

The probe microphone used was a Brüel & Kjær type 4182 connected to a power
supply Brüel & Kjær type 2807. The signal from the microphone was calibrated
before each experimental session using a sound level calibrator Brüel & Kjær
type 4230. This calibrator produces a signal at 1000 Hz and an RMS sound
pressure of 1 Pa. A digital function generator, Siglent SDG1032X, generated
sine sweeps at a constant voltage. The duration of each sweep cycle was 300
seconds. An audio amplifier, Yamaha P2500S (YA), modulated the signal fed to
the speaker. We used a multimeter, Fluke 289, to monitor the input voltage.
The sound pressure curves were estimated using the maximum peak mode. The
signal analyzer was a Stanford Research SR780.

We visualized the external flow using Planar Laser-induced Fluorescence
(PLIF). A fog generator (Dantec Dynamics, Rosco Mini-V) attached to the cavity
injected smoke through one of the walls via a needle valve. The typical size
of the particle tracers is 1 to 3 $\mu$m. A laser sheet, wavelength of 448-462
nm (blue), illuminated a cross-section of the flow downstream of the nozzle. A
laser engraver module modified in-house of 20 W produced the laser sheet. A
collimator lens focused the laser on a cylindrical lens. The minimum sheet
thickness we reached with this arrangement was 1 mm. We designed and built a
Pulsed-Width-Modulation (PWM) circuit to drive the laser module at a nominal
frequency of 20 kHz. We recorded the experiments using a high-speed camera
perpendicular to the flow: Chronos 2.1-HD from Krontech. The typical frame
rate was 4229 frames per second (fps). We used a 50 mm CCTV lens and a 30 mm
infinity tube to adjust the working distance and record the images. We tracked
the front of each vortex in a sequence of recorded frames to estimate its
velocity. Using linear regression, we fitted the positions as a function of
time and estimated the average velocity of the vortex at about 4 cm downstream
of the neck. Such an average relies on the mean position of the vortex through
a few frames, being more reliable than instantaneous velocity fields.

In the following section, we report on the speaker frequency response, the
cavity, and the effect of two nozzles coupled to the SJ actuator. Then, we
compare the behavior of each of these elements to measurements of the speed of
the jets and vortices expelled. Finally, in the discussion section, we
elaborate arguments that may enlighten the mechanism underlying the dynamics.

##  III Results

Here, we describe the main elements of the SJ actuator that may significantly
affect its dynamics. The speaker is, by itself, a complex device that operates
on three different levels: electric, mechanical, and acoustic. In this
section, we tested the response of each level independently. The impedance
curve illustrates the electric response of the speaker. Curves of the cone RMS
displacement as a function of the frequency depict the mechanical response of
the speaker. The sound pressure level reports the acoustic response of the
speaker. We repeated the measurement within the cavity of the SJ actuator.
Then, we measured the velocity of the vortices expelled from the nozzle for
different conditions.

###  III.1 Speaker characterization

A speaker is a device that converts alternate current to a mechanical
vibration through a coil in relative movement with a permanent magnet. The
mechanical vibration transforms into sound via the compression of the adjacent
air. The coil is associated with an electrical resistance and a reactance that
characterizes its electrical response. To estimate the speaker response, we
tested it independently by detaching it from the SJ actuator. We used an Audio
Test System (DATS V2 from Dayton Audio) to estimate the impedance curve at low
voltages and some Thiele-Small parameters. The magnitude of the impedance as a
function of the frequency in Figure 2(a) shows a spike at 32 Hz. The impedance
magnitude reaches its nominal value for lower and higher frequencies, up to
300 Hz. The inset of Figure 2(a) shows that the impedance magnitude increases
continuously for frequencies higher than 300 Hz. Since the frequency axis of
the inset is on a logarithmic scale, the increase in the impedance is gradual.

Figure 2: (Color online)(a) Magnitude of the electric impedance as a function
of the frequency measured with an Audio Test System. (b) RMS displacement of
the speaker as a function of the frequency for different voltages.

Here, we investigated if the impedance peak may infiltrate the mechanical or
acoustic responses of the speaker. Besides the electrical, the mechanical
components could be also conceived as elements of a DHO, i.e., the suspension
ring and the spider result in elastic and damping constants, and the coil, the
cone, and the dust cap have equivalent mass. Such elements may lead to
vibration modes on their own. Thus, we attached the accelerometer to the
center of the dust cap to measure its displacement as a function of the
driving frequency for different voltages. We performed this test with the
speaker fixed to a breadboard in an open room. Figure 2(b) shows the Root Mean
Square (RMS) displacement as a function of the frequency for different
voltages. The inset of Figure 2(b) shows the displacement in logarithmic
scales. Although the behavior in the main figure seems to fit an Arrhenius-
type equation, we observe in the inset that the exponent is not constant. The
speaker exhibits larger displacements for lower frequencies. This test reveals
that, for lower frequencies, no other DHO is present in the mechanical
response, including the magnitude of the electrical impedance. The higher
displacements occur on the lower frequency band on the non-audible spectrum.

Figure 3: (Color online) Speaker frequency response. Maximum sound pressure
level during a sine sweep measured in an anechoic chamber from 1 to 1600 Hz
for different voltages.

To test the speaker’s acoustic response, we placed it alone in the anechoic
chamber while reproducing the sine sweep and recording with the microphone.
Figure 3 shows a sound pressure level average as a function of the frequency
for a range of logarithmically spaced voltages. Despite some response below
the hearing range, the sound pressure level arises steadily when frequencies
reach about 20 Hz. The curve-minima appears between 24 and 32 Hz for the
voltages studied, in good agreement with the displacement measurements that
peak for low frequencies. For frequencies above the minimum, the sound
pressure level increases continuously and gradually until reaching a maximum
at 1498 $\pm$ 6 [Hz]. The curves shown in Figure 3 are typical of a flat
response of a real speaker. In anechoic conditions at the highest voltage, the
speaker produces a maximum sound pressure of 115 dB at 30 cm from the dust
cap.

###  III.2 Frequency response of the cavity

Figure 4: (Color online) Cavity frequency response drive by the speaker.
Maximum sound pressure level during a sine sweep measured in an anechoic
chamber from 1 to 1600 Hz for different voltages.

The cavity of the SJ actuator is cubic with an inner side of 20 cm. The
speaker attached to the wall seals the cavity, and air may only go in and out
through the neck of the resonator. We tested the frequency response of the
cavity by connecting the speaker to the audio amplifier while running a sine
sweep from 1 to 1600 Hz at different voltages. The cavity response shown in
Figure 4 shows that the cavity exhibits characteristics similar to rooms. For
example, a spike consistently appears at 817 $\pm$ 3 Hz for the voltages
studied. Such a spike corresponds to the first normal room mode, predicted at

$f=\frac{c_{0}}{\lambda_{1}}=\frac{c_{0}}{2L},$ (1)

where $c_{0}$ = 333 m/s is the local speed of sound estimated in the lab,
$\lambda_{1}=2L=40$ cm is the first wavelength that fits the cavity of the SJ
actuator. Such a small calculation leads to a peak frequency of 833 Hz. One of
the reasons the calculation does not conform to the measurement is the cone
recess, meaning there is a longer average distance between the dust cap and
the wall than the length of the wall. We estimated that an additional length
of 37 mm would be enough to match both frequencies, consistent with the length
associated with the recessed volume. Two more spikes appear at 1232 and 1504
Hz and correspond to the first tangential and oblique modes. Calculations to
estimate these modes can be performed similarly, as in Equation 1. Despite
peaks of similar magnitude, the third symmetric mode exhibits a higher level
than the first for the lowest voltage, which inverts as voltage increases. We
speculate that the maximum frequency of the speaker is related to this result.
Linear room acoustics predicts the optimum frequency of the three symmetric
spikes. However, Figure 3 shows another spike at lower frequencies. This spike
is not symmetric as it develops, reaches its maximum on tenths of Hertz, and
gradually reduces during hundreds of Hertz with its maximum at 103 $\pm$ 2 Hz.
Such behavior results in a shape more similar to a hump than a spike, and its
maximum value is about one order of magnitude ($\approx 20$ dB) below the
dominating room mode. The maximum frequency of this spike does not correspond
to any standard room mode.

At low sound intensities, the behavior of the room mode depicted in Figure 4
was strongly affected by placing an obstacle in different positions within the
cavity. The obstacle was a cylindrical bar of 5.5 cm in diameter and 9 mm in
height. Figure 5 shows the result of this test, where we measured the sound
pressure level within the cavity as a function of the frequency for an
intensity of 100 mV. The color curves correspond to different positions of an
obstacle within the cavity. The position labels in Figure 1(b) correspond to
the floor wall. The maroon curve corresponds to the sound pressure level
without an obstacle and is the highest of the curves. The maximum intensity
decreased by more than 10 dB when we positioned the bar in Corner 2.

Similarly, the intensity decreased more than 7 dB when we placed the obstacle
in Corner 1. When we placed the obstacle on the edges, the center, and Corner
3, there was a slum decrease in the intensity but a significant change in the
dispersion. The curves for Corner 1 and 2 show a sharper peak in the maximum
frequency of the spectrum. This behavior was not observed for higher voltages,
as all the curves took the shape of the maroon curve in Figure 5.

Figure 5: (Color online) Sound pressure curves when an obstacle is placed in
different positions of the cavity at 100 mV.

###  III.3 Helmholtz Resonance

Once the nozzle is open to the ambient, a Helmholtz resonator couples to the
SJ actuator. Here, we tested two nozzles with the same length and different
diameters. Coupling another resonator may add to the complexity of the system.
However, these DHOs couple in specific frequencies that can be estimated
theoretically as

$f_{He}=\frac{c_{0}}{2\pi}\sqrt{\frac{A_{n}}{V_{0}l_{n}}},$ (2)

where $A_{n}$ is the cross-sectional area of the nozzle, $V_{0}$ is the cavity
volume of the SJ actuator, and $l_{n}$ is the length of the nozzle. This
calculation leads to Helmholtz frequencies of 20 and 25 Hz for the 8 and 10 mm
nozzle diameters, respectively. Figure 6(a & b) shows the sound pressure level
spectra when the SJ actuator is open using the 8- and 10-mm nozzles. These
curves are, in essence, the same as the ones in Figure 4 up to 400 Hz and the
maroon curve without obstacle in Figure 5. Despite the similarity, there are
differences with the Helmholtz frequencies calculated previously. Figure 6 (c
& d) shows a close-up of the regions marked with black rectangles in Figure 6
(a & b). Figure 6 (c) shows the effect of an 8 mm nozzle on the lower (left)
and the higher (right) voltages. Figure 6 (d) shows the result of a 10 mm
nozzle on the lower (left) and the higher (right) voltages. In this figure,
Helmholtz resonators add a minor effect to the spectrum, even for the lower
voltages. The effect becomes negligible for voltages higher than 100 mV,
compared with the spectra measured when blocked. Thus, it becomes clear that
the nonlinear room mode developing within the cavity is the dominant effect
leading to the sound pressure level.

Figure 6: (Color online) Sound pressure level measured using frequency
analysis of the cavity open with a nozzle of (a & c) 8 mm and (b & d) 10 mm.
(c & d) show a close-up for the lowest and the highest voltages.

Commonly, the velocity of SJ mirrors the sound pressure curves. To test the
effect of the nonlinear room mode on the jet, we measured the jet velocity for
different conditions of voltage and frequency. To perform the velocity
measurements, we visualized the SJ flow at the exit of the SJ actuator. Figure
7 shows a panel where the frequency is increased from left to right while
keeping the voltage constant at 8 V. The arrow in Figure 7(c) depicts the
velocity direction. Figure 8 shows systematic velocity measurements as a
function of the frequency for voltages from 500 mV to 10 V. Smaller voltages
did not produce flow. The vertical lines of Figure 8 represent error bars
estimated as the standard deviation of the velocity of at least ten
measurements. We observed that the velocity increases with frequency, reaching
a maximum at intermediate frequencies. For lower voltages, 500 mV and 1 V, the
maximum velocity is located at about 20 Hz. However, for higher voltages, the
maximum velocity shifts toward higher frequencies. This behavior is similar
when we compare two nozzles of different diameters.

Figure 7: (Color online) Jets and vortex rings visualized at 8 V.
Visualization experiments at driving frequencies of (a) 40 Hz, (b) 80 Hz, (c)
110 Hz, (d) 130 Hz, (e) 140 Hz, (f) 150 Hz.

While a change of a diameter from 8 to 10 mm represents only a 25% increase,
the area of the cross-section is increased by more than 50%. The increased
area balances with the jet velocities decrease. The maximum velocities
measured in Figure 8 (a) are around 17 m/s, while such a velocity is reduced
to 12 m/s when the nozzle has a larger aperture, implying the flow rate keeps
constant.

Figure 8: (Color online) Velocity measurements of vortices expelled from the
the SJ actuator at different frequencies and voltages. Two nozzles were used:
(a). The diameter is 8 mm. (b). The diameter is 10 mm.

Each velocity curve in Figure 8 shows a maximum velocity at a given frequency.
The optimum frequency is voltage-dependent, shifting toward higher frequencies
for higher voltages. In Figure 8(a), the maximum frequency for 500 mV is about
20 Hz, and it gets closer to 50 Hz for 10 V. We observed a similar behavior
for the lower voltages in Figure 8(b). The velocity does not exhibit a clear
peak for higher voltages, and the maximum velocities are observed for 30 to 50
Hz if accounting for the error bar..

##  IV Discussion

The reason for the frequency shift from 20 to about 50 Hz documented in Figure
8 is not self-explanatory from the data presented up to this point. When
conditions lead to SJ formation, the maximum frequency at lower voltages is
misleadingly close to the Helmholtz frequency of the resonator. Despite using
two nozzles of different diameters, their resonating frequencies coincided
with the minimum response of the speaker. This condition limited the
contributions of the Helmholtz resonance from the beginning. Helmholtz
resonance only impacts the sound pressure for the lower voltages, 10 and 100
mV. Our SJ actuator only expells vortices when the voltage is at least 500 mV.
We did not measure any significant impact of the Helmholtz resonance on the
sound pressure at 500 mV nor at 1 V. The SJ regime of this device is far from
the regime where the Helmholtz resonance plays any role in the pressurized
cavity.

Furthermore, even at the two lower voltages, the Helmholtz resonance only
represents a small addition to the sound pressure within the cavity, as seen
in Figures 6(c & d), meaning the asymmetric mode is overwhelmingly dominant
even here. However, one more criterion plays a role in the performance of the
SJ actuator: the vortex formation criteria. This criterion depends on two
dimensionless parameters, the Stokes and the Reynolds number. The Stokes
number is defined as

$\textrm{S}=d\sqrt{\frac{2\pi f}{\nu}},$ (3)

where $d$ is the diameter of the nozzle, $f$ is the driving frequency, and
$\nu$ is the kinematic viscosity. For an SJ driven by a sinusoidal waveform, a
modified Reynolds number is defined as

$\textrm{Re}=\frac{2U_{v}d}{\nu}.$ (4)

The numerical prefactor on the definition of the Reynolds number corresponds
to the velocity integration during the time of discharge, i.e., half the cycle
of a sine wave. The formation criteriaBroučková et al. (2016) state that
vortices may only form if $\textrm{Re}>0.16\;\textrm{S}^{2}$. Figure 9 depicts
this space parameter, where the purple area corresponds to the region where
vortices do not form. The stroke length may be introduced as an aid to
interpreting Figure 9. The stroke length is defined as $L=U_{v}/f$, and the
dimensionless stroke length is defined as in the context of SJs. Thus, the
solid black line in Figure 9 corresponds not only to the formation criteria
but may also be interpreted as a dimensionless stroke length of $L/d=0.5$. In
this device, vortices do not stop forming at a constant dimensionless stroke
length. Despite curve $\textrm{Re}=0.16\;\textrm{S}^{2}$ representing a
transition from flow to no-flow, crossing through different curves above
signifies a propulsion decrease. When the 8 mm nozzle is attached, voltages of
500 mV and 1 V vortices stop forming at $L/d<6$, as seen in Figure 9(a). But,
for higher voltages, vortices still form for shorter stoke lengths,
approaching the formation criteria. Figure 9(b) shows a similar behavior for
the nozzle of 10 mm. The main difference between Figure 9(a & b) is that
vortices still form at stroke lengths smaller than $L/d<6$ for the lower
voltages. The red lines in Figure 9 correspond to the same boundary given by
the dimensionless stroke length. However, for the nozzle of 10 mm, vortices
form between the black and the red line, and the same does not occur for the 8
mm nozzle. This behavior is consistent with the transitional area reported by
Zhou et al.Zhou et al. (2009) (2009), although not defined on the same
boundaries.

Figure 9: (Color online) Stokes vs Reynolds transition diagram. (a). The
diameter is (a) 8 mm and (b) 10 mm.

Now, we elaborate on the conditions needed for an SJ actuator to produce flow
at a single dimensionless stroke length. First, we may assume that for a
laminar regime, the pressure drop between the cavity and the environment
scales linearly with speed during discharge, as $U_{v}\sim\Delta P$. The
complete scaling is known as the Hagen-Poiseuille equation. This scaling is
valid for Reynolds numbers up to 3000. As the flow transitions to turbulent,
the linear relation gradually shifts to $U_{v}\sim\Delta P^{1/2}$. The rate of
such transition depends on the relative roughness of the pipe. Fixing the
dimensionless stroke length in the Stokes-Reynolds chart means the relation
between velocity and frequency should be constant, translating to $U_{v}\sim
f$ for a given nozzle diameter. Thus, to preserve the stroke length, the
pressure should increase linearly with the frequency, $f\sim\Delta P$, when
the flow is laminar. This dependency is weaker when a turbulent jet fully
develops, $f\sim\Delta P^{1/2}$.

Unlike previous investigations on SJ actuators, the external flow is driven by
the pressure accumulated within the cavity by the non-linear resonant mode.
Hereunder, we present an argument to scale the maximum pressure growth
produced by the resonant mode to drive the flow at higher frequencies. Figure
10(a) shows the sound pressure as a function of the frequency for different
voltages in the log-log scale. The pressure accumulation in the cavity due to
the development of the resonant mode during low frequencies may be simplified
using a power law of the form

$\Delta P=af^{n},$ (5)

where $a$ is a prefactor that is a function of the voltage and $n$ is the
exponent. The solid lines in Figure 10(a) correspond to a power law fit. The
prefactor $a$ varies continuously as a function of the voltage, as seen in
Table 2. However, the exponent is relatively the same, with a mean value of
$\bar{n}=2.46\pm 0.03$. For simplicity, we may assume that the exponent is
$n=5/2$. According to the power-law-fit, the frequency scales with the
pressure as $f\sim\Delta P^{2/5}$. This scaling has a smaller exponent than
needed to keep a constant dimensionless stroke length, even for a fully
developed turbulent jet. Even at its highest growth rate, the pressure
developed in the cavity by the resonant mode is insufficient to maintain a
constant stroke, thus resulting in the jet weakening and ultimately inhibiting
its formation. The experimental points in Figure 9 exhibit the insufficient
growth rate as a lower exponent than a fixed dimensionless stroke length. The
direction change at higher Stokes numbers corresponds to the exponent decay of
the pressure curve. Lower voltages result in shorter initial stroke lengths,
being more sensitive to the stroke length drop for higher frequencies. The
combination of the asymmetric resonant mode and the formation criteria results
in the apparent frequency shift seen in Figure 9. But, this behavior is not
related to the Helmholtz resonance.

We suppose that the cubic shape plays a role in the development of the
asymmetric room mode since the three length scales related to the size are
equally important, which contrasts with the formation of resonant modes on
closed and narrow ducts, where the length of the duct is much more significant
than its cross-section.

Figure 10: (Color online). Sound pressure vs frequency in log-log scale. The solid lines correspond to the a power-law fit of the growth rate to estimate the exponent. Table 2: Pre-factors and exponents fitted to the maximum growth rate of the asymmetric room mode to a power law. V [mV] |  $a$[Pa] | $n[1]$
---|---|---
1$\times$101 | 1.0737$\times$10-6 | 2.3919
1$\times$102 | 9.2743$\times$10-6 | 2.4406
5$\times$102 | 4.4698$\times$10-5 | 2.4539
1$\times$103 | 8.5289$\times$10-5 | 2.4685
3$\times$103 | 2.6217$\times$10-4 | 2.4750
6$\times$103 | 5.0276$\times$10-4 | 2.4876
8$\times$103 | 6.8815$\times$10-4 | 2.4732
1$\times$104 | 8.7928$\times$10-4 | 2.4592

##  V Conclusions

We studied an SJ actuator driven by a speaker whose cavity operates in a
compressible regime. The different elements of the SJ actuator are studied
independently. Despite being a peak on its electric impedance, this does not
affect its mechanical or acoustic response. We tested the mechanical response
of the speaker using accelerometry to estimate the RMS displacement at
different voltages. The speaker showed large displacements for low frequencies
that gradually diminished as the frequency increased. We tested the acoustic
response by driving a sinusoidal frequency sweep at a constant voltage and
performing frequency analysis. The speaker exhibited a flat response up to
1600 Hz. The element of the SJ actuator that dominated its dynamics was its
large cavity.

The cavity volume is more than one order of magnitude larger in previous
reports. Such a volume is significantly larger than other cavities. But, it is
small compared with standard rooms. A typical theory implemented on SJ
actuators assumes the wavelengths are much larger than the device. Therefore,
their field properties collapse to their average values, and their elements
are simplified. Such an assumption is unused in standard rooms, where
reflection and diffraction form standing waves that result in room modes. A
nonlinear room mode develops within the cavity of our SJ actuator at very low
frequencies. The spectrum of this room mode is different from others, as its
frequency distribution is significantly asymmetric, and its maximum is one
order of magnitude weaker. Despite this fact, this resonant mode dominates the
dynamics, above the speaker and Helmholtz resonance. It may underlie that
despite the size of the cavity being considerably smaller than the wavelength,
it is not negligible anymore. We suppose the asymmetric room mode results from
a nonlinear effect within the cavity. The room-mode intensity can diminish for
low voltages, and its frequency distribution is focused when adding an
obstacle in different cavity positions. It may point to the formation of nodes
and antinodes, as in standard room acoustics. The velocity measurements in
Figure 8 show an apparent resonance shift to higher frequencies as voltage
increases. We showed the shift results from shorter stroke lengths as
frequency increases due to the driving mechanism that produces vortices and
jets. We believe this room mode could be included as part of the design of SJ
actuators to generate a flatter resonant response around resonance than other
devices.

## Acknowledgements

This work was supported by UNAM-PAPIIT IN107024. J.F. Hernández-Sánchez and S.
Sánchez acknowledge the CONAHCYT (Mexico) to finance the project
CF-2023-I-1344. Benjamin Valera Orozco, Antonio Pérez-López, and Ricardo
Dorantes-Escamilla provided technical support for the experimental work. The
authors thank Pablo Rendón for their support and insightful discussions on
this project.

## Data Availability Statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * Mohseni and Mittal (2014) K. Mohseni, R. Mittal, Synthetic jets: fundamentals and applications, CRC Press, 2014.
  * Krishnan and Mohseni (2009) G. Krishnan, K. Mohseni, Axisymmetric synthetic jets: An experimental and theoretical examination, Aiaa Journal 47 (10) (2009) 2273–2283.
  * Glezer and Amitay (2002) A. Glezer, M. Amitay, Synthetic jets, Annual review of fluid mechanics 34 (1) (2002) 503–529.
  * Smith and Glezer (1998) B. L. Smith, A. Glezer, The formation and evolution of synthetic jets, Physics of fluids 10 (9) (1998) 2281–2297.
  * Cater and Soria (2002) J. E. Cater, J. Soria, The evolution of round zero-net-mass-flux jets, Journal of Fluid Mechanics 472 (2002) 167–200.
  * Broučková et al. (2016) Z. Broučková, P. Šafařík, Z. Trávníček, A parameter map of synthetic jet regimes based on the Reynolds and Stokes numbers: Commentary on the article by Rimasauskiene et al., Mechanical Systems and Signal Processing 68 (2016) 620–623.
  * Chiatto et al. (2017) M. Chiatto, F. Capuano, G. Coppola, L. De Luca, LEM characterization of synthetic jet actuators driven by piezoelectric element: A review, Sensors 17 (6) (2017) 1216\.
  * Trávníček et al. (2015) Z. Trávníček, Z. Broučková, J. Kordík, T. Vít, Visualization of synthetic jet formation in air, Journal of Visualization 18 (2015) 595–609.
  * Amitay and Cannelle (2006) M. Amitay, F. Cannelle, Evolution of finite span synthetic jets, Physics of Fluids 18 (5).
  * Rayleigh (1916) L. Rayleigh, The theory of the Helmholtz resonator, Proceedings of the Royal Society of London. Series A, Containing Papers of a Mathematical and Physical Character 92 (638) (1916) 265–275.
  * Ahmed and Bangash (2009) A. Ahmed, Z. Bangash, Experimental investigation of axisymmetric coaxial synthetic jets, Experimental thermal and fluid science 33 (8) (2009) 1142–1148.
  * Alvi and Cattafesta III (2010) F. Alvi, L. Cattafesta III, The art and science of flow control–case studies using flow visualization methods, The European Physical Journal Special Topics 182 (1) (2010) 97–112.
  * Sharma (2007) R. N. Sharma, Fluid dynamics-based analytical model for synthetic jet actuation, AIAA journal 45 (8) (2007) 1841–1847.
  * Gallas et al. (2003) Q. Gallas, R. Holman, T. Nishida, B. Carroll, M. Sheplak, L. Cattafesta, Lumped element modeling of piezoelectric-driven synthetic jet actuators, AIAA journal 41 (2) (2003) 240–247.
  * Chiatto et al. (2018) M. Chiatto, A. Palumbo, L. De Luca, A calibrated lumped element model for the prediction of PSJ actuator efficiency performance, in: Actuators, vol. 7, MDPI, 10, 2018.
  * Chiatto et al. (2019) M. Chiatto, A. Palumbo, L. de Luca, Design approach to predict synthetic jet formation and resonance amplifications, Experimental Thermal and Fluid Science 107 (2019) 79–87.
  * Kordík and Trávníček (2017) J. Kordík, Z. Trávníček, Optimal diameter of nozzles of synthetic jet actuators based on electrodynamic transducers, Experimental Thermal and Fluid Science 86 (2017) 281–294.
  * Kordík and Trávníček (2018) J. Kordík, Z. Trávníček, Novel nozzle shapes for synthetic jet actuators intended to enhance jet momentum flux, in: Actuators, vol. 7, MDPI, 53, 2018.
  * Förner and Polifke (2015) K. Förner, W. Polifke, Aero-acoustic characterization of Helmholtz resonators in the linear regime with system identification, in: Proceedings of the 22nd International Conference on Sound and Vibration, vol. 596, 2015, 2015\.
  * Persoons and O’Donovan (2007) T. Persoons, T. S. O’Donovan, A pressure-based estimate of synthetic jet velocity, Physics of Fluids 19 (12).
  * Gil and Strzelczyk (2016) P. Gil, P. Strzelczyk, Performance and efficiency of loudspeaker driven synthetic jet actuator, Experimental Thermal and Fluid Science 76 (2016) 163–174.
  * Ingård and Ising (1967) U. Ingård, H. Ising, Acoustic nonlinearity of an orifice, The journal of the Acoustical Society of America 42 (1) (1967) 6–17.
  * Smyk et al. (2020) E. Smyk, P. Gil, R. Gałek, Ł. Przeszłowski, Acoustic and flow aspects of novel synthetic jet actuator, in: Actuators, vol. 9, MDPI, 100, 2020.
  * Chiang and Huang (2015) H.-Y. Chiang, Y.-H. Huang, Vibration and sound radiation of an electrostatic speaker based on circular diaphragm, The Journal of the Acoustical Society of America 137 (4) (2015) 1714–1721.
  * Crocker (1998) M. J. Crocker, Handbook of acoustics, John Wiley & Sons, 1998.
  * Kuttruff (2016) H. Kuttruff, Room acoustics, Crc Press, 2016.
  * Kleiner and Tichy (2014) M. Kleiner, J. Tichy, Acoustics of small rooms, CRC Press, 2014\.
  * Qiao et al. (2015) Z. Qiao, Y. Huang, N. Vincenzo, W. Dong, Aerosol manipulation by acoustic tunable phase-control at resonant frequency, Powder technology 281 (2015) 76–82.
  * Reyes et al. (2018) C. Reyes, L. Fu, P. P. Suthanthiraraj, C. E. Owens, C. W. Shields IV, G. P. López, P. Charbonneau, B. J. Wiley, The Limits of Primary Radiation Forces in Bulk Acoustic Standing Waves for Concentrating Nanoparticles, Particle & Particle Systems Characterization 35 (7) (2018) 1700470\.
  * Boullosa and Orduña-Bustamante (2010) R. R. Boullosa, F. Orduña-Bustamante, Acoustic levitation at very low frequencies, Acta Acustica united with Acustica 96 (2) (2010) 376–382.
  * Zhou et al. (2009) J. Zhou, H. Tang, S. Zhong, Vortex roll-up criterion for synthetic jets, AIAA journal 47 (5) (2009) 1252–1262.