# Choosing a Proxy Metric from Past Experiments

Nilesh Tripuraneni
Google DeepMind
<EMAIL_ADDRESS>Lee Richardson
Google
<EMAIL_ADDRESS>Alexander D’Amour
Google DeepMind
<EMAIL_ADDRESS>Jacopo Soriano
Google
<EMAIL_ADDRESS>Steve Yadlowsky
Google DeepMind
<EMAIL_ADDRESS>
###### Abstract

In many randomized experiments, the treatment effect of the long-term metric
(i.e. the primary outcome of interest) is often difficult or infeasible to
measure. Such long-term metrics are often slow to react to changes and
sufficiently noisy they are challenging to faithfully estimate in short-
horizon experiments. A common alternative is to measure several short-term
proxy metrics in the hope they closely track the long-term metric – so they
can be used to effectively guide decision-making in the near-term. We
introduce a new statistical framework to both define and construct an optimal
proxy metric for use in a homogeneous population of randomized experiments.
Our procedure first reduces the construction of an optimal proxy metric in a
given experiment to a portfolio optimization problem which depends on the true
latent treatment effects and noise level of experiment under consideration. We
then denoise the observed treatment effects of the long-term metric and a set
of proxies in a historical corpus of randomized experiments to extract
estimates of the latent treatment effects for use in the optimization problem.
One key insight derived from our approach is that the optimal proxy metric for
a given experiment is not apriori fixed; rather it should depend on the sample
size (or effective noise level) of the randomized experiment for which it is
deployed. To instantiate and evaluate our framework, we employ our methodology
in a large corpus of randomized experiments from an industrial recommendation
system and construct proxy metrics that perform favorably relative to several
baselines.

##  1 Introduction

Randomized controlled trials (RCTs) are the gold standard approach for
measuring the causal effect of an intervention (Hernán and Robins, 2010);
however, designing and analyzing high-quality RCTs requires various
considerations to ensure scientifically robust results. For example, an
experimenter must clearly define the intervention, control, and choose a
primary outcome for the study. In this work, we will assume that the
intervention and control are clearly defined, and consider the problem of
choosing a good primary outcome. A common approach is to choose the primary
outcome to be a key metric which drives downstream decision-making. Such
metrics are critical components in the decision-making pipelines of many
large-scale technology companies (Chen and Fu, 2017; Rachitsky, ) as well as
used to guide policy decisions in economics and medicine (Athey et al., 2019;
Elliott et al., 2015). Unfortunately, direct measurement of such a metric can
be impractical or infeasible. In many cases, they are long-term outcomes
observed with a significant temporal delay, making them slow to move (i.e.
insensitive) in the short term, and inherently noisy. Moreover, they may be
prohibitively expensive to query.

On the other hand, proxy metrics (or surrogates) that are easier to measure or
faster to react are often available to use in lieu of the long-term outcome.
For example in clinical settings, CD4 white-blood cell counts in blood serve
as a surrogate for mortality due to AIDS (Elliott et al., 2015), while in
online experimentation platforms diversity of consumed content serves a proxy
for long-term visitation frequencies (Wang et al., 2022). A significant
literature exists on designing and analyzing proxy metrics and experiments
that use them as a primary outcome. One important question addressed by this
literature is choosing (or combining) proxy metrics to be a good surrogate for
measuring the effect of the intervention on the long-term outcome (Prentice,
1989; Hohnhold et al., 2015; Parast et al., 2017; Athey et al., 2019; Wang et
al., 2022, 2023). To do so, one needs a principled reason for why the measured
treatment effect on the proxy outcome is related to the treatment effect on
the long-term outcome. Frequently, this is done by making _causal_ assumptions
about the relationship between the treatment, proxy outcome, and long-term
outcome (see, e.g., VanderWeele, 2013; Athey et al., 2019; Kallus and Mao,
2022). However, motivated by the unique way that trials are run in technology
product applications, we take a different approach based on statistical
regularity assumptions in a population of experiments, similar to meta-
analytic approaches such as that taken by Elliott et al. (2015).

RCTs performed in technology products are typically referred to as A/B tests.
They are used for a wide variety of applications in the technology industry,
however one of the most common applications is for assessing the effect of a
candidate launch of a new product feature or change on the user’s experience.
If the results of the A/B test suggest that the candidate launch has a
positive effect on the user experience, then it will be deployed to all users.
Depending on the scale of the product and engineering team, many candidate
launches requiring many A/B tests may be required on a regular basis. The
results of these A/B tests on long-term outcomes and proxy metrics are logged,
serving as a history of past candidate launches that we may use to guide the
choice of future proxy metrics to use for decision making.

This perspective has been studied in the technology research literature
previously, for example in Richardson et al. (2023) and Wang et al. (2022), to
develop useful heuristics for choosing a proxy metric for use in future A/B
tests. In this work, we define a precise statistical framework for choosing a
proxy metric based on this historical A/B test data, and develop a method for
optimizing a composite proxy, an affine combination of base proxy metrics,
that can be used as a primary outcome for future A/B tests.

The central contributions of our paper are the following:

  * •

We define a new notion of optimality for a proxy metric we term proxy quality,
for use in a homogeneous population of randomized experiments. Our definition
differs from the existing literature in that it phrases optimality as ensuring
the observed proxy metric closely tracks the unobserved population treatment
effect on the long-term outcome (see Figure 1). Conveniently, our definition
also packages two important considerations for a proxy metric – short-term
sensitivity of the proxy metric and directional alignment with the long-term
outcome – into a single objective (see Equation 2).

  * •

We show this new notion of proxy quality can be used to construct optimal
proxy metrics for new A/B tests via an efficient two-step procedure. The first
step reduces the construction of an optimal weighted combination of base
proxies, that maximize our definition of proxy quality, to a classic portfolio
optimization problem. This optimization problem is a function of the latent
variability of the unobservable population treatment effects and the noise
level of the new experiment under consideration (see Section 2.1.2). We then
use a hierarchical model to denoise the observed treatment effects on the
proxy and long-term outcome in a historical corpus of A/B tests to extract the
variation in the unobserved population treatment effects (see Section 2.2).
The variance estimates of the population TEs are then used as plug-ins to the
aforementioned optimization.

  * •

We highlight the adaptivity of our proxy metric procedure to the inherent
noise level of each experiment for which it will be used. In our framework the
optimal proxy metric for a given experiment is not apriori fixed. Rather it
should depend on the sample size (or effective noise level) of the randomized
experiment for which it is deployed in order to profitably trade-off bias from
disagreement with the long-term outcome and intrinsic variance (see Section
2.1.1 and Figure 3).

  * •

Finally, we instantiate and evaluate our framework on a set of 307 real A/B
tests from an industrial recommendation engine showing how the proxy metrics
we construct can improve decision-making in a large-scale system (see Section
3).

###  1.1 Statistical Framework

Consider a corpus of $K$ randomized experiments (or A/B tests) where the
$i\in\\{1,\ldots,K\\}$-th experiment is of sample size $n_{i}$. In each
experiment, there is a specific intervention that has some population
treatment effect (TE) that we denote $\Delta_{i}$.111This may be an average TE
(ATE) or relative ATE, however, we will not emphasize the differences between
these two as they are immaterial for this work. In the experiment, we measure
an estimated TE $\hat{\Delta}_{i}$ on the subset of the population included in
our experiment. Note that the entire population may be included in the
experiment, but $\hat{\Delta}_{i}$ remains a random quantity, given
$\Delta_{i}$, because of the random assignment of treatments in the
experiment.

To differentiate between the TE on the long-term outcome and the proxy
metrics, we will attach a superscript $N$ as in $\Delta_{i}^{N}$ or
$\hat{\Delta}_{i}^{N}$ for the long-term outcome (we use $N$ since these are
sometimes referred to as north star metrics), and a superscript $P$ as in
$\mathbf{\Delta}_{i}^{P}$ and $\hat{\bf{\Delta}}_{i}^{P}$ for the proxy
metrics. Note that there may be multiple base proxy metrics, so
$\mathbf{\Delta}_{i}^{P}\in\mathbb{R}^{d}$. Throughout, we assume that $n_{i}$
is large enough, and that the experimental design is sufficiently regular such
that conditional on $(\Delta_{i}^{N},\mathbf{\Delta}_{i}^{P})$,
$(\hat{\Delta}_{i}^{N},\mathbf{\hat{\Delta}}_{i}^{P})$ is well-approximated by
a Normal distribution centered around the population TE222Note that in our
framework, the notion of proxy quality in Section 2.1.1 and Section 2.1.2 only
relies on low-order moments and doesn’t explicitly require Gaussianity but the
estimation procedure in Section 2.2 makes explicit use of this structure. (due
to the central limit theorem); and that the joint (within-experiment)
covariance of $(\hat{\Delta}_{i}^{N},\mathbf{\hat{\Delta}}_{i}^{P})$, denoted
as $\bm{\Xi}$, has a good estimator, denoted as $\hat{\bm{\Xi}}$. Our
discussion is agnostic to the precise estimator of the TEs and their
covariances. We only require black-box access to their values. For the
historical corpus of $K$ randomized experiments, we assume that these
triplicates of measurements
$\\{(\hat{\Delta}_{i}^{N},\hat{\bm{\Delta}}_{i}^{P},\hat{\bm{\Xi}}_{i})\\}_{i=1}^{K}$
are available.

Our goal in this paper is to revisit the proxy metric problem: the selection
of short-term proxy metrics (or a weighted combination thereof) that track the
long-term outcome in a new $K+1$st experiment where measurements of the long-
term outcome are unavailable, but measurements of short-term proxy metrics
are. In order to develop a statistical framework to construct proxies in a new
experiment, we leverage a meta-analytic approach to model the relationship
between different experiments. To this end, we assume the population TEs for
each experiment are drawn i.i.d. from a common joint distribution
$\mathcal{D}$,

$\displaystyle\begin{pmatrix}\Delta^{N}_{i}\\\
\mathbf{\Delta}_{i}^{P}\end{pmatrix}$ $\displaystyle\sim\mathcal{D}(\cdot)\
\text{ i.i.d. }$ (1)

supported over $\mathbb{R}^{d+1}$. We acknowledge this assumption is strong
and not suitable for all applications. However, in our motivating application
of interest – studying a corpus of AB tests from a large technology company –
historical intuition and various tests do not provide significant evidence
this assumption is violated. The approach of placing a distributional prior on
the population ATE in similar settings of A/B testing at large-scale
technology companies (Deng and Shi, 2016; Deng, ) as well as other meta-
analytic studies of RCTs (Elliott et al., 2015; Elliott, 2023) has also been
advocated for as a useful assumption in prior work.

##  2 Methods

With the above setting in place we first define the measure of quality of a
proxy metric which relates the estimated TE on the short-term proxies to the
population TE on the long-term outcome for a new experiment. Subsequently, we
show how the relevant latent parameters contained in the definition of proxy
quality can be efficiently estimated via a hierarchical model.

###  2.1 Optimal Proxy Metrics

In order to judge the quality of a proxy metric we first define a new notion
of utility for a proxy metric. Our key insight is that in a new
experiment333In the following since we assume all the experiments are i.i.d.
we suppress index notation on this arbitary experiment drawn from
$\mathcal{D}(\cdot)$. where
$\begin{pmatrix}\Delta^{N}&\mathbf{\Delta}^{P}\end{pmatrix}^{\top}\sim\mathcal{D}(\cdot)$,
the observed TEs of the short-term proxies $\hat{\mathbf{\Delta}}^{P}$ should
closely track the latent TE of the long-term outcome (see Figure 1).444Note in
a new experiment the estimated treatment effect on the long-term outcome
$\hat{\Delta}^{N}$ may be unavailable. This is because decisions that are
intended to move $\Delta^{N}$ will be made on the basis of
$\hat{\mathbf{\Delta}}^{P}$. Thus, we would like these quantities to be well-
correlated.

Figure 1: In a new experiment, we view the observed TEs as being generated
from their corresponding (unobserved) latent values by a noisy channel which
adds independent, mean-zero experimental noise with covariance $\bm{\Xi}$. In
this new experiment the noisy, observed long-term outcome ${\hat{\Delta}^{N}}$
is inaccessible. We seek to find noisy proxy metrics whose TEs closely track
the population TEs on the long-term outcome.

####  2.1.1 Proxy Quality of a Single Short-Term Metric in a New Experiment

For simplicity, we first consider the case when the vector-valued sequence of
proxies reduces to a single scalar proxy. In order to capture the intuition
that the short-term estimated proxy TE, $\hat{\Delta}^{P}$ should track the
population long-term outcome TE, $\Delta^{N}$ we define the proxy quality as
the correlation between the aforementioned quantities. The correlation is a
simple and natural measure which captures the predictive relationship between
the proxy metric and long-term outcome. Under stronger conditions in Section
2.2, we also argue that optimizing for this measure of proxy quality minimizes
the probability of a (signed) decision error or surrogate paradox.

In our setting we consider the case where the estimated TEs are unbiased
estimators of their underlying latent population quantities – so we can
parameterize $\hat{\Delta}^{P}=\Delta^{P}+\sqrt{\Xi^{PP}}\cdot\epsilon$, where
$\epsilon$ is an independent random zero-mean, unit-variance random variable.
Hence we can define and simplify the proxy quality as:

$\displaystyle\mathrm{corr}(\Delta^{N},\hat{\Delta}^{P})=\frac{\mathrm{Cov}(\Delta^{N},\Delta^{P})}{\sqrt{\mathrm{Var}(\Delta^{N})(\mathrm{Var}(\Delta^{P})+\Xi^{P,P})}}=\frac{\mathrm{corr}(\Delta^{P},\Delta^{N})}{\sqrt{1+\frac{\Xi^{P,P}}{\mathrm{Var}(\Delta^{P})}}}.$
(2)

In our setting, the definition of proxy quality decomposes the predictive
relationship between the estimated proxy TE and population long-term outcome
TE into latent predictive correlation $\mathrm{corr}(\Delta^{P},\Delta^{N})$ –
a property of the distribution $\mathcal{D}(\cdot)$ – and an effective inverse
signal-to-noise ratio $\Xi^{PP}/\mathrm{Var}(\Delta^{P})$ – which is also a
function of the noise level of the experiment. We now make several comments on
the aforementioned quantities.

  * •

The latent predictive correlation $\mathrm{corr}(\Delta^{P},\Delta^{N})$
tracks the alignment between the population proxy metric TE and the population
long-term outcome TE. In particular, this correlation is reflective of the
intrinsic predictive quality of a fixed proxy metric. This quantity is not
easily accessible since we do not directly observe data sampled from
$\mathcal{D}(\cdot)$. We return to the issue of estimating such quantities in
Section 2.2.

  * •

The quantity $\Xi^{PP}/\mathrm{Var}(\Delta^{P})$ computes the ratio of the
within-experiment noise in the estimated proxy metric TE—due to fluctuations
across experimental units and treatment assignments—to the latent variation of
the population proxy metric TE across experiments. For the former quantity we
expect $\Xi^{PP}$ to depend on the size of the randomized experiment in
consideration (i.e. $\Xi^{PP}\sim\frac{1}{n}$, where $n$ is the sample size of
the experiment), since it is a variance over independent treatment units.
Meanwhile $\mathrm{Var}(\Delta^{P})$ captures how easily the population proxy
metric TE moves in the experiment population $\mathcal{D}(\cdot)$. In a (large
enough) given experiment, $\Xi^{PP}$ is easily estimated by $\hat{\Xi}^{PP}$
using the sample covariance estimator. Meanwhile, $\mathrm{Var}(\Delta^{P})$
is difficult to measure directly, just like
$\mathrm{corr}(\Delta^{P},\Delta^{N})$. We later show how to use a
hierarchical model to estimate these parameters (see Section 2.2). Finally, it
is worth noting this ratio term in the denominator is also closely related to
a formal definition of metric sensitivity which appears in the A/B testing
literature (Deng, ; Richardson et al., 2023).

Together the numerator and denominator in Equation 2 trade off two (often
competing) desiderata into a single objective: the numerator favors alignment
with the population TE on the long-term outcome while the denominator
downweights this by the signal-to-noise ratio of the proxy metric.555This
tradeoff between directional alignment of the proxy metric/long-term outcome
and sensitivity of the proxy metric is further discussed in (Richardson et
al., 2023). One unique property of this proxy quality measure is that, given a
set of base proxies, the “optimal” single proxy is not an intrinsic property
of the proxy metric or distribution of treatment effects captured by
$\mathcal{D}(\cdot)$. Rather, it also depends on the experimental design.
Specifically, it is a function of the experiment sample size $n$, which will
control the size of $\Xi^{PP}$. This behavior represents a form of bias-
variance trade-off. For large sample sizes, as $\Xi^{PP}\to 0$, Equation 2
will favor less biased metrics whose population-level TEs are aligned with the
long-term outcome (i.e. the numerator is large). Meanwhile, for small sample
sizes where $\Xi^{PP}$ is large, Equation 2 will favor less noisy metrics with
a high signal-to-noise ratio so the denominator is small.

####  2.1.2 Composite Proxy Quality in a New Experiment

The previous discussion on assessing the quality of a single proxy metric
captures many of the important features behind our approach. However, in
practice, we are often not restricted to picking a single proxy metric to
approximate the long-term outcome. Rather, we are free to construct a
composite proxy metric which is a convex combination of the TEs of a set of
base proxy metrics to best predict the effect on the long-term outcome.

In our framework, the natural extension to the vector-valued setting takes a
convex combination of the proxies
$\mathrm{corr}(\mathbf{w}^{\top}\mathbf{\hat{\Delta}}^{P},\Delta^{N})$ for a
normalized weight vector $\mathbf{w}$, instead of restricting ourselves a
single proxy. However, beyond just defining the quality of a weighted proxy
metric, we can also optimize for the quality of this weighted sum of base
proxy metrics:

$\displaystyle\max_{\mathbf{w}\in\mathbb{R}^{d}}\mathrm{corr}(\mathbf{w}^{\top}\mathbf{\hat{\Delta}}^{P},\Delta^{N}):\mathbf{1}^{\top}\mathbf{w}=1,\mathbf{w}\geq
0.$ (3)

Again, the estimated TEs are unbiased estimators of their underlying latent
population quantities, so we can parameterize
$\mathbf{\hat{\Delta}}^{P}=\mathbf{\Delta}^{P}+\left(\bm{\Xi}^{PP}\right)^{1/2}\cdot\bm{\epsilon}$,
where $\bm{\epsilon}$ is an independent random zero-mean, identity-covariance
random vector. So the objective expands to:

$\displaystyle\max_{\mathbf{w}\in\mathbb{R}^{d}}\frac{1}{\sqrt{\mathrm{Var}(\Delta^{N})}}\frac{\mathbf{w}^{\top}\mathrm{Cov}(\Delta^{N},\mathbf{\Delta}^{P})}{\sqrt{\mathbf{w}^{\top}(\mathrm{Cov}(\mathbf{\Delta}^{P},\mathbf{\Delta}^{P})+\bm{\Xi}^{P,P})\mathbf{w}}}:\mathbf{1}^{\top}\mathbf{w}=1,\mathbf{w}\geq
0.$ (4)

Essentially all considerations noted in the previous section translate to the
vector-valued setting mutatis mutandis. In particular, the numerator in
Equation 4 captures the alignment between the true latent weighted proxy and
the population long-term outcome, while the denominator downweights the
numerator by the effective noise in each particular experiment. As before we
expect $\bm{\Xi}^{P,P}\sim\frac{1}{n}$ with the sample size, $n$, of the
experiment. Hence, the optimal weights for a given experiment will adapt to
the noise level (or equivalently sample size) of the experiment run.

The formulation in Equation 4 raises the question of how to efficiently
compute $\mathbf{w}$. Indeed, at first glance the optimization problem as
phrased in Equation 4 is non-convex. Fortunately, the objective in Equation 4
(up to a constant pre-factor) maps exactly onto the Sharpe ratio (or reward-
to-volatility ratio) maximization problem often encountered in portfolio
optimization (Sharpe, 1966, 1998). As is well-known in the optimization
literature, the program in Equation 4 can be converted to an equivalent
(convex) quadratic programming problem which can be efficiently solved
(Cornuejols and Tütüncü, 2006, Section 8.2). We briefly detail this
equivalence explicitly in Appendix B.

The portfolio perspective also lends an additional interpretation to the
objective in Equation 4. If we analogize each specific proxy metric as an
asset to invest in, then $\mathrm{Cov}(\Delta^{N},\mathbf{\Delta}^{P})$ is the
returns vectors of our assets, and
$\mathrm{Cov}(\mathbf{\Delta}^{P},\mathbf{\Delta}^{P})+\bm{\Xi}^{P,P}$ is
their effective covariance–so
$\mathbf{w}^{\top}(\mathrm{Cov}(\mathbf{\Delta}^{P},\mathbf{\Delta}^{P})+\bm{\Xi}^{P,P})\mathbf{w}$
captures the risk of our portfolio of proxies. Just as in portfolio
optimization, where two highly-correlated assets should not be over-invested
in, if two proxy metrics are both strongly aligned with the long-term outcome,
but are themselves very correlated, the objective in Equation 4 will not
assign high weights to both of them.

###  2.2 Estimation of Latent Parameters via a Hierarchical Model

As the last piece of our framework, we finally turn to the question of
obtaining estimates of the unobservable latent quantities arising in Equation
4. While $\bm{\Xi}^{PP}$ is easily estimated from the within-experiment sample
covariance $\hat{\bm{\Xi}}^{PP}$, the quantities
$\mathrm{Cov}(\Delta^{N},\mathbf{\Delta}^{P})$,
$\mathrm{Cov}(\mathbf{\Delta}^{P},\mathbf{\Delta}^{P})$ are tied to the
latent, unobservable population TEs of the proxy metrics and long-term
outcome.

In order to gain a handle on these quantities, we take a meta-analytic
approach which combines two key pieces. First, as in our previous discussion,
we require the setting described in Equation 1 – that is we assume the true
population TEs are drawn i.i.d. from a common joint distribution. While only
this assumption was needed for our previous discussion, we now introduce
additional parametric structure in the form of an explicit generative model to
allow for tractable estimation of the parameters
$\mathrm{Cov}(\Delta^{N},\mathbf{\Delta}^{P})$ and
$\mathrm{Cov}(\mathbf{\Delta}^{P},\mathbf{\Delta}^{P})$. Second, we assume
access to a pool of homogeneous RCTs for which unbiased estimates of the TE on
the short-term proxy metrics and long-term outcome are available (i.e.
$(\hat{\Delta}^{N}_{i},\mathbf{\hat{\Delta}}_{i}^{P})$ for
$i\in\\{1,\ldots,K\\}$). With these two pieces we can construct a hierarchical
(or linear mixed) model to estimate the latent parameters:

$\displaystyle\begin{pmatrix}\Delta^{N}_{i}\\\
\mathbf{\Delta}^{P}_{i}\end{pmatrix}$
$\displaystyle\sim\mathrm{MVN}\left(\begin{pmatrix}\mu^{N}\\\
\bm{\mu}^{P}\end{pmatrix},\bm{\Lambda}\right)$ (5)
$\displaystyle\begin{pmatrix}\hat{\Delta}^{N}_{i}\\\
\hat{\mathbf{\Delta}}^{P}_{i}\end{pmatrix}\mid\begin{pmatrix}\Delta^{N}_{i}\\\
\mathbf{\Delta}^{P}_{i}\end{pmatrix}$
$\displaystyle\sim\mathrm{MVN}\left(\begin{pmatrix}\Delta^{N}_{i}\\\
\mathbf{\Delta}^{P}_{i}\end{pmatrix},\bm{\Xi}_{i}\right)$ (6)

which due to the joint Gaussianity we can marginalize as:

$\begin{pmatrix}\hat{\Delta}^{N}_{i}\\\
\hat{\mathbf{\Delta}}^{P}_{i}\end{pmatrix}\sim\mathrm{MVN}\left(\begin{pmatrix}\mu^{N}\\\
\bm{\mu}^{P}\end{pmatrix},\bm{\Sigma}_{i}:=\bm{\Lambda}+\bm{\Xi}_{i}\right),\quad\forall
i\in[K].$ (7)

We use the notation $\bm{\Lambda}$ to capture the latent covariance of the
joint distribution $\mathcal{D}(\cdot)$ which we parameterize by a
multivariate normal (i.e. MVN). So in our case,
$\mathrm{Cov}(\Delta^{N},\mathbf{\Delta}^{P})=\bm{\Lambda}^{NP}$ and
$\mathrm{Cov}(\mathbf{\Delta}^{P},\mathbf{\Delta}^{P})=\bm{\Lambda}^{PP}$ for
$\begin{pmatrix}\Delta^{N}&\mathbf{\Delta}^{P}\end{pmatrix}^{\top}\sim\mathcal{D}(\cdot)$.
Moreover, for the purposes of inference we simply use the plug-in estimate
$\hat{\bm{\Xi}}\approx\bm{\Xi}$ which is routinely done in similar
hierarchical modeling approaches (Gelman et al., 1995). While using
multivariate normality in Equation 5 is an assumption (albeit we believe
reasonable in our case), it is not essential to the content of our results.
Our proxy quality definition relies only on inferring low-order moments of
$\mathcal{D}(\cdot)$ for which this parametric structure is convenient. The
second approximation that the noisy TEs are multivariate normal around their
true latent values (Equation 6) is well-justified by the central limit theorem
in our case, since the experiments we consider all have at least $O(10^{5})$
treatment units. Since inference in this model is not closed-form, we
implement the aforementioned generative model in the open-source probabilistic
programming language NumPyro (Phan et al., 2019) to extract the latent
parameters. Additional details on the inference procedure are deferred to
Appendix C.

Figure 2 provides an example where the inference procedure is used to extract
the latent population variation in a synthetically generated dataset. Although
this example is synthetic (and exaggerated), empirically in our corpus we
observe many base proxy metrics with correlations to the long-term outcome of
$\sim 0.6$ in their experimental noise matrix $\hat{\bm{\Xi}}$. Thus, the
example shows a case where the raw correlation may provide an over-optimistic
estimate of the underlying alignment between a proxy metric and long-term
outcome. The denoising model we fit helps mitigate the impact of correlated
within-experiment noise in our setting. We schematically detail the end-to-end
algorithm which composes the denoising model fit and portfolio optimization to
construct a proxy for a new A/B test in Algorithm 1.

(a) Latent Population TEs of Proxy Metric / long-term Outcome

(b) Estimated TEs of Proxy Metric / long-term Outcome

(c) Estimated TEs with Raw / Estimated Covariance from Hierarchical Model

Figure 2: The panel visualizes the denoising effect of fitting a hierarchical
model to raw TEs to uncover their latent variation on synthetic data. We
generate 1500 synthetic datapoints sampled from the model in Equation 5 with
one proxy metric. Each datapoint represents a synthetic TE measurement from a
single A/B test. We use parameters with $\begin{pmatrix}\mu^{N}\\\
\mu^{P}\end{pmatrix}=\begin{pmatrix}0.0\\\ 0.0\end{pmatrix}$,
$\bm{\Lambda}=.01\cdot\begin{pmatrix}1&0.2\\\ 0.2&1\end{pmatrix}$ to generate
data in Figure 2(a). We add Gaussian noise with covariance
$\bm{\Xi}=.02\cdot\begin{pmatrix}1&0.7\\\ 0.7&1\end{pmatrix}$ to them in
Figure 2(b). Finally, we fit the generative model to the observed data in
Equation 7 using the within-experiment covariances $\bm{\Xi}$ in Figure 2(c).
Figure 2(c) illustrates how the hierarchical model denoises the raw observed
TEs to disentangle the latent variation in the population from the
experimental noise in each synthetic A/B test.

Lastly, with the generative model in Equations 5, LABEL: and 6 in place we can
provide an alternative interpretation of our definition of composite proxy
quality. Under the condition that $\begin{pmatrix}\mu^{N}\\\
\bm{\mu}^{P}\end{pmatrix}=\mathbf{0}$666This condition may not be true in all
applications but is approximately satisfied in our dataset with all metrics
having the ratio of their global mean to global standard deviation being
bounded by $0.1$ but often being even less., the probability of a signed
alignment (or equivalently the complement of a surrogate paradox Elliott et
al. (2015)) can be simplified too,

$\displaystyle\mathbb{P}(\Delta^{N}>0,\mathbf{w}^{\top}\mathbf{\hat{\Delta}^{P}}>0)=\mathbb{P}(\Delta^{N}<0,\mathbf{w}^{\top}\mathbf{\hat{\Delta}^{P}}<0)=\frac{1}{4}+\frac{\sin^{-1}(\rho)}{2\pi}$
(8)

for
$\rho=\frac{1}{\sqrt{\Lambda^{NN}}}\frac{\mathbf{w}^{\top}\bm{\Lambda}^{NP}}{\sqrt{\mathbf{w}^{\top}(\bm{\Lambda}^{PP}+\bm{\Xi}^{PP})\mathbf{w}}}$.
This computation relies on the joint Gaussianity of the model and centering
condition to explicitly compute the alignment probability. Given $\rho$ is our
definition of the vector-valued proxy quality and the inverse-sine function is
monotone increasing, optimizing the proxy quality over $\mathbf{w}$ can be
interpreted as minimizing the probability of a signed decision error in this
setting.

Algorithm 1 Composite Proxy Algorithm

0:
$\\{(\hat{\Delta}_{i}^{N},\hat{\bm{\Delta}}_{i}^{P},\hat{\bm{\Xi}}_{i})\\}_{i=1}^{K}$
(TE and Noise Estimates from Historical Tests), $\hat{\bm{\Xi}}_{K+1}^{PP}$
(Noise Estimate for New Test).

1: $\bm{\mu},\bm{\Lambda}\leftarrow$
HM($\\{(\hat{\Delta}_{i}^{N},\hat{\bm{\Delta}}_{i}^{P},\hat{\bm{\Xi}}_{i})\\}_{i=1}^{K}$).
Fit model (e.g. using Numpyro) defined in Equation 7.

2: $\mathbf{w}\leftarrow$ Sharpe
Ratio($\bm{\Lambda},\hat{\bm{\Xi}}_{K+1}^{PP}$). Solve convex program in
Equation 10 (e.g. using CVXPY).

2: $\mathbf{w}$, $\mathbf{w}^{\top}\hat{\bm{\Delta}}_{K+1}^{P}$ (Proxy Weights
and Composite Proxy for New Experiment).

##  3 Results

In this section, we turn to evaluating the performance of our composite proxy
procedure against several baselines. As raw proxy metrics to consider in our
evaluations we use a small set of 3 hand-selected proxy metrics which capture
different properties domain experts believe are relevant to long-term user
satisfaction in our setting. We first highlight a unique feature of our proxy
procedure – its adaptivity to the sample size of the experiment for which it
will be applied. We then perform a comparison of our new proxy procedure
against the raw proxy metrics and a baseline procedure (Richardson et al.,
2023) appearing in the literature.

###  3.1 Proxy Quality and Sample Size Dependence

One unique feature of our procedure is its adaptivity to the noise level (or
effectively sample size of the experiment); recall in Equation 4 the optimal
weights will depend on the latent parameters which are inferred from the pool
of homogeneous RCTs on which they are fit, but also the experiment noise
estimate $\hat{\bm{\Xi}}$ which depends on the new A/B test it is to be used
for. While in practice one could recompute a proxy metric depending on the
aposteriori results of each A/B test (so $\hat{\bm{\Xi}}$ is known), it is
also desirable to be able to fit a proxy for each A/B test apriori, without
knowledge of its results. To do so, we found that in our application,
$\bm{\Xi}^{PP}_{i}$ could be estimated with reasonable accuracy purely on the
basis of historical data of other A/B tests in our population of experiments
by postulating a scaling of the form
$\bm{\Xi}^{PP}_{i}=\bm{\Xi}^{PP}_{\mathrm{ref}}/n_{i}$. Here the reference
matrix $\bm{\Xi}^{PP}_{\mathrm{ref}}$ can be thought of as the within
experiment variance of an A/B test in the population with one sample. The
ansatz $\bm{\Xi}^{PP}_{i}=\bm{\Xi}^{PP}_{\mathrm{ref}}/n_{i}$, combines two
observations. The first is that the variance of a TE estimate decays as
$\sim\frac{1}{n_{i}}$ in the number of treatment units $n_{i}$, which is
immediate from the independence of treatment units. However, the second is
that the constant prefactor in the variance $\bm{\Xi}^{PP}_{\mathrm{ref}}$ is
approximately the same across different A/B tests in our corpus. The reference
matrix $\bm{\Xi}^{PP}_{\mathrm{ref}}$ can then be estimated as a weighted
average of $\hat{\bm{\Xi}}_{i}$ from the corpus. Additional details and
verification of these hypotheses are provided in Appendix A. The upshot of
this approach is that the computation of the optimal weights in Equation 4 for
a new A/B test can then be done using only the sample size ($n_{i}$) of this
new experiment (i.e. before the new experiment is “run”).

To understand the dependence of our new composite proxies weighting on the
experiment sample size, we fit the latent parameters $\bm{\Lambda}$ (from the
hierarchical model in Equations 5, LABEL: and 6) and
$\bm{\Xi}^{PP}_{\mathrm{ref}}$ on the entire corpus for the results in Figure
3. We then use the scaling $\bm{\Xi}=\bm{\Xi}^{PP}_{\mathrm{ref}}/n$ to
estimate the optimal weights of our new composite proxy from Equation 4 for
different sample sizes $n$ for a hypothetic new A/B test. Figure 3 shows how
as the sample size increases the new composite proxy smoothly increases its
weighting on raw metrics which are noisier but more strongly correlated with
the long term outcome. Moreover, while the Auxiliary Metric 3 is an
intuitively reasonable metric, its value as determined by the measure of proxy
quality is dominated by a mixture of the other two components.

Figure 3: The optimal weighting dependence on sample size for our new
composite proxy, represents a bias-variance trade-off. For large sample sizes
the weighting favors potentially noisier metrics that are more aligned with
the long term outcome. However, for smaller sample sizes the optimal weighting
backs off to metrics which are less noisy but also less aligned to the long
term outcome.

###  3.2 Held-out Evaluation of Proxy Procedures

The primary difficulty of this evaluation is that in TE estimation there is
the lack of ground-truth “labels” of the treatment effect (i.e. in our
framework the population latent TEs such as $\Delta^{N}$ are never observed).
However, in our setting we do have access to a large corpus of 307 A/B tests
as noted earlier. Hence, we use held-out/cross-validated evaluations of
certain criterion which depend on the noisy metrics aggregated over an
evaluation set, to gauge the performance of proxy metrics fit on a training
set.

We consider several relevant criteria for performance which have been used in
the literature. Two important measures which appear in (Richardson et al.,
2023) are the proxy score and sensitivity. To define the criteria, recall that
a TE metric is often used to make a downstream decision by thresholding its
t-statistic,
$\mathrm{tstat}=\frac{\hat{\Delta}}{\sqrt{\mathrm{Var}(\hat{\Delta})}}$ as
$\mathrm{tstat}>2\to+\text{ (positive)}$, $-2<\mathrm{tstat}<2\to 0\text{
(neutral)}$, and $\mathrm{tstat}<-2\to-\text{ (negative)}$. Given a corpus of
A/B tests we can then compute the decisions induced by a short-term proxy
metric, the decisions induced by the long-term outcome and check the number of
A/B tests for which they align. After normalization, the number of detections
(both metrics decisions are positive or negative) minus the number of mistakes
(one metric decision is positive while the other is negative) defines the
proxy score. Similarly, for a short-term proxy metric we can compute its
sensitivity – which is the number of times it triggers a statistically
significant decision by being positive or negative. Loosely speaking, these
two criterion function like the notions of precision and recall in information
retrieval. Ideally, a short-term metric would maximize both quantities by
being sensitive and triggering often (so as to not miss any A/B tests where
the TE for the long-term outcome is significant) but not over triggering and
leading to many false positives (or negatives). Additional details on these
metrics are provided in the Appendix D777It is worth noting these measures are
still imperfect in the sense that comparisons are made against the decisions
induced by the noisy estimate of the long-term outcome TE not the latent
population long-term outcome TE. This is further complicated by the fact that
we empirically find that experimental noise in the A/B tests is correlated
between short-term proxies and the long-term outcome (i.e. the phenomena
detailed in Figure 1). In fact, this phenomena provides partial motivation for
our definition of denoised proxy quality.. As another measure of performance,
we also compute and report our definition of our composite proxy quality for a
given composite proxy’s learned weights.

We use the same set of raw proxy metrics as before in our evaluation. As a
procedure to compare our methodology against, we use the baseline of
optimizing the convex combination of these 3 metrics to optimize the
aforementioned proxy score which is detailed in (Richardson et al., 2023). In
each case we use stratified 4-fold cross-validation (CV) to compute the
weights for each procedure on a training subset of the corpus and evaluate the
metrics on the held-out set by computing the aforementioned evaluation scores.
The baseline proxy method and base proxy metrics each learn a fixed set of
weights888the base proxy metrics simply place all their weight on themselves.
depending on the training fold, which is applied to each A/B test in the
evaluation fold. For our procedure we use the ansatz
$\bm{\Xi}^{PP}_{i}=\bm{\Xi}^{PP}_{\mathrm{ref}}/n_{i}$ mentioned in the
previous setting to calculate $\bm{\Xi}^{PP}_{\mathrm{ref}}$ as a simple
weighted average from data only in the training fold of our CV split. The
optimal proxy metric for each A/B test in the test fold can then be refit
using only the sample size $n_{i}$ of that A/B test. This strategy has the
additional benefit of enforcing strict separation of the data in train/test
set folds in our CV split.

Metric | Sensitivity | Proxy Score | Proxy Quality
---|---|---|---
New Composite Proxy | 0.181 | 0.666 | 0.302
Baseline Composite Proxy | 0.182 | 0.611 | 0.279
Auxiliary Metric 1 | 0.062 | 0.611 | 0.174
Auxiliary Metric 2 | 0.368 | 0.222 | 0.258
Auxiliary Metric 3 | 0.166 | 0.104 | 0.030
Table 1: Comparison of our new composite proxy against a baseline method, and
their constituent base proxies alongst several criterion which are computed on
a held-out evaluation. Our new composite proxy performs favorably across all
measures – notably achieving the highest proxy score and proxy quality amongst
all considered. All evaluation criteria are bounded in $[0,1]$ and for each
higher is better.

Results for our cross-validated evaluation are displayed in Table 1 across our
corpus of 307 A/B tests. Note that both our new composite proxy and the
baseline composite proxy improve significantly in the proxy score and proxy
quality over the raw metrics without sacrificing unduly on sensitivity
relative to Auxiliary Metric 2. Moreover, the new composite proxy achieves not
only the highest proxy quality but also proxy score despite not explicitly
optimizing for proxy score on the training set. We believe this may be a
feature of our new composite proxy whose weights are adaptive to the size of
each experiment in the A/B test (which vary in our corpus from approximately
$O(10^{6})$ to $O(10^{8})$ in size).

##  4 Conclusion

We have presented a framework for both defining and constructing optimal
composite proxy metrics, which are used to approximate the decisions induced
by a difficult-to-measure long-term metric. One key insight from our framework
is that the optimal proxy for a given experiment should depend on the noise
level (or equivalently the size) of that experiment. In the future, more
flexible modeling of the joint latent effect distribution – to accommodate
more structured latent effects could be useful. Similarly, extending our
framework to handle the construction of nonlinear composite proxy metrics is
an interesting avenue for further exploration.

##  Appendix A Within-Experiment Covariance Scaling

As we note before, one interesting feature of our composite proxy quality
procedure is its dependence on the noise level of $\bm{\Xi}$ of the randomized
experiment it will be applied too. Recall in Equation 4 the optimal weights
will depend not only on the latent parameters which are instrinsic properties
of $\mathcal{D}(\cdot)$, but the experiment noise estimate $\hat{\bm{\Xi}}$
which depends on the particular A/B test too which it is applied.

While in practice, one could recompute a composite proxy metric after an A/B
test is run (so $\hat{\bm{\Xi}}$ is known), in many applications it is also
desirable to be able to fit weights for a composite proxy metric before each
A/B test is run. In order to do this we found that we could build a simple
predictive model for the experimental noise level $\bm{\Xi}^{PP}_{i}$ in a
given experiment on the basis of historical data of other A/B tests in our
population of experiments. We did so by making an ansatz of the form
$\bm{\Xi}^{PP}_{i}=\bm{\Xi}^{PP}_{\mathrm{ref}}/n_{i}$, where
$\bm{\Xi}^{PP}_{\mathrm{ref}}$ can be thought of as the within experiment
variance of an A/B test in the population with one sample. This ansatz follows
from two facts. The first is that the variance of a TE estimate decays as
$\sim\frac{1}{n_{i}}$ in the number of treatment units $n_{i}$, which is
immediate from the independence of treatment units. However, the second – that
the constant prefactor in the variance $\bm{\Xi}^{PP}_{\mathrm{ref}}$ is
approximately the same across different A/B tests – is an empirical
observation due to underlying homogeneity in the population of A/B tests. This
approximate homogeneity is evidenced in Figure 4.

(a) Variance of A/B tests for Auxiliary Metric 1

(b) Variance of A/B tests for Auxiliary Metric 2

Figure 4: Both displays show the within-experiment marginal sample variance
(blue dots) for two different metrics computed across 307 different A/B tests
and their corresponding power-law fit (red line). Despite the underlying A/B
tests being different, we found that the variance were reasonably well modeled
by a single inverse-power law with the same constant prefactor over the entire
population.

On the basis of this ansatz for each A/B test,

$\displaystyle\bm{\Xi}_{i}=\frac{\bm{\Xi}_{\mathrm{ref}}}{n_{i}},\quad\\{1,\ldots,K\\}$

we can construct an estimator for the reference constant matrix
$\bm{\Xi}_{\mathrm{ref}}$ as,

$\displaystyle\hat{\bm{\Xi}}_{\mathrm{ref}}=\sum_{i=1}^{K}\gamma_{i}n_{i}\hat{\bm{\Xi}}_{i}$
(9)

for some convex combination of weights $\gamma_{i}$. While we can use an equal
weighting scheme where $\gamma_{i}=\frac{1}{K}$, since the sample variance
estimates $\hat{\bm{\Xi}}_{i}$ themselves are noisy, we instead use a
precision weighted combination of them to reduce variance by taking
$\gamma_{i}\propto n_{i}$.

With this estimate $\hat{\bm{\Xi}}_{\mathrm{ref}}$ in hand, for a new $K+1$st
A/B test with sample size $n_{K+1}$ we can approximates its within-experiment
covariance as,

$\displaystyle\hat{\bm{\Xi}}_{K+1}\approx\frac{\hat{\bm{\Xi}}_{\mathrm{ref}}}{n_{K+1}}$

under the implicit homogeneity assumption we make.

##  Appendix B Sharpe Ratio and Portfolio Optimization

Under the mild condition (which is always satisified for us that at least one
element of $\bm{\Lambda}^{NP}$ is positive), we can transform the Sharpe ratio
optimization objective to an equivalent convex quadratic program:

$\displaystyle\min_{\mathbf{x}\in\mathbb{R}^{d}}\mathbf{x}^{\top}\bm{\Sigma}\mathbf{x}\quad:\quad\mathbf{x}\geq
0,\quad\mathbf{r}^{\top}\mathbf{x}=1.$ (10)

where $\bm{\Sigma}=\bm{\Lambda}^{PP}+\hat{\bm{\Xi}}^{PP}$ and
$\mathbf{r}=\bm{\Lambda}^{NP}$. The solution to the original problem in
Equation 4 can be recovered by normalizing as
$\mathbf{w}=\frac{\mathbf{x}}{\|{\mathbf{x}}\|_{1}}$. The details of this
standard transformation can be found in (Cornuejols and Tütüncü, 2006, Section
8.2), although the original reduction is a generalization of the Charnes-
Cooper transformation (Charnes and Cooper, 1962) which dates back to at least
(Schaible, 1974).

##  Appendix C Inference in Hierarchical Model

In order to extract estimates of the latent parameter we perform full Bayesian
inference over the generative model in Equation 7 using Numpyro Phan et al.
(2019) which uses the NUTS sampler to perform MCMC on the posterior. We found
Bayesian inference to be more stable then estimating the MLE of the model. We
augmented the generative model in Equation 7 with the weak priors:

$\displaystyle\bm{\mu}\sim\mathcal{N}(0,1000\cdot\text{meanscale}\cdot\mathbf{I}_{d+1})$
$\displaystyle\mathbf{s}\sim 3\cdot\textsc{Half-Cauchy}(\text{devscale})$
$\displaystyle\mathbf{C}\sim\textsc{LKJ}(\text{concentration}=1)$
$\displaystyle\bm{\Lambda}=\sqrt{\mathbf{s}}^{\top}\circ\mathbf{C}\circ\sqrt{\mathbf{s}}$

where we use the operator $\circ$ to denote coordinatewise broadcasted
multiplication. Here the vector-valued parameters meanscale, and devscale are
set to match the overall scales of the raw mean and raw covariance of the
corpous A/B tests. We found the overall inferences to be robust to the choice
of scales in the Half-Cauchy prior on the pooled variance parameter and normal
prior mean, which are both weakly-informative. (Gelman, 2006) and (Polson and
Scott, 2012) both argue for the use of the Half-Cauchy prior for the top-level
scale parameter in hierarchical linear models as opposed to the more
traditional use of the Inverse-Wishart prior on both empirical and theoretical
grounds. The choice of the LKJ prior with concentration parameter set to 1 is
essentially a uniform prior over the space of correlation matrices (Gelman et
al., 1995; Lewandowski et al., 2009).

Inference in this model was performed using the default configuration of the
NUTS sampler in Numpyro (Phan et al., 2019). We also found it useful to
initialize the parameters $(\mu^{N},\bm{\mu}^{P})$ and $\bm{\Lambda}$ to the
scales of the raw mean and raw covariance of the corpus A/B tests. We
diagnosed convergence and mixing of the NUTS sampler using standard
diagnostics such as the r-hat statistic (Gelman et al., 1995). In all our
experiments we found the sampler mixed efficiently and we achieved a perfect
r-hat statistic for all parameters of 1.0. For each MCMC run we generated
10000 burn-in samples and 50000 MCMC samples for 4 parallel chains. We used
the posterior means of the samples to extract estimates of $\bm{\Lambda}$ for
use in our proxy quality score.

##  Appendix D Proxy Score and Sensitivity

Here we explain several performance criterion we use for proxy metrics which
are further detailed in the literature (Richardson et al., 2023). In order to
define both quantities, recall that a TE metric is often used to make a
downstream decision by thresholding its t-statistic,
$\mathrm{tstat}=\frac{\hat{\Delta}}{\sqrt{\mathrm{Var}(\hat{\Delta})}}$ as
$\mathrm{tstat}>2\to+\text{ (positive)}$, $-2<\mathrm{tstat}<2\to 0\text{
(neutral)}$, and $\mathrm{tstat}<-2\to-\text{ (negative)}$. The formal
definitions of proxy score and sensitivity are most easily defined in the
context of a contingency table visualized in Figure 5 which takes these
decisions as inputs. The contingency table tabulates the decisions induced by
a particular observed short term proxy metric and the observed long-term north
star metric jointly over 554 A/B tests.

Figure 5: A synthetic contingency table which captures the alignment of the
decisions induced by the t-statistics of the TEs of the north star metric and
a proxy metric.

The green cells in Figure 5 represent cases where the proxy and north star are
both statistically significant and move in the same direction (i.e.
_Detections_). The red cells in Figure 5 again represent cases where the proxy
and north star are both statistically significant, but where proxy and north
star are misaligned (i.e. _Mistakes_). The remaining cells correspond to cases
where at least one of the metrics is not statistically significant. The
relative importance of these cells is more ambiguous.

In this setting, the sensitivity can be defined as:

Metric Sensitivity $\displaystyle=$ $\displaystyle\frac{\text{Number of
experiments where the proxy is is significant}}{\text{Number of total
experiments}}.$

Here the numerator can be obtained by summing over the first and last rows of
the table. The proxy score can be defined as,

Proxy Score $\displaystyle=$
$\displaystyle\frac{\text{Detections}-\text{Mistakes}}{\text{Number of
experiments where the north star is significant}}.$

The denominator here can be obtained by summing over the first column and last
column. The sensitivity metric captures the ability of a metric to detect a
statistically significant effect – which inherently takes into account its
inherent moveability and susceptibility to experimental noise. Given that
north star metrics are often noisy and slow to react in the short-term the
goal of a proxy is to be sensitive.

The proxy score rewards metrics that are both sensitive and directionally
aligned with the north star. Sensitive metrics need only populate the first
and third rows of the contingency table. However, metrics in the first and
third rows can only increase the proxy score if they are in the same direction
as the long-term north star. A similar score, called _Label Agreement_ , has
been used by Dmitriev and Wu (2016).

## References

  * Athey et al. [2019] Susan Athey, Raj Chetty, Guido W Imbens, and Hyunseung Kang.  The surrogate index: Combining short-term proxies to estimate long-term treatment effects more rapidly and precisely.  Working Paper 26463, National Bureau of Economic Research, November 2019\.  URL http://www.nber.org/papers/w26463.
  * Charnes and Cooper [1962] A. Charnes and W. W. Cooper.  Programming with linear fractional functionals.  _Naval Research Logistics Quarterly_ , 9(3-4):181–186, 1962.  doi: https://doi.org/10.1002/nav.3800090303.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/nav.3800090303.
  * Chen and Fu [2017] Albert C. Chen and Xin Fu.  Data + intuition: A hybrid approach to developing product north star metrics.  In _Proceedings of the 26th International Conference on World Wide Web Companion_ , WWW ’17 Companion, page 617–625, Republic and Canton of Geneva, CHE, 2017. International World Wide Web Conferences Steering Committee.  ISBN 9781450349147.  doi: 10.1145/3041021.3054199.  URL https://doi.org/10.1145/3041021.3054199.
  * Cornuejols and Tütüncü [2006] Gerard Cornuejols and Reha Tütüncü.  _Optimization methods in finance_ , volume 5.  Cambridge University Press, 2006.
  * [5] Alex Deng.  Metric Sensitivity Decomposition. Causal Inference and Its Applications in Online Industry.  https://alexdeng.github.io/causal/sensitivity.html#metric-sensitivity-decomposition.  [Online; accessed 21-December-2022].
  * Deng and Shi [2016] Alex Deng and Xiaolin Shi.  Data-driven metric development for online controlled experiments: Seven lessons learned.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 77–86, 2016.
  * Dmitriev and Wu [2016] Pavel Dmitriev and Xian Wu.  Measuring metrics.  In _Proceedings of the 25th ACM international on conference on information and knowledge management_ , pages 429–437, 2016.
  * Elliott [2023] Michael R. Elliott.  Surrogate endpoints in clinical trials.  _Annual Review of Statistics and Its Application_ , 10(1):75–96, 2023.  doi: 10.1146/annurev-statistics-032921-035359.  URL https://doi.org/10.1146/annurev-statistics-032921-035359.
  * Elliott et al. [2015] Michael R Elliott, Anna SC Conlon, Yun Li, Nico Kaciroti, and Jeremy MG Taylor.  Surrogacy marker paradox measures in meta-analytic settings.  _Biostatistics_ , 16(2):400–412, 2015.
  * Gelman [2006] Andrew Gelman.  Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper).  _Bayesian Analysis_ , 1(3):515 – 534, 2006.  doi: 10.1214/06-BA117A.  URL https://doi.org/10.1214/06-BA117A.
  * Gelman et al. [1995] Andrew Gelman, John B Carlin, Hal S Stern, and Donald B Rubin.  _Bayesian data analysis_.  Chapman and Hall/CRC, 1995.
  * Hernán and Robins [2010] Miguel A Hernán and James M Robins.  Causal inference, 2010.
  * Hohnhold et al. [2015] Henning Hohnhold, Deirdre O’Brien, and Diane Tang.  Focus on the long-term: It’s better for users and business.  In _Proceedings 21st Conference on Knowledge Discovery and Data Mining_ , Sydney, Australia, 2015.  URL http://dl.acm.org/citation.cfm?doid=2783258.2788583.
  * Kallus and Mao [2022] Nathan Kallus and Xiaojie Mao.  On the role of surrogates in the efficient estimation of treatment effects with limited outcome data, 2022.
  * Lewandowski et al. [2009] Daniel Lewandowski, Dorota Kurowicka, and Harry Joe.  Generating random correlation matrices based on vines and extended onion method.  _Journal of Multivariate Analysis_ , 100(9):1989–2001, 2009.  ISSN 0047-259X.  doi: https://doi.org/10.1016/j.jmva.2009.04.008.  URL https://www.sciencedirect.com/science/article/pii/S0047259X09000876.
  * Parast et al. [2017] Layla Parast, Tianxi Cai, and Lu Tian.  Evaluating surrogate marker information using censored data.  _Statistics in medicine_ , 36(11):1767–1782, 2017\.
  * Phan et al. [2019] Du Phan, Neeraj Pradhan, and Martin Jankowiak.  Composable effects for flexible and accelerated probabilistic programming in numpyro.  _arXiv preprint arXiv:1912.11554_ , 2019.
  * Polson and Scott [2012] Nicholas G. Polson and James G. Scott.  On the Half-Cauchy Prior for a Global Scale Parameter.  _Bayesian Analysis_ , 7(4):887 – 902, 2012.  doi: 10.1214/12-BA730.  URL https://doi.org/10.1214/12-BA730.
  * Prentice [1989] Ross L. Prentice.  Surrogate endpoints in clinical trials: definition and operational criteria.  _Statistics in medicine_ , 8(4):431–440, 1989\.
  * [20] Lenny Rachitsky.  Choosing Your North Star Metric.  https://future.com/north-star-metrics/.
  * Richardson et al. [2023] Lee Richardson, Alessandro Zito, Dylan Greaves, and Jacopo Soriano.  Pareto optimal proxy metrics.  _arXiv preprint arXiv:2307.01000_ , 2023.
  * Schaible [1974] Siegfried Schaible.  Parameter-free convex equivalent and dual programs of fractional programming problems.  _Zeitschrift für Operations Research_ , 18:187–196, 1974.
  * Sharpe [1966] William F. Sharpe.  Mutual fund performance.  _The Journal of Business_ , 39(1):119–138, 1966\.  ISSN 00219398, 15375374.  URL http://www.jstor.org/stable/2351741.
  * Sharpe [1998] William F Sharpe.  The sharpe ratio.  _Streetwise–the Best of the Journal of Portfolio Management_ , 3:169–85, 1998.
  * VanderWeele [2013] Tyler J. VanderWeele.  Surrogate measures and consistent surrogates.  _Biometrics_ , 69(3):561–569, 2013.  ISSN 0006341X, 15410420.  URL http://www.jstor.org/stable/24538119.
  * Wang et al. [2023] Xuan Wang, Layla Parast, Larry Han, Lu Tian, and Tianxi Cai.  Robust approach to combining multiple markers to improve surrogacy.  _Biometrics_ , 79(2):788–798, 2023.
  * Wang et al. [2022] Yuyan Wang, Mohit Sharma, Can Xu, Sriraj Badam, Qian Sun, Lee Richardson, Lisa Chung, Ed H Chi, and Minmin Chen.  Surrogate for long-term user experience in recommender systems.  In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , pages 4100–4109, 2022.