# Incorporating Auxiliary Variables to Improve the Efficiency of Time-Varying
Treatment Effect Estimation

Jieru Shi, Zhenke Wu, Walter Dempsey
Department of Biostatistics, University of Michigan

###### Abstract

The use of smart devices (e.g., smartphones, smartwatches) and other wearables
for context sensing and delivery of digital interventions to improve health
outcomes has grown significantly in behavioral and psychiatric studies. Micro-
randomized trials (MRTs) are a common experimental design for obtaining data-
driven evidence on mobile health (mHealth) intervention effectiveness where
each individual is repeatedly randomized to receive treatments over numerous
time points. Individual characteristics and the contexts around randomizations
are also collected throughout the study, some may be pre-specified as
moderators when assessing time-varying causal effect moderation. Moreover, we
have access to abundant measurements beyond just the moderators. Our study
aims to leverage this auxiliary information to improve causal estimation and
better understand the intervention effect. Similar problems have been raised
in randomized control trials (RCTs), where extensive literature demonstrates
that baseline covariate information can be incorporated to alleviate chance
imbalances and increase asymptotic efficiency. However, covariate adjustment
in the context of time-varying treatments and repeated measurements, as seen
in MRTs, has not been studied. Recognizing the connection to Neyman
Orthogonality, we address this gap by introducing an intuitive approach to
incorporate auxiliary variables to improve the efficiency of moderated causal
excursion effect estimation. The efficiency gain of our approach is proved
theoretically and demonstrated through simulation studies and an analysis of
data from the Intern Health Study (NeCamp et al., 2020).

Keywords: Causal Inference; Asymptotic Efficiency; Micro-randomized Trials;
Mobile Health; Moderation Effect; Covariate Adjustment.

##  1 Introduction

The use and development of mobile interventions are experiencing rapid growth.
In “just-in-time” mobile interventions (Nahum-Shani et al., 2018), treatments
are provided via mobile devices that are intended to help individuals make
healthy decisions “in the moment” and thus have a proximal, near-term impact
on health outcomes. Micro-randomized trials (MRTs; Klasnja et al. (2015);
Dempsey et al. (2015)) provide data to guide the development of such mobile
interventions (Free et al., 2013), with each participant in an MRT
sequentially randomized to treatment numerous times, at possibly hundreds to
thousands of occasions. The weighted and centered least squares (WCLS) method
(Boruvka et al., 2018) is regarded as the benchmark method used for estimating
moderated causal excursion effects, ensuring a consistent estimate with
asymptotic normality. Estimates of these marginal comparisons are crucial to
domain scientists in deciding whether to include a treatment in an mHealth
intervention package.

In an MRT, measurements of individual characteristics, context, and response
to treatments are collected passively through sensors or actively by self-
report. Some of these measurements may be identified in advance as primary
moderators of interest when assessing time-varying causal effect moderation,
while the others are referred to as _auxiliary variables_. These auxiliary
variables are analogous to baseline covariates used in average treatment
effect (ATE) estimation in randomized controlled trials (RCTs). Extensive
research has demonstrated that incorporating baseline covariates can improve
efficiency for ATE estimation. In particular, Lin (2013) recommended using a
fully interacted linear regression model, and it was shown to be
asymptotically more efficient over both unadjusted and additive models. In
recent years, further generalizations of covariate adjustment have been made
to accommodate more complex randomization schemes (Su and Ding, 2021; Zhao and
Ding, 2024) and to study arbitrary functions of response, such as linear
contrasts, risk ratios, and odds ratios (Ye et al., 2023).

Despite these appealing properties, there is still a knowledge gap between
existing covariate adjustment approaches and applications to causal excursion
effect estimation using data arising from an MRT. More specifically, when the
treatment, response, and moderators are time-varying, it is unclear how to
adjust auxiliary variables to improve the estimation efficiency of the
moderated treatment effect.

In this paper, we make two key contributions to the evaluation of treatment
effects in longitudinal data with time-varying treatments, responses, and
moderators. First, we highlight the crucial role of incorporating auxiliary
variables in estimating time-specific causal effects, and illustrate that
doing so correctly can lead to a global efficiency gain. In addition, we
introduce a general theoretical framework for incorporating auxiliary
variables into the analysis of moderated causal effects. The efficiency gain
of our approach is proved theoretically and demonstrated through simulation
studies and an analysis of data from the Intern Health Study (NeCamp et al.,
2020).

###  1.1 Outline

The rest of the paper proceeds as follows. Section 2 provides a review of
current analytical methods for MRTs, as well as covariate adjustment
techniques used in estimating the ATE within classical RCT contexts. We
specifically summarize our main contributions in Section 2.4 and discuss the
challenges associated with the analysis of sequentially randomized experiments
like MRTs. In Section 3, we propose a general strategy of using auxiliary
variables to improve the efficiency of the time-varying treatment effect
estimation and present the asymptotic properties of the moderated causal
effect estimation. Section 4 presents the main theorem, showing theoretical
improvement in efficiency achieved by proper adjustment of auxiliary
variables. We summarize the statistical properties of the proposed method in
Section 5. Section 6 provides simulation studies that compare various
estimators, with a particular focus on the efficiency gains from using the
proposed approach. Section 7 illustrates the efficiency improvement using our
proposed method with a recent MRT: the Intern Health Study (NeCamp et al.,
2020). The paper concludes with a brief discussion in Section 8. All technical
proofs and additional simulation results are collected in the Supplementary
Materials.

##  2 Preliminaries

###  2.1 Notation

For a given individual $j$, let $A_{t,j}$ denote the treatment at the $t$-th
treatment occasion and $Y_{t+1,j}$ be the subsequent proximal response
$(t=1,\dots,T;j=1,\dots,N)$. Individual $j$’s contextual information at the
$t$-th treatment occasion is represented by $X_{t,j}$, which may contain
summaries of previous context, treatments, or response measurements. For
example, before each treatment occasion, the individual might report their
current mood status, then the vector $X_{t,j}$ could contain this measurement
or, with previous measurements, variation or change in mood. Hereafter, we may
omit the subscript $j$ when discussing the associated random variables. The
overbar is used to denote a sequence of random variables or realized values up
to a specific treatment occasion. For example, $\bar{A}_{t}=(A_{1},...,A_{t}$)
denotes the sequence of treatment up to and including decision time $t$.
Information accrued up to treatment occasion $t$ is represented by the history
$H_{t}=(\bar{A}_{t-1},\bar{X}_{t})$. For simplicity, the main results in this
paper assume a binary treatment $A_{t}\in\\{0,1\\}$ and it is randomized
according to a known sequence of randomization probabilities that may depend
on the complete observed history, denoted
$\\{p_{t}(A_{t}|H_{t})\\}_{t=1}^{T}$. In the following, random variables or
vectors are denoted with uppercase letters; lowercase letters denote their
realized values.

We assume that the longitudinal data are independent and identically
distributed across $N$ individuals. Note that this assumption would be
violated, if, for example, some of the treatments are used to enhance social
support between individuals in the study (Liao et al., 2016). The following
section defines the “causal excursion effect” estimand. Then we express the
causal estimand in terms of the observed data and provide causal assumptions
sufficient for these expressions.

###  2.2 Existing Inferential Methods Review

Many treatments are designed to influence an individual in the short term or
proximally in time (Heron and Smyth, 2010). To answer questions related to the
causal effect of time-varying treatments on the proximal response, we focus on
a class of estimands referred to as “causal excursion effects”. In particular,
a causal excursion effect is a function of the decision point $t$ and a set of
moderators $S_{t}$, marginalizing over all other observed and unobserved
variables (Boruvka et al., 2018; Qian et al., 2021). We provide formal
definitions below using potential outcomes (Rubin, 1978; Robins, 1986).

Let $Y_{t+1}(\bar{a}_{t-1})$ denote the potential outcome for the proximal
response under treatment sequence $\bar{a}_{t-1}$. Let $S_{t}(\bar{a}_{t-1})$
denote the potential outcome for a time-varying effect moderator which is a
deterministic function of the potential history up to time $t$,
$H_{t}(\bar{a}_{t-1})$. We consider the setting in which the potential
outcomes are i.i.d over individuals according to a distribution $\mathcal{P}$.
The causal excursion effect is

$\beta_{\bf p}(t;s)=\mathbb{E}_{{\bf
p}}\left[Y_{t+1}\left(\bar{A}_{t-1},A_{t}=1\right)-Y_{t+1}\left(\bar{A}_{t-1},A_{t}=0\right)\,|\,S_{t}(\bar{A}_{t-1})=s\right].$
(1)

Equation (1) is defined with respect to a reference distribution $\bf p$,
i.e., the distribution of treatments
$\bar{A}_{t-1}:=\\{A_{1},A_{2},\dots,A_{t-1}\\}$. We follow common practice in
observational mobile health studies where analyses such as GEEs (Liang and
Zeger, 1986) are conducted marginally over $\bf p$. To express the causal
excursion effect $\beta_{\bf p}(t;s)$ in terms of the observed data, we assume
positivity, consistency, and sequential ignorability (Robins, 1994, 1997):

######  Assumption 2.1 (Identification).

  * •

Consistency:
$\\{Y_{t+1,j}(\bar{A}_{t}),X_{t,j}(\bar{A}_{t-1}),A_{t,j}(\bar{A}_{t-1})\\}=\\{Y_{t+1,j},X_{t,j},A_{t,j}\\}$,
for each $t\leq T$ and $j$, i.e., observed values equal the corresponding
potential outcomes;

  * •

Positivity: if the joint density $\mathbb{P}\\{H_{t}=h_{t}\\}$ is greater than
zero, then $\mathbb{P}(A_{t}=a\,|\,H_{t}=h_{t})>0$, $a=0,1$;

  * •

Sequential ignorability: For each $t\leq T$, the potential outcomes
$\\{Y_{t+1}(\bar{a}_{t}),X_{t+1}(\bar{a}_{t}),A_{t+1}(\bar{a}_{t}),\dots,Y_{T+1}(\bar{a}_{T})\\}$
are independent of $A_{t}$ conditional on the observed history $H_{t}$.

Under Assumption 2.1, (1) can be re-expressed in terms of observables:

$\beta_{\bf p}(t;s)=\mathbb{E}\left[\mathbb{E}_{{\bf p}}\left[Y_{t+1}\mid
A_{t}=1,H_{t}\right]-\mathbb{E}_{{\bf p}}\left[Y_{t+1}\mid
A_{t}=0,H_{t}\right]\mid S_{t}=s\right].$ (2)

Assuming $\beta_{{\bf p}}(t;s)=f_{t}(s)^{\top}\beta_{0}^{\star}$ where
$f_{t}(s)\in\mathbb{R}^{q}$ is a feature vector comprised of a $q$-dimensional
summary of observed information depending only on state $s$ and decision point
$t$, a consistent estimator for $\beta_{0}^{\star}$ can be obtained by
minimizing the WCLS criterion (Boruvka et al., 2018):

$\mathbb{P}_{N}\Big{[}\sum_{t=1}^{T}W_{t}\times\Big{(}Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-\left(A_{t}-\tilde{p}_{t}(1\mid
S_{t})\right)f_{t}(S_{t})^{\top}\beta\Big{)}^{2}\Big{]},$ (3)

where $\mathbb{P}_{N}$ is an operator denoting the sample average,
$W_{t}=\tilde{p}_{t}(A_{t}\mid S_{t})/p_{t}(A_{t}\mid H_{t})$ is a weight
where the numerator is an arbitrary function with range $(0,1)$ that only
depends on $S_{t}$, and $g_{t}(H_{t})\in\mathbb{R}^{d}$ are $d$ control
variables chosen to help reduce noise and to construct more powerful test
statistics. See Boruvka et al. (2018) for more details on the estimand
formulation and consistency, asymptotic normality, and robustness properties
of the WCLS estimation method.

In addition to studying continuous proximal outcomes and defining the causal
excursion effect as a linear contrast of the outcomes under different
treatment allocations, as in Equation (2), there are also many MRTs concerning
longitudinal binary outcomes. Qian et al. (2021) proposed an estimator of the
marginal excursion effect (EMEE) by defining a log relative risk model for the
causal excursion effect, and we present a detailed review in Appendix M.

###  2.3 Covariate Adjustment Literature Review

The RCT methodology literature has extensively examined the use of covariate
adjustment to improve the precision of inference on ATEs. Consider, for
example, an RCT with a binary randomized treatment denoted $A$, and $N$
subjects in the study are the population of interest. Neyman (1923) showed
that the difference-in-means estimator is unbiased for the ATE. Fisher (1936)
proposed to use the ordinary least square (OLS) adjusted estimator of ATE,
which is the estimated coefficient on the treatment $A$ in the OLS regression
of $Y$ on $\\{1,A,X_{0}\\}$. This approach aims to leverage the prognostic
information in baseline covariates to improve estimation efficiency. However,
Freedman (2008) demonstrated that in specific scenarios, such adjustment could
potentially harm asymptotic precision.

Lin (2013) argued that in sufficiently large samples, the statistical problems
Freedman (2008) raised are either minor or easily fixed. In addition, Lin
(2013) shows that OLS adjustment with a full set of treatment $\times$
covariate interactions improves or does not hurt asymptotic precision, even
when the regression model is incorrect. The robust Eicker-Huber–White (Eicker,
1967; Huber, 1967; White, 2014) variance estimator (Freedman, 2006) is
consistent or asymptotically conservative (regardless of whether the
interactions are included) in estimating the true asymptotic variance. More
results on the asymptotic precision of treatment effect can be found in papers
by Yang and Tsiatis (2001), Tsiatis et al. (2008), Zhang et al. (2008),
Schochet (2010), and Negi and Wooldridge (2021).

More complex randomization schemes have led to recent developments in
covariate adjustment methods. For example, Su and Ding (2021) extended the
theory to cluster-randomized experiments, where randomization over units
within clusters is either unethical or logistically infeasible. Zhao and Ding
(2024) examined a variety of schemes for rerandomization based on p-values
(ReP), and corresponding ATE estimators from the unadjusted, additive, and
fully interacted linear regressions. It is found that the estimator from the
fully interacted regression is asymptotically most efficient. Ye et al. (2023)
used a model-assisted approach for covariate adjustment and generalized the
outcome types to linear contrast, risk ratios, and odds ratios. Their
conclusions are based on an asymptotic theory that provides a clear picture of
how covariate-adaptive randomization and regression adjustment alter
statistical efficiency.

###  2.4 Challenges

Based on these existing methods, we aim to develop a method to incorporate
covariate adjustment techniques into the analysis of MRTs. The main challenges
are twofold: First, in an MRT, the treatment, response, and moderators are all
time-varying, thus it remains questionable whether we can still center the
auxiliary variables using the overall average without producing inconsistent
estimates. Second, suppose a feasible alternative is to estimate the centering
parameters at each time point separately, the total number of such parameters
can become prohibitive as the number of decision points grows. Therefore, it
would be ideal to have a simpler approach that can be uniformly applied to all
time points.

Guo and Basse (2023) and Ye et al. (2023) both proposed checklists of ideal
properties for covariate adjustment methods. As we consider time-varying
treatment effect estimation, we build upon their lists of the ideal properties
the auxiliary variable adjustment method should have: (1) valid statistical
inference; (2) robust estimation; (3) strong finite-sample performance; (4)
wide applicability; and (5) computational simplicity. In the following
sections, we describe a general method for performing auxiliary variable
adjusted WCLS, termed “A2-WCLS”, on longitudinal data with time-varying
treatments to estimate causal excursion effects that satisfy (1)-(5).

##  3 Estimand and Inferential Method

In this section, we explore the asymptotic relative efficiency of various
candidate methods for adjusting auxiliary variables in the causal excursion
effect estimation. We aim to specify “how to account for auxiliary variables
in the analysis to improve precision” (ICH, 1998). We begin by making a
parametric assumption on the causal parameter of interest:

######  Assumption 3.1.

Assume the causal excursion effect
$\beta_{\mathbf{p}}(t;s)=f_{t}(s)^{\top}\beta_{0}^{\star}$, where
$f_{t}(s)\in\mathbb{R}^{q}$ is a feature vector comprised of a $q$-dimensional
summary of observed information depending only on state $s$ and decision point
$t$.

This parametric assumption does not necessarily translate to a correct causal
effect specification. Instead, we can interpret the proposed linear form as an
$L_{2}$ projection of the true causal excursion effect onto the space spanned
by a $q$-dimensional feature vector $f_{t}(s)$ that only depends on $t$ and
$s$ (Dempsey et al., 2020; Shi et al., 2023). The choice between these
interpretations, whether it is a correctly specified causal effect or a
projection, reflects a bias-variance trade-off. In practical applications, the
projection interpretation ensures a well-defined parameter with practical
interest.

###  3.1 Time-Specific Causal Excursion Effect Estimation

To establish a clear connection with previous research on covariate
adjustment, we start with the time-specific causal excursion effect estimand,
which shares similarities with the ATE estimand in RCTs. Recall that the ATE
estimand is typically defined as the linear difference between the expected
outcomes under different treatment allocations, i.e.,
$\mathbb{E}\big{[}\mathbb{E}[Y|X,A=1]-\mathbb{E}[Y|X,A=0]\big{]}$. In the
context of MRTs, the time-specific marginal causal effect estimand can be
defined as
$\beta_{\mathbf{p}}(t)=\mathbb{E}\big{[}\mathbb{E}_{\mathbf{p}}[Y_{t+1}|H_{t},A_{t}=1]-\mathbb{E}_{\mathbf{p}}[Y_{t+1}|H_{t},A_{t}=0]\big{]}$.
Moving forward, we drop the subscript ${\mathbf{p}}$ to simplify the notation.
The causal effect assumption stated in Assumption 3.1 is then reformulated as
$\beta(t;S_{t})=\beta_{0,t}^{\star}$, where $S_{t}=\emptyset$ and the model is
non-parametric in time $t$. As we are interested in the time-specific causal
effect, we consider the treatment randomization probability to be constant at
each time, denoted as $p_{t}(A_{t}|H_{t})=p_{t}$. In this section, the phrase
“more efficient” explicitly refers to an estimator of $\beta_{0,t}^{\star}$
that has a smaller asymptotic variance.

####  3.1.1 The Unadjusted Estimator

Analysis using solely the data from time-varying outcomes and treatments,
$\\{Y_{t,j},A_{t,j}\\}_{t=1}^{T},j=1,\dots,N$, without adjusting for auxiliary
variables yields the _unadjusted estimator_. Specifically, in our context, the
unadjusted estimator $\hat{\beta}_{0,t}^{\text{U}}$ can be obtained by
minimizing the following objective function:

$\mathbb{P}_{N}\Big{[}\sum_{t=1}^{T}\big{(}Y_{t+1}-\alpha_{0,t}-\left(A_{t}-p_{t}\right)\beta_{0,t}\big{)}^{2}\Big{]}.$
(4)

For a constant and known treatment randomization probability $p_{t}$, the
corresponding weight simplifies to $W_{t}=1$. The unadjusted estimator
$\hat{\beta}_{0,t}^{\text{U}}$ is shown to be consistent and asymptotically
normal (a special case as seen in Boruvka et al. (2018)), and it will serve as
the reference against which we will evaluate the relative efficiency of
alternative estimators.

####  3.1.2 The WCLS estimator

Recall that the WCLS estimator is derived by minimizing the objective function
(3). When estimating the time-specific treatment effect with
$\beta(t;S_{t})=\beta_{0,t}^{\star}$, the WCLS criterion can be reformulated
as follows:

$\mathbb{P}_{N}\Big{[}\sum_{t=1}^{T}\big{(}Y_{t+1}-\alpha_{0,t}-\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}-\left(A_{t}-p_{t}\right)\beta_{0,t}\big{)}^{2}\Big{]}.$
(5)

Here, $\tilde{g}_{t}(H_{t})=g_{t}(H_{t})-\mathbb{E}[g_{t}(H_{t})]$ denotes the
mean-centered control variables. Similar to $\hat{\beta}_{0,t}^{\text{U}}$,
the estimator $\hat{\beta}_{0,t}^{\text{WCLS}}$ obtained by minimizing (5) is
consistent and asymptotically normal (Boruvka et al., 2018). The distinction
lies in the inclusion of control variables $\tilde{g}_{t}(H_{t})^{\top}\alpha$
in (5), which is expected to reduce the estimation error of the causal
parameter. A natural question is whether this goal can be consistently
achieved. If not, it is important to identify conditions where incorporating
control variables $g_{t}(H_{t})$ may harm the asymptotic efficiency of
$\hat{\beta}_{0,t}^{\text{WCLS}}$ relative to $\hat{\beta}_{0,t}^{\text{U}}$.
To address this, we present the following lemma, with detailed proof in
Appendix A.2.

######  Lemma 3.2 (Asymptotic Variance Comparison between
$\hat{\beta}_{0,t}^{\text{WCLS}}$ and $\hat{\beta}_{0,t}^{\text{U}}$).

Denote
$\beta_{1,t}^{\star}=\Sigma(g_{t}(H_{t}))^{-1}\mathbb{E}[Y_{t+1}(A_{t}-p_{t})\tilde{g}_{t}(H_{t})]\in\mathbb{R}^{d}$,
where
$\Sigma(g_{t}(H_{t}))=\mathbb{E}\big{[}\tilde{g}_{t}(H_{t})\tilde{g}_{t}(H_{t})^{\top}\big{]}\in\mathbb{R}^{d\times
d}$. The difference in the asymptotic variance between the WCLS estimator
$\hat{\beta}_{0,t}^{\text{WCLS}}$ and the unadjusted estimator
$\hat{\beta}_{0,t}^{\text{U}}$ can be expressed as
$-p_{t}(1-p_{t})\alpha_{1,t}^{\star\top}\Sigma(g_{t}(H_{t}))\big{(}\alpha^{\star}_{1,t}+2(1-2p_{t})\beta^{\star}_{1,t}\big{)}$.

Here $\beta_{1,t}^{\star}$ represents the true moderation effect size of
variables $\tilde{g}_{t}(H_{t})$, and its least-squares solutions can be
obtained from minimizing the objective function below in Equation (6), where
we include the full interaction between treatment $A_{t}$ and
$\tilde{g}_{t}(H_{t})$. The $\Sigma(g_{t}(H_{t}))$ denotes the variance-
covariance matrix for the control variables $g_{t}(H_{t})$. If $p_{t}=1/2$,
the adjustment is either neutral or favorable. However, consider the case
where $g_{t}(H_{t})$ is positively correlated with the outcome and strongly
moderates the treatment effect (i.e., $\alpha^{\ast}_{1,t}>0$ and
$|\beta^{\ast}_{1,t}|$ is large), if $\beta^{\ast}_{1,t}>0$ and the
randomization probability $p_{t}>1/2$, the adjustment would lead to an
increase in asymptotic variance.

In conclusion, the use of the WCLS criterion presented in (5) cannot guarantee
efficiency improvement as compared to the unadjusted estimator presented in
(4). This aligns with the findings in Freedman (2008) regarding standard
covariate adjustment methods in RCTs.

####  3.1.3 A More Efficient Estimator

We consider an alternative way to adjust for auxiliary variables which
guarantees estimation efficiency improvement. Motivated by Lin (2013), we
incorporate similar ideas as in covariate adjustment literature into the WCLS
approach. The proposed estimator of the causal excursion effect can then be
obtained by minimizing the following criterion:

$\mathbb{P}_{N}\Big{[}\sum_{t=1}^{T}\big{(}Y_{t+1}-\alpha_{0,t}-\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}-\left(A_{t}-p_{t}\right)(\beta_{0,t}+\tilde{g}_{t}(H_{t})^{\top}\beta_{1,t})\big{)}^{2}\Big{]}.$
(6)

Equation (6) is a strict extension of the proposal in Lin (2013); we denote
the minimizer $\hat{\beta}_{0,t}^{\text{L}}$. Lemma 3.3 below establishes an
asymptotic efficiency gain of $\hat{\beta}_{0,t}^{\text{L}}$ over _both_ the
unadjusted $\hat{\beta}_{0,t}^{\text{U}}$ and the WCLS
$\hat{\beta}_{0,t}^{\text{WCLS}}$ estimators.

######  Lemma 3.3.

Suppose Assumptions 2.1 and 3.1 hold, and the randomization probability
$p_{t}$ is known. Given invertibility and regularity conditions, let
$(\hat{\alpha}_{0,t},\hat{\alpha}_{1,t},\hat{\beta}_{0,t}^{\text{L}},\hat{\beta}_{1,t})$
minimize objective function (6):

  1. 1.

$\hat{\beta}_{0,t}^{\text{L}}$ is consistent and asymptotically normal such
that
$\sqrt{N}(\hat{\beta}_{0,t}^{\text{L}}-\beta_{0,t}^{\star})\rightarrow\mathcal{N}(0,Q_{t}^{-1}\Sigma_{t}^{\text{L}}Q_{t}^{-1})$,
where $Q_{t},\Sigma_{t}^{\text{L}}$ are defined in Appendix B;

  2. 2.

$\hat{\beta}_{0,t}^{\text{L}}$ is at least as efficient as the unadjusted
estimator $\hat{\beta}_{0,t}^{\text{U}}$ described in (4), and the asymptotic
efficiency gain is
$\frac{\left(\alpha^{\star}_{1,t}+(1-2p_{t})\beta^{\star}_{1,t}\right)^{\top}\Sigma(g_{t}(H_{t}))\left(\alpha^{\star}_{1,t}+(1-2p_{t})\beta^{\star}_{1,t}\right)}{p_{t}(1-p_{t})}+\beta_{1,t}^{\star\top}\Sigma(g_{t}(H_{t}))\beta^{\star}_{1,t}$;

  3. 3.

$\hat{\beta}_{0,t}^{\text{L}}$ is at least as efficient as the WCLS estimator
$\hat{\beta}_{0,t}^{\text{WCLS}}$ described in (5), and the asymptotic
efficiency gain is
$\frac{1-3p_{t}+3p_{t}^{2}}{p_{t}(1-p_{t})}\beta_{1,t}^{\star\top}\Sigma(g_{t}(H_{t}))\beta^{\star}_{1,t}$.

Lemma 3.3 demonstrates that incorporating mean-centered auxiliary variables in
the estimation of the time-specific marginal treatment effect using the
objective function in Equation (6) guarantees an efficiency improvement over
both the unadjusted and WCLS estimators. A detailed proof for Lemma 3.3 is
available in Appendix B.

######  Remark 3.4.

The above discussion aims to connect two frameworks of covariate adjustment,
one under our setting of time-varying causal excursion effect estimation in
MRTs, and the other under the setting of classical RCTs. We here focused on
comparing the efficiency of time-specific marginal effect estimators, but this
comparison also applies to time-specific moderation analysis when considering
categorical moderators, i.e.,
$\beta(t;S_{t})=f_{t}(S_{t})^{\top}\beta_{0,t}^{\star}$ and $f_{t}(S_{t})$ are
categorical variables. Detailed discussion and proof are in Appendix F.

###  3.2 Moderation Effect Estimation

There are several limitations with (6). First, most MRTs have a relatively
large number of time points $T$. Therefore, estimating centering parameters
for each time point separately leads to a large number of additional
parameters. Second, centering the auxiliary variable at its mean is not a one-
size-fits-all remedy. For example, when we are interested in moderated
treatment effects (i.e., $S_{t}\neq\emptyset$), centering by time-specific
means can introduce bias.

To overcome these limitations, we introduce a general approach for
incorporating auxiliary variables into the estimation of time-varying causal
effects. Our method leverages a low-dimensional centering function, uniformly
applicable across all time points, to estimate the causal parameter
$\beta_{0}^{\star}$, ensuring both consistency and improved efficiency.

Here we consider auxiliary variable adjusted estimators of the causal
excursion effect, with a _primary moderator_ of interest
$f_{t}(S_{t})\in\mathbb{R}^{q}$ and a smoothed causal excursion effect model
$f_{t}(S_{t})^{\top}\beta_{0}$. Let $Z_{t}\in\mathbb{R}^{p}$ represent a set
of $p$-dimensional _auxiliary variables_ satisfying $Z_{t}\subseteq
g_{t}(H_{t})$ and $Z_{t}\cap f_{t}(S_{t})=\emptyset$. Previously in Section
2.2, we reviewed the estimator $\hat{\beta}_{0}^{\text{WCLS}}$ obtained from
the WCLS criterion outlined in display (3) is consistent and asymptotically
normal (Boruvka et al., 2018). Our goal is to improve the estimation
efficiency of $\hat{\beta}_{0}^{\text{WCLS}}$ by leveraging the auxiliary
information from $\\{Z_{t}\\}_{t=1}^{T}$. In the context of moderation effect
analysis, the term “more efficient” refers to an estimator that has a lower
asymptotic variance than the alternative for all linear combinations, which is
equivalent to the negative semidefiniteness of the difference between the
asymptotic variance matrices.

####  3.2.1 Auxiliary Variable Adjusted WCLS (A2-WCLS) Estimation

An auxiliary variable adjusted estimation method using a centering function
$\mu_{t}(S_{t})\in\mathbb{R}^{p}$ is developed below. Under Assumption 3.1,
the Auxiliary-variable Adjusted Weighted and Centered Least Square (A2-WCLS)
criterion is shown by minimizing the following:

$\mathbb{P}_{N}\Big{[}\sum_{t=1}^{T}W_{t}\times\Big{(}Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-\big{(}A_{t}-\tilde{p}_{t}(1\mid
S_{t})\big{)}\big{(}f_{t}(S_{t})^{\top}\beta_{0}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}\big{)}\Big{)}^{2}\Big{]}.$
(7)

The resulting minimizer is denoted as $\hat{\beta}_{0}^{\text{A2}}$, our
A2-WCLS estimator. An appropriate choice of the centering function
$\mu_{t}(S_{t})$ is key to obtaining a consistent estimation of the moderated
causal effect.  We need to make sure the minimizers of two objective
functions, one in Equation (7) and the other in Equation (3), with and without
auxiliary variable adjustment, converge asymptotically to the same limit value
$\beta_{0}^{\star}$. This can be formulated as a set of linear constraints on
the centering function $\mu_{t}(S_{t})$ as stated in Condition 3.5:

######  Condition 3.5 (The orthogonality condition).

For the $i$th component of $Z_{t}\in\mathbb{R}^{p}$, denoted
$Z^{i}_{t}\in\mathbb{R}$, $i=1,\ldots,p$, the associated centering function,
denoted $\mu_{t}^{i}(S_{t})$, must satisfy the following orthogonality
condition with respect to $f_{t}(S_{t})\in\mathbb{R}^{q}$:

$\mathbb{E}\Big{[}\sum_{t=1}^{T}\tilde{p}_{t}(1|S_{t})(1-\tilde{p}_{t}(1|S_{t}))(Z^{i}_{t}-\mu^{i}_{t}(S_{t}))f_{t}(S_{t})\Big{]}=\mathbf{0}_{q\times
1}.$ (8)

where $\tilde{p}_{t}(A_{t}|S_{t})$ is the numerator of the weight $W_{t}$ that
is an arbitrary function with range $(0,1)$ depending only on $S_{t}$.

Condition 3.5 is a sufficient condition for incorporating auxiliary variables
into the objective function (7). This is the key technical contribution
demonstrating “how to” include time-varying auxiliary variables without
harming the statistical consistency of the resulting estimator. There are two
novel aspects to this orthogonality condition. First, Equation (8) allows
flexibility in selecting $\mu^{i}_{t}(S_{t})$ beyond the conditional mean
$\mathbb{E}[Z^{i}_{t}|S_{t}]$. This flexibility eases the strict requirement
for consistent estimations of $\mathbb{E}[Z^{i}_{t}|S_{t}]$, which can be
challenging even in moderate dimensions. Second, Equation (8) represents a
system of $q$ linear equations, given the dimension of
$f_{t}(S_{t})\in\mathbb{R}^{q}$. This implies that for each auxiliary variable
$\\{Z_{t}^{i}\\}_{t=1}^{T}$, there are $q$ constraints on the associated
centering function $\\{\mu_{t}^{i}(S_{t})\\}_{t=1}^{T}$. As a result, the
dimension of $\\{\mu_{t}^{i}(S_{t})\\}_{t=1}^{T}$ can be as low as $q$,
avoiding prohibitive growth with increasing $T$. To illustrate Condition 3.5,
consider the following example:

######  Example 3.6 (Fully marginal causal effect estimation).

Consider estimating the fully marginal causal excursion effect where
$f_{t}(S_{t})=1$ (i.e., $S_{t}=\emptyset$). We want to potentially incorporate
auxiliary variable $Z_{t}^{i}\in\mathbb{R}$ to improve efficiency. Here we
compare three centering function choices: (a) $\mu_{t}^{i}=\bar{Z}_{t}^{i}$,
($t=1,2,\dots,T$), (b) $\mu^{i}_{t}=\mu^{i}=\bar{Z}^{i}$, and (c)
$\mu^{i}_{t}=\mu^{i}=\frac{\sum_{t=1}^{T}\tilde{p}_{t}(1|S_{t})(1-\tilde{p}_{t}(1|S_{t}))Z^{i}_{t}}{\sum_{t=1}^{T}\tilde{p}_{t}(1|S_{t})(1-\tilde{p}_{t}(1|S_{t}))}$.
Option (a) estimates the time-specific mean $\mathbb{E}[Z^{i}_{t}]$ following
the approach introduced by Lin (2013), indicating a correct model
specification. It satisfies Equation (8), but involves $T$ estimates, which
can be cumbersome when $T$ is large. Option (b), the global mean of the
auxiliary variable $\\{Z_{t}^{i}\\}_{t=1}^{T}$, is a scalar centering function
but can lead to inconsistent estimates of $\beta_{0}^{\star}$ since it does
not satisfy Equation (8). Please refer to Appendix K.1 for an empirical
illustration of the biased estimate. Option (c) is a scalar centering function
satisfying Equation (8), representing a weighted average of
$\\{Z_{t}^{i}\\}_{t=1}^{T}$. In conclusion, option (c) has the desirable
characteristics we seek — a low-dimensional centering function that is
uniformly applicable across all time points and consistently estimates the
causal parameter $\beta_{0}^{\star}$.

The following lemma describes the asymptotic properties of the A2-WCLS
estimator, allowing us to compare its estimation precision with the existing
WCLS estimator.

######  Lemma 3.7.

Suppose Assumptions 2.1 and 3.1 hold, centering function $\mu_{t}(S_{t})$
satisfies the orthogonality condition 3.5, and that the randomization
probability $p_{t}(A_{t}=1|H_{t})$ is known. Let
$(\hat{\alpha},\hat{\beta}_{0}^{\text{A2}},\hat{\beta}_{1})$ minimize
objective function (7). Under regularity conditions,
$\hat{\beta}_{0}^{\text{A2}}$ is consistent and asymptotically normal such
that
$\sqrt{N}(\hat{\beta}_{0}^{\text{A2}}-\beta_{0}^{\star})\rightarrow\mathcal{N}(0,Q^{-1}\Sigma^{\text{A2}}Q^{-1})$,
where $Q,\Sigma^{\text{A2}}$ are defined in Appendix D.

Note that the consistency property of $\hat{\beta}_{0}^{\text{A2}}$ does not
depend on the specific choice of $\mu_{t}(S_{t})$, as long as $\mu_{t}(S_{t})$
satisfies Condition 3.5. In addition, the A2-WCLS criterion inherits the
robustness of the WCLS method, which guarantees that
$\hat{\beta}_{0}^{\text{A2}}$ remains consistent even if the nuisance function
$g(H_{t})^{\top}\alpha$ is not correctly specified.

Equation (8) serves as a set of estimating equations to derive a plug-in
estimator of $\mu^{i}_{t}(S_{t})$. A straightforward option for a
$q$-dimensional working model is
$\mu^{i}_{t}(S_{t};\theta^{i})=f_{t}(S_{t})^{\top}\theta^{i}$, where
$i\in\\{1,\dots,p\\}$, and $\hat{\theta}^{i}$ can be computed by solving
Equation (8). This choice allows us to exploit the low-dimensional nature of
the causal effect model. Let
$\Theta=(\theta^{1},\dots,\theta^{p})\in\mathbb{R}^{q\times p}$ and
$\hat{\Theta}^{\top}=(\hat{\theta}^{1},\dots,\hat{\theta}^{p})$, we can
construct a straightforward and easy-to-implement A2-WCLS criterion as
follows:

$\mathbb{P}_{N}\bigg{[}\sum_{t=1}^{T}W_{t}\times\Big{(}Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-\big{(}A_{t}-\tilde{p}_{t}(1\mid
S_{t})\big{)}\big{(}f_{t}(S_{t})^{\top}\beta_{0}+(Z_{t}-\hat{\Theta}^{\top}f_{t}(S_{t}))^{\top}\beta_{1}\big{)}\Big{)}^{2}\bigg{]}$
(9)

######  Remark 3.8 (Connection to the Neyman Orthogonality).

By treating $(\alpha,\beta_{1})$ as nuisance parameters, we can establish a
connection between the Orthogonality Condition 3.5 and the Neyman
orthogonality of the score equations (Neyman, 1979; Chernozhukov et al.,
2018a). Intuitively, Neyman orthogonality implies that the moment conditions
used to identify $\beta_{0}$ are insensitive to the nuisance parameter
$\beta_{1}$. Likewise, our orthogonality condition here also aims to ensure
that the consistent estimation of $\beta_{0}^{\star}$ will not be affected by
the estimation of $\beta_{1}^{\star}$. It can be derived that Neyman
orthogonality leads to the same constraint as Condition 3.5. The proof is
presented in Appendix C.1.

######  Remark 3.9 (Connection to a semiparametric efficient estimator).

A particular scenario arises when $S_{t}$ is set to be the entire observed
history $H_{t}$. In this case, semiparametric efficiency theory can be used to
derive efficient estimators (Robins, 1994). According to the semiparametric
efficient influence function, we can view the A2-WCLS criterion in display (7)
as derived under a homogeneity assumption with stabilized inverse probability
weighting $W_{t}$. Then we express the fully conditional causal effect as a
function of the feature vector $f_{t}(S_{t})$ and its orthogonal complement
using Condition 3.5. As a result, we can directly read off the marginal effect
of interest. The detailed discussion is provided in Appendix H.

######  Remark 3.10 (Bidirectional asymptotic property).

The asymptotic property outlined in Lemma 3.7 considers $T$ as a fixed value,
with the sample size $N$ approaching infinity. In some cases, MRTs may involve
small sample sizes compared to the scale of total time points $T$. Our
simulations in Section 6 addressed this by applying a small sample correction
(Mancl and DeRouen, 2001) to the robust variance estimator, which yielded
strong empirical performance for small sample sizes ($N\approx 40$). In the
extreme case where the sample size is significantly small, we establish
bidirectional asymptotics for the proposed estimators in Lemma D.3, requiring
either the sample size $N$ or the number of decision points $T$ to approach
infinity (Yu et al., 2023; Liu et al., 2023). Additional details and proof are
provided in Appendix D.1.

######  Remark 3.11 (Important Extensions to Time-Lagged/Binary Outcomes).

Some MRT analyses might focus on evaluating the delayed effect of current
treatment $A_{t}$ on a future outcome $Y_{t+\Delta}$, where $\Delta>1$. The
proposed A2-WCLS method can be readily extended to account for these time-
lagged outcomes, enabling a novel approach to adjust post-treatment auxiliary
variables. Furthermore, our method applies to the causal excursion effect
represented not as a linear contrast, but as logarithmic relative risks, as
demonstrated in Qian et al. (2021). We have included these extensions in
Appendix L and Appendix M.

####  3.2.2 The Coefficients of the Adjusted Auxiliary Variables

The estimate $\hat{\beta}_{1}(\mu_{t})$ obtained from minimizing A2-WCLS in
criterion (7) converges to $\beta_{1}^{\star}(\mu_{t})$ given by:

$\mathbb{E}\Big{[}\sum_{t=1}^{T}\tilde{p}_{t}(1-\tilde{p}_{t})\Sigma_{\mu}(Z_{t})\Big{]}^{-1}\mathbb{E}\Big{[}\sum_{t=1}^{T}\tilde{p}_{t}(1-\tilde{p}_{t})\big{(}\beta(t;S_{t},Z_{t})-\beta(t;S_{t})\big{)}\big{(}Z_{t}-\mu_{t}(S_{t})\big{)}\Big{]},$
(10)

where
$\Sigma_{\mu}(Z_{t})=\mathbb{E}\big{[}(Z_{t}-\mu_{t}(S_{t}))(Z_{t}-\mu_{t}(S_{t}))^{\top}\big{]}\in\mathbb{R}^{p\times
p}$. Recall the definition in Equation (2),
$\beta(t;S_{t},Z_{t}):=\mathbb{E}\big{[}\mathbb{E}[Y_{t+1}|H_{1},1]-\mathbb{E}[Y_{t+1}|H_{1},0]|S_{t},Z_{t}\big{]}$
in expression (10) represents the moderated causal excursion effect at time
$t$, given the effect moderators $\\{S_{t},Z_{t}\\}$. Therefore, the contrast
$\beta(t;S_{t},Z_{t})-\beta(t;S_{t})$ represents the additional causal effect
attributed to including the auxiliary variable $Z_{t}$ in the primary
moderator set $S_{t}$. Expression (10) indicates that
$\beta_{1}^{\star}(\mu_{t})$ depends on both $Z_{t}$ and $\mu_{t}(S_{t})$, so
it can reflect how the inference changes based on different choices of
auxiliary variables and their corresponding centering functions when employing
the A2-WCLS method. The following lemma offers additional insights. Detailed
proofs and a simulation are provided in Appendix E.

######  Lemma 3.12.

The coefficient $\beta_{1}^{\star}(\mu_{t})$ can evaluate how the choice of
the auxiliary variable $Z_{t}$ and its corresponding centering function
$\mu_{t}(S_{t})$ affect moderation effect estimation:

  1. (a)

If $\beta(t;S_{t},Z_{t})-\beta(t;S_{t})=0$, then $\beta_{1}^{\star}=0$, i.e.,
the inclusion of non-effect-moderator auxiliary variables through the A2-WCLS
criterion has no impact on the asymptotic variance.

  2. (b)

For any function $\mu_{t}(S_{t})$ satisfying Condition 3.5,
$|\beta_{1}^{\star}(\mu_{t})|\leq|\beta_{1}^{\star}(\mathbb{E}[Z_{t}|S_{t}])|$;
i.e., parameter attenuation will occur under misspecification of the centering
function.

##  4 Theoretical Results

In this section, we provide theoretical results on incorporating auxiliary
variables to improve the estimation efficiency of time-varying causal effect
moderation. This differs from the global efficiency gain seen in Lemma 3.3 on
the estimation of time-specific causal effects and from the theorem presented
in Lin (2013), both of which focus only on fully marginal causal effects
($S_{t}=\emptyset$). Here modeling time-varying moderated causal effects
requires additional assumptions for guaranteed efficiency gains, because
treatments, moderators, and outcomes can exhibit time dependence, and the
estimators for moderated effects are functions smoothed over time.

Given the fully observed history $H_{t}$ and a selection of auxiliary
variables $Z_{t}$, with a known centering function $\mu_{t}(S_{t})$, we
introduce the following assumption:

######  Assumption 4.1 (Correct causal model specification).

The true causal effect model $\beta(t;H_{t})$ is correctly specified in
$H_{t}$, that is,
$\beta(t;H_{t})=f_{t}(S_{t})^{\top}\beta^{\star}_{0}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta^{\star}_{1}$.

Here, we do not impose specific distribution assumptions on potential outcomes
$\\{Y_{t+1}(\bar{A}_{t-1},a)\\},a\in\\{0,1\\}$, nor require the correct
specification of the true data-generating model
$\mathbb{E}[Y_{t+1}|H_{t},A_{t}]$. Our focus is only on the difference in
conditional outcomes
$\beta(t;H_{t})=\mathbb{E}[Y_{t+1}|H_{t},1]-\mathbb{E}[Y_{t+1}|H_{t},0]$.
Another assumption is regarding the time dependence:

######  Assumption 4.2 (Conditionally uncorrelated error and future states).

Define the residual as
$\epsilon_{t}^{\text{A2}}=Y_{t+1}-g_{t}(H_{t})^{\top}\alpha^{\star}-(A_{t}-\tilde{p}_{t}(1|S_{t}))\big{(}f_{t}(S_{t})^{\top}\beta^{\star}_{0}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta^{\star}_{1}\big{)}$.
After accounting for all the information from the observed history up to and
including the current time point, the unexplained residual in the current
outcome is not correlated with potential observations in future states, i.e.,
$\mathbb{E}[\epsilon_{t}^{\text{A2}}f_{t^{\prime}}(S_{t^{\prime}})(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))|H_{t},A_{t}]=0$,
for $t<t^{\prime}$.

This assumption implies that the residuals do not carry any additional
information beyond that which is already contained in the observed history.
This is fundamental in many statistical learning models, particularly in
Markov Decision Processes (MDPs), where the current state determines future
states, independent of any unobserved information. Based on the assumptions
outlined above, we can state the following theorem:

######  Theorem 4.3.

Under the assumptions of Lemma 3.7 as well as Assumptions 4.1, and 4.2, if the
centering function is known, incorporating auxiliary variables via A2-WCLS
guarantees either an improvement or no harm to the asymptotic precision of the
moderated treatment effect estimation. Specifically, the asymptotic efficiency
gain compared to the WCLS estimator is given by $Q^{-1}\Sigma Q^{-1}$, where

$\begin{split}Q&=\mathbb{E}\Big{[}\sum_{t=1}^{T}\tilde{p}_{t}(1-\tilde{p}_{t})f_{t}(S_{t})f_{t}(S_{t})^{\top}\Big{]},\\\
\Sigma&=\mathbb{E}\Big{[}\Big{(}\sum_{t=1}^{T}W_{t}(A_{t}-\tilde{p}_{t})^{2}f_{t}(S_{t})(Z_{t}-\mu_{t}(S_{t}))^{\top}\Big{)}\beta^{\star}_{1}\beta_{1}^{\star\top}\Big{(}\sum_{t=1}^{T}W_{t}(A_{t}-\tilde{p}_{t})^{2}f_{t}(S_{t})(Z_{t}-\mu_{t}(S_{t}))^{\top}\Big{)}^{\top}\Big{]}.\end{split}$
(11)

For ease of understanding, we consider (11) in the setting where
$f_{t}(S_{t})=1$ (i.e., $S_{t}=\emptyset$). In this case, the efficiency
improvement is proportional to the variance of a weighted sum of $Z_{t}$ over
time, multiplied by the square of $\beta_{1}^{\star}$. Therefore, if we
include auxiliary variables that are strong effect moderators with large
variances, we can substantially improve the precision of our estimates. Proof
of Theorem 4.3 can be found in Appendix G.

######  Remark 4.4 (Study design and causal modeling assumptions).

Theorem 4.3 is presented without imposing any constraints on the MRT study
design. This means that observations can exhibit dependence across all time
points, and treatment randomization can depend on the fully observed history.
As a result, stronger modeling assumptions are required to ensure efficiency
gains. This trade-off is further explored in Appendix G with two common study
designs, where we illustrate how Assumption 4.1 and 4.2 can be further relaxed
while still ensuring efficiency improvement in the estimation of causal
effects varying over time.

######  Remark 4.5 (Estimating centering functions).

Theorem 4.3 assumes known centering functions; however, in practice, these
functions may need to be estimated from observed data, and analysis should
account for this extra variance to ensure proper uncertainty quantification. A
‘stacking estimating equations’ approach, a type of M-estimator (Carroll et
al., 2006), is adopted to construct a robust asymptotic variance estimator
shown in Equation (49) of Appendix I.2. While the efficiency improvement may
be smaller compared to Theorem 4.3, our proposed method remains robust to the
estimated centering functions. We see empirical efficiency gains in Section 6.
See Appendix I for further discussion.

##  5 Properties

In this section, we return to the five desired properties in the checklist
discussed in Section 2.4 and demonstrate that A2-WCLS satisfies all of them.

Valid statistical inference. As demonstrated in Lemma 3.7 and Theorem 4.3, our
approach ensures consistent and asymptotically normal estimation for the
causal parameter. Moreover, under Assumption 4.1 and 4.2, A2-WCLS has
theoretical guarantees for achieving an efficiency gain relative to the
original WCLS method.

Robust estimation. (1) Lemma 3.12 indicates that the inclusion of non-effect-
moderator auxiliary variables in the causal effect model does not compromise
the precision of the estimation of the parameter of interest. (2) The A2-WCLS
method shares the same property as the WCLS in that if the true randomization
probability $p_{t}(A_{t}|H_{t})$ is known, a consistent estimator can be
obtained even when the nuisance function $g_{t}(H_{t})^{\top}\alpha$ is
misspecified (Boruvka et al., 2018). Robust standard errors are used to
account for model misspecification and heteroskedasticity.

Strong finite-sample performance. Our conclusions are derived based on
asymptotics, where the product of sample size and the total number of time
points $NT$ approaches infinity. However, our proposed method can still
provide reliable performance in a finite sample with a small sample correction
(Mancl and DeRouen, 2001) applied to the variance estimator, which is
elaborated in Appendix J.

Wide applicability. The proposed method is a comprehensive extension of the
existing literature on covariate adjustment that can be applied to study
designs with $T\geq 1$, as discussed in Lemma 3.3 and Theorem 4.3. When $T=1$,
the adjustment of auxiliary variables relies on baseline information, making
it equivalent to the covariate adjustment techniques discussed in Lin (2013)
and Ye et al. (2023).

Computational simplicity. Using linear working models
$\Theta^{\top}f_{t}(S_{t})$ for the centering function simplifies computation,
and can be easily implemented using existing routines found in standard
statistical software packages.

##  6 Simulation

Here, the empirical performance of the proposed A2-WCLS method is examined via
simulation. Results verify the theory that the A2-WCLS method can
significantly improve the efficiency of causal effect estimation, even when
accounting for the additional variance introduced by estimating the centering
function from data.

###  6.1 Simulation Setup

The proposed method is empirically evaluated via simulation where the data
generation model is based on Boruvka et al. (2018) with minor adjustments made
to demonstrate the improvement of our proposed method over the WCLS approach.
Consider an MRT with known randomization probability and the observation
vector being a single state variable $Z_{t}\in\\{-1,1\\}$ at each decision
time $t$, and the state dynamics are given by
$\mathbb{P}(Z_{t}=1|A_{t-1},H_{t-1})=\text{expit}(0.05t+\xi A_{t-1})$ with
$\xi=0.1$. Let

$Y_{t+1,j}=(\beta_{10}+\beta_{11}(Z_{t,j}-\mathbb{E}[Z_{t,j}]))\times(A_{t,j}-p_{t}(1|H_{t,j}))+0.8Z_{t,j}+\epsilon_{t,j}.$
(12)

The randomization probability is set to
$p_{t}(1|H_{t})=\text{expit}(\eta_{1}A_{t-1}+\eta_{2}Z_{t})$ where
$(\eta_{1},\eta_{2})=(-0.8,0.8)$ and $\text{expit}(x)=(1+\exp(-x))^{-1}$. The
independent error term satisfies $\epsilon_{t}\sim\mathcal{N}(0,1)$ with
$\text{Corr}(\epsilon_{u},\epsilon_{t})=0.5^{|u-t|/2}$. We set
$\beta_{10}=-0.2$, and $\beta_{11}\in\\{0.2,0.5,0.8\\}$ to represent a small,
medium, and large moderation effect.

###  6.2 Simulation Results

In the first simulation experiment, the fully marginal causal effect is
estimated, which is constant in time and given by
$\beta_{0}^{\star}=\beta_{10}=-0.2$. Results are reported with $N=250$
individuals and $T=30$ using the following estimation methods:

Estimation Method I: WCLS. The WCLS method (Boruvka et al., 2018) is used with
$Z_{t,j}$ as a control variable. This method guarantees to produce a
consistent estimate with a valid confidence interval. Thus, it will be used as
a reference for comparison of the estimation results from the following
method.

Estimation Method II: A2-WCLS. The proposed A2-WCLS method is used with
$Z_{t,j}$ adjusted as an auxiliary variable. Since the fully marginal causal
excursion effect (i.e. $S_{t}=\emptyset$, $f_{t}(S_{t})=1$) is of primary
interest, the working model of the centering function using A2-WCLS criterion
in (9) is $\mu_{t}(S_{t})=\theta$, and
$\hat{\theta}=\frac{\mathbb{P}_{n}\left[\sum_{t=1}^{T}\tilde{p}_{t,j}(1-\tilde{p}_{t,j})Z_{t,j}\right]}{\mathbb{P}_{n}\left[\sum_{t=1}^{T}\tilde{p}_{t,j}(1-\tilde{p}_{t,j})\right]}$.

The simulation experiment involves estimating $\theta$ using the data
generated, and this introduces non-negligible variation that affects the
asymptotic variance estimation for the fully marginal causal effect.
Therefore, the asymptotic variance estimator displayed in Equation (49) from
Appendix I.2 is used. Table 1 reports the simulation results. “%RE gain”
indicates the percentage of times we achieve an efficiency gain out of 1000
Monte Carlo replicates. “mRE” stands for the average relative efficiency, and
“RSD” represents the relative standard deviation between two estimates.
Despite having to account for the extra variance caused by estimating
$\theta$, the proposed A2-WCLS method still significantly improves the
efficiency of fully marginal causal effect estimation.

Table 1: Fully marginal causal effect estimation efficiency comparison. The true value of the parameters is $\beta_{0}^{\star}=-0.2$.  Method | $\beta_{11}$ | Est | SE | CP | %RE gain | mRE | RSD
---|---|---|---|---|---|---|---
WCLS | 0.2 | -0.197 | 0.029 | 0.952 | - | - | -
0.5 | -0.197 | 0.029 | 0.935 | - | - | -
0.8 | -0.194 | 0.031 | 0.936 | - | - | -
A2-WCLS | 0.2 | -0.199 | 0.026 | 0.955 | 100% | 1.207 | 1.198
0.5 | -0.200 | 0.027 | 0.936 | 100% | 1.202 | 1.196
0.8 | -0.200 | 0.028 | 0.940 | 100% | 1.196 | 1.198

Except for estimating the fully marginal effect, we conducted additional
simulations to assess the performance of the proposed A2-WCLS estimator. A
simulation using a higher-dimensional working model for $\mu_{t}(S_{t})$ can
be found in Appendix K.3. The results show that a higher dimension of
$\mu_{t}(S_{t})$ does not necessarily lead to a significant improvement in
relative efficiency compared to the use of the simpler working model
$\Theta^{\top}f_{t}(S_{t})$. This can be seen as a trade-off between having a
centering function that better approximates the conditional expectation
$\mathbb{E}[Z_{t}|S_{t}]$ and the additional variances introduced by
estimating more parameters. Further evaluations of A2-WCLS in estimating time-
varying treatment effects are outlined in Appendix K.4. The results
demonstrate consistent estimation and efficiency gains. In Appendix K.5, we
explore how efficiency gains vary with different choices for sample size $N$
and total time points $T$.

##  7 Application to Intern Health Study

The Intern Health Study (IHS) is a six-month microrandomized trial on medical
interns (NeCamp et al., 2020), which aimed to investigate when to provide
mHealth interventions to people in stressful work environments to improve
their behavior and mental health. In this section, we evaluate the
effectiveness of targeted notifications in improving an individual’s mood and
step counts. The data set contains 1562 participants.

The exploratory and MRT analyses conducted in this paper focus on weekly
randomization, thus, an individual was randomized to receive mood, activity,
sleep, or no notifications with equal probability ($1/4$ each) every week. We
choose the outcome $Y_{t+1,j}$ as the average self-reported mood score (a
Likert scale taking values from 1 to 10) and step count (cubic root
transformed) for individual $j$ in study week $t$. The average weekly mood
score when a notification is delivered is 7.14, and 7.16 when there is no
notification; The average weekly step count (cubic root) when a notification
is delivered is 19.1, and also 19.1 when there is no notification. In the
following analysis, the prior week’s outcome is chosen as the auxiliary
variable. We evaluate the targeted notification treatment effect for medical
interns using our proposed method (A2-WCLS) and WCLS.

###  7.1 Comparison of the Marginal Effect Estimation

First, the fully marginal excursion effect is assessed (i.e.,
$\beta(t)=\beta_{0}^{\star}$). For an individual $j$, the study week is coded
as a subscript $t$. $Y_{t+1,j}$ is the self-reported mood score or step count
of the individual $j$ in study week $t+1$. $A_{t,j}$ is defined as the
specific type of notification that targets improving the outcome. For example,
if the outcome is the self-reported mood score, sending mood notifications
would be the treatment, thus $\mathbb{P}(A_{t,j}=1)=0.25$. We analyze the
fully marginal causal effect of the targeted notifications on self-reported
mood score and step count.

We considered three estimators for the fully marginal treatment effect
$\beta^{\star}_{0}$. Estimator I is the unadjusted estimator that uses only
the time-varying outcome and treatment assignment data. Estimator II is
obtained using the WCLS method (Boruvka et al., 2018) with the outcome of the
previous week $Z_{t,j}:=Y_{t,j}$ as a control variable. Specifically, for mood
score outcomes, we selected the previous week’s average mood score as the
control variable, and for step count outcomes, we chose the prior week’s
average step count. Estimator III is obtained by further adjusting $Z_{t,j}$
as an auxiliary variable using the proposed A2-WCLS method, with the working
centering function defined as
$\mu_{t}=\mathbb{P}_{N}[Z_{t,j}]=\mathbb{P}_{N}[Y_{t,j}]$.

We report the estimates in Figure 1 and present more details in Table 12 of
Appendix N. Compared to Estimator I, Estimators II & III have a tangible
improvement in standard error estimates, with a relative estimation efficiency
of 2.32, 2.33 for the mood outcome and 1.57, 1.58 for the step outcome. Even
though Estimator III using A2-WCLS does not gain considerable efficiency over
Estimator II (WCLS) results, it is not likely to lose efficiency. We conclude
that sending activity notifications can increase (the cubic root of) step
counts by 0.072, with statistical significance at level 95%. However, we do
not observe a significant effect of sending mood notifications on users’
moods.

Figure 1: Causal effect estimates with the 95% confidence interval, and
standard errors

###  7.2 Time-Varying Treatment Effect Estimation

In most mobile health intervention studies, time-in-study has always been an
important moderator of treatment effects. Therefore, we include study week in
the marginal treatment effect model for further investigation:
$\beta(t)=\beta_{0}^{\star}+\beta_{1}^{\star}t$. The auxiliary variables are
still chosen to be the corresponding outcome in the previous week. Control
variables have been established in Section 7.1 as being beneficial in reducing
standard errors, so control variables (prior week’s outcome) will always be
included in the following analysis.

Figure 2 below shows estimated time-varying treatment moderation effects. We
compare our proposed approach against the WCLS method from Boruvka et al.
(2018). More details of the moderated analysis are presented in Table 13 of
Appendix N. The shaded area represents the 95% confidence band of the
moderation effects at varying values of the moderator (week-in-study). A
narrower confidence band is observed when A2-WCLS is used; specifically, the
relative efficiency for $\hat{\beta}_{0},\hat{\beta}_{1}$ in the mood model is
2.204 and 2.157, respectively, and in the step model is 1.562 and 1.475,
respectively.

There is an overall decreasing trend in the effectiveness of the treatments on
both proximal outcomes with study week. The scientific rationale for this
trend may be habituation to or burden caused by the targeted reminders.
Furthermore, the causal excursion effect of mobile prompts for step count
change is positive and statistically significant in the first several weeks of
the study. This suggests that sending targeted reminders may be beneficial to
increasing physical activity in the short term. Note that the estimated effect
using A2-WCLS is significant until week 18, while using WCLS the effect is
only significant until week 12. This demonstrates the importance of including
auxiliary variables for improved precision in causal moderation analysis. Both
analyses show that in later stages of the study, the effect is no longer
significant, which may be due to habituation to and/or burden caused by the
smartphone reminders.

Figure 2: Causal effect estimates with the 95% confidence interval.

##  8 Discussion

MRTs are frequently adopted in mobile health studies to assess the
effectiveness of just-in-time intervention components. Although abundant
auxiliary variables are often collected in these studies through wearables and
smartphones, their statistical utility for enhancing causal effect analysis
has not kept pace with its counterpart in classical RCTs. To fill this gap, we
introduce an approach (A2-WCLS) that incorporates auxiliary variables into the
estimation of moderated causal excursion effects. This work addresses the
long-standing problem of “how to” and “what is the benefit of” leveraging
auxiliary information in the context of repeated measurements and time-varying
treatment effects, subsuming existing popular covariate adjustment methods in
classical RCTs as a special case (Section 3).

Under some interpretable conditions in time-varying settings, the proposed
A2-WCLS method offers improved efficiency for moderated causal excursion
effect estimation in comparison to the benchmark WCLS methods. In Appendix O,
we provide clear guidelines for practitioners to implement auxiliary variable
adjustment in longitudinal studies.

Many open questions remain. First, we may also use feature selection methods
to regularize the centering function $\mu_{t}(S_{t})$, e.g., by the smooth
clipped absolute deviation method (SCAD) (Fan and Li, 2001), to pick an
optimal model for $\mu_{t}(S_{t})$. A non-linear centering function can also
be considered, similar to Guo and Basse (2023) proposed for RCTs. Second, in
addition to using auxiliary variables to improve estimation efficiency, we may
also consider flexible modeling of how the control variables $g(H_{t})$ enter
the objective function of A2-WCLS. There is a wide variety of machine learning
methods that can be used for this purpose. Third, if the MRT has a clustered
structure (interference within clusters and/or treatment heterogeneity at the
cluster level), it requires new centering techniques using cluster-level
auxiliary variables (Shi et al., 2023). Finally, more advanced and flexible
MRT designs should be explored so that they may use auxiliary variable
adjustment to maximize efficiency. For example, Van Lancker et al. (2022)
proposes using an information adaptive design, which adapts to the amount of
precision gain and can lead to faster, more efficient trials, without
sacrificing validity or power. We leave these directions for future work.

##  Appendix A Proof of Lemma 3.2

###  A.1 Asymptotic Properties of Nuisance Parameters $\alpha_{0,t}$ and
$\alpha_{1,t}$

Here we first sketch a proof that, given known randomization probability
$p_{t}$ and the centered control variables $\tilde{g}_{t}(H_{t})^{\top}$, the
time-specific intercepts $\alpha_{0,t}$ and the coefficients $\alpha_{1,t}$
have the same limit value under objective functions displayed in (4), (5), and
(6). Assume partial derivative and expectation calculations can always be
interchangeable. The closed-form estimation for $\alpha_{0,t}$ can be written
as:

$\displaystyle\alpha^{\star}_{0,t}$
$\displaystyle=\mathbb{E}\left[\left(Y_{t+1}-\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}-\left(A_{t}-p_{t}\right)(\beta_{0,t}+\tilde{g}_{t}(H_{t})^{\top}\beta_{1,t})\right)\right]$
$\displaystyle=\mathbb{E}\left[Y_{t+1}\right]-\mathbb{E}\left[\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}\right]-\mathbb{E}\left[\beta_{0,t}\left(A_{t}-p_{t}\right)\right]-\mathbb{E}\left[\left(A_{t}-p_{t}\right)\left(\tilde{g}_{t}(H_{t})^{\top}\beta_{1,t}\right)\right]$
$\displaystyle=\mathbb{E}\left[Y_{t+1}\right],$

which remains invariant regardless of the covariates adjusted in the
regression model. The rationale lies in both
$\mathbb{E}[\tilde{g}_{t}(H_{t})]=0$ and $\mathbb{E}[A_{t}-p_{t}]=0$. Besides,
we have the limit value for $\alpha_{1,t}$ as:

$\displaystyle\alpha^{\star}_{1,t}$
$\displaystyle=\Sigma(g_{t}(H_{t}))^{-1}\mathbb{E}\left[\tilde{g}_{t}(H_{t})\left(Y_{t+1}-\left(A_{t}-p_{t}\right)(\beta_{0,t}+\tilde{g}_{t}(H_{t})^{\top}\beta_{1,t})\right)\right]$
$\displaystyle=\Sigma(g_{t}(H_{t}))^{-1}\Big{\\{}\mathbb{E}\left[\tilde{g}_{t}(H_{t})Y_{t+1}\right]-\mathbb{E}\left[\beta_{0,t}\tilde{g}_{t}(H_{t})\left(A_{t}-p_{t}\right)\right]-\Sigma(g_{t}(H_{t}))\mathbb{E}\left[\left(A_{t}-p_{t}\right)\beta_{1,t}\right]\Big{\\}}$
$\displaystyle=\Sigma(g_{t}(H_{t}))^{-1}\mathbb{E}\left[\tilde{g}_{t}(H_{t})Y_{t+1}\right].$

As the objective functions presented in (5) and (6) both use the same _control
variables_ $\tilde{g}_{t}(H_{t})$, the resulting estimates
$\hat{\alpha}_{1,t}$ from different methods are expected to converge to a
common value.

###  A.2 Asymptotic Variance Comparison Between $\hat{\beta}_{0,t}^{\text{U}}$
and $\hat{\beta}_{0,t}^{\text{WCLS}}$

For the unadjusted estimator $\hat{\beta}_{0,t}^{\text{U}}$, the asymptotic
variance can be consistently estimated by
$Q_{t}^{-1}\Sigma_{t}^{\text{U}}Q_{t}^{-1}$:

$\begin{split}Q_{t}&=p_{t}(1-p_{t})\\\
\epsilon_{t}^{\text{U}}&=Y_{t+1}-\alpha_{0,t}-\left(A_{t}-p_{t}\right)\beta_{0,t}\\\
\Sigma_{t}^{\text{U}}&=\mathbb{E}\big{[}\left(\epsilon_{t}^{\text{U}}(A_{t}-p_{t})\right)^{2}\big{]}\end{split}$
(13)

For the WCLS estimator $\hat{\beta}_{0,t}^{\text{WCLS}}$, the asymptotic
variance can be consistently estimated by
$Q_{t}^{-1}\Sigma_{t}^{\text{WCLS}}Q_{t}^{-1}$:

$\begin{split}Q_{t}&=p_{t}(1-p_{t})\\\
\epsilon_{t}^{\text{WCLS}}&=Y_{t+1}-\alpha_{0,t}-\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}-\left(A_{t}-p_{t}\right)\beta_{0,t}\\\
\Sigma_{t}^{\text{WCLS}}&=\mathbb{E}\big{[}\left(\epsilon_{t}^{\text{WCLS}}(A_{t}-p_{t})\right)^{2}\big{]}\end{split}$
(14)

For convenience, we center the control variables $g_{t}(H_{t})$ around their
means, without loss of generality. Let $\tilde{g}_{t}(H_{t})$ represent the
centered variables. To ensure clarity in the proof, we assume that
$\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}$ does not include the intercept term.
Since the outer term $Q_{t}$ is the same for both asymptotic variance
expressions, therefore, we only need to investigate the difference between
$\Sigma_{t}^{\text{WCLS}}$ and $\Sigma_{t}^{\text{U}}$:

$\displaystyle\Sigma_{t}^{\text{WCLS}}$
$\displaystyle=\mathbb{E}\big{[}\left((\epsilon_{t}^{\text{U}}-\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t})(A_{t}-p_{t})\right)^{2}\big{]}$
$\displaystyle=\Sigma_{t}^{\text{U}}+\mathbb{E}\big{[}\left(\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}(A_{t}-p_{t})\right)^{2}\big{]}-2\mathbb{E}\big{[}\epsilon_{t}^{\text{U}}\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}(A_{t}-p_{t})^{2}\big{]}$

The expansion shows that the difference between $\Sigma_{t}^{\text{WCLS}}$ and
$\Sigma_{t}^{\text{U}}$ is:

$\mathbb{E}\big{[}\left(\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}(A_{t}-p_{t})\right)^{2}\big{]}-2\mathbb{E}\big{[}\epsilon_{t}^{\text{U}}\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}(A_{t}-p_{t})^{2}\big{]}$
(15)

The first quadratic term is given by:

$\displaystyle\mathbb{E}\big{[}(A_{t}-p_{t})^{2}\alpha_{1,t}^{\top}\tilde{g}_{t}(H_{t})\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}\big{]}=p_{t}(1-p_{t})\alpha_{1,t}^{\top}\Sigma(g_{t}(H_{t}))\alpha_{1,t}$

where
$\Sigma(g_{t}(H_{t}))=\mathbb{E}\big{[}\tilde{g}_{t}(H_{t})\tilde{g}_{t}(H_{t})^{\top}\big{]}\in\mathbb{R}^{d\times
d}$ represents the variance-covariance matrix for the control variables
$g_{t}(H_{t})$. The second interaction term could be further simplified as:

$\displaystyle\mathbb{E}\big{[}\epsilon_{t}^{\text{U}}\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}(A_{t}-p_{t})^{2}\big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\big{[}\left(\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,H_{t}]p_{t}(1-p_{t})^{2}+\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,H_{t}]p_{t}^{2}(1-p_{t})\right)\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}\big{]}$
$\displaystyle=$ $\displaystyle
p_{t}(1-p_{t})\mathbb{E}\big{[}\left(\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,H_{t}](1-p_{t})+E[\epsilon_{t}^{\text{U}}|A_{t}=0,H_{t}]p_{t}\right)\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}\big{]}$
$\displaystyle=$ $\displaystyle
p_{t}(1-p_{t})\Big{(}\mathbb{E}\big{[}\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,H_{t}]\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}\big{]}$
$\displaystyle-$ $\displaystyle
p_{t}\mathbb{E}\big{[}(\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,H_{t}]-\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,H_{t}])\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}\big{]}\Big{)}.$

It rests to study the following terms:

$\displaystyle\mathbb{E}\big{[}\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,H_{t}]\tilde{g}_{t}(H_{t})^{\top}\big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\big{[}\mathbb{E}[Y_{t+1}-(1-p_{t})\beta_{0,t}|A_{t}=1,H_{t}]\tilde{g}_{t}(H_{t})^{\top}\big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\big{[}\mathbb{E}[Y_{t+1}|A_{t}=1,H_{t}]\tilde{g}_{t}(H_{t})^{\top}\big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\big{[}(\mathbb{E}[Y_{t+1}|A_{t}=1,H_{t}]-\mathbb{E}[Y_{t+1}|H_{t}]+\mathbb{E}[Y_{t+1}|H_{t}])\tilde{g}_{t}(H_{t})^{\top}\big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\big{[}(\mathbb{E}[Y_{t+1}|A_{t}=1,H_{t}]-\mathbb{E}[Y_{t+1}|H_{t}])\tilde{g}_{t}(H_{t})^{\top}\big{]}+\alpha_{1,t}^{\top}\Sigma(g_{t}(H_{t}))$
$\displaystyle=$
$\displaystyle(1-p_{t})\beta_{1,t}^{\top}\Sigma(g_{t}(H_{t}))+\alpha_{1,t}^{\top}\Sigma(g_{t}(H_{t})),$

and similarly

$\displaystyle\mathbb{E}\big{[}\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,H_{t}]\tilde{g}_{t}(H_{t})^{\top}\big{]}=-p_{t}\beta_{1,t}^{\top}\Sigma(g_{t}(H_{t}))+\alpha_{1,t}^{\top}\Sigma(g_{t}(H_{t}))$

Then combining these two terms together, we have:

$\displaystyle\Sigma_{t}^{\text{WCLS}}-\Sigma_{t}^{\text{U}}$
$\displaystyle=-p_{t}(1-p_{t})\alpha_{1,t}^{\top}\Sigma(g_{t}(H_{t}))\left(\alpha_{1,t}+2(1-2p_{t})\beta_{1,t}\right)$
(16)

###  A.3 More on the Nuisance Parameters $\alpha_{0,t}$ and $\alpha_{1,t}$

It is natural to consider the estimation of nuisance parameters $\alpha_{0,t}$
and $\alpha_{1,t}$ and their potential optimality in minimizing the asymptotic
variances of $\hat{\beta}_{0,t}^{\text{U}}$ and
$\hat{\beta}_{0,t}^{\text{WCLS}}$. To address this, we provide the proof that
the estimators of $\alpha_{0,t}$ and $\alpha_{1,t}$ obtained using objective
functions (4) and (5) are exactly the optimal value one can get to minimize
the asymptotic variance of $\text{Var}(\hat{\beta}_{0,t}^{\text{U}})$ and
$\text{Var}(\hat{\beta}_{0,t}^{\text{WCLS}})$. Here is a sketch of the
illustration using the first case:

The asymptotic variance of the unadjusted estimator, which is shown in the
previous section, is the following:

$\displaystyle\Sigma_{t}^{\text{U}}=\mathbb{E}\big{[}\left(\epsilon_{t}^{\text{U}}(A_{t}-p_{t})\right)^{2}\big{]},$

where
$\epsilon_{t}^{\text{U}}=Y_{t+1}-\alpha_{0,t}-\left(A_{t}-p_{t}\right)\beta_{0,t}$.
Thus the “optimal” $\alpha_{0,t}$ can be obtained:

$\displaystyle\operatorname*{arg\,min}_{\alpha_{0,t}}\mathbb{E}\big{[}(A_{t}-p_{t})^{2}(Y_{t+1}-\alpha_{0,t}-\left(A_{t}-p_{t}\right)\beta_{0,t})^{2}\big{]},$

which is equivalent to solving:

$\mathbb{E}\big{[}(A_{t}-p_{t})^{2}(Y_{t+1}-\alpha_{0,t}-\left(A_{t}-p_{t}\right)\beta_{0,t})\big{]}=0.$
(17)

For the causal parameter $\beta_{0,t}$ to be consistently estimated, it is
essential for this condition to be met in conjunction with the following
estimating equation:

$\mathbb{E}\big{[}(A_{t}-p_{t})(Y_{t+1}-\alpha_{0,t}-\left(A_{t}-p_{t}\right)\beta_{0,t})\big{]}=0.$
(18)

By solving (17) and (18) simultaneously, we got
$\alpha_{0,t}=\mathbb{E}[Y_{t+1}]$ and
$\beta_{0,t}=\mathbb{E}[Y_{t+1}|A_{t}=1]-\mathbb{E}[Y_{t+1}|A_{t}=0]$, which
matches with the limit value of $\alpha_{0,t}$ provided in Appendix A.1. We
have the same conclusion for $\alpha_{1,t}$.

##  Appendix B Proof of Lemma 3.3

###  B.1 Asymptotic Properties of $\hat{\beta}_{0,t}^{\text{L}}$

As per Lin (2013); Boruvka et al. (2018), $\hat{\beta}_{0,t}^{\text{L}}$
obtained by minimizing (6) is consistent and asymptotically normal. We proceed
with the proof by centering $g_{t}(H_{t})$ at its mean and denoting it as
$\tilde{g}_{t}(H_{t})$. And the asymptotic variance can be consistently
estimated by $Q_{t}^{-1}\Sigma_{t}^{\text{L}}Q_{t}^{-1}$, where

$\begin{split}Q_{t}&=p_{t}(1-p_{t})\\\
\epsilon_{t}^{\text{L}}&=Y_{t+1}-\alpha_{0,t}-\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}-\left(A_{t}-p_{t}\right)(\beta_{0,t}+\tilde{g}_{t}(H_{t})\beta_{1,t})\\\
\Sigma_{t}^{\text{L}}&=\mathbb{E}\big{[}\left(\epsilon_{t}^{\text{L}}(A_{t}-p_{t})\right)^{2}\big{]}\end{split}$
(19)

###  B.2 Asymptotic Variance Comparison Between $\hat{\beta}_{0,t}^{\text{L}}$
and $\hat{\beta}_{0,t}^{\text{U}}$

Since the outer term $Q_{t}$ is the same for both asymptotic variance
expressions, therefore, we only need to investigate the difference between
$\Sigma_{t}^{\text{L}}$ and $\Sigma_{t}^{\text{U}}$.

$\displaystyle\Sigma_{t}^{\text{L}}$
$\displaystyle=\mathbb{E}\big{[}\left((\epsilon_{t}^{\text{U}}-\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}-(A_{t}-p_{t})\tilde{g}_{t}(H_{t})^{\top}\beta_{1,t})(A_{t}-p_{t})\right)^{2}\big{]}$
$\displaystyle=\Sigma_{t}^{\text{U}}+\mathbb{E}\big{[}\left(\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}+(A_{t}-p_{t})\tilde{g}_{t}(H_{t})^{\top}\beta_{1,t}\right)^{2}(A_{t}-p_{t})^{2}\big{]}$
$\displaystyle-2\mathbb{E}\big{[}\epsilon_{t}^{\text{U}}(\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}+(A_{t}-p_{t})\tilde{g}_{t}(H_{t})^{\top}\beta_{1,t})(A_{t}-p_{t})^{2}\big{]}$

The expansion shows that the difference between $\Sigma_{t}^{\text{L}}$ and
$\Sigma_{t}^{\text{U}}$ is:

$\begin{split}&\mathbb{E}\big{[}\left(\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}+(A_{t}-p_{t})\tilde{g}_{t}(H_{t})^{\top}\beta_{1,t}\right)^{2}(A_{t}-p_{t})^{2}\big{]}\\\
&-2\mathbb{E}\big{[}\epsilon_{t}^{\text{U}}(\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}+(A_{t}-p_{t})\tilde{g}_{t}(H_{t})^{\top}\beta_{1,t})(A_{t}-p_{t})^{2}\big{]}\end{split}$
(20)

The first quadratic term is given by:

$\displaystyle\mathbb{E}\big{[}\left(\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}+(A_{t}-p_{t})\tilde{g}_{t}(H_{t})^{\top}\beta_{1,t}\right)^{2}(A_{t}-p_{t})^{2}\big{]}$
$\displaystyle=$ $\displaystyle
p_{t}(1-p_{t})\big{[}\alpha_{1,t}^{\top}\Sigma(g_{t}(H_{t}))\alpha_{1,t}+2(1-2p_{t})\alpha_{1,t}^{\top}\Sigma(g_{t}(H_{t}))\beta_{1,t}+(1-3p_{t}+3p_{t}^{2})\beta_{1,t}^{\top}\Sigma(g_{t}(H_{t}))\beta_{1,t}\big{]}$

The second interaction term could be further simplified as:

$\displaystyle\mathbb{E}\big{[}\epsilon_{t}^{\text{U}}(\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}+(A_{t}-p_{t})\tilde{g}_{t}(H_{t})^{\top}\beta_{1,t})(A_{t}-p_{t})^{2}\big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\big{[}\left(\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,H_{t}]p_{t}(1-p_{t})^{2}+\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,H_{t}]p_{t}^{2}(1-p_{t})\right)\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}\big{]}$
$\displaystyle+\mathbb{E}\big{[}\left(\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,H_{t}]p_{t}(1-p_{t})^{3}-\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,H_{t}]p_{t}^{3}(1-p_{t})\right)\tilde{g}_{t}(H_{t})^{\top}\beta_{1,t}\big{]}$
$\displaystyle=$ $\displaystyle
p_{t}(1-p_{t})\mathbb{E}\big{[}\left(\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,H_{t}](1-p_{t})+\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,H_{t}]p_{t}\right)\tilde{g}_{t}(H_{t})^{\top}\alpha_{1,t}\big{]}$
$\displaystyle+p_{t}(1-p_{t})\mathbb{E}\big{[}\left(\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,H_{t}](1-p_{t})^{2}-\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,H_{t}]p^{2}_{t}\right)\tilde{g}_{t}(H_{t})^{\top}\beta_{1,t}\big{]}$

Recall:

$\displaystyle\mathbb{E}\big{[}\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,H_{t}]\tilde{g}_{t}(H_{t})^{\top}\big{]}$
$\displaystyle=(1-p_{t})\beta_{1,t}^{\top}\Sigma(g_{t}(H_{t}))+\alpha_{1,t}^{\top}\Sigma(g_{t}(H_{t}))$
$\displaystyle\mathbb{E}\big{[}\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,H_{t}]\tilde{g}_{t}(H_{t})^{\top}\big{]}$
$\displaystyle=-p_{t}\beta_{1,t}^{\top}\Sigma(g_{t}(H_{t}))+\alpha_{1,t}^{\top}\Sigma(g_{t}(H_{t}))$

Then combing these two terms together, we have
$\Sigma_{t}^{\text{L}}-\Sigma_{t}^{\text{U}}$ equals to:

$\displaystyle-
p_{t}(1-p_{t})\big{[}\left(\alpha_{1,t}+(1-2p_{t})\beta_{1,t}\right)^{\top}\Sigma(g_{t}(H_{t}))\left(\alpha_{1,t}+(1-2p_{t})\beta_{1,t}\right)+p_{t}(1-p_{t})\beta_{1,t}^{\top}\Sigma(g_{t}(H_{t}))\beta_{1,t}\big{]},$
(21)

which is guaranteed to be non-positive.

###  B.3 Efficiency Comparison Between $\hat{\beta}_{0,t}^{\text{L}}$ and
$\hat{\beta}_{0,t}^{\text{WCLS}}$

Simply applying the differences derived above, namely (16) and (21), we can
easily get the asymptotic variance difference:

$\displaystyle\Sigma_{t}^{\text{L}}-\Sigma_{t}^{\text{WCLS}}=-p_{t}(1-p_{t})\big{[}(1-3p_{t}+3p_{t}^{2})\beta_{1,t}^{\top}\Sigma(g_{t}(H_{t}))\beta_{1,t}\big{]},$
(22)

which is guaranteed to be non-positive.

##  Appendix C Proof of Condition 3.5

To illustrate the derivation of the orthogonality condition in detail, we
write down the derivatives of (7) and (3) with respect to $\beta_{0}$. The
derivative of (3) with respect to $\beta_{0}$ is:

$\mathbb{E}\left[\sum_{t=1}^{T}W_{t}(Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-(A_{t}-\tilde{p}_{t})f_{t}(S_{t})^{\top}\beta_{0})(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\right]$
(23)

The derivative of (7) with respect to $\beta_{0}$ is:

$\mathbb{E}\left[\sum_{t=1}^{T}W_{t}(Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-(A_{t}-\tilde{p}_{t})(f_{t}(S_{t})^{\top}\beta_{0}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}))(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\right]$
(24)

The intuition is that if the difference between (23) and (24) is always zero
(not depending on the form of $\mu_{t}(S_{t})$), then both of them should have
the same minimizer for $\beta_{0}$:

$\mathbb{E}\left[\sum_{t=1}^{T}W_{t}(A_{t}-\tilde{p}_{t})^{2}f_{t}(S_{t})(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}\right]=\mathbf{0}_{q\times
1}$

After calculating the expectation of the term above, we can obtain the form of
orthogonality constraint shown in Condition 3.5. For each
$Z^{i}_{t}-\mu^{i}_{t}(S_{t})$ we wish to include:

$\mathbb{E}\left[\sum_{t=1}^{T}\tilde{p}_{t}(1-\tilde{p}_{t})f_{t}(S_{t})(Z^{i}_{t}-\mu^{i}_{t}(S_{t}))^{\top}\right]=\mathbf{0}_{q\times
1}$

###  C.1 Neyman Orthogonality

We are interested in the true value $\beta_{0}^{\star}$ of the low-dimensional
target parameter $\beta_{0}$. Therefore in the estimation procedure,
$\beta_{1}$ and $\alpha$ are treated as nuisance parameters. First, we assume
that $\beta_{0}^{\star}$ satisfies the moment conditions:

$\mathbb{E}[\psi(\beta_{0}^{\star};\beta_{1}^{\star},\alpha^{\star})]=0$

where $\psi(\beta_{0};\beta_{1},\alpha)$ is a known score function. In our
case, $\psi(\beta_{0};\beta_{1},\alpha)$ is exactly the estimating equation
for $\beta_{0}$:

$\sum_{t=1}^{T}W_{t}\big{[}Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-(A_{t}-\tilde{p}_{t})(f_{t}(S_{t})^{\top}\beta_{0}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1})\big{]}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})$

Here we further require the Neyman orthogonality condition for the score
$\psi$. By the definition given in Chernozhukov et al. (2018b), the Gateaux
derivative operator with respect to $\beta_{1}$ is:

$G(\beta_{1})=\mathbb{E}\left[\sum_{t=1}^{T}W_{t}(A_{t}-\tilde{p}_{t})^{2}f_{t}(S_{t})(Z_{t}-\mu_{t}(S_{t}))^{\top}\right](\beta_{1}-\beta_{1,0})$

which should vanish when evaluated at the true parameter values
$\beta_{1,0}=\beta_{1}^{\star}$, which implies that:

$\mathbb{E}\left[\sum_{t=1}^{T}\tilde{p}_{t}(1-\tilde{p}_{t})f_{t}(S_{t})(Z_{t}-\mu_{t}(S_{t}))^{\top}\right]=\mathbf{0}_{q\times
p}$

Similarly, we can obtain the form of orthogonality constraint shown in
Condition 3.5.

##  Appendix D Proof of Lemma 3.7

The A2-WCLS criterion is:

$\mathbb{P}_{n}\left[\sum_{t=1}^{T}W_{t}\times\left(Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-\left(A_{t}-\tilde{p}_{t}(1\mid
S_{t})\right)(f_{t}(S_{t})^{\top}\beta_{0}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1})\right)^{2}\right]$

which can be expressed as the following estimating equation:

$\displaystyle
U(\alpha,\beta_{0},\beta_{1};\mu_{t})=\mathbb{E}\Bigg{[}\sum_{t=1}^{T}W_{t}(Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-$
$\displaystyle(A_{t}-\tilde{p}_{t})(f_{t}(S_{t})^{\top}\beta_{0}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}))$
$\displaystyle\times\left(\begin{array}[]{c}g_{t}(H_{t})\\\
(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\\\
(A_{t}-\tilde{p}_{t})(Z_{t}-\mu_{t}(S_{t}))\end{array}\right)\Bigg{]}$ (28)

Assuming the centering functions $\mu_{t}(S_{t})$ satisfy the orthogonality
condition 3.5, then we have:

$\displaystyle\mathbb{E}\left[\sum_{t=1}^{T}W_{t}(Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-(A_{t}-\tilde{p}_{t})(f_{t}(S_{t})^{\top}\beta_{0}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}))(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\right]$
$\displaystyle=$
$\displaystyle\mathbb{E}\left[\sum_{t=1}^{T}W_{t}(Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-(A_{t}-\tilde{p}_{t})f_{t}(S_{t})^{\top}\beta_{0})(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\right]$
$\displaystyle-\mathbb{E}\left[\sum_{t=1}^{T}W_{t}(A_{t}-\tilde{p}_{t})^{2}f_{t}(S_{t})(Z_{t}-\mu_{t}(S_{t}))^{\top}\right]\beta_{1}~{}~{}~{}~{}\text{(orthogonality
condition)}$ $\displaystyle=$
$\displaystyle\mathbb{E}\left[\sum_{t=1}^{T}W_{t}(Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-(A_{t}-\tilde{p}_{t})f_{t}(S_{t})^{\top}\beta_{0})(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\right]~{}~{}~{}~{}~{}\text{(WCLS
estimating equation)}$

Based on the asymptotic properties proved in Boruvka et al. (2018),
$\hat{\beta}_{0}^{\text{A2}}$ is consistent and asymptotically normal, and the
robust sandwich variance-covariance estimator is applied here to consistently
estimate the asymptotic variance of $\hat{\beta}_{0}^{\text{A2}}$, which can
be retrieved from the middle block diagonal $(q\times q)$ matrix:

$Q^{-1}\Sigma
Q^{-1}=\mathbb{E}\left[\dot{U}(\alpha,\beta_{0},\beta_{1};\mu_{t})\right]^{-1}\mathbb{E}\left[U(\alpha,\beta_{0},\beta_{1};\mu_{t})\right]^{\otimes
2}\mathbb{E}\left[\dot{U}(\alpha,\beta_{0},\beta_{1};\mu_{t})\right]^{-1}$

Specifically, for the WCLS criterion:

$\begin{split}Q_{\beta_{0}}&=\mathbb{E}\left[\sum_{t=1}^{T}\tilde{p}_{t}(1-\tilde{p}_{t})f_{t}(S_{t})f_{t}(S_{t})^{\top}\right],\\\
\epsilon_{t}^{\text{WCLS}}&=Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-\left(A_{t}-\tilde{p}_{t}\right)f_{t}(S_{t})^{\top}\beta_{0},\\\
\Sigma_{\beta_{0}}^{\text{WCLS}}&=\mathbb{E}\left[\left(\sum_{t=1}^{T}W_{t}\epsilon_{t}^{\text{WCLS}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\right)\left(\sum_{t=1}^{T}W_{t}\epsilon_{t}^{\text{WCLS}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\right)^{\top}\right].\end{split}$
(29)

For the A2-WCLS criterion:

$\begin{split}Q_{\beta_{0}}&=\mathbb{E}\left[\sum_{t=1}^{T}\tilde{p}_{t}(1-\tilde{p}_{t})f_{t}(S_{t})f_{t}(S_{t})^{\top}\right],\\\
\epsilon_{t}^{\text{A2}}&=Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-\left(A_{t}-\tilde{p}_{t}\right)(f_{t}(S_{t})^{\top}\beta_{0}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}),\\\
\Sigma_{\beta_{0}}^{\text{A2}}&=\mathbb{E}\left[\left(\sum_{t=1}^{T}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\right)\left(\sum_{t=1}^{T}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\right)^{\top}\right].\end{split}$
(30)

###  D.1 Bidirectional Asymptotics

The asymptotic property discussed in the main text relies on $N$ approaching
infinity. However, in certain situations, MRTs may have considerably smaller
sample sizes $N$ compared to the total number of time points $T$. Therefore,
it is crucial to examine how the proposed estimator behaves as $T$ increases.

In this section, we derive the bidirectional asymptotics for the proposed
A2-WCLS estimator, requiring either sample size or the number of decision
points to go to infinity (Yu et al., 2023; Liu et al., 2023). Several crucial
considerations come to the forefront when considering the scenario where $T$
approaches infinity. First, the sample must be sufficiently representative to
facilitate an accurate estimation of the population-level causal effect.
Second, while increasing $T$ may seem advantageous, it does not guarantee more
informative estimates of $\beta_{0}$ under the practical constraint of
declining engagement over time. Lastly, careful attention must be given to the
effect moderation model $f_{t}(S_{t})$, as an increasing $t$ may lead to a
diverging causal excursion effect estimation if $f_{t}(S_{t})$ is set to
change with time.

Building on the asymptotic property discussed in the main text, we now
consider the scenario when $T\rightarrow\infty$. The A2-WCLS criterion can be
modified by solving:

$\frac{1}{NT}\sum_{j=1}^{N}\sum_{t=1}^{T}m(H_{t,j},A_{t,j};\alpha,\beta_{0},\beta_{1})=0$
(31)

where:

$\displaystyle m(H_{t},A_{t};\alpha,\beta_{0},\beta_{1})$
$\displaystyle=W_{t}\Big{(}Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-\left(A_{t}-\tilde{p}_{t}\right)(f_{t}(S_{t})^{\top}\beta_{0}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1})\Big{)}$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\times\left(\begin{array}[]{c}g_{t}(H_{t})\\\
(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\\\
(A_{t}-\tilde{p}_{t})(Z_{t}-\mu_{t}(S_{t})\end{array}\right).$

We extract the estimating equations specifically for the causal parameter
$\beta_{0}$ and denote:

$\displaystyle m_{\beta_{0}}(H_{t},A_{t};\alpha,\beta_{0},\beta_{1})$
$\displaystyle=W_{t}\Big{(}Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-\left(A_{t}-\tilde{p}_{t}\right)(f_{t}(S_{t})^{\top}\beta_{0}+$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1})\Big{)}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})$

The centering function $\mu_{t}(S_{t})$ can be obtained by the following
modified orthogonality condition:

######  Condition D.1 (The orthogonality condition for bidirectional
asymptotics).

For the $i$th component of $Z_{t}\in\mathbb{R}^{p}$, denoted
$Z^{i}_{t}\in\mathbb{R}$, $i=1,\ldots,p$, the associated centering function,
denoted $\mu_{t}^{i}(S_{t})$, must satisfy the following orthogonality
condition with respect to $f_{t}(S_{t})\in\mathbb{R}^{q}$:

$\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\big{[}\tilde{p}_{t}(1|S_{t})(1-\tilde{p}_{t}(1|S_{t}))(Z^{i}_{t}-\mu^{i}_{t}(S_{t}))f_{t}(S_{t})\big{]}=\mathbf{0}_{q\times
1}.$ (32)

To make sure the solutions
$(\hat{\alpha},\hat{\beta}_{0}^{\text{A2}},\hat{\beta}_{1})$ of Equation (31)
consistently estimate their corresponding parameters, we require the following
assumption to hold:

######  Assumption D.2.

The estimating equation (31) satisfies

  1. 1.

There exists $\\{\alpha^{\star},\beta_{0}^{\star},\beta^{\star}_{1}\\}$, such
that
$\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}[m(H_{t},A_{t};\alpha^{\star},\beta^{\star}_{0},\beta^{\star}_{1})]=0$.

  2. 2.

The absolute value of the moderated causal effect
$|f_{t}(S_{t})^{\top}\beta_{0}^{\star}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta^{\star}_{1}|$
is bounded almost surely for $\forall t$ by some constant $c_{1}$.

  3. 3.

Define $\dot{m}_{\beta_{0}}(H_{t},A_{t};\alpha,\beta_{0},\beta_{1})=\partial
m_{\beta_{0}}(H_{t};\alpha,\beta_{0},\beta_{1})/\partial\beta_{0}$, then
$\mathbb{E}[\dot{m}_{\beta_{0}}(H_{t},A_{t};\alpha,\beta_{0},\beta_{1})]$ is
bounded almost surely for $\forall t$ by some constant $c_{2}$.

  4. 4.

$\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{t=1}^{T}\text{Var}\big{[}m_{\beta_{0}}(H_{t},A_{t};\alpha,\beta_{0},\beta_{1})|H_{t},A_{t}\big{]}\rightarrow\Gamma_{\beta_{0}}$,
where $\Gamma_{\beta_{0}}$ is a constant positive-definite matrix.

A sufficient condition for Assumption D.2 to hold is assuming that the outcome
regression linear model is a correct model specification for
$\mathbb{E}[Y_{t+1}|H_{t},A_{t}]$. Thus $m(H_{t};\alpha,\beta_{0},\beta_{1})$
is a martingale difference sequence with respect to sequence
$\\{H_{t},A_{t}\\}$. In contrast to the finite-horizon A2-WCLS estimating
equation defined in (7), Equation (31) requires a more stringent assumption on
modeling accuracy.

Assumption D.2 guarantees the following limiting values exist:

$\displaystyle B_{\beta_{0}}$
$\displaystyle=\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}[\dot{m}_{\beta_{0}}(H_{t},A_{t};\alpha,\beta_{0},\beta_{1})]$
$\displaystyle\Gamma_{\beta_{0}}$
$\displaystyle=\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\big{[}m_{\beta_{0}}(H_{t},A_{t};\alpha,\beta_{0},\beta_{1})m^{\top}_{\beta_{0}}(H_{t},A_{t};\alpha,\beta_{0},\beta_{1})|H_{t},A_{t}\big{]}$

Then we obtain the following lemma on the bidirectional asymptotic property of
the proposed A2-WCLS estimator:

######  Lemma D.3.

Suppose Assumptions 2.1, 3.1 and D.2 hold, centering function $\mu_{t}(S_{t})$
satisfies the orthogonality condition D.1, and that the randomization
probability $p_{t}(A_{t}|H_{t})$ is known. Given the notation mentioned above,
let $(\hat{\alpha},\hat{\beta}_{0}^{\text{A2}},\hat{\beta}_{1})$ solve
Equation (31). Under regularity conditions, $\hat{\beta}_{0}^{\text{A2}}$ is
consistent and asymptotically normal such that either $N\rightarrow\infty$ or
$T\rightarrow\infty$, we have
$\sqrt{NT}(\hat{\beta}^{\text{A2}}_{0}-\beta_{0}^{\star})\rightarrow\mathcal{N}(0,B_{\beta_{0}}^{-1}\Gamma_{\beta_{0}}B_{\beta_{0}}^{-1})$.

Here we sketch the proof. We use $\mathbb{P}_{NT}$ to denote
$\frac{1}{NT}\sum_{j=1}^{N}\sum_{t=1}^{T}$. The subscript $j$ is dropped for
notation ease, and the estimating equation
$m_{\beta_{0}}(H_{t},A_{t};\alpha,\beta_{0},\beta_{1})$ has the following
expansion:

$\displaystyle\mathbb{P}_{NT}\left[W_{t}\left(Y_{t+1}-g_{t}(H_{t})^{\top}\hat{\alpha}-(A_{t}-\tilde{p}_{t})\left(f_{t}(S_{t})^{\top}\hat{\beta}_{0}^{\text{(A2)}}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\hat{\beta}_{1})\right)\right)(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\right]$
$\displaystyle=\mathbb{P}_{NT}\Big{[}W_{t}\Big{(}Y_{t+1}-g_{t}(H_{t})^{\top}\alpha^{\star}-(A_{t}-\tilde{p}_{t}(1|S_{t}))\big{(}f_{t}(S_{t})^{\top}\beta_{0}^{\star}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta^{\star}_{1}\big{)}\Big{)}$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\times(A_{t}-\tilde{p}_{t}(1|S_{t}))f_{t}(S_{t})\Big{]}$
$\displaystyle~{}~{}~{}~{}~{}+\mathbb{P}_{NT}\left[W_{t}(A_{t}-\tilde{p}_{t}(1|S_{t}))g_{t}(H_{t})^{\top}(\alpha^{\star}-\hat{\alpha})f_{t}(S_{t})\right]$
(33)
$\displaystyle~{}~{}~{}~{}~{}-\mathbb{P}_{NT}\left[W_{t}(A_{t}-\tilde{p}_{t}(1|S_{t}))^{2}(Z_{t}-\mu_{t}(S_{t}))^{\top}(\hat{\beta}_{1}-\beta_{1}^{\star})f_{t}(S_{t})\right]$
(34)
$\displaystyle~{}~{}~{}~{}~{}-\mathbb{P}_{NT}\left[W_{t}(A_{t}-\tilde{p}_{t}(1|S_{t}))^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\right](\hat{\beta}_{0}^{\text{(A2)}}-\beta^{\star})$
$\displaystyle=0.$

Term (33) is $o_{p}(1)$ when $NT\rightarrow\infty$ by
$\mathbb{E}[W_{t}(A_{t}-\tilde{p}_{t}(1|S_{t}))f_{t}(S_{t})|H_{t}]=0$ and thus
form a martingale difference sequence; term (34) is $o_{p}(1)$ when
$NT\rightarrow\infty$ by the orthogonality condition D.1. Thus we obtain:

$\displaystyle(NT)^{1/2}(\hat{\beta}_{0}^{\text{(A2)}}-\beta^{\star})=$
$\displaystyle\left\\{\mathbb{P}_{NT}\left[W_{t}(A_{t}-\tilde{p}_{t}(1|S_{t}))^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\right]\right\\}^{-1}\times$
$\displaystyle(NT)^{1/2}\mathbb{P}_{NT}\Big{[}W_{t}\Big{(}Y_{t+1}-g_{t}(H_{t})^{\top}\alpha^{\star}-(A_{t}-\tilde{p}_{t}(1|S_{t}))\times$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\big{(}f_{t}(S_{t})^{\top}\beta_{0}^{\star}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta^{\star}_{1}\big{)}\Big{)}(A_{t}-\tilde{p}_{t}(1|S_{t}))f_{t}(S_{t})\Big{]}$

Furthermore, since we assumed the first inverse term’s limit value exists,
then

$\displaystyle(NT)^{1/2}(\hat{\beta}_{0}^{\text{(A2)}}-\beta^{\star})=$
$\displaystyle(NT)^{1/2}\mathbb{P}_{NT}\Bigg{[}\left\\{\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\left[\tilde{p}_{t}(1|S_{t})(1-\tilde{p}_{t}(1|S_{t}))f_{t}(S_{t})f_{t}(S_{t})^{\top}\right]\right\\}^{-1}\times$
$\displaystyle~{}~{}~{}\Big{(}W_{t}\big{(}Y_{t+1}-g_{t}(H_{t})^{\top}\alpha^{\star}-(A_{t}-\tilde{p}_{t}(1|S_{t}))\times$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}\left(f_{t}(S_{t})^{\top}\beta_{0}^{\star}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta^{\star}_{1}\right)\big{)}(A_{t}-\tilde{p}_{t}(1|S_{t}))f_{t}(S_{t})\Big{)}\Bigg{]}+o_{p}(1)$

Under Assumption D.2, we have:

$\displaystyle\lim_{NT\rightarrow\infty}\mathbb{P}_{NT}\Big{[}W_{t}\big{(}Y_{t+1}-g_{t}(H_{t})^{\top}\alpha^{\star}-(A_{t}-\tilde{p}_{t}(1|S_{t}))\times$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\left(f_{t}(S_{t})^{\top}\beta_{0}^{\star}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta^{\star}_{1}\right)\big{)}(A_{t}-\tilde{p}_{t}(1|S_{t}))f_{t}(S_{t})\Big{]}=0$

Consequently, under regularity conditions, the estimator
$\hat{\beta}^{(A2)}_{0}\overset{P}{\to}\beta^{\star}$; that is,
$\hat{\beta}^{(A2)}_{0}$ is consistent. The influence function for
$\hat{\beta}^{(A2)}_{0}$ is:

$\displaystyle\left\\{\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\left[\tilde{p}_{t}(1|S_{t})(1-\tilde{p}_{t}(1|S_{t}))f_{t}(S_{t})f_{t}(S_{t})^{\top}\right]\right\\}^{-1}\times$
$\displaystyle~{}~{}\Big{(}W_{t}\big{(}Y_{t+1}-g_{t}(H_{t})^{\top}\alpha^{\star}-(A_{t}-\tilde{p}_{t}(1|S_{t}))\left(f_{t}(S_{t})^{\top}\beta_{0}^{\star}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta^{\star}_{1}\right)\big{)}(A_{t}-\tilde{p}_{t}(1|S_{t}))f_{t}(S_{t})\Big{)},$
(35)

Then under moment conditions mentioned in Dvoretzky (1972) Theorem 2.2, we
have asymptotic normality of the estimator $\hat{\beta}^{(A2)}_{0}$ with
variance given by $B_{\beta_{0}}^{-1}\Gamma_{\beta_{0}}B_{\beta_{0}}^{-1}$.

##  Appendix E Proof of Lemma 3.12

The relationship between two residuals $\epsilon_{t}^{\text{A2}}$ and
$\epsilon_{t}^{\text{WCLS}}$ is:

$\epsilon_{t}^{\text{A2}}=\epsilon_{t}^{\text{WCLS}}-(A_{t}-\tilde{p}_{t})(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}$

Thus, the estimating equation for parameter $\beta_{1}$ from the A2-WCLS
criterion can be written as:

$0=\mathbb{E}\Big{[}\sum_{t=1}^{T}W_{t}(\epsilon_{t}^{\text{WCLS}}-(A_{t}-\tilde{p}_{t})(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1})(A_{t}-\tilde{p}_{t})(Z_{t}-\mu_{t}(S_{t}))\Big{]}$

Solving the equation above, we have:

$\beta_{1}^{\star}(\mu_{t}(S_{t}))=\mathbb{E}\Big{[}\sum_{t=1}^{T}\tilde{p}_{t}(1-\tilde{p}_{t})\Sigma_{\mu}(Z_{t})\Big{]}^{-1}\mathbb{E}\Big{[}\sum_{t=1}^{T}\tilde{p}_{t}(1-\tilde{p}_{t})(\beta(t;Z_{t},S_{t})-\beta(t;S_{t}))(Z_{t}-\mu_{t}(S_{t}))\Big{]}$
(36)

where
$\Sigma_{\mu}(Z_{t})=\mathbb{E}[(Z_{t}-\mu_{t}(S_{t}))(Z_{t}-\mu_{t}(S_{t}))^{\top}]$.
For Lemma 3.12 (a), if $Z_{t}$ is in fact not a moderator, then
$\beta(t;Z_{t},S_{t})-\beta(t;S_{t})=0$ for every $t$. Plug in Equation (36),
we have $\beta_{1}^{\star}=0$.

For Lemma 3.12 (b), we can demonstrate that the first term of
$\beta_{1}^{\star}(\mu_{t}(S_{t}))$ is less than or equal to that of
$\beta_{1}^{\star}$:

$\displaystyle\mathbb{E}\Big{[}\sum_{t=1}^{T}\tilde{p}_{t}(1-\tilde{p}_{t})\Big{(}Z_{t}-\mathbb{E}[Z_{t}|S_{t}]+\mathbb{E}[Z_{t}|S_{t}]-\mu_{t}(S_{t})\Big{)}\Big{(}Z_{t}-\mathbb{E}[Z_{t}|S_{t}]+\mathbb{E}[Z_{t}|S_{t}]-\mu_{t}(S_{t})\Big{)}^{\top}\Big{]}^{-1}$
$\displaystyle\leq$
$\displaystyle\mathbb{E}\Big{[}\sum_{t=1}^{T}\tilde{p}_{t}(1-\tilde{p}_{t})\Big{(}Z_{t}-\mathbb{E}[Z_{t}|S_{t}]\Big{)}\Big{(}Z_{t}-\mathbb{E}[Z_{t}|S_{t}]\Big{)}^{\top}\Big{]}^{-1}$

and the second terms are equal:

$\displaystyle\mathbb{E}\Big{[}\sum_{t=1}^{T}\tilde{p}_{t}(1-\tilde{p}_{t})\mathbb{E}[(\beta(t;Z_{t},S_{t})-\beta(t;S_{t}))(Z_{t}-\mu_{t}(S_{t}))]\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\sum_{t=1}^{T}\tilde{p}_{t}(1-\tilde{p}_{t})(\beta(t;Z_{t},S_{t})-\beta(t;S_{t}))(Z_{t}-\mathbb{E}[Z_{t}|S_{t}])\Big{]}$

Therefore,
$|\beta_{1}^{\star}(\mu_{t})|\leq|\beta_{1}(\mathbb{E}[Z_{t}|S_{t}])|$, which
means that $\beta_{1}^{\star}(\mu_{t})$ underestimates the true moderation
effect.

###  E.1 Simulation results for Lemma 3.12 (a)

In Lemma 3.12, we highlighted that incorporating an auxiliary variable $Z_{t}$
that does not serve as an effect moderator does not undermine the current
efficiency. To illustrate, we conducted a simulation study estimating the
fully marginal effect while including a non-effect moderator $Z_{t}$. The
outcomes are summarized in Table 2 below.

Table 2: Robustness of the proposed A2-WCLS method (with $N=250$ and $T=30$). The true value of the parameters is $\beta_{0}^{\star}=-0.2$.  Method | Est | SE | CP | %RE gain | mRE | RSD
---|---|---|---|---|---|---
WCLS | -0.200 | 0.024 | 0.941 | - | - | -
A2-WCLS | -0.200 | 0.024 | 0.941 | 47.3% | 1.000 | 1.000

##  Appendix F Proof of Remark 3.4

When discussing the efficiency gain of estimating time-specific _moderated_
causal effects, we observe a similar efficiency improvement pattern in
obtaining the causal estimator $\beta_{0,t}$ when the causal moderator
$f_{t}(S_{t})$ only contains categorical variables. Slightly abusing
notations, we redefine the previously mentioned notations as follows. First,
for the unadjusted estimator:

$\begin{split}Q_{t}&=\mathbb{E}[p_{t}(1-p_{t})f_{t}(S_{t})f_{t}(S_{t})^{\top}]\\\
\epsilon_{t}^{\text{U}}&=Y_{t+1}-f_{t}(S_{t})^{\top}\alpha_{0,t}-(A_{t}-p_{t})f_{t}(S_{t})^{\top}\beta_{0,t}\\\
\Sigma_{t}^{\text{U}}&=\mathbb{E}\big{[}\left(\epsilon_{t}^{\text{U}}(A_{t}-p_{t})\right)^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}\end{split}$
(37)

Next, for the WCLS estimator:

$\begin{split}\epsilon_{t}^{\text{WCLS}}&=Y_{t+1}-f_{t}(S_{t})^{\top}\alpha_{0,t}-(Z_{t}-\mu_{t}(S_{t}))^{\top}\alpha_{1,t}-(A_{t}-p_{t})f_{t}(S_{t})^{\top}\beta_{0,t}\\\
\Sigma_{t}^{\text{WCLS}}&=\mathbb{E}\big{[}\left(\epsilon_{t}^{\text{WCLS}}(A_{t}-p_{t})\right)^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}\end{split}$
(38)

And, for the Lin (2013) adjusted estimator:

$\begin{split}\epsilon_{t}^{\text{L}}&=Y_{t+1}-f_{t}(S_{t})^{\top}\alpha_{0,t}-(Z_{t}-\mu_{t}(S_{t}))^{\top}\alpha_{1,t}-(A_{t}-p_{t})\big{(}f_{t}(S_{t})^{\top}\beta_{0,t}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1,t}\big{)}\\\
\Sigma_{t}^{\text{L}}&=\mathbb{E}\big{[}\left(\epsilon_{t}^{\text{L}}(A_{t}-p_{t})\right)^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}\end{split}$
(39)

The intuition behind this lies in the fact that the feature vector
$f_{t}(S_{t})$ contains only indicator functions, thus it has only one element
that is non-zero with a value of $1$. Furthermore, the product
$f_{t}(S_{t})f_{t}(S_{t})^{\top}$ is a rank-one $q\times q$ diagonal matrix,
and there is only one diagonal entry that is $1$. Consequently, the same
efficiency comparison holds true for time-specific moderated causal effect
estimation. We provide more details in the following.

###  F.1 Asymptotic variance comparison between
$\hat{\beta}_{0,t}^{\text{WCLS}}$ and $\hat{\beta}_{0,t}^{\text{U}}$

To simplify the notation, let’s assume the auxiliary variable $Z_{t}$ has a
dimension of 1, making $\alpha_{1,t}$ and $\beta_{1,t}$ belong to
$\mathbb{R}$. If we have a p-dimensional auxiliary variable (where $p>1$), we
can iterate the proof p times to arrive at the same conclusion. Since the
outer term $Q_{t}$ is the same for both asymptotic variance expressions,
therefore, we only need to investigate the difference between
$\Sigma_{t}^{\text{WCLS}}$ and $\Sigma_{t}^{\text{U}}$.

$\displaystyle\Sigma_{t}^{\text{WCLS}}$
$\displaystyle=\mathbb{E}\big{[}\left((\epsilon_{t}^{\text{U}}-(Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t})(A_{t}-p_{t})\right)^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle=\Sigma_{t}^{\text{U}}+\mathbb{E}\big{[}\left((Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t}(A_{t}-p_{t})\right)^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}-$
$\displaystyle~{}~{}~{}~{}2\mathbb{E}\big{[}\epsilon_{t}^{\text{U}}(Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t}(A_{t}-p_{t})^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$

The expansion shows that the difference between $\Sigma_{t}^{\text{WCLS}}$ and
$\Sigma_{t}^{\text{U}}$ is:

$\begin{split}&\mathbb{E}\big{[}\left((Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t}(A_{t}-p_{t})\right)^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}-\\\
&~{}~{}~{}2\mathbb{E}\big{[}\epsilon_{t}^{\text{U}}(Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t}(A_{t}-p_{t})^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}\end{split}$
(40)

The first quadratic term is given by:

$\mathbb{E}\big{[}(A_{t}-p_{t})^{2}\alpha_{1,t}^{2}(Z_{t}-\mu_{t}(S_{t}))^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}=p_{t}(1-p_{t})\alpha_{1,t}^{2}\Sigma_{\mu}(Z_{t})f_{t}(S_{t})f_{t}(S_{t})^{\top}$

where
$\Sigma_{\mu}(Z_{t})=\mathbb{E}\big{[}(Z_{t}-\mu_{t}(S_{t}))^{2}\big{]}\in\mathbb{R}$.
The second interaction term could be further simplified as:

$\displaystyle\mathbb{E}\big{[}\epsilon_{t}^{\text{U}}(Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t}(A_{t}-p_{t})^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\big{[}\left(\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,Z_{t},S_{t}]p_{t}(1-p_{t})^{2}+\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,Z_{t},S_{t}]p_{t}^{2}(1-p_{t})\right)$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\times(Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle=$ $\displaystyle
p_{t}(1-p_{t})\mathbb{E}\big{[}\left(\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,Z_{t},S_{t}](1-p_{t})+\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,Z_{t},S_{t}]p_{t}\right)$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\times(Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$

It rests to study the following terms:

$\displaystyle\mathbb{E}\big{[}\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,Z_{t},S_{t}](Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\big{[}\mathbb{E}[Y_{t+1}|A_{t}=1,Z_{t},S_{t}](Z_{t}-\mu_{t}(S_{t}))f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\big{[}(\mathbb{E}[Y_{t+1}|A_{t}=1,Z_{t},S_{t}]-\mathbb{E}[Y_{t+1}|Z_{t},S_{t}]+\mathbb{E}[Y_{t+1}|Z_{t},S_{t}])(Z_{t}-\mu_{t}(S_{t}))f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\big{[}(\mathbb{E}[Y_{t+1}|A_{t}=1,H_{t}]-\mathbb{E}[Y_{t+1}|Z_{t},S_{t}])(Z_{t}-\mu_{t}(S_{t}))f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}+\alpha_{1,t}\Sigma_{\mu}(Z_{t})f_{t}(S_{t})f_{t}(S_{t})^{\top}$
$\displaystyle=$
$\displaystyle\big{(}(1-p_{t})\beta_{1,t}\Sigma_{\mu}(Z_{t})+\alpha_{1,t}\Sigma_{\mu}(Z_{t})\big{)}f_{t}(S_{t})f_{t}(S_{t})^{\top},$

and similarly

$\displaystyle\mathbb{E}\big{[}\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,H_{t}](Z_{t}-\mu_{t}(S_{t}))f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}=\big{(}-p_{t}\beta_{1,t}\Sigma_{\mu}(Z_{t})+\alpha_{1,t}\Sigma_{\mu}(Z_{t})\big{)}f_{t}(S_{t})f_{t}(S_{t})^{\top}.$

Then combining these two terms, we have:

$\displaystyle\Sigma_{t}^{\text{WCLS}}-\Sigma_{t}^{\text{U}}$
$\displaystyle=-p_{t}(1-p_{t})\alpha^{2}_{1,t}\Sigma_{\mu}(Z_{t})\left(1+2(1-2p_{t})\frac{\beta_{1,t}}{\alpha_{1,t}}\right)f_{t}(S_{t})f_{t}(S_{t})^{\top}$
(41)

###  F.2 Asymptotic Variance Comparison Between $\hat{\beta}_{0,t}^{\text{L}}$
and $\hat{\beta}_{0,t}^{\text{U}}$

Similarly, we only need to investigate the difference between
$\Sigma_{t}^{\text{L}}$ and $\Sigma_{t}^{\text{U}}$.

$\displaystyle\Sigma_{t}^{\text{L}}$
$\displaystyle=\mathbb{E}\big{[}\left((\epsilon_{t}^{\text{U}}-(Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t}-(A_{t}-p_{t})(Z_{t}-\mu_{t}(S_{t}))\beta_{1,t})(A_{t}-p_{t})\right)^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle=\Sigma_{t}^{\text{U}}+\mathbb{E}\big{[}\left((Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t}+(A_{t}-p_{t})(Z_{t}-\mu_{t}(S_{t}))\beta_{1,t}\right)^{2}(A_{t}-p_{t})^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle-2\mathbb{E}\big{[}\epsilon_{t}^{\text{U}}((Z_{t}-\mu_{t}(S_{t}))^{\top}\alpha_{1,t}+(A_{t}-p_{t})(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1,t})(A_{t}-p_{t})^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$

The expansion shows that the difference between $\Sigma_{t}^{\text{L}}$ and
$\Sigma_{t}^{\text{U}}$ is:

$\begin{split}&\mathbb{E}\big{[}\big{(}(Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t}+(A_{t}-p_{t})(Z_{t}-\mu_{t}(S_{t}))\beta_{1,t}\big{)}^{2}(A_{t}-p_{t})^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}\\\
&-2\mathbb{E}\big{[}\epsilon_{t}^{\text{U}}\big{(}(Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t}+(A_{t}-p_{t})(Z_{t}-\mu_{t}(S_{t}))\beta_{1,t}\big{)}(A_{t}-p_{t})^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}\end{split}$
(42)

The first quadratic term is given by:

$\displaystyle\mathbb{E}\big{[}\big{(}(Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t}+(A_{t}-p_{t})(Z_{t}-\mu_{t}(S_{t}))\beta_{1,t}\big{)}^{2}(A_{t}-p_{t})^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle=$ $\displaystyle
p_{t}(1-p_{t})\big{(}\alpha^{2}_{1,t}+\beta^{2}_{1,t}(1-3p_{t}+3p_{t}^{2})+2\alpha_{1,t}\beta_{1,t}(1-2p_{t})\big{)}\Sigma_{\mu}(Z_{t})f_{t}(S_{t})f_{t}(S_{t})^{\top}$
$\displaystyle=$ $\displaystyle
p_{t}(1-p_{t})\big{(}\alpha_{1,t}+\beta_{1,t}(1-2p_{t})\big{)}^{2}\Sigma_{\mu}(Z_{t})f_{t}(S_{t})f_{t}(S_{t})^{\top}$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}+p^{2}_{t}(1-p_{t})^{2}\beta^{2}_{1,t}\Sigma_{\mu}(Z_{t})f_{t}(S_{t})f_{t}(S_{t})^{\top}$

The second interaction term could be further simplified as:

$\displaystyle\mathbb{E}\big{[}\epsilon_{t}^{\text{U}}\big{(}(Z_{t}-\mu_{t}(S_{t}))\alpha_{1,t}+(A_{t}-p_{t})(Z_{t}-\mu_{t}(S_{t}))\beta_{1,t}\big{)}(A_{t}-p_{t})^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle=$
$\displaystyle\alpha_{1,t}\mathbb{E}\big{[}\left(\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,H_{t}]p_{t}(1-p_{t})^{2}+\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,H_{t}]p_{t}^{2}(1-p_{t})\right)(Z_{t}-\mu_{t}(S_{t}))f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle+\beta_{1,t}\mathbb{E}\big{[}\left(\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,H_{t}]p_{t}(1-p_{t})^{3}-\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,H_{t}]p_{t}^{3}(1-p_{t})\right)(Z_{t}-\mu_{t}(S_{t}))f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle=$
$\displaystyle\alpha_{1,t}p_{t}(1-p_{t})\mathbb{E}\big{[}\left(\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,H_{t}](1-p_{t})+\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,H_{t}]p_{t}\right)(Z_{t}-\mu_{t}(S_{t}))f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle+\beta_{1,t}p_{t}(1-p_{t})\mathbb{E}\big{[}\left(\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=1,H_{t}](1-p_{t})^{2}-\mathbb{E}[\epsilon_{t}^{\text{U}}|A_{t}=0,H_{t}]p^{2}_{t}\right)(Z_{t}-\mu_{t}(S_{t}))f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$

Then combing these two terms together, we have
$\Sigma_{t}^{\text{L}}-\Sigma_{t}^{\text{U}}$ equals to:

$\begin{split}-p_{t}(1-p_{t})\big{[}\left(\alpha_{1,t}+(1-2p_{t})\beta_{1,t}\right)^{2}\Sigma_{\mu}(Z_{t})+p_{t}(1-p_{t})\beta_{1,t}^{2}\Sigma_{\mu}(Z_{t})\big{]}f_{t}(S_{t})f_{t}(S_{t})^{\top}\end{split}$
(43)

which is guaranteed to be negative semi-definite.

###  F.3 Efficiency Comparison Between $\hat{\beta}_{0,t}^{\text{L}}$ and
$\hat{\beta}_{0,t}^{\text{WCLS}}$

Simply applying the differences derived above, namely (41) and (43), we can
easily get the asymptotic variance difference:

$\displaystyle\Sigma_{t}^{\text{L}}-\Sigma_{t}^{\text{WCLS}}=-p_{t}(1-p_{t})\big{[}(1-3p_{t}+3p_{t}^{2})\beta_{1,t}^{2}\Sigma_{\mu}(Z_{t})\big{]}f_{t}(S_{t})f_{t}(S_{t})^{\top},$
(44)

which is guaranteed to be negative semi-definite.

##  Appendix G Proof of Theorem 4.3

Recall the asymptotic variance expressions presented in (29) and (30), the
outer term $Q$ is the same for both, therefore, we only need to investigate
the difference between $\Sigma^{\text{A2}}$ and $\Sigma^{\text{WCLS}}$. And
$\Sigma^{\text{WCLS}}$ can be expanded as below:

$\displaystyle\Sigma^{\text{WCLS}}$
$\displaystyle=\mathbb{E}\Big{[}\Big{(}\sum_{t=1}^{T}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\Big{)}\Big{(}\sum_{t=1}^{T}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\Big{)}^{\top}\Big{]}$
$\displaystyle+2\mathbb{E}\Big{[}\sum_{t,t^{\prime}=1}^{T}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times
W_{t^{\prime}}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle+\mathbb{E}\Big{[}\Big{(}\sum_{t=1}^{T}W_{t}(A_{t}-\tilde{p}_{t})^{2}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})\Big{)}\Big{(}\sum_{t=1}^{T}W_{t}(A_{t}-\tilde{p}_{t})^{2}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})\Big{)}^{\top}\Big{]}.$

The expansion shows that the difference between $\Sigma^{\text{A2}}$ and
$\Sigma^{\text{WCLS}}$ is:

$\displaystyle-$ $\displaystyle
2\mathbb{E}\Big{[}\sum_{t,t^{\prime}=1}^{T}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times
W_{t^{\prime}}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle-$
$\displaystyle\mathbb{E}\Big{[}\Big{(}\sum_{t=1}^{T}W_{t}(A_{t}-\tilde{p}_{t})^{2}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})\Big{)}\Big{(}\sum_{t=1}^{T}W_{t}(A_{t}-\tilde{p}_{t})^{2}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})\Big{)}^{\top}\Big{]}$

The first term above indicates a requirement for assumptions regarding the
correlation between two random processes. If they are uncorrelated, we are
left with the final quadratic term, which is ensured to be negative semi-
definite. Nonetheless, this assumption may be less straightforward to
interpret.

Here, we first explore two specific scenarios that help clarify the two
sufficient conditions for efficiency gain mentioned in the main text:

###  G.1 Stochastic Contextual Bandit Environment

In stochastic bandit environment, the potential outcomes
$\\{X_{t},Y_{t}(a):a\in\\{0,1\\}\\}\overset{i.i.d}{\sim}\mathcal{P}$ over
$t\in\\{1,2,\dots,T\\}$. This implies that
$Y_{t+1}\perp\\!\\!\\!\perp\\{H_{t-1},A_{t-1}\\}|X_{t},A_{t}$, and the
conditional distribution $Y_{t+1}|X_{t},A_{t}$ is invariant over time. We
first study the interaction term with $t>t^{\prime}$:

$\displaystyle\mathbb{E}\Big{[}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times
W_{t^{\prime}}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\mathbb{E}\left[W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})|H_{t}\right]f_{t}(S_{t})\times
W_{t^{\prime}}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\tilde{p}_{t}(1-\tilde{p}_{t})\Big{(}\mathbb{E}\left[\epsilon_{t}^{\text{A2}}|H_{t},1\right]-\mathbb{E}\left[\epsilon_{t}^{\text{A2}}|H_{t},0\right]\Big{)}f_{t}(S_{t})\times
W_{t^{\prime}}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\tilde{p}_{t}(1-\tilde{p}_{t})\underbrace{\Big{(}\mathbb{E}\left[\epsilon_{t}^{\text{A2}}|X_{t},1\right]-\mathbb{E}\left[\epsilon_{t}^{\text{A2}}|X_{t},0\right]\Big{)}}_{Y_{t+1}\perp\\!\\!\\!\perp\\{H_{t-1},A_{t-1}\\}|X_{t},A_{t}}f_{t}(S_{t})\times
W_{t^{\prime}}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\tilde{p}_{t}(1-\tilde{p}_{t})\Big{(}\mathbb{E}\left[\epsilon_{t}^{\text{A2}}|X_{t},1\right]-\mathbb{E}\left[\epsilon_{t}^{\text{A2}}|X_{t},0\right]\Big{)}f_{t}(S_{t})\Big{]}\mathbb{E}\Big{[}W_{t^{\prime}}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\tilde{p}_{t}(1-\tilde{p}_{t})e(S_{t})f_{t}(S_{t})\Big{]}\mathbb{E}\Big{[}p_{t^{\prime}}(1-\tilde{p}_{t^{\prime}})(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$

When $t<t^{\prime}$:

$\displaystyle\mathbb{E}\Big{[}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times
W_{t^{\prime}}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times\mathbb{E}\Big{[}W_{t^{\prime}}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}|H_{t^{\prime}}\Big{]}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times\tilde{p}_{t^{\prime}}(1-\tilde{p}_{t^{\prime}})(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\Big{]}\mathbb{E}\Big{[}\tilde{p}_{t^{\prime}}(1-\tilde{p}_{t^{\prime}})(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\tilde{p}_{t}(1-\tilde{p}_{t})\Big{(}\mathbb{E}\left[\epsilon_{t}^{\text{A2}}|X_{t},1\right]-\mathbb{E}\left[\epsilon_{t}^{\text{A2}}|X_{t},0\right]\Big{)}f_{t}(S_{t})\Big{]}\mathbb{E}\Big{[}p_{t^{\prime}}(1-\tilde{p}_{t^{\prime}})(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\tilde{p}_{t}(1-\tilde{p}_{t})e(S_{t})f_{t}(S_{t})\Big{]}\mathbb{E}\Big{[}p_{t^{\prime}}(1-\tilde{p}_{t^{\prime}})(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$

Since the i.i.d assumption implies the conditional distribution of the outcome
is invariant over time, we have:

$\displaystyle\mathbb{E}[\tilde{p}_{t}(1-\tilde{p}_{t})e(S_{t})f_{t}(S_{t})]=0.$

Then the above terms when $t\neq t^{\prime}$ are equal 0. It rests on studying
the following terms when $t=t^{\prime}$:

$\displaystyle\mathbb{E}\Big{[}W_{t}^{2}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times(A_{t}-\tilde{p}_{t})^{2}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}W_{t}^{2}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})^{3}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})f_{t}(S_{t})^{\top}\Big{]}$

We assume treatment randomization depends on a subset of the fully observed
history, denoted as $\tilde{V}_{t}$ where $\tilde{V}_{t}\subset H_{t}$, and
define $V_{t}=\tilde{V}_{t}\cup\\{Z_{t}\\}\cup\\{S_{t}\\}$. The expression for
the above term is as follows:

$\displaystyle\mathbb{E}\Big{[}W^{2}_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})^{3}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})f_{t}(S_{t})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\Big{(}\frac{(1-\tilde{p}_{t})^{3}\tilde{p}^{2}_{t}}{p_{t}(V_{t})}\mathbb{E}[\epsilon_{t}^{\text{A2}}|V_{t},A_{t}=1]-\frac{\tilde{p}_{t}^{3}(1-\tilde{p}_{t})^{2}}{1-p_{t}(V_{t})}\mathbb{E}[\epsilon_{t}^{\text{A2}}|V_{t},A_{t}=0]\Big{)}\times$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})f_{t}(S_{t})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\Big{(}\frac{(1-\tilde{p}_{t})^{4}\tilde{p}^{2}_{t}}{p_{t}(V_{t})}+\frac{\tilde{p}_{t}^{4}(1-\tilde{p}_{t})^{2}}{1-p_{t}(V_{t})}\Big{)}e(V_{t})(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})f_{t}(S_{t})^{\top}\Big{]},$

since

$\displaystyle\mathbb{E}\big{[}\epsilon_{t}^{\text{A2}}|V_{t},A_{t}=1\big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\bigg{[}\mathbb{E}\Big{[}Y_{t+1}-g_{t}(H_{t})^{\top}\alpha-(1-p_{t})(f_{t}(S_{t})^{\top}\beta_{0}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1})|H_{t},A_{t}=1\Big{]}|V_{t},A_{t}=1\bigg{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\bigg{[}\mathbb{E}\Big{[}Y_{t+1}|H_{t},A_{t}=1\Big{]}-g_{t}(H_{t})^{\top}\alpha-(1-p_{t})(f_{t}(S_{t})^{\top}\beta_{0}+(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1})|V_{t}\bigg{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\bigg{[}(1-p_{t})\beta(t;H_{t})+\Big{(}\mathbb{E}\Big{[}W_{t}Y_{t+1}|H_{t}\Big{]}-g_{t}(H_{t})^{\top}\alpha\Big{)}-$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}(1-p_{t})(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}-(1-p_{t})f_{t}(S_{t})^{\top}\beta_{0}|V_{t}\bigg{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}(1-p_{t})\big{(}\beta(t;V_{t})-f_{t}(S_{t})^{\top}\beta_{0}-(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}\big{)}|V_{t}\Big{]}$
$\displaystyle=$ $\displaystyle\big{(}1-p_{t}(V_{t})\big{)}e(V_{t})$

and similarly:

$\mathbb{E}\big{[}\epsilon_{t}^{\text{A2}}|V_{t},A_{t}=0\big{]}=-p_{t}e(V_{t})$

Then we have the final expression for the interaction term:

$\displaystyle\mathbb{E}\big{[}\epsilon_{t}^{\text{A2}}W_{t}^{2}(A_{t}-\tilde{p}_{t})^{3}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})f_{t}(S_{t})^{\top}\big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\Big{(}\frac{(1-\tilde{p}_{t})^{4}\tilde{p}^{3}_{t}}{p_{t}(V_{t})}+\frac{\tilde{p}_{t}^{4}(1-\tilde{p}_{t})^{3}}{1-p_{t}(V_{t})}\Big{)}e(V_{t})(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})f_{t}(S_{t})^{\top}\Big{]}$

A sufficient condition for the improvement of efficiency in a general
moderated treatment effect is:

######  Assumption G.1.

The true causal effect model $\beta(t;H_{t})$ is correctly specified on
$V_{t}$.

This assumption implies that
$e(V_{t})=\beta(t;V_{t})-f_{t}(S_{t})^{\top}\beta_{0}-(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}=0$.
Thus the interaction term is 0. For the last term, also due to the i.i.d
assumption, we obtain:

$\displaystyle-\mathbb{E}\left[\sum_{t=1}^{T}W^{2}_{t}(A_{t}-\tilde{p}_{t})^{4}\big{(}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}\big{)}^{2}f_{t}(S_{t})f_{t}(S_{t})^{\top}\right]$

which is guaranteed to be negative semi-definite. Therefore, in the stochastic
contextual bandit environment, under Assumption G.1, the efficiency
improvement in a moderated treatment effect is guaranteed.

###  G.2 Longitudinal Dependent Data Environment

In mobile health trials, assuming a stochastic bandit environment might be
overly ambitious. Therefore, we investigate the sufficient condition for
efficiency improvement in scenarios where
$\\{Y_{t+1}(\bar{a}_{t}),X_{t}(\bar{a}_{t-1})\\}$, as mentioned in the
manuscript. This departure from the i.i.d. assumption of contextual variables
and potential outcomes over $t\in\\{1,2,\dots,T\\}$ complicates the analysis.
For simplicity in the proof, we assume $V_{t}=\emptyset$, implying that the
treatment randomization probability is a constant function without dependence
on the observed history. In this case, $p_{t}=\tilde{p}_{t}$, and $W_{t}=1$.

We first study the interaction term with $t>t^{\prime}$:

$\displaystyle\mathbb{E}\Big{[}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\tilde{p}_{t}(1-\tilde{p}_{t})\underbrace{\big{(}\mathbb{E}\left[\epsilon_{t}^{\text{A2}}|A_{t}=1\right]-\mathbb{E}\left[\epsilon_{t}^{\text{A2}}|A_{t}=0\right]\big{)}}_{=0}f_{t}(S_{t})\times$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$ $\displaystyle 0$

When $t<t^{\prime}$:

$\displaystyle\mathbb{E}\Big{[}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times\tilde{p}_{t^{\prime}}(1-\tilde{p}_{t^{\prime}})(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\tilde{p}_{t^{\prime}}(1-\tilde{p}_{t^{\prime}})(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\mathbb{E}\big{[}\epsilon_{t}^{\text{A2}}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}|A_{t}\big{]}\Big{]}$

Here a sufficient condition could be the interaction of future states
$(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}$
not correlated with current residual $\epsilon_{t}^{\text{A2}}$ given the
treatment allocation $A_{t}$, implying
$\mathbb{E}\big{[}\epsilon_{t}^{\text{A2}}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}|A_{t}\big{]}=0$.
In other words, it assumes that future unobserved states do not influence the
current treatment allocation path’s estimated effect, given the current
treatment. And we state formally below:

######  Assumption G.2.

After accounting for the current treatment allocation, any unexplained
variation in the current outcome is not correlated to potential observations
in future states.

A contrast here is to compare to the stronger assumption which assumes the
random process of the residuals is a martingale difference sequence (MDS) with
respect to the observed history information up to the current time,
essentially closing the door to any information from the future leaking into
the current period:

######  Assumption G.3.

After accounting for all observed history information up till the current
point in time, the unexplained variations in the current outcome are not
correlated to potential observations in future states.

Intuitively, it assumes the model accurately captures all relevant history
information that could correlate with both the current residuals and the
interaction of future states. If any relevant variable is omitted, the
assumption might be violated, as the omitted variable could correlate with
both the current errors and future states.

Clearly, Assumption G.2 implies Assumption G.3, making it a notably less
stringent requirement in terms of model specification. The conditional “lack
of correlation” is specifically imposed with respect to the current treatment,
excluding the necessity for conditioning on the entire historical context.

Then under Assumption G.2, the above terms when $t\neq t^{\prime}$ are equal
0. It rests on studying the following terms when $t=t^{\prime}$:

$\displaystyle\mathbb{E}\Big{[}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times(A_{t}-\tilde{p}_{t})^{2}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})^{3}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})f_{t}(S_{t})^{\top}\Big{]}$

Following the above derivation,

$\displaystyle\mathbb{E}\Big{[}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})^{3}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})f_{t}(S_{t})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\Big{(}(1-\tilde{p}_{t})^{3}\tilde{p}_{t}\mathbb{E}[\epsilon_{t}^{\text{A2}}|A_{t}=1]-\tilde{p}_{t}^{3}(1-\tilde{p}_{t})\mathbb{E}[\epsilon_{t}^{\text{A2}}|A_{t}=0]\Big{)}\times$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})f_{t}(S_{t})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}(1-\tilde{p}_{t})\tilde{p}_{t}\Big{(}(1-\tilde{p}_{t})^{2}\mathbb{E}[\epsilon_{t}^{\text{A2}}|A_{t}=1]-\tilde{p}_{t}^{2}\mathbb{E}[\epsilon_{t}^{\text{A2}}|A_{t}=0]\Big{)}\times$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})f_{t}(S_{t})^{\top}\Big{]}$
$\displaystyle=$ $\displaystyle 0$

For the last term, we can directly obtain the following:

$-\mathbb{E}\bigg{[}\Big{(}\sum_{t=1}^{T}W_{t}(A_{t}-\tilde{p}_{t})^{2}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})\Big{)}\Big{(}\sum_{t=1}^{T}W_{t}(A_{t}-\tilde{p}_{t})^{2}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})\Big{)}^{\top}\bigg{]}$

which is guaranteed to be negative semi-definite. Therefore, in the stochastic
contextual bandit environment, under Assumption G.2, the efficiency
improvement in a moderated treatment effect is guaranteed.

###  G.3 Longitudinal Dependent Data with Dynamic Treatment Randomization

The data collected for MRTs typically combines the two previously mentioned
special scenarios: the potential outcomes and state variables may demonstrate
time dependence, and the treatment randomization probability depends on a
subset of historical information. In such situations, we anticipate the need
for more stringent assumptions regarding the specification of the causal
effect and outcome model.

Based on the same routine, we first study the interaction term with
$t>t^{\prime}$:

$\displaystyle\mathbb{E}\Big{[}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times
W_{t^{\prime}}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\mathbb{E}\left[W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})|H_{t}\right]f_{t}(S_{t})\times
W_{t^{\prime}}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\tilde{p}_{t}(1-\tilde{p}_{t})\underbrace{\big{(}\mathbb{E}\left[\epsilon_{t}^{\text{A2}}|H_{t},1\right]-\mathbb{E}\left[\epsilon_{t}^{\text{A2}}|H_{t},0\right]\big{)}}f_{t}(S_{t})\times
W_{t^{\prime}}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$

A sufficient condition for the improvement of efficiency in a general
moderated treatment effect is:

######  Assumption G.4.

The true causal effect model $\beta(t;H_{t})$ is correctly specified.

This assumption implies that
$\mathbb{E}\left[\epsilon_{t}^{\text{A2}}|H_{t},1\right]-\mathbb{E}\left[\epsilon_{t}^{\text{A2}}|H_{t},0\right]=0$,
thus the above term is $0$. Besides, under Assumption G.3, for terms with
$t<t^{\prime}$:

$\displaystyle\mathbb{E}\Big{[}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times
W_{t^{\prime}}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times\mathbb{E}\Big{[}W_{t^{\prime}}(A_{t^{\prime}}-\tilde{p}_{t^{\prime}})^{2}|H_{t^{\prime}}\Big{]}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}W_{t}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\times\tilde{p}_{t^{\prime}}(1-\tilde{p}_{t^{\prime}})(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}W_{t}(A_{t}-\tilde{p}_{t})f_{t}(S_{t})\tilde{p}_{t^{\prime}}(1-\tilde{p}_{t^{\prime}})\underbrace{\mathbb{E}\big{[}\epsilon_{t}^{\text{A2}}(Z_{t^{\prime}}-\mu_{t^{\prime}}(S_{t^{\prime}}))^{\top}\beta_{1}f_{t^{\prime}}(S_{t^{\prime}})^{\top}|H_{t},A_{t}\big{]}}_{=0}\Big{]}$
$\displaystyle=$ $\displaystyle 0$

As for $t=t^{\prime}$, under Assumption G.4:

$\displaystyle\mathbb{E}\Big{[}W_{t}^{2}\epsilon_{t}^{\text{A2}}(A_{t}-\tilde{p}_{t})^{3}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})f_{t}(S_{t})^{\top}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\Big{(}\frac{(1-\tilde{p}_{t})^{4}\tilde{p}^{2}_{t}}{p_{t}(H_{t})}+\frac{\tilde{p}_{t}^{4}(1-\tilde{p}_{t})^{2}}{1-p_{t}(H_{t})}\Big{)}\underbrace{e(H_{t})}_{=0}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})f_{t}(S_{t})^{\top}\Big{]}$
$\displaystyle=$ $\displaystyle 0$

Therefore the interaction term equals 0, and
$\Sigma^{\text{A2}}-\Sigma^{\text{WCLS}}$ is left with a negative quadratic
term:

$-\mathbb{E}\Big{[}\Big{(}\sum_{t=1}^{T}W_{t}(A_{t}-\tilde{p}_{t})^{2}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})\Big{)}\Big{(}\sum_{t=1}^{T}W_{t}(A_{t}-\tilde{p}_{t})^{2}(Z_{t}-\mu_{t}(S_{t}))^{\top}\beta_{1}f_{t}(S_{t})\Big{)}^{\top}\Big{]}$

which is negative semidefinite. In conclusion, we proved that
$\hat{\beta}_{0}^{\text{A2}}$ is more efficient than
$\hat{\beta}_{0}^{\text{WCLS}}$.

##  Appendix H Connection to A Semiparametric Efficient Estimator

In the main text, we’ve outlined a connection between Neyman orthogonality and
our method, hinting at a connection with semiparametric efficiency theory. In
this section, we adopt a semiparametric efficiency theory perspective to offer
deeper insights into constructing the A2-WCLS estimator for $\beta_{0}$ based
on a semiparametric influence function.

A special case arises when $S_{t}$ is the complete observed history $H_{t}$,
meaning we are interested in a fully conditional causal effect. In such cases,