# Inference for relative sparsity

Samuel J. Weisenthal  Department of Biostatistics and Computational Biology,
University of Rochester School of Medicine and Dentistry, Rochester, NY
Medical Scientist Training Program, University of Rochester School of Medicine
and Dentistry, Rochester, NY  Sally W. Thurston  Department of Biostatistics
and Computational Biology, University of Rochester School of Medicine and
Dentistry, Rochester, NY  Ashkan Ertefaie  Department of Biostatistics and
Computational Biology, University of Rochester School of Medicine and
Dentistry, Rochester, NY

###### Abstract

In healthcare, there is much interest in estimating policies, or mappings from
covariates to treatment decisions. Recently, there is also interest in
constraining these estimated policies to the standard of care, which generated
the observed data. A relative sparsity penalty was proposed to derive policies
that have sparse, explainable differences from the standard of care,
facilitating justification of the new policy. However, the developers of this
penalty only considered estimation, not inference. Here, we develop inference
for the relative sparsity objective function, because characterizing
uncertainty is crucial to applications in medicine. Further, in the relative
sparsity work, the authors only considered the single-stage decision case;
here, we consider the more general, multi-stage case. Inference is difficult,
because the relative sparsity objective depends on the unpenalized value
function, which is unstable and has infinite estimands in the binary action
case. Further, one must deal with a non-differentiable penalty. To tackle
these issues, we nest a weighted Trust Region Policy Optimization function
within a relative sparsity objective, implement an adaptive relative sparsity
penalty, and propose a sample-splitting framework for post-selection
inference. We study the asymptotic behavior of our proposed approaches,
perform extensive simulations, and analyze a real, electronic health record
dataset.

##  1 Introduction

Treatment policies, or mappings from patient covariates to treatment
decisions, can help healthcare providers and patients make more informed,
data-driven decisions, and there is great interest in both the statistical and
reinforcement learning communities in developing methods for deriving these
policies (Chakraborty and Moodie, 2013; Futoma et al., 2020; Uehara et al.,
2022). There is particularly recent interest in deriving constrained versions
of these policies. While there has been work on the general theory of
constrained reinforcement learning (Le et al., 2019; Geist et al., 2019),
several methodologies can be more specifically categorized as “behavior-
constrained” policy optimization, an umbrella term used in Wu et al. (2019) to
encompass the array of methods that constrain the new, suggested policy to be
similar to the “behavioral” policy that generated the data. Examples of
‘behavior-constrained’ policy optimization include entropy-constrained policy
search (Haarnoja et al., 2017; Ziebart et al., 2008; Peters et al., 2010);
what Le et al. (2019) calls “conservative” policy improvement methods, such as
guided search and Trust Region Policy Optimization (TRPO) (Levine and Abbeel,
2014; Schulman et al., 2015, 2017; Achiam et al., 2017; Le et al., 2019),
which constrain optimization such that large divergences from the previous
policy are discouraged; and other approaches with similar goals, such as
likelihood weighting, entropy penalties (Fujimoto et al., 2019; Ueno et al.,
2012; Dayan and Hinton, 1997; Peters et al., 2010; Haarnoja et al., 2017;
Ziebart et al., 2008), imitation learning (Le et al., 2016), value constrained
model-based reinforcement learning Futoma et al. (2020); Farahmand et al.
(2017), and tilting (Kennedy, 2019; Kallus and Uehara, 2020). In Weisenthal et
al. (2023), a relative sparsity penalty was developed, which differs from
behavior constraints in existing studies in that it focuses on explainability
and relative interpretability between the suggested policy and the standard of
care. In Weisenthal et al. (2023), however, only estimation, not inference,
was considered for the relative sparsity objective function. In our work,
therefore, we consider the challenging problem of inference for the relative
sparsity objective. Further, in Weisenthal et al. (2023), the authors only
considered the single-stage decision setting; here, we consider the more
general, multi-stage decision setting. The objective function in relative
sparsity combines a raw value (expected reward) function with a relative Lasso
penalty, where both components pose challenges for inference. In the binary
action case, under a parameterized policy, the raw value function is optimized
by estimands that are infinite or arbitrarily large in magnitude, a
consequence of the the fact that the policy that solves the raw value
objective function is deterministic (Lei et al., 2017; Puterman, 2014;
Weisenthal et al., 2023).

The contribution of our work beyond the existing literature, can be summarized
in the following five ways. First, to address the issue of estimands that are
of infinite or arbitrarily large magnitude, we propose a double behavior
constraint, nesting a weighted TRPO behavior constraint within the relative
sparsity objective. Second, based on work in Zou (2006), we develop
methodology for an adaptive relative sparsity formulation, which improves
discernment of the penalty. Third, we provide a sample splitting framework
that is free of issues associated with post-selection inference (Cox, 1975;
Leamer, 1974; Kuchibhotla et al., 2022). Fourth, we rigorously study the
asymptotic properties of all frameworks: inference for existing TRPO methods
has not been studied due to the general focus on pure prediction in robotics
applications. To fill this gap, we develop novel theory for inference in the
TRPO framework and, in particular, for the weighted (Thomas, 2015; Owen, 2013)
TRPO estimator. Further, we rigorously study the asymptotic theory for the
adaptive relative sparsity penalty and develop theory for confidence intervals
in the sample splitting setting. We take special care to develop theory around
the nuisance, which appears not only in the denominator of the inverse
probability weighting expression but, also, in the sample splitting case,
within the suggested policy itself. Fifth, we consider the more general multi-
stage, Markov Decision Process (MDP), setting. We perform simulation studies,
revealing how the magnitude of the tuning parameter impacts inference, and
showing where the proposed methodology and theory succeeds, and where it might
fail. We conclude our work with a data analysis of a real, observational
dataset, derived from the MIMIC III database (Johnson et al., 2016b, a;
Goldberger et al., 2000), performing inference on a relatively sparse decision
policy for vasopressor administration. Although similar, routinely collected
health data has been used for prediction (e.g., (Futoma et al., 2015; Lipton
et al., 2015; Weisenthal et al., 2018)), there has been less work toward
developing rigorous decision models with this data, and we fill this gap as
well.

Developing the statistical inference properties of the relative sparsity
penalty allows us to better port this useful technique to healthcare, where
the uncertainty associated with the new, suggested policy is important in
order to guide healthcare providers, patients, and other interested parties as
they choose whether or not to adopt these new treatment strategies. This work
ultimately facilitates translation of data-driven treatment strategies from
the laboratory to the clinic, where they might substantially improve health
outcomes.

##  2 Notation

Throughout our work, we use subscript $0$ and $n$ to denote a true parameter
and an estimator derived from a sample of size $n$ (i.e., $\beta_{n}$ is an
estimator for $\beta_{0}$ based on a sample of size $n$), respectively. If we
have a vector, $v_{t},$ indexed at time $t,$ we index dimension $k$ as
$v_{t,k}.$ We index the components of a parameter vector, $\theta,$ as
$\theta_{k}$. In many cases, we will further subscript parameters according to
penalty tuning parameters (e.g., $\lambda$), as in $\theta_{\gamma,\lambda}$,
and, in this case, we will index dimension $k$ as $\theta_{\gamma,\lambda,k}$.
Let $\beta$ denote a parameter that indexes an arbitrary policy,
$\pi_{\beta}(a|s),$ where $a$ and $s$ are a binary action (treatment) and
state (patient covariates), respectively. Let similarly $b$ denote an
arbitrary nuisance parameter that indexes the existing, behavioral policy,
which corresponds to the standard of care. Let $E,E_{n}$ be the true and
empirical expectation operators, where, for some function $f,$
$E_{n}f=\frac{1}{n}\sum_{i}f(X_{i}).$ We will sometimes write, for arbitrary
functions $f$ and $g$,

$E_{n}\frac{f}{E_{n}g}=\frac{1}{n}\sum_{i}\frac{f_{i}}{E_{n}g}=\frac{1}{n}\sum_{i}\frac{f_{i}}{\frac{1}{n}\sum_{i}g_{i}}.$

We will often use capital letters to refer to an average and their lowercase
counterparts to refer to the elements being averaged; i.e., we write
$F_{n}=E_{n}f.$ Let $\pi_{\beta,b}$ denote a policy in which some components
of $\beta$ are fixed to components in $b$. Assuming random variable $A$ is
discrete and random variable $S$ is continuous, let
$E_{b}(f)=\sum_{a}\int_{s}f(a,s)\pi_{b}(a|s)p(s)ds$ denote an expectation of
some deterministic (non-random) function $f$ with respect to policy $\pi_{b}.$
Note that, in line with our subscript conventions for estimands and estimators
mentioned above, $\pi_{\beta_{n}}$ is an estimator for $\pi_{\beta_{0}}$.

##  3 Background

###  3.1 Markov decision processes (MDPs)

Consider the multi-stage, discrete-time, Markov decision process (MDP), a
general model for data that evolves over time based on the actions of some
agent as it interacts with an environment (Bellman, 1957). The MDP and its
extensions have been used to model many problems in the medical domain; for
examples related to diabetes, hypotension, and mobile health, see Chakraborty
and Moodie (2013); Ertefaie and Strawderman (2018); Futoma et al. (2020); Lei
et al. (2017); Luckett et al. (2019). We aim to address similar healthcare
problems here. Let us have a continuous, $K$-dimensional state,
$S\in\mathbb{R}^{K}$, which may, e.g., contain a patient’s covariates.Let us
also have a binary action, $A\in\\{0,1\\}$ which may be, e.g., the
administration of a medication. Let us sample $n$ independent and identically
distributed length-($T+1$) patient trajectories of states and actions (hence,
all trajectories must be of the same length). The random sample from a single
trajectory is then of the form
$\left\\{S_{i,0},A_{i,0},S_{i,1},A_{i,1},\dots,S_{i,T},A_{i,T},S_{i,T+1}\right\\}_{i=1,\dots,n}$,
where $S_{i,t}$ is the stage-$t$ state of patient $i$ and $A_{i,t}$ is the
stage-$t$ action for patient $i.$ A trajectory is sampled from a fixed
distribution denoted by $P_{0}$, which can be factored into an initial state
distribution, $P_{0}(S_{0})$, the transition probability,
$P_{0,t+1}(S_{t+1}|A_{t},S_{t},\dots,A_{0},S_{0}),$ and the data-generating
policy, $P_{0,t}(A_{t}|S_{t},\dots,A_{0},S_{0}).$ The latter is called a
“policy,” because we can imagine that a trajectory is constructed by a
healthcare provider (and/or patient) drawing an action conditional on the
patient history. We assume that the policy is Markov and does not change over
time, which are common assumptions in these problems (Sutton and Barto, 2018).

######  Assumption 1.

Markov property:
$P_{0,t}(A_{t}|S_{t},\dots,A_{0},S_{0})=P_{0,t}(A_{t}|S_{t}).$

######  Assumption 2.

Stationarity: $P_{0,t}(A_{t}|S_{t})=P_{0}(A_{t}|S_{t}).$

To facilitate interpretation, we parameterize $P(A_{t}|S_{t})$ with arbitrary
vector parameter $\theta$ (which can refer to $\theta=\beta$ or $\theta=b$
depending on whether we are referring to the parameter that we are optimizing
over or the behavioral policy parameter), so that it is
$P_{\theta}(A_{t}|S_{t}).$ We then denote $P_{\theta}(A_{t}|S_{t})$ as
$\pi_{\theta}(A_{t}|S_{t}),$ as is convention (Sutton and Barto, 2018). For
this parameterization, we propose the model

$\displaystyle\pi_{\theta}(A_{t}=1|S_{t}=s)=\text{expit}(\theta^{T}s)=\frac{\exp({\theta^{T}s})}{1+\exp({\theta^{T}s})}.$
(1)

Under Assumption 2, Assumption 1, and the model in (1), we have the sampling
distribution

$P_{0}(S_{0})\pi_{b_{0}}(A_{0}|S_{0})\prod_{t=1}^{T}\pi_{b_{0}}(A_{t}|S_{t})P_{0,t}(S_{t}|A_{t-1},S_{t-1},\dots,A_{0},S_{0}),$

where $P_{0}(A_{t}|S_{t})$, is parameterized by $b_{0}$ under (1) and becomes
$\pi_{b_{0}}(A_{t}|S_{t})$.

Let there also be a deterministic, stationary reward function

$R(S_{t},A_{t},S_{t+1}),$ (2)

which maps the state at some time to a utility. The total return for one
trajectory (or episode) is the sum of rewards from the first time step to the
end of the trajectory $\sum_{t=0}^{T}R(S_{t},A_{t},S_{t+1}).$ Because we focus
on finite-horizon cases, there is no need to consider discounted cumulative
rewards, in which the contribution of states that occur later in time is down-
weighted (Sutton and Barto, 2018).

Let us also parameterize an arbitrary policy $\pi_{\beta}$ with a vector of
coefficients $\beta$ using (1). Define the value of a policy, which is the
expected return when acting under the policy $\pi_{\beta}$, as

$V_{0}(\beta)=E_{\beta}\sum_{t=0}^{T}R(S_{t},A_{t},S_{t+1}).$ (3)

The reward-maximizing policy $\pi_{\beta_{0}}$ can be obtained by solving

$\beta_{0}=\arg\max_{\beta}V_{0}(\beta).$ (4)

######  Remark 1.

One cannot perform inference for $\beta_{0}.$ In Lemma 4 (Appendix A.3), we
show that, under the model for the policy in (1), because the optimal policy
$\pi_{\beta_{0}}$ is deterministic, $\beta_{0}$ diverges in magnitude.

The divergence issue precludes inference in Weisenthal et al. (2023), where
the authors optimize the “relative sparsity” objective,
$V_{0}(\beta)-\lambda||\beta-b_{0}||_{1}.$ We overcome this issue by replacing
the relative sparsity “base” objective, $V_{0},$ with the full TRPO objective,
as we will now describe.

###  3.2 On inference with Trust Region Policy Optimization (TRPO)

Inference for Trust Region Policy Optimization (TRPO) (Schulman et al., 2015)
has not previously been considered, to our knowledge, because the robotics
applications for which TRPO was developed might not benefit from inference in
the way that medical applications might. The remainder of this section,
therefore, contains what we believe to be novel insights.

The objective $V_{0}$ in (3) is not behavior constrained, which precludes
inference. To mitigate this issue, we add a Kullback-Leibler ($KL$) (Kullback
and Leibler, 1951) behavior constraint (Schulman et al., 2015). We choose $KL$
divergence because it is an expectation and therefore has favorable asymptotic
properties. More specifically, for fixed $\gamma,$ we will employ the
following objective as a new base objective for relative sparsity,

$M_{0}(\beta,b,\gamma)=V_{0}(\beta)-\gamma KL_{0}(\beta,b),$ (5)

where

$KL_{0}(\beta,b)=E_{b}\log\left({\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}\biggr{/}{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}\right).$
(6)

In general, finding a policy with minimal $KL$ divergence from the behavioral
policy is equivalent to finding a policy that maximizes likelihood (van der
Vaart, 2000). In the objective function (5), this would be achieved by setting
$\gamma=\infty.$

Now, the following estimand, $\beta_{0,\gamma},$ is behavior-constrained

$\beta_{0,\gamma}=\arg\max_{\beta}M_{0}(\beta,b_{0},\gamma).$ (7)

One can perform inference for $\beta_{0,\gamma},$ which will be finite in
magnitude, even though, as discussed in Remark 1, one cannot perform inference
for $\beta_{0},$ because $\beta_{0}$ is infinite in magnitude.

######  Remark 2.

If $\gamma>0,$ the behavior-constrained solution
$\beta_{0,\gamma}=\arg\max_{\beta}M_{0}(\beta,b_{0},\gamma)$ is finite in
magnitude, which allows for inference.

Our estimand, $\beta_{0,\gamma}$, depends on a parameter $\gamma;$ we are
targeting a behavior-constrained estimand. This is in contrast to typical
penalization, where one targets some estimand that does not depend on a
parameter and penalizes in order to reduce the variance of the estimator.
Having addressed the issue of infinite estimands, we can now augment the
behavior constrained objective function in (5) with a relative sparsity
objective.

##  4 Methodological Contributions

###  4.1 Adding (adaptive) relative sparsity to Trust Region Policy
Optimization (TRPO)

Having stated the Trust Region Policy Optimization (TRPO) objective in (5), we
will now add to it relative sparsity. As discussed in Remark 2, maximizing the
TRPO objective function defined by (5) allows for inference for a behavior-
constrained policy. Behavior constraints create closeness to the standard of
care, which facilitates adoption, since changes to practice guidelines can
pose challenges for healthcare providers (Gupta et al., 2017; Lipton, 2018;
Rudin, 2019). However, in a healthcare setting, one must convince the
healthcare provider (and possibly also the patient) to adopt a new treatment
policy. This is facilitated if the number of parameters that differ between
the two policies is small, which is related to considerations such as
cognitive burden (Miller, 2019; Du et al., 2019; Weisenthal et al., 2023). To
achieve relative sparsity, we propose

$\displaystyle W_{0}(\beta,\beta_{0,\gamma},b_{0})$
$\displaystyle=M_{0}(\beta,b_{0})-\lambda\sum_{k=1}^{K}{w}_{0,k}|\beta_{k}-b_{0,k}|$
$\displaystyle=V_{0}(\beta)-\gamma
KL_{0}(\beta,b_{0})-\lambda\sum_{k=1}^{K}{w}_{0,k}|\beta_{k}-b_{0,k}|,$ (8)

where $M_{0}$ is defined in (5) and
$w_{0,k}=1/|\beta_{0,\gamma,k}-b_{0,k}|^{\delta}.$ Accordingly, we define our
estimand as

$\beta_{0,\gamma,\lambda}=\arg\max_{\beta}W_{0}(\beta,\beta_{0,\gamma},b_{0}).$
(9)

The added Lasso penalty brings relative sparsity to our behavior constrained
estimand $\beta_{0,\gamma}.$ In practice, increasing the weight of the Lasso
penalty will also cause some behavior-constraint, but this is intended to be
minimal; unlike in the objective function of Weisenthal et al. (2023), where
the Lasso penalty jointly performs shrinkage to behavior and selection, the
Lasso penalty in (4.1) should only perform selection, while the $KL_{0}$
penalty performs shrinkage to behavior. In Equation (4.1), The degree of
shrinkage and selection will be controlled, respectively, by the tuning
parameters $\gamma,$ which controls the degree of closeness to the behavioral
policy, and $\lambda,$ which controls the degree of relative sparsity.
Further, $\delta,$ which is proposed in Zou (2006), controls the adaptivity of
the adaptive Lasso penalty (and is discussed more in Appendix A.5). We will
discuss how to choose these tuning parameters when we discuss estimation.

###  4.2 Sample splitting in the relative sparsity framework

Let $\mathcal{A}$ be a set containing the indices of selected (non-behavioral)
covariates. Let $1_{\mathcal{A}}$ be an indicator for the selected (non-
behavioral) covariates, so
$1_{\mathcal{A}}=(1_{1\in\mathcal{A}},\dots,1_{K\in\mathcal{A}})^{T}$. The
Lasso penalty in (4.1) performs selection, giving us $\mathcal{A}.$ However,
as with any selection, we must avoid issues with post-selection inference
(Leamer, 1974). For this, we use sample splitting (Cox, 1975), where we
perform selection on one split and then inference on a second, independent
split.

We first optimize (4.1) to obtain a selection, $\mathcal{A}$. In standard
sample splitting, one would eliminate the non-selected coefficients. In our
case, we keep non-selected variables, but we fix their parameters to their
behavioral counterparts, and then we perform inference only with respect to
the non-behavioral parameters. For this purpose, letting $\odot$ denote
element-wise multiplication, we propose a novel representation of a policy as

$\pi_{\beta,b}(1|s)=\text{expit}(\beta^{T}(s\odot
1_{\mathcal{A}})+b^{T}(s\odot(1_{K}-1_{\mathcal{A}}))),$ (10)

where $1_{K}$ is a length-$K$ vector of ones. We can then take partial
derivatives of $\pi_{\beta,b}$ with respect to $\beta$ or $b,$ as necessary,
while fixing some entries of $\beta$ to $b_{0}$ (in practice, we fix these
entries not to $b_{0}$ but to an estimator of $b_{0}$). The form of each
partial derivative, amended for the post-selection policy in (10), is included
in Appendix A.18. In particular, the cross derivatives now have extra terms,
because the nuisance, $b,$ appears in the suggested policy as well as in the
behavioral policy.

##  5 Estimation

###  5.1 Value

We now discuss estimation of the value, or expected return, under some
arbitrary policy, indexed by $\beta$. For this, we will need to take a
counterfactual expectation, which can be done using importance sampling or
inverse probability weighting (Kloek and Van Dijk, 1978; Precup, 2000; Thomas,
2015; Horvitz and Thompson, 1952; Robins et al., 1994; Chakraborty and Moodie,
2013; Precup, 2000). Starting with expressions that only depend on the
observed data, we rederive, in Appendix A.9, the well-known fact (Thomas,
2015) that an estimand for the potential value under a policy $\pi_{\beta}$,
can be written, as long as the denominator is never zero (which is formalized
as an assumption in Section 6.1), as

$\displaystyle
V_{0}(\beta)=E_{\beta}\sum_{t=0}^{T}R(S_{t},A_{t},S_{t+1})=E_{b}\left\\{\frac{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}\
\sum_{t=0}^{T}R(S_{t},A_{t},S_{t+1})\right\\}.$ (11)

Then $V_{0}$ can be estimated using an inverse probability weighted estimator.
However, in the multi-stage case, the vanilla inverse probability weighted
estimator is unstable. We therefore use a weighted, as it is called in Thomas
(2015), or self-normalized, as it is called in Owen (2013), importance
sampling estimator,

$\displaystyle{V}_{n}(\beta,b)=\frac{\frac{1}{n}\sum_{i=1}^{n}\frac{\prod_{t=0}^{T}\pi_{\beta}(A_{i,t}|S_{i,t})}{\prod_{t=0}^{T}\pi_{b}(A_{i,t}|S_{i,t})}\sum_{t=0}^{T}R(S_{i,t},A_{i,t},S_{i,t+1})}{\frac{1}{n}\sum_{i=1}^{n}\frac{\prod_{t=0}^{T}\pi_{\beta}(A_{i,t}|S_{i,t})}{\prod_{t=0}^{T}\pi_{b}(A_{i,t}|S_{i,t})}.}$
(12)

Note that (12) has two arguments while (3) has one, because (12) depends on
$b$ through the empirical inverse probability weighting ratio, whereas the
non-empirical terms involving $b$ cancel in (3). Let

$b_{n}=\arg\max_{b}\sum_{i=1}^{n}\sum_{t=0}^{T}\log\pi_{b}(A_{i,t}|S_{i,t})$

denote an estimator of $b_{0}$ (see Appendix A.8). Then, replace $b$ in (12)
with $b_{n}$ to obtain $V_{n}(\beta,b_{n}),$ an estimator for the potential
value, $V_{0}(\beta).$ For the parameter of the (unrestricted) optimal policy,
$\beta_{0}$, define the estimator
$\beta_{n}=\arg\max_{\beta}V_{n}(\beta,{b_{n}}).$

###  5.2 Trust Region Policy Optimization (TRPO)

Now that we have shown how to estimate $V_{0}$ in (3), we discuss how to
estimate $M_{0}$ in (5). For this, we write an estimator of of $KL_{0}$,
defined in (6), as

$KL_{n}(\beta,b)=\frac{1}{n}\sum_{i}\log\frac{\prod_{t=0}^{T}\pi_{b}(A_{i,t}|S_{i,t})}{\prod_{t=0}^{T}\pi_{\beta}(A_{i,t}|S_{i,t})}.$
(13)

We showed how we can use ${V}_{n}$ to estimate $V_{0}$ in (12). We hence
estimate $M_{0},$ defined in (5), with

$\displaystyle{M}_{n}(\beta,b_{n},\gamma)$
$\displaystyle={V}_{n}(\beta,b_{n})-\gamma KL_{n}(\beta,b_{n}).$ (14)

We estimate $\beta_{0,\gamma},$ defined in (7), with

$\beta_{n,\gamma}=\arg\max_{\beta}{M}_{n}(\beta,b_{n},\gamma).$ (15)

Increasing $\gamma$ increases the degree of behavior constraint of
$\pi_{\beta_{0,\gamma}},$ which is important for obtaining “closeness” to the
standard of care, as discussed in Section 3.2. As mentioned in Section 4, we
benefit from closeness to the standard of care when we translate a suggested
policy to the clinic, because the suggested policy will be more likely to be
adopted when its suggested treatment aligns with the established guidelines.
Increasing $\gamma$ also leads to stabilization of the objective, since
$V_{n}$ is typically more unstable than $KL_{n}$, where minimizing $KL_{n}$,
as discussed in Section 3.2, is equivalent to maximizing likelihood. This
stabilizes inference as well as estimation.

###  5.3 Adaptive relative sparsity

We finally define an estimator for $W_{0}$, defined in (4.1), as

$\displaystyle W_{n}(\beta,\beta_{n,\gamma},b_{n})$
$\displaystyle={M}_{n}(\beta,b_{n})-\lambda\sum_{k=1}^{K}w_{n,k}|\beta_{k}-b_{n,k}|,$
(16)

where we estimate $w_{0,k},$ defined in Section 4.1, with
$w_{n,k}={1}/{|\beta_{n,\gamma,k}-b_{n,k}|^{\delta}}.$ We then estimate
$\beta_{0,\gamma,\lambda},$ defined in (9), using

$\beta_{n,\gamma,\lambda}=\arg\max_{\beta}W_{n}(\beta,\beta_{n,\gamma},b_{n}).$
(17)

###  5.4 Tuning parameters

We now discuss the three tuning parameters in Equation (4.1):
$\gamma,\lambda,$ and $\delta$. The tuning parameter $\gamma$ impacts the
weight on the $KL$ divergence portion of the penalty in (5) and will determine
the closeness to the standard of care. From an estimation standpoint, $\gamma$
impacts the stability of the objective function, and should be chosen based on
the stability of the estimation in a training dataset, which we will
illustrate in simulations and in the real data analysis here. After selection,
in the post-selection inference step, there will be fewer free parameters, so
the estimation stability caused by $\gamma$ will be even larger; hence, it is
reasonable to choose a slightly smaller $\gamma$ but to still expect stability
in inference.

Given $\gamma,$ one can choose $\lambda$ as

$\lambda_{0}=\max\\{\lambda:V_{0}(\beta_{0,\lambda})\geq V^{min}\\},$ (18)

where $V^{min}=V_{0}(b_{0})+\sigma_{V}(b_{0}),$ and, if we use ${V}_{n},$
given in (12), as an estimator for $V_{0},$ then $\sigma_{V}$ is the
asymptotic standard deviation of $\sqrt{n}{V}_{n}.$ We take a $\max$ in (18)
to ensure maximum sparsity and closeness to behavior within the set of
policies that have acceptable value of at least $V^{min}$. We estimate
$\lambda_{0}$ from (18) using

$\lambda_{n}=\max\\{\lambda:{V}_{n}(\beta_{n,\lambda})\geq{V}_{n}^{min}\\},$
(19)

where ${V}_{n}^{min}={V}_{n}(b_{n},b_{n})+\sigma_{n,V}(b_{n}),$ and
$\sigma_{n,V}$ now refers to standard error of ${V}_{n}$, where estimation of
$\sigma_{n,V}$ is described in Appendix A.19. Note that the standard error is
not a general-purpose selection threshold, since the standard error will
decrease with increasing sample size. One might consider also the standard
deviation of the behavioral value or a certain percentage increase in value.

The tuning parameter $\delta>0,$ which is proposed in Zou (2006), impacts the
adaptivity of the adaptive Lasso penalty, and increasing $\delta$ should, in
theory, lead to a stronger penalty for the coefficients that are truly equal
to their behavioral counterparts and a weaker penalty for the coefficients
that truly diverge from their behavioral counterparts, as discussed in more
detail in Appendix A.5. The finite sample behavior of the adaptive Lasso
penalty, however, is sometimes unpredictable, which has been discussed in
e.g., Pötscher and Schneider (2009), so we recommend trying a few different
values of $\delta$ in a training set, as is done in Zou (2006), where the
authors use $\delta\in\\{.5,1,2\\}.$

##  6 Theory

We provide theory for inference for $\beta_{0,\gamma},$ which was defined in
(7). This is novel in its own right, since it applies to Trust Region Policy
Optimization (TRPO) (Schulman et al., 2015) whose inferential properties have
not been well characterized, and it further concerns a novel, weighted version
of TRPO, defined in (12). However, our overall goal is not to show results for
TRPO, but to operationalize this theory toward performing inference for
relative sparsity in the post-selection setting. We also provide theory that
will be used in the selection diagrams to visualize the variability our
estimators.

###  6.1 Assumptions

We make the following causal identifiability assumptions.

######  Assumption 3.

  1. (i)

Positivity: $\pi_{b_{0}}(A=a|S=s)>0\ \ \forall\ a,s.$

  2. (ii)

Consistency: $S_{t+1}(A_{0},\dots,A_{t})=S_{t+1}.$

  3. (iii)

No interference: Let $S_{t+1}(a_{0},\dots,a_{t})$ to be the potential state
under, possibly contrary to observation, $A_{0},\dots,A_{t}$ being fixed to
$a_{0},\dots,a_{t}.$ If $i$ and $j$ index different patients, we have that

$S_{i,t+1}(a_{i,0},\dots,a_{i,t},a_{j,0},\dots,a_{j,t})=S_{i,t+1}(a_{i,0},\dots,a_{i,t}).$

  4. (iv)

Sequential randomization:

$S_{t+1}(a_{0},\dots,a_{t}),S_{t+2}(a_{0},\dots,{a}_{t+1}),\dots,S_{T+1}(a_{0},\dots,{a}_{T})\perp\\!\\!\\!\perp
A_{t}|{S}_{t},{A}_{t-1}={a}_{t-1}.$

Under model 1, Assumption 3 (i) implies that $|b|<\infty$ and, consequently,
by Remark 2, that $|\beta|<\infty$. In our case, it is likely that Assumption
3 (ii), consistency, and Assumption 3 (iii), no intereference, are satisfied.
Assumption 3 (iv), sequential randomization, can be more problematic, although
we have included the same covariates that were used in the literature Futoma
et al. (2020); one could conceivably adjust for more covariates in future
work.

The following is necessary to establish asymptotic consistency.

######  Assumption 4.

Define $\bar{\mathcal{S}}=\mathcal{S}_{0},\dots,\mathcal{S}_{T+1},$
$\bar{\mathcal{A}}=\mathcal{A}_{0},\dots,\mathcal{A}_{T},$
$\bar{{S}}={S}_{0},\dots,{S}_{T+1},$ and $\bar{{A}}={A}_{0},\dots,{A}_{T}.$
For the objective defined in (35),
$m:\bar{\mathcal{S}}\times\bar{\mathcal{A}}\times B\times
B^{\prime}\mapsto\mathbb{R},$ where $\beta\in B$ and $b\in B^{\prime},$ we
have that $B\times B^{\prime}$ is compact. Moreover, for all $(\beta,b)\in
B\times B^{\prime},$ we have that $m(\cdot,\beta,b)$ is Borel measureable on
$\bar{\mathcal{S}}\times\bar{\mathcal{A}}$ and, for each
$(\bar{S},\bar{A})\in\bar{\mathcal{S}}\times\bar{\mathcal{A}}$ we have that
$m(\bar{S},\bar{A},\cdot)$ is continuous on $B\times B^{\prime}.$

######  Assumption 5.

The states are uniformly bounded; i.e., there exists $0<C<\infty$ such that
$P(|S_{t}|\geq C)=0,$ for all $t.$

Quantities such as mean arterial blood pressure (MAP), creatinine, or urine
output are physiologic quantities, and, therefore, random variables
representing these quantities will be restricted to take on finite values.

######  Assumption 6.

The reward is bounded; i.e., $\left|R(s,a,s^{\prime})\right|<\infty$ $\forall\
s,a,s^{\prime}.$

Often, the reward will be based on state variables, which are themselves
bounded by Assumption 5. It is therefore reasonable to assume boundedness of
the reward (although this would be violated if one were to, e.g., include
within the reward an infinite penalty for mortality).

Besides the Markov Decision Process (MDP) and causal assumptions (Assumption 2
and Assumption 3), we also make assumptions that allow us to extend Theorem
5.41 in van der Vaart (2000), which assumes boundedness of the partial
derivatives of $M_{0}$. We now argue that these boundedness assumptions are
reasonable, largely because of the causal assumption of positivity (Assumption
3 (i)) and the physiologic bounds on the state variables.

######  Assumption 7.

Let $\zeta=(\beta,b)^{T}\in\mathbb{R}^{2K}.$ The following partial derivatives
exist and, for any length-$T$ trajectory of states and actions, satisfy

$\displaystyle\frac{\partial^{3}}{\partial\zeta_{i}\zeta_{j}\zeta_{k}}\left(\frac{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}\sum_{t=0}^{T}R(S_{t},A_{t},S_{t+1})-\gamma\log\left(\frac{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}\right)\right)\leq\overset{\cdots}{\text{m}},$
(20)

for some integrable, measurable function $\overset{\cdots}{\text{m}},$ and for
every $\zeta=(\beta,b)^{T}$ in a neighborhood of
$\zeta_{0}=(\beta_{0,\gamma},b_{0})^{T}.$

We will argue in the following section that $\overset{\cdots}{\text{m}}$ in
Assumption 7 is a constant for the reinforcement learning problems that we are
interested in solving, which depend on the reward and the states, both of
which are usually bounded.

Let us consider the components of (20). We will start by discussing
$\sum_{t=0}^{T}R(S_{t},A_{t},S_{t+1}).$ Note that since $T$ is finite,
boundedness of the reward in Assumption 6 implies that

$\left|\sum_{0}^{T}R(s_{t},a_{t},s_{t+1})\right|<\infty,\forall(s_{0},\dots,s_{T+1},a_{0},\dots,a_{T}).$

Moreover, because we are considering the binary action case, the policies are
bounded above by 1 and below by 0. Following Assumption 3 (i), we have that
the inverse of the product of the policies is bounded (i.e.,
$\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})^{-1}<\infty$).

The derivatives in (20) are various combinations of the reward, states, and
policies and are therefore similarly bounded; one can see the forms of the
first and second order derivatives in Appendix A.18. Sometimes, a derivative
of the logarithm of a policy appears. In our generalized linear model setting,
because of model (1), we see that the differentiated logarithm turns into a
generalized linear model score of the form $(a-\pi(A=1|s))s,$ which is bounded
because the policies and state are bounded. The partial derivatives are over
$T$ time steps, and, because $T$ is finite, the summands or factors are
bounded.

The
$\log{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}\biggr{/}{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}$
term in (20) corresponds to the the Kullback-Leibler (KL) divergence between
the suggested policy and the data-generating, behavioral policy, defined in
(6), both of which are expit models according to (1). We have that minimizing
KL divergence is equivalent to maximizing the log likelihood, as discussed in
Section 5.5 of van der Vaart (2000). Hence, since the partial derivatives of
the log likelihood are well behaved, we have that the partial derivatives of
the $KL$ function are similarly well behaved.

Uniqueness of the maximizer of an objective function, $M_{0}$, is important
for establishing consistency of the maximizer of ${M}_{n}$.

######  Assumption 8.

We have uniqueness of the maximizer of $M_{0}(\beta,b_{0}),$ defined in (5);
i.e.,

$M_{0}(\beta_{0,\gamma},b_{0})>M_{0}(\beta,b_{0})$

for all $\beta\neq\beta_{0,\gamma}.$

Assumption 8 could be relaxed to a local uniqueness assumption (Loh and
Wainwright, 2013; Eltzner, 2020). In some problems, if taking the action
sequence $(A_{1}=1,A_{2}=0)$ and $(A_{1}=0,A_{2}=1)$ gives equivalent value,
the policy that maximizes value may not be unique, but, in many problems of
interest, the order of treatments matters.

######  Assumption 9.

We assume that $b_{n}$ is consistent for $b_{0},$ the data generating
behavioral policy; i.e. $b_{n}\overset{p}{\to}b_{0}$. Moreover, we assume that
the behavioral estimator is $\sqrt{n}$-consistent; i.e.,
$\sqrt{n}(b_{n}-b_{0})=O_{P}(1).$

Assumption 9 holds for the estimators we use for the behavioral policy,
assuming correct specification of the model in (1).

Define the Jacobian (gradient), Hessian, and cross derivative of ${M}_{n}$ and
$M_{0},$ respectively, as

$\displaystyle
J_{n}=\frac{\partial}{\partial\beta}{M}_{n}\in\mathbb{R}^{K},H_{n}=\frac{\partial^{2}}{\partial\beta^{2}}{M}_{n}\in\mathbb{R}^{K\times
K},X_{n}=\frac{\partial}{\partial
b}\frac{\partial}{\partial\beta}{M}_{n}\in\mathbb{R}^{K\times K},$
$\displaystyle
J_{0}=\frac{\partial}{\partial\beta}M_{0}\in\mathbb{R}^{K},H_{0}=\frac{\partial^{2}}{\partial\beta^{2}}M_{0}\in\mathbb{R}^{K\times
K},X_{0}=\frac{\partial}{\partial
b}\frac{\partial}{\partial\beta}M_{0}\in\mathbb{R}^{K\times K}.$ (21)

######  Assumption 10.

We have that $H_{0},$ which is defined in (6.1), exists and is non-singular.

Assumption 10 is standard and necessary to isolate the policy coefficients
from other terms in a Taylor expansion.

###  6.2 Preliminary remarks and results

We first include some preliminary remarks and results to aid us in proving the
theorems that will follow.

######  Remark 3.

Since the reward is a deterministic function of the states, under Assumptions
3 (i)-(iv), one can make arguments similar to those presented in Pearl (2009);
Muñoz and van der Laan (2012); van der Laan and Rose (2018); Ertefaie and
Strawderman (2018) and Parbhoo et al. (2022) to identify, as a function of the
observed data, the potential value $V_{0}(\beta),$ which is the value we would
obtain if we were to assign treatments based on the policy $\pi_{\beta}$. This
corresponds to the unweighted version of the estimator in 12.

######  Remark 4.

As shown in Thomas (2015), we have that

$E_{n}{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}\biggr{/}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}\overset{p}{\to}E{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}\biggr{/}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}$

by the Law of Large Numbers. Note further that
$E{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}\biggr{/}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}=1,$
as shown in Appendix A.4.

######  Remark 5.

By Slutsky’s theorem, since
$E_{n}{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}\biggr{/}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}\overset{p}{\to}1$
by Remark 4, we have that ${V}_{n}\overset{p}{\to}V_{0}$. This is also shown
in Thomas (2015).

The following consistency statements need to be verified, because we are using
weighted importance sampling, as shown in (12).

######  Lemma 1.

We have that ${M}_{n}$ is consistent for $M_{0},$ or that
${M}_{n}\overset{p}{\to}M_{0}.$

###### Proof.

See Appendix A.10 ∎

######  Lemma 2.

We have that

$E_{n}\frac{\partial}{\partial\beta}{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}\biggr{/}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}\overset{p}{\to}0,$
$E_{n}\frac{\partial}{\partial
b}{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}\biggr{/}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}\overset{p}{\to}0,$
$E_{n}\frac{\partial}{\partial
b}\frac{\partial}{\partial\beta}{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}\biggr{/}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}\overset{p}{\to}0.$

###### Proof.

See Appendix A.11. ∎

We will use Lemma 2 to show that the partial derivatives of ${M}_{n},$ defined
in (14), converge to the partial derivatives of $M_{0}$, defined in (5), which
requires verification because $M_{n}$ contains a weighting term, as shown in
(12). For the same reason, we verify the following Lemma.

######  Lemma 3.

We have that $J_{n}\overset{p}{\to}J_{0},H_{n}\overset{p}{\to}H_{0},$ and
$X_{n}\overset{p}{\to}X_{0}.$

###### Proof.

See Appendix A.12 ∎

###  6.3 Weighted Trust Region Policy Optimization (TRPO) consistency and
asymptotic normality

We will now ensure that $\beta_{n,\gamma}$ is well behaved asymptotically.
Recall that $\beta_{n,\gamma},$ is the maximizer of ${M}_{n},$ which is
defined in (14). Then $M_{n}$ serves as the “base” objective of the double-
penalized relative sparsity objective, $W_{n},$ which is defined in (16). We
need, essentially, that when the relative sparsity penalty goes away (in the
adaptive Lasso case), or is taken away (in the sample splitting, post-
selection case), we are left with an objective function that gives us a
maximizer, $\beta_{0,\gamma}$, that is amenable to inference.

To show consistency of $\beta_{n,\gamma},$ after making extensions to take
into account the weighting term in the importance sampling objective (12), we
can apply results from Wooldridge (2010) on two-step M-estimators. For
asymptotic normality of $\beta_{n,\gamma},$ we extend a classical proof,
versions of which can be found in e.g. van der Vaart (2000) and Wooldridge
(2010).

######  Theorem 1.

  1. (i)

We have consistency of the TRPO estimator $\beta_{n,\gamma}$ for
$\beta_{0,\gamma}$, or that
$\beta_{n,\gamma}\overset{p}{\to}\beta_{0,\gamma}$.

  2. (ii)

We have asymptotic normality for the weighted TRPO estimator,
$\beta_{n,\gamma}$, or that

$\sqrt{n}(\beta_{n,\gamma}-\beta_{0,\gamma})\overset{\mathcal{L}}{\to}-(H_{0})^{-1}\left(z_{0}+X_{0}q_{0}\right),$

where, scaling the gradient of (14), which is defined in (6.1), and taking a
limit,

$\sqrt{n}E_{n}\frac{\partial}{\partial\beta}\left(\frac{\frac{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}\sum_{t=0}^{T}R(S_{t},A_{t},S_{t+1})}{E_{n}\frac{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}}-\gamma\log\left(\frac{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}\right)\right)\bigg{|}_{\begin{subarray}{c}\beta=\beta_{0,\gamma}\\\
b=b_{0}\end{subarray}}\overset{\mathcal{L}}{\to}z_{0},$

$\sqrt{n}(b_{n}-b_{0})\overset{\mathcal{L}}{\to}q_{0},$ and the Hessian,
$H_{0},$ and the cross derivative, $X_{0},$ are defined in (6.1). Asymptotic
normality of $\beta_{n,\gamma}$ then follows from the fact that $H_{0}$ and
$X_{0}$ are constants and $z_{0}$ and $q_{0}$ are normally distributed random
variables.

###### Proof.

For Theorem 1 (i), see Appendix A.13. For Theorem 1 (ii), see Appendix A.14. ∎

###  6.4 Adaptive Lasso asymptotic normality

Having shown that $\beta_{n,\gamma},$ the maximizer of ${M}_{n},$ is well
behaved in the limit, we will now prove a similar result for
$\beta_{n,\gamma,\lambda},$ the maximizer of the full, double-penalized,
relative sparsity objective, $W_{n},$ which is defined in (16). We prove this
result by extending a result from Zou (2006).

######  Theorem 2.

Asymptotic normality of $\beta_{n,\gamma,\lambda}.$ Assume that, for
$\delta>0,$

$\frac{\lambda_{n}}{\sqrt{n}}\rightarrow 0\text{ and
}\lambda_{n}n^{(\delta-1)/2}\rightarrow\infty.$

Note that these conditions are necessary to obtain appropriate limiting
behavior (either disappearance or predominance) of the penalty term. Define
the active set, $\mathcal{A}$, to be the indices of the parameters that truly
differ from their behavioral counterparts,

$\mathcal{A}=\\{k:{(\beta_{0,\gamma,k}}-b_{0,k})\neq 0\\}.$ (22)

Note that $\mathcal{A}$ depends on $\gamma,$ since $\gamma$ defines the
estimand without the relative sparsity penalty. Define
$\beta_{0,\gamma,\mathcal{A}}$ as the coefficients indexed by $\mathcal{A}.$

Then for the coefficients that differ from their behavioral counterparts,

$\sqrt{n}(\beta_{n,\gamma,\mathcal{A}}-\beta_{0,\gamma,\mathcal{A}})\overset{\mathcal{L}}{\to}N(0,(H_{0,\mathcal{A}\mathcal{A}})^{-1}\text{var}(r_{v_{0}}^{T})((H_{0,\mathcal{A}\mathcal{A}})^{-1})^{T}).$

where
$r_{v_{0}}=[z_{0,\mathcal{A}}^{T}+u_{0,\mathcal{A}^{C}}^{T}H_{0,\mathcal{A}^{C}\mathcal{A}}+u_{0,\mathcal{A}^{C}}^{T}H_{0,\mathcal{A}\mathcal{A}^{C}}^{T}+v_{0,\mathcal{A}}^{T}(X_{0}^{T})_{\mathcal{A}\mathcal{A}}+v_{0,\mathcal{A}^{C}}^{T}(X_{0}^{T})_{\mathcal{A}^{C}\mathcal{A}}]^{T}$
is a normally distributed combination of $z_{0},H_{0},X_{0}$ (which were
defined in (6.1)), $u_{0},$ and $v_{0},$ and the subscripts
$\mathcal{A},\mathcal{A}^{C}$ indicate the indices associated with,
respectively, the non-behavioral and behavioral components of each vector or
matrix. The coefficients that do not differ from their behavioral
counterparts, $\beta_{0,\gamma,\mathcal{A}^{C}}$, converge to the limiting
random variables of the behavioral policy estimators.

###### Proof.

See Appendix A.15. ∎

Thus we obtain, for the truly non-behavior coefficients, which are the
solutions to the maximization of the base objective, ${M}_{n},$ asymptotic
normality, because these coefficients are untouched (in the limit) by the
adaptive penalty. We simultaneously drive the truly behavioral coefficients to
their behavioral counterparts, which are also asymptotically normal, because
they maximize likelihood.

##  7 Inference for policy coefficients

###  7.1 Estimator for the variance of the coefficients

We have given expressions for the asymptotic forms of the coefficients
$\beta_{n,\gamma}$ and $\beta_{n,\gamma,\lambda}$ based on the theory above,
but now we give more detail on how to estimate the variances of these
estimators in practice. To do so, we will need to build on the partial
derivatives of ${M}_{n},$ the forms of which are given in Lemma 3 and Appendix
A.18.

Recall that $H_{n},$ defined in (6.1), is an estimator for the Hessian
$H_{0}.$ Recall that $X_{n},$ defined in (A.12), is an estimator for the cross
derivative, $X_{0}$. Define the function over which we take an empirical
expectation in the gradient of (14) as

$z=\frac{\partial}{\partial\beta}\left(\frac{\frac{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}\sum_{t=0}^{T}R(S_{t},A_{t},S_{t+1})}{E_{n}\frac{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}}-\gamma\log\left(\frac{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}\right)\right)\bigg{|}_{\begin{subarray}{c}\beta=\beta_{0,\gamma}\\\
b=b_{0}\end{subarray}}.$ (23)

Note that $J_{n}=E_{n}z,$ for $J_{n}$ defined in (6.1). For $z$ defined in
(23), define $z_{i}$ such that $J_{n}=\frac{1}{n}\sum_{i}z_{i}$. Define

$q=(E_{n}l^{\prime\prime}(b_{n}))^{-1}l^{\prime}(b_{n})\text{ and
}q_{n}=E_{n}q.$ (24)

For $q$ defined in (24), define $q_{i}$ such that
$q_{n}=\frac{1}{n}\sum_{i}q_{i}$.

We then derive in Appendix A.17 the following estimator for $\sigma^{2}_{0},$
the asymptotic variance of $\beta_{n,\gamma},$

$\sigma^{2}_{n}=(H_{n})^{-1}\left(\frac{1}{n}\sum_{i}(z_{i}+X_{n}{q_{i}})^{2}\right)\left(H_{n}^{-1}\right)^{T},$
(25)

where all terms are evaluated at plug-in estimates, $\beta_{n,\gamma}$ and
$b_{n}$, of the true parameters, $\beta_{0,\gamma}$ and $b_{0}$, respectively.

We use (25) to perform rigorous inference in the post-selection step. We also
use the estimator (25) to obtain a rough visualization of the variability of
$\beta_{n,\gamma,\lambda}$ in the training data, which is heuristic, but can
be helpful for assessing the impact of $\gamma$ on the stability of
$M_{n}=V_{n}-\gamma KL_{n}$, defined in (14). The stability of $M_{n}$ is
important for inference in the post-selection step. To assess this stability,
we can also examine the variance of the value, an estimator for which is given
below.

###  7.2 Estimator for the variance of the value

To select $\lambda,$ the tuning parameter for the relative sparsity penalty in
(16), it is useful to estimate the variance of the value function. We use the
following plug-in estimator, which is derived in Appendix A.19,

$\sigma^{2}_{n,V,w}=\frac{\frac{1}{n}\sum_{i=1}^{n}\left(\frac{\prod_{t=0}^{T}\pi_{\beta}(A_{i,t}|S_{i,t})}{\prod_{t=0}^{T}\pi_{b}(A_{i,t}|S_{i,t})}\sum_{t=0}^{T}R(S_{i,t},A_{i,t},S_{i,t+1})-{V}_{n}\right)^{2}}{\left(\frac{1}{n}\sum_{i=1}^{n}\frac{\prod_{t=0}^{T}\pi_{\beta}(A_{i,t}|S_{i,t})}{\prod_{t=0}^{T}\pi_{b}(A_{i,t}|S_{i,t})}\right)^{2}}.$
(26)

This estimator will give us a sense of the uncertainty of the value function.
This estimator is especially important for the selection rule in (19), where
we require a standard error of the value under the behavioral policy. In other
words, we require the variance of $V_{n}(b).$ When $\beta=b,$ the estimator in
(26) reduces to a simple empirical variance.

###  7.3 Estimation algorithm

First, divide the dataset in half into split 1 (for selection) and split 2
(for post-selection inference).

Divide split 1 into a split 1 train and split 1 test.

In split 1 train, optimize (14) to obtain pilot estimators,
$\beta_{n,\gamma},$ and maximize likelihood to obtain $b_{n},$ Use these to
optimize (16) to obtain $\beta_{n,\gamma,\lambda}$ for three values of
$\gamma,$ three values of $\delta,$ and ten values of $\lambda.$

In split 1 train, visualize each $(\gamma,\delta)$ pair over varying
$\lambda.$

In split 1 train, choose $(\gamma,\delta)$ based on the visualizing the
variability of the estimates $\beta_{n,\gamma,\lambda}$ using estimator (25).

In split 1 train, simultaneously, view the value in (12) as a function of
$\gamma,\lambda,$ and $\delta,$ along with its variance, using the estimator
(26).

Also, using split 1 test, estimate the value.

Given $\gamma$ and $\delta$, select $\lambda$ using (19) in split 1 train.

Given a $\gamma,\delta,$ and $\lambda,$ which creates selection
$1_{\mathcal{A}},$ where $\mathcal{A}$ indexes the selected covariates:

In split 2 (the post-selection step), optimize (14) to obtain
$\beta_{n,\gamma,\mathcal{A}}$ (where $\mathcal{A}$ in
$\beta_{n,\gamma,\mathcal{A}}$ indexes the selected components) .

In split 2, perform inference using Theorem 1 and the corresponding estimator
(25), which provides confidence intervals for the selected components,
$\beta_{0,\gamma,\mathcal{A}}$.

##  8 Simulations

We perform simulations to better understand the behavior of our estimators in
Equations (25) and (26) as a function of the tuning parameters $\gamma$,
$\lambda$, and $\delta$ (recall that these parameters are discussed in Section
5.4).

###  8.1 Simulation scenario

For our simulations, Set $\alpha=P(\text{type 1 error})=0.05$. We will fix the
sample size to be $n=1000,$ the number of Monte-Carlo repetitions for the
selection to be $M_{S}=100,$ the number of Monte-Carlo repetitions for
coverage to be $M_{C}=500$, the trajectory length to be $T=2,$ the state to be
$S\in\mathbb{R}^{K},$ where $K=2$, and the true behavioral policy parameter to
be $b_{0}=(-0.3,0.2)^{T}$. We further define the reward to be

$R(s_{t},a_{t},s_{t+1})=-s_{t,2}a_{t}.$ (27)

The reward in (27) is convenient, because it illustrates our method and also
allows us to derive theoretical results with respect to $\beta_{0,\gamma}.$
These results are discussed in Appendix A.20 and Appendix A.21 and can be used
to evaluate coverage in the simulation studies.

###  8.2 Data generation

For our data generation, we require that the states have constant variance
over time, which is the case with many physiological measurements, and we
therefore use the generative model in Ertefaie and Strawderman (2018), which
we will describe briefly below. We set the standard deviation of $\epsilon,$
to the $K\times K$ identity matrix, $\sigma_{\epsilon}=I_{K},$ and we set the
treatment effect to be $\tau_{k}=0.1$ for all $k$. Let $\mu_{0}=1_{K},$ where
$1_{K}$ is a $K$-dimensional vector of ones, and draw the first state and
action, $S_{0}\sim N(0_{K},I_{K})$ and $A_{0}|S_{0}\sim
Bern(\text{expit}(b_{0}^{T}S_{0})).$ For each ensuing time step $t$, for
dimension $k$, draw ${\epsilon_{k}}\sim N(1,\sigma_{\epsilon,k}^{2}),$ where
$\sigma_{\epsilon,k}^{2}=1$. Then draw

$S_{t,k}|S_{t-1,k},A_{t}={(S_{t-1,k}-\mu_{t-1,k}+\epsilon_{k}})/{(1+\sigma^{2}_{\epsilon,k})^{1/2}}+\mu_{t,k},$

where $\mu_{t+1,k}=\mu_{t,k}(1+\tau_{k}A_{t}),$ and $A_{t}|S_{t}\sim
Bern(\text{expit}(b_{0}^{T}S_{t})).$ Note that $\sigma_{\epsilon,k}^{2}$ is
constant over time, and that the division by $\sigma_{\epsilon,k}^{2}$ in the
expression for $S_{t,k}$ ensures that the variance of the states are constant
over time. In simulations, because we set $\sigma^{2}_{\epsilon,k}=1,$ we
generate states with unit variance. We then do not have to scale the states to
have equal variance as discussed in A.6. This is useful because, as described
in Section A.20, we will be using the fact that we know the functional form of
the reward in (27) to analytically derive an expression to estimate
$\beta_{0,\gamma}$, but this expression will be impacted by scaling of the
state. When we just observe the reward , as we do in the real data, this is
not an issue, and we do scale the states.

###  8.3 Selection diagrams

Figure 1:  Selection diagrams for the simulated data
($n_{train}=250,n_{test}=250$). Over increasing $\gamma$ (going down) and
$\delta$ (going right), we show the average coefficients in the suggested
($\beta_{n,\gamma,\lambda})$ and behavioral ($b_{n}$) policies, the average
difference in probability of treatment between the two policies, and the
average value ($V_{n}$) for the suggested policy, all of which were computed
in the first split of the data . The dotted vertical line indicates
$\lambda_{n}$, a choice of $\lambda$ based on (19). Note that the average
suggested policy probability of treatment is
$\pi_{sugg}=(1/nT)\sum_{i}\sum_{t}{\pi}_{\beta_{n,\gamma,\lambda}}(A_{i,t}=1|s_{i,t})$
and vice versa for $\pi_{beh}$. The shaded regions in the coefficient
($\beta_{n,\gamma,\lambda}$) panels correspond to (25), and the dotted lines
show one standard error estimated empirically. The shaded regions in the value
panels show one standard error based on (26), which was used to select
$\lambda_{n}$ using (19), and the dotted lines show one standard error
estimated empirically.

Figure 1 shows results for $M_{S}=100$ Monte-Carlo datasets, and panels are
arranged in triplets indexed by $\gamma$ and $\delta$, where $\gamma$ and
$\delta$ increase as we descend the plot or go to the right, respectively. We
see that some variables approach their behavioral values less rapidly, giving
us relative sparsity, as observed in Weisenthal et al. (2023). Under the
assumption that (1) holds, and recalling that $\beta_{0}=\arg\max V_{0},$ we
expect to see results that align with the fact that the unconstrained
maximizer is $\beta_{0}=(0,-\infty),$ for which a proof is provided in
Appendix A.21. Hence, we expect the sign of the coefficient $\beta_{n,2}$ to
be negative and to become larger in magnitude as $\lambda\rightarrow 0,$ which
aligns with the results in Figure 1.

When $\gamma$ is large enough, and therefore the objective function is stable
enough, we see that the “empirical” standard errors of the coefficients, which
are shown as dotted black lines, align with variability shown by (25), which
are shown as shaded regions.

In the second panel of each triplet, which shows the difference in the
probability of treatment under the behavioral and suggested policies, we see
that increasing $\gamma$ gives us some baseline closeness to behavior, and the
gap is further closed by increasing $\lambda$. The central triplet in Figure 1
contains vertical dotted lines indicating the selected policy. For each
dataset, we automatically choose $\lambda_{n},$ based on (19), such that the
corresponding suggested policy is as sparse as possible but has an increase in
value of one standard error above the behavior policy. The dotted vertical
line in the central triplet indicates the choice of $\lambda_{n}$ when using
the coefficients averaged over Monte-Carlo datasets. Note that this selection,
using (19), was conducted only on one split of the data, which was itself
split into a training set and a test set. The former is used to estimate
coefficients and the latter to assess held-out value. We see good overall
closeness to behavior for the selected policy, and that the second covariate
was selected, as expected.

In general, for $V_{n}$ in Figure 1, for large enough $\gamma$, we see that
the shaded regions, which correspond to one standard error of $V_{n}$
according to the theoretical estimator in (26), are close to the dotted lines,
which correspond to one standard error estimated empirically. For both the
coefficients and the value, the objective function becomes more unstable for
small $\gamma.$ It is important not to attempt to attain too much value,
$V_{n}$, by making $\gamma$ small, because the more one upweights $V_{n},$ the
more unstable the estimation and inference become. However, Figure 1
exaggerates this instability, because it is based on only half of the data,
which is further divided in half (one half is used for training and one half
to compute held out value). Since the post-selection inference will be
conducted on a larger sample, and the number of covariates will decrease after
selection, both of which stabilize the estimation and inference, as discussed
in Section 5.4, one can select a $\gamma$ that is slightly smaller than what
Figure 1 might suggest. Having made selections of $\gamma$ and $\lambda$, we
then perform inference on a held out, independent split of the dataset, for
which we will now provide results in Section 8.4.

###  8.4 Post-selection inference

For each dataset, to avoid issues with post-selection inference that occur
when one selects and does inference on the same dataset (Pötscher and
Schneider, 2009), we first perform selection using (16) in one split of the
data, and then we use a second, independent split for inference. For the
latter, we perform inference on the non-behavioral components of
$\beta_{n,\gamma}$ within the suggested policy $\pi_{\beta,b},$ which is
defined in Section 4.2, Equation (10). Recall that each non-selected
(behavioral) coefficient, $\beta_{k}$ in $\pi_{\beta,b},$ which is defined in
(10), is fixed in advance to its behavioral counterpart, $b_{n,k}$. We
therefore re-estimate and perform inference on the non-behavioral components
of $\beta_{\gamma,n}$. We show post-selection inference results in Table 1.

| Suggested ($\beta_{n,\gamma})$ | Behavioral ($b_{n}$)
---|---|---
$S_{t,1}$ | set to $b_{n}$ | -0.302 (-0.428, -0.176)
$S_{t,2}$ | -0.129 (-0.261, 0.003) | 0.189 (0.066, 0.312)
Table 1: Estimated coefficients (95% confidence interval) for held out dataset
post-selection inference; $n=$500, $\gamma=$3.

We see that, for the selected second coefficient, the confidence intervals in
Table 1 show a significant difference from behavior . In the same way that we
showed our theoretical estimators for the coefficients and value in Figure 1
against an empirical reference, we do the same for the confidence intervals in
Table 1. We know that the coefficients that are set to their behavioral
counterparts have nominal coverage, because they are derived by maximum
likelihood estimation (Casella and Berger, 2002; van der Vaart, 2000), but we
must check the coverage for the non-behavioral coefficient corresponding to
$S_{t,2}$. In Section 8.5, we do so by conducting an additional Monte-Carlo
study in which we generate Table 1 $M_{C}=500$ times, and check whether the
confidence interval coverage for $\beta_{0,\gamma,2}$ is nominal.

###  8.5 Coverage of active parameters

We assess the results in Table 1 with a Monte-Carlo study. In the process, we
assess our estimator, $\sigma_{n}^{2}$, for the variance of $\beta_{n,\gamma}$
(Theorem 1 (ii) and Equation (25)). Details of the Monte-Carlo summary
statistics are given in Appendix A.22, but we summarize them briefly here. To
assess coverage, we must know $\beta_{0,\gamma}.$ However, in this policy
search setting, $\beta_{0,\gamma}$ is unknown to us, even when we are
simulating the data. Since we specified the functional form of the reward to
be (27), however, we can estimate $\beta_{0,\gamma}$ arbitrarily well by
sidestepping the need for importance sampling, as described in Appendix A.20.
We thus obtain the “true” parameter, $\beta_{0,n,\gamma}$, defined in (68) of
Appendix A.20. We index $\beta_{0,n,\gamma}$ with $0$ to designate its role as
a reference against which we will check our theory, and we treat
$\beta_{0,n,\gamma}$ as $\beta_{0,\gamma}$ when we evaluate coverage.

To check consistency (Theorem 1 (i)), we also compare the “true” estimand
$\beta_{0,n,\gamma}$ to the estimated coefficients,
$\beta_{n,\gamma}=\arg\max{M}_{n},$ averaged over Monte-Carlo datasets, which
we denote $\bar{\beta}_{n,\gamma},$ and define in (69) in Appendix A.20. We
also derive a Monte-Carlo estimator for the standard deviation of the
estimator $\beta_{n,\gamma},$ which we call $\sigma_{n}(\beta_{0,n,\gamma}),$
and which is just the standard deviation of the coefficient estimates over
Monte-Carlo datasets. More detail on the computation of this quantity is given
in Appendix A.22, Equation (71). We include a subscript 0 in
$\sigma_{0,n}(\beta_{0,n,\gamma})$ to designate its role as a “true reference”
against which we will check our theoretical results.

Define the estimated standard deviation that we obtain from (25) based on
Theorem 1 (ii), as $\bar{\sigma}_{n}(\beta_{n,\gamma}),$ where the overbar
indicates that this estimated variance was computed for each Monte-Carlo
dataset and then averaged (more detail is given in Appendix A.22, Equation
(70)). We will compare the estimated standard error,
$\bar{\sigma}_{n}(\beta_{n,\gamma}),$ to the “true” standard error,
$\sigma_{0,n}(\beta_{0,n,\gamma}),$ and we will also assess coverage. Note
that we are simulating “post-selection” inference, so we only assess coverage
for selected coefficients, where the selection was made in Figure 1, and, in
this case, concerns only $\beta_{n,\gamma,2}$, which corresponds to the second
covariate, $S_{t,2}$.

$\gamma$ | 0.01 | 3.00 | 6.00
---|---|---|---
True: $\bar{\beta}_{0,n,\gamma}$ | -6.33 | -0.13 | 0.03
Estimated: $\bar{\beta}_{n,\gamma}$ | -6.35 | -0.13 | 0.03
Bias | -0.03 | 0.00 | 0.00
True: $\sigma_{0,n}(\beta_{n,\gamma})$ | 13.62 | 1.60 | 1.50
Estimated: $\bar{\sigma}_{n}(\beta_{n,\gamma})$ | 16.08 | 1.53 | 1.41
Coverage | 0.98 | 0.95 | 0.93
Length CI | 2.82 | 0.27 | 0.25
Table 2:  Bias, standard deviation, and coverage for the coefficient,
$\beta_{0,\gamma,2}$, of the selected covariate, $S_{t,2}$. For simulation
settings $n=$ 500, $T=$ 2, $K=$ 2, and $M_{C}=$ 500, we show these performance
measures while varying $\gamma$. We also show the estimated (indexed by $n$
alone) and “true” (indexed by $0$ and $n$, indicating an empirical estimate of
the true value) coefficients, $\bar{\beta}_{n,\gamma}$ and
$\bar{\beta}_{0,n,\gamma}$, and standard deviations,
$\bar{\sigma}(\beta_{n,\gamma})$ and $\sigma_{0,n}(\beta_{n,\gamma})$, where
the overbar indicates an average over Monte-Carlo datasets.

Results are shown in Table 2, where we see roughly nominal coverage ($0.95$)
for the active covariate in this problem, supporting Theorem 1 (ii) and the
corresponding theoretical variance from (25). We see over-coverage for small
$\gamma.$ As discussed in Section 8.3, the parameter $\gamma$ impacts the
stability of the objective. If $\gamma$ is too small, ${V}_{n}$, which is
unstable, will dominate the objective in (14), and we will gain value but lose
stability. If $\gamma$ is larger, $KL_{n},$ which is more stable (as discussed
in Section 3.2), will be more prominent, and we will lose value, but we will
gain stability. However, note that while the coverage for the smallest
$\gamma$ is conservative (the variance is over-estimated), it is not as severe
as one would expect when viewing the selection diagram for the corresponding
$\gamma$ in the top row and middle column of Figure 1, in which it appears
that the coefficient estimates are quite unstable. This is because, as
discussed in Sections 5.4 and 5.4, there are fewer degrees of freedom and a
larger sample after selection, because some coefficients are fixed to their
behavioral counterparts, and an entire half of the dataset is devoted to post-
selection inference (rather than one quarter, which is the fraction devoted to
selection).

##  9 Real data analysis

We illustrate the proposed methodology and theory on a real dataset generated
by patients and their healthcare providers in the intensive care unit, as in
Weisenthal et al. (2023). We show that we can derive a relatively sparse
policy and perform inference for the coefficients.

###  9.1 Decision problem

We consider the same real data decision problem as in Weisenthal et al.
(2023), but in the multi-stage setting. There is variability in vasopressor
administration in the setting of hypotension (Der-Nigoghossian et al., 2020;
Russell et al., 2021; Lee et al., 2012). Although vasopressors can stabilize
blood pressure, they have a variety of adverse effects, making vasopressor
administration an important decision problem. We extend code from Weisenthal
et al. (2023), which was derived from Futoma et al. (2020); Gottesman et al.
(2020), to process the freely available, observational electronic health
record dataset, MIMIC III (Johnson et al., 2016b, a; Goldberger et al., 2000).
We include patients from the medical intensive care unit (MICU), as in
Weisenthal et al. (2023). We illustrate how we can provide inference for a
relatively sparse policy in this setting.

As in Weisenthal et al. (2023), we begin the trajectory at the beginning of
hypotension, which is defined as a mean arterial pressure (MAP) measurement
that is less than $60,$ a cutoff used in Futoma et al. (2020). We consider the
first 45 minutes after hypotension onset, where the first 15 minutes is
$S_{0}$, the second 15 minutes $S_{1},$ and the third $S_{2}.$ Eleven patients
left the MICU before 45 minutes, so we excluded those patients. Actions will
be $A_{0},$ taken after observing $S_{0}$ (we take the last measured
covariates in that time window), and $A_{1},$ taken after observing $S_{1}.$
We restrict ourselves to two stages, because it is important to stabilize MAP
early, and, also, because patients often leave the ICU due to death or
discharge (if we instead used e.g., 10 stages, many patients would leave,
leading to more missingness). We consider any vasopressor administration to be
an action, and different vasopressors are aggregated and normalized as in
Futoma et al. (2020); Komorowski et al. (2018) to their norepinephrine
equivalents. As in Weisenthal et al. (2023), because the norepinephrine
duration of action is short, we assume that a vasopressor administered in each
15 minute interval does not affect the blood pressure at the end of the
following 15 minute interval.

We consider the same set of covariates as those in Weisenthal et al. (2023)
and Futoma et al. (2020), which includes MAP, heart rate (HR), urine output
(Urine), lactate, Glasgow coma scale (GCS), serum creatinine, fraction of
inspired oxygen (FiO2), total bilirubin, and platelet count. As in Weisenthal
et al. (2023), based on Futoma et al. (2020), extreme, non-physiologically
reasonable values of covariates were floored or capped, and missing data was
imputed using the median or last observation carry forward. In terms of the
reward, we have that $S_{t}$ contains MAP as its first component; hence, we
define $R(S_{t},A_{t},S_{t+1})=S_{t+1,1}.$ In other words, we define the
reward as
$R(S_{t},A_{t},S_{t+1})=R((MAP_{t},\dots)^{T},A_{t},(MAP_{t+1},\dots)^{T})=MAP_{t+1},$
where the notation $(MAP_{t},\dots)$ indicates that the state depends on other
covariates besides $MAP$. This reward reflects the short-term goal of
increasing blood pressure in the setting of hypotension, and vasopressors
should increase this reward. This reward is imperfect but sensible, and, as
discussed in Weisenthal et al. (2023), by constraining to the behavioral
policy (the standard of care), we are able to derive a suggested policy that
improves outcomes with respect to this sensible reward, but does not
exclusively maximize this imperfect reward.

After excluding 39 patients who left the intensive care unit within the first
45 minutes of receiving a $MAP<60$ (which promps entry into the cohort), we
had $n=11,715$ patients. As in Weisenthal et al. (2023), we start by checking
that the behavioral policy model in (1) is specified correctly. To this end,
we estimate a calibration curve (Van Calster et al., 2016; Niculescu-Mizil and
Caruana, 2005), which is shown in Figure A.1 of Appendix A.23 and suggests
that the model specification in (1) is reasonable.

###  9.2 Selection diagrams

In Figure 2, we show selection diagrams, as we did in the simulations. In
Figure 2, we see that, for fixed $\delta$ and $\gamma,$ the suggested policy
coefficients, $\beta_{n,\gamma,\lambda},$ approach their behavioral
counterparts as $\lambda$ increases. We see that, like in Weisenthal et al.
(2023), MAP is isolated, and a relatively sparse policy is derived that still
has value that is one standard error above the behavioral value. The selected
$\lambda$ of this relatively sparse policy, which we denote $\lambda_{n},$ is
shown as the dotted, vertical line in the central triplet of Figure 2, and was
determined using (19).

The shaded regions in the coefficient panels show the variability of
$\beta_{n,\gamma,\lambda}$; we see that there is considerable variability,
especially with small $\gamma$. As noted in section 8.3, this variability is
likely exaggerated, since the coefficients are estimated on only one half of
the data, which is further split in half. In the post-selection step, we will
have a larger sample and fewer covariates, both of which stabilize the
problem, as discussed in Sections 5.4 and 8.5. We also see that the standard
error of $V_{n}$ (shaded) is small for large $\lambda,$ where the suggested
and the behavioral policy are the same. It is the standard error in this
region that is used to select $\lambda$ in (19). Given a selection,
$\lambda_{n},$ we now perform post-selection inference in a held-out split of
the data, results for which are shown in Table 3.

Figure 2: Selection diagrams for the real data
($n_{train}=1,176,n_{test}=1,176$). Over increasing $\gamma$ (going down) and
$\delta$ (going right), we show the average coefficients in the suggested
($\beta_{n,\gamma,\lambda})$ and behavioral ($b_{n}$) policies, the average
difference in probability of treatment between the two policies, and the
average value ($V_{n}$) for the suggested policy, all of which were computed
in the first split of the data . The dotted vertical line indicates,
$\lambda_{n}$, a choice of $\lambda$ based on (19). Note that the average
suggested policy probability of treatment is
$\pi_{sugg}=(1/nT)\sum_{i}\sum_{t}{\pi}_{\beta_{n,\gamma,\lambda}}(A_{i,t}=1|s_{i,t})$
and vice versa for $\pi_{beh}$. The shaded regions in the coefficient
($\beta_{n,\gamma,\lambda}$) panels correspond to (25) (to declutter the plot,
and because MAP was the only selected covariate, we show this only for MAP).
The shaded regions in the value panels show one standard error based on (26),
which was used to select $\lambda_{n}$ based on (19).

###  9.3 Post-selection inference

As shown in Figure 2, we selected $\gamma,$ $\lambda,$ and $\delta$ in the
first split of the real data. Given this selection, we now perform post-
selection inference on the second split of the data. We report results in
Table 3. In particular, as in Weisenthal et al. (2023), we see that all
coefficients except one are fixed to their behavioral counterparts, making it
easy to discuss and justify the suggested policy to the patients and providers
who may choose to adopt it. Unlike in Weisenthal et al. (2023), we now have a
95% confidence interval for the coefficient for MAP, which was derived from
Theorem 1 (ii) and the corresponding estimator (25). Note that the confidence
interval shown in Table 3 is much narrower than the shaded region shown in
Figure 2; the former was rigorously derived based on Theorem 1 (ii), whereas
the latter was just a heuristic way to visualize variability. Also note that,
because we are in the post-selection setting, the sample size used to derive
the confidence interval in Table 3 is twice as large as that used to estimate
(25) in Figure 2, and, because all but one parameter in the post-selection
step are set to their behavioral counterparts, there is one free parameter in
Table 3, while there are nine free parameters in Figure 2.

| Suggested ($\beta_{n,\gamma})$ | Behavioral ($b_{n}$)
---|---|---
MAP | 0.132 (-0.136, 0.400) | -0.070 (-0.159, 0.019)
HR | set to $b_{n}$ | 0.052 (-0.069, 0.173)
urine | set to $b_{n}$ | -0.300 (-0.478, -0.122)
lactate | set to $b_{n}$ | 0.426 ( 0.307, 0.546)
GCS | set to $b_{n}$ | -0.585 (-0.679, -0.490)
creatinine | set to $b_{n}$ | 0.186 ( 0.080, 0.293)
Fio2 | set to $b_{n}$ | 0.118 (-0.004, 0.241)
bilirubin | set to $b_{n}$ | 0.033 (-0.065, 0.131)
platelets | set to $b_{n}$ | -0.055 (-0.176, 0.065)
Table 3: Estimated coefficients (95% confidence interval) for held out dataset
post-selection inference; $n=$2,352 and $\gamma=$25.

In Table 3, we see that the confidence interval for the suggested policy
coefficient for MAP is shifted more toward a positive range, and is much
wider, with considerable negative and positive margins, than the confidence
interval for the behavioral policy coefficient for MAP, which is narrow and
almost entirely within a negative range.

##  10 Discussion

In our work, we developed methodology and theory to enable inference when
using the relative sparsity penalty developed in Weisenthal et al. (2023).
This inference framework allows one to construct confidence intervals for the
relative sparsity coefficients, improving the rigor of the method, and
ultimately facilitating safe translation into the clinic. To our knowledge, we
are the first to fully characterize the difficulties, such as infinite
estimands, with performing inference in the policy search setting under a
generalized linear model policy. We created finite, behavior-constrained
estimands by repurposing a weighted version of Trust Region Policy
Optimization (TRPO) (Schulman et al., 2015) as the “base” objective within the
relative sparsity framework. We proved novel theorems for weighted TRPO and
operationalized these results toward inference in a sample splitting
framework, which is free of issues associated with post-selection inference.
Unlike standard sample splitting techniques, our framework required that we
set non-selected parameters to some value other than zero, which considerably
complicated the partial derivatives of the objective functions, since the
nuisance began to appear in non-standard locations such as the numerator of
the inverse probability weighting ratio. We then developed an adaptive
relative sparsity penalty, which improved the discernment of the penalty. We
developed all of our methodology and theoretical results for the observational
data setting in the multi-stage, generalized linear model framework. Finally,
we illustrated our framework for inference for the relative sparsity penalty
on an intensive care unit, electronic health record, for which estimation was
non-trivial, and inference was even more difficult. In simulations, and in the
real data, we rigorously characterized sensitivity of the proposed inference
framework to the tuning parameters. We finally presented selection diagrams,
which are tools that help to select the tuning parameters using training data.

There are several opportunities for future work. For example, although we have
developed our method for large observational datasets, the current sample
splitting scheme could still be improved to make better use of the data (e.g.,
a bootstrap could be performed, rather than a single sample split). Also, as
discussed in Weisenthal et al. (2023), the real data analysis could be refined
by including a reward that takes into account mortality and morbidity.
Further, as discussed in Weisenthal et al. (2023), relaxing the linearity
assumption of the behavioral policy model (although we also checked the
reasonableness of this specification here) would be a good direction for
future work. The assumption that vasopressors administered in one time step
have a negligible effect on the MAP observed at the end of the next time step
is perhaps overly simplified, even though intravenous vasopressors have a
short duration of action. In general, more proximal administrations might have
more of an impact than more distal administrations. Discretizing time is also
a considerable simplification.

Other assumptions that we make in our work, such as the global uniqueness of
the maximizer of the objective function, might be overly restrictive; it would
be useful to consider a local uniqueness instead. Also, it would be
interesting to explore other “base” objective functions, such the weighted
likelihood objective of Ueno et al. (2012), which may have favorable
properties in this regard. The stationarity of the behavioral policy could be
relaxed by indexing the policy parameters by time step. The Markov property of
the behavioral policy might be more challenging to relax. Unlike many other
methods, we do not make Markov and stationarity assumptions for the transition
probabilities. One could increase the likelihood that the no unmeasured
confounders assumption holds by adjusting for more covariates, which are often
available in large electronic health record datasets. There are always
challenges associated with observational data, including unverifiable
assumptions. We emphasize that any suggested treatment from a policy derived
with the proposed method should be reviewed by the medical care team. As
discussed in Weisenthal et al. (2023), the transparency of the relative
sparsity framework facilitates this type of review, and the methodology and
theory for inference provided in our work increases the rigor of the relative
sparsity framework.

##  11 Acknowledgements

The authors thank Jeremiah Jones, Ben Baer, Michael McDermott, Brent Johnson,
and Kah Poh Loh for helpful discussions. This research, which is the sole
responsibility of the authors and not the National Institutes of Health (NIH),
was supported by the National Institute of Environmental Health Sciences
(NIEHS) and the National Institute of General Medical Sciences (NIGMS) under
T32ES007271 and T32GM007356.

##  12 Conflict of interest

Authors state no conflict of interest.

## References

  * Achiam et al. [2017] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel.  Constrained policy optimization.  In _International Conference on Machine Learning_ , pages 22–31. PMLR, 2017.
  * Bellman [1957] Richard Bellman.  A markovian decision process.  _Journal of mathematics and mechanics_ , pages 679–684, 1957.
  * Casella and Berger [2002] George Casella and Roger L Berger.  _Statistical inference_.  Cengage Learning, 2002.
  * Chakraborty and Moodie [2013] Bibhas Chakraborty and Erica E. M. Moodie.  _Statistical Reinforcement Learning_ , pages 31–52.  Springer New York, New York, NY, 2013.  ISBN 978-1-4614-7428-9.  doi: 10.1007/978-1-4614-7428-9˙3.  URL https://doi.org/10.1007/978-1-4614-7428-9_3.
  * Cox [1975] David R Cox.  A note on data-splitting for the evaluation of significance levels.  _Biometrika_ , 62(2):441–444, 1975.
  * Dayan and Hinton [1997] Peter Dayan and Geoffrey E Hinton.  Using expectation-maximization for reinforcement learning.  _Neural Computation_ , 9(2):271–278, 1997.
  * Der-Nigoghossian et al. [2020] Caroline Der-Nigoghossian, Drayton A Hammond, and Mahmoud A Ammar.  Narrative review of controversies involving vasopressin use in septic shock and practical considerations.  _Annals of Pharmacotherapy_ , 54(7):706–714, 2020\.
  * Du et al. [2019] Mengnan Du, Ninghao Liu, and Xia Hu.  Techniques for interpretable machine learning.  _Communications of the ACM_ , 63(1):68–77, 2019\.
  * Eltzner [2020] Benjamin Eltzner.  Testing for uniqueness of estimators.  _arXiv preprint arXiv:2011.14762_ , 2020.
  * Ertefaie and Strawderman [2018] Ashkan Ertefaie and Robert L Strawderman.  Constructing dynamic treatment regimes over indefinite time horizons.  _Biometrika_ , 105(4):963–977, 2018.
  * Farahmand et al. [2017] Amir-massoud Farahmand, Andre Barreto, and Daniel Nikovski.  Value-aware loss function for model-based reinforcement learning.  In _Artificial Intelligence and Statistics_ , pages 1486–1494. PMLR, 2017.
  * Fujimoto et al. [2019] Scott Fujimoto, David Meger, and Doina Precup.  Off-policy deep reinforcement learning without exploration.  In _International Conference on Machine Learning_ , pages 2052–2062. PMLR, 2019.
  * Futoma et al. [2015] Joseph Futoma, Jonathan Morris, and Joseph Lucas.  A comparison of models for predicting early hospital readmissions.  _Journal of biomedical informatics_ , 56:229–238, 2015\.
  * Futoma et al. [2020] Joseph Futoma, Michael C Hughes, and Finale Doshi-Velez.  Popcorn: Partially observed prediction constrained reinforcement learning.  _arXiv preprint arXiv:2001.04032_ , 2020.
  * Geist et al. [2019] Matthieu Geist, Bruno Scherrer, and Olivier Pietquin.  A theory of regularized markov decision processes.  In _International Conference on Machine Learning_ , pages 2160–2169. PMLR, 2019.
  * Geyer [1994] Charles J Geyer.  On the asymptotics of constrained m-estimation.  _The Annals of statistics_ , pages 1993–2010, 1994.
  * Goldberger et al. [2000] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley.  Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals.  _circulation_ , 101(23):e215–e220, 2000.
  * Gottesman et al. [2020] Omer Gottesman, Joseph Futoma, Yao Liu, Sonali Parbhoo, Leo Celi, Emma Brunskill, and Finale Doshi-Velez.  Interpretable off-policy evaluation in reinforcement learning by highlighting influential transitions.  In _International Conference on Machine Learning_ , pages 3658–3667. PMLR, 2020.
  * Gupta et al. [2017] Divya M Gupta, Richard J Boland, and David C Aron.  The physician’s experience of changing clinical practice: a struggle to unlearn.  _Implementation Science_ , 12:1–11, 2017.
  * Haarnoja et al. [2017] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine.  Reinforcement learning with deep energy-based policies.  In _International Conference on Machine Learning_ , pages 1352–1361. PMLR, 2017.
  * Hoffman et al. [2011] Matthew W Hoffman, Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos.  Regularized least squares temporal difference learning with nested l 2 and l 1 penalization.  In _European Workshop on Reinforcement Learning_ , pages 102–114. Springer, 2011.
  * Horvitz and Thompson [1952] Daniel G Horvitz and Donovan J Thompson.  A generalization of sampling without replacement from a finite universe.  _Journal of the American statistical Association_ , 47(260):663–685, 1952.
  * Johnson et al. [2016a] Alistair Johnson, Tom Pollard, and R Mark III.  Mimic-iii clinical database.  _Physio Net_ , 10:C2XW26, 2016a.
  * Johnson et al. [2016b] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9, 2016b.
  * Johnson et al. [2016c] Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9, 2016c.
  * Kallus and Uehara [2020] Nathan Kallus and Masatoshi Uehara.  Efficient evaluation of natural stochastic policies in offline reinforcement learning.  _arXiv preprint arXiv:2006.03886_ , 2020.
  * Kennedy [2019] Edward H Kennedy.  Nonparametric causal effects based on incremental propensity score interventions.  _Journal of the American Statistical Association_ , 114(526):645–656, 2019.
  * Kloek and Van Dijk [1978] Teun Kloek and Herman K Van Dijk.  Bayesian estimates of equation system parameters: an application of integration by monte carlo.  _Econometrica: Journal of the Econometric Society_ , pages 1–19, 1978\.
  * Knight and Fu [2000] Keith Knight and Wenjiang Fu.  Asymptotics for lasso-type estimators.  _Annals of statistics_ , pages 1356–1378, 2000.
  * Komorowski et al. [2018] Matthieu Komorowski, Leo A Celi, Omar Badawi, Anthony C Gordon, and A Aldo Faisal.  The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care.  _Nature medicine_ , 24(11):1716–1720, 2018.
  * Kuchibhotla et al. [2022] Arun K Kuchibhotla, John E Kolassa, and Todd A Kuffner.  Post-selection inference.  _Annual Review of Statistics and Its Application_ , 9:505–527, 2022.
  * Kullback and Leibler [1951] Solomon Kullback and Richard A Leibler.  On information and sufficiency.  _The annals of mathematical statistics_ , 22(1):79–86, 1951.
  * Le et al. [2016] Hoang Le, Andrew Kang, Yisong Yue, and Peter Carr.  Smooth imitation learning for online sequence prediction.  In _International Conference on Machine Learning_ , pages 680–688. PMLR, 2016.
  * Le et al. [2019] Hoang Le, Cameron Voloshin, and Yisong Yue.  Batch policy learning under constraints.  In _International Conference on Machine Learning_ , pages 3703–3712. PMLR, 2019.
  * Leamer [1974] Edward E Leamer.  False models and post-data model construction.  _Journal of the American Statistical Association_ , 69(345):122–131, 1974.
  * Lee et al. [2012] Joon Lee, Rishi Kothari, Joseph A Ladapo, Daniel J Scott, and Leo A Celi.  Interrogating a clinical database to study treatment of hypotension in the critically ill.  _BMJ open_ , 2(3):e000916, 2012.
  * Lei et al. [2017] Huitian Lei, Yangyi Lu, Ambuj Tewari, and Susan A Murphy.  An actor-critic contextual bandit algorithm for personalized mobile health interventions.  _arXiv preprint arXiv:1706.09090_ , 2017.
  * Levine and Abbeel [2014] Sergey Levine and Pieter Abbeel.  Learning neural network policies with guided policy search under unknown dynamics.  In _NIPS_ , volume 27, pages 1071–1079. Citeseer, 2014.
  * Lipton [2018] Zachary C Lipton.  The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.  _Queue_ , 16(3):31–57, 2018.
  * Lipton et al. [2015] Zachary C Lipton, David C Kale, Charles Elkan, and Randall Wetzel.  Learning to diagnose with lstm recurrent neural networks.  _arXiv preprint arXiv:1511.03677_ , 2015.
  * Loh and Wainwright [2013] Po-Ling Loh and Martin J Wainwright.  Regularized m-estimators with nonconvexity: Statistical and algorithmic theory for local optima.  _Advances in Neural Information Processing Systems_ , 26, 2013.
  * Luckett et al. [2019] Daniel J Luckett, Eric B Laber, Anna R Kahkoska, David M Maahs, Elizabeth Mayer-Davis, and Michael R Kosorok.  Estimating dynamic treatment regimes in mobile health using v-learning.  _Journal of the American Statistical Association_ , 2019.
  * Miller [2019] Tim Miller.  Explanation in artificial intelligence: Insights from the social sciences.  _Artificial intelligence_ , 267:1–38, 2019.
  * Muñoz and van der Laan [2012] Iván Díaz Muñoz and Mark van der Laan.  Population intervention causal effects based on stochastic interventions.  _Biometrics_ , 68(2):541–549, 2012.
  * Niculescu-Mizil and Caruana [2005] Alexandru Niculescu-Mizil and Rich Caruana.  Predicting good probabilities with supervised learning.  In _Proceedings of the 22nd international conference on Machine learning_ , pages 625–632, 2005.
  * Owen [2013] Art B. Owen.  _Monte Carlo theory, methods and examples_.  2013\.
  * Parbhoo et al. [2022] Sonali Parbhoo, Shalmali Joshi, and Finale Doshi-Velez.  Generalizing off-policy evaluation from a causal perspective for sequential decision-making.  _arXiv preprint arXiv:2201.08262_ , 2022.
  * Pearl [2009] Judea Pearl.  _Causality_.  Cambridge university press, 2009.
  * Peters et al. [2010] Jan Peters, Katharina Mulling, and Yasemin Altun.  Relative entropy policy search.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 24, 2010.
  * Pötscher and Schneider [2009] Benedikt M Pötscher and Ulrike Schneider.  On the distribution of the adaptive lasso estimator.  _Journal of Statistical Planning and Inference_ , 139(8):2775–2790, 2009.
  * Precup [2000] Doina Precup.  Eligibility traces for off-policy policy evaluation.  _Computer Science Department Faculty Publication Series_ , page 80, 2000.
  * Puterman [2014] Martin L Puterman.  _Markov decision processes: discrete stochastic dynamic programming_.  John Wiley & Sons, 2014.
  * Robins et al. [1994] James M Robins, Andrea Rotnitzky, and Lue Ping Zhao.  Estimation of regression coefficients when some regressors are not always observed.  _Journal of the American statistical Association_ , 89(427):846–866, 1994.
  * Rudin [2019] Cynthia Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1(5):206–215, 2019.
  * Russell et al. [2021] James A Russell, Anthony C Gordon, Mark D Williams, John H Boyd, Keith R Walley, and Niranjan Kissoon.  Vasopressor therapy in the intensive care unit.  In _Seminars in Respiratory and Critical Care Medicine_ , volume 42, pages 059–077. Thieme Medical Publishers, Inc., 2021.
  * Schulman et al. [2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz.  Trust region policy optimization.  In _International conference on machine learning_ , pages 1889–1897. PMLR, 2015.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Sutton and Barto [2018] Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  2018\.
  * Thomas [2015] Philip S Thomas.  _Safe reinforcement learning_.  PhD thesis, 2015.
  * Uehara et al. [2022] Masatoshi Uehara, Chengchun Shi, and Nathan Kallus.  A review of off-policy evaluation in reinforcement learning.  _arXiv preprint arXiv:2212.06355_ , 2022.
  * Ueno et al. [2012] Tsuyoshi Ueno, Kohei Hayashi, Takashi Washio, and Yoshinobu Kawahara.  Weighted likelihood policy search with model selection.  _Advances in Neural Information Processing Systems_ , 25:2357–2365, 2012.
  * Van Calster et al. [2016] Ben Van Calster, Daan Nieboer, Yvonne Vergouwe, Bavo De Cock, Michael J Pencina, and Ewout W Steyerberg.  A calibration hierarchy for risk models was defined: from utopia to empirical data.  _Journal of clinical epidemiology_ , 74:167–176, 2016.
  * van der Laan and Rose [2018] Mark J van der Laan and Sherri Rose.  _Targeted learning in data science: causal inference for complex longitudinal studies_.  Springer, 2018.
  * van der Vaart [2000] Aad W van der Vaart.  _Asymptotic statistics_ , volume 3.  Cambridge university press, 2000.
  * Weisenthal et al. [2018] Samuel J Weisenthal, Caroline Quill, Samir Farooq, Henry Kautz, and Martin S Zand.  Predicting acute kidney injury at hospital re-entry using high-dimensional electronic health record data.  _PloS one_ , 13(11):e0204920, 2018.
  * Weisenthal et al. [2023] Samuel J. Weisenthal, Sally W. Thurston, and Ashkan Ertefaie.  Relative sparsity for medical decision problems.  _Statistics in Medicine_ , 2023.
  * Wooldridge [2010] Jeffrey M Wooldridge.  _Econometric analysis of cross section and panel data_.  MIT press, 2010.
  * Wu et al. [2019] Yifan Wu, George Tucker, and Ofir Nachum.  Behavior regularized offline reinforcement learning.  _arXiv preprint arXiv:1911.11361_ , 2019.
  * Ziebart et al. [2008] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey.  Maximum entropy inverse reinforcement learning.  In _Aaai_ , volume 8, pages 1433–1438. Chicago, IL, USA, 2008.
  * Zou [2006] Hui Zou.  The adaptive lasso and its oracle properties.  _Journal of the American statistical association_ , 101(476):1418–1429, 2006.

##  Appendix A Appendix

###  A.1 Research code

Research code can be found at
https://github.com/samuelweisenthal/inference_for_relative_sparsity.

###  A.2 Data availability statement

The MIMIC [Johnson et al., 2016c] dataset that supports the findings of this
study is openly available at PhysioNet (doi: 10.13026/C2XW26) and can be found
online at https://physionet.org/content/mimiciii/1.4/.

###  A.3 Determinism of the reward-maximizing policy

######  Lemma 4.

Assuming (1), which defines the policy as an $\text{expit},$ and defining the
true value maximizing policy $\pi_{\beta_{0}}=\arg\max_{\pi}V_{0},$ let
$R_{m}=R(S_{m},A_{m},S_{m+1})$ we have that

$\displaystyle\pi_{\beta_{0}}(A_{t}=1|s_{t})$
$\displaystyle=I\left(E\left(\sum_{m=t}^{T}R_{m}|S_{t},A_{t}=1\right)-E\left(\sum_{m=t}^{T}R_{m}|S_{t},A_{t}=0\right)>0\right).$

###### Proof.

We will show that the policy that informs the decision $\pi(A_{t}=1|S_{t})$ is
equal to one or zero (i.e., it is deterministic). This has been shown in e.g.
Lei et al. [2017], Puterman [2014], but we specialize the proof to the binary
action, continuous state setting. Consider for now an arbitrary policy $\pi$,
which is not parameterized. We will show that the arbitrary policy must be
deterministic, and then we will argue that in order for the model that we use
for the policy in this work, defined in (1), to approximate this deterministic
policy, at least some of the policy coefficients must approach infinity in
magnitude.

We start by showing that, to maximize value, an arbitrary policy $\pi$ must be
deterministic. Let us consider the value starting at time $t$ and going
forward,

$\displaystyle V_{0,t:T}$
$\displaystyle=E\left(\sum_{m=t}^{T}R(S_{m},A_{m},S_{m+1})\right)$
$\displaystyle=E\left(E\left(\sum_{m=t}^{T}R(S_{m},A_{m},S_{m+1})|S_{t},A_{t}\right)\right)$
$\displaystyle=\int_{s_{t}}\sum_{a_{t}}E\left(\sum_{m=t}^{T}R(S_{m},A_{m},S_{m+1})|s_{t},A_{t}\right)\pi(a_{t}|s_{t})p(s_{t})ds_{t}$
$\displaystyle=\int_{s_{t}}E\left(\sum_{m=t}^{T}R_{m}|s_{t},A_{t}=1\right)\pi(A_{t}=1|s_{t})p(s_{t})ds_{t}$
$\displaystyle+E\left(\sum_{m=t}^{T}R_{m}|s_{t},A_{t}=0\right)\pi(A_{t}=0|s_{t})p(s_{t})ds_{t}.$

Where we have just written out the expectation. Now, the last expression is
equal to

$\displaystyle\int_{s_{t}}E\left(\sum_{m=t}^{T}R_{m}|s_{t},A_{t}=1\right)\pi(A_{t}=1|s_{t})p(s_{t})ds_{t}$
$\displaystyle+E\left(\sum_{m=t}^{T}R_{m}|s_{t},A_{t}=0\right)(1-\pi(A_{t}=1|s_{t}))p(s_{t})ds_{t}$
$\displaystyle=\int_{s_{t}}E\left(\sum_{m=t}^{T}R_{m}|s_{t},A_{t}=1\right)\pi(A_{t}=1|s_{t})p(s_{t})ds_{t}$
$\displaystyle+E\left(\sum_{m=t}^{T}R_{m}|s_{t},A_{t}=0\right)p(s_{t})ds_{t}$
$\displaystyle-E\left(\sum_{m=t}^{T}R_{m}|s_{t},A_{t}=0\right)\pi(A_{t}=1|s_{t})p(s_{t})ds_{t}.$

Where we have used the properties of model (1). This is further equal to

$\displaystyle\int_{s_{t}}\left(E\left(\sum_{m=t}^{T}R_{m}|s_{t},A_{t}=1\right)-E\left(\sum_{m=t}^{T}R_{m}|s_{t},A_{t}=0\right)\right)\pi(A_{t}=1|s_{t})p(s_{t})ds_{t}$
$\displaystyle+\int_{s_{t}}E\left(\sum_{m=t}^{T}R_{m}|s_{t},A_{t}=0\right)p(s_{t})ds_{t}$
$\displaystyle=\int_{s_{t}}\left(E\left(\sum_{m=t}^{T}R_{m}|S_{t},A_{t}=1\right)-E\left(\sum_{m=t}^{T}R_{m}|S_{t},A_{t}=0\right)\right)\pi(A_{t}=1|s_{t})p(s_{t})ds_{t}$
$\displaystyle\ \ +C$

where $C$ does not depend on $\pi(A_{t}=1|s_{t}).$ This implies that, since
$0\leq\pi_{\beta_{0}}(A_{t}=1|s_{t})\leq 1$, the maximizing policy is

$\displaystyle\pi_{\beta_{0}}(A_{t}=1|s_{t})=I\left(E\left(\sum_{m=t}^{T}R_{m}|S_{t},A_{t}=1\right)-E\left(\sum_{m=t}^{T}R_{m}|S_{t},A_{t}=0\right)>0\right).$
(28)

Note that the expected return at time $t$ depends on the actions taken from
time $t$ to $T,$ and so if the policy is stationary (Assumption 2), this
action will depend on the policy itself. We therefore start at time $T$ and go
backwards, applying (28) at each time step, so that the policy at time $t$ is
an indicator, and depends on a series of indicators that determine actions
taken up until time $T$. Hence, at any time step, the optimal policy is
deterministic.

We will now show that this determinism has implications for the magnitude of
the policy coefficients (to approximate an indicator function, an expit must
have arbitrarily large input, which can only occur if the coefficients of the
policy are arbitrarily large). Note that

$\pi_{\beta_{0}}(A_{t}=1|s_{t})=\text{expit}(\beta_{0}^{T}s_{t})$

and

$\text{expit}(\beta_{0}^{T}s_{t})=1\iff\exp(\beta_{0}^{T}s_{t})=1+\exp(\beta_{0}^{T}s_{t}),$

which is satisfied only if $\exp(\beta_{0}^{T}s_{t})=\infty,$ which is
satisfied if and only if, because $s_{t}$ is bounded, some entry of
$\beta_{0}$ is infinite (or, in practice, becomes arbitrarily large). ∎

###  A.4 The expected value of the importance sampling ratio

For intuition, we will show first in the single stage case

$Er=E_{b}\frac{\pi_{\beta}}{\pi_{b}}=\int_{s}\sum_{a}\frac{\pi_{\beta}(a|s)}{\pi_{b}(a|s)}\pi_{b}(a|s)p(s)ds=\int_{s}\sum_{a}\pi_{\beta}(a|s)p(s)ds=1.$

Now for the multi-stage case,

$\displaystyle Er$
$\displaystyle=E_{b}\frac{\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}$
$\displaystyle=\int_{s_{0}}\sum_{a_{0}}\dots\int_{s_{T}}\sum_{a_{T}}\frac{\pi_{\beta}(a_{0}|s_{0})\dots\pi_{\beta}(a_{T}|s_{t})}{\pi_{b}(a_{0}|s_{0})\dots\pi_{b}(a_{T}|s_{t})}\pi_{b}(a_{0}|s_{0})$
$\displaystyle\ \ \ \times p(s_{0})p(s_{1}|a_{0},s_{0})\dots
P_{t}(s_{t}|a_{t-1},s_{t-1},\dots,a_{0},s_{0})\pi_{b}(a_{T}|s_{t})ds_{0}\dots
ds_{T}$
$\displaystyle=\int_{s_{0}}\sum_{a_{0}}\dots\int_{s_{T}}\sum_{a_{T}}\pi_{\beta}(a_{0}|s_{0})p(s_{0})p(s_{1}|a_{0},s_{0})\dots\pi_{\beta}(a_{T}|s_{t})$
$\displaystyle\ \ \ \times
P_{t}(s_{t}|a_{t-1},s_{t-1},\dots,a_{0},s_{0})ds_{0}\dots ds_{T}$
$\displaystyle=1.$

###  A.5 On the adaptive lasso tuning parameter

In Zou [2006], the authors try $\delta=\\{.5,1,2\\}.$ Recall that the weight
of the penalty term is $w_{n}=1/|\beta_{n,\gamma}-b_{n}|^{\delta}.$ If
$\delta=1,$ then the weight for coefficient $k$ is
$|(\beta_{n,\gamma,k})-(b_{n,k})|,$ and coefficients that are truly non-
behavioral, $|(\beta_{0,\gamma,k})-(b_{0,k})|>0$, will have a smaller penalty
while the coefficients that are truly behavioral,
$|(\beta_{0,\gamma,k})-(b_{0,k})|=0$, will have a larger penalty (Zou [2006]
notes that this is related to Breiman’s non-negative Garotte). If $\delta>1,$
if $|(\beta_{n,\gamma,k})-(b_{n,k})|>1,$ then the penalty weight will the
inverse of the power of a number greater than one, and it will therefore be
small. If $|(\beta_{n,\gamma,k})-(b_{n,k})|<1,$ the penalty weight will be the
inverse of a power of a number less than one, and it will therefore be large.
Hence, setting $\delta>1$ should make the penalty more “adaptive,” and, as
$\delta\rightarrow\infty,$ the adaptivity should increase. If $\delta<1,$ and
$|(\beta_{0,\gamma,k})-(b_{0,k})|>1,$ then the denominator of the penalty will
be less large, and the penalty will be larger than it would be if $\delta=1$;
in the reverse case, if $|(\beta_{0,\gamma,k})-(b_{0,k})|<1,$ the denominator
will be larger, and the penalty will be smaller than it would be if
$\delta=1.$ Hence, the penalty will be tempered for all coefficients (the
adaptive penalty becomes less adaptive). As $\delta\rightarrow 0$ we approach
$|(\beta_{n,\gamma,k})-(b_{n,k})|^{0}=1,$ in which case the penalty weight
does not depend on the coefficients at all (it is completely non-adaptive),
and we return to a standard relative Lasso.

###  A.6 Scaling of the state covariates

We scale independently of time, because the policy coefficients are constant
over time by stationarity (Assumption 2). Concretely, for $i=1,\dots,n,\
t=1,\dots,T,\ k=1\dots,K,$ if $\sigma_{n}(S_{k})$ is an estimate for the
standard deviation of the dimension $k$ of the state, we scale $S_{itk}$ as
$\tilde{S}_{itk}={S_{itk}}/{\sigma_{n}(S_{k})}.$ Since we will be penalizing
the coefficients of the suggested policy to the behavioral policy
coefficients, and the $K$ covariates that comprise the state
$S_{t}=(S_{t,1},\dots,S_{t,K})^{T}$ must be scaled to put the coefficients on
a level playing field for penalization, we also scale before estimating the
behavioral policy $b_{n}$. Scaling is also discussed in Weisenthal et al.
[2023], Hoffman et al. [2011].

###  A.7 Additional notation

We first briefly introduce some notation that was not necessary in the
methodology section, but allows for more compact proofs here. Recall that $E$
and $E_{n}$ are the expectation and the empirical expectation operator,
respectively. Define the log of the data distribution, which can be written by
Assumptions 2 and 1 as

$l(b)=\log
P_{0}(S_{0})\pi_{b}(A_{0}|S_{0})\prod_{t=1}^{T}\pi_{b}(A_{t}|S_{t})P_{0,t}(S_{t}|A_{t-1},S_{t-1},\dots,A_{0},S_{0}).$
(29)

Note that the behavioral policy parameter estimator is

$b_{n}=\arg\max_{b}nE_{n}l(b).$ (30)

Define

$r={\prod_{t=0}^{T}\pi_{\beta}(A_{t}|S_{t})}\biggr{/}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})}.$
(31)

Note that we use lowercase $r$ to refer to the ratio here, which should not be
confused with capital $R,$ which refers to the reward in Equation (2). Note
that for $V_{0}$ defined in (3) and rewritten in (11), $V_{0}(\beta)=Ev,$
where

$\displaystyle v=r\sum_{t=0}^{T}R(S_{t},A_{t},S_{t+1}).$ (32)

Although $V_{0}$ as written in (11) depends on $b,$ we suppress this
dependence on $b,$ because the policies involving $b$ cancel. Then
${V}_{n}(\beta,b)={E_{n}v(\beta,b)}/{E_{n}r(\beta,b)}.$ Define the standard,
unweighted importance sampling estimator to be

$\displaystyle\tilde{V}_{n}(\beta,b)=E_{n}v(\beta,b).$ (33)

Note that for $KL_{0}$ defined in (6), $KL_{0}({\beta},b)=Ekl,$ where

$kl=\log\left(1/r\right).$ (34)

Note that for $KL_{n}$ defined in (13), $KL_{n}(\beta,b)=E_{n}kl$. Note that
if we define

$m=v-\gamma kl,$ (35)

then $M_{0}(\beta,b)=Em.$ Note that for ${M}_{n}$ defined in (14),
${M}_{n}(\beta,b_{n},\gamma)={V}_{n}(\beta,b_{n})-\gamma
KL_{n}(\beta,b_{n})={E_{n}v}/{E_{n}r}-\gamma
E_{n}kl=E_{n}\left({v}/{E_{n}r}-\gamma kl\right).$

###  A.8 Estimating the behavioral policy

Recall that $E_{n}$ is the empirical expectation operator, where
$E_{n}f(X)=\frac{1}{n}\sum_{i=1}^{n}f(X_{i})$. Recall the sampling
distribution defined in (29), which was

$l(b)=\log
P_{0}(S_{0})\pi_{b}(A_{0}|S_{0})\prod_{t=1}^{T}\pi_{b}(A_{t}|S_{t})P_{0,t}(S_{t}|A_{t-1},S_{t-1},\dots,A_{0},S_{0}).$

Define the log likelihood then as

$nE_{n}l=\sum_{i=1}^{n}\log P_{0,b}(a_{i,0:T},s_{i,0:T+1}).$ (36)

If we consider the maximizer,

$b_{n}=\arg\max_{b}nE_{n}l(b),$ (37)

then $b_{n}$ is an estimator for $b_{0},$ the true behavioral policy
parameter, and $\pi_{b_{n}}$ is an estimator for the data-generating
“behavioral” policy, $\pi_{b_{0}}$, where the form of $\pi$ is an
$\text{expit},$ as given in (1).

###  A.9 Estimating value with importance sampling

Define

$\left\\{A_{i,0:T},S_{i,0:T+1}\right\\}=\
\\{A_{i,0},\dots,A_{i,T},S_{i,0},\dots,S_{i,T+1}\\}.$

Define the likelihood of a trajectory under $P$ and the behavioral policy
$\pi$ indexed by an arbitrary parameter $b$ is

$P_{b}(A_{0:T},S_{0:T+1})=P_{0}(S_{0})\pi_{b}(A_{0}|S_{0})P_{b}(A_{1:T},S_{1:T})P_{0,T+1}(S_{T+1}|A_{T},S_{T},\dots,A_{0},S_{0}),$

where

$P_{b}(A_{1:T},S_{1:T})=\prod_{t=1}^{T}\pi_{b}(A_{t}|S_{t})P_{0,t}(S_{t}|A_{t-1},S_{t-1},\dots,A_{0},S_{0}).$

We obtain the potential value under policy $\pi_{\beta}$ by noting that

$\displaystyle V_{0}(\beta)$
$\displaystyle=E_{\beta}\left[\sum_{t=0}^{T}R(S_{t},A_{t},S_{t+1})\right]$
$\displaystyle=E_{b}\left\\{\rho(A_{0:T},S_{0:T+1};\beta,b)\sum_{t=0}^{T}R(S_{t},A_{t},S_{t+1})\right\\},$

where

$\displaystyle\rho(A_{0:T},S_{0:T+1};\beta,b)$
$\displaystyle=\frac{P_{\beta}(A_{0:T},S_{0:T+1})}{P_{b}(A_{0:T},S_{0:T+1})}$
$\displaystyle=\frac{P(S_{0})\pi_{\beta}(A_{0}|S_{0})P_{\beta}(A_{1:T},S_{1:T})P_{T+1}(S_{T+1}|A_{T},S_{T},\dots,A_{0},S_{0})}{P(S_{0})\pi_{b}(A_{0}|S_{0})P_{b}(A_{1:T},S_{1:T})P_{T+1}(S_{T+1}|A_{T},S_{T},\dots,A_{0},S_{0})}$
$\displaystyle=\frac{\pi_{\beta}(A_{0}|S_{0})P_{\beta}(A_{1:T},S_{1:T})}{\pi_{b}(A_{0}|S_{0})P_{b}(A_{1:T},S_{1:T})}$
$\displaystyle=\frac{\pi_{\beta}(A_{0}|S_{0})\prod_{t=1}^{T}\pi_{\beta}(A_{t}|S_{t})P_{t}(S_{t}|A_{t-1},S_{t-1},\dots,A_{0},S_{0})}{\pi_{b}(A_{0}|S_{0})\prod_{t=1}^{T}\pi_{b}(A_{t}|S_{t})P_{t}(S_{t}|A_{t-1},S_{t-1},\dots,A_{0},S_{0})}$
$\displaystyle=\frac{\pi_{\beta}(A_{0}|S_{0})\prod_{t=1}^{T}\pi_{\beta}(A_{t}|S_{t})}{\pi_{b}(A_{0}|S_{0})\prod_{t=1}^{T}\pi_{b}(A_{t}|S_{t})}$
$\displaystyle=\prod_{t=0}^{T}\frac{\pi_{\beta}(A_{t}|S_{t})}{\pi_{b}(A_{t}|S_{t})}.$

###  A.10 Proof of Lemma 1

We can by Assumption 9 on consistency of the nuisance, perform a Taylor
expansion around $b_{0},$ so that

$\displaystyle{{M}_{n}}(\beta,b_{n})$
$\displaystyle\approx{{M}_{n}}(\beta,b_{0})+(b_{n}-b_{0})^{T}\frac{\partial}{\partial
b}{M}_{n}(\beta_{n,\gamma},b_{0})$
$\displaystyle={{M}_{n}}(\beta,b_{0})+o_{P}(1)\frac{\partial}{\partial
b}{M}_{n}(\beta_{n,\gamma},b_{0})$
$\displaystyle={{M}_{n}}(\beta,b_{0})+o_{P}(1).$

where the last equality follows due to boundedness of the derivative by
Assumption 7 and the Central Limit Theorem, since the sample derivative is an
average. Hence, we only consider the convergence of ${M}_{n}(\beta,b_{0}).$
Note that ${M}_{n}={V}_{n}-\gamma KL_{n}.$ We have that ${V}_{n}$ converges to
$V_{0}$ by Remark 5, and $KL_{n}\overset{p}{\to}KL_{0}$ by the Law of Large
Numbers.

###  A.11 Proof of Lemma 2

Let us show that $\frac{\partial}{\partial\beta}E_{n}r\overset{p}{\to}0.$ By
Assumption 7, if we just set $\gamma=0$ and $R=1$, we have that
$|\frac{\partial}{\partial\beta}r|<C,$ for some constant $C$, which allows us
to apply Lebesgue’s Dominated Convergence Theorem. Then

$\displaystyle E_{n}\frac{\partial}{\partial\beta}r$
$\displaystyle\overset{p}{\to}E\frac{\partial}{\partial\beta}r,\text{\ \ \ \ \
\ \ \ \ \ (Law of Large Numbers)}$
$\displaystyle=\frac{\partial}{\partial\beta}E{r}\text{ \ \ \ \ \ \ \ \ \ \ \
(Dominated Convergence Theorem) }$
$\displaystyle=\frac{\partial}{\partial\beta}1\text{\ \ \ \ \ \ \ \ \ \ \ \ \
\ \ (Remark \ref{rem:Er1})}$ $\displaystyle=0.$

Identical arguments can be made to show that $E_{n}\frac{\partial}{\partial
b}r\overset{p}{\to}0$ and $E_{n}\frac{\partial}{\partial
b}\frac{\partial}{\partial\beta}r\overset{p}{\to}0.$

###  A.12 Proof of Lemma 3

We first show that
$J_{n}=\frac{\partial}{\partial\beta}{M}_{n}\overset{p}{\to}\frac{\partial}{\partial\beta}M_{0}=J_{0}.$

Note that if we write the arguments explicitly, we have
$\frac{\partial}{\partial\beta}{M}_{n}(\beta,b_{n}).$ We can, however, by
Assumption 9 on consistency of the nuisance, perform a Taylor expansion around
$b_{0},$ so that

$\displaystyle\frac{\partial}{\partial\beta}{M}_{n}(\beta,b_{n})$
$\displaystyle\approx\frac{\partial}{\partial\beta}{M}_{n}(\beta,b_{0})+(b_{n}-b_{0})^{T}\frac{\partial}{\partial
b}\frac{\partial}{\partial\beta}{M}_{n}(\beta_{n,\gamma},b_{0})$
$\displaystyle=\frac{\partial}{\partial\beta}{M}_{n}(\beta,b_{0})+o_{P}(1)\frac{\partial}{\partial
b}\frac{\partial}{\partial\beta}{M}_{n}(\beta,b_{0})$
$\displaystyle=\frac{\partial}{\partial\beta}{M}_{n}(\beta,b_{0})+o_{P}(1),$

where the last equality follows due to boundedness of the cross derivative by
Assumption 7 and the Central Limit Theorem, since the sample cross derivative
is an average. Hence, when we write $\frac{\partial}{\partial\beta}{M}_{n}$
below, we implicitly take this Taylor expansion and only focus on showing
convergence of the term $\frac{\partial}{\partial\beta}{M}_{n}(\beta,b_{0}),$
and we will do the same for the arguments concerning $H_{n}$ and $X_{n}$ that
follow.

Now we will show that $\frac{\partial}{\partial\beta}{M}_{n}$ converges to the
desired expectation. Recall that $z$ is defined in (23). Note

$\displaystyle
J_{n}=\frac{\partial}{\partial\beta}{M}_{n}=E_{n}z=E_{n}\left(\frac{\frac{\partial}{\partial\beta}v}{E_{n}r}-\frac{v(E_{n}\frac{\partial}{\partial\beta}r)}{(E_{n}r)^{2}}-\frac{\partial}{\partial\beta}\gamma
kl(\beta,b)\right).$ (38)

Define also

$J_{0}=\frac{\partial}{\partial\beta}M_{0}=E\left(\frac{\partial}{\partial\beta}v-\frac{\partial}{\partial\beta}\gamma
kl(\beta,b)\right).$

We need to therefore show that $J_{n}\overset{p}{\to}J_{0}.$ By Remark 4, we
have that $E_{n}r\overset{p}{\to}Er=1,$ and by Lemma 2 of Appendix A.11,
$E_{n}\frac{\partial}{\partial\beta}r\overset{p}{\to}0.$ Hence, we can
conclude, based on Slutsky’s theorem, that $J_{n}\overset{p}{\to}J_{0}.$

Let us now consider the Hessian, $H_{n}.$ Taking derivatives (the building
blocks of which are shown in Appendix A.18), we have that

$\displaystyle H_{n}$
$\displaystyle=\frac{E_{n}\frac{\partial^{2}}{\partial\beta^{2}}v}{E_{n}r}-\frac{E_{n}\frac{\partial}{\partial\beta}v}{E_{n}r}\frac{(E_{n}\frac{\partial}{\partial\beta}r)^{T}}{E_{n}r}-\frac{E_{n}\frac{\partial}{\partial\beta}r}{E_{n}r}\frac{(E_{n}\frac{\partial}{\partial\beta}v)^{T}}{E_{n}r}$
$\displaystyle-\frac{E_{n}v}{E_{n}r}\frac{E_{n}\frac{\partial^{2}}{\partial\beta^{2}}r}{E_{n}r}+2\frac{E_{n}v}{E_{n}r}\frac{E_{n}\frac{\partial}{\partial\beta}r}{E_{n}r}\frac{(E_{n}\frac{\partial}{\partial\beta}r)^{T}}{E_{n}r}-\frac{\partial^{2}}{\partial\beta^{2}}\gamma
KL_{n}(\beta,b)$ (39) $\displaystyle=(I)+(II)+(III)+(IV)+(V)+(VI).$ (40)

Note that by Lemma 2 and Slutsky’s theorem, the terms $(II-V)$ converge in
probability to zero. The term $E_{n}r\overset{p}{\to}1$ by Remark 4. Note then
that by the Law of Large Numbers, $H_{n}\overset{p}{\to}H_{0}.$

We also have that, similarly,

$\displaystyle X_{n}$ $\displaystyle=\frac{E_{n}\frac{\partial}{\partial
b}\frac{\partial}{\partial\beta}v}{E_{n}r}-\frac{E_{n}\frac{\partial}{\partial\beta}v}{E_{n}r}\frac{(E_{n}\frac{\partial}{\partial
b}r)^{T}}{E_{n}r}-\frac{E_{n}\frac{\partial}{\partial\beta}r}{E_{n}r}\frac{(E_{n}\frac{\partial}{\partial
b}v)^{T}}{E_{n}r}$
$\displaystyle-\frac{E_{n}v}{E_{n}r}\frac{E_{n}\frac{\partial}{\partial
b}\frac{\partial}{\partial\beta}r}{E_{n}r}+2\frac{E_{n}v}{E_{n}r}\frac{E_{n}\frac{\partial}{\partial\beta}r}{E_{n}r}\frac{(E_{n}\frac{\partial}{\partial
b}r)^{T}}{E_{n}r}-\frac{\partial}{\partial
b}\frac{\partial}{\partial\beta}\gamma KL_{n}(\beta,b)$ (41)
$\displaystyle=(I)+(II)+(III)+(IV)+(V)+(VI).$ (42)

Note that, as with $H_{n},$ by Lemma 2 and Slutsky’s theorem, the terms
$(II-V)$ converge in probability to zero. The term $E_{n}r\overset{p}{\to}1$
by Remark 4. Note then that by the law of Large Numbers and Slutsky’s theorem,
$X_{n}\overset{p}{\to}X_{0}.$

###  A.13 Proof of Theorem 1 (i)

To show consistency of $\beta_{n,\gamma},$ or that
$\beta_{n,\gamma}\overset{p}{\to}\beta_{0,\gamma}$, we can apply results from
Wooldridge [2010] on two-step M-estimators. By Wooldridge [2010], Section
12.4.1, to show consistency of
$\beta_{n,\gamma}=\arg\max_{\beta}{M}_{n}(\beta,b_{n})$ for
$\beta_{0,\gamma},$ we must first have that $b_{n}\overset{p}{\to}b_{0},$
which is true by Assumption 9. Given this, we then must show the following
conditions:

######  Condition 1.

We have that ${M}_{n}(\beta,b)$ satisfies the Uniform Weak Law of Large
Numbers over $B\times B^{\prime},$ where $\beta\in B$ and $b\in B^{\prime},$
or that $\sup_{\beta,b}|{M}_{n}(\beta,b)-M_{0}(\beta,b)|\overset{p}{\to}0,$

######  Condition 2.

We have that $M_{0}(\beta_{0,\gamma},b_{0})>M_{0}(\beta,b_{0})$ for all
$\beta\in B$ such that $\beta\neq\beta_{0,\gamma}$.

We will start by showing Condition 1. Note that $V_{n}$ and ${M}_{n}$ are the
weighted objectives in (12) and (14), and define $\tilde{V}_{n}=E_{n}v$ and
$\tilde{M}_{n}=E_{n}m$ as their unweighted counterparts, where $v$ and $m$ are
defined in (32) and (35). To show Condition 1, note that

$\displaystyle\sup_{\beta,b}|{M}_{n}(\beta,b)-M_{0}(\beta,b)|$
$\displaystyle=\sup_{\beta,b}|{V}_{n}(\beta,b)-\gamma KL_{0}-M_{0}(\beta,b)|$
$\displaystyle=\sup_{\beta,b}\left|{V}_{n}(\beta,b)-\tilde{V}_{n}(\beta,b)+\tilde{V}_{n}(\beta,b)-\gamma
KL_{0}-M_{0}(\beta,b)\right|$
$\displaystyle=\sup_{\beta,b}\left|\frac{E_{n}v}{E_{n}r}-\frac{E_{n}v}{Er}+\frac{E_{n}r}{Er}-\gamma
KL_{0}-M_{0}(\beta,b)\right|$
$\displaystyle=\sup_{\beta,b}\left|\frac{E_{n}r}{Er}-\frac{E_{n}v(E_{n}r-Er)}{E_{n}rEr}-\gamma
KL_{0}-M_{0}(\beta,b)\right|$
$\displaystyle\leq\sup_{\beta,b}\left|\frac{E_{n}r}{Er}-\gamma
KL_{0}-M_{0}(\beta,b)\right|+\sup_{\beta,b}\left|\frac{E_{n}v(E_{n}r-Er)}{E_{n}rEr}\right|$
$\displaystyle=\sup_{\beta,b}\left|\tilde{M}_{n}(\beta,b)-M_{0}(\beta,b)\right|+\sup_{\beta,b}\left|\frac{E_{n}v}{E_{n}r}(E_{n}r-Er)\right|$
$\displaystyle\leq\sup_{\beta,b}\left|\tilde{M}_{n}(\beta,b)-M_{0}(\beta,b)\right|+\sup_{\beta,b}\left|\frac{E_{n}v}{E_{n}r}\right|\sup_{\beta,b}\left|E_{n}r-Er\right|$
$\displaystyle=I+II.$

Where we have used that $E_{r}=1,$ which was shown in Remark 4. Theorem 12.1
in Wooldridge [2010] states that if $\beta\in B$ and $b\in B^{\prime}$ for
compact subsets of $\mathbb{R}^{K},$ $B$ and $B^{\prime},$ and if $m(\beta,b)$
is a measurable, real-valued function that is continuous and bounded on
$B\times B^{\prime},$ then $\sup_{\beta,b}|E_{n}m-Em|\overset{p}{\to}0.$ By
Assumptions 4 and 7, $m$ satisfies these conditions. Hence, we have

$\sup_{\beta,b}\left|\tilde{M}_{n}(\beta,b)-M_{0}(\beta,b)\right|=\sup_{\beta,b}\left|E_{n}m-Em\right|\overset{p}{\to}0.$

Hence (I) vanishes. For (II), Set $R(S_{t},A_{t},S_{t+1})=1/T,$ where $R$ is
defined in (2), and $\gamma=0,$ and note that

$m=v-\gamma\log kl=rR(S_{t},A_{t},S_{t+1})-\gamma\log kl=r,$

for $r$ defined in (31). Note then that Assumptions 4 and 7, which apply to
$m,$ apply equally to $r,$ which gives us that
$\sup_{\beta,b}|E_{n}r-Er|\overset{p}{\to}0$ by Theorem 12.1 in Wooldridge
[2010]. Note that $|b|<\infty$ by Assumption 3 (i) on positivity, and hence
$\sup_{\beta,b}|E_{n}v/E_{n}r|$ is bounded for all $\beta\in B$ and $b\in
B^{\prime}.$ Hence, we have that (II) vanishes. Hence, because (I) and (II)
vanish, we have uniform convergence of ${M}_{n}$. We further have that
Condition 2, the identification condition, holds by Assumption 8. We therefore
conclude, based on Wooldridge [2010], that the two-step M-estimator,
$\beta_{n,\gamma},$ is consistent for $\beta_{0,\gamma}$.

###  A.14 Proof of Theorem 1 (ii)

By Theorem 1 (i) and Assumptions 7-9, we can perform a multivariate Taylor
expansion of the gradient of the objective around the point
$(\beta_{0},b_{0}).$ We will now be extending Theorem 5.41 of van der Vaart
[2000] to the case where we have a nuisance parameter and a weighting
$E_{n}r.$ We will not write the higher order derivatives, as they will still
disappear in the limit, as in the proof of Theorem 5.41. The terms in which
the higher order derivatives appear contain either
$\beta_{n,\gamma}-\beta_{0,\gamma}$ or $b_{n}-b_{0},$ the first of which is
$o_{P}(1)$ by Theorem 1 (i) and the second of which is $o_{P}(1)$ by
Assumption 9, and the derivatives themselves are averages (as noted for the
use case in van der Vaart [2000] as well), and hence $O_{P}(1)$ by the Central
Limit Theorem, so they terms in which they appear will be
$o_{P}(1)O_{P}(1)=o_{P}(1)$.

We have therefore an expansion just concerning the terms of interest,

$\displaystyle
E_{n}\frac{\partial}{\partial\beta}\left(\frac{v}{E_{n}r}-\gamma
kl\right)\bigg{|}_{\begin{subarray}{c}\beta=\beta\\\ b=b\end{subarray}}$
$\displaystyle\approx
E_{n}\frac{\partial}{\partial\beta}\left(\frac{v}{E_{n}r}-\gamma
kl\right)\bigg{|}_{\begin{subarray}{c}\beta=\beta_{0,\gamma}\\\
b=b_{0}\end{subarray}}$
$\displaystyle+(\beta-\beta_{0,\gamma})^{T}E_{n}\frac{\partial^{2}}{\partial\beta^{2}}\left(\frac{v}{E_{n}r}-\gamma
kl\right)\bigg{|}_{\begin{subarray}{c}\beta=\beta_{0,\gamma}\\\
b=b_{0}\end{subarray}}$
$\displaystyle+(b-b_{0})^{T}E_{n}\frac{\partial}{\partial
b}\frac{\partial}{\partial\beta}\left(\frac{v}{E_{n}r}-\gamma
kl\right)\bigg{|}_{{\begin{subarray}{c}\beta=\beta_{0,\gamma}\\\
b=b_{0}\end{subarray}}}$

We can write this succinctly as

$J_{n}(\beta,b)\approx
J_{n}(\beta_{0,\gamma},b_{0})+H_{n}{(\beta-\beta_{0,\gamma})}+X_{n}{(b-b_{0})},$
(43)

where $J_{n}=E_{n}z,$ for $z$ defined in (23), $H_{n},$ and $X_{n}$ refer to
the empirical gradient, Hessian, and cross derivative, respectively, defined
in (6.1), and we only show the arguments for the gradient in (43) to
distinguish its form on the left-hand side of the equation from its form on
the right-hand side of the equation.

Since our objective function, $E_{n}\left({v}/{E_{n}r}-\gamma kl\right),$
given $b_{n}$, is maximized by $\beta_{n,\gamma},$ we can write (43) as

$0=J_{n}(\beta_{n,\gamma},b_{n})\approx
J_{n}(\beta_{0,\gamma},b_{0})+H_{n}(\beta_{n,\gamma}-\beta_{0,\gamma})+X_{n}(b_{n}-b_{0}).$

Rearranging,

$-(J_{n}(\beta_{0,\gamma},b_{0})+X_{n}(\beta_{n,\gamma}))(b_{n}-b_{0})\approx
H_{n}(\beta_{n,\gamma}-\beta_{0,\gamma}).$

By Lemma 3, we have that $H_{n}=H_{0}+o_{P}(1)$ where $H_{0}$ is the Hessian
of $M_{0}.$ Note that also by Lemma 3, $X_{n}=X_{0}+o_{P}(1),$ where $X_{0}$
is the cross derivative of the true objective $M_{0}.$ Thus we have that

$-(J_{n}+(X_{0}+o_{P}(1))(b_{n}-b_{0}))\approx(H_{0}+o_{P}(1))(\beta_{n,\gamma}-\beta_{0,\gamma}).$

Rearranging the terms involving $o_{P}(1),$ and noting that, by Assumption 9,
$o_{P}(1)(b_{n}-b_{0})=o_{P}(1)o_{P}(1)=o_{P}(1),$ we obtain

$-(J_{n}+X_{0}(b_{n}-b_{0}))\approx(H_{0}+o_{P}(1))(\beta_{n,\gamma}-\beta_{0,\gamma})+o_{P}(1)(b_{n}-b_{0}).$

By Assumption 10, the Hessian term becomes invertible in the limit at $H_{0}$,
and we multiply through and take transposes to get

$\displaystyle-(H_{0}+o_{P}(1))^{-1}(J_{n}+X_{0}(b_{n}-b_{0}))\approx(\beta_{n,\gamma}-\beta_{0,\gamma})+(H_{0}+o_{P}(1))^{-1}o_{P}(1)(b_{n}-b_{0}).$

Now, multiply both sides by $\sqrt{n}$ to obtain

$\displaystyle\sqrt{n}(\beta_{n,\gamma}-\beta_{0,\gamma})$
$\displaystyle\approx-(H_{0}+o_{P}(1))^{-1}(J_{n}+X_{0}\sqrt{n}(b_{n}-b_{0}))+(H_{0}+o_{P}(1))^{-1}o_{P}(1)\sqrt{n}(b_{n}-b_{0})$
$\displaystyle=-(H_{0}+o_{P}(1))^{-1}\sqrt{n}(J_{n}+X_{0}(b_{n}-b_{0}))+(H_{0}+o_{P}(1))^{-1}O_{P}(1)o_{P}(1)$
$\displaystyle=-(H_{0}+o_{P}(1))^{-1}(\sqrt{n}J_{n}+X_{0}\sqrt{n}(b_{n}-b_{0}))+o_{P}(1),$
(44)

where $\sqrt{n}(b_{n}-b_{0})=O_{P}(1)$ by Assumption 9 and the fact that
$H_{0}+o_{P}(1)$ is invertible by Assumption 10 implies that its inverse is
bounded in the limit, which allowed us to write that

$(H_{0}+o_{P}(1))^{-1}\sqrt{n}(b_{n}-b_{0})^{T}o_{P}(1)=(H_{0}+o_{P}(1))^{-1}O_{P}(1)o_{P}(1)=o_{P}(1).$

To show that (A.14) is asymptotically normal, we must show that
$\sqrt{n}J_{n}$ is asymptotically normal. Recall the definition of
$J_{n}=E_{n}z$ from (38). Note that

$\displaystyle\sqrt{n}J_{n}$
$\displaystyle=\frac{\sqrt{n}E_{n}\frac{\partial}{\partial\beta}v}{E_{n}r}-\frac{\sqrt{n}E_{n}v(E_{n}\frac{\partial}{\partial\beta}r)}{(E_{n}r)^{2}}-\sqrt{n}E_{n}\frac{\partial}{\partial\beta}\gamma
kl(\beta,b)$ (45)
$\displaystyle\overset{\mathcal{L}}{\to}\frac{z_{1}}{Er}-\frac{z_{2}(E\frac{\partial}{\partial\beta}r)}{(Er)^{2}}-z_{3},$

where $z_{1},z_{2},$ and $z_{3}$ are normally distributed by the Central Limit
Theorem. By Slutsky’s theorem and Remark 4, $\sqrt{n}J_{n}$ is therefore
asymptotically normal. By the discussion in Appendix A.16,
$\sqrt{n}(b_{n}-b_{0})$ is asymptotically normal. A final application of
Slutsky’s theorem to the expression in (A.14) proves the result.

###  A.15 Proof of Theorem 2

In Theorem 1 (ii), we followed the standard proof of normality for two-step
M-estimators that fulfill the classical conditions. For this double penalized
estimator ($\beta_{n,\gamma,\lambda}$) defined in (17) proof, we instead
follow Zou [2006]. Define, for ${M}_{n}$ in (14),

$\Psi_{n}(\beta,b)=n{M}_{n}-\lambda_{n}\sum_{k=1}^{K}{w_{n,k}}|\beta_{k}-b_{k}|$

and let

$\displaystyle\Gamma_{n}(\beta)$
$\displaystyle=\Psi_{n}(\beta,b)-\Psi_{n}(\beta_{0,\gamma},b_{0}).$

Note that, similarly to Theorem 1 (ii), we can write

$\displaystyle\Psi_{n}(\beta,b)-\Psi_{n}(\beta_{0,\gamma},b_{0})$
$\displaystyle\approx\sqrt{n}J_{n}\sqrt{n}(\beta-\beta_{0,\gamma})+\sqrt{n}z_{b,n}\sqrt{n}(b-b_{0})$
$\displaystyle+\frac{1}{2}\sqrt{n}(\beta-\beta_{0,\gamma})^{T}H_{n}\sqrt{n}(\beta-\beta_{0,\gamma})$
$\displaystyle+\frac{1}{2}\sqrt{n}(b-b_{0})^{T}H_{bb,n}\sqrt{n}(b-b_{0})$
$\displaystyle+\sqrt{n}(b-b_{0})^{T}X_{n}^{T}(\beta-\beta_{0,\gamma})$
$\displaystyle-\lambda_{n}\sum_{k=1}^{K}w_{n,k}\\{|\beta_{k}-b_{k}|-|\beta_{0,\gamma,k}-b_{0,k}|\\},$

where

$z_{b,n}=\frac{\partial}{\partial
b}{M}_{n}\bigg{|}_{\begin{subarray}{c}\beta=\beta_{0,\gamma}\\\
b=b_{0}\end{subarray}}$

and

$H_{bb,n}=\frac{\partial^{2}}{\partial
b^{2}}{M}_{n}\bigg{|}_{\begin{subarray}{c}\beta=\beta_{0,\gamma}\\\
b=b_{0}\end{subarray}}.$

Both $\sqrt{n}J_{n}$ and $\sqrt{n}z_{b,n}$ are $O_{P}(1)$ by the Central Limit
Theorem.

Defining $u=\sqrt{n}(\beta-\beta_{0,\gamma})$ and $v=\sqrt{n}(b-b_{0})$, let
us now rewrite the expansion above following the style in Zou [2006] by
substituting in $u,v$.

$\displaystyle n{M}_{n}(u,v)-n{M}_{n}(0,0)$
$\displaystyle\approx\sqrt{n}J_{n}u+\sqrt{n}z_{b,n}v+u^{T}H_{n}u+v^{T}H_{bb,n}v+v^{T}X_{n}^{T}u$

Note that the terms that depend only on $v$ (and thus only on $b$) are
constant with respect to $u$ (and by extension with respect to $\beta$) and,
therefore, we do not need to consider them further in the objective function,
which we will be maximizing with respect to $\beta$ only. Therefore, let us
focus only on the redefined objective

$\tilde{\Gamma}_{n}\approx\sqrt{n}J_{n}u+u^{T}H_{n}u+v^{T}X_{n}^{T}u,$

which, as discussed in Theorem 1 (ii), converges in probability to

$\tilde{\Gamma}_{0}\approx
z_{0}u_{0}+u_{0}^{T}H_{0}u_{0}+v_{0}^{T}X_{0}^{T}u_{0}.$

We can partition $\tilde{\Gamma}_{0}$ such that
$u_{0,1}=u_{0,\mathcal{A}},u_{0,2}=u_{0,\mathcal{A}^{c}},v_{0,1}=v_{0,\mathcal{A}},v_{0,2}=v_{0,\mathcal{A}^{c}},$
etc, and rewrite,

$\displaystyle\tilde{\Gamma}_{0}\approx\left[{\begin{array}[]{cc}z_{0,1}&z_{0,2}\\\
\end{array}}\right]\left[{\begin{array}[]{cc}u_{0,1}\\\ u_{0,2}\\\
\end{array}}\right]$
$\displaystyle+\left[{\begin{array}[]{cc}u_{0,1}&u_{0,2}\\\
\end{array}}\right]\left[{\begin{array}[]{cc}H_{0,11}&H_{0,12}\\\
H_{0,21}&H_{0,22}\\\ \end{array}}\right]\left[{\begin{array}[]{cc}u_{0,1}\\\
u_{0,2}\\\ \end{array}}\right]$ (54)
$\displaystyle+\left[{\begin{array}[]{cc}v_{0,1}&v_{0,2}\\\
\end{array}}\right]\left[{\begin{array}[]{cc}(X_{0}^{T})_{11}&(X_{0}^{T})_{12}\\\
(X_{0}^{T})_{21}&(X_{0}^{T})_{22}\\\
\end{array}}\right]\left[{\begin{array}[]{cc}u_{0,1}\\\ u_{0,2}\\\
\end{array}}\right].$ (60)

This partitioning, as in [Zou, 2006], helps us pinpoint the parts of the
objectives that are relevant to the non-behavioral coefficients. As we will
see now, the penalty will go to zero for the pieces of the objective that
depend on the non-behavioral parameters and to infinity for the non-behavioral
parameters. Note that a major innovation of Zou [2006] is to add dimension-
specific weights, $w_{n,j}$, to the penalty term, which ensure this type of
limiting behavior.

Let us now consider the penalty. We follow Zou [2006] closely here. Note that,
by the continuous mapping theorem, Assumption 9, and Theorem 1 (i),

$w_{n,j}=\frac{1}{|{\beta_{n,\gamma,j}}-b_{n,j}|^{\delta}}\overset{p}{\to}\frac{1}{|{\beta_{0,\gamma,j}}-b_{0,j}|^{\delta}}.$

Our relative sparsity penalty is

$\displaystyle\frac{\lambda_{n}}{\sqrt{n}}\sum_{k=1}^{K}w_{n,k}\sqrt{n}\\{|\beta_{k}-b_{k}|-|\beta_{0,\gamma,j}-b_{0,j}|\\}$
$\displaystyle=\frac{\lambda_{n}}{\sqrt{n}}\sum_{k=1}^{K}w_{n,k}\sqrt{n}\\{|\beta_{0,\gamma,k}+\frac{u_{k}}{\sqrt{n}}-(b_{0,k}+\frac{v_{k}}{\sqrt{n}})|-|\beta_{0,\gamma,k}-b_{0,k}|\\}$
$\displaystyle=\frac{\lambda_{n}}{\sqrt{n}}\sum_{k=1}^{K}w_{n,k}\sqrt{n}\\{|(\beta_{0,\gamma,k}-b_{0,k})+\frac{u_{k}-v_{k}}{\sqrt{n}}|-|\beta_{0,\gamma,k}-b_{0,k}|\\}.$

We will now use the results in [Zou, 2006] directly on the penalty term, but
we will treat our shifted coefficient $(\beta-b)$ as they treat their raw
coefficient $\beta$. Hence, following [Zou, 2006], if
$(\beta_{0,\gamma,j}-b_{0j})\neq 0,$ then
$w_{n,j}\overset{p}{\to}|\beta_{0,\gamma,j}-b_{0,j}|^{-\delta}$ and

$\displaystyle\sqrt{n}\\{|(\beta_{0,\gamma,j}-b_{0,j})+\frac{u_{j}-v_{j}}{\sqrt{n}}|-|\beta_{0,\gamma,j}-b_{0,j}|\\}\overset{p}{\to}(u_{0,j}-v_{0,j})sgn(\beta_{0,\gamma,j}-b_{0,j}).$

By Slutsky’s theorem, since $\lambda_{n}/\sqrt{n}\rightarrow 0,$

$\frac{\lambda_{n}}{\sqrt{n}}w_{n,j}\sqrt{n}\\{|(\beta_{0,\gamma,j}-b_{0,j})+\frac{u_{j}-v_{j}}{\sqrt{n}}|-|\beta_{0,\gamma,j}-b_{0,j}|\\}\overset{p}{\to}0.$

If, on the other hand, $(\beta_{0,\gamma,j}-b_{0,j})=0,$ then

$\displaystyle\sqrt{n}\\{|(\beta_{0,\gamma,j}-b_{0,j})+\frac{u_{j}-v_{j}}{\sqrt{n}}|-|\beta_{0,\gamma,j}-b_{0,j}|=|u_{j}-v_{j}|.$

Since in this case

$\frac{\lambda_{n}}{\sqrt{n}}w_{n,j}=\frac{\lambda_{n}}{\sqrt{n}}n^{\delta/2}(\sqrt{n}(\beta_{n,\gamma}-b_{n}))^{-\delta},$

where

$\sqrt{n}(\beta_{n,\gamma,j}-b_{n,j})=O_{P}(1)$

since

$\displaystyle\sqrt{n}(\beta_{n,\gamma,j}-b_{n,j})$
$\displaystyle=\sqrt{n}(\beta_{n,\gamma,j}-\beta_{0,\gamma,j}+\beta_{0,\gamma,j}-b_{n,j}-b_{0,j}+b_{0,j})$
$\displaystyle=\sqrt{n}(\beta_{n,\gamma,j}-\beta_{0,\gamma,j}-(b_{n,j}-b_{0,j})+\beta_{0,\gamma,j}-b_{0,j})$
$\displaystyle=\sqrt{n}(\beta_{n,\gamma,j}-\beta_{0,\gamma,j})-\sqrt{n}(b_{n,j}-b_{0,j})$
$\displaystyle=O_{P}(1)-O_{P}(1)=O_{P}(1).$

We then have, since $\lambda_{n}/\sqrt{n}n^{\delta/2}\rightarrow\infty,$ that,
by Slutsky’s theorem,

$\frac{\lambda_{n}}{\sqrt{n}}w_{n,j}\overset{p}{\to}\infty.$

We thus obtain complete penalization to the behavioral coefficient in this
case. Recall the previous partition of $\tilde{\Gamma},$ from (54). Coupled
with the assumption that $\lambda_{n}n^{(\delta-1)/2}\rightarrow\infty,$ we
have that

$\tilde{\Gamma}_{n}\overset{p}{\to}\tilde{\Gamma}_{0}=\begin{cases}z_{0}^{T}u_{0}+u_{0}^{T}H_{0}u_{0}+v_{0}^{T}X_{0}u_{0}&\mbox{if
}u_{0,j}-v_{0,j}=0,\forall j\not\in\mathcal{A}\\\
-\infty&otherwise\end{cases}.$

We see that even if the condition $u_{0,j}=v_{0,j}\forall j\not\in\mathcal{A}$
is satisfied, we do not have simplification of the objective, as is the case
in Zou [2006], since the terms $u_{0}$ and $v_{0}$ are not nonzero. For the
first case, we separate $u_{0,1}$ from $u_{0,2}$ to obtain

$\displaystyle z_{0}^{T}u_{0}+u_{0}^{T}H_{0}u_{0}/2+v_{0}^{T}(X_{0}^{T})u_{0}$
$\displaystyle=[z_{0,1}^{T}+u_{0,2}^{T}H_{0,21}/2+u_{0,2}^{T}H_{0,12}^{T}/2+v_{0,1}^{T}(X_{0}^{T})_{11}+v_{0,2}^{T}(X_{0}^{T})_{21}]^{T}u_{0,1}$
$\displaystyle\ \ \ \ +u_{0,1}^{T}H_{0,11}u_{0,1}/2$
$\displaystyle=r_{v_{0}}u_{0,1}+u_{0,1}^{T}H_{0,11}u_{0,1}/2,$

where

$r_{v_{0}}=[z_{0,1}^{T}+u_{0,2}^{T}H_{0,21}/2+u_{0,2}^{T}H_{0,12}^{T}/2+v_{0,1}^{T}(X_{0}^{T})_{11}+v_{0,2}^{T}(X_{0}^{T})_{21}].$

We then differentiate with respect to $u_{0,1}$ to see that the penalized
objective, $W_{0},$ is maximized at

$u_{0,\mathcal{A}}=-(H_{0,11})^{-1}r_{v_{0}}^{T}.$

If, on the other hand, $\beta_{0,\gamma,j}-b_{0,j}\neq 0,$ which occurs for
$j\not\in\mathcal{A},$ then the weight of the penalty goes off to negative
infinity. In this case, we must minimize $|u_{j}-v_{j}|,$ which drives $u_{j}$
to $v_{j}$. Finally, if we let $u_{n}=\arg\max_{u}\tilde{\Gamma}_{n}$ and
$u_{0}=\arg\max_{u}\tilde{\Gamma}_{0},$ then, since
$\tilde{\Gamma}_{n}\overset{p}{\to}\tilde{\Gamma}_{0},$ we have that, by epi-
convergence [Geyer, 1994, Knight and Fu, 2000], $u_{n}\rightarrow u_{0},$
where

$u_{0}=\left[u_{0,{\mathcal{A}}},u_{0,\mathcal{A}^{c}}\right]^{T}=\left[-(H_{0,11})^{-1}r_{v_{0}}^{T},v_{\mathcal{A}^{c}}\right]^{T}.$

Thus we have proved the theorem.

###  A.16 Behavioral policy (nuisance) influence function

We derive an estimator for $\sqrt{n}(b_{n}-b_{0})$ that does not depend on
$b_{0}.$ Let $\sqrt{n}(b_{n}-b_{0})\overset{\mathcal{L}}{\to}q_{0},$ which is
normally distributed [Casella and Berger, 2002, van der Vaart, 2000]. Observe,
by Taylor’s theorem, that

$\displaystyle\sqrt{n}(b_{n}-b_{0})$
$\displaystyle\approx-(E_{n}l^{\prime\prime}(b_{0}))^{-1}\sqrt{n}E_{n}l^{\prime}(b_{0})$
$\displaystyle=-\sqrt{n}E_{n}(E_{n}l^{\prime\prime}(b_{0}))^{-1}l^{\prime}(b_{0})$
$\displaystyle\approx-\sqrt{n}E_{n}(E_{n}l^{\prime\prime}(b_{n}))^{-1}l^{\prime}(b_{n})$
$\displaystyle=\sqrt{n}E_{n}q=\sqrt{n}q_{n},$

where $q$ is defined in (24) and $\sqrt{n}q_{n}$ does not depend on $b_{0},$
as desired.

###  A.17 Variance of the suggested policy coefficients

Let us now derive an estimator for the asymptotic variance of
$\beta_{n,\gamma}.$ Recall that $z_{0},$ $H_{0},$ and $X_{0}$ are the expected
gradient with respect to $\beta$, the expected Hessian with respect to
$\beta,$ and the expected cross derivative with respect to $b$ and $\beta,$
all of which are defined in (6.1). Recall that in Theorem 1 (ii) we used a
Taylor expansion in $\beta,b$ to obtain

$\sqrt{n}(\beta_{n,\gamma}-\beta_{0,\gamma})\approx-
H_{n}^{-1}(\sqrt{n}J_{n}+X_{n}\sqrt{n}q_{n}),$

where $J_{n}$ and $X_{n}$ are defined in (6.1), along with their expectations,
and $q_{n}$ is defined in (24). Recall that

$r=\frac{\prod_{t=0}^{T}\pi_{\beta,b}(A_{t}|S_{t})}{\prod_{t=0}^{T}\pi_{b}(A_{t}|S_{t})},v=r\sum_{t=0}^{T}R(S_{t},A_{t},S_{t+1}).$