#  Rejoinder: Learning Optimal Distributionally Robust Individualized
Treatment Rules

Weibin Mo, Zhengling Qi and Yufeng Liu Weibin Mo and Zhengling Qi are co-first
authors for the paper. Weibin Mo is a Ph.D. student, Department of Statistics
and Operations Research, University of North Carolina at Chapel Hill, Chapel
Hill, NC 27599, USA. E-mail<EMAIL_ADDRESS>Zhengling Qi is Assistant
Professor, Department of Decision Sciences, George Washington University,
Washington, D.C. 20052, USA. E-mail<EMAIL_ADDRESS>Yufeng Liu is
Professor, Department of Statistics and Operations Research, Department of
Genetics, Department of Biostatistics, Carolina Center for Genome Science,
Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel
Hill, NC 27599, USA. E-mail<EMAIL_ADDRESS>
We thank the opportunity offered by editors for this discussion and the
discussants for their insightful comments and thoughtful contributions. We
also want to congratulate Kallus (2020) for his inspiring work in improving
the efficiency of policy learning by retargeting. Motivated from the
discussion in Dukes and Vansteelandt (2020), we first point out interesting
connections and distinctions between our work and Kallus (2020) in Section 1.
In particular, the assumptions and sources of variation for consideration in
these two papers lead to different research problems with different scopes and
focuses. In Section 2, following the discussions in Li et al. (2020); Liang
and Zhao (2020), we also consider the efficient policy evaluation problem when
we have some data from the testing distribution available at the training
stage. We show that under the assumption that the sample sizes from training
and testing are growing in the same order, efficient value function estimates
can deliver competitive performance. We further show some connections of these
estimates with existing literature. However, when the growth of testing sample
size available for training is in a slower order, efficient value function
estimates may not perform well anymore. In contrast, the requirement of the
testing sample size for DRITR is not as strong as that of efficient policy
evaluation using the combined data. Finally, we highlight the general
applicability and usefulness of DRITR in Section 3.

##  1 Efficiency and Robustness

The discussion in Dukes and Vansteelandt (2020) highlighted the importance of
leveraging relevant data when inferring which treatment to assign. In
particular, the covariate weight functions considered in DRITR and retargeted
policy learning can imply different ways of utilizing relevant data during
training, and different target populations during testing. In this section, we
further clarify the differences and connections between DRITR and retargeted
policy learning.

DRITR and retargeted policy learning can be distinct from each other in terms
of the following two main perspectives:

  1. (I)

The assumptions used in these two papers are different. If the true
conditional treatment effect (CTE) function $C(\bm{x})$ induces a globally
optimal ITR $\bm{x}\mapsto\mathsf{sign}[C(\bm{x})]$ that belongs to the class
$\mathcal{D}$, then policy learning over $\mathcal{D}$ is not sensitive to
covariate reweighting/retargeting, as were discussed in Mo et al. (2020,
Remark 1) and Kallus (2020, Lemma 2.1). In particular, Kallus (2020) referred
this case as $\mathcal{D}$ being correctly specified, and focused on this case
to obtain the efficient covariate weighting function. In contrast, Mo et al.
(2020) studied the learning problem over a restricted ITR class $\mathcal{D}$
that is misspecified for the globally optimal ITR, and optimized the worst-
case reweighted value function as a robust objective;

  2. (II)

The sources of variation considered in these two papers are also different.
The optimality criteria for an efficient covariate weight function in Kallus
(2020) focuses on reducing the conditional-on-covariate variance of the
weighted outcome, i.e. the variance explained by $(A,Y)|\bm{X}$ as in their
Equation (6). In contrast, Mo et al. (2020) considered the robust criteria due
to covariate variations. In the formulation of DRITR, the effect of
$(A,Y)|\bm{X}$ is absorbed into the CTE function
$C(\bm{X})=\mathbb{E}[Y(1)-Y(-1)|\bm{X}]$ as a conditional mean function in
$\bm{X}$. The DR-value function in Mo et al. (2020, Equation (4)) robustifies
the underlying covariate distribution for evaluation of $C(\bm{X})$.

Two distinct assumptions on the ITR class mentioned above can explain
different goals of these two papers. When assuming a correctly specified ITR
class, Kallus (2020) can leverage the retargeting invariance property for
efficiency improvement. In contrast, when allowing the misspecified ITR class,
Mo et al. (2020) focused on the worst case of covariate changes to carry out
the robust policy optimization for generalizability. The phenomenon that
different model assumptions result in different goals can be remotely
analogous to semiparametric inference (Robins et al., 2000). Specifically,
when nuisance models are correct, the semiparametric efficient estimate can be
obtained. When either one of but not both nuisance models are correct, a
doubly robust estimate remains consistent. However, even for semiparametric
inference, the goals of efficiency and robustness may not coexist for a
specific estimate. For example, a generic construction of multiply robust
estimate for factorized likelihood models is generally not semiparametric
efficient under any model assumptions (Molina et al., 2017). It depends on the
main focuses of applications to choose either a semiparametric efficient
estimate or a multiply robust estimate. Analogously, the use of retargeted
policy learning or DRITR also depends on the goal for efficiency or
robustness, subject to the practitioners’ optimistic or pessimistic beliefs on
the working class of ITRs to learn from. In particular, DRITR is robust to
potential covariate changes under the misspecified ITR class assumption.

The distinctions on source of variation mentioned in (2) characterize two
different types of research questions. When questions are related to the
variations of $(A,Y)|\bm{X}$, such as limited overlap and heteroscedasticity,
retargeting weights in Kallus (2020) can provide an optimal way to control
conditional-on-covariate variances (Crump et al., 2006, 2009). However, such
optimal weights cannot control the variances of covariates themselves.
Therefore, retargeted policy learning may generally work well for the case
that conditional-on-covariate variances are the estimation bottleneck, while
covariate variations can be ignored. Such an example can be found in Athey and
Wager (2020, Section 4). Besides ignoring variances from covariates, the
violation of retargeting invariance, i.e. misspecifying the ITR class, can
also contribute biases due to reweighting. Kallus (2020, Section 5.2) also
pointed out that the levels of ITR class misspecification and covariate
changes need to be assumed mild when applying retargeting policy learning.

In contrast to the scope of retargeted policy learning, DRITR has an explicit
focus on covariate changes. This was motivated from the fact that the
challenges of generalizing causal estimands are mainly due to covariate
changes, and reweighting can correctly target the testing population of
interest (Stuart et al., 2011). Without prior information on the testing
population at the training stage, DRITR took the worst-case reweighting scheme
to guarantee robust performance. A similar strategy was also leveraged by Zhao
et al. (2019). However, given that reweighting in this case mainly aims for
covariate-change correction, DRITR is not intended for handling variations
from treatments and outcomes. Therefore, our formulation utilizes a
nonparametric estimate of the CTE function to remove the variations from
treatments and outcomes.

One potential research question is whether robust reweighting in DRITR can
also handle the limited overlap and heteroscedasticity problems considered in
retargeted policy learning. Unfortunately, we suspect that robust reweighting
and efficient retargeting may have opposite effects. In particular, we notice
that the optimal retargeting weight function from Kallus (2020) is inverse-
proportionate to $\sum_{a}{\sigma^{2}(\bm{x},a)\over\pi_{A}(a|\bm{x})}$, where
$\pi_{A}(a|\bm{x})$ is the propensity score function. In contrast, for robust
reweighting, Qi et al. (2019) studied a modified version of DRITR that focused
on the dual formulation. Although they did not explicitly link to the robust
reweigting, the resulting robust weight function in Qi et al. (2019) is
increasing in the variance function
$\sigma^{2}(\bm{x},a):=\mathbb{E}(Y|\bm{X}=\bm{x},A=a)$. Consequently, the
robust weight function may not be compatible with the retargeting weight
function. This suggests that DRITR and retargeted policy learning may need to
be utilized in different scenarios. It may be interesting to study a combined
version of retargeted policy learning and DRITR that can enjoy both efficiency
and robustness.

To conclude this section, we make a comparison between DRITR and retargeted
policy learning in the following Table 1.

| DRITR | Retargeted Policy Learning
---|---|---
Assumption | $\\{\bm{x}\mapsto\mathsf{sign}[C(\bm{x})]\\}\nsubseteq\mathcal{D}$ | $\\{\bm{x}\mapsto\mathsf{sign}[C(\bm{x})]\\}\subseteq\mathcal{D}$
Source of Variation | covariate $\bm{X}$ | conditional-on-covariate $(A,Y)|\bm{X}$
Weight Dependency |  | CTE function $C(\bm{x})$
---
underlying ITR $d(\bm{x})$
| propensity score function $\pi_{A}(a|\bm{x})$
---
variance function $\sigma^{2}(\bm{x},a)$
Optimality | maximizing worst-case value | minimizing conditional-on-covariate variance
Main Applications | covariate changes | limited overlap and heteroscedasticity
Table 1: Comparison of DRITR (Mo et al., 2020) and Retargeted Policy Learning
(Kallus, 2020)

##  2 Efficient Policy Evaluation under Specific Covariate Changes

DRITR aims for performance guarantee in presence of general covariate changes.
It assumes no access to any information from the testing distribution at the
training stage. When a small set of calibrating data is available from the
testing distribution, such information is only used for choosing a DR-constant
to determine the final DRITR. However, the problem of combining the training
and calibrating data during training as discussed in Li et al. (2020); Liang
and Zhao (2020) is also worthwhile to study. In this section, we focus on
efficient policy evaluation with training and calibrating data from a specific
testing distribution. It should be highlighted that the true covariate density
ratio of testing with respect to training is not readily available, and can
only be inferred from the observed training and calibrating data.

Consider two possible types of pooled datasets
$\mathcal{O}^{(1)}=\\{\bm{O}_{i}=(\bm{X}_{i},A_{i},Y_{i},S_{i})\\}_{i=1}^{n}$
and
$\mathcal{O}^{(2)}=\\{\bm{O}_{i}=(\bm{X}_{i},S_{i}A_{i},S_{i}Y_{i},S_{i})\\}_{i=1}^{n}$,
where $S_{i}=1$ indicates that $\bm{O}_{i}|(S_{i}=1)\sim\mathbb{P}_{\rm
train}$, and the $i$-th data point belongs to the training data; $S_{i}=0$
indicates that $\bm{O}_{i}|(S_{i}=0)\sim\mathbb{P}_{\rm test}$, and the $i$-th
data point belongs to the calibrating data. For the Type-1 dataset
$\mathcal{O}^{(1)}$, we observe covariates, treatment assignments and outcomes
in both training and calibrating data. For the Type-2 dataset
$\mathcal{O}^{(2)}$, treatment assignments and outcomes
$\\{(A_{i},Y_{i}):S_{i}=0\\}$ in calibrating data are missing. Let $Y_{i}(1)$
and $Y_{i}(-1)$ be the potential outcomes, and denote
$Y_{i}(d):=\sum_{a\in\\{1,-1\\}}Y_{i}(a){\mathbbm{1}}[d(\bm{X}_{i})=a]$. For a
fixed ITR $d:\mathcal{X}\to\\{1,-1\\}$, the goal for policy evaluation is to
estimate the following values of $d$ under the specific testing distribution:

$\displaystyle\theta:=\mathcal{V}_{\rm test}(d)=\mathbb{E}_{\rm
test}[Y_{i}(d)];\quad\theta_{1}:=\mathcal{V}_{1,\rm test}(d)=\mathbb{E}_{\rm
test}[Y_{i}(d)-Y_{i}(-d)].$ (1)

In order to identify the potential outcomes from observed outcomes, we make
the following assumptions (Rubin, 1974).

######  Assumption 1 (Consistency).

$Y_{i}=Y_{i}(A_{i})=\sum_{a\in\\{1,-1\\}}Y_{i}(a){\mathbbm{1}}(A_{i}=a)$.

######  Assumption 2 (Exchangeability over Treatment).

For $\bm{x}\in\mathcal{X}$, $s\in\\{1,0\\}$ and $a\in\\{1,-1\\}$, we have
$Y_{i}(a)\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}A_{i}|(\bm{X}_{i}=\bm{x},S_{i}=s)$.

######  Assumption 3 (Strong Ignorability of Treatment).

For $\bm{x}\in\mathcal{X}$, $s\in\\{1,0\\}$ and $a\in\\{1,-1\\}$, we have
$\pi_{A}(\bm{a}|\bm{x},s):=\mathbb{P}(A_{i}=a|\bm{X}_{i}=\bm{x},S_{i}=s)\geq\tau>0$.

Notice that the policy evaluation problem can be cast as a parameter
estimation problem. The key challenge here is that the estimand $\theta$ or
$\theta_{1}$ is evaluated under the target population $\mathbb{P}_{\rm test}$
that can be different from the distribution of the observed data
$\mathcal{O}^{(1)}$ or $\mathcal{O}^{(2)}$. In order to identify $\theta$ and
$\theta_{1}$ from the pooled data $\mathcal{O}^{(1)}$ or $\mathcal{O}^{(2)}$,
we consider the following mean exchangeability assumption used in Pearl and
Bareinboim (2014) to transport information from $\mathbb{P}_{\rm train}$ to
$\mathbb{P}_{\rm test}$.

######  Assumption 4 (Mean Exchangeability over Selection).

For $\bm{x}\in\mathcal{X}$, $a\in\\{1,-1\\}$ and $s\in\\{1,0\\}$, we have
$\mathbb{E}[Y_{i}(a)|\bm{X}_{i}=\bm{x},S_{i}=s]=\mathbb{E}[Y_{i}(a)|\bm{X}_{i}=\bm{x}]:=Q(\bm{x},a)$.

Assumption 4 implies that
$\mathbb{E}[Y_{i}(1)-Y_{i}(-1)|\bm{X}_{i},S_{i}]=\mathbb{E}[Y_{i}(1)-Y_{i}(-1)|\bm{X}_{i}]=Q(\bm{X}_{i},1)-Q(\bm{X}_{i},-1):=C(\bm{X}_{i})$,
which was the required condition in Mo et al. (2020, Remark 2). Covariate
changes in Mo et al. (2020, Assumption 1) is sufficient for Assumption 4. In
order to identify $\theta$ and $\theta_{1}$ from the training and pooled data,
we further consider the strong ignorability assumption on the conditional-on-
covariate selection probability function.

######  Assumption 5 (Strong Ignorability of Selection).

For $\bm{x}\in\mathcal{X}$ and $s\in\\{1,0\\}$, we have
$\pi_{S}(s|\bm{x}):=\mathbb{P}(S_{i}=s|\bm{X}_{i}=\bm{x})\geq\delta>0$.

Assumption 5 also implies the positivity of the marginal selection
probability: $\rho_{S}:=\mathbb{P}(S_{i}=1)\geq\delta$, and
$1-\rho_{S}=\mathbb{P}(S_{i}=0)\geq\delta$. Under Assumption 5, we can express
$\mathbb{E}_{\rm test}(\cdot)=\mathbb{E}(\cdot|S=0)$ under $\mathbb{E}_{\rm
train}(\cdot)=\mathbb{E}(\cdot|S=1)$ by reweighting, which is given in the
following Lemma 1.

######  Lemma 1.

Consider the data $(\bm{X},S)\in\mathcal{X}\times\\{1,0\\}$ satisfying
$\mathbb{P}(S=s|\bm{X})\geq\delta>0$ for $s\in\\{1,0\\}$. Then for any
$g:\mathcal{X}\to\mathbb{R}$ such that $\mathbb{E}[|g(\bm{X})|]<+\infty$, we
have

$\mathbb{E}[g(\bm{X})|S=0]=\mathbb{E}[w(\bm{X})g(\bm{X})|S=1],$

where

$w(\bm{X})={\mathbb{P}(S=1)\mathbb{P}(S=0|\bm{X})\over\mathbb{P}(S=0)\mathbb{P}(S=1|\bm{X})}.$

In particular, $w(\bm{X})\geq 0$ and $\mathbb{E}[w(\bm{X})|S=1]=1$.

Lemma 1 can be regarded as a parallel version of the weighted representation
in Mo et al. (2020, Assumption 1), and further suggests the form of the
theoretical weighting function $w(\bm{x})={\rho_{S}\over
1-\rho_{S}}{\pi_{S}(0|\bm{x})\over\pi_{S}(1|\bm{x})}$ for
$\bm{x}\in\mathcal{X}$.

###  2.1 Semiparametric Inference

In order to consider estimates for $\theta$ and $\theta_{1}$, we first study
the semiparametric inference properties for $\theta$ and $\theta_{1}$ based on
two types of data $\mathcal{O}^{(1)}$ and $\mathcal{O}^{(2)}$ respectively. In
Proposition 2, we establish the identification of $\theta$ and $\theta_{1}$
from the observed data. Then we derive the efficient influence functions for
$\theta$ and $\theta_{1}$ in Theorem 3, which is consistent with results from
Rudolph and van der Laan (2017), Dahabreh et al. (2019) and Uehara et al.
(2020). For $\bm{x}\in\mathcal{X}$, we denote
$Q(\bm{x},d):=\sum_{a\in\\{1,-1\\}}Q(\bm{x},a){\mathbbm{1}}[d(\bm{x})=a]$.

######  Proposition 2.

Consider the observed data $\mathcal{O}^{(1)}$ and $\mathcal{O}^{(2)}$, and
the parameters $\theta$ and $\theta_{1}$ in (1). Under Assumptions 1-5, we
have:

  1. (I)

$\displaystyle\theta$
$\displaystyle=\mathbb{E}\left\\{{{\mathbbm{1}}(S_{i}=0)\over
1-\rho_{S}}Q(\bm{X}_{i},d)\right\\}$
$\displaystyle=\mathbb{E}\Big{\\{}[w(\bm{X}_{i}){\mathbbm{1}}(S_{i}=1)+{\mathbbm{1}}(S_{i}=0)]Q(\bm{X}_{i},d)\Big{\\}}$
$\displaystyle(\text{on $\mathcal{O}^{(1)}$})$
$\displaystyle=\mathbb{E}\left\\{{w(\bm{X}_{i}){\mathbbm{1}}(S_{i}=1)\over\rho_{S}}Q(\bm{X}_{i},d)\right\\}$
$\displaystyle(\text{on $\mathcal{O}^{(2)}$}).$

The expressions for $\theta_{1}$ can be obtained by replacing
$Q(\bm{X}_{i},d)$ by $C(\bm{X}_{i})d(\bm{X}_{i})$;

  2. (II)

For $\bm{x}\in\mathcal{X}$,

$\displaystyle Q(\bm{X}_{i},d)$
$\displaystyle=\mathbb{E}\left\\{\left({{\mathbbm{1}}(S_{i}=1)\over\pi_{A}(A_{i}|\bm{X}_{i},1)}+{{\mathbbm{1}}(S_{i}=0)\over\pi_{A}(A_{i}|\bm{X}_{i},0)}\right){\mathbbm{1}}[d(\bm{X}_{i})=A_{i}]Y_{i}\middle|\bm{X}_{i}\right\\}$
$\displaystyle(\text{on $\mathcal{O}^{(1)}$})$
$\displaystyle=\mathbb{E}\left\\{{1\over\pi_{A}(A_{i}|\bm{X}_{i},1)}{\mathbbm{1}}[d(\bm{X}_{i})=A_{i}]Y_{i}\middle|\bm{X}_{i},S_{i}=1\right\\}$
$\displaystyle(\text{on $\mathcal{O}^{(2)}$}).$

The expressions for $\theta_{1}$ can be obtained by replacing
${\mathbbm{1}}[d(\bm{X}_{i})=A_{i}]$ by $d(\bm{X}_{i})A_{i}$.

######  Theorem 3 (Efficient Influence Functions).

Consider the observed data $\mathcal{O}^{(1)}$ and $\mathcal{O}^{(2)}$, and
the parameters $\theta$ and $\theta_{1}$ in (1). Under Assumptions 1-5 and
some regularity conditions, the corresponding semiparametric efficient
influence functions (EIFs) are:

$\displaystyle\begin{array}[]{rclclclcl}\mathrm{EIF}^{(1)}(\theta)&=&\left({w(\bm{X}){\mathbbm{1}}(S=1)\over\pi_{A}(A|\bm{X},1)}+{{\mathbbm{1}}(S=0)\over\pi_{A}(A|\bm{X},0)}\right)&\times&{\mathbbm{1}}[d(\bm{X})=A][Y-Q(\bm{X},A)]&+&{{\mathbbm{1}}(S=0)\over
1-\rho_{S}}[Q(\bm{X},d)-\theta];\\\
\mathrm{EIF}^{(2)}(\theta)&=&{w(\bm{X}){\mathbbm{1}}(S=1)\over\rho_{S}\pi_{A}(A|\bm{X},1)}&\times&{\mathbbm{1}}[d(\bm{X})=A][Y-Q(\bm{X},A)]&+&{{\mathbbm{1}}(S=0)\over
1-\rho_{S}}[Q(\bm{X},d)-\theta];\\\
\mathrm{EIF}^{(1)}(\theta_{1})&=&\left({w(\bm{X}){\mathbbm{1}}(S=1)\over\pi_{A}(A|\bm{X},1)}+{{\mathbbm{1}}(S=0)\over\pi_{A}(A|\bm{X},0)}\right)&\times&d(\bm{X})A[Y-Q(\bm{X},A)]&+&{{\mathbbm{1}}(S=0)\over
1-\rho_{S}}[C(\bm{X})d(\bm{X})-\theta_{1}];\\\
\mathrm{EIF}^{(2)}(\theta_{1})&=&{w(\bm{X}){\mathbbm{1}}(S=1)\over\rho_{S}\pi_{A}(A|\bm{X},1)}&\times&d(\bm{X})A[Y-Q(\bm{X},A)]&+&{{\mathbbm{1}}(S=0)\over
1-\rho_{S}}[C(\bm{X})d(\bm{X})-\theta_{1}].\end{array}$

Here, $\mathrm{EIF}^{(k)}$ represents the EIF based on $\mathcal{O}^{(k)}$ for
$k=1,2$.

Theorem 3 can be connected to existing literature. Rudolph and van der Laan
(2017, Section 4) obtained the EIF for $\theta_{1}$ on the Type-2 data
$\mathcal{O}^{(2)}$. Dahabreh et al. (2019, Appendix D) first obtained the EIF
for $\theta$ based on the Type-2 data $\mathcal{O}^{(2)}$. Then they further
argued that if Assumption 2 is relaxed to that
$Y_{i}(a)\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}A_{i}|(\bm{X}_{i}=\bm{x},S_{i}=1)$ holds on
the training data only, then the observed treatment assignments and outcomes
$\\{(A_{i},Y_{i}):S_{i}=0\\}$ from the calibrating data cannot be used. In
that case, the EIFs based on $\mathcal{O}^{(1)}$ and $\mathcal{O}^{(2)}$ are
the same. Uehara et al. (2020, Theorem 11) obtained the EIF for $\theta$ on
the Type-2 data $\mathcal{O}^{(2)}$ using a stratified sampling formulation
for $\bm{O}_{i}|(S_{i}=1)$ and $\bm{O}_{i}|(S_{i}=0)$, and treating
$w(\bm{X}_{i})$ as a general density ratio function of covariates. Finally, we
would like to point out that Kallus (2020, Lemma 5.1) considered the EIF for a
fixed weight function, while the weight function $w(\bm{x})$ as in Lemma 1 is
model-endogeneous in the sense that it corresponds to the conditional-on-
covariate selection odds or the covariate density ratio in the semiparametric
model. The EIF in Kallus (2020, Lemma 5.1) assumes that $w(\bm{x})$ is known
or the associated nuisance function $\pi_{S}(s|\bm{x},a)$ is known, while the
EIFs from our Theorem 3 account for additional variances contributed by
estimating $w(\bm{x})$.

Suppose $w(\bm{x})$, $\pi_{A}(a|\bm{x},s)$ and $Q(\bm{x},a)$ are known as
oracle. Recall that $C(\bm{x})=Q(\bm{x},1)-Q(\bm{x},-1)$. Denote
$n_{s}:=\\#\\{i:S_{i}=s\\}$ for $s\in\\{1,0\\}$. Then we have $n=n_{1}+n_{0}$
and $n_{1}/n\to\rho_{S}$ as $n\to\infty$. Theorem 3 can imply the following
oracle semiparametric efficient estimates of $\theta$ and $\theta_{1}$ based
on $\mathcal{O}^{(1)}$ and $\mathcal{O}^{(2)}$ respectively:

$\displaystyle\widehat{\theta}_{\rm eff}^{(1)}=\quad$ $\displaystyle{1\over
n_{1}}\sum_{i:S_{i}=1}{n_{1}\over
n}w(\bm{X}_{i}){{\mathbbm{1}}[d(\bm{X}_{i})=A_{i}]\over\pi_{A}(A_{i}|\bm{X}_{i},1)}[Y_{i}-Q(\bm{X}_{i},A_{i})]$
$\displaystyle(:=\phi_{n_{1}}^{(1)})$ $\displaystyle+$ $\displaystyle{1\over
n_{0}}\sum_{i:S_{i}=0}\left\\{{n_{0}\over
n}{{\mathbbm{1}}[d(\bm{X}_{i})=A_{i}]\over\pi_{A}(A_{i}|\bm{X}_{i},0)}[Y_{i}-Q(\bm{X}_{i},A_{i})]+Q(\bm{X}_{i},d)\right\\}$
$\displaystyle(:=\psi_{n_{0}}^{(1)}+\theta);$
$\displaystyle\widehat{\theta}_{\rm eff}^{(2)}=\quad$ $\displaystyle{1\over
n_{1}}\sum_{i:S_{i}=1}w(\bm{X}_{i}){{\mathbbm{1}}[d(\bm{X}_{i})=A_{i}]\over\pi_{A}(A|\bm{X},1)}[Y_{i}-Q(\bm{X}_{i},A_{i})]$
$\displaystyle(:=\phi_{n_{1}}^{(2)})$ $\displaystyle+$ $\displaystyle{1\over
n_{0}}\sum_{i:S_{i}=0}Q(\bm{X}_{i},d)$
$\displaystyle(:=\psi_{n_{0}}^{(2)}+\theta);$
$\displaystyle\widehat{\theta}_{1,\rm eff}^{(1)}=\quad$ $\displaystyle{1\over
n_{1}}\sum_{i:S_{i}=1}{n_{1}\over
n}w(\bm{X}_{i}){d(\bm{X}_{i})A_{i}\over\pi_{A}(A_{i}|\bm{X}_{i},1)}[Y_{i}-Q(\bm{X}_{i},A_{i})]$
$\displaystyle(:=\phi_{1,n_{1}}^{(1)})$ $\displaystyle+$ $\displaystyle{1\over
n_{0}}\sum_{i:S_{i}=0}\left\\{{n_{0}\over
n}{{\mathbbm{1}}[d(\bm{X}_{i})=A_{i}]\over\pi_{A}(A_{i}|\bm{X}_{i},0)}[Y_{i}-Q(\bm{X}_{i},A_{i})]+C(\bm{X}_{i})d(\bm{X}_{i})\right\\}$
$\displaystyle(:=\psi_{1,n_{0}}^{(1)}+\theta_{1});$
$\displaystyle\widehat{\theta}_{1,\rm eff}^{(2)}=\quad$ $\displaystyle{1\over
n_{1}}\sum_{i:S_{i}=1}w(\bm{X}_{i}){d(\bm{X}_{i})A_{i}\over\pi_{A}(A_{i}|\bm{X}_{i},1)}[Y_{i}-Q(\bm{X}_{i},A_{i})]$
$\displaystyle(:=\phi_{1,n_{1}}^{(2)})$ $\displaystyle+$ $\displaystyle{1\over
n_{0}}\sum_{i:S_{i}=0}C(\bm{X}_{i})d(\bm{X}_{i})$
$\displaystyle(:=\psi_{1,n_{0}}^{(2)}+\theta_{1}).$ (2)

Here, we use $n_{1}/n$ and $n_{0}/n$ to replace $\rho_{S}$ and $1-\rho_{S}$
respectively. The asymptotic variances of the oracle semiparametric estimates
in (2) are the smallest among all regular and asymptotic linear (RAL)
estimates (Tsiatis, 2007), which we establish in the following Theorem 4. For
$\bm{x}\in\mathcal{X}$, $s\in\\{1,0\\}$ and $a\in\\{1,-1\\}$, we denote
$\sigma^{2}(\bm{x},s,a):=\mathsf{Var}[Y_{i}(a)|\bm{X}_{i}=\bm{x},S_{i}=s]$,
$\sigma^{2}(\bm{x},s,d):=\sum_{a\in\\{1,-1\\}}\sigma^{2}(\bm{x},s,a){\mathbbm{1}}[d(\bm{x}_{i})=a]$
and $\pi_{A}(d|\bm{x},s):=\sum_{a\in\\{1,-1\\}}\pi_{A}(a|\bm{x},s)$. In order
to obtain asymptotic results for all these estimates, we make the following
integrability assumption.

######  Assumption 6 (Squared Integrability).

Assume that

$\mathbb{E}[Q(\bm{X}_{i},1)^{2}],~{}\mathbb{E}[Q(\bm{X}_{i},-1)^{2}]<+\infty;\quad\sup_{\bm{x}\in\mathcal{X},s\in\\{1,0\\},a\in\\{1,-1\\}}\sigma^{2}(\bm{x},s,a)<+\infty.$

######  Theorem 4 (Semiparametric Efficiency).

Consider the observed data $\mathcal{O}^{(1)}$ and $\mathcal{O}^{(2)}$, the
parameters $\theta$ and $\theta_{1}$ in (1), and the corresponding oracle
efficient estimates in (2). Under Assumptions 1-6, we have

$\displaystyle\begin{array}[]{lllll}\sqrt{n_{1}}\phi_{n_{1}}^{(1)}\stackrel{{\scriptstyle
n_{1}\to\infty}}{{\Longrightarrow}}Z_{1}^{(1)};&\sqrt{n_{0}}\psi_{n_{0}}^{(1)}\stackrel{{\scriptstyle
n_{0}\to\infty}}{{\Longrightarrow}}Z_{0}^{(1)};&Z_{1}^{(1)}\sim\mathcal{N}(0,\nu_{\rm
eff}^{(1)});&Z_{0}^{(1)}\sim\mathcal{N}(0,\zeta_{\rm
eff}^{(1)});&Z_{1}^{(1)}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}Z_{0}^{(1)};\\\
\sqrt{n_{1}}\phi_{n_{1}}^{(2)}\stackrel{{\scriptstyle
n_{1}\to\infty}}{{\Longrightarrow}}Z_{1}^{(2)};&\sqrt{n_{0}}\psi_{n_{0}}^{(2)}\stackrel{{\scriptstyle
n_{0}\to\infty}}{{\Longrightarrow}}Z_{0}^{(2)};&Z_{1}^{(2)}\sim\mathcal{N}(0,\nu_{\rm
eff}^{(2)});&Z_{0}^{(2)}\sim\mathcal{N}(0,\zeta_{\rm
eff}^{(2)});&Z_{1}^{(2)}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}Z_{0}^{(2)};\\\
\sqrt{n_{1}}\phi_{1,n_{1}}^{(1)}\stackrel{{\scriptstyle
n_{1}\to\infty}}{{\Longrightarrow}}Z_{11}^{(1)};&\sqrt{n_{0}}\psi_{1,n_{0}}^{(1)}\stackrel{{\scriptstyle
n_{0}\to\infty}}{{\Longrightarrow}}Z_{10}^{(1)};&Z_{11}^{(1)}\sim\mathcal{N}(0,\nu_{1,\rm
eff}^{(1)});&Z_{10}^{(1)}\sim\mathcal{N}(0,\zeta_{1,\rm
eff}^{(1)});&Z_{11}^{(1)}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}Z_{10}^{(1)};\\\
\sqrt{n_{1}}\phi_{1,n_{1}}^{(2)}\stackrel{{\scriptstyle
n_{1}\to\infty}}{{\Longrightarrow}}Z_{11}^{(2)};&\sqrt{n_{0}}\psi_{1,n_{0}}^{(2)}\stackrel{{\scriptstyle
n_{0}\to\infty}}{{\Longrightarrow}}Z_{10}^{(2)};&Z_{11}^{(2)}\sim\mathcal{N}(0,\nu_{1,\rm
eff}^{(2)});&Z_{10}^{(2)}\sim\mathcal{N}(0,\zeta_{1,\rm
eff}^{(2)});&Z_{11}^{(2)}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}Z_{10}^{(2)},\end{array}$

where

$\displaystyle\begin{array}[]{rlrl}\nu_{\rm
eff}^{(1)}&=\rho_{S}^{2}\mathbb{E}\left\\{w(\bm{X}_{i})^{2}{\sigma^{2}(\bm{X}_{i},1,d)\over\pi_{A}(d|\bm{X}_{i},1)}\middle|S_{i}=1\right\\};\\\
\zeta_{\rm
eff}^{(1)}&=(1-\rho_{S})^{2}\mathbb{E}\left\\{{\sigma^{2}(\bm{X}_{i},0,d)\over\pi_{A}(d|\bm{X}_{i},0)}\middle|S_{i}=0\right\\}+\mathsf{Var}[Q(\bm{X}_{i},d)|S_{i}=0];\\\
\nu_{\rm
eff}^{(2)}&=\mathbb{E}\left\\{w(\bm{X}_{i})^{2}{\sigma^{2}(\bm{X}_{i},1,d)\over\pi_{A}(d|\bm{X}_{i},1)}\middle|S_{i}=1\right\\};\\\
\zeta_{\rm eff}^{(2)}&=\mathsf{Var}[Q(\bm{X}_{i},d)|S_{i}=0];\\\ \nu_{1,\rm
eff}^{(1)}&=\rho_{S}^{2}\mathbb{E}\left\\{w(\bm{X}_{i})^{2}\sum_{a\in\\{1,-1\\}}{\sigma^{2}(\bm{X}_{i},1,a)\over\pi_{A}(a|\bm{X}_{i},1)}\middle|S_{i}=1\right\\};\\\
\zeta_{1,\rm
eff}^{(1)}&=(1-\rho_{S})^{2}\mathbb{E}\left\\{\sum_{a\in\\{1,-1\\}}{\sigma^{2}(\bm{X}_{i},0,a)\over\pi_{A}(a|\bm{X}_{i},0)}\middle|S_{i}=0\right\\}+\mathsf{Var}[C(\bm{X}_{i})d(\bm{X}_{i})|S_{i}=0];\\\
\nu_{1,\rm
eff}^{(2)}&=\mathbb{E}\left\\{w(\bm{X}_{i})^{2}\sum_{a\in\\{1,-1\\}}{\sigma^{2}(\bm{X}_{i},1,a)\over\pi_{A}(a|\bm{X}_{i},1)}\middle|S_{i}=1\right\\};\\\
\zeta_{1,\rm
eff}^{(2)}&=\mathsf{Var}[C(\bm{X}_{i})d(\bm{X}_{i})|S_{i}=0].\end{array}$

Moreover, for $\\{\alpha_{n}\\}$ such that
$\alpha_{n}/\sqrt{n_{1}}\to\gamma_{1}$ and
$\alpha_{n}/\sqrt{n_{0}}\to\gamma_{0}$ as $n\to\infty$, we have

$\displaystyle\begin{array}[]{rll}\alpha_{n}(\widehat{\theta}_{\rm
eff}^{(1)}-\theta)&=(\gamma_{1}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}(1))\times\sqrt{n_{1}}\phi_{n_{1}}^{(1)}+(\gamma_{0}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}(1))\times\sqrt{n_{0}}\psi_{n_{0}}^{(1)}&\stackrel{{\scriptstyle
n_{1},n_{0}\to\infty}}{{\Longrightarrow}}\gamma_{1}Z_{1}^{(1)}+\gamma_{0}Z_{0}^{(1)};\\\
\alpha_{n}(\widehat{\theta}_{\rm
eff}^{(2)}-\theta)&=(\gamma_{1}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}(1))\times\sqrt{n_{1}}\phi_{n_{1}}^{(2)}+(\gamma_{0}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}(1))\times\sqrt{n_{0}}\psi_{n_{0}}^{(2)}&\stackrel{{\scriptstyle
n_{1},n_{0}\to\infty}}{{\Longrightarrow}}\gamma_{1}Z_{1}^{(2)}+\gamma_{0}Z_{0}^{(2)};\\\
\alpha_{n}(\widehat{\theta}_{1,\rm
eff}^{(1)}-\theta_{1})&=(\gamma_{1}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}(1))\times\sqrt{n_{1}}\phi_{1,n_{1}}^{(1)}+(\gamma_{0}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}(1))\times\sqrt{n_{0}}\psi_{1,n_{0}}^{(1)}&\stackrel{{\scriptstyle
n_{1},n_{0}\to\infty}}{{\Longrightarrow}}\gamma_{1}Z_{11}^{(1)}+\gamma_{0}Z_{10}^{(1)};\\\
\alpha_{n}(\widehat{\theta}_{1,\rm
eff}^{(1)}-\theta_{1})&=(\gamma_{1}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}(1))\times\sqrt{n_{1}}\phi_{1,n_{1}}^{(2)}+(\gamma_{0}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}(1))\times\sqrt{n_{0}}\psi_{1,n_{0}}^{(2)}&\stackrel{{\scriptstyle
n_{1},n_{0}\to\infty}}{{\Longrightarrow}}\gamma_{1}Z_{11}^{(2)}+\gamma_{0}Z_{10}^{(2)}.\end{array}$

In particular, for $\alpha_{n}=\sqrt{n}$ (resp. $\sqrt{n_{1}}$ or
$\sqrt{n_{0}}$), the corresponding $\sqrt{n}$ (resp. $\sqrt{n_{1}}$ or
$\sqrt{n_{0}}$) asymptotic variances achieve the semiparametric
$\sqrt{n}$-(resp. $\sqrt{n_{1}}$-or $\sqrt{n_{0}}$-)variance lower bounds for
$\theta$ and $\theta_{1}$ on $\mathcal{O}^{(1)}$ and $\mathcal{O}^{(2)}$
respectively.

We notice that the efficient estimates (2) can depend on the unknown nuisance
functions $w(\bm{x})$, $\pi_{A}(a|\bm{x},s)$ and $Q(\bm{x},a)$. Implementable
efficient estimates of $\theta$ and $\theta_{1}$ are the plug-in versions with
the corresponding sample-dependent nuisance function estimates. Uehara et al.
(2020) took the following cross-fitting strategy (Chernozhukov et al., 2018)
when plugging in the nuisance function estimates: the pooled data are
stratified into $\\{i:S_{i}=1\\}$ and $\\{i:S_{i}=0\\}$, and the sample points
within each stratum are randomly divided into $K$ bags. For
$k\in\\{1,2,\cdots,K\\}$, we obtain out-of-bag estimates of the nuisance
functions from the pooled dataset that rules out the $k$-th bag of data
points. When constructing the cross-fitting estimates (2) of $\theta$ and
$\theta_{1}$, if the $i$-th data point belongs to the $k$-th bag, then it
utilizes the out-of-$k$-th-bag nuisance function estimates. Using the cross-
fitting strategy, we can follow Uehara et al. (2020, Theorem 2) and establish
$\sqrt{n}$-equivalences for plug-in efficient estimates. In the following
Theorem 5, we only consider the cross-fitting estimates for $\theta$, and the
same argument can be applied to $\theta_{1}$.

######  Theorem 5 ($\sqrt{n}$-Equivalence).

Consider the observed data $\mathcal{O}^{(1)}$ and $\mathcal{O}^{(2)}$ and the
parameter $\theta$ in (1). Denote $\Phi^{(1)}(\eta):=\widehat{\theta}_{\rm
eff}^{(1)}-\theta$ and $\Phi^{(2)}(\eta)=\widehat{\theta}_{\rm
eff}^{(2)}-\theta$ from (2) with
$\eta=\big{(}w(\bm{x}),\pi_{A}(a|\bm{x},s),Q(\bm{x},a)\big{)}$ as the nuisance
functions. For $k\in\\{1,2,\cdots,K\\}$, let $\widehat{\eta}_{\rm
cross}:=\\{(\widehat{w}^{(k)},\widehat{\pi}_{A}^{(k)},\widehat{Q}^{(k)})\\}_{k=1}^{K}$
be the out-of-bag nuisance function estimates, and
$\Phi^{(1)}(\widehat{\eta}_{\rm cross})$ and $\Phi^{(2)}(\widehat{\eta}_{\rm
cross})$ be the corresponding cross-fitting versions. Define

$\displaystyle\alpha_{n}^{(2)}$
$\displaystyle:=\max_{a\in\\{1,-1\\}}\max_{1\leq k\leq
K}\left\|{\widehat{w}^{(k)}(\cdot)\over\widehat{\pi}_{A}^{(k)}(a|\cdot,1)}-{w(\cdot)\over\pi_{A}(a|\cdot,1)}\right\|_{L^{2}(\mathbb{P})};$
$\displaystyle\alpha_{n}^{(1)}$ $\displaystyle:=\alpha_{n}^{(2)}+\max_{1\leq
k\leq
K}\left\|\widehat{\pi}_{A}^{(k)}(1|\cdot,0)-\pi_{A}(1|\cdot,0)\right\|_{L^{2}(\mathbb{P})};$
$\displaystyle\beta_{n}$ $\displaystyle:=\max_{a\in\\{1,-1\\}}\max_{1\leq
k\leq
K}\left\|\widehat{Q}^{(k)}(\cdot,a)-Q(\cdot,a)\right\|_{L^{2}(\mathbb{P})},$

where $\|\cdot\|_{L^{2}(\mathbb{P})}$ is the $L^{2}(\mathbb{P})$-norm with
respect to the covariate vector $\bm{X}$. Then under Assumptions 1-6 and that
$\alpha_{n}^{(1)},\alpha_{n}^{(2)},\beta_{n}=\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}_{\mathbb{P}}(1)$,
we have

$\sqrt{n}[\Phi^{(1)}(\widehat{\eta}_{\rm
cross})-\Phi^{(1)}(\eta)]=\mathcal{O}_{\mathbb{P}}(\sqrt{n}\alpha_{n}^{(1)}\beta_{n});\quad\sqrt{n}[\Phi^{(2)}(\widehat{\eta}_{\rm
cross})-\Phi^{(2)}(\eta)]=\mathcal{O}_{\mathbb{P}}(\sqrt{n}\alpha_{n}^{(2)}\beta_{n}).$

######  Remark 1.

Notice that in Theorem 5, by the fact that $\pi_{A}(a|\bm{x},s)\geq\tau$, we
further have

$\alpha_{n}^{(2)}\leq\text{constant}\times\max_{1\leq k\leq
K}\left\|\widehat{w}^{(k)}-w\right\|_{L^{2}(\mathbb{P})}+\text{constant}\times\max_{1\leq
k\leq
K}\left\|\widehat{\pi}_{A}^{(k)}(1|\cdot,1)-\pi_{A}(1|\cdot,1)\right\|_{L^{2}(\mathbb{P})}.$

Then a dominating rate of $\alpha_{n}^{(2)}$ can be chosen as the slower one
of $\widehat{w}(\cdot)$ and $\widehat{\pi}_{A}(1|\cdot,1)$.

One important implication of the remainder terms in Theorem 5 is that
$\alpha_{n}^{(1)}$, $\alpha_{n}^{(2)}$ and $\beta_{n}$ can be in slower orders
than the usual requirement
$\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}_{\mathbb{P}}(n^{-1/2})$
for negligibility. For example, if
$\alpha_{n}^{(1)}=\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}_{\mathbb{P}}(n^{-1/4})$
and
$\beta_{n}=\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}_{\mathbb{P}}(n^{-1/4})$,
then the plug-in effect for $\Phi^{(1)}(\widehat{\eta}_{\rm cross})$ is
negligible.

Finally, we notice that the $\sqrt{n}$-asymptotic results in Theorem 4 rely on
the implication from Assumption 5 that $n_{1}/n\to\rho_{S}\geq\delta$ and
$n_{0}/n\to 1-\rho_{S}\geq\delta$ as $n\to\infty$, so that both $n_{1}$ and
$n_{0}$ grow linearly in $n$. When the calibrating sample size $n_{0}$ is of a
smaller order in the training sample size $n_{1}$, the leading order terms for
the estimates in (2) are of order $\sqrt{n_{0}}$. We establish the
corresponding asymptotic results in the following Corollary 6 that are
different from those in Theorem 4.

######  Corollary 6 (Semiparametric Efficiency with Small Calibrating Data).

Consider the observed data $\mathcal{O}^{(1)}$ and $\mathcal{O}^{(2)}$, the
parameters $\theta$ and $\theta_{1}$ in (1), and the corresponding oracle
efficient estimates in (2). Suppose $n_{0}/n_{1}\to 0$ as
$n_{1},n_{0}\to\infty$. Then under Assumptions 1-4, 6 and that
$\sup_{\bm{x}\in\mathcal{X}}w(\bm{x})<+\infty$, we have

$\displaystyle\begin{array}[]{c}\sqrt{n_{0}}(\widehat{\theta}_{\rm
eff}^{(2)}-\theta)\underset{n_{0}\to\infty}{\stackrel{{\scriptstyle\mathcal{D}}}{{\longrightarrow}}}\mathcal{N}(0,\mathsf{Var}[Q(\bm{X}_{i},d)|S_{i}=0]);\quad\sqrt{n_{0}}(\widehat{\theta}_{\rm
eff}^{(2)}-\theta)\underset{n_{0}\to\infty}{\stackrel{{\scriptstyle\mathcal{D}}}{{\longrightarrow}}}\mathcal{N}(0,\mathsf{Var}[Q(\bm{X}_{i},d)|S_{i}=0]);\\\
\sqrt{n_{0}}(\widehat{\theta}_{1,\rm
eff}^{(1)}-\theta_{1})\underset{n_{0}\to\infty}{\stackrel{{\scriptstyle\mathcal{D}}}{{\longrightarrow}}}\mathcal{N}(0,\mathsf{Var}[C(\bm{X}_{i})d(\bm{X}_{i})|S_{i}=0]);\quad\sqrt{n_{0}}(\widehat{\theta}_{1,\rm
eff}^{(2)}-\theta_{1})\underset{n_{0}\to\infty}{\stackrel{{\scriptstyle\mathcal{D}}}{{\longrightarrow}}}\mathcal{N}(0,\mathsf{Var}[C(\bm{X}_{i})d(\bm{X}_{i})|S_{i}=0]),\end{array}$

which attain the corresponding semiparametric $\sqrt{n_{0}}$-asymptotic
variance lower bounds for $\theta$ and $\theta_{1}$ respectively.

Notice that the $\sqrt{n_{0}}$-asymptotic variances from Corollary 6 are the
same as ${1\over n_{0}}\sum_{i:S_{i}=0}Q(\bm{X}_{i},d)$ and ${1\over
n_{0}}\sum_{i:S_{i}=0}C(\bm{X}_{i})d(\bm{X}_{i})$ respectively, so that all
efficiency augmentation terms in (2) are negligible with respect to the
$\sqrt{n_{0}}$-order. Moreover, any estimate $\widehat{Q}(\bm{x},a)$ of
$Q(\bm{x},a)$ that satisfies
$\|\widehat{Q}(\cdot,a)-Q(\cdot,a)\|_{L^{2}(\mathbb{P})}=\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}_{\mathbb{P}}(n_{0}^{-1/2})$
for $a\in\\{1,-1\\}$ will not affect these $\sqrt{n_{0}}$-asymptotic results.
Since we have assumed that $n_{0}/n_{1}\to 0$, it can be relatively easier
than Theorem 5 to obtain an estimate $\widehat{Q}$ of order
$\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}_{\mathbb{P}}(n_{0}^{-1/2})$
based on the training data.

###  2.2 Estimating Weights

In Section 2.1, we obtain the efficient estimates for $\theta$ and
$\theta_{1}$ on two types of pooled data $\mathcal{O}^{(1)}$ and
$\mathcal{O}^{(2)}$ as in (2). We also establish the asymptotic properties in
Theorems 4 and 5 for the oracle and cross-fitting efficient estimates
respectively. In this section, we discuss different methods from literature
for estimating training sample weights $\\{w(\bm{X}_{i}):S_{i}=1\\}$ or the
nuisance function $\bm{x}\mapsto w(\bm{x})$. In particular, Augmented Inverse
Probability of Sampling Weight (AIPSW) and density ratio estimation are
related to the discussions in Li et al. (2020) and Liang and Zhao (2020)
respectively.

#### Augmented Inverse Probability of Sampling Weight (AIPSW)

Let $\widehat{\pi}_{S}(1|\bm{x})$ be an estimate of $\pi_{S}(1|\bm{x})$.
Define the estimated weights at training data points $\\{i:S_{i}=1\\}$ as

$\widehat{w}(\bm{X}_{i}):={n_{1}\widehat{\pi}_{S}(0|\bm{X}_{i})\over
n_{0}\widehat{\pi}_{S}(1|\bm{X}_{i})};\quad\forall i:S_{i}=1.$

Then the AIPSW estimates (Stuart et al., 2011; Buchanan et al., 2018) are
defined as in (2) with the AIPSWs $\\{\widehat{w}(\bm{X}_{i}):S_{i}=1\\}$ that
are obtained from the estimated conditional-on-covariate sampling probability
function $\widehat{\pi}_{S}(s|\bm{x})$. We further denote the AIPSW estimates
as $\widehat{\theta}_{\rm AIPSW}^{(1)}$, $\widehat{\theta}_{\rm AIPSW}^{(2)}$,
$\widehat{\theta}_{1,\rm AIPSW}^{(1)}$ and $\widehat{\theta}_{1,\rm
AIPSW}^{(2)}$ respectively. Notice that the AIPSW estimates have one-to-one
correspondence to the estimators proposed in Li et al. (2020, Section 3.2):
$\widehat{\theta}_{\rm AIPSW}^{(1)}=\widehat{V}_{\rm eff}^{*}(d)$,
$\widehat{\theta}_{\rm AIPSW}^{(2)}=\widehat{V}_{\rm onlyX}^{*}(d)$,
$\widehat{\theta}_{1,\rm AIPSW}^{(1)}=\widehat{R}_{\rm eff}^{*}(d)$, and
$\widehat{\theta}_{1,\rm AIPSW}^{(2)}=\widehat{R}_{\rm onlyX}^{*}(d)$.

In practice, $\widehat{\pi}_{S}(s|\bm{x})$ can be estimated by logistic
regression of $S_{i}$ on $\bm{X}_{i}$. Consider the simulation setup in Li et
al. (2020):
$\bm{X}_{i}|(S_{i}=1)\sim\mathcal{N}_{p}({\mathbf{0}},\mathbf{I}_{p})$ and
$\bm{X}_{i}|(S_{i}=0)\sim\mathcal{N}_{p}(\bm{\mu},\mathbf{I}_{p})$ for some
$\bm{\mu}\in\mathbb{R}^{p}$. For $s\in\\{1,0\\}$, denote $f(\bm{x}|s)$ as the
density of $\bm{X}_{i}|(S_{i}=s)$. The theoretical weight function in this
case is $w(\bm{x})={f_{X}(\bm{x}|0)\over
f_{X}(\bm{x}|1)}=\exp(\|\bm{\mu}\|_{2}^{2}/2-\bm{\mu}^{\intercal}\bm{x})$. The
corresponding log-odds of $S_{i}|(\bm{X}_{i}=\bm{x})$ is:
$\log\left({\pi_{S}(1|\bm{x})\over\pi_{S}(0|\bm{x})}\right)=\log\left({\rho_{S}\over
1-\rho_{S}}\right)-\log[w(\bm{x})]=\log\left({\rho_{S}\over
1-\rho_{S}}\right)-\|\bm{\mu}\|_{2}^{2}/2+\bm{\mu}^{\intercal}\bm{x}$. In this
case, logistic regression can correctly specify $S_{i}|\bm{X}_{i}$. Therefore,
the corresponding AIPSW estimates can enjoy semiparametric efficiency, which
was illustrated in the numerical studies of Li et al. (2020, Section 3.3).

#### Density Ratio Estimation

Instead of estimating $\pi_{S}(s|\bm{x})$ first to obtain
$w(\bm{x})={\rho_{S}\over
1-\rho_{S}}{\pi_{S}(0|\bm{x})\over\pi_{S}(1|\bm{x})}$, we can directly
consider $w(\bm{x})$ as the covariate density ratio function
${\mathbb{P}(\bm{X}_{i}=\bm{x}|S_{i}=0)\over\mathbb{P}(\bm{X}_{i}=\bm{x}|S_{i}=1)}$.
Uehara et al. (2020) proposed to estimate the covariate density ratio for
$\\{\bm{X}_{i}:S_{i}=1\\}$ using the Kernel-based unconstrained Least-Squares
Importance Fitting (KuLSIF) (Kanamori et al., 2012), which was also discussed
in Liang and Zhao (2020). Specifically, let $\mathcal{G}$ be a generic
function class, and consider the following least-squared problem:

$\min_{g\in\mathcal{G}}{1\over
2}\mathbb{E}\Big{\\{}[g(\bm{X}_{i})-w(\bm{X}_{i})]^{2}\Big{|}S_{i}=1\Big{\\}}=\min_{g\in\mathcal{G}}\left\\{{1\over
2}\mathbb{E}[g(\bm{X}_{i})^{2}|S_{i}=1]-\mathbb{E}[g(\bm{X}_{i})|S_{i}=0]\right\\}+{1\over
2}\mathbb{E}[w(\bm{X}_{i})^{2}|S_{i}=1].$

The empirical version of the above least-squared problem is as follows:

$\displaystyle\min_{g\in\mathcal{G}}\left\\{{1\over
2n_{1}}\sum_{i:S_{i}=1}g(\bm{X}_{i})^{2}-{1\over
n_{0}}\sum_{i:S_{i}=0}g(\bm{X}_{i})+{\lambda\over
2}\|g\|_{\mathcal{G}}^{2}\right\\},$ (3)

where $(\lambda/2)\|g\|_{\mathcal{G}}^{2}$ is a functional penalty on
$\mathcal{G}$. Let $K$ be a positive semi-definite kernel function on
$\mathcal{X}$ and $\mathcal{G}=\mathcal{H}_{K}$ be the corresponding
reproducing kernel Hilbert space (RKHS). Define
$\mathbf{K}_{11}:=[K(\bm{X}_{i},\bm{X}_{j}):S_{i}=S_{j}=1]$ and
$\mathbf{K}_{01}:=[K(\bm{X}_{i},\bm{X}_{j}):S_{i}=0,S_{j}=1]$. The solution to
(3) can be represented as
$g_{\bm{\alpha}}(\cdot)=\sum_{i:S_{i}=1}\alpha_{i}K(\bm{X}_{i},\cdot)+[1/(\lambda
n_{0})]\sum_{i:S_{i}=0}K(\bm{X}_{i},\cdot)$ (Kanamori et al., 2012, Theorem
1), and the dual optimization problem for
$\bm{\alpha}=(\alpha_{i}:S_{i}=1)^{\intercal}\in\mathbb{R}^{n_{1}}$ is:

$\min_{\bm{\alpha}\in\mathbb{R}^{n_{1}}}\left\\{{1\over
2}\bm{\alpha}^{\intercal}\left({1\over
n_{1}}\mathbf{K}_{11}+\lambda\mathbf{I}_{n_{1}}\right)\bm{\alpha}-{1\over\lambda
n_{0}n_{1}}{\mathbf{1}}_{n_{0}}^{\intercal}\mathbf{K}_{01}\bm{\alpha}\right\\}.$

Let $\widehat{w}(\bm{x})$ be the KuLSIF estimate of the covariate weight
function. Under certain conditions, Kanamori et al. (2012, Theorem 2) showed
that
$\|\widehat{w}-w\|_{L^{2}(\mathbb{P})}=\mathcal{O}_{\mathbb{P}}\left((n_{1}\wedge
n_{0})^{-1/(2+\gamma)}\right)$ for some $\gamma\in(0,2)$.

Based on the KuLSIF weights, Liang and Zhao (2020) considered
$\widehat{\theta}_{1}:=(1/n_{1})\sum_{i:S_{i}=1}\widehat{w}(\bm{X}_{i})\widehat{C}(\bm{X}_{i})d(\bm{X}_{i})$.
Under Assumptions 1-6, it requires that
$\|\widehat{w}-w\|_{L^{2}(\mathbb{P})}=\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}_{\mathbb{P}}(n^{-1/2})$
and
$\|\widehat{C}-C\|_{L^{2}(\mathbb{P})}=\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}_{\mathbb{P}}(n^{-1/2})$
to establish the $\sqrt{n}$-consistency of $\widehat{\theta}_{1}$. However,
the KuLSIF estimate of $w(\bm{x})$ cannot satisfy this rate condition. Even
when
$\|\widehat{w}-w\|_{L^{2}(\mathbb{P})}=\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}_{\mathbb{P}}(n^{-1/2})$
and
$\|\widehat{C}-C\|_{L^{2}(\mathbb{P})}=\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}_{\mathbb{P}}(n^{-1/2})$
so that $\widehat{\theta}_{1}$ is $\sqrt{n}$-consistent, the asymptotic
variance of $\widehat{\theta}_{1}$ is generally greater than the semi-
parametric efficiency bound in Theorem 4. In order to achieve the efficiency
bound, the KuLSIF density ratio estimate should be combined with a
semiparametric efficient estimate and the cross-fitting strategy.

#### Augmented Calibration Weight (ACW)

Let $\mathcal{G}$ be a function class on $\mathcal{X}$. Lemma 1 can also
motivate the following covariate balancing conditions among covariate
functions in $\mathcal{G}$ for the training sample weights
$\\{W_{i}:S_{i}=1\\}$ (Imai and Ratkovic, 2014):

$\displaystyle\underbrace{\sum_{i:S_{i}=1}W_{i}g(\bm{X}_{i})}_{\text{empirical
version of
$\mathbb{E}[w(\bm{X}_{i})g(\bm{X}_{i})|S_{i}=1]$}}=\underbrace{{1\over
n_{0}}\sum_{i:S_{i}=0}g(\bm{X}_{i})}_{\text{empirical version of
$\mathbb{E}[g(\bm{X}_{i})|S_{i}=0]$}};\quad\forall g\in\mathcal{G}.$ (4)

Based on the balancing conditions (4), Hainmueller (2012) proposed to solve
the entropy balancing problem for calibration weights:

$\displaystyle\begin{array}[]{cll}\displaystyle\min_{W_{i}:S_{i}=1}&\displaystyle\sum_{i:S_{i}=1}W_{i}\log
W_{i},\\\ \text{subject to}&W_{i}\geq 0;&\text{for all $i$ with $S_{i}=1$};\\\
&\displaystyle\sum_{i:S_{i}=1}W_{i}=1;\\\
&\displaystyle\sum_{i:S_{i}=1}W_{i}g(\bm{X}_{i})={1\over
n_{0}}\sum_{i:S_{i}=0}g(\bm{X}_{i});&\forall g\in\mathcal{G}.\end{array}$ (9)

Here, the objective function is the empirical Kullback–Leibler divergence of
$\mathbb{P}(\cdot|S_{i}=0)$ with respect to $\mathbb{P}(\cdot|S_{i}=1)$, and
$W_{i}$ is the covariate density ratio at $\bm{X}_{i}$. Therefore, the
optimization problem (9) seeks for the balancing weights
$\\{\widehat{W}_{i}:S_{i}=1\\}$ that satisfies (4) and minimizes the
discrepancy of the testing covariate distribution from training. If
$\mathcal{G}=\\{g_{j}\\}_{j=1}^{m}$ and denote
$\bm{g}=(g_{1},g_{2},\cdots,g_{m})^{\intercal}$ as an $m$-dimensional vector
of instrumental functions, then the calibration weights can be solved by the
following dual problem:

$\widehat{W}_{i}={\exp[\widehat{\bm{\lambda}}^{\intercal}\bm{g}(\bm{X}_{i})]\over\sum_{j:S_{j}=1}\exp[\widehat{\bm{\lambda}}^{\intercal}\bm{g}(\bm{X}_{j})]};\quad\text{for
all $i$ with $S_{i}=1$},$

where $\widehat{\bm{\lambda}}\in\mathbb{R}^{m}$ is solved by the balancing
equations (4). The dual solution can be further extended to the case when
$\mathcal{G}$ is an RKHS (Zhao, 2019). The calibration weights from (9) can
also correspond to a parametric model for $w(\bm{X}_{i})$. Specifically, if
$w(\bm{x};\bm{\eta})=\exp[\bm{\eta}^{\intercal}\bm{g}(\bm{x})]$, then
$\widehat{\bm{\lambda}}\stackrel{{\scriptstyle\mathbb{P}}}{{\to}}\bm{\eta}$,
and $\widehat{W}_{i}={w(\bm{X}_{i})\over
n_{1}}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}_{\mathbb{P}}(n^{-1})$
(Dong et al., 2020, Theorem 1). The following two cases can imply such a
parametric model:

  1. (I)

Logistic regression of $S_{i}$ on $\bm{g}(\bm{X}_{i})$ can imply such a
parametric model for $w(\bm{X}_{i})$;

  2. (II)

If $\bm{g}(\bm{X}_{i})$ is a sufficient statistic for $\bm{X}_{i}$ and
$\bm{g}(\bm{X}_{i})|(S_{i}=s)\sim\mathcal{N}_{m}(\bm{\mu}_{s},\Sigma)$ for
$s\in\\{1,0\\}$, then
$w(\bm{x})=\exp\left\\{(\bm{\mu}_{1}-\bm{\mu}_{0})^{\intercal}\left(\bm{g}(\bm{x})-{\bm{\mu}_{1}+\bm{\mu}_{0}\over
2}\right)\right\\}$. Because of the additional term
${\bm{\mu}_{1}+\bm{\mu}_{0}\over 2}$ in $w(\bm{x})$, a constant function is
required in the function class $\mathcal{G}$ for balancing conditions.

Finally, the ACW estimates are defined as the efficient estimates (2) with
$\\{w(\bm{X}_{i}):S_{i}=1\\}$ as $\\{n_{1}\widehat{W}_{i}:S_{i}=1\\}$ from (9)
(Dong et al., 2020). One advantage of such estimates is the implicit
nonparametric function class specification for $w(\bm{x})$.

###  2.3 Challenges for Efficient Policy Evaluation

To summarize, we have studied the policy evaluation problem when a set of
calibrating data from the testing distribution can be used for training. We
establish the efficient policy evaluation results based on the EIFs and
discussed the properties of efficient estimates that can be related to
existing literature and the discussions in Li et al. (2020); Liang and Zhao
(2020). However, when the calibrating sample size $n_{0}$ is small, our
discussion suggests the following two main challenges:

  1. (I)

Efficient policy evaluation requires a sufficiently large calibrating sample
size for useful efficiency gain. Assumption 5 can imply that the asymptotic
sampling rates $n_{1}/n\to\mathbb{P}(S_{i}=1)$ and
$n_{0}/n\to\mathbb{P}(S_{i}=0)$ are both at least $\delta$, which requires
that both the training and calibrating sample sizes $n_{1}$ and $n_{0}$ grow
linearly in $n$. When
$n_{0}=\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.5}{$\scriptscriptstyle\mathcal{O}$}}(n_{1})$,
Corollary 6 suggests that the $\sqrt{n_{0}}$-asymptotic efficient estimates in
(2) are equivalent to averaging the nonparametric estimates of
$Q(\bm{X}_{i},d)$ or $C(\bm{X}_{i})d(\bm{X}_{i})$ over the calibrating data.
The complicated forms in (2) may not be helpful for efficiency improvement in
this case. More importantly, the resulting estimates can be unstable due to
the limited calibrating sample size $n_{0}$;

  2. (II)

Nuisance function estimates can be hard to obtain in efficient policy
evaluation. Specifically, the optimality of efficient estimates requires that
the plug-in nuisance function estimates $\widehat{w}(\bm{x})$,
$\widehat{\pi}_{A}(a|\bm{x},s)$ and $\widehat{Q}(\bm{x},a)$ are
$\sqrt{n}$-negligible as in Theorem 5. However, the challenge for nuisance
function estimation mainly appears in the covariate weight function
$\widehat{w}(\bm{x})$, since its rate of convergence is determined by
$\min\\{n_{1},n_{0}\\}$. The same difficulty can appear when estimating the
conditional-on-covariate selection probability function $\pi_{S}(s|\bm{x})$ in
the AIPSW estimates, where a correctly specified parametric estimate
$\widehat{\pi}_{S}(s|\bm{x})$ is in the $(n_{1}\wedge n_{0})^{1/2}$-order.
Thus, it can be difficult to estimate the nuisance function when $n_{0}$ is
small.

Given the challenges of efficient policy evaluation with a limited calibrating
sample size $n_{0}$, DRITR can be less dependent on $n_{0}$. First of all,
DRITR utilizes the training data only to obtain the set of candidates
$\\{\widehat{d}_{c}\\}_{c\in\mathcal{C}}$, where $\mathcal{C}$ is a set of
candidate DR-constants, and each $\widehat{d}_{c}$ enjoys certain robust
performance guarantee. Then calibrating sample is used to choose the final
DRITR. In this way, DRITR is less affected by the size of $n_{0}$ compared to
the combined strategy in Section 2. Secondly, the value function estimates
used in evaluating candidate DRITRs on the calibrating dataset can still enjoy
certain properties with a small $n_{0}$. In Mo et al. (2020, Section 2.4), two
calibration procedures were proposed, one using the calibrating covariates
only, and another one using the calibrating covariates, treatments and
outcomes. In the calibration procedure based on calibrating covariates only,
the testing value function estimate for a given ITR $d$ is ${1\over
n_{0}}\sum_{i:S_{i}=0}\widehat{C}(\bm{X}_{i})d(\bm{X}_{i})$, which is
semiparametric $\sqrt{n_{0}}$-asymptotic efficient due to Corollary 6. For
another calibration procedure based on calibrating covariates, treatments and
outcomes, the testing value function estimate is ${1\over
n_{0}}\sum_{i:S_{i}=0}{{\mathbbm{1}}[d(\bm{X}_{i})=A_{i}]\over\pi_{A}(A_{i}|\bm{X}_{i},1)}Y_{i}$.
Although such an estimate may not achieve the $\sqrt{n_{0}}$-variance lower
bound, it can be robust if Assumption 4 is violated. Specifically, if
Assumption 4 is violated, the estimates (2) may not be consistent, while
${1\over
n_{0}}\sum_{i:S_{i}=0}{{\mathbbm{1}}[d(\bm{X}_{i})=A_{i}]\over\pi_{A}(A_{i}|\bm{X}_{i},1)}Y_{i}$
remains $\sqrt{n_{0}}$-consistent. Thirdly, we would like to point out that
DRITR avoids estimating the covariate weight function $w(\bm{x})$. Therefore,
it can bypass the challenge of nuisance function estimation given the limited
calibrating data.

##  3 Applicability of DRITR

In Section 1, we distinguish DRITR from retargeted policy learning as it
focuses on covariate changes. In Section 2, we consider the problem of
covariate changes with calibrating data from a specific testing distribution
being available at the training stage. In particular, we discuss the general
challenges for efficient policy evaluation when available information from
testing is limited, and how DRITR can avoid such challenges. As is mentioned
at the beginning of Section 2, DRITR focused on general covariate changes
instead of a specific testing distribution. In this section, we discuss the
general applicability of DRITR.

We first emphasize that DRITR aims for protecting scientific discoveries from
the general agnostic covariate changes. This explains why, in response to Li
et al. (2020), we proposed to work with the least favorable case among some
possible covariate changes. In fact, the concern on potential training-testing
distributional changes can be important in modern prediction methodology.
Efron (2020, Section 6) discussed their analysis on the prostate cancer
microarray study. If they randomly split data into training and testing, then
the testing error of a random forest classifier can be as low as 2%. However,
if they selected patients with the lowest ID numbers into the training
dataset, with the remaining for testing purpose, then the testing error would
be as high as 24%. We also performed similar analysis on the ACTG175 study. In
particular, we found that when testing on the female population, several other
existing methods can have poorer performance than DRITR. Such a violation of
the identical training and testing distributions can undermine an existing
scientific finding, and researchers may question the faithfulness of such a
finding when generalizing it to a much broader scope. On the contrary, a
scientific finding robust to all such violations can typically be closer to
universal, eternal truths and become long-lasting (Efron, 2020). The same
scientific principle has also been advocated in Yu and Kumbier (2020);
Bühlmann (2020), both of which established nice connections of such a
principle with adversarial perturbation and distributional robustness.

DRITR can correspond to a more “forgiving” but useful approach than precise
estimation. On one hand, we agree with Dukes and Vansteelandt (2020) that
making correct “causal predictions”, that is, estimating the CTE function
correctly, can be the most robust way of protecting from covariate changes. In
fact, we highlight in our Section 1 that general applicability of DRITR relies
on the assumption
$\\{\bm{x}\mapsto\mathsf{sign}[C(\bm{x})]\\}\nsubseteq\mathcal{D}$. One
example is when the true CTE function $C(\bm{x})$ is too complicated to be
estimable, the ITR class $\mathcal{D}$ can be misspecified. Another example is
that the CTE function $C(\bm{x})$ takes a complicated functional form, and
$\mathcal{D}$ is intended for a more parsimonious class of decision rules in
practice. In either of these two cases, DRITR can be a useful methodology with
tolerance on incorrect “causal predictions”. On the other hand, given our
combined data analysis in Section 2, efficient inference of parameters of
interests can have more restrictive requirements on data availability and
involve more assumptions. In contrast, the requirements for accurate
predictions, e.g., predicting which treatment to assign in our context, can
typically be less stringent than drawing efficient inference of parameter
estimates, as was discussed in Efron (2020, Criteria 6). These can distinguish
the prediction-driven focus and usefulness of DRITR. While an inference-based
criterion can only be applicable if all required assumptions hold, a
prediction-based criterion particularly focuses on some measurements of
testing performance and can be less restrictive. Therefore, even though DRITR
can be conservative by performing worst-case policy optimization, it can enjoy
less restrictions and more general applicability.

The last point we would like to point out is that the training of candidate
DRITRs can be performed before using calibrating data. This can provide more
privacy protection. Specifically, DRITR can utilize the training data to
obtain a class of candidate ITR estimates
$\\{\widehat{d}_{c}\\}_{c\in\mathcal{C}}$, where $\mathcal{C}$ is the set of
candidate DR-constants. When estimating the optimal DRITR on a specific
testing distribution, we only use the testing information to choose the best
ITR from $\\{\widehat{d}_{c}\\}_{c\in\mathcal{C}}$ without requesting for the
complete training data. In contrast, the combined analysis in Section 2
requires at least either $\\{\bm{X}_{i},\widehat{Q}(\bm{X}_{i},\pm
1):S_{i}=1\\}$ or $\\{\bm{X}_{i},\widehat{C}(\bm{X}_{i}):S_{i}=1\\}$ from the
training data. In this case, treatment effect information at the individual
level would be exposed to the testing agents. Therefore, the individualized
treatment effect information obtained from training can be kept privately when
applying DRITR, but cannot when using methods based on combined data in
Section 2.

## References

  * Athey and Wager (2020) Athey, S. and Wager, S. (2020), “Policy learning with observational data,” Econometrica, to appear.
  * Buchanan et al. (2018) Buchanan, A. L., Hudgens, M. G., Cole, S. R., Mollan, K. R., Sax, P. E., Daar, E. S., Adimora, A. A., Eron, J. J., and Mugavero, M. J. (2018), “Generalizing evidence from randomized trials using inverse probability of sampling weights,” Journal of the Royal Statistical Society: Series A (Statistics in Society), 181, 1193–1209.
  * Bühlmann (2020) Bühlmann, P. (2020), “Invariance, causality and robustness,” Statistical Science, 35, 404–426.
  * Chernozhukov et al. (2018) Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018), “Double/debiased machine learning for treatment and structural parameters,” The Econometrics Journal, 21, C1–C68.
  * Crump et al. (2006) Crump, R. K., Hotz, V. J., Imbens, G. W., and Mitnik, O. A. (2006), “Moving the goalposts: Addressing limited overlap in the estimation of average treatment effects by changing the estimand,” Tech. rep., National Bureau of Economic Research.
  * Crump et al. (2009) — (2009), “Dealing with limited overlap in estimation of average treatment effects,” Biometrika, 96, 187–199.
  * Dahabreh et al. (2019) Dahabreh, I. J., Robertson, S. E., Petito, L. C., Hernán, M. A., and Steingrimsson, J. A. (2019), “Efficient and robust methods for causally interpretable meta-analysis: transporting inferences from multiple randomized trials to a target population,” arXiv preprint arXiv:1908.09230.
  * Dong et al. (2020) Dong, L., Yang, S., Wang, X., Zeng, D., and Cai, J. (2020), “Integrative analysis of randomized clinical trials with real world evidence studies,” arXiv preprint arXiv:2003.01242.
  * Dukes and Vansteelandt (2020) Dukes, O. and Vansteelandt, S. (2020), Discussion of Kallus and Mo, Qi, and Liu: New Objectives for Policy Learning.
  * Efron (2020) Efron, B. (2020), “Prediction, Estimation, and Attribution,” Journal of the American Statistical Association, 115, 636–655.
  * Hainmueller (2012) Hainmueller, J. (2012), “Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies,” Political analysis, 20, 25–46.
  * Imai and Ratkovic (2014) Imai, K. and Ratkovic, M. (2014), “Covariate balancing propensity score,” Journal of the Royal Statistical Society: Series B: Statistical Methodology, 76, 243–263.
  * Kallus (2020) Kallus, N. (2020), “More Efficient Policy Learning via Optimal Retargeting,” Journal of the American Statistical Association, to appear.
  * Kanamori et al. (2012) Kanamori, T., Suzuki, T., and Sugiyama, M. (2012), “Statistical analysis of kernel-based least-squares density-ratio estimation,” Machine Learning, 86, 335–367.
  * Li et al. (2020) Li, S., Li, X., and Luedtke, A. (2020), “Discussion of Kallus (2020) and Mo, Qi, and Liu (2020): New Objectives for Policy Learning,” arXiv preprint arXiv:2010.04805.
  * Liang and Zhao (2020) Liang, M. and Zhao, Y. (2020), Discussion of Kallus (2020) and Mo et al (2020).
  * Mo et al. (2020) Mo, W., Qi, Z., and Liu, Y. (2020), “Learning Optimal Distributionally Robust Individualized Treatment Rules,” Journal of the American Statistical Association, to appear.
  * Molina et al. (2017) Molina, J., Rotnitzky, A., Sued, M., and Robins, J. (2017), “Multiple robustness in factorized likelihood models,” Biometrika, 104, 561–581.
  * Pearl and Bareinboim (2014) Pearl, J. and Bareinboim, E. (2014), “External validity: From do-calculus to transportability across populations,” Statistical Science, 29, 579–595.
  * Qi et al. (2019) Qi, Z., Cui, Y., Liu, Y., and Pang, J.-S. (2019), “Estimation of Individualized Decision Rules Based on an Optimized Covariate-Dependent Equivalent of Random Outcomes,” SIAM Journal on Optimization, 29, 2337–2362.
  * Robins et al. (2000) Robins, J. M., Rotnitzky, A., and van der Laan, M. (2000), “On profile likelihood: comment,” Journal of the American Statistical Association, 95, 477–482.
  * Rubin (1974) Rubin, D. B. (1974), “Estimating causal effects of treatments in randomized and nonrandomized studies,” Journal of Educational Psychology, 66, 688–701.
  * Rudolph and van der Laan (2017) Rudolph, K. E. and van der Laan, M. J. (2017), “Robust estimation of encouragement-design intervention effects transported across sites,” Journal of the Royal Statistical Society. Series B, Statistical methodology, 79, 1509–1525.
  * Stuart et al. (2011) Stuart, E. A., Cole, S. R., Bradshaw, C. P., and Leaf, P. J. (2011), “The use of propensity scores to assess the generalizability of results from randomized trials,” Journal of the Royal Statistical Society: Series A (Statistics in Society), 174, 369–386.
  * Tsiatis (2007) Tsiatis, A. (2007), Semiparametric theory and missing data, Springer Science & Business Media.
  * Uehara et al. (2020) Uehara, M., Kato, M., and Yasui, S. (2020), “Off-Policy Evaluation and Learning for External Validity under a Covariate Shift,” Advances in Neural Information Processing Systems, to appear.
  * Yu and Kumbier (2020) Yu, B. and Kumbier, K. (2020), “Veridical data science,” Proceedings of the National Academy of Sciences, 117, 3920–3929.
  * Zhao (2019) Zhao, Q. (2019), “Covariate balancing propensity score by tailored loss functions,” The Annals of Statistics, 47, 965–993.
  * Zhao et al. (2019) Zhao, Y.-Q., Zeng, D., Tangen, C. M., and Leblanc, M. L. (2019), “Robustifying trial-derived optimal treatment rules for a target population,” Electronic Journal of Statistics, 13, 1717–1743.