t+1}|\underline{X}^{\prime t},\underline{A}^{\prime t})$
$\displaystyle\overset{*_{3}}{=}$ $\displaystyle\sum_{\mathclap{a^{\prime
t+1},a^{t+1}}}p^{\prime}\left(C^{\prime}_{\boldsymbol{(\overline{\pi}^{t+2}_{\alpha})}}\Big{|}\underline{X}^{\prime
t},\underline{A}^{\prime t},\boldsymbol{a^{\prime
t+1}},\underline{X}^{t},\underline{A}^{t},a^{t+1}\right)\pi_{{id}}(a^{t+1}|\underline{X}^{\prime
t},\underline{A}^{\prime t},a^{\prime
t+1},\underline{X}^{t},\underline{A}^{t})\pi_{\alpha}(a^{\prime
t+1}|\underline{X}^{\prime t},\underline{A}^{\prime t})$
$\displaystyle\overset{*_{4}}{=}$ $\displaystyle\sum_{\mathclap{a^{\prime
t+1},a^{t+1}}}p^{\prime}\left(C^{\prime}_{(\overline{\pi}^{t+2}_{\alpha})}\Big{|}\boldsymbol{\underline{X}^{\prime
t},\underline{A}^{\prime t},a^{\prime
t+1},\underline{X}^{t},\underline{A}^{t}}\right)\underbrace{\pi_{{id}}(a^{t+1}|\underline{X}^{\prime
t},\underline{A}^{\prime t},a^{\prime
t+1},\underline{X}^{t},\underline{A}^{t})}_{\text{arbitrary dist. subject to
constraints}}\underbrace{\pi_{\alpha}(a^{\prime t+1}|\underline{X}^{\prime
t},\underline{A}^{\prime t})}_{\text{target policy}}$

with the following explanations:

  * •

$*1)$: We use that $C^{\prime}_{(\overline{\pi}^{t+2}_{\alpha},a^{\prime
t+1})}$ is independent of any interventions $\pi_{id}^{t+1}$ on $A^{t+1}$.

  * •

$*2)$: We use exchangeability :
$C^{\prime}_{(\overline{\pi}^{t+2}_{\alpha},a^{\prime
t+1},a^{t+1})}\perp\\!\\!\\!\perp A^{t+1}|\underline{X}^{\prime
t},\underline{A}^{\prime t},\underline{X}^{t},\underline{A}^{t}$ and
consistency:
$p^{\prime}\left(C^{\prime}_{(\overline{\pi}^{t+2}_{\alpha},a^{\prime
t+1},a^{t+1})}\Big{|}\underline{X}^{\prime t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t},a^{t+1}\right)=p^{\prime}\left(C^{\prime}_{(\overline{\pi}^{t+2}_{\alpha},a^{\prime
t+1})}\Big{|}\underline{X}^{\prime t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t},a^{t+1}\right)$ for $A^{t+1}$.

  * •

$*3)$: We use the exchangeability
$C^{\prime}_{(\overline{\pi}^{t+2}_{\alpha},a^{\prime
t+1})}\perp\\!\\!\\!\perp A^{\prime t+1}|\underline{X}^{\prime
t},\underline{A}^{\prime t},\underline{X}^{t},\underline{A}^{t},a^{t+1}$ and
consistency:
$p^{\prime}\left(C^{\prime}_{(\overline{\pi}^{t+2}_{\alpha},a^{\prime
t+1})}\Big{|}\underline{X}^{\prime t},\underline{A}^{\prime t},a^{\prime
t+1},\underline{X}^{t},\underline{A}^{t},a^{t+1}\right)=p^{\prime}\left(C^{\prime}_{(\overline{\pi}^{t+2}_{\alpha})}\Big{|}\underline{X}^{\prime
t},\underline{A}^{\prime t},a^{\prime
t+1},\underline{X}^{t},\underline{A}^{t},a^{t+1}\right)$ for $A^{\prime t+1}$.

  * •

$*4)$: We use the conditional independence
$C^{\prime}_{(\overline{\pi}^{t+2}_{\alpha})}\perp\\!\\!\\!\perp
A^{t+1}|\underline{X}^{\prime t},\underline{A}^{\prime t},a^{\prime
t+1},\underline{X}^{t},\underline{A}^{t}$

As before, we have to make sure in $*3)$ that
$p^{\prime}\left(C^{\prime}_{(\overline{\pi}^{t+2}_{\alpha},a^{\prime
t+1})}\Big{|}\underline{X}^{\prime t},\underline{A}^{\prime t},a^{\prime
t+1},\underline{X}^{t},\underline{A}^{t},a^{t+1}\right)$, i.e. conditioning on
$\underline{X}^{\prime t},\underline{A}^{\prime t},a^{\prime
t+1},\underline{X}^{t},\underline{A}^{t},a^{t+1}$, is well specified. To
understand what positivity requirements are necessary, we factorize the
"observational" (i.e. simulated) distribution for step $t$:

Observational factorization (step $t$):

$\displaystyle p^{\prime}\biggl{(}$ $\displaystyle
C^{\prime}_{(\overline{\pi}^{t+2}_{\alpha})}\Big{|}\underline{X}^{\prime
t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t}\biggl{)}=\sum_{X^{\prime
t},X^{t},a^{t+1},a^{\prime
t+1}}p^{\prime}\left(C^{\prime}_{(\overline{\pi}^{t+2}_{\alpha})}\Big{|}\underline{X}^{\prime
t},\underline{A}^{\prime t},a^{\prime
t+1},\underline{X}^{t},\underline{A}^{t}\right)$ $\displaystyle\cdot$
$\displaystyle\underbrace{\pi_{sim}^{\prime}(a^{\prime
t+1}|\underline{X}^{\prime t},\underline{A}^{\prime t},a^{t+1})}_{\text{known
simulation
policy}}\underbrace{\pi_{\beta}(a^{t+1}|\underline{X}^{t},\underline{A}^{t})}_{\text{retro.
acquisition policy}}\underbrace{g(X^{\prime t}|X^{t},A^{\prime
t})}_{\text{feature
revelation}}p(X^{t}|\underline{X}^{t-1},\underline{A}^{t})$

By comparing the observational and counterfactual factorizations, we see that
the following positivity assumption is required:

if $\displaystyle q^{\prime}(\underline{x}^{\prime t},\underline{a}^{\prime
t},a^{\prime
t+1},\underline{x}^{t},\underline{a}^{t},a^{t+1})=q^{\prime}(\underline{x}^{\prime
t},\underline{a}^{\prime
t},\underline{x}^{t},\underline{a}^{t})\pi_{\alpha}(a^{\prime
t+1}|\underline{x}^{\prime t},\underline{a}^{\prime
t})\pi_{{id}}(a^{t+1}|\underline{x}^{\prime t},\underline{a}^{\prime
t},a^{\prime t+1},\underline{x}^{t},\underline{a}^{t})>0$ then $\displaystyle
p^{\prime}(\underline{x}^{\prime t},\underline{x}^{\prime t},a^{\prime
t+1},\underline{x}^{t},\underline{a}^{t},a^{t+1})=p^{\prime}(\underline{x}^{\prime
t},\underline{a}^{\prime
t},\underline{x}^{t},\underline{a}^{t})\pi_{sim}^{\prime}(a^{\prime
t+1}|\underline{x}^{\prime t},\underline{a}^{\prime
t},a^{t+1})\pi_{\beta}(a^{t+1}|\underline{x}^{t},\underline{a}^{t})>0$
$\displaystyle\forall\underline{x}^{\prime
t},\underline{x}^{t},\underline{a}^{\prime t},\underline{a}^{t},a^{\prime
t+1},a^{t+1}$

with the following factorizations:

$\displaystyle q^{\prime}(\underline{X}^{\prime t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t})$
$\displaystyle=q^{\prime}(\underline{X}^{\prime t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t})g(X^{\prime t}|X^{t},A^{\prime
t})p(X^{t}|\underline{X}^{t-1},\underline{A}^{t})$ $\displaystyle
p^{\prime}(\underline{X}^{\prime t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t})$
$\displaystyle=p^{\prime}(\underline{X}^{\prime t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t})g(X^{\prime t}|X^{t},A^{\prime
t})p(X^{t}|\underline{X}^{t-1},\underline{A}^{t})$

The positivity condition can again be simplified through the two restrictions
on $\pi_{id}$:

Restrictions 1 and 2 for $\pi_{{id}}$ (step $t$):

if $\displaystyle a^{\prime t+1}\not\leq a^{t+1}\text{ or
}\pi_{\beta}(a^{t+1}|\underline{x}^{t},\underline{a}^{t})=0,$ then
$\displaystyle\pi_{{id}}(a^{t+1}|\underline{x}^{\prime
t},\underline{a}^{\prime t},a^{\prime
t+1},\underline{x}^{t},\underline{a}^{t})=0$ $\displaystyle\forall
a^{t+1},a^{\prime t+1}$

This imposes the requirement for $\pi_{\beta}$ that there exists at least one
value $a^{t+1}$ such that $a^{\prime t+1}\leq a^{t+1}$ and
$\pi_{\beta}(a^{t+1}|\underline{x}^{t},\underline{a}^{t})>0$ (i.e. local
positivity at $\underline{x}^{t},\underline{a}^{t},a^{\prime t}$). This has to
hold for all values $\underline{a}^{t}$ that were "allowed" in all previous
steps (i.e. all $a^{t}$, for all $\tau\leq t$, s.t.
$\pi_{id}(a^{\tau}|\underline{x}^{\prime\tau-1},\underline{a}^{\prime\tau-1},a^{\prime\tau},\underline{x}^{\tau-1},\underline{a}^{\tau-1})>0$
and which could later on have let to the current state). As $\pi_{id}$ only
needs to have support for at least one $a^{\tau}\in\mathcal{A}^{\tau}$ per
step, we can restrict $\pi_{id}$ at all previous steps to reduce the
positivity assumption for step $t$. Again, we do not want to restrict
$\pi_{id}$ too much, because if $\pi_{id}$ has wider support, this means that
more data points are used in the analysis. The regional positivity assumption
(from Definition 3) ensures in this case that only those simulations are used
(and exist) at all previous steps such that simulations of the desired actions
can be performed at step $t$ (and for future steps).

Full factorization

Bringing all time-steps $t=0,...,T$ together, one obtains the full
factorization of the identifying distribution $q^{\prime}$:

$\displaystyle q^{\prime}$
$\displaystyle(C^{\prime},Y,X^{\prime},A^{\prime},X,A)=g(C^{\prime}|\underline{X}^{\prime
T},\underline{A}^{\prime
T},Y)p(Y|\underline{X}^{T},\underline{A}^{T})q^{\prime}(\underline{X}^{\prime
T},\underline{A}^{\prime T},\underline{X}^{T},\underline{A}^{T})$

where

$\displaystyle q^{\prime}(\underline{X}^{\prime T},\underline{A}^{\prime T},$
$\displaystyle\underline{X}^{T},\underline{A}^{T})=$ (33)
$\displaystyle=\prod_{t=1}^{T}\underbrace{\pi_{\text{id}}(A^{t}|\underline{X}^{\prime
t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1})}_{\text{arbitrary distr. subject
to constraints}}\underbrace{\pi_{\alpha}(A^{\prime t}|\underline{X}^{\prime
t-1},\underline{A}^{\prime t-1})}_{\text{target
policy}}\prod_{t=0}^{T}p(X^{t}|\underline{X}^{t-1},\underline{A}^{t})\underbrace{g(X^{\prime
t}|X^{t},A^{\prime t})}_{\text{feature revelations}}$

In order for this expression to hold, $\pi_{id}$ must be restricted to have
support only on $\mathcal{\tilde{A}}$. This leads to the following
restriction:

$\displaystyle\pi_{id}(A^{t}|\underline{X}^{\prime t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1})=\underbrace{\mathbb{I}(A^{t}\in\mathcal{\tilde{A}}^{t}(\underline{X}^{\prime
t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1}))}_{\text{support
restriction}}f_{id}(\underline{X}^{\prime t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1}).$

where $f_{id}$ is an arbitrary function that ensures that $\pi_{id}$ is a
valid density. This concludes the proof of Theorem 3.

Bellman equation

Equations 31 and 32 correspond to the two parts of the semi-offline RL version
of the Bellman equation:

$\displaystyle\mathbb{E}\biggl{[}C^{\prime}_{(\overline{\pi}^{t+1}_{\alpha})}\Big{|}\underline{X}^{\prime
t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1}\biggl{]}$
$\displaystyle=\sum_{X^{\prime
t},X^{t}}\mathbb{E}\left[C^{\prime}_{(\overline{\pi}^{t+1}_{\alpha})}\Big{|}\underline{X}^{\prime
t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t-1},a^{t}\right]g(X^{\prime
t}|X^{t},A^{\prime t})p(X^{t}|\underline{X}^{t-1},\underline{A}^{t-1},a^{t})$
(34) $\displaystyle\text{ for any
}a^{t}\in\mathcal{A}^{t}(\underline{X}^{t-1},\underline{A}^{t-1},A^{\prime
t})$
$\displaystyle\mathbb{E}\biggl{[}C^{\prime}_{(\overline{\pi}^{t+1}_{\alpha})}\Big{|}\underline{X}^{\prime
t},\underline{A}^{\prime t},\underline{X}^{t},\underline{A}^{t}\biggl{]}$
$\displaystyle=\sum_{A^{\prime
t+1}}\mathbb{E}\left[C^{\prime}_{(\overline{\pi}^{t+2}_{\alpha})}\Big{|}\underline{X}^{\prime
t},\underline{A}^{\prime
t+1},\underline{X}^{t},\underline{A}^{t}\right]\pi_{\alpha}(A^{\prime
t+1}|\underline{X}^{\prime t},\underline{A}^{\prime t})$ (35)

The factorization holds under local positivity (if
$\mathcal{A}^{t}\neq\emptyset$ exists). Furthermore, the individual terms are
identified if regional positivity holds which concludes the proof of Theorem
4.

##  Appendix F Proof of Corollary 1

In this appendix, we proof Corollary 1, stating identification under the
maximal global positivity assumption.

Proof  Under the maximal global positivity assumption, we have
$\mathcal{\tilde{A}}^{t}(\underline{x}^{\prime t-1},\underline{a}^{\prime
t},\underline{x}^{t-1},\underline{a}^{t-1})=\mathcal{A}^{t}(\underline{x}^{t-1},\underline{a}^{t-1},a^{\prime
t})$. We can now insert this assumption into Eq. 18 which states the
identificiation of $J$ under global positivity:

$\displaystyle\pi_{id}(A^{t}|\underline{X}^{\prime t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1})$
$\displaystyle=\mathbb{I}(A^{t}\in\mathcal{\tilde{A}}^{t}(\underline{X}^{\prime
t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1}))f_{id}(\underline{X}^{\prime
t-1},\underline{A}^{\prime t},\underline{X}^{t-1},\underline{A}^{t-1})$
$\displaystyle=\mathbb{I}(A^{t}\in\mathcal{{A}}^{t}(\underline{X}^{t-1},\underline{A}^{t-1},A^{\prime
t}))f_{id}(\underline{X}^{\prime t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1})$
$\displaystyle=\mathbb{I}(A^{t}\geq a^{\prime
t})\pi_{\beta}(A^{t}|\underline{X}^{t-1},\underline{A}^{t-1})f_{id}^{*}(\underline{X}^{\prime
t-1},\underline{A}^{\prime t},\underline{X}^{t-1},\underline{A}^{t-1})$

where we let $f_{id}^{*}$ denote another arbitrary function that ensures that
$\pi_{id}$ is a valid density. This concludes the proof for Corollary 1.

##  Appendix G Comparison of the Semi-offline RL IPW estimator with Related
Methods

In this appendix, we demonstrate that our proposed IPW estimator
$J_{\textit{IPW-Semi}}$ is a more general version of an adapted version of the
IPW estimator introduced by [45]. We refer to this estimator as
$J_{\textit{IPW-Cen}}$ since it is derived from a censoring viewpoint. While
$J_{\textit{IPW-Cen}}$ was developed for a scenario where both feature
acquisition decisions and treatment decisions are made by the agent, it can be
adapted to the AFA setting. However, $J_{\textit{IPW-Cen}}$ is only applicable
to simpler settings with one acquisition option per time-point
($A^{t}\in\\{0,1\\}$). $J_{\textit{IPW-Cen}}$ is also only consistent if the
maximal global positivity assumption holds, as will be shown.

[45] derived $J_{\textit{IPW-Cen}}$ under the NDE and NUC assumptions. Instead
of using a semi-offline sampling policy that avoids the acquisition of non-
available features as proposed in this paper, [45] simply sample from
$\pi_{\alpha}$, even without knowledge about $X_{(1)}$. As the feature
revelation is not possible if a non-available feature is acquired, they treat
the resulting trajectory as censored. Known missing data methods are then
applied to adjust for this censoring. Hence, in the wording of this paper, we
would describe this viewpoint as an online RL + censoring viewpoint.

Adapted to the AFA setting under the consideration of deterministic AFA
policies $\pi_{\alpha}$, $J_{\textit{IPW-Cen}}$ becomes:

$\displaystyle J_{\textit{IPW-
Cen}}=\hat{\mathbb{E}}_{n,\textit{uncen}}[\rho^{T}_{\textit{Cen}}C_{(\pi_{\alpha})}]\text{
where }\rho^{T}_{\textit{Cen}}$
$\displaystyle=\prod_{t=1}^{T}\left(\frac{\mathbb{I}(A^{t}=1)}{\pi_{\beta}(A^{t}=1|\underline{X}^{t-1},\underline{A}^{t-1})}\right)^{A_{(\pi_{\alpha})}^{t}}$
(36)

where $\hat{\mathbb{E}}_{n^{\prime},\textit{uncen}}[.]$ denotes the empirical
average over the uncensored data points which have the known deterministic
counterfactuals $A_{(\pi_{\alpha})}^{t}$,$X_{(\pi_{\alpha})}^{t}$ and
$C_{(\pi_{\alpha})}$.

Since $A^{t}\in\\{0,1\\}$, it can be observed that the propensity score for a
specific time-point $t$ only appears in the factorization if the corresponding
action is $A_{(\pi_{\alpha})}^{t}=1$. An "acquire nothing" AFA policy (where
$\pi_{\alpha}(A^{t}=0|\underline{X}^{t-1},\underline{A}^{t-1})=1$ $\forall t$)
would thus require no adjustment ($\rho^{T}_{\textit{Cen}}=1$). In their
example, this estimator achieved a 50-fold increase in data efficiency
compared to the standard offline RL IPW estimator [45].

We establish the equivalence of our estimator $J_{\textit{IPW-Semi}}$ and
$J_{\textit{IPW-Cen}}$ in the following proposition:

######  Proposition 9.

(Equivalance of $J_{\textit{IPW-Cen}}$ and $J_{\textit{IPW-Semi}}$). The
estimators $J_{\textit{IPW-Cen}}$ and $J_{\textit{IPW-Semi}}$ are equivalent
for AFA settings with one action option per time-point, deterministic AFA
policies $\pi_{\alpha}$, the maximal global positivity assumption (Definition
6), and a simulation policy $\pi_{sim}=\pi_{\alpha}$.

Proof  Firstly, we clarify the blocking operation (from Definition 1) for this
setting:

$\displaystyle\pi_{sim}^{\prime}(A^{\prime t}=a^{\prime
t}|\underline{X}^{\prime t-1},\underline{A}^{\prime t-1},A^{t}=a^{t})$
$\displaystyle=\begin{cases}1,&\text{if $a^{\prime t}=0$ \& $a^{t}=0$}\\\
0,&\text{if $a^{\prime t}=1$ \& $a^{t}=0$}\\\ \pi_{\alpha}(A^{\prime
t}=a^{\prime t}|\underline{X}^{\prime t-1},\underline{A}^{\prime
t-1}),&\text{if $a^{t}=1$}.\end{cases}$

The inverse probability weights of $J_{\textit{IPW-Semi}}$ become:

$\displaystyle\rho^{T}_{\textit{Semi}}$
$\displaystyle=\prod_{t=1}^{T}\frac{\pi_{\alpha}(A^{\prime t}=a^{\prime
t}|\underline{X}^{\prime t-1},\underline{A}^{\prime
t-1})}{\pi^{\prime}_{sim}(A^{\prime t}=a^{\prime t}|\underline{X}^{\prime
t-1},\underline{A}^{\prime t-1},A^{t})}\frac{\mathbb{I}(A^{t}\geq a^{\prime
t})}{\pi_{\beta}(A^{t}\geq a^{\prime
t}|\underline{X}^{t-1},\underline{A}^{t-1})}$
$\displaystyle\overset{*_{1}}{=}\prod_{t=1}^{T}\left(\pi_{\alpha}(A^{\prime
t}=a^{\prime t}|\underline{X}^{\prime t-1},\underline{A}^{\prime
t-1})\right)^{1-a^{t}}\left(\frac{\mathbb{I}(A^{t}=1)}{\pi_{\beta}(A^{t}=1|\underline{X}^{t-1},\underline{A}^{t-1})}\right)^{a^{\prime
t}}$ $\displaystyle\overset{*_{2}}{=}\prod_{t=1}^{T}\mathbb{I}(A^{\prime
t}=A^{t}_{(\pi_{\alpha})})\left(\frac{\mathbb{I}(A^{t}=1)}{\pi_{\beta}(A^{t}=1|\underline{X}^{t-1},\underline{A}^{t-1})}\right)^{A^{t}_{(\pi_{\alpha})}}$

where we used in $*1)$ the above definition of $\pi^{\prime}_{sim}$ and that
$\mathbb{I}(A^{t}\geq 0)=1=\pi_{\beta}(A^{t}\geq
0|\underline{X}^{t-1},\underline{A}^{t-1})$. In $*2)$, we see that the first
term corresponds to whether $\pi_{\alpha}$ could be applied without running
into censoring. It thus gives 0 weights to all datapoints where blocking
occured (i.e. which are censored under the online RL + censoring viewpoint).
The second term then corresponds to the same weights as $\rho^{T}_{Cen}$ which
concludes the proof for Proposition 9.

We have demonstrated that, although the two estimators are derived from
different concepts (online RL with censoring vs semi-offline RL), they are
equal in this specific AFA setting of one action option per time-step,
deterministic policies and under the maximal global positivity assumption.
However, the key distinction lies in the generality of our estimator. Unlike
$J_{\textit{IPW-Cen}}$, which is limited to the described setting, we
developed an IPW estimator that can be applied for multiple acquisition
options (i.e. higher dimensional $A^{t}$), under the weaker global positivity
assumption and in a modified version for static features settings as we show
in our companion paper [56]. It can further be combined with a Q-model to
build the DRL estimator.

##  Appendix H Proof of Theorem 7

Proof  To proof the double robustness property of the semi-offline RL version
of the DRL estimator (i.e. Theorem 7), we decompose $J_{\textit{DRL-Semi}}$ in
two different ways:

Scenario 1: If $\hat{\pi}_{\beta}$ is correctly specified, we find

$\displaystyle J_{\textit{DRL-Semi}}$
$\displaystyle=\underbrace{\mathbb{E}_{p^{\prime}}[\rho_{\textit{Semi}}^{T}C^{\prime}]}_{=J}+\sum_{t=1}^{T}\underbrace{\mathbb{E}_{p^{\prime}}\left[-\rho_{\textit{Semi}}^{t}Q_{\textit{Semi}}^{t}+\rho_{\textit{Semi}}^{t-1}V_{\textit{Semi}}^{t-1}\right]}_{=0}.$

where the first term is just the IPW estimator. As $\hat{\pi}_{\beta}$ is
correctly specified, it consistently estimates $J$. The fact that the second
term becomes equal to 0 is shown in the following:

$\displaystyle\mathbb{E}_{p^{\prime}}$
$\displaystyle\left[-\rho_{\textit{Semi}}^{t}Q_{\textit{Semi}}^{t}+\rho_{\textit{Semi}}^{t-1}V_{\textit{Semi}}^{t-1}\right]=$
$\displaystyle\overset{*_{1}}{=}\mathbb{E}_{p^{\prime}}\biggl{[}\rho_{\textit{Semi}}^{\boldsymbol{t-1}}\boldsymbol{\biggl{(}-\frac{\pi_{\alpha}(A^{\prime
t}|\underline{X}^{\prime t-1},\underline{A}^{\prime
t-1})}{\pi^{\prime}_{sim}(A^{\prime t}|\underline{X}^{\prime
t-1},\underline{A}^{\prime
t-1},A^{t})}\frac{\pi_{id}(A^{t}|\underline{X}^{\prime
t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1})}{\pi_{\beta}(A^{t}|\underline{X}^{t-1},\underline{A}^{t-1})}}Q_{\textit{Semi}}^{t}$
$\displaystyle\quad\quad\quad\quad\quad\quad+\boldsymbol{\sum_{A^{\prime
t}}\pi_{\alpha}(A^{\prime t}|\underline{X}^{\prime t-1},\underline{A}^{\prime
t-1})Q_{\textbf{{Semi}}}^{t}\biggl{)}}\biggl{]}$
$\displaystyle\overset{*_{2}}{=}\mathbb{E}_{p^{\prime}}\biggl{[}\rho_{\textit{Semi}}^{t-1}\biggl{(}-\boldsymbol{\sum_{A^{\prime
t},A^{t}}\pi^{\prime}_{sim}(A^{\prime t}|\underline{X}^{\prime
t-1},\underline{A}^{\prime
t-1},A^{t})\pi_{\beta}(A^{t}|\underline{X}^{t-1},\underline{A}^{t-1})}\frac{\pi_{\alpha}(A^{\prime
t}|\underline{X}^{\prime t-1},\underline{A}^{\prime
t-1})}{\pi^{\prime}_{sim}(A^{\prime t}|\underline{X}^{\prime
t-1},\underline{A}^{\prime t-1},A^{t})}\cdot$
$\displaystyle\quad\quad\quad\quad\quad\quad\cdot\frac{\pi_{id}(A^{t}|\underline{X}^{\prime
t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1})}{\pi_{\beta}(A^{t}|\underline{X}^{t-1},\underline{A}^{t-1})}Q_{\textit{Semi}}^{t}+\sum_{A^{\prime
t}}\pi_{\alpha}(A^{\prime t}|\underline{X}^{\prime t-1},\underline{A}^{\prime
t-1})Q_{\textit{Semi}}^{t}\biggl{)}\biggl{]}$
$\displaystyle\overset{*_{3}}{=}\mathbb{E}_{p^{\prime}}\biggl{[}\rho_{\textit{Semi}}^{t-1}\biggl{(}-\boldsymbol{\sum_{A^{\prime
t}}\pi_{\alpha}(A^{\prime t}|\underline{X}^{\prime t-1},\underline{A}^{\prime
t-1})}Q_{\textit{Semi}}^{t}+\sum_{A^{\prime t}}\pi_{\alpha}(A^{\prime
t}|\underline{X}^{\prime t-1},\underline{A}^{\prime
t-1})Q_{\textit{Semi}}^{t}\biggl{)}\biggl{]}=0$

with the following explanations:

  * •

$*1)$: We use the relationship
$V_{\textit{Semi}}^{t-1}=\mathbb{E}_{\pi_{\alpha}}[Q_{\textit{Semi}}^{t}]$ and
the decomposition of $\rho_{\textit{Semi}}$.

  * •

$*2)$: We use the fact that one can pull the expected value with respect to
$\pi^{\prime}_{sim}(A^{\prime t}|\underline{X}^{\prime
t-1},\underline{A}^{\prime
t-1},A^{t})\pi_{\beta}(A^{t}|\underline{X}^{t-1},\underline{A}^{t-1})$ inside.

  * •

$*3)$: We use the fact that $Q^{t}_{Semi}$ is independent of
$\pi_{id}(A^{t}|\underline{X}^{\prime t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1})$ as long as it fulfills the
positivity assumption.

Scenario 2: If $Q_{\textit{Semi}}$ is correctly specified, we find

$\displaystyle J_{\textit{DRL-
Semi}}=\mathbb{E}_{p^{\prime}}[V_{\textit{Semi}}^{0}]+\mathbb{E}_{p^{\prime}}\left[\rho_{\textit{Semi}}^{T}\left(C^{\prime}-Q_{\textit{Semi}}^{T}\right)\right]+\mathbb{E}_{p^{\prime}}\left[\sum_{t=1}^{T-1}\rho_{\textit{Semi}}^{t}\left(-Q_{\textit{Semi}}^{t}+V_{\textit{Semi}}^{t}\right)]\right]$
$\displaystyle=\underbrace{\mathbb{E}_{p^{\prime}}\left[V_{\textit{Semi}}^{0}\right]}_{=J}+\mathbb{E}_{p^{\prime}}\left[\rho_{\textit{Semi}}^{T}\underbrace{\left(\sum_{C^{\prime}}C^{\prime}p(C^{\prime}|\underline{X}^{\prime
T-1},\underline{A}^{\prime
T},\underline{X}^{T-1},\underline{A}^{T-1})-Q_{\textit{Semi}}^{T}\right)}_{=0}\right]$
$\displaystyle+\mathbb{E}_{p^{\prime}}\biggl{[}\sum_{t=1}^{T-1}\rho_{\textit{Semi}}^{t}\underbrace{\biggl{(}-Q_{\textit{Semi}}^{t}+\sum_{X^{\prime
t},X^{t},}V_{\textit{Semi }}^{t}g(X^{\prime t}|X^{t},A^{\prime
t})p(X^{t}|\underline{X}^{t-1},\underline{A}^{t})\biggl{)}}_{=0}\biggl{]}$

where $\mathbb{E}_{p^{\prime}}\left[V_{\textit{Semi}}^{0}\right]$ corresponds
to the DM estimator which is consistent if $Q_{\textit{Semi}}$ is correctly
specified. For the last term, we used that we can pull in the expected value
with respect to the conditional distributions of $X^{t}$ and $X^{\prime t}$.
The resulting term equals the first part of the semi-offline RL version of
Bellman’s equation. This concludes the proof of Theorem 7.

##  Appendix I Proof of Theorem 8

In the following, we proof Theorem 8 by deriving an influence function for the
reformulated target $J$ from Eq. 15.

Proof  We derive an influence function following the path-derivative approach
described in our review of semi-parametric theory (Appendix B), similar to the
derivation of the influence function for offline RL [4]. We start by
specifying the parametric submodel and score function. The entire regular
parametric submodel for the curated dataset $\mathcal{D}^{\prime}$ is

$\displaystyle\\{g(C^{\prime}|\underline{X}^{\prime T},\underline{A}^{\prime
T},Y)p_{\theta}(Y|\underline{X}^{T},\underline{A}^{T})$
$\displaystyle\prod_{t=1}^{T}\pi^{\prime}_{sim}(A^{\prime
t}|\underline{X}^{\prime t-1},\underline{A}^{\prime
t-1},A^{t})\pi_{\beta,\theta}(A^{t}|\underline{X}^{t-1},\underline{A}^{t-1})\prod_{t=0}^{T}g(X^{\prime
t}|X^{t},A^{\prime t})$ $\displaystyle\cdot
p_{\theta}(X^{t}|\underline{X}^{t-1},\underline{A}^{t})\\}$

and it is equal to the true pdf at $\theta=0$. The corresponding score
function is:

$\displaystyle S\equiv S_{C^{\prime},Y,X^{\prime},A^{\prime},X,A}$
$\displaystyle=\nabla_{\theta}\text{log}p_{\theta}(Y|\underline{X}^{T},\underline{A}^{T})+\sum_{t=1}^{T}\nabla_{\theta}\text{log}\pi_{\beta,\theta}(A^{t}|\underline{X}^{t-1},\underline{A}^{t-1})+\sum_{t=0}^{T}\nabla_{\theta}\text{log}p_{\theta}(X^{t}|\underline{X}^{t-1},\underline{A}^{t})$
$\displaystyle=S_{Y|\underline{X}^{T},\underline{A}^{T}}+\sum_{t=1}^{T}S_{A^{t}|\underline{X}^{t-1},\underline{A}^{t-1}}+\sum_{t=0}^{T}S_{X^{t}|\underline{X}^{t-1},\underline{A}^{t-1}}$

One can see that, as expected, the score function only depends on scores of
the original dataset $\mathcal{D}$ as all other distributions are known and
artificially added for the simulation.

Computing the derivative $\nabla_{\theta}J$ gives:

$\displaystyle\nabla_{\theta}J\overset{*_{1}}{=}\nabla_{\theta}\mathbb{E}_{q^{\prime}}\left[C^{\prime}\right]=\nabla_{\theta}\left(\sum_{C^{\prime},Y,X^{\prime},A^{\prime},X,A}C^{\prime}q^{\prime}_{\theta}(C^{\prime},Y,X^{\prime},A^{\prime},X,A)\right)$
$\displaystyle=\sum_{C^{\prime},Y,X^{\prime},A^{\prime},X,A}C^{\prime}\nabla_{\theta}q^{\prime}_{\theta}(C^{\prime},Y,X^{\prime},A^{\prime},X,A)$
$\displaystyle\overset{*_{2}}{=}\sum_{C^{\prime},Y,X^{\prime},A^{\prime},X,A}C^{\prime}q^{\prime}_{\theta}(C^{\prime},Y,X^{\prime},A^{\prime},X,A)\nabla_{\theta}\text{log}q^{\prime}_{\theta}(C^{\prime},Y,X^{\prime},A^{\prime},X,A)$
$\displaystyle\overset{*_{3}}{=}\mathbb{E}_{q^{\prime}}\left[C^{\prime}\left(S_{Y|\underline{X}^{T},\underline{A}^{T}}+\sum_{t=0}^{T}S_{X^{t}|\underline{X}^{t-1},\underline{A}^{t}}+\sum_{t=1}^{T}S_{A^{t}|A^{t}\geq
A^{\prime t},\underline{X}^{t-1},\underline{A}^{t-1}}\right)\right]$

where we used in $*1)$ a simplification of notation:
$\mathbb{E}_{q^{\prime}}[C^{\prime}]\equiv\mathbb{E}_{q^{\prime}_{\theta}}[C^{\prime}]\equiv\mathbb{E}[C^{\prime}_{(\pi_{\alpha})}]$.
Similarly, we will denote
$\mathbb{E}[C^{\prime}_{(\pi_{\alpha})}|\underline{X}^{\prime
t},\underline{A}^{\prime t},\underline{X}^{t},\underline{A}^{t}]$ by
$\mathbb{E}_{q^{\prime}}[C^{\prime}|\underline{X}^{\prime
t},\underline{A}^{\prime t},\underline{X}^{t},\underline{A}^{t}]$. In $*2)$,
we used the property of the differentiation of the logarithm:

$\displaystyle\nabla_{\theta}\text{log}p_{\theta}(u)=\frac{\nabla_{\theta}p_{\theta}(u)}{p_{\theta}(u)}.$

In $*3)$, we used that $\pi_{\alpha}$, $\pi^{\prime}_{sim}$ and all
deterministic functions are known and independent of $\theta$. We also
inserted our choice for $\pi_{id}$ under the maximal global positivity
assumption.

Next, we handle each term separately:

Term 1:

$\displaystyle\mathbb{E}_{q^{\prime}}[C^{\prime}S_{Y|\underline{X}^{T},\underline{A}^{T}}]$
$\displaystyle=\mathbb{E}_{p^{\prime}}\left[\frac{q^{\prime}(C^{\prime},Y,X^{\prime},A^{\prime},X,A)}{p^{\prime}(C^{\prime},Y,X^{\prime},A^{\prime},X,A)}C^{\prime}S_{Y|\underline{X}^{T},\underline{A}^{T}}\right]$
$\displaystyle=\mathbb{E}_{p^{\prime}}\left[\rho_{\textit{Semi}}^{T}C^{\prime}S_{Y|\underline{X}^{T},\underline{A}^{T}}\right]$
$\displaystyle\overset{*_{1}}{=}\mathbb{E}_{p^{\prime}}\left[\rho_{\textit{Semi}}^{T}(C^{\prime}-\mathbb{E}_{q^{\prime}}[C^{\prime}|\underline{X}^{\prime
T},\underline{A}^{\prime
T},\underline{X}^{T},\underline{A}^{T}])S_{Y|\underline{X}^{T},\underline{A}^{T}}\right]$
$\displaystyle\overset{*_{2}}{=}\mathbb{E}_{p^{\prime}}\left[\rho_{\textit{Semi}}^{T}\left(C^{\prime}-\mathbb{E}_{q^{\prime}}\left[C^{\prime}|\underline{X}^{\prime
T},\underline{A}^{\prime
T},\underline{X}^{T},\underline{A}^{T}\right]\right)S\right]$

where we used in $*1)$ the fact that
$\mathbb{E}_{q^{\prime}}[C^{\prime}|\underline{X}^{\prime
T},\underline{A}^{\prime T},\underline{X}^{T},\underline{A}^{T}]$
$(=\mathbb{E}_{p^{\prime}}[C^{\prime}|\underline{X}^{\prime
T},\underline{A}^{\prime T},\underline{X}^{T},\underline{A}^{T}])$ and
$\rho_{\textit{Semi}}^{T}$ are independent of
$p^{\prime}(C^{\prime},Y|\underline{X}^{\prime T},\underline{A}^{\prime
T},\underline{X}^{T},\underline{A}^{T})$ and that the scores are mean zero,
i.e.
$\mathbb{E}_{p^{\prime}}\left[S_{Y|\underline{X}^{T},\underline{A}^{T}}\right]=0$.

To show $*2)$, we use the fact that all other scores
($S_{X^{t}|\underline{X}^{t-1},\underline{A}^{t}}$ and
$S_{A^{t}|\underline{X}^{t-1},\underline{A}^{t}}$) are independent of
$p^{\prime}(C^{\prime}|\underline{X}^{\prime T},\underline{A}^{\prime
T},\underline{X}^{T},\underline{A}^{T})$ such that this part of the expected
value can be pulled inside to find:

$\displaystyle\mathbb{E}_{p^{\prime}}\left[\left(C^{\prime}-\mathbb{E}_{q^{\prime}}\left[C^{\prime}|\underline{X}^{\prime
T},\underline{A}^{\prime
T},\underline{X}^{T},\underline{A}^{T}\right]\right)|\underline{X}^{\prime
T},\underline{A}^{\prime T},\underline{X}^{T},\underline{A}^{T}\right]=0$

Term 2:

$\displaystyle\mathbb{E}_{q^{\prime}}\biggl{[}C^{\prime}S_{X^{t}|\underline{X}^{t-1},\underline{A}^{t}}\biggr{]}$
$\displaystyle=\mathbb{E}_{p^{\prime}}\left[\rho_{\textit{Semi}}^{t}\mathbb{E}_{q^{\prime}}\left[C^{\prime}|\underline{X}^{\prime
t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t}\right]S_{X^{t}|\underline{X}^{t-1},\underline{A}^{t}}\right]$
$\displaystyle\overset{*_{1}}{=}\mathbb{E}_{p^{\prime}}\left[\rho_{\textit{Semi}}^{t}\left(\mathbb{E}_{q^{\prime}}\left[C^{\prime}|\underline{X}^{\prime
t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t}\right]-\mathbb{E}_{q^{\prime}}\left[C^{\prime}|\underline{X}^{\prime
t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1}\right]\right)S_{X^{t}|\underline{X}^{t-1},\underline{A}^{t}}\right]$
$\displaystyle\overset{*_{2}}{=}\mathbb{E}_{p^{\prime}}\left[\rho_{\textit{Semi}}^{t}\left(\mathbb{E}_{q^{\prime}}[C^{\prime}|\underline{X}^{\prime
t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t}]-\mathbb{E}_{q^{\prime}}[C^{\prime}|\underline{X}^{\prime
t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1}]\right)S\right]$

In $*1)$, we used the fact that
$\mathbb{E}_{q^{\prime}}[C^{\prime}|\underline{X}^{\prime
t-1},\underline{A}^{\prime t},\underline{X}^{t-1},\underline{A}^{t-1}]$ and
$\rho_{\textit{Semi}}^{t}$ are independent of
$p(X^{t}|\underline{X}^{t-1},\underline{A}^{t})$ and that the scores are mean
zero, i.e.
$\mathbb{E}_{p^{\prime}}\left[S_{X^{t}|\underline{X}^{t-1},\underline{A}^{t}}\right]=0$.
To show $*2)$, we split again by adding the scores separately to the equation.

  * •

We find that $\mathbb{E}_{p^{\prime}}[.]$ can be split:

$\displaystyle\mathbb{E}_{p^{\prime}}[.]=\mathbb{E}_{\underline{X}^{\prime
t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t}}\left[\mathbb{E}_{C^{\prime},Y,\overline{X}^{\prime
t+1},\overline{A}^{\prime
t+1},\overline{X}^{t+1},\overline{A}^{t+1}|\underline{X}^{\prime
t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t}}\left[.|\underline{X}^{\prime
t},\underline{A}^{\prime t},\underline{X}^{t},\underline{A}^{t}\right]\right]$

such that pulling the inner expectation inside and using the mean zero
property of the scores shows that $S_{Y|\underline{X}^{T},\underline{A}^{T}}$,
$S_{X^{\tau}|\underline{X}^{\tau-1},\underline{A}^{\tau}}$ and
$S_{A^{\tau}|\underline{X}^{\tau-1},\underline{A}^{\tau-1}}$ with $\tau>t$ can
be added.

  * •

We find that a further split of the expected value:

$\displaystyle\mathbb{E}_{\underline{X}^{\prime t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t}}[.]=\mathbb{E}_{\underline{X}^{\prime
t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t}}\left[\mathbb{E}_{X^{\prime
t},X^{t}|\underline{X}^{\prime t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t}}\left[.|\underline{X}^{\prime
t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t}\right]\right]$

leads to the difference of expected values in the brackets $(.)$ to become
zero, when the inner expectation is resolved first. This pulling inside of the
inner expectation is possible for the terms
$S_{X^{\tau}|\underline{X}^{\tau-1},\underline{A}^{\tau}}$ (for $\tau<t$) and
for $S_{A^{\tau}|\underline{X}^{\tau-1},\underline{A}^{\tau-1}}$ (for
$\tau\leq t$) as they are independent of the inner expected value.

Further we obtain for $t=0$:

$\displaystyle\mathbb{E}_{q^{\prime}}[C^{\prime}$ $\displaystyle
S_{X^{0}}]=\mathbb{E}_{p^{\prime}}\left[\left(\mathbb{E}_{q^{\prime}}[C^{\prime}|\underline{X}^{\prime
0},\underline{X}^{0}]-\mathbb{E}_{q^{\prime}}[C^{\prime}]\right)S\right]$

where $\mathbb{E}_{q^{\prime}}[C^{\prime}]=J$

Term 3:

$\displaystyle\mathbb{E}_{q^{\prime}}$
$\displaystyle\biggl{[}C^{\prime}S_{A^{t}|A^{t}\geq A^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1}}\biggr{]}=\mathbb{E}_{q^{\prime}}\biggl{[}\mathbb{E}_{q^{\prime}}[C^{\prime}|\underline{X}^{\prime
t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t}]S_{A^{t}|A^{t}\geq A^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1}}\biggr{]}=0$

where we leverage the fact that
$\mathbb{E}_{q^{\prime}}[C^{\prime}|\underline{X}^{\prime
t-1},\underline{A}^{\prime t},\underline{X}^{t-1},\underline{A}^{t}]$ is
independence of $A^{t}$, and the mean zero property of the scores.

Influence function:

Bringing all terms together, we see that

$\displaystyle\nabla_{\theta}J=$
$\displaystyle\mathbb{E}_{p^{\prime}}\biggl{[}\Bigl{(}\rho_{\textit{Semi}}^{T}\left(C^{\prime}-\mathbb{E}_{q^{\prime}}[C^{\prime}|\underline{X}^{\prime
T},\underline{A}^{\prime T},\underline{X}^{T},\underline{A}^{T}]\right)$
$\displaystyle\quad+\sum_{t=1}^{T}\rho_{\textit{Semi}}^{t}\left(\mathbb{E}_{q^{\prime}}[C^{\prime}|\underline{X}^{\prime
t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t}]-\mathbb{E}_{q^{\prime}}[C^{\prime}|\underline{X}^{\prime
t-1},\underline{A}^{\prime t},\underline{X}^{t-1},\underline{A}^{t-1}]\right)$
$\displaystyle\quad+\mathbb{E}_{q^{\prime}}[C^{\prime}|\underline{X}^{\prime
0},\underline{X}^{0}]-J\Bigr{)}S\biggr{]}$ $\displaystyle=$
$\displaystyle\mathbb{E}_{p^{\prime}}\biggl{[}\Bigl{(}\rho_{\textit{Semi}}^{T}\left(C^{\prime}-V_{\textit{Semi}}^{T}\right)+\sum_{t=1}^{T}\rho_{\textit{Semi}}^{t}\left(V_{\textit{Semi}}^{t}-Q_{\textit{Semi}}^{t}\right)+V_{\textit{Semi}}^{0}-J\Bigr{)}S\biggr{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}_{p^{\prime}}\biggl{[}\Bigl{(}\rho_{\textit{Semi}}^{T}C^{\prime}+\sum_{t=1}^{T}\left(-\rho_{\textit{Semi}}^{t}Q_{\textit{Semi}}^{t}+\rho_{\textit{Semi}}^{t-1}V_{\textit{Semi}}^{t-1}\right)-J\Bigr{)}S\biggr{]}$

Reading off the influence function $\Psi$ gives the influence from Theorem 8.
Note, that we do not claim efficiency of this influence function and leave the
investigation of the efficient influence function for future work.

##  Appendix J Estimation of Other Target Parameters from the Semi-offline RL
View

In this appendix, we extend the target parameter to include time-dependent
costs $C^{t}$ (s.t. $C=\overline{C}^{1}$). The newly defined target parameter
becomes $J=\mathbb{E}\left[\sum_{t=1}^{T}C^{t}_{(\pi_{\alpha})}\right]$. In
particular, these costs may include acquisition costs or misclassifications
costs for predictions at each time-step. The acquisition costs are given by
the known deterministic $g(C^{t}_{a}|A^{t})$. When considering
misclassification costs, we assume a per-step label $Y^{t}$ to be available at
each time step (s.t. $Y=\overline{Y}^{1}$). The per-step misclassification
costs can be computed by:

$\displaystyle
g(C^{t}_{mc}|\underline{X}^{t},\underline{A}^{t},Y^{t})=\sum_{Y^{*t}}\underbrace{g(C_{mc}|Y^{*t},Y^{t})}_{\text{misclassification
cost}}\underbrace{g(Y^{*t}|\underline{X}^{t},\underline{A}^{t})}_{\text{classifier}}.$

We combine both costs such that the target parameter is redefined to be:

$\displaystyle
J=\mathbb{E}\left[\sum_{t=1}^{T}C^{t}_{(\pi_{\alpha})}\right]=\mathbb{E}\left[\sum_{t=1}^{T}\left(C^{t}_{a,(\pi_{\alpha})}+C^{t}_{mc,(\pi_{\alpha})}\right)\right]$

with known $g(C^{t}|\underline{X}^{t},\underline{A}^{t},Y^{t})$.

The reformulation, identification and estimation steps from the semi-offline
RL view can be extended to per-step costs. We provide corollaries of the
identification and estimation theorems from the main body for this setting. We
do not provide additional proofs, as the extensions are straightforward.

###  J.1 Identification

We start with a corollary that extends Theorem 3 for the per-step costs
setting.

######  Corollary 2.

(Identification of $J$ (for per-step costs) for the semi-offline RL view). The
reformulated AFAPE problem of estimating $J$ (for per-step costs) under the
semi-offline RL view is under the no direct effect (NDE), no unobserved
confounding (NUC), consistency, no interference and the global positivity
assumption (from Definition 4) identified by

$\displaystyle
J=\mathbb{E}_{p^{\prime}}\left[\sum_{t=1}^{T}C^{\prime}_{(\pi_{\alpha})}\right]=\sum_{C^{\prime},Y,X^{\prime},A^{\prime},X,A}\left(\sum_{t=1}^{T}\mathbb{E}[C^{\prime
t}|\underline{X}^{\prime t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t}]\right)q^{\prime}(\underline{X}^{\prime
T},\underline{A}^{\prime T},\underline{X}^{T},\underline{A}^{T})$ (37)

where $q^{\prime}$ is given by Eq. 17 and

$\displaystyle\mathbb{E}[C^{\prime t}|\underline{X}^{\prime
t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t}]=\sum_{C^{\prime
t},Y^{t},Y^{\prime*t}}C^{\prime t}g(C^{\prime
t}|Y^{t},Y^{\prime*t})p(Y^{t}|\underline{X}^{t},\underline{A}^{t})g(Y^{\prime*t}|\underline{X}^{\prime
t},\underline{A}^{\prime t}).$

Note that the decomposition in Eq. 37 does not lead to the most efficient
estimators for this setting. There are multiple ways to change this
identification theorem to improve the derived estimators. Firstly, one may
leverage the correlation between $Y^{t}$/$C^{\prime t}$ between different
time-steps. Secondly, the choice / support restrictions of $\pi_{id}$ can
potentially be improved. The regional positivity assumption requires certain
support restrictions for $\pi_{id}$. These support restrictions are, however,
weaker for the target $\mathbb{E}[C^{\prime\tau}_{(\pi_{\alpha})}]$ than for
$\mathbb{E}[C^{\prime t}_{(\pi_{\alpha})}]$ (for $\tau<t$). One may thus vary
the choice for $\pi_{id}$ depending on the target. These potential
improvements are, however, outside the scope of this paper. We provide here
only identification and estimation results that can be easily derived from the
theorems in the main body.

Next, we continue with a corollary that extends Theorem 4 for the per-step
costs setting.

######  Corollary 3.

(Bellman equation for semi-offline RL (for per-step costs)). The semi-offline
RL view admits under the no direct effect (NDE), no unobserved confounding
(NUC), consistency, no interference, and the local positivity assumption at
datapoint $\underline{x}^{t},\underline{a}^{t},a^{\prime t}$ (from Definition
2), the following semi-offline RL version of the Bellman equation for per-step
costs:

$\displaystyle Q_{\textit{Semi}}(\underline{X}^{\prime t-1},$
$\displaystyle\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1})=\sum_{\mathclap{X^{\prime
t},X^{t}}}V_{\textit{Semi}}(\underline{X}^{\prime t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t-1},A^{t}=a^{t})g(X^{\prime
t}|X^{t},A^{\prime
t})p(X^{t}|\underline{X}^{t-1},\underline{A}^{t-1},A^{t}=a^{t})$
$\displaystyle\text{ for any
}a^{t}\in\mathcal{A}^{t}(\underline{X}^{t-1},\underline{A}^{t-1},A^{\prime
t})$ (38) $\displaystyle V_{\textit{Semi}}(\underline{X}^{\prime
t},\underline{A}^{\prime t},$
$\displaystyle\underline{X}^{t},\underline{A}^{t})=\mathbb{E}_{p^{\prime}}[C^{\prime
t}|\underline{X}^{\prime t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t}]+\sum_{A^{\prime
t+1}}Q_{\textit{Semi}}(\underline{X}^{\prime t},\underline{A}^{\prime
t+1},\underline{X}^{t},\underline{A}^{t-1},A^{t})\pi_{\alpha}(A^{\prime
t+1}|\underline{X}^{\prime t},\underline{A}^{\prime t})$ (39)

with semi-offline RL versions of the state-action value function
$Q_{\textit{Semi}}$ and state value function $V_{\textit{Semi}}$:

$\displaystyle Q_{\textit{Semi}}^{t}$ $\displaystyle\equiv
Q_{\textit{Semi}}(\underline{X}^{\prime t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1})\equiv\mathbb{E}_{p^{\prime}}\left[\sum_{\tau={t}}^{T}C^{\prime\tau}_{(\overline{\pi}^{t+1}_{\alpha})}|\underline{X}^{\prime
t-1},\underline{A}^{\prime t},\underline{X}^{t-1},\underline{A}^{t-1}\right]$
$\displaystyle V_{\textit{Semi}}^{t}$ $\displaystyle\equiv
V_{\textit{Semi}}(\underline{X}^{\prime t},\underline{A}^{\prime
t},\underline{X}^{t},\underline{A}^{t})\equiv\mathbb{E}_{p^{\prime}}\left[\sum_{\tau={t}}^{T}C^{\prime\tau}_{(\overline{\pi}^{t+1}_{\alpha})}|\underline{X}^{\prime
t},\underline{A}^{\prime t},\underline{X}^{t},\underline{A}^{t}\right]$

Furthermore, $Q_{\textit{Semi}}^{t}$ and $V_{\textit{Semi}}^{t}$ are
identified if the regional positivity assumption holds at
$\underline{X}^{\prime t-1},\underline{A}^{\prime
t},\underline{X}^{t-1},\underline{A}^{t-1}$ and
$a^{t}\in\mathcal{\tilde{A}}^{t}(\underline{X}^{\prime
t-1},\underline{A}^{\prime t},\underline{X}^{t-1},\underline{A}^{t-1})$.

###  J.2 Estimation

The estimation formulas can be extended to the per-step setting as follows:

1) Inverse probability weighting (IPW):

The target cost (for per-step costs) that is estimated by the semi-offline IPW
estimator is

$\displaystyle J_{\textit{IPW-Semi}}$
$\displaystyle=\hat{\mathbb{E}}_{n^{\prime}}\left[\sum_{t=1}^{T}\rho_{\textit{Semi}}^{t}\text{
}C^{\prime t}\right],$

with the same options for $\rho_{\textit{Semi}}^{t}$ as in the setting
described in the main body.

2) Direct method (DM):

The target cost (for per-step costs) is estimated by the semi-offline DM
estimator as

$\displaystyle J_{\textit{DM-
Semi}}=\hat{\mathbb{E}}_{n^{\prime}}[V_{\textit{Semi}}^{0}]$

with the adapted per-step cost version of $V_{\textit{Semi}}$ from Corollary
3.

3) Double reinforcement learning (DRL):

The target cost (for per-step costs) that is estimated by the semi-offline DRL
estimator is

$\displaystyle J_{\textit{DRL-
Semi}}=\hat{\mathbb{E}}_{n^{\prime}}\left[\sum_{t=1}^{T}\left(\rho_{\textit{Semi}}^{t}C^{\prime
t}-\rho_{\textit{Semi}}^{t}Q_{\textit{Semi}}^{t}+\rho_{\textit{Semi}}^{t-1}V_{\textit{Semi}}^{t-1}\right)\right].$

with the adapted per-step cost version of $V_{\textit{Semi}}$ and
$Q_{\textit{Semi}}$.

##  Appendix K Experiment Details

In this section, we first describe the experiment setup in more detail. We
also provide a detailed list of the parameters and configurations for each
experiment in Table 3.

###  K.1 Data, Costs and Missingness Mechanisms

For the experiments, we defined a "superfeature" as a feature that comprises
multiple subfeatures, which are usually acquired or skipped jointly and which
have a single cost. Furthermore, we assumed a subset of features is available
at no cost (free features) and set fixed acquisition costs $c_{acq}$ for the
remaining features. A prediction was to be performed at each time-step, which
corresponds to the setting described in Appendix J. We chose misclassification
costs such that good policies must find a balance between feature acquisition
cost and predictive value of the features.

We evaluated and compared the described methods on a synthetic dataset. The
features are distributed according to:

$\displaystyle
X_{(1),i}^{t}=\begin{cases}\gamma_{i}X_{(1),i}^{t-1}+(1-\gamma_{i})\epsilon_{i},&\text{if
}t>0\\\ \epsilon_{i},&\text{if }t=0.\end{cases}$

where $\epsilon_{i}\sim\mathcal{N}(0,\sigma)$. The labels are distributed
according to

$\displaystyle p(Y^{t}=1)=\begin{cases}1,&\text{if
}\zeta_{1}\sum_{i}X_{(1),i}^{t}+\zeta_{2}\sum_{i}X_{(1),i}^{t-1}>0\\\
0.3,&\text{otherwise}.\end{cases}$

This choice for $Y$ simulates a scenario where not all data points are equally
easy to classify.

The distribution of the missingness indicators for the MAR scenario follows
the logistic model as specified in Table 3. To evaluate the convergence of
different estimators, we consider the average cost of running the AFA agent on
the dataset over all data points in the ground truth test set (without
missingness) as the true expected cost $J$. This corresponds to estimating $J$
using Eq. 9 with a Monte Carlo estimate for
$\hat{\mathbb{E}}\left[C_{(\pi_{\alpha})}|X_{(1)},Y\right]$ and the ground
truth data without missingness (i.e. samples from $p(X_{(1)},Y)$).

###  K.2 Training

We used an impute-then-regress classifier [55] with unconditional mean
imputation and a logistic regression classifier for the classification task
and trained it on the available and further randomly subsampled data (where
$p(A_{i}^{t}=1)=0.5$). We tested random and fixed acquisition policies that
acquire each costly feature with a 50% or 100% probability. Furthermore, we
evaluated a vanilla Deep Q-Network (DQN) RL agent [54] which was trained on
the semi-offline sampling distribution $p^{\prime}$ using $\pi_{\alpha}$ as
the semi-offline sampling policy, but without adjustment for the blocking of
actions. The datasets were split into training set (for the training of the
DQN agent and the classifier), nuisance function training set, and test set,
where the estimators were evaluated. The splitting of the dataset in a
nuisance function training set and a test set is necessary due to the
complexity of the used nuisance model functions classes [67]. The resulting
loss of efficiency may, however, be avoided using a cross-fitting approach
[67].

Data and environment
---
Sample size $n_{D}$ |  $100^{\prime}000$ divided into $30\%$ training set (for DQN agent and classifier), $30\%$ nuisance function training set, and $40\%$ test set.
Superfeatures |  super$X_{0}$: $[X_{0}]$, super$X_{1}$: $[X_{1}]$, super$X_{2}$: $[X_{2},X_{3}]$
Label |  $Y^{t}\in\\{0,1\\}$ and for $t\leq T=3$.
Data generation parameters |  $\gamma_{i}=0.5$ $\forall i$, $\sigma=1$, $\zeta_{1}=1$, $\zeta_{2}=0.5$
Feature acquisition cost |  $c_{acq}=[0,1,1]$
Misclassification cost |  $c_{mc}=12$
Missingness mechanisms
MAR |  $p(A_{0}^{t}=1)=1.0,$ $p(A_{1}^{t}=1)=\sigma(0.6-0.6X_{(1),0}^{t-1}-0.1X_{(1),1}^{t-1})-0.1X_{(1),2}^{t-1}),$ $p(A_{2}^{t}=1)=\sigma(0.6-0.6X_{(1),0}^{t-1}-0.1X_{(1),1}^{t-1})-0.1X_{(1),2}^{t-1})$ Complete cases ratio: $p(A=\vec{1})=8.05\%$
MCAR |  $p(A_{0}^{t}=1)=1.0,$ $p(A_{1}^{t}=1)=0.2,$ $p(A_{2}^{t}=1)=0.2$ Complete cases ratio: $p(A=\vec{1})=0.007\%$
Models
Classifier |  Logistic regression
Agents |  Random $50\%$, Fixed $100\%$) DQN (learning rate: 0.0001, number of layers: 2, hidden layer neurons per layer: 16, hidden layer activation function: ReLU)
Nuisance functions |  $\hat{\pi}_{\beta}$ (logistic regression) $Q_{Semi}$ (learning rate: 0.001, number of layers: 2, hidden layer neurons per layer: 16, hidden layer activation function: ReLU )
Table 3: Full experiment details

## References

  * [1] Karthika Mohan, Judea Pearl, and Jin Tian.  Graphical Models for Inference with Missing Data.  In Advances in Neural Information Processing Systems, volume 26, 2013.
  * [2] Ilya Shpitser, Karthika Mohan, and Judea Pearl.  Missing data as a causal and probabilistic problem.  In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, UAI’15, pages 802–811, Arlington, Virginia, USA, July 2015. AUAI Press.
  * [3] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.  arXiv:2005.01643 [cs, stat], November 2020.  arXiv: 2005.01643.
  * [4] Nathan Kallus and Masatoshi Uehara.  Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes.  Journal of Machine Learning Research, 21(167):1–63, 2020.
  * [5] Irving H. LaValle.  On cash equivalents and information evaluation in decisions under uncertainty Part II: Incremental information decisions.  Journal of the American Statistical Association, 63(321):277–284, 1968.  Publisher: Taylor & Francis.
  * [6] Irving H. LaValle.  On cash equivalents and information evaluation in decisions under uncertainty part I: Basic theory.  Journal of the American Statistical Association, 63(321):252–276, 1968.  Publisher: Taylor & Francis.
  * [7] John P Gould.  Risk, stochastic preference, and the value of information.  Journal of Economic Theory, 8(1):64–84, May 1974.
  * [8] Ronald W. Hilton.  The Determinants of Cost Information Value: An Illustrative Analysis.  Journal of Accounting Research, 17(2):411–435, 1979.  Publisher: [Accounting Research Center, Booth School of Business, University of Chicago, Wiley].
  * [9] James Hess.  Risk and the Gain from Information.  Journal of Economic Theory, 27(1):231–238, 1982.
  * [10] Jeffrey M. Keisler, Zachary A. Collier, Eric Chu, Nina Sinatra, and Igor Linkov.  Value of information analysis: the state of application.  Environment Systems and Decisions, 34(1):3–23, March 2014.
  * [11] Alvin I. Mushlin and Lou Fintor.  Is screening for breast cancer cost-effective?  Cancer, 69(S7):1957–1962, 1992.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/1097-0142%2819920401%2969%3A7%2B%3C1957%3A%3AAID-CNCR2820691716%3E3.0.CO%3B2-T.
  * [12] Murray D. Krahn, John E. Mahoney, Mark H. Eckman, John Trachtenberg, Stephen G. Pauker, and Allan S. Detsky.  Screening for Prostate Cancer: A Decision Analytic View.  JAMA, 272(10):773–780, September 1994.
  * [13] Marc F. Botteman, Chris L. Pashos, Alberto Redaelli, Benjamin Laskin, and Robert Hauser.  The health economics of bladder cancer.  PharmacoEconomics, 21(18):1315–1330, December 2003.
  * [14] US Preventive Services Task Force*.  Screening for breast cancer: US Preventive Services Task Force recommendation statement.  Annals of internal medicine, 151(10):716–726, 2009.  Publisher: American College of Physicians.
  * [15] Lin Liu, Zach Shahn, James M. Robins, and Andrea Rotnitzky.  Efficient Estimation of Optimal Regimes Under a No Direct Effect Assumption.  Journal of the American Statistical Association, 116(533):224–239, January 2021.
  * [16] James Robins, Liliana Orellana, and Andrea Rotnitzky.  Estimation and extrapolation of optimal treatment and testing strategies.  Statistics in Medicine, 27(23):4678–4721, 2008.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.3301.
  * [17] Romain Neugebauer, Julie A. Schmittdiel, Alyce S. Adams, Richard W. Grant, and Mark J. van der Laan.  Identification of the Joint Effect of a Dynamic Treatment Intervention and a Stochastic Monitoring Intervention Under the No Direct Effect Assumption.  Journal of Causal Inference, 5(1):20160015, September 2017.
  * [18] Noémi Kreif, Oleg Sofrygin, Julie A. Schmittdiel, Alyce S. Adams, Richard W. Grant, Zheng Zhu, Mark J. van der Laan, and Romain Neugebauer.  Exploiting nonsystematic covariate monitoring to broaden the scope of evidence about the causal effects of adaptive treatment strategies.  Biometrics, 77(1):329–342, 2021.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/biom.13271.
  * [19] Qi An, Hui Li, Xuejun Liao, and Lawrence Carin.  Active feature acquisition with POMDP models.  Submitted to Pattern Recognition Letters, 2006.
  * [20] Yang Li and Junier Oliva.  Active Feature Acquisition with Generative Surrogate Models.  In Proceedings of the 38th International Conference on Machine Learning, pages 6450–6459. PMLR, July 2021.  ISSN: 2640-3498.
  * [21] Yang Li, Siyuan Shan, Qin Liu, and Junier B. Oliva.  Towards Robust Active Feature Acquisition.  arXiv:2107.04163 [cs], July 2021.
  * [22] Chun-Hao Chang, Mingjie Mai, and Anna Goldenberg.  Dynamic Measurement Scheduling for Event Forecasting using Deep RL.  In Proceedings of the 36th International Conference on Machine Learning, pages 951–960. PMLR, May 2019.
  * [23] Hajin Shim, Sung Ju Hwang, and Eunho Yang.  Joint Active Feature Acquisition and Classification with Variable-Size Set Encoding.  Advances in Neural Information Processing Systems, 31, 2018.
  * [24] Haiyan Yin, Yingzhen Li, Sinno Jialin Pan, Cheng Zhang, and Sebastian Tschiatschek.  Reinforcement Learning with Efficient Active Feature Acquisition.  arXiv:2011.00825 [cs], November 2020.
  * [25] Jinsung Yoon, James Jordon, and Mihaela Schaar.  ASAC: Active Sensing using Actor-Critic models.  In Machine Learning for Healthcare Conference, pages 451–473. PMLR, October 2019.  ISSN: 2640-3498.
  * [26] Jinsung Yoon, William R. Zame, and Mihaela Van Der Schaar.  Deep sensing: Active sensing using multi-directional recurrent neural networks.  In International Conference on Learning Representations, 2018.
  * [27] Fengyi Tang, Lifan Zeng, Fei Wang, and Jiayu Zhou.  Adversarial Precision Sensing with Healthcare Applications.  In 2020 IEEE International Conference on Data Mining (ICDM), pages 521–530, November 2020.
  * [28] Daniel Jarrett and Mihaela van der Schaar.  Inverse Active Sensing: Modeling and Understanding Timely Decision-Making.  arXiv:2006.14141 [cs, stat], June 2020.
  * [29] Sriraam Natarajan, Srijita Das, Nandini Ramanan, Gautam Kunapuli, and Predrag Radivojac.  On Whom Should I Perform this Lab Test Next? An Active Feature Elicitation Approach.  In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, pages 3498–3505, Stockholm, Sweden, July 2018. International Joint Conferences on Artificial Intelligence Organization.
  * [30] Srijita Das, Rishabh Iyer, and Sriraam Natarajan.  A Clustering based Selection Framework for Cost Aware and Test-time Feature Elicitation.  In 8th ACM IKDD CODS and 26th COMAD, pages 20–28. ACM, January 2021.
  * [31] Yang Li and Junier B. Oliva.  Dynamic Feature Acquisition with Arbitrary Conditional Flows.  arXiv:2006.07701 [cs, stat], March 2021.
  * [32] Pin Zhang.  A novel feature selection method based on global sensitivity analysis with application in machine learning-based prediction model.  Applied Soft Computing, 85:105859, 2019.  Publisher: Elsevier.
  * [33] Wenbo Gong, Sebastian Tschiatschek, Sebastian Nowozin, Richard E Turner, José Miguel Hernández-Lobato, and Cheng Zhang.  Icebreaker: Element-wise Efficient Information Acquisition with a Bayesian Deep Latent Gaussian Model.  In Advances in Neural Information Processing Systems, 2019.
  * [34] Jaromír Janisch, Tomáš Pevný, and Viliam Lisý.  Classification with costly features as a sequential decision-making problem.  Machine Learning, 109(8):1587–1615, August 2020.
  * [35] Xiaoyong Chai, Lin Deng, Qiang Yang, and C. X. Ling.  Test-cost sensitive naive Bayes classification.  In Fourth IEEE International Conference on Data Mining (ICDM’04), pages 51–58, November 2004.
  * [36] Miroslav Dudik, John Langford, and Lihong Li.  Doubly Robust Policy Evaluation and Learning, May 2011.  arXiv:1103.4601 [cs, stat].
  * [37] Philip Thomas and Emma Brunskill.  Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning.  In Proceedings of The 33rd International Conference on Machine Learning, pages 2139–2148. PMLR, June 2016.  ISSN: 1938-7228.
  * [38] James Robins.  A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect.  Mathematical Modelling, 7(9):1393–1512, January 1986.
  * [39] Daniel G. Horvitz and Donovan J. Thompson.  A generalization of sampling without replacement from a finite universe.  Journal of the American statistical Association, 47(260):663–685, 1952.  Publisher: Taylor & Francis.
  * [40] Shaun R. Seaman and Ian R. White.  Review of inverse probability weighting for dealing with missing data.  Statistical methods in medical research, 22(3):278–295, 2013.  Publisher: Sage Publications Sage UK: London, England.
  * [41] Jonathan A. C. Sterne, Ian R. White, John B. Carlin, Michael Spratt, Patrick Royston, Michael G. Kenward, Angela M. Wood, and James R. Carpenter.  Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls.  BMJ, 338:b2393, June 2009.  Publisher: British Medical Journal Publishing Group Section: Research Methods & Reporting.
  * [42] Li-Fang Cheng, Niranjani Prasad, and Barbara E. Engelhardt.  An Optimal Policy for Patient Laboratory Tests in Intensive Care Units.  In Biocomputing 2019, pages 320–331. WORLD SCIENTIFIC, October 2018.
  * [43] Chaojie An, Qifeng Zhou, and Shen Yang.  A reinforcement learning guided adaptive cost-sensitive feature acquisition method.  Applied Soft Computing, page 108437, January 2022.
  * [44] Gabriel Erion, Joseph D. Janizek, Carly Hudelson, Richard B. Utarnachitt, Andrew M. McCoy, Michael R. Sayre, Nathan J. White, and Su-In Lee.  CoAI: Cost-Aware Artificial Intelligence for Health Care.  Technical report, medRxiv, January 2021.
  * [45] Ellen C. Caniglia, James M. Robins, Lauren E. Cain, Caroline Sabin, Roger Logan, Sophie Abgrall, Michael J. Mugavero, Sonia Hernández-Díaz, Laurence Meyer, Remonie Seng, Daniel R. Drozd, George R. Seage III, Fabrice Bonnet, Fabien Le Marec, Richard D. Moore, Peter Reiss, Ard van Sighem, William C. Mathews, Inma Jarrín, Belén Alejos, Steven G. Deeks, Roberto Muga, Stephen L. Boswell, Elena Ferrer, Joseph J. Eron, John Gill, Antonio Pacheco, Beatriz Grinsztejn, Sonia Napravnik, Sophie Jose, Andrew Phillips, Amy Justice, Janet Tate, Heiner C. Bucher, Matthias Egger, Hansjakob Furrer, Jose M. Miro, Jordi Casabona, Kholoud Porter, Giota Touloumi, Heidi Crane, Dominique Costagliola, Michael Saag, and Miguel A. Hernán.  Emulating a trial of joint dynamic strategies: An application to monitoring and treatment of HIV-positive individuals.  Statistics in Medicine, 38(13):2428–2446, 2019.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.8120.
  * [46] Patrick Rockenschaub, Ela Marie Akay, Benjamin Gregory Carlisle, Adam Hilbert, Falk Meyer-Eschenbach, Anatol-Fiete Näher, Dietmar Frey, and Vince Istvan Madai.  Generalisability of AI-based scoring systems in the ICU: a systematic review and meta-analysis.  preprint, Intensive Care and Critical Care Medicine, October 2023.
  * [47] Patrick Rockenschaub, Adam Hilbert, Tabea Kossen, Falk von Dincklage, Vince Istvan Madai, and Dietmar Frey.  From Single-Hospital to Multi-Centre Applications: Enhancing the Generalisability of Deep Learning Models for Adverse Event Prediction in the ICU, April 2023.  arXiv:2303.15354 [cs].
  * [48] Thomas S. Verma and Judea Pearl.  Equivalence and Synthesis of Causal Models.  Technical Report R-150, Department of Computer Science, University of California, Los Angeles, 1990.
  * [49] Miguel A Hernán and James M Robins.  Causal Inference: What If.  CRC Boca Raton, FL, 2020.
  * [50] Dimitri Bertsekas.  Dynamic programming and optimal control: Volume I, volume 1.  Athena scientific, 2012.
  * [51] Yan Zhou, Roderick J. A. Little, and John D. Kalbfleisch.  Block-Conditional Missing at Random Models for Missing Data.  Statistical Science, 25(4), November 2010.
  * [52] Rohit Bhattacharya, Razieh Nabi, Ilya Shpitser, and James M. Robins.  Identification In Missing Data Models Represented By Directed Acyclic Graphs.  In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, pages 1149–1158. PMLR, August 2020.
  * [53] Razieh Nabi, Rohit Bhattacharya, and Ilya Shpitser.  Full Law Identification In Graphical Models Of Missing Data: Completeness Results.  arXiv:2004.04872 [cs, stat], August 2020.
  * [54] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533, February 2015.
  * [55] Marine Le Morvan, Julie Josse, Erwan Scornet, and Gael Varoquaux.  What’s a good imputation to predict with missing values?  In Advances in Neural Information Processing Systems, volume 34, pages 11530–11540, 2021.
  * [56] Henrik von Kleist, Alireza Zamanian, Ilya Shpitser, and Narges Ahmidi.  Evaluation of Active Feature Acquisition Methods for Static Feature Settings, December 2023.  arXiv:2312.03619 [cs, stat].
  * [57] Maya L Petersen, Kristin E Porter, Susan Gruber, Yue Wang, and Mark J van der Laan.  Diagnosing and responding to violations in the positivity assumption.  Statistical methods in medical research, 21(1):31–54, February 2012.
  * [58] Burr Settles.  Active learning literature survey.  2009\.
  * [59] Thomas T. Kok, Rachel M. Brouwer, Rene M. Mandl, Hugo G. Schnack, and Georg Krempl.  Active Selection of Classification Features.  arXiv:2102.13636 [cs], February 2021.  arXiv: 2102.13636.
  * [60] Sheng-Jun Huang, Miao Xu, Ming-Kun Xie, Masashi Sugiyama, Gang Niu, and Songcan Chen.  Active Feature Acquisition with Supervised Matrix Completion.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1571–1579, July 2018.
  * [61] Christian Beyer, Maik Büttner, Vishnu Unnikrishnan, Miro Schleicher, Eirini Ntoutsi, and Myra Spiliopoulou.  Active feature acquisition on data streams under feature drift.  Annals of Telecommunications, 75(9-10):597–611, October 2020.
  * [62] Chao Ma, Sebastian Tschiatschek, Konstantina Palla, Jose Miguel Hernandez-Lobato, Sebastian Nowozin, and Cheng Zhang.  EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAE.  In Proceedings of the 36th International Conference on Machine Learning, pages 4234–4243. PMLR, May 2019.
  * [63] Anastasios A. Tsiatis.  Semiparametric theory and missing data.  Springer series in statistics. Springer, New York, 2006.
  * [64] Peter J. Bickel, Chris AJ Klaassen, Peter J. Bickel, Ya’acov Ritov, J. Klaassen, Jon A. Wellner, and YA’Acov Ritov.  Efficient and adaptive estimation for semiparametric models, volume 4.  Springer, 1993.
  * [65] Edward H. Kennedy.  Semiparametric theory.  arXiv:1709.06418 [stat], September 2017.  arXiv: 1709.06418.
  * [66] Stef van Buuren.  Multiple imputation of discrete and continuous data by fully conditional specification.  Statistical Methods in Medical Research, 16(3):219–242, June 2007.
  * [67] Edward H. Kennedy.  Semiparametric doubly robust targeted double machine learning: a review.  arXiv:2203.06469 [stat], March 2022.  arXiv: 2203.06469.