# The costs and benefits of uniformly valid causal inference with high-
dimensional nuisance parameters

Niloofar Moosavi  Jenny Häggström  Xavier de Luna  Umeå School of Business,
Economics and Statistics at Umeå University  Department of Statistics, USBE,
Umeå University, 901 87, Umeå, Sweden<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Important advances have recently been achieved in developing procedures
yielding uniformly valid inference for a low dimensional causal parameter when
high-dimensional nuisance models must be estimated. In this paper, we review
the literature on uniformly valid causal inference and discuss the costs and
benefits of using uniformly valid inference procedures. Naive estimation
strategies based on regularisation, machine learning, or a preliminary model
selection stage for the nuisance models have finite sample distributions which
are badly approximated by their asymptotic distributions. To solve this
serious problem, estimators which converge uniformly in distribution over a
class of data generating mechanisms have been proposed in the literature. In
order to obtain uniformly valid results in high-dimensional situations,
sparsity conditions for the nuisance models need typically to be made,
although a double robustness property holds, whereby if one of the nuisance
model is more sparse, the other nuisance model is allowed to be less sparse.
While uniformly valid inference is a highly desirable property, uniformly
valid procedures pay a high price in terms of inflated variability. Our
discussion of this dilemma is illustrated by the study of a double-selection
outcome regression estimator, which we show is uniformly asymptotically
unbiased, but is less variable than uniformly valid estimators in the
numerical experiments conducted.

Double robustness,

Machine learning,

Post-model selection inference,

Regularization,

Superefficiency,

###### keywords:

, and

##  1 Introduction

High-dimensional situations, where the number of covariates is larger than the
number of observations are common in causal inference applications. Using
regularization type estimators such as lasso (Tibshirani, 1996) or other post-
model selection estimators are popular strategies in such cases. Important
advances have been achieved in developing procedures yielding uniformly valid
inference (defined below) for a low dimensional causal parameter when high-
dimensional nuisance models must be fitted (e.g., van der Laan and Rubin,
2006; Belloni et al., 2014; van der Laan, 2014; Farrell, 2015; Chernozhukov et
al., 2018). In this paper, we review the literature on uniformly valid causal
inference, and discuss the costs and benefits of using uniformly valid
inference procedures. This discussion is important since naive and invalid
post-model selection inference is to this day still common in statistical
practice.

Leeb and Pötscher (2005) demonstrated how a data-driven model selection step
can affect the distribution of the estimate of a parameter of interest.
Loosely, they show that the scaled ($\sqrt{n}$) bias of a naive two step
estimator, which does not take into account the selection step, goes to
infinity or stay bounded for a sequence of worst case scenario data generating
processes (DGPs), when relying on consistent or conservative model selection,
respectively. We say that such a naive estimator is not uniformly unbiased. An
estimator with associated uniformly valid inference, on the other hand, is
such that its distribution $F_{n}$ converges uniformly over a set of DGPs
$\mathcal{P}$, i.e., for any $u\in\mathbb{R}$

$\lim_{n\rightarrow\infty}\sup\limits_{P\in\mathcal{P}}|F_{n}(u)-F_{P}(u)|=0,$

where $F_{P}$ is the cumulative distribution derived from $P$. Hereafter, we
use the terms uniformly valid inference and valid inference interchangeably.

Valid inference for every single parameter in a linear regression model needs
careful consideration in high-dimensional settings. Some have considered
debiasing lasso for a valid inference that target the true data generating
process parameters (Javanmard and Montanari, 2014; Van de Geer et al., 2014;
Zhang and Zhang, 2014) and others have considered valid inference conditional
on the model that has been selected (Berk et al., 2013; Lee et al., 2016). In
a causal inference context, there is typically a low dimensional parameter of
interest, e.g., the average causal effect of a treatment, and high-dimensional
nuisance parameters. Belloni et al. (2014) proposed an estimator with valid
inference for a causal parameter in a linear model explaining outcome with a
treatment variable and a set of covariates, which can be of high dimension. To
achieve uniformly valid inference, they proposed to include the union of two
sets of covariates in the model: one obtained by selecting covariates relevant
when regressing the outcome on the covariates, and the second by selecting
covariates relevant when regressing the treatment on the covariates. Their
model implicitly implies a homogeneous causal effect. This can be relaxed to
allow for individual heterogeneous effects, using the potential outcome
framework (Neyman, 1923; Rubin, 1974), and nuisance models for both the
potential outcomes and for the treatment assignment given the covariates
(propensity score). For this general case, van der Laan and Rubin (2006) and
van der Laan (2014) obtained valid inference for a causal parameter using
targeted maximum likelihood estimation, where nuisance models are estimated
nonparametrically. Farrell (2015) considered the augmented inverse probability
of treatment weighting estimator (Robins et al., 1994), and showed that
uniformly valid inference is achieved when using post-lasso estimation for the
nuisance models. Similar results were derived in Chernozhukov et al. (2018)
using a double machine learning approach.

In both Belloni et al. (2014) and Farrell (2015), approximate sparsity is
assumed for the nuisance models. However, in Farrell (2015) the outcome model
can be less sparse if the propensity score is more sparse and vice versa
(called nonparametric double robustness property, Kennedy, 2016). Yet
consistency in the nonparametric estimation of all the nuisance models is
required, a condition relaxed in van der Laan (2014), and in more recent work
(Avagyan and Vansteelandt, 2021; Tan, 2020), where one of the nuisance models
may be inconsistently estimated.

Procedures yielding uniformly valid inference for a causal parameter, in the
general context of heterogeneous treatment effects, allow for the selection of
instruments (loosely, variables related to the treatment but not the outcome)
in the fit of the propensity score model. This is known to result in possibly
large inflation of the variance of the estimators (e.g., Hahn, 2004; de Luna
et al., 2011; Schnitzer et al., 2016; Rotnitzky and Smucler, 2020). Thus,
while uniformly valid inference is a highly desirable property, uniformly
valid procedures pay a high price in terms of inflated variability. We discuss
and illustrate this dilemma by studying a compromise solution, an outcome
regression estimator (e.g., Tan, 2007), which we allow to select instruments,
but which does not use the fitted propensity score, in contrast with uniformly
valid estimators proposed in the literature. The resulting post-model
selection estimator is shown to be uniformly asymptotically unbiased under a
commonly used product rate condition (Farrell, 2015), even though the
propensity score is not used in the estimator except for the covariate
selection step.

This paper is organised as follows. Section 2 presents a review of the
literature on uniformly valid causal inference. Section 3 gives a theoretical
discussion of the costs and benefits of uniformly valid inference, by studying
a double-selection outcome regression estimator. Section 4 illustrates this
discussion with a Monte Carlo study of finite sample properties of a
collection of estimators. Section 5 concludes the paper. All proofs are
delayed to an appendix.

##  2 Uniformly valid causal inference: a review

This is an extremely active research area. The focus is here on uniformly
valid inference on a low dimensional causal parameter after
regularization/model selection of high-dimensional nuisance models. We start
by introducing some general concepts, and then review first cornerstone work
on the homogeneous and then general heterogeneous case. A review of important
advances in recent years concludes this section.

The parameter of interest is a causal effect of a binary treatment variable
$T$ on an outcome $Y$ as defined below in different contexts. We use the
notation $X$ to denote a one dimensional pretreatment covariate and $\bm{X}$
to denote a set of pretreatment covariates which has dimension $p$, allowed to
grow with $n$. Note that the set may contain not only the covariates but also
transformations of them. We consider a set of identically and independently
distributed (i.i.d.) observations, $\\{(\bm{x}_{i},y_{i},t_{i})\\}^{n}_{i=1}$,
drawn from a distribution $P_{n}$. To study uniformly valid post-model
selection inference it is essential that the probability law $P_{n}$ is
allowed to vary with the sample size $n$. We use further the notations
$E_{n}[w_{i}]=\frac{1}{n}\Sigma_{i=1}^{n}w_{i}$ and $a\vee b=\max\\{a,b\\}$.
Moreover, $n_{t}$ denotes the number of individuals under treatment.

###  2.1 Hodges estimator and superefficiency

Superefficient estimators of a parameter of a model $\mathcal{M}$ are variants
of the well known Hodges estimator (van der Vaart, 1997) when a model
restriction holds; meaning that a model $\mathcal{M}_{0}\subset\mathcal{M}$
contains the true data generating process. The asymptotic variance of a
superefficient estimator is smaller than the efficiency bound for the class of
regular asymptotic linear (RAL) estimators of the parameter under model
$\mathcal{M}$, when there is a submodel $\mathcal{M}_{0}$ under which the same
bound is smaller. Superefficiency has a cost in the sense that the asymptotic
distribution of a superefficient estimator is valid only pointwise at
$\mathcal{M}_{0}$ instead of uniformly over a larger family of models
(uniformly valid inference).

This was highlighted by Leeb and Pötscher (2005) in a parametric setting (see
below for a detailed exposition), where a consistent model selection step that
selects out a ”redundant” variable (not part of $\mathcal{M}_{0}$) before a
maximum likelihood fit results in a superefficent estimator. Such an estimator
has an oracle property in the sense that it asymptotically (only pointwise at
$\mathcal{M}_{0}$ instead of uniformly over a larger family of models)
performs as well as a fictitious orcale estimator which can be constructed by
knowing $\mathcal{M}_{0}$ (Fan and Li, 2001).

###  2.2 Homogenous causal effect

As a primer, consider the parametric regression model $y_{i}=\alpha
t_{i}+\beta x_{i}+\epsilon_{i}$, with
$\epsilon_{i}\sim\mathcal{N}(0,\sigma^{2}),\,\sigma^{2}>0$. According to Leeb
and Pötscher (2005), if we are interested in $\alpha$, the post-selection
estimator which includes a preliminary consistent model selection step on $X$
(i.e., a test whether $\beta=0$) is more efficient than the simple OLS
estimator without this step if corr$(X,T)\neq 0$. However, if $\beta\neq 0$
and corr$(X,T)\neq 0$, the finite sample distribution of the post-selection
estimate is a mixture of two normal distributions, i.e., not well approximated
by the normal asymptotic distribution. This is because in the selection step
the nonzero coefficient can be detected for some samples and not detected for
others. If the resulting omitted variable bias is considerable, the empirical
coverage of a naive confidence interval can be far from the nominal coverage.
Leeb and Pötscher (2005) have shown that the minimal coverage of the naive
confidence interval with respect to all possible $\beta$ values goes to zero
as $n$ grows for consistent model selection steps, while the empirical
coverage for any fixed $\beta$ value goes to the nominal one. Their result
highlights the importance of uniformly valid inference compared to pointwise
asymptotic results.

For $\alpha$ to have a causal interpretation, the linear regression needs to
include all confounders as formally defined in next section, and it must be
correctly specified as a model for $E(Y\mid T,\bm{X})$ (in particular implying
a homogeneous/constant causal effect). Concerning the former condition, the
number of available covariates, hence potential confounders, may be very large
when using large observational databases. The latter condition, implicitly
requires that series expansions need to be used to approximate $E(Y\mid
T,\bm{X})$ increasingly well with increasing sample sizes. Therefore, these
two conditions often yield a high-dimensional setting in practice, i.e. where
the number of covariates is at least as large as the sample size.

Belloni et al. (2014) proposed a strategy for reaching valid inference in such
high-dimensional settings (Leeb and Pötscher, 2008). Their suggestion is a
two-step lasso-based method, where the union of covariate sets selected by two
distinct lasso regressions of $Y$ and $T$ on the covariates, respectively
(often called double selection), are utilized in a second step in the main
linear model including $T$ as a regressor. Instead of exact sparsity
conditions typically used in high-dimensional settings, they consider the
following approximate sparsity conditions. Let

$E(Y|T,\bm{X})=\alpha T+\beta_{Y}^{\prime}\bm{X}+R_{Y},$ (2.1)
$E(T|\bm{X})=\beta_{T}^{\prime}\bm{X}+R_{T},$ (2.2)

where $R_{f}$ is the specification error of using a sparse $\beta_{f}$ with
only $s_{f}$ nonzero elements, respectively for $f=Y,T$. The regularity
conditions to obtain uniform valid inference include

$E(E_{n}[R_{f,i}^{2}])^{1/2}=O\big{(}\sqrt{s_{f}/n}\big{)},$
$\log^{3}p/n=o(1),$ $s_{f}^{2}\log^{2}(p\vee n)/n=o(1).$

In other words, models (2.1) and (2.2) are assumed well approximated by a
sparse linear combination of the covariate vector $\bm{X}$. Under these
conditions (and other regularity conditions), Belloni et al. (2014, Corollary
2) showed that their estimator of $\alpha$ is asymptotically normal uniformly
over $P_{n}$, thereby uniformly valid inference can be made.

###  2.3 Heterogeneous causal effect

The effect of a binary treatment is now allowed to be heterogeneous using the
Neyman-Rubin potential outcome framework (Neyman, 1923; Rubin, 1974). For any
unit in the study, denote $Y(1)$ its potential outcome under treatment
($T=1$), and $Y(0)$ its potential outcome without treatment (or alternative
treatment). We assume that $Y=TY(1)+(1-T)Y(0)$ is the observed outcome, and no
interference between units is allowed (stable unit treatment value assumption;
Rubin, 1990). Each unit may have a different causal effect Y(1) - Y(0), and
the average causal effect $\tau=E(Y(1)-Y(0))$ is the parameter of interest in
the sequel. This parameter is identified given the following assumptions.

######  Assumption 1 (No unobserved confounding).

$Y(1),Y(0)\perp\\!\\!\\!\perp{T|\bm{X}}.$

######  Assumption 2 (Overlap).

$\mathbb{P}(T=t|\bm{X})\geq p_{min}>0,\qquad t=0,1.$

Thus, all confounding covariates are included in $\bm{X}$ and all units in the
study have non-zero probability to be included in both treatment groups. These
assumptions are made throughout the article. Note, however, that a sensitivity
analysis (see Díaz et al., 2018, and references therein) should accompany
inference based on Assumption 1, even when many covariates are available,
since this assumption is not testable without further information (e.g., de
Luna and Johansson, 2014). Assumption 2 has also important implications as was
recently demonstrated in D’Amour et al. (2021) for high-dimensional
situations, where overlap is linked to the sparsity conditions discussed
herein.

Let $E(Y|T=1,\bm{X})=m_{1}(\bm{X})$ and $E(Y|T=0,\bm{X})=m_{0}(\bm{X})$ denote
the outcome models, $E(T|\bm{X})=\mathbb{P}(T=1|\bm{X})=e(\bm{X})$ denotes the
propensity score model. One of the earliest proposals that addressed inference
on $\tau$ when estimating nuisance models nonparametrically is the targeted
maximum likelihood estimator (TMLE; van der Laan and Rubin, 2006; van der Laan
and Rose, 2011). Denote fits of the nuisance models $\mathbb{P}(T=1|\bm{X})$,
$E(Y(1)|\bm{X})$ and $E(Y(0)|\bm{X})$ by $\hat{e}(\bm{x}_{i})$,
$\hat{m}^{0}_{1}(\bm{x}_{i})$ and $\hat{m}^{0}_{0}(\bm{x}_{i})$, respectively.
Then, a fluctuated version of the predicted outcome values is used in the
following manner:

$\hat{\tau}_{TMLE}=E_{n}[\hat{m}^{1}_{1}(\bm{x}_{i})-\hat{m}^{1}_{0}(\bm{x}_{i})],$

where the fluctuations are found by

$\operatorname{logit}\hat{m}^{1}_{t}(\bm{x}_{i})=\operatorname{logit}\hat{m}^{0}_{t}(\bm{x}_{i})+\varepsilon_{n}h_{t}(\bm{x}_{i}),\qquad
t\in\\{0,1\\},$

where

$h_{t}(\bm{x}_{i})=\dfrac{\mathbb{1}\\{t=1\\}}{\hat{e}(\bm{x}_{i})}-\dfrac{\mathbb{1}\\{t=0\\}}{1-\hat{e}(\bm{x}_{i})},$

and $\varepsilon_{n}$ is found by running logistic regression of outcome $Y$
on $h_{T}(\bm{X})$ using $\operatorname{logit}\hat{m}^{0}_{T}(\bm{X})$ as
intercept. TMLE is consistent if either $e(\cdot)$ or $m_{0}(\cdot)$ and
$m_{1}(\cdot)$ are consistently estimated. Moreover, it is RAL and
semiparametrically efficient if all models are consistently estimated and a
product of rate of convergence similar to (2.9) hold. TMLE can also be
constructed by iteratively fluctuating the propensity score and the outcome
models, thereby yielding a RAL estimator when at most one of the models is
consistently estimated, i.e. this TMLE is then not only consistent but also
asymptotically normal (so called double robust statistical inference, van der
Laan, 2014).

In another major contribution, Farrell (2015) showed how to obtain uniformly
valid inference for the popular doubly robust augmented inverse probability of
treatment weighting (AIPW, Robins et al., 1994; Scharfstein et al., 1999)
estimator:

$\hat{\tau}_{DR}=E_{n}\left[\dfrac{t_{i}y_{i}-(t_{i}-\hat{e}(\bm{x}_{i}))\hat{m}_{1}(\bm{x}_{i})}{\hat{e}(\bm{x}_{i})}-\dfrac{(1-t_{i})y_{i}+(t_{i}-\hat{e}(\bm{x}_{i}))\hat{m}_{0}(\bm{x}_{i})}{1-\hat{e}(\bm{x}_{i})}\right].$
(2.3)

The fitted values $\hat{m}_{0}(\bm{x}_{i})$, $\hat{m}_{1}(\bm{x}_{i})$ and
$\hat{e}(\bm{x}_{i})$ are obtained using, e.g., post-lasso estimators. Farrell
(2015) proposed the use of group-lasso to benefit from grouped sparsity
patterns among potential outcomes and different treatment levels. Similar to
Belloni et al. (2014), Farrell (2015) assumed approximate sparsity, but for
logistic propensity score and linear potential outcome models, i.e.,

$m_{t}(\bm{X},\eta_{t})=\eta^{\prime}_{t}\bm{X}+R_{Y}^{t},\qquad
t\in\\{0,1\\},$ (2.4)
$e(\bm{X},\gamma)=\text{expit}(\gamma^{\prime}\bm{X}+R_{T}),$

where $R_{Y}^{t}$ and $R_{T}$ are approximation errors of estimating the true
models with sparse parameters $\eta_{t}$ and $\gamma$ that have $s_{Y}^{t}$
and $s_{T}$ nonzero elements, respectively. Farrell’s regularity conditions on
the specification errors are slightly different from those in Belloni et al.
(2014). In particular, he assumes

$E_{n}[(r_{Y,i}^{t})^{2}]^{1/2}\vee
E_{n_{t}}[(r_{Y,i}^{t})^{2}]^{1/2}\leq\mathcal{R}_{Y}^{t}=O(\sqrt{s_{Y}^{t}/n}),\qquad
t\in\\{0,1\\},$ (2.5)

and

$E_{n}[(\text{expit}(\gamma^{\prime}\bm{x}_{i})-\text{expit}(\gamma^{\prime}\bm{x}_{i}+r_{T,i}))^{2}]^{1/2}\leq\mathcal{R}_{T}=O(\sqrt{s_{T}/n}).$
(2.6)

More importantly, the sparsity assumption required for each nuisance model
separately is weaker compared to the one assumed by Belloni et al. (2014),
i.e.,

$s_{f}\log(p\vee n)^{3/2+\delta}=o(n),\qquad
s_{f}\in\\{s_{T},s_{Y}^{0},s_{Y}^{1}\\},$ (2.7)

for some $\delta>0$, since here we have a multiplicative rate condition

$s_{Y}^{t}s_{T}\log(p\vee n)^{3+2\delta}=o(n),\qquad t\in\\{0,1\\}.$ (2.8)

Thus, if the potential outcome models are more sparse, the propensity score
model is allowed to be more dense and the other way around. These sparsity
assumptions and other regularity conditions result in the following rates of
convergence for the post-lasso estimators of the nuisance models (Farrell,
2015, Section 6)

$E_{n}[(\hat{e}(\bm{x}_{i})-e(\bm{x}_{i}))^{2}]=o_{P_{n}}(1),\quad
E_{n}[(\hat{m}_{t}(\bm{x}_{i})-m_{t}(\bm{x}_{i}))^{2}]=o_{P_{n}}(1),$
$\displaystyle
E_{n}[(\hat{e}(\bm{x}_{i})-e(\bm{x}_{i}))^{2}]^{1/2}E_{n}[(\hat{m}_{t}(\bm{x}_{i})-m_{t}(\bm{x}_{i}))^{2}]^{1/2}=o_{P_{n}}(n^{-1/2}),$
(2.9)

for $t\in\\{0,1\\}$. Under these consistency and product rate conditions (and
other regularity assumptions) the estimator (2.3) is asymptotically normal
uniformly over $P_{n}$ (Farrell, 2015, Corollary 3). This result is not
restricted to the post-lasso estimator, but ensures $\sqrt{n}$-consistency of
the AIPW estimator of the low dimensional parameter of interest $\tau$ for any
estimator of nuisance models which fulfills the assumptions. High-dimensional
parametric or nonparametric nuisance models fit into this framework even
though the estimation cannot be done at the $\sqrt{n}$-rate.

The presentation above has focused on robustness to the danger of including
too few covariates (regularization bias), in settings where we believe in
sparsity assumptions. Another possible source of bias arises from the danger
of overfitting. This is a problem when the nuisance functions are too complex
(e.g., cannot be assumed to belong to a Donsker class; e.g., Díaz, 2019;
Kennedy, 2016). A general solution to avoid overfitting error, and thereby
obtain valid inference, is to use sample-splitting; see, Bickel (1982); Zheng
and van der Laan (2011); Cattaneo et al. (2018, 2019).

###  2.4 Advances in recent years

Recent years have witnessed a great deal of novel results in the field of
uniformly valid causal inference. Chernozhukov et al. (2018) readdressed valid
inference for the average causal effect under the framework of double/debiased
machine learners in light of the fact that the parameter $\tau$ satisfies a
Neyman orthogonal moment condition (Neyman, 1959, 1979); a moment condition
that is not sensitive to local errors in nuisance models and can be derived
using the first order influence function of the parameter (Bickel et al.,
1998; Tsiatis, 2007). They suggest sample splitting which together with the
above Neyman orthogonality leads to ignorable remainder terms even when
machine learning nuisance estimators are converging relatively slowly. Other
works have considered a Neyman orthogonal estimating equation of a
$l^{2}$-continuous functional of a conditional expectation (Chernozhukov et
al., 2020; Bradic et al., 2019). In this setting, both the conditional
expectation and the Reisz representer of the functional, which is the inverse
propensity score in the case of $\tau$, must be estimated. However, in the
latter case, the estimation of the inverse propensity score is performed
differently compared to Chernozhukov et al. (2018); i.e., using the equation
which characterize the nuisance model as a Riesz representer.

The notion of double robustness of an estimator has been widely used to
indicate that an estimator is consistent if at least one of the nuisance
models is estimated consistently, not necessarily both (Robins et al., 1994;
Bang and Robins, 2005). This property has also been called parametric double
robustness (Kennedy, 2016). As mentioned in the previous section, van der Laan
(2014, Section 4) was the first work which addressed what they called double
robust statistical inference (also called model double robustness in Smucler
et al., 2019), which indicates that inference on the parameter of interest
requires consistent estimation of only one of the nuisance models. The AIPW
estimator mentioned in the previous section based on regularized maximum
likelihood nuisance estimators has the nonparametric double robust property
(Kennedy, 2016), also called rate double robustness in Smucler et al. (2019),
whereby weak consistency of the nuisance models is required for uniformly
valid inference, but slower convergence in estimating the propensity score can
be bought out by faster convergence in the outcome models, and vice versa
(2.9). However, this AIPW estimator does not yield double robust statistical
inference. Alternative loss functions have been considered in the estimation
of nuisance models, which endow the AIPW estimator double robust statistical
inference (e.g. Avagyan and Vansteelandt, 2021; Tan, 2020; Ning et al., 2020;
Bradic et al., 2019). Smucler et al. (2019) extends this area of work by
generalizing the property of double robust inference to the estimation of all
parameters that belong to what is called the class of bilinear influence
function (BIF) functionals. These estimators are specific to sparse settings
and employ $l_{1}$-regularized estimators. Moreover, they consider a sparsity
condition even for the limit of a possibly inconsistent nuisance model
estimator.

The BIF class includes important causal parameters such as the average
treatment effect and the average treatment effect among treated and cover the
classes of parameters studied in Chernozhukov et al. (2020) and Robins et al.
(2008). However, the parameters who enjoy a uniform valid inference are not
limited to this class. For example, the continuously differentiable functions
of the functionals with bilinear influence function do not belong to the class
while validity of inference for those can be directly shown by the delta
method (Smucler et al., 2019). A different technique has been used for
constructing a valid confidence interval for a parameter outside the BIF
class, a conditional average treatment effect, which is to invert a chi
squared distributed double robust test statistic (Dukes and Vansteelandt,
2020).

Most of the above literature on uniformly valid causal inference is concerned
with high-dimensional settings, which can arise both because of a large set of
covariates is available, but also because functions/transformations of these
covariates are considered in generalized linear (in the parameters) nuisance
models with a sparsity property. However, alternative regularity conditions,
e.g. smoothness, may be considered attractive. For example, neural networks
can be used for smooth nuisance functions belonging to Sobolev spaces (Farrell
et al., 2021). Some of the results in the above-mentioned papers are not
specific to $l_{1}$-regularized estimators and apply to any well-behaved
nonparametric nuisance model estimation (van der Laan and Rubin, 2006;
Farrell, 2015; Chernozhukov et al., 2018). In van der Laan and Rubin (2006)
the choice of a single nonparametric estimator is not considered to be done
apriori, but a data-adaptive cross-validated combination of a set of
estimators (super learner, ensemble learner) was suggested. Finally, more
recently, Cui and Tchetgen Tchetgen (2019) suggest two novel selection
criteria, where the main focus is on getting smaller bias in the estimation of
the target parameter instead of the nuisance ones.

While the literature has focused on situations where the parameter (a causal
effect) is of low dimension, Semenova and Chernozhukov (2020) recently
addresses situations, where the Neyman orthogonal property can be applied to
obtain uniformly valid results in nonparametric situations, i.e. where the
parameter of interest is of infinite dimension. Examples include causal
effects conditional on a continuous covariate and causal effects of a
continuous valued treatment.

##  3 Cost of uniformity and a double-selection outcome regression estimator

One essential component of estimators with uniformly valid inference reviewed
in the latter section is that if there are instruments $-$ here covariates
that explain $T$ although they are not related to $Y$ conditional on the other
covariates included in $\bm{X}$ $-$ these may be part of the selected set of
covariates. This is also obviously true for propensity score centered methods;
see, e.g., Shortreed and Ertefaie (2017). This is unfortunate because the
semiparametric efficiency bound for the average treatment effect is lower if
we have knowledge on which variables are instruments (Hahn, 1998, 2004; de
Luna et al., 2011; Rotnitzky and Smucler, 2020; Tang et al., 2020). The
variance inflation due to instruments can be severe and this has been reported
in the literature numerous times, see, Schnitzer et al. (2016) and references
therein.

In the sequel, we provide a discussion and results which shed new light on
this issue by presenting an estimation strategy which seems to yield a
compromise between the estimators for which we have a uniformly valid
asymptotic distribution (using instruments) and superefficient estimators,
where irrelevant instruments are selected away by using the data.

For simplicity, consider as parameter of interest $\tau_{1}=E(Y(1))$. However,
the results for $\tau_{0}=E(Y(0))$ and thereby $\tau=\tau_{1}-\tau_{0}$ are
analogous. Let $\bm{x}=[\bm{x}_{1},\cdots,\bm{x}_{n}]^{\prime}$,
$\bm{y}=[y_{1},\cdots,y_{n}]^{\prime}$, superscript $T$ denotes subsetting
rows that correspond to treated individuals and subscript $S$ denotes
subsetting columns using the set $S$. Then,
$P^{T}_{S}=\bm{x}^{T}_{S}(\bm{x}^{T\prime}_{S}\bm{x}^{T}_{S})^{-1}\bm{x}^{T\prime}_{S}$
is the projection matrix onto the space spanned by $\bm{x}^{T}_{S}$, while
$\tilde{P}^{T}_{S}=\bm{x}_{S}(\bm{x}^{T\prime}_{S}\bm{x}^{T}_{S})^{-1}\bm{x}^{T\prime}_{S}$
is the matrix used in predicting $Y(1)$ for all $n$ individuals. Given a
selected covariate set $S$, we define the post-selection outcome regression
(OR) estimator as

$\hat{\tau}_{1,OR}(S)=E_{n}[(\tilde{P}^{T}_{S}\bm{y}^{T})_{i}]=\dfrac{1}{n}\Sigma_{i=1}^{n}[\bm{x}_{S,i}(\bm{x}^{T\prime}_{S}\bm{x}^{T}_{S})^{-1}\bm{x}^{T\prime}_{S}\bm{y}^{T}]:=\dfrac{1}{n}\Sigma_{i=1}^{n}[\hat{m}_{1}(\bm{x}_{{S},i})].$

A classical post-selection OR estimator is
$\hat{\tau}_{1,OR_{XY}}=\hat{\tau}_{1,OR}(X_{Y})$, where $X_{Y}$ is a set of
covariates derived from the $y-\bm{x}$ association using any covariate
selection strategy; for instance, the set of covariates that corresponds to
nonzero coefficients in a fitted lasso regression of $y$ versus $\bm{x}$ may
be considered. Instead, we study here theoretically the post-double-selection
OR estimator

$\hat{\tau}_{1,OR_{DS}}=\hat{\tau}_{1,OR}(X_{Y}\cup X_{T}),$ (3.1)

where $X_{T}$ is derived by fitting the $t-\bm{x}$ association using lasso or
any other covariate selection strategy. Note that this can be considered as a
generalization of Belloni et al. (2014) early estimator for the homogeneous
case presented in Section 2.2 to the general situation of an heterogeneous
treatment effect. Although this estimator is mentioned in the simulation
experiments run in Athey et al. (2018), no theoretical results are available
in the literature up to our knowledge.

Estimator (3.1) is not asymptotically linear and cannot be shown to yield
uniformly valid inference as was the case for the post-selection double robust
estimators described in Section 2.3. However, uniform fast rate of decay of
the bias can be guaranteed. The conditions used below are of the type used to
show uniform validity. In particular, a product convergence rate condition is
used.

######  Theorem 1.

Suppose

  1. (i)

$E_{n}[(1-t_{i}a(\bm{x}_{i}))(\tilde{P}^{T}_{S}m_{1}(\bm{x}^{T})-{m}_{1}(\bm{x}))_{i}]=o_{P_{n}}(n^{-v_{1}}),$

  2. (ii)

$E_{n_{t}}[((\mathbb{1}_{n_{t}\times
n_{t}}-P^{T}_{S})a(\bm{x}^{T}))_{i}^{2}]^{1/2}E_{n_{t}}[((\mathbb{1}_{n_{t}\times
n_{t}}-P^{T}_{S})m_{1}(\bm{x}^{T}))_{i}^{2}]^{1/2}=o_{p}(n^{-v_{2}}),$

where $\mathbb{1}_{n_{t}\times n_{t}}$ is the identity matrix of size $n_{t}$,
$a(\bm{X})=1/\mathbb{P}(T=1|\bm{X})$. Let $v=\min(v_{1},v_{2})$. Then,

$\displaystyle\text{Bias}(\hat{\tau}_{1,OR})=E(E_{n}[\hat{m}_{1}(\bm{x}_{S,i})]-\tau_{1})=o(n^{-v}).$

The first condition (i) requires that the order of the scaled error for all
the individuals is equal to the order of the scaled error on the treated
individuals weighted by inverse propensity scores, similar to Farrell (2015,
Assumption 3(c)). The second condition (ii) is a multiplicative rate condition
similar to the multiplicative rate condition in Farrell (2015, Assumption
3(b)). However, notice that here the rate must be fulfilled using the same set
of covariates in both nuisance models. This necessitates doing double
selection to get the double robustness property in terms of bias. The proof of
Theorem 1 can be found in Appendix A.1.

To illustrate how a double-selection procedure can benefit in terms of bias,
suppose

$m_{1}(\bm{X},\eta_{1})=\eta^{\prime}_{1}\bm{X}+R_{Y},$ (3.2)

and

$a(\bm{X},\gamma)=\gamma^{\prime}\bm{X}+R_{T}.$ (3.3)

Here, we consider a linear model for the inverse of propensity score model
(e.g. Imbens et al., 2005), and $R_{Y}$ and $R_{t}$ are approximation errors
of the true models with sparse coefficients $\eta_{1}$ and $\gamma$ in the
outcome and inverse propensity score models, respectively. The following
corollary is a direct consequence of Theorem 1 and the asymptotic results for
lasso regression in Farrell (2015, Section 6).

######  Corollary 1.

Suppose that models (3.2) and (3.3), hold, where the conditions (2.5), (2.6),
(2.7) and (2.8) are fulfilled. Moreover, assume the regularity conditions in
Farrell (2015, Corollary 5 and Appendix F.3). Consider
$\hat{\tau}_{1,OR_{DS}}$ in (3.1) where $X_{Y}$ and $X_{T}$ are estimated
using lasso regression of the observed potential outcome (for treated
individuals) and the inverse propensity score on $X$, respectively. Then,

$\text{Bias}(\hat{\tau}_{1,OR_{DS}})=o(n^{-1/2}).$

The above result shows that root-$n$ decay of the bias can be derived
uniformly over a set of DGPs. In this sense, the double-selection OR estimator
may be seen as a compromise between single selection estimators
(superefficient and no uniformly decaying bias) and the double-selection
estimators of Section 2 (with available uniformly valid asymptotic
distribution). The implications in terms of finite sample behaviour are
studied below with Monte Carlo experiments.

In practice, the inverse propensity score $a$ is not observed and (3.3) cannot
be fitted directly. Instead, we suggest to fit a lasso logistic regression for
the propensity score to retrieve the relevant covariates.

##  4 Simulation study

The aim of this simulation study is to illustrate the above theoretical
discussion on the cost and benefits of uniformly valid inference. While many
simulations studies are available in the literature reviewed above, their
focus is either to show the necessity of using uniformly valid procedures in
order to avoid regularization bias, or to illustrate the variance inflation
due to the use of instruments in the propensity score compared to
superefficient procedures. Here, we contrast these two aspects by considering
both uniformly valid and superefficient post-selection strategies, as well as
the double-selection outcome regression estimator (3.1).

###  4.1 Simulation design

We use 500 replicates in all situations, and consider sample sizes $n$ = 500,
1000, 1500, 2000. The covariate vector $\bm{X}$ is generated from a
multivariate normal distribution with zero mean and identity covariance
matrix. The dimension $p$ of the covariate vector equals $n$. Results for low
dimensional settings, $p<<n$, portray a similar general picture and are
available from the authors upon request. Data generation and all computations
are performed with the software R (R Core Team, 2019).

####  4.1.1 High-dimensional setting

We consider the following models

$Y(0)=m_{0}(\bm{X})+\epsilon_{0}=1+\eta_{0}^{\prime}\bm{X}+\epsilon_{0},$
$Y(1)=m_{1}(\bm{X})+\epsilon_{1}=2+\eta_{1}^{\prime}\bm{X}+\epsilon_{1},$
$\mathbb{P}(T=1|\bm{X})=e(\bm{X})=\text{expit}(\gamma^{\prime}\bm{X}),$

indexed by the parameter vectors

$\eta_{0}=\frac{k}{2}\cdot(1,1/2,1/3,1/4,1/5,0,0,0,0,0,1,1/2,1/3,1/4,1/5,0,\cdots,0),$
$\eta_{1}=k\cdot(1,1/2,1/3,1/4,1/5,0,0,0,0,0,1,1/2,1/3,1/4,1/5,0,\cdots,0),$
$\gamma=(1,1/2,1/3,1/4,1/5,1,1,1,1,1,0,\cdots,0),$

where $k\in\\{0.1,0.4,0.8,1.2\\}$ and the error terms,
$\epsilon_{t},\,\,t=0,1$, are generated from a normal distribution with
$E(\epsilon_{t}|\bm{X})=0$ and
$Var(\epsilon_{t}|\bm{X})=(1+p)^{-1}(1+\iota^{\prime}\cdot\bm{X}^{2})$ where
$\iota$ is the vector of ones. The parameter $k$ determines the strength of
the association between the outcome and the covariates.

####  4.1.2 Post-selection estimators of $\tau$

We use lasso as implemented in the R package hdm (Chernozhukov et al., 2016)
to estimate the nuisance models $m_{t}(\bm{X})$ and $e(\bm{X})$. We denote the
sets of variables which corresponds to nonzero coefficients in the estimated
sparse linear outcome models and logistic propensity score by
$X_{Y}=X_{Y_{0}}\cup X_{Y_{1}}$, the union of the sets estimated by each of
the two potential outcome models, and $X_{T}$, respectively. The lasso penalty
parameter is selected as
$\lambda=2.2\sqrt{n}\Phi^{-1}([\mbox{log}(n)-0.1][2p\mbox{log}(n)]^{-1})$
(Belloni et al., 2012).

With the purpose of estimating $\tau$, using different combinations of the
above covariate sets, we compare two versions of the OR estimator, three
versions of the doubly robust AIPW estimator and one version of the doubly
robust TMLE estimator. Specifically, OR${}_{X_{Y}}$ and ORDS use $X_{Y}$ and
the union $X_{T}\cup X_{Y}$ in the outcome models refitting steps,
respectively. AIPW${}_{X_{Y}}$ uses $X_{Y}$ in the propensity score and
outcome model refitting steps, AIPWDS uses the union $X_{T}\cup X_{Y}$ in both
the propensity score and outcome models (as in Belloni et al., 2014, Section
5) and AIPW${}_{X_{Y},X_{T}}$ uses $X_{T}$ in the propensity score model and
$X_{Y}$ in the outcome models (as in Farrell, 2015). TMLE${}_{X_{Y},X_{T}}$
uses $X_{T}$ in the propensity score model and $X_{Y}$ in the outcome models.

Given the same consistency and product rate conditions on the initial
estimators of nuisance models as in Farrell (2015), TMLE${}_{X_{Y},X_{T}}$ has
the same uniformly valid asymptotic distribution as AIPW${}_{X_{Y},X_{T}}$
(van der Laan and Rose, 2011, Chapter 27). Hence, if the refitted models used
in AIPW${}_{X_{Y},X_{T}}$ are used as initial model estimates in
TMLE${}_{X_{Y},X_{T}}$ we would expect similar results for large samples. In
summary, AIPW${}_{X_{Y},X_{T}}$, AIPWDS and TMLE${}_{X_{Y},X_{T}}$ have
uniformly valid asymptotic distributions, OR${}_{X_{Y}}$ and AIPW${}_{X_{Y}}$
have no such uniform validity, select away instruments and are superefficient,
while ORDS has uniformly decaying bias (Corollary 1). For TMLE, we use the R
package tmle (Gruber and van der Laan, 2012) and do not truncate the estimated
propensity scores, i.e., gbound = c(0,1). For the other estimators, we use own
written R code as well as the R package ui (Genbäck and de Luna, 2019) for the
variances.

###  4.2 Results

From 500 replicates we compute empirical biases, standard errors, root mean
squared errors (RMSE), and empirical coverages. We also compute mean estimated
standard errors. Table 1 presents results for $k=0.4$. Results for all values
of $k$ are given for bias, RMSE and coverages in the Appendix, Tables 2-4.

We see that estimators selecting away instruments (OR${}_{X_{Y}}$ and
AIPW${}_{X_{Y}}$) have lower Monte Carlo standard error but at the cost of
larger bias and poor empirical coverages (clearly lower than nominal level).
As expected the other estimators, which all have uniformly decaying bias, show
low bias, but at the cost of larger standard error. This cost is, however,
smallest for ORDS, and AIPW${}_{X_{Y},X_{T}}$ , AIPWDS and
TMLE${}_{X_{Y},X_{T}}$ have standard errors roughly up to five times as large
as OR${}_{X_{Y}}$, while for ORDS the increase in standard error is not as
severe (1.22 times the standard error of OR${}_{X_{Y}}$). All the low-bias
estimators have good empirical coverages, at least for sample sizes 1000 and
higher, although we do not have such theoretical guarantee for ORDS.

On a side note, we observe that the estimated standard errors of
AIPW${}_{X_{Y},X_{T}}$, AIPWDS and TMLE${}_{X_{Y},X_{T}}$ are distinctively
smaller than the Monte Carlo standard errors, but the difference is reduced
when increasing sample size. For ORDS no such underestimation of the variance
is observed.

Table 1: Results of 500 simulation replicates for estimators of $\tau$, for varying sample sizes $n$, number of covariates $p=n$ and $k=0.4$. RMSE, root mean-squared error; Bias; SE, Monte Carlo standard deviation; ESE, estimated standard error (influence curve based estimates, ignoring the variability in the selection step); CP, empirical coverage probability of 95% confidence intervals. $n$ | Estimator | RMSE | Bias | SE | ESE | CP
---|---|---|---|---|---|---
500 | OR${}_{X_{Y}}$ | 0.26 | -0.19 | 0.18 | 0.16 | 0.74
| AIPW${}_{X_{Y}}$ | 0.26 | -0.19 | 0.18 | 0.16 | 0.74
| ORDS | 0.22 | -0.05 | 0.22 | 0.21 | 0.94
| AIPW${}_{X_{Y},X_{T}}$ | 0.95 | -0.06 | 0.95 | 0.35 | 0.93
| AIPWDS | 0.88 | -0.06 | 0.88 | 0.34 | 0.91
| TMLE${}_{X_{Y},X_{T}}$ | 0.29 | -0.08 | 0.28 | 0.24 | 0.91
1000 | OR${}_{X_{Y}}$ | 0.19 | -0.14 | 0.14 | 0.11 | 0.72
| AIPW${}_{X_{Y}}$ | 0.19 | -0.14 | 0.14 | 0.11 | 0.73
| ORDS | 0.16 | -0.02 | 0.16 | 0.15 | 0.94
| AIPW${}_{X_{Y},X_{T}}$ | 0.38 | -0.00 | 0.38 | 0.25 | 0.97
| AIPWDS | 0.37 | -0.00 | 0.37 | 0.25 | 0.95
| TMLE${}_{X_{Y},X_{T}}$ | 0.21 | -0.04 | 0.20 | 0.18 | 0.95
1500 | OR${}_{X_{Y}}$ | 0.15 | -0.11 | 0.11 | 0.09 | 0.76
| AIPW${}_{X_{Y}}$ | 0.15 | -0.11 | 0.11 | 0.09 | 0.76
| ORDS | 0.12 | -0.02 | 0.12 | 0.12 | 0.95
| AIPW${}_{X_{Y},X_{T}}$ | 0.22 | -0.03 | 0.22 | 0.18 | 0.95
| AIPWDS | 0.24 | -0.02 | 0.24 | 0.18 | 0.95
| TMLE${}_{X_{Y},X_{T}}$ | 0.15 | -0.04 | 0.15 | 0.14 | 0.92
2000 | OR${}_{X_{Y}}$ | 0.13 | -0.09 | 0.10 | 0.08 | 0.80
| AIPW${}_{X_{Y}}$ | 0.13 | -0.09 | 0.10 | 0.08 | 0.80
| ORDS | 0.10 | -0.01 | 0.10 | 0.10 | 0.95
| AIPW${}_{X_{Y},X_{T}}$ | 0.19 | -0.02 | 0.19 | 0.16 | 0.94
| AIPWDS | 0.19 | -0.02 | 0.19 | 0.16 | 0.93
| TMLE${}_{X_{Y},X_{T}}$ | 0.15 | -0.03 | 0.15 | 0.13 | 0.92

##  5 Discussion

To obtain an unbiased estimator of the average causal effect of a treatment,
we need to control for all confounders. Knowing that some covariates are
related to the outcome, but not to the treatment does not change the
semiparametric efficiency bound. However, knowledge on the existence of
instruments has implications on the asymptotic variance that can be achieved
by unbiased estimators (Hahn, 2004; de Luna et al., 2011; Rotnitzky and
Smucler, 2020). In practice, we typically do not have such apriori (to data)
knowledge. Naively selecting away covariates (here instruments) using the data
at hand, by, e.g., regularization yields inference that is not uniformly valid
(may translate into large bias and incorrect coverage rates). On the other
hand, using methods which yield uniformly valid inference may yield large
variability compared to these naive (often superefficient) methods. In this
paper, we have reviewed the literature on uniformly valid causal inference,
and have discussed the costs and benefits of uniformly valid inference. The
latter discussion has been illustrated by studying a double-selection outcome
regression estimator, which is shown to be uniformly asymptotically unbiased
under a product rate condition. This seems to translate into finite sample
properties which are a compromise between uniformly valid and superefficient
estimators. The good properties of the double-selection OR estimator may
arguably be due to the designs considered in our simulations, for which the
outcome regression models can be consistently fitted. Consistency of the
outcome regression model is indeed assumed to show uniformly decaying bias of
the double-selection estimator. This assumption is also used by estimators
which have the nonparametric double robust property. On the other hand,
procedures that yield double robust statistical inference (van der Laan, 2014;
Tan, 2020; Avagyan and Vansteelandt, 2021), allow for one of the nuisance
models to be inconsistent (converges to an incorrect function of the
covariates). This alternative limit need, however, to be assumed sparse in the
covariates, a rather high-level assumption (Smucler et al., 2019).

## Acknowledgements

We acknowledge comments from Tetiana Gorbach, Mohammad Ghasempour and two
anonymous referees that have improved the paper. This work was supported by a
grant from the Marianne and Marcus Wallenberg Foundation.

##  Appendix A Appendix

###  A.1 Proof of Theorem 1

The bias of post selection outcome regression estimator can be expressed as
follows

$\displaystyle\text{Bias}(\hat{\tau}_{1,OR})$
$\displaystyle=E(E_{n}[\hat{m}_{1}(\bm{x}_{S,i})]-\tau_{1})$
$\displaystyle=E(E_{n}[\hat{m}_{1}(\bm{x}_{S,i})-{m}_{1}(\bm{x}_{i})])+E(E_{n}[{m}_{1}(\bm{x}_{i})]-\tau_{1})$
$\displaystyle=E(E_{n}[\hat{m}_{1}(\bm{x}_{S,i})-{m}_{1}(\bm{x}_{i})]),$

where the last equality follows by Assumption 1. To show that the scaled bias
term is asymptotically negligible we show that it is negligible conditional on
$S$ and $\\{t_{i},x_{i}\\}_{i=1}^{n}$.

$\displaystyle
n^{v}\text{Bias}(\hat{\tau}_{1,OR}|S,\\{t_{i},x_{i}\\}_{i=1}^{n})$
$\displaystyle:=n^{v}E(E_{n}[\hat{m}_{1}(\bm{x}_{S,i})-{m}_{1}(\bm{x}_{i})]|S,\\{t_{i},x_{i}\\}_{i=1}^{n})$
$\displaystyle=n^{v}E_{n}[\bm{x}_{S,i}(\bm{x}^{T\prime}_{S}\bm{x}^{T}_{S})^{-1}\bm{x}^{T\prime}_{S}m_{1}(\bm{x}^{T})-{m}_{1}(\bm{x}_{i})]$
$\displaystyle\approx
n^{v}E_{n}[t_{i}a(\bm{x}_{i})(\bm{x}_{S,i}(\bm{x}^{T\prime}_{S}\bm{x}^{T}_{S})^{-1}\bm{x}^{T\prime}_{S}m_{1}(\bm{x}^{T})-{m}_{1}(\bm{x}_{i}))]$
$\displaystyle=\dfrac{n^{v}}{n}a(\bm{x}^{T})^{\prime}(P^{T}_{S}-\mathbb{1}_{n_{t}\times
n_{t}}){m}_{1}(\bm{x}^{T})$
$\displaystyle=\dfrac{n^{v}}{n}((P^{T}_{S}-\mathbb{1}_{n_{t}\times
n_{t}})a(\bm{x}^{T}))^{\prime}(P^{T}_{S}-\mathbb{1}_{n_{t}\times
n_{t}}){m}_{1}(\bm{x}^{T}),$

where $\approx$ means that both side have the same limits which holds by 1(i).

$\displaystyle|n^{v}\text{Bias}(\hat{\tau}_{1,OR}|S,\\{t_{i},x_{i}\\}_{i=1}^{n})|$
$\displaystyle\leq n^{v}E_{n_{t}}[((P^{T}_{S}-\mathbb{1}_{n_{t}\times
n_{t}})a(\bm{x}^{T}))_{i}^{2}]^{1/2}$ $\displaystyle\ \ \ \ \times
E_{n_{t}}[((P^{T}_{S}-\mathbb{1}_{n_{t}\times
n_{t}})m_{1}(\bm{x}^{T}))_{i}^{2}]^{1/2}$ $\displaystyle=o_{P_{n}}(1),$

where the last equality holds by 1(ii). The statement in the theorem follows
by the above result on the order of decay of the conditional expectation and
uniform integrability of this conditional expectation.

###  A.2 Proof of Corollary 1

By construction $X_{Y}\subseteq S$. Therefore by Farrell (2015, Appendix F.3),
(2.5) and the sparsity condition (2.7) we have $v_{1}=1/2$. Moreover, using
Farrell (2015, Corollary 5) , $X_{Y}\subseteq S$, $X_{T}\subseteq S$, and
conditions (2.5), (2.6) and (2.8) we have $v_{2}=1/2$.

###  A.3 Simulation results

Table 2: $\sqrt{n}$-bias based on 500 simulation replicates, for estimators of $\tau$, with varying sample sizes $n$ and values for $k$, and number of covariates $p=n$. |  | $k$ |  |
---|---|---|---|---
$n$ | Estimator | 0.1 | 0.4 | 0.8 | 1.2
500 | OR${}_{X_{Y}}$ | -1.25 | -4.16 | -3.44 | -2.97
| AIPW${}_{X_{Y}}$ | -1.25 | -4.17 | -3.44 | -2.96
| ORDS | -0.26 | -1.08 | -2.09 | -2.56
| AIPW${}_{X_{Y},X_{T}}$ | -0.42 | -1.26 | -2.57 | -3.12
| AIPWDS | -0.35 | -1.45 | -2.49 | -2.44
| TMLE${}_{X_{Y},X_{T}}$ | -0.45 | -1.79 | -2.55 | -2.84
1000 | OR${}_{X_{Y}}$ | -1.81 | -4.32 | -3.14 | -2.28
| AIPW${}_{X_{Y}}$ | -1.81 | -4.31 | -3.15 | -2.27
| ORDS | -0.17 | -0.76 | -1.40 | -1.63
| AIPW${}_{X_{Y},X_{T}}$ | 0.59 | -0.06 | -0.86 | -1.12
| AIPWDS | 0.56 | -0.03 | -0.88 | -1.08
| TMLE${}_{X_{Y},X_{T}}$ | -0.30 | -1.23 | -1.76 | -1.83
1500 | OR${}_{X_{Y}}$ | -2.50 | -4.10 | -3.20 | -2.08
| AIPW${}_{X_{Y}}$ | -2.50 | -4.09 | -3.21 | -2.08
| ORDS | -0.50 | -0.91 | -1.44 | -1.47
| AIPW${}_{X_{Y},X_{T}}$ | -0.66 | -1.02 | -1.36 | -1.55
| AIPWDS | -0.42 | -0.89 | -1.39 | -1.48
| TMLE${}_{X_{Y},X_{T}}$ | -0.75 | -1.38 | -1.71 | -1.59
2000 | OR${}_{X_{Y}}$ | -2.83 | -3.91 | -2.91 | -1.56
| AIPW${}_{X_{Y}}$ | -2.82 | -3.92 | -2.89 | -1.53
| ORDS | -0.29 | -0.60 | -0.91 | -0.87
| AIPW${}_{X_{Y},X_{T}}$ | -0.54 | -0.91 | -1.21 | -1.02
| AIPWDS | -0.44 | -0.81 | -1.08 | -0.85
| TMLE${}_{X_{Y},X_{T}}$ | -0.68 | -1.15 | -1.33 | -1.04
Table 3: RMSE based on 500 simulation replicates, for estimators of $\tau$, with varying sample sizes $n$ and values for $k$, and number of covariates $p=n$. |  | $k$ |  |
---|---|---|---|---
$n$ | Estimator | 0.1 | 0.4 | 0.8 | 1.2
500 | OR${}_{X_{Y}}$ | 0.17 | 0.26 | 0.25 | 0.24
| AIPW${}_{X_{Y}}$ | 0.17 | 0.26 | 0.25 | 0.24
| ORDS | 0.21 | 0.22 | 0.24 | 0.26
| AIPW${}_{X_{Y},X_{T}}$ | 0.85 | 0.95 | 0.95 | 0.88
| AIPWDS | 0.79 | 0.88 | 0.88 | 0.76
| TMLE${}_{X_{Y},X_{T}}$ | 0.28 | 0.29 | 0.31 | 0.32
1000 | OR${}_{X_{Y}}$ | 0.12 | 0.19 | 0.17 | 0.15
| AIPW${}_{X_{Y}}$ | 0.12 | 0.19 | 0.17 | 0.15
| ORDS | 0.15 | 0.16 | 0.17 | 0.17
| AIPW${}_{X_{Y},X_{T}}$ | 0.37 | 0.38 | 0.37 | 0.38
| AIPWDS | 0.36 | 0.37 | 0.38 | 0.38
| TMLE${}_{X_{Y},X_{T}}$ | 0.20 | 0.21 | 0.22 | 0.22
1500 | OR${}_{X_{Y}}$ | 0.11 | 0.15 | 0.13 | 0.11
| AIPW${}_{X_{Y}}$ | 0.11 | 0.15 | 0.13 | 0.11
| ORDS | 0.12 | 0.12 | 0.13 | 0.13
| AIPW${}_{X_{Y},X_{T}}$ | 0.22 | 0.22 | 0.24 | 0.24
| AIPWDS | 0.22 | 0.24 | 0.25 | 0.25
| TMLE${}_{X_{Y},X_{T}}$ | 0.15 | 0.15 | 0.16 | 0.16
2000 | OR${}_{X_{Y}}$ | 0.10 | 0.13 | 0.11 | 0.10
| AIPW${}_{X_{Y}}$ | 0.10 | 0.13 | 0.11 | 0.10
| ORDS | 0.10 | 0.10 | 0.11 | 0.11
| AIPW${}_{X_{Y},X_{T}}$ | 0.18 | 0.19 | 0.19 | 0.19
| AIPWDS | 0.18 | 0.19 | 0.19 | 0.19
| TMLE${}_{X_{Y},X_{T}}$ | 0.15 | 0.15 | 0.15 | 0.15
Table 4: Empirical coverage probability of 95% confidence intervals based on 500 simulation replicates, for estimators of $\tau$, with varying sample sizes $n$ and values for $k$, and number of covariates $p=n$..  |  | $k$ |  |
---|---|---|---|---
$n$ | Estimator | 0.1 | 0.4 | 0.8 | 1.2
500 | OR${}_{X_{Y}}$ | 0.94 | 0.74 | 0.83 | 0.86
| AIPW${}_{X_{Y}}$ | 0.94 | 0.74 | 0.82 | 0.87
| ORDS | 0.93 | 0.94 | 0.92 | 0.92
| AIPW${}_{X_{Y},X_{T}}$ | 0.95 | 0.93 | 0.92 | 0.91
| AIPWDS | 0.92 | 0.91 | 0.90 | 0.90
| TMLE${}_{X_{Y},X_{T}}$ | 0.93 | 0.91 | 0.89 | 0.90
1000 | OR${}_{X_{Y}}$ | 0.93 | 0.72 | 0.84 | 0.89
| AIPW${}_{X_{Y}}$ | 0.93 | 0.73 | 0.84 | 0.90
| ORDS | 0.93 | 0.94 | 0.92 | 0.92
| AIPW${}_{X_{Y},X_{T}}$ | 0.97 | 0.97 | 0.95 | 0.95
| AIPWDS | 0.96 | 0.95 | 0.94 | 0.93
| TMLE${}_{X_{Y},X_{T}}$ | 0.95 | 0.95 | 0.94 | 0.94
1500 | OR${}_{X_{Y}}$ | 0.90 | 0.76 | 0.83 | 0.91
| AIPW${}_{X_{Y}}$ | 0.90 | 0.76 | 0.84 | 0.92
| ORDS | 0.96 | 0.95 | 0.96 | 0.95
| AIPW${}_{X_{Y},X_{T}}$ | 0.96 | 0.95 | 0.95 | 0.94
| AIPWDS | 0.95 | 0.95 | 0.94 | 0.93
| TMLE${}_{X_{Y},X_{T}}$ | 0.95 | 0.92 | 0.93 | 0.92
2000 | OR${}_{X_{Y}}$ | 0.88 | 0.80 | 0.86 | 0.92
| AIPW${}_{X_{Y}}$ | 0.88 | 0.80 | 0.87 | 0.92
| ORDS | 0.95 | 0.95 | 0.94 | 0.95
| AIPW${}_{X_{Y},X_{T}}$ | 0.94 | 0.94 | 0.94 | 0.95
| AIPWDS | 0.94 | 0.93 | 0.94 | 0.94
| TMLE${}_{X_{Y},X_{T}}$ | 0.92 | 0.92 | 0.92 | 0.93

## References

  * Athey et al. (2018) Athey, S., G. W. Imbens, and S. Wager (2018).  Approximate residual balancing: debiased inference of average treatment effects in high dimensions.  Journal of the Royal Statistical Society: Series B (Statistical Methodology) 80(4), 597–623.
  * Avagyan and Vansteelandt (2021) Avagyan, V. and S. Vansteelandt (2021).  High-dimensional inference for the average treatment effect under model misspecification using penalized bias-reduced double-robust estimation.  Biostatistics & Epidemiology doi.org/10.1080/24709360.2021.1898730, 1–18.
  * Bang and Robins (2005) Bang, H. and J. M. Robins (2005).  Doubly robust estimation in missing data and causal inference models.  Biometrics 61(4), 962–973.
  * Belloni et al. (2012) Belloni, A., D. Chen, V. Chernozhukov, and C. Hansen (2012).  Sparse models and methods for optimal instruments with an application to eminent domain.  Econometrica 80(6), 2369–2429.
  * Belloni et al. (2014) Belloni, A., V. Chernozhukov, and C. Hansen (2014).  Inference on treatment effects after selection among high-dimensional controls.  The Review of Economic Studies 81(2), 608–650.
  * Berk et al. (2013) Berk, R., L. Brown, A. Buja, K. Zhang, L. Zhao, et al. (2013).  Valid post-selection inference.  The Annals of Statistics 41(2), 802–837.
  * Bickel (1982) Bickel, P. J. (1982).  On adaptive estimation.  The Annals of Statistics 19, 647–671.
  * Bickel et al. (1998) Bickel, P. J., C. A. Klaassen, Y. Ritov, and J. A. Wellner (1998).  Efficient and adaptive estimation for semiparametric models.  Springer.
  * Bradic et al. (2019) Bradic, J., S. Wager, and Y. Zhu (2019).  Sparsity double robust inference of average treatment effects.  arXiv preprint arXiv:1905.00744.
  * Cattaneo et al. (2019) Cattaneo, M. D., M. Jansson, and X. Ma (2019).  Two-step estimation and inference with possibly many included covariates.  Review of Economic Studies 83(6), 1095–1122.
  * Cattaneo et al. (2018) Cattaneo, M. D., M. Jansson, and W. K. Newey (2018).  Inference in linear regression models with many covariates and heteroskedasticity.  Journal of the American Statistical Association 113(523), 1350–1361.
  * Chernozhukov et al. (2018) Chernozhukov, V., D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins (2018).  Double/debiased machine learning for treatment and structural parameters.  The Econometrics Journal 21(1), C1–C68.
  * Chernozhukov et al. (2016) Chernozhukov, V., C. Hansen, and M. Spindler (2016).  hdm: High-dimensional metrics.  arXiv preprint arXiv:1608.00354.
  * Chernozhukov et al. (2020) Chernozhukov, V., W. Newey, and R. Singh (2020).  De-biased machine learning of global and local parameters using regularized riesz representers.
  * Cui and Tchetgen Tchetgen (2019) Cui, Y. and E. Tchetgen Tchetgen (2019).  Selective machine learning of doubly robust functionals.  arXiv preprint arXiv:1911.02029.
  * D’Amour et al. (2021) D’Amour, A., P. Ding, A. Feller, L. Lei, and J. Sekhon (2021).  Overlap in observational studies with high-dimensional covariates.  Journal of Econometrics 221(2), 644–654.
  * de Luna and Johansson (2014) de Luna, X. and P. Johansson (2014).  Testing for the unconfoundedness assumption using an instrumental assumption.  Journal of Causal Inference 2(2), 187–199.
  * de Luna et al. (2011) de Luna, X., I. Waernbaum, and T. S. Richardson (2011).  Covariate selection for the nonparametric estimation of an average treatment effect.  Biometrika 98(4), 861–875.
  * Díaz (2019) Díaz, I. (2019).  Machine learning in the estimation of causal effects: targeted minimum loss-based estimation and double/debiased machine learning.  Biostatistics 21(2), 353–358.
  * Díaz et al. (2018) Díaz, I., A. R. Luedtke, and M. J. van der Laan (2018).  Sensitivity analysis.  In Targeted Learning in Data Science, pp. 511–522. Springer Series in Statistics. Springer.
  * Dukes and Vansteelandt (2020) Dukes, O. and S. Vansteelandt (2020).  Inference for treatment effect parameters in potentially misspecified high-dimensional models.  Biometrika 10.1093/biomet/asaa071.
  * Fan and Li (2001) Fan, J. and R. Li (2001).  Variable selection via nonconcave penalized likelihood and its oracle properties.  Journal of the American statistical Association 96(456), 1348–1360.
  * Farrell (2015) Farrell, M. H. (2015).  Robust inference on average treatment effects with possibly more covariates than observations.  Journal of Econometrics 189(1), 1–23.
  * Farrell et al. (2021) Farrell, M. H., T. Liang, and S. Misra (2021).  Deep neural networks for estimation and inference.  Econometrica 89(1), 181–213.
  * Genbäck and de Luna (2019) Genbäck, M. and X. de Luna (2019).  Causal inference accounting for unobserved confounding after outcome regression and doubly robust estimation.  Biometrics 75(2), 506–515.
  * Gruber and van der Laan (2012) Gruber, S. and M. J. van der Laan (2012).  tmle: An R package for targeted maximum likelihood estimation.  Journal of Statistical Software 51(13), 1–35.  doi:10.18637/jss.v051.i13.
  * Hahn (1998) Hahn, J. (1998).  On the role of the propensity score in efficient semiparametric estimation of average treatment effects.  Econometrica 66, 315–331.
  * Hahn (2004) Hahn, J. (2004).  Functional restriction and efficiency in causal inference.  Review of Economics and Statistics 86(1), 73–76.
  * Imbens et al. (2005) Imbens, G. W., W. K. Newey, and G. Ridder (2005).  Mean-square-error calculations for average treatment effects.  IEPR working paper 05.34.
  * Javanmard and Montanari (2014) Javanmard, A. and A. Montanari (2014).  Confidence intervals and hypothesis testing for high-dimensional regression.  The Journal of Machine Learning Research 15(1), 2869–2909.
  * Kennedy (2016) Kennedy, E. H. (2016).  Semiparametric theory and empirical processes in causal inference.  In He H., Wu P., Chen D.-G. (eds) Statistical causal inferences and their applications in public health research, pp. 141–167. Springer.
  * Lee et al. (2016) Lee, J. D., D. L. Sun, Y. Sun, J. E. Taylor, et al. (2016).  Exact post-selection inference, with application to the lasso.  The Annals of Statistics 44(3), 907–927.
  * Leeb and Pötscher (2005) Leeb, H. and B. M. Pötscher (2005).  Model selection and inference: Facts and fiction.  Econometric Theory 21(1), 21–59.
  * Leeb and Pötscher (2008) Leeb, H. and B. M. Pötscher (2008).  Sparse estimators and the oracle property, or the return of Hodges’ estimator.  Journal of Econometrics 142(1), 201–211.
  * Neyman (1959) Neyman, J. (1959).  Optimal asymptotic tests of composite statistical hypotheses.  In U. Grenander (Ed.), Probability and Statistics, pp. 416–444. John Wiley.
  * Neyman (1979) Neyman, J. (1979).  $c(\alpha)$ tests and their use.  Sankhya 41, 1–21.
  * Neyman (1923) Neyman, J. S. (1923).  On the application of probability theory to agricultural experiments. essay on principles. section 9.(tlanslated and edited by dm dabrowska and tp speed, statistical science (1990), 5, 465-480).  Annals of Agricultural Sciences 10, 1–51.
  * Ning et al. (2020) Ning, Y., P. Sida, and K. Imai (2020).  Robust estimation of causal effects via a high-dimensional covariate balancing propensity score.  Biometrika 107(3), 533–554.
  * R Core Team (2019) R Core Team (2019).  R: A Language and Environment for Statistical Computing.  Vienna, Austria: R Foundation for Statistical Computing.
  * Robins et al. (2008) Robins, J., L. Li, E. Tchetgen Tchetgen, A. van der Vaart, et al. (2008).  Higher order influence functions and minimax estimation of nonlinear functionals.  In Probability and statistics: essays in honor of David A. Freedman, pp. 335–421. Institute of Mathematical Statistics.
  * Robins et al. (1994) Robins, J. M., A. Rotnitzky, and L. P. Zhao (1994).  Estimation of regression coefficients when some regressors are not always observed.  Journal of the American statistical Association 89(427), 846–866.
  * Rotnitzky and Smucler (2020) Rotnitzky, A. and E. Smucler (2020).  Efficient adjustment sets for population average causal treatment effect estimation in graphical models.  Journal of Machine Learning Research 21(188), 1–86.
  * Rubin (1974) Rubin, D. B. (1974).  Estimating causal effects of treatments in randomized and nonrandomized studies.  Journal of educational Psychology 66(5), 688.
  * Rubin (1990) Rubin, D. B. (1990).  Formal modes of statistical inference for causal effects.  Journal of Statistical Planning and Inference 25, 279–292.
  * Scharfstein et al. (1999) Scharfstein, D., A. Rotnitzky, and J. Robins (1999).  Rejoinder to comments on “adjusting for non-ignorable drop-out using semiparametric non-response models?”.  Journal of the American Statistical Association 94, 1121–1146.
  * Schnitzer et al. (2016) Schnitzer, M. E., J. J. Lok, and S. Gruber (2016).  Variable selection for confounder control, flexible modeling and collaborative targeted minimum loss-based estimation in causal inference.  International Journal of Biostatistics 12(1), 97–115.
  * Semenova and Chernozhukov (2020) Semenova, V. and V. Chernozhukov (2020).  Debiased machine learning of conditional average treatment effects and other causal functions.
  * Shortreed and Ertefaie (2017) Shortreed, S. M. and A. Ertefaie (2017).  Outcome-adaptive lasso: Variable selection for causal inference.  Biometrics 73(4), 1111–1122.
  * Smucler et al. (2019) Smucler, E., A. Rotnitzky, and J. M. Robins (2019).  A unifying approach for doubly-robust $\backslash ell\\_1$ regularized estimation of causal contrasts.  arXiv preprint arXiv:1904.03737.
  * Tan (2007) Tan, Z. (2007).  Comment: Understanding OR, PS and DR.  Statistical Science 22(4), 560–568.
  * Tan (2020) Tan, Z. (2020).  Model-assisted inference for treatment effects using regularized calibrated estimation with high-dimensional data.  Annals of Statistics 48(2), 811–837.
  * Tang et al. (2020) Tang, D., D. Kong, W. Pan, and L. Wang (2020).  Outcome model free causal inference with ultra-high dimensional covariates.  arXiv preprint arXiv:2007.14190.
  * Tibshirani (1996) Tibshirani, R. (1996).  Regression shrinkage and selection via the lasso.  Journal of the Royal Statistical Society: Series B (Methodological) 58(1), 267–288.
  * Tsiatis (2007) Tsiatis, A. (2007).  Semiparametric theory and missing data.  Springer Science & Business Media.
  * Van de Geer et al. (2014) Van de Geer, S., P. Bühlmann, Y. Ritov, R. Dezeure, et al. (2014).  On asymptotically optimal confidence regions and tests for high-dimensional models.  The Annals of Statistics 42(3), 1166–1202.
  * van der Laan (2014) van der Laan, M. J. (2014).  Targeted estimation of nuisance parameters to obtain valid statistical inference.  The International Journal of Biostatistics 10(1), 29–57.
  * van der Laan and Rose (2011) van der Laan, M. J. and S. Rose (2011).  Targeted learning: causal inference for observational and experimental data.  Springer Science & Business Media.
  * van der Laan and Rubin (2006) van der Laan, M. J. and D. Rubin (2006).  Targeted maximum likelihood learning.  The International Journal of Biostatistics 2(1).
  * van der Vaart (1997) van der Vaart, A. W. (1997).  Superefficiency.  In Pollard D., Torgersen E., Yang G.L. (eds) Festschrift for Lucien Le Cam, pp. 397–410. Springer.
  * Zhang and Zhang (2014) Zhang, C.-H. and S. S. Zhang (2014).  Confidence intervals for low dimensional parameters in high dimensional linear models.  Journal of the Royal Statistical Society: Series B (Statistical Methodology) 76(1), 217–242.
  * Zheng and van der Laan (2011) Zheng, W. and M. J. van der Laan (2011).  Cross-validated targeted minimum-loss-based estimation.  In Targeted Learning, pp. 459–474. Springer.


# Legal Transformer Models May Not Always Help

Saibo Geng
EPFL, Switzerland
<EMAIL_ADDRESS>
Rémi Lebret
EPFL, Switzerland
<EMAIL_ADDRESS>
Karl Aberer
EPFL, Switzerland
<EMAIL_ADDRESS>

###### Abstract

Deep learning-based Natural Language Processing methods, especially
transformers, have achieved impressive performance in the last few years.
Applying those state-of-the-art NLP methods to legal activities to automate or
simplify some simple work is of great value. This work investigates the value
of domain adaptive pre-training and language adapters in legal NLP tasks. By
comparing the performance of language models with domain adaptive pre-training
on different tasks and different dataset splits, we show that domain adaptive
pre-training is only helpful with low-resource downstream tasks, thus far from
being a panacea. We also benchmark the performance of adapters in a typical
legal NLP task and show that they can yield similar performance to full model
tuning with much smaller training costs. As an additional result, we release
LegalRoBERTa, a RoBERTa model further pre-trained on legal corpora.

##  1 Introduction

The adoption of natural language processing in the legal domain has a long
history. The earliest systems for searching online legal content appeared in
the 1960s and 1970s, and legal expert systems were a hot topic of discussion
in the 1970s and 1980s.DALE, (2019) NLP has been applied to various legal
areas to automate activities, including Natural Language Understanding based
Contract review, question-answering based Legal advice DALE, (2019). Various
Natural Language tasks can be adapted to the legal domain, including

  * •

Question Answering

  * •

Argument detection, Definition Extraction

  * •

Semantic Annotation

  * •

Classification

In 2018, the release of BERT Devlin et al., (2019) as a pre-trained language
representation has achieved new state-of-the-art results on various NLP tasks.
Later on, domain-adaptive pretraining(DAPT) and task-adaptive
pretraining(TAPT)Gururangan et al., (2020) on the pre-trained model can
further improve the results. Based on this idea, researchers have attempted to
conduct DAPT with legal corpora, LegalBERT from Chalkidis et al., (2020) is a
good example. Later,Liu et al., (2019) argued that BERT is largely under-
trained and suggested a new setup of key hyper-parameters and training schema
to produce RoBERTa. RoBERTa significantly outperforms BERT’s performance on
various tasks. Following the same idea as LegalBERT, we believe a further
domain adaptation with RoBERTa should produce a new state-of-the-art model in
the legal domain. We present LegalRoBERTa, a RoBERTa model further pre-trained
on legal corpora.

However, unsupervised pretraining is a costly action that may take more than
several days, even with good computing resources. Chalkidis et al., (2020)
showed that compared to BERT, legalBERT achieves only slightly better
performances on three different legal NLP tasks:

  * •

Multilabel Classification task on EURLEX57K dataset Chalkidis et al., (2019)

  * •

Multilabel Classification task on ECHR-CASES dataset Chalkidis et al., (2019)

  * •

Named Entity Recognition on CONTRACTS-NER dataset Chalkidis et al., (2021)

Micheli et al., (2020) reported that corpus-specific MLM was not beneficial on
a French QuAD task. If the improvement from DAPT is not apparent compared to
the original model, it is doubtful whether researchers should spend time
collecting data and conducting DAPT every time a new language model is
available. We tested the popular open-source legal language models, including
LegalRoBERTa produced by ourselves, to see if a significant improvement could
be observed. As we will show in the section 5, both legalBERT and legalRoBERTa
demonstrated a limited boost compared to the original language model in the
legal text classification task.

Wang et al., (2020) pointed out that the improvement brought by the pre-
training was related to the data size of downstream tasks. The improvement is
more remarkable when the downstream task is low-resource and vice-versa. As
DAPT is also a particular type of pre-training; we believe that the benefit of
DAPT may also have similar relationships with downstream tasks. To demonstrate
this hypothesis, we tested legal language models on two different tasks: one
rich resource, one low resource as well as the same task with different sizes
of training data. The results showed that our hypothesis held and DAPT was
especially beneficial while downstream task suffers from a lack of data.

Finally, we investigate the performance of a new NLP technique adapters in
legal NLP tasks. The adapter is a more efficient way to fine-tune pre-trained
language models to downstream tasks. It is faster to train and takes less
space on disk. Our experimental results showed that the adapter was able to
produce a comparable performance as fine-tuning the full model.

##  2 Contributions

The contributions of this paper are:

  1. 1.

Inspired by the idea of legalBERT and the success of RoBERTa, we present
legalRoBERTa, a domain-adapted language representation for the legal area. It
was pre-trained on less legal corpora than legalBERT but produced a similar
performance as legalBERT.

  2. 2.

We adapted an existing legal summarization dataset Galgani and Hoffmann,
(2010) to a legal text retrieval task.

  3. 3.

We demonstrated that current open-source legal language models could only
bring marginal benefit or no improvement on a rich resource NLP task. On the
other hand, We showed that DAPT was beneficial when the downstream task was
low resource.

  4. 4.

We tested the performance of adapters Houlsby et al., (2019) on a legal text
classification task and show that they can produce comparable results as fine-
tuning the full model.

##  3 Related Work

Chalkidis et al., (2020) introduced legalBERT as the first transformer adapted
to the legal domain. Our work is trying to fill the gap between BERT and
RoBERTa in the legal area and investigate whether DAPT is truly helpful in
legal NLP tasks. Zheng et al., (2021) reported a similar conclusion on the
relation of DAPT and downstream tasks. Our work has been done simultaneously
with them, and we were not aware of their results until the work is mostly
finished.

##  4 Language Model Pre-training: LegalRoBERTa

###  4.1 Legal Corpora Description

Corpus | Size(raw) | Size(clean)
---|---|---
Patent Litigations | 1.57GB | 1.1GB
Caselaw Access Project | 5.6GB | 2.8GB
Google Patents Public Data | 1.1GB | 1.0GB
Total | 8.3GB | 4.9GB
Table 1: Pre-trained corpora

As the first step to build a legal language model, we tried to collect public
law-related corpora, but there were minimal available resources. As legal
documents could contain sensitive information, institutes usually only release
a small part of the data to the public, and a portion of them are in PDF
format. Finally, we obtained around 5 GB of clean legal text data to proceed
with the domain-adaptive pre-training.

###  4.2 Comparison with Other corpora

Domain | Corpus | #Token | Size(GB)
---|---|---|---
BIOMED | S2ORC | 7.55B | 47
CS | S2ORC | 8.10B | 48
NEWS | REALNEWS | 6.66B | 39
REVIEW | AMAZON reviews | 2.11B | 11
LEGAL | LEGAL-BERT | - | 12
LEGAL | LEGAL-ROBERTA | 1.01B | 4.9
Table 2: Corpora for various domain

Compared to other domain adaptive pre-training experiments, our legal corpora
is significantly smaller.

###  4.3 Pre-training Details

Following Devlin et al., (2019), we run additional pre-training steps of
RoBERTa-BASE on domain-specific corpora. While Devlin et al., (2019) suggested
additional steps up to 100k, our pre-training goes up to 446k as Chalkidis et
al., (2020) suggests that prolonged in-domain pre-training brings a positive
effect to future fine-tuning on downstream tasks.

Fine-tuning configuration:

  * •

learning rate = 5e-5(with learning rate decay, ends at 4.95e-8)

  * •

number of epochs = 3

  * •

Total steps = 446K

  * •

Total flops = 2.7365e18

  * •

Device: 2*GeForce GTX TITAN X (computeCapability= 5.2 )

  * •

RunTime 101 hours

  * •

Per GPU batch size = 2

Loss starts at 1.850 and ends at 0.880 The perplexity on legal corpus after
domain adaptive pre-training = 2.2735

However, given limited graphical memory space, our batch size is significantly
smaller than those used in similar domain adaptive pre-training. We think
further pre-training on the legal corpora should be considered and may be
beneficial, c.f. Table 10.1. RoBERTa-BASE has been pre-trained for
significantly more steps(1M) in generic corpora (e.g., Wikipedia, Children’s
Books); thus, it is highly skewed towards generic language.Chalkidis et al.,
(2020) In Appendix 10.2, we give two concrete examples to demonstrate how
differently is LegalRoBERTa behaving against RoBERTa and LegalBERT from
Chalkidis et al., (2020) performance on Next-Token-Prediction task.

##  5 Downstream Legal NLP Tasks and Model Testing

Model | Authors | HuggingFace url
---|---|---
Legal-BERT | Chalkidis et al., (2020) | nlpaueb/legal-bert-base-uncased
LegalRoBERTa | Our paper | saibo/legal-roberta-base
LegalBERT | Zheng et al., (2021) | zlucia/legalbert
RoBERTa-base | Liu et al., (2019) | roberta-base
BERT-base-unc | Devlin et al., (2019) | bert-base-uncased
rand-RoBERTa | Our paper | saibo/random-roberta-base

Table 3: Tested Language Models with HuggingFace URLs

Model | Precision | Recall | F1 | R@5 | P@5 | RP@5 | NDCG@5
---|---|---|---|---|---|---|---
Legal-BERT | 0.86 | 0.63 | 0.73 | 0.72 | 0.69 | 0.79 | 0.82
LegalRoBERTa | 0.84 | 0.63 | 0.72 | 0.70 | 0.67 | 0.78 | 0.80
LegalBERT | 0.86 | 0.61 | 0.71 | 0.71 | 0.68 | 0.78 | 0.81
RoBERTa-base | 0.85 | 0.65 | 0.74 | 0.72 | 0.69 | 0.79 | 0.82
BERT-base-uncased | 0.86 | 0.62 | 0.72 | 0.72 | 0.69 | 0.79 | 0.82
random-RoBERTa | 0.85 | 0.59 | 0.69 | 0.69 | 0.66 | 0.76 | 0.79

Table 4: Results on Large-Scale Multi-Label Text Classification on EU
Legislation

  * •

Only Legal-BERT from Chalkidis et al., (2020) has slightly outperformed
original BERT. The adapted model from Zheng et al., (2021) is slightly below
the original BERT, so is legalRoBERTa against original RoBERTa.

To investigate whether legal language models are better compared with normal
language models. We selected six open-source language models available on
HuggingFace, including two adapted BERT models from different authors: Legal-
BERT from Chalkidis et al., (2020) and LegalBERT from Zheng et al., (2021),
one adapted RoBERTa model from us, original BERT and RoBERTa models, and
finally, a randomly initialized RoBERTa model for comparison. All six models
are available via HuggingFace API. Their links on HuggingFace are listed in
the Table 3.

We evaluated these models on text classification and information retrieval
using two different datasets. EURLEX57K Chalkidis et al., (2019) is a large-
scale multi-label text classification(LMTC) dataset of EU laws. Legal Case
Reports Data Set Dua and Graff, (2017) is a dataset containing Australian
legal cases from the Federal Court of Australia (FCA) during 2006-2009, which
was built to experiment with automatic summarization and citation analysis.

Split | Documents(D) | Words/D | Labels/D
---|---|---|---
Train | 45k | 729 | 5
Dev | 6k | 714 | 5
Test | 6k | 725 | 5
Total | 57k | 727 | 5
Table 5: Statistics of the EUR-LEX dataset

###  5.1 Large-Scale Multi-Label Text Classification on EU Legislation(rich-
resource)

Experimental setup

Our dataset in this task contains 57K legislative documents from EUR-LEX,
annotated with around 4.3K labels. Each document can be labeled to more than
one label; thus, it is a multi-label classification task. The 4,271 labels are
divided into frequent (746 labels), few-shot (3,362), and zero-shot (163),
depending on whether they were assigned to more than 50, fewer than 50 but at
least one, or no training documents, respectively. The model is composed of a
language model as encoder and an extra classification layer on top. We use
binary cross-entropy as the loss function in this task.The metrics we used in
this task are identical as in the paper Chalkidis et al., (2019).

Train data ratio | Train samples | Model | Precision | Recall | F1 | R@5 | P@5 | RP@5 | NDCG@5
---|---|---|---|---|---|---|---|---|---
100% | 45000 | BERT | 0.86 | 0.62 | 0.72 | 0.72 | 0.69 | 0.79 | 0.82
|  | DAPT improvement | +0.00 | +0.00 | +0.01 | +0.00 | +0.00 | +0.00 | +0.00
|  | relative improvement(%) | 0.0 | 0.0 | 1.4 | 0.0 | 0.0 | 0.0 | 0.0
20% | 9000 | BERT | 0.66 | 0.19 | 0.29 | 0.39 | 0.35 | 0.43 | 0.46
|  | DAPT improvement | +0.04 | +0.00 | +0.00 | 0.01 | 0.00 | 0.00 | +0.00
|  | relative improvement(%) | +6.1 | 0.0 | 0.0 | +2.8 | 0.0 | 0.0 | 0.0
10% | 4500 | BERT | 0.58 | 0.09 | 0.15 | 0.30 | 0.27 | 0.33 | 0.35
|  | DAPT improvement | +0.06 | +0.02 | +0.03 | 0.02 | 0.01 | 0.01 | +0.02
|  | relative improvement(%) | +10.3 | +22 | +20 | +6.7 | +3.7 | +3.0 | +5.7
5% | 2250 | BERT | 0.49 | 0.06 | 0.11 | 0.22 | 0.20 | 0.24 | 0.26
|  | DAPT improvement | +0.0 | +0.01 | +0.02 | 0.02 | 0.02 | 0.02 | +0.02
|  | relative improvement(%) | 0.0 | +17 | +18 | +9.1 | +10 | +8.3 | +7.7
1% | 450 | BERT | 0.00 | 0.00 | 0.00 | 0.03 | 0.02 | 0.03 | 0.02
|  | DAPT improvement | +0.0 | +0.00 | +0.00 | 0.01 | 0.01 | 0.01 | +0.02
|  | relative improvement(%) | 0.0 | 0.0 | 0.0 | +33 | +50 | +33 | +100
Table 6: Improvement on LMTC with different sizes of training data

Results

As shown in the Table 4 only Legal-BERT from Chalkidis et al., (2020) has
slightly outperformed the original BERT. The adapted model LegalBERT from
Zheng et al., (2021) is slightly below the original BERT, so is LegalRoBERTa
against original RoBERTa. This slight margin could be due to statistical error
because we only ran the experiments once with a fixed random seed. However, we
could not pretend that pre-trained models have demonstrated significantly
better performance than the general language models. Chalkidis et al., (2020)
reported similar results to us on the same task: the difference between
legalBERT and BERT was not significant. One can also notice that the
difference between pre-trained RoBERTa and randomly initialized RoBERTa is
also relatively small. This is coherent with the argument from Wang et al.,
(2020): pretraining(or transfer learning in general) is only beneficial when
there is not enough data for downstream tasks.

###  5.2 Automatic Catchphrase Retrieval from Legal Court Case Documents(low-
resource)

Split | Cases
---|---
Train | 2807
Dev | 350
Test | 350
Total | 3507
Table 7: Statistics of the AUS-CASE dataset

Experimental setup

The datasetGalgani and Hoffmann, (2010) contains 3886 Australian legal cases
from the Federal Court of Australia (FCA). The cases are all annotated with
catchphrases, citations sentences, citation catchphrases, and citation
classes. The content which interests us most is the case sentences and
catchphrases. Initially, the catchphrases were used as the gold standard for
summarization experiments. We decide to adopt this dataset to a text retrieval
task because BERT as a bidirectional model is not well suited for text
generation tasks, including summarization. Our goal here is to retrieve the
correct catchphrases based on the description of a case(sentences).

We use a supervised approach to rank and retrieve catchphrases from court case
documents. Our method is inspired by VSE++: Improving Visual-Semantic
Embeddings with Hard Negatives from Faghri et al., (2018). In that work,
image-caption retrieval was conducted using image-caption embedding similarity
rank, on which the model is optimized via the minimization of triplet ranking
loss. We applied the same idea to our catchphrase retrieval task by replacing
the images with case descriptions and captions with catchphrases. An essential
difference between this task and the previous task is the volume of training
data. Comparing the Table 7 and Table 5, the retrieval task has only $1/10$
training data as the text classification task.

As shown in the Figure 10.3, the model in this task consists of an encoder
(pre-trained transformer in our case) to extract features from both
catchphrases and documents plus a dense neural network to transform features
to representation in a shared space. The loss function in this task is the
triplet loss function. Moreover, as we have only one ground true catchphrase,
our task is roughly five times more difficult than the popular MS-COCO image-
captioning task, where each image has five ground true captions on average.

The metric we used in this task is Recall@K (R@K), Mean Rank of the ground-
true label, and median rank of the ground-true label(MeanRank and MedRanks are
the lower, the better). We notice that this metric also depends on the size of
the test database. For example, searching over a one-thousand cases test set
is more challenging than a one-hundred cases test set. In this project, we
focus on the test set of size = 389 cases.

Model | R1 | R5 | R10 | MedRank | MeanRank
---|---|---|---|---|---
BERT | 14.4 | 33.7 | 45.7 | 13.0 | 49.8
| +-0.6 | +-0.8 | +-0.4 | +-0.0 | +-1.1
RoBERTa | 13.6 | 33.1 | 44.7 | 14.0 | 55.5
| +-0.6 | +-0.8 | +-0.9 | +-0.0 | +-2.9
Legal-BERT | 16.8 | 37.7 | 49.9 | 10.6 | 46.9
| +-1.0 | +-0.5 | +-0.3 | +-0.6 | +-1.3
LegalBERT | 15.4 | 34.8 | 46.2 | 12.8 | 47.3
| +-0.2 | +-1.3 | +-1.1 | +-0.4 | +-1.8
LegalRoBERTa | 15.1 | 34.6 | 46.3 | 13.5 | 52.9
| +-1.1 | +-0.9 | +-0.8 | +-1 | +-0.4
Table 8: Results on Automatic Catchphrase Retrieval from Legal Court Case
Documents

Results

As showed in the Table 8, all the domain-pre-trained models have outperformed
the corresponding original model. Legal-BERT from Chalkidis et al., (2020) is
significantly better than the original BERT. This result suggested that
domain-adapted models should be favored in the case of low-resource tasks.

Model | Precision | Recall | F1 | R@5 | P@5 | RP@5 | NDCG@5
---|---|---|---|---|---|---|---
Legal-BERT | 0.86 | 0.63 | 0.73 | 0.72 | 0.69 | 0.79 | 0.82
Adapter (diff) | +0.01 | -0.03 | -0.02 | -0.01 | -0.01 | 0.0 | 0.0
LegalRoBERTa | 0.84 | 0.63 | 0.72 | 0.70 | 0.67 | 0.78 | 0.80
Adapter (diff) | +0.02 | -0.04 | -0.02 | 0.0 | 0.0 | 0.0 | +0.01
LegalBERT | 0.86 | 0.61 | 0.71 | 0.71 | 0.68 | 0.78 | 0.81
Adapter (diff) | -0.01 | 0.0 | 0.0 | 0.0 | 0.0 | +0.01 | +0.01
RoBERTa-base | 0.85 | 0.65 | 0.74 | 0.72 | 0.69 | 0.79 | 0.82
Adapter (diff) | +0.01 | -0.06 | -0.04 | -0.01 | -0.01 | 0.00 | -0.01
Table 9: Performance of adapters on LMTC task

##  6 Varying Downstream task data size

Based on the observation from the previous experiments, we further investigate
the relation between domain-pretraining and downstream task data size. We vary
the size of training data of LMTC tasks and compare the performance of BERT
versus LegalBERT (see Table 10.5).

The results in Table 6 show that domain-pretraining’s performance improvement
is tightly related to the downstream task training data size. The benefit is
significant when the training data is scarce and negligible when the training
data is sufficient.

##  7 Adapters in legal text classification

When there are multiple downstream tasks, it is time-wise inefficient to fine-
tune the whole language model once per task, and it also requires much space
to save the models for each task. Adapter proposed by Houlsby et al., (2019)
is a good alternative to full model fine-tuning. Instead of updating all the
parameters contained in the model, we add a so-called adapter module into the
model. In the training step, only parameters contained in the adapter module
are updated while the language model itself remains unchanged. When training
is finished, one only needs to save the adapter module for each task.

This multi-task scenario is frequent in legal NLP tasks because a legal
activity could be assisted by several relatively simple downstream tasks. We
test the adapter module on various language models with or without domain pre-
training. From the performance perspective, adapters can produce the same
results as fine-tuning the whole model, cf Table 9. However, training adapters
is not faster than fine-tuning the full model because the forward pass and
back-propagation still have a similar amount of calculations as fine-tuning
the whole model.

##  8 Limitations and Future Work

The size of legal corpora available restricted the pre-training of
LegalRoBERTa. To better utilize the potential of RoBERTa, we should consider
collecting more data such as automatic scraping. In the meantime, Chalkidis et
al., (2020) has released some other legal corpora of UK and EU legislative of
roughly 2.5 GB. It should be beneficial to include those data into the pre-
training of LegalRoBERTa v2. Furthermore, the pre-training steps seem to be
insufficient compared with other related work (see 10.1). In the task of
Large-Scale Multi-Label Text Classification on EU Legislation, we evaluated
models with identical hyper-parameters due to limited time and limited
computing resources. A grid search of hyper-parameters and repeated
experiments several times with different random seeds could be considered.

In the task of legal case retrieval, paired statistical testing can be
conducted to conclude whether the domain pre-trained models are significantly
better than the original models.

##  9 Conclusion

In this work, we first tried to answer a critical question in legal NLP from
an empirical perspective: When does domain pre-training help the model to
yield better performance? Through a series of legal NLP experiments, we showed
that the existing three legal transformer models did not yield significant
improvement on a rich-resource task while did show considerable improvement on
a low-resource task or if we deliberately cut down the training data size. We
therefore recommend domain pre-trained language models only in case of low-
resource tasks. The second part showed that adapters, as an emerging
technique, are very suitable to solve legal NLP tasks. As an intermediate
result, we release LegalRoBERTa111https://huggingface.co/saibo/legal-roberta-
base, a Roberta model adapted to the legal domain.

##  10 Acknowledgments

This work was done in cooperation with the Federal Department of Foreign
Affairs(FDFA) of Switzerland and EPFL under a semester project.

## References

  * Chalkidis et al., (2019) Chalkidis, I., Fergadiotis, E., Malakasiotis, P., and Androutsopoulos, I. (2019).  Large-scale multi-label text classification on EU legislation.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6314–6322, Florence, Italy. Association for Computational Linguistics.
  * Chalkidis et al., (2020) Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., and Androutsopoulos, I. (2020).  Legal-bert: The muppets straight out of law school.
  * Chalkidis et al., (2021) Chalkidis, I., Fergadiotis, M., Malakasiotis, P., and Androutsopoulos, I. (2021).  Neural contract element extraction revisited: Letters from sesame street.
  * DALE, (2019) DALE, R. (2019).  Law and word order: Nlp in legal tech.  Natural Language Engineering, 25(1):211–217.
  * Devlin et al., (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019).  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Dua and Graff, (2017) Dua, D. and Graff, C. (2017).  UCI machine learning repository.
  * Faghri et al., (2018) Faghri, F., Fleet, D. J., Kiros, J. R., and Fidler, S. (2018).  Vse++: Improving visual-semantic embeddings with hard negatives.
  * Galgani and Hoffmann, (2010) Galgani, F. and Hoffmann, A. (2010).  Lexa: Towards automatic legal citation classification.  In Li, J., editor, AI 2010: Advances in Artificial Intelligence, volume 6464 of Lecture Notes in Computer Science, pages 445 –454. Springer Berlin Heidelberg.
  * Gururangan et al., (2020) Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., and Smith, N. A. (2020).  Don’t stop pretraining: Adapt language models to domains and tasks.
  * Houlsby et al., (2019) Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019).  Parameter-efficient transfer learning for nlp.
  * Lee et al., (2019) Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., and Kang, J. (2019).  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics.
  * Liu et al., (2019) Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019).  Roberta: A robustly optimized bert pretraining approach.
  * Micheli et al., (2020) Micheli, V., d’Hoffschmidt, M., and Fleuret, F. (2020).  On the importance of pre-training data volume for compact language models.
  * Wang et al., (2020) Wang, S., Khabsa, M., and Ma, H. (2020).  To pretrain or not to pretrain: Examining the benefits of pretraining on resource rich tasks.  CoRR, abs/2006.08671.
  * Zheng et al., (2021) Zheng, L., Guha, N., Anderson, B. R., Henderson, P., and Ho, D. E. (2021).  When does pretraining help? assessing self-supervised learning for law and the casehold dataset.  CoRR, abs/2104.08671.

## Appendix

###  10.1 Domain Adaptive Pre-training Details of Related Work

In the table below, we can see the Domain Adaptive Pre-training details of
related work.

Experiment | Authors | Step | # epoch | batch size
---|---|---|---|---
Various Domains | Gururangan et al., (2020) | 12.5K | 1 | 256
LegalBERT | Chalkidis et al., (2020) | 1000K | 40 | 256
BioBERT | Lee et al., (2019) | 200K-470K | - | 192
LegalROBERTa | Our paper | 446K | 3 | 4

###  10.2  Examples of Next-Token-Prediction Results of LegalRoBERTa

This {mask} Agreement is between General Motors and John Murray .

Model | Top1 | Top2 | Top3
---|---|---|---
BERT | new | current | proposed
LegalBERT | settlement | letter | dealer
LegalROBERTa | License | Settlement | Contract
Table 10: Next Token Prediction Example 1

The applicant submitted that her husband was subjected to treatment amounting
to {mask} whilst in the custody of Adana Security Directorate

Model | Top1 | Top2 | Top3
---|---|---|---
BERT | torture | rape | abuse
LegalBERT | torture | detention | arrest
LegalROBERTa | torture | abuse | insanity
Table 11: Next Token Prediction Example 2

###  10.3 Model Architecture in Catchphrase Retrieval task

Below is an illustration of the Automatic Catchphrase Retrieval from Legal
Court Case Documents described in Section 5.2.

###  10.4  Hyper-parameters in LMTC task

  1. 1.

lr: 3e-05

  2. 2.

random seed: 0

  3. 3.

batch size: 16

  4. 4.

max sequence size: 216

  5. 5.

epochs: 40

  6. 6.

dropout:0.1

  7. 7.

early stop:yes

  8. 8.

patience: 7

###  10.5 Results on LMTC with different sizes of training data

The results on LMTC for BERT and LegalBERT with different sizes of training
data are shown in the table below.

Training data ratio | Train samples | Model | Precision | Recall | F1 | R@5 | P@5 | RP@5 | NDCG@5
---|---|---|---|---|---|---|---|---|---
100% | 45000 | BERT | 0.86 | 0.62 | 0.72 | 0.72 | 0.69 | 0.79 | 0.82
|  | LegalBERT | 0.86 | 0.63 | 0.73 | 0.72 | 0.69 | 0.79 | 0.82
20% | 9000 | BERT | 0.66 | 0.19 | 0.29 | 0.39 | 0.35 | 0.43 | 0.46
|  | LegalBERT | 0.70 | 0.19 | 0.29 | 0.40 | 0.35 | 0.43 | 0.46
10% | 4500 | BERT | 0.58 | 0.09 | 0.15 | 0.30 | 0.27 | 0.33 | 0.35
|  | LegalBERT | 0.64 | 0.11 | 0.18 | 0.32 | 0.28 | 0.34 | 0.37
5% | 2250 | BERT | 0.49 | 0.06 | 0.11 | 0.22 | 0.20 | 0.24 | 0.26
|  | LegalBERT | 0.49 | 0.07 | 0.13 | 0.24 | 0.22 | 0.26 | 0.28
1% | 450 | BERT | 0.00 | 0.00 | 0.00 | 0.03 | 0.02 | 0.03 | 0.02
|  | LegalBERT | 0.00 | 0.00 | 0.00 | 0.04 | 0.03 | 0.04 | 0.04