# A One-Covariate-at-a-Time Multiple Testing Approach to Variable Selection in
Additive Models ††thanks: Su gratefully acknowledges the support from NSFC
under the grant No. 72133002. Zhang gratefully acknowledges the financial
support from NSFC (Projects No.71973141 and No.71873033). E-mail:
<EMAIL_ADDRESS>(L. Su<EMAIL_ADDRESS>(T. Yang),
<EMAIL_ADDRESS>(Y. Zhang), and<EMAIL_ADDRESS>(Q. Zhou).

Liangjun Su${}^{a},$ Thomas Tao Yangb, Yonghui Zhang${}^{c},$ and Qiankun
Zhoud
a School of Economics and Management, Tsinghua University, China
b Research School of Economics, Australian National University, Australia
c School of Economics, Renmin University of China, China
d Department of Economics, Louisiana State University, USA

###### Abstract

This paper proposes a one-covariate-at-a-time multiple testing (OCMT) approach
to choose significant variables in high-dimensional nonparametric additive
regression models. Similarly to Chudik, Kapetanios and Pesaran (2018), we
consider the statistical significance of individual nonparametric additive
components one at a time and take into account the multiple testing nature of
the problem. One-stage and multiple-stage procedures are both considered. The
former works well in terms of the true positive rate only if the marginal
effects of all signals are strong enough; the latter helps to pick up hidden
signals that have weak marginal effects. Simulations demonstrate the good
finite sample performance of the proposed procedures. As an empirical
application, we use the OCMT procedure on a dataset we extracted from the
Longitudinal Survey on Rural Urban Migration in China. We find that our
procedure works well in terms of the out-of-sample forecast root mean square
errors, compared with competing methods.

Keywords:

One covariate at a time, multiple testing, model selection, high
dimensionality, nonparametric, additive model.

_JEL classification:_

C12, C14, C21, C52.

##  1 Introduction

In this paper, we propose a multiple testing approach to variable selection
for high-dimensional nonparametric additive models. In our procedure, we
estimate the marginal effect of each variable on the dependent variable one by
one, possibly with some preselected variables. The variables we select are
those whose marginal effects exceed some threshold value, which we choose in a
way such that the probability of selecting the right model is very high, using
the idea from the multiple testing studies.

Variable selection has a pivotal role in econometrics and statistics for
statistical learning and scientific discoveries. Notable early contributions
include Akaike (1973), Akaike (1974), and Schwartz (1978), who suggested a
unified approach to model selection, that is, choosing a parameter vector
$\theta$ by minimizing the conventional criterion function plus a penalty to
penalize the model size via the use of $L_{0}$-norm. A natural generalization
is the penalized $L_{q}$-norm with the leading case given by the $L_{1}$-norm.
Since the seminal work of Tibshirani (1996), extensive studies have been
conducted in this field. Important contributions in the statistics literature
include those of Fan and Li (2001), Zhou and Hastie (2005), Zou (2006), Zou
and Li (2008), Zhang (2010), Fan and Lv (2013), and Fan and Tang (2013). The
important economics models developed include those by Belloni et al. (2012),
Belloni and Chernozhukov (2013), Belloni, Chernozhukov and Hansen (2014),
Belloni et al. (2017) and Chernozhukov et al. (2017). The basic idea of this
strand of literature is adding an appropriate penalty term for the model size
in the loss function to achieve the desirable model selection properties.

Our paper is more similar to another strand of literature on variable
selection, pioneered by Fan and Lv (2008), who proposed that for linear
models, the dependent variable be regressed on the independent variables one
at a time and only those variables be retained that are highly correlated with
the dependent variable. This idea, widely known as “screening”, has been
extended to numerous scenarios, including Fan, Samworth and Wu (2009), Fan and
Song (2010), and Fan, Feng and Song (2011); for a comprehensive review, see
Chapter 8 in Fan et al. (2020). The major advantage of this approach is its
fast and reliable algorithm to obtain estimates, thanks to its simplicity.

The paper most related to ours is that of Chudik, Kapetanios and Pesaran
(2018) (CKP hereafter), who proposed a one-covariate-at-a-time multiple
testing approach for linear models. CKP’s main contributions are twofold.
First, they proposed a criterion for variable selection by controlling the
probability of choosing the right (or approximate) model. Second, the
screening approach may miss some important “hidden” signals whose marginal
effects on the dependent variable are small. In contrast, the CKP procedure is
able to pick up hidden signals with very high probability. A recent related
paper, Kozbur (2020), considers a testing-based forward model selection
(TBFMS) procedure in linear regression models that inductively selects
covariates to add predictive power into a working statistical model. But this
latter paper mainly focuses on the error bound and shows that the proposed
procedure is able to achieve estimation rates matching those of Least Absolute
Shrinkage and Selection Operator (Lasso) and post-Lasso.

Our paper contributes to the literature by extending the CKP’s approach from
parametric models to nonparametric additive models. We emphasize that the
results in CKP are not applicable to our framework as CKP assume that the
number of signal variables is fixed. In contrast, the number of basis
functions for each nonparametric additive component is divergent to infinity
with the sample size, and the statistics constructed here are much more
complicated than the t-statistics in CKP and may not even have a well-behaved
limiting distribution. In addition to investigating a different model, our
paper differs from that of CKP in some other important aspects. First, hidden
signals defined by CKP as rare, but we generalize the definition of hidden
signals (in Table 2 in Section 2.5) such that they are not uncommon in
practice. Technical details are updated accordingly. Second, CKP chose tuning
parameters similar to a Bonferroni correction of the cumulative distribution
function of the standard normal. This choice implicitly requires a certain
degree of approximation of the standard normal to the t-statistic distribution
in their study. Since all the theoretical results in this paper are non-
asymptotic, we try to avoid this by choosing the tuning parameters using the
classic Bayesian information criterion (BIC). Third, we add an adaptive group-
Lasso-based post-OCMT step to eliminate pseudo-signals that cannot be
eliminated with very high probability in CKP. This step adds very little
computation burden because the dimension is reduced dramatically before the
last-step estimation.

One competing method to ours in the literature is the adaptive group Lasso
(AGLASSO) proposed by Huang, Horowitz and Wei (2010). The AGLASSO adds some
adaptive penalty term to the loss function in the spirit of Zou (2006), and it
belongs to the first strand of the literature. Even though we use AGLASSO to
eliminate the pseudo-signals after the OCMT procedure, our approach allows
much faster computation and provides more reliable estimates than their
approach owing to the simplicity of our procedure.

We consider various setups in simulation studies and compare the above post-
OCMT AGLASSO procedure with that based on the OCMT alone or the AGLASSO
procedure of Huang, Horowitz and Wei (2010) alone. We find that the former one
generally outperforms the latter two significantly. We apply our method on a
dataset from the Longitudinal Survey on Rural Urban Migration in China (RUMiC)
and the empirical results also demonstrate the excellent performance of our
procedure in finite samples.

The remainder of this paper is structured as follows. In the next section, we
illustrate our approach through a single-stage procedure that is silent to
hidden signals. In Section 3, we present the more powerful multiple-stage
procedure. We investigate the finite sample properties of our procedure
through Monte Carlo experiments in Section 4 and an empirical application in
Section 5. We conclude the paper in Section 6. We present proofs of all the
propositions and theorems in the paper in Appendix B. The online supplement
contains some additional technical materials that include the proofs of the
technical lemmas in Appendix A and some additional results in the simulation
and the application. To facilitate the reading of the proofs of the main
results, we present the idea of the proofs in Appendices A.1 and A.2.

Notation. For a generic real matrix
$\boldsymbol{A=}\left\\{a_{ij}\right\\}\boldsymbol{,}$ let
$\left\|\boldsymbol{A}\right\|=\left[\lambda_{\max}\left(\boldsymbol{A}^{\prime}\boldsymbol{A}\right)\right]^{1/2}$
denote the spectral norm and
$\left\|\boldsymbol{A}\right\|_{\infty}=\max_{ij}\left|a_{ij}\right|.$ When
$\boldsymbol{A}$ is symmetric, $\lambda_{\max}\left(\boldsymbol{A}\right)\
$and $\lambda_{\min}\left(\boldsymbol{A}\right)\ $denote its maximum and
minimum eigenvalues, respectively. For vector $\boldsymbol{x},$
$\left\|\boldsymbol{x}\right\|$ denotes its Euclidean norm. For the
deterministic series $\left\\{a_{n},b_{n}\right\\}_{n=1}^{\infty}$, we denote
$a_{n}\propto b_{n}$ if
$0<C_{1}\leq\lim\inf_{n\rightarrow\infty}\left|a_{n}/b_{n}\right|\leq\lim\sup_{n\rightarrow\infty}\left|a_{n}/b_{n}\right|\leq
C_{2}<\infty$ for some constants $C_{1}$ and $C_{2},$ $a_{n}\lesssim b_{n}$ if
$\lim\sup_{n\rightarrow\infty}\left|a_{n}/b_{n}\right|\leq C<\infty$ for some
$C$ that does not depend on $n,$ $a_{n}\gtrsim b_{n}$ if $b_{n}\lesssim
a_{n},$ $a_{n}\ll b_{n}$ if $a_{n}=o\left(b_{n}\right),$ and $a_{n}\gg b_{n}$
if $b_{n}\ll a_{n}.$ $\mathcal{A}^{c}$ denotes the complement of the set
$\mathcal{A}$. $\overset{P}{\rightarrow}$ denotes convergence in probability.
$C$ and $M$ denote some positive constants that may vary from line to line.

##  2 The Model and One-Stage Procedure

In this section we give the model and definitions of various versions of
signals and noises. Then we will provide the one-stage procedure for variable
selection, present the basic assumptions and study the non-asymptotic
properties of our one-stage procedure.

###  2.1 The Model

Recently, CKP proposed a powerful multiple-stage procedure in order to pick up
the hidden signals along with signals with non-zero net effects. In the case
where there are no hidden signals, only one stage is needed. Our paper aims to
generalize the results of the linear models in CKP to the nonparametric
additive models. The notations and technical details in CKP are already quite
tedious, and they would be even more so in this paper. To facilitate the
exposition, we start with a simple case where there are no pre-determined
variables, and we conduct only one-stage multiple testing. We will present the
more powerful multiple-stage procedure and show its validity in Section 3.

Suppose the model is

$Y=f^{\ast}\left(X_{1},X_{2},\ldots,X_{p^{\ast}}\right)+\varepsilon,$ (2.1)

where $Y$ is the dependent variable, $X_{1},$ $X_{2},\ldots,$ and
$X_{p^{\ast}}$ are random independent variables, $\varepsilon$ is an
unobserved error term, and $f^{\ast}$ is an unknown smooth function. Even
though we have only $p^{\ast}$ signal variables, namely $X_{1},$
$X_{2},\ldots,$ $X_{p^{\ast}}$, that should be included into the regression
model in (2.1), we do not know this truth before the data reveal the fact. The
realistic situation is that the $p^{\ast}$ signals are contained in a set
$\mathcal{S}_{n}=\left\\{X_{j},j=1,2,\ldots,p_{n}\right\\},$ where $p_{n}$ can
be much larger than the sample size $n$ and $p_{n}\propto n^{B_{p_{n}}}$ for
some $B_{p_{n}}>0$ or 1. $\mathcal{S}_{n}$ is the set for all candidate
variables, and we refer to it as the active set. We assume that
$E\left(\varepsilon|X_{1},X_{2},\ldots,X_{p^{\ast}},X_{p^{\ast}+1},\ldots,X_{p_{n}}\right)=0.$
The target of the model selection is to pick up those signals among
$\mathcal{S}_{n}.$

To avoid the curse of dimensionality in nonparametrics, we impose the additive
structure on $f^{\ast}$, that is,

$f^{\ast}\left(X_{1},X_{2},\ldots,X_{p^{\ast}}\right)=\sum_{j=1}^{p^{\ast}}f_{j}^{\ast}\left(X_{j}\right).$
(2.2)

Obviously, the individual functions $f_{j}^{\ast},$ $j=1,\ldots,p^{\ast},$
cannot be identified without certain suitable normalizations. We do the
normalization by assuming that $E[f_{j}^{\ast}\left(X_{j}\right)]=0$ for each
$j$ and rewrite the model as

$Y=\mu+\sum_{j=1}^{p^{\ast}}f_{j}^{\ast}\left(X_{j}\right)+\varepsilon,$ (2.3)

where $\mu=E\left(Y\right).$ Since $\mu$ can be estimated by the sample mean
of the $Y$ variable at the usual $\sqrt{n}$-rate and this estimation does not
affect the estimation of the nonparametric additive component, for the
simplicity of presentation, we assume $\mu=0$ below.

Since we will focus on the $B$-spline-based nonparametric theory, it is
standard to assume compact support for each regressor (see, e.g., Horowitz and
Mammen (2004), Chen (2007) and Huang, Horowitz and Wei (2010)). Without loss
of generality, we assume that the support for $X_{j}$ is $[0,1\mathcal{]}$ for
all $j$. Let $\alpha_{1}$ be a non-negative integer, $\alpha_{2}\in(0,1],$ and
$d=\alpha_{1}+\alpha_{2}.$ Denote the class of all $\alpha_{1}$ times
continuously differentiable real-valued functions on $[0,1]$ by
$C^{\alpha_{1}}\left([0,1]\right).$ Define $d$-th smooth real-valued functions
on $[0,1]$ as

$\Lambda^{d}\left([0,1]\right)=\left\\{h\in
C^{\alpha_{1}}\left([0,1]\right):\left|h^{\left(\alpha_{1}\right)}\left(t_{1}\right)-h^{\left(\alpha_{1}\right)}\left(t_{2}\right)\right|\leq
C\left|t_{1}-t_{2}\right|^{\alpha_{2}}\right\\}.$ (2.4)

For notational simplicity, we will restrict our attention to the case where
$f_{j}$’s are smooth enough and belong to $\Lambda^{d}\left([0,1]\right)$ with
$d>1$. The case of different smoothness parameters only complicates the
notation but does not bring in any new insight.

###  2.2 Signals, Hidden Signals, and Noises

The idea of one-stage procedure is that we estimate the impact of $X_{l}$ on
$Y,$ $l=1,2,...,p_{n},$ one by one. So we run $p_{n}$ estimations in total and
will keep those variables that are significant enough. Since in each
regression we only have one covariate, we are not estimating $f_{l}^{\ast}$
when the explanatory variable is $X_{l}$. Instead, we are estimating the
conditional expectation $f_{l}\left(X_{l}\right)\equiv E\left(Y|X_{l}\right).$
We define the net impact of $X_{l}$ on $Y$ as

$\theta_{l}\equiv\left\\{E\left[f_{l}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}=\left\\{E\left[\left(\sum_{j=1}^{p^{\ast}}\sigma_{lj}\right)^{2}\right]\right\\}^{1/2},$

where $\sigma_{lj}=E[f_{j}^{\ast}\left(X_{j}\right)|X_{l}].$ Here,
$\sigma_{lj}$ plays the role of the scaled covariance between $X_{j}$ and
$X_{l}$ for the linear model considered by CKP.

Obviously, $\theta_{l}$ can be 0 or close to 0 for signals, and $\theta_{l}$
can be nonzero or large for non-signals. As in CKP, we also have four
possibilities as tabulated in Table 1. Cases (I) and (IV) in Table 1 are
desirable cases. Case (III) happens when some non-signals are not independent
of the signals. The hidden signals defined in Case (II) are rare in the linear
case, and it is also rare in the nonparametric case. We generalize the
definition of hidden signals to the finite sample case in Table 2 in the next
section where $\theta_{l}$ is non-zero but small relative to the sample size.
Under this generalization, hidden signals are not uncommon.

Table 1: Population-based definitions of signals and noises | $\theta_{l}\neq 0$ | $\theta_{l}=0$
---|---|---
$\left\\{E[f_{l}^{\ast}\left(X_{l}\right)^{2}]\right\\}^{1/2}\neq 0$ | (I) Signals with nonzero net effect | (II) Hidden signals
$\left\\{E[f_{l}^{\ast}\left(X_{l}\right)^{2}]\right\\}^{1/2}=0$ | (III) Pseudo-signals | (IV) Noise variables

As we shall see, our one-stage procedure is silent on picking up hidden
signals and eliminating pseudo-signals. For the hidden signals, we will
propose a multiple-stage procedure in Section 3 that can effectively pick them
up. To eliminate the pseudo-signals, as a post-procedure we propose to re-
estimate the model using adaptive group Lasso in Section 3.4.

We assume that there are $p^{\ast\ast}$ pseudo-signals. Without loss of
generality, we denote them to be

$\left\\{X_{p^{\ast}+1},X_{p^{\ast}+2},\ldots,X_{p^{\ast}+p^{\ast\ast}}\right\\}.$

Below we focus the one-stage procedure in this section and postpone the multi-
stage case to the next section.

###  2.3 The Test Statistic and One-Stage Procedure

Suppose that we have $n$ observations
$\left\\{(y_{i},x_{1i},...,x_{p_{n}i})\right\\}_{i=1}^{n}$ that are drawn from
the distribution of $(Y,X_{1},...,X_{p_{n}}).$ The data are given in a
$n\times\left(p_{n}+1\right)$ matrix

$\left(\boldsymbol{y},\boldsymbol{x}_{1},\boldsymbol{x}_{2},\ldots,\boldsymbol{x}_{p_{n}}\right)$

where $\boldsymbol{y}=\left(y_{1},y_{2},\ldots,y_{n}\right)^{\prime}$ and
$\boldsymbol{x}_{l}=\left(x_{l1},x_{l2},\ldots,x_{ln}\right)^{\prime}$ for
$l=1,...,p_{n}$. We propose to choose finite order (e.g., cubic) B-spline
basis functions $\left\\{\psi_{l}\left(x\right)\right\\}_{l=1}^{m_{n}}$ on
$\left[0,1\right]$ to approximate the unknown functions $f_{l}$’s. For the
definition and properties of B-spline basis, see Stone (1985) and de Boor
(2001). We list some properties of the B-spline basis functions in Lemma A.3.
Since we normalize $E[f_{j}^{\ast}\left(X_{j}\right)]=0,$ we similarly
normalize the basis as

$\phi_{jl}\left(x\right)=\psi_{j}\left(x\right)-n^{-1}\sum_{i=1}^{n}\psi_{j}\left(x_{li}\right).$

This is a standard practice; see, e.g., Huang, Horowitz and Wei (2010). For
notational simplicity, we will write $\phi_{j}\left(x\right)$ for
$\phi_{jl}\left(x\right)$. Let
$P^{m_{n}}\left(x\right)=\left(\phi_{1}\left(x\right),\phi_{2}\left(x\right),\ldots,\phi_{m_{n}}\left(x\right)\right)^{\prime},$
an $m_{n}\times 1$ vector. Define

$\boldsymbol{\beta}_{l}=\left[E\left(P^{m_{n}}\left(X_{l}\right)P^{m_{n}}\left(X_{l}\right)^{\prime}\right)\right]^{-1}E\left(P^{m_{n}}\left(X_{l}\right)Y\right)\text{
and }U_{l}=Y-P^{m_{n}}\left(X_{l}\right)^{\prime}\boldsymbol{\beta}_{l},$
(2.5)

which are the population coefficient and the error term in the regression of
$Y$ on $P^{m_{n}}\left(X_{l}\right)$. Note that we suppress the dependence of
$\boldsymbol{\beta}_{l}$ on the sample size $n$.

For the one-stage procedure, we conduct the regression of $Y$ on
$P^{m_{n}}\left(X_{l}\right),$ $l=1,2,\ldots,p_{n},$ one by one. Let
$\mathbb{X}_{li}=P^{m_{n}}\left(x_{li}\right)$ be the approximating function
basis at the $i$th observation for $X_{l}$. Let
$\mathbb{X}_{l}=(\mathbb{X}_{l1},\mathbb{X}_{l2},\ldots,\mathbb{X}_{ln})^{\prime}$
be the $n\times m_{n}$ “design” matrix for $X_{l}.$ For $X_{l},$ we regress
$\boldsymbol{y}$ on $\mathbb{X}_{l}\ $to obtain

$\boldsymbol{\hat{\beta}}_{l}=\left(\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}\boldsymbol{y}.$

We construct the test statistic as

$\mathcal{\hat{X}}_{l}=\boldsymbol{\hat{\beta}}_{l}^{\prime}\left(\hat{\sigma}_{l}^{-2}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)\boldsymbol{\hat{\beta}}_{l},$
(2.6)

where $\hat{\sigma}_{l}^{2}=n^{-1}\sum_{i=1}^{n}\hat{u}_{li}^{2}$ and
$\hat{u}_{li}$ is the residual from the above regression. Then we define the
first-stage OCMT selection indictor as

$\widehat{\mathcal{J}}_{l}=\mathbf{1}\left(\mathcal{\hat{X}}_{l}>\varsigma_{n}\right)\text{
for }l=1,2,\ldots,p_{n},$ (2.7)

where $\mathbf{1}\left(\cdot\right)$ is the usual indicator function and
$\varsigma_{n}$ is a threshold value.

For the linear model in CKP with one covariate at a time,
$\mathcal{\hat{X}}_{l}$ is asymptotically $\chi^{2}\left(1\right)$ under
conditional homoskedasticity, and one can follow their lead to consider
threshold values for the associated $t$-statistics based on the adjusted
normal critical values. Nevertheless, such a result is not available in our
framework due to the divergent dimension of regressors in the sieve
estimation. In addition, the potential presence of conditional
heteroskedasticity greatly complicates our analysis too. What we really need
is to show that $\mathcal{\hat{X}}_{l}$ behaves distinctly for signals and
noises so that a suitable choice of the threshold value $\varsigma_{n}$ can
help us separate the signals from the noises. For these reasons, we do not
associate $\mathcal{\hat{X}}_{l}$ with any asymptotic distribution. Instead,
we will set $\varsigma_{n}\propto\kappa_{n}\log\left(m_{n}\right)m_{n}$ for a
positive series $\kappa_{n}$ that diverges to infinity slowly. For more
details, see the remark on Assumption 8 in the next subsection.

###  2.4 Basic Assumptions

To study the asymptotic properties of the one-stage procedure, we impose the
following assumptions.

###### Assumption 1

$\left\\{y_{i},x_{1i},x_{2i},\ldots,x_{p_{n}i}\right\\}_{i=1}^{n}$ are
independent and identically distributed (i.i.d.) across $i;$
$E(\varepsilon|X_{1},X_{2},\ldots,X_{p^{\ast}},X_{p^{\ast}+1},\ldots,$
$X_{p_{n}})$ $=0.$

###### Assumption 2

$p^{\ast}\ $is a positive integer that does not vary with $n.$
$p^{\ast\ast}\lesssim n^{B_{p^{\ast\ast}}}\ $and $p_{n}\propto n^{B_{p_{n}}}$
for some $B_{p_{n}}>B_{p^{\ast\ast}}\geq 0.$

###### Assumption 3

The support for $X_{l}$ is $\left[0,1\right],$ $l=1,\ldots,p_{n}$. The density
function for $X_{l}$ is bounded and bounded away from $0.$

###### Assumption 4

$\Pr\left(\left|\varepsilon\right|>t\right)\leq
C_{1}\exp\left(-C_{2}t^{s}\right)$ holds for all $t>0$ and some $s,$ $C_{1},$
$C_{2}>0.$

###### Assumption 5

$f_{l}\left(\cdot\right)=E\left(Y|X_{l}=\cdot\right)\in\Lambda^{d}\left([0,1]\right)\
$with $d>1$ for $l=1,\ldots,p_{n}$.

###### Assumption 6

$\left|f_{j}^{\ast}\left(\cdot\right)\right|,$ $j=1,...,p^{\ast},$ are
uniformly bounded. $E\left(\varepsilon^{2}|X_{1},X_{2},...,X_{p_{n}}\right)$
is uniformly bounded almost surely.

###### Assumption 7

$m_{n}\propto n^{B_{m}}$ with $1/\left(1+2d\right)<B_{m}<1/3.$

###### Assumption 8

$\varsigma_{n}\propto\kappa_{n}\log\left(m_{n}\right)m_{n}\ $for some
$\kappa_{n}$ such that $\kappa_{n}>1$ and $\kappa_{n}\rightarrow\infty$ slowly
as $n\rightarrow\infty$.

Assumption 1 imposes i.i.d. condition on the observations and standard
conditional moment restriction. The extension to weakly dependent observations
is possible but left for future research. In Assumption 2, we assume that the
number of signals is fixed; the number of pseudo-signals is allowed to
increase as $n$ increases but at a slower rate than that of the total
candidate variables. It is also possible to allow $p^{\ast}$ to diverge to
infinity (see Section 3.5). Assumption 3 restricts the support of $X$ to be
$\left[0,1\right].$ This is a very common condition for nonparametric additive
models; see, e.g., Li (2000) and Horowitz and Mammen (2004). Assumption 4
imposes some tail conditions $\varepsilon,$ which is also assumed in CKP but
weaker than the commonly used sub-exponential condition in the variable
selection literature and the one for the adaptive group Lasso in Huang,
Horowitz and Wei (2010). The primary use of this condition is to derive non-
asymptotic bound for errors.

Assumption 5 imposes fairly weak smooth condition on $f_{l}\left(x\right)$,
which is weaker than the commonly used condition $d\geq 2$. Note that $d>1$ is
needed for Assumption 7. Assumption 6 is a technical assumption needed to
simplify the proof. Specifically, the boundedness of $f_{j}^{\ast}$ implies
$U_{l}$ defined in equation (A.1) has the same tail behavior as $\varepsilon$.
The bounded conditional second moment of $\varepsilon$ is to ensure some nice
properties of $U_{l}\phi_{j}\left(X_{l}\right)$ and this assumption is also
common in the sieve literature (see, e.g., Newey (1997)). This assumption is
mild given that we assume all $X_{l}$’s have compact support and the tails of
$\varepsilon$ decay exponentially fast. Assumption 7 imposes conditions on
$m_{n}.$ First, we need $B_{m}<1/3$ such that $nm_{n}^{-3}\rightarrow\infty.$
The last condition is necessary for
$\left\|n^{-1}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right\|\propto m_{n}^{-1}$
to hold with very high probability. To see why, note that Lemma A.3 in
Appendix A.3 suggests that
$\left\|E\left[P^{m_{n}}\left(X_{l}\right)P^{m_{n}}\left(X_{l}\right)^{\prime}\right]\right\|\propto
m_{n}^{-1}.$ We need $m_{n}^{-1}\gg\left(m_{n}/n\right)^{1/2},$ or
equivalently, $nm_{n}^{-3}\rightarrow\infty,$ in order to ensure that
$n^{-1}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}$ is close to
$E\left[P^{m_{n}}\left(X_{l}\right)P^{m_{n}}\left(X_{l}\right)^{\prime}\right]$
with very high probability. Second, we need $B_{m}>1/\left(1+2d\right)$ to
ensure that the approximation bias is asymptotically negligible in comparison
with the asymptotic variance term: $m_{n}^{-d}\ll\left(m_{n}/n\right)^{1/2}$.

Assumption 8 imposes conditions on the threshold value $\varsigma_{n}$ that
ensures the separability of the signals from noises. If the true value of
$\boldsymbol{\beta}_{l}$ is $\boldsymbol{0}$, $\mathcal{\hat{X}}_{l}$ in
equation (2.6) is $O_{P}\left(m_{n}\right).$ In this case, to ensure
$\widehat{\mathcal{J}}_{l}=0$ with very high probability, we can take
$\varsigma_{n}\propto\kappa_{n}\log\left(m_{n}\right)m_{n}$ and lose some
power up to $\kappa_{n}\log\left(m_{n}\right).$ Here, $\kappa_{n}$ can be any
series diverging to infinity slowly, e.g.,
$\left[\log\left(m_{n}\right)\right]^{\epsilon}$ for some small $\epsilon>0.$
The loss of the power to some degree is inevitable because of the nature of
the multiple testing procedure when the number of tests goes to infinity. In
contrast, CKP choose their threshold by the Bonferroni correction of the
standard normal. We choose not to follow them because of the following
reasons. First, given the divergence of $m_{n},$ $\mathcal{\hat{X}}_{l}$ does
not converge to a chi-square distribution asymptotically even in the
homoskedastic case so that we cannot use chi-square distribution to
approximate the finite sample distribution of $\mathcal{\hat{X}}_{l}.$ Under
conditional heteroskedasticity, $\mathcal{\hat{X}}_{l}$ does not converge to a
chi-square distribution even if $m_{n}$ is held fixed. So our procedure does
not rely on the chi-square approximation. Second, even if we can do the
approximation, the cumulative density function (CDF) of a chi-square
distribution is very complicated. We do not have a rate for the inverse of its
CDF evaluated at a certain rate (e.g., $n^{-C}$) like the case of normal CDF.
For these reasons, we do our selection based on the non-asymptotic results.
Specifically, we will take
$\varsigma_{n}=C\kappa_{n}\log\left(m_{n}\right)m_{n}$ for some $\kappa_{n}$
and choose the value of $C$ by the classic BIC. The details can be found in
Section 4.2. This $\varsigma_{n}$ helps separate the signals from the noises
with very high probability, as demonstrated in the next section. Recall that
in Assumption 2 we assume that $p_{n}$ go to infinity at a polynomial rate of
$n,$ same as $m_{n}.$ This ensures that $\log\left(m_{n}\right)\propto\log
p_{n}.$ Therefore, theoretically we only need to put $\log\left(m_{n}\right)$
in $\varsigma_{n}$ to have a control of $p^{\ast\ast}$ and $p_{n}$ for the
TPR, FDR and FPR defined after Proposition 2.1 below. We postpone the
discussion on the comparison of technical conditions required for our
procedure and those required for the AGLASSO to Section 3.2.

###  2.5 The Non-asymptotic Properties of $\mathcal{\hat{X}}_{l}$ and the One-
Stage Procedure

We present the first theoretical result in this paper. It derives the
probability bounds for the “Type-I” and “Type-II” errors.

###### Proposition 2.1

Suppose that $Y$ is given by equation (2.1) and Assumptions 1 $-$ 8 hold.

(i) If
$\theta_{l}\lesssim\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2},$
then

$\displaystyle\Pr\left(\mathcal{\hat{X}}_{l}\geq\varsigma_{n}\right)$
$\displaystyle\leq\exp\left(-C_{1}m_{n}^{-1}\varsigma_{n}+\log
m_{n}\right)+C_{2}\exp\left(-C_{3}n^{C_{4}}\right)$ $\displaystyle\leq
n^{-M}+C_{2}\exp\left(-C_{3}n^{C_{4}}\right)$

for any fixed large $M>0$ and some positive constants $C_{1},$ $C_{2},$
$C_{3},$ and $C_{4}.$

(ii) If
$\theta_{l}\gtrsim\kappa_{n}\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2}$
with $\kappa_{n}$ specified in Assumption 8, then

$\Pr\left(\mathcal{\hat{X}}_{l}\geq\varsigma_{n}\right)\geq
1-n^{-M}-C_{5}\exp\left(-C_{6}n^{C_{7}}\right)$

for any fixed large $M>0$ and some positive constants $C_{5},$ $C_{6},$ and
$C_{7}.$

The proof of Proposition 2.1 is tedious. We provide some technical details in
Appendix A.1 before we formally prove the proposition in Appendix B. An
implication of Proposition 2.1 is that for the well-chosen threshold value
$\varsigma_{n},$ the above one-stage procedure can separate the signals with
$\theta_{l}\gtrsim\kappa_{n}\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2}$
from the noises with $\theta_{l}=0.$ Of course, we may have some intermediate
case where
$0<\theta_{l}\lesssim\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2}$
for which the above procedure fails to do so.

Following the literature and CKP, we define the true positive rates (TPR) and
the false positive rates (FPR) respectively as

$\displaystyle\text{TPR}_{n}$
$\displaystyle=\frac{\sum_{l=1}^{p_{n}}\mathbf{1}\left(\widehat{\mathcal{J}}_{l}=1\text{
and
}\left\\{E\left[f_{l}^{\ast}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}\neq
0\right)}{\sum_{l=1}^{p_{n}}\mathbf{1}\left(\left\\{E\left[f_{l}^{\ast}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}\neq
0\right)},\text{ and}$ $\displaystyle\text{ FPR}_{n}$
$\displaystyle=\frac{\sum_{l=1}^{p_{n}}\mathbf{1}\left(\widehat{\mathcal{J}}_{l}=1\text{
and
}\left\\{E\left[f_{l}^{\ast}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}=0\right)}{\sum_{l=1}^{p_{n}}\mathbf{1}\left(\left\\{E\left[f_{l}^{\ast}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}=0\right)}.$

Based on the test statistic defined in equation (2.6) and its property
developed in Proposition 2.1, we introduce the sample version of the
definitions of signals and noises in Table 2.

Table 2: Finite-sample-based definitions of signals and noises | $\theta_{l}\gg\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2}$ | $\theta_{l}\lesssim\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2}$
---|---|---
$\left\\{E[f_{l}^{\ast}\left(X_{l}\right)^{2}]\right\\}^{1/2}\neq 0$ | (I) Signals | (II) Hidden signals
$\left\\{E[f_{l}^{\ast}\left(X_{l}\right)^{2}]\right\\}^{1/2}=0$ | (III) Pseudo-signals | (IV) Noise variables

With a little abuse of notation, we continue to use $p^{\ast}$ and
$p^{\ast\ast}$ to denote the number of signals and pseudo-signals at the
sample level, and assume that they satisfy Assumption 2. Adopting the
definitions in Table 2, we define the false discovery rates (FDR) as

$\text{FDR}_{n}=\frac{\sum_{l=1}^{p_{n}}\mathbf{1}\left(\widehat{\mathcal{J}}_{l}=1\text{,
}\left\\{E\left[f_{l}^{\ast}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}=0,\text{
and
}\theta_{l}\lesssim\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2}\right)}{\sum_{l=1}^{p_{n}}\widehat{\mathcal{J}}_{l}+1}.$

Apparently, the definitions in Table 2 generalize the definitions in Table 1.
The Hidden Signals defined in Table 1 are rare, but those defined in Table 2
can be common in practice. Because of this, we adopt the definitions in Table
2 hereafter.

The one-stage procedure is valid in terms of TPR, only if the marginal effects
of all signals are strong enough. Specifically, we need the following
assumption.

###### Assumption 9

There are no hidden signals. That is,
$\theta_{j}\gtrsim\kappa_{n}\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2}$
for a series $\kappa_{n}\rightarrow\infty$ for all $j=1,2,...,p^{\ast}.\vskip
3.0pt plus 1.0pt minus 1.0pt$

If Assumption 9 fails to hold, we need the multiple-stage procedure to pick up
the hidden signals.

Based on the results in Proposition 2.1, we present the results for TPRn,
FPRn, and FDRn in the following theorem.

###### Theorem 2.1

Suppose that $Y$ is given by equation (2.1) and Assumptions 1 $-$ 9 hold. Then

(i) $E\left(\text{\emph{TPR}}_{n}\right)\geq
1-C_{1}\exp\left(-C_{2}n^{C_{3}}\right)$ for some positive constants
$C_{1},C_{2}$, and $C_{3};$

(ii) $E\left(\text{\emph{FPR}}_{n}\right)\leq
p^{\ast\ast}/\left(p_{n}-p^{\ast}\right)+C_{4}n^{-M}+C_{5}\exp\left(-C_{6}n^{C_{7}}\right)$
for any fixed positive large constant $M$ and some positive constants $C_{4},$
$C_{5},C_{6},$ and $C_{7};$

(iii) _FDR_ ${}_{n}\overset{P}{\rightarrow}0.$

Theorem 2.1 implies that all of TPR${}_{n},$ FPRn and FDRn can be well
controlled provided we assume away hidden signals at the sample level. Note
that Theorem 2.1(i)–(ii) focuses on the non-asymptotic properties of TPRn and
FPRn while the last part of Theorem 2.1 reveals that the false discovery rate
is asymptotically vanishing in large samples.

In the next section, we turn to the multiple-stage procedure that does not
rely on Assumption 9.

##  3 The Multiple-Stage Procedure

In this section we propose a multiple-state procedure to select variables for
the nonparametric additive models.

###  3.1 The Test Statistic with Pre-selected Variables

As mentioned above, we may not identify a signal $X_{l}$ whose marginal effect
satisfies
$\theta_{l}\lesssim\kappa_{n}\log\left(m_{n}\right)\left(m_{n}/n\right)^{1/2},$
even in the case where the joint effect of $X_{l}$ on $Y,$ namely,
$\left\\{E\left[f_{l}^{\ast}\left(X_{l}\right)^{2}\right]\right\\}^{1/2},$ is
large enough. Consequently, Theorem 2.1 does not hold without Assumption 9
which assumes away small sample hidden signals. In contrast, Lemma A.7 in
Appendix A.3 underpins the result that as long as
$\left\\{E\left[f_{l}^{\ast}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}\gtrsim\kappa_{n}\log\left(m_{n}\right)\left(m_{n}/n\right)^{1/2}$
for some slowly divergent series $\kappa_{n}$ and some full rank condition
holds, the marginal effect of $X_{l}$ will be strong enough to be picked up at
certain stage of a multiple-stage procedure.

To introduce the multiple-stage procedure, we add some notations. Suppose at
certain stage after stage 1, we have pre-selected $\iota_{n}$ variables from
the active set $\mathcal{S}_{n}=\left\\{X_{j},\text{ }j=1,...,p_{n}\right\\}$
based on some selection procedure to be described below. To avoid confusion,
we denote these pre-selected variables as $Z_{1},\ldots,Z_{\iota_{n}}$. Let
$\boldsymbol{Z}\equiv\left(Z_{1},\ldots,Z_{\iota_{n}}\right)^{\prime}\ $and
$P^{m_{n}}\left(\boldsymbol{Z}\right)=(P^{m_{n}}\left(Z_{1}\right)^{\prime},\ldots,$
$P^{m_{n}}\left(Z_{\iota_{n}}\right)^{\prime})^{\prime}.$ Note that
$P^{m_{n}}\left(\boldsymbol{Z}\right)$ is an $\iota_{n}m_{n}\times 1$ vector
for $\boldsymbol{Z}.$ At the next stage, we consider the nonparametric
additive regression of $Y$ on $\boldsymbol{Z}$ and an $X_{l}$ that has not
been selected so far and we do this one by one for all $p_{n}-\iota_{n}$ non-
selected variables $X_{l}.$ We define the impact of $X_{l}$ on $Y$ after
controlling $\boldsymbol{Z}$ as

$\displaystyle\theta_{l,\boldsymbol{Z}}$ $\displaystyle\equiv
E\left[\left(E\left[\left.Y-P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}\Phi_{\boldsymbol{Z}}^{-1}E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)Y\right]\right|X_{l}\right]\right)^{2}\right]^{1/2}$
$\displaystyle=E\left[\left(\sum_{j=1}^{p^{\ast}}E\left[\left.f_{j}^{\ast}\left(X_{j}\right)-P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}\Phi_{\boldsymbol{Z}}^{-1}E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)f_{j}^{\ast}\left(X_{j}\right)\right]\right|X_{l}\right]\right)^{2}\right]^{1/2}$
$\displaystyle=\left(E\left[\left(\sum_{j=1}^{p^{\ast}}\mu_{lj,\boldsymbol{Z}}\right)^{2}\right]\right)^{1/2},$

where $\Phi_{\boldsymbol{Z}}\equiv
E[P^{m_{n}}\left(\boldsymbol{Z}\right)P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}]$
and $\mu_{lj,\boldsymbol{Z}}\equiv
E\left[\left.f_{j}^{\ast}\left(X_{j}\right)-P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}\Phi_{\boldsymbol{Z}}^{-1}E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)f_{j}^{\ast}\left(X_{j}\right)\right]\right|X_{l}\right]$
denotes the effect of $X_{l}$ on $f_{j}^{\ast}\left(X_{j}\right)$ after
controlling the effects of $\boldsymbol{Z}$. Apparently, we suppress the
dependence of $\theta_{l,\boldsymbol{Z}}$ on the sample size $n$.

At the sample level, let $\mathbb{Z}$ and $\mathbb{X}_{l}$ denote the $n\times
m_{n}\iota_{n}$ and $n\times m_{n}$ “design matrices” for $\boldsymbol{Z}$ and
$X_{l},$ respectively. That is,

$\mathbb{Z}\equiv\left(\mathbb{Z}_{1},\ldots,\mathbb{Z}_{\iota_{n}}\right)\text{
and
}\mathbb{X}_{l}\equiv\left(\mathbb{X}_{l1},\ldots,\mathbb{X}_{ln}\right)^{\prime},$
(3.1)

where
$\mathbb{Z}_{l}=\left(\mathbb{Z}_{l1},\ldots,\mathbb{Z}_{ln}\right)^{\prime}\
$is a $n\times m_{n}$ matrix, $\mathbb{Z}_{li}=P^{m_{n}}\left(z_{li}\right)\
$and $\mathbb{X}_{li}=P^{m_{n}}\left(x_{li}\right).$ Define
$M_{\mathbb{Z}}\equiv
I_{n}-\mathbb{Z}\left(\mathbb{Z}^{\prime}\mathbb{Z}\right)^{-1}\mathbb{Z}^{\prime}.$
By the result of partitioned regressions, the coefficient of
$P^{m_{n}}\left(X_{l}\right)\ $is estimated by

$\boldsymbol{\hat{\beta}}_{l,\boldsymbol{Z}}=\left(\mathbb{X}_{l}^{\prime}M_{\mathbb{Z}}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}M_{\mathbb{Z}}\boldsymbol{y}.$

To determine whether $X_{l}$ should be treated as a signal variable, we
propose the following test statistic

$\mathcal{\hat{X}}_{l,\boldsymbol{Z}}=\boldsymbol{\hat{\beta}}_{l,\boldsymbol{Z}}^{\prime}\left(\hat{\sigma}_{l,\boldsymbol{Z}}^{-2}\mathbb{X}_{l}^{\prime}M_{\mathbb{Z}}\mathbb{X}_{l}\right)\boldsymbol{\hat{\beta}}_{l,\boldsymbol{Z}}=\left(\boldsymbol{y}^{\prime}M_{\mathbb{Z}}\mathbb{X}_{l}\right)\left(\hat{\sigma}_{l,\boldsymbol{Z}}^{2}\mathbb{X}_{l}^{\prime}M_{\mathbb{Z}}\mathbb{X}_{l}\right)^{-1}\left(\mathbb{X}_{l}^{\prime}M_{\mathbb{Z}}\boldsymbol{y}\right)$
(3.2)

where
$\hat{\sigma}_{l,\boldsymbol{Z}}^{2}=n^{-1}\sum_{i=1}^{n}\hat{\varepsilon}_{li}^{2}$
and $\hat{\varepsilon}_{li}$ is the residual from the regression
$\boldsymbol{y}$ on $\left(\mathbb{Z},\mathbb{X}_{l}\right)$.

We will study the non-asymptotic properties of
$\mathcal{\hat{X}}_{l,\boldsymbol{Z}}$ in the next subsection which lay down
the foundation for our multiple-stage procedure.

###  3.2 The Non-asymptotic Properties of
$\mathcal{\hat{X}}_{l,\boldsymbol{Z}}$

To study the non-asymptotic properties of
$\mathcal{\hat{X}}_{l,\boldsymbol{Z}},$ we impose the following technical
conditions.

Assumption 2’ $p^{\ast}\ $is a positive integer that does not vary with $n.$
$p^{\ast\ast}\lesssim n^{B_{p^{\ast\ast}}}\ $and $p_{n}\propto n^{B_{p_{n}}}$
for some $B_{p_{n}}>B_{p^{\ast\ast}}\geq 0.$ Further,
$B_{p^{\ast\ast}}<\left(1-3B_{m}\right)/2.$

Assumption 5’
$f_{l}\left(\cdot\right)=E\left(Y|X_{l}=\cdot\right)\in\Lambda^{d}\left([0,1]\right)\
$with $d>1$ for $l=1,\ldots,p_{n}$.
$f_{j}^{\ast}\in\Lambda^{d}\left([0,1]\right)\ $with $d>1$ for
$j=1,\ldots,p^{\ast}$.

Assumption 9’
$\left\\{E[f_{j}^{\ast}\left(X_{j}\right)^{2}]\right\\}^{1/2}\gtrsim\kappa_{n}\log\left(m_{n}\right)\left(m_{n}/n\right)^{1/2}$
for some slowly divergent series $\kappa_{n}$ and for $j=1,\ldots,p^{\ast}$.

###### Assumption 10

Let
$\boldsymbol{X}_{1}^{p^{\ast}+p^{\ast\ast}}\equiv\left(X_{1},\ldots,X_{p^{\ast}},X_{p^{\ast}+1},\ldots,X_{p^{\ast}+p^{\ast\ast}}\right)^{\prime},$
the vector of all signals and pseudo signals. Similarly, let
$P^{m_{n}}(\boldsymbol{X}_{1}^{p^{\ast}+p^{\ast\ast}})\equiv\left(P^{m_{n}}\left(X_{1}\right)^{\prime},\ldots,P^{m_{n}}\left(X_{p^{\ast}+p^{\ast\ast}}\right)^{\prime}\right)^{\prime}\
$and $\Phi_{\boldsymbol{X}_{1}^{p^{\ast}+p^{\ast\ast}}}\equiv
E[P^{m_{n}}(\boldsymbol{X}_{1}^{p^{\ast}+p^{\ast\ast}})$
$P^{m_{n}}(\boldsymbol{X}_{1}^{p^{\ast}+p^{\ast\ast}})^{\prime}].$ Assume that

$B_{X1}m_{n}^{-1}\leq\lambda_{\min}\left(\Phi_{\boldsymbol{X}_{1}^{p^{\ast}+p^{\ast\ast}}}\right)\leq\lambda_{\max}\left(\Phi_{\boldsymbol{X}_{1}^{p^{\ast}+p^{\ast\ast}}}\right)\leq
B_{X2}m_{n}^{-1}$

holds for some positive constants $B_{X1}$ and $B_{X2},$ and it also holds
when $\boldsymbol{X}_{1}^{p^{\ast}+p^{\ast\ast}}$ is augmented by an arbitrary
element $X_{l}$ with $l>p^{\ast}+p^{\ast\ast}.$

###### Assumption 11

If the pre-selected variables $\boldsymbol{Z}$ are either signals or pseudo-
signals, the effect of noise variable defined in Table 2 on $Y$ is still weak.
That is,
$\theta_{l,\boldsymbol{Z}}\lesssim\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2}$
for all $l\in[p^{\ast}+p^{\ast\ast}+1,p_{n}]$ when all variables in
$\boldsymbol{Z}$ are either signals or pseudo-signals.

Assumption 2’ strengthens Assumption 2 by adding one more condition on
$B_{p^{\ast\ast}}$. We impose it to ensure the good property of
$\boldsymbol{\tilde{u}}_{l,\boldsymbol{Z}}$ defined in equation (A.11) (see
Lemma A.10). Assumption 5’ strengthens Assumption 5 so that equation (A.4) in
Appendix A.2 holds. As remarked at the beginning of last subsection, we do not
impose Assumption 9 for the multiple-stage procedure, as long as the joint
effect of the signal is not too weak as imposed in Assumption 9’. This
assumption seems inevitable. It is analogous to Assumption 6 in CKP for the
linear regression model and similar to the so-called ‘beta-min’ condition that
is commonly assumed in the penalized regression literature (see, e.g., Chapter
7.4 of Buhlmann and van de Geer (2011)). Assumption 10 is the common rank
condition for nonparametric additive regressions. We also require it to hold
for the case when we add one noise variable. This seems inevitable because we
will run the regression with regressors being all signals and pseudo-signals
plus one noise variable in the multiple-stage procedure with very high
probability. Assumption 11 is also inevitable and implicitly imposed in CKP
for the linear regression models _._

It is worth mentioning that Assumption 10 plays a similar role to the
“restrictive eigenvalues” condition in Bickel, Ritov and Tsybakov (2009) and
Belloni et al. (2012). The restrictive-eigenvalues condition requires certain
full rank conditions on all possible design matrices composed of a certain
number of covariates. In contrast, our OCMT only requires full rank conditions
on the design matrices composed of signals and pseudo-signals, and permits
arbitrary correlations among the noise variables. Obviously, these two sets of
conditions are non-nested. In addition, Huang, Horowitz and Wei (2010) impose
essentially the same set of assumptions except the rank conditions discussed
here. The main advantage of the OCMT is that it does not require any numerical
min-search of an objective function, can be computed much faster, and deliver
more reliable results.

The following proposition presents the probability bounds for the “Type-I” and
“Type-II” errors when we have some pre-selected variables.

###### Proposition 3.1

Suppose that $Y$ is given by equation (2.1), Assumptions 1, 2’, 3, 4, 5’, 6,
7, 8, and 10 hold, and the pre-selected variables $\boldsymbol{Z}$ are either
signals or pseudo-signals.

(i) If
$\theta_{l,\boldsymbol{Z}}\lesssim\log\left(m_{n}\right)^{1/2}m_{n}^{1/2}n^{-1/2},$
then

$\displaystyle\Pr\left(\mathcal{\hat{X}}_{l,\boldsymbol{Z}}\geq\varsigma_{n}\right)$
$\displaystyle\leq\exp\left(-C_{1}m_{n}^{-1}\varsigma_{n}+\log
m_{n}\right)+C_{2}\exp\left(-C_{3}n^{C_{4}}\right)$ (3.3) $\displaystyle\leq
n^{-M}+C_{2}\exp\left(-C_{3}n^{C_{4}}\right)$

for any fixed large constant $M>0$ and some positive constants
$C_{1},C_{2},C_{3},$ and $C_{4}.$

(ii) If
$\theta_{l,\boldsymbol{Z}}\gtrsim\kappa_{n}\log\left(m_{n}\right)^{1/2}m_{n}^{1/2}n^{-1/2}$
with $\kappa_{n}$ specified in Assumption 8, then

$\Pr\left(\mathcal{\hat{X}}_{l,\boldsymbol{Z}}\geq\varsigma_{n}\right)\geq
1-n^{-M}-C_{5}\exp\left(-C_{6}n^{C_{7}}\right)$

for any fixed large constant $M>0$ and some positive constants $C_{5},C_{6},$
and $C_{7}.$

The proof of Proposition 3.1 is rather tedious. We provide some technical
discussions on the proof in Appendix A.2 before we formally prove it in
Appendix B.

Like Proposition 2.1, Proposition 3.1 implies that for the well-chosen
threshold value $\varsigma_{n},$ the use of the test statistic
$\mathcal{\hat{X}}_{l,\boldsymbol{Z}}$ helps to separate variables with large
value of $\theta_{l,\boldsymbol{Z}}$ from those with small value of
$\theta_{l,\boldsymbol{Z}}.$ This observation will be used in our multiple-
stage procedure to select all signal variables.

###  3.3 The Multiple-Stage Procedure

We present the multiple-stage procedure as follows.

We conduct the first-stage selection as in Section 2.3 by constructing the
test statistic $\mathcal{\hat{X}}_{l}$ as in equation (2.6) and using the
threshold value $\varsigma_{n}$ that satisfies the condition in Assumption 8.
We re-label the selection indicator in equation (2.7) as

$\widehat{\mathcal{J}}_{l,\left(1\right)}=\mathbf{1}\left(\mathcal{\hat{X}}_{l}>\varsigma_{n}\right)\text{
for }l=1,2,\ldots,p_{n}.$

We collect all the variables selected in stage 1 into the vector
$\boldsymbol{Z}_{\left(1\right)}$, and denote the index set of the selected
variables by $S_{\left(1\right)}.$ For the second stage, we denote the index
set of the active variables in stage 2 by $\Psi_{\left(2\right)}$ where
$\Psi_{\left(2\right)}=\left\\{1,2,\ldots,p_{n}\right\\}\backslash
S_{\left(1\right)}$. In the second stage, we regress $Y$ on
$P^{m_{n}}\left(X_{l}\right)$ with
$P^{m_{n}}\left(\boldsymbol{Z}_{\left(1\right)}\right)$ as pre-selected
variables one by one for $l\in\Psi_{\left(2\right)}.$ We construct the test
statistic $\mathcal{\hat{X}}_{l,\boldsymbol{Z}_{\left(1\right)}}$ as in
equation (3.2). We select the variable $X_{l}$ if
$\widehat{\mathcal{J}}_{l,\left(2\right)}=1,$ where

$\widehat{\mathcal{J}}_{l,\left(2\right)}=\mathbf{1}\left(\mathcal{\hat{X}}_{l,\boldsymbol{Z}_{\left(1\right)}}>\varsigma_{n}\right)\text{
for }l\in\Psi_{\left(2\right)}.$

We add all the variables selected in stage 2 into the set of variables
selected in stage 1 as a new vector, and we denote it as
$\boldsymbol{Z}_{\left(2\right)}.$ We denote the index set of the selected
variables ($\boldsymbol{Z}_{\left(2\right)}$) by $S_{\left(2\right)}$ and the
index set of the active variables for stage 3 as $\Psi_{\left(3\right)},$
where $\Psi_{\left(3\right)}=\left\\{1,2,\ldots,p_{n}\right\\}\backslash
S_{\left(2\right)}.$ And so on and so forth. For stage $k,$ we denote the pre-
selected variables as $\boldsymbol{Z}_{\left(k-1\right)},$ and the index set
of the active variables as $\Psi_{\left(k\right)}.$ Then we regress $Y$ on
$P^{m_{n}}\left(X_{l}\right)$ with
$P^{m_{n}}\left(\boldsymbol{Z}_{\left(k-1\right)}\right)$ as pre-selected
variables one by one for $l\in\Psi_{\left(k\right)}.$ We construct the test
statistic $\mathcal{\hat{X}}_{l,\boldsymbol{Z}_{\left(k-1\right)}}$ as in
equation (3.2). We select the variable $X_{l}$ if
$\widehat{\mathcal{J}}_{l,\left(k\right)}=1,$ where

$\widehat{\mathcal{J}}_{l,\left(k\right)}=\mathbf{1}\left(\mathcal{\hat{X}}_{l,\boldsymbol{Z}_{\left(k-1\right)}}>\varsigma_{n}\right)\text{
for }l\in\Psi_{\left(k\right)}.$

We add all the variables selected in stage $k$ into the set of variables
selected in stage $k-1$ as a new vector, and we denote it as
$\boldsymbol{Z}_{\left(k\right)}.$ We stop the procedure at a stage in which
no new variables are selected. We denote the stage, in which one or more
variables are selected but no new variables are selected after that, as
$\hat{k}_{s}$. So the OCMT procedures stops after stage $\hat{k}_{s}.$ The
selection indicator for variable $X_{l}$ of the OCMT procedure is defined as
follows

$\widehat{\mathcal{J}}_{l}=\sum_{k=1}^{\hat{k}_{s}}\widehat{\mathcal{J}}_{l,\left(k\right)}.$
(3.4)

By construction, $\widehat{\mathcal{J}}_{l}$ is either 1 or 0. It takes value
$1$ if $X_{l}$ is selected in the OCMT procedure and 0 otherwise.

The following theorem mainly studies the non-asymptotic properties of the
multiple-stage procedure in terms of TPR, FPR and FDR.

###### Theorem 3.1

Suppose that Assumptions 1, 2’, 3, 4, 5’, 6, 7, 8, 9’, 10, and 11 hold.

(i) $\Pr\left(\hat{k}_{s}>p^{\ast}\right)\leq
n^{-M_{6}}+C_{19}\exp\left(-C_{20}n^{C_{21}}\right)$ for some fixed large
positive number $M_{6}$ and some positive constants $C_{19},C_{20},$ and
$C_{21};$

(ii) $E\left(\text{\emph{TPR}}_{n}\right)\geq
1-C_{1}\exp\left(-C_{2}n^{C_{3}}\right)$ for some positive constants
$C_{1},C_{2}$, and $C_{3};$

(iii) $E\left(\text{\emph{FPR}}_{n}\right)\leq
p^{\ast\ast}/\left(p_{n}-p^{\ast}\right)+C_{4}n^{-M}$ for some positive
$C_{4}$ and any fixed positive large constant $M;$

(iv) _FDR_ ${}_{n}\overset{P}{\rightarrow}0.$

Theorem 3.1(i) implies the OCMT procedure can terminate at step $p^{\ast}$
with very high probability. Theorem 3.1(ii)-(iv) implies that all of
TPR${}_{n},$ FPRn and FDRn can be well controlled. All results but part (iv)
in the above theorem is a non-asymptotic result and works for all finite
sample sizes in principle. Of course, when $n\rightarrow\infty,$ we need to
conduct at most $p^{\ast}$-stage procedure to determine all signals and
eliminate all noise variables, with very high probability.

###  3.4 Dealing with Pseudo-signals

Because of the nature of the OCMT procedure, pseudo-signals cannot be excluded
from the selection list with high probability. To eliminate pseudo-signals, we
propose to employ the adaptive group Lasso after the OCMT. Then by the
properties of the adaptive group Lasso, the event that pseudo-signals are
excluded and signals are kept happens with probability approaching one
(w.p.a.1). We present this result in Theorem 3.2. We denote the set of the
variables selected by the OCMT procedure as $\hat{S}_{\text{OCMT}}\equiv
S_{\left(\hat{k}_{s}\right)},$ collect them into a vector
$\boldsymbol{Z}_{\text{OCMT}},$ and denote its dimension as
$\hat{p}_{\text{OCMT}}$. Similarly, let $\mathbb{Z}_{\text{OCMT}}$
($n\times\hat{p}_{\text{OCMT}}m_{n}$) denote the design matrix for
$\boldsymbol{Z}_{\text{OCMT}}$ as in equation (3.1). The post-OCMT adaptive
group Lasso procedure is as follows:

  1. 1.

Obtain the group Lasso estimator by searching $\boldsymbol{\beta}_{n}$
($\hat{p}_{\text{OCMT}}m_{n}\times 1$) to minimize

$L_{n1}\left(\boldsymbol{\beta}_{n},\lambda_{n1}\right)=\left\|\boldsymbol{y}-\mathbb{Z}_{\text{OCMT}}\boldsymbol{\beta}_{n}\right\|^{2}+\lambda_{n1}\sum_{j=1}^{\hat{p}_{\text{OCMT}}}\left\|\boldsymbol{\beta}_{nj}\right\|,$

where $\lambda_{n1}$ is a positive tuning parameter,
$\boldsymbol{\beta}_{n}=(\boldsymbol{\beta}_{n1}^{\prime},...,\boldsymbol{\beta}_{n\hat{p}_{\text{OCMT}}}^{\prime})^{\prime},$
$\boldsymbol{\beta}_{nj}$ is a $m_{n}\times 1$ vector of coefficients of the
B-spline basis for the $j$-th element in $\boldsymbol{Z}_{\text{OCMT}}.$
Denote the above estimator as $\boldsymbol{\tilde{\beta}}_{n}.$

  2. 2.

The adaptive group Lasso estimator is obtained by searching
$\boldsymbol{\beta}_{n}$ to minimize

$L_{n2}\left(\boldsymbol{\beta}_{n},\lambda_{n2}\right)=\left\|\boldsymbol{y}-\mathbb{Z}_{\text{OCMT}}\boldsymbol{\beta}_{n}\right\|^{2}+\lambda_{n2}\sum_{j=1}^{\hat{p}_{\text{OCMT}}}\frac{1}{\left\|\boldsymbol{\tilde{\beta}}_{nj}\right\|}\left\|\boldsymbol{\beta}_{nj}\right\|,$

where we use the convention that $0/0=0$. Denote the above estimator as
$\boldsymbol{\hat{\beta}}_{n}.$

The post-selection estimation proceeds as follows. Denote the selected
regressor from the above procedure as $\boldsymbol{Z}_{\text{AGLASSO}},$ and
similarly denote its B-spline basis and design matrix as
$P^{m_{n}}\left(\boldsymbol{Z}_{\text{AGLASSO}}\right)$ and
$\mathbb{Z}_{\text{AGLASSO}},$ respectively. The post selection estimator is
the OLS estimator of regressing $\boldsymbol{y}$ on
$\mathbb{Z}_{\text{AGLASSO}}$, which is

$\boldsymbol{\hat{\beta}}_{\text{post}}=\left(\mathbb{Z}_{\text{AGLASSO}}^{\prime}\mathbb{Z}_{\text{AGLASSO}}\right)^{-1}\mathbb{Z}_{\text{AGLASSO}}^{\prime}\boldsymbol{y.}$

###### Theorem 3.2

Suppose that Assumptions 1, 2’, 3, 4, 5’, 6, 7, 8, 9’, 10, and 11 hold.
Further, $\lambda_{n1}\geq C\sqrt{n\log\left(p^{\ast\ast}m_{n}\right)}$ for a
sufficient large $C$, and
$m_{n}^{1/2}\log\left(p^{\ast\ast}m_{n}\right)\ll\lambda_{n2}\ll
nm_{n}^{-1/4}.$ Then all signal variables are kept and all pseudo signals are
eliminated with probability approaching 1 (w.p.a.1).

The above theorem is almost the same as that in Huang, Horowitz and Wei (2010)
with the exception that the procedure starts with the covariates post OCMT.
Consequently, we only need to show that the adaptive group Lasso procedure is
still valid in the post-OCMT situation, and the rest follows immediately from
Huang, Horowitz and Wei (2010). Our procedure enjoys the same theoretical
property as the adaptive group Lasso. After the OCMT, the dimension of
candidate variables is reduced dramatically. Thus the additional computation
burden applying the post-OCMT adaptive group Lasso can be almost ignored. We
note that the main advantage of our procedure is fast and reliable
computation, which delivers better small sample performance as shown from our
simulation studies. An implication of the above theorem is that the post-OCMT
adaptive group procedure improves over the post-selection estimation results
in CKP (Theorem 2 in CKP) because pseudo-signals are now eliminated with very
high probability.

###  3.5 Diverging $p^{\ast}$

Allowing a diverging $p^{\ast}$ (number of true signals) is possible. The only
additional technical condition apart from the restriction on the speed of
$p^{\ast}$ is that $\sum_{j=1}^{p^{\ast}}f_{j}^{\ast}\left(X_{j}\right)$ is
uniformly bounded. Note that this condition naturally holds for a fixed
$p^{\ast}$ due to the boundedness of $f_{j}^{\ast}$. The main reason for the
requirement of this condition is technical: the uniform boundedness of
$\sum_{j=1}^{p^{\ast}}f_{j}^{\ast}\left(X_{j}\right)$ ensures that $U_{l}$
defined in equation (2.5) also satisfies the exponential decayed tail
condition,111For details, see the proof of Lemma A.5. and the tail property is
necessary to apply the main inequalities to obtain the probability bounds. The
uniform boundedness of $\sum_{j=1}^{p^{\ast}}f_{j}^{\ast}\left(X_{j}\right)$
was also imposed in Fan, Feng and Song (2011). Propositions 2.1 and 3.1 are on
individual $X_{l},$ but we do need Assumption 2” on $p^{\ast}$ so that
Proposition 3.1 holds. It is to ensure that we can have precise estimation on
a diverging design matrix.

Assumption 2” $p^{\ast}\lesssim n^{B_{p^{\ast}}}$ for some $B_{p^{\ast}}\geq
0.$ $p^{\ast\ast}\lesssim n^{B_{p^{\ast\ast}}}\ $and $p_{n}\propto
n^{B_{p_{n}}}$ for some $B_{p_{n}}>B_{p^{\ast\ast}},B_{p^{\ast}}\geq 0.$
Further, $B_{p^{\ast}}+B_{p^{\ast\ast}}<\left(1-3B_{m}\right)/2.$

We present the main results in the following theorem.

###### Theorem 3.3

Suppose that Assumptions 1, 2”, 3, 4, 5’, 6, 7, 8, 9’, 10, and 11 hold. In
addition, assume that $\sum_{j=1}^{p^{\ast}}f_{j}^{\ast}\left(X_{j}\right)$ is
uniformly bounded.. Then, the results in Theorem 3.1 continue to hold.

In the next section, we investigate the small sample performance of our
procedure by means of Monte Carlo experiment.

##  4 Monte Carlo Simulations

To investigate the small-sample performance of our procedure, we conduct Monte
Carlo experiments in this section.

###  4.1 Simulation Design

Following Huang, Horowitz and Wei (2010), we consider the following data
generating processes (DGPs). In what follows, we assume

$\displaystyle f_{1}\left(x\right)$ $\displaystyle=x;\text{\
}f_{2}\left(x\right)=(2x-1)^{2};\text{\
}f_{3}\left(x\right)=\frac{\sin\left(2\pi x\right)}{2-\sin\left(2\pi
x\right)};\text{ and}$ $\displaystyle f_{4}\left(x\right)$
$\displaystyle=0.1\sin\left(2\pi x\right)+0.2\cos\left(2\pi
x\right)+0.3\sin\left(2\pi x\right)^{2}+0.4\cos\left(2\pi
x\right)^{3}+0.5\sin\left(2\pi x\right)^{3}.$

For the errors, we assume $\varepsilon\sim$ i.i.d. $N\left(0,1\right)$ for
DGPs 1–6 and 9–10. In DGPs 7–8, we check the impact of heteroskedastic errors
on our methods. Specifically, we add a heteroskedastic error to the simplest
and the most complicated designs in DGPs 1–6 to form DGPs 7 and 8,
respectively. In DGPs 9–10, we consider the case with additive components of
binary variables that mimic the application in Section 5.

DGP 1: Four independent signals only. $Y$ is generated as follows:

$Y=2.55f_{1}\left(X_{1}\right)+2.57f_{2}\left(X_{2}\right)+1.68f_{3}\left(X_{3}\right)+f_{4}\left(X_{4}\right)+\varepsilon.$
(4.1)

Note the coefficients before $f_{i}$’s are set to make each signal have the
same strength in terms of variance for independent uniform
$X_{1},\ldots,X_{4}$. The covariates are generated as follows$:$

$\displaystyle X_{j}$ $\displaystyle=W_{j}\text{, }j=1,\ldots,4,\text{ and}$
$\displaystyle X_{j}$ $\displaystyle=\frac{W_{j}+U_{1}}{2}\text{, }j\geq 5,$

where $W_{j},$ $j=1,\ldots,p_{n}$, and $U_{1}$ are all independent draws from
$U(0,1)$. Thus, $p^{\ast}=4$ and $p^{\ast\ast}=0$ for DGP 1. Define the
Signal-to-noise ratio to be
$r_{sn}=\frac{\mathtt{sd}\left(f\right)}{\mathtt{sd}\left(\varepsilon\right)}$,
and $r_{sn}=1.5$ for DGP 1.

DGP 2: Four independent signals and two pseudo-signals. $Y$ is generated from
equation (4.1). The covariates are generated as follows$:$

$\displaystyle X_{j}$ $\displaystyle=W_{j}\text{, }j=1,\ldots,4,$
$\displaystyle X_{5}$ $\displaystyle=\frac{4X_{1}+U_{1}}{5}\text{,}$
$\displaystyle\text{ }X_{6}$ $\displaystyle=\frac{4X_{2}+U_{2}}{5},\text{
and}$ $\displaystyle X_{j}$ $\displaystyle=\frac{W_{j-2}+U_{3}}{2}\text{,\
}j\geq 7,$

where $W_{j},$ $j=1,\ldots,p_{n}-2$, $U_{1},$ and $U_{2}$ are all independent
draws from $U(0,1)$. Thus, $p^{\ast}=4$ and $p^{\ast\ast}=2$ for DGP 2.

DGP 3: Four signals, and one hidden signal. $Y$ is generated from

$Y=2.55f_{1}\left(X_{1}\right)+2.57f_{2}\left(X_{2}\right)+1.68f_{3}\left(X_{3}\right)+f_{4}\left(X_{4}\right)+f_{5}\left(X_{5}\right)+\varepsilon,$
(4.2)

where
$f_{5}\left(X_{5}\right)=-\mathbb{E}\left[2.55f_{1}\left(X_{1}\right)+2.57f_{2}\left(X_{2}\right)+1.68f_{3}\left(X_{3}\right)+f_{4}\left(X_{4}\right)|X_{5}\right].$
The covariates are generated as follows $:$

$\displaystyle X_{j}$ $\displaystyle=W_{j}\text{, }j=1,2,\text{ }$
$\displaystyle X_{j}$ $\displaystyle=\frac{W_{j}+U_{1}}{2}\text{, }j=3,4,$
$\displaystyle X_{5}$ $\displaystyle=U_{1}\text{, and}$ $\displaystyle X_{j}$
$\displaystyle=\frac{W_{j-1}+U_{2}}{2}\text{, }j\geq 6,$

where $W_{j},$ $j=1,\ldots,p_{n}-1,$ $U_{1},$ and $U_{2}$ are independent
draws from $U(0,1)$. Then, $p^{\ast}=5$ and $p^{\ast\ast}=0$ for DGP 3, and
the fifth signal is hidden by our definition.222By the distribution of
covariates,  $\displaystyle f_{5}\left(x\right)$ $\displaystyle\approx
0.97\pi-1.2\cos(\pi x)+\sin\left(\pi x\right)+0.6861\pi\arctan[2\left(\tan(\pi
x/2)-1\right)/\sqrt{3}]$ $\displaystyle-0.6861\pi\arctan[2\left(\tan(\pi
x/2+\pi/2)-1\right)/\sqrt{3}]+0.2778\cos^{3}(\pi x)-0.2222\sin^{3}(\pi x).$

DGP 4: Four signals, two pseudo-signals, and one hidden signal. $Y$ is
generated from equation (4.2). The covariates are generated as follows $:$

$\displaystyle X_{j}$ $\displaystyle=W_{j},\text{ }j=1,2,\text{ }$
$\displaystyle X_{j}$ $\displaystyle=\frac{W_{j}+U_{1}}{2},\text{ }j=3,4,$
$\displaystyle X_{5}$ $\displaystyle=U_{1},\text{\
}X_{6}=\frac{4X_{1}+U_{2}}{5}\text{, }X_{7}=\frac{4X_{2}+U_{3}}{5},\text{
and}$ $\displaystyle X_{j}$ $\displaystyle=\frac{W_{j-3}+U_{3}}{2},\text{
}j\geq 8,$

where $W_{j},$ $j=1,\ldots,p_{n}-3,$ $U_{1},$ $U_{2},$ and $U_{3}$ are
independent draws from $U(0,1)$. Then, $p^{\ast}=5$ and $p^{\ast\ast}=2$ for
DGP 4, and the fifth signal is a hidden signal.

DGP 5: Four correlated signals. $Y$ is generated from equation (4.1). The
covariates are generated as follows$:$

$\displaystyle X_{j}$ $\displaystyle=\frac{W_{j}+U_{1}}{2},\text{\
}j=1,\ldots,4,\text{ and}$ (4.3) $\displaystyle X_{j}$
$\displaystyle=\frac{W_{j}+U_{2}}{2},\text{\ }j\geq 5,$

where $W_{j},$ $j=1,\ldots,p_{n}$, $U_{1},$ and $U_{2}$ are independent draws
from $U(0,1)$. Thus, four signals are correlated with each other, and
$p^{\ast}=4$ and $p^{\ast\ast}=0$ for DGP 5.

DGP 6: Four signals, many pseudo-signals, and one hidden signal. $Y$ is
generated from equation (4.2). The covariates are generated as follows $:$

$\displaystyle X_{j}$ $\displaystyle=W_{j},\text{ }j=1,2,\text{ }$
$\displaystyle X_{j}$ $\displaystyle=\frac{W_{j}+U_{1}}{2},\text{ }j=3,4,$
$\displaystyle X_{5}$ $\displaystyle=U_{1},$ $\displaystyle X_{j}$
$\displaystyle=\frac{4X_{1}+\left(j-5\right)W_{j-1}}{j-1},\text{
}j=6,10,14,18,\ldots,$ $\displaystyle X_{j}$
$\displaystyle=\frac{4X_{2}+\left(j-5\right)W_{j-1}}{j-1},\text{
}j=7,11,15,19,\ldots,$ $\displaystyle X_{j}$
$\displaystyle=\frac{4X_{3}+\left(j-5\right)W_{j-1}}{j-1},\text{
}j=8,12,16,20,\ldots,\text{ and}$ $\displaystyle X_{j}$
$\displaystyle=\frac{4X_{4}+\left(j-5\right)W_{j-1}}{j-1},\text{
}j=9,13,17,21,\ldots$

where $W_{j},$ $j=1,\ldots,p_{n}-1,$ and $U_{1}$ are independent draws from
$U(0,1)$. Then, $p^{\ast}=5$ with one hidden signal for DGP 6.

DGP 7: Four independent signals with heteroskedastic errors. $Y$ is generated
from equation (4.1) with the same covariates as in DGP 1. We assume that
conditioning on $X,$ $\varepsilon$ is normal with mean 0 and variance
$0.436[1+\left(X_{1}+X_{2}+X_{3}+X_{4}\right)/4]^{2},$ and the unconditional
variance of $\varepsilon$ is approximately 1.

DGP 8: Four signals, many pseudo-signals, and one hidden signal with
heteroskedastic errors. $Y$ is generated from equation (4.2) with the same
covariates as in DGP 6. We assume that conditioning on $X,$ $\varepsilon$ is
normal with mean 0 and variance
$0.436[1+\left(X_{1}+X_{2}+X_{3}+X_{4}\right)/4]^{2}$.

DGP 9: Four independent signals with some binary variables. To mimic the
application, we consider the situation with some binary covariates. Note that
any function of a binary covariate can at most take two values. Without loss
of generality, we focus on the case in which those binary covariates enter the
model linearly. It is easy to see that our theoretical results continue to
hold in the presence of some linear additive components with the main
difference that they do not exhibit any approximation bias.333In this case,
the test statistics for the additive linear terms are the squares of
t-statistics, and the non-asymptotic inequality continues to hold by setting,
for example, $\varsigma_{n}\propto\left[\log p_{n}\right]^{1.1}$ for the case
when $p_{n}$ is a polynomial of $n.$ We set the threshold as
$\varsigma_{n}=C\left[\log p_{n}\right]^{1.1}$. $Y$ is generated from

$Y=2.57f_{2}\left(X_{1}\right)+1.68f_{3}\left(X_{2}\right)+1.47X_{3}+1.47X_{4}+\varepsilon,$

where $X_{j}=W_{j},$ $j=1,2,$ $X_{j}=V_{j-2},$ $j=3,4,$ $W_{1}$ and $W_{2}$
are independent $U(0,1),$ and $V_{1}$ and $V_{2}$ are independent Bernoulli
random variables with equal chances of taking value 0 or 1. The remaining
covariates are generated as follows:

$\displaystyle X_{j}$ $\displaystyle=\frac{W_{j-2}+U_{1}}{2},\text{
}j=5,6,...,\frac{p_{n}}{2},$ $\displaystyle X_{j}$
$\displaystyle=V_{j-\frac{p_{n}}{2}+2},\text{ }j=\frac{p_{n}}{2}+1,...,p_{n},$

where $W_{j},$ $j=5,6,...,\frac{p_{n}}{2}-2,$ and $U_{1},$ are independent
$U(0,1),$ $V_{j},$ $j=3,...,\frac{p_{n}}{2}+2$ are distributed the same as
$V_{1}.$ All $W$s$,V$s, and $U$ are independent of each other.

DGP 10: Four signals with one hidden signal in the presence of some binary
variables. For this DGP, $Y$ is generated from

$Y=2.57f_{2}\left(X_{1}\right)+1.5X_{2}+1.5X_{3}-X_{4}+\varepsilon,$

where $X_{1}=W_{1}$ is a $U(0,1),$ $X_{j}=V_{j-1},$ $j=2,3,4,$ are Bernoulli
random variables with equal chances of taking 0 or 1, and
Corr$\left(V_{1},V_{3}\right)=$Corr$\left(V_{1},V_{2}\right)=$Corr$\left(V_{2},V_{3}\right)=1/3$.
$X_{j},$ $j=5,6,...,p_{n}$ are the same as those in DGP 9.444An example of
$V_{1},V_{2},$ and $V_{3}$ is that
$V_{j}=\mathbf{1(}\tilde{U}_{j}+\tilde{U}_{4}>1)$ for $j=1,2,$ and $3,$ where
$\tilde{U}_{1},...,\tilde{U}_{4}$ are independent $U\left(0,1\right).$ Some
simple calculation implies that $X_{4}$ is a hidden signal.

###  4.2  Tuning Parameters and Procedure

The key tuning parameter in this study is the threshold $\varsigma_{n}.$ In
the simulations, we set

$\varsigma_{n}=Cm_{n}\left(\left[\log p_{n}\right]^{1.1}+\left[\log
m_{n}\right]^{1.1}\right)$

for continuous variables and

$\varsigma_{n}=C\left[\log p_{n}\right]^{1.1}$

for binary variables, which satisfies Assumption 8 when, in addition,
Assumptions 2 and 7 hold (both $m_{n}$ and $p_{n}$ grow at the polynomial rate
of $n$). In the small samples, we set this $\varsigma_{n}$ to have control
over both $m_{n}$ and $p_{n}$. CKP suggest using a larger threshold,
$\varsigma_{n}^{\ast}$, for subsequent stages, to improve the finite sample
performance. We follow their lead and set a $\varsigma_{n}^{\ast}$ larger than
$\varsigma_{n}$ for subsequent stages. That is, we replace $\varsigma_{n}$
with $\varsigma_{n}^{\ast}$ for
$\widehat{\mathcal{J}}_{l,\left(2\right)},\widehat{\mathcal{J}}_{l,\left(3\right)},...,\widehat{\mathcal{J}}_{l,\left(k\right)}$
in Section 3.3. The reason is that the chance of including a noise variable
increases quickly for subsequent stages, and we need a larger
$\varsigma_{n}^{\ast}$ for the OCMT to conclude more easily. We set
$\varsigma_{n}^{\ast}=3\varsigma_{n}$ for continuous variables. Note that CKP
take the threshold for later stages to be twice the threshold for the first
stage, and their test is based on $t$-statistics. Since ours is based on chi-
squared statistics, equivalently, we should set $\varsigma_{n}^{\ast}$ to be
$4\varsigma_{n}.$ However, some small-scale experiments suggest that setting
$\varsigma_{n}^{\ast}=3\varsigma_{n}$ can yield better small sample
performance, probably because our model is nonparametric and our test
statistic varies more than the parametric counterpart. Note that this change
does not affect our theoretical results because $\varsigma_{n}^{\ast}$ is
proportional to $\varsigma_{n}$. For binary additive components, we continue
to follow CKP and set $\varsigma_{n}^{\ast}=4\varsigma_{n}$ because they enter
the model linearly.

For $m_{n},$ we simply set $m_{n}=\left\lfloor n^{1/4}\right\rfloor+1,$ where
$\left\lfloor\cdot\right\rfloor$ is the floor operator. This $m_{n}$ satisfies
Assumption 7, if $d>3/2$. Other choices of sieve terms such as
$m_{n}=\left\lfloor n^{1/4}\right\rfloor+2$ are also considered in the
simulations, and they yield similar results.

For the $C$ in $\varsigma_{n}=Cm_{n}\left(\left[\log
p_{n}\right]^{1.1}+\left[\log m_{n}\right]^{1.1}\right)$ or
$\varsigma_{n}=C\left[\log p_{n}\right]^{1.1},$ we test $C$ in the range of
$0.5$ to $2.5$, specifically, $0.5,0.6,...,2.5$. We determine the value of $C$
by minimizing the following BIC:

$\text{BIC}\left(C\right)=n\log\left[\text{RSS}\left(C\right)/n\right]+\left(\text{number
of selected variables}\right)\cdot\log\left(n\right),$ (4.4)

where RSS$\left(C\right)$ denotes the residual sum of squares from the post-
OCMT ordinary least squares regression of $Y$ on
$P^{m_{n}}\left(\boldsymbol{Z}_{\left(\hat{k}_{s}\right)}\right)$, where
$\boldsymbol{Z}_{\left(\hat{k}_{s}\right)}$ are the variables selected by the
OCMT with $C$ in use. The tuning parameters for the adaptive group Lasso are
selected as in Section 4.3.

We present our procedure formally in the following:

  1. 1.

Set $\varsigma_{n}=0.5m_{n}\left(\left[\log p_{n}\right]^{1.1}+\left[\log
m_{n}\right]^{1.1}\right)$ and $\varsigma_{n}^{\ast}=3\varsigma_{n}$ for
continuous variables, $\varsigma_{n}=0.5\left[\log p_{n}\right]^{1.1}$ and
$\varsigma_{n}^{\ast}=4\varsigma_{n}$ for binary variables, and conduct the
OCMT procedure in Section 3.3;

  2. 2.

Re-estimate the model by regressing $Y$ on
$P^{m_{n}}\left(\boldsymbol{Z}_{\left(\hat{k}_{s}\right)}\right)$ (which is a
post-selection estimation);

  3. 3.

Compute the BIC from Step 2 and denote it as BIC${}_{1};$

  4. 4.

Repeat Steps 1 to 3, replacing $C=0.5$ in the definition of $\varsigma_{n}$
and $\varsigma_{n}^{\ast}$ by $0.6,$ $0.7,\ldots,$ $2.5$ and obtain the
corresponding BIC, denoted as BIC${}_{2},$ BIC${}_{3},\ldots,$ BIC${}_{21};$

  5. 5.

Find $\hat{\imath}=\arg\min_{1\leq\iota\leq 21}$ BICι and let the selected
variables be those obtained by using $\hat{\imath}$-th $\varsigma_{n}$ and
$\varsigma_{n}^{\ast};$

  6. 6.

Conduct the adaptive group Lasso for those variables selected in Step 5.

###  4.3 Estimator Compared

For comparison, we consider the adaptive group Lasso by Huang, Horowitz and
Wei (2010), which is designed for component selection in the nonparametric
additive model. It is a two-step approach—the first step is the usual group
Lasso and the second is the adaptive group Lasso with initial estimates from
the first step. We select tuning parameters $\lambda_{n1}$ in step 1 and
$\lambda_{n2}$ in step 2 ($\lambda_{n1}$ and $\lambda_{n2}$ are in the
notations of Huang, Horowitz and Wei (2010)) by BIC as well. Specifically, we
set $\lambda_{n1}=\lambda_{j}$ with

$\lambda_{j}=\exp\left\\{\log\left(\lambda_{\max}\right)+\left[\log\left(\lambda_{\min}\right)-\log\left(\lambda_{\max}\right)\right]\frac{j}{30}\right\\}$

for $j=0,1,...,30,$ where
$\lambda_{\min}=\max\left\\{0.05,10^{-5}\left\|\boldsymbol{y}\right\|\right\\}$
and $\lambda_{\max}=0.5\left\|\boldsymbol{y}\right\|.$ We calculate BICj based
on the estimation using $\lambda_{n1}=\lambda_{j},$ and we select the
$\hat{\lambda}_{n1}$ that minimizes the BICj among $j=0,1,...,30.$ We set the
estimates in the first step as the estimates using $\hat{\lambda}_{n1}.$ For
the second step, we set $\lambda_{n2}=\lambda_{j}$ for $j=0,1,...,30$ with the
initial estimates as the estimates from the first step using
$\hat{\lambda}_{n1}$. We again calculate BICj based on the estimation using
$\lambda_{n2}=\lambda_{j},$ and we select the $\hat{\lambda}_{n2}$ that
minimizes the BICj among $j=0,1,...,30.$ The final estimates are the adaptive
group Lasso estimates using $\hat{\lambda}_{n2}$. Note that the variables not
selected in the first step are not included in the second step, because the
penalty for those variables is infinity in the second step.

We find the solutions in both steps through the block coordinate descent
algorithm (i.e., the “shooting” algorithm). For the algorithm details, see Wu
and Lange (2008).555One implementation in MATLAB can be found at:
https://publish.illinois.edu/xiaohuichen/code/group-lasso-shooting/.

###  4.4 Estimation Results

We consider the combinations of $n=200$ or $400$ and $p_{n}=100,200$, or
$1000$ for each DGP. All results are based on 1,000 replications. We report
the results for three different methods. The first and second methods are our
post-OCMT procedure (Steps 1–6 in Section 4.2, denoted as “POST–OCMT”) and
OCMT procedure (Steps 1– 5 in Section 4.2, denoted as “OCMT”), respectively.
The third method is the adaptive group Lasso (denoted as “AGLASSO”).

Following the literature, we report the mean number of variables selected
(NV), the true positive rates (TPR), the false positive rates (FPR), the false
discovery rates (FDR), the percentage of the true model selected precisely
(CS), and the out-of-sample forecast root mean square errors (FRMSE) for all
the three methods.666The forecasts are based on the post-selection estimates
from each method. In each replication, we additionally independently generate
200 observations. Then, we calculate the root mean square errors (RMSE) of the
difference between the forecasts and $Y$ for the new observations. FRMSE is
the average of those RMSE for 1,000 replications. We report the average number
of stages (STEP) for our OCMT procedure only.

To save space, we report the results in the main body of this paper for only
DGPs 1 and 6 in Tables 3 and 4, respectively. These two designs are the
simplest and the most complicated designs in the simulation. Results for DGPs
2–5 and 7–10 are provided in Tables S1 to S8 in Appendix S2.

Table 3: DGP 1 Panel 1: $N=200$, $P=100$
---
| NV | TPR | FPR | FDR | CS | STEP | FRMSE
POST-OCMT | 4.0080 | 0.9998 | 0.0001 | 0.0015 | 0.9910 | - | 1.0718
OCMT | 4.0180 | 0.9998 | 0.0002 | 0.0031 | 0.9820 | 1.0260 | 1.0886
AGLASSO | 4.0410 | 0.9990 | 0.0005 | 0.0075 | 0.9550 | - | 1.0753
Panel 2: $N=200$, $P=200$
| NV | TPR | FPR | FDR | CS | STEP | FRMSE
POST-OCMT | 4.0150 | 0.9998 | 0.0001 | 0.0027 | 0.9830 | - | 1.0740
OCMT | 4.0200 | 0.9998 | 0.0001 | 0.0035 | 0.9790 | 1.0390 | 1.0939
AGLASSO | 4.0510 | 0.9950 | 0.0004 | 0.0114 | 0.9330 | - | 1.0808
Panel 3: $N=200$, $P=1000$
| NV | TPR | FPR | FDR | CS | STEP | FRMSE
POST-OCMT | 4.0010 | 0.9952 | 0.0000 | 0.0034 | 0.9640 | - | 1.0773
OCMT | 4.0200 | 0.9952 | 0.0000 | 0.0062 | 0.9510 | 1.0880 | 1.1173
AGLASSO | 3.9370 | 0.9620 | 0.0001 | 0.0145 | 0.8770 | - | 1.0993
Panel 4: $N=400$, $P=100$
| NV | TPR | FPR | FDR | CS | STEP | FRMSE
POST-OCMT | 4.0010 | 1.0000 | 0.0000 | 0.0002 | 0.9990 | - | 1.0470
OCMT | 4.0010 | 1.0000 | 0.0000 | 0.0002 | 0.9990 | 1.0000 | 1.0470
AGLASSO | 4.0080 | 0.9990 | 0.0001 | 0.0020 | 0.9870 | - | 1.0531
Panel 5: $N=400$, $P=200$
| NV | TPR | FPR | FDR | CS | STEP | FRMSE
POST-OCMT | 4.0010 | 1.0000 | 0.0000 | 0.0002 | 0.9990 | - | 1.0477
OCMT | 4.0010 | 1.0000 | 0.0000 | 0.0002 | 0.9990 | 1.0000 | 1.0477
AGLASSO | 4.0210 | 1.0000 | 0.0001 | 0.0035 | 0.9790 | - | 1.0520
Panel 6: $N=400$, $P=1000$
| NV | TPR | FPR | FDR | CS | STEP | FRMSE
POST-OCMT | 4.0040 | 1.0000 | 0.0000 | 0.0007 | 0.9960 | - | 1.0519
OCMT | 4.0040 | 1.0000 | 0.0000 | 0.0007 | 0.9960 | 1.0000 | 1.0519
AGALSSO | 4.0313 | 1.0000 | 0.0000 | 0.0052 | 0.9688 | - | 1.0524
Table 4: DGP 6 Panel 1: $N=200$, $P=100$
---
| NV | TPR | FPR | FDR | CS | STEP | FRMSE
POST-OCMT | 3.6300 | 0.6628 | 0.0033 | 0.0650 | 0.1770 | - | 1.2296
OCMT | 4.8450 | 0.6834 | 0.0149 | 0.2163 | 0.0000 | 1.2650 | 1.4003
AGLASSO | 2.9840 | 0.5522 | 0.0023 | 0.0432 | 0.0040 | - | 1.3670
Panel 2: $N=200$, $P=200$
| NV | TPR | FPR | FDR | CS | STEP | FRMSE
POST-OCMT | 3.5600 | 0.6438 | 0.0017 | 0.0682 | 0.1290 | - | 1.2396
OCMT | 4.6700 | 0.6642 | 0.0069 | 0.2107 | 0.0000 | 1.2220 | 1.3861
AGLASSO | 2.6340 | 0.4866 | 0.0010 | 0.0417 | 0.0000 | - | 1.3695
Panel 3: $N=200$, $P=1000$
| NV | TPR | FPR | FDR | CS | STEP | FRMSE
POST-OCMT | 3.2550 | 0.5892 | 0.0003 | 0.0669 | 0.0440 | - | 1.2674
OCMT | 4.2310 | 0.6088 | 0.0012 | 0.2040 | 0.0000 | 1.0880 | 1.3311
AGLASSO | 1.9780 | 0.3678 | 0.0001 | 0.0306 | 0.0000 | - | 1.4304
Panel 4: $N=400$, $P=100$
| NV | TPR | FPR | FDR | CS | STEP | FRMSE
POST-OCMT | 4.8840 | 0.9448 | 0.0017 | 0.0246 | 0.6960 | - | 1.0659
OCMT | 7.4660 | 0.9534 | 0.0281 | 0.3056 | 0.0000 | 1.8380 | 1.4894
AGLASSO | 4.3480 | 0.8276 | 0.0022 | 0.0331 | 0.1060 | - | 1.1294
Panel 5: $N=400$, $P=200$
| NV | TPR | FPR | FDR | CS | STEP | FRMSE
POST-OCMT | 4.8870 | 0.9488 | 0.0007 | 0.0213 | 0.7080 | - | 1.0723
OCMT | 7.4120 | 0.9542 | 0.0135 | 0.3019 | 0.0000 | 1.8360 | 1.5018
AGLASSO | 4.2240 | 0.8122 | 0.0008 | 0.0270 | 0.0630 | - | 1.1410
Panel 6: $N=400$, $P=1000$
| NV | TPR | FPR | FDR | CS | STEP | FRMSE
POST-OCMT | 4.8380 | 0.9356 | 0.0002 | 0.0242 | 0.6740 | - | 1.0784
OCMT | 7.4140 | 0.9398 | 0.0027 | 0.3082 | 0.0000 | 1.8030 | 1.4863
AGLASSO | 4.1340 | 0.7916 | 0.0002 | 0.0299 | 0.0110 | - | 1.1509

We can make several observations. First, POST–OCMT performs the best among the
three methods in most cases. POST–OCMT outperforms OCMT due to its ability to
eliminate pseudo-signals or noise variables. POST–OCMT even outperforms OCMT
slightly for DGPs without any pseudo-signals, especially for $n=200$. This
result confirms the necessity of conducting the additional post-OCMT step.
Moreover, the additional computation time for POST–OCMT compared with OCMT can
be almost ignored because the number of candidate variables after OCMT is very
small. Second, AGLASSO performs almost the same as our procedures for DGP 2.
Note that DGP 2 contains four independent signals, two pseudo-signals, and no
hidden signals. Then, this result is not surprising, because Lasso performs
very well for independent signals, and the biggest challenge for our procedure
is the possible presence of pseudo-signals. When the signals are correlated in
DGP 4, AGLASSO performs less well and is outperformed by POST–OCMT. Third, the
performance of all methods improves as $n$ increases from $200$ to $400$.
Fourth, OCMT is very successful at picking up hidden signals, especially for
$n=400$; see, for example, the results for DGPs 3, 4, 6, and 8. Fifth, our
methods perform well in the presence of heteroskedastic errors for DGP 7 and
8. Sixth, the CS of POST–OCMT performs well even for the complicated DGPs 6
and 8 at $n=400$, whereas AGLASSO performs poorly in terms of CS for these two
DGPs. Seventh, the number of stages for OCMT basically confirms our
theoretical findings. For example, for DGP 1, the mean number of stages is
slightly more than 1 for $n=200$, and is 1 for $n=400$. In the presence of
hidden signals, the mean number of stages is approximately 2 for $n=400$ for
DGPs 3, 4, 6, 8, and 10. Note the mean number of stages is approximately 2 for
$n=400$ for DGP 5 with correlated signals and no hidden signals. The reason is
that the correlation makes the marginal effect of $X_{1}$ on $Y$ very small
and $X_{1}$ almost behaves like a “hidden signal”. This result also confirms
the necessity of using multiple stages instead of a single-stage procedure.
Eighth, our method also works well for DGPs 9 and 10 that mimic the
application. A final remark is that our procedure can be implemented fast and
is much faster than AGLASSO. For example, when $p=1,000,$ our procedure took
less than half a minute for one replication on average, whereas the AGLASSO
took hours for one replication.

To summarize, our methods perform well in small samples, and we view it as a
useful alternative to existing methods in the literature.

##  5 An Application

In this section, we apply our method to a dataset extracted from
RUMiC.777RUMiC consists of three parts: the Urban Household Survey, the Rural
Household Survey, and the Migrant Household Survey. A group of researchers at
the Australian National University, the University of Queensland, and the
Beijing Normal University initiated this survey. The Institute for the Study
of Labor (IZA) supported it and provides the Scientific Use Files. RUMiC had
financial support from the Australian Research Council, the Australian Agency
for International Development, the Ford Foundation, IZA, and the Chinese
Foundation of Social Sciences. More information on the survey can be found at
https://datasets.iza.org/dataset/58/longitudinal-survey-on-rural-urban-
migration-in-china. The survey studied immigrants or workers moving from the
rural areas of China to its big cities. The survey asked interviewees
(immigrants or workers) a wide range of questions. For the detailed design of
the survey and other information, including on the construction of each
variable, see the survey website and Gong et al. (2008). Currently, the 2008
wave data are publicly available.

Economic reforms since the late 1970s have brought significant changes to
China’s economy. The government began relaxing its policy on population
mobility in the early 1980s. Gradually, peasants were allowed to leave
villages and work in big cities to earn higher incomes. Most migrant workers
may leave their spouses, children, or parents behind in their hometowns who
may need their financial support. This situation results in monetary
transfers, that is, remittances, from migrant workers to their family. In the
context of migration, family, and economic development, remittances are not
only an income source for recipients, but also reflect intrafamilial
relationships. Remittances clearly represent a dimension of family ties and
demonstrate high degrees of interaction between migrants and families at home.
In addition, remittances from the rural migrant workers also contribute
significantly to China’s agricultural productivity (c.f., Rozelle, Taylor and
deBrauw (1999)). For these reasons, it has long been of interest to model
remittances to families or relatives in the hometown; see Li (2001) and Cai
(2003), among others.

In this application, we take remittance as the dependent variable ($Y$); we
focus on the dataset from Guangdong Province (Guangzhou, Dongguan, and
Shenzhen cities) in the 2008 survey wave, and keep 78 covariates from the
dataset.888We keep covariates with relatively fewer missing observations.
After dropping observations with missing information, the number of
observations is 456. We provide the definitions of the dependent variable and
covariates, and the associated summary statistics, in Tables
LABEL:table:app_def and LABEL:table:summary, respectively. We report the
original labels of all covariates in the survey in the first column of Table
LABEL:table:app_def for reference. Among the 78 covariates, there are some
continuous variables with most observations as 0 (Panel C in Table
LABEL:table:app_def), and some discrete variables with very limited support
(Panel D in Table LABEL:table:app_def). For those variables, the design matrix
of the sieves generated are either singular or close to singular. For this
reason, we add those variables linearly into the model and treat them the same
as dummy variables (Panel E in Table LABEL:table:app_def) for modeling.
Consequently, the way we fit the dataset resembles the approach we used for
DGPs 9 and 10 in the simulation. We explain the reason our theoretical results
continue to hold in this situation in the simulation section (DGP 9). We take
natural logarithms for the $Y$ and continuous $X$ variables to offset the
effect of outliers; otherwise, the forecast can easily take some extreme
values.

We randomly select 400 observations as the training sample, and the remaining
56 observations as the test sample. The number of sieve terms is set as
$m_{n}=\left\lfloor 400^{1/4}\right\rfloor+1=5$ for the continuous variables
in Panel B of Table LABEL:table:app_def). We set
$\varsigma_{n}=Cm_{n}\left(\left[\log p_{n}\right]^{1.1}+\left[\log
m_{n}\right]^{1.1}\right)$ and $\varsigma_{n}^{\ast}=3\varsigma_{n}$ for
continuous variables, and $\varsigma_{n}=C\left[\log p_{n}\right]^{1.1}$ and
$\varsigma_{n}^{\ast}=4\varsigma_{n}$ for terms entering the model linearly
(see Section 4.2 for the reason). We set $C$ in the range of $0.5$ to $2.5$,
specifically, $0.5,0.6,...,2.5$, and choose $C$ to minimize the BIC for the
model selection. The competing methods are the group Lasso (labelled as
GLASSO) and the adaptive group Lasso (AGLASSO). The tuning parameters for
GLASSO and AGLASSO are selected as in Section 4.3. We evaluate the performance
of all methods based on the FRMSE of the test dataset using the fitted models
from different methods. We independently repeat the above procedure 100 times.

Table 5: Performance in Terms of FRMSE (100 Cases), benchmark: AGLASSO OCMT |  | POST-OCMT |  | GLASSO
---|---|---|---|---
Better | Same | Worse |  | Better | Same | Worse |  | Better | Same | Worse
77 | 17 | 6 |  | 76 | 21 | 3 |  | 5 | 88 | 7
Table 6: Average FRMSE ratio, benchmark: AGLASSO One Stage | OCMT | POST-OCMT | GLASSO | AGLASSO
---|---|---|---|---
0.873 | 0.817 | 0.814 | 1.005 | 1

We report the results in terms of the out-of-sample FRMSE in Tables 5 and 6,
with AGLASSO as the benchmark. Table 5 shows that OCMT and POST-OCMT
outperform AGLASSO in the majority of cases. Of course, our methods do not
outperform AGLASSO all the time. To highlight the necessity of the multiple
stages, we also report the results of the one-stage procedure (we selected the
tuning parameter also by minimizing the BIC) in Table 6. The OCMT stops at the
second stage for all 100 cases. We normalize the average FRMSE of AGLASSO as
1. Notably, the average FRMSE of our methods are lower than that of AGLASSO.
The OCMT also improves the FRMSE over the one-stage procedure, owing to some
hidden signals uncovered by our multiple-stage procedure. We report the
frequencies of variables (out of 100 cases) selected by all methods in Table
S11. It appears that G102 (monthly income) along with G133 (gifts to others,
including parents) and G137 (education cost for left-behind children)
contribute most to the model, as shown by all methods in general. We note that
AGLASSO tends to select more variables than our methods, on average, which was
also CKP’s finding in their application. The “over-fitting” is the main cause
of the relative inferior performance of AGLASSO.

##  6 Conclusion

In this paper, we examine the one-covariate-at-a-time multiple testing
approach to model selection in additive models. The properties of the TPR,
FPR, and FDR of our approach are established based on some non-asymptotic
bounds of Type-I and II errors. The simulation experiments and one application
on the RUMiC dataset showcase excellent small-sample properties of our
methods. Just as stated by CKP for linear models, we view our approach as a
useful alternative to the model selection methods for additive models in the
literature.

## References

  * (1)
  * Akaike (1973) Akaike, H. (1973). “Information Theory and an Extension of Maximum Likelihood Principle,” in _Proceedings of Second International Symposium on Information Theory,_ B. N. Petrov and F. Csaki (eds.), pp. 267-281.
  * (3)
  * Akaike (1974) Akaike, H. (1974). “A New Look at the Statistical Model Identification,”  _IEEE Transactions on Automatic Control,_ 19(6), 716-723.
  * (5)
  * Belloni et al. (2012) Belloni, A., Chen, D., Chernozhukov, V., and Hansen, C. (2012): “Sparse Models and Methods for Optimal Instruments with an Application to Eminent Domain,”  _Econometrica_ , 80, 6, 2369–2429.
  * (7)
  * Belloni and Chernozhukov (2013) Belloni, A., and Chernozhukov, V. (2013): “Least Squares after Model Selection in High-dimensional Sparse Models,”  _Bernoulli_ , 19, 2, 521-547.
  * (9)
  * Belloni, Chernozhukov and Hansen (2014) Belloni, A., Chernozhukov, V., and Hansen, C. (2014): “Inference on Treatment Effects after Selection among High-Dimensional Controls,”  _The Review of Economic Studies_ , 81, 2, 608–650.
  * (11)
  * Belloni et al. (2017) Belloni, A., Chernozhukov, V., Fernandez-val, I., and Hansen, C. (2017): “Program Evaluation and Causal Inference with High-Dimensional Data,”  _Econometrica_ , 85, 1, 233–298.
  * (13)
  * Bickel, Ritov and Tsybakov (2009) Bickel P. J., Ritov, Y., and Tsybakov, A. B. (2009): “Simultaneous Analysis of Lasso and Dantzig Selector,”  _The Annals of Statistics_ , 37, 4, 1705-1732.
  * (15)
  * Buhlmann and van de Geer (2011) Buhlmann, p., and van de geer, S. (2011): Statistics for High Dimensional Data: Methods, Theory and Applications. Springer, Heidelberg.
  * (17)
  * Cai (2003) Cai, Q. (2003): “Migrant Remittances and Family Ties: a Case Study in China,”  _International Journal of Population Geography_ , 9(6), 471-483.
  * (19)
  * Chen (2007) Chen, X. (2007): “Large Sample Sieve Estimation of Semi-nonparametric Models,”  _Handbook of Econometrics_ , Chapter 76, Vol 6B, North-Holland.
  * (21)
  * Chernozhukov et al. (2017) Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2017): “Double/debiased machine learning for treatment and structural parameters,”  _The Econometric Journal,_ 21, 1, C1-C68.
  * (23)
  * Chudik, Kapetanios and Pesaran (2018) Chudik, A., Kapetanios, G., and Pesaran, M. H. (2018): “A One Covariate at a Time, Multiple Testing Approach to Variable Selection in High-Dimensional Linear Regression Models,”  _Econometrica_ , 86(4), 1479–1512.
  * (25)
  * de Boor (2001) de Boor, C. (2001): A Practical Guide to Splines, Springer-Verlag, New York.
  * (27)
  * Fan and Li (2001) Fan, J. and Li, R. (2001): “Variable Selection via Nonconcave Penalized Likelihood and Its Oracle Properties,”  _Journal of the American Statistical Association_ , 96, 1348–1360.
  * (29)
  * Fan et al. (2020) Fan, J., Li, R., Zhang, C. H., and Zou, H. (2020): Statistical Foundations of Data Science, Chapman and Hall/CRC.
  * (31)
  * Fan and Lv (2008) Fan, J., and Lv, J. (2008), “Sure Independence Screening for Ultrahigh Dimensional Feature Space,”  _Journal of the Royal Statistical Society, Series B_ , 70, 849–911.
  * (33)
  * Fan and Lv (2013) Fan, J., and Lv, J. (2013): “Asymptotic Equivalence of Regularization Methods in Thresholded Parameter Space,”  _Journal of the American Statistical Association_ , 108, 1044–1061.
  * (35)
  * Fan, Feng and Song (2011) Fan, J., Feng, Y., Song, R. (2011): “Nonparametric Independence Screening in Sparse Ultra-High-Dimensional Additive Models,”  _Journal of the American Statistical Association_ , 106(494), 544-557.
  * (37)
  * Fan, Samworth and Wu (2009) Fan J., Samworth, R., and Wu, Y. (2009): “Ultrahigh Dimensional Feature Selection: Beyond the Linear Model,”  _Journal of Machine Learning Research_ , 10, 2013–2038.
  * (39)
  * Fan and Song (2010) Fan, J., and R. Song (2010): “Sure Independence Screening in Generalized Linear Models With NP-Dimensionality,”  _The Annals of Statistics,_ 38, 3567–3604.
  * (41)
  * Fan and Tang (2013) Fan, J., and Tang, C.(2013): “Tuning Parameter Selection in High Dimensional Penalized Likelihood,”  _Journal of the Royal Statistical Society, Series B_ , 75, 531–552.
  * (43)
  * Gong et al. (2008) Gong, X., Kong, S. T., Li, S., and Meng X. (2008): Rural-urban migrants: a driving force for growth, in L. Song and W. T. Woo (eds) China’s Dilemma, Canberra: Asia Pacific Press.
  * (45)
  * Horowitz and Mammen (2004) Horowitz, J. L., and Mammen, E. (2004). “Nonparametric Estimation of an Additive Model with a Link Function,”  _The Annals of Statistics_ , 32(6), 2412-2443.
  * (47)
  * Huang, Horowitz and Wei (2010) Huang, J., Horowitz, J. L., and Wei, F. (2010): “Variable Selection in Nonparametric Additive Models,”  _The Annals of Statistics_ , 38(4), 2282–2313.
  * (49)
  * Kozbur (2020) Kozbur, d. (2020): “Analysis of Testing-Based Forward Model Selection,”  _Econometrica_ , 38(4), 2282–2313.
  * (51)
  * Li (2000) Li, Q. (2000): “Efficient Estimation of Additive Partially Linear Models,”  _International Economic Review_ , Vol 41, No 4, 1073-1092.
  * (53)
  * Li (2001) Li, Q. (2001): “Analysis of the Remittance of the Out-Migrant Workers in China,”  _Sociological Research_ , 4, 64-76.
  * (55)
  * Newey (1997) Newey, W. K., (1997): “Convergence Rates and Asymptotic Normality for Series Estimators,”  _Journal of Econometrics_ , 79(1), 147-168.
  * (57)
  * Rozelle, Taylor and deBrauw (1999) Rozelle, S., Taylor, J. E, and DeBrauw, A. (1999): “Migration, Remittances, and Agricultural Productivity in China,”  _The American Economic Review_ , 89(2), 287-291.
  * (59)
  * Schwartz (1978) Schwarz, G. (1978) “Estimating the Dimension of a Model,”  _The Annals of Statistics_ , 6, 471.
  * (61)
  * Stone (1985) Stone, C. J. (1985): “Additive Regression and Other Nonparametric Models,”  _The Annals of Statistics_ , Vol 13, No 2, 689-705.
  * (63)
  * Tibshirani (1996) Tibshirani, R. (1996): “Regression Shrinkage and Selection via the Lasso,”  _Journal of the Royal Statistical Society, Series B_ , 58, 267–288
  * (65)
  * van der Vaart and Wellner (1996) van der Vaart, A. w., and Wellner, J. A. (1996): Weak Convergence and Empirical Processes, New York, Springer.
  * (67)
  * Wu and Lange (2008) Wu, T. T., and Lange, K. (1998): “Coordinate Descent Algorithms for Lasso Penalized Regression,”  _The Annals of Statistics_ , 2(1), 224-244.
  * (69)
  * Zhou and Hastie (2005) Zhou, H., and Hastie, T. (2005): “Regularization and Variable Selection via the Elastic Net,”  _Journal of the Royal Statistical Society, Series B_ , 67, 301–320.
  * (71)
  * Zou (2006) Zou, H. (2006), “The Adaptive Lasso and Its Oracle Properties,”  _Journal of the American Statistical Association,_ 101, 1418–1429.
  * (73)
  * Zou and Li (2008) Zou, H., and Li, R. (2008), “One-Step Sparse Estimates in Nonconcave Penalized Likelihood Models,”  _The Annals of Statistics_ , 36, 1509–1533.
  * (75)
  * Zhang (2010) Zhang, C. H. (2010), “Nearly Unbiased Variable Selection Under Minimax Concave Penalty,”  _The Annals of Statistics_ , 38, 894–942.
  * (77)
  * Zhou, Shen and Wolfe (1998) Zhou, S., Shen, X., and Wolfe, D. A., (1998): “Local Asymptotics for Regression Splines and Confidence Regions,”  _The Annals of Statistics_ , 26(5), 1760-1782.
  * (79)

Appendix

In this appendix we first provide some technical details for the one-stage and
multiple-stage procedures in Appendices A.1 and A.2, respectively. Then we
provide some technical lemmas that are used in the proofs of the main results
in Appendix A.3. Finally, we prove the main results in the paper in Appendix
B. The proofs of the technical lemmas and some additional simulation and
application results are relegated to the online supplement.

##  Appendix A Some Technical Details and Technical Lemmas

###  A.1 Some Technical Details for the One-Stage Procedure

The proof of Proposition 2.1 is tedious. We illustrate the main idea of the
proof by providing some brief technical details below.

Define

$f_{nl}\left(x\right)=P^{m_{n}}\left(x\right)^{\prime}\boldsymbol{\beta}_{l},\text{
and }U_{l}=Y-f_{nl}\left(X_{l}\right).$ (A.1)

Then $E\left[P^{m_{n}}\left(X_{l}\right)U_{l}\right]=0$ by the definition of
$f_{nl}\left(x\right).$ By Stone (1985), it holds that

$\left\\{E\left[\left(f_{nl}\left(X_{l}\right)-f_{l}\left(X_{l}\right)\right)^{2}\right]\right\\}^{1/2}\leq
C_{1}m_{n}^{-d}\text{ for some }C_{1}<\infty.$ (A.2)

If the bias is asymptotically negligible, by the properties listed in Lemma
A.3 below,

$\left\|\boldsymbol{\beta}_{l}\right\|\propto
m_{n}^{1/2}\left\\{E\left[f_{nl}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}\propto
m_{n}^{1/2}\left\\{E\left[f_{l}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}=m_{n}^{1/2}\theta_{l}.$

As a result $m_{n}^{-1/2}\left\|\boldsymbol{\beta}_{l}\right\|$ can be
equivalently used to measure the strength of net impact of $X_{l}$ on $Y$.

Let $u_{li}=y_{i}-f_{nl}\left(x_{li}\right)\ $and$\
\boldsymbol{u}_{l}=\left(u_{l1},u_{l2},\ldots,u_{ln}\right)^{\prime}$. Then
the estimator for $\boldsymbol{\beta}_{l}$ can be rewritten as

$\boldsymbol{\hat{\beta}}_{l}=\left(\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}\boldsymbol{y}=\boldsymbol{\beta}_{l}+\left(\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l},$

where recall that
$\mathbb{X}_{l}=(\mathbb{X}_{l1},\mathbb{X}_{l2},\ldots,\mathbb{X}_{ln})^{\prime}$
is the $n\times m_{n}$ “design” matrix for $X_{l}.$ With it,
$\mathcal{\hat{X}}_{l}$ can be rewritten as

$\mathcal{\hat{X}}_{l}=\boldsymbol{\hat{\beta}}_{l}^{\prime}\left(\hat{\sigma}_{l}^{-2}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)\boldsymbol{\hat{\beta}}_{l}=\boldsymbol{\beta}_{l}^{\prime}\left(\hat{\sigma}_{l}^{-2}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)\boldsymbol{\beta}_{l}+2\hat{\sigma}_{l}^{-2}\boldsymbol{\beta}_{l}^{\prime}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}+\boldsymbol{u}_{l}^{\prime}\mathbb{X}_{l}\left(\hat{\sigma}_{l}^{2}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}.$
(A.3)

Our analysis is based on the decomposition in (A.3). The idea is to show that
$\left\|n^{-1/2}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}\right\|,$
$\hat{\sigma}_{l}^{2},$ and
$\left\|n^{-1}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right\|$ are bounded by
some rate or value with very high probability. Based on these results, the
value of $\widehat{\mathcal{J}}_{l}$ is solely determined by the strength of
$\left\|\boldsymbol{\beta}_{l}\right\|$ with very high probability.

###  A.2 Some Technical Details for the Multiple-Stage Procedure

The proof of Proposition 3.1 is rather tedious. However, the main idea of the
proof is straightforward, and similar to the one for Proposition 2.1. To see
it, we provide some technical discussion below.

To simplify notations, let
$P^{m_{n}}\left(\boldsymbol{X}_{1}^{p^{\ast}}\right)\equiv\left(P^{m_{n}}\left(X_{1}\right)^{\prime},\ldots,P^{m_{n}}\left(X_{p^{\ast}}\right)^{\prime}\right)^{\prime},$
a $p^{\ast}m_{n}\times 1$ vector. Define the $n\times p^{\ast}m_{n}$ matrix

$\mathbb{X}_{1}^{p^{\ast}}\equiv\left(\mathbb{X}_{1},\ldots,\mathbb{X}_{p^{\ast}}\right),$

which is the design matrix for all the $p^{\ast}$ signal variables. To see the
approximation of the model more clearly, let

$\boldsymbol{\tilde{\beta}}_{1}^{p^{\ast}}=\left(E\left[P^{m_{n}}\left(\boldsymbol{X}_{1}^{p^{\ast}}\right)P^{m_{n}}\left(\boldsymbol{X}_{1}^{p^{\ast}}\right)^{\prime}\right]\right)^{-1}E\left[P^{m_{n}}\left(\boldsymbol{X}_{1}^{p^{\ast}}\right)Y\right],$

which is the population coefficient for the regression of $Y$ on
$P^{m_{n}}\left(\boldsymbol{X}_{1}^{p^{\ast}}\right)$. Write
$\boldsymbol{\tilde{\beta}}_{1}^{p^{\ast}}=\left(\boldsymbol{\tilde{\beta}}_{1}^{\prime},\ldots,\boldsymbol{\tilde{\beta}}_{p^{\ast}}^{\prime}\right)^{\prime},$
where $\boldsymbol{\tilde{\beta}}_{j}$ corresponds to the coefficient of
$P^{m_{n}}\left(X_{j}\right)$ in the regression. Let
$f_{nj}^{\ast}\left(x\right)=P^{m_{n}}\left(x\right)^{\prime}\boldsymbol{\tilde{\beta}}_{j}.$
Under Assumption 5’, Stone (1985) implies

$\left\\{E\left(\left[P^{m_{n}}\left(X_{j}\right)^{\prime}\boldsymbol{\tilde{\beta}}_{j}-f_{j}^{\ast}\left(X_{j}\right)\right]\right)^{2}\right\\}^{1/2}=O\left(m_{n}^{-d}\right).$
(A.4)

We rewrite $Y$ as

$\displaystyle Y$
$\displaystyle=\sum_{j=1}^{p^{\ast}}P^{m_{n}}\left(X_{j}\right)^{\prime}\boldsymbol{\tilde{\beta}}_{j}+\sum_{j=1}^{p^{\ast}}\left[f_{j}^{\ast}\left(X_{j}\right)-P^{m_{n}}\left(X_{j}\right)^{\prime}\boldsymbol{\tilde{\beta}}_{j}\right]+\varepsilon$
$\displaystyle\equiv\sum_{j=1}^{p^{\ast}}P^{m_{n}}\left(X_{j}\right)^{\prime}\boldsymbol{\tilde{\beta}}_{j}+R_{n}+\varepsilon.$
(A.5)

Equation (A.4) implies $R_{n}=O_{P}\left(m_{n}^{-d}\right)$ for a fixed
$p^{\ast}.$ Then the model looks like a linear model with an approximation
bias term $R_{n}$ and an error term $\varepsilon$.

Now we have a closer look at $\theta_{l,\boldsymbol{Z}},$ which measures the
impact of $X_{l}$ on $Y$ after controlling $\boldsymbol{Z}$. If we can ignore
the approximation bias, a finite sample approximation to the term inside
$\theta_{l,\boldsymbol{Z}},$ namely,

$E\left[\left.Y-P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}\left(E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}\right]\right)^{-1}E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)Y\right]\right|X_{l}\right],$

is

$\displaystyle P^{m_{n}}\left(X_{l}\right)^{\prime}\Phi_{X_{l}}^{-1}\cdot
E\left[P^{m_{n}}\left(X_{l}\right)\left(Y-P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}\Phi_{\boldsymbol{Z}}^{-1}E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)Y\right]\right)\right]$
$\displaystyle\propto m_{n}\cdot
P^{m_{n}}\left(X_{l}\right)^{\prime}E\left[P^{m_{n}}\left(X_{l}\right)\left(Y-P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}\Phi_{\boldsymbol{Z}}^{-1}E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)Y\right]\right)\right],$
(A.6)

where $\Phi_{X_{l}}\equiv
E[P^{m_{n}}\left(X_{l}\right)P^{m_{n}}\left(X_{l}\right)^{\prime}],$
$\Phi_{\boldsymbol{Z}}\equiv
E[P^{m_{n}}\left(\boldsymbol{Z}\right)P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}],$
and the last line holds by Lemma A.3. We let

$\boldsymbol{\eta}_{l,\boldsymbol{Z}}\equiv
E\left[P^{m_{n}}\left(X_{l}\right)\left(Y-P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}\Phi_{\boldsymbol{Z}}^{-1}E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)Y\right]\right)\right],$

which appears as the numerator in the population coefficient of the regression
of $Y-P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}\Phi_{\boldsymbol{Z}}^{-1}$
$\cdot E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)Y\right]$ on
$P^{m_{n}}\left(X_{l}\right)$. Then the last term in equation (A.6) becomes
$m_{n}P^{m_{n}}\left(X_{l}\right)^{\prime}\boldsymbol{\eta}_{l,\boldsymbol{Z}}$.
As a result,

$\theta_{l,\boldsymbol{Z}}\propto\left\\{E\left[\left(m_{n}P^{m_{n}}\left(X_{l}\right)^{\prime}\boldsymbol{\eta}_{l,\boldsymbol{Z}}\right)^{2}\right]\right\\}^{1/2}=\left\\{m_{n}^{2}\boldsymbol{\eta}_{l,\boldsymbol{Z}}^{\prime}\Phi_{X_{l}}\boldsymbol{\eta}_{l,\boldsymbol{Z}}\right\\}^{1/2}\propto
m_{n}^{1/2}\left\|\boldsymbol{\eta}_{l,\boldsymbol{Z}}\right\|$ (A.7)

by Lemma A.3. Note $\theta_{l,\boldsymbol{Z}}$ measures the impact of $X_{l}$
on $Y$ after controlling $\boldsymbol{Z}.$
$m_{n}^{1/2}\left\|\boldsymbol{\eta}_{l,\boldsymbol{Z}}\right\|$ can be
equivalently used to measure this strength.

To facilitate the analysis of $\boldsymbol{\eta}_{l,\boldsymbol{Z}},$ let
$\Phi_{X_{l}\boldsymbol{Z}}\equiv
E[P^{m_{n}}\left(X_{l}\right)P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}].$
By the orthogonality between the error term and regressor in the least squares
projection, we have

$\displaystyle\boldsymbol{\eta}_{l,\boldsymbol{Z}}$
$\displaystyle=E\left[\left\\{P^{m_{n}}\left(X_{l}\right)-\Phi_{X_{l}\boldsymbol{Z}}\Phi_{\boldsymbol{Z}}^{-1}P^{m_{n}}\left(\boldsymbol{Z}\right)\right\\}\left\\{Y-P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}\Phi_{\boldsymbol{Z}}^{-1}E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)Y\right]\right\\}\right]$
$\displaystyle=E\left[\left(P^{m_{n}}\left(X_{l}\right)-\boldsymbol{\gamma}_{X_{l},\boldsymbol{Z}}^{\prime}P^{m_{n}}\left(\boldsymbol{Z}\right)\right)\cdot\left(Y-\boldsymbol{\gamma}_{Y,\boldsymbol{Z}}^{\prime}P^{m_{n}}\left(\boldsymbol{Z}\right)\right)\right]$
$\displaystyle=E\left[\boldsymbol{U}_{X_{l},\boldsymbol{Z}}\cdot
U_{Y,\boldsymbol{Z}}\right],$ (A.8)

where
$\boldsymbol{\gamma}_{X_{l},\boldsymbol{Z}}\equiv\Phi_{\boldsymbol{Z}}^{-1}\Phi_{X_{l}\boldsymbol{Z}}^{\prime},$
$\boldsymbol{\gamma}_{Y,\boldsymbol{Z}}\equiv\Phi_{\boldsymbol{Z}}^{-1}E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)Y\right],$
and $\boldsymbol{U}_{X_{l},\boldsymbol{Z}}$ and $U_{Y,\boldsymbol{Z}}$ denote
the respective projection errors:

$\boldsymbol{U}_{X_{l},\boldsymbol{Z}}\equiv
P^{m_{n}}\left(X_{l}\right)-\boldsymbol{\gamma}_{X_{l},\boldsymbol{Z}}^{\prime}P^{m_{n}}\left(\boldsymbol{Z}\right),\text{
and }U_{Y,\boldsymbol{Z}}\equiv
Y-\boldsymbol{\gamma}_{Y,\boldsymbol{Z}}^{\prime}P^{m_{n}}\left(\boldsymbol{Z}\right).$
(A.9)

Note that $\boldsymbol{U}_{X_{l},\boldsymbol{Z}}$ is $m_{n}\times 1$ and
$U_{Y,\boldsymbol{Z}}$ is a scalar. Thus,
$\boldsymbol{\eta}_{l,\boldsymbol{Z}}$ measures the covariance between
$P^{m_{n}}\left(X_{l}\right)$ and $Y$ after controlling
$P^{m_{n}}\left(\boldsymbol{Z}\right)$.

At the sample level, we use the following notations. Let $\boldsymbol{z}_{i}$
denote the $i$-th observation for $\boldsymbol{Z}$. We let
$\boldsymbol{u}_{X_{l},\boldsymbol{Z}}\equiv\left(\boldsymbol{u}_{X_{l},\boldsymbol{Z},1},\ldots,\boldsymbol{u}_{X_{l},\boldsymbol{Z},n}\right)^{\prime}$
and
$\boldsymbol{u}_{Y,\boldsymbol{Z}}\equiv\left(u_{Y,\boldsymbol{Z,}1},\ldots,u_{Y,\boldsymbol{Z,}n}\right)^{\prime},\
$where
$\boldsymbol{u}_{X_{l},\boldsymbol{Z},i}=P^{m_{n}}\left(x_{li}\right)-\boldsymbol{\gamma}_{X_{l},\boldsymbol{Z}}^{\prime}P^{m_{n}}\left(\boldsymbol{z}_{i}\right)$
and
$u_{Y,\boldsymbol{Z,}i}=y_{i}-\boldsymbol{\gamma}_{Y,\boldsymbol{Z}}^{\prime}P^{m_{n}}\left(\boldsymbol{z}_{i}\right).$
The connection between $\mathcal{\hat{X}}_{l,\boldsymbol{Z}}$ and
$\boldsymbol{\eta}_{l,\boldsymbol{Z}}$ can be seen from the following. Noting
that

$\boldsymbol{y}=\mathbb{Z}\boldsymbol{\gamma}_{Y,\boldsymbol{Z}}+\boldsymbol{u}_{Y,\boldsymbol{Z}}\text{
and
}\mathbb{X}_{l}=\mathbb{Z}\boldsymbol{\gamma}_{X_{l},\boldsymbol{Z}}+\boldsymbol{u}_{X_{l},\boldsymbol{Z}},$

we have

$\displaystyle\mathcal{\hat{X}}_{l,\boldsymbol{Z}}$
$\displaystyle=\left[\boldsymbol{u}_{Y,\boldsymbol{Z}}^{\prime}M_{\mathbb{Z}}\boldsymbol{u}_{X_{l},\boldsymbol{Z}}\right]\left(\hat{\sigma}_{l,\boldsymbol{Z}}^{2}\mathbb{X}_{l}^{\prime}M_{\mathbb{Z}}\mathbb{X}_{l}\right)^{-1}\left[\boldsymbol{u}_{X_{l},\boldsymbol{Z}}^{\prime}M_{\mathbb{Z}}\boldsymbol{u}_{Y,\boldsymbol{Z}}\right]$
$\displaystyle=\left[(\boldsymbol{u}_{X_{l},\boldsymbol{Z}}^{\prime}\boldsymbol{u}_{Y,\boldsymbol{Z}}-n\boldsymbol{\eta}_{l,\boldsymbol{Z}}-\boldsymbol{u}_{X_{l},\boldsymbol{Z}}^{\prime}Q_{\mathbb{Z}}\boldsymbol{u}_{Y,\boldsymbol{Z}})+n\boldsymbol{\eta}_{l,\boldsymbol{Z}}\right]^{\prime}\left(\hat{\sigma}_{l,\boldsymbol{Z}}^{2}\mathbb{X}_{l}^{\prime}M_{\mathbb{Z}}\mathbb{X}_{l}\right)^{-1}$
$\displaystyle\cdot\left[(\boldsymbol{u}_{X_{l},\boldsymbol{Z}}^{\prime}\boldsymbol{u}_{Y,\boldsymbol{Z}}-n\boldsymbol{\eta}_{l,\boldsymbol{Z}}-\boldsymbol{u}_{X_{l},\boldsymbol{Z}}^{\prime}Q_{\mathbb{Z}}\boldsymbol{u}_{Y,\boldsymbol{Z}})+n\boldsymbol{\eta}_{l,\boldsymbol{Z}}\right]$
$\displaystyle=n^{2}\boldsymbol{\eta}_{l,\boldsymbol{Z}}^{\prime}\left(\hat{\sigma}_{l,\boldsymbol{Z}}^{2}\mathbb{X}_{l}^{\prime}M_{\mathbb{Z}}\mathbb{X}_{l}\right)^{-1}\boldsymbol{\eta}_{l,\boldsymbol{Z}}+2n\boldsymbol{\eta}_{l,\boldsymbol{Z}}^{\prime}\left(\hat{\sigma}_{l,\boldsymbol{Z}}^{2}\mathbb{X}_{l}^{\prime}M_{\mathbb{Z}}\mathbb{X}_{l}\right)^{-1}\boldsymbol{\tilde{u}}_{l,\boldsymbol{Z}}+\boldsymbol{\tilde{u}}_{l,\boldsymbol{Z}}^{\prime}\left(\hat{\sigma}_{l,\boldsymbol{Z}}^{2}\mathbb{X}_{l}^{\prime}M_{\mathbb{Z}}\mathbb{X}_{l}\right)^{-1}\boldsymbol{\tilde{u}}_{l,\boldsymbol{Z}}\boldsymbol{,}$
(A.10)

where
$Q_{\mathbb{Z}}=\mathbb{Z}\left(\mathbb{Z}^{\prime}\mathbb{Z}\right)^{-1}\mathbb{Z}^{\prime}$
and

$\boldsymbol{\tilde{u}}_{l,\boldsymbol{Z}}\equiv\boldsymbol{u}_{X_{l},\boldsymbol{Z}}^{\prime}\boldsymbol{u}_{Y,\boldsymbol{Z}}-n\boldsymbol{\eta}_{l,\boldsymbol{Z}}-\boldsymbol{u}_{X_{l},\boldsymbol{Z}}^{\prime}Q_{\mathbb{Z}}\boldsymbol{u}_{Y,\boldsymbol{Z}}.$
(A.11)

behaves like the counterpart of $\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}$ in
equation (A.3). To see the last point, first note that

$E\left[\boldsymbol{u}_{X_{l},\boldsymbol{Z}}^{\prime}\boldsymbol{u}_{Y,\boldsymbol{Z}}-n\boldsymbol{\eta}_{l,\boldsymbol{Z}}\right]=0.$

Next, it is easy to show that
$\boldsymbol{u}_{X_{l},\boldsymbol{Z}}^{\prime}Q_{\mathbb{Z}}\boldsymbol{u}_{Y,\boldsymbol{Z}}=n^{-1/2}\boldsymbol{u}_{X_{l},\boldsymbol{Z}}^{\prime}\mathbb{Z}\left(n^{-1}\mathbb{Z}^{\prime}\mathbb{Z}\right)^{-1}n^{-1/2}\mathbb{Z}^{\prime}\boldsymbol{u}_{Y,\boldsymbol{Z}}$
is of smaller order than
$\boldsymbol{u}_{X_{l},\boldsymbol{Z}}^{\prime}\boldsymbol{u}_{Y,\boldsymbol{Z}}-n\boldsymbol{\eta}_{l,\boldsymbol{Z}}$
because
$E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)\boldsymbol{U}_{X_{l},\boldsymbol{Z}}^{\prime}\right]=\boldsymbol{0}$
and
$E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)U_{Y,\boldsymbol{Z}}\right]=\boldsymbol{0}$
by the definitions in equation (A.9). The first term in (A.11) is the dominant
term of $\mathcal{\hat{X}}_{l,\boldsymbol{Z}}$ if
$\boldsymbol{\eta}_{l,\boldsymbol{Z}}\ $is large enough.

The analysis of $\mathcal{\hat{X}}_{l}\left(\boldsymbol{Z}\right)$ is based on
the decomposition in equation (A.10). The basic idea is similar to the one for
analyzing $\mathcal{\hat{X}}_{l}$ in equation (A.3): we show
$\hat{\sigma}_{l,\boldsymbol{Z}}^{2},$
$n^{-1}\mathbb{X}_{l}^{\prime}M_{\mathbb{Z}}\mathbb{X}_{l},$ and$\
n^{-1/2}\boldsymbol{\tilde{u}}_{l,\boldsymbol{Z}}$ are bounded by some value
or rate with very high probability, and as a result,
$\mathbf{1}\left(\mathcal{\hat{X}}_{l}\left(\boldsymbol{Z}\right)>\varsigma_{n}\right)$
is determined by the strength of $\theta_{l,\boldsymbol{Z}}$ (or equivalently,
$m_{n}^{1/2}\left\|\boldsymbol{\eta}_{l,\boldsymbol{Z}}\right\|$) with very
high probability.

###  A.3 Technical Lemmas

We frequently use the inequalities presented in Remark 1, Lemma A.1, and Lemma
A.2.

######  Remark 1 (Some inequalities)

_We may apply the following inequalities in the proofs directly without
referring them back to here. First_

$\Pr\left(X_{1}+X_{2}\geq C\right)\leq\Pr\left(X_{1}\geq\pi
C\right)+\Pr\left(X_{2}\geq\left(1-\pi\right)C\right)$

_for any constant_ $\pi.$_That is due to the fact that_
$\left\\{X_{1}+X_{2}\geq C\right\\}\subseteq\left\\{X_{1}\geq\pi
C\right\\}\cup\left\\{X_{1}\geq\left(1-\pi\right)C\right\\}.$_Similarly, we
have_

$\Pr\left(\sum_{i=1}^{n}X_{i}\geq C\right)\leq\sum_{i=1}^{n}\Pr\left(X_{i}\geq
Cn^{-1}\right)\ \text{\emph{and} }\Pr\left(\max_{1\leq i\leq n}X_{i}\geq
C\right)\leq\sum_{i=1}^{n}\Pr\left(X_{i}\geq C\right).$

_For any positive random variables_ $X_{1}$_and_ $X_{2}$_and positive
constants_ $C_{1}$_and_ $C_{2},$

$\Pr\left(X_{1}\cdot X_{2}\geq C_{1}\right)\leq\Pr\left(X_{1}\geq
C_{1}/C_{2}\right)+\Pr\left(X_{2}\geq C_{2}\right),$

_due to the fact that_ $\left\\{X_{1}\cdot X_{2}\geq
C_{1}\right\\}\subseteq\left\\{X_{1}\geq
C_{1}/C_{2}\right\\}\cup\left\\{X_{2}\geq C_{2}\right\\}.$__

######  Lemma A.1 (Bernstein’s Inequality)

For an i.i.d. series $\left\\{z_{i}\right\\}_{i=1}^{\infty}$ with zero mean
and bounded support $\left[-M,M\right]$, we have

$\Pr\left(\left|\sum_{i=1}^{n}z_{i}\right|>x\right)\leq
2\exp\left\\{-x^{2}/\left(2\left(v+Mx/3\right)\right)\right\\},$

for any $v\geq$_Var_ $\left(\sum_{i=1}^{n}z_{i}\right).$

The above lemma is Lemma 2.2.9 in van der Vaart and Wellner (1996).

###### Lemma A.2

Suppose that for an i.i.d. series $\left\\{z_{i}\right\\}_{i=1}^{\infty}$,
$\Pr\left(\left|z_{i}\right|>x\right)\leq
C_{1}\exp\left(-C_{2}x^{\alpha}\right)$ holds for all $x>0$ for some
$C_{1},C_{2},C_{3}>0.$ Further $E\left(z_{i}\right)=0$ and
$E\left(z_{i}^{2}\right)=\sigma_{z}^{2}$. Let
$\left\\{v_{n}\right\\}_{n=1}^{\infty}$ be a deterministic series such that
$v_{n}\propto n^{\lambda}.$ Then (i) if
$0<\lambda\leq\left(\alpha+1\right)/\left(\alpha+2\right)$, we have

$\Pr\left(\left|\sum_{i=1}^{n}z_{i}\right|>v_{n}\right)\leq\exp\left[\left.-\left(1-\pi\right)^{2}v_{n}^{2}\right/\left(2n\sigma_{z}^{2}\right)\right]\text{
for any }\pi\in\left(0,1\right);$

(ii) if $\lambda\geq\left(\alpha+1\right)/\left(\alpha+2\right),$ we have

$\Pr\left(\left|\sum_{i=1}^{n}z_{i}\right|>v_{n}\right)\leq\exp\left(-C_{4}v_{n}^{\alpha/\left(\alpha+1\right)}\right)\text{
for some }C_{4}>0.$

This lemma is Lemma A3 in CKP.

###### Lemma A.3

Recall that
$\Phi_{X_{l}}=E[P^{m_{n}}\left(X_{l}\right)P^{m_{n}}\left(X_{l}\right)^{\prime}].$
Suppose that Assumption 3 holds. Then there exist some positive constants
$B_{1},$ $B_{2},$ $B_{3},$ and $B_{4}$ such that

$\displaystyle\text{ }B_{1}m_{n}^{-1}$
$\displaystyle\leq\lambda_{\min}\left(\Phi_{X_{l}}\right)\leq\lambda_{\max}\left(\Phi_{X_{l}}\right)\leq
B_{2}m_{n}^{-1},$ (A.12) $\displaystyle B_{3}m_{n}^{-1}$ $\displaystyle\leq
E\left[\phi_{k}\left(X_{l}\right)^{2}\right]\leq B_{4}m_{n}^{-1},\text{ and }$
$\displaystyle E\left[\left|\phi_{k}\left(X_{l}\right)\right|\right]$
$\displaystyle\propto m_{n}^{-1},$

for all $k$ and $l$. For any
$f_{nl}\left(X_{l}\right)=P^{m_{n}}\left(X_{l}\right)^{\prime}\boldsymbol{\beta}$
where $X_{l}$ satisfies Assumption 3, we have

$\left\\{E\left[f_{nl}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}\propto
m_{n}^{-1/2}\left\|\boldsymbol{\beta}\right\|.$

For the first and second equations in the above lemma, see Lemma 6.2 in Zhou,
Shen and Wolfe (1998). The third equation is a direct result of those on pp.
91 and 133 in de Boor (2001). The last equation follows from (A.12) and the
fact

$\displaystyle\left\\{E\left[f_{nl}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}$
$\displaystyle=\left\\{\boldsymbol{\beta}^{\prime}E\left[P^{m_{n}}\left(X_{l}\right)P^{m_{n}}\left(X_{l}\right)^{\prime}\right]\boldsymbol{\beta}\right\\}^{1/2},\text{
and}$
$\displaystyle\left[\lambda_{\min}\left(\Phi_{X_{l}}\right)\right]^{1/2}\left\|\boldsymbol{\beta}\right\|$
$\displaystyle\leq\left\\{\boldsymbol{\beta}^{\prime}E\left[P^{m_{n}}\left(X_{l}\right)P^{m_{n}}\left(X_{l}\right)^{\prime}\right]\boldsymbol{\beta}\right\\}^{1/2}\leq\left[\lambda_{\max}\left(\Phi_{X_{l}}\right)\right]^{1/2}\left\|\boldsymbol{\beta}\right\|.$

###### Lemma A.4

Suppose that Assumptions 1 and 3 hold. Then

$\Pr\left(\left|\left\|\left(n^{-1}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\right\|-\left\|\Phi_{X_{l}}^{-1}\right\|\right|>\left.\left\|\Phi_{X_{l}}^{-1}\right\|\right/2\right)\leq
2m_{n}^{2}\exp\left\\{-C_{1}nm_{n}^{-3}\right\\}$ (A.13)

for some positive constant $C_{1}.$ If Assumption 7 also holds, then

$\Pr\left(\left|\left\|\left(n^{-1}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\right\|-\left\|\Phi_{X_{l}}^{-1}\right\|\right|>\left.\left\|\Phi_{X_{l}}^{-1}\right\|\right/2\right)\leq
C_{2}\exp\left\\{-C_{3}n^{C_{4}}\right\\}$

for some positive $C_{2},$ $C_{3},$ and $C_{4}.$

###### Lemma A.5

Suppose that Assumptions 1, 3, 4, 6, and 7 hold. Let
$\sigma_{l}^{2}=E\left(U_{l}^{2}\right),$
$\omega_{l}^{4}=E\left(U_{l}^{4}\right)-$
$\left[E\left(U_{l}^{2}\right)\right]^{2},$
$u_{li}=y_{i}-f_{nl}\left(x_{li}\right),$
$\hat{u}_{li}=y_{i}-P^{m_{n}}\left(x_{li}\right)^{\prime}\boldsymbol{\hat{\beta}}_{l},$
and $\hat{\sigma}_{l}^{2}=n^{-1}\sum_{i=1}^{n}\hat{u}_{li}^{2}.$ Then, for
$v_{n}\propto n^{\lambda}$ with $\lambda>1/2,$

$\Pr\left(\left|\hat{\sigma}_{l}^{2}-\sigma_{l}^{2}\right|\geq
v_{n}/n\right)\leq C_{1}\exp\left(-C_{2}n^{C_{3}}\right)$

holds for some $C_{1},$ $C_{2},$ and $C_{3}>0$.

###### Lemma A.6

Suppose Assumptions 1, 3, 4, 6, and 7 hold.

(i) If $v_{n}\lesssim n^{s/\left(s+2\right)}m_{n}^{2}$, then

$\Pr\left(\left|\boldsymbol{u}_{l}^{\prime}\mathbb{X}_{l}\left(\hat{\sigma}_{l}^{2}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}\right|\geq\frac{4}{3}\sigma_{l}^{2}v_{n}\right)\leq\exp\left(-C_{1}m_{n}^{-1}v_{n}+\log
m_{n}\right)+C_{2}\exp\left(-C_{3}n^{C_{4}}\right)$

for some positive constants $C_{1},$ $C_{2},$ $C_{3},$ and $C_{4}$.

(ii) If $v_{n}\gtrsim n^{s/\left(s+2\right)}m_{n}^{2},$ then

$\Pr\left(\left|\boldsymbol{u}_{l}^{\prime}\mathbb{X}_{l}\left(\hat{\sigma}_{l}^{2}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}\right|\geq\frac{4}{3}\sigma_{l}^{2}v_{n}\right)\leq
C_{5}\exp\left(-C_{6}n^{C_{7}}\right)$

for some positive constants $C_{5},$ $C_{6},$ and $C_{7}.$

###### Lemma A.7

Suppose that Assumptions 3, 5’, and 7 hold and that we pre-select
$\boldsymbol{Z}=\left(Z_{1},\ldots,Z_{\iota_{n}}\right)^{\prime}$ in previous
stages and all $Z$s are either signals or pseudo-signals. Assume that $b$
signals, without loss of generality say
$\boldsymbol{X}_{1}^{b}\equiv\left(X_{1},\ldots,X_{b}\right)^{\prime},$ are
not in the list of the pre-selected variables. Let

$\boldsymbol{\eta}_{1}^{b}=E\left[P^{m_{n}}\left(\boldsymbol{X}_{1}^{b}\right)\left(Y-P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}\left(E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}\right]\right)^{-1}E\left[P^{m_{n}}\left(\boldsymbol{Z}\right)Y\right]\right)\right]$

If there exist a $j,$ $1\leq j\leq b,$ such that
$\left\\{E\left[f_{nj}^{\ast}\left(X_{j}\right)^{2}\right]\right\\}^{1/2}\gtrsim\kappa_{n}\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2}$
for some $\kappa_{n}\rightarrow\infty$, we have

$\left\|\boldsymbol{\eta}_{1}^{b}\right\|\gtrsim\kappa_{n}\log\left(m_{n}\right)^{1/2}n^{-1/2}.$

###### Lemma A.8

Suppose Assumptions 1, 3, and 10 hold. Further suppose all covariates in
$\boldsymbol{Z}$ ($\iota_{n}\times 1$) are either signals or pseudo signals.
For a random variable $X_{l}$ that is not included in $\boldsymbol{Z},$ it
holds that

$\Pr\left(\left|\left\|\left(n^{-1}\left(\begin{array}[c]{c}\mathbb{Z}^{\prime}\\\
\mathbb{X}_{l}^{\prime}\end{array}\right)\left(\begin{array}[c]{cc}\mathbb{Z}&\mathbb{X}_{l}\end{array}\right)\right)^{-1}\right\|-\left\|\Phi^{-1}\right\|\right|>\left\|\Phi^{-1}\right\|/2\right)\leq
2m_{n}^{2}\iota_{n}^{2}\exp\left\\{-C_{1}nm_{n}^{-3}\iota_{n}^{-1}\right\\}$

for some positive constant $C_{1},$ where
$\Phi=\left[\begin{array}[c]{cc}\Phi_{\boldsymbol{Z}}&\Phi_{\boldsymbol{Z}X_{l}}\\\
\Phi_{\boldsymbol{Z}X_{l}}^{\prime}&\Phi_{X_{l}}\end{array}\right],$
$\Phi_{\boldsymbol{Z}X_{l}}=E\left[P^{m_{n}}(\boldsymbol{Z})^{\prime}P^{m_{n}}(X_{l})\right],$
and recall that
$\Phi_{\boldsymbol{Z}}=E[P^{m_{n}}\left(\boldsymbol{Z}\right)P^{m_{n}}\left(\boldsymbol{Z}\right)^{\prime}]$
and
$\Phi_{X_{l}}=E\left[P^{m_{n}}\left(X_{l}\right)P^{m_{n}}\left(X_{l}\right)^{\prime}\right].$
If in addition Assumptions 2’ and 7 hold, the right-hand side of the
inequality can be replaced by $C_{2}\exp\left(-C_{3}n^{C_{4}}\right).$

###### Lemma A.9

Suppose Assumptions 1, 2’, 3, 4, 6, 7, and 10 hold. Further suppose all
covariates in $\boldsymbol{Z}$ ($\iota_{n}\times 1$) are either signals or
pseudo signals. For a random variable $X_{l}$ that is not included in
$\boldsymbol{Z},$ then

$\displaystyle\Pr\left(\lambda_{\max}\left\\{\left(\hat{\sigma}_{l,\boldsymbol{Z}}^{2}n^{-1}\mathbb{X}_{l}^{\prime}M_{\mathbb{Z}}\mathbb{X}_{l}\right)^{-1}\right\\}\geq
2\sigma^{-2}B_{X1}^{-1}m_{n}\right)$ $\displaystyle\leq
C_{1}\exp(-C_{2}n^{C_{3}})\text{ and}$
$\displaystyle\Pr\left(\lambda_{\min}\left\\{\left(\hat{\sigma}_{l,\boldsymbol{Z}}^{2}n^{-1}\mathbb{X}_{l}^{\prime}M_{\mathbb{Z}}\mathbb{X}_{l}\right)^{-1}\right\\}\leq\frac{1}{4}\sigma^{-2}B_{X2}^{-1}m_{n}\right)$
$\displaystyle\leq C_{4}\exp(-C_{5}n^{C_{6}}),$

for some positive $C_{1},\ldots,C_{6},$ where $\sigma^{2}$ is defined in Lemma
A.11.

###### Lemma A.10

Suppose Assumptions 1, 2’, 3, 4, 5’, 6, 7, and 10 hold. Further suppose all
covariates in $\boldsymbol{Z}$ ($\iota_{n}\times 1$) are either signals or
pseudo signals. For a random variable $X_{l}$ that is not included in
$\boldsymbol{Z},$then

$\displaystyle\Pr\left(\left\|n^{-1/2}\left(\boldsymbol{u}_{X_{l},\boldsymbol{Z}}^{\prime}\boldsymbol{u}_{Y,\boldsymbol{Z}}-\boldsymbol{\eta}_{l,\boldsymbol{Z}}\right)\right\|^{2}\geq
C_{1}m_{n}^{-1}v_{n}\right)$
$\displaystyle\leq\left\\{\begin{array}[c]{c}2m_{n}\exp\left\\{-nm_{n}^{-2}v_{n}/[C_{2}(nm_{n}^{-1}+\iota_{n}n^{1/2}v_{n}^{1/2})]\right\\}+m_{n}\exp\left(-C_{3}m_{n}^{-1}v_{n}\right)\\\
2m_{n}\exp\left\\{-nm_{n}^{-2}v_{n}/[C_{2}(nm_{n}^{-1}+\iota_{n}n^{1/2}v_{n}^{1/2})]\right\\}+C_{4}\exp\left(-C_{5}n^{C_{6}}\right)\end{array}\right.\begin{array}[c]{c}\text{if
}v_{n}\lesssim n^{s/\left(s+2\right)}m_{n}^{2}\\\ \text{if }v_{n}\gtrsim
n^{s/\left(s+2\right)}m_{n}^{2}\end{array},$

and

$\displaystyle\Pr\left(\left\|n^{-1/2}\boldsymbol{u}_{X_{l},\boldsymbol{Z}}^{\prime}\mathbb{Z}\left(\mathbb{Z}^{\prime}\mathbb{Z}\right)^{-1}\mathbb{Z}^{\prime}\boldsymbol{u}_{Y,\boldsymbol{Z}}\right\|^{2}\geq
C_{1}m_{n}^{-1}v_{n}\right)$
$\displaystyle\leq\left\\{\begin{array}[c]{l}C_{13}\exp\left(-C_{14}n^{C_{15}}\right)+m_{n}^{2}\iota_{n}\exp\left(-C_{16}n^{1/2}m_{n}^{-1/6}\iota_{n}^{-1}\left(m_{n}^{-1}v_{n}\right)^{1/2}\right)\\\
C_{17}\exp\left(-C_{18}n^{C_{19}}\right)\end{array}\right.\begin{array}[c]{c}\text{if
}v_{n}\lesssim
n^{\left(s-2\right)/\left[4\left(s+2\right)\right]}m_{n}^{5/6}\iota_{n}^{2}\\\
\text{if }v_{n}\gtrsim
n^{\left(s-2\right)/\left[4\left(s+2\right)\right]}m_{n}^{5/6}\iota_{n}^{2}\end{array},$

for some positive $C_{2},C_{3},C_{4},C_{5},$ and $C_{6}$. If we set
$v_{n}=\varsigma_{n},$ then

$\Pr\left(\left\|n^{-1/2}\boldsymbol{\tilde{u}}_{l,\boldsymbol{Z}}\right\|^{2}\geq
C_{1}m_{n}^{-1}\varsigma_{n}\right)\leq
n^{-M}+C_{20}\exp\left(-C_{21}n^{C_{22}}\right)$

for any large positive constant $M$ and some positive constants
$C_{20},C_{21},$ and $C_{22}.$

###### Lemma A.11

Suppose Assumptions 1, 2’, 3, 4, 5’, 7, and 10 hold. Suppose all covariates in
$\boldsymbol{Z}$ ($\iota_{n}\times 1$) are either signals or pseudo signals
and $X_{l}$ is a variable that is not included in $\boldsymbol{Z}.$ Define

$U=Y-\left(\begin{array}[c]{c}P^{m_{n}}\left(\boldsymbol{Z}\right)\\\
P^{m_{n}}\left(X_{l}\right)\end{array}\right)^{\prime}\Phi^{-1}E\left[\left(\begin{array}[c]{c}P^{m_{n}}\left(\boldsymbol{Z}\right)\\\
P^{m_{n}}\left(X_{l}\right)\end{array}\right)Y\right],$

Let $\sigma^{2}=E\left(U^{2}\right),$
$\omega^{4}=E\left(U^{4}\right)-\left[E\left(U^{2}\right)\right]^{2}$. Then,
for $v_{n}\propto n^{\lambda}$ with $\lambda>1/2,$ we have

$\Pr\left(\left|\hat{\sigma}_{l,\boldsymbol{Z}}^{2}-\sigma^{2}\right|\geq
v_{n}/n\right)\leq C_{1}\exp\left(-C_{2}n^{C_{3}}\right)\text{ for some
}C_{1},C_{2},C_{3}>0.$

###### Lemma A.12

Suppose Assumptions 1, 2’, 3, 4, 6, 7, and 10 hold. Further suppose all
covariates in $\boldsymbol{Z}$ ($\iota_{n}\times 1$) are either signals or
pseudo signals. For a random variable $X_{l}$ that is not included in
$\boldsymbol{Z},$ we have

$\Pr\left(\boldsymbol{\tilde{u}}_{l,\boldsymbol{Z}}^{\prime}\left(\hat{\sigma}_{l,\boldsymbol{Z}}^{2}\mathbb{X}_{l}^{\prime}M_{\mathbb{Z}}\mathbb{X}_{l}\right)^{-1}\boldsymbol{\tilde{u}}_{l,\boldsymbol{Z}}\geq\varsigma_{n}\right)\leq\exp\left(-C_{1}m_{n}^{-1}\varsigma_{n}+\log
m_{n}\right)+C_{2}\exp(-C_{3}n^{C_{4}}).$

###### Lemma A.13

Suppose Assumptions 1, 2’, 3, 4, 5’, 6, 7, 8, and 10 hold. Then

$\Pr\left(\mathcal{D}_{k}\right)\geq
1-kn^{-M_{1}}-kC_{4}\exp\left(-C_{5}n^{C_{6}}\right)$

for any finite fixed positive integer $k,$ any fixed large positive constant
$M_{1},$ and some positive constants $C_{4},C_{5},$ and $C_{6}.$ When $k$ is
fixed or divergent to infinity at a rate no faster than $n^{a}$ for some
$a>0,$ we can write

$\Pr\left(\mathcal{D}_{k}\right)\geq
1-n^{-M_{2}}-C_{7}\exp\left(-C_{8}n^{C_{9}}\right)\text{ for any }k\leq
n^{a},$

for some large positive constant $M_{2}$ and some positive constants
$C_{7},C_{8},$ and $C_{9}.$

##  Appendix B Proofs of the Main Results

Proof of Proposition 2.1. To see the intuition, we begin with the case where
$\theta_{l}=0$. When $\theta_{l}=0,$ $\left\|\boldsymbol{\beta}_{l}\right\|=0$
and
$\mathcal{\hat{X}}_{l}=\boldsymbol{u}_{l}^{\prime}\mathbb{X}_{l}\left(\hat{\sigma}_{l}^{2}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}.$
Then,

$\displaystyle\Pr\left(\mathcal{\hat{X}}_{l}\geq\varsigma_{n}\right)$
$\displaystyle\leq\Pr\left(\boldsymbol{u}_{l}^{\prime}\mathbb{X}_{l}\left(\hat{\sigma}_{l}^{2}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}\geq\varsigma_{n}\right)$
$\displaystyle=\Pr\left(\boldsymbol{u}_{l}^{\prime}\mathbb{X}_{l}\left(\hat{\sigma}_{l}^{2}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}\geq\frac{4}{3}\sigma_{l}^{2}\left(\frac{3}{4}\sigma_{l}^{-2}\varsigma_{n}\right)\right)$
$\displaystyle\leq\exp\left(-C_{1}m_{n}^{-1}\varsigma_{n}+\log
m_{n}\right)+C_{2}\exp\left(-C_{3}n^{C_{4}}\right)$

for some positive constants $C_{1},C_{2},$ and $C_{3},$ where the last line
holds by applying the first part of Lemma A.6 with
$v_{n}=\frac{3}{4}\sigma_{l}^{-2}\varsigma_{n}$. Since
$\varsigma_{n}\propto\kappa_{n}\log\left(m_{n}\right)m_{n},$
$C_{1}m_{n}^{-1}\varsigma_{n}-\log m_{n}\gg M\log m_{n}$ for any fixed
positive $M.$ This shows the first part of the proposition when
$\theta_{l}=0.$

Note by Lemma A.3

$\left\\{E\left[f_{nl}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}\propto
m_{n}^{-1/2}\left\|\boldsymbol{\beta}_{l}\right\|.$

If $\theta_{l}\lesssim\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2},$
by the triangle inequality, equation (A.2) and Assumption 7,

$\displaystyle\left\\{E\left[f_{nl}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}$
$\displaystyle\leq\left\\{E\left[f_{l}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}+\left\\{E\left[\left|f_{nl}\left(X_{l}\right)-f_{l}\left(X_{l}\right)\right|^{2}\right]\right\\}^{1/2}$
$\displaystyle\lesssim\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2}+C_{1}m_{n}^{-d}\lesssim\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2},$

because the approximation bias is asymptotically negligible under Assumption
7. Therefore,
$\left\|\boldsymbol{\beta}_{l}\right\|\lesssim\log\left(m_{n}\right)^{1/2}m_{n}n^{-1/2}.$
Then,

$\displaystyle\Pr\left(\mathcal{\hat{X}}_{l}\geq\varsigma_{n}\right)$
$\displaystyle=\Pr\left(\left(n^{1/2}\boldsymbol{\beta}_{l}+n^{1/2}\left(\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}\right)^{\prime}\left(\hat{\sigma}_{l}^{-2}n^{-1}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)\left(n^{1/2}\boldsymbol{\beta}_{l}+n^{1/2}\left(\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}\right)\geq\varsigma_{n}\right)$
$\displaystyle\leq\Pr\left(\left\|n^{1/2}\boldsymbol{\beta}_{l}+n^{1/2}\left(\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}\right\|^{2}\hat{\sigma}_{l}^{-2}\lambda_{\max}\left\\{n^{-1}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right\\}\geq\varsigma_{n}\right)$
$\displaystyle\leq\Pr\left(\left\|n^{1/2}\boldsymbol{\beta}_{l}+n^{1/2}\left(\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}\right\|^{2}\geq\frac{1}{3}\sigma_{l}^{2}B_{2}^{-1}m_{n}\varsigma_{n}\right)+\Pr\left(\hat{\sigma}_{l}^{-2}\geq
2\sigma_{l}^{-2}\right)$
$\displaystyle+\Pr\left(\lambda_{\max}\left\\{\left(\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)\right\\}\geq\frac{3}{2}B_{2}m_{n}^{-1}\right)$
$\displaystyle\equiv A_{1}+A_{2}+A_{3}.$

By Lemma A.5 and Lemma A.4,
$A_{l}=\frac{1}{4}C_{2}\exp\left(-C_{3}n^{C_{4}}\right)$ for some constants
$C_{2},C_{3}$ and $C_{4}$ and $l=2,3.$ For $A_{1},$ we have with
$C\equiv\frac{1}{3}\sigma_{l}^{2}B_{2}^{-1},$

$\displaystyle A_{1}$
$\displaystyle\leq\Pr\left(\left\|n^{1/2}\left(\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}\right\|+\left\|n^{1/2}\boldsymbol{\beta}_{l}\right\|\geq
Cm_{n}^{1/2}\varsigma_{n}^{1/2}\right)$
$\displaystyle=\Pr\left(\left\|\left(n^{-1}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}n^{-1/2}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}\right\|\geq\frac{1}{2}Cm_{n}^{1/2}\varsigma_{n}^{1/2}+\frac{1}{2}Cm_{n}^{1/2}\varsigma_{n}^{1/2}-\left\|n^{1/2}\boldsymbol{\beta}_{l}\right\|\right)$
$\displaystyle\leq\Pr\left(\left\|\left(n^{-1}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}n^{-1/2}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}\right\|\geq\frac{1}{2}Cm_{n}^{1/2}\varsigma_{n}^{1/2}\right)$
$\displaystyle\leq\Pr\left(\left\|n^{-1/2}\mathbb{X}_{l}^{\prime}\boldsymbol{u}_{l}\right\|\geq\frac{1}{2}Cm_{n}^{-1/2}\varsigma_{n}^{1/2}\right)+\Pr\left(\left\|\lambda_{\max}\left(n^{-1}\mathbb{X}_{l}^{\prime}\mathbb{X}_{l}\right)^{-1}\right\|\geq\frac{3}{2}B_{1}^{-1}m_{n}\right)$
$\displaystyle\leq n^{-M}+\frac{1}{2}C_{2}\exp\left(-C_{3}n^{C_{4}}\right)$

for any arbitrarily large constant $M$ and some constants $C_{2},C_{3}$ and
$C_{4},$ where for the fourth line holds by the fact that
$m_{n}^{1/2}\varsigma_{n}^{1/2}=\kappa_{n}^{1/2}\log\left(m_{n}\right)^{1/2}m_{n}\gg\left\|n^{1/2}\boldsymbol{\beta}_{l}\right\|,$
and the last line holds by equation (S1.14), the fact that
$C_{1}m_{n}^{-1}\varsigma_{n}-\log m_{n}\gg M\log m_{n},$ and Lemma A.4. In
sum, we have $\Pr\left(\mathcal{\hat{X}}_{l}\geq\varsigma_{n}\right)\leq
n^{-M}+C_{2}\exp\left(-C_{3}n^{C_{4}}\right).$

We turn to the second part of the proposition. If
$\theta_{l}\gtrsim\kappa_{n}\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2},$
by equation (A.2) and Assumption 7,

$\left\\{E\left[f_{nl}\left(X_{l}\right)^{2}\right]\right\\}^{1/2}\gtrsim\kappa_{n}\log\left(m_{n}\right)^{1/2}\left(m_{n}/n\right)^{1/2},$

where we use the fact that the approximation bias is asymptotically negligible
under Assumption 7. Therefore,