# Robust and Adaptive Functional Logistic Regression

Ioannis Kalogridis

(Department of Mathematics, KU Leuven, Belgium
)

###### Abstract

We introduce and study a family of robust estimators for the functional
logistic regression model whose robustness automatically adapts to the data
thereby leading to estimators with high efficiency in clean data and a high
degree of resistance towards atypical observations. The estimators are based
on the concept of density power divergence between densities and may be formed
with any combination of lower rank approximations and penalties, as the need
arises. For these estimators we prove uniform convergence and high rates of
convergence with respect to the commonly used prediction error under fairly
general assumptions. The highly competitive practical performance of our
proposal is illustrated on a simulation study and a real data example which
includes atypical observations.

Keywords: Functional logistic regression, robustness, regularization,
asymptotics

MSC 2020: 62R10, 62G35, 62G20

##  1 Introduction

In recent years, advances in technology and improved storage capabilities have
enabled applied scientists to observe and record increasingly complex high-
dimensional objects, which may be assumed to be realizations of a random
function with values on some nicely behaved function space. Such data are
commonly referred to as functional data and their study has enjoyed notable
popularity following breakthrough works, such as Ramsay and Dalzell (1991),
Ramsay and Silverman (2005) and Ferraty and Vieu (2006). In many cases, it is
of interest to relate a random function $X$ with sample paths in
$\mathcal{L}^{2}(\mathcal{I})$ for some compact $\mathcal{I}\subset\mathbb{R}$
to a binary response $Y$ through the model

$\mathbb{E}\\{Y|X\\}=H\left(\alpha_{0}+\int_{\mathcal{I}}X(t)\beta_{0}(t)dt\right),$
(1)

with $H:\mathbbm{R}\to(0,1)$ a known bijective ”link” function and
$(\alpha_{0},\beta_{0})\in\mathbb{R}\times\mathcal{L}^{2}(\mathcal{I})$
unknown quantities that need to be estimated from the data. This scalar-on-
function logistic regression model originally proposed by James (2002) remains
popular to this day owing to a broad range of possible applications, such as
cancer gene detection and foetal risk assessment, see Ratcliffe et al. (2002)
and Wei et al. (2014) respectively.

Despite continuous interest by practitioners, the theoretical study of the
scalar-on-function logistic regression has significantly lagged behind and the
scalar-on-function linear regression model has instead garnered most of the
attention. For such a model, the interested practitioner would readily find a
wealth of theoretically grounded estimation techniques at her disposal. We
mention, for example, regression on the leading functional principal
components (Cardot et al., 1999; Hall et al., 2007), penalized spline
regression (Cardot et al., 2003), full-rank smoothing spline regression, as
proposed and studied by Crambes et al. (2009) and Yuan and Cai (2010), as well
as hybrid approaches involving both functional principal components and
penalized splines (Reiss and Ogden, 2007). Unfortunately, none of the
theoretical results developped in the aforementioned works carry over to the
functional logistic regression model and, to the best of our knowledge, the
only theoretical results for that model were developed by Cardot and Sarda
(2005) and Müller and Stadtmüller (2005). The former authors established the
consistency of penalized likelihood based spline estimators whereas the latter
authors proved the consistency and asymptotic normality of estimates derived
from a truncated functional principal component expansion of the coefficient
function $\beta_{0}$.

An important practical disadvantage of both the proposal of Cardot and Sarda
(2005) and that of Müller and Stadtmüller (2005) is their lack of resistance
towards atypical observations. The reason for this lack of resistance is their
reliance on the log-likelihood contributions of each observation
$(X_{i},Y_{i})$ so that any badly fitting pair of observations, e.g., an
$X_{i}$ with a large leverage and a misclassified $Y_{i}$, would significantly
distort the log-likelihood function as well as estimates derived from it. As
an attempt to address this lack of resistance, Denhere and Billor (2016)
proposed performing logistic regression on the leading robust principal
components obtained with the method of Hubert et al. (2005). This method is
only partially effective in practice, as it only addresses outlying predictor
functions and not misclassified responses, which can still lead to severe
distortions. More recently, Mutis et al. (2022) expanded on the proposal of
Denhere and Billor (2016) by incorporating weights based on the Pearson
residuals to the log-likelihood thereby obtaining estimates with greater
resistance. However, the theoretical properties of none of these methods are
understood so that it is impossible to decide on best practices.

In order to fill the consequential vacuum in the literature with respect to
estimators lacking either theoretical justification or resistance to outlying
observations, this paper introduces and studies a flexible family of penalized
lower-rank smoothers that can fulfil both roles. Resistance is achieved
through the use of the density power divergence objective function introduced
by Basu et al. (1998), which we flexibly combine with a lower rank
representation of the coefficient function and a penalty term in order to both
obtain easily computable estimates that avoid overfitting. We select the
tuning parameter of our objective function in a data-driven way so as to
achieve high efficiency in clean data and significant resistance to outlying
observations. Under general assumptions, which even permit random tuning
parameters, we establish the uniform convergence of our estimator and obtain a
useful error decomposition of the prediction error into the variance and the
twin biases resulting from modelling and penalization. These theoretical
results are the first among robust estimators for the scalar-on-functional
logistic model, but our modern methodology may also be used to strengthen
older results, such as those of Cardot and Sarda (2005) dealing with penalized
maximum likelihood estimation.

The rest of the paper is structured as follows. Section 2 motivates and
describes our proposal in detail. Section 3 is devoted to the study of the
asymptotic properties of the proposed class of estimators; we discuss both
consistency and rates of convergence with respect to the uniform and
prediction errors respectively. Practical selection of the size of the
approximating subspace, the penalty parameter and the tuning constant of the
objective function are discussed in Section 4. Sections 5 and 6 illustrate the
competitive finite-sample performance of our proposal in a Monte Carlo study
and in a real data example involving gait analysis, respectively. Finally,
section 7 contains a short concluding discussion while all proofs are
collected in the appendix.

##  2 The proposed family of estimators

In order to describe the proposed family of estimators in detail, it is first
necessary to review the concept of density power divergence, as introduced by
Basu et al. (1998). Consider two densities with respect to the counting
measure, $h$ and $f$. For $\kappa_{0}\geq 0$ the density power divergence
$d_{\kappa_{0}}(h,g)$ is defined as

$\displaystyle
d_{\kappa_{0}}(h,f)=\begin{dcases}\sum_{y\in\mathbbm{Z}}\left\\{f^{1+\kappa_{0}}(y)-\left(1+\frac{1}{\kappa_{0}}\right)h(y)f^{\kappa_{0}}(y)+\frac{1}{\kappa_{0}}h^{1+\kappa_{0}}(y)\right\\}&\kappa_{0}>0\\\
\sum_{y\in\mathbbm{Z}}h(y)\log\\{h(y)/f(y)\\}&\kappa_{0}=0.\end{dcases}$ (2)

It can be shown that $d_{\kappa_{0}}(h,f)\geq 0$ and that
$d_{\kappa_{0}}(h,f)=0$ if, and only if, $h=f$ on all of $\mathbbm{Z}$, see
Lemma 1 in the Appendix. If we assume that $h$ is the true density and
$\\{f_{\theta},\theta\in\Theta\\}$ is a family of model densities, density
power divergence estimators asymptotically minimize
$d_{\kappa_{0}}(h,f_{\theta})$ over $\theta\in\Theta$. It is worth noting that
$d_{0}(h,f_{\theta})$ is merely the Kullback-Leibler divergence which is
asymptotically minimized by maximum likelihood estimators. It is well-known,
however, that although maximum likelihood estimators are efficient, they are
not robust against outliers and model misspecification. Density power
divergence estimators based on $\kappa_{0}>0$ can be shown to be more
resistant, in the sense that their influence function (Hampel et al., 2011) is
bounded and redescending for large outliers, and are therefore better suited
for the analysis of complex datasets containing outliers. In fact, as
demonstrated by the analysis of Kalogridis et al. (2023), the robustness of
density power divergence estimators increases with higher values of
$\kappa_{0}$, but that comes at a cost for their efficiency so that in
practice a deal between efficiency and robustness needs to be struck.

To see how density divergence may be used in the context of functional
logistic regression, let us assume that we are in possession of a sample
$(X_{1},Y_{1}),\ldots,(X_{n},Y_{n})$ consisting of independent and identically
distributed tuples in $\mathcal{L}^{2}(\mathcal{I})\times\\{0,1\\}$ following
model (1). For simplicity and without loss of generality we shall identify
$\mathcal{I}$ with the $[0,1]$-interval. In view of (1), we may assume that
conditionally on $X_{i}$ each binary response $Y_{i}$ possesses a Bernoulli
density
$f_{p_{i}(\alpha_{0},\beta_{0})}(y)=p_{i}(\alpha_{0},\beta_{0})^{y}(1-p_{i}(\alpha_{0},\beta_{0}))^{1-y}$
where $p_{i}(\alpha_{0},\beta_{0})$, the probability of ”success”, is given by

$\displaystyle
p_{i}(\alpha_{0},\beta_{0})=H\left(\alpha_{0}+\int_{[0,1]}X_{i}(t)\beta_{0}(t)dt\right),\quad(i=1,\ldots,n).$

We show in the appendix that
$M((\alpha,\beta),\kappa_{0}):=\mathbb{E}\\{d_{\kappa_{0}}(f_{p_{1}(\alpha_{0},\beta_{0})},f_{p_{1}(\alpha,\beta)})\\}$
is bounded below by $0$ and bounded above by $1+\kappa_{0}^{-1}$. The upper
bound is attained if, and only if,
$|p_{1}(\alpha_{0},\beta_{0}))-p_{1}(\alpha,\beta)|=1$ almost everywhere.
Moreover, $M((\alpha,\beta),\kappa_{0})$ is a continuous function of
$(\alpha,\beta)$ whenever $H$, the link function, is also continuous. These
properties make $M((\alpha,\beta),\kappa_{0})$ an attractive objective
function for robust estimation, but unfortunately minimizing
$M((\alpha,\beta),\kappa_{0})$ directly is not a feasible task owing to our
lack of knowledge of either the distribution of $X$ or the population
parameters $(\alpha_{0},\beta_{0})$. Our approach thus consists of minimizing
a suitable approximation of this criterion.

A finite sample approximation to $M((\alpha,\beta),\kappa_{0})$ may be
obtained by replacing $\mathbb{E}\\{\cdot\\}$ with its empirical counterpart
weighing each $X_{i}$ with $n^{-1}$, as is typically done in the construction
of M-estimators (see, e.g., van der Vaart, 1998, Chapter 5). Next, from (2) it
may be seen that we may drop terms not depending on $(\alpha,\beta)$, as these
terms do not influence the value of the minimizer. Lastly, for every
$\kappa_{0}>0$, a conditionally unbiased estimator for
$\sum_{y}f^{\kappa_{0}}_{p_{i}(\alpha,\beta)}(y)f_{p_{i}(\alpha_{0},\beta_{0})}(y)$
is given by $f_{p_{i}(\alpha,\beta)}^{\kappa_{0}}(Y_{i})$. Putting everything
together, we arrive at the finite-sample approximation of
$M((\alpha,\beta),\kappa_{0})$ given by

$\displaystyle\frac{1}{n}\sum_{i=1}^{n}l_{\widehat{\kappa}_{n}}(Y_{i},p_{i}(\alpha,\beta))=\frac{1}{n}\sum_{i=1}^{n}\left[\sum_{y\in\\{0,1\\}}f^{1+\widehat{\kappa}_{n}}_{p_{i}(\alpha,\beta)}(y)-\left(1+\frac{1}{\widehat{\kappa}_{n}}\right)f^{\widehat{\kappa}_{n}}_{p_{i}(\alpha,\beta)}(Y_{i})\right],$
(3)

where we now also allow for changing, even random, tuning parameters
$\widehat{\kappa}_{n}$ that may depend on the sample itself. These parameters
determine the trade-off between robustness and efficiency, as for
$\widehat{\kappa}_{n}\to 0$ we obtain what is essentially the log-likelihood
of the Bernoulli distribution whereas for larger $\widehat{\kappa}_{n}$ we
obtain an objective function that leads to more robust but less efficient
estimators. The above formulation allows us in a sense to get the best of both
worlds, as $\widehat{\kappa}_{n}$ may be chosen small enough in clean samples
and larger in datasets containing outlying observations. A possible selection
scheme for $\widehat{\kappa}_{n}$ that we have found to be effective in
practice is outlined in Section 4 below.

We follow the recipe of Ramsay and Silverman (2005, Chapter 15) in order to
obtain estimators from the objective function (3). That is, rather than
minimizing (3) directly over the whole of
$\mathbbm{R}\times\mathcal{L}^{2}([0,1])$, which entails the risk of
overfitting, we instead restrict attention to the considerably smaller space
$\mathbbm{R}\times\Theta_{K_{n}}$ where $\Theta_{K_{n}}$ is a
$K_{n}$-dimensional linear subspace that is spanned by known real-valued
functions $\theta_{1},\ldots,\theta_{K_{n}}$. These functions can, for
example, be B-splines or Fourier bases, both of which are broadly used in
functional data analysis. Here, the dimension of the approximating subspace,
$K_{n}$, depends on $n$, as for the approximation bias to be vanish in the
limit we need to have $K_{n}\to\infty$ as $n\to\infty$ but not too fast, see
Theorem 1 below. Other than a finite-dimensional subspace, we also make use of
a penalty functional $\mathcal{J}:\Theta_{K_{n}}\to\mathbbm{R}_{+}$ with the
aim of further controlling the roughness of the slope estimates. With these
two refinements, our proposal amounts to estimating the unknown quantities
with

$\displaystyle(\widehat{\alpha}_{n},\widehat{\beta}_{n})=\operatorname*{argmin}_{(\alpha,\beta)\in\mathbbm{R}\times\Theta_{K_{n}}}\left[\frac{1}{n}\sum_{i=1}^{n}l_{\widehat{\kappa}_{n}}(Y_{i},p_{i}(\alpha,\beta))+\widehat{\lambda}_{n}\mathcal{J}(\beta)\right],$
(4)

with $\widehat{\lambda}_{n}$ a possibly random penalty parameter determining
the premium that is placed on the roughness of $\widehat{\beta}_{n}$, as
measured by $\mathcal{J}(\widehat{\beta}_{n})$. For $\widehat{\lambda}_{n}=0$
the estimator is an unpenalized linear combination of the $\theta_{j}$ whereas
for $\widehat{\lambda}_{n}\to\infty$ the penalty dominates the objective
function forcing the slope estimator to lie in its null-space. The null-space
of $\mathcal{J}(\cdot)$ is often low dimensional, e.g., it may consist of
polynomials up to a certain order, so that functions that lie on it tend to be
of simple form.

It is worth noting that the estimator in (4) is regularized both with respect
to its dimension and its complexity, as, in our experience, both types of
regularization are needed for a reliable all-around practical performance. In
particular, the present dual regularization scheme ensures that the estimator
can be made flexible enough by taking a large value of $K_{n}$, while also
remaining interpretable on account of the penalization by means of
$\mathcal{J}(\cdot)$. Moreover, the present set-up is very flexible and allows
for a wide variety of approximating subspaces $\Theta_{K_{n}}$ and penalties
$\mathcal{J}(\cdot)$, as the applied scientist sees fit. These may include the
frequently used B-splines with derivative or difference penalties, but also
wavelets with total variation or $l_{1}$ penalties for cases in which
$\beta_{0}$ is suspected to be less smooth. The interested reader is referred
to Kalogridis and Van Aelst (2023) for a discussion of a number of practically
useful alternatives.

At a first glance it may seem that the proposed estimator is computationally
intensive, as it depends on three parameters, namely the tuning parameter
$\widehat{\kappa}_{n}$, the dimension of the approximating subspace, $K_{n}$,
and the penalty parameter $\widehat{\lambda}_{n}$. However, practical
experience with lower-rank penalized estimators has shown that $K_{n}$ is not
as crucial as the other parameters of the problem and therefore it can be
selected in a semi-automatic manner, that is, almost independently of the
sample (see, e.g., Ruppert et al., 2003; Wood, 2017). An efficient method of
selecting $\widehat{\kappa}_{n}$ and $\widehat{\lambda}_{n}$ is discussed in
Section 4 below.

##  3 Theoretical properties

###  3.1 Consistency

We now study the asymptotic properties of penalized density power divergence
estimators. Throughout this section we assume that $\alpha_{0}=0$ and the
object of interest is the coefficient function $\beta_{0}$, as is typically
the case. The assumptions that we require for our theoretical development are
as follows.

  * (A1)

There exists a $\kappa_{0}\in(0,1]$ such that
$\widehat{\kappa}_{n}\xrightarrow{\mathbb{P}}\kappa_{0}$, as $n\to\infty$.

  * (A2)

The link function $H:\mathbbm{R}\to(0,1)$ is bijective, strictly increasing
and differentiable with its derivative $H^{\prime}$ satisfying
$\sup_{x\in\mathbbm{R}}H^{\prime}(x)\leq c_{0}$ for some $c_{0}<\infty$.

  * (A3)

The coefficient function $\beta_{0}$ belongs to a Banach space of functions,
$\mathcal{B}([0,1])$, with norm $\|\cdot\|_{\mathcal{B}}$ that is embeddable
in $\mathcal{C}([0,1])$ with its norm $\|\cdot\|_{\infty}$, i.e.,
$\mathcal{B}([0,1])\subset\mathcal{C}([0,1])$ and there exists a finite
$c_{1}>0$ such that

$\displaystyle\|f\|_{\infty}\leq c_{1}\|f\|_{\mathcal{B}},\quad\forall
f\in\mathcal{B}([0,1]).$

  * (A4)

$\Theta_{K_{n}}\subset\mathcal{B}([0,1])$ and $K_{n}\asymp n^{\gamma}$ for
some $\gamma\in(0,1)$. Letting
$\widetilde{\beta}_{K_{n}}=\operatorname*{argmin}_{\beta\in\Theta_{K_{n}}}\|\beta-\beta_{0}\|$
we require $\|\widetilde{\beta}_{K_{n}}-\beta_{0}\|\to 0$ and
$\widehat{\lambda}_{n}\mathcal{J}(\widetilde{\beta}_{K_{n}})\xrightarrow{\mathbb{P}}0$,
as $n\to\infty$.

  * (A5)

There exists $0<C<\infty$ such that $\mathbb{P}(\|X\|\leq C)=1$ and there
exists $0\leq c_{2}<1$ such that for every $\beta\in\mathcal{B}([0,1]$ with
$\beta\neq 0$ it holds that $\mathbb{P}(\langle X,\beta\rangle=0)\leq c_{2}$.

Assumption (A1) requires that the random tuning parameter
$\widehat{\kappa}_{n}$ converges in probability to some positive constant
$\kappa_{0}$. Assumptions of this type are commonly used in the theoretical
treatment of estimators involving nuisance (estimated) parameters, see, e.g.,
(van der Vaart, 1998, Sections 5.4 and 5.8.1). It is also possible to select
the tuning parameter deterministically, i.e.,
$\widehat{\kappa}_{n}=\kappa_{0}$ for some $\kappa_{0}>0$ for all
$n\in\mathbbm{N}$ in which case (A1) is clearly satisfied and all subsequent
arguments remain valid. Assumption (A2) concerning the link function, $H$, is
similarly very general and permits all commonly used link functions, namely
the logit, probit and Gumbel (complementary log-log) links.

Assumption (A3) is a mild assumption on the coefficient function, $\beta_{0}$.
It is required that that $\beta_{0}$ lies in a complete normed space of
functions on $[0,1]$, $\mathcal{B}([0,1])$, which is a subspace of the space
of continuous functions, $\mathcal{C}([0,1])$. This assumption is considerably
more general than the corresponding assumption of Kalogridis and Van Aelst
(2023), as we do not require the embedding of $\mathcal{B}([0,1])$ in
$\mathcal{C}([0,1])$ to be compact. As a result, (A3) is satisfied by a wider
variety of function spaces. For example, while in our framework we may take
$\mathcal{B}([0,1])=\mathcal{C}([0,1])$, this is not permitted by the
assumptions of Kalogridis and Van Aelst (2023), as the unit ball
$\\{f\in\mathcal{C}([0,1]):\|f\|_{\infty}\leq 1\\}$ is not compact in
$\mathcal{C}([0,1])$.

Assumption (A4) requires that the dimension of the approximating subspace,
$K_{n}$, does not grow too fast relative to the sample size $n$, so that
$K_{n}/n\to 0$. This is a common assumption for estimators based on
approximating subspaces (sieves), see, e.g., Eggermont and LaRiccia (2009,
Chapter 15) and Kalogridis and Van Aelst (2023). Moreover, it is required that
$\beta_{0}$ can be well approximated in the $\mathcal{L}^{2}([0,1])$-norm by
an element of $\Theta_{K_{n}}$. This sequence of deterministic approximations
of $\beta_{0}$ in $\Theta_{K_{n}}$, $\widetilde{\beta}_{K_{n}}$, should have
finite roughness, as measured by $\mathcal{J}(\cdot)$ so that
$\widehat{\lambda}_{n}\mathcal{J}(\widetilde{\beta}_{K_{n}})\xrightarrow{\mathbb{P}}0$.
It is thus required that $\mathcal{J}(\cdot)$ is selected appropriately
relative to $\Theta_{K_{n}}$. Both requirements are met by many popular
approximating subspaces and penalties, e.g., by B-splines with derivative-
based penalties and by Fourier bases with either derivative or harmonic
acceleration penalties (Ramsay and Silverman, 2005). It is worth noting that
contrary to other theoretical treatments of functional linear regression
models, such as (Cardot et al., 2003; Yuan and Cai, 2010; Shin and Lee, 2016),
$\widehat{\lambda}_{n}$ is herein more realistically treated as a sequence of
random variables rather than a sequence of constants. This is an important
difference, as most often $\widehat{\lambda}_{n}$ is selected in a data-driven
manner and is thus random rather than fixed.

Finally, assumption (A5) requires that the covariate, $X$, is almost surely
bounded when viewed as a random element of $\mathcal{L}^{2}([0,1])$. A similar
assumption exists in many treatments of functional linear regression models,
e.g., in (Cardot et al., 2003; Kalogridis and Van Aelst, 2023). It is
additionally required that $X$ is not concentrated on any finite-dimensional
subspace of $\mathcal{B}([0,1])$. Since
$\mathcal{B}([0,1])\subset\mathcal{L}^{2}([0,1])$ this requirement is met
whenever the null space of the covariance operator of $X$ consists only of the
zero element $\\{0\\}$ which holds if, and only if, all of its countably
infinite eigenvalues are strictly positive. This assumption is likewise more
general than assumption (A4) of Kalogridis and Van Aelst (2023), as it
involves only the space $\mathcal{B}([0,1])$ instead of the larger
$\mathcal{L}^{2}([0,1])$. This is possible because, by (A4), our estimator
lies in $\mathcal{B}([0,1])$, hence we can restrict attention to this smaller
space of functions.

As shown in the appendix (see Lemma 1 and Lemma 2 there), our estimator is
Fisher consistent, that is, it estimates the correct quantity at the
population level, under assumptions (A1), (A2) and (A5) alone. The practical
significance of assumptions (A3) and (A4) is that they allow us to
additionally obtain a strong form of consistency of our finite sample
estimator $\widehat{\beta}_{n}$ towards the population function $\beta_{0}$.
This result is presented in Theorem 1 below.

######  Theorem 1.

Suppose that assumptions (A1)–(A5) are satisfied. Then,
$\|\widehat{\beta}_{n}-\beta_{0}\|_{\mathcal{B}}\xrightarrow{\mathbb{P}}0$, as
$n\to\infty$.

Theorem 1 establishes the convergence of our estimator $\widehat{\beta}_{n}$
as an element of the Banach space $\mathcal{B}([0,1])$. To the best of our
knowledge, this is a strong mode of convergence that has not been established
previously for any robust functional regression estimators. In particular, the
results of Boente et al. (2020) and Kalogridis and Van Aelst (2023) only imply
that $\|\widehat{\beta}_{n}-\beta_{0}\|_{\mathcal{B}}=O_{\mathbb{P}}(1)$, as
$n\to\infty$, which is clearly a weaker result. Since, by our assumptions,
$\mathcal{B}([0,1]$ is embeddable in $\mathcal{C}([0,1])$, the result of
Theorem 1 minimally implies the uniform convergence
$\|\widehat{\beta}_{n}-\beta_{0}\|_{\infty}\xrightarrow{\mathbb{P}}0$, as
$n\to\infty$, but it may also imply convergence of certain derivatives if
$\mathcal{B}([0,1])$ is more regular, e.g., if $\mathcal{B}([0,1])$ is a
Hölder or a Sobolev space, as is often assumed with B-spline or Fourier
estimators. For example, if $\mathcal{B}([0,1])$ is the Hölder space with
derivatives up to order $m$ satisfying a Hölder condition with exponent
$0<r\leq 1$, then convergence is with respect to the norm

$\displaystyle\left\|f\right\|_{\mathcal{B}}=\left\|f\right\|_{\infty}+\max_{0\leq
j\leq m}\sup_{\begin{subarray}{c}x,y\in[0,1]\\\ x\neq
y\end{subarray}}\left|\frac{f^{(j)}(x)-f^{(j)}(y)}{(x-y)^{r}}\right|,\quad
f\in\mathcal{B}([0,1]).$

Besides its intrinsic theoretical interest, such a mode of convergence offers
important practical guarantees regarding the performance of the estimator, in
that the population function $\beta_{0}$ and its derivatives will be estimated
equally well throughout their domain.

###  3.2 Rates of convergence

To obtain a rate of convergence of the proposed family of estimators, we use
the distance criterion given by

$\displaystyle\left|\pi(\widehat{\beta}_{n},\beta_{0})\right|^{2}=\mathbb{E}\left\\{\left|\langle
X,\widehat{\beta}_{n}-\beta_{0}\rangle\right|^{2}\right\\},$

which may be rewritten as
$\pi(\widehat{\beta}_{n},\beta_{0})=\\{\langle\Gamma(\widehat{\beta}_{n}-\beta_{0}),\widehat{\beta}_{n}-\beta_{0}\rangle\\}^{1/2}$
with $\Gamma$ denoting the self-adjoint Hilbert-Schmidt covariance operator of
$X$. This is a natural criterion to consider and it has been extensively in
theoretical investigations of functional linear models, e.g., by Cardot and
Sarda (2005); Crambes et al. (2009); Boente et al. (2020) and Kalogridis and
Van Aelst (2023). Intuitively, $\pi(\widehat{\beta}_{n},\beta_{0})$
corresponds to the average prediction error that arises when using $\langle
X_{n+1},\widehat{\beta}_{n}\rangle$ to predict $\langle
X_{n+1},\beta_{0}\rangle$, where $X_{n+1}$ is a new random function possessing
the same distribution as $X$.

It is worth noting that, under (A5), $\pi(\widehat{\beta}_{n},\beta_{0})\leq
C\|\widehat{\beta}_{n}-\beta_{0}\|_{\infty}$, so that, by Theorem 1,
$\pi(\widehat{\beta}_{n},\beta_{0})\xrightarrow{\mathbb{P}}0$. Theorem 2 below
serves to delineate the roles of the subspace $\Theta_{K_{n}}$, the penalty
functional $\mathcal{J}(\cdot)$ and the penalty parameter
$\widehat{\lambda}_{n}$ in the convergence of the prediction error to $0$.

######  Theorem 2.

Suppose that assumptions (A1)–(A5) are satisfied. Then,

$\displaystyle\left|\pi(\widehat{\beta}_{n},\beta_{0})\right|^{2}=O_{\mathbb{P}}\left(\frac{K_{n}\log
n}{n}\right)+O_{\mathbb{P}}\left(\left\|\widetilde{\beta}_{K_{n}}-\beta_{0}\right\|^{2}\right)+O_{\mathbb{P}}\left(\widehat{\lambda}_{n}\mathcal{J}\left(\widetilde{\beta}_{K_{n}}\right)\right),\quad\text{as}\
n\to\infty.$

Theorem 2 presents the prediction error as a function of three quantities,
which respectively represent the variance, the squared approximation bias and
the regularization bias. This decomposition is the same as the decomposition
obtained by Kalogridis and Van Aelst (2023) for functional linear regression
estimators. However, as discussed earlier, our assumptions with respect to
$\beta_{0}$ and the finite-dimensional distributions of $X$ are significantly
weaker than the assumptions of those authors.

Examining the prediction error decomposition in detail, we see that the
variance term $K_{n}\log n/n$ depends only on the size of the approximating
subspace, $\Theta_{K_{n}}$, but not its type. This situation has a well-known
parallel in non-parametric regression (Eggermont and LaRiccia, 2009, Chapter
15). The $\log n$ term is not usually encountered in the non-parametric
setting and appears herein due to the more difficult nature of the problem. In
particular, it appears because of the infinite-dimensional predictor $X$.
Contrary to the variance, the approximation bias
$\|\widetilde{\beta}_{K_{n}}-\beta_{0}\|$ does depend on the type of
approximating subspace as well as $\beta_{0}$ itself. To achieve a good rate
of decay of the approximation error, we need to select a subspace that
approximates well the class of functions to which $\beta_{0}$ belongs. For
instance, a subspace of twice continuously differentiable functions, if we
suspect that $\beta_{0}$ is also twice continuously differentiable. Lastly,
the decay of $\widehat{\lambda}_{n}\mathcal{J}(\widetilde{\beta}_{K_{n}})$ to
$0$ essentially depends on the appropriateness of $\mathcal{J}(\cdot)$ as a
measure of roughness on $\Theta_{K_{n}}$.

As a concrete example, let us consider the case wherein $\mathcal{B}([0,1])$
is the space of functions having uniformly bounded derivatives up to order
$r\geq 1$ with $r$th derivative satisfying a Hölder condition of order
$v\in(0,1)$. For $\Theta_{K_{n}}$ we select the $(K_{n}+p)$-dimensional
B-spline subspace of order $p>r$ generated by $K_{n}$ equidistant interior
knots in $[0,1]$ and $\mathcal{J}(\beta)=\|\beta^{(q)}\|^{2}$ for $q<p$. Then,
$\mathcal{J}(\widetilde{\beta}_{K_{n}})=O(1)$ and, as shown in de Boor (2001,
p. 149), $\|\widetilde{\beta}_{K_{n}}-\beta_{0}\|=O(K_{n}^{-r-v})$. Hence,

$\displaystyle\left|\pi(\widehat{\beta}_{n},\beta_{0})\right|^{2}=O_{\mathbb{P}}\left(\frac{K_{n}\log
n}{n}\right)+O_{\mathbb{P}}\left(K_{n}^{-2(r+v)}\right)+O_{\mathbb{P}}\left(\widehat{\lambda}_{n}\right).$

For $K_{n}\asymp n^{1/(2(r+v)+1)}$ and
$\widehat{\lambda}_{n}=O_{\mathbb{P}}(n^{-c})$ with $c\geq 2(r+v)/(2(r+v)+1)$
we obtain the optimal rate
$|\pi(\widehat{\beta}_{n},\beta_{0})|^{2}=O_{\mathbb{P}}(n^{-2(r+v)/(2(r+v)+1)}\log
n)$. Interestingly, this rate of convergence is substantially faster than the
rate $O_{\mathbb{P}}(n^{-2(r+v)/(4(r+v)+1)})$ obtained by Cardot and Sarda
(2005) for their penalized spline likelihood estimator. The difference is
attributable to our use of modern empirical process theory in the proofs of
our results.

##  4 Practical implementation

###  4.1 Computational algorithm

The penalized density power divergence estimator defined in (4) depends on the
choice of the approximating subspace, $\Theta$, its dimension, $K_{n}$, the
tuning parameter $\widehat{\kappa}_{n}$ and the penalty parameter
$\widehat{\lambda}_{n}$. In this section we outline an effective strategy for
their selection, but first we discuss the computation of penalized density
power divergence estimates. To develop our algorithm, we assume with little
loss of generality that the penalty $\mathcal{J}(\beta)$ can be written in
quadratic form, i.e., we can write
$\mathcal{J}(\beta)=\boldsymbol{\beta}^{\top}\mathbf{P}\boldsymbol{\beta}$ for
some positive semi-definite $\mathbf{P}$, with
$\boldsymbol{\beta}\in\mathbbm{R}^{K_{n}}$ the coefficient vector of
$\beta\in\Theta_{K_{n}}$. While many penalties, such as penalties on
integrated squared derivatives, fulfil this assumption, $l_{1}$-type penalties
do not. However, for such penalties we may employ the computationally
convenient local quadratic approximation proposed by Fan and Li (2004) so that
the core idea of our algorithm remains valid.

With the quadratic simplification of the penalty, we solve (4) with a modified
Fisher-scoring procedure the updating steps of which may be reduced to
penalized iteratively reweighted least squares updates, in the manner outlined
by Green and Silverman (1994). In particular, the weights in our case are
given by

$\displaystyle
w_{i}=(1+\widehat{\kappa}_{n})\frac{\mathbb{E}_{p_{i}}\\{f_{p_{i}}^{\widehat{\kappa}_{n}}(Y_{i})|Y_{i}-p_{i}|^{2}|X_{i}\\}}{p_{i}^{2}(1-p_{i})^{2}}\left|\frac{\partial
p_{i}}{\partial\eta_{i}}\right|^{2},\quad(i=1,\ldots,n),$

where $\mathbb{E}_{p_{i}}\\{\cdot\\}$ denotes expectation with respect to a
Bernoulli density with probability of ”success” $p_{i}=H(\eta_{i})$ and the
linear components $\eta_{i}=\alpha+\langle X_{i},\beta\rangle,i=1,\ldots,n$.
At the same time, the vector of ”working” data $\mathbf{z}\in\mathbbm{R}^{n}$
is given by

$\displaystyle
z_{i}=\eta_{i}-\left(1+\widehat{\kappa}_{n}\right)\frac{\partial
p_{i}}{\partial\eta_{i}}\frac{\mathbb{E}_{p_{i}}\\{f_{p_{i}}^{\widehat{\kappa}_{n}}(Y_{i})(Y_{i}-p_{i})/(p_{i}(1-p_{i}))|X_{i}\\}-f_{p_{i}}^{\widehat{\kappa}_{n}}(Y_{i})(Y_{i}-p_{i})/(p_{i}(1-p_{i}))}{w_{i}}.$

The expressions simplify somewhat when the ”canonical” link
$H(x)=e^{x}/(1+e^{x})$ is used, as in that case $\partial
p_{i}/\partial\eta_{i}=p_{i}(1-p_{i})$. We initiate our Fisher scoring
algorithm at the density power divergence estimates obtained with
$\widehat{\kappa}_{n}=2$. For $\widehat{\kappa}_{n}=0$, it is easy to see that
the above formulae reduce to those for penalized likelihood estimators, as
given in Green and Silverman (1994, Chapter 5) highlighting yet again that
density power divergence estimation is an extension of maximum likelihood
estimation.

###  4.2 Tuning and smoothing parameter selection

As discussed previously, after the selection of $\Theta$, the estimator
requires the choice of $\widehat{\kappa}_{n}$, $K_{n}$ and
$\widehat{\lambda}_{n}$ which correspond to the tuning parameter, the
dimensional of the approximating subspace and the penalty parameter,
respectively. Among these, $K_{n}$ seems to be the least critical for the
success of the estimator, as extensive experience with penalized lower rank
estimators has shown that provided that the approximating subspace is rich
enough, but still smaller than the sample size $n$, the dimension makes little
difference. In our experience, a choice such as $K=[\min\\{30,n/4\\}]$, which
ensures at least $4$ observations per basis function and puts a cap at $30$
basis functions, is appropriate for many situations. The number of basis
functions can be increased beyond 30 in highly complex situations, but these
tend to be rather rare in practice. Hence, the rest of this subsection is
dedicated to the selection of $\widehat{\kappa}_{n}$ and
$\widehat{\lambda}_{n}$.

The parameter $\widehat{\kappa}_{n}$ is particularly important as it
determines the trade-off between robustness and efficiency of our penalized
power divergence estimators and as such it is important to select it properly.
Similar to Kalogridis et al. (2023), our strategy of selecting
$\widehat{\kappa}_{n}$ amounts to minimizing an approximation of the mean-
integrated squared error (MISE)
$\mathbb{E}\\{\|\widehat{\beta}_{n}-\beta_{0}\|^{2}\\}$. To clearly show the
dependence of $\widehat{\beta}_{n}$ on $\kappa$ we herein denote it with
$\widehat{\beta}_{n,\kappa}$. Standard calculations show that for each
$\kappa\geq 0$ the MISE can be decomposed as

$\displaystyle\operatorname{MISE}(\kappa)=\left\|\mathbb{E}\\{\widehat{\beta}_{n,\kappa}\\}-\beta_{0}\right\|^{2}+\mathbb{E}\left\\{\left\|\widehat{\beta}_{n,\kappa}-\mathbb{E}\\{\widehat{\beta}_{n,\kappa}\\}\right\|^{2}\right\\},$
(5)

where the first term on the RHS represents the integrated squared bias and the
second term represents the integrated variance of
$\widehat{\beta}_{n,\kappa}$. None of these terms can be computed explicitly,
as we lack knowledge of both $\beta_{0}$ and
$\mathbb{E}\\{\widehat{\beta}_{n,\kappa}\\}$ (recall that
$\widehat{\beta}_{n,\kappa}$ is defined implicitly as the solution of (4)). To
approximate the bias term of $\operatorname{MISE}(\kappa)$, we follow the
strategy of Warwick and Jones (2005) and replace
$\mathbb{E}\\{\widehat{\beta}_{n,\kappa}\\}$ with its unbiased estimator
$\widehat{\beta}_{n,\kappa}$. Moreover, we use a ”pilot” estimator in the
place of the unknown $\beta_{0}$. We have found in our numerical experiments
that that any density power divergence estimator with a large enough tuning
serves is an effective ”pilot” estimator in practice, as it is less
susceptible to outlying observations. Hence we use $\widehat{\beta}_{n,2}$ as
our initial estimator for $\beta_{0}$.

To approximate the variance term, observe first that
$\mathbb{E}\\{\|\widehat{\beta}_{n,\kappa}-\mathbb{E}\\{\widehat{\beta}_{n,\kappa}\\}\|^{2}\\}=\operatorname{Tr}\\{\mathbf{P}_{0}\operatorname{Cov}\\{\widehat{\boldsymbol{\beta}}_{n,\kappa}\\}\\}$
with $\mathbf{P}_{0}$ consisting of
$\langle\theta_{i},\theta_{j}\rangle,i,j=\ldots,K_{n}$. Let us next write

$\displaystyle l_{\kappa}^{\prime}(Y_{i},p_{i}(\alpha,\beta))=\frac{\partial
l_{\kappa}(Y_{i},x)}{\partial x}\bigg{|}_{x=p_{i}(\alpha,\beta)}\frac{\partial
H(x)}{\partial x}\bigg{|}_{x=\eta_{i}},\quad(i=1,\ldots,n),$

for the first derivatives and similarly
$l_{\kappa}^{\prime\prime}(Y_{i},p_{i}(\alpha,\beta))$ for the second
derivatives. Define $n\times n$ matrices
$\mathbf{C}_{\kappa}=\operatorname{diag}\\{|l^{\prime}_{\kappa}(Y_{i},p_{i}(\widehat{\alpha}_{n,\kappa},\widehat{\beta}_{n,\kappa}))|^{2}\\}$
and
$\mathbf{D}_{\kappa}=\operatorname{diag}\\{l^{\prime\prime}_{\kappa}(Y_{i},p_{i}(\widehat{\alpha}_{n,\kappa},\widehat{\beta}_{n,\kappa})\\}$.
A first order Taylor expansion of the score function of (4) yields the
approximation

$\displaystyle\operatorname{Cov}\\{(\widehat{\alpha}_{n,\kappa},\widehat{\boldsymbol{\beta}}_{n,\kappa})\\}\approx\left[\mathbf{B}^{\star\top}\mathbf{D}_{\kappa}\mathbf{B}^{\star}+2\widehat{\lambda}_{n,\kappa}\mathbf{P}\right]^{-1}\mathbf{B}^{\star\top}\mathbf{C}_{\kappa}\mathbf{B}^{\star}\left[\mathbf{B}^{\star\top}\mathbf{D}_{\kappa}\mathbf{B}^{\star}+2\widehat{\lambda}_{n,\kappa}\mathbf{P}^{\star}\right]^{-1},$

where $\mathbf{B}^{\star}$ has a column of ones and $(i,j)$th element $\langle
X_{i},\theta_{j}\rangle,i=1,\ldots,n,j=2\ldots,(K_{n}+1)$ and
$\mathbf{P}^{\star}$ is the penalty matrix $\mathbf{P}$ augmented with a row
and a column of zeroes. An approximation to
$\operatorname{Cov}\\{\widehat{\boldsymbol{\beta}}_{n,\kappa}\\}\\}$ may be
subsequently obtained by selecting the corresponding elements from the larger
covariance matrix
$\operatorname{Cov}\\{(\widehat{\alpha}_{n,\kappa},\widehat{\boldsymbol{\beta}}_{n,\kappa})\\}$.

With the foregoing approximations of the bias and variance we obtain the
approximate mean-squared error (AMISE) given by

$\displaystyle\operatorname{AMISE}_{2}(\kappa)$
$\displaystyle=\left(\widehat{\boldsymbol{\beta}}_{n,\kappa}-\widehat{\boldsymbol{\beta}}_{n,2}\right)^{\top}\mathbf{P}_{0}\left(\widehat{\boldsymbol{\beta}}_{n,\kappa}-\widehat{\boldsymbol{\beta}}_{n,2}\right)+\operatorname{Tr}\left\\{\mathbf{P}_{0}\operatorname{Cov}\\{\widehat{\boldsymbol{\beta}}_{n,\kappa}\\}\right\\},$

where the subscript $2$ indicates that $\widehat{\beta}_{n,2}$ is used in
place of the unknown $\beta_{0}$. As a first step, we select
$\widehat{\kappa}_{n}$ by minimizing $\operatorname{AMISE}_{1}(\kappa)$ over a
grid of $20$ equidistant values in $[0,2]$. This grid includes both penalized
maximum likelihood estimators ($\kappa=0$) as well as more robust estimators
($\kappa>0$). Both bias approximation and the subsequent selection of $\kappa$
depends on the pilot estimator, $\widehat{\beta}_{n,2}$. To reduce this
dependence we further follow the proposal of Basak et al. (2021) and iterate
the selection procedure. That is, after identifying the value of $\kappa$
minimizing $\operatorname{AMISE}_{2}(\kappa)$, $\widehat{\kappa}_{n}$, we use
$\widehat{\beta}_{n,\widehat{\kappa}_{n}}$ as the new pilot estimator and
minimize $\operatorname{AMISE}_{\widehat{\kappa}_{n}}(\kappa)$. This process
is repeated until $\widehat{\kappa}_{n}$ converges. This procedure requires
merely the computation of a number of estimators
$(\widehat{\alpha}_{n,\kappa},\widehat{\beta}_{n,\kappa})$ for different
values of $\kappa$ and, thanks to the rapid convergence of our modified
Fisher-scoring algorithm, the associated computational burden is minimal.

The computation of the estimator for a given value of $\kappa$ requires an
appropriate value of the penalty parameter $\lambda$. To determine $\lambda$,
we propose an appropriate modification of the AIC criterion adopted in
Kalogridis et al. (2023), viz,

$\displaystyle\operatorname{AIC}(\lambda)=2\sum_{i=1}^{n}l_{\kappa}(Y_{i},p_{i}(\widehat{\alpha}_{n,\kappa},\widehat{\beta}_{n,k}))+2\operatorname{Tr}\left\\{\left[\mathbf{B}^{\star\top}\mathbf{D}_{\kappa}\mathbf{B}^{\star}+2\lambda\mathbf{P}^{\star}\right]^{-1}\mathbf{B}^{\star\top}\mathbf{D}_{\kappa}\mathbf{B}^{\star}\right\\}.$

Implementations and illustrative examples of the functional density power
divergence in the R programming language (R Core Team, 2023) are available at
https://github.com/ioanniskalogridis/Robust-and-Adaptive-Functional-Logistic-
Regression.

##  5 Numerical examples

We now examine the practical performance of the proposed penalized density
power divergence via a numerical study. In our numerical experiments we are
primarily interested in assessing how well the coefficient function
$\beta_{0}$ in a variety of settings that include local behaviour of
$\beta_{0}$, noisy and outlying predictors $X_{i}$ and misclassified responses
$Y_{i}$. For this comparison, we consider the following estimators:

  * •

The adaptive density power divergence estimator discussed in the previous
section, denoted by DPD($\widehat{\kappa}_{n}$).

  * •

The adaptive density power divergence estimator corresponding to $\kappa=0$,
abbreviated as ML.

  * •

The density power divergence estimator with fixed $\kappa=1$, abbreviated as
DPD($1$)

  * •

The density power divergence estimator with fixed $\kappa=2$, abbreviated as
DPD($2$).

Recall that the adaptive estimator, DPD($\widehat{\kappa}_{n})$, uses DPD($2$)
as an initial guess for $\beta_{0}$ and the grid of candidate values for
$\widehat{\kappa}_{n}$ is contained in $[0,2]$. It is therefore of interest to
compare how DPD($\widehat{\kappa}_{n})$ performs against the estimators ML,
DPD($1$) and DPD($2$) in practice. For all estimators we have used the popular
cubic B-spline basis with $\mathcal{J}(f)=\|f^{\prime\prime}\|^{2}$. Moreover,
we have used the canonical logit link function in all subsequent computations.

We have generated the predictor curves according to the truncated Karhunen-
Loève decomposition given by

$\displaystyle X(t)=\sum_{j=1}^{50}j^{-1}Z_{j}\sqrt{2}\sin((j-1/2)\pi t),\quad
t\in[0,1],$ (6)

where the $Z_{j}$ are independent standard Gaussian random variables. To deal
with these curves practically we have discretized them in 200 equidistant
points $t_{j}$ within the $[0,1]$-interval and used Riemann sums to compute
inner products and norms whenever required. The predictor variables were
combined with each of the following three coefficient functions:

  1. 1.

$\beta_{1}(t)=3(t-0.3)^{2}+1$

  2. 2.

$\beta_{2}(t)=3\sin(3.4t^{2})$

  3. 3.

$\beta_{3}(t)=-\sin(5t/1.2)/0.5-1$.

The binary responses $Y_{i}$ were then generated from a Bernoulli distribution
with probability of ”success” equal to $H(\langle X,\beta\rangle)$ where
$H(x)=e^{x}/(1+e^{x})$. The coefficient functions are increasingly complex, as
$\beta_{2}$ possesses more local characteristics than $\beta_{1}$ and,
likewise, $\beta_{3}(t)$ exhibits more local behaviour than $\beta_{2}(t)$.

(a)

(b)

Figure 1: Two representative samples of simulated curves from clean and
contaminated $(\epsilon=0.02)$ data on the left and right respectively.

Next to the clean data generation described above, we have also perturbed a
number of observations $(X_{i},Y_{i})$ in the following manner. For
$\epsilon\in\\{0.01,0.02,0.05\\}$ we have replaced an $\epsilon$-fraction of
the $X_{i}$ with $5X_{i}$ and we have also replaced the corresponding $Y_{i}$
with $1-Y_{i}$ thereby changing an $\epsilon$-fraction of the responses from
$0$ to $1$ and vice versa. This contamination scenario corresponds to the
introduction of bad leverage points into the data and it can be expected a
priori that the performance of all estimators will be affected by these
outlying observations. For a better understanding of the effect of the
multiplication by $5$ on the shape and amplitude of the curves, Figure 1
presents two representative samples of curves with $\epsilon=0$ (clean data)
and $\epsilon=0.02$ (mild contamination).

For each coefficient function $\beta$ and level of contamination $\epsilon$,
we evaluate the performance of the estimators according to the mean squared
error given by

$\displaystyle\operatorname{MSE}=\frac{1}{200}\sum_{j=1}^{200}\left|\widehat{\beta}_{n}(t_{j})-\beta(t_{j})\right|^{2},$

which is an approximation to the $\mathcal{L}^{2}([0,1])$-distance. Since the
MSE is bounded below by zero but unbounded above, its distribution tends to be
right skewed and heavy-tailed. For this reason we have found that the median
is an overall better measure of centrality than the mean. Therefore, Table 1
below reports median MSEs (Median) for the four competing estimators along
with their bootstrapped standard errors (BSE) based on 1000 simulated datasets
with $n=400$.

|  | DPD($\widehat{\kappa}_{n})$ | ML | DPD(1) | DPD(2)
---|---|---|---|---|---
| $\epsilon$ | Median | BSE | Median | BSE | Median | BSE | Median | BSE
$\beta_{1}$ | 0 | 0.389 | 0.024 | 0.385 | 0.031 | 0.482 | 0.032 | 0.401 | 0.021
0.01 | 0.504 | 0.027 | 2.398 | 0.337 | 0.726 | 0.056 | 0.506 | 0.025
0.02 | 0.714 | 0.041 | 8.654 | 1.179 | 1.263 | 0.095 | 0.723 | 0.047
0.05 | 2.313 | 0.139 | 6.670 | 0.888 | 5.327 | 0.284 | 2.322 | 0.142
$\beta_{2}$ | 0 | 1.180 | 0.021 | 1.176 | 0.025 | 1.307 | 0.021 | 1.195 | 0.019
0.01 | 1.347 | 0.029 | 5.292 | 0.810 | 1.642 | 0.052 | 1.358 | 0.034
0.02 | 1.487 | 0.040 | 20.19 | 3.736 | 1.932 | 0.098 | 1.505 | 0.040
0.05 | 2.929 | 0.129 | 30.69 | 3.925 | 14.38 | 1.908 | 2.927 | 0.117
$\beta_{3}$ | 0 | 1.038 | 0.027 | 1.040 | 0.023 | 1.160 | 0.038 | 1.062 | 0.025
0.01 | 1.064 | 0.029 | 2.520 | 0.266 | 1.196 | 0.045 | 1.096 | 0.024
0.02 | 1.278 | 0.039 | 10.43 | 1.373 | 1.797 | 0.095 | 1.302 | 0.040
0.05 | 2.915 | 0.180 | 11.15 | 1.617 | 8.473 | 2.057 | 2.946 | 0.195

Table 1: Median and boostrapped standard errors (obtained from 10000 bootstrap
replications) of the mean-squared errors for the competing estimators over
1000 datasets of size $n=400$. Best median performances are in bold.

There are several interesting observations emerging from Table 1. It may be
immediately seen that while penalized maximum likelihood estimators can be
effective in estimating the true coefficient function in the absence of
contamination, their performance greatly deteriorates even with even a small
fraction of outliers. In particular, even for the smallest level of
contamination the performance of the penalized ML estimator decreases 2 or
even 3-fold. The lack of resistance of ML based estimators needs to be
contrasted with the stability of density power divergence estimators with
positive tuning parameters. The performance of these estimators almost matches
the performance of ML based estimators in clean data and barely deteriorates
for $\epsilon\in\\{0.01,0.02\\}$. Higher levels of contamination do take their
toll on the performance of density power divergence estimators, but clearly
not to the same extent as with ML based estimators. As a means of visualizing
the difference in practical performance of ML-based and density power
divergence estimators, Figure 2 presents the 1000 estimates for $\beta_{1}(t)$
of the ML and DPD($\widehat{\kappa}_{n})$ procedures in clean data as well as
in mildly contaminated data on the first and second rows respectively.

(a)

(b)

(c)

(d)

Figure 2: Top row: $1000$ ML and DPD($\widehat{\kappa}_{n})$-estimates for
$\beta_{1}(t)$ with $\epsilon=0$ on the left and right panels respectively.
Bottom row: $1000$ ML and DPD($\widehat{\kappa}_{n})$-estimates for
$\beta_{1}(t)$ with $\epsilon=0.01$ on the left and right panels respectively.
The solid black lines depict the true coefficient function $\beta_{1}(t)$.

Comparing the robust estimators DPD($\widehat{\kappa}_{n}$), DPD(1) and DPD(2)
in detail reveals that the adaptive nature of the
DPD($\widehat{\kappa}_{n}$)-estimator allows for higher efficiency than their
competitors in clean data without sacrificing robustness whenever outlying
observations are present among the data. In fact, the performance of
DPD($\widehat{\kappa}_{n}$) is similar to that of DPD(2) under high levels of
contamination and substantially better than the performance of the less robust
DPD(1)-estimator. The reason for this superior performance may be traced to
the construction of DPD($\widehat{\kappa}_{n})$-estimators, which ensures that
the tuning parameter is selected in such a way that the mean-integrated
squared error is approximately minimized. This criterion tends to yield a
large value of $\kappa$ in the presence of contamination and a smaller value
of $\kappa$ in clean data thereby ensuring the good all-round performance of
DPD($\widehat{\kappa}_{n}$)-estimators.

##  6 Application: Gait Analysis

It is frequently of interest to determine whether an individual walks normally
or her walk is hindered due to an underlying reason, for example, pain in the
abdomen or wounded toes. The present dataset consists of $166$ observations on
individuals performing a walk cycle of a few seconds during which the position
of the toes in the vertical axis is recorded via a motion sensor at $343$
points. These trajectories are depicted in the left panel of Figure 3. An
indicator variable with values of $1$ and $0$ depending on whether the walk is
normal or not accompanies these trajectories. The goal of this study is to
determine the probability of a normal walk based on the position of the toes
during this cycle. The dataset is freely available under the name
”ToeSegmentation2” in the UCR Time Series Classification Archive (Dau et al.,
2018).

(a)

(b)

Figure 3: Left: Position of the toes of $166$ subjects in the vertical axis
over a gait cycle. Right: Trajectories of observations with large Anscombe
residuals according to the DPD($\widehat{\kappa}_{n})$-estimates.

In most real-data situations, physiological differences among individuals can
be expected to lead to a number of outlying observations in the form of
vertical outliers, that is, observations that do not quite follow the model.
In our case, these may be either individuals that exhibit a normal walking
pattern yet suffer from pain or discomfort or individuals whose walk is
abnormal despite the fact that they are otherwise healthy. In order to
demonstrate the effects of such observations on non-robust estimators, we have
computed both the penalized B-spline based maximum likelihood estimator
proposed by Cardot and Sarda (2005) with the canonical logit link and the
analogous adaptive density power divergence estimator.

The coefficient function estimates are depicted in the left panel of Figure 4
in solid blue and dashed red lines respectively. The panel immediately reveals
a striking difference between these two estimates, in that while the
DPD($\widehat{\kappa}_{n})$-estimates exhibit a number of peaks, most notably
around the 100th and 300th points within the cycle, the maximum likelihood
estimates form what is essentially a straight line, which indicates severe
oversmoothing. As a result of this oversmoothing, the maximum likelihood
estimator is unable to identify the points of the cycle that are influential
in the determination of whether one’s walking is normal or abnormal. In view
of our prior discussion, it may be conjectured that a number of outlying
observations have exerted undue influence on both the maximum likelihood
estimates themselves and the selected penalty parameter, which in this case is
carried out in a non-resistant manner.

In order to ascertain whether some observations are indeed influential on the
ML estimates, we may look at the residuals produced by the resistant
DPD($\widehat{\kappa}_{n})$-estimates for which the algorithm described in
Section 4 selects $\widehat{\kappa}_{n}=1.9$ indicating the presence of
several outlying observations. To detect these observations we may look at the
Anscombe residuals (McCullagh and Nelder, 1983, p. 29), which for the
Bernoulli distribution are given by

$\displaystyle
r_{A,i}=\frac{\operatorname{IB}(Y_{i},2/3,2/3)-\operatorname{IB}(\widehat{\mu}_{i},2/3,2/3)}{\widehat{\mu}_{i}^{1/6}(1-\widehat{\mu}_{i})^{1/6}},\quad(i=1,\ldots,n),$

where $\operatorname{IB}(x,a,b)=\int_{0}^{x}t^{a-1}(1-t)^{b-1}dt$ and
$\widehat{\mu}_{i}$ is $i$th fitted value, i.e.,
$\widehat{\mu}_{i}=H(\widehat{\alpha}_{n}+\int_{1}^{343}X_{i}(t)\widehat{\beta}_{n}(t)dt)$.
The Anscombe residuals follow the standard Gaussian distribution much more
closely than their Pearson counterparts, hence we may apply conventional
outlying detection rules. In particular, we may classify an observation as an
outlier if $|r_{A,i}|\geq 2$. With this rule, the
DPD($\widehat{\kappa}_{n}$)-estimates point towards 19 outlying observations,
$13$ of which have Anscombe residuals larger than $10$ and can thus be
considered extreme outliers. The trajectories of the $19$ detected outlying
observations are plotted in the right panel of Figure 4. Interestingly, $18$
of these observations are classified as normal walk which indicates that in
the vast majority of cases it is individuals with underlying issues that are
misclassified as walking normally and not the other way around.

As a sensitivity check, one may decide to remove the detected outlying
observations from our data and recompute the estimates. Doing so results in
the revised estimates presented in the right panel of Figure 4. The panel
reveals that, while the DPD($\widehat{\kappa}_{n}$)-estimates have undergone
minimal adjustment, the ML-estimates have markedly changed and are now
completely identical with the DPD($\widehat{\kappa}_{n}$)-estimates. We may
thus conclude that the presence of outliers have negatively impacted the non-
resistant ML estimates, but not the much more resistant power density
divergence estimates, which yield reliable estimates both in the presence and
absence of outlying observations. Furthermore, as these estimates are much
less drawn towards outlying observations, they can also be used for the
purpose of outlier detection allowing us to identify influential observations
that would have been otherwise missed by ML-based estimates.

(a)

(b)

Figure 4: Left: Coefficient function estimates based on the full dataset from
the DPD($\widehat{\kappa}_{n}$) and ML estimators in solid blue and dashed red
lines respectively. Right: Coefficient function estimates obtained upon
removing observations with large Anscombe residuals according to the
DPD($\widehat{\kappa}_{n}$)-estimates.

##  7 Concluding remarks

This paper introduces a family of estimators for the functional logistic
regression model that serves as a more balanced alternative to popular
maximum-likelihood based estimators, whose lack of robustness makes them
particularly vulnerable to outlying observations. The proposed family of
estimators can achieve high efficiency in clean data and robustness against
outlying observations through appropriate tuning and, what is more, we have
proposed an algorithm that selects the tuning in an automatic fashion so that
the applied researcher only needs to select the approximating subspace and
penalty functional. The proposed estimators possess strong theoretical
properties in that they are uniformly consistent and can achieve high rates of
convergence with respect to the prediction error even with random tuning
parameters. These properties make the proposed methodology particularly well-
suited for the analysis of many complex datasets commonly encountered
nowadays, such as the gait dataset of Section 6.

There is a number of interesting and practically useful extensions we aim to
consider in future work, most notable of which is the generalization to other
GLMs. While our methodology based on penalized density power divergence can be
readily applied regardless of which distribution in the exponential family
that the response follows, the theoretical properties established herein are
specific to the Bernoulli distribution and do not immediately extend to other
GLMs. These theoretical properties likely remain true under the right set of
conditions and the investigation of such conditions is, in our opinion, a
worthwhile endeavour. Our work can also be extended to more versatile
distributions of the exponential family involving also a scale parameter, such
as the general gamma and log-normal distributions which are frequently used to
model claims in the insurance industry. Due to the frequent presence of
outlying observations in such datasets, the use of robust methods is well-
warranted and we are confident that penalized density power divergence will
likewise lead to computationally feasible estimators with strong theoretical
properties.

## Acknowledgements

The author gratefully acknowledges the support by the Research Foundation-
Flanders (project 1221122N).

##  8 Appendix: Proofs of the theoretical results

######  Lemma 1 (Bounds on the density power divergence).

Let h and g denote Bernoulli densities with probabilities of ”success” equal
to $p_{h}$ and $p_{g}$ respectively. For every $\kappa_{0}\in(0,1]$ we have

$\displaystyle 0\leq d_{\kappa_{0}}(h,f)\leq 1+\frac{1}{\kappa_{0}},$

and the upper bound is attained if, and only if, $|p_{f}-p_{h}|=1$.

###### Proof.

The lower bound follows by the arithmetic-geometric mean (AM-GM) inequality,
as the densities are nonnegative and for every $y\in\\{0,1\\}$ we have

$\displaystyle
h(y)f^{\kappa_{0}}(y)=\left\\{h^{1+\kappa_{0}}(y)\right\\}^{\frac{1}{1+\kappa_{0}}}\left\\{f^{1+\kappa_{0}}(y)\right\\}^{\frac{\kappa_{0}}{1+\kappa_{0}}}\leq\frac{1}{1+\kappa_{0}}h^{1+\kappa_{0}}(y)+\frac{\kappa_{0}}{1+\kappa_{0}}f^{1+\kappa_{0}}(y).$

The inequality is strict unless $h=f$ on $\\{0,1\\}$. Multiplying by
$(\kappa_{0}+1)/\kappa_{0}>0$ yields the result. For the upper bound, write

$\displaystyle
d_{\kappa_{0}}(h,f)=\sum_{y\in\\{0,1\\}}\\{f(y)-h(y)\\}f^{\kappa_{0}}(y)+\frac{1}{\kappa_{0}}\sum_{y\in\\{0,1\\}}\\{h^{\kappa_{0}}(y)-f^{\kappa_{0}}(y)\\}h(y),$

and observe that

$\displaystyle d_{\alpha_{0}}(h,f)$
$\displaystyle\leq\left|p_{h}-p_{f}\right|\left|(1-p_{f})^{\kappa_{0}}-p_{f}^{\kappa_{0}}\right|+\frac{1}{\kappa_{0}}\sum_{y\in\\{0,1\\}}\left|h^{\kappa_{0}}(y)-f^{\kappa_{0}}(y)\right|h(y)$
$\displaystyle\leq 1+\frac{1}{\kappa_{0}},$

by the reverse triangle inequality and the facts that $|p_{h}-p_{f}|\leq 1$
and $|h^{\kappa_{0}}(y)-f^{\kappa_{0}}(y)|\leq 1$. To verify that the upper
bound is attainable for $|p_{f}-p_{h}|=1$ notice that in that case it must be
that either $p_{f}=0$ and $p_{h}=1$ or $p_{f}=1$ and $p_{h}=1$. Simple algebra
shows that the above inequalities become in both cases equalities and the
reverse implication is also true.

∎

######  Lemma 2 (Fisher consistency).

For any $\kappa_{0}>0$ define

$\displaystyle
M(\beta,\kappa_{0})=\mathbb{E}\\{d_{\kappa_{0}}(f_{p_{1}(\beta_{0})},f_{p_{1}(\beta)})\\},\quad
g\in\mathcal{B}([0,1]),$

with $\beta_{0}$ denoting the true coefficient function. Under assumptions
(A2) and (A5), $M(\beta,\kappa_{0})\geq 0$ and $M(\beta,\kappa_{0})=0$ if, and
only if, $\beta=\beta_{0}$.

###### Proof.

Observe initially that, by Lemma 1, $M(\beta,\kappa_{0})\geq 0$ for all
$\beta\in\mathcal{B}([0,1])$ as expectation is a monotone operation. For
$\beta\neq\beta_{0}$ let $A_{\beta}=\\{\omega\in\Omega:\langle
X_{1}(\omega),\beta-\beta_{0}\rangle=0\\}$. By the measurability of the map
$X_{1}:\Omega\to\mathcal{L}^{2}([0,1])$, we have $A_{\beta}\in\mathcal{A}$
and, by (A2), $\mathbb{P}(A_{\beta}^{c})>0$. Thus,

$\displaystyle M(\beta,\kappa_{0})$
$\displaystyle=\int_{\Omega}d_{\kappa_{0}}(f_{p_{1}(\beta_{0})},f_{p_{1}(\beta)})d\mathbb{P}$
$\displaystyle=\int_{A_{\beta}}d_{\kappa_{0}}(f_{p_{1}(\beta_{0})},f_{p_{1}(\beta)})d\mathbb{P}+\int_{A_{\beta}^{c}}d_{\kappa_{0}}(f_{p_{1}(\beta_{0})},f_{\beta_{1}(g)})d\mathbb{P}$
$\displaystyle=\int_{A_{\beta}^{c}}d_{\kappa_{0}}(f_{p_{1}(\beta_{0})},f_{p_{1}(\beta)})d\mathbb{P}$
$\displaystyle>0,$

as, by (A2), $d_{\kappa_{0}}(f_{p_{1}(\beta_{0})},f_{p_{1}(\beta)})$ vanishes
on $A_{\beta}$ and $d_{\kappa_{0}}(f_{p_{1}(\beta_{0})},f_{p_{1}(\beta)})>0$
on $A_{\beta}^{c}$. But, as $\mathbb{P}(A_{\beta}^{c})>0$, integration on
$A_{\beta}^{c}$ preserves the strict inequality. The proof is complete.

∎

For each fixed $\kappa_{0}>0$ and $\beta\in\mathcal{L}^{2}([0,1])$, let
$M_{n}(\beta,\kappa_{0})$ denote the empirical version of
$M(\beta,\kappa_{0})$, that is,

$\displaystyle
M_{n}(\beta,\kappa_{0})=\frac{1}{n}\sum_{i=1}^{n}\left[\sum_{y\in\\{0,1\\}}f^{1+\kappa_{0}}_{p_{i}(\beta)}(y)-\left(1+\frac{1}{\kappa_{0}}\right)f_{p_{i}(\beta)}^{\kappa_{0}}(Y_{i})+\frac{1}{\kappa_{0}}\sum_{y\in\\{0,1\\}}f_{p_{i}(\beta_{0})}^{1+\kappa_{0}}(y)\right]$
(7)

Notice that $M_{n}(\beta,\kappa_{0})$ is the average of bounded i.i.d. random
variables, hence, by the weak law of large numbers (WLLN),
$M_{n}(\beta,\kappa_{0})\xrightarrow{\mathbb{P}}M(\beta,\kappa_{0})$. Lemma 3
below provides a uniform law according to which
$M_{n}(\beta,\kappa_{0})-M(\beta,\kappa_{0})\xrightarrow{\mathbb{P}}0$
uniformly over $\beta\in\Theta_{K_{n}}$, as $n\to\infty$.

######  Lemma 3.

Suppose that (A2) and (A4) are satisfied. Then, for every
$0<\kappa_{0}<\infty$,

$\displaystyle\sup_{\beta\in\Theta_{K_{n}}}|M_{n}(\beta,\kappa_{0})-M(\beta,\kappa_{0})|\xrightarrow{\mathbb{P}}0.$

###### Proof.

By the triangle inequality and the WLLN, it suffices to establish the uniform
convergence for the classes of functions on
$\mathcal{L}^{2}([0,1])\times\\{0,1\\}$ given by

$\displaystyle\mathcal{F}_{n}$
$\displaystyle=\left\\{f^{\kappa_{0}}_{H(\langle
x,\beta\rangle)}(y),\beta\in\Theta_{K_{n}}\right\\}$
$\displaystyle\mathcal{F}^{\prime}_{n}$
$\displaystyle=\left\\{\sum_{y\in\\{0,1\\}}f^{1+\kappa_{0}}_{H(\langle
x,\beta\rangle)}(y),\beta\in\Theta_{K_{n}}\right\\}.$

Here, as in Section 2, $f_{H(\langle x,\beta\rangle)}(y)$ denotes a Bernoulli
density with probability of ”success” $H(\langle x,\beta\rangle)$.

Let $\mathbb{P}_{n}$ denote the empirical measure of
$(X_{i},Y_{i}),\ldots,(X_{n},Y_{n})$. We first show that $\mathcal{F}_{n}$ is
a Glivenko-Cantelli class of functions (van der Vaart, 1998, p. 269) so that

$\displaystyle\sup_{\beta\in\Theta_{K_{n}}}\left|\int
f^{\kappa_{0}}_{H(\langle x,\beta\rangle)}(y)d\mathbb{P}_{n}-\int
f^{\kappa_{0}}_{H(\langle
x,\beta\rangle)}(y)d\mathbb{P}\right|\xrightarrow{\mathbb{P}}0,$ (8)

To prove (8), we develop a bound on the $\epsilon$-covering number of
$\mathcal{F}_{n}$ in the $L_{1}(\mathbb{P}_{n})$-norm,
$N_{1}(\epsilon,\mathcal{F}_{n},\mathbb{P}_{n})$. Note that
$N_{1}(\epsilon,\mathcal{F}_{n},\mathbb{P}_{n})$ is random, as it depends on
the empirical measure $\mathbb{P}_{n}$. To bound this covering number,
consider the set of real-valued functions on $\mathcal{L}^{2}([0,1])$ given by

$\displaystyle\mathcal{G}_{n}=\left\\{\langle
x,\beta\rangle,\beta\in\Theta_{K_{n}}\right\\}.$

Clearly, $\mathcal{G}_{n}$ is at most a $K_{n}$-dimensional vector space of
functions on $\mathcal{L}^{2}([0,1])$ and is therefore a Vapnik-Chervonenkis
(VC) class of functions with VC index bounded above by $K_{n}+2$, see, e.g.,
(van der Vaart, 1998, pp. 275–276). The class of functions
$H(\mathcal{G}_{n})$ is also VC with the same index, by the monotonicity of
$H$ assumed in (A2) and Lemma 9.9 (viii) in (Kosorok, 2008). Furthermore, the
class of functions $H(\mathcal{G}_{n})$ has envelope equal to $1$, hence, by
Lemma 19.15 in van der Vaart (1998), there exists a universal constant $C$
such that

$\displaystyle\log(N_{1}(\epsilon,H(\mathcal{G}_{n}),\mathbb{P}_{n}))\leq
C(K_{n}+1)\log\left(\frac{1}{\epsilon}\right),$

for all small $\epsilon>0$. Observe next that the function
$f_{p}(y)=p^{y}(1-p)^{1-y},y\in\\{0,1\\}$, is Lipschitz with respect to $p$
with Lipschitz constant bounded above by $1$ so that the same bound holds for
the set of functions on $\mathcal{L}^{2}([0,1])\times\\{0,1\\}$ given by
$\\{f_{H(\langle x,\beta\rangle)}(y),\beta\in\Theta_{K_{n}}\\}$. To derive
from this a bound on the covering number of $\mathcal{F}_{n}$ in the
$L_{1}(\mathbb{P}_{n})$-norm, simply note that the function $x\mapsto
x^{\kappa_{0}}$ is increasing and bounded by $1$ for all $0\leq x\leq 1$.
Therefore, by Lemma 19.15 of van der Vaart (1998) yet again,

$\displaystyle\log(N_{1}(\epsilon,\mathcal{F}_{n},\mathbb{P}_{n}))\leq
C(K_{n}+1)\log\left(\frac{1}{\epsilon}\right),$

for small $\epsilon>0$. By (A4), it now follows that, for such $\epsilon>0$,
$\log(N_{1}(\epsilon,\mathcal{F})_{n},\mathbb{P}_{n}))/n\xrightarrow{\mathbb{P}}0$.
The uniform convergence in (8) now follows from van de Geer (2000, Exercise
3.6) with $b_{n}=2$ in that result, as the functions in $\mathcal{F}$ are
uniformly bounded by $1$ for all $\beta\in\Theta_{K_{n}}$.

To complete the proof, we need to likewise establish that the set of functions
$\mathcal{F}^{\prime}$ is Glivenko-Cantelli. For this we may consider the
cases $y=0$ and $y=1$ separately. The proof may be devised along similar
lines: first bound the covering number of $H(\mathcal{G}_{n})$ and then notice
that the functions $x\mapsto x^{k_{0}}$ and $x\mapsto(1-x)^{k_{0}}$ are
monotone and bounded for $0\leq x\leq 1$. It follows that there exists a $C>0$
such that

$\displaystyle\log(N_{1}(\epsilon,\mathcal{F}_{n}^{\prime},\mathbb{P}_{n}))\leq
C(K_{n}+1)\log\left(\frac{1}{\epsilon}\right),$

for all small $\epsilon>0$. Under (A4), the result follows again by an appeal
to van de Geer (2000, Exercise 3.6). ∎

######  Lemma 4.

Suppose that assumptions (A1)–(A5) are satisfied. Then, for every
$\kappa_{0}>0$,

$\displaystyle
M(\widehat{\beta}_{n},\kappa_{0})\xrightarrow{\mathbb{P}}M(\beta_{0},\kappa_{0}).$

###### Proof.

By Lemma 1, $\beta_{0}$ uniquely minimizes $M(\beta,\kappa_{0})$ over all
$\beta\in\mathcal{B}([0,1])$. Since
$\widehat{\beta}_{n}\in\mathcal{B}([0,1])\subset\mathcal{L}^{2}([0,1])$ for
all $n\in\mathbbm{N}$, we clearly have

$\displaystyle
0<M(\widehat{\beta}_{n},\kappa_{0})-M(\beta_{0},\kappa_{0})=I+II+III,$ (9)

where

$\displaystyle
I=M(\widehat{\beta}_{n},\kappa_{0})-M_{n}(\widehat{\beta}_{n},\kappa_{0}),\
II=M_{n}(\widehat{\beta}_{n},\kappa_{0})-M_{n}(\widehat{\beta}_{n},\widehat{\kappa}_{n}),\
III=M_{n}(\widehat{\beta}_{n},\widehat{\kappa}_{n})-M(g_{0},\kappa_{0}).$

Our proof consists of showing that each one of the terms in RHS of (9)
converge to $0$ in probability. Starting with $I$, by definition of
$\widehat{\beta}_{n}$ we have $\widehat{\beta}_{n}\in\Theta_{K_{n}}$, whence,
by Lemma 2,

$\displaystyle\left|I\right|\leq\sup_{\beta\in\Theta_{K_{n}}}|M_{n}(\beta,\kappa_{0})-M(\beta,\kappa_{0})|\xrightarrow{\mathbb{P}}0.$

To establish the convergence of $II$ to $0$, notice that, by (A1), we can
assume that $\widehat{\kappa}_{n}\in(\kappa_{0}/2,3\kappa_{0}/2)$ for all
large $n$ with high probability. By the mean-value theorem, for every
$x\in\mathcal{L}^{2}([0,1])$ and $y\in\\{0,1\\}$ there exists a
$\widetilde{\kappa}_{n}=\widetilde{\kappa}_{n}(x,y)$ such that
$|\widetilde{\kappa}_{n}-\kappa_{0}|\leq|\widehat{\kappa}_{n}-\kappa_{0}|<\kappa_{0}/2$
and

$\displaystyle f^{\widehat{\kappa}_{n}}_{H(\langle
x,\widehat{\beta}_{n}\rangle)}(y)-f^{\kappa_{0}}_{H(\langle
x,\widehat{\beta}_{n}\rangle)}(y)=(\widehat{\kappa}_{n}-\kappa_{0})f^{\widetilde{\kappa}_{n}}_{H(\langle
x,\widehat{\beta}_{n}\rangle)}(y)\log\left(f_{H(\langle
x,\widehat{\beta}_{n}\rangle)}(y)\right).$

Now,

$\displaystyle\left|f^{\widetilde{\kappa}_{n}}_{H(\langle
x,\widehat{\beta}_{n}\rangle)}(y)\log\left(f_{H(\langle
x,\widehat{\beta}_{n}\rangle)}(y)\right)\right|$
$\displaystyle\leq\sup_{|\kappa-\kappa_{0}|<\kappa_{0}/2,\
p\in[0,1]}\left|f^{\kappa}_{p}(y)\log\left(f_{p}(y)\right)\right|$
$\displaystyle\leq\sup_{|\kappa-\kappa_{0}|<\kappa_{0}/2,\
p\in[0,1]}(1-p)^{\kappa}\left|\log(1-p)\right|+\sup_{|\kappa-\kappa_{0}|<\kappa_{0}/2,\
p\in[0,1]}p^{\kappa}\left|\log(p)\right|$ $\displaystyle\leq\sup_{\
p\in[0,1]}(1-p)^{\kappa_{0}/2}\left|\log(1-p)\right|+\sup_{\
p\in[0,1]}p^{3\kappa_{0}/2}\left|\log(p)\right|$ $\displaystyle\leq M,$

for some finite $D=D(\kappa_{0})$. Applying this together with the facts that
$f_{p}(y)\leq 1$ and $\sum_{y\in\\{0,1\\}}f_{p}(y)=1$ for every $p\in[0,1]$
yields

$\displaystyle|II|$
$\displaystyle\leq\frac{1}{n}\sum_{i=1}^{n}\left[\sum_{y\in\\{0,1\\}}\left|f_{H(\langle
x,\widehat{\beta}_{n}\rangle)}^{\widehat{\kappa}_{n}}(y)-f_{H(\langle
x,\widehat{\beta}_{n}\rangle)}^{\kappa_{0}}(y)\right|f_{H(\langle
x,\widehat{\beta}_{n}\rangle)}(y)+\frac{|\widehat{\kappa}_{n}-\kappa_{0}|}{\kappa_{0}\widehat{\kappa}_{n}}f_{H(\langle
X_{i},\widehat{\beta}_{n}\rangle)}^{\widehat{\kappa}_{n}}(Y_{i})\right]$
$\displaystyle\quad+\left(1+\frac{1}{\widehat{\kappa}_{n}}\right)\frac{1}{n}\sum_{i=1}^{n}\left|f_{H(\langle
X_{i},\widehat{\beta}_{n}\rangle)}^{\widehat{\kappa}_{n}}(Y_{i})-f_{H(\langle
X_{i},\widehat{\beta}_{n}\rangle)}^{\kappa_{0}}(Y_{i})\right|$
$\displaystyle\leq
D\left|\widehat{\kappa}_{n}-\kappa_{0}\right|+\frac{\left|\widehat{\kappa}_{n}-\kappa_{0}\right|}{(\kappa_{0}-\delta)\kappa_{0}}+\left(1+\frac{1}{\kappa_{0}-\delta}\right)D\left|\widehat{\kappa}_{n}-\kappa_{0}\right|,$

for all large $n$ with high probability. Conclude that
$II\xrightarrow{\mathbb{P}}0$.

To complete the proof we now show that $III\xrightarrow{\mathbb{P}}0$. To see
this, recall first that $\widehat{\beta}_{n}$ minimizes
$M_{n}(\beta,\widehat{\kappa}_{n})+\widehat{\lambda}_{n}\mathcal{J}(\beta)$
over $\beta\in\Theta_{K_{n}}$ and the abstract approximation to $\beta_{0}$ in
$\Theta_{K_{n}}$, $\widetilde{\beta}_{K_{n}}$, satisfies
$\widetilde{\beta}_{K_{n}}\in\Theta_{K_{n}}$. Hence, by the non-negativity of
the map $\beta\mapsto\widehat{\lambda}_{n}\mathcal{J}(\beta)$ and (A4), we
obtain

$\displaystyle III$ $\displaystyle\leq
M_{n}(\widetilde{\beta}_{K_{n}},\widehat{\kappa}_{n})+\widehat{\lambda}_{n}\mathcal{J}(\widetilde{\beta}_{K_{n}})-M(\beta_{0},\kappa_{0})$
$\displaystyle=M_{n}(\widetilde{\beta}_{K_{n}},\widehat{\kappa}_{n})-M(\beta_{0},\kappa_{0})+o_{\mathbb{P}}(1)$
$\displaystyle=\left\\{M_{n}(\widetilde{\beta}_{K_{n}},\widehat{\kappa}_{n})-M_{n}(\beta_{0},\kappa_{0})\right\\}+\left\\{M_{n}(\beta_{0},\kappa_{0})-M(\beta_{0},\kappa_{0})\right\\}+o_{\mathbb{P}}(1).$
(10)

By the WLLN,
$M_{n}(\beta_{0},\kappa_{0})\xrightarrow{\mathbb{P}}M(\beta_{0},\kappa_{0})$.
To show the convergence of the first term in the RHS of (8) to $0$ notice
that, by the triangle inequality,

$\displaystyle\left|M_{n}(\widetilde{\beta}_{K_{n}},\widehat{\kappa}_{n})-M_{n}(\beta_{0},\kappa_{0})\right|\leq\left|M_{n}(\widetilde{\beta}_{K_{n}},\widehat{\kappa}_{n})-M_{n}(\widetilde{\beta}_{K_{n}},\kappa_{0})\right|+\left|M_{n}(\widetilde{\beta}_{K_{n}},\kappa_{0})-M_{n}(\beta_{0},\kappa_{0})\right|$
(11)

Our previous argument employed in the treatment of $II$ immediately shows that
the first term in the RHS of (11) is $o_{\mathbb{P}}(1)$, as $n\to\infty$.
Furthermore, for every fixed $\kappa_{0}>0$, the mapping
$[0,1]\to[0,1]:p\mapsto f_{p}^{\kappa_{0}}(y),y\in\\{0,1\\},$ is Lipschitz.
Hence, there exists a finite $D^{\prime}=D^{\prime}(\kappa_{0})$ such that

$\displaystyle\left|f^{\kappa_{0}}_{H(\langle
X_{i},\widetilde{\beta}_{K_{n}}\rangle)}(y)-f^{\kappa_{0}}_{H(\langle
X_{i},\beta_{0}\rangle)}(y)\right|\leq D^{\prime}(\kappa_{0})\left|H(\langle
X_{i},\widetilde{\beta}_{K_{n}}\rangle)-H(\langle
X_{i},\beta_{0}\rangle)\right|.$

With the mean-value theorem, (A2) and (A5), we find

$\displaystyle\left|f^{\kappa_{0}}_{H(\langle
X_{i},\widetilde{\beta}_{K_{n}}\rangle)}(y)-f^{\kappa_{0}}_{H(\langle
X_{i},\beta_{0}\rangle)}(y)\right|$ $\displaystyle\leq
CD^{\prime}(\kappa_{0})\sup_{x\in\mathbbm{R}}H^{\prime}(x)\left\|\widetilde{\beta}_{K_{n}}-\beta_{0}\right\|,$

which leads to

$\displaystyle\left|M_{n}(\widetilde{\beta}_{K_{n}},\kappa_{0})-M_{n}(\beta_{0},\kappa_{0})\right|\leq
c_{0}\left\|\widetilde{\beta}_{K_{n}}-\beta_{0}\right\|,$

for some $c_{0}>0$. By (A4), $\|\widetilde{\beta}_{K_{n}}-\beta_{0}\|\to 0$ as
$K_{n}\to\infty$ so that
$M_{n}(\widetilde{\beta}_{K_{n}},\kappa_{0})-M_{n}(\beta_{0},\kappa_{0})\xrightarrow{\mathbb{P}}0$.
We have thus shown that the RHS of (11) converges to zero in probability.
Taking into account the convergence of $I$ and $II$ to $0$, we see that

$\displaystyle o_{\mathbb{P}}(1)<III\leq o_{\mathbb{P}}(1).$

Consequently, $III=o_{\mathbb{P}}(1)$ and the proof is complete. ∎

###### Proof of Theorem 1.

The proof relies on Lemmas 2 and 4 as well as on Lemma 14.3 and Theorem 2.12
of Kosorok (2008). First, notice that, by Lemma 2, $M(\beta,\kappa_{0})$
possesses a unique minimum at $\beta_{0}$ and by the continuity of the link
function $H$, Lemma 1, (A5) and dominated convergence $M(\beta,\kappa_{0})$ is
continuous on $\mathcal{B}([0,1])$. By Lemma 14.3 of (Kosorok, 2008) it
follows that for any sequence $\beta_{k}\in\mathcal{B}([0,1])$,
$\lim_{k\to\infty}M(\beta_{k},\kappa_{0})=M(\beta_{0},\kappa_{0})$ implies
$\|\beta_{k}-\beta_{0}\|_{\mathcal{B}}\to 0$. By exercise 14.6.2 of (Kosorok,
2008) there exists an increasing function cadlag (right continuous with left
limits) function $f:[0,\infty]\to[0,\infty]$ that satisfies $f(0)=0$, is
continuous at $0$ and $\|\beta-\beta_{0}\|_{\mathcal{B}}\leq
f(|M(\beta,\kappa_{0})-M(\beta_{0},\kappa_{0})|)$ for all
$\beta\in\mathcal{B}([0,1])$. Noting that our estimator
$\widehat{\beta}_{n}\in\mathcal{B}([0,1])$ for all $n\in\mathbbm{N}$, we find

$\displaystyle\left\|\widehat{\beta}_{n}-\beta_{0}\right\|_{\mathcal{B}}\leq
f\left(\left|M(\widehat{\beta}_{n},\kappa_{0})-M(\beta_{0},\kappa_{0})\right|\right).$
(12)

By Lemma 4,
$|M(\widehat{\beta}_{n},\kappa_{0})-M(\beta_{0},\kappa_{0})|\xrightarrow{\mathbb{P}}0$.
Therefore, by the continuity of $f$ at $0$ and the fact that $f(0)=0$, (12)
entails that
$\|\widehat{\beta}_{n}-\beta_{0}\|_{\mathcal{B}}\xrightarrow{\mathbb{P}}0$, as
was to be proved. ∎

We now introduce some useful notation that will be useful in the proof of
Theorem 2. Let $\mathcal{G}$ denote a class of real-valued functions on
$\mathcal{L}^{2}([0,1])\times\mathbb{R}$. For $g\in\mathcal{G}$ we define

$\displaystyle\|g\|_{\infty}=\sup_{x\in\mathcal{L}^{2}([0,1]),y\in\mathbb{R}}|g(x,y)|.$

The covering number in this uniform metric,
$N_{\infty}(\epsilon,\mathcal{G})$, is defined as the smallest value of
$N\in\mathbb{N}$ such that there exists a sequence $\\{g_{j}\\}_{j=1}^{N}$
with the property that

$\displaystyle\sup_{g\in\mathcal{G}}\min_{j=1,\ldots,N}\|g-g_{j}\|_{\infty}\leq\epsilon.$

The corresponding entropy, $H_{\infty}(\epsilon,\mathcal{G})$, is defined as
the logarithm of the covering number, i.e.,
$H_{\infty}(\epsilon,\mathcal{G})=\log(N_{\infty}(\epsilon,\mathcal{G}))$.

For a probability measure $\mathbb{P}$ we also define the bracketing number in
the $\mathcal{L}^{2}(\mathbb{P})$-metric,
$N_{B}(\epsilon,\mathcal{G},\mathbb{P})$, as the smallest value of
$N\in\mathbb{N}$ for which there exist $N$ pairs of functions
$\\{[g_{j}^{L},g_{j}^{U}]\\}$ such that
$\|g_{j}^{U}-g_{j}^{L}\|_{\mathcal{L}^{2}(\mathbb{P})}\leq\epsilon$ for all
$j=1,\ldots,N,$ and such that for each $g\in\mathcal{G}$, there is a
$j=j(g)\in\\{1,\ldots,N\\}$ such that

$\displaystyle g_{j}^{L}(x,y)\leq g(x,y)\leq g_{j}^{U}(x,y).$

The corresponding bracketing entropy,
$H_{B}(\epsilon,\mathcal{G},\mathbb{P})$, is defined as the logarithm of the
bracketing number, i.e.,
$H_{B}(\epsilon,\mathcal{G},\mathbb{P})=\log(N_{B}(\epsilon,\mathcal{G},\mathbb{P}))$.

For convenience throughout the rest of our proofs we use $c_{0}$ to denote
generic positive constants whenever their value is not important for our
argument. Thus, $c_{0}$ may change from appearance to appearance.

######  Lemma 5 (Bracketing entropy).

For $\beta\in\Theta_{K_{n}}$ and
$\kappa\in(\kappa_{0}-\delta_{2},\kappa_{0}+\delta_{2})$ for $\delta_{2}>0$
such that $\kappa_{0}-\delta_{2}>0$ define the real-valued function
$z:\mathcal{L}^{2}([0,1])\times\\{0,1\\}\to\mathbbm{R}$,

$\displaystyle z_{\beta,\kappa}(x,y)=\sum_{y\in\\{0,1\\}}\left\\{f_{H(\langle
x,\widetilde{\beta}_{K_{n}}\rangle)}^{1+\alpha}(y)-f_{H(\langle
x,\beta\rangle)}^{1+\kappa}(y)\right\\}-\left(1+\frac{1}{\kappa}\right)\left\\{f_{H(\langle
x,\widetilde{\beta}_{K_{n}}\rangle)}^{\kappa}(y)-f_{H(\langle
x,\beta\rangle)}^{\kappa}(y)\right\\},$

and the class of functions

$\displaystyle\mathcal{Z}_{n,\delta_{1},\delta_{2}}=\left\\{z_{\beta,\kappa}(x,y),\beta\in\Theta_{K_{n}},\|\beta-\widetilde{\beta}_{K_{n}}\|\leq\delta_{1},|\kappa-\kappa_{0}|<\delta_{2}\right\\}.$

Let $\mathbb{P}_{X,Y}$ denote the probability measure (distribution) induced
by the random variables $(X,Y)\in\mathcal{L}^{2}([0,1])\times\\{0,1\\}$. Then,
for each sufficiently small $\delta_{1}>0$ and $\delta_{2}>0$, there exists a
positive constant $A=A(\delta_{1},\delta_{2})$ such that for all $n\geq n_{0}$
we have

$\displaystyle
H_{B}\left(\epsilon,\mathcal{Z}_{n,\delta_{1},\delta_{2}},\mathbb{P}_{X,Y}\right)\leq
3K_{n}\log\left(1+\frac{A}{\epsilon}\right),\quad\epsilon>0.$

###### Proof.

By Lemma 2.1 of van de Geer (2000), we have

$\displaystyle
H_{B}\left(\epsilon,\mathcal{Z}_{n,\delta_{1},\delta_{2}},\mathbb{P}_{X,Y}\right)\leq
H_{\infty}\left(\epsilon/2,\mathcal{Z}_{n,\delta_{1},\delta_{2}}\right),\quad\epsilon>0.$

Hence it suffices to bound the uniform entropy. By (A5), we may assume without
loss of generality that $\|x\|\leq C$. For any two functions in
$\mathcal{Z}_{n,\delta_{1},\delta_{2}}$, $z_{\beta_{1},\kappa_{1}}(x,y)$ and
$z_{\beta_{2},\kappa_{2}}(x,y)$, we have

$\displaystyle\left|z_{\beta_{1},\kappa_{1}}(x,y)-z_{\beta_{2},\kappa_{2}}(x,y)\right|$
$\displaystyle\leq\left|z_{\beta_{1},\kappa_{1}}(x,y)-z_{\beta_{1},\kappa_{2}}(x,y)\right|+\left|z_{\beta_{1},\kappa_{2}}(x,y)-z_{\beta_{2},\kappa_{2}}(x,y)\right|$
$\displaystyle\leq\sum_{y\in\\{0,1\\}}\left\\{\left|f_{H(\langle
x,\beta_{1}\rangle)}^{1+\kappa_{1}}(y)-f_{H(\langle
x,\beta_{1}\rangle)}^{1+\kappa_{2}}(y)\right|+\left|f_{H(\langle
x,\widetilde{\beta}_{K_{n}}\rangle)}^{1+\kappa_{1}}(y)-f_{H(\langle
x,\widetilde{\beta}_{K_{n}}\rangle)}^{1+\kappa_{2}}(y)\right|\right\\}$
$\displaystyle\quad+\frac{\left|\kappa_{1}-\kappa_{2}\right|}{|\kappa_{0}-\delta|^{2}}+\left|f_{H(\langle
x,\beta_{1}\rangle)}^{\kappa_{1}}(y)-f_{H(\langle
x,\beta_{1}\rangle)}^{\kappa_{2}}(y)\right|+\left|f_{H(\langle
x,\widetilde{\beta}_{K_{n}}\rangle)}^{\kappa_{1}}(y)-f_{H(\langle
x,\widetilde{\beta}_{K_{n}}\rangle)}^{\kappa_{2}}(y)\right|$
$\displaystyle\quad+\sum_{y\in\\{0,1\\}}\left|f_{H(\langle
x,\beta_{1}\rangle)}^{1+\kappa_{2}}(y)-f_{H(\langle
x,\beta_{2}\rangle)}^{1+\kappa_{2}}(y)\right|+\frac{\kappa_{2}+1}{\kappa_{0}-\delta}\left|f_{H(\langle
x,\beta_{1}\rangle)}^{\kappa_{2}}(y)-f_{H(\langle
x,\beta_{2}\rangle)}^{\kappa_{2}}(y)\right|$

For all sufficiently small $\delta_{1}>0$,
$\|\beta-\widetilde{\beta}_{K_{n}}\|<\delta_{1}$ implies that $H(\langle
x,\beta\rangle)$ is bounded away from $0$ for all large $n$. To see this,
first observe that, by (A2), $H(\langle x,\beta_{0}\rangle)$ is bounded away
from $0$ and $1$, as $|\langle x,\beta_{0}\rangle|\leq C\|\beta_{0}\|$ and $H$
is bijective. Additionally,

$\displaystyle\left|H(\langle x,\beta\rangle)-H(\langle
x,\beta_{0}\rangle)\right|$ $\displaystyle\leq\left|H(\langle
x,\beta\rangle)-H(\langle
x,\widetilde{\beta}_{K_{n}}\rangle)\right|+\left|H(\langle
x,\widetilde{\beta}_{K_{n}}\rangle)-H(\langle x,\beta_{0}\rangle)\right|$
$\displaystyle\leq\sup_{x\in\mathbbm{R}}H^{\prime}(x)C\delta_{1}+o(1),$

as $n\to\infty$. Now, for $p$ bounded away from $0$ and $1$, say
$0<p_{\min}\leq p\leq p_{\max}<1$, it is easy to see that the function
$p\mapsto f_{p}^{\kappa}(y)$ is Lipschitz for all
$\kappa\in(\kappa_{0}-\delta_{2},\kappa_{0}+\delta_{2})$ with Lipschitz
constant equal to $(k_{0}+\delta_{2})(1+p_{\max})/(p_{\min}(1-p_{\max}))$.
Therefore, we further find

$\displaystyle\left|z_{\beta_{1},\kappa_{1}}(x,y)-z_{\beta_{2},\kappa_{2}}(x,y)\right|$
$\displaystyle\leq
4\sup_{y\in\\{0,1\\},p\in[0,1],|\kappa-\kappa_{0}|<\delta_{2}}\left|f^{1+\kappa}_{p}(y)\log(f_{p}(y))\right|\left|\kappa_{1}-\kappa_{2}\right|+\frac{\left|\kappa_{1}-\kappa_{2}\right|}{|\kappa_{0}-\delta_{2}|^{2}}$
$\displaystyle\quad+2\sup_{y\in\\{0,1\\},p\in[0,1],|\kappa-\kappa_{0}|<\delta_{2}}\left|f^{\kappa}_{p}(y)\log(f_{p}(y))\right|\left|\kappa_{1}-\kappa_{2}\right|$
$\displaystyle\quad+\sup_{x\in\mathbbm{R}}H^{\prime}(x)C\frac{(1+k_{0}+\delta_{2})(1+p_{\max})}{p_{\min}(1-p_{\max})}\left\|\beta_{1}-\beta_{2}\right\|$
$\displaystyle\leq
c_{0}\left|\kappa_{1}-\kappa_{2}\right|+c_{0}\left\|\beta_{1}-\beta_{\widetilde{K}_{n}}\right\|+c_{0}\left\|\beta_{2}-\beta_{\widetilde{K}_{n}}\right\|$

for some global $c_{0}>0$ depending only on $\delta_{2}$. Conclude that

$\displaystyle
H_{\infty}\left(\epsilon,\mathcal{Z}_{n,\delta_{1},\delta_{2}}\right)\leq
H\left(\epsilon/(2c_{0}),\\{|\kappa-\kappa_{0}|\leq\delta_{2}\\}\right)+2H\left(\epsilon/(4c_{0}),\\{\beta\in\Theta_{K_{n}},\|\beta-\widetilde{\beta}_{K_{n}}\|\leq\delta_{1}\\}\right).$

These two entropy numbers may be bounded by Lemma 2.5 and Corollary 2.6 of van
de Geer (2000) respectively leading to

$\displaystyle
H_{\infty}\left(\epsilon,\mathcal{Z}_{n,\delta_{1},\delta_{2}}\right)\leq\log\left(\frac{8c_{0}\delta_{2}}{\epsilon}+1\right)+2K_{n}\log\left(\frac{16c_{0}\delta_{1}}{\epsilon}+1\right)\leq
3K_{n}\log\left(\frac{A}{\epsilon}+1\right),$

for $A=8c_{0}\delta_{2}\vee 16c_{0}\delta_{1}$. The proof is complete. ∎

###### Proof of Theorem 2.

Observe that, since $\widehat{\beta}_{n}$ minimizes
$M_{n}(\beta,\widehat{\kappa}_{n})+\widehat{\lambda}_{n}\mathcal{J}(\beta)$
over $\beta\in\Theta_{K_{n}}$ and $\widetilde{g}_{K_{n}}\in\Theta_{K_{n}}$, we
have

$\displaystyle
M\left(\widehat{\beta}_{n},\widehat{\kappa}_{n}\right)-M\left(\widetilde{\beta}_{K_{n}},\widehat{\kappa}_{n}\right)\leq
U_{n}\left(\widetilde{\beta}_{K_{n}},\widehat{\beta}_{n},\widehat{\kappa}_{n}\right)+\widehat{\lambda}_{n}\mathcal{J}(\widetilde{\beta}_{K_{n}}),$
(13)

where we have dropped the non-negative term
$\widehat{\lambda}_{n}\mathcal{J}(\widehat{g}_{n})$ and rearranged terms.
Here, $U_{n}(f,g,\alpha):\Theta_{K_{n}}\times\Theta_{K_{n}}\times\mathbbm{R}$
denotes the centered process given by

$\displaystyle
U_{n}(f,g,\alpha)=\\{M_{n}(f,\alpha)-M(f,\alpha\\}-\\{M_{n}(g,\alpha)-M(g,\alpha)\\}.$

Our proof consists of two steps. In the first step we establish a quadratic
lower bound of the LHS of (13) in terms of
$\pi(\widehat{\beta}_{n},\widetilde{\beta}_{K_{n}})$. In particular, we will
show that there exist constants $\eta,L>0$ such that

$\displaystyle
M\left(\widehat{\beta}_{n},\widehat{\kappa}_{n}\right)-M\left(\widetilde{\beta}_{K_{n}},\widehat{\kappa}_{n}\right)\geq\eta\left|\pi(\widehat{\beta}_{n},\widetilde{\beta}_{K_{n}})\right|^{2}-L\left\|\widetilde{\beta}_{K_{n}}-\beta_{0}\right\|\pi(\widehat{\beta}_{n},\widetilde{\beta}_{K_{n}}),$
(14)

for all large $n$ with high probability. In the second step we derive the
modulus continuity of
$U_{n}\left(\widetilde{\beta}_{K_{n}},\widehat{\beta}_{n},\widehat{\kappa}_{n}\right)$
in terms of $\pi(\widehat{\beta}_{n},\widetilde{\beta}_{K_{n}})$ and show that

$\displaystyle
U_{n}\left(\widetilde{\beta}_{K_{n}},\widehat{\beta}_{n},\widehat{\kappa}_{n}\right)=O_{\mathbb{P}}(1)\left\\{\gamma_{n}\pi(\widehat{\beta}_{n},\widetilde{\beta}_{K_{n}})\vee\gamma_{n}^{2}\right\\},$
(15)

where $\gamma_{n}^{2}=K_{n}\log n/n$. In combination, (14) and (15) imply the
result of the theorem. To see this, plug (14) and (15) into (13) and rearrange
to get

$\displaystyle\left|\pi(\widehat{g}_{n},\widetilde{g}_{K_{n}})\right|^{2}\leq
L^{\prime}\left(\gamma_{n}\pi(\widehat{\beta}_{n},\widetilde{\beta}_{K_{n}})\vee\gamma_{n}^{2}\right)+L^{\prime}\left\|\widetilde{\beta}_{K_{n}}-\beta_{0}\right\|\pi(\widehat{\beta}_{n},\widetilde{\beta}_{K_{n}})+L^{\prime}\widehat{\lambda}_{n}\mathcal{J}\left(\widetilde{\beta}_{K_{n}}\right),$
(16)

for some sufficiently large $L^{\prime}$ and all large $n$ with high
probability. Notice then that we only need to prove the theorem for
$\gamma_{n}\pi(\widehat{g}_{n},\widetilde{g}_{K_{n}})\geq\gamma_{n}^{2}$, or,
equivalently, $\pi(\widehat{g}_{n},\widetilde{g}_{K_{n}})\geq\gamma_{n}$,
otherwise the result clearly holds. With this simplification, (16) is an
inequality of the form $x_{0}^{2}\leq bx_{0}+c$ for
$x_{0}=\pi(\widehat{\beta}_{n},\widetilde{\beta}_{K_{n}})\geq 0$,
$b=L^{\prime}(\left\|\widetilde{\beta}_{K_{n}}-\beta_{0}\right\|+\gamma_{n})$
and $c=L^{\prime}\mathcal{J}\left(\widetilde{\beta}_{K_{n}}\right)$. This
means that $x_{0}$ must be less than or equal to the positive root of
$x^{2}-bx-c=0$, that is,

$\displaystyle 0\leq x_{0}\leq\frac{b+\sqrt{b^{2}+4c}}{2}\leq b+\sqrt{c},$

and after substituting for $x_{0}$, $b$ and $c$,

$\displaystyle\pi(\widehat{\beta}_{n},\widetilde{\beta}_{K_{n}})\leq
L^{\prime}\left(\left\|\widetilde{\beta}_{K_{n}}-\beta_{0}\right\|+\gamma_{n}\right)+\sqrt{L^{\prime}\widehat{\lambda}_{n}\mathcal{J}\left(\widetilde{\beta}_{K_{n}}\right)}.$

Squaring and using the inequality $(x+y)^{2}\leq 2x^{2}+2y^{2}$ twice, we get

$\displaystyle\left|\pi(\widehat{\beta}_{n},\widetilde{\beta}_{K_{n}})\right|^{2}\leq
O_{\mathbb{P}}(1)\gamma_{n}^{2}+O_{\mathbb{P}}(1)\left\|\widetilde{\beta}_{K_{n}}-\beta_{0}\right\|^{2}+O_{\mathbb{P}}(1)\widehat{\lambda}_{n}\mathcal{J}\left(\widetilde{\beta}_{K_{n}}\right).$

To pass from $\pi(\widehat{\beta}_{n},\widetilde{\beta}_{K_{n}})$ to
$\pi(\widehat{\beta}_{n},g_{0})$ simply note that

$\displaystyle\left|\pi(\widehat{\beta}_{n},\beta_{0})\right|^{2}$
$\displaystyle\leq
2\left|\pi(\widehat{\beta}_{n},\widetilde{\beta}_{K_{n}})\right|^{2}+2C^{2}\left\|\widetilde{\beta}_{K_{n}}-\beta_{0}\right\|^{2}$
$\displaystyle=O_{\mathbb{P}}(1)\gamma_{n}^{2}+O_{\mathbb{P}}(1)\left\|\widetilde{\beta}_{K_{n}}-\beta_{0}\right\|^{2}+O_{\mathbb{P}}(1)\widehat{\lambda}_{n}\mathcal{J}\left(\widetilde{\beta}_{K_{n}}\right),$

as we wanted to show.

To prove (14) and (15) let us first observe that by Theorem 1 and (A1) we may
restrict attention to the set
$F_{\delta}\subset\mathcal{B}([0,1])\times\mathbbm{R}$ given by

$\displaystyle
F_{\delta}=\left\\{\left\|\beta-\beta_{0}\right\|_{\infty}<\delta,\left|\kappa-\kappa_{0}\right|<\delta\right\\},$
(17)

for some sufficiently small $\delta>0$. To establish (14) it thus suffices to
prove

$\displaystyle
M\left(\beta,\kappa\right)-M\left(\widetilde{\beta}_{K_{n}},\kappa\right)\geq\eta\left|\pi(\beta,\widetilde{\beta}_{K_{n}})\right|^{2}-L\left\|\beta-\widetilde{\beta}_{K_{n}}\right\|\pi(\beta,\widetilde{\beta}_{K_{n}})$
(18)

for some $\eta,L>0$, uniformly in $(\beta,\kappa)\in F_{\delta}$. To prove
(18), first observe that

$\displaystyle
M\left(\beta,\kappa\right)-M\left(\widetilde{\beta}_{K_{n}},\kappa\right)$
$\displaystyle=\sum_{y\in\\{0,1\\}}\mathbb{E}\left\\{f_{p_{1}(\beta)}^{1+\kappa}(y)-f_{p_{1}(\widetilde{\beta}_{K_{n}})}^{1+\kappa}(y)\right\\}$
$\displaystyle\quad-\left(1+\frac{1}{\kappa}\right)\sum_{y\in\\{0,1\\}}\mathbbm{E}\left\\{f_{p_{1}(\beta_{0})}(y)\left[f_{p_{1}(\beta)}^{\kappa}(y)-f_{p_{1}(\widetilde{\beta}_{K_{n}})}^{\kappa}(y)\right]\right\\}.$
(19)

Consider next a first order Taylor expansion with Lagrange remainder of
$f_{p_{1}(\beta)}^{1+\kappa}(y)$ and $f_{p_{1}(\beta)}^{\kappa}(y)$ about
$f_{p_{1}(\widetilde{\beta}_{K_{n}})}^{1+\kappa}(y)$ and
$f_{p_{1}(\widetilde{\beta}_{K_{n}})}^{\kappa}(y)$, respectively. With the
formulas

$\displaystyle\frac{\partial f_{p}^{\kappa}(y)}{\partial p}$
$\displaystyle=\kappa f_{p}^{\kappa}(y)\left\\{\frac{y-p}{p(1-p)}\right\\}$
$\displaystyle\frac{\partial^{2}f_{p}^{\kappa}(y)}{\partial p^{2}}$
$\displaystyle=\kappa\frac{f_{p}^{\kappa}(y)}{p^{2}(1-p)^{2}}\left\\{(\kappa-1)p^{2}+2(1-\kappa)yp+\kappa
y^{2}-y\right\\},$

which are valid for every $\kappa>0$, we deduce the existence of random
variables $p_{\star}=p_{\star}(X_{1},y,\kappa,\beta)$ and
$p_{\star\star}=p_{\star\star}(X_{1},Y_{1},\kappa,\beta)$ such that
$\max\\{|p_{\star}-p_{1}(\widetilde{\beta}_{K_{n}})|,|p_{\star\star}-p_{1}(\widetilde{\beta}_{K_{n}})|\\}\leq|p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}})|$
and

$\displaystyle
f_{p_{1}(\beta)}^{1+\kappa}(y)-f_{p_{1}(\widetilde{\beta}_{K_{n}})}^{1+\kappa}(y)$
$\displaystyle=(1+\kappa)f_{p_{1}(\widetilde{g}_{K_{n}})}^{1+\kappa}(y)\left\\{\frac{y-p_{1}(\widetilde{\beta}_{K_{n}})}{p_{1}(\widetilde{\beta}_{K_{n}})(1-p_{1}(\widetilde{\beta}_{K_{n}}))}\right\\}(p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}}))$
$\displaystyle\quad+(1+\kappa)\frac{f_{p_{\star}}^{1+\kappa}(y)}{2p_{\star}^{2}(1-p_{\star})^{2}}\left\\{\alpha
p_{\star}^{2}-2\alpha
yp_{\star}+(\kappa+1)y^{2}-y\right\\}\left|p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}})\right|^{2}$
$\displaystyle=(1+\kappa)f_{p_{1}(\beta_{0})}^{1+\kappa}(y)\left\\{\frac{y-p_{1}(\beta_{0})}{p_{1}(\beta_{0})(1-p_{1}(\beta_{0}))}\right\\}(p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}}))$
$\displaystyle\quad+(1+\kappa)\frac{f_{p_{\star}}^{1+\kappa}(y)}{2p_{\star}^{2}(1-p_{\star})^{2}}\left\\{\kappa
p_{\star}^{2}-2\kappa
yp_{\star}+(\kappa+1)y^{2}-y\right\\}\left|p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}})\right|^{2}$
$\displaystyle\quad+(1+\kappa)\left[f_{p_{1}(\widetilde{\beta}_{K_{n}})}^{1+\kappa}(y)\left\\{\frac{y-p_{1}(\widetilde{\beta}_{K_{n}})}{p_{1}(\widetilde{\beta}_{K_{n}})(1-p_{1}(\widetilde{\beta}_{K_{n}}))}\right\\}-f_{p_{1}(\beta_{0})}^{1+\kappa}(y)\left\\{\frac{y-p_{1}(\beta_{0})}{p_{1}(\beta_{0})(1-p_{1}(\beta_{0}))}\right\\}\right]$
$\displaystyle\quad\times(p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}}))$

as well as

$\displaystyle
f_{p_{1}(\beta)}^{\kappa}(y)-f_{p_{1}(\widetilde{\beta}_{K_{n}})}^{\kappa}(y)$
$\displaystyle=\kappa
f_{p_{1}(\widetilde{\beta}_{K_{n}})}^{\kappa}(y)\left\\{\frac{y-p_{1}(\widetilde{\beta}_{K_{n}})}{p_{1}(\beta_{K_{n}})(1-p_{1}(\beta_{K_{n}}))}\right\\}\left(p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}}))\right)$
$\displaystyle\quad+\kappa\frac{f_{p_{\star\star}}^{\kappa}(y)}{2p_{\star\star}^{2}(1-p_{\star\star})^{2}}\left\\{(\kappa-1)p_{\star\star}^{2}+2(1-\kappa)yp_{\star\star}+\kappa
y^{2}-y\right\\}\left|p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}})\right|^{2}$
$\displaystyle=\kappa
f_{p_{1}(\beta_{0})}^{\kappa}(y)\left\\{\frac{y-p_{1}(\beta_{0})}{p_{1}(\beta_{0})(1-p_{1}(\beta_{0}))}\right\\}(p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}}))$
$\displaystyle\quad+\kappa\frac{f_{p_{\star\star}}^{\kappa}(y)}{2p_{\star\star}^{2}(1-p_{\star\star})^{2}}\left\\{(\kappa-1)p_{\star\star}^{2}+2(1-\kappa)yp_{\star\star}+\kappa
y^{2}-y\right\\}\left|p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}})\right|^{2}$
$\displaystyle\quad+\kappa\left[f_{p_{1}(\widetilde{\beta}_{K_{n}})}^{\kappa}(y)\left\\{\frac{Y_{1}-p_{1}(\widetilde{\beta}_{K_{n}})}{p_{1}(\widetilde{\beta}_{K_{n}})(1-p_{1}(\widetilde{\beta}_{K_{n}}))}\right\\}-f_{p_{1}(\beta_{0})}^{\kappa}(y)\left\\{\frac{y-p_{1}(\beta_{0})}{p_{1}(\beta_{0})(1-p_{1}(\beta_{0}))}\right\\}\right]$
$\displaystyle\quad\times\left\\{p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}})\right\\}.$

Plugging these expansions into (8) and simplifying we get

$\displaystyle
M\left(\beta,\kappa\right)-M\left(\widetilde{\beta}_{K_{n}},\kappa\right)$
$\displaystyle=(1+\kappa)\sum_{y\in\\{0,1\\}}\mathbb{E}\left\\{\left(u_{n}(X_{1},y,\widetilde{\beta}_{K_{n}},\kappa)-v_{n}(X_{1},y,\widetilde{\beta}_{K_{n}},\kappa)\right)\left(p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}})\right)\right\\}$
$\displaystyle\quad+(1+\kappa)\sum_{y\in\\{0,1\\}}\mathbb{E}\left\\{\left(u_{n}^{\prime}(X_{1},y,\beta,\kappa)-v_{n}^{\prime}(X_{1},y,\beta,\kappa)\right)\left|p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}})\right|^{2}\right\\},$

where for convenience we have defined

$\displaystyle u_{n}(X_{1},y,\widetilde{\beta}_{K_{n}},\alpha)$
$\displaystyle=f_{p_{1}(\widetilde{\beta}_{K_{n}})}^{1+\kappa}(y)\left\\{\frac{y-p_{1}(\widetilde{\beta}_{K_{n}})}{p_{1}(\widetilde{\beta}_{K_{n}})(1-p_{1}(\widetilde{\beta}_{K_{n}}))}\right\\}-f_{p_{1}(\beta_{0})}^{1+\kappa}(y)\left\\{\frac{y-p_{1}(\beta_{0})}{p_{1}(\beta_{0})(1-p_{1}(\beta_{0}))}\right\\}$
$\displaystyle v_{n}(X_{1},y,\widetilde{\beta}_{K_{n}},\kappa)$
$\displaystyle=f_{p_{1}(\widetilde{\beta}_{K_{n}})}^{\kappa}(y)f_{p_{1}(\beta_{0})}(y)\left\\{\frac{y-p_{1}(\widetilde{\beta}_{K_{n}})}{p_{1}(\widetilde{\beta}_{K_{n}})(1-p_{1}(\widetilde{\beta}_{K_{n}}))}\right\\}-f_{p_{1}(\beta_{0})}^{1+\kappa}(y)\left\\{\frac{y-p_{1}(\beta_{0})}{p_{1}(\beta_{0})(1-p_{1}(\beta_{0}))}\right\\}$
$\displaystyle u_{n}^{\prime}(X_{1},y,\beta,\kappa)$
$\displaystyle=\frac{f_{p_{\star}}^{1+\kappa}(y)}{2p_{\star}^{2}(1-p_{\star})^{2}}\left\\{\kappa
p_{\star}^{2}-2\kappa yp_{\star}+(\kappa+1)y^{2}-y\right\\}$ $\displaystyle
v_{n}^{\prime}(X_{1},y,\beta,\kappa)$
$\displaystyle=\frac{f_{p_{\star\star}}^{\kappa}(y)f_{p_{1}(\beta_{0})}(y)}{2p_{\star\star}^{2}(1-p_{\star\star})^{2}}\left\\{(\kappa-1)p_{\star\star}^{2}+2(1-\kappa)yp_{\star\star}+\kappa
y^{2}-y\right\\},$

where $u_{n}^{\prime}(X_{1},y,\beta,\kappa)$ and
$v_{n}^{\prime}(X_{1},y,\beta,\kappa)$ depend on $\beta$ through $p_{\star}$
and $p_{\star\star}$ respectively.

We first show that for sufficiently small $\delta$ in the definition of
$F_{\delta}$, there exists an $\eta>0$

$\displaystyle(1+\kappa)\sum_{y\in\\{0,1\\}}\mathbb{E}\left\\{\left(u_{n}^{\prime}(X_{1},y,\beta,\kappa)-v_{n}^{\prime}(X_{1},y,\beta,\kappa)\right)\left|p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}})\right|^{2}\right\\}\geq\eta\left|\pi(\beta,\widetilde{\beta}_{K_{n}})\right|^{2},$
(20)

for all sufficiently large $n$, uniformly in $(\beta,\kappa)\in F$. To see
this, observe first that, by definition of $p_{\star}$ and our assumptions, we
find

$\displaystyle\left|p_{\star}-p_{1}(\beta_{0})\right|$
$\displaystyle\leq\left|p_{\star}-p_{1}(\widetilde{\beta}_{K_{n}})\right|+\left|p_{1}(\widetilde{\beta}_{K_{n}})-p_{1}(\beta_{0})\right|$
$\displaystyle\leq
c_{0}\left\|\beta-\beta_{0}\right\|_{\infty}+c_{0}\left\|\widetilde{\beta}_{K_{n}}-\beta_{0}\right\|$
$\displaystyle\leq c_{0}\delta+o(1)$

A similar argument also shows that
$|p_{\star\star}-p_{1}(g_{0})|<c_{0}\delta+o(1)$. Recall next that
$p_{1}(\beta_{0})$ is bounded away from $0$ and $1$ with probability one (see
the proof of Lemma 5). Hence, $p_{\star}$ and $p_{\star\star}$ are also
bounded away from $0$ and $1$ for all $\delta\leq\delta_{0}$ and $n\geq n_{0}$
with probability one. By continuity, there exists a
$\delta_{0}^{\prime}\leq\delta_{0}$ such that for all
$\delta\leq\delta_{0}^{\prime}$ we have

$\displaystyle
u_{n}^{\prime}(X_{1},y,\beta,\kappa)-v_{n}^{\prime}(X_{1},y,\beta,\kappa)$
$\displaystyle\geq\frac{f_{p_{1}(\beta_{0})}^{1+\kappa}(y)}{8p_{1}^{2}(\beta_{0})(1-p_{1}(\beta_{0}))^{2}}\left\\{p_{1}^{2}(\beta_{0})-2yp_{1}(\beta_{0})+y^{2}\right\\}$
$\displaystyle=\frac{f_{p_{1}(\beta_{0})}^{1+\kappa}(y)}{8p_{1}^{2}(\beta_{0})(1-p_{1}(\beta_{0}))^{2}}\left|y-p_{1}(\beta_{0})\right|^{2},$

for all $n\geq n_{0}$ with probability one. Summing over $y\in\\{0,1\\}$ we
see that there exists a $c_{0}>0$ such that

$\displaystyle\sum_{y\in\\{0,1\\}}\left\\{u_{n}^{\prime}(X_{1},y,\beta,\kappa)-v_{n}^{\prime}(X_{1},y,\beta,\kappa)\right\\}\geq
c_{0}$

for all $n\geq n_{0}$ with probability one. Therefore,

$\displaystyle\sum_{y\in\\{0,1\\}}\mathbb{E}\left\\{\left(u_{n}^{\prime}(X_{1},y,\beta,\kappa)-v_{n}^{\prime}(X_{1},y,\beta,\kappa)\right)\left|p_{1}(\beta)-p_{1}(\beta_{K_{n}})\right|^{2}\right\\}\geq
c_{0}\mathbb{E}\left\\{\left|p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}})\right|^{2}\right\\}.$

Now, by the mean-value theorem, there exists a (random)
$x^{\star}\in\mathbbm{R}$ such that $|x^{\star}-\langle
X_{1},\widetilde{\beta}_{K_{n}}\rangle|\leq|\langle X_{1},\beta\rangle-\langle
X_{1},\widetilde{\beta}_{K_{n}}\rangle|$ and

$\displaystyle\left|p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}})\right|^{2}=\left|H\left(\langle
X_{1},\beta\rangle\right)-H\left(\langle
X_{1},\widetilde{\beta}_{K_{n}}\rangle\right)\right|^{2}=\left|H^{\prime}(x^{\star})\right|^{2}\left|\langle
X_{1},\beta-\widetilde{\beta}_{K_{n}}\rangle\right|^{2}.$

For
$\beta\in\\{f\in\mathcal{B}([0,1]):\|\beta-\beta_{0}\|_{\infty}<\delta\\}$, by
(A5) and (A4), we find

$\displaystyle\left|x^{\star}-\langle X_{1},\beta_{0}\rangle\right|\leq
c_{0}\delta+o(1),$

for some $c_{0}>0$ with probability one. Since $|\langle
X_{1},\beta_{0}\rangle|$ is bounded away from zero and one with probability
one and $H$ is strictly increasing, this entails the existence of a
$\delta_{0}^{\prime\prime}$ such that $H^{\prime}(x^{\star})\geq c_{0}>0$ with
probability one for all
$\delta\leq\delta_{0}^{\prime\prime}\leq\delta_{0}^{\prime}$ and $n\geq
n_{0}^{\prime}$, whence we obtain

$\displaystyle\mathbb{E}\left\\{\left|p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}})\right|^{2}\right\\}\geq
c_{0}^{2}\mathbb{E}\left\\{\left|\langle
X_{1},\beta-\widetilde{\beta}_{K_{n}}\rangle\right|^{2}\right\\}=c_{0}^{2}|\pi(\beta,\widetilde{\beta}_{K_{n}})|^{2},$

for all $\delta\leq\delta_{0}^{\prime\prime}$ and $n\geq n_{0}^{\prime}$ with
probability one, for all large $n$. Combining the above lower bounds, we find

$\displaystyle(1+\kappa)\sum_{y\in\\{0,1\\}}\mathbb{E}\left\\{\left(u_{n}^{\prime}(X_{1},y,\beta,\kappa)-v_{n}^{\prime}(X_{1},y,\beta,\kappa)\right)\left|p_{1}(\beta)-p_{1}(\beta_{K_{n}})\right|^{2}\right\\}\geq\eta\left|\pi(\beta,\widetilde{\beta}_{K_{n}})\right|^{2},$

for all $n\geq n_{0}\vee n_{0}^{\prime}$ and
$\delta\leq\delta_{0}^{\prime\prime}$. Here,
$\eta=c_{0}(1+\kappa_{0}-\delta)>0$ and $(1+\kappa_{0}-\delta)$ is strictly
positive for all $\delta\leq(1+\kappa_{0})/2$, as $c_{0}>0$. Choosing
$\delta\leq\delta_{0}^{\prime\prime}\wedge(\kappa_{0}+1)/2$ ensures the
positivity of $\eta>0$, completing the proof of (20).

We next show that

$\displaystyle\sum_{y\in\\{0,1\\}}$
$\displaystyle\left|\mathbb{E}\left\\{\left(u_{n}(X_{1},y,\widetilde{\beta}_{K_{n}},\kappa)-v_{n}(X_{1},y,\widetilde{\beta}_{K_{n}},\kappa)\right)\left(p_{1}(\beta)-p_{1}(\widetilde{\beta}_{K_{n}})\right)\right\\}\right|$
$\displaystyle\quad\leq
L\left\|\widetilde{\beta}_{K_{n}}-\beta_{0}\right\|\pi(\beta,\widetilde{\beta}_{K_{n}}),$
(21)

for some finite $L>0$, uniformly in $(\beta,\kappa)\in F_{\delta}$. To prove
this, notice first that by the boundedness of $p_{1}(\beta_{0})$ away from $0$
and $1$ and the fact that
$|p_{1}(\widetilde{\beta}_{K_{n}})-p_{1}(\beta_{0})|\to 0$ as $n\to\infty$
with probability one, it may also be assumed that
$p_{1}(\widetilde{\beta}_{K_{n}})$ is also bounded away from $0$ and $1$ for
all large $n$. For $p$ bounded away from $0$ and $1$ the functions $p\mapsto
f_{p}^{\kappa}(y)$ and $p\mapsto(y-p)/p(1-p)$ are for all
$\kappa\in(\kappa_{0}-\delta,\kappa_{0}+\delta)$ with small $\delta>0$ and all
$y\in\\{0,1\\}$ are bounded and Lipschitz. By boundedness, their product is
also Lipschitz. Conclude that for all $n\geq n_{0}^{\prime\prime}$ there
exists a $c_{0}>0$ depending only on $\delta$ such that

$\displaystyle\left|u_{n}(X_{1},y,\widetilde{\beta}_{K_{n}},\kappa)-v_{n}(X_{1},y,\widetilde{\beta}_{K_{n}},\kappa)\right|\leq
c_{0}\left|p_{1}(\widetilde{\beta}_{K_{n}})-p_{1}(\beta_{0})\right|\leq
c_{0}\left\|\widetilde{\beta}_{K_{n}}-\beta_{0}\right\|,$

with probability one. Moreover, (A2) and the Schwarz inequality imply that

$\displaystyle\mathbb{E}\left\\{\left|p_{1}(\widetilde{\beta}_{K_{n}})-p_{1}(\beta)\right|\right\\}\leq
c_{0}\mathbb{E}\left\\{\left|\langle
X_{1},\widetilde{\beta}_{K_{n}}-\beta\rangle\right|\right\\}\leq
c_{0}\pi(\widetilde{\beta}_{K_{n}},\beta).$

These two bounds jointly imply (18), thus completing the first part of our
proof.

To complete the proof we now establish (15) and for this we argue as in
Kalogridis and Van Aelst (2023). First note that on $F_{\delta}$ we have
$\|\beta-\widetilde{\beta}_{K_{n}}\|\leq\delta+o(1)\leq 2\delta$ for all large
$n$, by (A4) and the fact that the uniform norm dominates the
$\mathcal{L}^{2}([0,1])$-norm. Thus,

$\displaystyle\left|\frac{U_{n}(\widetilde{\beta}_{K_{n}},\widehat{\beta}_{n},\widehat{\kappa}_{n})}{\gamma_{n}\pi\left(\widetilde{\beta}_{K_{n}},\widehat{\beta}_{n}\right)\vee\gamma_{n}^{2}}\right|\leq\sup_{\begin{subarray}{c}\beta\in\Theta_{K_{n}}:\|\beta-\widetilde{\beta}_{K_{n}}\|\leq
2\delta\\\
|\kappa-\kappa_{0}|\leq\delta\end{subarray}}\left|\frac{U_{n}(\widetilde{\beta}_{K_{n}},\beta,\kappa)}{\gamma_{n}\pi\left(\widetilde{\beta}_{K_{n}},\beta\right)\vee\gamma_{n}^{2}}\right|,$
(22)

for all large $n$ with high probability. We will show that the random variable
on the RHS of (22) is bounded in probability, which is equivalent to showing
that

$\displaystyle\lim_{T\to\infty}\limsup_{n\to\infty}\mathbb{P}\left(\sup_{\begin{subarray}{c}\beta\in\Theta_{K_{n},\delta},\kappa\in
V_{\delta}\\\
\pi\left(\widetilde{\beta}_{K_{n}},\beta\right)\leq\gamma_{n}\end{subarray}}\left|U_{n}(\widetilde{\beta}_{K_{n}},\beta,\alpha)\right|\geq
T\gamma_{n}^{2}\right)=0,$ (23)

as well as

$\displaystyle\lim_{T\to\infty}\limsup_{n\to\infty}\mathbb{P}\left(\sup_{\begin{subarray}{c}\beta\in\Theta_{K_{n},\delta},\kappa\in
V_{\delta}\\\
\pi\left(\widetilde{\beta}_{K_{n}},\beta\right)>\gamma_{n}\end{subarray}}\left|\frac{U_{n}(\widetilde{\beta}_{K_{n}},\beta,\kappa)}{\pi\left(\widetilde{\beta}_{K_{n}},\beta\right)}\right|\geq
T\gamma_{n}\right)=0,$ (24)

where for notational simplicity we have defined the balls

$\displaystyle\Theta_{K_{n},\delta}$
$\displaystyle=\left\\{\beta\in\Theta_{K_{n}}:\|\beta-\widetilde{\beta}_{K_{n}}\|\leq
2\delta\right\\}$ $\displaystyle V_{\delta}$
$\displaystyle=\left\\{\kappa\in\mathbbm{R}:|\kappa-\kappa_{0}|\leq\delta\right\\}.$

For the proof of both (23) and (24) we will use the fact that for all
$\epsilon>0$ sufficiently small, say $\epsilon\leq\epsilon_{0}$, there exists
a $B>0$ such that

$\displaystyle\int_{0}^{\epsilon}\log^{1/2}\left(1+\frac{1}{u}\right)du\leq
B\epsilon\log^{1/2}\left(\frac{1}{\epsilon}\right).$

(Use, e.g., L’Hospital’s rule). To show (23) we will apply Theorem 5.11 of van
de Geer (2000) on the process $U_{n}(\widetilde{\beta}_{K_{n}},\beta,\kappa)$
for $\beta\in\Theta_{K_{n},\delta}$, $\kappa\in V_{\delta}$ and
$\pi(\beta,\widetilde{\beta}_{K_{n}})\leq\gamma_{n}$. To prepare for this
application, write $U_{n}(\widetilde{\beta}_{K_{n}},\beta,\kappa)$ as

$\displaystyle U_{n}(\widetilde{\beta}_{K_{n}},\beta,\kappa)=\int
z_{\beta,\kappa}d\left(\mathbb{P}_{n}-\mathbb{P}_{X,Y}\right)=n^{-1/2}v_{n}(z_{\beta,\kappa}),$

where $z_{\beta,\kappa}$ and $\mathbb{P}_{X,Y}$ are as in Lemma 5 and
$v_{n}(\cdot)$ denotes the empirical process. By our definitions,
$z_{\beta,\kappa}\in\mathcal{Z}(n,2\delta,\delta)$ and each $z_{\beta,\kappa}$
is uniformly bounded by $2+1/(\kappa_{0}-\delta)$. Let
$\mathcal{H}_{B,K}(\epsilon,\mathcal{Z}(n,2\delta,\delta),\mathbb{P}_{X,Y})$
denote the generalized entropy with bracketing (in the Bernstein ”norm”
$\rho_{K}$) of the class of functions $\mathcal{Z}(n,2\delta,\delta)$, as
given in Definition 5.1 in van de Geer (2000). By the uniform boundedness of
the elements of $\mathcal{Z}(n,2\delta,\delta)$, Lemma 5.10 of van de Geer
(2000) yields

$\displaystyle\mathcal{H}_{B,8+4/(\alpha_{0}-\delta)}\left(\epsilon,\mathcal{Z}(n,2\delta,\delta)\right)\leq
H_{B}(\epsilon/\sqrt{2},\mathcal{Z}(n,2\delta,\delta),\mathbb{P}_{X,Y}),\quad\epsilon>0.$
(25)

Observe next that by our assumptions for all
$\|\beta-\widetilde{\beta}_{K_{n}}\|\leq 2\delta$ we have $|H(\langle
X,\beta\rangle))-H(\langle X,\beta_{0}\rangle))|\leq c_{0}\delta+o(1)$, as
$n\to\infty$. As discussed, previously, $H(\langle X,\beta_{0}\rangle)$ is
bounded away from $0$ and $1$ with probability one. Hence, $H(\langle
X,\beta\rangle))$ is also bounded away from $0$ and $1$ for all
$\beta\in\Theta_{K_{n},\delta}$ when $\delta$ is chosen small enough and $n$
is large with probability one. From this we may deduce that for all
$\kappa\in(\kappa_{0}-\delta,\kappa_{0}+\delta)$ and $y\in\\{0,1\\}$,
$f_{H(\langle X,\beta\rangle)}^{\kappa}(y)$ and $f_{H(\langle
X,g\rangle)}^{1+\kappa}(y)$ are Lipschitz with respect to the probabilities
$H(\langle X,\beta\rangle)$ for $\|\beta-\widetilde{\beta}_{K_{n}}\|\leq
2\delta$. A straightforward calculation now shows that there exists a global
$c_{0}>0$ such that

$\displaystyle\left|z_{\beta,\kappa}(X,Y)\right|\leq c_{0}\left|\langle
X,\beta-\widetilde{\beta}_{K_{n}}\rangle\right|.$

Taking expectations yields

$\displaystyle\mathbb{E}\left\\{\left|z_{\beta,\kappa}(X,Y)\right|^{2}\right\\}\leq
c_{0}^{2}\left|\pi\left(\beta,\widetilde{\beta}_{K_{n}}\right)\right|^{2}\leq
c_{0}^{2}\gamma_{n}^{2}.$ (26)

It follows from Lemma 5.8 of van de Geer (2000) that we may take
$R_{n}=c_{0}\gamma_{n}$ in Theorem 5.11 of van de Geer (2000). Assumption (A5)
implies that $R_{n}\to 0$ as $n\to\infty$. Hence $R_{n}\leq\epsilon_{0}$ for
all large $n$. Thus, taking square roots, integrating (25) from $0$ to $R_{n}$
and applying Lemma 5, we get

$\displaystyle\int_{0}^{R_{n}}\mathcal{H}_{B,8+4/(\alpha_{0}-\delta)}^{1/2}(u,\mathcal{Z}(n,2\delta,\delta)du\leq
c_{0}K_{n}^{1/2}\gamma_{n}\log^{1/2}(n)=c_{0}\frac{K_{n}\log(n)}{n^{1/2}}.$

Conditions (5.31), (5.33) and (5.34) of Theorem 5.11 van de Geer (2000) are
satisfied if we take $C_{0}=T/c_{0}$ and
$C_{1}=(8+4/(\alpha_{0}-\delta))T/c_{0}^{2}$. An application of that theorem
then yields

$\displaystyle\mathbb{P}\left(\sup_{\begin{subarray}{c}\beta\in\Theta_{K_{n},\delta},\kappa\in
V_{\delta}\\\
\pi\left(\widetilde{\beta}_{K_{n}},\beta\right)\leq\gamma_{n}\end{subarray}}\left|U_{n}(\widetilde{\beta}_{K_{n}},\beta,\kappa)\right|\geq
T\gamma_{n}^{2}\right)$
$\displaystyle=\mathbb{P}\left(\sup_{\begin{subarray}{c}\beta\in\Theta_{K_{n},\delta},\kappa\in
V_{\delta}\\\
\pi\left(\widetilde{\beta}_{K_{n}},\beta\right)\leq\gamma_{n}\end{subarray}}\left|v_{n}(z_{\beta,\kappa})\right|\geq
Tn^{1/2}\gamma_{n}^{2}\right)$ $\displaystyle\leq
C\exp\left[-\frac{T^{2}K_{n}\log(n)}{C^{2}(C_{1}+1)}\right].$

for some global $C>0$. The exponential tends to $0$, as, by (A4), $K_{n}\log
n\to\infty$. Furthermore, as this holds for all sufficiently large $T$, (23)
holds and therefore the first step of our proof is complete.

Finally, we establish (24). We begin by observing that, by (A5),
$\pi(\beta,\widetilde{\beta}_{K_{n}})\leq
C\|\beta-\widetilde{\beta}_{K_{n}}\|\leq 2C\delta\leq\epsilon_{0}/c_{0}$
whenever $\beta\in\Theta_{K_{n},\delta}$ with
$\delta\leq\epsilon_{0}/(c_{0}2C)$. Here, $c_{0}$ is the constant appearing in
(26). Therefore, it suffices to prove

$\displaystyle\lim_{T\to\infty}\limsup_{n\to\infty}\mathbb{P}\left(\sup_{\begin{subarray}{c}\beta\in\Theta_{K_{n},\delta},\kappa\in
V_{\delta}\\\
\gamma_{n}<\pi\left(\widetilde{\beta}_{K_{n}},\beta\right)\leq\epsilon_{0}/c_{0}\end{subarray}}\left|\frac{U_{n}(\widetilde{\beta}_{K_{n}},\beta,\kappa)}{\pi\left(\widetilde{\beta}_{K_{n}},\beta\right)}\right|\geq
T\gamma_{n}\right)=0.$ (27)

Let $S=\min\\{s>1:2^{-s}\epsilon_{0}/c_{0}<\gamma_{n}\\}$. By (A4),
$K_{n}\asymp n^{\gamma}$ for some $\gamma\in(0,1)$. Hence, we can have
$S\leq\lceil c_{0}\log_{2}(n)+1\rceil$ for some $c_{0}>0$. Boole’s inequality
gives

$\displaystyle\mathbb{P}\left(\sup_{\begin{subarray}{c}\beta\in\Theta_{K_{n},\delta},\kappa\in
V_{\delta}\\\
\gamma_{n}<\pi\left(\widetilde{\beta}_{K_{n}},g\right)\leq\epsilon_{0}/c_{0}\end{subarray}}\left|\frac{U_{n}(\widetilde{\beta}_{K_{n}},\beta,\kappa)}{\pi\left(\widetilde{\beta}_{K_{n}},\beta\right)}\right|\geq
T\gamma_{n}\right)$
$\displaystyle\leq\sum_{s=1}^{S}\mathbb{P}\left(\sup_{\begin{subarray}{c}\beta\in\Theta_{K_{n},\delta},\kappa\in
V_{\delta}\\\
2^{-s}\epsilon_{0}/c_{0}<\pi\left(\widetilde{\beta}_{K_{n}},\beta\right)\leq
2^{-s+1}\epsilon_{0}/c_{0}\end{subarray}}\left|\frac{U_{n}(\widetilde{\beta}_{K_{n}},\beta,\kappa)}{\pi\left(\widetilde{\beta}_{K_{n}},\beta\right)}\right|\geq
T\gamma_{n}\right)$
$\displaystyle\leq\sum_{s=1}^{S}\mathbb{P}\left(\sup_{\begin{subarray}{c}\beta\in\Theta_{K_{n},\delta},\kappa\in
V_{\delta}\\\ \pi\left(\widetilde{\beta}_{K_{n}},\beta\right)\leq
2^{-s+1}\epsilon_{0}/c_{0}\end{subarray}}\left|U_{n}(\widetilde{\beta}_{K_{n}},\beta,\alpha)\right|\geq
T2^{-s}\gamma_{n}\epsilon_{0}/c_{0}\right)$

Clearly, for every $s\geq 1$ we have $2^{-s+1}\epsilon_{0}\leq\epsilon_{0}$.
Hence, by (26) and Lemma 5,

$\displaystyle\int_{0}^{2^{-s+1}\epsilon_{0}}\mathcal{H}^{1/2}_{B,8+4/(\alpha_{0}-\delta)}\left(u,\mathcal{Z}(n,2\delta,\delta)\right)du\leq
2^{-s+1}B^{\star}K_{n}^{1/2}\log^{1/2}(n),$

for some global $B^{\star}>0$, where we have used the fact that $2^{s}\leq
2^{S}\leq n^{2c}$ for all large $n$. We will apply Theorem 5.11 of van de Geer
(2000) individually to each summand, see also the proof of (23). The
conditions of that theorem are satisfied for
$C_{0}=T\epsilon_{0}/(2B^{\star}c_{0})$ and
$C_{1}=T(8+4/(\alpha_{0}-\delta))/c_{0}$ with $T$ sufficiently large, as, by
definition of $S$, we have $\gamma_{n}\leq 2^{-s+1}\epsilon_{0}/c_{0}$ for all
$s=1,\ldots,S$. Therefore, by Theorem 5.11 of van de Geer (2000), there exists
a global constant $C{{}^{\prime}}>0$ such that

$\displaystyle\mathbb{P}\left(\sup_{\begin{subarray}{c}\beta\in\Theta_{K_{n},\delta},\kappa\in
V_{\delta}\\\ \pi\left(\widetilde{\beta}_{K_{n}},\beta\right)\leq
2^{-s+1}\epsilon_{0}/c_{0}\end{subarray}}\left|U_{n}(\widetilde{\beta}_{K_{n}},\beta,\kappa)\right|\geq
T2^{-s}\gamma_{n}\epsilon_{0}/c_{0}\right)\leq
C^{\prime}\exp\left[-\frac{T^{2}K_{n}\log(n)}{4|C^{\prime}|^{2}(C_{1}+1)c_{0}^{2}}\right].$

The exponential does not depend on $s=1,\ldots,S$, hence upon summing and
recalling that $S\leq\lceil c_{0}\log_{2}(n)+1\rceil$ for some $c_{0}>0$ we
arrive at

$\displaystyle\mathbb{P}\left(\sup_{\begin{subarray}{c}\beta\in\Theta_{K_{n},\delta},\kappa\in
V_{\delta}\\\
\gamma_{n}<\pi\left(\widetilde{\beta}_{K_{n}},\beta\right)\leq\epsilon_{0}/c_{0}\end{subarray}}\left|\frac{U_{n}(\widetilde{\beta}_{K_{n}},\beta,\kappa)}{\pi\left(\widetilde{\beta}_{K_{n}},\beta\right)}\right|\geq
T\gamma_{n}\right)\leq C^{\prime}\lceil
c_{0}\log_{2}(n)+1\rceil\exp\left[-\frac{T^{2}K_{n}\log(n)}{4|C^{\prime}|^{2}(C_{1}+1)c_{0}^{2}}\right],$

for all large $n$. For all $T$ large enough, say $T\geq T_{0}$, the RHS tends
to $0$, as $n\to\infty$, by virtue of $K_{n}\asymp n^{\gamma}$. We have thus
established (24), thereby completing the proof. ∎

## References

## References

  * Basak et al. (2021) authorBasak, S., authorBasu, A., and authorJones M.C. (year2021) titleOn the ”optimal” density power divergence tuning parameter, journalJ. Appl. Stat. volume48 pages536–556.
  * Basu et al. (1998) authorBasu, A., authorHarris, I.R., authorHjort, N.L, and authorJones, M.C. (year1998) titleRobust and efficient estimation by minimising a density power divergence, journalBiometrika volume85 pages549–559.
  * Boente et al. (2020) authorBoente, G., authorSalibián-Barrera, M., and authorVena, P. (year2020) titleRobust estimation for semi-functional linear regression models, journalComp. Statist. Data Anal. volume152.
  * Cardot et al. (1999) authorCardot, H., authorFerraty, F., and authorSarda, P. (year1999) titleFunctional linear model, journalStatist. Probab. Lett. volume45 pages11–22.
  * Cardot et al. (2003) authorCardot, H., authorFerraty, F., and authorSarda, P. (year2003) titleSpline estimators for the functional linear model, journalStatist. Sinica volume13 pages571–591.
  * Cardot and Sarda (2005) authorCardot, H., and authorSarda, P. (year2005) titleEstimation in generalized linear models for functional data via penalized likelihood, journalJ. Multivariate Anal. volume92 pages24–41.
  * Crambes et al. (2009) authorCrambes, C., authorKneip, A., and authorSarda, P. (year2009) titleSmoothing splines estimators for functional linear regression, journalAnn. Statist. volume37 pages35–72.
  * Dau et al. (2018) authorDau, H.A., authorKeogh, E., authorKamgar, K., authorYeh, C.-C. M., authorZhu, Y, authorGharghabi, S., authorRatanamahatana, C. A., authorYanping, D., authorHu, B., authorBegum, N., authorBagnall, A., authorMueen, A., authorBatista, G., and authorHexagon-ML (year2018) titleThe UCR Time Series Classification Archive, https://www.cs.ucr.edu/~eamonn/time_series_data_2018/.
  * de Boor (2001) authorde Boor, C. (year2001) titleA Practical Guide to Splines, Revised ed., publisherSpringer, New York.
  * Denhere and Billor (2016) authorDenhere, M., and authorBillor, N. (year2016) titleRobust Principal Component Functional Logistic Regression, journalComm. Statist. Simulation Comput. volume45 pages264–281.
  * Eggermont and LaRiccia (2009) authorEggermont, P.P.B., and authorLaRiccia, V.N. (year2009) titleMaximum Penalized Likelihood Estimation, Volume II: Regression, publisherSpringer, New York.
  * Fan and Li (2004) authorFan, J., and authorLi, R. (year2001) titleVariable Selection via Nonconcave Penalized Likelihood and its Oracle Properties, journalJ. Amer. Statist. Assoc. volume96 pages1348–1360.
  * Ferraty and Vieu (2006) authorFerraty, F., and authorVieu, P. (year2006) titleNonparametric Functional Data Analysis: Theory and Practice, publisherSpringer, New York.
  * Green and Silverman (1994) authorGreen, P.J., and authorSilverman, B.W. (year1994) titleNonparametric Regression and Generalized Linear Models, publisherChapman & Hall, London.
  * Hall et al. (2007) authorHall, P., and authorHorowitz, J.L. (year2007) titleMethodology and convergence rates for functional linear regression, journalAnn. Statist. volume35 pages70–91.
  * Hampel et al. (2011) authorHampel, F.R., authorRonchetti, E.M., authorRousseeuw, P.J., and authorStahel, W.A. (year2011) titleRobust Statistics: The Approach Based on Influence Functions, publisherWiley, New York.
  * Hubert et al. (2005) authorHubert, M., authorRousseeuw, P.J., and authorVanden Branden, K. (year2005) titleROBPCA: a new approach to robust principal component analysis, journalTechnometrics volume47 pages64–79.
  * James (2002) authorJames, G.M. (year2002) titleGeneralized Linear Models with Functional Predictors, journalJ. R. Stat. Soc. Ser. B. Stat. Methodol. volume64 pages411–432.
  * Kalogridis et al. (2023) authorKalogridis, I., authorClaeskens, G., and authorVan Aelst, S. (year2022) titleRobust and efficient estimation of nonparametric generalized linear models, journalTEST, accepted.
  * Kalogridis and Van Aelst (2023) authorKalogridis, I., and authorVan Aelst, S. (year2023) titleRobust penalized estimators for functional linear regression, journalJ. Multivariate Anal. volume194.
  * Kosorok (2008) authorKosorok, M. R. (year2008) titleIntroduction to Empirical Processes and Semiparametric inference, publisherSpringer, New York.
  * McCullagh and Nelder (1983) authorMcCullagh, P., and authorNelder, J.A. (year1983) titleGeneralized Linear Models, publisherChapman & Hall, London.
  * Müller and Stadtmüller (2005) authorMüller, H.-G., and authorStadtmüller, U. (year2005) titleGeneralized functional linear models, journalAnn. Statist. volume33 pages774–805.
  * Mutis et al. (2022) authorMutis, M., authorBeyaztas, U., authorSimsek, G.G., and authorShang, H.L. (year2022) titleA robust scalar-on-function logistic regression for classification, journalComm. Statist. Theory Methods volumeappeared online.
  * R Core Team (2023) authorR Core Team (year2023). titleR: A language and environment for statistical computing. organizationR Foundation for Statistical Computing, addressVienna, Austria.
  * Ramsay and Dalzell (1991) authorRamsay, J.O., and authorDalzell, C.J. (year1991) titleSome Tools for Functional Data Analysis, journalJ. R. Stat. Soc. Ser. B. Stat. Methodol volume53 pages539–561.
  * Ramsay and Silverman (2005) authorRamsay, J.O., and authorSilverman, B.W. (year2005) titleFunctional Data Analysis, publisherWiley, New York.
  * Ratcliffe et al. (2002) authorRatcliffe, S.J., authorHeller, G.Z. and authorLeader, L.R. (year2002) titleFunctional data analysis with application to periodically stimulated foetal heart rate data. II: Functional logistic regression, journalStat. Med. volume21 pages1115–1127.
  * Reiss and Ogden (2007) authorReiss, P.T., and authorOgden, R.T. (year2007) titleFunctional principal component regression and functional partial least squares, journalJ. Amer. Statist. Assoc. volume102 pages984–996.
  * Ruppert et al. (2003) authorRuppert, D., authorWand, M.P., and authorCarroll, R.J. (year2003) titleSemiparametric Regression, publisherCambridge, New York.
  * Shin and Lee (2016) authorShin, H., and authorLee, S. (year2016) titleAn RKHS approach to robust functional linear regression, journalStatist. Sinica volume26 pages255–272.
  * van de Geer (2000) authorvan de Geer, S. (year2000) titleEmpirical Processes in M-Estimation, publisherCambridge University Press, New York, NY.
  * van der Vaart (1998) authorvan der Vaart, A.W. (year1998) titleAsymptotic Statistics, publisherCambridge University Press, New York, NY.
  * Warwick and Jones (2005) authorWarwick, J., and authorJones, M.C. (year2005) titleChoosing a robustness tuning parameter, journalJ. Stat. Comput. Simul. volume75 pages581–588.
  * Wei et al. (2014) authorWei, P., authorTang, H. and authorLi, D. (author2014) titleFunctional Logistic Regression Approach to Detecting Gene by Longitudinal Environmental Exposure Interaction in a Case-Control Study, journalGen. Epidem. volume38 pages638–651.
  * Wood (2017) authorWood, S. (year2017) titleGeneralized Additive Models, 2nd ed., publisherCRC Press, Boca Raton, FL.
  * Yuan and Cai (2010) authorYuan, M., and authorCai, T.T. (year2010) titleA reproducing kernel Hilbert space approach to functional linear regression, journalAnn. Statist. volume38 pages3412–3444.