asmA.0

# Empirical Risk Minimization for Time Series:
Nonparametric Performance Bounds for Prediction

Christian Brownlees∗ Jordi Llorens-Terrazas Universitat Pompeu Fabra and
Barcelona Graduate School of Economics.
e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
We would like to thank Gabor Lugosi, Geert Mesters, David Rossell, Francesco
Violante and Piotr Zwiernik for providing numerous helpful comments. We would
also like to thank participants at the finance research unit seminar,
univeristy of Copenhagen (May 10, 2021) and finance and insurance seminar,
CREST (May 27, 2021). Christian Brownlees acknowledges support from the
Spanish Ministry of Science and Technology (Grant MTM2012-37195) and the
Spanish Ministry of Economy and Competitiveness through the Severo Ochoa
Programme for Centres of Excellence in R&D (SEV-2011-0075).

(First Draft: July 1, 2021
This Draft: )

###### Abstract

Empirical risk minimization is a standard principle for choosing algorithms in
learning theory. In this paper we study the properties of empirical risk
minimization for time series. The analysis is carried out in a general
framework that covers different types of forecasting applications encountered
in the literature. We are concerned with 1-step-ahead prediction of a
univariate time series generated by a parameter-driven process. A class of
recursive algorithms is available to forecast the time series. The algorithms
are recursive in the sense that the forecast produced in a given period is a
function of the lagged values of the forecast and of the time series. The
relationship between the generating mechanism of the time series and the class
of algorithms is unspecified. Our main result establishes that the algorithm
chosen by empirical risk minimization achieves asymptotically the optimal
predictive performance that is attainable within the class of algorithms.

Keywords: Empirical risk minimization, oracle inequality, time series,
forecasting, Markov chain.

JEL: C14, C22, C53, C58.

##  1 Introduction

Empirical risk minimization is a standard principle for choosing algorithms in
learning theory (Vapnik and Chervonenkis, 1971; Devroye et al., 1996). Simply
put, empirical risk minimization consists in choosing the algorithm that
minimizes the empirical risk. One of the main goals of learning theory is to
establish bounds on the predictive performance of the algorithm that minimizes
the empirical risk relative to the optimal performance attainable in a given
class of algorithms. A key feature of learning theory is its nonparametric
nature, in the sense that performance bounds are typically obtained under the
assumption that the generating mechanism of the data is unknown. Despite the
fact that empirical risk minimization is a general principle and widely
applicable, the majority of contributions in this area focus on the analysis
of i.i.d. data.

In this paper we study empirical risk minimization for time series. Our
analysis is carried out in a general framework that allows to study different
types of forecasting applications. We are concerned with 1-step-ahead
prediction of a univariate stationary time series generated by a (possibly
nonlinear) parameter-driven process. The class of processes we entertain is
fairly broad and it includes linear state space and stochastic volatility
models. A class of recursive algorithms is available to predict the time
series. The algorithms are recursive in the sense that the forecast produced
in a given period is a function of the lagged values of the forecast and the
time series. The class we consider is inspired by threshold models (Tong,
1990) and it includes as special cases the prediction formulae/filters of ARMA
and GARCH models. The prediction accuracy of the forecasts is measured by a
loss function in the Bregman class (Bregman, 1967; Banerjee et al., 2005;
Laurent et al., 2013; Patton, 2020), which includes the loss functions
typically used for the estimation of ARMA and GARCH models. Our analysis is
nonparametric in the sense that the relationship between the data generating
mechanism of the time series and the class of algorithms is unspecified.

The main result of this paper consists in establishing an oracle inequality
that provides non-asymptotic guarantees on the predictive performance of
empirical risk minimization. The oracle inequality implies that empirical risk
minimization is consistent, in the sense that the algorithm chosen by
empirical risk minimization achieves asymptotically the optimal predictive
performance that can be attained within the class of algorithms considered. In
particular, our result implies that ARMA/GARCH prediction based on the
standard Gaussian maximum likelihood estimator achieves the optimal predictive
performance even when the conditional mean/conditional variance equation of
the model is misspecified.

The main result is illustrated by a number of basic applications of the
general framework. We consider forecasting time series generated by an AR(1)
plus noise model as well as a stochastic volatility model. In the case of
stochastic volatility we consider forecasting on the basis of returns or a
realized volatility measure. Last, as a side result, we show that our class of
algorithms can be interpreted as the solution of a sequential optimization
problem that consists in minimizing an appropriately defined measure of
tracking error of the algorithm.

The main result follows from five intermediate propositions. We begin by
establishing existence of moments and strong mixing conditions of a joint
process that includes the time series and the algorithm (Proposition 1).
Importantly, the strong mixing coefficients are bounded by a function with
geometric decay uniformly over the class of algorithms. Next we establish a
general inequality that states that the performance of empirical risk
minimization can be controlled by the sum of two quantities (Proposition 2).
The first is the supremum of an average of differences between conditional and
unconditional expectations and the second is the supremum of the empirical
process associated with the prediction loss of the algorithm. The first term
is bounded using an inequality from Ibragimov (Proposition 3). The second term
is bounded using a covering argument (Proposition 4) and a concentration
inequality for strong mixing processes (Proposition 5).

Proposition 1 contains the main novel idea of the paper. The result builds
upon the literature on nonlinear time series models and Markov chains
(Bougerol and Picard, 1992; Lanne and Saikkonen, 2005; Francq and Zakoïan,
2006; Meitz and Saikkonen, 2008; Kristensen, 2009). The novelty with respect
to the literature consists in using Markov chain theory to establish moment
and dependence properties of an _algorithm_ , as opposed to a _model_. More
precisely, the strategy consists in embedding the time series and the
algorithm in what we name a companion Markov chain. We then show that the
companion Markov chain is $V$-geometric ergodic, which implies existence of
moments and strong mixing of the time series and the algorithm (Meyn and
Tweedie, 1993). The uniform bound on the strong mixing coefficients is
established using results by Roberts and Rosenthal (2004). This approach is
motivated by the fact that while it can be challenging to characterize the
moment and dependence properties of general nonlinear processes, a number of
tools are available to establish these properties for Markov nonlinear
processes (Carrasco and Chen, 2002). We emphasize that the result does not
hinge on the approximation properties of the class of algorithms.

Four remarks are in order before we proceed. First, empirical risk
minimization has a number of analogies with quasi-maximum likelihood
estimation for ARMA/GARCH models. Important research in the area includes Lee
and Hansen (1994), Lumsdaine (1996), Ling and McAleer (2003), Francq and
Zakoïan (2004), Kristensen and Rahbek (2005) and Straumann and Mikosch (2006).
Contributions in this literature typically assume that the conditional
mean/conditional variance equation of the time series is known whereas the
innovation distribution is not. Interest then lies in estimating the
parameters of the conditional mean/conditional variance equation. The main
difference with these contributions is that in this paper the relationship
between the data generating process and the algorithm is unspecified. In
particular, the class of algorithms may not contain the conditional
mean/conditional variance of the time series.

Second, empirical risk minimization for time series is closely related to
M-estimation for dependent data. Classic references in this area include
Gallant and White (1988) and Pötscher and Prucha (1997), which develop general
theory on the basis of fairly high-level assumptions. Gallant and White (1988)
rely, among other requirements, on uniform NED and dominance conditions on the
objective function of M-estimation. We remark that checking that these
conditions hold is not always straightforward. Instead, in this paper we rely
on primitive assumptions to establish that conditions akin uniform NED and
dominance hold.

Third, empirical risk minimization is close in spirit to nonparametric time
series modeling. Important research in this area includes Pagan and Schwert
(1990), Masry and Tjøstheim (1995), Härdle and Tsybakov (1997) and Linton and
Mammen (2005). This literature focuses on developing nonparametric estimation
techniques to estimate the time series model that has generated the data. The
main difference with these contributions is that we are not concerned with
estimating the time series model that has generated the data and that interest
solely lies in choosing an optimal algorithm for prediction within a given
class.

Fourth, this paper contributes to the literature on empirical risk
minimization for dependent data. Besides a number of notable contributions,
this literature is not extensive.111We remark that nontrivial technical
challenges arise with dependent data. Mendelson (2015) argues that some of the
standard techniques used in learning theory cannot be extended beyond i.i.d.
and bounded data setup. Two closely related contributions are Jiang and Tanner
(2010) and Brownlees and Guðmundsson (2021), which study empirical risk
minimization for regression. The class of algorithms considered in these
papers depends on a finite number of lags of the time series. In such a
setting it is typically straightforward to obtain the dependence properties of
the joint system composed of the times series and the algorithm. The strategy
adopted in both papers consists in assuming that the time series is strong
mixing and then applying standard results for functions of strong mixing
processes to obtain that the joint process is also strong mixing. Such an
approach is not viable in the framework of this paper. In our setup forecasts
depend on the entire past history of the time series. In this case standard
results for functions of strong mixing processes do not provide useful
results.

The rest of the paper is structured as follows. Section 2 introduces the basic
framework. Section 3 presents empirical risk minimization and the main result
of the paper. Section 4 contains applications. Section 5 outlines the proof of
the main result. Concluding remarks follow in Section 6. Proofs are in the
Appendix.

##  2 Basic Definitions and Assumptions

#### Data generating process.

We are concerned with 1-step-ahead prediction of a stationary time series
$\\{Y_{t},t\geq 0\\}$ generated by a parameter-driven process. The process
$\\{Y_{t},t\geq 0\\}$ takes values in $\mathcal{Y}\subseteq\mathbb{R}$ and is
defined as $Y_{0}=y\in\mathcal{Y}$, and

$\displaystyle Y_{t}$ $\displaystyle=$ $\displaystyle
g_{y1}(H_{t})+g_{y2}(H_{t})\epsilon_{Y\,t}~{},\hskip 10.00002ptt\geq 1~{},$
(1)

where $\\{H_{t},t\geq 0\\}$ is a hidden process, $\\{\epsilon_{Y\,t},t\geq
1\\}$ is an i.i.d. sequence of random variables and $g_{y1}$ and $g_{y2}$ are
Borel-measurable real functions. The process $\\{H_{t},t\geq 0\\}$ takes
values in $\mathcal{H}=\mbox{int}(\mathcal{Y})$ and is defined as
$H_{0}=h\in\mathcal{H}$, and

$\displaystyle H_{t}$ $\displaystyle=$ $\displaystyle
g_{h1}(H_{t-1})+g_{h2}(H_{t-1})\epsilon_{H\,t}~{},\hskip 10.00002ptt\geq
1~{},$ (2)

where $\\{\epsilon_{H\,t},t\geq 1\\}$ is an i.i.d. sequence of random
variables and $g_{h1}$ and $g_{h2}$ are Borel-measurable real functions. We
remark that in our framework, depending on the application, the target time
series $\\{Y_{t},t\geq 0\\}$ may denote some appropriate transformation of the
data. For example, in volatility forecasting using stock returns, where
interest lies in predicting the 1-step-ahead scale of stock returns, the time
series $\\{Y_{t},t\geq 0\\}$ may be defined as the squared return process.

The data generating process satisfies the following set of assumptions.

###### A.1 (Data generating process).

  1. (i)

The functions $g_{h1}$ and $g_{h2}$ are bounded on bounded subsets of
$\mathbb{R}$. There exist positive constants $a_{h}$ and $b_{h}$ such that
$|g_{h1}(h)|\leq a_{h}|h|+o(|h|)$ as $|h|\rightarrow\infty$ and
$|g_{h2}(h)|\leq b_{h}|h|+o(|h|)$ as $|h|\rightarrow\infty$. The function
$g_{h2}$ satisfies $\inf_{h}{|g_{h2}(h)|}>0$.

  2. (ii)

The functions $g_{y1}$ and $g_{y2}$ are bounded on bounded subsets of
$\mathbb{R}$. There exist positive constants $C_{y1}$ and $C_{y2}$ such that
$|g_{y1}(h)|\leq C_{y1}|h|$ and $|g_{y2}(h)|\leq C_{y2}(1\vee|h|)$. The
function $g_{y1}$ satisfies $\inf_{h}g_{y1}(h)\geq 0$ when
$\mathcal{Y}=\mathbb{R}_{+}$. The function $g_{y2}$ satisfies
$\inf_{h}g_{y2}(h)>0$.

  3. (iii)

The random vector $\epsilon_{t}=(\epsilon_{Y\,t},\epsilon_{H\,t})^{\prime}$
has a distribution that is absolutely continuous with respect to Lebesgue
measure on $\mathbb{R}^{2}$ and is supported on
$(\underline{\epsilon},\infty)^{2}$ with $\underline{\epsilon}=-\infty$ when
$\mathcal{Y}=\mathbb{R}$ and $\underline{\epsilon}=0$ when
$\mathcal{Y}=\mathbb{R}_{+}$. The joint density $\phi(\epsilon_{t})$ of the
random vector $\epsilon_{t}$ satisfies
$\phi(\epsilon_{t})=\phi_{Y}(\epsilon_{Y\,t})\phi_{H}(\epsilon_{H\,t})$, where
$\phi_{Y}$ and $\phi_{H}$ are densities that are bounded away from zero on
compact subsets of $(\underline{\epsilon},\infty)$. The random variables
$\epsilon_{Y\,t}$ and $\epsilon_{H\,t}$ satisfy
$\mathbb{E}\epsilon_{H\,t}^{2r_{m}}<\infty$,
$\mathbb{E}\epsilon_{Y\,t}^{2r_{m}}<\infty$ for some $r_{m}\geq 6$. The random
variable $\epsilon_{Y\,t}$ satisfies
$\mathbb{E}\left(\log\epsilon_{Y\,t}\right)^{2r_{m}}<\infty$ when
$\mathcal{Y}=\mathbb{R}_{+}$.

  4. (iv)

The condition $\mathbb{E}(a_{h}+b_{h}|\epsilon_{H\,t}|)^{2r_{m}}<1$ holds.

Assumption 1 is similar to standard assumptions used to establish geometric
ergodicity of nonlinear time series models (Masry and Tjøstheim, 1995; Lu and
Jiang, 2001; Lanne and Saikkonen, 2005; Meitz and Saikkonen, 2008) and it
allows for a fairly broad class of parameter-driven processes. Note that the
$\\{Y_{t},t\geq 0\\}$ process can take values on either
$\mathcal{Y}=\mathbb{R}$ or $\mathcal{Y}=\mathbb{R}_{+}$ (assumptions differ
slightly depending on these two cases). This allows us to cover different
types of forecasting applications encountered in the literature.

Assumption 1$(i)$ is similar to Assumption 3.2 in Masry and Tjøstheim (1995)
and it implies that (2) is dominated asymptotically by a stable linear model.
As Masry and Tjøstheim (1995) emphasize, such a requirement is mild, since
functions that grow everywhere faster than a stable linear model are
nonstationary.

Assumption 1$(ii)$ allows for a fair amount of flexibility in equation (1). In
particular, it requires $|Y_{t}|$ to be bounded from above by a linear
function of $|H_{t}|$.

Assumption 1$(iii)$ imposes conditions on the random variables
$\epsilon_{H\,t}$ and $\epsilon_{Y\,t}$ that are, for the most part, analogous
to standard conditions used in the literature. The less standard requirement
is assuming, when $\mathcal{Y}=\mathbb{R}_{+}$, that $2r_{m}$ moments of
$\log\epsilon_{Y\,t}$ exist. This guarantees that the moments of some of the
loss functions considered in this paper are finite. The assumption is
relatively mild. For example, it allows for distributions with density bounded
from above in a neighborhood of zero (e.g. the exponential) as well as certain
distributions with unbounded density (e.g. chi-square with one degree of
freedom). Finally, the independence assumption between $\epsilon_{H\,t}$ and
$\epsilon_{Y\,t}$ may be relaxed at the expense of more tedious proofs.

Assumption 1$(iv)$ is a stability condition analogous to the one assumed in
Masry and Tjøstheim (1995) or Lanne and Saikkonen (2005).

#### Algorithms.

A class of recursive algorithms indexed by
$\theta\in\Theta\subset\mathbb{R}^{p}$ and denoted by $\\{f_{\theta\,t},t\geq
0\\}$ is available to predict 1-step-ahead the time series $\\{Y_{t},t\geq
0\\}$. The process $\\{f_{\theta\,t},t\geq 0\\}$ takes values in
$\mathcal{F}\subseteq\mathbb{R}$ and is defined as
$f_{\theta\,0}=f\in\mathcal{F}$ and

$f_{\theta\,t}=\sum_{k=1}^{K}(\alpha_{0\,k}+\alpha_{1\,k}Y_{t-1}+\beta_{1\,k}f_{\theta\,t-1})\mathbbm{1}_{t-1\,k}~{},\hskip
10.00002ptt\geq 1~{},$ (3)

where
$\theta=(\alpha_{0\,1},\ldots,\alpha_{0\,K},\alpha_{1\,1},\ldots,\alpha_{1\,K},\beta_{1\,1},\ldots,\beta_{1\,K})^{\prime}$
with $K=p/3$, $\mathbbm{1}_{t\,k}=\mathbbm{1}_{\\{Y_{t}\in\mathcal{Y}_{k}\\}}$
and $\\{\mathcal{Y}_{1},\ldots,\mathcal{Y}_{K}\\}$ is a known partition of
$\mathcal{Y}$ made of $K$ sets referred to as regimes. The partition is of the
form $\\{(r_{1},r_{2}),[r_{2},r_{3}),\dots,[r_{K},\infty)\\}$ with
$-\infty=r_{1}<r_{2}<\dots<r_{K}<\infty$ when $\mathcal{Y}=\mathbb{R}$ and
$\\{[r_{1},r_{2}),[r_{2},r_{3}),\dots,[r_{K},\infty)\\}$ with
$0=r_{1}<r_{2}<\dots<r_{K}<\infty$ when $\mathcal{Y}=\mathbb{R}_{+}$. The
parameter vector $\theta$ is referred to as a prediction rule. We remark that
the class of prediction algorithms defined in (3) corresponds to the class of
1-step-ahead prediction formulae induced by the self-exciting threshold
autoregressive moving average model (SETARMA) (Tong, 1990). As it is customary
in the learning literature, the relationship between $Y_{t}$ and
$f_{\theta\,t}$ is unspecified and (3) is simply an algorithm to predict
$Y_{t}$.

The class of algorithms satisfies the following set of assumptions.

###### A.2 (Algorithms).

$(i)$ The set $\Theta\subset\mathbb{R}^{p}$ with $p=3K$ is nonempty and such
that
$\Theta\subseteq[\underline{\alpha}_{0},\overline{\alpha}_{0}]^{K}\times[\underline{\alpha}_{1},\overline{\alpha}_{1}]^{K}\times[0,\overline{\beta}_{1}]^{K}$
with
$\underline{\epsilon}<\underline{\alpha}_{0}<\overline{\alpha}_{0}<\infty$,
$0<\underline{\alpha}_{1}<\overline{\alpha}_{1}<\infty$ and
$\overline{\beta}_{1}<1$. $(ii)$ The number of regimes $K$ satisfies
$K<(r_{m}-2)/3$.

The process $\\{f_{\theta\,t},t\geq 0\\}$ takes values in
$\mathcal{F}=\mathbb{R}$ when $\mathcal{Y}=\mathbb{R}$ and
$\mathcal{F}=[\underline{\alpha}_{0},\infty)$ when
$\mathcal{Y}=\mathbb{R}_{+}$.

Assumption 2$(i)$ is mild and imposes constraints on the class of prediction
rules $\Theta$ that are analogous to standard constraints imposed in the
analysis of quasi-maximum likelihood estimators of ARMA and GARCH models
(Francq and Zakoïan, 2010). We remark that when $\mathcal{Y}=\mathbb{R}$ the
constraint $\beta_{1\,k}\in[0,\overline{\beta}_{1}]$ may be relaxed to
$\beta_{1\,k}\in[-\overline{\beta}_{1},\overline{\beta}_{1}]$ at the expense
of more tedious proofs.

Assumption 2$(ii)$ states that the size of the class of prediction rules is
bounded by a linear function of the number of moments of $\epsilon_{Y\,t}$ and
$\epsilon_{H\,t}$.

#### Loss function.

The prediction accuracy of the algorithm is measured by a loss function that
belongs to the Bregman class. Let $\psi:\mathcal{S}\rightarrow\mathbb{R}$ be a
strictly convex and continuously differentiable function defined over a convex
set $\mathcal{S}\subseteq\mathbb{R}$. Then, the Bregman loss associated with
$\psi$ for predicting $Y_{t}$ with $f_{\theta\,t}$ is defined as

$L(Y_{t},f_{\theta\,t})=\psi(Y_{t})-\psi(f_{\theta\,t})-\nabla\psi(f_{\theta\,t})(Y_{t}-f_{\theta\,t})~{}.$
(4)

The Bregman class is a fairly large and tractable family of losses. In
particular, the log-likelihood of random variables in the regular exponential
family can be expressed as the (negative) sum of Bregman losses (up to a
constant term) (Banerjee et al., 2005). Thus, the Bregman class includes the
standard loss functions used for quasi-maximum likelihood estimation of time
series models.

In this paper we focus exclusively on losses that satisfy the following
condition.

######  Condition 1 (Bregman).

The loss $L$ is such that (i) $Y_{t}\in\mathcal{S}$ a.s. for all $t\geq 0$,
(ii) $\sup_{\theta\in\Theta}\mathbb{E}(L(Y_{t},f_{\theta\,t}))^{r_{m}}<\infty$
for all $t\geq 0$ and (iii) $L(f_{\theta_{1}\,t},f_{\theta_{2}\,t})\leq
C_{\psi}(f_{\theta_{1}\,t}-f_{\theta_{2}\,t})^{2}$ a.s. for all $t\geq 0$, for
any $\theta_{1},\theta_{2}\in\Theta$ and for some positive constant
$C_{\psi}$.

Table 1 contains a number of Bregman losses that satisfy Condition 1 given
Assumptions 1 and 2. We remark that when $\mathcal{Y}=\mathbb{R}$ only the
first two losses are admissible whereas when $\mathcal{Y}=\mathbb{R}_{+}$ all
the losses in the table are allowed. The table contains both well known and
lesser known loss functions.222The random variables listed in Table 1 are all
the random variables in the natural exponential family with quadratic variance
function and unbounded support (Morris, 1982). The table includes the loss
that corresponds to the log-likelihood of the Gaussian (with known variance)
with respect to the mean parameter, which is the classic square loss. This
loss function is typically used for maximum likelihood estimation of ARMA
models. The table also contains the loss associated with the log-likelihood of
the NEF-GHS (with known number of convolutions) with respect to the natural
parameter (Morris, 1982). The NEF-GHS is a flexible distribution taking values
on the real line that allows for skewness and higher kurtosis than the
Gaussian. In addition, compared to the square loss it has the advantage that
the $r$-th moment of the loss only requires the existence of the $r$-th moment
of its arguments. To the best of our knowledge this loss function/distribution
has not been used extensively in the time series literature. Next, the table
includes the loss associated with the log-likelihood of the gamma (with known
shape) with respect to the mean parameter, which in the volatility forecasting
literature is known as the QLIKE loss (Patton, 2011).333The standard
definition of the QLIKE is $L(Y_{t},f_{\theta\,t})=Y_{t}/f_{\theta\,t}+\log
f_{\theta\,t}$. This is equivalent to our definition for optimization purposes
with respect to $\theta$. We recall that by appropriately constraining the
shape parameter, the gamma distribution nests the exponential and chi-square
distributions. This loss function is typically used for maximum likelihood
estimation of MEM (Engle and Gallo, 2006), ACD models (Engle and Russell,
1998) and GARCH models. Finally, the table includes the losses associated with
the log-likelihoods of the Poisson and negative binomial (with known number of
failures) with respect to the mean parameter.444We follow the convention that
$0\log 0=0$, hence $\mbox{dom}(\psi)=\mathbb{R}_{+}$ in both cases. These loss
functions are typically used for maximum likelihood estimation of dynamic
models for count data (Agosto:Cavaliere:Kristensen:Rahbek:2016; Davis et al.,
2021). We remark that our framework does not allow for $\\{Y_{t},t\geq 0\\}$
to take values on a countable set. That said, these losses satisfy our
regularity conditions and may be used for empirical risk minimization. The
analysis of empirical risk minimization when $\\{Y_{t},t\geq 0\\}$ takes
values in a countable set can be carried out using the same strategy developed
in this paper, but some of the proofs would differ.

Table 1: Regular Bregman Losses

$\mathcal{S}$ | $\psi(u)$ | $L(u,v)$ | Log-likelihood
---|---|---|---
$\mathbb{R}$ | $u^{2}$ | $(u-v)^{2}$ | Gaussian
$\mathbb{R}$ | $u\tan^{-1}(u)-{1\over 2}\log(1+u^{2})$ | $u\left[\tan^{-1}(u)-\tan^{-1}(v)\right]+{1\over 2}\log{1+v^{2}\over 1+u^{2}}$ | NEF-GHS
$\mathbb{R}_{++}$ | $-\log u$ | ${u\over v}-\log{u\over v}-1$ | Gamma
$\mathbb{R}_{+}$ | $u\log u-u$ | $u\log{u\over v}-(u-v)$ | Poisson
$\mathbb{R}_{+}$ | $u\log{u\over 1+u}-\log(1+u)$ | $u\log{u\over v}+(1+u)\log{1+v\over 1+u}$ | Negative Binomial

The table lists Bregman losses that satisfy Condition 1 given Assumptions 1
and 2.

#### Dominating process.

We introduce a dominating process $\\{d_{\theta\,t},t\geq 0\\}$ that plays a
key role in the theoretical analysis of this paper. This process bounds the
absolute difference between the forecast processes associated with two
different prediction rules. The process $\\{d_{\theta\,t},t\geq 0\\}$ takes
values in $\mathcal{D}=[1,\infty)$ and is defined as
$d_{\theta\,0}=d\in\mathcal{D}$ and

$\displaystyle
d_{\theta\,t}=1+|Y_{t-1}|+|f_{\theta\,t-1}|+\overline{\beta}_{1}d_{\theta\,t-1}~{},\hskip
10.00002ptt\geq 1~{}.$ (5)

As it is established in one of the intermediate results of this paper, this
process has the property that for any $\delta\in(0,1]$ and for any
$\theta,\dot{\theta}\in\Theta$ such that
$\|\theta-\dot{\theta}\|_{2}\leq\delta$ it holds that
$|f_{\theta\,t}-f_{\dot{\theta}\,t}|\leq\delta d_{\dot{\theta}\,t}$ for all
$t\geq 0$. This property and the generalized triangular equality for Bregman
losses imply that

$L(Y_{t},f_{\theta\,t})\leq L(Y_{t},f_{\dot{\theta}\,t})+\delta
C_{\psi}(d_{\dot{\theta}\,t}^{2}+2|Y_{t}-f_{\dot{\theta}\,t}|d_{\dot{\theta}\,t})~{},\hskip
10.00002ptt\geq 0~{}.$ (6)

##  3 Empirical Risk Minimization

We are interested in choosing a prediction rule $\theta$ from a sequence of
“in-sample” observations to forecast 1-step-ahead a sequence of “out-of-
sample” observations. The sequences of in-sample and out-of-sample
observations are respectively defined as $\\{Y_{1},\ldots,Y_{T}\\}$ and
$\\{Y_{T+1},\ldots.Y_{T+M}\\}$. The number of out-of-sample observations is
such that $M=\lceil\gamma T\rceil$ for some $\gamma>0$.

The accuracy of a prediction rule $\theta$ is measured by the out-of-sample
1-step-ahead conditional risk, which is defined as

$R(\theta)=\mathbb{E}\left[\left.{1\over
M}\sum_{t=T+1}^{T+M}L(Y_{t},f_{\theta\,t})\right|Y_{T},\ldots,Y_{1}\right]~{}.$
(7)

A natural strategy for choosing a prediction rule $\theta$ consists in picking
the one that minimizes the in-sample 1-step-ahead empirical risk. The
empirical risk minimizer (ERM) is defined as

$\hat{\theta}\in\arg\min_{\theta\in\Theta}R_{T}(\theta)~{},\text{ where
}R_{T}(\theta)={1\over T}\sum_{t=1}^{T}L(Y_{t},f_{\theta\,t})~{}.$ (8)

If more than one prediction rule achieves the minimum we may pick one
arbitrarily. In (7) and (8) we remark that $f_{\theta\,1}$ is computed using
$Y_{0}=y$ and $f_{\theta\,0}=f$ that are fixed, known and that do not depend
on $\theta$.555The initial value $Y_{0}$ can be a pre-sample observation
assumed to be fixed or a fixed value set at the outset of the analysis. Note
that when $Y_{0}$ is a pre-sample observation then the empirical risk in (8)
can be thought of as the analog of the conditional log-likelihood of $\theta$
given $Y_{0}$.

One of the goals of learning theory is to establish a bound on the performance
of the ERM relative to the optimal risk that can be achieved within the class
of prediction rules considered. We measure the accuracy of the ERM on the
basis of the conditional out-of-sample risk, which is defined as

$R(\hat{\theta})=\mathbb{E}\left[\left.{1\over
M}\sum_{t=1}^{T+M}L(Y_{t},\hat{f}_{t})\right|Y_{T},\ldots,Y_{1}\right]~{},$
(9)

where $\hat{f}_{t}=f_{\hat{\theta}\,t}$. The performance measure in (9) can be
interpreted as the out-of-sample conditional risk of the ERM obtained from the
in-sample observations. The following theorem establishes such a bound and is
our main result.

######  Theorem 1.

Suppose Assumptions 1 and 2 are satisfied. Then there exists a constant
$\sigma^{2}$ such that, for all $T$ sufficiently large, we have that

$R(\hat{\theta})\leq\inf_{\theta\in\Theta}R(\theta)+47\sigma\sqrt{p\log T\over
T}$

holds at least with probability $1-\log^{-1}T-o(\log^{-1}T)$ as
$T\rightarrow\infty$.

The inequality in Theorem 1 is commonly referred to as an _oracle inequality_
, and it provides non-asymptotic guarantees on the performance of the
ERM.666We remark that it is possible to obtain explicit bounds for the minimum
$T$ and for the probability of the oracle inequality. Moreover, it is
straightforward to see from the intermediate results of this paper that it is
possible to sharpen the rate of the probability upper bound of the oracle
inequality as well as the absolute constant. However, we have not pursued this
and we have solely focused on recovering the “classic” rate of convergence
$\sqrt{\log T/T}$. The constant $\sigma^{2}$ is application-specific and may
be interpreted as an upper bound for the long run variance of the loss
process. We define the constant precisely in Proposition 5. The rate of
convergence $\sqrt{\log T/T}$ is sometimes referred to as the classical rate
of convergence of empirical risk minimization in the learning literature for
classification with i.i.d. data (Devroye et al., 1996, Ch. 12). The theorem
implies that in our framework the ERM is consistent with respect to the class
of prediction rules $\Theta$, meaning that
$|R(\hat{\theta})-\inf_{\theta\in\Theta}R(\theta)|\stackrel{{\scriptstyle
p}}{{\rightarrow}}0$. In other words, the ERM achieves asymptotically the
optimal forecasting performance attainable within the class of algorithms
considered. We emphasize that the existence of an optimal prediction rule
$\theta^{*}=\arg\min_{\theta\in\Theta}R(\theta)$ is not required by the
theorem.

Let us highlight the stability conditions required by the theorem. These are
the stability of the data generating process (Assumption 1.$(iv)$) and the
condition $\beta_{1k}\leq\overline{\beta}_{1}<1$ for $k=1,\ldots,K$ on the
class of algorithms (Assumptions 2.$(i)$). We remark that this latter
condition differs from the standard stability conditions of SETARMA models
and, as a consequence, of ARMA and GARCH models. For simplicity, we discuss
this in the case of an ARMA(1,1). For an ARMA(1,1) model a necessary condition
for stability is $\alpha_{1\,1}+\beta_{1\,1}<1$ (using the notation of this
work). However, in our framework this constraint is not required by the class
of algorithms that corresponds to the ARMA(1,1).

###  3.1 Additional Discussion

A number of additional remarks on the framework of this paper are in order.

Our analysis studies the properties of the ERM when the time series is
generated by a parameter-driven process. Clearly, an observation-driven
process may be entertained instead. In this case, the analysis of the
performance of the ERM can be carried out using the same strategy developed in
this paper. However, some of the proofs will differ and we leave the analysis
of this case for future research.

The class of recursive algorithms we entertain is fairly flexible and builds
upon the class of threshold models that have a well established tradition in
the time series literature. We remark that our results may be extended to
alternative classes of recursive algorithms and do not inherently depend on
the functional form of the algorithmic class we consider in this paper. In
particular, our analysis does not require the class of algorithms to have
special approximation properties or to include the optimal 1-step-ahead
forecast associated with the data generating process and the loss function.
What is key in our framework is that, loosely speaking, the algorithms “forget
the past sufficiently fast”.

Instead of comparing the performance of the ERM against the optimal risk
attainable in the class, one may wish to compare against the risk of the
optimal 1-step-ahead forecast. For loss functions in the Bregman class the
optimal 1-step-ahead forecast is the conditional mean (assuming it exists)
(Banerjee et al., 2005). Thus, the risk of the optimal 1-step-ahead forecast
may be defined as

$R^{*}=\mathbb{E}\left[\left.{1\over
M}\sum_{t=T+1}^{T+M}L(Y_{t},\mu_{t})\right|Y_{T},\ldots,Y_{1}\right],$

where $\mu_{t}=\mathbb{E}(Y_{t}|Y_{t-1},\ldots,Y_{1})$ for $t>1$ and
$\mu_{1}=\mathbb{E}(Y_{1})$. The performance of the ERM relative to the risk
of the optimal 1-step-ahead foreast may be expressed as

$R(\hat{\theta})-R^{*}=\left[\inf_{\theta\in\Theta}R(\theta)-R^{*}\right]+\left[R(\hat{\theta})-\inf_{\theta\in\Theta}R(\theta)\right].$

The first term is called the approximation error and the second term is called
the estimation error (Devroye et al., 1996, Ch. 12). Notice that oracle
inequalities control the estimation error. The approximation error is
typically difficult to control, especially in a data dependent setting. There
are a number of contributions that, in some sense, attempt to control the
approximation error (Nelson, 1992). In general, the analysis of the
approximation error requires additional assumptions. For this reason learning
theory typically focuses on studying the estimation error, which is the
approach pursued in this paper.

##  4 Applications

We illustrate our framework with a number of applications. For simplicity, we
shall always consider forecasting using the class of algorithms defined in (3)
when the number of regimes is set to $K=1$.777The detailed analysis of these
applications is carried out in the Online Appendix.

#### Forecasting an AR(1) plus noise.

Consider the AR(1) plus noise model given by $Y_{0}=y\in\mathbb{R}$,
$H_{0}=h\in\mathbb{R}$ and

$\displaystyle Y_{t}$ $\displaystyle=$ $\displaystyle
H_{t}+\epsilon_{Y\,t}~{},$ $\displaystyle H_{t}$ $\displaystyle=$
$\displaystyle\mu_{H}+\varrho(H_{t-1}-\mu_{H})+\epsilon_{H\,t}~{},$

for $t\geq 1$, where $\\{\epsilon_{Y\,t},t\geq 1\\}$ and
$\\{\epsilon_{H\,t},t\geq 1\\}$ are i.i.d. sequences of Gaussian random
variables and $\varrho\in[0,1)$. The class of algorithms defined in (3) is
used for forecasting. Prediction accuracy is measured by the square loss.
Then, Assumptions 1 and 2 are satisfied and Theorem 1 holds.

We remark that in this application we have that the approximation error of the
class of algorithms converges to zero when $T$ is large (when $\Theta$ is
suitably chosen). The class of algorithms includes the steady state Kalman
filter, which implies that the class of algorithms includes a forecast process
that converges to the conditional mean of $Y_{t}$ given the past when $t$ is
large.

#### Forecasting volatility using returns.

Consider the stochastic volatility model for the return process
$\\{r_{t},t\geq 0\\}$ given by $r_{0}\in\mathbb{R}$,
$\sigma^{2}_{0}\in\mathbb{R}_{++}$ and

$\displaystyle r_{t}$ $\displaystyle=$
$\displaystyle\sqrt{\sigma^{2}_{t}}~{}z_{t}~{},$
$\displaystyle\log\sigma^{2}_{t}$ $\displaystyle=$
$\displaystyle\mu_{H}+\varrho(\log\sigma^{2}_{t-1}-\mu_{H})+\eta_{t}~{},$

for $t\geq 1$, where $\\{z_{t},t\geq 1\\}$ and $\\{\eta_{t},t\geq 1\\}$ are
i.i.d. sequences of Gaussian random variables and $\varrho\in(0,1)$. It is
straightforward to see that this model belongs to the class of data generating
processes considered in this paper (using $Y_{t}=r_{t}^{2}$,
$H_{t}=\sigma^{2}_{t}$, $\epsilon_{Y\,t}=z_{t}^{2}$ and
$\epsilon_{H\,t}=\exp(\eta_{t})$). The class of algorithms defined in (3) is
used for forecasting (using $Y_{t}=r_{t}^{2}$), which corresponds to the
1-step-ahead prediction formula of the GARCH(1,1). Prediction accuracy is
measured by the QLIKE loss. Then, Assumptions 1 and 2 are satisfied and
Theorem 1 holds.

We remark that in this application the ERM coincides with the Gaussian quasi-
maximum likelihood estimator of the GARCH(1,1). Despite the fact that the
conditional variance equation implied the GARCH(1,1) is misspecified in this
setting, our main theorem implies that the ERM chooses an algorithm with
asymptotically optimal predictive performance within the class of algorithms
considered.

#### Forecasting volatility using realized volatility.

Over the last two decades realized volatility measures have enhanced
volatility prediction (Andersen et al., 2003). Realized volatility measures
are precise estimators of the (latent) volatility computed from intra-daily
stock prices. Consider the (nonlinear) stochastic volatility model for the
realized volatility process $\\{RV_{t},t\geq 0\\}$ given by
$RV_{0}\in\mathbb{R}_{++}$, $\sigma^{2}_{0}\in\mathbb{R}_{++}$ and

$\displaystyle RV_{t}$ $\displaystyle=$
$\displaystyle\sigma^{2}_{t}\epsilon_{RV\,t}~{},$
$\displaystyle\sigma^{2}_{t}$ $\displaystyle=$ $\displaystyle
g_{h1}(\sigma^{2}_{t-1})+g_{h2}(\sigma^{2}_{t-1})\epsilon_{\sigma^{2}\,t}~{},$

for $t\geq 1$, where $\\{\epsilon_{RV\,t},t\geq 1\\}$ and
$\\{\epsilon_{\sigma^{2}\,t},t\geq 1\\}$ are i.i.d. sequences of gamma random
variables and $g_{y1}$ and $g_{y2}$ are Borel-measurable real functions that
satisfy Assumption 1.$(i)$. We assume that $\epsilon_{RV\,t}$ is unit mean,
which implies that $RV_{t}$ is a conditionally unbiased proxy for the
volatility $\sigma^{2}_{t}$.888The realized volatility measurement error in
the model is multiplicative. The analysis of this section can also be carried
out in the case of an additive measurement error. The class of algorithms
defined in (3) is used for forecasting, which corresponds to the 1-step-ahead
prediction formula of the MEM(1,1) or ARMA(1,1). Prediction accuracy is
measured by the QLIKE or the square loss. Then, Assumptions 1 and 2 are
satisfied and Theorem 1 holds.

We remark that the model considered in this application should be interpreted
as a reduced form approximation. Meddahi (2003) has derived the discrete-time
representation of the realized volatility process implied by a fairly general
class of continuous-time stochastic volatility models commonly encountered in
the literature. These results imply that a more flexible framework than the
one considered here is required to allow for such data generating processes,
and we have not pursued to accommodate this.999Meddahi (2002) establishes that
for a general diffusion, the measurement error of realized volatility depends
on the entire path of the spot volatility. Our framework is consistent with
the continuous-time stochastic volatility model with constant intra-daily
volatility used in Patton (2011).

Theorem 1 implies that the ERM achieves the optimal performance for realized
volatility prediction. However, interest typically lies in forecasting the
latent volatility process $\\{\sigma^{2}_{t},t\geq 0\\}$ rather than its noisy
measurement. Building upon (Hansen and Lunde, 2006; Patton, 2011) we establish
further properties of the ERM. We measure the accuracy of a prediction rule
$\theta$ for predicting the volatility process $\\{\sigma^{2}_{t},t\geq 0\\}$
using the out-of-sample 1-step-ahead conditional risk

$R_{\mathrm{Vol}}(\theta)=\mathbb{E}\left[\left.{1\over
M}\sum_{t=1}^{T+M}L(\sigma^{2}_{t},f_{\theta\,t})\right|RV_{T},\ldots,RV_{1}\right]~{}.$

The loss in predicting the volatility process $\\{\sigma^{2}_{t},t\geq 0\\}$
satisfies

$L(\sigma^{2}_{t},f_{\theta\,t})=L(RV_{t},f_{\theta\,t})+L(\sigma^{2}_{t},RV_{t})-(\sigma^{2}_{t}-RV_{t})(\nabla\psi(f_{\theta\,t})-\nabla\psi(RV_{t}))~{},$
(10)

which follows from the generalized triangular equality for Bregman losses. In
(10) we have that the second term does not depend on the algorithm and the
third term has a conditional expectation of zero given the past. The
decomposition in (10) and Theorem 1 imply that

$\left|R_{\mathrm{Vol}}(\hat{\theta})-\inf_{\theta\in\Theta}R_{\mathrm{Vol}}(\theta)\right|=\left|R(\hat{\theta})-\inf_{\theta\in\Theta}R(\theta)\right|\stackrel{{\scriptstyle
p}}{{\rightarrow}}0~{}.$

Thus, the ERM based on the noisy realized volatility measure chooses an
algorithm with optimal performance for volatility forecasting (within the
class of algorithms) provided that the realized volatility measure is
conditionally unbiased.

#### Recursive prediction as a solution of a sequential optimization problem.

The class of algorithms defined in (3) was introduced without any
justification other than its close connection to standard models used in the
literature. In this section we show that this class of algorithms may be
motivated as the solution of a sequential optimization problem. The analysis
is inspired by the research by Creal et al. (2013) and Harvey (2013) on
GAS/DCS models and by Gijbels et al. (1999); Harvey and Chakravarty (2008) on
the relation between nonparametric estimators and time series models.101010The
1-step-ahead prediction formula implied by GAS/DCS models is sometimes
motivated as the approximate solution of a local estimation problem based on a
generic (and sufficiently regular) likelihood function. The class of
algorithms we introduce for 1-step-ahead prediction can be interpreted as the
exact solution of a local estimation problem based on a Bregman loss.

Let $\\{f_{t},t\geq 0\\}$ be defined as $f_{0}=f\in\mathcal{F}$ and

$\displaystyle f_{t}=\arg\min_{f\in\text{int}(\mathcal{S})}Q_{t}(f)~{},$ (11)

where $Q_{t}$ is the tracking error function defined as

$Q_{t}(f)=w_{1}L(\bar{f},f)+w_{2}L(Y_{t-1},f)+w_{3}L(f_{t-1},f)~{},$

where $L$ denotes a loss in the Bregman family and
$(w_{1},w_{2},w_{3})=w\in\Delta^{3}$ with $\Delta^{3}$ denoting the
3-dimensional simplex. The tracking error is a convex combination of the
divergences with respect to the constant $\bar{f}$, the previous observation
and the previous forecast. If $f_{0}=\bar{f}$,111111We remark that the choice
$f_{0}=\bar{f}$ is made only for expository purposes, as it simplifies the
notation in (12). This would imply that the initial value for the forecast
process is determined by empirical risk minimization, which we do not cover in
our framework.  it is straightforward to verify that

$\displaystyle Q_{t}(f)\propto\sum_{i=0}^{t-1}k\left({x_{t}-x_{t-i}\over
h}\right)L\left(Y_{t-i-1},f\right)+\lambda L(\bar{f},f)~{},$ (12)

where $\\{x_{t},t\geq 0\\}$ is a deterministic sequence defined as $x_{t}=t$
for each $t\geq 0$, $k(u)=\exp(u)\mathbbm{1}_{\\{u\leq 0\\}}$,
$h=1/\ln(w_{3})$ and $\lambda=w_{2}^{-1}-\sum_{i=1}^{t}w_{3}^{i-1}$. Thus, the
tracking error can equivalently be thought of as the objective function of a
local constant regression plus a regularization term that penalizes deviations
from the constant $\bar{f}$. The solution of this optimization problem is

$\displaystyle f_{t}=w_{1}\bar{f}+w_{2}Y_{t-1}+w_{3}f_{t-1}~{},$ (13)

which coincides with the class of algorithms in (3) provided that
$\theta=(\alpha_{0},\alpha_{1},\beta_{1})^{\prime}$ with
$\alpha_{0}=w_{1}\bar{f}$, $\alpha_{1}=w_{2}$ and $\beta_{1}=w_{3}$. Note that
empirical risk minimization may be interpreted as choosing the set of weights
$w$ and the constant $\bar{f}$ in the objective function $Q_{t}$ that minimize
the in-sample empirical prediction loss.

##  5 Proof of Theorem 1

###  5.1 Companion Markov Chain

The first step of our analysis consists in introducing a companion Markov
chain associated with the process $\\{(Y_{t},f_{\theta\,t})^{\prime},t\geq
0\\}$. We recall a number of notions from Markov chain theory. Notation and
definitions are based on Meyn and Tweedie (1993). The discrete-time process
$\\{X_{t},t\geq 0\\}$ is a time-homogeneous Markov chain with state space
$\mathcal{X}\subseteq\mathbb{R}^{d}$ and equipped with a Borel
$\sigma$-algebra $\mathcal{B}(\mathcal{X})$ if for each $n\in\mathbb{N}$ there
exists an $n$-step transition probability kernel
$P^{n}_{X}:\mathcal{X}\times\mathcal{B}(\mathcal{X})\rightarrow[0,1]$ such
that $P^{n}_{X}(x,A)=\mathbb{P}(X_{t+n}\in A|X_{t}=x)$ for all
$t\in\mathbb{Z}_{+}$. As customary, $P^{1}_{X}(x,A)$ is denoted by
$P_{X}(x,A)$. We use $\pi_{X}:\mathcal{B}(\mathcal{X})\rightarrow[0,1]$ to
denote the invariant measure of the Markov chain (assuming it exists), that
is, the probability measure such that for each $A\in\mathcal{B}(\mathcal{X})$
it holds that $\pi_{X}(A)=\int_{\mathcal{X}}\pi_{X}(dx)P_{X}(x,A)$.

Define the companion Markov chain $\\{X_{\theta\,t},t\geq 0\\}$ that takes
values in $\mathcal{X}=\mathcal{H}\times\mathcal{F}\times\mathcal{D}$ and is
given by
$X_{\theta\,0}=x=(h,f,d)^{\prime}\in\mathcal{H}\times\mathcal{F}\times\mathcal{D}$
and

$\displaystyle X_{\theta\,t}=\begin{bmatrix}H_{t}\\\ f_{\theta\,t}\\\
d_{\theta\,t}\end{bmatrix}=\begin{bmatrix}g_{h1}(H_{t-1})+g_{h2}(H_{t-1})Z_{1\,t}\\\
\sum_{k=1}^{K}\\{\alpha_{0\,k}+\alpha_{1\,k}\left[g_{y1}(H_{t-1})+g_{y2}(H_{t-1})Z_{2\,t}\right]+\beta_{1\,k}f_{\theta\,t-1}\\}\mathbbm{1}_{t-1\,k}\\\
1+|g_{y1}(H_{t-1})+g_{y2}(H_{t-1})Z_{2\,t}|+|f_{\theta\,t-1}|+\overline{\beta}_{1}d_{\theta\,t-1}\end{bmatrix}$
(14)

for $t\geq 1$, where
$\mathbbm{1}_{t-1\,k}=\mathbbm{1}_{\\{g_{y1}(H_{t-1})+g_{y2}(H_{t-1})Z_{2\,t}\in\mathcal{Y}_{k}\\}}$,
$Z_{1\,t}=\epsilon_{H\,t}$ and $Z_{2\,t}=\epsilon_{Y\,t-1}$. We are interested
in establishing that the companion Markov chain $\\{X_{\theta\,t},t\geq 0\\}$
is $V_{X}$-geometrically ergodic (Meyn and Tweedie, 1993; Meitz and Saikkonen,
2008).

######  Definition 1 ($V_{X}$-geometric ergodicity).

A Markov chain $\\{X_{t},t\geq 0\\}$ is $V_{X}$-geometrically ergodic if there
exists a real valued function $V_{X}:\mathcal{X}\rightarrow[1,\infty)$, a
probability measure $\pi_{X}$ on $\mathcal{B}(\mathcal{X})$, and constants
$\rho<1$ and $M_{x}<\infty$ (depending on $x$) such that

$\sup_{v:|v|\leq
V_{X}}\left|\int_{\mathcal{X}}P^{n}_{X}(x,dx_{n})v(x_{n})-\int_{\mathcal{X}}\pi_{X}(dx_{n})v(x_{n})\right|\leq\rho^{n}M_{x}~{},$
(15)

for all $x\in\mathcal{X}$ and all $n\geq 1$.

A number of remarks are in order. First, the definition implicitly assumes
that the expectation of the function $V_{X}$ with respect to the measure
$\pi_{X}$ exists. Second, a Markov chain that is $V_{X}$-geometric ergodic has
convenient moment and dependence properties. If we choose $V_{X}=1$ then we
have that (15) coincides with the definition of geometric ergodicity, which,
in turn, allows to establish $\beta$\- and $\alpha$-mixing. Moreover,
$V_{X}$-geometric ergodicity implies that the unconditional expectation of
$v(X)$ exists for any function $v$ such that $|v|\leq V_{X}$.

The following lemma establishes that the companion Markov chain
$\\{X_{\theta\,t},t\geq 0\\}$ is $V_{X}$-geometrically ergodic.

######  Lemma 1.

Suppose Assumptions 1 and 2 are satisfied. Then $\\{X_{\theta\,t},t\geq 0\\}$
is $V_{X}$-geometrically ergodic with $V_{X}(x)=1+\|x\|_{1}^{2r_{m}}$.

The proof of Lemma 1 is based on establishing that the Markov chain is
irreducible, aperiodic and satisfies the so-called drift criterion. The claim
then follows from Theorem 15.0.1 of Meyn and Tweedie (1993), which is a
classic result that is routinely employed to establish stability of nonlinear
time series models.

The following lemma establishes that the constants $\rho$ and $M_{x}$ in
Definition 1 in the case of geometric ergodicity (that is, when $V_{X}=1$) can
be chosen so that they do not depend on $\theta$.121212We omit the subscript
$\theta$ from $x$ to simplify the notation, but the dependence on $\theta$ is
understood.

######  Lemma 2.

Suppose Assumptions 1 and 2 are satisfied. Then, there exist positive
constants $\rho\in(0,1)$ and $R<\infty$ that do not depend on $\theta$ such
that $\\{X_{\theta\,t},t\geq 0\\}$ satisfies

$\sup_{v:|v|\leq
1}\left|\int_{\mathcal{X}}P^{n}_{X}(x,dx_{n})v(x_{n})-\int_{\mathcal{X}}\pi_{X}(dx_{n})v(x_{n})\right|\leq
R\tilde{V}_{X}(x)\rho^{n}~{},$ (16)

for all $x\in\mathcal{X}$ and all $n\geq 1$, and
$\tilde{V}_{X}(x)=1+\|x\|_{1}$.

The proof of Lemma 2 consists in an application of Theorem 12 of Roberts and
Rosenthal (2004). We remark that the MCMC literature has developed a number of
results that allow to establish explicit geometric ergodicity convergence
rates (Rosenthal, 1995). The important implication of Lemma 2 is that the
dependence properties of the companion Markov chain $\\{X_{\theta\,t},t\geq
0\\}$ can be characterized independently of $\theta$.

The second step of the analysis consists is using the properties of the
companion Markov chain $\\{X_{\theta\,t},t\geq 0\\}$ to establish the
properties of the joint process $\\{(Y_{t},X_{\theta\,t})^{\prime},t\geq
0\\}$. The following lemma establishes the connection between the transition
kernels of $\\{X_{\theta\,t},t\geq 0\\}$ and
$\\{(Y_{t},X_{\theta\,t})^{\prime},t\geq 0\\}$.

######  Lemma 3.

Consider the Markov chain $\\{(Y_{t},X_{\theta\,t})^{\prime},t\geq 0\\}$
defined above. Let $\pi_{Y|X}(dy|x_{t})$ denote the (invariant) conditional
distribution of $Y_{t}$ given $X_{\theta\,t}=x_{t}$. Then, its $n$-step
transition kernel is given by

$\displaystyle
P^{n}_{Y,X}((y,x),d(y_{n},x_{n}))=\pi_{Y|X}(dy_{n}|x_{n})\int_{\mathcal{H}}P_{X}^{n-1}(\tilde{x},dx_{n})P_{H}(h,dh_{1}),\quad
n\geq 2,$ (17)

where $P_{H}$ is the transition kernel of $\\{H_{t},t\geq 0\\}$, and
$\tilde{x}=\tilde{x}(y,x,h_{1})=(h_{1},\sum_{k=1}^{K}(\alpha_{0\,k}+\alpha_{1\,k}y+\beta_{1\,k}f)\mathbbm{1}_{\\{y\in\mathcal{Y}_{k}\\}},1+|y|+|f|+\overline{\beta}_{1}d)^{\prime}$.

The proof of the lemma builds upon the analysis of GARCH models of Meitz and
Saikkonen (2008). The structure given by equations (1), (2), (3) and (5)
allows us to cast $\\{(Y_{t},X_{\theta\,t})^{\prime},t\geq 0\\}$ as a Markov
chain with Dirac measure as the initial distribution. We remark that the
analysis of $\\{X_{\theta\,t},t\geq 0\\}$ differs depending on whether the
process is studied in isolation or jointly with the process $\\{Y_{t},t\geq
0\\}$. The random vector $X_{\theta\,t}$ depends on $Y_{t-1}$. When the
process $\\{X_{\theta\,t},t\geq 0\\}$ is analyzed in the joint system
$\\{(Y_{t},X_{\theta\,t})^{\prime},t\geq 0\\}$ we have that the 1-step-ahead
transition kernel of the process conditions on $Y_{t-1}$. However, when
$\\{X_{\theta\,t},t\geq 0\\}$ is analyzed in isolation we have that the
1-step-ahead transition kernel of the process does not condition on $Y_{t-1}$.

The following lemma establishes that $\\{(Y_{t},X_{\theta\,t})^{\prime},t\geq
0\\}$ inherits the moment and dependence properties of the companion Markov
chain $\\{X_{\theta\,t},t\geq 0\\}$.

######  Lemma 4.

Suppose Assumptions 1 and 2 are satisfied. Then $(i)$
$\\{(Y_{t},X_{\theta\,t})^{\prime},t\geq 0\\}$ is $V_{Y,X}$-geometrically
ergodic with $V_{Y,X}(y,x)=1+|y|^{2r_{m}}+\|x\|^{2r_{m}}_{1}$; and $(ii)$
there exist positive constants $\rho\in(0,1)$ and $R<\infty$ that do not
depend on $\theta$ such that $\\{(Y_{t},X_{\theta\,t})^{\prime},t\geq 0\\}$
satisfies

$\displaystyle\sup_{v:|v|\leq
1}\left\lvert\int_{\mathcal{Y}\times\mathcal{X}}[P^{n}_{Y,X}((y,x),d(y_{n},x_{n}))-\pi_{Y,X}(d(y_{n},x_{n}))]v(y_{n},x_{n})\right\rvert\leq
R\tilde{V}_{X}(\check{x})\rho^{n},$

for all $(y,x)^{\prime}\in\mathcal{Y}\times\mathcal{X}$ and for all $n\geq 2$,
and
$\check{x}=(h,\overline{\alpha}_{0}+\overline{\alpha}_{1}|y|+\overline{\beta}_{1}|f|,1+|y|+|f|+\overline{\beta}_{1}d)^{\prime}$.

Finally, we establish the moment and dependence properties of
$\\{(Y_{t},X_{\theta\,t})^{\prime},t\geq 0\\}$. We introduce some further
notation. We define the $L_{r}$ norm of a random variable $X$ as
$\|X\|_{L_{r}}=\left(\mathbb{E}|X|^{r}\right)^{1/r}$ for any $r\in[1,\infty)$.
We define the $\alpha$-mixing coefficients of the process
$\\{(Y_{t},X_{\theta\,t}^{\prime})^{\prime},t\geq 0\\}$ as

$\alpha(l)=\sup_{A\in\mathcal{F}_{-\infty}^{s},B\in\mathcal{F}_{s+l}^{\infty}}\left|{\mathbb{P}\left(A\cap
B\right)-\mathbb{P}\left(A\right)\mathbb{P}\left(B\right)}\right|~{},$

where $\mathcal{F}_{-\infty}^{s}$ and $\mathcal{F}_{s+l}^{\infty}$ denote the
$\sigma$-algebras generated by
$\\{(Y_{t},X_{\theta\,t}^{\prime})^{\prime}:0\leq t\leq s\\}$ and
$\\{(Y_{t},X_{\theta\,t}^{\prime})^{\prime}:s+l\leq t\leq\infty\\}$
respectively.

######  Proposition 1.

Suppose Assumptions 1 and 2 are satisfied. Then, the process
$\\{(Y_{t},X_{\theta\,t})^{\prime},t\geq 0\\}$ $(i)$ satisfies
$\|Y_{t}\|_{L_{2r_{m}}}<\infty$, $\|H_{t}\|_{L_{2r_{m}}}<\infty$,
$\sup_{\theta\in\Theta}\|f_{\theta\,t}\|_{L_{2r_{m}}}<\infty$ and
$\sup_{\theta\in\Theta}\|d_{\theta\,t}\|_{L_{2r_{m}}}<\infty$; and $(ii)$ has
$\alpha$-mixing coefficients that satisfy
$\alpha(l)\leq\exp\left(-C_{\alpha}l^{r_{\alpha}}\right)$ for some
$C_{\alpha}>0$ and $r_{\alpha}>0$ that do not depend on $\theta$.

###  5.2 Establishing Performance Bounds for the ERM

We introduce a general inequality to bound the performance of the ERM.

######  Proposition 2.

Let $\overline{R}(\theta)=\mathbb{E}R(\theta)$. Then, it holds that

$\displaystyle R(\hat{\theta})-\inf_{\theta\in\Theta}R(\theta)\leq
2\sup_{\theta\in\Theta}|R(\theta)-\overline{R}(\theta)|+2\sup_{\theta\in\Theta}|R_{T}(\theta)-\overline{R}(\theta)|~{}.$
(18)

It is important to emphasize that Proposition 2 is a general result that only
requires the loss process to be a stationary sequence. We note that when the
data is i.i.d. we have that $R(\theta)=\overline{R}(\theta)$ and the
inequality in Proposition 2 corresponds to the classic inequality derived in
Vapnik and Chervonenkis (1974) (Devroye et al., 1996), which is routinely used
to derive bounds on the performance of the ERM.

The first term of the inequality in (18) is the supremum of a difference
between an average of conditional and unconditional expectations. Proposition
3 bounds this term using Ibragimov’s inequality (Davidson, 1994, Theorem
14.2).

######  Proposition 3.

Suppose Assumptions 1 and 2 are satisfied. Then, for all $T$ and for
$C={12\over\gamma}\sup_{\theta\in\Theta}\|L(Y_{t},f_{\theta\,t})\|_{L_{r_{m}}}\sum_{l=1}^{\infty}e^{-\left((r_{m}-1)/r_{m}\right)C_{\alpha}l^{r_{\alpha}}}$,
it holds that

$\displaystyle\mathbb{P}\left(\sup_{\theta\in\Theta}|R(\theta)-\overline{R}(\theta)|>{C\over
2\sqrt{T}}\right)\leq{1\over\sqrt{T}}~{}.$ (19)

The second term of the inequality in (18) is the supremum of the empirical
process associated with the prediction loss process. We bound this using a
covering argument and a concentration inequality for $\alpha$-mixing
processes. More precisely, Proposition 4 is based on a covering argument
similar to Jiang and Tanner (2010). Importantly, the proof of Proposition 4
relies on the dominating process $\\{d_{\theta\,t},t\geq 0\\}$ and, in
particular, on the property spelled out in (6). Proposition 5 is based on a
Bernstein-type inequality for $\alpha$-mixing sequences from Liebscher (1996).

######  Proposition 4.

Suppose Assumptions 1 and 2 are satisfied. Then, for any
$\varepsilon\in(0,16C_{d}]$ it holds that

$\displaystyle\mathbb{P}\left(\sup_{\theta\in\Theta}|R_{T}(\theta)-\overline{R}(\theta)|>{\varepsilon\over
2}\right)$
$\displaystyle\quad\leq\left(1+{32C_{\Theta}C_{d}\over\varepsilon}\right)^{p}\sup_{\theta\in\Theta}\left[\mathbb{P}\left(\left|{1\over
T}\sum_{t=1}^{T}\widetilde{U}_{\theta\,t}\right|>{\varepsilon\over
4}\right)+\mathbb{P}\left(\left|{1\over
T}\sum_{t=1}^{T}\widetilde{V}_{\theta\,t}\right|>2C_{d}\right)\right]~{},$

where $C_{\Theta}=\sup_{\theta\in\Theta}\|\theta\|_{2}$,
$C_{d}=C_{\psi}\sup_{\theta}\|d_{\theta\,t}^{2}+2|Y_{t}-f_{\theta\,t}|d_{\theta\,t}\|_{L_{1}}$,
$\widetilde{U}_{\theta\,t}=U_{\theta\,t}-\mathbb{E}U_{\theta\,t}$,
$\widetilde{V}_{\theta\,t}=V_{\theta\,t}-\mathbb{E}V_{\theta\,t}$,
$U_{\theta\,t}=L(Y_{t},f_{\theta\,t})$ and
$V_{\theta\,t}=C_{\psi}\left(d_{\theta\,t}^{2}+2|Y_{t}-f_{\theta\,t}|d_{\theta\,t}\right)$.

######  Proposition 5.

Suppose Assumptions 1 and 2 are satisfied. Then, for all $T$ sufficiently
large and for $\varepsilon_{T}=46\sigma\sqrt{p\log T/T}$, it holds that

$\displaystyle\left(1+{32C_{\Theta}C_{d}\over\varepsilon_{T}}\right)^{p}\sup_{\theta\in\Theta}\mathbb{P}\left(\left|{1\over
T}\sum_{t=1}^{T}\widetilde{U}_{\theta\,t}\right|>{\varepsilon_{T}\over
4}\right)\leq{1\over\log T}\text{ and }$
$\displaystyle\left(1+{32C_{\Theta}C_{d}\over\varepsilon_{T}}\right)^{p}\sup_{\theta\in\Theta}\mathbb{P}\left(\left|{1\over
T}\sum_{t=1}^{T}\widetilde{V}_{\theta\,t}\right|>2C_{d}\right)\leq
o\left({1\over\log T}\right)\text{ as }T\rightarrow\infty~{},$

where $\sigma^{2}=16{r_{m}\over
r_{m}-2}\sup_{\theta\in\Theta}\|L(Y_{t},f_{\theta\,t})-\mathbb{E}L(Y_{t},f_{\theta\,t})\|_{L_{r_{m}}}^{2}(1+2\sum_{l=1}^{\infty}\exp\left(-C_{\alpha}l^{r_{\alpha}}\right)^{1-{2\over
r_{m}}})$.

It follows from Proposition 3, 4 and 5 that, for $T$ sufficiently large,

$2\sup_{\theta\in\Theta}|R(\theta)-\overline{R}(\theta)|+2\sup_{\theta\in\Theta}|R_{T}(\theta)-\overline{R}(\theta)|\leq
47\sigma\sqrt{p\log T\over T}$

holds with high probability. This fact and Proposition 2 imply Theorem 1.

##  6 Conclusions

Leo Breiman forcefully argued that there are two main philosophies to analyze
data (Breiman, 2001), the data modeling and the algorithmic modeling cultures.
The data modeling culture is based on assuming that the data is generated by a
(partially) known model whereas the algorithmic modeling culture pursues to be
agnostic about the data generating mechanism. It is fair to say that the
majority of research in the time series literature is typically carried out
trough the lens of the data modeling culture, whereas the fraction of
contributions from the algorithmic modeling perspective is meager.

In this work we take the algorithmic standpoint and study the performance of
empirical risk minimization to choose an algorithm to forecast 1-step-ahead a
time series. A key feature of the analysis is that the relationship between
the time series and the class of algorithms is unspecified. Our main result
implies that the algorithm chosen by empirical risk minimization achieves
asymptotically the optimal predictive performance that is attainable within
the class.

The algorithmic modeling culture paves the way for the development of new
forecasting strategies for time series applications. Using the tools
introduced in the nonlinear time series literature it is possible to develop
general nonparametric theory to study the properties of these algorithmic
forecasting strategies from primitive assumptions.

##  Appendix A Proofs of Section 3 and 5

To simplify the analysis and without loss of generality we assume that
$\Theta=[\underline{\alpha}_{0},\overline{\alpha}_{0}]^{K}\times[\underline{\alpha}_{1},\overline{\alpha}_{1}]^{K}\times[0,\overline{\beta}_{1}]^{K}$.
To simplify notation we write $W_{\theta\,t}=(Y_{t},X_{\theta\,t})^{\prime}$.

###### Proof of Theorem 1.

The claim follows from Propositions 1, 2, 3, 4 and 5. ∎

###### Proof of Lemma 1.

We apply Lemmas B.1, B.2 and B.3 together with Theorem 15.0.1 of Meyn and
Tweedie (1993) to obtain that $\\{X_{\theta\,t},t\geq 0\\}$ is
$q_{X}$-geometrically ergodic with
$q_{X}(x)=1+(\kappa^{\prime}\dot{x})^{2r_{m}}$, where
$\dot{x}=(|h|,|f|,|d|)^{\prime}$, and the vector $\kappa\in(0,1)^{3}$ is
defined in Lemma B.1. Moreover, it is easy to see that Lemma B.3 still holds
with $q_{X}(x)$ replaced with $q_{X}(x)/\underline{\kappa}^{2r_{m}}$, where
$\underline{\kappa}$ is the minimum of the components of $\kappa$. The claim
follows by noting that $V_{X}(x)=1+\|x\|_{1}^{2r_{m}}\leq
q_{X}(x)/\underline{\kappa}^{2r_{m}}$. ∎

###### Proof of Lemma 2.

The claim of the Lemma follows from an application of Theorem 12 by Roberts
and Rosenthal (2004). Define
$\tilde{q}_{X}(x)=1+\tilde{\kappa}_{h}|h|+\tilde{\kappa}_{f}|f|+\tilde{\kappa}_{d}|d|=1+\tilde{\kappa}^{\prime}\dot{x}$
where $\tilde{\kappa}\in(0,1)^{3}$ as well as the set
$\tilde{S}_{2\,\epsilon}=\left\\{x\in\mathcal{X}:\tilde{\kappa}^{\prime}\dot{x}\leq\tilde{M}\right\\}$.
By arguments analogous to those used to claim that $S_{2\,\epsilon}$ defined
in Lemma B.1 is small we can show that we can choose a $\tilde{\kappa}$ such
that for any $x\in\tilde{S}_{2\,\epsilon}$ and any
$A\in\mathcal{B}(\mathcal{X})$ it holds that
$P_{X}^{2}(x,A)\geq\tilde{c}_{*}\tilde{\varphi}(A)$, where
$\tilde{c}_{*}\in(0,1)$ and $\tilde{\varphi}(A)=\mu_{Leb}(A\cap\tilde{D})$ is
Lebesgue measure restricted to an open rectangular region $\tilde{D}$, which
is the analogue of $D$ defined in Lemma B.1. As we remark in the proof of
Lemma B.1 $\tilde{c}_{*}$ and $\tilde{D}$ do not depend on $\theta$.

It is easily verified that $\tilde{q}_{X}(x)$ satisfies the drift criterion by
the same arguments as in Lemma B.3, and that
$\tilde{q}_{X}(x)\leq\tilde{V}_{X}(x)$. Define
$\lambda^{-1}=1-\tilde{\gamma}_{1}+\tilde{\gamma}_{2}/(2+\tilde{M}),$ where
$\tilde{\gamma}_{1}$ and $\tilde{\gamma}_{2}$ are analogous to $\gamma_{1}$
and $\gamma_{2}$ in Lemma B.3 and do not depend on $\theta$, and
$\tilde{M}=\inf_{x\in S_{2\,\epsilon}^{c}}\tilde{q}_{X}(x)$. The proof
strategy of Theorem 12 by Roberts and Rosenthal (2004) is based on a coupling
argument. To this end we use $\\{X_{\theta\,t}^{G},t\geq 0\\}$ to denote an
independent copy of the Markov chain $\\{X_{\theta\,t},t\geq 0\\}$ started at
the stationary distribution, namely $X_{\theta\,0}^{G}\sim\pi_{X}$. We define
$B=\max\\{1,\lambda^{2}(1-\tilde{c}_{*})\overline{R}\\},$ where the constant
$\overline{R}$ is computed in Lemma LABEL:lem:integral_rosenthal from the
Online Appendix. We distinguish two cases. Note that in both cases we are
applying Lemma LABEL:lem:integral_rosenthal from the Online Appendix.
$(i)$ Suppose that $\lambda^{-1}<1$. Then the assumptions of Proposition 11
and Theorem 12 by Roberts and Rosenthal (2004) are satisfied, thus applying
the theorem we have that for any $j\in\\{1,\ldots,n\\}$,

$\sup_{v:|v|\leq
1}\left|\int_{\mathcal{X}}\left[P^{n}_{X}(x,dx_{1})-\pi_{X}(dx_{1})\right]v(x_{1})\right|\leq(1-\tilde{c}_{*})^{j}+\lambda^{-n}{B^{j-1}\over
2}\left(\tilde{q}_{X}(x)+\mathbb{E}\tilde{q}_{X}(X_{\theta\,t})\right)$

holds for all $x\in\mathcal{X}$ and all $n\geq 1$. Let
$\overline{V}=1+\|H_{t}\|_{L_{1}}+\sup_{\theta\in\Theta}\|f_{\theta\,t}\|_{L_{1}}+\sup_{\theta\in\Theta}\|d_{\theta\,t}\|_{L_{1}}$.
Obviously, $\overline{V}\geq\mathbb{E}\tilde{q}_{X}(X_{\theta\,t})$.
Furthermore, $\overline{V}<\infty$ by Lemma 1 and Proposition 1$(i)$. Set
$j=\lfloor rn\rfloor$ for sufficiently small $r>0$ so that the bound converges
to zero at a geometric rate. We now have that (16) holds with
$\rho=(1-\tilde{c}_{*})^{r}\vee\left(\lambda^{-1}B^{r}\right)<1$ and
$R=2\overline{V}$ (note that $\tilde{q}_{X}\geq 1$). The result follows since
$\rho$ and $R$ do not depend on $\theta$.
$(ii)$ In the case $\lambda^{-1}\geq 1$, we can find an enlargement of
$\tilde{S}_{2\,\epsilon}$ for which the result in $(i)$ still holds (Roberts
and Rosenthal, 2004). We choose $M^{\prime}$ such that
$2+M^{\prime}>\tilde{\gamma}_{2}/\tilde{\gamma}_{1}$. Note that
$\tilde{S}^{\prime}_{2\,\epsilon}=\\{x\in\mathcal{X}:\tilde{\kappa}^{\prime}\dot{x}\leq
M^{\prime}\\}$ is still a small set by the same arguments used in the proof of
Part II of Lemma B.1. Consequently,
$P_{X}^{2}(x,A)\geq\tilde{c}_{*}^{\prime}\tilde{\varphi}^{\prime}(A)$ for all
$x\in\tilde{S}_{2\,\epsilon}^{\prime}$, where $\tilde{c}_{*}^{\prime}$ is
possibly smaller than $\tilde{c}_{*}$ but strictly positive (and independent
of $\theta$),
$\tilde{\varphi}^{\prime}(A)=\mu_{Leb}(A\cap\tilde{D}^{\prime})$, and
$\tilde{D}^{\prime}$ is analogous to $\tilde{D}$ in part $(i)$. Clearly,
$\lambda^{\prime-1}=1-\tilde{\gamma}_{1}+{\tilde{\gamma}_{2}\over
2+M^{\prime}}<1$. The result now follows by using the same arguments as in
part $(i)$ with $\lambda$ and $\tilde{c}_{*}$ replaced by $\lambda^{\prime}$
and $\tilde{c}_{*}^{\prime}$. ∎

###### Proof of Lemma 3.

For all $n\geq 2$ we write

$\displaystyle P^{n}_{W}(w,dw_{n})$
$\displaystyle=\pi_{Y|X}(dy_{n}|x_{n})\mathbb{P}(dx_{n}|w)=\pi_{Y|X}(dy_{n}|x_{n})\int_{\mathcal{H}}\mathbb{P}(dx_{n}|w,h_{1})P_{H}(h,dh_{1})~{},$

where the last equality follows because the $H_{t}$ component of
$W_{\theta\,t}$ is a Markov chain of its own. Define
$\tilde{f}_{\theta}=f_{\theta\,1}=\sum_{k}\\{\alpha_{0k}+\alpha_{1k}y+\beta_{1}f\\}\mathbbm{1}_{\\{y\in\mathcal{Y}_{k}\\}}$,
and $\tilde{d}=d_{1}=1+|y|+|f|+\overline{\beta}_{1}d$. Note that by the i.i.d.
assumption on the innovations $Z_{1\,t}$ and $Z_{2\,t}$ we have that the
$X_{\theta\,t}$ component of $\\{W_{\theta\,t},t\geq 0\\}$ has a 2-step
transition mechanism which is entirely similar to the 1-step transition
mechanism of the companion Markov chain defined in (14) with initial value
given by $\tilde{x}(w,h_{1})$. We denote
$\tilde{P}_{X}^{n}((w,h_{1}),dx_{n})=\mathbb{P}(dx_{n}|w,h_{1})$. Note that
$\tilde{P}_{X}^{2}((w,h_{1}),dx_{2})=P_{X}^{1}(\tilde{x},dx_{2})$ where
$\tilde{x}=\tilde{x}(w,h_{1})=(h_{1},\tilde{f}_{\theta},\tilde{d})$. Next,
$\tilde{P}_{X}^{3}((w,h_{1}),dx_{3})=\int_{\mathcal{X}}P_{X}(\tilde{x},dx_{2})P_{X}(x_{2},dx_{3})=P_{X}^{2}(\tilde{x},dx_{3}).$
By induction,
$\tilde{P}_{X}^{n}((w,h_{1}),dx_{n})=P_{X}^{n-1}(\tilde{x},dx_{n})$, and the
result follows. ∎

###### Proof of Lemma 4.

$(i)$ First, $\\{X_{\theta\,t},t\geq 0\\}$ viewed as a separate Markov chain
is $V_{X}$-geometrically ergodic by Lemma 1. We begin by showing that

$\displaystyle\mathbb{E}_{Y|X}\left(V_{Y,X}(W_{\theta\,t})\right)\equiv\int_{\mathcal{Y}}V_{Y,X}(y,x)\pi_{Y|X}(dy|x)<C\cdot
V_{X}(x)~{}.$ (20)

For all $x\in\mathcal{X}$, by Assumption 1$(ii)$ we have that

$\displaystyle\mathbb{E}_{Y|X}\left(V_{Y,X}(W_{\theta\,t})\right)$
$\displaystyle=V_{X}(x)+\mathbb{E}_{Y|X}\left(|g_{y1}(h)+g_{y2}(h)\epsilon_{Y\,t}|^{2r_{m}}\right)$
$\displaystyle\leq
V_{X}(x)+2^{2r_{m}-1}|g_{y1}(h)|^{2r_{m}}+2^{2r_{m}-1}|g_{y2}(h)|^{2r_{m}}\mathbb{E}_{Y|X}(|\epsilon_{Y\,t}|^{2r_{m}})$
$\displaystyle\leq V_{X}(x)+C\|x\|_{1}^{2r_{m}}\leq C\cdot V_{X}(x)~{},$

where the constant $0<C<\infty$ may change from line to line. To satisfy the
definition of $V_{Y,X}$-geometric ergodicity, we must have that
$\mathbb{E}\left(V_{Y,X}(Y_{t},X_{\theta\,t})\right)<\infty$, where the
expectation is taken with respect to the invariant measure $\pi_{Y,X}$. By
(20) we have that

$\displaystyle\mathbb{E}\left(V_{Y,X}(W_{\theta\,t})\right)=\int_{\mathcal{X}}\pi_{X}(dx)\int_{\mathcal{Y}}V_{Y,X}(y,x)\pi_{Y|X}(dy|x)\leq\int_{\mathcal{X}}\pi_{X}(dx)C\cdot
V_{X}(x)<\infty,$

as expected. For any $w=(y,x)^{\prime}\in\mathcal{Y}\times\mathcal{X}$ and all
$n\geq 2$ we have that

$\displaystyle\sup_{v:|v|\leq
V_{Y,X}}\left\lvert\int_{\mathcal{Y}\times\mathcal{X}}\left[P_{Y,X}^{n}(w,dw_{n})-\pi_{Y,X}(dw_{n})\right]v(w_{n})\right\rvert$
$\displaystyle=\sup_{v:|v|\leq
V_{Y,X}}\left\lvert\int_{\mathcal{X}}\int_{\mathcal{H}}P_{X}^{n-1}(\tilde{x},dx_{n})P_{H}(h,dh_{1})-\pi_{X}(dx_{n})\int_{\mathcal{Y}}\pi_{Y|X}(dy_{n}|x_{n})v(y_{n},x_{n})\right\rvert$
$\displaystyle\quad\leq C\sup_{v^{\prime}:|v^{\prime}|\leq
V_{X}}\left\lvert\int_{\mathcal{H}}\left(\int_{\mathcal{X}}\left[P_{X}^{n-1}(\tilde{x},dx_{n})-\pi_{X}(dx_{n})\right]v^{\prime}(x_{n})\right)P_{H}(h,dh_{1})\right\rvert$
$\displaystyle\quad\leq C\int_{\mathcal{H}}\sup_{v^{\prime}:|v^{\prime}|\leq
V_{X}}\left\lvert\int_{\mathcal{X}}\left[P_{X}^{n-1}(\tilde{x},dx_{n})-\pi_{X}(dx_{n})\right]v^{\prime}(x_{n})\right\rvert
P_{H}(h,dh_{1})$ $\displaystyle\quad\leq
CR_{\theta}\rho_{\theta}^{n-1}\mathbb{E}\left(V_{X}(\tilde{x})|H_{0}=h\right)~{},$
(21)

where $R_{\theta}<\infty$, $\rho_{\theta}<1$. The equality follows by Lemma 3,
the first inequality is a consequence of (20) and the last inequality is
implied by the drift criterion that we have used in the proof of Lemma 1.
Furthermore, note that by Assumptions 1$(i)$, $(iii)$, $(iv)$ and (25) we have

$\displaystyle\mathbb{E}\left(V_{X}(\tilde{x})|H_{0}=h\right)$
$\displaystyle\leq
1+3^{2r_{m}-1}\left(\mathbb{E}\left(|H_{1}|^{2r_{m}}|H_{0}=h\right)+|\tilde{f}_{\theta}|^{2r_{m}}+|\tilde{d}|^{2r_{m}}\right)$
$\displaystyle<1+C\cdot\begin{cases}|\tilde{f}_{\theta}|^{2r_{m}}+|\tilde{d}|^{2r_{m}}+|h|^{2r_{m}}~{},&|h|>M_{\epsilon}\\\
1+|\tilde{f}_{\theta}|^{2r_{m}}+|\tilde{d}|^{2r_{m}}~{},&|h|\leq
M_{\epsilon}\end{cases}$ $\displaystyle<CV_{X}(\check{x})~{},$

where $1<C<\infty$ may change from line to line and the choice of $\epsilon$
is such that $\mathbb{E}(a_{h}+b_{h}|\epsilon_{H\,t}|+\epsilon)^{2r_{m}}<1$
(Assumption 1$(iv)$).
$(ii)$ Repeating the same arguments as in $(i)$ with $2r_{m}=1$ and with
$\sup_{v:|v|\leq 1}$ instead of $\sup_{v:|v|\leq V_{Y,X}}$, we can use Lemma 2
in the last inequality of (A) instead of the standard drift criterion to
obtain constants $\rho\in(0,1)$ and $R<\infty$ that do not depend on $\theta$.
The proof is completed by noting that we can redefine $R$ to absorb
$C\rho^{-1}$. ∎

###### Proof of Proposition 1.

$(i)$ We begin by noting that Lemma 4 implies that $\|Y_{t}\|_{L_{2r_{m}}}$,
$\|H_{t}\|_{L_{2r_{m}}}$, $\|f_{\theta\,t}\|_{L_{2r_{m}}}$ and
$\|d_{\theta\,t}\|_{L_{2r_{m}}}$ exist. To complete the proof we need to
establish that $\sup_{\theta\in\Theta}\|f_{\theta\,t}\|_{L_{2r_{m}}}$ and
$\sup_{\theta\in\Theta}\|d_{\theta\,t}\|_{L_{2r_{m}}}$ are finite. Since
$\alpha_{1\,k}>0$, we have that
$\|f_{\theta\,t}\|_{L_{2r_{m}}}\leq|\overline{\alpha}_{0}|+\overline{\alpha}_{1}\|Y_{t}\|_{L_{2r_{m}}}+\overline{\beta}_{1}\|f_{\theta\,t}\|_{L_{2r_{m}}}$
by the existence of the stationary distribution of $\\{W_{\theta\,t},t\geq
0\\}$ which follows from Lemma 4. The last inequality and Assumption 2 imply
that
$\sup_{\theta\in\Theta}\|f_{\theta\,t}\|_{L_{2r_{m}}}\leq(1-\overline{\beta}_{1})^{-1}\left(|\overline{\alpha}_{0}|+\overline{\alpha}_{1}\|Y_{t}\|_{L_{2r_{m}}}\right)<\infty~{}.$
Analogously, we have that $\|d_{\theta\,t}\|_{L_{2r_{m}}}\leq
1+\|Y_{t}\|_{L_{2r_{m}}}+\|f_{\theta\,t}\|_{L_{2r_{m}}}+\overline{\beta}_{1}\|d_{\theta\,t}\|_{L_{2r_{m}}}.$
Thus,
$\sup_{\theta\in\Theta}\|d_{\theta\,t}\|_{L_{2r_{m}}}\leq(1-\overline{\beta}_{1})^{-1}\left(1+\|Y_{t}\|_{L_{2r_{m}}}+\sup_{\theta\in\Theta}\|f_{\theta\,t}\|_{L_{2r_{m}}}\right)<\infty.$
$(ii)$ It is enough to show that $\\{W_{\theta\,t},t\geq 0\\}$ is
geometrically $\beta$-mixing, since $\alpha(l)\leq\beta(l)$, where
$\beta(l)=\sup_{t\in\mathbb{Z}}{1\over
2}\sup\sum_{i=1}^{I}\sum_{j=1}^{J}\left\lvert\mathbb{P}(A_{i}\cap
B_{i})-\mathbb{P}(A_{i})\mathbb{P}(B_{i})\right\rvert,$ and the supremum is
taken over all pairs of finite partitions $\\{A_{1},\dots,A_{I}\\}$ and
$\\{B_{1},\dots,B_{J}\\}$ of $\Omega$ such that
$A_{i}\in\sigma\\{W_{\theta\,s}:s\leq t\\}$, $i=1,\dots,I$, and
$B_{j}\in\sigma\\{W_{\theta\,s}:s\geq t+l\\}$, $j=1,\dots,J$. Let
$\delta_{w}(A)=\mathbbm{1}\\{w\in A\\}$ for any
$A\in\mathcal{B}(\mathcal{Y}\times\mathcal{X})$. By Proposition 4 in Liebscher
(2005), $\\{W_{\theta\,t},t\geq 0\\}$ is $\beta$-mixing with geometrically
decaying mixing numbers if $(a)$
$\int_{\mathcal{Y}\times\mathcal{X}}V_{X}(x_{0})\delta_{y,x}(dw_{0})\\}=V_{X}(x)<\infty$,
and $(b)$ $\\{W_{\theta\,t},t\geq 0\\}$ is $Q$-geometrically ergodic in the
sense of Liebscher (2005) with $Q(w)=V_{X}(x)$. Condition $(a)$ holds for all
$w\in\mathcal{Y} \times\mathcal{X}$. For condition $(b)$, we first need to
show that
$\int_{\mathcal{Y}\times\mathcal{X}}V_{X}(x_{n})\pi_{Y,X}(dw_{n})<\infty$.
This is easily obtained by noting that

$\displaystyle\int_{\mathcal{Y}\times\mathcal{X}}V_{X}(x_{n})\pi_{Y,X}(dw_{n})=\int_{\mathcal{X}}V_{X}(x_{n})\pi_{X}(dx_{n})\int_{\mathcal{Y}}\pi_{Y|X}(dy_{n}|x_{n})<\infty,$

where the last inequality follows from the $V_{X}$-geometric ergodicity of
$\\{X_{\theta\,t},t\geq 0\\}$. As for the remaining part of condition $(b)$,
notice that from Lemma 4$(ii)$ we have that
$\left\|P_{Y,X}^{l}(w,\cdot)-\pi_{Y,X}\right\|_{TV}\leq
R\tilde{V}_{X}(\check{x})\rho^{l}\wedge 1$, where

$\left\|P_{Y,X}^{l}(w,\cdot)-\pi_{Y,X}\right\|_{TV}=\sup_{v:|v|\leq
1}\left\lvert\int_{\mathcal{Y}\times\mathcal{X}}\left[P^{l}_{Y,X}\left(w,dw_{l}\right)-\pi_{Y,X}(dw_{l})\right]v(w_{l})\right\rvert~{},$

which completes the proof of condition $(b)$. It remains to be shown that the
rate of decay does not depend on $\theta$. For any probability measure $\tau$
on $\mathcal{Y}\times\mathcal{X}$, define
$\xi_{l}(\tau)=\int_{\mathcal{Y}\times\mathcal{X}}\left\|P_{Y,X}^{l}(w,\cdot)-\pi_{Y,X}\right\|_{TV}\cdot\tau(dw).$
By virtue of part $(ii)$ of Lemma 4 we compute that $\xi_{l}(\pi_{Y,X})\leq
R\check{V}\rho^{l}$, where

$\displaystyle\check{V}=\left(2+\|H_{t}\|_{L_{1}}+(1+\overline{\alpha}_{1})\|Y_{t}\|_{L_{1}}+(1+\overline{\beta}_{1})\sup_{\theta\in\Theta}\|f_{\theta\,t}\|_{L_{1}}+\overline{\beta}_{1}\sup_{\theta\in\Theta}\|d_{\theta\,t}\|_{L_{1}}\right)~{},$

and
$\xi_{l}(\delta_{y,x})=\left\|P_{Y,X}^{l}(w,\cdot)-\pi_{Y,X}\right\|_{TV}\leq
R\tilde{V}_{X}(\check{x})\rho^{l}\wedge 1.$ Now, by Proposition 3 in Liebscher
(2005) we have that for any $w\in\mathcal{Y}\times\mathcal{X}$, and $m=\lfloor
l/2\rfloor$, $\beta(l)\leq 3\xi_{m}(\delta_{y,x})+\xi_{m}(\pi_{Y,X})\leq
R\left(\check{V}+3\tilde{V}_{X}(\check{x})\right)\rho^{m}\wedge 1.$ It is not
difficult to verify that
$\alpha(l)\leq\beta(l)\leq\exp\left(-C_{\alpha}l^{r_{\alpha}}\right)\wedge 1$
for all $l\geq 1$. The choice of $C_{\alpha}$ and $r_{\alpha}$ depends on
$R,\check{V},\rho$ and $\tilde{V}_{X}(\check{x})$. Since none of those depend
on $\theta$ by Lemma 2, neither does the rate of decay of the uniform bound
for the $\alpha$-mixing coefficients. The claim follows by redefining $R$ and
noting that $\tilde{V}_{X}\geq 1$. ∎

###### Proof of Proposition 2.

Let $\\{(Y^{G}_{t},f^{G}_{\theta\,t})^{\prime},t\geq 0\\}$ be an independent
copy of the process $\\{(Y_{t},f_{\theta\,t})^{\prime},t\geq 0\\}$ and define
$\overline{R}(\hat{\theta})=\mathbb{E}\left[{1\over
M}\sum_{t=T+1}^{T+M}L(Y_{t}^{G},f^{G}_{\hat{\theta}\,t})\right]$. By the
properties of infimum and supremum and the definition of empirical risk
minimizer (i.e. $R_{T}(\theta)\geq R_{T}(\hat{\theta})$ for all
$\theta\in\Theta$), we have that

$\displaystyle R(\hat{\theta})-\inf_{\theta\in\Theta}R(\theta)$
$\displaystyle=R(\hat{\theta})-\overline{R}(\hat{\theta})+\overline{R}(\hat{\theta})-\inf_{\theta\in\Theta}\left[\overline{R}(\theta)+R(\theta)-\overline{R}(\theta)\right]$
$\displaystyle\leq\left[R(\hat{\theta})-\overline{R}(\hat{\theta})\right]+\left[\overline{R}(\hat{\theta})-\inf_{\theta\in\Theta}\overline{R}(\theta)\right]-\inf_{\theta\in\Theta}\left[R(\theta)-\overline{R}(\theta)\right]$
$\displaystyle\leq
2\sup_{\theta\in\Theta}|R(\theta)-\overline{R}(\theta)|+2\sup_{\theta\in\Theta}|R_{T}(\theta)-\overline{R}(\theta)|~{},$

where the last inequality follows from Lemma 8.2 in Devroye et al. (1996). ∎

###### Proof of Proposition 3.

First, by the triangular inequality we have

$\displaystyle\sup_{\theta\in\Theta}\left|R(\theta)-\overline{R}(\theta)\right|$
$\displaystyle\leq\sup_{\theta\in\Theta}{1\over
M}\sum_{t=T+1}^{T+M}\left|\mathbb{E}_{T}L\left(Y_{t},f_{\theta\,t}\right)-\mathbb{E}L\left(Y_{t},f_{\theta\,t}\right)\right|~{},$
(22)

where $\mathbb{E}_{T}(\cdot)=\mathbb{E}(\cdot|Y_{T},\ldots,Y_{1})$. For each
$t=T+1,\ldots,T+M$, it follows from Ibragimov’s inequality (Davidson, 1994,
Theorem 14.2) that

$\displaystyle\mathbb{E}\left|\mathbb{E}_{T}L(Y_{t},f_{\theta\,t})-\mathbb{E}L(Y_{t},f_{\theta\,t})\right|$
$\displaystyle\leq
6\alpha(l)^{1-1/r}\|L(Y_{t},f_{\theta\,t})\|_{L_{r}}~{},\quad l=t-T,$ (23)

for any $r\in\\{1,\ldots,r_{m}\\}$. By combining (23) with Markov’s inequality
applied to the non-negative random variable
$\left|\mathbb{E}_{T}L(Y_{t},f_{\theta\,t})-\mathbb{E}L(Y_{t},f_{\theta\,t})\right|$,
Proposition 1, averaging and taking the supremum on both sides, we have that

$\displaystyle\sup_{\theta\in\Theta}{1\over
M}\sum_{t=T+1}^{T+M}\left|\mathbb{E}_{T}L(Y_{t},f_{\theta\,t})-\mathbb{E}L(Y_{t},f_{\theta\,t})\right|
\leq{6\sup_{\theta\in\Theta}\|L(Y_{t},f_{\theta\,t})\|_{L_{r}}\over\delta
M}\sum_{l=1}^{M}e^{-\left(1-1/r\right)C_{\alpha}l^{r_{\alpha}}}$ (24)

with probability at least $1-\delta$. Note that Proposition 1 implies that
$\sup_{\theta\in\Theta}\|L(Y_{t},f_{\theta\,t})\|_{L_{r}}<\infty$. It follows
from (22), (23) and (24) that if we set $\delta=T^{-1/2}$

$\displaystyle\sup_{\theta\in\Theta}\left|R(\theta)-\overline{R}(\theta)\right|$
$\displaystyle\geq{1\over\sqrt{T}}{6\sup_{\theta\in\Theta}\|L(Y_{t},f_{\theta\,t})\|_{L_{r}}\over\gamma}\sum_{l=1}^{\infty}e^{-\left(1-1/r\right)C_{\alpha}l^{r_{\alpha}}}$

holds with probability at most $T^{-{1\over 2}}$. ∎

###### Proof of Proposition 4.

Define $g_{\theta\,t}=L(Y_{t},f_{\theta\,t})$. Let
$\Theta_{i}=\\{\theta\in\mathbb{R}^{p}:\|\theta-\theta_{i}\|_{2}\leq\delta\\}$
with $\theta_{i}\in\Theta$ for $i=1,\ldots,N_{\delta}$ denote a
$\delta$-covering of $\Theta$ for some $\delta\in(0,1]$. Then, we have that

$\displaystyle\mathbb{P}\left(\sup_{\theta\in\Theta}\left|R_{T}(\theta)-\overline{R}(\theta)\right|>{\varepsilon\over
2}\right)\leq\sum_{i=1}^{N_{\delta}}\mathbb{P}\left(\sup_{\theta\in\Theta_{i}}\left|{1\over
T}\sum_{t=1}^{T}g_{\theta\,t}-\mathbb{E}g_{\theta\,t}\right|>{\varepsilon\over
2}\right)$
$\displaystyle\quad\leq\sum_{i=1}^{N_{\delta}}\mathbb{P}\left(\left|{1\over
T}\sum_{t=1}^{T}g_{i\,t}-\mathbb{E}g_{i\,t}\right|>{\varepsilon\over
4}\right)$
$\displaystyle\quad+\sum_{i=1}^{N_{\delta}}\mathbb{P}\left(\sup_{\theta\in\Theta_{i}}\left|{1\over
T}\sum_{t=1}^{T}g_{\theta\,t}-\mathbb{E}g_{\theta\,t}-\left({1\over
T}\sum_{t=1}^{T}g_{i\,t}-\mathbb{E}g_{i\,t}\right)\right|>{\varepsilon\over
4}\right)~{},$

where $g_{i\,t}=g_{\theta_{i}\,t}$. Lemma LABEL:lem:gdiffbound from the Online
Appendix establishes that $(i)$ for each $\theta\in\Theta_{i}$ we have
$|g_{\theta\,t}-g_{i\,t}|\leq\bar{g}_{i\,t}=\delta
C_{\psi}(d_{\theta_{i}\,t}^{2}+2|Y_{t}-f_{\theta_{i}\,t}|d_{\theta_{i}\,t})$
and $(ii)$ there exists a positive constant $C_{d}$ (that does not depend on
$i$ and $t$) such that for all $\delta\in(0,1]$ we have that
$\mathbb{E}\bar{g}_{i\,t}\leq\delta C_{d}$. Set
$\delta=\varepsilon/(16C_{d})$. Then, for all $\varepsilon<16C_{d}$

$\displaystyle\mathbb{P}\left(\sup_{\theta\in\Theta_{i}}\left|{1\over
T}\sum_{t=1}^{T}(g_{\theta\,t}-g_{i\,t})-\mathbb{E}(g_{\theta\,t}-g_{i\,t})\right|>{\varepsilon\over
4}\right)\leq\mathbb{P}\left({1\over
T}\sum_{t=1}^{T}(\bar{g}_{i\,t}+\mathbb{E}\bar{g}_{i\,t})>{\varepsilon\over
4}\right)$ $\displaystyle\quad=\mathbb{P}\left({1\over
T}\sum_{t=1}^{T}(\bar{g}_{i\,t}-\mathbb{E}\bar{g}_{i\,t})>{\varepsilon\over
4}-2\mathbb{E}\bar{g}_{i\,t}\right)\leq\mathbb{P}\left({1\over
T}\sum_{t=1}^{T}(\bar{g}_{i\,t}-\mathbb{E}\bar{g}_{i\,t})>{\varepsilon\over
8}\right)$

holds, where the last inequality follows from the fact that
$\mathbb{E}\bar{g}_{i\,t}\leq\varepsilon/16$. The claim follows after defining
$U_{\theta_{i}\,t}=g_{i\,t}$ and
$V_{\theta_{i}\,t}=C_{\psi}\left(d_{\theta_{i}\,t}^{2}+2|Y_{t}-f_{\theta_{i}\,t}|d_{\theta_{i}\,t}\right)=16C_{d}\bar{g}_{i\,t}/\varepsilon$
and noting that
$N_{\delta}\leq\left(1+2C_{\Theta}/\delta\right)^{p}=\left(1+32C_{\Theta}C_{d}/\varepsilon\right)^{p}$.
∎

###### Proof of Proposition 5.

The analysis of the sequences $\\{\widetilde{U}_{\theta\,t},t\geq 0\\}$ and
$\\{\widetilde{V}_{\theta\,t},t\geq 0\\}$ is analogous. Here we focus on
$\\{\widetilde{U}_{\theta\,t},t\geq 0\\}$. To simplify notation we omit the
subscript $\theta$ in the notation of the sequence
$\\{\widetilde{U}_{\theta\,t},t\geq 0\\}$.

Let
$\sum_{t=1}^{T}\widetilde{U}_{t}=\sum_{t=1}^{T}U^{\prime}_{t}+\sum_{t=1}^{T}U^{\prime\prime}_{t}$
where $U^{\prime}_{t}=\widetilde{U}_{t}\mathbbm{1}_{\\{|\widetilde{U}_{t}|\leq
b_{T}\\}}-\mathbb{E}\left(\widetilde{U}_{t}\mathbbm{1}_{\\{|\widetilde{U}_{t}|\leq
b_{T}\\}}\right)$ and
$U^{\prime\prime}_{t}=\widetilde{U}_{t}\mathbbm{1}_{\\{|\widetilde{U}_{t}|>b_{T}\\}}-\mathbb{E}\left(\widetilde{U}_{t}\mathbbm{1}_{\\{|\widetilde{U}_{t}|>b_{T}\\}}\right)$.
We then have that

$\displaystyle\mathbb{P}\left(\left|{1\over
T}\sum_{t=1}^{T}\widetilde{U}_{t}\right|>{\varepsilon_{T}\over
4}\right)\leq\mathbb{P}\left(\left|\sum_{t=1}^{T}U^{\prime}_{t}\right|>{T\varepsilon_{T}\over
8}\right)+\mathbb{P}\left(\left|\sum_{t=1}^{T}U^{\prime\prime}_{t}\right|>{T\varepsilon_{T}\over
8}\right)~{}.$

Define $M_{T}=\lfloor T^{{1\over 2}-{p+1\over 2(r_{m}-1)}}\log^{-{1\over
2}}T\rfloor$ and $b_{T}=C_{b}T^{p+1\over 2(r_{m}-1)}(p\log T)^{-{p-1\over
2(r_{m}-1)}}$ where $C_{b}$ is a positive constant to be chosen in what
follows. The sequence $\\{U^{\prime}_{t}\\}_{t=1}^{T}$ has the same mixing
properties of $\\{\widetilde{U}_{t}\\}_{t=1}^{T}$ and
$\|U_{t}^{\prime}\|_{L_{\infty}}<2b_{T}$. Then for all $T$ sufficiently large
and $p<r_{m}-2$ the conditions of Theorem 2.1 in Liebscher (1996) are
satisfied since $M_{T}\in\\{1,\dots,T\\}$ and
$8M_{T}b_{T}\leq{T\varepsilon_{T}/2}$. Then, we have

$\displaystyle\mathbb{P}\left(\left|\sum_{t=1}^{T}U^{\prime}_{t}\right|>{T\varepsilon_{T}\over
2}\right)$ $\displaystyle\leq 4\exp\left(-{T^{2}\varepsilon_{T}^{2}\over
4096{T\over M_{T}}\mathbb{E}(\sum_{t=1}^{M_{T}}U^{\prime}_{t})^{2}+{64\over
3}M_{T}b_{T}T\varepsilon_{T}}\right)$ $\displaystyle\quad+4{T\over
M_{T}}\exp\left(-C_{\alpha}M_{T}^{r_{\alpha}}\right)~{}.$

If we define $\gamma(l)=|\mbox{Cov}(U^{\prime}_{t},U^{\prime}_{t+l})|$ for
$l=0,\ldots,T-1$, then we have that
$\mathbb{E}(\sum_{t=1}^{M_{T}}U^{\prime}_{t})^{2}\leq
M_{T}(\gamma(0)+2\sum_{l=1}^{\infty}\gamma(l))$. Let
$C_{m}=\sup_{\theta\in\Theta}\|\widetilde{U}_{t}\|_{L_{r_{m}}}$. Noting that
$L(Y_{t},f_{\theta\,t})\geq 0$, Davydov’s inequality implies

$\displaystyle\gamma(l)$ $\displaystyle\leq$ $\displaystyle 2{r_{m}\over
r_{m}-2}2^{1-{2\over r_{m}}}\alpha(l)^{1-{2\over
r_{m}}}\|\widetilde{U}_{t}^{\prime}\|_{L_{r_{m}}}\|\widetilde{U}_{t+l}^{\prime}\|_{L_{r_{m}}}\leq
16C_{m}^{2}{r_{m}\over r_{m}-2}\alpha(l)^{1-{2\over r_{m}}}~{},$

for $l=0,\ldots,T-1$, and we use the fact that for any $r$ we have
$\|\widetilde{U}_{t}^{\prime}\|_{L_{r}}\leq 2\|\widetilde{U}_{t}\|_{L_{r}}$.
Thus, we have $\mathbb{E}(\sum_{t=1}^{M_{T}}U^{\prime}_{t})^{2}\leq
M_{T}16C_{m}^{2}{r_{m}\over
r_{m}-2}(1+2\sum_{l=1}^{\infty}\exp\left(-C_{\alpha}l^{r_{\alpha}}\right)^{1-{2\over
r_{m}}})=M_{T}\sigma^{2}$. For all $T$ sufficiently large, we have that
$\log\left(1+{32C_{\Theta}C_{d}\over\varepsilon_{T}}\right)^{p}\leq{1\over
2}p\log T$. Then, for all $T$ sufficiently large, it holds that

$\displaystyle\left(1+{32C_{\Theta}C_{d}\over\varepsilon_{T}}\right)^{p}\mathbb{P}\left(\left|\sum_{t=1}^{T}U^{\prime}_{t}\right|>{T\varepsilon_{T}\over
2}\right)$ $\displaystyle\quad\leq
4\left(1+{32C_{\Theta}C_{d}\over\varepsilon_{T}}\right)^{p}\left[\exp\left(-{2116\sigma^{2}p\log
T\over 4096\sigma^{2}+{2944\over 3}\sigma C_{b}(p\log T)^{-{p-1\over
2(r_{m}-1)}}}\right)+{T\over M_{T}}e^{-C_{\alpha}M_{T}^{r_{\alpha}}}\right]$
$\displaystyle\quad=o(\log^{-1}T)~{}.$

Let $C_{m}=\sup_{\theta\in\Theta}\|\widetilde{U}_{t}\|_{L_{r_{m}}}$. We note
that for all $T$ sufficiently large,

$\displaystyle\left(1+{32C_{\Theta}C_{d}\over\varepsilon_{T}}\right)^{p}\mathbb{P}\left(\left|\sum_{t=1}^{T}U^{\prime\prime}_{t}\right|>{{T\varepsilon_{T}}\over
2}\right)\stackrel{{\scriptstyle(a)}}{{\leq}}\left(1+{32C_{\Theta}C_{d}\over\varepsilon_{T}}\right)^{p}{8\over
T\varepsilon_{T}}\mathbb{E}\left|\sum_{t=1}^{T}U^{\prime\prime}_{t}\right|$
$\displaystyle\quad\leq\left(1+{32C_{\Theta}C_{d}\over\varepsilon_{T}}\right)^{p}{16\over\varepsilon_{T}}\mathbb{E}\left|\widetilde{U}_{t}\mathbbm{1}_{\\{|\widetilde{U}_{t}|>b_{T}\\}}\right|\stackrel{{\scriptstyle(b)}}{{\leq}}\left(1+{32C_{\Theta}C_{d}\over\varepsilon_{T}}\right)^{p}{16\over\varepsilon_{T}}{C_{m}^{r_{m}}\over
b_{T}^{r_{m}-1}}$
$\displaystyle\quad\leq\left(1+32C_{\Theta}C_{d}\right)^{p}{16\over\varepsilon_{T}^{p+1}}{C_{m}^{r_{m}}\over
b_{T}^{r_{m}-1}}\stackrel{{\scriptstyle(c)}}{{\leq}}\log^{-1}T~{},$

where $(a)$ follows from Markov’s inequality $(b)$ from the inequality
$\mathbb{E}(|X\mathbbm{1}_{\\{|X|>b\\}}|)\leq\mathbb{E}(|X|^{r})/b^{r-1}$ for
any random variable $X$ wih finite $r$-th moment and positive constant $b$,
and $(c)$ from a sufficiently large choice of the constant $C_{b}$. The
sequence $\tilde{V}_{\theta\,t}$ can be analysed using the exact same strategy
(using the same choice of $M_{T}$ and $b_{T}$ used for $\tilde{U}_{T}$). ∎

##  Appendix B Irreducibility, Aperiodicity, Drift Criterion

Before we proceed, we establish upper bounds on $|H_{t}|$, $|f_{\theta\,t}|$
and $|d_{\theta\,t}|$. By Assumption 1$(i)$ we have that for any $\epsilon>0$
there exists some $1<M_{\epsilon}<\infty$ such that
$|H_{t}|\leq(a_{h}+b_{h}|Z_{1\,t}|+\epsilon)|H_{t-1}|$ for all
$|H_{t-1}|>M_{\epsilon}$. The same assumption also implies that when
$|H_{t-1}|\leq M_{\epsilon}$ we have
$|H_{t}|\leq|g_{h1}(H_{t-1})|+|g_{h2}(H_{t-1})|~{}|Z_{1\,t}|\leq\overline{g}_{h}^{\epsilon}(1+|Z_{1\,t}|),$
where $\overline{g}_{h}^{\epsilon}=\sup|g_{h1}(h)|\vee\sup|g_{h2}(h)|<\infty$
and the supremums are taken with respect to
$h\in[-M_{\epsilon},M_{\epsilon}]$. Hence we have that

$\displaystyle|H_{t}|\leq(a_{h}+b_{h}|Z_{1\,t}|+\epsilon)|H_{t-1}|\mathbbm{1}_{\\{|H_{t-1}|>M_{\epsilon}\\}}+\overline{g}_{h}^{\epsilon}(1+|Z_{1\,t}|)\mathbbm{1}_{\\{|H_{t-1}|\leq
M_{\epsilon}\\}}~{}.$ (25)

By 2 we have
$|f_{\theta\,t}|\leq\overline{\alpha}_{0}+\overline{\alpha}_{1}|g_{y1}(H_{t-1})|+\overline{\alpha}_{1}|g_{y2}(H_{t-1})|~{}|Z_{2\,t}|+\overline{\beta}_{1}|f_{\theta\,t-1}|$.
Furthermore, it follows from Assumption 1$(i)$ and $(ii)$ that

$\displaystyle|f_{\theta\,t}|$
$\displaystyle\leq\begin{cases}\overline{\alpha}_{1}C_{y}(1+\epsilon+|Z_{2\,t}|)~{}|H_{t-1}|+\overline{\beta}_{1}|f_{\theta\,t-1}|&|H_{t-1}|>M_{\epsilon}\\\
\overline{\alpha}_{0}+\overline{\alpha}_{1}\overline{g}_{y}^{\epsilon}(1+|Z_{2\,t}|)+\overline{\beta}_{1}|f_{\theta\,t-1}|&|H_{t-1}|\leq
M_{\epsilon}\end{cases}~{},$ (26)

where $C_{y}=C_{y1}\vee C_{y2}$,
$\overline{g}_{y}^{\epsilon}=\sup|g_{y1}(h)|\vee\sup|g_{y2}(h)|<\infty$ and
the supremums are taken with respect to $h\in[-M_{\epsilon},M_{\epsilon}]$. We
note that $M_{\epsilon}$ can be redefined if necessary in order to remove the
constant $\overline{\alpha}_{0}$ from the bound when $|H_{t-1}|>M_{\epsilon}$.
Lastly, using analogous arguments we have

$\displaystyle|d_{\theta\,t}|$
$\displaystyle\leq\begin{cases}C_{y}(1+\epsilon+|Z_{2\,t}|)~{}|H_{t-1}|+|f_{\theta\,t-1}|+\overline{\beta}_{1}|d_{\theta\,t-1}|~{},&|H_{t-1}|>M_{\epsilon}\\\
1+\overline{g}_{y}^{\epsilon}(1+|Z_{2\,t}|)+|f_{\theta\,t-1}|+\overline{\beta}_{1}|d_{\theta\,t-1}|~{},&|H_{t-1}|\leq
M_{\epsilon}\end{cases}$ (27)

where again $M_{\epsilon}$ may be redefined if necessary in order to remove
the constant $1$ from the bound when $|H_{t-1}|>M_{\epsilon}$.
Second, we introduce a partition of the state space $\mathcal{X}$ that plays a
key role in the subsequent proofs. Let
$\kappa=(\kappa_{h},\kappa_{f},\kappa_{d})^{\prime}\in(0,1)^{3}$ where the
specific choice of this vector will be determined in what follows. We define
the sets

$\displaystyle
S_{2\,\epsilon}=\\{(h,f,d)\in\mathcal{X}:\kappa_{h}|h|+\kappa_{f}|f|+\kappa_{d}|d|\leq
M\\}\quad\text{and}\quad S_{1\,\epsilon}=\mathcal{X}\setminus
S_{2\,\epsilon},$ (28)

where $M$ is a positive constant (note that in general $M\neq M_{\epsilon}$).
Third, let $\rho_{z\,\epsilon}=a_{h}+b_{h}|Z_{1\,t}|+\epsilon$,
$C_{y,z}^{\epsilon}=C_{y}(1+\epsilon+|Z_{2\,t}|)$ and define the matrix

$\mathbf{C}_{\epsilon}(Z_{t})=\begin{bmatrix}\rho_{z\,\epsilon}&0&0\\\
\overline{\alpha}_{1}\tilde{C}_{y,z}^{\epsilon}&\overline{\beta}_{1}+\epsilon&0\\\
\tilde{C}_{y,z}^{\epsilon}&1&\overline{\beta}_{1}+\epsilon\end{bmatrix}~{}.$

Assumption 1$(iii)$ implies that
$\mathbb{E}\left(\mathbf{C}_{\epsilon}(Z_{t})^{\otimes 2r_{m}}\right)$ exists.
Finally, we set $\epsilon>0$ in a way such that
$\mathbb{E}(a_{h}+b_{h}|Z_{1\,t}|+\epsilon)^{2r_{m}}<1$ and
$\overline{\beta}_{1}+\epsilon<1$. Assumptions 1$(iv)$ and 2 imply that an
$\epsilon$ that satisfies these constraints exists. For this particular choice
of $\epsilon$, we have that the spectral radius of
$\mathbb{E}\left(\mathbf{C}_{\epsilon}(Z_{t})^{\otimes 2r_{m}}\right)$ is
strictly less than unity. Such a choice of $\epsilon$ will be assumed
throughtout.

######  Lemma B.1 (Irreducibility).

Consider the setting of Proposition 1. There exists an open rectangular region
$D \subset\mathcal{X}$ that does not depend on $\theta$ or $x$ such that the
Markov chain $\\{X_{\theta\,t},t\geq 0\\}$ is $\varphi$-irreducible with
$\varphi(A)=\mu_{Leb}(A\cap D)$ for any $A\in\mathcal{B}(\mathcal{X})$.

###### Proof.

We follow the strategy of Lanne and Saikkonen (2005) and Meitz and Saikkonen
(2008). It suffices to show the following three intermediate results.

  1. I.

For any $x\in S_{1\,\epsilon}$ there exists an $n\in\mathbb{Z}_{+}$ such that
$P_{X}^{n}(x,S_{2\,\epsilon})>0$.

  2. II.

For any $A\in\mathcal{B}(\mathcal{X})$ it holds that $\inf_{x\in
S_{2\,\epsilon}}P_{X}^{2}(x,A\cap D)\geq c_{*}\mu_{Leb}(A\cap D)$, where $D$
is an open rectangular region to be specified in what follows and $c_{*}$ is a
positive scalar that does not depend on $\theta$ or $x$.

  3. III.

For any $x\in S_{1\,\epsilon}$ there exists an $n\in\mathbb{Z}_{+}$ such that
for any $A\in\mathcal{B}(\mathcal{X})$ it holds that $P_{X}^{n+2}(x,A\cap
D)>0$ whenever $\mu_{Leb}(A\cap D)>0$.

I. Define the event
$\Omega_{n}=\left\\{\omega\in\Omega:|Z_{1\,t}|\leq\mathbb{E}|Z_{1\,t}|\text{
and }|Z_{2\,t}|\leq\mathbb{E}|Z_{2\,t}|~{},t=1,\dots,n\right\\}$ for an
arbitrary $n$ and note that $\mathbb{P}(\Omega_{n})>0$. Define the auxiliary
vector
$\dot{X}_{\theta\,t}=(|H_{t}|,|f_{\theta\,t}|,|d_{\theta\,t}|)^{\prime}$. To
establish part I we show that for any $\kappa\in(0,1)^{3}$ and for each
$t=1,\ldots,n$ we have that when $X_{\theta\,t-1}\in S_{1\,\epsilon}$ the
inequality

$\displaystyle\left(\kappa^{\prime}\dot{X}_{\theta\,t}\right)^{2r_{m}}\leq\left(\kappa^{\otimes
2r_{m}}\right)^{\prime}\mathbb{E}\left(\mathbf{C}_{\epsilon}(Z_{t})^{\otimes
2r_{m}}\right)\dot{X}_{\theta\,t-1}^{\otimes 2r_{m}}~{}$ (29)

holds given $\Omega_{n}$. We distinguish the cases $(i)$
$|H_{t-1}|>M_{\epsilon}$ and $(ii)$ $|H_{t-1}|\leq M_{\epsilon}$.
$(i)$ From (25), (26) and (27) we have that

$\displaystyle\left(\kappa^{\prime}\dot{X}_{\theta\,t}\right)^{2r_{m}}$
$\displaystyle\leq\left(\kappa^{\prime}\mathbf{C}_{\epsilon}(Z_{t})\dot{X}_{\theta\,t-1}\right)^{2r_{m}}=\left(\kappa^{\otimes
2r_{m}}\right)^{\prime}\mathbf{C}_{\epsilon}(Z_{t})^{\otimes
2r_{m}}\dot{X}_{\theta\,t-1}^{\otimes 2r_{m}},$ (30)

where the last equality follows from properties of Kronecker products. By
adding and subtracting $\mathbb{E}\left(\mathbf{C}_{\epsilon}(Z_{t})^{\otimes
2r_{m}}\right)$ in (30) we obtain

$\displaystyle\left(\kappa^{\prime}\dot{X}_{\theta\,t}\right)^{2r_{m}}$
$\displaystyle\leq\left(\kappa^{\otimes
2r_{m}}\right)^{\prime}\mathbb{E}\left(\mathbf{C}_{\epsilon}(Z_{t})^{\otimes
2r_{m}}\right)\dot{X}_{\theta\,t-1}^{\otimes 2r_{m}}$
$\displaystyle+\left(\kappa^{\otimes
2r_{m}}\right)^{\prime}\left\\{\mathbf{C}_{\epsilon}(Z_{t})^{\otimes
2r_{m}}-\mathbb{E}\left(\mathbf{C}_{\epsilon}(Z_{t})^{\otimes
2r_{m}}\right)\right\\}\dot{X}_{\theta\,t-1}^{\otimes 2r_{m}}~{}.$ (31)

The random elements of the matrix $\mathbf{C}_{\epsilon}(Z_{t})^{\otimes
2r_{m}}$ are of the form
$C(a_{h}+b_{h}|Z_{1\,t}|+\epsilon)^{j_{1}}(1+\epsilon+|Z_{2\,t}|)^{j_{2}}$ for
$j_{1},j_{2}\geq 0$ such that $j_{1}+j_{2}\leq 2r_{m}$, where $C$ denotes some
positive constant (that depends on $\epsilon$). Conditionally on $\Omega_{n}$,
it follows from the independence between $Z_{1\,t}$ and $Z_{2\,t}$ as well as
Jensen’s inequality that the random elements are bounded from above by their
expectations. This establishes that the bound in (29) holds in case $(i)$.
$(ii)$ From (25), (26) and (27) we can write
$\left(\kappa^{\prime}\dot{X}_{\theta\,t}\right)^{2r_{m}}\leq\left(\overline{C}_{z\,\epsilon}+\kappa_{\mathbf{f}}^{\prime}\overline{B}\mathbf{f}_{\theta\,t-1}\right)^{2r_{m}},$
where
$\overline{C}_{z\,\epsilon}={\kappa_{h}\overline{H}_{z\,\epsilon}+\kappa_{f}(\overline{\alpha}_{0}+\overline{\alpha}_{1}\overline{Y}_{z\,\epsilon})+\kappa_{d}(1+\overline{Y}_{z\,\epsilon})}$,
$\overline{H}_{z\,\epsilon}=\overline{g}_{h}^{\epsilon}(1+|Z_{1\,t}|)$,
$\overline{Y}_{z\,\epsilon}=\overline{g}_{y}^{\epsilon}(1+|Z_{2\,t}|)$,
$\kappa_{\mathbf{f}}=(\kappa_{f},\kappa_{d})^{\prime}$,
$\mathbf{f}_{\theta\,t-1}=\left(|f_{\theta\,t-1}|,|d_{\theta\,t-1}|\right)^{\prime}$
and $\overline{B}$ is a $2\times 2$ lower triangular matrix with
$\overline{B}_{11}=\overline{B}_{22}=\overline{\beta}_{1}$ and
$\overline{B}_{21}=1$. On the event $\Omega_{n}$, it is straightforward to
verify that
$\overline{C}_{z\,\epsilon}\leq\left\|\overline{C}_{z\,\epsilon}\right\|_{L_{1}}$.
Combining conditions $X_{\theta\,t-1}\in S_{1\,\epsilon}$ and $|H_{t-1}|\leq
M_{\epsilon}$ we have
$\kappa_{\mathbf{f}}^{\prime}\mathbf{f}_{\theta\,t-1}>M-\kappa_{h}|H_{t-1}|\geq
M-\kappa_{h}M_{\epsilon}$. Thus, we can choose $M$ large enough such that
$\epsilon\left(M-\kappa_{h}M_{\epsilon}\right)>\left\|\overline{C}_{z\,\epsilon}\right\|_{L_{2r_{m}}}\geq\left\|\overline{C}_{z\,\epsilon}\right\|_{L_{1}}$.
Note that this choice of $M$ is independent of $t$. Such a choice of $M$ is
kept fixed throughtout our derivations.131313We remark that Assumption 1
implies that $\left\|\overline{C}_{z\,\epsilon}\right\|_{L_{2r_{m}}}$ exists.
Then, conditionally on $\Omega_{n}$ and whenever $X_{\theta\,t-1}\in
S_{1\,\epsilon}$, we have

$\displaystyle\left(\overline{C}_{z\,\epsilon}+\kappa_{\mathbf{f}}^{\prime}\overline{B}\mathbf{f}_{\theta\,t-1}\right)^{2r_{m}}\leq\left(\left\|\overline{C}_{z\,\epsilon}\right\|_{L_{2r_{m}}}+\kappa_{\mathbf{f}}^{\prime}\overline{B}\mathbf{f}_{\theta\,t-1}\right)^{2r_{m}}\leq\left(\kappa_{\mathbf{f}}^{\prime}\overline{B}_{\epsilon}\mathbf{f}_{\theta\,t-1}\right)^{2r_{m}}~{},$

where $\overline{B}_{\epsilon}=\overline{B}+\epsilon I$. We note that
$\kappa_{\mathbf{f}}^{\prime}\overline{B}_{\epsilon}\mathbf{f}_{\theta\,t-1}=\kappa^{\prime}\mathbf{C}_{\epsilon}(Z_{t})\ddot{X}_{\theta\,t-1}$,
where
$\ddot{X}_{\theta\,t-1}=\left(0,\mathbf{f}_{\theta\,t-1}^{\prime}\right)^{\prime}$,
and we have that

$\displaystyle\left(\kappa^{\prime}\mathbf{C}_{\epsilon}(Z_{t})\ddot{X}_{\theta\,t-1}\right)^{2r_{m}}\leq(\kappa^{\otimes
2r_{m}})^{\prime}\mathbb{E}\left(\mathbf{C}_{\epsilon}(Z_{t})^{\otimes
2r_{m}}\right)\dot{X}_{\theta\,t-1}^{\otimes 2r_{m}}~{},$ (32)

where we use the definition of $\ddot{X}_{\theta\,t-1}$. This establishes that
the bound in (29) holds in case $(ii)$. By Lemma A.2. of Ling and McAleer
(2003) we can choose $\kappa\in(0,1)^{3}$ such that the vector
$v=\left(I-\mathbb{E}\left(\mathbf{C}_{\epsilon}(Z_{t})^{\otimes
2r_{m}}\right)\right)^{\prime}\kappa^{\otimes 2r_{m}}$ has positive
components.141414Recall that the matrix
$\mathbb{E}\left(\mathbf{C}_{\epsilon}(Z_{t})^{\otimes 2r_{m}}\right)$ has a
spectral radius that is strictly less than unity. As noted by Lanne and
Saikkonen (2005), the given proof makes clear that it means no loss of
generality to assume that the components of $\kappa$ are bounded by unity. In
particular, we remark that the vector $v$ does not depend on $\theta$. We use
$\underline{v}$ to denote the minimum of the components of $v$. Thus from (29)
it follows that

$\displaystyle\left(\kappa^{\prime}\dot{X}_{\theta\,t}\right)^{2r_{m}}$
$\displaystyle\leq\left(\kappa^{\prime}\dot{X}_{\theta\,t-1}\right)^{2r_{m}}-v^{\prime}\dot{X}_{t-1}^{\otimes
2r_{m}}=\left(\kappa^{\prime}\dot{X}_{\theta\,t-1}\right)^{2r_{m}}\left(1-{v^{\prime}\dot{X}_{\theta\,t-1}^{\otimes
2r_{m}}\over\left(\kappa^{\otimes
2r_{m}}\right)^{\prime}\dot{X}_{\theta\,t-1}^{\otimes 2r_{m}}}\right)$
$\displaystyle\leq(1-\underline{v})\left(\kappa^{\prime}\dot{X}_{\theta\,t-1}\right)^{2r_{m}}~{},$
(33)

where $\underline{v}\in(0,1)$. By repeated application of (B) starting from
$X_{\theta\,0}=x\in S_{1\,\epsilon}$ we have
$\left(\kappa^{\prime}\dot{X}_{\theta\,n}\right)^{2r_{m}}\leq(1-\underline{v})^{n}\left(\kappa^{\prime}\dot{x}\right)^{2r_{m}}.$
Since $\underline{v}\in(0,1)$ we have that for any $x\in S_{1\epsilon}$ there
exists a sufficiently large $n$ such that the right hand side of the
inequality is smaller than $M^{2r_{m}}$. Thus, we have that $X_{\theta\,n}\in
S_{2\,\epsilon}$ with positive probability.
II. First we write $P_{X}^{2}(x,A\cap
D)=\mathbb{E}(\;\mathbb{E}(\mathbbm{1}_{\\{X_{\theta\,2}\in A\cap
D\\}}|H_{1},X_{\theta\,0})\;|\;X_{\theta\,0}=x\;),$ for any $x\in
S_{2\,\epsilon}$, $A\in\mathcal{B}(\mathcal{X})$, $D\subset\mathcal{X}$ such
that $D$ is an open rectangular region (to be specified in what follows). Let
$\underline{h}_{1}\geq\sup_{|h|<M/\kappa_{h}}g_{h1}(h)$. The result is
obtained by showing the following intermediate results. $(i)$
$\inf_{h_{1}\in[\underline{h}_{1},\underline{h}_{1}+1]}\mathbb{E}(\mathbbm{1}_{\\{X_{\theta\,2}\in
A\cap D\\}}|H_{1}=h_{1},X_{\theta\,0}=x)\geq c^{\prime}\mu_{Leb}(A\cap D),$
where $c^{\prime}$ is a positive scalar that does not depend on $\theta$ or
$x$. $(ii)$ $P_{X}^{2}(x,A\cap D)\geq
c^{\prime\prime}\inf_{h_{1}\in[\underline{h}_{1},\underline{h}_{1}+1]}\mathbb{E}(\mathbbm{1}_{\\{X_{\theta\,2}\in
A\cap D\\}}|H_{1}=h_{1},X_{\theta\,0}=x)$, where $c^{\prime\prime}$ is a
positive scalar that does not depend on $\theta$ or $x$.
$(i)$ Set
$\underline{Z}_{2\,1}=\sup_{|h|<M/\kappa_{h}}{(R-g_{y1}(h))/g_{y2}(h)}$ and
$\underline{Z}_{2\,2}=\sup_{h_{1}\in[\underline{h}_{1},\underline{h}_{1}+1]}{(R-g_{y1}(h_{1}))/g_{y2}(h_{1})}$,
where
$R>{\overline{\beta}_{1}M/\kappa_{f}-\underline{\alpha}_{0}\over\underline{\alpha}_{1}}\vee
r_{K}\vee\sup_{|h|<M/\kappa_{h}}g_{y1}(h)\vee\sup_{h_{1}\in[\underline{h}_{1},\underline{h}_{1}+1]}g_{y1}(h_{1}),$
and note that $\underline{Z}_{2\,1}$ and $\underline{Z}_{2\,2}$ do not depend
on $x$, $h_{1}$ or $\theta$. Then it holds that

$\displaystyle\inf_{h_{1}\in[\underline{h}_{1},\underline{h}_{1}+1]}\mathbb{E}(\mathbbm{1}_{\\{X_{\theta\,2}\in
A\cap D\\}}|H_{1}=h_{1},X_{\theta\,0}=x)$
$\displaystyle\quad\geq\inf_{h_{1}\in[\underline{h}_{1},\underline{h}_{1}+1]}\int_{0}^{\infty}\int_{\underline{Z}_{2\,1}}^{\infty}\int_{\underline{Z}_{2\,2}}^{\infty}\mathbbm{1}_{\\{X_{\theta\,2}\in
A\cap
D\\}}\phi_{H}(Z_{1\,2})\phi_{Y}(Z_{2\,1})\phi_{Y}(Z_{2\,2})dZ_{1\,2}dZ_{2\,1}dZ_{2\,2}~{}.$

Over the integration range of the right hand side a number of properties hold.
First, we have that $g_{y1}(h)+g_{y2}(h)Z_{2\,1}>R$ and
$g_{y1}(h_{1})+g_{y2}(h_{1})Z_{2\,2}>R$, and $f_{\theta\,1}>0$. Furthermore,
we have that the map between $X_{\theta\,2}$ and
$(Z_{1\,2},Z_{2\,2},Z_{2\,1})^{\prime}$ is linear and is given by

$X_{\theta\,2}=c+G\begin{bmatrix}Z_{1\,2}&Z_{2\,2}&Z_{2\,1}\end{bmatrix}^{\prime}~{},$
(34)

where the expression for the vector $c=(c_{h},c_{f},c_{d})^{\prime}$ and the 3
$\times$ 3 block-diagonal matrix $G$ are given in (LABEL:eqn:c_vector) and
(LABEL:eqn:G_matrix) from the Online Appendix. Furthermore, $G$ is invertible
uniformly over $S_{2\,\epsilon}$, $\Theta$ and
$[\underline{h}_{1},\underline{h}_{1}+1]$. In fact, we have that $\det
G\in[G_{l},G_{u}]$, and it is clear that $G_{l}>0$ by asssumptions 1 and
2.151515See (LABEL:eqn:G_l) and (LABEL:eqn:G_u) from the Online Appendix for
the expression for $G_{l}$ and $G_{u}$.  Define
$Z_{1\,2}(X_{\theta\,2})={H_{2}-c_{h}\over g_{h2}(h_{1})},$ and

$\displaystyle Z_{2\,2}(X_{\theta\,2})$ $\displaystyle=$
$\displaystyle{g_{h2}(h_{1})g_{y2}(h)[\alpha_{1\,K}+\overline{\beta}_{1})(f_{\theta\,2}-c_{f})-\alpha_{1\,K}\beta_{1\,K}(d_{\theta\,2}-c_{d})]\over\det
G}$ $\displaystyle Z_{2\,1}(X_{\theta\,2})$ $\displaystyle=$
$\displaystyle{g_{h2}(h_{1})g_{y2}(h_{1})[\alpha_{1\,K}(d_{\theta\,2}-c_{d})-(f_{\theta\,2}-c_{f})]\over\det
G}~{}.$

We observe that the constraints $Z_{2\,1}(X_{\theta\,2})>\underline{Z}_{1\,2}$
and $Z_{2\,2}(X_{\theta\,2})>\underline{Z}_{2\,2}$ impose upper and lower
bounds on $d_{\theta\,2}$ which are linear functions of $f_{\theta\,2}$ with
positive slopes. In fact, from Assumption 2 we have that the minimum
discrepancy between slopes is given by
$\inf_{\theta\in\Theta}{\alpha_{1\,K}+\overline{\beta}_{1}\over\alpha_{1\,K}\beta_{1\,K}}-{1\over\alpha_{1\,K}}=\inf_{\theta\in\Theta}{\alpha_{1\,K}+\overline{\beta}_{1}-\beta_{1\,K}\over\alpha_{1\,K}\beta_{1\,K}}={1\over\overline{\beta}_{1}}>0.$
It follows that the intersection of images of the map defined in (34) with
respect to $\theta\in\Theta$, $x\in S_{2\,\epsilon}$ and
$h_{1}\in[\underline{h}_{1},\underline{h}_{1}+1]$ contains the following
set161616See (LABEL:eqn:H2_lb), (LABEL:eqn:f2_lb),(LABEL:eqn:d2_lb) and
(LABEL:eqn:d2_ub) from the Online Appendix for the expressions for
$\underline{H}_{2}$, $\underline{f}_{2}$, $\underline{d}_{2}(f_{\theta\,2})$
and $\overline{d}_{2}(f_{\theta\,2})$.

$\\{X_{\theta\,2}\in\mathcal{X}:H_{2}>\underline{H_{2}},f_{\theta\,2}>\underline{f}_{2},\underline{d}_{2}(f_{\theta\,2})<d_{\theta\,2}<\overline{d}_{2}(f_{\theta\,2})\\}~{}.$
(35)

We remark that Assumption 2 implies that such a set is non-empty and it
contains sets of positive Lebesgue measure. Thus, we can pick $D$ as an open
rectangular region in the intersection of (35) and $S_{1\,\epsilon}$. Clearly,
$D$ does not depend on $\theta$, $x$ or $h_{1}$. Next, by the change of
variable theorem we obtain that

$\displaystyle\inf_{h_{1}\in[\underline{h}_{1},\underline{h}_{1}+1]}\int_{0}^{\infty}\int_{\underline{Z}_{2\,1}}^{\infty}\int_{\underline{Z}_{2\,2}}^{\infty}\mathbbm{1}_{\\{X_{\theta\,2}\in
A\cap
D\\}}\phi_{H}(Z_{1\,2})\phi_{Y}(Z_{2\,1})\phi_{Y}(Z_{2\,2})dZ_{1\,2}dZ_{2\,1}dZ_{2\,2}$
$\displaystyle\quad\geq\inf_{\begin{subarray}{c}x\in S_{2\,\epsilon}\\\
h_{1}\in[\underline{h}_{1},\underline{h}_{1}+1]\\\ \theta\in\Theta\\\
X_{\theta\,2}\in A\cap D\end{subarray}}\det
G^{-1}\phi_{H}(Z_{1\,2}(X_{\theta\,2}))\phi_{Y}(Z_{2\,1}(X_{\theta\,2}))\phi_{Y}(Z_{2\,2}(X_{\theta\,2}))\int_{A\cap
D}dX_{\theta\,2}~{}.$

The boundedness conditions on $g_{h2},g_{y1},g_{y2},\phi_{H}$ and $\phi_{Y}$
imply that

$\inf_{\begin{subarray}{c}x\in S_{2\,\epsilon}\\\
h_{1}\in[\underline{h}_{1},\underline{h}_{1}+1]\\\ \theta\in\Theta\\\
X_{\theta\,2}\in A\cap D\end{subarray}}\det
G^{-1}\phi_{H}(Z_{1\,2}(X_{\theta\,2}))\phi_{Y}(Z_{2\,1}(X_{\theta\,2}))\phi_{Y}(Z_{2\,2}(X_{\theta\,2}))\geq
c^{\prime}>0~{},$

where $c^{\prime}$ does not depend on $\theta$, $x$ or $h_{1}$. The claim of
part $(i)$ then follows.
$(ii)$ We have that

$\displaystyle P_{X}^{2}(x,A\cap D)$
$\displaystyle\geq\int_{0}^{\infty}\mathbb{E}(\mathbbm{1}_{\\{X_{\theta\,2}\in
A\cap D\\}}|H_{1}=h_{1},X_{\theta\,0}=x)\phi_{H}(Z_{1\,1})dZ_{1\,1}$
$\displaystyle\geq\inf_{h_{1}\in[\underline{h}_{1},\underline{h}_{1}+1]}\mathbb{E}(\mathbbm{1}_{\\{X_{\theta\,2}\in
A\cap D\\}}|H_{1}=h_{1},X_{\theta\,0}=x)$
$\displaystyle\quad\times\int_{\underline{h}_{1}}^{\underline{h}_{1}+1}{1\over
g_{h2}(h)}\phi_{H}\left({h_{1}-g_{h1}(h)\over g_{h2}(h)}\right)dh_{1}~{},$

where the last inequality follows by the choice of $\underline{h}_{1}$, and
$g_{h2}(h)$ is strictly positive by assumption. Moreover, the boundedness
conditions on $g_{h1}$, $g_{h2}$, and $\phi_{H}$ imply that
$\inf_{|h|<M/\kappa_{h}}g_{h2}(h)^{-1}\phi_{H}\left(g_{h2}(h)^{-1}(h_{1}-g_{h1}(h))\right)\geq
c^{\prime}>0,$ where we emphasize that $c^{\prime}$ does not depend on $x$ or
$\theta$. This concludes the second part. Combining parts $(i)$ and $(ii)$,
the result in II holds with $c_{*}=c^{\prime}c^{\prime\prime}>0$.

III. The Chapman-Kolmogorov equations imply that for any $x\in
S_{1\,\epsilon}$ it holds that

$\displaystyle P_{X}^{n+2}(x,A\cap
D)\geq\int_{S_{2\,\epsilon}}P_{X}^{n}(x,dx_{n})P_{X}^{2}(x_{n},A\cap D)\geq
c_{*}\mu_{Leb}(A\cap D)P_{X}^{n}(x,S_{2\,\epsilon})>0,$

where the last two inequalities follow, respectively, from Parts II and I (for
a sufficiently large $n$). ∎

######  Lemma B.2 (Aperiodicity).

Consider the setting of Lemma 1. Then, the Markov chain
$\\{X_{\theta\,t},t\geq 0\\}$ is aperiodic.

###### Proof.

It follows from Proposition A1.1 in Chan (1990), that to establish
aperiodicity it suffices to show that that for each $x\in D$ there exists an
$n\in\mathbb{Z}_{+}$ such that $P_{X}^{n+2}(x,D)>0$ and $P_{X}^{n+3}(x,D)>0$,
where $D$ is a small set. We divide the proof in three parts. In part $(i)$ we
show that the set $D$ defined in Lemma B.1 is a small set. In part $(ii)$ we
show that for each $x \in D$ there exists an $n$ such that
$P_{X}^{n+2}(x,D)>0$. In part $(iii)$ we show that for the same $x$ and same
$n$ defined in part $(ii)$ it holds that $P_{X}^{n+3}(x,D)>0$.
$(i)$ We note that by repeating the arguments in Part II of Lemma B.1 with
$S_{2\,\epsilon}$ replaced by $D$ we have that for any
$A\in\mathcal{B}(\mathcal{X})$ there exist $c_{*}^{\prime}>0$ and an open
rectangular region $D^{\prime}$ such that $\inf_{x\in D}P_{X}^{2}(x,A\cap
D^{\prime})\geq c_{*}^{\prime}\mu_{Leb}(A\cap D^{\prime}).$
$(ii)$ It follows from Parts I and II of Lemma B.1 that for any $x\in D$ there
exists an $n$ such that $P_{X}^{n}(x,S_{2\,\epsilon})>0$ and for any $x\in
S_{2\,\epsilon}$ we have that $P_{X}^{2}(x,D)>0$. The claim follows from an
application of the Chapman-Kolmogorov equation.
$(iii)$ Note that in the proof of Lemma B.1 we can choose an $M$ and
$M^{\prime}$ with $M>M^{\prime}$ such that $P_{X}^{n}(x,S_{2\,\epsilon})>0$
and $P_{X}^{n+1}(x,S_{2\,\epsilon}^{\prime})>0$ where
$S_{2\,\epsilon}^{\prime}=\\{(h,f,d)^{\prime}\in\mathcal{X}:\kappa_{h}|h|+\kappa_{f}|f|+\kappa_{d}|d|\leq
M^{\prime}\\}$. It is straightforward to see in the proof of Lemma B.1 that
$M$ can be chosen as any sufficiently large constant. Furthermore, we have
that $\inf_{x\in S_{2\,\epsilon}^{\prime}}P_{X}^{2}(x,D)\geq\inf_{x\in
S_{2\,\epsilon}}P_{X}^{2}(x,D)\geq c_{*}\mu_{Leb}(D)>0$. The Chapman-
Kolmogorov equation implies the claim since we have that
$P_{X}^{n+3}(x,D)\geq\int_{S_{2\,\epsilon}^{\prime}}P_{X}^{n+1}(x,dx_{n+1})P_{X}^{2}(x_{n+1},D)\geq
c_{*}\mu_{Leb}(D)P^{n+1}_{X}(x,S_{2\,\epsilon}^{\prime})>0.$ ∎

######  Lemma B.3 (Drift Criterion).

Consider the setting of Lemma 1. Then, the Markov chain
$\\{X_{\theta\,t},t\geq 0\\}$ satisfies
$\mathbb{E}\left(q_{X}(X_{\theta\,t})|X_{\theta\,t-1}=x\right)\leq(1-\gamma_{1})q_{X}(x)+\gamma_{2}\mathbbm{1}_{\\{x\in
C\\}}$ for some $\gamma_{1}>0$ and $\gamma_{2}<\infty$ where $C$ is a small
set. Furthermore, $\gamma_{1}$, $\gamma_{2}$ and $C$ do not depend on
$\theta$.

###### Proof.

Set $C$ equal to $S_{2\,\epsilon}$ and note that Part II in the proof of Lemma
B.1 establishes that $S_{2\,\epsilon}$ is a small set that does not depend on
$\theta$.
When $x\in S_{1\,\epsilon}$, we distinguish two cases: $(i)$
$|h|>M_{\epsilon}$ or $(ii)$ $|h|\leq M_{\epsilon}$.
Case $(i)$. From (30) we have that

$\displaystyle\mathbb{E}_{x}(q_{X}(X_{\theta\,t}))-1$
$\displaystyle\leq\mathbb{E}_{x}\left(\kappa^{\prime}\mathbf{C}_{\epsilon}(Z_{t})\dot{x}\right)^{2r_{m}}=\left(\kappa^{\otimes
2r_{m}}\right)^{\prime}\mathbb{E}\left(\mathbf{C}_{\epsilon}(Z_{t})^{\otimes
2r_{m}}\right)\dot{x}^{\otimes 2r_{m}}~{}.$ (36)

Following steps analogous to the ones used to go from (29) to (B) we have that

$\displaystyle\mathbb{E}_{x}\left(q_{X}(X_{\theta\,t})|X_{\theta\,t-1}=x\right)$
$\displaystyle\leq
1+(\kappa^{\prime}\dot{x})^{2r_{m}}-v^{\prime}\dot{x}^{\otimes
2r_{m}}\leq(1-\gamma_{1})q_{X}(x)~{},$

where $\gamma_{1}\in(0,1)$ and does not depend $\theta$.
Case $(ii)$. From Part I of Lemma B.1 (case $(ii)$) it follows that
$\mathbb{E}_{x}\left(q_{X}(X_{\theta\,t})\right)-1\leq\mathbb{E}_{x}\left(\overline{C}_{z\,\epsilon}+\kappa_{\mathbf{f}}^{\prime}\overline{B}\mathbf{f}\right)^{2r_{m}}$.
We observe that

$\displaystyle\mathbb{E}_{x}\left(\overline{C}_{z\,\epsilon}+\kappa_{\mathbf{f}}^{\prime}\overline{B}\mathbf{f}\right)^{2r_{m}}=\left(\left(\mathbb{E}_{x}\left(\overline{C}_{z\,\epsilon}+\kappa_{\mathbf{f}}^{\prime}\overline{B}\mathbf{f}\right)^{2r_{m}}\right)^{1\over
2r_{m}}\right)^{2r_{m}}$
$\displaystyle\leq\left(\left\|\overline{C}_{z\,\epsilon}\right\|_{L_{2r_{m}}}+\kappa_{\mathbf{f}}^{\prime}\overline{B}\mathbf{f}\right)^{2r_{m}}~{},$

and note that the assumptions on the innovations imply that
$\left\|\overline{C}_{z\,\epsilon}\right\|_{L_{2r_{m}}}$ exists. Using steps
analogous to those used to get to (32) we have that

$\displaystyle\left(\left\|\overline{C}_{z\,\epsilon}\right\|_{L_{2r_{m}}}+\kappa_{\mathbf{f}}^{\prime}\overline{B}\mathbf{f}\right)^{2r_{m}}$
$\displaystyle\leq\mathbb{E}_{x}\left(\kappa^{\prime}\mathbf{C}_{\epsilon}(Z_{t})\ddot{x}\right)^{2r_{m}}\leq(\kappa^{\otimes
2r_{m}})^{\prime}\mathbb{E}\left(\mathbf{C}_{\epsilon}(Z_{t})^{\otimes
2r_{m}}\right)\dot{x}^{\otimes 2r_{m}}~{}.$

The claim of case $(ii)$ then follows using the same steps of case $(i)$ after
equation (36). When $x\in S_{2\,\epsilon}$ it follows from Assumption 1 and
the definition of $S_{2\,\epsilon}$ that $\sup_{\begin{subarray}{c}x\in
S_{2\,\epsilon}\\\
\theta\in\Theta\end{subarray}}\mathbb{E}\left(q_{X}(X_{\theta\,t})|X_{\theta\,t-1}=x\right)\leq\gamma_{2}<\infty,$
where we have used the fact the expectation exists and it is bounded over
$\Theta$ for every $x\in S_{2\,\epsilon}$ provided that $Z_{1\,t}$ and
$Z_{2\,t}$ have $2r_{m}$ moments. Since $(1-\gamma_{1})q_{X}(x)$ is positive
the claim of the proposition holds when $x\in S_{2\,\epsilon}$. ∎

## References

  * Andersen et al. (2003) Andersen, T. G., Bollerslev, T., Diebold, F. X., and Labys, P. (2003).  Modeling and Forecasting Realized Volatility.  Econometrica, 71, 579–625.
  * Banerjee et al. (2005) Banerjee, A., Merugu, S., Dhillon, I. S., and Ghosh, J. (2005).  Clustering with Bregman Divergences.  Journal of Machine Learning Research, 6, 1705–1749.
  * Bougerol and Picard (1992) Bougerol, P. and Picard, N. (1992).  Stationarity of GARCH processes and of some nonnegative time series.  Journal of Econometrics, 52, 115–127.
  * Bregman (1967) Bregman, L. M. (1967).  The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming.  USSR Computational Mathematics and Mathematical Physics, 7, 200–217.
  * Breiman (2001) Breiman, L. (2001).  Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author).  Statistical Science, 16, 199 – 231.
  * Brownlees and Guðmundsson (2021) Brownlees, C. and Guðmundsson, G. S. (2021).  Performance of Empirical Risk Minimization for Linear Regression with Dependent Data.  Technical report.
  * Carrasco and Chen (2002) Carrasco, M. and Chen, X. (2002).  Mixing and Moment Properties of various GARCH and Stochastic Volatility Models.  Econometric Theory, 18, 17–39.
  * Chan (1990) Chan, K. (1990).  Deterministic stability, stochastic stability, and ergodicity.  In H. Tong, editor, Non-linear Time Series: A Dynamical System Approach, pages 448–466. Clarendon Press.
  * Creal et al. (2013) Creal, D., Koopman, S. J., and Lucas, A. (2013).  Generalized Autoregressive Score Models with Applications.  Journal of Applied Econometrics, 28, 777–795.
  * Davidson (1994) Davidson, J. (1994).  Stochastic Limit Theory.  Oxford University Press, New York.
  * Davis et al. (2021) Davis, R. A., Fokianos, K., Holan, S. H., Joe, H., Livsey, J., Lund, R., Pipiras, V., and Ravishanker, N. (2021).  Count Time Series: A Methodological Review.  Journal of the American Statistical Association, pages 1–15.
  * Devroye et al. (1996) Devroye, L., Györfi, L., and Lugosi, G. (1996).  A Probabililstic Theory of Pattern Recognition.  Springer, New York.
  * Engle and Russell (1998) Engle, R. and Russell, J. R. (1998).  Autoregressive Conditional Duration: A New Model for Irregularly Spaced Transaction Data.  Econometrica, 66, 1127–1162.
  * Engle and Gallo (2006) Engle, R. F. and Gallo, G. M. (2006).  A multiple indicators model for volatility using intra-daily data.  Journal of Econometrics, 131, 3–27.
  * Francq and Zakoïan (2010) Francq, C. and Zakoïan, J. (2010).  GARCH Models: Structure, Statistical Inference and Financial Applications.  Wiley.
  * Francq and Zakoïan (2004) Francq, C. and Zakoïan, J.-M. (2004).  Maximum likelihood estimation of pure GARCH and ARMA-GARCH processes.  Bernoulli, 10, 605 – 637.
  * Francq and Zakoïan (2006) Francq, C. and Zakoïan, J.-M. (2006).  Mixing Properties of a General Class of GARCH(1,1) Models without Moment Assumptions on the Observed Process.  Econometric Theory, 22, 815–834.
  * Gallant and White (1988) Gallant, A. R. and White, H. (1988).  A Unified Theory of Estimation and Inference for Nonlinear Dynamic Models.  Basil Blackwell.
  * Gijbels et al. (1999) Gijbels, I., Pope, A., and Wand, M. P. (1999).  Understanding Exponential Smoothing via Kernel Regression.  Journal of the Royal Statistical Society: Series B, 61, 39–50.
  * Hansen and Lunde (2006) Hansen, P. R. and Lunde, A. (2006).  Consistent ranking of volatility models.  Journal of Econometrics, 131, 97–121.
  * Härdle and Tsybakov (1997) Härdle, W. and Tsybakov, A. (1997).  Local polynomial estimators of the volatility function in nonparametric autoregression.  Journal of Econometrics, 81, 223–242.
  * Harvey and Chakravarty (2008) Harvey, A. and Chakravarty, T. (2008).  Beta-t-(E)GARCH.  Cambridge working papers in economics, Faculty of Economics, University of Cambridge.
  * Harvey (2013) Harvey, A. C. (2013).  Dynamic Models for Volatility and Heavy Tails With Applications to Financial and Economic Time Series.  Cambridge University Press.
  * Jiang and Tanner (2010) Jiang, W. and Tanner, M. (2010).  Risk Minimization for Time Series Binary Choice with Variable Selection.  Econometric Theory, 26, 1437–1452.
  * Kristensen (2009) Kristensen, D. (2009).  On stationarity and ergodicity of the bilinear model with applications to GARCH models.  Journal of Time Series Analysis, 30, 125–144.
  * Kristensen and Rahbek (2005) Kristensen, D. and Rahbek, A. (2005).  Asymptotics of the QMLE for a Class of ARCH(q) Models.  Econometric Theory, 21, 946–961.
  * Lanne and Saikkonen (2005) Lanne, M. and Saikkonen, P. (2005).  Non-linear GARCH models for highly persistent volatility.  The Econometrics Journal, 8, 251–276.
  * Laurent et al. (2013) Laurent, S., Rombouts, J. V., and Violante, F. (2013).  On loss functions and ranking forecasting performances of multivariate volatility models.  Journal of Econometrics, 173, 1–10.
  * Lee and Hansen (1994) Lee, S.-W. and Hansen, B. E. (1994).  Asymptotic Theory for the Garch(1,1) Quasi-Maximum Likelihood Estimator.  Econometric Theory, 10, 29–52.
  * Liebscher (1996) Liebscher, E. (1996).  Strong convergence of sums of $\alpha$-mixing random variables with applications to density estimation.  Stochastic Processes and their Applications, 65, 69–80.
  * Liebscher (2005) Liebscher, E. (2005).  Towards a Unified Approach for Proving Geometric Ergodicity and Mixing Properties of Nonlinear Autoregressive Processes.  Journal of Time Series Analysis, 25, 669–689.
  * Ling and McAleer (2003) Ling, S. and McAleer, M. (2003).  Asymptotic Theory for a Vector ARMA-GARCH Model.  Econometric Theory, 19, 280–310.
  * Linton and Mammen (2005) Linton, O. and Mammen, E. (2005).  Estimating Semiparametric ARCH($\infty$) Models by Kernel Smoothing Methods.  Econometrica, 73, 771–836.
  * Lu and Jiang (2001) Lu, Z. and Jiang, Z. (2001).  $L_{1}$ geometric ergodicity of a multivariate nonlinear AR model with an ARCH term.  Statistics & Probability Letters, 51, 121–130.
  * Lumsdaine (1996) Lumsdaine, R. L. (1996).  Consistency and Asymptotic Normality of the Quasi-maximum Likelihood Estimator in IGARCH(1,1) and Covariance Stationary GARCH(1,1) Models.  Econometrica, 64, 575–596.
  * Masry and Tjøstheim (1995) Masry, E. and Tjøstheim, D. (1995).  Nonparametric Estimation and Identification of Nonlinear ARCH Time Series: Strong Convergence and Asymptotic Normality.  Econometric Theory, 11, 258–289.
  * Meddahi (2002) Meddahi, N. (2002).  A Theoretical Comparison Between Integrated and Realized Volatility.  Journal of Applied Econometrics, 17, 479–508.
  * Meddahi (2003) Meddahi, N. (2003).  ARMA representation of integrated and realized variances.  The Econometrics Journal, 6, 335–356.
  * Meitz and Saikkonen (2008) Meitz, M. and Saikkonen, P. (2008).  Ergodicity, Mixing and Existence of Moments of a Class of Markov Models with Applications to GARCH and ACD Models.  Econometric Theory, 24, 1291–1320.
  * Mendelson (2015) Mendelson, S. (2015).  Learning without concentration.  Journal of the ACM, 62, 1–25.
  * Meyn and Tweedie (1993) Meyn, S. and Tweedie, R. (1993).  Markov Chains and Stochastic Stability.  Springer, London.
  * Morris (1982) Morris, C. N. (1982).  Natural Exponential Families with Quadratic Variance Functions.  The Annals of Statistics, 10, 65 – 80.
  * Nelson (1992) Nelson, D. B. (1992).  Filtering and forecasting with misspecified ARCH models I: Getting the right variance with the wrong model.  Journal of Econometrics, 52, 61–90.
  * Pagan and Schwert (1990) Pagan, A. R. and Schwert, G. (1990).  Alternative models for conditional stock volatility.  Journal of Econometrics, 45, 267–290.
  * Patton (2011) Patton, A. J. (2011).  Volatility forecast comparison using imperfect volatility proxies.  Journal of Econometrics, 160, 246–256.
  * Patton (2020) Patton, A. J. (2020).  Comparing Possibly Misspecified Forecasts.  Journal of Business & Economic Statistics, 38, 796–809.
  * Pötscher and Prucha (1997) Pötscher, B. M. and Prucha, I. R. (1997).  Dynamic Nonlinear Econometric Models: Asymptotic Theory.  Springer.
  * Roberts and Rosenthal (2004) Roberts, G. and Rosenthal, J. (2004).  General State Space Markov Chains and MCMC algorithms.  Probability Surveys, 1, 20–71.
  * Rosenthal (1995) Rosenthal, J. (1995).  Minorization Conditions and Convergence Rates for Markov Chain Monte Carlo.  Journal of the Americal Statistical Association, 90, 558–566.
  * Straumann and Mikosch (2006) Straumann, D. and Mikosch, T. (2006).  Quasi-maximum-likelihood estimation in conditionally heteroscedastic time series: A stochastic recurrence equations approach.  The Annals of Statistics, 34, 2449 – 2495.
  * Tong (1990) Tong, H. (1990).  Non-linear Time Series: A Dynamical System Approach.  Clarendon Press.
  * Vapnik and Chervonenkis (1971) Vapnik, V. and Chervonenkis, A. (1971).  On the uniform convergence of relative frequencies of events to their probabilities.  Theory of Probability and its Applications, 197, 264–280.
  * Vapnik and Chervonenkis (1974) Vapnik, V. N. and Chervonenkis, A. Y. (1974).  Theory of Pattern Recognition [in Russian].  Nauka.