# On the stability test for
reproducing kernel Hilbert spaces

Mauro Bisiacco Department of Information Engineering, University of Padova,
Padova, Italy<EMAIL_ADDRESS>Gianluigi Pillonetto Department of
Information Engineering, University of Padova, Padova, Italy
(giapi@dei.unipd.it). This work has been partially supported by the Italian
SIR project RBSI14JYM2Learn4AP and by the PRIN project 2015 2015PJ28EP.

###### Abstract

Reproducing kernel Hilbert spaces (RKHSs) are special Hilbert spaces where all
the evaluation functionals are linear and bounded. They are in one-to-one
correspondence with positive definite maps called kernels. Stable RKHSs enjoy
the additional property of containing only functions defined over the real
line and absolutely integrable. Necessary and sufficient conditions for RKHS
stability are known in the literature: the integral operator induced by the
kernel must be bounded as map between $\mathcal{L}_{\infty}$, the space of
essentially bounded (test) functions, and $\mathcal{L}_{1}$, the space of
absolutely integrable functions. Considering Mercer (continuous) kernels in
continuous-time and the entire discrete-time class, we show that the stability
test can be reduced to the study of the kernel operator over test functions
which assume (almost everywhere) only the values $\pm 1$. They represent the
same functions needed to investigate stability of any single element in the
RKHS. In this way, the RKHS stability test becomes an elegant generalization
of a straightforward result concerning Bounded-Input Bounded-Output (BIBO)
stability of a single linear time-invariant system.

Keywords: Linear time-invariant dynamic systems; BIBO stability; kernel-based
regularization; reproducing kernel Hilbert spaces

##  1 Introduction

In control literature, the term system identification is used to indicate
procedures that learn models of dynamic systems from input-output data [32,
2]. When the system under study is linear and time-invariant, the problem
reduces to estimating particular functions known as system impulse responses.
Given any input, they permit to calculate the corresponding system output via
a convolution. For physical (causal) continous-time systems, impulse responses
are real-valued functions defined over the positive real axis
$\mathbb{R}_{+}$.
Over the years, many approaches for system identification have been suggested.
One of the most popular relies on traditional statistics and can be called the
_classical system identification framework_ as described in [16] and [26]. It
uses finite-dimensional parameter structures and apply techniques, like
maximum likelihood or prediction error methods (PEM), to estimate the
parameters. The concept of discrete model order is then adopted to control
their complexity.
In recent years, the regularization techniques described in [17, 21] have
proved to be a powerful alternative to these classical procedures based on
PEM. Instead of postulating parametric structures, impulse responses are
directly searched for in high-dimensional (possibly infinite-dimensional)
spaces. Ill-posedness is circumvented by including information on the physics
of the problem. Important regularizers proposed in the last years include
notions of stability [20, 19, 10, 8]. In particular, Bounded-Input Bounded-
Output (BIBO) stability is a fundamental concept encountered in control
theory. It ensures that a dynamic system, solicited by any bounded input,
returns only bounded outputs. The related necessary and sufficient condition
is especially simple: a linear and time-invariant system is BIBO stable if and
only if the corresponding impulse response is absolutely integrable [14]. For
illustration purposes, it is now useful to reformulate this condition. Let
$\mathcal{L}_{\infty}$ indicate the space of essentially bounded functions of
norm $\|\cdot\|_{\infty}$ while $\mathcal{L}_{1}$ is the space of absolutely
integrable functions of norm $\|\cdot\|_{1}$. Define also
$\mathcal{U}_{\infty}$ as the subset of $\mathcal{L}_{\infty}$ containing test
functions $u$ as follows:

$\mathcal{U}_{\infty}:=\left\\{u\in{\mathcal{L}}_{\infty}\ \text{with}\
u:\mathbb{R}_{+}\rightarrow\mathbb{R}\ \text{and}\ |u(t)|=1\
\mbox{a.e.}\right\\}.$ (1)

Then, using $f$ to denote a system impulse response, it is immediate to prove
that

$\text{BIBO stability}\ \iff\
\sup_{u\in\mathcal{U}_{\infty}}\int_{0}^{+\infty}f(t)u(t)dt=\|f\|_{1}<+\infty,$
(2)

where the equality on the r.h.s. holds since the supremum is obtained setting
$u(t)$ to the sign of $f(t)$ almost everywhere. In the regularized framework
for system identification, one key tool to include BIBO stability in the
estimation process is the use of particular spaces of functions, known as BIBO
stable (or just stable) Reproducing kernel Hilbert spaces (RKHSs) [21, 7, 18].
Any function contained in such spaces enjoys the condition (2).
RKHSs theory is described in the fundamental works [1, 4]. Their simplest
formulation says that they are Hilbert spaces where any pointwise evaluation
of a function is a linear and bounded functional. They are also in one-to-one
correspondence with positive definite kernels. In our setting, a kernel $K$ is
a real-valued map over $\mathbb{R}_{+}\times\mathbb{R}_{+}$ such that, for any
$n$ and $n$-uple of nonnegative scalars $\\{x_{1},\ldots,x_{n}\\}$, the
$n\times n$ matrix $\mathbf{K}$, with $K(x_{i},x_{j})$ as $(i,j)$ entry, is
positive semidefinite. While first applications of RKHSs regarding statistics,
approximation theory and computer vision can be found in the eighties [5, 22,
31], such spaces were then introduced in machine learning by Federico Girosi
in [13]. Combining RKHSs and Tikhonov regularization theory [29, 28] leads to
powerful algorithms for function estimation like regularization
networks/kernel-ridge regression and support vector machines [30, 25, 27, 3].
In system identification, the concept of stable RKHSs for impulse response
estimation was introduced in [20], see also e.g. [20, 10, 12, 21] for further
developments. As already mentioned above, these spaces, induced by the so
called stable kernels, are RKHSs containing only absolutely integrable
functions. They permit to define impulse response estimators looking for
solutions that balance adherence to experimental data and a penalty term
accounting for BIBO stability. This makes the search space manageable by
inducing a ranking of possible solutions: among dynamic systems that describe
the data in a similar way, the one that is, in some sense, more stable will be
selected.
A complete characterization of stable RKHSs can be found in [9, 7]. It makes
use of the kernel operator $L_{K}$ which, for any given kernel $K$, maps
functions $u$ in $y_{u}$ where

$y_{u}=L_{K}[u]\ \iff\ y_{u}(t)=\int_{0}^{+\infty}\ K(t,\tau)u(\tau)d\tau,\
t\geq 0.$

Letting $\mathcal{H}$ be the RKHS induced by $K$, one then has

$\text{BIBO stable $\mathcal{H}$}\ \iff\ y_{u}\in\mathcal{L}_{1}\ \ \forall
u\in\mathcal{L}_{\infty}.$ (3)

A simple application of the closed graph theorem, along the same line e.g. of
that described in Lemma 4.1 contained in [11], shows that such condition is
equivalent to the continuity of the kernel operator as a map from
$\mathcal{L}_{1}$ to $\mathcal{L}_{\infty}$. In terms of its operator norm
$\|K\|_{\infty,1}$, this means that one must have $\|K\|_{\infty,1}<+\infty$.
In light of this observation, an equivalent stability condition can be now
introduced. We use $\|\cdot\|_{\infty}$ to indicate the norm in
$\mathcal{L}_{\infty}$ and denote the boundary of the unit ball in such space
as

$\mathcal{B}_{\infty}=\Big{\\{}\
u\in\mathcal{L}_{\infty}:\|u\|_{\infty}=1\Big{\\}}.$ (4)

Then, the following reformulation of (3) holds:

$\text{BIBO stable $\mathcal{H}$}\ \iff\ \sup_{u\in\mathcal{B}_{\infty}}\
\big{\|}\int_{0}^{+\infty}K(\cdot,x)u(x)dx\big{\|}_{\mathcal{L}_{1}}=\|K\|_{\infty,1}<+\infty.$
(5)

This equivalence is, in some sense, closer to the condition (2). The
requirement on the single system, represented by the impulse response $f$, has
become a condition on the kernel $K$ which embeds an infinite set of impulse
responses (all those belonging to the induced RKHS $\mathcal{H}$). However,
something that _disturbs_ the symmetry is the fact that the supremum is taken
over $\mathcal{B}_{\infty}$, a larger set which strictly contains
$\mathcal{U}_{\infty}$. To fix this issue, we work in continuous-time and
consider continuous kernels $K$. More formally, the following assumption is
adopted.

######  Assumption 1.

The kernel $K:\mathbb{R}_{+}\times\mathbb{R}_{+}\rightarrow\mathbb{R}$ is
continuous. In addition, let $\mathcal{I}$ be the set containing all the $t$
such that the kernel sections $K(t,\cdot)$ are not absolutely integrable.
Then, the Lebesgue measure of $\mathcal{I}$ is null.

The first point in the assumption says that we consider the fundamental Mercer
class which contains in practice all the kernels adopted in machine learning
and system identification [25, 17]. The second part regarding the Lebesgue
measure of $\mathcal{I}$ guarantees that the kernel operator makes sense.
Then, we will show that also in the RKHS setting the test functions can be
confined to $\mathcal{U}_{\infty}$. Beyond its theoretical interest, one
additional advantage of this outcome is quite obvious: any procedure becomes
simpler by restricting its domain of competency. An example can be found in
[6], where this result was first proved to hold for block-diagonal (discrete-
time) kernels. Then, it was used for studying this class and obtaining new
insights on stable kernels theory. Hence, it sounds likely that our result,
reported formally in the following theorem, can be important in the future to
face similar problems involving explicit computation of the kernel operator
norm.

######  Theorem 2.

Let $\mathcal{H}$ be the RKHS induced by the continuous (Mercer) kernel
$K:\mathbb{R}_{+}\times\mathbb{R}_{+}\rightarrow\mathbb{R}$. Then, if
Assumption 1 holds, one has

$\text{BIBO stable $\mathcal{H}$}\ \iff\ \sup_{u\in\mathcal{U}_{\infty}}\
\big{\|}\int_{0}^{+\infty}K(\cdot,x)u(x)dx\big{\|}_{\mathcal{L}_{1}}=\|K\|_{\infty,1}<+\infty$
(6)

with $\mathcal{U}_{\infty}$ defined in (1).

The proof is reported below, in Section 2. It is worth also remarking that
such result holds also considering the entire class of discrete-time kernels
defined over $\mathbb{N}\times\mathbb{N}$. This is discussed in Section 3.

##  2 Proof of Theorem 2

As detailed below, the proof of Theorem 2 is not trivial also because the well
known Bauer’s theorem cannot be used in our context. Such principle states
that the maximum of a convex function over a closed convex set is obtained at
the boundary of the optimization domain, e.g. see [23][Part 4] or [15]. But in
our case $\mathcal{U}_{\infty}$ is a proper subset of the boundary
$\mathcal{B}_{\infty}$ of the unit ball.
It is natural to divide the proof in two parts, assuming that the kernel $K$
is either stable or unstable.

###  2.1 The case of stable $K$

Let $K$ be a Mercer and stable kernel over
$\mathbb{R}_{+}\times\mathbb{R}_{+}$. It comes that
$L_{K}:{\mathcal{L}}_{\infty}\rightarrow{\mathcal{L}}_{1}$ is well defined
with finite operator norm, i.e.

$\|K\|_{\infty,1}=\sup_{u\in\mathcal{B}_{\infty}}\ \|L_{K}[u]\|_{1}<+\infty.$

Now, we want to show that

$\sup_{u\in{\mathcal{B}}_{\infty}}\
\|L_{K}[u]\|_{1}=\sup_{u\in{\mathcal{U}}_{\infty}}\ \|L_{K}[u]\|_{1}.$ (7)

Since ${\mathcal{U}}_{\infty}\subset{\mathcal{B}}_{\infty}$, the problem
reduces to proving that the following statement holds true.

######  Proposition 3.

It holds that

$\sup_{u\in\mathcal{U}_{\infty}}\
\|L_{K}[u]\|_{1}\geq\sup_{u\in\mathcal{B}_{\infty}}\ \|L_{K}[u]\|_{1}.$

To prove this, some useful lemmas are first introduced.

######  Lemma 4.

For any couple of functions $y,w\in{\mathcal{L}}_{1}$, the map
$F:{\mathbb{R}}\ \rightarrow\ {\mathbb{R}}_{+}$ defined by

$F(x):=\int_{0}^{+\infty}\ |y(t)+xw(t)|dt,\ x\in{\mathbb{R}}$

is convex.

###### Proof.

We need to prove that

$F(ax_{1}+(1-a)x_{2})\leq aF(x_{1})+(1-a)F(x_{2}),\ \forall a:\ 0\leq a\leq 1$

which is equivalent to showing that

$\displaystyle\int_{0}^{+\infty}\ |y(t)+(ax_{1}+(1-a)x_{2})w(t)|dt\leq
a\int_{0}^{+\infty}\ |y(t)+x_{1}w(t)|dt$
$\displaystyle\qquad\qquad\qquad+(1-a)\int_{0}^{+\infty}\
|y(t)+x_{2}w(t)|dt,\quad\forall a:\ 0\leq a\leq 1.$

Letting

$z(t):=y(t)+x_{2}w(t)\ \ \text{and}\ \ x:=x_{1}-x_{2},$

in turn this is equivalent to

$\displaystyle\int_{0}^{+\infty}\ |z(t)+axw(t)|dt\leq a\int_{0}^{+\infty}\
|z(t)+xw(t)|dt$ $\displaystyle\qquad\qquad\qquad+(1-a)\int_{0}^{+\infty}\
|z(t)|dt,\quad\forall a:\ 0\leq a\leq 1.$

and also to

$\displaystyle\int_{0}^{+\infty}\ (|z(t)+axw(t)|-|az(t)+axw(t)|)dt$
$\displaystyle\qquad\qquad\qquad\leq(1-a)\int_{0}^{+\infty}\
|z(t)|dt,\quad\forall a:\ 0\leq a\leq 1.$

From $-|a-b|\leq|a|-|b|\leq|a-b|$, the lhs satisfies

$\int_{0}^{+\infty}(|z(t)+axw(t)|-|az(t)+axw(t)|)dt\leq$
$\leq\int_{0}^{+\infty}\ |(z(t)+axw(t))-(az(t)+axw(t))\
|dt=|1-a|\int_{0}^{+\infty}\ |z(t)|dt.$

Using $a\leq 1$, this can be also rewritten as

$\int_{0}^{+\infty}(|z(t)+axw(t)|-|az(t)+axw(t)|)dt\leq(1-a)\int_{0}^{+\infty}\
|z(t)|dt$

and this concludes the proof.
∎

An immediate consequence of Lemma 4 is that, if

$v=u+xw\ \text{with}\ u,w\in{\mathcal{L}}_{\infty}\ \text{and}\
x\in{\mathbb{R}},$

then the function

$G(x):=\|L_{K}[v]\|_{1}=\|L_{K}[u]+xL_{K}[w]\|_{1}$

is convex. Therefore, using Bauer’s maximum principle, if $x\in[a,b]$ its
maximum value is attained at either $x=a$ or $x=b$. For any
$u\in{\mathcal{B}}_{\infty}$ we define

$E_{u}(a,b):=\left\\{t\in{\mathbb{R}}_{+}:\ a\leq|u(t)|<b,\right\\},\ \
\forall a,b:\ 0\leq a<b\leq 1.$

The next lemma then shows that the test inputs can be restricted to a proper
subset of ${\mathcal{B}}_{\infty}$.

######  Lemma 5.

For any $u\in{\mathcal{B}}_{\infty}$, there exists
$v\in{\mathcal{B}}_{\infty}$ such that $\|L_{K}[v]\|_{1}\geq\|L_{K}[u]\|_{1}$
and for any $t\in{\mathbb{R}}_{+}$ it holds that $|v(t)|\geq\frac{1}{2}$.

###### Proof.

Let $\psi_{E}(x)$ denote the indicator function of the set $E$ and consider

$w(t,x):=u(t)+(x-1)\psi_{E_{u}(\frac{1}{4},\frac{1}{2})}(t)u(t).$

One can easily see that $w(t,x)=u(t)$ outside
$E_{u}(\frac{1}{4},\frac{1}{2})$, while $w(t,x)=xu(t)$ in
$E_{u}(\frac{1}{4},\frac{1}{2})$. Therefore
$w(\cdot,x)\in{\mathcal{B}}_{\infty}$ at least in the interval $x\in[-2,2]$.
So, by the convexity property discussed before, one has

$\|L_{K}[w(\cdot,x)]\|_{1}\geq\|L_{K}[u]\|_{1}$

for $x=x_{0}$, where either $x_{0}=-2$ or $x_{0}=2$. Now, define
$z(t):=w(t,x_{0})$ and note that

$z\in{\mathcal{B}}_{\infty},\ \|L_{K}[z]\|_{1}\geq\|L_{K}[u]\|_{1},\ \
E_{z}(\frac{1}{4},\frac{1}{2})=\emptyset$

since all the values in the interval $[\frac{1}{4},\frac{1}{2})$ have been
moved to the interval $[\frac{1}{2},1)$. Define also

$y(t,x):=z(t)+x\psi_{E_{z}(0,\frac{1}{4})}(t)$

and note that such function belongs to ${\mathcal{B}}_{\infty}$ (at least) if
$x\in[-\frac{3}{4},\frac{3}{4}]$. Choosing again $x_{0}$ as the maximizing
point, it comes that $v(t):=y(t,x_{0})$ satisfies

$v\in{\mathcal{B}}_{\infty},\
\|L_{K}[v]\|_{1}\geq\|L_{K}[z]\|_{1}\geq\|L_{K}[u]\|_{1},\
|v(t)|\geq\frac{1}{2},\ \forall t\geq 0$

and this concludes the proof.
∎

######  Remark 6.

If the set $E_{u}(\frac{1}{4},\frac{1}{2})$ has null Lebesgue measure, it
would be pointless to replace $u$ with $z$ since only what happens a.e. is
relevant. Such situation would e.g. arise if $u=0$ a.e. or
$u\in{\mathcal{U}}_{\infty}$ (which would lead to an empty set). In these two
limit cases, one could directly assume either $v=1$ a.e. or $v=u$ everywhere.

The next lemma allows us to conclude that the set of test inputs can be
further restricted to ${\mathcal{U}}_{\infty}$.

######  Lemma 7.

For any $u\in{\mathcal{B}}_{\infty}$ such that $|u(t)|\geq\frac{1}{2}$ for any
$t\in{\mathbb{R}}_{+}$, and for any $\epsilon>0$, there exists
$v\in{\mathcal{U}}_{\infty}$ such that
$\|L_{K}[v]\|_{1}\geq\|L_{K}[u]\|_{1}-\epsilon$.

###### Proof.

First, we introduce two key sequences of functions denoted by $w_{n}(t,x)$ and
$v_{n}\in{\mathcal{B}}_{\infty},\ n\in{\mathbb{N}}$. As clear in what follows,
they are built recursively, one after the other. First, the two sequences
satisfy

$\displaystyle v_{1}(t)$ $\displaystyle:=$ $\displaystyle u(t)$ $\displaystyle
w_{n+1}(t,x)$ $\displaystyle=$ $\displaystyle
v_{n}(t)+(x-1)\psi_{E_{v_{n}}(\frac{2^{n}-1}{2^{n}},\frac{2(2^{n}-1)}{2^{n+1}-1})}(t)v_{n}(t),\
\forall n\geq 1.$

Just using this partial information on their nature, one can already see that
$v_{n}(\cdot)\in{\mathcal{B}}_{\infty}$ implies

$w_{n+1}(\cdot,x)\in{\mathcal{B}}_{\infty}$

at least in the interval

$x\in\Big{[}\frac{1-2^{n+1}}{2(2^{n}-1)},\frac{2^{n+1}-1}{2(2^{n}-1)}\Big{]}.$

So, again by the convexity property discussed before, it holds that

$\|L_{K}[w_{n+1}(\cdot,x)]\|_{1}\geq\|L_{K}[v_{n}(\cdot)]\|_{1}\ \ \text{for
$x=x_{n}$}$ (8)

where

$x_{n}=\frac{1-2^{n+1}}{2(2^{n}-1)}\ \ \text{or}\ \
x_{n}=\frac{2^{n+1}-1}{2(2^{n}-1)}.$

Now, we complete the definition of the sequences by letting

$v_{n+1}(t):=w_{n+1}(t,x_{n})$

where $x_{n}$ is recursively defined as that value satisfying (8). One can
easily see that

$E_{v_{n+1}}(0,\frac{2^{n+1}-1}{2^{n+1}})=\emptyset.$

Furthermore, introducing

$u_{n}(t):=\text{sign}[v_{n}(t)],\ \forall n\geq 1\ \text{and}\ t\geq 0$

one can also see that

$\|u_{n}-v_{n}\|_{\infty}\leq\frac{1}{2^{n+1}},\
u_{n}\in{\mathcal{U}}_{\infty}.$

Therefore, by the stability assumption (continuity of the operator $L_{K}$),
it follows that

$\|L_{K}[u_{n}]-L_{K}[v_{n}]\|_{1}\leq\frac{\|K\|_{\infty,1}}{2^{n+1}}=:a_{n}\
\Rightarrow\ \|L_{K}[u_{n}]\|_{1}\geq\|L_{K}[v_{n}]\|_{1}-a_{n}$

with $a_{n}\geq 0$ monotone non-decreasing and converging to zero. This,
together with the monotone non-decreasing property of $\|L_{K}[v_{n}]\|_{1}$,
implies $\|L_{K}[u_{n}]\|_{1}\geq\|L_{K}[u]\|_{1}-a_{n}$. We can now choose
any $n$ such that $a_{n}\leq\epsilon$, obtaining that $v:=u_{n}$ satisfies the
statement present in this lemma.
∎

######  Remark 8.

The same caveats contained in Remark 6 hold here. Specifically, if
$u\in{\mathcal{U}}_{\infty}$ one could just assume $v=u$.

Now we are ready to prove Statement 3. Let $u_{n}\in{\mathcal{B}}_{\infty}$ be
any sequence such that $\|L_{K}[u_{n}]\|_{1}$ tends to $\|K\|_{\infty,1}$ and
define $\epsilon_{n}:=\frac{1}{n}$. We also build
$v_{n}\in{\mathcal{B}}_{\infty}$ in accordance with Lemma 5 and then, starting
from $v_{n}$, the sequence $w_{n}\in{\mathcal{U}}_{\infty}$ in accordance with
Lemma 7, which makes

$\|L_{K}[w_{n}]\|_{1}\geq\|L_{K}[u_{n}]\|_{1}-\frac{1}{n}.$

It is now immediate to see that $\|L_{K}[w_{n}]\|_{1}$ also tends to
$\|K\|_{\infty,1}$. This completes the proof.

###  2.2 The unstable case

In presence of instability, we can now consider, in place of
${\mathcal{L}}_{1}$, the space ${\mathcal{L}}$ containing all the Lebesgue-
measurable functions from ${\mathbb{R}}_{+}\setminus\mathcal{I}$ to
${\mathbb{R}}$, i.e. the space of Lebesgue measurable and real-valued
functions defined a.e. over ${\mathbb{R}}_{+}$. Abusing notation, we still use

$\|y\|_{1}:=\int_{0}^{+\infty}\ |y(t)|dt,\ \forall y\in{\mathcal{L}}$

noticing that $\|\cdot\|_{1}$ is no more a true norm since one could have
$\|y\|_{1}=+\infty$.
With these facts in mind, since instability is equivalent to

$\sup_{u\in{\mathcal{B}}_{\infty}}\ \|L_{K}[u]\|_{1}=+\infty,$

we now show that

$\sup_{u\in{\mathcal{B}}_{\infty}}\ \|L_{K}[u]\|_{1}=+\infty\ \Rightarrow\
\sup_{u\in{\mathcal{U}}_{\infty}}\ \|L_{K}[u]\|_{1}=+\infty.$ (9)

For this purpose, consider the restriction $K_{n}$ of $K$ to the square
${\mathcal{Q}}_{n}=[0,n]\times[0,n]$. More precisely, $K_{n}$ coincides with
$K$ in ${\mathcal{Q}}_{n}$ and assume null values elsewhere. Since $K$ is
continuous (Mercer), $K_{n}$ is locally bounded over ${\mathcal{Q}}_{n}$ and
hence, absolutely integrable. This implies that $K_{n}$ is a stable kernel.
Define

$a_{n}:=\|K_{n}\|_{\infty,1}=\sup_{u\in{\mathcal{B}}_{\infty}}\
\|L_{K_{n}}[u]\|_{1}=\sup_{u\in{\mathcal{U}}_{\infty}}\ \|L_{K_{n}}[u]\|_{1}$
(10)

where the last equality derives from the result obtained in the first part of
the proof applied to the stable kernel $K_{n}$. We now show that the sequence
$a_{n}$ is divergent reporting the first of two instrumental lemmas useful to
obtain the desired result.

######  Lemma 9.

Let $a_{n}$ be defined in (10). Then, one has

$a:=\lim_{n\rightarrow+\infty}\ a_{n}=+\infty.$ (11)

###### Proof.

The sequence $a_{n}$ is monotonically non decreasing. To see this, one can
e.g. apply inputs with support over $[0,n]$ to kernel operators induced by
$K_{m}$, with $m\geq n$, obtaining outputs with $1$-norms not less than those
defined by the same inputs applied to kernel operators associated with
$K_{n}$. It follows that there exists

$a=\lim_{n\rightarrow+\infty}\ a_{n}\geq a_{n},\ \forall n.$

For the sake of contradiction, assume $a<+\infty$. Since $K$ is unstable,
there exists $v\in{\mathcal{B}}_{\infty}$ such that $\|L_{K}[v]\|_{1}\geq a+1$
(possibly $\|L_{K}[v]\|_{1}=+\infty$). Now consider the restrictions $v_{n}$
of $v$ over $[0,n]$ with null values outside this interval. Let us introduce
the sets

${\mathcal{R}}_{n}(p):=\left\\{(x,y):0\leq x\leq n+p,\ 0\leq y\leq n\right\\}$

and

${\mathcal{R}}_{n}:=\\{\ (x,y):0\leq x<+\infty,\ 0\leq y\leq n\ \\}$

with $K_{n}(p)$ and $K_{n}(+\infty)$ to denote the corresponding “kernels” (we
call these objects kernels improperly since they are not even symmetric). One
has

$L_{K_{n}(p)}[v_{n}]=L_{K_{n+p}}[v_{n}]\ \Rightarrow\
\|L_{K_{n}(p)}[v_{n}]\|_{1}\leq a_{n+p}\leq a,\ \forall p$

where the equality holds since $v_{n}(t)=0$ for $t\in(n,n+p]$ while the
inequality follows from the fact that all the $a_{n}$ are upper bounded by
$a$. Hence,

$\|L_{K_{n}(p)}[v_{n}]\|_{1}=\int_{0}^{n+p}\ |y_{n}(t)|dt\leq a,\ \forall p\
\Rightarrow\ \int_{0}^{+\infty}\ |y_{n}(t)|dt\leq a$ $\Rightarrow\
\|L_{K_{n}(+\infty)}[v]\|_{1}=\|L_{K_{n}(+\infty)}[v_{n}]\|_{1}\leq a$

(note that $L_{K_{n}(+\infty)}[v]=L_{K_{n}(+\infty)}[v_{n}]$ since the support
of $K_{n}$ does not extend beyond $n$). Now, for any $t$ such that the kernel
section is absolutely integrable we define

$y(t):=L_{K}[v](t)=\int_{0}^{+\infty}\ K(t,\tau)v(\tau)d\tau,$
$y_{n}(t):=L_{K_{n}(+\infty)}[v](t)=\int_{0}^{n}\ K(t,\tau)v(\tau)d\tau.$

Then, the term

$|y(t)-y_{n}(t)|\leq\int_{n}^{+\infty}\ |K(t,\tau)|d\tau$

is infinitesimal as $n$ grows to infinity, so that $[L_{K_{n}(+\infty)}[v](t)$
pointwise converges to $y(t)$ a.e. (in fact the kernel sections are non
absolutely summable only if $t$ belongs to ${\mathcal{I}}$ whose Lebesgue
measure has been assumed null). Summarizing, one has

$\|y_{k}(t)\|_{1}\leq a,\ \|y(t)\|_{1}\geq a+1,\ y_{k}(t)\rightarrow y(t)\
\mbox{(pointwise a.e.)}.$

But if $y_{k}(t)$ converges pointwise a.e. to $y(t)$, a fortiori one has

$y(t)=\lim_{k\rightarrow+\infty}\ \inf_{m\geq k}\
y_{m}(t)=\lim_{k\rightarrow+\infty}\ y_{k}(t)\ \mbox{a.e.}$

and, using the Fatou’s Lemma [24], one obtains

$\|y\|_{1}=\int_{0}^{+\infty}\ |y(t)|dt\leq\lim_{k\rightarrow+\infty}\
\inf_{m\geq k}\ \int_{0}^{+\infty}\ |y_{m}(t)|dt=$
$=\lim_{k\rightarrow+\infty}\ \inf_{m\geq k}\ \|y_{m}(t)\|_{1}\leq a.$

This shows that $\|y\|_{1}\leq a$ and $\|y\|_{1}\geq a+1$, leading to the
desired contradiction if $a<+\infty$. Hence, one must have $a=+\infty$ and
this completes the proof.
∎

We now introduce the second lemma useful for our purposes. It provides an
extension of the convexity arguments contained in Lemma 4 to handle functions
not necessarily in ${\mathcal{L}}_{1}$.

######  Lemma 10.

Consider the (possibly not absolutely integrable) functions $y(t),w(t)$ and
define

$F(x):=\int_{0}^{+\infty}\ |y(t)+xw(t)|dt,\ x\in[-x_{1},x_{2}],\
x_{1},x_{2}>0.$

Then, at least one of $x_{1},x_{2}$ satisfies the inequality $F(x_{i})\geq
F(0)$.

###### Proof.

The possible situations are

  * •

$w(t)\in{\mathcal{L}}_{1},\ y(t)\in{\mathcal{L}}_{1}$. In this case the result
follows from Lemma 4 and basic properties of convex functions.

  * •

$w(t)\in{\mathcal{L}}_{1},\ y(t)\notin{\mathcal{L}}_{1}$. In this case, one
has

$\displaystyle F(x):=\int_{0}^{+\infty}\ |y(t)+xw(t)|dt$
$\displaystyle\qquad\geq\int_{0}^{+\infty}\ |y(t)|dt-|x|\int_{0}^{+\infty}\
|w(t)|dt=+\infty,\ \forall x\in{\mathbb{R}}$

and the result immediately follows.

  * •

$w(t)\notin{\mathcal{L}}_{1},\ y(t)\in{\mathcal{L}}_{1}$. In this case,

$\displaystyle F(x):=\int_{0}^{+\infty}\ |y(t)+xw(t)|dt$
$\displaystyle\qquad\geq|x|\int_{0}^{+\infty}\ |w(t)|dt-\int_{0}^{+\infty}\
|y(t)|dt=+\infty,\ \forall x\in{\mathbb{R}}/\\{\ 0\\}$

so that both $F(x_{1})$ and $F(x_{2})$ are larger than $F(0)$.

  * •

$w(t)\notin{\mathcal{L}}_{1},\ y(t)\notin{\mathcal{L}}_{1}$ and, in addition,
there exists $x_{0}\in{\mathbb{R}}_{+}$ such that
$y(t)+x_{0}w(t)\in{\mathcal{L}}_{1}$. Now

$\displaystyle F(x):=\int_{0}^{+\infty}\ |y(t)+xw(t)|dt$
$\displaystyle\qquad=\int_{0}^{+\infty}\ |[y(t)+x_{0}w(t)]+(x-x_{0})w(t)|dt$

so that we can consider the second case just replacing $y(t)$ with
$y(t)+x_{0}w(t)$ and $xw(t)$ with $(x-x_{0})w(t)$. So, $F(x)$ is always equal
to $+\infty$ except when $x=x_{0}$. So, $F(x_{1})\geq F(0)$ and/or
$F(x_{2})\geq F(0)$.

  * •

$w(t)\notin{\mathcal{L}}_{1},\ y(t)\notin{\mathcal{L}}_{1}$ and, in addition,
there does not exist $x_{0}\in{\mathbb{R}}_{+}$ such that
$y(t)+x_{0}w(t)\in{\mathcal{L}}_{1}$. In this case, $F(x)$ is always equal to
$+\infty$.

Summarizing, we have seen that, except in the first case already treated in
Lemma 4, $F(x)$ is always equal to $+\infty$ except at most when $x=x_{0}$.
So, when $x=x_{1}$ and/or $x=x_{2}$ it assumes an infinite value. This
completes the proof.
∎

We are now in a position to conclude the proof. Exploiting Lemma 9, in
particular (10) and (11), we can find a sequence
$w_{n}\in{\mathcal{B}}_{\infty}$ with bounded support over $[0,n]$ satisfying

$\|y_{n}\|_{1}:=\|L_{K}[w_{n}]\|_{1}\geq a_{n}-\epsilon,\ |w_{n}(t)|=1,\
\forall t:\ 0\leq t\leq n.$

In fact, given any $\epsilon>0$, the 1-norm just of $y_{n}(t)$ restricted to
$[0,n]$ would not be smaller than $a_{n}-\epsilon$ in view of the definition
of supremum applied to the kernel operator associated with $K_{n}$. Thus, one
has

$\lim_{n\rightarrow+\infty}\|L_{K}[w_{n}]\|_{1}=+\infty.$

Now, we extend $w_{n}$ for $t>n$ as follows

$v_{n}(x):=w_{n}+x\psi_{(n,+\infty)}$

and note that $v_{n}(x)\in{\mathcal{B}}_{\infty}$ for any $x\in[-1,1]$ and
$v_{n}(x)\in{\mathcal{U}}_{\infty}$ for $x=\pm 1$. Now, we can exploit Lemma
10, interpreting $v_{n}(x)$ as $y(t)+xw(t)$. One obtains

$\|L_{K}[v_{n}](x_{n})\|_{1}\geq\|L_{K}[w_{n}]\|_{1}\ \ \text{if either}\ \
x_{n}=+1\ \ \text{or}\ \ x_{n}=-1.$

Then, choosing

$s_{n}:=v_{n}(x_{n}),$

with $x_{n}$ depending on $n$ and chosen in order to satisfy the above
inequality for any $n$, one has

$\|L_{K}[s_{n}]\|_{1}\geq a_{n}-\epsilon.$

This permits to conclude that

$\lim_{n\rightarrow+\infty}\|L_{K}[s_{n}]\|_{1}=+\infty\ \ \text{with all
the}\ \ s_{n}\in{\mathcal{U}}_{\infty},$

hence completing the proof (note that $\|L_{K}[s_{n}]\|_{1}$ could be already
equal to $+\infty$, making not even necessary the building of the entire
sequence).

##  3 The discrete-time case

In discrete-time, the kernel $K$ can be interpreted as an infinite-dimensional
matrix. Using notation of ordinary algebra also to handle objects of infinite
dimension, the kernel operator $L_{K}[u]$ becomes $Ku$ where $u$ is an
infinite-dimensional column vector. It is then immediate to see that all the
arguments developed in the previous section to prove Theorem 2 still hold. One
has just to consider the corresponding discrete-time versions of
$\mathcal{L}_{1},\mathcal{L}_{\infty},\mathcal{B}_{\infty}$ and
${\mathcal{U}}_{\infty}$. Furthermore, when discussing the unstable case, the
space of functions ${\mathcal{L}}$ with the Lebesgue measure is replaced by
the measure space containing all the sequences over $\mathbb{N}$ equipped with
the counting measure. Then, all the arguments contained in the proof hold true
with integrals that become sums. One concludes that even in discrete-time the
sup of the kernel operator norm computed over ${\mathcal{B}}_{\infty}$
coincides with that over ${\mathcal{U}}_{\infty}$, i.e.

$\sup_{u\in{\mathcal{B}}_{\infty}}\|Ku\|_{1}=\sup_{u\in{\mathcal{U}}_{\infty}}\|Ku\|_{1}.$

##  4 Conclusions

The result obtained in this paper gives additional insights on the
relationship between stability of a single linear dynamic systems and
stability of an ensemble of systems embedded in a positive definite kernel.
Stability of a RKHS containing time-invariant linear systems can now be
assessed using the same functions needed to investigate BIBO stability of any
single element in the RKHS. We envision that the availability of this new
stability test (over a smaller subset of test functions) could facilitate the
development of new theory regarding the use of RKHSs in system identification
and control theory.

## References

  * [1] N. Aronszajn. Theory of reproducing kernels. Trans. of the American Mathematical Society, 68:337–404, 1950.
  * [2] K.J. Åström and P. Eykhoff. System identification-a survey. Automatica, 7(2):123–162, 1971.
  * [3] B.M. Bell and G. Pillonetto Estimating parameters and stochastic functions of one variable using nonlinear measurement models. Inverse Problems 20 (3), 2004.
  * [4] S. Bergman. The Kernel Function and Conformal Mapping. Mathematical Surveys and Monographs, AMS, 1950.
  * [5] M. Bertero, T. Poggio, and V. Torre. Ill-posed problems in early vision. Proceedings of IEEE, 1988.
  * [6] M. Bisiacco and G. Pillonetto. Kernel absolute summability is sufficient but not necessary for RKHS stability. SIAM journal on control and optimization, 2020.
  * [7] M. Bisiacco and G. Pillonetto. On the mathematical foundations of stable RKHSs. Automatica, 2020.
  * [8] G. Bottegal, H. Hjalmarsson and G. Pillonetto. A new kernel-based approach to system identification with quantized output data. Automatica, 85, 145-15, 2017.
  * [9] C. Carmeli, E. De Vito, and A. Toigo. Vector valued reproducing kernel Hilbert spaces of integrable functions and Mercer theorem. Analysis and Applications, 4:377–408, 2006.
  * [10] T. Chen, H. Ohlsson, and L. Ljung. On the estimation of transfer functions, regularizations and Gaussian processes - Revisited. Automatica, 48:1525–1535, 2012.
  * [11] T. Chen and G. Pillonetto. On the stability of reproducing kernel Hilbert spaces of discrete-time impulse responses. Automatica, 2018.
  * [12] F. Dinuzzo. Kernels for linear time invariant system identification. SIAM Journal on Control and Optimization, 53(5):3299–3317, 2015\.
  * [13] F. Girosi. An equivalence between sparse approximation and support vector machines. Neural Comput., 10(6):1455–1480, August 1998.
  * [14] T. Kailath. Linear systems. Prentice-Hall, 1979.
  * [15] M. Kruzik. Bauer’s maximum principle and hulls of sets. Calculus of Variations and Partial Differential Equations, 11:321–332, 2000.
  * [16] L. Ljung. System Identification - Theory for the User. Prentice-Hall, Upper Saddle River, N.J., 2nd edition, 1999.
  * [17] G. Pillonetto, T. Chen, A. Chiuso, G. De Nicolao, and L. Ljung. Regularized System Identification. Springer, 2022.
  * [18] G. Pillonetto and L. Ljung. Full Bayesian identification of linear dynamic systems using stable kernels. Proceedings of the National Academy of Sciences 120 (18), 2023.
  * [19] G. Pillonetto, A. Chiuso, and G. De Nicolao. Prediction error identification of linear systems: a nonparametric Gaussian regression approach. Automatica, 47(2):291–305, 2011.
  * [20] G. Pillonetto and G. De Nicolao. A new kernel-based approach for linear system identification. Automatica, 46(1):81–93, 2010.
  * [21] G. Pillonetto, F. Dinuzzo, T. Chen, G. De Nicolao, and L. Ljung. Kernel methods in system identification, machine learning and function estimation: a survey. Automatica, 50(3):657–682, 2014.
  * [22] T. Poggio and F. Girosi. Networks for approximation and learning. In Proceedings of the IEEE, volume 78, pages 1481–1497, 1990\.
  * [23] R.T. Rockafellar. Convex Analysis. Princeton Landmarks in Mathematics. Princeton University Press, 1970.
  * [24] W. Rudin. Real and Complex Analysis. McGraw-Hill, Singapore, 1987.
  * [25] B. Schölkopf and A. J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. (Adaptive Computation and Machine Learning). MIT Press, 2001.
  * [26] T. Söderström and P. Stoica. System Identification. Prentice-Hall, 1989.
  * [27] J.A.K. Suykens, T. Van Gestel, J. De Brabanter, B. De Moor, and J. Vandewalle. Least Squares Support Vector Machines. World Scientific, Singapore, 2002.
  * [28] A. N. Tikhonov and V. Y. Arsenin. Solutions of Ill-posed Problems. Winston/Wiley, Washington, D.C., 1977.
  * [29] A.N. Tikhonov. On the solution of incorrectly formulated problems and the regularization method. Doklady Akademii Nauk SSSR, 151:501–504, 1963.
  * [30] V. Vapnik. Statistical Learning Theory. Wiley, New York, NY, USA, 1998.
  * [31] G. Wahba. Spline models for observational data. SIAM, Philadelphia, 1990.
  * [32] L.A. Zadeh. On the identification problem. IRE Transactions on Circuits Theory, 3(4):277–281, 1956.