# Kernel-Based Identification with Frequency Domain Side-Information

Mohammad Khosravi  Roy S. Smith

###### Abstract

This paper discusses the problem of system identification when frequency
domain side-information is available. Initially, we consider the case where
the side-information is provided as the ${\mathcal{H}}_{\infty}$-norm of the
system being bounded by a given scalar. This framework allows considering
different forms of frequency domain side-information, such as the
dissipativity of the system. We propose a nonparametric identification
approach for estimating the impulse response of the system under the given
side-information. The estimation problem is formulated as a constrained
optimization in a stable reproducing kernel Hilbert space, where suitable
constraints are considered for incorporating the desired frequency domain
features. The resulting optimization has an infinite-dimensional feasible set
with an infinite number of constraints. We show that this problem is a well-
defined convex program with a unique solution. We propose a heuristic that
tightly approximates this unique solution. The proposed approach is equivalent
to solving a finite-dimensional convex quadratically constrained quadratic
program. The efficiency of the discussed method is verified by several
numerical examples.

##  I Introduction

System identification problem, initially introduced in [1], deals with the
theory and techniques for estimating suitable mathematical models describing
dynamical systems using measurement data. The topic has received substantial
attention due to its broad applicability in numerous phenomena in science and
technology [2, 3, 4]. In many situations, identifying a dynamical system goes
beyond fitting a mathematical model to the input-output measurement data. We
may additionally need to integrate a specific attribute or a known feature of
the system into the model. This side-information is possibly provided from our
general understanding of the behavior of the system based on its inherent
physical nature, or acquired from qualitative characteristics and phenomena
observed from historical or experimental data. For example, in the
identification of nonlinear dynamics various forms of side-information such as
stability, region of attraction, dissipativity and many others are
incorporated [5, 6, 7, 8, 9].

The incorporation of side-information has been considered in the
identification of linear dynamics, e.g., the low complexity of the model is
imposed by considering sparsity promoting regularizations [10, 11, 12, 13,
14]. For penalizing the order of systems, the rank and the nuclear norm of the
corresponding Hankel matrix are utilized in [14, 13, 15, 16]. For the same
purpose, the notion of atomic transfer functions and regularization based on
the atomic norm are employed [12, 10]. Identification with the side-
information involving positivity features of the system such as compartmental
structure and the internal or external positivity are also discussed in the
literature [17, 18, 19, 20]. Other forms of side-information are studied in
[21, 22, 23, 24, 25, 26], e.g., including information on the location of the
eigenvalues [23, 24], the moments of transfer function [25], and the positive-
realness [21, 22]. The subspace identification method is employed to include
information on the steady-state behavior [27, 28, 29] such as the stability of
the system [29].

Starting from the seminal work Pillonetto and De Nicolao [30], a paradigm
shift known as the _kernel-based_ approach has emerged in system
identification which allows integrating side-information [31, 32, 33, 34]. In
this framework, the identification problem is formulated as a regularized
regression in a reproducing kernel Hilbert space (RKHS) [35] where the
regularization term, based on the norm of RKHS, penalizes solutions not
compatible with the side-information. By a suitable choice of the kernel
function, or by imposing appropriate constraints in the regression problem,
one can incorporate a variety of side-information such as stability, resonant
frequencies, smoothness of the impulse response, steady-state gain, and,
internal or external positivity of the system [36, 37, 38, 39, 40, 41].
Moreover, employing a Tikhonov-like regularization in this framework leads to
an improvement in the bias-variance trade-off [32].

Input-output behavioral properties such as the the
${\mathcal{H}}_{\infty}$-norm of the plant, or more generally frequency domain
properties like the dissipativity of the system [42], can be considerably
useful in feedback controller design [43, 44]. These features are a priori
known for many systems due to their inherent physical nature, e.g., the
electrical circuits where the energy is dissipated by the resistors [42]. In
addition, they can be verified using recently developed data-driven methods
[45, 46, 47, 48, 49]. Information about these attributes is potentially
available for later use as side-information. However, existing research on LTI
system identification with side-information [26] does not adequately address
the inclusion of this frequency domain side-information.

The main goals of this paper are to develop identification methods utilizing
frequency domain side-information in a numerically tractable fashion and to
provide suitable theoretical guarantees. First, we consider the case where the
side-information is the ${\mathcal{H}}_{\infty}$-norm of the system being
bounded by a given scalar. The framework considers various forms of frequency
domain side-information, such as the dissipativity of the system. As kernel-
based approaches provide a powerful framework, the identification problem is
formulated as a constrained regularized regression problem in a stable RKHS
[32, 50] with suitable constraints to impose the frequency domain side-
information on the estimation. The optimization problem is infinite-
dimensional with an infinite number of constraints. We show that this problem
is a convex program with a unique solution. Towards deriving a tractable
solution scheme, we consider a suitable finite set of frequencies, and
subsequently, an approximate estimation problem is formulated as an
optimization problem with the same objective function but with constraints
imposed according to these frequencies. The new problem attains a unique
solution with a specific parametric form. Subsequently, an equivalent finite-
dimensional convex optimization is derived as a convex quadratically
constrained quadratic program (QCQP). By solving this optimization problem,
the solution of the approximate problem is obtained. We derive proper bounds
on the tightness of this approximation and provide theoretical guarantees on
the convergence of the approximate solution to the solution of the original
problem. Several numerical examples verify the efficiency of the proposed
method.

##  II Notations and Preliminaries

The set of natural numbers, the set of non-negative integers, the set of real
numbers, the set of non-negative real numbers, the set of complex numbers, the
$n$-dimensional Euclidean space and the space of $n$ by $m$ real matrices are
denoted by ${\mathbb{N}}$, ${\mathbb{Z}}_{+}$, ${\mathbb{R}}$,
${\mathbb{R}}_{+}$, ${\mathbb{C}}$, ${\mathbb{R}}^{n}$ and
${\mathbb{R}}^{n\times m}$ respectively. For any $z\in{\mathbb{C}}$, the real
and imaginary part of $z$ are denoted by $\mathrm{real}(()z)$ and
$\mathrm{imag}(()z)$, respectively. The inner product and norm of Hilbert
space ${\mathscr{H}}$ is denoted by
${\langle{\cdot,\cdot}\rangle}_{{\mathscr{H}}}$ and
$\|\cdot\|_{{\mathscr{H}}}$, respectively, and when it is clear from the
context, we drop the subscript. To handle discrete and continuous time in the
same formulation, ${\mathbb{T}}$ denotes either ${\mathbb{Z}}_{+}$ or
${\mathbb{R}}_{+}$, and ${\mathbb{T}}_{\pm}$ is the set of scalars $t$ where
either $t\in{\mathbb{T}}$ or $-t\in{\mathbb{T}}$. Given measure space
${\mathscr{X}}$, the space of measurable functions
$g:{\mathscr{X}}\to{\mathbb{R}}$ is denoted by ${\mathbb{R}}^{{\mathscr{X}}}$.
The element ${\mathrm{u}}\in{\mathbb{R}}^{{\mathscr{X}}}$ is shown entry-wise
as ${\mathrm{u}}=(u_{x})_{x\in{\mathscr{X}}}$, or equivalently as
${\mathrm{u}}=\big{(}u(x)\big{)}_{x\in{\mathscr{X}}}$. Depending on the
context of discussion, ${\mathscr{L}}^{\infty}$ refers either to
$\ell^{\infty}({\mathbb{Z}})$ or $L^{\infty}({\mathbb{R}})$. Similarly,
${\mathscr{L}}^{1}$ is either $\ell^{1}({\mathbb{Z}}_{+})$ or
$L^{1}({\mathbb{R}}_{+})$. For $p\in\\{1,\infty\\}$, the norm in
${\mathscr{L}}^{p}$ is denoted by $\|\cdot\|_{p}$. Let
$({\mathbb{X}},\|\cdot\|_{{\mathbb{X}}})$ and
$({\mathbb{Y}},\|\cdot\|_{{\mathbb{Y}}})$ be two normed vector spaces. The set
of linear bounded (continuous) operators
${\mathrm{A}}:{\mathbb{X}}\to{\mathbb{Y}}$, denoted by
${\mathcal{L}}({\mathbb{X}},{\mathbb{Y}})$, is a normed vector space with the
norm defined as
$\|{\mathrm{A}}\|_{{\mathcal{L}}({\mathbb{X}},{\mathbb{Y}})}:=\sup_{{\mathrm{x}}\in{\mathbb{X}},\|{\mathrm{x}}\|_{{\mathbb{X}}}\leq
1}\ \|{\mathrm{A}}{\mathrm{x}}\|_{{\mathbb{Y}}}$. The identity matrix/operator
and the zero vector are denoted by $\mathbb{I}$ and $\mathbf{0}$ respectively.
Given ${\mathscr{V}}\subseteq{\mathbb{X}}$, the linear span of
${\mathscr{V}}$, denoted by $\mathrm{span}{\mathscr{V}}$, is a linear subspace
of ${\mathbb{X}}$ containing linear combination of the elements of
${\mathscr{V}}$. Let ${\mathcal{Y}}$ be a set and
${\mathcal{C}}\subseteq{\mathcal{Y}}$. We define the function
$\delta_{{\mathcal{C}}}$ as $\delta_{{\mathcal{C}}}(y)=0$, if
$y\in{\mathcal{C}}$ and $\delta_{{\mathcal{C}}}(y)=\infty$, otherwise.
Similarly, function $\mathbf{1}_{{\mathcal{C}}}$ is defined as
$\mathbf{1}_{{\mathcal{C}}}(y)=1$, if $y\in{\mathcal{C}}$ and
$\mathbf{1}_{{\mathcal{C}}}(y)=0$, otherwise.

##  III System Identification with Frequency Domain Side-Information

Consider a stable LTI system ${\mathcal{S}}$ described with impulse response
${\mathrm{g}}^{{\scalebox{0.5}{$({\mathcal{S}})$}}}:=({g}^{{\scalebox{0.5}{$({\mathcal{S}})$}}}_{t})_{t\in{\mathbb{T}}}\in{\mathbb{R}}^{\mathbb{T}}$
and transfer function $G^{{\scalebox{0.5}{$({\mathcal{S}})$}}}$. For the case
of discrete-time and the case of continuous-time, we have here respectively
${\mathbb{T}}:={\mathbb{Z}}_{+}$ and ${\mathbb{T}}:={\mathbb{R}}_{+}$. Let the
system be actuated with a bounded input signal denoted by
${\mathrm{u}}=(u_{t})_{t\in{\mathbb{T}}}\in{\mathscr{L}}^{\infty}$.
Accordingly, for any $t\in{\mathbb{T}}_{\pm}$, one can define linear map
${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}$ over the space of stable causal impulse
responses, for the discrete-time case, as

${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}}):=\sum_{s\in{\mathbb{Z}}_{+}}g_{s}u_{t-s},$
(1)

and similarly, for the case of continuous-time, as

${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}}):=\int_{{\mathbb{R}}_{+}}\\!\\!g_{s}u_{t-s}\
\\!\mathrm{d}s.$ (2)

Let the output of the system be measured at time instants
${\mathscr{T}}:=\\{t_{i}\ \\!|\\!\
i=0,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}\\!-\\!1\\}$, for a given
$n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}\in{\mathbb{N}}$. More precisely,
define $y_{t}$ as

$y_{t}:={\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}}^{{\scalebox{0.5}{$({\mathcal{S}})$}}})+w_{t},\qquad
t\in{\mathscr{T}},$ (3)

where, for any $t\in{\mathscr{T}}$, $w_{t}$ denotes the measurement
uncertainty. Subsequently, let ${\mathscr{D}}$ be the set of input-output
pairs, i.e., ${\mathscr{D}}$ is defined as ${\mathscr{D}}=\\{(u_{t},y_{t})\ |\
t\in{\mathscr{T}}\\}$.

Let assume we know that the ${\mathcal{H}}_{\infty}$-norm of system
${\mathcal{S}}$ is bounded by a given scalar $\rho\in{\mathbb{R}}_{+}$. The
question is whether this side-information is naturally encoded in the
identification problem. The following example elaborates this issue.

######  Example.

Let ${\mathcal{S}}$ be a discrete-time system described by the following
transfer function

$G^{{\scalebox{0.5}{$({\mathcal{S}})$}}}(z)=\frac{1}{2z-1}\ +\ \frac{3}{100}\
\frac{z-1}{z^{2}+z+0.9}.$ (4)

To obtain set of data ${\mathscr{D}}$, we actuate the system with a random
white Gaussian signal of length
$n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}=150$, and then, the output of system
is measured where SNR $=14.5\\!\ $dB (see Figure 1). Additionally, assume we
are given the side-information
$\|G^{{\scalebox{0.5}{$({\mathcal{S}})$}}}\|_{{\mathcal{H}}_{\infty}}\leq 1$.
We employ Matlab’s System Identification Toolbox [51] to estimate models
$\hat{G}_{1}$ and $\hat{G}_{2}$ for the system using impulseest and n4sid,
respectively. The results are shown in Figure 1 where we have
$\|\hat{G}_{1}\|_{{\mathcal{H}}_{\infty}}=1.24$ and
$\|\hat{G}_{2}\|_{{\mathcal{H}}_{\infty}}=1.38$. One can see that the
estimated models do not comply the side-information.

Figure 1: The estimated models $\hat{G}_{1}$ and $\hat{G}_{2}$ for the system
(4) are compared with the true transfer function.

Motivated by this example, we introduce the main problem discussed in this
paper as system identification with side-information on the
${\mathcal{H}}_{\infty}$-norm of the system.

######  Problem 1.

Given the set of input-output data ${\mathscr{D}}$, estimate the impulse
response of system ${\mathcal{S}}$ satisfying the side-information
$\|G^{{\scalebox{0.5}{$({\mathcal{S}})$}}}\|_{{\mathcal{H}}_{\infty}}\leq\rho$,
where $\rho$ is a given non-negative real scalar.

This problem can be extended to the identification problem with more general
characterization of dissipativity side-information. More precisely, let
${\mathrm{Q}}\in{\mathbb{S}}^{2}$ be an indefinite matrix and assume that in
addition to the given set of data ${\mathscr{D}}$, we know that the system
${\mathcal{S}}$ is _dissipative_ [52] with respect to quadratic supply rate
function $s_{{\mathrm{Q}}}(u,y)$ defined as

$\begin{split}s_{{\mathrm{Q}}}(u,y)&:=\begin{bmatrix}u&y\end{bmatrix}{\mathrm{Q}}\begin{bmatrix}u\\\
y\end{bmatrix}=\begin{bmatrix}u&y\end{bmatrix}\begin{bmatrix}q_{u}&q_{uy}\\\
q_{uy}&q_{y}\end{bmatrix}\begin{bmatrix}u\\\ y\end{bmatrix}\\\
&=q_{u}u^{2}+2q_{uy}uy+q_{y}y^{2},\end{split}$ (5)

where $q_{y}<0$. For the case of $s_{{\mathrm{Q}}}(u,y)=\rho^{2}u^{2}-y^{2}$
where $\rho\in{\mathbb{R}}_{+}$, this dissipativity prior knowledge is
equivalent to being ${\mathscr{L}}_{2}$-gain or ${\mathcal{H}}_{\infty}$-norm
of the system not larger than $\rho$. Given the above side-information on the
dissipativity of system ${\mathcal{S}}$, a desired identification procedure
for estimating the impulse response of the system should suitably utilize this
information and also guarantee that the identified model satisfies the given
feature. The more general problem is the following.

######  Problem 2.

Given the set of data ${\mathscr{D}}$ and considering the side-information on
the dissipativity of system with respect to the supply rate
$s_{{\mathrm{Q}}}(u,y)$, estimate the impulse response of system
${\mathcal{S}}$ satisfying the dissipativity side-information.

Note that we have

${\mathrm{Q}}=\begin{bmatrix}q_{u}&q_{uy}\\\
q_{uy}&q_{y}\end{bmatrix}=\begin{bmatrix}l_{1}&l_{2}\\\
0&l_{3}\end{bmatrix}\begin{bmatrix}1&0\\\
0&-1\end{bmatrix}\begin{bmatrix}l_{1}&0\\\ l_{2}&l_{3}\end{bmatrix}$ (6)

where $l_{1}=(\det{\mathrm{Q}}/q_{y})^{\frac{1}{2}}$,
$l_{2}=-q_{uy}/(-q_{y})^{\frac{1}{2}}$, and $l_{3}=(-q_{y})^{\frac{1}{2}}$,
and subsequently, one can define the system $\tilde{{\mathcal{S}}}$ with input
$v=l_{1}u$ and output $z=l_{2}u+l_{3}y$ as shown in Figure 2. Then,
$\tilde{{\mathcal{S}}}$ is a dissipative system with respect to supply rate
function $s(v,z)=u^{2}-y^{2}$, or equivalently,
$\|G^{{\scalebox{0.5}{$(\tilde{{\mathcal{S}}})$}}}\|_{{\mathcal{H}}_{\infty}}\leq
1$ where $G^{{\scalebox{0.5}{$(\tilde{{\mathcal{S}}})$}}}$ is the transfer
function of system $\tilde{{\mathcal{S}}}$. Meanwhile, using the introduced
change of variables and based on ${\mathscr{D}}$, we can define set of data
${\mathscr{D}}_{\tilde{{\mathcal{S}}}}:=\\{(v_{t},z_{t})\ |\
t\in{\mathscr{T}}\\},$ where $v_{t}:=l_{1}u_{t}$ and
$z_{t}:=l_{2}u_{t}+l_{3}y_{t}$, for $t\in{\mathscr{T}}$. Accordingly, the
problem is equivalent to identifying system $\tilde{{\mathcal{S}}}$ given the
set of data ${\mathscr{D}}_{\tilde{{\mathcal{S}}}}$ and the side-information
$\|G^{{\scalebox{0.5}{$(\tilde{{\mathcal{S}}})$}}}\|_{{\mathcal{H}}_{\infty}}\leq
1$. Therefore, in order to address Problem 2 it suffices to find a solution
approach for Problem 1.

Figure 2: The equivalent system $\tilde{{\mathcal{S}}}$ with
${\mathcal{H}}_{\infty}$-norm less than or equal to $1$.

Problem 1 can be further extended to the case where system ${\mathcal{S}}$ is
approximately known. More precisely, let $\overline{{\mathcal{S}}}$ be a known
system with transfer function
$G^{{\scalebox{0.5}{$(\overline{{\mathcal{S}}})$}}}$ and we are given that
$\|G^{{\scalebox{0.5}{$({\mathcal{S}})$}}}-G^{{\scalebox{0.5}{$(\overline{{\mathcal{S}}})$}}}\|_{{\mathcal{H}}_{\infty}}\leq\rho$,
for a known $\rho\in{\mathbb{R}}_{+}$. This can be the case where the system
might have been previously modeled or identified as
$\overline{{\mathcal{S}}}$, and we would now like to identify an improved
version of the model. The following identification problem addresses this
case.

######  Problem 3.

Given the set of input-output data ${\mathscr{D}}$, estimate the impulse
response of system ${\mathcal{S}}$ satisfying the side-information
$\|G^{{\scalebox{0.5}{$({\mathcal{S}})$}}}-G^{{\scalebox{0.5}{$(\overline{{\mathcal{S}}})$}}}\|_{{\mathcal{H}}_{\infty}}\leq\rho$,
where $\overline{{\mathcal{S}}}$ is a known system with transfer function
$G^{{\scalebox{0.5}{$(\overline{{\mathcal{S}}})$}}}$ and
$\rho\in{\mathbb{R}}_{+}$ is a given scalar.

Let $\Delta{\mathcal{S}}$ be the system with the transfer function
$G^{{\scalebox{0.5}{$(\Delta{\mathcal{S}})$}}}:=G^{{\scalebox{0.5}{$({\mathcal{S}})$}}}-G^{{\scalebox{0.5}{$(\overline{{\mathcal{S}}})$}}}$
and the impulse response
${\mathrm{g}}^{{\scalebox{0.5}{$(\Delta{\mathcal{S}})$}}}:={\mathrm{g}}^{{\scalebox{0.5}{$({\mathcal{S}})$}}}-{\mathrm{g}}^{{\scalebox{0.5}{$(\overline{{\mathcal{S}}})$}}}$,
where ${\mathrm{g}}^{{\scalebox{0.5}{$(\overline{{\mathcal{S}}})$}}}$ is the
impulse response of system $\bar{{\mathcal{S}}}$. Since
${\mathrm{g}}^{{\scalebox{0.5}{$(\overline{{\mathcal{S}}})$}}}$ is known, one
can obtain
${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}}^{{\scalebox{0.5}{$(\overline{{\mathcal{S}}})$}}})$,
and subsequently, define $d_{t}$ as
$d_{t}:=y_{t}-{\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}}^{{\scalebox{0.5}{$(\overline{{\mathcal{S}}})$}}})$,
for any $t\in{\mathscr{T}}$. Due to (3) and the linearity of
${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}$, we have

$d_{t}={\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}}^{{\scalebox{0.5}{$({\mathcal{S}})$}}})+w_{t}-{\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}}^{{\scalebox{0.5}{$(\overline{{\mathcal{S}}})$}}})={\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}}^{{\scalebox{0.5}{$(\Delta{\mathcal{S}})$}}})+w_{t},\quad
t\in{\mathscr{T}}.$ (7)

Accordingly, we can define
${\mathscr{D}}_{\Delta{\mathcal{S}}}:=\\{(u_{t},d_{t})\ |\
t\in{\mathscr{T}}\\}$ as the input-output pairs of data for the system
$\Delta{\mathcal{S}}$. Therefore, the Problem 3 is equivalent to identifying
system $\Delta{\mathcal{S}}$ given the set of data
${\mathscr{D}}_{\Delta{\mathcal{S}}}$ and the side-information
$\|G^{{\scalebox{0.5}{$(\Delta{\mathcal{S}})$}}}\|_{{\mathcal{H}}_{\infty}}\leq\rho$,
which is in form of Problem 1.

Let ${\mathbb{T}}={\mathbb{Z}}_{+}$,
${\mathscr{T}}:=\\{i=0,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}\\!-\\!1\\}$,
and consider the following identification problem:

######  Problem 4.

Given the set of input-output data ${\mathscr{D}}$, estimate the impulse
response of system ${\mathcal{S}}$ satisfying the frequency domain side-
information
$\|WG^{{\scalebox{0.5}{$({\mathcal{S}})$}}}\|_{{\mathcal{H}}_{\infty}}\leq\rho$,
where $\rho\in{\mathbb{R}}_{+}$ is a given scalar, and weight $W$ is a known
stable transfer function with stable casual inverse.

Let the output signal
${\mathrm{y}}=(y_{t})_{t=0}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1}$ be
filtered by system $W$ and
${\mathrm{p}}:=(p_{t})_{t=0}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1}$ be
the resulting filtered signal. One can see that
${\mathscr{D}}_{H}:=\\{(u_{t},p_{t})|t\in{\mathscr{T}}\\}$ is a set of input-
output data for the system with transfer function
$H:=WG^{{\scalebox{0.5}{$({\mathcal{S}})$}}}$, where we know that
$\|H\|_{{\mathcal{H}}_{\infty}}\leq\rho$. Let $\hat{H}$ be the solution of
Problem 1 for this setting. Then, $\hat{G}:=W^{-1}\hat{H}$ is a solution to
Problem 4.

Considering the above discussion, a solution approach for Problem 1 also
addresses Problems 2, 3 and 4. In the remainder of the paper, we discuss
solving Problem 1.

##  IV The Estimation Problem: Existence and Uniqueness of the Solution

We investigate the existence and uniqueness properties for the solution of
Problem 1.

Let ${\mathscr{F}}\subseteq{\mathbb{R}}^{\mathbb{T}}$ be a suitable functional
space of stable impulse responses taken as the hypothesis set for the
estimation Problem 1. Given bounded signal
${\mathrm{u}}\in{\mathbb{R}}^{\mathbb{T}}$, with respect to each
$t\in{\mathbb{T}}$, we have the linear map
${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}:{\mathscr{F}}\to{\mathbb{R}}$ as defined
in (1) and (2). Based on this definition and the set of data ${\mathscr{D}}$,
one can define the empirical loss function or the fitting error function,
${\mathcal{E}}_{{\mathscr{D}}}:{\mathscr{F}}\to{\mathbb{R}}$, as the sum of
squared error. In other words, for a given candidate impulse response
${\mathrm{g}}\in{\mathscr{F}}$, we have that

${\mathcal{E}}_{{\mathscr{D}}}({\mathrm{g}}):=\operatornamewithlimits{\sum}\limits_{t\in{\mathscr{T}}}({\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}})-y_{t})^{2}.$
(8)

Define ${\mathscr{G}}\subseteq{\mathscr{L}}^{1}$ as the set of impulse
responses corresponding to the systems with ${\mathcal{H}}_{\infty}$-norm less
than or equal to $\rho$. More precisely, we have

$\\!\\!{\mathscr{G}}\\!:=\\!\bigg{\\{}{\mathrm{g}}\\!=\\!(g_{t})_{t\in{\mathbb{Z}}_{+}}\\!\in\\!{\mathscr{L}}^{1}\bigg{|}\operatornamewithlimits{sup}\limits_{\omega\in[0,\pi]}\Big{|}\sum_{t\in{\mathbb{Z}}_{+}}g_{t}\mathrm{e}^{-\mathrm{j}\omega
t}\Big{|}\\!\leq\\!\rho\bigg{\\}},$ (9)

and

${\mathscr{G}}\\!:=\\!\bigg{\\{}{\mathrm{g}}\\!=\\!(g_{t})_{t\in{\mathbb{R}}_{+}}\\!\in\\!{\mathscr{L}}^{1}\bigg{|}\operatornamewithlimits{sup}\limits_{\omega\in{\mathbb{R}}_{+}}\Big{|}\int_{{\mathbb{R}}_{+}}g_{t}\mathrm{e}^{-\mathrm{j}\omega
t}\mathrm{d}t\Big{|}\\!\leq\\!\rho\bigg{\\}},$ (10)

respectively for ${\mathbb{T}}={\mathbb{Z}}_{+}$ and
${\mathbb{T}}={\mathbb{R}}_{+}$. Note that since each element of
${\mathscr{G}}$ belongs to ${\mathscr{L}}^{1}$ the summation in (9) and the
integration in (10) are well-defined. Accordingly, in order to address Problem
1, we solve the following optimization problem

$\begin{array}[]{cl}\operatornamewithlimits{min}\limits_{{\mathrm{g}}\in{\mathscr{F}}}&{\mathcal{E}}_{{\mathscr{D}}}({\mathrm{g}})+\lambda{\mathcal{R}}({\mathrm{g}})\\\
\mathrm{s.t.}&{\mathrm{g}}\in{\mathscr{G}},\end{array}$ (11)

where ${\mathcal{R}}:{\mathscr{F}}\to{\mathbb{R}}_{+}$ is a suitable
regularization function and $\lambda>0$ is the regularization weight. Due to
the definition of ${\mathscr{G}}$, the optimization problem (11) is an
infinite-dimensional program with an uncountably infinite number of inequality
constraints, which is not tractable in the current form. Accordingly, we need
to address the following questions:

  1. 1.

_What is a suitable candidate for hypothesis set ${\mathscr{F}}$?_

  2. 2.

_Does optimization problem ( 11) admit a solution? Is this solution unique?_

  3. 3.

_How can we obtain the solution of ( 11) or a tight approximation for it?_

The first two questions are addressed throughout the remainder of this
section. The last question is postponed to Section V.

###  IV-A Stable Reproducing Kernel Hilbert Spaces

The hypothesis space taken for estimating the unknown impulse response is a
type of Hilbert spaces known as reproducing kernel Hilbert spaces (RKHS) which
are introduced briefly below (see [53, 35] for more details). The structure of
RKHS provides a suitable framework for investigating the problem and obtaining
a tractable scheme for solving (11).

######  Definition 1 ([35]).

Let ${\mathscr{H}}\subseteq{\mathbb{R}}^{{\mathbb{T}}}$ be a Hilbert space
endowed with inner product ${\langle{\cdot,\cdot}\rangle}_{{\mathscr{H}}}$ and
induced norm $\|\cdot\|_{{\mathscr{H}}}$. Then, ${\mathscr{H}}$ is a
reproducing kernel Hilbert space (RKHS) if for any $t\in{\mathbb{T}}$, we have
$\sup\big{\\{}|g_{t}|\ \big{|}\
{\mathrm{g}}:=(g_{t})_{t\in{\mathbb{T}}}\in{\mathscr{H}},\|{\mathrm{g}}\|_{{\mathscr{H}}}\leq
1\big{\\}}<\infty$.

Along with the RKHS, the Mercer kernel is introduced in the next definition.

######  Definition 2 ([35]).

The continuous function
$\mathbbm{k}:{\mathbb{T}}\times{\mathbb{T}}\to{\mathbb{R}}$ is a Mercer kernel
or positive-definite kernel when for any $m\in{\mathbb{N}}$,
$t,s,t_{1},\ldots,t_{m}\in{\mathbb{T}}$ and
$a_{1},\ldots,a_{m}\in{\mathbb{R}}$, we have
$\mathbbm{k}(t,s)=\mathbbm{k}(s,t)$ and $\sum_{1\leq i,j\leq
m}a_{i}a_{j}\mathbbm{k}(t_{i},t_{j})\geq 0$. For each $t\in{\mathbb{T}}$, the
section of kernel $\mathbbm{k}$ at $t$ is defined as the function
$\mathbbm{k}(t,\cdot):{\mathbb{T}}\to{\mathbb{R}}$ and denoted by
$\mathbbm{k}_{t}$.

The next theorem shows the connection between RKHSs and the Mercer kernels.

######  Theorem 1 ([35]).

Given a Mercer kernel
$\mathbbm{k}:{\mathbb{T}}\times{\mathbb{T}}\to{\mathbb{R}}$, there exists a
RKHS
${\mathcal{H}}_{\mathbbm{k}}\subseteq{\mathrm{g}}\in{\mathbb{R}}^{{\mathbb{T}}}$,
endowed with inner product
${\langle{\cdot,\cdot}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$ and norm
$\|\cdot\|_{{\mathcal{H}}_{\mathbbm{k}}}$, such that for any
$t\in{\mathbb{T}}$, we have $\mathbbm{k}_{t}\in{\mathcal{H}}_{\mathbbm{k}}$,
and
${\langle{{\mathrm{g}},\mathbbm{k}_{t}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}=g_{t}$,
for all
${\mathrm{g}}=(g_{t})_{t\in{\mathbb{T}}}\in{\mathcal{H}}_{\mathbbm{k}}$. The
second feature is called the reproducing property.

Based on Theorem 1, we know that a RKHS is uniquely characterized by a Mercer
kernel. Therefore, in the current context, the kernel $\mathbbm{k}$ is chosen
suitably such that each impulse response
${\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}$ represents a stable system in the
bounded-input-bounded-output (BIBO) sense. More precisely, we should have
${\mathcal{H}}_{\mathbbm{k}}\subseteq{\mathscr{L}}^{1}$. When kernel
$\mathbbm{k}$ satisfies this feature, it is called a stable kernel [50]. The
following theorem provides a necessary and sufficient condition for the
stability of a given kernel.

######  Theorem 2 ([50, 54]).

Let $\mathbbm{k}:{\mathbb{T}}\times{\mathbb{T}}\to{\mathbb{R}}$ be a Mercer
kernel. Then, $\mathbbm{k}$ is stable if and only if, when
${\mathbb{T}}={\mathbb{Z}}_{+}$, we have
$\sum_{t\in{\mathbb{Z}}_{+}}\big{|}\sum_{s\in{\mathbb{Z}}_{+}}u_{s}\mathbbm{k}(t,s)\big{|}<\infty$,
and, when ${\mathbb{T}}={\mathbb{R}}_{+}$, we have
$\int_{{\mathbb{R}}_{+}}\big{|}\int_{{\mathbb{R}}_{+}}u_{s}\mathbbm{k}(t,s)\mathrm{d}s\big{|}\mathrm{d}t<\infty$,
for any ${\mathrm{u}}=(u_{t})_{t\in{\mathbb{T}}}\in{\mathscr{L}}^{\infty}$.

The most common stable kernel in the literature [32] is the _tuned/correlated_
(TC) kernel defined as follows

$\mathbbm{k}(s,t)=\begin{cases}\alpha^{\max(s,t)}&\text{ if
${\mathbb{T}}={\mathbb{Z}}_{+}$,}\\\ \mathrm{e}^{-\beta\max(s,t)}&\text{ if
${\mathbb{T}}={\mathbb{R}}_{+}$,}\end{cases}$ (12)

where $\alpha\in[0,1)$ and $\beta>0$. By setting $\alpha$ as
$\mathrm{e}^{-\beta}$, one can obtain same definition for both cases in (12).
This is the default kernel employed in impulseest function of Matlab’s System
Identification Toolbox. One can easily see that if kernel $\mathds{h}$ is
_dominated_ by TC kernel $\mathbbm{k}$, i.e., there exists
$\gamma\in{\mathbb{R}}_{+}$ such that for any $s,t\in{\mathbb{T}}$ we have
$|\mathds{h}(s,t)|\leq\gamma\mathbbm{k}(s,t)$, then $\mathds{h}$ is a stable
kernel. This highlights the importance of TC kernels.

###  IV-B The Estimation Problem in Stable Reproducing Kernel Hilbert Spaces

Let $\mathbbm{k}$ be a stable kernel and ${\mathcal{H}}_{\mathbbm{k}}$ be the
corresponding RKHS. Motivated by [30, 32, 31] and the above discussion, we set
${\mathcal{H}}_{\mathbbm{k}}$ as the hypothesis space for the estimation
problem, i.e., ${\mathscr{F}}={\mathcal{H}}_{\mathbbm{k}}$. Subsequently, we
can introduce a suitable kernel-based regularization. More precisely, let the
regularization function
${\mathcal{R}}:{\mathcal{H}}_{\mathbbm{k}}\to{\mathbb{R}}_{+}$ be defined as
${\mathcal{R}}({\mathrm{g}}):=\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}$.
Therefore, in correspondence with the estimation Problem 1, we have the
following optimization problem:

$\begin{array}[]{cl}\operatornamewithlimits{min}\limits_{{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}}&\operatornamewithlimits{\sum}\limits_{t\in{\mathscr{T}}}({\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}})-y_{t})^{2}+\lambda\
\\!\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2},\\\
\mathrm{s.t.}&|G(\mathrm{j}\omega)|\leq\rho,\quad\forall\omega\in\Omega_{{\mathbb{T}}},\end{array}$
(13)

where $G$ denotes the transfer function corresponding to the impulse response
${\mathrm{g}}$, $\Omega_{{\mathbb{T}}}:=[0,\pi]$ when
${\mathbb{T}}={\mathbb{Z}}_{+}$, and $\Omega_{{\mathbb{T}}}:={\mathbb{R}}_{+}$
when ${\mathbb{T}}={\mathbb{R}}_{+}$. Note that by abuse of notation, for both
cases of discrete-time and continuous-time, we employ same expression
$G(\mathrm{j}\omega)$ for the transfer function of the system.

The constraints introduced in (13) are the Fourier transform of the impulse
response ${\mathrm{g}}$ at different frequencies. Since ${\mathrm{g}}$ belongs
to RKHS ${\mathcal{H}}_{\mathbbm{k}}$, one study the frequency response and
its properties in the domain of RKHS ${\mathcal{H}}_{\mathbbm{k}}$.

######  Definition 3.

With respect each $\omega\in\Omega_{{\mathbb{T}}}$, we define maps
${\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}:{\mathcal{H}}_{\mathbbm{k}}\to{\mathbb{R}}$
and
${\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}:{\mathcal{H}}_{\mathbbm{k}}\to{\mathbb{R}}$
such that for any
${\mathrm{g}}=(g_{t})_{t\in{\mathbb{T}}}\in{\mathcal{H}}_{\mathbbm{k}}$, we
have

$\\!\\!\\!\\!\\!\\!\\!\\!{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})\\!:=\\!\sum_{t\in{\mathbb{Z}}_{+}}\\!g_{t}\cos(\omega
t),\quad{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})\\!:=\\!-\\!\\!\sum_{t\in{\mathbb{Z}}_{+}}\\!g_{t}\sin(\omega
t),\\!\\!\\!\\!$ (14)

when ${\mathbb{T}}={\mathbb{Z}}_{+}$, and

$\\!\\!\\!\\!\\!\\!{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})\\!:=\\!\\!\int_{{\mathbb{R}}_{+}}\\!\\!\\!g_{t}\cos(\omega
t)\mathrm{d}t,\quad{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})\\!:=\\!-\\!\\!\int_{{\mathbb{R}}_{+}}\\!\\!\\!g_{t}\sin(\omega
t)\mathrm{d}t,\\!\\!$ (15)

when ${\mathbb{T}}={\mathbb{R}}_{+}$. Also
${\mathcal{F}}_{\omega}:{\mathcal{H}}_{\mathbbm{k}}\to{\mathbb{C}}$ is defined
as
${\mathcal{F}}_{\omega}({\mathrm{g}})={\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})+\mathrm{j}{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})$,
for any ${\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}$.

Based on the definition of ${\mathcal{F}}_{\omega}$, for any
${\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}$ and
$\omega\in\Omega_{{\mathbb{T}}}$, we have
$G(\mathrm{j}\omega)={\mathcal{F}}_{\omega}({\mathrm{g}})$, where $G$ is the
transfer function corresponding to the impulse response ${\mathrm{g}}$.
Accordingly, the problem (13) can be re-written in the following form

$\begin{array}[]{cl}\operatornamewithlimits{min}\limits_{{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}}&\operatornamewithlimits{\sum}\limits_{t\in{\mathscr{T}}}({\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}})-y_{t})^{2}+\lambda\
\\!\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2},\\\
\mathrm{s.t.}&|{\mathcal{F}}_{\omega}({\mathrm{g}})|\leq\rho,\quad\forall\omega\in\Omega_{{\mathbb{T}}}.\end{array}$
(16)

######  Remark 1.

One can see that ${\mathcal{F}}_{\omega}$ is the Fourier transform restricted
to the RKHS ${\mathcal{H}}_{\mathbbm{k}}$ evaluated for the frequency
$\omega\in\Omega_{{\mathbb{T}}}$. However, we should note that it does not
necessarily inherit the same properties of standard Fourier transform. More
precisely, the structure of ${\mathcal{H}}_{\mathbbm{k}}$ plays a key role.

Before proceeding further, we need to introduce a notion based on the kernel
$\mathbbm{k}$. For any $n\in{\mathbb{Z}}_{+}$, define $\mu_{n}\in[0,\infty]$
as

$\mu_{n}:=\begin{cases}\operatornamewithlimits{\sum}\limits_{t\in{\mathbb{Z}}_{+}}t^{n}\
\\!\mathbbm{k}(t,t)^{\frac{1}{2}},&\text{ if
}{\mathbb{T}}={\mathbb{Z}}_{+},\\\ \int_{{\mathbb{R}}_{+}}t^{n}\
\\!\mathbbm{k}(t,t)^{\frac{1}{2}}\ \\!\mathrm{d}t,&\text{ if
}{\mathbb{T}}={\mathbb{R}}_{+}.\end{cases}$ (17)

The next lemma introduces properties of
${\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}{}$,
${\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}{}$ and ${\mathcal{F}}_{\omega}{}$.

######  Lemma 3.

Assume that $\mu_{0}<\infty$. Then, the following hold:
1) The maps
${\mathcal{F}}_{\omega}^{\text{\rm{(r)}}},{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}:{\mathcal{H}}_{\mathbbm{k}}\to{\mathbb{R}}$
and ${\mathcal{F}}_{\omega}:{\mathcal{H}}_{\mathbbm{k}}\to{\mathbb{C}}$ are
linear continuous with
$\|{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}\|_{{\mathcal{L}}({\mathcal{H}}_{\mathbbm{k}},{\mathbb{R}})},\|{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}\|_{{\mathcal{L}}({\mathcal{H}}_{\mathbbm{k}},{\mathbb{R}})}\
\leq\ \mu_{0}.$
2) There exist unique elements
$\varphi_{\omega}^{\text{\rm{(r)}}}=(\varphi_{\omega,t}^{\text{\rm{(r)}}})_{t\in{\mathbb{T}}}$
and
$\varphi_{\omega}^{\text{\rm{(i)}}}=(\varphi_{\omega,t}^{\text{\rm{(i)}}})_{t\in{\mathbb{T}}}$
in ${\mathcal{H}}_{\mathbbm{k}}$ such that, for any
${\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}$, we have

${\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})={\langle{\varphi_{\omega}^{\text{\rm{(r)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}},\qquad{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})={\langle{\varphi_{\omega}^{\text{\rm{(i)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}},$
(18)

3) For any $t\in{\mathbb{T}}$, when ${\mathbb{T}}={\mathbb{Z}}_{+}$, we have

$\\!\\!\\!\\!\\!\\!\varphi_{\omega,t}^{\text{\rm{(r)}}}\\!=\\!\\!\\!\sum_{s\in{\mathbb{Z}}_{+}}\mathbbm{k}(t,s)\cos(\omega
s),\
\varphi_{\omega,t}^{\text{\rm{(i)}}}\\!=\\!-\\!\\!\sum_{s\in{\mathbb{Z}}_{+}}\mathbbm{k}(t,s)\sin(\omega
s),\\!\\!$ (19)

and, when ${\mathbb{T}}={\mathbb{R}}_{+}$, we have

$\\!\\!\\!\\!\\!\\!\varphi_{\omega,t}^{\text{\rm{(r)}}}\\!=\\!\\!\\!\int_{{\mathbb{R}}_{+}}\\!\\!\\!\\!\\!\mathbbm{k}(t,s)\cos(\omega
s)\mathrm{d}s,\
\varphi_{\omega,t}^{\text{\rm{(i)}}}\\!=\\!-\\!\\!\int_{{\mathbb{R}}_{+}}\\!\\!\\!\\!\\!\mathbbm{k}(t,s)\sin(\omega
s)\mathrm{d}s,\\!\\!$ (20)

###### Proof.

See Appendix A-A. ∎

Let $\Omega\subseteq\Omega_{{\mathbb{T}}}$ and $\eta\in{\mathbb{R}}_{+}$.
Define set ${\mathscr{G}}_{\mathbbm{k}}(\eta,\Omega)$ as

${\mathscr{G}}_{\mathbbm{k}}(\eta,\Omega):=\bigg{\\{}{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}\
\bigg{|}\
|{\mathcal{F}}_{\omega}({\mathrm{g}})|\leq\eta,\forall\omega\in\Omega\bigg{\\}}.$
(21)

One can see that the feasible set in (16) is a special case of this set. Based
on Lemma 3, we study the main properties of
${\mathscr{G}}_{\mathbbm{k}}(\eta,\Omega)$ in the next theorem.

######  Theorem 4.

Assume that $\mu_{0}<\infty$. Then, ${\mathscr{G}}_{\mathbbm{k}}(\eta,\Omega)$
is non-empty, closed and convex.

###### Proof.

From (21), one can see that

$\begin{split}&{\mathscr{G}}_{\mathbbm{k}}(\eta,\Omega)=\\{{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}\
|\ |{\mathcal{F}}_{\omega}({\mathrm{g}})|\leq\eta,\forall\omega\in\Omega\\}\\\
&=\bigcap\limits_{\omega\in\Omega}\
\\{{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}\ |\
|{\mathcal{F}}_{\omega}({\mathrm{g}})|\leq\eta\\}=\bigcap\limits_{\omega\in\Omega}\
{\mathscr{G}}_{\mathbbm{k}}(\eta,\\{\omega\\}).\end{split}$

Therefore, it is enough to show that, for any
$\omega\in\Omega_{{\mathbb{T}}}$,
${\mathscr{G}}_{\mathbbm{k}}(\eta,\\{\omega\\})$ is a closed and convex subset
of ${\mathcal{H}}_{\mathbbm{k}}$. Since
$|{\mathcal{F}}_{\omega}({\mathrm{g}})|^{2}=|{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})|^{2}+|{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})|^{2}$,
we have that

${\mathscr{G}}_{\mathbbm{k}}(\eta,\\{\omega\\})=\bigg{\\{}{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}\
\bigg{|}\
|{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})|^{2}+|{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})|\leq\eta^{2}\bigg{\\}}.$
(22)

Due to Lemma 3, we know that
${\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})={\langle{\varphi_{\omega}^{\text{\rm{(r)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$
and
${\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})={\langle{\varphi_{\omega}^{\text{\rm{(i)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$,
for any ${\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}$. With respect to each
$\theta\in[0,\frac{\pi}{2}]$, define set
${\mathscr{G}}_{\mathbbm{k}}(\eta,\omega,\theta)$ as

$\begin{split}{\mathscr{G}}_{\mathbbm{k}}(\eta,\omega,\theta):=\Big{\\{}{\mathrm{g}}\in\\!{\mathcal{H}}_{\mathbbm{k}}\Big{|}&|{\langle{\varphi_{\omega}^{\text{\rm{(r)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}|\leq\eta\cos\theta,\\\
&\qquad|{\langle{\varphi_{\omega}^{\text{\rm{(i)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}|\leq\eta\sin\theta\Big{\\}}.\end{split}$

Note that ${\mathscr{G}}_{\mathbbm{k}}(\eta,\omega,\theta)$ is the
intersection of
$\\{{\mathrm{g}}\\!\in\\!{\mathcal{H}}_{\mathbbm{k}}|{\langle{\varphi_{\omega}^{\text{\rm{(r)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}\\!\leq\\!\eta\cos\theta\\}$,
$\\{{\mathrm{g}}\\!\in\\!{\mathcal{H}}_{\mathbbm{k}}|{\langle{\varphi_{\omega}^{\text{\rm{(r)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}\\!\geq\\!-\eta\cos\theta\\}$,
$\\{{\mathrm{g}}\\!\in\\!{\mathcal{H}}_{\mathbbm{k}}|{\langle{\varphi_{\omega}^{\text{\rm{(r)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}\\!\leq\\!\eta\sin\theta\\}$,
and,
$\\{{\mathrm{g}}\\!\in\\!{\mathcal{H}}_{\mathbbm{k}}|{\langle{\varphi_{\omega}^{\text{\rm{(r)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}\\!\geq\\!-\eta\sin\theta\\}$,
which are closed half-spaces in ${\mathcal{H}}_{\mathbbm{k}}$. Therefore,
${\mathscr{G}}_{\mathbbm{k}}(\eta,\omega,\theta)$ is a closed and convex set.
Since we have
${\mathscr{G}}_{\mathbbm{k}}(\eta,\\{\omega\\})=\bigcap_{\theta\in[0,\frac{\pi}{2}]}{\mathscr{G}}_{\mathbbm{k}}(\eta,\omega,\theta)$,
the set ${\mathscr{G}}_{\mathbbm{k}}(\eta,\\{\omega\\})$ is closed and convex
as well. Note that for $\mathbf{0}\in{\mathcal{H}}_{\mathbbm{k}}$ and any
$\omega\in\Omega$, we have ${\mathcal{F}}_{\omega}(\mathbf{0})=0$. Therefore,
$|{\mathcal{F}}_{\omega}(\mathbf{0})|=0\leq\eta$ and subsequently,
$\mathbf{0}\in{\mathscr{G}}(\eta,\Omega)$. This shows that
${\mathscr{G}}(\eta,\Omega)$ is a non-empty set and concludes the proof. ∎

Before proceeding to the main theorem of this section, we need to present an
auxiliary lemma.

######  Lemma 5.

Assume that $\mu_{0}<\infty$ and ${\mathrm{u}}\in{\mathscr{L}}^{\infty}$.
Then, for any $t\in{\mathbb{T}}$, the map
${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}:{\mathcal{H}}_{\mathbbm{k}}\to{\mathbb{R}}$,
defined in (1) and (2), is linear and continuous with
$\|{\mathrm{L}}^{\\!{\mathrm{u}}}_{t}\|_{{\mathcal{L}}({\mathcal{H}}_{\mathbbm{k}},{\mathbb{R}})}\leq\mu_{0}\|{\mathrm{u}}\|_{\infty}$.
Also, there exists unique
$\varphi_{t}^{\text{\rm{(u)}}}:=(\varphi_{t,s}^{\text{\rm{(u)}}})_{s\in{\mathbb{T}}}\in{\mathcal{H}}_{\mathbbm{k}}$
such that
${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}})={\langle{\varphi_{t}^{\text{\rm{(u)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$,
for any ${\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}$. Moreover, for any
$s\in{\mathbb{T}}$, we have

$\varphi_{t,s}^{\text{\rm{(u)}}}={\mathrm{L}}^{\\!{\mathrm{u}}}_{t}(\mathbbm{k}_{s})=\begin{cases}\sum_{\tau\in{\mathbb{Z}}_{+}}\mathbbm{k}(s,\tau)u_{t-\tau},&\text{
if }{\mathbb{T}}={\mathbb{Z}}_{+},\\\
\int_{{\mathbb{R}}_{+}}\mathbbm{k}(s,\tau)u_{t-\tau}\mathrm{d}\tau,&\text{ if
}{\mathbb{T}}={\mathbb{R}}_{+}.\\\ \end{cases}$ (23)

###### Proof.

See Appendix A-B. ∎

######  Theorem 6.

Assume that $\mu_{0}>0$ and consider the following program

$\operatornamewithlimits{min}\limits_{{\mathrm{g}}\in{\mathscr{G}}_{\mathbbm{k}}(\eta,\Omega)}\
\operatornamewithlimits{\sum}\limits_{t\in{\mathscr{T}}}({\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}})-y_{t})^{2}+\lambda\
\\!\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}.$ (24)

Then, (24) is a convex optimization problem with a _unique_ solution
${\mathrm{g}}^{\star}_{\Omega}$. Moreover, we have

$\|{\mathrm{g}}^{\star}_{\Omega}\|_{{\mathcal{H}}_{\mathbbm{k}}}\ \\!\leq\
\\!\lambda^{-\frac{1}{2}}\|{\mathrm{y}}\|.$ (25)

where vector ${\mathrm{y}}$ is defined as
${\mathrm{y}}=[y_{t}]_{t\in{\mathscr{T}}}$.

###### Proof.

Define
${\mathcal{J}}:{\mathcal{H}}_{\mathbbm{k}}\to{\mathbb{R}}\cup\\{+\infty\\}$
such that for any ${\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}$ we have

${\mathcal{J}}({\mathrm{g}})=\sum_{t\in{\mathscr{T}}}({\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}})-y_{t})^{2}+\lambda\|g\|^{2}_{{\mathcal{H}}_{\mathbbm{k}}}+\delta_{{\mathscr{G}}_{\mathbbm{k}}(\eta,\Omega)}({\mathrm{g}}).$
(26)

Since $\mathbf{0}\in{\mathscr{G}}_{\mathbbm{k}}(\eta,\Omega)$, we have
$\delta_{{\mathscr{G}}_{\mathbbm{k}}(\eta,\Omega)}(\mathbf{0})=0$. From the
definition of ${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}$ in (1) and (2), it is
implied that ${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}(\mathbf{0})=0$, for each
$t\in{\mathscr{T}}$, and subsequently, we have
${\mathcal{J}}(\mathbf{0})=\sum_{t\in{\mathscr{T}}}y_{t}^{2}=\|{\mathrm{y}}\|^{2}<\infty.$
From Theorem 4, we know that ${\mathscr{G}}_{\mathbbm{k}}(\eta,\Omega)$ is a
convex and closed set, and consequently,
$\delta_{{\mathscr{G}}_{\mathbbm{k}}(\eta,\Omega)}$ is a proper lower semi-
continuous convex function [55]. Due to Lemma 5, we know that
${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}:{\mathcal{H}}_{\mathbbm{k}}\to{\mathbb{R}}$
is a continuous linear map, for each $t\in{\mathscr{T}}$. Therefore, function
${\mathcal{E}}_{{\mathscr{D}}}:{\mathcal{H}}_{\mathbbm{k}}\to{\mathbb{R}}$,
defined in (8), is a convex and continuous function. Since $\lambda>0$, we
know that ${\mathcal{J}}$ is a proper and lower semi-continuous strongly
convex function. Therefore,
$\min_{g\in{\mathcal{H}}_{\mathbbm{k}}}{\mathcal{J}}(g)$ has a unique (finite)
solution [55], and subsequently, (24) is a convex program with a unique
solution ${\mathrm{g}}^{\star}_{\Omega}$ with finite cost. Since
$\mathbf{0}\in{\mathscr{G}}_{\mathbbm{k}}(\eta,\Omega)$ and due to optimality
of ${\mathrm{g}}^{\star}_{\Omega}$, we have
${\mathcal{J}}({\mathrm{g}}^{\star}_{\Omega})\leq{\mathcal{J}}(\mathbf{0})$.
Subsequently, one can see

$\lambda\|{\mathrm{g}}^{\star}_{\Omega}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}\\!\leq\\!\operatornamewithlimits{\sum}\limits_{t\in{\mathscr{T}}}({\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}}^{\star}_{\Omega})-y_{t})^{2}+\lambda\
\\!\|{\mathrm{g}}^{\star}_{\Omega}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}\leq{\mathcal{J}}(\mathbf{0})\\!=\\!\sum_{t\in{\mathscr{T}}}y_{t}^{2},$

which implies (25). This concludes the proof. ∎

######  Corollary 7.

Assume that $\mu_{0}<\infty$. Then, (16) is a convex optimization with a
unique solution denoted by ${\mathrm{g}}^{\star}$. Moreover,
${\mathrm{g}}^{\star}$ satisfies inequality (25). A similar property holds for
(13).

###### Proof.

In Theorem 6, set $\Omega$ and $\eta$ respectively to $\Omega_{{\mathbb{T}}}$
and $\rho$. Then, the convexity of (16) as well as the existence and
uniqueness of its solution immediately follows. Since (13) and (16) are
equivalent, the same claim holds for (13). ∎

In the above discussion, we have assumed that $\mu_{0}<\infty$. The next
theorem shows that the boundedness of $\mu_{0}$ is a valid assumption for most
of the stable kernels introduced in the literature. In particular, it holds
for TC kernel.

######  Theorem 8.

Consider kernel $\mathbbm{k}$ and assume there exist $\beta,\gamma>0$ such
that $|\mathbbm{k}(s,t)|\leq\gamma\mathrm{e}^{-\beta\max(s,t)}$, for any
$s,t\in{\mathbb{T}}$. Then, for any $n\in{\mathbb{Z}}_{+}$, we have
$\mu_{n}\leq\gamma^{\frac{1}{2}}(\frac{2}{\beta})^{n+1}n!$, where
$n!:=\prod_{k=1}^{n}k$, when $n\geq 1$, and, $0!:=1$.

###### Proof.

See Appendix A-C. ∎

Based on Corollary 7, we know that the optimization problem (13) has a unique
solution addressing Problem 1. Since this optimization problem is defined over
an infinite-dimensional space with uncountably infinite number of constraints,
obtaining this solution is not yet tractable in the current form. This is
discussed in the next section.

##  V Towards a Tractable Scheme

In this section, we present a tractable approach to derive the solution of the
nonparametric estimation problem introduced in Section IV.

Consider optimization problem (16). For ease of discussion and without loss of
generality, we assume $\rho=1$. More precisely, one can use a change of
variable and replace $y_{t}$ with $\rho^{-1}y_{t}$ in ${\mathscr{D}}$ and
then, set $\rho=1$ in Problem 1 as well as in (16), or equivalently in (13).
With respect to each $\omega\in\Omega_{{\mathbb{T}}}$, there exists a
constraint in (16). Thus, we have uncountably infinite number of constraints
which makes the problem intractable. To resolve this issue, we approximate the
problem by considering a suitable finite subset of $\Omega_{{\mathbb{T}}}$. In
order to investigate this possibility, we need the notion of partition
introduced in the next definition.

######  Definition 4.

We say ${\mathscr{P}}$ is a partition of interval $[a,b]$ if ${\mathscr{P}}$
is a finite subset of $[a,b]$ as ${\mathscr{P}}=\\{\omega_{i}\ |\
i=0,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}\\}$ where

$a=\omega_{0}<\omega_{1}<\ldots<\omega_{n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}}=b.$
(27)

With respect to partition ${\mathscr{P}}$, the mesh of ${\mathscr{P}}$,
denoted by $\mathrm{mesh}({\mathscr{P}})$, is defined as

$\mathrm{mesh}({\mathscr{P}}):=\max\\{|\omega_{i}-\omega_{i-1}|\ |\
i=1,2,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}\\}.$ (28)

Now, let ${\mathscr{P}}=\\{\omega_{i}\ |\
i=0,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}\\}$ be a given partition.
One can see that satisfying the constraints
$|{\mathcal{F}}_{\omega}({\mathrm{g}})|\leq 1$, for all $w\in{\mathscr{P}}$,
does not necessarily imply that the desired feature
$\sup_{\omega\in\Omega_{{\mathbb{T}}}}|{\mathcal{F}}_{\omega}({\mathrm{g}})|\leq
1$. More precisely, these constraints do not guarantee that for all
$\omega\in\Omega_{{\mathbb{T}}}\backslash{\mathscr{P}}$ we have
$|{\mathcal{F}}_{\omega}({\mathrm{g}})|\leq 1$ as well. Accordingly, in order
to approximate problem (16), we take $\varepsilon>0$ and consider the
following problem as our approximation

$\begin{array}[]{cl}\operatornamewithlimits{min}\limits_{{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}}&\operatornamewithlimits{\sum}\limits_{t\in{\mathscr{T}}}({\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}})-y_{t})^{2}+\lambda\
\\!\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2},\\\
\mathrm{s.t.}&|{\mathcal{F}}_{\omega}({\mathrm{g}})|^{2}\leq
1-\varepsilon,\quad\forall\omega\in{\mathscr{P}}.\end{array}$ (29)

In the following, we provide appropriate conditions on partition
${\mathscr{P}}$ and $\varepsilon>0$ for ensuring that the solution of (29)
satisfies
$\sup_{\omega\in\Omega_{{\mathbb{T}}}}|{\mathcal{F}}_{\omega}({\mathrm{g}})|\leq
1$. These conditions depend on the bandwidth of system ${\mathrm{g}}$ and the
rate of change of $|{\mathcal{F}}_{\omega}({\mathrm{g}})|$ with respect to
$\omega$. Accordingly, we need to introduce necessary definitions and
preliminaries.

######  Definition 5.

Let ${\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}$ be a given impulse response.
With respect to ${\mathrm{g}}$, we define function
$m_{{\mathrm{g}}}:\Omega_{{\mathbb{T}}}\to{\mathbb{R}}$ as

$m_{{\mathrm{g}}}(\omega)=|{\mathcal{F}}_{\omega}({\mathrm{g}})|^{2}.$ (30)

Given impulse response ${\mathrm{g}}$, the function $m_{{\mathrm{g}}}$ shows
the squared magnitude of transfer function corresponding to ${\mathrm{g}}$ at
different frequencies. The following lemma introduces a bound for the rate of
changes of $m_{{\mathrm{g}}}$ in terms of its Lipschitz constant, which in
turn can be expressed in terms of the properties of the regularization kernel.
This will be used later in the analysis of problem (29).

######  Lemma 9.

Assume that $\mu_{0},\mu_{1}<\infty$. Then, for any
${\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}$ and
$\omega_{1},\omega_{2}\in\Omega_{{\mathbb{T}}}$, we have

$|m_{{\mathrm{g}}}(\omega_{2})-m_{{\mathrm{g}}}(\omega_{1})|\leq
L_{{\mathrm{g}}}|\omega_{2}-\omega_{1}|,$ (31)

where
$L_{{\mathrm{g}}}:=4\mu_{0}\mu_{1}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}$.

###### Proof.

See Appendix A-D. ∎

For the case of discrete-time systems, i.e., ${\mathbb{T}}={\mathbb{Z}}_{+}$,
the frequency range $\Omega_{{\mathbb{T}}}$ is the bounded interval
$[0,\omega_{\mathrm{max}}]$, where $\omega_{\mathrm{max}}=\pi$. In order to
introduce analogous of $\omega_{\mathrm{max}}$ for the continuous-time
systems, we need the next assumption.

######  Assumption 1.

For the case ${\mathbb{T}}={\mathbb{R}}_{+}$, we have

$\lim_{\omega\to\infty}\int_{{\mathbb{R}}_{+}}\int_{{\mathbb{R}}_{+}}\mathbbm{k}(s,t)\mathrm{e}^{-\mathrm{j}\omega(s-t)}\mathrm{d}s\mathrm{d}t=0.$
(32)

This assumption says that the bandwidth of kernel $\mathbbm{k}$ is bounded.
Moreover, from (32), we know that there exists
$\omega_{\mathrm{max}}\in{\mathbb{R}}_{+}$ such that

$\bigg{|}\int_{{\mathbb{R}}_{+}}\int_{{\mathbb{R}}_{+}}\mathbbm{k}(s,t)\mathrm{e}^{-\mathrm{j}\omega(s-t)}\mathrm{d}s\mathrm{d}t\bigg{|}\leq\frac{\lambda}{\sum_{t\in{\mathscr{T}}}y_{t}^{2}},$
(33)

for all $\omega>\omega_{\mathrm{max}}$. In the following, it is shown that the
frequency value $\omega_{\mathrm{max}}$ plays the role of an upper bound for
the bandwidth of the system of interest. The next theorem shows that for TC
kernels Assumption 1 holds. One may show similar results for other stable
kernels in the literature.

######  Theorem 10.

Let $\beta>0$ and
$\mathbbm{k}:{\mathbb{R}}_{+}\times{\mathbb{R}}_{+}\to{\mathbb{R}}_{+}$ be the
TC kernel $\mathbbm{k}(s,t)=\mathrm{e}^{-\beta\max(s,t)}$. Then, we have

$\int_{{\mathbb{R}}_{+}}\int_{{\mathbb{R}}_{+}}\mathbbm{k}(s,t)\mathrm{e}^{-\mathrm{j}\omega(s-t)}\mathrm{d}s\mathrm{d}t=\frac{2}{\omega^{2}+\beta^{2}}.$
(34)

###### Proof.

See Appendix A-E. ∎

Based on the discussion above, we can present the main theorem of this
section.

######  Theorem 11.

Assume that $\mu_{0},\mu_{1}$, defined in (17), are finite. Let
${\mathscr{P}}$ be a given partition of $[0,\omega_{\mathrm{max}}]$ and
$\varepsilon>0$. Then, optimization problem (29) is a convex program with a
unique solution ${\mathrm{g}}^{(\star,\varepsilon)}$. For
${\mathrm{g}}^{(\star,\varepsilon)}$, we have

$\|{\mathrm{g}}^{(\star,\varepsilon)}\|_{{\mathcal{H}}_{\mathbbm{k}}}\
\\!\leq\ \\!\lambda^{-\frac{1}{2}}\|{\mathrm{y}}\|.$ (35)

where ${\mathrm{y}}$ is the vector defined as
${\mathrm{y}}=[y_{t}]_{t\in{\mathscr{T}}}$. Moreover, if
$\mathrm{mesh}({\mathscr{P}})\leq\frac{2\varepsilon}{L}$ where
$L:=\frac{1}{\lambda}4\mu_{0}\mu_{1}\sum_{t\in{\mathscr{T}}}y_{i}^{2}$, then
we have

$\|G^{(\star,\varepsilon)}\|_{{\mathcal{H}}_{\infty}}=\sup_{\omega\in\Omega_{{\mathbb{T}}}}|{\mathcal{F}}_{\omega}({\mathrm{g}}^{(\star,\varepsilon)})|\leq
1.$ (36)

where $G^{(\star,\varepsilon)}$ is the transfer function corresponding to
${\mathrm{g}}^{(\star,\varepsilon)}$.

###### Proof.

Due to the definition of the set ${\mathscr{G}}_{\mathbbm{k}}$ in (21), we
know that the feasible set of optimization problem (29) is
${\mathscr{G}}_{\mathbbm{k}}((1-\varepsilon)^{\frac{1}{2}},{\mathscr{P}})$.
Subsequently, problem (29) can be written as

$\operatornamewithlimits{min}\limits_{{\mathrm{g}}\in{\mathscr{G}}_{\mathbbm{k}}((1-\varepsilon)^{\frac{1}{2}},{\mathscr{P}})}\
\
\operatornamewithlimits{\sum}\limits_{t\in{\mathscr{T}}}({\mathrm{L}}^{\\!{\mathrm{u}}}_{i}({\mathrm{g}})-y_{i})^{2}+\lambda\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}.$
(37)

Then, according to Theorem 6, the optimization problem (37) as well as (29),
is a convex program with unique solution, denoted by
${\mathrm{g}}^{(\star,\varepsilon)}$, which satisfies (35).

Let $\omega\in[0,\omega_{\mathrm{max}}]$. If $\omega\in{\mathscr{P}}$, then
$|{\mathcal{F}}_{\omega}({\mathrm{g}}^{(\star,\varepsilon)})|\leq(1-\varepsilon)^{\frac{1}{2}}\leq
1$. If $\omega\notin{\mathscr{P}}$, then there exists $i\in\\{1,\ldots,n\\}$,
such that $\omega\in(\omega_{i-1},\omega_{i})$. If
$|{\mathcal{F}}_{\omega}({\mathrm{g}}^{(\star,\varepsilon)})|>1$, then, due to
Lemma 9 and
$|{\mathcal{F}}_{\omega_{i-1}}({\mathrm{g}}^{(\star,\varepsilon)})|^{2},|{\mathcal{F}}_{\omega_{i}}({\mathrm{g}}^{(\star,\varepsilon)})|^{2}\leq
1-\varepsilon$, we know that

$\begin{split}\varepsilon&<\Big{|}\
|{\mathcal{F}}_{\omega_{i}}({\mathrm{g}}^{(\star,\varepsilon)})|^{2}-|{\mathcal{F}}_{\omega}({\mathrm{g}}^{(\star,\varepsilon)})|^{2}\Big{|}\\\
&\ \
=|m_{{\mathrm{g}}^{(\star,\varepsilon)}}(\omega_{i})-m_{{\mathrm{g}}^{(\star,\varepsilon)}}(\omega)|\\!\leq\\!L_{{\mathrm{g}}^{(\star,\varepsilon)}}|\omega_{i}-\omega|,\\\
\varepsilon&<\Big{|}\
|{\mathcal{F}}_{\omega}({\mathrm{g}}^{(\star,\varepsilon)})|^{2}-|{\mathcal{F}}_{\omega_{i-1}}({\mathrm{g}}^{(\star,\varepsilon)})|^{2}\Big{|}\\\
&\ \
=|m_{{\mathrm{g}}^{(\star,\varepsilon)}}(\omega)-m_{{\mathrm{g}}^{(\star,\varepsilon)}}(\omega_{i-1})|\\!\leq\\!L_{{\mathrm{g}}^{(\star,\varepsilon)}}|\omega-\omega_{i-1}|,\end{split}$
(38)

where
$L_{{\mathrm{g}}^{(\star,\varepsilon)}}=4\mu_{0}\mu_{1}\|{\mathrm{g}}^{(\star,\varepsilon)}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}$.
Therefore, since $|\omega_{i}-\omega_{i-1}|\leq\mathrm{mesh}({\mathscr{P}})$,
we have that
$2\varepsilon<L_{{\mathrm{g}}^{(\star,\varepsilon)}}\mathrm{mesh}({\mathscr{P}})$.
Subsequently, it follows that

$\begin{split}\mathrm{mesh}({\mathscr{P}})>\frac{2\varepsilon}{L_{{\mathrm{g}}^{(\varepsilon)}}}&=\frac{2\varepsilon}{4\mu_{0}\mu_{1}\|{\mathrm{g}}^{(\varepsilon)}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}}\\\
&\geq\frac{2\varepsilon\lambda}{4\mu_{0}\mu_{1}\\!\sum_{t\in{\mathscr{T}}}\\!y_{t}^{2}}=\frac{2\varepsilon}{L},\end{split}$
(39)

which contradicts the specification that
$\mathrm{mesh}({\mathscr{P}})\leq\frac{2\varepsilon}{L}$. Therefore, we have
$|{\mathcal{F}}_{\omega}({\mathrm{g}}^{(\varepsilon)})|\leq 1$. This shows
that
$\|G^{(\star,\varepsilon)}\|_{{\mathcal{H}}_{\infty}}=\sup_{\omega\in[0,\omega_{\mathrm{max}}]}|{\mathcal{F}}_{\omega}({\mathrm{g}}^{(\star,\varepsilon)})|\leq
1$, which concludes the proof for the case ${\mathbb{T}}={\mathbb{Z}}_{+}$
where we have $\omega_{\mathrm{max}}=\pi$. Now, we consider the case
${\mathbb{T}}={\mathbb{R}}_{+}$ and let $\omega>\omega_{\mathrm{max}}$. Due to
Lemma 3 and the Cauchy-Schwartz inequality, we know that

$\begin{split}|{\mathcal{F}}_{\omega}({\mathrm{g}}^{(\star,\varepsilon)})|^{2}&=|{\langle{{\mathrm{g}}^{(\star,\varepsilon)},\varphi_{\omega}^{\text{\rm{(r)}}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}|^{2}+|{\langle{{\mathrm{g}}^{(\star,\varepsilon)},\varphi_{\omega}^{\text{\rm{(i)}}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}|^{2}\\\
&\leq\|{\mathrm{g}}^{(\star,\varepsilon)}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}\
\Big{[}\|\varphi_{\omega}^{\text{\rm{(r)}}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}+\|\varphi_{\omega}^{\text{\rm{(i)}}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}\Big{]}.\end{split}$
(40)

On the other hand, from (18), (15), and (33), we have

$\begin{split}\|\varphi_{\omega}^{\text{\rm{(r)}}}&\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}+\|\varphi_{\omega}^{\text{\rm{(i)}}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}={\langle{\varphi_{\omega}^{\text{\rm{(r)}}},\varphi_{\omega}^{\text{\rm{(r)}}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}^{2}+{\langle{\varphi_{\omega}^{\text{\rm{(i)}}},\varphi_{\omega}^{\text{\rm{(i)}}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}^{2}\\\
&=\int_{{\mathbb{R}}_{+}}\\!\\!\\!\varphi_{\omega,t}^{\text{\rm{(r)}}}\cos(\omega
t)\mathrm{d}t+\int_{{\mathbb{R}}_{+}}\\!\\!\\!\varphi_{\omega,t}^{\text{\rm{(i)}}}\sin(\omega
t)\mathrm{d}t\\\
&=\int_{{\mathbb{R}}_{+}}\\!\int_{{\mathbb{R}}_{+}}\\!\mathbbm{k}(t,s)\cos(\omega
s)\cos(\omega t)\mathrm{d}s\mathrm{d}t\\\
&\qquad+\int_{{\mathbb{R}}_{+}}\\!\int_{{\mathbb{R}}_{+}}\\!\mathbbm{k}(t,s)\sin(\omega
s)\sin(\omega t)\mathrm{d}s\mathrm{d}t\\\
&=\int_{{\mathbb{R}}_{+}}\\!\int_{{\mathbb{R}}_{+}}\\!\mathbbm{k}(t,s)\mathrm{e}^{-\mathrm{j}\omega(s-t)}\mathrm{d}s\mathrm{d}t\leq\frac{\lambda}{\sum_{t\in{\mathscr{T}}}y_{t}^{2}},\end{split}$
(41)

where the last equality is due to the fact that
$\int_{{\mathbb{R}}_{+}}\\!\int_{{\mathbb{R}}_{+}}\\!\mathbbm{k}(t,s)\sin(\omega(s-t))\mathrm{d}s\mathrm{d}t=0$
which is a result of $\mathbbm{k}(t,s)=\mathbbm{k}(s,t)$ and
$\sin(\omega(s-t))=-\sin(\omega(t-s))$. Accordingly, due to (40) and (35), we
have $|{\mathcal{F}}_{\omega}({\mathrm{g}}^{(\star,\varepsilon)})|^{2}\leq 1$.
This shows that
$\|G^{(\star,\varepsilon)}\|_{{\mathcal{H}}_{\infty}}=\sup_{\omega\in\Omega_{{\mathbb{T}}}}|{\mathcal{F}}_{\omega}({\mathrm{g}}^{(\star,\varepsilon)})|\leq
1$ and concludes the proof. ∎

Due to Theorem 11, we know that (29) admits a unique solution. While this
optimization problem is defined over the infinite-dimensional Hilbert space
${\mathcal{H}}_{\mathbbm{k}}$, we can find its unique solution
${\mathrm{g}}^{(\star,\varepsilon)}$ by solving an equivalent convex finite-
dimensional program. This feature, which makes (29) a tractable problem, is
due to the structure of the RKHS ${\mathcal{H}}_{\mathbbm{k}}$ and the
Representer Theorem, provided below.

######  Theorem 12 (Representer Theorem, [56, 57]).

Let $e:{\mathbb{R}}^{m}\to{\mathbb{R}}\cup\\{+\infty\\}$ and
$r:{\mathbb{R}}_{+}\to{\mathbb{R}}$ be functions such that $r$ is an
increasing function. Also, let ${\mathcal{H}}$ be a Hilbert space with inner
product ${\langle{\cdot,\cdot}\rangle}_{{\mathcal{H}}}$. Consider the
optimization problem

$\min_{{\mathrm{w}}\in{\mathcal{H}}}\
e({\langle{{\mathrm{w}}_{1},{\mathrm{w}}}\rangle}_{{\mathcal{H}}},\ldots,{\langle{{\mathrm{w}}_{m},{\mathrm{w}}}\rangle}_{{\mathcal{H}}})+r(\|{\mathrm{w}}\|_{{\mathcal{H}}}),$
(42)

where ${\mathrm{w}}_{1},\ldots,{\mathrm{w}}_{m}\in{\mathcal{H}}$ are given
vectors. Then, if (42) admits a solution, it has also a solution in
${\mathscr{W}}:=\mathrm{span}\\{{\mathrm{w}}_{i}\\}_{i=1}^{m}$.

In order to present the tractable finite-dimensional optimization problem
equivalent to (29), additional definitions are required. Define
$m:=n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}+2n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}+2$
and the index sets
${\mathcal{I}}^{{\scalebox{0.63}{$(\mathrm{u})$}}}:=\\{0,1,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1\\}$,
${\mathcal{I}}^{{\scalebox{0.63}{$(\mathscr{P})$}}}:=\\{0,1,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}\\}$,
and ${\mathcal{I}}:=\\{0,1,\ldots,m-1\\}$. Let $\\{\varphi_{i}\\}_{i=0}^{m-1}$
be vectors defined as

$\begin{cases}\varphi_{i}:=\varphi_{t_{i}}^{\text{\rm{(u)}}},&\text{ for
}i\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathrm{u})$}}},\\\
\varphi_{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}+2j}:=\varphi_{\omega_{j}}^{\text{\rm{(r)}}},&\text{
for }j\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathscr{P})$}}},\\\
\varphi_{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}+2j+1}:=\varphi_{\omega_{j}}^{\text{\rm{(i)}}},&\text{
for }j\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathscr{P})$}}}.\end{cases}$ (43)

Let $\Phi\in{\mathbb{R}}^{m\times m}$ be a symmetric matrix such that its
entry at the $i^{\text{\tiny{th}}}$ row and the $j^{\text{\tiny{th}}}$ column
is
${\langle{\varphi_{i-1},\varphi_{j-1}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$,
for $i,j=1,\ldots,m$. One can see that $\Phi$ is the Gram matrix of vectors
$\varphi_{0},\ldots,\varphi_{m-1}$. Moreover, for
$i\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathrm{u})$}}}$, we define vector
${\mathrm{a}}_{i}\in{\mathbb{R}}^{m}$ as the $(i+1)^{\text{\tiny{th}}}$ column
of $\Phi$. Similarly, vectors ${\mathrm{b}}_{j}\in{\mathbb{R}}^{m}$ and
${\mathrm{c}}_{j}\in{\mathbb{R}}^{m}$ are defined respectively as the
$(n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}+2j+1)^{\text{\tiny{th}}}$ and the
$(n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}+2j+2)^{\text{\tiny{th}}}$ column of
$\Phi$, for $j\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathscr{P})$}}}$. In the
following, without loss of generality, we assume that
$\varphi_{0},\ldots,\varphi_{m-1}$ are linearly independent. Indeed, if for
some $i\in{\mathcal{I}}$, the vector $\varphi_{i}$ belongs to
$\mathrm{span}\\{\varphi_{j}|j\in{\mathcal{I}}\backslash\\{i\\}\\}$, it does
not provide any additional information and one can replace it with a linear
combination of $\\{\varphi_{j}|j\in{\mathcal{I}}\backslash\\{i\\}\\}$.

We can now present the theorem on the equivalent tractable finite-dimensional
program.

######  Theorem 13.

The unique solution of optimization problem (29),
${\mathrm{g}}^{(\star,\varepsilon)}$, is in the linear form

${\mathrm{g}}^{(\star,\varepsilon)}=x_{0}\varphi_{0}+\ldots+x_{m-1}\varphi_{m-1},$
(44)

where
${\mathrm{x}}:=\begin{bmatrix}x_{0}&x_{1}&\ldots&x_{m-1}\end{bmatrix}^{\mathsf{\scriptscriptstyle
T}}\in{\mathbb{R}}^{m}$ is the unique solution of the following convex program

$\begin{array}[]{cl}\operatornamewithlimits{min}\limits_{{\mathrm{x}}\in{\mathbb{R}}^{m}}&\sum_{i\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathrm{u})$}}}}({\mathrm{a}}_{i}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}}-y_{i})^{2}+\lambda{\mathrm{x}}^{\mathsf{\scriptscriptstyle
T}}\Phi{\mathrm{x}}\\\
\mathrm{s.t.}&({\mathrm{b}}_{j}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}})^{2}+({\mathrm{c}}_{j}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}})^{2}\leq 1-\varepsilon,\quad\forall
j\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathscr{P})$}}}.\end{array}$ (45)

###### Proof.

Define
${\mathcal{J}}:{\mathcal{H}}_{\mathbbm{k}}\to{\mathbb{R}}\cup\\{+\infty\\}$ as

${\mathcal{J}}({\mathrm{g}}):=\sum_{t\in{\mathscr{T}}}({\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}})-y_{t})^{2}+\sum_{\omega\in{\mathscr{P}}}\delta_{\\{|{\mathcal{F}}_{\omega}|^{2}\leq
1-\varepsilon\\}}({\mathrm{g}}).$ (46)

Since
${\mathscr{P}}=\\{\omega_{0},\ldots,\omega_{n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}}\\}$
is a finite set, the summation in (46) is well-defined. We know that
optimization problem (29) is equivalent to
$\min_{{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}}{\mathcal{J}}({\mathrm{g}})+\lambda{\mathcal{R}}({\mathrm{g}})$,
where ${\mathcal{R}}:{\mathcal{H}}_{\mathbbm{k}}\to{\mathbb{R}}$ is defined as
${\mathcal{R}}({\mathrm{g}})=\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}$
(see Section IV-B). Let the function
$e:{\mathbb{R}}^{m}\to{\mathbb{R}}\cup\\{+\infty\\}$ be defined such that for
any ${\mathrm{z}}=(z_{i})_{i=0}^{m-1}\in{\mathbb{R}}^{m}$ we have

$e({\mathrm{z}}):=\sum_{i\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathrm{u})$}}}}(z_{i}-y_{t_{i}})^{2}+\sum_{j\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathscr{P})$}}}}\delta_{{\mathscr{A}}_{j}}({\mathrm{z}}),$
(47)

where, for each $j\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathscr{P})$}}}$,
${\mathscr{A}}_{j}\subseteq{\mathbb{R}}^{m}$ is the following set

${\mathscr{A}}_{j}:=\bigg{\\{}(z_{i})_{i=0}^{m-1}\in{\mathbb{R}}^{n}\bigg{|}z_{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}+2j}^{2}+z_{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}+2j+1}^{2}\leq
1-\varepsilon\bigg{\\}}.$ (48)

Also, let $r:{\mathbb{R}}_{+}\to{\mathbb{R}}$ be the increasing function
defined as $r(z)=\lambda z^{2}$. For
$i\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathrm{u})$}}}$ and
$j\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathscr{P})$}}}$, we know that
${\mathrm{L}}^{\\!{\mathrm{u}}}_{t_{i}}({\mathrm{g}})={\langle{\varphi_{i},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$,
${\mathcal{F}}_{\omega_{j}}^{\text{\rm{(r)}}}({\mathrm{g}})={\langle{\varphi_{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}+2j},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$
and
${\mathcal{F}}_{\omega_{j}}^{\text{\rm{(r)}}}({\mathrm{g}})={\langle{\varphi_{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}+2j+1},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$.
Accordingly, we have

${\mathcal{J}}({\mathrm{g}})+\lambda{\mathcal{R}}({\mathrm{g}})=e({\langle{\varphi_{0},{\mathrm{g}}}\rangle},\ldots,{\langle{\varphi_{m-1},{\mathrm{g}}}\rangle})+r(\|{\mathrm{g}}\|).$

From Theorem 11, it follows that
$\min_{{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}}{\mathcal{J}}({\mathrm{g}})+\lambda{\mathcal{R}}({\mathrm{g}})$
has a unique solution denoted by ${\mathrm{g}}^{(\star,\varepsilon)}$.
Therefore, due to Theorem 12, it has a solution which belongs to
${\mathscr{W}}=\ \mathrm{span}\\{\varphi_{i}\\}_{i=0}^{m-1}$. From the
uniqueness of the solution of (29), it follows that
${\mathrm{g}}^{(\star,\varepsilon)}$ belongs to ${\mathscr{W}}$. Therefore, in
order to find ${\mathrm{g}}^{(\star,\varepsilon)}$, we need to obtain the
corresponding coefficients in the linear representation (44). Accordingly, we
replace ${\mathrm{g}}$ in
$\min_{{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}}{\mathcal{J}}({\mathrm{g}})+\lambda{\mathcal{R}}({\mathrm{g}})$,
or equivalently in (29), by ${\mathrm{g}}=\sum_{i=0}^{m-1}x_{i}\varphi_{i}$,
and solve the problem for
${\mathrm{x}}:=[x_{i}]_{i=0}^{m-1}\in{\mathbb{R}}^{m}$. Due to the linearity
of inner product and the definition of matrix $\Phi$, one has

$\begin{split}{\mathcal{R}}({\mathrm{g}})={\langle{{\mathrm{g}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}=\sum_{i=0}^{m-1}\sum_{j=0}^{m-1}x_{i}x_{j}{\langle{\varphi_{i},\varphi_{j}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}={\mathrm{x}}^{\mathsf{\scriptscriptstyle
T}}\Phi{\mathrm{x}}.\end{split}$ (49)

Also, for each $i\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathrm{u})$}}}$, due to
the definition of ${\mathrm{a}}_{i}$, one can see that

${\mathrm{L}}^{\\!{\mathrm{u}}}_{t_{i}}({\mathrm{g}})\\!=\\!\Big{\langle}{\varphi_{i}},{\sum_{k=0}^{m-1}\\!x_{k}\varphi_{k}}\Big{\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}\\!\\!\\!=\\!\sum_{k=0}^{m-1}\\!x_{k}{\langle{\varphi_{i},\varphi_{k}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}\\!=\\!{\mathrm{a}}_{i}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}}.$ (50)

Similarly, from the definition of vectors ${\mathrm{b}}_{j}$ and
${\mathrm{c}}_{j}$, it follows that
${\mathcal{F}}_{\omega_{j}}^{\text{\rm{(r)}}}({\mathrm{g}})={\mathrm{b}}_{j}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}}$ and
${\mathcal{F}}_{\omega_{j}}^{\text{\rm{(i)}}}({\mathrm{g}})={\mathrm{c}}_{j}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}}$, for each
$j\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathscr{P})$}}}$. Therefore, from
(46), one has

${\mathcal{J}}({\mathrm{g}})={\mathcal{J}}\left(\sum_{k=0}^{m-1}x_{k}\varphi_{k}\right)=\sum_{i=0}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1}({\mathrm{a}}_{i}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}}-y_{i})^{2}+\sum_{j=0}^{n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}}\delta_{{\mathscr{B}}_{j}}({\mathrm{x}}),$

where, for each $j\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathscr{P})$}}}$,
${\mathscr{B}}_{j}\subseteq{\mathbb{R}}^{m}$ is the set defined as

$\\!\\!{\mathscr{B}}_{j}:=\bigg{\\{}{\mathrm{x}}\\!=\\!(x_{i})_{i=0}^{m-1}\in{\mathbb{R}}^{m}\
\\!\bigg{|}\ \\!({\mathrm{b}}_{j}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}})^{2}+({\mathrm{c}}_{j}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}})^{2}\leq 1-\varepsilon\bigg{\\}}.$

Accordingly, solving
$\min_{{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}}{\mathcal{J}}({\mathrm{g}})+\lambda{\mathcal{R}}({\mathrm{g}})$
reduces to

$\min_{{\mathrm{x}}\in{\mathbb{R}}^{m}}\sum_{i=0}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1}({\mathrm{a}}_{i}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}}-y_{i})^{2}+\sum_{j=0}^{n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}}\delta_{{\mathscr{B}}_{j}}({\mathrm{x}})+\lambda{\mathrm{x}}^{\mathsf{\scriptscriptstyle
T}}\Phi{\mathrm{x}},$ (51)

which is equivalent to (45). Since ${\mathrm{x}}^{\mathsf{\scriptscriptstyle
T}}\Phi{\mathrm{x}}=\|\sum_{i=0}^{m-1}x_{i}\varphi_{i}\|^{2}$ and
$\\{\varphi_{i}\\}_{i=0}^{m-1}$ are linearly independent, $\Phi$ is a positive
definite matrix. Therefore, the cost function in (45) is strongly convex.
Also, for each $j=0,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}$, we have

$({\mathrm{b}}_{j}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}})^{2}+({\mathrm{c}}_{j}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}})^{2}={\mathrm{x}}^{\mathsf{\scriptscriptstyle
T}}({\mathrm{b}}_{j}{\mathrm{b}}_{j}^{\mathsf{\scriptscriptstyle
T}}+{\mathrm{c}}_{j}{\mathrm{c}}_{j}^{\mathsf{\scriptscriptstyle
T}}){\mathrm{x}},$ (52)

and ${\mathrm{b}}_{j}{\mathrm{b}}_{j}^{\mathsf{\scriptscriptstyle
T}}+{\mathrm{c}}_{j}{\mathrm{c}}_{j}^{\mathsf{\scriptscriptstyle T}}$ is a
positive semi-definite matrix. Consequently, the feasible set in (45) is a
convex and closed set. Moreover, since $\varepsilon<1$, we know that
${\mathrm{g}}=\mathbf{0}$ is a feasible point of (45). Therefore, the
optimization problem (45) is a convex program with a unique solution. This
concludes the proof. ∎

Before proceeding further, we present a corollary which is useful in the
implementation of the proposed approach. Let
$\overline{{\mathscr{P}}}:=\\{\overline{\omega}_{i}\\}_{i=0}^{n_{{{\scalebox{0.5}{$\overline{{\mathscr{P}}}$}}}}}\subseteq{\mathscr{P}}$,
${\mathcal{I}}^{{\scalebox{0.63}{$(\overline{\mathscr{P}})$}}}:=\\{0,1,\ldots,n_{{{\scalebox{0.5}{$\overline{{\mathscr{P}}}$}}}}\\}$
and
$\overline{m}:=n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}+2n_{{{\scalebox{0.5}{$\overline{{\mathscr{P}}}$}}}}+2$.
Analogously to $\Phi$ and $\\{\varphi_{i}\\}_{i=0}^{m-1}$, we define
$\overline{\Phi}$ and $\\{\overline{\varphi}_{i}\\}_{i=0}^{\overline{m}-1}$.
Also, for $i\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathrm{u})$}}}$ and
$j\in{\mathcal{I}}^{{\scalebox{0.63}{$(\overline{\mathscr{P}})$}}}$, let
$\overline{{\mathrm{a}}}_{i}$, $\overline{{\mathrm{b}}}_{j}$ and
$\overline{{\mathrm{c}}}_{j}$ be defined in the same manner as
${\mathrm{a}}_{i}$, ${\mathrm{b}}_{j}$ and ${\mathrm{c}}_{j}$, respectively.

######  Corollary 14.

Consider the following optimization problem

$\begin{array}[]{cl}\operatornamewithlimits{min}\limits_{{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}}&\operatornamewithlimits{\sum}\limits_{t\in{\mathscr{T}}}({\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}})-y_{t})^{2}+\lambda\
\\!\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2},\\\
\mathrm{s.t.}&|{\mathcal{F}}_{\omega}({\mathrm{g}})|^{2}\leq
1-\varepsilon,\quad\forall\omega\in\overline{{\mathscr{P}}}.\end{array}$ (53)

Then, (53) has a unique solution denoted by
$\overline{{\mathrm{g}}}^{(\star,\varepsilon)}$. This solution satisfies (35),
and admits a parametric form as
$\overline{{\mathrm{g}}}^{(\star,\varepsilon)}=\sum_{i=0}^{\overline{m}-1}x_{i}\overline{\varphi}_{i}$
where ${\mathrm{x}}=[x_{i}]_{i=0}^{\overline{m}-1}$ is the solution of the
following convex program

$\begin{array}[]{cl}\operatornamewithlimits{min}\limits_{{\mathrm{x}}\in{\mathbb{R}}^{\overline{m}}}&\sum_{i\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathrm{u})$}}}}(\overline{{\mathrm{a}}}_{i}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}}-y_{i})^{2}+\lambda{\mathrm{x}}^{\mathsf{\scriptscriptstyle
T}}\overline{\Phi}{\mathrm{x}}\\\
\mathrm{s.t.}&(\overline{{\mathrm{b}}}_{j}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}})^{2}+(\overline{{\mathrm{c}}}_{j}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}})^{2}\leq 1-\varepsilon,\quad\forall
j\in{\mathcal{I}}^{{\scalebox{0.63}{$(\overline{\mathscr{P}})$}}}.\end{array}$
(54)

Moreover, if
$|{\mathcal{F}}_{\omega}(\overline{{\mathrm{g}}}^{(\star,\varepsilon)})|^{2}\leq
1-\varepsilon$, for all $\omega\in{\mathscr{P}}$, then
$\overline{{\mathrm{g}}}^{(\star,\varepsilon)}$ coincides with
${\mathrm{g}}^{(\star,\varepsilon)}$.

###### Proof.

Using similar lines of argument to the proof of Theorem 11, one can show the
existence and uniqueness of the solution of (53). The parametric form of
$\overline{{\mathrm{g}}}^{(\star,\varepsilon)}$ and the fact that
${\mathrm{x}}$ is the solution of (54) can be concluded based on a proof
similar to the proof of Theorem 13.

Since, for each $\omega\in{\mathscr{P}}$, one has
$|{\mathcal{F}}_{\omega}(\overline{{\mathrm{g}}}^{(\star,\varepsilon)})|^{2}\leq
1-\varepsilon$, we know that $\overline{{\mathrm{g}}}^{(\star,\varepsilon)}$
is feasible for optimization (29). Therefore, as
${\mathrm{g}}^{(\star,\varepsilon)}$ is the optimal solution of (29), it
follows that
${\mathcal{E}}_{{\mathscr{D}}}({\mathrm{g}}^{(\star,\varepsilon)})+\lambda{\mathcal{R}}({\mathrm{g}}^{(\star,\varepsilon)})\leq{\mathcal{E}}_{{\mathscr{D}}}(\overline{{\mathrm{g}}}^{(\star,\varepsilon)})+\lambda{\mathcal{R}}(\overline{{\mathrm{g}}}^{(\star,\varepsilon)})$,
where ${\mathcal{E}}_{{\mathscr{D}}}$ is defined in (8). Also, we know that
the feasible set in (29) is a subset of feasible set of (53). Accordingly,
since $\overline{{\mathrm{g}}}^{(\star,\varepsilon)}$ is the optimal solution
for (53), we have
${\mathcal{E}}_{{\mathscr{D}}}(\overline{{\mathrm{g}}}^{(\star,\varepsilon)})+\lambda{\mathcal{R}}(\overline{{\mathrm{g}}}^{(\star,\varepsilon)})\leq{\mathcal{E}}_{{\mathscr{D}}}({\mathrm{g}}^{(\star,\varepsilon)})+\lambda{\mathcal{R}}({\mathrm{g}}^{(\star,\varepsilon)})$.
Therefore, $\overline{{\mathrm{g}}}^{(\star,\varepsilon)}$ is an optimizer of
(29). Consequently, from the uniqueness of the solution of (29), we have
$\overline{{\mathrm{g}}}^{(\star,\varepsilon)}={\mathrm{g}}^{(\star,\varepsilon)}$,
concluding the proof. ∎

Theorem 13 introduces the finite-dimensional convex program (45) as a
tractable problem equivalent to optimization (29). The solution of (45) is
discussed in the next section. However, we should first verify that solving
optimization problem (29), for small enough $\varepsilon>0$, provides a close
approximation to the solution of the main optimization problem (16). This is
addressed by the next theorem.

######  Theorem 15.

Let
${\mathrm{L}}^{\\!{\mathrm{u}}}:{\mathcal{H}}_{\mathbbm{k}}\to{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}$
be a linear operator defined as
${\mathrm{L}}^{\\!{\mathrm{u}}}({\mathrm{g}}):=[{\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}})]_{t\in{\mathscr{T}}}$,
for any ${\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}$. Then, we have

$\|{\mathrm{g}}^{(\star,\varepsilon)}-{\mathrm{g}}^{\star}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}\leq
4\varepsilon(\|{\mathrm{L}}^{\\!{\mathrm{u}}}\|_{{\mathcal{L}}({\mathcal{H}}_{\mathbbm{k}},{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}})}^{2}+1)^{\frac{1}{2}}\|{\mathrm{y}}\|^{2}.$
(55)

where ${\mathrm{y}}$ is the vector defined as
${\mathrm{y}}=[y_{t}]_{t\in{\mathscr{T}}}$. Moreover, we have
$\operatornamewithlimits{lim}\limits_{\varepsilon\to\mathbf{0}}{\mathrm{g}}^{(\star,\varepsilon)}={\mathrm{g}}^{\star}$
in ${\mathcal{H}}_{\mathbbm{k}}$. Furthermore, if
$\sup_{t\in{\mathbb{T}}}\mathbbm{k}(t,t)^{\frac{1}{2}}$ is finite, one has
$g_{t}^{(\star,\varepsilon)}\operatornamewithlimits{\longrightarrow}\limits^{\varepsilon\to
0}g_{t}^{\star}$, uniformly in ${\mathbb{T}}$.

###### Proof.

See Appendix A-F. ∎

######  Remark 2.

The property $\sup_{t\in{\mathbb{T}}}\mathbbm{k}(t,t)^{\frac{1}{2}}<\infty$ is
satisfied by the TC kernel and other common kernels in the literature.

##  VI Optimization Algorithm

Due to Theorem 13, the problem to be solved is

$\begin{array}[]{cl}\operatornamewithlimits{min}\limits_{{\mathrm{x}}\in{\mathbb{R}}^{m}}&\|{\mathrm{A}}{\mathrm{x}}-{\mathrm{y}}\|^{2}+\lambda{\mathrm{x}}^{\mathsf{\scriptscriptstyle
T}}\Phi{\mathrm{x}}\\\ \mathrm{s.t.}&{\mathrm{x}}^{\mathsf{\scriptscriptstyle
T}}({\mathrm{b}}_{j}{\mathrm{b}}_{j}^{\mathsf{\scriptscriptstyle
T}}+{\mathrm{c}}_{j}{\mathrm{c}}_{j}^{\mathsf{\scriptscriptstyle
T}}){\mathrm{x}}\leq 1-\varepsilon,\quad\forall
j\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathscr{P})$}}},\end{array}$ (56)

where
${\mathrm{A}}:=[{\mathrm{a}}_{0},\ldots,{\mathrm{a}}_{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1}]^{\mathsf{\scriptscriptstyle
T}}$ and ${\mathrm{y}}:=[y_{t}]_{t\in{\mathscr{T}}}$. From the definition of
the matrix ${\mathrm{A}}$, we know that ${\mathrm{A}}$ is the first
$n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}$ rows of $\Phi$. The vectors
${\mathrm{b}}_{j}$ and ${\mathrm{c}}_{j}$ are respectively the
$(n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}+2j+1)^{\text{\tiny{th}}}$ and the
$(n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}+2j+2)^{\text{\tiny{th}}}$ columns
of $\Phi$, for $j\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathscr{P})$}}}$.
Therefore, matrix $\Phi$ is sufficient for establishing the optimization
problem (56). To this end, for each $i,j\in{\mathcal{I}}$, we need to obtain
the value of
${\langle{\varphi_{i},\varphi_{j}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$,
which demands calculating an infinite double summation, when
${\mathbb{T}}={\mathbb{Z}}_{+}$, or an improper double integral, when
${\mathbb{T}}={\mathbb{R}}_{+}$. In general, to obtain these values one should
employ numerical methods which are essentially inexact and also
computationally demanding. On the other hand, one can derive these values
analytically in specific but rather general situations, e.g., when the TC
kernel is employed (see Appendix A-G for more details). Note that (56) is a
convex quadratically constrained quadratic program program, for which there
exist various efficient methods [58]. For example, we can utilize methods
which are based on log-barrier functions [58] and adapt them suitably to this
application (see Appendix A-H for more details).

In many situations, such as in the example given in Section III, the
constraint
$|G(\mathrm{j}\omega)|=|{\mathcal{F}}_{\omega}({\mathrm{g}})|^{2}\leq
1-\varepsilon$ is not binding on the whole frequency range
$\Omega_{{\mathbb{T}}}$. Accordingly, it is not required to impose this
constraint for each $\omega\in{\mathscr{P}}$. Motivated by this fact and
Corollary 14, we can introduce an iterative scheme. More precisely, let
$\overline{{\mathscr{P}}}_{0}:=\emptyset$, and at iteration $k$, let
$\overline{{\mathscr{P}}}_{k}$ be a given subset of ${\mathscr{P}}$. Consider
the optimization problem (29) where only the constraints corresponding to the
frequencies in $\overline{{\mathscr{P}}}_{k}$ are imposed, i.e., we have the
following program

$\begin{array}[]{cl}\operatornamewithlimits{min}\limits_{{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}}&\operatornamewithlimits{\sum}\limits_{t\in{\mathscr{T}}}({\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}})-y_{t})^{2}+\lambda\
\\!\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2},\\\
\mathrm{s.t.}&|{\mathcal{F}}_{\omega}({\mathrm{g}})|^{2}\leq
1-\varepsilon,\quad\forall\omega\in\overline{{\mathscr{P}}}_{k}.\end{array}$
(57)

Due to Corollary 14, we know that (57) has a unique solution, denoted by
${\mathrm{g}}_{k}$, which can be obtained by solving an equivalent finite-
dimensional convex program as in (54). Given ${\mathrm{g}}_{k}$, one can check
whether the constraint $|{\mathcal{F}}_{\omega}({\mathrm{g}}_{k})|^{2}\leq
1-\varepsilon$ is violated on the remaining frequencies in ${\mathscr{P}}$.
Accordingly, one can obtain the following set

$\Delta_{k}{\mathscr{P}}:=\Big{\\{}\omega\in{\mathscr{P}}\backslash\overline{{\mathscr{P}}}_{k}\
\Big{|}\
|{\mathcal{F}}_{\omega}({\mathrm{g}}_{k})|^{2}>1-\varepsilon\Big{\\}},$ (58)

Subsequently, we update the frequency set as
$\overline{{\mathscr{P}}}_{k+1}:=\overline{{\mathscr{P}}}_{k}\cup\Delta_{k}{\mathscr{P}}$,
and proceed to the iteration $k+1$. The iterative scheme stops when
$\Delta_{k}{\mathscr{P}}=\emptyset$. This happens either when
$\overline{{\mathscr{P}}}_{k}={\mathscr{P}}$ or the solution
${\mathrm{g}}_{k}$ satisfies the constraint
$|{\mathcal{F}}_{\omega}({\mathrm{g}}_{k})|^{2}\leq 1-\varepsilon$ for all
$\omega\in{\mathscr{P}}$. Consequently, due to Corollary 14, we have
${\mathrm{g}}_{k}={\mathrm{g}}^{(\star,\varepsilon)}$ when the stopping
condition is met. It is noteworthy that given the matrix $\Phi$, one can
extract $\overline{\Phi}_{k}$ as a sub-matrix of $\Phi$, and subsequently, the
corresponding vectors $\overline{{\mathrm{a}}}_{i}$,
$\overline{{\mathrm{b}}}_{j}$ and $\overline{{\mathrm{c}}}_{j}$ in (54) are
obtained as columns of $\overline{\Phi}_{k}$. This fact improves the
computational tractability of the proposed approach. Algorithm 1 summarizes
this iterative scheme.

Algorithm 1 Kernel-Based Identification with Frequency Domain Side-Information
$\|G^{({\mathcal{S}})}\|_{{\mathcal{H}}_{\infty}}\leq 1$

1:input: data set ${\mathscr{D}}$, partition set ${\mathscr{P}}$, kernel
$\mathbbm{k}$, regularization weight $\lambda$, tolerance
$\varepsilon\in(0,1)$, matrix $\Phi$, unconstrained estimation
${\mathrm{g}}_{0}$

2:$k\leftarrow 0$ and ${\mathrm{g}}_{k}\leftarrow{\mathrm{g}}_{0}$.

3:$\overline{{\mathscr{P}}}_{k}\leftarrow\emptyset$ and get
$\Delta_{k}{\mathscr{P}}$ due to (58).

4:while $\Delta_{k}{\mathscr{P}}\neq\emptyset$ do

5:
$\overline{{\mathscr{P}}}\leftarrow\overline{{\mathscr{P}}}_{k}\cup\Delta{\mathscr{P}}_{k}$.

6: update $\overline{\Phi}$, $\overline{{\mathrm{a}}}_{i}$ for
$i\in{\mathcal{I}}^{{\scalebox{0.63}{$(\mathrm{u})$}}}$,
$\overline{{\mathrm{b}}}_{j}$ and $\overline{{\mathrm{c}}}_{j}$, for
$j\in{\mathcal{I}}^{{\scalebox{0.63}{$(\overline{\mathscr{P}})$}}}$.

7: solve optimization problem (54) and
${\mathrm{g}}_{k}\leftarrow\sum_{i}x_{i}\overline{\varphi}_{i}$.

8: ${\mathscr{P}}_{k}\leftarrow\overline{{\mathscr{P}}}$ and get
$\Delta_{k}{\mathscr{P}}$ due to (58).

9:end

10:Output: ${\mathrm{g}}^{(\star,\varepsilon)}$

This procedure generates a strictly increasing sequence of sets
$\emptyset=\overline{{\mathscr{P}}}_{0}\subset\overline{{\mathscr{P}}}_{1}\subset\overline{{\mathscr{P}}}_{2}\subset\ldots\subset{\mathscr{P}}$.
Since ${\mathscr{P}}$ is a finite set, this sequence is also finite and
therefore, Algorithm 1 stops after finite number of iterations.

##  VII Numerical Examples

In this section, we provide numerical examples demonstrating the performance
of the proposed scheme in Algorithm 1.

######  Example 1.

We consider the settings of the example given in Section III. We set
$\varepsilon=10^{-5}$ and take partition set
${\mathscr{P}}=\\{\omega_{i}|i=0,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}\\}$
for interval $[0,\pi]$ such that
$\omega_{i}=\frac{\pi}{n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}}i$, for
$i=0,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}$, with
$n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}=3141$. We employ the TC kernel and
apply Algorithm 1. In order to tune the hyperparameters $\alpha$ and
$\lambda$, we utilize Bayesian optimization with a lower-confidence-bound
acquisition function [59] to find the hyperparameters minimizing an objective
function defined based on a cross-validation procedure. More precisely, for a
choice of hyperparameters, the first $100$ points of ${\mathscr{D}}$ are used
for training the model and following this, the cost function to be optimized
by Bayesian optimization is defined as the validation error calculated using
the remaining points of ${\mathscr{D}}$ More details are provide in Appendix
A-I. Starting from $\overline{{\mathscr{P}}}_{0}=\emptyset$, we estimate
${\mathrm{g}}_{0}$ as the solution of the unconstrained problem. This solution
violates the constraints on
$\overline{{\mathscr{P}}}_{1}:=\\{\omega_{i}|i=36,\ldots,140\\}$. Proceeding
from this partition set, we obtain ${\mathrm{g}}_{1}$ which satisfies the
constraints for all of the frequencies in ${\mathscr{P}}$. Therefore,
$\Delta_{1}{\mathscr{P}}=\emptyset$ and the algorithm terminates with
${\mathrm{g}}^{(\star,\varepsilon)}={\mathrm{g}}_{1}$.

Figure 3 shows the transfer function of $G^{(\star,\varepsilon)}$ along with
the estimated models $\hat{G}_{1}$ and $\hat{G}_{2}$, obtained in Section III,
and also the estimated model $\hat{G}_{3}$ resulted from the method in [26]
which is briefly reviewed later in this section. One can see that the result
of proposed scheme satisfies the side-information constraint, and also fits
better to the true transfer function
$G^{{\scalebox{0.5}{$({\mathcal{S}})$}}}$. For quantitative evaluation and
comparison of the estimated impulse response, we use _coefficient of
determination_ , also known as _R-squared_ , which is defined as following

$\mathrm{fit}({\mathrm{g}})=100\times\left(1-\frac{\|{\mathrm{g}}-{\mathrm{g}}^{{\scalebox{0.5}{$({\mathcal{S}})$}}}\|_{2}}{\|{\mathrm{g}}^{{\scalebox{0.5}{$({\mathcal{S}})$}}}\|_{2}}\right),$
(59)

where ${\mathrm{g}}$ is the estimated impulse response. Here, we have
$\mathrm{fit}(\hat{{\mathrm{g}}}_{1})=87.13$%,
$\mathrm{fit}(\hat{{\mathrm{g}}}_{2})=87.15$%, and
$\mathrm{fit}(\hat{{\mathrm{g}}}_{3})=79.08$%, where $\hat{{\mathrm{g}}}_{i}$
is the impulse responses corresponding to $\hat{G}_{i}$, for $i=1,2,3$. On the
other hand, we have $\mathrm{fit}({\mathrm{g}}^{(\star,\varepsilon)})=90.84$%
which shows an improvement in the estimation as well as satisfying the given
side-information. Note that while $\hat{G}_{3}$ also satisfies the side-
information constraint, the proposed method performs significantly better in
terms of fitting performance.  $\triangle$

Figure 3: The transfer function of system (4), the model estimated using the
proposed approach, $G^{(\star,\varepsilon)}$, and the estimated models
$\hat{G}_{1}$, $\hat{G}_{2}$ and $\hat{G}_{3}$.

The method in [26] is a variant of the subspace identification method
incorporating frequency domain side-information, and is denoted by
$\text{FDI}_{\text{sub}}$ in the following. In this approach, initially a
sequence of state variables is estimated, and then, the subspace
identification method is formulated with matrix inequalities, coming from the
Kalman-Yakubovich-Popov (KYP) lemma, imposing the side-information. The
resulting nonlinear program is reduced to a convex one using appropriate
transformations.

######  Example 2.

In this example, we perform numerical experiments to compare the proposed
method with [26]. We generate randomly four sets of $150$ stable systems using
drss Matlab’s function with orders in the range of $\\{10,\ldots,30\\}$ and
poles not larger than $0.98$. The systems are normalized with their
${\mathcal{H}}_{\infty}$-norm, and then, actuated with a realization of
standard random white Gaussian signal
${\mathrm{u}}=(u_{t})_{t=0}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1}$,
i.e., $u_{0},\ldots,u_{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1}$ are i.i.d.
samples of ${\mathcal{N}}(0,1)$. The output of system is corrupted with
additive Gaussian noise. The variance of output noise is chosen such that we
have $10\\!\ $dB, $20\\!\ $dB, $30\\!\ $dB and $40\\!\ $dB signal-to-noise
ratio (SNR) in the respective sets of the systems. Similar to the previous
example, we tune the hyperparameters of both the proposed approach and the
$\text{FDI}_{\text{sub}}$ using Bayesian optimization [59] and cross-
validation. The settings for the proposed approach are similar to ones
employed in Example 1. The box plots of results are shown in Figure 4. While,
we can see that the performance of the both of these approaches improves when
SNR increases, the kernel-based approach significantly outperforms the
subspace-based method $\text{FDI}_{\text{sub}}$.

Figure 4: The performance of $\text{FDI}_{\text{sub}}$ [26] is compared with
the proposed method, $\text{FDI}_{\text{RKHS}}$.

Discussion: The $\text{FDI}_{\text{sub}}$ approach [26] suffers from the well-
known model order selection issue [31]. On the other hand, the proposed scheme
is a derivation of kernel-based regularization methods, and therefore, tuning
the complexity of model is performed by the powerful concept of estimating
continuous regularization hyperparameters rather than picking an integer order
based on a selection rule [32, 31]. Moreover, while the proposed approach
works directly with input-output data, $\text{FDI}_{\text{sub}}$ employs an
estimation of the state trajectory which leads to model estimation prone to
high variance and noisy results.  $\triangle$

The next example considers Problem 1 in the continuous-time case and
demonstrates the role of proposed approach for the continuous-time systems.

######  Example 3.

Consider the _unknown_ continuous-time system ${\mathcal{S}}$ with transfer
function $G^{{\scalebox{0.5}{$({\mathcal{S}})$}}}(s)$ defined [60] as

$\\!\\!\\!\\!G^{{\scalebox{0.5}{$({\mathcal{S}})$}}}(s)\\!=\\!-\frac{2s^{3}\\!+\\!3.6s^{2}\\!+\\!2.095s\\!+\\!0.396s}{0.461s^{4}\\!+\\!2.628s^{3}\\!+\\!4.389s^{2}\\!+\\!2.662s\\!+\\!0.519},\\!\\!$
(60)

with side-information
$\|G^{{\scalebox{0.5}{$({\mathcal{S}})$}}}\|_{{\mathcal{H}}_{\infty}}\leq 1$.
Let the system be initially at rest and actuated with a random switching pulse
signal as shown in Figure 5. Then, the output of system is measure at
$n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}=250$ time instants
$t_{0},\ldots,t_{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1}\in[0,10]$, where
$t_{k}=kT_{\mathrm{s}}+\delta_{k}$, for
$k=0,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1$, with
$T_{\mathrm{s}}=0.04$ and
$\delta_{k}\sim\mathrm{Uniform}([0,T_{\mathrm{s}}])$. The output measurements,
$\\{y_{t_{i}}\\}_{i=0}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1}$, are
subject to additive white Gaussian noise such that the SNR is $20\\!\ $dB. We
interpolate the output measurement values at time instants
$\bar{t}_{k}=(k\\!+\\!1)T_{\mathrm{s}}$, for
$k=0,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1$, using shape-
preserving piecewise cubic interpolation available in Matlab’s function
interp1 and pchip option. The SNR in the interpolated outputs,
$\\{\bar{y}_{t_{i}}\\}_{i=0}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1}$, is
$20.68\\!\ $dB.

We identify the system through _direct_ and _indirect_ approaches. For the
direct approaches, we employ the interpolated data and CONTSID Toolbox [61] to
obtain transfer function estimations $\hat{G}_{1}$ and $\hat{G}_{2}$,
respectively, using tfsrivc and rivc functions with known orders of system. In
indirect approaches, a discrete-time impulse response is estimated using the
interpolated data, and then the continuous-time version is derived by the
shape-preserving piecewise cubic interpolation method. To this end, we use the
subspace method [26] explained in Example 2 and also, the discrete-time
version of the proposed method. Let the corresponding transfer functions be
denoted respectively by $\hat{G}_{\mathrm{sub}}$ and
$G^{(\star,\varepsilon)}_{\mathrm{ind}}$. In obtaining
$G^{(\star,\varepsilon)}_{\mathrm{ind}}$, $\varepsilon$ is set to $10^{-3}$
and the rest of settings are similar to Example 1. Starting from
$\overline{{\mathscr{P}}}_{0}=\emptyset$, the algorithm terminates in the
second iteration with
$\overline{{\mathscr{P}}}_{1}:=\\{\omega_{i}|i=50,\ldots,76\\}$.

In addition to the above methods, we identify the system in a direct approach
based on the proposed algorithm for the case of continuous-time, and using the
original uninterpolated measurement data. Accordingly, we apply Algorithm 1
where $\varepsilon$ is chosen as $10^{-3}$, the partition set is taken as
${\mathscr{P}}=\\{\omega_{i}=10^{-2}i|i=0,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{P}}$}}}=10^{4}\\}$,
a TC kernel is employed, and the hyperparameters are tuned similarly to
Example 1. Initially, we have $\overline{{\mathscr{P}}}_{0}=\emptyset$ where
the resulting solution, ${\mathrm{g}}_{0}$, violates the constraints on
$\Delta{\mathscr{P}}_{0}:=\\{\omega_{i}|i=159,\ldots,205\\}$. The algorithm
terminates in the second iteration. Denote the estimated impulse response and
the corresponding transfer function respectively by
${\mathrm{g}}^{(\star,\varepsilon)}_{\mathrm{dir}}\\!$ and
$G^{(\star,\varepsilon)}_{\mathrm{dir}}\\!$.

Figure 5: The transfer function of system (60), the model estimated using the
proposed approach, $G^{(\star,\varepsilon)}_{\mathrm{dir}}$, and the estimated
models $\hat{G}_{\mathrm{sub}}$ and $G^{(\star,\varepsilon)}_{\mathrm{ind}}$.

In Figure 5, the estimated transfer functions $\hat{G}_{\mathrm{sub}}$,
$G^{(\star,\varepsilon)}_{\mathrm{ind}}$ and
$G^{(\star,\varepsilon)}_{\mathrm{dir}}$ are shown and graphically compared
with $G^{{\scalebox{0.5}{$({\mathcal{S}})$}}}$. The side-information is
satisfied only by $\hat{G}_{\mathrm{sub}}$,
$G^{(\star,\varepsilon)}_{\mathrm{ind}}$ and
$G^{(\star,\varepsilon)}_{\mathrm{ind}}$, and it is violated by $\hat{G}_{1}$
and $\hat{G}_{2}$. Indeed, we have
$\|\hat{G}_{1}\|_{{\mathcal{H}}_{\infty}}=1.11$ and
$\|\hat{G}_{2}\|_{{\mathcal{H}}_{\infty}}=1.17$. To evaluate quantitatively
the estimation results, we employ _R-squared_ metric defined in (59). The
fitting results are $\mathrm{fit}(\hat{{\mathrm{g}}}_{1})=78.93$%,
$\mathrm{fit}(\hat{{\mathrm{g}}}_{2})=63.37$%,
$\mathrm{fit}(\hat{{\mathrm{g}}}_{\mathrm{sub}})=73.76$%,
$\mathrm{fit}({\mathrm{g}}^{(\star,\varepsilon)}_{\mathrm{ind}})=87.15$%, and
$\mathrm{fit}({\mathrm{g}}^{(\star,\varepsilon)}_{\mathrm{dir}})=91.86$%,
where $\hat{{\mathrm{g}}}_{1}$, $\hat{{\mathrm{g}}}_{2}$,
$\hat{{\mathrm{g}}}_{\mathrm{sub}}$ and
${\mathrm{g}}^{(\star,\varepsilon)}_{\mathrm{ind}}$ are the impulse responses
corresponding to $\hat{G}_{1}$, $\hat{G}_{2}$, $\hat{G}_{\mathrm{sub}}$ and
$G^{(\star,\varepsilon)}_{\mathrm{ind}}$, respectively. Therefore, the
proposed method significantly outperforms the other schemes and also satisfies
the side-information.

For further comparison, we perform a Monte Carlo experiment with a set of
$100$ runs and settings similar to the numerical experiment above, where the
SNR level is equal to $20\\!\ $dB and the regular sampling is employed. We
estimate the impulse response of the system in each run using the direct and
indirect implementation of the proposed method and the previously mentioned
subspace identification approach, i.e., we obtain impulse response estimations
${\mathrm{g}}^{(\star,\varepsilon)}_{\mathrm{dir}}$,
${\mathrm{g}}^{(\star,\varepsilon)}_{\mathrm{ind}}$ and
$\hat{{\mathrm{g}}}_{\mathrm{sub}}$. Figure 6 shows the boxplot comparing the
estimation performance of these methods. Also, Table I provides the bias,
variance and mean squared error (MSE) of the estimations.

Figure 6: Box plots of the R-squared metric for the estimation results of the
methods discussed in Example 3.

method $\text{Bias}^{2}$$(\hat{{\mathrm{g}}})$ Var$(\hat{{\mathrm{g}}})$
MSE$(\hat{{\mathrm{g}}})$ $[\times 10^{-6}]$ $[\times 10^{-6}]$ $[\times
10^{-6}]$ $\hat{G}_{\mathrm{sub}}$ 6.39 7.97 14.36
$G^{(\star,\varepsilon)}_{\mathrm{ind}}$ 5.33 1.73 7.07
$G^{(\star,\varepsilon)}_{\mathrm{dir}}$ 2.25 0.63 2.88

Table I:  Bias, variance and MSE comparison between the continuous-time model
estimation methods.

Discussion: For the proposed method, we observe that the direct implementation
significantly outperforms the indirect approach. This observation is in
alignment with the literature on continuous-time system identification [62].
Moreover, one can see that the kernel-based approaches show better estimation
performance compared to the subspace method, as shown in Example 2.
Furthermore, the proposed method can be implemented directly even when in the
case of irregular output sampling. These features highlight the importance of
the developed identification scheme for the estimation of continuous-time
impulse responses.  $\triangle$

##  VIII Conclusion

System identification with frequency domain side-information has been studied
in this paper. We have employed an RKHS framework for both discrete-time and
continuous-time dynamics. The problem is formulated as an infinite-dimensional
optimization problem for fitting a stable impulse response to the data and
satisfying an ${\mathcal{H}}_{\infty}$-norm constraint. The problem is well-
defined and convex with a unique solution. We have proposed a finite-
dimensional convex quadratically constrained quadratic program that tightly
approximates the solution. It is shown that the approximation is uniformly
exact, where this bound can be calculated a priori from the kernel
hyperparameters and the measurement data. The effectiveness of the discussed
method is illustrated by several numerical examples.

##  Appendix A Appendix

###  A-A Proof of Lemma 3

Consider the case ${\mathbb{T}}={\mathbb{R}}_{+}$ and let
${\mathrm{g}}^{{\scalebox{0.5}{$(1)$}}}:=(g^{{\scalebox{0.5}{$(1)$}}}_{t})_{t\in{\mathbb{R}}_{+}},{\mathrm{g}}^{{\scalebox{0.5}{$(2)$}}}:=(g^{{\scalebox{0.5}{$(2)$}}}_{t})_{t\in{\mathbb{R}}_{+}}\in{\mathcal{H}}_{\mathbbm{k}}$
and $\alpha_{1},\alpha_{2}\in{\mathbb{C}}$. Since
${\mathcal{H}}_{\mathbbm{k}}\subseteq{\mathscr{L}}^{1}$, we know that
$\|{\mathrm{g}}^{{\scalebox{0.5}{$(1)$}}}\|_{1}$ and
$\|{\mathrm{g}}^{{\scalebox{0.5}{$(2)$}}}\|_{1}$ are finite. Also, for any
$t\in{\mathbb{R}}_{+}$ and any $\omega\in{\mathbb{R}}_{+}$, from triangle
inequality, we have

$|(\alpha_{1}g^{{\scalebox{0.5}{$(1)$}}}_{t}+\alpha_{2}g^{{\scalebox{0.5}{$(2)$}}}_{t})\
\mathrm{e}^{-\mathrm{j}\omega
t}|\leq|\alpha_{1}||g^{{\scalebox{0.5}{$(1)$}}}_{t}|+|\alpha_{2}||g^{{\scalebox{0.5}{$(2)$}}}_{t}|.$
(61)

Hence,
$\big{(}(\alpha_{1}g^{{\scalebox{0.5}{$(1)$}}}_{t}+\alpha_{2}g^{{\scalebox{0.5}{$(2)$}}}_{t})\
\mathrm{e}^{-\mathrm{j}\omega t}\big{)}_{t\in{\mathbb{R}}_{+}}$ is absolutely
integrable. Therefore, the improper integral

${\mathcal{F}}_{\omega}(\alpha_{1}{\mathrm{g}}^{{\scalebox{0.5}{$(1)$}}}+\alpha_{2}{\mathrm{g}}^{{\scalebox{0.5}{$(2)$}}}):=\int_{{\mathbb{R}}_{+}}(\alpha_{1}g^{{\scalebox{0.5}{$(1)$}}}_{t}+\alpha_{2}g^{{\scalebox{0.5}{$(2)$}}}_{t})\
\mathrm{e}^{-\mathrm{j}\omega t}\mathrm{d}t$ (62)

converges and we have

${\mathcal{F}}_{\omega}(\alpha_{1}{\mathrm{g}}^{{\scalebox{0.5}{$(1)$}}}+\alpha_{2}{\mathrm{g}}^{{\scalebox{0.5}{$(2)$}}}):=\alpha_{1}{\mathcal{F}}_{\omega}({\mathrm{g}}^{{\scalebox{0.5}{$(1)$}}})+\alpha_{2}{\mathcal{F}}_{\omega}({\mathrm{g}}^{{\scalebox{0.5}{$(2)$}}}).$
(63)

Therefore,
${\mathcal{F}}_{\omega}(\alpha_{1}{\mathrm{g}}^{{\scalebox{0.5}{$(1)$}}}+\alpha_{2}{\mathrm{g}}^{{\scalebox{0.5}{$(2)$}}})$
is a well-defined ${\mathbb{C}}$-valued linear map. Moreover, from
$\mathrm{e}^{-\mathrm{j}\omega t}=\cos(\omega t)-\mathrm{j}\sin(\omega t)$,
one can easily see that
${\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})=\mathrm{real}({\mathcal{F}}_{\omega}({\mathrm{g}}))$,
${\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})=\mathrm{imag}({\mathcal{F}}_{\omega}({\mathrm{g}}))$,
and
${\mathcal{F}}_{\omega}({\mathrm{g}})={\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})+\mathrm{j}{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})$,
for any ${\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}$. Let
$\alpha_{1},\alpha_{2}\in{\mathbb{R}}$ and compare the real and imaginary
parts of the left-hand side and the right-hand side of (63). Hence, we have

$\begin{split}{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}(\alpha_{1}{\mathrm{g}}^{{\scalebox{0.5}{$(1)$}}}+\alpha_{2}{\mathrm{g}}^{{\scalebox{0.5}{$(2)$}}})=&\alpha_{1}{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}}^{{\scalebox{0.5}{$(1)$}}})+\alpha_{2}{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}}^{{\scalebox{0.5}{$(2)$}}}),\\\
{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}(\alpha_{1}{\mathrm{g}}^{{\scalebox{0.5}{$(1)$}}}+\alpha_{2}{\mathrm{g}}^{{\scalebox{0.5}{$(2)$}}})=&\alpha_{1}{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}}^{{\scalebox{0.5}{$(1)$}}})+\alpha_{2}{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}}^{{\scalebox{0.5}{$(2)$}}}),\end{split}$
(64)

and also
$|{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}(\alpha_{1}{\mathrm{g}}^{{\scalebox{0.5}{$(1)$}}}+\alpha_{2}{\mathrm{g}}^{{\scalebox{0.5}{$(2)$}}})|,|{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}(\alpha_{1}{\mathrm{g}}^{{\scalebox{0.5}{$(1)$}}}+\alpha_{2}{\mathrm{g}}^{{\scalebox{0.5}{$(2)$}}})|<\infty$.
This shows that ${\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}$ and
${\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}$ are well-defined linear maps.

Since
${\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})=\mathrm{real}({\mathcal{F}}_{\omega}({\mathrm{g}}))$,
from triangle inequality and the definition of ${\mathcal{F}}_{\omega}$, one
can see

$\begin{split}|{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})|&=|\mathrm{real}({\mathcal{F}}_{\omega}({\mathrm{g}}))|\\\
&\leq|{\mathcal{F}}_{\omega}({\mathrm{g}})|\\\
&=\bigg{|}\int_{{\mathbb{R}}_{+}}g_{t}\mathrm{e}^{-\mathrm{j}\omega
t}\mathrm{d}t\bigg{|}\\\
&\leq\int_{{\mathbb{R}}_{+}}|g_{t}|\mathrm{d}t,\end{split}$ (65)

for any ${\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}$. Subsequently, due to the
reproducing property and the Cauchy-Schwartz inequality, we have

$|{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})|\leq\int_{{\mathbb{R}}_{+}}|{\langle{{\mathrm{g}},\mathbbm{k}_{t}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}|\mathrm{d}t\leq\int_{{\mathbb{R}}_{+}}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}\|\mathbbm{k}_{t}\|_{{\mathcal{H}}_{\mathbbm{k}}}\mathrm{d}t.$

From
$\|\mathbbm{k}_{t}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}={\langle{\mathbbm{k}_{t},\mathbbm{k}_{t}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}=\mathbbm{k}(t,t)$
and the definition of $\mu_{0}$, one can see that
$|{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})|\leq\mu_{0}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}$,
and hence, we have

$\|{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}\|_{{\mathcal{L}}({\mathcal{H}}_{\mathbbm{k}},{\mathbb{R}})}:=\\!\\!\sup_{\begin{subarray}{c}{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}\\\
\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}\leq
1\end{subarray}}\\!\\!|{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})|\leq\\!\\!\sup_{\begin{subarray}{c}{\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}\\\
\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}\leq
1\end{subarray}}\\!\\!\mu_{0}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}=\mu_{0}.$

Similarly, one can show that
$\|{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}\|_{{\mathcal{L}}({\mathcal{H}}_{\mathbbm{k}},{\mathbb{R}})}\leq\mu_{0}$.
Therefore, ${\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}$ and
${\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}$ are linear bounded functionals,
i.e.,
${\mathcal{F}}_{\omega}^{\text{\rm{(r)}}},{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}\in{\mathcal{L}}({\mathcal{H}}_{\mathbbm{k}},{\mathbb{R}})$.
Since ${\mathcal{H}}_{\mathbbm{k}}$ is a Hilbert space, due to the Riesz’s
representation theorem, we know that there exist unique
$\varphi_{\omega}^{\text{\rm{(r)}}}:=(\varphi_{\omega,t}^{\text{\rm{(r)}}})_{t\in{\mathbb{R}}_{+}}$
and
$\varphi_{\omega}^{\text{\rm{(i)}}}:=(\varphi_{\omega,t}^{\text{\rm{(i)}}})_{t\in{\mathbb{R}}_{+}}$
in ${\mathcal{H}}_{\mathbbm{k}}$ such that, for any
${\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}$, we have
${\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})={\langle{\varphi_{\omega}^{\text{\rm{(r)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$
and
${\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})={\langle{\varphi_{\omega}^{\text{\rm{(i)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$.
Subsequently, from the reproducing property of kernel, for any
$t\in{\mathbb{R}}_{+}$, we have that

$\begin{array}[]{ll}&\varphi_{\omega,t}^{\text{\rm{(r)}}}={\langle{\varphi_{\omega}^{\text{\rm{(r)}}},\mathbbm{k}_{t}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}={\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}(\mathbbm{k}_{t})=\int_{{\mathbb{R}}_{+}}\\!\\!\\!\mathbbm{k}(t,s)\cos(\omega
s)\mathrm{d}s,\\\
&\varphi_{\omega,t}^{\text{\rm{(i)}}}={\langle{\varphi_{\omega}^{\text{\rm{(i)}}},\mathbbm{k}_{t}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}\\!=\\!{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}(\mathbbm{k}_{t})=-\\!\int_{{\mathbb{R}}_{+}}\\!\\!\\!\mathbbm{k}(t,s)\sin(\omega
s)\mathrm{d}s.\end{array}$

This concludes the proof for the case ${\mathbb{T}}={\mathbb{R}}_{+}$. Similar
lines of argument holds when ${\mathbb{T}}={\mathbb{Z}}_{+}$ and the proof is
concluded. ∎

###  A-B Proof of Lemma 5

Let assume ${\mathbb{T}}={\mathbb{R}}_{+}$. For any
${\mathrm{g}}:=(g_{s})_{s\in{\mathbb{R}}_{+}}\in{\mathcal{H}}_{\mathbbm{k}}$,
from the triangle inequality and the reproducing property, it follows that

$|{\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}})|\leq\int_{{\mathbb{R}}_{+}}\\!\\!|g_{s}||u_{t-s}|\mathrm{d}s=\int_{{\mathbb{R}}_{+}}\\!\\!|{\langle{{\mathrm{g}},\mathbbm{k}_{s}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}||u_{t-s}|\mathrm{d}s.$

Then, due to the Cauchy-Schwartz inequality and since
${\mathrm{u}}\in{\mathscr{L}}^{\infty}$, we have

$\begin{split}|{\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}})|&\leq\int_{{\mathbb{R}}_{+}}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}\mathbbm{k}(s,s)^{\frac{1}{2}}|u_{t-s}|\mathrm{d}s\\\
&\leq\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}\|{\mathrm{u}}\|_{\infty}\int_{{\mathbb{R}}_{+}}\mathbbm{k}(s,s)^{\frac{1}{2}}\mathrm{d}s,\end{split}$
(66)

where we have used the fact that
$\|\mathbbm{k}_{s}\|_{{\mathcal{H}}_{\mathbbm{k}}}=\mathbbm{k}(s,s)^{\frac{1}{2}}$,
for any $s\in{\mathbb{R}}_{+}$. This shows that
$(g_{s}u_{t-s})_{s\in{\mathbb{R}}_{+}}$ is absolutely integrable, for any
${\mathrm{g}}=(g_{t})_{t\in{\mathbb{R}}_{+}}\in{\mathcal{H}}_{\mathbbm{k}}$,
and therefore ${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}$ is a well-defined linear
map. Moreover, from the definition of
$\|{\mathrm{L}}^{\\!{\mathrm{u}}}_{t}\|_{{\mathcal{L}}({\mathcal{H}}_{\mathbbm{k}},{\mathbb{R}})}$
and (66), we have

$\|{\mathrm{L}}^{\\!{\mathrm{u}}}_{t}\|_{{\mathcal{L}}({\mathcal{H}}_{\mathbbm{k}},{\mathbb{R}})}\leq\|{\mathrm{u}}\|_{\infty}\int_{{\mathbb{R}}_{+}}\\!\\!\mathbbm{k}(s,s)^{\frac{1}{2}}\mathrm{d}s=\mu_{0}\|{\mathrm{u}}\|_{\infty}.$

Therefore, ${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}$ is a bounded linear
functional. Subsequently, due to Riesz’s representation theorem, there exists
a unique element
$\varphi_{t}^{\text{\rm{(u)}}}=(\varphi_{t,s}^{\text{\rm{(u)}}})_{s\in{\mathbb{R}}_{+}}\in{\mathcal{H}}_{\mathbbm{k}}$
such that
${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}})={\langle{\varphi_{t}^{\text{\rm{(u)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$,
for any ${\mathrm{g}}\in{\mathcal{H}}_{\mathbbm{k}}$. From reproducing
property of kernel, we know that
$\varphi_{t,s}^{\text{\rm{(u)}}}={\langle{\varphi_{t}^{\text{\rm{(u)}}},\mathbbm{k}_{s}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$,
for any $s\in{\mathbb{R}}_{+}$. Since
${\langle{\varphi_{t}^{\text{\rm{(u)}}},\mathbbm{k}_{s}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}={\mathrm{L}}^{\\!{\mathrm{u}}}_{t}(\mathbbm{k}_{s})$
and due to the definition of ${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}$, we have

$\\!\\!\\!\varphi_{t,s}^{\text{\rm{(u)}}}={\mathrm{L}}^{\\!{\mathrm{u}}}_{t}(\mathbbm{k}_{s})=\\!\\!\int_{{\mathbb{R}}_{+}}\\!\\!\\!\mathbbm{k}_{s}(\tau)u_{t-\tau}\mathrm{d}\tau=\\!\\!\int_{{\mathbb{R}}_{+}}\\!\\!\\!\mathbbm{k}(s,\tau)u_{t-\tau}\mathrm{d}\tau.$

Similar argument holds for ${\mathbb{T}}\\!=\\!{\mathbb{Z}}_{+}$. ∎

###  A-C Proof of Theorem 8

When ${\mathbb{T}}={\mathbb{Z}}_{+}$, from the upper bound of kernel, we have

$\mu_{n}\leq\sum_{t\in{\mathbb{Z}}_{+}}t^{n}\
\\!\mathbbm{k}(t,t)^{\frac{1}{2}}\leq\int_{{\mathbb{R}}_{+}}t^{n}(\gamma\mathrm{e}^{-\beta
t})^{\frac{1}{2}}\mathrm{d}t,$

where the second inequality is due to being
$f(t):=\mathrm{e}^{-\frac{1}{2}\beta t}$ a non-increasing function. This
inequality holds for the case of ${\mathbb{T}}={\mathbb{R}}_{+}$ as well.
Using change of variable $s=\frac{\beta}{2}t$, we have

$\mu_{n}\leq\gamma^{\frac{1}{2}}\frac{2^{n+1}}{\beta^{n+1}}\int_{{\mathbb{R}}_{+}}s^{n}\mathrm{e}^{-s}\mathrm{d}s=\gamma^{\frac{1}{2}}\frac{2^{n+1}}{\beta^{n+1}}\Gamma(n+1),$

where the equality is due to the definition of Gamma function. The claim
follows from $\Gamma(k+1)=k!$, for $k\in{\mathbb{Z}}_{+}$. ∎

###  A-D Proof of Lemma 9

Given
${\mathrm{g}}:=(g_{t})_{t\in{\mathbb{T}}}\in{\mathcal{H}}_{\mathbbm{k}}$,
since ${\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})$ and
${\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})$ are functions of
$\omega$, we define functions
$m_{{\mathrm{g}}}^{(\mathrm{r})},m_{{\mathrm{g}}}^{(\mathrm{i})}:\Omega_{{\mathbb{T}}}\to{\mathbb{R}}$
as

$\begin{split}m_{{\mathrm{g}}}^{(\mathrm{r})}(\omega)&:={\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})={\langle{\varphi_{\omega}^{\text{\rm{(r)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}},\
\\\
m_{{\mathrm{g}}}^{(\mathrm{i})}(\omega)&:={\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})={\langle{\varphi_{\omega}^{\text{\rm{(i)}}},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}},\end{split}$

for any $\omega\in\Omega_{{\mathbb{T}}}$. We know that
$\frac{\mathrm{d}}{\mathrm{d}\omega}\left(g_{t}\cos(\omega
t)\right)=-tg_{t}\sin(\omega t)$ and
$\frac{\mathrm{d}}{\mathrm{d}\omega}\left(g_{t}\sin(\omega
t)\right)=tg_{t}\cos(\omega t)$. From the reproducing property and the Cauchy-
Schwartz inequality, we have

$\begin{split}\Big{|}\frac{\mathrm{d}}{\mathrm{d}\omega}(g_{t}\cos(\omega
t))\Big{|}&=|tg_{t}\sin(\omega t)|\\\ &\leq t|g_{t}|\\\
&=t|{\langle{\mathbbm{k}_{t},{\mathrm{g}}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}|\\\
&\leq
t\|\mathbbm{k}_{t}\|_{{\mathcal{H}}_{\mathbbm{k}}}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}\\\
&=t{\langle{\mathbbm{k}_{t},\mathbbm{k}_{t}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}^{\frac{1}{2}}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}\\\
&=t\mathbbm{k}(t,t)^{\frac{1}{2}}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}.\end{split}$
(67)

Similarly, one can see

$\begin{split}\Big{|}\frac{\mathrm{d}}{\mathrm{d}\omega}(g_{t}\sin(\omega
t))\Big{|}\leq t\\!\
\mathbbm{k}(t,t)^{\frac{1}{2}}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}.\end{split}$
(68)

Due to $\mu_{1}<\infty$ and the Weierstrass M-test theorem, we have the
uniform and absolute convergence for the series
$\sum_{t\in{\mathbb{Z}}_{+}}\frac{\mathrm{d}}{\mathrm{d}\omega}(g_{t}\cos(\omega
t))$ and
$\sum_{t\in{\mathbb{Z}}_{+}}\frac{\mathrm{d}}{\mathrm{d}\omega}(g_{t}\sin(\omega
t))$, when ${\mathbb{T}}={\mathbb{Z}}_{+}$, as well as for the improper
integrals
$\int_{{\mathbb{R}}_{+}}\frac{\mathrm{d}}{\mathrm{d}\omega}(g_{t}\cos(\omega
t))\mathrm{d}t$ and
$\int_{{\mathbb{R}}_{+}}\frac{\mathrm{d}}{\mathrm{d}\omega}(g_{t}\sin(\omega
t))\mathrm{d}t$, when ${\mathbb{T}}={\mathbb{R}}_{+}$. Consequently, from the
definition of
$m_{{\mathrm{g}}}^{(\mathrm{r})},m_{{\mathrm{g}}}^{(\mathrm{i})}$, it follows
that

$\begin{split}&\frac{\mathrm{d}}{\mathrm{d}\omega}m_{{\mathrm{g}}}^{(\mathrm{r})}(\omega)=\frac{\mathrm{d}}{\mathrm{d}\omega}\sum_{t\in{\mathbb{Z}}_{+}}\\!g_{t}\cos(\omega
t)=-\\!\\!\sum_{t\in{\mathbb{Z}}_{+}}\\!tg_{t}\sin(\omega t),\\\
&\frac{\mathrm{d}}{\mathrm{d}\omega}m_{{\mathrm{g}}}^{(\mathrm{i})}(\omega)=\frac{\mathrm{d}}{\mathrm{d}\omega}\sum_{t\in{\mathbb{Z}}_{+}}\\!g_{t}\sin(\omega
t)=\sum_{t\in{\mathbb{Z}}_{+}}\\!tg_{t}\cos(\omega t),\end{split}$

when ${\mathbb{T}}={\mathbb{Z}}_{+}$, and

$\begin{split}&\frac{\mathrm{d}}{\mathrm{d}\omega}m_{{\mathrm{g}}}^{(\mathrm{r})}(\omega)=\frac{\mathrm{d}}{\mathrm{d}\omega}\int_{{\mathbb{R}}_{+}}\\!\\!g_{t}\cos(\omega
t)\mathrm{d}t=-\\!\int_{{\mathbb{R}}_{+}}\\!\\!tg_{t}\sin(\omega
t)\mathrm{d}t,\\\
&\frac{\mathrm{d}}{\mathrm{d}\omega}m_{{\mathrm{g}}}^{(\mathrm{i})}(\omega)=\frac{\mathrm{d}}{\mathrm{d}\omega}\int_{{\mathbb{R}}_{+}}\\!\\!g_{t}\sin(\omega
t)\mathrm{d}t=\int_{{\mathbb{R}}_{+}}\\!\\!tg_{t}\cos(\omega
t)\mathrm{d}t,\end{split}$

when ${\mathbb{T}}={\mathbb{R}}_{+}$. Subsequently, from (67), (68), the
triangle inequality and the definition of $\mu_{1}$ in (17), it follows that

$|\frac{\mathrm{d}}{\mathrm{d}\omega}m_{{\mathrm{g}}}^{(\mathrm{r})}(\omega)|\leq\mu_{1}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}},$
(69)

and

$|\frac{\mathrm{d}}{\mathrm{d}\omega}m_{{\mathrm{g}}}^{(\mathrm{i})}(\omega)|\leq\mu_{1}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}.$
(70)

From
$m_{{\mathrm{g}}}(\omega)=(m_{{\mathrm{g}}}^{(\mathrm{r})}(\omega))^{2}+(m_{{\mathrm{g}}}^{(\mathrm{i})}(\omega))^{2}$,
it follows that $m_{{\mathrm{g}}}$ is a differentiable function with
derivative given as following

$\\!\\!\\!\\!\begin{split}\frac{\mathrm{d}}{\mathrm{d}\omega}&m_{{\mathrm{g}}}(\omega)=\frac{\mathrm{d}}{\mathrm{d}\omega}({\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})^{2}+{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})^{2})\\\
&=2{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})\frac{\mathrm{d}}{\mathrm{d}\omega}m_{{\mathrm{g}}}^{(\mathrm{r})}(\omega)+2{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})\frac{\mathrm{d}}{\mathrm{d}\omega}m_{{\mathrm{g}}}^{(\mathrm{i})}(\omega).\end{split}$
(71)

Due to Lemma 3, we know that
$|{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})|\leq\mu_{0}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}$
and
$|{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})|\leq\mu_{0}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}$.
Subsequently, from (71), we have

$\begin{split}&\left|\frac{\mathrm{d}}{\mathrm{d}\omega}m_{{\mathrm{g}}}(\omega)\right|\\\
&\leq
2\left[\left|{\mathcal{F}}_{\omega}^{\text{\rm{(r)}}}({\mathrm{g}})\right|\\!\
\left|\frac{\mathrm{d}}{\mathrm{d}\omega}m_{{\mathrm{g}}}^{(\mathrm{r})}(\omega)\right|+\left|{\mathcal{F}}_{\omega}^{\text{\rm{(i)}}}({\mathrm{g}})\right|\\!\
\left|\frac{\mathrm{d}}{\mathrm{d}\omega}m_{{\mathrm{g}}}^{(\mathrm{i})}\\!(\omega)\right|\right]\\\
&\leq
2\big{[}\mu_{0}\,\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}\,\mu_{1}\,\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}+\mu_{0}\,\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}\,\mu_{1}\,\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}\big{]}\\\
&=4\mu_{0}\mu_{1}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}.\end{split}$

Now, let $\omega_{1},\omega_{2}\in[0,\pi]$. Without loss of generality, we
assume $\omega_{2}>\omega_{1}$. Consequently, we have

$\begin{split}|m_{{\mathrm{g}}}(\omega_{2})&-m_{{\mathrm{g}}}(\omega_{1})|=|\int_{\omega_{1}}^{\omega_{2}}\frac{\mathrm{d}}{\mathrm{d}\omega}m_{{\mathrm{g}}}(\omega)\mathrm{d}\omega|\\\
&\leq\int_{\omega_{1}}^{\omega_{2}}\left|\frac{\mathrm{d}}{\mathrm{d}\omega}m_{{\mathrm{g}}}(\omega)\right|\mathrm{d}\omega\leq
4\mu_{0}\mu_{1}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}|\omega_{2}-\omega_{1}|.\end{split}$

In other words, we have
$|m_{{\mathrm{g}}}(\omega_{2})-m_{{\mathrm{g}}}(\omega_{1})|\leq
L_{{\mathrm{g}}}|\omega_{2}-\omega_{1}|$ where
$L_{{\mathrm{g}}}=4\mu_{0}\mu_{1}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}$.
∎

###  A-E Proof of Theorem 10

We know that $\max(s,t)=\frac{1}{2}\big{(}s+t+|s-t|\big{)}$. Accordingly,
using change of variable $\tau=s-t$, it follows that

$\int_{{\mathbb{R}}_{+}}\\!\int_{{\mathbb{R}}_{+}}\\!\\!\mathbbm{k}(s,t)\mathrm{e}^{-\mathrm{j}\omega(s-t)}\mathrm{d}s\mathrm{d}t=\\!\int_{{\mathbb{R}}_{+}}\\!\mathrm{e}^{-\beta
t}\\!\\!\int_{-t}^{\infty}\\!\\!\mathrm{e}^{-\beta\tau^{+}-\mathrm{j}\omega\tau}\mathrm{d}\tau\mathrm{d}t,$
(72)

where $\tau^{+}:=\frac{1}{2}\big{(}\tau+|\tau|\big{)}$. Also, we have

$\begin{split}&\\!\\!\int_{-t}^{\infty}\mathrm{e}^{-\beta\tau^{+}-\mathrm{j}\omega\tau}\mathrm{d}\tau=\int_{-t}^{0}\\!\mathrm{e}^{-\mathrm{j}\omega\tau}\mathrm{d}\tau+\int_{0}^{\infty}\\!\mathrm{e}^{-\beta\tau-\mathrm{j}\omega\tau}\mathrm{d}\tau\\\
&=\Big{[}-\frac{1}{\mathrm{j}\omega}+\frac{\mathrm{e}^{\mathrm{j}\omega
t}}{\mathrm{j}\omega}\Big{]}+\frac{1}{\beta+\mathrm{j}\omega}=\frac{\mathrm{e}^{\mathrm{j}\omega
t}}{\mathrm{j}\omega}-\frac{\beta}{\mathrm{j}\omega(\beta+\mathrm{j}\omega)}.\end{split}$
(73)

Subsequently, by replacing the right-hand side of (73) in (72) and simplifying
the integrals, we obtain

$\begin{split}\int_{{\mathbb{R}}_{+}}&\int_{{\mathbb{R}}_{+}}\mathbbm{k}(s,t)\mathrm{e}^{-\mathrm{j}\omega(s-t)}\mathrm{d}s\mathrm{d}t\\\
&=-\frac{1}{\mathrm{j}\omega(-\beta+\mathrm{j}\omega)}-\frac{1}{\mathrm{j}\omega(\beta+\mathrm{j}\omega)}=\frac{2}{\beta^{2}+\omega^{2}}.\end{split}$

This concludes the proof. ∎

###  A-F Proof of Theorem 15

Let
${\mathscr{V}}_{\mathbbm{k}}:={\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}\times{\mathcal{H}}_{\mathbbm{k}}$
be the Hilbert space endowed with the inner product defined as

${\langle{({\mathrm{x}}_{1},{\mathrm{g}}_{1}),({\mathrm{x}}_{2},{\mathrm{g}}_{2})}\rangle}_{{\mathscr{V}}_{\mathbbm{k}}}:={\mathrm{x}}_{1}^{\mathsf{\scriptscriptstyle
T}}{\mathrm{x}}_{2}+\lambda{\langle{{\mathrm{g}}_{1},{\mathrm{g}}_{2}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}},$
(74)

for all $({\mathrm{x}}_{1},{\mathrm{g}}_{1})$ and
$({\mathrm{x}}_{2},{\mathrm{g}}_{2})$ in ${\mathscr{V}}_{\mathbbm{k}}$. Given
$\Omega\subseteq\Omega_{{\mathbb{T}}}$ and $\rho\in{\mathbb{R}}_{+}$, define
set
${\mathscr{U}}_{\mathbbm{k}}(\rho,\Omega)\subseteq{\mathscr{V}}_{\mathbbm{k}}$
as

${\mathscr{U}}_{\mathbbm{k}}(\rho,\Omega):=\Big{\\{}({\mathrm{x}},{\mathrm{g}})\Big{|}{\mathrm{L}}^{\\!{\mathrm{u}}}({\mathrm{g}})-{\mathrm{x}}={\mathrm{y}},|{\mathcal{F}}_{\omega}({\mathrm{g}})|\leq\rho,\forall\omega\in\Omega\Big{\\}},$

where
${\mathrm{y}}:=[y_{t}]_{t\in{\mathscr{T}}}\in{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}$.
Let
${\mathscr{U}}_{\varepsilon}:={\mathscr{U}}_{\mathbbm{k}}(1-\varepsilon,[0,\pi])$
and
${\mathscr{U}}_{{\mathscr{P}}}:={\mathscr{U}}_{\mathbbm{k}}(1-\varepsilon,{\mathscr{P}})$.
Accordingly, replacing $\eta$ with $\rho$, (24) is equivalent to the following
optimization problem

$\operatornamewithlimits{min}\limits_{({\mathrm{x}},{\mathrm{g}})\in{\mathscr{U}}_{\mathbbm{k}}(\rho,\Omega)}\|({\mathrm{x}},{\mathrm{g}})\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2},$
(75)

and therefore, it is a convex optimization with the same unique solution. Let
${\mathrm{v}}^{(\varepsilon)}:=({\mathrm{x}}^{(\varepsilon)},{\mathrm{g}}^{(\varepsilon)})$
be the solution for (75) for ${\mathscr{U}}_{\varepsilon}$ for
$\varepsilon\in[0,1)$. When $\varepsilon=0$, we simply write
${\mathrm{v}}^{\star}:=({\mathrm{x}}^{\star},{\mathrm{g}}^{\star})$. Also, let
${\mathrm{v}}^{(\star,\varepsilon)}:=({\mathrm{x}}^{(\star,\varepsilon)},{\mathrm{g}}^{(\star,\varepsilon)})$
is the solution of (75) for ${\mathscr{U}}_{{\mathscr{P}}}$. These notations
is consistent with our previous ones due to the equivalency of (24) and (75)
and the uniqueness of the solution. One can easily see that
${\mathscr{U}}_{\varepsilon}\subseteq{\mathscr{U}}_{{\mathscr{P}}}$. Also,
based on an argument similar to those provided in the proof of Theorem 11, we
know that ${\mathscr{U}}_{{\mathscr{P}}}\subseteq{\mathscr{U}}_{0}$.
Accordingly, we have

$\begin{split}\sup_{{\mathrm{w}}\in{\mathscr{U}}_{\mathscr{P}}}\min_{{\mathrm{v}}\in{\mathscr{U}}_{\varepsilon}}\|{\mathrm{v}}-{\mathrm{w}}\|_{{\mathscr{V}}_{\mathbbm{k}}}&\leq\sup_{{\mathrm{w}}\in{\mathscr{U}}_{0}}\min_{{\mathrm{v}}\in{\mathscr{U}}_{\varepsilon}}\|{\mathrm{v}}-{\mathrm{w}}\|_{{\mathscr{V}}_{\mathbbm{k}}},\\\
\sup_{{\mathrm{w}}\in{\mathscr{U}}_{0}}\min_{{\mathrm{v}}\in{\mathscr{U}}_{\mathscr{P}}}\|{\mathrm{v}}-{\mathrm{w}}\|_{{\mathscr{V}}_{\mathbbm{k}}}&\leq\sup_{{\mathrm{w}}\in{\mathscr{U}}_{0}}\min_{{\mathrm{v}}\in{\mathscr{U}}_{\varepsilon}}\|{\mathrm{v}}-{\mathrm{w}}\|_{{\mathscr{V}}_{\mathbbm{k}}}.\end{split}$

Let ${\mathrm{v}}_{1}\in{\mathscr{V}}_{\mathbbm{k}}$ be defined as

${\mathrm{v}}_{1}:=\mathrm{proj}_{{\mathscr{U}}_{\varepsilon}}({\mathrm{v}}^{\star})={\mathrm{argmin}}_{{\mathrm{v}}\in{\mathscr{U}}_{\varepsilon}}\|{\mathrm{v}}-{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}.$
(76)

Since ${\mathrm{v}}_{1}\in{\mathscr{U}}_{\varepsilon}$ and
${\mathscr{U}}_{\varepsilon}\subseteq{\mathscr{U}}_{0}$, due to the definition
of ${\mathrm{v}}^{(\varepsilon)}$, we know that
$\|{\mathrm{v}}^{(\varepsilon)}\|_{{\mathscr{V}}_{\mathbbm{k}}}\leq\|{\mathrm{v}}_{1}\|_{{\mathscr{V}}_{\mathbbm{k}}}$.
Therefore, from triangle inequality and (76), we have

$\begin{split}\|{\mathrm{v}}^{(\varepsilon)}\|_{{\mathscr{V}}_{\mathbbm{k}}}&\leq\|{\mathrm{v}}_{1}\|_{{\mathscr{V}}_{\mathbbm{k}}}\leq\|{\mathrm{v}}_{1}-{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}+\|{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}\\\
&=\min_{{\mathrm{v}}\in{\mathscr{U}}_{\varepsilon}}\|{\mathrm{v}}-{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}+\|{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}\\\
&\leq\sup_{{\mathrm{w}}\in{\mathscr{U}}_{0}}\min_{{\mathrm{v}}\in{\mathscr{U}}_{\varepsilon}}\|{\mathrm{v}}-{\mathrm{w}}\|_{{\mathscr{V}}_{\mathbbm{k}}}+\|{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}.\end{split}$

Consequently, it follows that

$0\leq\|{\mathrm{v}}^{(\varepsilon)}\|_{{\mathscr{V}}_{\mathbbm{k}}}-\|{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}\leq\sup_{{\mathrm{w}}\in{\mathscr{U}}_{0}}\min_{{\mathrm{v}}\in{\mathscr{U}}_{\varepsilon}}\|{\mathrm{v}}-{\mathrm{w}}\|_{{\mathscr{V}}_{\mathbbm{k}}}.$

Due to the convexity of ${\mathscr{U}}_{0}$ and
${\mathscr{U}}_{\varepsilon}\subseteq{\mathscr{U}}_{0}$, we know that
$\frac{1}{2}({\mathrm{v}}^{(\varepsilon)}+{\mathrm{v}}^{\star})\in{\mathscr{U}}_{0}$,
and therefore, we have

$\|{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}\leq\|\frac{1}{2}({\mathrm{v}}^{(\varepsilon)}+{\mathrm{v}}^{\star})\|_{{\mathscr{V}}_{\mathbbm{k}}}\
=\frac{1}{2}\|{\mathrm{v}}^{(\varepsilon)}+{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}.$
(77)

Subsequently, one can see that

$\begin{split}\|{\mathrm{v}}^{(\varepsilon)}&-{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}=2\|{\mathrm{v}}^{(\varepsilon)}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}+2\|{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}-\|{\mathrm{v}}^{(\varepsilon)}+{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}\\\
&\leq
2\|{\mathrm{v}}^{(\varepsilon)}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}+2\|{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}-4\|{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}\\\
&=2\|{\mathrm{v}}^{(\varepsilon)}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}-2\|{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}\\\
&=2(\|{\mathrm{v}}^{(\star,\varepsilon)}\|_{{\mathscr{V}}_{\mathbbm{k}}}-\|{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}})(\|{\mathrm{v}}^{(\star,\varepsilon)}\|_{{\mathscr{V}}_{\mathbbm{k}}}+\|{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}})\\\
&\leq
2(\sup_{{\mathrm{w}}\in{\mathscr{U}}_{0}}\min_{{\mathrm{v}}\in{\mathscr{U}}_{\varepsilon}}\|{\mathrm{v}}-{\mathrm{w}}\|_{{\mathscr{V}}_{\mathbbm{k}}})(\|{\mathrm{v}}^{(\star,\varepsilon)}\|_{{\mathscr{V}}_{\mathbbm{k}}}+\|{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}).\end{split}$

Since $\mathbf{0}$ belongs to ${\mathscr{U}}_{\varepsilon}$,
${\mathscr{U}}_{{\mathscr{P}}}$ and ${\mathscr{U}}_{0}$, one has
$\|{\mathrm{y}}\|\leq\|{\mathrm{g}}^{(\varepsilon)}\|_{{\mathcal{H}}_{\mathbbm{k}}}$,
$\|{\mathrm{y}}\|\leq\|{\mathrm{g}}^{(\star,\varepsilon)}\|_{{\mathcal{H}}_{\mathbbm{k}}}$
and
$\|{\mathrm{y}}\|\leq\|{\mathrm{g}}^{\star}\|_{{\mathcal{H}}_{\mathbbm{k}}}$.
Therefore, we have

$\|{\mathrm{v}}^{(\varepsilon)}-{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}\leq
4(\sup_{{\mathrm{w}}\in{\mathscr{U}}_{0}}\min_{{\mathrm{v}}\in{\mathscr{U}}_{\varepsilon}}\|{\mathrm{v}}-{\mathrm{w}}\|_{{\mathscr{V}}_{\mathbbm{k}}})\|{\mathrm{y}}\|.$

Similarly, one can show the following inequalities

$\displaystyle\|{\mathrm{v}}^{(\varepsilon)}-{\mathrm{v}}^{(\star,\varepsilon)}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}$
$\displaystyle\leq
4(\sup_{{\mathrm{w}}\in{\mathscr{U}}_{{\mathscr{P}}}}\min_{{\mathrm{v}}\in{\mathscr{U}}_{\varepsilon}}\|{\mathrm{v}}-{\mathrm{w}}\|_{{\mathscr{V}}_{\mathbbm{k}}})\|{\mathrm{y}}\|,$
(78)
$\displaystyle\|{\mathrm{v}}^{(\star,\varepsilon)}-{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}$
$\displaystyle\leq
4(\sup_{{\mathrm{w}}\in{\mathscr{U}}_{0}}\min_{{\mathrm{v}}\in{\mathscr{U}}_{{\mathscr{P}}}}\|{\mathrm{v}}-{\mathrm{w}}\|_{{\mathscr{V}}_{\mathbbm{k}}})\|{\mathrm{y}}\|.$
(79)

Now, let ${\mathrm{v}}=({\mathrm{x}},{\mathrm{g}})$ be an arbitrary element of
${\mathscr{U}}_{0}$. Then, we know that
$(1-\varepsilon){\mathrm{g}}\in{\mathscr{G}}(1-\varepsilon,\Omega_{{\mathbb{T}}})$.
Also, we have

${\mathrm{L}}^{\\!{\mathrm{u}}}((1-\varepsilon){\mathrm{g}})-({\mathrm{x}}-\varepsilon{\mathrm{L}}^{\\!{\mathrm{u}}}({\mathrm{g}}))={\mathrm{L}}^{\\!{\mathrm{u}}}({\mathrm{g}})-{\mathrm{x}}={\mathrm{y}}.$

Therefore
${\mathrm{w}}:=({\mathrm{x}}-\varepsilon{\mathrm{L}}^{\\!{\mathrm{u}}}({\mathrm{g}}),(1-\varepsilon){\mathrm{g}})$
is an element of ${\mathscr{U}}_{0}$. Moreover, we have

$\begin{split}\|{\mathrm{v}}-{\mathrm{w}}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}&=\varepsilon(\|{\mathrm{L}}^{\\!{\mathrm{u}}}({\mathrm{g}})\|^{2}+\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2})^{\frac{1}{2}}\\\
&\leq\varepsilon(\|{\mathrm{L}}^{\\!{\mathrm{u}}}\|_{{\mathcal{L}}({\mathcal{H}}_{\mathbbm{k}},{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}})}^{2}+1)^{\frac{1}{2}}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}.\end{split}$

Accordingly, it follows that

$\begin{split}\sup_{{\mathrm{w}}\in{\mathscr{U}}_{0}}\min_{{\mathrm{v}}\in{\mathscr{U}}_{\varepsilon}}&\|{\mathrm{v}}-{\mathrm{w}}\|_{{\mathscr{V}}_{\mathbbm{k}}}\\\
&\leq\sup_{({\mathrm{x}},{\mathrm{g}})\in{\mathscr{U}}_{0}}\varepsilon(\|{\mathrm{L}}^{\\!{\mathrm{u}}}\|_{{\mathcal{L}}({\mathcal{H}}_{\mathbbm{k}},{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}})}^{2}+1)^{\frac{1}{2}}\|{\mathrm{g}}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}\\\
&\leq\varepsilon(\|{\mathrm{L}}^{\\!{\mathrm{u}}}\|_{{\mathcal{L}}({\mathcal{H}}_{\mathbbm{k}},{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}})}^{2}+1)^{\frac{1}{2}}\|{\mathrm{y}}\|.\end{split}$
(80)

Since
${\mathscr{U}}_{\varepsilon}\subseteq{\mathscr{U}}_{{\mathscr{P}}}\subseteq{\mathscr{U}}_{0}$,
we know that

$\sup_{{\mathrm{w}}\in{\mathscr{U}}_{0}}\min_{{\mathrm{v}}\in{\mathscr{U}}_{{\mathscr{P}}}}\|{\mathrm{v}}-{\mathrm{w}}\|_{{\mathscr{V}}_{\mathbbm{k}}}\leq\sup_{{\mathrm{w}}\in{\mathscr{U}}_{0}}\min_{{\mathrm{v}}\in{\mathscr{U}}_{\varepsilon}}\|{\mathrm{v}}-{\mathrm{w}}\|_{{\mathscr{V}}_{\mathbbm{k}}}.$

Subsequently, due to (79) and (80), we have

$\|{\mathrm{v}}^{(\star,\varepsilon)}-{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}\leq\varepsilon(\|{\mathrm{L}}^{\\!{\mathrm{u}}}\|_{{\mathcal{L}}({\mathcal{H}}_{\mathbbm{k}},{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}})}^{2}+1)^{\frac{1}{2}}\|{\mathrm{y}}\|.$
(81)

Consequently, it follows that

$\begin{split}\|{\mathrm{g}}^{(\star,\varepsilon)}-{\mathrm{g}}^{\star}\|_{{\mathcal{H}}_{\mathbbm{k}}}^{2}&\leq\|{\mathrm{v}}^{(\star,\varepsilon)}-{\mathrm{v}}^{\star}\|_{{\mathscr{V}}_{\mathbbm{k}}}^{2}\\\
&\leq
4\varepsilon(\|{\mathrm{L}}^{\\!{\mathrm{u}}}\|_{{\mathcal{L}}({\mathcal{H}}_{\mathbbm{k}},{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}})}^{2}+1)^{\frac{1}{2}}\|{\mathrm{y}}\|^{2}.\end{split}$

This also shows that
$\lim_{\varepsilon\to\mathbf{0}}{\mathrm{g}}^{(\star,\varepsilon)}={\mathrm{g}}^{\star}$
in ${\mathcal{H}}_{\mathbbm{k}}$. Note that, for any $t\in{\mathbb{T}}$, we
have
$g_{t}^{(\star,\varepsilon)}={\langle{{\mathrm{g}}^{(\star,\varepsilon)},\mathbbm{k}_{t}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$
and
$g_{t}^{\star}={\langle{{\mathrm{g}}^{\star},\mathbbm{k}_{t}}\rangle}_{{\mathcal{H}}_{\mathbbm{k}}}$.
Therefore, from the Cauchy-Schwartz inequality and the reproducing property,
one has