# Attacks on Perception-Based Control Systems: Modeling and Fundamental Limits

Amir Khazraei, Henry Pfister, and Miroslav Pajic This work is sponsored in
part by the ONR under the agreement N00014-20-1-2745, AFOSR under the award
number FA9550-19-1-0169, by the NSF under the CNS-1652544 award as well as the
National AI Institute for Edge Computing Leveraging Next Generation Wireless
Networks, Grant CNS-2112562. Some of preliminary results in this paper
appeared in [1].The authors are with the Department of Electrical and Computer
Engineering, Duke University, Durham, NC 27708 USA (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>miroslav.pajic@duke.edu).

###### Abstract

We study the performance of perception-based control systems in the presence
of attacks, and provide methods for modeling and analysis of their resiliency
to stealthy attacks on both physical and perception-based sensing.
Specifically, we consider a general setup with a nonlinear affine physical
plant controlled with a perception-based controller that maps both the
physical (e.g., IMUs) and perceptual (e.g., camera) sensing to the control
input; the system is also equipped with a statistical or learning-based
anomaly detector (AD). We model the attacks in the most general form, and
introduce the notions of attack effectiveness and stealthiness independent of
the used AD.

In such setting, we consider attacks with different levels of runtime
knowledge about the plant. We find sufficient conditions for existence of
stealthy effective attacks that force the plant into an unsafe region without
being detected by _any_ AD. We show that as the open-loop unstable plant
dynamics diverges faster and the closed-loop system converges faster to an
equilibrium point, the system is more vulnerable to effective stealthy
attacks. Also, depending on runtime information available to the attacker, the
probability of attack remaining stealthy can be arbitrarily close to one, if
the attacker’s estimate of the plant’s state is arbitrarily close to the true
state; when an accurate estimate of the plant state is not available, the
stealthiness level depends on the control performance in attack-free
operation.

##  I Introduction

The recent progress in deep-learning and computer vision has created a new
generation of control systems that incorporate perceptual data for control and
decision-making. For example, a camera or a light detection and ranging
(LiDAR) sensor can provide the controller with information about plant states
(e.g., distance to the obstacles, position in a local frame). Deep neural
networks (DNNs) have shown capability to extract information from the complex
perception data such as images.

Two main approaches – i.e., _modular_ and _end-to-end_ perception-based
controllers have been recently developed [2]. With the end-to-end control
approach, perception data (e.g., camera images, LiDAR 3D-point clouds),
combined with other physical sensor information, is directly mapped to control
inputs at runtime (e.g., see [3, 4, 5]). The controller is then either learned
via supervised methods, such as imitation learning that mimics a pre-obtained
optimal control input, or using deep-reinforcement learning techniques that
design the control policy by maximizing the cumulative reward in an
unsupervised fashion. On the other hand, with the modular control approach, a
subset of state information is extracted from the perception data (e.g.,
images) and then combined with other physical sensor information, followed by
the use of classic feedback controllers (e.g., [6, 7, 8]).

Despite the tremendous promise that DNN-based perception brings to controls,
resiliency of perception-based controllers to well-documented adversarial
threats remains a challenge, limiting their applicability in real-world
scenarios. The main focus of adversarial machine learning methods has been on
vulnerability of DNNs to small input perturbation, effectively focusing on
robustness analysis of DNNs; e.g., targeting DNNs classification or control
performance when a small and bounded noise is added to the images in camera-
based control systems (e.g., [9, 10]). However, an attacker capable of
compromising the system’s perception/sensing would not limit their actions
(i.e., injected data) to small bounded measurement perturbation; the reason is
that control theoretic methods for designing stealthy attacks suggests that
the perturbation vector should gradually increase in size over time (e.g.,
[11, 12, 13, 14]).

Little consideration has been given on the potential impact of stealthy (i.e.,
undetectable) attacks, which are especially dangerous in the control context
as many systems have recovery-based defenses triggered once an attack is
detected. Model-based vulnerability analysis methods, designed from the
control theory perspective, have been used to analyze the impact stealthy
attacks could have on systems with linear time-invariant (LTI) dynamics and
simple physical sensing, without perception, (e.g., [15, 16, 11, 14]).
However, such analysis cannot be easily extended to systems with complex
dynamics and sensing model that includes perception-based sensing.

Consequently, this work studies the impact that stealthy attacks on system
sensing could have on perception-based control. We assume the attack goal is
to move the system into an unsafe region while remaining stealthy from any
anomaly detector (AD). Even though our notion of stealthiness is related to
the work [17], we do not restrict the attack impact to the infinite time
horizon. We show how such attacks can be modeled in a general form of an
_additive term_ for physical (i.e., non-perception) sensors and a _generative
model_ for perception sensors, without any assumptions about a bound on the
perturbation values.

Perception-based controllers (either end-to-end or modular controllers)
implicitly extract the state information from perception sensing (e.g., camera
images) to derive suitable control inputs. Hence, the goal of this work is to
evaluate their vulnerability to sensing attacks, by investigating whether
there exist adversarial perception outputs that convey desired ‘falsified’
state information while remaining stealthy, rather than blindly adding noise
or e.g., a patch to the current image. For example, for a vehicle with lane-
keeping control (whose goal is to remain in the lane center), we find a
sufficient stealthy attack sequence based on suitable adversarial image
sequences conveying the desired falsified state information that is stealthy
while fooling the controller into unsafe steering decisions.

Depending on attacker’s level of knowledge about the plant’s states, we derive
conditions for which there exists a stealthy and effective (i.e., impactful)
attack sequence that forces the system far from the operating point in the
safe region. In particular, we assume that the attacker has full knowledge
about the system’s open-loop dynamics, and consider two cases where at runtime
the attacker (_i_) has or (_ii_) does not have access to the estimation of the
plant’s state. We show that in the first case, the attack can approach
arbitrarily close to the strict stealthiness condition as long as the
estimation error is small. For the latter, the stealthiness level of the
attack depends on the system’s performance in attack-free operation as well as
the level of impact that the attacker expects to impose on the system (in
terms of the distance from the system’s desired operating point due to the
attack). Thus, there is a trade-off between the stealthiness guarantees and
the performance degradation caused by the attack.

Moreover, for LTI plants, we show that these two cases fall in the same
category, which means the attacker does not need to have access to an accurate
estimate of the plant’s states. We also show that unlike in systems with LTI
plants and linear controllers, where the design of stealthy and effective
attacks is independent of the control design, for nonlinear plants the level
of stealthiness is closely related to the level of closed-loop system
stability – i.e., if the closed-loop system ‘is more stable’ (i.e., its
trajectory converges faster to the equilibrium point), the attack can have
stronger stealthiness guarantees. On the other hand, the attack impact (i.e.,
control degradation) fully depends on the level of open-loop system
instability (e.g., the size of unstable eigenvalues for LTI systems).

###  I-A Related Work

The initial work [18] on adversarial example generation showed that DNNs are
vulnerable to small input perturbations. Afterward, the majority of works has
applied this idea to adversarial attacks on physical world such as malicious
stickers on traffic signs to fool the detectors and/or classifiers [19, 20,
21]. For example, design of adversarial attacks for regression task has been
studied in [22], where the goal of the attacker is to alter the geometrical
prediction of the scene and the predicted distances from the camera. Yet, all
these methods only consider classification or regression tasks in a static
manner; i.e., the target only depends on its input, without consideration of
the longitudinal (i.e., over time) system behaviours.

The vulnerability of perception-based vehicle controls has been recently
studied in longitudinal way (e.g., [9, 10, 23, 24]). For instance, [9]
considers autonomous vehicles with end-to-end DNN controllers that directly
map perceptual inputs into the vehicle steering angle, and target the systems
by painting black lines on the road. On the other hand, [23, 10] introduced
online attacks on streams of images in the tracking task, while [24] considers
longitudinal attacks on camera-LiDAR fusion. Specifically, [10] uses the idea
of adding a small patch into the image for a couple of consecutive frames to
change the position of the bounding box around each object, where the location
and the size of the patch is obtained by solving an optimization problem.

However, these works only consider specific applications and analyze the
attack impact in an ad-hoc manner, limiting the use of their results in other
systems/domains. Further, they lack any consideration of attack stealthiness,
as injecting e.g., adversarial patches that only maximises the disruptive
impact on the control can be detected by most ADs. For instance, [25]
introduced an AD that easily detects the adversarial attacks from [9]. On the
other hand, in this work, we focus on systems with nonlinear system dynamics,
define general notions of attack stealthiness, and introduce sufficient
conditions for a perception-based control system to be vulnerable to effective
yet stealthy perception and sensing attacks. We show that to launch a
successful stealthy attack, the attacker may need to compromise other sensing
information besides perception.

Finally, for _non-perception_ control systems, stealthy attacks have been
well-defined in e.g., [26, 11, 27, 16, 28, 17, 29, 30, 13, 31, 32, 33, 34],
including replay [26], covert [28], zero-dynamic [27], and false data
injection attacks [11, 12, 16]. However, all these work also only focus on LTI
systems and linear controllers, as well as on specific AD design (e.g.,
$\chi^{2}$ detector). The problem of resilient state estimation for a class of
nonlinear control systems has been considered in e.g., [35], focusing on
systems that do not include perception as part of the closed-loop.

The notion of attack stealthiness independent of the employed AD (i.e.,
remaining stealthy for _all_ existing/potential ADs) has been studied in [17].
We additionally differentiate our work in the following – our notion of
stealthiness is stronger than the one in [17] as stealthiness there depends on
time; i.e., there exists only a bounded time that the attack can stay
undetected by an AD. However, the notion of stealthiness in our work is
independent of time and the attack is guaranteed to be stealthy for all time
steps after initiating the attack. Moreover, the performance degradation
metric used in [17] is the error covariance of a Kalman filter estimator as
opposed in our work; we assume the attacker’s goal is to force the system
states into an unsafe region. In addition, we consider systems with nonlinear
dynamics as well as with both perception and physical sensors, unlike [17]
where only LTI plants with physical (i.e., without perception) sensors are
investigated.

Further, this work considers attacks on both perception (e.g., images) and
sensor measurements while the existing vulnerability analyses do not take into
account perception. Besides, these works all assume LTI model and linear
controller for the system dynamics while we consider a class of systems with
nonlinear dynamical model and control. In addition, they focus on
detectability of attacks with respect to specific detection schemes employed
by the controller, such as the classic $\chi^{2}$ anomaly detection algorithm.
Recently [30] has introduced a learning-based attack design for systems with
nonlinear dynamics; yet, in addition to providing no formal analysis, the work
only considers stealthiness with respect to the $\chi^{2}$-based AD, and does
not consider perception-based controllers.

###  I-B Paper Contribution and Organization

The contribution of this work is twofold. First, we consider attacks on
perception-based control in the presence of ADs. We define a new notion of
stealthiness using Neyman-Pearson Lemma and its relation with total variation
and Kullback-Leibler (KL) divergence, where an attack is considered stealthy
if it is stealthy from _any_ AD. We find a sufficient condition for which the
perception-based control systems are vulnerable to highly effective, in terms
of moving the system from the desired operating point, yet stealthy attacks.
Second, unlike all previous works in control literature, we consider highly
impactful stealthy attacks on plants with _nonlinear_ dynamics, controlled by
a perception-based controller potentially employing an end-to-end nonlinear
controller (e.g., DNN).

The paper is organized as follows. In Section III, we present the system
model, as well as a model of attacks on perception before introducing the
concept of stealthy yet effective attacks on the control systems. Section IV
introduces two attack strategies and provides conditions under which the
system is vulnerable to highly effective stealthy attacks. Finally, in Section
V, we provide case studies to illustrate these conditions, before concluding
remarks in Section VI.

##  II Preliminaries

In this section, we introduce employed notation before presenting properties
of KL divergence known as _data processing inequality_ , _monotonicity_ and
_chain-rule_ as presented in [36].

##### Notation

$\mathbb{R}$ denotes the set of reals, whereas $\mathbb{P}$ and $\mathbb{E}$
denote the probability and expectation of a random variable. For a square
matrix $A$, $\lambda_{max}(A)$ is the maximum eigenvalue. For a vector
$x\in{\mathbb{R}^{n}}$, $||x||_{p}$ denotes the $p$-norm of $x$; when $p$ is
not specified, the 2-norm is implied. For a vector sequence, $x_{0}:x_{t}$
denotes the set $\\{x_{0},x_{1},...,x_{t}\\}$. A function
$f:\mathbb{R}^{n}\to\mathbb{R}^{p}$ is Lipschitz on the set $\mathcal{D}$ with
constant $L$ if for any $x,y\in\mathcal{D}\subseteq\mathbb{R}^{n}$ it holds
that $||f(x)-f(y)||\leq L||x-y||$; it is globally Lipschitz with constant $L$
if $\mathcal{D}=\mathbb{R}^{n}$. If $X$ and $Y$ are two sets, $X-Y$ includes
the elements in $X$ that are not in $Y$. For a set $X$, $\partial X$ and
$X^{o}$ define the boundary and the interior of the set, respectively. $B_{r}$
denotes a closed ball centered at zero with radius $r$; i.e.,
$B_{r}=\\{x\in\mathbb{R}^{n}\,\,|\,\,\|x\|\leq r\\}$, whereas $\mathbf{1}_{A}$
is the indicator function on a set $A$. For a function $f$, we denote
$f^{\prime}=\frac{\partial f}{\partial x}$ as the partial derivative of $f$
with respect to $x$ and $\nabla f_{i}(x)$ is the gradient of the function
$f_{i}$ ($i$-th element of the function $f$). Finally, if $\mathbf{P}$ and
$\mathbf{Q}$ are probability distributions relative to Lebesgue measure with
densities $\mathbf{p}$ and $\mathbf{q}$, respectively, then the total
variation between them is defined as
$TV(\mathbf{P},\mathbf{Q})=\frac{1}{2}\int|\mathbf{p}(x)-\mathbf{q}(x)|dx$.
The KL divergence between $\mathbf{P}$ and $\mathbf{Q}$ is defined as
$KL(\mathbf{P}||\mathbf{Q})=\int\mathbf{p}(x)\log{\frac{\mathbf{p}(x)}{\mathbf{q}(x)}}dx$.

### Properties of KL Divergence and Other Preliminaries

In the following lemmas, $X$ and $Y$ are assumed to be random variables
(Lemmas 1-4 proofs can be found in [36]).

######  Lemma 1.

[Data Processing Inequality] Assume $Y$ is produced given $X$ based on the
conditional law $W_{Y|X}$. Let $P_{Y}$ (respectively $Q_{Y}$) denote the
distribution of $Y$ when $X$ is distributed as $P_{X}$ (respectively $Q_{X}$).
Then

$KL(Q_{Y}||P_{Y})\leq KL(Q_{X}||P_{X}).$ (1)

######  Lemma 2.

[Monotonicity] Let $P_{X,Y}$ and $Q_{X,Y}$ be two distributions for a pair of
variables $X$ and $Y$, and $P_{X}$ and $Q_{X}$ be the marginal distributions
for variable $X$. Then,

$KL(Q_{X}||P_{X})\leq KL(Q_{X,Y}||P_{X,Y}).$ (2)

######  Lemma 3.

[Chain rule] Let $P_{X,Y}$ and $Q_{X,Y}$ be two distributions for a pair of
variables $X$ and $Y$. Then,

$KL(Q_{X,Y}||P_{X,Y})=KL(Q_{X}||P_{X})+KL(Q_{Y|X}||P_{Y|X}),$ (3)

where $KL(Q_{Y|X}||P_{Y|X})$ is defined as

$KL(Q_{Y|X}||P_{Y|X})=\mathbb{E}_{x\sim Q_{X}}KL(Q_{Y|X=x}||P_{Y|X=x}).$ (4)

######  Lemma 4.

Let $P_{X}$ and $Q_{X}$ be two Gaussian distributions with the same covariance
$\Sigma$ and different means $\mu_{Q}$ and $\mu_{P}$, respectively. Then, it
holds that

$KL(Q_{X}||P_{X})=\mu_{Q}^{T}\Sigma^{-1}\mu_{P}.$ (5)

######  Lemma 5.

Let $Q_{X}$ be a distribution for a scalar random variable $X$, and that
$X\leq M$ for some $M>0$. Then,

$\mathbb{E}_{Q_{X}}\\{X\\}\leq M.$ (6)

###### Proof.

The proof is straightforward from the definition of expectation and some
properties of the integral. ∎

##  III Modeling Perception-Based Control Systems in the Presence of Attacks

In this section, we introduce the system model and show how to capture attacks
on system sensing, including perception. Specifically, we consider the setup
from Fig. 1 where each of the components is modeled as follows.

###  III-A Plant and Perception Model

We assume the plant has nonlinear dynamics in the standard state-space form

$\begin{split}x_{t+1}&=f(x_{t})+Bu_{t}+w_{t},\quad
y_{t}^{s}=C_{s}x_{t}+v_{t}^{s},\\\ z_{t}&=G(x_{t}).\end{split}$ (7)

Here, $x_{t}\in{\mathbb{R}^{n}}$, $u_{t}\in{\mathbb{R}^{m}}$,
$w_{t}\in{\mathbb{R}^{n}}$, $z_{t}\in{\mathbb{R}^{l}}$,
$y_{t}^{s}\in{\mathbb{R}^{s}}$ and $v^{s}\in{\mathbb{R}^{s}}$ denote the
state, input, system disturbance, observations from perception-based sensors,
(non-perception) sensor measurements, and sensor noise, at time $t$,
respectively. The perception-based sensing is modeled by an unknown generative
model $G$, which is nonlinear and potentially high-dimensional. For example,
consider a camera-based lane keeping system. Here, the observations $z_{t}$
are the captured images; the map $G$ generates the images based on the
vehicle’s position. Without loss of generality, we assume that $f(0)=0$.
Finally, the process and measurement noise vectors $w$ and $v^{s}$ are assumed
independent and identically distributed (iid) Gaussian processes
$w\sim\mathcal{N}(0,\,\Sigma_{w})$ and
$v^{s}\sim\mathcal{N}(0,\,\Sigma_{v^{s}})$.

###  III-B Control Unit

The control unit, shown in Fig. 1, consists of perception, controller, and
anomaly detector units. We assume that the control unit receives $y^{c,s}$ and
$z^{c}$ as the sensor measurements and perception sensing (e.g., images),
respectively as an input. Thus, without malicious activity, it holds that
$y^{c,s}=y^{s}$ and $z^{c}=z$. Now, we describe in detail each of the
components.

####  III-B1 Perception

We assume that there exists a perception map $P$ that imperfectly estimates
the partial state information from from perception sensing (e.g., images) –
i.e.,

$y_{t}^{P}=P(z_{t}^{c})=C_{P}x_{t}+v^{P}(x_{t});$ (8)

here, $P$ denotes a deep neural network (DNN) trained using any supervised
learning method on a data set $\mathcal{X}=\\{(z_{i},x_{i})\\}_{i=1}^{N}$
collected densely around the operating point $x_{o}$ of the system, as in [37,
38]. In addition, $v^{P}\in\mathbb{R}^{p}$ is the perception map error that
depends on the state of the system – i.e., smaller around the training data
set. To capture perception guarantees, we employ the model for robust
perception-based control from [6], and the standard model of the perception
map (8) from [7] that can capture well DNN-based perception modules.
Specifically, if the model is trained effectively, we assume that the
perception error $v^{P}$ around the operating point $x_{o}$ is bounded, i.e.,
the following assumption from [7] holds.

######  Assumption 1.

There exists a safe set $\mathcal{S}$ with the radius $R_{\mathcal{S}}$ (i.e.,
$\mathcal{S}=\\{x\in\mathbb{R}^{n}\,\,|\,\,\|x\|\leq R_{\mathcal{S}}\\}$)
around the operating point such that for all $x\in\mathcal{S}$, it holds that
$\|P(z)-C_{P}x\|\leq\gamma$, where $z=G(x)$ – i.e., for all $x\in\mathcal{S}$,
$\|v^{P}(x)\|\leq\gamma$. Without loss of generality, in this work, we
consider the origin as the operating point – i.e., $x_{o}=0$.

Figure 1: The architecture of a perception-based control system under attack
on system sensing, including perception. Independently of the way attacks are
actually implemented (i.e., directly compromising a sensor or modifying the
measurements delivered over the network to the controller), the same impact on
the control performance is obtained.

######  Remark 1.

As will be shown later in this work, the assumption on boundedness of
$\|v^{P}(x)\|$ and having a good estimate of such bound is only essential for
the defender because the perception map is also used for anomaly detection. We
should note that such bound will not be employed by the attacker to design
stealthy impactful attacks. A systematic method to find the bound on
$\|v^{P}(x)\|$ is discussed in [7].

####  III-B2 Controller

The physical plant (7) is controlled by a (general) nonlinear controller
$u_{t}=\pi(z_{t}^{c},y_{t}^{c,s})$ that maps the image and physical sensor
information to the input control. Using (7), one can write the above control
law in an equivalent form of $u_{t}=\Pi(x_{t},v_{t}^{s})$ that absorbs
$z_{t}^{c}=G(x_{t})$ in the control function $\pi$. Hence, for

$h(x_{t},v_{t}^{s})=f(x_{t})+B\pi(z_{t}^{c},y_{t}^{c,s})=f(x_{t})+B\Pi(x_{t},v_{t}^{s}),$

the evolution of the closed-loop system can be captured as

$\begin{split}x_{t+1}&=h(x_{t},v_{t}^{s})+w_{t}.\end{split}$ (9)

In the general form, the controller can employ any end-end control policy that
uses the image and sensor measurements. For noiseless systems, the state
dynamics can be captured as111With slight abuse of notation, $x$ is used to
denote the states of the noiseless system; yet, in the rest of the paper we
use $x$ to denote the states of the actual physical system with noise and we
clarify if the other case is implied.

$x_{t+1}=h(x_{t},0).$ (10)

######  Definition 1.

The origin of the system (10) is exponentially stable on a set
$\mathcal{D}\subseteq\mathbb{R}^{n}$ if for any $x_{0}\in\mathcal{D}$, there
exist $0<\alpha<1$ and $M>0$, such that $\|x_{t}\|\leq M\alpha^{t}\|x_{0}\|$
for all $t\geq 0$.

######  Lemma 6 ([39]).

For the system from (10), if there exists a function
$V:\mathbb{R}^{n}\to\mathbb{R}$ such that for any
$x_{t}\in\mathcal{D}\subseteq\mathbb{R}^{n}$, the following holds

$\begin{split}c_{1}\|x_{t}\|^{2}\leq V(x_{t})&\leq c_{2}\|x_{t}\|^{2},\\\
V(x_{t+1})-V(x_{t})&\leq-c_{3}\|x_{t}\|^{2},\\\ \|\frac{\partial
V(x)}{\partial x}\|&\leq c_{4}\|x\|,\end{split}$ (11)

for some positive $c_{1},c_{2}$, $c_{3}$ and $c_{4}$, then the origin is
exponentially stable.

######  Assumption 2.

We assume that for the closed-loop control system (10) the origin is
exponentially stable on a set $\mathcal{D}=B_{d}$. Using the converse Lyapunov
theorem [40], there exists a Lyapunov function that satisfies the inequalities
in (11) with constants $c_{1}$, $c_{2}$, $c_{3}$, and $c_{4}$ on a set
$\mathcal{D}=B_{d}$. As a result, it follows that $\Pi(0,0)=0$.

######  Remark 2.

The assumptions for closed-loop system are critical for system guarantees
without the attack; i.e., if the system does not satisfy the stability
property in attack-free scenarios, then an effective strategy for the attacker
would be to wait until the system fails by itself. We refer the reader to the
recent work e.g., [6, 7] on design of such controllers.

######  Remark 3.

Note that the exponential stability assumption on the closed-loop system (10)
is only considered due to the simplicity of the notation and can be relaxed to
control systems with asymptotic stability that satisfies the converse Lyapunov
theorem conditions [40] (Theorem 3.14). Specifically, a similar result as in
Lemma 8 can be obtained; however, the needed notation would be significantly
more cumbersome.

######  Definition 2.

Let $\mathcal{U}_{\rho}$ be the set of all functions
$f:\mathbb{R}^{n}\to\mathbb{R}^{n}$ such that the dynamics
$x_{t+1}=f(x_{t})+d_{t}$, where $d_{t}$ satisfies $\|d_{t}\|\leq\rho$, reaches
arbitrarily large states for some nonzero initial state $x_{0}$. For a
function $f$ from $\mathcal{U}_{\rho}$ and an initial condition $x_{0}$, we
define

$T_{f}(\alpha,x_{0})=\min\\{t\,|\,\|x_{t}\|\geq\alpha\\}.$ (12)

i.e., $T_{f}(\alpha,x_{0})$ is the minimal number of time-steps needed for an
unstable dynamic $f$, starting from the initial condition $x_{0}$, to leave a
bounded ball with the center at zero and radius $\alpha$.222To simplify our
notation, and since we consider specific $f$ from the plant dynamics (7), we
drop the the subscript $f$.

####  III-B3 Anomaly Detector

The system is equipped with an anomaly detector (AD) designed to detect the
presence of any abnormal behaviours. We use
$Y_{t}=\left[\begin{smallmatrix}y_{t}^{P}\\\
y_{t}^{c,s}\end{smallmatrix}\right]$ and
$Y_{t}^{a}=\left[\begin{smallmatrix}y_{t}^{P,a}\\\
y_{t}^{c,s,a}\end{smallmatrix}\right]$ to capture sensor (from (7)) and
perception-based (from (8)) values without and under attack, respectively – we
use the superscript $a$ to differentiate all signals of the attacked system,
with the full attack model introduced in the next subsection. Now, by denoting
$Y_{-\infty}^{-1}=Y_{-\infty}:Y_{-1}$, we consider the classical binary
hypothesis testing problem:

$H_{0}$: normal condition (the AD receives $Y_{-\infty}:Y_{t}$);

$H_{1}$: abnormal behaviour (the AD receives
$Y_{-\infty}^{-1},Y_{0}^{a}:Y_{t}^{a}$).

Effectively, the AD uses both the extracted state information from the
perception map (i.e., (8)) as well as sensor measurements. Given a random
sequence $\bar{Y}^{t}=(\bar{Y}_{-\infty}:\bar{Y}_{t})$, it either comes from
the distribution $\mathbf{P}$ (null hypothesis $H_{0}$), which is determined
by system uncertainties, or from a distribution $\mathbf{Q}$ (the alternative
hypothesis $H_{1}$); note that the _unknown_ distribution $\mathbf{Q}$ is
controlled by the attacker.

For a given AD specified by a function $D:\bar{Y}^{t}\to\\{0,1\\}$, two types
of error may occur. Error type ($\@slowromancap i@$), also referred as _false
alarm_ , occurs if $D(\bar{Y}^{t})=1$ when $\bar{Y}^{t}\sim\mathbf{P}$;
whereas type ($\@slowromancap ii@$) error (_miss detection_) occurs if
$D(\bar{Y}^{t})=0$ when $\bar{Y}^{t}\sim\mathbf{Q}$. Hence, the sum of the
_conditional error probabilities_ of AD $D$ for a given random sequence
$\bar{Y}^{t}$ is

$p_{t}^{e}(D)=\mathbb{P}(D(\bar{Y}^{t})=0|\bar{Y}^{t}\sim\mathbf{Q})+\mathbb{P}(D(\bar{Y}^{t})=1|\bar{Y}^{t}\sim\mathbf{P}).$
(13)

Note that $p_{t}^{e}(D)$ is not a probability measure as it can take values
larger than one. Let us define
$p_{t}^{TD}(D)=\mathbb{P}(D(\bar{Y}^{t})=1|\bar{Y}^{t}\sim\mathbf{Q})$ as the
probability of true detection, and
$p_{t}^{FA}(D)=\mathbb{P}(D(\bar{Y}^{t})=1|\bar{Y}^{t}\sim\mathbf{P})$ as the
probability of false alarm for the detector $D$. We say that an AD (defined by
$D$) to be better than a random guess-based AD (defined by $D_{RG}$) if
$p^{FA}(D)<p^{TD}(D)$; as with the random guess it holds that

$\begin{split}p^{FA}(D_{RG})&=\mathbb{P}(D_{RG}(\bar{Y}^{t})=1|\bar{Y}^{t}\sim\mathbf{P})=\mathbb{P}(D_{RG}(\bar{Y}^{t})=1)\\\
&=\mathbb{P}(D_{RG}(\bar{Y}^{t})=1|\bar{Y}^{t}\sim\mathbf{Q})=p^{TD}(D_{RG}).\end{split}$

###  III-C Attack Modeling

We assume that the attacker has the ability to compromise perception-based
sensing (e.g., camera images) as well as (potentially) the sensor measurements
$y^{s}_{t}$ delivered to the controller (see Fig. 1). Such attacks on physical
sensors can be achieved by directly compromising the sensing (or the
environment of the sensors) or the communication between the sensors and the
controller [41, 42]. On the other hand, implementing the attack on the images
delivered to the controller may not be feasible using physical spoofing
attacks such as LiDAR spoofing by injecting laser data points [21, 24]. Hence,
the attacker needs to compromise the images in cyber domain (e.g., as
discussed in [43]). This can be achieved by modifying the firmware directly on
the camera or the driver code, as done in Hyundai’s Gen5W and Tesla’s Model 3
attacks where custom firmware was installed [44, 45], or the Tesla Model S
attacks that sent custom messages by compromising application drivers [46].
Similarly, such cyber attacks can be achieved using _Man-in-the-Middle_
attacks that target the connection between the camera and the controller;
examples include intercepting legitimate messages and manipulating their
contents, before resending to the intended destination (e.g., as described in
inter-vehicle attack scenarios in [41, 42] or vehicle attacks that manipulated
timing on V2V/V2I messages, causing out-of-date situational awareness [47].

Moreover, we assume the attack starts at $t=0$, and as previously mentioned,
we use the superscript $a$ to differentiate all signals of the attacked
system, for all $t\geq 0$; the attack sequence is
$\\{z_{t}^{c,a},y_{t}^{c,s,a}\\}_{t\geq 0}$, where e.g., the value of
observation delivered to the perception unit at time $t$ is denoted by
$z_{t}^{c,a}$. Note that due to nonlinearity of the operators ($P$ and $G$),
we do not employ the additive false-data injection model for perception
attacks, widely used for LTI systems with non-perception sensing (e.g., [11]).

Thus, the system dynamics under attack can be modeled as

$\begin{split}x_{t+1}^{a}&=f(x_{t}^{a})+Bu_{t}^{a}+w_{t}^{a},\\\
u_{t}^{a}&=\pi(z_{t}^{c,a},{y_{t}^{c,s,a}}).\end{split}$ (14)

In this work, we assume the attacker has full knowledge of the system, its
dynamics, and employed architecture. Further, the attacker has the required
computation power to calculate suitable attack signals to inject, planning
ahead as needed.

######  Remark 4.

The assumption that the attacker has knowledge of the system dynamics (i.e.,
function $f$ and matrix $C_{s}$) is commonly used in existing studies focused
on the worst-case analysis of the attack impact on control systems [11, 12,
16, 14]. In particular, by focusing on resourceful attackers who possess
extensive knowledge of the targeted systems, we can effectively assess the
consequences of worst-case attacks and develop appropriate defensive
strategies. However, the attacks presented in Section IV do not require
knowledge of the noise profile, significantly improving their applicability.

######  Remark 5.

In our notation, $x_{0}^{a}:x_{t}^{a}$ denotes a state trajectory of the
system under attack (for an attack starting at $t=0$), while $x_{0}:x_{t}$
denotes the state trajectory of the attack-free system; we refer to such state
trajectory as the _attack-free_ trajectory. Thus, when comparing the attack-
free trajectory and the system trajectory under attack (i.e., from (14)), we
assume that $w_{t}^{a}=w_{t}$ and $v_{t}^{s,a}=v_{t}^{s}$. However, we cannot
make such assumption for $v^{P}(x)$ as it is a function of states and the
states are compromised due to the attack.

We define an attack to be stealthy if the best strategy for the AD is to
ignore the measurements and make a random guess between the hypotheses; i.e.,
that there is _no_ AD $D$ that satisfies $p^{TD}(D)>p^{FA}(D)$. However,
reaching such stealthiness guarantees may not be possible in general.
Therefore, we define the notion of $\epsilon$-_stealthiness_ , which as we
will show later, is attainable for a large class of nonlinear systems.
Formally, we define the notions of _strict stealthiness_ and _$\epsilon$
-stealthiness_ as follows.

######  Definition 3.

Consider the system defined in (7). An attack sequence is _strictly stealthy_
if there exists no detector for which $p_{t}^{FA}<p_{t}^{TD}$ holds, for any
$t\geq 0$. An attack is $\epsilon$-_stealthy_ if for a given $\epsilon>0$,
there exists no detector such that $p_{t}^{FA}<p_{t}^{TD}-\epsilon$ holds, for
any $t\geq 0$.

Before introducing the sufficient condition for the above notion of
stealthiness, we consider the following lemma.

######  Lemma 7.

The anomaly detector $D$ satisfies $p^{FA}(D)<p^{TD}(D)-\epsilon$ if and only
if $p^{e}(D)<1-\epsilon$. Also, $p^{e}(D)=1$ if and only if $D$ performs as a
random-guess detector.

###### Proof.

First, we consider the case $p^{e}(D)<1-\epsilon$. From (13), it holds that

$\begin{split}&p^{e}(D)=\mathbb{P}(D(\bar{Y})=0|\bar{Y}\sim\mathbf{Q})+\mathbb{P}(D(\bar{Y})=1|\bar{Y}\sim\mathbf{P})\\\
&=1-\mathbb{P}(D(\bar{Y})=1|\bar{Y}\sim\mathbf{Q})+\mathbb{P}(D(\bar{Y})=1|\bar{Y}\sim\mathbf{P})<1-\epsilon\\\
\end{split}$ (15)

Thus,
$\mathbb{P}(D(\bar{Y})=1|\bar{Y}\sim\mathbf{P})<\mathbb{P}(D(\bar{Y})=1|\bar{Y}\sim\mathbf{Q})-\epsilon$
or $p^{FA}(D)<p^{TD}(D)-\epsilon$.

Now, if we have $p^{e}(D)=1$, then we get
$\mathbb{P}(D(\bar{Y})=1|\bar{Y}\sim\mathbf{P})=\mathbb{P}(D(\bar{Y})=1|\bar{Y}\sim\mathbf{Q})$
where the decision of the detector $D$ is independent of the distribution of
$\bar{Y}$ and therefore, the detector performs as the random guess detector.
Since the reverse of all these implications hold, the other (i.e., necessary)
conditions of the theorem also hold. ∎

Now, we can capture stealthiness conditions in terms of KL divergence of the
corresponding distributions.

######  Theorem 1.

An attack sequence is strictly stealthy if and only if
$KL\big{(}\mathbf{Q}(Y_{-\infty}^{-1},Y_{0}^{a}:Y_{t}^{a})||\mathbf{P}(Y_{-\infty}:Y_{t})\big{)}=0$
for all $t\geq 0$. An attack sequence is $\epsilon$-stealthy if the
corresponding observation sequence $Y_{0}^{a}:Y_{t}^{a}$ satisfies

$\begin{split}KL\big{(}\mathbf{Q}(Y_{-\infty}^{-1},Y_{0}^{a}:Y_{t}^{a})||\mathbf{P}(Y_{-\infty}:Y_{t})\big{)}\leq\log(\frac{1}{1-\epsilon^{2}}).\end{split}$
(16)

###### Proof.

With some abuse of notation only specific to this theorem, $\mathbf{Q}$ and
$\mathbf{P}$ are used to denote
$\mathbf{Q}(Y_{-\infty}^{-1},Y_{0}^{a}:Y_{t}^{a})$ and
$\mathbf{P}(Y_{-\infty}:Y_{t})$, respectively. First we prove the strictly
stealthy case.

Using Neyman-Pearson lemma for any existing detector $D$, it follows that

$p_{t}^{e}(D)\geq\int\min\\{\mathbf{Q}(y),\mathbf{P}(y)\\}dy,$ (17)

where the equality holds for the Likelihood Ratio function as
$D^{*}=\mathbf{1}_{\mathbf{Q}\geq\mathbf{P}}$ [48]. Since
$1-\int\min\\{\mathbf{Q}(y),\mathbf{P}(y)\\}dy=\frac{1}{2}\int|\mathbf{q}(x)-\mathbf{p}(x)|dx$,
from [49] and the definition of total variation distance between $\mathbf{Q}$
and $\mathbf{P}$, it holds that

$p_{t}^{e}(D)\geq 1-TV(\mathbf{Q},\mathbf{P}),$ (18)

where $TV$ denotes the total variation distance between the distributions.

Now, it holds that
$TV(\mathbf{Q},\mathbf{P})\leq\sqrt{1-e^{-KL(\mathbf{Q}||\mathbf{P})}}$ (Eq.
(14.11) in [50]). Thus, if $KL\big{(}\mathbf{Q}||\mathbf{P}\big{)}=0$ holds,
then $p_{t}^{e}(D)\geq 1$ for any detector $D$. Therefore, according to Lemma
7 the attack is strictly stealthy. On the other hand, if for any detector $D$
$p_{t}^{e}(D)\geq 1$ holds, then the equality holds for
$TV(\mathbf{Q},\mathbf{P})=0$; this is equivalent to $\mathbf{Q}=\mathbf{P}$
and therefore, $KL\big{(}\mathbf{Q}||\mathbf{P}\big{)}=0$.

For the $\epsilon$-stealthy case, we combine (18) with the inequality
$TV(\mathbf{Q},\mathbf{P})\leq\sqrt{1-e^{-KL(\mathbf{Q}||\mathbf{P})}}$ and
the $\epsilon$-stealthy condition (16), to show

$p_{t}^{e}(D)\geq 1-TV(\mathbf{Q},\mathbf{P})\geq
1-\sqrt{1-e^{-KL(\mathbf{Q}||\mathbf{P})}}\geq 1-\epsilon,$

therefore, according to Lemma 7 the attack is $\epsilon$-stealthy and this
concludes the proof. ∎

######  Remark 6.

The $\epsilon$-stealthiness condition defined in [17] requires that

$\lim_{t\to\infty}\frac{KL\big{(}\mathbf{Q}(Y_{0}^{a}:Y_{t}^{a})||\mathbf{P}(Y_{0}:Y_{t})\big{)}}{t}\leq\epsilon.$

This allows for the KL divergence to linearly increase over time for any
$\epsilon>0$, and as a result, after large-enough time period the attack may
be detected even though it satisfies the definition of stealthiness from [17].
Yet, the $\epsilon$-stealthiness from Definition 3 only depends on $\epsilon$
and is fixed for any time $t$; thus, it introduces a stronger notion of
stealthiness for the attack.

#### Attack Goal

We capture the attacker’s goal as _maximizing_ degradation of the control
performance. Specifically, as we consider the origin as the desired operating
point of the closed-loop system, the attack objective is to maximize the (norm
of) states $x_{t}$. Moreover, the attacker wants _to remain stealthy – i.e.,
undetected by any employed AD_, as formalized below.

######  Definition 4.

The attack sequence, denoted by
$\\{z_{0}^{c,a},y_{0}^{c,s,a}\\},\\{z_{1}^{c,a},y_{1}^{c,s,a}\\},...$ is
referred to $(\epsilon,\alpha)$_-successful attack_ if there exists
$t^{\prime}\geq 0$ such that $\|x_{t^{\prime}}\|\geq\alpha$ and the attack is
$\epsilon$-stealthy for all $t\geq 0$. When such a sequence exists for a
system, the system is called $(\epsilon,\alpha)$-_attackable_. Finally, when
the system is $(\epsilon,\alpha)$-attackable for arbitrarily large $\alpha$,
the system is referred to as _perfectly attackable_.

In the rest of this work, our goal is to derive methods to capture the impact
of stealthy attacks; specifically, in the next section we derive conditions
for existence of a _stealthy_ yet _effective_ attack sequence
$\\{z_{0}^{c,a},y_{0}^{c,s,a}\\},\\{z_{1}^{c,a},y_{1}^{c,s,a}\\},...$
resulting in $\|x_{t}\|\geq\alpha$ for some $t\geq 0$ – i.e., we find
conditions for a system to be $(\epsilon,\alpha)$-attackable. Here, for an
attack to be stealthy, we focus on the $\epsilon-$stealthy notion; i.e., that
the best AD could only improve the probability detection by $\epsilon$
compared to random-guess baseline detector.

##  IV Conditions for $(\epsilon,\alpha)$-Attackable Systems

To provide sufficient conditions for a system to be
$(\epsilon,\alpha)$-attackable, in this section, we introduce two
methodologies to design attack sequences on perception and (classical) sensing
data. The difference in these strategies is the level of information that the
attacker has about the system; we show that the stronger attack impact can be
achieved with the attacker having full knowledge of the system state.

Specifically, we start with the attack strategy where the attacker has access
to the current estimation of state; in such case, we show that the
stealthiness condition is less restrictive, simplifying design of
$\epsilon$-stealthy attacks. For the second attack strategy, we show that the
attacker can launch the attack sequence with only knowing the function $f$
(i.e., plant model); however, achieving $\epsilon$-stealthy attack in this
case is harder as more restrictive conditions are imposed on the attacker.

###  IV-A Attack Strategy i@: Using Estimate of the Plant State

Consider the attack sequence where $z^{c,a}_{t}$ and $y_{t}^{c,s,a}$ injected
at time $t$, for all $t\geq 0$, satisfy

$z^{c,a}_{t}=G(x_{t}^{a}-s_{t}),\,\,\,y_{t}^{c,s,a}=C_{s}(x_{t}^{a}-s_{t})+v_{t}^{s},$
(19)

with $s_{t+1}=f(\hat{x}_{t}^{a})-f(\hat{x}_{t}^{a}-s_{t})$, and for a nonzero
$s_{0}$. Here, $\hat{x}_{t}^{a}$ denotes an estimation of the plant’s state
(in the presence of attacks), and thus ${\zeta}_{t}=\hat{x}_{t}^{a}-x_{t}^{a}$
is the corresponding state estimation error. Note that the attacker can obtain
$\hat{x}_{t}^{a}$ by e.g., running a local estimator using the true
measurements before injecting the false values; i.e.,
$y_{t}^{s,a}=C_{s}x_{t}^{a}+v_{t}^{s}$ and $z_{t}^{a}=G(x_{t}^{a})$. We assume
that the estimation error is bounded by $b_{\zeta}$ – i.e., $\|\zeta_{t}\|\leq
b_{\zeta}$, for all $t\geq 0$.

On the other hand, the above attack design may not require access to the true
plant state $x_{t}^{a}$, since only the ‘shifted’ (i.e., $x_{t}^{a}-s_{t}$)
outputs of the real sensing/perception are injected. For instance, in the lane
centring control (i.e., keeping the vehicle between the lanes),
$G(x_{t}-s_{t})$ only shifts the actual image $s_{t}$ to the right or left
depending on the coordinate definition. Similarly, the attack on physical
(i.e., non-perception) sensors can be implemented as
$y_{t}^{c,s,a}=C_{s}x_{t}^{a}+v_{t}^{s,a}-C_{s}s_{t}=y_{t}^{s,a}-C_{s}s_{t}$
where the attacker only needs to subtract $Cs_{t}$ from the current true
measurements.

The idea behind the above attacks is to have the system believe that its
(plant) state is equal to the state
$e_{t}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}x_{t}^{a}-s_{t}$;
thus, referred to as the _fake state_. Note that effectively both $z_{t}^{a}$
and $y_{t}^{s,a}$ used by an AD are directly functions of the fake state
$e_{t}$. Thus, if the distribution of $e_{0}:e_{t}$ is close to $x_{0}:x_{t}$
(i.e., attack-free trajectory), then the attacker will be successful in
injecting a stealthy attack sequence.

######  Definition 5.

For an attack-free state trajectory $x_{0}:x_{t}$, and for any $T\geq 0$ and
$b_{x},b_{v}>0$, $\delta(T,b_{x},b_{v})$ is the probability that the system
state and physical sensor noise $v^{s}$ remain in the zero center ball with
radius $b_{x}$ and $b_{v}$, respectively, during time period $0\leq t\leq T$,
i.e.,

$\delta(T,b_{x},b_{v})\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\mathbb{P}\big{(}\sup_{0\leq
t\leq T}\|x_{t}\|\leq b_{x},\sup_{0\leq t\leq T}\|v_{t}^{s}\|\leq
b_{v}\big{)}.$ (20)

When the system with exponentially stable closed-loop control dynamics333Note
that under similar conditions the same analysis also holds for asymptotic
stability condition [40] is affected by a bounded perturbation, one can show
that the state of the system will remain in a bounded set. The following lemma
from [40] provides the condition and the upper bound on the norm of the state.

######  Lemma 8 ([40]).

Let $x=0$ be an exponentially stable equilibrium point of the nominal system
(10). Also, let $V(x_{t})$ be a Lyapunov function of the nominal system that
satisfies (11) in $\mathcal{D}$, where $\mathcal{D}=B_{d}$. Suppose the system
is affected by additive perturbation term $g(x_{t})$ that satisfies
$\|g(x_{t})\|\leq\delta+\gamma\|x_{t}\|$. If $c_{3}-\gamma c_{4}>0$ with
$\delta<\frac{c_{3}-\gamma c_{4}}{c_{4}}\sqrt{\frac{c_{1}}{c_{2}}}\theta d$
holds for all $x\in\mathcal{D}$ and some positive $\theta<1$, then for all
$\|x_{0}\|<\sqrt{\frac{c_{1}}{c_{2}}}d$, there exists $t_{1}>0$ such that for
all $0\leq t\leq t_{1}$ $\|x_{t}\|\leq\kappa e^{-\beta t}\|x_{0}\|$ holds with
$\kappa=\sqrt{\frac{c_{2}}{c_{1}}}$, $\beta=\frac{(1-\theta)(c_{3}-\gamma
c_{4})}{2c_{2}}$ and for $t\geq t_{1}$ it holds that $\|x_{t}\|\leq b$ with
$b=\frac{c_{4}}{c_{3}-\gamma
c_{4}}\sqrt{\frac{c_{2}}{c_{1}}}\frac{\delta}{\theta}$.

The next result captures conditions under which a perception-based control
system is not resilient to attacks, in sense that it is
($\epsilon$,$\alpha$)-attackable.

######  Theorem 2.

Consider the system (7) with closed-loop control as in Assumption 2. Assume
that the functions $f$, $f^{\prime}$ (derivative of $f$) and $\Pi^{\prime}$
(derivative of $\Pi$) are Lipschitz, with constants $L_{f}$, $L^{\prime}_{f}$
and $L^{\prime}_{\Pi}$, respectively, and let us define
$L_{1}=L^{\prime}_{f}(b_{x}+2b_{\zeta}+\phi)$,
$L_{2}=\min\\{2L_{f},L^{\prime}_{f}(\alpha+b_{x}+b_{\zeta})\\}$ and
$L_{3}=L^{\prime}_{\Pi}(b_{x}+\phi+b_{v})$ for some $\phi>0$. Moreover, assume
that there exists $\phi>0$ such that the inequalities
$L_{1}+L_{3}\|B\|<\frac{c_{3}}{c_{4}}$ and
$L_{2}b_{\zeta}<\frac{c_{3}-(L_{1}+L_{3}\|B\|)c_{4}}{c_{4}}\sqrt{\frac{c_{1}}{c_{2}}}\theta
d$, for some $0<\theta<1$, are satisfied. Then, the system (7) is
($\epsilon$,$\alpha$)-attackable with probability
$\delta(T(\alpha+b+b_{x},s_{0}),b_{x},b_{v})$ for some $\epsilon>0$, if it
holds that ${\phi}>{b}$ and $f\in\mathcal{U}_{\rho}$ with
$\rho=2L_{f}(b_{x}+b+b_{\zeta})$, $\|s_{0}\|\leq\phi$ and
$b=\frac{c_{4}}{c_{3}-(L_{1}+L_{3}\|B\|)c_{4}}\sqrt{\frac{c_{2}}{c_{1}}}\frac{L_{2}b_{\zeta}}{\theta}$.

The theorem proof is provided in Appendix -A.

From (11), $c_{3}$ can be viewed as a ‘measure’ of the closed-loop system
stability (larger $c_{3}$ means the system converges faster to the equilibrium
point); on the other hand, from Theorem 2, closed-loop perception-based
systems with larger $c_{3}$ are more vulnerable to stealthy attacks as the
conditions of the theorem are easier to satisfy. However, if the plant’s
dynamics is very unstable, $T(\alpha+b_{x}+b,s_{0})$ is smaller for a fixed
$\alpha$ and $s_{0}$. Thus, the probability of attack success
$\delta(T(\alpha+b_{x}+b,s_{0}),b_{x},b_{v})$ is larger for a fixed $b_{x}$
and $b_{v}$.

It should be further noted that $L^{\prime}_{f}$, used in Theorem 2
conditions, would be equal to zero for LTI systems; thus, causing $L_{1}$ to
become zero. Similarly when the mapping $\Pi$ approaches more towards linear
behaviour, $L_{3}$ will go to zero. Therefore, the inequality
$L_{1}+L_{3}\|B\|<\frac{c_{3}}{c_{4}}$ holds for linear systems and linear
controllers. However, larger values of $c_{3}$ will help the inequality to be
satisfied even for nonlinear control systems. In simulation results described
in Section V, we discuss when these conditions are satisfied in more detail.

Moreover, in the extreme case when $b_{\zeta}=0$ (i.e., the attacker can
exactly estimate the plant state), the condition
$L_{2}b_{\zeta}<\frac{c_{3}-(L_{1}+L_{3}\|B\|)c_{4}}{c_{4}}\sqrt{\frac{c_{1}}{c_{2}}}\theta
d$ will be relaxed and the other condition
$L_{1}+L_{3}\|B\|<\frac{c_{3}}{c_{4}}$ becomes less restrictive as $L_{1}$
becomes smaller. Thus, in this case, if the attacker initiate the attack with
arbitrarily small $s_{0}$, then _$\epsilon$ can be arbitrarily close to zero
and the attack will be very close to being strictly stealthy_. Hence, the
following result holds.

######  Corollary 1.

Assume $b_{\zeta}=0$, $L_{1}+L_{3}\|B\|<\frac{c_{3}}{c_{4}}$ with
$L_{1}=L^{\prime}_{f}b_{x}$, $L_{3}=L^{\prime}_{\Pi}(b_{x}+b_{v})$ and the
functions $f$, $f^{\prime}$ (derivative of $f$) and $\Pi^{\prime}$ (derivative
of $\Pi$) are Lipschitz, with constants $L_{f}$, $L^{\prime}_{f}$ and
$L^{\prime}_{\Pi}$, respectively. If $f\in\mathcal{U}_{\rho}$ with
$\rho=2L_{f}(b_{x}+\|s_{0}\|)$, and
$\|s_{0}\|\leq\frac{c_{3}-c_{4}(L_{1}+L_{3}\|B\|)}{c_{4}(L^{\prime}_{f}+\|B\|L^{\prime}_{\Pi})}$
holds, then the system (7) is ($\epsilon$,$\alpha$)-attackable with
probability $\delta(T(\alpha+b_{x}+\|s_{0}\|,s_{0}),b_{x},b_{v})$, where
$\epsilon=\sqrt{1-e^{-b_{\epsilon}}}$ for
$b_{\epsilon}=\Big{(}\lambda_{max}(\Sigma_{w}^{-1})+\lambda_{max}(C_{s}^{T}\Sigma^{-1}_{v}C_{s}+\Sigma^{-1}_{w})\times\min\\{T(\alpha+b_{x}+\|s_{0}\|,s_{0}),\sqrt{\frac{c_{2}}{c_{1}}}\frac{e^{-\beta}}{1-e^{-\beta}}\\}\Big{)}\|s_{0}\|^{2}$
and some $\beta>0$.

Finally, the above results depend on determining $\rho$ such that
$f\in\mathcal{U}_{\rho}$. Hence, the following result provides a sufficient
condition for $f\in\mathcal{U}_{\rho}$.

######  Proposition 1.

Let $V:\mathbb{R}^{n}\to\mathbb{R}$ be a continuously differentiable function
satisfying $V(0)=0$ and define $U_{r_{1}}=\\{x\in
B_{r_{1}}\,\,|\,\,V(x)>0\\}$. Assume that
$\|\frac{\partial{V}(x)}{\partial{x}}\|\leq\beta(\|x\|)$, and for any $x\in
U_{r_{1}}$ it holds $V(f(x))-V(x)\geq\alpha(\|x\|)$, where $\alpha(\|x\|)$ and
$\beta(\|x\|)$ are in class of $\mathcal{K}$ functions [40]. Further, assume
that $r_{1}$ can be chosen arbitrarily large. Now,

  * •

if $\lim_{\|x\|\to\infty}\frac{\alpha(\|x\|)}{\beta(\|x\|)}\to\infty$, then
$f\in\mathcal{U}_{\rho}$ for any $\rho>0$.

  * •

However, if $\lim_{\|x\|\to\infty}\frac{\alpha(\|x\|)}{\beta(\|x\|)}=\gamma$,
then $f\in\mathcal{U}_{\rho}$ for any $\rho<\gamma$.

###### Proof.

We prove the first case and the second case can be shown in a similar way.

If $\lim_{\|x\|\to\infty}\frac{\alpha(\|x\|)}{\beta(\|x\|)}\to\infty$, there
exists a bounded ball with radius $r_{2}$ and center at zero (referred to as
$B_{r_{2}}$) such that for all $x\in S$ with $S=\\{U_{r_{1}}-B_{r_{2}}^{o}\\}$
it holds that $\frac{\alpha(\|x\|)}{\beta(\|x\|)}>\rho$. Since the function
$V$ is differentiable, using the Mean-value theorem for the dynamics
$x_{t+1}=f(x_{t})+d_{t}$ with $\|d_{t}\|\leq\rho$ and for any $x_{t}\in
U_{r_{1}}$, we have that

$\begin{split}V(x_{t+1})-V(x_{t})=&V(f(x_{t})+d_{t})-V(x_{t})\\\
=&V(f(x_{t}))+(\frac{\partial{V}(x)}{\partial{x}})^{T}d_{t}-V(x_{t})\\\
\geq&\alpha(\|x\|)-\beta(\|x\|)\rho.\end{split}$ (21)

Thus, for any $x_{t}\in S$, we have $V(x_{t+1})-V(x_{t})>0$. Let us define
$\eta=\min\\{V(x_{t+1})-V(x_{t})\,\,|\,\,x_{t}\in
S\,\,\text{and}\,\,V(x_{t})\geq a_{r_{2}}\\}$ where $a_{r}=\max_{x\in{\partial
B_{r}}}V(x)$ for any $r>0$. Such minimum exists as the considered set is
compact and we have $\eta>0$ in $U_{r_{1}}$. Let us also assume $x_{0}=\\{x\in
B_{r_{2}}\,\,|\,\,V(x)=a_{r_{2}}\\}$. Now, we claim that the trajectories
starting from $x_{0}$ should leave the set $U_{r_{1}}$ through the boundaries
of $B_{r_{1}}$.

To show this, we know that for any $x_{t}\in S$, $V(x_{t})\geq a_{r_{2}}$,
since $V(x_{t+1})-V(x_{t})\geq\eta>0$. Then, for any $t>0$

$V(x_{t})\geq V(x_{0})+t\eta=a_{r_{2}}+\eta t.$ (22)

The above inequality shows that $x_{t}$ cannot stay in the set $S$ forever as
$V(x)$ is bounded on the compact set $S$. On the other hand, $x_{t}$ cannot
leave the set $S$ through the boundaries satisfying $V(x)=0$ or the surface of
$B_{r_{2}}$ because $V(x_{t})>a_{r_{2}}$. Thus, the trajectories should leave
the set $S$ through the surface of $B_{r_{1}}$, and as $r_{1}$ can be chosen
arbitrarily large, the trajectories of $x_{t}$ will diverge to becoming
arbitrarily large. ∎

######  Example 1.

Consider the dynamical system

$\begin{split}x_{1,t+1}&=2x_{1,t}+x_{1,t}x_{2,t}^{2}\\\
x_{2,t+1}&=0.5x_{2,t},\\\ \end{split}$ (23)

and let us consider the function $V(x)=x_{1}^{2}-x_{2}^{2}$. Hence, we have
$\|\frac{\partial{V}(x)}{\partial{x}}\|\leq 2\|x\|=\beta(\|x\|)$ and

$\begin{split}V(f(x_{t}))-V(x_{t})=&4x_{1,t}^{2}+x_{1,t}^{2}x_{2,t}^{4}+2x_{1,t}^{2}x_{2,t}^{2}-0.25x_{2,t}^{2}\\\
-x_{1,t}^{2}+x_{2,t}^{2}=&3x_{1,t}^{2}+0.75x_{2,t}^{2}+x_{1,t}^{2}x_{2,t}^{4}+2x_{1,t}^{2}x_{2,t}^{2}\\\
\geq&0.75\|x_{t}\|^{2}=\alpha(\|x\|)\end{split}$

Since $\lim_{\|x\|\to\infty}\frac{\alpha(\|x\|)}{\beta(\|x\|)}\to\infty$, for
any $\rho>0$, there exists $x$ such that $V(x_{t+1})-V(x_{t})>0$ for the
dynamics $x_{t+1}=f(x_{t})+d_{t}$ with $\|d\|\leq\rho$. The shaded region in
Fig. 2 shows the area where $V>0$ outside $B_{r_{2}}$ and inside of
$B_{r_{1}}$. Moreover, any area outside of the ball $B_{r_{2}}$ satisfies
$V(x_{t+1})-V(x_{t})>0$ (i.e., $V(x_{t})$ is increasing over time). Now, by
denoting the state value at point $A$ with $x_{A}$, we have
$V(x_{A})=x_{1,A}^{2}>0$. As $V(x_{t}$) is increasing outside $B_{r_{2}}$ as
$t$ increases, any trajectory starting at $A$ cannot leave the shaded region
from the surface of corresponding to $V=0$ or $B_{r_{2}}$; it also cannot stay
in the region for all $t$, as the trajectory evolves. Thus, the system states
will eventually leave the shaded region, ‘exiting’ from the surface of the
ball $B_{r_{1}}$.

Figure 2: The trajectory of the dynamical system in Example 1.

Theorem 2 provides conditions on the closed-loop system stability that may not
hold in general for any system with closed-loop exponential stability. In the
following theorem, we show that one can still obtain a similar
($\epsilon$,$\alpha$)-successful attack (possibly with a larger $\epsilon$)
even when the closed-loop system only satisfies exponential stability.

######  Theorem 3.

Assume that the closed-loop control system (10) is exponentially stable. Then,
the system (7) is ($\epsilon$,$\alpha$)-attackable with probability
$\delta(T(\alpha+b+b_{x},s_{0}),b_{x})$ for some $\epsilon>0$, if
$f\in\mathcal{U}_{\rho}$ with $\rho=2L_{f}(b_{x}+b+b_{\zeta})$ and
$b=\frac{c_{4}}{c_{3}}\sqrt{\frac{c_{2}}{c_{1}}}\frac{(L_{f}(2b_{x}+b_{\zeta})+2L_{\Pi}(b_{x}+b_{v}))}{\theta}$.

###### Proof.

Using the same definition as in (9) we have

$\begin{split}r_{t+1}=&h(r_{t},0)+f(x_{t}^{a})-f(x_{t}^{a}+\zeta_{t})+f(e_{t}+\zeta_{t})-f(x_{t})\\\
&-f(r_{t})+B\Pi(e_{t},v_{t}^{s})-B\Pi(x_{t},v_{t}^{s})\\\
&-B\Pi(r_{t},0)=h(r_{t},0)+\sigma_{t},\end{split}$

From the Lipschitz property of function $f$, it follows that
$\|f(x_{t}^{a})-f(x_{t}^{a}+\zeta_{t})\|\leq L_{f}b_{\zeta}$, and with
probability $\delta(T(\alpha+b+b_{x},s_{0}),b_{x})$ we have

$\begin{split}\|f(x_{t})\|&\leq L_{f}\|x_{t}\|\leq L_{f}b_{x},\\\
\|f(e_{t}+\zeta_{t})-f(r_{t})\|&\leq L_{f}\|x_{t}+\zeta_{t}\|\leq
L_{f}(b_{\zeta}+b_{x}),\\\ \|\Pi(x_{t},v_{t}^{s})\|&\leq
L_{\Pi}(|x_{t}\|+\|v_{t}^{s}\|)\leq L_{\Pi}(b_{x}+b_{v}),\\\
\|\Pi(e_{t},v_{t}^{s})-\Pi(r_{t},0)\|&\leq L_{\Pi}(|x_{t}\|+\|v_{t}^{s}\|)\leq
L_{\Pi}(b_{x}+b_{v}).\end{split}$

Therefore, with probability $\delta(T(\alpha+b+b_{x},s_{0}),b_{x})$ it holds
that $\|\sigma_{t}\|\leq L_{f}(2b_{x}+b_{\zeta})+2L_{\Pi}(b_{x}+b_{v})$ for
all $0\leq t\leq T(\alpha+b+b_{x})$.

Since the closed-loop system $r_{t+1}=h(r_{t},0)$ is exponentially stable, we
use Lemma 8 to show that the dynamics $r_{t+1}=h(r_{t},0)+\sigma_{t}$ will
remain in a bounded ball centered at zero with such probability for all $0\leq
t\leq T(\alpha+b+b_{x})$. The bound is obtained by
$b=\frac{c_{4}}{c_{3}}\sqrt{\frac{c_{2}}{c_{1}}}\frac{(L_{f}(2b_{x}+b_{\zeta})+2L_{\Pi}(b_{x}+b_{v}))}{\theta}$.
Using the Data-processing inequality of KL divergence and following the same
procedure as in Theorem 2 we obtain

$\begin{split}K&L\big{(}\mathbf{Q}({Y}_{-\infty}^{-1},Y_{0}^{a}:Y_{T(\alpha+b_{x}+b,s_{0})}^{a})||\mathbf{P}(Y_{-\infty}:Y_{T(\alpha+b_{x}+b,s_{0})})\big{)}\\\
&\leq\sum_{i=0}^{T(\alpha+b_{x}+b,s_{0})}\lambda_{max}(C_{s}^{T}\Sigma^{-1}_{v}C_{s}+\Sigma^{-1}_{w})\|r_{i}\|^{2}\\\
&\leq\sum_{i=0}^{T(\alpha+b_{x}+b,s_{0})}\lambda_{max}(C_{s}^{T}\Sigma^{-1}_{v}C_{s}+\Sigma^{-1}_{w})b=b_{\epsilon}\\\
&\leq\lambda_{max}(C_{s}^{T}\Sigma^{-1}_{v}C_{s}+\Sigma^{-1}_{w})b(T(\alpha+b_{x}+b,s_{0})+1)=b_{\epsilon}.\end{split}$

This means that the system is ($\epsilon$,$\alpha$)-attackable with
probability $\delta(T(\alpha+b_{x}+b,s_{0}),b_{x},b_{v})$ and
$\epsilon=\sqrt{1-e^{-b_{\epsilon}}}$. On the other hand, since we have
$f\in\mathcal{U}_{\rho}$ with $\rho=2L_{f}(b_{x}+b+b_{\zeta})$, similarly as
in Theorem 2, we can show that for $t\geq T(\alpha+b+b_{x},s_{0})$ the states
of the under-attack system will satisfy $\|x_{t}^{a}\|\geq\alpha$ with
probability $\delta(T(\alpha+b+b_{x},s_{0}),b_{x})$. ∎

Note that our results only focus on the existence of perception measurements
$G(x_{t}^{a}-s_{t})$, obtained by shifting the current perception scene by
$s_{t}$, that results in $(\epsilon,\alpha)$-successful attack, and not how to
compute it. Further, to derive attack sequence using Attack Strategy i@, the
attacker needs the estimation of the plant states. Thus, Attack Strategy ii@
relaxes this assumption, with the attacker only needing to have knowledge
about the plant’s (open-loop) dynamics $f$ and the computation power to
calculate $s_{t+1}=f(s_{t})$ ahead of time.

###  IV-B Attack Strategy ii@: Using Plant Dynamics

Similarly to Attack Strategy i@, consider the attack sequence where
$z^{a}_{t}$ and $y_{t}^{s,a}$, for all $t\geq 0$, satisfy

$\begin{split}z^{a}_{t}&=G(x_{t}^{a}-s_{t}),\,\,\,y_{t}^{s,a}=C_{s}(x_{t}^{a}-s_{t})+v_{t}^{s,a},\\\
s_{t+1}&=f(s_{t}),\end{split}$ (24)

for some nonzero $s_{0}$. However, here the attacker does not need an estimate
of the plant’s state; they simply follow plant dynamics $s_{t+1}=f(s_{t})$ to
find the desired measurements’ transformation. Now, we define the state
$e_{t}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}x_{t}^{a}-s_{t}$
as the _fake state_ , and the attacker’s intention is to make the system
believe that the plant state is equal to $e_{t}$.

The following theorem captures the condition for which the system is not
resilient to attack strategy ii@, in the sense that it is
$(\epsilon,\alpha)$-successful attackable.

######  Theorem 4.

Consider the system (7) that satisfies Assumption 2. Assume that both
functions $f^{\prime}$ and $\Pi^{\prime}$ are Lipschitz, with constants
$L^{\prime}_{f}$ and $L^{\prime}_{\Pi}$, respectively. Moreover, assume that
there exists $\phi>0$ such that the inequalities
$L_{1}+L_{3}\|B\|<\frac{c_{3}}{c_{4}}$ and
$L_{2}b_{x}<\frac{c_{3}-(L_{1}+L_{3}\|B\|)c_{4}}{c_{4}}\sqrt{\frac{c_{1}}{c_{2}}}\theta
d$ with $0<\theta<1$ are satisfied, where
$L_{2}=L^{\prime}_{f}(\alpha+b_{x})$, $L_{1}=L^{\prime}_{f}(\alpha+\phi)$ and
$L_{3}=L^{\prime}_{\Pi}(b_{x}+\phi+b_{v})$. Then, the system (7) is
($\epsilon$,$\alpha$)-attackable with probability
$\delta(T(\alpha+b_{x}+b,s_{0}),{b_{x}})$ and
$b=\frac{c_{4}}{c_{3}-(L_{1}+L_{3}\|B\|)c_{4}}\sqrt{\frac{c_{2}}{c_{1}}}\frac{L_{2}b_{x}}{\theta}$,
for some $\epsilon>0$, if it holds that $\frac{\phi}{b}>1$ and
$f\in\mathcal{U}_{0}$.

The proof of the theorem is provided in Appendix -B.

Unlike in Theorem 2, $L_{1}$ and $L_{2}$ in Theorem 4 increase as $\alpha$
increases. Therefore, unless $L^{\prime}_{f}=0$, one cannot claim that the
attack can be $\epsilon$-stealthy for arbitrarily large $\alpha$ as the
inequality $L_{1}+L_{3}\|B\|\leq\frac{c_{3}}{c_{4}}$ might not be satisfied.
Therefore, there is a trade-off between the stealthiness guarantees
($\epsilon$) and the performance degradation caused by the attack ($\alpha$).
However, in an extreme case where the system is linear, it holds that
$L^{\prime}_{f}=0$ and $L_{1}=L_{2}=0$; before introducing the results for LTI
systems in the next subsection, we remark on the following.

######  Remark 7.

The fact that we considered the control input in a general end-to-end form can
help us verify that our results also hold when the perception-based control
system is modular; i.e., when the states are extracted from the perception
module and physical sensors and then used with the classic control methods.
This is because the end-to-end formulation of the controller is very general,
also covering the modular methods. In such case, the traditional control
methods can be used such that the noiseless closed-loop system becomes
exponentially or asymptotically stable.

###  IV-C Attack on LTI Systems

To derive sufficient conditions for which stealthy yet effective attacks
exist, we have designed such attacks for perception-based control systems
where plants have input affine nonlinear dynamics. However, for LTI plants,
the obtained conditions can be significantly relaxed. Specifically, for LTI
systems, because the system dynamics takes the form of $f(s_{t})=As_{t}$,
Attack Strategies i@ and ii@ become identical as

$\begin{split}s_{t+1}&=f(\hat{x}_{t}^{a})-f(\hat{x}_{t}^{a}-s_{t})=A(\hat{x}^{a}_{t})-A(\hat{x}^{a}_{t}-s_{t})\\\
&=As_{t}=f(s_{t}).\end{split}$ (25)

Therefore, the attacker does not need to estimate the state of the system and
they can use the Attack Strategy ii@ to design the attack sequence.

In general, assume that for such systems we use the controller satisfying
Assumption 2, where due to complexity of perception (e.g., image
observations), a nonlinear function maps the perception sensing (e.g., the
image) and sensors to the control input. Hence, from Corollary 1, we directly
obtain the following result.

######  Corollary 2.

Consider an LTI perception-based control system with $f(x_{t})=Ax_{t}$. Assume
that $L_{3}\|B\|<\frac{c_{3}}{c_{4}}$ with
$L_{3}=L^{\prime}_{\Pi}(b_{x}+b_{v})$,
$\|s_{0}\|\leq\frac{c_{3}-c_{4}L_{3}\|B\|}{c_{4}\|B\|L^{\prime}_{\Pi}}$, and
the matrix $A$ is unstable. Then the system is
($\epsilon$,$\alpha$)-attackable with probability
$\delta(T(\alpha+b_{x}+\|s_{0}\|,s_{0}),{b_{x}},b_{v})$, for arbitrarily large
$\alpha$ and $\epsilon=\sqrt{1-e^{-b_{\epsilon}}}$, with
$b_{\epsilon}=\Big{(}\lambda_{max}(\Sigma_{w}^{-1})+\lambda_{max}(C_{s}^{T}\Sigma^{-1}_{v}C_{s}+\Sigma^{-1}_{w})\times\min\\{T(\alpha+b_{x}+\|s_{0}\|,s_{0}),\sqrt{\frac{c_{2}}{c_{1}}}\frac{e^{-\beta}}{1-e^{-\beta}}\\}\Big{)}\|s_{0}\|^{2}$
and some $\beta>0$.

Note that even though the above corollary considers LTI plants, the above
requirement $L_{3}\|B\|<\frac{c_{3}}{c_{4}}$ is due to the nonlinearity of
employed controllers.

Both Theorem 2 and Theorem 4 assume _end-to-end_ controller that directly maps
the perception and sensor measurements to the control input. However, there
are controllers that first extract the state information using the perception
module $P$ and then use a feedback controller to find the control input (e.g.,
[7, 6]. For instance, consider a linear feedback controller with gain
$K=\begin{bmatrix}K_{P}&K_{s}\end{bmatrix}$, resulting in

$\begin{split}u_{t}=KY_{t}&=K_{s}y^{c,s}_{t}+K_{P}y^{P}_{t}\\\
&=KCx_{t}+K_{P}v^{P}(x_{t})+K_{s}v^{s}_{t}.\end{split}$ (26)

Applying the above control input to the attack-free system

$\begin{split}x_{t+1}&=Ax_{t}+Bu_{t}+w_{t}\\\
&=(A+BKC)x_{t}+BK_{P}v^{P}(x_{t})+BK_{s}v^{s}_{t}.\end{split}$ (27)

Let us assume that Assumptions 1 and 2 still hold. It is easy to show that
$(A+BKC)$ needs to be a stable matrix (i.e., all eigenvalues are inside the
unit circle). Then, we obtained the following result.

######  Theorem 5.

Consider perception-based control of an LTI plant with dynamics
$f(x_{t})=Ax_{t}$, controlled with a linear feedback controller from (26). If
the matrix $A$ is unstable, the system is ($\epsilon$,$\alpha$)-attackable
with probability one for arbitrarily large $\alpha$ and
$\epsilon=\sqrt{1-e^{-b_{\epsilon}}}$, where

$b_{\epsilon}=\lambda_{max}(C_{s}^{T}\Sigma^{-1}_{v}C_{s}+\Sigma^{-1}_{w})(\frac{2\gamma
T(\alpha+R_{\mathcal{S}},s_{0})+\|s_{0}\|}{1-\lambda_{max}(A+BKC)}),$ (28)

and $\lambda_{max}(A+BKC)$ is the largest eigenvalue of the matrix $A+BKC$.

###### Proof.

Let us assume that the attack dynamics are generated using (25); by defining
$e_{t}=x_{t}^{a}-s_{t}$, we obtain $z_{t}^{c,a}=G(x_{t}^{a}-s_{t})=G(e_{t})$
and $y_{t}^{c,s,a}=y_{t}^{s,a}-C_{s}s_{t}=C_{s}e_{t}+v_{t}^{s}$. Therefore,

$\begin{split}e_{t+1}&=x_{t+1}^{a}-s_{t+1}=Ax_{t}^{a}+BKY_{t}-As_{t}\\\
&=Ae_{t}+BK\begin{bmatrix}C_{P}e_{t}+v^{P}(e_{t})\\\
C_{s}e_{t}+v_{t}^{s}\end{bmatrix}\\\
&=(A+BKC)e_{t}+BK_{P}v^{P}(e_{t})+BK_{s}v^{s}_{t}.\end{split}$ (29)

The above dynamics follows the same dynamics as in (27). Thus, if the initial
condition $s_{0}$ is chosen small enough, $e$ will remain in the set
$\mathcal{S}$ and it would holds that $\|v^{P}(e_{t})\|\leq\gamma$ for all
$t\geq 0$. Now, by defining $r_{t}=e_{t}-x_{t}$, we have

$r_{t+1}=(A+BKC)r_{t}+v^{P}(e_{t})-v^{P}(x_{t})$ (30)

Since we have $\|v^{P}(e_{t})\|\leq\gamma$ and $\|v^{P}(x_{t})\|\leq\gamma$,
and the matrix $(A+BKC)$ is stable, we have

$\begin{split}\|r_{t}\|=\|(A+&BKC)^{t}r_{0}+\\\
+&\sum_{i=0}^{t-1}(A+BKC)^{t-i-1}(v^{P}(e_{i})-v^{P}(x_{i}))\|\\\
\leq|\lambda_{max}&(A+BKC)|^{t}\|s_{0}\|+\frac{2\gamma}{1-\lambda_{max}(A+BKC)},\end{split}$

where we used $r_{0}=s_{0}$ and the squared matrix property of
$\|A^{t}v\|\leq|\lambda_{max}(A)|^{t}\|v\|$ for any $v\in\mathbb{R}^{n}$ and
$A\in\mathbb{R}^{n\times n}$. From the proof of Theorem 2, we obtain

$\begin{split}KL&\big{(}\mathbf{Q}(Y_{0}^{a}:Y_{T(\alpha+R_{\mathcal{S}},s_{0})}^{a})||\mathbf{P}(Y_{0}:Y_{T(\alpha+R_{\mathcal{S}},s_{0})})\big{)}\\\
&\leq\sum_{i=0}^{T(\alpha+R_{\mathcal{S}},s_{0})}\lambda_{max}(C_{s}^{T}\Sigma^{-1}_{v}C_{s}+\Sigma^{-1}_{w})\|r_{i}\|^{2}\\\
&=\lambda_{max}(C_{s}^{T}\Sigma^{-1}_{v}C_{s}+\Sigma^{-1}_{w})\sum_{i=0}^{T(\alpha+R_{\mathcal{S}},s_{0})}\|r_{i}\|^{2}\\\
&=\lambda_{max}(C_{s}^{T}\Sigma^{-1}_{v}C_{s}+\Sigma^{-1}_{w})(\frac{2\gamma
T(\alpha+R_{\mathcal{S}},s_{0})+\|s_{0}\|}{1-\lambda_{max}(A+BKC)}).\end{split}$

Since the matrix $A$ is unstable, choosing $s_{0}=cq_{i}$ where $c>0$ is a
scalar and $q_{i}$ is the unstable eigenvector with associated eigenvalue
$|\lambda_{i}|>1$, results in $s_{t}=c\lambda_{i}^{t}q_{i}$; thus, $s_{t}$
becomes arbitrarily large for large enough $t$.

Now, as $T(\alpha+R_{\mathcal{S}},s_{0})$ is the time such that
$\|s_{t}\|\geq\alpha+R_{\mathcal{S}}$, for all $t\geq
T(\alpha+R_{\mathcal{S}},s_{0})$, it holds that

$\begin{split}\|s_{t}\|-\|x_{t}^{a}\|&\leq\|x_{t}^{a}-s_{t}\|=\|e_{t}\|\leq
R_{\mathcal{S}}\Rightarrow\\\ &\Rightarrow\|x_{t}^{a}\|\geq
R_{\mathcal{S}}+\alpha-R_{\mathcal{S}}=\alpha,\end{split}$ (31)

which concludes the proof. ∎

Finally, note that for LTI plants, the attacker will be effective if and only
if $s_{0}$ is not orthogonal to all unstable eigenvectors of the matrix $A$.
Therefore, if $\gamma$ is small enough, by choosing such $s_{0}$ that is also
arbitrarily close to zero, the attacker can be $\epsilon$-stealthy with
$\epsilon$ being arbitrarily small. On the other hand, $\gamma$ is not
controlled by the attacker (rather, it is property of the controller design)
and if $\gamma$ is large, there may be no stealthiness guarantee for the
attacker. While this implies that having large $\gamma$ improves resiliency of
the systems, it should be noted that large $\gamma$ is not desirable from the
control’s perspective, as it would degrade the control performance when the
system is free of attack.

The instability condition for LTI plants in Theorem 5 is inline with the
results from [11, 14] focused on LTI systems with linear controllers and
without perception-based control, as well as with the notion of perfect
attackability introduced there for the specific ADs (i.e., $\chi^{2}$)
considered in those works. On the other hand, in this work, we consider a
general notion of stealthiness (being stealthy from any AD) and provide
analysis for perception-based control systems.

######  Remark 8.

It should be noted the results derived in this work are sufficient conditions
for the existence of $(\epsilon,\alpha)$ attacks and it does not mean the
systems whose system model parameters does not satisfy these conditions are
secure against attacks. However, one can prove that open-loop stability of the
system can provide security for the system, however, providing a formal
analysis is beyond the scope of this work.

##  V Simulation Results

We illustrate and evaluate our methodology for vulnerability analysis of
perception-based control systems on two case studies, inverted pendulum and
autonomous vehicles (AVs).

###  V-A Inverted Pendulum

We consider a fixed-base inverted pendulum equipped with an end-to-end
controller and a perception module that estimates the pendulum angle from
camera images. By using $x_{1}=\theta$ and $x_{2}=\dot{\theta}$, the inverted
pendulum dynamics can be modeled in the state-space form as

$\begin{split}\dot{x}_{1}&=x_{2}\\\
\dot{x}_{2}&=\frac{g}{r}\sin{x_{1}}-\frac{b}{mr^{2}}x_{2}+\frac{L}{mr^{2}};\end{split}$
(32)

here, $\theta$ is the angle of the pendulum rod from the vertical axis
measured clockwise, $b$ is the Viscous friction coefficient, $r$ is the radius
of inertia of the pendulum about the fixed point, $m$ is the mass of the
pendulum, $g$ is the acceleration due to gravity, and $L$ is the external
torque that is applied at the fixed base [51]. Finally, we assumed $m=0.2Kg$,
$b=0.1\frac{m\times Kg}{s}$, $r=0.3m$ and discretized the model with
$T_{s}=0.01\,\,s$.

Using Lyapunov’s indirect method one can show that the origin of the above
system is unstable because the linearized model has an unstable eigenvalue.
However, the direct Lyapunov method can help us to find the whole unstable
region $-\pi<\theta<\pi$ (see [40]). We used a data set $\mathcal{S}$ with
$500$ sample pictures of the fixed-base inverted pendulum with different
angles in $(-\pi,\pi)$ to train a DNN $P$ (perception module) to estimate the
angle of the rod. The angular velocity is also measured directly by the sensor
with noise variance of $\Sigma_{v}=0.0001$. We also trained a deep
reinforcement learning-based controller directly mapping the image pixels and
angular velocity values to the input control. For anomaly detection, we
designed a standard $\chi^{2}$ Extended Kalman filter AD that receives the
perception module’s P output and angular velocity, and outputs the
residue/anomaly alarm.

Figure 3: (a) Evolution of the angle’s ($\theta$) absolute value over time for
different levels of $b_{\zeta}$; (b) The norm of the residue over time when
the attack starts at time $t=0$.

Figure 4: (a) Angle’s ($\theta$) absolute value over time for Attack Strategy
ii@ (red) and normal condition (blue); (b) The residue norm over time for both
under-attack and attack-free systems.

After discretizing the model with sampling rate of $dt=0.01sec$, one can
verify that $L^{\prime}_{f}=0.33$, $\|B\|=0.556$ and we also found that
$L^{\prime}_{\Pi}=0.12$ for the controller. We also considered a Lyapunov
function with $c_{1}=c_{2}=0.5$, $c_{3}=0.057$ and $c_{4}=1$ that satisfies
the inequalities from (11). On the other hand, in more than 100 experiments
without the attack, each simulated for a period of 5 seconds (i.e., 500 time
steps), we found that for $b_{x}=0.2$ and $b_{v}=0.05$ the inequalities in
Definition 5 were always satisfied – i.e., holding with probability of almost
one. Now, one can verify that the condition in Theorem 2 is satisfied if
$b_{\zeta}<.051$.

We first choose $s_{0}=\begin{bmatrix}0.001&0.001\end{bmatrix}^{T}$ and used
Attack Strategy i@ to design attacks. To derive the current adversarial image
at each time step, the attacker exploits the actual image and compromise it by
deviating the pendulum rod by $s_{t}$ degrees. This compromised image is then
delivered to the controller, to be used by the perception module to evaluate
the system state. The other sensor’s measurements are also compromised
accordingly. Again, the attacker does not need to have access to the
perception map $P$; the knowledge about the dynamics $f$ and estimate of the
current plant state $\hat{x}_{t}^{a}$ is sufficient to craft the perturbed
images.

Fig. 3(a) shows the actual pendulum rod angle for different estimation
uncertainty level $b_{\zeta}$ (by the attacker) when the attack starts at
$t=0$. In both cases, the attacker can drive the pendulum rod into an unsafe
region. Fig. 3(b) shows the residue signal over time; the attack stealthiness
level decreases as $b_{\zeta}$ increases, consistent with our results in
Section IV.

Fig. 4(b) presents the residue of the system in normal operating condition
(i.e., without attack) as well as under Attack Strategy<EMAIL_ADDRESS>We can see that
the residue level of both Attack Strategy i@ with $b_{\zeta}=.05$ and Attack
Strategy ii@ are the same as for the system without attack. The red and blue
line in Fig. 4(a) also show the pendulum rod angle trajectory for Attack
Strategy ii@ and normal condition, respectively.

###  V-B Autonomous Vehicle

We consider the nonlinear dynamical model of an AV from [52], with four states
$\left[x~{}~{}~{}y~{}~{}~{}\psi~{}~{}~{}v\right]^{T}$ in the form

$\begin{split}\dot{x}&=v\cos({\psi+\beta}),~{}~{}~{}~{}\dot{y}=v\sin({\psi+\beta}),\\\
\dot{\psi}&=\frac{v}{l_{r}}\sin(\beta),~{}~{}\dot{v}=a,~{}~{}\beta=\tan^{-1}(\frac{l_{r}}{l_{f}+l_{r}}\tan(\delta_{f}));\end{split}$
(33)

here, $x$ and $y$ represent the position of the center of mass in $x$ and $y$
axis, respectively, $\psi$ is the inertial heading, $v$ is the velocity of the
vehicle, and $\beta$ is the angle of the current velocity of the center of
mass with respect to the longitudinal axis of the car. Also, $l_{f}=1.1$ and
$l_{r}=1.73m$ are the distance from the center of the mass of the vehicle to
the front and rear axles, respectively, and $a$ is the acceleration. The
control inputs are the front steering angle $\delta_{f}$ and acceleration $a$.
We assume only $\psi$ is measured directly using noisy sensors, with zero-mean
noise with variance $\Sigma_{v}=0.0001$. Moreover, there is a camera affixed
to dashboard that receives the image of the scene in front of the car. The
system noise is assumed to be zero-mean, with covariance $\Sigma_{w}=0.0001$.

We consider the scenario where the car has a constant speed of $25m/s$. Since
the control objective is to keep the vehicle within the lane (i.e., between
the lines of the road), the relative position of the camera with respect to
the two lines is the essential information used by the controller. Therefore,
even if the image has some other backgrounds, we assume that the lines of the
road would be detected, to allow for extracting the relevant information about
the states of the system (including the position of the car with respect to
the lane center). Using deep reinforcement learning, we trained a controller
that takes the images (containing the lines) and the inertial heading
measurements $\psi$ as the observations, and maps them to the control input
$\delta_{f}$ that keeps the car between the lanes. As the reward function, we
assign a higher reward when the car is between the lines and a lower reward
when the car is moving further from the center of the lane. To find the
position of the car with respect to the lane center, we trained another DNN
(perception map) that takes the images and renders the position y in state
space as follows $P(z_{t})=y_{t}+v^{P}(y_{t}).$

This information is used in AD, where we used $P(z_{t})$ and measured $\psi$
to find the residue for both $\chi^{2}$ and CUSUM detectors. We assume the
system is equipped with $\chi^{2}$[12] and CUSUM anomaly detectors [53], where
an Extended Kalman filter is used to find the residue signal. The thresholds
are set to have $p^{FA}=0.05$ as false alarm rate. Although the system
dynamics (33) is not in the form of (7), the car’s kinematics in lateral
movement can be approximated well by an LTI model with matrix $A$ around
operating point $(x_{e},0,0,0)$ as $A=\bigl{[}\begin{smallmatrix}1&0&0&dt\\\
0&1&25dt&0\\\ 0&0&1&0\\\ 0&0&0&1\end{smallmatrix}\bigr{]};$ the model was
obtained by linearizing the state dynamics with sampling time of $dt=0.01sec$,
where $x_{e}$ is any arbitrary value. Then, we considered the LTI-based attack
design using as $s_{t+1}=As_{t}$ for some small nonzero initial condition
$s_{0}=0.001\begin{bmatrix}0&1&1&0\end{bmatrix}^{T}$. Since the open-loop
model is linear we verify the conditions of Corollary 2; the only condition
that needs to be satisfied in this case is $L_{3}\|B\|<c_{3}$.

We obtained $L^{\prime}_{\Pi}=.23$ and considered a Lyapunov function with
$c_{1}=c_{2}=0.5$, $c_{3}=0.032$ and $c_{4}=1$ that satisfies the conditions
from (11). Also, when running 100 experiments without the attack for, each for
a simulation period of 30 seconds (i.e., 3000 time steps) in all experiments
we obtained $\sup_{0\leq t\leq 3000}\|x_{t}\|<0.2,~{}~{}\sup_{0\leq t\leq
3000}\|v^{s}_{t}\|<0.05$, which means that for $b_{x}=0.2$ and $b_{v}=0.05$,
with probability of almost one the system stays in the zero center ball, as in
Definition 5. Now, we have $L_{3}=L^{\prime}_{\Pi}(b_{x}+b_{v}+\phi)=0.0532$
where the upper bound on the norm of the initial condition of attack states
$s$ is considered to be $0.0014$. Thus, having $\|B\|=0.556$ (obtained by
linearizing and discretization),

$L_{3}\|B\|=0.0106<.032.$

Now, $y_{t}^{a}-s_{t,2}$ provides the desired position of the camera (center
of the car) in Y-axis (where $s_{t,2}$ is the second element of $s_{t}$). To
find the attacked image, we distorted the current image by shifting it some
pixels to right or left (depending on sign of $s_{t,2}$) in order to have
camera be placed in $y_{t}^{a}-s_{t,2}$. Another approach to find the attacked
image is to use some prerecorded images of the road where the car is placed at
with different distances from the lane center. Then, the goal would be to find
an image from those pre-recorded images whose distance from the lane center is
closest to $y_{t}^{a}-s_{t,2}$.

Fig. 5 shows the position of the center of the car with respect to the road
center when the attack starts at time zero. If the attacker chooses the
initial condition $s_{0}=0.001\begin{bmatrix}0&1&1&0\end{bmatrix}^{T}$ the car
will deviate to the left side of the road (left figure), while choosing
$s_{0}=-.001\begin{bmatrix}0&1&1&0\end{bmatrix}^{T}$ will push the car to the
right hand side (right figure).

Fig. 6 illustrates the average number of alarms at each time step for both
$\chi^{2}$ (left figure) and CUSUM (right figure) anomaly detectors in 1000
experiments when the attack starts at time $t=0$. As shown, the values of true
alarm averages (for $t>0$) are the same as the false alarm averages (for
$t\leq 0$) for both ADs, which indicates the stealthiness of the attack
according to Definition 3.

Figure 5: (left) The position of the car with respect to the road center over
time for the initial condition
$s_{0}=0.001\begin{bmatrix}0&1&1&0\end{bmatrix}^{T}$ when the attack starts at
time $t=0$; (right) the position of the car with respect to the road center
over time for the initial condition
$s_{0}=-0.001\begin{bmatrix}0&1&1&0\end{bmatrix}^{T}$ when the attack starts
at time $t=0$.

Figure 6: (left) The average number of alarms at each time step for
$\chi^{2}$-based intrusion detector when the attack starts at time $t=0$;
(right) the average number of alarms at each time step for CUSUM-based anomaly
detector when the attack starts at time $t=0$.

##  VI Conclusion

In this work, we have considered the problem of resiliency under sensing and
perception attacks for perception-based control systems, focusing on a general
class of nonlinear dynamical plants. We have assumed that the noiseless
closed-loop system equipped with an end-to-end controller and anomaly
detector, is exponentially stable on a set around the equilibrium point. We
have introduced a general notion of $\epsilon$-stealthiness as a measure of
difficulty in attack detection from the set of perception measurements and
sensor values. Further, we have derived sufficient conditions for an effective
yet $\epsilon$-stealthy attack sequences to exist. Here, the control
performance degradation has been considered as moving the system state outside
of the safe region defined by a bounded ball with radius $\alpha$, resulting
in an $(\epsilon,\alpha)$-successful attack. Finally, we illustrated our
results on two case studies, fixed-base inverted pendulum and vehicle lane-
keeping.

###  -A Proof of Theorem 2

###### Theorem 2.

We need to show that the sequence of falsified perception (e.g., images) and
physical sensor values
$\\{z_{0}^{c,a},y_{0}^{c,s,a}\\},\\{z_{1}^{c,a},y_{1}^{c,s,a}\\},...$ obtained
by Attack Strategy i@ is ($\epsilon$,$\alpha$)-successful attack for
arbitrarily large $\alpha$. By defining $e_{t}=x_{t}^{a}-s_{t}$ and
$r_{t}=e_{t}-x_{t}$, we get $z^{c,a}_{t}=G(e_{t})$,
$y_{t}^{c,s,a}=C_{s}e_{t}+v_{t}^{s}$ and

$\begin{split}e_{t+1}=&f(x_{t}^{a})-f(x_{t}^{a}+\zeta_{t})+f(e_{t}+\zeta_{t})+B\Pi(e_{t},v_{t}^{s})\\\
&+w_{t}^{a},\\\
r_{t+1}=&h(r_{t},0)+f(x_{t}^{a})-f(x_{t}^{a}+\zeta_{t})+f(e_{t}+\zeta_{t})-f(x_{t})\\\
&-f(r_{t})+B\Pi(e_{t},v_{t}^{s})-B\Pi(x_{t},v_{t}^{s})-B\Pi(r_{t},0)\\\
=&h(r_{t},0)+\sigma_{1}^{\prime}+\sigma_{2}^{\prime},\end{split}$ (34)

with
$\sigma_{1}^{\prime}=f(x_{t}^{a})-f(x_{t}^{a}+\zeta_{t})+f(e_{t}+\zeta_{t})-f(x_{t})-f(r_{t})$
and
$\sigma_{2}^{\prime}=B\Pi(G(e_{t}),y_{t}^{s,a})-B\Pi(G(x_{t}),y_{t}^{s})-B\Pi(G(r_{t}),C_{s}r_{t})$.

Using the Mean value theorem and equality $x_{t}^{a}=r_{t}+s_{t}+x_{t}$ we
have that

$\begin{split}f(x_{t}^{a}+\zeta_{t})&=f(x_{t}^{a})+\frac{\partial f}{\partial
x}|_{\Theta_{(x^{a},x^{a}+\zeta)}}\zeta_{t},\\\
f(e_{t}+\zeta_{t})&=f(x_{t})+\frac{\partial f}{\partial
x}|_{\Theta_{(x,x+r+\zeta)}}(r_{t}+\zeta_{t}),\\\
f(r_{t})&=f(0)+\frac{\partial f}{\partial
x}|_{\Theta_{(0,r)}}r_{t}=\frac{\partial f}{\partial
x}|_{\Theta_{(0,r)}}r_{t},\end{split}$ (35)

where for instance, $\frac{\partial f}{\partial
x}|_{\Theta_{(x,y)}}=\begin{bmatrix}\nabla
f_{1}\big{(}c_{1}x+(1-c_{1})y\big{)}\\\ \vdots\\\ \nabla
f_{n}\big{(}c_{n}x+(1-c_{n})y\big{)}\end{bmatrix}$ for some $0\leq
c_{1},...,c_{n}\leq 1$ where $f_{i}$ is the $i$-th element of function $f$.
Therefore, we get that

$\begin{split}\sigma_{1}^{\prime}=&(\frac{\partial f}{\partial
x}|_{\Theta_{(x^{a},x^{a}+\zeta)}}-\frac{\partial f}{\partial
x}|_{\Theta_{(x,x+r+\zeta)}})\zeta_{t}+\\\ &(\frac{\partial f}{\partial
x}|_{\Theta_{(x,x+r+\zeta)}}-\frac{\partial f}{\partial
x}|_{\Theta_{(0,r)}})r_{t}.\end{split}$ (36)

Similarly, using the Mean Value theorem and $e_{t}=r_{t}+x_{t}$ we obtain

$\begin{split}\Pi(e_{t},v_{t}^{s})&=\Pi(x_{t},v_{t}^{s})+\frac{\partial\Pi}{\partial
e_{t}}|_{\Theta_{(x,x+r)}}r_{t}\\\
\Pi(r_{t},0)&=\Pi(0,0)+\frac{\partial\Pi}{\partial
r_{t}}|_{\Theta_{(0,r)}}r_{t},\end{split}$ (37)

where $\Pi(0,0)=0$. Therefore, we get

$\sigma_{2}^{\prime}=B(\frac{\partial\Pi}{\partial
e_{t}}|_{\Theta_{((x,v),(x+r,v))}}-\frac{\partial\Pi}{\partial
r_{t}}|_{\Theta_{((0,0),(r,0))}})r_{t}$ (38)

Using the fact that the functions $f$ and $\frac{\partial f}{\partial x}(x)$
are Lipschitz, for all $0\leq t\leq T(\alpha+b+b_{x},s_{0})$, it holds that

$\begin{split}&\left\|\frac{\partial f}{\partial
x}|_{\Theta_{(x^{a},x^{a}+\zeta)}}-\frac{\partial f}{\partial
x}|_{\Theta_{(x,x+r+\zeta)}}\right\|\leq\\\
&L^{\prime}_{f}\big{(}\Theta_{(x^{a},x^{a}+\zeta)}-\Theta_{(x,x+r+\zeta)}\big{)}\leq
L^{\prime}_{f}(\|x_{t}^{a}\|+\|x_{t}\|+\|r_{t}\|\\\ &+\|\zeta_{t}\|)\leq
L^{\prime}_{f}(\alpha+b_{x}+b_{\zeta})+L^{\prime}_{f}\|r_{t}\|,\end{split}$
(39)

where we used our assumption that $\|x_{t}\|\leq b_{x}$ with probability
$\delta(T(\alpha+b+b_{x},s_{0}),b_{x},b_{v})$. Moreover, Lipschitz assumption
on $f$ and the boundedness of $\zeta$ also results in

$\begin{split}\left\|\frac{\partial f}{\partial
x}|_{\Theta_{(x^{a},x^{a}+\zeta)}}-\frac{\partial f}{\partial
x}|_{\Theta_{(x,x+r+\zeta)}}\right\|\leq 2L_{f}\end{split}$ (40)

Therefore, we get $\|\frac{\partial f}{\partial
x}|_{\Theta_{(x^{a},x^{a}+\zeta)}}-\frac{\partial f}{\partial
x}|_{\Theta_{(x,x+r+\zeta)}}\|\leq\min\\{2L_{f},L^{\prime}_{f}(\alpha+b_{x}+b_{\zeta})+L^{\prime}_{f}\|r_{t}\|\\}\leq\min\\{2L_{f},L^{\prime}_{f}(\alpha+b_{x}+b_{\zeta})\\}+L^{\prime}_{f}\|r_{t}\|$.
Similarly, we have

$\begin{split}\|\frac{\partial f}{\partial
x}|&{}_{\Theta_{(x,x+r+\zeta)}}-\frac{\partial f}{\partial
x}|_{\Theta_{(0,r)}}\|\leq
L^{\prime}_{f}\big{(}\Theta_{(x,x+r+\zeta)}-\Theta_{(0,r)}\big{)}\leq\\\
&L^{\prime}_{f}(\|x_{t}\|+\|r_{t}\|+\|\zeta_{t}\|)\leq
L^{\prime}_{f}(b_{x}+b_{\zeta})+L^{\prime}_{f}\|r_{t}\|\end{split}$ (41)

Therefore,
$\|\sigma_{1}^{\prime}\|\leq\min\\{2L_{f},L^{\prime}_{f}(\alpha+b_{x}+b_{\zeta})\\}b_{\zeta}+L^{\prime}_{f}b_{\zeta}\|r_{t}\|+L^{\prime}_{f}(b_{x}+b_{\zeta})\|r_{t}\|+L^{\prime}_{f}\|r_{t}\|^{2}$.
Now, consider the bounded set of $B_{\phi}$ satisfying
$B_{\phi}\subset\mathcal{D}$, where we know $\|r_{t}\|^{2}\leq\phi\|r_{t}\|$
for all $r_{t}\in B_{\phi}$, and we get

$\begin{split}\|\sigma^{\prime}_{1}\|\leq&\min\\{2L_{f},L^{\prime}_{f}(\alpha+b_{x}+b_{\zeta})\\}b_{\zeta}\\\
&+L^{\prime}_{f}(b_{x}+2b_{\zeta}+\phi)\|r_{t}\|=L_{1}\|r_{t}\|+L_{2}b_{\zeta}.\end{split}$
(42)

In addition, since the function $\Pi^{\prime}$ is Lipschitz, for all
$x\in\mathcal{D}$ and $r_{t}\in B_{\phi}$ with probability
$\delta(T(\alpha+b+b_{x},s_{0}),b_{x},b_{v})$ it holds that

$\displaystyle\left\|\frac{\partial\Pi}{\partial
e_{t}}|_{\Theta_{((x,v),(x+r,v))}}-\frac{\partial\Pi}{\partial
r_{t}}|_{\Theta_{((0,0),(0,r))}}\right\|$ $\displaystyle\leq
L^{\prime}_{\Pi}(\|x_{t}+r_{t}+v_{t}^{s}\|)$ $\displaystyle\leq
L^{\prime}_{\Pi}(\phi+b_{x}+b_{v}),$

and we get $\|\sigma_{2}^{\prime}\|\leq L_{3}\|B\|\|r_{t}\|$ with
$L_{3}=L^{\prime}_{\Pi}(b_{x}+\phi+b_{v})$; this results in

$\|\sigma^{\prime}_{1}+\sigma^{\prime}_{2}\|\leq
L_{2}b_{\zeta}+(L_{1}+L_{3}\|B\|)\|r_{t}\|.$ (43)

Since for $r_{t}\in B_{\phi}$ we have that
$L_{1}+L_{3}\|B\|<\frac{c_{3}}{c_{4}}$ and
$L_{2}b_{\zeta}<\frac{c_{3}-(L_{1}+L_{3}\|B\|)c_{4}}{c_{4}}\sqrt{\frac{c_{1}}{c_{2}}}\theta
d$, using Lemma 8 for all
$\|r_{0}\|=\|s_{0}\|<\min\\{\sqrt{\frac{c_{1}}{c_{2}}}d,\phi\\}$, there exists
$t_{1}>0$, such that for all $t<t_{1}$ it holds that
$\|r_{t}\|\leq\sqrt{\frac{c_{2}}{c_{1}}}e^{-\beta t}\|s_{0}\|$ with
$\beta=\frac{(1-\theta)\big{(}c_{3}-(L_{1}+L_{3}\|B\|)c_{4}\big{)}}{2c_{2}}$,
and $\|r_{t}\|\leq b$ with
$b=\frac{c_{4}}{c_{3}-(L_{1}+L_{3}\|B\|)c_{4}}\sqrt{\frac{c_{2}}{c_{1}}}\frac{L_{2}b_{\zeta}}{\theta}$
for $t\geq t_{1}$. It should be noted that our assumption $\phi>b$ (or
equivalently $B_{b}\subset B_{\phi}$) guarantees that the trajectory of
$r_{t}$ will remain in the set $B_{\phi}$.

Now, we need to show that for $t\geq T(\alpha+b_{x}+b,s_{0})$ we get that
$\|x_{t}^{a}\|\geq\alpha$. Since the function $f$ is differentiable, using the
Mean-value theorem we obtain

$\begin{split}s_{t+1}=&f(s_{t}+\zeta_{t}+e_{t})-f(\zeta_{t}+e_{t})=\\\
=&f(s_{t})+\frac{\partial{f}}{\partial{x}}|_{(s_{t},s_{t}+e_{t}+\zeta_{t})}(e_{t}+\zeta_{t})-f(e_{t}+\zeta_{t}).\end{split}$
(44)

Since $e_{t}=r_{t}+x_{t}$, for $0\leq t\leq T(\alpha+b_{x}+b,s_{0})$ with
probability $\delta(T(\alpha+b_{x}+b,s_{0}),b_{x},b_{v})$ we have that

$\displaystyle\left\|\frac{\partial{f}}{\partial{x}}|_{(s_{t},s_{t}+e_{t}+\zeta_{t})}(e_{t}+\zeta_{t})-f(e_{t}+\zeta_{t})\right\|$
$\displaystyle\leq 2L_{f}\|(r_{t}+x_{t}+\zeta_{t})\|$ $\displaystyle\leq
2L_{f}(b_{x}+b+b_{\zeta}).$

Since we assume that $f\in\mathcal{U}_{\rho}$ with
$\rho=2L_{f}(b_{x}+b+b_{\zeta})$, there exists $s_{0}$ such that $s_{t}$
becomes arbitrarily large. Using Definition 2, $T(\alpha+b_{x}+b,s_{0})$ is
defined as the first time step that satisfies $\|s_{t}\|\geq\alpha+b_{x}+b$.
On the other hand, using $e_{t}=x_{t}^{a}-s_{t}$ and $e_{t}=x_{t}+r_{t}$, it
holds that

$\|x_{t}^{a}\|\geq\|s_{t}\|-\|e_{t}\|\geq\|s_{t}\|-\|x_{t}\|-\|r_{t}\|\geq\alpha.$

We now need to show that the designed $Y_{t}^{a}$ for $t\geq 0$ satisfies the
stealthiness condition; i.e., we need to show that
$KL\big{(}\mathbf{Q}(Y_{-\infty}^{-1},Y_{0}^{a}:Y_{t}^{a})||\mathbf{P}(Y_{-\infty}:Y_{t})\big{)}\leq\log(\frac{1}{1-\epsilon^{2}})$
for some $\epsilon>0$. Since the sequences $Y_{0}^{a}:Y_{t}^{a}$,
$Y_{0}:Y_{t}$, and $Y_{-\infty}^{-1}$ are generated by $e_{0}:e_{t}$,
$x_{0}:x_{t}$, and $x_{-\infty}:x_{-1}$, respectively, using the Data-
processing inequality of the KL divergence it holds that

$\begin{split}&KL\big{(}\mathbf{Q}({Y}_{-\infty}^{-1},{Y}_{0}^{a}:{Y}_{t}^{a})||\mathbf{P}({Y}_{-\infty}:{Y}_{t})\big{)}\leq\\\
&KL\big{(}\mathbf{Q}(x_{-\infty}:x_{-1},e_{0}:e_{t})||\mathbf{P}(x_{-\infty}:x_{t})\big{)}.\end{split}$
(45)

On the other hand, by defining ${Z}_{t}=\begin{bmatrix}x_{t}\\\
y_{t}^{c,s}\end{bmatrix}$ and ${Z}_{t}^{e}=\begin{bmatrix}e_{t}\\\
y_{t}^{c,s,a}\end{bmatrix}$, and using monotonicity from Lemma 2 it holds that

$\begin{split}KL\big{(}\mathbf{Q}(&x_{-\infty}:x_{-1},e_{0}:e_{t})||\mathbf{P}(x_{-\infty}:x_{t})\big{)}\leq\\\
&KL\big{(}\mathbf{Q}(Z_{-\infty}:Z_{-1},Z_{0}^{e}:Z_{t}^{e})||\mathbf{P}(Z_{-\infty}:Z_{t})\big{)}.\end{split}$
(46)

Then, we apply the chain-rule property of KL-divergence on the right-hand side
of (46) to obtain the following

$\begin{split}&KL\big{(}\mathbf{Q}(Z_{-\infty}:Z_{-1},Z_{0}^{e}:Z_{t}^{e})||\mathbf{P}(Z_{-\infty}:Z_{t})\big{)}=\\\
&KL\big{(}\mathbf{Q}({Z}_{-\infty}:{Z}_{-1})||\mathbf{P}({Z}_{-\infty}:{Z}_{-1})\big{)}+\\\
&\,\,\,\,\,\,\,KL\big{(}\mathbf{Q}({Z}_{0}^{e}:{Z}_{t}^{e}|{Z}_{-\infty}:{Z}_{-1})||\mathbf{P}({Z}_{0}:{Z}_{t}|{Z}_{-\infty}:{Z}_{-1})\big{)}\\\
&=KL\big{(}\mathbf{Q}({Z}_{0}^{e}:{Z}_{t}^{e}|{Z}_{-\infty}:{Z}_{-1})||\mathbf{P}({Z}_{0}:{Z}_{t}|{Z}_{-\infty}:{Z}_{-1})\big{)};\end{split}$
(47)

here, we used the fact that the KL-divergence of two identical distributions
(i.e., $\mathbf{Q}({Z}_{-\infty}:{Z}_{-1})$ and
$\mathbf{P}({Z}_{-\infty}:{Z}_{-1})$ since the system is not under attack for
$t<0$) is zero. Using the chain rule property of the KL divergence we have
that

$\begin{split}K&L\big{(}\mathbf{Q}({Z}_{0}^{e}:{Z}_{t}^{e}|{Z}_{-\infty}:{Z}_{-1})||\mathbf{P}({Z}_{0}:{Z}_{t}|{Z}_{-\infty}:{Z}_{-1})\big{)}\\\
&=KL\big{(}\mathbf{Q}(e_{0}|{Z}_{-\infty}:{Z}_{-1})||\mathbf{P}(x_{0}|{Z}_{-\infty}:{Z}_{-1})\big{)}\\\
&+KL\big{(}\mathbf{Q}(y_{0}^{c,s,a}|e_{0},{Z}_{-\infty}:{Z}_{-1})||\mathbf{P}(y_{0}^{c,s}|x_{0},{Z}_{-\infty}:\mathbf{Z}_{-1})\big{)}\\\
&+...+KL\big{(}\mathbf{Q}(e_{t}|{Z}_{-\infty}:{Z}_{t-1}^{e})||\mathbf{P}(x_{t}|{Z}_{-\infty}:{Z}_{t-1})\big{)}\\\
&+KL\big{(}\mathbf{Q}(y_{t}^{c,s,a}|e_{t},{Z}_{-\infty}:{Z}^{e}_{t-1})||\mathbf{P}(y_{t}^{c,s}|x_{t},{Z}_{-\infty}:{Z}_{t-1})\big{)}.\end{split}$

Given ${Z}_{-\infty}:{Z}_{t-1}$, the distribution of $x_{t}$ is a Gaussian
with some mean $\mu(x_{t-1})$ and covariance $\Sigma_{w}$ written as
$x_{t}=\mu(x_{t-1})+w_{t-1}$. Similarly using (34) given
${Z}_{-\infty}:{Z}_{-1},{Z}_{0}^{e}:{Z}_{t-1}^{e}$, the distribution of
$e_{t}$ is a Gaussian with some mean $\mu(e_{t-1})$ and covariance
$\Sigma_{w}$ written as $e_{t}=\mu(e_{t-1})+w_{t-1}$. Therefore, we get
$r_{t}=e_{t}-x_{t}=\mu(e_{t-1})-\mu(x_{t-1})$. Using Lemma 4 and Lemma 3 for
all $t\geq 0$ it holds that

$\begin{split}KL\big{(}\mathbf{Q}&(e_{t}|{Z}_{-\infty}:{Z}_{t-1}^{e})||\mathbf{P}(x_{t}|{Z}_{-\infty}:{Z}_{t-1})\big{)}=\\\
&=\mathbf{E}_{\mathbf{Q}({Z}_{-\infty}:{Z}_{t-1}^{e})}\\{r_{t}^{T}\Sigma_{w}^{-1}r_{t}\\}\leq\\\
&\leq\mathbf{E}_{\mathbf{Q}(e_{t}|{Z}_{-\infty}:{Z}_{t-1}^{e})}\\{\lambda_{max}(\Sigma_{w}^{-1})\|r_{t}\|^{2}\\}.\end{split}$
(48)

On the other hand, given $x_{t}$, the distribution of $y_{t}^{c,s}$ is a
Gaussian with mean $C_{s}x_{t}$ and covariance $\Sigma_{v^{s}}$. Similarly,
given $e_{t}$, the distribution of $y_{t}^{c,s,a}$ is a Gaussian with mean
$C_{s}e_{t}$ and covariance $\Sigma_{v^{s}}$. Using Lemma 4 and Lemma 3 for
all $t\geq 0$

$\begin{split}KL\big{(}\mathbf{Q}&(y_{t}^{c,s,a}|e_{t},{Z}_{-\infty}:{Z}^{e}_{t-1})||\mathbf{P}(y_{t}^{c,s}|x_{t},{Z}_{-\infty}:{Z}_{t-1})\big{)}=\\\
&KL\big{(}\mathbf{Q}(y_{t}^{c,s,a}|e_{t}||\mathbf{P}(y_{t}^{c,s}|x_{t})\big{)}\leq\\\
&\mathbf{E}_{\mathbf{Q}(y_{t}^{c,s,a}|e_{t})}\\{r_{t}^{T}C_{s}^{T}\Sigma_{v^{s}}^{-1}C_{s}r_{t}\\}\end{split}$
(49)

The arguments inside the expectations in the right hand side of the
inequalities (48) and (49) are upper bounded by
$\lambda_{max}(\Sigma_{w}^{-1})\|r_{t}\|^{2}$ and
$\lambda_{max}(C_{s}^{T}\Sigma_{v^{s}}^{-1}C_{s})\|r_{t}\|^{2}$, respectively,
using the norm property $x^{T}Qx\leq\lambda_{max}(Q)\|x\|^{2}$. On the other
hand, since $\|r_{t}\|$ is bounded from above, using Lemma 5 we can find an
upper bound for (48) and (49).

Specifically, if $T(\alpha+b_{x}+b,s_{0})<t_{1}$, then

$\displaystyle\sum_{i=0}^{T(\alpha+b_{x}+b,s_{0})}\|r_{i}\|^{2}\leq$
$\displaystyle\min\left\\{T(\alpha+b_{x}+b,s_{0})+1,\sqrt{\frac{c_{2}}{c_{1}}}\frac{e^{-\beta}}{1-e^{-\beta}}\right\\}$
$\displaystyle\times\|s_{0}\|^{2}$

with probability $\delta(T(\alpha+b_{x}+b,s_{0}),b_{x},b_{v})$. However, if
$T(\alpha+b_{x}+b,s_{0})\geq t_{1}$ then

$\displaystyle\sum_{i=0}^{T(\alpha+b_{x}+b,s_{0})}\|r_{i}\|^{2}$
$\displaystyle\leq\min\left\\{t_{1},\sqrt{\frac{c_{2}}{c_{1}}}\frac{e^{-\beta}}{1-e^{-\beta}}\right\\}\|s_{0}\|^{2}$
$\displaystyle+(T(\alpha+b_{x}+b,s_{0})+1-t_{1})b$

with probability $\delta(T(\alpha+b_{x}+b,s_{0}),b_{x},b_{v})$. Using the
inequalities (45)–(49), we get

$\begin{split}K&L\big{(}\mathbf{Q}({Y}_{-\infty}^{-1},Y_{0}^{a}:Y_{T(\alpha+b_{x}+b,s_{0})}^{a})||\mathbf{P}(Y_{-\infty}:Y_{T(\alpha+b_{x}+b,s_{0})})\big{)}\\\
&\leq\lambda_{max}(C_{s}^{T}\Sigma^{-1}_{v}C_{s}+\Sigma^{-1}_{w})\times\\\
&\max\Big{\\{}\min\big{\\{}T(\alpha+b_{x}+b,s_{0})+1,\sqrt{\frac{c_{2}}{c_{1}}}\frac{e^{-\beta}}{1-e^{-\beta}}\big{\\}}\|s_{0}\|^{2},\\\
&\min\big{\\{}t_{1},\sqrt{\frac{c_{2}}{c_{1}}}\frac{e^{-\beta}}{1-e^{-\beta}}\big{\\}}\|s_{0}\|^{2}+(T(\alpha+b_{x}+b,s_{0})+1-t_{1})b\Big{\\}}\\\
&=b_{\epsilon},\end{split}$

which means that the system is ($\epsilon$,$\alpha$)-attackable with
probability of $\delta(T(\alpha+b_{x}+b,s_{0}),b_{x},b_{v})$ and
$\epsilon=\sqrt{1-e^{-b_{\epsilon}}}$. ∎

###  -B Proof of Theorem 4

###### Theorem 4.

We need to show that the sequence of compromised perception (e.g., images) and
sensor values $\\{z_{0}^{a},y_{0}^{s,a}\\},\\{z_{1}^{a},y_{1}^{s,a}\\},...$
obtained by Attack Strategy ii@ are ($\epsilon$,$\alpha$)-successful attack.
By defining $e_{t}=x_{t}^{a}-s_{t}$ and $r_{t}=e_{t}-x_{t}$, we get
$z^{a}_{t}=G(e_{t})$, $y_{t}^{s,a}=C_{s}e_{t}+v_{t}^{s,a}$ and

$\begin{split}e_{t+1}=&f(x_{t}^{a})-f(s_{t})+B\Pi(e_{t},v_{t}^{s})+w_{t}^{a},\\\
r_{t+1}=&f(r_{t})+B\Pi(r_{t},0)+f(x_{t}^{a})-f(s_{t})-f(x_{t})\\\
&-f(r_{t})+B\Pi(e_{t},v_{t}^{s})-B\Pi(x_{t},v_{t}^{s})-B\Pi(r_{t},0)\\\
&=h(r_{t},0)+\sigma_{1}+\sigma_{2},\end{split}$

with $\sigma_{1}=f(x_{t}^{a})-f(s_{t})-f(x_{t})-f(r_{t})$ and
$\sigma_{2}=B\Pi(e_{t},v_{t}^{s})-B\Pi(x_{t},v_{t}^{s})-B\Pi(r_{t},0)$. Using
the Mean value theorem and equality $x_{t}^{a}=r_{t}+s_{t}+x_{t}$, we obtain

$\begin{split}f(x_{t}^{a})&=f(x_{t}+s_{t}+r_{t})=f(s_{t})+\frac{\partial
f}{\partial x}|_{\Theta_{(s,s+x+r)}}(x_{t}+r_{t})\\\
f(r_{t})&=f(0)+\frac{\partial f}{\partial
x}|_{\Theta_{(0,r)}}r_{t}=\frac{\partial f}{\partial
x}|_{\Theta_{(0,r)}}r_{t}\\\ f(x_{t})&=f(0)+\frac{\partial f}{\partial
x}|_{\Theta_{(0,x)}}x_{t}=\frac{\partial f}{\partial
x}|_{\Theta_{(0,x)}}x_{t}\end{split}$

Therefore, we get

$\begin{split}\sigma_{1}=&(\frac{\partial f}{\partial
x}|_{\Theta_{(s,s+x+r)}}-\frac{\partial f}{\partial
x}|_{\Theta_{(0,x)}})x_{t}\\\ &+(\frac{\partial f}{\partial
x}|_{\Theta_{(s,s+x+r)}}-\frac{\partial f}{\partial
x}|_{\Theta_{(0,r)}})r_{t}\end{split}$ (50)

Similarly, using the Mean Value theorem and $e_{t}=r_{t}+x_{t}$, we obtain
that

$\begin{split}\Pi(e_{t},v_{t}^{s})&=\Pi(x_{t},v_{t}^{s})+\frac{\partial\Pi}{\partial
e_{t}}|_{\Theta_{(x,x+r)}}r_{t}\\\
\Pi(r_{t},0)&=\Pi(0,0)+\frac{\partial\Pi}{\partial
r_{t}}|_{\Theta_{(0,r)}}r_{t}\end{split}$ (51)

Therefore, we get

$\sigma_{2}=B(\frac{\partial\Pi}{\partial
e_{t}}|_{\Theta_{((x,v),(x+r,v))}}-\frac{\partial\Pi}{\partial
r_{t}}|_{\Theta_{((0,0),(r,0))}})r_{t}.$ (52)

Using the fact that the function $\frac{\partial f}{\partial x}(x)$ is
Lipschitz, for all $0\leq t\leq T(\alpha+b_{x}+b,s_{0})$ we have

$\left\|\frac{\partial f}{\partial x}|_{\Theta_{(s,s+x+r)}}-\frac{\partial
f}{\partial x}|_{\Theta_{(0,x)}}\right\|\leq
L^{\prime}_{f}(\|x_{t}^{a}\|+\|x_{t}\|)\leq L^{\prime}_{f}(\alpha+b_{x}),$

and

$\left\|\frac{\partial f}{\partial x}|_{\Theta_{(s,s+x+r)}}-\frac{\partial
f}{\partial x}|_{\Theta_{(0,r)}}\right\|\leq
L^{\prime}_{f}(\|x_{t}^{a}\|+\|r_{t}\|)\leq L^{\prime}_{f}(\alpha+\phi).$

Therefore, $\|\sigma_{1}\|\leq L_{2}b_{x}+L_{1}\|r_{t}\|$.

Similarly, $\left\|\frac{\partial\Pi}{\partial
e_{t}}|_{\Theta_{((x,v),(x+r,v))}}-\frac{\partial\Pi}{\partial
r_{t}}|_{\Theta_{((0,0),(r,0))}}\right\|\leq
L^{\prime}_{\Pi}(\|x_{t}+r_{t}\|\leq L^{\prime}_{\Pi}(\phi+b_{x}),$ and we get
$L_{3}=L^{\prime}_{\Pi}(b_{x}+\phi+b_{v})$ and $\|\sigma_{2}\|\leq
L_{3}\|r_{t}\|$; this results in $\|\sigma_{1}+\sigma_{2}\|\leq
L_{2}b_{x}+(L_{1}+L_{3}\|B\|)\|r_{t}\|$. Since we have
$L_{1}+L_{3}\|B\|<\frac{c_{3}}{c_{4}}$ and
$L_{2}b_{x}<\frac{c_{3}-(L_{1}+L_{3}\|B\|)c_{4}}{c_{4}}\sqrt{\frac{c_{1}}{c_{2}}}\theta
d$, using Lemma 8 for all
$\|r_{0}\|=\|s_{0}\|<\min\\{\phi,\sqrt{\frac{c_{1}}{c_{2}}}d\\}$ there exists
$t_{1}>0$, such that for all $t\geq t_{1}$ we have $\|r_{t}\|\leq b$ with
$b=\frac{c_{4}}{c_{3}-(L_{1}+L_{3}\|B\|)c_{4}}\sqrt{\frac{c_{2}}{c_{1}}}\frac{L_{2}b_{x}}{\theta}$;
also, for all $0\leq t<t_{1}$, it holds that
$\|r_{t}\|\leq\sqrt{\frac{c_{2}}{c_{1}}}e^{-\beta t}\|s_{0}\|$ with
$\beta=\frac{(1-\theta)\big{(}c_{3}-(L_{1}+L_{3}\|B\|)c_{4}\big{)}}{2c_{2}}$.

On the other hand, the dynamics $s_{t+1}=f(s_{t})$ with nonzero $s_{0}$ will
reach to $\|s_{t}\|\geq\alpha+b_{x}+b$ for some $t\geq
T(\alpha+b_{x}+b,s_{0})$ as $f\in\mathcal{U}_{0}$. Using the reverse triangle
inequality, we obtain

$\begin{split}\|s_{t}\|-\|x_{t}^{a}\|&\leq\|x_{t}^{a}-s_{t}\|=\|e_{t}\|=\|x_{t}+r_{t}\|\\\
&\leq b_{x}+b\Rightarrow-b-
b_{x}+b+b_{x}+\alpha=\alpha\leq\|x_{t}^{a}\|,\end{split}$

with probability $\delta(T(\alpha+b_{x}+b,s_{0}),{b_{x}},b_{v})$.

Now, we need to show that the designed $y_{t}^{a}$ satisfies the stealthiness
condition. In other words, we need to show that
$KL\big{(}\mathbf{Q}(Y_{0}^{a}:Y_{T(\alpha+b_{x}+b,s_{0})}^{a})||\mathbf{P}(Y_{0}:Y_{T(\alpha+b_{x}+b,s_{0})})\big{)}\leq\log(\frac{1}{1-\epsilon^{2}})$
for some $\epsilon>0$. Since the sequences $Y_{0}^{a},...,Y_{t}^{a}$ and
$Y_{0},...,Y_{t}$ are generated by $e_{0},...,e_{t}$ and $x_{0},...,x_{t}$,
respectively, using the Data-processing inequality of KL divergence and
following the same procedure as for Theorem 2, we obtain

$\begin{split}KL&\big{(}\mathbf{Q}(Y_{0}^{a}:Y_{T(\alpha+b_{x}+b,s_{0})}^{a})||\mathbf{P}(Y_{0}:Y_{T(\alpha+b_{x}+b,s_{0})})\big{)}\\\
&\leq\sum_{i=0}^{T(\alpha+b_{x}+b,s_{0})}\lambda_{max}(C_{s}^{T}\Sigma^{-1}_{v}C_{s}+\Sigma^{-1}_{w})\|r_{i}\|^{2}.\end{split}$
(53)

Similar argument as in the proof of Theorem 2 results in

$\begin{split}&KL\big{(}\mathbf{Q}(Y_{0}^{a}:Y_{T(\alpha+b_{x}+b,s_{0})}^{a})||\mathbf{P}(Y_{0}:Y_{T(\alpha+b_{x}+b,s_{0})})\big{)}\leq\\\
&\lambda_{max}(\Sigma_{w}^{-1})\|s_{0}\|^{2}+\lambda_{max}(C_{s}^{T}\Sigma^{-1}_{v}C_{s}+\Sigma^{-1}_{w})\times\\\
&\max\Big{\\{}\min\big{\\{}T(\alpha+b_{x}+b,s_{0})+1,\sqrt{\frac{c_{2}}{c_{1}}}\frac{e^{-\beta}}{1-e^{-\beta}}\big{\\}}\|s_{0}\|^{2},\\\
&\min\big{\\{}t_{1},\sqrt{\frac{c_{2}}{c_{1}}}\frac{e^{-\beta}}{1-e^{-\beta}}\big{\\}}\|s_{0}\|^{2}+(T(\alpha+b_{x}+b,s_{0})-t_{1})b\Big{\\}}\\\
&=b_{\epsilon};\end{split}$

this means that the system is ($\epsilon$,$\alpha$)-attackable with
probability of $\delta(T(\alpha+b_{x}+b,s_{0}),b_{x})$ with
$\epsilon=\sqrt{1-e^{-b_{\epsilon}}}$. ∎

## References

  * [1] A. Khazraei, H. Pfister, and M. Pajic, “Resiliency of perception-based controllers against attacks,” in _Learning for Dynamics and Control Conference_. PMLR, 2022, pp. 713–725.
  * [2] A. Tampuu, T. Matiisen, M. Semikin, D. Fishman, and N. Muhammad, “A survey of end-to-end driving: Architectures and training methods,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2020.
  * [3] V. Rausch, A. Hansen, E. Solowjow, C. Liu, E. Kreuzer, and J. K. Hedrick, “Learning a deep neural net policy for end-to-end control of autonomous vehicles,” in _2017 American Control Conference (ACC)_. IEEE, 2017, pp. 4914–4919.
  * [4] M. Jaritz, R. De Charette, M. Toromanoff, E. Perot, and F. Nashashibi, “End-to-end race driving with deep reinforcement learning,” in _IEEE Int. Conf. on Robotics and Automation (ICRA)_ , 2018, pp. 2070–2075.
  * [5] F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy, “End-to-end driving via conditional imitation learning,” in _2018 IEEE Int. Conf. on Robotics and Automation (ICRA)_ , 2018, pp. 4693–4700.
  * [6] S. Dean and B. Recht, “Certainty equivalent perception-based control,” in _Learning for Dynamics and Control_. PMLR, 2021, pp. 399–411.
  * [7] S. Dean, N. Matni, B. Recht, and V. Ye, “Robust guarantees for perception-based control,” in _Learning for Dynamics and Control_. PMLR, 2020, pp. 350–360.
  * [8] A. A. Al Makdah, V. Katewa, and F. Pasqualetti, “Accuracy prevents robustness in perception-based control,” in _2020 American Control Conference (ACC)_. IEEE, 2020, pp. 3940–3946.
  * [9] A. Boloor, K. Garimella, X. He, C. Gill, Y. Vorobeychik, and X. Zhang, “Attacking vision-based perception in end-to-end autonomous driving models,” _Journal of Systems Architecture_ , vol. 110, p. 101766, 2020.
  * [10] Y. Jia, Y. Lu, J. Shen, Q. A. Chen, Z. Zhong, and T. Wei, “Fooling detection alone is not enough: First adversarial attack against multiple object tracking,” in _Int. Conf. on Learning Representations (ICLR)_ , 2020.
  * [11] Mo, Yilin and Sinopoli, Bruno, “False data injection attacks in control systems,” in _First workshop on Secure Control Systems_ , 2010, pp. 1–6.
  * [12] I. Jovanov and M. Pajic, “Relaxing integrity requirements for attack-resilient cyber-physical systems,” _IEEE Transactions on Automatic Control_ , vol. 64, no. 12, pp. 4843–4858, Dec 2019.
  * [13] A. Khazraei and M. Pajic, “Perfect attackability of linear dynamical systems with bounded noise,” in _2020 ACC_. IEEE, 2020, pp. 749–754.
  * [14] C. Kwon, W. Liu, and I. Hwang, “Analysis and design of stealthy cyber attacks on unmanned aerial systems,” _Journal of Aerospace Information Systems_ , vol. 11, no. 8, pp. 525–539, 2014.
  * [15] I. Jovanov and M. Pajic, “Sporadic data integrity for secure state estimation,” in _IEEE 56th CDC_ , 2017, pp. 163–169.
  * [16] A. Khazraei and M. Pajic, “Attack-resilient state estimation with intermittent data authentication,” _Automatica_ , vol. 138, p. 110035, 2022.
  * [17] C.-Z. Bai, F. Pasqualetti, and V. Gupta, “Data-injection attacks in stochastic control systems: Detectability and performance tradeoffs,” _Automatica_ , vol. 82, pp. 251–260, 2017.
  * [18] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” _arXiv preprint arXiv:1312.6199_ , 2013.
  * [19] K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. Song, “Robust physical-world attacks on deep learning visual classification,” in _Proceedings of the IEEE conf. on computer vision and pattern recognition_ , 2018, pp. 1625–1634.
  * [20] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in _Proceedings of the 2017 ACM on Asia conference on computer and communications security_ , 2017, pp. 506–519.
  * [21] J. Sun, Y. Cao, Q. A. Chen, and Z. M. Mao, “Towards robust lidar-based perception in autonomous driving: General black-box adversarial sensor attack and countermeasures,” in _29th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 20)_, 2020, pp. 877–894.
  * [22] A. Wong, S. Cicek, and S. Soatto, “Targeted adversarial perturbations for monocular depth prediction,” _Advances in neural information processing systems_ , vol. 33, pp. 8486–8497, 2020.
  * [23] H.-J. Yoon, H. J. Sani, and P. Voulgaris, “Learning image attacks toward vision guided autonomous vehicles,” _arXiv:2105.03834_ , 2021.
  * [24] R. S. Hallyburton, Y. Liu, Y. Cao, Z. M. Mao, and M. Pajic, “Security analysis of camera-lidar fusion against black-box attacks on autonomous vehicles,” in _31st USENIX Security Symposium_ , 2022, pp. 1903–1920.
  * [25] F. Cai and X. Koutsoukos, “Real-time out-of-distribution detection in learning-enabled cyber-physical systems,” in _2020 ACM/IEEE 11th Int. Conf. on Cyber-Physical Systems (ICCPS)_ , 2020, pp. 174–183.
  * [26] Y. Mo and B. Sinopoli, “Secure control against replay attacks,” in _47th Allerton Conf. on Communication, Control, and Computing_. IEEE, 2009, pp. 911–918.
  * [27] A. Teixeira, I. Shames, H. Sandberg, and K. H. Johansson, “Revealing stealthy attacks in control systems,” in _50th Annual Allerton Conf. on Communication, Control, and Computing_. IEEE, 2012, pp. 1806–1813.
  * [28] R. S. Smith, “Covert misappropriation of networked control systems: Presenting a feedback structure,” _IEEE Control Systems Magazine_ , vol. 35, no. 1, pp. 82–92, 2015.
  * [29] T. Sui, Y. Mo, D. Marelli, X. Sun, and M. Fu, “The vulnerability of cyber-physical system under stealthy attacks,” _IEEE Transactions on Automatic Control_ , vol. 66, no. 2, pp. 637–650, 2020.
  * [30] A. Khazraei, R. S. Hallyburton, Q. Gao, Y. Wang, and M. Pajic, “Learning-based vulnerability analysis of cyber-physical systems,” in _13th EEE/ACM Int. Conf. on Cyber-Physical Systems (ICCPS)_ , 2022.
  * [31] M. Pajic, I. Lee, and G. J. Pappas, “Attack-resilient state estimation for noisy dynamical systems,” _IEEE Transactions on Control of Network Systems_ , vol. 4, no. 1, pp. 82–92, March 2017.
  * [32] Y.-C. Liu, G. Bianchin, and F. Pasqualetti, “Secure trajectory planning against undetectable spoofing attacks,” _Automatica_ , vol. 112, p. 108655, 2020.
  * [33] G. Bianchin, Y.-C. Liu, and F. Pasqualetti, “Secure navigation of robots in adversarial environments,” _IEEE Control Systems Letters_ , vol. 4, no. 1, pp. 1–6, 2019.
  * [34] D. Gadginmath, V. Krishnan, and F. Pasqualetti, “Direct vs indirect methods for behavior-based attack detection,” in _2022 IEEE 61st Conference on Decision and Control (CDC)_. IEEE, 2022, pp. 7090–7096.
  * [35] Q. Hu, D. Fooladivanda, Y. H. Chang, and C. J. Tomlin, “Secure state estimation and control for cyber security of the nonlinear power systems,” _IEEE Transactions on Control of Network Systems_ , vol. 5, no. 3, pp. 1310–1321, 2017.
  * [36] Y. Polyanskiy and Y. Wu, “Information theory: From coding to learning,” 2022.
  * [37] T. v. Dijk and G. d. Croon, “How do neural networks see depth in single images?” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 2183–2191.
  * [38] A. Lambert, A. Shaban, A. Raj, Z. Liu, and B. Boots, “Deep forward and inverse perceptual models for tracking and prediction,” in _2018 IEEE Int. Conf. on Robotics and Automation (ICRA)_ , 2018, pp. 675–682.
  * [39] H. J. Kushner, “A partial history of the early development of continuous-time nonlinear stochastic systems theory,” _Automatica_ , vol. 50, no. 2, pp. 303–334, 2014.
  * [40] H. K. Khalil, “Nonlinear systems,” 2002.
  * [41] V. Lesi, I. Jovanov, and M. Pajic, “Network scheduling for secure cyber-physical systems,” in _2017 IEEE Real-Time Systems Symposium (RTSS)_ , Dec 2017, pp. 45–55.
  * [42] V. Lesi, I. Jovanov, and M. Pajic, “Integrating security in resource-constrained cyber-physical systems,” _ACM Trans. Cyber-Phys. Syst._ , vol. 4, no. 3, May 2020.
  * [43] R. S. Hallyburton, Q. Zhang, Z. M. Mao, and M. Pajic, “Partial-information, longitudinal cyber attacks on lidar in autonomous vehicles,” _arXiv preprint arXiv:2303.03470_ , 2023.
  * [44] MITRE. [Online]. Available: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-26246
  * [45] ——. [Online]. Available: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-9977
  * [46] ——. [Online]. Available: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-9337
  * [47] I. A. Sumra, J.-L. Ab Manan, and H. Hasbullah, “Timing attack in vehicular network,” in _Proc. of the 15th WSEAS_ , 2011, pp. 151–155.
  * [48] A. Krishnamurthy, “Lecture 21: Minimax theory,” 2017.
  * [49] Y. Polyanskiy and Y. Wu, “Lecture notes on information theory,” _Lecture Notes for ECE563 (UIUC) and_ , vol. 6, no. 2012-2016, p. 7, 2014.
  * [50] T. Lattimore and C. Szepesvári, _Bandit algorithms_. Cambridge University Press, 2020.
  * [51] A. Formal’skii, “An inverted pendulum on a fixed and a moving base,” _J. of applied mathematics an mechanics_ , vol. 70, no. 1, pp. 56–64, 2006.
  * [52] J. Kong, M. Pfeiffer, G. Schildbach, and F. Borrelli, “Kinematic and dynamic vehicle models for autonomous driving control design,” in _2015 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2015, pp. 1094–1099.
  * [53] D. Umsonst, H. Sandberg, and A. A. Cárdenas, “Security analysis of control system anomaly detectors,” in _IEEE American Control Conference (ACC)_ , 2017, pp. 5500–5506.