$\to$ full_access(2). This path, coincidentally, corresponds to the shortest
attack path in this instance of the game. It may alternatively also come up as
the “easiest” path to penetrate according to CVSS or RVSS ratings, depending
on how the game was defined.

  * •

equilibrium utility $U^{*}$: in the given setup, this is the probability
(distribution) of the attacker location over the 10 possible positions in the
attack graph, and we get numbers for these likelihoods from the equilibrium
computation, being

$U^{*}\approx\left\\{\begin{array}[]{c|c}\text{node}&\text{probability of the
attacker being there}\\\ \hline\cr 1&0.573\\\ 2&0\\\ 3&0\\\ 4&0\\\ 5&0\\\
6&0.001\\\ 7&0.111\\\ 8&0.083\\\ 9&0.228\\\ 10&0.001\\\ \end{array}\right.$
(5.1)

which is the expected effect of the defender’s original duty, _i.e._ , the
adversary can get close to the part or machine represented by node $v_{0}$,
but has only a very small chance of conquering it.

Further aspects to include in the consideration relate to the possibility (and
perhaps likely event) to see an optimized defense _fail_ from time to time.
Intrinsic to the concept, with reasons exposed more visibly later, the
defender may suffer a “disappointment” by missing the attacker although the
game-theoretically best defense was implemented. Including the possibility of
such events and minimizing the chances for a defense to fail at all is a more
complex matter and theoretically challenging. We refer to the work of [44, 16,
146] for methods in this regard. Much easier to include are costs of changing
configurations for security. While patching a node’s software is typically
part of the regular maintenance duties, a change of access credentials or
changing a node’s configuration is something with the risk of causing service
disruptions, and hence often avoided. One can (and would need to) include such
costs in the design of the respective utility functions, and generic methods
to do so have been described by [117].

###  5.2 Introduction to Security Games and Strategic Defenses

We have seen in recent years that attackers are becoming increasingly
sophisticated and intelligent. Traditional security solutions that rely on
cryptography, firewalls, and intrusion detection systems are _necessary_ (cf.
Section 2.3) but not _sufficient_ to guarantee the security of the robots.
There are many ways that an attacker can circumvent these technologies and
gain access to the targeted systems. The design objective of perfect security
is not possible as the system designers are always constrained by resources.
The attack graph from Section 5.1 is one step towards this: instead of aiming
for perfect security, one reasonable security solution is to understand the
specific system features and their objectives and take into account the
strategic behaviors of the attacks and the constraints on the attack-and-
defense resources. In robotic systems, the consequences of a compromised
system differ depending on the domain of the applications. For example, a
service robot that interacts with humans, e.g., self-driving cars and
autonomous vehicles, can be turned into a deadly weapon that hurts human
users. Manufacturing robots on assembly lines can break down and cause a
significant economic loss due to reduced production. Hence understanding and
quantifying the system-specific objectives and the available resource is key
to developing an effective defense mechanism against attackers.

To this end, game theory provides a modeling and reasoning framework for the
design of effective security solutions [71]. First, game-theoretic models can
capture the competitive and strategic behaviors of the players and their
constraints. Second, there are a rich set of game-theoretic algorithms and
tools that enable the prediction of the outcomes through the analysis and the
computation of the equilibrium. Third, game models provide ways to incorporate
human factors, including bounded rationality, cognitive biases, and human
perception. Fourth, game models can take different forms at multiple layers of
the system and for various attack models. They can be composed and integrated
to create a game of games to provide a holistic view of the security issues
across the layers of the system and enable a design of system-wide security
solutions. Game theory has been used in a wide variety of cybersecurity
contexts. A few application areas include intrusion detection systems [168,
182, 167], adversarial machine learning [156, 100, 165, 158, 161], proactive
and adaptive defense [140, 38, 30, 183, 185, 175, 50], cyber deception [170,
164, 56, 96, 93, 95], communications channel jamming [8, 180, 176, 151, 181,
171], secure industrial control systems [80, 172, 170, 4] and critical
infrastructure security and resilience [127, 20, 55, 21, 53, 51, 52].

####  5.2.1 Models and Security Games

Let us reconsider the intuitive description of games laid out in Section 5, in
more rigorous and general terms: A normal form game of complete information is
defined by three elements. The first one is the set of players, denoted by
$\mathcal{N}$. In security games, there are often two players in the game. One
is the attacker $A$. The other one is the defender $D$. The second element is
the action set of the players, denoted by $\mathcal{A}_{i},i\in\mathcal{N}$.
The action set captures the feasible actions that are available to the
players. It can naturally incorporate the system and knowledge constraints of
the players, and the rules of the games. The third element is the preference
or the payoffs of the players $U_{i},i\in\mathcal{N}$, which depends on the
actions played by all players, $\\{a_{i},i\in\mathcal{N}\\}$, known as the
action profile. Each player chooses to play the action that maximizes his
payoff. We are interested in the outcome of this game when the players have
complete information of this game and choose action $a_{i}\in\mathcal{A}_{i}$
to maximize their own payoff. The normal-form game of two players with finite
actions, say the row player $A$ and the column player $D$, can be represented
by a matrix. Each row $k\in\\{1,2,\ldots,n\\}$ corresponds to an action in the
action set of player $A$; each column $l\in\\{1,2,\ldots,m\\}$ corresponds to
an action in the action set of player $D$. Matrices $F,G\in\mathds{R}^{n\times
m}$ are the payoff matrices for players $A,D$, respectively. Entries
$F_{kl},G_{kl}$ represent the payoff to players $A,D$, respectively, when
actions that correspond to $k$-th row and $l$-th column are played.

This outcome is predicted by the solution concept called Nash equilibrium. An
action profile $\\{a_{i}^{*}\in\mathcal{A}_{i},i\in\mathcal{N}\\}$ constitutes
a (pure-strategy) Nash equilibrium when no player can deviate from it
unilaterally; in other words,

$U_{i}(a^{*}_{i},a^{*}_{-i})\geq U_{i}(a_{i},a^{*}_{-i}),$

for all $a_{i}\in\mathcal{A}_{i},i\in\mathcal{N}$. Here, $a^{*}_{-i}$ is the
set of all equilibrium actions $\\{a^{*}_{i},i\in\mathcal{N}\\}$ excluding the
equilibrium action of player $i$, i.e., $a_{i}^{*}$. The Nash equilibrium of
an $N$-person game defined by the triplet
$\left(\mathcal{N},\\{\mathcal{A}_{i}\\}_{i\in\mathcal{N}},\\{U_{i}\\}_{i\in\mathcal{N}}\right)$
may not exist. However, the existence issue is resolved when we extend the
strategy space to mixed strategies, which are essentially _probability
distributions_ over the action spaces that describe _random choice rules_ for
taking actions in the game: Let $x_{i},i\in\mathcal{N},$ be the mixed
strategies of player $i$. Its $j$-th component $x_{i}(a_{j})$ can be
interpreted as the probability of player $i$ choosing action $a_{j}$ from the
discrete action set $\mathcal{A}_{i}$. It is clear that $x_{i}(a_{j})$ is
nonnegative and $\sum_{a_{j}\in\mathcal{A}_{i}}x_{i}(a_{j})=1$. Under the
mixed strategy profile $\\{x_{i},i\in\mathcal{N}\\}$, the payoff received by
the player is the average payoff $\bar{U}_{i}(x_{i},x_{-i})$, which is merely
a weighted sum using the payoffs from the actions, multiplied with their
corresponding probabilities from the mixed strategy. In the case of two-player
games, let $x_{1},x_{2}$ be the mixed strategies represented as a finite-
dimensional vectors (of appropriate dimension) of the row player and the
column player, respectively. The $k$-th component of $x_{1}$ and the $l$-th
component of $x_{2}$ correspond to the probabilities of the row player (resp.
column player) choosing actions associated with $k$-th row (resp. $l$-th
column). The average payoff to the row player is given by
$\bar{U}_{1}=x_{1}^{T}Fx_{2}$; the average payoff to the column player is
given by $\bar{U}_{2}=x_{1}^{T}Gx_{2}$.

The mixed-strategy Nash equilibrium can be defined in a similar way as the
pure-strategy Nash equilibrium. The mixed-strategy profile
$\\{x_{i}^{*},i\in\mathcal{N}\\}$ constitutes a mixed-strategy Nash
equilibrium if for all admissible mixed strategy $x_{i},i\in\mathcal{N}$,

$\bar{U}_{i}(x^{*}_{i},x^{*}_{-i})\geq\bar{U}_{i}(x_{i},x^{*}_{-i}).$

It has been known that there exists a mixed-strategy Nash equilibrium for
every finite normal-form game [6, 85].

Zero-sum games are a special class of games that are often used to model
strictly competitive behaviors between two players. One player’s gain is the
other player’s loss. In other words, let
$U_{1}(a_{1},a_{2})=-U_{2}(a_{1},a_{2})=U(a_{1},a_{2})$. Player 1’s objective
is to maximize the payoff $U$ while Player 2’s objective is to minimize it.
The roles of who maximizes and who minimizes can, however, be freely
exchanged, and the game Cut-The-Rope is an example where the defender is a
minimizer: indeed, the defender’s payoff in Section 5.1 is simply the
probability for the attacker to hit the vital target asset, which naturally
should be small if the defense is good. In turn, the attacker obviously seeks
to maximize this probability, and we have a zero-sum competition here.

The solution concept of zero-sum games is _saddle-point equilibrium_ : this is
a joint strategy $(a_{1}^{*},a_{2}^{*})$ if for all $a_{1}\in\mathcal{A}_{1}$
and $a_{2}\in\mathcal{A}_{2}$, $U(a_{1},a_{2}^{*})\leq
U(a_{1}^{*},a_{2}^{*})\leq U(a_{1}^{*},a_{2})$, and called _pure_ if the
strategies $a_{1}^{*},a_{2}^{*}$ are pure. A two-person zero-sum game can be
represented by one matrix $H$. Row $i$ of the matrix corresponds to the $i$-th
action of the row player, say the defender. Column $j$ of the matrix
corresponds to the $j$-th action of the column player, say the attacker. The
entry of the matrix $H_{ij}$ is the payoff to the defender, i.e., the loss to
the attacker when the defender plays the $i$-th action and the attacker plays
the $j$-th action. Let $x_{1}$ and $x_{2}$ be the mixed strategies of the
players. The average payoff or loss to player 1 or player 2, respectively, is
given by $\bar{U}(x_{1},x_{2})=x_{1}^{T}Hx_{2}$. Here, $x_{1},x_{2}$, and $H$
are vectors and matrix of appropriate dimensions. A mixed strategy
$(x_{1}^{*},x_{2}^{*})$ is a mixed-strategy saddle-point equilibrium if for
all admissible $x_{1},x_{2}$,
$\bar{U}(x_{1},x_{2}^{*})\leq\bar{U}(x_{1}^{*},x_{2}^{*})\leq\bar{U}(x_{1}^{*},x_{2})$.
The value $\bar{U}$ achieved under the equilibrium profile is called the
_value_ of the game.

Returning to Cut-The-Rope (Section 5.1) again for illustration, the value
would be the last number $v=0.001$ in (5.1), since the game is primarily about
minimizing the attacker’s chances to hit its target. Any deviation towards a
different defense than prescribed by the game would just increase the success
chances for the adversary to more than $0.001$. This is important for the
defender to bear in mind, since an attempt to further decrease the protection
in other places may open the door wider for the attacker: for example, if the
defender is okay with the probability of $0.001$ for the attacker to hit node
10, but then strives to decrease the – seemingly high – probability of $0.228$
for the attacker to be at node 9 instead, any change in the defense strategy
for the sake of lowering the number $0.228$ would imply an increase of the
attacker’s chance to hit node 10 perhaps on other ways, say, bypassing node 9
at all! This effect is due to the equilibrium property formalized above.

One important property of saddle-point equilibrium is the exchangeability;
i.e., when $(x_{1}^{*},x_{2}^{*})$ and $(x_{1}^{\circ},x_{2}^{\circ})$ are two
distinct saddle-point equilibria of the zero-sum game, then
$(x_{1}^{*},x_{2}^{\circ})$ and $(x_{1}^{\circ},x_{2}^{*})$ are also saddle-
point equilibria of the game and yield the same game value. This is the
theoretical reason why it is safe for the defender to use _any_ of the
existing equilibria for its defensive purpose, but at the same time dangerous
to rely on the adversary’s equilibrium as a hint on where to defend: the
exchangeability property lets the adversary pick any of (perhaps many) optimal
attack strategies to gain the best possible success rates, which can easily
annihilate the defender’s precautions if they were based on the attacker’s
equilibrium behavior instead of the (better) defender’s equilibrium
strategies.

####  5.2.2 Structural and Operational Security

Zero-sum games are useful to capture many security scenarios. For example, a
jamming game between a team of UAV and a jammer has been investigated in [26].
Illustrated in Fig. 5.2, a team of UAV is controlled to maximize the
connectivity among themselves in an adversarial environment where an attacker
can choose a subset of communication links to jam. The game between the
operator of the team and the attacker is described by the zero-sum game at
time $k$:

$\max_{x(k+c)}\min_{e\in\mathcal{E}}\lambda_{2}(e;x(k+c)).$ (5.2)

Here, $x(k)$ is the position of the UAV at time $k$. Two UAV can form a link
when they are sufficiently close within a desirable range of communications.
The connectivity of the UAV team is described by the algebraic connectivity of
the network, denoted by $\lambda_{2}$ (i.e., the second-smallest eigenvalue of
the associated Laplacian matrix). When $\lambda_{2}$ is zero, the network has
disjoint partitions. Otherwise, the network is connected, i.e., there exists a
path from one node to any node in the network. A higher value of $\lambda_{2}$
indicates that there are a larger number of paths on average that connect
between two arbitrary nodes in this network. At each time step $k$, the
operator determines where the agents should move to in the next time step
$x(k+c)$, where $c$ is a time interval. The control is constrained by the
physical dynamics of UAV. The attacker can jam a subset of links from all the
communication links of the team, denoted by $\mathcal{E}$. The attacker’s
capability is described by the number of links that he can jam at time $k$.
This zero-sum security game can be played repeatedly at every time step $k$.

Figure 5.2: A team of UAV collaborate on a mission. They can communication
with each other when one is in the range of communication of the other. An
attacker can jam the signals between two UAV.

In transportation networks, the class of interdiction games is similar to the
jamming games in communications. One player (e.g., an attacker) aims to remove
the links of a network to minimize the throughput or disrupt the operation of
the infrastructure subject to resource constraints. In other words, the
attacker’s capability is assumed to be bounded and he can only remove a small
subset of links in the network. The other player (e.g., planner or defender)
aims to design a robust network and invest resources to protect against the
attacks on the network and maintain the service of the infrastructure. This
type of games has been commonly used in scenarios of the infrastructure
protections [18, 51, 27], multi-agent robotic systems [88, 24, 26], and IoT
networks [17, 19].

Another example of security game is the system configuration game [168, 167,
182]. In this game, we consider one system defender and one attacker as two
players. The system defender configures its network and IoT in Fig. 5.1 by
choosing the setting of the software, security rules/policies, and network
topologies. Each system configuration inevitably has known or zero-day
vulnerabilities. An attacker aims to find the vulnerabilities of the entry-
point system and exploit them to penetrate and infect further parts of the
system. Let $\mathcal{C}=\\{c_{1},c_{2},\cdots,c_{m}\\}$ be the set of
configuration that the system can choose from. Let $\mathcal{V}$ be the set of
vulnerabilities that the system can have. Each configuration is associated
with a subset of vulnerabilities of $\mathcal{V}$. We let
$\pi:\mathcal{C}\rightarrow 2^{\mathcal{V}}$ be the point-to-set mapping
between configurations and the subsets of vulnerabilities;
$\pi(c),\subseteq\mathcal{V},c\in\mathcal{C},$ is called the attack surface
when the system is configured to $c$. An attacker can choose an attack that
exploits several vulnerabilities of the system. Let
$\mathcal{A}=\\{a_{1},a_{2},\cdots,a_{n}\\}$ be the set of attack actions. Let
$\gamma:\mathcal{A}\rightarrow 2^{\mathcal{V}}$ be the point-to-set mapping
between attack actions and the subset of vulnerabilities;
$\gamma(a)\subseteq\mathcal{V},a\in\mathcal{A},$ is the set of vulnerabilities
exploited by the attack action $a\in\mathcal{A}$. When one of the
vulnerabilities exploited by the attacker is in the attack surface under
configuration $c$, then the attacker is successful and receives a reward. More
formally, when $\gamma(a)\cap\pi(c)\neq\emptyset$, the reward to the attacker,
which is also the loss to the defender, is given by $R(\gamma(a)\cap\pi(c))$,
where $R$ is a set-valued function that quantifies the impact of the
successfully exploited vulnerabilities. This configuration game is a normal-
form zero-sum game. An example of this game is represented by the following
matrix:

H:  $c_{1}$ $c_{2}$ $c_{3}$ $c_{3}$ $a_{1}$ $h_{11}$ $h_{12}$ $h_{13}$
$h_{14}$ $a_{2}$ $h_{21}$ $h_{22}$ $h_{23}$ $h_{24}$

Here, the row player is the attacker with $2$ attack actions. The column
player is the defender with $4$ configurations. The reward/loss to the players
are described by the matrix entries $h_{ij},i\in\\{1,2\\},j\in\\{1,2,3,4\\},$
which are the rewards to the attacker when he uses $a_{i}$ to attack and the
defender uses configures the system at $c_{j}$. The defender can relies on
this model and assesses his best-effort worst-case security. The saddle-point
equilibrium of this game yields a game value that quantifies the level of the
security under the best-effort of the defender. It also leads an insight for
the defender on how to choose a secure configuration to safeguard the system
for a prescribed attack model.

The analysis of the saddle-point equilibria of the security game has the
following implications. First, the equilibrium strategies provide a security
strategy for the defenders and protect the system in the worst-case scenario
that is assumed by the defender. Such strategies are computed ahead of time.
The operator can use them to maintain the connectivity of the UAV at each time
$k$ robust to the worst-case adversarial behaviors within a range of attack
behaviors. In many cases, the exact knowledge of the worst-case may not always
be available. The overestimate of the capability of the attacker will result
in a conservative solution while the underestimate will lead to a successful
attacker and failure in the operation when the attack is not correctly
anticipated. There is a need to consider strategies other than protections or
preventions to safeguard the system. One type of strategy that can be built on
top of the robust mechanism is the resiliency mechanism. In the case of the
underestimate, the system is well prepared and designed to quickly recover
from the attack. In the case of the overestimate, the resources used to
strengthen the network for extremely low likelihood events can be used for the
repair of the links and the restoration of the services. With limited
resources, the defender needs to find an optimal tradeoff between the
robustness and the resiliency to mitigate the impact of the attacks and
maintain an acceptable level of system performance. This joint robust and
resilient mechanism has been studied in [18] and applied to multi-agent
robotic systems in [89, 88].

Second, the value of the game obtained from the equilibrium analysis provides
a predicted outcome and performance of the system. It provides a worst-case
performance guarantee and a quantified assessment of the risks. In the example
of UAV networks, the solution to the zero-sum game from solving (5.2) provides
a way for the designer to assess whether the network is still connected under
the worst-case adversary. If it is, the designer can assess the security
margin from being disconnected. Otherwise, the designer needs to find
mechanisms other than the control variable $u(k+c)$ to strengthen the network.
For example, instead of using mobility to create connectivity, the designer
can introduce additional communication resources, e.g., construction of ad hoc
base stations, or the use of satellite communications. This design choice is
another layer of optimal planning of resources since additional mechanisms are
also constrained by limited resources. In [182], the authors see the value of
games as the security capacity of a system. This is because when the computed
value is below the targeted value, it means that it is impossible for the
system to be secure for the given attack model unless additional resources are
invested in the system. Games have also been studied for the overall design of
secure communication layers as networks by [109].

###  5.3 Multi-Stage and Multi-Phase Games

In Section 5.2, we have presented game theory as a tool to understand
cybersecurity. In this section, we extend the game-theoretic technique
developed for cyber attacks and connect it with the physical models of robots.
The target of many APT is to create malfunction of the physical assets,
including a power plant, an autonomous vehicle, or a water treatment plant. By
incorporating the physical models into the security game framework, we can
provide a cross-layer security framework for robots and develop tailored cyber
protection for the given robot systems that have specific operational system
specifications and requirements.

To illustrate this concept, we use a generic nonlinear dynamical system in
(5.3) to capture the mechanical behaviors of the robots. Let $x(t)$ be the
state of the physical system and $y(t)$ be the output of the system. The
physical dynamics of the robot systems, such as mechanical arms, walking
robots, UAV, can all be written into the following form:

$\displaystyle\dot{x}(t)$ $\displaystyle=$ $\displaystyle
f(t,x,u;\theta(t,a,d)),$ (5.3) $\displaystyle y(t)$ $\displaystyle=$
$\displaystyle h(t,x,u;\theta(t,a,d)).$ (5.4)

Here, $f$ and $h$ are continuous functions in $(t,x,u)$. The physical system
is controlled by the feedback law $u$ to achieve stabilization or targeted
performances. $\theta(t,a,d)$ is the cyber state of the robot. It can
represent the state on the attack graph or the high-level description of the
well-being of the cyber system. The state of the cyber system is influenced by
the attack strategies $a$ and the defense strategies $d$. A well-designed
defense can reduce the probability of the system in a compromised cyber state
and allow the cyber system to recover quickly once it is attacked. From (5.3),
it is clear that the cyber defense and attack not only directly affect the
cyber state but also indirectly creates an impact on the physical system. For
example, when the attacker gains access to the ROS nodes, he can modify the
control logic and turn the robot into a deadly weapon [31, 154]. In the
scenarios of multi-agent systems, one robot can be misled by a compromised
robot to put the team into jeopardy and fail the mission [148, 173].

Figure 5.3: Multi-stage and multi-phase interactions between an attacker and a
defender: The attacker changes the cyber state $\theta$ to affect the physical
state $x$ at the last stage of Phase $3$.

The goal of the extended game framework is to capture this impact so that the
defense designed at the cyber layer will reduce the cyber-physical risks and
the control designed at the physical layer will be able to quickly mitigate
the physical damages when an attacker succeeds at the cyber layer. To capture
these multiple layers of effects, authors in [177, 54, 127] have created a
multi-stage and multi-phase game model. The entire attack process is
decomposed into multiple phases that represent multiple rounds or stages of
interactions between the attacker and the system at different layers. At Phase
1, the attacker aims to create social engineering approaches to infect the
system. To defense against this attack, defenders can raise security
awareness, provide training to users and employees, or developing incident
documentation and alert system to prevent malicious outsiders from entering
the system or the insider to behave abnormally.

At Phase 2, the attacker aims to maximize the infection, search for its
targeted asset and get closer to it. The defender at this phase can leverage
spot-checking to detect virus/malware, change system configurations, or
develop proactive defense mechanisms (e.g., honeypots [60, 83] and moving
target defenses [171, 59]) to reduce the system risks. At Phase 3, the
attacker aims to create physical damage on the system on the asset. It is
already late for the defender to prevent the asset at this stage from damages.
However, the defender can detect anomalous behaviors and reconfigure the
control at the physical layer to reduce the impact of the attack and develop
mechanisms to recover the system from the attacks.

The multi-stage multi-phase interactions are illustrated in Fig. 5.3. Each
phase contains several stages of interactions. The success of an attacker in
one phase will lead him to the next phase until he takes over the control of
the physical assets. The state of the cyber system $\theta$ evolves over these
multi-round interactions. In Phase 3, a compromised cyber state will influence
the physical state $x$. The control taken at the end of Phase 3 can mitigate
the physical impact of the attacker.

Figure 5.4: Illustration of FlipIt games: The attacker and the defender
compete to control a shared resource. Both players can choose when to move at
any time. Each move incurs a cost. The player controls the resources for a
period of time after his move till the next move of the other player.

Each phase has unique attacker-defender interactions. They can be modeled
using a suitable game-theoretic framework. In the first phase, the game often
involves a human user and an attacker. The goal of the attacker is to use
social engineering techniques to deceive the users to gain credentials for
access. In [140], FlipIt games have been proposed to understanding many
cybersecurity scenarios. Consider the scenario where a user can choose when to
change his passwords and an attacker can choose the time to hack the account.
A weak password that has not been changed for a long time can be eventually
leaked to the attacker. One way to protect a user’s account to frequently
change the password. However, it would create a perceived overhead if a user
changes the password too frequently, and [115] gives a game model to find an
optimal tradeoff between security and usability here. From the attacker’s
perspective, there is a cost for him to gain reconnaissance and hack the
account. FlipIt games capture the strategic decision of both players. The game
analysis provides a risk assessment of the system and the development of
defense strategies. The applications of FlipIt games have been extended to
many applications including cloud computing [98, 149, 22], cybercrime [13, 7],
and IoT systems [23, 94].

In the second phase, an intelligent attacker can move stealthily and
strategically in the network to gain access to the targeted asset. APT are
this type of threat that is capable of customizing their strategies against
specific targets and disguise themselves for a prolonged period. Once the APT
attackers enter the system, they escalate their privilege and propagate
laterally in the network, compromising other nodes to gain deeper access to
find their target. The goal of the defender is to detect the compromise nodes
and respond quickly to prevent the attacker from going deeper and reaching
critical assets. A game modeling this type of interactions is Cut-The-Rope
(Section 5.1), but other models have also been proposed, using sequential
games [57, 55, 87]. One important application of these models is to develop
proactive defenses. They provide a precautious and strategic way to increase
the cost of attack while mitigating the potential damage attacker could bring
to the final target. An effective proactive response system can delay the
attack and give network administrators a sufficient amount of time to
meticulously analyze data and deploy effective responses to the threats.

In the third phase, an attacker has successfully gained access to the critical
asset and aims to create maximum impact. The goal of the defender in this
phase is to reduce the damages that can be created by the attacker. An example
of games that capture this scenario is the Flip the Cloud game described in
[98]. An APT attacker can take hold of the cloud and sends falsified
information to mislead a robot that relies on the computations in the cloud.
The analysis of the game between the cloud that is taken over by the attacker
and the system leads to a strategic trust mechanism [94] that can filter and
reject misleading information and an event-triggered control mechanism [149]
to switching the control laws to maintain an acceptable level of performance.
Here, the goal of physical control is to strengthen the resiliency of the
robots. With a suitable design, the robots can still carry on their missions
and complete their tasks despite the compromised cyber state and the
unanticipated events. The resilient control problem has been discussed in
[179, 128, 129]. Game-theoretic techniques to achieve resiliency of the
control system performance have been studied in [174, 57, 170, 18, 186].

Generally, it is advisable to consider APT models relative to what the
adversary tries to accomplish in the long run, as [124] distinguishes two
types of APT:

  * •

One type is about _gaining long-run control_ over the victim, but without
ultimately destroying it. This can be the case when an industrial robotics-
enhanced production line is hacked for the purpose of quality dropout
increase, or to induce flaws in the products, up to inserting malicious parts
or similar. Other scenarios include the overtake of an infrastructure of
unintended purposes, e.g., cryptocurrency mining or similar. FlipIt is a class
of game models to defend against this type of APT.

  * •

The other type aims at _killing the victim_ , which entails a slow and
ubiquitous penetration staying beneath the detection radar so that it is too
late for the defender to react when the attacker becomes visibly active.
Examples of such incidents have been reported on large critical
infrastructures, with Stuxnet being an early and famous example. Game models
for this type of APT are, among others, Cut-The-Rope.

####  5.3.1 Signaling Games for Multi-Phase Security

In the security games across the three phases, the players often have
incomplete information regarding the payoffs, action sets, and the type of
opponents the players interact with. It is essential for security games to
capture these uncertainties in the game. Signaling games are a common class of
games that have been used to model the sequential interactions between two
players under incomplete information. They have been used in many applications
such as cyber deception [103, 96, 188, 97, 99], communication networks [108,
14], and trust management [15, 82, 102]. In this class of games, one player is
the sender, denoted by $S$, and the other player is the receiver, denoted by
$D$. The sender has private information $\theta\in\Theta$ unknown to the
receiver and sends a signal111The literature also uses the term “message” in
the context of signalling games, which we avoid here to prevent ambiguities
with the term “message” as data in transit like in Chapter 2.
$m\in\mathcal{M}$ to the receiver. The goal of the receiver observes the
signal $m$ and chooses an action $a\in\mathcal{A}$ to respond to the signal so
that his reward $U_{S}(\theta,m,a)$ is maximized. The goal of the sender is to
pick a signal that will lead to a desirable action chosen by the receiver so
that his reward $U_{R}(\theta,m,a)$ is maximized. Both players have the
knowledge of how this game is played. More specifically, the players know the
reward functions and action sets of both players. The private information
$\theta$ is modeled as a random variable. Both players have knowledge of the
distribution of the random variable. However, only the sender knows the
realization of $\theta$.

This game is illustrated by an extensive-form game in Fig. 5.5. Nature first
chooses $\theta$ according to the distribution known to the players. The
sender who observes $\theta_{1}$ or $\theta_{2}$ will pick a signal
$m\in\\{m_{1},m_{2}\\}$. The receiver cannot distinguish between the type of
the players (indicated in the figure by the information set of the receiver)
but can only choose an action $\\{a_{1},a_{2}\\}$ based on his observation of
the signal. The strategies of the players are described by the policies
$\mu_{S}:\Theta\rightarrow\mathcal{M}$ and
$\mu_{R}:\mathcal{M}\rightarrow\mathcal{A}$ that are determined prior to the
start of the game. The players use the policies to determine their actions
based on their private observations. Bayesian perfect Nash equilibrium is
commonly used as the solution concept for the signaling games. An equilibrium
profile $(\mu^{*}_{S},\mu^{*}_{R})$ is a Bayesian perfect Nash equilibrium if
it satisfies sequential rationality and there exists a consistent belief
system, a distribution over the information set, that supports this
equilibrium profile. Readers can refer to the mathematical details in [42] for
the analysis and the computation of the equilibrium.

Figure 5.5: Signaling games between one sender and one receiver. The sender
has private information $\theta$ and sends a signal
$m\in\mathcal{M}=\\{m_{1},m_{2}\\}$ to the receiver to achieve an outcome that
optimizes his reward. The receiver determines action
$a\in\mathcal{A}=\\{a_{1},a_{2}\\}$ to maximize his reward. The dotted line
indicates an information set of player 2.

Signaling games can be used to capture information asymmetry, where one player
has more information than the other player. It is a pervasive phenomenon in
cybersecurity. Across the three phases depicted in Fig. 5.3, the system
defender may not distinguish the attacker from the normal users. In contrast,
the attacker can observe the behaviors of the system. In [101], signaling
games have been used to model phishing. An attacker sends a phishing email to
a population of receivers while a user relies on spam and scam detection
systems to filter out a suspicious email from the primary inbox.

An extension of the signaling games to multiple rounds of interactions has
been studied in [38, 57]. The multi-round game models are used to study the
Phase 2 interaction where the attacker aims to escalate his privilege and gain
access to the targeted asset. In [148], a trust mechanism based on signaling
games has been developed for UAV at Phase 3 as the last shield to defend
against the attacker. Once an attacker has an access to the remote control
station, he can send a falsified control command to direct the UAV to hit a
building. The trust mechanism enables the UAV to make onboard decisions of
following or rejecting the command when they predict that following the
command would lead to catastrophic consequences.

Figure 5.6: G$1$, G$2$, and G$3$ represent games at three phases. The three
games are nested. The outcome of the game at earlier phases will affect the
structure of the game in the later phases. The defense strategies need to be
planned backward from the last phase.

####  5.3.2 Games-in-Games Model

The games in the three phases are interdependent. The actions chosen by the
defender and the attacker in the first phase will affect the cyber state and
the structure of the game played in the second phase. When planning the
defense at the first phase, it is essential to understand its consequences on
the following phases and make an effective planning decision at the first
phase. The games at the three phases can be integrated into a game-in-games
[57, 172, 150, 26, 89, 154, 152], in which the game at an earlier phase is
nested in the game at a later phase. Illustrated in Fig. 5.6, the game-of-
games integration gives a holistic view of the security issues across multiple
layers of robotic systems and provides a cross-layer risk assessment and
design methodology of security mechanisms.

Security games for sophisticated attacks often require an integrated model
that composes interactions at different layers, stages, or phases of the
system. The game-in-games leverage the sequential nature of the cyber attacks
and provide a framework to compose local-stage games into an integrated large-
scale game for a holistic analysis of the risks. The computation of the
equilibrium solutions at each phase is backward. The defense strategies in
Phase 1 depends on the defense strategies in Phase 2, which is determined by
the strategies in Phase 3. This backward computation will guarantee that the
defense strategies are strategically optimal across the phases rather than
myopically optimal within one single stage. Readers can refer to the recent
book [187] for a comprehensive introduction of the game-theoretic techniques
for cross-layer designs.

####  5.3.3 Resilient Control Mechanisms and Real-Time System Performance

In Section 5.3, we have used a multi-stage and multi-phase game to capture how
an attacker moves from the cyber layer to the physical layer. The physical
layer of robotics consists of the real-time dynamics represented by the system
model in (5.3). It also corresponds to level $0$ of the ROS architecture,
illustrated in Fig. 3.2. The defense at the physical layer heavily relies on
the resilient control mechanisms when the attacker has successfully taken
control of the devices at the field network. The purpose of resilient control
is to enable the robotic systems to maintain a satisfactory level of
performance when the robotic system is attacked by unanticipated threats in
real-time. An example of such resilient control mechanisms is introduced in
[149] for cloud robotics. A UAV that relies on the cloud for communication and
information processing can switch from an optimal mode of operations to a safe
mode when a man-in-the-middle attack is detected.

As discussed in [166], resilient control is divided into three stages: ex-ante
planning, interim execution, and ex-post recovery. The ex-ante stage is the
resilience planning that designs contingency plans to prepare for the
anticipated attacks. The interim execution stage is the operation stage of the
control system, which executes the resilience plans in real-time. A resilient
operation includes online learning for anomaly detection and adaptive
decision-making for responding to the anomaly. The ex-post recovery refers to
the recovery process in which the robots can still maintain critical functions
or heal themselves to complete the tasks.

The three-stage resilient control mechanism is the last resort to safeguard
the robotic systems and mitigate the impact of physical damages. This approach
is complementary to the cyber defense designed at the penultimate level to
prevent an attacker to reach the final level. Perfect security is not
practicable in real-time systems as it would significantly increase the cost
and reduce the usability of cybersecurity and resilient control mechanisms can
be designed jointly to effectively reduce the impact of cyber threats. The
cyber defense in the joint design needs to anticipate the consequence when the
attacker successfully evades the defense and reaches the physical asset.
Meanwhile, the design of a resilient control mechanism needs to take into
account the effectiveness of the cyber defense and design resiliency in
response to possible successful attacks. This joint design methodology aligns
with the games-in-games defense paradigm introduced in Section 5.3.2. The
resilient control is subsumed in the last stage design, or G3 in Fig. 5.6,
while the cyber defense is viewed as the outcome of G1 and G2. In [170, 169],
resilient control is viewed as a game between the controller and the worst-
case scenario that can occur to the real-time system. Therefore, the games-in-
games design paradigm provides a holistic view to understand the impact of
cyber defense on the real-time system performance and design cross-layer
defense and resilient control mechanisms.

###  5.4 Examples of Game-Theoretic Analysis

We provide two case studies to elaborate on the application of game theory to
robot security. The first one introduces the application of signal games to
UAV and develops a cyber-physical trust interface between the IT-level signals
and the OT-level operations and controls. The second one continues the example
described in Fig. 5.7 and discusses how to design control mechanisms that can
fend off jamming attacks while maintaining connectivity.

####  5.4.1 Signaling Games and UAV

This case study presents a team of multi-agent UAV with $n$ autonomous agents
(ASs) and a control station (CS). Each agent has two components. One is the
physical layer which implements real-time control to achieve its control
objectives. The other one is the cyber layer which sends information and
signals to the agents as inputs for the controller. At the physical layer, a
min-max model predictive control (MPC) problem is formulated to handle the
worst-case disturbances based on the model. For AS agent $i$ at time $k$, the
problem is formulated as a zero-sum game between the controller and the
disturbance:

$\displaystyle\mathcal{P}_{k}^{i}:\min_{\hat{u}^{i}_{k}}\max_{\hat{w}^{i}_{k}}\
\ \ J_{c}\left(x_{k}^{i},r_{k}^{i},\hat{u}^{i}_{k},\hat{w}^{i}_{k}\right).$
(5.5)

Here, $J_{c}$ is the accumulated stage cost until horizon-window $N$;
$x_{k}^{i}$ is the state vector; $r_{k}^{i}$ is the reference trajectory given
by the CS; $\hat{u}^{i}_{k}$ and $\hat{w}^{i}_{k}$ are the estimated control
and disturbance vectors. An adversary can fabricate a fake reference signal
$r^{i}$ to deviate agent $i$ from its real trajectory to achieve Suicidal
Attack (SA) or Collision Attack (CA).

At the cyber layer of ASs, we use a signaling game method to capture the
information asymmetry and multi-stage behaviors of these players. The CS
(sender $S$) has a binary private type $\theta$ denotes whether $S$ is normal
or malicious. $S$ sends a signal $r^{i}$ to each AS (receiver $R^{i}$). Before
choosing action $a^{i}$, AS updates its beliefs about the type $\theta$ using
Bayes’ rule and prior belief $p^{i}(\theta)$. The goal of $R^{i}$ is to choose
an action $a^{i}$ to minimize its expected cost $c_{R}^{i}$ given a posterior
belief $\mu^{i}(\theta|r^{i})$, while the goal of the sender is to choose a
signal $r^{i}$ to minimize the cost $c_{S}$ by anticipating the behavior of
the receiver $R^{i}$. The game admits a PBNE, which is a strategy profile
$\\{\sigma(S),\sigma_{R}^{i}\\}$ and posterior beliefs $\mu^{i}(\theta|r^{i})$
such that

$\displaystyle\forall\theta,\quad\sigma_{R}^{i}(r^{i})\in\arg\min_{a^{i}}\sum_{\theta}\mu^{i}(\theta|r^{i})c_{R}^{i}(r^{i},a^{i},\theta),$
(5.6)
$\displaystyle\forall\theta,\quad\mathbf{\sigma}_{S}(\theta)\in\arg\min_{\mathbf{r}}c_{S}(\mathbf{r},\mathbf{\sigma}_{R},\theta)$
(5.7)

where posterior beliefs $\mu^{i}(\theta|r^{i})$ are updated according to
Bayes’ rule. There are two PBNE that exist in this cyber-physical signaling
game. One is a separating equilibrium, and the other is a pooling equilibrium.
Both equilibria can lead to the protection of ASs from collisions as the
equilibria can guarantee that $R^{i}$ only accepts reference trajectory
$r^{i}$ if it is out of the danger zones. The designed framework yields an
intelligent control of each agent to avoid collisions. Illustrated in Fig.
5.7, a group of UAV reject the falsified command and switch the system to a
safe control mode. The UAV hover in the air and keep a safe distance from each
other and the building. The results indicate that the integrative framework
enables the co-design of cyber-physical systems to minimize the damages,
leading to online updating the cyber defense and physical layer control
decisions. Interested readers can refer to [148] for more details on this case
study.

Figure 5.7: Trust mechanism implemented in the UAV control system. The UAV
start to hover before they hit the building.

####  5.4.2 Jamming Games and Multi-Agent Systems

Multi-agent systems provide a framework for studying distributed decision-
making problems as a number of agents make local decisions by interacting with
each other over networks. One of the common security threats in networked
systems is jamming attacks. The adversary can simply transmit interference
signals to interrupt communication among agents. Non-cooperative game theory
approaches can be used to find the optimal defense mechanism to prevent and
restore the network from successful attacks.

We model the interaction between an attacker and a defender in a two-player
two-stage game setting. The attacker is motivated to disrupt the communication
by attacking individual links. The attack model consists of a jammer who
chooses the links and the durations of the attack with the knowledge of the
communication graph of the UAV and the energy constraints. The defender can
recover a subset of links that are important for maintaining the connectivity
of the graph with limited energy.

In the game, both players attempt to choose the best strategies to maximize
their own utility functions. The utilities for the attacker $U^{A}$ and the
defender $U^{D}$ are defined as the total generalized edge connectivity (with
the negative sign for the attacker), plus the cost for jamming (attacker) or
recovering (defender). The two-stage game is played as follows. The jammer
first attacks and then the defender recovers in the subgame. Let $m^{A}$ be
the attacked edges and $\sigma^{A}$ be the attack intervals; let $m^{D}$ be
the edges recovered and $\sigma^{D}$ be the recovery intervals. The strategies
of the attacker and the defender are in terms of $(m^{A},\delta^{A})$ and
$(m^{D},\delta^{D})$, respectively.

The subgame perfect Nash equilibria are obtained using backward induction.
Given the attacker’s strategy $(m^{A},\delta^{A})$, the defender decides the
best response strategy as

$\displaystyle\left(m^{D*}(m^{A},\delta^{A}),\delta^{D*}(m^{A},\delta^{A})\right)\in\arg\max_{(m^{D},\delta^{D})}U^{D}((m^{A},\delta^{A}),(m^{D},\delta^{D}))$
(5.8)

Likewise, given the initial network graph $\mathcal{G}$, the attacker decides
the strategy as

$\displaystyle\left(m^{A*},\delta^{A*}\right)\in\arg\max_{(m^{A},\delta^{A})}U^{D}((m^{A},\delta^{A}),\left(m^{D*}(m^{A},\delta^{A}),\delta^{D*}(m^{A},\delta^{A})\right))$
(5.9)

This game can be applied to a multi-agent consensus problem, where the game is
played repeatedly over time. In such a case, the energy constraints are
extended to satisfy continuous communications. Fig. 5.8 shows the states of
the agents and properties of the players, with the agents achieving
approximate consensus at $t\approx 4$ with tolerance $\epsilon=0.5$. This
framework enables the study of how the attacks and recovery strategies affect
the consensus process of the multi-agent systems. By analyzing the games, we
can find the optimal strategies for the attacker and the defender in terms of
edge connectivity and the number of connected components of the graph.
Interested readers can refer to [88] for more details on this case study.

Figure 5.8: The state trajectories of the UAV. The green areas indicate the
intervals where the defender recovers. The red areas indicate the intervals
where the attacker attacks. The four agents reach consensus after $t\approx 4$
[88].

##  Chapter 6 Discussions and Conclusions

Securing robot systems has its unique challenges, since their interaction with
the world is virtual (related to information) and physical, which extends the
usual threat landscape considerably. Consequently, the tools to address
security need to meet the diversity of threats, and game theory, applied to
security scoring systems, can provide a powerful mechanism to orchestrate and
assemble security mechanisms that each cover their specific threat spectrum,
but which only in totality can provide comprehensive protection.

The steps taken in this book are only preliminary and yet point out a gap
between what theory can offer and what robot designers could use in the
future. Since systems are heterogeneous and with components from many vendors
combined, it can be tempting and easy to just delegate responsibility to
somebody else. This is an issue on the organizational level, and risk
management standards can be very helpful here to address issues of ownership,
responsibility, and incident management. The complexity of bringing a
development project into standard compliance is yet another motivation to
employ optimization, such as game theory.

The complementation of technical security mechanisms by adequate
organizational precautions pervasive throughout the whole robot life cycle is
an issue that we only touched lightly here, but demanding more in-depth
research in the future. The problem with robotic security may partly be
attributed to the lack of responsibility assignment when it comes to an
incident.

##### Security and Performance Tradeoffs

One important challenge to address with robotic security is to tradeoff
between adding security and the performance requirements in the overall
system. Real-time processes will need to account for it to To harden the
security in robotic systems. For example, the real-time control loops can be
subject to stochastic latency due to the addition of encryption and access
control mechanisms. To cope with it, the robot will require additional
computational resources. The off-the-self and traditional solutions would not
work for all robots. It is essential to tailor the security solutions for
different robot application domains and take into account the performance
specifications. The security solutions for a teleoperated medical robot should
be significantly different from the ones for a domestic cleaning robot. The
security models and the threat consequences are drastically different in two
cases.

There is a need to prioritize the security objectives and develop bespoke
frameworks for the system-specific tradeoffs and designs. Such priorities can
be added to a model as importance scores (see, e.g., [118]), or with explicit
rankings among the goals. One such extension towards the latter is
lexicographic optimization as described in [66, 126]. Quantitative metrics and
design methodologies play an important role to achieve these objectives. One
important future direction is for robot designers to develop customized
metrics and methodologies to understand the security-performance trade-offs
and the design of optimal resource-constrained solutions. The specification of
metrics and quantification of risks, thereby induces an operational difficulty
perhaps, since engineers but also security specialists may find it difficult
to quantify security for an optimization. Likewise difficult is the general
specification of probability parameters as appear throughout the majority of
stochastic models, not only to describe robot security.

Helping robot designers with security requires more than just proposing yet
another security model, but also helps the practitioner to reason about how to
instantiate the models for their use. Work in this direction is relatively
scarce, but the problem of systematic parameter learning is addressable by
machine learning techniques. See [123] for an example application in the
context of critical infrastructures that are transferable to robots as
infrastructures too, or [63] and [114] for online learning and reasoning about
the trustworthiness of components in a joint system. Further help is offered
by scoring systems like RVSS, as these provide a systematic tool to quantify
security and, as [67] describe, also get ideas about how to specify
probabilities if a stochastic model or decision making requires them. This can
be complemented by other than numeric quantification techniques, such as
graphical risk specification as proposed by [145].

Security defense is often an add-on solution in today’s robotic systems.
Oftentimes, the security solutions are based on traditional and off-the-shelf
solutions, e.g. cryptography, firewalls, and intrusion detection systems.
Advanced defense strategies, such as cyber deception and moving target
defenses, will require a careful evaluation of the threat models and
additional system resources to enable such defenses. Without a deliberate
built-in design, our robotic systems will always be in a vulnerable state as
the attacker can eventually map out the system and launch successful attacks.
Built-in defense mechanisms aim to outsmart and deter the attacker by
leveraging the system resources to introduce uncertainties and make the attack
more costly. Including uncertainty in optimization is its issue but doable
with game models that adopt a more complex payoff modeling than crisp numbers.
Specifically, it is possible to optimize actions for defense and resource
investment when consequences are uncertain [113], even in light of multiple
conflicting goals [112], interdependencies and network effects [157, 21, 25,
27, 81].

##### Security vs. Safety

This book has discussed the cybersecurity frameworks and models for robotic
systems. It is essential to distinguish security from safety and reliability,
which have been relatively well studied in the robotics literature. The first
key difference is that security is an issue strategically created by an
adversary. The safety issues are often related to natural causes. Some of them
can exceed expectations but they are not associated with objectives and
malicious intentions. Often, we tackle the safety issues by specifying a
tolerable set of uncertainties and design systems under the worst cases among
these uncertainties. The attack is an outcome of the purposefully planned
actions and the exploitation of the vulnerabilities. We need to understand the
attack models through the objectives, the incentives, and the capabilities of
an attacker when developing security solutions for robots.

Second, the impact of the damage created by an attacker may not directly
observable at the physical layer at an early stage of the compromise. Safety
often refers to the last-mile physical protection at the OT-level. It is often
too late when the attacker succeeds in penetrating the cyber layers, controls
the physical assets, and can manipulate them at his will. Security defense
goes beyond the OT and protects the system at the IT-level. In this book, we
have described the challenges and quantitative methods that can be used to
address the IT-level security and its induced impact on the OT. Safety and
security issues are not mutually exclusive. They can be treated together
within a holistic framework that considers the cross-layer effects. Ensuring
IT-level security is an important step toward improving the safety of the
system, especially when major OT-safety concerns arise from IT-security.

##### Emerging Attack Models and Defense Solutions

This book has presented several attack models and solutions to counteract
them. There are many emerging threats and advanced techniques that would be of
interest to investigate. For example, adversarial machine learning is an
increasingly important topic. Many robotic systems rely on learning models for
pattern recognition, detection, and perception of the environment. An attacker
can manipulate the input data and mislead the robot to erroneous learning
results [58, 160, 159, 158, 64]. This attack can lead to misinformed decisions
and control, which would result in catastrophic consequences. It is imperative
to assess the trustworthiness of learning models and develop contingent
solutions when the learning is not trusted.

New technologies in robotics also inspire new attack models. For example,
cloud robotics is a new paradigm of robotic systems that integrate the
technologies of cloud computing and storage into robotics [65]. It empowers
the robots with the powerful computation, storage, and communication resources
in the cloud and enables information sharing and communication among a group
of robots and devices. However, the confidentiality and the integrity of the
data communicated between the cloud and the robot can be compromised by an
attacker. Furthermore, an attacker at the cloud can falsify the computations
to mislead the robots or create a denial of service so that the robot does not
have sufficient inputs to act in an unknown environment [98, 153, 149].

New attack vectors and more sophisticated attackers would galvanize the
defender to develop new defense solutions. One promising direction of cyber
defense is the deception technology, which employs decoys (e.g., honeypots) or
introduces uncertainties (e.g., moving target defense) to deceive, detect, and
deter the attacker. Deception technologies provide a proactive way to defend
against zero-day and advanced attacks and enable an automated way to respond
in real-time to the threats. Design of deception techniques often relies on a
clear understanding of the system tradeoffs involving resource constraints,
security objectives, and attack models. Game theory has been used as a primary
tool to address this tradeoff and develop an optimal cyber deception mechanism
[103]. Interested readers can develop new security solutions for robots by
making connections between these advanced cyber defense solutions with the new
attack threats in robotics.

Beyond the technical solutions to security issues in robots, economic policies
and tools can also be used to mitigate their adversarial impact on society.
Cyber insurance is such a product that protects owners and users of the robots
from cyberattack-induced damages. The coverage of cyber insurance allows the
risks to be transferred and distributed fairly at the cost of premiums.
Damages such as injuries, collisions, theft, and extortion can be possibly
covered by the insurance. The premiums and the incentives of the insurance
need to be carefully designed to reduce moral hazards and increase social
welfare. Design methodologies of insurance design developed in [163, 10, 47,
162] can be applied and customized to different robotic applications in the
future as an additional layer of risk protection.

##### Bridging Game Theory and Practice

Chapter 5 has provided an overview of the game-theoretic methods and their
applications in cybersecurity and robotics. We have seen that game-theoretic
frameworks can capture the defense mechanisms and the attack models. The games
take different forms to describe the distinct features at a specific layer of
the robotic system. The formulation of the game models builds on the system
designer’s knowledge and assumption about the attacker. The assumption of the
attack model may not perfectly align with the practice. One important reason
is that the designer and the attacker have asymmetric information about each
other. Furthermore, the players may not act rationally even if the game is
known to both. These questions are reasonable concerns when we apply the
solutions from idealized game models. The idealized models provide a canonical
form of descriptions. Many sophisticated methods can enrich these models to
provide practical security solutions.

One method to enrich the baseline game models is reinforcement learning (RL).
The defender can learn and react to the attacker’s behaviors in real-time. The
RL does not require the defender to know the games ahead of time but uses his
observations to adapt his strategies without knowing the underlying model. In
[55] has developed RL algorithms to assimilate the data collected by honeypots
to create an attack model and learn about the attacker’s intention and
capabilities. [184] and [185] have also presented several RL mechanisms which
are used to model different styles of learning in terms of rationality and the
intelligence of the learner. They can be used to capture human factors such as
constraints on cognition, perception, and reasoning.

RL techniques have also been used as part of the OT to control and monitor
robots in real-time. The OT-level RL allows the robots to learn the cyber-
induced changes in the physical systems and respond to them to achieve agility
and resiliency (e.g. see [170, 57, 169]). It is possible to compose the RL
algorithms at IT and OT levels to achieve holistic security learning and
monitoring of the robotic systems.

Besides RL, the baseline game models can be enriched by directly incorporating
information incompleteness. In large-scale finite security games, it is not
practical for the players to know every entry of the payoff matrix. The
players can estimate the unknown payoffs by leveraging information from
historical or real-time plays [84, 92, 104]. For example, [92] has presented a
gradient method to estimate the payoff matrices by finding the closest one to
the game matrices played in the past. Incorporating uncertainties and bounded
rationality into game models is a major step toward bridging game theory and
practice. This cross-disciplinary approach will benefit from fruitful
collaborations between game theorists, cybersecurity experts, and roboticists.

## Acknowledgements

The authors would like to thank the support that we receive from our
institutions. We thank many of our friends and colleagues for their inputs and
suggestions.

Stefan Rass, Bernhard Dieber and Víctor Mayoral Vilches thank Endika Gil
Uriarte, Martin Pinzger, Nacim Ramdani, Alcino Cunha, Francisco Rodriguez Lera
and Roberto Guzman for invaluable discussions and suggestions on the DevOps
and DevSecOps cycle in the robotics context, as outlined in this book.

Special thanks from Quanyan Zhu go to the members of the Laboratory of Agile
and Resilient Complex Systems (LARX) at NYU, including Jeffrey Pawlick, Juntao
Chen, Rui Zhang, Tao Zhang, Linan Huang, Yunhan Huang, and Guanze Peng. Their
encouragement and support have provided an exciting intellectual environment
for us where the major part of the work presented in this book was completed.
Quanyan Zhu would like to acknowledge support from several funding agencies,
including the National Science Foundation (NSF), Army Research Office (ARO),
and the Critical Infrastructure Resilience Institute (CIRI) at the University
of Illinois at Urbana-Champaign for making this book possible.

Materials in this book were in part first presented at the Workshop on
Security and Privacy in Robotics at International Conference on Robotics and
Automation (ICRA), held virtually from May 31 – June 4, 2020. We are grateful
to the ICRA conference organizers who have made this workshop possible despite
the difficult times of the pandemics. We appreciate the speakers and the
audience who made this workshop possible. Interested readers can refer to [1]
for materials from the presentations and the panel discussion.

## References

  * [1] ICRA 2020 Workshop, 2020.  https://sites.google.com/view/icra-workshop-2020.
  * [2] M. Aghassi and D. Bertsimas.  Robust game theory.  Mathematical Programming, 107(1-2):231–273, 2006.
  * [3] Alias Robotics.  Red teaming ros-industrial, extended version.  https://aliasrobotics.com/files/red_teaming_rosindustrial.pdf, 2020.
  * [4] T. Alpcan and T. Başar.  A game theoretic analysis of intrusion detection in access control systems.  In Decision and Control, 2004. CDC. 43rd IEEE Conference on, volume 2, pages 1568–1573. IEEE, 2004.
  * [5] N. Alzahrani and N. Bulusu.  Towards True Decentralization: A Blockchain Consensus Protocol Based on Game Theory and Randomness.  In L. Bushnell, R. Poovendran, and T. Başar, editors, Decision and Game Theory for Security, volume 11199, pages 465–485. Springer International Publishing, Cham, 2018.
  * [6] T. Başar and G. J. Olsder.  Dynamic noncooperative game theory, volume 23.  Siam, 1999.
  * [7] A. Basak, J. Černỳ, M. Gutierrez, S. Curtis, C. Kamhoua, D. Jones, B. Bošanskỳ, and C. Kiekintveld.  An initial study of targeted personality models in the flipit game.  In International conference on decision and game theory for security, pages 623–636. Springer, 2018.
  * [8] T. Basar.  The gaussian test channel with an intelligent jammer.  IEEE Transactions on Information Theory, 29(1):152–157, 1983.
  * [9] G. Bianchin, Y.-C. Liu, and F. Pasqualetti.  Secure Navigation of Robots in Adversarial Environments.  IEEE Control Systems Letters, 4(1):1–6, Jan. 2020.
  * [10] J. Bolot and M. Lelarge.  Cyber insurance as an incentivefor internet security.  In Managing information risk and the economics of security, pages 269–290. Springer, 2009.
  * [11] N. Boumkheld, S. Panda, S. Rass, and E. Panaousis.  Honeypot Type Selection Games for Smart Grid Networks.  In T. Alpcan, Y. Vorobeychik, J. S. Baras, and G. Dán, editors, Decision and Game Theory for Security, pages 85–96, Cham, 2019. Springer International Publishing.
  * [12] J. Buchmann and J. Ding.  Post-quantum cryptography.  Lecture Notes in Computer Science 5299. Springer, 2008.
  * [13] E. Canzani.  Risk management in (cyber-) terrorism: Modeling insights and perspectives.  Countering Terrorist Activities in Cyberspace, 139:131, 2018.
  * [14] T. E. Carroll and D. Grosu.  A game theoretic investigation of deception in network security.  Security and Communication Networks, 4(10):1162–1172, 2011.
  * [15] W. Casey, J. A. Morales, E. Wright, Q. Zhu, and B. Mishra.  Compliance signaling games: toward modeling the deterrence of insider threats.  Computational and Mathematical Organization Theory, 22(3):318–349, 2016.
  * [16] T. Chauveau.  Subjective risk and disappointment.
  * [17] J. Chen, C. Touati, and Q. Zhu.  Heterogeneous multi-layer adversarial network design for the iot-enabled infrastructures.  In GLOBECOM 2017-2017 IEEE Global Communications Conference, pages 1–6. IEEE, 2017.
  * [18] J. Chen, C. Touati, and Q. Zhu.  A dynamic game approach to strategic design of secure and resilient infrastructure network.  IEEE Transactions on Information Forensics and Security, 15:462–474, 2019.
  * [19] J. Chen, C. Touati, and Q. Zhu.  Optimal secure two-layer iot network design.  IEEE Transactions on Control of Network Systems, pages 1–1, 2019\.
  * [20] J. Chen and Q. Zhu.  Interdependent Network Formation Games.  arXiv preprint arXiv:1602.07745, 2016.
  * [21] J. Chen and Q. Zhu.  Interdependent network formation games with an application to critical infrastructures.  In American Control Conference (ACC), 2016, pages 2870–2875. IEEE, 2016.
  * [22] J. Chen and Q. Zhu.  Optimal contract design under asymmetric information for cloud-enabled internet of controlled things.  In International Conference on Decision and Game Theory for Security, pages 329–348. Springer, 2016.
  * [23] J. Chen and Q. Zhu.  Optimal Contract Design Under Asymmetric Information for Cloud-Enabled Internet of Controlled Things.  In International Conference on Decision and Game Theory for Security, pages 329–348. Springer International Publishing, 2016.
  * [24] J. Chen and Q. Zhu.  Resilient and decentralized control of multi-level cooperative mobile networks to maintain connectivity under adversarial environment.  In IEEE Conference on Decision and Control (CDC), pages 5183–5188, 2016.
  * [25] J. Chen and Q. Zhu.  Interdependent strategic cyber defense and robust switching control design for wind energy systems.  In 2017 IEEE Power & Energy Society General Meeting, pages 1–5. IEEE, 2017.
  * [26] J. Chen and Q. Zhu.  Control of multi-layer mobile autonomous systems in adversarial environments: A games-in-games approach.  IEEE Transactions on Control of Network Systems, submitted, 2019\.
  * [27] J. Chen and Q. Zhu.  A Game-and Decision-Theoretic Approach to Resilient Interdependent Network Analysis and Design.  Springer, 2019.
  * [28] A. Choudhury, D. McGrew, and J. Salowey.  AES galois counter mode (GCM) cipher suites for TLS.
  * [29] C. Cichy and S. Rass.  An Overview of Data Quality Frameworks.  IEEE Access, 7:24634–24648, 2019.
  * [30] A. Clark, Q. Zhu, R. Poovendran, and T. Başar.  Deceptive routing in relay networks.  In Decision and Game Theory for Security, pages 171–185. Springer, 2012.
  * [31] A. Clark, Q. Zhu, R. Poovendran, and T. Başar.  An impact-aware defense against stuxnet.  In 2013 American Control Conference, pages 4140–4147. IEEE, 2013\.
  * [32] I. E. Commission et al.  Industrial communication networks network and system security part 1-1: Terminology, concepts and models, iec.  Technical report, TS 62443-1-1 ed1. 0, Geneva, Switzerland, 2009.
  * [33] CyVision Technologies.  Cauldron.
  * [34] B. Dieber, B. Breiling, S. Taurer, S. Kacianka, S. Rass, and P. Schartner.  Security for the robot operating system.  98:192–203.
  * [35] B. Dieber, S. Kacianka, S. Rass, and P. Schartner.  Application-level security for ros-based applications.  In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4477–4482, Oct 2016.
  * [36] B. Dieber, R. White, S. Taurer, B. Breiling, G. Caiazza, H. Christensen, and A. Cortesi.  Penetration testing ROS.  In A. Koubaa, editor, Robot Operating System (ROS), volume 831, pages 183–225. Springer International Publishing.  Series Title: Studies in Computational Intelligence.
  * [37] E. M. Directive.  42/ec of the european parliament and the council of 17 may 2006 on machinery, and amending directive 95/16/ec (recast).  Official Journal of the European Union L, 157(09):07, 2006.
  * [38] S. Farhang, M. H. Manshaei, M. N. Esfahani, and Q. Zhu.  A dynamic bayesian security game framework for strategic defense mechanism design.  In International conference on decision and game theory for security, pages 319–328. Springer, 2014.
  * [39] C. Fung, Q. Zhu, R. Boutaba, and T. Başar.  Smurfen: A system framework for rule sharing collaborative intrusion detection.  In 2011 7th International Conference on Network and Service Management, pages 1–6. IEEE, 2011.
  * [40] C. J. Fung and Q. Zhu.  Facid: A trust-based collaborative decision framework for intrusion detection networks.  Ad Hoc Networks, 53:17–31, 2016.
  * [41] C. J. Fung, Q. Zhu, R. Boutaba, and T. Başar.  Bayesian decision aggregation in collaborative intrusion detection networks.  In 2010 IEEE Network Operations and Management Symposium-NOMS 2010, pages 349–356. IEEE, 2010.
  * [42] R. S. Gibbons.  Game theory for applied economists.  Princeton University Press, 1992.
  * [43] Greenbone Networks GmbH.  OpenVAS - OpenVAS - open vulnerability assessment scanner, 2020.
  * [44] F. Gul.  A theory of disappointment aversion.  Econometrica, 59(3):667.
  * [45] T. Hansen and D. E. Eastlake.  US secure hash algorithms (SHA and HMAC-SHA).  https://tools.ietf.org/html/rfc4634.
  * [46] T. Hansen and D. E. Eastlake.  US secure hash algorithms (SHA and SHA-based HMAC and HKDF).  https://tools.ietf.org/html/rfc6234.
  * [47] Y. Hayel and Q. Zhu.  Attack-aware cyber insurance for risk sharing in computer networks.  In Decision and Game Theory for Security, pages 22–34. Springer, 2015.
  * [48] R. Heartfield, G. Loukas, S. Budimir, A. Bezemskij, J. R. Fontaine, A. Filippoupolitis, and E. Roesch.  A taxonomy of cyber-physical threats and impact in the smart home.  Computers & Security, 78:398–428, Sept. 2018.
  * [49] S. H. Houmb and V. N. L. Franqueira.  Estimating ToE risk level using CVSS.  pages 718–725. IEEE Computer Society Press.
  * [50] J. W. Huang, Q. Zhu, V. Krishnamurthy, and T. Başar.  Distributed correlated q-learning for dynamic transmission control of sensor networks.  In Acoustics Speech and Signal Processing (ICASSP), 2010 IEEE International Conference on, pages 1982–1985. IEEE, 2010.
  * [51] L. Huang, J. Chen, and Q. Zhu.  A large-scale markov game approach to dynamic protection of interdependent infrastructure networks.  In International Conference on Decision and Game Theory for Security, pages 357–376. Springer, 2017.
  * [52] L. Huang, J. Chen, and Q. Zhu.  Distributed and optimal resilient planning of large-scale interdependent critical infrastructures.  In 2018 Winter Simulation Conference (WSC), pages 1096–1107. IEEE, 2018.
  * [53] L. Huang and Q. Zhu.  Analysis and computation of adaptive defense strategies against advanced persistent threats for cyber-physical systems.  In International Conference on Decision and Game Theory for Security, pages 205–226. Springer, 2018.
  * [54] L. Huang and Q. Zhu.  Analysis and computation of adaptive defense strategies against advanced persistent threats for cyber-physical systems.  In International Conference on Decision and Game Theory for Security, 2018.
  * [55] L. Huang and Q. Zhu.  Adaptive honeypot engagement through reinforcement learning of semi-markov decision processes.  In International Conference on Decision and Game Theory for Security, pages 196–216. Springer, 2019.
  * [56] L. Huang and Q. Zhu.  Dynamic bayesian games for adversarial and defensive cyber deception.  In E. Al-Shaer, J. Wei, K. W. Hamlen, and C. Wang, editors, Autonomous Cyber Deception: Reasoning, Adaptive Planning, and Evaluation of HoneyThings, pages 75–97. Springer International Publishing, Cham, 2019.
  * [57] Y. Huang, J. Chen, L. Huang, and Q. Zhu.  Dynamic games for secure and resilient control system design.  National Science Review, 7(7):1125–1141, 2020.
  * [58] Y. Huang and Q. Zhu.  Deceptive reinforcement learning under adversarial manipulations on cost signals.  In International Conference on Decision and Game Theory for Security, pages 217–237. Springer, 2019.
  * [59] S. Jajodia, A. K. Ghosh, V. Swarup, C. Wang, and X. S. Wang.  Moving target defense: creating asymmetric uncertainty for cyber threats, volume 54.  Springer Science & Business Media, 2011.
  * [60] S. Jajodia, V. Subrahmanian, V. Swarup, and C. Wang.  Cyber deception, volume 6.  Springer, 2016.
  * [61] Joanneum ROBOTICS.  jr-robotics/ROSPenTo.  2020\.  https://github.com/jr-robotics/ROSPenTo.
  * [62] J. Jonsson, K. Moriarty, B. Kaliski, and A. Rusch.  PKCS #1: RSA cryptography specifications version 2.2.
  * [63] A. Josang and R. Ismail.  The beta reputation system.  In Proceedings of the 15th bled electronic commerce conference, volume 5, pages 2502–2511, 2002.
  * [64] A. D. Joseph, B. Nelson, B. I. Rubinstein, and J. Tygar.  Adversarial machine learning.  Cambridge University Press, 2018.
  * [65] B. Kehoe, S. Patil, P. Abbeel, and K. Goldberg.  A survey of research on cloud robotics and automation.  IEEE Transactions on automation science and engineering, 12(2):398–409, 2015.
  * [66] I. Konnov.  On Lexicographic Vector Equilibrium Problems.  Journal of Optimization Theory and Applications, 118(3):681–688, Sept. 2003.
  * [67] S. König, A. Gouglidis, B. Green, and A. Solar.  Assessing the Impact of Malware Attacks in Utility Networks.  In Rass, S. and Schauer, S. (eds.) Game Theory for Security and Risk Management: From Theory to Practice, pages 335–351. Springer International Publishing, Cham, 2018.
  * [68] Q. D. La, T. Q. S. Quek, and J. Lee.  A game theoretic model for enabling honeypots in IoT networks.  In 2016 IEEE International Conference on Communications (ICC), pages 1–6, Kuala Lumpur, Malaysia, May 2016. IEEE.
  * [69] F. J. R. Lera, C. F. Llamas, A. M. Guerrero, and V. M. Olivera.  Cybersecurity of Robotics and Autonomous Systems: Privacy and Safety.  In G. Dekoulis, editor, Robotics - Legal, Ethical and Socioeconomic Impacts. InTech, Dec. 2017.
  * [70] N. Liu, M. Du, and X. Hu.  Adversarial Machine Learning: An Interpretation Perspective.  arXiv:2004.11488 [cs, stat], Apr. 2020.  arXiv: 2004.11488.
  * [71] M. H. Manshaei, Q. Zhu, T. Alpcan, T. Bacşar, and J.-P. Hubaux.  Game theory meets network security and privacy.  ACM Computing Surveys (CSUR), 45(3):25, 2013.
  * [72] L. Masinter, T. Berners-Lee, and R. T. Fielding.  Uniform resource identifier (URI): Generic syntax, 2016.
  * [73] V. Mayoral-Vilches, I. Abad-Fernández, M. Pinzger, S. Rass, B. Dieber, A. Cunha, F. J. Rodríguez-Lera, G. Lacava, A. Marotta, F. Martinelli, et al.  alurity, a toolbox for robot cybersecurity.  arXiv preprint arXiv:2010.07759, 2020.
  * [74] V. Mayoral-Vilches, N. García-Maestro, M. Towers, and E. Gil-Uriarte.  Devsecops in robotics.  arXiv preprint arXiv:2003.10402, 2020.
  * [75] V. Mayoral-Vilches, E. Gil-Uriarte, I. Zamalloa Ugarte, G. Olalde Mendia, R. Izquierdo Pisón, L. Alzola Kirschgens, A. Bilbao Calvo, A. Hernández Cordero, L. Apa, and C. Cerrudo.  Towards an open standard for assessing the severity of robot security vulnerabilities, the Robot Vulnerability Scoring System (RVSS).  ArXiv e-prints, July 2018.
  * [76] V. Mayoral-Vilches, L. A. Kirschgens, A. B. Calvo, A. H. Cordero, R. I. Pisón, D. M. Vilches, A. M. Rosas, G. O. Mendia, L. U. S. Juan, I. Z. Ugarte, E. Gil-Uriarte, E. Tews, and A. Peter.  Introducing the robot security framework (RSF), a standardized methodology to perform security assessments in robotics, 2019.
  * [77] V. Mayoral-Vilches, M. Pinzger, S. Rass, B. Dieber, and E. Gil-Uriarte.  Can ROS be used securely in industry? red teaming ROS-Industrial.  arXiv preprint arXiv:2009.08211, 2020.
  * [78] J. McClean, C. Stull, C. Farrar, and D. Mascarenas.  A preliminary cyber-physical security assessment of the robot operating system (ros).  In Unmanned Systems Technology XV, volume 8741, page 874110. International Society for Optics and Photonics, 2013.
  * [79] K. McCloghrie, U. Blumenthal, and F. Maino.  The advanced encryption standard (AES) cipher algorithm in the SNMP user-based security model, 2004.  https://tools.ietf.org/html/rfc3826.
  * [80] F. Miao and Q. Zhu.  A moving-horizon hybrid stochastic game for secure control of cyber-physical systems.  In Decision and Control (CDC), 2014 IEEE 53rd Annual Conference on, pages 517–522. IEEE, 2014.
  * [81] R. Miura-Ko, B. Yolken, J. Mitchell, and N. Bambos.  Security decision-making among interdependent organizations.  In Computer Security Foundations Symposium, 2008. CSF’08. IEEE 21st, pages 66–80. IEEE, 2008.
  * [82] M. M. Moghaddam, M. H. Manshaei, and Q. Zhu.  To trust or not: A security signaling game between service provider and client.  In Decision and Game Theory for Security, pages 322–333. Springer, 2015.
  * [83] I. Mokube and M. Adams.  Honeypots: concepts, approaches, and challenges.  In Proceedings of the 45th annual southeast regional conference, pages 321–326, 2007.
  * [84] A. Monga and Q. Zhu.  On solving large-scale low-rank zero-sum security games of incomplete information.  In Information Forensics and Security (WIFS), 2016 IEEE International Workshop on, pages 1–6. IEEE, 2016.
  * [85] J. F. Nash et al.  Equilibrium points in n-person games.  Proceedings of the national academy of sciences, 36(1):48–49, 1950\.
  * [86] C. Neumann, T. Yu, S. Hartman, and K. Raeburn.  The Kerberos Network Authentication Service (V5).  2005\.
  * [87] M. A. Noureddine, A. Fawaz, W. H. Sanders, and T. Başar.  A game-theoretic approach to respond to attacker lateral movement.  In International Conference on Decision and Game Theory for Security, pages 294–313. Springer, 2016.
  * [88] Y. Nugraha, A. Cetinkaya, T. Hayakawa, H. Ishii, and Q. Zhu.  Dynamic resilient network games considering connectivity.  In 2020 59th IEEE Conference on Decision and Control (CDC), pages 3779–3784. IEEE, 2020.
  * [89] Y. Nugraha, T. Hayakawa, A. Cetinkaya, H. Ishii, and Q. Zhu.  Subgame perfect equilibrium analysis for jamming attacks on resilient graphs.  In 2019 American Control Conference (ACC), pages 2060–2065. IEEE, 2019.
  * [90] Open Robotics.  roswtf - ROS wiki, 2020.
  * [91] Open Source Robotics Foundation, Inc.  ROS 2 Robotic Systems Threat Model, 2021.  retrieved: March 4, 2021.
  * [92] Y. Pan, G. Peng, J. Chen, and Q. Zhu.  Masage: Model-agnostic sequential and adaptive game estimation.  In International Conference on Decision and Game Theory for Security, pages 365–384. Springer, 2020.
  * [93] J. Pawlick.  A Systems Science Perspective on Deception for Cybersecurity in the Internet of Things.  PhD thesis, 2018.  Copyright - Database copyright ProQuest LLC; ProQuest does not claim copyright in the individual underlying works; Last updated - 2018-08-09.
  * [94] J. Pawlick, J. Chen, and Q. Zhu.  istrict: An interdependent strategic trust mechanism for the cloud-enabled internet of controlled things.  IEEE Transactions on Information Forensics and Security, 14(6):1654–1669, 2018.
  * [95] J. Pawlick, E. Colbert, and Q. Zhu.  A game-theoretic taxonomy and survey of defensive deception for cybersecurity and privacy.  arXiv preprint arXiv:1712.05441, 2017.
  * [96] J. Pawlick, E. Colbert, and Q. Zhu.  Modeling and analysis of leaky deception using signaling games with evidence.  IEEE Transactions on Information Forensics and Security, 14(7):1871–1886, 2018.
  * [97] J. Pawlick, E. Colbert, and Q. Zhu.  A game-theoretic taxonomy and survey of defensive deception for cybersecurity and privacy.  ACM Computing Surveys (CSUR), 52(4):82, 2019.
  * [98] J. Pawlick, S. Farhang, and Q. Zhu.  Flip the cloud: Cyber-physical signaling games in the presence of advanced persistent threats.  In International Conference on Decision and Game Theory for Security, pages 289–308. Springer, 2015.
  * [99] J. Pawlick and Q. Zhu.  Deception by design: evidence-based signaling games for network defense.  arXiv preprint arXiv:1503.05458, 2015.
  * [100] J. Pawlick and Q. Zhu.  A Stackelberg game perspective on the conflict between machine learning and data obfuscation.  In Information Forensics and Security (WIFS), 2016 IEEE International Workshop on, pages 1–6. IEEE, 2016.
  * [101] J. Pawlick and Q. Zhu.  Phishing for Phools in the Internet of Things: Modeling One-to-Many Deception using Poisson Signaling Games.  arXiv preprint arXiv:1703.05234, 2017.
  * [102] J. Pawlick and Q. Zhu.  Strategic trust in cloud-enabled cyber-physical systems with an application to glucose control.  IEEE Transactions on Information Forensics and Security, 12(12):2906–2919, 2017.
  * [103] J. Pawlick and Q. Zhu.  Game Theory for Cyber Deception: From Theory to Applications.  Springer Nature, 2021.
  * [104] G. Peng, T. Zhang, and Q. Zhu.  A data-driven distributionally robust game using wasserstein distance.  In International Conference on Decision and Game Theory for Security, pages 405–421. Springer, 2020.
  * [105] C. Petschnigg, M. Brandstotter, H. Pichler, M. Hofbaur, and B. Dieber.  Quantum computation in robotic science and applications.  In 2019 International Conference on Robotics and Automation (ICRA), pages 803–810. IEEE.
  * [106] T. Pornin.  Deterministic usage of the digital signature algorithm (DSA) and elliptic curve digital signature algorithm (ECDSA).
  * [107] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng.  Ros: an open-source robot operating system.  In ICRA workshop on open source software, volume 3, page 5. Kobe, Japan, 2009.
  * [108] M. A. Rahman, M. H. Manshaei, and E. Al-Shaer.  A game-theoretic approach for deceiving remote operating system fingerprinting.  In 2013 IEEE Conference on Communications and Network Security (CNS), pages 73–81. IEEE, 2013.
  * [109] S. Rass.  Complexity of Network Design for Private Communication and the P-vs-NP question.  International Journal of Advanced Computer Science and Applications, 5(2):148–157, 2014.
  * [110] S. Rass.  Perfectly Secure Communication, based on Graph-Topological Addressing in Unique-Neighborhood Networks.  arXiv preprint arXiv:1810.05602, 2018.
  * [111] S. Rass.  Security Strategies and Multi-Criteria Decision Making.  In S. Rass and S. Schauer, editors, Game Theory for Security and Risk Management.From Theory to Practice, Static & dynamic game theory : foundations & applications, pages 47–74. Birkhäuser, Cham, Switzerland, 2018.
  * [112] S. Rass.  Security Strategies and Multi-Criteria Decision Making.  In S. Rass and S. Schauer, editors, Game Theory for Security and Risk Management, pages 47–74. Springer International Publishing, Cham, 2018.
  * [113] S. Rass, S. Konig, and S. Schauer.  Defending Against Advanced Persistent Threats Using Game-Theory.  PLoS ONE, 12(1):e0168675, 2017.
  * [114] S. Rass and S. Kurowski.  On Bayesian Trust and Risk Forecasting for Compound Systems.  In Proceedings of the 7th International Conference on IT Security Incident Management & IT Forensics (IMF), pages 69–82. IEEE Computer Society, 2013.
  * [115] S. Rass and S. König.  Password Security as a Game of Entropies.  Entropy, 20(5):312, 2018.
  * [116] S. Rass, S. König, and E. Panaousis.  Cut-the-rope: A game of stealthy intrusion.  In T. Alpcan, Y. Vorobeychik, J. S. Baras, and G. Dán, editors, Decision and Game Theory for Security, volume 11836, pages 404–416. Springer International Publishing.
  * [117] S. Rass, S. König, and S. Schauer.  On the cost of game playing: How to control the expenses in mixed strategies.  In Decision and Game Theory for Security.8th International Conference, GameSec 2017, pages 494–505. Springer.
  * [118] S. Rass and B. Rainer.  Numerical Computation of Multi-Goal Security Strategies.  In R. Poovendran and W. Saad, editors, Decision and Game Theory for Security, LNCS 8840, pages 118–133. Springer, 2014.
  * [119] S. Rass, B. Rainer, M. Vavti, J. Göllner, A. Peer, and S. Schauer.  Secure Communication over Software-Defined Networks.  Mobile Networks and Applications, 20(1):105–110, 2015.
  * [120] S. Rass, B. Rainer, M. Vavti, and S. Schauer.  A Network Modeling and Analysis Tool for Perfectly Secure Communication.  In Proceedings of the 27th IEEE International Conference on Advanced Information Networking and Applications, pages 267–275. IEEE Computer Society Press, 2013.
  * [121] S. Rass and P. Schartner.  Multipath Authentication without shared Secrets and with Applications in Quantum Networks.  In Proceedings of the International Conference on Security and Management (SAM), volume 1, pages 111–115. CSREA Press, 2010.
  * [122] S. Rass and P. Schartner.  Authentic Quantum Nonces.  In C. Kollmitzer, S. Schauer, S. Rass, and B. Rainer, editors, Quantum Random Number Generation: Theory and Practice, Quantum Science and Technology, pages 35–44. Springer International Publishing, Cham, 2020.
  * [123] S. Rass and S. Schauer.  Refining Stochastic Models of Critical Infrastructures by Observation.  In Proceedings of the 56th ESReDA Seminar, European Atomic Energy Community, number No JRC118427 in JRC Publications, pages 212–223. Publications Office of the European Union, 2019.
  * [124] S. Rass, S. Schauer, S. König, and Q. Zhu.  Cyber-Security in Critical Infrastructures: A Game-Theoretic Approach.  SpringerNature.
  * [125] S. Rass, A. Schorn, and F. Skopik.  Trust and Distrust: On Sense and Nonsense in Big Data.  In E. Kosta, J. Pierson, D. Slamanig, S. Fischer-Hübner, and S. Krenn, editors, Privacy and Identity Management. Fairness, Accountability, and Transparency in the Age of Big Data, volume 547, pages 81–94. Springer International Publishing, Cham, 2019.
  * [126] S. Rass, A. Wiegele, and S. König.  Security Games over Lexicographic Orders.  In Q. Zhu, J. S. Baras, R. Poovendran, and J. Chen, editors, Decision and Game Theory for Security, volume 12513, pages 422–441. Springer International Publishing, Cham, 2020.  Series Title: Lecture Notes in Computer Science.
  * [127] S. Rass and Q. Zhu.  GADAPT: a sequential game-theoretic framework for designing defense-in-depth strategies against advanced persistent threats.  In International Conference on Decision and Game Theory for Security, pages 314–326. Springer International Publishing, 2016.
  * [128] C. Rieger, I. Ray, Q. Zhu, and M. Haney.  Industrial Control Systems Security and Resiliency: Practice and Theory.  Advances in Information Security. Springer, 2019.
  * [129] C. Rieger, Q. Zhu, and T. Başar.  Agent-based cyber control strategy design for resilient control systems: Concepts, architecture and methodologies.  In Resilient Control Systems (ISRCS), 2012 5th International Symposium on, pages 40–47. IEEE, 2012.
  * [130] Y. Ruan, S. Kalyanasundaram, and X. Zou.  Survey of return-oriented programming defense mechanisms: Survey of return-oriented programming defense mechanisms.  Security and Communication Networks, 9(10):1247–1265, July 2016\.
  * [131] P. W. Shor.  Polynomial-time algorithms for prime factorization and discrete logarithms on a quantum computer.  26(5):1484–1509.
  * [132] C. B. Simmons, C. Ellis, S. Shiva, D. Dasgupta, and Q. Wu.  Avoidit: A cyber attack taxonomy.  CTIT technical reports series, 2009.
  * [133] A. Singhal and X. Ou.  Security risk analysis of enterprise networks using probabilistic attack graphs, 2011.
  * [134] F. Skopik, M. Landauer, M. Wurzenberger, G. Vormayr, J. Milosevic, J. Fabini, W. Prüggler, O. Kruschitz, B. Widmann, K. Truckenthanner, S. Rass, M. Simmer, and C. Zauner.  synERGY: Cross-correlation of operational and contextual data to timely detect and mitigate attacks to cyber-physical systems.  Journal of Information Security and Applications, 54:102544, Oct. 2020.
  * [135] K. Stouffer, J. Falco, and K. Scarfone.  Guide to industrial control systems (ics) security.  NIST special publication, 800(82):16–16, 2011.
  * [136] L. Stroetmann and H. Pohl.  Robot operating system (ros): Safe & insecure.  2014\.
  * [137] System Security Research Group.  Implementation of the cut-the-rope game” of stealthy intrusion | SYSSEC.
  * [138] S. Taurer, B. Breiling, S. Svrta, and B. Dieber.  Case study: Remote attack to disable MiR100 safety.  In Proceedings of the first Cybersecurity for Robotics 2019 Conference (CSfR2019), 2019.
  * [139] Tenable.  Nessus vulnerability assessment, 2020.
  * [140] M. Van Dijk, A. Juels, A. Oprea, and R. L. Rivest.  Flipit: The game of “stealthy takeover”.  Journal of Cryptology, 26(4):655–713, 2013.
  * [141] R. Varma, C. Melville, C. Pinello, and T. Sahai.  Post quantum secure command and control of mobile agents : Inserting quantum-resistant encryption schemes in the secure robot operating system.
  * [142] V. M. Vilches.  IT, OT, IoT and Robotics, a security comparison, Jun 2020.  https://cybersecurityrobotics.net/it-ot-iot-and-robotics-security-comparison/.
  * [143] V. M. Vilches, E. Gil-Uriarte, I. Z. Ugarte, G. O. Mendia, R. I. Pisón, L. A. Kirschgens, A. B. Calvo, A. H. Cordero, L. Apa, and C. Cerrudo.  Towards an open standard for assessing the severity of robot security vulnerabilities, the robot vulnerability scoring system (RVSS).
  * [144] Y. Vorobeychik and M. Kantarcioglu.  Adversarial Machine Learning.  Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):1–169, Aug. 2018.
  * [145] J. Wachter, T. Grafenauer, and S. Rass.  Visual Risk Specification and Aggregation.  In IARIA, editor, SECURWARE 2017: The Eleventh International Conference on Emerging Security Information, Systems and Technologies, pages 93–98. 2017.
  * [146] J. Wachter, S. Rass, S. König, and S. Schauer.  Disappointment-aversion in security games.  In International Conference on Decision and Game Theory for Security, pages 314–325. Springer.
  * [147] R. White.  ruffsl/roschaos, 2018.  original-date: 2017-09-29T17:24:54Z.
  * [148] Z. Xu and Q. Zhu.  A cyber-physical game framework for secure and resilient multi-agent autonomous systems.  In 2015 54th IEEE Conference on Decision and Control (CDC), pages 5156–5161. IEEE, 2015.
  * [149] Z. Xu and Q. Zhu.  Secure and resilient control design for cloud enabled networked control systems.  In Proceedings of the first ACM workshop on cyber-physical systems-security and/or privacy, pages 31–42, 2015.
  * [150] Z. Xu and Q. Zhu.  Cross-layer secure cyber-physical control system design for networked 3d printers.  In 2016 American Control Conference (ACC), pages 1191–1196. IEEE, 2016.
  * [151] Z. Xu and Q. Zhu.  A Game-Theoretic Approach to Secure Control of Communication-Based Train Control Systems Under Jamming Attacks.  In Proceedings of the 1st International Workshop on Safe Control of Connected and Autonomous Vehicles, pages 27–34. ACM, 2017\.
  * [152] Z. Xu and Q. Zhu.  A game-theoretic approach to secure control of communication-based train control systems under jamming attacks.  In Proceedings of the 1st International Workshop on Safe Control of Connected and Autonomous Vehicles, pages 27–34. ACM, 2017.
  * [153] Z. Xu and Q. Zhu.  Secure and practical output feedback control for cloud-enabled cyber-physical systems.  In Communications and Network Security (CNS), 2017 IEEE Conference on, pages 416–420. IEEE, 2017.
  * [154] Z. Xu and Q. Zhu.  Cross-layer secure and resilient control of delay-sensitive networked robot operating systems.  In 2018 IEEE Conference on Control Technology and Applications (CCTA), pages 1712–1717. IEEE, 2018.
  * [155] X. Yuan, P. He, Q. Zhu, and X. Li.  Adversarial Examples: Attacks and Defenses for Deep Learning.  arXiv:1712.07107 [cs, stat], July 2018.  arXiv: 1712.07107.
  * [156] R. Zhang and Q. Zhu.  Secure and resilient distributed machine learning under adversarial environments.  In 2015 18th International Conference on Information Fusion (Fusion), pages 644–651. IEEE, 2015.
  * [157] R. Zhang and Q. Zhu.  Attack-Aware Cyber Insurance of Interdependent Computer Networks.  2016\.
  * [158] R. Zhang and Q. Zhu.  A game-theoretic analysis of label flipping attacks on distributed support vector machines.  In Information Sciences and Systems (CISS), 2017 51st Annual Conference on, pages 1–6. IEEE, 2017.
  * [159] R. Zhang and Q. Zhu.  A game-theoretic defense against data poisoning attacks in distributed support vector machines.  In Decision and Control (CDC), 2017 IEEE 56th Annual Conference on, pages 4582–4587. IEEE, 2017.
  * [160] R. Zhang and Q. Zhu.  A game-theoretic approach to design secure and resilient distributed support vector machines.  IEEE transactions on neural networks and learning systems, 29(11):5512–5527, 2018.
  * [161] R. Zhang and Q. Zhu.  A game-theoretic approach to design secure and resilient distributed support vector machines.  IEEE transactions on neural networks and learning systems, 29(11):5512–5527, 2018.
  * [162] R. Zhang and Q. Zhu.  Flipin: A game-theoretic cyber insurance framework for incentive-compatible cyber risk management of internet of things.  IEEE Transactions on Information Forensics and Security, 2019.
  * [163] R. Zhang, Q. Zhu, and Y. Hayel.  A bi-level game approach to attack-aware cyber insurance of computer networks.  IEEE Journal on Selected Areas in Communications, 35(3):779–794, 2017.
  * [164] T. Zhang, L. Huang, J. Pawlick, and Q. Zhu.  Game-theoretic analysis of cyber deception: Evidence-based strategies and dynamic risk mitigation.  arXiv preprint arXiv:1902.03925, 2019.
  * [165] T. Zhang and Q. Zhu.  Dynamic differential privacy for admm-based distributed classification learning.  IEEE Transactions on Information Forensics and Security, 12(1):172–187, 2016.
  * [166] Q. Zhu.  Control challenges.  In C. Rieger, editor, Resilient Control Architectures and Power Systems, page Chapter 14 (in press). Wiley-IEEE Press, 2021.
  * [167] Q. Zhu and T. Başar.  Dynamic policy-based ids configuration.  In Decision and Control, 2009 held jointly with the 2009 28th Chinese Control Conference. CDC/CCC 2009. Proceedings of the 48th IEEE Conference on, pages 8600–8605. IEEE, 2009.
  * [168] Q. Zhu and T. Başar.  Indices of power in optimal ids default configuration: theory and examples.  In Decision and Game Theory for Security, pages 7–21. Springer, 2011.
  * [169] Q. Zhu and T. Başar.  Robust and resilient control design for cyber-physical systems with an application to power systems.  In 2011 50th IEEE Conference on Decision and Control and European Control Conference, pages 4066–4071. IEEE, 2011.
  * [170] Q. Zhu and T. Başar.  A dynamic game-theoretic approach to resilient control system design for cascading failures.  In Proceedings of the 1st international conference on High Confidence Networked Systems, pages 41–46, 2012.
  * [171] Q. Zhu and T. Başar.  Game-theoretic approach to feedback-driven multi-stage moving target defense.  In International Conference on Decision and Game Theory for Security, pages 246–263. Springer, 2013.
  * [172] Q. Zhu and T. Basar.  Game-theoretic methods for robustness, security, and resilience of cyberphysical control systems: games-in-games principle for optimal cross-layer resilient control systems.  IEEE Control Systems Magazine, 35(1):46–65, 2015.
  * [173] Q. Zhu, L. Bushnell, and T. Başar.  Resilient distributed control of multi-agent cyber-physical systems.  In Control of Cyber-Physical Systems, pages 301–316. Springer, 2013\.
  * [174] Q. Zhu, L. Bushnell, and T. Başar.  Resilient distributed control of multi-agent cyber-physical systems.  In Control of Cyber-Physical Systems, pages 301–316. Springer, 2013\.
  * [175] Q. Zhu, Z. Han, and T. Başar.  No-regret learning in collaborative spectrum sensing with malicious nodes.  In Communications (ICC), 2010 IEEE International Conference on, pages 1–6. IEEE, 2010.
  * [176] Q. Zhu, H. Li, Z. Han, and T. Başar.  A stochastic game model for jamming in multi-channel cognitive radio systems.  In 2010 IEEE International Conference on Communications, pages 1–6. IEEE, 2010.
  * [177] Q. Zhu and S. Rass.  On multi-phase and multi-stage game-theoretic modeling of advanced persistent threats.  IEEE Access, 6:13958–13971, 2018.
  * [178] Q. Zhu, S. Rass, and P. Schartner.  Community-Based Security for the Internet of Things.  In Smart Cities Cybersecurity and Privacy, pages 11–19. Elsevier, 2019.
  * [179] Q. Zhu, C. Rieger, and T. Başar.  A hierarchical security architecture for cyber-physical systems.  In 2011 4th international symposium on resilient control systems, pages 15–20. IEEE, 2011.
  * [180] Q. Zhu, W. Saad, Z. Han, H. V. Poor, and T. Başar.  Eavesdropping and jamming in next-generation wireless networks: A game-theoretic approach.  In Military Communications Conference (MILCOM), 2011, pages 119–124. IEEE, 2011.
  * [181] Q. Zhu, J. B. Song, and T. Başar.  Dynamic secure routing game in distributed cognitive radio networks.  In Global Telecommunications Conference (GLOBECOM 2011), 2011 IEEE, pages 1–6. IEEE, 2011.
  * [182] Q. Zhu, H. Tembine, and T. Başar.  Network security configurations: A nonzero-sum stochastic game approach.  In American Control Conference (ACC), 2010, pages 1059–1064. IEEE, 2010.
  * [183] Q. Zhu, H. Tembine, and T. Başar.  Distributed strategic learning with application to network security.  In American Control Conference (ACC), 2011, pages 4057–4062. IEEE, 2011.
  * [184] Q. Zhu, H. Tembine, and T. Başar.  Hybrid learning in stochastic games and its applications in network security.  Reinforcement Learning and Approximate Dynamic Programming for Feedback Control, pages 305–329, 2013.
  * [185] Q. Zhu, H. Tembine, and T. Başar.  Heterogeneous learning in zero-sum stochastic games with incomplete information.  In 49th IEEE conference on decision and control (CDC), pages 219–224. IEEE, 2010.
  * [186] Q. Zhu, D. Wei, and K. Ji.  Hierarchical architectures of resilient control systems: Concepts, metrics and design principles.  In J. C. P. Cheng, H. Zhang, editor, Cyber Security for Industrial Control Systems: from the viewpoint of close-loop. CRC Press, 2015\.
  * [187] Q. Zhu and Z. Xu.  Cross-Layer Design for Secure and Resilient Cyber-Physical Systems: A Decision and Game Theoretic Approach.  Springer Nature, 2020.
  * [188] J. Zhuang, V. M. Bier, and O. Alagoz.  Modeling secrecy and deception in a multiple-period attacker–defender signaling game.  European Journal of Operational Research, 203(2):409–418, 2010\.