printacmref=false 2022 2022 383 Duke University Durham U.S.A Duke University
Durham U.S.A Duke University Durham U.S.A Duke University Durham U.S.A

# Multi-Agent Adversarial Attacks for Multi-Channel Communications

Juncheng Dong<EMAIL_ADDRESS>,  Suya Wu<EMAIL_ADDRESS>,
Mohammadreza Soltani<EMAIL_ADDRESS>and  Vahid Tarokh
<EMAIL_ADDRESS>
###### Abstract.

Recently Reinforcement Learning (RL) has been applied as an anti-adversarial
remedy in wireless communication networks. However studying the RL-based
approaches from the adversary’s perspective has received little attention.
Additionally, RL-based approaches in an anti-adversary or adversarial paradigm
mostly consider single-channel communication (either channel selection or
single channel power control), while multi-channel communication is more
common in practice. In this paper, we propose a multi-agent adversary system
(MAAS) for modeling and analyzing adversaries in a wireless communication
scenario by careful design of the reward function under realistic
communication scenarios. In particular, by modeling the adversaries as
learning agents, we show that the proposed MAAS is able to successfully choose
the transmitted channel(s) and their respective allocated power(s) without any
prior knowledge of the sender strategy. Compared to the single-agent adversary
(SAA), multi-agents in MAAS can achieve significant reduction in signal-to-
noise ratio (SINR) under the same power constraints and partial observability,
while providing improved stability and a more efficient learning process.
Moreover, through empirical studies we show that the results in simulation are
close to the ones in communication in reality, a conclusion that is pivotal to
the validity of performance of agents evaluated in simulations.

###### Key words and phrases:

RL, Communication, Multiagent System, Adversary Attacks

##  1\. Introduction

In recent years, we have witnessed the success of reinforcement learning (RL)
in many sequential decision-making problems in machine learning, optimal
control, and wireless communication Mao et al. (2018); Ye et al. (2020); Li
(2018). This is mainly due to the use of deep neural networks (DNNs) and their
expressive power in the traditional RL algorithms such as Deep Q-Network
(DQN), Double DQN (DDQN), and Deep Deterministic Policy Gradient (DDPG) Mnih
et al. (2013); van Hasselt et al. (2015); Lillicrap et al. (2019). However,
many practical sequential decision-making problems involve modeling the
interaction between multiple agents and the environment. For instance,
defending wireless communication networks against adversary attacks typically
requires the collaboration between different nodes in the network. Wireless
networks are highly vulnerable to adversarial attacks due to the open and
sharing nature of the wireless medium. One of the challenges in these networks
is to detect/predict adversarial activities and designing proper defense
mechanisms, which typically require cooperation/competition among multiple
friendly or adversary agents Aref et al. (2017); Zhou et al. (2021). Early
attempts to prevent the adversarial attacks in wireless communication included
the frequency hopping techniques. By choosing transmission frequency from a
set of available frequency chunks (either randomly or by using some learning
procedure) the defender attempts to avoid the compromised frequencies Prasadh
and Jindal (2020); Grover et al. (2014).

Figure 1. Multi-channel wireless communication system with $M$ adversaries and
one pair of sender/receiver. At time $t$, both sender/receiver and adversaries
select a channel and a power level respectively as
$\left(P_{S}^{(t)},C_{S}^{(t)}\right)$ and
$\left(P_{j}^{(t)},C_{j}^{(t)}\right)$. Adversaries observe state $s^{(t)}$
and receive rewards $r_{j}^{(t)}$ from communication environment.

Recently RL has been successfully applied for designing algorithms for
defending against adversary attacks in a hostile wireless communication
environment Ye et al. (2020); Liu et al. (2018); Xu et al. (2020); Chen et al.
(2018). However, using the RL-based methods from the adversary’s perspective
has received little attention. This is especially important since a better
understanding of adversaries’ behavior can lead to better design of defense
mechanisms as well as active defense. Furthermore, the current endeavor for
utilizing the RL-based methods either in anti-adversary or adversary paradigm
mostly consider communication within a single channel (either channel
selection or single channel power control), while multi-channel communication
is more common in real scenarios.

In this paper, we propose a multi-agent adversary system (MAAS) based on RL
for modeling and analyzing adversaries (e.g., jammers) in a wireless
communication scenario by a careful design of the reward function for
realistic multi-channel communication scenarios. Through extensive
simulations, we show that the proposed MAAS learns to choose the transmit
channel and the most efficient power allocation for attack without any prior
knowledge of the sender strategy. In particular, our results demonstrate that
using MAAS and implicit collaboration between the adversaries can provide
better performance and success rates compared to the single-agent adversary
(SAA) case. In addition, MAAS is fault-tolerant and robust to the failure of
some agents since other agents may take over the failed agent’s duty and
continue their operations. Our contribution can be summarized as follows:

  * •

We use an RL-based approach for modeling and analyzing the adversaries in a
multi-channel and realistic wireless communication scenario.

  * •

We show the superiority of multi-agent adversaries versus the single-agent by
achieving a significant reduction in the signal-to-interference-plus-noise
ratio (SINR) under the same power constraint and partial observability.
Specifically, we show that MAAS is able to learn more efficiently and stably
by reducing the exponential complexity of SAA’s action space to a linear
complexity through distributing actions to multi-agents.

The rest of this paper is organized as follows. In Section 2, we define our
system model of communication, provide some background about RL, and briefly
discuss existing work on RL-based anti-adversaries. In Section 3, we propose
and discuss the MAAS in detail together with studying the multi-channel system
model and the advantages of using multi-agents instead of single-agent. In
Section 4, we present simulation results on various types of sender strategy
and demonstrate the superior performance of the proposed MAAS against baseline
approaches, including the single-agent case, random adversaries, and greedy
adversaries. We conclude in Section 5.

##  2\. Backgrounds

In this section, we first review some required background for the single-agent
RL as well as multi-agent RL, and then provide some related works of using RL
techniques and multi-agent systems in the wireless communication.

###  2.1. Single-agent RL

A single-agent RL is an agent that learns how to make sequential decisions by
interacting with the environment through receiving a feedback signal called
_reward_. To choose the best action in each time step, the agent tries to
choose an action, maximizing its long-term (discounted) rewards. In this
scenario, the environment is modeled by a Markov Decision Process (MDP) Sutton
and Barto (2018) defined by a 5-tuple of
$(\mathcal{A},\mathcal{S},p,R,\gamma)$, where $\mathcal{A}$ and $\mathcal{S}$
denote the action space and state-space respectively. $p$ is the transition
probability, $R$ denotes the reward function and $\gamma$ is the discount
factor that trades off the immediate and future rewards. In a single-agent RL,
at time step $t$, the agent chooses its action $a_{t}$ according to a _policy_
function $\pi$. As a result of choosing the action $a_{t}$, the environment
changes its current state $s_{t}$ to the next state $s_{t+1}$, and returns a
real-valued reward $r_{t}$ (feedback) to the agent. Hence, the interaction of
an agent with an environment modeled by the above MDP is defined as follows:

  1. (1)

The state of the environment at time $t$: $s_{t}\in\mathcal{S}$

  2. (2)

Agent’s action at time $t$: $a_{t}\in\mathcal{A}$

  3. (3)

State transition probability:
$p:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$

  4. (4)

Reward function: $R:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$

  5. (5)

Agent’s policy: $\pi:\mathcal{S}\rightarrow\mathcal{A}$

Here, policy $\pi$ is a mapping from the current state $s_{t}$ to the current
action $a_{t}$, i.e., $\pi=\pi(a_{t}|s_{t})$. Moreover, $p$ captures the
dynamics of the environment and is written as a conditional probability of
next state given the current state and action, that is
$p(s^{(t+1)}|s^{(t)},a^{(t)})$. Also, the reward function $R$ gets the current
state and action as input and returns the agent’s reward, i.e.
$r^{(t)}:=R(s^{(t)},a^{(t)})$. As mentioned above, the objective function of a
single-agent RL is to maximize the discounted cumulative rewards by selecting
the best policy $\pi$. That is,

$\displaystyle\max_{\pi}\mathbb{E}\Big{[}\sum_{t=0}^{\infty}\gamma^{t}R^{(t)};\pi\Big{]}.$

If the state transition probability and the reward function are known (i.e.,
planning problem), then the optimal policy $\pi^{*}$ can be solved through
iterative algorithms such as the value iteration algorithm Sutton and Barto
(2018). However, in many applications, the dynamics of the environment, $p$,
and the reward function $R$ are not known. Model-free algorithms Mnih et al.
(2013) are RL algorithms that solve MDP without requiring to know $p$ or $R$
in advance. One of the most widely used model-free RL algorithm is the
Q-learning algorithm that finds the optimal policy $\pi^{*}$ by maximizing
action-value (also called Q-function)
$Q^{\pi}(s_{t},a_{t})=\mathbb{E}\Big{[}\sum_{t=0}^{\infty}\gamma^{t}R^{(t)}\rvert
s_{t},a_{t};\pi\Big{]}$. The action-value function returns the expected
discounted cumulative reward when an agent takes an action $a_{t}$ in state
$s_{t}$, and following the policy $\pi$ afterward. As a result, the optimal
policy $\pi^{*}$ is the one that maximizes $Q(s_{t},a_{t})$ for all $s_{t}$
and $a_{t}$. That is,

$\displaystyle\pi^{*}=\operatorname*{argmax}_{\pi}Q^{\pi}(s_{t},a_{t}).$

The classical Q-learning algorithm maintains a Q-table to store the
corresponding Q-values for all state-action pairs. At each time step $t$,
values in Q-table is updated based on the Bellman optimally equation,
$Q^{\pi^{*}}(s_{t},a_{t})=r^{(t)}+\gamma\mathbb{E}[Q^{\pi^{*}}(s_{t+1},a_{t+1})]$.
Values in Q-table converges to the optimal value $Q^{*}(s,t)$ after enough
iterations. However, for many problems, the state space grows exponentially
with the dimension of state and the state space can often be continuous;
hence, making the Q-learning impractical. To get around these issues, deep
Q-learning applies a neural network to estimate the Q-values and to replace
the Q-table.

###  2.2. Multi-agent RL (MARL)

Existing Multi-agent RL algorithms can be categorized by how the agents are
trained and how they make a decision after training (execution). Both training
and execution schemes can be divided into centralized and distributed
approaches Gronauer and Diepold (2021). Distributed adversaries are usually
more robust than the centralized ones when there is a counterattack from the
anti-adversary side, or when some adversaries fail to operate properly. On the
other hand, distributed training/execution can be expensive and unstable in a
hostile environment due to the communication overhead between agents. As a
result, distributed training and execution are more desired for multi-agent
adversary systems. We refer the readers to OroojlooyJadid and Hajinezhad
(2021); Gronauer and Diepold (2021) for a detailed discussion about MARL.

###  2.3. Applications of RL in Communication

The progress in the field of RL spurred a new interest in intelligent wireless
communication and networking Mao et al. (2018); Luong et al. (2019). The rapid
proliferation of security concerns has made considerable attention for
investigating adversary attacks and mitigating the jamming interference,
resulting in extensive studies in intelligent anti-adversary (anti-jamming)
techniques Pirayesh and Zeng (2021). Previous methods have used the game
theory to find the optimal anti-jamming strategies Wu et al. (2012); Hanawal
et al. (2014); Gwon et al. (2013); Wang et al. (2011); Tang et al. (2015).
Most of these theoretical studies formulated the interaction of a sender with
a jammer/adversary as a zero-sum Markov game Wu et al. (2012); Hanawal et al.
(2014); Gwon et al. (2013) or a Stackelberg game Tang et al. (2015). The two
players in the game dynamically decide actions to optimize the opposite
objectives. The actions typically include which channel to switch and what
transmission rate to use. Then the optimal strategy of the sender can be
derived by applying theoretical results from the MDP and the prior knowledge
about jammers. In these methods, the transition probability is necessary for
such a derivation; therefore, it cannot be used in more challenging and
realistic scenarios.

Since RL methods are an appealing framework for selecting an optimal action
that yields the highest reward for a given state, they have been used
extensively in modeling many anti-adversary wireless communication systems,
including cognitive radio networks Han et al. (2017); Shi et al. (2018); Liu
et al. (2018); Xu et al. (2020); Ye et al. (2020), in-air unmanned aerial
vehicular (UAV) Lv et al. (2017); Xiao et al. (2018); Lu et al. (2020),
Internet of Things (IoT) services Chen et al. (2018). For example, Han et al.
(2017) has proposed a deep Q-learning-based anti-jamming strategy for
cognitive radio networks. In this system, the secondary users (SUs), learned
by convolutional neural networks (CNN) choose a frequency channel that avoids
interference with the primary users (PUs). Along the same line, an anti-
jamming deep reinforcement learning algorithm (ADRLA) has been proposed by Liu
et al. (2018) based on recursive convolutional neural networks (RCNN). For a
more realistic scenario with a large state and action space, some authors
incorporate the power allocation into users’ decision-making by utilizing
advanced DDQN Bi et al. (2019); Xu et al. (2020) and fast convergence
investigation Ye et al. (2020). To adapt to the multi-agent environment, a
sequence of multi-agent reinforcement learning (MARL)-based schemes have been
designed to find anti-jamming policies in multi-channel systems Aref et al.
(2017); Yao and Jia (2019); Elleuch et al. (2021); Zhou et al. (2021). In
addition to the RL-based methods, Erpek et al. (2019) has used a deep
classifier-based jamming attack and anti-jamming defense to distinguish the
success of attacks from the genuine transmissions by deep classifiers.

Regarding the jamming strategies, there are a few works about anti-jamming
strategies in the presence of intelligent attacks Arjoune and Faruque (2020).
Most of the existing anti-jamming schemes are developed to confront
conventional types of jammers, for example, random jammers, sweeping jammers,
and reactive jammers Erpek et al. (2019).

In this paper, we propose a MARL-based system for multi-channel communications
from the adversary/jammer perspective, where the jammers are smart and are
able to adapt to a multi-channel environment efficiently, especially when
current communication networks tend to be decentralized and ad-hoc Luong et
al. (2019).

##  3\. Multi-agent Adversary System (MAAS)

In this section, we first describe our proposed system model for simulating
multi-channel communication. Based on the system model, reward functions are
carefully designed for the correct guidance of MAAS. We also discuss in detail
the learning process of MAAS. Figure 2 illustrates the workflow of our
proposed MAAS and algorithm 1 presents the pseudo-codes for training and
prediction of MAAS.

###  3.1. System Model of Communication

We now define our communication model. In our scenario, we assume there is a
pair of sender/receiver and there are $N$ available channels for sending the
signal (see figure 1). We assume that the communication between the sender and
the receiver and also the adversaries happens only at discrete time steps
$t=0,1,2,...$. At each time step $t$, the sender is allowed to choose multiple
channels to send its signal to the receiver. The amount of power allocated to
each channel is assumed to be fixed during the time that channel is used for
communication. We also assume that there are $M$ adversaries (indexed by the
index $j=0,1,...,M-1$), each selects one channel out of $N$ available channels
and a power level from a set $P=\\{P_{0},P_{1},\ldots,P_{K}\\}$, where $0\leq
P_{0}<P_{1}<\dots<P_{K}$). This means that the adversary $j$’s action at time
$t$ is a 2-dimensional vector $a_{j}^{(t)}=[C_{j}^{(t)},P_{j}^{(t)}]^{T}$,
where $C_{j}^{(t)}$ is the channel selected by adversary $j$, and
$P_{j}^{(t)}\in P$ is its selected power level at time $t$.

In our scenario, the goal is that adversaries can select their actions to
maximize decrease of the quality of communication (QoC) between the sender and
the receiver. The (QoC) at each time step $t$ is defined by the signal-to-
interference-plus-noise ratio (SINR):

$\displaystyle\text{SINR}^{(t)}=\frac{P_{S}^{(t)}*h_{S}}{\eta+\sum_{j=0}^{M-1}P_{j}^{(t)}*h_{j}*I(C_{j}^{(t)}=C_{S}^{(t)})},$
(1)

where $\eta$ denotes the communication noise, and $h_{S}$ and $h_{j}$ are
power gains for the sender and adversary $j$, respectively.

In order to evaluate the performance of an adversary in our communication
model, we define _Success of Attack_ (SA) as a binary-valued function as
follows:

$SA=\mathbbm{1}{(SINR^{(t)}<\tau SNR^{(t)})},$ (2)

where $\tau$ denotes a pre-defined threshold, and SNR (signal-to-noise-ratio)
is the maximal achievable SINR for the receiver, calculated as
$SNR=\frac{P_{s}*h_{s}}{\eta}$ (i.e., the SINR without any interference).
Also, $\mathbbm{1}(condition)$ denotes the indicator function defined as 1 if
the condition is true, and 0 otherwise. We also define the _Success Rate of
Attack_ (SRA) as the ratio of the number of SA over a time interval $T>0$:

$\displaystyle\text{SRA}=\frac{\sum_{t=0}^{T-1}\mathbbm{1}{(SINR^{(t)}<\tau
SNR^{(t)})}}{T}.$ (3)

The choice of $\tau$ is problem-specific. However, as we will see in the
experimental results, the change of $\tau$ will not have a dramatic effect in
changing the SRA value of the proposed MASS.

###  3.2. Design of Reward Function

In RL, designing a reward function is a crucial step as it is the only
feedback that an agent can receive from the environment; as a result, it has a
huge effect on learning the best policy Buşoniu et al. (2010) and improving
the overall performance. A standard choice for the reward function in many
anti-adversary RL-based approaches is given by the SINR. However, directly
using the SINR as the reward function may be misleading since it may increase
the reward function artificially; thus, achieving a higher SRA for the MAAS
compared to the single-agent systems. To overcome this issue, our reward
function includes two parts: the portion of channels blocked by adversaries,
and the power cost incurred by adversaries for attacking the communication
between the sender and the receiver.

  * •

Channels blocked by adversaries. This can be computed by the decrease in the
Shannon channel capacity calculated as
$B*\left(\log_{2}(1+\text{SNR}^{(t)})-\log_{2}(1+\text{SINR}^{(t)})\right)$.

  * •

Power cost. The reward function should include a term, indicating the cost of
power to penalize adversaries if they use the allocated power inefficiently.
We consider a constant power cost, $Cost_{power}$ for all the adversaries and
assume that the cost is known to adversaries throughout the communication.

Hence, the total reward function for the whole multi-agent system is given by:

$\displaystyle
R^{(t)}=B*\left(\log_{2}(1+\text{SNR}^{(t)})-\log_{2}(1+\text{SINR}^{(t)})\right)$
$\displaystyle\hskip 105.2751pt-C_{\text{power}}*\sum_{j=0}^{M-1}P_{j}^{(t)},$
(4)

where $P_{j}$ is the power used by adversary $j$.

###  3.3. RL for Multi-Agent Adversary System

Unlike the episodic RL problems in which there is a terminal state indicating
the end of one episode (e.g., game playing Lowe et al. (2020)), modeling
adversaries in wireless communication requires continuous learning. Due to the
continuous state and non-Markovian dynamic of environment caused by
simultaneous learning of multi-agents as well as the requirement for
continuous learning, we choose the Double Deep Q-learning with prioritized
experience replay as RL agent for each adversary in MAAS for faster adaption
to the environment van Hasselt et al. (2015); Schaul et al. (2016). Next, we
describe in detail the learning process.

State. Similar to existing works Ye et al. (2020), we use the SINR at time
$t-1$ as the state for the environment at time $t$. That is,
$s^{(t)}=SINR^{(t-1)}$ for $t=1,2,3,\ldots.$ We note that in wireless
communication, the true value of SINR is not known for any agent other than
the receiver. Here, we first follow the current practice for using the
expected SINR for the rewards. Later, we show through simulations that the
success rate of adversaries without using the true value of SINR stays at the
same level as the former case. To our knowledge, conducting this set of
simulations is novel and shows the stability as well as reliability of the
performance of the proposed RL method.

Reward. In addition to the multi-agent system reward defined in equation
(3.2), a learning agent (e.g., adversary $j$ for $j=0,1,\dots,M-1$) in MARL
can have its own reward for distributed training at time $t$ defined as
follows:

$\displaystyle
R_{j}^{(t)}=B*\left(\log_{2}(1+\text{SNR}^{(t)})-\log_{2}(1+\text{SINR}^{(t)})\right)$
$\displaystyle\hskip 105.2751pt-Cost_{power}*P_{j}^{(t)}.$ (5)

Learning. For training of the MAAS, we follow the distributed learning
paradigm in which there is no central entity to coordinate the information
exchange between adversaries. in addition, all adversaries interact with the
environment simultaneously and can observe the SINR values sequentially. They
also select their actions independently from each other. Each adversary $j$ is
equipped by an experience memory $MEM_{j}$ for storing the adversary’s
experience for faster learning, and a pair of actor network $Q_{j}^{actor}$
and target network $Q_{j}^{target}$) which are initialized randomly at the
beginning of communication. At the very beginning ($t=0$), adversaries make
random actions. At any other time $t>0$, each adversary selects its action
$a_{j}^{(t)}$ using its actor network with the current state $s^{(t)}$ (the
SINR value from the last time step, $t-1$) as input. The sequence of SINR
values is also used for computing individual rewards $r_{j}^{(t)}$ which is
used to train actor networks and target networks $Q_{j}^{(actor)}$ and
$Q_{j}^{(target)}$ (See figure 2 and algorithm 1 for details of the learning
algorithm and training process).

Figure 2. Flow of distributed learning distributed execution MAAS at time $t$.
Each adversary $j$ chooses its action independently according to the observed
state and all adversaries attack simultaneously. The corresponding SINR values
are used for the individual reward of each adversary and as the state for time
$t+1$. Training is done separately on each adversary.

Input: $\gamma,$
$h_{s},h_{j},\epsilon_{j},decay_{j},\alpha_{j},j\in{0,2,\ldots,M-1}$
$MEM_{j}(\text{experience storage of adversary j})$

Output: $Q_{j}^{actor}$

1

/* Initialization */

2 for  _$j=0,2,...,M-1$_ do

3 Initialize $Q_{j}^{actor},Q_{j}^{target}$

4 Choose $a_{j}^{(0)}$ as random action

5$SINR^{(0)}$ is computed based on
$[a_{0}^{(0)},a_{2}^{(0)},...,a_{M-1}^{(0)}]$

6 $s^{(1)}=SINR^{(0)}$

7

8for  _$t=1,2,3,\ldots$_ do

9

/* Adversaries choose action */

10 for  _$j=0,2,...,M-1$_ do

11 if  _$uniform(0,1) >\epsilon_{j}$_ then

12 $a_{j}^{(t)}=\operatorname*{argmax}_{a_{j}}Q_{j}^{actor}(s^{(t)},a_{j})$

13 else

14 Choose $a_{j}^{(t)}$ as a random action

15

16 $SINR^{(t)}$ is computed based on
$[a_{0}^{(t)},a_{2}^{(t)},...,a_{M-1}^{(t)}]$

17 $s^{(t+1)}=SINR^{(t)}$

/* Receive individual reward and storage the experience */

18 for  _$j=0,2,...,M-1$_ do

19 Compute $r_{j}^{(t)}$

20 Storage the experience tuple $(s^{(t)},a_{j}^{(t)},r_{j}^{(t)},s^{(t+1)})$
into $MEM_{j}$

/* Training */

21 for  _$j=0,2,...,M-1$_ do

22 Sample experience batch $E$ from $MEM_{j}$

/* for each experience tuple in batch $E$ */

23 for  _ $e=(s,a,r,s^{\prime})\in E$_ do

24 $a^{*}=\operatorname*{argmax}_{a_{j}}Q_{j}^{actor}(s,a_{j})$

25 $Target=r+\gamma Q_{j}^{target}(s^{\prime},a^{*})$

26 $Loss=\Big{(}Q_{j}^{actor}(s,a_{j})-Target\Big{)}^{2}$

27 Train $Q_{j}^{actor}$ through gradient descent on $Loss$

/* Weight Shift for Double DQN */

28 $Q_{j}^{target}=\alpha_{j}Q_{j}^{actor}+(1-\alpha_{j})Q_{j}^{target}$

/* Decreasing the probability of exploration for more exploitation */

29 $\epsilon_{j}=\epsilon_{j}*decay_{j}$

Algorithm 1 Multi-Agent Adversary System

###  3.4. Advantages of MAAS

Compared to the single-agent system, MAAS is more robust to the failure of any
adversary due to the counterattacks of anti-adversaries, communication
problems, hardware issues, etc. This is because of the nature of distributed
training and executing of agents: if one or more adversaries fail to operate,
others can continue their training or execute independently without the
failure of the whole system. Furthermore, multi-channel communication in SAA
may become a major challenge. The action space of a single adversary grows
exponentially with the number of channels it can choose (curse of
dimensionality Bellman (1966)). Suppose there are in total $N$ channels and
$K$ power levels to choose from, and an adversary is allowed to select $X$
channels at each time step. Therefore, the size of action space is given by
$(NK)^{X}$. If $N=15$, $K=5$ and $X=4$, then the size of action space is
$31,640,625$ which is already beyond the capability of most powerful
computational resources, not to mention that most adversary devices are small
in memory and computation power. Exponentially growing action space implies
exponentially more experience to search; thus, exponentially longer time to
learn. By distributing the possibility of $X$ channels among multiple agents
such that each agent can select less than $X$ channel at each time, MAAS
reduces the exponential growth of the action space. For example, if each agent
in MAAS is allowed to choose only one channel at each time step, the
complexity of the action space will be linear, $O(NK)$, leading to a more
efficient and stable learning process. We show this empirically in the
experimental section.

##  4\. Simulations

In this section, we conduct extensive simulation studies to show the
performance of the proposed MAAS in both single-channel and multiple-channel
wireless communication. In each communication scenario, we consider four types
of transmitters: (1) Constant sender, (2) Sweep sender, (3) Autoregressive-
type (AR-type) sender, and (4) Pulse sender. For each type of sender, MAAS is
compared with four benchmark adversaries: (a) Random adversary, (b)Greedy
adversary, (c) Single RL adversary that attacks only one channel (SSRL), and
(d) Single RL adversary that attacks multiple channels (MSRL). We present more
details in the following sections.

###  4.1. Sender Strategies

Let $N$ denote the total number of available channels, we consider a
complementary set of sender strategies in simulations. The first choice is
constant sender, which represents the most straightforward strategies. Then,
we consider strategies that change with a deterministic pattern that is the
sweep sender. Next, the pulse sender simulates the communication environment
with abrupt changes. Last, but not least, we use the AR-type sender to
represent complicated strategies with extra randomness. Each type of sender
selects the channel for signal transmissions by the mechanisms described
below.

  1. (1)

Constant sender The simplest and easiest strategy of transmission is to stay
at one frequency all the time.

$C_{S}^{(t)}=C_{S}^{(0)},$ (6)

where $C_{S}^{(0)}$ is chosen uniformly from all available channels at initial
time $t=0$.

  2. (2)

Sweep sender The choices of channels sweep among all available channels
through time.

$C_{S}^{(t)}=t\%N.$ (7)

  3. (3)

Pulse sender We expect this type of sender to evaluate the jammer capability
of tracking the abrupt changes.

$C_{S}^{(t)}=\left\\{\begin{array}[]{ll}5,&\text{if}\;t\%N\leq 2\\\
1,&o.w.\end{array}\right.$ (8)

  4. (4)

AR-type sender On the contrary to the senders above that with deterministic
schemes of frequency hopping. The AR-type sender investigates the impact of a
non-stationary communication environment.

$C_{S}^{(t)}=\left\\{\begin{array}[]{ll}C_{S}^{(t-1)}+i\%N,&\text{if}\;C_{S}^{(t-1)}\%2=0\\\
C_{S}^{(t-1)}-i\%N,&\text{if}\;C_{S}^{(t-1)}\%2=1\\\
X_{t}\in\\{1,N\\},&\text{if}\;C_{S}^{(t-1)}>N\\\
Y_{t}\in\\{1,N\\},&\text{if}\;C_{S}^{(t-1)}<1\\\ \end{array}\right.$ (9)

where $X_{t}$ and $Y_{t}$ are binary random variables with the probabilities
$p(X_{t}=1)=0.1$ and $p(Y_{t}=1)=0.9$ respectively. At time $t=0$,
$C_{S}^{(0)}$ is initialized by an uniformly random choice from all available
channels.

For each type of senders, the sender’s power $P_{S}$ is assumed to be fixed
throughout the communication.

###  4.2. Benchmark Adversaries

We compare the attacking performance of the proposed MAAS with four benchmark
adversaries. Let $M$ denote the number of adversaries in MAAS. The schemes of
these adversary attacks are described below.

  1. (a)

Random The random adversary interferes by choosing the channel index and
allocating power level randomly at each time.

  2. (b)

Greedy The greedy adversary records the history rewards of each action in each
iteration. Then, it performs jamming by greedily choosing the action
(including channel selection and power allocation) with the highest averaged
history rewards. Note that the greedy adversary is a variant of a multi-armed
bandit agent.

  3. (c)

SSRL The single-channel and single-agent RL (SSRL) adversary is a learning-
based intelligent jammer. SSRL performs jamming on one channel at each time.

  4. (d)

MSRL The multi-channel and single-agent RL (MSRL) is an intelligent jammer as
well. However, in contrast to SSRL, MSRL can produce interference on multiple
channels at each time.

We denote the proposed MAAS as the multi-channel and multi-agent RL (MMRL) in
all following tables and figures.

###  4.3. Simulation Results

In the simulation studies follows, we set up a synthetic communication
environment. There are $N=5$ available channels and four levels of powers
$P_{S}\in\\{0,1,3,5\\}$. In the single-channel communication system, we fix
the power of sender $P_{S}=5$. While in the multi-channel communication
system, the senders can send signals through two channels with the
corresponding power $P_{S}=[1,5]$. The channel gains $h_{j}$, $h_{s}$ and cost
of power $C_{\text{power}}$ are set to be $1$ without loss of generality.

As for the RL-type adversaries, we set the initial exploration rate
$\epsilon=1$ and the decay rate as $0.996$. During training, the Adam Kingma
and Ba (2015) optimizer is applied with a learning rate of $0.0001$ and no
weight decay. All RL-type agents are trained in 2000 epochs. To avoid the
uncertainty caused by randomness in the training procedure and random
initialization, we report the averaged experimental results over multiple runs
for each communication scenario.

Table 1 and Figure 3 illustrate the overall performance of adversaries in
different scenarios. As shown in Table 1, MAAS has a consistently increased
success rate of attacks significantly comparing to all benchmark adversaries
regardless of threshold $\tau$.

Figure 3. The averaged rewards of various adversaries in different scenarios. The figure shows that MAAS consistently gains over the other adversaries. In particular, it outperforms MSRL in all cases even though we enforce its total power to be the same as the MSRL. Table 1. The averaged success rates (%) of various adversaries on different scenarios. $\tau$ is the threshold for identifying the success of attacks. | Random | Greedy | SS RL | MS RL | MM RL
---|---|---|---|---|---
Sweep Single-Channel
$\tau=0.2$ | 46.2 | 39.9 | 31.4 | 72.5 | 79.1
$\tau=0.5$ | 65.5 | 39.9 | 31.4 | 72.5 | 79.1
Sweep Multi-Channel
$\tau=0.2$ | 49.1 | 40.5 | 33.5 | 49.4 | 65.9
$\tau=0.5$ | 62.3 | 71.2 | 67.7 | 77.2 | 84.5
Autoregressive Single-Channel
$\tau=0.2$ | 36.7 | 57.0 | 35.1 | 66.6 | 75.3
$\tau=0.5$ | 47.7 | 57.0 | 35.1 | 66.6 | 75.3
Autoregressive Multi-Channel
$\tau=0.2$ | 30.7 | 34.5 | 25.1 | 49.3 | 62.3
$\tau=0.5$ | 46.5 | 67.2 | 71.5 | 72.5 | 81.5
Pulse Single-Channel
$\tau=0.2$ | 48.0 | 40.0 | 53.5 | 82.4 | 93.2
$\tau=0.5$ | 65.1 | 40.0 | 53.5 | 82.4 | 93.2
Pulse Multi-Channel
$\tau=0.2$ | 64.4 | 60.1 | 66.4 | 87.4 | 94.4
$\tau=0.5$ | 81.2 | 60.1 | 66.4 | 87.4 | 94.4

####  4.3.1. Constant Sender

While the constant sender is the simplest sender for adversaries, performance
on a constant sender can be an excellent indication of the learning process of
different RL adversaries. It is apparent from Figure 4 that all three RL-type
adversaries converge to the theoretical optimum. In particular, MAAS converges
significantly fast than the others. This efficient learning behavior can lead
to overall better performance of jamming, as shown in Table 2. On the other
hand, Figure 4 provides the evidence of a fair comparison in our experiments,
that is, the multi-channel RL-type adversaries having a similar optimal reward
function as it of the single-channel RL-type adversary. It is beneficial from
the reward function we designed, and therefore the optimal reward value for RL
agents only has a minuscule increase with more channel capability.

Figure 4. The averaged rewards of RL-type adversaries on constant sender scenarios. On top, the figure shows the performance of attacks in single-channel communications. At the bottom, it shows the performance in multi-channel communications. Table 2. The averaged success rates (%) of RL-type adversaries on constant sender scenarios. $\tau$ is the threshold for identifying success of attacks. | SS RL | Single RL | Multi RL
---|---|---|---
Single-Channel
$\tau=0.2$ | 87.5 | 90.7 | 94.1
$\tau=0.5$ | 87.5 | 90.7 | 94.1
Multi-Channel
$\tau=0.2$ | 44.9 | 90.1 | 92.7
$\tau=0.5$ | 44.9 | 90.1 | 92.7

####  4.3.2. Sweep Sender.

The most exciting aspect of Figure 5 is the clear demonstration of the
training advantage of MAAS, especially in complicated communication scenarios.
For example, in the single-channel scenario, the top half of Figure 5 shows
that the learning of MAAS is stable, and the convergence of MAAS is fast. At
the same time, from the bottom half of Figure 5, it is observed that MAAS has
immense rewards by escaping from the local minimum to which a single agent
adversary may converge.

Figure 5. The averaged rewards of various adversaries agents on sweep sender
scenarios. On top, the figure shows the performance of attacks in single-
channel communications. At the bottom, it shows the performance in multi-
channel communications.

####  4.3.3. AR-type Sender.

The AR-type sender is the most complex and challenging sender of all synthetic
cases. Nevertheless, MAAS still has a better performance than other types of
adversaries facing such a challenge, as shown in Figure 6. For example, MAAS
maintains itself in a high-reward region in single-channel communications.
Moreover, MAAS tends to converge to a higher reward region in multi-channel
communications.

Figure 6. The averaged rewards of various adversaries agents on autoregressive
sender scenarios. On top, the figure shows the performance of attacks in
single-channel communications. At the bottom, it is the performance in multi-
channel communications.

####  4.3.4. Pulse Sender.

For the pulse sender scenarios, where the communication environment changes
abruptly, MAAS has demonstrated its characteristic of efficient learning and
stability again. In addition, it has shown faster convergence to optimal and
stays stably in high-reward regions in single-channel and multi-channel
communications (shown in Figure 7). Also, Table 1 indicates that the success
of MAAS is solid. That is, the success rate of MAAS does not change for
different values of threshold $\tau$. In fact, random jammers is the only one
suffering from this issue.

Figure 7. The averaged rewards of various adversaries agents on pulse sender
scenarios. On top, the figure shows the performance of attacks in single-
channel communications. At the bottom, it is the performance in multi-channel
communications.

##  5\. One Step Forward: Learning under Real Communication Scenario

The previous experiments demonstrate the paradigm of the performance of RL-
type adversaries in different synthetic communication systems. However, in
real-world scenarios, the channel gains $h_{s}$ and $h_{j}$ are random
variables, while the previous simulation studies assume the constant values of
$h_{s}$ and $h_{j}$. In this section, we provide some experiments that show
that even with random channel gains, the performance of the proposed MAAS
remains close to the case of the previous experiments using the constant
channel gains.

We first experiment by setting $h_{s}$ and $h_{j}$ as Rayleigh random
variables Sklar (1997). Therefore, the corresponding SINR ($SINR_{real}$) is
calculated as follows:

$\displaystyle SINR_{real}^{(t)}$
$\displaystyle=\frac{P_{S}^{(t)}*h_{S}*\beta_{S}}{n+\sum_{j=1}^{N}P_{j}*h_{j}*\beta_{j}*I(C_{j}^{(t)}=C_{S}^{(t)})},$
(10)

where $\beta_{S}$ and $\beta_{j}\sim Rayleigh(0,\sqrt{0.5})$. We denote the
SINR defined in equation (1) as $SINR_{cons}^{(t)}$ for the use of constant
values of the channel gains. We trained the MAAS with both $SINR_{real}$ and
$SINR_{cons}$. we kept all the experimental settings, like the choices of
hyper-parameters, the same to ensure that our comparison was fair. Also, we
ran our experiments for 8 Monte Carlo trials and reported the average over all
the trials. The t-test is then applied to test the difference between the
averaged SRAs with degrees of freedom as $7$. As shown in Table 3, the results
indicate that the MAAS framework is robust to even using random channel gains
as MAAS achieves almost the same SRA as the experiment with $SINR_{cons}$.

Table 3. The averaged SRAs (%) of MAAS trained with/without raw SINR. (Standard deviations are in the bracket.) P-value closer to 0 indicates more likelihood of significant difference while p-value closer to 1 indicates no significant difference between two averaged SRAs. | Constant | Sweep | AR-Type | Pulse
---|---|---|---|---
Single-Channel $\tau=0.5$
$SINR_{real}$ | 93.4(7.4) | 78.2(4.0) | 77.3(6.7) | 93.6(2.4)
$SINR_{cons}$ | 93.1(6.9) | 78.4(4.3) | 76.4 (5.9) | 94.1(2.6)
p-value | 0.934 | 0.925 | 0.78 | 0.695
Multi-Channel $\tau=0.5$
$SINR_{real}$ | 92.5(6.6) | 84.5(4.3) | 81.2(7.2) | 94.5(2.9)
$SINR_{cons}$ | 92.1(6.2) | 85.3(4.8) | 81.7 (7.5) | 93.9(2.2)
p-value | 0.902 | 0.731 | 0.894 | 0.649

##  6\. Conclusion

We have proposed a MARL-based multi-agent adversary system (MAAS) along with a
system model for multi-channel communication. We have also defined an
appropriate reward function for multi-agent adversaries. MAAS has shown an
outstanding performance compared to various baselines, even with power
constraints and without partial observability. By distributing actions to
multiple agents, MAAS can learn more efficiently and stably than single-agent
adversary (SAA) systems. This is due to the linear complexity of the action
space in MAAS. The proposed MAAS has its value in active defense as well as
understanding the behaviors of multi-agent RL adversaries in designing the
defense mechanisms.

##  7\. Acknowledgement

This work was supported in part by Air Force Research Lab Award #FA
8750-20-2-0504.

## References

  * (1)
  * Aref et al. (2017) Mohamed A. Aref, Sudharman K. Jayaweera, and Stephen Machuzak. 2017.  Multi-Agent Reinforcement Learning based Cognitive Anti-jamming. In _IEEE Wireless Communications and Networking Conference, WCNC_. Institute of Electrical and Electronics Engineers Inc.  https://doi.org/10.1109/WCNC.2017.7925694
  * Arjoune and Faruque (2020) Youness Arjoune and Saleh Faruque. 2020.  Smart Jamming Attacks in 5G New Radio: A Review.  _2020 10th Annual Computing and Communication Workshop and Conference, CCWC 2020_ (2020), 1010–1015.  https://doi.org/10.1109/CCWC47524.2020.9031175
  * Bellman (1966) Richard Bellman. 1966\.  Dynamic Programming.  _Science_ 153, 3731 (1966), 34–37.
  * Bi et al. (2019) Yue Bi, Yue Wu, and Cunqing Hua. 2019.  Deep Reinforcement Learning Based Multi-User Anti-Jamming Strategy.  _IEEE International Conference on Communications_ 2019-May (may 2019).  https://doi.org/10.1109/ICC.2019.8761848
  * Buşoniu et al. (2010) Lucian Buşoniu, Robert Babuška, and Bart De Schutter. 2010.  _Multi-agent Reinforcement Learning: An Overview_.  Springer Berlin Heidelberg, Berlin, Heidelberg, 183–221.  https://doi.org/10.1007/978-3-642-14435-6_7
  * Chen et al. (2018) Ye Chen, Yanda Li, Dongjin Xu, and Liang Xiao. 2018\.  DQN-based Power Control for IoT Transmission against Jamming.  _IEEE Vehicular Technology Conference_ 2018-June (jul 2018), 1–5.  https://doi.org/10.1109/VTCSpring.2018.8417695
  * Elleuch et al. (2021) Ibrahim Elleuch, Ali Pourranjbar, and Georges Kaddoum. 2021\.  A Novel Distributed Multi-Agent Reinforcement Learning Algorithm Against Jamming Attacks.  _IEEE Communications Letters_ (2021).  https://doi.org/10.1109/LCOMM.2021.3097290
  * Erpek et al. (2019) Tugba Erpek, Yalin E. Sagduyu, and Yi Shi. 2019.  Deep Learning for Launching and Mitigating Wireless Jamming Attacks.  _IEEE Transactions on Cognitive Communications and Networking_ 5, 1 (jul 2019), 2–14.  https://doi.org/10.1109/TCCN.2018.2884910 arXiv:1807.02567
  * Gronauer and Diepold (2021) Sven Gronauer and Klaus Diepold. 2021.  Multi-Agent Deep Reinforcement Learning: A Survey.  _Artificial Intelligence Review_ (2021), 1–49.
  * Grover et al. (2014) Kanika Grover, Alvin Lim, and Qing Yang. 2014.  Jamming and anti-jamming techniques in wireless networks: A survey.  _International Journal of Ad Hoc and Ubiquitous Computing_ 17 (01 2014), 197.  https://doi.org/10.1504/IJAHUC.2014.066419
  * Gwon et al. (2013) Youngjune Gwon, Siamak Dastangoo, Carl Fossa, and H. T. Kung. 2013.  Competing Mobile Network Game: Embracing Anti-jamming and Jamming Strategies with Reinforcement Learning.  _2013 IEEE Conference on Communications and Network Security, CNS 2013_ (2013), 28–36.  https://doi.org/10.1109/CNS.2013.6682689
  * Han et al. (2017) Guoan Han, Liang Xiao, and H. Vincent Poor. 2017.  Two-dimensional Anti-jamming Communication based on Deep Reinforcement Learning. In _ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings_. 2087–2091.  https://doi.org/10.1109/ICASSP.2017.7952524
  * Hanawal et al. (2014) Manjesh K Hanawal, Mohammad J Abdel-Rahman, and Marwan Krunz. 2014.  Game Theoretic Anti-jamming Dynamic Frequency Hopping and Rate Adaptation in Wireless Systems. In _2014 12th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks, WiOpt 2014_. 247–254.  https://doi.org/10.1109/WIOPT.2014.6850306
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Lei Ba. 2015.  Adam: A method for Stochastic Optimization.  _3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings_ (2015).  arXiv:1412.6980 http://arxiv.org/abs/1412.6980
  * Li (2018) Yuxi Li. 2018.  Deep Reinforcement Learning.  arXiv:1810.06339 [cs.LG]
  * Lillicrap et al. (2019) Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2019.  Continuous Control with Deep Reinforcement Learning.  arXiv:1509.02971 [cs.LG]
  * Liu et al. (2018) Xin Liu, Yuhua Xu, Luliang Jia, Qihui Wu, and Alagan Anpalagan. 2018.  Anti-jamming Communications Using Spectrum Waterfall: A Deep Reinforcement Learning Approach.  _IEEE Communications Letters_ 22, 5 (may 2018), 998–1001.  https://doi.org/10.1109/LCOMM.2018.2815018 arXiv:1710.04830
  * Lowe et al. (2020) Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. 2020\.  Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments.  arXiv:1706.02275 [cs.LG]
  * Lu et al. (2020) Xiaozhen Lu, Liang Xiao, Canhuang Dai, and Huaiyu Dai. 2020\.  UAV-Aided Cellular Communications with Deep Reinforcement Learning against Jamming.  _IEEE Wireless Communications_ 27, 4 (2020), 48–53.  https://doi.org/10.1109/MWC.001.1900207 arXiv:1805.06628
  * Luong et al. (2019) Nguyen Cong Luong, Dinh Thai Hoang, Shimin Gong, Dusit Niyato, Ping Wang, Ying Chang Liang, and Dong In Kim. 2019\.  Applications of Deep Reinforcement Learning in Communications and Networking: A Survey.  _IEEE Communications Surveys and Tutorials_ 21, 4 (2019), 3133–3174.  https://doi.org/10.1109/COMST.2019.2916583 arXiv:1810.07862
  * Lv et al. (2017) Shichao Lv, Liang Xiao, Qing Hu, Xiaoshan Wang, Changzhen Hu, and Limin Sun. 2017\.  Anti-Jamming Power Control Game in Unmanned Aerial Vehicle Networks. In _2017 IEEE Global Communications Conference, GLOBECOM 2017 - Proceedings_ , Vol. 2018-Janua. Institute of Electrical and Electronics Engineers Inc., 1–6.  https://doi.org/10.1109/GLOCOM.2017.8253988
  * Mao et al. (2018) Qian Mao, Fei Hu, and Qi Hao. 2018.  Deep learning for intelligent wireless networks: A comprehensive survey.  _IEEE Communications Surveys and Tutorials_ 20, 4 (2018), 2595–2621.  https://doi.org/10.1109/COMST.2018.2846401
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013.  Playing Atari with Deep Reinforcement Learning.  arXiv:1312.5602 [cs.LG]
  * OroojlooyJadid and Hajinezhad (2021) Afshin OroojlooyJadid and Davood Hajinezhad. 2021.  A Review of Cooperative Multi-Agent Deep Reinforcement Learning.  arXiv:1908.03963 [cs.LG]
  * Pirayesh and Zeng (2021) Hossein Pirayesh and Huacheng Zeng. 2021.  Jamming Attacks and Anti-Jamming Strategies in Wireless Networks: A Comprehensive Survey.  (2021).  arXiv:2101.00292 http://arxiv.org/abs/2101.00292
  * Prasadh and Jindal (2020) S. Prasadh and Sumit Jindal. 2020.  _Security and Efficiency Analysis of Anti-jamming Techniques_.  251–259.  https://doi.org/10.1007/978-3-030-39875-0_27
  * Schaul et al. (2016) Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. 2016\.  Prioritized Experience Replay.  arXiv:1511.05952 [cs.LG]
  * Shi et al. (2018) Yi Shi, Yalin E. Sagduyu, Tugba Erpek, Kemal Davaslioglu, Zhuo Lu, and Jason H. Li. 2018\.  Adversarial Deep Learning for Cognitive Radio Security: Jamming Attack and Defense Strategies.  _2018 IEEE International Conference on Communications Workshops, ICC Workshops 2018 - Proceedings_ (jul 2018), 1–6.  https://doi.org/10.1109/ICCW.2018.8403655
  * Sklar (1997) B. Sklar. 1997.  Rayleigh Fading Channels in Mobile Digital Communication Systems .I. Characterization.  _IEEE Communications Magazine_ 35, 7 (1997), 90–100.  https://doi.org/10.1109/35.601747
  * Sutton and Barto (2018) Richard S. Sutton and Andrew G. Barto. 2018.  _Reinforcement Learning: An Introduction_.  A Bradford Book, Cambridge, MA, USA.
  * Tang et al. (2015) Xiao Tang, Pinyi Ren, Yichen Wang, Qinghe Du, and Li Sun. 2015.  Securing Wireless Transmission against Reactive Jamming: A Stackelberg Game Framework. In _2015 IEEE Global Communications Conference (GLOBECOM)_. 1–6.  https://doi.org/10.1109/GLOCOM.2015.7417368
  * van Hasselt et al. (2015) Hado van Hasselt, Arthur Guez, and David Silver. 2015\.  Deep Reinforcement Learning with Double Q-learning.  arXiv:1509.06461 [cs.LG]
  * Wang et al. (2011) Beibei Wang, Yongle Wu, K. J.Ray Liu, and T. Charles Clancy. 2011\.  An Anti-jamming Stochastic Game for Cognitive Radio Networks.  _IEEE Journal on Selected Areas in Communications_ 29, 4 (apr 2011), 877–889.  https://doi.org/10.1109/JSAC.2011.110418
  * Wu et al. (2012) Yongle Wu, Beibei Wang, K. J.Ray Liu, and T. Charles Clancy. 2012\.  Anti-jamming Games in Multi-channel Cognitive Radio Networks.  _IEEE Journal on Selected Areas in Communications_ 30, 1 (2012), 4–15.  https://doi.org/10.1109/JSAC.2012.120102
  * Xiao et al. (2018) Liang Xiao, Caixia Xie, Minghui Min, and Weihua Zhuang. 2018\.  User-Centric View of Unmanned Aerial Vehicle Transmission Against Smart Attacks.  _IEEE Transactions on Vehicular Technology_ 67, 4 (apr 2018), 3420–3430.  https://doi.org/10.1109/TVT.2017.2785414
  * Xu et al. (2020) Jianliang Xu, Huaxun Lou, Weifeng Zhang, and Gaoli Sang. 2020\.  An Intelligent Anti-jamming Scheme for Cognitive Radio based on Deep Reinforcement Learning.  _IEEE Access_ 8 (2020), 202563–202572.  https://doi.org/10.1109/ACCESS.2020.3036027
  * Yao and Jia (2019) Fuqiang Yao and Luliang Jia. 2019.  A Collaborative Multi-Agent Reinforcement Learning Anti-Jamming Algorithm in Wireless Networks.  _IEEE Wireless Communications Letters_ 8, 4 (2019), 1024–1027.  https://doi.org/10.1109/LWC.2019.2904486 arXiv:1809.04374
  * Ye et al. (2020) Pei Gen Ye, Yuan Gen Wang, Jin Li, Liang Xiao, and Guopu Zhu. 2020.  Fast Reinforcement Learning for Anti-jamming Communications. In _2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings_.  https://doi.org/10.1109/GLOBECOM42002.2020.9322486 arXiv:2002.05364v1
  * Zhou et al. (2021) Quan Zhou, Yonggui Li, and Yingtao Niu. 2021.  Intelligent Anti-Jamming Communication for Wireless Sensor Networks: A Multi-Agent Reinforcement Learning Approach.  _IEEE Open Journal of the Communications Society_ 2 (feb 2021), 775–784.  https://doi.org/10.1109/ojcoms.2021.3056113