# Defeating Proactive Jammers Using Deep Reinforcement Learning for Resource-
Constrained IoT Networks

Abubakar S. Ali1 Shimaa Naser1 and Sami Muhaidat*1  2]

1Department of Electrical Engineering and Computer Science, Khalifa
University, Abu Dhabi 127788, UAE
2Department of Systems and Computer Engineering, Carleton University, Ottawa,
ON K1S 5B6, Canada

###### Abstract

Traditional anti-jamming techniques like spread spectrum, adaptive power/rate
control, and cognitive radio, have demonstrated effectiveness in mitigating
jamming attacks. However, their robustness against the growing complexity of
internet-of-thing (IoT) networks and diverse jamming attacks is still limited.
To address these challenges, machine learning (ML)-based techniques have
emerged as promising solutions. By offering adaptive and intelligent anti-
jamming capabilities, ML-based approaches can effectively adapt to dynamic
attack scenarios and overcome the limitations of traditional methods. In this
paper, we propose a deep reinforcement learning (DRL)-based approach that
utilizes state input from realistic wireless network interface cards. We train
five different variants of deep Q-network (DQN) agents to mitigate the effects
of jamming with the aim of identifying the most sample-efficient, lightweight,
robust, and least complex agent that is tailored for power-constrained
devices. The simulation results demonstrate the effectiveness of the proposed
DRL-based anti-jamming approach against proactive jammers, regardless of their
jamming strategy which eliminates the need for a pattern recognition or
jamming strategy detection step. Our findings present a promising solution for
securing IoT networks against jamming attacks and highlights substantial
opportunities for continued investigation and advancement within this field.

###### Index Terms:

Jamming, anti-jamming, cognitive radio, deep reinforcement learning

##  I Introduction

Cognitive radio networks (CRNs) have emerged as a revolutionary paradigm in
wireless communication, offering intelligent means to optimizate the available
spectrum resources through dynamic channel identification [1]. Neverthless,
the open nature of wireless communication channels exposes CRNs to potential
security breaches, particularly jamming attacks which can degrade network
performance and significantly reduce the throughput [2]. Traditional jamming
countermeasures, such as frequency hopping or direct sequence spread spectrum
(DSSS), have inherent limitations, especially when confronted with advanced
jammers that are capable of detecting and disrupting these techniques [3].
Although, game-theoretical strategies have been explored to address this
issue, such techniques assume impractical preconditions like a priori
knowledge of the perturbation pattern and can falter when faced with rapidly
changing jamming strategies [4, 5, 6].

Deep reinforcement learning (DRL), a blend of reinforcement learning and deep
learning, has been spotlighted due to its adaptability to dynamic environments
and ability to learn from raw data, without the need for pre-existing
knowledge. In the context of anti-jamming systems, DRL has been employed in
various ways in multiple works. For instance, the authors of [7] proposed a
deep anti-jamming reinforcement learning algorithm (DARLA) that used raw
spectrum data as the environmental state, addressing the anti-jamming problem
in a dynamic environment. Similarly, the work in [8] proposed a sequential
deep reinforcement learning algorithm (SDRLA) to improve anti-jamming
performance. Other research has introduced wideband autonomous cognitive
radios [9], transformer encoder-like Q-networks [10], and unmanned aerial
vehicle (UAV) jammers modeled as partially observable Markov decision
processes [11]. Some studies have also used the signal-to-interference-plus-
noise ratio (SINR) to enhance anti-jamming techniques [12, 13]. However, the
aforementioned studies relied on supplementary equipment or data such as raw
spectrum data or SINR which can be energy-inefficient and difficult to
acquire, rendering them unsuitable for resource-constrained internet-of-things
(IoT) networks.
In our prior study [14], we introduced a novel approach that uses a single
vector of clear channel assessment (CCA) information as the state input. This
simplifies the environmental state representation, hence, reducing the
computational complexity of the neural network. Our previous work also was a
departure from the approach presented in [8] as it involved a generic DRL
agent capable of effectively operating within dynamic jamming pattern
environments without requiring a preliminary pattern recognition process.
However, despite these capabilites, the CCA-based method faces some
challenges, particularly related to the information extraction from WLAN
network interface cards (NICs) and its efficacy against random channel hopping
jamming. In this paper, we strive to overcome these challenges by proposing an
improved anti-jamming scheme. In specific, we exploit a novel radio frequency
(RF)-jamming detection testbed [15], utilize the spectrum sensing capabilities
of WLAN NICs, and apply ML algorithms to detect and avoid jamming attacks.
Additionally, we conduct a comprehensive investigation of different agent
alternatives to optimize the anti-jamming performance in dynamic pattern
jamming scenarios.

Figure 1: System topology is composed of the transmitter, receiver, and
jammer. The transmitter tries to communicate with the receiver in the presence
of a jamming attack.

##  II System Model and Formulation

In this section, we describe the system, jammer, and signal models under
jamming attack as illustrated in Fig. 1. We consider the UNII-1 band of the
5GHz radio spectrum and assume that the radio environment consists of one user
(a transmitter-receiver pair) against one jammer. A novel aspect of our model
is the presence of an agent at the transmission end, which formulates real-
time anti-jamming strategies. These strategies are then shared with the
receiver through a reliable control link. We also assume that the transmitter
possesses broad-band spectrum sensing capabilities [14]. For ease of analysis,
we segment the continuous time into discrete time slots, assuming that both
the user and the jammer operate within the same time slot. In each time slot
$t$, the user selects a frequency $f_{T,t}$ from the range
$\left[f_{L},f_{U}\right]$ for data transmission to the receiver, using power
$P_{T,t}$. Concurrently, the jammer attempts to interrupt this transmission by
selecting a frequency $f_{J,t}$ and power $P_{J,t}$ according to a predefined
jamming pattern.

###  II-A Jammer Model

To investigate proactive jamming attack mitigation, we adopt a range of
jamming strategies to effectively counter such threats. Specifically, we
employ four distinct approaches: constant, sweeping, random, and dynamic
jamming techniques. In this model, we assume that the jammer jams a single
frequency $f_{J,t}$ with a varying distance $d_{JT}$ between the jammer and
transmitter and varying jamming powers $P_{J,t}$. Given the proactive nature
of the jammer, it is assumed to be unaware of the current state of the
channel. In the case of the constant jamming strategy, at the beginning of a
transmission session, the jammer picks one of the available channels of the RF
spectrum to jam consistently. Operating in a manner similar to the constant
jammer, the combined jammer possesses the ability to disrupt multiple
channels. However, it should be noted that not all channels can be jammed
simultaneously by this particular type of jammer. On the other hand, in the
sweeping jamming strategy, the jammer starts jamming the RF spectrum starting
from $f_{L}$ (i.e. $f_{J,t}=f_{L})$ and gradually increasing its jamming
frequency until it reaches $f_{U}$ (i.e. $f_{J,t}=f_{U}$) in a sweeping
fashion. The change of frequency from one to the adjacent occurs at the
beginning of each time slot. In contrast, in the random jamming strategy, the
jammer randomly selects a frequency $f_{J,t}$ from the set of the available
frequencies $\left\\{f_{L},\cdots,f_{U}\right\\}$ and jams at the beginning of
every time slot. Finally, in the dynamic pattern jamming strategy, the jammer
has the capability of selecting one of the three aforementioned jamming
strategies (i.e. constant, sweeping, or random) at the beginning of each
transmission session.

###  II-B Signal Model

The received discrete baseband signal $r[n]$ at the receiver after matched
filtering and sampling at the symbol intervals can be expressed as follows

$r\left[n\right]=\sqrt{P^{rx}_{T}}\;x\left[n\right]+\sqrt{P^{rx}_{J}}\;j\left[n\right]+w\left[n\right],$
(1)

where $x[n]$ and $j[n]$ represent the discrete-time baseband signals
transmitted by the transmitter and the jammer, respectively. Furthermore,
$w[n]$ denotes the zero-mean additive white Gaussian noise (AWGN) with
variance $\sigma^{2}$. Finally, $P^{rx}_{T}$ and $P^{rx}_{J}$ represent the
average received power from the transmit and the jamming signals,
respectively, which can be written as follows

$P^{rx}_{J}=\phi^{JR}P_{J,t},$ (2)

and

$P^{rx}_{T}=\phi^{TR}P_{T,t},$ (3)

where $\phi^{JR}=\gamma_{0}d_{JR}^{-\epsilon}$ and
$\phi^{TR}=\gamma_{0}d_{TR}^{-\epsilon}$ are the channel power gains of the
jammer-receiver and transmitter-receiver links, respectively. Also,
$\gamma_{0}$ represents the channel power gain at a reference distance of 1m.
$d_{JR}$ and $d_{TR}$ are the distances of the jammer-receiver and
transmitter-receiver links, respectively. Finally, $\epsilon\geq 2$ denotes
the path loss exponent.

###  II-C Problem Formulation

The received SINR can be therefore expressed as follows

$\Theta=\frac{P^{R}}{P^{rx}_{J}+\sigma^{2}},$ (4)

where $P^{R}$ is the power received from the transmitted signal at the
receiver.

Consider $\Theta_{th}$ as the SINR threshold required for successful
transmission. The objective at time slot $t$ is to maximize the normalized
throughput, defined as $\mathcal{U}(f_{T,t})=\delta(\theta\geq\theta_{th})$,
where $\delta(x)$ is a function that equals 1 if $x$ is true, and 0 otherwise.

##  III Proposed DRL-Based Approach

In this section, we introduce a DRL-based anti-jamming scheme that obtains its
state information by scanning the entire spectrum.

###  III-A MDP Formulation

We utilize the received power feature from the generated dataset to represent
the state vector $\mathbf{P_{t}}$. Specifically, the state vector is
represented as $\mathbf{P_{t}}=[p_{t,1},p_{t,2},\cdots,p_{t,N_{c}}]$, where
$p_{t,i}$ is the received power at time $t$ for frequency $i$. The size of the
state space is $\left|\mathcal{S}\right|=N_{c}$. In our formulation, the
action $a_{t}\in\\{f_{1},f_{2},\cdots,f_{N_{c}}\\}$ represents the selection
of frequency $i$ at time slot $t$. Similarly, the action space size is
$\left|\mathcal{A}\right|=N_{c}$. The transmitter-receiver pair aims to
achieve successful transmission with a low channel switching cost $\Gamma$.
Therefore, the reward at time slot $t$ can be expressed as

$r_{t}=\begin{cases}\mathcal{U}(f_{T,t})-\Gamma\delta(a_{t}\neq
a_{t-1})&\text{ if }f_{T,t}\neq f_{J,t}\\\ 0&\text{ if
}f_{T,t}=f_{J,t}.\end{cases}$ (5)

The reward function presented in (5) takes into account the throughput factor
and ignores the energy consumption factor. This is due to the fact that in the
current anti-jamming strategy, the transmit power is fixed. Furthermore, the
normalization of the reward values to 1 and 0 is valid since the considered
jammer is proactive. Based on this, upon obtaining the reward $r_{t}$, the
environment transitions to the next state $s_{t+1}$ based on a transition
probability $p(s_{t+1}|s_{t},a_{t})$. This probability represents the
likelihood of transitioning from state $s_{t}$ to state $s_{t+1}$ given the
action $a_{t}$. The initial state is denoted by $s_{0}$ and the terminal state
is the state at which the agent ceases decision-making, which is denoted by
$s_{T}$. The goal of the agent is to find the optimal policy,
$\pi{(s)}=\arg\max_{a}Q(s,a)$, that maps the state to the best action. The
optimal policy is found by learning the optimal action-value function,
$Q^{*}(s,a)$, using an RL algorithm such as DRL.

Figure 2: Architecture of the proposed DDQN Q-network. Figure 3: Overview of
the training and deployment phases of the proposed DRL-based anti-jamming
approach.

###  III-B Agent Design

We train five different agents to determine the most suitable strategy for
power-constrained devices. These agents include DQN, DQN with fixed targets,
DDQN, Dueling DQN, and DDQN with prioritized replay. Each agent has a unique
combination of neural network architecture, experience replay mechanism, and
target network update frequency. By training and evaluating the performance of
these agents, we aim to identify the most appropriate approach for power-
constrained devices in effectively countering proactive jamming attacks.

####  III-B1 DQN

The DQN algorithm is a model-free, online, off-policy RL method in which a
value-based RL agent is employed to train a Q-network that estimates and
returns future rewards [16, 17]. The selection of this type of agent is
motivated by the fact that our observation space is continuous, and our action
space is discrete. Our DQN algorithm implementation is presented in Algorithm
1.

The implemented DQN agent uses a function approximator in the form of a neural
network, whose weights $\theta_{Q}$ are updated with every iteration. The
Q-network is used to determine the Q-value of the action. The Q-network
comprises two hidden layers, as illustrated in Fig. 2, and a ReLU activation
function $f(x)=\textup{max}(0,x)$ is chosen [18]. The experience reply buffer
$\mathcal{D}$ stores the agent’s experience, which is the transition pair at
time-step $t$ and is defined as $(s_{t},a_{t},r_{t},s_{t+1})$.

The stochastic gradient descent (SGD) algorithm [19] is used during training
to update the weights $\theta_{t}$ at every time-step $t$.

$\mathbb{Initialize}$ $\theta_{Q},\epsilon_{t}=1,\delta,i=j=0,\textup{ and
}K$;

while  _j $<\left|\mathcal{E}\right|$_ do

set $s_{t}=s_{t_{0}}$;

while  _t $<\left|\mathcal{T}\right|$_ do

$X_{t}\sim U\left(0,1\right)$;

if  _$X_{t} <\epsilon_{t}$_ then

$a_{t}=\textup{random}(1,\cdots,N_{c})$;

else

$a_{t}=\underset{a_{t}}{\textup{arg max}}Q(s_{t},a{t}\mid\theta_{Q})$;

end if

$a_{t}\mapsto\mathbb{T}$;

Obtain $r_{t}$ and $s_{t+1}$;

Store the experience $[s_{t},a_{t},r_{t},s_{t+1}]$ in $\mathcal{D}$;

Sample a random mini-batch of $K$ experiences from $\mathcal{D}$;

if  _$s_{t}==s_{t_{f}}$_ then

$y_{t}=r_{t}$;

else

$y_{t}=\mathbb{E}_{s_{t},a_{t}}[r_{s_{t},s_{t+1},a_{t}}+\gamma
Q_{\pi}(s_{t+1},a_{t_{\textup{max}}}\mid\theta_{Q})\mid s_{t},a_{t}]$;

end if

Update Q-network parameters $\theta_{Q}=\theta_{t}-\eta\nabla
L_{t}(\theta_{t})$;

where
$L_{t}(\theta_{t})=\mathbb{E}_{s_{t},a_{t}}[(y_{t}-Q_{\pi}(s_{t},a_{t};\theta_{t}))^{2}]$;

Update the exploration rate $\epsilon_{t+1}=\epsilon_{t}-\delta$;

Set $s_{t}=s_{t+1}$;

$t=t+1$;

end while

$j=j+1$;

end while

$\mathbf{Output}$ optimal policy $\overset{*}{\pi}$;

Algorithm 1 DQN Algorithm for Anti-Jamming.

####  III-B2 DQN with Fixed Targets

This variant of DQN updates the target network less frequently, reducing the
risk of oscillations and instability during learning. The algorithm is similar
to the DQN, but the target network is updated less frequently. This can be
achieved by increasing the value of $C$ (the number of steps between target
network updates). The neural network architecture and other components remain
unchanged from the DQN architecture.

####  III-B3 DDQN

The Double Deep Q-Network (DDQN) is an improvement over DQN that reduces the
overestimation of Q-values by using two separate networks to estimate the
current and target Q-values. The neural network architecture and other
components remain unchanged from the DQN architecture.

####  III-B4 Dueling DQN

This algorithm is similar to the DQN, but with a different neural network
architecture that decouples the estimation of state values and action
advantages, potentially leading to better performance and stability. To
implement this, the architecture of the Q-network in Fig. 2 is modified to
include two separate streams for state values and action advantages, and then
these streams are combined to obtain the final Q-values. The other components
remain unchanged from the DQN architecture.

####  III-B5 DQN with Prioritized Replay

This approach combines DQN with prioritized experience replay, which samples
more important experiences more frequently during learning, potentially
improving learning efficiency. To implement this, the uniform sampling of
experiences from the replay buffer $\mathcal{D}$ is replaced with prioritized
sampling based on the absolute TD-error of each experience. Additionally, the
loss function $L_{t}(\theta_{t})$ is updated to include importance-sampling
weights to correct for the bias introduced by the prioritized sampling. The
neural network architecture remains unchanged from the DQN architecture.

###  III-C Training and Deployment of the Agent

In this section, we detail the training and deployment of our proposed DRL-
based anti-jamming approach, which aims to mitigate jamming attacks in power-
constrained devices. Fig. 3 presents an overview of the training and
deployment phases of the proposed DRL-based anti-jamming approach. The
training phase involves the setup of the system, loading the corresponding
data from the spectral scan dataset, obtaining the received power (dBm)
feature of each channel, and training the agents based on the reward value
obtained from the selected channel. At the beginning, a system setup is made
to specify the type of jammer (i.e., sweeping, random, constant, or dynamic
pattern jammers), the jamming power, and the distance. Based on this setup,
the corresponding data is loaded from the spectral scan dataset. Depending on
the type of jammer, the received power (dBm) feature of each channel is
obtained. For instance, if the jammer is constant, and the jamming frequency
is 5180 MHz at 20 cm with a jamming power of 10 dBm, then the dataset with the
corresponding filename will be loaded. This ensures that the 5180 MHz
frequency will have the highest received power compared to the other
frequencies. Based on this state information, the agent will select a channel
and receive a reward value based on the selected channel, as defined in (5).
Using this reward value, the agent’s network is trained and then the
environment transitions to the next state. It is worth noting that this
process repeats until convergence or a terminal state is reached.

During the deployment phase, the trained agent is implemented within the
environment it was originally trained on. However, in this phase, the agent
does not undergo further training as it exploits the knowledge gained from the
training phase. Given a system setup and the current channel $f_{T,t}$, the
agent takes in the state vector, which describes the whole spectrum, as input
and selects the best channel $f_{T,t+1}$ to switch to. If the selected channel
$f_{T,t+1}$ is the same as the current channel $f_{T,t}$, then transmission
continues on $f_{T,t}$. If $f_{T,t+1}\neq f_{T,t}$, a channel switch
announcement (CSA) is carried out, and the subsequent transmission switches to
$f_{T,t+1}$. This process keeps repeating until all data is transmitted or the
terminal state is reached.

##  IV Results and Discussions

To evaluate the proposed DRL-based anti-jamming solution, we aim to
investigate its performance under dynamic pattern jamming, where the jammer
randomly selects one of the three jamming patterns namely, sweep, random, and
combined at the beginning of each transmission session. This evaluation is
important as our primary objective is to develop a generic anti-jamming agent
capable of mitigating various jamming patterns. We perform the simulations
using a custom-based simulator designed based on the collected dataset in
[15]. Also, unless otherwise stated, the simulation parameters used in our
study are presented in Table I. Furthermore, we tune the hyper-parameters of
the proposed DRL-based anti-jamming scheme during training to achieve a good
policy for the agent, as shown in Table II. Finally, we investigate the
effects of the $\Gamma$ parameter on the total throughput of the proposed
framework, and we compare the results obtained by using different values of
$\Gamma$.

TABLE I: Simulation Parameters Parameter | Value
---|---
RF spectrum band | 5GHz UNII-1
Bandwidth of communication signal | 20 MHz
Bandwidth of jamming signal | 20 MHz
Number of channels $N_{c}$ | 8
Initial channel center frequency $f_{T,0}$ | 5.180 GHz
Distance between channel frequencies | 20 MHz
Distance between jammer and transmitter $d_{JT}$ | 20 cm
Jamming power $P_{J,t}$ | 10d Bm
TABLE II: DRL Hyper-parameters. Parameter | Value
---|---
Number of training episodes $\left|\mathcal{E}\right|$ | 100
Number of testing episodes $\left|\mathcal{E}\right|$ | 100
Number of time-steps $\left|\mathcal{T}\right|$ | 100
Discount factor $\gamma$ | 0.95
Initial exploration rate $\zeta$ | 1
Exploration decay $\delta$ | 0.005
Minimum exploration rate $\zeta_{\textup{min}}$ | 0.01
Experience buffer size $\mathcal{D}$ | 10000
Minimum batch size $K$ | 32
Averaging window size | 10
Early termination criterion | Average reward = 90
Channel switching cost $\Gamma$ | [0, 0.05, 0.1, 0.15]

(a) $\Gamma=0.00$.

(b) $\Gamma=0.05$.

(c) $\Gamma=0.10$.

(d) $\Gamma=0.15$.

Figure 4: Learning performance of the investigated DRL-based anti-jamming
agents under dynamic pattern jamming with $\Gamma=0,0.05,0.1,0.15.$

Fig. 4 depicts the learning performance of the DRL-based anti-jamming agents
under dynamic pattern jamming, with different values of $\Gamma$. We observe
that DQN with fixed Q-targets, DDQN, and DDQN with prioritized replay achieve
a mean reward of approximately 100, while Dueling DQN achieves a mean reward
of around 95. However, the DQN agent only manages to obtain a mean reward of
approximately 86, and this failure persists for all values of $\Gamma$. Unlike
in our prior work, [14], in this work all the agents were able to learn the
dynamics of the system and evade the jammer. Importantly, we note that all the
trained DRL agents, except for DQN, can learn a policy to escape the dynamic
pattern jamming. Moreover, we observe that for all types of jammers, the DRL
agents can make intelligent channel selection decisions to evade jamming.
Interestingly, the DDQN with prioritized replay achieved the most stable
learning convergence across all values of $\Gamma$.

In Fig. 5, we present the normalized mean throughput of the legitimate user
under various jamming patterns. We observe that, for all values of $\Gamma$,
all the evaluated agents, except DQN, have the ability to completely evade
dynamic pattern jamming. Moreover, for all agents, we observe a reduction in
throughput as the value of $\Gamma$ increases, with a greater reduction for
higher values of $\Gamma$. As seen in the case of the learning performance,
the DDQN with prioritized replay achieved a consistently high throughput over
all values of $\Gamma$.

The impact of $\Gamma$ on the channel switching behavior of the agents is
demonstrated in Fig. 6. It is observed that the agents switch channels 100% of
the time, regardless of the values of $\Gamma$. This indicates that in order
to evade dynamic pattern jamming, the agents develop a policy that maps the
states to the optimal action and ignores the jamming pattern. This leads to
continuous channel switching even under values of $\Gamma>0$. In other words,
the agents choose to be penalized by the channel switching cost and experience
a reduction in overall throughput instead of remaining on a single channel and
losing 1/8 of their total throughput.

Figure 5: Normalized throughput performance of the DRL-based anti-jamming
agent under dynamic pattern jamming. Figure 6: Impact of channel switching
cost ($\Gamma$) on the DRL-based anti-jamming agent under dynamic pattern
jamming.

Finally, we study the convergence times and inference speeds of the five DRL
agents as shown in Table III. During training, the DQN agent demonstrated the
fastest convergence speed among all the agents, with an average convergence
time of 388.28 seconds. The speed of convergence and inference in DRL agents
is determined by the complexity of the learning algorithm and the efficiency
of the exploration strategy. DQN, with its simpler learning algorithm and
efficient exploration, converges faster. On the other hand, DDQN with
prioritized replay memory involves more complex computations and a more
sophisticated memory management system, which slows down both the convergence
and the inference speed.

TABLE III: Comparison of the Convergence and inference times for the five
Agents. The results are present in the format of mean ($\pm$ std.) obtained
from 10-folds.

Agent | Convergence Time (sec) | Inference Speed (KHz)
---|---|---
DQN |  388.28 ($\pm$ 3.62) |  507.23 ($\pm$ 4.30)
DQN with Fixed Targets |  405.37 ($\pm$ 1.74) |  472.43 ($\pm$ 2.15)
DDQN |  457.42 ($\pm$ 3.26) |  437.78 ($\pm$ 3.43)
Dueling DQN |  405.79 ($\pm$ 6.54) |  464.58 ($\pm$ 2.87)
DDQN with Prioritized Replay |  532.85 ($\pm$ 3.91) |  382.31 ($\pm$ 5.25)

Overall, all the algorithms investigated showed good performance in jamming
detection and avoidance. The inference speed of the algorithms varied, with
DQN being the fastest during training. Among all DRL-based approaches, DDQN
with prioritized replay memory offers the best trade-off between throughput
and speed.

##  V Conclusions

This paper investigates the intelligent anti-jamming problem within a dynamic
jamming environment. In our endeavor to construct a more practical scheme, we
incorporated a jamming detection testbed and jamming data acquired from actual
WLAN network interface cards. Utilizing this dataset, we developed a custom
simulation and introduced a DRL agent with a fully connected neural network
architecture to navigate the intricate decision-making problem inherent to
anti-jamming. With our proposed scheme, the agent is capable of learning the
most effective anti-jamming strategy through a continuous process of trial and
error, testing various actions, and observing their environmental impact. We
used simulation results from a variety of environmental settings to
corroborate the effectiveness of the proposed DRL-based anti-jamming scheme.
It’s important to note, however, that a high-power wideband jammer leaves no
room for evasion. Consequently, future research will involve creating an anti-
jamming technique focused on confronting the jammer at the same frequency, as
opposed to evasion or concealment.

## References

  * [1] S. Haykin, “Cognitive radio: brain-empowered wireless communications,” _IEEE J. Sel. Areas Commun._ , vol. 23, no. 2, pp. 201–220, 2005.
  * [2] A. G. Fragkiadakis, E. Z. Tragos, and I. G. Askoxylakis, “A survey on security threats and detection techniques in cognitive radio networks,” _IEEE Commun. Surv. Tutor._ , vol. 15, no. 1, pp. 428–445, 2012.
  * [3] D. Torrieri, _Principles of spread-spectrum communication systems_. Springer, 2005, vol. 1.
  * [4] Y. Xu, G. Ren, J. Chen, Y. Luo, L. Jia, X. Liu, Y. Yang, and Y. Xu, “A one-leader multi-follower bayesian-stackelberg game for anti-jamming transmission in UAV communication networks,” _IEEE Access_ , vol. 6, pp. 21 697–21 709, 2018.
  * [5] H. Noori and S. Sadeghi Vilni, “Jamming and anti-jamming in interference channels: A stochastic game approach,” _IET Commun._ , vol. 14, no. 4, pp. 682–692, 2020.
  * [6] I. K. Ahmed and A. O. Fapojuwo, “Stackelberg equilibria of an anti-jamming game in cooperative cognitive radio networks,” _IEEE Trans. Cogn. Commun._ , vol. 4, no. 1, pp. 121–134, 2017.
  * [7] X. Liu, Y. Xu, L. Jia, Q. Wu, and A. Anpalagan, “Anti-jamming communications using spectrum waterfall: A deep reinforcement learning approach,” _IEEE Commun. Lett._ , vol. 22, no. 5, pp. 998–1001, 2018.
  * [8] S. Liu, Y. Xu, X. Chen, X. Wang, M. Wang, W. Li, Y. Li, and Y. Xu, “Pattern-aware intelligent anti-jamming communication: A sequential deep reinforcement learning approach,” _IEEE Access_ , vol. 7, pp. 169 204–169 216, 2019.
  * [9] S. Machuzak and S. K. Jayaweera, “Reinforcement learning based anti-jamming with wideband autonomous cognitive radios,” in _Proc. IEEE Int. Conf. Commun. China (ICCC)_. IEEE, 2016, pp. 1–5.
  * [10] J. Xu, H. Lou, W. Zhang, and G. Sang, “An intelligent anti-jamming scheme for cognitive radio based on deep reinforcement learning,” _IEEE Access_ , vol. 8, pp. 202 563–202 572, 2020.
  * [11] N. Gao, Z. Qin, X. Jing, Q. Ni, and S. Jin, “Anti-intelligent UAV jamming strategy via deep Q-networks,” _IEEE Trans. Commun._ , vol. 68, no. 1, pp. 569–581, 2019.
  * [12] L. Xiao, D. Jiang, D. Xu, H. Zhu, Y. Zhang, and H. V. Poor, “Two-dimensional antijamming mobile communication based on reinforcement learning,” _IEEE Trans. Veh. Technol._ , vol. 67, no. 10, pp. 9499–9512, 2018.
  * [13] Y. Bi, Y. Wu, and C. Hua, “Deep reinforcement learning based multi-user anti-jamming strategy,” in _Proc. IEEE Int. Conf. Commun. (ICC)_. IEEE, 2019, pp. 1–6.
  * [14] A. S. Ali, W. T. Lunardi, L. Bariah, M. Baddeley, M. A. Lopez, J.-P. Giacalone, and S. Muhaidat, “Deep reinforcement learning based anti-jamming using clear channel assessment information in a cognitive radio environment,” in _Proc. 5th Int. Conf. Adv. Commun. Tech. Netw. (CommNet)_ , 2022, pp. 1–6.
  * [15] A. S. Ali, G. Singh, W. T. Lunardi, L. Bariah, M. Baddeley, M. Andreoni _et al._ , “RF jamming dataset: A wireless spectral scan approach for malicious interference detection,” _TechRxiv. Preprint_ , 2022.
  * [16] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [17] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” _arXiv_ , vol. abs/1312.5602, 2013.
  * [18] V. Nair and E. G. Hinton, “Rectified linear units improve restricted boltzmann machines,” in _Proc. Int. Conf. Mach. Learn. (ICML)_ , Jul. 2010, pp. 2094–2100.
  * [19] L. Bottou and O. Bousquet, “The tradeoffs of large scale learning,” in _Proc. Int. Conf. Adv. Neural. Inf. Process Syst._ , 2008, pp. 161–168.