# DRL-Based Dynamic Channel Access and SCLAR Maximization for Networks Under
Jamming

Abdul Basit1, Muddasir Rahim1, Georges Kaddoum1,3, Tri Nhu Do2, and Nadir
Adam1 3 Artificial Intelligence & Cyber Systems Research Center, Lebanese
American University
2 Department of Electrical Engineering, Polytechnique Montreal, QC, Canada,
Email<EMAIL_ADDRESS>1 Department of Electrical Engineering, École de
Technologie Supérieure (ÉTS), Université du Québec, Montréal, QC, Canada,
Emails<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This paper investigates a deep reinforcement learning (DRL)-based approach for
managing channel access in wireless networks. Specifically, we consider a
scenario in which an intelligent user device (iUD) shares a time-varying
uplink wireless channel with several fixed transmission schedule user devices
(fUDs) and an unknown-schedule malicious jammer. The iUD aims to harmoniously
coexist with the fUDs, avoid the jammer, and adaptively learn an optimal
channel access strategy in the face of dynamic channel conditions, to maximize
the network’s sum cross-layer achievable rate (SCLAR). Through extensive
simulations, we demonstrate that when we appropriately define the state space,
action space, and rewards within the DRL framework, the iUD can effectively
coexist with other UDs and optimize the network’s SCLAR. We show that the
proposed algorithm outperforms the tabular Q-learning and a fully connected
deep neural network approach.

###### Index Terms:

Deep reinforcement learning, medium access control, jamming attacks, residual
neural network.

##  I Introduction

The expansion of wireless technologies and the growing demand for high-speed
data are contributing to making modern networks more complex. As networks
become increasingly complex, effective network management becomes more crucial
for operational efficiency. The medium access control (MAC) layer is pivotal
for optimizing data transmission because it effectively manages multiple
users’ access to a shared channel. In wireless networks, the fair and optimal
allocation of limited resources among users is essential, but malicious users
like jammers disrupt this process. Jammers, in particular, severely compromise
channel utilization and network performance, which impacts throughput, delay,
and quality of service (QoS) [1]. Conventional channel access methods like
time-division multiple access (TDMA) and carrier sense multiple access (CSMA)
often underperform in heterogeneous networks due to their inability to adapt
to changing network conditions.

Prior research has extensively investigated using artificial intelligence
(AI)-based solutions to address MAC-layer challenges. For instance, [2]
advocates improving the performance of wireless networks using machine
learning (ML)-based solutions to manage radio resources. In [3], a Q-learning
random-access strategy is introduced for ultra-dense networks with non-
orthogonal multiple access (NOMA). It involves a centralized base station (BS)
utilizing observed channel status to select optimal transmission power.
Similarly, in [4], deep reinforcement learning (DRL) is employed to predict
spectrum usage in cognitive radio (CR) networks to reduce collisions with
neighboring networks, thus enhancing network performance.

The authors of [5] propose a deep learning-based CSMA protocol for Wi-Fi
networks, and the authors of [6] focus on using supervised learning to detect
MAC-layer attacks in cognitive radio networks (CRNs) to improve performance in
their respective types of networks. The authors of [7], on the other hand, put
forward a DRL-based unmanned aerial vehicle (UAV)-altitude random access
approach for Internet of Things (IoT) networks. Finally, the authors of [8]
propose to use a DRL-based smart node that can coexist with other nodes
utilizing slot-based protocols to ensure adaptive access. Addressing
performance degradation involves introducing intelligent user devices (iUDs)
that can learn optimal channel access strategies on the fly and enhance
network performance by reducing collisions and jamming susceptibility. More
specifically, AI and ML techniques enable iUDs to make informed transmission
decisions in dynamic wireless settings.

To this end, we propose a DRL-based iUD that can coexist with legitimate
fixed-schedule user devices (fUDs) and jammers in a quasi-static network. In
the network considered, all UDs, including fUDs and the iUD, transmit to an
access point (AP) through a shared time-slotted uplink channel, and the jammer
disrupts communication between the UDs and the AP. The uplink data channel
undergoes frequent transformations, which causes the transmit power, channel
coefficients, and path loss parameters of all UDs and jammers to vary.
Consequently, each UD’s signal-to-interference-plus-noise ratio (SINR) and
achievable rate continually fluctuate. We formulate the channel access problem
as a partially observable Markov decision process (POMDP) in a constantly
changing channel environment with jammers and use a deep Q-network (DQN) to
approximate the optimal action-value function [9].

Our proposed approach enables the iUD to effectively coexist with fUDs and
avoid jammers, thus enhancing channel utilization and the network’s sum cross-
layer achievable rate (SCLAR) by minimizing collisions and jamming
susceptibility. The methodology involves formulating the channel access
problem as a POMDP in dynamic channel conditions with jammers. We employ a
residual deep neural network (ResDNN) for optimal action-value function
approximation and conduct simulations to demonstrate that our proposed
approach outperforms the iUD that uses a tabular Q-learning approach, or a
fully connected deep neural network (FC-DNN) to approximate the action-value
function in terms of channel utilization, SCLAR, and jamming robustness.

##  II System Model

We consider uplink transmission in a quasi-static wireless network in which
each UD’s transmission schedule changes slowly over time. There is one
$\mathrm{AP}$ with $K$ antennas, $N$ single-antenna UDs, and $M$ jammers in
the network in question. The UDs transmit data packets to the $\mathrm{AP}$
via a shared time slotted ($t=\\{0,1,2,3,\cdots\\}$) uplink channel, and the
jammer always tries to attack the data channel, as illustrated in Fig. 1. Let
$[\boldsymbol{\mathrm{A}}]_{n,n}\in\\{0,1\\}$ denote the transmission status
of the $n^{th}$ UD, where $1$ means that the UD is transmitting a data packet
in time slot $t$ and $0$ means that it is not transmitting any packet.
Furthermore, the status of the $m^{th}$ jammer $\mathrm{J}_{m}$ in time slot
$t$ can be denoted by
$[\boldsymbol{\mathrm{I}}_{\mathrm{J}}]_{m,m}\in\\{0,1\\}$ with $1$ meaning
the jammer is active and $0$ meaning it is not. At the end of each time slot,
the $\mathrm{AP}$ broadcasts an acknowledgment packet (ACK) to the UDs over a
separate control channel. It is assumed that the jammer has no impact on the
control channel.

Figure 1: Illustration of the network with legitimate and malicious user
devices communicating with the access point.

###  II-A Slotted Time-Based MAC

It is assumed that all the UDs are synchronized with the $\mathrm{AP}$. The
total time is split into $T=\\{T_{1},T_{2},\cdots,T_{f},\cdots,T_{F}\\}$
repeated frames, where the subscript $\\{1,2,\cdots,f,\cdots,F\\}$ denotes the
frame number, and $F$ is the total number of frames. Moreover, a single frame
consists of a fixed number of time slots. For instance, the $f^{th}$ frame is
divided into
$t_{f}=\\{t^{1}_{f},t^{2}_{f},\cdots,t^{s}_{f},\cdots,t^{S}_{f}\\}$ slots,
where the superscript $\\{1,2,\cdots,s,\cdots,S\\}$ represents the time slot
number and $S$ denotes the total number of time slots in the frame, i.e.,
$t^{s}_{f}$ is the $s^{th}$ time slot in the $f^{th}$ frame. Packet
transmission starts at the beginning of a time slot, and a single packet takes
up to a one time slot to transmit. Furthermore, each UD can transmit multiple
packets in a single frame. Thus, multiple UDs can share a radio frequency
channel at time slot $t$. In the event multiple UDs, i.e., $\mathrm{UD}_{n}$
and $\mathrm{UD}_{n^{\prime}}\in\mathcal{N}\backslash\\{\mathrm{UD}_{n}\\}$
try to transmit simultaneously in the time slot $t$, a collision happens.
Additionally, if at a given time slot $t$ $[\boldsymbol{\mathrm{I}}_{\tt
J}]_{m,m}$ = 1 and $\mathrm{UD}_{n}$ transmits a packet, jamming occurs and
$\mathrm{UD}_{n}$’s packet is lost.

####  II-A1 Stochastic fUDs

In the network in question, an fUD is a type of UD that has a predefined
transmission schedule that has been assigned in a stochastic manner. Consider
a coefficient [$\chi$] that equals 1 if $\mathrm{UD}_{n}$ transmits a packet
in time slot $t^{s}_{f}$ and 0 otherwise. This coefficient models the
transmission status of $\mathrm{UD}_{n}$, which follows a Bernoulli
distribution that is parameterized by $\Omega$, i.e.,
$[\chi]\sim$Bern$(\Omega)$. Therefore, the signal to generate an fUD’s
transmission vector in a frame $t_{f}=[0,1,0,0,1,...]$ can be represented
mathematically as

$\displaystyle
P([\boldsymbol{\mathrm{A}}]_{n,n})=\begin{cases}1-\Omega&[\chi]=0\\\
\Omega&[\chi]=1.\end{cases}$ (1)

####  II-A2 Random Jamming

We consider there to be a random jammer present in the network that transmits
jamming signals intermittently according to a predetermined attack strategy
with the sole intention of disrupting fUDs and iUD’s transmission. The jammer
transmits in predefined $t_{f}^{s}$ time slots out of total $T_{F}$ slots and
repeats its jamming pattern from frame to frame. The jammer’s jamming schedule
$I_{\tt J}$ in the $F^{th}$ frame is denoted as
$\boldsymbol{\mathrm{I}}^{F}_{J}$ and can be expressed as

$\displaystyle\boldsymbol{\mathrm{I}}^{F}_{\tt
J}=\begin{cases}1,&\big{[}(n-1)S_{o}\big{]}\leq
t^{s}_{f}\leq\big{[}(n-1)S_{o}+S_{o^{\prime}}\big{]}\\\
0,&\big{[}(n-1)S_{o}+S_{o^{\prime}}\big{]}\leq
t^{s}_{f}\leq[nS_{o}],\end{cases}$ (2)

where $n\in\mathbb{N}$ is the period number, which is set to a predefined
value. Furthermore, $S_{o^{\prime}}\in S,S_{o^{\prime}}<S_{o}$, and
$S_{o^{\prime}}=[0,S_{o^{\prime}}]$ is the time period in which the jammer is
inactive and the transmission of $\mathrm{UD}_{n}$ is thus unaffected.
Similarly, $S_{o}\in S$, and $S_{o}=[S_{o},S]$ is the time period in which the
jammer is active and any packets transmitted by $\mathrm{UD}_{n}$ are
destroyed. Furthermore, it holds that $S_{o^{\prime}}+S_{o}=S$.

####  II-A3 DRL-Based iUD

The DRL-based iUD is a UD that uses our proposed channel access policy. The
iUD is oblivious to 1) the number of coexisting legitimate UDs, 2) the
transmission schedules of coexisting legitimate UDs, 3) the presence and
quantity of jammers, and 4) the jammers’ operating mechanism. The iUD’s goal
is to learn the transmission schedule of coexisting legitimate UDs and jammers
in order to opportunistically enact an optimal transmission schedule and
enhance network performance.

###  II-B Signal Modelling of the Network Considered

####  II-B1 Signal Received at the $\mathrm{AP}$

Let $\boldsymbol{\mathrm{h}}_{n}\in\mathbb{C}^{K\times 1}$ and
$\boldsymbol{\mathrm{g}}_{m}\in\mathbb{C}^{K\times 1}$ be the channel vectors
between the $n^{th}$ UD and the $\mathrm{AP}$ and the $m^{th}$ jammer and the
$\mathrm{AP}$, respectively. Furthermore, the channel matrices between the $N$
UDs and the $\mathrm{AP}$ and the $M$ jammers and the $\mathrm{AP}$ are
written as $\boldsymbol{\mathrm{H}}\in\mathbb{C}^{K\times N}$ and
$\boldsymbol{\mathrm{G}}\in\mathbb{C}^{K\times M}$, respectively. We ignore
large-scale fading and assume that the elements of $\boldsymbol{\mathrm{H}}$
are independent and identically Gaussian distributed with zero mean and unit
variance. Let $\boldsymbol{\mathrm{A}}$ denote the network operation-action
matrix of $N$ UDs and, as defined earlier, $[\boldsymbol{\mathrm{A}}]_{n,n}$
is the transmission status of the $n^{th}$ UD. Let $x_{n}$ be the signal
transmitted by the $\mathrm{UD}_{n}$ to the $\mathrm{AP}$. Furthermore, the
transmitted signal vector
$\boldsymbol{\mathrm{x}}_{\mathrm{UD}}=[x_{1},\cdots,x_{n},\cdots,x_{N}]\in\mathbb{C}^{N\times
1}$ is an $[N\times 1]$ column vector that contains the transmitted signals.
The transmit power matrix
$\boldsymbol{\mathrm{P}}_{\mathtt{UD}}=\operatorname{diag}(P_{1},P_{2},\cdots,P_{n},\cdots,P_{N})$
is an $[N\times N]$ diagonal matrix that contains the transmit power of each
UD in the diagonal elements where ${P}_{n}$ represents the transmit power of
$\mathrm{UD}_{n}$. Since $N$ UDs share the same time-frequency resources, the
received signal vector $\boldsymbol{\mathrm{y}}$ is a $[K\times 1]$ column
vector that contains the signals received at the $\mathrm{AP}$ from all $N$
UDs. Therefore, the signal received at the $\mathrm{AP}$ is given by

$\displaystyle\boldsymbol{\mathrm{y}}=\\!\\!{\sum_{n=1}^{N}[\boldsymbol{\mathrm{A}}]_{n,n}\sqrt{{P_{n}}}\boldsymbol{\mathrm{h}}_{n}x_{n}}+\sum_{m=1}^{M}[\boldsymbol{\mathrm{I}}_{\tt
J}]_{m,m}\sqrt{{P_{m}}}\boldsymbol{\mathrm{g}}_{m}x_{m}\\!\\!\\!+\\!\boldsymbol{\mathrm{n}},$
(3)

where $\boldsymbol{\mathrm{n}}\in\mathbb{R}^{K\times 1}$ denotes the additive
white Gaussian noise (AWGN) vector at the $\mathrm{AP}$ with
$E[|\boldsymbol{\mathrm{n}}\boldsymbol{\mathrm{n}}^{T}|^{2}]=\sigma^{2}\boldsymbol{\mathrm{I}}_{K}$,
$\boldsymbol{\mathrm{A}}=\operatorname{diag}\big{[}[\boldsymbol{\mathrm{A}}]_{1,1},[\boldsymbol{\mathrm{A}}]_{2,2},\cdots,[\boldsymbol{\mathrm{A}}]_{{N},{N}}\big{]}$
is an $[N\times N]$ diagonal matrix and $\boldsymbol{\mathrm{I}}_{\tt
J}=\operatorname{diag}\big{[}[\boldsymbol{\mathrm{I}}_{\tt
J}]_{1,1},[\boldsymbol{\mathrm{I}}_{\tt
J}]_{2,2},\cdots,[\boldsymbol{\mathrm{I}}_{\tt J}]_{M_{\tt J},M_{\tt
J}}\big{]}$ is an $[M_{\tt J}\times M_{\tt J}]$ diagonal matrix.

####  II-B2 Linear Detection at the $\mathrm{AP}$ and SINR Formulation

Let $\hat{\boldsymbol{\mathrm{x}}}_{\mathtt{UD}}\in\mathbb{C}^{N\times 1}$ be
estimated using a linear receiver. If we consider the linear detection matrix
$\boldsymbol{\mathrm{V}}\in\mathbb{C}^{K\times N}$, which is used to separate
the signal received at the $\mathrm{AP}$ into $N$ streams,
$\hat{\boldsymbol{\mathrm{x}}}_{\mathtt{UD}}$ can be expressed as

$\displaystyle\scalebox{0.94}{$\hat{\boldsymbol{\mathrm{x}}}_{\mathtt{UD}}=\boldsymbol{\mathrm{V}}^{\sf
H}\boldsymbol{\mathrm{y}}=\textstyle\boldsymbol{\mathrm{V}}^{\sf
H}\boldsymbol{\mathrm{H}}\sqrt{{\boldsymbol{\mathrm{P}}_{\mathtt{UD}}}}\boldsymbol{\mathrm{A}}\boldsymbol{\mathrm{x}}_{\mathtt{UD}}+\boldsymbol{\mathrm{V}}^{\sf
H}\boldsymbol{\mathrm{G}}\sqrt{{\boldsymbol{\mathrm{P}}_{\mathtt{J}}}}\boldsymbol{\mathrm{I}}_{\mathtt{J}}\boldsymbol{\mathrm{x}}_{\tt
J}+\boldsymbol{\mathrm{V}}^{\sf H}\boldsymbol{\mathrm{n}}$}.$ (4)

If we consider matched filter successive interference cancellation (MF-SIC),
the SINR for $\mathrm{UD}_{n}$, $\Gamma_{n}$, can be obtained as follows by
removing the interference from other UDs with MF-SIC.

(5)

##  III Problem Formulation

###  III-A Sum Cross-Layer Achievable Rate Formulation

Cross-layer achievable rate (CLAR) is a technique that jointly considers
physical-layer and MAC-layer issues in an integrated framework. The physical-
layer achievable rate of $\mathrm{UD}_{n}$ in a single time slot $t^{s}_{f}$
can be represented as

$\displaystyle C_{{}_{n}}[t^{s}_{f}]=\log_{2}(1+\Gamma_{n}[t^{s}_{f}]),$ (6)

where $C_{n}[t^{s}_{f}]$ is the achievable rate in bits per slot per hertz.
Furthermore, the achievable rate of $\mathtt{UD}_{n}$ for the frame $t_{f}$
can be expressed as

$\displaystyle\boldsymbol{\mathrm{C}}_{n}[t_{f}]$
$\displaystyle=\textstyle\sum_{n=1}^{{N}}\sum_{s=1}^{S}C_{n}[t^{s}_{f}].$ (7)

Successful transmission rate ($\xi$) is a MAC-layer metric that measures the
number of data packets that are successfully transmitted by $\mathrm{UD}_{n}$
to the $\mathrm{AP}$ over a given period of time and can be expressed as

$\displaystyle\xi_{n}[t_{f}]=\frac{p_{\mathtt{S}-\mathtt{UD}_{n}}[t^{1}_{f}]+p_{\mathtt{S}-\mathtt{UD}_{n}}[t^{2}_{f}],\cdots,p_{\mathtt{S}-\mathtt{UD}_{n}}[t^{S}_{f}]}{S},$
(8)

where $p_{\mathtt{S}-\mathtt{UD}_{n}}[t^{s}_{f}]$ is the probability of
successful transmission of $\mathtt{UD}_{n}\rightarrow\mathtt{C}$ in time slot
$t^{s}_{f}$. The CLAR of $\mathtt{UD}_{n}$ in time slot $(t^{s}_{f})$ is
denoted by $R_{n}[t^{s}_{f}]$ and can be expressed as

$\displaystyle
R_{n}[t^{s}_{f}]=\xi_{\mathtt{UD}_{n}}[t^{s}_{f}]C_{n}[t^{s}_{f}].$ (9)

Furthermore, $\boldsymbol{\mathrm{r}}_{n}\in\mathbb{R}^{T\times 1}$ represents
the CLAR of an entire frame and can be expressed as

$\displaystyle\boldsymbol{\mathrm{r}}_{n}=[R_{n}[t^{1}_{f}],R_{n}[t^{2}_{f}],\cdots,R_{n}[t^{s}_{f}],\cdots,R_{n}[t^{S}_{f}]].$
(10)

The SCLAR maximization (SCLARM) is an optimization problem that aims to find
the optimal resource allocation to maximize the SCLAR which we elaborate on
next.

###  III-B SCLARM Problem Formulation

The network contains $N$ legitimate UDs, i.e., $|\mathcal{N}|=N$, of which
$(N-1)$ are fUDs and one is an iUD. The UD action vector
$\boldsymbol{\mathrm{a}}_{n}\in\\{0,1\\}^{T\times 1}$, where
$\boldsymbol{\mathrm{a}}_{n}=\\{[\boldsymbol{\mathrm{A}}_{n}(t):k=1,...,|\mathcal{C}|,n=1,...,N,t=0,...,T-1\\}$,
is the network’s UD action vector of a given $\mathrm{UD}_{n}$ over a given
time period $T$ and describes the UD’s operation. The SCLAR maximization
(SCLARM) optimization problem can be written as

$\displaystyle\underset{\boldsymbol{\mathrm{A}}}{\mathrm{maximize}}$
$\displaystyle\textstyle\sum_{n=1}^{|\mathcal{N}|}(\boldsymbol{\mathrm{r}}_{n})^{\sf
T}\boldsymbol{\mathrm{a}}_{n},$ (11a) subject to $\displaystyle
a_{n}(t)\in\\{0,1\\},\forall t\in\\{0,...,T-1\\},$ (11b)
$\displaystyle\text{unknown }\boldsymbol{\mathrm{a}}_{\tt J}(t),\forall
t\in\\{0,...,T-1\\},$ (11c)

where $\boldsymbol{\mathrm{A}}$ in (11a) can be expressed as

$\displaystyle\boldsymbol{\mathrm{A}}=[\boldsymbol{\mathrm{a}}_{{\tt
UD}_{1}},\cdots,\boldsymbol{\mathrm{a}}_{{\tt
UD}_{n}},\cdots,\boldsymbol{\mathrm{a}}_{\mathtt{iUD}}].$ (12)

##  IV The POMDP Formulated and the DQN Solution Proposed

In modern wireless networks, decision-making problems are generally modeled
using the MDP [10]. A variant, the POMDP, is used when the agent has
incomplete information about its environment. To be able to apply
reinforcement learning (RL) algorithms to the channel allocation optimization
problem, the problem, including the agent, the action space $\mathcal{A}$, the
state space $\mathcal{S}$, the instantaneous reward function $r$, the
transition probability $\mathcal{P}$, and the policy $\pi$ must be converted
to the POMDP framework.

At time $t$, the agent observes a state $s_{t}\in\mathcal{S}$ and takes action
$a_{t}\in\mathcal{A}$, resulting in a new state $s_{t+1}$ with a transition
probability $\mathcal{P}$ and an instantaneous reward $r_{t+1}$. The
interaction continues until the agent learns the optimal policy $\pi^{*}$. The
agent’s policy $\pi$ maps states to actions
$\mathcal{S}\leftarrow\mathcal{A}$, i.e., $a_{t}=\pi(s_{t})$. The long-term
reward is defined as the expected accumulated discounted instantaneous reward
and given by $\mathcal{R}=\mathbb{E}[\sum_{t=1}^{T}\gamma
r_{t+1}(s_{t},\pi(s_{t}))]$. The parameter $0\leq\gamma\leq 1$ is the discount
factor. The main goal is to obtain the optimal decision policy $\pi^{*}$ that
maximizes the long-term reward, i.e., $\pi^{*}=\max_{\pi}\mathcal{R}$.

###  IV-A The Proposed POMDP to Address the Formulated Optimization Problem

In our SCLARM problem, the iUD can only see the current channel state at each
time slot. However, it can infer a distribution over the system state based on
past actions and observations. The SCLARM problem is therefore compatible with
the POMDP framework [11]. Next, we define the SCLARM problem’s important
parameters in terms of the POMDP framework. The agent is the iUD that uses our
algorithm to maximize its objective function. $\pi^{*}$(s) serves to utilize
the spare time slots while avoiding collisions and evading jamming. In the
$t^{th}$ time slot, the iUD determines whether to transmit over the channel or
wait depending on the observations it receives from the $\mathrm{AP}$.

####  IV-A1 Action Space

A discrete action space $\mathcal{A}\dot{=}\\{a_{i}:i=1,2\\}$ represents the
iUD’s transmission decisions at each time slot. The transmission decision at
time step $t$ is defined as $a_{i}\in\\{\tt{dispatch},\tt{hold}\\}$, where
$\mathtt{dispatch}$ implies packet transmission and $\mathtt{hold}$ means no
transmission.

####  IV-A2 Channel Observation

After the iUD’s action $a_{t}$ has been taken, the $\mathrm{AP}$ broadcasts
its feedback $\mathrm{ACK}_{t+1}$, which represents the transmission status,
over a control channel. The channel observation vector is an action-
observation pair that is expressed as
$\boldsymbol{\mathrm{c}}_{t+1}=(a_{t},\mathrm{ACK}_{t+1})$ and becomes a
component of the system state.

####  IV-A3 State space

The state space
$\boldsymbol{\mathrm{\mathcal{S}}}=[\boldsymbol{\mathrm{s}}_{1},\cdots,\boldsymbol{\mathrm{s}}_{t},\cdots,\boldsymbol{\mathrm{s}}_{T}]$
contains all the possible states that the environment can transition into. The
state at time step $t+1$, which is denoted by
$\boldsymbol{\mathrm{s}}_{t+1}\in\boldsymbol{\mathrm{\mathcal{S}}}$,
represents the real-time transmission status and transmission rate of the UDs
and is expressed as

(13)

####  IV-A4 Utility Functions

We define the utility function of a legitimate $\mathrm{UD}_{n}$ $\in
N\\{n:1,2,\cdots,N\\}$ as a function of the UD’s achievable rate given by
$\mathbb{U}_{t}^{\mathtt{UD}_{n}}=\nu^{\mathtt{UD}}\times R^{\mathtt{UD}_{n}}$
and $\mathbb{U}^{\tt iUD}_{t}=\nu^{\tt UD}\times R^{\tt iUD}$, where
${\nu^{\tt UD}}$ is the scaling factor that is given in Table I. More
specifically, ${\nu^{\tt UD}}$ is used for iUD to distinguish among the
networks’ different scenarios.

TABLE I: Scaling factor $\nu^{\tt UD}$ & $\nu^{\tt net}$ and iUD decision Slot Status |  $a_{t}$ by iUD | iUD Decision | $\nu^{\tt UD}$ | $\nu^{\tt net}$
---|---|---|---|---
Jammed | hold | Good (G) | 4 | 5
fUD Transmitting | hold | Good (G) | 4 | 5
Free | hold | Worst (W) | 1 | -10
Jammed | dispatch | Worst (W) | 1 | -10
fUD Transmitting | dispatch | Bad (B) | 3 | -5
Free | dispatch | Excellent (E) | 5 | 10

####  IV-A5 Agent’s Reward Function

Since the iUD’s objective is to maximize SCLAR, we associate the definition of
reward with the utility values of all the legitimate UDs scaled by the scalar
reward ${\nu^{\tt net}}$ and an observation from the $\mathrm{AP}$. The value
of reward $r_{t+1}$ at different $O_{t+1}$ is influenced by
$\mathbb{U}^{A_{g}}_{t}$ and $\sum_{n=1}^{N}\mathbb{U}_{t}^{\mathtt{UD}_{n}}$
such that $r_{t+1}=\nu^{\tt net}(\mathbb{U}_{t}^{\tt
iUD}+\sum_{n=1}^{N}\mathbb{U}_{t}^{\tt{UD}_{n}})$ where $\nu^{\tt net}$ is the
iUD’s scaling factor set out in Table I.

####  IV-A6 Transition Probability

The state transition function $\mathcal{P}:\mathcal{S}\times
A\rightarrow\Pi(\mathcal{S})$ represents how the state changes
$s_{t}\rightarrow s_{t+1}$ with uncertainty. In our SCLARM formulation,
$\mathcal{P}$ depends on the UD’s actions and the jammer’s status and is given
by

$\displaystyle\mathcal{P}(\boldsymbol{\mathrm{s}}_{t+1}|\boldsymbol{\mathrm{s}}_{t},a_{t})$
$\displaystyle=p(a_{\tt UD_{1}}|\boldsymbol{\mathrm{s}}_{t})\times p(a_{\tt
UD_{2}}|\boldsymbol{\mathrm{s}}_{t})\times\cdots\times p(a_{\tt
iUD}|\boldsymbol{\mathrm{s}}_{t})$ $\displaystyle\quad\times\cdots\times
p(a_{\tt UD_{N}}|\boldsymbol{\mathrm{s}}_{t})\times p(a_{\tt
J_{m}}|\boldsymbol{\mathrm{s}}_{t}).$ (14)

In our case, determining (IV-A6) presents a significant challenge when it
comes to solving the optimization problem formulated in (11a) because the iUD
is not aware of network dynamics. However, as it is explained in [12],
historical trajectories generated by the iUD’s interactions with the
environment can be utilized to search for $\pi^{*}$. Thus, the iUD can learn a
policy $\pi$ directly from its experience instead of learning $\mathcal{P}$.

###  IV-B The Proposed Approach to Solve the POMDP formulated

Value-based methods are used to estimate the agent’s value function. The value
function is then utilized to implicitly obtain $\pi^{*}$. Two value functions
are used in the literature to measure the expected accumulated discounted
rewards of being in a certain state or taking a certain action in a state
based on a policy $\pi$; the state-value function $V^{\pi}(s)$ and the action-
value function $Q(s_{t},a_{t})$. More specifically, the function
$V^{\pi}(s_{t})$ indicates the value of being in a state $s_{t}$ when
following a policy $\pi$. Mathematically speaking, the state-value function
$V^{\pi}(s)$, the action-value function $Q(s_{t},a_{t})$, the optimal value
function $V^{*}(s)$, and the optimal action-value function
$Q^{*}(s_{t},a_{t})$ can be expressed as
$V^{\pi}(s)\\!=\\!\mathbb{E}\big{[}\sum_{t=0}^{\infty}\\!\gamma
r_{t+1}(s_{t},a_{t},s_{t+1})|a_{t}\\!\sim\\!\pi(.|s_{t}),s_{0}\\!=\\!s\big{]},Q^{\pi}(s,a)=\mathbb{E}\big{[}\sum_{t=0}^{\infty}\gamma
r_{t+1}(s_{t},a_{t},s_{t+1})|a_{t}\sim\pi(.|s_{t}),s_{0}=s,a_{0}=a\big{]},V^{*}(s)=\max_{a_{t}}[r_{t+1}(s_{t},a_{t})+\gamma\mathbb{E}_{\pi}V^{*}(s_{t+1})],\,\text{and}\,Q^{*}(s,a)=r_{t+1}(s_{t},a_{t})+\gamma\mathbb{E}[\max_{a_{t+1}}Q^{*}(s_{t+1},a_{t+1})]$,
respectively.

In general, the RL agent’s goal is to obtain the optimal policy
$\pi^{*}(s)=\operatorname*{argmax}_{a}Q^{*}(s,a)$. Various methodologies can
be employed to approximate $\pi^{*}(s)$, each with its unique approach. Value-
based methods, for instance, determine $\pi^{*}(s)$ by updating the $Q(s,a)$
until convergence is reached where $Q(s,a)\approx Q^{*}(s,a)$. Subsequently,
the $\pi^{*}(s)$ is then derived from $Q^{*}(s,a)$. A prevalent value-based
method used in wireless communication is Q-learning [9]. We elaborate on
Q-learning and its extensions below.

####  IV-B1 Q-Learning

Q-learning is one of the algorithms most widely used to solve problems modeled
using MDP. It is a model-free online method that learns by direct interaction
with the environment. It updates estimates using observed rewards and
transitions at each iteration and focuses on evaluating the quality of actions
in each state to gain comprehensive experience. Q-learning obtains the optimal
values of the Q-function using the following update rule that is based on the
Bellman equation

$\displaystyle Q(s_{t},a_{t})$ $\displaystyle\leftarrow
Q(s_{t},a_{t})+\alpha[r_{t+1}(s_{t},a_{t})$
$\displaystyle\qquad+\gamma\max_{a_{t+1}}Q(s_{t+1},a_{t+1})-Q(s_{t},a_{t})],$
(15)

where $\alpha$ is the learning rate, $\gamma$ is the discount factor,
$r_{t+1}(s_{t},a_{t})$ is the reward at time step $t+1$ for action $a_{t}$ in
state $s_{t}$, and $s_{t+1}$ is the subsequent state after action $a_{t}$ is
taken in state $s_{t}$. This iterative process gradually produces a Q-table
containing optimal Q-values that facilitate effective decision-making.
However, traditional Q-learning suffers from slow convergence and has a
limited capacity to handle high-dimensional and dynamic state and action
spaces [13]. These limitations are overcome by deep neural networks to
approximate $Q(s,a)$, as discussed below.

####  IV-B2 Deep Q Network

A deep Q network (DQN) algorithm integrates the favorable characteristics of
Q-learning and deep learning techniques [14]. It replaces the Q-table in
Q-learning with a deep neural network (DNN) that approximates
$Q(s,a;\boldsymbol{\mathrm{\theta}})$ by adjusting
$\boldsymbol{\mathrm{\theta}}$, which represents the neural network
parameters, through training. The action-value function for a given
$\boldsymbol{\mathrm{\theta}}$ can be computed approximately for unseen state-
action pairs $(s_{t},a_{t})$. The DQN learns $\boldsymbol{\mathrm{\theta}}$
instead of the $|\mathcal{S}|\times|\mathcal{A}|$ matrix of Q-values. The
neural network tries to find $\boldsymbol{\mathrm{\theta}}$ that approximates
$V^{*}(s)$ and $Q^{*}(s,a)$ in every state-action pair
$(s,a)\in\mathcal{S}\times\mathcal{A}$. This process is referred to as the DQN
training cycle, which is elaborated next.

####  IV-B3 The DQN Training Cycle

At time step $t$, the state of the environment is $s_{t}$, and the ResDNN’s
parameters are $\boldsymbol{\mathrm{\theta}}$. The agent takes an action
$a_{t}=\operatorname*{argmax}_{a}Q(s_{t},a_{t};\boldsymbol{\mathrm{\theta}})$
for which the DQN gives an output of
$Q(s_{t},a_{t};\boldsymbol{\mathrm{\theta}})$ for all actions in
$\mathcal{A}$. With reward $r_{t+1}$ and next state $s_{t+1}$, the tuple
$(s_{t},a_{t},r_{t+1},s_{t+1})$ forms an experience tuple that is utilized by
the DQN for training. To find the optimal $\boldsymbol{\mathrm{\theta}}$, the
DQN minimizes the prediction errors of
$Q(s_{t},a_{t};\boldsymbol{\mathrm{\theta}})$ using the loss function [9]
given by

$\displaystyle\mathcal{L}_{\tt MSE}(\boldsymbol{\mathrm{\theta}})$
$\displaystyle=\big{(}y^{\tt
True}_{r_{t+1},s_{t+1}}-Q(s_{t},a_{t};\boldsymbol{\mathrm{\theta}})\big{)}^{2},$
(16)

where $Q(s_{t},a_{t};\boldsymbol{\mathrm{\theta}})$ is the estimated Q-value
and $y^{\tt True}_{r_{t+1},s_{t+1}}$ is the DQN’s target output, which can be
calculated using the Bellman equation $y^{\tt
True}_{r_{t+1},s_{t+1}}=r_{t+1}+\gamma\max_{a_{t+1}}Q(s_{t+1},a_{t+1};\boldsymbol{\mathrm{\theta}}).$
The DQN is trained by repeatedly updating $\boldsymbol{\mathrm{\theta}}$ using
a semi-gradient algorithm given by
$\boldsymbol{\mathrm{\theta}}_{k+1}\leftarrow\boldsymbol{\mathrm{\theta}}_{k}-\alpha_{k}\nabla_{\boldsymbol{\mathrm{\theta}}}\mathcal{L}_{\tt
MSE}(\boldsymbol{\mathrm{\theta}}_{k})$, where $\alpha_{k}$ is the step size
at iteration $k$ and $\nabla_{\boldsymbol{\mathrm{\theta}}}\mathcal{L}_{\tt
MSE}(\boldsymbol{\mathrm{\theta}}_{k})$ is the gradient of the loss function
with respect to $\boldsymbol{\mathrm{\theta}}$ at iteration $k$.

The DQN algorithm uses a “quasi-static target network” technique to maintain
two networks: a prediction network
$Q(s_{t},a_{t},\boldsymbol{\mathrm{\theta}}^{\tt pred})$ and a target network
$Q(s_{t},a_{t};\boldsymbol{\mathrm{\theta}}^{\tt target})$, where
$\boldsymbol{\mathrm{\theta}}^{\tt pred}$ and
$\boldsymbol{\mathrm{\theta}}^{\tt target}$ refer to the weight parameters of
their respective network. The target network helps stabilize the learning
process by reducing the variance and correlation of the target values. The DQN
trains the prediction network at each learning step to decrease the loss
function in (16).

The target network is typically a copy of the original DQN with
$\boldsymbol{\mathrm{\theta}}^{\tt target}$ replaced by
$\boldsymbol{\mathrm{\theta}}^{\tt pred}$. Instead of immediately following
$\boldsymbol{\mathrm{\theta}}^{\tt target}$ at each training iteration, the
target network updates the values of $\boldsymbol{\mathrm{\theta}}^{target}$
using a soft update rule [15], which is expressed as

$\displaystyle\boldsymbol{\mathrm{\theta}}^{\tt
target}\leftarrow(1-\tau)\boldsymbol{\mathrm{\theta}}^{\tt
target}+(\tau)\boldsymbol{\mathrm{\theta}}^{\tt pred},$ (17)

where $\tau$ is a hyper-parameter that controls the frequency of target
updates. This technique results in the target network being updated gradually
and smoothly after a few iterations. Thus, the loss function (16) and update
equation lead to

$\displaystyle\mathcal{L}_{\tt
MSE_{B}}(\boldsymbol{\mathrm{\theta}})=\frac{1}{\tt N_{\tt BE}}\sum_{e\in\tt
EB}{\big{(}y^{\tt
True}_{r_{t+1},s_{t+1}}-q(s_{t},a_{t};\boldsymbol{\mathrm{\theta}}^{\tt
target})}\big{)}^{2},$ (18)

where $y^{\tt
True}_{r_{t+1},s_{t+1}}=r_{t+1}+\gamma\max_{a_{t+1}}q(s_{t+1},a_{t+1};\boldsymbol{\mathrm{\theta}}^{\tt
pred})$ and

$\displaystyle\boldsymbol{\mathrm{\theta}}^{\tt
pred}_{k+1}\leftarrow\boldsymbol{\mathrm{\theta}}^{\tt
pred}_{k}-\alpha_{k}\nabla_{\boldsymbol{\mathrm{\theta}}}\mathcal{L}_{\tt
MSE_{\tt B}}(\boldsymbol{\mathrm{\theta}}^{\tt
pred}_{k};\boldsymbol{\mathrm{\theta}}^{\tt target}_{k}).$ (19)

In this work, to stabilize the loss function in (18) we employ a ResDNN-based
function approximation. The pseudocode followed by the iUD is given in
Algorithm 1.

1 Input:  $\tt{N_{UDs}},p,\mathcal{A},S,\epsilon,\gamma,\tau,\tt EB,\tt
N_{BE},\boldsymbol{\mathrm{\theta}}^{pred},\boldsymbol{\mathrm{\theta}}^{target}$

2 Initialize:  $s_{0}\in\mathcal{S},a_{0}\in\mathcal{A}$, set
$s_{t+1}=0,r_{t+1}=0$

43 $\tt env\leftarrow\tt
Env(p,\alpha,s_{0},\epsilon,\epsilon_{min},\epsilon_{decay},\gamma,\tau,N_{UDs})$

5 for  _$X_{f}:\\{f=1,2,\cdots,F\\}$_ do

6 for  _$t\in X_{f}=\\{0,1,2,\cdots\,T\\}$_ do

7 Set $\lambda=1,\tt PLE=1,\Lambda=1$,

8 $NC\sim\mathcal{U}(2\,\tt dBm,\,5\,\tt dBm)$

9 $[P_{1},\cdots,P_{N},P_{\tt J},P_{\tt{iUD}}]\sim\mathcal{U}(20\,\tt
dBm,\,25\,\tt dBm)$

10 Compute $[h_{n},g_{m},h_{\tt{iUD}}]$ using $\lambda$ & $\Lambda$ $\forall
n\in N,\,\forall m\in M$.

11

12 Obtain $\Gamma_{n}$ via (5) $\forall n\in N$

13 Calculate $C_{n}$ via (7) $\forall n\in N$

14 Observe $s_{0}$ & update $a_{t}\leftarrow\epsilon$-greedy([$s_{0}$])

15 Get $r_{t+1},\mathrm{ACK}^{i}_{t+1},s_{t+1}$ from $\tt env.step$
([$a_{t}$])

16 Deposit $(s_{t},a_{t},r_{t+1},s_{t+1})$ to $\tt EB$

17 Sample $\tt{E^{N_{BE}}}$ experiences [$s_{t}^{\tt N_{BE}}$, $a_{t}^{\tt
N_{BE}}$, $r_{t+1}^{\tt N_{BE}}$, $s_{t+1}^{\tt N_{BE}}]$

18

19 for  _each $e\in\tt{E^{N_{BE}}}$ = $(s_{t},a_{t},r_{t+1},s_{t+1})$ _ do

20 Compute $Q(s_{t},a_{t};\theta^{\tt pred})$

21 Compute $Q(s_{t+1},a_{t+1};\theta^{\tt target})$

22

23 Perform Gradient Descent to update $\boldsymbol{\mathrm{\theta}}^{\tt
pred}$

24 Update $\boldsymbol{\mathrm{\theta}}^{\tt pred}$ using (19)

25 if  _$t\%\tau==0$_ then

26 Update $\boldsymbol{\mathrm{\theta}}^{\tt target}$ using (17)

27

28 Optimal $\theta^{\tt pred}\rightarrow\pi^{*}(s)$

29 Update $s_{t}\leftarrow s_{t+1}$

30 Choose $a^{*}_{t}\leftarrow\pi^{*}(s)$

31

Algorithm 1 DRL-based iUD channel access for varying frame sizes

##  V Numerical Results and Discussion

In this section, we evaluate the proposed solution’s performance. First, we
examine the DRL agent’s learning curve and training loss. Subsequently, we
demonstrate iUD’s performance at maximizing the network’s SCLAR. We used
Python for all our experimental simulations. The RL framework was simulated
using Keras and TensorFlow.

Figure 2: The convergence curves of the learning agents.

Fig. 2 illustrates the convergence of agents using ResNet, FC-DNN, and tabular
Q-learning for $Q(s_{t},a_{t})$ estimation. The ResNet-based agent converges
at high average rewards regardless of frame size variations due to its
superior capacity to learn complex patterns and generalize across states. In
contrast, the FC-DNN-based agent settles at sub-optimal rewards since it lacks
hierarchical feature learning, resulting in sub-optimal policies. Finally, the
tabular Q-learning-based agent fails to converge due to the curse of
dimensionality in large state spaces.

Fig. 3 shows the neural network’s epoch loss against the number of training
epochs. For all the frame sizes considered, the ResDNN-based DRL agent
successfully reduces the loss in each successive epoch, which indicates
improvement in its prediction capability and performance for all training
episodes.

Figure 3: ResDNN loss under different frame settings. Figure 4: SCLAR plot of
the considered network.

In Fig. 4, we analyze the iUD’s performance in terms of improving network
SCLAR considering the frame size of 5 time slots. In the initial time slots,
the iUD’s packets frequently collide with those of the coexisting fUDs and are
jammed by the jammer, resulting in the lowest SCLAR values. However, after a
few time slots, the iUD can maximize the network’s SCLAR and channel
utilization by learning and implementing an optimal transmission strategy
featuring negligible collisions and improved jamming robustness.

##  VI Conclusion

This paper presents a DRL-based iUD in a quasi-static network that learns a
transmission strategy to maximize the network’s SCLAR. The proposed approach
reduces collisions with legitimate UDs and avoids jamming. Simulation results
show that the iUD learns to coexist with other UDs under changing channel
conditions. After training, the iUD is able to select time slots for data
packet transmission and maximize the network’s SCLAR. To validate the proposed
algorithm we compare our results with tabular Q-learning and a fully connected
deep neural network approach.

## References

  * [1] M. H. Khadr, H. B. Salameh, M. Ayyash, H. Elgala, and S. Almajali, “Jamming resilient multi-channel transmission for cognitive radio IoT-based medical networks,” _Journal of Communications and Networks_ , vol. 24, no. 6, pp. 666–678, 2022.
  * [2] L. Miuccio, S. Riolo, S. Samarakoon, D. Panno, and M. Bennis, “Learning generalized wireless MAC communication protocols via abstraction,” in _IEEE GLOBECOM 2022_. IEEE, 2022, pp. 2322–2327.
  * [3] M. V. da Silva, R. D. Souza, H. Alves, and T. Abrão, “A NOMA-based Q-learning random access method for machine type communications,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 10, pp. 1720–1724, 2020.
  * [4] R. Mennes, F. A. De Figueiredo, and S. Latré, “Multi-agent deep learning for multi-channel access in slotted wireless networks,” _IEEE Access_ , vol. 8, pp. 95 032–95 045, 2020.
  * [5] J. Xin, W. Xu, Y. Cai, T. Wang, S. Zhang, P. Liu, Z. Guo, and J. Luo, “Deep learning based MAC via joint channel access and rate adaptation,” in _2022 IEEE 95th VTC2022-Spring_. IEEE, 2022, pp. 1–7.
  * [6] W. F. Fihri, H. El Ghazi, B. Abou El Majd, and F. El Bouanani, “A machine learning approach for backoff manipulation attack detection in cognitive radio,” _IEEE Access_ , vol. 8, pp. 227 349–227 359, 2020.
  * [7] S. Khairy, P. Balaprakash, L. X. Cai, and Y. Cheng, “Constrained deep reinforcement learning for energy sustainable multi-UAV based random access IoT networks with NOMA,” _IEEE J. Sel. Areas Commun._ , vol. 39, no. 4, pp. 1101–1115, 2020.
  * [8] Y. Yu, T. Wang, and S. C. Liew, “Deep-reinforcement learning multiple access for heterogeneous wireless networks,” _IEEE J. Sel. Areas Commun._ , vol. 37, no. 6, pp. 1277–1290, 2019.
  * [9] M. Morales, _Grokking deep reinforcement learning_. Manning Publications, 2020.
  * [10] T. Bozkus and U. Mitra, “Link analysis for solving multiple-access MDPs with large state spaces,” _IEEE Trans. Signal Process._ , vol. 71, pp. 947–962, 2023.
  * [11] S. Wang, H. Liu, P. H. Gomes, and B. Krishnamachari, “Deep reinforcement learning for dynamic multichannel access in wireless networks,” _IEEE Trans. Cognit. Commun. Networking_ , vol. 4, no. 2, pp. 257–265, 2018.
  * [12] S. Chen and Y. Li, “An overview of robust reinforcement learning,” in _2020 IEEE International Conference on Networking, Sensing and Control (ICNSC)_. IEEE, 2020, pp. 1–6.
  * [13] W. Yu, T. J. Chua, and J. Zhao, “User-centric heterogeneous-action deep reinforcement learning for virtual reality in the metaverse over wireless networks,” _IEEE Trans. Wireless Commun._ , 2023.
  * [14] A. Sivaranjani and B. Vinod, “Artificial potential field incorporated deep Q-network algorithm for mobile robot path prediction.” _Intelligent Automation & Soft Computing_, vol. 35, no. 1, 2023.
  * [15] T. Kobayashi and W. E. L. Ilboudo, “T-soft update of target network for deep reinforcement learning,” _Neural Networks_ , vol. 136, pp. 63–71, 2021.


# Measuring the speed of sound
using cumulants of baryon number††thanks: Presented at the XXIXth
International Conference on Ultra-relativistic Nucleus-Nucleus Collisions,
April 2022

Agnieszka Sorensen  Dmytro Oliinychenko  Larry McLerran
Volker Koch  Institute for Nuclear Theory, University of Washington, Box
351550, Seattle, Washington 98195, USA  Lawrence Berkeley National Laboratory,
1 Cyclotron Road, Berkeley, California 94720, USA

###### Abstract

We show that the values of the first three cumulants of the baryon number
distribution can be used to calculate the isothermal speed of sound and its
logarithmic derivative with respect to the baryon number density. We discuss
applications of this result to heavy-ion collision experiments and address
possible challenges, including effects due to baryon number conservation,
differences between proton and baryon cumulants, and the influence of finite
number statistics on fluctuation observables in both experiment and hadronic
transport simulations. In particular, we investigate the relation between
quantities calculated in infinite, continuous matter and observables obtained
in simulations using a finite number of particles.

##  1 Introduction

The speed of sound in dense nuclear matter has been given considerable
attention in recent years. Measurements and global analyses of neutron star
data [1, 2, 3, 4, 5, 6, 7, 8, 9] suggest that at moderate and high baryon
densities, the speed of sound squared $c_{s}^{2}$ may exceed the conformal
limit of $1/3$. While more studies dedicated to the cold asymmetric nuclear
matter will be necessary to confirm this striking behavior, here we propose a
complimentary method of studying the speed of sound in dense (nearly
symmetric) nuclear matter using observables accessible in heavy-ion
collisions.

We are aware of three constraints on $c_{s}^{2}$ obtained with heavy-ion
collision data. In [10], the speed of sound in matter created in ultra-
relativistic collisions, where the chemical potential $\mu_{B}\approx 0$, has
been estimated using the proportionality of the entropy density and
temperature to the charge particle multiplicity and mean transverse momentum,
respectively, yielding $c_{s}^{2}=0.24\pm 0.04$, consistent with lattice QCD
data [11]. In [12], which used data from collisions probing regions of finite
baryon number density $n_{B}$, requiring the Landau model as well as hybrid
hydrodynamics and hadronic transport simulations to reproduce the widths of
the negatively charged pion rapidity distribution leads to a prediction for a
minimum of $c_{s}^{2}$ within the collision energy range
$\sqrt{s_{NN}}=4\textrm{--}9\ \textrm{GeV}$. Finally, in [13], a Bayesian
analysis of flow observables utilizing hadronic transport simulations with
flexible vector mean-field potentials (tuned to reproduce chosen values of the
incompressibility $K_{0}$ and $c_{s}^{2}$ in given density ranges) constrains
$c_{s}^{2}$ within $0.47\pm 0.12$ for $n_{B}\in(2,3)n_{0}$ and within
$-0.08\pm 0.14$ for $n_{B}\in(3,4)n_{0}$, where $n_{0}\approx
0.16~{}\textrm{fm}^{-3}$ is the saturation density of nuclear matter.

In the presented work [14], we suggest another way of constraining $c_{s}^{2}$
based on heavy-ion collision data: by utilizing cumulants of the baryon number
distribution. Cumulants are one of the most prominent heavy-ion observables
due to their characteristic qualitative behavior in the vicinity of a critical
point [15, 16]. Below, we will use the fact that also the quantitative
behavior of the cumulants follows directly from the underlying EOS and, in
particular, may shed light on the isothermal speed of sound squared
$c_{T}^{2}$.

##  2 Connecting cumulants of baryon number and the isothermal speed of sound

Baryon number cumulants are defined as
$\kappa_{j}=VT^{j-1}\left(d^{j}P/d\mu_{B}^{j}\right)_{T}$, where $P$ is the
pressure and $V$ is the volume. In particular, we have

$\displaystyle\kappa_{1}=Vn_{B}~{},\hskip
22.76219pt\kappa_{2}=VT\left(\frac{d{n_{B}}}{d{\mu_{B}}}\right)_{T}~{}.$ (1)

Importantly, cumulants are related to moments of the baryon number
distribution, measurable in experiments; e.g., for $j\leq 3$,
$\kappa_{j}\equiv\Big{\langle}\Big{(}N_{B}-\big{\langle}N_{B}\big{\rangle}\Big{)}^{j}\Big{\rangle}$.

The isothermal speed of sound is defined as
$c_{T}^{2}\equiv\left(d{P}/d{\mathcal{E}}\right)_{T}$ (where $\mathcal{E}$ is
the energy density), and by using Eq. (1) and utilizing Maxwell’s relations,
it can be written as

$\displaystyle
c_{T}^{2}=\bigg{[}\frac{T}{\kappa_{1}}\left(\frac{d{\kappa_{1}}}{d{T}}\right)_{\mu_{B}}+\frac{\mu_{B}}{T}\frac{\kappa_{2}}{\kappa_{1}}\bigg{]}^{-1}~{},$
(2)

directly connecting it to the cumulants. However, estimating the first term in
the above equation from experimental data poses a significant challenge. At
the same time, the second term becomes dominant whenever $\mu_{B}/T\gg 1$, so
that for matter at high baryon densities we can write

$\displaystyle c_{T}^{2}\approx\frac{T\kappa_{1}}{\mu_{B}\kappa_{2}}~{}.$ (3)

In a similar fashion, one can connect cumulants with the logarithmic
derivative of $c_{T}^{2}$ with respect to $n_{B}$,

$\displaystyle\bigg{(}\frac{d\ln c_{T}^{2}}{d\ln
n_{B}}\bigg{)}_{T}+c_{T}^{2}=1-\frac{\kappa_{3}\kappa_{1}}{\kappa_{2}^{2}}-c_{T}^{2}\bigg{(}\frac{d\ln(\kappa_{2}/T)}{d\ln
T}\bigg{)}_{n_{B}}~{},$ (4)

and neglecting the last term on the right-hand side, which is valid for
$\mu_{B}/T\gg 1$, yields

$\displaystyle\left(\frac{d\ln c_{T}^{2}}{d\ln
n_{B}}\right)_{T}+c_{T}^{2}\approx
1-\frac{\kappa_{3}\kappa_{1}}{\kappa_{2}^{2}}~{}.$ (5)

Since formulas in Eqs. (3) and (5) are only approximate, it makes sense to ask
what is the region of validity of these equations. As discussed in [14], based
on comparisons of the exact and approximate expressions for $c_{T}^{2}$ and
its logarithmic derivative in two models of dense nuclear matter, one can
expect Eq. (3) to be valid for small to moderate temperatures, $T\lesssim
100~{}\textrm{MeV}$, and moderate to high baryon chemical potentials,
$\mu_{B}\gtrsim 600\ \textrm{MeV}$, while Eq. (5) is appropriate as long as
$\mu_{B}\gtrsim 200\ \textrm{MeV}$. We also note here that in the region of
validity of our approximations, the difference between $c_{T}^{2}$ and the
isentropic speed of sound squared $c_{\sigma}^{2}$, where the latter is more
often discussed in the context of heavy-ion collisions, is not substantial.

##  3 Experimental data and discussion

To apply our method to heavy-ion collisions, we use experimental data on the
cumulants of the net baryon number and chemical freeze-out parameters
$(T_{\textrm{fo}},\mu_{\textrm{fo}})$, determined by the solenoidal tracker at
RHIC (STAR) [17] and high acceptance dielectron spectrometer (HADES) [18, 19]
in collisions at 0-5% centrality and for rapidity $|y|<0.5$ ($|y|<0.4$ in the
case of HADES). 111We note that recently, the STAR collaboration has published
results on cumulants of proton number in collisions at
$\sqrt{s_{\textrm{NN}}}=3~{}\textrm{GeV}$ [20]. However, in this measurement a
non-symmetric rapidity window of $-0.5<y<0$ has been used, which does not
enable one to make a faithful comparison with the other STAR and HADES data;
therefore, we decided against including this data in the current analysis.
With these data, we plot the right-hand sides of Eqs. (3) and (5) in Fig. 1
(red triangles); we also show the corresponding results calculated for an
ideal gas (small gray circles) and results obtained within the vector density
functional (VDF) model [21], where the latter have been calculated at points
$(T,\mu_{B})$ in the phase diagram that allowed us to best reproduce the
experimental data. We note here that while the experimental results agree
reasonably well with the ideal gas predictions even for small values of
$\mu_{B}$, the fact that these points lie beyond the region of validity means
that they do not carry any information about $c_{T}^{2}$.

Figure 1: Comparison of the right-hand sides of Eqs. (3) (left panel) and (5)
(right panel) for experimental data (red triangles), the ideal gas at freeze-
out (small gray circles), and VDF model results (purple stars); teal lines
along the $\mu_{B}$-axes of the plots indicate the regions of validity of
approximations used in Eqs. (3) and (5).

Let us discuss these results. First, as seen in the left panel of Fig. 1, the
points obtained with the experimental data indicate a relatively small value
of the speed of sound, $c_{T}^{2}\approx 0.04$, at large $\mu_{B}$,
$\mu_{B}\approx 800~{}\textrm{MeV}$ (about half of the corresponding ideal gas
value, as shown in the insert). Second, as seen in the right panel of Fig. 1,
the experimental data indicate a very dramatic change in the behavior of the
speed of sound between moderate chemical potentials (shown by experimental
results from collisions at
$\sqrt{s_{\textrm{NN}}}=\big{\\{}14.5,~{}11.5,~{}7.7\big{\\}}~{}\textrm{GeV}$)
and high chemical potentials (shown by experimental results from collisions at
$\sqrt{s_{\textrm{NN}}}=2.4~{}\textrm{GeV}$): while at moderate $\mu_{B}$ the
logarithmic derivative of the speed of sound, $d\ln c_{T}^{2}/d\ln
n_{B}\approx 1-\kappa_{3}\kappa_{1}/\kappa_{2}^{2}$, is positive and
decreasing, indicating that $c_{T}^{2}$ maybe be plateauing as a function of
$\mu_{B}$, at high $\mu_{B}$, $\mu_{B}\approx 800~{}\textrm{MeV}$, we have
$1-\kappa_{3}\kappa_{1}/\kappa_{2}^{2}\approx 2.3$, indicating a very sharp
increase in $c_{T}^{2}$ with $n_{B}$ at this region of the phase diagram. As
already mentioned in the introduction, such a steep increase in the speed of
sound as a function of $n_{B}$ is consistent with neutron star studies [1, 2,
4, 5, 6, 7, 8, 9], as well as the recent constraint on the speed of sound at
$n_{B}\in(2,3)n_{0}$ from a Bayesian analysis of flow observables [13].
Interestingly, the matching of the experimental results done within the VDF
model (purple stars) suggests that the observed behavior of the cumulants of
baryon number and, consequently, the isothermal speed of sound may be
primarily due to the ordinary nuclear liquid-gas phase transition instead of
the hadron–QGP transition (see Fig. 3 in [14] and the associated text);
however, more extensive model studies are needed to either confirm or disprove
this assertion.

Several comments are in order. First, in our preliminary study we assumed that
the cumulants of the net proton number are a good proxy for the cumulants of
the net baryon number [22], which is likely too crude of an approximation
(see, e.g., [23]). Furthermore, we assumed that the momentum space cumulants,
measured in the experiment, can be used in place of the coordinate space
cumulants, Eq. (1); this is also likely incorrect. Moreover, we did not take
the influence of the rapidity bin width into account (which can be done by,
e.g., employing finite volume corrections [24]). Interestingly, however, a
recent, more in-depth analysis [25] of proton number cumulants measured at
$\sqrt{s_{\textrm{NN}}}=2.4~{}\textrm{GeV}$, taking into account some of the
issues mentioned above, obtains (using Eqs. (3) and (5)) $c_{T}^{2}\approx
0.0086\pm 0.0004$ and $d\ln c_{T}^{2}/d\ln n_{B}\approx 5.2\pm 0.8$,
indicating a behavior that qualitatively agrees with our analysis while being
even more striking.

Further studies, in particular utilizing hadronic transport simulations, are
planned to evaluate to what extent the presented method is applicable to
dynamical systems created in heavy-ion collisions and, applying appropriate
corrections, to infer the behavior of the speed of sound from cumulants of the
baryon number distribution.

## References

  * [1] P. Bedaque and A. W. Steiner, Phys. Rev. Lett. 114, no.3, 031103 (2015) doi:10.1103/PhysRevLett.114.031103 [arXiv:1408.5116 [nucl-th]].
  * [2] I. Tews, J. Carlson, S. Gandolfi and S. Reddy, Astrophys. J. 860, no.2, 149 (2018) doi:10.3847/1538-4357/aac267 [arXiv:1801.01923 [nucl-th]].
  * [3] L. McLerran and S. Reddy, Phys. Rev. Lett. 122, no.12, 122701 (2019) doi:10.1103/PhysRevLett.122.122701 [arXiv:1811.12503 [nucl-th]].
  * [4] Y. Fujimoto, K. Fukushima and K. Murase, Phys. Rev. D 101, no.5, 054016 (2020) doi:10.1103/PhysRevD.101.054016 [arXiv:1903.03400 [nucl-th]].
  * [5] E. Annala, T. Gorda, A. Kurkela, J. Nättilä and A. Vuorinen, Nature Phys. 16, no.9, 907-910 (2020) doi:10.1038/s41567-020-0914-9 [arXiv:1903.09121 [astro-ph.HE]].
  * [6] R. Essick, I. Tews, P. Landry, S. Reddy and D. E. Holz, Phys. Rev. C 102, no.5, 055803 (2020) doi:10.1103/PhysRevC.102.055803 [arXiv:2004.07744 [astro-ph.HE]].
  * [7] M. C. Miller, F. K. Lamb, A. J. Dittmann, S. Bogdanov, Z. Arzoumanian, K. C. Gendreau, S. Guillot, W. C. G. Ho, J. M. Lattimer and M. Loewenstein, et al. Astrophys. J. Lett. 918, no.2, L28 (2021) doi:10.3847/2041-8213/ac089b [arXiv:2105.06979 [astro-ph.HE]].
  * [8] T. Gorda, O. Komoltsev and A. Kurkela, [arXiv:2204.11877 [nucl-th]].
  * [9] M. Marczenko, L. McLerran, K. Redlich and C. Sasaki, [arXiv:2207.13059 [nucl-th]].
  * [10] F. G. Gardim, G. Giacalone, M. Luzum and J. Y. Ollitrault, Nature Phys. 16, no.6, 615-619 (2020) doi:10.1038/s41567-020-0846-4 [arXiv:1908.09728 [nucl-th]].
  * [11] S. Borsanyi, Z. Fodor, C. Hoelbling, S. D. Katz, S. Krieg and K. K. Szabo, Phys. Lett. B 730, 99-104 (2014) doi:10.1016/j.physletb.2014.01.007 [arXiv:1309.5258 [hep-lat]].
  * [12] J. Steinheimer and M. Bleicher, Eur. Phys. J. A 48, 100 (2012) doi:10.1140/epja/i2012-12100-0 [arXiv:1207.2792 [nucl-th]].
  * [13] D. Oliinychenko, A. Sorensen, V. Koch and L. McLerran, [arXiv:2208.11996 [nucl-th]].
  * [14] A. Sorensen, D. Oliinychenko, V. Koch and L. McLerran, Phys. Rev. Lett. 127, no.4, 042303 (2021) doi:10.1103/PhysRevLett.127.042303 [arXiv:2103.07365 [nucl-th]].
  * [15] M. Asakawa, S. Ejiri and M. Kitazawa, Phys. Rev. Lett. 103, 262301 (2009) doi:10.1103/PhysRevLett.103.262301 [arXiv:0904.2089 [nucl-th]].
  * [16] M. A. Stephanov, Phys. Rev. Lett. 107, 052301 (2011) doi:10.1103/PhysRevLett.107.052301 [arXiv:1104.1627 [hep-ph]].
  * [17] M. Abdallah et al. [STAR], Phys. Rev. C 104, no.2, 024902 (2021) doi:10.1103/PhysRevC.104.024902 [arXiv:2101.12413 [nucl-ex]].
  * [18] J. Adamczewski-Musch et al. [HADES], Phys. Rev. C 102, no.2, 024914 (2020) doi:10.1103/PhysRevC.102.024914 [arXiv:2002.08701 [nucl-ex]].
  * [19] M. Lorenz, https://indico.gsi.de/event/9423/contributions/40798/, 3rd EMMI Workshop (2019).
  * [20] M. S. Abdallah et al. [STAR], Phys. Rev. Lett. 128, no.20, 202303 (2022) doi:10.1103/PhysRevLett.128.202303 [arXiv:2112.00240 [nucl-ex]].
  * [21] A. Sorensen and V. Koch, Phys. Rev. C 104, no.3, 034904 (2021) doi:10.1103/PhysRevC.104.034904 [arXiv:2011.06635 [nucl-th]].
  * [22] Y. Hatta and M. A. Stephanov, Phys. Rev. Lett. 91, 102003 (2003) [erratum: Phys. Rev. Lett. 91, 129901 (2003)] doi:10.1103/PhysRevLett.91.102003 [arXiv:hep-ph/0302002 [hep-ph]].
  * [23] M. Kitazawa and M. Asakawa, Phys. Rev. C 85, 021901 (2012) doi:10.1103/PhysRevC.85.021901 [arXiv:1107.2755 [nucl-th]].
  * [24] V. Vovchenko, O. Savchuk, R. V. Poberezhnyuk, M. I. Gorenstein and V. Koch, Phys. Lett. B 811, 135868 (2020) doi:10.1016/j.physletb.2020.135868 [arXiv:2003.13905 [hep-ph]].
  * [25] V. Vovchenko and V. Koch, Phys. Lett. B 833, 137368 (2022) doi:10.1016/j.physletb.2022.137368 [arXiv:2204.00137 [hep-ph]].