satisfactory generalization to unseen data. The drift rates positively
correlate with choice-RTs, whereas the boundaries and non-decision time
negatively correlate with choice-RTs. The model successfully learns to extract
the neural features that account for the choice-RT variability at each trial.
To evaluate whether obtaining trial estimates of cognitive parameters improved
the model of choice and choice-RT data, Table 5 presents the Wiener likelihood
test for the neurocognitive generalization ability to unseen data. The results
show that the use of single-trial predictions of cognitive parameters
$\boldsymbol{\omega}_{i}$ provides higher likelihood than the median estimates
$\boldsymbol{\bar{\omega}}$ fitted from the training data. This implies that
single-trial estimates better account for new data compared to median
estimates.

Figure 13(a) shows the average of signals generated by the neurocognitive
autoencoder when given a set of approximately 800 test choice-RTs compared to
the average of actual signals associated with the same choice-RTs. At the
selected electrodes, the window of interest is 100 ms pre-stimulus to 500 ms
post-stimulus, which captures the N200 waveform. The generated and original
signals appear visually similar in the timing and amplitudes of the peaks and
troughs. Figures 13(b), 13(c), and 13(d) depict the trial-averaged frequency
spectra and corresponding ERP waveforms of the reconstructed signals.
Regarding the frequency spectra, the most important features are the 30 and 40
Hz peaks, which correspond to the flicker frequency of the signal (Gabor
patch) and spatial white noise, respectively. Interestingly, the generative
model learns to structure output the steady-state visually evoked potentials
(SSVEPs) that occur in response to a visual stimulus flickering at different
frequencies, even though it was never explicitly encoded in the model.
Moreover, in the low noise condition (b), the 30 Hz peak is large and the 40
Hz is small, while in the high noise condition (d), the 30 Hz peak is reduced
and the 40 Hz peak is enhanced. In terms of ERP waveforms, the model captures
the relationships of the N200 peak latencies with respect to the additive
noise conditions. Higher additive white noise in the stimulus effectively
increases the latency and decreases the amplitude of the N200. We focus on the
N200 signal because the original study (Nunez et al.,, 2019) found strong
relationships between N200 latency and choice-RT, and thus the N200 is a good
validation of our model. These prove the convergence of the model in
optimizing the lower bound of the conditional likelihood mapping from
behavioral data to EEG features, which effectively encodes differences in the
stimuli presented to the subjects in the latent variable space.

In addition to evaluating traditional ERP estimates (trial-averaged), we also
assess the single-trial ERP estimate (channel-averaged). To increase the
signal-to-noise ratio to better detect the N200, the first singular-value
decomposition (SVD) component obtained from the ERP response is taken as a
channel weighting function. More details of the SVD method can be seen at
(Nunez et al.,, 2019). Figure 15 shows the performance of the model in
learning the N200 feature in each trial. As shown in Figure 15, the
distributions of the single-trial N200 peak latencies, as well as the
amplitudes calculated from the generated signals, closely match those of the
original signals at three different noise levels. The peak amplitude
distribution is somewhat broader than the original data’s generated
distribution. Importantly, the model can generate the variability of the N200
latency with the experimental manipulation of low, medium, and high noise,
systematically increasing the N200 latency in the generated signals.

Figure 17 represents the sensitivity analysis of the choice-RT and drift-
diffusion parameters regardless of the noise conditions. In the left column,
we examine the sensitivity of the neural signals generated by the choice-RTs.
We can see similar patterns across subjects where the increases in choice-RTs
lead to significant declines in the 30 Hz and the rises of the N200 latencies.
This confirms the minimization approach of the KL divergence between the
latent spaces inferred from the behavioral data and the neural signals. Power
at 40 Hz reflecting the neural response to the noise also changes according to
the choice-RTs, though the pattern is not as strong as the subjects suppressed
the noise signal in all conditions.

One of the powerful tools for exploring the relationship between cognitive
processes is to examine the sensitivity of neural signals to cognitive
parameters. The middle and right columns of Figure 17 depict the effect of
_hypothetical_ modulations of drift rates and non-decision time on the
generated neural signals. The results show that our model reveals the
intricate interactions between cognitive parameters and neural signals, which
is consistent with prior discoveries in the cognitive modeling literature. As
the non-decision time is faster, the N200 latencies are shorter, and the 30 Hz
peaks are larger. Accordingly, the amplitudes of the N200 peaks are more
prominent, though not shown in the figures for clarity. The same interactions
are observed with the increase in drift rates, representing evidence
accumulation. Again, the effects on 40 Hz peaks are weaker and depend on the
subjects. We did not observe the effects of the boundary separation (caution)
on the neural signals. The effect can be reversed with slower non-decision
times and lower drift rates. The strongest effects can be seen when both
parameters influence neural signals. This demonstrates the effectiveness of
the designs of the hierarchical latent variables inferred from choice-RTs and
the disentangled latent space produced by the EEG data.

(a) Fitted from EEG (training data)

(b) Predicted from EEG (test data)

(c) Fitted from choice-RTs (training data)

(d) Predicted from choice-RTs (test data)

Figure 12: Drift-diffusion single-trial parameter estimations from correct
responses of subject s1. The parameters are constrained by the subject priors
resulting from a Bayesian MCMC modeling (without EEG data). Scatter plots
illustrate the relationship between the parameters and the observed choice-RTs
for each trial. The top two rows are posterior inferences from neural signals,
while the bottom two are from behaviors. The left column shows the drift-rate
($\delta$) estimates, the middle column shows boundary ($\alpha$) estimates,
and the right column presents non-decision time ($ndt$) estimates. The
correlations between the choice-RTs and the inferred DDM parameters are
consistent with what is expected. On top of each panel are the Spearman
correlation coefficients ($\rho$). The covariances of the inferred parameters
are indicated by circles, which correspond to contours having one standard
deviation. For clarity, each circle is magnified 300 times.

(a) EEG data at the selected electrodes

(b) Low noise condition

(c) Medium noise condition

(d) High noise condition

Figure 14: Performance of the model in reconstructing 98 EEG channels of
subject s1 by averaging $\approx$ 800 predicted EEG trials from $\approx$ 800
choice-RTs in the test set. Time point zero denotes the time point of stimulus
onset. The first row displays the original (blue) and generated (orange)
trial-averaged EEG data at the pooled electrodes. The x-axis denotes the time
in milliseconds from stimulus onset, and the y-axis denotes the signal
amplitude. The second, third, and fourth rows are (left) frequency spectra and
(right) EEG signals averaged over all test choice-RT trials ($\approx 800/3$
per condition). The signals on the right are low-pass filtered at 15 Hz for
clarity of N200 peaks. Each colored line corresponds to one reconstructed EEG
channel. In low-noise conditions, the spectra show a strong peak at the Gabor
flicker frequency of 30 Hz, and the ERP waveform shows a shorter N200 latency
and larger peak amplitude. Under high-noise conditions, the spectra show a
strong peak at the noise flicker frequency of 40 Hz, and the ERP waveform
shows a longer N200 latency and a smaller peak amplitude.

(a) Subject s1

(b) Subject s2

(c) Subject s3

(d) Subject s4

Figure 15: Performance of the model in reconstructing single-trial N200 peaks
from choice-RTs in four subjects. The dotted lines are references to the
original data. The distributions of (left) single-trial N200 peak latencies
across three noise conditions and (right) the N200 peak amplitude statistics
are shown. Single-trial observations of the peak latency of N200 are found
using the SVD method (Nunez et al.,, 2019) for each subject and noise
condition.

(a) Subject s1

(b) Subject s2

(c) Subject s3

(d) Subject s4

Figure 17: Sensitivity analysis of choice-RTs and latent drift-diffusion
parameters on EEG signal generation in four subjects. The left column presents
the effects of choice-RTs on the output neural signals. The blue bars
represent the power at 30 Hz, while the red bars represent the power at 40 Hz.
The orange bars show the N200 latencies. The middle column shows the changes
in the single-trial N200 distribution w.r.t to _hypothetical_ changes in the
cognitive parameters. The yellow distribution represents the reference data,
while the blue and red ones correspond to modified parameter settings that
decrease or increase the N200 latencies, respectively. The modification in
subject s4 (ndt $\pm$ 0.05, $\delta\pm$ 0.3) is different from other subjects.
The right column characterizes the changes in 30 Hz and 40 Hz peaks w.r.t to
the changes in the same cognitive parameters.

###  17 Conclusion

In this work, we proposed a joint behavioral and EEG modeling approach driven
by a cognitive model of decision making. The experimental results demonstrate
the effectiveness of our Neurocognitive VAE in simultaneously modeling high-
dimensional EEG signals and low-dimensional behavioral data. Remarkably, the
model learns essential task-relevant neural features, e.g. N200 peaks and
SSVEP, without explicit specification in the optimization objective.
Furthermore, the model captures how these features modulate behavior,
specifically discovering relationships between brain activity and behavior
consistent with other models based on prior knowledge. This suggests that the
Neurocognitive VAE helps uncover neural signals linked to behavioral data by
mapping to a structured latent space. Compared to the aforementioned published
joint models (Nunez et al.,, 2015, 2017, 2019; Lui et al.,, 2021; Turner et
al.,, 2013, 2016), our end-to-end model is capable of inferring task-relevant
EEG features from behavior without prior knowledge of which features to
optimize. The structured latent space allows the learning of behavioral
variability to drive the EEG data generation process, leading to the
prediction of the structure of EEG features in relation to the stimuli used in
the experiments (N200 and SSVEP) and the behavioral performance (choice-RT).
In addition, the model allows us to directly map the variability of cognitive
parameters to neural signals, allowing for theoretical predictions that guide
future experimental studies. It should be noted that our framework does not
serve to refine the functional form of process-oriented computational models.
Instead, it presumes a set of fixed assumptions; in the DDM, a constant drift
rate and boundary separation within trials. Importantly, our framework can be
generalized to encompass any other neural measures combined with any cognitive
model to explain behavior, provided that the cognitive model expresses a
closed-form likelihood of behavioral data. Importantly, by parameterizing the
likelihood by a deep neural network receiving neural data as input, trial-
level parameter inferences are made possible. In this research, we assume a
DDM posterior with a diagonal covariance matrix. This could lead to an
overestimation of the variance of the marginal posteriors if the true
posterior has dependencies. It would be beneficial to investigate the use of a
full covariance matrix as an alternative. It is important to mention that our
validation process focused on correct responses. Due to the low number of
incorrect responses compared to correct ones, we lack confidence in
interpreting the results in this study for the incorrect trials, although the
direction of the trial-level parameter fits was consistent with the results
for correct trials. We anticipate future research to explore strategies to
address the class imbalance problem in deep learning models (Johnson and
Khoshgoftaar,, 2019). Further work with a larger dataset is needed to
demonstrate that we can extend the model to new individuals. In principle,
this would potentially allow us to predict brain activity in clinical
populations with known behavioral differences.

### Data and Code Availability Statement

The dataset analyzed during the current study is available on
https://zenodo.org/record/8381751, and the implementation of the model is in
the following repository https://github.com/khuongav/neurocognitive_vae.

###  18 Supplementary Materials

####  18.1 Neural Network Architectures and Training Hyperparameters

The inferential and generative processes are parameterized by deep neural
networks, as shown by the flows in Figure 10. Table 6 details the
architectures of the five networks. The input EEG signals are of size 98 x 250
(1 second of data of 98 channels at 250 Hz). The feature extraction layers in
the EEG and cognitive encoders are similar to Vo et al., (2022). All the
feature maps have 128 channels. Leaky ReLU (lReLU) activation functions are
applied to all layers, with a slope of 0.1 to stimulate easier gradient flow.
Batch normalizations (BN) (Ioffe and Szegedy,, 2015) are used in each
convolutional layer of the encoders and decoders. Self-attention layers (Zhang
et al.,, 2019) are applied in the encoders and decoders to better account for
long-range relationships in time series. $\boldsymbol{c}$ are noise condition
embeddings as one-hot vectors (size 3). The size of $\boldsymbol{z}_{N}$ is
set at 32 as increasing the dimension did not lead to any improvement in
performance on a validation set.

In Equation 5.4, the term $\log p(\boldsymbol{y}\mid\boldsymbol{z}_{C})$ is
weighted by $\lambda=2$ to scale up the likelihood of low-dimensional
behavior. The KL terms are weighted by $\beta=20$. The KL terms are normalized
to balance the KL divergence loss and the reconstruction loss. Please refer to
Sections 4.2 and A6 of (Higgins et al.,, 2016) for further information. The
optimization of
$q_{\phi_{C}}\left(\boldsymbol{z}_{C}\mid\boldsymbol{x}\right)$ is divided
into two stages. We first optimize the network w.r.t drift rate $\delta$ and
boundary $\alpha$, while non-decision time $\tau$ is set to $0.93\cdot
RT_{min}$ for each subject, approximating the results of the Bayesian MCMC
modeling Nunez et al., (2019). Having trained $\phi_{C}$ for $\delta$ and
$\alpha$, we can proceed to train only the last fully connected layer that
predicts $\tau$. This procedure is to circumvent the difficulty of
simultaneously optimizing the network for the boundary and the non-decision
time on the experimental data. We used Adam (Kingma and Ba,, 2014) for
optimizations, with a learning rate of 5e-4 and exponential decay rates
$\beta_{1}$ = 0.9 and $\beta_{2}$ = 0.999.

Table 6: Neural network parametrization $\text{Encoder}_{\text{N}}-q_{\phi_{N}}(\boldsymbol{z}_{N}\mid\boldsymbol{x})$ |  | $\text{Encoder}_{\text{C}}-q_{\phi_{C}}(\boldsymbol{z}_{C}\mid\boldsymbol{x})$ | Decoder - $p_{\theta}(\boldsymbol{x}\mid\boldsymbol{z}_{N},\boldsymbol{z}_{C})$
---|---|---|---
maps EEG signals to neural latents |  | maps EEG signals to cognitive latents | reconstructs EEG signals
Dropout(0.3) |  |  |  Get $\boldsymbol{z}_{C}$
Conv 1, lReLU, 128 x 250 |  | Conv 1, lReLU, 128 x 250 | Linear 128, lReLU
Conv 6, BN, lReLU | X 2 | Conv 6, BN, lReLU, Dropout(0.7) | Linear 32, lReLU
Conv 6, Stride 2, BN, lReLU | Conv 6, Stride 2, BN, lReLU, Dropout(0.7) |  Concat $\boldsymbol{z}_{N}$, $\boldsymbol{c}$
Self Attention |  | Self Attention | Conv Transp 8, Stride 4, 512 Channels, BN, lReLU
Conv 6, BN, lReLU | X 2 | Conv 6, BN, lReLU, Dropout(0.7) | Conv Transp 8, Stride 4, 256 Channels, BN, lReLU
Conv 6, Stride 2, BN, lReLU | Conv 6, Stride 2, BN, lReLU, Dropout(0.7) | Self Attention
Reshape 2048, Concat $\boldsymbol{c}$ |  |  Reshape 2048, Concat $\boldsymbol{c}$ | Conv Transp 6, Stride 3, 128 Channels, BN, lReLU
Linear 32 (mean $\boldsymbol{z}_{N}$) |  |  Linear 1 (mean $\delta$), Linear 1 (logvar $\delta$) | Conv Transp 6, Stride 3, 128 Channels, BN, lReLU
Linear 32 (logvar $\boldsymbol{z}_{N}$) |  |  Linear 1, Softplus (mean $\alpha$) | Self Attention
|  |  Linear 1 (logvar $\alpha$) | Conv Transp 10, Stride 2, 98 Channels
|  |  Linear 1, Softplus (mean $ndt$) |
|  |  Linear 1 (logvar $ndt$) |
|  |  |
$\text{Encoder}^{2}_{\beta}-q^{2}_{\beta}\left(\boldsymbol{z}_{N}\mid\boldsymbol{z}_{C}\right)$ |  | $\text{Encoder}^{1}_{\beta}-q^{1}_{\beta}\left(\boldsymbol{z}_{C}\mid\boldsymbol{y}_{i}\right)$ |
maps cognitive latents to neural latents |  | maps behaviors to cognitive latents |
Linear 128, lReLU |  | Linear 128, lReLU |
Linear 128, lReLU |  | Linear 128, lReLU |
Concat $\boldsymbol{c}$ |  |  Concat $\boldsymbol{c}$ |
Linear 64 |  | Linear 6 |

####  18.2 Simulation Studies

We assessed our ability to recover true non-decision time (NDT) and drift rate
by simulating response time data and EEG signals. Response time data were
simulated from a drift-diffusion model with trial-to-trial variability in NDT
and evidence accumulation rate (i.e., drift rate). To simulate EEG signals
with a known relationship with DDM parameters, we specifically focused on N200
due to the significant associations between N200 latency and NDT reported by
Nunez et al., (2019). In our new experiments, we additionally observed a
substantial relationship between drift rate and N200 latency, which we
included in the simulation. Boundary separation was not included in the
simulation, as we did not find any neural correlates of variability in
boundary separation, and those are usually only found in tasks with trial-
level accuracy feedback (Cavanagh and Frank,, 2014; Nunez et al.,, 2024).

To simulate single-trial EEG signals, we shifted the true averaged ERP
waveform based on each sample of trial-level NDT, using a linear regression
slope of 1, as in Nunez et al., (2019). EEG noise was obtained from the
original data, using independently sampled segments that did not include
responses to stimuli. The resulting ERP and EEG waveforms were then combined
to generate artificial EEG signals for each trial that carried the N200
latency information and was associated with choice and response time.

It is evident from the results in Figure 18 that the model can accurately
recover the original distributions of trial-specific parameters. In
particular, the generating and recovered distributions strongly overlap, and
the correlation plots indicate that our single-trial estimates of cognitive
parameters exhibit good correlations with the reference parameters.

(a) Parameter distributions

(b) Parameter correlations

Figure 18: Drift-diffusion parameter estimates from neural signals in a
simulation of trial-level choice RTs and EEG signals. The top panels show the
overlap between the recovered and the original distributions of trial-specific
drift-rate and NDT. The reference values for the drift rate and NDT are drawn
from the normal distributions $\mathcal{N}(1.5,0.2)$ and
$\mathcal{N}(0.3,0.05)$, respectively. The bottom scatter plots illustrate the
relationship between the recovered parameters and the original parameters each
trial. $\rho$ are the Spearman correlation coefficients.

####  18.3 Experimental Tasks

Nunez et al., (2019) incorporated data from two experiments to test the
hypothesis that N200 peak-latencies track Visual Encoding Time (VET). Both
experiments required participants to determine whether a Gabor stimulus had
high or low spatial frequency content. The tasks took place in a dark room
with participants fixating on a small spot while responding to the stimuli
presented on a 61 cm LED monitor.

Figure 19: Example stimuli of the cue and response intervals of medium noise
conditions (Nunez et al.,, 2019). In the response phase, participants
identified the spatial-frequency target represented by each Gabor, using their
left hand to press a button for a target with a low spatial frequency (2.4
cpd) and their right hand for a target with a high spatial frequency (2.6
cpd). N200 waveforms were calculated time-locked to the onset of the Gabor
stimulus during the response intervals. The visual noise altered at a
frequency of 40 Hz, while the Gabor signal modulated at 30 Hz, inducing 40 Hz
and 30 Hz electrocortical responses that monitor attention to both noise and
signal.

In these experiments, Gabors were sinusoidal grating patterns with a Gaussian
falloff of contrast. The high and low spatial frequencies of the target Gabors
were 2.4 and 2.6 cycles per degree visual angle (cpd) respectively. The
experiments involved three conditions of visual noise contrast: high, medium,
and low. Visual noise was displayed both before and concurrently with the
Gabor targets at regular intervals. Example stimuli are given in Figure 19.
Participants used a button box to respond, pressing with the left hand for low
spatial frequency targets and the right hand for high spatial frequency
targets. They maintained fixation on a central spot while identifying the
spatial frequency of the Gabor stimuli embedded in noise.

EEG data was recorded using a 128-channel Geodesic sensor net. The visual
noise changed at 40 Hz, and the Gabor signal flickered at 30 Hz, evoking
specific electrocortical responses. The primary objective was to assess
whether N200 peak-latencies recorded by EEG reflected VET across varying
visual noise conditions, thereby shedding light on the timing involved in
perceptual decision-making processes.

####  18.4 Decision-Making Models - The Drift-Diffusion Model (DDMs)

The Drift-Diffusion Model (DDM) is a sequential sampling model of decision
making. The model assumes that decision-making is the result of the
accumulation of evidence in favor of one option or another. The evidence is
represented by a random walk process, where the evidence accumulates over
time, and the decision is made when the accumulated evidence exceeds a
threshold.

Figure 20: The DDM is illustrated in action during a two-choice task, with
non-decision time shown in green. Following the visual encoding period, the
decision variable (DV) begins evidence accumulation and reaches either the
upper or lower limit for each trial. The black vector depicts the average rate
of evidence accumulation. The blue curve depicts the distribution of response
times when choice 1 is successfully picked, while the red curve depicts the
distribution of reaction times when choice 2 is correctly selected. When the
DV drifts towards the incorrect boundary owing to random noise, incorrect
decisions are made. The distribution of reaction times for incorrect trials is
depicted by the dotted curve. EEG data for each trial, processed using
singular value decomposition to highlight N200, is shown on top that track the
start of evidence accumulation.

Mathematically, the DDM is described by a set of equations that govern the
accumulation of evidence and the decision-making process. The basic equation
for the DDM following a Wiener process (Figure 20) is

$dx=\delta dt+\varsigma dW$ (60)

where $dx$ is the evidence step, $dt$ is the time step and $\varsigma dW$
denotes a Gaussian noise with a scale $\varsigma$. The drift rate $\delta$ and
the diffusion coefficient $\varsigma$ are parameters that describe the average
rise in change over a unit interval during evidence accumulation and the
instantaneous variation in the rate of change, respectively. The variance can
be fixed at 1 for mathematical explicitness and simplicity. The distance
between two options is described as the boundary separation, or $\alpha$. The
beginning position of evidence accumulation, which shows a bias toward one of
the two options, is encoded by the parameter $\beta$. When $\beta$ is 0.5, the
beginning point is halfway between the two borders, and the evidence-building
process can begin unbiasedly between the two options. Visual encoding time
prior to evidence accumulation and motor execution time following evidence
accumulation could be written as $\tau_{e}$ and $\tau_{m}$, respectively. Only
the total of the two processes, non-decision time $\tau$, can be observed with
behavior alone.

The probability density function (pdf) of the Wiener diffusion model is
bivariate (with one dimension for the latency ($t$) and one for the binary
choice ($c$)); its analytical form can be approximated as

$\left\\{\begin{aligned} &\text{ Wiener }(t,c=0;\alpha,\beta,\tau,\delta)\\\
&=\frac{\pi}{\alpha^{2}}e^{-\frac{1}{2}\left(2\alpha\beta\delta+\delta^{2}(t-\tau)\right)}\times\sum_{k=1}^{+\infty}\left[k\sin(\pi
k\beta)e^{-\frac{1}{2}\frac{k^{2}\pi^{2}}{\alpha^{2}}(t-\tau)}\right]\\\
&\text{ Wiener }(t,c=1;\alpha,\beta,\tau,\delta)\\\ &=\text{ Wiener
}(t,c=0\mid\alpha,1-\beta,\tau,-\delta)\end{aligned}\right.$ (61)

Efficient methods for the computation of the Wiener diffusion model density
and distribution functions exist (Navarro and Fuss,, 2009), making it a highly
tractable model. In this work, $\beta$ is set at 0.5, so that the starting
point is always unbiased at $z$ = $\beta\alpha$. $\tau$ is set to be
$0.93\cdot RT_{min}$ for each subject, approximating Bayesian MCMC modeling
results (Nunez et al.,, 2019).

##  Chapter 6 Conclusions

###  19 Contributions

The growing accessibility of medical time series data is propelling the
advancement of mathematical models and techniques that can analyze it broadly
and efficiently. This dissertation is a move towards this goal, motivated by
the recent achievements in probabilistic modeling and deep learning.

In this dissertation, we have constructed integrated frameworks that combine
concepts from latent variable models, state-space models, and deep learning to
model multidimensional dependencies in physiological signals. These frameworks
are capable of modeling complex data distributions across various
applications. This is accomplished through the development of probabilistic
models that utilize deep neural networks to parameterize the underlying
conditional distributions. Deep learning architectures serve as powerful
function approximators, enabling the model to automatically extract features
essential for wide applicability. Recent developments in deep learning can be
seamlessly integrated into this framework.

The approaches proposed in this dissertation offer versatile frameworks for
representing and learning from diverse types of physiological measures. By
efficiently processing unlabeled datasets, these models enable the discovery
of hidden structures and patterns, facilitating data-driven hypothesis
generation:

  1. 1.

Deep State-Space Model for Heart Electrical Waveforms (Vo et al.,, 2023). This
application has significant potential for clinical diagnoses, especially since
it allows for heart disease assessment through wearable devices. The use of
optically obtained signals as inputs adds to the innovation, potentially
making diagnosis simpler and more accessible.

  2. 2.

Brain Signal Modeling with Probabilistic Graphical Models and Deep Adversarial
Learning (Vo et al.,, 2022). Combining these two approaches is promising for
encoding neural oscillations’ complexity while maintaining interpretability.
Moreover, applying these techniques to epilepsy seizure detection as an
unsupervised learning problem could lead to earlier and more accurate
diagnoses, improving patient outcomes.

  3. 3.

Joint Modeling of Physiological Measures and Behavior (Vo et al.,, 2024). This
approach shows potential in tackling the modern challenge in amalgamation of
diverse medical data sources. By analyzing the relationship between
physiological measures and behavior, our method could uncover new insights
into brain function and potentially revolutionize our understanding of
neurocognitive processes.

###  20 Future Work

Every progression in a field brings forth a set of unanswered queries, usually
more complex than the ones preceding them. Regarding the deep latent variable
models (DLVMs) presented in this thesis, several open questions exist, the
resolution of which could enhance our comprehension of their working
principles as well as to better exploit their modeling power.

  * •

Ensuring patient safety when implementing medical machine learning methods for
clinical applications necessitates robust models. A key characteristic of
robust models is their resilience to out-of-distribution (OOD) data, meaning
they can still provide accurate predictions when encountering data that differ
from the training set. Such OOD data might include samples from varied patient
demographics, different medical equipment and laboratory methods. Future
research should focus on assessing the adaptability of the DLVMs to OOD data
and enhancing model resilience against such data by leveraging established
strategies in the field (Zhou et al.,, 2022; Wang et al.,, 2022).

  * •

Probabilistic graphical models offer a principled approach to incorporating
prior knowledge and structured frameworks into the model, utilizing current
message-passing algorithms for approximate inference. A crucial yet difficult
task for broader adoption of DLVMs is developing a message-passing library
that seamlessly works with existing deep learning libraries (Johnson et al.,
2016b, ; Bendekgey et al.,, 2024).

  * •

Choosing the optimal model parameterization for a specific application can be
challenging. This category of models derives all the complexities associated
with defining the precise network architecture, such as the number of layers,
units, and activation functions, from its deep learning components. Therefore,
identifying a systematic method for hyperparameter optimization that is
effective across different applications is essential (Yu and Zhu,, 2020; He et
al.,, 2021).

  * •

Recent progress in physics-informed deep learning (Raissi et al.,, 2019;
Nabian and Meidani,, 2020) integrates the advantages of deep learning methods
with physical principles to improve both model efficacy and generalization. In
this approach, deep learning models are enhanced with a regularization term
that serves as prior knowledge, reflecting the fundamental laws and penalizing
deviations from these governing equations. Investigating DLVM approaches that
adhere to any specified law of electrophysics, as characterized by stochastic
differential equations, would be advantageous.

## References

  * Allen, (2007) Allen, J. (2007).  Photoplethysmography and its application in clinical physiological measurement.  Physiol Meas., 28(3):1–39.
  * Aschbacher et al., (2020) Aschbacher, K., Yilmaz, D., Kerem, Y., Crawford, S., Benaron, D., Liu, J., Eaton, M., Tison, G. H., Olgin, J. E., Li, Y., et al. (2020).  Atrial fibrillation detection from raw photoplethysmography waveforms: A deep learning application.  Heart rhythm O2, 1(1):3–9.
  * Aznan et al., (2019) Aznan, N. K. N., Atapour-Abarghouei, A., Bonner, S., Connolly, J. D., Al Moubayed, N., and Breckon, T. P. (2019).  Simulating brain signals: Creating synthetic eeg data via neural-based generative models for improved ssvep classification.  In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.
  * Bahdanau et al., (2014) Bahdanau, D., Cho, K., and Bengio, Y. (2014).  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473.
  * Banerjee et al., (2014) Banerjee, R., Sinha, A., Choudhury, A. D., and Visvanathan, A. (2014).  Photoecg: Photoplethysmographyto estimate ecg parameters.  In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4404–4408. IEEE.
  * Bau et al., (2019) Bau, D., Zhu, J.-Y., Wulff, J., Peebles, W., Strobelt, H., Zhou, B., and Torralba, A. (2019).  Seeing what a gan cannot generate.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 4502–4511.
  * Begoli et al., (2019) Begoli, E., Bhattacharya, T., and Kusnezov, D. (2019).  The need for uncertainty quantification in machine-assisted medical decision making.  Nature Machine Intelligence, 1(1):20–23.
  * Bendekgey et al., (2024) Bendekgey, H. C., Hope, G., and Sudderth, E. (2024).  Unbiased learning of deep generative models with structured discrete representations.  Advances in Neural Information Processing Systems, 36.
  * Bonneel et al., (2015) Bonneel, N., Rabin, J., Peyré, G., and Pfister, H. (2015).  Sliced and radon wasserstein barycenters of measures.  Journal of Mathematical Imaging and Vision, 51(1):22–45.
  * Cavanagh and Frank, (2014) Cavanagh, J. F. and Frank, M. J. (2014).  Frontal theta as a mechanism for cognitive control.  Trends in cognitive sciences, 18(8):414–421.
  * Chiu et al., (2020) Chiu, H.-Y., Shuai, H.-H., and Chao, P. C.-P. (2020).  Reconstructing qrs complex from ppg by transformed attentional neural networks.  IEEE Sensors Journal, 20(20):12374–12383.
  * Chung et al., (2014) Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014).  Empirical evaluation of gated recurrent neural networks on sequence modeling.  arXiv preprint arXiv:1412.3555.
  * Dempster et al., (1977) Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977).  Maximum likelihood from incomplete data via the em algorithm.  Journal of the royal statistical society: series B (methodological), 39(1):1–22.
  * Donahue et al., (2016) Donahue, J., Krähenbühl, P., and Darrell, T. (2016).  Adversarial feature learning.  arXiv preprint arXiv:1605.09782.
  * Dumoulin et al., (2016) Dumoulin, V., Belghazi, I., Poole, B., Mastropietro, O., Lamb, A., Arjovsky, M., and Courville, A. (2016).  Adversarially learned inference.  arXiv preprint arXiv:1606.00704.
  * Flamary et al., (2021) Flamary, R., Courty, N., Gramfort, A., Alaya, M. Z., Boisbunon, A., Chambon, S., Chapel, L., Corenflos, A., Fatras, K., Fournier, N., et al. (2021).  Pot: Python optimal transport.  Journal of Machine Learning Research, 22(78):1–8.
  * Forstmann and Wagenmakers, (2015) Forstmann, B. U. and Wagenmakers, E.-J., editors (2015).  An Introduction to Model-Based Cognitive Neuroscience.  Springer New York, New York, NY.
  * Glomb et al., (2020) Glomb, K., Cabral, J., Cattani, A., Mazzoni, A., Raj, A., and Franceschiello, B. (2020).  Computational models in electroencephalography.  arXiv preprint arXiv:2009.08385.
  * Goodfellow et al., (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014).  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680.
  * Hartmann et al., (2018) Hartmann, K. G., Schirrmeister, R. T., and Ball, T. (2018).  Eeg-gan: Generative adversarial networks for electroencephalograhic (eeg) brain signals.  arXiv preprint arXiv:1806.01875.
  * He et al., (2021) He, X., Zhao, K., and Chu, X. (2021).  Automl: A survey of the state-of-the-art.  Knowledge-based systems, 212:106622.
  * Heusel et al., (2017) Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017).  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30.
  * Higgins et al., (2016) Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2016).  beta-vae: Learning basic visual concepts with a constrained variational framework.  In International conference on learning representations.
  * Hoffman and Johnson, (2016) Hoffman, M. D. and Johnson, M. J. (2016).  Elbo surgery: yet another way to carve up the variational evidence lower bound.  In Workshop in Advances in Approximate Bayesian Inference, NIPS, volume 1.
  * Hong et al., (2019) Hong, S., Xiao, C., Ma, T., Li, H., and Sun, J. (2019).  Mina: multilevel knowledge-guided attention for modeling electrocardiography signals.  arXiv preprint arXiv:1905.11333.
  * Hong et al., (2020) Hong, S., Zhou, Y., Shang, J., Xiao, C., and Sun, J. (2020).  Opportunities and challenges of deep learning methods for electrocardiogram data: A systematic review.  Computers in biology and medicine, 122:103801.
  * Ioffe and Szegedy, (2015) Ioffe, S. and Szegedy, C. (2015).  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. pmlr.
  * Itthipuripat et al., (2019) Itthipuripat, S., Sprague, T. C., and Serences, J. T. (2019).  Functional MRI and EEG index complementary attentional modulations.  Journal of Neuroscience, 39(31):6162–6179.
  * Jabbar et al., (2020) Jabbar, A., Li, X., and Omar, B. (2020).  A survey on generative adversarial networks: Variants, applications, and training.  arXiv preprint arXiv:2006.05132.
  * Jang et al., (2016) Jang, E., Gu, S., and Poole, B. (2016).  Categorical reparameterization with gumbel-softmax.  arXiv preprint arXiv:1611.01144.
  * Jansen and Rit, (1995) Jansen, B. H. and Rit, V. G. (1995).  Electroencephalogram and visual evoked potential generation in a mathematical model of coupled cortical columns.  Biological cybernetics, 73(4):357–366.
  * Jansen et al., (1993) Jansen, B. H., Zouridakis, G., and Brandt, M. E. (1993).  A neurophysiologically-based mathematical model of flash visual evoked potentials.  Biological cybernetics, 68(3):275–283.
  * (33) Johnson, A. E., Pollard, T. J., Shen, L., Lehman, L.-w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., and Mark, R. G. (2016a).  Mimic-iii, a freely accessible critical care database.  Scientific data, 3(1):1–9.
  * Johnson and Khoshgoftaar, (2019) Johnson, J. M. and Khoshgoftaar, T. M. (2019).  Survey on deep learning with class imbalance.  Journal of Big Data, 6(1):1–54.
  * (35) Johnson, M. J., Duvenaud, D. K., Wiltschko, A., Adams, R. P., and Datta, S. R. (2016b).  Composing graphical models with neural networks for structured representations and fast inference.  Advances in neural information processing systems, 29.
  * Jordan et al., (1999) Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. (1999).  An introduction to variational methods for graphical models.  Machine learning, 37(2):183–233.
  * Kingma and Ba, (2014) Kingma, D. P. and Ba, J. (2014).  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * Kingma and Welling, (2013) Kingma, D. P. and Welling, M. (2013).  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114.
  * Koller and Friedman, (2009) Koller, D. and Friedman, N. (2009).  Probabilistic graphical models: principles and techniques.  MIT press.
  * Krishnan et al., (2017) Krishnan, R., Shalit, U., and Sontag, D. (2017).  Structured inference networks for nonlinear state space models.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
  * Krishnan et al., (2015) Krishnan, R. G., Shalit, U., and Sontag, D. (2015).  Deep kalman filters.  arXiv preprint arXiv:1511.05121.
  * Lee and Wagenmakers, (2014) Lee, M. D. and Wagenmakers, E.-J. (2014).  Bayesian cognitive modeling: A practical course.  Cambridge university press.
  * Lee et al., (2020) Lee, T.-Y., Vo, K., Baek, W., Khine, M., and Dutt, N. (2020).  Stint: selective transmission for low-energy physiological monitoring.  In Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design, pages 115–120.
  * Li et al., (2018) Li, C., Welling, M., Zhu, J., and Zhang, B. (2018).  Graphical generative adversarial networks.  Advances in neural information processing systems, 31.
  * Lu et al., (2009) Lu, G., Yang, F., Taylor, J., and Stein, J. F. (2009).  A comparison of photoplethysmography and ecg recording to analyse heart rate variability in healthy subjects.  Journal of medical engineering & technology, 33(8):634–641.
  * Lui et al., (2021) Lui, K. K., Nunez, M. D., Cassidy, J. M., Vandekerckhove, J., Cramer, S. C., and Srinivasan, R. (2021).  Timing of readiness potentials reflect a decision-making process in the human brain.  Computational Brain & Behavior, 4(3):264–283.
  * McSharry et al., (2003) McSharry, P. E., Clifford, G. D., Tarassenko, L., and Smith, L. A. (2003).  A dynamical model for generating synthetic electrocardiogram signals.  IEEE transactions on biomedical engineering, 50(3):289–294.
  * Minka, (2013) Minka, T. P. (2013).  Expectation propagation for approximate bayesian inference.  arXiv preprint arXiv:1301.2294.
  * Miyato et al., (2018) Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. (2018).  Spectral normalization for generative adversarial networks.  arXiv preprint arXiv:1802.05957.
  * Mohamed and Lakshminarayanan, (2016) Mohamed, S. and Lakshminarayanan, B. (2016).  Learning in implicit generative models.  arXiv preprint arXiv:1610.03483.
  * Moody et al., (2020) Moody, B., Moody, G., Villarroel, M., Clifford, G., and Silva, I. (2020).  Mimic-iii waveform database matched subset.  MIMIC-III Waveform Database Matched Subset v1. 0.
  * Nabian and Meidani, (2020) Nabian, M. A. and Meidani, H. (2020).  Physics-driven regularization of deep neural networks for enhanced engineering design and analysis.  Journal of Computing and Information Science in Engineering, 20(1).
  * Navarro and Fuss, (2009) Navarro, D. J. and Fuss, I. G. (2009).  Fast and accurate calculations for first-passage times in wiener diffusion models.  Journal of mathematical psychology, 53(4):222–230.
  * Niederer et al., (2019) Niederer, S. A., Lumens, J., and Trayanova, N. A. (2019).  Computational models in cardiology.  Nature Reviews Cardiology, 16(2):100–111.
  * Nunez et al., (2024) Nunez, M. D., Fernandez, K., Srinivasan, R., and Vandekerckhove, J. (2024).  A tutorial on fitting joint models of m/eeg and behavior to understand cognition.  PsyArXiv.
  * Nunez et al., (2019) Nunez, M. D., Gosai, A., Vandekerckhove, J., and Srinivasan, R. (2019).  The latency of a visual evoked potential tracks the onset of decision making.  Neuroimage, 197:93–108.
  * Nunez et al., (2015) Nunez, M. D., Srinivasan, R., and Vandekerckhove, J. (2015).  Individual differences in attention influence perceptual decision making.  Frontiers in psychology, 8:18.
  * Nunez et al., (2017) Nunez, M. D., Vandekerckhove, J., and Srinivasan, R. (2017).  How attention influences perceptual decision making: Single-trial EEG correlates of drift-diffusion model parameters.  Journal of mathematical psychology, 76:117–130.
  * Nunez and Srinivasan, (2006) Nunez, P. L. and Srinivasan, R. (2006).  Electric fields of the brain: the neurophysics of EEG.  Oxford University Press, USA.
  * Olier et al., (2021) Olier, I., Ortega-Martorell, S., Pieroni, M., and Lip, G. Y. (2021).  How machine learning is impacting research in atrial fibrillation: implications for risk prediction and future management.  Cardiovascular Research, 117(7):1700–1717.
  * Pascual et al., (2019) Pascual, D., Aminifar, A., Atienza, D., Ryvlin, P., and Wattenhofer, R. (2019).  Synthetic epileptic brain activities using gans.  Machine Learning for Health (ML4H) at NeurIPS.
  * Raissi et al., (2019) Raissi, M., Perdikaris, P., and Karniadakis, G. E. (2019).  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational Physics, 378:686–707.
  * Ratcliff and McKoon, (2008) Ratcliff, R. and McKoon, G. (2008).  The diffusion decision model: theory and data for two-choice decision tasks.  Neural computation, 20(4):873–922.
  * Reisner et al., (2008) Reisner, A., Shaltis, P. A., McCombie, D., and Asada, H. H. (2008).  Utility of the photoplethysmogram in circulatory monitoring.  Anesthesiology: The Journal of the American Society of Anesthesiologists, 108(5):950–958.
  * Rosiek and Leksowski, (2016) Rosiek, A. and Leksowski, K. (2016).  The risk factors and prevention of cardiovascular disease: the importance of electrocardiogram in the diagnosis and treatment of acute coronary syndrome.  Therapeutics and clinical risk management, 12:1223.
  * Roy et al., (2019) Roy, Y., Banville, H., Albuquerque, I., Gramfort, A., Falk, T. H., and Faubert, J. (2019).  Deep learning-based electroencephalography analysis: a systematic review.  Journal of neural engineering, 16(5):051001.
  * Sarkar and Etemad, (2021) Sarkar, P. and Etemad, A. (2021).  Cardiogan: Attentive generative adversarial network with dual discriminators for synthesis of ecg from ppg.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 488–496.
  * Shoeb, (2009) Shoeb, A. H. (2009).  Application of machine learning to epileptic seizure onset detection and treatment.  PhD thesis, Massachusetts Institute of Technology.
  * Siddiqui et al., (2020) Siddiqui, M. K., Morales-Menendez, R., Huang, X., and Hussain, N. (2020).  A review of epileptic seizure detection using machine learning classifiers.  Brain informatics, 7:1–18.
  * Sohn et al., (2015) Sohn, K., Lee, H., and Yan, X. (2015).  Learning structured output representation using deep conditional generative models.  Advances in neural information processing systems, 28.
  * sudden, (2020) sudden (Retrieved on September 2020).  Sudden death in young people: Heart problems often blamed.  https://www.mayoclinic.org/diseases-conditions/sudden-cardiac-arrest/in-depth/sudden-death/art-20047571.
  * Sun et al., (2022) Sun, Q. J., Vo, K., Lui, K., Nunez, M., Vandekerckhove, J., and Srinivasan, R. (2022).  Decision sincnet: Neurocognitive models of decision making that predict cognitive processes from neural signals.  In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–9. IEEE.
  * Suzuki et al., (2016) Suzuki, M., Nakayama, K., and Matsuo, Y. (2016).  Joint multimodal learning with deep generative models.  arXiv preprint arXiv:1611.01891.
  * Tang et al., (2022) Tang, Q., Chen, Z., Guo, Y., Liang, Y., Ward, R., Menon, C., and Elgendi, M. (2022).  Robust reconstruction of electrocardiogram using photoplethysmography: A subject-based model.  Frontiers in Physiology, page 645.
  * Tian et al., (2020) Tian, X., Zhu, Q., Li, Y., and Wu, M. (2020).  Cross-domain joint dictionary learning for ecg reconstruction from ppg.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 936–940. IEEE.
  * Tran et al., (2023) Tran, B.-H., Shahbaba, B., Mandt, S., and Filippone, M. (2023).  Fully bayesian autoencoders with latent sparse gaussian processes.  In International Conference on Machine Learning, pages 34409–34430. PMLR.
  * Turner et al., (2013) Turner, B. M., Forstmann, B. U., Wagenmakers, E.-J., Brown, S. D., Sederberg, P. B., and Steyvers, M. (2013).  A bayesian framework for simultaneously modeling neural and behavioral data.  NeuroImage, 72:193–206.
  * Turner et al., (2016) Turner, B. M., Rodriguez, C. A., Norcia, T. M., McClure, S. M., and Steyvers, M. (2016).  Why more is better: Simultaneous modeling of EEG, fMRI, and behavioral data.  NeuroImage, 128:96–115.
  * van Gent et al., (2019) van Gent, P., Farah, H., van Nes, N., and van Arem, B. (2019).  Analysing noisy driver physiology real-time using off-the-shelf sensors: Heart rate analysis software from the taking the fast lane project.  Journal of Open Research Software, 7(1).
  * Van Gent et al., (2019) Van Gent, P., Farah, H., Van Nes, N., and Van Arem, B. (2019).  Heartpy: A novel heart rate algorithm for the analysis of noisy signals.  Transportation research part F: traffic psychology and behaviour, 66:368–378.
  * Vedantam et al., (2017) Vedantam, R., Fischer, I., Huang, J., and Murphy, K. (2017).  Generative models of visually grounded imagination.  arXiv preprint arXiv:1705.10762.
  * Vo et al., (2023) Vo, K., El-Khamy, M., and Choi, Y. (2023).  Ppg to ecg signal translation for continuous atrial fibrillation detection via attention-based deep state-space modeling.  arXiv preprint arXiv:2309.15375.
  * Vo et al., (2021) Vo, K., Naeini, E. K., Naderi, A., Jilani, D., Rahmani, A. M., Dutt, N., and Cao, H. (2021).  P2e-wgan: Ecg waveform synthesis from ppg with conditional wasserstein generative adversarial networks.  In Proceedings of the 36th Annual ACM Symposium on Applied Computing, pages 1030–1036.
  * Vo et al., (2024) Vo, K., Sun, Q. J., Nunez, M. D., Vandekerckhove, J., and Srinivasan, R. (2024).  Deep latent variable joint cognitive modeling of neural signals and human behavior.  NeuroImage, page 120559.
  * Vo et al., (2022) Vo, K., Vishwanath, M., Srinivasan, R., Dutt, N., and Cao, H. (2022).  Composing graphical models with generative adversarial networks for EEG signal modeling.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1231–1235. IEEE.
  * Wainwright et al., (2008) Wainwright, M. J., Jordan, M. I., et al. (2008).  Graphical models, exponential families, and variational inference.  Foundations and Trends® in Machine Learning, 1(1–2):1–305.
  * Wang et al., (2022) Wang, J., Lan, C., Liu, C., Ouyang, Y., Qin, T., Lu, W., Chen, Y., Zeng, W., and Philip, S. Y. (2022).  Generalizing to unseen domains: A survey on domain generalization.  IEEE transactions on knowledge and data engineering, 35(8):8052–8072.
  * Wu et al., (2015) Wu, W., Nagarajan, S., and Chen, Z. (2015).  Bayesian machine learning: Eeg$\backslash$/meg signal processing measurements.  IEEE Signal Processing Magazine, 33(1):14–36.
  * Yu and Zhu, (2020) Yu, T. and Zhu, H. (2020).  Hyper-parameter optimization: A review of algorithms and applications.  arXiv preprint arXiv:2003.05689.
  * Zhang et al., (2019) Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A. (2019).  Self-attention generative adversarial networks.  In International conference on machine learning, pages 7354–7363. PMLR.
  * Zhou et al., (2022) Zhou, K., Liu, Z., Qiao, Y., Xiang, T., and Loy, C. C. (2022).  Domain generalization: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4396–4415.
  * Zhu et al., (2019) Zhu, Q., Tian, X., Wong, C.-W., and Wu, M. (2019).  Ecg reconstruction via ppg: A pilot study.  In 2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), pages 1–4. IEEE.


#  AA-DRL: AoI-Aware Deep Reinforcement Learning Approach for D2D-Assisted
Industrial IoT

Hossam Farag1, Mohamed Ragab2, and Čedomir Stefanović1 1 Department of
Electronic Systems, Aalborg University, Denmark
2 Centre for Frontier AI Research, Agency for Science Technology and Research
(A*STAR), Singapore
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In real-time Industrial Internet of Things (IIoT), e.g., monitoring and
control scenarios, the freshness of data is crucial to maintain the system
functionality and stability. In this paper, we propose an AoI-Aware Deep
Reinforcement Learning (AA-DRL) approach to minimize the Peak Age of
Information (PAoI) in D2D-assisted IIoT networks. Particularly, we analyzed
the success probability and the average PAoI via stochastic geometry, and
formulate an optimization problem with the objective to find the optimal
scheduling policy that minimizes PAoI. In order to solve the non-convex
scheduling problem, we develop a Neural Network (NN) structure that exploits
the Geographic Location Information (GLI) along with feedback stages to
perform unsupervised learning over randomly deployed networks. Our motivation
is based on the observation that in various transmission contexts, the
wireless channel intensity is mainly influenced by distance-dependant path
loss, which could be calculated using the GLI of each link. The performance of
the AA-DRL method is evaluated via numerical results that demonstrate the
effectiveness of our proposed method to improve the PAoI performance compared
to a recent benchmark while maintains lower complexity against the
conventional iterative optimization method.

###### Index Terms:

Industrial IoT, neural networks, age of information

##  I Introduction

The Internet of Things (IoT) technology evolves rapidly as a worldwide network
of interconnected intelligent devices that are capable of sensing,
communicating, and processing to support a variety of applications, such as
industrial monitoring, health monitoring and vehicular networks [1]. Different
from consumer IoT, Industrial IoT (IIoT) networks are characterized by strict
communication requirements to maintain production efficiency and avoid safety-
critical situation [2]. IIoT networks are evolving from the typical star
network configuration to Device-to-Device (D2D) communications where sensor-
actuator pairs in propinquity communicate directly without evolving a central
node (e.g., access point or a Base Station (BS) [3]. In outband D2D
communication [3], the D2D pairs operate in full frequency reuse model where
the they communication over the same frequency band in uncoordinated fashion.
For IIoT scenarios that comprise enormous D2D pairs, this incurs significant
interference when D2D links are activated at the same time. Such uncoordinated
access would cause noticeable degradation in the network performance in terms
of delay, throughput and most importantly the information freshness.
Information freshness is crucial for a typical real-time control and
monitoring scenario as it affects the derived intelligent and autonomous
decisions and the system stability as well. For instance, in oil refineries,
valve actuators should acquire timely monitoring of oil level to avoid oil
tank spillage [4]. Information freshness is quantified by Age-of-Information
(AoI) [5], a process-level metric which, from the receiver perspective, counts
the time elapsed since the latest received information was generated.
Efficient D2D links scheduling has been proved to effectively enhance various
performance metrics [3]. However, this approach is not necessarily effective
to improve and optimize AoI. In addition, scheduling problems for wireless
networks with complex interference are usually non-convex and NP-hard [6]. The
conventional method for link scheduling first involves estimating the
interference of channels and then optimizing the schedule using these
estimates [7]. Estimating Channel State Information (CSI) in densely deployed
networks can be costly, and even achieving a near-optimal solution to the
resulting optimization problem can be intricate. For instance, in network
consisting of $N$ D2D links, $N^{2}$ CSI are required in the path-loss matrix
within each coherence block, which corresponds to a computational complexity
of at least $O(N^{2})$. To that end, machine learning and artificial
intelligence techniques are employed to find optimal schedules in dense
networks [8].

In this work, we develop an AoI-Aware Deep Reinforcement Learning (AA-DRL)
approach to minimize Peak AoI (PAoI) in dense D2D-assisted IIoT networks. The
learning approach is based on the collected Geographic Location Information
(GLI) to select the optimal scheduling policy. Our motivation is based on the
fact that in various transmission contexts, the wireless channel intensity is
mainly influenced by distance-dependant path loss. Moreover, the pattern of
interference within a network largely depends on how transmitters and
receivers are positioned relative to one another. We derive the successful
transmission probability and the average PAoI considering a preemptive queuing
policy and spatial coupled interference between the D2D pairs. Then, we
formulate the PAoI optimization problem under stationary randomized policy. We
solve the non-convex scheduling problem via a proposed Neural Network (NN)
structure where the scheduling policy is mapped to the GLI while explicit CSI
is not required. The NN is trained via an unsupervised training process by
utilizing the GLI along with feedback stages to obtain the optimal scheduling
policy. Our obtained numerical results show that the proposed AA-DRL approach
achieves improved PAoI performance compared to a recent benchmark [9] while
maintains lower complexity against the conventional iterative optimization
method.

The remainder of the paper is organized as follows. Related work is presented
in Section II. Section III describes the system model. In Section IV, we
introduce our proposed DRL algorithm and the NN structure. Performance
evaluations and conclusion are presented in Sections V and Section VI,
respectively.

##  II Related Work

Several research works have been conducted with the goal of minimizing AoI in
wireless networks. The authors in [10] studied the average AoI under different
queue management schemes which concluded that packet replacement can promote
reduced AoI compared to the conventional First Come First Served (FCFS)
approach. The authors in [11, 12, 13] introduced centralized scheduling
methods to minimize the average AoI. However, the centralized scheduling
approach is inefficient in D2D-enabled IIoT networks as it would incur high
overhead and extended delay. Efficient scheduling in D2D networks was tackled
by different works that focus on the analysis and optimization of resource
allocation [14], traffic density [15], or user fairness [16], while less
attention was paid to the minimization of AoI. Moreover, other works apply
stochastic geometry to model the spatial relationship of D2D devices, and
adopting AoI-aware decentralized scheduling with the assumption of having full
CSI [17]. However, it is shown that it is very challenging to obtain a global
CSI in D2D networks [18]. A backlog-aware protocol was presented in [19] to
minimize the average AoI subject to a delay constraint. The work in [9]
presented a locally-adaptive slotted-ALOHA protocol where a link-wise access
probability is dynamically selected to minimize the AoI considering a unit-
size buffers and Last Come First Served (LCFS) queuing discipline. Both [19]
and [9] require the exchange of queue-status information between neighboring
nodes to find the optimal access probability, which implies significant
overhead and complexity in dense IIoT networks. Different from these works,
our proposed AA-DRL has the potential to optimize the AoI performance of IIoT
networks while maintaining low complexity with no explicit need for CSI.

Figure 1: The hierarchy of the considered IIoT network.

##  III System Model

We consider an IIoT network with a three-layer hierarchy as depicted in Fig.
1. The first layer is the field network that consists of $N$ D2D pairs
transmitting status updates via a shared time-slotted channel, where each
transmission attempt fits within the duration of one time slot. The second
layer includes edge computing nodes with storage, computing, communication,
and other resources that are used for computing-intensive tasks. The third
layer is the industrial cloud where historical data about the field and edge
nodes is stored in the cloud for long-term data analysis. The considered model
is common in most remote data acquisition and distributed control
applications, and is shown to have promising advantages in supporting
efficient resource management [20]. In the field network layer, each
transmitter generates status update following a Poisson process with average
rate $\lambda$ packets/slot. Such arrival model captures the scenario where
such traffic can be triggered by the occurrence of some random incident [21].
The D2D transmitters have a single-occupancy backlog, i.e., the output buffer
can accommodate only one packet. Moreover, we consider a preemptive queuing
policy where an arriving packet can preempt the one currently in the service
(if there is such). We consider a stationary randomized scheduling policy
[22], in which the D2D transmitters are activated to transmit in each time
slot with a given slot-access probability that is subject to optimization. Let
$\pi=\\{p_{1},p_{2},...,p_{N}\\}$ denotes the the randomized scheduling policy
and $\Pi$ is the class that represents all the possible policies where
$\pi\in\Pi$. Particularly, for a given scheduling policy $\pi$, the D2D
transmitter $n_{i}$, $i\in\\{1,2,...,N\\}$, is activated with a probability
$p_{i}$ across all the time slots.

###  III-A Successful Update Probability

The collisions among simultaneously active D2D pairs are not necessarily
destructive, due to the capture effect [23]. In that sense, a packet is
decoded successfully at the end of a time slot when the Signal-to-Interference
plus-Noise ratio (SINR) at the corresponding receiver exceeds the capture
ratio $\beta$. The SINR at the AP in a time slot $t$ given a set $\mathcal{V}$
of interfering links can be written as [17]

${\text{SINR}_{i}(t)=\frac{P_{t}|h_{i,i}|^{2}d_{i,i}^{-\alpha}}{\sigma^{2}+\sum_{j\in\mathcal{V}\backslash\\{i\\}}P_{t}|h_{j,i}|^{2}d_{j,i}^{-\alpha}}},$
(1)

where $P_{t}$ is the transmission power (assumed fixed for all D2D pairs),
$h_{j,i}$ is the random variable that represents the Rayleigh fading of the
channel between transmitter of link $j$ and the receiver of link $i$ with
$h_{j,i}\sim\mathrm{exp}(1)$, $d_{j,i}$ is the distance between between
transmitter of link $j$ and the receiver of link $i$, $\sigma^{2}$ denotes the
power of AWGN, and $\alpha$ is the path loss exponent. According to the
considered channel model, for an arbitrary node $n_{i}$, the conditional
successful decoding probability in timeslot $t$ given a certain scheduling
policy $\pi$ can be obtained as

$\begin{split}\phi_{i}(t|\mathcal{\pi})&=\mathrm{Pr(SINR}_{i}>\beta\,|\pi)\\\
&=\mathrm{Pr}\left(\frac{P_{t}|h_{i,i}|^{2}d_{i}^{-\alpha}}{\sigma^{2}+\sum_{j\in\mathcal{V}\backslash\\{i\\}}P_{t}|h_{j,i}|^{2}d_{j}^{-\alpha}a_{j}(t)}>\beta|\pi\right)\\\
&=\mathrm{exp}\left(-\frac{\beta\sigma^{2}}{P_{t}d_{i}^{-\alpha}}\right)\prod_{j\in\mathcal{V}\backslash\\{i\\}}\left(\frac{1}{1+a_{j}(t)\frac{\beta
d_{j}^{-\alpha}}{d_{i}^{-\alpha}}}\right).\end{split}$ (2)

Using (2) and considering the adopted randomized scheduling policy, the
average successful probability $\overline{\phi_{i}(t)}$ is given as

$\overline{\phi_{i}(t)}=p_{i}\mathrm{exp}\left(-\frac{\beta\sigma^{2}}{P_{t}d_{i}^{-\alpha}}\right)\prod_{j\in\mathcal{V}\backslash\\{i\\}}\left(1-\frac{p_{j}}{1+\frac{d_{i}^{-\alpha}}{\beta
d_{j}^{-\alpha}}}\right).$ (3)

###  III-B Analysis of the Peak AoI

Figure 2: Evaluation of the AoI of an arbitrary node $n_{i}$ with $D=6$.

The evaluation of the PAoI is statistically identical for all the D2D pairs,
hence, in the following we focus on the derivation of the PAoI of an arbitrary
D2D pair. Fig. 2 shows an example evolution of the AoI of the considered
preemptive queuing scheme. Let $t_{j}$, $j=1,2,3,....,$ the generation time of
the $j$th update. We denote $X_{j}$ as the random variable that represents the
interarrival time between consecutive updates, $X_{j}=t_{j+1}-t_{j}$, which
follows an exponentially distribution with mean $1/\lambda$. Note that an
update may not be received correctly by the D2D receiver due to transmission
failures or preemption. Hereafter, we use a different index $i\leq j$ to refer
to the successfully received updates. Let $d_{i}$ refers to the departure time
of $i$th update that is successfully received by the corresponding D2D
receiver, and $S_{i}$ is its corresponding service time. We denote $g_{i}$ as
the generation time of the first generated update after $d_{i-1}$, and is
given as

$g_{i}\triangleq\mathrm{min}\\{t_{j}\mid t_{j}>d_{i-1}\\}.$ (4)

Therefore, we can see that the indices $i\,\textrm{and}\,j$ in general do not
refer to the same update. For instance, in Fig. 2, the generated update at
$t_{3}$ is not received, and the successfully received update at $d_{2}$ is
the one generated at $t_{4}$. We define $W_{i}=g_{i}-d_{i-1}$ as the interval
between the reception of the $(i-1)$th update until the generation of the next
update. We define the interval $T_{i}=d_{i}-g_{i}$, which represents the
interval from $g_{i}$ until the next update received successfully. Note that
$T_{i}$ spans the generation instants of failed updates. We also define the
interdeparture time between two consecutive successfully received status
updates $Y_{i}=d_{i}-d_{i-1}$. From the definition of $W_{i}$ and $T_{i}$, we
have $Y_{i}=T_{i}+W_{i}$. Therefore, the PAoI , denoted as $A_{P_{i}}$ (the
value of AoI just before receiving the update at $d_{i}$), can be given as

$A_{P_{i}}=Y_{i}+S_{i-1},$ (5)

where $S_{i-1}$ represents the service time of the update received before
$d_{i}$. For instance, in Fig. 2, the PAoI at $d_{2}$ is equal to
$Y_{2}+S_{1}$, where $S_{1}$ is the service time of the update received before
$d_{2}$, which is $d_{1}$. The average PAoI
($\operatorname{\mathbb{E}}[A_{P_{i}}]$) is given by

$\operatorname{\mathbb{E}}[A_{P_{i}}]=\operatorname{\mathbb{E}}[Y]+\operatorname{\mathbb{E}}[S]=\operatorname{\mathbb{E}}[T]+\operatorname{\mathbb{E}}[W]+\operatorname{\mathbb{E}}[S],$
(6)

where
$\operatorname{\mathbb{E}}[Y]=\operatorname{\mathbb{E}}[T]+\operatorname{\mathbb{E}}[W]$.
We denote $\gamma$ as the probability that an update is preempted. The value
of $\gamma$ is given as a function of the arrival intensity and the service
time as follows

$\gamma=1-e^{(-\mu\lambda)},$ (7)

where $\mu$ is the duration of one time slot, which represents the
deterministic service time $S$ of an update.

For the considered queuing system, when a packet departs, it leaves the system
empty, hence, $W$ will follow the same distribution as the interarrival time,
i.e., $\operatorname{\mathbb{E}}[W]=1/\lambda$. Moreover, we have
$\operatorname{\mathbb{E}}[S]=\mu$. Therefore, the term
$\operatorname{\mathbb{E}}[T]$ can be evaluated using the following recursive
method [24] as

$\begin{split}\operatorname{\mathbb{E}}[T]=&\underbrace{(1-\gamma)(1-\alpha)\operatorname{\mathbb{E}}[S]}_{R_{1}}\\\
&+\underbrace{(1-\gamma)\alpha\left(\operatorname{\mathbb{E}}[S]+\operatorname{\mathbb{E}}[W]+\operatorname{\mathbb{E}}[\hat{T}]\right)}_{R_{2}}\\\
&+\underbrace{\gamma(\operatorname{\mathbb{E}}[X\mid
X<S]+\operatorname{\mathbb{E}}[\hat{T}])}_{R_{3}},\end{split}$ (8)

where $\alpha=1-\overline{\phi_{i}(t)}$ represents the failed transmission
probability at time slot $t$. The first term $R_{1}$ in (8) denotes the case
when the first update (generated at $g_{i}$) is not preempted ($1-\gamma$) by
other updates and is received successfully. The term $R_{2}$ refers to the
case when the first update is not preempted, but the transmission fails. In
this case, the system spends the service time $S$ for the first update, then
waits for the period $W$ until the next update is generated. The evaluation of
$\hat{T}$ is the same as $T$, hence
$\operatorname{\mathbb{E}}[\hat{T}]=\operatorname{\mathbb{E}}[T]$. The term
$R_{3}$ represents the case that the first generated update is preempted by a
new update. In that case, the effective generation interval, i.e., the
generation interval given that a packet, is preempted can be given as

$\operatorname{\mathbb{E}}[X\mid X<S]=\frac{\int_{0}^{\mu}s\lambda
e^{-s\lambda}\,ds}{1-e^{(-\mu\lambda)}}=\frac{1}{\lambda}+\mu\left(1-\frac{1}{\gamma}\right).$
(9)

Using (9) in (8) and substituting $\operatorname{\mathbb{E}}[S]$ and
$\operatorname{\mathbb{E}}[W]$, $\operatorname{\mathbb{E}}[T]$ can be obtained
as

$\operatorname{\mathbb{E}}[T]=\frac{\gamma+\alpha-\gamma\alpha}{\lambda(1-\beta)(1-\alpha)}.$
(10)

Then, $\operatorname{\mathbb{E}}[Y]$ becomes

$\begin{split}\operatorname{\mathbb{E}}[Y]&=\operatorname{\mathbb{E}}[T]+\operatorname{\mathbb{E}}[W]\\\
&=\frac{\gamma+\alpha-\gamma\alpha}{\lambda(1-\gamma)(1-\alpha)}+\frac{1}{\lambda}=\frac{1}{\lambda(1-\gamma)(1-\alpha)}.\end{split}$
(11)

Based on (6) and (11), we obtain the average PAoI for the PR scheme
$\operatorname{\mathbb{E}}[A_{P_{i}}]$ as

$\operatorname{\mathbb{E}}[A_{P_{i}}]=\frac{1}{\lambda(1-\gamma)(1-\alpha)}+\mu.$
(12)

For a given $\lambda$ and $\mu$, $\operatorname{\mathbb{E}}[A_{P_{i}}]$ is
mainly influenced by $\overline{\phi_{i}(t)}$, which depends on $p_{i}$. Our
goal in this work is to find the optimal scheduling policy $\pi$ that
minimizes the average PAoI $\operatorname{\mathbb{E}}[A_{P_{i}}]$. Therefore,
our optimization problem can be formulated as follows

$\begin{split}&\underset{\pi\in\Pi}{\mathrm{min}}\,\,\,\operatorname{\mathbb{E}}[A_{P_{i}}]\\\
&\mathrm{s.t.}\,\,\,0<p_{i}\leq 1,\,\,\forall i\in\\{1,2,....,N\\}.\\\
\end{split}$ (13)

Deriving a closed for expression for the solution of (13) a hard problem due
to the spatio-temporal correlation of link states. In the following section,
we introduce a DRL algorithm to solve optimization problem in (13).

##  IV The GLI-Based Deep Learning Approach

In this section, we present the design structure of the GLI-based NN to
approximate the solution of (13) by mapping the GLI to the scheduling policy.
The conventional method to solve the optimization problem in (13) requires a
full information about the CSI, which incurs $O(N^{2})$ computational
complexity for a network of $N$ D2D pairs. The CSI could be mapped using GLI,
which is considered as a function of CSI that captures the main feature of the
wireless channels (the path loss and shadowing of a wireless link are mostly
functions of distance and location). In that sense, we use the GLI as input to
the NN to acquire the optimal scheduling policy $\pi^{*}$. The structure of NN
is depicted in Fig. 3 and is illustrated in more details as follows.

Figure 3: The structure of considered NN.

###  IV-A D2D Density Grid

First, we construct grid matrices to quantize the continuous form of the
locations of the transmitters and receivers as shown in Fig. 4. We assume that
the considered network is distributed in a square-shaped area with a side
length of $L$. The whole layout is then partitioned into square cells, where
the GLI information is represented as the tuple
$\\{(x_{i}^{tx},y_{i}^{tx}),(x_{i}^{rx},y_{i}^{rx})\\}_{i=1}^{N}$, where
$(x_{i},y_{i})$ is the index of the cell and the coordinate values ranges from
0 to $L$. We consider two sub-grids, $G^{Tx}$ and $G^{Rx}$ that represent the
activation state of the transmitters and receivers, respectively. For a grid
size of $R\times R$, the transmitter matrix $G_{i}^{Tx}$ of link $i$ is
defined as

$G_{i}^{Tx}=\begin{cases}1&\text{if}(x,y)=\lceil(x_{i}^{tx},y_{i}^{tx})*R/L\rceil\\\
0&\text{otherwise.}\end{cases}$ (14)

Based on the activation probability $p_{i}$, we have

$G^{Tx}=\sum_{j=i}^{N}p_{j}G_{j}^{Tx}.$ (15)

Note that the same applies to $G^{Rx}$. In this case, the matrices $G^{Tx}$
and $G^{Rx}$ represent GLI information to solve the problem in (13) where
probability $p_{i}$ reflects the interference to other links, hence they could
be regarded as feature matrices to the convolution layer presented in the next
subsection.

###  IV-B The 3-Layer Convolution Stage

The two matrices $G^{Tx}$ and $G^{Rx}$ are processed via three connected
convolution layers and the output is a set of extracted feature after each
layer. Each entry in the resulting matrix comes from a unique convolution
positioned at the corresponding index of the input matrix using the
convolution filter. When the index represents the receiver’s location, then
the convolution essentially extracts features from all the transmitters in
proximity to this receiver, based on the size of the convolution filter. Both
$G^{Tx}$ and $G^{Rx}$ undergo the convolution phase concurrently, producing
three matrices each. Then, every link on the D2D plane gleans a total of six
features from these matrices, based on the index of its individual receiver or
transmitter.

Figure 4: Construction of the transmitter and receiver density grids.

###  IV-C The Fully Connected Stage

The second stage is fully connected stage that comprises two hidden layers. We
consider the Rectified Linear Unit (ReLU) as the activation function of each
neuron where a sigmoid non-linearity is used at the output node to produce the
activation probability $p_{i}$. For the considered D2D configuration with $N$
links, the feature vectors for each link are processed through the fully
connected layer. This results in a collection of activation probability
vectors $\pi$. In order to consider the Tx-Rx features in the learning
process, we include the $p_{i}$ from the previous iteration and the distance
between the Tx and Rx of the link as two features into the fully connected
stage. Particularly, the output probabilities $p_{i}$ are used as inputs to
get new GLI using (15). This in turn helps to improve the convergence of the
NN and enhance the training process in general.

###  IV-D The NN Training Process

We train the NN using a randomly generated set of D2D layouts to minimize the
AoI via gradient descent on the convolutional filter weights and the NN weight
parameters. Specifically, the locations of $N$ transmitters are first
generated uniformly within the region $L\times L$, and then the locations of
the corresponding receivers are generated following a uniform distribution
within a pairwise distances of $\\{d_{min},d_{max}\\}$. Although the training
stage would require the channel gains, it will no longer be required after the
network is well-trained and only $N$ GLI is required to obtain the optimal
schedule. This way, for each iteration, the NN tends to improve the scheduling
policy of the previous iteration.

TABLE I: Evaluation parameters Parameter | Value
---|---
Deployment area |  $600$ meters $\times$ $600$ meters
Hidden layer |  $30$ neurons
GLI grid length $R$ | $150$
Size of convolution filter | $10\times 10$
Path loss exponent ($\alpha$) | $3$
SINR threshold ($\beta$) |  $0$ dB
Noise power ($\sigma^{2}$) |  $-90$ dBm
Transmission power |  $100$ mW

##  V Performance Evaluations

In this section, we present the setup of the training and test processes of
the considered NN, and evaluate the performance of the proposed AA-DRL via
numerical results with the relevant parameters listed in Table I.

###  V-A Network Setup and Training Process

We consider a D2D-assisted IIoT network where $N$ D2D pairs are distributed in
$600$ meters $\times$ $600$ meters region. The D2D transmitters are uniformly
positioned within the deployment area while the locations of the corresponding
receivers are generated following a uniform distribution within the pairwise
distances $\\{d_{min}=2$ meters and $d_{max}=80$ meters$\\}$. We generate
10000 sets of such layout to train the NN and 5000 sets for testing. The NN
consists of 3 convolution filters with each filter of size $20\times$ $20$.
Each hidden layer in the fully connected stage comprises 30 neurons utilizing
ReLU and sigmoid functions.

Figure 5: Comparison of the average PAoI under different values of $\lambda$
with $N=100$ and $N=400$.  Figure 6: CDF of AA-DRL and benchmark with $N=100$
and $\lambda=0.2$ of 1000 different network layouts.

###  V-B Numerical Results

The following results show the performance of the proposed AA-DRL scheme in
terms of the average PAoI and the computational complexity. Moreover, we use
the work in [9] as a benchmark to prove the effectiveness of our proposed
approach. Fig.5 compares the average PAoI of AA-DRL and the benchmark in [9]
under varying $\lambda$. The figure shows superior performance of the proposed
AA-DRL over the benchmark, especially when the network size increases from
$N=100$ to $N=400$. For instance, while AA-DRL achieves $55\%$ reduction in
the AoI compared to the benchmark at $N=100$ and $\lambda=0.71$, this
percentage increases to $75\%$ at $N=400$ and $\lambda=0.81$. The
effectiveness of our proposed AA-DRL is attained through incorporating and
mapping of the GLI in the scheduling policy, while the benchmark work is based
only on local observations of the backlog status of the users, which would be
inefficient in high interference regimes.

To further emphasize the effectiveness of the proposed AA-DRL considering
different layouts, we plot the Cumulative Distribution Function (CDF) of the
average PAoI in Fig. 6 with $N=100$ and $\lambda=0.2$. Although different PAoI
values are obtained under different layouts, we can observe that the CDF
curves of both approaches have the same trend, hence the performance
improvements of AA-DRL could be guaranteed under different network
distributions (i.e., different spatial locations).

Figure 7: Log scale values of the computation time required to optimize a
single layout under varying $N$.

One concern regarding the proposed AA-DRL would be its corresponding
computational complexity. In the following, we roughly analyze the
computational complexity of AA-DRL and compare it with an optimal iterative
approach. The iterative approach uses a convex optimization solver (e.g., CVX)
to obtain a local optimal probability $p_{i}$. The access probability $p_{i}$
is updated in each iteration towards a minimum PAoI based on the obtained
$p_{i}$ from the previous iteration. This way, $N$ one-variable problems are
solved in a single iteration. For each iteration, the problem requires the
collection of $N^{2}$ elements of the CSI matrix, leading to at least
$O(N^{2})$ computational complexity of this algorithm. For our proposed AA-DRL
algorithm, the overall computation is
$f\times[R^{2}\times(c_{1}^{2}+c_{2}^{2}+c_{3}^{2})+N\times 10\times
h_{1}\times h_{2}]$, where $f$ is the number of feedback rounds, $c_{i}$ is
the size of the convolution filter of stage $i$ and $h_{1}$ and $h_{2}$ are
the number of neurons in each hidden layer. Therefore, for a given D2D layout,
the time complexity of AA-DRL scales as $O(N)$. Fig. 7 shows the computation
time in log scale of AA-DRL and the iterative method under different values of
$N$. For the sake of reasonable comparison, we chose hardware configurations
that are optimally compatible with each algorithm. For AA-DRL, we used Nvidia
GPU GeForce GTX 1080Ti, while for the iterative algorithm we used Intel CPU
Core i7-8700K @ 3.70GHz. The design of the NN in the proposed AA-DRL is
inherently amenable to parallel processing, gaining substantial advantages
from the parallel computational capabilities of GPUs. Conversely, the
iterative algorithm exhibits inherently sequential computation patterns,
making it more suited to CPUs, which offer higher clock speeds. As it is
demonstrated by Fig. 7, our proposed AA-DRL shows considerable computational
advantages over the iterative approach in large-scale deployments while
achieving optimal AoI performance. It is also worth mentioning that the
considered IIoT architecture shown in Fig. 1 provides computational advantage
where the complexity of AA-DRL would be further relaxed taking advantage of
the computational capabilities of the edge node.

##  VI Conclusion

In this paper, we proposed a DRL-based approach to optimize AoI in
D2D-assisted IIoT networks. We formulated a scheduling problem to minimize the
average PAoI and developed a NN that maps the GLI to the optimal scheduling
policy. The obtained results showed that our proposed approach achieves
improved PAoI compared to a recent benchmark, all while exhibiting reduced
computational complexity in contrast to the traditional iterative minimization
algorithm.

## Acknowledgement

This paper has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No. 883315.

## References

  * [1] J. Lin, W. Yu, N. Zhang, X. Yang, H. Zhang, and W. Zhao, “A survey on internet of things: Architecture, enabling technologies, security and privacy, and applications,” _IEEE Internet of Things Journal_ , vol. 4, no. 5, pp. 1125–1142, 2017.
  * [2] H. Farag, E. Sisinni, M. Gidlund, and P. Österberg, “Priority-aware wireless fieldbus protocol for mixed-criticality industrial wireless sensor networks,” _IEEE Sensors Journal_ , vol. 19, no. 7, pp. 2767–2780, 2019\.
  * [3] L. Liu and W. Yu, “A d2d-based protocol for ultra-reliable wireless communications for industrial automation,” _IEEE Transactions on Wireless Communications_ , vol. 17, no. 8, pp. 5045–5058, 2018.
  * [4] P. Gil, A. Santos, and A. Cardoso, “Dealing with outliers in wireless sensor networks: An oil refinery application,” _IEEE Transactions on Control Systems Technology_ , vol. 22, no. 4, pp. 1589–1596, 2014.
  * [5] Y. Sun _et al._ , “Update or wait: How to keep your data fresh,” _IEEE Trans. Inf. Theory_ , vol. 63, no. 11, pp. 7492–7508, 2017.
  * [6] Q. He, D. Yuan, and A. Ephremides, “Optimal link scheduling for age minimization in wireless systems,” _IEEE Transactions on Information Theory_ , vol. 64, no. 7, pp. 5381–5394, 2018.
  * [7] J. Peng, H. Qiu, J. Cai, W. Xu, and J. Wang, “D2d-assisted multi-user cooperative partial offloading, transmission scheduling and computation allocating for mec,” _IEEE Transactions on Wireless Communications_ , vol. 20, no. 8, pp. 4858–4873, 2021.
  * [8] W. Cui, K. Shen, and W. Yu, “Spatial deep learning for wireless scheduling,” _IEEE Journal on Selected Areas in Communications_ , vol. 37, no. 6, pp. 1248–1261, 2019.
  * [9] H. H. Yang, A. Arafa, T. Q. S. Quek, and H. V. Poor, “Spatiotemporal analysis for age of information in random access networks under last-come first-serve with replacement protocol,” _IEEE Transactions on Wireless Communications_ , vol. 21, no. 4, pp. 2813–2829, 2022.
  * [10] M. Costa, M. Codreanu, and A. Ephremides, “On the age of information in status update systems with packet management,” _IEEE Transactions on Information Theory_ , vol. 62, no. 4, pp. 1897–1910, 2016.
  * [11] R. Talak, S. Karaman, and E. Modiano, “Improving age of information in wireless networks with perfect channel state information,” _IEEE/ACM Transactions on Networking_ , vol. 28, no. 4, pp. 1765–1778, 2020.
  * [12] C. Li, Q. Liu, S. Li, Y. Chen, Y. T. Hou, W. Lou, and S. Kompella, “Scheduling with age of information guarantee,” _IEEE/ACM Transactions on Networking_ , pp. 1–14, 2022.
  * [13] H. Farag, M. Gidlund, and Č. Stefanović, “A deep reinforcement learning approach for improving age of information in mission-critical iot,” in _2021 IEEE Global Conference on Artificial Intelligence and Internet of Things (GCAIoT)_ , 2021, pp. 14–18.
  * [14] M. Elnourani, S. Deshmukh, and B. Beferull-Lozano, “Distributed resource allocation in underlay multicast d2d communications,” _IEEE Transactions on Communications_ , vol. 69, no. 5, pp. 3409–3422, 2021.
  * [15] G. Chisci, H. Elsawy, A. Conti, M.-S. Alouini, and M. Z. Win, “Uncoordinated massive wireless networks: Spatiotemporal models and multiaccess strategies,” _IEEE/ACM Transactions on Networking_ , vol. 27, no. 3, pp. 918–931, 2019.
  * [16] M. Liu and L. Zhang, “Resource allocation for d2d underlay communications with proportional fairness using iterative-based approach,” _IEEE Access_ , vol. 8, pp. 143 787–143 801, 2020.
  * [17] H. H. Yang, A. Arafa, T. Q. S. Quek, and H. V. Poor, “Optimizing information freshness in wireless networks: A stochastic geometry approach,” _IEEE Transactions on Mobile Computing_ , vol. 20, no. 6, pp. 2269–2280, 2021.
  * [18] M. Li, C. Chen, H. Wu, X. Guan, and X. Shen, “Age-of-information aware scheduling for edge-assisted industrial wireless networks,” _IEEE Transactions on Industrial Informatics_ , vol. 17, no. 8, pp. 5562–5571, 2021\.
  * [19] H. Farag, C. Stefanovic, and M. Gidlund, “Distributed backlog-aware protocol for heterogeneous d2d communication-assisted wireless sensor networks,” _IEEE Transactions on Mobile Computing_ , pp. 1–12, 2023.
  * [20] C. Chen, L. Lyu, S. Zhu, and X. Guan, “On-demand transmission for edge-assisted remote control in industrial network systems,” _IEEE Transactions on Industrial Informatics_ , vol. 16, no. 7, pp. 4842–4854, 2020\.
  * [21] E. T. Ceran, D. Gündüz, and A. György, “Average age of information with hybrid arq under a resource constraint,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 3, pp. 1900–1913, 2019.
  * [22] R. Talak, S. Karaman, and E. Modiano, “Optimizing information freshness in wireless networks under general interference constraints,” _IEEE/ACM Transactions on Networking_ , vol. 28, no. 1, pp. 15–28, 2020.
  * [23] M. Zorzi and R. Rao, “Capture and retransmission control in mobile radio,” _IEEE Journal on Selected Areas in Communications_ , vol. 12, no. 8, pp. 1289–1298, 1994.
  * [24] Y. Gu, H. Chen, Y. Zhou, Y. Li, and B. Vucetic, “Timely status update in internet of things monitoring systems: An age-energy tradeoff,” _IEEE Internet of Things Journal_ , vol. 6, no. 3, pp. 5324–5335, 2019.