# Non-orthogonal Age-Optimal Information Dissemination in Vehicular Networks:
A Meta Multi-Objective Reinforcement Learning Approach

Ahmed A. Al-Habob, Hina Tabassum, and Omer Waqar This work was supported by
two Discovery Grants funded by the Natural Sciences and Engineering Research
Council of Canada (NSERC).At the time of this work, A. A. Habob was with the
Department of Electrical Engineering and Computer Science at York University,
Toronto, Canada. He is currently with the Memorial University of Newfoundland,
Canada.H. Tabassum is with the Department of Electrical Engineering and
Computer Science, York University, Toronto, ON, Canada.O. Waqar is with the
School of Computing, University of the Fraser Valley, BC, Canada and also
affiliated as an adjunct faculty member with the Department of Electrical
Engineering and Computer Science, York University, Toronto, ON, Canada.
E-mails<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This paper considers minimizing the age-of-information (AoI) and transmit
power consumption in a vehicular network, where a roadside unit (RSU) provides
timely updates about a set of physical processes to vehicles. We consider non-
orthogonal multi-modal information dissemination, which is based on superposed
message transmission from RSU and successive interference cancellation (SIC)
at vehicles. The formulated problem is a multi-objective mixed-integer
nonlinear programming problem; thus, a Pareto-optimal front is very
challenging to obtain. First, we leverage the weighted-sum approach to
decompose the multi-objective problem into a set of multiple single-objective
sub-problems corresponding to each predefined objective preference weight.
Then, we develop a hybrid deep Q-network (DQN)-deep deterministic policy
gradient (DDPG) model to solve each optimization sub-problem respective to
predefined objective-preference weight. The DQN optimizes the decoding order,
while the DDPG solves the continuous power allocation. The model needs to be
retrained for each sub-problem. We then present a two-stage meta-multi-
objective reinforcement learning solution to estimate the Pareto front with a
few fine-tuning update steps without retraining the model for each sub-
problem. Simulation results illustrate the efficacy of the proposed solutions
compared to the existing benchmarks and that the meta-multi-objective
reinforcement learning model estimates a high-quality Pareto frontier with
reduced training time.

###### Index Terms:

Age-of-information (AoI), deep reinforcement learning (DRL), meta deep
reinforcement learning (meta-DRL), multi-objective optimization, successive
interference cancellation (SIC).

##  1 Introduction

Vehicular communication networks enable a wide range of applications which
require real-time updates, such as highly prioritized road safety applications
(such as collision warning and adaptive cruise control) and the infotainment
services such as news, media and social entertainments, which require real-
time updates [1]. With the increasing diversity of vehicular applications that
require real-time information updates, such as blind spot/lane change and
forward collision warnings, communications in vehicular networks become time-
critical, and thus, fresh status updates are of high importance [2].

Although the conventional communication latency and throughput are effective
metrics to evaluate the performance of the vehicular communication networks,
these metrics do not capture the information freshness which is critical to
obtain the real-time knowledge about the location, orientation, and speed of
the vehicles. To this end, the age-of-information (AoI) is a useful metric to
quantify the freshness of the information while taking into account the
transmission latency, update generation time, and inter-update time interval.
Specifically, AoI is defined as the elapsed time between the received
information at the destination and the time when it was generated at the
source [3]. It should be noted that the inter-update time —which is a
scheduling parameter— is a crucial parameter in the AoI [3], and thus
optimizing AoI is totally different from optimizing other metrics such as the
throughput and latency.

Along another note, the existing state-of-the-art considers minimizing the AoI
in uni-modal information dissemination scenario, where the destination/vehicle
receives updates about a single application or physical process. However, a
more practical scenario that should be addressed is the multi-modal
information dissemination, in which each vehicle is interested in maintaining
fresh status updates for one or more applications (physical processes). The
straightforward strategy is to broadcast updates about all physical processes
to all vehicles at each time slot. However, such a strategy consumes the
transmitter’s power. Consequently, the trade-off between maintaining
information freshness and reserving the power consumption should be handled to
satisfy the decision-maker’s preference. Such a trade-off mandates a proper
multi-objective optimization framework that handles the decision-maker’s
preference by considering efficient messages’ encoding scheme to
unicast/multicast updates and allocating the power in vehicular networks.

Traditionally, optimization approaches tackle the resource allocation in
vehicular networks [4]. However, the communication channels in vehicular
networks are rapidly varying. Furthermore, optimizing time-dependent metrics
such as AoI involves a sequential decision-making over time. Thus, iterative
and computationally complex optimization approaches are not well-suited. To
this end, deep reinforcement learning (DRL) has been considered as a promising
solution to learn a better policy on decision-making problems that are
evolving over time[5, 6].

DRL paradigm has witnessed dramatic evolution to handle complex scenarios such
as discrete-continuous hybrid action space and different training and testing
environments [7]. A variant of the DRL is the multi-objective reinforcement
learning (MORL) [8], in which the agent aims at optimizing multiple
conflicting objectives. MORL inherits the well-known challenges of multi-
objective optimization111The conventional strategy to handle multi-objective
optimization is to convert the problem into a single objective optimization
problem, which can be implemented using $\epsilon$-constraint approach [9],
Tchebycheff approach [10], or weighted sum approach [11, 12, 13]., including
the trade-off between objectives with different units, ranges, and order of
magnitude [12].

To address the aforementioned issues, a DRL agent can be trained and tested to
find the preferred Pareto optimal solution based on the decision-maker’s
preference. The drawback is that the DRL agent needs to be retrained if the
objectives’ preference is modified or the vehicular communication environment
is changed. Recently, meta-DRL concept has been introduced to enable the agent
to quickly adapt to new environments by learning the agent a meta-policy that
solves multiple tasks from a given distribution [14]. Meta-learning (or the
learning to learn) can leverage knowledge from previous experiences to rapidly
adapt to new/unseen tasks with few training (fine-tuning) steps [14, 15]. In
this context, meta-DRL can be considered a posteriori paradigm to address the
objectives’ preference issue, in which the meta-DRL is trained to find the
Pareto optimal solutions of a set of objectives’ preferences which can be
adapted or fine-tuned to the desirable solution [16, 15].

###  1.1 Related Works

Optimizing AoI in vehicular networks has been investigated in different
scenarios [17, 18, 5, 19, 2]. In [17], a greedy algorithm was developed to
minimize the expected sum AoI in a vehicular beacon broadcasting system and
mitigate beacons’ signals collision. In [18], a Lyapunov optimization solution
was developed to minimize the transmit power in a vehicle-to-vehicle (V2V)
network to facilitate ultra-reliable low-latency V2V communications subject to
probabilistic AoI constraints. A proactive DRL technique was proposed in [5]
to provide an AoI-aware radio resource management in vehicular networks with
Manhattan grid road topology. In [19], the impact of inter-update generation,
selection of fog/cloud servers, and processing delay on the AoI was studied in
a vehicular shuttle system. A deep Q-learning algorithm was developed to
optimize the vehicles’ routing to improve the average AoI. In [2], the social
relations among users in the vehicle network were considered in an AoI-centric
information dissemination model. The authors considered joint optimization of
the information update rate and the transmit probabilities to minimize AoI.

Different unicast–multicast scenarios were considered in literature to
transmit multiple messages to a group of users [20, 21, 22, 23, 24]. In [20],
a non-orthogonal multiple access (NOMA) unicast–multicast scenario was
designed, in which a set of unicast users (each requires unique message) and a
set of multicast users (those who require an identical message) shares the
same time/space/frequency resource. In this scenario, the messages of unicast
users were encoded according to their channels’ quality and the multicast
message was superposed as the last encoded message. An integrated multicast-
unicast scenario was proposed in [21, 22], such that each user receives a
private message and a common message was broadcasted to all users. Sum-rate
maximization was considered in [21], while the authors of [22] considered
maximizing the energy efficiency in a simultaneous wireless information and
power transfer protocol. In [23], a joint unicast and multi-group multicast
transmission has been considered, in which a user is either a unicast or
belongs to a group of multicast users. In [24], a fixed interval transmission
strategy was considered to minimize the average AoI in a unilateral multicast
network. The considered network consists of a base-station (BS) broadcasting
time-sensitive updates to a set of users, in which the BS decides to broadcast
an update packet at time. In [25], a digital twin-driven vehicular task
offloading was studied to provide augmented computing capacities for internet
of vehicles (IoV) scenario by considering mobile edge computing (MEC) and
intelligent reflective surface (IRS). To reduce the overall delay and energy
consumption, a two-stage optimization approach was developed for Jointly
Optimizing Task Offloading and IRS Configuration. Another IoV scenario was
studied in [26], in which an asynchronous federated broad learning (FBL)
framework integrates broad learning (BL) into federated learning (FL).

Studying AoI in unicasts/multicasts scenarios has been considered in
literature [27, 28, 29, 30, 31]. In [27], the average and peak AoI of
multicast transmission with deadlines has been considered. An access point
transmits timestamped status updates to multiple devices and the status update
is terminated if either a subset of devices devices successfully receive the
status update or the deadline expires. In [28], a multicast scheduling
strategy has been proposed to improve the energy efficiency and AoI. The
server transmits information to multiple users by queuing and bundling the
requests from different users and serving users requesting the same contents.
In [29], the AoI in multicast networks with retransmissions has been studied,
in which the updates are encoded as a short blocklength packets and is
broadcasted to multiple destinations via independent and identically
distributed error-prone channels. The stopping threshold, the average AoI and
EE expressions for stopping and wait-for-all schemes are derived as a function
of the packet length. In [30], an architecture of AoI in
multicast/unicast/device-to-device (D2D) transmission with a cell-free massive
multiple-input multiple-output (MIMO) was studeied. An age-optimum unicast-
multicast scheduling of multiple update messages to vehicles has been
considered in [31]. In this framework, at most an update about one physical
process is scheduled to a vehicle at each time slot. Ant colony optimization
algorithm and deep Q-learning model were developed to solve a weighted sum
optimization problem of the AoI and power consumption.

TABLE I: Existing related work.

Ref. Year Vehicular network AoI NOMA multi-modal dissemination DRL-based
solution Hybrid DQN-DDPG meta-DRL Pareto front using meta-MORL [17] $2018$ ✓ ✓
X X X X X [18] $2020$ ✓ ✓ X X X X X [5] $2020$ ✓ ✓ X ✓ X X X [19] $2020$ ✓ ✓ X
✓ X X X [2] $2022$ ✓ ✓ X X X X X [27] $2020$ X ✓ X X X X X [28] $2018$ X ✓ X X
X X X [29] $2021$ X ✓ X X X X X [30] $2023$ X ✓ X X X X X [7] $2021$ ✓ X X ✓ ✓
✓ X [31] $2022$ ✓ ✓ X ✓ X X X Ours – ✓ ✓ ✓ ✓ ✓ ✓ ✓

###  1.2 Motivation and Contributions

None of the research works to date considered the non-orthogonal multi-modal
information dissemination in a vehicular network with multiple conflicting
objectives such as AoI minimization and power consumption minimization
simultaneously. We cast this problem as a multi-objective optimization problem
(MOOP) and demonstrate the efficacy and the generalization capability of the
meta multi-objective RL for estimating the Pareto front of the MOOP.
Estimating the Pareto front in MOOP is essential, as it enables the decision-
maker to optimize the conflicting objectives without the need to pre-determine
the preference weight corresponding to each Pareto point. It is shown that the
proposed model can estimate the entire Pareto front using few fine-tuning
update steps, without the need to retrain a new DRL model for each point in
the Pareto front.  To summarize the existing work and highlight our key
contributions, Table I summarizes the related work in terms of considering key
techniques such as minimizing AoI, meta-DRL, meta-MORL, etc.  The main
contributions of this paper can then be summarized as follows:

  * •

This paper proposes a non-orthogonal multi-modal information dissemination
framework222The non-orthogonal multi-modal information dissemination is based
on superposed message transmission from RSU and successive interference
cancellation (SIC)-enabled decoding at vehicles.  in which each vehicle can
receive updates about one or more physical processes at a time. A roadside
unit (RSU) schedules updates on multiple processes such that the average AoI
and the RSU’s power consumption can be minimized at the same time. The two
objectives are coupled in a conflicting manner due to the transmit power
allocations.

  * •

We develop a meta multi-objective reinforcement learning (meta-MORL) framework
to minimize both the AoI at the vehicles and the RSU’s power expenditure while
optimizing the messages’ decoding order and their corresponding power
allocations. In this context, we

    * –

first design a hybrid DRL model, namely, hybrid deep Q-network (DQN)-deep
deterministic policy gradient (DDPG) model to obtain the Pareto front of the
considered multi-objective problem. The DQN solves the messages’ decoding
order and the DDPG handles the continuous power allocation decision. The model
needs to be retrained for each point of the Pareto front.

    * –

then, we develop a two-stage meta-MORL solution to deal with the multiple sub-
problems determined by the preference weight of the objectives and to estimate
the Pareto front without retraining. The first stage trains a policy with a
good generalization of the preference weight of the objectives. The fine-
tuning stage is then applied to quickly adapt the trained policy for an unseen
preference weight of the objectives.

  * •

Extensive simulations are provided to evaluate the performance of the proposed
algorithm and the generalization capability of the proposed meta-MORL
solution. The results demonstrate that the proposed algorithms can efficiently
optimize the messages’ decoding order and power allocation issues, and the
meta-multi-objective RL adapts quickly to the problem instances with
unseen/new objective-preference weight.

The remainder of this paper is organized as follows. Section 2 presents the
system model and the performance metrics. The problem is formulated in Section
3. The proposed hybrid DQN-DDPG DRL model is introduced in Section 4 and the
multi-objective meta-DRL solution is introduced in Section 5. Section 6
illustrates simulation results and Section 7 concludes the paper.

TABLE II: Main notations used in the paper.

Notation Description Notation Description $V$ Number of vehicles in the
vehicles set $\mathcal{V}$ $N$ Number of antenna elements at the RSU $F$
Number of physical processes in the processes set $\mathcal{F}$ $L$ Payload
size of an update (in bits) $\mathcal{R}_{i}$ Processes of interest to vehicle
$v_{i}$ ($\mathcal{R}_{i}\subseteq\mathcal{F}$)
$\bm{R}=[r_{i,l}]_{\footnotesize{V\times F}}$ $r_{i,l}=1$ if vehicle $v_{i}$
is interested in process $f_{l}$, $r_{i,l}=0$ otherwise $T/\delta$ Number of
time slots/Duration of each slot $\psi_{0}=\\{x_{0},y_{0}\\}$ Coordinates of
the RSU $\psi^{(t)}_{i}=\\{x^{(t)}_{i},y^{(t)}_{i}\\}$ Coordinates of vehicle
$i$ at time slot $t$ $\phi_{i}^{(t)}$ Angle of vehicle $i$ relative to the RSU
at time slot $t$ $\ell_{i}^{(t)}$ Distance between the vehicle $i$ and the RSU
$\mbox{{h}}_{i}^{(t)}$ Communication channel between the RSU and vehicle $i$
at time slot $t$ $f_{c}/c_{0}/c_{i}$ Carrier frequency/Speed of light/Speed of
vehicle $i$ $\varrho_{i}^{(t)}$ Doppler shift due to the movement of vehicle
$i$ $\mbox{{a}}(\phi_{i}^{(t)})/\phi_{i}^{(t)}$ Transmit steering
vector/Azimuth angle between RSU and $v_{i}$ at slot $t$ $\bm{\pi}^{(t)}$
Decoding order decision of the messages at time slot $t$ $\mbox{{p}}^{(t)}$
Power allocation decision at time slot $t$ $z_{i}^{(t)}/n_{i}$ Received
signal/Additive white Gaussian noise at vehicle $i$ w Beamforming vector at
time slot $t$ $\chi^{(t)}_{i}/\tilde{\xi}$ Large-scale channel attenuation of
vehicle $i$/Normalization factor
$\gamma_{i,\pi_{l^{\prime}}^{(t)}}^{(t)}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)})$
SINR experienced at vehicle $i$ to decode the $\pi_{l^{\prime}}^{(t)}$-th
message $\varepsilon^{\mbox{\scriptsize
max}}_{i}/\varepsilon_{i}(\gamma_{i}^{(t)})$ Maximum allowed error
probability/Decoding error probability $\Phi\left(\cdot\right)/\omega$ Channel
dispersion/Channel bandwidth $\delta_{1}/\delta_{2}$ Vehicles’ parameters
acquisition time/Information transmission time
$\Delta_{i\pi_{l^{\prime}}}^{(t)}/\bar{\Delta}_{{i,l}}$ Instantaneous
AoI/Time-average AoI of $f_{l}$ at vehicle $i$
$\bar{\Delta}^{\mbox{\scriptsize max}}/\bar{\Delta}^{\mbox{\scriptsize min}}$
Maximum/Minimum value of the total time-average AoI $\bar{{p}}^{(t)}$ Time-
average power consumption at RSU
${O}(\bm{\pi}^{(t)}\\!,\mbox{{p}}^{(t)})/\zeta$ Objective function/Relative
objective’s preference weight $\mathcal{S}/\mathcal{A}/\rho^{(t)}$ State
space/Action space/Immediate reward $\theta^{Q}/\theta^{Q_{c}}/\theta^{\mu}$
DQL network weights/Critic network weights/Actor network weights

##  2 System Model and Performance Metrics

This section introduces the considered system model, communication model, and
performance metrics. The notations used throughout this paper are listed in
Table II.

###  2.1 Network Model

The considered system consists of a set $\mathcal{V}=\\{v_{i}\\}_{i=1}^{V}$ of
$V$ vehicles supported by an RSU that disseminates timely status updates to
the vehicles. The RSU is equipped with a uniform linear array of $N$ antennas.
A multi-modal data dissemination scenario is considered, in which the RSU is
capable of providing timely status updates about a set
$\mathcal{F}=\\{f_{l}\\}_{l=1}^{F}$ of $F$ physical processes. The payload
size of an update is $L$ bits.  Each vehicle is interested in maintaining
freshness of its information status about a subset of processes
$\mathcal{R}_{i}\subseteq\mathcal{F}$. To represent the information demands of
the vehicles, we define $\bm{R}=[r_{i,l}]_{\footnotesize{V\times F}}$ such
that

$r_{i,l}=\begin{cases}1,&\mbox{if vehicle }i~{}\mbox{is interested in
process}~{}l,\\\ 0,&\mbox{otherwise}.\end{cases}$ (1)

The time is divided into $T$ time slots each of duration $\delta$. Let
$\psi_{0}=\\{x_{0},y_{0}\\}$ be the coordinates of the RSU and
$\psi^{(t)}_{i}=\\{x^{(t)}_{i},y^{(t)}_{i}\\}$ be the coordinates of vehicle
$i$ at time slot $t$. The angle of vehicle $i$ relative to the RSU at time
slot $t$ can be expressed as follows:

$\phi_{i}^{(t)}=\arccos\frac{x^{(t)}_{i}-x_{0}}{\ell_{i}^{(t)}},$ (2)

where $\ell_{i}^{(t)}=\lVert\psi^{(t)}_{i}-\psi_{0}\rVert$ is the distance
between the vehicle $i$ and the RSU. The communication channel between the RSU
and vehicle $i$ at time slot $t$ is modeled as follows:

$\mbox{{h}}_{i}^{(t)}=\sqrt{\frac{c_{0}}{4\pi
f_{c}\ell_{i}^{(t)^{2}}}}\mbox{{a}}^{H}(\phi_{i}^{(t)})e^{j2\pi\varrho_{i}^{(t)}},$
(3)

where $f_{c}$ is the carrier frequency, $c_{0}$ is the speed of light, and
$\varrho_{i}^{(t)}$ is the Doppler shift due to the movement of vehicle $i$
expressed as $\varrho_{i}^{(t)}=\frac{c_{i}f_{c}\cos\phi_{i}^{(t)}}{c_{0}},$
where $c_{i}$ is the speed of vehicle $i$ [32]. Assuming a uniform linear
antenna array at the RSU, the transmit array steering vector
$\mbox{{a}}(\phi_{i}^{(t)})\in\mathbb{C}^{N\times 1}$ (with $\phi_{i}^{(t)}$
as the azimuth angle between the RSU and vehicle $i$ at time slot $t$) can be
expressed as follows:

$\mbox{{a}}(\phi_{i}^{(t)})\\!=\\![1,e^{j\pi\sin\phi_{i}^{(t)}}\\!,e^{j2\pi\sin\phi_{i}^{(t)}}\\!,\cdots\\!,e^{j(N-1)\pi\sin\phi_{i}^{(t)}}],$
(4)

where $j=\sqrt{-1}$ and the antenna spacing is $\lambda/2$ with $\lambda$ as
the carrier wavelength.

###  2.2 Received Signal and SINR Model

Let $[f^{(t)}_{1},f^{(t)}_{2},\cdots,f^{(t)}_{F}]$ be the raw messages of the
physical processes $\mathcal{F}$ at time slot $t$,  the RSU sends a superposed
message $\sum_{l=1}^{F}\sqrt{p^{(t)}_{\pi_{l}^{(t)}}}f^{(t)}_{\pi_{l}^{(t)}}$
according to a decoding order decision
$\bm{\pi}^{(t)}=[\pi^{(t)}_{1},\pi^{(t)}_{2},\cdots,\pi^{(t)}_{F}]$ and power
allocation decision $\mbox{{p}}^{(t)}=[p^{(t)}_{1},\cdots,p^{(t)}_{F}]$, where
$\pi^{(t)}_{l}$ is the $l$-th elements of the decoding order decision
$\bm{\pi}^{(t)}$. The RSU broadcasts the superposed message to the vehicles.
Therefore, the received signal at vehicle $i$ can be modeled as follows:

$z_{i}^{(t)}=\mbox{{{w}}}^{(t)}\left(\sum_{l=1}^{F}\sqrt{p^{(t)}_{\pi_{l}^{(t)}}}f^{(t)}_{\pi_{l}^{(t)}}\right)\mbox{{h}}_{i}^{(t)^{H}}+n_{i},$
(5)

where $n_{i}\sim\mathcal{CN}(0,\sigma^{2})$ is the additive white Gaussian
noise (AWGN), $H$ denotes the Hermitian transpose, and $\mbox{{{w}}}^{(t)}$ is
the beamforming vector. The maximum ratio transmission (MRT) beamforming
scheme is considered [33], in which the asymptotically optimal beamformer
vector for a set of vehicles $\mathcal{V}$ is a linear combination of channels
of these vehicles [34, 23]. Consequently, the MRT beamforming vector is
expressed as follows:

${\mbox{{{w}}}}^{(t)}=\sum_{i=1}^{V}\frac{\mbox{{h}}_{i}^{(t)}}{\sqrt{N\chi^{(t)}_{i}\tilde{\xi}}},$
(6)

where $\chi^{(t)}_{i}={\frac{c_{0}}{4\pi
f_{c}\ell_{i}^{(t)^{2}}}}e^{-j2\pi\varrho_{i}^{(t)}}$ is the large-scale
channel attenuation of vehicle $i$ and $\tilde{\xi}$ is a normalization factor
[23, 35]. For a given decoding order decision $\bm{\pi}^{(t)}$, the vehicles
decode the messages such that the message corresponding to the
$\pi_{l^{\prime}}^{(t)}$-th physical process is decoded before the message of
the $\pi_{m^{\prime}}^{(t)}$-th process,
$\forall~{}{l^{\prime}}\leq{m^{\prime}}$. Consequently, the signal-to-
interference plus-noise ratio (SINR) experienced at vehicle $i$ to decode the
$\pi_{l^{\prime}}^{(t)}$-th message is given by

$\gamma_{i,\pi_{l^{\prime}}^{(t)}}^{(t)}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)})=\frac{p^{(t)}_{\pi_{l^{\prime}}^{(t)}}|\mbox{{h}}_{i}^{(t)^{H}}{\mbox{{{w}}}}^{(t)}|^{2}}{\sum\limits_{\begin{subarray}{c}{m^{\prime}}={l^{\prime}}+1\end{subarray}}^{F}p^{(t)}_{\pi_{m^{\prime}}^{(t)}}|\mbox{{h}}_{i}^{(t)^{H}}{\mbox{{{w}}}}^{(t)}|^{2}+\sigma^{2}},\forall
1\leq{l^{\prime}}\leq F.$ (7)

###  2.3 Decoding Error Probability

Note that SIC is performed at the vehicles to obtain the required updates,
such that to successfully estimate the $\pi_{l^{\prime}}^{(t)}$-th message at
vehicle $i$, it has to first perform SIC to estimate and remove all the
messages $\pi_{m}^{(t)}$ $\forall 1\leq m\leq l^{\prime}-1$. Once these
messages are removed, vehicle $i$ can perform direct decoding by treating the
data of messages $\pi_{l^{\prime}+1},\pi_{l^{\prime}+2},\cdots F$ as
interference according to (7). Consequently, the $\pi_{l^{\prime}}^{(t)}$-th
message can be estimated at vehicle $i$ if

$\varepsilon_{i}(\gamma_{i,\pi_{m}^{(t)}}^{(t)}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)}))\\!\leq\varepsilon^{\mbox{\scriptsize
max}}_{i},~{}\forall~{}1\leq m\leq{l^{\prime}},$ (8)

where $\varepsilon^{\mbox{\scriptsize max}}_{i}$ is the maximum allowed error
probability at vehicle $i$ and  the decoding error probability can be
expressed as follows [36]:

$\begin{split}&\\!\\!\\!\varepsilon_{i}(\gamma_{i,\pi_{m}^{(t)}}^{(t)}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)}))=\\\
&\\!\\!\\!\Phi\left(\sqrt{\frac{\delta_{2}\omega}{\Gamma^{(t)}_{i,\pi_{m}^{(t)}}}}\left[\ln\left(1+\gamma_{i,\pi_{m}^{(t)}}^{(t)}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)})\right)-\frac{L\ln
2}{\delta_{2}\omega}\right]\right)\\!\\!,\end{split}$ (9)

where
$\Phi\left(q\right)\triangleq\frac{1}{\sqrt{2\pi}}\int_{q}^{\infty}\exp(-\frac{u^{2}}{2})du$,
$\Gamma^{(t)}_{i}\triangleq
1-\frac{1}{(1+\gamma_{i,\pi_{m}^{(t)}}^{(t)}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)}))^{2}}$
is the channel dispersion, $\gamma_{i,\pi_{m}^{(t)}}^{(t)}$ is the SINR at
vehicle $i$ at time slot $t$, $\omega$ is the bandwidth of the channel, and
$\delta_{2}\triangleq\delta-\delta_{1}$ is the information transmission time,
with $\delta_{1}$ as the dedicated time to acquire the vehicles’ angular
parameters (i.e., location and speed).  It is worth noting that according to
the SIC mechanism, for a given decoding order decision $\bm{\pi}^{(t)}$, a
vehicle can correctly estimate the $\pi_{l^{\prime}}^{(t)}$-th message if all
the previous messages $\pi_{m}^{(t)}$ $\forall 1\leq m\leq l^{\prime}-1$ have
been correctly estimated and removed regardless of whether this vehicle is
interested of previous $\pi_{m}^{(t)}$ $\forall 1\leq m\leq l^{\prime}-1$
messages or not.

###  2.4 Age of Information

The instantaneous AoI of the $\pi_{l^{\prime}}^{(t)}$-th physical process at
vehicle $i$ evolves according to

$\Delta_{i\pi_{l^{\prime}}}^{(t)}\\!(\bm{\pi}^{(t)}\\!,\\!\mbox{{p}}^{(t)}\\!)\\!=\\!\\!\begin{cases}\delta,&\mbox{if}~{}\mbox{\eqref{Condd}
is satisfied},\\\
\Delta_{i\pi_{l^{\prime}}}^{(t-1)}\\!+\\!\delta,&\mbox{otherwise}.^{\prime\prime}\end{cases}$
(10)

The time-average AoI of $f_{l}$ at vehicle $i$ over $T$ time slots is
$\bar{\Delta}_{{i,l}}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)})\triangleq\mathbb{E}_{T}\\!\\!\left[\Delta_{i,l}^{(t)}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)})\right]=$
$\frac{1}{T}\sum_{t=1}^{T}\Delta_{i,l}^{(t)}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)})$.
Consequently, the total time-average AoI can be expressed as follows:

$\begin{split}\bar{\Delta}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)})&=\sum_{i=1}^{V}\sum_{l=1}^{F}r_{i,l}\bar{\Delta}_{{i,l}}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)})\\\
&=\frac{1}{T}\sum_{i=1}^{V}\sum_{t=1}^{T}\sum_{l=1}^{F}r_{i,l}\Delta_{i,l}^{(t)}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)}).\end{split}$
(11)

Note that the maximum value of $\bar{\Delta}_{{i,l}}$ is $\delta(T+1)/2$,
which corresponds the case of no update about $f_{l}$ is received at vehicle
$i$ over the $T$ time slots. Thus, the maximum value (upper bound) of the
total time-average AoI $\bar{\Delta}^{\mbox{\scriptsize max}}$ can be
expressed as follows:

$\bar{\Delta}^{\mbox{\scriptsize
max}}=\frac{\delta(T+1)}{2}\sum_{i=1}^{V}\sum_{l=1}^{F}r_{i,l},$ (12)

which corresponds the case of no update is received by any vehicle during the
$T$ time slots. The minimum value (lower bound) of the total time-average AoI
$\bar{\Delta}^{\mbox{\scriptsize min}}$ corresponds the case of each vehicle
is able to decode its required messages in each time slot. Consequently,
$\bar{\Delta}^{\mbox{\scriptsize min}}$ can be expressed as:

$\bar{\Delta}^{\mbox{\scriptsize
min}}=\delta\sum_{i=1}^{V}\sum_{l=1}^{F}r_{i,l}.$ (13)

###  2.5 A Toy Example

Fig. 1 illustrates a schematic diagram of the considered system model with
$F=4$ processes and $V=4$ vehicles, the vehicles’ information demand matrix is

$\mathbf{R}=[r_{i,l}]_{\footnotesize{4\times
4}}=\blockarray{ccccc}f_{1}&f_{2}f_{3}f_{4}\\\ \block{[cccc]c}1100v_{1}\\\
0011v_{2}\\\ 0101v_{3}\\\ 1000v_{4}\\\ .$ (14)

The messages are superposed at the RSU according to a dummy decoding order
decision $\bm{\pi}^{(t)}=[1,2,3,4]$. Let us assume that the power is properly
allocated, the SIC at the vehicles is performed as follows. At a given vehicle
$v_{i}$, the message $f^{(t)}_{1}$ can be obtained by applying direct decoding
from the received signal while treating the other messages as interference. If
the condition in (8) is satisfied and $f^{(t)}_{1}$ is correctly decoded, the
AoI of process $f_{1}$ at vehicle $v_{i}$ will be updated if $r_{i,1}=1$
(i.e., process $f_{1}$ is of interest to $v_{i}$). The SIC procedure is
performed to get message $f^{(t)}_{2}$ if the previous message $f^{(t)}_{1}$
was correctly decoded, such that $f^{(t)}_{1}$ is removed and $f^{(t)}_{3}$
and $f^{(t)}_{4}$ are treated as interference, and so on. It is worth noting
that the iterations of the SIC at each vehicle depends on its requirements and
the decoding order decision $\bm{\pi}^{(t)}$. For example, in Fig. 1 vehicle
$v_{1}$ applies direct decoding to get $f^{(t)}_{1}$ and one SIC iteration to
get $f^{(t)}_{2}$, vehicles $v_{2}$ and $v_{3}$ apply direct decoding followed
by three SIC iterations to get $f^{(t)}_{4}$, and vehicle $v_{4}$ applies only
the direct decoding to get $f^{(t)}_{1}$.

It is also worth mentioning that, unlike the conventional NOMA scenarios where
the messages can be superposed according to the ranking of the communication
channel conditions, both the power allocation and the decoding order decision
$\bm{\pi}^{(t)}$ should be optimized to consider the vehicles demands and the
value of the AoI at each vehicle.

Figure 1: System model with $V=4$ vehicles, $F=4$ processes, information
demand $\mathbf{R}$ in (14), and a decoding order decision
$\bm{\pi}^{(t)}=[1,2,3,4]$.

##  3 Multi-objective Problem Statement

We consider minimizing the time-average AoI of each process at the vehicles
$\bar{\Delta}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)})$ as well as the time-average
power consumption at RSU,
$\bar{{p}}^{(t)}=\frac{1}{T}\sum_{t=1}^{T}\sum_{l=1}^{F}p^{(t)}_{l}$. The
multi-objective problem is thus formulated as follows:

P1 $\displaystyle\min_{\bm{\pi}^{(t)},\mbox{{p}}^{(t)}}$
$\displaystyle\left\\{\bar{\Delta}(\bm{\pi}^{(t)}\\!,\mbox{{p}}^{(t)}\\!),\bar{{p}}^{(t)}\right\\}$
(15a) s.t. $\displaystyle{\sum_{l=1}^{F}p_{l}^{(t)}\leq P^{\mbox{\scriptsize
max}}},$ (15b) $\displaystyle p_{l}^{(t)}\geq 0,~{}\forall~{}1\leq l\leq F,$
(15c) $\displaystyle\pi_{l^{\prime}}^{(t)}\in\mathcal{F},~{}\forall~{}1\leq
l^{\prime}\leq F,$ (15d) $\displaystyle\mbox{card}(\bm{\pi}^{(t)})\leq F.$
(15e)

Constraints (15b) and (15c) guarantee that the allocated power is positive and
less than the maximum transmission power of the RSU. Constraint (15d)
guarantees that the decoding order decision captures only the physical
processes and (15e) guarantees there is no repeated processes, with
$\mbox{card}(\cdot)$ as a cardinality operator.

Keeping in mind the trade-off between these two objectives and the fact that
they have different units, ranges, and orders of magnitude, they should be
normalized such that they have similar ranges [12]. The common strategy to
handle multi-objective optimization is to convert the problem into a single
objective optimization problem, which can be implemented using three
approaches. The first approach referred to as $\epsilon$-constraint approach,
in which one objective is selected to be the primary objective and the others
appear as constraints with respect to the auxiliary parameter $\epsilon$ [9].
This approach has three major shortcomings: (i) The value of the auxiliary
parameter $\epsilon$ should be carefully adjusted as it may affect the problem
feasibility; (ii) The selection of the primary objective function; (iii) More
than one auxiliary parameter may be required to constrain objectives of
different ranges. The second approach is Tchebycheff approach, in which the
primary objective is to optimize (maximize or minimize) an auxiliary parameter
and all objectives appear as weighted constraints with respect to the
auxiliary parameter [10]. The third approach is the weighted sum, in which the
objectives are combined into a single function using prefixed weights [11, 12,
13].

The weighted-sum is an effective approach to handle a multi-objective
optimization problem, where each weight corresponds to a particular sub-
problem. The solutions of $J$ sub-problems constitute a set of the Pareto
optimal solutions or the Pareto optimal front [37].

Definition 1. Let $\mathcal{X}$ represents the feasible space of the
optimization problem

$\min_{\mbox{{x}}}\mbox{{o}}(\mbox{{x}})=\left\\{o_{1}(\mbox{{x}}),o_{2}(\mbox{{x}}),\cdots,o_{K}(\mbox{{x}})\right\\}~{}~{}\mbox{subject
to: }~{}\mbox{{x}}\in\mathcal{X}$

A point $\mbox{{x}}^{*}\in\mathcal{X}$ is a Pareto optimal point if and only
if there does not exist another point, $\mbox{{x}}\in\mathcal{X}$, such that
$\mbox{{o}}(\mbox{{x}})\leq\mbox{{o}}(\mbox{{x}}^{*})$, and
$o_{k}(\mbox{{x}})<o_{k}(\mbox{{x}}^{*})$ for at least one objective function.
The set of all Pareto optimal points is called the Pareto frontier.

To obtain the Pareto optimal fronts for (15), we consider the normalized
weighted metric method, which entails minimizing the difference between the
objectives and the corresponding utopia solutions. Fortunately, we can obtain
both the Utopia and Nadir solutions of the two objectives. The Utopia and
Nadir solutions of time-average AoI are $\bar{\Delta}^{\mbox{\scriptsize
max}}$ and $\bar{\Delta}^{\mbox{\scriptsize min}}$ as expressed in (12) and
(13), respectively. While for the time-average power consumption, the Nadir
and Utopia solutions are the maximum transmission power of the RSU
$P^{\mbox{\scriptsize max}}$ and the minimum transmission power of the RSU
($P^{\mbox{\scriptsize min}}=0$), respectively. Consequently, we define the
objective function as:

$\begin{split}{O}(\bm{\pi}^{(t)}\\!,\mbox{{p}}^{(t)})\\!=&\zeta\frac{\bar{\Delta}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)})\\!-\\!\bar{\Delta}^{\mbox{\scriptsize
min}}}{\bar{\Delta}^{\mbox{\scriptsize max}}-\bar{\Delta}^{\mbox{\scriptsize
min}}}\\!+\\!(1\\!-\\!\zeta)\frac{\bar{{p}}^{(t)}\\!-\\!P^{\mbox{\scriptsize
min}}}{P^{\mbox{\scriptsize max}}\\!-\\!P^{\mbox{\scriptsize
min}}},\end{split}$ (16)

where $0\leq\zeta\leq 1$ is a relative weight to give preference to minimize
the AoI or the power. It is worth noting that the objective function in (16)
is dimensionless, bounded by $[0,1]$, and the decision-maker can set the
objective-preference weight to any desired value $0\leq\zeta\leq 1$. This is
not the case in other multi-objective optimization methods, such as
$\epsilon$-constraint method, in which the value of $\epsilon$ should be
selected carefully to avoid infeasibility issues.

Henceforth, the multi-objective optimization problem P1 can be transformed as
follows:

P2 $\displaystyle\min_{\bm{\pi}^{(t)},\mbox{{p}}^{(t)}}$
$\displaystyle{O}(\bm{\pi}^{(t)}\\!,\mbox{{p}}^{(t)})$ (17a) s.t.
$\displaystyle\eqref{con11}-\eqref{con4}.$ (17b)

The optimization problem in (17) is a mixed-integer non-linear programming
(MINLP) problem that involves discrete (decoding order decision
$\bm{\pi}^{(t)}$) and continuous (power allocation $\mbox{{p}}^{(t)}$)
decision making over multiple transmission time slots. Conventional
optimization methods are not appropriate for solving the problem (17) which
evolves over time, hence we resort to the DRL strategies for solving problem
(17).

##  4 Hybrid DQN-DDPG -Based DRL Solution

In this section, a hybrid DQN-DDPG DRL model is introduced to solve the
optimization problem in (17) for a given value of the objective-preference
weight $\zeta$.

###  4.1 Theoretical Preliminaries

Reinforcement learning is the process of learning by an agent to maximize the
discounted reward over the learning time horizon by interacting with an
environment. At each learning epoch, given the current state of the
environment, an action is performed by the agent on the environment which
transits to the next state and returns an immediate reward. A key metric in
training the agent is the $Q$-function which estimates the future reward of
taking an action $\bm{a}$ at a given state $\bm{s}$. It has been shown that
learning based on $Q$-function (also referred to as $Q$-learning) converges
towards an optimal solution after visiting each state-action pair with
sufficient number of learning iterations [38]. Such a learning approach is
impractical in the following scenarios: (1) The number of state-action pairs
is very large; (2) The action space and/or state space are/is continuous.
Utilizing deep neural networks (DNN) to approximate the $Q$-function is
referred to as DRL, in which the $Q$-function is written as
$Q(\bm{s},\bm{a}\mid\bm{\theta})$, where $\bm{\theta}$ represents the weight
vector of the DNN [39]. Actor-critic networks comprising of two deep
$Q$-learning networks are able to deal with environments that have continuous
action spaces [40]. DDPG is an actor-critic algorithm involves two networks,
namely, the actor and critic networks. The actor network learns to obtain the
best action at a given state, while the critic evaluates the reward of the
state-action pair [40].

It is worth mentioning that the DRL model is commonly formulated as a Markov
Decision Process (MDP) problem. An MDP is represented by a tuple
$\\{\mathcal{S},\mathcal{A},\vartheta,\rho\\}$, where $\mathcal{S}$ is the
state space that consists of the set of all possible states, $\mathcal{A}$ is
a finite set of actions from which the agent can choose,
$\vartheta\mathrel{\mathop{\mathchar
58\relax}}\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$ is a
transition probability which defines the probability of observing a state
after executing an action at a given environment’s state, and
$\rho\mathrel{\mathop{\mathchar
58\relax}}\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the expected
reward of performing an action at a given state. We note that discretizing
continuous action space increases the problem’s dimensionality. Thus, DDPG
[40] is applied to handle continuous action space. The DDPG algorithm
maintains a parameterized actor policy, which maps states to a probability
distribution over the actions [40]. In this context, a hybrid DQN-DDPG DRL
model is developed such that the DQN handles the decoding order decision
(discrete value) and the actor-critic DDPG handles the power allocation
(continuous value) decision. The following section defines the state space,
action space, and reward of the designed DRL model.

###  4.2 DRL Model

Our goal is to design a DRL system that jointly optimizes the decoding order
decision (discrete value) and the power allocation (continuous value) decision
to minimize the objective function in (17). In this context, we develop a DRL
model that involves the definition of the environment state, the action, and
the immediate reward function $\rho^{(t)}$ as follows:

####  4.2.1 The Environment State Space

The state space is denoted as $\mathcal{S}$, in which the state at time slot
$t$ captures the communication channel between the RSU and the vehicles as
well as the AoI of the required processes at the vehicles. Thus, the state at
time slot $t$ is given by

$\bm{s}^{(t)}=\left\\{\\{\sum_{i=1}^{V}r_{i,l}\chi^{(t)}_{i}\\},\\{\sum_{i=1}^{V}r_{i,l}\Delta_{i,l}^{(t)}\\}\right\\}_{l=1}^{F}.$
(18)

####  4.2.2 Action Space

At each decision-making instant, based on the observed state, the DQN and
actor-critic models make an action to order the physical process for encoding
and to allocate the power, respectively. Under this setup, the action space
$\mathcal{A}$ represents pairs of actions
$\mbox{{a}}^{(t)}=(\mbox{{a}}_{\bm{\pi}}^{(t)},\mbox{{a}}_{\bm{p}}^{(t)})$,
such that

  * •

$\mbox{{a}}_{\bm{\pi}}^{(t)}=[\pi^{(t)}_{1},\pi^{(t)}_{2},\cdots,\pi^{(t)}_{F}]$
is the decoding order decision which is obtained using the DQN agent.

  * •

$\mbox{{a}}_{\bm{p}}^{(t)}=[\alpha_{1}^{(t)},\alpha_{2}^{(t)},\cdot,\alpha_{F}^{(t)}]P^{max}$
is the power allocation decision, with
$\bm{\alpha}^{(t)}=[\alpha_{1}^{(t)},\alpha_{2}^{(t)},\cdot,\alpha_{F}^{(t)}]$
as the output of the actor-critic/DDPG agent.

####  4.2.3 Reward

Based on the observed state and the agents’ action, an immediate reward is
returned by the environment that reflects the suitability of the action to
minimize the objective function with a given value of the objective preference
weight. For the considered optimization problem, a good decision on process
decoding order and power allocation can minimize the AoI and power consumption
with the given preference weight $\zeta$. To reflect the quality of the action
taken by the agents, the immediate reward at time slot $t$ is expressed as:

$\begin{split}\rho^{(t)}\\!\\!=&\exp\left(-\zeta\frac{\bar{\Delta}_{t}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)})\\!-\\!\bar{\Delta}^{\mbox{\scriptsize
min}}}{\bar{\Delta}_{t}^{\mbox{\scriptsize
max}}-\bar{\Delta}^{\mbox{\scriptsize
min}}}\\!-\\!(1\\!-\zeta)\frac{\hat{p}^{(t)}-P^{\mbox{\scriptsize
min}}}{P^{\mbox{\scriptsize max}}-P^{\mbox{\scriptsize min}}}\right)\\\
&-\Upsilon^{(t)},\end{split}$ (19)

where $\Upsilon^{(t)}=\kappa[\sum_{l=1}^{F}\alpha_{l}^{(t)}-1]^{+}$ is a
penalty function with $[x]^{+}=\max\\{x,0\\}$ and $\kappa$ as a penalty
constant,
$\hat{p}^{(t)}=\frac{1}{t}\sum_{t^{\prime}=1}^{t}\sum_{l=1}^{F}{p_{l}}^{(t^{\prime})}$,
$\bar{\Delta}_{t}(\bm{\pi}^{(t)},\mbox{{p}}^{(t)})$, and
$\bar{\Delta}_{t}^{\mbox{\scriptsize max}}$, are obtained by replacing $T$ by
$t$ in (11) and (12), respectively. It is worth noting that the parameters in
equation (16) are redefined in (19) to represent the AoI and power consumption
at time instant $t$. Keeping in mind that our aim is to minimize the objective
function, while the DRL learns to maximize the accumulative reward, the
exponential function is chosen to define the immediate reward which stabilizes
the offline training [5]. Finally, $\Upsilon^{(t)}$ represents a penalty of
violating the constraint (15b).

###  4.3 Hybrid DQN-DDPG Algorithm

A schematic block diagram of the proposed hybrid DQN-DDPG DRL model is shown
in Fig. 2, which illustrates the action-making agents for decoding order and
power allocation as well as the learning process for both agents for a given
value of the objective-preference weight $\zeta$. The training algorithm of
the hybrid DQN-DDPG model is illustrated in Algorithm 1. The objective-
preference weight $\zeta$, and DQN weights $\theta^{Q}$, critic network
weights $\theta^{Q_{c}}$, actor network weights $\theta^{\mu}$ are the input
of the algorithm. For each training episode, the environment is initialized
and an initial state is obtained (line 4). At every time step $t$, the DQN
model obtains a decoding order action $\bm{a}_{\bm{\pi}}^{(t)}$ using the
$\bar{\epsilon}$-greedy policy to balance the exploitation of known actions
and the exploration of new actions (line 6). The DDPG model obtains a power
allocation action $\bm{a}_{\bm{P}}^{(t)}$ using the actor network
$\mu(\bm{s}^{(t)}\mid\theta^{\mu})$ (lines 7-8). A combined action
$\bm{a}^{(t)}=(\bm{a}_{\bm{\pi}}^{(t)},\bm{a}_{\bm{P}}^{(t)})$ is applied on
the environment which returns an immediate reward $\rho^{(t)}$ and transits to
a new state $\bm{s}^{({t+1})}$ (line 9). The transition tuple
$\\{\bm{s}^{({t})},\bm{a}^{(t)},\rho^{(t)},\bm{s}^{({t+1})}\\}$ is added to
the replay buffer B (line 10). A mini-batch of transitions are sampled from
the replay buffer to update the deep networks (line 12).

Figure 2: Hybrid DQN-DDPG model for age-optimum information dissemination.
Algorithm 1 Hybrid DQN-DDPG training algorithm for age-optimum information
dissemination.

1:Input: Objective-preference weight $\zeta$, No. of episodes, DQL network
weights $\theta^{Q}$, critic network weights $\theta^{Q_{c}}$, and actor
network weights $\theta^{\mu}$;

2:Initialize target network $\theta^{Q^{\prime}}\leftarrow\theta^{Q}$,
$\theta^{Q^{\prime}_{c}}\leftarrow\theta^{Q_{c}}$, and
$\theta^{\mu^{\prime}}\leftarrow\theta^{\mu}$;

3:For $episode=1$ to No. of episodes do

4:Initialize the environment and receive the initial state $\bm{s}^{(1)}$;

5: Repeat:

6: With probability $\bar{\epsilon}$, select randomly an action
$\bm{a}^{(t)}_{\bm{\pi}}\in\mathcal{A}_{\bm{\pi}}$; Otherwise, select
$\bm{a}_{\bm{\pi}}^{(t)}=\arg\max\limits_{\forall\bm{a}_{\bm{\pi}}^{(t)}\in\bm{\mathcal{A}}_{\bm{\pi}}}Q(\bm{s}^{(t)},\bm{a}^{(t)}_{\bm{\pi}}\mid\bm{\theta}^{Q})$;

7: Obtain
$\bm{\alpha}^{(t)}=[\alpha_{1}^{(t)},\alpha_{2}^{(t)},\cdots,\alpha_{F}^{(t)}]\triangleq\mu(\bm{s}^{(t)}\mid\theta^{\mu}){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}+N}$,
where $N$ is the exploration noise [40];

8: Obtain the power allocation action
$\bm{a}_{\bm{P}}^{(t)}=\bm{\alpha}^{(t)}P^{max}$;

9: Observe the reward $\rho^{(t)}$ using (19) and next state
$\bm{s}^{({t+1})}$ by applying the action
$\bm{a}^{(t)}=(\bm{a}_{\bm{\pi}}^{(t)},\bm{a}_{\bm{P}}^{(t)})$;

10: Store the transition
$\\{\bm{s}^{({t})},\bm{a}^{(t)},\rho^{(t)},\bm{s}^{({t+1})}\\}$ in B; $t=t+1$;

11: Until terminal state $t=T$;

12: Sample a mini-batch of $M$ transitions from B;

13: Update the weight $\theta^{Q}$, $\theta^{Q_{c}}$ and $\theta^{\mu}$ by
minimizing the corresponding loss functions in (20);

14:Update the target networks
$\begin{split}&\theta^{Q^{\prime}}\leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}},\\\
&\theta^{Q^{\prime}_{c}}\leftarrow\tau\theta^{Q_{c}}+(1-\tau)\theta^{Q^{\prime}_{c}},\\\
&\theta^{\mu^{\prime}}\leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}}.\end{split}$

15:End for

The Adam optimizer [41] is considered to update the weights of the DNNs with
an aim to minimize the following loss functions

$\begin{split}&L({\theta}^{Q})\\!=\\!\sum_{\iota}(\rho^{(\iota)}\\!+\\!\\!\max_{\forall\bm{a}_{\bm{\pi}}^{{}^{\prime}}\in\mathcal{A}}\\!Q^{\prime}(\bm{s}^{{}^{\prime}},\bm{a}_{\bm{\pi}}^{{}^{\prime}},{\theta}^{Q^{\prime}})\\!-\\!Q(\bm{s}^{(\iota)},\bm{a}_{\bm{\pi}}^{(\iota)},{\theta}^{Q}))^{2},\\\
&L({\theta}^{Q_{c}})=\sum_{\iota}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}y^{(\iota)}}-Q(\bm{s}^{(\iota)},\bm{a}_{\bm{P}}^{(\iota)},{\theta}^{Q_{c}}))^{2},\\\
&L({\theta}^{\mu})=-Q(\bm{s}^{(\iota)},\bm{a}_{\bm{P}}^{(\iota)},{\theta}^{Q_{c}},{\theta}^{\mu}),\end{split}$
(20)

where
$y^{(\iota)}=\rho^{(\iota)}+Q_{c}^{\prime}(\bm{s}^{{}^{\prime}},\bm{a}_{\bm{P}}^{{}^{\prime}},{\theta}^{Q_{c}^{\prime}})$
[40].

It is worth noting that both the DQN and DDPG agents work jointly such that
they observe the same observation or environment state and the shared reward
is not sparse and its value depends on the actions of both agents, which
prevents a lazy agent situation. In other words, let us assume that the DQN
model tends to become a lazy agent. In such a case, the reward of DQN model
will decrease and the DDPG will not be able to compensate this behavior, i.e.,
cannot force the DQN model to learn a good policy. The same applies if the
DDPG model tends to become a lazy agent, then it will not take advantage of
the successful actions of DQN. That is, if the action of the DDPG is bad the
reward will decrease regardless of how good is the action of DQN model. Such a
joint effect of the actions of the two agents on the value of the reward and
the shared environment observation prevent both agents from becoming a lazy
agent, which may arise due to partial observability [42, 43] or sparse reward
scenarios [44].”

##  5 Meta Reinforcement Learning

In Section 4, a hybrid DQN-DDPG DRL model has been developed to solve the age-
optimum information dissemination in vehicular networks problem. However, it
is worth noting that a predetermined objective-preference weight is required
before training the model. Consequently, the quality of the inferred solution
depends on whether the objective-preference weight has been observed during
the training stage. If an unseen objective-preference weight is encountered, a
new model should be trained from scratch which is inefficient approach to
construct the Pareto front as it requires a large number of models to be
trained and stored. Inspired by recently proposed algorithms for fast
adaptation in DRL models[14, 15], this section introduces a meta reinforcement
learning approach to increase the diversity and quality of the solutions.  The
training of the meta RL is performed based on the multi step model-agnostic
meta-learning (MAML) approach [45], which consists two nested stages. The
inner stage performs multiple gradient descent steps to update the deep
notworks parameters for a given value of the relative weight, while the outer
stage enables the updating of the deep notworks’ parameters over all the
sampled values of relative weight.

###  5.1 Meta Reinforcement Algorithm

Training the meta-DRL model is an important part of the meta-based algorithm
as it yields the parameters for the neural network that can adapt quickly to a
new task. The goal of meta-learning is to ensure that the meta-based model is
capable of optimizing the objective function with any objective-preference
weight after a small number of fine-tuning updates.  Suppose a set $J$
tasks333Each task is a scalar optimization sub-problem with an objective-
preference weight are available for learning and the corresponding objective-
preference weights are sampled based on a probability distribution
$\bm{\Lambda}$, the $j$-th task is associated with loss functions
$L_{j}(\tilde{\theta}^{Q})$, $L_{j}(\tilde{\theta}^{Q_{c}})$, and
$L_{j}(\tilde{\theta}^{\mu})$ parameterized by the DQL network weights
$\tilde{\theta}^{Q}$, critic network weights $\tilde{\theta}^{Q_{c}}$, and
actor network weights $\tilde{\theta}^{\mu}$, respectively. MAML approach
guarantees convergence [45], which aims at finding good initial parameters
($\tilde{\theta}^{Q^{*}}$, $\tilde{\theta}^{Q_{c}^{*}}$,
$\tilde{\theta}^{\mu^{*}}$) such that after observing a new task, a few
gradient descend steps (fine-tuning steps) starting from such initial
parameters can efficiently approach the optimizer of the corresponding loss
functions. The multi-step MAML consists of two nested stages, the inner stage
performs multiple gradient descent steps for each individual task, while the
meta parameters over all the sampled tasks are updated using the outer stage.
These two nested stages provide diverse and representative training, which
mitigates the bias in the training [45]. Consequently, the inner stage of each
task initializes at the meta parameters, (i.e.,
$\tilde{\tilde{\theta}}^{Q}_{j_{0}}=\tilde{{\theta}}^{Q}$,
$\tilde{\tilde{\theta}}^{Q_{c}}_{j_{0}}={\tilde{\theta}}^{Q_{c}}$,
$\tilde{\tilde{\theta}}^{\mu}_{j_{0}}={\tilde{\theta}}^{\mu}$) and runs $M$
gradient descent steps as

$\displaystyle\tilde{\tilde{\theta}}^{Q}_{j_{k+1}}=\tilde{\tilde{\theta}}^{Q}_{j_{k}}-\alpha_{Q}\nabla
L_{j}(\tilde{\tilde{\theta}}^{Q}_{j_{k}}),$ (21)
$\displaystyle\tilde{\tilde{\theta}}^{Q_{c}}_{j_{k+1}}=\tilde{\tilde{\theta}}^{Q_{c}}_{j_{k}}-\alpha_{Q_{c}}\nabla
L_{j}(\tilde{\tilde{\theta}}^{Q_{c}}_{j_{k}}),$
$\displaystyle\tilde{\tilde{\theta}}^{\mu}_{j_{k+1}}=\tilde{\tilde{\theta}}^{\mu}_{j_{k}}-\alpha_{\mu}\nabla
L_{j}(\tilde{\tilde{\theta}}^{\mu}_{j_{k}}),$

where $\alpha_{Q}$, $\alpha_{Q_{c}}$, and $\alpha_{\mu}$ represent the inner
step size of the DQL, critic, and actor networks, respectively. Consequently,
the overall meta goal is given by

$\displaystyle\min_{\tilde{{\theta}}^{Q}}\mathcal{L}(\tilde{{\theta}}^{Q})\coloneqq\mathbb{E}_{j\sim\bm{\Lambda}}\left[\mathcal{L}_{j}(\tilde{{\theta}}^{Q})\right]\coloneqq\mathbb{E}_{j\sim\bm{\Lambda}}\left[L_{j}(\tilde{\tilde{\theta}}^{Q}_{j_{M}}(\tilde{\theta}^{Q}))\right],$
(22)
$\displaystyle\min_{\tilde{{\theta}}^{Q_{c}}}\mathcal{L}(\tilde{{\theta}}^{Q_{c}})\coloneqq\mathbb{E}_{j\sim\bm{\Lambda}}\left[\mathcal{L}_{j}(\tilde{{\theta}}^{Q_{c}})\right]\coloneqq\mathbb{E}_{j\sim\bm{\Lambda}}\left[L_{j}(\tilde{\tilde{\theta}}^{Q_{c}}_{j_{M}}(\tilde{\theta}^{Q_{c}}))\right],$
$\displaystyle\min_{\tilde{{\theta}}^{\mu}}\mathcal{L}(\tilde{{\theta}}^{\mu})\coloneqq\mathbb{E}_{j\sim\bm{\Lambda}}\left[\mathcal{L}_{j}(\tilde{{\theta}}^{\mu})\right]\coloneqq\mathbb{E}_{j\sim\bm{\Lambda}}\left[L_{j}(\tilde{\tilde{\theta}}^{\mu}_{j_{M}}(\tilde{\theta}^{\mu}))\right].$

Then the outer stage of meta update is gradient decent steps to minimize (22).
In [45], a simplified form of gradient of the losses has been derived using
the chain rule, which can be expressed as

$\displaystyle\nabla\mathcal{L}_{j}(\tilde{{\theta}}^{Q})=\left[\prod_{k=0}^{M-1}(1-\alpha_{Q}\nabla^{2}L_{j}(\tilde{\tilde{\theta}}^{Q}_{j_{k}}))\right]\nabla
L_{j}(\tilde{\tilde{\theta}}^{Q}_{j_{M}}),$ (23)
$\displaystyle\nabla\mathcal{L}_{j}(\tilde{{\theta}}^{Q_{c}})=\left[\prod_{k=0}^{M-1}(1-\alpha_{{Q_{c}}}\nabla^{2}L_{j}(\tilde{\tilde{\theta}}^{{Q_{c}}}_{j_{k}}))\right]\nabla
L_{j}(\tilde{\tilde{\theta}}^{{Q_{c}}}_{j_{M}}),$
$\displaystyle\nabla\mathcal{L}_{j}(\tilde{{\theta}}^{\mu})=\left[\prod_{k=0}^{M-1}(1-\alpha_{\mu}\nabla^{2}L_{j}(\tilde{\tilde{\theta}}^{\mu}_{j_{k}}))\right]\nabla
L_{j}(\tilde{\tilde{\theta}}^{\mu}_{j_{M}}),$

where $\nabla L_{j}(\cdot)$ and $\nabla^{2}L_{j}(\cdot)$ represent the
gradient and Hessian operators of the loss function $L_{j}(\cdot)$,
respectively. Finally, the full gradient descent step of the outer stage can
be expressed as

$\displaystyle\tilde{{\theta}}^{Q}_{{k^{\prime}+1}}=\tilde{{\theta}}^{Q}_{{k^{\prime}}}-\beta_{Q}\mathbb{E}_{j\sim\bm{\Lambda}}\nabla\mathcal{L}_{j,k^{\prime}}(\tilde{{\theta}}^{Q}),$
(24)
$\displaystyle\tilde{{\theta}}^{Q_{c}}_{{k^{\prime}+1}}=\tilde{{\theta}}^{Q_{c}}_{{k^{\prime}}}-\beta_{{Q_{c}}}\mathbb{E}_{j\sim\bm{\Lambda}}\nabla\mathcal{L}_{j,k^{\prime}}(\tilde{{\theta}}^{Q_{c}}),$
$\displaystyle\tilde{{\theta}}^{\mu}_{{k^{\prime}+1}}=\tilde{{\theta}}^{\mu}_{{k^{\prime}}}-\beta_{\mu}\mathbb{E}_{j\sim\bm{\Lambda}}\nabla\mathcal{L}_{j,k^{\prime}}(\tilde{{\theta}}^{\mu}),$

where $\beta_{Q}$, $\beta_{Q_{c}}$, and $\beta_{\mu}$ represent the outer step
size of the DQL, critic, and actor networks, respectively.

Algorithm 2 The meta-learning algorithm for age-optimum information
dissemination.

1:Input: Distribution over the objective-preference weight $\bm{\Lambda}$,
number of meta-learning iterations, number of sampled tasks $J$;

2:Initialize the DQL network weights $\tilde{\theta}^{Q}$, critic network
weights $\tilde{\theta}^{Q_{c}}$, and actor network weights
$\tilde{\theta}^{\mu}$;

3:For $k^{\prime}=0$ to No. of meta-learning iterations do

4:Sample $J$ tasks by distribution $\bm{\Lambda}$;

5: For $j=1$ to $J$ do

6: For $k=0$ to $M$ do

7: Obtain a training set of transition tuples;

8: Update the DRL networks parameters using (LABEL:inner_grad);

9: End for

10: End for

11:Update the DRL networks parameters using (24);

12:End for

13:Output $\tilde{\theta}^{Q^{*}}\longleftarrow\tilde{\theta}^{Q}$;
$\tilde{\theta}^{Q_{c}^{*}}\longleftarrow\tilde{\theta}^{Q_{c}}$;
$\tilde{\theta}^{\mu^{*}}\longleftarrow\tilde{\theta}^{\mu^{*}}$.

Algorithm 3 Fine-tuning algorithm to obtain the Pareto front using the meta-
model.

1:Input: The objective-preference weight vector of the Pareto front
$\bm{\zeta}=[\zeta_{1},\zeta_{2},\cdots,\zeta_{\hat{J}}]$, number of fine-
tuning iterations, well-trained meta-model DQL network weights
$\tilde{\theta}^{Q^{*}}$, well-trained meta-model critic network weights
$\tilde{\theta}^{Q_{c}^{*}}$, and well-trained meta-model actor network
weights $\tilde{\theta}^{\mu^{*}}$;

2: For $j=1$ to $\hat{J}$ do

3: $\theta^{Q}_{j},\theta^{Q_{c}}_{j},\theta^{\mu}_{j}$ $\longleftarrow$
$\tilde{\theta}^{Q^{*}},\tilde{\theta}^{Q_{c}^{*}},\tilde{\theta}^{\mu^{*}}$;

4: For $episode=1$ to No. of fine-tuning steps do

5: $(\theta^{Q}_{j},\theta^{Q_{c}}_{j},\theta^{\mu}_{j})$ $\longleftarrow$
Hybrid DQN-
DDPG$(\zeta_{j},\theta^{Q}_{j},\theta^{Q_{c}}_{j},\theta^{\mu}_{j})$ using
Algorithm 1;

6: End for

7: Estimate the $j$-th Pareto point using
$\theta^{Q}_{j},\theta^{Q_{c}}_{j},\theta^{\mu}_{j}$, and $\zeta_{j}$;

8: End for

9:Output the Pareto points.

The meta-learning algorithm is illustrated in Algorithm 2, in which at the
$k$-th inner stage iteration a training set of transition tuples is sampled
and utilized to update the meta parameters using (LABEL:inner_grad). At the
$k^{\prime}$-th outer stage iteration, two training sets of transition tuples
are independently sampled and utilized to estimate the gradient and Hessian of
the loss and update meta parameters using (24).

###  5.2 Fine-tuning and Inference

In the meta-training stage, we have learned the initial networks parameters,
which have good generalization ability. Pareto solution for any objective-
preference weight can be inferred from the well-trained meta-model after
undergoing a few fine-tuning steps. Algorithm 3 illustrates how to infer a
Pareto frontier of $\hat{J}$ Pareto points corresponding to a set of
objective-preference weights
$\bm{\zeta}=[\zeta_{1},\zeta_{2},\cdots,\zeta_{\hat{J}}]$.

##  6 Simulation Results and Discussions

This section introduces simulation results to evaluate the proposed framework
and solution approaches.

###  6.1 Simulation Parameters

Without loss of generality, the considered vehicular network consists of a
segment of a two-lane road along the x-axis and the vehicles move along the
positive or negative directions of the x-axis. The length of the road segment
is 3 km and the width of each lane is 3 m. The RSU is located at
$\\{1500,50\\}$ m and the vehicles’ are initialized randomly on the road in
both directions. The vehicles’ speeds are randomly drawn from $U(10,15)$ m/s.
The initial value of the instantaneous AoI of each process of interest at each
vehicle is $\delta$. Unless otherwise stated, the considered values of the
system parameters are listed in Table III.

TABLE III: Simulation Parameters.

Parameter Value Parameter Value Parameter Value $P^{\mbox{{\footnotesize
max}}}$ $0$ dB $c_{i}$ $\sim U(10,15)$ m/s [32] $\omega$ $10$ MHz [32] $f_{c}$
$3$ GHz [32] $N$ $64$ [32] $L$ $128$ byte $\varepsilon^{\mbox{{\footnotesize
max}}}$ $10^{-}6$ $c_{0}$ $2.99\times 10^{8}$ m/s $\sigma^{2}$ $0.1$

The implementation of the DQN and DDPG networks involve three hidden layers of
512, 256, and 128 neurons, respectively. Each layer is followed with a
rectified linear unit (ReLU) activation function. The learning rates of the
DQN, actor, and critic networks are $0.001$, $0.0001$, and $0.001$,
respectively. The mini-batch size for training the hybrid DQN-DDPG model is
$64$, the discount factor is $0.5$, and the penalty constant is $1$. The
number of training episodes is $1000$. For the meta training, the inner step
sizes of the DQL, critic, and actor networks are $\alpha_{Q}=0.1$,
$\alpha_{Q_{c}}=0.01$, and $\alpha_{\mu}=0.001$ while the outer step sizes are
$\beta_{Q}=0.01$, $\beta_{Q_{c}}=0.01$, and $\beta_{\mu}=0.001$, respectively.
The number of meta learning episodes is $500$ and the number of gradient
descent steps and fine-tuning steps is $50$. Fig. 3 shows the learning curves
for the hybrid DQN-DDPG DRL model versus the number of training episodes. It
can be noticed that the reward gradually increases, and its average value
saturates after sufficient training episodes without a major drop in the
average reward. This indicates that the adopted training parameters provide
stable learning without over-fitting.

Figure 3: The learning curves for the hybrid DQN-DDPG DRL model versus the
number of training episodes.

###  6.2 Results

We compare the performance of the following approaches:

  * •

Hybrid DQN-DDPG DRL which represents the results of the proposed DRL algorithm
described in Algorithm 1 that requires a value of the object-preference weight
$\zeta$.

  * •

The meta-DRL which represents the results of the proposed meta-DRL algorithm
described in Algorithm 2, without the fine-tuning step.

  * •

The meta-DRL with fine-tuning which represents the results of the proposed
meta-DRL algorithm described in Section 5.2 with the fine-tuning stage in
Algorithm 3.

  * •

The random solution in which the decoding order and power allocation decisions
are randomly selected.

  * •

The exhaustive search solution in which the power allocation decision is
discretized into $10$ levels and all the combinations of the decoding order
and the discretized power allocation decisions are examined.

Figure 4 illustrates the objective function versus the objective-preference
weight $\zeta$ for the proposed framework obtained using the four solution
approaches. It is seen that the hybrid DQN-DDPG approach achieves near-optimum
performance in comparison with the exhaustive search solution and the random
approach provides the worst performance. The performance of the meta-based DRL
is better than that of the random solution and the fine-tuning stage improves
the performance of the meta-based model and provides close to optimum
solutions. It is worth mentioning that a hybrid DQN-DDPG model is trained for
each value of $\zeta$ in Fig. 4.

Figure 4: Objective function in (16) versus the relative weight $\zeta$ with
$V=10$ vehicles, $F=4$ processes and
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}=2$.

To get more insight into this result, Fig. 5 illustrates the corresponding
average AoI and power consumption versus $\zeta$. It is noticed that the
proposed framework provides a good trade-off between AoI and power expenditure
as for low values of $\zeta$ it minimizes the power expenditure and as $\zeta$
increases it minimizes the AoI. The trade-off provided by the hybrid DQN-DDPG
is close to that of the exhaustive search and the meta-DRL with fine-tuning
provides a remarkably good performance. That is not the case for the random
solution, in which both the AoI and power expenditure are not function of the
relative weight. It is worth noting the meta-DRL solution also not a function
of $\zeta$ and its performance is better than the random solution, as it is
trained to minimize the objective function for randomly sampled objective-
preference weights.

Figure 5: Average AoI and power expenditure versus the relative weight $\zeta$
with $V=10$ vehicles, processes $F=4$ processes and
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}=2$. Figure 6: Pareto
fronts with $V=10$ vehicles, $F=4$ processes,
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}=2$, and
$\zeta=0\mathrel{\mathop{\mathchar 58\relax}}0.1\mathrel{\mathop{\mathchar
58\relax}}1$. Figure 7: Total training steps to obtain the Pareto front.

Figure 6 illustrates the Pareto fronts obtained by the four solution
approaches. It can be noticed that the hybrid DQN-DDPG and the meta-DRL with
fine-tuning approaches provide non-dominated solutions constitute diverse and
evenly spread of Pareto fronts, which gives the decision-maker a set of
satisfactory trade-off solutions. That is not the case for the random and
meta-DRL approaches where the Pareto points are concentrated in close-
proximity apart from the lower left part of the figure. To quantify the
quality of the Pareto fronts in Fig. 6, the hypervolume indicator (Lebesgue
measure) of each Pareto front is calculated using Monte-Carlo approximation.
The hypervolume indicator maps the set of points in a Pareto front to a single
real value measure that represents the region dominated by that Pareto front
and bounded above (for minimization problems) by a given reference point [46].
The higher the value of the hypervolume indicator the better the Pareto front
to give the decision-maker satisfactory trade-off solutions. Assuming the
upper bound of the two objectives as a reference point i.e., (average AoI =
$\bar{\Delta}^{\mbox{\scriptsize max}}$, average power = $P^{\mbox{\scriptsize
max}}$), the hypervolume indicators of the random, meta-DRL, meta-DRL with
fine-tuning, and hybrid DQN-DDPG approaches in Fig. 6 equal $150$, $315$,
$660$, and $710$, respectively. It is worth mentioning that the desirable
performance of the hybrid DQN-DDPG approach comes with a cost of training a
model to obtain each Pareto point. However, the meta-DRL with fine-tuning
approach trains only one model and to obtain a Pareto point it runs a few
fine-tuning steps.

To illustrate the required training time to obtain the Pareto front, Fig. 7
illustrates the total training time versus the number of Pareto points. It is
evident that the total training time of the meta-based solutions is
significantly less than that of the hybrid DQN-DDPG approach, particularly for
large number of Pareto points. Nonetheless, the meta-DRL with fine-tuning
approach achieves a Pareto front close to that of the hybrid DQN-DDPG
approach, albeit having much less total training time.

Figure 8: Objective function in (16) versus the number of vehicles with $V=10$
vehicles, $F=4$ processes,
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}=2$, and $\zeta=0.5$ for
the hybrid DQN-DDPG and meta-DRL with fine-tuning. Figure 9: Average AoI and
power expenditure versus the number of vehicles with $V=10$ vehicles, $F=4$
processes and $\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}=2$, and
$\zeta=0.5$ for the hybrid DQN-DDPG and meta-DRL with fine-tuning. Figure 10:
Objective function in (16) versus the total number of processes $F$ with
number of process of interest per vehicle
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}=\lceil\frac{F}{2}\rceil$,
$V=10$ vehicles, and $\zeta=0.5$ for the hybrid DQN-DDPG and meta-DRL with
fine-tuning. Figure 11: Average AoI and power expenditure versus the total
number of processes $F$ with number of process of interest per vehicle
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}=\lceil\frac{F}{2}\rceil$,
$V=10$ vehicles, and $\zeta=0.5$ for the hybrid DQN-DDPG and meta-DRL with
fine-tuning.

To illustrate the effectiveness of the proposed framework and solution
approaches versus the number of vehicles $V$, Fig. 8 and Fig. 9 illustrate the
objective function in (16) and the corresponding objectives over a range of
the number of vehicles, respectively. It can be seen that the proposed
framework with the hybrid DQN-DDPG DRL minimizes both the AoI and power
consumption, while the AoI in the random solution increases rapidly as the
number of vehicles increases. It can also noticed that the meta-DRL solution
minimizes the AoI and power consumption even though it was trained using
randomly selected objective-preference weights. The adaptation capability of
the the meta-DRL model can be inferred as well as the performance of the meta-
DRL with fine-tuning is close to that of the conventional DRL.

Figures 10 and 11 illustrate the objective function in (16) and the
corresponding average AoI and power consumption over a range of the number of
physical process, respectively. It is worth mentioning that in these figures
as the number of physical process increases the number of required process per
vehicle is also increases (i.e.,
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}=\lceil\frac{F}{2}\rceil$).
It can be noticed that both the AoI and power consumption increase as the
number of physical process increases, and the meta-DRL with fine-tuning
solution provides better performance in comparison with the meta-DRL and
random solutions.

Figures 12 and 13 show the performance of the solution approaches for a number
of physical process over a range of the number of required process per
vehicle. It can be noticed that both the AoI and power consumption increase as
the number of required process per vehicle increases, and the meta-DRL with
fine-tuning solution provides better performance in comparison with the meta-
DRL and random solutions.

Figure 12: Objective function in (16) versus number of process of interest per
vehicle $\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}$ with $F=6$
processes, $V=10$ vehicles, and $\zeta=0.5$ for the hybrid DQN-DDPG and meta-
DRL with fine-tuning. Figure 13: Average AoI power expenditure versus number
of process of interest per vehicle
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}$ with $F=6$ processes,
$V=10$ vehicles, and $\zeta=0.5$ for the hybrid DQN-DDPG and meta-DRL with
fine-tuning.

To study the effect of the fine-tuning on the performance of the meta-based
DRL solution, Fig. 14 illustrates the performance of the hybrid DQN-DDPG
model, meta-based DRL, and meta-based DRL with fine-tuning solutions versus
the number of fine-tuning steps. The hybrid DQN-DDPG model is trained with the
objective-preference weight of $\zeta=0.5$ while the meta-based DRL model is
trained using randomly selected values of $\zeta$. It can be seen that as the
number of the fine-tuning steps increases the performance of meta-DRL solution
with fine-tuning improves.

Figure 14: Objective function in (16) versus the number of fine-tuning steps
with $V=10$ vehicles, $F=4$ processes,
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}=2$, and $\zeta=0.5$ for
the hybrid DQN-DDPG and meta-DRL with fine-tuning.

To study the effect of the vehicles’ mobility and the convergence of the meta-
based DRL algorithm, Fig. 15 illustrates the performance when the DRL based
model is trained with vehicle speed $c_{i}\sim U(10,15)$ m/s and deployed on
networks with three different vehicle speed values $20,25$, and $30$ m/s. It
can be noticed that the algorithm provides a good convergence performance and
can adapt to the change in the network mobility after a few fine-tuning steps.
It can be also inferred that the power required to deliver timely updates
increases with the vehicles’ speed, which consequently leads to a higher value
of the objective function.

Figure 15: Objective function in (16) versus the number of fine-tuning steps
for vehicles’ speeds $20,25$, and $30$ m/s with $\zeta=0.5$, $V=10$ vehicles,
$F=4$ processes, and
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}=2$. Figure 16:
Objective function in (16) versus the number of fine-tuning steps with
$\zeta=0.5$, total number of processes $F=4$ processes and
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}=2$ for two different
environments (1) two lanes with $c_{i}\sim U(10,15)$ m/s and (2) four lanes
with $c_{i}\sim U(25,35)$ m/s.

To study the generalization ability of the meta-based DRL algorithm to new
physical environments, we considered two different vehicular network scenarios
in Fig. 16, namely, urban environment which considers a two lanes road with
velocity of the vehicle is $c_{i}\sim U(10,15)$ m/s and highway environment
which considers a four lanes road with velocity of the vehicle is $c_{i}\sim
U(25,35)$ m/s. Fig. 16 illustrates the performance of the DRL model for three
different cases (1) the same-environment solution which represents the case
when the DRL is trained and deployed on the same environment (i.e., the urban
environment); (2) the different-environment solution which represents the case
when the DRL is trained on the urban environment and deployed on the highway
environment; and (3) the meta-DRL with fine-tuning solution which represents
the case when the DRL is trained on the urban environment fine-tuned and
deployed on the highway environment. It can be seen that the meta-DRL model
has a good generalization ability and can fast adapt to new environments via a
few fine-tuning steps.

##  7 Conclusion

This paper has proposed an SIC-enabled message superposition framework to
disseminate timely status updates about a set of physical processes to a set
of vehicles. A multi-objective mixed integer optimization problem has been
formulated to minimize both the average AoI and power consumption objectives
in vehicular networks. A hybrid DQN-DDPG DRL model has been developed to solve
the optimization problem. To address the tradeoff between the two objectives,
a meta-based DRL algorithm is trained using a range of objective-preference
weights and undergoes a few fine-tuning update steps to estimate the Pareto
points of the problem. Simulation results have illustrated that the hybrid
DQN-DDPG solution minimizes both the AoI and power consumption for the given
objective-preference weight. Moreover, the meta-model demonstrates a rapid and
excellent adaptability to estimate a high quality Pareto frontier for all
problem instances, including those with either unseen objective-preference
weights or the new vehicular environments. Considering DRL-based solution to
handle more than two conflicting metrics in vehicular networks is considered
as a future work.

## References

  * [1] S. Zhang, J. Chen, F. Lyu, N. Cheng, W. Shi, and X. Shen, “Vehicular communication networks in the automated driving era,” _IEEE Commun. Mag._ , vol. 56, no. 9, pp. 26–32, 2018.
  * [2] Z. Li, L. Xiang, and X. Ge, “Age of information modeling and optimization for fast information dissemination in vehicular social networks,” _IEEE Trans. Veh. Technol._ , vol. 71, no. 5, pp. 5445–5459, 2022.
  * [3] S. Kaul, R. Yates, and M. Gruteser, “Real-time status: How often should one update,” in _Proc. IEEE INFOCOM_ , 2012, pp. 2731–2735.
  * [4] Y. Ren, F. Liu, Z. Liu, C. Wang, and Y. Ji, “Power control in D2D-based vehicular communication networks,” _IEEE Trans. Veh. Technol._ , vol. 64, no. 12, pp. 5547–5562, 2015.
  * [5] X. Chen, C. Wu, T. Chen, H. Zhang, Z. Liu, Y. Zhang, and M. Bennis, “Age of information aware radio resource management in vehicular networks: A proactive deep reinforcement learning perspective,” _IEEE Trans. Wirel. Commun._ , vol. 19, no. 4, pp. 2268–2281, 2020.
  * [6] M. Samir, C. Assi, S. Sharafeddine, D. Ebrahimi, and A. Ghrayeb, “Age of information aware trajectory planning of UAVs in intelligent transportation systems: A deep learning approach,” _IEEE Trans. Veh. Technol._ , vol. 69, no. 11, pp. 12 382–12 395, 2020.
  * [7] Y. Yuan, G. Zheng, K.-K. Wong, and K. B. Letaief, “Meta-reinforcement learning based resource allocation for dynamic V2X communications,” _IEEE Trans. Veh. Technol._ , vol. 70, no. 9, pp. 8964–8977, 2021.
  * [8] C. Liu, X. Xu, and D. Hu, “Multiobjective reinforcement learning: A comprehensive overview,” _IEEE Trans. Syst. Man Cybern. Syst._ , vol. 45, no. 3, pp. 385–398, 2015.
  * [9] S. Zarandi, A. Khalili, M. Rasti, and H. Tabassum, “Multi-objective energy efficient resource allocation and user association for in-band full duplex small-cells,” _IEEE Trans. Green Commun. Netw._ , vol. 4, no. 4, pp. 1048–1060, 2020.
  * [10] R. Hashemi, H. Beyranvand, M. R. Mili, A. Khalili, H. Tabassum, and D. W. K. Ng, “Energy efficiency maximization in the uplink Delta-OMA networks,” _IEEE Trans. Veh. Technol._ , vol. 70, no. 9, pp. 9566–9571, 2021.
  * [11] I. Y. Kim and O. De Weck, “Adaptive weighted sum method for multiobjective optimization: A new method for Pareto front generation,” _Struct. Multidiscipl. Optim._ , vol. 31, no. 2, pp. 105–116, 2006.
  * [12] R. T. Marler and J. S. Arora, “Survey of multi-objective optimization methods for engineering,” _Structural and multidisciplinary optimization_ , vol. 26, no. 6, pp. 369–395, 2004.
  * [13] O. B. Augusto, F. Bennis, and S. Caro, “A new method for decision making in multi-objective optimization problems,” _Pesquisa Operacional_ , vol. 32, pp. 331–369, 2012.
  * [14] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _Proc. International conference on machine learning_. PMLR, 2017, pp. 1126–1135.
  * [15] Z. Zhang, Z. Wu, H. Zhang, and J. Wang, “Meta-learning-based deep reinforcement learning for multiobjective optimization problems,” _IEEE Trans. Neural Netw. Learn. Syst._ , pp. 1–14, 2022.
  * [16] X. Chen, A. Ghadirzadeh, M. Björkman, and P. Jensfelt, “Meta-learning for multi-objective reinforcement learning,” in _Proc. IEEE International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 977–983.
  * [17] Y. Ni, L. Cai, and Y. Bo, “Vehicular beacon broadcast scheduling based on age of information (AoI),” _China Commun._ , vol. 15, no. 7, pp. 67–76, 2018\.
  * [18] M. K. Abdel-Aziz, S. Samarakoon, C.-F. Liu, M. Bennis, and W. Saad, “Optimized age of information tail for ultra-reliable low-latency communications in vehicular networks,” _IEEE Trans. Commun._ , vol. 68, no. 3, pp. 1911–1924, 2020.
  * [19] M. Chen, Y. Xiao, Q. Li, and K.-c. Chen, “Minimizing age-of-information for fog computing-supported vehicular networks with deep Q-learning,” in _Proc. IEEE International Conference on Communications (ICC)_ , 2020, pp. 1–6.
  * [20] L. Yang, J. Chen, Q. Ni, J. Shi, and X. Xue, “NOMA-enabled cooperative unicast–multicast: Design and outage analysis,” _IEEE Trans. Wirel. Commun._ , vol. 16, no. 12, pp. 7870–7889, 2017.
  * [21] J. Wang, H. Xu, B. Zhu, L. Fan, and A. Zhou, “Hybrid beamforming design for mmWave joint unicast and multicast transmission,” _IEEE Commun. Lett._ , vol. 22, no. 10, pp. 2012–2015, 2018.
  * [22] W. Hao, G. Sun, F. Zhou, D. Mi, J. Shi, P. Xiao, and V. C. M. Leung, “Energy-efficient hybrid precoding design for integrated multicast-unicast millimeter wave communications with SWIPT,” _IEEE Trans. Veh. Technol._ , vol. 68, no. 11, pp. 10 956–10 968, 2019.
  * [23] M. Sadeghi, E. Björnson, E. G. Larsson, C. Yuen, and T. Marzetta, “Joint unicast and multi-group multicast transmission in massive MIMO systems,” _IEEE Trans. Wirel. Commun._ , vol. 17, no. 10, pp. 6375–6388, 2018.
  * [24] Y. Chen, H. Tang, and J. Wang, “Optimizing age of information in multicast unilateral networks,” in _Proc. IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB)_ , 2020, pp. 1–4.
  * [25] X. Yuan, J. Chen, N. Zhang, J. Ni, F. R. Yu, and V. C. M. Leung, “Digital twin-driven vehicular task offloading and IRS configuration in the internet of vehicles,” _IEEE Trans. Intell. Transp. Syst._ , vol. 23, no. 12, pp. 24 290–24 304, 2022.
  * [26] X. Yuan, J. Chen, N. Zhang, Q. Ye, C. Li, C. Zhu, and X. S. Shen, “Low-cost federated broad learning for privacy-preserved knowledge sharing in the RIS-aided internet of vehicles,” _Engineering_ , 2023.
  * [27] J. Li, Y. Zhou, and H. Chen, “Age of information for multicast transmission with fixed and random deadlines in IoT systems,” _IEEE Internet Things J._ , vol. 7, no. 9, pp. 8178–8191, 2020.
  * [28] S. Nath, J. Wu, and J. Yang, “Optimum energy efficiency and age-of-information tradeoff in multicast scheduling,” in _Proc. IEEE International Conference on Communications (ICC)_ , 2018, pp. 1–6.
  * [29] M. Xie, J. Gong, X. Jia, and X. Ma, “Age and energy tradeoff for multicast networks with short packet transmissions,” _IEEE Trans. Commun._ , vol. 69, no. 9, pp. 6106–6119, 2021.
  * [30] M. Zhou, J. Li, J. Yuan, M. Xie, W. Tan, R. Yin, and L. Yang, “An architecture for AoI and cache hybrid multicast/unicast/D2D with cell-free massive MIMO systems,” _IEEE Access_ , vol. 11, pp. 43 080–43 088, 2023.
  * [31] A. Al-Habob, H. Tabassum, and O. Waqar, “Dynamic unicast-multicast scheduling for age-optimal information dissemination in vehicular networks,” in _Proc. IEEE Globecom Workshops (GC Wkshps)_ , 2022, pp. 1218–1223.
  * [32] W. Yuan, Z. Wei, S. Li, J. Yuan, and D. W. K. Ng, “Integrated sensing and communication-assisted orthogonal time frequency space transmission for vehicular networks,” _IEEE J. Sel. Top. Signal Process._ , vol. 15, no. 6, pp. 1515–1528, 2021.
  * [33] Z. Yang, J. A. Hussein, P. Xu, Z. Ding, and Y. Wu, “Power allocation study for non-orthogonal multiple access networks with multicast-unicast transmission,” _IEEE Trans. Wirel. Commun._ , vol. 17, no. 6, pp. 3588–3599, 2018.
  * [34] M. Sadeghi and C. Yuen, “Multi-cell multi-group massive MIMO multicasting: An asymptotic analysis,” in _Proc. IEEE Global Communications Conference (GLOBECOM)_ , 2015, pp. 1–6.
  * [35] M. Sadeghi, E. Björnson, E. G. Larsson, C. Yuen, and T. L. Marzetta, “Max–min fair transmit precoding for multi-group multicasting in massive MIMO,” _IEEE Trans. Wirel. Commun._ , vol. 17, no. 2, pp. 1358–1373, 2018\.
  * [36] C. She, C. Sun, Z. Gu, Y. Li, C. Yang, H. V. Poor, and B. Vucetic, “A tutorial on ultrareliable and low-latency communications in 6G: Integrating domain knowledge into deep learning,” _Proc. IEEE_ , vol. 109, no. 3, pp. 204–246, 2021.
  * [37] X. Ma, Y. Yu, X. Li, Y. Qi, and Z. Zhu, “A survey of weight vector adjustment methods for decomposition-based multiobjective evolutionary algorithms,” _IEEE Trans. Evol. Comput._ , vol. 24, no. 4, pp. 634–649, 2020.
  * [38] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. MIT Press, 2018.
  * [39] V. Mnih _et al._ , “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, Feb. 2015.
  * [40] T. P. Lillicrap _et al._ , “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [41] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proc. Int. Conf. Learn. Representations_ , 2015.
  * [42] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls _et al._ , “Value-decomposition networks for cooperative multi-agent learning,” _arXiv preprint arXiv:1706.05296_ , 2017.
  * [43] ——, “Value-decomposition networks for cooperative multi-agent learning based on team reward,” in _Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems_ , 2018, pp. 2085–2087.
  * [44] H. Kim, S. Kim, D. Lee, and I. Jang, “Avoiding collaborative paradox in multi-agent reinforcement learning,” _ETRI Journal_ , vol. 43, no. 6, pp. 1004–1012, 2021.
  * [45] K. Ji, J. Yang, and Y. Liang, “Theoretical convergence of multi-step model-agnostic meta-learning.” _J. Mach. Learn. Res._ , vol. 23, pp. 29–70, 2022.
  * [46] A. P. Guerreiro, C. M. Fonseca, and L. Paquete, “The hypervolume indicator: Problems and algorithms,” _arXiv preprint arXiv:2005.00515_ , 2020.

|  Ahmed A. Al-habob (S’15-M’22) received the BSc degree in telecommunications
and computer engineering from Taiz University, Yemen, in 2009. He received the
MSc degree in telecommunications from the Electrical Engineering Department,
King Fahd University of Petroleum and Minerals (KFUPM), Saudi Arabia, in 2016.
He received the Ph.D. degree in Electrical Engineering from the Faculty of
Engineering and Applied Science, Memorial University, St. John’s, NL, Canada,
in 2022. He was a Postdoctoral Visitor at Lassonde School of Engineering, York
University, Toronto, ON, Canada. He is currently a Postdoctoral Fellow at
Memorial University of Newfoundland, NL, Canada. His research interest
includes wireless communications and networking.
---|---
|  Hina Tabassum (Senior Member, IEEE) received the Ph.D. degree from the King
Abdullah University of Science and Technology (KAUST). She is currently an
Associate Professor with the Lassonde School of Engineering, York University,
Canada, where she joined as an Assistant Professor, in 2018. She is also
appointed as the York Research Chair of 5G/6G-enabled mobility and sensing
applications (2023 - 2028). She was a postdoctoral research associate at
University of Manitoba, Canada. She has published over 100 refereed papers in
well-reputed IEEE journals, magazines, and conferences. She received the
Lassonde Innovation Early-Career Researcher Award in 2023 and the N2Women:
Rising Stars in Computer Networking and Communications in 2022. She was listed
in the Stanford’s list of the World’s Top $2\%$ Researchers in 2021, 2022, and
2023. She is the Founding Chair of the Special Interest Group on THz
communications in IEEE Communications Society (ComSoc)-Radio Communications
Committee (RCC). She served as an Associate Editor for IEEE Communications
Letters (2019–2023), IEEE Open Journal of the Communications Society (OJCOMS)
(2019–2023), and IEEE Transactions on Green Communications and Networking
(TGCN) (2020–2023). Currently, she is also serving as an Area Editor for IEEE
OJCOMS and an Associate Editor for IEEE Transactions on Communications, IEEE
Transactions on Wireless Communications, and IEEE Communications Surveys and
Tutorials. She has been recognized as an Exemplary Editor by the IEEE
Communications Letters (2020), IEEE OJCOMS (2023), and IEEE TGCN (2023). Her
research interests include multi-band optical, mm-wave, and THz networks and
cutting-edge machine learning solutions for next generation wireless
communication and sensing networks.
---|---
|  Omer Waqar received the B.Sc. degree in electrical engineering from the
University of Engineering and Technology (UET), Lahore, Pakistan, in 2007 and
the Ph.D. degree in electrical and electronic engineering from the University
of Leeds, Leeds, U.K., in November 2011. From January 2012 to July 2013, he
was a Research Fellow with the Center for Communications Systems Research and
5G Innovation Center (5GIC), University of Surrey, Guildford, U.K. He worked
as an Assistant Professor in UET, Lahore, Pakistan from August 2013 to June
2018. He worked as a researcher in the department of Electrical and Computer
Engineering, University of Toronto, Canada from July 2018 to June 2019. He
worked as an Assistant Professor in the department of Engineering, Thompson
Rivers University (TRU), British Columbia (BC), Canada from August 2019 to
July 2023. Since August 2023, he has been working as an Assistant Professor in
the School of Computing, University of the Fraser Valley, BC, Canada and holds
an adjunct faculty position at York university, Ontario, Canada. He has
authored or co-authored 35+ peer-reviewed articles including top-tier journals
such as IEEE Transactions on Vehicular Technology. He has secured over $200K
in research grants from the Tri-Council agency i.e., NSERC Discovery grant and
NSERC Alliance grants. Currently, he is serving as an Associate Editor for the
IEEE Open Journal of the Communications Society and IEEE Canadian Journal of
Electrical and Computer Engineering. His current research interests include,
intelligent reflecting surface aided communication systems, Deep-Learning for
next generation communication networks, wireless sensing and resource
allocation of wireless networks for several distributed machine learning
paradigms.
---|---