# Optimizing Distributed Reinforcement Learning with Reactor Model and Lingua
Franca

Jacky Kwok
UC Berkeley
<EMAIL_ADDRESS>
Marten Lohstroh
UC Berkeley
<EMAIL_ADDRESS>
Edward A. Lee
UC Berkeley
<EMAIL_ADDRESS>

###### Abstract

Distributed Reinforcement Learning (RL) frameworks are essential for mapping
RL workloads to multiple computational resources, allowing for faster
generation of samples, estimation of values, and policy improvement. These
computational paradigms require a seamless integration of training, serving,
and simulation workloads. Existing frameworks, such as Ray, are not managing
this orchestration efficiently. In this study, we’ve proposed a solution
implementing Reactor Model, which enforces a set of actors to have a fixed
communication pattern. This allows the scheduler to eliminate works needed for
synchronization, such as acquiring and releasing locks for each actor or
sending and processing coordination-related messages. Our framework, Lingua
Franca (LF), a coordination language based on the Reactor Model, also provides
a unified interface that allows users to automatically generate dataflow
graphs for distributed RL. On average, LF outperformed Ray in generating
samples from OpenAI Gym and Atari environments by 1.21x and 11.62x, reduced
the average training time of synchronized parallel Q-learning by 31.2%, and
accelerated Multi-Agent RL inference by 5.12x.

##  1 Introduction

The field of machine learning (ML) has witnessed an exponential increase in
computational requirements for training complex models, particularly
supervised learning powered by deep neural networks. This complexity has
necessitated the creation of new frameworks focused on training these networks
and leveraging specialized hardware to reduce training times. Examples include
TensorFlow [1], MXNet [7], and PyTorch [26]. Beyond classical supervised
learning, emerging AI applications are increasingly required to operate in
dynamic environments and pursue long-term goals, problems that reinforcement
learning (RL) are well suited for. RL involves learning to behave in uncertain
environments based on feedback from interaction with the environment. Optimal
behavior can be specified via appropriately chosen reward functions of the
data. This approach has already led to significant achievements such AlphaGo
[34], and more recently, the success of ChatGPT [23]. RL applications span
various domains, including traffic systems [6], UAVs [13], large language
models (LLMs) [23], and dexterous manipulation [2].

Deep RL presents new additional challenges beyond traditional RL for
distributed frameworks. Deep RL, unlike its traditional counterparts, demands
a sequential interaction between the agent and the environment, making it
predominantly utilize the CPU (central processing unit) [3]. However, the
scalability of deep RL models, particularly in handling complex state-action
associations, hinges on efficiently leveraging both CPUs and graphical
processing units (GPUs) [12, 8]. The combination of RL with deep neural
networks in deep RL models necessitates a balanced approach in computational
resource allocation. The processing speed, especially when updating policies
involving millions of parameters, becomes a critical factor. The need to
balance CPU and GPU resources, coupled with the limitations of existing
frameworks, highlights the need for innovative approaches in developing
efficient, scalable, and versatile systems capable of supporting the dynamic
and complex nature of modern AI and ML applications. This paper aims to delve
into these challenges and explore potential solutions, paving the way for more
efficient and effective reinforcement learning systems.

To this end, we introduce an optimized version of Lingua Franca (LF) [17], a
polyglot coordination language for concurrent and time-sensitive applications
initially designed for embedded systems, which was tailored to address the
unique challenges of RL applications. LF stands out in its ability to
effectively handle a diverse range of workloads, from lightweight, stateless
computations needed for simulation to the more intensive, long-running
computations required for training.

A key feature of LF is its unified interface, which is adept at expressing
reactor-based computations (e.g. rollout worker reactors, replay buffer
reactors, and learner reactors). This adaptability is crucial for efficiently
managing the diverse computational demands of RL applications. The
computational paradigm in LF allows for effective load balancing of
simulations, handling inputs and state spaces in shared memory, and provides
robustness against failures. On the other hand, the reactor-based model is
optimized for stateful computations, such as those found in model training,
and it facilitates shared mutable state to clients, like in a parameter server
architecture.

LF is a distributed framework that successfully unifies training, simulation,
and serving. Our framework also provides a unified interface that enables
users to easily define abstractions in RL. On average, LF outperformed Ray
[22] in generating samples from OpenAI Gym [5] and Atari environments by 1.21x
and 11.62x, reduced the average training time of synchronized parallel
Q-learning by 31.2%, and accelerated Multi-Agent RL inference by 5.12x.

Contributions: In this paper, we demonstrate how the reactor-oriented paradigm
implemented in Lingua Franca optimizes utilization of computational resources
and efficiently parallelizes RL workloads. We then describe the mechanisms of
LF runtime, illustrating how it exploits parallelism and differs from the
actor-based framework, Ray. Additionally, our investigation reveals that
multithreading offers greater advantages than multiprocessing in distributed
RL. We then introduce an optimized Reactor C runtime with customized Python,
enabling true parallelism and efficient coordination of threads. We further
present an extensive evaluation on the generations of samples from OpenAI Gym
Environments, inference of multi-Agent RL, and synchronized parallel actor
critic demonstrating that the LF runtime outperforms Ray in the training,
inference, and simulation for RL.

##  2 Distributed Reinforcement Learning and Requirements

Reinforcement Learning (RL) algorithms aim to enhance an agent’s policy
performance within a given environment, often represented through a simulator.
These algorithms alternate cycles of data collection with the latest policy,
value estimation using the latest data (and possibly data collected from
previous policies), and policy improvement. Depending on the signal provided
by the reward function on the collected data and the efficiency of the
algorithm itself, the reinforcement learning process can vary in the amount of
data/computational resources needed to run until satisfactory performance is
achieved. For most tasks of interest, these data and compute requirements are
large, motivating distributed reinforcement learning frameworks that can more
efficiently leverage parallel computational paradigms.

Many aspects of the reinforcement learning problem can be parallelized [29].
Data collection can be split across multiple worker threads, learning can be
split across multiple worker threads with each thread maintaining its own
value function and policy parameters, updates to the neural networks can be
performed in parallelized fashions using popular deep learning frameworks [1,
4, 25], and simulators can themselves be parallelized, for example on
accelerated computing infrastructure [27, 36, 30, 33].

###  2.1 Variants in RL Algorithms

Single-Agent Training: The most fundamental scenario in RL is training a
single agent, which involves repeatedly applying the steps of rollout, replay,
and optimization. Synchronous algorithms like A2C [19] and PPO [31] follow
these steps in sequence, whereas asynchronous variants (e.g., A3C [19], Ape-X
[11], APPO [37], IMPALA [9]) overlap rollout and optimization steps to enhance
data throughput.

Multi-Agent Training: Multi-agent training involves multiple agents
interacting within the environment, either cooperatively or competitively.
While the dataflow structure resembles that of single-agent training,
complexities arise when customizing training for individual agents. For
instance, if agents require optimization at different frequencies or are
trained with distinct algorithms, the training dataflow must accommodate
multiple iterative loops with varying parameters.

Model-Based Algorithms: Model-based RL algorithms aim to learn the transition
dynamics of the environment to increase training efficiency [21]. This adds a
supervised training component to standard distributed RL, involving training
one or more dynamics models with environment-generated data.

Challenges and Opportunities in Framework Optimization: Frameworks like Ray
have made strides in CPU/GPU orchestration for RL, but they face efficiency
challenges due to the actor model’s inherent communication overhead and
synchronization demands. Ideal frameworks for RL should support heterogeneous
computations, flexibility in computational models, dynamic execution, and
large data handling, while integrating seamlessly with deep learning libraries
and simulation frameworks.

We argue that distributed RL frameworks can better handle millions of tasks
per second by providing deterministic concurrency. Current frameworks, on the
other hand, fall short in efficiently meeting the evolving demands of AI
applications, indicating a significant gap in the field.

This research aims to propose a novel system architecture that addresses these
comprehensive requirements, bridging the gap in the current landscape and
pushing the boundaries of what’s possible in distributed RL.

##  3 Introduction to Actor and Reactor Model

###  3.1 Actor Model

The Actor model [10], a foundational concept in concurrent computing, emerged
in the 1970s as a response to the increasing complexity and interactivity in
computer systems. It introduced a novel way to handle concurrent operations by
conceptualizing "actors" as the primary units of computation. These actors are
analogous to objects in object-oriented programming (OOP) [28], but they are
designed specifically to address the challenges of concurrency. Each actor
represents a self-contained unit with its own local state, and actors interact
with each other exclusively through asynchronous message passing. This model
contrasts with traditional approaches that often rely on shared state and
synchronization mechanisms like locks, which can lead to issues like deadlocks
and race conditions.

Actors in the model operate independently and concurrently, providing a
natural framework for distributed and parallel systems. When an actor receives
a message, it can perform several actions: it can create more actors, send
messages to other actors, modify its own internal state, or decide how to
respond to the next message it receives. This makes the model highly adaptable
and scalable, suitable for applications ranging from simple concurrent
programs to complex distributed systems. The asynchronous nature of message
passing in the Actor model is key to its effectiveness in dealing with
concurrency. It allows actors to send and receive messages without waiting for
a response, thereby preventing bottlenecks and enabling continuous operation
even when certain components are busy or delayed.

Furthermore, the Actor model introduces a flexible approach to message
processing, without enforcing any strict order in which messages must be
processed. This characteristic is particularly beneficial in distributed
systems where message delivery times can vary unpredictably. Actors can
process incoming messages in different sequences, and the model does not
guarantee that messages will arrive in the order they were sent. This
flexibility allows for more efficient utilization of resources and can lead to
more robust system designs that are tolerant of delays and variable message
delivery times. Overall, the Actor model’s emphasis on independent, concurrent
actors and asynchronous communication makes it a powerful paradigm for
building scalable and resilient concurrent and distributed systems.

###  3.2 Reactor Model

The Reactor model [16] shows great potential as an alternative to the actor
model, enabling efficient and deterministic concurrency. The newly proposed
Reactor model represents an advancement in deterministic reactive systems,
providing a structured framework for creating complex, reactive RL
applications. Central to this model are the concepts of "reactors" and
"reactions." Reactors can be interpreted as deterministic actors, but instead
of responding to messages, they react to discrete events, each linked to a
specific logical time, denoted by a "tag." These events can trigger reactions
within a reactor, similar to message handlers in actor systems, but with a key
difference: reactions in reactors are governed by a defined order, ensuring
determinism. Reactions share some similarities with message handlers in
traditional actor models but differ in their triggering mechanism. Instead of
messages, they are activated by discrete events, which can also be generated
by reactions. Each event associates a value with a tag, representing its
logical release time within the system. Reactions can access and modify state
shared with other reactions in the same reactor, but interaction between
different reactors is solely through events. This design choice illustrates
the model’s deterministic nature, as the order of reaction executions is
predictable and subject to strict constraints, such as tag order and execution
order for reactions within the same reactor.

The Reactor model can be thought of as a "sparse synchronous model." This
means that synchronous-reactive interactions at a particular logical time may
be confined to isolated parts of the system. When a reaction executes, it
exclusively accesses the reactor’s state. Moreover, for reactions triggered by
the same event (or events with the same tag), their execution order is pre-
defined, reinforcing deterministic behavior. Reactors are composed of ports
(inputs and outputs), hierarchy, and actions. The term "reactors" not only
relate to actors in actor systems but also aligns with the synchronous
reactive programming paradigm, prominent in languages like Esterel, Signal,
and Lustre. Unlike traditional actors, reactors don’t directly reference their
peers. Instead, they use named and typed ports for interconnection, enabling a
hierarchical design. This hierarchy, besides facilitating deterministic
behavior, also serves as a scoping mechanism for ports and imposes constraints
on connection types.

Additionally, reactors feature actions, a variant of ports used for scheduling
future events within the same reactor or as a synchronization mechanism
between internal logic and asynchronous external events. This design choice
provides a bridge between deterministic internal logic and the
nondeterministic external world, like sensor data, environment states, or
network messages. The Reactor model also incorporates state variables within
reactors. The shared resources, like replay buffers, model parameters,
environment states in RL, are key motivators for grouping reactions in a
single reactor. However, reactors themselves do not share state, ensuring
isolation between reactors and allowing parallel execution of reactions in
different reactors, unless a connection necessitates sequential execution.

Connections between reactors establish explicit communication channels. These
connections reveal dependencies that are crucial for scheduling decisions
honoring data dependencies. The Reactor model simplifies the declaration of
these dependencies by breaking down functionality into reactions with well-
defined lexical scopes, thus eliminating dependencies out of scope. The
Reactor model’s execution is governed by a run time environment, which adheres
to a discrete-event model to ensure determinacy. This environment is
responsible for maintaining a global event queue and a reaction queue,
managing logical time, and executing reactions in a predetermined order.
Details regarding the scheduler are described in the next section.

###  3.3 Ray

Ray [15, 22] is an open-source framework designed for distributed computing
and provides a simple, universal API for building distributed applications.
Ray utilizes the Actor model, which is central to its architecture and
operation, making it highly effective for concurrent and distributed computing
tasks. The Actor model in Ray is used to encapsulate state and behavior, with
actors being distributed across a node or a cluster and communicating through
asynchronous message passing. Actors can be used for performance reasons (like
caching soft state or ML models), or they could be used for managing long
living connections to databases or to web sockets. They can maintain state
across multiple tasks, which is particularly useful for applications that
require managing large, mutable states, such as machine learning models or
large datasets.

The local scheduler in Ray is a component of a worker node called Raylet.
Raylet manages the worker processes and consists of two components, a task
scheduler and an object store. The task scheduler takes care of scheduling and
executing work on a node. It addresses issues such as a worker being busy, not
having the proper resources to run a task, or not having the values it needs
to run a given task. Expensive serialization and deserialization as well as
data copying are a common performance bottleneck in distributed computing.
Shared memory, specifically that which is managed directly by the operating
system kernel, emerges as a superior mechanism compared to conventional
approaches like socket connections. This kind of memory management boasts
efficiency that transcends even memory-mapped file-based sharing. The
fundamental attribute of distributed object store is that all objects within
the store are immutable and retained in shared memory. The object store has
its eviction policy, removing objects from the store or transferring them to
other nodes when the allocated size limit is exceeded. This design choice
ensures optimal access speeds, particularly when multiple workers on a
singular node need to engage with the data. It’s noteworthy that every node
possesses an independent object store. Each node of a Ray is equipped with an
object store, within that node’s Raylet, and all objects stored collectively
form the distributed object store of a cluster. The object store manages a
shared pool of memory across workers on the same node. The object store is
implemented in Plasma, which belongs to the Apache Arrow project.
Functionally, the object store takes care of memory management and ultimately
makes sure workers have access to the objects they need in Ray.

###  3.4 Lingua Franca

Lingua Franca (LF) [18, 32] is a polyglot coordination language designed to
facilitate the development of concurrent systems by focusing on deterministic
interactions with their environment. It operates on the reactor model, where
reactors are the fundamental units of composition, each encapsulating
reactions to external stimuli. LF’s main feature is its deterministic nature,
meaning that given a set of inputs, a LF program will always produce the same
outputs, greatly enhancing testability and efficiency. This determinism is
achieved by using a superdense model of time (where events are ordered by time
and microstep), ensuring causality and the absence of non-deterministic
feedback loops in the reaction network.

The architecture of LF includes a compiler (lfc). The compiler process
involves parsing and validating the LF code, checking for syntax errors,
instantiation cycles, and cyclic dependencies among reactions. Valid code can
then be transpiled into target code in languages like C, C++, TypeScript, and
Python, which is then combined with a runtime system to manage the execution
of reactors. In this paper, we focus on the Python target as most RL
applications are written in Python. LF also allows for graphical program
representation, enhancing program structure understanding and error
identification.

In LF, reactors can be defined with parameters (immutable after
initialization), ports (for data input/output), actions (for scheduling
internal events), and timers (for periodic events). Reactors can also declare
state variables to maintain state across logical time. Reactions in LF, which
contain the core logic, are defined with triggers (conditions under which they
execute), sources (additional data inputs), and effects (outputs or actions
they can trigger). LF supports reaction deadlines, where an alternative code
block executes if a reaction misses its specified deadline, thus enforcing
timing constraints.

LF allows for flexibility in connecting reactors through multiports (ports
handling multiple data channels) and banks (multiple reactor instances).
Connections can be logical, implying synchronization between ports, or
physical, introducing intentional non-determinism for scenarios where strict
event ordering is not required. LF’s syntax permits the creation of complex
interaction patterns among reactors, offering a robust toolset for
constructing scalable and maintainable concurrent systems.

The formal semantics of LF are grounded in the theory of discrete-event
systems, utilizing a generalized ultrametric space for modeling the behavior
of LF programs. This approach guarantees that each LF program is deterministic
and adheres to causality, essential for reliability in concurrent system
design. LF’s semantic model is fully abstract, providing both an operational
perspective (how the program executes) and a denotational perspective (the
meaning of program constructs), ensuring consistency.

###  3.5 Representing Distributed RL As Dataflow Graph with Lingua Franca

Figure 1: Generated Dataflow Graph for distributed RL

LF provides a streamlined process to represent distributed RL as dataflow
graphs [17]. By simply initializing reactors and setting their input and
output ports, LF can automatically generate the corresponding dataflow graph.
This diagrammatic feature is seamlessly integrated with Visual Studio
extensions and Eclipse, offering a visual representation of the underlying RL
processes. The resulting diagram (Fig. 1) clearly displays the reactors,
ports, state variables, and their data dependencies, simplifying the
understanding of the system’s structure and aiding in the debugging of
distributed RL systems. Users are therefore better positioned to manage the
parallelization of RL processes. Detailed information about the implementation
is provided in the appendix.

RolloutReactor: This reactor is pivotal in interacting with the environment.
It gathers trajectories by executing a policy and recording the resulting
states, rewards, and other pertinent outcomes. It receives gradients from the
LearnerReactor for policy updates and sends experience (e.g. trajectories) to
the ReplayBufferReactor.

  * •

EnvironmentState: The present state of the environment.

  * •

PolicyState: The internal state of the policy, significant for stateful
policies such as those using recurrent neural networks.

  * •

ActionBuffer: A temporary repository for actions decided by the policy.

  * •

RewardBuffer: Temporary storage for rewards post-action.

  * •

ObservationBuffer: Temporary storage for new environmental observations post-
action.

ReplayBufferReactor: Acting as a centralized experience replay buffer, this
reactor stores trajectories from the RolloutReactor for subsequent sampling.
It provides sampled experience batches to the LearnerReactor for policy
updates.

  * •

ExperienceData: An accumulation of experiences (state, action, reward,
subsequent state, and termination info).

  * •

SamplingPointer: Indices or pointers facilitating efficient experience
sampling.

  * •

PrioritizedInfo: For prioritized replay buffers, additional data to manage
experience priorities.

LearnerReactor: The LearnerReactor updates policies based on sampled
experiences from the ReplayBufferReactor or directly from the RolloutReactor,
and broadcast updated gradients to the RolloutReactor.

  * •

ModelParameter: The neural network parameters, including weights and biases.

  * •

OptimizerState: Elements related to the optimization process, such as momentum
and learning rates.

  * •

LearningRate: The current learning rate, either static or dynamically
adjusted.

  * •

TargetNetworkParameters: In algorithms like DQN [20], these are slowly updated
parameters offering a stable learning target.

##  4 Optimizations for Distributed Reinforcement Learning

###  4.1 Scheduling Algorithm

The execution process of LF programs involves a scheduler responsible for
overseeing all scheduled future events, managing the logical time progression,
and executing triggered reactions in the order dictated by the dependency
graph. The scheduling mechanism in the LF model, depicted in fig. 2, operates
with an event queue that strictly follows a tag order for processing upcoming
events. Upon adding an event to the queue and processing it, the scheduler
identifies triggered reactions, placing them in a reaction queue. These
reactions are subsequently moved to a ready queue and executed by worker
threads once all dependencies, as outlined by the Action-Port Graph (APG), are
satisfied. Within this scheduling framework, the scheduler ensures a
sequential execution of reactions, considering dependencies and aiming to
maximize parallelism.

The scheduling mechanism is similar to Directed Acyclic Graph (DAG)-based
strategies but differs by accommodating reactions within the Action-Port Graph
(APG) that may not always need execution. Since the scheduler cannot predict
in advance which reactions will be triggered at a given tag, it cannot
precompute an optimal schedule. To address this, the scheduler assigns a level
to each reaction, allowing reactions at the same level to execute
concurrently. This heuristic eliminates the need for runtime APG analysis,
streamlining the process. The scheduler processes reactions in a stepwise way,
advancing to the next level only after completing all reactions at the current
level. This approach substantially reduces synchronization overhead and
contention on shared resources, contributing to its efficiency.

Figure 2: Scheduling mechanism in the LF runtime [18]

###  4.2 Scheduling Optimizations

The scheduling algorithm described in the above section is relatively
straightforward to implement, but achieving competitive performance requires
additional optimizations. These optimizations, implemented in the C runtime,
are detailed below.

Coordination of worker threads: Conceptually, the scheduler and workers are
distinct; however, in practice, having a central scheduler with separate
worker threads can cause significant synchronization overhead. To mitigate
this, our implementation allows any worker thread to become the scheduler,
allowing it to move ready reactions to the queue or advance to the logical
time once all reactions are processed. Additionally, LF leverage the fact that
we know the number of parallel reactions to execute from the APG, and thus can
use a counting semaphore to regulate the number of active workers.

Lock-free data structure: The three queues (event, reaction, and ready queue)
and other data structures are shared across all workers. Mutex-based
synchronization would be inefficient due to high contention over these shared
resources. Accordingly, LF adopted lock-free data structures wherever
feasible. For example, the ready queue is a fixed-size buffer with an atomic
counter, which corresponds to the maximum number of parallel reactions defined
by the APG level. Each time a worker tries to execute a reaction it atomically
decrements the counter. If the counter is negative, it indicates an empty
queue, and the worker should proceed accordingly. Otherwise, the counter
provides the index within the buffer from which to read. This operation is
safe without additional synchronization, as all workers are waiting for
upcoming new reactions.

###  4.3 Enabling Multi-threading for Reinforcement Learning

Parallel processing is a cornerstone in the field of machine learning,
enabling the handling of computationally intensive tasks and large-scale data.
The parallel execution models, namely multithreading and multiprocessing,
provide different advantages. This section aims to dissect these models to
guide the selection of an appropriate parallel execution strategy for RL.

Multithreading in RL:

  1. 1.

Shared Memory Space: Multithreading involves threads operating within the same
memory space, which facilitates faster and more efficient data sharing among
threads compared to multiprocessing.

  2. 2.

Resource Efficiency: The creation and management of threads consume fewer
resources than processes. This efficiency stems from the shared memory space
and the absence of a need for complex inter-process communication mechanisms.

  3. 3.

I/O Bound Task Optimization: Multithreading proves advantageous in I/O-bound
operations, where the ability to perform other tasks while waiting for I/O
operations enhances efficiency.

  4. 4.

Context Switching: The shared process and memory space of threads enable
faster context switching than multiprocessing, as less information needs to be
saved and restored.

Multiprocessing in RL

  1. 1.

CPU Bound Task Optimization: Multiprocessing is typically more suitable for
CPU-bound tasks, allowing for the distribution of tasks across multiple CPU
cores. However, most of the CPU-bound tasks (e.g. gradient updates) in RL
should be offloaded to GPU instead.

  2. 2.

Fault Tolerance and Stability: The isolated nature of processes in
multiprocessing ensures that a crash in one process does not impact others,
thus enhancing application stability. The scheduler in LF guarantees the
safety of threads and permits users to specify actions in the event of a
failure.

  3. 3.

Bypassing the GIL in Python: In Python, the Global Interpreter Lock (GIL)
limits thread execution, multiprocessing provides a viable alternative for
parallel CPU computations. This has been resolved by leveraging the No GIL
version of Python.

  4. 4.

Multi-Core Utilization: Multiprocessing enables the full utilization of multi-
core processors by running separate processes on each core. However, it’s
important to consider that when sending excessively large objects, the
processor may spend a significant amount of time on I/O overhead. LF, on the
other hand, is able to bring threads to its full utilization

###  4.4 Optimizing Thread Allocations

The principle that running a task with multiple threads can speed up the
process is generally true, as each thread can handle a portion of the work
simultaneously. However, the scaling is not always linear due to various
factors like thread allocation and CPU architecture. In our benchmarks, the
Intel CPU has a hybrid architecture, featuring a combination of performance
and efficiency cores. Performance cores are designed for high-speed and
intensive tasks, while efficiency cores are optimized for lower power
consumption and handling background tasks. Current implementation of LF
runtime randomly allocates threads for workers. As a result, when synchronous
tasks are distributed across these cores, the overall speed is bounded by the
slower efficiency cores. To achieve linear scaling, the Reactor C runtime has
been optimized to prioritize using the performance cores before spawning
threads.

##  5 Performance Comparison Between LF and Ray

The actor model is a popular approach for developing distributed RL
applications, with Ray being notable for their usability and efficiency in
handling multiple actors. However, Lingua Franca (LF) introduces a different
model of computation, imposing more restrictions compared to the actor model.
We hypothesize that the reactor model can outperform the traditional actor
models used in distributed RL frameworks (e.g. Ray). Drawing evidence from
this study, the reactor model has been shown to surpass the performance of the
traditional Actor Framework, Akka and C++ Actor Framework, by factors of 1.86x
and 1.42x, respectively. More specifically, the reactor model with a fixed set
of actors and fixed communication patterns allows the scheduler to eliminate
works needed for synchronization (e.g. acquiring and releasing locks for each
actor, or sending and processing coordination-related messages between
environment and policy). Furthermore, LF leverages No GIL Python for
multithreading, which offers several benefits, including efficient data
sharing within a shared memory space, resource efficiency due to lower
resource consumption compared to processes, and faster context switching
thanks to the shared process and memory space. The reduced overhead is
demonstrated by both figure 7 and 8. All measurements were performed on AWS
EC2 m5.8xlarge instance, equipped with an Intel® Xeon® Platinum 8175M CPU @
2.50GHz featuring 32 vCPUs. This setup includes 128 GiB of RAM and offers a 10
Gbps network bandwidth. The system runs on Ubuntu 20.04 and use Python 3.9.10,
with numpy version 1.22.3 and gym version 0.19.0. Benchmarks are available at
this GitHub repository.

Figure 3: Mean Overhead of Broadcast and Gather 10MB Object with Different
Number of Actors using Ray and LF.

Figure 4: Mean Overhead of Broadcast and Gather on 16 actors with Different
Object Sizes using Ray and LF.

###  5.1 Number of Actors

Fig. 4 illustrates the mean overhead of broadcasting and gathering a 10MB
object across different numbers of actors in a distributed RL setup. Two
frameworks are compared: Ray and Lingua Franca (LF). The x-axis represents the
number of actors, which are 2, 4, 8, and 16, while the y-axis shows the
overhead in milliseconds. Both frameworks exhibit an increase in overhead as
the number of actors grows, but Ray consistently has a higher overhead than
LF. For instance, with 16 actors, Ray’s overhead is close to 20 milliseconds,
whereas LF’s is just above 5 milliseconds. The error bars suggest variability
in the measurements, with Ray showing larger variance as the number of actors
increases.

###  5.2 Object Size

Fig. 4 presents a comparison of the mean overhead for broadcasting and
gathering operations on objects of varying sizes using 16 actors, between Ray
and Lingua Franca (LF) frameworks, including a 99% confidence interval (CI).
The x-axis displays the object size in megabytes (MB), ranging from 0 to 500
MB, while the y-axis indicates the overhead in milliseconds. From the graph,
we can observe that as the object size increases, the overhead for both Ray
and LF also increases. However, Ray’s overhead grows at a higher rate than
LF’s. For instance, with the largest object size of 500 MB, Ray’s overhead
approaches 800 milliseconds, whereas LF’s overhead is about half of that,
around 400 milliseconds.

Both figures 4 and 4 support the hypothesis that the reactor model employed by
LF can outperform traditional actor models like Ray, particularly in scenarios
with a greater number of actors and larger object size, due to optimizations
such as reduced synchronization work and efficient multithreading with No GIL
Python.

Figure 5: Simulation Throughput of Ray and LF in Various Gym Environments.

Figure 6: Simulation Throughput of Ray and LF in Various Atari Environments.

###  5.3 Open AI Gym Environments

In Fig. 6, we see that LF outperforms Ray by 1.21x on average, with a
particularly significant lead in the Blackjack environment. This suggests that
LF is more efficient, especially in situations where there is a lower CPU
demand for action updates within the environment. Considering the CPU
architecture in m5x8.xlarge, which has a mix of 8 performance cores and 16
efficiency cores, the observed performance is not ideal due to the differing
speeds of the cores and potential thermal throttling. The efficiency cores are
slower and can bottleneck the system as the benchmark is limited by the
slowest thread. Moreover, when multiple cores are active, the CPU may not run
at its highest frequency due to thermal constraints. To optimize performance
and achieve linear scaling, it is recommended to utilize only the 8
performance cores and disabled. This can be done by setting CPU affinity for
performance cores, ensuring that CPU-bound tasks, which require
synchronization, do not get restricted by the slower efficiency cores. By
running tasks only on the performance cores, one can avoid the inconsistencies
that come with hybrid architectures and get a more linear scaling with the
number of actors or threads in use. It’s important to note that vectorized
environments are asynchronous and do not parallelize inference of policy.
Therefore, it is not included in our benchmarks.

###  5.4 Atari Environments

In fig. 6, we compare the performance of Ray and LF on Atari environments,
specifically on Pacman, Pong, and SpaceInvader. Here, LF again significantly
outperforms Ray in terms of observations per second. On average, LF is roughly
11.62x faster than Ray across these environments. The substantial performance
difference can be attributed to LF’s efficiency in handling high I/O
(input/output) bound tasks. Atari environments are more complex than the
previously mentioned OpenAI gym environments. They represent each state as an
80x80 numpy array, which requires more computational resources to serialize
and deserialize, especially when data needs to be sent over a network. Ray’s
use of pickle5 for serialization does help to increase throughput by
efficiently serializing numpy arrays, but it still introduces overhead during
network transmission and the serialization/deserialization process. This
overhead is particularly significant in environments where state updates are
frequent and must be communicated quickly. LF’s ability to handle the demands
of complex simulation environments is thus a key advantage.

Figure 7: Dataflow Graph of Parallel Q-learning Figure 8: Synchronized
Parallel Q-learning with Different Batch Sizes from a Replay Buffer

###  5.5 Synchronized Parallel Q-learning

Deep Q-Networks (DQNs) [20] are an advancement in reinforcement learning that
utilize deep neural networks to estimate Q-values. The Q-values are
predictions of the expected rewards for taking certain actions given
particular states in an environment. DQNs extend the capabilities of
traditional Q-learning by handling larger state and action spaces, which are
common in complex problems. These networks can scale up effectively with more
data or increased model complexity; therefore, DQN usually utilizes GPU for
gradient updates. However, since AWS EC2 m5.8xlarge doesn’t include a GPU,
benchmarking was conducted on a workstation with an Intel i9-13950HX CPU @
2.20GHz featuring 32 vCPUs, NVIDIA RTX4090, and 32 GiB of RAM. The system also
runs on Ubuntu 20.04 and uses Python 3.9.10, with numpy 1.22.3, torch 1.9.0,
and gym 0.20.0.

DQN has been implemented in a synchronized parallel Q-learning setup within a
BlackJack environment [35]. The dataflow graph is shown in fig. 7. The network
is trained to take a blackjack hand as input and output scores for each of the
possible actions in the game, which represent the expected rewards of taking
those actions. The computational tasks are distributed with the rollout and
replay buffer being executed on a CPU, while the DQN Reactor uses a GPU. This
setup takes advantage of GPUs for complex matrix operations and multicore CPUs
for sequential decision-making simulations in Open AI Gym environments.

The benchmark results, as seen in fig. 8, demonstrate performance improvements
using LF over Ray. In tests with a 500 sample batch size from the replay
buffer, the average training time decreased by 31.2%. The data also shows that
while Ray’s training time increases with larger mini-batch sizes, LF’s
performance remains stable, indicating LF’s ability to handle larger batches
efficiently without significantly increasing resource utilization.

Figure 9: TrafficJunction4-v0 environment. The goal is to pass a four-way
junction without any collisions.

Figure 10: Multi-Agent Reinforcement Learning

###  5.6 Multi-Agent RL Inference Comparison

In Multi-Agent RL (MARL), multiple agents operate in a common environment,
where each of them tries to optimize its own return by interacting with the
environment and other agents [38] (See fig. 10). In centralized MARL, a
central controller aggregates information across the agents, including joint
actions, rewards, and observations, and policies across different agents. In
decentralized MARL, which is more common in cooperative situations, each agent
makes decisions based on its local observations [24].

We validate LF in a MARL setting with ma-gym [14], a MARL library based on
OpenAI Gym [5]. We use the TrafficJunction4-v0 environment, in which four
agents are trying to pass a crossroad without crashing (see Fig. 10). This is
a decentralized setting where each agent gets it’s own local observation. See
in fig. 12, LF requires significantly less inference time than Ray (less than
1/2 the fitted line’s slope). It is also noteworthy that as the number of
agents increases as shown in fig. 12, LF’s inference time scales better
(again, less than 1/2 the rate of increase than that of Ray). In
TrafficJunction environments with 10 Agents, LF achieves a 5.12x speed up
compared to Ray.

Figure 11: Inference Time Between Ray and LF over Episode Counts

Figure 12: Inference Time Comparison between Ray and LF across Various Numbers
of Agents

##  6 Discussion and Conclusion

In this study, we introduced LF, a novel distributed RL framework based on the
Reactor model, which outperforms widely used frameworks like Ray in handling
training, serving, and simulation tasks in RL. It achieves this through
reducing work needed for synchronization using Reactor model and lock-free
data structure. Our empirical evaluations demonstrate LF’s superior
performance: a 1.21x and 11.62x faster sample generation in OpenAI Gym and
Atari environments, a 31.2% reduction in average training time for
synchronized parallel Q-learning, and a 5.12x acceleration in multi-agent RL
inference. We aim to incorporate the optimizations into LF’s federated
execution, enabling efficient distributed training and serving across
clusters. With this work, we make important contributions to the field of
distributed RL.

## References

  * [1] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. $\\{$TensorFlow$\\}$: a system for $\\{$Large-Scale$\\}$ machine learning.  In 12th USENIX symposium on operating systems design and implementation (OSDI 16) (2016), pp. 265–283.
  * [2] Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., et al. Solving rubik’s cube with a robot hand.  arXiv preprint arXiv:1910.07113 (2019).
  * [3] Ali, H., et al. A view on deep reinforcement learning in system optimization.  arXiv preprint arXiv:1908.01275 (2019).
  * [4] Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., et al. Jax: composable transformations of python+ numpy programs, 2018.
  * [5] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym.  arXiv preprint arXiv:1606.01540 (2016).
  * [6] Chen, L., Lingys, J., Chen, K., and Liu, F. Auto: Scaling deep reinforcement learning for datacenter-scale automatic traffic optimization.  In Proceedings of the 2018 conference of the ACM special interest group on data communication (2018), pp. 191–205.
  * [7] Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., and Zhang, Z. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems.  arXiv preprint arXiv:1512.01274 (2015).
  * [8] Espeholt, L., Marinier, R., Stanczyk, P., Wang, K., and Michalski, M. Seed rl: Scalable and efficient deep-rl with accelerated central inference.  arXiv preprint arXiv:1910.06591 (2019).
  * [9] Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.  In International conference on machine learning (2018), PMLR, pp. 1407–1416.
  * [10] Hewitt, C. Actor model of computation: scalable robust information systems.  arXiv preprint arXiv:1008.1459 (2010).
  * [11] Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., Van Hasselt, H., and Silver, D. Distributed prioritized experience replay.  arXiv preprint arXiv:1803.00933 (2018).
  * [12] Inci, A., Bolotin, E., Fu, Y., Dalal, G., Mannor, S., Nellans, D., and Marculescu, D. The architectural implications of distributed reinforcement learning on cpu-gpu systems.  arXiv preprint arXiv:2012.04210 (2020).
  * [13] Kaufmann, E., Bauersfeld, L., Loquercio, A., Müller, M., Koltun, V., and Scaramuzza, D. Champion-level drone racing using deep reinforcement learning.  Nature 620, 7976 (2023), 982–987.
  * [14] Koul, A. ma-gym: Collection of multi-agent environments based on openai gym.  https://github.com/koulanurag/ma-gym, 2019.
  * [15] Liang, E., Wu, Z., Luo, M., Mika, S., Gonzalez, J. E., and Stoica, I. Rllib flow: Distributed reinforcement learning is a dataflow problem.  Advances in Neural Information Processing Systems 34 (2021), 5506–5517.
  * [16] Lohstroh, H. M. F. Reactors: A deterministic model of concurrent computation for reactive systems.  University of California, Berkeley, 2020.
  * [17] Lohstroh, M., Menard, C., Bateni, S., and Lee, E. A. Toward a lingua franca for deterministic concurrent systems.  ACM Transactions on Embedded Computing Systems (TECS) 20, 4 (2021), 1–27.
  * [18] Menard, C., Lohstroh, M., Bateni, S., Chorlian, M., Deng, A., Donovan, P., Fournier, C., Lin, S., Suchert, F., Tanneberger, T., et al. High-performance deterministic concurrency using lingua franca.  ACM Transactions on Architecture and Code Optimization 20, 4 (2023), 1–29.
  * [19] Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning.  In International conference on machine learning (2016), PMLR, pp. 1928–1937.
  * [20] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning.  arXiv preprint arXiv:1312.5602 (2013).
  * [21] Moerland, T. M., Broekens, J., Plaat, A., Jonker, C. M., et al. Model-based reinforcement learning: A survey.  Foundations and Trends® in Machine Learning 16, 1 (2023), 1–118.
  * [22] Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Elibol, M., Yang, Z., Paul, W., Jordan, M. I., et al. Ray: A distributed framework for emerging $\\{$AI$\\}$ applications.  In 13th USENIX symposium on operating systems design and implementation (OSDI 18) (2018), pp. 561–577.
  * [23] OpenAI.  Gpt-4 technical report.  ArXiv abs/2303.08774 (2023).
  * [24] Oroojlooy, A., and Hajinezhad, D. A review of cooperative multi-agent deep reinforcement learning.  Applied Intelligence 53, 11 (2023), 13677–13722.
  * [25] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in pytorch, 2017.
  * [26] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library.  arXiv preprint arXiv:1912.01703 (2019).
  * [27] Puig, X., Undersander, E., Szot, A., Cote, M. D., Partsey, R., Yang, J., Desai, R., Clegg, A. W., Hlavac, M., Min, T., Gervet, T., Vondruš, V., Berges, V.-P., Turner, J., Maksymets, O., Kira, Z., Kalakrishnan, M., Malik, J., Chaplot, D. S., Jain, U., Batra, D., Rai, A., and Mottaghi, R. Habitat 3.0: A co-habitat for humans, avatars and robots, 2023.
  * [28] Rentsch, T. Object oriented programming.  ACM Sigplan Notices 17, 9 (1982), 51–57.
  * [29] Samsami, M. R., and Alimadad, H. Distributed deep reinforcement learning: An overview.  arXiv preprint arXiv:2011.11012 (2020).
  * [30] Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., Parikh, D., and Batra, D. Habitat: A Platform for Embodied AI Research.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019).
  * [31] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347 (2017).
  * [32] Schulz-Rosengarten, A., von Hanxleden, R., Lohstroh, M., Lee, E. A., and Bateni, S. Polyglot modal models through lingua franca.  In Proceedings of Cyber-Physical Systems and Internet of Things Week 2023. IEEE, 2023, pp. 337–342.
  * [33] Shacklett, B., Rosenzweig, L. G., Xie, Z., Sarkar, B., Szot, A., Wijmans, E., Koltun, V., Batra, D., and Fatahalian, K. An extensible, data-oriented architecture for high-performance, many-world simulation.  ACM Transactions on Graphics (TOG) 42, 4 (2023), 1–13.
  * [34] Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search.  nature 529, 7587 (2016), 484–489.
  * [35] Silver, D., Wierstra, A. G. I. A. D., and Riedmiller, M. Playing atari with deep reinforcement learning.  DeepMind Lab. arXiv 1312 (2013).
  * [36] Szot, A., Clegg, A., Undersander, E., Wijmans, E., Zhao, Y., Turner, J., Maestre, N., Mukadam, M., Chaplot, D., Maksymets, O., Gokaslan, A., Vondrus, V., Dharur, S., Meier, F., Galuba, W., Chang, A., Kira, Z., Koltun, V., Malik, J., Savva, M., and Batra, D. Habitat 2.0: Training home assistants to rearrange their habitat.  In Advances in Neural Information Processing Systems (NeurIPS) (2021).
  * [37] Zeng, F., and Wang, C. Visual navigation with asynchronous proximal policy optimization in artificial agents.  Journal of Robotics 2020 (2020), 1–7.
  * [38] Zhang, K., Yang, Z., and Başar, T. Multi-agent reinforcement learning: A selective overview of theories and algorithms.  Handbook of reinforcement learning and control (2021), 321–384.

##  7 Reactor Code Appendix

⬇

1target Python {

2 threading: True

3}

4

5preamble {=

6 #Import packages

7 #e.g Torch, Gym, Numpy

8=}

9

10reactor RolloutReactor {

11 input[6] gradients

12 output[6] trajectories

13

14 state EnvironmentState

15 state PolicyState

16 state ActionBuffer

17 state RewardBuffer

18 state ObservationBuffer

19

20 reaction(startup) {=

21 #Initialize Environment

22 =}

23

24 reaction(gradients) -> trajectories {=

25 #Perform rollouts for the Environment

26 =}

27}

28

29reactor ReplayBufferReactor {

30 input[6] trajectories

31 output[6] dataset

32

33 state ExperienceData

34 state SamplingPointer

35 state PrioritizedInfo

36

37 reaction(startup) {=

38 #Initialize ReplayBuffer

39 =}

40

41 reaction(trajectories) -> gradients {=

42 #Append Trajectories into ReplayBuffer

43 =}

44}

45

46reactor LearnerReactor {

47 output[6] gradients

48 input[6] dataset

49

50 state ModelParameter

51 state OptimizerState

52 state LearningRate

53 state TargetNetworkParameters

54

55 reaction(startup) -> gradients {=

56 # Initialize the policy

57 =}

58

59 reaction(dataset) -> gradients {=

60 # Update the policy

61 =}

62}

63

64main reactor {

65 rollout = new[6] RolloutReactor()

66 replay = new[6] ReplayBufferReactor()

67 learner = new[6] LearnerReactor()

68

69 # Specifiy

70 (learner.gradients)+ -> rollout.gradients

71 (rollout.trajectories)+ -> replay.trajectories

72 (replay.dataset)+ -> learner.dataset

73}


Towards a native, pervasive AI in IEEE 802.11 WLANs: The evolution path for
Wi-Fi sec:vision

The AIML TIG technical report described above does not finalize the
standardization activities within the 802.11. The TIG has recently transformed
into the AIML standing committee (SC), which will continue reviewing and
analyzing the feasibility and need for AI/ML in Wi-Fi. This, obviously,
dampens the euphoria in AI for Wi-Fi, but at the same time reveals that the
overarching goal of AI-native Wi-Fi must be taken cautiously. From now on, the
AIML SC will serve as a liaison point on AI/ML topics, which can lead to new
collaborations with other standards-developing organizations (SDOs) like ITU,
ETSI, or 3GPP, who have already worked on AI/ML standardization for several
years shehzad2022artificial, ouyang2021next, thus boosting innovation and
bringing AI/ML adoption in Wi-Fi closer. Next, we overview the main challenges
of AI/ML adoption in Wi-Fi in the post-AIML TIG landscape, from which we
derive a possible standardization roadmap towards AI nativeness (see Fig.
fig:roadmap) and highlight key enablers for adoption.

AI/ML in Wi-Fi: Standardization Challenges

Backward compatibility

One Wi-Fi-specific challenge is the backward-compatible premise of 802.11,
whereby AI/ML features should take into account legacy devices, thus ensuring
fairness in access to shared resources. Hence, AI/ML solutions must be
properly designed and exhaustively tested to ensure that the AI/ML-operating
devices are not disrupting legacy devices. For that, entities on top of 802.11
would be required to be involved, including the Wi-Fi Alliance, which is in
charge of Wi-Fi certifications and which would need to introduce novel testing
and certification track procedures for AI/ML. Alternatively, new greenfield
bands can be used for AI/ML-based protocols, hence not disrupting legacy
devices operating in different bands. This approach, however, entails
involving spectrum regulators like ITU or FCC and requires strongly motivating
the release of novel bands.

ML computations and overheads

The data-driven condition of AI/ML methods makes them really computationally
intensive, thus leading to extra computation and communication needs. In terms
of computation, hardware-constrained devices would need to handle most of the
ML operations, including data processing and ML model training and inference.
Therefore, either their computational capabilities would need to be
significantly extended, or computational support would be required (e.g., from
the cloud). In terms of storage and communication, ML requires from tens to
hundreds of megabytes. Moreover, depending on the training and inference
approach (e.g., distributed learning), the overheads may dramatically increase
due to the communication among ML components. Therefore, the use of ML models
may turn out to be unfeasible if the computation and communication
requirements are unbearable.

Continuous development and integration

ML model performance highly depends on the data with which the models are
trained, thus requiring continuous monitoring and updating. Such volatility
calls for flexible ML model deployment options, so that models can be updated
at the targeted hardware on a plug-and-play basis. The plug-and-play approach
has strong implications for the way 802.11 operates, as it would require the
unification and opening of rooted interfaces on the chipsets. With
standardized ML-specialized procedure interfaces, vendors could implement
their own AI/ML-based functionalities on top. This is aligned with 3GPP’s
vision for Rel-18, where the definition of AI/ML models is expected to be part
of the implementation and left out of the standard RP221348. For example, for
data rate selection, the 802.11 only defines transmission rates available,
thus the actual selection can be done by proprietary ML methods.

Interoperability

The heterogeneity of ML solutions can cause conflicts and lead to fairness
issues, especially if ML optimizations involve channel access. Additionally,
if ML is applied to manipulate how information is transmitted (e.g., build
fields in transmitted frames), interoperability issues may arise between
devices from different vendors. In this regard, it is important from the point
of view of 802.11 standardization to clearly define the bounds of operation
for AI/ML and to standardize new fields, so that the underlying principles of
Wi-Fi (e.g., fair channel access) are guaranteed and interoperability is
provided. Furthermore, the developed AI/ML methods will need to meet regional
laws, e.g., ETSI’s channel access rules in European unlicensed bands.

General concerns on AI/ML

Besides networking-specific challenges, AI/ML may raise broad concerns in the
areas of security, explainability, and ethics leslie2019understanding. AI has
progressed at a fast pace in recent years, even faster than regulations on AI
use. As a downside, AI may cause harm due to a large set of reasons (e.g.,
lack of explainability, misuse, poor data used for training, bad design,
etc.), thus leading to misbehavior. As a result, AI may lead to unexpected
insecure actions, thus potentially compromising security for mission-critical
applications. Other relevant threats of AI include aspects related to privacy
(e.g., potentially derived from ML model and data transfer). In addition, the
proliferation of specialized parties with expertise in AI/ML may lead to
marketplaces for trading raw ML models, trained models, and training data. The
main issue of this is that manufacturers, vendors, and operators might be
reluctant to run unknown code on their devices and networks. To address the
above-mentioned perils of AI and make it secure, ethical, and trustworthy,
substantial standardization efforts (i.e., developed outside the 802.11
standard) are required.

AI/ML in Wi-Fi: Adoption and Standardization Roadmap

Knowing that AI/ML can improve Wi-Fi’s performance and unlock novel use cases
and functionalities, it is crucial to examine how the 802.11 standard can
sustain such evolution. Owing to the challenges listed above, the journey
towards AI-native Wi-Fi is not simple and short, and will only be realized as
a result of several years of research and joint efforts in a complex
regulatory, standardization, and industrial ecosystem. In our roadmap, we
identify three potential stages of adoption (highlighted at the bottom of Fig.
fig:roadmap), based on the degree of integration and involvement of AI/ML in
the Wi-Fi protocol:

  1. 1.

Early AI/ML adoption: AI/ML solutions developed on top of the 802.11 standard
for providing additional support (e.g., enhanced troubleshooting, analytics)
and limited optimization capabilities (e.g., dynamic channel selection, radio
resource management). At this stage, commercial solutions such as smart APs or
cloud-based Wi-Fi managers include basic AI/ML functionalities.

  2. 2.

Consolidation of AI/ML: The latest efforts in 802.11 standardization and the
proliferation of Wi-Fi products with AI/ML (e.g., Qualcomm’s AI-optimized Wi-
Fi 7 chipset FastConnect 7900) underscore the willingness to integrate AI/ML
deeper into Wi-Fi. Such efforts suggest that the 802.11 specification can be
updated in the following years to allow the integration of vendor-specific
AI/ML features (e.g., ML-based CSI compression).

  3. 3.

AI nativeness: The last stage of adoption envisages AI/ML built-in 802.11 PHY
and MAC features. This stage goes beyond the current way of developing
standards and solutions (building self-AI/ML protocols [rial2023role]), so its
success depends on many advances in the regulation, improvement,
trustworthiness, and optimization of AI/ML. From standardization, AI
nativeness can be achieved by expanding the current protocols to support AI/ML
operations, thus leading to new procedures and interfaces to enable ML
pipelines (cf. 3GPP’s AI-native air interface [RP221348]).

##  Key Enablers for AI/ML Adoption in Wi-Fi

We next identify key enablers for AI/ML adoption in Wi-Fi.

###  AI/ML-aware infrastructure

AI/ML processes—data collection, data processing, model deployment, model
training/inference, ML output application—need to be considered to extend the
current network’s infrastructure, thus requiring the creation of new elements
and interfaces [wilhelmi2020flexible]. Several SDOs have already provided
guidelines and defined requirements for adopting AI/ML in telecommunications
systems (cf. the ITU-T Y.3172 and ETSI GS ENI 005 frameworks). In Wi-Fi,
several key requirements for supporting AI/ML are discussed in
[aiml_tig_use_cases]. Likewise important is data handling and processing,
which are crucial to the effective functioning of AI/ML but also to equip
networks with cognitive capabilities. Several standardization activities
started to prepare the ground in 2017 by introducing elements and procedures
for introducing data gathering and processing capabilities to networks (cf.
the 3GPP network data analytics function and ITU-T Y.3174 specification).
Finally, the adoption of AI/ML in networks calls for a paradigm shift in the
telecommunication ecosystem, leading, for instance, to the proliferation of ML
marketplaces (cf. ITU-T Y.3176).

###  Trustworthy AI/ML

Trust in AI/ML is a must. For that, AI ethics are important for defining
procedures and tools to develop secure AI applications (cf. the US leadership
in AI plan or the EU’s coordinated plan on AI). Also important is explainable
AI (XAI), which allows understanding black-box ML methods better by explaining
their behavior and outputs. Other appealing paradigms for trustworthy AI/ML
are ML sandboxes and network digital twins (NDT). An ML sandbox allows for
secure training, testing, and evaluating ML solutions before they are applied
to production environments [wilhelmi2021usage], whereas NDTs are digital
replicas of a network that can fulfill the purposes of ML sandboxes. ML
sandboxes and NDTs have started to be considered by different SDOs, cf. ITU-T
Y.3181 and IETF’s document on NDT concepts and architecture. Finally,
trustworthy AI/ML involves the definition and generation of high-quality data,
which is essential to ensure the proper training of robust and ethical AI/ML
models. For that, there must be progress on the standardization of datasets
and evaluation scenarios for validating AI/ML solutions (cf. the work in
progress ISO/IEC CD TR 5259-6).

###  Advances in AI/ML

The progress made in the field of AI/ML strongly influences the interest in
adopting it in networks. For instance, the outbreaks in AI computation with
the introduction of optimized hardware for deep learning around 2015 generated
excitement in many other areas, including telecommunications. More recently,
the outbreaks of generative AI and large language models (LLMs) have fueled
the telecom industry with new use cases and extended capabilities
[bariah2024large], thus re-shifting the previous vision on AI/ML adoption.
Apart from the advances in AI/ML per se, AI/ML optimization (e.g., model
compression, quantization) is indispensable to make AI/ML adoption feasible.
In Wi-Fi, for instance, AI/ML components are expected to mainly run in
hardware-constrained AP and stations, which prevents using heavy models and
keeping large datasets in memory. The communication overheads required to
sustain the AI/ML operation are also critical in networks, so specialized
types of signaling (e.g., new traffic classes for ML) might be defined in the
upcoming years.