# Scalable Multiuser Immersive Communications with Multi-numerology and Mini-
slot

Ming Hu, Jiazhi Peng, Lifeng Wang, and Kai-Kit Wong M. Hu, J. Peng, and L.
Wang are with the School of Information Science and Engineering, Fudan
University, Shanghai 200433, China (e-mail: $\rm
lifengwang@fudan.edu.cn$).K.-K. Wong is with the Department of Electronic and
Electrical Engineering, University College London, London WC1E 7JE, U.K.
(e-mail: kai-kit.wong@ucl.ac.uk).

###### Abstract

This paper studies multiuser immersive communications networks in which
different user equipment may demand various extended reality (XR) services. In
such heterogeneous networks, time-frequency resource allocation needs to be
more adaptive since XR services are usually multi-modal and latency-sensitive.
To this end, we develop a scalable time-frequency resource allocation method
based on multi-numerology and mini-slot. To appropriately determining the
discrete parameters of multi-numerology and mini-slot for multiuser immersive
communications, the proposed method first presents a novel flexible time-
frequency resource block configuration, then it leverages the deep
reinforcement learning to maximize the total quality-of-experience (QoE) under
different users’ QoE constraints. The results confirm the efficiency and
scalability of the proposed time-frequency resource allocation method.

###### Index Terms:

Immersive communications, quality of experience, numerology, mini-slot.

##  I Introduction

The sixth-generation wireless (6G) communications systems seek to boost
scalable and resilient transmissions for addressing new challenges, which come
from the emerging multi-modal extended reality (XR) services. As one of 6G key
use cases, immersive communications provide remote XR services including mixed
reality and augmented reality, however, these immmersive services impose
stringent requirements on data rate and communications latency [1, 2]. Multi-
numerology and mini-slot are compelling access technologies for supporting
diverse scenarios and requirements, which have been applied in 5G systems [3].

Since sub-6 GHz, millimeter wave (mmWave) and terahertz (THz) frequency bands
shall be widely-used in 6G, the frequency bandwidths at different carrier
frequencies are dramatic, which may substantially increase system complexity
and peak-to-average power ratio (PAPR). In light of hardware constraints and
various XR service requirements, multi-numerology is introduced to make
flexible resource allocation, compared to the uniformly distributing frequency
resource allocation in 4G systems [4]. Its gist is that subcarrier spacing can
be tuned according to the specific frequency bandwidths and service
requirements. To meet the requirements of mission-critical services, [5]
proposes a scalable resource block based on flexible numerology. In [6],
mixed-numerology is adopted in windowed orthogonal frequency division
multiplexing (OFDM) systems. Like traditional single-numerology case, multi-
numerology OFDM systems are still susceptible to the critical PAPR, hence [7]
proposes a numerology scheduling method to reduce the PAPR. The numerology
scheduler of [8] is designed to be adaptive for satisfying different slices’
requirements. In vehicle-to-everything networks, recent work [9] shows that
the use of 5G numerology can improve the vehicles’ Quality-of-Service. On the
other hand, mini-slot is an alternative for further enhancing the network’s
scalability, which achieves much lower latency since it allows shorter
transmission slot duration to contain only a few OFDM symbols, therefore,
mini-slot plays a promising role in ultra-reliable low-latency communication
(URLLC). When URLLC and the enhanced mobile broadband (eMBB) traffics coexist
in multiuser communications systems, [10] provides a media access control
(MAC) layer scheduling for maximizing eMBB utility under URLLC latency
constraint. To flexibly manage the eMBB and URLLC network slices, [11]
investigates the numerology and mini-slot for maximizing the network
throughput while satisfying service level agreement. Meanwhile, [12] develops
a hybrid puncturing and superposition scheme which can maximize the minimum
average throughput of eMBB and URLLC users.

In immersive communications systems, 360∘ video is an important component for
XR applications [13, 2]. However, 360∘ video streaming demands large frequency
bandwidths for low-latency transmissions. Existing research contributions have
attempted to improve the efficiency of 360∘ video streaming under limited
resources constraints. In [14], a two-tier 360-degree video streaming scheme
is investigated to address the network dynamics, such a scheme is source
representation independent and outperforms the DASH streaming and Field of
View (FoV) streaming schemes. Since videos in the FoV region must be
delivered, [15] analyzes the future FoV region in a closed-form under a given
confidence level. By considering the tile-based 360-degree video streaming,
[16] adopts a machine learning method to optimize different Quality of
Experience (QoE) objectives. A viewport-aware adaptive tiling scheme is
studied in [17], where the long-term user QoE is maximized. In [18], energy
efficiency of delivering multi-quality tiled 360 virtual reality (VR) videos
is enhanced by jointly determining appropriate beamforming, subcarrier and the
choices of quality level. Latest work [2] designs flexible frame structures
for achieving the two-tier 360-degree video delivery and improving the system
robustness.

Motivated by the aforementioned studies, this paper aims to establish the
scalable two-tier 360-degree video streaming in multiuser immersive
communications systems by leveraging multi-numerology and mini-slot. To the
best of our knowledge, this is the first work to exploit the merits of joint
multi-numerology and mini-slot design for 360-degree video delivery. We
orchestrate a flexible time-frequency resource block structure and propose a
novel time-frequency allocation method to maximize the total quality-of-
experience (QoE) under diverse levels of UE’s QoE constraint, which
significantly outperforms the equal resource allocation.

##  II System Descriptions

In a downlink immersive communications system, base station (BS) delivers
360-degree video contents to $N$ user equipment (UEs). To meet different UEs’
QoE, we manage time-frequency resources via numerology and mini-slot.
According to the 5G new radio (5G NR) [3, 19, 20], each subframe duration is
fixed value of 1ms, the OFDM’s subcarrier spacing is $15\times 2^{\mu}$kHz
with the numerology $\mu$ (a non-negative integer value for low-latency XR
services), and each resource block has 12 consecutive subcarriers with the
same numerology in the frequency domain. Note that each slot contains 14 OFDM
symbols, and 4G LTE’s numerology is $\mu=0$ with the slot duration of 1ms [3].
For generality, we consider that the $n$-th UE’s mini-slot duration can be
selected to contain $\eta_{n}$ OFDM symbols, and each slot is divided into
several mini-slots whose lengths may be different. As such, we design a new
time-frequency resource block configuration (the unit is
ms$\times$kHz=s$\times$Hz) to achieve scalable multiuser immersive
communications, which is defined as follows:

$\displaystyle{\rm C}^{\rm RB}$ $\displaystyle=\Delta T_{\rm min}\times\Delta
B_{\rm min}$ $\displaystyle=\frac{1}{14\times 2^{\mu_{\rm
max}}}\times\left(12\times 15\times 2^{\mu_{\rm min}}\right),$ (1)

where $\Delta T_{\rm min}=\frac{1}{14\times 2^{\mu_{\rm max}}}$ms is the
minimum OFDM symbol duration that can be supported in the system with the
maximum numerology value $\mu_{\rm max}$, considering the fact that the OFDM
symbol duration changes inversely to its subcarrier spacing; $\Delta B_{\rm
min}=12\times 15\times 2^{\mu_{\rm min}}$kHz is the minimum bandwidth part in
the frequency domain with the minimum numerology value $\mu_{\rm min}$, here,
the bandwidth part is a subset of resource blocks for a given numerology [20].
The advantage of such a resource block design is that it solely depends on the
system’s maximum and minimum numerology values regardless of UEs’ specific XR
service requirements, in this case, the size of a bandwidth part (the unit is
ms$\times$kHz=s$\times$Hz) occupied by the $n$-th UE with the required
numerology $\mu_{n}$ and mini-slot $\eta_{n}$ can be easily calculated as

$\displaystyle{\rm C}^{\rm BWP}$ $\displaystyle=\left(\eta_{n}\times
2^{\mu_{\rm max}-\mu_{n}}\Delta T_{\rm
min}\right)\times\left(2^{\mu_{n}-\mu_{\rm min}}\Delta B_{\rm min}\right)$
$\displaystyle=\eta_{n}\times 2^{\mu_{\rm max}-\mu_{\rm min}}\times{\rm
C}^{\rm RB}.$ (2)

Each bandwidth part is dedicated to one UE and there is no overlap between
them. From (II), the number of resource blocks in each UE’s allocated
bandwidth part only depends on its mini-slot for a given immersive
communications system, as illustrated in Fig. 1.

Figure 1: An illustration of time-frequency resource structure with multi-
numerology ($\mu\in\\{0,1,2\\}$ in this figure) and the same mini-slot value
$\eta=2$ for all the UEs, in which each rectangle of the resource grid
represents a resource block given by (II), and each UE’s allocated bandwidth
part has the same number of resource blocks given by ${\rm C}^{\rm BWP}=8{\rm
C}^{\rm RB}$ according to (II), regardless of each UE’s numerology.

In the two-tier 360-degree video streaming system [2], each video frame has
the duration of $T$ (ms) and frequency bandwidth $B$ (kHz), which is
partitioned into basic tier (BT) video chunk phase and and enhancement tier
(ET) video chunk phase [14]. Let $\mu_{n}^{\rm BT}$, $\eta_{n}^{\rm BT}$,
$\mu_{n}^{\rm ET}$, and $\eta_{n}^{\rm ET}$ denote the BT’s and ET’s
numerology and mini-slot, respectively. Based on (II), we have the following
time resource constraint

$\displaystyle\sum\limits_{(i,i^{\prime})\in\mathcal{\\{L,R\\}},\hfill\atop(i,i^{\prime})\notin\mathcal{\\{B,A\\}}}\eta_{n,i}^{\rm
BT}2^{\mu_{\rm max}-\mu_{n,i}^{\rm BT}}\Delta T_{\rm min}+$
$\displaystyle\quad\sum\limits_{(j,j^{\prime})\in\mathcal{\\{L,R\\}},\hfill\atop(j,j^{\prime})\notin\mathcal{\\{B,A\\}}}\eta_{n,j}^{\rm
ET}2^{\mu_{\rm max}-\mu_{n,j}^{\rm ET}}\Delta T_{\rm min}\leq T,\;\;\;\forall
n,$ (3)

and frequency resource constraint

$\displaystyle\sum\limits_{(i,i^{\prime})\in\mathcal{\\{B,A\\}},\hfill\atop(i,i^{\prime})\notin\mathcal{\\{L,R\\}}}2^{\mu_{n,i}^{\nu}-\mu_{\rm
min}}\Delta B_{\rm min}\leq B,\;\;\;\forall n,\nu\in\\{\rm BT,\rm ET\\},$ (4)

where $\eta_{n,i}^{\rm BT}$ are the numerology of the $i$-th bandwidth part at
BT and $\eta_{n,j}^{\rm ET}$ is the numerology of the $j$-th bandwidth part at
ET for the UE $n$, the relative positioning constraint
$(i,i^{\prime})\in\mathcal{\\{L,R\\}}$ means that the bandwidth part $i$ is
the left or right of the bandwidth part $i^{\prime}$, i.e., there is no
overlap between them in the time domain, the relative positioning constraint
$(i,i^{\prime})\in\mathcal{\\{B,A\\}}$ means that the bandwidth part $i$ is
the below or above of bandwidth part $i^{\prime}$, i.e., there is no overlap
between them in the frequency domain,

Since the time-frequency resources are orthogonally allocated to the UEs (no
co-channel interference), the BT’s transmission rate (bits per frame) is given
by

$\displaystyle R_{n}^{\rm BT}=\sum\limits_{i}{\rm{C}}_{{\rm
BT},n,i}^{\rm{BWP}}\log_{2}\left(1+\frac{G_{n}^{\rm t}G_{n}^{\rm
r}\left|\hbar_{n}^{\rm BT}\right|^{2}p_{n}}{\delta^{2}}\right),$ (5)

where ${\rm{C}}_{{\rm BT},n,i}^{\rm{BWP}}$ is the $i$-th bandwidth part
allocated to the $n$-th UE at BT with the mini-slot $\eta_{n,i}^{\rm BT}$,
$G_{n}^{\rm t}$ and $G_{n}^{\rm r}$ are the effective transmit antenna gain
and receive antenna gain obtained by the UE $n$, respectively,
$\left|\hbar_{n}^{\rm BT}\right|^{2}$ and $\left|\hbar_{n}^{\rm
ET}\right|^{2}$ are the large-scale fading channel power gains111In the highly
directional beamforming transmissions such as millimeter wave (mmWave), the
effect of small-scale fading on the channel power gain could be ignored.
$p_{n}$ is the transmit power spectral density (PSD) of the downlink channel
between the BS and the $n$-th UE, and $\delta^{2}$ is the noise’s PSD.
Likewise, ET’s transmission rate (bits per frame) is given by

$\displaystyle R_{n}^{\rm ET}=\sum\limits_{j}{\rm{C}}_{{\rm
ET},n,j}^{\rm{BWP}}\log_{2}\left(1+\frac{G_{n}^{\rm t}G_{n}^{\rm
r}\left|\hbar_{n}^{\rm BT}\right|^{2}p_{n}}{\delta^{2}}\right),$ (6)

where ${\rm{C}}_{{\rm ET},n,j}^{\rm{BWP}}$ is the $j$-th bandwidth part
allocated to the $n$-th UE at ET with the mini-slot $s_{n,j}^{\rm ET}$. We
further highlight that the proposed resource block and bandwidth part designs
facilitate the analysis of transmission rate in multiuser communications since
(5) and (6) are available for arbitrary UEs’ numerologies.

One of the key performance indicators (KPIs) in the two-tier video delivery
system is the QoE, which is commonly-measured in a logarithmic manner [14,
17], i.e., the QoE for the $n$-th UE is calculated as

$\displaystyle\widetilde{Q}_{n}=\left(1-\rho_{n}\right)Q_{n}\left(\hat{R}_{n}^{\rm
BT}\right)+\rho_{n}Q_{n}\left(\Xi_{n}\right),$ (7)

where $\rho_{n}$ ($0\leq\rho_{n}\leq 1$) is the probability of precisely
predicting UE’s FoV, $Q_{n}\left(x\right)=a_{n}+b_{n}\log\left(x\right)$ with
specific XR video-dependent constant parameters $a_{n}$ and $b_{n}$,
$\hat{R}_{n}^{\rm BT}=\frac{R_{n}^{\rm BT}}{\mathcal{C}^{\rm BT}}$ with the
coverage area of the 360∘ video $\mathcal{C}^{\rm BT}$,
$\Xi_{n}=\hat{R}_{n}^{\rm BT}+\hat{R}_{n}^{\rm ET}$ denotes the effective rate
after using layered coding to generate the ET chunks, here $\hat{R}_{n}^{\rm
ET}=\frac{R_{n}^{\rm ET}}{\mathcal{C}^{\rm ET}}$ with the coverage area of the
ET chunk $\mathcal{C}^{\rm ET}$.

By appropriately selecting numerology and mini-slot, our aim is to maximize
the total QoE while achieving the minimum QoE per active UE, thus the time-
frequency resource allocation problem is formulated as

$\displaystyle\mathop{\max}\limits_{\mathbf{x},\boldsymbol{\eta,\mu}}\sum\limits_{n=1}^{N}x_{n}\widetilde{Q}_{n}$
(8) $\displaystyle\mathrm{s.t.}\leavevmode\nobreak\
\mathrm{C1:}\leavevmode\nobreak\
x_{n}\in\\{0,1\\},x_{n}\left(Q_{n}\left(\hat{R}_{n}^{\rm
BT}\right)-\overline{Q}_{n}^{\rm min}\right)\geq 0,\;\;\;\forall n,$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \mathrm{C2:}\leavevmode\nobreak\
Q_{n}\left(\hat{R}_{n}^{\rm BT}\right)\leq\gamma_{n}^{\rm
peak}\overline{Q}_{n}^{\rm min},\;\;\;\forall n,$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \mathrm{C3:}\leavevmode\nobreak\
{\rm Eq.(3)},\;\;{\rm Eq.(4)},$ $\displaystyle\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \mathrm{C4:}\leavevmode\nobreak\
{\rm{C}}_{{\nu},n,i}^{\rm{BWP}}\bigcap{\rm{C}}_{{\nu^{\prime}},n^{\prime},i^{\prime}}^{\rm{BWP}}=\emptyset,\;\;\;\forall
n,\nu,\nu^{\prime}\in\\{\rm BT,\rm ET\\},$ $\displaystyle\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \mathrm{C5:}\leavevmode\nobreak\
\eta_{n,i}^{\nu}\in\mathbb{Z},\eta_{n,i}^{\nu}\leq
14,\mu_{n,i}^{\nu}\in\mathbb{Z},\;\;\;\forall n,\nu,\in\\{\rm BT,\rm ET\\},$

where $\mathbf{x}=[x_{n}]$ are binary values that indicate whether UE is
served or not, $\boldsymbol{\eta}=[\eta_{n,i}^{\rm BT},\eta_{n,i}^{\rm ET}]$
and $\boldsymbol{\mu}=[\mu_{n,i}^{\rm BT},\mu_{n,i}^{\rm ET}]$. Constraint
$\mathrm{C1}$ makes sure the served UEs meet the minimum QoE value
$\overline{Q}_{n}^{\rm min}$ and the solutions of problem (8) are always
feasible; $\mathrm{C2}$ shows that the QoE for entire 360∘ view at BT needs to
be restricted to below a peak value $\gamma_{n}^{\rm
peak}\overline{Q}_{n}^{\rm min}$ in practice, since QoE for viewport videos at
ET is more important; $\mathrm{C3}$ is the limitation of available time-
frequency resources; $\mathrm{C4}$ illustrates that there is no overlap among
bandwidth parts; In constraint $\mathrm{C5}$, mini-slot $\eta_{n,i}^{\nu}\leq
14$ means that each slot consists of several mini-slots and there are always
14 OFDM symbols per slot regardless of the specific numerology [3, 19].

##  III DRL-based Algorithm Design

Problem (8) is combinatorial with non-overlap constraints, which is
challenging to solve. Since deep reinforcement learning (DRL) is a powerful
machine learning tool to deal with discrete decision-making problems, we
provide a DRL-based solution, in which an agent (link between the UE and BS)
interacts with its environment, and the key components of RL are detailed as
follows:

  * •

State: In a given resource grid with time and frequency boundaries, the
observed state space at BS is denoted by
$\mathcal{S}=\\{\boldsymbol{\Theta,\Psi,\aleph,\Phi}\\}$, where
$\boldsymbol{\Theta}$ contains the allocated bandwidth parts and idle ones,
$\boldsymbol{\Psi}$ contains the allocated BT and ET’s bandwidth parts for
each UE and the corresponding QoE $Q\left(\hat{R}^{\rm BT}\right)$ and
$\widetilde{Q}$ given by (7), $\aleph$ contains the order of serving UE at
each time-step, and $\Phi$ contains the time and frequency boundary lines of
the allocated resources at the beginning of each episode. The state vector at
each time-step is $\mathbf{s}_{t}\in\mathcal{S}$.

  * •

Actions: The action $a_{t}$ taken by the agent include the numerology $\mu$,
mini-slot $\eta$ and their corresponding bandwidth part ${\rm C}^{\rm BWP}$
given by (II) (as illustrated Fig. 1). Given the state $\mathbf{s}_{t}$, agent
chooses feasible numerology and mini-slot values based on constraints
$\mathrm{C1}$ and $\mathrm{C5}$ (Note that both the objective and constraint
$\mathrm{C1}$ of problem (8) only depend on the selected mini-slot thanks to
our resource block and bandwidth part designs in Section II.), and thus
determines feasible bandwidth parts based on the current time and frequency
boundary lines, to satisfy the constraints $\mathrm{C3}$-$\mathrm{C4}$.

  * •

Reward: When the action is executed, agent obtains a reward. Let $M$ denote
the maximum allowable number of time-steps before terminating an episode, and
the reward function is given by

$\displaystyle
r_{n,t}=\left\\{\begin{array}[]{l}\varrho\Delta\widetilde{Q}_{n}\left(t\right)+\left(1-\varrho\right)\mathcal{Z},\;{\rm{if}}\;{\rm{C1-C5\leavevmode\nobreak\
are\leavevmode\nobreak\ met}},\\\
{\rm\mathcal{H}},\;{\rm{if\leavevmode\nobreak\ episode\leavevmode\nobreak\
terminates}}\;{\rm{and\leavevmode\nobreak\ C1-C5\leavevmode\nobreak\
are\leavevmode\nobreak\ met}},\\\ \Theta,\;{\rm otherwise},\\\
\end{array}\right.$ (12)

where
$\Delta\widetilde{Q}_{n}\left(t\right)=\widetilde{Q}_{n}\left(t\right)-\widetilde{Q}_{n}\left(t-1\right)$
is the incremental value of the agent’s QoE at time-step $t$, $\mathcal{Z}$ is
a penalty to enable that the agent is time-aware (namely limited number of
time-steps for the agent’s interaction with the environment) [8, 21], which is
much helpful for system stability and low-latency immmersive communications,
$\varrho$ is the weighting parameter, a high reward value ${\rm\mathcal{H}}$
is obtained if the episode terminates and constraints ${\rm{C1}}$-${\rm{C5}}$
are met, otherwise incurring a penalty $\Theta$.

Figure 2: Deep convolutional neural network structure in the proposed DQN-
based method.

In light of deep Q-learning [22], we propose a DRL-based algorithm to solve
the problem (8), which is detailed in Algorithm 1. The deep Q-network of
Algorithm 1 is illustrated in Fig. 2.

1 Initialize the maximum number of episodes $E$, the maximum allowable number
of time-steps $M$ per episode, and deep Q-network (shown in Fig. 2); the
episode index $e=1$

2 while  _$e\leq E$_ do

3 initialize the state space $\mathcal{S}$, in which $\aleph$ is initialized
by sorting the UEs in descending order of their QoE levels under equal
resource allocation;

4 for  _$t=1,\cdots,M$_ do

5 a) Select a random action $a_{n,t}$ according to $\varepsilon$-greedy
policy. Note that some actions need to be excluded when the constraints
$\mathrm{C1}$-$\mathrm{C5}$ are not met;

6 b) After executing the selected action, obtain a reward given by (12) and
observe the next state $\mathbf{s}_{t+1}$;

7 if  _constraint C2 is violated_ then

8 the corresponding link is no longer considered as an agent at BT phase in
this episode;

9 end if

10 c) Store the transitions
$\\{\mathbf{s}_{t},{\mathbf{a}}_{t},\mathbf{r}_{t},\mathbf{s}_{t+1}\\}$ into
memory;

11 d) Train the deep Q-network using minibatch of transitions data from the
memory;

12 e) Update the policy $\pi$: $\mathcal{S}\rightarrow\mathbf{a}$;

13

14 end for

15 iii) $e=e+1$;

16 end while

The corresponding action $\mathbf{a}$ is obtained

Algorithm 1 DRL based Time-frequency Resource Allocation Table I: Simulation parameters BT view coverage | $\mathcal{C}^{{\rm{BT}}}=360^{\circ}\times 180^{\circ}$
---|---
ET view coverage | $\mathcal{C}^{{\rm{ET}}}=135^{\circ}\times 135^{\circ}$
mmWave carrier frequency |  $f_{c}=28$GHz
System bandwidth |  $B=69.12$MHz
Each video frame duration |  $T=0.0625$ms
Effective transmit antenna gain per UE video |  $G_{\rm t}=15$dBi
Effective receive antenna gain per UE video |  $G_{\rm r}=10$dBi
Large-scale channel fading power gain | $\left|\hbar_{n}\right|^{2}=\left({\frac{3\times 10^{8}}{{4\pi f_{c}}}}\right)^{2}\times d_{n}^{-2}$
Noise’s PSD |  $\delta^{2}=-169$dBm/Hz
Total transmit PSD |  $p_{\rm total}=-47$dBm/Hz
Vector of four UEs’ minimum QoE levels | $\overline{\mathbf{Q}}_{1\times 4}=\left[4.9,4.6,4.8,4,6\right]$
Peak levels of four UEs’ QoE at BT |  $5\overline{\mathbf{Q}}_{1\times 4}$ with $\gamma_{n}^{\rm peak}=5$
Numerology | $\mu=\\{4,5,6\\}$
Mini-slot | $\eta=\\{2,4,7\\}$

##  IV Simulation Results

This section provides numerical results to confirm the efficiency of the
proposed DRL-based time-frequency resource allocation in Section III. The
learning parameters include: learning rate is 0.001, the maximum allowable
number of time-steps per episode is $M=1000$, the parameters in the reward
function are set as $\varrho=0.5$, $\mathcal{Z}=-0.01$,
${\rm\mathcal{H}}=500$, and $\Theta=-2$, respectively, the discount factor
$\gamma=0.99$, and deep Q-network in Fig. 2 are trained using Adam optimizer.
The system parameters are detailed in Table I. In the simulations, the
communication distance $d_{n}\geq 1$ from the BS to the $n$-th UE is uniformly
distributed with the cell radius $200$m, the probability of precisely
predicting UE’s FoV $\rho_{n}$ follows the truncated normal distribution with
the truncation interval [0.6,1] (its mean and variance of the parent general
normal probability density function are 0.8 and 0.49, respectively), and
$a_{n}=0$, $b_{n}=1$, $\forall n$. The results are obtained by averaging over
2000 trials.

###  IV-A Convergence

Figure 3: The convergence of the proposed DRL-based method.

Fig. 3 confirms that the proposed DRL-based method converges at a fast speed.
Moreover, the input dimensionality is not large based on our resource block
design given by (II) ($56\times 24$ resource blocks in the simulations), which
relieves the burden of computation and memory. The maximum average award
becomes stable when the number of episodes is greater than $E=310$.

Figure 4: The feasibility of the proposed DRL-based method.

###  IV-B Feasibility

Fig. 4 confirms that the proposed method achieves the feasibility and
satisfies the constraints $\mathrm{C1}$-$\mathrm{C5}$ of problem (8), namely
the minimum QoE of the served UEs are guaranteed, and the percentage of the
served UEs is almost 100$\%$ as the number of episodes is larger than 310,

###  IV-C Efficiency

Figure 5: The performance of the proposed DRL-based method.

Fig. 5 confirms the efficiency of the proposed method compared to the equal
resource allocation cases. In the simulations, the number of episodes is set
as $E=510$ based on the empirical results in Figs. 3 and 4, the equal
bandwidth allocation case considers free-viewpoint without FoV predication [2]
(BS sends the entire 360-degree videos to the UEs.), and the equal time-
frequency resource allocation case considers that the frequency bandwidths are
equally allocated to the UEs and time of each UE’s BT is the half of a video
frame duration. We see that the proposed DRL-based method outperforms the
benchmark cases. In addition, the equal time-frequency resource allocation
case obtains the worst QoE performance, due to the fact that in the equal
time-frequency resource allocation case, the minimum QoE of some UEs cannot be
met when time resources are not enough for BT transmissions, i.e., less UEs
are served in this case.

##  V Conclusions

In multiuser immersive communications, diverse service requirements posed a
great challenge to the time-frequency resource management. To address this
issue, this paper focused on the scalable delivery of 360-degree video. Based
on the numerology and mini-slot techniques, a flexible resource block
structure has been designed, which only depends on the system’s maximum and
minimum numerology. Then, a DRL-based method was developed to optimize the
numerology and mini-slot values for maximizing the network QoE. The results
confirmed the advantages of our proposed method, moreover, the proposed
resource block design was very useful for facilitating the performance
analysis and reducing the system costs.

## References

  * [1] J. G. Apostolopoulos, P. A. Chou, B. Culbertson, T. Kalker, M. D. Trott, and S. Wee, “The road to immersive communication,” _Proc. IEEE_ , vol. 100, no. 4, pp. 974–990, April 2012.
  * [2] M. Hu, L. Wang, B. Tan, and S. Jin, “Two-tier 360-degree video delivery control in multiuser immersive communications systems,” _IEEE Trans. Veh. Technol._ , vol. 72, no. 3, pp. 4119–4123, Mar. 2023.
  * [3] A. A. Zaidi, R. Baldemair, V. Moles-Cases, N. He, K. Werner, and A. Cedergren, “OFDM numerology design for 5G new radio to support IoT, eMBB, and MBSFN,” _IEEE Commun. Stand. Mag._ , vol. 2, no. 2, pp. 78–83, 2018.
  * [4] P. Guan, D. Wu, T. Tian, J. Zhou, X. Zhang, L. Gu, A. Benjebbour, M. Iwabuchi, and Y. Kishiyama, “5G field trials: OFDM-based waveforms and mixed numerologies,” _IEEE J. Sel. Areas Commun._ , vol. 35, no. 6, pp. 1234–1243, June 2017.
  * [5] L. You, Q. Liao, N. Pappas, and D. Yuan, “Resource optimization with flexible numerology and frame structure for heterogeneous services,” _IEEE Commun. Lett._ , vol. 22, no. 12, pp. 2579–2582, Dec 2018.
  * [6] X. Zhang, L. Zhang, P. Xiao, D. Ma, J. Wei, and Y. Xin, “Mixed numerologies interference analysis and inter-numerology interference cancellation for windowed OFDM systems,” _IEEE Trans. Veh. Technol._ , vol. 67, no. 8, pp. 7047–7061, Aug. 2018.
  * [7] E. Memisoglu, A. E. Duranay, and H. Arslan, “Numerology scheduling for PAPR reduction in mixed numerologies,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 6, pp. 1197–1201, June 2021.
  * [8] K. Boutiba, M. Bagaa, and A. Ksentini, “Radio resource management in multi-numerology 5G new radio featuring network slicing,” in _Proc. IEEE ICC_ , 2022, pp. 359–364.
  * [9] T.-S.-L. Nguyen, S. Kallel, N. Aitsaadi, C. Adjih, and I. Fajjari, “A flexible numerology configuration for efficient resource allocation in 3GPP V2X 5G new radio,” in _IEEE GLOBECOM_ , 2022, pp. 4449–4454.
  * [10] H. Yin, L. Zhang, and S. Roy, “Multiplexing URLLC traffic within eMBB services in 5G NR: Fair scheduling,” _IEEE Trans. Commun._ , vol. 69, no. 2, pp. 1080–1093, Feb. 2021.
  * [11] M. Setayesh, S. Bahrami, and V. W. Wong, “Resource slicing for eMBB and URLLC services in radio access network using hierarchical deep learning,” _IEEE Trans. Wireless Commun._ , vol. 21, no. 11, pp. 8950–8966, Nov. 2022\.
  * [12] M. Darabi, V. Jamali, L. Lampe, and R. Schober, “Hybrid puncturing and superposition scheme for joint scheduling of URLLC and eMBB traffic,” _IEEE Commun. Lett._ , vol. 26, no. 5, pp. 1081–1085, May 2022.
  * [13] A. Yaqoob, T. Bi, and G.-M. Muntean, “A survey on adaptive 360∘ video streaming: Solutions, challenges and opportunities,” _IEEE Commun. Surv. Tutor._ , vol. 22, no. 4, pp. 2801–2838, 2020.
  * [14] L. Sun, F. Duanmu, Y. Liu, Y. Wang, Y. Ye, H. Shi, and D. Dai, “A two-tier system for on-demand streaming of 360 degree video over dynamic networks,” _IEEE J. Emerg. Sel. Topics Circuits Syst._ , vol. 9, no. 1, pp. 43–57, July 2019.
  * [15] T. C. Nguyen and J.-H. Yun, “Predictive tile selection for 360-degree VR video streaming in bandwidth-limited networks,” _IEEE Commun. Lett._ , vol. 22, no. 9, pp. 1858–1861, Sept. 2018.
  * [16] Y. Zhang, P. Zhao, K. Bian, Y. Liu, L. Song, and X. Li, “DRL360: 360-degree video streaming with deep reinforcement learning,” in _Proc. IEEE INFOCOM_ , 2019, pp. 1252–1260.
  * [17] N. Kan, J. Zou, C. Li, W. Dai, and H. Xiong, “Rapt360: Reinforcement learning-based rate adaptation for 360-degree video streaming with adaptive prediction and tiling,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 32, no. 3, pp. 1607–1623, Mar. 2022.
  * [18] C. Guo, L. Zhao, Y. Cui, Z. Liu, and D. W. K. Ng, “Power-efficient wireless streaming of multi-quality tiled 360 VR video in MIMO-OFDMA systems,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 8, pp. 5408–5422, Aug. 2021\.
  * [19] L. Marijanović, S. Schwarz, and M. Rupp, “Multiplexing services in 5G and beyond: Optimal resource allocation based on mixed numerology and mini-slots,” _IEEE Access_ , vol. 8, pp. 209 537–209 555, Nov 2020.
  * [20] 3GPP TS 38.211, “NR; Physical channels and modulation (Release 176),” June 2023\.
  * [21] F. Pardo, A. Tavakoli, V. Levdik, and P. Kormushev, “Time limits in reinforcement learning,” in _Pro. ICML_ , 2018, pp. 1–10.
  * [22] V. Mnih _et al._ , “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, Feb. 2015.


# HEP Software Foundation Community White Paper Working Group – Training,
Staffing and Careers

Dario Berzano 2 Riccardo Maria Bianchi 3 Peter Elmer 4 Sergei V. Gleyzer 1
John Harvey 5 Roger Jones 6 Michel Jouvin 7 Daniel S. Katz 8 Sudhir Malik 9
Dario Menasce 7 Mark Neubauer 10 Fernanda Psihas 11,a Albert Puig Navarro 1
Graeme A. Stewart 12 Christopher Tunnell 10 Justin A. Vasel 4 and Sean-Jiun
Wang CERN, Geneva, SwitzerlandDepartment of Physics and Astronomy, University
of Pittsburgh, Pittsburgh, PA, USAPrinceton University, Princeton, NJ,
USAUniversity of Florida, Gainesville, FL, USALancaster University, Lancaster,
UKLAL, Université Paris-Sud and CNRS/IN2P3, Orsay, FranceUniversity of
Illinois Urbana-Champaign, Urbana, IL, USAUniversity of Puerto Rico, Mayaguez,
PR, USAINFN Sezione di Milano-Bicocca, ItalyDepartment of Physics, Indiana
University, Bloomington, IN, USAPhysik-Institut, Universität Zürich, Zürich,
SwitzerlandUniversity of Chicago, Chicago, IL, USASupported by SNF under
contract 168169.

###### Abstract

The rapid evolution of technology and the parallel increasing complexity of
algorithmic analysis in HEP requires developers to acquire a much larger
portfolio of programming skills. Young researchers graduating from
universities worldwide currently do not receive adequate preparation in the
very diverse fields of modern computing to respond to growing needs of the
most advanced experimental challenges. There is a growing consensus in the HEP
community on the need for training programmes to bring researchers up to date
with new software technologies, in particular in the domains of concurrent
programming and artificial intelligence. We review some of the initiatives
under way for introducing new training programmes and highlight some of the
issues that need to be taken into account for these to be successful.

|  | HSF-CWP-2017-02
---|---|---
|  | June 20, 2018

##  1 Introduction

The field of high energy physics (HEP) has consistently exploited the latest
innovations in computational tools and technologies for processing data. The
software toolset of the particle physicist is ever-growing, and the problem
set increasingly complex. Young researchers will likely encounter a number of
programming techniques with which they are totally unfamiliar, and will
therefore need “on-the-job” training in order for them to be productive.

Evidently a strong research community comprises independent thinkers who ask
research questions and direct their own analyses, instead of blindly following
prescriptions and recipes. It is therefore imperative that training
opportunities and resources are made available, such that adequate career path
possibilities exist for people in HEP who would otherwise leave the field due
to limited advancement opportunities.

To produce the best possible science, it is important that physicists can
easily acquire essential software engineering skills. The nature and level of
the expertise a physicist needs to acquire will vary according to the problems
that need to be tackled. For example, basic data analysis requires familiarity
with the Python programming language and with a broad range of data analysis
libraries in current usage. However, those wishing to contribute to large open
source projects, such as ROOT, require greater training in the use of software
engineering practices that cover the full development life-cycle. Clearly a
thorough understanding of the physics problem domain is essential, making it
necessary to provide a career path for those young physicists willing to
invest time and effort in becoming software specialists.

In each HSF workshop, a consensus on the need for proper tutoring, training,
and relevant career possibilities emerged as the basic ingredients for
ensuring success. The main requirements and challenges to be faced were
identified as follows:

  * •

to encourage and provide incentives to the members of the HEP community to
train students and other collaborators,

  * •

to properly assign credit to software development as a scientific discipline,
including training activities,

  * •

to establish policies for the hiring and long-term retention of researchers
specialising in computing,

  * •

to address the gap in formal software training that is not always given by
universities as part of a physicist’s education.

##  2 Career Paths for Software Experts

The introduction and development of sophisticated software tools has led to
improvements in the performance of our data processing software. The personnel
who drive these advances through novel implementations work on a broad range
of tasks, including the development of high speed DAQ systems, of modern
databases for storing the state of a detector, the management of data flow
from the first trigger level to the final analysis dataset, as well as the
development of algorithms and mathematical tools for extracting publishable
physics results.

As tools become more complex, physicists must be continuously retrained in
order to utilize them effectively. It is important to find a way to deploy
training efforts holistically and cross-experimentally, whenever possible, in
order to avoid unnecessary duplication of the effort involved. This will
require organization and coordination to ensure that different software teams
can agree on common standards, thereby allowing different experiments to adopt
the same solutions in an efficient way.

Beyond the immediate benefit to the scientific community in having well-
trained collaborators, most people who start a physics graduate programme will
have careers outside of academia. This trend is reflected in several calls for
proposals from national scientific funding agencies that promote training in
computer infrastructure areas, such as the EU’s Horizon 2020 programme and
those from the National Science Foundation (NSF) NSF1 (1, 2).

A key goal is to find incentives that will encourage increasing numbers of
people to dedicate their time and effort to train their colleagues. To be
effective, the training must be done by people who work at the cutting edge of
the technology and who in many cases are found among the youngest researchers.
These are also the very same people who are most in need of officially
recognized credits for the advancement of their careers. Currently, visibility
and recognition is given mainly to those working on data analysis projects,
rather than to the development and support of the underlying software. One
solution to this problem would be the establishment of specific career paths
for researchers specialising in scientific software development. However, the
practical implementation of such an approach is extremely challenging, as
there are a wide range of institutions involved that belong to different
countries with their own policies, priorities, and funding strategies.

Broadly speaking, two scientific profiles of researchers could be envisaged. A
physicst could make a detailed plan of what they need, expressed in a tidy
requirements document, and a computer scientist could use this document to
provide the required solution. This is how the software development process
typically works in the software industry. However, in HEP requirements are
rapidly changing, forcing developers and physicsts to interact closely during
the entire process of software development. This works smoothly only when both
communities have the same goals and speak the same language, hence the need
for physicists with a good knowledge of computer science.

The first possible career path would be that of a _physicist with computing
science specialisation_ , also known as a physicist-programmer in some
communities. (It would be conceptually different to that of a _computer
scientist_ as the two have different goals, seek different paths to the
solution of their problems, and usually do not even share the same language.)
Such people would have an active role in physics analysis, and so meet many of
the criteria needed for an academic career path.

The second would be the path of a domain-specific _software engineer_. Such a
person has the primary role of developing software and finding software
solutions, but with a large domain-specific knowledge to understand relevant
use cases and the available solutions. Such posts are almost impossible to
establish in current academic institutions; rather being seen by many funding
agencies and universities as short term technical positions. This does not
allow domain specific expertise to be aquired or retained in our field.

Both career paths suffer in academic terms from few or poorly cited
publications, few opportunities to win grants in their own right and little
opportunity for impact with commerce and wider society. These three failings
make an academic appointment challenging or disfavoured as compared to other
applicants. However, there are measures that can be taken to address all three
weaknesses, which we expand on in the remainder of this document.

##  3 Training needs of the community

The HEP community consists of people with diverse software experience,
interests, and time availablity to learn new techniques. Any training
programme must take into account the variation in target audiences. For
example, an undergraduate doing a summer research project has different needs
and skills than their professor.

###  3.1 Classification of trainees

For purposes of training we can broadly classify four different experience
levels:

  * •

_Beginner_ : New collaborators with no knowledge of the tools or techniques
they are expected to use. These are people in need of some kind of formal
training in modern computing techniques, such as compiled and scripting
languages, together with operating system basics such as filesystems, version
control, and command-line shells.

  * •

_Intermediate_ : People with some experience in concepts and tools, but
looking to supplement their experience with more recent and modern approaches.

  * •

_Advanced_ : Experts who have mastered current technologies and
implementations and who want to stay up-to-date with new advanced
developments.

  * •

_Software Specialists_ : HEP scientists in charge of software development in
areas not limited only to analysis, such as DAQ systems, computing
infrastructure, databases, pattern recongition, and complete frameworks.

Each of these groups has very different training needs. However, _whenever
possible_ , any training programme should take advantage of developments in
pedagogy, such as active learning ActiveLearning (3), peer learning
PeerLearning (4), and web-based training.111See §4.1. In some cases, it may
even be advantageous to hand out code samples that are purposely broken or
flawed, and ask students to fix or improve them. Learning the material in a
way that sticks is difficult and challenging for both the students and the
instructor and often takes more time than we would prefer. However, this is
the best way to educate scientists who can fully contribute to the physics
programmes at large, which is really the ultimate goal of any training
programme.

###  3.2 Knowledge that needs to be transferred

At all stages of software and computing training, we should take care to
encourage _good practices across the community_ , such as ensuring error
checking, modularity of code design, version control, writing tests, etc. All
the key concepts addressed in training should not be specific to a particular
experiment or field of application, but general enough to be useful to the
whole HEP community and beyond. A number of specific concepts need to be
taught, in order to guarantee the basic level of competence needed to write
efficient code for the various tasks that need to be performed in HEP
experiments. These include programming concepts, data structures, basics of
code design, error checking, code management tools, validation and debugging
tools. More advanced topics include modularity of code design, advanced data
structures, evaluation metrics, writing tests and working with different types
of hardware accelerator. Additionally, special emphasis should be given to
reporting results and documenting them.

Some of the training subjects considered important to pursue are listed in
Appendix A.

##  4 Advancing Training in HEP

The implementation of training should employ different training formats, such
as videos, wikis, lectures, jupyter notebooks and advanced visualizations,
etc., so that people can learn in a familiar and effective manner and in such
a way that experts are encouraged to share their knowledge.

An important point to consider is the difficulty, often experienced in the
past, in developing large software programs across different experiment
collaborations. While there already are experiment-specific training efforts
in place, there are many needs that are in common. Establishing a common
training programme could help facilitate the sharing of exprience amongst
different experiments. This would reduce duplication of efforts and enable
growth of a shared training culture by accumulating and sharing expertise. To
realise this goal, a possibility could be the creation of a _federation_ of
training initiatives, aimed at improving the efficiency and cost-effectivness
of this important activity. An appropriate incentive programme to reward those
who train the community might help to facilitate the goals of such an
initiative.

When trying to exchange training materials within a group the first problem is
to convince authors to contribute their work. This issue can be alleviated by
addressing the correct assignment of intellectual property. Another problem is
that trainers do not like to reuse given material as-is. They usually want to
refactor it, building their own training history. This makes it difficult to
have everyone agree on common approaches. It is even challenging to agree to
host material in the same centralized place. A way to overcome this could be
to settle for a centralized catalog, with each author being free to host their
material in their location of choice.

Another challenge is the wide range of student competence. Special care must
be given to setup a training structure that can manage both introductory and
advanced material. This may be addressed by organising course material as a
large number of independant topics. The IN2P3 authors try to restrict their
tutorials to 25 minutes, as inspired by the Pomodoro
techniquePomodoroTechnique (5), so that one can easily jump and compose one’s
own curriculum. However, it turns out to be tricky to keep such small
tutorials really independant and meaningful when they can be selected at will.

All trainers have also faced the tremendous loss of time in software
installation for any practical exercises as many students have not managed to
prepare their machines in advance. Here containers may bring real progress,
provided that everyone has at least DockerDocker (6) installed on their
machine. JupyterHubJupyterHub (7) is a technology which supports training
sessions without the need for specialist installations.

###  4.1 Initiatives for Future Training Programmes

Some methods that can be used for location independent training include
Massive Online Open Courses (MOOCs), hands-on workshops, online knowledge
bases, expert trainer volunteer networks, and web-based training approaches.

MOOCs can be used to develop an open-source set of tutorials and tools.
Existing online courses such as Udacity Udacity (8) and Coursera Coursera (9)
can be evaluated and exploited by the community. A novel approach such as
WikiToLearn WikiToLearn (10) could also be explored to assess potential
benefits (as has already been attempted in the context of the GridKa School of
Computing GridKa (11), see below).

Experiment-specific and global knowledge bases can be established with
incentives for experts to contribute. They can be open-source so that a lot of
knowledge can be added by the trainees themselves as information is learned;
this is the particular context where an approach such as WikiToLearn could be
of great help.

Question-and-answer websites such as Stack Overflow Stackoverflow (12) for HEP
are also a very useful resource for common problems and questions. This
approach has already been considered by HSF start-up members, and turned out
to be difficult to pursue due to the lack of a critical mass needed by Stack
Overflow, but in the future, boundary conditions might change, making this
approach viable.

Hands-on workshops are an invaluable part of learning how to apply theoretical
concepts in practice. Identifying which of the current workshops are
productive and useful, and if they cover all the topics that need to be
transferred and that are in demand by the students, is an important action
item.

Creating an expert tutor volunteer network is another way to provide training
and support to the community. This of course requires proper recognition, at
least in terms of career prospects as an incentive for young researchers. It
is clear that the best possible tutors are, in principle, those people who are
actively engaged in modern software developments: these are often young
researchers in the first steps of their careers and, in order to be attracted
to devoting substantial time to training and tutoring, they must be assured
that such an effort would be properly recognised in an official way. Such kind
of recognition is not currently implemented, at least not in a standardized or
official way. A possible structure for such a network could be the
establishment of a federation of existing schools, as discussed in §4.3.

###  4.2 Web Based Training

Difficulties that have emerged in the past with respect to implementing
training courses are the lack of funding and the lack of available time by
experts in the field. People with enough expertise or insight usually don’t
have time to devote to prolonged periods of student training, and, even when
they can find time, the cost of setting up a training course in an effective
way is often beyond what is made available by funding agencies (funds for
travel, hosting, setting up a room with a computing infrastructure to allow
interactive hands-on session, etc.) A possible solution is a completely
different approach to training, using a web-based platform to provide training
materials to students. This would be complementary to the already existing and
successful efforts such as the CERN School of Computing’s Bertinoro and KIT
ones.

The web based approach has several advantages over traditional ones:

  * •

Tutors can add material to the web site at a very slow pace (whenever they
find time to do it, one slide a week or a chapter a day) .

  * •

Their material, publicly available on a web-site, can be further expanded by
collaborators (also at their own pace) or, even better and more productively,
by students who decide to contribute new additions, examples, exercises etc.
Such a collaborative effort allows more people to be exposed to training at
any given time, creates a sense of community, and creates bridges between
people in contiguous areas of research. Students can use the same platform to
exchange their own examples, make suggestions and point out interesting
concepts. In such a model, the possible contribution from others to the
training material needs to be moderated and validated with appropriate
policies.

  * •

If complemented by the availability of remote virtual machines (possibly via a
browser), students could have access to examples and exercises that are
already embedded in their own natural environment: all the necessary tools and
libraries needed to implement the exercise will be already available in the
virtual machine (possibly a Docker container). With just a web browser,
students could run complex examples from home, taking advantage of a remote
facility that provides some storage and computing power. Important here is the
concept of “environment”: a Docker container could be set up in such a way
that students will work in an exact replica of the environment they will be
exposed to in their experiment. Moreover, students could be provided with
Docker containers that preserve their modified environment across sessions,
allowing them to develop their skills over a prolonged period of time by
accessing all the files that were made persistent day by day during the
training.

  * •

There would no longer be a need to find the resources to host a school and pay
the tutor(s) (and eventually subsidize the students to participate in training
in a remote location). Students could follow the training at their own pace
from wherever they happen to be. A traditional school only lasts for 5 days
(10 at most) and it is difficult to cover a subject to any significant depth
in such a short time. The web approach, instead, would allow for very long and
in-depth coverage of any kind of subject, and in this sense it could be a
complementary approach to a traditional school. Of particular interest could
be courses such as “Machine Learning”, “Statistical Analyis with ROOT”, or
even just “Good practices in C++” or “Python Programming for scientific
computing”.

  * •

Finally, this approach could allow the creation of browsable repositories of
all training materials, grouped by argument, by relevance, by experiment or
whatever other critieria. Students from all over the world could be exposed to
a large repository of examples, exercises and in general training material
from their own home.

An example of a such a web-based platform already exists, and has been
implemented as an Open Source project (backed by Wikimedia) by a group of more
than 30 Italian students. The project is WikiToLearn WikiToLearn (10). It
hosts training material in several languages, for several disciplines, ranging
from Economics to Physics, Mathematics, and several others. Because it is
based upon wikimedia WikiMedia (13) software, it is very easy to add material
to the site, and to make it appear under a specific topic (such as
Software/Techniques/Machine-Learning) and to manipulate it as if it were a
single document. In the end, students can selectively choose individual
chapters from the site and have the corresponding pdf sent them as a book,
complete with index, content, and chapters.

The adoption of such an approach is made rather easy in WikiToLearn by the
relative simplicity of the wikimedia-based toolset: users contribute their
training material using just a web-browser, and in order to do this
efficiently, the necessary learning curve has been kept appropriately shallow.
An interesting exercise in this context has recently been made by colleagues
of the GridKa School of computing: the material from this year (2017) has been
made publicly available on WTL WikiToLearnGridka (14). This constitutes an
interesting example of what can be accomplished using this platform; it is
just a first example of what is possible, but an inspiring one.

Another example of web-based platform is a collection of online tutorials
(mostly written in French) hosted on the Gitlab IN2P3 server Code-Swim-Coaches
(15). Taking advantage of the Gitlab ability to host Docker images, those
tutorials aim to avoid the “damned installation step” that often absorbs half
of a training session. Similarly to the GridKa School for WikiToLearn, the
annual IN2P3 Computing Days are an opportunity to refresh and extend the
collection of tutorials every year. Future work will focus on English
translation, and the development of a web site which will index the above
tutorials, together with the best recommended external tutorials.

Finally it should be important to evaluate, if and to which extent, the
complementary approaches to training, such as schools and dedicated web-site
courses, could be of mutual benefit, in other words how to make them
efficiently cooperate in the development of a complete training program.

###  4.3 Enhancing Current Training Programmes

To achieve our goals for training the community, we can take advantage of
existing training forums. Resources such as conferences, workshops, and
schools (in person and online) can provide a lot of value for our training
purposes with little effort to set up. We should leverage the existing
training forums that most closely match the HEP community’s needs.

Within the HEP community, there are already some working examples of dedicated
training environments that alternate between general topics and experiment-
specific topics. The LHC Physics Center (LPC) at Fermilab hosts Hands-on
Advanced Tutorial Sessions (HATS) LPCHandsOn (16) throughout the year to
introduce and train participants in topics as diverse as the latest
$b$-tagging algorithms, Git/GitHub, and machine learning. These HATS provide
face-to-face time with instructors and participants at Fermilab, and also
allow remote collaborators to join in by video and complete the same online
exercises. A similar approach is in use in the CMS Data Analysis Schools
(CMSDAS) LPCDataAnalysis (17), a series of week-long workshops that now take
place at multiple labs all over the world and are designed to ramp up new
collaborators in CMS-specific analysis tools while providing some discussion
of the physics as well.

Other examples are CERN School of Computing CERNSchoolOfComputing (18), CERN
OpenLab Software workshops OPENLab (19) education in collaboration with
industry partners, and a series of more advanced topical training courses
provided by MPI Munich and DESY that focus on advanced programming, use of
acceleration hardware and statistical tools including machine learning. This
list includes the Bertinoro Bertinoro (20), GridKa GridKa (11), and CODAS-HEP
CODAS-HEP (21) Schools of Computing.

Over the past decade, MOOCs have been developed by universities and private
organizations. They have been well received by industry and academia. In
addition, they provide a lot of flexibility in terms of cost and use of time;
they are typically free and open for enrollment at any time of the year. Since
the material can be accessed at any time and revisited at any time, they can
be completed at a pace that makes sense, for example, for a physicist who
needs to learn machine leanrning in a piecemeal way.

There are a growing set of MOOCs teaching various subjects. Since there are
many options, there is a wide variety with respect to the depth of the
material and specific tools taught. Exploring these options allows us to
choose which is the right offering for the knowledge needed to work on a
specific experiment. We can pick and choose modules to tailor an appropriate
roadmap of skills to learn. More difficult will be to assemble specific
training material not already avaible elsewhere in an efficient and organized
way, since this requires adequate organization, volunteers and a suitable
infrastructure.

Several industry conferences already exist that bring together those in
academia and industry who are at the cutting edge of these techniques.
Conferences such as NIPS NIPS (22) and PyData PyData (23) provide a focused
place where attendees can learn a lot about machine learning in a short period
of time. Machine learning concepts such as current methods, tools, and
problems facing industry and academia can be learned at conferences. In
addition, conferences are an excellent networking opportunity; attendees can
meet and share ideas with fellow learners and experts. Bonds can be formed
quickly at conferences that can be maintained after the duration of the
conference. These connections to the outside community can be essential since
we will be evolving training materials to ensure that they stay relevant over
time.

For example, at the time of this writing, Coursera Coursera (9) and Udacity
Udacity (8) both provide great machine learning massive open online courses at
no cost. These two courses both provide a great foundation for assimilating
machine learning fundamentals. However, Coursera’s approach emphasizes more
theory (with more math background necessary) and uses MATLAB/Octave while
Udacity’s approach emphasizes more practical aspects using machine learning
techniques using Python tools.

###  4.4 Resources and Incentives

It should be considered that some graduate student advisors, might need to be
encouraged to make sure their students are properly trained. Sometimes,
students are instead pushed to learn the bare minimum to get the work done, at
the expense of a broader training/education curriculum that could actually
yield improved results further down the line. One incentive would be to
provide training programmes that also count as course credit, perhaps as an
elective. This model is already in limited use with some online solutions
TrainingProgram (24), but this is not universal. Discussions should be started
with collaborators at higher education institutions to see what the roadblocks
or opportunities would be for these training sessions to serve double duty. It
should also be considered that not all students will end their career in
research or academia: their contribution to the research activity, as
students, should therefore also provide them knowledge, know-how, and skills
considered a valuable asset by industry, in order to increase their chances of
a career outside research.

Training is something that a large cross-section of the community understands
to be important, but finding time and effort to contribute to this project is
not actively on the radar of most potential volunteers. Providing incentives
for their participation and creating the appropriate platforms can go a long
way to reach a productive training environment. It can be as simple as
inviting someone to give a software tutorial on the subject that they are
familiar with, give a lecture or seminar or contribute to a growing knowledge
base.

Another important incentive is recognition. For younger members of the
community, having the opportunity to create a training resource, such as a
software tutorial or a knowledge base on a particular topic, is very
empowering and motivational to continue the efforts of training others.
Engaging younger members of the community is crucial to long-term success of
HEP training endeavors.

In the context of web-based training, if the HSF helps to constitute a living
collection of online tutorials, we could organize regular events such as a
“Tutorial of the month”, or some sort of “like” system for the students to
support their favorite tutorials. The best authors must be recognized, so that
they can showcase their most popular online tutorials, just the same as their
research publications.

It is also critical to incorporate training into grant proposals so that it
can be connected with other areas such as research and development. Efforts
like DIANA-HEP DIANA-HEP (25) and AMVA4NewPhysics AMVA4NewPhysics (26) that
combine training and software development are good examples of such ideas in
practice. More examples of such efforts are needed.

##  5 Other resources

Software Carpentry SoftwareCarpentry (27) and Data Carpentry DataCarpentry
(28) are two parts of The Carpentries organization that collaboratively build
and teach some of the basic concepts in developing and maintaining software,
and analyzing data, respectively. The materials that they develop and use are
open, and can be customized for science domains (e.g., HEP), or participant
groups (e.g., undergraduates). Their model is that they offer training of
trainers, and then the trainers who have graduated can offer training under
the SC/DC names, though of course, anyone can use the SC/DC materials without
doing it under the SC/DC names. Subjects covered in a typical Data Carpentry
school are given in Appendix B.

While Software Carpentry is leveraged to build a foundation of knowledge for
later more advanced concepts, it is important to note much of this material is
developed specifically for this course and not a part of a larger Software
Carpentry programme. This course focuses on a shallow but wide building of
foundational skills approach, introducing many base concepts but not going
deep into any one concept, with a through line of Open Science and Ethical
Data Usage.

##  6 Conclusions

The HEP community by and large acknowledges and recognizes the great
importance of training in the field of scientific computing. This activity
should encompass several types of _students_ , from undergraduates, to young
researchers, up to senior physicists, all of them in need of an appropriately
designed training path in order to be proficent in their scientific
endeavours.

We have identified a certain number of problems that need to be overcome in
setting up an appropriate training programme:

  * •

Costs and relative funding

  * •

Incentives

  * •

Career paths

  * •

Overall organization across experiments, countries and corresponding Funding
Agencies

For each of these points we have provided an overview of the current situation
and made proposals to improve the situation in HEP. The ideas presented need
to be developed further and concrete actions in the community need to be
implemented, which we will undertake in the HSF Training Working
GroupHSFTrainingWG (29).

##  Appendix A Training Topics of Interest

  * •

Basic and Advanced Programming Concepts

    * –

Object oriented paradigm

    * –

Compiled langauges (C++)

    * –

Scripting languages (Python, Javascript,…)

  * •

Algorithms

    * –

Boost library

    * –

STL algorithms for containers

    * –

R and/or ROOT

  * •

Frameworks (development or application level)

    * –

Qt

    * –

ROOT

    * –

experiment specific frameworks (possibly if of potential interest outside the
originating experiment)

  * •

Code design (design patterns)

  * •

Development tools

    * –

IDEs (Integrated Development Environment)

    * –

Debuggers

    * –

Profilers

  * •

Evaluation metrics

  * •

“Trust” metrics such as data driven tests

  * •

Specific software implementation training

  * •

Good practices

  * •

Code style and clarity

  * •

Scripting and data cleaning

  * •

Reporting results reproducibly

  * •

Writing Documentation

##  Appendix B Research Data Science Curriculum

This list is taken from the curriculum of the CODATA-RDA Research Data Science
Summer School in progress in Trieste, Italy during July 2017.
(http://indico.ictp.it/event/7974/):

  * •

Introduction

  * •

UNIX Shell programming (Software Carpentry Module)

  * •

GitHub (Software Carpentry Module)

  * •

R (Software Carpentry Module)

  * •

BYOD (Bring Your Own Data) best practices

  * •

Data Management with SQL (Software Carpentry Module)

  * •

RDM Storage Management

  * •

Visualisation

  * •

Machine Learning Overview - Recommendation

  * •

Recommender Systems

  * •

Artificial Neural Networks and other Machine Learning Systems

  * •

Research Computational Infrastructure

## References

  * (1) “Training-based Workforce Development for Advanced Cyberinfrastructure (CyberTraining) Webinar”  URL: https://www.nsf.gov/events/event_summ.jsp?cntn_id=190179&org=NSF
  * (2) “Cyberinfrastructure Training, Education, Advancement, and Mentoring for Our 21st Century Workforce (CI-TEAM)”  URL: https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=12782
  * (3) “Active learning approach”  URL: http://www.crlt.umich.edu/tstrategies/tsal
  * (4) “Peer learning approach”  URL: https://en.wikipedia.org/wiki/Peer_learning
  * (5) “Pomodoro Technique”  URL: https://en.wikipedia.org/wiki/Pomodoro_Technique
  * (6) “Docker Containers”  URL: https://www.docker.com/
  * (7) “Jupyter Hub”  URL: https://jupyterhub.readthedocs.io/en/stable/
  * (8) “Udacity”  URL: https://www.udacity.com/course/intro-to-machine-learning--ud120
  * (9) “Coursera”  URL: https://www.coursera.org/learn/machine-learning
  * (10) “WikiToLearn: a web-based collaborative tool to share knowledge”  URL: https://it.wikitolearn.org/
  * (11) “The GridKa School of Computing”  URL: http://gridka-school.scc.kit.edu/2017/
  * (12) “Stackoverflow”  URL: https://stackoverflow.com/
  * (13) “WikiMedia”  URL: https://www.wikimedia.org/
  * (14) “Gridka learing material published on WikiToLearn”  URL: https://en.wikitolearn.org/GridKa2017
  * (15) “The Code Swim Coaches”  URL: https://gitlab.in2p3.fr/MaitresNageurs
  * (16) “The LHC Physics Center (LPC) Hands-on Advanced Tutorial Sessions”  URL: http://lpc.fnal.gov/programs/schools-workshops/
  * (17) “The CMS Data Analysis Schools (CMSDAS)”  URL: http://lpc.fnal.gov/programs/schools-workshops/cmsdas.shtml
  * (18) “The CERN School of Computing”  URL: https://csc.web.cern.ch/
  * (19) “The CERN OpenLab Software workshops”  URL: http://openlab.cern/
  * (20) “The Bertinoro School of Computing”  URL: https://web.infn.it/esc17/index.php?lang=en
  * (21) “The GridKa School of Computing”  URL: http://codas-hep.org
  * (22) “NIPS”  URL: https://nips.cc
  * (23) “PyData”  URL: https://pydata.org
  * (24) “Training program”  URL: https://www.edx.org/credit
  * (25) “The DIANA/HEP project”  URL: http://diana-hep.org/
  * (26) “Advanced Multi-Variate Analysis for New Physics Searches at the LHC”  URL: https://amva4newphysics.wordpress.com/
  * (27) “Software Carpentry”  URL: https://software-carpentry.org
  * (28) “Data Carpentry”  URL: https://www.datacarpentry.org
  * (29) “HEP Software Foundation Working Group”  URL: https://hepsoftwarefoundation.org/activities/training.html


# Looking for Complexity at Phase Boundaries in Continuous Cellular Automata

Vassilis Papadopoulos  Ecole Polytechnique Fédérale de Lausanne
(EPFL)Switzerland 0000-0003-3270-5219<EMAIL_ADDRESS>,  Guilhem
Doat  Ecole Polytechnique Fédérale de Lausanne (EPFL)Switzerland
<EMAIL_ADDRESS>,  Arthur Renard  Ecole Polytechnique Fédérale de Lausanne
(EPFL)Switzerland<EMAIL_ADDRESS>and  Clément Hongler  Ecole
Polytechnique Fédérale de Lausanne (EPFL)Switzerland<EMAIL_ADDRESS>
###### Abstract.

One key challenge in Artificial Life is designing systems that display an
emergence of complex behaviors. Many such systems depend on a high-dimensional
parameter space, only a small subset of which displays interesting dynamics.
Focusing on the case of continuous systems, we introduce the ’Phase Transition
Finder’(PTF) algorithm, which can be used to efficiently generate parameters
lying at the border between two phases. We argue that such points are more
likely to display complex behaviors, and confirm this by applying PTF to Lenia
showing it can increase the frequency of interesting behaviors more than two-
fold, while remaining efficient enough for large-scale searches.

Lenia, Artificial, Life, Alife, Complex Systems, Phases

Figure 1.  Cell-like creatures in Lenia, displaying behaviour reminiscing of
mitosis

Lenia

##  1\. Introduction

Among the main goals of the field of Artificial Life (Alife) lies the
discovery, or design, of systems that display ’interesting’ life-like
behaviors, such as the spontaneous emergence of creatures that interact in
complex manners (Gardner, 1970). To this end, a plethora of Alife systems were
carefully designed to exhibit some kind of emergent property; be it Turing
completeness (Von Neumann and Burks, 1966; Wolfram, 1984), self-organization
(Mordvintsev et al., 2020), self-reproduction (Langton, 1984) or simply the
appearance of solitons (Chan, 2020). Despite those impressive results,
manually designing such systems is very difficult, as it is often hard to
predict what parameters will result in interesting emerging behaviors.

To address this problem, some more automatized approaches are being developed,
leveraging the increase in computing power, as well as the recent advances in
machine learning. For instance, genetic algorithms are effective to
efficiently traverse the parameter space of an Alife system, when we are
looking for a specific behaviour(Hamon et al., 2022). Such methods can also be
combined with ’novelty maximization’ techniques(Yang, 2023), which presumably
allow exploring the parameter space ’efficiently’, visiting only novel
behaviors. One can also employ gradient-descent based methods to steer the
dynamics towards a desired goal (Mordvintsev et al., 2020), or combine it with
a more abstract target to again obtain an efficient ’parameter-exploring’
algorithm (Etcheverry et al., 2020).

While such methods produce ’interesting’ systems much more reliably than
random sampling, they remain relatively slow; usually, the generation of a new
set of parameters requires a new training run, which is slow (especially for
genetic algorithms), and often expensive to scale. In this paper, we explore a
complementary method to generate interesting systems at a much higher rate
than random sampling, while remaining scalable and lightweight as it does not
rely on machine learning methods.

The main idea we seek to exploit is the observation that _phase transition
regions_ often display complex dynamics. One potential reason for this
peculiar observation comes from statistical physics, from which we learn that
(infinite) systems at a phase transition point display scale
invariance(Landau, 1937). In particular, this implies the existence of
structures at every scale, which is certainly non-trivial emergent behaviour
if the dynamics are local. Ideas from self-organized criticality (Bak et al.,
1989) also seem to suggest that critical points are an important driver of
complexity. In this paper, we design a ’proof-of-concept’ algorithm for homing
in these regions in parameter space and confirm empirically that the
proportion of ’interesting’ behaviors is much higher than random sampling.

The structure of this paper is as follows. In section 2, we describe the
’Phase Transition Finder’ (PTF) algorithm in more detail and develop a
concrete implementation for parameter exploration in Lenia. In section 3, we
present the results of the application of PTF on Lenia, and assess its
efficacy and scalability. We conclude by discussing potential applications and
future research directions.

##  2\. Idea and algorithm

It is a well-known empirical fact that the ’interesting’ dynamics in Alife
systems are much rarer than their chaotic or periodic counterparts. For
instance, in the class of outer-holistic 2D cellular automata estimations put
the number of interesting dynamics at $\sim 1\%$(Hudcova and Mikolov, 2020).
In continuous systems such as Lenia(Chan, 2019), mapping of a slice of
parameter space shows that creatures appear only in a very thin part of the
full region. Moreover, whenever a set of parameters that displays emergence of
creatures is discovered, they are usually quite fragile; a slight change in
parameters can make the creature suddenly die, or expand uncontrollably. Such
fine-tuning is again reminiscent of a critical point between two phases, where
very slight changes can topple the system in one phase or the other.

###  2.1. Phase Transition Finder Algorithm

These observations lead us to conjecture that at the very least, points lying
at the edge between two phases are worth exploring for interesting dynamics.
We seek to design an effective method of homing into these regions which we
will call ’Phase Transition Finder’ (PTF). The first step in this direction is
of course determining two (or more) mutually exclusive _phases_. We choose a
definition of ’phase’ that is intentionally very loose such that these ideas
are applicable to a wide range of models. We define it simply as a
quantitative description of some aspects of the dynamics of the model. To give
a few examples, we could say we are in a ’dead’ phase if all activations of a
cellular automaton go to $0$ after a certain time. Conversely, we could define
an ’exploding’ phase, if more than $80\%$ of cells are alive after a certain
time. For multi-channel models, we could define different phases according to
which channel is dominant, according to the growth rate in the initial stages,
or the magnitude of the fluctuations in the final stages… The possibilities
are endless.

The second step is to sample points from the transition regions of the phases
we have defined. This is relatively straightforward; we begin by randomly
sampling points from a pre-determined region of parameter space (the prior
region), until we find two points that lie in different phases. Next, we
connect them by an arbitrary path (in parameter space), and by dichotomy, we
efficiently find the point lying at the interface between the two phases (see
Fig.2)111Note that here we could also use gradient descent. However, that
might not always be applicable as the definition of the phases (which is
binary in nature) does not always provide with a differentiable proxy..
Crucially, this step relies on the _continuity_ of the parameter
space222Still, it would be interesting to apply it on discrete parameter
landscapes, as it may provide interesting results, although the ’transition’
point is not guaranteed to exist.. For this method to work, we need to choose
a set of mutually exclusive phases that cover the full parameter space.
Additionally, if the algorithm is to be efficient, determining to what phase a
point belongs should be fast. For this, it is okay to use heuristic methods,
as having perfect accuracy is not necessary.

Note that the choice of the prior region is very important; ideally one should
pick a prior region that displays an equal amount of each of the chosen
phases. This is a weakness of this method, as the choice of the prior must be
done manually. This can be mitigated whenever the chosen phases are
sufficiently simple such that one can roughly guess their locations and
extent. In the discussion at the end we mention some directions that could be
useful in solving this weakness.

Figure 2. 2-dimensional parameter space for an Alife model. Two points in
different phases are located, and then by dichotomy, we rapidly approach the
phase transition region.

Figure showing a path in 2-dimensional alife model parameters space.

###  2.2. An illustrative example: multi-channel Lenia

To illustrate PTF and verify its usefulness, we design a proof-of-concept
experiment by focusing on the class of continuous Alife systems known as
(multi-channel) Lenia. Lenia was developed by Bert Chan (Chan, 2020), first as
a natural continuous extension of the Game of Life. This model displays a wide
variety of emerging creatures and gliders, and was later extended from a
single channel to multiple ones, increasing further the diversity of its
’fauna’. We remind here how multi-channel Lenia works, as it will be the focus
of our experiments.

The state of the world at time $t$ is fully determined by a 3-channel field
where each channel is represented by $A^{t}_{i}(\vec{x})$ taking values in
$[0,1]$, where $i$ indexes the three (R,G,B) channels, and $\vec{x}=(x,y)$ is
the coordinates of the 2D world. The state at $t+dt$ is computed according to
the following equation :

(1)
$A^{t+dt}_{i}(\vec{x})=\bigg{[}A^{t}_{i}(\vec{x})+dt\sum_{j}B_{ji}G_{ji}(K_{ji}\ast
A^{t}_{j})\bigg{]}^{1}_{0}$

Where $K_{ji}(\vec{x})\in\mathbb{R}^{2}\rightarrow\mathbb{R}$ are kernels that
compute the contribution of channel $j$ for channel $i$ through a convolution.
They can be arbitrary radially symmetric functions, but to reduce the
parameter space dimensionality, they are entirely determined by the position
and magnitudes of three bump functions
$\exp{-\frac{(x-\mu)^{2}}{2\sigma^{2}}}$, see Fig.3. Similarly,
$G_{ji}=2\exp{\frac{(x-\mu_{ji})^{2}}{2\sigma_{ji}}}-1\in[-1,1]$ are growth
functions, which are there to introduce non-linearities in the dynamics.
Again, they are fully determined by specifying $\mu_{ij}$ and $\sigma_{ij}$.
The $B_{ij}\in[0,1]\mbox{s.t.}\sum_{j}B_{ij}=1$ weight the contributions of
the different channels. Finally, $\left[\right]^{1}_{0}$ denotes clipping to
$[0,1]$ such that the values of $A$ remain in the same range (which also
introduces an additional non-linearity).

Figure 3. Top: Example kernels $K_{ji}$. Each kernel is convoluted with the
corresponding labeled channel on top. The colors of the circles indicates to
which channel the convolution contributes (i.e., the ’$i$’ in $K_{ji}$).
Bottom : Cross-section of one kernel $K_{ji}$, displaying the 8 free
parameters ($\sum_{j}\beta_{j}=1$).

Figure depicting a Lenia Kernel and how it is constructed.

In total (excluding the choice of $dt$, which we will keep fixed), this
amounts to $106$ real numbers in $[0,1]$ as free parameters. Navigating such a
high-dimensional manifold manually in search of interesting dynamics is not an
easy task. Luckily, this particular model suggests natural phases for the
phase-transition search, those being ’dead’ and ’living’ phases. This is
because moving creatures in Lenia exist necessary in the balance between death
and expansion, as they destroy ’matter’ behind them and recreate it in front,
simulating locomotion.

More precisely, we formally define the ’dead’ phase as
$\lim_{t\rightarrow\infty}\bar{A}^{t}_{i}<\alpha$, with $\alpha$ a
hyperparameter and $\bar{A}^{t}_{i}$ denotes the average over the spatial
positions. The ’live’ phase is defined as the complement of the ’dead’ phase.
In practice, to determine efficiently which phase we are in, we approximate
the definition by cutting at finite time $T$, since it is otherwise
technically undecidable.

When we attempt to evaluate by inspection dynamics that we find using this
technique, we will be looking for the following qualities, which aim to
capture why some dynamics look interesting:

  * •

Dynamics that do not settle quickly

  * •

Several coexisting and interacting colors

  * •

Presence of solitons, gliders

  * •

Self-reproduction and other interesting interactions between solitons

##  3\. Results on Lenia

In this section, we present some results stemming from the application of the
algorithm to multi-channel Lenia. For these experiments, we decided to explore
only the space of parameters affecting the _physics_ of the system, namely the
parameters appearing in 1. One could extend the search to consider the initial
condition $A^{0}_{i}(\vec{x})$, which can also be impactful, but we wanted to
focus on finding interesting ’physics’, rather than interesting initial
conditions. Thus, we will be initializing all runs with Perlin noise in all
channels. In this way, we are implicitly selecting for interesting dynamics
that develop for this class of random initial conditions.

We implement multi-channel Lenia using Pytorch (code provided in supplemental
material). For the discretization, we set the scale by choosing the kernel to
be of size $1\times 1$ (from here on, unless specified, all units are in
’kernel size’). We fix $dt=0.1$ and discretize the kernel to a $31\times 31$
pixels square, equivalent to $dx\approx 0.03$. We perform the visual
evaluation on $15$ second clips ($1800$ timesteps at $120$ fps), on a $\approx
16\times 16$ toroidal world.

To speed up the search for the alive and dead phases, we restrict to a
$5\times 5$ world, evolved for 800 timesteps. This can be justified because
the scale of emergent structures is rarely much larger than a few kernel
sizes, so restricting to a smaller world does not affect the outcome for which
phase we are in. In any case, a few misclassifications are unproblematic,
since we do not aim for $100\%$ success rate in the parameters we discover.

We begin with a relatively ’neutral’ prior. We set all parameters to be
generated uniformly in $[0,1]$, except $\mu$ and $\sigma$ for the growth
functions, which are uniform in $[0,0.7]$, $[0,0.2]$ respectively (specifics
in the provided code), otherwise ’dead phases’ are extremely rare. We set the
threshold to $\alpha=0.05$ and generate 200 phase transition points. We then
qualitatively assess the amount of ’interesting’ dynamics for these parameters
against 200 random parameters.

Next, we repeat the experiment with a slightly ’smarter’ prior. One can see
that if any of the $G_{ij}(0)>0$, then the ’ground state’ is unstable; there
can be no ’dead’ phase, as a dead world will have a non-negative growth! The
critical value in this case is given by
$\sigma_{ij}=\frac{\mu_{ij}}{2\ln(2)}$, we thus constrain the prior for
$\sigma_{ij}\in[0,0.9\frac{\mu_{ij}}{2\ln(2)}]$. This choice ensures both the
a phase transition is present in the prior, and that the proportion of
dead/alive phases will be roughly balanced, speeding the initial phase search.
We repeat the experiment as before displaying results in Table 1.

Prior | % Dead | % Int. rand | % Int. phase | Time (m:s)
---|---|---|---|---
neutral | 32% | 5.5% | 13.0% | 4:40
smart | 65% | 14.0% | 30.5% | 4:20
Table 1.  Table with results for the experiments. ”$\%$ Dead” denotes the
proportion of ”dead” phase when sampling from the prior. ”$\%$ Int. rand”
(resp. phase) denotes the proportion of dynamics deemed ’interesting’ by
visual inspection, when sampling from the prior (resp. from phase transition
points). Time columne is the time taken to generate 100 phase transition
points on a RTX 3080 GPU.

Overall, we can see that the points in the phase transition region display a
significantly higher percentage of ’interesting’ configurations (see Figure 4
for a glimpse). Additionally to these controlled tests, we extensively tested
the algorithm for different priors. Overall, qualitatively we observe that for
phase transition points, the presence of solitons, static or even moving is
not uncommon, this is in stark contrast to a random parameter search, where
finding gliders is extremely uncommon. Some videos of remarkable dynamics that
were found using this method can be found in the supplementary material.

Figure 4. Final snapshot for a simulation of 1000 steps in Lenia. Left of the
red line: parameters obtained with PTF. Right : Parameters obtained with
prior.

Figure showing a path in 2-dimensional alife model parameters space.

One thing we noticed is that the phase transitions are often not ’sharp’, but
rather continuous. This is to be expected as nothing guarantees our choice of
phase to match in any way the ’rigorous’ definition in statistical physics or
thermodynamics, especially for arbitrary $\alpha>0$. When $\alpha$ is close to
zero, we sometimes see behavior reminiscent of ’critical slowing down’: close
to the transition point, the dynamics slow down and die out (or conversely,
expand) slowly, such that at the cutoff time $T$, we are indeed at
$\lim_{t\rightarrow\infty}\bar{A}^{t}_{i}\approx\alpha$. This is not what we
are looking for as it generally does not give rise to interesting dynamics,
but there is no way to discard those points a priori.

In contrast, when the parameters display solitons, the phase transition is
usually sharp : small fluctuations in parameter space usually kill off the
solitons, or make them unstable to nucleation. One last remark is that self-
reproduction behaviour (usually through duplication of the solitons) is quite
a common occurence. It is unclear if this is a specificity of Lenia or of the
phase-transition regions, and it would be interesting to test this in other
continuous models.

##  4\. Discussion and research directions

In this short paper, we presented a novel method for the automatic search of
interesting sets of parameters in continuous Alife systems. We presented some
encouraging results by applying it to Lenia, validating the potential of this
method. In particular, the rate at which new points in parameter space can be
constructed is relatively high ($\sim$40 minutes for 1000 examples, although
this can highly vary depending on the hyperparameters we choose), which opens
many avenues to apply deep learning techniques. Indeed, the main problem with
pure ’random’ generation of parameters is that the ’signal’ in them is small;
generally, the amount of interesting dynamics is extremely low. In this case,
using an efficient method like the one presented in this paper can be useful
for increasing the ’signal’ in the dataset to a more tractable level for
machine learning applications.

The algorithm does however have some limitations, the first of which is that
the choice of the prior region still has an important impact on the results,
and it still requires some degree of manual tuning. This can potentially be
mitigated by developing heuristics for the choice of the prior region, such as
modifying it until the two phases cover approximately the same volume. Another
problem is that a large part of the generated parameters still display trivial
dynamics. Some (like dying dynamics) can be easily detected and discarded. For
other less obvious cases, one possible direction to pursue is to train a ML
model to discard uninteresting dynamics, by learning through human
feedback(Christiano et al., 2023). While it would be difficult to train such a
model on random data (as most datapoints would be trivial), the pre-filtration
through our method makes the dataset more diverse and thus more likely to
provide the model with useful examples.

Some other research directions involve certainly applying this method in a
wider context, both considering other phase types, and other continuous
systems. Systems such as Flow-Lenia (Plantec et al., 2023) where mass is
conserved Fare particularly interesting because they require the design of a
different type of phase (as ’dying’ configurations don’t exist). Another
worthwhile direction would certainly be to include the initial condition in
the varying parameter set, to see if phase transition region also increase the
amount of interesting behaviors in that context.

In conclusion, this method opens avenues for parameter exploration which are
mostly orthogonal to other methods such as genetic algorithms and
reinforcement learning, and in combination they open exciting possibilities
for automatic discovery of Artificial Life systems !

##  5\. Acknowledgements

The authors would like to thank João Penedones, Eugéne Bergeron and Sanyam
Jain for interesting discussions. Support from the Blavatnik Family
Foundation, the Latsis Foundation, the NCCR SwissMAP, and from an EPFL FSB
Seed Funding Grant are gratefully acknowledged.

## References

  * (1)
  * Bak et al. (1989) Per Bak, Kan Chen, and Michael Creutz. 1989.  Self-organized criticality in the ’Game of Life.  _Nature_ 342, 6251 (Dec. 1989), 780–782.  https://doi.org/10.1038/342780a0 Number: 6251 Publisher: Nature Publishing Group.
  * Chan (2019) Bert Wang-Chak Chan. 2019.  Lenia - Biology of Artificial Life.  _Complex Systems_ 28, 3 (Oct. 2019), 251–286.  https://doi.org/10.25088/ComplexSystems.28.3.251 arXiv:1812.05433 [nlin].
  * Chan (2020) Bert Wang-Chak Chan. 2020.  Lenia and Expanded Universe. In _The 2020 Conference on Artificial Life_. 221–229.  https://doi.org/10.1162/isal_a_00297 arXiv:2005.03742 [nlin].
  * Christiano et al. (2023) Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2023.  Deep reinforcement learning from human preferences.  https://doi.org/10.48550/arXiv.1706.03741 arXiv:1706.03741 [cs, stat].
  * Etcheverry et al. (2020) Mayalen Etcheverry, Clément Moulin-Frier, and Pierre-Yves Oudeyer. 2020.  Hierarchically Organized Latent Modules for Exploratory Search in Morphogenetic Systems. In _Advances in Neural Information Processing Systems_ , Vol. 33. Curran Associates, Inc., 4846–4859.  https://proceedings.neurips.cc/paper/2020/hash/33a5435d4f945aa6154b31a73bab3b73-Abstract.html
  * Gardner (1970) Martin Gardner. 1970.  Mathematical Games.  https://www.scientificamerican.com/article/mathematical-games-1970-10/
  * Hamon et al. (2022) Gautier Hamon, Mayalen Etcheverry, Bert Wang-Chak Chan, Clément Moulin-Frier, and Pierre-Yves Oudeyer. 2022.  Learning Sensorimotor Agency in Cellular Automata.  (2022).  https://inria.hal.science/hal-03519319
  * Hudcova and Mikolov (2020) Barbora Hudcova and Tomas Mikolov. 2020.  Classification of Complex Systems Based on Transients. In _The 2020 Conference on Artificial Life_. 367–375.  https://doi.org/10.1162/isal_a_00260 arXiv:2008.13503 [nlin].
  * Landau (1937) L. D. Landau. 1937.  On the theory of phase transitions.  _Zh. Eksp. Teor. Fiz._ 7 (1937), 19–32.  https://doi.org/10.1016/B978-0-08-010586-4.50034-1
  * Langton (1984) Christopher G. Langton. 1984.  Self-reproduction in cellular automata.  _Physica D: Nonlinear Phenomena_ 10, 1 (Jan. 1984), 135–144.  https://doi.org/10.1016/0167-2789(84)90256-2
  * Mordvintsev et al. (2020) Alexander Mordvintsev, Ettore Randazzo, Eyvind Niklasson, and Michael Levin. 2020.  Growing Neural Cellular Automata.  _Distill_ 5, 2 (Feb. 2020), e23.  https://doi.org/10.23915/distill.00023
  * Plantec et al. (2023) Erwan Plantec, Gautier Hamon, Mayalen Etcheverry, Pierre-Yves Oudeyer, Clément Moulin-Frier, and Bert Wang-Chak Chan. 2023.  Flow-Lenia.  https://doi.org/10.48550/arXiv.2212.07906 arXiv:2212.07906 [nlin].
  * Von Neumann and Burks (1966) J. Von Neumann and A. W. Burks. 1966.  _Theory of self-reproducing automata_.  Urbana, University of Illinois Press.
  * Wolfram (1984) Stephen Wolfram. 1984.  Universality and complexity in cellular automata.  _Physica D: Nonlinear Phenomena_ 10, 1-2 (Jan. 1984), 1–35.  https://doi.org/10.1016/0167-2789(84)90245-8
  * Yang (2023) Bo Yang. 2023.  Self-Replicating Hierarchical Structures Emerge in a Binary Cellular Automaton.  http://arxiv.org/abs/2305.19504 arXiv:2305.19504 [nlin].

##  Appendix A Code and simulations

The full code can be found at https://github.com/frotaur/TempGECCO. It
includes the code to run the PTF algorithm on Lenia, as well as some
particularly interesting parameter sets found with the method. The dynamics
for these parameters can be simulated and viewed in real time with the
codebase.

Alternatively, https://vassi.life/alife/teaching/teaser showcases a subset of
the dynamics under the section ’Lenia’.