# Exploring Randomly Wired Neural Networks for Climate Model Emulation

###### Abstract

Exploring the climate impacts of various anthropogenic emissions scenarios is
key to making informed decisions for climate change mitigation and adaptation.
State-of-the-art Earth system models can provide detailed insight into these
impacts, but have a large associated computational cost on a per-scenario
basis. This large computational burden has driven recent interest in
developing cheap machine learning models for the task of climate model
emulation. In this manuscript, we explore the efficacy of randomly wired
neural networks for this task. We describe how they can be constructed and
compare them to their standard feedforward counterparts using the ClimateBench
dataset. Specifically, we replace the serially connected dense layers in
multilayer perceptrons, convolutional neural networks, and convolutional long
short-term memory networks with randomly wired dense layers and assess the
impact on model performance for models with 1 million and 10 million
parameters. We find that models with less complex architectures see the
greatest performance improvement with the addition of random wiring (up to
30.4% for multilayer perceptrons). Furthermore, out of 24 different model
architecture, parameter count, and prediction task combinations, only one saw
a statistically significant performance deficit in randomly wired networks
compared to their standard counterparts, with 14 cases showing statistically
significant improvement. We also find no significant difference in prediction
speed between networks with standard feedforward dense layers and those with
randomly wired layers. These findings indicate that randomly wired neural
networks may be suitable direct replacements for traditional dense layers in
many standard models.

Modeling various greenhouse gas and aerosol emissions scenarios is important
for both understanding climate change and making informed political and
economic decisions. However, accomplishing this with large Earth system models
is a complex and computationally expensive task. As such, data-driven machine
learning models have risen in prevalence as cheap emulators of Earth system
models. In this work, we explore a special type of machine learning model
called randomly wired neural networks and find that they perform competitively
for the task of climate model emulation. This indicates that future machine
learning models for emulation may significantly benefit from using randomly
wired neural networks as opposed to their more standard counterparts.

##  1 Introduction

Characterizing the response of the Earth system to future emissions scenarios
is key to informing climate change mitigation and adaptation strategies.
Modern Earth system models (ESMs) are incredibly useful tools for this task,
providing detailed climate projections far into the future for scenario-based
analysis. However, as the process complexity and spatial resolution of such
ESMs increases, so does their computational cost (Collins et al., 2012).
Consequently, it becomes impractical to explore a wide range of possible
emissions scenarios with modern ESMs, limiting their applicability to a
restricted set of future scenarios (O’Neill et al., 2016).

To address this large computational cost, the climate modeling research
community frequently makes use of emulators. Emulators are a set of
statistical tools that aim to approximate the complex physical relationships
in a full ESM at a fraction of the computational cost (Meinshausen et al.,
2011; Tebaldi et al., 2020). Recent developments have demonstrated the
considerable promise of machine learning (ML) techniques in these emulation
tasks for both individual model components (Seifert and Rasp, 2020; Silva et
al., 2021; Mooers et al., 2021; Chantry et al., 2021; Lee et al., 2019) and
entire ESM predictions (Rasp et al., 2020; Watson-Parris et al., 2022;
Mansfield et al., 2020; Beusch et al., 2020). Such ML models often enable
accurate predictions at greatly reduced computational cost relative to full
ESMs.

The model classes investigated in state-of-the-art ML research in the climate
sciences range widely, and include linear models (Rasp, 2020; Silva et al.,
2020; Mansfield et al., 2020), tree based methods (Yuval and O’Gorman, 2020;
Yuval et al., 2021; Silva et al., 2021), and various implementations of neural
networks (Price and Rasp, 2022; Bretherton et al., 2022; Krasnopolsky et al.,
2005; Rasp and Thuerey, 2021). This is broadly indicative of a very large
potential design space for ML model architectures. Here, we investigate
“randomly wired neural networks” in an effort to further explore this
architecture design space for climate mode emulation. Randomly wired neural
networks are a special class of neural network where components are connected
in a random, rather than structured, manner. This is in direct contrast to
widely-used feed forward multi-layer perceptron (FFMLP) style architectures,
where components are connected in series. At its core, random wiring is a form
of neural architecture search (NAS) (Xie et al., 2019; Kasim et al., 2021)
that searches a more complete space of connectivity patterns than other common
NAS strategies (Elsken et al., 2019) to identify high-performing model
architectures. This class of neural network introduces novel connectivity
patterns between layers rather than novel types of layers themselves, and they
have demonstrated skill in a variety of domains, including handwriting
recognition (Gelenbe and Yin, 2016), internet network attack detection (Brun
et al., 2018), and emulation of aerosol optics (Geiss et al., 2023).

In this work, we evaluate the suitability of randomly wired neural networks
for climate emulation using the ClimateBench benchmarking dataset. We
specifically investigate the use of random wiring to predict future
temperature and precipitation statistics. To that end, we compare the
performance of randomly wired networks against their serially connected
counterparts within three types of neural network models covering a wide range
of complexities: multilayer perceptrons (MLPs), convolutional neural networks
(CNNs), and convolutional long short-term memory networks (CNN-LSTMs). We find
that random network wiring shows competitive results across model
architectures and prediction tasks with the greatest improvements (up to
30.4%) in MLPs. These performance improvements for randomly wired models
decreases with increasing model architecture complexity, but out of 24 model
architecture, parameter count, and prediction task combinations, only one saw
a statistically significant performance deficit for randomly wired models. On
the other hand, 14 saw statistically significant performance improvement.
Furthermore, we find that randomly wired neural networks take no longer to
make predictions than their standard serially connected counterparts. Our work
indicates that in many cases, random wiring can serve as direct replacements
for traditional feedforward neural networks to improve predictive skill.

The remainder of the paper is structured as follows. In Section 2 we provide
the details of how we construct our randomly wired neural networks and
highlight their differences with more standard networks. Section 3 describes
the ClimateBench dataset and how we use it to train, validate, test, and
evaluate our models. Our experimental setup to compare randomly wired neural
networks with their conventional counterparts is discussed in Section 4.
Section 5 provides the results of these experiments, with further discussion
and analysis in Section 6. Lastly, Section 7 concludes the manuscript with
discussion of future work.

##  2 Randomly Wired Neural Networks

A typical multilayer perceptron (MLP) neural network is structured such that
information (a tensor) only flows in a single direction from one dense layer
to the next. This is illustrated in Figure 1 which shows a graph
representation of an MLP with six hidden dense layers. In this graph
representation, each layer is represented as a node in a directed graph where
edges indicate the flow of tensors. We use this graph representation and
terminology in part for consistency with previous work (Xie et al., 2019). In
our random networks, as well as those of Xie et al. (2019) and Geiss et al.
(2023), data still flows through the network in a feedforward manner. However,
unlike dense layers in MLPs, dense layers in our random networks may receive
inputs from any number of preceding layers and pass their outputs to any
number of subsequent layers. That is, there may be “skip connections” between
dense layers. An example of this class of random neural network, which we will
henceforth refer to as “RandDense” networks, is illustrated in Figure 1. With
this in mind, we will detail our random network generation process and further
highlight differences between our networks and typical MLPs.

Figure 1: Graph representations of a six layer MLP (left) and six layer
RandDense network (right) shown side-by-side with their respective node
operations in the insets. White circles represent hidden dense layers and
their activation functions. Brown circles represent the output layer discussed
in Section 2. Lastly, the upper rectangular block represents neural network
layers preceding the dense layers. In this case, it is a simple input layer
which performs no operation, but other choices such as a convolutional block
are possible (see Section 4). For the RandDense network, aggregation from
three previous input nodes is done via weighted sum with weights $w_{0}$,
$w_{1}$, and $w_{2}$. The summation is followed by ReLU activation and the
dense layer. Lastly, two identical copies of the output are sent to two
separate nodes downstream.

Network connectivity. Since the layers within our RandDense networks are
randomly wired, the first step in generating such a network is determining
which layers are connected to which, or the network connectivity. Following
Geiss et al. (2023), given a fixed number of layers $n$, we randomly select
the number of neurons per dense layer and generate an adjacency matrix
representing the connections between layers in our RandDense network. Since
our RandDense networks are still feedforward in nature, several constraints
may be placed on the adjacency matrix. If we let each row represent a layer,
and column values represent connections (inbound edges) from previous layers,
then the adjacency matrix must be lower triangular. Thus, there are $n(n+1)/2$
possible connections for an $n$ layer random network. For each randomly wired
model, we generate an adjacency matrix from these possibilities by randomly
assigning 1’s and 0’s to each entry of an $n\times n$ lower triangular matrix.
We do not do any further filtering or selection on the connectivity patterns
within the randomly wired models. The corresponding adjacency matrices for the
networks in Figure 1 are shown in Figure 2. Some of these connectivity
patterns may have layers without an inbound or outbound tensor. Thus, to
ensure the network is valid, we follow Geiss et al. (2023) and iterate through
each row and column, randomly activating an edge in each row/column if it has
no active edges.

$\displaystyle\begin{bmatrix}1&0&0&0&0&0\\\ 0&1&0&0&0&0\\\ 0&0&1&0&0&0\\\
0&0&0&1&0&0\\\ 0&0&0&0&1&0\\\ 0&0&0&0&0&1\end{bmatrix}$

$\displaystyle\begin{bmatrix}1&0&0&0&0&0\\\ 0&1&0&0&0&0\\\ 1&0&0&0&0&0\\\
0&1&1&1&0&0\\\ 1&0&0&1&1&0\\\ 0&0&1&1&1&1\end{bmatrix}$

Figure 2: Graph representations of the six layer MLP (left) and six layer
RandDense network (right) from Figure 1 shown side-by-side with their
respective adjacency matrices.

Node operations. Nodes in a RandDense network may have multiple inbound or
outbound edges, unlike the nodes of an MLP which only have one of each (see
Figure 1). To treat this difference, we must define a new operation that
occurs at every node.

The node operation of our RandDense networks begins with an aggregation of the
incoming tensors. If a node has more than one inbound edge, we follow Xie et
al. (2019) and aggregate the incoming tensors via a weighted sum with
learnable, positive weights. The weights are kept positive by applying the
sigmoid function. Exclusively summing inbound tensors instead of concatenating
them (Geiss et al., 2023) requires a consistent number of neurons per layer
throughout the network, but avoids extremely large input tensors that might
occur using concatenation. In the style of Xie et al. (2019), our random
networks have the ReLU activation function before the dense layer so that the
outbound tensor can contain both positive and negative values. This avoids
extremely large weighted sums when the number of inputs to a given layer is
high. Figure 1 illustrates the differences between a node, which contains the
dense layer and its activation, in a standard MLP and our random networks.
Notice that a node in the graph representation of an MLP in Figure 1 contains
a dense layer followed by ReLU activation, while a node of the RandDense
network in Figure 1 contains an weighted summation followed by ReLU activation
then the dense layer. Lastly, if a node is connected to multiple downstream
nodes, it will simply send out identical copies of its output along each
outbound edge.

Input and output nodes. In our implementation of RandDense networks, each
hidden layer has the same number of neurons. To accommodate smaller input
vectors, the node immediately following the input layer will contain a number
of neurons equal to the difference between the selected layer size and the
input size. For example, if the selected layer size is 100 and the input size
is 12, the first randomly wired node following the input layer will contain a
dense layer of size 88. Then, this dense layer is concatenated with the input
to create a layer of the correct size. This is done so that any nodes
downstream of the first node may still have direct access to the input (Geiss
et al., 2023).

For the output node which may have multiple inbound edges, we simply take the
average of all inbound tensors and send this value to a final dense layer of
the desired output size. This final output node, which is colored brown in
Figure 1, has linear activation instead of ReLU so that both positive and
negative values may be output. Our randomly wired neural networks are
implemented using Tensorflow 2 in Python (Abadi et al., 2015), with code
available on Github (see data availability statement).

##  3 Data

Standard benchmarks are a valuable tool for the intercomparison of ML methods.
The ClimateBench dataset, along with its associated evaluation criteria and
metrics seeks to provide a standardized framework for objectively evaluating
ML-driven climate model emulators (Watson-Parris et al., 2022). We use this
ClimateBench dataset to train, validate, and test our models in this work. The
dataset is constructed from several simulations performed by Norwegian Earth
System Model version 2 (NorESM2) (Seland et al., 2020) as part of the Scenario
Model Intercomparison Project (ScenarioMIP) (O’Neill et al., 2016), Coupled
Model Intercomparison Project Phase 6 (CMIP6) (Eyring et al., 2016), and
Detection and Attribution Model Intercomparison Project (DAMIP) (Gillett et
al., 2016). ClimateBench provides four main anthropogenic forcing agents as
predictors: carbon dioxide (CO2), sulfur dioxide (SO2), black carbon (BC), and
methane (CH4), with the goal of predicting the annual mean surface air
temperature (TAS), annual mean diurnal temperature range (max TAS $-$ min TAS)
(DTR), annual mean total precipitation (PR), and 90th percentile of daily
precipitation (PR90) from 2015-2100. SO2, BC, and the predictands are provided
as annual mean spatial distributions across 96 latitude $\times$ 144 longitude
global grid points, while the longer lived and well mixed CO2 and CH4 inputs
are provided as annual global total concentration and global average
emissions, respectively. For each experiment, the ClimateBench dataset also
includes the post-processed output of three NorESM2 ensemble members which
sample internal variability of the model using different initial model states.

Our models are trained to predict one of TAS, DTR, PR, and PR90 following the
ClimateBench benchmarking framework. Specifically, model inputs are global
total concentrations of CO2, global average CH4 emissions, as well as spatial
distributions ($96\times 144$) of annual average BC and SO2 for a range of
years from a given set of experiments. The models predict an annual average
output variable value for each of the $96\times 144$ grid points for each of
the years 1850-2100, though they are evaluated on a smaller window as
specified in the following subsections.

###  3.1 Training and Validation

Following the training, validation, and testing strategy in the original
ClimateBench manuscript, we select the historical, ssp126, ssp370, and ssp585
experiments from CMIP6 (Eyring et al., 2016) as well as the hist-GHG and hist-
aer experiments from DAMIP (Gillett et al., 2016) for training. Each
historical experiment spans the years 1850-2014, and each of the ssp
experiments contain data for the years 2015-2100 for a total of 753 years.
These experiments together cover a wide range of values for each of the input
anthropogenic forcers and predicted output variables, making it an ideal suite
to train our models on.

In any machine learning application, it is good practice to reserve a portion
of the training data for validation so that model performance may be monitored
on both sets to prevent overfitting. Since climate data is highly correlated
in time, it is recommended to select a continuous portion of the dataset as
validation, rather than a random subset. We choose to use the first two years
of every decade from the historical, hist-GHG, hist-aer, ssp126, ssp370, and
ssp585 training datasets as validation. Here, the validation set is used for
early-stopping, a form of regularization where training is halted once
performance on the validation set stops improving.

###  3.2 Testing and Evaluation

In addition to the validation set, a third test dataset is used as a final
evaluation to ensure that neural network hyperparameter tuning or model
selection has not overfit the validation data. As in the original ClimateBench
manuscript, we select the ssp245 experiment from ScenarioMIP for the years
2080-2100 as our test dataset. This experiment lies between the extremes of
the ssp126 and ssp585 experiments and represents a medium forcing and
mitigation scenario.

Following the original work by Watson-Parris et al. (2022), we choose to
evaluate our models using “total root-mean square error” ($NRMSE_{t}$), a
combination of the normalized, global mean RMSE ($NRMSE_{s}$) and the NRMSE in
the global mean ($NRMSE_{g}$). These are calculated as

$\displaystyle NRMSE_{s}$
$\displaystyle=\sqrt{\langle\left(\left|x_{i,j,t}\right|_{t}-\left|y_{i,j,n,t}\right|_{t,n}\right)^{2}\rangle}\Big{/}\left|\langle
y_{i,j}\rangle\right|_{t,n},$ $\displaystyle NRMSE_{g}$
$\displaystyle=\sqrt{\left|\left(\langle
x_{i,j,t}\rangle-\langle\left|y_{i,j,n,t}\right|_{n}\rangle\right)^{2}\right|_{t}}\Big{/}\left|\langle
y_{i,j}\rangle\right|_{t,n},$ $\displaystyle NRMSE_{t}$
$\displaystyle=NRMSE_{s}+\alpha\cdot NRMSE_{g},$

where the global mean denoted $\langle x_{i,j}\rangle$ is cosine weighted by
latitude to account for the decreasing grid-cell area towards the poles such
that $\langle
x_{i,j}\rangle=\frac{1}{N_{lat}N_{lon}}\sum_{i}^{N_{lat}}\sum_{j}^{N_{lon}}\cos(lat(i))x_{i,j}$,
$|x|_{t}$ is the average of $x$ over all $t$, $n$ is the NorESM2 ensemble
members, $t$ is time in years, and $\alpha$ is an empirically chosen factor to
give each NRMSE component equal weight (Watson-Parris et al., 2022).

##  4 Experimental Setup

In order to explore the effects of random wiring, we first define three
baseline model architectures which contain MLP dense layers. These models are
meant to represent realistic deep learning architectures a climate scientist
would use for emulation tasks, and include a standard MLP, convolutional
neural network, and a convolutional-LSTM network. For each of these model
architectures, we explore performance differences across the four output
variables (TAS, DTR, PR, and PR90) described in Section 3 at two different
parameter counts within the dense layers of each respective network: 1 million
and 10 million. With each parameter count, we also explore performance for
nine different numbers of hidden dense layers (2-10). Then, the standard MLP
dense layers are replaced with randomly wired ones for comparison. We expect
that the three baseline model architectures should cover a wide range of
performances, illustrating how random wiring impacts performance when placed
within both good and bad models. Example networks (with six hidden layers) for
each of the three baseline models along with example randomly wired
variations, are shown in Figure 3.

Figure 3: Each of the three baseline models with six MLP dense layers shown
next to an example RandDense variation.

###  4.1 MLP Network

A standard MLP model is the most common and basic type of neural network which
only contains dense layers. They have been extensively used in climate
modelling tasks such as subgrid process representation (Rasp et al., 2018),
rainfall downscaling (Tran Anh et al., 2019), longwave raditaion emulation
(Krasnopolsky et al., 2005), pan evaporation prediction (Ghorbani et al.,
2018), and stochastic synthesis in hydrology simulations (Rozos et al., 2021).

We begin with the first parameter limit of 1 million. We generate MLP models
for a given number of hidden layers by selecting a fixed layer size for all of
the hidden layers such that the network’s parameter count is 1 million $\pm
10\%$. We generate 50 such MLP models with 2 hidden dense layers, 50 MLPs with
3 hidden dense layers, and so on up to 10 hidden dense layers. This gives 450
MLP models, each with different parameter initializations. The process is
repeated for a parameter limit of 10 million, yielding a total of 900 MLPs.
For the RandDense comparison networks, we repeat a similar process. Using the
network generation method described in Section 2, we generate 50 RandDense
networks for 2-10 hidden layers at both the 1 million and 10 million parameter
count for a total of 900 RandDense networks. For small numbers of hidden
layers, many of these models will have identical layer connectivity patterns
since there are very few pattern choices. However, for larger numbers of
hidden layers, many of the generated models will have unique wirings.

As discussed in Section 3, the inputs to the models are global CO2 and CH4, as
well as $96\times 144$ grids of SO2 and BC. However, in MLPs each input is
handled by one neuron in the input layer. As such, if we directly fed the
anthropogenic forcer inputs to the model, the input layer would be over 26,000
neurons wide. This is a extremely high layer size, and so we follow Watson-
Parris et al. (2022) and perform dimensionality reduction on the SO2 and BC
inputs. Specifically, we use the first five empirical orthogonal functions
(EOFs) of each field. Thus, the total input size to the MLP is 12: the first
five EOFs for SO2 and BC, plus global CO2 and CH4. The output of the MLP is a
flattened map of the predicated variable, which is then reshaped to the
correct grid for final predictions.

Both the MLPs and RandDense networks are trained with a batch size of 25 and
use the Adam optimizer (Kingma and Ba, 2014) and mean squared error loss for
100 epochs. We also use early stopping during training with a patience of 10
epochs.

###  4.2 Convolutional Network

Convolutional neural networks (CNNs) have been widely adopted in object
tracking and image recognition tasks because they are able to model spacial
dependencies (Li et al., 2021; O’Shea and Nash, 2015). CNNs have also been
recently adopted for various weather and climate tasks such as precipitation
nowcasting (Trebing et al., 2021), bow echo detection (Mounier et al., 2022),
climate zone classification (Yoo et al., 2019) and ocean modelling (Nikolaev
et al., 2020).

Following a similar process as in the previous subsection, we generate 50 MLPs
and 50 RandDense networks with 2-10 hidden dense layers for both 1 and 10
million parameter counts. However, rather than being standalone networks,
these models are appended to a convolutional block. This block consists of a
single convolutional layer with 20 filters, a kernal size of 3, ReLU
activation, and $L_{2}$ regularization (Cortes et al., 2012). The
convolutional layer is followed by global average pooling.

Since CNNs are designed to handle images with multiple channels, we preserve
the original dimensionality of the input variables, unlike the transformation
we conducted for the standalone MLP and RandDense networks in the previous
subsection. Specifically, we are able to feed the $96\times 144$ maps of SO2
and BC to the convolutional network, as well as global CO2 and CH4. In order
to treat the four input variables as four channels of one $96\times 144$
“image” of the globe, the CO2 and CH4 data are transformed to $96\times 144$
grids where each grid cell has the same value.

The CNNs are followed by either conventional MLP or RandDense blocks, which we
will simply refer to as “CNN networks,” and “CNN RandDense networks.” The CNNs
use a batch size of 25 and are trained with the Adam optimizer using the mean
squared error loss function for 100 epochs. We again use early stopping during
training with a patience of 10 epochs.

###  4.3 Convolutional-LSTM Network

Long short-term memory (LSTM) networks are a type of recurrent neural network
(RNN) which model temporal dependencies. This time-aware property makes them
useful for a range of forecasting tasks such as precipitation downscaling
(Tran Anh et al., 2019), soil moisture estimation (Ahmed et al., 2021), and
flood forecasting (Liu et al., 2020). For the specific task of climate model
emulation, a combined CNN-LSTM model has been shown to outperform CNN and LSTM
models in isolation (Watson-Parris et al., 2022).

As in the previous two subsections, for this architecture we generate 50 MLPs
and 50 RandDense networks with 2-10 hidden layers for 1 and 10 million
parameter counts. These models are then appended to a CNN-LSTM block. This
block first consists of the same convolutional, average pooling, and global
average pooling layers as in the previous subsection, but with each of them
time distributed (applied in the same way) across every 10 year time window
within the training samples. These time distributed layers are followed by an
LSTM with 25 units.

In order to enable the CNN-LSTM block to make time-aware predictions, the
training data is transformed slightly by slicing it into 10-year moving
windows with a 1-year stride. For example, two such windows might be 2015-2024
and 2016-2025. This results in each historical dataset losing 9 data points
since the last starting year of the 10-year window is 2005 instead of 2014.
Thus, the CNN-LSTM dataset contains $753-3\cdot 9=726$ data points.

The CNN-LSTMs are followed by either conventional MLP or RandDense blocks,
which we will simply refer to as “CNN-LSTM networks,” and “CNN-LSTM RandDense
networks.” We train them with a batch size of 25 using Adam optimizer and mean
squared error loss for 100 epochs. We also use early stopping with a patience
of 5 epochs.

##  5 Results

Table 1: Best total RMSE performance for each model class and predicted variable across all generated models, along with the original CNN-LSTM and pattern scaling models from Watson-Parris et al. (2022). Lower is better, and the better RMSE between the standard and RandDense models is bolded. |  | TAS | DTR | PR | PR90
---|---|---|---|---|---
MLP | Standard | 1.928 | 15.62 | 4.663 | 5.651
RandDense | 1.612 | 14.67 | 4.472 | 5.206
CNN | Standard | 3.350 | 23.15 | 9.235 | 10.30
RandDense | 3.353 | 22.92 | 8.681 | 9.964
CNN-LSTM | Standard | 0.262 | 11.85 | 2.861 | 3.880
RandDense | 0.263 | 11.66 | 2.775 | 3.810
ClimateBench | Neural Network | 0.327 | 16.78 | 3.175 | 4.339
Pattern Scaling | 0.320 | 19.72 | 3.662 | 4.461

A summary of our experimental results is shown in Table 1, which shows the
best total RMSE ($NRMSE_{t}$) performance for each model class across all
generated models, including all numbers of hidden layers and both parameter
counts. For nearly every model architecture and predicted variable, the
RandDense variations outperformed their standard counterparts, the two
exceptions being the TAS prediction task in both convolutional models. In
general, the best performance differences were more stark for the
precipitation variables (PR and PR90). Additionally, the margin of difference
for instances when the RandDense networks performed better were generally
larger than those when the standard networks performed better. A more detailed
breakdown is shown in Table 2, which shows the relative total RMSE performance
improvement provided by the addition of random wiring. There are a few key
trends. First, we see that the performance improvement provided by the
addition of random wiring is most stark for MLPs with 1M parameters, but
decreases with increasing model architecture complexity with some variety with
respect to parameter count. We hypothesize that this is due to the fact that
the randomly wired layers in the more complex convolutional architectures are
farther removed from the input, and so they do less direct computation on the
input as opposed to the CNN or LSTM which directly handle spatial and temporal
dependencies in the data directly. Second, there appears to be a more
consistent statistically significant benefit provided by random wiring for the
precipitation prediction tasks. Lastly, while not every performance change was
statistically significant at the $\alpha=0.05$ significance level, there were
far more instances of statistically significant performance improvements
provided by random wiring than deficits. In fact, there was only one instance
(MLPs with 10M parameters predicting DTR) where the addition of random wiring
incurred a statistically significant performance loss, compared to the 14
cases where random wiring showed a statistically significant performance
improvement.

Table 2: Relative change in best RMSE performance with the addition of random wiring averaged over all generated models for each architecture, parameter count, and predicted variable. Statistically significant changes (p-value ¡ $0.05$) are bolded. |  | TAS | DTR | PR | PR90
---|---|---|---|---|---
MLP | 1M | 30.4% | 12.8% | 20.9% | 21.8%
10M | 5.39% | -11.0% | 2.57% | -0.21%
CNN | 1M | -0.10% | 1.11% | 1.06% | 0.41%
10M | 6.36% | 2.18% | 5.20% | 2.94%
CNN-LSTM | 1M | 0.01% | 0.76% | 2.90% | 1.54%
10M | -0.89% | 1.47% | 4.47% | 3.37%

When compared with the CNN-LSTM model from the original ClimateBench
manuscript (Watson-Parris et al., 2022) (see Table 1), we find that the CNN-
LSTM RandDense models outperform across all prediction tasks, with the
greatest improvements for temperature variables. As in Watson-Parris et al.
(2022), the CNN-LSTM models also consistently outperform a linear pattern
scaling approach which uses independent linear regressions to predict each
variable within each grid box. Additional results, including best performances
for spatial and global RMSE, can be found in the Appendix. The following
subsections are dedicated to discussing the experimental results for each of
the baseline model architectures in greater detail.

###  5.1 MLP

Figure 4: Mean total RMSE of 50 MLP models vs. mean RMSE of 50 RandDense
models for both TAS and DTR. The color heatmaps to the right indicate the
number of hidden layers. Errorbars show $\pm$ standard error of the mean.

A summary of our MLP and RandDense experiments is shown in Figure 4. Broadly,
we find that on average RandDense networks outperform their standard
counterparts at 1M parameters with mixed results at 10M, especially for the
DTR prediction task. Points below the black $y=x$ line indicate that the
RandDense networks outperformed the MLP networks on average, and vice versa
for points above. The farther a point is from the $y=x$ line, the more drastic
the mean performance difference. For the TAS prediction task, the RandDense
networks had better mean RMSE performance across nearly all hidden layer and
parameter counts, with the largest mean performance differences being between
MLP and RandDense networks with 1M parameters. For the 1M parameter models,
both the standard and RandDense variations perform worse with more hidden
layers, but at 10M parameters, the trend reverses for standard networks and
vanishes for the RandDense networks. For the DTR prediction task, the
RandDense models performed better than the MLPs at 1M parameters, and vice
versa at 10M parameters. For both parameter counts, mean performance decreases
as the number of hidden layers increases. Trends for both precipitation
variables are quite similar. In general, the RandDense networks outperform
their MLP counterparts, with some exceptions for networks with high numbers of
hidden layers at 10M parameters. For the MLPs with 1M parameters, performance
decreased slightly with higher numbers of hidden layers, but increased
slightly with more layers for 10M parameter MLPs. Across both parameter
counts, performance for RandDense networks decreased as more layers were
added.

###  5.2 CNN

Figure 5: Mean total RMSE of 50 CNN models vs. mean RMSE of 50 CNN RandDense
models for both TAS and DTR. The color heatmaps to the right indicate the
number of hidden layers. Errorbars show $\pm$ standard error of the mean.

A summary of our CNN and CNN RandDense experiments is shown in Figure 5. In
general, the CNN RandDense networks outperform the CNNs for precipitation
variables, with mixed results for the temperature variables. For the TAS
prediction task, the CNN RandDense networks have overall better mean
performance than their standard counterparts at 10M parameters and vice versa
at 1M parameters. Interestingly, this pattern is reversed for the DTR
prediction task. At 1M parameters, both the CNN and CNN RandDense models had
increased TAS mean performance at higher numbers of layers, with the opposite
trend at 10M parameters. For DTR, however, neither network architecture showed
much change in mean performance with different network depths.

Like the MLP/RandDense networks, the trends in CNN/CNN RandDense mean
performance for both precipitation prediction tasks are generally similar.
With few exceptions, the CNN RandDense networks outperform their standard
counterparts, with the largest mean performance differences at 10M parameters.
Furthermore, the mean performance gains for the RandDense models in the PR
prediction task are generally greater than in the PR90 prediction task. Mean
performance with respect to the number of hidden layers shows no clear trends
across prediction variable or number of parameters.

###  5.3 CNN-LSTM

Figure 6: Mean RMSE of 50 CNN-LSTM models vs. mean RMSE of 50 CNN-LSTM
RandDense models for both TAS and DTR. The color heatmaps to the right
indicate the number of hidden layers. Errorbars show $\pm$ standard error of
the mean.

A summary of our CNN-LSTM and CNN-LSTM RandDense experiments is shown in
Figure 6. Broadly, the CNN-LSTM RandDense networks outperform, on average,
their standard counterparts for precipitation prediction tasks, but show mixed
average performance for temperature prediction tasks. Unlike the previously
discussed model architectures, the RandDense variations of CNN-LSTM models
generally perform worse than their standard counterparts in the TAS prediction
task, regardless of network depth or number of parameters. For the DTR
prediction task, trends in mean performance differ by parameter count. At 1M
parameters, CNN-LSTM models have slightly better mean performance, with the
difference becoming less stark as the the number of layers increases. At 10M
parameters, the CNN-LSTM RandDense models have generally better mean
performance, with larger performance gains at higher numbers of layers.

For both precipitation variables, the CNN-LSTM RandDense networks outperform
their standard counterparts with few exceptions across all network depths and
parameter counts. Mean performance differences are most stark for the PR90
prediction task at high numbers of layers and 10M parameters. For both PR and
PR90, the CNN-LSTM RandDense networks have slight performance gains for deeper
networks at 1M parameters, and the slight performance losses for deeper
networks at 10M parameters.

Since the CNN-LSTM RandDense model had the best performance overall (see Table
1), we further analyze its difference from the NorESM2 ground truth
predictions. Figure 7 shows the NorESM2 predictions alongside the best CNN-
LSTM RandDense predictions, as well as the mean difference between the two.
Across prediction tasks, the mean difference in predictions during the testing
period (2080-2100) was statistically insignificant in most locations.
Furthermore, significant differences, when present, were relativey small. For
example, about 54% of the significant differences in TAS prediction are within
0.1K ($\sim 4.5\%$ error) and 90% of significant differences are within 0.22 K
($\sim 11.8\%$ error).

In general, the CNN-LSTM RandDense model is more performant over the ocean
than land for temperature variables. This is especially true for DTR. Among
statistically significant errors (see Figure 7), the CNN-LSTM RandDense model
had an average DTR error of 0.05K over sea compared to a 0.17K over land.
Additionally, it tends to overestimate warming in the Northern hemisphere but
underestimates in the Southern hemisphere. For the precipitation variables,
the model generally makes better predictions over land. The area which the
models makes the worst predictions in is the intertropical convergence zone
(ITCZ) between South America and Africa. Specifically, the model
underestimates both average and extreme precipitation over the ITCZ’s winter
location (Schneider et al., 2014). We hypothesize that it struggles in this
region due to variability in climate change-induced ITCZ shift across the
simulations the model was trained on.

Figure 7: NorESM2 ground truth predictions for each variable averaged from
2080-2100, alongside the predictions averaged over the same time period of the
best performing CNN-LSTM RandDense models. Also shown is the mean difference
between the NorESM2 and CNN-LSTM RandDense predictions, once again averaged
over the 2080-2100 testing period. Statistically insignificant differences
$(p>0.05)$ are masked.

##  6 Discussion

Each model architecture experiment set shares a common theme across the
experimental results presented here: there is at least one performance metric
and prediction task where randomization provides clear benefits. For example,
even though randomization did not provide clear mean DTR performance benefits,
Table 1 shows that the best performance metric favored the RandDense models
across all tested architectures. This suggests that for the task of climate
model emulation, scientists and practitioners are likely to achieve some
performance benefit from generating a limited number of randomly wired
variations of their existing models with a similar number of parameters. In
scenarios where finely-tuned and optimized model predictive skill is of
critical importance, investigating network randomization is an important
design choice to consider. With this in mind, the remainder of this section is
dedicated to further exploring random networks and why they might achieve the
results they do.

###  6.1 Prediction Speed

The speed with which neural networks make their predictions is important for
both standalone models and models within a larger system. Researchers seeking
to quickly construct, validate, and test standalone models for multiple
applications would clearly benefit from faster models. Similarly, researchers
seeking to use a neural network to model a subprocess in a large climate model
(Irrgang et al., 2021; Holder et al., 2022; Silva et al., 2021; Geiss et al.,
2023) would also benefit from quicker predictions, since such submodels are
often called many thousands of times per timestep in a full Earth system
model. As such, we investigate the prediction speed of randomly wired neural
networks and compare them with their feedforward counterparts.

For fair comparison, we took the a best performing standard model, say a 6
layer MLP, and the best performing RandDense equivalent and used them to make
predictions on the test data 10 times. This is done because the time required
for just one prediction is quite small. We repeated nine more times for a
total of ten trials. Lastly, we conducted an independent sample t-test on the
two samples of 10 run times to investigate whether the mean run times of the
two models were different in a statistically significant way.

Across the several different model architectures, layer counts, and prediction
tasks we find no consistent difference between the run times of the standard
models and their randomly wired variations. In a large majority of
comparisons, the difference was statistically insignificant. This is partially
illustrated in Figure 8, which shows 95% confidence intervals for PR
prediction speed across model architectures. Notice that in nearly every case,
the confidence intervals for the standard and RandDense models overlap.
Furthermore, neither the standard or RandDense networks consistently had
faster average runtimes. We conclude that random wiring neither benefits nor
detracts from prediction speed, making RandDense networks suitable
replacements for standard MLPs either in standalone or sub-model applications.

Figure 8: The average PR prediction times for the best performing models
within each model class along with 95% confidence intervals.

###  6.2 Sensitivity to the Node Operation

The node operation of the random networks discussed in Section 2 and
illustrated in Figure 1 was a specific design choice by the authors, inspired
by the choices of Xie et al. (2019) and Geiss et al. (2023). There are many
valid alternative designs, however. Here, we investigate two modifications to
our node operation:

  1. 1.

Placing the ReLU activation after the Dense layer instead of before.

  2. 2.

Replacing the weighted sum with a simple unweighted sum.

With the ReLU activation after the dense layer, we found different results
across the three model architectures. 1M parameter RandDense network
performance decreased for the all prediction tasks by about $20\%$, but in
many cases the RandDense networks still outperformed their standard
counterparts. The same was observed for 10M parameter networks, except when
predicting DTR, where RandDense performance improved by $20\%$. For the CNN
RandDense models, there were mixed mean performance changes for the
temperature variables, just a few percentage points increase at 1M parameters
and the opposite at 10M parameters. There was little change in performance
($<1\%$) predicting precipitation. Lastly, for the CNN-LSTM RandDense models,
when the ReLU activation was placed after the dense layer there was a $6\%$
improvement TAS prediction performance, $8\%$ decrease in DTR prediction
performance, and limited change in performance ($<1\%$) for both precipitation
variables.

Removing the weighted sum and replacing it with a simple unweighted sum made
no significant performance difference across all model types, predicted
variables, layer counts, or parameter counts. These performance changes, or
lack thereof, as a result of node operation modifications highlight the
importance of specific components of the operation.

###  6.3 Further Investigating the Effects of Random Wiring

We hypothesize that random wiring may affect a neural network’s performance by
either

  1. 1.

Benefiting/hindering the training of the full model, or

  2. 2.

Processing the output of the previous layers better/worse.

To explore these hypotheses, we attempted to isolate the source of performance
benefits or deficits for randomly wired neural networks. Specifically, for the
CNN and CNN-LSTM architectures we trained the best performing conventional
models from scratch. We then isolated the trained CNN or CNN-LSTM block and
froze its weights. Lastly, we attached to this block the best performing
randomly wired architecture with randomly initialized weights and retrained
the model. This is a similar idea to that of previous work which configured
artificially evolved nanoparticle systems into Boolean logic gates (Bose et
al., 2015). Since the CNN or CNN-LSTM block has been frozen, only the weights
in the randomly wired layers are trained in this iteration. If the performance
of this new model was different from the original random model which had both
its convolutional block and random layers trained from scratch, then it would
indicate that random wiring was beneficial or detrimental to the training
process. On the other hand, if the frozen model and the original random model
performed the same, it would indicate that the randomly wired layers are
helping or hurting the model in the way they process the output of the
CNN/CNN-LSTM block.

We repeated this experimental procedure 10 times for both the CNN and CNN-
LSTM, analyzing both 1 million and 10 million parameter counts at 2, 6, and 10
layers. All four prediction tasks were analyzed for this experiment. Figure 9
shows a summary of the results for the CNN RandDense networks. The figure
illustrating the CNN-LSTM RandDense results may be found in the Appendix.
Overall, we find no consistent trend in performance differences between the
RandDense models trained from scratch and those with frozen CNN or CNN-LSTM
blocks with only one exception: the DTR prediction task for CNN networks.
While the confidence intervals for RandDense networks and their frozen
counterparts may not always overlap, neither performs better in general. This
provides evidence that random wiring impacts performance due to the way it
handles the output of the convolutional block, and not by it affecting the
CNN’s training process.

Figure 9: Mean RMSE performance of 10 CNN RandDense networks vs. mean RMSE
performance of 10 CNN RandDense networks with frozen weights from the best
performing standard CNN network. The x-axis indicates the number of hidden
layers and parameters. For example, “6, 1M” means 6 hidden layers and 1M
parameters

##  7 Conclusion

Motivated by the promise of randomly wired neural networks in related
applications, we explore random wirings between dense layers of neural
networks for the task of climate model emulation. We replaced the traditional
feedforward dense layers in several model architectures with our randomly
wired networks, coined “RandDense” networks, and conduct performance
experiments using the ClimateBench dataset. Across several architectures,
model complexities, and predicted variables, we find performance benefits for
models containing randomly wired layers, indicating that in many cases,
standard feedforward networks may be effectively replaced with RandDense
networks which achieve better performance at no additional computational cost.

Additionally, we explore various node operations for RandDense networks and
conduct preliminary experiments to understand their performance (Section 6).
However, this leaves the door open for future work. Many more node operation
modifications, such as the inclusion of batch normalization and various
activation functions, remain to be evaluated. Furthermore, our method of
generating random graph structures described in Section 2 is only one way of
accomplishing the task. Special defined types of random graphs have been
explored in other applications (Xie et al., 2019), but assessing their
efficacy for climate model emulation remains a topic for future work. Lastly,
while the experiments in Section 6 provide a brief exploration into why
RandDense networks perform better or worse in certain scenarios, there may be
other reasons why RandDense networks perform differently than their standard
counterparts such as alleviating the vanishing gradient problem (Orhan and
Pitkow, 2017). Further analysis into RandDense networks may shed light on
additional reasons behind their performance discrepancies.

###### Acknowledgements.

The authors would like to thank Joseph Hardin for helpful discussions and
feedback. DWP acknowledges funding from the European Union’s Horizon 2020
research and innovation programme iMIRACLI under Marie Skłodowska-Curie grant
agreement No 860100. The ClimateBench data is available here:
https://doi.org/10.5281/zenodo.5196512. The code used to generate, train, and
evaluate our models can be found here: https://github.com/yikwill/randomly-
wired-nn. Additional Results In this section, we present additional results
from our RMSE and analytical experiments. The Tables 3 and 4 show the best
spatial, global, and total RMSE performance of each model at each parameter
count tested. Figure 10 shows the results of the weigh freezing analysis
experiment for the CNN-LSTM RandDense architecture.

Table 3: Best spatial, global, and total RMSE performance for each model class
and predicted variable across all network depths at 1M parameters, along with
the original CNN-LSTM model from Watson-Parris et al. (2022). Lower is better,
and the better RMSE between the standard and RandDense models is bolded.

|  | TAS | DTR | PR | PR90
---|---|---|---|---|---
|  |  Spatial |  Global |  Total |  Spatial |  Global |  Total |  Spatial |  Global |  Total |  Spatial |  Global |  Total
MLP | Standard | 0.433 | 0.380 | 2.331 | 10.76 | 0.782 | 15.62 | 2.655 | 0.188 | 4.663 | 4.137 | 0.456 | 6.741
RandDense | 0.301 | 0.262 | 1.612 | 9.918 | 0.795 | 14.67 | 2.729 | 0.266 | 4.472 | 3.249 | 0.411 | 5.306
CNN | Standard | 0.698 | 0.610 | 3.747 | 16.44 | 1.307 | 23.61 | 5.078 | 0.811 | 9.235 | 5.957 | 0.867 | 10.30
RandDense | 0.697 | 0.608 | 3.737 | 16.56 | 1.273 | 23.45 | 5.081 | 0.800 | 9.117 | 5.999 | 0.852 | 10.31
CNN-LSTM | Standard | 0.055 | 0.041 | 0.262 | 7.376 | 0.802 | 12.18 | 1.948 | 0.169 | 2.861 | 2.346 | 0.302 | 3.880
RandDense | 0.058 | 0.041 | 0.263 | 7.572 | 0.812 | 12.13 | 1.875 | 0.169 | 2.775 | 2.263 | 0.303 | 3.810
ClimateBench |  | 0.107 | 0.044 | 0.327 | 9.917 | 1.372 | 16.78 | 2.128 | 0.209 | 3.175 | 2.610 | 0.346 | 4.339

Table 4: Best spatial, global, and total RMSE performance for each model class
and predicted variable across all network depths at 10M parameters, along with
the original CNN-LSTM model from Watson-Parris et al. (2022). Lower is better,
and the better RMSE between the standard and RandDense models is bolded.

|  | TAS | DTR | PR | PR90
---|---|---|---|---|---
|  |  Spatial |  Global |  Total |  Spatial |  Global |  Total |  Spatial |  Global |  Total |  Spatial |  Global |  Total
MLP | Standard | 0.354 | 0.315 | 1.928 | 10.30 | 1.083 | 18.41 | 2.484 | 0.453 | 5.146 | 3.017 | 0.496 | 5.651
RandDense | 0.410 | 0.338 | 2.100 | 11.05 | 1.369 | 18.58 | 2.880 | 0.426 | 5.094 | 3.472 | 0.511 | 6.704
CNN | Standard | 0.641 | 0.542 | 3.350 | 15.94 | 1.211 | 23.15 | 5.117 | 0.811 | 9.481 | 5.936 | 0.884 | 10.398
RandDense | 0.643 | 0.542 | 3.353 | 16.56 | 1.176 | 22.92 | 5.114 | 0.711 | 8.681 | 5.905 | 0.785 | 9.964
CNN-LSTM | Standard | 0.061 | 0.041 | 0.264 | 7.273 | 0.815 | 11.85 | 2.003 | 0.169 | 2.911 | 2.342 | 0.301 | 3.906
RandDense | 0.063 | 0.041 | 0.268 | 7.525 | 0.795 | 11.66 | 1.926 | 0.169 | 2.816 | 2.295 | 0.304 | 3.821
ClimateBench |  | 0.107 | 0.044 | 0.327 | 9.917 | 1.372 | 16.78 | 2.128 | 0.209 | 3.175 | 2.610 | 0.346 | 4.339

Figure 10: Mean RMSE performance of 10 CNN-LSTM RandDense networks vs. mean
RMSE performance of 10 CNN-LSTM RandDense networks with frozen weights from
the best performing standard CNN-LSTM network. The x-axis indicates the number
of hidden layers and parameters. For example, “6, 1M” means 6 hidden layers
and 1M parameters

## References

  * Abadi et al. (2015) Abadi, M., and Coauthors, 2015: TensorFlow: Large-scale machine learning on heterogeneous systems. URL https://www.tensorflow.org/, software available from tensorflow.org.
  * Ahmed et al. (2021) Ahmed, A., R. C. Deo, A. Ghahramani, N. Raj, Q. Feng, Z. Yin, and L. Yang, 2021: LSTM integrated with Boruta-random forest optimiser for soil moisture estimation under RCP4. 5 and RCP8. 5 global warming scenarios. Stochastic Environmental Research and Risk Assessment, 35 (9), 1851–1881.
  * Beusch et al. (2020) Beusch, L., L. Gudmundsson, and S. I. Seneviratne, 2020: Emulating Earth system model temperatures with MESMER: from global mean temperature trajectories to grid-point-level realizations on land. Earth System Dynamics, 11 (1), 139–159.
  * Bose et al. (2015) Bose, S., C. P. Lawrence, Z. Liu, K. Makarenko, R. M. van Damme, H. J. Broersma, and W. G. van der Wiel, 2015: Evolution of a designless nanoparticle network into reconfigurable Boolean logic. Nature nanotechnology, 10 (12), 1048–1052.
  * Bretherton et al. (2022) Bretherton, C. S., and Coauthors, 2022: Correcting Coarse-Grid Weather and Climate Models by Machine Learning From Global Storm-Resolving Simulations. Journal of Advances in Modeling Earth Systems, 14 (2), e2021MS002 794.
  * Brun et al. (2018) Brun, O., Y. Yin, E. Gelenbe, Y. M. Kadioglu, J. Augusto-Gonzalez, and M. Ramos, 2018: Deep learning with dense random neural networks for detecting attacks against IoT-connected home environments. International ISCIS Security Workshop, Springer, Cham, 79–89.
  * Chantry et al. (2021) Chantry, M., S. Hatfield, P. Dueben, I. Polichtchouk, and T. Palmer, 2021: Machine learning emulation of gravity wave drag in numerical weather forecasting. Journal of Advances in Modeling Earth Systems, 13 (7), e2021MS002 477.
  * Collins et al. (2012) Collins, M., R. E. Chandler, P. M. Cox, J. M. Huthnance, J. Rougier, and D. B. Stephenson, 2012: Quantifying future climate change. Nature Climate Change, 2 (6), 403–409.
  * Cortes et al. (2012) Cortes, C., M. Mohri, and A. Rostamizadeh, 2012: L2 regularization for learning kernels. arXiv preprint arXiv:1205.2653.
  * Elsken et al. (2019) Elsken, T., J. H. Metzen, and F. Hutter, 2019: Neural architecture search: A survey. The Journal of Machine Learning Research, 20 (1), 1997–2017.
  * Eyring et al. (2016) Eyring, V., S. Bony, G. A. Meehl, C. A. Senior, B. Stevens, R. J. Stouffer, and K. E. Taylor, 2016: Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. Geoscientific Model Development, 9 (5), 1937–1958.
  * Geiss et al. (2023) Geiss, A., P.-L. Ma, B. Singh, and J. C. Hardin, 2023: Emulating aerosol optics with randomly generated neural networks. Geoscientific Model Development, 16 (9), 2355–2370, 10.5194/gmd-16-2355-2023, URL https://gmd.copernicus.org/articles/16/2355/2023/.
  * Gelenbe and Yin (2016) Gelenbe, E., and Y. Yin, 2016: Deep learning with random neural networks. 2016 International Joint Conference on Neural Networks (IJCNN), IEEE, 1633–1638.
  * Ghorbani et al. (2018) Ghorbani, M., R. C. Deo, Z. M. Yaseen, M. H Kashani, and B. Mohammadi, 2018: Pan evaporation prediction using a hybrid multilayer perceptron-firefly algorithm (MLP-FFA) model: case study in North Iran. Theoretical and applied climatology, 133 (3), 1119–1131.
  * Gillett et al. (2016) Gillett, N. P., and Coauthors, 2016: The detection and attribution model intercomparison project (DAMIP v1. 0) contribution to CMIP6. Geoscientific Model Development, 9 (10), 3685–3697.
  * Holder et al. (2022) Holder, C., A. Gnanadesikan, and M. Aude-Pradal, 2022: Using neural network ensembles to separate ocean biogeochemical and physical drivers of phytoplankton biogeography in Earth system models. Geoscientific Model Development, 15 (4), 1595–1617.
  * Irrgang et al. (2021) Irrgang, C., N. Boers, M. Sonnewald, E. A. Barnes, C. Kadow, J. Staneva, and J. Saynisch-Wagner, 2021: Towards neural Earth system modelling by integrating artificial intelligence in Earth system science. Nature Machine Intelligence, 3 (8), 667–674.
  * Kasim et al. (2021) Kasim, M., and Coauthors, 2021: Building high accuracy emulators for scientific simulations with deep neural architecture search. Machine Learning: Science and Technology, 3 (1), 015 013.
  * Kingma and Ba (2014) Kingma, D. P., and J. Ba, 2014: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  * Krasnopolsky et al. (2005) Krasnopolsky, V. M., M. S. Fox-Rabinovitz, and D. V. Chalikov, 2005: New approach to calculation of atmospheric model physics: Accurate and fast neural network emulation of longwave radiation in a climate model. Monthly Weather Review, 133 (5), 1370–1383.
  * Lee et al. (2019) Lee, C. S., E. Sohn, J. D. Park, and J.-D. Jang, 2019: Estimation of soil moisture using deep learning based on satellite data: A case study of South Korea. GIScience & Remote Sensing, 56 (1), 43–67.
  * Li et al. (2021) Li, Z., F. Liu, W. Yang, S. Peng, and J. Zhou, 2021: A survey of convolutional neural networks: analysis, applications, and prospects. IEEE transactions on neural networks and learning systems.
  * Liu et al. (2020) Liu, M., Y. Huang, Z. Li, B. Tong, Z. Liu, M. Sun, F. Jiang, and H. Zhang, 2020: The applicability of LSTM-KNN model for real-time flood forecasting in different climate zones in China. Water, 12 (2), 440.
  * Mansfield et al. (2020) Mansfield, L. A., P. J. Nowack, M. Kasoar, R. G. Everitt, W. J. Collins, and A. Voulgarakis, 2020: Predicting global patterns of long-term climate change from short-term simulations using machine learning. npj Climate and Atmospheric Science, 3 (1), 1–9.
  * Meinshausen et al. (2011) Meinshausen, M., S. C. Raper, and T. M. Wigley, 2011: Emulating coupled atmosphere-ocean and carbon cycle models with a simpler model, MAGICC6–Part 1: Model description and calibration. Atmospheric Chemistry and Physics, 11 (4), 1417–1456.
  * Mooers et al. (2021) Mooers, G., M. Pritchard, T. Beucler, J. Ott, G. Yacalis, P. Baldi, and P. Gentine, 2021: Assessing the Potential of Deep Learning for Emulating Cloud Superparameterization in Climate Models With Real-Geography Boundary Conditions. Journal of Advances in Modeling Earth Systems, 13 (5), e2020MS002 385.
  * Mounier et al. (2022) Mounier, A., L. Raynaud, L. Rottner, M. Plu, P. Arbogast, M. Kreitz, L. Mignan, and B. Touzé, 2022: Detection of bow echoes in kilometer-scale forecasts using a convolutional neural network. Artificial Intelligence for the Earth Systems, 1–66.
  * Nikolaev et al. (2020) Nikolaev, A., I. Richter, and P. Sadowski, 2020: Deep learning for climate models of the Atlantic ocean. AAAI Spring Symposium: MLPS, 2020.
  * O’Neill et al. (2016) O’Neill, B. C., and Coauthors, 2016: The scenario model intercomparison project (ScenarioMIP) for CMIP6. Geoscientific Model Development, 9 (9), 3461–3482.
  * Orhan and Pitkow (2017) Orhan, A. E., and X. Pitkow, 2017: Skip connections eliminate singularities. arXiv preprint arXiv:1701.09175.
  * O’Shea and Nash (2015) O’Shea, K., and R. Nash, 2015: An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458.
  * Price and Rasp (2022) Price, I., and S. Rasp, 2022: Increasing the accuracy and resolution of precipitation forecasts using deep generative models. International Conference on Artificial Intelligence and Statistics, PMLR, 10 555–10 571.
  * Rasp (2020) Rasp, S., 2020: Coupled online learning as a way to tackle instabilities and biases in neural network parameterizations: general algorithms and Lorenz 96 case study (v1. 0). Geoscientific Model Development, 13 (5), 2185–2196.
  * Rasp et al. (2020) Rasp, S., P. D. Dueben, S. Scher, J. A. Weyn, S. Mouatadid, and N. Thuerey, 2020: WeatherBench: a benchmark data set for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems, 12 (11), e2020MS002 203.
  * Rasp et al. (2018) Rasp, S., M. S. Pritchard, and P. Gentine, 2018: Deep learning to represent subgrid processes in climate models. Proceedings of the National Academy of Sciences, 115 (39), 9684–9689.
  * Rasp and Thuerey (2021) Rasp, S., and N. Thuerey, 2021: Data-driven medium-range weather prediction with a resnet pretrained on climate simulations: A new model for weatherbench. Journal of Advances in Modeling Earth Systems, 13 (2), e2020MS002 405.
  * Rozos et al. (2021) Rozos, E., P. Dimitriadis, K. Mazi, and A. D. Koussis, 2021: A multilayer perceptron model for stochastic synthesis. Hydrology, 8 (2), 67.
  * Schneider et al. (2014) Schneider, T., T. Bischoff, and G. H. Haug, 2014: Migrations and dynamics of the intertropical convergence zone. Nature, 513 (7516), 45–53.
  * Seifert and Rasp (2020) Seifert, A., and S. Rasp, 2020: Potential and limitations of machine learning for modeling warm-rain cloud microphysical processes. Journal of Advances in Modeling Earth Systems, 12 (12), e2020MS002 301.
  * Seland et al. (2020) Seland, Ø., and Coauthors, 2020: Overview of the Norwegian Earth System Model (NorESM2) and key climate response of CMIP6 DECK, historical, and scenario simulations. Geoscientific Model Development, 13 (12), 6165–6200.
  * Silva et al. (2020) Silva, S. J., C. L. Heald, and A. B. Guenther, 2020: Development of a reduced-complexity plant canopy physics surrogate model for use in chemical transport models: A case study with GEOS-Chem v12. 3.0. Geoscientific Model Development, 13 (6), 2569–2585.
  * Silva et al. (2021) Silva, S. J., P.-L. Ma, J. C. Hardin, and D. Rothenberg, 2021: Physically regularized machine learning emulators of aerosol activation. Geoscientific Model Development, 14 (5), 3067–3077.
  * Tebaldi et al. (2020) Tebaldi, C., A. Armbruster, H. Engler, and R. Link, 2020: Emulating climate extreme indices. Environmental Research Letters, 15 (7), 074 006.
  * Tran Anh et al. (2019) Tran Anh, D., S. P. Van, T. D. Dang, and L. P. Hoang, 2019: Downscaling rainfall using deep learning long short-term memory and feedforward neural network. International Journal of Climatology, 39 (10), 4170–4188.
  * Trebing et al. (2021) Trebing, K., T. Staǹczyk, and S. Mehrkanoon, 2021: SmaAt-UNet: Precipitation nowcasting using a small attention-UNet architecture. Pattern Recognition Letters, 145, 178–186.
  * Watson-Parris et al. (2022) Watson-Parris, D., and Coauthors, 2022: ClimateBench v1.0: A Benchmark for Data-Driven Climate Projections. Journal of Advances in Modeling Earth Systems, 14 (10), e2021MS002 954, https://doi.org/10.1029/2021MS002954, URL https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2021MS002954, e2021MS002954 2021MS002954, https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2021MS002954.
  * Xie et al. (2019) Xie, S., A. Kirillov, R. Girshick, and K. He, 2019: Exploring randomly wired neural networks for image recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, 1284–1293.
  * Yoo et al. (2019) Yoo, C., D. Han, J. Im, and B. Bechtel, 2019: Comparison between convolutional neural networks and random forest for local climate zone classification in mega urban areas using Landsat images. ISPRS Journal of Photogrammetry and Remote Sensing, 157, 155–170.
  * Yuval et al. (2021) Yuval, J., P. A. O’Gorman, and C. N. Hill, 2021: Use of neural networks for stable, accurate and physically consistent parameterization of subgrid atmospheric processes with good performance at reduced precision. Geophysical Research Letters, 48 (6), e2020GL091 363.
  * Yuval and O’Gorman (2020) Yuval, J., and P. A. O’Gorman, 2020: Stable machine-learning parameterization of subgrid processes for climate modeling at a range of resolutions. Nature communications, 11 (1), 1–10.


# Self-compression of stimulated Raman backscattering by flying focus

Z. Wu  Y. Zuo  Z. Zhang  X. Wang  J. Mu  X.D. Wang  B. Hu  J. Su  Z. Li  X.
Wei  X. Zeng<EMAIL_ADDRESS>Science and Technology on Plasma Physics
Laboratory, Research Center of Laser Fusion, China Academic of Engineering
Physics, Mianyang, Sichuan, China, 621900.

###### Abstract

A novel regime of self-compression is proposed for plasma-based backward Raman
amplification(BRA) upon flying focus. By using a pumping focus moving with a
speed equal to the group velocity of stimulated Raman backscattering(SRBS),
only a short part of SRBS which does always synchronize with the flying focus
can be amplified. Due to the asymmetrical amplification, the pulse can be
directly compressed in the linear stage of BRA. Therefore, instead of a short
pulse, the Raman spontaneous or a long pulse can seed the BRA amplifiers. The
regime is supported by the 2D particle-in-cell(PIC) simulation without a seed,
presenting that the pump pulse is compressed from 26ps to 116fs, with an
output amplitude comparable with the case of a well-synchronized short seed.
This method provides a significant way to simplify the Raman amplifiers and
overcome the issue of synchronization jitter between the pump and the seed.

Plasma, flying focus, chromatic Raman compression, Backward Raman scattering

††preprint: AIP/123-QED

Currently, laser powers of multiply petawatt has been reported by large laser
facilities relying on chirped pulse amplification(CPA)Strickland and Mourou
(1985). A further enhancement of laser power is impeded by the material damage
of compression gratings. The difficult is proposed to be overcame by using
plasma for laser amplification. The plasma amplifiers, based on backward Raman
amplification (BRA)Malkin, Shvets, and Fisch (1999, 2000); Mourou  _et al._
(2012); Ping, Cheng, and Suckewer (2004); Cheng  _et al._ (2005); Ren  _et
al._ (2008) or strongly coupled stimulated Brillouin scattering(scSBS)Andreev
_et al._ (2006); Lancia  _et al._ (2010, 2016); Marqués  _et al._ (2019) have
shown the amplified intensities over $10^{16}~{}\rm{W/cm^{2}}$, indicating
exawatt laser powers can be obtain within centimeter-diameter amplifiers.

The BRA experiment has shown an amplified seed intensity far exceed that of
the pump, however, the transfer efficiency(6.4% for double passes, and 5.1%
for a single pass)Ren  _et al._ (2008); Turnbull  _et al._ (2012a, b) and pump
depletion (more than 70% of the pump energy remained) were still low. One
explanation attributes the low efficiency to the thermal effects from the pump
before the seed arrived. Plasma heating can influence BRA by accelerating the
electrons close to the phase velocity of plasma wave(about c/20). As a
consequence, BRA would be suppressed by several negative effects such as
particle trapping, Landau damping, and plasma wavebreaking. Another
explanation is the idea condition for the BRA can not be fully satisfied by
the experiment due to an extremely complex setup. For instance, for typical
plasma amplifiers, several laser pulses are used in the experiment for the
ionization pulses, pump and seed, respectively. The spatiotemporal overlap of
them can hardly be kept due to the perturbation of the laser system.

Flying focus has been proposed to overcome the precursors and plasma heating
in front of the seedTurnbull  _et al._ (2018a); Froula  _et al._ (2018). In
this scheme, a laser focus is able to move with an artificial velocity after a
chromatically focusing system. By setting the focusing pump intensity just
above the ionization threshold of the background gas, a ionization wave is
generated with a moving velocity equal to the flying focusTurnbull  _et al._
(2018b). For a flying velocity around $v=c$, the seed pulse can always be
arranged just behind the ionization wave. Therefore, a clean environment,
without pump heating and Raman spontaneous, is created in front of the seed.
This scheme, however, requires a more precise synchronization of the pump and
the seed, or the amplified seed will be extremely unstable. A stationary
plasma-wave seed with a density structure of two-peaks is able to solve the
issue of synchronization jitterKenan Qu and Fisch (2017). However, it is still
a challenge job to produce such a plasma structure. To our knowledge, it has
not be experimentally realized yet.

Figure 1: Illustration of the self-compression regime due to SRBS by flying
focus. The flying focus with a growth curve is set the same velocity(c) as
SRBS but counter to the pump velocity(-c). (a)The plasma and SRBS is initially
created at the entrance by flying focus. (b) The SRBS propagating with flying
focus is amplified to a short pulse.

In this paper, we show for the first time the 2d particle-in-cell(PIC)
simulation of BRA with flying focus. Moreover, it is proposed a self-
compression regime to simplify the Raman amplifiers and solve the issue of the
synchronization jitter. The scheme is displayed in Fig.1. As the focal
intensity of the pump is slightly higher than the ionization threshold of the
background gas, SRBS mainly occurs within the pump focus. By adjusting the
flying velocity the same as the SRBS, the growth curve $\gamma$ would always
overlap with a small part of initial SRBS, leading to an asymmetrical
amplification. Consequently, the amplified pulse is compressed to around the
width of $\gamma$ curve in the linear stage of BRA, as shown in Fig.1(b). A
self-compression regime is generated to transfer the long disordered Raman
noisy to a short coherent seed. Similarly, it can transfer a long seed to a
short one, indicating the Raman spontaneous or a long pulse can seed Raman
amplifiers. As Raman spontaneous is a parasitic pulse of the pump, and a long
duration seed can always partly overlap with the pump focus, the
synchronization jitter can be naturally avoided.

In general, flying focus can be described by focusing a collimated laser beam
with chirp and pulse group delay(PDG). The spatio-frequency E-field of the
flying focus consists of a superposition of Gaussian beams of different
frequencies, with there waist located at different positions. Commonly, the
spatio-temporal E-field can only be numerically obtained by the inverse
Fourier transform. However, in the limit of large pulse chirp, it can be
estimated analytically asSAINTE-MARIE, GOBERT, and QUÉRÉ (2017)

$\displaystyle\begin{array}[]{rcl}a(z,t^{\prime})=\frac{a_{0}(z,t^{\prime}/\beta)}{\sqrt{1+z(t^{\prime})^{2}}}e^{i\phi(z,t^{\prime})}\end{array}$
(2)

With

$\displaystyle\begin{array}[]{rcl}\phi(z,t^{\prime})=arctan[z(t^{\prime})]+\frac{kr^{2}}{2z(t^{\prime})[1+z(t^{\prime})^{2}]}+\frac{{t^{\prime}}^{2}}{2\beta}\end{array}$
(4)

where $t^{\prime}=t-z/c$, $z(t^{\prime})=t^{\prime}\tau/\beta-z/z_{R}$,
$\beta$ is the pulse chirp, $\tau$ is the parameter of PGD, and $z_{R}$ is the
Rayleigh length. The beam waist locates at the position of $z=t^{\prime}\tau
z_{R}/\beta$ for $z(t^{\prime})=0$, indicating the focus spot is not static
but variable along the propagating direction. By adjusting the pulse chirp
$\beta$ and PGD $\tau$, the focus can move with a velocity from subluminal to
superluminalSAINTE-MARIE, GOBERT, and QUÉRÉ (2017); Froula  _et al._ (2018,
2019).

As the growth rate of the Langmuir wave and SRBS are the same in the linear
stage, the initial Raman spontaneous $b_{s}$ can be estimated by the
magnification of the energy intensity of Langmuir wave from thermal noisy to
pump depletionMalkin and Fisch (2014),

$\displaystyle\begin{array}[]{rcl}b_{s}\approx
a_{0}\sqrt{\frac{W_{eT}}{W_{ek}}}=\frac{a_{0}e}{m_{e}c^{2}}\sqrt{\frac{T_{e}}{8\pi\varepsilon_{0}\lambda_{a}}}\end{array}$
(6)

where $T_{e}$ is the electron temperature, $e$ and $m_{e}$ is the electron
charge and mass, and $\varepsilon_{0}$ is the permittivity of free space, and
c is speed of light in vacuum,
$W_{eT}\sim\frac{T_{e}\omega_{p}\omega_{a}}{4\pi\lambda_{a}}$ and
$W_{ek}\sim\frac{I_{0}\omega_{p}}{\omega_{a}}$ are the energy density of
Langmuir wave for the equilibrium thermal noise and pump depletion,
respectively, for $I_{0}$ is the pump intensity,
$\omega_{p}=\sqrt{\frac{n_{e}e^{2}}{m_{e}\varepsilon_{0}}}$ and $\omega_{a}$
are the frequency of the Langmuir wave and pump, and $n_{e}$ is the electron
plasma density.

Figure 2: (a)Numerical solution of SRBS growth rate with various plasma
densities and temperatures. (b)On-axial $|b|$ from spontaneous at different
plasma densities. (c)Spectrum of on-axial $|b|$ from spontaneous at different
plasma densities. (d)On-axial $|b|$ at t=900T for plasma density of
0.015$n_{c}$ and 0.03$n_{c}$. In the simulation, $|b|$ is the absolute
amplitude of SRBS, $T_{e}=25$eV, $a_{0}=0.01$,
$n_{c}=\omega^{2}_{a}m_{e}\varepsilon_{0}/e^{2}$ is the critical plasma
density.

An optimal window of parameter is expected for the self-compression regime.
The pump intensity has little space to be improved because it has to be
slightly above the ionization threshold(0.01 in our simulation). Therefore,
the BRA performance is mainly optimized from plasma density and temperature.
The idea linear SRBS growth rate is $\gamma=\sqrt{\omega_{p}\omega_{a}}/2$.
With damping and frequency detuning, it changes to $Re(\lambda)$ for a
langmuir wave $f\propto e^{\lambda t}$. $\lambda$ satisfiesBalakin, Levin, and
Skobelev (2020)

$\displaystyle\begin{array}[]{rcl}\lambda^{2}+(\nu+i\delta\omega)\lambda-\gamma^{2}=0,\end{array}$
(8)

where $\nu=\nu_{ei}/4+\nu_{Ld}$ is the total damping, for collision damping
$\nu_{ei}=2.9\times 10^{6}Zn_{e}[cm^{-3}]\Lambda[eV]^{-3/2}$, where $\Lambda$
is the Coulomb logarithm, $Z$ is the plasma ion chargeBalakin  _et al._
(2011); Clark and Fisch (2003); Johnson  _et al._ (2017); Malkin and Fisch
(2009), Landua damping
$\nu_{Ld}=\omega_{p}\sqrt{\pi}/{{(2q_{T})}^{3/2}}exp(-1/2q_{T}-3/2)$,
$\delta\omega=k_{f}T_{e}/\omega_{p}m_{e}$ is the frequency detuning due to
thermal chirp, where $q_{T}=k_{f}T_{e}/\omega_{p}m_{e}$, $k_{f}$ is the wave
number of the Langmuir waveBalakin  _et al._ (2011); Malkin and Fisch (2009).
The numerical solution of SRBS growth rate is shown in Fig.2, illustrating an
optimal temperature can be found for each plasma density.

In order to further optimize the plasma density, we used a 2D particle-in-
cell(PIC) code OPIC2DZhang  _et al._ (2012, 2014, 2017) to simulate the growth
of Raman spontaneous with different plasma densities. The best amplification
is obtained at density of 0.15$n_{c}$, as shown in Fig.2(b), corresponding to
an optimal plasma temperature around 25eV. The major limitation of plasma
density is from scSBS effect. As the pump is also used for ionization pulse,
the ionization wave in front of SRBS can reflect pump to form an scSBS seed.
Although the reflectivity is low, the intensity accumulating with the
ionization wave can achieve to the same level as the pump, as shown in the on-
axis SRBS at the density of 0.03$n_{c}$[Fig.2(c)]. Despite of a low pump
intensity(the amplitude is 0.01, corresponding to an intensity of $2.7\times
10^{14}~{}\rm{W/cm^{2}}$ for the wavelength of 1 $\mu$m), the condition of
scSBS can still be satisfied because of a low plasma temperature(25 eV),
according to the threshold of scSBS $I>1.1\times
10^{13}T_{e}^{3/2}n_{c}/n\sqrt{1-n/n_{c}}\lambda~{}\rm{W/cm^{2}}$. The scSBS
effect becomes significant as the plasma density increases over 0.025nc.
Typically, the interfringes generated by SRBS and scSBS appear in the
amplified pulse, and SBS takes up an increasing proportion in the spectrum, as
shown in Fig.2(b)and(c).

In the 2D simulation, the ionization and BRA process with flying focus are
simulated in a moving window of $600\lambda\times 100\lambda$ for the z-y
plane, with cells of $\Delta_{z}=0.1\lambda$ and $\Delta_{y}=0.2\lambda$, and
$16\times 2$ particles for each cell, where $z$ and $y$ are the prorogating
and transverse direction, respectively. A 4-mm-long and 50-$\mu$m-wide
hydrogen is applied for the background gas. For the interaction length of 4
mm, the pump duration is about 26 ps, with a central wavelength of 1 $\mu$m.
The initial pump focus has a waist of 5 $\mu$m, with a maximum amplitude of
0.02. After it countering propagates along the z axial of the moving window,
the beam waist is magnified to about 10 $\mu$m due to ionization-induced
defocusing, so the peak amplitude is reduced to 0.01, which is slightly above
the ionization threshold of Hydrogen, as shown in Fig.3(a). Beyond the focus
about 200 $\mu$m, the background gas at the right boundary of the moving
window is kept neutral, as shown in the plasma density[Fig.3(b)] obtained by
an Ammosov-Delone-Krainov tunnel ionization model(ADK).

Figure 3: (a)Flying focus at t=900T, where T=3.3fs is the single cycle of 1
$\mu$m laser, $|a|$ is absolute valvule of normalized pump amplitude. The
velocity of flying focus and moving window is the same(0.99c), so the focus is
static in the simulation window. (b)Plasma density at t=900T, the gas at right
boundary of the moving window is kept neutral. (c)2D PIC simulation result of
SRBS amplitude $|b|$ developed from Raman spontaneous for $n_{e}=0.015n_{c}$,
$T_{e}=25~{}\rm{eV}$.

The optimal 2d simulation result of self-compression process is shown in
Fig3(c). The initial spontaneous amplitude of $\sim 3\times
10^{-5}~{}\rm{W/cm^{2}}$ is obtained by inverse fast fourier transform(IFFT)
of the wavelength from 1100 nm-1200 nm. After 2 mm amplification, the pump
starts to be depleted as the interaction reaches the nonlinear stage. The
pulse amplitude is further amplified to about 0.066 after 4-mm amplification.
Due to the diffractive effect, the amplified SRBS beam is diverged from 10
$\mu m$ to 20 $\mu m$. The total transfer efficiency is about 40%. The
simulation result indicates a short laser pulse with an amplitude far exceed
that of the pump can be obtained from Raman spontaneous upon flying focus. As
only one pump pulse is applied for the amplification, the Raman amplifiers can
be significantly simplified.

The simulation results with a well-synchronized short seed and a long-duration
seed are displayed in Fig.4. Both the long and short seeds have a central
wavelength 1140 nm, an amplitude of 0.001, a beam waist of 100 $\mu$m(for a
Rayleigh length more than 4 mm). The long seed with a duration of 2 ps covers
the full moving window while the short seed with a duration of 200 fs locates
at 400$\mu$m of the moving window. As shown in Fig.4(a), although the initial
amplitude of the Raman spontaneous is 1/30 as the seeds, the amplified
amplitudes tend to be comparable after the 4-mm amplification. This is mainly
due to the growth rate $\gamma$ degrades from exponential to near linear after
the pump is depleted. Despite a strong seed enable the interaction to reach
the nonlinear stage earlier, the growth then is approached by the case of a
Raman noisy seed. The evolution of pulse durations are displayed in Fig.4(b):
both the Raman spontaneous and 2ps seed are directly compressed in the linear
stage, despite the duration of Raman spontaneous slightly returns back during
a short time. The 200fs seed is stretched in the linear stage because it is
already narrower than the growth rate curve. In the nonlinear, all pulses are
further compressed to 116 fs, 65 fs, and 40 fs by the Raman spontaneous, the
200 fs seed and the 20 ps seed, respectively.

The best result is obtained by the long-duration seed, with an output peak
amplitude and a transfer efficiency of 0.12 and 28.3% respectively, which are
higher than 0.09 and 24.4% obtained by the short seed. This is mainly due to a
better envelope match of the SRBS maximum with gain maximum. The concept of
envelope match has been proposed in the seed-ionizing BRAZhang  _et al._
(2014). Here we show a similar regime by using a long-duration seed and flying
focus. In the linear stage, the SRBS maximum has a subluminous speedMalkin,
Shvets, and Fisch (1999), a long-duration seed can accelerate the maximum
moving by providing a plasma wave to the following interaction from the pulse
front. Therefore, the linear amplified seed peak is more close to the pump
focus(400$\lambda$), as shown in Fig.4(c). Conversely, in the nonlinear stage
when the SRBS peak has a superluminous speedMalkin, Shvets, and Fisch (1999),
the acceleration of the SRBS maximum moving becomes a negative factor.
However, this issue can be overcame here because the SRBS can not exceed the
ionization wave producing by the flying focus. As shown in Fig.4(d), the
amplified peak of the 2 ps seed still does not exceed the pump focus at 4000T.
Therefore, different from the previous conclusion that a short seed with a
sharp front is preferred for BRA, a long-duration seed here achieves to an
even better result.

Figure 4: Simulation result of (a) peak intensity and (b) FWHM duration with a
2 ps seed, a 200 fs seed, and Raman spontaneous.(c) On-axial pulse shapes of
SRBS at t=900T. (d) On-axial pulse shapes of SRBS at t=4000T.

One thing to be noted is that the numerical noisy in the PIC code is several
orders of magnitude as the thermal fluctuation in plasma. The numerical noisy
can be reduced by increasing particles per cell, however, it brings huge
calculation for the simulation. Alternatively, as the amplification of thermal
fluctuation to the numerical noisy is very close to the idea linear
amplification, an extra interaction length can be estimated as
$L=ln(\frac{b_{n}}{b_{s}})\frac{c}{\gamma}$, where $b_{s}$ and $b_{n}$ are the
SRBS due to thermal fluctuation and numerical noisy, respectively. For
$a_{0}=0.01$, $n_{e}=0.02n_{c}$, $T_{e}=5$ eV, $\lambda_{a}=1~{}\mu$m, it has
$b_{s}\approx 10^{-9}\approx b_{n}/4000$ according to Eq.6, which agrees well
with the result obtained by $\delta-$correct noiseBalakin, Levin, and Skobelev
(2020). Hence, the extra interaction length is estimated bout $L=0.85$ mm. The
transfer efficiency will be reduced to about 16% because of this extra
interaction length. However, as Raman spontaneous can almost perfectly match
the resonant frequency and spatio-temporal synchronization with the pump
pulse, probably it can lead to an even higher transfer efficiency in the
experiment. For instance, the highest efficiency reported by plasma-based
Raman amplifiers so far is from Raman spontaneousVieux  _et al._ (2017).

The amplification of Raman spontaneous may be suppressed by the frequency and
velocity detuning between the flying focus and SRBS. Frequency detuning is
usually produced by plasma inhomogeneity, pulse chirp and plasma heating.
Here, we mainly investigate the influence from the plasma inhomogeneity as the
pulse chirp can be compensated by a well-designed plasma gradient, and plasma
heating can be avoided by flying focusTurnbull  _et al._ (2018a). With
frequency detuning, the linear growth rate becomes
$\gamma^{\prime}=\sqrt{\gamma^{2}-\delta\omega^{2}/4}$ according to Eq.8,
indicating ${\delta\omega}$ can not exceed a threshold of $2\gamma$ for SRBS
growth. However, in a nonuniform plasma, the SRBS can not be fully broken down
even for a density fluctuation far exceed the threshold because there is
always part of plasma satisfies $\delta\omega<2\gamma$. To investigate the
influence from plasma inhomogeneity, a random density fluctuation from
$10\%-30\%$ was used in the PIC model. As shown in Tab.1, the amplification
still keep well at the density fluctuation of $10\%$. The amplified $|b|$ is
more sensitive to the velocity of flying focus, as shown in Tab.1. Especially
for the velocity above $c$, the amplified amplitude decreases to 0.038 at a
velocity of 1.02c. In the experiment, the precisely control of the focus
velocity can be realized by adjusting the distance of 2 compression
gratingsFroula  _et al._ (2018).

Table 1:  Amplified peak amplitudes with various plasma density fluctuation and flying velocities, for $a_{0}=0.01$, $T_{e}=25$eV, $n=0.015n_{c}$. Density fluctuation($\%$) | 0 | 5 | 10 | 20 | 30
---|---|---|---|---|---
Amplified amplitude($|b|$) | 0.066 | 0.064 | 0.058 | 0.050 | 0.047
Flying velocity(c) | 0.98 | 0.99 | 1 | 1.02 | 1.05
Amplified amplitude($|b|$) | 0.053 | 0.066 | 0.066 | 0.038 | 0.027

Another issue of flying focus is the limitation of interaction aperture. In
order to migrate precursors and plasma heating, it requires producing an
ionization wave a static short distance in front of SRBSTurnbull  _et al._
(2018a). Commonly, the Rayleigh length of the pump focus has to be short, or
the intensity would be sufficiently strong to generate plasma far ahead of the
beam waist. For instance of a Gaussian beam, $z_{R}=w_{0}^{2}/\lambda\approx
6.4$ mm for $w_{0}=100~{}\rm{\mu}$m,$\lambda=1~{}\rm{\mu}$m, where $w_{0}$ is
the beam waist,$\lambda$ is the wavelength, implying the pump will start to
produce plasma 6.4 mm ahead of the flying focus. As a result, both the
precursors and plasma heating can not be migrated. Also, it can not be used
for self-compression. To solve the problem, a random phase plate can be
employed before the achromatical lens to magnify the beam waist without
changing the Rayleigh length. The random phase plate is widely used in large
laser facilities for a beam waist more than 200 $\mu m$Ping  _et al._ (2009).
To further enlarge the beam focus to millimeter or centimeter, a microns array
can be used to merge a series small focus into a large one.

In summary, we propose a new self-compression regime for plasma-based Raman
amplifiers. The use of flying focus can generate a growth rate moving with the
same velocity as SRBS, so only a short part of SRBS always synchronizing with
the flying focus can be amplified, leading to the self-compression in the
linear stage of BRA. This regime is supported by a 2D PIC simulation without a
seed, showing a transfer efficiency around 18% after 4 mm interaction.
Meanwhile, a long-duration seed is demonstrated possible for BRA in this
regime. It leads to an even a better result than a short seed by improving the
envelope match of the SRBS maximum with gain maximum. Above of all, a new way
to generate short laser pulses by Raman scattering is presented. Moveover, by
reducing the complexity and avoiding the synchronization jitter of plasma-
based Raman amplifiers, it is a promising method to generate ultrahigh laser
power.

This work was partly supported by National Key Program for S and T Research
and Development (Grant No. 2018YFA0404804), the National Natural Science
Foundation of China(Grant No. 11875240) and the Science and Technology on
Plasma Physics Laboratory Fund (Grant No. 6142A0403010417).

## References

  * Strickland and Mourou (1985) D. Strickland and G. Mourou, Opt. Commun. 56, 219 (1985).
  * Malkin, Shvets, and Fisch (1999) V. M. Malkin, G. Shvets, and N. J. Fisch, Phys. Rev. Lett 82, 4448 (1999).
  * Malkin, Shvets, and Fisch (2000) V. M. Malkin, G. Shvets, and N. J. Fisch, Phys. Rev. Lett 84, 1208 (2000).
  * Mourou  _et al._ (2012) G. A. Mourou, N. J. Fisch, V. M. Malkin, Z. Toroker, E. A. Khazanov, A. M. Sergeev, T. Tajima, and B. L. Garrec, Optics Communications 285, 720 (2012).
  * Ping, Cheng, and Suckewer (2004) Y. Ping, W. Cheng, and S. Suckewer, Phys. Rev. Lett 92, 175007 (2004).
  * Cheng  _et al._ (2005) W. Cheng, Y. Avitzour, Y. Ping, and S. Suckewer, Phys. Rev. Lett 94, 045003 (2005).
  * Ren  _et al._ (2008) J. Ren, S. Li, A. Morozov, S. Suckewer, N. A. Yampolsky, V. M. Malkin, and N. J. Fisch, Phys. Plasmas 15, 056702 (2008).
  * Andreev  _et al._ (2006) A. A. Andreev, C. Riconda, V. T. Tikhonchuk, and S. Weber, Phys. Plasma 13, 053110 (2006).
  * Lancia  _et al._ (2010) L. Lancia, J. R. Marquès, M. Nakatsutsumi, C. Riconda, S. Weber, S. Hüller, A.Mančić, P. Antici, V. T. Tikhonchuk, A. Héron, P. Audebert, and J. Fuchs, Phys. Rev. Lett. 104, 025001 (2010).
  * Lancia  _et al._ (2016) L. Lancia, A. Giribono, L. Vassura, M. Chiaramello, C. Riconda, S. Weber, A. Castan, A. Chatelain, A. Frank, T. Gangolf, M. N. Quinn, J. Fuchs, and J.-R. Marquès, Phys. Rev. Lett. 116, 075001 (2016).
  * Marqués  _et al._ (2019) J.-R. Marqués, L. Lancia, T. Gangolf, M. Blecher, S. B. nos, J. Fuchs, O. Willi, F. Amiranoff, R. L. Berger, M. Chiaramello, S. Weber, and C. Riconda, PHYSICAL REVIEW X 9, 021008 (2019).
  * Turnbull  _et al._ (2012a) D. Turnbull, S. Li, A. Morozov, and S.Suckewer, Phys. Plasmas 19, 073103 (2012a).
  * Turnbull  _et al._ (2012b) D. Turnbull, S. Li, A. Morozov, and S.Suckewer, Phys. Plasmas 19, 083109 (2012b).
  * Turnbull  _et al._ (2018a) D. Turnbull, S. Bucht, A. Davies, D. Haberberger, T. Kessler, J. L. Shaw, and D. H. Froula1, Phys. Rev. Lett. 120, 024801 (2018a).
  * Froula  _et al._ (2018) D. H. Froula, D. Turnbull, A. S. Davies, T. J. Kessler, D. Haberberger, J. P. Palastro, S.-W. Bahk, I. A. Begishev, R. Boni, S. Bucht, J. Katz1, and J. L. Shaw, Nature photonics 12, 262 (2018).
  * Turnbull  _et al._ (2018b) D. Turnbull, Franke, J. Katz, J. P. Palastro, I. A. Begishev, R. Boni, J. Bromage, A. L. Milder, J. L. Shaw, and D. H. Froula, Phys. Rev. Lett. 120, 225001 (2018b).
  * Kenan Qu and Fisch (2017) I. B. Kenan Qu and N. J. Fisch, Phys. Rev. Lett. 118, 164801 (2017).
  * SAINTE-MARIE, GOBERT, and QUÉRÉ (2017) A. SAINTE-MARIE, O. GOBERT, and F. QUÉRÉ, Optica 4, 1298 (2017).
  * Froula  _et al._ (2019) D. H. Froula, J. P. Palastro, D. Turnbull, A. Davies, L. Nguyen, A. Howard, D. Ramsey, P. Franke, S.-W. Bahk, I. A. Begishev, R. Boni, J. Bromage, S. Bucht, R. K. Follett, D. Haberberger, G. W. Jenkins, J. Katz, T. J. Kessler, J. L. Shaw, and J. Vieira, Phys. Plasmas 26, 032109 (2019).
  * Malkin and Fisch (2014) V. M. Malkin and N. J. Fisch, Eur. Phys. J. Special Topics 223, 1157 (2014).
  * Balakin, Levin, and Skobelev (2020) A. A. Balakin, D. S. Levin, and S. A. Skobelev, Phys. Rev. A 102, 013516 (2020).
  * Balakin  _et al._ (2011) A. A. Balakin, N. J. Fisch, G. M. Fraiman, V. M. Malkin, and Z. Toroker, Phys. Plasmas 18, 102311 (2011).
  * Clark and Fisch (2003) D. S. Clark and N. J. Fisch, Phys. Plasmas 10, 4848 (2003).
  * Johnson  _et al._ (2017) L. A. Johnson, D. F. Gordon, J. P. Palastro, and B. Hafizi, Phys. Plasma 24, 033107 (2017).
  * Malkin and Fisch (2009) V. M. Malkin and N. J. Fisch, Phys. Plasma 80, 046409 (2009).
  * Zhang  _et al._ (2012) Z. M. Zhang, X. T. He, Z. M. Sheng, and M. Y. Yu, Appl. Phys. Lett 100, 134103 (2012).
  * Zhang  _et al._ (2014) Z. M. Zhang, B. Zhang, W. Hong, M. Y. Yu, J. Teng, S. K. He, and a. Y. Q. Gu1, Phys. Plasmas 21, 123109 (2014).
  * Zhang  _et al._ (2017) Z. M. Zhang, B. Zhang, W. Hong, Z. G. Deng, J. Teng, S. K. He, W. M. Zhou, and Y. Q. Gu, Phys. Plasmas 23, 113104 (2017).
  * Vieux  _et al._ (2017) G. Vieux, S. Cipiccia, D. W. Grant, N. Lemos, P. Grant, C. Ciocarlan, B. Ersfeld1, M. S. Hur, P. Lepipas, G. G. Manahan, G. Raj, D. R. Gil, A. Subiel, G. H. Welsh, S. M. Wiggins, S. R. Yoffe, J. P. Farmer, C. Aniculaesei, E. Brunetti1, and X. Yang, Scientific report 7, 2399 (2017).
  * Ping  _et al._ (2009) Y. Ping, R. K. Kirkwood, T.-L. Wang, D. S. Clark, S. C. Wilks, S. C. Wilks, N. Meezan, R. L. Berger, J. Wurtele, N. J. Fisch, V. M. Malkin, E. J. Valeo, S. F. Martins, and C. Joshi, Phys. Plasmas 16, 123113 (2009).