# Graph Neural Network Interatomic Potential Ensembles with Calibrated
Aleatoric and Epistemic Uncertainty on Energy and Forces

Jonas Buska,∗ Mikkel N. Schmidtb Ole Wintherb,c,d Tejs Veggea Peter Bjørn
Jørgensena

( aDepartment of Energy Conversion and Storage, Technical University of
Denmark, Kongens Lyngby, Denmark. E-mail<EMAIL_ADDRESS>
bDepartment of Applied Mathematics and Computer Science, Technical University
of Denmark, Kongens Lyngby, Denmark. E-mail<EMAIL_ADDRESS>
cCenter for Genomic Medicine, Rigshospitalet, Copenhagen University Hospital,
Denmark.
dBioinformatics Centre, Department of Biology, University of Copenhagen,
Denmark.
∗Corresponding author<EMAIL_ADDRESS>)

###### Abstract

Inexpensive machine learning potentials are increasingly being used to speed
up structural optimization and molecular dynamics simulations of materials by
iteratively predicting and applying interatomic forces. In these settings, it
is crucial to detect when predictions are unreliable to avoid wrong or
misleading results. Here, we present a complete framework for training and
recalibrating graph neural network ensemble models to produce accurate
predictions of energy and forces with calibrated uncertainty estimates. The
proposed method considers both epistemic and aleatoric uncertainty and the
total uncertainties are recalibrated post hoc using a nonlinear scaling
function to achieve good calibration on previously unseen data, without loss
of predictive accuracy. The method is demonstrated and evaluated on two
challenging, publicly available datasets, ANI-1x (Smith _et al._ 1) and
Transition1x (Schreiner _et al._ 2), both containing diverse conformations far
from equilibrium. A detailed analysis of the predictive performance and
uncertainty calibration is provided. In all experiments, the proposed method
achieved low prediction error and good uncertainty calibration, with predicted
uncertainty correlating with expected error, on energy and forces. To the best
of our knowledge, the method presented in this paper is the first to consider
a complete framework for obtaining calibrated epistemic and aleatoric
uncertainty predictions on both energy and forces in ML potentials.

##  1 Introduction

Accurate and computationally inexpensive machine learning (ML) potentials are
increasingly being used to accelerate atomic structure optimization and
molecular dynamics simulations by iteratively predicting and applying
interatomic energies and forces 3, 4. This development has the potential to
revolutionise disciplines of computational chemistry such as predicting
molecular properties and structures, predicting reaction mechanisms and
networks, as well as discovering new materials, e.g., for energy conversion
and storage of renewable energy. In these settings, it is crucial to asses the
confidence of predictions and to detect when predictions are unreliable, to
avoid wrong or misleading results by either ending the simulation early or
enabling recovery by falling back to higher fidelity but also more
computationally expensive methods, such as density functional theory (DFT) 5.
Uncertainty quantification (UQ) methods can enable assessment of the
confidence in predictions and thus make applications of ML potentials more
robust and reliable. To ensure uncertainty estimates are useful and
informative, they need to be calibrated, i.e. there should be an agreement
between the predictive distribution and the empirical distribution. Especially
if the uncertainty is expected to indicate the range of plausible values of
the predictions. Good calibration thus ensures that ML-based uncertainty
estimates are interpretable and actionable, and enable the selection of a
suitable confidence threshold for a given application on the original unit
scale of the quantity of interest.

A widely used UQ method for ML potentials is to apply an ensemble of models
and use the agreement of the predictions of the ensemble members as a measure
of confidence in the prediction 6. This approach relies on the observation
that randomly initialised models will often provide increasingly different
predictions further away from the training data distribution. From a Bayesian
perspective, if the individual ensemble members are seen as draws from the
posterior distribution, the variance between predictions of the ensemble
members are a measure of the posterior uncertainty 7, 8, 9. This type of
uncertainty in the model parameters is often referred to as model uncertainty
or _epistemic uncertainty_. Other popular approaches for estimating the
epistemic uncertainty with neural networks include Bayesian neural networks
10, 11 that can directly learn a probability distribution over the neural
network parameters, and Monte Carlo dropout 12 that estimates the predictive
distribution through multiple stochastic forward passes. However, ensembles as
well as these other UQ methods are not inherently calibrated and do not
account for uncertainty in the data, which makes it difficult to select an
appropriate confidence threshold for a given application. Data uncertainty,
also known as _aleatoric uncertainty_ , can originate from noisy observations
and can be estimated with a mean-variance model 13 that treats the uncertainty
variance as an additional model output, or more recently with evidential
learning 14, 15 that learns the parameters of a higher order distribution over
the likelihood parameters, and conformal prediction 16, a distribution-free
approach that estimates a prediction interval directly. The deep ensemble
approach 17 combines an ensemble of mean-variance networks to estimate both
aleatoric and epistemic uncertainty in a unified model.

As discussed above, an important aspect of predictive uncertainty is the
concept of _calibration_ , which implies an agreement between the predicted
uncertainty and the expected empirical error. When evaluating calibration, it
is important to consider the asymmetric relationship between errors and
uncertainties. By the common assumption that errors are drawn from a
distribution (usually Gaussian), small uncertainties should be associated with
small errors and large errors should be associated with large uncertainties,
but large uncertainties can be associated with both small and large errors.
Therefore there is no direct correlation between uncertainties and the
magnitude of errors. However, there should be a correlation between the
uncertainties and the expected magnitude of errors. Several works have
proposed methods for evaluating and validation calibration of regression
models. Kuleshov _et al._ 18 proposed evaluating the coverage of the errors by
the predictive distribution averaged over the data using a calibration curve.
Later, Levi _et al._ 19 proposed checking the correlation of uncertainties and
expected errors computed in bins of increasing uncertainty in a reliability
diagram. Recently, Pernot 20 highlighted the limitations of the previous
approaches and proposed an additional analysis of z-scores (standard scores)
for variance-based UQ methods. UQ and calibration for molecular property
prediction and interatomic ML potentials has been explored in recent
literature 21, 22, 23, 24, 16, but often the training and inference methods
used do not inherently ensure good calibration. The method presented in this
paper is, to the best of our knowledge, the first to consider a complete
framework for obtaining calibrated epistemic and aleatoric uncertainty
predictions on both energy and forces.

In previous work, we have shown how to extend a graph neural network model for
predicting formation energy of molecules to also provide calibrated
uncertainty estimates that can be decomposed into aleatoric and epistemic
uncertainty 25. The method works by combining an ensemble of models with mean-
variance outputs and applying post hoc recalibration with isotonic regression
on data not used for the training. In this work, we further extend the
approach to include calibrated uncertainty on the force predictions.
Specifically, we extend a neural network potential with a probabilistic
predictive distribution on energy and forces, and consider a deep ensemble of
models 17 to express the aleatoric and epistemic uncertainty about the energy
and force predictions. The uncalibrated predictive distributions are then
recalibrated post hoc to fit the error distribution on previously unseen data.
An added benefit of this approach is that ensemble models are generally known
to produce more accurate predictions than single models 6. Through computer
experiments, we demonstrate that the proposed method results in accurate and
calibrated predictions on two publicly available datasets, ANI-1x 1 and
Transitions1x 2 containing out-of-equilibrium and near-transition-state
structures, respectively. The main contribution of the work is a complete
framework for training and evaluating neural network potentials with accurate
predictions and calibrated aleatoric and epistemic uncertainty on both
energies and forces.

The rest of the paper is structured as follows. The proposed method including
the extended graph neural network model and the recalibration procedure is
described in Section 2. The datasets, experiments and results are presented in
Section 3. Finally, the main findings and perspectives are discussed in
Section 4 and we conclude in Section 5.

##  2 Methods

###  2.1 Graph neural network model

As the base model for our ensemble we use PaiNN 26, an equivariant message
passing neural network (MPNN) model designed specifically for predicting
properties of molecules and materials. The model provides a mapping from sets
of atomic species and positions $\\{(Z_{i},\vec{r_{i}})\\}$ to potential
energy $E$ and interatomic forces $\\{\vec{F_{i}}\\}$. The potential energy is
modelled as a sum over the atomic contributions $E_{i}$:

$E=\sum_{i}E_{i}\,,$ (1)

and the forces are computed as the derivative of the potential energy with
respect to the atomic positions, ensuring conservation of energy:

$\vec{F}_{i}=-\partial E/\partial\vec{r}_{i}\,.$ (2)

Specifically, the model input is represented as a graph, where there is an
edge between a pair atoms if the mutual distance between the atoms is below a
certain cutoff. The cutoff distance is treated as a hyperparameter and is
fixed at 5.0 Å in all of our experiments. The neural network architecture
consists of a number of interaction layers, where information, or “messages”,
are exchanged along the edges of the input graph to update the hidden node
states, followed by a readout function represented by a fully connected neural
network that outputs the atom-wise quantities. The number of interaction
layers and the size of the node hidden states are hyperparameters of the
model.

###  2.2 Extended model with aleatoric uncertainty

We extend the base model with additional outputs representing the aleatoric
energy uncertainty $\sigma^{2}_{E}=\sum_{i}\sigma^{2}_{E_{i}}$ and atom-wise
aleatoric force uncertainties $\\{\sigma^{2}_{F_{i}}\\}$. The atom-wise
quantities, $\sigma^{2}_{E_{i}}$ and $\sigma^{2}_{F_{i}}$, are constrained to
be positive by passing them through a softplus activation function,
$\log(1+\exp(\cdot))$, and adding a small constant for numerical stability.
Note that here we chose to represent the atom-wise aleatoric force uncertainty
by a single scalar even though the force vectors are 3-dimensional. This
simplifying assumption means that we consider the noise scale in the spatial
dimensions to be isotropic, i.e., uniform in all directions. Other options
would be to represent the aleatoric force uncertainty by a common scalar for
all atoms or as atom-wise vectors representing the uncertainty in each
direction. However we found the isotropic approach to be a reasonable
compromise and to work well in practice and did not study the other solutions
further.

###  2.3 Model training procedure

Each network in the ensemble is initialized with random weight parameters
$\theta$ and trained individually on the same training dataset using a loss
function composed of a weighted sum of the energy and force loss terms:

$\mathcal{L}(\theta)=\lambda_{E}\mathcal{L}_{E}(\theta)+\lambda_{F}\mathcal{L}_{F}(\theta)\,,$
(3)

where the weight $\lambda_{F}$ is between 0 and 1 and
$\lambda_{E}=(1-\lambda_{F})$.

ML potentials are usually trained with mean squared error (MSE) loss for both
the energy and forces. Using a negative log likelihood (NLL) loss function
provides a natural way of training mean-variance models that also consider
uncertainty 13. The mean squared error (MSE) loss for the energy is straight
forward. The MSE loss for the forces is evaluated per atom and component-wise
over the spatial dimensions and is then averaged over the number of atoms. The
negative log likelihood (NLL) loss for energy, assuming a normally distributed
error, is given for a single instance by the following expression where
$x=\\{(Z_{i},\vec{r_{i}})\\}$ represents the model input and the observed
values of energy and forces are denoted by $E^{\text{obs}}$ and
$F^{\text{obs}}$, respectively:

$\displaystyle\text{NLL}_{E}(\theta)$ $\displaystyle=-\log
p(E^{\text{obs}}|x,\theta)$ (4)
$\displaystyle=\frac{1}{2}\Bigg{(}\frac{\big{(}E^{\text{obs}}-E(x)\big{)}^{2}}{\sigma_{E}^{2}(x)}+\log\sigma_{E}^{2}(x)+\log
2\pi\Bigg{)}\,.$ (5)

The instance-wise energy losses are then averaged over the number of
instances.

Analogous to the MSE loss for forces, the NLL loss for forces is evaluated per
atom $i$ and component-wise over the spatial dimensions $D$ (recall that the
predicted atom-wise force uncertainty $\sigma_{F_{i}}^{2}$ is a single scalar
applied over all spatial dimensions):

$\displaystyle\text{NLL}_{F_{i}}(\theta)$ $\displaystyle=\sum_{d=1}^{D}-\log
p(F^{\text{obs}}_{i,d}|x,\theta)$ (6)
$\displaystyle=\sum_{d=1}^{D}\frac{1}{2}\Bigg{(}\frac{\big{(}F^{\text{obs}}_{i,d}-F_{i,d}(x)\big{)}^{2}}{\sigma_{F_{i}}^{2}(x)}+\log\sigma_{F_{i}}^{2}(x)+\log
2\pi\Bigg{)}\,.$ (7)

The atom-wise force losses are then averaged over the total number of atoms.
Note that NLL with fixed variance is equivalent to (scaled) MSE and the $\log
2\pi$ terms are constant and can be omitted in training. Here, for models that
are trained with a combination of MSE and NLL loss on either energy or forces,
we scale the NLL loss by the expected uncertainty (determined empirically) to
avoid the NLL loss dominating.

Training directly with NLL loss can be unstable due to interactions between
the mean and variance in the loss function, so we apply a training procedure
similar to previous work 25, where the model is always trained with MSE loss
for an initial warmup period before linearly interpolating to the NLL loss.
Other more sophisticated methods for training with NLL loss exist 27, 28, but
we found this simple approach to be sufficient to achieve training stability
in our experiments.

###  2.4 Ensemble model with epistemic uncertainty

To estimate the epistemic uncertainty, we follow the approach of 17 and make
an ensemble approximation by combining the predictions of $M$ individual
models. Using a Bayesian interpretation of deep ensemble models 7, 8, 9, we
can interpret the model weights $\theta^{(m)}$ of each ensemble member $m$ as
samples from an approximate posterior distribution $q(\theta)\approx
p(\theta|\mathcal{D})$, where $\mathcal{D}$ is the training data. For a
regression model with input $x$ and output $y$ trained on a dataset
$\mathcal{D}$ we have:

$\displaystyle p(y|x,\theta)$ $\displaystyle=\int
p(y|x,\theta)p(\theta|\mathcal{D})d\theta,$ (8)
$\displaystyle\approx\frac{1}{M}\sum_{m=1}^{M}p(y|x,\theta^{(m)}),\quad\theta^{(m)}\sim
p(\theta|\mathcal{D}),$ (9)
$\displaystyle\approx\frac{1}{M}\sum_{m=1}^{M}p(y|x,\theta^{(m)}),\quad\theta^{(m)}\sim
q(\theta).$ (10)

The first approximation is to estimate the integral with $M$ samples from the
distribution $p(\theta|\mathcal{D})$ and the second approximation comes from
approximating the true posterior $p(\theta|\mathcal{D})$ with the distribution
$q(\theta)$. The uncertainty arising from $p(y|x,\theta)$ is the aleatoric
uncertainty, while the epistemic uncertainty is modeled as the uncertainty
arising from the distribution of the model parameters $q(\theta)$.

When applying this interpretation to a ML potential energy model, the
underlying assumption of the model is that energy and force observations are
generated by first sampling $\theta$ and then sampling the normally
distributed noise, i.e., for a single molecular energy and atomic force we
then have:

$\displaystyle\theta$ $\displaystyle\sim q(\theta),$ (11) $\displaystyle
E^{\text{obs}}(x)$
$\displaystyle\sim\mathcal{N}\left(E_{\theta}(x),\sigma_{\theta,E}^{2}(x)\right),$
(12) $\displaystyle\vec{F}^{\text{obs}}_{i}(x)$
$\displaystyle\sim\mathcal{N}\left(-\frac{\partial
E_{\theta}}{\partial\vec{r}_{i}}(x),\mathbf{I}\sigma^{2}_{\theta,F_{i}}(x)\right)\
.$ (13)

We have two levels of stochastic variables, so to calculate the variance we
can use the law of total variance:

$\mathrm{Var}\left[Y\right]=\underbrace{\mathbb{E}\left[\mathrm{Var}\left[Y|X\right]\right]}_{\text{aleatoric}}+\underbrace{\mathrm{Var}\left[\mathbb{E}\left[Y|X\right]\right]}_{\text{epistemic}}\
.$ (14)

Using the law of total variance, we get the following expression for the
observed energy variance:

$\displaystyle\mathrm{Var}\left[E^{\text{obs}}\right]$
$\displaystyle=\mathbb{E}\left[\mathrm{Var}\left[E^{\text{obs}}|\theta\right]\right]+\mathrm{Var}\left[\mathbb{E}\left[E^{\text{obs}}|\theta\right]\right]\
,$ (15)
$\displaystyle=\underbrace{\mathbb{E}_{\theta}\left[\sigma^{2}_{\theta,E}(x)\right]}_{\text{aleatoric}}+\underbrace{\mathrm{Var}_{\theta}\left(E_{\theta}(x)\right)}_{\text{epistemic}}\
.$ (16)

Since the force observation is a vector, we compute the total variance
element-wise:

$\displaystyle\mathrm{Var}\left[F^{\text{obs}}_{i,d},F^{\text{obs}}_{i,d}\right]$
$\displaystyle=\mathbb{E}\left[\mathrm{Var}\left[F^{\text{obs}}_{i,d},F^{\text{obs}}_{i,d}|\theta\right]\right]$
(17)
$\displaystyle\phantom{=}+\mathrm{Var}\left[\mathbb{E}\left[F^{\text{obs}}_{i,d}|\theta\right],\mathbb{E}\left[F^{\text{obs}}_{i,d}|\theta\right]\right]$
$\displaystyle=\underbrace{\mathbb{E}_{\theta}\left[\sigma_{\theta,F_{i}}^{2}(x)\right]}_{\text{aleatoric}}+\underbrace{\mathrm{Var}_{\theta}\left(-\frac{\partial
E_{\theta}(x)}{\partial r_{i,d}},-\frac{\partial E_{\theta}(x)}{\partial
r_{i,d}}\right)}_{\text{epistemic}}\,.$ (18)

Treating the parameters of the ensemble member as samples from an approximate
posterior $q(\theta)$ and using the samples to approximate the expectations,
we get the following expressions for the energy mean and variance:

$\displaystyle E^{(*)}$ $\displaystyle=\frac{1}{M}\sum_{m=1}^{M}E^{(m)}\,,$
(19) $\displaystyle\sigma_{E^{(*)}}^{2}$
$\displaystyle=\underbrace{\frac{1}{M}\sum_{m=1}^{M}\sigma^{2}_{E^{(m)}}}_{\text{aleatoric}}+\underbrace{\frac{1}{M}\sum_{m=1}^{M}\left(E^{(m)}-E^{(*)}\right)^{2}}_{\text{epistemic}}\,.$
(20)

Similarly, the mean and variance of the forces for a single atom $i$ are given
by the following expressions:

$\displaystyle\vec{F}_{i}^{(*)}$
$\displaystyle=\frac{1}{M}\sum_{m=1}^{M}\vec{F}_{i}^{(m)}\,,$ (21)
$\displaystyle\sigma^{2}_{F_{i}^{(*)}}$
$\displaystyle=\underbrace{\frac{1}{M}\sum_{m=1}^{M}\sigma^{2}_{F_{i}^{(m)}}}_{\text{aleatoric}}+\underbrace{\frac{1}{M}\sum_{m=1}^{M}\frac{1}{D}\left\|\vec{F}_{i}^{(m)}-\vec{F}_{i}^{(*)}\right\|^{2}}_{\text{epistemic}}\,,$
(22)

where $D$ denotes the spatial dimensions. With the assumption of isotropic
force variance, we average across the spatial dimension in (22) when
estimating the epistemic force variance of (18) using the sample variance. The
mean energy and forces represent the ensemble prediction and the energy and
force variances represent the ensemble total uncertainties, which can be
decomposed into aleatoric and epistemic components as shown above.

When we want to evaluate the likelihood of an observation, we need access to
the predictive distribution and knowing the mean and variance is not
sufficient. Following 17, we parameterize a normal distribution with the mean
and variance. With the mean and variance specified, the normal distribution is
the maximum entropy probability distribution, i.e., we make the least
assumptions about the data by using a normal distribution following the
maximum entropy principle 29, 30. However, to hold true, this would require
all the variance outputs of the ensemble members to be equal. If the variances
follow an inverse-gamma distribution, the predictive distribution would be a
student-t in the infinite ensemble limit, which is the assumption used in deep
evidential regression 14, 27.

###  2.5 Uncertainty calibration

Several methods exist for evaluating the calibration of regression models. NLL
provides a standard metric for quantifying the overall quality of
probabilistic models by measuring the probability of observing the data given
the predicted distribution. However, NLL depends on both the predicted mean
and uncertainty (see eq. 5 and 7) and it can be useful to evaluate only the
quality of the uncertainty estimates. For example, it is often informative to
visually compare the predicted uncertainties with the empirical errors by
plotting them. Since the total uncertainty of the ensemble model can be
interpreted as a variance of a normal distribution, we expect most of the
errors to lie within 2-3 standard deviations of the predictive distribution.
The variance-based approach also allows us to evaluate standard scores, also
known as z-scores, defined as the empirical error divided by the standard
deviation of the predictive distribution 20:

$z=\frac{y^{\text{obs}}-y(x)}{\sigma(x)}\,.$ (23)

A z-score variance (ZV) close to 1 indicates that the predicted uncertainty on
average corresponds to the variance of the error and thereby is an indication
of good average calibration. The same approach can be applied to subsets of
the data leading to a local z-score variance (LZV) analysis, for example by
evaluating ZV in bins of increasing uncertainty to test the consistency of the
uncertainty estimates. For plotting, we found it useful to report the square
root of the z-variance (RZV). Additionally, we can assess how well the the
uncertainty estimates correspond to the expected error locally by sorting the
predictions in equal size bins by increasing uncertainty and plotting the root
mean variance (RMV) of the uncertainty versus the empirical root mean squared
error (RMSE), also known as an error-calibration plot or reliability diagram.
The error-calibration can be summarized by the expected normalized calibration
error (ENCE) 19, which measures the mean difference between RMV and RMSE
normalised by RMV:

$\text{ENCE}=\frac{1}{K}\sum_{k=1}^{K}\frac{|\text{RMV}_{k}-\text{RMSE}_{k}|}{\text{RMV}_{k}}\,,$
(24)

where $k=1,\dots,K$ iterates the bins. LZV analysis and the reliability
diagram provide two useful ways to evaluate the local consistency of the
uncertainty estimates.

Good average or local calibration is not sufficient to ensure that individual
uncertainty estimates are informative, i.e., if the uncertainty estimates are
homoscedastic, they are not very useful. Thus it is generally desirable for
uncertainty estimates to be as small as possible while also having some
variation, which is also known as sharpness. To measure sharpness, we report
the root mean variance (RMV) of the uncertainty, which should be small and
correspond to the RMSE, and the coefficient of variation (CV), which
quantifies the ratio of the standard deviation of the uncertainties with the
mean uncertainty and thus the overall dispersion, or heteroscedasticity, of
the predicted uncertainty:

$\text{CV}=\frac{\sqrt{N^{-1}\sum_{n=1}^{N}(\sigma(x_{n})-\overline{\sigma})^{2}}}{\overline{\sigma}}\,,$
(25)

where $n=1,\dots,N$ in this case iterates the test dataset, $\sigma(x_{n})$ is
the predicted standard deviation (uncertainty) of instance $n$ and
$\overline{\sigma}=N^{-1}\sum_{n=1}^{N}\sigma(x_{n})$ is the mean predicted
standard deviation. If uncertainties are heteroscedastic while having good
local calibration, it is also an indication of good ranking ability which is
important in certain applications such as active learning 6.

###  2.6 Uncertainty recalibration

The model training procedure described above does not by itself ensure good
uncertainty calibration when the model is applied to unseen data. The
individual models may overfit to the training data and the total ensemble
uncertainties (eq. 20 and 22) are strictly greater than any of the individual
model uncertainties, and not fitted on any data. Therefore, following the
approach of our previous work 25, we recalibrate the ensemble uncertainty
estimates post hoc by using a recalibration function that maps the
uncalibrated predictive distribution to a well calibrated distribution. The
recalibration function is a non-linear uncertainty scaling function based on
isotonic regression fitted to predict empirical squared errors on the
validation set. Specifically, the recalibration function maps the uncalibrated
uncertainty estimates $\sigma^{2}$ to scaled uncertainty estimates
$s^{2}\sigma^{2}$, where $s^{2}$ is the predicted scaling factor. In our
experiments, both energy uncertainties $\sigma^{2}_{E^{(*)}}$ and force
uncertainties $\sigma^{2}_{F^{(*)}}$ are recalibrated in this way.

Because the recalibration function is a scaling function, the recalibration
procedure does not change the mean of the predictive distribution and thus
does not change the prediction. Additionally, the isotonic regression results
in a monotonic increasing scaling function and thus preserves the ordering of
the uncertainty estimates and thereby the ranking. We use the implementation
of isotonic regression available in the scikit-learn Python package 31.

##  3 Experiments and Results

Table 1: Test results of ensemble models ($M=5$) trained on the ANI-1x (A1x)
and Transition1x (T1x) datasets with different combinations of mean squared
error (MSE) and negative log likelihood (NLL) loss functions on the energies
and forces. Energy errors are averaged over molecules, while force errors are
computed component-wise and averaged over the spatial dimensions and atoms.

Data Loss Energy (eV) Forces (eV/Å) $\mathcal{L}_{E}$ $\mathcal{L}_{F}$
MAE$\downarrow$ RMSE$\downarrow$ NLL$\downarrow$ RZV ENCE$\downarrow$ CV RMV
MAE$\downarrow$ RMSE$\downarrow$ NLL$\downarrow$ RZV ENCE$\downarrow$ CV RMV
A1x MSE MSE 0.0123 0.0278 -2.81 0.97 0.0773 1.27 0.0283 0.0180 0.0362 -2.56
0.98 0.0243 1.24 0.0386 NLL MSE 0.0118 0.0256 -3.04 1.00 0.0411 1.00 0.0209
0.0179 0.0364 -2.56 0.98 0.0229 1.22 0.0381 MSE NLL 0.0117 0.0305 -2.91 0.97
0.0928 1.53 0.0305 0.0175 0.0399 -2.77 1.00 0.0099 1.51 0.0410 NLL NLL 0.0105
0.0296 -3.26 1.02 0.0600 1.51 0.0237 0.0171 0.0402 -2.79 1.00 0.0093 1.56
0.0409 T1x MSE MSE 0.0344 0.0612 -1.79 0.89 0.1144 0.82 0.0682 0.0370 0.0743
-1.94 0.99 0.0292 1.21 0.0804 NLL MSE 0.0318 0.0578 -1.94 0.92 0.1383 0.86
0.0655 0.0366 0.0744 -1.97 0.98 0.0293 1.27 0.0831 MSE NLL 0.0332 0.0600 -1.84
0.95 0.1108 0.83 0.0628 0.0369 0.0745 -1.99 0.94 0.0615 1.18 0.0817 NLL NLL
0.0303 0.0574 -2.09 0.98 0.0906 0.96 0.0562 0.0359 0.0751 -2.05 0.94 0.0645
1.15 0.0773

###  3.1 Datasets

The proposed method was evaluated on two publicly available datasets designed
specifically for the development and evaluation of ML potentials, ANI-1x 1 and
Transitions1x 2. The datasets include out-of-equilibrium and near-transition-
state structures, respectively, and represent varied energy and force
distributions. The ANI-1x dataset consists of Density Functional Theory (DFT)
calculations for approximately 5 million diverse molecular conformations with
an average of 8 heavy atoms (C, N, O) and an average of 15 total atoms
(including H) along with multiple properties including total energy and
interatomic forces computed at the B97x/6-31G(d) level of theory. The dataset
was generated by perturbing equilibrium configurations using an active
learning procedure to ensure conformational diversity with the aim of
developing an accurate and transferable ML potential. The Transition1x dataset
contains DFT calculations of energy and forces, for 9.6 million molecular
conformations with up to 7 heavy atoms (C, N, O) and an average of 14 total
atoms (including H), likewise computed at the B97x/6-31G(d) level of theory.
Here, the structures were sampled on and around full reaction pathways, thus
including conformations far from equilibrium and near transition states. The
dataset was generated by running a nudged elastic band (NEB) 32 algorithm with
DFT on a set of approximately 10 thousand organic reactions with up to 6 bond
changes while saving intermediate calculations with the aim of improving ML
potentials around transition states. Transition1x is more varied in terms of
interatomic distances between pairs of heavy atoms than ANI-1x, but less
varied in terms of the distribution of forces, since forces are generally
minimised along reaction pathways 2, 32.

###  3.2 Model hyperparameters

The same general model configuration was used in all experiments. Each
individual graph neural network model was configured with 3 interaction
layers, a hidden node state size of 256 and a 2-layer atom-wise readout
network with 3 outputs representing $E_{i}$, $\sigma^{2}_{E_{i}}$ and
$\sigma^{2}_{F_{i}}$. The input molecular graphs were generated with an edge
cutoff radius of 5.0 Å. Models were trained using the Adam optimizer with an
initial learning rate of $10^{-4}$, an exponential decay learning rate
scheduler, a batch size of 64 molecular graphs, force loss weight
$\lambda_{F}=0.5$ and an early stopping criterion on the validation loss to
prevent overfitting. In each experiment, models were trained individually with
the same hyperparameters on the same training data, but with different random
parameter initialisation and random shuffling of the training data to induce
model diversity in the ensembles. For each ensemble model, after the training
was completed, a recalibration function was fitted using predictions on the
respective validation dataset following the procedure described in Section 2.6
and applied to the predictions on the test data. For ensemble models trained
with MSE loss, only the epistemic uncertainty is considered.

###  3.3 ANI-1x results

Ensembles of graph neural network models were trained on ANI-1x using the data
splits from Schreiner _et al._ 2, where the validation and test datasets
consist of approximately 5% of the data each and the training set consists of
the remaining 90% of the data. The splits are stratified by chemical formula
to ensure different splits do not contain configurations made up of exactly
the same atoms and selected such that all splits include all species of heavy
atoms. Individual models were trained for up to 10 million gradient steps
(approximately 144 epochs) with an initial warmup period of 2 million steps
where the model was trained only with MSE loss followed by an interpolation
period of 1 million steps, where the loss was interpolated linearly to NLL
loss (only NLL models). The ensemble predictions were then recalibrated post
hoc using a recalibration function fitted using the validation dataset. The
trade-off between validation performance and ensemble size $M$ using models
trained with NLL loss on both energy and forces is illustrated in Figure 1.
Using a larger ensemble size results in lower error, as expected, but comes at
the cost of additional computations. We observe that a reasonably low error is
obtained at $M=5$ and only small improvements are gained beyond that, which is
similar to what we found in previous work 25.

Figure 1: Trade-off between performance and ensemble size on the ANI-1x
validation dataset using ensembles of models trained with NLL loss on both
energy and forces.

Test results for $M=5$ ensembles trained with different combinations of MSE
and NLL loss on energy and forces are presented in the first four rows of
Table 1. The ensemble trained with MSE loss on both energy and forces is a
standard ensemble model with post hoc recalibration. The other three ensembles
show the effect of training with NLL loss on either energy or forces or both.
All four ensembles achieved a low error on energy and forces in terms of MAE
and RMSE compared to the results reported by Schreiner _et al._ 2 using a
PaiNN model similar to the base model of our ensembles (MAE=0.023 on energy
and MAE=0.039 on forces). Importantly, training with NLL loss on either energy
or forces or both did not result in worse performance in terms of prediction
error.

All four ensemble models achieved good average calibration on ANI-1x in terms
of NLL and ZV after recalibration, but ensembles trained with NLL loss
performed slightly better which is observed both on energy and forces and the
ensemble trained with NLL on both energy and forces performs best overall.
Additionally, all ensembles scored a high CV indicating uncertainty estimates
are heteroscedastic and thus informative. Calibration plots for the ensemble
trained on ANI-1x with NLL loss on energy and forces are presented in Figure
2. The uncertainty vs. error plots show that in general large errors are
associated with large uncertainties and most errors are within 2-3 standard
deviations of uncertainty as desired. For the energy, the model appears to be
biased for some examples with large errors, but these are relatively few and
are correctly identified as problematic by high uncertainty. For the forces,
the distribution of errors looks more symmetrical around zero. This is also
clearly shown by the local z-score analysis plots where for the energy, the
variance of the z-scores is slightly off for very low and very high
uncertainties, although still centered around 1, whereas for the forces the
variance of the z-scores is close to 1 for all uncertainties which indicates
high consistency. Finally, the reliability diagrams show the relation between
predicted uncertainty and expected error for the energy and forces,
respectively. Both plots show a clear correlation between the uncertainty and
the expected error as the curves lie close to the identity line. Again, the
model very slightly underestimates the expected error of the energy at low and
high uncertainties, and the curve for the forces is near perfect. The
reliability diagrams are summarised by ENCE scores in Table 1.

Figure 2: Calibration results on the ANI-1x dataset of energy (top row) and
forces (bottom row) for an ensemble of $M=5$ models trained with NLL loss on
both energy and forces. Transparent curves show results before applying
recalibration.

###  3.4 Transition1x results

Analogous to the first experiment, ensembles of graph neural network models
were trained on Transition1x using data splits from Schreiner _et al._ 2 based
on the same splitting criteria as ANI-1x described above. Models were trained
for up to 3 million gradient steps (approximately 21 epochs) with an initial
warmup period of 2 million steps followed by an interpolation period of 2
million steps (training was stopped before finishing the full interpolation
period). When training for longer on this dataset, we observed severe
overfitting. We believe this is because the data was generated from a
relatively small set of chemical reactions making the models prone to overfit
the many similar configurations associated with the same reactions in the
training data. The ensemble predictions were recalibrated post hoc using a
recalibration function fitted using the validation dataset. Varying the
ensemble size yielded similar results to the first experiment (Figure 1) and
$M=5$ was selected as a good compromise between performance and computational
cost.

Test results for $M=5$ ensembles are presented in the last four rows of Table
1. As in the first experiment, all ensembles achieved a low error on energy
and forces in terms of MAE and RMSE compared to the results reported in
Schreiner _et al._ 2 (MAE=0.048 on energy and MAE=0.058 on forces) and
training with NLL loss did not decrease performance in terms of prediction
error in any case. All four ensembles achieved acceptable average calibration
in terms of NLL and ZV on the Transition1x test data. Surprisingly, ensembles
trained with MSE loss were as well or better calibrated than ensembles trained
with NLL loss on this dataset. All ensembles score similar high CV indicating
uncertainty estimates are heteroscedastic and thus informative. Calibration
plots for an ensemble trained on Transition1x with NLL loss on energy and
forces are presented in Figure 3. The uncertainty vs. error plots show that in
general large errors are associated with large uncertainties as desired. For
both energy and forces some errors extend beyond 2-3 standard deviations of
uncertainty indicating the error distributions have wider tails and may not be
Gaussian in this case. Similar to the ANI-1x experiment, it looks like the
model is biased for some instances with large energy errors, but these cases
are correctly identified as problematic by high uncertainty. For the forces,
the error distribution appears more symmetrical around zero but with wide
tails. The local z-score analysis plot for the energy indicate some
inconsistencies in the energy uncertainties. Plotting the root variance of the
z-scores as a function of the observed molecular energies (Figure 4) shows
that the uncertainty is underestimated for low energies and overestimated for
high energies on average. Taking a closer look at the energy distribution
reveals significant differences between the validation and test set which
could be the reason for this problem. The variance of the local z-scores for
the forces are more consistent, but values below one indicate that the model
generally overestimates the uncertainty on the forces. The reliability diagram
for the energy also shows signs of some inconsistencies, as the curve does not
form a straight line along the diagonal, but overall the uncertainties are
correlated with the expected error. The corresponding reliability diagram for
the forces shows a more consistent result, only with a tiny overestimation of
the force uncertainty. The reliability diagrams are summarised by ENCE scores
in Table 1.

Figure 3: Calibration results on the Transition1x dataset of energy (top row)
and forces (bottom row) for an ensemble of $M=5$ models trained with NLL loss
on both energy and forces. Transparent curves show results before applying
recalibration.

Figure 4: Local root z-score variance (RZV) conditioned on observed energy on
the Transition1x test dataset (top). Energy distribution in the Transition1x
data split (bottom).

##  4 Discussion

The proposed method achieved good predictive performance as well as calibrated
and consistent uncertainty estimates in experiments on two challenging,
publicly available molecular datasets. A major advantage of the approach is
that it considers both epistemic and aleatoric uncertainty through an ensemble
approximation of mean-variance models. We believe that considering both
aleatoric and epistemic uncertainty is critical to ensure good calibration in
and out of the training data distribution. Often the training procedures of
uncertainty aware models do not inherently ensure good calibration on unseen
data. For example, ensemble members trained on the same data will often fit
the same mean prediction without accounting for noise in the data and mean-
variance methods will estimate the noise on the training data but do not
guarantee good extrapolation of the uncertainty estimates to previously unseen
data. Therefore, the post hoc recalibration procedure is key to achieve good
calibration on unseen data in our experiments, but is not commonly applied by
other UQ methods in the literature.

The computational overhead of training and evaluating ensemble models is
sometimes pointed out as a major disadvantage of using ensembles. However, it
is important to note that most of this computation can be performed in
parallel and thus only leads to a small overhead of computing the ensemble
approximation and recalibration in real time. Some works have proposed methods
for speeding up the training of ensembles, such as snapshot ensembles 33, 34,
which could also be applied in this case. Another widely accepted advantage of
ensembles is that they often improve prediction accuracy (see Figure 1 as an
example), which can be considered a positive side effect of the proposed
method. Here, we have used ensembles of size 5, but larger ensembles can be
expected to further improve performance (up to a limit) at the cost of more
computation. The approach could also potentially benefit from other recent
extensions to ensembles such as using randomized priors 35 to improve the
quality of especially the epistemic uncertainty estimates.

Evaluation of uncertainty calibration for regression models is an active area
of research 18, 19, 21, 20. Standard procedures for assessing the quality of
uncertainty estimates are necessary within the field to establish confidence
in uncertainty quantification methods and establish fair comparison. We
recommend recent work by Pernot 36 which provides a good overview of
calibration assessment methods and a detailed approach for evaluating
uncertainty. Our experiments show that the uncertainty estimates obtained with
the proposed method are largely consistent with the expected error for varying
size of the uncertainty (Figures 2 and 3). However, we observed indications
that uncertainties are not equally well calibrated along different molecular
energies (Figure 4). The current recalibration method only considers the
magnitude of the predicted uncertainty. It would be an interesting direction
for future work to design a recalibration function that can account for
additional input features such as the (predicted) energy, while remaining a
monotonic increasing scaling function, with the aim of achieving equally good
calibration throughout the input space. Applying the calibration evaluation
framework proposed by Pernot 20 could help provide additional insights into
the consistency and adaptive of predictive uncertainty.

##  5 Conclusion

In this work, we have presented a complete framework for training neural
network potentials with calibrated uncertainty estimates on both energy and
forces. The proposed method was demonstrated and evaluated on two challenging,
publicly available molecular datasets containing diverse conformations far
from equilibrium. In all cases, the proposed method achieved low prediction
error and good uncertainty calibration. On the ANI-1x dataset training with
NLL loss improved the calibration over training with standard MSE loss. On the
Transition1x dataset, the same improvement was not observed and good
calibration was achieved by training with standard MSE loss and applying post
hoc nonlinear recalibration. This could be because the validation and test
data are more out of distribution in this case. The proposed method does not
depend on the particular architecture of the neural network model, and can
thus easily be adapted to new models in the future. We hope that this work
will contribute to better calibrated ML potentials and enable more robust and
reliable applications.

## References

  * Smith _et al._ 2018 J. S. Smith, B. Nebgen, N. Lubbers, O. Isayev and A. E. Roitberg, _The Journal of Chemical Physics_ , 2018, 148, 241733.
  * Schreiner _et al._ 2022 M. Schreiner, A. Bhowmik, T. Vegge, J. Busk and O. Winther, _Scientific Data_ , 2022, 9, 779.
  * Dral 2020 P. O. Dral, _J. Phys. Chem. Lett._ , 2020, 11, 2336–2347.
  * von Lilienfeld and Burke 2020 O. A. von Lilienfeld and K. Burke, _Nature Communications_ , 2020, 11, 4895.
  * Peterson _et al._ 2017 A. A. Peterson, R. Christensen and A. Khorshidi, _Phys. Chem. Chem. Phys._ , 2017, 19, 10978–10985.
  * Schran _et al._ 2020 C. Schran, K. Brezina and O. Marsalek, _The Journal of Chemical Physics_ , 2020, 153, 104105.
  * Wilson and Izmailov 2020 A. G. Wilson and P. Izmailov, Proceedings of the 34th International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2020, pp. 4697–4708.
  * Hoffmann and Elster 2021 L. Hoffmann and C. Elster, _Deep Ensembles from a Bayesian Perspective_ , 2021.
  * Gustafsson _et al._ 2020 F. K. Gustafsson, M. Danelljan and T. B. Schon, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Los Alamitos, CA, USA, 2020, pp. 1289–1298.
  * Neal 1992 R. Neal, Advances in Neural Information Processing Systems, 1992.
  * Blundell _et al._ 2015 C. Blundell, J. Cornebise, K. Kavukcuoglu and D. Wierstra, Proceedings of the 32nd International Conference on International Conference on Machine Learning \- Volume 37, 2015, p. 1613–1622.
  * Gal and Ghahramani 2016 Y. Gal and Z. Ghahramani, Proceedings of The 33rd International Conference on Machine Learning, New York, New York, USA, 2016, pp. 1050–1059.
  * Nix and Weigend 1994 D. A. Nix and A. S. Weigend, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94), 1994, pp. 55–60 vol.1.
  * Amini _et al._ 2020 A. Amini, W. Schwarting, A. Soleimany and D. Rus, Advances in Neural Information Processing Systems, 2020, pp. 14927–14937.
  * Soleimany _et al._ 2021 A. P. Soleimany, A. Amini, S. Goldman, D. Rus, S. N. Bhatia and C. W. Coley, _ACS Central Science_ , 2021, 7, 1356–1367.
  * Hu _et al._ 2022 Y. Hu, J. Musielewicz, Z. W. Ulissi and A. J. Medford, _Machine Learning: Science and Technology_ , 2022, 3, 045028.
  * Lakshminarayanan _et al._ 2017 B. Lakshminarayanan, A. Pritzel and C. Blundell, _Advances in neural information processing systems_ , 2017, 30, 6402–6413.
  * Kuleshov _et al._ 2018 V. Kuleshov, N. Fenner and S. Ermon, Proceedings of the 35th International Conference on Machine Learning, 2018, pp. 2796–2804.
  * Levi _et al._ 2022 D. Levi, L. Gispan, N. Giladi and E. Fetaya, _Sensors_ , 2022, 22, .
  * Pernot 2022 P. Pernot, _The Journal of Chemical Physics_ , 2022, 157, 144103\.
  * Tran _et al._ 2020 K. Tran, W. Neiswanger, J. Yoon, Q. Zhang, E. Xing and Z. W. Ulissi, _Machine Learning: Science and Technology_ , 2020, 1, 025006\.
  * Hirschfeld _et al._ 2020 L. Hirschfeld, K. Swanson, K. Yang, R. Barzilay and C. W. Coley, _Journal of Chemical Information and Modeling_ , 2020, 60, 3770–3780.
  * Scalia _et al._ 2020 G. Scalia, C. A. Grambow, B. Pernici, Y.-P. Li and W. H. Green, _Journal of Chemical Information and Modeling_ , 2020, 60, 2697–2717.
  * Nigam _et al._ 2021 A. Nigam, R. Pollice, M. F. D. Hurley, R. J. Hickman, M. Aldeghi, N. Yoshikawa, S. Chithrananda, V. A. Voelz and A. Aspuru-Guzik, _CoRR_ , 2021, abs/2102.11439, .
  * Busk _et al._ 2021 J. Busk, P. B. Jørgensen, A. Bhowmik, M. N. Schmidt, O. Winther and T. Vegge, _Machine Learning: Science and Technology_ , 2021, 3, 015012\.
  * Schütt _et al._ 2021 K. Schütt, O. Unke and M. Gastegger, International Conference on Machine Learning, 2021, pp. 9377–9388.
  * Skafte _et al._ 2019 N. Skafte, M. Jørgensen and S. Hauberg, Advances in Neural Information Processing Systems, 2019.
  * Seitzer _et al._ 2022 M. Seitzer, A. Tavakoli, D. Antic and G. Martius, International Conference on Learning Representations, 2022.
  * Jaynes 1957 E. T. Jaynes, _Phys. Rev._ , 1957, 106, 620–630.
  * Blower 2013 D. D. J. Blower, _Information Processing: The Maximum Entropy Principle_ , CreateSpace Independent Publishing Platform, 2013.
  * Pedregosa _et al._ 2011 F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot and E. Duchesnay, _Journal of Machine Learning Research_ , 2011, 12, 2825–2830.
  * Sheppard _et al._ 2008 D. Sheppard, R. Terrell and G. Henkelman, _The Journal of Chemical Physics_ , 2008, 128, .
  * Huang _et al._ 2017 G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft and K. Q. Weinberger, _Snapshot Ensembles: Train 1, get M for free_ , 2017.
  * Wang _et al._ 2021 F. Wang, G. Wei, Q. Liu, J. Ou, H. Lv _et al._ , _Advances in Neural Information Processing Systems_ , 2021, 34, 19719–19729.
  * Osband _et al._ 2018 I. Osband, J. Aslanides and A. Cassirer, Advances in Neural Information Processing Systems, 2018, pp. 8617–8629.
  * Pernot 2023 P. Pernot, _Validation of uncertainty quantification metrics: a primer_ , 2023\.


††thanks: These two authors contributed equally††thanks: These two authors
contributed equally

# Measurement of Stimulated Raman Side-Scattering Predominance and Energetic
Importance in the Compression Stage of the Double-Cone Ignition Approach to
Inertial Confinement Fusion

K. Glize  X. Zhao  Key Laboratory for Laser Plasmas (MoE) and School of
Physics and Astronomy, Shanghai Jiao Tong University, Shanghai 200240, China
Collaborative Innovation Center of IFSA (CICIFSA), Shanghai Jiao Tong
University, Shanghai 200240, China  Y. H. Zhang  Beijing National Laboratory
for Condensed Matter Physics, Institute of Physics, Chinese Academy of
Sciences, Beijing 100190, China  Collaborative Innovation Center of IFSA
(CICIFSA), Shanghai Jiao Tong University, Shanghai 200240, China  C. W. Lian
Key Laboratory of Basic Plasma Physics and Department of Engineering and
Applied Physics, University of Science and Technology of China, Hefei 230026,
China  S. Tan  Key Laboratory for Micro-/Nano-Optoelectronic Devices of
Ministry of Education, School of Physics and Electronics, Hunan University,
Changsha, 410082, China  F. Y. Wu  Key Laboratory for Laser Plasmas (MoE) and
School of Physics and Astronomy, Shanghai Jiao Tong University, Shanghai
200240, China  Collaborative Innovation Center of IFSA (CICIFSA), Shanghai
Jiao Tong University, Shanghai 200240, China  C. Z. Xiao  Key Laboratory for
Micro-/Nano-Optoelectronic Devices of Ministry of Education, School of Physics
and Electronics, Hunan University, Changsha, 410082, China  R. Yan  Key
Laboratory of Basic Plasma Physics and Department of Engineering and Applied
Physics, University of Science and Technology of China, Hefei 230026, China
Z. Zhang  Beijing National Laboratory for Condensed Matter Physics, Institute
of Physics, Chinese Academy of Sciences, Beijing 100190, China  Collaborative
Innovation Center of IFSA (CICIFSA), Shanghai Jiao Tong University, Shanghai
200240, China  Songshan Lake Materials Laboratory, Dongguan, Guangdong 523808,
China  X. H. Yuan  Authors to whom correspondence should be addressed
<EMAIL_ADDRESS>and<EMAIL_ADDRESS>Key Laboratory for Laser
Plasmas (MoE) and School of Physics and Astronomy, Shanghai Jiao Tong
University, Shanghai 200240, China  Collaborative Innovation Center of IFSA
(CICIFSA), Shanghai Jiao Tong University, Shanghai 200240, China  J. Zhang
Authors to whom correspondence should be addressed<EMAIL_ADDRESS>and<EMAIL_ADDRESS>Key Laboratory for Laser Plasmas (MoE) and School of
Physics and Astronomy, Shanghai Jiao Tong University, Shanghai 200240, China
Collaborative Innovation Center of IFSA (CICIFSA), Shanghai Jiao Tong
University, Shanghai 200240, China  Beijing National Laboratory for Condensed
Matter Physics, Institute of Physics, Chinese Academy of Sciences, Beijing
100190, China

###### Abstract

Due to its particular geometry, stimulated Raman side-scattering (SRSS) drives
scattered light emission at non-conventional directions, leading to scarce and
complex experimental observations. Experimental campaigns at the SG-II UP
facility have measured the scattered light driven by SRSS over a wide range of
angles, showing an emission at large polar angles, sensitive to the plasma
profile and laser polarization. Furthermore, direct comparison with back-
scattering measurement has evidenced SRSS as the dominant Raman scattering
process in the compression stage, leading to the scattering loss of about 5%
of the total laser energy. The predominance of SRSS was confirmed by 2D
particle-in-cell simulations, and its angular spread has been corroborated by
ray-tracing simulations. The main implication is that a complete
characterization of the SRS instability and an accurate measurement of the
energy losses require the collection of the scattered light in a broad range
of directions. Otherwise, spatially limited measurement could lead to an
underestimation of the energetic importance of stimulated Raman scattering.

In inertial confinement fusion (ICF) experiments Nuckolls  _et al._ (1972),
one of the main obstacles that prevents the implosion from reaching the
required conditions to obtain ignition of the fuel are the laser-plasma
instabilities (LPIs) Kruer (1991), consisting of non-linear couplings of the
laser driver beam with plasma modes. Stimulated Raman scattering (SRS) is a
three-wave coupling resonantly driving an electron plasma wave (EPW) Comisar
(1966). This process leads to the scattering of a part of the incident laser
reducing the energy coupling, and generation of a hot electron population that
can preheat the fuel core. This instability is of primary concern in most of
the ICF schemes, such as Indirect-Drive Kirkwood  _et al._ (2013); Montgomery
(2016); Hall  _et al._ (2017), Direct-Drive Rosenberg  _et al._ (2018, 2020)
or Shock Ignition Scott  _et al._ (2021); Cristoforetti  _et al._ (2019,
2021); Baton  _et al._ (2020). Stimulated Raman side-scattering (SRSS) is a
particular SRS geometry in which the scattered light is emitted perpendicular
to the density gradient, enabling an absolute growth (exponential growth in
time at a localised spatial position) at density lower than $n_{c}/4$, where
$n_{c}$ is the critical density. Despite extensive theoretical investigations
in the late 70s Liu  _et al._ (1974); Mostrom and Kaufman (1979); Afeyan and
Williams (1985), most of the interest has been focused on stimulated Raman
back-scattering (SRBS) Montgomery (2016), due to the experimental complexity
to measure SRSS and the largest SRS growth rate for the backward geometry.
Recently, there has been a renewed interest due to observations of SRSS on
several planar experiments Rosenberg  _et al._ (2018, 2020); Depierreux  _et
al._ (2016, 2019); Cristoforetti  _et al._ (2019), either from single beam
interaction, or by multiple beams Short (2020). Multi-beam processes happen
when the laser beams are sharing a common symmetry axis enabling to drive a
shared daughter wave, being either an EPW Michel  _et al._ (2015) or a
scattered wave Depierreux  _et al._ (2016, 2019). These experimental
observations have led to the development of a more complete analytical
description of the SRSS, accounting for the convective nature (finite spatial
amplification while propagating through the resonant region) of the
instability near the turning point, in order to explain the SRSS growth in
region below the absolute threshold Michel  _et al._ (2019). It highlighted
that ICF experiments are prone to being SRSS unstable as the instability can
extend to lower densities in the convective regime due to the large dimensions
of the interaction, namely long density scale-length, large laser focal spot
and high temperature. Therefore a complete understanding of this detrimental
process is imperative as leading to additional laser energy coupling loss and
hot electron generation. However, experimental observations have been
restricted to a limited number of directions since ICF laser facilities are
usually not designed to measure SRS in directions other than back-scattering.
Thus, comprehensive measurements of this mechanism and related losses are
still not available. In order to improve the overall characterisation of SRS
at non-typical angles, crucial new diagnostics are currently being implemented
in order to provide further observations at additional angles of observations
Rosenberg  _et al._ (2021).

In this Letter, we present, up to our knowledge, the first highly-resolved 2D
angular measurement of SRS in ICF experiment. These measurements have been
obtained in the context of the Double-Cone Ignition (DCI) scheme Zhang  _et
al._ (2020), a newly proposed approach to Direct-Drive (DD) ignition McCrory
_et al._ (2008), in which the compression and ignition stages are separated.
During the preliminary experimental campaigns, focused on the compression
stage at low laser intensity, a new diagnostic has been designed and
implemented to provide an angularly resolved measurement of the SRS emission
spectrum: Angular-Resolved Scattered-light Diagnostic Station (ARSDS) Zhao
_et al._ (2022). It enabled to observe broadband SRS light emitted over a wide
spatial area. Direct comparison with the back-scattered light collected in the
aperture of one of the driver beam evidenced SRSS as the dominant SRS process.
2D PIC simulations confirmed that in this regime, the interaction was below
the threshold for the typical SRBS to grow and only SRSS was responsible for
the SRS emission. Due to its broad spatial emission, SRSS was identified to be
responsible for the scattering of up to $5\pm 2\%$ of the total laser energy
and requires to be measured over the whole interaction volume.

Figure 1:  (a) Schematic of the irradiation geometry on the conical target.
(b) Pulse shape of the overlapped power delivered on target, both design
(blue) and experimental (dashed red). (c) Spherical coordinate map depicting
the positions of the laser beams (red squares), the ARSDS fibers (black dots)
and the additional 50 fibers (blue diamonds).

The experiment was performed at the SG-II UP laser facility Xu  _et al._
(2008) using the setup displayed on Fig. 1(a). For clarity, we define here
that beam #7 azimuthal position is the origin $\varphi=0\degree$ of the
azimuth axis and that the north pole is the origin $\theta=0\degree$ of the
polar axis. A cone of four beams were used to irradiate a
45$\leavevmode\nobreak\ \upmu$m-thick CH spherical cap target, with an inner
radius of 450$\leavevmode\nobreak\ \upmu$m. The CH cap is contained within a
20$\leavevmode\nobreak\ \upmu$m-thick, open-ended Au cone. The cone of four
beams, #1, #3, #5 and #7, were incident on the CH target at a polar angle of
$\theta=50\degree$, uniformly distributed in the azimuthal direction. The
beams are primarily p-polarised plus an angle of $7\degree$ for beams #3, #5
and $23\degree$ for beams #1, #7. Each beam delivers 1.5 kJ at 351 nm focused
by a $f/7.1$ wedged lens on target within a 525$\leavevmode\nobreak\ \upmu$m
($1/e^{2}$), CPP smoothed focal spot, with a pulse shape presented on Fig.
1(b), reaching a peak overlapped intensity of $\approx 1.8\times
10^{15}\leavevmode\nobreak\ $W.cm-2 in vacuum. At peak power, plasma
parameters were estimated using the hydro-radiative code MULTI2D Ramis  _et
al._ (2009), predicting a density scale-length of $L_{n_{c}/4}\approx
200\leavevmode\nobreak\ \upmu$m at $n_{c}/4$ up to $L_{n_{c}/10}\approx
250\leavevmode\nobreak\ \upmu$m in the coronal plasma, with an uniform
electron temperature from $T_{e}\approx 2.0$ to $2.5\leavevmode\nobreak\ $keV.
The electron temperature near $n_{c}/4$ was confirmed to be $\leq
2.2\leavevmode\nobreak\ $keV from the red shifted-spectral feature related to
Two-Plasmon Decay (TPD) instability Seka  _et al._ (1985), as shown by the
inset in Fig. 2(a). A Full-Aperture Back-scattering Station (FABS) was
installed on #7 in order to collect the light scattered in the backward
direction $[\theta,\varphi]=[50\degree,0\degree]$. ARSDS was used to collect
the SRS scattered light in both azimuthal and polar axes, at the coordinates
depicted by the black dots in Fig. 1(c), in order to obtain temporally
integrated spectra resolved in angle.

Figure 2:  Typical ARSDS spectrum of the SRS emission against (a) the polar
angle collected at an azimuthal angle of $33.7\degree$; (b) the azimuthal
angle collected at polar angles displayed in Fig. 1(c). The inset in (a)
presents the contrast enhanced TPD features. (c) (resp. (d)) Red plain line is
the scattered energy against the exit angle, integrated over the spectrum (a)
(resp. (b)) and normalized to the collection solid angle. Back-scattered
energy measured by FABS is displayed as a black dot on (c). The discontinuity
in (d) is due to the different polar coordinate, as shown in Fig. 1(c). On (c)
and (d), blue dotted, green dashed and black dash-dot lines present the
scattered light signal in arbitrary units from the ray-tracing simulations
with respectively ideal spherical compression, flattened compression and
rotated polarization.

A typical result is presented on Fig. 2, displaying the scattered light
spectrum angularly resolved in the (a) polar and (b) azimuthal directions. The
overall signal is dominated by light with a spectrum ranging from 540 nm up to
640 nm emitted within a wide area ranging from $30\degree$ up to $90\degree$
polar angles, over the whole azimuth, corresponding to a density range of
$\approx[0.1-0.2]n_{c}$. The plain red curve on Fig. 2(c) (resp. Fig. 2(d))
presents the signal from Fig. 2(a) (resp. Fig. 2(b)) integrated over the
spectrum, calibrated in energy and normalised to the solid angle. It shows
that most of the scattered light is emitted at large polar angles, peaked
around $\theta=70\degree$. Such signal corresponds to an SRS emission at
shorter wavelength and larger angle than previously reported, where usually
the detection of SRS light is limited to angles $\leq 50\degree$. Furthermore,
Fig. 2(d) shows that the amount of scattered energy is sensitive to the
azimuthal position. The maximum emission is not contained in the azimuthal
plane of the laser beam but is offset by an angle $\Delta\varphi\approx
10\degree$.

Considering that the broadband signal is emitted at polar angles larger than
the incident driver beams, over a large azimuthal section, it evidences that
side-scattering is responsible for such emission. The SRSS scattered light is
emitted orthogonal to the density gradient and experiences refraction on its
way out of the plasma Afeyan and Williams (1985); Michel  _et al._ (2019).
This results in a correlation between the scattered light wavelength and the
polar exit angle, as observed on Fig. 2(a). Moreover, SRSS scattered light is
also emitted perpendicular to the laser polarization plane, as reported in
theoretical Kruer (1991); Afeyan and Williams (1985); Menyuk  _et al._ (1985)
and numerical Xiao  _et al._ (2016, 2018); Gu  _et al._ (2021) studies.
Considering the 2D spherical nature of our target, the inherent sensitivity of
SRSS to the plasma profile and laser polarization, numerical simulations are
required to confirm the observed angular spread of the scattered light. A new
3D ray-tracing code PHANTAM Ji  _et al._ (2022), based on the method published
by Kaiser Kaiser (2000), has been developed to simulate the propagation of the
SRSS light, also accounting for collisional absorption. The simulation box is
0.4cm x 0.4cm x 0.3cm with a 200x200x300 mesh, containing a plasma obtained
from the hydrodynamic simulations. SRSS light is generated when incident rays
propagate into the the electron density range $[0.12,0.20]n_{c}$, at the
associated wavelength, consistent with the experimental data, without gain or
threshold consideration. SRSS rays are initialized perpendicular to both the
local density gradient and the local polarization of the incident ray with a
9-degree random spreading angle. Upon exiting the plasma, SRSS rays are
collected at the same positions than ARSDS detectors. As displayed by the blue
dotted curves in Fig. 2(c) and (d), such simulations are confirming an
emission at large polar angles, maximized near the beam azimuth, despite
several discrepancies. These are likely the result of difference in plasma
profiles between the ideal spherical compression from the simulation Yang  _et
al._ (2022) and the actual experiment. This assumption can be confirmed by
reducing the discrepancies when considering a plasma profile flattened along
the target normal, as depicted by the green dashed curves in Fig. 2(c) and
(d). The importance of the laser polarization is highlighted by the black
dash-dot curves, showing a completely different scattering profile when the
polarization of each beam is rotated by 90$\degree$. These simulations show
that the scattering is a complex combination of plasma profile, polarization
and single-beam contribution from beam #7 and the neighbour beam #5. Further
investigations, beyond the scope of this paper, are necessary to improve
numerical agreement with the experiment and will be the subject of future
work.

From these spectra, it appears that the scattering is driven by single beam
SRSS, as multi-beam process would drive scattered light either in the bisector
plan at $\varphi\approx$ 45$\degree$ for a shared scattered electromagnetic
wave Depierreux  _et al._ (2016, 2019), or constrained to density $n_{e}\leq
0.12n_{c}$ for a shared EPW due to the large angle between two neighbouring
beams Michel  _et al._ (2015). This was further confirmed by experimentally
measuring a two-orders-of-magnitude decrease of the signal in the polar
direction when #7 is switched off, and retrieving the overall emission profile
in the ray-tracing simulations from independent beam contribution. Thus,
single-beam intensity is considered in the following discussion.

In ICF experiments, SRSS can experience both absolute or convective growth,
depending on the interaction conditions. Figure 3 presents SRSS convective
gain Michel  _et al._ (2019) and absolute threshold Afeyan and Williams (1985)
against the scattered light wavelength and incident laser intensity, for our
conditions considering oblique incidence and damping. It appears that the
absolute threshold is overcome for most of the density range. The lower
density part can be interpreted considering the convective regime, having a
moderate gain in our intensity range, where the white curve highlights the
iso-contour corresponding to a gain of 1. The absence of signal above 640 nm
in our data is likely due to the significant re-absorption at the associated
high density, as previously reported Depierreux  _et al._ (2016); Michel  _et
al._ (2019). At low density, Michel Michel  _et al._ (2019) reported that the
finite size of the beam is limiting the convective growth of SRSS, which
requires a large transverse amplification length. Considering a
525$\leavevmode\nobreak\ \upmu$m focal spot, this corresponds to a cutoff
around 530 nm in our conditions, consistent with our experimental
observations.

Figure 3:  SRSS convective gain against laser intensity and scattered light
wavelength, for our experimental plasma conditions $L_{n_{c}/10}\approx
250\leavevmode\nobreak\ \upmu$m and $T_{e}=2.0\leavevmode\nobreak\ $keV. The
white curve highlights the iso-contour corresponding to a gain of 1. The white
region corresponds to the absolute regime. The dashed blue rectangle
highlights our experimental measurements.

FABS measurements typically show a slightly higher amount of scattered energy
than ARSDS ($\approx 1.5$ higher), with similar spectrum, for the same polar
angle as shown on the Fig. 2(c), despite the difference in azimuth, consistent
with Fig. 2(d). Such measurement evidences that there is no stronger emission
localised in the backward direction which could be attributed to SRBS. This is
expected as the laser intensity is one order of magnitude lower than the
predicted SRBS threshold of $\approx\leavevmode\nobreak\ 4.5\times
10^{15}\leavevmode\nobreak\ $W.cm-2 for our conditions Estabrook and Kruer
(1983); Kruer (1991); Montgomery (2016). Such low laser intensity also
prevents SRBS to grow from the high intensity speckles Rousseaux  _et al._
(2016); Glize  _et al._ (2017), which have been inferred to reach up to
$\approx 2.5\times 10^{15}\leavevmode\nobreak\ $W.cm-2 . This results in a
negligible reflectivity measured by FABS, $\leq 0.15\pm 0.05$%, as being only
a fraction of the total scattered energy. In order to account for the large
scattering angular spread, 50 additional fibers Zhang (2022) were used along
with ARSDS and FABS in order to extend the spatial measurement over
$\frac{4}{3}\pi$, as depicted on Fig. 1(c). This diagnostic setup is similar
to ARSDS, providing an energy measurement of SRS scattering integrated in
time. Despite inherent uncertainties due to the limited number of directions
probed, a reflectivity up to $5\pm 2\%$ of the total laser energy was
estimated, without considering the absorption. This evidences that SRSS is
energetically significant and necessitates to be measured over the whole
interaction volume. This implies that SRSS can lead to additional losses not
usually accounted for and could be one candidate to explain some “missing”
energy reported in recent experiments Hall  _et al._ (2017); Turnbull  _et
al._ (2022). Indeed, it appears that SRSS losses can be easily overlooked as
only scattering a low amount of energy locally, which would normally not be
detected or considered.

Figure 4:  (a) 2D spectrum of the scattered light averaged over 2 ps, (b) and
associated spectrum from PIC simulations.

To confirm the predominance of SRSS over SRBS, 2D plane-wave PIC simulations
have been performed using the code EPOCH Arber  _et al._ (2015). An
s-polarized plane wave is normally incident into a linear-density-profile
plasma slab ranging from $0.1n_{c}$ to $0.2n_{c}$, with a density scale-length
of $250\leavevmode\nobreak\ \upmu$m. The full simulation box is
$200\leavevmode\nobreak\ \upmu$m in length (x-axis) and
$80\leavevmode\nobreak\ \upmu$m in width (y-axis), and there is a
$10\leavevmode\nobreak\ \upmu$m vacuum at the left boundary. The longitudinal
boundary conditions are open boundary for fields and thermal boundary for
particles, and the transverse boundary condition is periodic. Interaction
parameters are as follow: the laser intensity $I_{0}=4.5\times
10^{14}\leavevmode\nobreak\ $W.cm-2, the electron temperature
$T_{e}=2\leavevmode\nobreak\ $keV, and fixed ions. Figure 4(a) shows a time-
averaged 2D spectrum of the scattered light wave over $2\leavevmode\nobreak\
$ps. The brightest signals are the near $90\degree$ scattered light
perpendicular to the density gradient. As expected, there is no measurable
SRBS due to its negligible convective gain. Figure 4(b) shows the scattered
light spectrum confirming SRSS growth over the whole range of density
simulated, consistent with the experiment.

In conclusion, it was experimentally evidenced and confirmed by 2D PIC
simulations that side-scattering is the dominant stimulated Raman scattering
geometry in the compression stage of DCI experiment. Due to the sensitivity to
both plasma profile and laser polarization, SRSS scattered light is emitted
over a wide volume, as observed experimentally and confirmed by ray-tracing
simulations. Such broad angular scattering results in a small amount of energy
being scattered locally, while being energetically significant when measured
over the whole interaction volume. Thus, relying on limited directions to
diagnose SRS, such as back-scattering, could lead to a strong underestimation
of the actual energy losses, by a factor of 35 in our conditions. Even though
the results discussed here were obtained using conical targets, it was
verified experimentally that our conclusions were still valid for planar
targets. Thus, the importance and relevance of our observations can be
extended beyond the DCI scheme. However, to assess the impact and significance
of side-scattering on other ICF schemes, further investigations are required
such as: (i) SRSS behavior at higher intensities; (ii) the competition between
Raman side- and back-scattering when SRBS threshold is overcome; (iii) the
effect of laser smoothing techniques such as smoothing by spectral dispersion
(SSD) Skupsky  _et al._ (1989) which could reportedly mitigate SRSS Kang  _et
al._ (2021).

We thank the SG-II UP operating group and target fabrication team for their
assistance. This work was supported by the Strategic Priority Research Program
of Chinese Academy of Sciences (grants No. XDA25010100, No. XDA25030200 & No.
XDA25050700).

## References

  * Nuckolls  _et al._ (1972) J. Nuckolls, L. Wood, A. Thiessen, and G. Zimmeran, Nature 239, 139 (1972).
  * Kruer (1991) W. L. Kruer, Journal of Plasma Physics 45, 135 (1991).
  * Comisar (1966) G. G. Comisar, Phys. Rev. 141, 200 (1966).
  * Kirkwood  _et al._ (2013) R. K. Kirkwood, J. D. Moody, J. Kline, E. Dewald, S. Glenzer, L. Divol, P. Michel, D. Hinkel, R. Berger, E. Williams, J. Milovich, L. Yin, H. Rose, B. MacGowan, O. Landen, M. Rosen, and J. Lindl, Plasma Physics and Controlled Fusion 55, 103001 (2013).
  * Montgomery (2016) D. S. Montgomery, Physics of Plasmas 23, 055601 (2016), https://doi.org/10.1063/1.4946016 .
  * Hall  _et al._ (2017) G. N. Hall, O. S. Jones, D. J. Strozzi, J. D. Moody, D. Turnbull, J. Ralph, P. A. Michel, M. Hohenberger, A. S. Moore, O. L. Landen, L. Divol, D. K. Bradley, D. E. Hinkel, A. J. Mackinnon, R. P. J. Town, N. B. Meezan, L. Berzak Hopkins, and N. Izumi, Physics of Plasmas 24, 052706 (2017).
  * Rosenberg  _et al._ (2018) M. J. Rosenberg, A. A. Solodov, J. F. Myatt, W. Seka, P. Michel, M. Hohenberger, R. W. Short, R. Epstein, S. P. Regan, E. M. Campbell, T. Chapman, C. Goyon, J. E. Ralph, M. A. Barrios, J. D. Moody, and J. W. Bates, Phys. Rev. Lett. 120, 055001 (2018).
  * Rosenberg  _et al._ (2020) M. J. Rosenberg, A. A. Solodov, W. Seka, R. K. Follett, J. F. Myatt, A. V. Maximov, C. Ren, S. Cao, P. Michel, M. Hohenberger, J. P. Palastro, C. Goyon, T. Chapman, J. E. Ralph, J. D. Moody, R. H. H. Scott, K. Glize, and S. P. Regan, Physics of Plasmas 27, 042705 (2020), https://doi.org/10.1063/1.5139226 .
  * Scott  _et al._ (2021) R. H. H. Scott, K. Glize, L. Antonelli, M. Khan, W. Theobald, M. Wei, R. Betti, C. Stoeckl, A. G. Seaton, T. D. Arber, D. Barlow, T. Goffrey, K. Bennett, W. Garbett, S. Atzeni, A. Casner, D. Batani, C. Li, and N. Woolsey, Phys. Rev. Lett. 127, 065001 (2021).
  * Cristoforetti  _et al._ (2019) G. Cristoforetti, L. Antonelli, D. Mancelli, S. Atzeni, F. Baffigi, F. Barbato, D. Batani, G. Boutoux, F. D’Amato, J. Dostal, R. Dudzak, E. Filippov, Y. J. Gu, L. Juha, O. Klimo, M. Krus, S. Malko, A. S. Martynenko, P. Nicolai, V. Ospina, S. Pikuz, O. Renner, J. Santos, V. T. Tikhonchuk, J. Trela, S. Viciani, L. Volpe, S. Weber, and L. A. Gizzi, High Power Laser Science and Engineering 7 (2019), 10.1017/hpl.2019.37.
  * Cristoforetti  _et al._ (2021) G. Cristoforetti, S. Hüller, P. Koester, L. Antonelli, S. Atzeni, F. Baffigi, D. Batani, C. Baird, N. Booth, M. Galimberti, K. Glize, A. Héron, M. Khan, P. Loiseau, D. Mancelli, M. Notley, P. Oliveira, O. Renner, M. Smid, A. Schiavi, G. Tran, N. C. Woolsey, and L. A. Gizzi, High Power Laser Science and Engineering 9, e60 (2021).
  * Baton  _et al._ (2020) S. Baton, A. Colaïtis, C. Rousseaux, G. Boutoux, S. Brygoo, L. Jacquet, M. Koenig, D. Batani, A. Casner, E. L. Bel, D. Raffestin, A. Tentori, V. Tikhonchuk, J. Trela, C. Reverdin, L. Le-Deroff, W. Theobald, G. Cristoforetti, L. Gizzi, P. Koester, L. Labate, and K. Shigemori, High Energy Density Physics 36, 100796 (2020).
  * Liu  _et al._ (1974) C. S. Liu, M. N. Rosenbluth, and R. B. White, The Physics of Fluids 17, 1211 (1974), https://aip.scitation.org/doi/pdf/10.1063/1.1694867 .
  * Mostrom and Kaufman (1979) M. A. Mostrom and A. N. Kaufman, Phys. Rev. Lett. 42, 644 (1979).
  * Afeyan and Williams (1985) B. B. Afeyan and E. A. Williams, The Physics of Fluids 28, 3397 (1985), https://aip.scitation.org/doi/pdf/10.1063/1.865340 .
  * Depierreux  _et al._ (2016) S. Depierreux, C. Neuville, C. Baccou, V. Tassin, M. Casanova, P.-E. Masson-Laborde, N. Borisenko, A. Orekhov, A. Colaitis, A. Debayle, G. Duchateau, A. Heron, S. Huller, P. Loiseau, P. Nicolaï, D. Pesme, C. Riconda, G. Tran, R. Bahr, J. Katz, C. Stoeckl, W. Seka, V. Tikhonchuk, and C. Labaune, Phys. Rev. Lett. 117, 235002 (2016).
  * Depierreux  _et al._ (2019) S. Depierreux, C. Neuville, V. Tassin, M.-C. Monteil, P.-E. Masson-Laborde, C. Baccou, P. Fremerye, F. Philippe, P. Seytor, D. Teychenné, J. Katz, R. Bahr, M. Casanova, N. Borisenko, L. Borisenko, A. Orekhov, A. Colaitis, A. Debayle, G. Duchateau, A. Heron, S. Huller, P. Loiseau, P. Nicolai, C. Riconda, G. Tran, C. Stoeckl, W. Seka, V. Tikhonchuk, D. Pesme, and C. Labaune, Plasma Physics and Controlled Fusion 62, 014024 (2019).
  * Short (2020) R. W. Short, Physics of Plasmas 27, 042703 (2020), https://doi.org/10.1063/1.5131158 .
  * Michel  _et al._ (2015) P. Michel, L. Divol, E. L. Dewald, J. L. Milovich, M. Hohenberger, O. S. Jones, L. B. Hopkins, R. L. Berger, W. L. Kruer, and J. D. Moody, Phys. Rev. Lett. 115, 055003 (2015).
  * Michel  _et al._ (2019) P. Michel, M. J. Rosenberg, W. Seka, A. A. Solodov, R. W. Short, T. Chapman, C. Goyon, N. Lemos, M. Hohenberger, J. D. Moody, S. P. Regan, and J. F. Myatt, Phys. Rev. E 99, 033203 (2019).
  * Rosenberg  _et al._ (2021) M. J. Rosenberg, J. E. Hernandez, N. Butler, T. Filkins, R. E. Bahr, R. K. Jungquist, M. Bedzyk, G. Swadling, J. S. Ross, P. Michel, N. Lemos, J. Eichmiller, R. Sommers, P. Nyholm, R. Boni, J. A. Marozas, R. S. Craxton, P. W. McKenty, A. Sharma, P. B. Radha, D. H. Froula, P. Datte, M. Gorman, J. D. Moody, J. M. Heinmiller, J. Fornes, P. Hillyard, and S. P. Regan, Review of Scientific Instruments 92, 033511 (2021), https://doi.org/10.1063/5.0040558 .
  * Zhang  _et al._ (2020) J. Zhang, W. M. Wang, X. H. Yang, D. Wu, Y. Y. Ma, J. L. Jiao, Z. Zhang, F. Y. Wu, X. H. Yuan, Y. T. Li, and J. Q. Zhu, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 378, 20200015 (2020), https://doi/pdf/10.1098/rsta.2020.0015 .
  * McCrory  _et al._ (2008) R. L. McCrory, D. D. Meyerhofer, R. Betti, R. S. Craxton, J. A. Delettrez, D. H. Edgell, V. Y. Glebov, V. N. Goncharov, D. R. Harding, D. W. Jacobs-Perkins, J. P. Knauer, F. J. Marshall, P. W. McKenty, P. B. Radha, S. P. Regan, T. C. Sangster, W. Seka, R. W. Short, S. Skupsky, V. A. Smalyuk, J. M. Soures, C. Stoeckl, B. Yaakobi, D. Shvarts, J. A. Frenje, C. K. Li, R. D. Petrasso, and F. H. Séguin, Physics of Plasmas 15, 055503 (2008), https://doi.org/10.1063/1.2837048 .
  * Zhao  _et al._ (2022) X. Zhao, X. H. Yuan, J. Zheng, Y. F. Dong, K. Glize, Y. H. Zhang, Z. Zhang, and J. Zhang, Review of Scientific Instruments 93, 053505 (2022), https://doi.org/10.1063/5.0090841 .
  * Xu  _et al._ (2008) G. Xu, T. Wang, Z. Li, Y. Dai, Z. Lin, Y. Gu, and J. Zhu, The Review of Laser Engineering 36, 1172 (2008).
  * Ramis  _et al._ (2009) R. Ramis, J. M. ter Vehn, and J. Ramírez, Computer Physics Communications 180, 977 (2009).
  * Seka  _et al._ (1985) W. Seka, B. B. Afeyan, R. Boni, L. M. Goldman, R. W. Short, K. Tanaka, and T. W. Johnston, The Physics of Fluids 28, 2570 (1985), https://aip.scitation.org/doi/pdf/10.1063/1.865265 .
  * Menyuk  _et al._ (1985) C. R. Menyuk, N. M. El‐Siragy, and W. M. Manheimer, The Physics of Fluids 28, 3409 (1985), https://aip.scitation.org/doi/pdf/10.1063/1.865341 .
  * Xiao  _et al._ (2016) C. Z. Xiao, Z. J. Liu, C. Y. Zheng, and X. T. He, Physics of Plasmas 23, 022704 (2016), https://doi.org/10.1063/1.4941969 .
  * Xiao  _et al._ (2018) C. Z. Xiao, H. B. Zhuo, Y. Yin, Z. J. Liu, C. Y. Zheng, Y. Zhao, and X. T. He, Plasma Physics and Controlled Fusion 60, 025020 (2018).
  * Gu  _et al._ (2021) Y.-J. Gu, O. Klimo, V. Tikhonchuk, and S. Weber, Nuclear Fusion 61, 066014 (2021).
  * Ji  _et al._ (2022) Y. Ji, C. W. Lian, and R. Yan, in preparation (2022).
  * Kaiser (2000) T. B. Kaiser, Phys. Rev. E 61, 895 (2000).
  * Yang  _et al._ (2022) M. Yang, F. Y. Wu, Z. B. Chen, Y. X. Zhang, Y. Chen, J. C. Zhang, Z. Z. Chen, Z. F. Fang, R. Ramis, and J. Zhang, Acta Physica Sinica submitted (2022).
  * Estabrook and Kruer (1983) K. Estabrook and W. L. Kruer, The Physics of Fluids 26, 1892 (1983), https://aip.scitation.org/doi/pdf/10.1063/1.864336 .
  * Rousseaux  _et al._ (2016) C. Rousseaux, K. Glize, S. D. Baton, L. Lancia, D. Bénisti, and L. Gremillet, Phys. Rev. Lett. 117, 015002 (2016).
  * Glize  _et al._ (2017) K. Glize, C. Rousseaux, D. Bénisti, V. Dervieux, L. Gremillet, S. D. Baton, and L. Lancia, Physics of Plasmas 24, 032708 (2017), https://doi.org/10.1063/1.4978879 .
  * Zhang (2022) Y. H. Zhang, Review of Scientific Instruments in preparation (2022).
  * Turnbull  _et al._ (2022) D. Turnbull, J. Katz, D. E. Hinkel, P. Michel, T. Chapman, L. Divol, E. Kur, S. MacLaren, A. L. Milder, M. Rosen, A. Shvydky, G. B. Zimmerman, and D. H. Froula, Phys. Rev. Lett. 129, 025001 (2022).
  * Arber  _et al._ (2015) T. D. Arber, K. Bennett, C. S. Brady, A. Lawrence-Douglas, M. G. Ramsay, N. J. Sircombe, P. Gillies, R. G. Evans, H. Schmitz, A. R. Bell, and C. P. Ridgers, Plasma Physics and Controlled Fusion 57, 113001 (2015).
  * Skupsky  _et al._ (1989) S. Skupsky, R. W. Short, T. Kessler, R. S. Craxton, S. Letzring, and J. M. Soures, Journal of Applied Physics 66, 3456 (1989), https://doi.org/10.1063/1.344101 .
  * Kang  _et al._ (2021) N. Kang, H. Liu, S. Zhou, Y. Zhao, and A. Lei, J. Opt. Soc. Am. B 38, 3567 (2021).