* Braun-Munzinger  _et al._ (2017) P. Braun-Munzinger, A. Rustamov, and J. Stachel, “Bridging the gap between event-by-event fluctuation measurements and theory predictions in relativistic nuclear collisions,” Nucl. Phys. A 960, 114–130 (2017), arXiv:1612.00702 [nucl-th] .
  * Vovchenko  _et al._ (2020) Volodymyr Vovchenko, Oleh Savchuk, Roman V. Poberezhnyuk, Mark I. Gorenstein, and Volker Koch, “Connecting fluctuation measurements in heavy-ion collisions with the grand-canonical susceptibilities,” Phys. Lett. B 811, 135868 (2020), arXiv:2003.13905 [hep-ph] .
  * Nahrgang  _et al._ (2015) Marlene Nahrgang, Marcus Bluhm, Paolo Alba, Rene Bellwied, and Claudia Ratti, “Impact of resonance regeneration and decay on the net-proton fluctuations in a hadron resonance gas,” Eur. Phys. J. C 75, 573 (2015), arXiv:1402.1238 [hep-ph] .
  * Zhang  _et al._ (2020) Yu Zhang, Shu He, Hui Liu, Zhenzhen Yang, and Xiaofeng Luo, “Effects of resonance weak decay and hadronic rescattering on the proton number fluctuations in Au+Au collisions at $\sqrt{s_{\mathrm{NN}}}=5$ from a microscopic hadronic transport (JAM) model,” Phys. Rev. C 101, 034909 (2020), arXiv:1905.01095 [nucl-ex] .
  * Berdnikov and Rajagopal (2000) Boris Berdnikov and Krishna Rajagopal, “Slowing out-of-equilibrium near the QCD critical point,” Phys. Rev. D 61, 105017 (2000), arXiv:hep-ph/9912274 .
  * Bluhm  _et al._ (2020a) Marcus Bluhm _et al._ , “Dynamics of critical fluctuations: Theory – phenomenology – heavy-ion collisions,” Nucl. Phys. A 1003, 122016 (2020a), arXiv:2001.08831 [nucl-th] .
  * Bluhm  _et al._ (2020b) Marcus Bluhm, Marlene Nahrgang, and Jan M. Pawlowski, “Locating the freeze-out curve in heavy-ion collisions,” (2020b), arXiv:2004.08608 [nucl-th] .
  * Bluhm  _et al._ (2019) M. Bluhm, Y. Jiang, M. Nahrgang, J. M. Pawlowski, F. Rennecke, and N. Wink, “Time-evolution of fluctuations as signal of the phase transition dynamics in a QCD-assisted transport approach,”  _Proceedings, 27th International Conference on Ultrarelativistic Nucleus-Nucleus Collisions (Quark Matter 2018): Venice, Italy, May 14-19, 2018_ , Nucl. Phys. A982, 871–874 (2019), arXiv:1808.01377 [hep-ph] .
  * Braun-Munzinger  _et al._ (2021) P. Braun-Munzinger, B. Friman, K. Redlich, A. Rustamov, and J. Stachel, “Relativistic nuclear collisions: Establishing a non-critical baseline for fluctuation measurements,” Nucl. Phys. A 1008, 122141 (2021), arXiv:2007.02463 [nucl-th] .
  * Braun (2009) Jens Braun, “The QCD Phase Boundary from Quark-Gluon Dynamics,” Eur. Phys. J. C64, 459–482 (2009), arXiv:0810.1727 [hep-ph] .
  * Braun  _et al._ (2010) Jens Braun, Holger Gies, and Jan M. Pawlowski, “Quark Confinement from Color Confinement,” Phys.Lett. B684, 262–267 (2010), arXiv:0708.2413 [hep-th] .
  * Cyrol  _et al._ (2016) Anton K. Cyrol, Leonard Fister, Mario Mitter, Jan M. Pawlowski, and Nils Strodthoff, “Landau gauge Yang-Mills correlation functions,” Phys. Rev. D94, 054005 (2016), arXiv:1605.01856 [hep-ph] .
  * Cyrol  _et al._ (2018b) Anton K. Cyrol, Mario Mitter, Jan M. Pawlowski, and Nils Strodthoff, “Nonperturbative finite-temperature Yang-Mills theory,” Phys. Rev. D97, 054015 (2018b), arXiv:1708.03482 [hep-ph] .
  * Braun  _et al._ (2020b) Jens Braun, Wei-jie Fu, Jan M. Pawlowski, Fabian Rennecke, Daniel Rosenblüh, and Shi Yin, “Chiral susceptibility in ( 2+1 )-flavor QCD,” Phys. Rev. D 102, 056010 (2020b), arXiv:2003.13112 [hep-ph] .
  * Braun  _et al._ (2020c) Jens Braun, Marc Leonhardt, Jan M. Pawlowski, and Daniel Rosenblüh, “Chiral and effective $U(1)_{\rm A}$ symmetry restoration in QCD,” (2020c), arXiv:2012.06231 [hep-ph] .
  * Litim and Pawlowski (1998) Daniel F. Litim and Jan M. Pawlowski, “On gauge invariant Wilsonian flows,” in  _The exact renormalization group. Proceedings, Workshop, Faro, Portugal, September 10-12, 1998_ (1998) pp. 168–185, arXiv:hep-th/9901063 [hep-th] .
  * Berges  _et al._ (2002) Juergen Berges, Nikolaos Tetradis, and Christof Wetterich, “Nonperturbative renormalization flow in quantum field theory and statistical physics,” Phys. Rept. 363, 223–386 (2002), arXiv:hep-ph/0005122 [hep-ph] .
  * Schaefer and Wambach (2008) Bernd-Jochen Schaefer and Jochen Wambach, “Renormalization group approach towards the QCD phase diagram,”  _Helmholtz International Summer School on Dense Matter in Heavy Ion Collisions and Astrophysics Dubna, Russia, August 21-September 1, 2006_ , Phys. Part. Nucl. 39, 1025–1032 (2008), arXiv:hep-ph/0611191 [hep-ph] .
  * Gies (2012) Holger Gies, “Introduction to the functional RG and applications to gauge theories,”  _Renormalization group and effective field theory approaches to many-body systems_ , Lect. Notes Phys. 852, 287–348 (2012), arXiv:hep-ph/0611146 [hep-ph] .
  * Rosten (2012) Oliver J. Rosten, “Fundamentals of the Exact Renormalization Group,” Phys. Rept. 511, 177–272 (2012), arXiv:1003.1366 [hep-th] .
  * Braun (2012) Jens Braun, “Fermion Interactions and Universal Behavior in Strongly Interacting Theories,” J. Phys. G39, 033001 (2012), arXiv:1108.4449 [hep-ph] .
  * Pawlowski (2014) Jan M. Pawlowski, “Equation of state and phase diagram of strongly interacting matter,”  _Proceedings, 24th International Conference on Ultra-Relativistic Nucleus-Nucleus Collisions (Quark Matter 2014): Darmstadt, Germany, May 19-24, 2014_ , Nucl. Phys. A931, 113–124 (2014).
  * Litim (2000) Daniel F. Litim, “Optimization of the exact renormalization group,” Phys. Lett. B486, 92–99 (2000), arXiv:hep-th/0005245 [hep-th] .
  * Litim (2001) Daniel F. Litim, “Optimized renormalization group flows,” Phys. Rev. D64, 105007 (2001), arXiv:hep-th/0103195 [hep-th] .
  * Litim and Pawlowski (2006) Daniel F. Litim and Jan M. Pawlowski, “Non-perturbative thermal flows and resummations,” JHEP 11, 026 (2006), arXiv:hep-th/0609122 .
  * Khan  _et al._ (2015) Naseemuddin Khan, Jan M. Pawlowski, Fabian Rennecke, and Michael M. Scherer, “The Phase Diagram of QC2D from Functional Methods,” (2015), arXiv:1512.03673 [hep-ph] .
  * Pawlowski and Rennecke (2014) Jan M. Pawlowski and Fabian Rennecke, “Higher order quark-mesonic scattering processes and the phase structure of QCD,” Phys. Rev. D90, 076002 (2014), arXiv:1403.1179 [hep-ph] .
  * Rennecke (2015b) Fabian Rennecke,  _The Chiral Phase Transition of QCD._ , Ph.D. thesis, U. Heidelberg (main) (2015b).
  * Rennecke and Schaefer (2017) Fabian Rennecke and Bernd-Jochen Schaefer, “Fluctuation-induced modifications of the phase structure in (2+1)-flavor QCD,” Phys. Rev. D96, 016009 (2017), arXiv:1610.08748 [hep-ph] .
  * Lo  _et al._ (2013) Pok Man Lo, Bengt Friman, Olaf Kaczmarek, Krzysztof Redlich, and Chihiro Sasaki, “Polyakov loop fluctuations in SU(3) lattice gauge theory and an effective gluon potential,” Phys. Rev. D88, 074502 (2013), arXiv:1307.5958 [hep-lat] .
  * Pawlowski (2011) Jan M. Pawlowski, “The QCD phase diagram: Results and challenges,”  _Proceedings, 9th Conference on Quark Confinement and the Hadron Spectrum: Madrid, Spain, 30 Aug-3 Sep 2010_ , AIP Conf. Proc. 1343, 75–80 (2011), arXiv:1012.5075 [hep-ph] .


# Inertial Confinement Fusion Forecasting
via LLMs

Mingkai Chen
Rochester Institute of Technology
<EMAIL_ADDRESS>
&Taowen Wang
Rochester Institute of Technology
<EMAIL_ADDRESS>
&James Chenhao Liang
Rochester Institute of Technology
<EMAIL_ADDRESS>
&Chuan Liu
University of Rochester
<EMAIL_ADDRESS>
&Chunshu Wu
University of Rochester
<EMAIL_ADDRESS>
&Qifan Wang
Meta AI
<EMAIL_ADDRESS>
&Ying Nian Wu
University of California, Los Angeles
<EMAIL_ADDRESS>
&Michael Huang
University of Rochester
<EMAIL_ADDRESS>
&Chuang Ren
University of Rochester
<EMAIL_ADDRESS>
&Ang Li
Pacific Northwest National Laboratory
<EMAIL_ADDRESS>
&Tong Geng
University of Rochester
<EMAIL_ADDRESS>
&Dongfang Liu
Rochester Institute of Technology
<EMAIL_ADDRESS>
Corresponding author.

###### Abstract

Controlled fusion energy is deemed pivotal for the advancement of human
civilization. In this study, we introduce Fusion-LLM, a novel integration of
Large Language Models (LLMs) with classical reservoir computing paradigms
tailored to address challenges in Inertial Confinement Fusion (ICF). Our
approach offers several key contributions: Firstly, we propose the LLM-
anchored Reservoir, augmented with a fusion-specific prompt, enabling accurate
forecasting of hot electron dynamics during implosion. Secondly, we develop
Signal-Digesting Channels to temporally and spatially describe the laser
intensity across time, capturing the unique characteristics of ICF inputs.
Lastly, we design the Confidence Scanner to quantify the confidence level in
forecasting, providing valuable insights for domain experts to design the ICF
process. Extensive experiments demonstrate the superior performance of our
method, achieving 1.90 CAE, 0.14 top-1 MAE, and 0.11 top-5 MAE in predicting
Hard X-ray (HXR) energies of ICF tasks, which presents state-of-the-art
comparisons against concurrent best systems. Additionally, we present
Fusion4AI, the first ICF benchmark based on physical experiments, aimed at
fostering novel ideas in plasma physics research and enhancing the utility of
LLMs in scientific exploration. Overall, our work strives to forge an
innovative synergy between AI and plasma science for advancing fusion energy.

##  1 Introduction

> _“…human society remains at a Type 0, a primitive form of civilization…”_
>
> $-$ The Kardashev scale [34]

After National Ignition Facility (NIF) achieving ignition in December 2022
[1], the focus of current inertial confinement fusion (ICF) research shifts to
exploring high gain schemes required to make fusion a practical and
sustainable energy source for humankind. Fusion represents a potential key
enabler for advancing humanity towards a Type I civilization on the Kardashev
scale [91], offering a virtually limitless and clean energy source that could
power our civilization globally. This advancement could potentially resolve
numerous crises we currently face — $e.g.,$ economic recessions and climate
change — by eliminating the need for finite resources like fossil fuels.

The optimization of ICF designs and the achievement of reliable ignition face
formidable constraints [8, 14] due to laser-plasma instabilities (LPI)[22,
54]. Efficient and symmetrical driving of the target, vital for ICF, is
impeded by LPI phenomena such as stimulated Raman and Brillouin
backscatterings (SRS and SBS), which can disrupt implosion symmetry and reduce
efficiency through cross-beam energy transfer (CBET) [68, 21]. Hot electrons,
a byproduct of LPI processes like SRS and Two-Plasmon-Decay (TPD), can both
hinder and assist ignition, showcasing the complexity of the interaction.
Despite efforts to measure and simulate hot electrons, obtaining predictive
scaling laws remains challenging due to the dynamic nature of laser/plasma
conditions and computational limitations [38, 57, 89, 66, 42], highlighting
the current gap in establishing a predictive capability based on first
principles that aligns with experimental data. These constraints highlight the
need for an innovative approach to overcome these obstacles.

Figure 1: The overview of Fusion-LLM.

Currently, Large Language Models (LLMs) exhibit versatile capabilities across
diverse disciplines ($e.g.,$ robotics [44, 73, 90, 26], medical health [41,
67, 77, 50], agriculture [56, 80]), adeptly capturing intricate patterns in
multimodal data. Due to their success in other domains [39, 93, 46], we are
convinced that LLMs may also potentially excel in generalizing to plasma
physics, particularly in forecasting the behavior of hot electrons generated
during implosion in the real-world, physics experiments. Leveraging their vast
pre-trained knowledge base, LLMs could optimize ICF designs by efficiently
evaluating numerous scenarios, aiding researchers in identifying promising
approaches expediently, and generating insights to enhance our understanding
and control of the fusion process.

In light of this perspective, we intend to harness the power of LLMs to
overcome the barrier to the vertical advancement of the next stage of human
civilization: fusion energy. This science problem manifests as two sub-
problems in a unique and challenging setup: ❶ how to tailor pre-trained LLMs
in order to accurately predict the behavior of hot electrons based on laser
intensity inputs? and ❷ how to evaluate the trustworthiness of the LLM’s
predictions in order to guide the ICF design?

We conceptualize LLMs as a computational reservoir to unlock their potential
for robust domain adaptation and generalization for ICF task, titled Fusion-
LLM (see Fig. 1). In order to address question ❶, we propose the LLM-anchored
Reservoir, augmented with a fusion-specific prompt, to facilitate the
interpretation of plasma physics. The incorporated prompts encompass domain-
specific knowledge, instructional cues, and statistical information, thereby
enabling the LLMs to accommodate the specific demands of the given task.
Additionally, we develop the Signal-Digesting Channels to capture the
distinctive characteristics of ICF inputs. It features a temporal encoder to
better align the laser signals with the pre-trained, time-series space, and a
spatial encoder to provide a global description of the input landscape. To
tackle question ❷, we introduce the Confidence Scanner to assess the
trustworthiness in predictions. Specifically, we couple the gradient saliency
in prediction head and token entropy in LLM outputs to obtain the model’s
confidence scores.

Overall, this study aims to provide an exciting synergy between the domain of
AI and plasma science for fusion energy development. The core contributions of
this paper can be summarized as follows:

  * •

We represent a pioneer investigation into the use of LLMs for analyzing hot
electron dynamics in ICF. LLMs offer a cost-effective alternative compared to
both plasma-physics experiments and simulations at the ignition scale.
Empirical evidence (see §3/S2) showcases the efficacy of the application of
LLM for plasma physics study, which directly benefits the practical design of
ICF.

  * •

We construct an LLM-anchored reservoir computing framework to predict the hot
electron dynamics in ICF implosions. Compared to prior arts in reservoir
computing [43, 20] and time-series-based LLM [31], our approach requires less
data (see Table 2d) and shorter training schedule (see Table 2e), while
achieving superior performance.

  * •

We develop the first fusion benchmark — Fusion4AI (see §3.1) based on physical
experiments, to facilitate new ideas in plasma physics research and the use of
LLMs in scientific exploration.

##  2 Methodology

###  2.1 Preliminary

Figure 2: The ICF pipeline.

The ICF Overview. A ICF shot is depicted in Fig. 2. Laser beams  are directed
towards a target fuel pellet , rapidly heating its surface to generate a
plasma envelope . This plasma envelope subsequently undergoes implosion
through a rocket-like expulsion of material, compressing the pellet to
densities exceeding that of lead by twenty-fold and attaining ignition
temperatures of approximately 100,000,000∘C, thus initiating fusion reactions.
Throughout this process, laser-induced plasma instabilities give rise to hot
electrons, which may prematurely heat the pellet and emit Hard X-Rays (HXR)
emissions  as they interact within the plasma environment. Understanding the
hot electron dynamics is crucial for guiding the design of the ICF. In the
context, HXR diagnostics have been extensively applied in studies at
facilities such as OMEGA [69, 88, 71, 87] and the NIF [58, 59, 70]. It is
however worth noting that conducting these experiments is extremely expensive
(around $1m for one shot).

Experiment and Data Collection. To collect the fusion data, we conduct 100
real-world ignitions of ICF. Each ignition is accompanied by detailed
configurations including the size of the fuel pellet, the phase plate of the
laser, and the input laser profile. We also measure raw sensor readings
regarding HXR, which can be further converted into charges of hot electrons,
allowing for the subsequent calculation of the total energy of these electrons
based on their temperature. In this study, we focus on forecasting the energy
from HXR emissions by given laser intensity inputs throughout the ignition.
This data is indispensable for physicists to ascertain and analyze the state
of ICF.

###  2.2 Fusion-LLM

In this section, we introduce Fusion-LLM, a novel approach for predicting hot-
electron energy in ICF. Illustrated in Fig. 3, the inputs comprise fusion-
specific prompts and HXR signals, which are processed through LLM for feature
extraction. Subsequently, the output module makes predictions of hot-electron
energy along with confidence scores. Our approach is characterized by three
core modules: LLM-anchored Reservoir, which establishes a reservoir foundation
using LLMs to comprehend the dynamic impact of laser intensity on hot-electron
energy emission; Signal-Digesting Channels is responsible for encoding HXR
signals, capturing the temporal characteristics of sequential details, and
spatial distinctiveness of data landscapes; and Confidence Scanner, tasked
with estimating prediction confidence for each shot, thereby providing
trustworthy guidance for practical ICF design.

Figure 3:  Fusion-LLM framework. (a) Fusion-specific prompts structure domain
textual prompts with context, task and input descriptions (see §2.2.1 for
details). (b) Signal-digesting channels comprise a pre-trained temporal
encoder to extract time-series features of laser signals, and a spatial
encoder to encode critical landscapes of the inputs (see §2.2.2 for details).
For simplicity, we skip some architectural modules. We provide implementation
details in appendix (see §S1).

####  2.2.1 LLM-anchored Reservoir

In classic reservoir computing (RC), a fixed, randomly generated reservoir
($e.g.$, RNN) transforms input data into a high-dimensional representation. A
trainable output layer then maps this representation to the desired output. RC
has gained popularity in the scientific community due to its ability to enable
efficient processing of sequential data with simple training methods.
Following the RC convention, we construct a reservoir using LLM with a shallow
prediction head (see §S1). Due to the extensive pre-training, LLMs are
equipped with robust generalization capabilities for in-context reasoning and
time-series forecasting. To leverage the physics knowledge embedded in LLMs,
we design fusion-specific prompts (FSP) that strategically connect the LLM’s
vast knowledge base to the specific nuances of the ICF domain. Our LLM-
anchored reservoir can be given by:

$s[t+1]=f(s[:t],E_{las},^{**}\texttt{Prompt}),$ (1)

where $s[t]$ denotes the state of the reservoir at time step $t$ and $E_{las}$
represents input laser intensity.

The ${}^{**}\texttt{Prompt}$ specifically denotes the structured FSP,
comprising three descriptors (see Fig. 3a) designed with particular focus: ➤
Context descriptor provides a detailed overview of the ICF process,
highlighting the nature, sources, and characteristics of the data. It
elaborates on the experimental procedures (see §2.1) used in data collection,
emphasizing principles and methodologies specific to ICF. This descriptor
enhances the LLMs’ comprehension of the experimental context. ➤ Task
descriptor outlines explicit instructions for the prediction tasks, including
the format and expected length of the output. It guides the inference and
forecasting process by specifying imperative considerations, ensuring that the
forecasting align with domain-specific insights. ➤ Input descriptor presents a
concise statistical summary of the input data, offering insights into its
distribution and key characteristics such as minimum, maximum, and median
values. This descriptor is vital for informing the LLMs about the underlying
statistical properties of the input signals. Collectively, this prompting
strategy facilitates the LLMs’ ability to mine intricate input signals and
produce predictions that are scientifically robust and contextually coherent
within the phase space of the reservoir.

In general, our LLM-based design extends the territorial boundary of reservoir
computing, offering two significant advantages in leveraging artificial
intelligence for scientific exploration:

  * •

Adaptability for science. The utilization of LLMs in ICF forcasting exhibits
notable adaptability in confronting a pivotal scientific challenge. Later, we
will present empirical evidence to substantiate the systemic efficacy (see
§3.1). The success of this methodology is poised to provide a generic
alternative for other adjacent scientific domains in pursuit of LLM-based
solutions.

  * •

Efficiency for data scarcity. Gratitude is extended for the extensive pre-
trained knowledge base and the precise delineation of prompt descriptors.
Leveraging these assets, the LLM-based system mitigates data dependency to the
K-shot level — 80 shots in our experiments — exemplifying the advantageous
efficiency in addressing the persistent challenge of data scarcity in
scientific inquiry.

####  2.2.2 Signal-Digesting Channels

With the establishment of the LLM-based approach described in §2.2.1, robust
predictions can be immediately attained (see Table 2c). Due to the uniqueness
of the ICF data, the input HXR signal exhibits its time-series, temporal
landscape. Recognizing the sensitivity of LLMs to input data [64, 65], we
introduce Signal-Digesting Channels (SDC), conceptually aligned with our
reservoir design, to capture crucial input characteristics and further augment
the performance of LLMs.

For the ICF process, the hot electron energy emitted during the initial and
terminal phases is characterized by relatively uniform values, in contrast to
the target impact phase, where peak values are observed. ${}_{\\!}$This laser-
energy signal landscape exhibits significant discrepancies between the
uniformity of the plain phase and the peaks of the impact phase.
${}_{\\!}$This physics insight guides our SDC design (see Fig. 3b), which
comprises two components: a temporal encoder to align the laser signals with
the pre-trained time-series space, and a spatial encoder to delineate the
landscape of input data.

➤ Temporal encoder is designed to extract time-series features using a
windowing mechanism that constructs consistent signal patches across
sequential time steps. It employs a set of Transformer layers [85] to capture
time-series distributions over the forecast horizon. This process is
formulated as $f_{tmp}:(X_{t-l:t+h},Y_{t-l:t})\mapsto\hat{\psi}$, where $X$
and $Y$ represent the input data and target data spanning time windows of
length $l$ and $h$ at time $t$. The encoder is pre-trained on a large-scale
time-series dataset (see §S1 for details) and is optimized using the log-
likelihood of the forecast:

$\arg\max_{\theta}\quad\mathbb{E}_{(X,Y)\sim p(D)}\log
p(Y_{t:t+h}|f_{tmp}(X_{t-l:t+h},Y_{t-l:t})),$

where $p(D)$ represents the data distribution from which the time-series
samples are drawn. During fine-tuning on the target-domain ICF data, all pre-
trained parameters are frozen, except for the last linear layer. The temporal
encoder is utilized for feature extraction over the input laser signals $I$,
producing temporal tokens denoted as $E_{tmp}=f_{tmp}(I)$ for subsequent
processing.

➤ ${}_{\\!}$Spatial encoder is designed to analyze the input signals by
providing a qualitative overview of the input landscape. Specifically, it is
structured to characterize the spatial patterns of laser intensity signals
throughout the ICF process. We utilize the projection block $f_{LLM}$ to
project sets of critical contexts into spatial features, denoted as
$E_{spt}=\\{f_{LLM}(\text{``pulse''}),f_{LLM}(\text{``peak''}),...,f_{LLM}(\text{``trailing''})\\}$.
In practice, $E_{spt}$ is further processed by a cross-attention layer with
temporal features $E_{tmp}$. This step couples the contextual description to
the actual signal distribution within the ICF, enabling the LLM to better
capture the observed physical phenomena for predictions.

After acquiring the features from both the temporal and spatial encoders, we
concatenate them $\dot{E}$ = [$E_{tmp}$;$E_{spt}$] to form the output of DSC.
Here, we use $\dot{E}$ as augmented inputs to replace the vanilla $E_{las}$ in
Eq.1. Fundamentally, DSC introduces a novel method for input encoding in
reservoir computing, which enhances overall system performance. This design
offers the following advantages:

  * •

Discernment for temporal pattern. With the pre-trained temporal encoder, SDC
adeptly captures crucial time-series features of laser signals that correlate
with HXR outputs. This merit enables the LLM to recognize distinct patterns
across various time steps, representing an improvement over strong reservoir
models (see Table 2c), which struggle to manage ICF’s temporal patterns.

  * •

In-context reasoning in signal processing. SDC tackles the complexity of
processing signals with diverse attributes, such as those found in the uniform
and peak phases within ICF. Through the integration of contextual disciplinary
knowledge, SDC, equipped with in-context reasoning, significantly boosts the
effectiveness of LLM backbone (see Table 2c). This enhancement enables robust
performance even in the face of inherent fluctuations (i.e., uniform $vs$.
peak in ICF).

####  2.2.3 Confidence Scanner

Figure 4: Pipeline of Confidence Scanner. We re-calibrate token confidence to
align with the energy-prediction head.

Trustworthiness is pivotal for AI in science. Under-confident predictions may
lead to misguided conclusions or improper actions. Some approaches [45, 27]
directly utilize the entropy observed in the output tokens of the LLMs to
gauge the confidence. However, these methods encounter a challenge in our
study of ICF whereby the entropy of LLM’s output tokens does not consistently
reflect confidence of prediction at each time step. This discrepancy arises
due to the non-linear transformation undertaken by the multi-layer perception
within the prediction head, which alters the embedding of token counterparts,
thereby distorting the relation between tokens and their corresponding
confidence estimations.

To this end, we propose Confidence Scanner that incorporates a confidence
reweighing mechanism to assess the confidence level of each prediction
systematically. Concretely, our approach re-calibrates the allocation of
confidence across tokens to implicitly reflect their actual influence on
predictions. As shown in Fig. 4, we extract the embedding
$E_{k}=\\{e_{n-k},\dots,e_{n}\\}$ for the last layer of LLM, which
specifically analyze the embedding of the last $k$ tokens. The confidence
level $H$ is then formulated as:

$H=\left[h(e_{n-k}),\dots,h(e_{n})\right],$ (2)

where $h(\cdot)$ map the embedding to word probabilities, which are
subsequently used to calculate the entropy. Furthermore, we derive a reweigh
matrix $S$ from the task prediction head $H_{task}(\cdot)$ by performing a
forward process to predict hot-electron energy by $P=H_{task}(E_{k})$. The
matrix $S$ is then obtained through saliency, reflecting the contribution to
the $i$-length of predictions:

$S=\left[\sigma\left(\frac{\partial P_{0}}{\partial
E_{k}}\right),\dots,\sigma\left(\frac{\partial P_{i}}{\partial
E_{k}}\right)\right],$ (3)

where the $\sigma$ is the softmax function to normalize the saliency scale.
Finally, entropy $H$ is combined with reweigh matrix $S$ to produce the
confidence score $C=-H\times S$ for the hot electron energy predictions.
Through our design, we align the entropy derived from LLM embeddings with the
hot electron energy forecasting. This alignment allows us to directly obtain a
confidence level that can serve as a trustworthiness indicator for our system.
We provide empirical evidence in Fig. 6.

##  3 Empirical Findings

Figure 5: Qualitative results of hot electron prediction. We plot Ground Truth
and the predictions of Ours, Time-LLM, LSTM and Autoformer. Y and X axes
denote energy and time steps respectively.

###  3.1 Main Results

Dataset. We develop a new benchmark, Fusion4AI, to support AI research in ICF.
This benchmark consists of 40,000 LPI samples containing 100 shot sequences
with 400 time steps/shot. Each shot is documented with key parameters such as
target size, laser intensity, and energy of hot electrons (see §2.1). The
dataset has been systematically divided into 80/10/10 for train/val/test
splits respectively. We will release this dataset upon acceptance to advance
research for the fusion reaction.

Baselines. We choose a classic physical Particle-In-Cell (PIC) simulation
method [13], a number of classic AI models (i.e., LSTM [25], Autoformer [86]),
reservoir computing models (i.e., HoGRC [43], RCRK [17], NGRC [20]), and
concurrent time-series LLM-based models (i.e., GPT4TS [94] and Time-LLM [31])
as baseline models for performance comparison on proposed Fusion4AI dataset.

Experimental Setup. Our experiments are trained with 100 epochs and a batch
size of 5, which is adequate to achieve convergence based on our empirical
findings. In addition, we utilize a fixed learning rate of $0.0004$ and the
Adam optimizer [36]. A loss function defined by the cumulative absolute error
across each time-steps
$f_{\text{loss}}(Y,G)=\sum_{n=1}^{\text{pred\\_len}}|y_{n}-g_{n}|$ is used,
where $Y$ and $G$ represent the sequences of predictions and ground truths,
respectively, $y_{n}$ and $g_{n}$ denote the values at the $n$-th time-step,
and pred_len is the length of prediction. For all other counterpart methods,
we follow the original experimental setting and training configurations to
reproduce the results.

Metric. We employ cumulative absolute error (CAE) as our primary metric. The
sole distinction in its implementation involves nullifying values that are
less than 0.03 of the predicted value. In addition, we incorporate two
supplementary metrics: top-1 and top-5 MAE, which represent mean absolute
error focusing exclusively on the top one percent or five percent errors,
respectively, thereby highlighting the performance where the highest errors
are observed.

Table 1: Quantitative results on Fusion4AI test split for hot electron energy
predictions (see §3.1 for details). Refer to the metrics section for details
of CAE, top-1 and top-5 MAE.

Method | CAE$\downarrow$ | top-1 MAE$\downarrow$ | top-5 MAE$\downarrow$
---|---|---|---
PIC Simulation [13] | 2.88 | 0.20 | 0.13
LSTM [25] | 5.82±0.06 | 0.35±0.01 | 0.35±0.01
Autoformer [86] | 5.79±0.04 | 0.35±0.01 | 0.34±0.01
HoGRC [43] | 4.20±0.79 | 0.25±0.05 | 0.22±0.02
RCRK [17] | 4.31±0.46 | 0.28±0.04 | 0.22±0.01
NGRC [20] | 4.28±0.68 | 0.27±0.04 | 0.23±0.02
GPT4TS [94] | 3.34±0.58 | 0.18±0.05 | 0.14±0.04
Time-LLM [31] | 3.48±0.72 | 0.18±0.05 | 0.15±0.05
Fusion-LLM (Llama-2-7B) | 2.15±0.26 | 0.14±0.01 | 0.12±0.01
Fusion-LLM (Llama-3-8B) | 1.90±0.33 | 0.14±0.01 | 0.11±0.01

Performance Comparison. The primary challenge in forecasting for ICF lies in
developing a robust yet efficient predictive model. Table 1 empirically
demonstrates the effectiveness of our system in predicting output energy in
ICF, showcasing a significant advantage over all other baseline methods across
all metrics. Specifically, our approach with Llama-3 surpasses PIC simulation
model [13] by 0.98 in terms of CAE. Additionally, compared to classic AI
methods, our approach outperforms LSTM [25] and Autoformer [86] by 3.92 and
3.89, respectively. Our margins in CAE over three other reservoir computing
methods, HoGRC [43], RCRK [17], NGRC [20], are 2.30, 2.41, and 2.38,
respectively. Moreover, our method outperforms concurrent LLM-based approaches
GPT4TS [94] and Time-LLM [31] by 1.44 and 1.58 on CAE, ${}_{\\!}$respectively,
${}_{\\!}$while ${}_{\\!}$maintaining ${}_{\\!}$comparable ${}_{\\!}$speed.
${}_{\\!}$Detailed ${}_{\\!}$analysis ${}_{\\!}$is ${}_{\\!}$supplemented
${}_{\\!}$in ${}_{\\!}$§S2.

These results underscore the efficacy and efficiency of our LLM-based solution
in predicting hot electron dynamics in ICF. By extending LLM’s successful
adaptability to the new and exciting domain of fusion energy, our empirical
findings represent just the beginning of the innovative opportunities
presented by applying LLM algorithms to challenging subjects in scientific
exploration.

Figure 6: Visualization of confidence score and prediction error. Samples
correspond to Fig. 5.

We further present qualitative results in Fig. 5, aligning with our
quantitative findings that our method surpasses all comparative baselines in
predictive accuracy. Additionally, Fig. 6 illustrates the confidence scores
associated with our predictions. The visualization elucidates a clear
correlation between predictive error and confidence scores, indicating high
confidence corresponding to low errors and conversely. Notably, our approach
consistently demonstrates a heightened level of confidence, particularly in
forecasting peak values across the temporal sequence, a critical phase in ICF.

Table 2: A set of ablative studies. The experiments are evaluted on val split.

Algorithm Component | CAE$\downarrow$
---|---
Baseline | 3.57
$+$ Fusion-Specific Prompt | 2.59
$+$ Signal-Digesting Channels | 2.01
Ours (both) | 1.19

a Key Component Analysis

# of samples | Ours | HoGRC [43] | NGRC [20] | Time-LLM [31]
---|---|---|---|---
80 | 1.19 | 3.80 | 3.73 | 3.60
60 | 1.73 | 3.87 | 3.84 | 3.69
40 | 2.56 | 4.01 | 4.08 | 3.94
20 | 3.47 | 4.35 | 4.19 | 4.10

d Different # of Samples

Prompt Type | CAE$\downarrow$
---|---
Baseline | 2.01
$+$ Discipline-related Prompt | 1.46
$+$ Input Statistics | 1.58
Ours (both) | 1.19

b Fusion-Specific Prompt

# of epochs | Ours | HoGRC [43] | NGRC [20] | Time-LLM [31]
---|---|---|---|---
100 | 1.19 | 3.80 | 3.73 | 3.60
50 | 1.19 | 3.99 | 3.83 | 3.60
20 | 1.56 | 4.12 | 4.35 | 4.01
10 | 2.79 | 5.23 | 5.50 | 4.52

e Different # of Epochs

Algorithm Component | CAE$\downarrow$
---|---
Baseline | 2.59
$+$ Temporal Encoder | 2.47
$+$ Spatial Encoder | 1.41
Ours (both) | 1.19

c Signal-Digesting Channels

Head Dimension | # Params | CAE$\downarrow$ | top-1 MAE$\downarrow$ | top-5 MAE$\downarrow$
---|---|---|---|---
256 | 102.4 K | 1.23 | 0.14 | 0.12
128 | 51.2 K | 1.19 | 0.14 | 0.11
64 | 25.6 K | 1.34 | 0.14 | 0.11
32 | 12.8 K | 1.57 | 0.14 | 0.13

f Different Head Dimension

###  3.2 Diagnostic Experiment

This section ablates Fusion-LLM’s systemic design on val split of Fusion4AI.
All experiments use the Llama 3 8B variant. Appendix §S2 has more experimental
results.

Key Component Analysis. We first investigate the two principal modules of
Fusion-LLM, specifically, Fusion-Specific Prompt and Signal-Digesting
Channels. We construct a baseline model with generic dummy prompts which only
provide broad, non-specific instructions regarding fusion, and a rudimentary
encoder composed of a single linear layer. As shown in Table 2a, the baseline
model achieves 3.57 CAE. Upon applying Fusion-Specific Prompt to the baseline,
we observe significant improvements for CAE from 3.57 to 2.59. Furthermore,
after incorporating Signal-Digesting Channels into the baseline model, we
achieve significant gains of 1.56 CAE. Finally, by integrating both core
techniques, our Fusion-LLM delivers the best performance of 1.19 CAE. These
findings affirm that the proposed Fusion-Specific Prompt and Signal-Digesting
Channels operate synergistically, and validate the effectiveness of our
comprehensive model design.

Fusion-Specific Prompt. We next study the impact of our Fusion-Specific Prompt
by contrasting it with a constructed baseline. This baseline incorporates
Signal-Digesting Channels and employs generic prompts that provide broad,
nonspecific instructions unrelated to the process of fusion. As shown in Table
2b, the baseline yields a performance measure of 2.01 in terms of CAE. Upon
substituting the generic prompt with one that integrates discipline-specific
information, including background knowledge, task instructions, etc, there is
an observable enhancement in performance, achieving an improvement of 0.55 in
CAE over the baseline. Additionally, a further analysis involving the
integration of input statistics, containing the maximum and minimum values,
etc, of the input time series, demonstrates superior performance,
outperforming the baseline by 0.43 in CAE. The most notable enhancement is
recorded when employing our Fusion-Specific Prompt, which amalgamates both the
discipline-related information and input statistics, culminating in a peak
performance of 1.19 CAE. This outcome highlights the essential function of the
Fusion-Specific Prompt within our approach, significantly impacting the
performance of the overall model.

Signal-Digesting Channels. We then examine the influence of Signal-Digesting
Channels in Table 2c. For the baseline, we use a basic approach comprising
solely a single linear layer. Under this setting, the baseline model achieves
2.59 in terms of CAE. Integration of either the Temporal Encoder or the
Spatial Encoder independently results in performance improvements of 0.12 and
1.18 above the baseline respectively. Conversely, the integration of both
Encoders in SDC substantially surpasses all alternative counterparts,
achieving a CAE of 1.19. These results substantiate the hypothesis that the
proposed Signal-Digesting Channels augment the capability of our approach to
more accurately interpret input time series data.

Reservoir and LLM Comparison. To thoroughly explore the training effectiveness
of our methodology under conditions of limited sample availability, we perform
a comparative analysis using a variable number of training samples with two
concurrent reservoir methods [43, 20] and one LLM-based method [31] in Table
2d using CAE. The empirical findings demonstrate that our approach
consistently outperforms all competing strategies across various sample
configurations. Notably, this superior performance and training effectiveness
are evident even with as few as 20 samples. Such robust efficacy is critical
in scientific AI applications, where datasets are often constrained in size.

In addition, we study the training efficiency of our approach in contrast to
the above strong baselines [43, 20, 31] in Table 2e, across various training
epochs. The experimental outcomes illustrate that our approach not only
outperforms its counterparts but also demonstrates superior efficiency.
Specifically, our method is capable of achieving comparable or superior
performance in a significantly reduced training duration. For instance, our
model requires only 10 epochs to achieve better performance than other methods
that require 100 epochs. This enhanced efficiency in training is particularly
significant, as it demonstrates the potential of our approach to deliver
robust performance swiftly, thereby facilitating more expedient research in
practical scenarios.

Prediction Head Dimension. Lastly, we conduct additional experiments to
evaluate the impact of varying dimensions in the prediction head. As shown in
Table 2f, our approach demonstrates an enhancement in CAE, reducing from 1.57
to 1.34, concomitant upon augmenting the head dimension from 32 to 64. This
improvement continues, culminating in a CAE of 1.19 at a head dimension of
128, where it stabilizes, indicating this as the optimal head dimension for
balancing effectiveness and parameter-efficiency. We therefore select the
dimension of 128 as the default setting.

##  4 Related Work and Discussion

AI ${}_{\\!}$for ${}_{\\!}$Science. ${}_{\\!}$AI ${}_{\\!}$is
${}_{\\!}$increasingly ${}_{\\!}$acknowledged ${}_{\\!}$as ${}_{\\!}$a
${}_{\\!}$critical ${}_{\\!}$instrument and recently ${}_{\\!}$led trending of
${}_{\\!}$significant scientific discovery ${}_{\\!}$[32, 40, 55, 5].
${}_{\\!}$The ${}_{\\!}$trajectory ${}_{\\!}$of ${}_{\\!}$AI in
${}_{\\!}$scientific ${}_{\\!}$research ${}_{\\!}$commenced ${}_{\\!}$with
${}_{\\!}$different ${}_{\\!}$elementary data analysis methods (e.g., Rule-
based [11, 60], Bayesian [19], Analogy [24, 30, 76], Evolutionary [35, 16],
Connectionism [83, 40], etc) and has evolved into the advanced foundation
models [23, 81, 15, 18, 10]. This development is underscored by considerable
progress in AI recently, particularly the emergence of sophisticated
constructs such as LLMs [2, 78, 75, 95], which have fundamentally altered our
approach to scientific challenges. By incorporating expansive knowledge bases,
AI is indispensable for deciphering vast and varied scientific data.

The uniqueness of our work lies in the challenging nature of accessing ICF
data, given the difficulty in observing LPI, which renders conventional AI
tools less effective in this crucial area of physics. Building on the success
of LLMs, this study harnesses their extensive knowledge base to unlock robust
predictive capabilities for physical phenomena. Our empirical findings
validate our methodological approach, positioning our work as pioneering and
innovative. This not only demonstrates the applicability of LLMs in the
scientific domain but also fosters a dynamic interplay between AI and
scientific exploration. We showcase the potential of our LLM-based solution to
make meaningful contributions in addressing complex scientific challenges,
thus advancing both AI and science.

Plasma Physics for Fusion. In the realm of plasma physics, particularly in
ICF, efficient and symmetric driving of a target is imperative. Specifically,
LPI represents significant impediments due to the complex dynamics of
phenomena in fusion. For instance, Stimulated Raman and Brillouin Scattering
(SRS and SBS) can cause the reflection of laser beams [37]. Moreover, Cross-
Beam Energy Transfer [28] may adversely influence the symmetry of implosion .
A critical issue particularly in direct drive scenarios is the preheating
caused by hot electrons produced via SRS and Two-Plasmon Decay, which
potentially increases the shell entropy and diminishes the implosion
efficiency [68, 21, 14, 54]. Conversely, these hot electrons might contribute
positively by depositing their energy in the compressed shell, thereby
enhancing the ignition shock and aiding ignition processes such as those
observed in shock ignition, a novel high-gain strategy [9, 53]. Understanding
the dynamics ofLPI and establishing predictive models for hot electron
generation in direct drive configurations are crucial.D In response to these
issues and challenges, extensive physical experimentation and simulations [68,
21, 14, 54, 9, 53] are necessary, which are both costly and time-consuming.

To address the above challenges, our AI-based approach emerges as a potential
alternative. By reformulating LLMs with a simple yet effective pipeline, our
model can effectively engage in such ICF tasks as Computational Reservoir to
encapsulate domain-specific knowledge and generalize within the domain. This
capability potentially allows LLMs to guide ICF design, circumventing the need
for extensive and expensive experimental setups and simulations. Empirical
evidence indicates that the incorporation of LLMs had promising to
revolutionize the predictive modeling in plasma physics by providing quick,
cost-effective insights that are grounded in generalizable knowledge.

Reservior Computing. The ICF data exhibits a time-series nature, where laser
intensity influences the target, correlating with the volume of hot electron
energy emission. Traditional machine learning approaches [72, 7, 3, 84, 61,
62, 33] for time-series data typically involve dynamically transforming
temporal inputs into a high-dimensional state space through nonlinear
mappings. Revolutionary advancements in the field, such as echo state networks
[29, 47] and liquid state machines [49, 92], have introduced the concept of
Reservoir Computing (RC). RC presents an innovative framework [51, 63, 20, 48]
characterized by a static “reservoir”. This reservoir is pre-trained, reducing
the computational expenses associated with training only the readout
parameters, rather than the entire network. This feature makes RC advantageous
for applications involving temporal data [48, 82]. In this paradigm, the input
signals are transformed non-linearly into a dynamic time-series system’s state
space, from which outputs are linearly extracted [74].

In contrast to traditional reservoirs (e.g., RNN-based [20, 48]) that often
struggle with processing highly dynamic temporal data, and contemporary time-
series Large Language Models (LLMs) [79, 4, 31, 94] that employ massive
parameters but focus solely on temporal sequences, often neglecting the vast
knowledge embedded within LLMs, we propose a re-imagining of LLMs within the
RC paradigm. This innovative integration not only leverages extensive domain-
specific knowledge and adept handling of temporal data but also reduces the
need for extensive meta-parameter tuning by maintaining most parameters as
static during training. Consequently, our approach emerges as a compelling
alternative, effectively addressing the challenges posed by both traditional
RNN-based reservoirs and modern time-series LLMs, showcasing superior
capability in handling intricate and large-scale temporal patterns.

##  5 Conclusion

Fusion energy stands as a pivotal pathway toward advancing human civilization
to a Type I status on the Kardashev scale [34]. The key to realizing this
potential lies in mastering Inertial Confinement Fusion, where understanding
laser-plasma instabilities is paramount. To address this challenge, we present
Fusion-LLM, a groundbreaking framework merging LLMs with reservoir computing.
Our approach not only provides a cost-effective solution but also emerges as a
top-tier contender in forecasting hot electron dynamics, offering invaluable
insights for plasma scientists in ${}_{\\!}$refining ${}_{\\!}$ICF
${}_{\\!}$designs. ${}_{\\!}$Beyond ${}_{\\!}$its ${}_{\\!}$immediate
${}_{\\!}$impact ${}_{\\!}$on ${}_{\\!}$ICF, ${}_{\\!}$employing
${}_{\\!}$LLMs for ${}_{\\!}$scientific ${}_{\\!}$exploration ${}_{\\!}$holds
${}_{\\!}$promise ${}_{\\!}$for ${}_{\\!}$${}_{\\!}$cross-domain
${}_{\\!}$applications, ${}_{\\!}$potentially ${}_{\\!}$catalyzing
${}_{\\!}$advancements ${}_{\\!}$in ${}_{\\!}$AI-driven ${}_{\\!}$scientific
${}_{\\!}$endeavors.

## References

  * [1] Abu-Shawareb, H., Acree, R., Adams, P., Adams, J., Addis, B., Aden, R., Adrian, P., Afeyan, B., Aggleton, M., Aghaian, L., et al.: Achievement of target gain larger than unity in an inertial fusion experiment. Physical Review Letters 132(6), 065102 (2024)
  * [2] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
  * [3] Ahmed, N.K., Atiya, A.F., Gayar, N.E., El-Shishiny, H.: An empirical comparison of machine learning models for time series forecasting. Econometric Reviews 29(5-6), 594–621 (2010)
  * [4] AI, M.: Llama 3 model card. Meta (2024), https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md
  * [5] Ali, S., Ravi, P., Williams, R., DiPaola, D., Breazeal, C.: Constructing dreams using generative ai. In: AAAI (2024)
  * [6] Anthropic: The claude 3 model family: Opus, sonnet, haiku. Anthropic (2024), https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf
  * [7] Arel, I., Rose, D.C., Karnowski, T.P.: Deep machine learning-a new frontier in artificial intelligence research [research frontier]. IEEE Computational Intelligence Magazine 5(4), 13–18 (2010)
  * [8] Betti, R., Hurricane, O.: Inertial-confinement fusion with lasers. Nature Physics 12, 435–448 (2016)
  * [9] Betti, R., Zhou, C.D., Anderson, K.S., Perkins, L.J., Theobald, W., Solodov, A.A.: Shock ignition of thermonuclear fuel with high areal density. Physical Review Letters 98, 155001 (2007)
  * [10] Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E., et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)
  * [11] Breiman, L.: Random forests. Machine Learning 45, 5–32 (2001)
  * [12] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. In: NeurIPS (2020)
  * [13] Cao, S., Patel, D., Lees, A., Stoeckl, C., Rosenberg, M., Gopalaswamy, V., Wen, H., Huang, H., Shvydky, A., Betti, R., et al.: Predicting hot electron generation in inertial confinement fusion with particle-in-cell simulations. Physical Review E 106(5), 055214 (2022)
  * [14] Craxton, R.S., Anderson, K.S., Boehly, T.R., Goncharov, V.N., Harding, D.R., Knauer, J.P., McCrory, R.L., McKenty, P.W., Meyerhofer, D.D., Myatt, J.F., Schmitt, A.J., Sethian, J.D., Short, R.W., Skupsky, S., Theobald, W., Kruer, W.L., Tanaka, K., Betti, R., Collins, T.J.B., Delettrez, J.A., Hu, S.X., Marozas, J.A., Maximov, A.V., Michel, D.T., Radha, P.B., Regan, S.P., Sangster, T.C., Seka, W., Solodov, A.A., Soures, J.M., Stoeckl, C., Zuegel, J.D.: Direct-drive inertial confinement fusion: A review. Physics of Plasmas 22(11), 110501 (2015)
  * [15] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [16] Dietterich, T.G.: Ensemble methods in machine learning. In: International Workshop on Multiple Classifier Systems. pp. 1–15. Springer (2000)
  * [17] Dong, J., Ohana, R., Rafayelyan, M., Krzakala, F.: Reservoir computing meets recurrent kernels and structured transforms. In: NeurIPS (2020)
  * [18] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)
  * [19] Frank, E., Trigg, L., Holmes, G., Witten, I.H.: Naive bayes for regression. Machine Learning 41, 5–25 (2000)
  * [20] Gauthier, D.J., Bollt, E., Griffith, A., Barbosa, W.A.: Next generation reservoir computing. Nature Communications 12(1), 1–8 (2021)
  * [21] Goncharov, V.N., Sangster, T.C., Radha, P.B., Betti, R., Boehly, T.R., Collins, T.J.B., Craxton, R.S., Delettrez, J.A., Epstein, R., Glebov, V.Y., Hu, S.X., Igumenshchev, I.V., Knauer, J.P., Loucks, S.J., Marozas, J.A., Marshall, F.J., McCrory, R.L., McKenty, P.W., Meyerhofer, D.D., Regan, S.P., Seka, W., Skupsky, S., Smalyuk, V.A., Soures, J.M., Stoeckl, C., Shvarts, D., Frenje, J.A., Petrasso, R.D., Li, C.K., Seguin, F., Manheimer, W., Colombant, D.G.: Performance of direct-drive cryogenic targets on omega. Physics of Plasmas 15(5), 056310 (2008)
  * [22] Gopalaswamy, V., Williams, C., Betti, R., Patel, D., Knauer, J., Lees, A., Cao, D., Campbell, E., Farmakis, P., Ejaz, R., et al.: Demonstration of a hydrodynamically equivalent burning plasma in direct-drive inertial confinement fusion. Nature Physics pp. 1–7 (2024)
  * [23] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
  * [24] Hearst, M.A., Dumais, S.T., Osuna, E., Platt, J., Scholkopf, B.: Support vector machines. IEEE Intelligent Systems and their Applications 13(4), 18–28 (1998)
  * [25] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation 9(8), 1735–1780 (1997)
  * [26] Huang, W., Abbeel, P., Pathak, D., Mordatch, I.: Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In: ICML (2022)
  * [27] Huang, Y., Song, J., Wang, Z., Chen, H., Ma, L.: Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv preprint arXiv:2307.10236 (2023)
  * [28] Igumenshchev, I.V., Edgell, D.H., Goncharov, V.N., Delettrez, J.A., Maximov, A.V., Myatt, J.F., Seka, W., Shvydky, A., Skupsky, S., Stoeckl, C.: Crossed-beam energy transfer in implosion experiments on omega. Physics of Plasmas 17(12), 122708 (2010)
  * [29] Jaeger, H.: Echo state network. scholarpedia 2(9), 2330 (2007)
  * [30] Jain, A.K., Murty, M.N., Flynn, P.J.: Data clustering: a review. ACM Computing Surveys 31(3), 264–323 (1999)
  * [31] Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J.Y., Shi, X., Chen, P.Y., Liang, Y., Li, Y.F., Pan, S., et al.: Time-llm: Time series forecasting by reprogramming large language models. In: ICLR (2023)
  * [32] Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al.: Highly accurate protein structure prediction with alphafold. Nature 596(7873), 583–589 (2021)
  * [33] Kadous, M.W.: Learning comprehensible descriptions of multivariate time series. In: ICML (1999)
  * [34] Kardashev, N.S.: Transmission of Information by Extraterrestrial Civilizations. Soviet Astronomy 8, 217 (1964)
  * [35] Kennedy, J., Eberhart, R.: Particle swarm optimization. In: ICNN (1995)
  * [36] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015)
  * [37] Kirkwood, R.K., Montgomery, D.S., Afeyan, B.B., Moody, J.D., MacGowan, B.J., Joshi, C., Wharton, K.B., Glenzer, S.H., Williams, E.A., Young, P.E., Kruer, W.L., Estabrook, K.G., Berger, R.L.: Observation of the nonlinear saturation of langmuir waves driven by ponderomotive force in a large scale plasma. Physical Review Letters 83, 2965–2968 (1999)
  * [38] Klimo, O., Weber, S., Tikhonchuk, V.T., Limpouch, J.: Particle-in-cell simulations of laser–plasma interaction for the shock ignition scenario. Plasma Physics and Controlled Fusion 52(5), 055013 (2010)
  * [39] Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. In: NeurIPS (2022)
  * [40] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
  * [41] Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., Gao, J.: Llava-med: Training a large language-and-vision assistant for biomedicine in one day. In: NeurIPS (2023)
  * [42] Li, J., Zhang, S., Krauland, C.M., Wen, H., Beg, F.N., Ren, C., Wei, M.S.: Pump depletion and hot-electron generation in long-density-scale-length plasma with shock-ignition high-intensity laser. Physical Review E 101, 033206 (2020)
  * [43] Li, X., Zhu, Q., Zhao, C., Duan, X., Zhao, B., Zhang, X., Ma, H., Sun, J., Lin, W.: Higher-order granger reservoir computing: simultaneously achieving scalable complex structures inference and accurate dynamics prediction. Nature Communications 15(1), 2506 (2024)
  * [44] Lin, B., Zhu, Y., Chen, Z., Liang, X., Liu, J., Liang, X.: ADAPT: vision-language navigation with modality-aligned action prompts. In: CVPR (2022)
  * [45] Lin, Z., Trivedi, S., Sun, J.: Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187 (2023)
  * [46] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: NeurIPS (2023)
  * [47] Lukoševičius, M.: A practical guide to applying echo state networks. In: Neural Networks: Tricks of the Trade: Second Edition, pp. 659–686. Springer (2012)
  * [48] Lukoševičius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Computer Science Review 3(3), 127–149 (2009)
  * [49] Maass, W.: Liquid state machines: motivation, theory, and applications. Computability in context: computation and logic in the real world pp. 275–296 (2011)
  * [50] Moor, M., Banerjee, O., Abad, Z.S.H., Krumholz, H.M., Leskovec, J., Topol, E.J., Rajpurkar, P.: Foundation models for generalist medical artificial intelligence. Nature 616(7956), 259–265 (2023)
  * [51] Nakajima, K., Fischer, I.: Reservoir computing. Springer (2021)
  * [52] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. In: NeurIPS (2019)
  * [53] Perkins, L.J., Betti, R., LaFortune, K.N., Williams, W.H.: Shock ignition: A new approach to high gain inertial confinement fusion on the national ignition facility. Physical Review Letters 103, 045004 (2009)
  * [54] Radha, P., Hohenberger, M., Edgell, D., Marozas, J., Marshall, F., Michel, D., Rosenberg, M., Seka, W., Shvydky, A., Boehly, T., et al.: Direct drive: Simulations and results from the national ignition facility. Physics of Plasmas 23(5), 056305 (2016)
  * [55] Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., Prabhat, f.: Deep learning and process understanding for data-driven earth system science. Nature 566(7743), 195–204 (2019)
  * [56] Rezayi, S., Liu, Z., Wu, Z., Dhakal, C., Ge, B., Zhen, C., Liu, T., Li, S.: Agribert: Knowledge-infused agricultural language models for matching food and nutrition. In: IJCAI (2022)
  * [57] Riconda, C., Weber, S., Tikhonchuk, V., Héron, A.: Kinetic simulations of stimulated raman backscattering and related processes for the shock-ignition approach to inertial confinement fusion. Physics of Plasmas 18(9), 092701 (2011)
  * [58] Rosenberg, M., Solodov, A., Myatt, J., Seka, W., Michel, P., Hohenberger, M., Short, R., Epstein, R., Regan, S., Campbell, E., et al.: Origins and scaling of hot-electron preheat in ignition-scale direct-drive inertial confinement fusion experiments. Physical Review Letters 120(5), 055001 (2018)
  * [59] Rosenberg, M., Solodov, A., Seka, W., Follett, R., Myatt, J., Maximov, A., Ren, C., Cao, S., Michel, P., Hohenberger, M., et al.: Stimulated raman scattering mechanisms and scaling behavior in planar direct-drive experiments at the national ignition facility. Physics of Plasmas 27(4) (2020)
  * [60] Safavian, S.R., Landgrebe, D.: A survey of decision tree classifier methodology. IEEE Transactions on Systems, Man, and Cybernetics 21(3), 660–674 (1991)
  * [61] Samuel, A.L.: Some studies in machine learning using the game of checkers. IBM Journal of Research and Development 3(3), 210–229 (1959)
  * [62] Sapankevych, N.I., Sankar, R.: Time series prediction using support vector machines: a survey. IEEE Computational Intelligence Magazine 4(2), 24–38 (2009)
  * [63] Schrauwen, B., Verstraeten, D., Van Campenhout, J.: An overview of reservoir computing: theory, applications and implementations. In: European Symposium on Artificial Neural Networks. pp. 471–482 (2007)
  * [64] Sclar, M., Choi, Y., Tsvetkov, Y., Suhr, A.: Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324 (2023)
  * [65] Sclar, M., Choi, Y., Tsvetkov, Y., Suhr, A.: Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In: ICLR (2024)
  * [66] Shang, W.L., Betti, R., Hu, S.X., Woo, K., Hao, L., Ren, C., Christopherson, A.R., Bose, A., Theobald, W.: Electron shock ignition of inertial fusion targets. Physical Review Letters 119, 195001 (2017)
  * [67] Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., Wei, J., Chung, H.W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al.: Large language models encode clinical knowledge. Nature 620(7972), 172–180 (2023)
  * [68] Smalyuk, V.A., Shvarts, D., Betti, R., Delettrez, J.A., Edgell, D.H., Glebov, V.Y., Goncharov, V.N., McCrory, R.L., Meyerhofer, D.D., Radha, P.B., Regan, S.P., Sangster, T.C., Seka, W., Skupsky, S., Stoeckl, C., Yaakobi, B., Frenje, J.A., Li, C.K., Petrasso, R.D., Séguin, F.H.: Role of hot-electron preheating in the compression of direct-drive imploding targets with cryogenic ${\mathrm{d}}_{2}$ ablators. Physical Review Letters 100, 185005 (2008)
  * [69] Smalyuk, V., Shvarts, D., Betti, R., Delettrez, J., Edgell, D., Glebov, V.Y., Goncharov, V., McCrory, R., Meyerhofer, D., Radha, P., et al.: Role of hot-electron preheating in the compression of direct-drive imploding targets with cryogenic d 2 ablators. Physical Review Letters 100(18), 185005 (2008)
  * [70] Solodov, A., Rosenberg, M., Seka, W., Myatt, J., Hohenberger, M., Epstein, R., Stoeckl, C., Short, R., Regan, S., Michel, P., et al.: Hot-electron generation at direct-drive ignition-relevant plasma conditions at the national ignition facility. Physics of Plasmas 27(5) (2020)
  * [71] Stoeckl, C., Bahr, R., Yaakobi, B., Seka, W., Regan, S., Craxton, R., Delettrez, J., Short, R., Myatt, J., Maximov, A., et al.: Multibeam effects on fast-electron generation from two-plasmon-decay instability. Physical Review Letters 90(23), 235002 (2003)
  * [72] Sutton, R.S.: Learning to predict by the methods of temporal differences. Machine Learning 3, 9–44 (1988)
  * [73] Szot, A., Clegg, A., Undersander, E., Wijmans, E., Zhao, Y., Turner, J.M., Maestre, N.D., Mukadam, M., Chaplot, D.S., Maksymets, O., Gokaslan, A., Vondruš, V., Dharur, S., Meier, F., Galuba, W., Chang, A.X., Kira, Z., Koltun, V., Malik, J., Savva, M., Batra, D.: Habitat 2.0: Training home assistants to rearrange their habitat. In: NeurIPS (2021)
  * [74] Tanaka, G., Yamane, T., Héroux, J.B., Nakane, R., Kanazawa, N., Takeda, S., Numata, H., Nakano, D., Hirose, A.: Recent advances in physical reservoir computing: A review. Neural Networks 115, 100–123 (2019)
  * [75] Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)
  * [76] Tenenbaum, J.B., Silva, V.d., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290(5500), 2319–2323 (2000)
  * [77] Thirunavukarasu, A.J., Ting, D.S.J., Elangovan, K., Gutierrez, L., Tan, T.F., Ting, D.S.W.: Large language models in medicine. Nature Medicine 29(8), 1930–1940 (2023)
  * [78] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
  * [79] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P.S., Lachaux, M., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E.M., Subramanian, R., Tan, X.E., Tang, B., Taylor, R., Williams, A., Kuan, J.X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., Scialom, T.: Llama 2: Open foundation and fine-tuned chat models. CoRR (2023)
  * [80] Tzachor, A., Devare, M., Richards, C., Pypers, P., Ghosh, A., Koo, J., Johal, S., King, B.: Large language models and agricultural extension services. Nature Food 4(11), 941–948 (2023)
  * [81] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017)
  * [82] Verstraeten, D., Schrauwen, B., d’Haene, M., Stroobandt, D.: An experimental unification of reservoir computing methods. Neural Networks 20(3), 391–403 (2007)
  * [83] Weisberg, S.: Applied linear regression, vol. 528. John Wiley & Sons (2005)
  * [84] Williams, C.K., Rasmussen, C.E.: Gaussian processes for machine learning. MIT press Cambridge, MA (2006)
  * [85] Woo, G., Liu, C., Kumar, A., Xiong, C., Savarese, S., Sahoo, D.: Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592 (2024)
  * [86] Wu, H., Xu, J., Wang, J., Long, M.: Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting. In: NeurIPS (2021)
  * [87] Yaakobi, B., Stoeckl, C., Seka, W., Delettrez, J., Sangster, T., Meyerhofer, D.: Measurement of preheat due to fast electrons in laser implosions of cryogenic deuterium targets. Physics of Plasmas 12(6) (2005)
  * [88] Yaakobi, B., Stoeckl, C., Boehly, T., Meyerhofer, D., Seka, W.: Measurement of preheat due to fast electrons in laser implosions. Physics of Plasmas 7(9), 3714–3720 (2000)
  * [89] Yan, R., Li, J., Ren, C.: Intermittent laser-plasma interactions and hot electron generation in shock ignition. Physics of Plasmas 21(6), 062705 (2014)
  * [90] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K.R., Cao, Y.: React: Synergizing reasoning and acting in language models. In: ICLR (2023)
  * [91] Zhang, A., Yang, J., Luo, Y., Fan, S.: Forecasting the progression of human civilization on the kardashev scale through 2060 with a machine learning approach. Scientific Reports 13(1), 11305 (2023)
  * [92] Zhang, Y., Li, P., Jin, Y., Choe, Y.: A digital liquid state machine with biologically inspired learning and its application to speech recognition. IEEE Transactions on Neural Networks and Learning Systems 26(11), 2635–2649 (2015)
  * [93] Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E.P., Zhang, H., Gonzalez, J.E., Stoica, I.: Judging llm-as-a-judge with mt-bench and chatbot arena. In: NeurIPS (2023)
  * [94] Zhou, T., Niu, P., Sun, L., Jin, R., et al.: One fits all: Power general time series analysis by pretrained lm. In: NeurIPS (2023)
  * [95] Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023)

SUMMARY OF THE APPENDIX

This supplementary contains additional experimental results and discussions of
our NeurIPS 2024 submission: Inertial Confinement Fusion Forecasting via LLMs,
organized as follows:

  * •

§S1 provides Implementation Details.

  * •

§S2 reports more Quantitative Results with Runtime Analysis.

  * •

§S3 shows more Qualitative Results.

  * •

§S4 analyzes Failure Case.

  * •

§S5 conducts Confidence Analysis.

  * •

§S6 discusses the Social Impact & Limitation of our research.

  * •

§S7 offers Ethical Guard or our dataset.

  * •

§S8 claims Reproducibility of our approach.

  * •

§S9 supplies Data License for the methods we used for comparison.

##  Appendix S1 Implementation Details

The overall pipeline of Fusion-LLM is shown in Fig. 3. Experiments are
conducted on two NVIDIA A100-40GB GPUs. For our approach, we keep all
parameters of the LLMs and most of the SDC frozen during the fine-tuning. Only
parameters pertaining to the Prediction Head and partial Spatial Encoder are
trainable. The codes and dataset shall be publicly released upon paper
acceptance.

  * •

Fusion-LLM is built from Llama-2-7B-hf / 3-8B [78, 4] to construct reservoir
without tuning.

  * •

Fusion-specific prompts structure the textual prompts with three descriptors:
context descriptor, task descriptor, and input descriptor. Each descriptor is
initialized with specialized tokens for indication (e.g.,
$<|$begin_of_text$|>$, $<|$eot_id$|>$, $<|$start_header_id$|>$, etc) and input
scalars as context descriptions (e.g., $<$seq_len$>$, $<$pred_len$>$,
$<$phase_plate$>$, etc). These prompts are subsequently concatenated and input
into the projection layer from LLMs for feature embedding.

  * •

Signal-digesting channels are composed of two components, temporal encoder and
spatial encoder. The ${}_{\\!}$former ${}_{\\!}$one, ${}_{\\!}$which
incorporates 24 Transformer layers and a linear layer, captures temporal
features ${}_{\\!}$over ${}_{\\!}$the ${}_{\\!}$input ${}_{\\!}$laser
${}_{\\!}$signal. ${}_{\\!}$This ${}_{\\!}$module ${}_{\\!}$has ${}_{\\!}$been
${}_{\\!}$pre-trained ${}_{\\!}$on ${}_{\\!}$the ${}_{\\!}$Large-scale
${}_{\\!}$Open ${}_{\\!}$Time ${}_{\\!}$Series ${}_{\\!}$Archive
${}_{\\!}$(LOTSA) ${}_{\\!}$dataset ${}_{\\!}$[85], ${}_{\\!}$which
${}_{\\!}$covers ${}_{\\!}$nine ${}_{\\!}$varied ${}_{\\!}$domains
${}_{\\!}$and ${}_{\\!}$compiles ${}_{\\!}$over ${}_{\\!}$27 ${}_{\\!}$billion
${}_{\\!}$timestamped ${}_{\\!}$instances. ${}_{\\!}$The ${}_{\\!}$spatial
${}_{\\!}$decoder ${}_{\\!}$first ${}_{\\!}$uses ${}_{\\!}$a
${}_{\\!}$projection ${}_{\\!}$block ${}_{\\!}$from ${}_{\\!}$Llama
${}_{\\!}$[78, 4] ${}_{\\!}$to ${}_{\\!}$encode ${}_{\\!}$the
${}_{\\!}$context ${}_{\\!}$description ${}_{\\!}$of ${}_{\\!}$the
${}_{\\!}$input ${}_{\\!}$signal, ${}_{\\!}$followed ${}_{\\!}$by ${}_{\\!}$a
${}_{\\!}$leaner ${}_{\\!}$transformation. ${}_{\\!}$Outputs ${}_{\\!}$are
${}_{\\!}$fed ${}_{\\!}$into ${}_{\\!}$a ${}_{\\!}$cross-attention
${}_{\\!}$layer, ${}_{\\!}$where ${}_{\\!}$Key ${}_{\\!}$and ${}_{\\!}$Value
${}_{\\!}$are ${}_{\\!}$derived ${}_{\\!}$from ${}_{\\!}$the
${}_{\\!}$contextual ${}_{\\!}$embedding ${}_{\\!}$and ${}_{\\!}$query
${}_{\\!}$stems ${}_{\\!}$from ${}_{\\!}$temporal ${}_{\\!}$features,
${}_{\\!}$to ${}_{\\!}$generate ${}_{\\!}$the ${}_{\\!}$final
${}_{\\!}$spatial ${}_{\\!}$features. ${}_{\\!}$We concatenate ${}_{\\!}$the
${}_{\\!}$spatial ${}_{\\!}$and ${}_{\\!}$temporal ${}_{\\!}$features
${}_{\\!}$before ${}_{\\!}$feeding ${}_{\\!}$them ${}_{\\!}$into ${}_{\\!}$a
${}_{\\!}$linear ${}_{\\!}$layer ${}_{\\!}$to ${}_{\\!}$produce ${}_{\\!}$the
${}_{\\!}$final, ${}_{\\!}$augmented ${}_{\\!}$input ${}_{\\!}$signals.

  * •

Confidence scanner has been described in §2.2.3 and it has no consumption of
parameters. The default number of tokens $k$ used in confidence calculation is
set to 50 in the implementation.

  * •

Prediction head consists of two layers: a convolution layer with the kernel
size of 32 and stride of 32, followed by batch normalization and GELU
activation, connected to LLM, then fed to a linear layer with the input
dimension of 128 that produces the final prediction.

##  Appendix S2 Quantitative Results

This section elaborates on a detailed analysis of quantitative results in
Table S2, focusing specifically on in-context learning [12] performance and a
runtime assessment of the models under investigation. Initially, we present
supplementary in-context learning results obtained directly from various LLMs
(i.e., Llama 2 [79], Llama 3 [4], and Claude 3 Opus [6]). These findings
indicate that, even without an additional fine-tuning process, the LLMs
exhibit substantial proficiency in the in-context learning scheme within the
ICF task. For instance, Claude 3 Opus [6] achieves CAE scores of 12.19, 10.67,
and 9.46 for the 1-shot, 2-shot, and 3-shot scenarios, respectively. It is
amply demonstrated that the vanilla LLM has the ability to make inferences and
predictions on empirical scientific data even if it is not fine-tuned at all.
This underscores that our approach, leveraging these LLMs, represents a
notable advancement, particularly in forecasting the energy dynamics of hot
electrons.

Furthermore, it is pertinent to emphasize the economic and operational
advantages of our computational approach over traditional physical
experiments. Specifically, conducting a single ICF experiment typically incurs
costs upwards of one million US dollars. Conversely, computational simulations
such as a $150\mu m$ PIC simulation [13] require extensive computational
resources, amounting to the utilization of 19,584 cores of CPU over a period
of 10 hours. In stark contrast, our model necessitates significantly less
computational time and resources, requiring only 30 minutes on 2 NVIDIA A100
GPUs for training and only 3~4 seconds for inference with much higher
predictive accuracy compared to the PIC simulation. This comparison not only
underscores the cost-effectiveness of our approach but also its efficiency and
practicality in other scientific applications where computational resource
constraints are a critical factor.

Table S1: Quantitative results on Fusion4AI test split for hot electron energy
forecasting (see §3.1 for details). Train Time refer to training for the
designated task, and Infer Time refer to the amount of time used to predict
one case. Note that 3-shot experiments could not be performed on Llama series
of models due to the limitation of context window.
Method | # Params | Train Time | Infer Time | CAE$\downarrow$ | top-1 MAE$\downarrow$ | top-5 MAE$\downarrow$
---|---|---|---|---|---|---
PIC Simulation [13] | - | - |  $>10$ hrs | 2.88 | 0.20 | 0.13
LSTM [25] | 81.6K | ~5 mins |  $<1$ s | 5.82 | 0.35 | 0.35
Autoformer [86] | 120.4K | ~8 mins |  $<1$ s | 5.79 | 0.35 | 0.34
GPT4TS [94] | 1.5 B | ~22 mins | ~2 s | 3.34 | 0.18 | 0.14
Time-LLM [31] | 7 B | ~20 mins | ~3 s | 3.48 | 0.18 | 0.15
Llama 2 (1-Shot) [79] | 7 B | - | ~3 s | 471.22 | 15.80 | 14.92
Llama 2 (2-Shot) [79] | 7 B | - | ~5 s | 30.33 | 0.95 | 0.91
Llama 2 (1-Shot) [79] | 70 B | - | ~7 s | 20.63 | 0.64 | 0.63
Llama 2 (2-Shot) [79] | 70 B | - | ~8 s | 16.42 | 0.51 | 0.50
Llama 3 (1-Shot) [4] | 8 B | - | ~4 s | 583.14 | 17.70 | 16.97
Llama 3 (2-Shot) [4] | 8 B | - | ~6 s | 26.66 | 0.83 | 0.81
Llama 3 (1-Shot) [4] | 70 B | - | ~14 s | 72.35 | 1.02 | 1.15
Llama 3 (2-Shot) [4] | 70 B | - | ~19 s | 13.62 | 0.40 | 0.39
Claude 3 Opus (1-Shot) [6] | 137 B | - | ~12 s | 12.19 | 0.39 | 0.39
Claude 3 Opus (2-Shot) [6] | 137 B | - | ~17 s | 10.67 | 0.38 | 0.37
Claude 3 Opus (3-Shot) [6] | 137 B | - | ~20 s | 9.46 | 0.37 | 0.36
RCRK [17] | 106K | ~2 mins |  $<1$ s | 4.31 | 0.28 | 0.22
HoGRC [43] | 394K | ~4 mins |  $<1$ s | 4.20 | 0.25 | 0.22
NGRC [20] | 157K | ~2 mins |  $<1$ s | 4.28 | 0.27 | 0.23
Fusion-LLM (Llama-2) | 7B | ~30 mins | ~3 s | 2.15 | 0.14 | 0.12
Fusion-LLM (Llama-3) | 8B | ~30 mins | ~4 s | 1.90 | 0.14 | 0.11
Figure S1: Predictions of LLMs with In-Context Learning. We plot Ground Truth
and the predictions of Claude 3 Opus (3-shot) and Llama 3 70B (2-shot) with
the comparison of trained methods LSTM and Autoformer. Y and X axes denote
energy and time steps.

##  Appendix S3 More Qualitative Result

This section expands to include more qualitative results that help to
understand the capabilities and effectiveness of this model. Initially, we
release all visualized prediction results of our model Fusion-LLM on test
split of our dataset Fusion4AI in Fig. S2. From these qualitative results, it
can be found that our model achieves accurate predictions on all unseen data,
especially conforming to the temporal and spatial characteristics of the
predicted targets, which is crucial for physicists to apply our model as a
tool in the design of real-world ICF ignitions.

Figure S2: Visualization of hot electron prediction in test split. We plot
Ground Truth and the predictions of Ours, Time-LLM, LSTM and Autoformer. Y and
X axes denote energy and time steps, respectively.

Recall that we describe in §S2, the direct prediction of ICF tasks by vanilla
LLM use in-context learning method without fine-tuning on our data is not as
good as fine-tuned methods in terms of the quantitative value of metric, but
it can in fact predict more meaningful patterns than traditional methods such
as LSTM [25] and Autoformer [86]. As illustrated in Figure S1, the LLM without
fine-tune can infer approximate predictions with reference to the 1 to 3
examples provided, as compared to LSTM [25] and Autoformer [86] which can only
predict straight lines close to 0 with patterns. This proves that the vanilla
LLMs themselves already contain the capability to infer specific empirical
scientific data, which is the core reason we chose LLMs as the reservoir of
our Fusion-LLM.

##  Appendix S4 Failure Case Analysis

Figure S3: The Qualitative result of Shot 80937. We plot Input Laser
Intensity, Ground Truth and Prediction. Y axis denotes laser intensity / hot
electron energy, and X axis denotes time step.

In this section, we examine a significant outlier with the largest error
forecasts generated by the Fusion-LLM on the test split. This particular
instance serves as a critical case study for understanding the limitations and
challenges faced by our model. Fig. S3 illustrates that the shot markedly
deviates from the typical scenarios. Notably, this shot exhibits an
exceptionally low peak hot electron energy, registering less than 0.15,
whereas the majority of other cases yield values ranging between 0.25 and 0.5
under a comparable input laser profile. This anomaly categorizes this shot as
an out-of-distribution (OOD) instance. The limited volume of training data
available for Fusion-LLM is a plausible explanation for the model’s diminished
performance on this OOD data. In scenarios where training data is sparse, the
model’s capability to generalize to new, especially atypical, data points is
inherently restricted. Consequently, this case highlights the importance of
enhancing the dataset’s diversity and volume for ICF tasks. We hope the
community can share more data points to improve and enlarge Fusion4AI dataset
(§3.1) together.

##  Appendix S5 Confidence Analysis

In this section, we provide a comprehensive discussion and analysis of the
confidence scores associated with the Fusion-LLM. As elaborated in §2.2.3, our
confidence scores offer per-step evaluations, thereby aiding physicists to
gain deeper insights into the reliability across various segments of
predictions. This functionality is particularly vital for understanding the
model’s performance dynamics within specific contexts of its predictive
output.

To visually represent this, we have plotted the prediction errors at each time
step for the four test sets alongside their respective confidence scores in
Fig. S4. The visual analysis reveals that the confidence scores exhibit a
discernible decline in the intervals where the model’s predictions incur
larger errors compared to other intervals. This observation substantiates the
existence of a correlation between the model’s diminished confidence and the
occurrence of higher prediction inaccuracies.

Such an analysis underscores the importance of confidence scores as a
diagnostic tool, highlighting intervals where the model’s predictions are
potentially less reliable. By mapping these confidence scores to the
corresponding prediction errors, physicists can identify specific phases
within the prediction and temporal sequence where the model’s forecasting
should be interpreted with caution. This capability not only enhances the
trustworthiness of the Fusion-LLM but also provides critical feedback for
further refinement of the model in the future research.

Moreover, the integration of confidence scores into the model’s predictive
framework offers a robust mechanism for assessing the model’s performance in
real-time applications. By continuously monitoring these scores, physicists
can make informed decisions about the reliability of the predictions, ensuring
that critical assessments and subsequent actions are based on the most
credible forecasting.

Figure S4: Qualitative results of confidence score and prediction error.

##  Appendix S6 Social Impacts and Limitations

The introduction of Fusion-LLM represents a significant advancement in
integrating LLMs with classical reservoir computing paradigms to enhance
predictive capabilities in Inertial Confinement Fusion. This novel approach
not only meets but exceeds several existing state-of-the-art models in
performance benchmarks. From a societal perspective, the implications of
Fusion-LLM are profoundly beneficial, as our approach provides a valuable tool
for advancing our understanding and capabilities in harnessing fusion energy —
a potential key to long-term sustainable energy solutions. However, it is
imperative to acknowledge and critically assess the potential drawbacks
associated with this technology. Similar to other predictive models, Fusion-
LLM faces challenges when dealing with out-of-distribution data or scenarios
that have not been previously encountered. This limitation underscores the
need for ongoing research and refinement, particularly in its application to
real-world ICF scenarios where unpredictable behaviors might emerge.
Therefore, while the model demonstrates promising applications, its deployment
in practical settings must be approached with caution, ensuring continuous
evaluation and adaptation to maintain reliability and safety in its predictive
assertions.

##  Appendix S7 Ethical Safeguards

In our paper, which involves a new dataset, we will establish comprehensive
ethical safeguards to mitigate potential misuse and ensure responsible
utilization, as outlined in the detailed protocols in the final release of
models and datasets. These protocols include strict usage guidelines, access
restrictions, integration of safety filters, and monitoring mechanisms. We
conduct thorough risk assessments to identify potential misuse scenarios,
developing tailored mitigation strategies such as robust data governance
frameworks. Although not all research may require stringent safeguards, we
adhere to best practices, promoting ethical awareness encouraging researchers
to consider the broader impacts of their work and maintain detailed
documentation for transparency and accountability. These efforts demonstrate
our commitment to upholding the highest standards of ethical conduct in
scientific inquiry, aiming to safeguard the interests and privacy of all
people involved.

##  Appendix S8 Reproducibility

Fusion-LLM is implemented in PyTorch [52]. Experiments are conducted on two
NVIDIA A100-40GB GPUs. To guarantee reproducibility, our full implementation
shall be publicly released upon paper acceptance.

##  Appendix S9 Licenses for existing assets

All the methods we used for comparison are publicly available for academic
usage. PIC Simulation is implemented based on the reproducing by osiris-
code/osiris with AGPL-3.0 license. We use huggingface/transformers for the
implementations of Autoformer [86], Llama 2 [79] and Llama 3 [4] with
Apache-2.0 license and Llama 2/3 Community License Agreement. We used the
official repositories DAMO-DI-ML/NeurIPS2023-One-Fits-All (GPT4TS [94]),
KimMeen/Time-LLM [31], rubenohana/Reservoir-computing-kernels (RCRK [17]),
CsnowyLstar/HoGRC [43] and quantinfo/ng-rc-paper-code (NGRC [20]) for our
comparison experiments, where Time-LLM [31] is licensed under Apache-2.0,
HoGRC [43] and NGRC [20] are licensed under MIT, the rest did not mention
their licenses.