# An Ensemble-Based Deep Framework for Estimating Thermo-Chemical State
Variables from Flamelet Generated Manifolds

Amol Salunkhe
University at Buffalo, 338 Davis Hall
Buffalo, New York 14260
& Georgios Georgalis
Data Intensive Studies Center,
Tufts University, Medford, MA 02155
& Abani Patra
Data Intensive Studies Center,
Tufts University, Medford, MA 02155
& Varun Chandola
University at Buffalo, 338 Davis Hall
Buffalo, New York 14260

###### Abstract

Complete computation of turbulent combustion flow involves two separate steps:
mapping reaction kinetics to low-dimensional manifolds and looking-up this
approximate manifold during CFD run-time to estimate the thermo-chemical state
variables. In our previous work, we showed that using a deep architecture to
learn the two steps jointly, instead of separately, is 73 $\%$ more accurate
at estimating the source energy, a key state variable, compared to benchmarks
and can be integrated within a DNS turbulent combustion framework. In their
natural form, such deep architectures do not allow for uncertainty
quantification of the quantities of interest: the source energy and key
species source terms. In this paper, we expand on such architectures,
specifically ChemTab, by introducing deep ensembles to approximate the
posterior distribution of the quantities of interest. We investigate two
strategies of creating these ensemble models: one that keeps the flamelet
origin information (Flamelets strategy) and one that ignores the origin and
considers all the data independently (Points strategy). To train these models
we used flamelet data generated by the GRI–Mech 3.0 methane mechanism, which
consists of 53 chemical species and 325 reactions. Our results demonstrate
that the Flamelets strategy is superior in terms of the absolute prediction
error for the quantities of interest, but is reliant on the types of flamelets
used to train the ensemble. The Points strategy is best at capturing the
variability of the quantities of interest, independent of the flamelet types.
We conclude that, overall, ChemTab Deep Ensembles allows for a more accurate
representation of the source energy and key species source terms, compared to
the model without these modifications.

_Keywords_ Combustion $\cdot$ Deep Ensembles $\cdot$ Physics Guided Machine
Learning $\cdot$ Chemistry Tabulation

##  1 Introduction

Accurate computation of turbulent combustion flow is necessary for the
development of propulsion and energy generation applications (e.g., rocket
motors [1] [9], internal combustion engines [18], nuclear reactors [4]).
Executing such simulations comes with a common, but critical choice: either to
make simplifying assumptions [7] (the simulation does not exactly model all
the physics and chemistry, but is feasible) or to use direct numerical
simulations (DNS) which take long to run and consume large memory resources
(simulation accurately models the phenomena, but is often infeasible.) For
example, a single DNS simulation of $CO/H_{2}/N_{2}$-air jet flames produces
several terabytes of raw data and requires 120,000 hours of CPU-time [8].
Quantifying model and parameter uncertainty remains a challenge when
representing such systems due to the different scales and non-linearity [3,
15, 10].

Given the difference in scales of the flow and chemistry calculation, the two
sub–systems are typically (re)solved separately within a DNS framework.
Approaches such as the Flamelet Generated Manifolds (FGM) use a two–step
strategy where the governing reaction kinetics are pre–computed and mapped to
a low–dimensional manifold, characterized by a few reaction progress variables
(model reduction) and the manifold is then “looked–up” during CFD run–time to
estimate the high–dimensional system state used by the flow system (Fig. 1).
Having these two separate calculations computed independently may reduce the
estimation accuracy of the thermo-chemical state variables due to the
assumption that a low-dimensional manifold exists and is an accurate mapping
for the combustion chemistry [13]. In previous work, we developed ChemTab
[19], a deep neural network architecture, that jointly performs the look-up
process by learning the appropriate embedding of the reaction progress
variables of laminar flames via an encoder and applying a deep regressor to
approximate the manifolds and retrieve the thermo-chemical state variables.
The joint process is 73$\%$ more accurate at estimating the source energy
compared to benchmarks and is easily coupled within a DNS turbulent combustion
framework.

Figure 1: (Re)solving flow and chemistry systems separately during a DNS. Table 1: Nomenclature FGM | Flamelet Generated Manifolds
---|---
$\rho$ | Density of the mixture
$\dot{S_{i}}$ | Source term of the $i^{th}$ species
$S_{e}$ | Source energy
$T$ | Temperature of the mixture
$Y_{i}$ | Mass fraction of $i^{th}$ species
$\mathcal{D}_{i}$ | Diffusivity of the $i^{th}$ species
$k$ | Thermal conductivity
$h_{i}$ | Enthalpy of the $i^{th}$ species
$h_{f,i}^{o}$ | Heat of formation of the $i^{th}$ species
$Z_{mix}$ | Mixture Fraction

While deep neural network architectures have, in general, become the go-to
approach in tasks where the sole goal is accurate prediction, these models do
not communicate any inherent uncertainty in their predictions. The task of
manifold “look-up” within a turbulent combustion DNS is an approximation,
which can lead to errors in the chemistry species calculations outside those
used for training [6], leading to discrepancy between the approximate “look-
up” model and truth, and is therefore a major source of uncertainty.
Predictions from complex computational models with uncertainty quantification
(UQ) increase trustworthiness, both in the model and underlying assumptions.

The goal of this paper, is to expand the regressor part of the ChemTab deep
architecture (Fig. 2) to be able to quantify the uncertainty in the quantities
of interest (the species source energy $S_{e}$ and the key source terms
$\dot{S}_{O2},\dot{S}_{CO},\dot{S}_{CO2},\dot{S}_{H2O},\dot{S}_{OH},\dot{S}_{H2},\dot{S}_{CH4}$)
using Deep Ensembles (DE) [11] and to study how two different data strategies
for network training affect the response for the quantities of interest. The
two strategies are to create training data ensembles keeping flamelet
information (Flamelets strategy) or to create training data ensembles without
considering the underlying flamelet information (Points strategy). The
hypothesis is that Points strategy results in models that, in general, predict
the quantities of interest better, but Flamelets strategy may be superior when
predicting specific flamelet types.

Figure 2: ChemTab + Deep Ensembles Architecture: Jointly generate embedded
progress variables via linear dimensionality reduction (encoder) and
appoximate the manifold via ensembled regressors to get the thermo-chemical
state variables. The model predicts the key species source terms, which allow
to calculate the source energy.

###  1.1 ChemTab Pre-training and Data Generation

The governing conservation equations for mass, species, momentum and energy
for the 1-D, fully compressible, and viscous flames, the steady-state solution
with zero mass flux ($u_{x}=0$) is given by:

$\pdv{}{x}(\mathcal{\rho}\mathcal{D}_{i}\pdv{Y_{i}}{x})+\dot{S_{i}}=0$ (1)
$\pdv{}{x}(k\pdv{T}{x}+\sum{\mathcal{\rho}\mathcal{D}_{i}\pdv{Y_{i}}{x}h_{i}})-\sum{\dot{S_{i}}h_{f,i}^{o}}=0$
(2)

Where $\rho$ is the density of the mixture, $\mathcal{D}_{i}$ is the
diffusivity of the $i^{th}$ species, $Y_{i}$ is the $i^{th}$ species mass
fraction, and $\dot{S_{i}}$ is the source term of the $i^{th}$ species. $k$ is
the thermal conductivity, $T$ is the temperature of the mixture, $h_{i}$ is
the enthalpy of the $i^{th}$ species, and $h_{f,i}^{o}$ is the heat of
formation of the $i^{th}$ species.

In Eq. 2, the final term $\sum{\dot{S_{i}}h_{f,i}^{o}}$ is the source energy
$S_{e}$: the total sum of the product of all the source terms for all species
and their respective heat of formation. The source terms and source energy are
the quantities of interest (QoIs) in this problem.

Solving these differential equations using a finite volume PDE solver provides
the reaction progress variables. To model the chemical kinetics reaction
rates, a variety of mechanisms are adopted in the combustion community. For
this dataset, we used the GRI–Mech 3.0 methane mechanism, which consists of 53
chemical species and 325 reactions. The PDE flamelet solver discretizes the
domain into $299$ grid points (299 observations on the axial coordinate
$X_{pos}$) in between the fuel and the air boundary and $100$ flamelets were
solved to steady–state. Once the solution reaches steady–state the solver
completes one iteration. For the next iteration, the flame solution is
strained by reducing the domain by $0.99$ and the process is continued until
the flame extinguishes. Each flamelet is then tagged with the corresponding
strain rate, the flamelet key. We thus received 29,900 data points (100 flames
and 299 grid points) for a single pressure setting. Some of the generated data
that represent extinguished flames (approximately 45 flames) were discarded,
which led to exclusion of approximately 13,455 data points. So the final
dataset that we used to train, test and validate the models consisted of 55
flamelets constituting 16,445 data points.

Available Data from Chemistry Solver: $Y$ is a matrix of the $s$ species mass
fractions, $\dot{S}$ is a matrix of the source terms of the $s$ species,
$S_{e}$ is a vector of the source energy for the mixture, and $Z_{mix}$ is a
vector of the mixture fraction. $\displaystyle
Y=\begin{bmatrix}Y_{11}&..&..&Y_{1s}\\\ ..&..&..&..\\\ ..&..&..&..\\\
..&..&..&..\\\
Y_{n1}&..&..&Y_{ns}\end{bmatrix},\quad\dot{S}=\begin{bmatrix}\dot{S}_{11}&..&..&\dot{S}_{1s}\\\
..&..&..&..\\\ ..&..&..&..\\\ ..&..&..&..\\\
\dot{S}_{n1}&..&..&\dot{S}_{ns}\end{bmatrix},\quad{S}_{e}=\begin{bmatrix}{S}_{e_{1}}\\\
..\\\ ..\\\ ..\\\ {S}_{e_{n}}\end{bmatrix},\quad
Z_{mix}=\begin{bmatrix}Z_{mix_{1}}\\\ ..\\\ ..\\\ ..\\\
Z_{mix_{n}}\end{bmatrix}$ (3a)

Embedded Data used by the Regressor(s) of ChemTab: $Y$ is a matrix of the $s$
species mass fractions and $W$ is the weight matrix that encodes the original
data into $p<<s$ dimensions.

$\displaystyle C_{pv}=\begin{bmatrix}C_{pv_{11}}&..&C_{pv_{1p}}\\\ ..&..&..\\\
..&..&..\\\ ..&..&..\\\ C_{pv_{11}}&..&C_{pv_{np}}\end{bmatrix}=Y\times
W=\begin{bmatrix}Y_{11}&..&..&Y_{1s}\\\ ..&..&..&..\\\ ..&..&..&..\\\
..&..&..&..\\\
Y_{n1}&..&..&Y_{ns}\end{bmatrix}\quad\times\quad\begin{bmatrix}W_{11}&..&W_{1p}\\\
..&..&..\\\ ..&..&..\\\ ..&..&..\\\ W_{1s}&..&W_{sp}\end{bmatrix}$ (3b)

Predicted Quantities of Interest: The ChemTab Regressor(s) use $C=C_{pv}\oplus
Z_{mix}$ as inputs to predict the $k$ key source terms ($\hat{\dot{S}}$) and
the source energy ($\hat{S_{e}}$).

$\displaystyle\hat{\dot{S}}=\begin{bmatrix}\hat{S}_{11}&..&\hat{S}_{1k}\\\
..&..&..\\\ ..&..&..\\\ ..&..&..\\\
\hat{S}_{n1}&..&\hat{S}_{nk}\end{bmatrix},\quad\hat{{S}_{e}}=\begin{bmatrix}\hat{{S}_{e_{1}}}\\\
..\\\ ..\\\ ..\\\ \hat{{S}_{e_{n}}}\end{bmatrix},\quad$ (3c)

In Eqs. 3 the value of $n$ is 16,445 data points and the value of $s$ is 53.
In Eq. 3b the value of $p$ is 4 and in Eq. 3c the value of $k$ is 7.

With the data generated, we previously trained the encoder part of ChemTab,
i.e., the embedding weights $W$ are pre-computed. Through this linear
dimensionality reduction, we use the optimal reactive scalars for the
embedding $C_{pv}$ together with the given mixture fraction $Z_{mix}$ to form
the embedded progress variables, which are then the inputs to the deep
ensemble regressors.

###  1.2 Uncertainty Quantification in ChemTab

Quantifying uncertainty in deep neural networks is a topic of increasing
interest in aerospace applications (e.g., Monte-Carlo dropout layers in a
U-net for hybrid rocket regression rate estimation [20], deep ensembles for
the inertia estimation of target and servicing spacecraft [2], and Bayesian
networks for risk assessment of aerospace bolts [12]). In Bayesian inference,
predictive uncertainty is often decomposed into the distinct types of
aleatoric and epistemic uncertainty. Aleatoric uncertainty refers to inherent
uncertainty in the data, and is considered intractable and irreducible [14].
Epistemic uncertainty refers to inadequate knowledge about the data
distribution, and theoretically can be reduced by collecting more data or
understanding the problem better. In the context of a deep network, however,
these two traditional uncertainty definitions are not easily separable because
any prediction from the trained model is impacted by the model form
uncertainty, the aleatoric uncertainty in the data used for training, and the
epistemic uncertainty in the input data of interest.

In the hypothetical scenario where a fully Bayesian Neural Network could be
deployed, the goal would be to exactly predict the posterior distribution of
the source energy $S_{e}$ and the key source terms $\dot{S}_{i}$, expressed as
$\mathcal{S}=\\{S_{e},\dot{S}_{i}\\}$ :

$p(\mathcal{S}|C,D)=\int p(\mathcal{S}|C,\theta)p(\theta|D)d\theta$ (4)

Where $C$ is the input embedded progress variables ($C=C_{pv}\oplus Z_{mix}$),
$\theta$ are the weights of the deep network, and $D$ is the training data.

However, given that a fully Bayesian Neural Network is computationally
infeasible [17] to estimate the posterior distribution, we use Deep Ensembles
as a Bayesian approximation [5]. Deep Ensembles refer to the technique of
training a number of deep neural networks from ensembles of the available data
(in this case the embedded progress variables). There are various ways of
generating these ensembles: e.g., bagging/bootstrap aggregating or stacking
[16]. For this case, given that the original ChemTab model is low bias (i.e.,
no underlying assumptions about the form of the target function) and high
variance (i.e., high likelihood to overfit the training set), bagging is the
best option. We train homogeneous weak learners in different regimes of the
available embedded flamelet progress variables and aggregate predictions from
these weak learners such that the overall resulting output is more robust
(i.e., lower variance than the original single learner.) In this case, the
goal becomes to approximate the posterior distribution of the source energy
$S_{e}$ and the key source terms $\dot{S}_{i}$, expressed as
$\mathcal{S}=\\{S_{e},\dot{S}_{i}\\}$ from the $N$ deep ensemble models:

$p(\mathcal{S}|C,D)=\int
p(\mathcal{S}|C,\theta)p(\theta|D)d\theta\approx\frac{1}{N}\sum_{i=1}^{N}p(\mathcal{S}|C,\theta_{i})\,,~{}~{}\theta_{i}\sim
p(\theta|D)$ (5)

Where $C$ is the input embedded progress variables, $\theta_{i}$ are the
weights of the $i^{th}$ ensembled network, and $D$ is the available training
data.

To characterize the posterior distribution $p(\mathcal{S}|C,D)$, we use the
mean of the predictions from the ensemble:

$\mu_{\mathcal{S}|C,D}=\frac{1}{N}\sum_{i=1}^{N}\hat{\mathcal{S}_{i}}$ (6)

And the sample standard deviation as the uncertainty in the predictions:

$U_{\mathcal{S}}=\sqrt{\frac{\sum_{i=1}^{N}(\hat{\mathcal{S}_{i}}-\mu_{\mathcal{S}|C,D})^{2}}{N-1}}$
(7)

##  2 Problem Formulation and Ensemble Strategies

Each of the deep regressor models trained from a data ensemble, can be
expressed as the following layer-wise problem formulation:

$f_{\theta_{i}}^{[0]}=C=C_{pv}\oplus Z_{mix}$ (8)

The subsequent layers together make up the regressor that learns a non-linear
function between the embedded progress variables and the quantities of
interest (the source energy $S_{e}$ and the source terms $\dot{S}_{i}$).

$f_{\theta_{i}}^{[l]}=\theta_{i}^{[l-1]}f_{\theta_{i}}^{[l-1]}+b_{i}^{[l-1]}\qquad\forall\quad
l\in 2\leq l\leq L-1$ (9)
$f_{\theta_{i}}^{[L]}=\sigma(\theta_{i}^{[L-1]}f_{\theta_{i}}^{[L-1]}+b_{i}^{[L-1]})$
(10)

Where , $\theta_{i}^{l}$ are the weights of the ensembled network $i$ at layer
$l$, $b_{i}^{l}$ is the bias of the ensembled network $i$ at layer $l$, and
$\sigma$ is the ReLU scalar activation function.

As mentioned before, we considered two separate strategies to train these
ensembled deep networks:

  1. 1.

Flamelets Strategy: 80 % of the available 55 flamelets (including all the data
points belonging to those flamelets) are used for training–validation of the
models and 20% of the 55 flamelets are used as holdout for the performance
analysis.

  2. 2.

Points Strategy: 80 % of the available 16,445 data points (without considering
which flamelets they belong to) are used for training–validation of the models
and 20% of the 16,445 data points are used as holdout for the performance
analysis.

In each of these two strategies, each member of the ensemble is trained from
randomly sampled data with replacement from the training–validation set. Each
sample includes 80 % of the corresponding training set (i.e., the probability
of a single data point or flame being included in a member model is 64 %). For
the performance analysis, the ensembles are then evaluated on the entire
holdout set.

##  3 Comparison of Ensemble Strategies

###  3.1 Flamelets Strategy

####  3.1.1 Ensemble Size Ablation Study

As a first step, we conducted an ablation study to select the optimal number
of member models in the ensemble. We investigated ensemble sizes $N$ between
$2\leq N\leq 12$.

Figure 3: Flamelets Strategy - Ensemble Ablation

The problem solved by the ensemble model here is highly non-linear and multi-
objective and hence we do not see a clear optimal ensemble size that reduces
the error consistently for all species source terms $\hat{\dot{S}}$ and source
energy $\hat{S_{e}}$. Also, this strategy is highly biased towards the
flamelets that were selected for the member models during the
training–validation process. We observe, however, that the optimal membership
size is between 6 to 10 for most of the source terms and source energy tests.
We choose 8 as the optimal membership size for the performance analysis that
follows.

####  3.1.2 Performance Analysis

To evaluate whether the ensembled models are adequate at predicting the
quantities of interest, we compared the mean absolute prediction error
(compared to truth) between the Single Model and Deep Ensembles architectures
for the key source terms and source energy. All the comparisons are performed
on the entirety of the holdout dataset containing the 20% flamelets that none
of the models have been trained with.

Table 2: Mean Absolute Prediction Error by Key Source Terms Term | Single Model | Deep Ensemble
---|---|---
${S}_{e}$ | 2.390641e+09 | 1.325913e+09
$\dot{S}_{O_{2}}$ | 7.614887e+02 | 4.567584e+02
$\dot{S}_{CO}$ | 5.408012e+02 | 4.370647e+02
$\dot{S}_{CO_{2}}$ | 5.789108e+01 | 3.189488e+01
$\dot{S}_{H_{2}O}$ | 1.792300e+02 | 1.024356e+02
$\dot{S}_{OH}$ | 1.663385e+02 | 1.416341e+02
$\dot{S}_{H_{2}}$ | 2.617136e+01 | 1.791482e+01
$\dot{S}_{CH_{4}}$ | 1.906412e+02 | 9.576332e+01

When comparing by output quantity type, we observe in 2 that the Deep Ensemble
improves (decreases) the mean absolute error, compared to the single model,
for all the quantities of interest.

Table 3: Total Absolute Prediction Error on Source Energy (${S}_{e}$) by Flamelet Key Flamelet Key | Single Model | Deep Ensemble
---|---|---
0.00024894 | 5.506834e+12 | 1.908204e+12
0.00046070 | 7.157733e+11 | 6.082280e+11
0.00059539 | 5.062102e+11 | 4.530053e+11
0.00065971 | 3.284762e+11 | 2.980305e+11
0.00291989 | 1.514082e+11 | 2.100692e+11
0.00085258 | 1.443497e+11 | 2.128944e+11
0.00115982 | 1.213801e+11 | 1.711747e+11
0.00174825 | 1.047254e+11 | 1.322080e+11
0.00193711 | 1.004693e+11 | 1.289064e+11
0.00225936 | 9.594018e+10 | 1.238106e+11

When comparing by flamelet keys, we observe in Table 3 that the Deep Ensemble
improves (decreases) the residuals of the first few flames and then fails to
improve on the subsequent flames. Given this result, it is evident that the
ensemble models focus on reducing the absolute error of the flamelets that
contribute the most absolute error, but fail to do so for the flamelets that
contribute less. Another contributor to this result is that the prediction
outputs from the models in the ensemble are equally weighted when taking a
simple average. A different, chemistry-based weighting scheme, is likely more
appropriate for this strategy. For example, if some members of the ensemble
are better at predicting certain types of flamelets (because they were trained
with such types of flamelets), then these members should be weighted more when
predicting the quantities of interest for certain types of flamelets.

Table 4: Total Absolute Prediction Error on (${S}_{e}$) by $X_{pos}$ Bins $X_{pos}$ \- Bins | Single Model | Deep Ensemble
---|---|---
0.0 - 0.11 | 9.220320e+11 | 1.202144e+12
0.11 - 0.22 | 1.680561e+12 | 5.679709e+11
0.22 - 0.33 | 1.939765e+12 | 1.639737e+12
0.33 - 0.44 | 1.695730e+12 | 3.512237e+11
0.44 - 0.55 | 4.271018e+11 | 2.188931e+11
0.55 - 0.66 | 4.738706e+11 | 1.227035e+11
0.66 - 0.77 | 4.508939e+11 | 1.009125e+11
0.77 - 0.88 | 2.092276e+11 | 8.492034e+10
0.88 - 0.99 | 6.044873e+10 | 6.821203e+10
0.99 - 1.0 | 3.187248e+09 | 4.210067e+09

When comparing by grid position of the flamelet, we observe in Table 4 that
the Deep Ensemble improves (decreases) the residuals for most of the $X_{pos}$
bins apart from the very beginning and end of the grid.

In the following pages, we show comparisons for 2 flamelets (Key 0.000024894
and 0.00046070) from the holdout set between truth and the Single Model / Deep
Ensemble. The 95% confidence interval is defined as $1.96$ times the empirical
standard deviation from the mean prediction.

Figure 4: Single Model vs. Deep Ensemble comparison (Flamelet Key =
0.000024894)

We observe in Fig. 4 that the mean prediction of the ensemble is a much more
accurate representation for the source terms and the source energy response.
For example, for source terms $\dot{S}_{O_{2}},\dot{S}_{CH_{4}}$, the single
model appears to over and under estimate the true value, whereas the ensembled
models capture the behavior smoothly. We also note that the 95% confidence
interval does not always fully encapsulate the true value.

Figure 5: Single Model vs. Deep Ensemble comparison (Flamelet Key =
0.00046070)

We observe in Fig. 5 that the differences between the single and ensembled
models are less pronounced that the previous flamelet. We note that both
architectures do not capture the peaks in some of the source terms, e.g.,
$\dot{S}_{H_{2}O}$. It is, however, valuable that the ensembled models provide
a uncertainty measure, which clearly communicates the existence of peaks or
hills in the responses, e.g., for $\dot{S}_{O_{2}},\dot{S}_{H_{2}O}$. From
these predictions, we also observe the larger errors in the beginning portions
of the flamelet for some quantities, e.g., in the source energy $S_{e}$ and
$\dot{S}_{O_{2}}$ plots.

###  3.2 Points Strategy

####  3.2.1 Ensemble Size Ablation Study

Similarly to the Flamelet strategy, we conducted an ablation study to select
the optimal number of member models in the ensemble for this case. We
investigated ensemble sizes $N$ between $2\leq N\leq 12$.

Figure 6: Points Strategy - Ensemble Ablation

We observe in Fig. 6 that several source terms show an optimal performance
when the optimal membership size is 10. We choose 10 as the optimal membership
size for the performance analysis that follows.

####  3.2.2 Performance Analysis

To evaluate whether the ensembled models are better at predicting the
quantities of interest, we compared the mean absolute prediction error
(compared to truth) between the Single Model and Deep Ensembles architectures
for the key source terms and source energy. All the comparisons are performed
on the entirety of the holdout dataset containing the 20% data points that
none of the models have been trained with.

Table 5: Mean Absolute Prediction Error by Key Source Terms Term | Single Model | Deep Ensemble
---|---|---
${S}_{e}$ | 4.679949e+08 | 2.635308e+08
$\dot{S}_{O_{2}}$ | 1.951080e+02 | 2.208963e+02
$\dot{S}_{CO}$ | 3.353040e+01 | 3.229429e+01
$\dot{S}_{CO_{2}}$ | 3.011369e+01 | 1.199201e+01
$\dot{S}_{H_{2}O}$ | 3.879371e+01 | 2.450645e+01
$\dot{S}_{OH}$ | 1.280294e+02 | 1.151302e+02
$\dot{S}_{H_{2}}$ | 9.770577e+00 | 3.982494e+00
$\dot{S}_{CH_{4}}$ | 4.241771e+01 | 1.780133e+01

When comparing by output quantity type, we observe in Table 5 that the Deep
Ensemble improves (decreases) the mean absolute error, compared to the single
model, for all the quantities of interest except $\dot{S}_{O2}$.

Table 6: Total Absolute Prediction Error on Source Energy (${S}_{e}$) by Flamelet Key Flamelet Key | Single Model | Deep Ensemble
---|---|---
0.00023650 | 7.904599e+10 | 2.853646e+10
0.00020277 | 6.357353e+10 | 3.267808e+10
0.00033866 | 6.025435e+10 | 4.111402e+10
0.00030564 | 5.756831e+10 | 3.318879e+10
0.00022467 | 5.533021e+10 | 3.334996e+10
0.00039499 | 5.167164e+10 | 4.540711e+10
0.00032172 | 5.037682e+10 | 3.497139e+10
0.00027584 | 4.817240e+10 | 2.317479e+10
0.00024894 | 4.783312e+10 | 2.263936e+10
0.00021344 | 4.749524e+10 | 3.102894e+10

When comparing by flamelet keys, we observe in Table 6 that the Deep Ensemble
improves (decreases) the residuals consistently across all flamelets,
including the ones that previously the single model was better for. With this
result, our initial hypothesis, that overall the Points strategy will perform
better than the Flamelets strategy given the equal weighting of the ensemble
models, is true.

Table 7: Total Absolute Prediction Error on (${S}_{e}$) by $X_{pos}$ Bins  $X_{pos}$ \- Bins | Single Model | Deep Ensemble
---|---|---
0.0 - 0.11 | 3.364565e+11 | 3.162390e+11
0.11 - 0.22 | 2.284694e+11 | 1.254176e+11
0.22 - 0.33 | 4.133929e+11 | 2.176143e+11
0.33 - 0.44 | 2.058881e+11 | 1.079951e+11
0.44 - 0.55 | 8.971007e+10 | 3.211109e+10
0.55 - 0.66 | 1.102133e+11 | 2.920763e+10
0.66 - 0.77 | 8.144498e+10 | 2.514758e+10
0.77 - 0.88 | 4.431664e+10 | 9.388437e+09
0.88 - 0.99 | 2.765116e+10 | 3.611705e+09
0.99 - 1.1 | 1.691913e+09 | 2.038095e+07

When comparing by grid position of the flamelet, we observe in 7 that the Deep
Ensemble improves (decreases) the residuals forall of the $X_{pos}$ bins from
the very beginning to the end of the grid.

In the following pages, we show comparisons for 2 flamelets (Key 0.00023650
and 0.00020277) from the holdout set between truth and the Single Model / Deep
Ensembles. The 95% confidence interval is defined as $1.96$ times the
empirical standard deviation from the mean prediction.

Figure 7: Single Model vs Deep Ensemble comparison (Flamelet Key = 0.00023650)

Initially, we observe in Fig. 7 that the Points strategy, both for the single
and for the ensembled models follows the quantities of interest closer
compared to the Flamelets strategy. Compared to the single model, the ensemble
approach, offers the added advantage of knowledge where the model has higher
uncertainty, which is visible for example in the plot for the source energy
$S_{e}$ in the initial part of the flamelet.

Figure 8: Single Model vs Deep Ensemble comparison (Flamelet Key = 0.00020277)

For this flamelet, we observe in Fig. 8 that the single model has some
discrepancy and does not capture the variability for some of the source terms,
e.g., $\dot{S}_{CH_{4}}$ in certain flamelet positions. However, the ensemble
mean predictions not only captures that variability, but also includes the
truth within the range of the confidence interval, which makes the predictions
more trustworthy compared to the single model case.

##  4 Conclusions

In this paper we presented ChemTab Deep Ensembles, a deep uncertainty-enabled
architecture capable of predicting the key source terms and source energy from
flamelet generated manifolds. We compared the mean absolute error for these
quantities of interest with the original version of the architecture, which
included a single regressor. We generated results from two strategies on
creating the ensembles, one that keeps the flamelet origin information
(Flamelets strategy) and one that ignores the origin and considers all the
data points independently (Points strategy). Our results demonstrate that the
(Flamelets strategy) is superior in terms of the absolute prediction error for
the quantities of interest, but is notably reliant on the types of flamelets
used to train the ensemble. We noted that for certain flamelet keys and for
certain positions within those flamelets (particularly the beginning or end),
the (Flamelets strategy) produced weaker results compared to the single model.
In future work, we will explore a different, chemistry-based weighting scheme,
for the Flamelets strategy instead of averaging the ensemble model outputs,
which may remedy these issues. The Points strategy, as was our initial
hypothesis, performs better in terms of absolute errors than the single model
for every flamelet and position we tested on, as well as capturing the
variability of the quantities of interest well. In future work, we will
investigate methods to further capture the uncertainty throughout the deep
architectures, including the encoder part.

## Acknowledgments

This work is supported by the United States Department of Energy’s (DoE)
National Nuclear Security Administration (NNSA) under the Predictive Science
Academic Alliance Program III (PSAAP III) at the University at Buffalo, under
contract number DE-NA0003961.

## References

  * [1] Chedevergne, F., Casalis, G., Majdalani, J.: Dns investigation for the taylor-culick flow stability analysis. In 43rd AIAA/ASME/SAE/ASEE Joint Propulsion Conference & Exhibit p. 5796 (2007)
  * [2] Chu, W., Wu, S., Wu, Z., Wang, Y.: Least square based ensemble deep learning for inertia tensor identification of combined spacecraft. Aerospace Science and Technology 106, 106189 (2020). doi:https://doi.org/10.1016/j.ast.2020.106189, https://www.sciencedirect.com/science/article/pii/S1270963820308713
  * [3] Duraisamy, K., Alonso, J.: Adjoint based techniques for uncertainty quantification in turbulent flows with combustion. In: In 42nd AIAA Fluid Dynamics Conference and Exhibit. p. 2711. AIAA (2012)
  * [4] Fischer, P., Merzari, E., Min, M., Kerkemeier, S., Lan, Y.H., Phillips, M., Rathnayake, T., Novak, A., Gaston, D., Chalmers, N., Warburton, T.: Highly optimized full-core reactor simulations on summit (2021). doi:10.48550/ARXIV.2110.01716, https://arxiv.org/abs/2110.01716
  * [5] Fort, S., Hu, H., Lakshminarayanan, B.: Deep ensembles: A loss landscape perspective (2019). doi:10.48550/ARXIV.1912.02757, https://arxiv.org/abs/1912.02757
  * [6] Freitas, R.S., Rochinha, F.A., Mira, D., Jiang, X.: Parametric and model uncertainties induced by reduced order chemical mechanisms for biogas combustion. Chemical Engineering Science 227, 115949 (2020). doi:https://doi.org/10.1016/j.ces.2020.115949, https://www.sciencedirect.com/science/article/pii/S0009250920304814
  * [7] Gonzalez-Juez, E.D., Kerstein, A.R., Ranjan, R., Menon, S.: Advances and challenges in modeling high-speed turbulent combustion in propulsion systems. Progress in Energy and Combustion Science 60, 26–67 (2017). doi:https://doi.org/10.1016/j.pecs.2016.12.003, https://www.sciencedirect.com/science/article/pii/S0360128515300563
  * [8] Hawkes, E.R., Sankaran, R., Sutherland, J.C., Chen, J.H.: Direct numerical simulation of turbulent combustion: fundamental insights towards predictive models. Journal of Physics: Conference Series 16, 65–79 (jan 2005). doi:10.1088/1742-6596/16/1/009, https://doi.org/10.1088/1742-6596/16/1/009
  * [9] Center for Hybrid Rocket Exascale Simulation Technology, U.o.B.: Ablative boundary layers at the exascale, https://ablate.dev/
  * [10] Ji, W., Ren, Z., Marzouk, Y., Law, C.K.: Quantifying kinetic uncertainty in turbulent combustion simulations using active subspaces. Proceedings of the Combustion Institute 37(2), 2175–2182 (2019). doi:https://doi.org/10.1016/j.proci.2018.06.206, https://www.sciencedirect.com/science/article/pii/S1540748918303924
  * [11] Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 30, p. 6405–6416. Curran Associates, Inc. (2017), https://proceedings.neurips.cc/paper/2017/file/9ef2ed4b7fd2c810847ffa5fa85bce38-Paper.pdf
  * [12] Lee, D., Kim, N., Kim, M.: Dynamic bayesian network for fatigue-crack-related risk assessment of aerospace bolts. Journal of Aerospace Information Systems pp. 1–10 (2022)
  * [13] Maas, U., Pope, S.: Implementation of simplified chemical kinetics based on intrinsic low-dimensional manifolds. In: Symposium (International) on Combustion 24, 103–112 (1992)
  * [14] Moloud, A., Farhad, P., Sadiq, H., Dana, R., Li, L., Mohammad, G., Paul, F., Xiaochun, C., Abbas, K., U. Rajendra, A., Vladimir, M., Saeid, N.: A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion 76, 243–297 (2021). doi:https://doi.org/10.1016/j.inffus.2021.05.008, https://www.sciencedirect.com/science/article/pii/S1566253521001081
  * [15] Mueller, M.E., Raman, V.: Model form uncertainty quantification in turbulent combustion simulations: Peer models. Combustion and Flame 187, 137–146 (2018). doi:https://doi.org/10.1016/j.combustflame.2017.09.011, https://www.sciencedirect.com/science/article/pii/S0010218017303413
  * [16] Opitz, D., Maclin, R.: Popular ensemble methods: An empirical study. Journal of Artificial Intelligence Research 11, 169–198 (1999)
  * [17] Papamarkou, T., Hinkle, J., Young, M.T., Womble, D.: Challenges in markov chain monte carlo for bayesian neural networks (2019). doi:10.48550/ARXIV.1910.06539, https://arxiv.org/abs/1910.06539
  * [18] Posch, S., Gößnitzer, C., Ofner, A.B., Pirker, G., Wimmer, A.: Modeling cycle-to-cycle variations of a spark-ignited gas engine using artificial flow fields generated by a variational autoencoder. Energies 15(7) (2022). doi:10.3390/en15072325
  * [19] Salunkhe, A., Deighan, D., DesJardin, P., Chandola, V.: Chemtab: A physics guided chemistry modeling framework (2022). doi:10.48550/ARXIV.2202.09855, https://arxiv.org/abs/2202.09855
  * [20] Surina, G., Georgalis, G., Aphale, S.S., Patra, A., DesJardin, P.E.: Measurement of hybrid rocket solid fuel regression rate for a slab burner using deep learning. Acta Astronautica 190, 160–175 (2022). doi:https://doi.org/10.1016/j.actaastro.2021.09.046, https://www.sciencedirect.com/science/article/pii/S0094576521005324