# Neural Stellar Population Synthesis Emulator for the DESI PROVABGS

K.J. Kwon  Department of Physics, University of California, Santa Barbara,
Broida Hall, Santa Barbara, CA 93106, USA  ChangHoon Hahn  Department of
Astrophysical Sciences, Princeton University, Peyton Hall, Princeton, NJ
08544, USA  Justin Alsing  Oskar Klein Centre for Cosmoparticle Physics,
Department of Physics, Stockholm University, Stockholm SE-106 91, Sweden

(Received 2022 September 28; Revised 2023 February 3; Accepted 2023 February
6)

###### Abstract

The Probabilistic Value-Added Bright Galaxy Survey (PROVABGS) catalog will
provide the posterior distributions of physical properties of $>10$ million
DESI Bright Galaxy Survey (BGS) galaxies. Each posterior distribution will be
inferred from joint Bayesian modeling of observed photometry and spectroscopy
using Markov Chain Monte Carlo sampling and the Hahn et al. (2022a) stellar
population synthesis (SPS) model. To make this computationally feasible,
PROVABGS will use a neural emulator for the SPS model to accelerate the
posterior inference. In this work, we present how we construct the emulator
using the Alsing et al. (2020) approach and verify that it can be used to
accurately infer galaxy properties. We confirm that the emulator is in
excellent agreement with the original SPS model with $\ll 1\%$ error and is
$100\times$ faster. In addition, we demonstrate that the posteriors of galaxy
properties derived using the emulator are also in excellent agreement with
those inferred using the original model. The neural emulator presented in this
work is essential in bypassing the computational challenge posed in
constructing the PROVABGS catalog. Furthermore, it demonstrates the advantages
of emulation for scaling sophisticated analyses to millions of galaxies.

Galaxies (573), Astronomy data modeling (1859), Neural networks (1933),
Spectral energy distribution (2129)

2023 March 7

##  1 Introduction

Over the next decade, spectroscopic galaxy surveys such as the Dark Energy
Spectroscopic Instrument (DESI; DESI Collaboration et al., 2016; Abareshi et
al., 2022), the Prime Focus Spectrograph (PFS; Takada et al., 2014), William
Herschel Telescope Enhanced Area Velocity Explorer (WEAVE; Dalton et al.,
2012), and the 4-metre Multi-Object Spectroscopic Telescope (4MOST; de Jong et
al., 2019) will measure the spectra of millions of galaxies. They will provide
statistically powerful samples of galaxies at higher redshifts, low stellar
mass ranges, and other less explored regimes. The DESI Bright Galaxy Survey
(BGS; Hahn et al., 2022b), for instance, will provide a magnitude-limited
galaxy sample out to $z\sim 0.6$ and probe dwarf galaxies down to $\sim
10^{7}M_{\odot}$ in the local universe. From the spectra of galaxies in these
new regimes, we can measure their detailed physical properties and shed light
on their formation and evolution.

We currently measure the physical properties of a galaxy, such as its stellar
mass ($M_{*}$), star formation rate (SFR), metallicity ($Z$), and age ($t_{\rm
age}$), by modeling its spectral energy distribution (SED), or spectra, using
stellar population synthesis (SPS; _e.g._ Bruzual & Charlot, 2003; Maraston,
2005; Conroy et al., 2009, see Walcher et al. 2011; Conroy 2013 for a
comprehensive review). Theoretical SPS models are compared to observed spectra
using Bayesian inference to accurately quantify parameter uncertainties and
degeneracies (Acquaviva et al., 2011; Chevallard & Charlot, 2016; Leja et al.,
2017; Carnall et al., 2018; Johnson et al., 2021). The Bayesian approach
enables marginalization over nuisance parameters that model the effects of
observational systematics (_e.g._ flux calibration). Posteriors of galaxies in
a population can also be combined using a Bayesian hierarchical approach for
more accurate population inference (_e.g._ Leja et al., 2019, 2021; Alsing et
al., 2022; Leistedt et al., 2022). However, one major limitation of current
Bayesian SED modeling methods is the computational resources that they
require. Current methods, which use Markov Chain Monte Carlo (MCMC) techniques
to sample the posterior, take 10–100 CPU hours per galaxy (_e.g._ Carnall et
al., 2019; Tacchella et al., 2021). This makes it prohibitive to apply state-
of-the-art galaxy SED modeling to the next-generation galaxy surveys.

Alsing et al. (2020) recently demonstrated that neural emulators can be used
to accelerate SED model evaluations by three to four orders of magnitude. With
the speed-up of neural emulators, posterior inference can be reduced from
hours to minutes per galaxy. In this work, we present a neural emulator
specifically designed for the SPS model that will be used to construct the
PRObabilistic Value-Added Bright Galaxy Survey (Hahn et al., 2022a,
PROVABGS111https://changhoonhahn.github.io/provabgs) catalog. PROVABGS will
provide the posterior distributions of physical properties of $>10$ million
DESI BGS galaxies from joint SED modeling of their optical spectrophotometry.
Its SED model is based on an SPS model with a non-parametric star formation
history (SFH), a non-parametric metallicity history (ZH) that varies with
time, and a flexible dust attenuation prescription. The goal of this work is
to demonstrate that our emulator accurately reproduces the SED model (with
$\ll 1\%$ errors) as well as the posterior on galaxy properties.

We begin in Section 2 with a brief description of the PROVABGS SED model. We
then present how we design and train our neural emulator in Section 3. In
Section 4, we validate the emulator.

##  2 Stellar Population Synthesis

The PROVABGS catalog will use the state-of-the-art Hahn et al. (2022a) SPS
model to infer the posteriors of the physical properties of BGS galaxies. In
this section, we briefly introduce the PROVABGS SPS model that we emulate. In
the PROVABGS SPS model, a galaxy SED is modeled as a combination of the single
stellar populations (SSPs), a group of stars of the same age and metallicity,
formed over the lifetime of the galaxy.

The PROVABGS SPS model uses non-parametric prescriptions for its SFH and ZH.
They are treated as linear combinations of basis functions derived from
applying non-negative matrix factorization (NMF; Lee & Seung 1999; Cichocki &
Phan 2009; Févotte & Idier 2011) to the SFHs and ZHs of simulated galaxies in
the Illustris cosmological hydrodynamic simulation (Vogelsberger et al., 2014;
Nelson et al., 2015). For SFH, the model uses four NMF bases with
coefficients, $\beta_{1},\beta_{2},\beta_{3},\beta_{4}$, and include an SSP
starburst component parameterized with $f_{\text{burst}}$ and
$t_{\text{burst}}$ for additional stochasticity. $f_{\text{burst}}$ denotes
the fraction of the total stellar mass formed during the starburst and
$t_{\text{burst}}$ is the time of the starburst event. For the ZH that varies
with time, the model uses two NMF bases with coefficients $\gamma_{1}$ and
$\gamma_{2}$.

From the SFH and ZH, we calculate the rest-frame luminosity of a galaxy. The
SFH and ZH are binned into logarithmically-spaced lookback time
($t_{\text{lookback}}$) that are truncated at the age of the galaxy. Then, for
each $t_{\text{lookback}}$ bin, the luminosity of an SSP is computed, based on
the metallicity in the bin, using Flexible Stellar Population Synthesis222we
use Python wrapper of FSPS, Python-Fsps (Foreman-Mackey et al.,
2014),333https://github.com/dfm/python-fsps(hereafter FSPS; Conroy et al.
2009; Conroy & Gunn 2010) with MIST isochrones (Paxton et al. 2011, 2013,
2015; Choi et al. 2016; Dotter 2016), the Chabrier (2003) initial mass
function, MILES stellar library (Sánchez-Blázquez et al., 2006) for the
wavelengths between 3800–7100Å and BaSeL library (Lejeune et al., 1997, 1998;
Westera et al., 2002) outside of that range. The SSP luminosity is attenuated
using the two-component dust attenuation by Charlot & Fall (2000). SSPs
younger than 100 Myr are first attenuated by birth clouds. Afterward, all SSPs
are attenuated by the Kriek & Conroy (2013) attenuation curve. The SSP
luminosities are then normalized by the total stellar mass formed in their
$t_{\rm lookback}$ bins and then combined to obtain the dust-attenuated SED.

In summary, apart from nuisance parameters, the PROVABGS SPS model has 12
model parameters: total stellar mass, four NMF SFH basis coefficients, two
starburst components, two NMF ZH basis coefficients, and three dust
attenuation parameters. For the full list of the PROVABGS model parameters and
more details on the PROVABGS SPS model, we refer readers to Hahn et al.
(2022a).

##  3 PROVABGS Emulator

The PROVABGS catalog will infer $>10$ million posteriors using MCMC sampling.
As we later demonstrate, accurately estimating the posterior requires $\gtrsim
120$,000 SPS model evaluations, given the dimensionality of our SPS model. If
we evaluate the PROVABGS SPS model using FSPS, it requires $>10$ CPU hours per
galaxy and makes PROVABGS computationally infeasible (we will refer to the
PROVABGS SPS model evaluated using FSPS as PROVABGS-FSPS). We, therefore,
construct a neural emulator for the SPS model that will accelerate the model
evaluations. In this section, we describe how we construct the emulator.

We follow the neural emulation framework from Alsing et al. (2020). The
emulator consists of a neural network that takes SPS model parameters as input
and predicts coefficients of a fixed set of $N_{\rm PCA}$ Principal Component
Analysis (PCA) basis vectors for galaxy SEDs. The PCA coefficients and basis
vectors are then linearly combined to predict the model SED as a function of
wavelength. In Alsing et al. (2020), they use a single emulator over the full
SED wavelength coverage. In this work, however, we divide the wavelength
coverage into four contiguous ranges, 2000–3600, 3600–5500, 5500–7410, and
7410–60000Å, to achieve higher accuracy. The wavelength ranges have 127, 2109,
2113, and 549 spectral bins, respectively. We also use separate emulators for
the NMF and burst contributions to the SFH. In total, we use 8 emulators. For
each emulator, we use $N_{\text{PCA}}=\\{50$, $50$, $50$, $30\\}$ PCA
components, which we determined through experimentation to achieve $\ll 1$%
PCA reconstruction error throughout the entire wavelength. For the neural
networks, we use the same fully-connected architecture and activation function
as Alsing et al. (2020). We determine the number of hidden layers and units
through experimentation: 8 hidden layers and 256 units for the NMF emulators
and 6 hidden layers and 512 units for the burst emulators. Our architecture is
deeper than the emulator in Alsing et al. (2020), because we have a more
complex SPS model.

To train the emulators, we sample 1 million parameters from the priors in
Table 1 of Hahn et al. (2022a) and use PROVABGS-FSPS to generate a model SED
for each parameter. We then split the dataset into a training and validation
set using a 90/10 split. From the training set, we derive the $N_{\rm PCA}$
basis vectors and use them to transform the model SEDs to PCA coefficients.
Before training, the weights of the neural network are randomly initialized
from a normal distribution and biases are initially set to zero. Afterward, we
train the neural network using the mean squared error between the predicted
and true PCA coefficients as the loss function with the Adam optimizer (Kingma
& Ba, 2014). The initial learning rate is set to $10^{-3}$. If the loss
evaluated on the validation set does not improve after 20 consecutive epochs,
the learning rate is gradually decreased following the sequence: $10^{-3},\
5\times 10^{-4},\ 10^{-4},\ldots,\ 5\times 10^{-6},\ 10^{-6}$. After the
learning rate reaches $10^{-6}$, we terminate the training when the validation
loss does not improve for 20 consecutive epochs.

Figure 1: Fractional error of the PROVABGS emulator based on the 100,000 test
SEDs. The fractional errors are calculated with respect to the model SEDs
calculated using PROVABGS-FSPS. Each panel represents a separate wavelength
bin with a unique emulator. The fractional errors for 68, 95, 99, and 99.9% of
the test SEDs are shown with different shades. The dashed lines represent $\pm
1\%$ errors. The emulator can reconstruct SEDs to sub-percent accuracy for
$95$% of the test SEDs over the full wavelength range, and for 99.9% of the
test SEDs over $\lambda>3000\text{\AA}$. Although there is a relatively large
error on $\lambda<3000\text{\AA}$, we find that this does not impact the
accuracy of the posteriors of the properties of forward-modeled BGS-like
spectra, because BGS spectra generally have low SNR at these wavelengths.

##  4 Results

###  4.1 Validating Emulator SED

In this section, we demonstrate that the emulator accurately reproduces the
SEDs computed by PROVABGS-FSPS. To test the emulator, we sample 100,000
parameters from the priors and calculate their SEDs using the emulator and
PROVABGS-FSPS. Then, we compute the fractional difference of the SEDs:
$({f_{\text{emul.}}-f_{\text{fsps}}})/f_{\text{fsps}}$, where $f_{\rm emul.}$
and $f_{\rm fsps}$ denotes the emulator and PROVABGS-FSPS SED fluxes,
respectively. In Figure 1, we present the fractional errors of the emulator in
each of the four wavelength bins. We mark the percentiles of the emulator
reconstruction error with different shades (68, 95, 99, 99.9%, from the
darkest to lightest). For $\sim 95$% of the test SEDs, the fractional error is
consistently within $<1\%$ in all wavelength bins. In fact, the SEDs from the
emulator and PROVABGS-FSPS agree to a sub-percent level for $99.9$% of the
test SEDs in all but the first wavelength bin. One may be concerned regarding
relatively higher fractional errors at $\lambda<3000\text{\AA}$; however, we
demonstrate in Section 4.2 that this does not significantly impact the
accuracy of the posteriors because BGS spectra have lower signal-to-noise
ratios (SNRs) near-UV wavelengths (Hahn et al., 2022b). Also, SPS modeling has
larger uncertainties in the near-UV due to factors such as metallicities,
stellar age, and dust (Conroy, 2013; Leja et al., 2019; Johnson et al., 2021),
which alleviates potential concerns. We also note that since we train separate
emulators for the different wavelength bins, the SEDs predicted by our
emulator can be discontinuous. However, given the accuracy of our emulator,
this discontinuity does not have a significant impact.

Figure 2: Cumulative distribution of the absolute fractional error of the
PROVABGS emulator, averaged over the wavelength $2400<\lambda<10000$Å. Each
shaded region marks the percentiles of the emulator reconstruction errors (68,
95, 99, 99.9%, from darkest to lightest). The emulator produces the mean
absolute fractional error $\ll$1% on 99.9% test SEDs.

As an additional illustration of the emulator’s accuracy, we present the
cumulative distribution of the absolute fractional error averaged over the
wavelength $2400<\lambda<10000$Å in Figure 2. The shaded regions correspond to
the same percentiles as Figure 1. The emulator produces the mean fractional
error $\ll 1\%$ on $>99$.9% of the test SEDs.

###  4.2 Validating Emulator Posterior

Next, we validate the posteriors of galaxy properties derived using the
PROVABGS emulator against posteriors derived using PROVABGS-FSPS. We infer the
posteriors from synthetic observations of BGS-like spectra from Hahn et al.
(2022a). The synthetic spectra are constructed using $\sim 2$,000 mock
galaxies from the L-galaxies semi-analytic galaxy formation model (Lgal,
hereafter; Henriques et al. 2015). They are calculated with FSPS using the SFH
and ZH of the simulated galaxies. Then, velocity dispersion, dust attenuation,
realistic noise model, and DESI BGS instrumental effect are applied to the
spectra to produce realistic DESI-like spectra. We refer readers to Hahn et
al. (2022a) for the details about the forward modeling. Out of the $\sim
2$,000 Lgal galaxies, we select a representative subset of 93 galaxies. In
Figure 3, we present the distribution of the physical properties of these
galaxies (red). We focus on the following galaxy properties: total stellar
mass ($M_{*}$), star formation rate averaged over past 1 Gyr
($\overline{\text{SFR}}_{\text{1Gyr}}$), mass-weighted metallicity
($Z_{\text{MW}}$), and mass-weighted age ($t_{\text{age,MW}}$). The galaxy
properties are computed in the same way as in Hahn et al. (2022a). For
reference, we present the galaxy property distribution of the full Lgal
samples in black contours and mark the 68 and 95${\rm th}$ percentiles. The
selected 93 samples span the full distribution of Lgal galaxy properties.

Figure 3: The distribution of the physical properties of 93 Lgal simulated
galaxies that we use for our emulator versus PROVABGS-FSPS posterior
comparison (red). We focus on the total stellar mass, average star formation
rate over 1 Gyr, mass-weighted metallicity, and mass-weighted stellar age. The
black contours mark 68 and 95 percentiles of each property for the entire Lgal
samples. The selected Lgal galaxies are a representative sample of the full
Lgal sample. We infer the posteriors of galaxy properties using both our
emulator and the PROVABGS-FSPS from realistic BGS-like spectra constructed
from these simulated galaxies.

We infer the posteriors of the 93 simulated galaxies from their spectra using
a Gaussian likelihood, priors from Hahn et al. (2022a), and MCMC sampling. For
our MCMC sampler, we use the Zeus ensemble slice sampler (Karamanis & Beutler,
2020; Karamanis et al., 2021). Using PROVABGS-FSPS to compute SEDs, we
initialize 30 walkers. We discard the first 1000 iterations as burn-in for 89
of the galaxies. For the other four galaxies, we use 6000 iterations as burn-
in due to inaccurate initialization. The 3000 iterations after burn-in are
used as samples of the posterior distributions. To infer the posteriors using
the emulator, we initialize the walkers at the same positions as the walkers
in the PROVABGS-FSPS chains after burn-in. This is to remove any impact of
initialization when we later compare the two sets of posterior distributions.
We confirm that our posterior samples converge within 3,000 iterations after
burn-in using visual inspection of the distributions of walkers and by
examining the autocorrelation time. For the latter, we use integrated
autocorrelation time (IAT), which is the number of iterations needed to sample
two points that are independent of each other. If
$N_{\text{iteration}}>K\tau/N_{\text{walkers}}$, where $\tau$ is the converged
IAT estimate and $K$ is a constant, the walkers sampled a sufficient number of
independent points to capture posterior distributions. We take $K\sim 50$ and
calculate the mean of $\tau$ for each walker at every 50 iterations following
Goodman & Weare (2010). With $N_{\text{walkers}}=30$, we obtain $\tau\approx
1000$. Hence, $N_{\text{iteration}}\approx 3000$ is sufficient.

Next, we compare the posteriors inferred using the emulator and PROVABGS-FSPS.
From the posteriors of SPS parameters, we derive posteriors of galaxy
properties $M_{*}$, $\overline{\text{SFR}}_{\text{1Gyr}}$, $Z_{\text{MW}}$,
and $t_{\text{age,MW}}$. In Figure 4, we present the emulator (red) and
PROVABGS-FSPS (black) posteriors inferred from the same simulated galaxy. The
contours represent the 68, 95, and 99.7 percentiles. The medians of the
marginalized 1D posteriors are marked with vertical dashed lines in the
histograms. The posteriors derived using the emulator and PROVABGS-FSPS are in
excellent agreement. We also compare the medians and $1\sigma$ uncertainties
of the 1D marginalized posteriors from the emulator and PROVABGS-FSPS for the
selected LGal galaxies in Figure 5. From the leftmost panel, we compare $\log
M_{*}$, $\log\overline{\text{SFR}}_{\text{1Gyr}}$, $\log Z_{\text{MW}}$, $\log
t_{\text{age,MW}}$. There is excellent agreement on all 93 Lgal galaxies in
the posterior medians and uncertainties: we find $<0.02\sigma$ discrepancies
between the medians of the emulator and PROVABGS-FSPS posteriors.

Figure 4: Posterior distributions of galaxy properties derived from a
synthetic BGS-like spectrum inferred using the emulator (red) and PROVABGS-
FSPS (black). For a typical BGS-like spectrum, see Hahn et al. (2022a). The
contours mark 68, 95, and 99.7 percentiles and the vertical dashed lines in
the histograms denote the median of 1D marginalized posteriors. The posteriors
inferred using the emulator and PROVABGS-FSPS are in excellent agreement.
Figure 5: Comparison of galaxy properties inferred using the emulator and
PROVABGS-FSPS. From the left to right panels, we compare the total stellar
mass ($\log M_{*}$), star formation rate averaged over past 1 Gyr
($\log\overline{\text{SFR}}_{\text{1Gyr}}$), mass-weighted metallicity ($\log
Z_{\text{MW}}$), mass-weighted age ($t_{\text{age,MW}}$). These galaxy
properties are computed using the PROVABGS SPS model based on the inferred
posteriors of PROVABGS model parameters. We present the medians of inferred
posteriors in the scatter points and mark the 1-$\sigma$ range with the error
bars. For all samples, the posterior medians derived using the emulator agree
with PROVABGS-FSPS within $\sigma_{\text{fsps}}$, and $\sigma_{\text{emul.}}$
is consistent with $\sigma_{\text{fsps}}$. As illustrated, there is an
excellent agreement between the posteriors derived using the emulator and
PROVABGS-FSPS.

As the main motivation for the emulator to evaluate the PROVABGS SPS model
faster, we profile the computational advantage of the emulator. We randomly
sample 1,000 test parameters from priors and profile the elapsed time for the
emulator and PROVABGS-FSPS to calculate all 1,000 SEDs. On a 2.6 GHz 6-Core
Intel Core i7 CPU, the emulator took an average of 2.3 ms per evaluation,
while PROVABGS-FSPS took 240 ms per evaluation. Thus, to sample a posterior
for a single galaxy, which requires $\sim 120$,000 model evaluations, the
emulator takes $\sim 4$ minutes while PROVABGS-FSPS takes $\sim 10$ hours. The
emulator is $>100\times$ faster than the original SPS model.

##  5 Conclusion

We presented the neural emulator for Hahn et al. (2022a) SPS model that will
be used to construct the DESI PROVABGS catalog. The emulator is constructed
using the neural emulation framework of Alsing et al. (2020). It consists of a
fully-connected neural network that takes SPS model parameters as input and
predicts PCA coefficients, which are combined with PCA basis vectors to
predict the model SED. The neural network and PCA basis vectors are trained
using a training set of 1 million SPS parameters and model SEDs. We use a
similar neural network as the emulators in Alsing et al. (2020) and determine
their architecture (number of hidden layers and units) through
experimentation.

We validate the emulator using a test dataset of 100,000 model SEDs and
confirm that the emulator reconstructs the SEDs computed using PROVABGS-FSPS
with $\ll 1\%$ error for $\lambda>3000$Å. The emulator reproduces the original
SPS model with mean absolute fraction errors of $<0.005$ on 99.9% of the test
SEDs. As further validation of our emulator, we compare posteriors of galaxy
properties derived using the emulator and PROVABGS-FSPS. The posteriors are
inferred from Bayesian SED modeling of synthetic BGS-like spectra constructed
from 93 simulated galaxies in the L-galaxies semi-analytic model. Based on
autocorrelation time analysis, the MCMC sampling converges within 3,000
iterations after burn-in. We find excellent agreement between the emulator and
PROVABGS-FSPS posteriors. The medians of the posteriors deviate by
$<0.02\sigma$ and are in good agreement within the uncertainties. We also
compare the computational cost of the emulator and PROVABGS-FSPS. The emulator
takes 2–3 ms to model a single SED while the PROVABGS-FSPS requires 300–400
ms. To derive a posterior for a single galaxy, our emulator requires
$100\times$ less CPU time, from 10–13 hours to 4–6 minutes. Additional speed-
up will be achievable using GPUs (_e.g._ Alsing et al., 2022) and with more
parallel MCMC methods (_e.g._
$\mathtt{affine}$444https://github.com/justinalsing/affine). For
reproducibility, the codebase for the training, validation, and benchmarking
of the emulator is publicly
available.555https://github.com/changhoonhahn/gqp_mc
https://github.com/changhoonhahn/provabgs

Our PROVABGS emulator is an accurate and $100\times$ faster alternative to
direct SPS computation using PROVABGS-FSPS. It will be essential for
constructing PROVABGS and providing posteriors of galaxy properties for $>10$
million BGS galaxies. More broadly, neural emulation offers a way to scale
rigorous Bayesian inference to the tens of millions of galaxies that will be
observed by upcoming galaxy surveys (_e.g._ DESI, PFS, WEAVE, 4MOST, and
Rubin). Applications of neural emulation, such as the one presented in this
work, will enable future galaxy studies to combine the most rigorous and
accurate measurements of galaxy properties with the unprecedented statistical
power of upcoming observations.

## Acknowledgement

The authors would like to thank Rita Tojeiro for her helpful comments and
feedback on the draft. KJK acknowledges support from the National Science
Foundation under Grant No. 2012086. CH is supported by the AI Accelerator
program of the Schmidt Futures Foundation. This project has received funding
from the European Research Council (ERC) under the European Union’s Horizon
2020 research and innovation programme (grant agreement no. 101018897
CosmicExplorer). JA was partially supported by the research project grant
“Fundamental Physics from Cosmological Surveys” funded by the Swedish Research
Council (VR) under Dnr 2017- 04212.

## References

  * Abareshi et al. (2022) Abareshi, B., Aguilar, J., Ahlen, S., et al. 2022, Overview of the Instrumentation for the Dark Energy Spectroscopic Instrument, arXiv, doi: 10.48550/ARXIV.2205.10939
  * Acquaviva et al. (2011) Acquaviva, V., Gawiser, E., & Guaita, L. 2011, The Astrophysical Journal, 737, 47, doi: 10.1088/0004-637X/737/2/47
  * Alsing et al. (2022) Alsing, J., Peiris, H., Mortlock, D., Leja, J., & Leistedt, B. 2022, arXiv e-prints, arXiv:2207.05819.  https://arxiv.org/abs/2207.05819
  * Alsing et al. (2020) Alsing, J., Peiris, H., Leja, J., et al. 2020, ApJS, 249, 5, doi: 10.3847/1538-4365/ab917f
  * Bruzual & Charlot (2003) Bruzual, G., & Charlot, S. 2003, Monthly Notices of the Royal Astronomical Society, 344, 1000, doi: 10.1046/j.1365-8711.2003.06897.x
  * Carnall et al. (2019) Carnall, A. C., Leja, J., Johnson, B. D., et al. 2019, The Astrophysical Journal, 873, 44, doi: 10.3847/1538-4357/ab04a2
  * Carnall et al. (2018) Carnall, A. C., McLure, R. J., Dunlop, J. S., & Davé, R. 2018, Monthly Notices of the Royal Astronomical Society, 480, 4379, doi: 10.1093/mnras/sty2169
  * Chabrier (2003) Chabrier, G. 2003, PASP, 115, 763, doi: 10.1086/376392
  * Charlot & Fall (2000) Charlot, S., & Fall, S. M. 2000, ApJ, 539, 718, doi: 10.1086/309250
  * Chevallard & Charlot (2016) Chevallard, J., & Charlot, S. 2016, Monthly Notices of the Royal Astronomical Society, 462, 1415, doi: 10.1093/mnras/stw1756
  * Choi et al. (2016) Choi, J., Dotter, A., Conroy, C., et al. 2016, ApJ, 823, 102, doi: 10.3847/0004-637X/823/2/102
  * Cichocki & Phan (2009) Cichocki, A., & Phan, A.-H. 2009, IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, E92.A, 708, doi: 10.1587/transfun.E92.A.708
  * Conroy (2013) Conroy, C. 2013, Annual Review of Astronomy and Astrophysics, 51, 393, doi: 10.1146/annurev-astro-082812-141017
  * Conroy & Gunn (2010) Conroy, C., & Gunn, J. E. 2010, ApJ, 712, 833, doi: 10.1088/0004-637X/712/2/833
  * Conroy et al. (2009) Conroy, C., Gunn, J. E., & White, M. 2009, The Astrophysical Journal, 699, 486, doi: 10.1088/0004-637X/699/1/486
  * Dalton et al. (2012) Dalton, G., Trager, S. C., Abrams, D. C., et al. 2012, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 8446, Ground-based and Airborne Instrumentation for Astronomy IV, ed. I. S. McLean, S. K. Ramsay, & H. Takami, 84460P, doi: 10.1117/12.925950
  * de Jong et al. (2019) de Jong, R. S., Agertz, O., Berbel, A. A., et al. 2019, The Messenger, 175, 3, doi: 10.18727/0722-6691/5117
  * DESI Collaboration et al. (2016) DESI Collaboration, Aghamousa, A., Aguilar, J., et al. 2016, arXiv:1611.00036 [astro-ph].  http://arxiv.org/abs/1611.00036
  * Dotter (2016) Dotter, A. 2016, ApJS, 222, 8, doi: 10.3847/0067-0049/222/1/8
  * Foreman-Mackey et al. (2014) Foreman-Mackey, D., Sick, J., & Johnson, B. 2014, python-fsps: Python bindings to FSPS (v0.1.1), v0.1.1, Zenodo, doi: 10.5281/zenodo.12157
  * Févotte & Idier (2011) Févotte, C., & Idier, J. 2011, Algorithms for nonnegative matrix factorization with the beta-divergence.  https://arxiv.org/abs/1010.1763
  * Goodman & Weare (2010) Goodman, J., & Weare, J. 2010, Communications in Applied Mathematics and Computational Science, 5, 65, doi: 10.2140/camcos.2010.5.65
  * Hahn et al. (2022a) Hahn, C., Kwon, K. J., Tojeiro, R., et al. 2022a, arXiv e-prints, arXiv:2202.01809.  https://arxiv.org/abs/2202.01809
  * Hahn et al. (2022b) Hahn, C., Wilson, M. J., Ruiz-Macias, O., et al. 2022b, arXiv e-prints, arXiv:2208.08512.  https://arxiv.org/abs/2208.08512
  * Henriques et al. (2015) Henriques, B. M. B., White, S. D. M., Thomas, P. A., et al. 2015, Monthly Notices of the Royal Astronomical Society, 451, 2663, doi: 10.1093/mnras/stv705
  * Johnson et al. (2021) Johnson, B. D., Leja, J., Conroy, C., & Speagle, J. S. 2021, The Astrophysical Journal Supplement Series, 254, 22, doi: 10.3847/1538-4365/abef67
  * Karamanis & Beutler (2020) Karamanis, M., & Beutler, F. 2020, arXiv preprint arXiv: 2002.06212
  * Karamanis et al. (2021) Karamanis, M., Beutler, F., & Peacock, J. A. 2021, arXiv preprint arXiv:2105.03468
  * Kingma & Ba (2014) Kingma, D. P., & Ba, J. 2014, arXiv e-prints, arXiv:1412.6980.  https://arxiv.org/abs/1412.6980
  * Kriek & Conroy (2013) Kriek, M., & Conroy, C. 2013, ApJ, 775, L16, doi: 10.1088/2041-8205/775/1/L16
  * Lee & Seung (1999) Lee, D. D., & Seung, H. S. 1999, Nature, 401, 788, doi: 10.1038/44565
  * Leistedt et al. (2022) Leistedt, B., Alsing, J., Peiris, H., Mortlock, D., & Leja, J. 2022, arXiv e-prints, arXiv:2207.07673.  https://arxiv.org/abs/2207.07673
  * Leja et al. (2017) Leja, J., Johnson, B. D., Conroy, C., van Dokkum, P. G., & Byler, N. 2017, The Astrophysical Journal, 837, 170, doi: 10.3847/1538-4357/aa5ffe
  * Leja et al. (2019) Leja, J., Speagle, J. S., Johnson, B. D., et al. 2019, arXiv, arXiv:1910.04168.  https://ui.adsabs.harvard.edu/abs/2019arXiv191004168L/abstract
  * Leja et al. (2019) Leja, J., Johnson, B. D., Conroy, C., et al. 2019, ApJ, 877, 140, doi: 10.3847/1538-4357/ab1d5a
  * Leja et al. (2021) Leja, J., Speagle, J. S., Ting, Y.-S., et al. 2021, arXiv e-prints, arXiv:2110.04314.  https://arxiv.org/abs/2110.04314
  * Lejeune et al. (1997) Lejeune, T., Cuisinier, F., & Buser, R. 1997, A&AS, 125, 229, doi: 10.1051/aas:1997373
  * Lejeune et al. (1998) —. 1998, A&AS, 130, 65, doi: 10.1051/aas:1998405
  * Maraston (2005) Maraston, C. 2005, Monthly Notices of the Royal Astronomical Society, 362, 799, doi: 10.1111/j.1365-2966.2005.09270.x
  * Nelson et al. (2015) Nelson, D., Pillepich, A., Genel, S., et al. 2015, Astronomy and Computing, 13, 12, doi: 10.1016/j.ascom.2015.09.003
  * Paxton et al. (2011) Paxton, B., Bildsten, L., Dotter, A., et al. 2011, ApJS, 192, 3, doi: 10.1088/0067-0049/192/1/3
  * Paxton et al. (2013) Paxton, B., Cantiello, M., Arras, P., et al. 2013, ApJS, 208, 4, doi: 10.1088/0067-0049/208/1/4
  * Paxton et al. (2015) Paxton, B., Marchant, P., Schwab, J., et al. 2015, ApJS, 220, 15, doi: 10.1088/0067-0049/220/1/15
  * Sánchez-Blázquez et al. (2006) Sánchez-Blázquez, P., Peletier, R. F., Jiménez-Vicente, J., et al. 2006, Monthly Notices of the Royal Astronomical Society, 371, 703, doi: 10.1111/j.1365-2966.2006.10699.x
  * Tacchella et al. (2021) Tacchella, S., Conroy, C., Faber, S. M., et al. 2021, arXiv e-prints, 2102, arXiv:2102.12494.  http://adsabs.harvard.edu/abs/2021arXiv210212494T
  * Takada et al. (2014) Takada, M., Ellis, R. S., Chiba, M., et al. 2014, Publications of the Astronomical Society of Japan, 66, R1, doi: 10.1093/pasj/pst019
  * Vogelsberger et al. (2014) Vogelsberger, M., Genel, S., Springel, V., et al. 2014, Nature, 509, 177, doi: 10.1038/nature13316
  * Walcher et al. (2011) Walcher, J., Groves, B., Budavári, T., & Dale, D. 2011, Astrophysics and Space Science, 331, 1, doi: 10.1007/s10509-010-0458-z
  * Westera et al. (2002) Westera, P., Lejeune, T., Buser, R., Cuisinier, F., & Bruzual, G. 2002, A&A, 381, 524, doi: 10.1051/0004-6361:20011493


# Multimodal Grounding for Embodied AI via Augmented Reality Headsets for
Natural Language Driven Task Planning.

Selma Wanna a, Fabian Parraa, Robert Valnerb, Karl Kruusamäeb and Mitch Pryora
CONTACT Selma Wanna Email<EMAIL_ADDRESS>aUniversity of Texas at Austin;
bUniversity of Tartu

###### Abstract

Recent advances in generative modeling have spurred a resurgence in the field
of Embodied Artificial Intelligence (EAI). EAI systems typically deploy large
language models to physical systems capable of interacting with their
environment. In our exploration of EAI for industrial domains, we successfully
demonstrate the feasibility of co-located, human-robot teaming. Specifically,
we construct an experiment where an Augmented Reality (AR) headset mediates
information exchange between an EAI agent and human operator for a variety of
inspection tasks. To our knowledge the use of an AR headset for multimodal
grounding and the application of EAI to industrial tasks are novel
contributions within Embodied AI research. In addition, we highlight potential
pitfalls in EAI’s construction by providing quantitative and qualitative
analysis on prompt robustness.

###### keywords:

Natural Language Processing; Foundation Models; Language Grounding;
Multimodality; Human Robot Collaboration

††articletype: full paper

##  1 Introduction

Offloading dangerous inspection, surveillance, and manipulation tasks to
robots in unstructured environments, e.g., industrial task domains, incident
response, etc. has been the driving motivation for utilizing robots in human-
robot teams. However, the supervision of such a team requires the human
operator to work and lead the robots simultaneously. To do so effectively
requires an intuitive and minimally restrictive control interface that also
provides sufficient situational awareness of the robots and the environment.
Mixed Reality (MR) technology offers a capable platform for designing such
control interfaces, as allows overlaying the operator’s view with, e.g., heat-
maps, structural weak-spots, and the location of other human and robot team-
members in no line-of-sight or low visibility.

MR tools, such as Augmented Reality (AR) headsets often come equipped with
hand-tracking and speech recognition capabilities, allowing the operator to
utilize multiple naturalistic communication modalities that can reduce
ambiguity of unimodal interaction in HRI (Fig. 1). The challenge, however, is
understanding the operator’s intent from the combination of these modalities.
Commands issued via speech and gestures have to be robustly grounded into
executable tasks.

We address this issue of multimodal grounding by adapting prior work in
Embodied AI research: the study of artificial intelligence deployed to
physical systems capable of interacting with their environments [1, 2] to AR
headsets. Specifically, we inject visual and language information obtained via
the AR headset directly into the language prompt of GPT-3 [3]. However, to
extend EAI to generalize to industrial settings, we enforced a co-located,
human-robot teaming paradigm where an AR headset mediated dialogue between the
EAI agent and human operator.

To our knowledge, this demonstration is novel to EAI deployments with respect
to the industrial domain and the use of the AR headset. We contribute a
successful demonstration of EAI and AR integration, provide studies on prompt
design which highlight potential pitfalls in common EAI constructions [4, 5,
6, 7, 8, 9] with respect to prompt fragility, and conclude with a holistic
discussion on the merits of adopting EAI for multimodal task planning.

Figure 1: Conceptual overview of mixed reality headset utilized as an
intuitive and minimally restrictive multimodal control interface.

##  2 Related Work

Prior designs for EAI deployments have largely converged on a common
architecture which leverages hard prompt learning techniques in conjunction
with object detectors to generate an agent’s next action prediction. A
summarization of prior work is provided in Table 1.

Method, Year & Reference | Prompt Type | Model(s) | Tasks
---|---|---|---
ProgPrompt, 2022 [4] | Hard | GPT-3 [3] | VirtualHome [10] \+ Physical
InnerMonologue, 2022 [11, 5] | Hard | PaLM [12] \+ InstructGPT [13] | Physical
SayCan, 2022 [6] | Hard | PaLM [12] | ALFRED [14] \+ BEHAVIOR [15] \+ Physical
SocraticModels, 2022 [7] | Hard | ViLD [16] | Simulated Tabletop
ProbES, 2022 [17] | Soft | ViLBERT [18] with LSTM and MLP Head | REVERIE [19] \+ R2R [20]

Table 1:  Summarization of recent effort to leverage LLM prompting techniques
for Embodied AI.

Most methods that rely on hard prompt engineering employ human-engineered
prompts with little quantitative design justification. This is problematic
because prompt engineering is a non-robust process where different but
semantically equivalent prompts may cause task performance to vary between
pure chance and state-of-the-art performance [21]. The fickleness of prompt
design manifests itself in non-robust prompt preference [22], prompt sample
selection bias [21], and sample ordering bias [21]. Additionally, discrete
prompt design often requires extensive human-engineering [23] thus leading to
the recent creation of PromptCraft, an open-source platform for robotics
researchers to share their prompting strategies [9].

The emergence of multimodal foundation models [24] such as CLIP [25] have
reinforced the reliance on LLMs for Embodied AI task planning [26, 27, 17].
These methods, which distill image information into text, are commonly
leveraged for tasks such as object detection [16] and scene description [28]
as a means of world and agent state tracking. These systems are uniquely
compatible with the LLM prompting paradigm because the LLM is restricted to
text modalities by design. Within this LLM-driven design, there are largely
two multimodal fusion techniques: injecting visual information via image-to-
text algorithms into the language prompt [4, 5, 11, 7] or synthesizing the
information downstream [8, 6]. We adopt the former approach; however, we
abandon image-to-text models in favor of human generated virtual reality (VR)
markers.

##  3 Background

###  3.1 Large Language Models

Large Language Models (LLMs) come in a variety of sizes and architectures;
however, this discussion centers on autoregressive models capable of in-
context learning [3] because they are best suited for text generation tasks
[23], e.g., task planning [6, 4, 5]. For autoregressive language models, the
most common training objective is to maximize the log-likelihood of the next
token prediction at a decoding step, $t$, based on the context provided by the
previous $t-1$ tokens. This is formalized in Equation 1 where, $\mathbf{y}$,
represents the decoded text that is generated as a result of conditioning on
the input text, $\mathbf{x}$, and latent features, $h$. Equation 1 aims to
solve for the LLM parameters, $\theta$, that maximize the log likelihood of
the observations, $y$.

$\max_{\theta}\ log\
p(\mathbf{y}|\mathbf{x};\theta)=\max_{\theta}\sum_{y_{t}}log\
p(y_{t}|h_{<t};\theta)$ (1)

This training objective is performed under self-supervision tasks [24]. Under
this construction, the model learns language information by solving a variety
of de-noising tasks such as masked token prediction [29], next sentence
prediction [29], next token prediction (language modeling) [3], long range
dependency modeling [30], etc. For a comprehensive overiew of de-noising
objectives and functions, please refer to [23].

###  3.2 Prompt Learning

The discovery of in-context learning [3] in conjunction with the expensiveness
of training enormous language models drove the field of natural language
research toward prompt based learning [24]. Additionally, the companies that
offer LLMs as a service restrict access to LLM feature and gradient
information, rendering other transfer learning techniques, such as fine
tuning, infeasible [31]. As such, this work focuses on tuning-free prompt
learning: a type of prompt engineering that involves searching for the optimal
prompting function for a LLM with frozen model parameters [23].

Formally, prompt learning involves taking natural language, $\mathbf{x}$, as
an input to a prompt function, $f_{prompt}(\cdot)$, to generate a prompt:
$\mathbf{x}^{\prime}$. While the theory behind prompt learning lags behind its
empirical findings, it is speculated that prompting incantations [32] can
prime [3] LLMs to activate relevant neurons for a desired task [33].

The simplest method for prompt searching is to perform an exhaustive search
along the axes of few-shot example ordering, example selection, and number of
examples. This search space is limited by the maximum token request and rate
limits API constraints provided by OpenAI’s text-davinci-003 model [3]. As
such, the search space comprised of the set of 2-length permutations of
multimodal UMRF decoding examples. Each example permutation was scored by its
BLEU score [34] accuracy against its natural language instruction’s ground-
truth UMRF decoding in the validation set. The prompt permutation with the
highest accuracy was chosen as the optimal prompt.

###  3.3 Unified Meaning Representation Format

The Unified Meaning Representation Format (UMRF) [35] is a platform
independent task description format based on JSON notation, designed to
decouple robot’s autonomous capabilities and its command interface (Fig. 2a).
Thus a robot could be controlled via any system or command interface that
outputs commands in UMRF, increasing both the robot’s and command interface’s
modularity and reusability. Tasks are defined as graphs of interconnected
actions, described via parent/child relations (Fig. 2b). UMRF supports
sequential, concurrent and cyclical graphs and parametrization, i.e., actions
can accept and produce data (Fig. 2c). A thorough coverage of UMRF can be
found in [35].

Figure 2: (a) UMRF graph based robot commanding pipeline where arbitrary
command modality, e.g., voice command, is parsed and converted to UMRF graph
notation via a dedicated parser. Each UMRF node in the UMRF graph is then
mapped to a known executable action that implements the desired behaviour. (b)
Example of an UMRF graph in JSON notation outlining an inspection task that
contains three sequential actions. (c) A detailed UMRF JSON notation of a
parametrized navigation action that accepts coordinates and the name of the
target location.

##  4 UMRF Prompt Design

Our discrete prompt methodology is most similar to InnerMonologue [5] where we
incorporate relevant world contexts and objects in conjunction with few shot
examples within the prompt. However, we depart from their effort by (1)
incorporating chain-of-thought prompting techniques within in-context examples
[36], similar to ProgPrompt [4]; (2) adopt the UMRF [35] formalism for action
decoding instead of pythonic code generations; and lastly (3) provide language
and visual feedback via the AR headset to develop a multimodal prompt. For
greater detail, please refer for Figure 3.

Figure 3: Summarization of the UMRF prompt design with multimodal input and
output streams labeled.

##  5 Prompt Design Experiments

###  5.1 Experiment 1. Greedy & Exhaustive Search

The exhaustive search algorithm explored a space of ${}_{10}P_{2}$ prompts
evaluated on a validation set of five examples. To emphasize the expense of
the greedy search, the average query time to OpenAI was 1.5 minutes. Thus,
running a search of over 450 queries cost roughly 11 hours of computation.
This expense motivates the need for more efficient search algorithms for
prompt design such as the method suggested in Section 5.2.

The experiment outcomes support our hypothesis that prompt design is highly
non-robust even for simple tasks that can be solved using traditional grammars
or simpler neural networks, such as the UMRF decomposition task. This is
clearly demonstrated in appendix 11. The top ten performing prompts are
provided in Figure 4.

Figure 4: The variation in performance of the top 10 performing prompt
designs. Note that despite being stronger prompts overall, there is a lack of
robust generalization for each design to the validation set.

While information ordering and example selection account for high variation
(see Figures 13 and 12 respectively), the top performing prompts most commonly
share combinations of example types: 1, 4, and 5 as described in Table 3. In
every case, barring one, example type 5 performed best as the first example in
the sequence. This may be because example 5 is both the longest and most
informative training example. While its complexity may allow for the best
generalization performance when ordered first, its placement toward the end of
the prompt may confuse the LLM as it begins to decode the validation query.

The strongest performing prompt design with an average BLEU score of 0.662 was
prompt 70 with the structure: example 4 + example 5 (please refer to Table 4.)
The prompt with the highest BLEU score (0.850) had the following structure:
example 1 + example 4. For more details regarding the prompt design of the top
ten prompts, please refer to Table 4.

###  5.2 Experiment 2. Assessing Prompt Fragility

In an attempt to better explore the prompt search space, we extended prior
work in textual data augmentation [37] to find an optimal discrete prompt.
Specifically, we tried to learn a prompt policy by searching for compositional
augmentations which maximize BLEU score [34] on a validation set through
Bayesian optimization. The value in this methodology is in learning a
surrogate model that can more cheaply mimic the outputs of an LLM, allowing
for more efficient search space exploration. However our adaptation of Text
AutoAugment [37] to generative language tasks was unsuccessful due to weak
reward signals and high performance sensitivity to heuristic augmentations as
discussed in [37].

Additional experiments regarding prompt fragility were conducted to measure
BLEU score accuracy given a single application of text augmentation with
varied magnitudes of the operation [38]. As shown in Figure 5, GPT-3 is robust
to random deletion and insertion operations. However, random swapping of words
and synonym replacements tend to have a larger effect on performance.

Figure 5: Prompt sensitivity analysis to EDA [38] perturbations at various
magnitudes.

A more representative experiment was performed to measure the BLEU score
accuracy after applying compositional augmentations to the prompt. The results
are shown in Figure 6. Generally, compositional augmentations widened
performance variation across the board. As a future recommendation, the search
space for the magnitude parameter for Text AutoAugment [37] should be
constrained to less than 0.1.

Figure 6: Prompt sensitivity analysis to EDA [38] after compositional
perturbations were performed at various magnitudes.

Despite this setback, further analyses was conducted to investigate whether a
given prompt’s similarity to UMRF examples present in GPT-3’s pretraining
dataset could be a predictor of a prompt’s performance. Semantic similarity
was measured as the cosine similarity between all-MiniLM-L6-v2 model
embeddings [39] of a prompt versus the UMRF examples present in The Pile [40].
In our low-data regime, we could not identify a correlation (see Figures 7 and
8).

Figure 7: The semantic similarity of the 90 unique prompts generated using the
greedy search algorithm from Section 3.2 measured against UMRF examples seen
in GPT-3’s pretraining dataset [40]. Figure 8: The semantic similarity of
prompts sampled from our adaptation of Text Autoaugment against UMRF examples
seen in GPT-3’s pretraining dataset [40]. Policy 1 did not apply heuristic
data augmentation techniques [38] while policy 2 did through application of
synonym replacement.

From this analysis we are able to identify a legitimate danger in carelessly
deploying EAI in safety-critical environments. Specifically, semantically
equivalent prompts can vary greatly in performance. By qualitative inspection,
seemingly harmless applications of synonym replacement such as converting
numerical representations to their written forms as well as replacing similar
words, e.g., ‘move forward’ to ‘approach’ and ‘table’ to ‘bureau’, noticeably
harmed task performance. Potentially adversarial augmentations, including
converting the coordinate variable, ‘y’, to ‘yttrium’ or ‘atomic number 39’
were detrimental to task performance. Unfortunately, there is no immediate
path forward for roboticists developing systems in niche domains.
Specifically, we cannot rely on robustness techniques employed for language
tasks with greater task representation in LLM pretraining datasets [41].

##  6 Demonstration

This section demonstrates the functionality of the multimodal speech and
augmented reality based UMRF graph parser, outlined in Section 4. The
demonstration depicts a remote inspection scenario, where a mobile manipulator
robot (Clearpath Husky + two Universal Robots UR5’s), equipped with a camera
(Intel RealSense D435) attached to the end-effector, has to navigate to and
inspect specific areas defined by the operator. The operator is equipped with
an AR headset (Microsoft HoloLens 2) that is able to capture voice commands
(Fig. 9a) and allows defining goal locations via gesture-operated virtual
markers (Fig. 9b). Inspection and task execution feedback is overlaid to the
operator’s field of view in real-time (Fig. 9c). The Azure Spatial Anchors
plugin [42] was used on HoloLens to allow the robot and AR-devices to co-
localize and share same the same reference frame. Target poses are generated
by spawning a coordinate frame in the world, and dragging it to the desired
pose. The Natural language command is captured by pressing the microphone icon
on the HoloLens app.

Fig. 10 shows the software setup of the demo, containing three main
components: HoloLens, which captures the operator’s input and provides
feedback; command server, which hosts the UMRF parser; and the robot, that is
able to execute tasks outlined in UMRF notation. The Robot Operating System
(ROS) [43] and RoboFleet [44] were used for data distribution between the
components. The operator interface on HoloLens2 was implemented in Unreal
Engine 4.26 [45], which combines an operator’s voice command with the
coordinates of the virtual marker to a string format. The combined input is
sent to the command server via RoboFleetUnrealClient. The UMRF parser
(available on GitHub111github.com/temoto-framework/gpt_umrf_parser) on the
command server, implemented as a ROS Python node, receives the command string
and constructs the prompt (see Section 4). Each prompt embeds five operator
command + UMRF graph pair examples (Table 2), which then is sent to OpenAI via
openai v.0.25.0 Python API. The API call was configured for text-davinci-003
model, with max_tokens=1024, and rest of the settings on default values. The
UMRF JSON string, retured by OpenAI, then is sent to the robot via ROS
message. TeMoto Framework [46] was used to both control the mobile base,
camera, and manipulators of the robot, as well as TeMoto Action Engine was
used to ground the UMRF graphs to executable actions (setup files available on
GitHub222github.com/temoto-framework-demos/gpt_temoto_demo).

Figure 9: AR interface from the operator’s perspective. (a) interactive marker that user can drag & drop (b) Natural Language interface to send voice commands. (c) UMRF graph and video feedback is shown in the AR space Figure 10: Software setup of the demonstration Table 2: Examples used for constructing the prompts. Only abstract description of the output is provided, all examples in full detail available in online materials. |  Examples
---|---
1 |  INPUT: ‘Move to the main hall [x=14; y=3.2; yaw=1.26]’
|  OUTPUT: single ‘navigation’ action
2 |  INPUT: ‘Go to the workshop [x=-33.9; y=12.1; yaw=0.04]’
|  OUTPUT: single ‘navigation’ action
3 |  INPUT: ‘robot go observe the valve [x=-93.6; y=11.0; yaw=-0.85]’
|  OUTPUT: sequence of:
$\text{`navigate'}\rightarrow\text{`manipulate'}\rightarrow\text{`scan'}\rightarrow\text{`manipulate'}\rightarrow\text{`scan'}$
4 |  INPUT: ‘robot go inspect the workshop [x=74.2; y=-223.6; yaw=2.72]’
|  OUTPUT: sequence of:
$\text{`navigate'}\rightarrow\text{`manipulate'}\rightarrow\text{`scan'}\rightarrow\text{`manipulate'}\rightarrow\text{`scan'}$
5 |  INPUT: ‘Scan the area’
|  OUTPUT: single ‘scan’ action

##  7 Discussion & Future Work

In this paper we provide successful demonstrations of inspection tasks in
industrial settings by mediating multimodal information through an AR headset.
Despite the efficacy of the EAI construction, the prompting paradigm
necessitates greater scrutiny from robotics researchers. Specifically, prompt
designs that leverage in-context learning are not token-space efficient. This
prevents LLMs from observing larger quantities of training examples within the
hard prompt construction. Additionally, extensive human-engineering is
required to develop ‘optimal’ discrete prompts. This is partially due to
prompt fragility. Furthermore, the nonrobustness of prompts to natural and
adversarial perturbations add on additional vulnerabilities to physical
systems. Lastly, there are technical challenges when relying on a third party
to serve a LLM, particularly during periods of high demand. Commonly, OpenAI
API request and rate limits were exceeded and led to no API responses or
incomplete parses.

We outline avenues of future work as follows. First, there is a need to
conduct additional studies on human operator agency and quality-of-life in our
collaborative human-robot team setup versus the traditional human-in-the-loop
paradigm where the operator’s role is relegated to correcting erroneous vision
algorithm outputs. Second, it is necessary to quantify the gap in EAI
performance when using human-assisted AR markers versus imperfect object
detectors. Third, the robot dialogue responses which indicate agent and world
state information are entirely visual. Efforts to implement text-to-speech
algorithms may improve operator enjoyment of using the system. Fourth, to
address the issue of token space efficiency, exploring multi-step LLM
prompting techniques is necessary. A potential path forward is to first query
for a compact representation of the task graph then query the LLM to fill in
the nodal information. Fifth, it is necessary to conduct a study on task
performance and robustness of various representations for LLM task planners to
decode natural language into, e.g., UMRFs, pythonic code [4] or lists [5] as a
function of their representation frequency in GPT-3’s pretraining dataset. We
speculate more representative formalisms may allow the use of information
retrieval solutions to the prompt-robustness challenge [41]. Lastly, given the
preliminary outcomes on prompt fragility, it is imperative that a sensitivity
analysis be conducted between the magnitude of prompt perturbations and their
effects on validation BLEU scores. This is our immediate next step toward
characterizing prompt robustness for EAI systems.

## Acknowledgement

We thank Los Alamos National Laboratory for their support. LA-UR-23-22632.

## Funding

This research has been in part supported by European Social Fund via IT
Academy programme, Estonian Centre of Excellence in IT (EXCITE) funded by the
European Regional Development Fund, and AI & Robotics Estonia co-funded by the
EU and Ministry of Economic Affairs and Communications in Estonia.

Additionally, we thank the support of Los Alamos National Laboratory and the
Center for Nonlinear Studies.

## Notes on contributors

Selma Wanna received a B.Sc. degree in electrical engineering and M.Sc. degree
in mechanical engineering from the University of Texas at Austin. Her research
interests include quantifying the robustness of deep learning systems
operating in out-of-distribution domains.

Fabian Parra received his bachelor’s degree in mechatronic engineering from
the Nueva Granada Military University in Bogota, and the M.Sc. degree in
Robotics and Computer engineering from the University of Tartu in Estonia. His
research interests include mobile manipulator robots, supervised autonomy and
human-machine interfaces.

Robert Valner is a junior researcher and a Ph.D. candidate at University of
Tartu. He received B.Sc. and M.Sc. degrees in physics and computer engineering
respectively from University of Tartu. His current research interests include
fault tolerant and adaptive robotic architectures, multi-robot systems, human-
robot interaction and autonomous robotic systems.

Dr. Karl Kruusamäe is an associate professor of robotics engineering at the
University of Tartu. He received the M.S. degree in information technology and
the Ph.D. in physics from the University of Tartu, Tartu, Estonia, in 2008 and
2012, respectively. His research interests include human-robot interaction and
shared autonomy.

Dr. Mitch Pryor is a Senior Research Scientist and Lecturer for the Cockrell
School of Engineering at the University of Texas at Austin. Dr. Pryor earned
his BSME at Southern Methodist University in 1993. He completed his Masters
(1999) and PhD (2002) at UT Austin with an emphasis on the modeling,
simulation, and operation of redundant manipulators. He has worked for
numerous research sponsors including, NASA, DARPA, DOE, INL, LANL, ORNL, Y-12,
and many industrial partners. He is a co-founder of the Nuclear Robotics Group
and the Drilling & Rig Automation Group. Both are interdisciplinary research
efforts to deploy robotics in hazardous, uncertain environments to perform
manufacturing, material handling and other tasks. He is a member of ROS-
Industrial, IEEE, ASME, PGE, and ANS.

## References

  * [1] Duan J, Yu S, Tan HL, et al. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence. 2022;6(2):230–244.
  * [2] Deitke M, Batra D, Bisk Y, et al. Retrospectives on the embodied ai workshop ; 2022\. Available from: https://arxiv.org/abs/2210.06849.
  * [3] Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. In: Larochelle H, Ranzato M, Hadsell R, et al., editors. Advances in Neural Information Processing Systems; Vol. 33. Curran Associates, Inc.; 2020. p. 1877–1901. Available from: https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * [4] Singh I, Blukis V, Mousavian A, et al. Progprompt: Generating situated robot task plans using large language models. In: Second Workshop on Language and Reinforcement Learning; 2022. Available from: https://openreview.net/forum?id=aflRdmGOhw1.
  * [5] Huang W, Xia F, Xiao T, et al. Innermonologue: Embodied reasoning through planning with language models; 2022. CoRL 2022 (to appear); Available from: https://innermonologue.github.io/.
  * [6] Ichter B, Brohan A, Chebotar Y, et al. Do as i can, not as i say: Grounding language in robotic affordances. In: 6th Annual Conference on Robot Learning; 2022\. Available from: https://openreview.net/forum?id=bdHkMjBJG_w.
  * [7] Zeng A, Attarian M, Ichter B, et al. Socratic models: Composing zero-shot multimodal reasoning with language ; 2022. Available from: https://arxiv.org/abs/2204.00598.
  * [8] Liang X, Zhu F, Lingling L, et al. Visual-language navigation pretraining via prompt-based environmental self-exploration. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); May; Dublin, Ireland. Association for Computational Linguistics; 2022. p. 4837–4851. Available from: https://aclanthology.org/2022.acl-long.332.
  * [9] Vemprala S, Bonatti R, Bucker A, et al. Chatgpt for robotics: Design principles and model abilities. Microsoft; 2023. MSR-TR-2023-8. Available from: https://www.microsoft.com/en-us/research/publication/chatgpt-for-robotics-design-principles-and-model-abilities/.
  * [10] Puig X, Ra K, Boben M, et al. Virtualhome: Simulating household activities via programs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); June; 2018.
  * [11] Huang W, Abbeel P, Pathak D, et al. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents ; 2022. Available from: https://arxiv.org/abs/2201.07207.
  * [12] Chowdhery A, Narang S, Devlin J, et al. Palm: Scaling language modeling with pathways ; 2022. Available from: https://arxiv.org/abs/2204.02311.
  * [13] Ouyang L, Wu J, Jiang X, et al. Training language models to follow instructions with human feedback ; 2022. Available from: https://arxiv.org/abs/2203.02155.
  * [14] Shridhar M, Thomason J, Gordon D, et al. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2020. Available from: https://arxiv.org/abs/1912.01734.
  * [15] Li C, Gokmen C, Levine G, et al. BEHAVIOR-1k: A benchmark for embodied AI with 1,000 everyday activities and realistic simulation. In: 6th Annual Conference on Robot Learning; 2022. Available from: https://openreview.net/forum?id=_8DoIe8G3t.
  * [16] Gu X, Lin TY, Kuo W, et al. Open-vocabulary object detection via vision and language knowledge distillation. In: International Conference on Learning Representations; 2022. Available from: https://openreview.net/forum?id=lL3lnMbR4WU.
  * [17] Liang X, Zhu F, Lingling L, et al. Visual-language navigation pretraining via prompt-based environmental self-exploration. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); May; Dublin, Ireland. Association for Computational Linguistics; 2022. p. 4837–4851. Available from: https://aclanthology.org/2022.acl-long.332.
  * [18] Lu J, Batra D, Parikh D, et al. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Wallach H, Larochelle H, Beygelzimer A, et al., editors. Advances in Neural Information Processing Systems; Vol. 32. Curran Associates, Inc.; 2019. Available from: https://proceedings.neurips.cc/paper/2019/file/c74d97b01eae257e44aa9d5bade97baf-Paper.pdf.
  * [19] Qi Y, Wu Q, Anderson P, et al. Reverie: Remote embodied visual referring expression in real indoor environments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); June; 2020.
  * [20] Anderson P, Wu Q, Teney D, et al. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2018.
  * [21] Lu Y, Bartolo M, Moore A, et al. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); May; Dublin, Ireland. Association for Computational Linguistics; 2022. p. 8086–8098. Available from: https://aclanthology.org/2022.acl-long.556.
  * [22] Cao B, Lin H, Han X, et al. Can prompt probe pretrained language models? understanding the invisible risks from a causal view. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); May; Dublin, Ireland. Association for Computational Linguistics; 2022. p. 5796–5808. Available from: https://aclanthology.org/2022.acl-long.398.
  * [23] Liu P, Yuan W, Fu J, et al. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput Surv. 2022 sep;Just Accepted; Available from: https://doi.org/10.1145/3560815.
  * [24] Bommasani R, Hudson DA, Adeli E, et al. On the opportunities and risks of foundation models ; 2021. Available from: https://arxiv.org/abs/2108.07258.
  * [25] Radford A, Kim JW, Hallacy C, et al. Learning transferable visual models from natural language supervision ; 2021. Available from: https://arxiv.org/abs/2103.00020.
  * [26] Khandelwal A, Weihs L, Mottaghi R, et al. Simple but effective: Clip embeddings for embodied ai. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); June; 2022. p. 14829–14838.
  * [27] Dorbala VS, Sigurdsson G, Piramuthu R, et al. Clip-nav: Using clip for zero-shot vision-and-language navigation ; 2022. Available from: https://arxiv.org/abs/2211.16649.
  * [28] Kamath A, Singh M, LeCun Y, et al. Mdetr–modulated detection for end-to-end multi-modal understanding. arXiv preprint arXiv:210412763. 2021;.
  * [29] Devlin J, Chang MW, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers); Jun.; Minneapolis, Minnesota. Association for Computational Linguistics; 2019. p. 4171–4186. Available from: https://aclanthology.org/N19-1423.
  * [30] Paperno D, Kruszewski G, Lazaridou A, et al. The LAMBADA dataset: Word prediction requiring a broad discourse context. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); Aug.; Berlin, Germany. Association for Computational Linguistics; 2016. p. 1525–1534. Available from: https://aclanthology.org/P16-1144.
  * [31] Ye X, Durrett G. The unreliability of explanations in few-shot prompting for textual reasoning. In: Oh AH, Agarwal A, Belgrave D, et al., editors. Advances in Neural Information Processing Systems; 2022. Available from: https://openreview.net/forum?id=Bct2f8fRd8S.
  * [32] Liang P, Bommasani R, Lee T, et al. Holistic evaluation of language models ; 2022\. Available from: https://arxiv.org/abs/2211.09110.
  * [33] Su Y, Wang X, Qin Y, et al. On transferability of prompt tuning for natural language processing. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; Jul.; Seattle, United States. Association for Computational Linguistics; 2022. p. 3949–3969. Available from: https://aclanthology.org/2022.naacl-main.290.
  * [34] Papineni K, Roukos S, Ward T, et al. Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics; Jul.; Philadelphia, Pennsylvania, USA. Association for Computational Linguistics; 2002. p. 311–318. Available from: https://aclanthology.org/P02-1040.
  * [35] Valner R, Wanna S, Kruusamäe K, et al. Unified meaning representation format (umrf) - a task description and execution formalism for hri. J Hum-Robot Interact. 2022 jan;Just Accepted; Available from: https://doi-org.ezproxy.lib.utexas.edu/10.1145/3522580.
  * [36] Wei J, Wang X, Schuurmans D, et al. Chain of thought prompting elicits reasoning in large language models ; 2022. Available from: https://arxiv.org/abs/2201.11903.
  * [37] Ren S, Zhang J, Li L, et al. Text AutoAugment: Learning compositional augmentation policy for text classification. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing; 2021.
  * [38] Wei J, Zou K. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); Nov.; Hong Kong, China. Association for Computational Linguistics; 2019. p. 6382–6388. Available from: https://aclanthology.org/D19-1670.
  * [39] Reimers N, Gurevych I. Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing; 11. Association for Computational Linguistics; 2019\. Available from: https://arxiv.org/abs/1908.10084.
  * [40] Gao L, Biderman S, Black S, et al. The pile: An 800gb dataset of diverse text for language modeling ; 2021. Available from: https://arxiv.org/abs/2101.00027.
  * [41] Arora S, Narayan A, Chen MF, et al. Ask me anything: A simple strategy for prompting language models. In: International Conference on Learning Representations; 2023. Available from: https://openreview.net/forum?id=bhUPJnS2g0X.
  * [42] Argüelles R, Wojciakowski M, Fowler C, et al. Azure spatial anchors overview. 2019;Available from: docs.microsoft.com/en-gb/azure/ spatial-anchors/overview.
  * [43] Stanford Artificial Intelligence Laboratory et al. Robotic operating system ; ???? Available from: https://www.ros.org.
  * [44] Sikand KS, Zartman L, Rabiee S, et al. Robofleet: Secure open source communication and management for fleets of autonomous robots. arXiv preprint arXiv:210306993. 2021;.
  * [45] Epic Games. Unreal engine ; ???? Available from: https://www.unrealengine.com.
  * [46] Valner R, Vunder V, Aabloo A, et al. Temoto: A software framework for adaptive and dependable robotic autonomy with dynamic resource management. IEEE Access. 2022;10:51889–51907.

##  Appendix A Additional Figures

Figure 11: A lack of prompt robustness is shown in this figure. Figure 12: A
lack of prompt robustness toward chosen training example is shown in this
figure. Figure 13: A lack of prompt robustness toward visual versus natural
language command cues are shown in this figure.

Prompt Type |  Prompt Structure
---|---
1 |  [x=-9.074; y=-1.89; yaw=2.97] the left side of the same desk + Turn left and approach the left side of the same desk + $\langle$umrf_label$\rangle$ \+ …
2 |  [x=4.76; y=-6.78; yaw=7.687] the bed + Turn right and face the bed. + $\langle$umrf_label$\rangle$ \+ …
3 |  [x=-9.15; y=4.316; yaw=2.168] the wall [x=1.26; y=7.61; yaw=-0.214] the table + Turn right and walk to the wall then turn left and walk to the table. + $\langle$umrf_label$\rangle$ \+ …
4 |  [x=-6.74; y=-4.67; yaw=3.086] the right side of the wooden desk + Walk over to the right side of the wooden desk. + $\langle$umrf_label$\rangle$ \+ …
5 |  [x=1.12; y=-1.749; yaw=6.01] the middle of the side of the bed [x=-7.14; y=-3.14; yaw=3.14] the bed + Turn right and take a small step forward then turn right and walk until you’re even with the middle of the side of the bed then when you are turn right and walk to the bed. + $\langle$umrf_label$\rangle$ \+ …

Table 3:  Prompt structure of example types.

Prompt Type |  Prompt Structure |  Average BLEU Score
---|---|---
61 |  Type 4 V + Type 5 L |  0.588
65 |  Type 4 L + Type 2 L |  0.652
66 |  Type 4 L + Type 2 V |  0.632
70 |  Type 5 L + Type 4 V |  0.662
73 |  Type 5 V + Type 1 V |  0.538
74 |  Type 5 V + Type 2 L |  0.538
75 |  Type 5 V + Type 2 V |  0.538
78 |  Type 5 V + Type 4 L |  0.561
82 |  Type 5 L + Type 1 V |  0.524
84 |  Type 5 L + Type 2 V |  0.560

Table 4:  Prompt structure of the top 10 best generalizing prompts. The V
versus L distinction indicates whether the visual information or the natural
language command came first in the example.


# A Note on Proton Stability in the Standard Model

Seth Koren  Enrico Fermi Institute, University of Chicago, Chicago, IL 60637,
U.S.A.

###### Abstract

In this short note we describe the symmetry responsible for absolute,
nonperturbative proton stability in the Standard Model. The SM with $N_{c}$
colors and ${N_{g}}$ generations has an exact, anomaly-free, generation-
independent, global symmetry group
$U(1)_{B-N_{c}L}\times\mathbb{Z}_{N_{g}}^{L}$, which contains a subgroup of
baryon plus lepton number of order $2N_{c}{N_{g}}$. This disallows proton
decay for ${N_{g}}>1$. Many well-studied models beyond the SM explicitly break
this global symmetry, and the alternative deserves further attention.

Everything not forbidden is compulsory. So which symmetry forbids proton decay
in the Standard Model? It’s the lightest baryon, but baryon number is
anomalous and not a symmetry of the quantum SM. The difference between baryon
and lepton number is anomaly-free, but allows e.g. $p^{+}\rightarrow
e^{+}\pi^{0}$. In fact there is a discrete subgroup of baryon plus lepton
number which is anomaly free by virtue of the SM having more than one
generation. This symmetry imposes the selection rule $\Delta
B=N_{c}{N_{\text{g}}},\Delta L={N_{\text{g}}}$ on the SM with $N_{c}$ colors
and ${N_{\text{g}}}$ generations. In the following we briefly review the topic
of mixed anomalies in the SM selectively aimed toward evincing the anomaly-
free discrete global symmetries. Field theoretic calculations we have omitted
can be found in standard QFT textbooks or in Bertlmann’s monograph [1].

| $\quad Q\quad$ | $\quad\bar{u}\quad$ | $\quad\bar{d}\quad$ | $\quad L\quad$ | $\quad\bar{e}\quad$
---|---|---|---|---|---
$SU(3)_{C}$ | $3$ | $\bar{3}$ | $\bar{3}$ | – | –
$SU(2)_{L}$ | $2$ | – | – | $2$ | –
$U(1)_{Y}$ | $+1$ | $-4$ | $+2$ | $-3$ | $+6$
$U(1)_{B}$ | $+1$ | $-1$ | $-1$ | – | –
$U(1)_{L}$ | – | – | – | $+1$ | $-1$
Table 1: Representations of the SM Weyl fermions under the classical
symmetries of the SM. We normalize each $U(1)$ so the least-charged particle
has unit charge, $B\equiv 3B_{\text{usual}}$, $Y\equiv 6Y_{\text{usual}}$, and
$L\equiv L_{\text{usual}}$.

The Standard Model of particle physics is defined as the gauge theory of the
non-Abelian symmetry group $SU(3)_{C}\times SU(2)_{L}\times U(1)_{Y}$ with
three ‘generations’ (or ‘families’) of left-handed Weyl fermions in the
representations shown in Table 1. There is additionally a scalar electroweak
Higgs doublet which has Yukawa couplings providing masses to the electrically-
charged fermions in the broken phase.

The SM so defined contains additional ‘accidental’ generation-independent
exact classical global symmetries corresponding to baryon and lepton number,
whose charges are also listed in Table 1. These are accidental in that the
most general renormalizable Lagrangian one may write down automatically
preserves them. However, the Lagrangian is a classical object and a good
classical global symmetry $U(1)_{X}$ may be broken by the path integral
measure upon quantization if the fermions charged under $U(1)_{X}$ are in a
chiral representation of a gauge group $G$ [2].

One may check whether the classical global symmetry survives quantization by
examining the ‘anomaly conditions’, which in four dimensions consists
essentially of evaluating the three-point correlator of the symmetry currents
at one loop and checking if the Ward-Takahashi identity is satisfied. Our case
of interest will have one global symmetry current and two gauge symmetry
currents, such that the condition for global current conservation in the
quantum theory is

$\partial_{\mu}\langle J^{\mu}_{X}J^{\nu}_{G}J^{\rho}_{G}\rangle=0.$ (1)

If this condition is satisfied, then the symmetry $U(1)_{X}$ is ‘anomaly-free’
and the classical current conservation $\partial_{\mu}J^{\mu}_{X}=0$ may be
upgraded to a Ward identity in the full quantum theory. If this condition is
violated then nonperturbative $G$ gauge theory effects inevitably lead to
$U(1)_{X}$ symmetry violation, and furthermore, $U(1)_{X}$ cannot itself be
consistently gauged.***The absence of ABJ anomalies is necessary to gauge
$U(1)_{X}$, but not sufficient. We note in particular that the $\left\langle
J_{X}J_{X}J_{X}\right\rangle$ ‘'t Hooft anomaly’ need also vanish, and in the
presence of dynamical gravity, a mixed anomaly with gravitational currents
must also be avoided.

This type of anomaly is sometimes referred to as a ‘mixed anomaly’ between
$U(1)_{X}$ and $G$ or an ‘ABJ anomaly’ after its discovery by Adler-Bell-
Jackiw [3, 4] as the microphysical explanation for the decay of the neutral
pion $\pi^{0}\rightarrow\gamma\gamma$. In that application one relates the
failure of the Ward identity in the three-point correlator to a three-point
amplitude using the LSZ formula. The pion, as a pseudo-Goldstone of the axial
symmetry $U(1)_{A}$, has nonzero overlap with its global symmetry current
$J_{A}^{\mu}$, and the photons have nonzero overlap with the $U(1)_{Q}$ gauge
symmetry current $J_{Q}^{\mu}$, so this amplitude provides for pion decay.

When such an anomaly is present, a global rotation of the charged fermions
does not leave the action invariant but rather changes the effective
$\theta$-term for the $G$ gauge field. If we perform a rotation of the
fermions charged under the global $U(1)_{X}$ by an angle $\alpha$, the action
is shifted as

$\psi_{i}\rightarrow\psi_{i}e^{iq_{i}\alpha}\quad\Rightarrow\quad\delta
S=\alpha\mathcal{A}\int\frac{F\tilde{F}}{16\pi^{2}},$ (2)

where $\psi_{i}$ are left-handed Weyl fermions with charge $q_{i}$ under
$U(1)_{X}$, $F$ is the field strength of the gauge group $G$ and $\tilde{F}$
is its Hodge dual. If this transformation changes the partition function of
the quantum theory, then it is no longer a symmetry.

The integrand may be recognized as the Chern-Pontryagin density of the gauge
field configuration, and its integral is an integer topological invariant
which measures the winding of the gauge field around the sphere at Euclidean
spacetime infinity. $\mathcal{A}$ is also an integer which is the sum of the
‘anomaly coefficients’ of all left-handed Weyl fermions

$\mathcal{A}\
\delta^{ab}=\sum\limits_{i}\text{Tr}\left[q_{i}T^{a}_{R_{i}}T^{b}_{R_{i}}\right],$
(3)

where $T^{a}_{R_{i}}$ are the generators of the representation $R_{i}$ of $G$
and for a non-Abelian group the normalization is such that in the fundamental
representation $F$, $\text{Tr}\left[T_{F}^{a}T_{F}^{b}\right]=\delta^{ab}$.
That is, a $G$ fundamental with unit $X$ charge contributes $\mathcal{A}=1$.
For an Abelian $G$ there is only one generator and $\delta^{ab}\mapsto 1$. The
integral nature of the anomaly derives from the number of zero modes of
fermions in the background spacetime—these are counted by the index of the
Dirac operator, which is directly related to the anomaly by a theorem of
Atiyah and Singer [5, 6].

If $G$ is non-Abelian and the only nontrivial representations of $G$ are
fundamentals, as for example with $SU(2)_{L}$ in the SM, then we have simply

$\mathcal{A}_{XSU(2)^{2}_{L}}=\sum\limits_{\text{Fund}\ i}q_{i},$ (4)

while if $G$ is abelian as for example with $U(1)_{Y}$ in the SM, we have
simply a trilinear in charges

$\mathcal{A}_{XU(1)_{Y}^{2}}=\sum\limits_{i}q_{i}Y_{i}^{2}.$ (5)

If the fermions charged under $U(1)_{X}$ are in a vector-like representation
of $G$, as all the SM fermions are with $SU(3)_{C}$ (or electromagnetism),
then for each left-handed Weyl fermion $\psi_{i}$ there is another right-
handed Weyl fermion $\psi_{\bar{i}}^{\dagger}$ with the same quantum numbers,
such that they pair up into a Dirac spinor. Then $\psi_{i}$ and
$\psi_{\bar{i}}$ are in complex conjugate representations of the gauge group
and their contributions to the anomaly coefficient are related by three
negative signs and cancel out.

| $U(1)_{B}$ | $U(1)_{L}$
---|---|---
$SU(2)_{L}^{2}$ | $N_{c}{N_{\text{g}}}$ | ${N_{\text{g}}}$
$U(1)_{Y}^{2}$ | $-18N_{c}{N_{\text{g}}}$ | $-18{N_{\text{g}}}$
Table 2: Mixed anomalies of the classical accidental symmetries with the
chiral gauge symmetries of the SM. $N_{c}$ is the number of colors and $N_{g}$
the number of generations.

In Table 2 we give the ABJ anomalies of baryon and lepton number with the
chiral factors of the SM gauge group. $\mathcal{A}\neq 0$ indicates the
presence of an anomaly and the classical $U(1)_{X}$ symmetry is broken, since
the action is no longer invariant under the transformation as in Equation 2.
As is familiar, while both baryon and lepton number have anomalies, we may
form the anomaly-free current $B-N_{c}L$ of baryon minus lepton number. This
is the only anomaly-free, generation-independent continuous global symmetry of
the SM. On the other hand, baryon plus lepton number is violated in
nonperturbative processes involving the gauge fields given by the new term in
the action Equation 2, which effects (see e.g. [7] for lucid discussion)

$\left\langle\partial_{\mu}J^{\mu}_{B+N_{c}L}\right\rangle=2N_{c}{N_{\text{g}}}\int\frac{W\tilde{W}}{16\pi^{2}},$
(6)

where the expectation value is taken in a given background gauge field, $W$ is
the $SU(2)_{L}$ field strength in that configuration, and we have left off the
similar $U(1)_{Y}$ term as it has no effects in $d=4$ flat space for reasons
of topology. This nonperturbative effect is central to electroweak
baryogenesis [8, 9] wherein the thermal configurations giving dynamical
symmetry violation are called ‘sphalerons’ [10, 11].

While we have exhausted the anomaly-free continuous global symmetries, let us
now relax our symmetry of interest from the full $U(1)_{X}$ of rotations by
arbitrary angles to the subgroup of transformations by $\alpha=2\pi k/N$ for
some $N\in\mathbb{N},k=0..N-1$. If we choose $N=\mathcal{A}$, then under any
rotation the action changes by a multiple of $2\pi i$ in Equation 2 and the
partition function is invariant. This $\mathbb{Z}_{N}$ subgroup of $U(1)_{X}$
then remains a good symmetry of the quantum theory.

In the case of the SM, this means that there is an additional discrete,
anomaly-free $\mathbb{Z}_{N_{\text{g}}}$ worth of symmetries for the SM with
${N_{\text{g}}}$ generations of fermions. Of course there is some freedom to
describe the additional generator, since any addition of $Y$ or $B-N_{c}L$
would work just as well. But we may non-redundantly identify this as the
$\mathbb{Z}_{N_{\text{g}}}^{L}$ subgroup of lepton number $U(1)_{L}$, which is
manifestly independent of both—whereas there is a $\mathbb{Z}_{N_{c}}$
subgroup of $B-N_{c}L$ in which the leptons transform trivially and the
transformation is equivalent to one of $U(1)_{B}$.

Consequently, the anomaly-free, generation-independent, global symmetry group
of the SM is $U(1)_{B-N_{c}L}\times\mathbb{Z}_{N_{\text{g}}}^{L}$. A few more
remarks are in order on the structure of this symmetry group. Firstly, we note
that baryon plus lepton number $U(1)_{B+N_{c}L}$ intersects this group in a
$\mathbb{Z}_{2N_{c}{N_{\text{g}}}}$ subgroup generated by
$(1,1)\in\mathbb{Z}_{2N_{c}{N_{\text{g}}}}^{B-N_{c}L}\times\mathbb{Z}_{{N_{\text{g}}}}^{L}$.

This $\mathbb{Z}_{2N_{c}{N_{\text{g}}}}$ is the maximal anomaly-free subgroup
of baryon plus lepton number in the SM. We see that the appearance of the
anomaly coefficient in Equation 6 expressing current nonconservation enforces
dynamically the $\Delta L={N_{\text{g}}},\Delta B=N_{c}{N_{\text{g}}}$
selection rule imposed by the existence of this exact discrete symmetry.
Indeed, SM sphaleron processes all respect this selection rule.

Further within this, we note that there is a $\mathbb{Z}_{2N_{c}}$ subgroup in
which $U(1)_{B+N_{c}L}$ intersects $U(1)_{B-N_{c}L}$ directly, since leptons
and antileptons have the same charge $\text{mod}\ 2N_{c}$. And inside of this,
fermion number can be realized as the order two subgroup of $B\pm N_{c}L$
rotations by $e^{i\pi F}$, since the only fields charged under $B$ or $L$ in
the SM have odd $B$ or $L$ charges and are fermions, and $N_{c}$ is also odd.
Summarizing these relationships, we have

$U(1)_{B-N_{c}L}\times\mathbb{Z}^{L}_{N_{\text{g}}}\supset\mathbb{Z}_{2N_{c}{N_{\text{g}}}}^{B+N_{c}L}\supset\mathbb{Z}_{2N_{c}}^{B\pm
N_{c}L}\supset(-1)^{F},$

among anomaly-free global symmetries of the SM.

We note also that the $U(1)_{B}$ symmetry we have defined in the SM at high
energies may more accurately be named ‘quark number’. It is in the confined
phase that $B_{\text{usual}}\equiv B/N_{c}$ really counts baryons, which must
be constructed with the $SU(N_{c})$ invariant tensor
$\varepsilon_{i_{1}i_{2}\dots i_{N_{c}}}$ to be colorless. If we strictly work
in an effective theory below nuclear energy scales then there are no
$B-N_{c}L$ unit charges, and it’s sensible to work with baryon minus lepton
number as usually defined $B/N_{c}-L$. The above subgroup series is then
modified to

$U(1)_{B/N_{c}-L}\times\mathbb{Z}^{L}_{N_{\text{g}}}\supset\mathbb{Z}_{2{N_{\text{g}}}}^{B/N_{c}+L}\supset(-1)^{F}.$

The proton, with $B/N_{c}=1$ and as the lightest baryon in the broken phase,
then cannot decay while satisfying both $\Delta(B/N_{c}-L)=0$ and
$\Delta(B/N_{c}+L)=0\ (\text{mod}\ 2{N_{\text{g}}})$.

While many theories beyond the SM explicitly break these global symmetries, in
the face of increasingly stringent constraints on the lifetime of the proton
it may be worth reconsidering the prospects that it is absolutely stable. I,
for one, would welcome the possibility of there being one fewer looming
existential threat.

## Earlier Work

Work on anomalies of discrete symmetries began with [12, 13]. A variety of
authors have considered exotic gauged $\mathbb{Z}_{3}$ symmetries to stabilize
the proton in the context of the Minimal Supersymmetric Standard Model (MSSM)
and extensions thereof (e.g. [14, 15, 16, 17, 18, 19, 20, 21, 22]), where
baryon and lepton number are no longer classical global symmetries. Other
interesting related work includes [23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
33]. I note especially that, while conducting extensive literature review, I
found that [34] on the global structure of the SM gauge group noted the
existence of a $\mathbb{Z}_{N_{\text{g}}}$ anomaly-free subgroup of
$B/N_{c}+L$, and [35] on discrete symmetries in the MSSM mentioned in their
Footnote 7 that a $\mathbb{Z}_{N_{\text{g}}}$ subgroup of $B/N_{c}$ protects
the proton in the SM. Soon after this work appeared, [36] explored the
incompatibility of the $B/N_{c}+L$ symmetry with a variety of grand
unification schemes. I beg the pardon of any experts who know the facts
explained in this manuscript already, and I hope the preceding dedicated
discussion remains of use to the community.

## Acknowledgements

I am grateful to Clay Cordova, Carlos Wagner, Liantao Wang, and especially T.
Daniel Brennan and Sungwoo Hong for helpful discussions. This work was
supported by an Oehme Postdoctoral Fellowship from the Enrico Fermi Institute
at the University of Chicago.

## References

  * Bertlmann [1996] R. A. Bertlmann,  _Anomalies in quantum field theory_ (1996).
  * Fujikawa [1979] K. Fujikawa, Path Integral Measure for Gauge Invariant Fermion Theories, Phys. Rev. Lett. 42, 1195 (1979).
  * Adler [1969] S. L. Adler, Axial vector vertex in spinor electrodynamics, Phys. Rev. 177, 2426 (1969).
  * Bell and Jackiw [1969] J. S. Bell and R. Jackiw, A PCAC puzzle: $\pi^{0}\to\gamma\gamma$ in the $\sigma$ model, Nuovo Cim. A 60, 47 (1969).
  * Atiyah and Singer [1963] M. F. Atiyah and I. M. Singer, The index of elliptic operators on compact manifolds, Bulletin of the American Mathematical Society 69, 422 (1963).
  * Fujikawa [1980] K. Fujikawa, Path Integral for Gauge Theories with Fermions, Phys. Rev. D 21, 2848 (1980), [Erratum: Phys.Rev.D 22, 1499 (1980)].
  * Morrissey  _et al._ [2005] D. E. Morrissey, T. M. P. Tait, and C. E. M. Wagner, Proton lifetime and baryon number violating signatures at the CERN LHC in gauge extended models, Phys. Rev. D 72, 095003 (2005), arXiv:hep-ph/0508123 .
  * Kuzmin  _et al._ [1985] V. A. Kuzmin, V. A. Rubakov, and M. E. Shaposhnikov, On the Anomalous Electroweak Baryon Number Nonconservation in the Early Universe, Phys. Lett. B 155, 36 (1985).
  * Shaposhnikov [1987] M. E. Shaposhnikov, Baryon Asymmetry of the Universe in Standard Electroweak Theory, Nucl. Phys. B 287, 757 (1987).
  * Manton [1983] N. S. Manton, Topology in the Weinberg-Salam Theory, Phys. Rev. D 28, 2019 (1983).
  * Klinkhamer and Manton [1984] F. R. Klinkhamer and N. S. Manton, A Saddle Point Solution in the Weinberg-Salam Theory, Phys. Rev. D 30, 2212 (1984).
  * Ibanez and Ross [1991] L. E. Ibanez and G. G. Ross, Discrete gauge symmetry anomalies, Phys. Lett. B 260, 291 (1991).
  * Ibanez and Ross [1992] L. E. Ibanez and G. G. Ross, Discrete gauge symmetries and the origin of baryon and lepton number conservation in supersymmetric versions of the standard model, Nucl. Phys. B 368, 3 (1992).
  * Ibanez [1993] L. E. Ibanez, More about discrete gauge anomalies, Nucl. Phys. B 398, 301 (1993), arXiv:hep-ph/9210211 .
  * Hinchliffe and Kaeding [1993] I. Hinchliffe and T. Kaeding, B+L violating couplings in the minimal supersymmetric Standard Model, Phys. Rev. D 47, 279 (1993).
  * Kubo and Suematsu [2001] J. Kubo and D. Suematsu, Suppressing the mu and neutrino masses by a superconformal force, Phys. Rev. D 64, 115014 (2001), arXiv:hep-ph/0107133 .
  * Dreiner  _et al._ [2006] H. K. Dreiner, C. Luhn, and M. Thormeier, What is the discrete gauge symmetry of the MSSM?, Phys. Rev. D 73, 075007 (2006), arXiv:hep-ph/0512163 .
  * Dreiner  _et al._ [2007] H. K. Dreiner, C. Luhn, H. Murayama, and M. Thormeier, Baryon triality and neutrino masses from an anomalous flavor U(1), Nucl. Phys. B 774, 127 (2007), arXiv:hep-ph/0610026 .
  * Lee  _et al._ [2008] H.-S. Lee, C. Luhn, and K. T. Matchev, Discrete gauge symmetries and proton stability in the U(1)-prime - extended MSSM, JHEP 07, 065, arXiv:0712.3505 [hep-ph] .
  * Luhn and Thormeier [2008] C. Luhn and M. Thormeier, Dirac neutrinos and anomaly-free discrete gauge symmetries, Phys. Rev. D 77, 056002 (2008), arXiv:0711.0756 [hep-ph] .
  * Lee [2011] H.-S. Lee, Minimal gauge origin of baryon triality and flavorful signatures at the LHC, Phys. Lett. B 704, 316 (2011), arXiv:1007.1040 [hep-ph] .
  * Anastasopoulos  _et al._ [2015] P. Anastasopoulos, R. Richter, and A. N. Schellekens, Discrete symmetries from hidden sectors, JHEP 06, 189, arXiv:1502.02686 [hep-th] .
  * Banks and Dine [1992] T. Banks and M. Dine, Note on discrete gauge anomalies, Phys. Rev. D 45, 1424 (1992), arXiv:hep-th/9109045 .
  * Preskill  _et al._ [1991] J. Preskill, S. P. Trivedi, F. Wilczek, and M. B. Wise, Cosmology and broken discrete symmetry, Nucl. Phys. B 363, 207 (1991).
  * Csaki and Murayama [1998] C. Csaki and H. Murayama, Discrete anomaly matching, Nucl. Phys. B 515, 114 (1998), arXiv:hep-th/9710105 .
  * Dine [2016] M. Dine,  _Supersymmetry and String Theory: Beyond the Standard Model_ (Cambridge University Press, 2016).
  * Araki  _et al._ [2008] T. Araki, T. Kobayashi, J. Kubo, S. Ramos-Sanchez, M. Ratz, and P. K. S. Vaudrevange, (Non-)Abelian discrete anomalies, Nucl. Phys. B 805, 124 (2008), arXiv:0805.0207 [hep-th] .
  * Berasaluce-Gonzalez  _et al._ [2011] M. Berasaluce-Gonzalez, L. E. Ibanez, P. Soler, and A. M. Uranga, Discrete gauge symmetries in D-brane models, JHEP 12, 113, arXiv:1106.4169 [hep-th] .
  * Evans  _et al._ [2012] J. L. Evans, M. Ibe, J. Kehayias, and T. T. Yanagida, Nonanomalous discrete $r$ symmetry decrees three generations, Phys. Rev. Lett. 109, 181801 (2012).
  * Tachikawa [2020] Y. Tachikawa, On gauging finite subgroups, SciPost Phys. 8, 015 (2020), arXiv:1712.09542 [hep-th] .
  * Kobayashi and Uchida [2022] T. Kobayashi and H. Uchida, Anomaly of non-Abelian discrete symmetries, Phys. Rev. D 105, 036018 (2022), arXiv:2111.10811 [hep-th] .
  * Anber and Poppitz [2021] M. M. Anber and E. Poppitz, Nonperturbative effects in the Standard Model with gauged 1-form symmetry, JHEP 12, 055, arXiv:2110.02981 [hep-th] .
  * Davighi  _et al._ [2022] J. Davighi, A. Greljo, and A. E. Thomsen, Leptoquarks with Exactly Stable Protons, (2022), arXiv:2202.05275 [hep-ph] .
  * Tong [2017] D. Tong, Line Operators in the Standard Model, JHEP 07, 104, arXiv:1705.01853 [hep-th] .
  * Byakti  _et al._ [2018] P. Byakti, D. Ghosh, and T. Sharma, Note on gauge and gravitational anomalies of discrete ZN symmetries, JHEP 01, 015, arXiv:1707.03837 [hep-ph] .
  * Wang  _et al._ [2022] J. Wang, Z. Wan, and Y.-Z. You, Proton Stability: From the Standard Model to Ultra Unification, (2022), arXiv:2204.08393 [hep-ph] .