# Finite and infinite primes in models of PA

Daniele Mundici  Department of Mathematics and Computer Science “Ulisse Dini”
University of Florence
Viale Morgagni 67/A
I-50134 Florence
Italy<EMAIL_ADDRESS>
###### Abstract.

We characterize models of Peano arithmetic ($\mathsf{PA}$) with infinitely
many infinite primes $p$ such that $p+2$ has no finite prime divisor.

###### Key words and phrases:

Peano arithmetic, prime number, Löwenheim-Skolem theorem, Gödel compactness
theorem.

###### 2000 Mathematics Subject Classification:

Primary: 03H15, 11N13 Secondary: 03C07, 11A07, 11M06, 11U10, 11B25.

##  1\. Statement of the main result

We refer to [1] for background in model theory. The standard model of first-
order Peano arithmetic ($\mathsf{PA}$) is
$\langle\mathbb{N},+,\cdot,S,0\rangle$ where $\mathbb{N}=\\{0,1,2,\dots\\}$,
$S$ is the successor function and $+,\cdot,0$ have their usual meaning, [1,
1.4.11, p.42]. Any other (nonisomorphic) model is said to be nonstandard.

We assume that $\mathsf{PA}$ is consistent. From Gödel’s second incompleteness
theorem it follows that the consistency of $\mathsf{PA}$ can be formalized as
a sentence $\phi$ in the language of $\mathsf{PA}$, but neither $\phi$ nor
$\neg\phi$ is a consequence of $\mathsf{PA}$.

A prime in a model $M$ of $\mathsf{PA}$ is an element $q\geq 2$ of the
universe $U$ of $M$ such that for all $x,y\in U$, if $x\cdot y=q$ then either
$x=q$ or $y=q.$ Any nonstandard model $L$ of $\mathsf{PA}$ has an initial
segment $N_{L}$ isomorphic to the standard model. The primes of $L$ in $N_{L}$
are said to be finite. $L$ also has an infinite set of infinite (also known as
nonstandard) primes.

Throughout, the symbol $p$ will be reserved for a finite prime, and the
adjective “finite” will be omitted when it is clear from the context. For
$k=1,2,\dots$ we let

$p_{k}=\mbox{the $k$th prime in the list 2,3,5,7,11,\ldots}\,.$ (1)

We have not found in the literature a proof of the following result:

######  Theorem 1.1.

There is a countable model $M$ of Peano arithmetic containing an infinite
sequence $q_{1}<q_{2}<\cdots<q_{i}<\cdots$ of infinite primes such that for
each $i=1,2,\dots,$ no finite prime of $M$ is a divisor of $q_{i}+2$.

###### Proof.

Let the language $\mathcal{L}=\\{+,\cdot,S,{0}\\}$ of $\mathsf{PA}$ be
enriched with new constant symbols $c_{1},c_{2},\dots$ . Let
$\mathcal{L}^{*}=\mathcal{L}\cup\\{c_{1},c_{2},\dots\\}$ and
$\mathcal{L}_{n}=\mathcal{L}\cup\\{c_{1},c_{2},\dots,c_{n}\\}$. By an
$\mathcal{L}^{*}$-model of $\mathsf{PA}$ we mean a model $M$ of $\mathsf{PA}$
for the language $\mathcal{L}^{*}$, where each constant symbol
$c_{i}\in\mathcal{L}^{*}$ is interpreted as an element $\bar{c}_{i}$ of the
universe of $M$. If the $\mathcal{L}$-reduct of $M$ is the standard model of
$\mathsf{PA}$ we say that $M$ is a standard $\mathcal{L}^{*}$-model of
$\mathsf{PA}$. Otherwise, $M$ is a nonstandard $\mathcal{L}^{*}$-model of
$\mathsf{PA}$. Standard and nonstandard $\mathcal{L}_{n}$-models of
$\mathsf{PA}$ are similarly defined for the language $\mathcal{L}_{n}$.

To avoid confusion with the notation $\equiv$ in modular arithmetic, we will
let = denote the identity relation symbol (denoted $\equiv$ in [1, 1.3.7,
p.25]). While the relation symbol $\leq$ is not in $\mathcal{L},$ we will use
the abbreviation $x\leq y$ for $\exists z\,\,x+z=y$. The derived relations
$<,>,\geq$ are also freely used.

Let $\alpha_{1}$ be the sentence

$c_{1}\geq SS0,$ (2)

and for $i=2,3,4,\dots,$ let $\alpha_{i}$ be the sentence

$c_{i}>c_{i-1}\ .$ (3)

Next for every $i=1,2,\dots$, let the sentence $\beta_{i}$ be defined by

$\forall x\,\forall y\,\,\,\,x\cdot
y=c_{i}\,\,\,\to\,\,\,(x=c_{i}\,\,\vee\,\,y=c_{i}).$ (4)

Finally, for every $i=1,2,\dots$ and prime $p=2,3,5,\dots$ let $\gamma_{i,p}$
be the sentence

$\forall z\,\,\,\,c_{i}+SS0\not=z\cdot\underbrace{SS\cdots
S}_{\mbox{\it\footnotesize p\, \rm times}}0.$ (5)

With $\alpha_{i},\beta_{i},\gamma_{i,p}$ shorthand for the sentences in
(2)-(5), let

$\Theta^{*}=\\{\alpha_{i},\beta_{i},\gamma_{i,p}\mid
i=1,2,\dots;\,\,\,p=2,3,5,\dots\\}.$ (6)

To prove the theorem we must construct an $\mathcal{L}^{*}$-model of
$\mathsf{PA}$ satisfying $\Theta^{*}$.

To this purpose, for every $i=1,2,\dots$ and prime $p=2,3,5,\dots$ let the
sentence $\omega_{i,p}$ be defined by

$\exists x\,\,\,\,\,\,c_{i}=x\cdot\underbrace{SS\cdots
S}_{\mbox{\it\footnotesize p\, \rm times}}0+\underbrace{SS\cdots
S}_{\mbox{\it\footnotesize p$-1$\, \rm times}}0.$ (7)

Let $N=\langle\mathbb{N},+,\cdot,S,0,\bar{c}_{1},\bar{c}_{2},\dots\rangle$ be
a standard $\mathcal{L}^{*}$-model of $\mathsf{PA}$ with $\bar{c}_{i}$ the
interpretation of the constant symbol $c_{i}$. Then $N$ satisfies
$\omega_{i,p}$ if and only if $\bar{c}_{i}$ is congruent to $p-1$ modulo $p$.
With the notation of [1, 1.3.14, p.28] and [2, p.29],

$N\models\omega_{i,p}\,\,\,\,\,\,\mbox{if and only
if}\,\,\,\,\,\,\bar{c}_{i}\equiv p-1\,\,\,(p).$ (8)

In particular, from $N\models\omega_{i,p}$ it follows that
$\bar{c}_{i}+2\equiv 1\,\,\,(p)$, whence $\bar{c}_{i}+2$ is not a multiple of
$p$. Recalling (5) we may write

$N\models\omega_{i,p}\to\gamma_{i,p}.$

More generally, for each prime $p=2,3,5,\dots,$ the standard model
$\langle\mathbb{N},+,\cdot,S,0\rangle$ satisfies the sentence $\sigma$ given
by

$\forall z[(\exists x\,z=x\cdot\underbrace{SS\cdots S}_{\mbox{\it\footnotesize
p\, \rm times}}0+\underbrace{SS\cdots S}_{\mbox{\it\footnotesize p$-1$\, \rm
times}}0)\to(\exists y\,z+SS0=y\cdot\underbrace{SS\cdots
S}_{\mbox{\it\footnotesize p\, \rm times}}0+S0)].$

The sentence $\sigma$ is derivable from the axioms of $\mathsf{PA}$ and the
rules of first-order logic, $\mathsf{PA}\vdash\sigma.$ 111Most elementary
number theory can be carried over in $\mathsf{PA}.$ Examples include the basic
properties of addition and multiplication and the uniqueness of remainder and
quotient, from which $\mathsf{PA}\vdash\sigma$ routinely follows. For details
see, e.g., Theorem 1.3 in Van Oosten’s lectures “Introduction to Peano
Arithmetic, Gödel Incompleteness and Nonstandard Models”, available online at
https://webspace.science.uu.nl/$\sim$ooste110/syllabi/peanomoeder.pdf
Instantiation $z\mapsto c_{i}$ now yields a proof in $\mathsf{PA}$ of the
sentence $\omega_{i,p}\to\gamma_{i,p}$. In symbols,
$\mathsf{PA}\vdash\omega_{i,p}\to\gamma_{i,p}$ for every $i=1,2,\dots$ and
prime $p=2,3,5,\dots$. Therefore, for every (standard or nonstandard)
$\mathcal{L}^{*}$-model $M$ of $\mathsf{PA}$,

$M\models\omega_{i,p}\to\gamma_{i,p}\,\,\,\,\,\,\,\mbox{for every
$i=1,2,\dots$ and prime $p=2,3,5,\dots$.}$ (9)

Claim: Let $n$ and $k$ be fixed but otherwise arbitrary natural numbers $\geq
1$. Let $p_{k}$ the $k$th prime in the list (1). Then there exists a standard
$\mathcal{L}_{n}$-model
$N_{n,k}=\langle\mathbb{N},+,\cdot,S,0,\bar{c}_{1},\dots,\bar{c}_{n}\rangle$
of $\mathsf{PA}$ such that

$N_{n,k}\models\bigwedge_{i=1}^{n}\alpha_{i}\wedge\beta_{i}\wedge\omega_{i,2}\wedge\omega_{i,3}\wedge\omega_{i,5}\wedge\dots\wedge\omega_{i,p_{k}}.\,$
(10)

In other words, $\bar{c}_{1}<\dots<\bar{c}_{n}$ and for each
$i=1,\dots,n,\,\,\,$ $\bar{c}_{i}$ is a prime number such that $\bar{c}_{i}$
is congruent to $p-1$ modulo $p$ for any prime $p\leq p_{k}.$

To prove this, for each $i=1,\dots,n$ let the sentences
$\omega_{i,2},\omega_{i,3},\dots,\omega_{i,p_{k}}$ in (7) be displayed as
follows:

$\exists x\,c_{i}=x\cdot SS0+S0,\,\,\,\exists x\,c_{i}=x\cdot
SSS0+SS0,\dots,\exists x\,c_{i}=x\cdot\underbrace{SS\cdots
S}_{\mbox{\tiny$p_{k}$\, \rm times}}\,0\,+\underbrace{SS\cdots
S}_{\mbox{\tiny$p_{k}-1$\, \rm times}}0.$

Recalling (8), a standard $\mathcal{L}_{n}$-model satisfies the conjunction

$\bigwedge_{i=1}^{n}\omega_{i,2}\wedge\omega_{i,3}\wedge\omega_{i,5}\wedge\dots\wedge\omega_{i,p_{k}}$

if and only if $\bar{c}_{1},\dots,\bar{c}_{n}$ are solutions of the following
system $\mathcal{S}$ of $n\times k$ modular equations, respectively in the
unknowns $y_{1},\dots,y_{n}\in\mathbb{N}$:

$\begin{array}[]{llll}y_{1}\equiv 1\,\,\,\quad\quad(2)&\quad\quad y_{2}\equiv
1\,\,\,\quad\quad(2)&\quad\cdots&\quad\quad y_{n}\equiv
1\,\,\,\quad\quad(2)\\\ y_{1}\equiv 2\,\,\,\quad\quad(3)&\quad\quad
y_{2}\equiv 2\,\,\,\quad\quad(3)&\quad\cdots&\quad\quad y_{n}\equiv
2\,\,\,\quad\quad(3)\\\ y_{1}\equiv 4\,\,\,\quad\quad(5)&\quad\quad
y_{2}\equiv 4\,\,\,\quad\quad(5)&\quad\cdots&\quad\quad y_{n}\equiv
4\,\,\,\quad\quad(5)\\\
\quad\cdots&\quad\quad\quad\cdots&\quad\cdots&\quad\quad\quad\cdots\\\
y_{1}\equiv p_{k}-1\,\,\,(p_{k})&\quad\quad y_{2}\equiv
p_{k}-1\,\,\,(p_{k})&\quad\cdots&\quad\quad y_{n}\equiv
p_{k}-1\,\,\,(p_{k}).\end{array}$

The Chinese Remainder Theorem, [2, Theorem 1, p.34], provides infinitely many
solutions of $\mathcal{S}$. By direct inspection of the present special case,
the sequence

$p_{1}p_{2}\cdots p_{k}-1,\,\,\,\,\,\,\quad 2p_{1}p_{2}\cdots
p_{k}-1,\,\,\,\,\,\,\quad 3p_{1}p_{2}\cdots p_{k}-1,\,\,\,\,\,\,\quad
4p_{1}p_{2}\cdots p_{k}-1,\dots$

yields all solutions of $\mathcal{S}$. Dirichlet’s theorem on arithmetic
progressions [2, §16, Theorem 1, p.251] ensures that this sequence contains
infinitely many primes. Thus we can extract from it an $n$-tuple
$\bar{c}_{1}<\dots<\bar{c}_{n}$ of primes solving the system $\mathcal{S}.$ We
then obtain a standard $\mathcal{L}_{n}$-model
$N_{n,k}=\langle\mathbb{N},+,\cdot,S,0,\bar{c}_{1},\dots,\bar{c}_{n}\rangle$
of $\mathsf{PA}$ satisfying (10), as required to settle our claim.

From (9) and (10), for all $n,k=1,2,\dots$ we obtain:

$N_{n,k}\models\bigwedge_{i=1}^{n}\alpha_{i}\wedge\beta_{i}\wedge\gamma_{i,2}\wedge\gamma_{i,3}\wedge\gamma_{i,5}\wedge\dots\wedge\gamma_{i,p_{k}}\,.$
(11)

In other words, $\bar{c}_{1}<\dots<\bar{c}_{n}$ and for each
$i=1,\dots,n,\,\,\,$ $\bar{c}_{i}$ is a prime number of $N_{n,k}$ such that
$\bar{c}_{i}+2$ is not divisible by any prime $\leq p_{k}.$

For every $n,k=1,2,3,\dots$ let us define the following subset of the set
$\Theta^{*}$ of $\mathcal{L}^{*}$-sentences introduced in (6):

$\Theta_{n,k}=\\{\alpha_{i},\beta_{i},\gamma_{i,p}\mid
i=1,2,\dots,n;\,\,\,p=2,3,5,\dots p_{k}\\}.$

Any finite subset $\Theta$ of $\Theta^{*}$ is contained in $\Theta_{n,k}$ for
all suitably large $n$ and $k$. From (11) it follows that the standard
$\mathcal{L}_{n}$-model $N_{n,k}$ satisfies $\Theta.$ Therefore, for any
finite subset $\Pi$ of the set $\Pi^{*}$ of axioms of $\mathsf{PA},$ $N_{k,p}$
satisfies $\Pi\cup\Theta$.

Having thus shown that $\Theta^{*}\cup\,\,\Pi^{*}$ is finitely satisfiable, by
Gödel’s compactness theorem [1, Theorem 1.3.22, p.67] there is an an
$\mathcal{L}^{*}$-model $M$ of $\mathsf{PA}$ satisfying $\Theta^{*}$.

By the Löwenheim-Skolem theorem [1, Corollaries 2.1.6-2.1.7, pp.67-68], $M$
may be assumed countable. For each $i=1,2,\dots,$ upon setting
$q_{i}=\bar{c}_{i}$, the proof is complete. ∎

## References

  * [1] C.C.Chang, H.J.Keisler, Model Theory, Studies in Logic and the Foundations of Mathematics, Vol. 73, Third impression. Elsevier, Amsterdam, 1992.
  * [2] K.Ireland, M.Rosen, A Classical Introduction to Modern Number Theory, Second edition, fifth corrected printing. Graduate Texts in Mathematics, Vol. 84, Springer-Verlag, New York, Heidelberg, Berlin, 1990\.


#  21cmVAE: A Very Accurate Emulator of the 21-cm Global Signal

Christian H. Bye Department of Astronomy, University of California, Berkeley,
CA 94720, USA  Department of Physics, McGill University, Montréal, QC H3A 2T8,
Canada  Stephen K. N. Portillo DIRAC Institute, Department of Astronomy,
University of Washington, 3910 15th Ave. NE, Seattle, WA 98195, USA  Anastasia
Fialkov Institute of Astronomy, University of Cambridge, Madingley Road,
Cambridge CB3 0HA, United Kingdom  Kavli Institute for Cosmology, Madingley
Road, Cambridge CB3 0HA, UK

(Received Month Date, 2021; Revised Month Date, 2022; Accepted Month Date,
2022)

###### Abstract

Considerable observational efforts are being dedicated to measuring the sky-
averaged (global) 21-cm signal of neutral hydrogen from Cosmic Dawn and the
Epoch of Reionization. Deriving observational constraints on the astrophysics
of this era requires modeling tools that can quickly and accurately generate
theoretical signals across the wide astrophysical parameter space. For this
purpose artificial neural networks were used to create the only two existing
global signal emulators, 21cmGEM and globalemu. In this paper we introduce
21cmVAE, a neural network-based global signal emulator, trained on the same
dataset of $\sim 30,000$ global signals as the other two emulators, but with a
more direct prediction algorithm that prioritizes accuracy and simplicity.
Using neural networks, we compute derivatives of the signals with respect to
the astrophysical parameters and establish the most important astrophysical
processes that drive the global 21-cm signal at different epochs. 21cmVAE has
a relative rms error of only 0.34%—equivalently 0.54 mK—on average, which is a
significant improvement compared to the existing emulators, and a run time of
0.04 seconds per parameter set. The emulator, the code, and the processed
datasets are publicly available at https://github.com/christianhbye/21cmVAE
and through https://zenodo.org/record/5904939.

early universe — cosmology — astronomy software

††journal: ApJ

##  1 Introduction

The 21-cm line of neutral hydrogen is one of the most promising probes of
Cosmic Dawn and the Epoch of Reionization (EoR). Emitted by neutral hydrogen
at redshifts $z>6$, the signal is redshifted to frequencies below 200 MHz, and
can therefore be observed by radio telescopes (Tozzi et al., 2000). The global
21-cm signal, obtained by averaging the spectrum across all sky, traces the
cosmology and astrophysics of the high-redshift universe. The intensity of the
signal is observed in contrast to the background radiation, which is normally
assumed to be the CMB, and is quantified by the differential brightness
temperature. This temperature depends on the excitation temperature of the
21-cm transition (the spin temperature), the temperature of the background
radiation, and the abundance of neutral hydrogen. During Cosmic Dawn and the
Epoch of Reionization, processes including the Wouthuysen-Field effect
(Wouthuysen, 1952; Field, 1959), X-ray heating (Madau et al., 1997), and
reionization of neutral hydrogen affect the differential brightness
temperature and leave characteristic features in the global signal (Shaver et
al., 1999). The Wouthuysen-Field effect describes how the spin states of
neutral hydrogen are mixed through absorption and re-emission of Ly$\alpha$
photons, thus coupling the spin temperature to the kinetic temperature of the
hydrogen gas. This effect was dominant in the beginning of the Cosmic Dawn
when the gas temperature was cooler than the background radiation temperature,
hence making the differential brightness temperature negative. Later, the
first X-ray sources heated the intergalactic medium (IGM), making the gas go
from absorption to emission against the background. Finally, the abundance of
neutral hydrogen decreased due to reionization of the gas and the signal
vanished. We refer to Furlanetto et al. (2006), Pritchard & Loeb (2012), and
Barkana (2016) for in-depth reviews of these physical processes.

There are several ongoing efforts to detect the global signal with one
reported detection (Bowman et al., 2018), made by the Experiment to Detect the
Global EoR Signature (EDGES, Bowman & Rogers, 2010; Monsalve et al., 2017,
2018, 2019). Other experiments include PRIZM (Probing Radio Intensity at
high-Z from Marion, Philip et al., 2019), MIST (Mapper of the IGM Spin
Temperature, http://www.physics.mcgill.ca/mist/), SARAS (Shaped Antenna
measurement of the background RAdio Spectrum, Singh et al., 2021), REACH
(Radio Experiment for the Analysis of Cosmic Hydrogen, de Lera Acedo, 2019;
Cumner et al., 2021), and DAPPER (Dark Ages Polarimeter PathfindER, Burns et
al., 2019). The EDGES detection showed a deep and narrow absorption profile,
which cannot be explained by standard cosmological models. In particular, the
lower 99% confidence bound on the best-fit amplitude was 0.3 K, which is
approximately 50% greater than the largest predicted amplitude (Bowman et al.,
2018).

With the surprising first results and awaited new measurements, it is
important to model the range of possible 21-cm signals from the early universe
and explore the associated astrophysical parameter space. To this end, a
flexible method is required to realize the global signal efficiently, allowing
parameters to be constrained from measurements with the use of sampling
techniques such as MCMC (Liu & Shaw, 2020). Simulations of the 21-cm signal
(e.g. Visbal et al., 2012; Fialkov et al., 2014) take a few hours to run and,
thus, cannot be directly employed in a parameter estimation pipeline. Instead,
emulators trained on the results of these simulations are becoming a popular
tool for fast model generation. Currently available software include emupy
(Kern et al., 2017), which emulates the 21-cm power spectrum; 21cmGAN (List &
Lewis, 2020), which generates tomographic samples of the 21-cm brightness
temperature; and the two global signal emulators 21cmGEM (Cohen et al., 2020)
and globalemu (Bevins et al., 2021). Given seven astrophysical parameters,
21cmGEM calculates five auxiliary parameters and uses a series of neural
networks, a bagged tree classifier, and principal component analysis to
emulate the global signal over redshifts $z=5-50$. It is fast and accurate:
the emulator has a running time of 0.16 seconds per parameter set and an
average relative error of 1.59%. The emulator has already been used to
constrain the astrophysical parameters using the data of the EDGES High Band
experiment (Monsalve et al., 2019). globalemu emulates the global signal from
the same parameters as 21cmGEM, using just a single neural network. The
simpler prediction algorithm emulates the global signals significantly faster,
with a run time of only 1.3 ms, and with a smaller average relative error of
1.12%.

In this paper we present 21cmVAE, a new emulator of the global signal from the
same seven astrophysical parameters used in 21cmGEM and globalemu: the star
formation efficiency ($f_{*}$), the minimum circular velocity of star-forming
halos ($V_{c}$), the X-ray radiation efficiency ($f_{X}$), the optical depth
($\tau$) of the Cosmic Microwave Background radiation (CMB), the power-law
slope ($\alpha$) and low energy cutoff ($\nu_{\small\textrm{min}}$) of the
X-ray spectral energy distribution (SED), and the mean free path of ionizing
photons ($R_{\small\textrm{mfp}}$). The emulator uses the same dataset as
21cmGEM and globalemu, with global signals simulated from the parameters using
the method described in e.g. Cohen et al. (2020). We refer to Cohen et al.
(2020) for more details on the modeling.

The objective of this work is to use artificial neural networks to learn
relationships between parameters and signals without enforcing any physical
models. Compared to 21cmGEM we aim to use a simpler prediction algorithm that
does not calculate auxiliary parameters or depend on the input parameters.
However, unlike globalemu, we do not aim to make the fastest or simplest model
that meets a target performance—instead we explore a wider range of models and
prioritize very accurate predictions. As a result, our emulator uses only one
neural network which predicts global signals given the seven astrophysical
parameters. This allows 21cmVAE to be flexible and predict a wide range of
signals. The emulator is written in Python, using the machine learning
libraries TensorFlow (Abadi et al., 2015) and Keras (Chollet et al., 2015),
making it easy to modify or retrain. 21cmVAE is available on GitHub, where the
dataset and documentation also can be found.

This paper has six sections. After the introduction, section 2 discusses how
we designed and optimized our models and the performance and speed of the
emulator is presented in section 3. In section 4, we evaluate the impact of
the astrophysical parameters on the global signal and in section 5, we use the
emulator to interpret an intermediate (latent) representation of the signals.
We conclude in section 6.

##  2 Methods

This section assumes familiarity with basic principles and terminology of
neural networks, we refer to Goodfellow et al. (2016) for a review of this
topic.

###  2.1 Architecture

The emulator has a 7-dimensional input layer (for 7 astrophysical parameters)
and a 451-dimensional output layer, since it outputs realizations of global
signals at 451 redshifts. It also has four hidden layers—of 288, 352, 288, and
224 dimensions, respectively—which all are fully connected and use the
activation function ReLU (Fukushima, 1969). We also trained an emulator which
utilized an autoencoder to create a low-dimensional representation of the
global signals; this emulator appeared to be both less accurate and slower,
but is described in Appendix A. We earlier tried to use a variational
autoencoder (Kingma & Welling, 2013), which uses a regularized latent space
that we believed would preserve data structures better than the latent space
of a vanilla autoencoder and thus work better in the emulator. However, we
found that this regularization came at the cost of increased reconstruction
error of the autoencoder and that the emulator error was larger with a
variational autoencoder than a standard autoencoder.

We use the root mean squared (rms) error as a fraction of signal amplitude to
evaluate the performance of the emulator. This is defined for each signal by:

$\textrm{Error}=\frac{\sqrt{{\langle(T(\nu)-\hat{T}(\nu))^{2}\rangle}}}{\max(|T(\nu)|)}.$
(1)

Here, $\hat{T}(\nu)$ denotes the frequency-dependent signal predicted by the
emulator, whereas $T(\nu)$ represents the simulated signal from the dataset.
The figure of merit is the same as used by 21cmGEM, making the results
directly comparable.

###  2.2 Dataset

We use the publicly available dataset created for 21cmGEM of $29,641$ signals,
including a training set with $27,455$ signals and a test set with $2,186$
signals. We restrict the datasets to only include signals in the parameter
ranges given by Cohen et al. (2020) (the ranges for the test set in brackets):
$f_{*}=0.0001-0.50\;(0.0003-0.50)$, $V_{c}=4.2-100\;(4.2-76.5)$ km
$\textrm{s}^{-1}$, $f_{X}=0-1000\;(0-10)$, $\tau=0.04-0.2\;(0.055-0.1)$,
$\alpha=1.0-1.5\;(1.0-1.5)$, $\nu_{\small\textrm{min}}=0.1-3.0\;(0.1-3.0)$
keV, $R_{\textrm{\small mfp}}=10-50\;(10-50)$ Mpc. Figure 1 shows a
representative sample of the global signal models in the training set.

Figure 1: 500 global signal models randomly drawn from the training set.

Approximately $10\%$ of the training set is used as a validation set. This
results in a total of $28,996$ global signals split into a training set with
24,562 signals, a validation set with $2,730$ signals, and a test set with
$1,704$ signals. The three datasets were divided randomly, ensuring
distributions that are statistically similar, with the only difference being
the restrictions on the parameter ranges. The larger parameter ranges on the
training and validation set make the emulator capable of exploring a variety
of astrophysical scenarios, but we keep narrower ranges on the parameters in
the test set to make the results comparable to 21cmGEM and globalemu.

Before training the emulator, the training signals are preprocessed. This is
done by computing the mean temperature at each frequency across all signals
and subtracting this from the respective signals. Afterwards, every signal is
divided by the standard deviation across all signals and frequencies. The
preprocessed signals thus have zero mean in every frequency bin and are scaled
to units of standard deviation. Standard practice is to divide by the standard
deviation in each frequency bin; we deviate from this because the low standard
deviation between signals at small frequencies would make the preprocessed
signals blow up.

We train the emulator on the signals in the training set, while monitoring the
performance on the training set and validation set. Only the errors on the
training set are being back-propagated through the neural networks; thus, the
errors on the validation set measures whether the model is able to emulate
signals that are statistically similar to the training signals but not
propagated through the network. The validation error is therefore an important
metric for overfitting and the ability of the emulator to generalize to unseen
signals. We use it to regulate the learning rate of the model, to trigger
early stopping, and during hyperparameter tuning: see sections 2.3 and 2.4 for
more details.

The test set is not seen by the emulator until after training and
hyperparameter tuning. The performance on the test set is therefore a measure
of how well the emulator can predict global signals from parameters it was not
optimized for. Hence, the test error is the best estimate of the performance
of the emulator in a real setting and is therefore the figure of merit we use.

###  2.3 Training

We trained the emulator for 350 epochs, using minibatches with 256 signals in
each, and a loss function defined as the square of the figure of merit (Eq.
1). We used the Adam (Kingma & Ba, 2014) optimizer and an initial learning
rate of 0.01. Together, these define the gradient descent algorithm used to
minimize the loss function.

During training, we also used a learning rate schedule, which reduces the
learning rate if the validation loss does not decrease over 5 epochs, and
early stopping which stops the training if the validation loss does not
decrease over 15 epochs. We also saved the model weights and biases after each
epoch and loaded the weights and biases that correspond to the epoch with the
smallest validation loss after training. This is normally towards the end of
the training, but will be at an earlier epoch in the case of overfitting.

###  2.4 Hyperparameter Tuning

In our neural networks, there are several parameters that cannot be optimized
during training. These are called hyperparameters; examples include the number
of layers in the network, dimensionality of each layer, and the loss
functions. We performed hyperparameter tuning to optimize the number of layers
and dimensionality of each layer. Although there exist dedicated Python
packages to do this, we wrote our own script based on a random grid search to
get full control over the tuning. We specified an appropriate hyperparameter
space, used the script to randomly generate parameters in the space and built
the emulator with these parameters. The model was trained and the validation
loss of the emulator was saved. We trained 500 emulators with different
hyperparameters. The hyperparameters associated with the emulator with the
lowest validation loss were used to build the final product; these define the
architecture which is described in section 2.1.

##  3 Performance

###  3.1 Test Errors

The training algorithm is stochastic, hence training the same model twice on
the same dataset may yield different results. We therefore trained the
emulator 20 times after optimizing the architecture with hyperparameter
tuning. For each of the 20 trials, we computed the error for each signal in
the test set. We get a mean error across signals for each trial; the
distribution of these mean errors is showed in Appendix A, in Figure 6.
21cmVAE has a mean error across the 20 trials of $0.354\%\pm 0.001\%$ and a
median error of $0.305\%\pm 0.001\%$. The results of the best trial of 21cmVAE
are displayed in the histogram in Figure 2, which also shows how the
performance compares to 21cmGEM and globalemu.

Figure 2: The histogram shows the distribution of errors for the test set of
1704 astrophysical signals with mean, median, and 95th percentile marked
(solid lines). It also shows the results for globalemu (dash-dotted line) and
21cmGEM (dashed lines). Note that globalemu does not report the median error
and that the 95th percentile errors of globalemu and 21cmGEM are 2.41% and
3.49% respectively, hence out of the range used for the x-axis here.

We list the relative (as defined in Eq. 1) and absolute (the rms in mK) errors
of the best trial across the entire simulated frequency range (approximately
28-237 MHz) and for selected frequency bands in Table 1: 50-100 MHz
(corresponding to EDGES Low Band), 60-120 MHz (corresponding to EDGES Mid
Band), 90-200 MHz (corresponding to EDGES High Band), and 50-200 MHz.

$\nu$ (MHz) | $z$ | Mean Error (%) | Median Error (%) | Mean Error (mK) | Median Error (mK)
---|---|---|---|---|---
28-237 | 5.0-50.0 | 0.34 | 0.29 | 0.54 | 0.50
50-100 | 13.3-27.4 | 0.36 | 0.29 | 0.50 | 0.45
60-120 | 10.9-22.6 | 0.38 | 0.30 | 0.57 | 0.52
90-200 | 6.2-14.7 | 0.82 | 0.56 | 0.90 | 0.79
50-200 | 6.2-27.4 | 0.45 | 0.39 | 0.71 | 0.65
Table 1: The mean and median of the error of 21cmVAE across the full frequency
range the global signals are sampled at, and across examples of frequency
bands used by the EDGES experiment.

The mean error across the entire frequency range is 0.54 mK or 0.34% of the
signal amplitude. For comparison, the rms between the global signal predicted
from all parameters in the middle of the test range and the global signals of
a model with all the parameters simultaneously shifted by $\pm 1\%$ is 0.70
mK. Hence, the emulator is sensitive to $\mathcal{O}(1\%)$ changes in the
astrophysical input parameters.

For a sample view of the performance of 21cmVAE on the test set, we show in
Figure 3 the predicted signals with error closest to the 10th percentile, the
median, and the 95th percentile, as well as the signal with the largest error.
This can be directly compared to the analogous Figure 9 of Bevins et al.
(2021) and Figure 13 of Cohen et al. (2020), which show the performances of
globalemu and 21cmGEM, respectively.

Figure 3: Comparison of true signal (blue solid line) in the test set and
emulated signals (orange dashed line) for the case with error closest to the
10th percentile, median, 95th percentile, and the largest error. The errors
and the values of the astrophysical parameters of each corresponding model are
shown in each panel of the figure.

###  3.2 Speed

We measure the time between inputting astrophysical parameters to when an
output is produced on a computer with a GeForce RTX 2080 Ti Rev. A. GPU. By
randomly drawing parameters from the parameter ranges of the training set a
thousand times, we find that it takes on average $0.0414\pm 0.0007$ s to
predict one global signal. We also measured the time it takes to predict 1000
signals simultaneously—that is, inputting 1000 parameter combinations to the
emulator at once—obtaining an average time of $0.0418\pm 0.0002$ s.

##  4 Impact of Astrophysical Parameters

We use 21cmVAE to investigate the impact of each astrophysical parameter on
the global signal, both qualitatively and quantitatively. First, we show
visually how each parameter affects the emulated global signal. We use the
signal emulated from the mean parameters in the test set as our nominal model
and vary each parameter uniformly between its minimal and maximal value in the
test set. The mean parameters are given by
$\bar{X}\equiv\\{\log{f_{*}}=-0.800$, $\log{(V_{c}/1\textrm{km
s}^{-1})}=1.4803$, $\log{f_{X}}=-0.1190$, $\tau=0.07$, $\alpha=1.25$,
$\nu_{\small\textrm{min}}=0.85$ keV, $R_{\textrm{\small mfp}}=29.60$ Mpc}.

Figure 4: The effect of each astrophysical parameter on the emulated global
signal. The black, dashed line in each panel is the nominal model. Each panel
shows the global signal obtained by varying one parameter, while keeping all
the others fixed at their nominal values. The color of each signal represents
the value of the varied parameter—on a logarithmic (base 10) scale in the
cases of $f_{*}$, $V_{c}$, and $f_{X}$.

Figure 4 shows the effect of each parameter on the global signal. In general,
larger values of $f_{*}$ and smaller values of $V_{c}$ are both associated
with the center of the absorption trough being shifted to earlier times (lower
frequencies); this is as expected since this combination of parameters
corresponds to earlier star formation. Furthermore, we note that the amplitude
of the global signal is regulated by X-ray heating: it increases with
decreasing $f_{X}$ and increasing $\nu_{\small\textrm{min}}$, both of which
correspond to less X-ray heating. Less heating also lead to later absorption,
as can be seen in the figure. Whereas the four parameters discussed have
combined a significant impact on the center, amplitude, and width of the
absorption trough, the global signal is much less sensitive to the last three
parameters. As anticipated, $\tau$ is only important at late times and
regulates the amplitude during reionization. The variations in amplitude are
below the 10 mK-level, but the 21-cm line can be seen in emission at low
redshifts for sufficiently small values of $\tau$. Being associated with X-ray
heating, the global signal varies with $\alpha$ along the same trends as it
varies with $f_{X}$ and $\nu_{\small\textrm{min}}$, but the variations are
smaller and $\alpha$ is more constrained than the other parameters. Finally,
$R_{\textrm{\small mfp}}$ has no apparent effect on the global signal at the
scales considered here.

Since 21cmVAE maps astrophysical parameters to realizations of the global
signal, we can compute the derivative of this map with respect to the input
parameters. This allows us to quantify the impact I of each astrophysical
parameter on the global signal, which we define as the rms of the derivative
of the mean global signal—that is, the global signal corresponding to the mean
parameters of the test set ($\bar{X}$)—with respect to its inputs. We define
the impact of each parameter $\bar{X}_{i}$ by

$I(\bar{X}_{i})\equiv\sqrt{{\Bigg{\langle}\left(\frac{\rm{d}\bar{T}(\nu)}{\rm{d}X_{i}}\right)^{2}\bigg{\rangle}}_{\nu}}\Delta
X_{i}.$ (2)

We also compute the impact of the combinations $\log{(f_{*}\cdot f_{X})}$,
which is proportional (up to the minor effect of X-ray SED) to the total
amount of X-rays that goes into heating of the IGM, and
$\log{(f_{*}/V_{c}^{3})}$, which measures the efficiency of the Wouthuysen-
Field effect and is proportional to the total mass of gas in dark matter halos
converted into stars (see e.g Barkana, 2016). The first factor in the above
equation,
$\sqrt{{\Bigg{\langle}\left(\frac{\rm{d}\bar{T}(\nu)}{\rm{d}X_{i}}\right)^{2}\bigg{\rangle}}_{\nu}}$,
is proportional to the Fisher information for an instrument that has constant
error bars for each pixel in the given frequency band. The other factor,
$\Delta X_{i}$, is the uncertainty on the parameter, which we take to be the
width of the parameter range in the test set. Table 2 show the impact of each
parameter in the frequency bands that we reported the results for in Table 1.
While this analysis considers derivatives at the mean parameters in our test
set, it could be repeated at any point of interest in the parameter space.

$\nu$ (MHz) | z | I($\log{f_{*}}$) | I($\log{V_{c}}$) | I($\log{f_{X}}$) | I($\tau$) | I($\alpha$) | I($\nu_{\small\textrm{min}}$) | I($R_{\textrm{\small mfp}}$) | I($\log{(f_{*}\cdot f_{X})}$) | I($\log{(f_{*}/V_{c}^{3})}$)
---|---|---|---|---|---|---|---|---|---|---
28-237 | 5.0-50.0 | 64.56 | 91.64 | 121.46 | 6.42 | 13.88 | 114.28 | 0.49 | 269.28 | 218.68
50-100 | 13.3-27.4 | 104.76 | 156.36 | 189.19 | 6.71 | 21.57 | 177.64 | 0.79 | 420.62 | 365.40
60-120 | 10.9-22.6 | 124.60 | 175.36 | 236.13 | 8.27 | 26.99 | 222.06 | 0.85 | 521.96 | 420.31
90-200 | 6.2-14.7 | 98.47 | 107.93 | 224.83 | 13.09 | 25.77 | 211.76 | 0.47 | 486.01 | 293.91
50-200 | 6.2-27.4 | 93.88 | 133.26 | 176.63 | 9.27 | 20.19 | 166.18 | 0.69 | 391.53 | 318.00
Table 2: The impact in mK of each parameter on the mean global signal across
the frequency bands considered in Table 1.

The results in Table 2 indicate that the global signal at redshifts $z=5-50$
is most sensitive to changes in the combinations $f_{*}\cdot f_{X}$ and
$f_{*}/V_{c}^{3}$. The four parameters with the greatest impact in this
redshift range are $f_{X}$, $\nu_{\small\textrm{min}}$, $V_{c}$, and $f_{*}$
and the three parameters with the smallest impact are $R_{\textrm{\small
mfp}}$, $\tau$, and $\alpha$. The four parameters with the largest
quantitative impact are the same as the parameters that visually appeared to
have the most impact on the global signals. Thus, the quantitative impact
analysis confirms the intuition provided in Figure 4: The global signal is
most sensitive to the strength and time of onset of star formation and heating
in the redshift range $z=5-50$.

Table 2 also shows how the impact of the parameters changes with the observed
frequency band, hence how it changes with time. All the parameters except for
$\tau$ have their largest impact in the redshift range $z=10.9-22.6$, which is
expected since the global signal used in this analysis both have its
absorption through and go into emission in this redshift range. The most
important parameters in this range are $\log{(f_{*}\cdot f_{X})}$ and
$\log{(f_{*}/V_{c}^{3})}$ with impact values of 521.96 mK and 420.31 mK,
respectively. These also have the greatest impact in the range $z=6.2-14.7$,
but their impact decrease to about 90% and 70% respectively of their values in
the range $z=10.9-22.6$. The only parameter that increases in impact is
$\tau$, which goes from 8.27 mK to 13.09 mK. This shows that $\tau$ becomes
comparatively more important at lower redshifts. On the other hand, the other
reionization parameter, $R_{\textrm{\small mfp}}$, has consistently the
smallest impact and does not appear to become more important at later times.

##  5 Interpreting the Latent Space

Having seen how the global signals vary with the astrophysical parameters, we
analyze how the last hidden layer change with the parameters. We denote this
representation the latent representation or the latent space. It is
224-dimensional, since the last hidden layer has 224 neurons (in comparison,
the frequency representation is 451-dimensional since the global signals are
sampled at 451 frequencies). The latent representation is related to the
output global signals by only one linear transformation and the additive bias
of the neural network. Investigating the structure of this representation can
therefore illustrate which of the seven astrophysical parameters have the
strongest effect on the global signals in the frequency range $\nu=28-237$
MHz.

Figure 5: A t-SNE visualization of the latent representation colored by the
seven astrophysical input parameters (panels a)-g)) and the combinations
$f_{*}\cdot f_{X}$ (panel h)) and $f_{*}/V_{c}^{3}$ (panel i)). The points
shown are the values of the neurons in the last hidden layer of the emulator
associated with each global signal. The first three parameters and the two
combinations are given on log scales (base 10), otherwise the units are the
same as used elsewhere in the text. In general, the parameters that are most
important for the global signal in the frequency band $\nu=28-237$ MHz are
most separated in the t-SNE representation of the latent space.

As an illustration, we plot (Figure 5) the value of each of the 224 neurons in
the last hidden layer for each of the $1,704$ signals from the test set
projected to a plane using the t-SNE technique (t-Stochastic Neighbor
Embedding, van der Maaten & Hinton, 2008). This technique preserves relative
distances, meaning that points close in the latent space are also close in the
2D projection. We require that the 200 closest points, or about 10%, are
counted as neighbors (i.e., setting the perplexity parameter for t-SNE to
200). For each of the seven astrophysical parameters, we color a copy of this
plot according to the value of the parameters and show the results in Figure
5. To gain further insight we also show the combinations $f_{*}\cdot f_{X}$
and $f_{*}/V_{c}^{3}$ discussed in Section 4.

Since Figure 5 is an attempt at visualizing a 224-dimensional space in 2D, the
shapes and structures are not very meaningful. However, as t-SNE preserves
relative distances, the relative placement of the points contains information.
By visually exploring Figure 5, we find, as anticipated, that the structure of
the latent space is most sensitive to changes in the astrophysical parameters
regulating star formation (namely $f_{*}$ and $V_{c}$ shown in panels a and b
respectively) and X-ray heating (via the dependence on $f_{X}$ and, to a
lesser degree, on $\nu_{\small\textrm{min}}$, panels c and f respectively) as
these parameters have the sharpest effect on the global signal within the
explored frequency range. We can also see how the latent space is structured:
The upper right part of the space is a region with large $f_{*}$ (panel a) and
small $V_{c}$ (panel b), indicating that this part of the latent space is
associated with early star formation. This behavior is even more clearly seen
on panel (i) which is colorcoded with respect to $f_{*}/V_{c}^{3}$. Similarly,
the latent space separates $f_{X}$ (panel c), with large values to the upper
left and smaller values to the lower right and $\nu_{\textrm{\small min}}$
(panel f) with smaller values to the upper left and larger values to the lower
right, effectively corresponding to stronger and earlier heating respectively
(Fialkov et al., 2014) and, thus, agreeing with the distribution of $f_{X}$.
In other words, the amount of heating generally increases upwards and to the
left in the t-SNE representation of the latent space, which is highlighted in
panel (h) corresponding to $f_{*}\cdot f_{X}$.

We also find that the effects of the parameters regulating reionization (in
our case it is the CMB optical depth and the mean free path of the ionizing
photons, panels d and g respectively) as well as the slope of X-ray SED (panel
e) is not reflected in the t-SNE plots and there is no apparent structure
across the corresponding panels. In order to minimize the loss of the
emulator, it was not necessary to organize the hidden layer in a way that
sorts these parameters in the t-SNE projection, indicating that they have
smaller effects on the global signals when explored across the broad frequency
range $\nu\approx 28-237$ MHz.

In summary, we find—as expected—that processes of X-ray heating and
Wouthuysen-Field coupling have the strongest impact on the global signal in
the explored frequency range with the latent space separating the amount of
star formation along one diagonal in the t-SNE representation and the X-ray
heating along the other diagonal. In contrast, the parameters regulating
reionization as well as the slope of X-ray SED have a subdominant effect on
the apparent structure of the latent space within the broad frequency range
considered.

This is largely consistent with our findings in section 4, where we saw
visually that the center, amplitude, and width of the global signals are most
sensitive to the amount of X-ray heating and star formation, and
quantitatively that the parameters regulating these effects have the greatest
impacts. Larger amplitudes are associated with less X-ray heating and, to a
lesser degree, more star formation. Since these are largely separated along
different diagonals, the t-SNE representation does not separate amplitudes to
the same extent as it separates the amount of X-rays and star formation.
However, we may loosely identify the lower, right region of the latent space
with global signals of large amplitude, since it is a region with low $f_{X}$
and, in general, above average values of $f_{*}$ and
$\nu_{\small\textrm{min}}$. The opposite side of the plot is a region of
smaller amplitudes, with large $f_{X}$, more small values of $f_{*}$ and small
$\nu_{\small\textrm{min}}$. On the other hand, the center and width of the
global signals appear to be separated with increasing width and the center
shifted to high frequencies in the bottom half of the latent space (following
generally the separation of $f_{*}\cdot f_{X}$).

The parameters $\tau$, $\alpha$, and $R_{\textrm{\small mfp}}$ both have the
smallest effect on the global signal and do not separate the latent space in
the t-SNE representation. This shows that the emulator, as anticipated, does
not prioritize separating parameters that do not significantly change the
shape of the global signal.

##  6 Conclusions

We have presented 21cmVAE, a new emulator of the 21-cm global signal, which
predicts signals based on a seven-parameter input. The prediction pipeline is
simple, with one neural network, whose architecture is optimized with
hyperparameter tuning. The code used to optimize the network is publicly
available, together with the final product. 21cmVAE emulates global signals in
approximately 0.04 seconds on average and with a mean rms error of 0.34% of
the signal amplitude, a significant improvement in error from existing
emulators. The absolute error is on average smaller than the rms between two
signals with parameters changed by 1%.

Compared to globalemu, which is the fastest and most accurate existing
emulator of the global 21-cm signal, as well as the most similar in
architecture, 21cmVAE has overall smaller errors in prediction of the same
global signals: both the mean and maximum errors of 21cmVAE are more than a
factor of 3 smaller than the mean and maximum errors of globalemu over the
redshift range $z=5-50$. This is due to different priorities: globalemu aims
to be as simple and fast as possible given a target accuracy of about 10% of
the expected noise of the REACH experiment, whereas 21cmVAE aims to be as
accurate as possible without compromises. globalemu is indeed able to emulate
the global signals in 1.3 ms, which is even faster than 21cmVAE.

By exploring different representation of the global signal and its
derivatives, we were able to qualitatively and quantitatively establish which
of the model parameters create the most significant changes in the global
21-cm signal across a broad frequency range. As anticipated, we find that
processes of X-ray heating and Wouthuysen-Field coupling have the strongest
impact, while the parameters regulating reionization as well as the slope of
X-ray SED have no apparent effect on the latent representation for the
redshift range $z=5-50$. As the analysis in Section 4 show, the impact of each
parameter depends on the frequency band considered, and the latent
representation will therefore depend on the frequency sampling of the global
signals in the training set. The visual latent representation and the
quantitative derivative analysis are a potentially powerful diagnostic that
can point out dominant astrophysical processes and help optimizing theoretical
modeling when targeting specific frequency bands of different experiments.

In summary, 21cmVAE achieves unprecedentedly small errors for a range of 21-cm
models across a wide frequency band. Combined with the short running time and
the implementation in Python, this makes 21cmVAE ideal for parameter fitting
such as MCMC.

## Acknowledgments

SKNP acknowledges support from the DIRAC Institute in the Department of
Astronomy at the University of Washington. The DIRAC Institute is supported
through generous gifts from the Charles and Lisa Simonyi Fund for Arts and
Sciences, and the Washington Research Foundation. AF was supported by the
Royal Society University Research Fellowship.

##  Appendix A Autoencoder-based Emulator

In addition to 21cmVAE, we trained an autoencoder to generate a low-
dimensional—latent—representation of the global signals and an emulator based
on the autoencoder. These two networks were trained the same way as 21cmVAE
and the hyperparameters were optimized with the same hyperparameter tuner. The
emulator maps astrophysical parameters to the latent representation, which we
set to 9 dimensions after hyperparameter tuning, and then decoded by the
autoencoder. By compressing the data to the latent space, the autoencoder is
forced to learn the most robust features of the global signal. We believed
that this would aid the emulator in predicting the global signals and that the
9 latent parameters would be easier to interpret as combinations of the input
astrophysical parameters than the hidden layers of 21cmVAE which all have more
than 200 dimensions. Despite this, 21cmVAE actually performs better than the
autoencoder-based emulator and can be interpreted both qualitatively and
quantitatively, as done in sections 4 and 5. We still show the methods and
results here for comparison.

The autoencoder attempts to reconstruct global signals but has a 9-dimensional
latent layer which forces it to reduce the dimensionality of each global
signal from 451 to 9. The encoder has one hidden layer with 352 dimensions,
whereas the decoder has two hidden layers with 32 and 352 dimensions,
respectively. The emulator takes in the 7 astrophysical parameters and outputs
the latent representation, using four hidden layers of dimensions 352, 352,
352, and 224.

As with 21cmVAE, we trained the tuned autoencoder-based emulator 20 times and
computed the mean test error for each trial. These are shown in Figure 6.

Figure 6: Histogram of the mean errors of the 20 trials for the autoencoder
(blue), 21cmVAE (orange), and the autoencoder-based emulator (gray).

21cmVAE has a mean error across trials of $0.354\%\pm 0.001\%$ and a median
error of $0.305\%\pm 0.001\%$, whereas the autoencoder-based emulator has a
mean error of $0.44\%\pm 0.01\%$ and a median error of $0.39\%\pm 0.01\%$. We
see that the distribution of mean errors of 21cmVAE is strictly at smaller
errors than the distribution of errors of the autoencoder-based emulator.
However, the errors due to the autoencoder itself are in general smaller than
the errors of 21cmVAE, as shown in Figure 6. The autoencoder has a mean error
of $0.332\%\pm 0.003\%$ and a median error of $0.292\%\pm 0.002\%$. The errors
of autoencoder-based emulator would be identical to the autoencoder errors if
it could perfectly map the astrophysical parameters to the autoencoder latent
parameters; thus, the difference between these distributions is due to errors
in the map from astrophysical parameters to latent parameters. An improved
autoencoder-based emulator could reduce this difference, but the autoencoder
errors are likely the limit of how small the autoencoder-based emulator errors
can be. For the best trial, the mean error of the autoencoder-based emulator
on the test set is $0.39\%$ and the median error is $0.35\%$. This is worse
than 21cmVAE (mean: $0.34\%$, median: $0.29\%$), but a significant improvement
over 21cmGEM (mean: $1.59\%$, median: $1.30\%$). The autoencoder that is used
by that emulator has a mean error of $0.33\%$ and a median error of $0.29\%$.

##  Appendix B Dataset size

The performance of the emulator is correlated with the size of the training
dataset. To test this, we randomly sample subsets of the training set and
evaluate the performance of the emulator trained on each of the subsets. The
subsets range in size from 5% to the full dataset in increments of 5%. We
sample the subsets at each given size ten times and average the error across
the samples. This ensures that the results only depends on the size of the
subsets and not the composition of the subsets. This relationship is displayed
in Figure 7. We see that the emulator error steadily decreases with
increasingly more signals until the dataset size is $\sim 35\%$ (about 8600
signals), after which it only slowly decrease. Therefore, the marginal effect
of increasing the size of the training set used to train 21cmVAE would likely
be small.

Figure 7: Test error versus the size of the training set as a percentage of
the full training set for the emulator. The points and errorbars represent the
mean and standard error in the mean respectively.

## References

  * Abadi et al. (2015) Abadi, M., Agarwal, A., Barham, P., et al. 2015, TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.  https://www.tensorflow.org/
  * Barkana (2016) Barkana, R. 2016, Phys. Rep., 645, 1, doi: 10.1016/j.physrep.2016.06.006
  * Bevins et al. (2021) Bevins, H. T. J., Handley, W. J., Fialkov, A., de Lera Acedo, E., & Javid, K. 2021, MNRAS, 508, 2923, doi: 10.1093/mnras/stab2737
  * Bowman & Rogers (2010) Bowman, J. D., & Rogers, A. E. E. 2010, Nature, 468, 796, doi: 10.1038/nature09601
  * Bowman et al. (2018) Bowman, J. D., Rogers, A. E. E., Monsalve, R. A., Mozdzen, T. J., & Mahesh, N. 2018, Nature, 555, 67, doi: 10.1038/nature25792
  * Burns et al. (2019) Burns, J. O., Bale, S., & Bradley, R. F. 2019, in American Astronomical Society Meeting Abstracts, Vol. 234, American Astronomical Society Meeting Abstracts #234, 212.02
  * Chollet et al. (2015) Chollet, F., et al. 2015, Keras, https://keras.io
  * Cohen et al. (2020) Cohen, A., Fialkov, A., Barkana, R., & Monsalve, R. A. 2020, MNRAS, 495, 4845, doi: 10.1093/mnras/staa1530
  * Cumner et al. (2021) Cumner, J., De Lera Acedo, E., de Villiers, D. I. L., et al. 2021, arXiv e-prints, arXiv:2109.10098.  https://arxiv.org/abs/2109.10098
  * de Lera Acedo (2019) de Lera Acedo, E. 2019, in 2019 International Conference on Electromagnetics in Advanced Applications (ICEAA) (IEEE), doi: 10.1109/iceaa.2019.8879199
  * Fialkov et al. (2014) Fialkov, A., Barkana, R., & Visbal, E. 2014, Nature, 506, 197, doi: 10.1038/nature12999
  * Field (1959) Field, G. B. 1959, ApJ, 129, 536, doi: 10.1086/146653
  * Fukushima (1969) Fukushima, K. 1969, IEEE Transactions on Systems Science and Cybernetics, 5, 322, doi: 10.1109/TSSC.1969.300225
  * Furlanetto et al. (2006) Furlanetto, S. R., Oh, S. P., & Briggs, F. H. 2006, Phys. Rep., 433, 181, doi: 10.1016/j.physrep.2006.08.002
  * Goodfellow et al. (2016) Goodfellow, I., Bengio, Y., & Courville, A. 2016, Deep Learning (MIT Press)
  * Kern et al. (2017) Kern, N. S., Liu, A., Parsons, A. R., Mesinger, A., & Greig, B. 2017, ApJ, 848, 23, doi: 10.3847/1538-4357/aa8bb4
  * Kingma & Ba (2014) Kingma, D. P., & Ba, J. 2014, arXiv e-prints, arXiv:1412.6980.  https://arxiv.org/abs/1412.6980
  * Kingma & Welling (2013) Kingma, D. P., & Welling, M. 2013, arXiv e-prints, arXiv:1312.6114.  https://arxiv.org/abs/1312.6114
  * List & Lewis (2020) List, F., & Lewis, G. F. 2020, MNRAS, 493, 5913, doi: 10.1093/mnras/staa523
  * Liu & Shaw (2020) Liu, A., & Shaw, J. R. 2020, PASP, 132, 062001, doi: 10.1088/1538-3873/ab5bfd
  * Madau et al. (1997) Madau, P., Meiksin, A., & Rees, M. J. 1997, ApJ, 475, 429, doi: 10.1086/303549
  * Monsalve et al. (2019) Monsalve, R. A., Fialkov, A., Bowman, J. D., et al. 2019, ApJ, 875, 67, doi: 10.3847/1538-4357/ab07be
  * Monsalve et al. (2018) Monsalve, R. A., Greig, B., Bowman, J. D., et al. 2018, ApJ, 863, 11, doi: 10.3847/1538-4357/aace54
  * Monsalve et al. (2017) Monsalve, R. A., Rogers, A. E. E., Bowman, J. D., & Mozdzen, T. J. 2017, ApJ, 847, 64, doi: 10.3847/1538-4357/aa88d1
  * Philip et al. (2019) Philip, L., Abdurashidova, Z., Chiang, H. C., et al. 2019, Journal of Astronomical Instrumentation, 8, 1950004, doi: 10.1142/S2251171719500041
  * Pritchard & Loeb (2012) Pritchard, J. R., & Loeb, A. 2012, Reports on Progress in Physics, 75, 086901, doi: 10.1088/0034-4885/75/8/086901
  * Shaver et al. (1999) Shaver, P. A., Windhorst, R. A., Madau, P., & de Bruyn, A. G. 1999, A&A, 345, 380.  https://arxiv.org/abs/astro-ph/9901320
  * Singh et al. (2021) Singh, S., Nambissan T., J., Subrahmanyan, R., et al. 2021, arXiv e-prints, arXiv:2112.06778.  https://arxiv.org/abs/2112.06778
  * Tozzi et al. (2000) Tozzi, P., Madau, P., Meiksin, A., & Rees, M. J. 2000, ApJ, 528, 597, doi: 10.1086/308196
  * van der Maaten & Hinton (2008) van der Maaten, L., & Hinton, G. 2008, Journal of Machine Learning Research, 9, 2579.  http://jmlr.org/papers/v9/vandermaaten08a.html
  * Visbal et al. (2012) Visbal, E., Barkana, R., Fialkov, A., Tseliakhovich, D., & Hirata, C. M. 2012, Nature, 487, 70, doi: 10.1038/nature11177
  * Wouthuysen (1952) Wouthuysen, S. A. 1952, AJ, 57, 31, doi: 10.1086/106661