# An information-based metric for observing strategy optimization,
demonstrated in the context of photometric redshifts with applications in
cosmology

Alex I. Malz,1 Francois Lanusse,2 John Franklin Crenshaw,3 & Melissa L.
Graham4
1Ruhr-University Bochum, German Centre for Cosmological Lensing,
Universitätsstraße 150, 44801 Bochum, Germany
2AIM, CEA, CNRS, Université Paris-Saclay, Université Paris Diderot, Sorbonne
Paris Cité, F-91191 Gif-sur-Yvette, France
3Department of Physics, University of Washington, Box 351560, Seattle, WA
98195
4DiRAC Institute, Department of Astronomy, University of Washington, Box
351580, U.W., Seattle, WA 98195, USA
E-mail<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

The observing strategy of a galaxy survey influences the degree to which its
resulting data can be used to accomplish any science goal. LSST is thus
seeking metrics of observing strategies for multiple science cases in order to
optimally choose a cadence. Photometric redshifts are essential for many
extragalactic science applications of LSST’s data, including but not limited
to cosmology, but there are few metrics available, and they are not
straightforwardly integrated with metrics of other cadence-dependent
quantities that may influence any given use case. We propose a metric for
observing strategy optimization based on the potentially recoverable mutual
information about redshift from a photometric sample under the constraints of
a realistic observing strategy. We demonstrate a tractable estimation of a
variational lower bound of this mutual information implemented in a public
code using conditional normalizing flows. By comparing the recoverable
redshift information across observing strategies, we can distinguish between
those that preclude robust redshift constraints and those whose data will
preserve more redshift information, to be generically utilized in a downstream
analysis. We recommend the use of this versatile metric to observing strategy
optimization for redshift-dependent extragalactic use cases, including but not
limited to cosmology, as well as any other science applications for which
photometry may be modeled from true parameter values beyond redshift.
111https://github.com/aimalz/TheLastMetric

###### keywords:

surveys – galaxies: distances and redshifts – methods: statistical

††pubyear: 2021††pagerange: An information-based metric for observing strategy
optimization, demonstrated in the context of photometric redshifts with
applications in cosmology–References

##  1 Introduction

The Vera C. Rubin Observatory will produce a catalog of tens of billions of
astronomical objects over the course of the ten-year Legacy Survey of Space
and Time (LSST; Ivezić et al. 2019). The quality and quantity of resulting
data will depend on LSST’s observing strategy (OS), which encompasses the
choice of frequency and duration of visits to each portion of the night sky
across each of LSST’s $ugrizy$ filters as a function of the survey’s duration.
As the OS directly impacts the science one can accomplish with the resulting
data (Jones et al., 2020), LSST’s Science Collaborations (SCs) are directing
considerable effort to optimizing the choice of OS (e.g., LSST Science
Collaboration et al., 2017; Graham et al., 2018; Lochner et al., 2021, to name
but a few).

Though the space of all OSs is very high-dimensional, a decision of OS may be
informed by how each science goal is affected by each OS considered. LSST has
developed two tools222https://pstn-051.lsst.io/ to facilitate an optimal
choice of OS. OpSim (LSST, 2016; Delgado et al., 2014) forecasts the impact of
an OS on the properties of the resulting photometric observations LSST will
deliver. The MAF (Metrics Analysis Framework) (LSST, 2017) was established to
ensure that the choice of OS would be well-informed by all science cases,
whose proponents are invited to include one or more metrics to be
automatically evaluated for each set of simulated observational conditions.
Though each science application may favor a different OS, a fair choice may be
made by reviewing how all science goals are affected.

The utility of a gargantuan catalog of extragalactic objects, such as what
LSST will provide, relies on the accuracy and precision of its constraints on
their redshifts, which, for a photometric survey such as LSST, represent a
dominant factor in the error budgets of most if not all extragalactic science
applications. Without access to high-fidelity spectroscopic redshift
measurements, users of LSST’s extragalactic catalog will rely on photometric
redshift (photo-$z$) estimates, which suffer from multiple forms of
uncertainty, even under idealized conditions (see Schmidt et al., 2020, and
extensive references therein).

Redshift uncertainty thus represents one of LSST’s greatest liabilities and
one of the utmost importance to multiple LSST SCs (Awan et al., 2016; Lochner
et al., 2018; Scolnic et al., 2018; Almoubayyed et al., 2020), particularly
the Dark Energy SC (DESC), Transients and Variable Stars (TVS) SC, and
Galaxies SC, but photo-$z$ metrics remain underdeveloped. Some metrics are
straightforward, such as the 10-year coadded depth or the number of supernovae
with more than 10 epochs, and can be directly predicted from the OpSim
metadata and visualized as sky maps with HEALpix (Górski et al., 2005).
However, photo-$z$ performance is quantified by derived statistics of a
particular photo-$z$ estimator on an entire simulated galaxy catalog, often as
a function of true redshift. The holy grail of OS metrics would be a map of
“goodness of photo-$z$ quality" insensitive to photo-$z$ estimator for each
HEALpix pixel, a goal that has not yet been achieved, let alone with enough
computational efficiency for practical MAF integration.

Multiple aspects of the current approach to photo-$z$ metrics for OS
optimization would benefit from improvement, ideally addressing as many as
possible of the following needs:

  * •

An observing strategy metric for photo-$z$s should be agnostic to the choice
of estimation method as well as whether point estimates or photo-$z$
posteriors are used.

  * •

A metric for photo-$z$ should be adaptable to integrate with metrics of
additional quantities sensitive to observing strategy.

  * •

An observing strategy metric for photo-$z$s should not preclude direct
comparison of overall metrics between science goals nor between analysis
approaches for a shared science goal.

  * •

Any metric included in the MAF should be fast and scalable; simulation and
propagation of mock data through an entire analysis pipeline is not feasible.

This paper explores a potential OS metric of estimated redshift quality that
represents an improvement upon established metrics along the above axes. Our
metric relies directly on estimating the mutual information between photometry
and redshift, in other words, quantifying how much information is gained on
the redshift of galaxies by having access to the photometry under a given OS.

This paper is not the first use of an information criterion for optimization
in the context of redshift estimation. Malz et al. (2018) used an information-
theoretic metric to optimize the storage parameterization of photo-$z$
posteriors, however, Kalmbach et al. (2020) used a metric of mutual
information more closely related to that of this work to optimize filter
design for photo-$z$s. In their work, photometric data with Gaussian error was
simulated using a simple redshift prior and a small number of galaxy SED
templates. This simple forward model provided an analytic method for
calculating the mutual information.

In this paper, we leverage recent advances in machine learning to enable the
calculation of the mutual information contained in more complex simulations
for which an analytic model may be unavailable, as could be anticipated of
even idealized data processed through OpSim under a realistically complex OS.
In Section 2, we introduce the mathematical framework for the variational
mutual information lower bound that serves as the basis for our metric. In
Section 3, we demonstrate the metric in the context of OS optimization for
LSST. And in Section 4, we summarize its strengths and future directions for
its development and application.

##  2 Method: Variational Mutual Information Lower Bound

This section introduces TheLastMetric (TLM), denoted as $\mathchar
1140\relax$,333the last letter of the Hebrew alphabet, pronounced “tav” an
estimator-independent metric for parameters of interest conditioned on
photometry under any OS. Though we derive it in the context of redshift-
dependent cosmological probes, its mathematical structure is not inherently
restricted to redshift; as it is so broadly applicable across science cases,
one may jokingly exaggerate that it’s the penultimate OS metric, hence its
name.

In Section 2.1, we introduce the mathematical formalism and review relevant
information-theoretic concepts. In Section 2.2 we derive the metric itself,
and in Section 2.3 we describe the model by which the metric is calculated.

###  2.1 An information theoretic view of observing strategies

Every science case seeks a metric of how informative the data corresponding to
each OS is with respect to some physical parameters of interest. The
properties of the telescope and its OS correspond to a transformation of the
underlying true data in the universe. Such true data would correspond to
photons that could be observed only by an impossibly perfect, idealized
instrument that collects complete, noiseless data, in contrast to what we can
observe from any real telescope under a given OS, a subset of those photons
restricted by the OS and convolved with instrumental errors. Typically, the
information content of recovered physical parameters is determined by running
the observed data through an end-to-end analysis pipeline, which is
impractical for a MAF. This is particularly true for cosmological
applications, but while our derived cosmological constraints depend on the
analysis procedure (Chang et al., 2018), the potentially recoverable
cosmological information content does not.444Though we derive and demonstrate
our metric with this application in mind, such a statement is no less true of
other physical parameters of interest to a variety of science cases, making
this metric broadly applicable beyond cosmology or even redshifts.

Let us begin by quantifying the total cosmological information content of an
idealized survey with perfect photometry using information theory. We denote
the random variable representing cosmological parameters as $\Theta$ and a
random variable representing survey photometry as $X_{phot}$. The information
content about cosmological parameters $\theta\sim\Theta$ due to photometry
$x_{phot}\sim X_{phot}$ can be described by the mutual information between
$\Theta$ and $X_{phot}$, defined as

$I(\Theta\ ;X_{phot})\equiv\mathbb{E}_{p(\theta,x_{phot})}\
\left[\log\frac{p(\theta,x_{phot})}{p(\theta)p(x_{phot})}\right].$ (1)

The mutual information between two random variables represents the reduction
in uncertainty in one due to knowing the other. It is desirable for this
mutual information to be large, i.e. we want the photometry $x_{phot}$ to be
very informative about the cosmological parameters $\theta$.

In practice, of course, we do not have access to perfect photometry, and a
typical cosmological analysis pipeline does not directly relate photometry to
cosmology in a single step; instead, typical analyses constitute a pipeline
from observed photometry $X_{phot}^{obs}$ to cosmology $\Theta$ via a number
of intermediate quantities. The relationship between cosmological parameters
$\Theta$, redshift distribution $Z$, true photometry $X_{phot}^{true}$, and
observed photometry $X_{phot}^{obs}$ is an example of a Markov chain
$\Theta\rightarrow Z\rightarrow X_{phot}^{true}\rightarrow X_{phot}^{obs}$
satisfying

$p(\theta,z,x_{phot}^{true},x_{phot}^{obs})=p(\theta)\ p(z|\theta)\
p(x_{phot}^{true}|z)\ p(x_{phot}^{obs}|x_{phot}^{true}).$ (2)

The amount of information each stage of this chain retains about cosmology can
be expressed using the data processing inequality (Cover & Thomas, 2006),

$I(\Theta\ ;\ Z)\geq I(\Theta\ ;\ X_{phot}^{true})\geq I(\Theta\ ;\
X_{phot}^{obs})\;,$ (3)

which can be intuitively understood as saying that information can only be
lost through the steps of a chain.

Every application of LSST data aims to minimize the information loss in the
steps of this chain, or one analogous to it, that are under our control. An
example of a step we as experimenters control is the choice of OS that min
information loss in the stages of this chain that relates galaxies’ true
redshifts and their observed photometry $I(Z\ ;\ X_{phot}^{obs})$, which
depends on the specific OS that transforms $X_{phot}^{true}\rightarrow
X_{phot}^{obs}$. Again following the data processing inequality, we expect the
following:

$I(Z\ ;\ X_{phot}^{true})\geq I(Z\ ;\ X_{phot}^{obs})\;,$ (4)

i.e. that the mutual information between redshift and observed photometry
would be saturated if perfect, true, unobservable photometry were available,
but the OS determines how closely we can approximate that bound.

Our goal is to compare the mutual information $I(Z\ ;\ X_{phot}^{obs})$555As
was suggested in the preamble, this paper concerns the mutual information of
redshift and photometry, but the derivation is just as valid for the mutual
information $I(\Psi;X_{phot}^{obs})$ of observed photometry with some other
parameter $\Psi$, for example e.g. stellar mass, used in any science
application, even beyond cosmology. for different OSs, with the understanding
that the OS that can achieve the highest mutual information at this level of
the chain will also maximize the overall mutual information with respect to
cosmology $I(\Theta\ |\ X_{phot}^{obs})$ for a fixed analysis pipeline. Given
this scope, we henceforth use $X_{phot}$ as shorthand for $X_{phot}^{obs}$.
The next challenge to discuss is a practical computation of the mutual
information.

###  2.2 Tractable variational lower bound on the mutual information

Evaluating the mutual information as defined in Equation 1 is in general
extremely challenging, as in most cases only samples from the distributions
involved as accessible, not the underlying distributions themselves. In recent
years however, the concept of mutual information has found many applications
in the machine learning literature (e.g. Tishby & Zaslavsky, 2015; Alemi et
al., 2017; Bachman et al., 2019, as well as the recent review of Poole et al.
(2019)), which has driven significant research into tractable estimators of
lower bounds.

To achieve a tractable expression for the mutual information $I(Z;X_{phot})$
of redshift and photometry, let us first introduce the entropy

$H(Z)\equiv-\int dz\ p(z)\log p(z)$ (5)

of the redshift distribution $p(z)$, which quantifies our uncertainty on the
random variable $Z$. Using Equation 5, we rewrite the mutual information
$I(Z;X_{phot})$ of redshift and photometry in the following way:

$\displaystyle I(Z;X_{phot})$ $\displaystyle=\mathbb{E}_{p(z,x_{phot})}\
\left[\log\frac{p(z|x_{phot})}{p(z)}\right]$
$\displaystyle=\mathbb{E}_{p(z,x_{phot})}\left[\log
p(z|x_{phot})\right]-\mathbb{E}_{p(z)}\left[\log p(z)\right]$
$\displaystyle=\mathbb{E}_{p(z,x_{phot})}\left[\log
p(z|x_{phot})\right]+H(Z).$ (6)

As we do not directly have access to the posterior distribution
$p(z|x_{phot})$, the first term is unknown, making the mutual information
itself intractable.

However, the overall expression for the mutual information can be bounded from
below by introducing a variational approximation $q_{\varphi}(z|x_{phot})$ for
the posterior $p(z|x_{phot})$, which leads to the variational lower bound
introduced in Barber & Agakov (2003):

$\displaystyle I(Z;X_{phot})$
$\displaystyle=\mathbb{E}_{p(z,x_{phot})}\left[\log\frac{p(z|x_{phot})}{q_{\varphi}(z|x_{phot})}\right]$
$\displaystyle+\mathbb{E}_{p(z,x_{phot})}\left[\log
q_{\varphi}(z|x_{phot})\right]+H(Z)$ (7)
$\displaystyle=\mathcal{D}_{KL}\left[p(z|x_{phot})||q_{\varphi}(z|x_{phot})\right]$
$\displaystyle+\mathbb{E}_{p(z,x_{phot})}\left[\log
q_{\varphi}(z|x_{phot})\right]+H(Z).$ (8)

In this expression, $\mathcal{D}_{KL}[p(z|x_{phot})||q_{\varphi}(z|x_{phot})]$
is the Kullback-Leibler Divergence (KLD), a directional measure of the loss of
information due to using the variational model $q_{\varphi}(z|x_{phot})$ as an
approximation to the true, unknown, posterior distribution $p(z|x_{phot})$.
Because this KLD is non-negative, this last expression can be used to provide
the following lower bound on the mutual information:

$\displaystyle I(Z;X_{phot})$
$\displaystyle\geq\mathbb{E}_{p(z,x_{phot})}\left[\log
q_{\varphi}(z|x_{phot})\right]+H(Z)\equiv\mathchar 1140\relax,$ (9)

thereby providing the definition of TheLastMetric (TLM).

Not only is this expression now tractable, as it can be estimated simply by
optimization of the variational parameters $\varphi$, but, moreover, the bound
is tight when $q_{\varphi}(z|x_{phot})=p(z|x_{phot})$ is true. Like the KLD,
$\mathchar 1140\relax$ has units of information, which are nats in the
base-$e$ convention used in this paper but can be trivially converted to, e.g.
base-2 bits.

###  2.3 Lower bound implementation using conditional normalizing flows

The lower bound on mutual information introduced in the previous section
relies on having access to a parametric conditional density
$q_{\varphi}(z|x_{phot})$, also known as the variational distribution, which
is optimized to match the true posterior $p(z|x_{phot})$. Any parametric
conditional density estimator could be used for this purpose, but the more
expressive the model, the tighter the lower bound will be.

In this work, we approximate $p(z|x_{phot})$ with a Normalizing Flow (NF)
(Jimenez Rezende & Mohamed, 2015; Dinh et al., 2015), a flexible class of deep
generative models that represents the current state-of-the-art on many density
estimation tasks (Kobyzev et al., 2020). NFs are Latent Variable Models
(LVMs), which model a given distribution $p(x)$ over a target variable $x$ by
introducing 1) a latent variable $z$ that follows a known prior distribution
(typically a multivariate normal distribution) $p(z)$ as $z\sim p(z)$ and 2) a
parametric mapping $f_{\varphi}$ that maps this latent variable $z$ to a point
in the target distribution $x$ according to $x=f_{\varphi}(z)$. This is no
different from other deep generative latent variable models like Variational
Autoencoders (Kingma & Welling, 2013) or Generative Adversarial Networks
(Goodfellow et al., 2014), but what sets NFs apart is that they are
specifically designed using a bijective mapping $f_{\varphi}$. In the case of
a bijection, the probability density function $q_{\varphi}$ of the model can
be expressed as

$q_{\varphi}(x)=p(z=f^{-1}_{\varphi}(x))\left|\det\frac{\partial
f_{\varphi}}{\partial x}(x)\right|^{-1},$ (10)

where $\det\frac{\partial f_{\varphi}}{\partial x}$ is the Jacobian
determinant of $f_{\varphi}$, which accounts for how $f_{\varphi}$ distorts
volume elements. This expression is nothing more than the change of variable
formula for probabilities, but it gives NFs a crucial advantage over other LVM
models: their probability density function has an explicit closed-form
expression. In other words, for a given set of parameters $\varphi$ we can
explicitly compute the probability $q_{\varphi}(x)$ of a data point $x$ under
the model.

A Conditional NF (CNF) can be trivially made by introducing a conditional
variable $y$ in the mapping $f_{\varphi}(z;y)$ (Winkler et al., 2019) that
preserves the bijectivity of the mapping for all $y$, so Equation 10 still
applies. Thus the conditional distribution modeled by the flow is simply:

$q_{\varphi}(x|y)=p(z=f^{-1}_{\varphi}(x;y))\left|\det\frac{\partial
f_{\varphi}}{\partial x}(x;y)\right|^{-1}.$ (11)

In practice, as the mapping $f_{\varphi}$ is typically implemented using a
neural network, making a normalizing flow conditional simply amounts to adding
the variable $y$ as an input to the networks parameterizing the flow.

Because (C)NFs have tractable likelihoods, they can be trained by directly
optimizing the probability of the training set under the model. For CNFs, the
training loss takes the following form

$\mathcal{L}=-\mathbb{E}_{p(x,y)}\left[\log q_{\varphi}(x|y)\right],$ (12)

which can be shown to minimize the KLD
$\mathcal{D}_{KL}[p(x|y)||q_{\varphi}(x|y)]$, i.e. driving the (approximating)
model distribution to be close to the (true) data distribution. We note that
the CNF’s loss function given by Equation 12 is equal to the first term of the
variational lower bound Equation 2.2. In other terms, training the CNF with
this loss function is exactly equivalent to maximizing the variational mutual
information lower bound.

The approach presented here to build a practical lower bound is agnostic to
the NF architecture employed, but the specific choice should be appropriate to
the details of the problem at hand. We defer the details of the model used in
this work to Section 3.2.

##  3 Demonstration in the context of redshifts for LSST

The necessary and sufficient conditions favoring the use of TLM are that it
must be at least as effective as established metrics and have some additional
advantage(s), such as utility, interpretability, and/or efficiency. To
demonstrate the former, we perform controlled experiments on appropriate data,
described in Section 3.1, using an implementation of a variational
approximation to $\mathchar 1140\relax$, presented in Section 3.2. The latter
is discussed in Section 3.3, where the restults of this experiment are
presented.

Figure 1: CMNN-estimated photo-$z$ statistics for a wide variety of OpSim conditions (grey), the baseline OpSim (black), and the exemplary OpSim OS simulations used in this pilot study (colors as in legend). The baseline_v1.5 OS has mid-quality results, and the simulations used in this analysis were chosen because they represent a range of photo-$z$ qualities that stand out as better and worse than baseline_v1.5. These results should not be taken as representative of the future absolute quality of LSST photo-$z$, but the differences in the results are representative of the impact of different OpSim strategies on the photo-$z$ results. OpSim OS Simulation | m5_u | m5_g | m5_r | m5_i | m5_z | m5_y
---|---|---|---|---|---|---
baseline_v1.5 | 25.86 | 27.02 | 26.99 | 26.42 | 25.70 | 24.94
footprint_stuck_rolling_v1.5 | 25.56 | 26.68 | 26.62 | 26.06 | 25.33 | 24.61
ddf_heavy_nexp2_v1.6 | 25.57 | 26.82 | 26.84 | 26.26 | 25.57 | 24.82
footprint_newA_v1.5 | 25.75 | 26.87 | 26.85 | 26.29 | 25.55 | 24.78
third_obs_pt60_v1.5 | 25.87 | 27.03 | 26.99 | 26.43 | 25.70 | 24.93
barebones_v1.6 | 26.00 | 27.13 | 27.07 | 26.57 | 25.78 | 25.05
Table 1:  Simulated median 5$\sigma$ limiting magnitudes in extragalactic
regions for coadded images from the wide-fast-deep survey, from the
baseline_v1.5 OpSim OS simulation and the five exemplary OpSim OS simulations
used in this work.  Figure 2:  The true catalog redshift distribution for the
mock test galaxy sets used to simulate photo-$z$ results for each OpSim OS
simulation. Differences between the redshift distributions across OpSim OS
simulations are due to statistical fluctuations and correspond to a $0.5\%$
difference in entropy $H(Z)$, meaning they are statistically
indistinguishable.

###  3.1 Data: simulated photo-$z$ catalogs

For this work, we use simulations based on the same mock galaxy catalog to
which the Color-Matched Nearest-Neighbors (CMNN) photo-$z$ estimator666A
demonstration of the CMNN photo-$z$ estimator is available on GitHub at
https://github.com/dirac-institute/CMNN_Photoz_Estimator. was applied to
produce LSST-like photo-$z$ results in Graham et al. (2018, 2020) and the same
set of OpSim conditions Lochner et al. (2021) used for DESC’s assessment of
the impact of OS on multi-probe cosmological constraints. As an overview, we
use CMNN to generate mock photometry for a given OpSim simulation, then use
the traditional photo-$z$ metrics evaluated on CMNN’s photo-$z$ estimates to
identify exemplary OSs to which we then apply TLM.

For each OpSim OS simulation, we first determine the 5$\sigma$ limiting
magnitude of the 10-year coadded images from the wide-fast-deep program in sky
regions (HEALpix $\sim$220 arcmin wide). We identify regions as extragalactic
fields (i.e., appropriate for cosmological studies) if their Galactic dust
extinction was E(B-V)$<$0.2 mag and if they received at least 5 visits per
year in all six $ugrizy$ filters, and then calculate the median 10-year depths
over all extragalactic fields, which are reported in Table 1. For every OpSim
OS simulation, these 5$\sigma$ depths in each filter, $m_{5}$, are passed to
the CMNN Estimator, which uses them to calculate magnitude errors for a
catalog of mock galaxies: $\sigma_{\rm rand}^{2}=(0.04-\gamma)x+\gamma x^{2}$,
where $x=^{0.4(m_{\rm true}-m_{5})}$, $m_{\rm true}$ is the true catalog
apparent magnitude, and $\gamma$ is a filter-dependent factor which account
for the effect of, e.g., sky brightness (see Section 3.2 of Ivezić et al.,
2019). The CMNN Estimator then simulates observed apparent magnitudes by
drawing a random value from a normal distribution with a standard deviation
equal to $\sigma_{\rm rand}$ and adding it to the true catalog magnitude.
Test- and training-sets are drawn randomly (without replacement) from the mock
galaxy catalog, and photo-$z$ estimates for the test set are generated by
identifying training-set galaxies with similar colors, i.e. a subset of color-
matched nearest-neighbors in 5-dimensional color space, and adopting as the
true redshift one test-set galaxy’s photo-$z$ chosen at random. In order to
standardize the mock catalogs used for all simulations, we applied the same
cuts on the observed apparent magnitudes of 25.0, 26.0, 26.0, 25.0, 24.8, and
24.0 mag in filters ugrizy to both the test and training sets, about half a
magnitude brighter than the brightest 5$\sigma$ limiting depth of any given
OpSim OS simulation, as in Lochner et al. (2021).

It is important to note that for all of our simulations, the test- and
training-sets were drawn from the same intrinsic mock catalog, which means
they are perfectly matched in terms of their redshift and apparent magnitude
distributions. While this would be concerning if we aimed to evaluate the
realistic performance of the CMNN estimator, its role in this study is not to
produce LSST’s “official" or even “best" photo-$z$s; rather, we use it as a
forward model of the relationship between the relative quality of photo-$z$
estimates given the quality of input photometry, which provides us with mock
photometry under a given OS upon which we demonstrate TLM.

Several statistical measures are commonly used to evaluate the quality of
test-set photo-$z$ estimates based on the photo-$z$ error, $\Delta
z\equiv(z_{true}-z_{phot})/(1+z_{phot})$. While the bias $\langle\Delta
z\rangle$, representing systematic over- or under-estimates of point
estimates, was calculated, the values under these idealized conditions of a
perfectly representative and complete training-set were too small and similar
for bias to discriminate between OSs. However, our experimental design does
produce simulated photo-$z$ results for which the standard deviation and
fraction of outliers are relatively improved or degraded in a way that
correlates with the 5$\sigma$ depths in the six LSST filters. For a robust
standard deviation in $\Delta z$ we use the interquartile range (IQR) divided
by 1.349, making the fraction of outliers the fraction of test-set galaxies
with $|\Delta z|>3\times$ the robust standard deviation or $>3\times 0.06$,
whichever is larger (matching the definition of the LSST Science Requirements
Document, Ivezić & the LSST Science Collaboration, 2013).

In Figure 1, we show the robust standard deviation and fraction of
outliers777See Graham et al. (2020) for a full description of the standard
deviation and fraction of outliers statistics. in bins of photo-$z$ for a wide
variety of OpSim OS simulations, highlighting the baseline OpSim OS simulation
and a selection of five additional OpSim OS simulations which produced notably
better or worse results than the baseline that we use in this work, whose
median 10-year depths are provided in Table 1.

In Figure 2, we show the distribution of true catalog redshift for the test-
set galaxies used for each simulation. Because the same magnitude cuts were
applied to the test and training sets for all simulations, the training sets
have similar redshift distributions. The differences between the lines are
just the statistical random fluctuations caused by drawing a new test subset
(of 50000) from the greater mock galaxy catalog (of millions of galaxies) for
each simulation. The drop in the number of galaxies in high redshift bins is
realistic for the applied cuts on apparent magnitude, and is the cause for
some of the observed scatter in the photo-$z$ statistical results seen in
Figure 1.

Figure 3: Histogram of the redshift distributions for baseline_v1.5 under
normalizing flows with different values of the tuning parameter $K$. The true
distribution is in gray, while the distribution learned by normalizing flows
with various values of $K$ are displayed in color. Aside from a high-redshift
artifact for the extreme $K=2$ flow, the choice of $K$ appears to have little
effect.

###  3.2 Implementation: density estimation with pzflow

We build the NF lower bound discussed in Section 2.3 using pzflow (Crenshaw,
2021), a GPU-enabled python package for normalizing flows built on Jax
(Bradbury et al., 2018). We refer to our public implementation as
TheLastMetric 888https://github.com/aimalz/TheLastMetric.

For the latent distribution $p(z)$, we use the uniform distribution
$\mathcal{U}(0,3.2)$, as sampling and density estimation are trivial, and the
domain matches the compact support of the redshifts in our data set. Matching
the features of the latent space to the data eases training and prevents any
potential unphysical outliers.

For the bijection $f_{\varphi}$, we use a rational-quadratic neural spline
coupling (RQ-NSC; Durkan et al., 2019), which is a state-of-the-art bijection
both capable of modeling high-dimensional distributions with hundreds of modes
and efficient at both sampling and density estimation. The RQ-NSC transforms
galaxy redshifts with monotonically-increasing piecewise combinations of $K$
segments, each of which is a rational-quadratic function. The bijection
parameters $\varphi$ are the values and derivatives of these rational-
quadratic functions at $K+1$ spline knots. The value of $K$ impacts the
resolution of the distribution learned by the NF, with large $K$ corresponding
to high resolution. After fixing $K$, a neural network calculates the values
and derivatives of the $K+1$ knots from the conditional variables: the five
LSST colors ($u-g$, $g-r$, $r-i$, $i-z$, $z-y$) and the $r$ band magnitude
(which serves as a proxy for overall luminosity). After assessing several
configurations for the neural architecture of our NFs, we adopted a single RQ-
NSC coupling layer, parameterized by a dense neural network with 2 layers of
size 128 and ReLu non-linearities.

To confirm robustness to the choice of tuning parameter $K$, we train flows
with $K=$ 2, 8, 16, and 32 for each of the six OSs under consideration. Each
flow is trained by minimizing the loss of Equation 12 with respect to the
parameters $\varphi$ using the Adam optimizer (Kingma & Ba, 2014). We train
with a learning rate (lr) of $10^{-3}$ for 100 epochs, followed by lr
$=2\times 10^{-4}$ for 100 epochs, followed by lr $=10^{-4}$ for 50 epochs.
For each flow, this takes about 1 minute on a Tesla P100 12GB GPU (or about
twice that on a CPU). Figure 3 shows the redshift distributions learned by the
four flows trained on the baseline_v1.5 OS. Aside from a high-redshift
artifact for $K=2$, the choice of $K$ has little effect on the redshift
distribution learned by the flow. Similar behavior is observed for the
redshift distributions of the other OSs, as well as the cross-correlations
with galaxy colors. Thus, for the remainder of this work, we will just use
$K=16$.

Now that we have set the value of $K$, for each OS we train nine additional
flows with $K=16$ and the same training schedule. The 10 flows per OS will be
used as a deep ensemble (Lakshminarayanan et al., 2016) to account for the
epistemic uncertainty of the model. Deep ensembles perform approximate
Bayesian marginalization (Wilson & Izmailov, 2020) over network parameters by
independently initializing and training neural network parameters a number of
different times. In the case of a non-convex loss function, this procedure
often results in solutions that live in distinct basins of attraction in
parameter space (Fort et al., 2020) and is therefore preferable to methods
that approximately marginalize over single basins of attraction, such as the
Laplace (MacKay, 1992) and SWAG (Maddox et al., 2019) approximations. We
calculate a distribution and report a mean of $\mathchar 1140\relax$ for each
OS based on the ten trained NFs.

Figure 4:  Redshift-binned average of negative log-posterior $-\log
q_{\varphi}(z|x_{phot})$ under each OpSim OS simulation, analogous to Figure 1
(i.e. lower is better). This component of TLM indicates the uncertainty on
redshifts given observed photometry, for instance, the reduction in redshift
uncertainty as the Balmer break passes between the LSST photometric filters,
and otherwise follows the dominant trend of the traditional photo-$z$ metrics
by worsening at high redshift. Though the general ranking of OSs matches that
of the traditional metrics, the reorderings between redshift bins is not as
severe, an indication of robustness of TLM.  Figure 5: The distribution of
$\mathchar 1140\relax$ estimates for each OS. The shaded distributions
represent an estimate of the epistemic errors in the evaluation of the metric,
obtained using a deep ensemble approach. Mean $\mathchar 1140\relax$ estimates
indicated by vertical lines are obtained by averaging of the deep ensemble
values. The stochasticity exceeds the difference between metric values for two
pairs of OSs, but stratification by TLM is otherwise robust.  Figure 6:  A
comparison between the mean $\mathchar 1140\relax$ and the traditional
photo-$z$ metrics of intrinsic scatter (top panel) and outlier rate (bottom
panel) from the CMNN estimator, calculated for the redshift range $0.3<z_{\rm
phot}<3.0$ for each OS (colors). Both TLM and traditional photo-$z$ metrics
penalize the footprint_stuck_rolling_v1.5 and ddf_heavy_nexp2_v1.6 OSs and
favor barebones_v1.6 and generally agree on the ranking of OSs.

###  3.3 Results

As a hypothesis, we expect that TLM will confirm the hierarchy of OSs
corresponding to the traditional photo-$z$ statistics; as will be discussed in
Section 4, the advantages of TLM are its potential extensions, but we first
establish its consistency with our intuition based on the established
photo-$z$ metrics.

Before considering TLM’s value for each OS, we aim to build some intuition of
how it behaves in practice. We therefore begin by considering the behavior of
a key component of $\mathchar 1140\relax$: the per-galaxy log-posterior $\log
q_{\varphi}(z|x_{phot})$, which quantifies how probable the true redshift of a
galaxy with photometry $x_{phot}$ is under the approximated posterior redshift
distribution. This value would be high for narrow posteriors centered on the
true redshifts, indicating that the photometry is very constraining of
redshift. Alternatively, a low value indicates that the posterior is not very
concentrated and/or offset from the true redshift, meaning that the photometry
is not very constraining.

Figure 4 shows the redshift-binned negative expected value $\langle-\log
q_{\varphi}(z|x_{phot})\rangle$ for different OSs; the minus sign is included
for easier comparison with Figure 1, i.e. lower is better. We confirm the
conclusions of the CMNN statistics of Figure 1: towards higher redshifts,
photometry becomes less constraining, in the same way that the scatter and
outlier rate increases for CMNN, and the barebones_v1.6 OS consistently
outperforms the footprint_stuck_rolling_v1.5 OS at all redshifts. We also
observe from Figure 4 that the ordering of OSs by these curves does not
significantly depend on redshift, which is largely consistent with the
findings of the traditional metrics of Figure 1. This indicates that there is
not a sub-range of redshifts for which a given strategy would outperform the
others, which implies that a single OS could be optimal for both low-redshift
and high-redshift science use cases.

In addition, $\langle-\log q_{\varphi}(z|x_{phot})\rangle$ achieves a series
of local minima, corresponding to increased information content, around the
redshifts where the 4000 Å Balmer break, a broad feature of galaxy spectra
important for photo-$z$ estimation, crosses between the LSST photometric
filters. This provides a strong consistency check that our implementation of
the metric is capturing the real, physical mutual information between
photometry and redshift. It also demonstrates consistency with the results of
Kalmbach et al. (2020), who found that using information theory to optimize
filters for photo-$z$s corresponds to designing filters that can optimally
constrain the location of the Balmer break as it moves across the optical
wavelength range.

As described in Section 2.2, $\mathchar 1140\relax$ itself is obtained by
taking the expectation of the log-posterior $\log q_{\varphi}(z|x_{phot})$
over the entire sample, thus capturing how informative the observed photometry
is about the redshifts across the whole population, and adding an entropy term
$H(Z)$ which only depends on the redshift distribution.

The distribution of $\mathchar 1140\relax$ from the deep ensembles for each OS
are shown in Figure 5, recalling that higher $\mathchar 1140\relax$ is better.
The epistemic uncertainty in TheLastMetric dominates $\mathchar 1140\relax$’s
discriminatory power for two pairs of OSs, but there is a clear four-tiered
hierarchy of the redshift information each OS’s photometry preserves.

Figure 6 shows the mean values $\langle\mathchar 1140\relax\rangle$ of these
distributions, plotted against the two relevant canonical photo-$z$ metrics of
Figure 1 for each OS, confirming that the behavior of $\mathchar 1140\relax$
is qualitatively similar to that of the traditional photo-$z$ metrics999It
also confirms the close correlation between the traditional intrinsic scatter
and outlier rate, as the latter is defined in terms of the former., favoring
barebones_v1.6 and disfavoring footprint_stuck_rolling_v1.5.

##  4 Discussion & conclusions

In this paper, we introduce TheLastMetric, $\mathchar 1140\relax$, a metric of
mutual information, and apply it in the context of observing strategy
optimization for an astronomical survey with diverse scientific goals. We also
present TheLastMetric to the community, an implementation of a variational
lower bound on the mutual information of photometry with respect to redshift.
We demonstrate the calculation of TheLastMetric on mock photometric galaxy
catalogs corresponding to exemplary observing strategies for LSST, confirming
that it is qualitatively similar to conventional photo-$z$ metrics.

TheLastMetric offers distinct advantages addressing key needs for observing
strategy metrics for LSST’s diverse extragalactic goals:

  * •

TheLastMetric is a measure of information in units of nats, meaning it and any
science-case specific extensions thereof are directly comparable, enabling the
isolation of the relative importance of science goals from the raw values of
their observing strategy metrics.

  * •

TheLastMetric does not assume any photo-$z$ estimator, freeing it from
assumptions of photo-$z$ template libraries, training sets, and other priors,
as well as from the computational overhead associated with many popular
photo-$z$ estimators.

  * •

TheLastMetric is applicable across redshift-dependent science cases as well as
to other quantities informed by photometry that is influenced by observing
strategy.

TheLastMetric is not without its own assumptions of course. We show that it is
robust to the tuning parameters of the TheLastMetric back-end, but evaluation
on draws from the conditional normalizing flow model rather than the same data
upon which it was trained would, strictly speaking, be more self-consistent.
Furthermore, though TheLastMetric eliminates the computational expense of
estimating photo-$z$s, it retains the traditional metrics’ computational
overhead of simulating a mock galaxy catalog from OpSim parameters.

We note that the entropy $H(Z)$ of the redshift distribution of the mock
galaxy catalog, which factors into $\mathchar 1140\relax$ in Equation 9 and
thus influences Figures 5 and 6, may differ between cosmological probes or
other science cases that use subsamples of galaxies under different selection
functions. Though the entropy term $H(Z)$ would need to be recomputed for the
anticipated redshift distribution of the science-motivated subsample, that
term is subdominant in magnitude as well as trivial to calculate. Since the
expected value of Equation 9 is defined in terms of $p(z,x_{phot})$,
$\mathchar 1140\relax$ could be recalculated under a different redshift
distribution without requiring retraining of TheLastMetric to obtain the $\log
q_{\varphi}(z|x_{phot})$ for each mock galaxy. Thus TheLastMetric is
extensible to redshift-dependent science cases without increasing
computational expense beyond what is required of the current photo-$z$
metrics.

Mathematically, TheLastMetric is not inherently exclusive to redshift and may
be extended to any parameter of interest available in the truth catalog of a
mock galaxy sample to yield an interpretable metric of how informative the
photometry is about that parameter. Use of TheLastMetric and potential
extensions thereof, within and beyond cosmological applications, will enable
the identification of an appropriate observing strategy for LSST. We thus
recommend TheLastMetric’s inclusion in the MAF and motivate future development
into further mutual information metrics specific to individual science cases
or probes of a single science application.

## Acknowledgements

This work was incubated at the August 2020 TVS SC MAF
Hackathon101010https://lsst-tvssc.github.io/metricshackathon2020, which was
supported by an LSSTC Enabling Science small programs grant.

AIM acknowledges support from the Max Planck Society and the Alexander von
Humboldt Foundation in the framework of the Max Planck-Humboldt Research Award
endowed by the Federal Ministry of Education and Research. JFC is supported by
the U.S. Department of Energy, Office of Science, under Award DE-SC-0011635,
as well as the National Science Foundation, Division Of Astronomical Sciences,
under Award AST-1715122, and the Office of Advanced Cyberinfrastructure, under
Award OAC-1739419. MLG acknowledges support from the DIRAC Institute in the
Department of Astronomy at the University of Washington. The DIRAC Institute
is supported through generous gifts from the Charles and Lisa Simonyi Fund for
Arts and Sciences, and the Washington Research Foundation.

## References

  * Alemi et al. (2017) Alemi A., Fischer I., Dillon J., Murphy K., 2017, in ICLR. https://arxiv.org/abs/1612.00410
  * Almoubayyed et al. (2020) Almoubayyed H., et al., 2020, Monthly Notices of the Royal Astronomical Society, 499, 1140
  * Awan et al. (2016) Awan H., et al., 2016, ApJ, 829, 50
  * Bachman et al. (2019) Bachman P., Devon Hjelm R., Buchwalter W., 2019, arXiv e-prints, p. arXiv:1906.00910
  * Barber & Agakov (2003) Barber D., Agakov F., 2003, in Proceedings of the 16th International Conference on Neural Information Processing Systems. NIPS’03. MIT Press, Cambridge, MA, USA, p. 201–208
  * Bradbury et al. (2018) Bradbury J., et al., 2018, JAX: composable transformations of Python+NumPy programs, http://github.com/google/jax
  * Chang et al. (2018) Chang C., et al., 2018, Mon Not R Astron Soc, 482, 3696
  * Cover & Thomas (2006) Cover T. M., Thomas J. A., 2006, ELEMENTS OF INFORMATION THEORY. Telecommunication, Wiley
  * Crenshaw (2021) Crenshaw J. F., 2021, jfcrenshaw/pzflow: v1.6.0, doi:10.5281/zenodo.4679913, https://doi.org/10.5281/zenodo.4679913
  * Delgado et al. (2014) Delgado F., Saha A., Chandrasekharan S., Cook K., Petry C., Ridgway S., 2014, in Modeling, Systems Engineering, and Project Management for Astronomy VI. International Society for Optics and Photonics, p. 915015
  * Dinh et al. (2015) Dinh L., Krueger D., Bengio Y., 2015, in Bengio Y., LeCun Y., eds, Proceedings of the 3rd International Conference on Learning Representations. San Diego, CA (arXiv:1410.8516)
  * Durkan et al. (2019) Durkan C., Bekasov A., Murray I., Papamakarios G., 2019, in Wallach H. M., Larochelle H., Beygelzimer A., d’Alché-Buc F., Fox E. B., Garnett R., eds, Advances in Neural Information Processing Systems 32. Curran Associates, Inc., Vancouver, Canada, pp 7511–7522 (arXiv:1906.04032)
  * Fort et al. (2020) Fort S., Hu H., Lakshminarayanan B., 2020, arXiv:1912.02757 [cs, stat]
  * Goodfellow et al. (2014) Goodfellow I. J., Pouget-Abadie J., Mirza M., Xu B., Warde-Farley D., Ozair S., Courville A., Bengio Y., 2014, arXiv e-prints, p. arXiv:1406.2661
  * Górski et al. (2005) Górski K. M., Hivon E., Banday A. J., Wandelt B. D., Hansen F. K., Reinecke M., Bartelmann M., 2005, ApJ, 622, 759
  * Graham et al. (2018) Graham M. L., Connolly A. J., Ivezić Ž., Schmidt S. J., Jones R. L., Mario Jurić Daniel S. F., Yoachim P., 2018, AJ, 155, 1
  * Graham et al. (2020) Graham M. L., et al., 2020, AJ, 159, 258
  * Ivezić & the LSST Science Collaboration (2013) Ivezić Ž., the LSST Science Collaboration 2013, http://ls.st/LPM-17
  * Ivezić et al. (2019) Ivezić Ž., et al., 2019, ApJ, 873, 111
  * Jimenez Rezende & Mohamed (2015) Jimenez Rezende D., Mohamed S., 2015, arXiv e-prints, p. arXiv:1505.05770
  * Jones et al. (2020) Jones R. L., Yoachim P., Ivezic Z., Neilsen E. H., Ribeiro T., 2020, Technical Report 051, Survey Strategy and Cadence Choices for the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST). LSST
  * Kalmbach et al. (2020) Kalmbach J. B., VanderPlas J. T., Connolly A. J., 2020, The Astrophysical Journal, 890, 74
  * Kingma & Ba (2014) Kingma D. P., Ba J., 2014, arXiv e-prints, p. arXiv:1412.6980
  * Kingma & Welling (2013) Kingma D. P., Welling M., 2013, arXiv e-prints, p. arXiv:1312.6114
  * Kobyzev et al. (2020) Kobyzev I., Prince S. J. D., Brubaker M. A., 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence, pp 1–1
  * LSST (2016) LSST 2016, OpSim 3.3.8
  * LSST (2017) LSST 2017, MAF 2.4.0
  * LSST Science Collaboration et al. (2017) LSST Science Collaboration et al., 2017, arXiv e-prints, p. arXiv:1708.04058
  * Lakshminarayanan et al. (2016) Lakshminarayanan B., Pritzel A., Blundell C., 2016, arXiv e-prints, p. arXiv:1612.01474
  * Lochner et al. (2018) Lochner M., et al., 2018, arXiv:1812.00515 [astro-ph]
  * Lochner et al. (2021) Lochner M., et al., 2021, arXiv:2104.05676 [astro-ph]
  * MacKay (1992) MacKay D. J., 1992, PhD thesis, California Institute of Technology
  * Maddox et al. (2019) Maddox W., Garipov T., Izmailov P., Vetrov D., Wilson A. G., 2019, arXiv:1902.02476 [cs, stat]
  * Malz et al. (2018) Malz A. I., Marshall P. J., DeRose J., Graham M. L., Schmidt S. J., Wechsler R., Collaboration) L. D. E. S., 2018, AJ, 156, 35
  * Poole et al. (2019) Poole B., Ozair S., van den Oord A., Alemi A. A., Tucker G., 2019, CoRR, abs/1905.06922
  * Schmidt et al. (2020) Schmidt S. J., et al., 2020, Mon Not R Astron Soc, 499, 1587
  * Scolnic et al. (2018) Scolnic D. M., et al., 2018, arXiv:1812.00516 [astro-ph]
  * Tishby & Zaslavsky (2015) Tishby N., Zaslavsky N., 2015, CoRR, abs/1503.02406
  * Wilson & Izmailov (2020) Wilson A. G., Izmailov P., 2020, arXiv:2002.08791 [cs, stat]
  * Winkler et al. (2019) Winkler C., Worrall D., Hoogeboom E., Welling M., 2019, arXiv:1912.00042 [cs, stat]