# Validating Synthetic Galaxy Catalogs for Dark Energy Science in the LSST Era

Eve Kovacs Argonne National Laboratory, Lemont, IL 60439, USA  Yao-Yuan Mao
NHFP Einstein Fellow  Department of Physics and Astronomy, Rutgers, The State
University of New Jersey, Piscataway, NJ 08854, USA  Michel Aguena
Laboratoire d’Annecy de Physique des Particules, Université Savoie Mont Blanc,
CNRS/IN2P3, F-74941 Annecy, France  Anita Bahmanyar  Astronomy and
Astrophysics, University of Toronto, Toronto, Ontario M5S 1A1, Canada  Adam
Broussard  Department of Physics and Astronomy, Rutgers, The State University
of New Jersey, Piscataway, NJ 08854, USA  James Butler  Department of Physics,
University of Chicago, Chicago, IL 60637, USA  Duncan Campbell  McWilliams
Center for Cosmology, Department of Physics, Carnegie Mellon University,
Pittsburgh, PA 15213, USA  Chihway Chang  Department of Astronomy and
Astrophysics, University of Chicago, Chicago, IL 60637, USA  Kavli Institute
for Cosmological Physics, University of Chicago, Chicago, IL 60637, USA
Shenming Fu Department of Physics, Brown University, 182 Hope Street, Box
1843, Providence, RI 02912, USA  NSF’s National Optical-Infrared Astronomy
Research Laboratory, 950 North Cherry Avenue, Tucson, AZ 85719, USA  Katrin
Heitmann Argonne National Laboratory, Lemont, IL 60439, USA  Danila Korytov
Argonne National Laboratory, Lemont, IL 60439, USA  François Lanusse AIM, CEA,
CNRS, Université Paris-Saclay, Université Paris Diderot, Sorbonne Paris Cité,
F-91191 Gif-sur-Yvette, France  Patricia Larsen Argonne National Laboratory,
Lemont, IL 60439, USA  Rachel Mandelbaum McWilliams Center for Cosmology,
Department of Physics, Carnegie Mellon University, Pittsburgh, PA 15213, USA
Christopher B. Morrison  Department of Astronomy, University of Washington,
Box 351580, Seattle, WA 98195, USA  Constantin Payerne  Laboratoire de
Physique Subatomique et de Cosmologie, Université Grenoble-Alpes, CNRS/IN2P3,
53 avenue des Martyrs, 38026 Grenoble, France  Marina Ricci Laboratoire
d’Annecy de Physique des Particules, Université Savoie Mont Blanc, CNRS/IN2P3,
F-74941 Annecy, France  Eli Rykoff  SLAC National Accelerator Laboratory,
Menlo Park, CA 94025, USA  Kavli Institute for Particle Astrophysics and
Cosmology, Stanford University, Stanford, CA 94305, USA  F. Javier Sánchez
Fermi National Accelerator Laboratory, PO Box 500, Batavia, IL 60510, USA
Ignacio Sevilla-Noarbe Centro de Investigaciones Energéticas, Medioambientales
y Tecnológicas (CIEMAT), E-28040 Madrid, Spain  Melanie Simet  University of
California, Riverside, 900 University Avenue, Riverside, CA 92521, USA  Chun-
Hao To  SLAC National Accelerator Laboratory, Menlo Park, CA 94025, USA  Kavli
Institute for Particle Astrophysics and Cosmology, Stanford University,
Stanford, CA 94305, USA  Vinu Vikraman  Argonne National Laboratory, Lemont,
IL 60439, USA  Rongpu Zhou  Department of Physics and Astronomy, University of
Pittsburgh, Pittsburgh, PA 15260, USA  Camille Avestruz Department of Physics,
University of Michigan, Ann Arbor, MI 48109, USA  Leinweber Center for
Theoretical Physics, University of Michigan, Ann Arbor, MI 48109, USA
Christophe Benoist  Université Côte d’Azur, OCA, CNRS, Lagrange, UMR 7293, CS
34229, 06304, Nice Cedex 4, France  Andrew J. Benson Carnegie Observatories,
Pasadena, CA 91101, USA  Lindsey Bleem Argonne National Laboratory, Lemont, IL
60439, USA  Aleksandra Ćiprijanović  Fermi National Accelerator Laboratory, PO
Box 500, Batavia, IL 60510, USA  Céline Combet Laboratoire de Physique
Subatomique et de Cosmologie, Université Grenoble-Alpes, CNRS/IN2P3, 53 avenue
des Martyrs, 38026 Grenoble, France  Eric Gawiser Department of Physics and
Astronomy, Rutgers, The State University of New Jersey, Piscataway, NJ 08854,
USA  Shiyuan He  Institute of Statistics and Big Data, Renmin University of
China, Beijing, China  Remy Joseph Department of Astrophysical Sciences,
Princeton University, Princeton, NJ 08544, USA  Jeffrey A. Newman  Department
of Physics and Astronomy, University of Pittsburgh, Pittsburgh, PA 15260, USA
Pittsburgh Particle Physics, Astrophysics and Cosmology Center, University of
Pittsburgh, Pittsburgh, PA 15260, USA  Judit Prat  Department of Physics,
University of Chicago, Chicago, IL 60637, USA  Samuel Schmidt Department of
Physics and Astronomy, University of California, Davis, One Shields Ave,
Davis, CA, 95616, USA  Anže Slosar Physics Department, Brookhaven National
Laboratory, Upton, NY 11973, USA  Joe Zuntz Institute for Astronomy,
University of Edinburgh, Edinburgh EH9 3HJ, United Kingdom  The LSST Dark
Energy Science Collaboration

###### Abstract

Large simulation efforts are required to provide synthetic galaxy catalogs for
ongoing and upcoming cosmology surveys. These extragalactic catalogs are being
used for many diverse purposes covering a wide range of scientific topics. In
order to be useful, they must offer realistically complex information about
the galaxies they contain. Hence, it is critical to implement a rigorous
validation procedure that ensures that the simulated galaxy properties
faithfully capture observations and delivers an assessment of the level of
realism attained by the catalog. We present here a suite of validation tests
that have been developed by the Rubin Observatory Legacy Survey of Space and
Time (LSST) Dark Energy Science Collaboration (DESC). We discuss how the
inclusion of each test is driven by the scientific targets for static ground-
based dark energy science and by the availability of suitable validation data.
The validation criteria that are used to assess the performance of a catalog
are flexible and depend on the science goals. We illustrate the utility of
this suite by showing examples for the validation of cosmoDC2, the
extragalactic catalog recently released for the LSST DESC second Data
Challenge.

methods: numerical – methods: statistical – dark energy – large-scale
structure of the universe

##  1 Introduction

The upcoming Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST;
Ivezić et al., 2019; LSST Science Collaboration et al., 2009) is an ambitious
ground-based imaging survey covering approximately half the sky and beginning
as early as 2023. The advent of this wide and deep data set will usher in a
new era of cosmology characterized by small statistical errors and better
control of systematic errors. The Rubin Observatory LSST Dark Energy Science
Collaboration (LSST DESC)111https://lsstdesc.org has been formed to prepare
for the wealth of scientific opportunities offered by this large influx of
data (LSST Dark Energy Science Collaboration, 2012). The collaboration was
established with an initial focus on probes of the fundamental properties of
dark energy (Weinberg et al., 2013), including measurements of weak lensing,
strong lensing, large-scale structure, galaxy clusters and supernovae. Within
the collaboration, Working Groups (WGs) have been convened to develop and test
a variety of scientific analyses and make forecasts for the precision with
which the parameters describing the behavior of dark energy can be constrained
(The LSST Dark Energy Science Collaboration, 2018a).

These activities would not be possible without an extensive and
contemporaneous simulation campaign that is designed to provide synthetic
galaxy catalogs with various levels of realism. As with any simulation
campaign, it is critical to perform validation to assess the catalog’s level
of fidelity compared to the real universe and determine the extent of its
utility for testing scientific analyses. The validation process relies on a
series of tests that compare the catalog data with suitably chosen validation
data sets. The three considerations that underlie the construction of a test
suite are: (1) the scientific goals that will be pursued with the catalog, (2)
the availability of suitable validation data and (3) an evaluation of how
realistic the catalog must be in order to meet those goals.

A number of simulated catalogs with various levels of complexity have been
produced for ongoing and past surveys (e.g., Crocce et al., 2015; Alam et al.,
2017; Avila et al., 2018; DeRose et al., 2019; Shirasaki et al., 2019;
Safonova et al., 2020; Lin et al., 2020; DeRose et al., 2021; Wechsler et al.,
2021). These catalogs range from large general-purpose catalogs (Crocce et
al., 2015; DeRose et al., 2019, 2021; Wechsler et al., 2021) intended to cover
a wide range of science targets for surveys such as the Dark Energy Survey
(DES), VISTA and WISE, to more specialized catalogs intended for specific
surveys or analyses. For example, catalogs can be produced to provide
synthetic data for specific scientific analyses such as the analysis of shear
autocorrelations (Shirasaki et al., 2019), or to mimic a specific data set
such as those of the Dark Energy Survey BAO sample (Avila et al., 2018) or the
Dark Energy Spectroscopic Instrument Bright Galaxy Survey (Safonova et al.,
2020) or the emission line galaxy (ELG) sample from the extended Baryon
Oscillation Spectroscopic Survey (Lin et al., 2020). The three considerations
listed above enter into the validation efforts that accompanied the production
of these catalogs.

Turning to the test suite that we present here, the nature of the tests that
are of interest to us is driven by our goal of studying dark energy science.
The suite must include checks that the statistical distributions of galaxies
on the simulated sky are realistic and that the statistical distributions of
the properties relevant for dark energy science agree with those that are
observed in the real universe. The level of agreement required must be
specified for each distribution.

The range and availability of validation data necessarily has a big impact on
the tests that can be performed. Hence the assembly and curation of an
appropriate validation data set is a fundamental step in the construction of
any validation test suite. Understanding which data sets can be used to
validate which galaxy property and determining what ranges of the property
values can be constrained are key factors in constructing the suite.

A specific example that illustrates this point is the validation of synthetic
galaxy properties as a function of redshift. Synthetic catalogs are available
which provide galaxies out to higher redshifts, $z\gtrsim 1.5$ (DeRose et al.,
2019; Korytov et al., 2019), but there are very few complete, magnitude-
limited observational data samples available with sufficient statistics at
LSST depths to be useful for validation purposes. While there are several
high-redshift spectroscopic galaxy surveys such as the VIMOS VLT Deep Survey
(VVDS) (Le Fèvre et al., 2013), the VIMOS Ultra Deep Survey (VUDS) (Le Fèvre
et al., 2014) and zCOSMOS (Lilly et al., 2007), these data typically suffer
from incompleteness issues and low statistics at higher redshifts. Photometric
catalogs assembled from COSMOS data (Laigle et al., 2016; Weaver et al., 2021)
suffer from significant outlier rates, particularly at fainter magnitudes,
and, due to the template-fitting methods employed to estimate the redshifts,
their color and redshift distributions are not unbiased. The ever-present
issue of cosmic variance that accompanies small-area survey data may also be a
consideration for specific validation tests. Consequently, the validation of
full catalog distributions at high redshift is quite problematic and we must,
in many cases, rely on extrapolations of existing data.

Another aspect of catalog validation is the verification of the catalog
contents. Rather than making comparisons with external data, the verification
process consists of checks to ensure that the simulated catalog delivers the
expected results for given quantities. Such checks include tests for the
uniqueness of object identifiers, tests of the statistics of object-property
distributions and comparisons of object properties with theoretical
predictions based on the input parameters for the underlying cosmological
simulations.

As mentioned above, in addition to developing the validation tests, a set of
validation criteria has to be defined. These criteria are informed by the
specific science goals that will be pursued using the synthetic catalog.
Different goals may provide different requirements for the same validation
test and may even evolve with time. In this sense, the criteria can be viewed
independently of the test. A test suite could be adapted for other surveys by
retaining the tests and validation data sets but altering the validation
criteria to require different levels of fidelity with the observational data.
The entire validation process, which encompasses the tests and the criteria,
is in fact the key to understanding how a catalog can be used and what its
limitations are.

The validation test suite that we present in this paper provides a set of
comparisons of carefully selected measurements from a synthetic galaxy catalog
with a curated set of validation data. The tests have been developed in
conjunction with LSST DESC WGs222https://lsstdesc.org/pages/organization.html
and focus on the validation of static galaxy properties. The validation of
time-domain properties is discussed elsewhere in LSST Dark Energy Science
Collaboration (2020). Due to the broad range of dark energy science targets
within LSST DESC, we expect that our validation test suite is representative
of the set of tests required to evaluate the realism of synthetic galaxy
catalogs targeted for studying dark energy science in optical imaging surveys,
both now and in the future. The test suite for spectroscopic surveys would
have some overlap with the suite presented here.

Our suite of tests has been used to validate cosmoDC2 (Korytov et al.,
2019)333cosmoDC2 is publicly available from the NERSC website:
https://portal.nersc.gov/project/lsst/cosmoDC2/_README.html, the extragalactic
catalog underlying the LSST DESC second data challenge, DC2 (LSST Dark Energy
Science Collaboration, 2020) This data challenge is one of two challenges that
have been instituted by the LSST DESC in order to assess and quantify the
progress towards preparedness for the arrival of LSST data. DC2 is an
ambitious program to produce a realistic LSST-like data set that can be used
for a multitude of science goals such as testing analysis and production
pipelines, studying a variety of systematic effects, testing the Rubin LSST
Science Pipelines (Bosch et al., 2019), studying time-domain objects and
running joint analyses that involve multiple observational probes. Our
validation process is an integral part of catalog development and production
because (1) the underlying model for the galaxy-halo connection used to
populate the synthetic catalog typically needs to be adjusted to meet the
validation criteria and (2) the production of the catalog requires significant
human effort and computing resources to produce, so a staged and iterative
approach is necessary.

In this work, we use DESCQA (Mao et al., 2018), which is the automated
validation framework developed by LSST DESC to compare simulated catalogs with
a variety of validation data. This framework is quite general in that it
ingests sets of catalogs and sets of validation data (which may be
observational data or theoretical predictions) and then compares the results,
in the form of summary statistics and comparison plots, for a specified set of
tests. Each test can incorporate validation criteria by producing a numerical
score derived from a comparison of selected summary statistics for the catalog
and validation data and issuing a pass or fail based on the score.

In summary, we present here a validation test suite designed to evaluate
simulated galaxy catalogs that have been targeted for ground-based imaging
surveys focused on dark energy science. The suite is based on a curated set of
validation data that includes both observational data and theoretical
predictions. We demonstrate the utility of the suite with the validation of
cosmoDC2 using the DESCQA framework and the DC2 validation criteria supplied
by LSST DESC.

This paper is organized as follows. In Section 2, we discuss the construction
of our validation test suite by considering the scientific goals of our
planned analyses and the availability of suitable validation data sets. The
reader who is primarily interested in the test results for cosmoDC2 can skip
to Section 4. In Section 3, we briefly describe the cosmoDC2 extragalactic
catalog and the validation framework DESCQA. Next, in Section 4, we show
examples of the validation tests comprising the suite and discuss the
validation criteria used to evaluate the fidelity of cosmoDC2 with the
observational data. Finally, we summarize our results and discuss the
implications and future work in Section 5.

##  2 Construction of the Validation Test Suite

Here we discuss the development of the validation test suite that is presented
in this paper. The scientific goals of the analyses planned by the LSST DESC
WGs444https://lsstdesc.org/assets/pdf/docs/DESC_SRM_latest.pdf act as drivers
for identifying critical properties of the simulated data that must be
realistically rendered in the synthetic galaxy catalog. Although these
scientific goals, and hence the intended uses of the catalog, are varied and
complex, a relatively small set of critical properties have emerged from this
exercise. We find considerable overlap in the requirements coming from the
different analysis WGs. In the following subsections, for each of these
science goals, we present: (1) the science drivers and the critical properties
that must be accurately rendered by the catalog, (2) the data sets that are
available to validate these properties and (3) the tests that we construct by
matching the available data with the galaxy properties that require
validation.

###  2.1 Galaxy Number Densities

####  2.1.1 Scientific Drivers and Critical Properties

The galaxy number-density distribution is one of the most fundamental
properties of a synthetic galaxy catalog. Many targets for dark energy science
rely on the assumption that this distribution is realistically rendered. In
the real universe, the observed number density of galaxies can be viewed as a
multi-dimensional distribution that is dependent on redshift and the spectral
energy distribution (SED) of each galaxy, as well as other factors such as
Galactic reddening and atmospheric and instrumental effects. In practice, for
imaging surveys, the incoming light is integrated over the band passes of the
filters in the survey and this high dimensional space is reduced to redshift
plus the luminosities in these filters. For LSST, there will be 6 filters $u$,
$g$, $r$, $i$, $z$, and $y$. Any validation test suite must incorporate tests
that compare the number density as a function of both redshift and observed
magnitudes in the available filters.

The primary scientific drivers that require realistic galaxy number densities
are large-scale structure (LSS) and weak lensing (WL) analyses. These analyses
extract information from measurements of two-point correlation functions such
as galaxy clustering, shear auto-correlations and galaxy-density – shear
cross-correlations, all of which are used in the so-called $3\times 2$pt
analysis (e.g., DES Collaboration, 2018, 2021; Joachimi et al., 2021). Most
importantly, for a single redshift slice, the noise on a correlation function
measurement is inversely proportional to number density. If the redshift
distribution of galaxies, $N(z)$, is wrong, then the mean redshift of
tomographic slices will be wrong, but more subtly, the number of source and
lens galaxies will be modulated and the effective redshift, from which the
signal in both the auto-correlation and the cross-correlation is derived, will
be affected in an unpredictable way.

Another critical ingredient of LSS and WL analyses is an understanding of the
accuracy with which photometric redshifts can be determined. This
understanding is limited by uncertainties in instrumental calibration and the
realism of the distribution of galaxy spectral energy distributions (SEDs) and
their evolution with redshift. The population of galaxies has evolved as the
universe has aged, and the resulting changes in their luminosity functions,
combined with observational selection effects, are reflected in the evolution
of the observed redshift distribution as a function of magnitude. Photometric
redshift algorithms estimate galaxy redshifts by comparing observed magnitudes
to either model templates (e.g., Benítez, 2000) or machine learning algorithms
(e.g., Sadeh et al., 2016; Izbicki & Lee, 2017) using training sets of
observed galaxies with known redshifts (Salvato et al., 2019). These redshift
estimates are subject to biases and degeneracies due to the limited
information available, e.g. in LSST we will have only six magnitudes from
broad-band filters with which to estimate a redshift. In order to avoid being
dominated by systematic uncertainties, we must fully characterize such
degeneracies and thus it is vital to have a galaxy population that is
reflective of that observed in the real universe: systematic differences
between the model galaxy populations and the actual observed populations could
introduce biases in the analysis. In addition to one-point distributions, the
spatial clustering of galaxies is a critical component in calibrating
photo-$z$-derived redshift distributions, so properly modeling the spatial
correlations of galaxies is also important (see Section 2.3.1).

One of the principal uses of the cosmoDC2 catalog has been to provide the
extragalactic catalog inputs for the DC2 image simulation campaign (LSST Dark
Energy Science Collaboration, 2020). The simulated images enable a large array
of science goals based on the processing and analysis of these images. One
particularly important scientific target that is strongly affected by galaxy
number densities is the phenomenon of blending (Chang et al., 2013; Sanchez et
al., 2021). This is the apparent overlap, on the plane of the sky, of the
light profiles of different stars and galaxies. Blending is observed, either
because objects at different redshifts appear on the same line of sight, or
because galaxies are in close physical proximity. Overlapping light profiles
in a blended scene therefore affect the measurement of galaxy shapes and
magnitudes as well as the number and spatial distribution of galaxies. With
$\sim 60\%$ of galaxies found to be affected by blending in the HSC Wide
survey (Bosch et al., 2018) and a similar fraction expected in LSST (Newman et
al., 2019; Sanchez et al., 2021), blending will act as a major source of
systematic effects for shear estimations, flux measurements, galaxy number
densities and photometric redshifts estimations. In a recent study, Sanchez et
al. (2021) showed that $62\%$ of galaxies in LSST will have at least $1\%$ of
their flux coming from overlapping, neighbouring sources. This effect,
combined with shot noise, decreases the effective number density of galaxies
by $18\%$.

Because the occurrence of blending is so high, it is unlikely that WL and LSS
analyses can be conducted by removing blended objects. Even with state-of-the-
art deblenders (see Melchior et al., 2018, for instance) to model the light
profiles of blended objects, we will have to calibrate the effects of residual
blending as well as the effect of the deblender itself on dark energy
measurements. The accurate characterization of the performance of these
deblending algorithms is a critical component of the systematics analysis and
depends on realistic galaxy number densities as a function of luminosity and
redshift.

One particularly problematic aspect of deblending is that of unrecognized
blends (Dawson et al., 2015; Martinet et al., 2019). Most deblending
algorithms rely on the detection of each galaxy before their models can be
extracted. In cases where galaxies are too close to each other, detection and
deblending algorithms cannot determine whether one or more objects are
present. Deblending algorithms are likely to erroneously model the scene as
one object whose properties (flux, shape, color) will not be representative of
the underlying truth. The rate and impact of such unrecognized blends on shear
and photometric redshift estimation have to be characterised in the context of
LSST, and simulations will play a key role in understanding this effect and
mitigating this systematic. Depending on the deblending strategy, it is also
possible that modeling errors might introduce biases in the size, shape, flux
and colors of deblended galaxies. Simulations will allow us to identify which
deblending strategies are expected to be successful and help us understand the
impact of these strategies on measurements relevant for higher level science.

####  2.1.2  Galaxy Number-Density Data Sets

We now examine the available data sets that can be used to validate galaxy
number densities as a function of redshift and magnitude. It is critical to
have at least one data set that provides a check on the absolute normalization
of the galaxy number densities in the catalog. One simple possibility is to
validate the cumulative number density as a function of magnitude. The
construction of such a test requires data from a survey that is both wide and
deep. The first requirement suppresses the uncertainty from cosmic variance,
while the second ensures that we can validate the number density of very faint
galaxies. The data set from the first data release (Aihara et al., 2018a) of
the Deep layer of the Hyper Suprime-Cam (HSC) Subaru Strategic Program (Aihara
et al., 2018b) provides the deepest measurements available from an ongoing
survey with an area exceeding a few square degrees.

A convenient way to characterize the behavior of this data set is to obtain
power-law fits to the measured cumulative number counts of galaxies in the
$g$, $r$, $i$, $z$ and $y$ bands. These fits are valid for CModel (Lupton et
al., 2001; Bosch et al., 2018) magnitudes $\lesssim 25$. The extrapolation of
these fits to fainter magnitudes is justified by measurements from deep
pencil-beam surveys, which appear to have power-law number counts for CModel
magnitudes $\lesssim 28$ (e.g. Beckwith et al., 2006). We note that although
the HSC data have star-galaxy separation selections applied, we expect the
systematic errors associated with these selections to grow with fainter
magnitudes. Hence, if a stringent validation test for the number density is
required, it may be necessary to apply a more careful extrapolation procedure
that takes such systematic effects into account.

Additional verification of our extrapolations can be done by comparing the
fitted number counts against the counts obtained from deep pencil beam
surveys. For example, very deep Subaru observations in the Cosmic Evolution
Survey (COSMOS) field (Capak et al., 2007) yield a cumulative number density
of 150/arcmin2 or $5.4\times 10^{5}$/deg2 for $I<26.5$, which is quite close
to our extrapolated $i$-band555The mean effective wavelengths for COSMOS
$I$-band (F814W) and LSST i-band are $\approx 8100$ and $\approx 7500$ Å,
respectively. The transmission curve for F814W is asymmetrical, so the peak of
its transmission occurs at $7500$ Å HSC fit of $(5.17\pm 0.04)\times
10^{5}$/deg2. We caution the reader, however, that data from pencil beam
surveys are subject to considerable cosmic variance and hence are not as
reliable or useful as our extrapolated fits.

Observational data for galaxy redshift distributions are available from Coil
et al. (2004) and Newman et al. (2013). These observations are reported as
parameterized fits to the shape of $dN/dz$ distributions for a variety of
magnitude-limited samples. The selection cuts for these samples were imposed
on the CFHT $R$\- and $I$-band magnitudes and range from $18-27$ and $22-24$,
respectively, for the works cited above. The parameterized fits have a simple
analytic form that can be readily integrated to compute probability
distributions over the range of redshifts being validated. In order to make
meaningful comparisons, the cuts used to select the validation data must also
be applied to the catalog data. In practice, the filters simulated for a given
catalog will be different from those in the validation data, so depending on
the precision required by the validation criteria, it may be necessary to
apply correction factors to account for this difference. The fits to the
observational data are available out to a redshift of $z\lesssim 1.5$, which
as mentioned in Section 1, limits the range of the validation that can be
performed.

We reiterate that although there are additional data available from other
spectroscopic surveys such as VVDS (Le Fèvre et al., 2013), VUDS (Le Fèvre et
al., 2014) and zCOSMOS (Lilly et al., 2007) and from photometric catalogs that
are obtained from COSMOS data (Laigle et al., 2016; Weaver et al., 2021),
these data are not as suitable for validating the full catalog redshift
distribution. For example, VVDS has very few highly-secure redshifts at $z>1$,
VUDS does not provide a complete magnitude-limited sample in redshift ranges
relevant for LSST, and, due to the VUDS color selections, the statistics in
the redshift range $1<z<2$ are low. Similarly, zCOSMOS-deep data is color-
selected and does not span the full galaxy population and VANDELS (Pentericci
et al., 2018) targets only red galaxies in the range $1.5<z<2.5$. COSMOS data
have a photometric redshift outlier rate of $>10$% for galaxies at high
redshift or with $I$-band magnitudes in the range $24<I<25$. All of these
issues have the potential to confuse any conclusions about the level of
agreement between the catalog and validation data.

Two observational data sets are available to validate galaxy SED
distributions. The first is the data set from the Sloan Digital Sky Survey
(SDSS) main galaxy sample which provides measurements of the $ugriz$ colors
for an apparent magnitude-limited sample from SDSS Data Release (DR) 13
(Albareti et al., 2017). We select galaxies from this sample with well defined
redshifts666We used the following SQL query: survey=“SDSS” AND class=“GALAXY”
AND z¿0 AND zWarning=0 that lie in the range $0.06<z<0.09$, and use this data
set to validate color distributions at low redshifts.

The second data set is a combination of DEEP2 (Newman et al., 2013) and DEEP3
spectroscopic redshifts and Canada-France-Hawaii Telescope Legacy Survey
(CFHTLS) (Hudelot et al., 2012) photometry in $ugriz$ bands; this combined
data set is described in Zhou et al. (2019). The depth of the CFHTLS-Deep
imaging is comparable to that of the LSST 10-year stack and the DEEP2 and
DEEP3 spectroscopy is complete to $R_{AB}=24.1$ over the redshift range of
$0<z\lesssim 1.4$. The area covered by both CFHTLS-Deep imaging and DEEP2/3
spectroscopic is roughly 0.3 square degrees. We use this data set for
validating the colors of galaxies at a redshift of $z\sim 0.9$. Although
Subaru $Y$-band data are also included in this data set, they are not used
here due to their incomplete and non-uniform spatial coverage. The photometry
of both data sets is corrected for (Milky-Way) Galactic extinction before
being used for validation.

Similar to the caveats discussed for the the validation of redshift
distributions, the color completeness of data from other spectroscopic and
photometric surveys limits their utility for our validation applications. None
of the aforementioned alternative surveys enables an assessment of the full
range of galaxy colors at fixed redshift and magnitude. In contrast, the
DEEP2/DEEP3 data includes objects of all colors at $z>0.7$ by design. It is
the single data set, readily available at this time, that is best suited for a
validation of the full catalog. In the future, with additional effort and
access to catalogs with well-matched photometry and redshifts, further
comparisons based on other data sets could be made in the regimes where those
data sets are relatively complete.

####  2.1.3 Number Density Tests

As we have seen in Section 2.1.2, not all projections of galaxy number
densities in the multi-dimensional space of redshift and available filter
magnitudes have corresponding validation data sets available. In order to
simplify the test suite and match the available validation data sets, we have
chosen to implement tests based on various projections of this multi-
dimensional distribution. We include:

  * •

the cumulative number density of galaxies as a function of magnitude (Section
4.2),

  * •

the probability distribution as a function of redshift for magnitude-limited
samples of galaxies (Section 4.3),

  * •

color distributions (Section 4.4):

    * –

the probability distribution as a function of galaxy colors for magnitude-
limited samples of galaxies in slices of redshift (Section 4.4.1),

    * –

the 2-dimensional probability distributions of galaxy colors in slices of
redshift (Section 4.4.2).

The first test checks the realism of the absolute number density as a function
of magnitude but does not provide a detailed check on the redshift
distribution. The cumulative distribution used in this test is sufficient to
ensure that the catalog number counts scale approximately as expected with
survey depth. A more stringent test would employ differential counts. The
second test checks the realism of the redshift distribution as a function of
magnitude but is insensitive to the absolute normalization of the data. The
remaining tests check the realism of various projected distributions of galaxy
colors and hence provide validation tests for galaxy SEDs. Because galaxy
colors and magnitudes are correlated and vary with redshift, a comprehensive
set of tests to validate galaxy SEDs should cover this multi-dimensional
color-magnitude space. Many projections of this space are possible. As a first
test, we use simple one-point distributions as a basic check of the realism of
galaxy colors, but this test alone is not sufficient to determine if a catalog
captures the full complexity of the SED distributions in the real universe.
Other more complicated color distributions provide further diagnostics for
catalog performance. In our suite, we include color-color distribution tests
to capture information on the correlations between colors.

###  2.2 Galaxy Shapes, Sizes and Morphologies

####  2.2.1 Scientific Drivers and Critical Properties

The observable quantity for WL is the subtle distortion (or shear) of observed
galaxy shapes due to the deflection of light due to the large-scale matter
distribution between those galaxies and the observers (Bartelmann & Schneider,
2001). In the two-point correlation function described in Section 2.3.1, as
well as other WL statistics, we estimate shear for an ensemble of galaxies.
Operationally, we average galaxy ellipticities (inferred from the light
profile) over a sample of galaxies. As the mean ellipticities of most galaxies
are more than 10 times larger than the WL-induced shears, one needs to average
over a large number of galaxies to suppress this intrinsic shape noise and
reveal the cosmological shear signal. This means that we cannot only use the
large bright galaxies for WL measurements. Rather, the small and faint
galaxies, due to their large numbers, dominate the galaxy samples used for WL
measurements, and present additional challenges for the estimation of shear.

At the accuracy level required for Stage IV galaxy surveys such as LSST (The
LSST Dark Energy Science Collaboration, 2018b), shear estimation is
challenging. Multiplicative and additive biases in the shear estimation can
arise from each step of the estimation. In particular, the modeling and
interpolation of the point spread function (PSF) across the focal plane, the
assumed model for the galaxy profiles, and the selection used to arrive at the
WL galaxy sample all have the potential to introduce biases in the shear
estimation. A number of approaches have been taken to calibrate these biases,
ranging from image simulations to more empirical approaches using the data
themselves (for a review of systematic effects and state-of-the-art shear
estimation algorithms, see Mandelbaum, 2018; Sheldon et al., 2020)

For WL science, several aspects of the simulations are important. In addition
to having realistic number density and magnitude distributions for galaxies,
the morphology (size and shape) distributions are also critical. The
importance of this can be seen in previous work such as Fenech Conti et al.
(2017), Mandelbaum et al. (2018) and Zuntz et al. (2018), where image
simulations were specifically developed to calibrate specific shear
measurement algorithms. As discussed above, since the universe produces many
more small, faint galaxies than large, bright galaxies, any lensing sample
will be dominated in number by galaxies close to our minimum cutoffs in size
and brightness. As such, an unrealistic model for the size of the galaxies
relative to the size of the PSF could result in a mismatch of this cutoff
compared to the real world. Even more recent shear estimation methods that
attempt to self-calibrate the shear biases require basic validation against
simulations with a reasonable level of complexity in the galaxy population
(MacCrann et al., 2020). On the other hand, the shape of the galaxies is the
dominant source of statistical error in shape measurements. The apparent shape
of the galaxies is a combination of the intrinsic galaxy shape and measurement
errors, both could be redshift and magnitude dependent. Overall, a coherent
framework that encompasses all the relevant correlations between the different
galaxy properties is essential for testing systematic errors that could
couple, e.g., weak lensing and photo-z systematic effects.

Another effect impacting shear measurement is that of objects in close
proximity on the plane of the sky, or blending. This contribution has two
major effects on shear measurements. Firstly, flux contamination between well-
identified close neighbours will lead to biased estimates of galaxy shapes
(Dawson et al., 2015; Martinet et al., 2019; Hoekstra et al., 2017). Secondly,
in the regime of unrecognized blends, not only the measured shapes of galaxies
will be biased, but the number of galaxies available for weak lensing will be
decreased, as a group of galaxies is mistakenly identified as one (Sanchez et
al., 2021).

It is expected that blending will not only be a function of density and
distance between galaxies, but will also depend on the shapes and alignments
of galaxies. For instance, Dawson et al. (2015) showed that unrecognized
blending will affect more strongly highly elliptical objects. This is of
particular relevance for WL analysis as it probes alignments and ellipticities
of galaxies. In order to account for blending properly in WL and other probes,
it is paramount that simulations provide a realistic account of the spatial
distribution of galaxy shapes.

Since deblending is a promising avenue to mitigate the impact of blending,
simulations should serve as a realistic testing ground for algorithms.
Deblenders can make various assumptions as to the morphological properties and
intrinsic alignments of the objects they intend to model. Providing deblender
developers with realistic distributions of shapes and sizes will help refine
their algorithms in view of upcoming real images (Sheldon et al., 2020).

####  2.2.2 Galaxy Shape, Size and Morphology Data Sets

The available data that we use to validate the sizes of faint galaxies is
obtained from parametric, two-dimensional, intensity-profile fits (Mandelbaum
et al., 2014) to postage stamps of COSMOS galaxies (Mandelbaum et al., 2012)
with $i<25.2$. The fitting procedure used in this work was first presented in
Lackner & Gunn (2012).

For validating galaxy sizes as a function of redshift, we use the measurements
of van der Wel et al. (2014) that combine data from 3D-HST (Brammer et al.,
2012) and CANDELS (Koekemoer et al., 2011; Grogin et al., 2011) to provide
size distributions as a function of rest-frame V-band luminosity in six
redshift bins spanning the range $0<z<3$ (see Table 5, van der Wel et al.,
2014). For validating the sizes of the galaxy disk and bulge components, we
use the fits provided in Mandelbaum et al. (2014), obtained from COSMOS data,
which give estimates of the size distributions for each galaxy component as a
function of rest-frame F814W luminosities for four redshift bins spanning the
range $0<z<2$.

Validation data for the distribution of galaxy ellipticities is available from
the COSMOS survey (Joachimi et al., 2013). These data are available for four
morphological classes: luminous red galaxies (LRG), disk, early and late type
galaxies in the redshift range $0<z<2$ and are selected based on $i$-band and
absolute $V$-band magnitude cuts of $i<24$ and $M_{V}<-19.0$ and
$-21.0<M_{V}<-17.0$ for the LRG and other morphology classes respectively. The
selection cuts to determine the morphology class are quite complex and would
be challenging to implement for a synthetic catalog. Instead, as described in
Section 4.8, in order to approximate the morphological selections in the
COSMOS sample, we implement selections on $B/T$, the ratio of bulge-to-total
fluxes in $i$-band.

####  2.2.3 Size, Shape and Morphology Tests

The galaxy orientation, size and shape tests that we implement are comparisons
with both theoretical expectations and observational data for the following
distributions:

  * •

the position angle distribution (Section 4.5),

  * •

the cumulative distribution function (CDF) of galaxy sizes for a faint-galaxy
sample (Section 4.6),

  * •

the size-luminosity relation as a function of absolute magnitude and redshift
(Section 4.7)

    * –

for the total light profile (Section 4.7.1),

    * –

for the disk- and bulge-component light profiles (Section 4.7.2),

  * •

the ellipticity distributions for selected morphological classifications and
luminosities (Section 4.8).

The first test is designed to check that the position angle of galaxies is
randomly distributed with a uniform probability distribution. Although this is
an assumption that is normally built into catalogs, it is sufficiently
important to require a test ensuring that the uniformity has propagated to the
final catalog.

The second test ensures that the slope of the galaxy-size distribution near
our faint-magnitude resolution cutoff is approximately realistic, so that any
selection effects in the catalogs derived from images are similar to what they
would be in real data.

The third and fourth tests check that galaxy sizes scale with luminosity as
expected and that the ellipticities of selected sub-samples of galaxies follow
the observed distributions, respectively.

###  2.3 Galaxy Correlation Functions

####  2.3.1 Scientific Drivers and Critical Properties

Two-point correlation functions measure the scale- and time-dependent
properties of the clustering of galaxies. In particular, two sets of
correlation functions will be particularly well-measured by LSST: the density
(or number count) of galaxies, and the correlation between their shapes. The
former, galaxy clustering, uses the galaxy density as a (biased) tracer of the
underlying dark matter field. The latter, cosmic shear, measures the
gravitational lensing applied to galaxy populations by intervening matter
fields. In imaging surveys such as LSST, these correlation functions are two-
dimensional: they are measured on thick tomographic bins of galaxies
determined via photmometric redshifts.

Galaxy clustering measurements provide a very high signal-to-noise but biased
probe of the underlying matter field (Davis et al., 1978; Dressler, 1980). The
bias is expected to evolve as a function of redshift and scale and this
unknown multiplicative uncertainty limits the power of galaxy clustering on
its own.

Lensing measurements provide a nearly-unbiased measurement of the underlying
matter density, integrated over the line of sight to each galaxy (Mandelbaum,
2018). Lensing tomography is sensitive to a combination of the growth of
structure, since higher mass densities act as stronger lenses, and to the
expansion history of the universe, since the expansion modifies the geometry
of the observer-lens-source system.

As mentioned in Section 2.1.1, the $3\times 2$pt analysis combines two-point
clustering measurements with lensing auto- and cross-correlations. The power
of this analysis lies in the observation that the unknown galaxy bias and
possible systematic photometric redshift errors can be constrained by a
combination of measurements (DES Collaboration, 2018). Since these systematic
effects contribute a large component of the overall error budget in the
analysis, we can significantly improve our cosmological constraints using this
combination. The three parts of this analysis each place demands on the
fidelity of the catalog; errors in any of them will distort the recovered
cosmological parameters.

Galaxy clustering correlations, and especially tomographic auto-
correlations777In “perfect” surveys, density correlations between different
redshift bins would be negligibly small, but in real surveys redshift errors
induce measurable correlations. in simulated catalogs must be as realistic as
possible to avoid either spuriously powerful constraints (if the model used is
too simple) or incorrect ones. In particular, the galaxy bias prescription
used to populate the simulation with galaxies is critical.

Many different prescriptions to describe galaxy bias have been developed
(Kaiser, 1984; Bardeen et al., 1986; Bernardeau, 1996; Manera & Gaztañaga,
2011; Desjacques et al., 2018). The uncertainties in the constraints on this
relation strongly increase the errors in the dark energy equation of state or
gravitational growth index (Eriksen & Gaztanaga, 2015). Thus, having realistic
models for galaxy bias is key for studying how to obtain the maximum
performance in current and future data analyses. In particular, given the
increased statistical power of modern surveys, linear biasing models (Gruen et
al., 2018) are no longer sufficient. Furthermore, since galaxy bias depends on
galaxy formation and evolution, it necessarily depends on the galaxy
population selected for the analysis. The validation of galaxy bias is
therefore a complex and challenging issue. However, a minimal expectation for
the catalog data is that the galaxy bias inferred from fitting a linear bias
model to the galaxy 2-pt correlation functions shows increasing bias with
redshift. This behavior is expected because the sample of galaxies used to
evaluate the 2-pt correlation functions is flux limited and hence includes
more faint, low-bias galaxies at low redshift.

There is less ambiguity about the model that should be used to obtain faithful
galaxy lensing from a simulation: it should match the integrated distortion
along the line of sight. The practical calculation of this quantity, however,
can be computationally demanding. Errors in going from snapshot information to
integrated shear can induce multiplicative biases in overall shear measurement
(Petri et al., 2017). Furthermore, since some cosmic shear statistics probe
small physical scales, down to $k\sim 10h^{-1}$Mpc, the fidelity of the
simulation’s matter power spectrum on these scales must also be well-
understood in order to account for the possible biases that result from
uncertainties in the simulations at small scales.

The final component of the $3\times 2$pt analysis is the cross-correlation of
foreground (lens) galaxy positions and background (source) galaxy shear,
commonly known as galaxy-galaxy lensing. This measurement probes the average
halo profile for a given foreground galaxy sample (Bartelmann & Schneider,
2001) and provides a powerful tool for studying the galaxy-halo connection as
shown, for example, in Mandelbaum et al. (2006). Since this cross-correlation,
as the third probe in the $3\times 2$pt scheme, is crucial for breaking
degeneracies between cosmological parameters and nuisance parameters
(photometric redshift errors, galaxy bias, baryonic effects and intrinsic
alignments), it too must be realised to high accuracy in synthetic catalogs.

A final scientific application that is dependent on the fidelity of the large-
scale structure of synthetic catalogs is the determination and calibration of
redshifts. Many cosmological analyses rely on detailed knowledge of the
redshift distribution for tomographic subsets of the overall sample. A
powerful set of techniques have been developed that use the fact that galaxies
are embedded in large scale structure and thus have strong spatial
correlations with other objects that are nearby. Newman (2008) presented a
method to determine the redshift distribution for a set of objects precisely
and accurately by measuring their spatial cross-correlation with a smaller set
of objects with known spectroscopic redshifts. More recent techniques (McQuinn
& White, 2013; Rahman et al., 2015; Sánchez & Bernstein, 2019; Rau et al.,
2020) have combined photometric and clustering information into a single
redshift estimator. While powerful, these methods are dependent on
complementary estimates of both the weak lensing magnification signal and the
evolving galaxy bias of the samples. In order to model these systematic
effects, any simulation must include realistically complex galaxy clustering
and magnification signals, as well as a complex galaxy bias evolution model
mapping how the galaxies populate dark matter halos.

####  2.3.2 Correlation-Function Data Sets

Several measurements are available for validating galaxy clustering at low
redshifts. From the SDSS DR7 galaxy sample, Wang et al. (2013) present results
for the angular correlation function $w(\theta)$, the over-abundance of galaxy
pairs at an angular separation $\theta$, relative to a random distribution.
Table 2 in that work gives $w(\theta)$ for four magnitude limited samples
based on SDSS $r$-band magnitudes. The angular separations vary from $\sim
0.005$ to $\sim 10$ degrees. The median redshift for this sample is 0.22. The
values of $w(\theta)$ are measured using the Landy-Szalay estimator (Landy &
Szalay, 1993) and are in agreement with earlier measurements, but offer much
higher precision due to the large size of the galaxy sample and the uniformity
of the data.

To check the estimated galaxy bias, $b(z)$, we use the empirical fit obtained
by Nicola et al. (2020), which analyzes the clustering of magnitude-limited
samples of galaxies in tomographic redshift bins from the first data release
of HSC Subaru Strategic Program (HSCSSP). Equation 4.12 from that work
provides a simple fitting function for the linear galaxy bias which is given
by

$b(z)=(b_{1}*(m_{\rm lim}-24)+b_{0})*D(z)^{\alpha},$ (1)

where $b_{1}=-0.0624\pm 0.0070$, $b_{0}=0.8346\pm 0.161$, $\alpha=1.30\pm
0.19$, $D(z)$ is the cosmology-dependent linear growth factor and $m_{\rm
lim}$ is the limiting magnitude of the data sample. Equation 1 was obtained
for an HSCSSP data sample with $m_{\rm lim}=24.5$.

SDSS data for the projected correlation function, $w_{p}(r_{p})$, is available
for low redshifts in the range $0.02<z<0.25$ and for various selection cuts on
the magnitude and colors of the galaxy sample (Zehavi et al., 2011). Tables
C7, C9, and C10 in Zehavi et al. (2011) provide measurements of volume-limited
samples of galaxies in absolute $r$-band (Hubble parameter $h=1$) magnitude
bins ranging from $-23<M_{r}^{h}<-17$ for the entire sample and
$-23<M_{r}^{h}<-19$ for the blue sample and red samples, respectively.
Galaxies are separated into blue and red samples by using a magnitude-
dependent color cut defined by
$(M_{g}^{h}-M_{r}^{h})_{\rm{cut}}=0.21-0.03M_{r}^{h}$. In order to ensure that
the galaxy samples in each magnitude bin are volume limited, bin-specific
redshift cuts are applied. All of these cuts must be reproduced for the
catalog selections. We reiterate that the absolute magnitude values in Zehavi
et al. (2011) are quoted for $h=1$, and need to be corrected using

$M_{r}=M_{r}^{h}+5\log_{10}h,$ (2)

for the cosmology used to simulate the synthetic catalog.

For testing the evolution of galaxy clustering at higher redshifts than those
found in SDSS galaxy samples, we use data available from the DEEP2 survey
(Mostek et al., 2013). The projected auto-correlation functions measured from
these data have been fit to power-laws for stellar-mass limited samples and
are valid for redshifts $0.74<z<1.05$. These data are particularly convenient
for validating synthetic catalogs because stellar-mass cuts are very
straightforward to implement and interpret at the catalog level. Additional
validation data for magnitude-limited samples are available from the VIPERS
survey (de la Torre et al., 2013; Marulli et al., 2013). Marulli et al. (2013)
also present results for stellar-mass limited samples of the data, which have
slopes similar to those of the DEEP2 data, but whose normalizations are
somewhat higher.

We now turn to the available data sets and theoretical predictions for
validating weak-lensing quantities. First, we use the predictions for the
shear-shear auto-correlations as a function of angular separation to validate
the catalog shear-shear signal at large angular scales. These predictions are
obtained from linear theory in CAMB (Lewis et al., 2000) and the Core
Cosmology Library (CCL; Chisari et al., 2019).

For validating the galaxy-galaxy lensing signal, there are several reference
sources available. A crucial feature in the implementation of any test based
on observational data is to ensure that the observed lens galaxy selection can
be reproduced in the synthetic catalog thereby checking how the galaxy-halo
connection in the synthetic catalog matches with that of data. Our first data
set is the galaxy-galaxy lensing signal from the Baryon Oscillation
Spectroscopic Survey (BOSS) LOWZ sample presented in (Singh & Mandelbaum,
2016). This sample consists of LRGs with $z<0.4$ selected from SDSS DR8. The
selection cuts to reproduce this sample are presented in Reid et al. (2016)
and consist of color selections based on SDSS $g-r$ and $r-i$ colors, a color-
dependent $r$-band magnitude cut and a selection cut on $r$-band magnitudes
requiring that $16<r<19$.

A second set of validation data uses the galaxy-galaxy lensing signal from the
SDSS Locally Brightest Galaxies (LBG) sample presented in Mandelbaum et al.
(2016). The LBG sample is defined in Anderson et al. (2015) and is a subsample
of the flux-limited SDSS Main galaxy sample (Blanton et al., 2005). This LBG
sample has accurate stellar mass and color measurements, which allow us to
test synthetic catalogs in these different parameter spaces. The selection
cuts that must be reproduced for the catalog lens sample include redshifts in
the range $0.03<z<0.2$, SDSS $r$-band magnitudes with $r<17.7$ and stellar
masses in the range $\log_{10}(M^{*}/M_{\odot})>10$. The measurements of
CFHTLenS (Velander et al., 2014) also provide further more qualitative checks
of the galaxy-galaxy lensing signal at higher redshifts.

####  2.3.3 Correlation Function Tests

The correlation function tests we implement in our suite are based directly on
the available validation data and are as follows:

  * •

galaxy-galaxy two-point correlation as a function of angular separation for
magnitude limited samples at low redshfit (Section 4.9),

  * •

comparison of galaxy bias with a linear bias model (Section 4.10),

  * •

galaxy-galaxy two-point correlation as a function of projected distance
(Section 4.11):

    * –

for magnitude limited samples at low redshift (Section 4.11.1),

    * –

for color-selected samples at low redshift (Section 4.11.2),

    * –

for samples selected by stellar mass at higher redshift (Section 4.11.3),

The first test, which validates the two-point angular correlation function,
provides a relatively simple test of galaxy clustering because it does not
require the implementation of a complicated set of selection cuts to reproduce
the observed data sample.

The galaxy bias test compares the galaxy angular power spectrum with
theoretical predictions to obtain an approximate estimate of the linear bias.

The third set of tests, which validate the spatial clustering of galaxies as a
function of projected distance for different galaxy sub-samples, provides more
complicated tests of the two-point clustering because of the selection cuts
that must be applied to the catalog in order to mimic those used in the
observational data. This means that the test results will depend on the
details of these selection cuts and therefore this set of tests constitute
more stringent checks of the fidelity of the catalog.

Turning now to weak-lensing statistics, we implement two tests to check the
validity of shear correlations:

  * •

shear-shear auto-correlations as a function of redshift (Section 4.12),

  * •

the galaxy-galaxy lensing signal as a function of projected distance (Section
4.13),

    * –

for a mock LOWZ (LRG) galaxy sample (Section 4.13.1),

    * –

for a mock LBG galaxy sample in bins of stellar mass and color (Section
4.13.2).

The first test is a key validation test for catalogs intended to mimic weak
lensing surveys and verifies that the auto-correlations of the cosmological
shear values in the synthetic catalog match the theoretical predictions from
linear theory for sufficiently large scales.

The two galaxy-galaxy lensing tests validate the galaxy-shear correlations for
different sub-samples of lower redshift galaxies with $z\lesssim 0.4$. The
first test validates the galaxy-halo connection for the arguably important LRG
galaxy sample. The second test provides a more stringent validation of the
galaxy-halo connection owing to the further subdivisions of the sample by
stellar mass and color.

###  2.4 Galaxy Clusters

####  2.4.1 Scientific Drivers and Critical Properties

Galaxy clusters form in the most massive dark matter halos and are the
largest-known gravitationally collapsed objects in the universe. Clusters act
as high-density tracers of large scale structure. The number density and
spatial distribution of clusters as a function of their masses and redshifts
are sensitive probes of cosmology. In particular, the measurement of cluster
abundance is sensitive both to the growth of structure and to the geometry of
the universe (Allen et al., 2011, and references therein).

Galaxy clusters are detected from their peculiar observational signature with
respect to the background. In the optical and infrared, clusters appear as
characteristic spatial concentration of galaxies forming a peak in the
redshift distribution, and displaying similar colors and a characteristic
luminosity distribution. Their center is taken to be the most likely central
galaxy, that is often also the brightest cluster galaxy (BCG).

The LSST DESC reference cluster detection algorithm, redMaPPer, finds galaxy
clusters in optical surveys by identifying over-densities of red-sequence (on
a color-magnitude diagram) member galaxies. The redMaPPer richness, $\lambda$,
is a cluster observable defined as the sum of the membership probabilities of
the galaxies within a specified radius and above a luminosity threshold whose
scatter in the mass-richness relation is minimized (Rykoff et al., 2014). The
membership probability of one galaxy is determined by its position, color and
magnitude (Rykoff et al., 2012, 2014, 2016). For each detection, redMaPPer
outputs a list of potential member galaxies, as well as estimates of the
richness, redshift and most likely central galaxy. In order to provide a
redshift independent mass estimator, optical cluster finders limit the
membership assignment based on $L_{*}(z)$, the knee of the luminosity function
(Rykoff et al., 2016; Aguena et al., 2021b).

In surveys, cluster masses are inferred from observables, such as cluster
member galaxy counts (i.e., richness), X-ray luminosities, weak lensing
measurements, galaxy velocity dispersions or the flux of the
Sunyaev–Zel’dovich (SZ) effect on the Cosmic Microwave Background. Those
observables are then linked to the true cluster masses via mass-observable
relations (MOR), which need to be calibrated. This is one of the most critical
aspects of using clusters as robust cosmological probes. The MOR calibration
often relies on gravitational lensing, since this method does not require
assumptions about the cluster dynamical state and is believed to be less
biased (at least for high mass systems) than, e.g., hydrostatic mass estimates
from X-ray measurements (von der Linden et al., 2014; Mantz et al., 2015;
Becker & Kravtsov, 2011). In the optical, the lensing masses are usually
obtained by fitting the radial profile of the shear signal of background
sources around clusters, by assuming a given form of the underlying radial
mass profile. In order to enhance the signal-to-noise, which is low for an
individual cluster, the radial shear signal is often stacked, by grouping
clusters in bins of richness and redshift.

Another critical element for the use of cluster samples is the determination
of their selection function (i.e., how well the detected sample represents the
true population). For optical surveys, apart from survey design and detection
algorithm specification, the selection function depends not only on the true
mass (or observable proxy) and redshift of the systems, but also on many
aspects of the physics of clusters (see next paragraph for examples) and the
underlying covariance between their measured masses and observables.

The assembly of cluster samples usable for cosmological analyses thus requires
knowledge of the detection efficiency, the correct measurement of their
observables, an estimation of their masses and redshifts and the evaluation of
the sample selection function. All steps should be extensively tested and
validated by using simulated data sets. This is particularly challenging
because cluster analyses require many aspects of the synthetic catalog to be
as realistic as possible, e.g., member galaxy redshift, luminosity and color
distributions, member galaxy spatial distributions and their variation from
the field to the inner regions of clusters, the gravitational lensing
implementation, the ability to determine high-quality photometric redshifts
and the link between the galaxy distribution and properties and the underlying
dark matter distributions.

In this analysis we focus on validating the most important cluster properties
responsible for their detection and mass estimation in the future LSST data.
In particular, we base many of our requirements on the use of the red-sequence
Matched-filter Probabilistic Percolation cluster-finder (redMaPPer) detection
algorithm (Rykoff et al., 2014, 2016). However, as we test for generic cluster
properties, our validation scheme should also enable tests of other detection
methods.

####  2.4.2 Galaxy Cluster Validation Data

For the validation of synthetic clusters, we use both observational data
samples and parametrizations measured from observed data or other simulations.
We include measurements of observed clusters, simulated halos and cluster-
member galaxies.

Polynomial fits to measurements of the mean $g-r$, $r-i$ and $i-z$ colors, as
a function of redshift, for red-sequence galaxies observed by DES (Rykoff et
al., 2016) provide a convenient summary of the expected dependence of the
color of cluster-member galaxies with redshift.888These fits were also used in
the cosmoDC2 production pipeline as described in Korytov et al. (2019). We
also assemble a validation data sample of 2661 redMaPPer clusters by selecting
clusters from the publicly available redMaPPer catalog (version 6.4) for DES
Y1 data 999 https://des.ncsa.illinois.edu/releases/y1a1/key-catalogs/key-
redmapper. For each cluster, this catalog provides a redshift and color
information and membership probabilites for the cluster members. In selecting
this sample, we impose a richness cut of $\lambda>30$ and we assign a median
color to the cluster by computing the median color for all cluster member
galaxies whose membership probability is greater than 70%.

The evolution of the luminosity of member galaxies as a function of redshift,
which impacts the detection efficiency for cluster members, requires an
estimation of the evolution of $L_{*}(z)$ (the knee of the luminosity
function). We derive the corresponding apparent magnitude $m_{*}(z)$ (assuming
that LSST $i$-band is used for the detection) from the passive evolution of a
burst galaxy with a formation redshift $z=3$, taken from the PEGASE2 library
(burst_sc86_zo.sed, Fioc & Rocca-Volmerange, 1997). The relationship is
calibrated using the value of ${\rm K}^{\star}$($z=0.25$) (the value of
$m_{*}(z)$ in K band) derived by Lin et al. (2006) from an observed cluster
sample. We use this estimate as a prediction for $m_{*}(z)$.

In order to estimate the luminosity distribution of member galaxies within
galaxy clusters, we begin with the redMaPPer v5.10 cluster catalog derived
from the SDSS DR8 data set, which covers approximately $10405$ $\rm{deg}^{2}$
with $i$-band depth of $\sim 20.9$ (Rozo et al., 2015). We select clusters
with redshift ($z_{\lambda}$) less than $0.3$ to avoid the selection effects
from the SDSS survey depth and richness $\lambda$ greater than $5$ to maximize
the signal to noise. This results in a catalog consisting of $80507$ clusters.
We then compute the conditional luminosity function following the method
described in To et al. (2020).

For the validation of cluster galaxy-density profiles, we use the
parametrization of galaxy density-profiles from Hennig et al. (2017), measured
on a set of SZ selected galaxy clusters. High mass SZ selected cluster samples
are known to present high purity and to be close to pure mass selection (Bleem
et al., 2020; Huang et al., 2020; Hilton et al., 2021). The mean galaxy-
density profiles in these clusters can thus be compared to that of massive
halos in simulations. The clusters used in Hennig et al. (2017) are detected
from the South Pole Telescope survey (Bleem et al., 2015) and their galaxy
density profiles are measured from optical data taken during the science
verification phase of the Dark Energy Survey (Jarvis et al., 2015). The
evolution of the profiles with mass and redshift are studied by using NFW
parametrizations (Navarro et al., 1997) of the profiles.

The data sets available for validating the mass-richness relation of galaxy
clusters are obtained from the DES (Year 1 and Science Verification) and the
SDSS surveys. McClintock et al. (2019) use the DES Y1 data set ($\sim$1500
deg2), with redMaPPer depth $0.2L_{*}$ at $z=0.7$, to obtain $>6500$ clusters
with richness $\lambda>20$. They divide these redMaPPer clusters into $4\times
3$ richness ($\lambda\geq 20$) and redshift ($0.2\leq z\leq 0.65$) bins and
measure the mean masses in these bins via the stacked weak lensing signal. The
modeled mass-richness relation (i.e., the expectation value of the halo mass
at a given richness and redshift) is determined to be:

$\langle M_{\rm
200m}|\lambda,z\rangle=M_{0}\left(\frac{\lambda}{\lambda_{0}}\right)^{F}\left(\frac{1+z}{1+z_{0}}\right)^{G},$
(3)

where $M_{0}=[3.081\pm 0.075(\rm stat)\pm 0.133(\rm sys)]\cdot
10^{14}M_{\odot}$, $F=1.356\pm 0.051(\rm stat)\pm 0.008(\rm sys)$, $G=-0.30\pm
0.30(\rm stat)\pm 0.06(\rm sys)$. Equation 3 uses the halo mass definition
$M_{\rm 200m}$, which denotes the spherical overdensity (SO) mass within a
sphere whose mean internal density is 200 times the mean matter density of the
universe at redshift $z$. The pivot values $\lambda_{0}=40$ and $z_{0}=0.35$
are close to the median values for the McClintock et al. (2019) cluster
sample. For the DES SV and SDSS surveys, in order to facilitate easy
comparisons of the data, McClintock et al. (2019) provide corrections for the
impact of analysis details that could cause the definitions of richness to
differ among the surveys. We use these corrected results as additional
validation data.

A comparison of the halo masses recovered from measured weak-lensing
quantities with those obtained directly from a synthetic catalog provides a
consistency test that, ideally, would result in an exact match between the two
quantities for any given halo. In reality, there will be scatter in this
relationship and the amount of scatter will depend on the realism of the
catalog and the underlying systematic effects that may or may not have been
included. This type of test does not require any external data, but does
require a number of theoretical inputs to obtain best-fit mass estimates from,
e.g., excess surface-density profiles. For this case, theoretical predictions
for the excess surface-density are obtained using the DESC
CLMM101010https://github.com/LSSTDESC/CLMM [github.com] library assuming an
NFW dark-matter profile (Navarro et al., 1997) and a concentration-mass
relation (Duffy et al., 2008). We note that such an analysis delivers
estimates of $M_{\rm 200m}$, whereas the synthetic catalog may use an
alternate mass definition such as a Friends-Of-Friends (FoF) halo mass. In
order to obtain the most meaningful consistency check, it is necessary either
to estimate the effect of using different mass definitions or, if possible, to
convert the masses from one definition to another.

Another proxy for the mass of a galaxy cluster is the velocity dispersion of
member galaxies. Evrard et al. (2008) present an estimate of the bulk virial
scaling relation for dark matter halos that is determined from an ensemble of
simulations. They determine that the dark matter velocity dispersion,
$\sigma_{DM}(M,z)$, in halos with masses larger than
$10^{14}h^{-1}\text{M}_{\odot}$ scales with the halo mass as

$\sigma_{DM}=\sigma_{DM,15}\left(\frac{h_{100}(z)M_{\rm
200c}}{10^{15}\textup{M}_{\odot}}\right)^{\alpha},$ (4)

where $\sigma_{DM,15}$ = 1082.9 $\pm$ 4.0 km/sec, $\alpha$ = 0.3361 $\pm$
.0026, $h_{100}(z)$ denotes the dimensionless Hubble parameter normalized by
100 ${\rm km}~{}{\rm sec}^{-1}{\rm Mpc}^{-1}$ and $M_{\rm 200c}$ denotes the
SO mass defined as the enclosed mass within a sphere whose mean internal
density is 200 times the critical density of the universe at redshift $z$. The
dark matter velocity dispersion is related to the galaxy velocity dispersion
via a dimensionless factor, the velocity bias, which is defined as
$\sigma_{gal}/\sigma_{DM}$. The values of velocity bias can be determined from
simulations and are scattered in the range 0.9 to 1.1, with averages that vary
from 0.96 at high redshift to 1.04 at low redshift (Evrard et al., 2008). A
more recent study (Gifford et al., 2013) has determined the velocity bias for
a number of semi-analytic galaxy models and for a variety of galaxy samples
with varying fractions of red, blue, bright and dim galaxies. The values of
the bias in most of these cases vary between 0.95 and 1.05 except for samples
with low fractions of red galaxies which have biases as high as 1.1. For
simplicity, we choose the velocity bias to be equal to 1 for validation test
in this paper.

####  2.4.3 Galaxy Cluster Tests

Based on the requirements presented in Section 2.4.1 and the available
validation data sets, we develop 7 validation tests for clusters. These are
based either on the properties of their galaxies:

  * •

the color-redshift relation of central galaxies (Section 4.14),

  * •

the redshift evolution of member-galaxy magnitudes in comparison to $m_{*}$,
(i.e., $L_{*}$ converted to the corresponding $i$-band apparent
magnitude111111See Section 2.4.2) (Section 4.15),

  * •

the shape of the cluster galaxy conditional luminosity function (Section
4.16),

  * •

the shape of the number-density profile of member galaxies (Section 4.17));

or on the connection between the true halo mass and different observables:

  * •

the mass-richness relation for redMaPPer selected clusters (Section 4.18),

  * •

the weak lensing cluster profiles and the associated measured mass (Section
4.19),

  * •

the velocity-dispersion–halo-mass relation (Section 4.20).

The ability to run redMaPPer to identify clusters in a synthetic catalog is a
non-trivial requirement that demands the existence of a clearly identifiable
red sequence with tightly constrained scatter on a color-magnitude diagram for
galaxies in halos with masses $M_{halo}\gtrsim 10^{13}M_{\odot}/h$. The first
test checks for the presence of the red sequence and is a minimal requirement
for a synthetic cluster catalog intended for use by optical surveys.

As described in Section 2.4.2, the evolution of the luminosity of member
galaxies with redshift impacts the performance of optical cluster finders.
Therefore, to test for the consistency of cluster detection as a function of
redshift, the second test checks how well the redshift evolution of halo-
member magnitudes is traced by $m_{*}$. This test ensures that a richness
estimation based on $m_{*}$ is redshift independent and is also an important
requirement for alternative optical cluster finders such as WaZP (Aguena et
al., 2021b).

The performance of redMaPPer also depends on the relative luminosity
distribution of cluster central and satellite galaxies. Thus, in order to
ensure that the performance of redMaPPer on the synthetic catalog is
sufficiently realistic, it is important to check the fidelity of the
luminosity functions against the observed data. Ideally, we want to compare
the luminosity function in bins of halo masses; however, halo masses are not
an observable in the real data. Instead, in the third test, we measure the
luminosity function of centrals and satellites in bins of richness
($\lambda$), which serves as a low-scatter halo mass proxy for redMaPPer
clusters.

The fourth test checks the number-density profile of member galaxies, which is
one of the drivers of cluster detection (see, e.g., Euclid Collaboration,
2019). The distribution of member galaxies not only impacts the detection but
also the richness estimation (Rykoff et al., 2014, 2016). Moreover, it can be
used as a proxy of the underlying dark matter distribution (Guo et al., 2012;
Agustsson, 2010) and at larger radii, it is also important for testing weak-
lensing mass measurements and possible splash-back effects.

Apart from the cluster member properties, it is also crucial to validate the
link between cluster observables and mass. The fifth test provides a way to
investigate this link by determining the cluster mass-richness relation in the
synthetic catalog and comparing it to that measured with redMaPPer catalogs
from, e.g., DES (see Section 2.4.2) Gravitational lensing will be the primary
probe of mass for LSST clusters, it is therefore mandatory to test the
validity of the shear profiles around massive halos. Moreover, the sixth test
then naturally arises from examining the relation between the true halo mass
and the estimate of the mass recovered from weak lensing measurements

Finally, as mentioned in Section 2.4.2 the velocity dispersion of a dark
matter halo acts as another proxy for its mass and therefore the seventh test
which examines the validity of the velocity dispersion-mass relation thus
provides another significant validity test for cluster cosmological analyses.

###  2.5 Emission Line Galaxies

####  2.5.1 Scientific Drivers and Critical Properties

Nebular emission lines in galaxies are a critical tracer of many elements of
galaxy physics, including gas phase metallicity, star formation rate, and dust
content (Runco et al., 2021). Additionally, they have been found to have a
significant effect on galaxy colors used for photometric redshift fitting
(Csörnyei et al., 2021), indicating that improper emission line modeling could
lead to a reduction on photometric redshift precision and accuracy.
Measurements of large-scale structure (LSS) with LSST will rely heavily on the
ability to recover galaxy redshifts accurately from photometry using
photometric redshift fitting, as galaxy clustering analyses require splitting
the sample into narrow redshift bins to perform angular clustering analysis
(Nicola et al., 2020).

Typically, so-called red sequence galaxies are used for such fits because the
Balmer break – the strong decrease in flux density at wavelengths bluer than
rest-frame $4000$ Å – greatly increases the precision of photometric redshift
fits (Gladders & Yee, 2000; Rozo et al., 2016; Stoughton et al., 2002).
However, an alternative analysis that is being actively pursued within LSST
DESC focuses on another sub-population of galaxies that could yield high-
quality photometric redshifts by utilizing strong emission line features,
rather than continuum features. We refer to such objects as extreme emission
line (EEL) galaxies. Because emission lines are isolated to very specific
wavelengths, it is possible that EEL galaxies could yield comparable or higher
redshift precision than red sequence galaxies, allowing for improved
measurements of galaxy clustering by reducing the redshift width of redshift
bins and reducing contamination between adjacent bins. Indeed, Csörnyei et al.
(2021) find that galaxy emission lines often have a more pronounced effect on
galaxy colors than photometric uncertainties. As a result, galaxy catalogs
with realistic emission-line properties are crucial to the development of this
analysis.

####  2.5.2 Emission Line Data Sets

Data for emission line ratio validation are sourced from SDSS DR12 (Alam et
al., 2015). SDSS DR12 provides data from the culmination of SDSS-III
observations, which includes data from BOSS (Dawson et al., 2013). BOSS
provides optical spectra for over 2 million galaxies across 10,000 deg2 at
$z<0.7$. We select galaxies with non-zero measured continuum and emission line
fluxes in H$\alpha$, H$\beta$, [OIII]4959,5007, and [OII]3727,3729 to generate
a catalog of 87,949 objects at $z\lesssim 0.4$. While this SDSS catalog covers
a much smaller range in redshift than that of the LSST, we can use it broadly
as a test of the shapes of emission line ratio distributions.

####  2.5.3 Emission-Line Tests

We consider a combination of hydrogen (H$\beta$) and oxygen ([OII]3727,3729,
[OIII]4959,5007) emission lines for our emission line ratio test. These
emission lines are considered because (1) they are relatively bright in
$ugrizy$ bands at $z\lesssim 1.6$ and (2) ratios of different combinations of
emission line fluxes can function as proxies for physical properties of
interest. The ratio between $F_{\mathrm{H\alpha}}$ and $F_{\mathrm{H\beta}}$,
for example, probes dust attenuation of nebular light. Meanwhile,
$F_{\mathrm{[OIII]}}/F_{\mathrm{[OII]}}$ traces the ionization parameter and
is commonly known as $O_{23}$ (Nakajima et al., 2013). Oxygen and hydrogen
lines can be used together to calculate $R_{23}$, or
$(F_{\mathrm{[OIII]}}+F_{\mathrm{[OII]}}/F_{\mathrm{[H\beta]}})$, which is a
tracer of metallicity (Lilly et al., 2003).

###  2.6 Stellar Mass Function

####  2.6.1 Scientific Drivers and Critical Properties

The stellar mass function (SMF), which is defined as the number density of
galaxies as a function of their stellar mass, is one of the most fundamental
products of a synthetic galaxy catalog and acts as an important probe of the
galaxy-halo connection. Although the SMF was not identified by the LSST DESC
WGs as a critical static property requiring a specific validation criterion,
it is strongly correlated with the behavior of other observable properties
across cosmic time and therefore should be validated.

The stellar mass of a galaxy is assembled over time, with each burst of star
formation contributing an evolving component to its SED, as described by
stellar population synthesis models (Bruzual & Charlot, 2003). In fact, these
models form the basis for the inference of galaxy stellar masses from
observations of their SEDs and are a critical component in determining the SMF
from observed data. Recently, Chaves-Montero & Hearin (2020) have demonstrated
in model universes that galaxy colors are largely regulated by their recent
star-formation history and that variations in these histories are the dominant
influence on the distributions of galaxy colors. Thus there is a direct, but
complex, connection between the SMF and the SEDs, which are the fundamental
galaxy observables that are sampled by optical surveys. The SMF at a given
redshift provides a summary statistic for the galaxy content of the universe
at a given time, and the evolution of the SMF as a function of redshift traces
the evolution of the galaxy population and its properties over the history of
the universe (Behroozi et al., 2019). If the SMF as a function of redshift
does not agree with the observational data then the luminosity and color
distributions of synthetic galaxies can, at most, be right only at one
redshift.

The distributions of stellar masses and star-formation rates directly impact
the simulation of time-domain objects such as supernovae and strong lenses.
Volumetric supernova (SN) rates depend on the star-formation rates of their
host galaxies. The properties of Type Ia SN, which are of cosmological
interest, are known to depend on their host environments and are strongly
correlated with the stellar mass of their host galaxies (Sullivan et al.,
2010; Kelly et al., 2010; Lampeitl et al., 2010). Furthermore, Type Ia SN
occur more frequently in galaxies with high stellar masses so the properties
of these galaxies are of particular relevance for SN science. Although we do
not discuss the validation of time-domain objects in this paper, from the
above discussion it is clear that an accurate SMF over the range of redshifts
of interest is a crucial driver for enabling realistic simulations of the host
assignments of SN.

####  2.6.2 Stellar Mass Function Data Sets

The data available for the validation of the SMF comes from two sources.
Moustakas et al. (2013) use data from the PRIMUS and SDSS surveys to measure
the SMF in redshift bins with widths varying from $0.1$ to $0.2$ and spanning
the redshift range $0<z<1$. Tables 3 and 4 in Moustakas et al. (2013) present
SMF measurements over a wide range of stellar-mass bins spanning the range
$8.8<\log(M/M_{\odot}h_{70}^{-2})<12.0$. These data are reasonably consistent
with earlier measurements, given the differences in how the stellar masses
were defined in some of the earlier studies. More recent high-redshift
($0.5<z<1$) measurements of the SMF from the VIPERS survey (Davidzon et al.,
2013) are almost a factor two lower than the PRIMUS results and lie on the
lower boundary of earlier measurements for all redshift bins. We do not
include these data in our validation test.

Data from the BOSS DR12 CMASS galaxy sample are available to validate the SMF
for galaxies in the redshift range $0.4<z<0.7$. This data sample includes
specific selection cuts on their SDSS $g$, $r$, $i$ band-magnitudes and colors
as described in Reid et al. (2016). These cuts, given by:

  1. 1.

$d_{\perp}>0.55$ where $d_{\perp}=(r-i)-(g-r)/8$,

  2. 2.

$i<19.86+1.6(d_{\perp}-0.8)$,

  3. 3.

$17.5<i<19.9$,

  4. 4.

$(r-i)<2$,

select a sample of galaxies with high stellar masses and can be used to
validate a CMASS-like sample in the synthetic catalog.

####  2.6.3 Stellar Mass Function Tests

Our test suite includes two tests of the SMF that are based on the available
validation data described in Section 2.6.2:

  * •

the SMF in redshift bins matching those of the PRIMUS data (Section 4.22.1),
and

  * •

the stellar mass distribution for a mock CMASS galaxy sample (Section 4.22.2).

The first test checks the redshift dependence of the SMF for redshifts in the
range $0<z<1$. The second test narrows the selection to a bright galaxy sample
that provides a specific validation test for high stellar-mass galaxies.

###  2.7 Catalog Verification

As mentioned in Section 1, verification tests ensure that the contents of the
catalog have been correctly produced. Synthetic catalogs provide data for a
variety of analysis needs, one of the most important of which is the
extragalactic input for image simulations. This is a particularly costly use
of the catalog both in terms of computer time and human effort. It is
therefore critical to ensure that values of the galaxy properties supplied by
the catalog are finite, have reasonable distributions that lie within
acceptable physical ranges, that object identifiers are unique and that
quantities that should be related to each other in specific ways are related
as expected. Our validation test suite therefore includes a set of tests that
we dub readiness tests (Section 4.1), to verify that the catalog contents are
as expected.

###  2.8 Summary

In Table 1, we present a summary of the tests comprising the validation test
suite that we have discussed in this section. The tests are grouped by the
type of test (one-point distributions, two-point correlations, relations). The
science drivers relevant for each test and the section showing results for the
cosmoDC2 catalog are also listed, along with the sources of validation data
for each test. The tests listed in italics have quantitative validation
criteria.

We provide a complementary summary of the tests in the validation suite in
Table 2, where we have listed the selection cuts that have been applied to the
data for each test, as well as the validation criteria that have been
implemented. We also summarize the results obtained for the application of
these tests to the cosmoDC2 catalog, which will be discussed in Section 4. The
selections of redshift, magnitudes, colors and other quantities listed in this
table define the ranges of these quantities that are possible to validate with
the currently available observational data sets. Although there are
significant amounts of observational data in regions outside the ranges listed
in Table 2, they are problematic to use for validation purposes due to the
difficulty of understanding the selection effects and hence the completeness
of the data. As previously mentioned, the validation criteria listed in Table
2 have been developed by the LSST DESC WGs, and are specific to the validation
of the cosmoDC2 catalog for LSST science. In most cases, we have required
qualitative agreement between the catalog and validation data over all regimes
where suitable validation data exists. A few tests have more quantitative
validation criteria and the motivations for these are discussed in Section 4.

Test | Section | Science Drivers | Validation Data Sources
---|---|---|---
verification tests |  |  |
readiness | 4.1 |  |
one-point distributions |  |  |
cumulative number density | 4.2 | WL, BL, LSS |  Aihara et al. (2018a); Beckwith et al. (2006);
|  |  | Capak et al. (2007)
redshift | 4.3 | PZ, LSS, BL | Coil et al. (2004); Newman et al. (2013)
color (SDSS data) | 4.4.1 | PZ, CL, BL | Albareti et al. (2017)
color (DEEP2 data) |  |  |  Newman et al. (2013); Hudelot et al. (2012);
|  |  | Zhou et al. (2019)
position angle | 4.5 | WL |  Jones et al. (2001–) (uniform distributions)
size | 4.6 | WL, BL | Mandelbaum et al. (2014); Lackner & Gunn (2012)
ellipticity | 4.8 | WL, BL | Joachimi et al. (2013)
stellar mass (redshift dependence) | 4.22.1 | SN, SL, CL | Moustakas et al. (2013)
stellar mass (mock CMASS sample) | 4.22.2 | SN, SL, CL | Reid et al. (2016)
two-point correlations |  |  |
galaxy–galaxy angular | 4.9 | LSS, PZ, BL | Wang et al. (2013)
galaxy–density (bias) | 4.10 | LSS, BL | Nicola et al. (2020); Chisari et al. (2019)
galaxy–galaxy projected distance | 4.11.1 | WL, LSS, PZ, | Zehavi et al. (2011)
(luminosity dependence) |  | BL |
galaxy–galaxy projected distance | 4.11.2 | WL, LSS, PZ, | Zehavi et al. (2011)
(color dependence) |  | BL |
galaxy–galaxy projected distance | 4.11.3 | WL, LSS, PZ, | Mostek et al. (2013)
(redshift dependence) |  | BL |
shear–shear | 4.12 | WL | Lewis et al. (2000); Chisari et al. (2019)
galaxy–shear (mock LOWZ sample) | 4.13.1 | WL | Singh et al. (2015)
galaxy–shear (mock LBG sample) | 4.13.2 | WL | Mandelbaum et al. (2016)
relations (binned statistics) |  |  |
color–color | 4.4.2 | PZ, CL |  Newman et al. (2013); Albareti et al. (2017);
|  |  | Hudelot et al. (2012); Zhou et al. (2019)
size–luminosity (total light profile) | 4.7.1 | WL, BL | van der Wel et al. (2014)
size–luminosity (disk and bulge) | 4.7.2 | WL, BL | Mandelbaum et al. (2014)
color–redshift (red sequence) | 4.14 | PZ, CL | Rykoff et al. (2016)
cluster-member magnitude–redshift | 4.15 | CL | Fioc & Rocca-Volmerange (1997); Lin et al. (2006)
magnitude–halo mass (CLF) | 4.16 | CL | Rozo et al. (2015); To et al. (2020)
galaxy-density profile–halo mass | 4.17 | CL | Hennig et al. (2017)
halo mass–richness | 4.18 | CL | McClintock et al. (2019)
cluster weak-lensing masses | 4.19 | CL | Navarro et al. (1997); Duffy et al. (2008)
velocity dispersion–halo mass | 4.20 | CL | Evrard et al. (2008)
emission-line ratios | 4.21 | PZ, LSS | Alam et al. (2015)
Table 1: Summary table of tests, science drivers and validation data sources
for the tests in our test suite. The acronyms used for the science drivers
are: PZ: photometric redshift determination; WL: weak lensing; BL: blending;
SL: strong lensing; CL: cluster cosmology; LSS: large-scale structure
clustering; SN: supernovae. The sections in the paper in which each test is
described in detail are also given. The tests listed in italics are those with
specific quantitative validation criteria supplied by the LSST DESC WGs. See
Table 2 for a complementary summary of the data selections, more detailed
validation criteria and cosmoDC2 results for each of the validation tests.

=-1in  Test Redshift Other Selections Validation Criteria CosmoDC2 Selection
Status verification tests readiness none none visual inspection one-point
distributions cumulative number density none $\rm{mag}<27.5$ within $\pm 40$%
of HSCa $g$b, $r$, $i$, $z$ pass redshift $0.0<z<1.5$ $18<r<27$ qualitative
agreement fail for $r<=21$ color (SDSS data) $0.05<z<0.1$ $r<17.7$ Cramér-von
Mises $r-i$ passes color (DEEP2 data) $0.85<z<0.95$ $r<24$ statistic: $w<.05$
$g-r$ passes position angle none none uniform pass size none $i<25.2$c
$1<dN/dx<4$ pass ellipticity $0.0<z<2.0$ $i<24$, $-21.0<V<-17.0$d qualitative
agreement pass stellar mass (redshift dependence) $0.0<z<1.0$ none qualitative
agreement pass stellar mass (mock CMASS sample) $0.4<z<0.7$ $17.5<i<19.9$,
$r-i<2$e qualitative agreement pass two-point correlations galaxy–galaxy
angular none $17<r<21$ qualitative agreement pass galaxy–density (bias)
$0.2<z<1.2$ $i<25.3$ qualitative agreement pass galaxy–galaxy projected
distance none $-23<M_{r}^{h}<-19$ qualitative agreement pass (luminosity
dependence) galaxy–galaxy projected distance none $-23<M_{r}^{h}<-19$
qualitative agreement pass (red galaxies) (color dependence)
$(M_{g}^{h}-M_{r}^{h})_{\rm{cut}}=0.21-0.03M_{r}^{h}$ fail (blue galaxies)
galaxy–galaxy projected distance $0.74<z<1.05$
$\log_{10}(M^{*}/M_{\odot})>10.5,10.8$ $\chi^{2}<2$ for pass (redshift
dependence) $1-10$ $h^{-1}$Mpc shear–shear $0.5<z<3.0$ $M_{r}<-19$ qualitative
agreement pass galaxy–shear (mock LOWZ sample) $0.4<z<1.0$ $16<r<19$f
qualitative agreement pass for $r_{p}>0.4\rm{Mpc}/h$ galaxy–shear (mock LBG
sample) $0.03<z<0.2$ $r<17.7$, $\log_{10}(M^{*}/M_{\odot})>10$ qualitative
agreement pass (red galaxies); fail (blue galaxies) relations (binned
statistics) color–color $0.1<z<0.3$ $r<24$ Wasserstein distance fail
size–luminosity (total light profile) $0.0<z<2.5$ none qualitative agreement
pass size–luminosity (disk and bulge) $0.0<z<2.0$ none qualitative agreement
pass color–redshift (red sequence) $0.0<z<1.5$ $M_{\rm
halo}>10^{13.5}M_{\odot}$g qualitative agreement pass cluster-member
magnitude–redshift $0.0<z<1.5$ $M_{\rm halo}>10^{13.2}M_{\odot}$ qualitative
agreement pass magnitude–halo mass (CLF) $0.1<z<0.2$ $5<\lambda<100$h
qualitative agreement pass galaxy-density profile–halo mass $\langle
z\rangle$=0.46 $\langle M\rangle$=$6\times 10^{14}M_{\odot}$, $i<22$
qualitative agreement pass halo mass–richness $0.3<z<0.4$ $M_{FoF}>5.10^{13}$,
$\lambda>20$i qualitative agreement pass cluster weak-lensing masses
$0.1<z<0.7$ $M_{FoF}>10^{14}M_{\odot}$, $i<25$j qualitative agreement pass
velocity dispersion–halo mass none $M_{FoF}>10^{14}M_{\odot}$, qualitative
agreement pass $M^{*}>10^{9}M_{\odot}$, $N_{g}>=5$ emission-line ratios
$0.0<z<0.7$ $r<19.5$ $(z<0.4)$ 2d K-S test fail $i<19.9$ $(z>0.4)$

  * a

Extrapolated fit to HSC data in magnitude range $24.0<\rm{mag}<27.5$

  * b

$g$-band passes almost all of the required range. See Section 4.2 for details.

  * c

Galaxies are further selected such that their size is less than the median
size of the sample. See Section 4.6.

  * d

Additional selections on $B/T$ are described in Section 4.8.

  * e

Additional color and magnitude selections are described in Section 2.6.2.

  * f

Additional color selections are described in Reid et al. (2016).

  * g

Galaxies are further selected to be on the red sequence. See Section 4.14.

  * h

Test is run on redMaPPer-selected clusters from cosmoDC2

  * i

Additional selections are described in Section 4.18

  * j

Additional selections are described in Section 4.19

Table 2: Summary table of tests showing the redshift and other selection cuts
applied to the catalog data, the validation criteria and the test status for
the cosmoDC2 catalog. The selections listed in this table span the ranges of
availability of suitable validation data. See Table 1 for a complementary
summary of the science drivers, the validation data sources and the paper
section describing each test.

##  3 Simulated Catalogs and Validation Test Framework

In this section, we briefly describe the catalog and the underlying simulation
we employ to demonstrate our validation procedure, as well as the DESCQA
validation framework which we use to develop and run all of the validation
tests shown in Section 4.

###  3.1 Simulations

The cosmoDC2 catalog (Korytov et al., 2019; Hearin et al., 2020), which was
produced for the LSST DESC DC2, is based on a trillion-particle, ($4.225$
Gpc)3 cosmological, gravity-only N-body simulation, the ‘Outer Rim’ (Heitmann
et al., 2019). The mass resolution of this simulation is $2.6\cdot 10^{9}$M⊙.
The underlying cosmological model parameters are similar to those of WMAP7 and
the simulation identifies halos using a parallel FoF-based halo finder with a
linking length of b= 0.168 and the requirement of a minimum of 20 particles
per halo. Using a recent determination of the stellar-to-halo-mass relation
(Girelli et al., 2020), we expect that the stellar mass resolution of a
catalog based on this simulation to be of the order of a few $\times
10^{8}$M⊙. The cosmoDC2 catalog covers $440$ deg2 of sky area to a redshift of
$z=3$ and is complete to a magnitude depth of 28 in the LSST $r$-band. Each
galaxy is characterized by a multitude of galaxy properties including stellar
mass, morphology, SEDs, broadband filters magnitudes, host halo information
and weak lensing shear. The catalog has been constructed using a new hybrid
technique that combines data-based empirical approaches with semi-analytic
galaxy modeling. The empirical model establishes a restricted set of
fundamental galaxy properties, which are then completed by adding the
properties of matching galaxies obtained from the semi-analytic model (SAM).
In this approach, the empirical model is tuned to match the observational data
and the SAM galaxies function as a library from which to draw the large and
complex set of properties required by DC2. The method captures the highly
nonlinear correlations between galaxy properties that are built into the SAM
but preserves agreement with observational data by selective sampling of the
SAM library. The underlying assumption in the approach is that the quantities
tuned in the empirical model are sufficiently correlated with the quantities
available from the SAM so that the statistical distributions of the latter
will be realistic. Our validation suite tests the validity of this assumption.

The empirical model used for cosmoDC2 is based on UniverseMachine (Behroozi et
al., 2019) and is augmented with additional rest-frame magnitude and color
modeling. Since the parameters of UniverseMachine are optimized using two-
point correlation function data, the model is able to preserve stellar-mass
and star-formation-rate correlations as a function of environment. This model
is applied to Outer Rim halos to populate halo lightcones with galaxies. At
this stage, the galaxies have redshifts, positions, stellar masses, star-
formation rates, LSST $r$-band rest-frame magnitudes and $g-r$ and $r-i$
colors. Then, these galaxies are matched to those that have been obtained by
running the Galacticus SAM (Benson, 2012) on a small companion simulation to
Outer Rim. Owing to the computing time required to run the SAM, this companion
simulation is about 1600 times smaller than Outer Rim ($360$ Mpc)3. The
matching is done on the rest-frame magnitudes and colors. The properties of
the matched SAM galaxies supply the full complement of required properties
including LSST and SDSS rest-frame and observer-frame magnitudes, as well as
coarse-grained SEDs obtained from a set of top-hat filters spanning a
wavelength range from 100 nm to 2 $\mu$m. All of the magnitudes are available
separately for disk and bulge components, with and without host-galaxy
extinction corrections and with and without emission-line corrections. After
the match, we use additional empirical models, based on properties obtained
from the SAM galaxies (e.g., the bulge-to-total ratio), to provide values for
galaxy shapes, orientations and sizes.

In general, the observational data sets that are used to tune the galaxy
properties in the catalog are different from those used in the validation
tests. The exceptions are the PRIMUS data for the stellar mass function
(Moustakas et al., 2013) and the SDSS data for the two point correlation
functions (Zehavi et al., 2011), which were used to tune the UniverseMachine
empirical model.

The weak-lensing quantities in the catalog are constructed from the particle-
lightcone data from the Outer Rim simulation by projecting particles onto a
series of mass sheets (lens planes) and performing a full ray-tracing
calculation to produce maps of the weak-lensing distortions. The surface
densities on the mass sheets are computed on a
HEALPix121212https://sourceforge.net/projects/healpix/ (Górski et al., 2005)
grid with Nside=4096. The mass sheets have a median width of approximately
$114~{}\rm{Mpc}$. Ray-tracing follows photon paths backwards in time from an
observer grid to a source plane and adding deflections that depend on the
surface density of particles at each lens plane between the observer and the
source. The shears and convergence for each galaxy are obtained by first
shifting the galaxy to its observed position and then interpolating the source
map to that position.

###  3.2 The Validation Framework: DESCQA

The DESCQA validation framework (Mao et al., 2018) provides an efficient,
flexible and convenient infrastructure that allows developers to produce and
run a wide range of validation tests.131313https://github.com/LSSTDESC/descqa
[github.com] The streamlined workflow enables non-expert users to run the
available tests very simply. The framework is composed of three principal
parts: (1) a catalog reader interface, (2) a validation test interface and (3)
a web interface. A main script141414This script was originally called the
master script in Mao et al. (2018) runs the selected catalogs and tests and
outputs the test results. These results are viewed on a dedicated web page
that is generated by the web-interface portion of the framework. Each test
typically produces comparison plots of the catalog and validation data and
delivers a status and an optional score that is the result of applying a test-
specific metric comparing the catalog and validation data.

The catalog reader interface employs the Python package
GCRCatalogs151515https://github.com/LSSTDESC/gcr-catalogs [github.com] to
supply a unified interface for ingesting a diverse range of catalogs.
GCRCatalogs in turn uses the Generic Catalog
Reader161616https://github.com/yymao/generic-catalog-reader [github.com] (GCR)
base class to provide many convenient features for filtering the data, which
renders it especially suited for use in the validation test suite.

The validation test interface contains a class (the BaseValidationClass) which
provides a unified infrastructure for constructing a test. Test developers
need only to write a module that runs the desired test on an arbitrary catalog
and provide an input configuration file that contains any required auxiliary
information. The latter can include paths to the validation data, selection
cuts required to match the catalog to the validation data and plotting and
processing options that must be passed to the test.

##  4 Validation Test Results for cosmoDC2

We now provide a concrete use case of applying our validation test suite to
assess a synthetic catalog. In this section, we present the results of running
the tests summarised in Table 1 on the cosmoDC2 extragalactic catalog. For
each test, we describe the implementation details and discuss the validation
criteria set by the relevant LSST DESC WGs.

###  4.1 Readiness Tests

The readiness tests provide a range of general checks that enable a very
simple first quality check of the catalog data. These quality checks can be
easily summarized and include various statistics evaluated on the
distributions that are included in the test. Statistics include minimum and
maximum values, means and standard deviations, fractions of zero values,
fractions of infinite values etc., as well as warnings for outlying values
that exceed the expected ranges. Other checks are implemented to check if data
columns are repeated, that all galaxy_ids are unique, that the halo_ids of
central galaxies are unique and that the expected relationships between galaxy
properties are found. For example, the relationship between the galaxy
ellipticities and the lengths of the major and minor axes and between the
shears, magnification and convergence are tested. The configuration file for
the tests allows the user to choose which quantities are checked and to set
values for the expected ranges in which the statistics for each distribution
should lie. In order to illustrate the capabilities of these tests, in
Appendix A we show selected screen shots from the DESCQA web interface for the
test outputs produced for the cosmoDC2 catalog. Problems with the catalog data
are readily identified by visual inspection of the test outputs.

###  4.2 Cumulative Galaxy Number Density

In Figure 1, we compare the cosmoDC2 galaxy number counts with the HSC
extrapolations described in Section 2.1.2. The top panel shows the cumulative
number counts per square degree for galaxies with $r$-band magnitude less than
the value of the abscissa, as a function of that value. The validation suite
contains similar tests for $g$-, $i$-, $z$\- and $y$-band magnitudes and the
results for all of these tests are comparable to the one shown here.

The grey-shaded band around the HSC extrapolated curve shows the region
corresponding to a $\pm 40\%$ fractional difference in the number counts. The
validation criteria set by the LSST DESC WL and LSS groups required that the
number counts for cosmoDC2 lie within this grey band in the magnitude range
$24<r<27.5$, which is indicated by the vertical shade band in the figure. This
loose validation criterion is designed to check that the catalog has
reasonably realistic cumulative number counts at the expected depths of the
LSST WL and LSS data samples. The number counts of galaxies fainter than
$r=27.5$ are not critical for the scientific analyses planned for cosmoDC2,
since such galaxies are generally below the detection limits of the LSST
survey. We note, however, that there is a limitation in the comparison between
the HSC extrapolation and the cosmoDC2 data. The HSC galaxy detections come
from measurements of images for which blending can impact the number of
detections and their fluxes (because deblended objects can potentially add to
or fall out of the sample), so both the shape and magnitude of the
extrapolation could be affected by undetected blends. The loose validation
criteria for this test partially compensate for this additional uncertainty.

In the validation region, the maximum fractional difference between the
catalog data and HSC extrapolation is determined and the catalog passes the
test if this difference is below the specified tolerance (0.4 in our case).
The maximum fractional differences for cosmoDC2 for $g$-, $r$\- $i$-, $z$\-
and $y$-bands are 0.49, 0.34, 0.17, 0.10, and 0.27, respectively. We note that
cosmoDC2 passes this important validation test in every band except $g$-band,
where the passing criterion is only marginally exceeded due to a small deficit
in the number density above $g=27$. We do not expect that this minor deficit
will significantly affect any of our verification tests.

In Figure 1, the decrease in the cosmoDC2 number counts at fainter magnitudes
is an effect of the finite mass resolution of the underlying N-body
simulation. Very faint galaxies reside in halos whose mass is too small to be
resolved in the simulation. In order for cosmoDC2 to pass this test, it was
necessary to add synthetic ultra-faint galaxies to the catalog to boost the
number counts for magnitudes $\gtrsim 27$ as described in Korytov et al.
(2019). The required number of such galaxies was estimated by using a low-mass
power-law extrapolation of the halo mass function to determine the number of
missing halos and then populating each of these with a single galaxy. Without
these additional galaxies, cosmoDC2 would have fallen well below the
validation criteria shown in Figure 1. This test provides an example of why it
is critical that validation testing is an integral part of the catalog
production process. In this case, it was necessary to alter the production
methodology so that the science goals for the catalog could be realized.

Figure 1: Top panel: Observed cumulative $r$-band number counts per square
degree as a function of magnitude from cosmoDC2 (blue) and extrapolated from
the HSC survey (black) (see text for more details). The grey shaded band shows
a $\pm 40\%$ uncertainty around the HSC extrapolation. The vertical shaded
region shows the magnitude range within which the two curves are compared.
Bottom panel: Relative difference between the two curves in the top panel.

###  4.3 Redshift Distributions

In Figure 2, we show the comparison between a $\sim$60 deg2 patch of cosmoDC2
and the DEEP2 data fits from Coil et al. (2004) as described in Section 2.1.2
for eight magnitude bins in the range $18<r<27$. These fits to the DEEP2 data
are valid for $z\lesssim 1.5$. Our test suite includes several similar tests
for different combinations of magnitude cuts on both $r$\- and $i$-band
magnitudes. (A similar test for magnitude-limited galaxy samples with
$r<22-24$ is shown in Figure 13 of Korytov et al. 2019). We note that we have
imposed the magnitude selection cuts on the LSST filter magnitudes in the
catalog data, with no additional corrections to account for the differences
between LSST and CFHT filters.

Figure 2: Redshift distribution of cosmoDC2 galaxies compared with fits to
DEEP2 data for a redshift range of $0<z<1.5$ for eight magnitude bins. The
selection cuts are on LSST $r$-band magnitudes with values ranging from 18 to
27 as indicated in the legends. The fits to the DEEP2 data are valid for
$z\lesssim 1.5$.

A $\chi^{2}$ test between the catalog and validation data is a convenient way
to quantify the level of agreement between the two data sets. A summary score
for the test is computed by calculating the average of the reduced $\chi^{2}$
between the catalog data and observed fits for each of the magnitude-limited
samples. Some care is required in implementing this test due to the smallness
of the statistical errors in the binned catalog data. These statistical errors
do not properly account for the errors due to cosmic variance, particularly
for catalogs covering only small sky areas. Instead, we implement a jackknife
procedure where we estimate the error, $\sigma_{i}$, on the $i$-th redshift
bin by excluding regions of RA and Dec from the catalog data using a
k-means171717https://scikit-learn.org/stable/ algorithm. The elements of the
covariance matrix are then given by

$\sigma^{2}_{ij}=\frac{N_{\rm{jack}}-1}{N_{\rm{jack}}}\sum_{k}(\bar{N}_{i}-N^{k}_{i})\cdot(\bar{N}_{j}-N^{k}_{j}),$
(5)

where $N_{\rm{jack}}$ denotes the number of jackknife regions and
$\bar{N}_{i}$ and $N^{k}_{i}$ denote the number of galaxies in the $i$-th
redshift bin for the full sample and for the sample with the $k$-th region
excluded, respectively. The computation of covariances is lengthy so this test
is faster if run over smaller sky areas (e.g., $\sim$100 deg2). For these
smaller areas, the above covariance matrix is often not invertible due to
instabilities in the off-diagonal matrix elements, so we use only the diagonal
elements in the $\chi^{2}$ computation. Consequently, the $\chi^{2}$ may be
underestimated.

Except for the brightest galaxy samples ($r<21$) at high redshift, the
cosmoDC2 redshift distributions are in reasonable agreement with the DEEP2
fits, and we obtain $\chi^{2}/\rm{d.o.f}$ values of 79210, 8085, 13.3, .53,
.62, .93, .96 and 2.2 for $r$-band magnitude ranges between 18 and 20 - 27,
respectively. The large $\chi^{2}$ values for the brightest galaxies shown in
the upper two panels of Figure 2 are due to a deficit of these galaxies at
redshifts $z\gtrsim 0.6$. The validation criteria for cosmoDC2 did not specify
a quantitative limit for $\chi^{2}/\rm{d.o.f}$, so redshift distributions that
are broadly in agreement with the observational data are considered
acceptable. For cosmoDC2, all but the brighter galaxies with $r<21$ and
redshifts in the range $0.6\lesssim z\lesssim 1$ approximately follow the
expected distributions.

###  4.4 Color Distributions

####  4.4.1 1-d Color Distributions

The 1-d color-distribution test in our suite is similar to the color test
described in Mao et al. (2018), but has been adapted for use on a lightcone
catalog. For cosmoDC2, we select galaxies within a specified redshift range
and having apparent magnitudes brighter than a specified limit, and compare
the resulting color distributions to those of observed galaxies, selected
using the same cuts. It should be noted that although we only compare the
distributions of colors, the magnitude limits on the catalog samples implies
that it is only possible to pass the test if the magnitude distributions are
also sufficiently realistic. In Figure 3, we show an example of the color test
for high-redshift galaxies. For this example, cosmoDC2 galaxies have $r$-band
magnitudes $<24$ and redshifts in the range $0.85<z<0.95$ and are compared
with DEEP2 data. The suite contains a similar test comparing synthetic
galaxies having $r$-band magnitudes $<17.7$ and redshifts in the range
$0.05<z<0.10$ with SDSS data. The results for this test for the cosmoDC2
catalog are shown in Figure 14 in Korytov et al. (2019).

Figure 3: Color distributions of cosmoDC2 galaxies compared with those of
DEEP2 data for $u-g$, $g-r$, $r-i$ and $i-z$ colors. The cosmoDC2 galaxies
comprise a magnitude- and redshift-limited sample with $r$-band magnitude
$<24$ and a redshift range of $0.85<z<0.95$. The SDSS filter magnitudes in
cosmoDC2 have been corrected to compensate for the differences with the DEEP2
CFHT filters.

As described in Mao et al. (2018), we obtain a quantitative estimate of the
level of difference between the validation data and the synthetic color
distributions, by calculating the Cramér-von Mises (CvM) statistic, $\omega$
(Anderson, 1962). This statistic provides a non-parametric method to test if
two data samples are drawn from the same underlying distribution by measuring
the average $L^{2}$ distance between the two cumulative distribution
functions. The values of $\omega$ in Figure 3 are 0.28, 0.0425, 0.15 and 0.229
for $u-g$, $g-r$, $r-i$ and $i-z$ color distributions, respectively. Lower
values for this statistic correspond to better agreement between the
distributions and values of $\omega\lesssim 0.05$ indicate that the two
distributions are in very good agreement. Only the $g-r$ distribution
satisfies this stringent criterion.

For DC2, no quantitative requirement has been given for $\omega$. As can be
seen from Figure 3, the cosmoDC2 color distributions, except for $g-r$, do not
agree well with the validation data. There are notable differences,
particularly in the numbers of redder galaxies. As discussed in Korytov et al.
(2019), in order to facilitate the study of cluster systematic effects, the
empirical color model for cosmoDC2 has been tuned to enhance the efficiency
with which the redMaPPer algorithm is able to identify synthetic clusters.
This tuning results in an enhancement, particularly evident in the $r-i$ color
distribution, of the red fraction in cosmoDC2.

####  4.4.2 Color–Color Distributions

We have expanded the test suite to include 2-dimensional distributions as
well. It is well known that, unlike the CvM statistic, there is not a well-
defined metric to compare multivariate datasets in a unique way. In this case,
we have instead used the so-called Wasserstein metric (Dobrushin, 1970, in
Russian) measured across several random directions in color-color space, where
the distributions are projected. These different measurements are averaged and
provided as additional outputs for our color-color plot comparisons, which
include a heatmap versus contour plot visualization as well. We implemented a
kernel two sample test from Gretton et al. (2012) as an additional cross-
check.

In Figure 4, we show the two dimensional distribution of $g-r$ and $r-i$
cosmoDC2 colors compared with that of the DEEP2 data for the redshift range
$0.1<z<0.3$. The“Compare" metric refers to the averaged Wasserstein distances,
which in this case indicate a non-zero bias is present in this color space.
The kernel comparison MMD (Maximum Mean Discrepancy) test also shows that the
difference between the two distributions is significant (p value is 0.001 for
both coming from the same underlying true distribution).

Figure 4: The 2 dimensional distribution of $g-r$ (y-axis) and $r-i$ (x-axis)
cosmoDC2 colors compared with those of the DEEP2 data for the redshift range
$0.1<z<0.3$. The color bar and the black contours shows the scales for the
densities of cosmoDC2 and DEEP2 galaxies, respectively. The black contours are
shown at density levels of 0.3, 3.0, 30.0 and 100.0. The results for the
comparison metrics discussed in the text are given in the legend.

###  4.5 Position Angle Distribution

In Figure 5, we show the galaxy position-angle distribution for 60 deg2 of
cosmoDC2. The uniformity of the distribution can be verified by visual
inspection. This test also provides a more quantitative measure of the
uniformity by using a statistic based on the two-sample Kolmogorov–Smirnov
(K–S) test, as implemented in NumPy (van der Walt et al., 2011), to compare
the galaxy position-angle distribution to a uniform distribution. The default
requirement on the $p$-value of the test statistic was chosen to be $p\geq
0.05$. However, as we discuss below, the $p$-value is not a particularly
reliable or robust statistic, so we prefer to use visual inspection of the
distribution to verify its uniformity.

Historically, this test was developed for a small catalog for which the
$p$-value statistic provided a reasonably robust measure of uniformity.
However, for the large sample sizes that are obtained with catalogs such as
cosmoDC2, the $p$-value statistic is sensitive to the sample size and we find
that its value decreases with increasing sample size. When running this test
on larger catalogs it is necessary to down-sample the distribution uniformly
to remain in a sample-size regime where the $p$-values are reasonably
consistent. We tested a variety of catalog sizes and down-sampling factors to
determine that the size $N_{\theta}$ of the position-angle array used in the
K–S test should satisfy $N_{\theta}\lesssim 5\times 10^{6}$.

Figure 5: The distribution of position angles for cosmoDC2. The $p$-value
statistic is obtained by using the two-sample K–S test to compare a uniformly
down-sampled distribution of catalog position angles to a uniform
distribution. The validation criterion requires that $p>0.05$.

###  4.6 Size Distribution

The modeling of galaxy sizes in cosmoDC2 is described at length in Korytov et
al. (2019). Here, we summarize the main ingredients. The size of the disk and
bulge component of each galaxy is obtained by using the fits to the mean
relation for size versus SDSS absolute $r$-band magnitude presented in Zhang &
Yang (2019) for $B/T<0.5$ and $B/T>0.5$, respectively. A scatter about the
mean relation of 0.2 dex and 0.15 dex (Somerville et al., 2018) for the disk
and bulge components, respectively is introduced in the size assignments.
Redshift dependence is included by imposing a smooth decrease in size with
redshift such that mean sizes of both disks and bulges are reduced by a factor
of two at $z=1$. In cosmoDC2, an approximate prescription given by the
luminosity-weighted sum of the disk- and bulge-component sizes is used to
estimate the total size of each galaxy and serves as a proxy for other methods
of determining size. Since the galaxy sizes in cosmoDC2 are derived from
empirical data, the tests described in this section and Section 4.7 therefore
constitute a check on the cosmoDC2 modeling procedure and verify that the mean
relationships used are consistent with those derived from other data sets and
that the redshift dependence and the scatter are reasonable.

For the size-distribution test presented here, a comparison with the available
validation data requires an estimate of the half-light radius, $R_{\rm eff}$,
which is the size parameter characterizing the Sérsic (single-component)
profile of the galaxy. Since this parameter is not available in cosmoDC2, we
use instead the approximation for $R_{\rm eff}$ described above, but note that
it does not give the same result as fitting a single Sersic profile to the sum
of the disk and bulge profiles.

In Figure 6, we show the size distribution for cosmoDC2 galaxies (left panel)
and a comparison of the sizes for cosmoDC2 and COSMOS galaxies (right panel).
Sizes are measured in units of arcseconds. To mimic the COSMOS data selection,
we apply an $i<25.2$ magnitude cut to the cosmoDC2 galaxies. We measure the
cumulative distribution function (CDF) of these galaxy sizes in thin bins of
$R_{\rm eff}$, with a variable cutoff to match the validation data set, and
use the slope of the CDF as our comparison statistic.

In order to ensure we are probing the the small-size slope only, we discard
all galaxies with a size greater than the median size. We then bin both data
sets in optimally spaced size bins and measure the number of cosmoDC2 and
COSMOS galaxies in each bin. We fit a line to both $dN/dx$ measurements and
compare the slopes of the best-fit line. This linear description is not very
accurate, so we require simply that the two slopes agree within a factor of 2.
This criterion is limited by the quality of the metric more than the quality
of the data or the simulations. Since our sensitivity to this slope is low,
and the fidelity of galaxy sizes is also checked by the size-luminosity
relation (see Section 4.7), this loose criterion is sufficient for DC2
analyses.

Figure 6: True (unlensed) size distribution of small cosmoDC2 galaxies (left
panel) and the comparison of cosmoDC2 galaxy sizes (blue) with a similar
sample of COSMOS galaxy sizes (grey) (right panel). The sizes of cosmoDC2
galaxies are estimated by computing the luminosity-weighted sums of the disk
and bulge component sizes. Sizes are given in arcseconds. CosmoDC2 galaxies
are selected such that their $i$-band magnitude $<25.2$ and small galaxies are
defined such that their size is smaller than the median size of the sample. In
the right panel, the values of the slopes for linear fits to $dN/dx$ for
cosmoDC2 and COSMOS data are given in the legend. The validation criterion for
this test is that the slopes agree to within a factor of 2.

###  4.7 Size–Luminosity Relations

For these tests we compare the observed size-luminosity relations as described
in Section 2.2.2, which have been derived from the total light profiles
(single components) for 3D-HST and CANDELS data and the disk and bulge light
profiles (two components) for COSMOS data. For the simulated catalog samples,
we mimic the redshift selections in the observed data. We also convert the
absolute magnitudes of galaxies to luminosities as described in detail below.
As discussed in Section 4.6, the total size of a cosmoDC2 galaxy is obtained
by computing the luminosity-weighted sum of the disk and bulge components. We
remind the reader that this is an approximate prescription for estimating
galaxy size and does not give the same result as fitting a single Sersic
profile to the sum of the disk and bulge profiles.

####  4.7.1 Total Light Profile

For the comparison of sizes based on the total light profile, we use $M_{V}$,
the cosmoDC2 absolute $V$-band magnitudes to obtain luminosities using the
relation $\log_{10}(L/L_{\odot})=(4.80-M_{V})/2.5$ (Willmer, 2018, Table
3)1818184.80 is the absolute AB magnitude of the sun measured in $V$-band. In
Figure 7, we show the size-luminosity relation for cosmoDC2 (orange points)
and the mean relation for the 3D-HST + CANDELS data (blue lines), with the
$1-\sigma$ population uncertainties in the validation data depicted by blue
shaded bands. The catalog data agree with the mean validation data within the
uncertainties.

Figure 7: The size-luminosity relation for cosmoDC2 galaxies (orange points)
compared to 3D-HST + CANDELS data (blue solid lines) for five redshift ranges
as indicated in each panel. The shaded bands show the $1-\sigma$ population
uncertainties in the data.

####  4.7.2 Disk and Bulge Light Profiles

For size estimates based on the disk and bulge components of the light
profile, we use the cosmoDC2 LSST absolute $i$-band magnitudes to obtain
luminosities using the relation $\log_{10}(L/L_{\odot})=(4.52-M_{i})/2.5$
(Willmer, 2018, Table 3). Here, we have assumed that LSST $i$-band and COSMOS
F814W are approximately equivalent. In Figure 8 we show the disk and bulge
data in blue and red, respectively. The cosmoDC2 and the COSMOS data are shown
as points and lines, respectively. The $1-\sigma$ population scatter in the
validation data is depicted by shaded bands. Overall, the catalog data are in
good agreement with the mean relationship for the validation data, with some
deviations for bright galaxies at redshifts $z>1$. The deviations are within
the measured uncertainites, which unfortunately, are quite large.

Figure 8: The size-luminosity relation for the bulge (blue) and disk (red)
components of cosmoDC2 galaxies (points) compared to COSMOS data (solid lines)
for four redshift bins as indicated in each panel. The shaded bands show the
$1-\sigma$ population scatter in the COSMOS data. We convert the absolute
$i$-band magnitudes for the catalog data to units of solar luminosities as
described in the text.

###  4.8 Ellipticity Distribution

We compare the ellipticity distributions of cosmoDC2 galaxies with data from
the COSMOS survey (Joachimi et al., 2013). In Figure 9 we show distributions
for the four morphology selections and $V$-band magnitude ranges given in
Joachimi et al. (2013). The morphologies of the cosmoDC2 galaxies are matched
to those of the COSMOS data by selecting subsamples with $i$-band bulge-to-
total ratios ($B/T$) given by $0.7<B/T<1.0$, $0.0<B/T<0.2$ and $0.4<B/T<0.7$
for LRG, early, disk and late types, respectively. The magnitudes of the
ellipticities for cosmoDC2 galaxies are defined as the $r$-band luminosity-
weighted average of the disk and bulge ellipticities. This prescription for
estimating the ellipticities provides an approximation for measuring the
ellipticity from the shape of the total light profile of the galaxy. The disk
and bulge ellipticities in cosmoDC2 are drawn from distributions that have
been tuned to match the COSMOS data as described in Korytov et al. (2019).
Hence this test amounts to a consistency check that the assignment is
consistent with the expected distributions. From Figure 9, we see that all the
distributions agree well except for early type galaxies. This is probably due
to the fact that our selection on $B/T$ is too crude to match the more
complicated morphological selections for early type galaxies in the COSMOS
data.

Figure 9: A comparison of the ellipticity distributions for magnitude-limited
samples of cosmoDC2 galaxies and COSMOS galaxies. Since the COSMOS data are
used to assign ellipticities to cosmoDC2 galaxies, this test is a consistency
check that the assigned values match the intended distributions.

###  4.9 Galaxy-Galaxy Angular Correlation Function

In this test we use the
TreeCorr191919https://rmjarvis.github.io/TreeCorr/_build/html/index.html
package (Jarvis, 2015) to compute the two-point angular correlation function,
$w(\theta)$, the over-abundance of galaxy pairs at some angular separation
$\theta$ relative to a random distribution. We apply selection cuts to the
catalog sample to mimic measurements from Wang et al. (2013) of $w(\theta)$ in
bins of SDSS $r$-band magnitudes in the range $17<r<21$. The computation of
$w(\theta)$ uses the estimator of Landy & Szalay (1993). This test has been
described previously in Korytov et al. (2019), so we do not reproduce it here.
The results for the test showed that the amplitudes and slopes of the
clustering signals for the catalog data were in reasonable agreement with the
validation data, except at very small angular scales ($\theta\lesssim
0.4^{\circ}$), where the catalog clustering signal was slightly steeper than
the SDSS data.

###  4.10 Galaxy Bias

For this test, we divide the catalog galaxy sample into tomographic redshift
bins in the range $0.2<z<1.2$ and select galaxies whose observed $i$-band
magnitudes satisfy $i<25.3$, which is the expected $i$-band limiting magnitude
for the 10-year LSST gold sample (The LSST Dark Energy Science Collaboration,
2018b). For each tomographic bin, the redshift distribution is determined and
the angular correlation function is computed using TreeCorr as in Section 4.9.
Next, the values of the cosmological parameters used in the simulated catalog
are input to CCL (Chisari et al., 2019) to compute the theoretical prediction
for the angular correlation function. A linear bias model is independently fit
in each tomographic bin by minimizing $\chi^{2}_{b}$, which is defined as:

$\chi^{2}_{b}=\sum_{i=1,N_{\text{bins}}}\frac{\left(w_{\text{data}}(\theta_{i})-b\,w_{\text{th}}(\theta_{i})\right)^{2}}{\left(\Delta
w(\theta_{i})\right)^{2}}.$ (6)

Here $\Delta w(\theta)$ are the error bars reported by TreeCorr (using 80
jackknife patches) and $N_{\text{bins}}$ is the number of angular bins
considered in each tomographic bin. We note that in Equation 6, we are