The MegaMapper September 7, 2022

Submitted to the Proceedings of the US Community Study

on the Future of Particle Physics (Snowmass 2021)

The MegaMapper: A Stage-5 Spectroscopic Instrument Concept for the Study of
Inflation and Dark Energy

Principal Authors
David J. Schlegel1, Juna A. Kollmeier 2,32

Team Members

Greg Aldering1, Stephen Bailey1, Charles Baltay31, Christopher Bebek1, Segev
BenZvi30, Robert Besuner1, Guillermo Blanc2, Adam S. Bolton20, Ana Bonaca2,
Mohamed Bouri4, David Brooks24, Elizabeth Buckley-Geer29, Zheng Cai5, Jeffrey
Crane2, Regina Demina30, Joseph DeRose1, Arjun Dey20, Peter Doel24, Xiaohui
Fan6, Simone Ferraro1, Douglas Finkbeiner35, Andreu Font-Ribera24, Satya
Gontcho A Gontcho1, Daniel Green36, Gaston Gutierrez29, Julien Guy1, Henry
Heetderks3, Dragan Huterer11, Leopoldo Infante2, Patrick Jelinsky3, Dionysios
Karagiannis34, Stephen M. Kent29, Alex G. Kim1, Jean-Paul Kneib8, Anthony
Kremin1, Luzius Kronig8, Nick Konidaris2, Ofer Lahav24, Michael L. Lampton3,
Martin Landriau1, Dustin Lang16, Alexie Leauthaud22, Michael E. Levi1, Michele
Liguori7, Eric V. Linder3, Christophe Magneville9, Paul Martini19, Mario
Mateo11, Patrick McDonald1, Christopher J. Miller11, John Moustakas27, Adam D.
Myers21, John Mulchaey2, Jeffrey A. Newman14, Peter E. Nugent1, Nikhil
Padmanabhan31, Nathalie Palanque-Delabrouille9, Antonella Palmese${12,1}$,
Anthony L. Piro2, Claire Poppett3, Jason X. Prochaska22, Anthony R. Pullen18,
David Rabinowitz31, Anand Raichoor1, Solange Ramirez2, Hans-Walter Rix10,
Ashley J. Ross19, Lado Samushia28, Emmanuel Schaan1, Michael Schubnell11, Uros
Seljak1,12,13, Hee-Jong Seo17, Stephen A. Shectman2, Edward F. Schlafly33,
Joseph Silber1, Joshua D. Simon2, Zachary Slepian23, Anže Slosar37, Marcelle
Soares-Santos15, Greg Tarlé11, Ian Thompson2, Monica Valluri11, Risa H.
Wechsler25,26, Martin White1,13, Michael J. Wilson1, Christophe Yèche9, Dennis
Zaritsky6, Rongpu Zhou1

1 Lawrence Berkeley National Laboratory, Berkeley, CA 94720
2 The Observatories of the Carnegie Institution for Science, 813 Santa Barbara
St., Pasadena, CA 91101, USA
3 Space Sciences Laboratory, University of California Berkeley, Berkeley, CA
94720, USA
4 STI IMT, École Polytechnique Fédérale de Lausanne (EPFL), 1015 Lausanne,
Switzerland
5 Department of Physics and Tsinghua Center for Astrophysics, Tsinghua
University, Beijing 100084, P R China
6 Department of Astronomy/Steward Observatory, University of Arizona, Tucson,
AZ 85721
7 Dipartimento di Fisica e Astronomia “G. Galilei”,Università degli Studi di
Padova, via Marzolo 8, I-35131, Padova, Italy
8 Institute of Physics, Laboratory of Astrophysics, Ecole Polytechnique
Fédérale de Lausanne (EPFL), Observatoire de Sauverny, 1290 Versoix,
Switzerland
9 IRFU, CEA, Université Paris-Saclay, F-91191 Gif-sur-Yvette, France
10 Max Planck Institut für Astronomie, Königstuhl 17, D–69117 Heidelberg,
Germany
11 University of Michigan, Ann Arbor, MI 48109
12 Department of Physics, University of California Berkeley, Berkeley, CA
94720, USA
13 Department of Astronomy, University of California Berkeley, Berkeley, CA
94720, USA
14 University of Pittsburgh and PITT PACC, Pittsburgh, PA 15260
15 Department of Physics, Brandeis University, Waltham, MA 02453
16 Perimeter Institute, Waterloo, Ontario N2L 2Y5, Canada
17 Department of Physics and Astronomy, Ohio University, Clippinger Labs,
Athens, OH 45701, USA
18 New York University, New York, NY 10003
19 The Ohio State University, Columbus, OH 43212
20 NSF’s NOIRLab, 950 N. Cherry Ave., Tucson, AZ 85719 USA
21 Department of Physics and Astronomy, University of Wyoming, Laramie, WY
82071, USA
22 University of California at Santa Cruz, Santa Cruz, CA 95064
23 University of Florida, Gainesville, FL 32611
24 University College London, WC1E 6BT London, United Kingdom
25 Stanford University, Stanford, CA 94305
26 SLAC National Accelerator Laboratory, Menlo Park, CA 94025
27 Siena College, Department of Physics & Astronomy, 515 Loudon Road,
Loudonville, NY 12211, USA
28 Kansas State University, Manhattan, KS 66506
29 Fermi National Accelerator Laboratory, Batavia, IL 60510
30 Department of Physics and Astronomy, University of Rochester, 500 Joseph C.
Wilson Boulevard, Rochester, NY 14627, USA
31 Department of Physics, Yale University, New Haven, CT 06520
32 Canadian Institute for Theoretical Astrophysics, 60 St. George St., Toronto
Canada
33 Space Telescope Science Institute, Baltimore, MD 21218
34 Department of Physics & Astronomy, University of the Western Cape, Cape
Town 7535, South Africa
35 Harvard-Smithsonian Center for Astrophysics, Cambridge, MA 02138
36 University of California San Diego, La Jolla, CA 92093
37 Brookhaven National Laboratory, Upton, NY 11973

> ABSTRACT
>
>
>
>
> In this white paper, we present the MegaMapper concept. The MegaMapper is a
> proposed ground-based experiment to measure Inflation parameters and Dark
> Energy from galaxy redshifts at $2<z<5$. In order to achieve path-breaking
> results with a mid-scale investment, the MegaMapper combines existing
> technologies for critical path elements and pushes innovative development in
> other design areas. To this aim, we envision a 6.5-m Magellan-like
> telescope, with a newly designed wide field, coupled with DESI
> spectrographs, and small-pitch robots to achieve multiplexing of 26,100.
> This will match the expected achievable target density in the redshift range
> of interest and provide a 15x capability over the existing state-of the art,
> without a 15x increase in project budget.

##  1 Introduction & Objectives

The mechanisms driving the accelerated expansion of the Universe in its very
first moments (Inflation) and at late times (Dark Energy) represent some of
the most important open problems in fundamental physics, and have been the
subject of several of the Snowmass Community Science White Papers [e.g., 1, 2,
3, 4, 5]. We refer readers to those papers, and the questions therein, which
have informed the development of the MegaMapper concept. Below, we sketch in
brief the concept itself and how this concept aims to address these open
problems with a singular platform. At root, the fundamental challenge for
future spectroscopic experiments is mapping speed (at fixed spectroscopic
fidelity). The MegaMapper concept combines a very wide field, a large primary
aperture, and a densely packed focal plane to solve the next order of
magnitude in the Mapping Speed progression. The main principles that guide the
development of the MegaMapper concept are that 1) mapping speed envisioned for
the next decade should aim for order-of-magnitude transformative change (see
Figure 1), and 2) the development of projects of this nature should endeavor
to remain “mid-scale” as long as practical in order to efficiently benefit
from technical innovation, reduce project risk, and control cost and schedule.

###  1.1 Cosmology and Fundamental Physics

Primordial non-Gaussianity [6] has been identified as one of the most powerful
tools to study Inflation and the origin of the primordial fluctuations.
Although the simplest inflationary models predict Gaussian initial conditions,
large classes of models predict deviations from Gaussianity that leave
specific imprints on the galaxy power spectrum and bispectrum [6, 7], thus
probing physics at scales inaccessible to Earth-based colliders.
Distinguishing multi-field from single-field inflation requires bounds on the
“local” non-Gaussianity parameter to be considerably better than
$\sigma(f_{NL}^{\rm local})\approx 1$, or about an order of magnitude
improvement over current constraints [8]. Moreover, a model independent
reconstruction of the spectrum of matter perturbations will significantly
improve our understanding of the primordial fluctuations, as well as any
oscillation or feature in the primordial power spectrum, often affected by the
physics of inflation [9, 10].

Our understanding of Dark Energy [11] will greatly benefit from measuring the
expansion and growth of fluctuations throughout cosmic history. A lever arm
that extends from low redshift (Dark Energy era) to high redshifts (matter
domination era) will tightly constrain large classes of theories, and test
possible modifications to General Relativity. Moreover, precision measurements
of the matter power spectrum will provide tight constraints on Early Dark
Energy models, providing a few percent measurement out to $z\sim 10^{5}$ [10,
5].

The large volume available at $z>2$ will increase the number of modes measured
by over an order of magnitude over current experiments, opening up an
uncharted territory with huge discovery potential. Over the next decade, the
Rubin Observatory’s Legacy Survey of Space and Time (LSST) [e.g. 12] will
begin to enable hundreds of millions of tracers to be targeted for the
MegaMapper.

The increased mapping speed by more than one order of magnitude, together with
targets selected from the LSST imaging, will allow us to observe a high-
redshift sample at $2\lesssim z\lesssim 5$. The large volume available over
this redshift range will allow the MegaMapper to explore primordial physics to
unprecedented precision, beyond the CMB cosmic variance limit [5]. Dramatic
improvements in the constraints on Inflation and Dark Energy [13, 10, 5] are
possible, while relaxing assumptions such as a power-law primordial power
spectrum [9, 14]. LSST imaging will enable the selection of $\approx 100$
million spectroscopic targets spanning this redshift range. These targets are
a combination of massive galaxies and highly star-forming systems, both of
which can be efficiently selected from broad-band photometric surveys [15].

The MegaMapper represents a cost-effective, low-risk version of such a survey,
capable of achieving the stated science goals using proven technologies.
Simultaneously, experience from the pioneering SDSS suite of surveys
demonstrates that such a facility would confer a tremendous opportunity for
ancillary science explorations that help answer a broad array of astronomical
questions from the formation of the Milky Way to the evolution and growth of
galaxies.

Possible sites for the MegaMapper include Las Campanas Observatory (Chile),
Cerro Tololo Observatory (Chile), San Pedro Martir (Mexico) and Kitt Peak
National Observatory (Arizona). The Chilean sites would have full overlap with
LSST imaging, whereas the northern sites would be limited to 15,000 sq. deg.
of suitable extragalactic overlap in the $-30<\delta<+30$ equatorial region.

Figure 1: Number of galaxy redshifts as a function of time for the largest
cosmology surveys. The dotted line represents an increase of survey size by a
factor of 10 every decade. Fielding the MegaMapper in ten years maintains this
pace into the 2030s, and enables the Inflation and Dark Energy measures
proposed in this and other white papers.

###  1.2 Cosmology Science Forecasts: Inflation and Dark Energy

In [13] we have identified two samples, a more optimistic “idealized” sample
based on the LSST target density of Lyman-$\alpha$ emitters and Lyman-Break
galaxies, and a “fiducial” sample, based on conservative redshift success
rates and assumptions about line strengths (see [13] for the sample
specifications and [15] for background about selection and sample properties).

We have found that both samples can cross the theoretical threshold
$\sigma(f_{NL}^{\rm local})\lesssim 1$, from measurement of the galaxy power
spectrum on large scales, surpassing the current CMB bounds by an order of
magnitude [13, 10]. Potentially large improvements are possible when including
the analysis of the bispectrum. Improvements by a factor of two or larger over
the current bounds are also expected for the equilateral and orthogonal shapes
[13]. Particular care should be taken in the telescope and survey design to
control systematics to allow such precise measurements of primordial non-
Gaussianity.

Using a combination of Redshift-Space Distortions (RSD) and Baryon Acoustic
Oscillations (BAO), the fraction of Dark Energy $\Omega_{DE}$ can be measured
to better than 1% all the way to $z\approx 4.5$ for the “idealized” sample,
and better than 2% up to $z\approx 5$ for the “fiducial” sample (see Figure 1
in [13]). Other notable improvements include a factor of two better
determination of the spatial Curvature (compared to DESI + Planck), and a
factor of $\gtrsim 2.5$ improvement in the Dark Energy figure of merit (Table
4 in [13]). Early Dark Energy (EDE) has been proposed as a solution to the
Hubble tension [16]. MegaMapper will be able to constrain the fraction of EDE
to better than $2\%$ all the way to $z\sim 10^{5}$ [10], when the Universe was
only a few years old, definitely testing this hypothesis, and more generally
providing percent-level expansion constraints throughout cosmic history.

Table 1: Survey speeds for multi-fiber spectrographs as measured by the product of the telescope clear aperture, number of fibers and losses from mirror reflections. This speed assumes a dedicated facility, which would not be possible in all cases. Keck/FOBOS[17], MSE[18], SpecTel[19] and MegaMapper[20] are proposed experiments. LSSTspec[21, 22] is a notional number using MegaMapper positioners on the LSST focal plane, if optical design limitations could be overcome injecting f/1.2 light into fibers.  Instrument (year) | Primary/m2 | Nfiber | Reflections | Product | Speed vs. | SDSS
---|---|---|---|---|---|---
SDSS (1999) | 3.68 | 640 | 0.92 | 1908 | 1. | 00
BOSS (2009) | 3.68 | 1000 | 0.92 | 2980 | 1. | 56
DESI (2020) | 9.5 | 5000 | 0.91 | 42,750 | 22. | 4
PFS (2023) | 50 | 2400 | 0.91 | 108,000 | 56. | 6
4MOST (2023) | 12 | 1624 | 0.92 | 15,800 | 8. | 3
DESI-Upgrade (2027) | 9.5 | 11,250 | 0.91 | 96,200 | 50. | 4
MegaMapper | 28 | 26,100 | 0.92 | 590,000 |  309. |
Keck/FOBOS | 77.9 | 1800 | 0.93 | 102,000 | 53. | 6
MSE | 78 | 3249 | 0.91 | 228,000 | 119. |
LSSTspec | 35.3 | 8640 | 0.93 | 222,000 | 116. |
SpecTel | 87.9 | 15,000 | 0.92 | 1,070,000 | 560. |
Table 2: Survey speeds as measured by the raw product of collecting area and field-of-view. This is the appropriate metric for a wide-area survey with sparse targets. Even without taking full advantage of multiplexing, the MegaMapper survey speed is competitive with larger telescopes owing to its large field-of-view. Instrument (year) | Primary/m2 | FOV/deg2 | Reflections | Product | Speed vs. | SDSS
---|---|---|---|---|---|---
SDSS (1999) | 3.68 | 7.06 | 0.92 | 21.0 | 1. | 00
BOSS (2009) | 3.68 | 7.06 | 0.92 | 21.0 | 1. | 00
DESI (2020) | 9.5 | 8.04 | 0.91 | 68.7 | 3. | 27
PFS (2023) | 50 | 1.33 | 0.91 | 59.9 | 2. | 85
4MOST (2023) | 12 | 4.90 | 0.92 | 58.8 | 2. | 80
MegaMapper | 28 | 7.06 | 0.92 | 160. |  7. | 62
Keck/FOBOS | 77.9 | 0.087 | 0.93 | 4.94 | 0. | 23
MSE | 78 | 1.52 | 0.91 | 107. | 5. | 10
LSSTspec | 35.3 | 9.60 | 0.93 | 247. | 11. | 76
SpecTel | 87.9 | 4.91 | 0.92 | 350. | 16. | 65

###  1.3 Additional Science Opportunities

The facility constructed in pursuit of the cosmology program described above
also serves a broad range of additional astrophysical and cosmological
objectives. Some of these would be addressed coincident with the 5-year
cosmology key project, while others could be pursued after completion of that
project by the broader community. Some of these science cases have been
articulated in the National Academy of Sciences “Astro 2020” report. We
summarize here the synergies of the MegaMapper with the recommendations of
Astro2020 here.

####  1.3.1 Maximizing the community investment in LSST

A wide-field spectroscopic survey will greatly enhance the LSST science
returns, as identified by several Astro 2020 science white papers [23, 24,
25]. Calibration of photometric redshifts is possible over the whole range of
LSST sources through cross-correlation techniques with the spectroscopic
sample. A large overlap in survey area will enable a reduction in the
statistical errors to meet the stringent LSST requirements [23]. The
availability of a large overlapping spectroscopic sample will allow cross
correlation of these galaxies with the faint LSST sources to better constrain
the Intrinsic Alignment effect in weak lensing measurements. Moreover, a
combination of lensing amplitude provided by LSST, together with growth
measurements through RSD can provide a powerful test of General Relativity on
cosmological scales. Finally, the MegaMapper would provide redshifts for
strong gravitational lenses and type Ia supernovae discovered in LSST,
allowing their cosmological interpretation. Wide-field spectroscopy is
therefore a keystone for fully extracting the information content from other
cosmological methodologies.

####  1.3.2 The Milky Way as a Dark Matter Experiment

The MegaMapper, with its tremendously capable focal plane, will be a key
complement to the astrophysical studies of Dark Matter [25, 26, 27]: by
measuring the velocity dispersion of faint Milky Way satellites, the mass and
density can be inferred and compared to theoretical predictions, in an
environment where baryon effects are expected to be minimal. Similarly, the
perturbations and gaps in the Milky Way stellar streams created by encounters
with Dark Matter substructure create a characteristic velocity signal that can
be measured with spectroscopy to determine the perturber mass [26]. Moreover,
the merger dynamics in galaxy clusters can be studied, providing constraints
on self-interacting Dark Matter [28]. In particular, the MegaMapper concept is
already inspiring cosmologists to think about new methodological frameworks
(beyond “streams” and “dwarf galaxies”) to consider more comprehensive dark
matter experiments that can be done in the Milky Way.

Additionally, of great interest in the modern era and particularly with the
successful launch of the James Webb Space Telescope, the MegaMapper can enable
the industrial, wide-field study of galaxy evolution to $z\sim 2$ and stellar
population and kinematics and dense tomography of the intergalactic medium
[29]. While it is beyond the scope of what is presented here, future
instrumentation on such a facility in the post-survey period could enable a
tremendous range of science from planet to galaxy formation and evolution. The
concept thus represents an enabling platform in the spirit of the “SDSS-DESI”
family of projects.

##  2 Technical Overview

The overall MegaMapper design seeks to optimize survey speed with a judicious
choice of telescope aperture and instrument multiplexing. A new set of optical
models achieves a speed (as measured by $A\cdot\Omega$) that is difficult to
match with either smaller or larger telescopes (see Table 1). The overall
telescope design is nearly identical to the existing Magellan 6.5-m
telescopes, although the preferred design modifies the central hole to be
larger to accommodate the wider corrected field of the new optical design. The
instrument can accommodate a multiplexing of 26,100 and would feed between 600
and 675 fibers to each DESI spectrograph. 16 of these spectrographs already
exist for the DESI and SDSS-V projects (see [30] and [31]), with another 23 to
28 spectrographs required for the full capability of the Stage-5 MegaMapper
instrument (See Figure 2).

Figure 2: Focal plane layout for 348 fiber robot rafts (with 26,100 robots) on
the MegaMapper focal plane. The triangular rafts offer mechanical stiffness,
and could be invdividually inserted from the backside of the circular mounting
plate. The six unpopulated locations at the edge of the focal plane are
reserved for guide/focus cameras.

Telescope:

The telescope concept is based on the highly successful Magellan telescope
design. This constitutes a lightweight, honeycomb structure, 6.5 m
borosilicate glass primary mirror built by the University of Arizona Mirror
Lab. A baseline design is considered that adopts the same optical prescription
as Magellan I and II (i.e., a f/1.25 paraboloid), equipped with a 2.4 m
hyperbolic secondary mirror ($\sim$70% larger in diameter than the current
f/11 Gregorian secondaries), and a 5-lens wide field corrector that provides a
3.0 deg diameter field-of-view on a Cassegrain focal plane fed at f/3.6 (see
Figure 4). An ADC is designed as part of the corrector. The first and largest
element in the corrector is 1.8 m in diameter, with the other lenses also
being meter sized. The large secondary and sky baffles imply a central
obscuration of $\sim$20% of the area of the primary mirror, significantly
larger than the $\sim 5$% obscuration in the current Magellans but still less
than either SDSS or DESI. This baseline design requires a larger central hole
in the primary mirror than the 1.3 m hole in Magellan I and II, either by
enlarging the central hole in an existing mirror or casting a new mirror with
a custom mold.

Figure 3: Rendering of the Magellan-style telescope with the secondary mirror
and 5-element corrector, pointed towards the horizon. The spectrographs are
parked on the base with a fiber run that is substantially shorter than the
51-meter fiber run for DESI.

The f/3.6 telecentric focal plane has a diameter of 1230 mm, which at a plate
scale of 0.113 mm/” corresponds to a 3.0 deg diameter FOV. Figure 5 presents
spot diagrams at different zenith angles for the telescope’s preliminary
optical design described above, with $<23$ $\mu$m rms radius ($\sim 0.4$” FWHM
on sky) across the full FOV, which has a maximum of 2.7% vignetting at the
field edge. At this platescale, a 107 $\mu$m optical fiber (identical to the
DESI fibers) subtends a diameter of 0.94” on the sky, which is near the
optimal fiber size for point sources or compact, high-redshift galaxies in the
sky-noise limit.

The telescope mount, enclosure, and auxiliary facilities could be identical to
the current Magellan design. However, particular sub-systems would be
redesigned and updated. Figure 3 shows the MegaMapper in the Magellan CAD.
Some subsystems would benefit from advances of technology since the Magellan 1
& 2 builds, in particular for the sensor and control systems.

Figure 4: The optical systems on the revised wide-field Magellan telescope to
be used for the MegaMapper survey. Design studies are underway on the
corrector elements indicating that there are numerous vendors ready and
capable of manufacturing each element of the system. This deceptively simple
diagram represents the largest objective technical risk, now largely retired
since the publication of Astro2020 (Smee, private communication). Figure 5:
Spot diagram for the telescope and corrector optics. This demonstrates the
excellent image quality achieved under the current optical design. (courtesy
S. Smee, R. Barkhauser) Figure 6: Prototype Trillium Mark II fiber positioner
under development for MegaMapper, capable of fast re-configuration of fibers
to micron-scale accuracy (courtesy J. Silber). Figure 7: Mechanical drawing of
a 75-fiber robot raft, where the MegaMapper focal plane would consist of 348
of these rafts. Internally, the raft is further subdivided into 3 logical and
mechanical groups of 25 fibers (courtesy J. Silber).

Focal Plane: The focal plane is physically large and accommodates 26,100 zonal
fiber positioners with a center-to-center pitch of 6.1 mm.

Each individual fiber positioner is composed of two precision, mechanical
gearmotors. Such a “theta-phi” motion is the basis of both the DESI and SDSS-V
positioner designs. These have the benefit of fast positioning time, high
accuracy, and maintaining optical telecentricity. DESI and SDSS-V have both
completed their focal planes (of 5000 single-fiber positioners, and 1000 tri-
fiber positioners respectively), demonstrating that such a system can be mass
produced, reliably controlled, positioned to an accuracy of $2$ micron, and
reconfigured in a time envelope of less than one minute.

MegaMapper represents an evolution of these focal planes with smaller
positioners and other design modifications based upon our experiences building
the DESI and SDSS-V focal planes to facilitate mass production, integration,
testing, and servicing. Several designs have been prototyped for positioners
with a center-to-center spacing of 6.2 mm, with the second generation of the
Trillium positioners shown in Figure 6. The focal plane will be divided into
348 identical 75-fiber rafts as shown in Figure 7, with 8 or 9 rafts feeding
each spectrograph. Each of these 75-fiber raft modules should be a complete
working miniature instrument, including robotics, fibers, support structure,
and electronics. Installation and servicing of these raft modules will not
require the heavy support equipment and critical lifts required for the
installation of the much larger DESI focal plane petals.

Spectrographs: The spectrographs would be identical to those successfully
built and tested for DESI and SDSS-V. These spectrographs went through
extensive design and trade studies, and are optimized to measure redshifts of
faint targets in the sky-noise limit. The performance of these spectrographs
has been shown to exceed their design goals in delivered optical quality and
throughput. We would choose to somewhat increase the number of fibers feeding
each spectrograph from 500 to 600 or 675 by decreasing the fiber spacing at
the spectrograph slit. This is supported by the as-delivered spot quality in
the as-built spectrographs.

Each spectrograph is fed by a pseudo-slit with 600-675 fibers, with dichroics
dividing collimated light into three cameras. Each camera has gratings, optics
and CCDs that are optimized for its wavelength range in 360–555, 555–656, and
656–980 nm channels. The spectral resolution runs from 2000 on the blue end
(at 360 nm) increasing to a resolution of 5500 on the red end (at 980 nm) in
order to work between bright sky lines. The as-built efficiencies are 70–90%
across the full optical range.

Figure 8: Optical model of the DESI and SDSS-V spectrographs, which are
identical systems aside from the CCD packages. As-built efficiencies are
70–90% across the full optical spectrum.

Data System: The MegaMapper data systems will be a continuation of the data
system as developed within the SDSS+DESI family of projects and operated on
the NERSC high-performance computing platform. The DESI data reduction from
the raw pixel level to fully-calibrated spectra and redshift-fitting is state-
of-the-art today, and will be maintained and updated at least through the DESI
key project from 2021-2026. Poisson-limited spectra for faint targets is
achieved through a combination of stability of the spectrographs, stability of
the PSF with theta-phi positioners, and a rigorous forward-modeling of the
spectral extraction and sky-subtraction [32].

##  3 Conclusions

The MegaMapper is designed to be a high-reward, low-risk, cost-effective
approach to achieving a Stage-5 Spectroscopic experiment and achieving survey
speed 15$\times$ faster than the current best-in-class Dark Energy
Spectroscopic Instrument (DESI). The 6.5-m primary mirror has 3$\times$ the
collecting area of DESI, yet is no more challenging in the optical design for
the focal plane and the spectrographs. Our approach is to harness the power of
efficient mid-scale facility construction (i.e., the Magellan telescopes) and
survey design and operation (i.e., the SDSS/DESI surveys). In combination with
LSST, CMB-S4, and other cosmological probes, MegaMapper promises to explore
the full range of cosmological and fundamental physics that remains essential
to our understanding of the universe.

## References

  * [1] A. Achúcarro, M. Biagetti, M. Braglia, G. Cabass, E. Castorina, R. Caldwell, X. Chen, W. Coulton, R. Flauger, J. Fumagalli, M. M. Ivanov, H. Lee, A. Maleknejad, P. D. Meerburg, A. Moradinezhad Dizgah, G. A. Palma, S. Renaux-Petel, G. L. Pimentel, B. Wallisch, B. D. Wandelt, L. T. Witkowski, and W. L. Kimmy Wu, “Inflation: Theory and Observations,” arXiv e-prints (Mar., 2022) arXiv:2203.08128, arXiv:2203.08128 [astro-ph.CO].
  * [2] M. A. Amin, F.-Y. Cyr-Racine, T. Eifler, R. Flauger, M. M. Ivanov, M. LoVerde, C. B. d. S. Nascimento, A. H. G. Peter, M. Vogelsberger, S. Watson, and R. Wechsler, “Snowmass2021 Theory Frontier White Paper: Data-Driven Cosmology,” arXiv e-prints (Mar., 2022) arXiv:2203.07946, arXiv:2203.07946 [astro-ph.CO].
  * [3] J. A. Blazek, D. Clowe, T. E. Collett, I. P. Dell’Antonio, M. Dickinson, L. Galbany, E. Gawiser, K. Heitmann, R. Hložek, M. Ishak, S. W. Jha, A. G. Kim, C. D. Leonard, A. von der Linden, M. Lochner, R. Mandelbaum, P. Melchior, J. Meyers, J. A. Newman, P. Nugent, S. Perlmutter, D. J. Perrefort, J. Sánchez, S. J. Schmidt, S. Singh, M. Sullivan, A. Verma, and R. Zhou, “Snowmass2021 Cosmic Frontier White Paper: Enabling Flagship Dark Energy Experiments to Reach their Full Potential,” arXiv e-prints (Apr., 2022) arXiv:2204.01992, arXiv:2204.01992 [astro-ph.CO].
  * [4] K. Dawson, A. Hearin, K. Heitmann, M. Ishak, J. Ulf Lange, M. White, and R. Zhou, “Snowmass2021 Cosmic Frontier White Paper: High Density Galaxy Clustering in the Regime of Cosmic Acceleration,” arXiv e-prints (Mar., 2022) arXiv:2203.07291, arXiv:2203.07291 [astro-ph.CO].
  * [5] S. Ferraro, N. Sailer, A. Slosar, and M. White, “Snowmass2021 Cosmic Frontier White Paper: Cosmology and Fundamental Physics from the three-dimensional Large Scale Structure,” arXiv:2203.07506 [astro-ph.CO].
  * [6] P. D. Meerburg et al., “Primordial Non-Gaussianity,” arXiv:1903.04409 [astro-ph.CO].
  * [7] M. Alvarez et al., “Testing Inflation with Large Scale Structure: Connecting Hopes with Reality,” arXiv:1412.4671 [astro-ph.CO].
  * [8] Planck Collaboration, Y. Akrami et al., “Planck 2018 results. IX. Constraints on primordial non-Gaussianity,” arXiv:1905.05697 [astro-ph.CO].
  * [9] A. Slosar et al., “Scratches from the Past: Inflationary Archaeology through Features in the Power Spectrum of Primordial Fluctuations,” arXiv:1903.09883 [astro-ph.CO].
  * [10] N. Sailer, E. Castorina, S. Ferraro, and M. White, “Cosmology at high redshift — a probe of fundamental physics,” JCAP 12 no. 12, (2021) 049, arXiv:2106.09713 [astro-ph.CO].
  * [11] A. Slosar et al., “Dark Energy and Modified Gravity,” arXiv:1903.12016 [astro-ph.CO].
  * [12] Ž. Ivezić, S. M. Kahn, J. A. Tyson, B. Abel, E. Acosta, R. Allsman, D. Alonso, Y. AlSayyad, S. F. Anderson, J. Andrew, J. R. P. Angel, G. Z. Angeli, R. Ansari, P. Antilogus, C. Araujo, R. Armstrong, K. T. Arndt, P. Astier, É. Aubourg, N. Auza, T. S. Axelrod, D. J. Bard, J. D. Barr, A. Barrau, J. G. Bartlett, A. E. Bauer, B. J. Bauman, S. Baumont, E. Bechtol, K. Bechtol, A. C. Becker, J. Becla, C. Beldica, S. Bellavia, F. B. Bianco, R. Biswas, G. Blanc, J. Blazek, R. D. Blandford, J. S. Bloom, J. Bogart, T. W. Bond, M. T. Booth, A. W. Borgland, K. Borne, J. F. Bosch, D. Boutigny, C. A. Brackett, A. Bradshaw, W. N. Brandt, M. E. Brown, J. S. Bullock, P. Burchat, D. L. Burke, G. Cagnoli, D. Calabrese, S. Callahan, A. L. Callen, J. L. Carlin, E. L. Carlson, S. Chandrasekharan, G. Charles-Emerson, S. Chesley, E. C. Cheu, H.-F. Chiang, J. Chiang, C. Chirino, D. Chow, D. R. Ciardi, C. F. Claver, J. Cohen-Tanugi, J. J. Cockrum, R. Coles, A. J. Connolly, K. H. Cook, A. Cooray, K. R. Covey, C. Cribbs, W. Cui, R. Cutri, P. N. Daly, S. F. Daniel, F. Daruich, G. Daubard, G. Daues, W. Dawson, F. Delgado, A. Dellapenna, R. de Peyster, M. de Val-Borro, S. W. Digel, P. Doherty, R. Dubois, G. P. Dubois-Felsmann, J. Durech, F. Economou, T. Eifler, M. Eracleous, B. L. Emmons, A. Fausti Neto, H. Ferguson, E. Figueroa, M. Fisher-Levine, W. Focke, M. D. Foss, J. Frank, M. D. Freemon, E. Gangler, E. Gawiser, J. C. Geary, P. Gee, M. Geha, C. J. B. Gessner, R. R. Gibson, D. K. Gilmore, T. Glanzman, W. Glick, T. Goldina, D. A. Goldstein, I. Goodenow, M. L. Graham, W. J. Gressler, P. Gris, L. P. Guy, A. Guyonnet, G. Haller, R. Harris, P. A. Hascall, J. Haupt, F. Hernandez, S. Herrmann, E. Hileman, J. Hoblitt, J. A. Hodgson, C. Hogan, J. D. Howard, D. Huang, M. E. Huffer, P. Ingraham, W. R. Innes, S. H. Jacoby, B. Jain, F. Jammes, M. J. Jee, T. Jenness, G. Jernigan, D. Jevremović, K. Johns, A. S. Johnson, M. W. G. Johnson, R. L. Jones, C. Juramy-Gilles, M. Jurić, J. S. Kalirai, N. J. Kallivayalil, B. Kalmbach, J. P. Kantor, P. Karst, M. M. Kasliwal, H. Kelly, R. Kessler, V. Kinnison, D. Kirkby, L. Knox, I. V. Kotov, V. L. Krabbendam, K. S. Krughoff, P. Kubánek, J. Kuczewski, S. Kulkarni, J. Ku, N. R. Kurita, C. S. Lage, R. Lambert, T. Lange, J. B. Langton, L. Le Guillou, D. Levine, M. Liang, K.-T. Lim, C. J. Lintott, K. E. Long, M. Lopez, P. J. Lotz, R. H. Lupton, N. B. Lust, L. A. MacArthur, A. Mahabal, R. Mandelbaum, T. W. Markiewicz, D. S. Marsh, P. J. Marshall, S. Marshall, M. May, R. McKercher, M. McQueen, J. Meyers, M. Migliore, M. Miller, D. J. Mills, C. Miraval, J. Moeyens, F. E. Moolekamp, D. G. Monet, M. Moniez, S. Monkewitz, C. Montgomery, C. B. Morrison, F. Mueller, G. P. Muller, F. Muñoz Arancibia, D. R. Neill, S. P. Newbry, J.-Y. Nief, A. Nomerotski, M. Nordby, P. O’Connor, J. Oliver, S. S. Olivier, K. Olsen, W. O’Mullane, S. Ortiz, S. Osier, R. E. Owen, R. Pain, P. E. Palecek, J. K. Parejko, J. B. Parsons, N. M. Pease, J. M. Peterson, J. R. Peterson, D. L. Petravick, M. E. Libby Petrick, C. E. Petry, F. Pierfederici, S. Pietrowicz, R. Pike, P. A. Pinto, R. Plante, S. Plate, J. P. Plutchak, P. A. Price, M. Prouza, V. Radeka, J. Rajagopal, A. P. Rasmussen, N. Regnault, K. A. Reil, D. J. Reiss, M. A. Reuter, S. T. Ridgway, V. J. Riot, S. Ritz, S. Robinson, W. Roby, A. Roodman, W. Rosing, C. Roucelle, M. R. Rumore, S. Russo, A. Saha, B. Sassolas, T. L. Schalk, P. Schellart, R. H. Schindler, S. Schmidt, D. P. Schneider, M. D. Schneider, W. Schoening, G. Schumacher, M. E. Schwamb, J. Sebag, B. Selvy, G. H. Sembroski, L. G. Seppala, A. Serio, E. Serrano, R. A. Shaw, I. Shipsey, J. Sick, N. Silvestri, C. T. Slater, J. A. Smith, R. C. Smith, S. Sobhani, C. Soldahl, L. Storrie-Lombardi, E. Stover, M. A. Strauss, R. A. Street, C. W. Stubbs, I. S. Sullivan, D. Sweeney, J. D. Swinbank, A. Szalay, P. Takacs, S. A. Tether, J. J. Thaler, J. G. Thayer, S. Thomas, A. J. Thornton, V. Thukral, J. Tice, D. E. Trilling, M. Turri, R. Van Berg, D. Vanden Berk, K. Vetter, F. Virieux, T. Vucina, W. Wahl, L. Walkowicz, B. Walsh, C. W. Walter, D. L. Wang, S.-Y. Wang, M. Warner, O. Wiecha, B. Willman, S. E. Winters, D. Wittman, S. C. Wolff, W. M. Wood-Vasey, X. Wu, B. Xin, P. Yoachim, and H. Zhan, “LSST: From Science Drivers to Reference Design and Anticipated Data Products,” Astrophysical Journal 873 no. 2, (Mar., 2019) 111, arXiv:0805.2366 [astro-ph].
  * [13] S. Ferraro et al., “Inflation and Dark Energy from spectroscopy at $z>2$,” arXiv:1903.09208 [astro-ph.CO].
  * [14] R. de Putter, E. V. Linder, and A. Mishra, “Inflationary Freedom and Cosmological Neutrino Constraints,” Phys. Rev. D89 no. 10, (2014) 103502, arXiv:1401.7022 [astro-ph.CO].
  * [15] M. J. Wilson and M. White, “Cosmology with dropout selection: Straw-man surveys and CMB lensing,” arXiv:1904.13378 [astro-ph.CO].
  * [16] E. Abdalla et al., “Cosmology intertwined: A review of the particle physics, astrophysics, and cosmology associated with the cosmological tensions and anomalies,” JHEAp 34 (2022) 49–211, arXiv:2203.06142 [astro-ph.CO].
  * [17] K. Bundy, K. Westfall, N. MacDonald, R. Kupke, M. Savage, C. Poppett, A. Alabi, G. Becker, J. Burchett, P. Capak, A. Coil, M. Cooper, D. Cowley, W. Deich, D. Dillon, J. Edelstein, P. Guhathakurta, J. Hennawi, M. Kassis, K. G. Lee, D. Masters, T. Miller, J. Newman, J. O’Meara, J. X. Prochaska, M. Rau, J. Rhodes, R. M. Rich, C. Rockosi, A. Romanowsky, C. Schafer, D. Schlegel, A. Shapley, B. Siana, Y. S. Ting, D. Weisz, M. White, B. Williams, G. Wilson, M. Wilson, and R. Yan, “FOBOS: A Next-Generation Spectroscopic Facility at the W. M. Keck Observatory,” arXiv e-prints (Jul, 2019) arXiv:1907.07195, arXiv:1907.07195 [astro-ph.IM].
  * [18] J. Marshall, J. Bullock, A. Burgasser, K. Chambers, D. DePoy, A. Dey, N. Flagey, A. Hill, L. Hillenbrand, D. Huber, T. Li, S. Juneau, M. Kaplinghat, M. Mateo, A. McConnachie, J. Newman, A. Petric, D. Schlegel, A. Sheinis, Y. Shen, D. Simons, M. Strauss, K. Szeto, K.-V. Tran, C. Yèche, and the MSE Science Team, “The Maunakea Spectroscopic Explorer,” arXiv e-prints (Jul, 2019) arXiv:1907.07192, arXiv:1907.07192 [astro-ph.IM].
  * [19] R. Ellis, K. Dawson, J. Bland-Hawthorn, R. Bacon, A. Bolton, M. Bremer, J. Brinchmann, K. Bundy, C. Conroy, B. Delabre, A. Dey, A. Drlica-Wagner, J. Greene, L. Guzzo, J. Johnson, A. Leauthaud, K.-G. Lee, L. Pasquini, L. Pentericci, J. Richard, H.-W. Rix, C. Rockosi, D. Schlegel, A. Slosar, M. Strauss, M. Takada, E. Tolstoy, and D. Watson, “SpecTel: A 10-12 meter class Spectroscopic Survey Telescope,” arXiv e-prints (Jul, 2019) arXiv:1907.06797, arXiv:1907.06797 [astro-ph.IM].
  * [20] D. Schlegel, J. A. Kollmeier, and S. Ferraro, “The MegaMapper: a z¿2 spectroscopic instrument for the study of Inflation and Dark Energy,” in Bulletin of the American Astronomical Society, vol. 51, p. 229.  Sept., 2019.  arXiv:1907.11171 [astro-ph.IM].
  * [21] C. W. Stubbs and K. Heitmann, “Report on LSST Next-generation Instrumentation Workshop, April 11, 12 2019,” arXiv e-prints (May, 2019) arXiv:1905.04669, arXiv:1905.04669 [astro-ph.IM].
  * [22] B. Blum, S. W. Digel, A. Drlica-Wagner, S. Habib, K. Heitmann, M. Ishak, S. W. Jha, S. M. Kahn, R. Mandelbaum, P. Marshall, J. A. Newman, A. Roodman, and C. W. Stubbs, “Snowmass2021 Cosmic Frontier White Paper: Rubin Observatory after LSST,” arXiv e-prints (Mar., 2022) arXiv:2203.07220, arXiv:2203.07220 [astro-ph.CO].
  * [23] LSST Dark Energy Science Collaboration, R. Mandelbaum et al., “Wide-field Multi-object Spectroscopy to Enhance Dark Energy Science from LSST,” arXiv:1903.09323 [astro-ph.CO].
  * [24] LSST Dark Energy Science Collaboration, J. A. Newman et al., “Deep Multi-object Spectroscopy to Enhance Dark Energy Science from LSST,” arXiv:1903.09325 [astro-ph.CO].
  * [25] K. Bechtol et al., “Dark Matter Science in the Era of LSST,” arXiv:1903.04425 [astro-ph.CO].
  * [26] LSST Dark Matter Group Collaboration, A. Drlica-Wagner et al., “Probing the Fundamental Nature of Dark Matter with the Large Synoptic Survey Telescope,” arXiv:1902.01055 [astro-ph.CO].
  * [27] S. Chakrabarti, A. Drlica-Wagner, T. S. Li, N. Sehgal, J. D. Simon, S. Birrer, D. A. Brown, R. Bernstein, A. D. Bolatto, P. Chang, K. Dawson, P. Demorest, D. Grin, D. L. Kaplan, J. Lazio, J. Marshall, E. J. Murphy, S. Ransom, B. E. Robertson, R. Singh, A. Slosar, T. Treu, Y.-D. Tsai, and B. F. Williams, “Snowmass2021 Cosmic Frontier White Paper: Observational Facilities to Study Dark Matter,” arXiv e-prints (Mar., 2022) arXiv:2203.06200, arXiv:2203.06200 [astro-ph.CO].
  * [28] K. Bechtol, S. Birrer, F.-Y. Cyr-Racine, K. Schutz, S. Adhikari, A. Banerjee, S. Bird, N. Blinov, K. K. Boddy, C. Boehm, K. Bundy, M. Buschmann, S. Chakrabarti, D. Curtin, L. Dai, A. Drlica-Wagner, C. Dvorkin, A. L. Erickcek, D. Gilman, S. Heeba, S. Kim, V. Iršič, A. Leauthaud, M. Lovell, Z. Lukić, Y.-Y. Mao, S. Mau, A. Mitridate, P. Mocz, J. B. Muñoz, E. O. Nadler, A. H. G. Peter, A. Price-Whelan, A. Robertson, N. Sabti, N. Sehgal, N. Shipp, J. D. Simon, R. Singh, K. Van Tilburg, R. H. Wechsler, A. Widmark, and H.-B. Yu, “Snowmass2021 Cosmic Frontier White Paper: Dark Matter Physics from Halo Measurements,” arXiv e-prints (Mar., 2022) arXiv:2203.07354, arXiv:2203.07354 [hep-ph].
  * [29] J. Najita, B. Willman, D. P. Finkbeiner, R. J. Foley, S. Hawley, J. A. Newman, G. Rudnick, J. D. Simon, D. Trilling, R. Street, A. Bolton, R. Angus, E. F. Bell, D. Buzasi, D. Ciardi, J. R. A. Davenport, W. Dawson, M. Dickinson, A. Drlica-Wagner, J. Elias, D. Erb, L. Feaga, W.-f. Fong, E. Gawiser, M. Giampapa, P. Guhathakurta, J. L. Hoffman, H. Hsieh, E. Jennings, K. V. Johnston, V. Kashyap, T. S. Li, E. Linder, R. Mandelbaum, P. Marshall, T. Matheson, S. Meibom, B. W. Miller, J. O’Meara, V. Reddy, S. Ridgway, C. M. Rockosi, D. J. Sand, C. Schafer, S. Schmidt, B. Sesar, S. S. Sheppard, C. A. Thomas, E. J. Tollerud, J. Trump, and A. von der Linden, “Maximizing Science in the Era of LSST: A Community-Based Study of Needed US Capabilities,” arXiv e-prints (Oct., 2016) arXiv:1610.01661, arXiv:1610.01661 [astro-ph.IM].
  * [30] DESI Collaboration, A. Aghamousa, J. Aguilar, S. Ahlen, S. Alam, L. E. Allen, C. Allende Prieto, J. Annis, S. Bailey, and C. Balland, “The DESI Experiment Part II: Instrument Design,” arXiv e-prints (Oct, 2016) arXiv:1611.00037, arXiv:1611.00037 [astro-ph.IM].
  * [31] J. A. Kollmeier, G. Zasowski, H.-W. Rix, M. Johns, S. F. Anderson, N. Drory, J. A. Johnson, R. W. Pogge, J. C. Bird, G. A. Blanc, J. R. Brownstein, J. D. Crane, N. M. De Lee, M. A. Klaene, K. Kreckel, N. MacDonald, A. Merloni, M. K. Ness, T. O’Brien, J. R. Sanchez-Gallego, C. C. Sayres, Y. Shen, A. R. Thakar, A. Tkachenko, C. Aerts, M. R. Blanton, D. J. Eisenstein, J. A. Holtzman, D. Maoz, K. Nandra, C. Rockosi, D. H. Weinberg, J. Bovy, A. R. Casey, J. Chaname, N. Clerc, C. Conroy, M. Eracleous, B. T. Gänsicke, S. Hekker, K. Horne, J. Kauffmann, K. B. W. McQuinn, E. W. Pellegrini, E. Schinnerer, E. F. Schlafly, A. D. Schwope, M. Seibert, J. K. Teske, and J. L. van Saders, “SDSS-V: Pioneering Panoptic Spectroscopy,” arXiv e-prints (Nov., 2017) arXiv:1711.03234, arXiv:1711.03234 [astro-ph.GA].
  * [32] A. S. Bolton and D. J. Schlegel, “Spectro-Perfectionism: An Algorithmic Framework for Photon Noise-Limited Extraction of Optical Fiber Spectroscopy,” Publications of the ASP 122 no. 888, (Feb, 2010) 248, arXiv:0911.2689 [astro-ph.IM].


# Data Shapley Value for Handling Noisy Labels: An application in Screening
COVID-19 Pneumonia from Chest CT Scans

###### Abstract

A long-standing challenge of deep learning models involves how to handle noisy
labels, especially in applications where human lives are at stake. Adoption of
the data Shapley Value (SV), a cooperative game theoretical approach, is an
intelligent valuation solution to tackle the issue of noisy labels. Data SV
can be used together with a learning model and an evaluation metric to
validate each training point’s contribution to the model’s performance. The SV
of a data point, however, is not unique and depends on the learning model, the
evaluation metric, and other data points collaborating in the training game.
However, effects of utilizing different evaluation metrics for computation of
the SV, detecting the noisy labels, and measuring the data points’ importance
has not yet been thoroughly investigated. In this context, we performed a
series of comparative analyses to assess SV’s capabilities to detect noisy
input labels when measured by different evaluation metrics. Our experiments on
COVID-19-infected of CT images illustrate that although the data SV can
effectively identify noisy labels, adoption of different evaluation metric can
significantly influence its ability to identify noisy labels from different
data classes. Specifically, we demonstrate that the SV greatly depends on the
associated evaluation metric.

Index Terms: Data Shapley value, Noisy Labels, Data Valuation, Medical
Imaging, Capsule Networks.

##  1 Introduction

Deep learning has proven remarkable success in several medical fields,
including medical imaging, where advanced computer vision algorithms have
delivered near human-level performance in specific tasks. Several factors may
affect the success of a deep learning algorithm, including model structure,
initialization of the model parameters, training methods, and computational
hardware. However, the most influential factor is having access to large-scale
data sets with reliable labels. Collecting large-scale data sets with reliable
labels is, however, a significantly challenging task in several applications,
particularly within the medical domain. On the one hand, patients’ privacy
preservation and data sharing protocols prohibit hospitals and clinical
institutions from releasing their in-house data sets. On the other hand,
labeling medical images requires knowledge and expertise from radiologists and
physicians, making the labeling process resource-intensive. To overcome this
challenge, extracting pathology labels of large-scale medical images from
radiology reports with the help of text mining [1], human-machine
collaborative techniques [2, 3], and the use of non-expert annotators are some
of the proposed solutions in the literature [4]. However, the provided labels
by such methods are noisy and inaccurate compared to manually labeled images
[5]. Consequently, handling noisy labels in medical imaging data sets is
crucial for developing high-performance processing/learning models.

Prior work: Learning from a training data set with noisy labels has been a
challenging task within the deep learning domain for a long time. Research
studies indicate that noisy labels may have a more significant adverse effect
on the performance of deep learning models than the noises in data
attributes/measurements [6]. Proposed techniques for handling noisy labels in
medical imaging data sets include weakly supervised learning [7, 8, 9],
customized training methods [4], and re-weighting training samples [10].
Another approach to tackle the noisy label problem is to adopt Shapley Value
(SV), a cooperative game theoretical method, where each training point is
considered a player in the training game, and its contribution to any subset
of players is measured using a performance evaluation metric. Researchers have
leveraged the SV for data valuation and measuring the quality of data points
in different deep learning applications [11, 12, 13, 14, 15, 16, 17, 18]. In
[19] and [20], authors utilized the SV in a federated learning study to
measure the quality of data from each participator. A recent study [17] has
investigated the capability of the SV in quantifying the importance of
training data points in the performance of the learning model in a large-scale
X-ray data set. It should be noted that the SV of a data point is not a unique
value and depends on the learning model, the evaluation metric, and other data
points collaborating in the training game. So far, research studies have used
only one specific evaluation metric in their SV computation process, such as
accuracy metric in [19, 17] and Area under the ROC curve (AUC-ROC) metric in
[20] to demonstrate the data SV capability in measuring the quality of data
points. However, the effect of utilizing different evaluation metrics in
computing the SV, detecting the noisy labels, and measuring the data points’
importance in the training process has never been investigated.

Contributions: This study investigates effects of incorporating different
evaluation metrics in determining the data SV and quantifying the importance
of each training point in the model’s performance. To the best of our
knowledge, this is the first study that explores adoption of various
evaluation metrics in measuring the data SV and detecting noisy input labels.
The data SV measures the contribution of each training point to all possible
subsets of the training set, having an exponential complexity in the size of
the training set. Therefore, we use a permutation sampling approach to
approximate the data SV. We assess the capability of data SV obtained by
commonly-used classification metrics, including accuracy, recall, and
specificity in detecting the noisy labels through a set of experiments with
different noise levels. The experiments are performed based on a COVID- 19
screening task from chest CT scans by implementing a lightweight Capsule-
network-based classifier [21] to extract discriminative features from chest CT
scans and distinguish COVID-19-infected CT images from normal ones [22]. The
Capsule-network-based classifier can represent each CT image via a small
feature vector. Since calculating the SV requires retraining the learning
model on multiples coalitions of training samples, we use a fast classifier to
make our data valuation process time-efficient. Therefore, the extracted
feature vector is fed into a logistic regression classifier for final decision
making. The results indicate that the measured data SV is not unique and is
highly dependent on the evaluation metric. We demonstrate that despite the
great potential of the data SV in detecting noisy labels, its performance is
extremely affected by the adopted evaluation metric.

##  2 Data Shapley Value

This study implements the data SV in its data valuation approach for
identifying training samples with noisy labels. Data SV [23] is a well-known
measure in cooperative game theory for assigning a fair payoff to each player
of the game by taking into account its contribution to all possible coalitions
of the players. The training process of an ML/DL model can be assumed as a
game where training data points are players that collaborate to achieve the
highest model performance. Therefore, the SV can measure the importance of
training samples in the performance of the ML/DL model [11, 12, 13, 14, 15,
16, 17, 18]. Having a training set D, the SV of a data point $i$ $\epsilon$ D
is calculated as

$\text{$SV_{i}$}=\frac{1}{N}\sum_{S\subseteq
D-\\{i\\}}\frac{1}{\binom{N-1}{|S|}}[V\\{S\cup i\\}-V\\{S\\}],$ (1)

where $S$ is any possible subset of $D$ not including $i$, $N$ is the size of
the training set $D$, and $V$ is an evaluation metric for measuring the model
performance. $V\\{S\\}$ is the performance of the model trained on the subset
$S$ and measured by the evaluation metric $V$. In other words, the SV of a
training data point $i$ is its average marginal contribution to all subsets of
$S$, which can be interpreted as a quality measure for data assessment.
However, computing SV for a player requires computing its marginal
contribution to all possible coalitions of game players, which has exponential
complexity in the number of players. Moreover, in data SV computation,
calculating each marginal contribution, $V\\{S\\}$ requires training the model
on S that makes the computation process more time-consuming. To overcome this
challenge, we use a permutation sampling method [18] to approximate the data
SV. For this purpose, we use an equivalent form of Eq. 1 as follows

$\text{$SV_{i}$}=\frac{1}{N!}\sum_{\pi\epsilon\Pi}[V\\{S_{i}^{\pi}\cup
i\\}-V\\{S_{i}^{\pi}\\}],$ (2)

where $\Pi$ is all possible permutations of data points, $\pi\epsilon\Pi$ is a
sample permutation of data points, and $S_{i}^{\pi}$ is the set of training
points coming before data point $i$ in permutation $\pi$. Indeed, in each
random permutation, we can assume that data points are joining the training
game in random order. Then, each data point will receive the marginal
contribution that his collaboration brings to the training points which have
already played their role in the game. The data SV for each training point
would be its average marginal contributions over all possible permutations. In
practice, using a permutation sampling approach, the data SV will converge
after $3N$ permutations [16]. It is worth mentioning that the SV of a data
point is not a unique value and depends on the learning model, the evaluation
metric, and other data points collaborating in the training game. So far,
other studies have used a specific evaluation metric in their SV computation
process and demonstrated the data SV capability in measuring the quality of
data points. Here, we perform a set of experiments to investigate the effect
of different evaluation metrics in measuring the SV, quantifying the quality
of training points, and detecting noisy labels. We use the standard evaluation
metrics in classification tasks, including accuracy, recall, and specificity,
and discuss their effectiveness in detecting noisy labels in both classes of
data. Our data valuation process, illustrated in Fig. 1, includes: i) training
a deep classifier on a training set, ii) extracting high-resolution features
from training CT images using the trained deep classifier and feeding them
into a fast classifier, which is logistic regression in this study, iii)
calculating the SV of each training point based on different evaluation
metric, iv) analyzing the obtained SVs and detecting noisy labels.

The deep classifier used for discriminating COVID-19 infected CT images from
the normal ones is a lightweight DL model containing two convolutional and two
Capsule layers. A batch-normalization layer follows the first convolutional
layer, and a max-pooling layer follows the second one. The ReLU activation
function is applied after each convolutional layer to capture non-linear
patterns. The number of channels for each convolutional layer is $64$. The
output of the max-pooling layer is reshaped and fed to a Capsule layer to
extract high-resolution features from CT images. Finally, the last Capsule
layer predicts the probability that each CT image belongs to infected or non-
infected classes. It is noteworthy that screening COVID-19-infected CT images
on a slice-level basis can be a primary step in detecting COVID-19 patients
from healthy cases. We use a weighted loss function to tackle the imbalance
dataset, considering a higher penalty to the samples from minority class,
which is COVID-19 infected slices in our case. By extracting high-resolution
features from the last Capsule layer, each CT image with a matrix size of
$512\times 512$ is presented in a $1\times 16$ vector. Next, extracted
features from the training set and their corresponding labels are used in the
SV computation process. For more information about Capsule network-based
classifiers and the weighted loss function, please refer to Reference [24].

##  3 Dataset description

Fig. 1: The proposed framework to quantify the quality of data and discriminate COVID-19 infected CT images from normal ones.  Table 1: Training, validation and test sets used in our experiments. | COVID-19 | Normal | Total
---|---|---|---
Training set | $100$ | $400$ | $500$
Validation Set | $50$ | $200$ | $250$
Test Set | $241$ | $759$ | $1000$

The dataset used in our experiments is a combination of two different open-
access datasets. For non-COVID-19 CT images, we used a subset of $50$ normal
cases from our recently released dataset referred to as the “COVID-CT-MD”
[25], which is available through Figshare
111https://figshare.com/s/c20215f3d42c98f09ad0. For CT images with the
evidence of COVID-19 lesions, we used a public dataset containing chest CT
volumes of 10 COVID-19 patients [26], where three expert radiologists have
annotated COVID-19 manifestations. In both datasets, the matrix size of the CT
images is $512\times 512$ pixels. We randomly split the dataset into three
independent groups, as presented in table 1, including $100$, $50$, and $241$
COVID-19 infected CT scans, and $400$, $200$, and $759$ normal CT scans for
training, validation, and test sets. The extracted features from the training
set are used to train the logistic regression model over multiple permutation
sampling, and the test set is used for measuring the model performance during
the SV computation process.

##  4 Experimental Results

Fig. 2: The convergence of SV for randomly selected training samples. The
x-axis indicates the number of iteration in SV computation process and the
y-axis indicates the estimate of SV.

As the pre-processing step, each CT slice is normalized based on its pixel
intensities’ mean and standard deviation. Furthermore, we utilize the real-
time data augmentation method to enhance the model’s performance on unseen
data. Each mini-batch of original images during the training process is
transformed into synthetic images using conventional data augmentation
strategies such as zooming, shifting, and shearing. Over the training’ epochs,
the model will observe each augmented image only once, resulting in an
improvement in the model’s generalization. In each mini-batch, 16 CT images
are fed to the network. The Adam optimization algorithm with an initial
learning rate of $0.001$ minimizes the loss function over the training
process. The number of passes through the training set is set to $100$.
However, to mitigate the model over-fitting, the training process will stop
whenever the loss function on the validation set is not reduced over five
epochs. Finally, the SV computation is performed using the training set as the
players for training the logistic regression classifier and the test set for
measuring the model performance trained on any possible subset of training
points. As mentioned previously, we have $100$ COVID-19 infected CT slices and
$400$ normal CT images as our training set. Since we aim to deal with noisy
labels in the training set, we manually add some noises in the labels of our
training set and investigate the capability of the SV obtained based on
different evaluation metrics in detecting noisy labels. We run three
experiments with three different noise levels, including $10\%$, $20\%$, and
$30\%$ in each class of data. Therefore, 10, 20, and 30 positive CT images and
40, 80, and 120 normal CT scans have incorrect labels in experiment I, II, and
III, respectively.

Table 2: The efficiency of data SV obtained based on different evaluation
metrics in detecting noisy labels of each class of data in Exp $I$, $II$, and
$III$ with the noise level of $10\%$, $20\%$, and $30\%$. Fig. 3: Mapping
between the data SV and the type of data class derived via different
evaluation metrics (Plots are related to the Exp. III).

In each experiment, first, we train the Capsule-network-based classifier using
the training set with noisy labels. Next, the extracted features from the
trained classifier are fed into a logistic regression classifier to compute
the data SV. It is worth mentioning that computing the SV for each training
point requires re-training the classifier on multiple subsets of training
points, making the data evaluation process time-intensive. Hence, we use a
fast classifier such as logistic regression to make the computation process
affordable. We adopt different evaluation metrics, including accuracy, recall,
and specificity, to calculate the data SV. We run the computation process for
$2000$ permutations to assure the convergence of the estimated data SVs. Fig.
2 presents the convergence of the SV for two randomly selected training
samples. As can be observed, the estimated SV converges after $3N=1500$
permutations, which is in agreement with previous works [16].

Next, we order the training points based on their estimated SV and mapped them
with the corresponding label in our input labels that contains some noisy
labels (green points in column left) and the ground-truth labels, which are
correct labels (blue lines in right column), as visualized in Fig. 3. As can
be observed, when adopting accuracy or recall metrics, all training points
with the highest data SV belong to the positive class. In contrast, the
specificity metric ranks all training points with the negative input labels
before the data points with positive input labels. The plots also indicate
that although the specificity metric puts all images with the positive input
label among the lowest data SVs, the least valuable data points are the ones
mislabeled as positive classes (their ground-truth label is negative). This
reveals the capability of the SV in identifying noisy labels with the negative
ground-truth labels (mislabeled as the positive images in the input label)
when adopting the specificity metric. On the contrary, the majority of the
least valuable training points obtained by the accuracy and recall metrics
belong to negative input labels. However, according to the plots, the accuracy
metric assigns more value to the images with negative input labels than recall
metric. In addition, by comparing the plots in left and right columns, we can
conclude that the least valuable data points obtained by both accuracy and
recall metrics are the images mislabeled as the negative ones (have positive
ground-truth labels). This indicates the possibility of detecting noisy labels
with the positive ground-truth labels when using the accuracy or recall
metrics in computing the SV.

Furthermore, we investigate the effect of utilizing different evaluation
metrics in the SV computation process for identifying noisy labels. We
determined the fraction of noisy labels ranked as the $10\%$, $20\%$, and
$30\%$ of the lowest data SVs for experiments $I$, $II$, and $III$,
respectively. We also consider the fraction of noisy labels ranked as the
lowest $30\%$ data SVs in all experiments. As demonstrated in Table 2, when
adopting accuracy or recall metrics, all noisy labels with the positive
ground-truth label (which had been incorrectly labeled as negative ones in
input labels) have been correctly identified in experiments $I$ and $II$.
Conversely, when utilizing specificity metric in the SV computation process,
no noisy label with the positive ground-truth label (mislabeled as negative
images in the input labels) is ranked between the lowest $30\%$ of data SVs.
The results indicate that the adoption of the specificity metric has been more
successful in detecting noisy labels with the negative ground-truth label
(incorrectly labeled as positive ones in input labels). All noisy labels with
the negative ground-truth labels (both correctly labeled and mislabeled ones)
have been ranked in the lowest $30\%$ data SVs. Although the accuracy and
recall metrics show the same performance in detecting noisy labels with a
negative ground-truth label, the performance of the accuracy metric succeeds
the recall metric as the noise level increases.

It should be mentioned that, to the best of our knowledge, the adoption of
different evaluation metrics in measuring the data SV has not been discussed
in previous researches. However, Reference [17], which has performed an SV-
based data valuation study using the accuracy metric on an X-ray data set
containing $2000$ training points, illustrates that all the $100$ training
points with the highest SVs belong to the positive class. Besides, with the
help of three radiologists, they figured out that out of the $100$ lowest data
SVs in their experiment, there were $65$ mislabeled images where $80\%$ of
them were positive images that had been incorrectly labeled as negative ones.
Indeed, their experiments are in accordance with our results, confirming that
by utilizing the accuracy metric in computing the SV, the images with positive
labels will receive the highest values. In addition, it would be more likely
to detect mislabeled images with ground-truth positive labels (mislabeled as
negative ones). Our findings show that, while the data SV has a lot of
potential for detecting noisy labels in training sets, it is extremely
dependent on the evaluation metric used.

##  5 Conclusion

This research explores the effect of using different evaluation metrics in
data SV computation and its capability and limitation in identifying noisy
labels in a training set. We examine the standard evaluation metrics in
classification tasks, including accuracy, recall, and specificity in
calculating the data SV on a chest CT scan data set. Our findings show that,
while the data SV has a lot of potential for detecting noisy labels in
training sets, it depends highly on the evaluation metric used. We also
demonstrate that the data SV is not a unique value and will differ by
incorporating different evaluation metrics. This research study conducted
experiments on a binary classification task and a limited data set. We leave
the implementation of data SV based on various evaluation metrics and
investigating its capabilities and limitations in detecting noisy labels in a
multi-institutional data set and a multi-class classification problem to the
future. Exploring the dependency of the data SV on different types of learning
algorithms is another future direction of the present work.

## References

  * [1] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097–2106, 2017.
  * [2] K. Zhang, X. Liu, J. Shen, Z. Li, Y. Sang, X. Wu, Y. Zha, W. Liang, C. Wang, K. Wang, et al., “Clinically applicable ai system for accurate diagnosis, quantitative measurements, and prognosis of covid-19 pneumonia using computed tomography,” Cell, vol. 181, no. 6, pp. 1423–1433, 2020\.
  * [3] A. Degerli, M. Ahishali, M. Yamac, S. Kiranyaz, M. E. Chowdhury, K. Hameed, T. Hamid, R. Mazhar, and M. Gabbouj, “Covid-19 infection map generation and detection from chest x-ray images,” Health Information Science and Systems, vol. 9, no. 1, pp. 1–16, 2021.
  * [4] G. Wang, X. Liu, C. Li, Z. Xu, J. Ruan, H. Zhu, T. Meng, K. Li, N. Huang, and S. Zhang, “A noise-robust framework for automatic segmentation of covid-19 pneumonia lesions from ct images,” IEEE Transactions on Medical Imaging, vol. 39, no. 8, pp. 2653–2663, 2020.
  * [5] L. Oakden-Rayner, “Exploring large-scale public medical image datasets,” Academic radiology, vol. 27, no. 1, pp. 106–112, 2020.
  * [6] X. Zhu and X. Wu, “Class noise vs. attribute noise: A quantitative study,” Artificial intelligence review, vol. 22, no. 3, pp. 177–210, 2004.
  * [7] S. Hu, Y. Gao, Z. Niu, Y. Jiang, L. Li, X. Xiao, M. Wang, E. F. Fang, W. Menpes-Smith, J. Xia, et al., “Weakly supervised deep learning for covid-19 infection detection and classification from ct images,” IEEE Access, vol. 8, pp. 118869–118883, 2020.
  * [8] X. Wang, X. Deng, Q. Fu, Q. Zhou, J. Feng, H. Ma, W. Liu, and C. Zheng, “A weakly-supervised framework for covid-19 classification and lesion localization from chest ct,” IEEE transactions on medical imaging, vol. 39, no. 8, pp. 2615–2625, 2020.
  * [9] I. Laradji, P. Rodriguez, O. Manas, K. Lensink, M. Law, L. Kurzman, W. Parker, D. Vazquez, and D. Nowrouzezahrai, “A weakly supervised consistency-based learning method for covid-19 segmentation in ct images,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2453–2462, 2021.
  * [10] C. Xue, Q. Dou, X. Shi, H. Chen, and P.-A. Heng, “Robust learning at noisy labeled medical images: Applied to skin lesion classification,” in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 1280–1283, IEEE, 2019.
  * [11] S. Maleki, L. Tran-Thanh, G. Hines, T. Rahwan, and A. Rogers, “Bounding the estimation error of sampling-based shapley value approximation,” arXiv preprint arXiv:1306.4265, 2013.
  * [12] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in Proceedings of the 31st international conference on neural information processing systems, pp. 4768–4777, 2017.
  * [13] L. Merrick and A. Taly, “The explanation game: Explaining machine learning models using shapley values,” in International Cross-Domain Conference for Machine Learning and Knowledge Extraction, pp. 17–38, Springer, 2020.
  * [14] E. Song, B. L. Nelson, and J. Staum, “Shapley effects for global sensitivity analysis: Theory and computation,” SIAM/ASA Journal on Uncertainty Quantification, vol. 4, no. 1, pp. 1060–1083, 2016.
  * [15] E. Štrumbelj and I. Kononenko, “Explaining prediction models and individual predictions with feature contributions,” Knowledge and information systems, vol. 41, no. 3, pp. 647–665, 2014.
  * [16] A. Ghorbani and J. Zou, “Data shapley: Equitable valuation of data for machine learning,” in International Conference on Machine Learning, pp. 2242–2251, PMLR, 2019.
  * [17] S. Tang, A. Ghorbani, R. Yamashita, S. Rehman, J. A. Dunnmon, J. Zou, and D. L. Rubin, “Data valuation for medical imaging using shapley value and application to a large-scale chest x-ray dataset,” Scientific reports, vol. 11, no. 1, pp. 1–9, 2021.
  * [18] R. Jia, D. Dao, B. Wang, F. A. Hubis, N. Hynes, N. M. Gürel, B. Li, C. Zhang, D. Song, and C. J. Spanos, “Towards efficient data valuation based on the shapley value,” in The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1167–1176, PMLR, 2019.
  * [19] T. Wang, J. Rausch, C. Zhang, R. Jia, and D. Song, “A principled approach to data valuation for federated learning,” in Federated Learning, pp. 153–167, Springer, 2020.
  * [20] N. Khuri and S. Parsons, “A value-based approach for training of classifiers with high-throughput small molecule screening data,” in Proceedings of the 12th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics, pp. 1–10, 2021.
  * [21] G. Hinton, S. Sabour, and N. Frosst, “Matrix capsules with EM routing,” 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings, pp. 1–29, 2018.
  * [22] A. Mohammadi, Y. Wang, N. Enshaei, P. Afshar, F. Naderkhani, A. Oikonomou, J. Rafiee, H. Rodrigues de Oliveira, S. Yanushkevich, and K. N. Plataniotis, “Diagnosis/Prognosis of COVID-19 Chest Images via Machine Learning and Hypersignal Processing: Challenges, opportunities, and applications,” IEEE Signal Processing Magazine, vol. 38, pp. 37–66, sep 2021.
  * [23] L. S. Shapley, 17\. A value for n-person games.  Princeton University Press, 2016.
  * [24] S. Heidarian, P. Afshar, N. Enshaei, F. Naderkhani, M. J. Rafiee, F. B. Fard, K. Samimi, S. F. Atashzar, A. Oikonomou, K. N. Plataniotis, et al., “Covid-fact: A fully-automated capsule network-based framework for identification of covid-19 cases from chest ct scans,” Frontiers in Artificial Intelligence, vol. 4, 2021.
  * [25] P. Afshar, S. Heidarian, N. Enshaei, F. Naderkhani, M. J. Rafiee, A. Oikonomou, F. Babaki Fard, K. Samimi, K. Plataniotis, and A. Mohammadi, “COVID-CT-MD: COVID-19 Computed Tomography (CT) Scan Dataset Applicable in Machine Learning and Deep Learning,” 2020.
  * [26] J. Ma, Y. Wang, X. An, C. Ge, Z. Yu, J. Chen, Q. Zhu, G. Dong, J. He, Z. He, et al., “Towards efficient covid-19 ct annotation: A benchmark for lung and infection segmentation,” arXiv e-prints, pp. arXiv–2004, 2020\.