Submitted to the Proceedings of the US Community Study

on the Future of Particle Physics (Snowmass 2021)

# The REDTOP experiment: Rare $\eta/\eta^{\prime}$ Decays To Probe New
Physics.

J. Elam, A. Mane  Argonne National Laboratory, (USA)  J. Comfort, P. Mauskopf,
D. McFarland, L. Thomas  Arizona State University, (USA)  I. Pedraza, D. Leon,
S. Escobar, D. Herrera, D. Silverio  Benemerita Universidad Autonoma de
Puebla, (Mexico)  W. Abdallah  Department of Mathematics, Faculty of Science,
Cairo University, Giza (Egypt)  D. Winn  Fairfield University, (USA)  M.
Spannowsky  Durham University, (UK)  J. Dey, V. Di Benedetto, B. Dobrescu, D.
Fagan, E. Gianfelice-Wendt, E. Hahn, D. Jensen, C. Johnstone, J. Johnstone, J.
Kilmer, T. Kobilarcik, A. Kronfeld, K. Krempetz, S. Los, M. May, A. Mazzacane,
N. Mokhov, W. Pellico, A. Pla-Dalmau, V. Pronskikh, E. Ramberg, J. Rauch, L.
Ristori, E. Schmidt, G. Sellberg, G. Tassotto, Y. D. Tsai  Fermi National
Accelerator Laboratory, (USA)  A. Alqahtani  Georgetown University, (USA)  J.
Shi  Guangdong Provincial Key Laboratory of Nuclear Science, Institute of
Quantum Matter, South China Normal University, Guangzhou 510006, (China)
Guangdong-Hong Kong Joint Laboratory of Quantum Matter, Southern Nuclear
Science Computing Center, South China Normal University, Guangzhou 510006,
(China)  R. Gandhi  Harish-Chandra Research Institute, HBNI, Jhunsi (India)
S. Homiller  Harvard University, Cambridge, MA (USA)  X. Chen, Q. Hu
Institute of Modern Physics, Chinese Academy of Sciences, Lanzhou (China)  E.
Passemar  Indiana University (USA)  P. Sánchez-Puertas  Institut de Física
d’Altes Energies - Barcelona (Spain)  S. Roy  Institute of Physics,
Sachivalaya Marg, Sainik School Post, Bhubaneswar, (India)  C. Gatto
[Corresponding author] Istituto Nazionale di Fisica Nucleare – Sezione di
Napoli, (Italy)  Northern Illinois University, (USA)  W. Baldini  Istituto
Nazionale di Fisica Nucleare – Sezione di Ferrara, (Italy)  R. Carosi, A.
Kievsky, M. Viviani  Istituto Nazionale di Fisica Nucleare – Sezione di Pisa,
(Italy)  W. Krzemień, M. Silarski, M. Zielinski  Institute of Physics,
Jagiellonian University, 30-348 Krakow, (Poland)  D. Guadagnoli  Laboratoire
d’Annecy-le-Vieux de Physique Théorique, (France)  D. S. M. Alves, S.
González-Solís, S. Pastore  Los Alamos National Laboratory, (USA)  M.
Berlowski  National Centre for Nuclear Research – Warsaw, (Poland)  G. Blazey,
A. Dychkant, K. Francis, M. Syphers, V. Zutshi, P. Chintalapati, T. Malla, M.
Figora, T. Fletcher  Northern Illinois University, (USA)  A. Ismail  Oklahoma
State University, (USA)  D. Egaña-Ugrinovic  Perimeter Institute for
Theoretical Physics, Waterloo, (Canada)  Y. Kahn  Princeton University,
Princeton, (USA)  D. McKeen  TRIUMF, (Canada)  P. Meade  Stony Brook
University – New York, (USA)  A. Gutierrez, M. A. Hernandez-Ruiz  Universidad
Autonoma de Zacatecas  R. Escribano, P. Masjuan, E. Royo  Universitat Autónoma
de Barcelona  Institut de Física d’Altes Energies - Barcelona (Spain)  B.
Kubis  Universität Bonn, Helmholtz-Institut für Strahlen- und Kernphysik
(Theorie) and Bethe Center for Theoretical Physics (Germany)  J. Jaeckel
Universität Heidelberg, (Germany)  L. E. Marcucci  Universita’ di Pisa and
INFN, (Italy)  C. Siligardi, S. Barbi, C. Mugoni  Universita’ di Modena e
Reggio Emilia, (Italy)  M. Guida  Universita’ di Salerno and INFN – Sezione di
Napoli, (Italy)  S. Charlebois, J. F. Pratte  Université de Sherbrooke,
(Canada)  L. Harland-Lang  University of Oxford, (UK)  J. M. Berryman
University of California Berkeley, (USA)  Institute for Nuclear Theory,
University of Washington, (USA)  R. Gardner, P. Paschos  University of
Chicago, (USA)  J. Konisberg  University of Florida, (USA)  C. Mills, Z. Ye
University of Illinois Chicago, (USA)  M. Murray, C. Rogan, C. Royon, N.
Minafra, A. Novikov, F. Gautier, T. Isidori  C. Mills, Z. Ye  University of
Illinois Chicago, (USA)  S. Gardner, X. Yan  University of Kentucky, (USA)  Y.
Onel  University of Iowa, (USA)  M. Pospelov  University of Minnesota, (USA)
B. Batell, A. Freitas, M. Rai  University of Pittsburgh, (USA)  D. N. Gao
University of Science and Technology of China, (China)  K. Maamari  University
of Southern California, (USA)  A. Kupść  University of Uppsala, (Sweden)  B.
Fabela-Enriquez  Vanderbilt University, (USA)  A. Petrov  Wayne State
University, (USA)  S. Tulin  York University, (Canada) [

###### Abstract

The $\eta$ and $\eta^{\prime}$ mesons are nearly unique in the particle
universe since they are almost Goldstone bosons and the dynamics of their
decays are strongly constrained. The integrated $\eta$-meson samples collected
in earlier experiments amount to $\sim 10^{9}$ events. A new experiment,
REDTOP (Rare Eta Decays To Observe Physics Beyond the Standard Model), is
being proposed, with the intent of collecting a data sample of order 1014
$\eta$ (1012 $\eta^{\prime}$) for studying very rare decays. Such statistics
are sufficient for investigating several symmetry violations, and for
searching for particles and fields beyond the Standard Model. In this work we
present several studies evaluating REDTOP sensitivity to processes that couple
the Standard Model to New Physics through all four of the so-called _portals_
: the Vector, the Scalar, the Axion and the Heavy Lepton portal. The
sensitivity of the experiment is also adequate for probing several
conservation laws, in particular $CP$, $T$ and Lepton Universality, and for
the determination of the $\eta$ form factors, which is crucial for the
interpretation of the recent measurement of muon $g-2$.
Preprint numbers: FERMILAB-FN-1153-AD-PPD-T, LA-UR-22-22208

REDTOP Collaboration Homepage: ]https://redtop.fnal.gov

###### Contents

  1. I Introduction
  2. II Motivations for a high luminosity $\eta/\eta^{\prime}$ factory
  3. III BSM physics with an $\eta/\eta^{\prime}$ factory
    1. III.1 Searches for new particles and fields
      1. III.1.1 Vector portal models
      2. III.1.2 Scalar portal models
      3. III.1.3 Heavy neutral lepton portal models
      4. III.1.4 The visible QCD axion
      5. III.1.5 Axion-like particles
      6. III.1.6 Probing a BSM origin of the proton radius anomaly
    2. III.2 Tests of conservation laws
      1. III.2.1 C and CP violation from Dalitz asymmetries in $\eta\rightarrow\pi^{+}\pi^{-}\pi^{0}$
      2. III.2.2 CP violation in $\eta\rightarrow\pi^{+}\pi^{-}e^{+}e^{-}$
      3. III.2.3 Tests of CP invariance via $\gamma^{*}$ polarization studies in $\eta\to\pi^{+}\pi^{-}\gamma^{*}$
      4. III.2.4 CP violation in $\eta\to\mu^{+}\mu^{-}$
      5. III.2.5 CP violation in double Dalitz decays $\eta\to\ell^{+}\ell^{-}\ell^{\prime+}\ell^{\prime-}$
      6. III.2.6 CP violation in $\eta\rightarrow\pi^{0}\mu^{+}\mu^{-}$
      7. III.2.7 Lepton flavor violation measurements
      8. III.2.8 Tests of lepton flavor universality with $\eta/\eta^{\prime}$ mesons
  4. IV Non-perturbative QCD
    1. IV.1 $\eta\to 3\pi$ and light quark mass extraction
    2. IV.2 $\eta^{\prime}\to\eta\pi\pi$ and $\eta^{\prime}\to 3\pi$
    3. IV.3 Form factors and their applications
      1. IV.3.1 $P\to\gamma\gamma$ decays
      2. IV.3.2 Dalitz decay
      3. IV.3.3 Double-Dalitz decays
      4. IV.3.4 Contribution to the anomalous magnetic moment of the muon, $(g-2)_{\mu}$
  5. V Muon polarimetry at REDTOP
    1. V.1 CP violation via longitudinal polarization in $\eta\to\mu^{+}\mu^{-}$
    2. V.2 CPT violation in transverse polarization
    3. V.3 Polarization in Dalitz decays
    4. V.4 Polarization in $\eta\to\pi^{0}\mu^{+}\mu^{-}$ decays
  6. VI The REDTOP experiment
    1. VI.1 Hadro-production of $\eta$ mesons
    2. VI.2 Beam and target requirements
    3. VI.3 Detector requirements
    4. VI.4 The REDTOP detector
      1. VI.4.1 The vertex detector
      2. VI.4.2 The central tracker
      3. VI.4.3 Electromagnetic and hadronic calorimeter: ADRIANO2
      4. VI.4.4 The Threshold Cerenkov Radiator (TCR)
      5. VI.4.5 The Timing Layer ($TL$, optional)
      6. VI.4.6 The Muon Polarimeter (MuPol, optional)
      7. VI.4.7 Superconducting solenoid
    5. VI.5 The event trigger systems
    6. VI.6 The event structure
    7. VI.7 Level-0 trigger
    8. VI.8 Level-1 trigger
    9. VI.9 Level-2 trigger
    10. VI.10 Digitize and Compress: summary of trigger performance
    11. VI.11 The Computing Model
  7. VII The Simulation Framework
    1. VII.1 The event generator: $GenieHad$
    2. VII.2 The particle transport software: $slic$
    3. VII.3 The reconstruction and analysis software: $lcsim$
    4. VII.4 The simulation strategy
      1. VII.4.1 Beam parameters
      2. VII.4.2 Digitization
      3. VII.4.3 Simulation of the trigger systems
      4. VII.4.4 Pattern recognition and PID
      5. VII.4.5 Track reconstruction in the Fiber Tracker and in the Central Tracker (helix fit and Kalman filter)
      6. VII.4.6 Shower reconstruction in the TCR and ADRIANO2 calorimeter (parametric)
  8. VIII Sensitivity Studies to Physics Beyond the Standard Model
    1. VIII.1 Estimation of the sensitivity curves
    2. VIII.2 Standard Model background
    3. VIII.3 Generation of signal events
  9. IX Sensitivity to the Vector portal
    1. IX.1 $A^{\prime}\rightarrow e^{+}e^{-}$: Bump-hunt analysis
    2. IX.2 $A^{\prime}\rightarrow e^{+}e^{-}$: Detached-vertex analysis
    3. IX.3 $A^{\prime}\rightarrow\mu^{+}\mu^{-}$: Bump-hunt analysis
    4. IX.4 $A^{\prime}\rightarrow\mu^{+}\mu^{-}$: Detached-vertex analysis
    5. IX.5 $B\rightarrow\pi^{+}\pi^{-}$: Bump-hunt analysis
    6. IX.6 $B\rightarrow\pi^{+}\pi^{-}$: Detached-vertex analysis
    7. IX.7 Sensitivity to selected theoretical models
      1. IX.7.1 Minimal Dark Photon Model
      2. IX.7.2 Leptophobic B boson Model
      3. IX.7.3 Protophobic Fifth Force Model
  10. X Sensitivity to the Scalar portal
    1. X.1 $h\rightarrow e^{+}e^{-}$: $Bump-hunt$ analysis
    2. X.2 $h\rightarrow e^{+}e^{-}$: Detached-vertex analysis
    3. X.3 $h\rightarrow\mu^{+}\mu^{-}$: Bump-hunt analysis
    4. X.4 $h\rightarrow\mu^{+}\mu^{-}$: Detached-vertex analysis
    5. X.5 $h\rightarrow\pi^{+}\pi^{-}$: $Bump-hunt$ analysis
    6. X.6 $h\rightarrow\pi^{+}\pi^{-}$: _Detached-_ vertex analysis
    7. X.7 Sensitivity to selected theoretical models
      1. X.7.1 Hadrophilic Scalar Mediator (including Spontaneous Flavor Violation models)
      2. X.7.2 Two-Higgs doublet model
      3. X.7.3 Minimal scalar model
  11. XI Sensitivity to the Pseudoscalar portal
    1. XI.1 Leptonic decays: $a\rightarrow e^{+}e^{-}$ and $a\rightarrow\mu^{+}\mu^{-}$ final states
      1. XI.1.1 $\eta\rightarrow\pi^{+}\pi^{-}a$: bump-hunt analysis
      2. XI.1.2 $\eta\rightarrow\pi^{0}\pi^{0}a$: bump-hunt analysis
      3. XI.1.3 $\eta\rightarrow\pi^{0}\pi^{0}a$: detached-vertex analysis
    2. XI.2 Radiative decay: $a\rightarrow\gamma\gamma$ final states
    3. XI.3 Sensitivity to selected theoretical models
      1. XI.3.1 Sensitivity to Axion-Like Particles
      2. XI.3.2 _Piophobic QCD axion with mass of 17 MeV/ $c^{2}$ _
  12. XII Sensitivity to the Heavy Neutral Lepton portal
    1. XII.1 $\eta\rightarrow\pi^{0}H\;;\;H\rightarrow\nu N_{2}\;;\;N_{2}\rightarrow N_{1}h^{\prime}\;;\;h^{\prime}\rightarrow e^{+}e^{-}$: Detached vertex analysis
    2. XII.2 Sensitivity to selected theoretical models
      1. XII.2.1 Two-Higgs doublet model
    3. XII.3 Concluding remark
  13. XIII Tests of Conservation Laws
    1. XIII.1 _CP-_ violation studies from Dalitz plot mirror asymmetry in $\eta\rightarrow\pi^{+}\pi^{-}\pi^{0}$
    2. XIII.2 _CP-_ violation studies from the asymmetry of the decay planes of $\eta\rightarrow\pi^{+}\pi^{-}e^{+}e^{-}$
    3. XIII.3 _CP-_ violation studies in the decays $\eta\rightarrow\mu^{+}\mu^{-}$
    4. XIII.4 _CP-_ violation studies in the decays $\eta\rightarrow\gamma\mu^{+}\mu^{-}$
    5. XIII.5 _CP-_ violation studies from the asymmetry of the decay planes of $\eta\rightarrow\mu^{+}\mu^{-}e^{+}e^{-}$
    6. XIII.6 _CP-_ violation studies in the decay $\eta\rightarrow\pi^{0}\mu^{+}\mu^{-}$
    7. XIII.7 Lepton flavor violation in the decay $\eta\rightarrow e^{+}\mu^{-}$\+ c.c.
      1. XIII.7.1 Concluding remark
    8. XIII.8 Test of Lepton Flavor Universality
      1. XIII.8.1 $\eta\rightarrow 4\>leptons$ decays
      2. XIII.8.2 Test of Lepton Flavor Universality with $\eta\rightarrow\gamma\>2\>leptons$ decays
      3. XIII.8.3 Remarks on LFU measurements with $\eta$ mesons
  14. XIV Sensitivity to Non-perturbative QCD
    1. XIV.1 Form factor studies
      1. XIV.1.1 Concluding remark
  15. XV Discussion of the results
  16. XVI Conclusions
  17. XVII Appendix I: The Acceleration Scheme
    1. XVII.1 Fermilab configuration.
    2. XVII.2 CERN configuration.
    3. XVII.3 BNL configuration.
    4. XVII.4 HIAF configuration.
    5. XVII.5 GSI configuration.
  18. XVIII Appendix II: Tagged $\eta$-factory
  19. XIX Appendix III: Radiation Damage and Detector Aging
  20. XX Appendix IV: Radiation Shielding and Safety

## Executive Summary

  * •

The next decade represents an almost unique opportunity for laboratories with
high intensity proton accelerators to uncover Dark Matter or New Physics.

  * •

There are strong theoretical reasons to search for New Physics in the MeV–GeV
range.

  * •

The $\eta$ and $\eta^{\prime}$ mesons are almost unique since they carry he
same quantum numbers as the Higgs (except for parity), and have no Standard
Model charges. Their decays are flavor-conserving and most of them forbidden
at leading order (in various symmetry-breaking parameters) within the Standard
Model.

  * •

Rare decays are therefore enhanced compared to the remaining flavor-neutral
mesons. Thus an $\eta/\eta^{\prime}$ factory is an excellent laboratory for
studying rare processes and Beyond Standard Model physics at low energy.

  * •

A sample of order $10^{14}$($10^{12}$ ) $\eta(\eta^{\prime})$ mesons can
address most of the recent theoretical models. Such an experiment would have
enough sensitivity to explore a very large portion of the unexplored parameter
space for all the four portals connecting the Dark Sector with the Standard
Model. Lepton Universality and the $CP$ and $T$ symmetries can also be probed
with excellent sensitivity.

  * •

Many other studies can be conducted with such a large data sample, including,
for example, the determination of the $\eta$ form factors, which is crucial to
understanding the $(g-2)_{\mu}$ measurement.

  * •

The REDTOP Collaboration is proposing an $\eta/\eta^{\prime}$ factory with
such parameters. No similar experiment exists or is currently planned by the
international community.

  * •

A full detector simulation and reconstruction has been implemented to study
several processes driven by New Physics, and many theoretical models have been
bench marked. About $5\times 10{}^{10}$ background events have been generated
and fully reconstructed to estimate the sensitivity of REDTOP. This took over
three years to complete and required about $4\times 10{}^{7}$ core-hours of
computing on the Open Science Grid.

  * •

The Physics case for REDTOP is presented in the first part of this work. In
Sec. III.1 we discuss the four portals and the latest theoretical models
proposed to explain outstanding experimental anomalies. In Sec. III.2 we
discuss several tests of conservation laws which could be explored at REDTOP.

  * •

A description of the experiment, including the sub-detectors, the trigger
systems and the computing model, is presented in the central part of this
document. Radiation damage, detector aging and shielding issues have also been
considered. They are discussed in some detail in the appendices.

##  I Introduction

It is generally accepted that the Standard Model (SM) is not a complete
description of all quantum interactions. The exact nature of dark energy and
dark matter, and the baryon asymmetry of the universe, the observation of a
Universe in accelerating expansion, and neutrino masses, are among the very
interesting questions that cannot be answered within the framework of the SM.

There is a strong indication that the physics Beyond the Standard Model (BSM)
could contain new particles and/or force mediators, which significantly
violate some discrete symmetries of the universe, in particular _CP_.

The High Energy Physics community has engaged in an unprecedented experimental
effort, with the construction of the LHC and its four detectors, to observe
physics BSM in the High Energy domain. The absence of conclusive evidence so
far may suggest that: a) the New Physics most immediately accessible to
experimenters is at _low_ energies, rather than at high energies, and b) such
New Physics is elusive, in particular it couples to SM matter too faintly to
be detected by experiments at colliders. To detect such interactions, one may
need huge luminosities, which are one of the most attractive features of
fixed-target experiments.

As a consequence of fact a), a plethora of new theoretical models have
flourished, that extend the SM with light gauge bosons, in the MeV–GeV mass
range (see e.g., (Batell et al., 2009a; Reece and Wang, 2009; Bjorken et al.,
2009)). Such efforts have, on the one hand, been encouraged by the fact that
recently observed astrophysical anomalies point to such mass range as a
promising area of exploration. In addition, in such mass regime, otherwise
strong astrophysical and cosmological constraints are weakened or eliminated,
while constraints from high energy colliders are, in most cases, inapplicable.

On the other side, fact b) puts the spotlight on experimental searches with
techniques resting on an integrated luminosity that exceeds by several orders
of magnitudes those currently available at colliders. As pointed out in Ref.
Batell et al. (2009b), while the characteristic integrated luminosity for
high-energy colliders is of order 1041cm-2, the analogue of integrated
luminosity for a moderate intensity (namely, $\sim$1021 protons on target
(POT)) fixed target experiment with a 1 mm thick target is of the order of
$\sim$1044 cm-2. By applying formula (4) in Ref. Batell et al. (2009b) at a
fixed target experiment with Elab=1 GeV, one obtains the following comparison
between the production rates for neutral GeV-scale states at LHC and low
energy fixed targets:

$\frac{N_{LHC}}{N_{fixed\>target}}\sim 10^{-8+6n}$ (1)

where the interaction between the standard and dark matter is assumed being
mediated by marginal or irrelevant operators of dimension $4+n$, with $n\geq
0$.

For the kind of New Physics mentioned above, fixed-target experiments with
hadronic beams thus have well-defined advantages with respect to high-energy
colliders — let alone the tremendous difference in construction and operating
costs. At the same time, several factors are limiting the realization of a
high intensity fixed target experiment. Two of the most limiting factors are:
a) the sheer production rate of events from inelastic interaction of the beam
onto the target and, b) the large background from neutrons (either primary or
secondary) which make a signal in the detector. Regarding point a), the
technologies implemented in the present generation of detectors are not fast
enough to cope with proton beam intensities even as modest as few tens of
watts. Regarding point b), a neutron and a photon have very similar signatures
in a conventional, single-readout calorimeter, hindering the ability to
disentangle such particles unless novel detector techniques are implemented.
Last, but not least, intense neutron fluxes could damage quickly a detector if
non radiation-hard materials are used.

The design of the REDTOP experiment is based on the above considerations.
REDTOP is a high yield $\eta$/$\eta^{\prime}$-factory, operated in a fixed
target configuration with beam luminosity of order 1034 cm-2sec-1. The mass
range for potential discoveries is approximately [14 MeV-950 MeV], limited on
the lower side by the resolution of the detector and on the upper side by the
meson mass.

REDTOP is also a frontier experiment, aiming at measuring the
$\eta/\eta^{\prime}$ decay rates or their asymmetries for very rare processes,
and with a precision several orders of magnitude higher than the present
measurements. These decays would provide direct tests of conservation laws
and, along with other measurements, will open up new windows to discover
physics beyond the Standard Model. In addition to searches for new physics,
the experiment will involve development and first use of innovative detectors.
Novel instrumentation will include a super-light Frisch (2014) or an LGAD
tracker with unprecedentedly low material budget et al. (2020), an ADRIANO2
calorimeter, and a Threshold Cerenkov Radiator (TCR). An optional Active Muon
Polarimeter is being considered to improve the measurements of the muon
polarization. With the information obtained from the highly granular
calorimeter as well as from the other subdetectors, an extended Particle Flow
Analysis (PFA) M.Thomson (2006) could be implemented. The 5-D measurement
performed on the showers (energy, space, and time) will facilitate the
disentangling of complex or overlapping events.

The development of all such detector techniques will require a substantial
effort. On the other hand, future experiments, operating with similar event
rates or requiring similar levels of background rejection, will certainly
benefit from the pioneering R&D carried by the REDTOP Collaboration.

##  II Motivations for a high luminosity $\eta/\eta^{\prime}$ factory

It has been recently noted that “Light dark matter (LDM) must be neutral under
SM charges, otherwise it would have been discovered at previous colliders”
Krnjaic (2020). Under such circumstance, the study of processes originated by
particles carrying no SM charges is, intuitively, more appropriate in LDM
searches, as no charged currents are present, which could potentially
interfere with Beyond Standard Model (BSM) processes.

The $\eta$ and $\eta^{\prime}$ mesons have been widely studied in the past, as
their special nature has attracted the curiosity of the scientists (Nefkens,
1996). The $\eta$ is a Goldstone boson, therefore its QCD dynamics is strongly
constrained by that property. In nature, there are only few Goldstone bosons.
Furthermore, the $\eta$ is, at the same time, an eigenstate of the C, P, CP, I
and G operators with all zero eigenvalues (namely: $I^{G}J^{PC}=0^{+}0^{-+}$)
which makes it identical (except for parity) to the vacuum or the Higgs boson.
In that respect, it is a very pure state, carrying no Standard Model charges
and, as noted above, its decays do not involve charge-changing currents: all
decays of $\eta$ and $\eta^{\prime}$ mesons are flavor-conserving. Therefore,
a $\eta/\eta^{\prime}$-factory could be interpreted as a “low-energy Higgs-
factory”, anticipating much of the exploration achievable at a high-energy
Higgs factory. Any coupling to BSM states, therefore, does not interfere with
Standard Model charge-changing operators (as it occurs, for example, with
mesons carrying flavor). From the experimental point of view, the
$\eta/\eta^{\prime}$ dynamics is particularly favorable to the exploration of
small BSM effects since, as a consequence of the properties mentioned above,
they have an unusually small decay width ($\Gamma_{\eta}$=1.3 KeV vs
$\Gamma_{\rho}$=149 MeV, for example). Electromagnetic and strong decays are
suppressed up to order $\mathcal{O}(10^{-6})$ favoring the study of more rare
decays, especially those related to BSM particles and to violation of discrete
symmetries. This helps considerably in reconstructing the kinematic of the
event and in reducing the Standard Model background by requiring that the
invariant mass of the final state particles is consistent with the $\eta$
mass.

Another reason to investigate more precisely the $\eta/\eta^{\prime}$ mesons
is that their structure has never been fully understood. Recent work Singh and
Patel (2011) indicates that such mesons are unique among the pseudoscalars as
they have anomalously large masses which are contributed by quarks only about
80% of the momentum, leaving considerable room for potential contribution by
New Physics.

A summary of the processes that can be studied at REDTOP for exploring New
Physics is shown in Fig. 1. The processes are grouped by their physics topic,
and will be discussed in more details in the next sections.

Figure 1: Physics landscape for a $\eta/\eta^{\prime}$ factory.

Considering the present limits on the parameters associated to BSM physics and
the practical limitations of current detector technologies, the next
generation of experiments should be designed with the goal of producing no
less than $10^{13}$ $\eta$ mesons and $10^{11}$ $\eta^{\prime}$ mesons. The
physics reach of an experiment with such statistics is very broad, spanning
several aspects of physics BSM. The most relevant processes to be explored
fall into two main fields of research: _Search for New Particle and Fields_ ,
and _Test of Conservation laws_. Along with BSM Physics, the availability of
such a large sample of flavor-conserving mesons will also allow probing the
isospin violating sector of low energy QCD to an unprecedented degree of
precision. The large number of processes that could be studied at the proposed
$\eta/\eta^{\prime}$-factory will provide not only a nice scientific
laboratory but also the source of many topics for Ph.D. thesis.

Few of the BSM processes accessible with an $\eta/\eta^{\prime}$-factory of
REDTOP class have been selected for detailed sensitivity studies. They will be
discussed later in this paper, along with several recent theoretical models
which are explaining some of the anomalies that have been observed by the
experiments.

##  III BSM physics with an $\eta/\eta^{\prime}$ factory

Large samples of $\eta/\eta^{\prime}$ decays open new avenues for the study of
BSM physics. This is particularly true for weakly coupled hidden sectors, in
which the new fields are SM singlets, as well as studies of fundamental
symmetries and their breaking.

Models of new hidden sectors are typified by so-called portals, in which a new
field, either a vector, scalar, or a heavy neutral lepton, appears in an SM
gauge-singlet interaction of mass dimension four or less. As a result, the
additional interactions do not spoil the UV properties and hence the
renormalizability of the SM. Portals of higher mass dimension, such as axion
models, are also very interesting. REDTOP is capable of probing all of these
portals.

Turning to symmetry tests, REDTOP offers new opportunities for searches for CP
violation, as well as for tests of both lepton flavor violation and
universality. Within the standard model CP violation is described by one
complex phase in the Cabibbo-Kobayashi-Maskawa (CKM) quark-mixing matrix. All
three generations of quarks contribute in order to realize a non-zero CP-
violating effect. It has long been suspected new sources of CP violation must
exist in order to explain baryon asymmetry of the universe. Tantalizing hints
of lepton-flavor-universality violation have also been seen in $B$-meson
decays, and it is important to search for these effects in light-quark systems
as well. Searches for lepton-flavor violation in $\eta,\eta^{\prime}$ decays
complement searches for $\mu-e$ conversion in the field of a nucleus, for
which sensitive searches are being mounted worldwide.

REDTOP is well suited for all of these studies. Figure 1 provides a compact
illustration of the physics possibilities. The overview that follows describes
these possibilities, noting the _golden channels_ that have a higher signal to
noise ratio within REDTOP.

###  III.1 Searches for new particles and fields

One of the most prominent ways to accommodate NP is the so called _“hidden
sector physics_ ”, comprising new particles with masses below the electroweak
(EW) scale coupled very weakly to the SM world via so-called _portals_ Batell
et al. (2009b). Such schema are characterized by new particles which are
either heavy or interact indirectly with the SM sector. These _hidden sectors_
may be experimentally accessible via particles in the MeV–GeV mass range,
which are coupled to the Standard Model sectors via renormalizable
interactions with dimensionless coupling constants (the _portals_) or by
higher-dimensional operators. The latter, however, are suppressed by the
dimensional couplings of the order $\Lambda^{-n}$, associated to a new energy
scale of the hidden sector.

Three such portals are renormalizable within the Standard Model: the _Vector
portal_ , the _Scalar portal_ , and the _Heavy Neutral Lepton portal_. They
differ by the mass dimension of the SM operator which is coupled to the dark
sector. A fourth portal: the _Pseudoscalar (_ or _Axion) portal_ , is not, in
general, renormalizable, and the models falling in this category are often
_Effective Field Theories_. Nonetheless, the discovery of the Higgs boson
indicates that fundamental scalar bosons exist in nature, justifying the
search for more light pseudoscalar particles. Several classes of models exist,
accommodating such additional states: extensions of the Higgs sector Gunion et
al. (2000), models with extra non-doublet scalars Bae et al. (2012), or
pseudo-Nambu-Goldstone bosons (PNGB) of a spontaneously broken U(1) symmetry
Peccei and Quinn (1977a); Weinberg (1978a). All these models have new light
states that couple only weakly to Standard Model particles.

Several $\eta$/$\eta^{\prime}$-related processes have been selected to study
the sensitivity of REDTOP to such portals. Some of them could also shed some
light on anomalies observed in recent experiments. These are discussed below.
These searches are golden opportunities for REDTOP.

####  III.1.1 Vector portal models

Several extensions of the standard model are based on the interaction of new
light vector particles, resulting, for example, from extra gauge symmetries.
New vector states can mediate interactions both with the SM fields, and with
extra fields in the dark sector. It is speculated that the gauge structure of
the SM derives from a larger gauge group, as, for example, in the Grand
Unified Theories (GUTs), where new vector states exist. If these particles
exist, their mass is expected to be of the order of $10^{16}$ GeV, an energy
well beyond the direct reach of present accelerators. Other models assume that
the SM has additional gauge structures able to accommodate gauge bosons with
masses below the TeV scale Langacker (2009). The current results from LHC
experiments has put very strong bounds on the existence of such new vector
states, with the hypothesis that the coupling of the latter to the SM is large
enough. To cope with such observations, more recent theoretical models assume
the existence of relatively light vector states (e.g., in the MeV–GeV mass
range) with small couplings to the SM. This mass range is poorly constrained
by the LHC experiments, and it could be probed easily with dedicated
experiments with high intensity beams such as $\eta/\eta^{\prime}$-factories.

The _Vector portal_ spans several classes of models. Typical examples are:
kinetically mixed dark photons in the GeV mass range, gauge bosons coupled to
baryons, _dark Higgs_ bosons generated through the portal or via
_Higgsstrahlung_ , _heavy neutral leptons_ (HNL) generated through the portal.
Models currently under study by the REDTOP Collaboration include the _Minimal
dark photon model_(kinetically mixed dark photons), the _Leptophobic B boson
model_ , and the _Protophobic Fifth Force model_.

##### _Minimal dark photon model_

One of the most popular models in the _Vector portal_ is commonly referred to
as: _Minimal dark photon model._ In this case, the SM is augmented by a single
new state $A^{\prime}$ which couples to visible matter via a _kinetic mixing
parameter $\varepsilon$_ Holdom (1986). REDTOP could observe new vector
particles in the decays: $\eta/\eta^{\prime}\rightarrow\gamma
A^{\prime}\rightarrow\gamma e^{+}e^{-}$ and
$\eta/\eta^{\prime}\rightarrow\gamma
A^{\prime}\rightarrow\gamma\mu^{+}\mu^{-}$.

This process has a relatively large branching ratio $\sim 7\times 10^{-3}$ et
al. (2016). Consequently, REDTOP will be able to detect a number of such final
states with samples larger than $10^{8}$. Two experiments are pursuing
dedicated dark photon searches with $e^{+}e^{-}$ colliding beams: the HPS at
JLAB Adrian et al. (2018) and PADME at Laboratori Nazionali di Frascati Raggi
et al. (2015). REDTOP, on the other hand, will perform a similar search with
hadron-produced $\eta$ mesons, producing very different statistical and
systematic uncertainties. Preliminary sensitivity studies on the _Minimal dark
photon model_ have been performed as part of CERN’s “Physics Beyond Collider”
program et al. (2019) indicating that REDTOP is sensitive to a large portion
of the unexplored region of the _ $\varepsilon$ _parameter space. New studies,
based on a much better detector that can identify a detached vertices are in
progress.

A search for dark photons can be carried out at an
$\eta/\eta^{\prime}$-factory by looking for final states with a photon and two
leptons (Dalitz decay). Considering the process

$\eta\rightarrow\gamma A^{\prime}\rightarrow\gamma+l^{+}l^{-}$ (2)

in the hypothesis that the mass of this dark photon is smaller than the mass
of the $\eta/\eta^{\prime}$ meson, it will be relatively straightforward to
observe it. A detailed study of this process is presented in Sec. VIII.

##### _Leptophobic B boson model_

We consider a model for a leptophobic gauge boson $B$ that couples to baryon
number through the following interaction Lagrangian Tulin (2014)

$\mathcal{L}_{\rm{int}}=\left(\frac{1}{3}g_{B}+\varepsilon
Q_{q}e\right)\bar{q}\gamma^{\mu}qB_{\mu}-\varepsilon
e\bar{\ell}\gamma^{\mu}\ell B_{\mu}\,,$ (3)

where $B_{\mu}$ is the new gauge boson field, $g_{B}$ is the new gauge
coupling (considered here universal for all quarks $q$) and
$\alpha_{B}=g_{B}^{2}/4\pi$ is the fine structure associated to the baryonic
force. This interaction preserves the low-energy symmetries of QCD, i.e.,
charge conjugation $(C)$, parity $(P)$, and $T$ invariance, as well as isospin
and $SU(3)$ flavor symmetry. In the MeV–GeV mass range, $m_{\pi^{0}}\lesssim
m_{B}\lesssim 1$ GeV, the $B$ boson decays predominantly to $\pi^{0}\gamma$,
or to $\pi^{0}\pi^{+}\pi^{-}$ when kinematically allowed, very much like the
$\omega$ meson; in fact, the $\omega$ quantum numbers,
$I^{G}(J^{PC})=0^{-}(1^{--})$, can be assigned to the $B$ boson. In addition,
the Lagrangian in Eq. (3) is not completely decoupled from leptons as it
contains subleading photon-like couplings to fermions proportional to
$\varepsilon=eg_{B}/(4\pi)^{2}$. This effect allows the purely leptonic decay
$B\to e^{+}e^{-}$, which dominates below single pion threshold.

Rare $\eta$ and $\eta^{\prime}$ decays are specially suited to search for $B$
signatures in the MeV–GeV mass range. Here we concentrate first on the doubly
radiative decays $\eta^{(\prime)}\to\pi^{0}\gamma\gamma$ and
$\eta^{\prime}\to\eta\gamma\gamma$. The current layout of REDTOP is not
sensitive to completly neutral final states. However the discovery potential
offered by these processes is very promising and that could strengthen the
case for an upgrade of the experiment. In fact, an improved version of REDTOP
is planned, with the $\eta$ being tagged and in which final states with
$\gamma$’s and $\pi^{0}$’s could be detected. In these decays, the new boson
would appear as an intermediate state resonance in the decay chain
$\eta^{(\prime)}\to B\gamma\to\pi^{0}\gamma\gamma$, thus producing a peak at
around $m_{B}$ in the $\pi^{0}\gamma$ invariant mass spectrum.

This search requires both experimental precision as well as a robust SM
prediction. In Escribano et al. (2020), a VMD framework was used to describe
vector-meson exchange contributions to these decays, as well as the L$\sigma$M
to describe scalar-meson exchanges. In analogy to VMD, we now incorporate an
intermediate $B$ boson exchange contribution through the Feynman diagram
depicted on the left hand side of Fig. 2. This contribution can be derived
from the standard VMD vector-vector-pseudoscalar and vector-photon
Lagrangians, supplemented by an effective vector-$B$ boson vertex. The
standard VMD interaction Lagrangians are given by

$\displaystyle\mathcal{L}_{VVP}$ $\displaystyle=$
$\displaystyle\frac{G}{\sqrt{2}}\epsilon^{\mu\nu\alpha\beta}{\rm{tr}}\left[\partial_{\mu}V_{\nu}\partial_{\alpha}V_{\beta}P\right]\,,$
$\displaystyle\mathcal{L}_{V\gamma}$ $\displaystyle=$
$\displaystyle-2egf_{\pi}^{2}A^{\mu}{\rm{tr}}\left[QV_{\mu}\right]\,,$ (4)

where $\epsilon_{\mu\nu\alpha\beta}$ is the totally antisymmetric Levi-Civita
tensor, $V^{\mu}$ and $P$ are the matrices for the vector and pseudoscalar
meson fields, $A^{\mu}$ is the photon field and,
$Q={\rm{diag}}\\{2/3,-1/3,-1/3\\}$ is the quark-charge matrix Bramon et al.
(1992).

Figure 2: Schematic diagram for the $B$ boson exchange mechanism to the decays
$\eta\to\pi^{0}\gamma\gamma$ (left) and $\eta\to\pi^{+}\pi^{-}\gamma$ (right).

The Lagrangian that describes the $VB$ interaction is formally identical to
the $V\gamma$ one in Eq. (4), with the substitutions $A^{\mu}\to B^{\mu}$,
$e\to g_{B}$ and $Q\to{\rm{diag}}\\{1/3,1/3,1/3\\}$. It is given by

$\displaystyle\mathcal{L}_{VB}$ $\displaystyle=$
$\displaystyle-2\frac{1}{3}g_{B}gf_{\pi}^{2}B^{\mu}{\rm{tr}}\left[V^{\mu}\right]\,.$
(5)

From the Lagrangians in Eqs. (4) and (5), it is straightforward to obtain
expressions for the $g_{BP\gamma}$ couplings in terms of $g_{B}$, which are
found to be:

$\displaystyle g_{B\pi^{0}\gamma}(q^{2})$ $\displaystyle=$
$\displaystyle\frac{\sqrt{2}eg_{B}}{4\pi^{2}f_{\pi}}\left(\frac{m_{\omega}^{2}}{m_{\omega}^{2}-q^{2}-im_{\omega}\Gamma_{\omega}}\right)\,,$
(6) $\displaystyle g_{B\eta\gamma}(q^{2})$ $\displaystyle=$
$\displaystyle\frac{eg_{B}}{12\pi^{2}f_{\pi}}\frac{1}{\sqrt{6}}\left[(c_{\theta}-\sqrt{2}s_{\theta})\left(\frac{m_{\omega}^{2}}{m_{\omega}^{2}-q^{2}-im_{\omega}\Gamma_{\omega}}\right)+(2c_{\theta}+\sqrt{2}s_{\theta})\left(\frac{m_{\phi}^{2}}{m_{\phi}^{2}-q^{2}-im_{\phi}\Gamma_{\phi}}\right)\right],$
(7) $\displaystyle g_{B\eta^{\prime}\gamma}(q^{2})$ $\displaystyle=$
$\displaystyle\frac{eg_{B}}{12\pi^{2}f_{\pi}}\frac{1}{\sqrt{6}}\left[(s_{\theta}+\sqrt{2}c_{\theta})\left(\frac{m_{\omega}^{2}}{m_{\omega}^{2}-q^{2}-im_{\omega}\Gamma_{\omega}}\right)+(2s_{\theta}-\sqrt{2}c_{\theta})\left(\frac{m_{\phi}^{2}}{m_{\phi}^{2}-q^{2}-im_{\phi}\Gamma_{\phi}}\right)\right],$
(8)

where $\theta$ is the $\eta$-$\eta^{\prime}$ mixing angle,
$s_{\theta}\equiv\sin\theta$, and $c_{\theta}\equiv\cos\theta$.

Combining the $g_{B\eta\gamma}$ and $g_{B\pi^{0}\gamma}$ couplings from Eqs.
(6) and (7) with the propagator of the $B$ boson, one can calculate the $B$
boson exchange contribution to the amplitude of the decay
$\eta\to\pi^{0}\gamma\gamma$. This is given by

$\displaystyle{\cal
A}^{B\,\mathrm{boson}}_{\eta\to\pi^{0}\gamma\gamma}=g_{B\eta\gamma}(t)g_{B\pi^{0}\gamma}(t)\left[\frac{(P\cdot
q_{2}-m_{\eta}^{2})\\{a\\}-\\{b\\}}{m_{B}^{2}-t-im_{B}\Gamma_{B}}+\bigg{\\{}\begin{array}[]{c}q_{2}\leftrightarrow
q_{1}\\\ t\leftrightarrow u\end{array}\bigg{\\}}\right]\ ,$ (11)

where $t,u=(P-q_{2,1})^{2}=m_{\eta}^{2}-2P\cdot q_{2,1}$ are the Mandelstam
variables and $\\{a\\}$ and $\\{b\\}$ are the Lorentz structures. These are
defined as

$\displaystyle\\{a\\}$
$\displaystyle=(\epsilon_{1}\cdot\epsilon_{2})(q_{1}\cdot
q_{2})-(\epsilon_{1}\cdot q_{2})(\epsilon_{2}\cdot q_{1})\ ,$ (12)
$\displaystyle\\{b\\}$ $\displaystyle=(\epsilon_{1}\cdot
q_{2})(\epsilon_{2}\cdot P)(P\cdot q_{1})+(\epsilon_{2}\cdot
q_{1})(\epsilon_{1}\cdot P)(P\cdot q_{2})$
$\displaystyle-(\epsilon_{1}\cdot\epsilon_{2})(P\cdot q_{1})(P\cdot
q_{2})-(\epsilon_{1}\cdot P)(\epsilon_{2}\cdot P)(q_{1}\cdot q_{2})\,,$

where $P$ is the four-momentum of the decaying $\eta$ meson, and
$\epsilon_{1,2}$ and $q_{1,2}$ are, respectively, the polarization and four-
momentum vectors of the final photons. The amplitudes for the decays
$\eta^{\prime}\to\pi^{0}\gamma\gamma$ and $\eta^{\prime}\to\eta\gamma\gamma$
have a similar structure to that of Eq. (11), with the replacements
$m_{\eta}^{2}\to m_{\eta^{\prime}}^{2}$, and
$g_{B\eta\gamma}g_{B\pi^{0}\gamma}\to
g_{B\eta^{\prime}\gamma}g_{B\pi^{0}\gamma}$ for the
$\eta^{\prime}\to\pi^{0}\gamma\gamma$ decay and
$g_{B\eta\gamma}g_{B\pi^{0}\gamma}\to g_{B\eta^{\prime}\gamma}g_{B\eta\gamma}$
for $\eta^{\prime}\to\eta\gamma\gamma$.

We also consider $B$ boson signals in the decay $\eta\to\pi^{+}\pi^{-}\gamma$.
In this case, the $B$ boson is produced as in the doubly radiative process
discussed above but it decays instead into $\pi^{+}\pi^{-}$ through
$\rho$-$\omega$ mixing as depicted in Fig. 2 (right diagram). This process is
suppressed since it depends on $\varepsilon$. We can write the amplitude for
the $\eta(q)\to\pi^{+}(p_{1})\pi^{-}(p_{2})\gamma(k)$ in terms of a scalar
function $\mathcal{F}(s,t,u)$ according to

${\cal
A}(s,t,u)=i\mathcal{F}(s,t,u)\varepsilon_{\mu\nu\alpha\beta}\epsilon^{\mu}(k)p_{1}^{\nu}p_{2}^{\alpha}q^{\beta}\,,$
(13)

with the Mandelstam variables $s=(q-p_{1})^{2}\,,t=(p_{1}+p_{2})^{2}$ and
$u=(q-p_{2})^{2}$. In $P$-wave approximation, one has
$\mathcal{F}(s,t,u)=F(t)$ Kubis and Plenter (2015), and the decay rate is
given by

$\Gamma(\eta\to\pi^{+}\pi^{-}\gamma)=\int_{4m_{\pi}^{2}}^{M_{\eta}^{2}}dt\,\frac{t\sigma^{3}_{\pi}(M_{\eta}^{2}-t)^{3}}{12(8\pi
M_{\eta})^{3}}|F(t)|^{2}\,,$ (14)

where $\sigma_{\pi}=\sqrt{1-4m_{\pi}^{2}/t}$. For a discussion of the Standard
Model amplitude $F(t)$, see Refs. Stollenwerk et al. (2012); Kubis and Plenter
(2015).

The $B$ boson exchange contribution to the form factor $F(t)$ in Eq. (14) can
be written as

$F(t)=g_{B\eta\gamma}(t)\frac{1}{m_{B}^{2}-t-im_{B}\Gamma_{B}}g_{B\pi\pi}(t)\,,$
(15)

where $g_{B\eta\gamma}(t)$ is given in Eq. (7) and $g_{B\pi\pi}(t)$ reads

$g_{B\pi\pi}(t)=\sqrt{4\pi\alpha_{{\rm{em}}}\varepsilon^{2}}|F_{\pi}(t)|\,.$
(16)

The pion form factor in Eq. (16) can, in turn, be expressed as

$F_{\pi}(t)=F_{\rho}(t)\left[1+\frac{1+\delta}{3}\frac{\tilde{\Pi}(t)}{t-m_{\omega}^{2}+im_{\omega}\Gamma_{\omega}}\right]\,,$
(17)

where $\delta=2g_{B}/(\varepsilon e)$ Tulin (2014) accounts for the
$B$-$\omega$ mixing and
$F_{\rho}(t)=m_{\rho}^{2}/(m_{\rho}^{2}-t-im_{\rho}\Gamma_{\rho})$ is the pion
form factor associated to the $\rho$ exchange only. For the $\rho$-$\omega$
mixing parameter in Eq. (17), we assume
$\tilde{\Pi}(t)=\tilde{\Pi}(m_{\omega}^{2})=-3500(300)\,\text{MeV}^{2}$ Tulin
(2014).

##### _Protophobic Fifth Force model_

Another interesting model to challenge has been proposed Feng et al. (2016,
2017) to explain a $6.8\sigma$ anomaly in the invariant mass distributions of
$e^{+}e^{-}$ pairs produced in 8Be discrete nuclear transitions Krasznahorkay
et al. (2016). The mass of such a gauge boson ($X_{17}$) is determined to be
about 17 MeV, which is below the sensitivity of WASA and KLOE, but accessible
to REDTOP thanks to the slight boost imparted to the $\eta$ meson in the lab
frame. The same fifth force would be able to reconcile the $3.6\sigma$
discrepancy between the predicted and measured values of the muon’s anomalous
magnetic moment Feng et al. (2016). In this respect, REDTOP will be a nice
complement to the $(g-2)_{\mu}$ experiment currently running at Fermilab
Miller et al. (2007).

The $X_{17}$ has been the subject of much experimental and theoretical study,
with the NA64 experiment at the CERN SPS, searching for $X_{17}\to e^{+}e^{-}$
decay, finding only negative results Banerjee et al. (2020), with a probe of
the remaining parameter space possible Depero et al. (2020). Evidence for
$X_{17}$ has also been observed in 4He decay Krasznahorkay et al. (2021), and
the quantum number selectivity associated with its emission from an excited
nuclear state supports its interpretation as a vector particle Krasznahorkay
et al. (2021); Feng et al. (2020). However, it has been argued $X_{17}$
production is dominated by a non-resonant process, obviating these
conclusions, with the nonobservation of $X_{17}$ via bremsstrahlung arguing
against a vector interpretation Zhang and Miller (2021). REDTOP can provide a
definitive test of this issue. Preliminary studies show that REDTOP has an
excellent sensitivity to this model via the production channel:
$\eta/\eta^{\prime}\rightarrow\gamma X_{17}\rightarrow\gamma e^{+}e^{-}$. This
sensitivity is mainly due to the high granularity ADRIANO2 calorimeter.
Particular to the protophobic model is the distinct couplings that the new
gauge boson possesses to $u$ and $d$ quarks. In this sense, the model
represents an explicit example of a generalized $B$ boson model.

We present and contrast the decay width for $\eta\to\gamma X$, where $X$ is a
generic gauge boson with couplings to $u$, $d$ and $s$ quarks, in two
calculation schemes. The first is derived from the traditional triangle
anomaly of Adler, Bell and Jackiw (ABJ) Adler (1969); Bell and Jackiw (1969)
where one photon leg has been replaced by an $X$. In this scheme, the ratio of
branching ratios of the $\eta$ to normal and dark photons is given by

$\left(\frac{BR_{\eta\to\gamma
X}}{BR_{\eta\to\gamma\gamma}}\right)_{\text{ABJ}}=\frac{1}{2\pi\alpha}\left(1-\frac{m_{X}^{2}}{m_{\eta}^{2}}\right)^{3}\left[\frac{(c_{\theta}-\sqrt{2}s_{\theta})(2\varepsilon_{u}-\varepsilon_{d})+(2c_{\theta}+\sqrt{2}s_{\theta})\varepsilon_{s}}{c_{\theta}-2\sqrt{2}s_{\theta}}\right]^{2},$
(18)

where $m_{X}$ is the mass of the new gauge boson; $\varepsilon_{u}$,
$\varepsilon_{d}$ and $\varepsilon_{s}$ are respectively the up-, down- and
strange-quark charges under the new interaction; $\theta\approx-19.5^{\circ}$
is the $\eta$–$\eta^{\prime}$ mixing angle, $c_{\theta}\equiv\cos\theta$, and
$s_{\theta}\equiv\sin\theta$.

Alternatively, the decay width can be calculated in the scheme of vector meson
dominance (VMD) Fujiwara et al. (1985), in which interactions of the
pseudoscalar meson octet are described in terms of a single interaction vertex
with the vector meson nonet. The vector mesons then mix kinetically with the
SM photon and the new boson $X$. The ratio of branching ratios then becomes

$\displaystyle\left(\dfrac{BR_{\eta\to\gamma
X}}{BR_{\eta\to\gamma\gamma}}\right)_{\text{VMD}}=\dfrac{1}{8\pi\alpha}\left(1-\dfrac{m_{X}^{2}}{m_{\eta}^{2}}\right)^{3}\times$
(19)
$\displaystyle\left|\dfrac{\left(c_{\theta}-\sqrt{2}s_{\theta}\right)\left[3(\varepsilon_{u}-\varepsilon_{d})F_{\rho}(m_{X}^{2})+(\varepsilon_{u}+\varepsilon_{d})F_{\omega}(m_{X}^{2})\right]+2\left(2c_{\theta}+\sqrt{2}s_{\theta}\right)\varepsilon_{s}F_{\phi}(m_{X}^{2})}{c_{\theta}-2\sqrt{2}s_{\theta}}\right|^{2},$

where $F_{V}(q^{2})$ is the vector meson form factor, given by

$F_{V}(q^{2})=\left(1-\frac{q^{2}}{m_{V}^{2}}-i\frac{\Gamma_{V}}{m_{V}}\right)^{-1},$
(20)

$m_{V}$ the vector meson mass and $\Gamma_{V}$ its corresponding total width.
In the limit $m_{X}\to 0$ and $\Gamma_{V}\to 0$, Eq. (19) is equivalent to Eq.
(18). The case of the leptophobic $B$ boson is recovered in the limit
$\varepsilon_{u},\,\varepsilon_{d},\,\varepsilon_{s}\to g_{B}/3$.

For the case of the protophobic gauge boson,
$2\varepsilon_{u}=-\varepsilon_{d}$; and we can define
$\varepsilon_{n}\equiv\varepsilon_{u}+2\varepsilon_{d}=-\varepsilon_{u}=\varepsilon_{d}/2$.
However, this model is not prescriptive regarding $\varepsilon_{s}$;
therefore, the branching ratio depends on three parameters, which we take to
be $\\{m_{X},\,\varepsilon_{n},\,\varepsilon_{s}/\varepsilon_{n}\\}$. Figure 3
shows the dependence of the ratio of branching ratios on
$\varepsilon_{s}/\varepsilon_{n}$ in the limit $m_{X}\to 0$. We note a
cancellation that occurs near $\varepsilon_{s}/\varepsilon_{n}\approx 4$; this
cancellation is perfect in the ABJ scheme, but the vector boson widths
(particularly $\Gamma_{\rho}$) prevent this from being identically zero in the
VMD scheme, even if $m_{X}$ vanishes.

Figure 3: The dependence of $BR_{\eta\to\gamma X}/BR_{\eta\to\gamma\gamma}$ on
the ratio of the strange-quark and neutron charges,
$\varepsilon_{s}/\varepsilon_{n}$, in the protophobic gauge boson model in the
limit $m_{X}\to 0$. The relative strengths of the interactions have been
divided out.

####  III.1.2 Scalar portal models

In the so called scalar or Higgs portal, the dark sector couples to the SM via
the Higgs boson or an extension of the latter. Dark scalars _S_ can be
explored in REDTOP via $\eta\rightarrow S\rightarrow\pi^{0}e^{+}e^{-}$ and
$\eta\rightarrow\pi^{0}S\rightarrow\pi^{0}\mu^{+}\mu^{-}$processes. Three
complementary models are currently under consideration by REDTOP: the _Minimal
dark scalar model, Spontaneous Flavor Violation model_ or _Flavor-Specific
Scalar model_ (which have similar REDTOP phenomenology), and the _Two-Higgs
doublet model._ In the former, the dark scalar mimics a light Higgs and,
consequently, it couples prevalently to heavy quarks. The latter has larger
coupling, instead, to light quarks. The predicted branching ratios differ by
more than two orders of magnitude.

##### _Minimal scalar model_

The minimal scalar portal model operates with one extra singlet field $S$ and
two types of couplings, $\mu$ and $\lambda$. The mechanism for the
$\eta\to\pi^{0}\mu^{+}\mu^{-}$ (or $e^{+}e^{-}$) decay is usually described
via a 2-photon intermediate state to conserve $C$:
$\eta\to\pi^{0}\gamma\gamma$ along with $\gamma\gamma\to\mu\bar{\mu}$ via a
triangle diagram. Branching ratios are calculated to be of the order of
$10^{-9}$ Ng and Peters (1992, 1993); Shabalin (2002); Escribano and Royo
(2020), which should be well within the sensitivity of REDTOP. Preliminary
sensitivity studies on the _Minimal scalar model_ have been performed as part
of CERN’s “Physics Beyond Collider” program et al. (2019) indicating that
REDTOP has modest sensitivity to $\mu$ and $\lambda$, as it should be expected
from the low quark content of the $\eta$/$\eta^{\prime}$ mesons. New studies,
based on a much better performing detector, also capable of identifying a
detached vertex, are in progress.

##### _Spontaneous Flavor Violation model_

The limitations in the REDTOP reach to the minimal scalar model are due to the
smallness of the scalar’s couplings to the $\eta$ and $\eta^{\prime}$ mesons.
In this minimal model, the scalar couples preferentially to the third
generation, while couplings to the light quarks that compose the $\eta$ and
$\eta^{\prime}$ are suppressed. A variety of beyond the Standard Model
theories share this feature, mostly as a consequence of imposing that the new-
physics flavored interactions follow the Standard Model flavor hierarchies in
order to avoid stringent bounds from flavor-changing neutral currents (FCNCs)
D’Ambrosio et al. (2002). In a recent paper Egana-Ugrinovic et al. (2019a),
however, it was demonstrated that a novel flavor mechanism called Spontaneous
Flavor Violation (SFV) allows to construct natural and well-motivated BSM
models, where New Physics may couple preferentially to light quarks while
avoiding bounds from FCNCs. Models with light or heavy scalars based on SFV
can be explored both at low-energy experiments Egana-Ugrinovic et al. (2020);
Batell et al. (2019) and the LHC Egana-Ugrinovic et al. (2019b, 2021). These
two approaches are complimentary, as models containing sub-GeV scalars that
couple to light quarks require UV completions that include heavier states,
which themselves have sizeable couplings to light quarks.

The implications of finding New Physics with novel flavor hierarchies would
have profound consequences for our understanding of the flavor sector.
Moreover, such New Physics could be important for other fundamental issues,
such as the dark matter problem. An excellent example that illustrates the
broad relevance of looking for extra scalars with novel flavor hierarchies was
presented by Batell et al. in Batell et al. (2019). In this work, it was shown
that a light scalar (order 1 GeV and below) coupling preferentially to light
quarks may serve as a mediator to the dark sector. Batell et al. found that
REDTOP would have unique discovery potential in the mass range $m_{K}\lesssim
m_{S}\lesssim m_{\eta^{\prime}}$, with $m_{S}$ being the mass of the new
scalar. The complementarity with the LHC and the implications for flavor
physics were independently explored in Egana-Ugrinovic et al. Egana-Ugrinovic
et al. (2019b), where the phenomenology of scalars with masses of the order of
hundreds of GeV (that arise in UV completions of the model presented in Batell
et al. (2019)) was explored.

These phenomenological studies have shown the need to further explore New
Physics with preferential couplings to light quarks. REDTOP represents an
exquisite opportunity to look for such models.

##### _Flavor-Specific and Hadrophilic Scalars_

As motivated above, it is of interest to explore new scalars with couplings
patterns that are qualitatively distinct to those in the Higgs portal model. A
general effective field theory investigation of scalar mediators with flavor-
specific interactions, i.e., a scalar coupling dominantly to a single SM
fermion mass eigenstate, was initiated in Ref. Batell et al. (2018).
Subsequent work in Ref. Batell et al. (2019) explored the phenomenology of a
concrete scenario in which a scalar with hadrophilic couplings mediated
interactions with dark matter. Furthermore, Ref. Batell et al. (2021)
investigated simple renormalizable models of flavor-specific scalars involving
a vector-like fermion or scalar doublet, focusing on the complementarity
between low and high energy observables.

As discussed in Ref. Batell et al. (2019), REDTOP has excellent prospects to
probe scalars that couple dominantly to first generation quarks. Here we will
consider a coupling of a scalar $S$ to up quarks. The low energy Lagrangian is
given by

${\cal
L}\supset\frac{1}{2}(\partial_{\mu}S)^{2}-\frac{1}{2}m_{S}^{2}S^{2}-g_{u}\,S\,\overline{u}\,u,$
(21)

where $m_{S}$ is the scalar mass and $g_{u}$ is the effective coupling of the
scalar to up quarks. This model faces strong constraints from cosmology,
astrophysics, beam dumps, and past $\eta$ decay searches for $m_{S}<2m_{\pi}$.
However, the constraints are significantly weaker if the scalar mass is above
than the two pion threshold. For REDTOP, this singles out
$2m_{\pi}<m_{S}<m_{\eta}-m_{\pi}$ as the mass range of interest in this model.
The scalar will be produced at REDTOP via $\eta\rightarrow\pi^{0}S$, with
branching ratio

$\displaystyle{\rm Br}(\eta\rightarrow\pi^{0}S)$ $\displaystyle=$
$\displaystyle\frac{c_{S\pi^{0}\eta}^{2}g_{u}^{2}B^{2}}{16\pi
m_{\eta}\Gamma_{\eta}}\lambda^{1/2}\left(1,\frac{m_{S}^{2}}{m_{\eta}^{2}},\frac{m_{\pi^{0}}^{2}}{m_{\eta}^{2}}\right),$
(22)

where $\lambda(a,b,c)=a^{2}+b^{2}+c^{2}-2ab-2ac-2bc$, $B\simeq
m_{\pi}^{2}/(m_{u}+m_{d})\approx 2.6$ GeV, and the coefficients
$c_{S\pi^{0}\eta}=\frac{1}{\sqrt{3}}\cos\theta-\sqrt{\frac{2}{3}}\sin\theta$
parametrize the effects of $\eta-\eta^{\prime}$ mixing, with
$\theta\approx-20^{\circ}$. Once produced, the scalar will decay promptly via
$S\rightarrow\pi^{+}\pi^{-}$, leading to the final state
$\pi^{0}\pi^{+}\pi^{-}$. In Fig. 4 we show the sensitivity of REDTOP to
scalars $S$ in this channel, using the results of the
$\pi^{0}(\pi^{+}\pi^{-})$ bump hunt analysis presented in Section X.5 for the
three mass points $m_{S}={300,350,400}$ MeV. As can be seen in the plot,
REDTOP has the potential to significantly extend the reach in this mass range
beyond the limits from the KLOE experiment. Furthermore, one observes an
interesting complementarity with future long-lived particle searches at FASER
Kling and Trojanowski (2021), FASER2 Kling and Trojanowski (2021) and SHiP
Batell et al. (2019), which will probe longer lifetimes and smaller couplings.

Figure 4: REDTOP sensitivity to the the hadrophilic scalar model, Eq. (21), in
the $\eta\rightarrow\pi^{0}S$, $S\rightarrow\pi^{+}\pi^{-}$ mode. We display
the projected sensitivity of REDTOP (red) in the $m_{S}-g_{u}$ plane to the
bump hunt analysis presented in Section X.5 based on $3.3\times{10}^{18}$ POT
for the three mass points $m_{S}={300,350,400}$ MeV. Also shown are various
existing constraints and projections from other planned or proposed
experiments; see Refs. (Batell et al., 2019; Kling and Trojanowski, 2021) for
further details.

##### _Two-Higgs doublet model_

This model Abdallah et al. (2021) is one of the simplest possible extensions
of the SM, assuming the existence of a second Higgs doublet $H$ and a dark
singlet real scalar $h^{\prime}$. It was initially introduced to explain the
anomalies observed by LSND, MiniBooNE and muon g−2 experiment, and it is being
extended to the decays of the $\eta\rightarrow\pi^{0}h^{\prime}/H$ and
$\eta^{\prime}\rightarrow\pi^{0}h^{\prime}/H$. Preliminary calculations
indicate that BR($\eta\rightarrow\pi^{0}h^{\prime}$)$\sim$10-9 while
BR($\eta^{\prime}\rightarrow\pi^{0}H$)$\sim$10-10, in both cases within REDTOP
sensitivity.

A scalar $H$ could be observed in an $\eta/\eta^{\prime}$ final state in
association with a $\pi^{0}$ by detecting the following processes:

$\displaystyle\eta\rightarrow\pi^{0}H\rightarrow\pi^{0}+l^{+}l^{-},$ (23)
$\displaystyle\eta\rightarrow\pi^{0}H\rightarrow\pi^{0}+\pi^{+}\pi^{-}.$ (24)

Within the SM, process (23) can only occur via a two-photon exchange diagram
with a branching ratio of the order of $10^{-9}$. If such a light particle
exists, even with a mass larger than the $\eta/\eta^{\prime}$ meson, which
couples the leptons to the quarks, the probability for this process could be
increased by several orders of magnitude, changing dramatically the dynamics
of the process. Two groups of theoretical models postulating a BSM light
scalar are receiving great attention lately: the Minimal Extension of the
Standard Model Scalar Sector (O’Connell et al., 2007a; Krnjaic, 2016) and the
models containing Higgs bosons with large couplings to light quarks (Batell et
al., 2019; Egana-Ugrinovic et al., 2019b). From the experimental point of
view, these models are complementary: the former predicting large coupling to
the $b$-quark and to gluons but a small one to the light quarks, while the
latter predicts a large coupling to light quarks. An observation at an
$\eta/\eta^{\prime}$ factory of the process (23) would be an indication that
the second set of models would be the most likely extension to the SM. Vice-
versa, an observation of a scalar at a $B$-factory but not at REDTOP would
favor the first group of models.

A detailed simulation of the process (23) and of the foreseen background,
including many instrumental effects, has been performed by the Collaboration
within the “Physics Beyond Collider” program (Alemany et al., 2019) assuming
the Minimal Extension of the Standard Model Scalar model.

An integrated beam flux of $10^{12}$ POT (as available at CERN, see Sec. XVII
below) has been assumed. A simple “ _bump-hunt_ ” analysis was performed,
looking at the invariant mass of di-leptons associated to a prompt photon. The
sensitivity to the $\sin^{2}\theta$ parameter (O’Connell et al., 2007a;
Krnjaic, 2016) is shown in Fig. 4. The largest contributing background was
found to be from the 3-body decay $\eta\rightarrow\gamma+l^{+}l^{-}$ where an
extra $\gamma$ fakes a $\pi^{0}$ in the final state.

A preliminary sensitivity analysis for several experiments (including REDTOP)
based on the second set of models can be found in Ref. (Batell et al., 2019)

LSND and MiniBooNE (MB) observed an excess of electron-like events over the
expected background. These excesses can be understood in the context of the
decay of a heavy neutral lepton ($N_{2}$) produced via up-scattering of the
beam $\nu_{\mu}$ in the detector Abdallah et al. (2021). Once $N_{2}$ is
produced, it decays instantaneously to another neutral lepton ($N_{1}$) and a
light scalar ($h^{\prime}$). The $h^{\prime}$ subsequently decays promptly to
a collimated $e^{+}e^{-}$ pair producing a signal in the detectors. The
solution is set in the context of a two-Higgs doublet model (2HDM)
$(\phi_{h},\phi_{H})$ and a dark singlet real scalar $\phi_{h^{\prime}}$.
After diagonalization of the mass matrix, one SM-like higgs ($h$) and two
other light scalars ($h^{\prime},H$) have been obtained. In this model, only
the $\phi_{h}$ acquires a vacuum expectation value (VEV). Hence, the coupling
of $\phi_{H}$ with fermions is an independent parameter. The model has three
right-handed neutrinos $N_{i}$ ($i=1,2,3$) which are responsible for producing
light neutrino masses via a type-I seesaw. Two of the heavy neutral leptons
participate in the production of excess events in LSND and MB, as mentioned
above. The model also resolves the discrepancy between theory and experiment
in the anomalous magnetic moment of muon via the contributions of the light
scalars ($h^{\prime},H$). Details of the model are given in Abdallah et al.
(2021). The benchmark point (BP) of the parameters is shown in Table 1.

$m_{N_{1}}$ | $m_{N_{2}}$ | $m_{N_{3}}$ | $y_{u}^{h^{\prime}(H)}\\!\\!\times\\!\\!10^{6}$ | $y_{e(\mu)}^{h^{\prime}}\\!\\!\times\\!\\!10^{4}$ | $y_{e(\mu)}^{H}\\!\\!\times\\!\\!10^{4}$
---|---|---|---|---|---
85 MeV |  $130$ MeV |  $10$ GeV | $0.8(8)$ | $0.23(1.6)$ | $2.29(15.9)$
$m_{h^{\prime}}$ | $m_{H}$ | $\sin\delta$ | $y_{d}^{h^{\prime}(H)}\\!\\!\times\\!\\!10^{6}$ | $y_{\nu_{\\!i2}}^{h^{\prime}(H)}\\!\\!\times\\!\\!10^{3}$ | $\lambda_{N_{\\!12}}^{h^{\prime}(H)}\\!\\!\times\\!\\!10^{3}$
17 MeV | 750 MeV | $0.1$ | $0.8(8)$ | $1.25(12.4)$ | $74.6(-7.5)$
Table 1: Benchmark point for LSND and MB events, and muon $g-2$ calculation.

The light scalars ($h^{\prime},H$) couple to $u$ and $d$ quarks. Hence these
scalars could be probed via the decay channels of $\eta\,(\eta^{\prime})$ in
REDTOP. The decay amplitudes of $\eta,\eta^{\prime}\to\pi^{0}S$ and
$\eta^{\prime}\to\eta S$, where $S$ refers to the light scalars
$h^{\prime},H$, are Gan et al. (2022):

$\displaystyle\mathcal{A}(\eta^{(\prime)}\to\pi^{0}S)$
$\displaystyle=-B_{0}\left[(\lambda_{u}-\lambda_{d})\Gamma^{u-d}_{\pi\eta^{(\prime)}}(m_{S}^{2})+(\lambda_{u}+\lambda_{d})\Gamma^{u+d}_{\pi\eta^{(\prime)}}(m_{S}^{2})+\lambda_{s}\Gamma^{s}_{\pi\eta^{(\prime)}}(m_{S}^{2})\right],$
(25) $\displaystyle\mathcal{A}(\eta^{\prime}\to\eta S)$
$\displaystyle=-B_{0}\left\\{(\lambda_{u}+\lambda_{d})\Gamma^{u+d}_{\eta\eta^{(\prime)}}(m_{S}^{2})+\lambda_{s}\Gamma^{s}_{\eta\eta^{(\prime)}}(m_{S}^{2})\right\\}.$
(26)

The values of $B_{0}$ and other form factor are given in Gan et al. (2022). In
this model $S$ couples exclusively to both $u$ and $d$ quarks, therefore the
partial widths of $\eta,\eta^{\prime}\to\pi^{0}S$ and $\eta^{\prime}\to\eta S$
are Gan et al. (2022) given by:

$\displaystyle\Gamma\big{(}\eta^{(\prime)}\to\pi^{0}S\big{)}$
$\displaystyle=\frac{|\mathcal{A}(\eta^{(\prime)}\to\pi^{0}S)|^{2}}{16\pi
m_{\eta^{(\prime)}}}\lambda^{1/2}\left(1,\frac{m_{\pi^{0}}^{2}}{m_{\eta^{(\prime)}}^{2}},\frac{m_{S}^{2}}{m_{\eta^{(\prime)}}^{2}}\right)\,,$
(27) $\displaystyle\Gamma\big{(}\eta^{\prime}\to\eta S\big{)}$
$\displaystyle=\frac{|\mathcal{A}(\eta^{\prime}\to\eta S)|^{2}}{16\pi
m_{\eta^{(\prime)}}}\lambda^{1/2}\left(1,\frac{m_{\pi^{0}}^{2}}{m_{\eta^{(\prime)}}^{2}},\frac{m_{S}^{2}}{m_{\eta^{(\prime)}}^{2}}\right)\,,$
(28)

where $\lambda(a,b,c)=a^{2}+b^{2}+c^{2}-2(ab+ac+bc)$.

For the benchmark parameter values, as shown in Table 1, where
$\lambda^{h^{\prime}(H)}_{q}=y_{q}^{h^{\prime}(H)},\leavevmode\nobreak\
q=u,d$, and $\lambda^{h^{\prime}(H)}_{s}=0$, the partial decay widths and the
branching ratios of the decay modes of $\eta$ and $\eta^{\prime}$ are
presented in Tables 2, 3.

|  $\Gamma(\eta\to\pi^{0}S)$ [GeV] | ${\rm BR}(\eta\to\pi^{0}S)$ |  $\Gamma(\eta^{\prime}\to\pi^{0}S)$ [GeV] | ${\rm BR}(\eta^{\prime}\to\pi^{0}S)$
---|---|---|---|---
$S=h^{\prime}$ | $1.59\times 10^{-17}$ | $1.22\times 10^{-11}$ | $1.91\times 10^{-17}$ | $1.01\times 10^{-13}$
$S=H$ | $0$ | $0$ | $5.70\times 10^{-16}$ | $3.03\times 10^{-12}$
Table 2: The partial decay widths and the branching ratios of the decay modes $\eta^{(\prime)}\to\pi^{0}S$, where $S=h^{\prime},H$.  |  $\Gamma(\eta^{\prime}\to\eta S)$ [GeV] | ${\rm BR}(\eta^{\prime}\to\eta S)$
---|---|---
$S=h^{\prime}$ | $4.54\times 10^{-14}$ | $2.41\times 10^{-10}$
$S=H$ | $0$ | $0$
Table 3: The partial decay width and the branching ratio of the decay mode
$\eta^{\prime}\to\eta S$.

The $\pi^{0}S$ production is proportional to $(\lambda_{u}-\lambda_{d})^{2}$
which equals to zero for this BP, due to
$\lambda^{h^{\prime}(H)}_{u}=\lambda^{h^{\prime}(H)}_{d}$. This assumption of
equal couplings to $u$ and $d$ quarks was made for simplicity. However, we now
consider a case different from the BP in Table 1 which assumes unequal
couplings, bringing the dominant isovector term into play, but still keeps the
LSND and MB fits intact and also leaves the muon $g-2$ calculation unaltered.
We assume $\lambda^{h^{\prime}(H)}_{u}\times 10^{6}=0.86(8.6)$ and
$\lambda^{h^{\prime}(H)}_{d}\times 10^{6}=0.74(7.4)$, i.e.,
$(\lambda^{h^{\prime}(H)}_{u}-\lambda^{h^{\prime}(H)}_{d})\times
10^{6}=0.12(1.2)$ and $\lambda^{h^{\prime}(H)}_{s}=0$. For this case, we have
calculated the branching ratios of the decay modes of
$\eta^{(\prime)}\to\pi^{0}S$ which are shown in Table 4. In Fig. 5, we show
the branching ratios of the decay modes $\eta^{(\prime)}\to\pi^{0}S$ as a
function of $(\lambda_{u}-\lambda_{d})^{2}$.

| ${\rm BR}(\eta\to\pi^{0}S)$ | ${\rm BR}(\eta^{\prime}\to\pi^{0}S)$
---|---|---
$S=h^{\prime}$ | $1.18\times 10^{-9}$ | $4.17\times 10^{-12}$
$S=H$ | $0$ | $1.25\times 10^{-10}$
Table 4: The branching ratios of the decay modes $\eta^{(\prime)}\to\pi^{0}S$,
where $S=h^{\prime},H$. Figure 5: The branching ratios of the decay modes
$\eta^{(\prime)}\to\pi^{0}S$ as a function of $(\lambda_{u}-\lambda_{d})^{2}$.

####  III.1.3 Heavy neutral lepton portal models

This portal operates with one or several dark heavy neutral leptons (HNLs).
Among the several models existing under this portal, the _Two-Higgs doublet
model_ is the only one considered, at present, by REDTOP. The process, in this
case, would be: $\eta/\eta^{\prime}\rightarrow\pi^{0}H$ with $H\rightarrow\nu
N_{2}$ and $N_{2}\rightarrow h^{\prime}N_{1}$ followed by
$h^{\prime}\rightarrow e^{+}e^{-}$. The process is identified by the presence
of a $\pi^{0}$ and an $e^{+}e^{-}$ pair in the final state and a peak in the
missing mass of the $\eta/\eta^{\prime}$ spectrum.

_Two-Higgs doublet model_

The 2HDM discussed in Sec. III.1.2 introduces, besides the $H$ and
$h^{\prime}$ scalars, also two heavy neutral leptons, $N_{1}$ and $N_{2}$
which represent invisible components of the HNL portal. This portal could be
explored by studying the process
$p+Li\rightarrow\eta+X\;$with$\;\eta\rightarrow\pi^{0}H\;;\;H\rightarrow\nu
N_{2}\;;\;N_{2}\rightarrow N_{1}h^{\prime}\;;\;h^{\prime}\rightarrow
e^{+}e^{-},$ which is, however, particularly challenging at REDTOP, because
the decay chains contains two undetected particles: $N_{1}$ and $N_{2}$. The
following points may be relevant when considering detectability of the
(visible) scalars in REDTOP:

  * •

In this model Abdallah et al. (2021), the branching ratios of the decay modes
of $H$, $h^{\prime}$, and $N_{2}$ for the BP in Table 1 are as follows:

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!{\rm
BR}(H\rightarrow\nu_{i}N_{2})=89.36\%,\leavevmode\nobreak\ {\rm
BR}(H\rightarrow N_{1}N_{2})=10.17\%,\leavevmode\nobreak\ {\rm
BR}(H\rightarrow\mu^{+}\mu^{-})=0.46\%,$ (29)
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!{\rm
BR}(H\rightarrow e^{+}e^{-})=0.011\%,\leavevmode\nobreak\ \leavevmode\nobreak\
\\!{\rm BR}(H\rightarrow\gamma\gamma)=3.54\times 10^{-6}\%,$ (30)
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!{\rm
BR}(h^{\prime}\rightarrow e^{+}e^{-})=99.9998\%,\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ {\rm BR}(h^{\prime}\rightarrow\gamma\gamma)=2\times
10^{-4}\%,$ (31)
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!{\rm
BR}(N_{2}\rightarrow N_{1}h^{\prime})=99.94\%,\leavevmode\nobreak\
\leavevmode\nobreak\ {\rm
BR}(N_{2}\rightarrow\nu_{i}h^{\prime})=0.06\%.\leavevmode\nobreak\
\leavevmode\nobreak\ $ (32)

  * •

It is assumed that $N_{1}$ could decay to invisible particles and also there
is a possibility to tune the parameters such that $N_{1}$ becomes a long-lived
particle.

  * •

Thus, an $H$, when produced in the detector, decays with $89.36\%$ BR to
$N_{2}$ and an active SM neutrino. The former, in turn will promptly decay to
an $N_{1}$ and $h^{\prime}$, leading to a visible $e^{+}e^{-}$ pair with
missing energy.

  * •

For the BP in Table 1, the total decay width of $H$, $h^{\prime}$ and $N_{2}$
are $1.45\times 10^{-5}$ GeV, $3.56\times 10^{-13}$ GeV, and $2.07\times
10^{-5}$ GeV, respectively. Hence all the particles will decay inside the
REDTOP.

####  III.1.4 The visible QCD axion

The original incarnation of the QCD axion (the so-called ‘PQWW’ axion Peccei
and Quinn (1977b, c); Weinberg (1978b); Wilczek (1978)) was a simple Two-
Higgs-Doublet Model (2HDM) with a common breaking mechanism for the
Electroweak and PQ symmetries. By the late 80s, the parameter space of the
PQWW axion was fully excluded by searches for axionic production in hadronic
decays and beam dump experiments. Still, many variants of the original QCD
axion have been explored (see, e.g., Bardeen et al. (1987)), and more
recently, Alves and Weiner have shown that a variant of the QCD axion with
$m_{a}\sim\mathcal{O}(1-10)$ MeV and $f_{a}\sim\mathcal{O}(1-10)$ GeV remains
viable Alves and Weiner (2018).

In this variant, the PQ mechanism is implemented by new dynamics at the GeV
scale coupling predominantly to the first generation of Standard Model
fermions. The resulting QCD axion is short-lived and decays to $e^{+}e^{-}$
with a lifetime $\tau_{a}\lesssim 10^{-13}$ s Andreev et al. (2021), avoiding
constraints from beam dumps and fixed target experiments, as well as from
upper bounds on rare meson/quarkonium decays to a long-lived axion that
escapes detection. Furthermore, this axion couples to the first generation of
SM quarks with a special relation between the ratios of light quark masses and
their PQ charges, namely,

$\frac{m_{u}}{m_{d}}\leavevmode\nobreak\ \simeq\leavevmode\nobreak\
\frac{Q^{{}_{\text{PQ}}}_{d}}{Q^{{}_{\text{PQ}}}_{u}}\leavevmode\nobreak\
=\leavevmode\nobreak\ \frac{1}{2}\;.$

This relation causes an accidental cancellation of the leading order $\chi$PT
contribution to axion-pion mixing,

$\theta_{a\pi}\leavevmode\nobreak\ \leavevmode\nobreak\ =\leavevmode\nobreak\
\leavevmode\nobreak\ -\leavevmode\nobreak\
\frac{f_{\pi}}{f_{a}}\,\left[\frac{(m_{u}\,Q^{{}_{\text{PQ}}}_{u}-m_{d}\,Q^{{}_{\text{PQ}}}_{d})}{m_{u}+m_{d}}\leavevmode\nobreak\
+\leavevmode\nobreak\
\mathcal{O}\left(\frac{m_{ud}}{m_{s}}\right)\right]\leavevmode\nobreak\
\leavevmode\nobreak\ \simeq\leavevmode\nobreak\ \leavevmode\nobreak\
\frac{\leavevmode\nobreak\ (-0.2\pm 30)\times 10^{-4}\leavevmode\nobreak\
}{(f_{a}/\text{GeV)}}\,,$ (33)

which results in a QCD axion with suppressed isovector couplings. This
piophobic axion is therefore safe from bounds from $\pi^{+}\rightarrow
a\,e^{+}\nu_{e}$ mediated via $a-\pi^{0}$ mixing. In addition, bounds from
$K^{+}\rightarrow\pi^{+}a$, previously believed to be severe, were shown to
suffer from large hadronic uncertainties that preclude the exclusion of
significant portions of the piophobic axion parameter space.

There are several motivations for searching for such an axion Alves (2021):

  * •

First and foremost, the QCD axion is tied to the solution of the strong CP
problem, which is one of the most significant puzzles in theoretical physics.

  * •

In addition, a piophobic QCD axion with mass of $\sim 16-17$ MeV could explain
the recent anomalies in isoscalar magnetic transitions of ${}^{8}\text{Be}$
and ${}^{4}\text{He}$ nuclei Krasznahorkay et al. (2016, 2019), while
simultaneously explaining the absence of anomalous emissions in isovector
and/or electric radiative nuclear processes Zhang and Miller (2021).

  * •

Finally, the suppressed axion-pion mixing in (33) could explain the anomalous
rate for $\pi^{0}\to e^{+}e^{-}$ measured by the KTeV collaboration Abouzaid
et al. (2007).

Because the QCD axion couples to quarks and/or gluons, it should invariably
lead to new signals in $\eta^{(\prime)}$ decays. The simplest such signal is a
contribution to $\eta^{(\prime)}\to e^{+}e^{-}$ due to $a-\eta^{(\prime)}$
mixing. Since the SM expectation for $\text{Br}\big{(}\eta^{(\prime)}\to
e^{+}e^{-}\big{)}$ is still two orders of magnitude below current experimental
sensitivity, there is significant room for new physics contributions to this
final state. REDTOP should have an expected sensitivity to
$\text{Br}\big{(}\eta\to e^{+}e^{-}\big{)}$ with $\mathcal{O}(10^{-10})$
precision, and to $\text{Br}\big{(}\eta^{\prime}\to e^{+}e^{-}\big{)}$ with
$\mathcal{O}(10^{-9})$ precision, which should lead to a sensitivity to
$a-\eta^{(\prime)}$ mixing angles of order
$\theta_{a\eta}\sim\mathcal{O}(10^{-4})$ and
$\theta_{a\eta^{\prime}}\sim\mathcal{O}(10^{-2})$, respectively (see Alves
(2021), Sec. VIII, and Table 34).

Firmer evidence for the QCD axion, however, would come as an observation of
axio-hadronic $\eta^{(\prime)}\to\pi\pi a$ decays. In fact, a new pseudoscalar
particle appearing in $\eta^{(\prime)}$ decays would only be possible if it
coupled to quarks and/or gluons. As such, it would affect the QCD topological
vacuum by either (i) contributing to or (ii) canceling the strong CP phase
$\theta_{\text{QCD}}$. In case (i), such pseudoscalar would be characterized
as a generic ‘axion-like particle’ (ALP), albeit this would be an extremely ad
hoc and fine-tuned scenario since another BSM mechanism would then have to be
concocted to cancel the ALP’s contribution to $\theta_{\text{QCD}}$.
Therefore, option (ii) would be a more compelling interpretation for such an
observation, namely, that a new light pseudoscalar $a$ appearing in
$\eta^{(\prime)}\to\pi\pi a$ must be the QCD axion which dynamically relaxes
$\theta_{\text{QCD}}$ to zero and solves the strong CP problem.

Figure 6: Contributions to the amplitude $\mathcal{A}(\eta^{(\prime)}\to\pi\pi
a)$ in the framework of Resonance Chiral Theory. Left graph: leading order
quartic term. Middle and right graphs: exchange of low-lying scalar
resonances. The dimensionless couplings $\widehat{c}_{d}$ and
$\widehat{c}_{m}$ are expected to deviate from the large-$N_{c}$ limit of
$|\widehat{c}_{d}|=|\widehat{c}_{m}|=1$ by $\mathcal{O}(10\%)$.

Low-energy strong interaction effects, however, introduce large uncertainties
to the calculation of axio-hadronic $\eta$ and $\eta^{\prime}$ decays. In
Alves (2021), this calculation was performed in the framework of Resonance
Chiral Theory (R$\chi$T), an interpolating effective theory between the low-
energy $\chi$PT framework and the microscopic QCD description. R$\chi$T
encodes the most prominent features of nonperturbative strong dynamics by
incorporating the low-lying QCD resonances and extending the principle of
vector meson dominance Ecker et al. (1989). In this framework, exchanges of
$0^{++}$ resonances such as the $a_{0}(980)$ and $f_{0}(980)$ contribute to
the amplitude for $\eta^{(\prime)}\to\pi\pi a$, see Fig. 6. These
contributions are of the same order of magnitude as the $\chi$PT quartic
coupling contribution, and significant destructive interference between these
amplitudes can take place within the expected range of couplings between
$a_{0}(980)$, $f_{0}(980)$ and $\pi$, $a$, $\eta^{(\prime)}$. The resulting
branching ratios for axio-hadronic $\eta^{(\prime)}$ decays can vary
significantly depending on the degree of destructive interference between
amplitudes. For the case of the piophobic QCD axion explaining the
${}^{8}\text{Be}$ and ${}^{4}\text{He}$ anomalies, the branching ratios for
$\eta^{(\prime)}\to\pi\pi a$ are shown in Fig. 7 as a function
$\widehat{c}_{d}$ and $\widehat{c}_{m}$, which parametrize the couplings
between the octet scalars and the chiral mesons. Previous
$\eta^{(\prime)}$-decay searches in $\pi\pi e^{+}e^{-}$ final states have not
been able to probe this scenario. In fact, an excess of 27 events (vs 7.7
expected) in that mass region was observed by CELSIUS/WASA in the
$\eta\rightarrow\pi^{+}\pi^{-}e^{+}e^{-}$ process Bargholtz et al. (2007). A
similar excess was also observed by BESIII in the
$\eta^{\prime}\rightarrow\pi^{+}\pi^{-}e^{+}e^{-}$ process Ablikim et al.
(2013), although it was dismissed as background from $\gamma$-conversion.
Despite the two-orders-of-magnitude variation in these branching ratios, they
are fully within the expected REDTOP sensitivity to
$\eta^{(\prime)}\to\pi\pi(a\to e^{+}e^{-})$ final states. Therefore, REDTOP
will definitively probe the remaining parameter space of the visible QCD
axion.

Figure 7: Estimated branching ratios for $\eta^{(\prime)}\to\pi\,\pi\,a$ as a
function of the scalar octet couplings to the light pseudoscalar mesons (taken
from Alves (2021)). The width of the blue and brown bands result from varying
the masses and widths of the scalar resonances, $a_{0}$ and $f_{0}$, within
their experimental uncertainties. For the narrow bands with darker shading,
the scalar masses were fixed to $m_{a_{0}}=m_{f_{0}}=980\,$MeV, and their
widths were varied within the ranges $\Gamma_{a_{0}}=(40-100)\,$MeV,
$\Gamma_{f_{0}}=(10-200)\,$MeV. The broader, lighter shaded bands resulted
from additionally varying the scalar masses within the ranges
$m_{a_{0}},\,m_{f_{0}}=(960-1000)\,$MeV.

####  III.1.5 Axion-like particles

More generally, axio-hadronic decays of the $\eta$ and $\eta^{\prime}$ can
also probe axion-like particles (ALPs), which have the same types of
interactions as the QCD axion but receive an additional, PQ-breaking
contribution to their masses. As such, ALP models have a broader parameter
space than the QCD axion since the ALP mass and decay constant are independent
parameters.

The ALP interactions that contribute most significantly to axio-hadronic
$\eta^{(\prime)}$ decays are:

$\mathcal{L}_{\text{ALP}}^{(Y)}\leavevmode\nobreak\
\supset\leavevmode\nobreak\
-c_{GG}\,\frac{\alpha_{s}}{4\pi}\,\frac{a}{f_{a}}G\widetilde{G}\leavevmode\nobreak\
+\leavevmode\nobreak\ \sum_{q=u,d,s}\leavevmode\nobreak\
m_{q}\,\leavevmode\nobreak\
\bar{q}\,e^{i\,c_{q}\frac{a}{f_{a}}\gamma_{5}}q\leavevmode\nobreak\
-\leavevmode\nobreak\ M_{\;{/\\!\\!\\!\\!\\!\text{\tiny
PQ}}}^{2}\,\frac{a^{2}}{2}\,,$ (34)

where $a$ denotes an ALP with decay constant $f_{a}$,
$M_{\;{/\\!\\!\\!\\!\\!\text{\tiny PQ}}}$ is a PQ-breaking contribution to the
ALP mass, and the quarks $q=u,d,s$ are written in the mass eigenstate basis.
Furthermore, (34) is defined at the GeV scale, the heavy-flavor quarks $c,b,t$
having been integrated out. If they carry PQ charges $c_{c}$, $c_{b}$, and
$c_{t}$, respectively, then they will implicitly contribute to the gluonic ALP
coupling through

$c_{GG}\leavevmode\nobreak\ =\leavevmode\nobreak\ c_{GG}^{\text{\tiny
UV}}\leavevmode\nobreak\ +\leavevmode\nobreak\
\sum_{\;q=c,b,t}\,\frac{c_{q}}{2}\,.$ (35)

By performing ALP-dependent quark chiral rotations, the ALP interactions in
(34) can be recast in another commonly considered basis in the recent
literature:

$\displaystyle\mathcal{L}_{\text{ALP}}^{(\partial)}\leavevmode\nobreak\
\supset\leavevmode\nobreak\ $
$\displaystyle-\left(c_{GG}+\sum_{q}\,\frac{c_{q}}{2}\right)\frac{\alpha_{s}}{4\pi}\,\frac{a}{f_{a}}G\widetilde{G}\leavevmode\nobreak\
+\leavevmode\nobreak\ \frac{\partial_{\mu}a}{f_{a}}\leavevmode\nobreak\
\sum_{q}\,\frac{c_{q}}{2}\;\bar{q}\,\gamma^{\mu}\gamma_{5}\,q\leavevmode\nobreak\
-\leavevmode\nobreak\ M_{\;{/\\!\\!\\!\\!\\!\text{\tiny
PQ}}}^{2}\,\frac{a^{2}}{2}$ (36a) $\displaystyle\leavevmode\nobreak\
\leavevmode\nobreak\ +\leavevmode\nobreak\ \sum_{i,j}\leavevmode\nobreak\
\frac{g_{2}}{\sqrt{2}\,}\;V^{\text{\tiny CKM}}_{ij}\leavevmode\nobreak\
W_{\mu}^{+}\leavevmode\nobreak\ \leavevmode\nobreak\
\bar{u}_{L}^{i}\,e^{-i\Big{(}\\!\frac{\,c_{u_{i}}-c_{d_{j}}\,}{2}\frac{a}{f_{a}}\gamma_{5}\\!\Big{)}}\gamma^{\mu}\,d_{L}^{j}\leavevmode\nobreak\
+\leavevmode\nobreak\ \text{h.c.}$ (36b)

Note that the “Yukawa basis” in (34) and the “derivative basis” in (36a) are
equivalent as long as the weak interactions are neglected. (The failure to
account for the additional terms in (36b) has led to much confusion in the
literature and convoluted treatments of axio-hadronic Kaon decays, e.g., Bauer
et al. (2021).)

However, because axio-hadronic $\eta^{(\prime)}$ decays do not violate flavor,
this basis equivalency issue between (34) and (36a) is irrelevant for the
decays we will consider in this subsection. For the same reason, more generic
derivative ALP couplings to axial and vector flavor-changing neutral currents
are also irrelevant. Therefore, for the remainder of this subsection we will
adopt the simpler, “Yukawa basis” parametrization of the ALP couplings in
(34).

In order to extract the amplitude for $\eta^{(\prime)}\to\pi\pi a$ from (34),
three basic steps are needed:

  1. 1.

The QCD level ALP-interactions in (34) must be mapped into chiral perturbation
theory ($\chi$PT), i.e., they must be re-expressed in terms of ALP couplings
to mesonic degrees of freedom.

  2. 2.

The mass matrix and kinetic mixing terms must be diagonalized in order to
obtain the physical ALP state $a_{\text{\tiny phys}}$ and the low energy meson
states $\pi^{0}$, $\eta$, and $\eta^{\prime}$.

  3. 3.

The $\chi$PT Lagrangian must be re-expressed in terms of the physical states,
so that the quartic interactions $\eta^{(\prime)}$-$\pi$-$\pi$-$a_{\text{\tiny
phys}}$ can be obtained.

A proper treatment and execution of steps 1$-$3 outlined above is still
missing in the ALP literature. Firstly, to properly describe the octet-singlet
composition of the $\eta$ and the $\eta^{\prime}$, as well as their mixing
with the ALP, one needs to go beyond leading order in $\chi$PT. Secondly,
(axio)-hadronic $\eta^{(\prime)}$ decays receive significant nonperturbative
corrections which require going beyond $\chi$PT, such as, e.g., accounting for
strong final state rescattering via dispersive relations, or including the
exchange of the low energy QCD resonances via Resonance Chiral Theory. Here,
we will not address either of these issues (work in preparation in these
directions will appear in Alves and Gonzàlez-Solís (2022)). We will simply
follow recent naïve, leading order $\chi$PT treatments to obtain the amplitude
for axio-hadronic $\eta^{(\prime)}$ decays. As such, our results should be
considered a rough, order-of-magnitude estimate of the rate for
$\eta^{(\prime)}\to\pi\pi a$ to qualitatively assess REDTOP’s sensitivity
reach to the parameter space of hadronic ALPs.

The implementation of step 1 above at leading order in $\chi$PT amounts to
mapping (34) into:

$\mathcal{L}_{\text{ALP}}^{(\chi\text{PT}\text{@LO})}\,=\leavevmode\nobreak\
-\frac{M_{0}^{2}}{2}\left(\eta_{0}-\frac{c_{GG}}{\sqrt{3/2}\,}\frac{f_{\pi}}{f_{a}}\,a\right)^{2}\,+\;\frac{f_{\pi}^{2}}{4}\,\text{Tr}\big{[}2B_{0}M_{q}(a)U\,+\,\text{h.c.}\big{]}\,-\,M_{\;{/\\!\\!\\!\\!\\!\text{\tiny
PQ}}}^{2}\frac{a^{2}}{2}\,,$ (37)

where $M_{0}\sim\mathcal{O}$(GeV) parametrizes the large strong anomaly
contribution to the mass of the $\eta^{\prime}$, $B_{0}=-\langle
q\bar{q}\rangle/f_{\pi}^{2}$, $M_{q}(a)$ is the ALP-dependent quark mass
matrix,

$M_{q}(a)\equiv\begin{pmatrix}\,m_{u}\,e^{i\,c_{u}a/f_{a}}\\!\\!\\!&&\\\
&\\!\\!\\!m_{d}\,e^{i\,c_{d}\,a/f_{a}}\\!\\!\\!&\\\
&&\\!\\!\\!m_{s}\,\,e^{i\,c_{s}\,a/f_{a}}\end{pmatrix},$ (38)

and $U$ denotes the non-linear representation of the chiral meson nonet,

$\displaystyle U\leavevmode\nobreak\ =\leavevmode\nobreak\
\text{Exp}\frac{i}{f_{\pi}}\,\begin{pmatrix}\vspace{0.17cm}\pi^{0}+\frac{\eta_{8}}{\sqrt{3}}+\frac{\eta_{0}}{\sqrt{3/2}}&\sqrt{2}\,\pi^{+}&\sqrt{2}\,K^{+}\\\
\vspace{0.17cm}\sqrt{2}\,\pi^{-}&-\pi^{0}+\frac{\eta_{8}}{\sqrt{3}}+\frac{\eta_{0}}{\sqrt{3/2}}&\sqrt{2}\,K^{0}\\\
\sqrt{2}\,K^{-}&\sqrt{2}\,\overline{K}^{0}&-\frac{2\,\eta_{8}}{\sqrt{3}}+\frac{\eta_{0}}{\sqrt{3/2}}\leavevmode\nobreak\
\end{pmatrix}.$ (39)

Even though we are restricting our calculation to the naïve leading order
$\chi$PT treatment, we will introduce a small improvement by adopting the two-
mixing-angle scheme to express the physical $\eta$ and $\eta^{\prime}$ states
in the octet-singlet basis as

$\displaystyle\eta_{0}\leavevmode\nobreak\ =$
$\displaystyle\leavevmode\nobreak\
\;\frac{f_{\pi}}{f_{0}}\,\big{(}\\!\cos\theta_{0}\,\eta^{\prime}\,-\,\sin\theta_{0}\,\eta\big{)}\,,$
(40a) $\displaystyle\eta_{8}\leavevmode\nobreak\ =$
$\displaystyle\leavevmode\nobreak\
\;\frac{f_{\pi}}{f_{8}}\,\big{(}\\!\sin\theta_{8}\,\eta^{\prime}\,+\,\cos\theta_{8}\,\eta\big{)}\,.$
(40b)

For concreteness, we will take the values $\theta_{8}=-24^{\circ}$,
$\theta_{0}=-2.5^{\circ}$, $f_{8}=1.51\,f_{\pi}$, and $f_{0}=1.29\,f_{\pi}$
from the unconstrained fit in Escribano and Frere (2005).

Step 2 is the non-trivial part of our calculation, and will involve obtaining
the ALP mixing angles $\theta_{a\pi}$, $\theta_{a\eta_{8}}$,
$\theta_{a\eta_{0}}$ with $\pi^{0}$, $\eta_{8}$, $\eta_{0}$, respectively.
Because of our judicious choice of ALP basis with no kinetic mixing terms, our
work is greatly simplified: the ALP-meson mixing angles can be simply obtained
by diagonalizing the ALP-meson mass matrix in (37). While this can be done
straightforwardly numerically, a parametric dependence of the mixing angles on
$m_{a}$, $f_{a}$, $c_{q}$ and $c_{GG}$ is desirable. This can be obtained by
considering the following.

First, in the PQ preserving limit of $M_{\;{/\\!\\!\\!\\!\\!\text{\tiny
PQ}}}=0$ and $m_{a}\ll m_{\pi}$ (i.e., when $a$ is the QCD axion), we have

$(m_{a}^{\text{\tiny(PQ)}})^{2}\leavevmode\nobreak\ =\leavevmode\nobreak\
\frac{(c_{u}+c_{d}+c_{s}+2\,c_{GG})^{2}}{(1+\epsilon_{\eta\eta^{\prime}})}\,\frac{m_{u}m_{d}}{(m_{u}+m_{d})^{2}}\,\frac{m_{\pi}^{2}\,f_{\pi}^{2}}{f_{a}^{2}}\,,$
(41)

and

$\displaystyle\theta_{a\pi}^{{}^{\text{\tiny(PQ)}}}=$
$\displaystyle-\frac{f_{\pi}}{f_{a}}\,\frac{1}{(1+\epsilon_{\eta\eta^{\prime}})}\,\bigg{(}\frac{\,(c_{u}m_{u}-c_{d}m_{d})\,+\,(c_{s}+2c_{GG})(m_{u}-m_{d})/2\,}{m_{u}+m_{d}}+\epsilon_{\eta\eta^{\prime}}\,\frac{\,(c_{u}-c_{d})\,}{2}\bigg{)}$
(42a) $\displaystyle\theta_{a\eta_{8}}^{{}^{\text{\tiny(PQ)}}}=$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\
\frac{f_{\pi}}{f_{a}}\,\frac{\sqrt{3}}{\leavevmode\nobreak\
2}\;\frac{1}{(1+\epsilon_{\eta\eta^{\prime}})}\;\bigg{(}c_{s}+\frac{2}{3}\,c_{GG}-\epsilon_{\eta\eta^{\prime}}\frac{\,(c_{u}+c_{d}+4c_{GG}/3)+(c_{u}+c_{d}-2c_{s})\,2m_{\\!K}^{2}/m_{\\!\eta^{\prime}}^{2}\,}{1+6m_{\\!K}^{2}/m_{\\!\eta^{\prime}}^{2}}\bigg{)}$
(42b) $\displaystyle\theta_{a\eta_{0}}^{{}^{\text{\tiny(PQ)}}}=$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\
\frac{f_{\pi}}{f_{a}}\;\bigg{(}\frac{c_{GG}}{\sqrt{3/2}\,}\,-\,\frac{m_{\pi}^{2}}{m_{\eta^{\prime}}^{2}}\,\frac{\sqrt{6}\,(c_{u}+c_{d}+c_{s}+2c_{GG})}{1+\epsilon_{\eta\eta^{\prime}}}\frac{m_{u}m_{d}}{(m_{u}+m_{d})^{2}}\bigg{)}$
(42c)

where

$\epsilon_{\eta\eta^{\prime}}\leavevmode\nobreak\ \equiv\leavevmode\nobreak\
\frac{m_{u}m_{d}}{m_{s}(m_{u}+m_{d})}\;\bigg{(}1+6\,\frac{m_{K}^{2}}{m_{\eta^{\prime}}^{2}}\bigg{)}\leavevmode\nobreak\
\simeq\leavevmode\nobreak\ 0.04\,.$ (43)

This can be generalized to the PQ-breaking case of a generic ALP as

$m_{a}^{2}\leavevmode\nobreak\ =\leavevmode\nobreak\
(m_{a}^{\text{\tiny(PQ)}})^{2}\,+\,M_{\;{/\\!\\!\\!\\!\\!\text{\tiny
PQ}}}^{2}\,,$ (44)

and

$\displaystyle\theta_{a\pi}=\theta_{a\pi}^{{}^{\text{\tiny(PQ)}}}\bigg{(}1\,+\,\frac{m_{a}^{2}}{m_{\pi}^{2}-m_{a}^{2}}\bigg{)}$
(45a)
$\displaystyle\theta_{a\eta_{8}}=\frac{1}{f_{8}}\,\bigg{[}\\!\cos{\theta_{8}}\bigg{(}\frac{f_{8}\cos{\theta_{0}}\,\theta_{a\eta_{8}}^{{}^{\text{\tiny(PQ)}}}-f_{0}\sin{\theta_{8}}\,\theta_{a\eta_{0}}^{{}^{\text{\tiny(PQ)}}}}{\cos{(\theta_{8}-\theta_{0})}}\bigg{)}\bigg{(}1\,+\,\frac{m_{a}^{2}}{\,m_{\eta}^{2}-m_{a}^{2}\,}\bigg{)}\,+$
$\displaystyle\qquad\qquad\quad\sin{\theta_{8}}\bigg{(}\frac{f_{0}\cos{\theta_{8}}\,\theta_{a\eta_{0}}^{{}^{\text{\tiny(PQ)}}}+f_{8}\sin{\theta_{0}}\,\theta_{a\eta_{8}}^{{}^{\text{\tiny(PQ)}}}}{\cos{(\theta_{8}-\theta_{0})}}\bigg{)}\bigg{(}1\,+\,\frac{m_{a}^{2}}{\,m_{\eta\prime}^{2}-m_{a}^{2}\,}\bigg{)}\bigg{]}$
(45b)
$\displaystyle\theta_{a\eta_{0}}=\frac{1}{f_{0}}\,\bigg{[}\\!-\sin{\theta_{0}}\bigg{(}\frac{f_{8}\cos{\theta_{0}}\,\theta_{a\eta_{8}}^{{}^{\text{\tiny(PQ)}}}-f_{0}\sin{\theta_{8}}\,\theta_{a\eta_{0}}^{{}^{\text{\tiny(PQ)}}}}{\cos{(\theta_{8}-\theta_{0})}}\bigg{)}\bigg{(}1\,+\,\frac{m_{a}^{2}}{\,m_{\eta}^{2}-m_{a}^{2}\,}\bigg{)}\,+$
$\displaystyle\qquad\qquad\quad\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\
\cos{\theta_{0}}\bigg{(}\frac{f_{0}\cos{\theta_{8}}\,\theta_{a\eta_{0}}^{{}^{\text{\tiny(PQ)}}}+f_{8}\sin{\theta_{0}}\,\theta_{a\eta_{8}}^{{}^{\text{\tiny(PQ)}}}}{\cos{(\theta_{8}-\theta_{0})}}\bigg{)}\bigg{(}1\,+\,\frac{m_{a}^{2}}{\,m_{\eta\prime}^{2}-m_{a}^{2}\,}\bigg{)}\bigg{]}.$
(45c)

Expression (45a) for the ALP-pion mixing angle $\theta_{a\pi}$ holds as long
as $m_{a}$ is not too close to $m_{\pi^{0}}$. Similarly, (45b) and (45c) hold
as long as $m_{a}$ is not too close to either $m_{\eta}$ or
$m_{\eta^{\prime}}$.

We can finally proceed to step 3 by re-expressing the ALP Lagrangian in terms
of the physical ALP:

$\mathcal{L}_{\text{ALP}}^{(\chi\text{PT}\text{@LO})}\bigg{|}_{\pi^{0}\,\to\,\pi^{0}+\theta_{a\pi}a\leavevmode\nobreak\
\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\
\eta_{8}\,\to\,\eta_{8}+\theta_{a\eta_{8}}a\leavevmode\nobreak\
\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\
\eta_{0}\,\to\,\eta_{0}+\theta_{a\eta_{0}}a\leavevmode\nobreak\
\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\
a\,\to\,a-\theta_{a\pi}\pi^{0}-\theta_{a\eta_{8}}\eta_{8}-\theta_{a\eta_{0}}\eta_{0}}\,,$
(46)

from which we obtain the leading order $a$-$\pi$-$\pi$-$\eta^{(\prime)}$
quartic couplings,

$\mathcal{L}_{\text{ALP}}^{(\chi\text{PT}\text{@LO})}\leavevmode\nobreak\
\supset\leavevmode\nobreak\
\lambda_{0}\,a\,\left(\pi^{+}\pi^{-}+\frac{\pi^{0}\pi^{0}}{2}\right)\left(\\!\frac{\eta_{8}}{\sqrt{3}\,}+\frac{\eta_{0}}{\sqrt{3/2}\,}\\!\right)\,,$
(47)

where

$\lambda_{0}\,\leavevmode\nobreak\ =\leavevmode\nobreak\
\frac{m_{\pi}^{2}}{f_{\pi}^{2}}\,\left(\frac{\,(c_{u}m_{u}+c_{d}m_{d})\,}{(m_{u}+m_{d})}\frac{f_{\pi}}{f_{a}}\,+\,\frac{\,(m_{u}-m_{d})\,}{(m_{u}+m_{d})}\,\theta_{a\pi}\,+\,\frac{\theta_{a\eta_{8}}}{\sqrt{3}\,}+\,\frac{\theta_{a\eta_{0}}}{\sqrt{3/2}\,}\right)\,.$
(48)

We can now turn our focus specifically to $\eta\to\pi\pi a$. The amplitude for
this decay then follows from (47) and (48) straightforwardly:

$\displaystyle\mathcal{A}_{\eta\to\pi\pi a}\leavevmode\nobreak\ $
$\displaystyle\equiv$ $\displaystyle\leavevmode\nobreak\
\mathcal{A}(\eta\to\pi^{0}\pi^{0}a)\leavevmode\nobreak\ =\leavevmode\nobreak\
\mathcal{A}(\eta\to\pi^{+}\pi^{-}a)$ (49) $\displaystyle=$
$\displaystyle\leavevmode\nobreak\
\lambda_{0}\,\left(\frac{f_{\pi}}{f_{8}}\,\frac{\cos\theta_{8}}{\sqrt{3}}\,-\,\frac{f_{\pi}}{f_{0}}\,\frac{\sin\theta_{0}}{\sqrt{3/2}}\right).$

Since the decay amplitude (49) is flat in the Dalitz phase-space, the
differential rate for $\eta\to\pi\pi a$ as a function of the ALP 3-momentum
$|\vec{p}_{a}|$ in the $\eta$ rest frame has a closed analytical form:

$\displaystyle\frac{d\Gamma}{d|\vec{p}_{a}|}\leavevmode\nobreak\ $
$\displaystyle=$ $\displaystyle\leavevmode\nobreak\
\frac{1}{S_{\pi\pi}}\,\frac{|\mathcal{A}_{\eta\to\pi\pi
a}|^{2}}{(4\pi)^{3}}\;\frac{|\vec{p}_{a}|^{2}}{m_{\eta}\,E_{a}}\,\sqrt{1-\frac{4\,m_{\pi}^{2}}{(p_{\eta}-p_{a})^{2}}}$
$\displaystyle=$ $\displaystyle\leavevmode\nobreak\
\frac{1}{S_{\pi\pi}}\,\frac{|\mathcal{A}_{\eta\to\pi\pi
a}|^{2}}{(4\pi)^{3}}\;\frac{|\vec{p}_{a}|^{2}}{m_{\eta}}\,\sqrt{\frac{m_{\eta}^{2}+m_{a}^{2}-2\,m_{\eta}\sqrt{m_{a}^{2}+|\vec{p}_{a}|^{2}}-4\,m_{\pi}^{2}}{(\,m_{\eta}^{2}+m_{a}^{2}-2\,m_{\eta}\sqrt{m_{a}^{2}+|\vec{p}_{a}|^{2}\,}\,)(\,m_{a}^{2}+|\vec{p}_{a}|^{2})}}\,,$

where the combinatorial factor $S_{\pi\pi}$ takes the values
$S_{\pi^{0}\pi^{0}}=2$ and $S_{\pi^{+}\pi^{-}\\!}=1$.

To assess REDTOP’s reach in the ALP parameter space, we consider two common
benchmark scenarios that have been broadly adopted in Snowmass studies of
ALPs:

  * •

_Gluon dominance,_ which assumes that the ALP couples predominantly to heavy
BSM colored fermions at UV scales above $f_{a}$. Once these fermions are
integrated out, the ALP couplings expressed in terms of (34) are given by
$c_{GG}^{\text{\tiny UV}}\neq 0$ and $c_{q}=0$ for all six SM quark flavors.

  * •

_Quark dominance,_ which assumes that the ALP couples predominantly to SM
quarks, with no BSM contributions to the gluonic ALP coupling, i.e.,
$c_{GG}^{\text{\tiny UV}}=0$. We further restrict the parameter space of this
benchmark scenario by imposing flavor blindness, i.e., we assume that all six
quark flavors couple identically to the ALP, as parametrized by a universal
coupling $c_{q}$ in (34) and (35).

Fig. 8 illustrates REDTOP’s futuristic reach in the ALP parameter space for
the _gluon dominance_ and _quark dominance_ benchmark scenarios, assuming
branching sensitivities of $\text{Br}[\eta\to\pi^{0}\pi^{0}a]=10^{-12}$ (green
curves) and $\text{Br}[\eta\to\pi^{+}\pi^{-}a]=10^{-12}$ (orange curves). It
is important to emphasize, however, that these futuristic projections assume
that REDTOP would be sensitive to prompt, displaced, and invisible ALP decays
alike.

For a more realistic near term sensitivity assuming the current REDTOP
detector design, we can also estimate REDTOP’s reach to _visibly_ -decaying
ALPs, i.e., ALPs that decay to visible final states within the detector’s
fiducial volume. We will make the simplified assumption that ALPs decaying
within 100 cm of the $\eta$ decay vertex in the $\eta$ rest frame are visibly-
decaying ALPs.

To proceed, further considerations are needed regarding the ALP couplings to
photons and leptons in addition to its hadronic couplings defined in (34):

$\mathcal{L}_{\text{ALP}}^{(Y)}\leavevmode\nobreak\ \leavevmode\nobreak\
\supset\leavevmode\nobreak\ \leavevmode\nobreak\
c_{\gamma\gamma}\,\frac{\alpha}{4\pi}\,\frac{a}{f_{a}}F\widetilde{F}\leavevmode\nobreak\
+\leavevmode\nobreak\ \sum_{\ell=e,\mu,\tau}\leavevmode\nobreak\
m_{\ell}\,\leavevmode\nobreak\
\bar{\ell}\,e^{i\,c_{\ell}\frac{a}{f_{a}}\gamma_{5}}\ell\,.$ (51)

In particular, the ALP coupling to photons receives contributions from high UV
scale dynamics, as well as from its mixing with the neutral pseudoscalar
mesons and its couplings to heavy quarks and charged leptons,

$\displaystyle c_{\gamma\gamma}\leavevmode\nobreak\ =\leavevmode\nobreak\
c_{\gamma\gamma}^{\text{\tiny UV}}\leavevmode\nobreak\ $ $\displaystyle+$
$\displaystyle\leavevmode\nobreak\
\frac{f_{a}}{f_{\pi}}\left(\theta_{a\pi}+\frac{5}{3}\,\theta_{a\eta_{{}_{ud}}}+\frac{\sqrt{2}}{\leavevmode\nobreak\
3}\,\theta_{a\eta_{{}_{s}}}\right)$ $\displaystyle+$
$\displaystyle\leavevmode\nobreak\ \sum_{q=c,b,t}3\,c_{q}\,(Q^{\text{\tiny
em}}_{q})^{2}\;\left|F_{1/2}(\tau_{q})\right|\leavevmode\nobreak\
+\sum_{\ell=e,\mu,\tau}c_{\ell}\,(Q^{\text{\tiny
em}}_{\ell})^{2}\;\left|F_{1/2}(\tau_{\ell})\right|\,,$

where $\tau_{f}\equiv 4m_{f}^{2}/m_{a}^{2}$, and

$F_{1/2}(\tau)\leavevmode\nobreak\ \equiv\leavevmode\nobreak\ -\tau
f(\tau)\,,\qquad\leavevmode\nobreak\ \leavevmode\nobreak\
f(\tau)=\begin{cases}&\\!\\!\\!\\!\leavevmode\nobreak\ \leavevmode\nobreak\
\left(\sin^{-1}\\!\sqrt{1/\tau}\right)^{2}\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \text{if}\leavevmode\nobreak\
\leavevmode\nobreak\ \tau\geq 1\,,\\\
&\\!\\!\\!\\!\\!-\frac{1}{4}\left[\ln\left(\frac{1+\sqrt{1-\tau}}{1-\sqrt{1-\tau}}\right)-i\pi\right]^{2}\leavevmode\nobreak\
\leavevmode\nobreak\ \text{if}\leavevmode\nobreak\ \leavevmode\nobreak\
\tau<1\,.\end{cases}$ (53)

Figure 8: Reach to the ALP parameter space by probes of axio-hadronic $\eta$
decays, such as REDTOP, assuming a branching ratio sensitivity of
$\text{Br}[\eta\to\pi\pi a]=10^{-12}$ for all benchmark scenarios. Top panel:
_gluon dominance_ scenario, with the green and orange curves denoting the
reach to visible∗ and invisible ALP decays; the dark blue shaded region
denoting sensitivity to visible∗ $a\to\gamma\gamma$ decays of a leptophobic
ALP; and the light blue shaded region denoting sensitivity to visible∗
$a\to\ell^{+}\ell^{-},\gamma\gamma$ decays of a leptophilic ALP. Bottom panel:
analogous to the top panel, but assuming the _quark dominance_ scenario. ∗In
these plots, the ALP decay is considered visible if the ALP decay vertex is
displaced by less than 100 cm from the $\eta$ decay vertex in the $\eta$ rest
frame.

Since we are considering ALPs in the context of axio-hadronic $\eta$ decays,
they will never be heavy enough to decay hadronically. Therefore, the ALP
decay width will be given by:

$\Gamma_{a}\leavevmode\nobreak\ =\leavevmode\nobreak\
\Gamma(a\to\gamma\gamma)\leavevmode\nobreak\
+\;\sum_{\ell}\;\Gamma(a\to\ell^{+}\ell^{-})\,,$ (54)

where

$\Gamma(a\to\gamma\gamma)\leavevmode\nobreak\ =\leavevmode\nobreak\
\frac{m_{a}^{3}}{4\pi}\,\left(\\!c_{\gamma\gamma}\;\frac{\alpha}{4\pi
f_{a}}\right)^{\\!2}\,,$ (55)

and the sum over lepton flavors in (54) of course only runs over leptons for
which the decay $a\to\ell^{+}\ell^{-}$ is kinematically allowed, i.e., for
$m_{\ell}<m_{a}/2$, so that

$\Gamma(a\to\ell^{+}\ell^{-})\leavevmode\nobreak\ =\leavevmode\nobreak\
\frac{m_{a}}{\,8\pi\,}\,\left(\frac{c_{\ell}\,m_{\ell}}{f_{a}}\right)^{\\!2}\,\sqrt{1-\frac{4\,m_{\ell}^{2}}{m_{a}^{2}}}\,.$
(56)

With (51)-(56), we can now complement our definition of the _gluon dominance_
and _quark dominance_ benchmark scenarios in the following way:

  * •

_Gluon dominance, leptophobic,_

$c_{GG}^{\text{\tiny UV}}\neq 0,\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ c_{q}=0\leavevmode\nobreak\
\forall\leavevmode\nobreak\ q,\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ c_{\ell}=0\leavevmode\nobreak\
\forall\leavevmode\nobreak\ \ell,\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ c_{\gamma\gamma}^{\text{\tiny
UV}}=0.$ (57)

  * •

_Gluon dominance, flavor universal, leptophilic,_

$c_{GG}^{\text{\tiny UV}}\neq 0,\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ c_{q}=0\leavevmode\nobreak\
\forall\leavevmode\nobreak\ q,\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ c_{\ell}=2c_{GG}^{\text{\tiny
UV}}\leavevmode\nobreak\ \forall\leavevmode\nobreak\ \ell,\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
c_{\gamma\gamma}^{\text{\tiny UV}}=0.$ (58)

  * •

_Quark dominance, flavor universal, leptophobic,_

$c_{GG}^{\text{\tiny UV}}=0,\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ c_{q}\neq 0\leavevmode\nobreak\
\forall\leavevmode\nobreak\ q,\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ c_{\ell}=0\leavevmode\nobreak\
\forall\leavevmode\nobreak\ \ell,\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ c_{\gamma\gamma}^{\text{\tiny
UV}}=0.$ (59)

  * •

_Quark dominance, flavor universal, leptophilic,_

$c_{GG}^{\text{\tiny UV}}=0,\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ c_{q}\neq 0\leavevmode\nobreak\
\forall\leavevmode\nobreak\ q,\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ c_{\ell}=c_{q}\leavevmode\nobreak\
\forall\leavevmode\nobreak\ \ell,\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ c_{\gamma\gamma}^{\text{\tiny
UV}}=0.$ (60)

Fig. 8 also illustrates the REDTOP sensitivity reach in these four scenarios
(shaded regions) assuming a _visible_ branching ratio sensitivity of
$\text{Br}[\eta\to\pi\pi(a\to\text{visible})]=10^{-12}$.

####  III.1.6 Probing a BSM origin of the proton radius anomaly

One of the existing and still unexplained anomalies present in the Standard
Model is related to the measurements of the proton radius $R_{p}$ with
electron and muon probes.

The processes involved are $\eta$-Dalitz decays, and lepton-pair decays:

$\eta\rightarrow\gamma e^{+}e^{-}\,,\eta\rightarrow\gamma\mu^{+}\mu^{-}\text{
and }\eta\rightarrow e^{+}e^{-}\,,\eta\rightarrow\mu^{+}\mu^{-}$ (61)

Other types of $R_{p}$ measurements, using muonic atoms (see, for example,
CODATA-2012), or using elastic scattering of electrons and muons on hydrogen
atoms, have found a discrepancy corresponding to several sigma in the electron
vs muon case. It is worth noting that such processes occur mainly through the
exchange of one virtual photon, but subleading corrections to that—needed to
better clarify the situation—may include pseudoscalar-exchange contribution to
the 2S hyperfine splitting in the muonic hydrogen, in a t-channel
contribution. One would expect the $\pi^{0}$ to dominate such subleading
correction but being a $t$-channel exchange, all pseudoscalars may contribute,
being the processes of the type $\eta(\eta{{}^{\prime}})\rightarrow\gamma
l^{+}l^{-}$ a background to them, as shown in Fig. 9 diagram a), versus the
two-photon contribution, as shown in Fig. 9 diagram b).

Figure 9:  Diagrams contributing to the proton radius

A light scalar particle S with a different coupling with electrons and muons
would mediate this process, as shown in diagram c) of figure 9, would explain
this anomaly of the Standard Model. Therefore, an experiment able to precisely
measure the branching ratios of this particle might help in explaining the
$R_{p}$ anomaly.

In REDTOP, the processes (61) are detected simultaneously and within the same
experimental apparatus. Consequently, most of the systematic errors are common
to the two processes and they factor out in the ratio of the corresponding
branching ratios, enhancing the precision of the overall measurement.
$\eta^{{}^{\prime}}$ contributions may play a role as well.

###  III.2 Tests of conservation laws

Conservation laws with their underlying symmetry principles are at the heart
of physics, from the classical space-time conservation laws of introductory
courses through the symmetries and additive quantum numbers of modern particle
physics. The Crystal Ball experiment at the Brookhaven AGS was able to provide
a few times $10^{7}$ $\eta$ (as for the $4\pi^{0}$ decay study). It was
subsequently moved to MAMI, and a goal there is to achieve another order of
magnitude in $\eta$ yield. Other facilities include KLOE (for
$\phi(1020)\to\eta\gamma$), and GlueX at JLAB, all at the few times $10^{7}$
level. Recently, WASA at COSY reached a milestone, by collecting about
$10^{9}$, although with conventional, background prone, detector technologies,
still insufficient for exploring successfully the realm BSM. To reach the more
exacting levels needed for symmetry violations, the usable $\eta$ flux must be
increased by several orders of magnitude.

To achieve this goal, the REDTOP experiment is being designed to provide a sea
change in the number of $\eta$ samples to $1.1\times 10^{14}$ and
$\eta^{\prime}$ samples to $1\times 10^{12}$, along with a nearly $4\pi$
detector to study a broad range of fore-front physics. The facility will
provide vastly reduced upper limits for $\eta$ and $\eta^{\prime}$ decays, as
well as studies of processes that can lead to New Physics Beyond the Standard
Model.

The light pseudoscalar mesons $\pi^{0}$, $\eta$, and $\eta^{\prime}$ have very
special roles for exploring and testing the conservation laws. The $\pi^{0}$
has a long history of such tests and has established tight upper limits of
charge ($C$) and lepton flavor ($LF$) violations et al. (2016). Unlike the
isospin $I=1$ for the $\pi^{0}$, all the additive quantum numbers for the
$\eta$ and $\eta^{\prime}$ are zero, and they differ from the vacuum only in
terms of parity. Due to the opposite $G$ parities of the the $\pi^{0}$ and
$\eta$, couplings to strong interactions are suppressed. Thus, tests of $C$
and $CP$ in electromagnetic interactions are much more directly accessible in
$\eta$ and $\eta^{\prime}$ decays, limited mainly by the flux of such mesons
Nef (1994). In addition, such decays can provide tests of $P$, $T$, $CT$,
$PT$, and even $CPT$. Among other possibilities are searches for lepton family
violation, leptoquarks, and significant tests of the parameterization of
chiral perturbation theory.

Almost all searches for symmetry violations in $\eta$/$\eta^{\prime}$ decays
are upper limits in the range of $10^{-5}$ or higher et al. (2016). An
exception is the decay $\eta\to 4\pi^{0}$ at $<6.9\times 10^{-7}$, based on
$3\times 10^{7}$ $\eta$ mesons Prakhov et al. (2000). One-sigma errors have
been reported for some asymmetries in the Dalitz distribution of
$\eta\to\pi^{+}\pi^{-}\pi^{0}$ (which are consistent with zero at the level of
$10^{-2}$) collaboration et al. (2008). Most models of symmetry violations for
various decay processes are at or below the level of $10^{-9}$, typically by
several orders of magnitude.

_CP_ violation has been extensively studied in the flavor-changing decays of
the neutral $K$\- and $B$-mesons. The origin of the violation is still not
fully understood. The standard model predicts that the source of _CP_
violation is a single phase in the Cabbibo-Kobayashi-Maskawa (CKM) mixing
matrix of quarks couplings. At present, the predictions based on CKM mechanism
are consistent with the observations in $K$ and $B$ systems, but tensions are
arising. We propose to explore other sources of _CP_ violation beyond the CKM
mechanism and with flavor-conserving processes, especially through
measurements not bound by EDM limits. Rare $\eta$/$\eta^{\prime}$ decays
provide a good laboratory for that. All decays of the $\eta$/$\eta^{\prime}$
mesons into multiple photons or into photons plus $\pi^{0}$s provides a direct
test of $C$ invariance. Each photon in the final state, including the two from
$\pi^{0}$ decay, has $C=-1$. Because the $\eta$ has $C=+1$, final states with
an odd number of photons are forbidden. However, the branching ratio for these
processes are bound by EDM measurements and they would explore aspects of _CP_
violations not accessible even at a $\eta$-factory. We propose, instead,
studying three processes that are not bound by current EDM measurements and
that are probing different operators which would induce a violation of _CP_
from sources Beyond the Standard Model.

Several $\eta$/$\eta^{\prime}$-related processes have been selected to study
the sensitivity of REDTOP to Conservation Laws. These studies are restricted,
for now, only to the CP violation (CP) and Lepton Flavor violation, where more
solid theoretical models exists which are consistent, at the same time, with
bounds from EDM measurements and with outstanding experimental anomalies.
These are discussed below, alongside with the theoretical models supporting
them.

####  III.2.1 C and CP violation from Dalitz asymmetries in
$\eta\rightarrow\pi^{+}\pi^{-}\pi^{0}$

The decay $\eta\to\pi^{+}\pi^{-}\pi^{0}$ can only occur when isospin or/and
charge conjugation (C) is/are broken. Thus the interference of a C-conserving
but isospin-breaking amplitude with a C-violating one would give rise to a
charge asymmetry in the Dalitz plot of the 3$\pi$ final state for this
process. Since parity P is conserved in this decay, the existence of a nonzero
charge asymmetry would attest to the breaking of C and CP symmetry. In
contrast, searches for a nonzero permanent electric dipole moment (EDM) probe
the possibility of new sources of P and CP violation. Moreover, the SM
mechanism of CP violation, vis-a-vis the flavor-changing weak interactions of
quarks, is expected to be completely negligible in this context. Although the
charge asymmetry observable was first proposed long ago Lee and Wolfenstein
(1965); Nauenberg (1965); Lee (1965), we revisit and refine it not only
because it is a flavor-diagonal, C- and CP-violating observable, but it is
also, moreover, an effect that scales linearly, rather than quadratically, in
the underlying CP-violating parameter Gardner and Shi (2020). Moreover, a
study of the new physics sources in Standard Model Effective Field Theory
(SMEFT), starting from the compilation of Ref. Grzadkowski et al. (2010),
shows that the sources of CP violation that stem from C- or P- violating
effects in the flavor diagonal sector are distinct Shi (2020); Gardner and Shi
(2022). A charge asymmetry in the $B^{0}(\bar{B}^{0})\to\pi^{+}\pi^{-}\pi^{0}$
Dalitz plot also probes CP violation Gardner and Tandean (2004), and the SMEFT
operators in that case are mass dimension six Shi (2020); Gardner and Shi
(2022). In our case the SMEFT operators are mass dimension eight, but could be
of dimension six in numerical size, which would signal the existence of
dynamics beyond SMEFT Shi (2020); Gardner and Shi (2022). (We note Ref.
Burgess et al. (2021) for a discussion of analogous ideas in $b\to
c\tau\bar{\nu}$ decay.) Thus the measurement of the charge asymmetry in
$\eta\to\pi^{+}\pi^{-}\pi^{0}$ decay is an ideal probe with which to explore
the possibility of physics beyond the SM. The REDTOP experimental concept is
well suited to careful measurements of the charge asymmetry observable, and we
consider it a golden channel for study at REDTOP.

Recently, theoretical work has been done Gardner and Shi (2020); Akdag et al.
(2022) investigating the patterns of C and CP violation in this process from
mirror asymmetry breaking of the Dalitz plot. As long known, when plotting the
Dalitz plot in terms of Mandelstam variables
$t\equiv(p_{\pi^{-}}+p_{\pi^{0}})^{2}$ and
$u\equiv(p_{\pi^{+}}+p_{\pi^{0}})^{2}$, the charge asymmetry would correspond
to a breaking of mirror symmetry, i.e., $t\leftrightarrow u$ exchange, in the
Dalitz plot. The charge asymmetry can be probed through the measurement of a
left-right asymmetry, $A_{LR}$ Layter et al. (1972)

$A_{LR}\equiv\frac{N_{+}-N_{-}}{N_{+}+N_{-}},$ (62)

where $N_{\pm}$ is the number of events when
$u\stackrel{{\scriptstyle>}}{{{}_{<}}}t$, i.e., the $\pi^{+}$ has more (less)
energy than the $\pi^{-}$ in the $\eta$ rest frame. More conveniently, we can
describe the Dalitz plot in terms of variables $X$ and $Y$

$\displaystyle X$ $\displaystyle\equiv$
$\displaystyle\sqrt{3}\frac{T_{\pi^{+}}-T_{\pi^{-}}}{Q_{\eta}}=\frac{\sqrt{3}}{2M_{\eta}Q_{\eta}}(u-t),$
$\displaystyle Y$ $\displaystyle\equiv$
$\displaystyle\frac{3T_{\pi^{0}}}{Q_{\eta}}-1=\frac{3}{2M_{\eta}Q_{\eta}}[(M_{\eta}-M_{\pi^{0}})^{2}-s]-1\,,$
(63)

where
$Q_{\eta}=T_{\pi^{+}}+T_{\pi^{-}}+T_{\pi^{0}}=M_{\eta}-2M_{\pi^{+}}-M_{\pi^{0}}$,
and $T_{\pi^{i}}$ is the $\pi^{i}$ kinetic energy in the $\eta$ rest frame.
The decay amplitude square can be parametrized in a polynomial expansion
around $(X,\leavevmode\nobreak\ Y)=(0,\leavevmode\nobreak\ 0)$ Anastasi et al.
(2016)

$\displaystyle|A(s,t,u)|^{2}\\!$ $\displaystyle=$
$\displaystyle\\!N(1+aY+bY^{2}+cX+dX^{2}+eXY+fY^{3}+gX^{2}Y$ (64)
$\displaystyle+$ $\displaystyle hXY^{2}+lX^{3}+\ldots)\,.$

The C transformation on the decay is equivalent to $t\leftrightarrow u$ in the
amplitude. As a result, the appearance of terms that are odd in $X$ would
serve as evidence of C and CP violation in this process Gardner and Tandean
(2004); Gardner and Shi (2020).

Although an experimental determination of non-zero coefficients for the terms
with odd powers of $X$ ($c$,$e$,$h$) in the study of the
$\eta\to\pi^{+}\pi^{-}\pi^{0}$ Dalitz plot would signal both $C$ and $CP$
violation, it is possible to gain insight into the isospin structure of the
new physics sources through consideration of their chiral dynamics. We follow
Ref. Gardner and Shi (2020) for our discussion. The decay amplitude in the SM,
working to leading order in isospin breaking, can be expressed as Gasser and
Leutwyler (1985); Anisovich and Leutwyler (1996)

$\displaystyle A(s,t,u)$ $\displaystyle=$
$\displaystyle-\frac{1}{Q^{2}}\frac{M_{K}^{2}}{M_{\pi}^{2}}\frac{M_{K}^{2}-M_{\pi}^{2}}{3\sqrt{3}F_{\pi}^{2}}M(s,t,u)\,.$
(65)

Since $C=-(-1)^{I}$ in $\eta\to 3\pi$ decay Lee (1965), the C- and CP-even
transition amplitude with a $\Delta I=1$ isospin-breaking prefactor must have
$I=1$. The amplitude $M(s,t,u)$ thus corresponds to the total isospin $I=1$
component of the $\pi^{+}\pi^{-}\pi^{0}$ state and can be expressed as
Anisovich and Leutwyler (1996); Lanz (2011)

$\displaystyle M^{C}_{1}\\!(s,t,u)$ $\displaystyle=$ $\displaystyle
M_{0}^{0}(s)+(s-u)M_{1}^{1}(t)+(s-t)M_{1}^{1}(u)$ (66) $\displaystyle+$
$\displaystyle M_{2}^{0}(t)+M_{2}^{0}(u)-\frac{2}{3}M_{2}^{0}(s)\,,$

where $M_{I}^{\ell}$(z) is an amplitude with $\pi-\pi$ rescattering in the
$z$-channel with isospin $I$ (and orbital angular momentum $\ell$). We refer
to Sec. IVA for a nuanced discussion of their construction, pertinent to a
high-precision extraction of the light quark mass difference. The
decomposition can be recovered under isospin symmetry in chiral perturbation
theory (ChPT) up to next-to-next-to-leading order (NNLO), ${\cal O}(p^{6})$,
because the only absorptive parts that can appear are in the $\pi-\pi$ $S-$
and $P$-wave amplitudes Bijnens and Ghorbani (2007).

Since we are considering C and CP violation, additional amplitudes can appear
— namely, total $I=0$ and $I=2$ amplitudes. The complete amplitude is thus

$\displaystyle A(s,t,u)$ $\displaystyle=$
$\displaystyle-\frac{1}{Q^{2}}\frac{M_{K}^{2}}{M_{\pi}^{2}}\frac{M_{K}^{2}-M_{\pi}^{2}}{3\sqrt{3}F_{\pi}^{2}}M^{C}_{1}\\!(s,t,u)$
(67) $\displaystyle+\alpha M^{\not C}_{0}\\!(s,t,u)+\beta M^{\not
C}_{2}\\!(s,t,u)\,,$

where $\alpha$ and $\beta$ are unknown, low-energy constants — complex numbers
to be determined by fits to the experimental event populations in the Dalitz
plot. If they are determined to be non-zero, they signal the appearance of C-
and CP-violation. Following the expectations of Watson’s theorem Watson
(1954); Gardner et al. (2001) we write Gardner and Shi (2020)

$\\!\\!M^{\not
C}_{0}\\!(s,t,u)\\!=\\!(s-t)\\!M_{1}^{1}(u)+(u-s)\\!M_{1}^{1}(t)-(u-t)\\!M_{1}^{1}(s)$
(68)

and

$\displaystyle\\!\\!\\!\\!\\!\\!M^{\not C}_{2}\\!(s,t,u)\\!$ $\displaystyle=$
$\displaystyle\\!(s-t)M_{1}^{1}(u)+(u-s)M_{1}^{1}(t)$ (69)
$\displaystyle\\!\\!\\!+2(u-t)M_{1}^{1}(s)+\sqrt{5}[M_{2}^{0}(u)-M_{2}^{0}(t)]\,.$

In what follows we adopt the NLO analyses of Refs. Gasser and Leutwyler
(1985); Bijnens and Ghorbani (2007) to construct particular forms for the
$M_{I}^{\ell}$ and refer to Ref. Gardner and Shi (2020) for all details. The
analysis we have outlined here has been employed in the sensitivity analysis
of Sec. XIII. We emphasize that the normalization factor $N$ as in Eq. (64)
drops out in the experimental asymmetry $A_{LR}$, Eq. (62).

Besides, it is also possible to measure the asymmetries which can probe the
isospin of the CP violating final state: a sextant asymmetry $A_{S}$ which is
sensitive to the $I=0$ state Lee (1965); Nauenberg (1965) and a quadrant
asymmetry which is sensitive to the $I=2$ final state Lee (1965); Layter et
al. (1972), which are illustrated in Fig. 10. However, an alternate, more
sensitive discriminant of the isospin structure of the new CP-violating
sources can be found by studying the pattern of the Dalitz plot in realizing
the left-right asymmetry, Eq. (62) Gardner and Shi (2020). See also Ref. Akdag
et al. (2022) for a comparison of $C$-violating effects of different isospin
in similar $\eta^{\prime}$ three-body decays.

Figure 10: The Dalitz plot geometry in $\eta\to\pi^{+}\pi^{-}\pi^{0}$ decay,
split into regions for probes of its symmetries. The solid line is the
boundary of the physically accessible region. The asymmetry $A_{LR}$, Eq.
(62), compares the population $N_{+}$ ($X>0$) with $N_{-}$ ($X<0$). The
quadrant asymmetry $A_{Q}$ probes $I=2$ contributions, $N_{\rm tot}A_{Q}\equiv
N({\rm A})+N({\rm C})-N({\rm B})-N({\rm D})$ Lee (1965), and the sextant
asymmetry $A_{S}$ probes $I=0$ contributions, $N_{\rm tot}A_{S}\equiv N({\rm
I})+N({\rm III})+N({\rm V})-N({\rm II})-N({\rm IV})-N({\rm VI})$ Barrett et
al. (1966); Lee (1965). All asymmetries probe C and CP violation. Figure from
Ref. Gardner and Shi (2020).

Analysis of the data is complex. Monte Carlo calculations must include
adjustments for experimental efficiencies and $\pi\pi$ interactions. The
results for the left-right (LR or $X$), quadrant (Q), and sextant (S)
asymmetries for $1.34\times 10^{6}$ $\eta\to\pi^{+}\pi^{-}\pi^{0}$ decays
collaboration et al. (2008) are:

$\displaystyle A_{LR}$ $\displaystyle=$ $\displaystyle(+0.09\pm
0.10\,_{-0.14}^{+0.10})\times 10^{-2}$ $\displaystyle A_{Q}$ $\displaystyle=$
$\displaystyle(-0.05\pm 0.10\,_{-0.05}^{+0.05})\times 10^{-2}$ $\displaystyle
A_{S}$ $\displaystyle=$ $\displaystyle(+0.08\pm 0.10\,_{-0.13}^{+0.08})\times
10^{-2}$ (70)

Observation of statistically significant asymmetries would be an evidence of
$C$ and $CP$ violation. REDTOP is particularly suitable for this measurement
since the reconstruction of charged pions in a Optical-TPC does not rely on a
magnetic field, which is usually the largest source of systematic asymmetries.
Therefore, it can vastly improve the accuracy of the measurements and resolve
the discrepancies.

Figure 11: 3-D representation of the Dalitz plot for the
$\eta\rightarrow\pi^{+}\pi^{-}\pi^{0}$ process.

A 3-dimensional representation of the X and Y variables for the
$\eta\to\pi^{+}\pi^{-}\pi^{0}$ process is shown in Fig. 11. The appearance of
terms that are odd in X would indicate both C and CP violation. The detection
of charged pions in REDTOP is based on the measurement of the Cerenkov angle
of the photons radiated in the aerogel (cf. Sec. VI.4.2). Therefore, the non-
uniformity of the magnetic field, which in general corresponds the largest
contribution to the systematic error in the asymmetry of kinematics variables
of positive and negative charged particles, plays no role at REDTOP. The
expected sensitivity for this process is currently under study with the REDTOP
detector, although it is expected to be higher than that with more
traditional, magnetic spectrometers.

####  III.2.2 CP violation in $\eta\rightarrow\pi^{+}\pi^{-}e^{+}e^{-}$

__

P and CP violation in the decay of $\eta\to\pi^{+}\pi^{-}\gamma$ has been
discussed nearly fifty years before Herczeg and Singer (1973). More recent
study Geng et al. (2002) has analyzed the CP-violating effects in this decay
by considering the photon polarizations, and predicted that a sizable linear
photon polarization could be expected in some new physics scenarios. In order
to avoid measuring the photon polarization, one can consider, as shown in Ref.
Gao (2002), the decay $\eta\rightarrow\pi^{+}\pi^{-}e^{+}e^{-}$ resulting from
the internal conversion of the photon into an $e^{+}e^{-}$ pair, and the CP-
violating effects hidden in the polarization of the photon now can be
translated into the CP asymmetry in the angular correlation of the
$e^{+}e^{-}$ plane relative to the $\pi^{+}\pi^{-}$ plane. This is actually
analogous to the neutral $K$ system, in which a large CP asymmetry, due to the
interference between the parity-conserving magnetic amplitudes and the parity-
violating electric amplitudes of
$K_{L}\rightarrow\pi^{+}\pi^{-}\gamma^{*}\rightarrow\pi^{+}\pi^{-}e^{+}e^{-}$,
has already been predicted theoretically and confirmed experimentally. Thus
the asymmetry in $\eta\rightarrow\pi^{+}\pi^{-}e^{+}e^{-}$ transition could be
found by analyzing its angular distribution Gao (2002), which is given by

${A}_{\phi}=\langle
sign(\sin\phi\cos\phi)\rangle=\frac{\displaystyle\int_{0}^{2\pi}\frac{d\Gamma(\eta\rightarrow\pi^{+}\pi^{-}e^{+}e^{-})}{d\phi}d\phi\leavevmode\nobreak\
sign(\sin\phi\cos\phi)}{\displaystyle\int_{0}^{2\pi}\frac{d\Gamma(\eta\rightarrow\pi^{+}\pi^{-}e^{+}e^{-})}{d\phi}d\phi},$
(71)

where $\phi$ is the angle between the $e^{+}e^{-}$ and $\pi^{+}\pi^{-}$ planes
in the $\eta$ rest frame (see Fig. 63).

It is obvious that the asymmetry in the flavor-conserving
$\eta\rightarrow\pi^{+}\pi^{-}e^{+}e^{-}$ decay, different from flavoring-
changing processes like $K_{L}\rightarrow\pi^{+}\pi^{-}e^{+}e^{-}$, indicates
the presence of non-standard CP-violation. It has been shown in Geng et al.
(2002); Gao (2002) that this asymmetry will arise if a relevant parity-
violating electric transition exists, and such manifestation of CP violation
in some New Physics scenarios might not be bounded by existing EDM
measurements; cf. however the discussion in Ref. Gan et al. (2022).

Experimentally, the first measurement of such asymmetry has been done by the
KLOE Collaboration in 2009 Ambrosino et al. (2009). The current best
measurement has been performed by the WASA-at-COSY Collaboration Adlarson et
al. (2016) and it is consistent with zero. However, the statistical error
(based on the production of $\sim 10^{9}$ $\eta$-mesons) largely dominates the
measurement. REDTOP’s larger statistics will improve on the systematic error
by almost two orders of magnitude, bringing the sensitivity to a level where
CP-violation could be observed. Thus, further experimental investigation of
this asymmetry might be helpful to increase our knowledge on CP violation, or
to impose some interesting constraints on some theoretical models.

Similar work can be directly generalized to
$\eta/\eta^{\prime}\to\pi^{+}\pi^{-}\ell^{+}\ell^{-}$ decays including
$\ell=e,\mu$. Very recently, the experimental measurement of such asymmetry in
$\eta^{\prime}\to\pi^{+}\pi^{-}e^{+}e^{-}$ has been performed by the BESIII
Collaboration Ablikim et al. (2021).

__

####  III.2.3 Tests of CP invariance via $\gamma^{*}$ polarization studies in
$\eta\to\pi^{+}\pi^{-}\gamma^{*}$

CP-violation can also be investigated with a virtual photon decaying into a
lepton-antilepton pair, as in

$\eta\rightarrow\pi^{+}\pi^{-}\gamma^{*}\quad\text{with}\quad\gamma^{*}\rightarrow
l^{+}+l^{-},$ (72)

by considering the asymmetry

$A_{\Phi}=\frac{N(\sin\phi\cos\phi>0)-N(\sin\phi\cos\phi<0)}{N(\sin\phi\cos\phi>0)+N(\sin\phi\cos\phi<0)}$
(73)

where $\phi$ is the angle between the decay planes of the lepton-antilepton
pair and the two charged pions. CP invariance requires $A_{\Phi}$ to vanish.
At the present, the measurement of such asymmetry performed by the WASA
collaboration Adlarson et al. (2015) is the best available, and it is
consistent with zero within the measurement errors. Unfortunately, that
measurement is largely dominated by the statistical error, from the production
of only $\sim 10^{9}\>\eta$-mesons. The larger statistics of REDTOP will
improve the systematic error by almost two orders of magnitude.

####  III.2.4 CP violation in $\eta\to\mu^{+}\mu^{-}$

The most general amplitude for $\eta\to\ell^{+}\ell^{-}$ can be effectively
parametrized as

$\mathcal{M}=g_{P}(\bar{u}i\gamma^{5}v)+g_{S}(\bar{u}v),$ (74)

with $g_{P,S}$ dimensionless parameters. The first contribution is $CP$ even
and has been computed in the SM (see Sanchez-Puertas (2019); Masjuan and
Sanchez-Puertas (2016) and references therein), while the latter represents a
$P$-odd $C$-even contribution, that is negligible in the SM. As such, any
indication of a nonzero $g_{S}$ would be a clear signal of New Physics. A
possible way to access this is through $CP$-odd lepton polarization
observables, that are of order $\mathcal{O}(g_{S}g_{P})$ and require of muon
polarimetry techniques — otherwise, the first contribution appears at
$\mathcal{O}(g_{S}^{2})$. A general caveat of these searches is that, in
general, $P$-odd $C$-even observables are tightly bound by neutron and lepton
EDMs, that make such a positive finding a priori unexpected at any foreseeable
meson factory. An exception to this was found in Ref. Sanchez-Puertas (2019),
where the New Physics were studied using the SMEFT D=6 operators. There it was
shown that, while EDM operators (or hadronic $CP$-odd operators inducing
$CP$-violating transition form factors) are severely constrained by neutron
and lepton EDMs, quark-lepton Fermi operators are less severely constrained
for the case of muons (additional constraints exists for electrons Yanase et
al. (2019)). As such, we foucs in the latter scenario which is the most
appealing case (find comments regarding $CP$-violating TFFs in double-Dalitz
decays, Sect. III.2.5). In particular, the single operators of such kind
inducing $CP$-violation are $\mathcal{O}_{\ell equ}^{(1)}$ and
$\mathcal{O}_{\ell edq}$ Grzadkowski et al. (2010), whose EDM contribution
starts at two loops, providing the necessary suppression to avoid EDM
constraints.

Figure 12: Kinematics of the process. The decaying muons’ momenta in the
$\eta$ rest frame are noted as $p_{\mu^{\pm}}$, while the $e^{\pm}$ momenta,
$p^{*}_{e^{\pm}}$, is shown in the corresponding $\mu^{\pm}$ reference frame
along with the momenta of the $\nu\bar{\nu}$ system. The $\hat{z}$ axis is
chosen along $p_{\mu^{+}}$.

Ref. Sanchez-Puertas (2019) introduced two different muon’s polarization
asymmetries,

$\displaystyle
A_{L}=\frac{N(\cos\theta>0)-N(\cos\theta<0)}{N}=\operatorname{Im}[4.1c_{\ell
edq}^{2222}-2.7(c_{\ell equ}^{(1)2211}+c_{\ell edq}^{2211})]\times 10^{-2},$
(75) $\displaystyle
A_{\times}=\frac{N(\sin\Phi>0)-N(\sin\Phi<0)}{N}=\operatorname{Im}[2.5c_{\ell
edq}^{2222}-1.6(c_{\ell equ}^{(1)2211}+c_{\ell edq}^{2211})]\times 10^{-3},$
(76)

with the first one identical if replacing $\theta\to\bar{\theta}$. Above,
$\theta(\bar{\theta})$ represents the polar angle of the $e^{+}(e^{-})$ in the
$\mu^{+}(\mu^{-})$ reference frame, with the $\hat{z}$ axis fixed by the
$\mu^{+}$ direction in the $\eta$ rest frame, see Fig. 12. The angle
$\Phi=\phi-\bar{\phi}$ is defined in Fig. 12 and reflects the sign of
$(\vec{p}_{e^{-}}\times\vec{p}_{e^{+}})\cdot\vec{p}_{\mu^{+}}$. The right hand
side in the equations above provides the contribution from the aforementioned
operators, with $c_{\mathcal{O}}^{(llqq)}$ the corresponding Wilson
coefficient for the $l(q)$-th lepton(quark) generation, respectively. The nEDM
bounds derived in Ref. Sanchez-Puertas (2019) (bounds from $D_{s}$ decays
analyzed in Ref. Sanchez-Puertas (2021) are similar, but less severe) updated
with the most recent nEDM measurement Abel et al. (2020) imply

$\operatorname{Im}c_{\ell equ}^{(1)2211}<0.001,\operatorname{Im}c_{\ell
edq}^{2211}<0.008,\operatorname{Im}c_{\ell edq}^{2222}<0.02.$ (77)

Clearly, the highest sensitivity happens for the coefficient involving muons
and strange quarks, $c_{\ell equ}^{(1)2222}$, and sets the target sensitivity
at REDTOP since one expects $A_{L}<10^{-3}$ from the bounds above.

####  III.2.5 CP violation in double Dalitz decays
$\eta\to\ell^{+}\ell^{-}\ell^{\prime+}\ell^{\prime-}$

Similar to $\eta\to\mu^{+}\mu^{-}$ decays, Ref. Sanchez-Puertas (2019) showed
that $CP$ violation effects encoded in either quark/lepton EDM as well as in
$CP$-violating hadronic operators driving $CP$-odd transition form factors are
unexpected due to EDM bounds. In addition, the sensitivity to quark-lepton
$CP$ violating interactions was studied there, while this involves an $\alpha$