# Characterization of photoreceivers for LISA

Felipe Guzmán Cervantes<EMAIL_ADDRESS><EMAIL_ADDRESS>Jeffrey Livas  Robert Silverberg  Ernest Buchanan  Robin Stebbins  NASA
Goddard Space Flight Center, Code 663, 8800 Greenbelt Road, Greenbelt, MD
20771, USA

###### Abstract

LISA will use quadrant photoreceivers as front-end devices for the phasemeter
measuring the motion of drag-free test masses in both angular orientation and
separation. We have set up a laboratory testbed for the characterization of
photoreceivers. Some of the limiting noise sources have been identified and
their contribution has been either measured or derived from the measured data.
We have built a photoreceiver with a 0.5 mm diameter quadrant photodiode with
an equivalent input current noise of better than
$1.8\,\mathrm{pA}/\sqrt{\mathrm{Hz}}$ below 20 MHz and a 3 dB bandwidth of 34
MHz.

###### pacs:

04.80.Nn, 07.60.-j, 07.87.+v, 85.60.-q, 85.60.Gz, 85.60.Dw, 85.60.Bt, 95.55.-n

##  I Introduction

The Laser Interferometer Space Antenna (LISA) is a planned gravitational wave
observatory in the frequency range of 0.1 mHz–100 mHz that consists of three
spacecraft separated by 5 million km in a nearly equilateral triangle whose
center follows the Earth in a heliocentric orbit with an orbital phase offset
of 20 degrees. Gravitational waves will be detected as distance fluctuations
between test masses moving along geodetic trajectories that are located in
different spacecraft. LISA will require low power ultra-low noise
photoreceivers for precision inter-spacecraft heterodyne laser interferometry.
Quadrant photoreceivers will be used to measure the test mass motion with a
sensitivity of 8 $\mathrm{nrad}/\sqrt{\mathrm{Hz}}$ in angular orientation and
10 $\mathrm{pm}/\sqrt{\mathrm{Hz}}$ in displacement over the frequency range
of 0.1 mHz–100 mHz jennrich-2009 . The laser beam at the transmitting
spacecraft will have a diameter of approximately 40 cm and an output laser
power at the telescope of the order of 1 W. Given the laser beam propagation
over $5\times 10^{9}$ m and accounting for losses on the beam path, from the
remote optical signal approximately 50 pW will be detected on the entire
quadrant photoreceiver. LISA will use heterodyne laser interferometry (see
Figure 1) for the inter-spacecraft displacement measurement. The incoming weak
signal will optically interfere with a stronger local oscillator $P_{LO}$. The
combined signal $P(t)$ measured at the photoreceiver can be expressed as

$P(t)=\underbrace{P_{LO}+P_{sig}}_{P_{DC}:\mathrm{\,\,DC\,\,power}\,\sim\,P_{LO}}+\underbrace{2\sqrt{P_{LO}\,P_{sig}}\,\cos\left(\Delta\omega\,t+\varphi\right)}_{P_{AC}:\mathrm{\,\,heterodyne\,\,signal}},$
(1)

where $\Delta\omega$ is the frequency difference between the interfering laser
beams (heterodyne frequency), and $\varphi$ is the interferometer phase
containing the gravitational wave information. Both ports of the beamsplitter
will be measured by quadrant detectors. Their combined information can also be
used for common-mode rejection of laser amplitude noise. The main task of the
photoreceiver development is to maintain nearly shot-noise limited performance
over a measurement bandwidth from 2–20 MHz shaddock-2006 ; bykov-2009 . This
frequency range is driven by the Doppler induced frequency variations of the
optical beat note signal due to the LISA constellation armlength changes,
given by the orbits of each spacecraft.

Figure 1: Descriptive diagram of laser heterodyne interferometric detection.

The local oscillator power $P_{LO}$ can be adjusted, according to the required
photoreceiver performance. To reduce power consumption, temperature gradients
at the optical bench due to hot spots at the photoreceivers, and to provide
additional design margin, we aim for a low-noise wide-bandwidth photoreceiver
development operating at low $P_{LO}$ levels. Using 0.5 mW local oscillator
optical power on the entire quadrant photoreceiver (of the order of 100 $\mu$W
per quadrant), and assuming a responsivity $\rho$ of 0.7 A/W for InGaAs
photodiodes at a laser wavelength of 1064 nm, the shot-noise $i_{SN}$ can be
computed as

$i_{SN}=\sqrt{2e\rho P_{DC}}\approx 10\,\mathrm{pA}/\sqrt{\mathrm{Hz}}.$ (2)

Allocating $30\%$ of the shot-noise level –
$3\,\mathrm{pA}/\sqrt{\mathrm{Hz}}$ – to the input current noise contribution
of a quadrant photoreceiver, and considering this is the quadrature sum of the
current noise of the individual quadrants, we set an input current noise goal
of $1.5\,\mathrm{pA}/\sqrt{\mathrm{Hz}}$ for the single-quadrant photoreceiver
transimpedance amplifier (TIA).

##  II Photodetector transimpedance amplifier

We have chosen a conventional DC-coupled TIA topology with a single ultra-low
noise / wide-bandwidth operational amplifier (op-amp), as shown in Figure 2.

Figure 2: Topology and noise model of photoreceiver TIA.

###  II.1 Noise model

For the TIA topology shown in Figure 2, two main input current noise sources
have been identified in the electronics:

  * •

Johnson noise ($i_{J}$) from feedback resistor $R_{f}$:

$i_{J}=\sqrt{\frac{4\,k\,T}{R_{f}}},$ (3)

where $k$ is the Boltzmann’s constant, and $T$ is the temperature (in Kelvin).

  * •

Op-amp noise properties: The current $i_{n}$ and voltage $e_{n}$ noise
properties of the op-amp contribute to the total TIA input current noise.

    * –

The op-amp current noise $i_{n}$ sums directly to the TIA input.

    * –

The op-amp voltage noise $e_{n}$ translates to current noise at the TIA input
$i_{TIA}(f)$ over the input and feedback impedances as

$i_{TIA}(f)=e_{n}\frac{\sqrt{1+(2\pi\,f\,R_{f}\,C_{T})^{2}}}{R_{f}},$ (4)

where $f$ is the frequency, and $C_{T}$ is the total circuit capacitance

$C_{T}=C_{d}+C_{f}+C_{op}+C_{s},$ (5)

including the photodiode capacitance $C_{d}$, feedback impedance $C_{f}$, op-
amp common-mode input capacitance $C_{op}$, and stray capacitances $C_{s}$
from the board, components and packaging.

The bandwidth $BW$ of the photoreceiver can be estimated as

$BW=\sqrt{\frac{GBWP}{2\pi R_{f}\,C_{T}}},$ (6)

where $GBWP$ is the gain-bandwidth product of the operational amplifier. The
total TIA input current noise $I_{\mathrm{noise}}(f)$ model can be expressed
as

$I_{noise}(f)=\sqrt{i_{T}^{2}+i_{TIA}^{2}(f)}\,\cdot\|\overline{TF}(f)\|,$ (7)

where $\|\overline{TF}(f)\|$ is the normalized TIA transfer function,
$i_{TIA}(f)$ is a frequency dependent component of the input current noise
(see Equation 4), and $i_{T}$ is the quadrature sum of various contributors
that can be approximated by neglecting their frequency dependency for modeling
purposes. For example, the expected current noise $i_{T}$ in the photoreceiver
shown in Figure 2, can be computed as

$i_{T}=\sqrt{i_{n}^{2}+i_{J}^{2}+i_{d}^{2}},$ (8)

where $i_{n}$ is the op-amp current noise, $i_{J}$ is the Johnson noise of the
feedback resistor $R_{f}$ (see Equation 3), and $i_{d}$ is the shot-noise from
the photodiode dark current. It can be seen from Equation 5 that the
photodiode capacitance and the op-amp common-mode input capacitance are
crucial factors in the total noise budget. The challenge for the photodiode
manufacture lays in achieving a minimum capacitance per unit area while
maintaining high responsivity and low leakage if reverse-biased. For the TIA
electronics, it is necessary to identify an op-amp with minimal common-mode
input capacitance, current and voltage noise, and a gain-bandwidth product
large enough to maintain the required sensitivity over the required
measurement bandwidth of 2–20 MHz.

##  III Prototype photoreceivers

###  III.1 Collaboration with industry

Under a Small Business Innovation Research (SBIR) grant, the company Discovery
Semiconductors has developed a large-area quadrant photodiode (QPD) of 1 mm
diameter and a quadrant capacitance of 2.5 pF when reverse-biased at 5 V. A
first fully integrated quadrant photoreceiver (Figure 3: QPD + TIA
electronics) performs with an equivalent input current noise of less than 3.2
pA/$\sqrt{\mathrm{Hz}}$ below 20 MHz joshi-2009 . The characteristics of this
prototype quadrant photoreceiver are:

  * •

Diameter of 1 mm with a 20 $\mu$m inter-quadrant gap.

  * •

Individual quadrant capacitance $C_{d}$ = 2.5 pF when reverse-biased at 5 V.

  * •

Dark current: 140 nA when reverse-biased at 5 V.

  * •

Responsivity at 1064 nm: $\sim 0.7$ A/W (quantum efficiency of 0.8).

  * •

TIA characteristics:

    * –

feedback impedance: $R_{f}=51\,\mathrm{k\Omega}$, $C_{f}=0.1\,\mathrm{pF}$.

    * –

op-amp ADA4817: $e_{n}=4\,\mathrm{nV}/\sqrt{\mathrm{Hz}}$,
$i_{n}=2.5\,\mathrm{fA}/\sqrt{\mathrm{Hz}}$, $C_{op}$ = 1.4 pF,
$GBWP=~{}410\,\mathrm{MHz}$.

Discovery Semiconductors has been awarded a second stage grant to further
develop quadrant photoreceivers. Given the successful development of a large-
area low-capacitance QPD in the first step, the next stage will be focused on
the noise reduction of the electronics, e.g. by integrating op-amps with
better noise properties and studying alternative TIA topologies. We expect to
receive additional devices with lower noise electronics at a later date.

Figure 3: Photograph of a prototype quadrant photoreceiver manufactured by
Discovery Semiconductors.

###  III.2 Laboratory prototypes

Working in parallel with Discovery Semiconductors to try to understand the
noise and bandwidth trade-offs in more detail, we have identified the ultra-
low noise / high-bandwidth op-amp EL5135 from Intersil intersil_el5135  with
the following nominal noise properties:

$e_{n}=1.5\,\mathrm{nV}/\sqrt{\mathrm{Hz}}$,
$i_{n}=0.9\,\mathrm{pA}/\sqrt{\mathrm{Hz}}$, $C_{op}$ = 1 pF,
$GBWP=1500\,\mathrm{MHz}$.

A significant reduction in the op-amp voltage noise together with the
bandwidth enhancement (given by the higher $GBWP$) were the main factors
considered for selecting the EL5135 for laboratory prototype photoreceivers.
Despite the higher op-amp current noise $i_{n}$ of the EL5135 compared to the
ADA4817 (mentioned above), the lower voltage noise $e_{n}$ of the EL5135
enables us to achieve significantly better performance over the entire
bandwidth from 2–20 MHz. The penalty is somewhat higher noise at low
frequencies. According to Equation 7, the term $i_{T}$ (corresponding to the
quadrature sum of various current noise contributions, including the op-amp
current noise $i_{n}$) dominates at lower frequencies, while the frequency
dependent term $i_{TIA}$ (consisting of the op-amp voltage noise $e_{n}$ swing
across the total equivalent TIA impedance) becomes the dominant noise
contribution at higher frequencies.
We have designed a TIA with a feedback impedance $R_{f}=40\,\mathrm{k\Omega}$,
$C_{f}=0.1\,\mathrm{pF}$, and an expected bandwidth of 40 MHz (according to
Equation 6). We have built two different prototype boards to test the noise
properties of a TIA with the EL5135 op-amp:

  * •

GAP500Q photoreceiver board: we have chosen the commercially available QPD
GAP500Q from GPD Optoelectronics gpdopto_gap500q  with a diameter of 0.5 mm
and a responsivity of approximately 0.7 A/W (quantum efficiency of 0.8) at
1064 nm. When reverse-biased at 5 V, this device has a nominal quadrant
capacitance $C_{d}$ = 2.0 pF and a dark current of 2.0 nA, according to the
manufacturer. The purpose of this investigation is to operate the TIA
electronics with a photodiode that approximates the per-quadrant capacitance
and package parasitic capacitance of the larger area Discovery Semiconductors
detector.

  * •

Mock-up TIA board: we have built a board for controlled noise investigations
of the TIA performance, shown in Figure 4.

Figure 4: Schematics of TIA mock-up board for noise measurements and frequency
response measurements.

This board has two inputs:

    1. 1.

input 1: an input capacitor (2.4 pF) of similar quadrant capacitance is used
to replace the photodiode. For noise measurements, this input can be grounded
while maintaining input 2 open.

    2. 2.

input 2: this is used to measure the expected photoreceiver transfer function
(TF) by injecting a signal (maintaining input 1 open), and scaling it
accordingly by the feedback gain ($40\mathrm{k}$). The equivalent input
current noise can be obtained, by dividing the output voltage noise by the
scaled transfer function.

The GAP500Q photodiode was reverse-biased with a battery power supply at 5 V,
and the op-amps in both circuits were driven with a power supply at $\pm$5 V.

##  IV Performance measurements of prototype photoreceivers

We operated the photoreceiver using only one quadrant of the GAP500Q QPD with
the EL5135 op-amp for the TIA electronics. The photoreceiver output voltage
noise $V_{n}(f)$ is given by

$V_{n}(f)=TF(f)\cdot\sqrt{i_{SN}^{2}+i_{EN}^{2}(f)},$ (9)

where $i_{SN}$ is the photocurrent shot-noise of the incident light, $TF(f)$
is the photoreceiver transfer function, and $i_{EN}(f)$ is the equivalent
input current noise of the TIA electronics. Operating under dark conditions,
the photoreceiver output voltage noise $V_{EN}(f)$ is given by

$V_{EN}(f)=TF(f)\cdot i_{EN}(f),$ (10)

By dividing Equations 9 and 10 diekmann-2008 , we obtain that the input
current noise of the TIA electronics can be computed as

$i_{EN}(f)=\sqrt{\frac{i_{SN}^{2}}{\left(\frac{V_{n}(f)}{V_{EN}(f)}\right)^{2}-1}}.$
(11)

For equivalent input current noise measurements, we used a light-emitting
diode (LED) at a center wavelength of 1050 nm ($\pm$50 nm) thorlabs_led1050e
as shot-noise-limited light source. We measured the GAP500Q photoreceiver
output voltage noise with ($V_{n}(f)$) and without ($V_{EN}(f)$) LED light,
operating at two different optical power levels of 90 $\mu$W and 60 $\mu$W
that are representative for the expected nominal 100 $\mu$W per quadrant.
These measurements showed equivalent input current noise levels $i_{EN}(f)$
that did not scale with the DC optical power level (90 $\mu$W and 60 $\mu$W).
This also shows that the measured current noise $i_{EN}(f)$ upon subtraction
of the shot-noise contribution, is not dependent on the optical power, which
is consistent with a shot-noise behavior of the light source.
We also measured the output voltage noise and the transfer function ($TF(f)$)
of our mock-up TIA board (see Figure 4). Analogous to Equation 10, the input
current noise can be computed by referring the output voltage noise (upon
subtraction in quadrature of the RF spectrum analyzer voltage noise floor) to
the input dividing by the transfer function. Figure 5 shows the noise
measurements.

Figure 5: Input current noise measurements of photoreceiver prototypes. The
dashed trace is the photoreceiver with one quadrant of the GAP500Q QPD and the
EL5135 op-amp TIA. The solid trace is the mock-up test board (see Figure 4).
The dashed-dotted trace is the photoreceiver noise model for a TIA design with
an EL5135 op-amp and a quadrant capacitance of 2.5 pF (Equations 7). The
traces with crosses and circles show the corresponding photoreceiver noise
models using parameter values obtained from a fit to the data. The dotted
trace is the equivalent current noise floor of the RF spectrum analyzer used
as measurement instrument, referred to the input by the TIA transfer function.
The thick solid traced is our TIA input current noise goal of
$1.5\,\mathrm{pA}/\mathrm{\sqrt{Hz}}$ in the measurement band 2-20 MHz.

The GAP500Q photoreceiver reaches a level of about
$1.5\,\mathrm{pA}/\mathrm{\sqrt{Hz}}$ up to $\sim$10 MHz, increasing at higher
frequencies. It exceeds the noise goal by approximately 20%
($1.8\,\mathrm{pA}/\mathrm{\sqrt{Hz}}$) at 20 MHz. The mock-up TIA circuit
meets our noise goal over the entire bandwidth (2-20 MHz), however, it shows
noise in excess of the model. At lower frequencies the equivalent input noise
is determined by excess current noise ($i_{T}$: assumed to have no frequency
dependence, Equation 8), while at higher frequencies, the increasing slope is
dominated by the op-amp voltage noise swing across the total circuit
capacitance ($i_{TIA}$: dependent on frequency, Equation 4). We run a set of
measurements in order to determine some of the involved unknowns:

  1. 1.

op-amp voltage noise ($e_{n}$): we measured the op-amp voltage noise on a
separate sample of the EL5135, by driving the op-amp as a voltage follower
with grounded input. A low-noise 10x amplifier in series with the op-amp
output was necessary for a voltage noise measurement above the spectrum
analyzer noise floor. The voltage noise level measured was
$e_{n}=2.1\,\mathrm{nV}/\mathrm{\sqrt{Hz}}$ at 10 MHz, which is significantly
higher than the specified $1.5\,\mathrm{nV}/\mathrm{\sqrt{Hz}}$.

  2. 2.

photodiode and feedback capacitances ($C_{d}$, $C_{f}$): using a LCR-bridge
impedance measurement instrument, we measured the photodiode quadrant
capacitance $C_{d}$ on a separate sample of the GAP500Q to be 3.2 pF with a 5
V reversed bias, which is higher than the nominal 2 pF. We also measured the
capacitor $C_{d}$ of the mock-up board and the feedback capacitor $C_{f}$ to
be 2.2 pF (nominally 2.4 pF) and 0.1 pF, respectively.

These noise properties are higher than specified and may account for part of
the excess noise. A direct measurement of the op-amp current noise $i_{n}$,
the op-amp input capacitance $C_{op}$, and the stray capacitances $C_{s}$ of
the circuit involve the development of dedicated electronic boards (currently
on-going) for well-controlled measurements. These measurements will be
conducted at a later stage. However, it is possible to obtain an estimate of
these values by fitting them as parameters of the noise model (Equations 7 and
8) to the two measured data sets. The noise level difference (offset) in the
data of the GAP500Q photoreceiver and the mock-up TIA is an indicator of a
significant excess current noise contribution, $i_{X}$, present in the photo-
measurement and not in the measurement of the mock-up TIA. This can also be
included into the fit by considering the following non-frequency dependent
contributions $i_{T}$ (Equation 8) for each case:

$\displaystyle\mathrm{GAP500Q\,\,photoreceiver:}$ $\displaystyle i_{T_{PD}}$
$\displaystyle=\sqrt{i_{n}^{2}+i_{J}^{2}+i_{d}^{2}+i_{X}^{2}},$ (12)
$\displaystyle\mathrm{Mock-up\,\,TIA:}$ $\displaystyle i_{T_{MU}}$
$\displaystyle=\sqrt{i_{n}^{2}+i_{J}^{2}}.$ (13)

The capacitance values $C_{d}$ and $C_{f}$ are assumed to be known from the
LCR-bridge measurements. We also assume similar op-amp and board noise
properties ($e_{n}$, $i_{n}$, $C_{op}+C_{s}$) for the two circuits. From the
fit, we obtained an op-amp current noise of $i_{n}\approx
1.1\,\mathrm{pA}/\mathrm{\sqrt{Hz}}$ (about 20% higher than nominal
$0.9\,\mathrm{pA}/\mathrm{\sqrt{Hz}}$) and a combined stray plus op-amp input
capacitance $C_{op}+C_{s}\approx 1.3\,$pF (1 pF nominal $C_{op}$). We also fit
a common op-amp voltage noise for the two data sets to be $e_{n}\approx
1.9\,\mathrm{nV}/\mathrm{\sqrt{Hz}}$, which is comparable (within $<10\%$) to
the independent measurement ($2.1\,\mathrm{nV}/\mathrm{\sqrt{Hz}}$), but about
25% higher than nominal ($1.5\,\mathrm{nV}/\mathrm{\sqrt{Hz}}$). Table 1
summarizes the current best estimates (CBE) of the photoreceiver noise
properties.

parameter | nominal | CBE | method
---|---|---|---
$C_{d}\,\left[\mathrm{pF}\right]$ | 2.0 | 3.2 | measured
$e_{n}\,\left[\mathrm{nV}/\mathrm{\sqrt{Hz}}\right]$ | 1.5 | 1.9 | fit
$i_{n}\,\left[\mathrm{pA}/\mathrm{\sqrt{Hz}}\right]$ | 0.9 | 1.1 | fit
$C_{op}+C_{s}\,\left[\mathrm{pF}\right]$ | 1.0 | 1.3 | fit
$i_{X}\,\left[\mathrm{pA}/\mathrm{\sqrt{Hz}}\right]$ | - | 0.7 | fit
Table 1: Noise parameters: comparison between nominal values and current best
estimates (CBE).

The excess current noise contribution, $i_{X}$, present in the GAP500Q
photoreceiver data was determined to be of the order of $i_{X}\approx
0.7\,\mathrm{pA}/\mathrm{\sqrt{Hz}}$. We have reversed-biased the photodiode
with a battery power supply, therefore, noise on the bias voltage translating
to excess current noise is not the cause. Additional tests are required to
determine the origin of this contribution.

##  V Conclusions and Outlook

We have presented the results of noise measurements conducted on different
photoreceiver prototypes. The measurements showed approximately 20% noise in
excess of our goal between 10–20 MHz. Direct measurements of the op-amp
voltage noise and the reverse-biased QPD quadrant capacitance evidenced noise
levels higher than nominal, accounting for part of the excess noise at higher
frequencies. By fitting the parameters of the noise model to the data, we
obtained estimates for the combined stray plus op-amp input capacitance and
the op-amp current noise $i_{n}$, which was determined to be approximately 20%
higher than nominal. Significant excess current noise (50% of total) $i_{X}$
was determined between photoconductive (GAP500Q photoreceiver) and electronic
(mock-up TIA) noise measurements. Additional testing is required to determine
its origin. The measured photoreceiver performance is of the order of
$1.5\,\mathrm{pA}/\mathrm{\sqrt{Hz}}$ below 10 MHz, increasing up to
$1.8\,\mathrm{pA}/\mathrm{\sqrt{Hz}}$ at 20 MHz with a 3 dB bandwidth of 34
MHz. However, the mock-up TIA performs at a level of
$1.35\,\mathrm{pA}/\mathrm{\sqrt{Hz}}$ below 20 MHz (10% higher than expected
from the nominal model) with a measured 3 dB bandwidth of 38 MHz. This
suggests a significantly better performance of a real photoreceiver with the
current TIA design, depending upon clarification and, if viable, mitigation of
the excess current noise $i_{X}$. In addition, as following steps, we plan to
conduct spatial scanning of the photodiode surfaces, measurement of inter-
quadrant cross-talk, and differential wavefront sensing angle measurements.

##  VI Acknowledgements

This research was supported in part by NASA contract ATFP07-0127. F. Guzmán
Cervantes is supported by an appointment to the NASA Postdoctoral Program at
the Goddard Space Flight Center, administered by Oak Ridge Associated
Universities (ORAU) through a contract with NASA. We thank A. Joshi, S. Datta
and J. Rue for stimulating discussions.

## References

  * (1) Jennrich O, LISA technology and instrumentation, Class. Quantum Grav. 26 (2009).
  * (2) Shaddock D, Ware B, Halverson P, Spero R, Klipstein B, Overview of the LISA phasemeter, AIP Conf Proc. 873 654-60 (2006).
  * (3) Bykov I, Esteban Delgado J, García Marín A, Heinzel G, Danzmann K, LISA phasemeter development: Advanced prototyping, J. Phys.: Conf. Ser. 154 (2009).
  * (4) Joshi A, Rue J, and Datta S, Low-Noise Large-Area Quad Photoreceivers Based on Low-Capacitance Quad InGaAs Photodiodes, IEEE Photonics Technology Letters, Vol.21, No.21 (2009).
  * (5) Datasheet of Intersil device EL5135: http://www.intersil.com/data/fn/fn7383.pdf
  * (6) Datasheet of GPD Optoelectonics device GAP500Q: http://www.gpd-ir.com/
  * (7) Diekmann C, Phasenstabilisierung und -auslesung für LISA, Diploma Thesis, Leibniz Universität Hannover, Germany (2008).
  * (8) Datasheet of THORLABS device LED1050E: http://www.thorlabs.com/Thorcat/16300/16388-S01.pdf


††thanks<EMAIL_ADDRESS>
# Comment on “Reproducibility and Replication of Experimental Particle
Physics Results”

Andrew Fowlie  Department of Physics and Institute of Theoretical Physics,
Nanjing Normal University, Nanjing, Jiangsu 210023, China

I would like to thank Junk and Lyons (2020) for beginning a discussion about
replication in high-energy physics (HEP). Junk and Lyons ultimately argue that
HEP learned its lessons the hard way through past failures and that other
fields could learn from our procedures. They emphasize that experimental
collaborations would risk their legacies were they to make a type-1 error in a
search for new physics and outline the vigilance taken to avoid one, such as
data blinding and a strict $5\sigma$ threshold.

The discussion, however, ignores an elephant in the room: there are regularly
anomalies in searches for new physics that result in substantial scientific
activity but don’t replicate with more data. For example, in 2015 ATLAS and
CMS showed evidence for a new particle with a mass of about $750\,\text{GeV}$
that decayed into two photons (CERN, 2015). Whilst the statistical
significance was never greater than $5\sigma$ (Khachatryan  _et al._ , 2016;
Aaboud  _et al._ , 2016), the results motivated about 500 publications about
the new particle, and countless special seminars and talks (Garisto, 2016).
The effect did not replicate when the experimental teams analyzed a larger
dataset about six months later (Aaboud  _et al._ , 2017; Khachatryan  _et al._
, 2017). Although this was a particularly egregious example, experimental
anomalies that garner considerable interest before vanishing are annual events
(Garisto, 2020).

We are motivated to attempt to control the type-1 error rate because type-1
errors damage our credibility and lead to us squandering our time and
resources on spurious effects. Whilst these non-replications aren’t strictly
type-1 errors as the statistical significance didn’t reach the $5\sigma$
threshold and no discoveries were announced, we incur similar damaging
consequences, so they cannot be ignored. I shall refer to these errors —
substantial scientific activity including publicly doubting the null and
speculating about new effects when the null was in fact true — as type-1′
errors. Whilst type-1 errors appear to be under control in HEP, type-1′ errors
are rampant. In the following sections, I discuss these errors in the context
statistical practices at the LHC.

##  I Evidence and error rates

Searches for new physics at the LHC are performed by comparing a _p-_ value,
$p$, against a pre-specified threshold, $\alpha$. There are two common
interpretations of this procedure (Hubbard and Bayarri, 2003):

  1. 1.

Error theoretic (Neyman and Pearson (1933)): By rejecting the null if
$p<\alpha$, we ensure a long-run type-1 error rate of $\alpha$. The threshold
$\alpha$ specified the desired type-1 error rate and the _p-_ value was a
means to achieving it.

  2. 2.

Evidential (Fisher (1925)): The _p-_ value is a measure of evidence of against
the null hypothesis. The threshold $\alpha$ specified a desired level of
evidence.

Even among adherents of _p-_ values, the latter interpretation is considered
unwarranted (Lakens, 2021), and it is almost never accompanied by a
theoretical framework or justification, or a discussion of the desired and
actual properties of $p$ as a measure of evidence.

Unfortunately, Junk and Lyons repeatedly implicitly switch from one to the
other. Indeed, the authors interpret $p$ as a measure of evidence and $\alpha$
as a threshold in evidence, e.g., justifying $5\sigma$ by “extraordinary
claims require extraordinary evidence” and stating that “[$3\sigma$] or
greater constitutes ‘evidence’.” We know, however, that interpreted as a
measure of evidence, $p$ is incoherent (Schervish, 1996; Wagenmakers, 2007)
and usually overstates the evidence against the null (Sellke  _et al._ , 2001;
Berger and Sellke, 1987; Berger and Delampady, 1987). For example, there
exists a famous bound (Vovk, 1993; Sellke  _et al._ , 2001) implying that
under mild assumptions $p=0.05$ corresponds to at least about $30\%$ posterior
probability of the null. This was in fact the primary criticism in Benjamin
_et al._ (2017). Consequently, one factor in the prevalence of type-1′ errors
may be that

  1. 1.

Physicists interpret _p-_ values as evidence (as do Junk and Lyons)

  2. 2.

Based on _p-_ values, physicists overestimate the evidence for new effects

  3. 3.

Substantial scientific activity on what turn out to be spurious effects

Unfortunately, _p-_ values simply can’t give researchers (including Junk and
Lyons) what they want — a measure of evidence — leading to wishful and
misleading interpretations of $p$ as evidence (Cohen, 1994). This cannot be
overcome by better statistical training; it is in inherent deficiency of _p-_
values and no amount of education about them will imbue them with a coherent
evidential meaning.

##  II Controlling errors

Controlling error rates depends critically on knowing the data collection and
analysis plan — the intentions of the researchers and what statistical tests
would be performed under what circumstances — and adjusting the _p-_ value to
reflect that. There are, however, an extraordinary number of tests performed
by ATLAS, CMS and LHCb at the LHC and elsewhere. This already makes it
challenging to interpret a _p-_ value at all and undoubtedly contributes to
the prevalence of type-1′ errors.

Junk and Lyons rightly celebrate the trend in HEP to publicly release datasets
and tools for analyzing them. This, however, raises the specter of data
dredging. Massive public datasets (CERN, 2020) combined with recent
developments in machine learning (Kasieczka  _et al._ , 2021) could enable
dredging at an unprecedented scale. We must think about what precautions we
need to prevent misleading inferences being drawn in the future; e.g., pre-
registration of planned analyses as a requisite to accessing otherwise open
data. Other more radical proposals, to the problems here and elsewhere,
include moving away from an error theoretic approach or any approach based on
_p-_ values.

##  III Final words on “Response to Andrew Fowlie’s Comments”

I would like to thank Junk and Lyons (2021) for their detailed response to my
comments. In the interests of brevity, I respond to only a few of the points
(labeled A – H in Junk and Lyons (2021)). First, I acknowledge that Junk and
Lyons (2020) aren’t mistakenly equating $p$ with the posterior of the null
(C). My previous comment is at fault if it implied otherwise. Similarly, by
“evidence” I assumed that the authors meant nothing more than observations
that should change our opinion (Kelly, 2016; Morey  _et al._ , 2016). The fact
that $p$ doesn’t equal the posterior of the null is trivial and not that
interesting. The fact that $p$ is typically much less than the posterior and
that there exist theorems demonstrating it across broad classes of priors
isn’t trivial and shouldn’t be taken lightly (B).

Second, I find it an over-simplification to say that experimental particle
physics usually only considers an error-theoretic interpretation of _p-_
values (A). In reality, as in Junk and Lyons (2020), our interpretation of $p$
is an “anonymous hybrid” (Hubbard and Bayarri, 2003) of evidential and error
theoretic. Why else would we respond at all to anomalies below $5\sigma$ if we
didn’t consider them evidence for new effects?

Lastly, there are two elements to consider regarding our responses to
anomalies. First, what is the appropriate response from our community to a
particular strength of evidence for a new effect? and second, are we misled by
$p$ about that strength of evidence, and would we respond differently if we
recognized that $p$ typically overstated the evidence or if we were able to
easily incorporate the prior plausibility of the new effect?

## References

  * Junk and Lyons (2020) Thomas R. Junk and Louis Lyons, “Reproducibility and Replication of Experimental Particle Physics Results,” Harvard Data Science Review 2, 4 (2020), arXiv:2009.06864 [physics.data-an] .
  * CERN (2015) CERN, “ATLAS and CMS physics results from Run 2,” (2015).
  * Khachatryan  _et al._ (2016) Vardan Khachatryan _et al._ (CMS), “Search for Resonant Production of High-Mass Photon Pairs in Proton-Proton Collisions at $\sqrt{s}=8$ and 13 TeV,” Phys. Rev. Lett. 117, 051802 (2016), arXiv:1606.04093 [hep-ex] .
  * Aaboud  _et al._ (2016) Morad Aaboud _et al._ (ATLAS), “Search for resonances in diphoton events at $\sqrt{s}=13$ TeV with the ATLAS detector,” JHEP 09, 001 (2016), arXiv:1606.03833 [hep-ex] .
  * Garisto (2016) Robert Garisto, “Editorial: Theorists React to the CERN 750 GeV Diphoton Data,” Phys. Rev. Lett. 116, 150001 (2016).
  * Aaboud  _et al._ (2017) Morad Aaboud _et al._ (ATLAS), “Search for new phenomena in high-mass diphoton final states using 37 fb-1 of proton–proton collisions collected at $\sqrt{s}=13$ TeV with the ATLAS detector,” Phys. Lett. B 775, 105–125 (2017), arXiv:1707.04147 [hep-ex] .
  * Khachatryan  _et al._ (2017) Vardan Khachatryan _et al._ (CMS), “Search for high-mass diphoton resonances in proton–proton collisions at 13 TeV and combination with 8 TeV search,” Phys. Lett. B 767, 147–170 (2017), arXiv:1609.02507 [hep-ex] .
  * Garisto (2020) Dan Garisto, “The Era of Anomalies,” Physics 13, 79 (2020).
  * Hubbard and Bayarri (2003) Raymond Hubbard and M. J Bayarri, “Confusion over measures of evidence ($p$’s) versus errors ($\alpha$’s) in classical statistical testing,” Am. Stat. 57, 171–178 (2003).
  * Neyman and Pearson (1933) J. Neyman and E. S. Pearson, “On the problem of the most efficient tests of statistical hypotheses,” Philos. Trans. Roy. Soc. London Ser. A 231, 289–337 (1933).
  * Fisher (1925) R. A. Fisher,  _Statistical Methods for Research Workers_ (Oliver & Boyd, 1925).
  * Lakens (2021) Daniël Lakens, “The practical alternative to the $p$ value is the correctly used $p$ value,” Perspect. Psychol. Sci. (2021).
  * Schervish (1996) Mark J. Schervish, “$P$ Values: What They are and What They are Not,” Am. Stat. 50, 203–206 (1996).
  * Wagenmakers (2007) Eric-Jan Wagenmakers, “A practical solution to the pervasive problems of $p$ values,” Psychon. Bull. Rev. 14, 779 – 804 (2007).
  * Sellke  _et al._ (2001) Thomas Sellke, M. J Bayarri, and J. O Berger, “Calibration of $p$ values for testing precise null hypotheses,” Am. Stat. 55, 62–71 (2001).
  * Berger and Sellke (1987) J. O Berger and Thomas Sellke, “Testing a Point Null Hypothesis: The Irreconcilability of $P$ Values and Evidence,” J. Am. Stat. Assoc. 82, 112–122 (1987).
  * Berger and Delampady (1987) J. O Berger and Mohan Delampady, “Testing precise hypotheses,” Statist. Sci. 2, 317–335 (1987).
  * Vovk (1993) V. G. Vovk, “A logic of probability, with application to the foundations of statistics,” J. Royal Stat. Soc. B55, 317–341 (1993).
  * Benjamin  _et al._ (2017) D. J. Benjamin, J. O Berger, M. Johannesson, B. A. Nosek, E. Wagenmakers, _et al._ , “Redefine statistical significance,” Nat. Hum. Behav. 2, 6–10 (2017).
  * Cohen (1994) Jacob Cohen, “The earth is round ($p<.05$),” Am. Psychol. 49, 997 – 1003 (1994).
  * CERN (2020) CERN, “CERN announces new open data policy in support of open science,” (2020).
  * Kasieczka  _et al._ (2021) Gregor Kasieczka _et al._ , “The LHC Olympics 2020: A Community Challenge for Anomaly Detection in High Energy Physics,” (2021), arXiv:2101.08320 [hep-ph] .
  * Junk and Lyons (2021) Thomas Junk and Louis Lyons, “Response to Andrew Fowlie’s Comments,” Harvard Data Science Review (2021), 10.1162/99608f92.ebd73e8c.
  * Kelly (2016) Thomas Kelly, “Evidence,” in  _The Stanford Encyclopedia of Philosophy_ , edited by Edward N. Zalta (Metaphysics Research Lab, Stanford University, 2016) winter 2016 ed.
  * Morey  _et al._ (2016) Richard D. Morey, Jan-Willem Romeijn, and Jeffrey N. Rouder, “The philosophy of Bayes factors and the quantification of statistical evidence,” J. Math. Psychol. 72, 6–18 (2016).