∎

e1e-mail<EMAIL_ADDRESS>
11institutetext: Max-Planck-Institut für Kernphysik, P.O. Box 103980, D 69029,
Heidelberg, Germany 22institutetext: Friedrich-Alexander-Universität Erlangen-
Nürnberg, Erlangen Centre for Astroparticle Physics, 91058 Erlangen, Germany

# Background rejection using image residuals from large telescopes in imaging
atmospheric Cherenkov telescope arrays

L. Olivera-Nieto e1,addr1  0000-0002-9105-0518 H. X. Ren addr1
0000-0003-0221-2560 A. M. W. Mitchell addr2  0000-0003-3631-5648 V. Marandon
addr1  0000-0001-9077-4058 J. A. Hinton addr1  0000-0002-1031-7760

(Received: 10 August 2022 / Accepted: 23 November 2022)

###### Abstract

Identification of Cherenkov light generated by muons has been suggested as a
promising way to dramatically improve the background rejection power of
Imaging Atmospheric Cherenkov Telescope (IACT) arrays at high energies.
However, muon identification remains a challenging task, for which efficient
algorithms are still being developed. We present an approach in which, rather
than identifying Cherenkov light from muons, we simply consider the presence
of Cherenkov light other than the main shower image in IACTs with large mirror
area. We show that in the case of the H.E.S.S. array of five telescopes this
approach results in background rejection improvements at all energies above 1
TeV. In particular, the rejection power can be improved by a factor $\sim 3-4$
at energies above 20 TeV while keeping $\sim 90\%$ of the original gamma-ray
efficiency.

††journal: Eur. Phys. J. C

##  1 Introduction

Figure 1: Large telescope images of rejected events that would be labelled as
gamma-ray candidates in the small telescope reconstruction. In both cases, the
right panel shows the ImPACT prediction associated with that event, whereas
the left panel shows the actual recorded event image. The event image shown in
the panel labelled as ”a” contains a clear muon arc, whereas the additional
feature in the image in panel ”b” is much smaller and has a simpler shape.
However, the maximum pixel intensity in this feature is more than double what
is expected from NSB noise. Note that the colorbar has been restricted so that
the fainter features are visible, since the main shower is much brighter in
both cases.

The Earth’s atmosphere is opaque to very-high-energy (VHE) radiation. Incoming
gamma-ray photons lose their energy by initiating electromagnetic particle
cascades in the atmosphere. The particles in these cascades are highly
relativistic and cause the production of Cherenkov light, which is in turn
collected by the telescope dishes of Imaging Atmospheric Cherenkov Telescopes
(IACTs). Given the huge rates of background cosmic-rays, a factor $10^{4}$
greater than those of gamma-rays background-factor , differentiating between
hadronic and electromagnetic cascades is a task critical to gamma-ray
astronomy.

IACTs have superior rejection power to other ground-based arrays in the domain
around 1 TeV, exploiting primarily the differences in shower width and
substructure between electromagnetic and hadronic showers stefanBDT . At
higher energies, of around tens of TeVs the rejection power worsens. High-
energy events are often not fully contained in the camera, which makes the
determination of the image parameters difficult. Additionally, this
parameterisation becomes dominated by the very bright central component of
high-energy showers, with additional faint emission that would indicate a
hadronic origin being overlooked.

Large numbers of muons, primarily resulting from the decay of charged pions,
are usually produced in the hadronic cascades initiated by cosmic-rays. The
potential of muons as a tool to separate between these two classes of showers
has long been recognized (see, for example Gaisser ). Recently, ourpaper
showed that for very large ($\gtrsim$20 m mirror diameter) telescopes
efficient identification of muon light can potentially lead to background
rejection levels up to $10^{-5}$ at energies above 10 TeV.

However, muon identification is not a trivial task. Muons that pass directly
through the telescope dish produce a distinguishable ring-like image in the
telescope camera. But once the ground impact distance increases beyond some
tens of meters only a fraction of the ring arc is detected (see top panel of
Figure 1). Therefore muons which arrive far from the telescopes can easily be
confused for low-energy shower images, or optical night-sky-background (NSB)
noise. Alternative approaches, like making use of the signature of arrival
time of muon light are promising, yet even then the task of muon
identification remains a challenge timing .

Nevertheless, given the magnitude of the achievable background rejection power
shown in ourpaper , less-than efficient muon identification could still result
in a significant improvement to the current background rejection power of
IACTs. We present an approach in which, rather than being concerned with
whether a recorded event contains muon light or not, we simply consider the
presence of light other than the main shower image (see bottom panel of Figure
1). This, of course, has the downside of being less precise in the sense that
light from particles other than muons - electrons, for example - may be used
to reject events. Additionally, unusually high NSB noise could lead to an
event being rejected. However, it results in a much simpler implementation
which, as shown below, still leads to improved background rejection power.

For this task we used data and simulations from the telescopes in the High
Energy Stereoscopic System (H.E.S.S.) hess2 . The H.E.S.S. experiment is
comprised of a total of five telescopes: four with a dish of 12 m diameter
referred to as CT1-4 and a central one, CT5, with a dish of 28 m diameter. The
event reconstruction is referred to as stereo when only the small CT1-4
telescopes are used, and hybrid when data from the entire array is used. In
this work we present an alternative approach in which only the data of CT1-4
is used for the event reconstruction, that is, stereo mode, but the data of
CT5 is used for an extra step of background rejection.

##  2 Background rejection with image residuals: ABRIR

ABRIR (Algorithm for Background Rejection using Image Residuals) is a
background rejection algorithm which draws additional information from the
event image taken by a large telescope, such as CT5 for the case of the
H.E.S.S. array. This algorithm is applied after the usual H.E.S.S. stereo
reconstruction, which includes an initial step of background rejection based
on Boosted Decision Trees (BDTs) stefanBDT . In particular, we use two sets of
initial cuts: the H.E.S.S. standard selection cuts (see Section 4.2 of
stefanBDT ), optimized for a Crab Nebula-like source and the so-called hard
cuts, optimized for a faint hard source. The main difference between both sets
of cuts is that while the standard cuts only require images to have a total
intensity larger than 60 photoelectrons (p.e.), the hard cuts require a
minimum of 200 p.e.. This translates into a higher energy threshold but also
higher quality events and more precise reconstructed parameters, such as core
location or energy. Additionally the threshold of the BDT parameter stefanBDT
is different, with 0.84 for the standard case and 0.8 for the hard cut. We
will denote the gamma-ray and background efficiency of this first cut as
$\eta_{BDT,\gamma}$ and $\eta_{BDT,B}$ respectively. We will apply ABRIR only
to the events that survive this initial cut. Only at this step is the data
from CT5 used, and it is exclusively CT5 images being considered by the
algorithm due to the advantaged muon detection capabilities of large
telescopes. Note that the total gamma-ray and background efficiency
($\eta_{tot,\gamma}$ and $\eta_{tot,B}$) needs to be computed as the product
of that of the initial cut and that of ABRIR, so:

$\eta_{tot,\gamma}=\eta_{\gamma}\cdot\eta_{BDT,\gamma}$
$\eta_{tot,B}=\eta_{B}\cdot\eta_{BDT,B}$

where $\eta_{\gamma}$ and $\eta_{B}$ are the gamma-ray and background
efficiency of ABRIR (see Section 3).

###  2.1 The ImPACT templates

In order to identify light as not a part of the main shower, we need to
identify the main shower itself. We do this with the help of the Image Pixel-
wise fit for Atmospheric Cherenkov Telescopes (ImPACT) algorithm impact ,
which is routinely used by the H.E.S.S experiment. ImPACT is a gamma-ray event
reconstruction algorithm that is based on the likelihood fitting of camera
pixel amplitudes to an expected image template. These templates are built with
a full Monte Carlo gamma-ray air shower simulation, followed by ray-tracing of
the telescope optics and simulation of the instrument electronics sim-telarray
; corsika . This fit is computationally expensive, so it is only performed
after a first round of background rejection is applied. For each event this
process results in an image of what the most similar gamma-ray event would
look like in each of the telescopes. We use this prediction to mask the main
component of CT5 image in order to better identify residual features. Note
that the CT5 image is not used in this likelihood fit, with the CT5 template
being derived only from the best-fit parameters of the CT1-4 fit.

###  2.2 The rejection criteria

After the main shower is masked, the remaining residual image is searched for
clusters of pixels (N). A cluster is only considered if it is comprised of
more than three neighboring pixels. For each of the clusters we compute the
maximum intensity $I_{\mathrm{max,N}}$, the total intensity
$I_{\mathrm{tot,N}}$ and the distance to the main shower $d_{\mathrm{N}}$. In
the next step, two conditions are checked against each of the clusters:

  1. (C1)

$I_{\mathrm{tot,N}}\cdot d_{\mathrm{N}}^{2}>I_{\mathrm{tot}}\cdot d^{2}$

  2. (C2)

$I_{\mathrm{max,N}}>I_{\mathrm{max}}$

where the threshold values used here for $I_{\mathrm{max}}$ and
$I_{\mathrm{tot}}\cdot d^{2}$ are 9 p.e. and 2 p.e $\cdot$ pixel2
respectively. These values were selected to maintain a gamma-ray efficiency of
around 90%, which in turn results in the background rejection performance
described in Section 3. A different requirement on the gamma-ray efficiency
would naturally impact the background rejection power, with better rejection
achievable if a higher fraction of gamma-rays is lost. Additionally, these
values are specific to the current state of the central telescope in the
H.E.S.S. array and should be adjusted as appropriate for application to other
telescopes or cameras.

The first condition (C1) selects clusters by their total intensity with a
penalization on proximity to the main shower. This is done to reject clusters
created solely by a mismatch between the outer row of pixels of the main
shower image and the predicted template. Such clusters might have a high total
intensity, but they are also very close to the main shower, which means the
product of intensity and distance will fall below the required threshold.
Additionally, this cut is always survived by relatively bright clusters that
are far from the shower, and thus have a high probability of not originating
from a mismatch between the data and the template. The second condition (C2)
aims to reject clusters resulting from uncleaned noise from the NSB. It does
so by requiring the maximum pixel intensity of a given cluster to be above a
threshold defined by the 5$\sigma$ level of the pedestal width of a typical
run. Note that this specific value depends on the image cleaning method used,
as well as on the camera performance and expected NSB level.

Events that fail either of the two conditions are kept, as well as events with
too small or no clusters outside of the main image. A schematic view of the
algorithm is shown in Figure 2. Additionally, events for which no light in the
large telescope was predicted by the template, but something is seen are also
marked as rejected, as well as the opposite case, events in which an shower
image was predicted, but all pixels in the data are below a minimal threshold
of 1 photoelectron.

Figure 2: Schematic diagram representing the algorithm structure.

##  3 Performance

Figure 3: Fraction of events kept by the ABRIR cut applied after the H.E.S.S.
standard cuts for simulated gamma-rays (blue dots), background data from off-
runs (red squares) and events taken from a radius of 0.2° from bright gamma-
ray sources (green stars), in particular the blazar PKS 2155-304 at zenith
angles of 20° and the Crab Nebula for the 40° zenith range. Note that since
PKS 2155-304 is an extragalactic source, no gamma-rays are expected to arrive
from it above a few TeV due to absorption on the extragalactic background
light. When zero events survive the cut, the 68% containment limit is drawn
assuming Poissonian statistics. Figure 4: Fraction of events kept by the ABRIR
cut applied after the H.E.S.S. hard cuts. Meaning of different panels and
colors is the same as in 3. When zero events survive the cut, the 68%
containment limit is drawn assuming Poissonian statistics.

We asses the performance of the algorithm by testing it on different types of
events. Note that both the background rejection power and the gamma-ray
efficiency presented in this section are a relative improvement on the value
for the initial H.E.S.S. standard and hard selection cuts. Those curves are
shown in Figure A1.

###  3.1 Background rejection power

In order to test the performance of the algorithm on the cosmic-ray
background, we use what is referred to as off-runs, that is, observations in
which the telescopes are pointed at fields without a known gamma-ray source.
These off-runs are typically the result of dedicated observations of known
empty fields, or also observations of extragalactic objects like dwarf
spheroidal galaxies that did not yield a gamma-ray detection dwarfsph . These
are not completely free of gamma-ray events, due to large-scale diffuse
emission like the extragalactic gamma-ray background and also due to possible
undetected faint gamma-ray sources. However, the relative fluxes of background
cosmic-rays to these gamma-ray sources make it safe to neglect this
contribution. It would also be possible to use simulations of proton-initiated
showers in order to eliminate this contamination. However, due to
uncertainties in the hadronic models used for high-energy particle
interactions harmdan ; alvaro , there are significant differences between the
properties of measured background events and those that result from
simulations. For this reason we decide to use real data only to characterize
the background, as it is more realistic and not dependent on the agreement
between data and simulations.

Figure 1 shows two example rejected events from one of these off runs which
pass the standard H.E.S.S. cuts. The right panel shows the ImPACT prediction
associated with each event, whereas the left panel shows the actual recorded
event image. For both cases, the conditions C1 and C2 are fulfilled.

Figures 3 and 4 shows the ABRIR cut efficiency for different samples as a
function of the event reconstructed energy for several observation zenith
angles. Figure 3 corresponds to the case where events surviving the H.E.S.S.
standard cuts are given as input, whereas the input events for the curves in
Figure 4 survive the hard cuts. The efficiency shown in the figures is
calculated simply as the ratio of the number of events before and after the
application of ABRIR. As can be seen, the cut rejects a significant fraction
of off-run events in both cases (red line) at all energies. Background rate
reductions of up to a factor 2.5 are obtained for energies above a few TeV for
the standard cuts case, while when using only the higher quality events
selected by the hard cut as input, this improvement goes up to factors of 3
and even 4 for energies above tens of TeV.

###  3.2 Gamma-ray efficiency

Besides rejecting a lot of suspected background events, it is important to
keep a high fraction of the true gamma-ray events. Some gamma-ray events are
expected to be flagged as rejected by the algorithm, and they belong to three
different categories:

  1. 1.

Gamma-ray events that contain muon light. As shown by ourpaper , a small
fraction of high-energy gamma-ray initiated showers will actually contain
muons, which can be detected by the large telescope. These events make up an
irreducible set of lost gamma-ray events associated with muon-tagging based
rejection.

  2. 2.

Gamma-ray events with low-altitude electrons. Since our approach does not
identify muon light as such, the light from any other particle that would
create a component in the image besides the gamma-ray shower could lead to an
event being rejected. Camera images of gamma-ray showers can contain light
from scattered electrons from the shower that emit close to the telescope
dish. This can create additional image components separated from the main
shower image. The likelihood of this effect drops rapidly as the shower core
ground impact point moves away from the telescope position.

  3. 3.

Gamma-ray events with unusually high NSB noise. A fraction of gamma-ray events
will contain noise from the NSB that is brighter than the threshold set by
condition C2 from Section 2.2. This can happen in the simulations due to
fluctuations in the NSB level, but it is also important in real data,
especially since the NSB is not the same for all regions of the sky. This
fraction is thus heavily dependent on the NSB conditions that were assumed
when the algorithm’s parameters were chosen, and also on the type of cleaning
that is used to remove the NSB contribution to the images.

We compute the gamma-ray efficiency of ABRIR by running it on simulated gamma-
ray events, using the CORSIKA package corsika  for shower and Cherenkov light
simulation and the sim-telarray package sim-telarray  for the telescope
response and camera simulation. In order to check the consistency of the
result in real data, we also check with a sample of events which reconstructed
direction falls within 0.2$\degree$ of the Crab Nebula and PKS 2155-304, which
are known bright gamma-ray sources. Note that this sample is expected to
contain some small fraction of cosmic-ray background, as cosmic-ray events are
distributed roughly uniformly on the sky. However, due to the relatively high
gamma-ray flux of these sources, the majority of events in this sample will be
true gamma-rays, and thus can be used to check the performance on real data
events. Note that this comparison is restricted to the zenith angles in which
these sources can be observed by the H.E.S.S. experiment and also by the
energies reached by the sources themselves.

Figures 3 and 4 shows the ABRIR cut efficiency for the simulated gamma-ray
sample (blue lines) as well as the events selected around known gamma-ray
sources (green line). Note that the gamma-ray efficiency is mostly flat as a
function of energy. This indicates that the group dominating the rejected
gamma-rays are those with residual features caused by the NSB, since, unlike
the expected number of muons, this does not depend on the event energy. The
gamma-ray efficiency computed with real data is consistent with that found in
simulations, which in turn confirms that the simulated NSB level is consistent
with that encountered in a typical field.

###  3.3 Application to a gamma-ray source: the Crab Nebula

Figure 5: Verification of the performance using data from the Crab Nebula. All
ratios shown in this figure are computed as the quantity after applying the
ABRIR cut divided by the same quantity before the cut. Left: Ratio of
background counts with and without the use of ABRIR for both the standard
(dark blue) and hard (orange) cuts. Middle: Gamma-ray efficiency computed as
the ratio of the resulting effective area for the datasets with and without
ABRIR (solid lines) and as the ratio of the measured excess (data points),
again for both sets of cuts using the same color scheme as in the left panel.
Right: Ratio of the measured flux from the Crab Nebula as a function of energy
with and without the use of ABRIR for both sets of initial cuts.

In order to verify the expected improved performance, we perform an analysis
of the same dataset from a real gamma-ray source with and without the use of
ABRIR and compare the results. For this, we use a total of 30 h of
observations of the Crab Nebula by the full array of the H.E.S.S. telescopes.
Note that since the goal of this paper is to demonstrate the relative
performance of this analysis technique, only ratios of the relevant quantities
will be shown.

In both cases, we first perform a stereo reconstruction of the data which
excludes the large central telescope. This results in a list of events which,
in the case without ABRIR are directly bundled with the relevant instrument
response functions (IRFs) via the Gammapy gammapy  package, as described in
lars . When applying ABRIR, an extra step is performed in which the algorithm
is applied, the flagged events are removed from the event list and the
effective areas are corrected by the gamma-ray efficiency computed from
simulations. The new event lists and IRFs are bundled in the same way as
before.

Once both datasets are ready, we can extract and compare different quantities
from them. First, we estimate the background level in a region of 0.3° radius
around the Crab location using the so-called reflected-region background
method (see berge  for a detailed description). This choice of region size is
relatively large for a standard H.E.S.S. point-source analysis, but allows for
increased statistics on the background measurement. The ratio of the measured
background before and after applying ABRIR as a function of energy can be seen
in the left panel of Figure 5, for both sets of initial cuts. The resulting
background efficiency is consistent with the one derived from the off runs
shown in Figures 3 and 4 within statistical errors. Using this measurement of
the background, we can compute the excess counts in the source region, and
compare it to the ratio of the effective areas. This is an additional
verification that the performance on simulated gamma-rays is observed in real
data. The result can be seen in the middle panel of Figure 5. Finally we can
compute the flux from the Crab nebula as a function of energy. The rightmost
panel of Figure 5 shows the ratio of flux measured before and after applying
ABRIR as a function of energy for both sets of cuts. As can be seen, they are
consistent with a ratio of unity within statistical errors, with the exception
of the lowest energy point in the standard result. This indicates a possible
mischaracterization of the effective area at the lowest energies. Given that
effective areas were not produced from scratch for the ABRIR dataset, but
rather derived from existing ones using an energy-dependent correction, it is
possible that the region near threshold, where the effective area raises
rapidly, is less accurately characterized. Dedicated IRFs or an increased
threshold would solve this issue.

###  3.4 Comparison to hybrid reconstruction

The background rejection powers achieved with the combination of the baseline
stereo algorithms and ABRIR are superior than those obtained when data from
the large telescope is included in the reconstruction from the beginning -
i.e. hybrid reconstruction. This can be seen when comparing the combination of
the ABRIR efficiencies shown in Figures 3 and 4 and the baseline stereo
efficiencies shown in Figure A1 with those of the hybrid analysis shown in
Figure A2. The reason for this lack of performance is that combining
information from different telescope sizes at the reconstruction level is a
non-trivial task, especially for the algorithms that rely on image shape
parameters. Ongoing improvements to the H.E.S.S. hybrid chain have shown
promising results and may achieve a performance comparable to that of ABRIR in
the low energy range. However, at high energies ABRIR will likely continue to
provide improved performance beyond that achieved by a hybrid analysis based
on the standard image parameters.

##  4 Discussion

We have presented an algorithm that makes use of large-dish telescopes in an
IACT array as a veto step in order to improve the background rejection. Its
use improves the background rejection power of the baseline H.E.S.S. stereo
reconstruction by a factor ranging between 2 and 4, depending on energy and
the specifics of the initial cut. The combined efficiency of this extra cut
with the baseline rejection power of the stereo reconstruction can reach
background rejection powers higher than 104 for high energies. Improved
background rejection is crucial for the detection and characterization of
extended and faint sources. Reducing the background rate also reduces the
uncertainties associated with it, leading to improved precision at high
energies. This improvement can be decisive in order to determine, for example,
the presence of energy-dependent morphology.

The different performance at high energies depending on the chosen initial cut
for the stereo reconstruction is due to their different image amplitude
thresholds. The standard cuts keep events whose image has a total brightness
of more than 60 p.e. in two of the four telescopes at least. The threshold for
the hard cuts is 200 p.e. The image of an event can have low brightness either
because the primary particle had relatively low energy, or because the
location of the shower core is far from the array. This means that some events
that are below the 200 p.e. threshold will be reconstructed with high energies
but large core distances. At large core distances, the number of muons that
are detectable by the telescope will be reduced ourpaper , meaning that the
power of the veto approach is reduced.

All results shown here are based on images for which the noise has been
cleaned using the so-called tail-cuts cleaning method, which requires each
pixel to have an intensity exceeding a threshold $I_{1}$ and a neighbour
exceeding a threshold $I_{2}$ 2006A&A…457..899A . In particular we have used
$I_{1}=7$ p.e. and $I_{2}=4$ p.e., as required for the ImPACT algorithm impact
. As mentioned in Section 3, remaining NSB noise seems to be the leading cause
of loss of gamma-ray events when applying ABRIR. A different image cleaning
approach, such as those based on the time information of the shower image
2013ICRC…33.3000S  could improve these results.

The presence of broken or unusable pixels in the camera, that is regions of
the camera which do not take data, could in principle impact the ability to
identify the presence of additional charge away from the shower
2006A&A…457..899A . However, in the case of the H.E.S.S. CT5 camera, the
number of such pixels is usually very small. There are just 6 isolated
permanently turned off pixels, and for certain sky regions, the presence of
bright stars can result in one or two more pixels being turned off Bi2022 .
This leads to a very small effect given that the camera has 1764 pixels in
total and muons are uniformly distributed across it.

We have shown here the performance of ABRIR when applied to the H.E.S.S.
array, which is made up of four middle-sized telescopes and one large, central
telescope. The future Cherenkov Telescope Array (CTA) will be made up of
telescopes of three sizes, the largest of which is smaller than the central
H.E.S.S. telescope yet still large enough to efficiently detect muons. The
idea behind the algorithm will thus be applicable to the CTA array, although
whether or not will it be a competitive technique will depend on the analysis
approaches used for mixed-telescope types and sub-array selection of the
observations. Large telescopes are typically seen as an asset only at the low
energies due to their reduced threshold. However, as we show here, including
large telescopes when observing targets for which the spectrum is expected to
extend to high energies is worthwhile given the achievable improvements
especially in background rejection.

##  Appendix A Efficiencies of the baseline H.E.S.S. cuts

Figure A1: Cut efficiencies of the stereo reconstruction for the standard and
hard cuts. Figure A2: Cut efficiencies of the hybrid reconstruction using the
standard-hybrid cuts.

We computed the background and gamma-ray efficiencies by running the H.E.S.S.
analysis on the off run events and simulated gamma-rays with and without the
baseline Boosted Decision Tree (BDT) cuts. The efficiency is then computed as
the ratio of the events passing the BDT cut to the initial number of events.
Note that this initial number comprises only events surviving the respective
image amplitude threshold for each of the cut sets (60 p.e. for standard cuts
and 200 p.e. for the hard cuts). The resulting efficiencies are shown for
different energies and zenith ranges in Figures A1 (stereo reconstruction) and
A2 (hybrid reconstruction). Note that the cuts used in the hybrid case, so-
called standard-hybrid cuts, are different from the stereo ones in terms of
thresholds, but are optimized for the same science case: a Crab Nebula-like
source.

## Declarations

Funding This research was supported by the Max Planck Society. AMWM is
supported by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) – Project Number 452934793

###### Acknowledgements.

The authors would like to thank the H.E.S.S. Collaboration for allowing the
use of H.E.S.S. data and simulations in this publication, as well as providing
useful discussions and input to the paper. This work made use of numpy numpy ,
scipy scipy , pandas pandas , matplotlib matplotlib , uproot uproot  and
ctapipe ctapipe .

## References

  * (1) W. Benbow & H. Collaboration.  _The H.E.S.S. standard analysis technique_.  In _Proc. Conf. Towards a Network of Atmospheric Cherenkov Detectors VII_ , volume 163-172 of _Proc. Conf. Towards a Network of Atmospheric Cherenkov Detectors VII_ (2005).
  * (2) S. Ohm, C. van Eldik, & K. Egberts.  _$\gamma$ /hadron separation in very-high-energy $\gamma$-ray astronomy using a multivariate analysis method_.  Astroparticle Physics, 31(5), 383 (2009).  doi:10.1016/j.astropartphys.2009.04.001.
  * (3) T. K. Gaisser, T. Stanev, F. Halzen et al.  _Gamma-ray astronomy above 50 TeV with muon-poor showers_.  Physical Review D, 43(2), 314 (1991).  doi:10.1103/PhysRevD.43.314.
  * (4) L. Olivera-Nieto, A. M. W. Mitchell, K. Bernlöhr et al.  _Muons as a tool for background rejection in imaging atmospheric Cherenkov telescope arrays_.  European Physical Journal C, 81(12), 1101 (2021).  doi:10.1140/epjc/s10052-021-09869-0.
  * (5) R. Mirzoyan, D. Sobczynska, E. Lorenz et al.  _Tagging single muons and other long-flying relativistic charged particles by ultra-fast timing in air Cherenkov telescopes_.  Astroparticle Physics, 25(5), 342 (2006).  doi:10.1016/j.astropartphys.2006.03.006.
  * (6) M. de Naurois.  _H.E.S.S.-II - Gamma ray astronomy from 20 GeV to hundreds of TeV’s_.  In _European Physical Journal Web of Conferences_ , volume 136 of _European Physical Journal Web of Conferences_ , 03001 (2017).  doi:10.1051/epjconf/201713603001.
  * (7) R. Parsons & J. Hinton.  _A Monte Carlo template based analysis for air-Cherenkov arrays_.  Astroparticle Physics, 56, 26 (2014).  ISSN 0927-6505.  doi:http://dx.doi.org/10.1016/j.astropartphys.2014.03.002.
  * (8) K. Bernlöhr.  _Simulation of imaging atmospheric Cherenkov telescopes with CORSIKA and sim_ telarray_.  Astroparticle Physics, 30(3), 149 (2008).  doi:10.1016/j.astropartphys.2008.07.009.
  * (9) D. Heck, J. Knapp, J. N. Capdevielle et al.  _CORSIKA: a Monte Carlo code to simulate extensive air showers._ Forschungszentrum Karlsruhe GmbH (1998).
  * (10) A. Abramowski, F. Aharonian, F. Ait Benkhali et al.  _Search for dark matter annihilation signatures in H.E.S.S. observations of dwarf spheroidal galaxies_.  Phys. Rev. D, 90, 112012 (2014).  doi:10.1103/PhysRevD.90.112012.
  * (11) R. D. Parsons & H. Schoorlemmer.  _Systematic differences due to high energy hadronic interaction models in air shower simulations in the 100 GeV-100 TeV range_.  Phys. Rev. D, 100(2), 023010 (2019).  doi:10.1103/PhysRevD.100.023010.
  * (12) Á. Pastor-Gutiérrez, H. Schoorlemmer, R. D. Parsons et al.  _Sub-TeV hadronic interaction model differences and their impact on air showers_.  European Physical Journal C, 81(4), 369 (2021).  doi:10.1140/epjc/s10052-021-09160-2.
  * (13) C. Deil, A. Donath, R. Terrier et al.  _gammapy/gammapy: v.0.18.2_ (2020).  doi:10.5281/zenodo.4701500.
  * (14) L. Mohrmann, A. Specovius, D. Tiziani et al.  _Validation of open-source science tools and background model construction in $\gamma$-ray astronomy_.  Astronomy and Astrophysics, 632, A72 (2019).  doi:10.1051/0004-6361/201936452.
  * (15) D. Berge, S. Funk, & J. Hinton.  _Background modelling in very-high-energy $\gamma$-ray astronomy_.  Astronomy and Astrophysics, 466(3), 1219 (2007).  doi:10.1051/0004-6361:20066674.
  * (16) F. Aharonian, A. G. Akhperjanian, A. R. Bazer-Bachi et al.  _Observations of the Crab nebula with HESS_.  A&A, 457(3), 899 (2006).  doi:10.1051/0004-6361:20065351.
  * (17) M. Shayduk & C. Consortium.  _Optimized Next-neighbour Image Cleaning Method for Cherenkov Telescopes_.  In _International Cosmic Ray Conference_ , volume 33 of _International Cosmic Ray Conference_ , 3000 (2013).
  * (18) B. Bi, M. Barcelo, C. Bauer et al.  _Performance of the new FlashCam-based camera in the 28m telescope of H.E.S.S._ In _37th International Cosmic Ray Conference_ , 743 (2022).  doi:10.22323/1.395.0743.
  * (19) C. R. Harris, K. J. Millman, S. J. van der Walt et al.  _Array programming with NumPy_.  Nature, 585, 357–362 (2020).  doi:10.1038/s41586-020-2649-2.
  * (20) P. Virtanen, R. Gommers, T. E. Oliphant et al.  _SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python_.  Nature Methods, 17, 261 (2020).  doi:10.1038/s41592-019-0686-2.
  * (21) W. McKinney et al.  _Data structures for statistical computing in python_.  In _Proceedings of the 9th Python in Science Conference_ , volume 445, 51–56. Austin, TX (2010).
  * (22) J. D. Hunter.  _Matplotlib: A 2D graphics environment_.  Computing in science & engineering, 9(3), 90 (2007).
  * (23) J. Pivarski, H. Schreiner, N. Smith et al.  _scikit-hep/uproot4: 4.2.3_ (2022).  doi:10.5281/zenodo.6540484.
  * (24) K. Kosack, J. Watson, M. Nöthe et al.  _cta-observatory/ctapipe: v0.11.0_ (2021).  doi:10.5281/zenodo.4746317.


# Correlations of sieve weights and distributions of zeros

Aled Walker

###### Abstract.

In this note we give two small results concerning the correlations of the
Selberg sieve weights. We then use these estimates to derive a new
(conditional) lower bound on the variance of the primes in short intervals,
and also on the so-called ‘form factor’ for the pair correlations of the zeros
of the Riemann zeta function. Our bounds ultimately rely on the estimates of
Bettin–Chandee for trilinear Kloosterman fractions.

##  1\. Introduction

For $z\geqslant 1$, we define the weight

$\rho_{z,d}:=\frac{d\mu(d)}{\varphi(d)}\sum\limits_{\begin{subarray}{c}q\leqslant
z/d\\\ (q,d)=1\end{subarray}}\frac{\mu^{2}(q)}{\varphi(q)},$ (1.1)

writing

$\lambda_{z}(n):=\sum\limits_{d|n}\rho_{z,d}.$ (1.2)

These weights arise in the theory of the Selberg sieve. A classical estimate
(proved in [13, Lemma 2], for instance) reads

$\sum\limits_{n\leqslant X}\lambda_{z}(n)^{2}=XL(z)+O(z^{2}),$ (1.3)

where

$L(z):=\sum\limits_{q\leqslant z}\frac{\mu^{2}(q)}{\varphi(q)}.$ (1.4)

Since $\lambda_{z}(n)=L(z)$ if all of the prime factors of $n$ are greater
than $z$, expression (1.3) may be used as an upper-bound sieve for an
interval.

However, sieving is not our present concern. We are motivated instead by a
different strand of the literature, in which $\lambda_{z}(n)$ acts as a proxy
for the von Mangoldt function $\Lambda(n)$ and information concerning
difficult properties of $\Lambda(n)$ may be gleaned from studying the more
tractable properties of $\lambda_{z}(n)$. Elements of this approach date back
to Heath-Brown and his work on the ternary Goldbach problem [18], and it also
appears in work of Goldston [13] on the variance of primes in short intervals,
Friedlander–Goldston [12] on the distribution of primes in arithmetic
progressions, Goldston–Yildrim [17] on the variance of primes in short
arithmetic progressions, as well as in the work of Goldston–Gonek–Özlük–Snyder
[15] on the distribution of the zeros of the Riemann zeta function (which we
will address at length below).

In certain contexts the functions derived from $\lambda_{z}(n)$ seem to have
been called ‘Vaughan’s approximation’, after Vaughan’s papers [29, 30] which
considered in detail the quality of the approximation

$\Big{|}\sum\limits_{\begin{subarray}{c}n\leqslant x\\\ n\equiv a\,(\text{mod
}q)\end{subarray}}\lambda_{z}(n)-\psi(x;q,a)\Big{|}$

for small $z$. This terminology is used in Fiorilli’s work [9], say.

We will be interested in two estimates for $\lambda_{z}(n)$: the estimate
(1.3) and the off-diagonal correlations $\lambda_{z}(n)\lambda_{z}(n+k)$. An
old result for these off-diagonal correlations is as follows:

######  Proposition 1.1.

Let $1\leqslant z\leqslant X$, and let $k$ be an integer such that
$1\leqslant|k|\leqslant X$. Let $X_{1}=\max(0,-k)$ and $X_{2}=\min(X,X-k)$.
Then

$\sum\limits_{X_{1}<n\leqslant
X_{2}}\lambda_{z}(n)\lambda_{z}(n+k)=\mathfrak{S}(k)(X-|k|)+O\Big{(}\frac{k\tau(k)X}{\varphi(k)z}\Big{)}+O(z^{2}),$
(1.5)

where

$\mathfrak{S}(k)=\begin{cases}2\prod\limits_{p>2}\Big{(}1-\frac{1}{(p-1)^{2}}\Big{)}\cdot\prod\limits_{\begin{subarray}{c}p>2\\\
p|k\end{subarray}}\Big{(}\frac{p-1}{p-2}\Big{)}&k\text{ even;}\\\ 0&k\text{
odd}.\end{cases}$ (1.6)

###### Proof.

See [13, Lemma 2]. ∎

Estimates (1.3) and (1.5) are non-trivial only in the range $\log X/\log z>2$,
which is the familiar ‘square-root barrier’ for the upper-bound sieve. Our
main technical results in this note extend the range of applicable $z$ beyond
$X^{1/2}$.

For the off-diagonal case, we can proceed unconditionally.

######  Proposition 1.2.

With notation as in Proposition 1.1, for any $\varepsilon>0$ we have

$\sum\limits_{X_{1}<n\leqslant
X_{2}}\lambda_{z}(n)\lambda_{z}(n+k)=\mathfrak{S}(k)(X-|k|)+O\Big{(}\frac{k\tau(k)X}{\varphi(k)z}\Big{)}+O_{\varepsilon}(\min(z^{2},X^{\frac{3}{4}+\varepsilon}z^{\frac{15}{32}})).$

For fixed $k$ this bound is non-trivial if $X^{\varepsilon}\leqslant
z\leqslant X^{\frac{8}{15}-\varepsilon}$. (Here and throughout we follow the
convention that $\varepsilon$ refers to a small parameter that may change from
line to line.)

Our proof of Proposition 1.2 is short, and ultimately relies on the deep
results of Bettin–Chandee on trilinear Kloosterman fractions [1] (which are
themselves based on the pioneering work of Duke–Friedlander–Iwaniec [6] on
bilinear Kloosterman fractions). The result is new, but it came to our
attention while preparing the final version of our manuscript that something
similar does appear in earlier work of Coppola–Laporta [5, Lemma 3]. Their
work only used the older DFI bound, and therefore only produced a non-trivial
error term in the weaker range $z\leqslant X^{48/95}$; furthermore their way
of proceeding from the Kloosterman fraction bounds to correlation bounds was
rather complicated (as they didn’t smooth their sums, as we will). However,
the core ideas seem to be broadly similar.

The work of Bettin–Chandee was also used by Fouvry–Radziwiłł [10], who have
established very strong equidstribution results for bilinear sums in
arithmetic progressions. In turn, they used these bounds (in [10, Corollary
1.5]) to establish equidistribution of sieve weights over certain arithmetic
progressions with large modulus, where the level of distribution of the sieve
weight can be as large as $X^{1/2+\delta}$ for some $\delta>0$. Their
Corollary 1.5 cannot be directly applied for our purposes (the range of
permissible arithmetic progressions is too restrictive and they only save a
power of $\log X$), but it seems likely that their methods could be adapted to
yield a version of Proposition 1.2. On balance, then, we feel that the our
main contribution here is the direct manner of proof rather than the statement
itself.

For the diagonal case, we are forced to proceed conditionally to obtain a
power saving.

######  Proposition 1.3 (Method of Hooley, [19]).

Assume the Riemann Hypothesis. Then, for all $1\leqslant z\leqslant X$ and for
all $\varepsilon>0$,

$\sum\limits_{n\leqslant
X}\lambda_{z}^{2}(n)=XL(z)+O_{\varepsilon}(\min(z^{2},X^{1/2+\varepsilon}z^{1/2})).$

This bound is non-trivial if $z\leqslant X^{1-\varepsilon}$. As we will
explain in Section 3, the proof of Proposition 1.3 is a straightforward
adaptation of a method of Hooley, who studied the weight $\lambda_{z}(n)$ as
part of his long series of papers on the Barban–Davenport–Halberstam theorem.
Assuming RH is of course regrettable, but, for the applications we have in
mind, the assumption of RH is par for the course.

The proofs of Propositions 1.2 and 1.3 will be deferred to Sections 2 and 3
respectively. For the rest of this introduction, we will discuss our two
applications.

Firstly, we wish to study the pair correlations of the zeros of the Riemann
zeta function. In particular we wish to understand the behaviour of the
function $F_{T}(\alpha)$, which was first introduced by Montgomery in his
seminal paper [24].

######  Definition 1.4 (Two point form factor111This is terminology from
random matrix theory, which we learnt from Lagarias–Rogers [21].).

For $\alpha\in\mathbb{R}$ and $T\geqslant 2$ we define

$F_{T}(\alpha):=\Big{(}\frac{T\log
T}{2\pi}\Big{)}^{-1}\sum\limits_{0<\gamma,\gamma^{\prime}\leqslant
T}e\Big{(}\alpha\frac{\log
T}{2\pi}(\gamma-\gamma^{\prime})\Big{)}w(\gamma-\gamma^{\prime}),$

where $w(u)=4/(4+u^{2})$, and where $\gamma,\gamma^{\prime}$ range over the
imaginary parts of the zeros of the Riemann zeta function $\zeta(s)$ (counted
with multiplicity). Here, as throughout, we let $e(\alpha)$ be a shorthand
notation for $e^{2\pi i\alpha}$.

Apart from the weight function $w$, which seems a little odd at first sight,
the function $F_{T}(\alpha)$ is evidently a natural object to consider.
Indeed, it is an old fact, going back to Riemann, that the number of zeros
with imaginary part between $0$ and $T$ is asymptotically equal to
$\frac{1}{2\pi}T\log T$. The average gap between consecutive zeros is
therefore $2\pi/\log T$, and so the function $F_{T}(\alpha)$ measures the
(weighted) Fourier transform of the gaps between the zeros, measured at the
scale of the average gap.

We observe the trivial facts that
$F_{T}(\alpha)=F_{T}(-\alpha)=\overline{F_{T}(\alpha)}$, so from now on we
only consider $\alpha\geqslant 0$. In [24], Montgomery proved the following
asymptotic result for $F_{T}(\alpha)$:

######  Theorem 1.5 (Theorem 1, [24]).

Assume the Riemann Hypothesis. Let $\varepsilon>0$ and let
$0\leqslant\alpha\leqslant 1-\varepsilon$. Then

$F_{T}(\alpha)=T^{-2\alpha}(\log
T)(1+o_{\varepsilon}(1))+\alpha+o_{\varepsilon}(1)$

as $T\rightarrow\infty$.

In particular, for a fixed $\alpha\in(0,1)$ one has

$F_{T}(\alpha)=\alpha+o(1)$

as $T\rightarrow\infty$.

Montgomery’s proof of Theorem 1.5 used a version of the approximate functional
equation for $\zeta(s)$, in order to link $F_{T}(\alpha)$ to a variance
estimate for the Dirichlet polynomial $\sum_{n\leqslant
T^{\alpha}}\Lambda(n)n^{s}$. Based on this method, together with some
conjectures on the size of the error term in the twin prime conjecture,
Montgomery was moved to posit the following:

######  Conjecture 1.6 (Montgomery’s pair correlation conjecture).

For each fixed $\alpha\in[1,\infty)$ one has

$F_{T}(\alpha)=1+o(1)$

as $T\rightarrow\infty$.

There are now substantial computations of zeros which suggest that Conjecture
1.6 is true (see [27]).

As we will discuss later in this introduction, Conjecture 1.6 is tied up with
deep issues concerning the variance of the number of primes in short intervals
[16]. Nonetheless, there are some intriguing partial results, particularly the
following lower-bound of Goldston–Gonek–Özlük–Snyder.

######  Theorem 1.7 (G–G–Ö–S,[15]).

Assume the Generalised Riemann Hypothesis for Dirichlet $L$-functions. Then
for any $\varepsilon>0$ one has

$F_{T}(\alpha)\geqslant\frac{3}{2}-\alpha-\varepsilon,$ (1.7)

provided $1\leqslant\alpha\leqslant\frac{3}{2}-\varepsilon$ and $T\geqslant
T_{0}(\varepsilon)$.

In particular, for fixed $\alpha\in[1,3/2)$ we have

$\liminf\limits_{T\rightarrow\infty}F_{T}(\alpha)\geqslant\frac{3}{2}-\alpha.$
(1.8)

This bound can be compared with the bound under the so-called ‘Alternative
Hypothesis’ (AH). This is a pathological distribution for the imaginary parts
of the zeros of $\zeta(s)$, which, among other things, would imply that

$\lim\limits_{T\rightarrow\infty}F_{T}(\alpha)=2-\alpha$

for $\alpha\in(1,2).$ The relationship between the bounds in this paper and AH
will be discussed in Section 5 of this note.

Theorem 1.7 seems to be the limit of what one can prove without assuming a
stronger hypothesis than GRH. The question of whether a better lower bound can
be proved under stronger hypotheses was taken up in Juhas’ thesis [20], in
which the following theorem was proved:

######  Theorem 1.8 (Theorem 9 of [20]).

Assume that for all $X$, for all $1\leqslant m\leqslant X^{1/2}$, and for all
non-principal Dirichlet characters $\chi\,(\text{mod }m)$ one has

$|\psi(X,\chi)|\ll_{\varepsilon}X^{1/2+\varepsilon}m^{-1/4},$ (1.9)

where $\psi(X,\chi)=\sum_{n\leqslant X}\Lambda(n)\chi(n)$ as usual. (This is
‘Hypothesis M’ in [20].) Then, for all $\varepsilon>0$ one has

$F_{T}(\alpha)\geqslant\frac{5}{4}-\frac{3}{4}\alpha-\varepsilon,$

provided $1\leqslant\alpha\leqslant\frac{5}{3}-\varepsilon$ and $T\geqslant
T_{0}(\varepsilon)$.

This argument makes use of some ingenious extra averaging over $k$ for the
correlations $\sum_{n}\lambda_{z}(n)\lambda_{z}(n+k)$ that is not present in
[15] (see Remark 2.3). However, the hypothesis (1.9) seems to us to be
unreasonably strong, as it would represent stronger than square-root
cancellation in the sum $\psi(X,\chi)$ and would contradict various other
conjectures in the literature. For instance, (1.9) would imply that

$\sum_{Q<q\leqslant 2Q}\sum\limits_{a\leqslant
q}\Big{|}\psi(X;q,a)-\delta_{(a,q)=1}\frac{X}{\varphi(q)}\Big{|}^{2}\ll_{\varepsilon}Q^{1/2}X^{1+\varepsilon}$

when $Q<X^{1/2}$. This would contradict Conjecture 1.1 of Fiorilli’s paper [8]
regarding the range over which the asymptotic Barban–Davenport–Halberstam
theorem can be expected to hold. If (1.9) were true for $m\leqslant
X^{2/3+\delta}$ for some small positive $\delta$ then it would contradict what
is known on GRH regarding the variance of the primes in arithmetic
progressions (see expression (5) of [8] and the references therein). And
finally, if (1.9) were true for $m\leqslant X^{1/2+\delta}$ for some small
positive $\delta$ then it would contradict Theorem 3 of Friedlander-Goldston
[12]. All in all, we feel that there is merit in improving the lower bound in
Theorem 1.7 subject to a more standard conjecture than (1.9).

Our bound will be conditional upon the following conjecture:

######  Conjecture 1.9 (Strong error term for primes in APs, $AP(\theta)$).

Let $0<\theta\leqslant 1$. Then $AP(\theta)$ is the conjecture that for all
$x\geqslant 1$, $q<x^{\theta}$, and $a\leqslant q$,

$\psi(x;q,a)=1_{(a,q)=1}\frac{x}{\varphi(q)}+O_{\varepsilon,\theta}\Big{(}\frac{x^{1/2+\varepsilon}}{q^{1/2}}\Big{)}.$

Although $AP(\theta)$ is of course far beyond what is currently known, even on
average over $q$, there is a history of this conjecture in the literature.
With $\theta=1$, $AP(\theta)$ was first formulated by Friedlander–Granville
[11, Conjecture 1(b)]. It is often attributed to Montgomery, however, since in
[23, (15.9)] Montgomery had proposed a similar version (the same as $AP(1)$
except with an error term of $(x/q)^{1/2+\varepsilon}$). By using an argument
based on the Maier matrix method, Friedlander–Granville showed that
Montgomery’s original conjecture was unreasonably strong for the very largest
moduli $q$. They proposed $AP(1)$ as a sensible alternative, one which would
nonetheless represent approximately square-root cancellation in the error
term.

Our lower-bound on $F_{T}(\alpha)$, conditional on $AP(\theta)$, is as
follows:

######  Theorem 1.10 (Lower bound on $F_{T}(\alpha)$).

Assume Conjecture 1.9 for $\theta=8/15$. Then for all for all $\varepsilon>0$,
for all $\alpha\in[1,49/48-\varepsilon)$, and for all $T\geqslant
T_{0}(\varepsilon)$, we have

$F_{T}(\alpha)\geqslant\frac{47}{15}-\frac{39\alpha}{15}-\varepsilon.$

Beyond $\alpha=49/48$, we are not able to offer an improvement over Theorem
1.7.

On the one hand, one could consider it unsurprising that the assumption of a
conjecture such as $AP(\theta)$, for some $\theta>1/2$, could lead to a
stronger conclusion than the assumption of GRH alone. However, as will become
clear in Section 4 when we describe the method of G–G–Ö–S, several different
terms naturally occur in this approach, and $AP(\theta)$ only helps us to
estimate one of them.

One might nonetheless be concerned that $AP(\theta)$ lies deeper than
Conjecture 1.6, thus rendering ‘content-free’ any partial result towards
Conjecture 1.6 that is conditional upon $AP(\theta)$. Of course, given two
conjectures which both lie far beyond the reach of the present field, it is
hard to say categorically which one lies deeper than the other. However, we
have a line of argument that suggests to us that assuming strong results on
the distribution of primes in arithmetic progressions is not a ridiculous
move. Indeed, all the prior results that establish Conjecture 1.6 for some
range of $\alpha\geqslant 1$ have been contingent on one of two conjectures:
either the authors assume an asymptotic for the variance of the prime counting
function $\psi(X)$ in short intervals, which on RH turns out to be equivalent
to Conjecture 1.6 (see Goldston–Montgomery [16]); or the authors assume a
power-saving in the correlations

$\sum\limits_{n\leqslant
X}\Lambda(n)\Lambda(n+k)=\mathfrak{S}(k)X+O(X^{\eta}),$ (1.10)

which, on RH, establishes Conjecture 1.6 in the range $|\alpha|<\eta^{-1}$.
One may consult Theorem 3 of Montgomery– Soundararajan’s paper [25] for an
example in which power-saving asymptotics for the full Hardy–Littlewood
$k$-tuple conjecture are assumed in order to get some handle on the
distribution of primes in short intervals (and hence on Conjecture 1.6). So it
seems reasonable to suggest that Conjecture 1.6 lies deeper than estimates
such as (1.10). Now, according to the present state of the field, a parity-
breaking conjecture such as the Hardy–Littlewood prime $k$-tuples conjecture
is not implied by even the strongest of conjectures on the distribution of
primes in arithmetic progressions. One may consider the classical Bombieri
sieve [2], say, for an example of this fact.

In summary, a ‘cheap’ direct path from strong results on primes in arithmetic
progressions to statements concerning the function $F_{T}(\alpha)$ does not
seem to presently exist.

The proof of Theorem 1.10 will be given in Section 4. We do not offer any
structural innovations on the technique of G–G–Ö–S from Theorem 1.7; instead,
our improvement comes from inputting our stronger auxiliary estimates into
their scheme.

Our second application of Propositions 1.2 and 1.3 is to the task of proving a
lower bound on the variance of the number of primes in short intervals. This
variance is closely related to the pair correlations of the zeros of the
Riemann zeta function. As we have already mentioned, on RH the following
conjecture is equivalent to Conjecture 1.6 (see Goldston–Montgomery [16]).

######  Conjecture 1.11.

For all $\varepsilon>0$, if $h\leqslant X^{1-\varepsilon}$ then

$\int\limits_{0}^{X}(\psi(x+h)-\psi(x)-h)^{2}\,dx=(1+o_{\varepsilon}(1))hx\log(x/h)$

as $X\rightarrow\infty$.

Similar methods to those used to prove Theorem 1.7 can be used to provide
lower bounds on this variance. As far as we are aware, the strongest result
that is currently known, on GRH, is the following (due to Goldston–Yıldırım):

######  Theorem 1.12 (Theorem 1 of [17] with $q=1$).

Assume the Generalised Riemann Hypothesis for Dirichlet $L$-functions. For all
$\varepsilon>0$, if $1\leqslant h\leqslant X^{1/3-\varepsilon}$ then

$\int\limits_{0}^{X}(\psi(x+h)-\psi(x)-h)^{2}\,dx\geqslant\Big{(}\frac{1}{2}-o_{\varepsilon}(1)\Big{)}hx\log(x/h^{3})$

On conjecture $AP(\theta)$, we are able to increase the lower bound in Theorem
1.12 (at least for small $h$).

######  Theorem 1.13.

Assume Conjecture $AP(\theta)$ for $\theta=\frac{8}{15}$. Then, for all
$\varepsilon>0$, if $1\leqslant h\leqslant X^{\frac{1}{49}-\varepsilon}$ then

$\int\limits_{X}^{2X}(\psi(x+h)-\psi(x)-h)^{2}\,dx\geqslant\Big{(}\frac{8}{15}-o_{\varepsilon}(1)\Big{)}hx\log(x/h^{4}).$

We stress again that there are several terms which occur in the method from
[17], only one of which is directly improved by assuming $AP(\theta)$ over
assuming GRH.

Acknowledgements: We would like to thank Thomas Bloom for making us aware of
the work in [15] and for many interesting conversations on the topic of
$\lambda_{z}(n)$ and $F_{T}(\alpha)$. We would also like to thank Andrew
Granville and Dimitris Koukoulopoulos for their comments and advice, and to
Jöni Teräväinen for comments on an earlier draft of the manuscript.

##  2\. Proof of Proposition 1.2

We will prove the following general result concerning correlations of sieve
weights.

######  Proposition 2.1.

Let $1\leqslant|k|\leqslant X$ and let $X_{1}=\max(0,-k)$ and
$X_{2}=\min(X,X-k)$. Let $2\leqslant z\leqslant X$, and let
$\rho^{*}_{z}:\mathbb{N}\longrightarrow\mathbb{R}$ be any function such that
$\rho_{z}^{*}(d)=0$ for $d>z$ and $|\rho_{z}^{*}(d)|\leqslant B$ for all $d$.
Let $\lambda^{*}_{z}(n)=\sum_{d|n}\rho_{z}^{*}(d)$. Then

$\sum\limits_{X_{1}<n\leqslant
X_{2}}\lambda_{z}^{*}(n)\lambda_{z}^{*}(n+k)=(X-|k|)\sum\limits_{\begin{subarray}{c}d_{1},d_{2}\leqslant
z\\\
(d_{1},d_{2})|k\end{subarray}}\frac{\rho^{*}_{z}(d_{1})}{d_{1}}\frac{\rho^{*}_{z}(d_{2})}{d_{2}}(d_{1},d_{2})+O_{\varepsilon}(B^{2}X^{\frac{3}{4}+\varepsilon}z^{\frac{15}{32}})).$

###### Proof of Proposition 1.2 assuming Proposition 2.1.

For all $d$ we have

$|\rho_{z,d}|\leqslant L(z).$

Indeed, $\rho_{z,d}$ is only supported on square-free $d$, and for such $d$ we
have

$\displaystyle
L(z)=\sum\limits_{k|d}\sum\limits_{\begin{subarray}{c}q\leqslant z\\\
(q,d)=k\end{subarray}}\frac{\mu^{2}(q)}{\varphi(q)}=\sum\limits_{k|d}\frac{\mu^{2}(k)}{\varphi(k)}\sum\limits_{\begin{subarray}{c}q\leqslant
z/k\\\
(q,d)=1\end{subarray}}\frac{\mu^{2}(q)}{\varphi(q)}\geqslant\Big{(}\sum\limits_{k|d}\frac{\mu^{2}(k)}{\varphi(k)}\Big{)}\Big{(}\sum\limits_{\begin{subarray}{c}q\leqslant
z/d\\\ (q,d)=1\end{subarray}}\frac{\mu^{2}(q)}{\varphi(q)}\Big{)}$
$\displaystyle=\frac{d}{\varphi(d)}\sum\limits_{\begin{subarray}{c}q\leqslant
z/d\\\ (q,d)=1\end{subarray}}\frac{\mu^{2}(q)}{\varphi(q)}=|\rho_{z,d}|.$

Now, from the definition of $\rho_{z,d}$ and swapping orders of summation,

$\sum\limits_{\begin{subarray}{c}d_{1},d_{2}\leqslant z\\\
(d_{1},d_{2})|k\end{subarray}}\frac{\rho_{z,d_{1}}}{d_{1}}\frac{\rho_{z,d_{2}}}{d_{2}}(d_{1},d_{2})=\sum\limits_{q_{1},q_{2}\leqslant
z}\frac{\mu^{2}(q_{1})}{q_{1}}\frac{\mu^{2}(q_{2})}{q_{2}}\sum\limits_{\begin{subarray}{c}d_{1}|q_{1}\\\
d_{2}|q_{2}\\\
(d_{1},d_{2})|k\end{subarray}}\mu(d_{1})\mu(d_{2})(d_{1},d_{2}).$

By expression (2.10) of Goldston’s paper [13], this equals

$\mathfrak{S}(k)+O\Big{(}\frac{k\tau(k)}{\varphi(k)z}\Big{)},$ (2.1)

where $\mathfrak{S}(k)$ is as in (1.6). Hence Proposition 1.2 follows from
Proposition 2.1. ∎

###### Proof of Proposition 2.1.

We begin by introducing a smooth cut-off function. Let $\delta>0$. It is a
well-known construction that there is a function $f_{\delta}\in
C_{c}^{\infty}(\mathbb{R})$ such that $f_{\delta}(x)\in[0,1]$ for all $x$,
$f_{\delta}(x)\equiv 1$ for all $x\in[1/2,1]$, $f_{\delta}(x)\equiv 0$ for all
$x<1/2-\delta$ and for all $x>1+\delta$, and

$|\widehat{f_{\delta}}(\alpha)|\ll_{K}\delta^{-K}|\alpha|^{-K}$ (2.2)

for all $\alpha\in\mathbb{R}$ and natural numbers $K$. For instance, one can
use the construction in Lemma 3.1 of [31], adapted to the interval $[1/2,1]$
rather than $[0,1]$.

Let $N$ be a dyadic scale for the variable $n$, such that
$X_{1}<N/2<N\leqslant X_{2}$. Owing to the bound

$|\lambda^{*}_{z}(n)|\leqslant B\tau(n),$

we have

$\displaystyle\sum\limits_{N/2<n\leqslant
N}\lambda_{z}^{*}(n)\lambda_{z}^{*}(n+k)=$
$\displaystyle\sum\limits_{n\in\mathbb{Z}}\lambda_{z}^{*}(n)\lambda_{z}^{*}(n+k)f_{\delta}(n/N)$
$\displaystyle+O\Big{(}B^{2}\Big{(}\sum\limits_{\begin{subarray}{c}N/2-\delta
N\leqslant n<N/2\\\ N<n\leqslant N+\delta
N\end{subarray}}\tau(n)\tau(n+k)\Big{)}\Big{)}.$ (2.3)

The divisor sum correlation is at most

$\sum\limits_{N/2-\delta N\leqslant n<N/2}\tau(n)^{2}+\sum\limits_{N/2-\delta
N+k<n\leqslant N/2+k}\tau(n)^{2},$

plus an analogous sum over the interval $N<n\leqslant N+\delta N$. Since

$\sum\limits_{m\leqslant M}\tau(n)^{2}=MP_{3}(\log
M)+O_{\varepsilon}(M^{1/2+\varepsilon}),$

for some polynomial $P_{3}$ of degree $3$, we observe that the error term in
(2) is at most

$O_{\varepsilon}(B^{2}(\delta X(\log X)^{3}+X^{1/2+\varepsilon})).$ (2.4)

We proceed to the main term. We have

$\displaystyle\sum\limits_{n\in\mathbb{Z}}\lambda_{z}^{*}(n)\lambda_{z}^{*}(n+k)f_{\delta}(n/N)=\sum\limits_{\begin{subarray}{c}d_{1}\leqslant
z\\\ d_{2}\leqslant z\\\
(d_{1},d_{2})|k\end{subarray}}\rho_{z}^{*}(d_{1})\rho_{z}^{*}(d_{2})\sum\limits_{\begin{subarray}{c}n\in\mathbb{Z}\\\
d_{1}|n\\\ d_{2}|n+k\end{subarray}}f_{\delta}(n/N)$
$\displaystyle=\sum\limits_{d|k}\sum\limits_{\begin{subarray}{c}d_{1}\leqslant
z/d\\\ d_{2}\leqslant z/d\\\
(d_{1},d_{2})=1\end{subarray}}\rho_{z}^{*}(dd_{1})\rho_{z}^{*}(dd_{2})\sum\limits_{\begin{subarray}{c}n\in\mathbb{Z}\\\
n\equiv(-k/d)\overline{d_{1}}\,\text{mod
}d_{2}\end{subarray}}f_{\delta}\Big{(}\frac{dd_{1}n}{N}\Big{)},$

where $\overline{d_{1}}$ refers to the multiplicative inverse of $d_{1}$
modulo $d_{2}$ that lies in the range $1\leqslant\overline{d_{1}}\leqslant
d_{2}-1$. The above is equal to

$\sum\limits_{d|k}\sum\limits_{\begin{subarray}{c}d_{1}\leqslant z/d\\\
d_{2}\leqslant z/d\\\
(d_{1},d_{2})=1\end{subarray}}\rho_{z}^{*}(dd_{1})\rho_{z}^{*}(dd_{2})\sum\limits_{l\in\mathbb{Z}}f_{\delta}\Big{(}l\frac{dd_{1}d_{2}}{N}-\frac{kd_{1}\overline{d_{1}}}{N}\Big{)},$

and applying the Poisson summation formula to the inner sum we obtain

$N\sum\limits_{d|k}\frac{1}{d}\sum\limits_{\begin{subarray}{c}d_{1}\leqslant
z/d\\\ d_{2}\leqslant z/d\\\
(d_{1},d_{2})=1\end{subarray}}\frac{\rho_{z}^{*}(dd_{1})}{d_{1}}\frac{\rho_{z}^{*}(dd_{2})}{d_{2}}\sum\limits_{c\in\mathbb{Z}}\widehat{f_{\delta}}\Big{(}c\frac{N}{dd_{1}d_{2}}\Big{)}e\Big{(}-c\frac{k}{d}\frac{\overline{d_{1}}}{d_{2}}\Big{)}.$

The term with $c=0$ is equal to

$(1+O(\delta))\frac{N}{2}\sum\limits_{\begin{subarray}{c}d_{1}\leqslant z\\\
d_{2}\leqslant z\\\
(d_{1},d_{2})|k\end{subarray}}\frac{\rho_{z}^{*}(d_{1})}{d_{1}}\frac{\rho_{z}^{*}(d_{2})}{d_{2}}(d_{1},d_{2}),$
(2.5)

since $\widehat{f_{\delta}}(0)=\int f=1/2+O(\delta)$.

Now we consider the terms with $c\neq 0$, which will go into the error. First
we deal with the contribution from those $c$ with $|c|\geqslant
dd_{1}d_{2}N^{-1}\delta^{-1}X^{\varepsilon}$. For these terms, the bound (2.2)
yields

$\sum\limits_{|c|\geqslant
dd_{1}d_{2}N^{-1}\delta^{-1}X^{\varepsilon}}|\widehat{f_{\delta}}(cN/dd_{1}d_{2})|\ll_{\varepsilon}X^{-10}$

and so the contribution to the final error term is negligible. To deal with
the remaining terms, we split the variables $d_{1}$ and $d_{2}$ into dyadic
ranges, seeking (for each fixed divisor $d|k$) an upper bound on

$\frac{N}{d}\sum\limits_{\begin{subarray}{c}c\in\mathbb{Z}\\\
1\leqslant|c|\leqslant
dM_{1}M_{2}N^{-1}\delta^{-1}X^{\varepsilon}\end{subarray}}\Big{|}\sum\limits_{\begin{subarray}{c}M_{1}/2<d_{1}\leqslant
M_{1}\\\ M_{2}/2<d_{2}\leqslant M_{2}\\\
(d_{1},d_{2})=1\end{subarray}}\frac{\rho_{z}^{*}(d_{1})}{d_{1}}\frac{\rho_{z}^{*}(d_{2})}{d_{2}}\widehat{f_{\delta}}\Big{(}c\frac{N}{dd_{1}d_{2}}\Big{)}e\Big{(}-c\frac{k}{d}\frac{\overline{d_{1}}}{d_{2}}\Big{)}\Big{|}.$

Expanding the Fourier transform of $f_{\delta}$, this is bounded above by a
constant times

$M_{1}M_{2}\delta^{-1}X^{\varepsilon}\max\limits_{\begin{subarray}{c}c\in\mathbb{Z}\\\
1\leqslant|c|\leqslant
dM_{1}M_{2}N^{-1}\delta^{-1}X^{\varepsilon}\end{subarray}}\max\limits_{x\in[0,2]}\Big{|}\sum\limits_{\begin{subarray}{c}M_{1}/2<d_{1}\leqslant
M_{1}\\\ M_{2}/2<d_{2}\leqslant M_{2}\\\
(d_{1},d_{2})=1\end{subarray}}\frac{\rho_{z}^{*}(d_{1})}{d_{1}}\frac{\rho_{z}^{*}(d_{2})}{d_{2}}e\Big{(}-c\frac{k}{d}\frac{\overline{d_{1}}}{d_{2}}-cx\frac{N}{dd_{1}d_{2}}\Big{)}\Big{|}.$
(2.6)

By considering complex conjugation, without loss of generality we may assume
that $c$ is positive.

Now we apply the bound of Bettin–Chandee on trilinear Kloosterman fractions.

######  Theorem 2.2 (Bettin-Chandee, Remark 1, [1]).

Let $\theta\in\mathbb{\mathbb{Z}}\setminus\\{0\\}$. Let $M_{1},M_{2},C$ be
dyadic scales, and for each integer $c$ in the range $C/2<c\leqslant C$ we let
$g_{c,\theta}\in C^{1}(\mathbb{R}^{2})$ be some continuously differentiable
function. Then, for some arbitrary weight functions $\alpha_{d_{1}}$,
$\beta_{d_{2}}$, and $\nu_{c}$, let

$\mathcal{B}_{g,\theta}(M_{1},M_{2},C):=\sum\limits_{\begin{subarray}{c}M_{1}/2<d_{1}\leqslant
M_{1}\\\ M_{2}/2<d_{2}\leqslant M_{2}\\\ (d_{1},d_{2})=1\\\ C/2<c\leqslant
C\end{subarray}}\alpha_{d_{1}}\beta_{d_{2}}\nu_{c}e\Big{(}\theta\frac{c\overline{d_{1}}}{d_{2}}+g_{c,\theta}(d_{1},d_{2})\Big{)}.$

Suppose that for some parameter $K$, for all integers $c$ in the range
$C/2<c\leqslant C$, for all $x\in(M_{1}/2,M_{1}]$ and for all
$y\in(M_{2}/2,M_{2}]$ we have

$\frac{\partial}{\partial
x}g_{c,\theta}(x,y)\ll\frac{K}{x^{2}y},\qquad\text{and
}\qquad\frac{\partial}{\partial y}g_{c,\theta}(x,y)\ll\frac{K}{xy^{2}}.$

Finally, let

$\|\alpha\|=\Big{(}\sum\limits_{M_{1}/2<d_{1}\leqslant
M_{1}}|\alpha_{d_{1}}|^{2}\Big{)}^{1/2},$

and similarly for the other weights. Let $\varepsilon>0$. Then
$\mathcal{B}_{g,\theta}(M_{1},M_{2},C)$ is at most a constant (depending on
$\varepsilon$) times

$\|\alpha\|\|\beta\|\|\nu\|\Big{(}1+\frac{|\theta|C+K}{M_{1}M_{2}}\Big{)}^{1/2}((CM_{1}M_{2})^{\frac{7}{20}+\varepsilon}(M_{1}+M_{2})^{\frac{1}{4}}+(CM_{1}M_{2})^{\frac{3}{8}+\varepsilon}(CM_{1}+CM_{2})^{\frac{1}{8}}).$

To apply this bound in our setting, we let

$\alpha_{d_{1}}:=\frac{\rho_{z}^{*}(d_{1})}{d_{1}},\quad\beta_{d_{2}}:=\frac{\rho_{z}^{*}(d_{2})}{d_{2}},\qquad\theta=-\frac{k}{d},\qquad
g_{c,\theta}(d_{1},d_{2})=-cx\frac{N}{dd_{1}d_{2}},$

and we take $C=c$ and the weight $\nu_{c}$ to be supported at a single element
$c$ (and equal to $1$ there). Observe that we may take $K=cN/d$. Since
$c\leqslant dM_{1}M_{2}N^{-1}\delta^{-1}X^{\varepsilon}$, $\|\alpha\|\ll
B/M_{1}$, and $\|\beta\|\ll B/M_{2}$, we infer that (2.6) is at most a
constant (depending on $\varepsilon$) times

$\displaystyle
B^{2}\delta^{-1}X^{\varepsilon}\Big{(}1+\delta^{-1}\Big{(}1+\frac{|k|}{N}\Big{)}\Big{)}^{\frac{1}{2}}$
$\displaystyle\Big{(}d^{\frac{7}{20}+\varepsilon}\delta^{-\frac{7}{20}-\varepsilon}N^{-\frac{7}{20}-\varepsilon}(M_{1}^{\frac{19}{20}+\varepsilon}M_{2}^{\frac{7}{10}+\varepsilon}+M_{1}^{\frac{7}{10}+\varepsilon}M_{2}^{\frac{19}{20}+\varepsilon})$
$\displaystyle+(d^{\frac{1}{2}+\varepsilon}\delta^{-\frac{1}{2}-\varepsilon}N^{-\frac{1}{2}-\varepsilon}(M_{1}^{1+\varepsilon}M_{2}^{\frac{7}{8}+\varepsilon}+M_{1}^{\frac{7}{8}+2\varepsilon}M_{2}^{1+\varepsilon})\Big{)}.$
(2.7)

What remains is the (somewhat intricate!) task of adding together the
contributions from various dyadic scales, and optimising for $\delta$.

Without loss of generality we assume $k>0$ (the case $k<0$ is symmetric), so
$X_{1}=0$ and $X_{2}=X-k$. Let $\eta>0$ be a positive parameter to be
optimised later. We let $N_{l}=2^{-l+1}(X-k)$, for $1\leqslant
l\leqslant\log(X^{\eta})/(\log 2)$. Then

$\sum\limits_{0<n\leqslant
X-k}\lambda_{z}^{*}(n)\lambda_{z}^{*}(n+k)=\sum\limits_{1\leqslant
l\leqslant(\log(X^{\eta}))/(\log 2)}\sum\limits_{N_{l}/2<n\leqslant
N_{l}}\lambda_{z}^{*}(n)\lambda_{z}^{*}(n+k)+O_{\varepsilon}(B^{2}X^{1-\eta+\varepsilon}).$

Adding together the main terms from (2.5), we get

$(1+O(\delta))(X-k)\Big{(}\sum\limits_{1\leqslant
l\leqslant\log(X^{\eta})/\log
2}2^{-l}\Big{)}\Big{(}\sum\limits_{\begin{subarray}{c}d_{1},d_{2}\leqslant
z\\\
(d_{1},d_{2})|k\end{subarray}}\frac{\rho_{z}^{*}(d_{1})}{d_{1}}\frac{\rho_{z}^{*}(d_{2})}{d_{2}}(d_{1},d_{2})\Big{)},$

which is

$(X-k)\Big{(}\sum\limits_{\begin{subarray}{c}d_{1},d_{2}\leqslant z\\\
(d_{1},d_{2})|k\end{subarray}}\frac{\rho_{z}^{*}(d_{1})}{d_{1}}\frac{\rho_{z}^{*}(d_{2})}{d_{2}}(d_{1},d_{2})\Big{)}+O_{\varepsilon}(B^{2}(\log
z)^{2}\tau_{-1}(k)(X^{1-\eta}+\delta X))$

where $\tau_{-1}(k):=\sum_{d|k}d^{-1}$. This has the desired main term.

Now we need to consider those terms with $c\neq 0$, i.e. expression (2). By
considering the worst case scenario ($k=d=X$ and $N=X^{1-\eta}$), one may
upper-bound the contribution to (2) by

$O_{\varepsilon}(B^{2}\delta^{-2-\varepsilon}X^{\eta+\varepsilon}(M_{1}^{1+\varepsilon}M_{2}^{\frac{7}{8}+\varepsilon}+M_{1}^{\frac{7}{8}+\varepsilon}M_{2}^{1+\varepsilon})).$

Summing over $d|k$ and absorbing the resulting $\tau(k)$ term into the
$X^{\varepsilon}$ term, and asumming over dyadic scales $M_{1}$ and $M_{2}$,
we get a total error from this part of

$O_{\varepsilon}(B^{2}\delta^{-2-\varepsilon}X^{\eta+\varepsilon}z^{\frac{15}{8}}).$

Combining all the errors, we arrive at an error term which is at most a
constant (depending on $\varepsilon$) times

$B^{2}(\delta
X^{1+\varepsilon}+X^{\frac{1}{2}+\varepsilon}+X^{1-\eta+\varepsilon}+\delta^{-2-\varepsilon}X^{\eta+\varepsilon}z^{\frac{15}{8}}).$

It remains to optimise $\delta$ and $\eta$. Picking

$\delta=X^{\frac{\eta-1}{3}}z^{\frac{5}{8}},$

we get an error of

$O_{\varepsilon}(B^{2}(X^{1-\eta+\varepsilon}+z^{\frac{5}{8}}X^{\frac{2}{3}+\frac{\eta}{3}+\varepsilon})).$

Picking

$\eta=\frac{1}{4}-\frac{15}{32}\frac{\log z}{\log X},$

so long as this is positive, we get an error of

$O_{\varepsilon}(B^{2}X^{\frac{3}{4}+\varepsilon}z^{\frac{15}{32}}).$

However, if

$\frac{1}{4}-\frac{15}{32}\frac{\log z}{\log X}<0$

then $z>X^{8/15}$, which means that the claimed error term in Proposition 2.1
is at least $O_{\varepsilon}(B^{2}X^{1+\varepsilon})$ and so the whole
proposition is trivial anyway. So we have derived an error term of
$O_{\varepsilon}(B^{2}X^{\frac{3}{4}+\varepsilon}z^{\frac{15}{32}})$, as
desired. ∎

######  Remark 2.3.

_For applications to pair correlations of zeros of the Riemann zets function
one has some extra averaging over $k$ for the correlations
$\lambda_{z}(n)\lambda_{z}(n+k)$, and some authors have taken advantage of
this. Lemma 1.11 of [20], for example, is a bound for the sum_

$\sum\limits_{k\leqslant h}(h-k)\sum\limits_{n\leqslant
X}\lambda_{z}(n)\lambda_{z}(n+k)$

_with a dominant error term of $O(h^{3/2}z^{2}X^{\varepsilon})$, which for
large $h$ this is a stronger result than what may be proved by applying
Proposition 1.2 to the inner sum and adding up the error terms naïvely._

##  3\. Proof of Proposition 1.3

As we mentioned in the introduction, much of what we do here was first
accomplished by Hooley in [19, Lemma 2]. That said, this result does lie
somewhat buried in the literature, and we hope too that we may provide the
reader with some motivation as to why this approach might be expected to
succeed, which Hooley did not include in [19].

We let

$\mathbb{D}_{z}(s):=\sum\limits_{n=1}^{\infty}\frac{\lambda_{z}^{2}(n)}{n^{s}}$

be the Dirichlet series associated to $\lambda_{z}^{2}$. Since

$|\lambda_{z}(n)|\leqslant L(z)\tau(n),$ (3.1)

the series for $\mathbb{D}_{z}(s)$ is absolutely convergent in the region $\Re
s>1$. Swapping sums reveals that

$\mathbb{D}_{z}(s)=\zeta(s)\sum\limits_{d_{1},d_{2}\leqslant
z}\frac{\rho_{z,d_{1}}\rho_{z,d_{2}}}{[d_{1},d_{2}]^{s}},$

and so $\mathbb{D}_{z}(s)$ has a meromorphic continuation to the whole of the
complex plane. We write

$\mathbb{F}_{z}(s):=\sum\limits_{d_{1},d_{2}\leqslant
z}\frac{\rho_{z,d_{1}}\rho_{z,d_{2}}}{[d_{1},d_{2}]^{s}},$

and a key element of the proof will be the establishing of a non-trivial bound
on $|\mathbb{F}_{z}(s)|$ to the left of the $1$-line.

######  Lemma 3.1.

Assume RH and let $\varepsilon>0$. Denoting $s=\sigma+it$, if
$\sigma>1/2+\varepsilon$ then for all $\delta>0$

$|\mathbb{F}_{z}(s)|\ll_{\varepsilon,\delta}(|t|+2)^{\delta}z^{1-\sigma+\delta}.$

###### Proof.

Following Hooley, from combinatorial manipulation one derives

$\mathbb{F}_{z}(s)=\sum\limits_{u\leqslant
z}\frac{\varphi(u,s)\mu^{2}(u)}{u^{2s}}\Big{(}\sum\limits_{\begin{subarray}{c}d\leqslant
z/u\\\ (d,u)=1\end{subarray}}\frac{\rho_{z,du}}{d^{s}}\Big{)}^{2},$ (3.2)

where

$\varphi(u,s):=\prod\limits_{p|u}(p^{s}-1).$

Let us now describe a heuristic for the remainder of the argument. If we were
to undertake a naïve approach, bounding the inner sum of (3.2) trivially, then
one would derive

$\displaystyle|\mathbb{F}_{z}(s)|$ $\displaystyle\ll(\log
z)^{2}\sum\limits_{u\leqslant
z}\frac{\mu^{2}(u)}{u^{\sigma}}\Big{(}\sum\limits_{d\leqslant
z/u}\frac{1}{d^{\sigma}}\Big{)}^{2}$ $\displaystyle\ll(\log
z)^{2}\sum\limits_{u\leqslant
z}\frac{\mu^{2}(u)}{u^{\sigma}}\Big{(}\frac{z}{u}\Big{)}^{2(1-\sigma)}$
$\displaystyle\ll(\log z)^{2}z^{2(1-\sigma)}.$

The exponent $2$ for the power of $z^{1-\sigma}$ would lead, ultimately, to an
error term in Proposition 1.3 involving $z^{2}$, with no hope of improving
upon (1.3).

However, since $L(z)^{-1}\rho_{z,du}$ is a sieve weight we might reasonably
expect to be able to approximate the behaviour of $L(z)^{-1}\rho_{z,du}$ by
the behaviour of $\mu(du)$. Were we to make this replacement, the inner
Dirichlet polynomial in (3.2) would become

$L(z)\mu(u)\sum\limits_{\begin{subarray}{c}d\leqslant z/u\\\
(d,u)=1\end{subarray}}\frac{\mu(d)}{d^{s}}.$

Using RH, the Dirichlet polynomial $\sum_{d\leqslant D}\mu(d)d^{-s}$ may in
turn be approximated by $\zeta(s)^{-1}$. Since RH also implies that
$|\zeta(s)|^{-1}\ll_{\varepsilon,\delta}(|t|+2)^{\delta}$, this leads to the
agreable

$\displaystyle|\mathbb{F}_{z}(s)|$
$\displaystyle\ll_{\varepsilon,\delta}(|t|+2)^{\delta}(\log
z)^{2}\sum\limits_{u\leqslant z}\frac{\mu^{2}(u)}{u^{\sigma}}$
$\displaystyle\ll(|t|+2)^{\delta}z^{1-\sigma}(\log z)^{2}.$

This would settle Lemma 3.1.

What remains is for us to make these approximations precise. The main
auxiliary lemma is the following.

######  Lemma 3.2.

Assume RH. Let $D\geqslant 1$ and $\varepsilon>0$ be reals, and let
$k\leqslant D$ be a natural number. Define

$M_{D,k}(s):=\sum\limits_{\begin{subarray}{c}d\leqslant D\\\
(d,k)=1\end{subarray}}\frac{\mu(d)d}{\varphi(d)d^{s}}$

and let

$\tau_{\theta}(k):=\sum\limits_{d|k}d^{\theta}.$

Then, if $s=\sigma+it$ with $1/2+\varepsilon<\sigma<1$, for all $\delta>0$ we
have

$|M_{D,k}(s)|=O_{\varepsilon,\delta}((|t|+2)^{\delta}\tau_{-1/2+\delta}(k)).$

We note for future reference the trivial bound
$\tau_{-1/2+\delta}(k)\leqslant\tau(k)$.

###### Proof.

Let

$G_{k}(w):=\sum\limits_{\begin{subarray}{c}d\geqslant 1\\\
(d,k)=1\end{subarray}}\frac{\mu(d)d}{\varphi(d)d^{w}}.$

The series defining $G_{k}(w)$ is absolutely convergent in the region $\Re
w>1$. Following Hooley, we also have (in the same region)

$G_{k}(w)=\frac{1}{\zeta(w)}H_{k}(w)B(w),$

where

$H_{k}(w):=\prod\limits_{p|k}\Big{(}1-\Big{(}1-\frac{1}{p}\Big{)}^{-1}\frac{1}{p^{w}}\Big{)}$

and

$B(w):=\prod\limits_{p}\Big{(}1-\frac{1}{p^{w}(p-1)}\Big{(}1-\frac{1}{p^{w}}\Big{)}^{-1}\Big{)}.$

Therefore, since we are assuming RH, $G_{k}(w)$ has an analytic continuation
to the region $\Re w>1/2+\varepsilon/2$. In this region, we have the bounds

$\displaystyle|H_{k}(w)|$
$\displaystyle\ll\sum\limits_{d|k}\frac{\mu^{2}(d)d}{d^{\Re w}\varphi(d)}$
$\displaystyle\ll_{\delta}\sum\limits_{d|k}d^{-\Re w+\delta}$
$\displaystyle\ll\tau_{-1/2+\delta}(k),$

and

$|B(w)|=O(1).$

Furthermore, by RH, in this region we have

$\Big{|}\frac{1}{\zeta(w)}\Big{|}=O_{\varepsilon,\delta}((|\Im
w|+2)^{\delta}),$

see [28, Theorem 14.2]. Therefore, we have the overall bound

$|G_{k}(w)|=O_{\varepsilon,\delta}((|\Im
w|+2)^{\delta}\tau_{-1/2+\delta}(k)).$ (3.3)

Now, pick $c$ such that $1+\delta<\sigma+c<2$ and, without loss of generality,
assume that $D$ is a half-integer. Then, by Perron’s formula (see [28, Lemma
3.12] for this particular formulation),

$M_{D,k}(s)=\frac{1}{2\pi
i}\int\limits_{c-iT}^{c+iT}G_{k}(s+s^{\prime})\frac{D^{s^{\prime}}}{s^{\prime}}\,ds^{\prime}+O_{\delta}(T^{-1}(D^{c}+D^{1-\sigma+\delta})).$

Now we move the contour $\Re s^{\prime}=c$ to the line $\Re
s^{\prime}=-\varepsilon/2$. There is a main-term contribution of $G_{k}(s)$,
coming from the pole at $s^{\prime}=0$. Regarding the error term, note that
$1/2+\varepsilon/2<\Re s-\varepsilon/2$, and so using the bound (3.3) we get a
combined error term of

$O_{\varepsilon,\delta}(k^{\delta}D^{c}T^{-1}(T^{\delta}+(|t|+1)^{\delta}))+O_{\varepsilon,\delta}(k^{\delta}D^{-\varepsilon/2}(T^{\delta}+(|t|+1)^{\delta}))+O_{\delta}(T^{-1}(D^{c}+D^{1-\sigma+\delta})).$

All these errors are inferior to the claimed error in the lemma if $T=D^{10}$.

Finally, note that $G_{k}(s)$ itself is bounded by

$|G_{k}(s)|\ll_{\varepsilon,\delta}(|t|+1)^{\delta}\tau_{-1/2+\delta}(k).$

The lemma follows. ∎

To use Lemma 3.2, we write

$\Gamma_{z}(u,s):=\sum\limits_{\begin{subarray}{c}d\leqslant z/u\\\
(d,u)=1\end{subarray}}\frac{\rho_{z,du}}{d^{s}},$

so that from (3.2) we get

$\mathbb{F}_{z}(s)=\sum\limits_{u\leqslant
z}\frac{\varphi(u,s)\mu^{2}(u)}{u^{2s}}\Gamma_{z}(u,s)^{2}.$ (3.4)

Then, following Hooley, by expanding the definition of $\rho_{z,du}$ we
establish

$\displaystyle\Gamma_{z}(u,s)$
$\displaystyle=\frac{u\mu(u)}{\varphi(u)}\sum\limits_{\begin{subarray}{c}u^{\prime}\leqslant
z/u\\\
(u^{\prime},u)=1\end{subarray}}\frac{\mu^{2}(u^{\prime})}{\varphi(u^{\prime})}\sum\limits_{\begin{subarray}{c}d\leqslant
z/uu^{\prime}\\\
(d,uu^{\prime})=1\end{subarray}}\frac{\mu(d)d}{\varphi(d)d^{s}}$
$\displaystyle=\frac{u\mu(u)}{\varphi(u)}\sum\limits_{\begin{subarray}{c}u^{\prime}\leqslant
z/u\\\
(u^{\prime},u)=1\end{subarray}}\frac{\mu^{2}(u^{\prime})}{\varphi(u^{\prime})}M_{z/uu^{\prime},uu^{\prime}}(s).$
(3.5)

By Lemma 3.2, we conclude that if $1/2+\varepsilon<\sigma<1$ then

$|\Gamma_{z}(u,s)|=O_{\varepsilon,\delta}((|t|+2)^{\delta}z^{\delta}u\varphi(u)^{-1}).$

Plugging this bound into (3.4), if $1/2+\varepsilon<\sigma<1$ then

$|\mathbb{F}_{z}(s)|\ll_{\varepsilon,\delta}z^{\delta}(|t|+2)^{\delta}\sum\limits_{u\leqslant
z}\frac{u^{2}}{u^{\sigma}\varphi(u)^{2}}\ll_{\varepsilon}(|t|+2)^{\delta}z^{1-\sigma+\delta}.$

Lemma 3.1 is proved. ∎

Proposition 1.3 follows quickly. Indeed, pick some $c>1$. Then, by Perron’s
formula again, we have

$\displaystyle\sum\limits_{n\leqslant X}\lambda_{z}^{2}(n)$
$\displaystyle=\frac{1}{2\pi
i}\int\limits_{c-iT}^{c+iT}\mathbb{D}_{z}(s)\frac{X^{s}}{s}\,ds+O\Big{(}(\log
z)^{2}\tau(X)^{2}1_{X\in\mathbb{N}}+X^{c}\sum\limits_{n\geqslant
1}\frac{\lambda_{z}^{2}(n)}{n^{c}}\min\Big{(}1,\frac{1}{T|\log(X/n)|}\Big{)}\Big{)}$
$\displaystyle=\frac{1}{2\pi
i}\int\limits_{c-iT}^{c+iT}\zeta(s)\mathbb{F}_{z}(s)\frac{X^{s}}{s}\,ds+O_{\delta}(X^{c+\delta}T^{-1}+X^{\delta}).$

Moving the contour to the line $\Re s=\frac{1}{2}+\varepsilon$, we pick up a
main term of $X\mathbb{F}_{z}(1)$ from the pole at $s=1$. It is a standard
calculation in the theory of the Selberg sieve that

$\mathbb{F}_{z}(1)=\sum\limits_{d_{1},d_{2}\leqslant
z}\frac{\rho_{z,d_{1}}\rho_{z,d_{2}}}{[d_{1},d_{2}]}=L(z),$ (3.6)

see [26, Exercise 9.3.6]. Therefore the main term is $XL(z)$.

Regarding the error terms, the bound from Lemma 3.1 together with the
Lindelhöf bound

$|\zeta(s)|\ll_{\delta}(|t|+2)^{\delta}$

gives

$\Big{|}\int\limits_{1/2+\varepsilon+iT}^{1/2+\varepsilon-
iT}\frac{\zeta(s)\mathbb{F}_{z}(s)X^{s}}{s}ds\Big{|}=O_{\varepsilon,\delta}(X^{\frac{1}{2}+\varepsilon}z^{\frac{1}{2}-\varepsilon+\delta}T^{\delta}).$

The remaining two integrals are

$O(X^{O(1)}T^{-1}).$

Picking $T$ to be a suitably large (but fixed) power of $X$, we derive

$\sum\limits_{n\leqslant
X}\lambda_{z}^{2}(n)=XL(z)+O_{\varepsilon,\delta}(X^{\frac{1}{2}+\varepsilon+\delta}z^{\frac{1}{2}}).$

Since $\varepsilon$ is arbitrary, Proposition 1.3 follows (after combining the
bound above with the classical bound (1.3)). ∎

##  4\. Proof of applications

In this section we discuss the proof of Theorems 1.10 and 1.13. For experts,
we can say that our proof consists of nothing more than substituting our
stronger auxiliary estimates on $\lambda_{z}(n)$ into the general framework
provided by G–G–Ö–S [15] and Goldston–Yildrim [17]. We have no desire to
repeat large tracts of the existing literature, but similarly it seems remiss
not to give the general reader at least some indication of how our
Propositions 1.2 and 1.3 are used. What follows is, we hope, an informative
balance.

We focus on Theorem 1.10. Fix $\alpha\in[1,2)$, let $X=T^{\alpha}$, and let

$F(X,T):=\Big{(}\frac{T}{2\pi}\log T\Big{)}F_{T}(\alpha).$

Since we are assuming $AP(8/15)$ we may also assume RH, as RH is equivalent to
the $q=1$ form of Conjecture 1.9. Let
$\Psi_{U}:\mathbb{R}\longrightarrow[0,1]$ be a smooth approximation to the
indicator function of the interval $[0,1]$, supported on $[-1/U,1+1/U]$, with
the properties that $\Psi_{U}(t)\in[0,1]$ and $\Psi^{(j)}(t)\ll U^{j}$ for
$j=1,2,3...$. (We use this notation here in order to match the notation in
[15], to allow for easier comparisons.) We begin with the following lemma of
G-G-Ö-S, which is proved using the same approximate functional equation
technique of Montgomery [24]:

######  Lemma 4.1 (Lemma 1, [15]).

Assume RH. If $X\gg T$ and $U=\log^{B}T$ with $B>1$, then we have

$F(X,T)=\frac{1}{2\pi
X^{2}}I_{1}(X,T)+\frac{X^{2}}{2\pi}I_{2}(X,T)+O\Big{(}\frac{T\log^{2}T}{U}\Big{)}+O_{\varepsilon}\Big{(}\frac{X^{1+\varepsilon}}{T}\Big{)},$
(4.1)

where

$I_{1}(X,T)=\int\limits_{-\infty}^{\infty}\Psi_{U}\Big{(}\frac{t}{T}\Big{)}\Big{|}\sum\limits_{n\leqslant
X}\Lambda(n)n^{\frac{1}{2}-it}-\int\limits_{1}^{X}u^{1/2-it}\,du\Big{|}^{2}\,dt$
(4.2)

and

$I_{2}(X,T)=\int\limits_{-\infty}^{\infty}\Psi_{U}\Big{(}\frac{t}{T}\Big{)}\Big{|}\sum\limits_{n>X}\frac{\Lambda(n)}{n^{3/2+it}}-\int\limits_{X}^{\infty}u^{-3/2-it}\,du\Big{|}^{2}\,dt.$
(4.3)

The action then shifts to estimating the integrals $I_{1}(X,T)$ and
$I_{2}(X,T)$. If one expands the definitions, one quickly confronts the
problem of having to control the values of the correlations $\sum_{n\leqslant
N}\Lambda(n)\Lambda(n+k)$. To get around this issue, the idea of G–G–Ö–S is to
note that $I_{1}(X,T)$ is at least

$\displaystyle\int\limits_{-\infty}^{\infty}\Psi_{U}\Big{(}\frac{t}{T}\Big{)}\Big{|}\sum\limits_{n\leqslant
X}\Lambda(n)n^{\frac{1}{2}-it}-$
$\displaystyle\int\limits_{1}^{X}u^{1/2-it}\,du\Big{|}^{2}\,dt\,-$
$\displaystyle\int\limits_{-\infty}^{\infty}\Psi_{U}\Big{(}\frac{t}{T}\Big{)}\Big{|}\sum\limits_{n\leqslant
X}\lambda_{z}(n)n^{\frac{1}{2}-it}-\sum\limits_{n\leqslant
X}\Lambda(n)n^{\frac{1}{2}-it}du\Big{|}^{2}\,dt,$

and therefore to conclude that $I_{1}(X,T)$ is at least

$\displaystyle\int\limits_{-\infty}^{\infty}\Psi_{U}\Big{(}\frac{t}{T}\Big{)}\Big{(}\sum\limits_{n\leqslant
X}\Lambda(n)n^{1/2-it}-\int\limits_{1}^{X}u^{1/2-it}\,du\Big{)}\Big{(}\overline{\sum\limits_{n\leqslant
X}\lambda_{z}(n)n^{1/2-it}-\int\limits_{1}^{X}u^{1/2-it}\,du}\Big{)}\,dt+c.c.$
$\displaystyle-\int\limits_{-\infty}^{\infty}\Psi_{U}\Big{(}\frac{t}{T}\Big{)}\Big{|}\sum\limits_{n\leqslant
X}\lambda_{z}(n)n^{1/2-it}-\int\limits_{1}^{X}u^{1/2-it}\,du\Big{|}^{2}\,dt.$
(4.4)

There is a similar expression for $I_{2}(X,T)$. Expanding out the brackets,
one sees that the relevant correlations are now
$\sum_{n}\Lambda(n)\lambda_{z}(n)$, $\sum_{n}\Lambda(n)\lambda_{z}(n+k)$,
$\sum_{n}\lambda_{z}(n)^{2}$, and $\sum_{n}\lambda_{z}(n)\lambda_{z}(n+k)$,
which promise to be more easily tractable than
$\sum_{n}\Lambda(n)\Lambda(n+k)$.

Let

$z=X^{\nu}$

for some parameter $\nu$ to be chosen later. Corollary 1 of Goldston-Gonek
[14] then provides us with a general way of taking bounds from diagonal and
off-diagonal correlations of the coefficients of Dirichlet polynomials and
using them to generate estimates for expressions such as (4). More precisely,
this corollary applies to estimating

$\int\limits_{-\infty}^{\infty}\Psi_{U}\Big{(}\frac{t}{T}\Big{)}\Big{(}\sum\limits_{n\leqslant
X}a_{n}n^{-\sigma-it}-\int\limits_{1}^{X}u^{-\sigma-
it}\,du\Big{)}\overline{\Big{(}\sum\limits_{n\leqslant X}b_{n}n^{-\sigma-
it}-\int\limits_{1}^{X}u^{-\sigma-it}\,du\Big{)}}\,dt,$

provided the coefficients $a_{n},b_{n}$ satisfy various hypotheses on the
summatory functions $\sum_{n\leqslant X}a_{n}$, $\sum_{n\leqslant X}b_{n}$, as
well as on the correlations $\sum_{n\leqslant X}a_{n}b_{n+h}$. These
hypotheses are given by $(A1)$ to $(A4)$ in [14].

To formulate these hypotheses for $\Lambda(n)$ and $\lambda_{z}(n)$, we will
use Propositions 1.2 and 1.3, as well as the following estimates. The bound

$\sum\limits_{n\leqslant X}\Lambda(n)=X+O_{\varepsilon}(X^{1/2+\varepsilon})$
(4.5)

follows from RH. The bounds

$\sum\limits_{n\leqslant X}\lambda_{z}(n)=X+O(X^{\nu})$ (4.6)

and

$\sum\limits_{n\leqslant
X}\Lambda(n)\lambda_{z}(n)=XL(z)+O_{\varepsilon}(X^{\nu+\varepsilon})$ (4.7)

are elementary and may be found in [15]. We also note a simple consequence of
the conjecture $AP(\theta)$.

######  Lemma 4.2.

Let $1\leqslant|k|\leqslant X$ and let $X_{1}=\max(0,-k)$ and
$X_{2}=\min(X,X-k)$. Assume $AP(\theta)$. Then if $\nu<\theta$ we have

$\sum\limits_{X_{1}<n\leqslant
X_{2}}\lambda_{z}(n)\Lambda(n+k)=\mathfrak{S}(k)(X-|k|)+O_{\varepsilon,\theta}(X^{\max(1/2+\nu/2,1-\nu)+\varepsilon}),$

###### Proof.

We have

$\displaystyle\sum\limits_{X_{1}<n\leqslant X_{2}}\lambda_{z}(n)\Lambda(n+k)$
$\displaystyle=\sum\limits_{d\leqslant
z}\rho_{z,d}\sum\limits_{\begin{subarray}{c}X_{1}<n\leqslant X_{2}\\\
d|n\end{subarray}}\Lambda(n+k)$
$\displaystyle=(X-|k|)\sum\limits_{\begin{subarray}{c}d\leqslant z\\\
(d,k)=1\end{subarray}}\frac{\rho_{z,d}}{\varphi(d)}+O_{\varepsilon,\theta}\Big{(}X^{1/2+\varepsilon}\sum\limits_{d\leqslant
z}\frac{|\rho_{z,d}|}{d^{1/2}}\Big{)}$
$\displaystyle=\mathfrak{S}(k)(X-|k|)+O\Big{(}\frac{k\tau(k)X}{\varphi(k)z}\Big{)}+O_{\varepsilon,\theta}(X^{1/2+\varepsilon}z^{1/2}L(z))$
$\displaystyle=\mathfrak{S}(k)(X-|k|)+O_{\varepsilon,\theta}(X^{\max(1/2+\nu/2,1-\nu)+\varepsilon})$

as claimed. To estimate the sum of $\rho_{z,d}\varphi(d)^{-1}$ we have used
the calculation from the proof of [13, Lemma 1], which in turn is taken from
[18, p. 57]. ∎

Plugging everything into Corollary 1 of [14], as applied to the integrals in
(4), we have

$\displaystyle I_{1}(X,T)\geqslant$
$\displaystyle\widehat{\Psi_{U}}(0)T\Big{(}2\sum\limits_{n\leqslant
X}\Lambda(n)\lambda_{z}(n)n-\sum\limits_{n\leqslant
X}\lambda_{z}^{2}(n)n\Big{)}$
$\displaystyle+4\pi\Big{(}\frac{T}{2\pi}\Big{)}^{3}\int\limits_{T/2\pi
X}^{\infty}\Big{(}\sum\limits_{h\leqslant 2\pi
Xv/T}\mathfrak{S}(h)h^{2}\Big{)}\Re\widehat{\Psi_{U}}(v)\frac{1}{v^{3}}\,dv$
$\displaystyle-4\pi\Big{(}\frac{T}{2\pi}\Big{)}^{3}\int\limits_{XT^{\varepsilon}/2\pi}^{\infty}\Big{(}\int\limits_{0}^{2\pi
Xv/T}u^{2}\,du\Big{)}\Re\widehat{\Psi_{U}}(v)\frac{1}{v^{3}}\,dv$
$\displaystyle+O_{\varepsilon}(T^{-1}X^{3+\varepsilon})+O_{\varepsilon}(X^{5/2+\varepsilon})+O_{\varepsilon}(X^{\varepsilon})+O_{\varepsilon,\theta}(X^{2+\varepsilon+\omega}),$
(4.8)

where

$\displaystyle\omega$
$\displaystyle=\max\Big{(}\frac{1}{2},\nu,\max\Big{(}1-\nu,\frac{1}{2}+\frac{\nu}{2}\Big{)},\max\Big{(}1-\nu,\min\Big{(}2\nu,\frac{3}{4}+\frac{15\nu}{32}\Big{)}\Big{)}\Big{)}$
$\displaystyle=\max\Big{(}\nu,1-\nu,\frac{1}{2}+\frac{\nu}{2},\min\Big{(}2\nu,\frac{3}{4}+\frac{15\nu}{32}\Big{)}\Big{)}.$

The size of the main terms in (4) is calculated in [15, Section 7]. Here,
provided that the error term in Proposition 1.3 is at most $X$, these terms
are shown222We feel we should point out that there is a confusing misprint in
this section of [15], in the display equation before [15, equation (7.3)],
which is incorrect by a factor or $2$. But this factor is then corrected at
the end of the calculation. to have a combined size of

$\frac{1}{2}X^{2}T\log(Tz/X)+O(TX^{2}(\log X)^{2/3}\log\log T).$ (4.9)

So the error term in (4) will be negligible provided

$\nu<1$

and

$\alpha\omega<1.$

A similar argument may be used to bound $I_{2}(X,T)$ from below, deferring to
Corollary 2 of [14] instead of Corollary 1. We direct the reader to [15] and
[14] for the details. The only aspect that it is important for us to note is
that the parameter $\eta$ from [15], which is taken to be $1/2-\varepsilon$ in
expression (6.3) of that paper, may in fact be taken to be $1$. (Here the
parameter $\eta$ comes from hypothesis $(A3)$ of [14], where $X^{\eta}$
measures the range of uniformity in $k$ for the error terms in the estimates
for the correlations $\sum_{n}\lambda_{z}(n)\lambda_{z}(n+k)$ and
$\sum_{n}\lambda_{z}(n)\Lambda(n+k)$.)

One derives that, uniformly for all $T\ll X\ll T^{2-\varepsilon}$,

$I_{2}(X,T)\geqslant\frac{T}{2X^{2}}\log(Tz/X)+O\Big{(}\frac{T}{X^{2}}(\log
X)^{2/3}\log\log T\Big{)}+E,$ (4.10)

where $E$ is an error term of size at most

$O_{\varepsilon}(T^{-1}X^{-1+\varepsilon})+O_{\varepsilon}(X^{-2+\omega+\varepsilon}).$

Thus the main term dominates provided that

$\alpha\omega<1.$

Therefore from Lemma 4.1 we conclude that

$F_{T}(\alpha)\geqslant(1-\alpha(1-\nu))+o_{\theta}(1)$ (4.11)

as $T\rightarrow\infty$, provided that

$\nu<\theta,\text{ and }\omega<\frac{1}{\alpha}.$ (4.12)

All that remains is to maximise the lower bound (4.11) subject to the
constraints (4.12). If $\theta\geqslant 8/15$, a straightforward calculation
yields

$\nu=\frac{32}{15\alpha}-\frac{24}{15}-\varepsilon$ (4.13)

as a solution, provided $1\leqslant\alpha\leqslant 49/48-\varepsilon$. This
should be contrasted with [15], in which the authors take
$z=T^{1/2-\varepsilon}$, i.e. $\nu=\frac{1}{2\alpha}-\varepsilon$, throughout,
which is a smaller threshold.

Plugging this value of $\nu$ into expression (4.11), we derive Theorem 1.10.∎

The proof of Theorem 1.13 is a similarly straightforward adaptation of [17],
easier even, as one does not need to defer to mean value estimates proved
elsewhere.

In the notation of [17], $R$ is used for our parameter $z$, and we take $q=1$.
The key expression from [17] is (4.37), in which there are several error
terms. It may be easily seen that $AP(\theta)$, Proposition 1.2 and
Proposition 1.3 imply that we may remove the

$O(X^{\frac{1}{2}}h^{\frac{3}{2}}R\log^{2}X)+O(h^{2}R^{2})$

error term and replace it with a single

$O_{\varepsilon}(h^{2}X^{\frac{3}{4}+\varepsilon}R^{\frac{15}{32}+\varepsilon})$

term, provided $R<X^{\theta}$. Provided $h\leqslant
X^{\frac{1}{49}-\varepsilon}$, if one chooses

$R=\frac{X^{\frac{8}{15}-\varepsilon}}{h^{\frac{32}{15}}}$

then these error terms are of a lower order of magnitude than the main term
(which has order of magnitude $hX\log X$). So assuming $AP(8/15)$ and plugging
in this value of $R$ into expression (4.37) of [17], one achieves the lower
bound in Theorem 1.13. ∎

##  5\. Comparison with the alternative hypothesis

In this short concluding section, we will reflect upon how the various
thresholds that we used in studying $F_{T}(\alpha)$ interact with the so-
called Alternative Hypothesis. This hypothesis (as formulated in expression
(2.2) of [7], say) contends that the distribution of the imaginary parts of
the zeros of the Riemann zeta function is exceedingly regular. More precisely,
AH states that if $\gamma,\gamma^{\prime}\geqslant T_{0}$ are such imaginary
parts then

$\widetilde{\gamma}-\widetilde{\gamma^{\prime}}\in\frac{1}{2}\mathbb{Z},$

where $\widetilde{\gamma}:=\frac{1}{2\pi}\gamma\log(\gamma/2\pi)$ is the
normalised version of $\gamma$. (Certain formulations, see [21, Conjecture
2.2], allow
$\widetilde{\gamma_{j}}-\widetilde{\gamma}_{j+1}\in\frac{1}{2}\mathbb{Z}+o_{j\rightarrow\infty}(1)$,
where $\gamma_{1},\gamma_{2},\gamma_{3},\dots$ are the positive imaginary
parts given in ascending order.) This distribution would contradict Conjecture
1.6. Indeed, under AH, it is immediate that the function $F_{T}(\alpha)$ is
periodic with period $2$. Therefore Theorem 1.5 implies that
$\lim_{T\rightarrow\infty}F_{T}(\alpha)$ is determined, and in particular
expression (1.8) holds. However, as is proven in [21], AH is nonetheless
consistent with everything that is presently known about the correlations of
the zeros (both pair correlations and higher correlations).

A sufficiently strong disproof of AH would rule out the existence of
exceptional zeros for Dirichlet $L$-functions. This fact was known to
Montgomery, and is remarked upon at the end of [24], but the full proof
doesn’t seem to have been written down until Conrey–Iwaniec [4]. Given that
the results of this paper are dependent on strong uniformity for the
distribution of primes in arithmetic progressions (so in particular they
assume the non-existence of exceptional zeros), disproving AH using these
methods would say nothing about the exceptional zero problem. However, we
nonetheless found it interesting to observe what correlation estimates would
be necessary in order to be able to conclude that
$\liminf_{T\rightarrow\infty}F_{T}(\alpha)\geqslant 2-\alpha$ for
$\alpha\in[1,2)$, which is the bound given by (1.8).

Let us assume RH, and also write $X_{1},X_{2}$ as in Proposition 1.2. Suppose
first that $AP(1)$ holds, and also assume that the weight $\lambda_{z}(n)$ is
as well distributed in arithmetic progressions as we assume $\Lambda(n)$ to
be, i.e. assume that if $z=X^{\nu}$ then

$\sum\limits_{X_{1}<n\leqslant
X_{2}}\lambda_{z}(n)\lambda_{z}(n+k)=\mathfrak{S}(k)(X-|k|)+O_{\varepsilon}\Big{(}\frac{\tau(k)kX}{\varphi(k)z}\Big{)}+O_{\varepsilon}(X^{\frac{1}{2}+\frac{\nu}{2}+\varepsilon}),$

for all $\nu<1$, for all $1\leqslant|k|\leqslant X$. Then the G–G–Ö–S method
shows that, for each fixed $\alpha\in[1,3/2)$,

$\liminf\limits_{T\rightarrow\infty}F_{T}(\alpha)\geqslant 3-2\alpha.$

This bound still vanishes at $\alpha=3/2$. In order to approach the AH bound,
and move beyond $3/2$, one needs to suppose some extra cancellation. As we
remarked in the introduction, if for some $\eta\geqslant 1/2$ one would go as
far as assuming asymptotics of the form

$\sum\limits_{X_{1}\leqslant n\leqslant
X_{2}}\Lambda(n)\Lambda(n+k)=\mathfrak{S}(k)(X-k)+O_{\varepsilon}(X^{\eta+\varepsilon}),$
(5.1)

uniform in $1\leqslant|k|\leqslant X^{1-\eta}$, then one would resolve
Conjecture 1.6 in the range $\alpha<\eta^{-1}$. (This follows from the mean
value theorems of Goldston-Gonek [14], for instance, or from Goldston-
Montgomery [16].) However, suppose one only assumes results on correlations
that seem to be weaker than the twin-prime conjecture, namely some bounds for
correlations with $\lambda_{z}$ that look like Proposition 1.2 and Lemma 4.2
but with stronger error terms, such as:

$\sum\limits_{X_{1}<n\leqslant
X_{2}}\lambda_{z}(n)\Lambda(n+k)=\mathfrak{S}(k)(X-|k|)+O_{\varepsilon}(zX^{\varepsilon})$
(5.2)

and

$\sum\limits_{X_{1}<n\leqslant
X_{2}}\lambda_{z}(n)\lambda_{z}(n+k)=\mathfrak{S}(k)(X-|k|)+O\Big{(}\frac{\tau(k)kX}{\varphi(k)z}\Big{)}+O_{\varepsilon}(zX^{\varepsilon})$
(5.3)

for all $z>X^{1/2}$. Then the method of Section 4 shows that we may take
$z=T^{1-\varepsilon}$, i.e. $\nu=\frac{1}{\alpha}-\varepsilon$, and establish
that

$F_{T}(\alpha)\geqslant 1-\alpha(1-\nu)-o(1)\geqslant
2-\alpha-\varepsilon-o(1).$

And since $\varepsilon$ is arbitrary, this establishes
$\liminf_{T\rightarrow\infty}F_{T}(\alpha)\geqslant 2-\alpha$, i.e. a lower
bound which matches AH.

Thus it seems reasonable to describe Theorem 1.7 of G–G–Ö–S as ‘half’ the AH
bound. Indeed, G–G–Ö–S work with a sieving level
$z=T^{\frac{1}{2}-\varepsilon}$; if one could double the exponent and take
$z=T^{1-\varepsilon}$ then one could achieve the AH bound.

The asymptotics (5.2) and (5.3) do not seem quite as formidable as (5.1).
However, to prove them – even assuming $AP(1)$ – one would nonetheless need to
detect extra cancellations in the sum

$\sum\limits_{d\leqslant
z}\rho_{z,d}\Big{(}\sum\limits_{\begin{subarray}{c}X_{1}<n\leqslant X_{2}\\\
d|n\end{subarray}}\Lambda(n+k)-1_{(k,d)=1}\frac{(X-|k|)}{\varphi(d)}\Big{)}$

coming from the oscillations in $\rho_{z,d}$, above and beyond the
cancellation from Conjecture 1.9. Of course there is a history of using ‘well-
factorability’ of certain linear sieve weights to prove better distribution in
arithmetic progressions than one might otherwise expect (the celebrated work
of Bombieri–Friedlander–Iwaniec [3], say, and more modern work by Maynard
[22]), though the cancellation one would need here is outrageously strong.

At such wild speculation, it seems appropriate to close the discussion.

## References

  * [1] S. Bettin and V. Chandee.  Trilinear forms with Kloosterman fractions.  Adv. Math., 328:1234–1262, 2018.
  * [2] E. Bombieri.  On twin almost primes.  Acta Arith., 28(2):177–193, 1975/76.
  * [3] E. Bombieri, J. B. Friedlander, and H. Iwaniec.  Primes in arithmetic progressions to large moduli.  Acta Math., 156(3-4):203–251, 1986.
  * [4] B. Conrey and H. Iwaniec.  Spacing of zeros of Hecke $L$-functions and the class number problem.  Acta Arith., 103(3):259–312, 2002.
  * [5] G. Coppola and M. Laporta.  On the correlations, Selberg integral and symmetry of sieve functions in short intervals, III.  Mosc. J. Comb. Number Theory, 6(1):3–24, 2016.
  * [6] W. Duke, J. Friedlander, and H. Iwaniec.  Bilinear forms with Kloosterman fractions.  Invent. Math., 128(1):23–43, 1997.
  * [7] D. W. Farmer, S. M. Gonek, and Y. Lee.  Pair correlation of the zeros of the derivative of the Riemann $\xi$-function.  J. Lond. Math. Soc. (2), 90(1):241–269, 2014.
  * [8] D. Fiorilli.  The distribution of the variance of primes in arithmetic progressions.  Int. Math. Res. Not. IMRN, (12):4421–4448, 2015.
  * [9] D. Fiorilli.  On Vaughan’s approximation: the first moment.  J. Lond. Math. Soc. (2), 95(1):305–322, 2017.
  * [10] É. Fouvry and M. Radziwiłł.  Level of distribution of unbalanced convolutions.  Preprint at arXiv:1811.08672.
  * [11] J. Friedlander and A. Granville.  Limitations to the equi-distribution of primes. I.  Ann. of Math. (2), 129(2):363–382, 1989.
  * [12] J. B. Friedlander and D. A. Goldston.  Variance of distribution of primes in residue classes.  Quart. J. Math. Oxford Ser. (2), 47(187):313–336, 1996.
  * [13] D. A. Goldston.  A lower bound for the second moment of primes in short intervals.  Exposition. Math., 13(4):366–376, 1995.
  * [14] D. A. Goldston and S. M. Gonek.  Mean value theorems for long Dirichlet polynomials and tails of Dirichlet series.  Acta Arith., 84(2):155–192, 1998.
  * [15] D. A. Goldston, S. M. Gonek, A. E. Özlük, and C. Snyder.  On the pair correlation of zeros of the Riemann zeta-function.  Proc. London Math. Soc. (3), 80(1):31–49, 2000.
  * [16] D. A. Goldston and H. L. Montgomery.  Pair correlation of zeros and primes in short intervals.  In Analytic number theory and Diophantine problems (Stillwater, OK, 1984), volume 70 of Progr. Math., pages 183–203. Birkhäuser Boston, Boston, MA, 1987.
  * [17] D. A. Goldston and C. Y. Yıldı rım.  On the second moment for primes in an arithmetic progression.  Acta Arith., 100(1):85–104, 2001.
  * [18] D. R. Heath-Brown.  The ternary Goldbach problem.  Rev. Mat. Iberoamericana, 1(1):45–59, 1985.
  * [19] C. Hooley.  On the Barban-Davenport-Halberstam theorem. XIII.  Acta Arith., 94(1):53–86, 2000.
  * [20] A. Juhas.  Lower bound estimates related to primes in short intervals.  PhD thesis, Universität Münster, 2016.
  * [21] J. C. Lagarias and B. Rodgers.  Higher Correlations and the Alternative Hypothesis.  Q. J. Math., 71(1):257–280, 2020.
  * [22] J. Maynard.  Primes in arithmetic progressions to large moduli ii: Well-factorable estimates.  Preprint at arXiv:2006.07088, 2020.
  * [23] H. L. Montgomery.  Topics in multiplicative number theory.  Lecture Notes in Mathematics, Vol. 227. Springer-Verlag, Berlin-New York, 1971.
  * [24] H. L. Montgomery.  The pair correlation of zeros of the zeta function.  In Analytic number theory (Proc. Sympos. Pure Math., Vol. XXIV, St. Louis Univ., St. Louis, Mo., 1972), pages 181–193, 1973.
  * [25] H. L. Montgomery and K. Soundararajan.  Primes in short intervals.  Comm. Math. Phys., 252(1-3):589–617, 2004.
  * [26] M. Ram Murty.  Problems in analytic number theory, volume 206 of Graduate Texts in Mathematics.  Springer, New York, second edition, 2008.  Readings in Mathematics.
  * [27] A. M. Odlyzko.  On the distribution of spacings between zeros of the zeta function.  Math. Comp., 48(177):273–308, 1987.
  * [28] E. C. Titchmarsh.  The theory of the Riemann zeta-function.  The Clarendon Press, Oxford University Press, New York, second edition, 1986.  Edited and with a preface by D. R. Heath-Brown.
  * [29] R. C. Vaughan.  Moments for primes in arithmetic progressions. I.  Duke Math. J., 120(2):371–383, 2003.
  * [30] R. C. Vaughan.  Moments for primes in arithmetic progressions. II.  Duke Math. J., 120(2):385–403, 2003.
  * [31] A. Walker.  Linear inequalities in primes.  Preprint at arxiv:1901.04855, to appear in Journal d’Analyse.