# Modelling cross-correlations of ultra-high-energy cosmic rays and galaxies

Federico R. Urban  Stefano Camera  and David Alonso

###### Abstract

The astrophysical engines that power ultra-high-energy cosmic rays (UHECRs)
remain to date unknown. Since the propagation horizon of UHECRs is limited to
the local, anisotropic Universe, the distribution of UHECR arrival directions
should be anisotropic. In this paper we expand the analysis of the potential
for the angular, harmonic cross-correlation between UHECRs and galaxies to
detect such anisotropies. We do so by studying proton, oxygen and silicium
injection models, as well as by extending the analytic treatment of the
magnetic deflections. Quantitatively, we find that, while the correlations for
each given multipole are generally weak, (1) the total harmonic power summed
over multipoles is detectable with signal-to-noise ratios well above 5 for
both the auto-correlation and the cross-correlation (once optimal weights are
applied) in most cases studied here, with peaks of signal-to-noise ratio
around between 8 and 10 at the highest energies; (2) if we combine the UHECR
auto-correlation and the cross-correlation we are able to reach detection
levels of $3\sigma$ and above for individual multipoles at the largest scales,
especially for heavy composition. In particular, we predict that the combined-
analysis quadrupole could be detected already with existing data.

##  1 Introduction

Sixty-two years ago the first ultra-high-energy cosmic ray (UHECR) was
detected; in the decades since observations have continued to accumulate
UHECRs up to extreme energies exceeding $E\geq 100\,\mathrm{EeV}$
($10^{20}\,\mathrm{EeV}$) [1]. The mystery of how to accelerate particles,
which appear to be overwhelmingly charged nuclei, to such extraordinary
energies remains as elusive as ever [2]. There is no compelling theoretical
model for the UHECR acceleration mechanism among several promising candidates.
More importantly, despite tremendous recent experimental progress [3, 4, 5, 6,
7, 8, 9, 10, 11, 12], we have not yet been able to trace the UHECRs back to
their sources. The difficulty in identifying the sources of UHECRs from the
data stems from two coupled problems: it is very hard to measure the charge,
or atomic number $Z$, of UHECRs with good accuracy [13, 14] and it is very
hard to realistically model the three-dimensional structure of the Galactic
magnetic field (GMF) from integrated two-dimensional information [15].111There
is an additional problem: the accuracy with which the energy of a UHECR event
is determined has a systematic uncertainty of approximately 15% to 20% [16].
This is, however, not as relevant as the uncertainties on UHECR charge and
intervening magnetic fields. Thus, the arrival directions of UHECRs as
observed from Earth are displaced from their arrival directions before
encountering the GMF distorting “lens” by a mostly unknown angle
$\theta\propto ZB/E$, where $B$ is the GMF strength, in a mostly unknown
direction.

In order to try to bypass some of these issues and to extract as much
information as possible from the sky distribution of arrival directions of
UHECRs, in particular from its anisotropies, in a previous work we have
proposed a new observable, the angular, harmonic cross-correlation (XC)
between the angular distribution of galaxies and that of UHECR arrival
directions, and assessed its performance in a proton-only injection model
[17]. In that work we showed how the XC is a complementary observable to the
more familiar angular, harmonic UHECR auto-correlation (AC): (i) the XC is
more sensitive to small-scale angular anisotropies and in certain conditions
can be easier to detect than the AC—conversely, the AC is more sensitive to
large-scale anisotropies; (ii) because systematic uncertainties do not cross-
correlate and, under some conditions, statistical noises do not strongly
cross-correlate either, the XC is an experimentally cleaner observable; (iii)
the XC encodes more astrophysical information than the AC, and, since the two
respond differently to model parameters, taken together they can break
degeneracies. The AC and XC have been already recognised as useful tools to
detect the angular anisotropy of the UHECR flux [18, 19, 20, 21, 22, 23].

In this paper we extend and improve our analysis in several ways, yet keeping
with a mostly analytic treatment. (1) Beyond the pure hydrogen model (H1) with
injection slope $\gamma=2.6$, we extend the analysis to an oxygen model (O16)
with injection slope $\gamma=2.1$ at all energies, and a silicium (Si28) model
with $\gamma=1.5$ and a sharp $E_{\text{max}}=280\,\mathrm{EeV}$ energy
cutoff, inspired by the analysis of [24]. (2) We extend the analysis to
different number of detected events to forecast the progress that can be made
with future observations. (3) We develop a more realistic description of
galaxy catalogues, including the impact of flux limits and incompleteness and
incomplete sky coverage. (4) We extend our treatment of the GMF, and refine it
by accounting for the latitude-dependence of the expected deflections [25].

The rest of the paper is structured as follows. In Section 2 we summarise the
theory of the AC and XC, as well as describe the UHECR injection models and
our treatment of the GMF. Section 3 collects our results, and Section 4
concludes with a discussion and an outlook for future work.

##  2 Method

###  2.1 Theoretical model

####  2.1.1 Sources

We assume that UHECR sources are numerous (that is, with a comoving density
larger than $10^{-5}\,\text{Mpc}^{-3}$) steady but faint, so that the
probability to observe an event from each source within the life-cycle of a
typical detector is much less than one [26, 27].222In the opposite case UHECR
multiplets are expected, see for example [28]. We also assume that their
distribution is correlated with the distribution of galaxies detected by a
given survey. This can happen in two opposite ways: either UHECR emitters are
a subset of the galaxies in the survey catalogue, or they are an entirely
different population of galaxies, which will nonetheless exhibit some
correlation with the former ones because they are hosted within the same
underlying cosmic large-scale structure.

Each source injects a flux of UHECRs with a power-law spectrum $\varPhi\propto
E^{-\gamma}$ up to a very high energy, unless an end-of-spectrum energy cutoff
$E_{\text{max}}$ is explicitly indicated. Therefore, the integral UHECR flux,
(i.e. the number of UHECRs detected per unit detector area, time, steradian
and energy) above a certain energy cut $E_{\text{cut}}$, in a given direction
$\hat{\bm{n}}$, and for a single species with atomic number $Z$, reads

$\displaystyle\varPhi(E_{\text{cut}},\hat{\bm{n}};\gamma,Z)$
$\displaystyle\coloneqq\frac{\mathcal{E}_{0}}{4\,\pi}\,\frac{\bar{n}_{\rm
s,c}\,E_{\text{cut}}}{1-\gamma}\left(\frac{E_{\text{cut}}}{E_{0}}\right)^{-\gamma}\int\mathrm{d}z\,\frac{\alpha(z,E_{\text{cut}};\gamma,Z)}{H(z)\,(1+z)}\,\left[1+\delta_{\rm
s}(z,\chi\,\hat{\bm{n}})\right]\;,$ (2.1)

where $\bar{n}_{\rm s,c}$ is the comoving average number density of sources
(which is assumed not to be evolving with redshift, but readily generalised to
such a case), $\mathcal{E}_{0}$ is the overall emissivity normalisation
factor, $E_{0}$ the energy at which the flux is normalised,
$\chi\equiv\chi(z)$ is the radial comoving distance to redshift $z$, and
$\delta_{\rm s}(z,\chi\,\hat{\bm{n}})$ is the UHECR sources density contrast
(see also [17]). The function $\alpha(z,E_{\text{cut}};\gamma,Z)$ is the
attenuation function that gives the probability that a UHECR detected with
energy above $E_{\text{cut}}$ had originated from a source located at redshift
$z$ and which emits UHECRs with atomic number $Z$ and injection slope
$\gamma$. We calculated the attenuation function by following $10^{6}$
simulated UHECRs with _SimProp_ v2r4 [29] with energies above
$E=10\,\mathrm{EeV}$ (with an upper cutoff of
$E_{\text{max}}=10^{5}\,\mathrm{EeV}$ for H1 and O16, and
$E_{\text{max}}=280\,\mathrm{EeV}$ for Si28), for redshifts $z\leq 0.3$. With
_SimProp_ , we accounted for all energy losses, namely adiabatic losses and
losses due to interactions with cosmic microwave background photons and with
extra-Galactic background photons according to the model of [30].

We are interested in the anisotropies in the UHECR flux,

$\displaystyle\varDelta(E_{\text{cut}},\hat{\bm{n}};\gamma,Z)$
$\displaystyle\coloneqq\frac{\varPhi(E_{\text{cut}},\hat{\bm{n}};\gamma,Z)-\bar{\varPhi}(E_{\text{cut}};\gamma,Z)}{\bar{\varPhi}(E_{\text{cut}};\gamma,Z)}$
(2.2)
$\displaystyle=\int\text{d}\chi\,\phi(E_{\text{cut}},\chi;\gamma,Z)\,\delta_{\rm
s}(z,\chi\,\hat{\bm{n}})\;,$ (2.3)

where
$4\,\pi\,\bar{\varPhi}(E_{\text{cut}};\gamma,Z)\coloneqq\int\text{d}^{2}\hat{n}\,\varPhi(E_{\text{cut}},\hat{\bm{n}};\gamma,Z)$
is the average flux across the sky. In the second line, we have recast the
anisotropy as an integral of the UHECR radial kernel

$\displaystyle\phi(E_{\text{cut}},z;\gamma,Z)$
$\displaystyle=\left[\int\text{d}\tilde{z}\,\frac{\alpha(\tilde{z},E_{\text{cut}};\gamma,Z)}{H(\tilde{z})\,(1+\tilde{z})}\right]^{-1}\frac{\alpha(z,E_{\text{cut}};\gamma,Z)}{1+z}\;,$
(2.4)

where $H(z)$ is the Hubble parameter and $\text{d}\chi/\text{d}z=1/H$.

####  2.1.2 Injection models

We work with three injection models: H1 with injection slope $\gamma=2.6$ at
all energies; O16 with injection slope $\gamma=2.1$ at all energies; Si28 with
$\gamma=1.5$ and a sharp $E_{\text{max}}=280\,\mathrm{EeV}$ energy cutoff.
These are the same three models that were analysed in [24] and are
representative of the realistic fits to the observed UHECR energy spectrum.

Heavy nuclei with energy $E\gg A\,E_{\text{cut}}$, with $A$ their mass number,
disintegrate rapidly on their way from the source to the Earth. This means
that the attenuation function $\alpha(z,E_{\text{cut}};\gamma,Z)$ for $Z>1$ is
going to be broken up into several attenuation functions, one for each species
that the original nucleus breaks into. For simplicity, and because it is well
within our uncertainties (that are driven by the incomplete knowledge of the
magnetic fields), we follow [24] and approximate the flux coming from a heavy
$(A,Z)$ nucleus as the sum of a fraction
$f_{\text{p}}=A^{2-\gamma}/(A^{2-\gamma}+1)$ from “light” elements (which in
_SimProp_ we count as protons), and the remaining $1-f_{\text{p}}$ part from
“heavy” elements (which in _SimProp_ we count as the original nuclei). Notice
that we can still use this formula even for $A=1$, although physically nothing
happens in this case. Therefore, in this approximation we substitute

$\displaystyle\alpha(z,E_{\text{cut}};\gamma,Z)$ $\displaystyle\rightarrow
f_{\text{p}}\,\alpha(z,E_{\text{cut}};\gamma,1)+(1-f_{\text{p}})\,\alpha(z,E_{\text{cut}};\gamma,Z)\;.$
(2.5)

Because of our choice of maximum energy cutoff at
$E_{\text{max}}=280\,\mathrm{EeV}$ for Si28, and because we are always
interested in integral UHECR spectra with energy cuts $E_{\text{cut}}\gg
10\,\mathrm{EeV}$ (in fact we will always work with $E_{\text{cut}}\geq
40\,\mathrm{EeV}$), the near entirety of Si28 nuclei which arrive at the
detector with $E\geq E_{\text{cut}}$ do not break up on their journey to the
Earth. In the O16 model instead we find $f_{\text{p}}\approx 0.43$—this
component will be visible in the AC and XC.

####  2.1.3 Energy spectrum

The energy spectrum on Earth, and with it the expected number of UHECR events
used to estimate the shot-noise contribution to the UHECR anisotropy map,
should be self-consistently derived from the injection spectrum propagated
from the source. However, within our level of accuracy, we can more simply
choose the final energy spectrum in such a way that it is representative of
what is observed in current experimental facilities. Therefore, in order to
determine the number of UHECR events which in turn fix the levels of Poisson
noise in the angular power spectra, we define the differential observed energy
spectrum as

$J(E)=\begin{cases}J_{0}\,\left(\dfrac{E}{\text{EeV}}\right)^{-\gamma_{1}}&E\leq
E_{1}\\\
J_{0}\,\left(\dfrac{E_{1}}{\text{EeV}}\right)^{-\gamma_{1}}\,\left(\dfrac{E}{E_{1}}\right)^{-\gamma_{2}}&E>E_{1}\end{cases}\;,$
(2.6)

where $\gamma_{1}=3$, $\gamma_{2}=5$, $E_{1}=10^{19.725}\,\mathrm{eV}$, and
$J_{0}=4.28\times 10^{6}/\text{EeV}$ as benchmark values. This choice gives a
total number of events above $E_{\text{cut}}$, defined as
$N_{\text{ev}}\coloneqq\int_{E_{\text{cut}}}\text{d}E\,J(E)$, of 1000, 200, 30
events for $E_{\text{cut}}=10^{19.6}\,\mathrm{eV}\approx 40\,\mathrm{EeV}$,
$E_{\text{cut}}=10^{19.8}\,\mathrm{eV}\approx 63\,\mathrm{EeV}$,
$E_{\text{cut}}=10^{20}\,\mathrm{eV}=100\,\mathrm{EeV}$, respectively. These
are the number of events we can expect to have for a full-sky observatory with
statistics comparable with running facilities [31, 32].

####  2.1.4 Galaxies

The anisotropy in galaxy number counts, using the same language, is defined as

$\displaystyle\varDelta_{\rm g}(\hat{\bm{n}})$
$\displaystyle\coloneqq\frac{N_{\rm g}(\hat{\bm{n}})-\bar{N}_{\rm
g}}{\bar{N}_{\rm g}}=\int\mathrm{d}\chi\,\phi_{\rm g}(\chi)\,\delta_{\rm
g}(z,\chi\,\hat{\bm{n}})\;,$ (2.7)

where $N_{\rm g}(\hat{\bm{n}})$ is the number of galaxies in a given direction
$\hat{\bm{n}}$ and $\bar{N}_{\rm g}$ its average over the celestial sphere. In
analogy to the UHECR anisotropies of Eq. 2.2, we define the three-dimensional
galaxy overdensity as $\delta_{\rm g}(z,\chi\,\hat{\bm{n}})$, and $\phi_{\rm
g}(\chi)$ is kernel of the galaxy distribution, i.e. the weighed distribution
of galaxy distances (cfr. Eq. 2.4). The galaxy kernel $\phi_{\rm g}(\chi)$ is
given by

$\displaystyle\phi_{\rm
g}(\chi)\coloneqq\left[\int\mathrm{d}\tilde{\chi}\,\tilde{\chi}^{2}\,w(\tilde{\chi})\,\bar{n}_{\rm
g,c}(\tilde{\chi})\right]^{-1}\,\chi^{2}\,w(\chi)\,\bar{n}_{\rm g,c}(\chi)\;,$
(2.8)

where $\bar{n}_{\rm g,c}(\chi)=\chi^{-2}\,\mathrm{d}N_{\rm g}/\mathrm{d}\chi$
is the comoving (volumetric) number density of galaxies in the sample,
$\mathrm{d}N_{\rm g}$ being the angular number density of galaxies in a bin of
radial width $\mathrm{d}\chi$. The quantity $w(\chi)$ is an optional distance-
dependent weight that can be applied to all the objects in the galaxy
catalogue, provided their redshifts are known. Assuming Poisson statistics, it
can be shown that the optimal weights that maximise the signal of the galaxy-
UHECR XC are given by

$\displaystyle w(\chi)$
$\displaystyle=\frac{\alpha(z,E_{\text{cut}};\gamma,Z)}{(1+z)\,\chi^{2}\,\bar{n}_{\rm
g,c}(\chi)}\;.$ (2.9)

The properties of the galaxy sample are modelled after the 2MASS Redshift
Survey (2MRS) [33], which constitutes one of the most complete full-sky
spectroscopic low-redshift surveys, covering the full sky. After masking out
the Galaxy, the 70% sky coverage for the actual 2MRS catalogue only degrades
our signal by a factor of $\sqrt{0.7}\approx 0.84$. For simplicity we also
assume that the catalogue is flux-complete, and comment on incompleteness in
Section 3.1. The redshift distribution of galaxies is well fitted by

$\displaystyle\frac{\text{d}N_{\rm g}}{\text{d}z}=\frac{N_{\rm
g}\,\beta}{\varGamma\left[(m+1)/\beta\right]}\,\frac{z^{m}}{z_{0}^{m+1}}\,\exp{\left[-{\left(\frac{z}{z_{0}}\right)}^{\beta}\right]}\;,$
(2.10)

with $N_{\rm g}=43\,182$ the total number of sources, $\beta=1.64$, $m=1.31$
and $z_{0}=0.0266$ [34].

Given that the number of detected UHECRs is small (for $E\gtrsim
40\,\mathrm{EeV}$ it is about 1000) we assume that on average each UHECR comes
from a different source, which emits UHECRs isotropically, and that all UHECR
sources are in the galaxy catalogue. For lack of a better model, we will
assume that $\delta_{\rm g}(z,\chi\,\hat{\bm{n}})=\delta_{\rm
s}(z,\chi\,\hat{\bm{n}})$. This relation may vary by a factor of $O(1)$ if the
UHECR flux correlates with other astrophysical properties (e.g. optical
luminosity, halo mass), as is likely the case.

The fit Eq. 2.10 does not extend to $z\rightarrow 0$ because an actual
catalogue will be limited at the distance of the first object. Moreover, the
many-sources assumption is valid only above a minimum source density of $\sim
10^{-5}\,\mathrm{Mpc}^{-3}$ [26, 27]. This is essentially because, otherwise,
a few powerful and nearby sources would outshine the UHECRs flux from more
distant ones, and we would observe multiplets of UHECRs from one (or more)
directions. Therefore, in all that follows we assume a minimum redshift of
$z_{\rm min}=0.0012$, which roughly corresponds to a minimum distance to the
closest galaxy of $5\,\mathrm{Mpc}$, as is customary in the literature. This
is especially important because formally the UHECR kernel Eq. 2.4
monotonically increases with inverse redshift and peaks at $z=0$—indeed, if we
did not truncate the galaxy kernel at the minimum $z_{\rm min}$, we would
introduce an unphysical cancellation of cosmic variance (see discussion about
Fig. 3 in Section 3).

####  2.1.5 Harmonic spectra

The two-dimensional fields $\varDelta_{a}(\hat{\bm{n}})$ can be decomposed
into their harmonic coefficients

$\displaystyle\varDelta_{\ell m}^{a}$
$\displaystyle\coloneqq\int\text{d}^{2}\hat{n}\,Y^{*}_{\ell
m}(\hat{\bm{n}})\,\varDelta_{a}(\hat{\bm{n}})\;,$ (2.11)

where $Y_{\ell m}$ are the spherical harmonic functions and
$a\in\left\\{\text{CR},\,{\rm g}\right\\}$ for UHECRs and galaxies,
respectively. The covariance of the $\varDelta_{\ell m}$ is the angular power
spectrum $\mathcal{S}^{ab}_{\ell}$, defined as

$\displaystyle\left\langle\varDelta^{a}_{\ell
m}\,\varDelta^{b\ast}_{\ell^{\prime}m^{\prime}}\right\rangle$
$\displaystyle\coloneqq\delta^{\rm(K)}_{\ell\ell^{\prime}}\,\delta^{\rm(K)}_{mm^{\prime}}\,\mathcal{S}^{ab}_{\ell}\;,$
(2.12)

with $\delta^{\rm(K)}$ being the Kronecker symbol.

For broad kernels such as Eqs. 2.4, LABEL: and 2.8 the harmonic, angular power
spectrum $\mathcal{S}_{\ell}^{ab}$ between two projected quantities
$\varDelta_{a}$ and $\varDelta_{b}$ is related to their three-dimensional
Fourier-space power spectrum $P_{ab}(z,k)$ by

$\displaystyle\mathcal{S}^{ab}_{\ell}=\int\frac{\mathrm{d}\chi}{\chi^{2}}\,\phi_{a}(\chi)\,\phi_{b}(\chi)\,P_{ab}\left[k=\frac{\ell+1/2}{\chi},z(\chi)\right]\;,$
(2.13)

where $\phi_{a}$ and $\phi_{b}$ are the radial kernels of both quantities. The
synthetic $P_{ab}(k,z)$ is modelled according to the halo distribution model
prescription, adapted to the specifics of the 2MRS catalogue [35, 34].

A given UHECR or galaxy observation consist of both the signal, whose
harmonic, angular power spectrum is given by Eq. 2.13, and a noise power
spectrum $\mathcal{N}^{ab}_{\ell}$, which combined give the observed angular
power spectrum:

$\displaystyle C^{ab}_{\ell}$
$\displaystyle\coloneqq\mathcal{S}^{ab}_{\ell}+\mathcal{N}^{ab}_{\ell}\;.$
(2.14)

The noise, being the fields $\varDelta_{a}(\hat{\bm{n}})$ associated to
discrete point processes represented by the angular positions of the UHECRs
and the galaxies in each sample, is given by

$\displaystyle\mathcal{N}^{ab}_{\ell}=\frac{\bar{N}_{\varOmega,a\wedge
b}}{\bar{N}_{\varOmega,a}\,\bar{N}_{\varOmega,b}}\;,$ (2.15)

where $\bar{N}_{\varOmega,a}$ and $\bar{N}_{\varOmega,b}$ are respectively the
angular number density of points in sample $a$ and $b$, and
$\bar{N}_{\varOmega,a\wedge b}$ is the angular number density of points shared
in common.

For the galaxy clustering auto-correlation, the noise power spectrum reads
(see Appendix B for a derivation)

$\displaystyle\mathcal{N}^{\rm
g\,g}_{\ell}=\left[\int\mathrm{d}\chi\,\chi^{2}\,w(\chi)\,\bar{n}_{\rm
g,c}(\chi)\right]^{-2}\,\int\mathrm{d}\chi\,\chi^{2}\,w^{2}(\chi)\,\bar{n}_{\rm
g,c}(\chi)\;.$ (2.16)

###  2.2 Signal-to-noise ratio

We estimate the signal-to-noise ratio (SNR) of the UHECR anisotropies under
the assumption that the fields being correlated ($\varDelta_{\rm CR}$,
$\varDelta_{\rm g}$ in our case) are Gaussian. The explicit expressions for
the AC and XC per-$\ell$ SNR read

$\displaystyle\text{SNR}^{\rm CR\,CR}_{\ell}$
$\displaystyle=\sqrt{2\,(2\ell+1)\,\left(\frac{\mathcal{S}^{{\rm
CR\,CR}}_{\ell}}{C^{{\rm CR\,CR}}_{\ell}}\right)^{2}}\;,$ (2.17)
$\displaystyle\text{SNR}^{{\rm g\,CR}}_{\ell}$
$\displaystyle=\sqrt{(2\ell+1)\,\frac{(\mathcal{S}^{{\rm
g\,CR}}_{\ell})^{2}}{C^{\rm g\,g}_{\ell}C^{{\rm CR\,CR}}_{\ell}+(C^{{\rm
g\,CR}}_{\ell})^{2}}}\;,$ (2.18)

and the total SNR is the sum in quadrature, i.e.

$\displaystyle\text{SNR}^{ab}\coloneqq\sqrt{\sum_{\ell=\ell_{\rm
min}}^{\ell_{\rm max}}\left(\text{SNR}^{ab}_{\ell}\right)^{2}}\;.$ (2.19)

Given the relatively small number of UHECRs currently measured, the shot noise
in the UHECR flux is the dominant contribution to the uncertainties, and we
are in a situation wherein $\mathcal{S}\ll\mathcal{N}$ for both the AC and the
XC. This means that the per-$\ell$ SNR can be approximated as
$\text{SNR}^{ab}_{\ell}\sim\mathcal{S}^{ab}_{\ell}/\mathcal{N}^{ab}_{\ell}$ to
a good accuracy: whereas we retain the full expressions in our computations,
this simplified expression guides our analytic intuition when incorporating
new effects, such as the magnetic smearing and leaking, and sky cuts.

###  2.3 Magnetic fields

####  2.3.1 Magnetic beam

Upon reaching the Milky Way the UHECRs meet the GMF screen which perturbs
their trajectories on their way to the Earth, thus obfuscating the original
anisotropy of their arrival direction distribution. The GMF has an amplitude
of about a few $\text{\textmu}\mathrm{G}$, and a complex three-dimensional
structure which to date is still poorly understood [36, 15, 37, 38]. In a
simplified treatment we can account for the effects of the GMF by smearing the
map of sources below a certain angular scale. The magnetic beam reads

$\displaystyle{\cal B}(r)$
$\displaystyle\coloneqq\frac{1}{2\,\pi\,\sigma^{2}}\exp\left[-\frac{r^{2}}{2\,\sigma^{2}}\right]\;,$
(2.20) $\displaystyle{\cal B}_{\ell}$
$\displaystyle\simeq\exp\left[-\frac{\ell\,(\ell+1)\,\sigma^{2}}{2}\right]\;,$
(2.21)

in position and harmonic space, ${\cal
B}_{\ell}\coloneqq\int\text{d}^{2}\hat{r}\,{\cal
B}(\cos\omega)\,P_{\ell}(\cos\omega)$, respectively. Here
$r^{2}=2\,(1-\hat{\bm{n}}_{1}\cdot\hat{\bm{n}}_{2})\coloneqq
2\,(1-\cos\omega)$ for two directions $\hat{\bm{n}}_{1}$ and
$\hat{\bm{n}}_{2}$, $P_{\ell}(\cos\omega)$ is a Legendre polynomial of order
$\ell$ and the width of the Gaussian beam, the displacement $\sigma$,333The
deflection angle $\theta$, which is the angle by which a charged particle is
deflected from its trajectory in a magnetic field, is not the same as the
observable displacement angle $\sigma$, which is the angle in the sky between
the original and actual directions of the charged particle. The two quantities
are related as $\langle\sigma^{2}\rangle=\langle\theta^{2}\rangle/3$ [39]. is
expressed as

$\displaystyle\sigma(b)\coloneqq\frac{1}{\sqrt{2}}\left(\frac{40\,\text{EV}}{E/Z}\right)\frac{1\,\text{deg}}{\sin^{2}b+0.15}\;,$
(2.22)

where $b$ is galactic latitude, see [24, 25] (notice the factor $1/\sqrt{2}$
difference with the definition and normalisation of that work)—in the last
step of Eq. 2.21 we have formally assumed that $\ell\gg 1$ but we have
verified that within our approximations this can be applied even for
$\ell\sim{\cal O}(1)$.444The expression Eq. 2.22 was obtained by mapping the
observed variance of rotation measures in the sky to the predicted cosmic-ray
deflections, bypassing almost entirely the specifics of the magnetic field
model itself.

In order to be able to treat the problem of magnetic deflections mostly
analytically, in this work we smear uniformly across the patch of the sky we
are interested in with a displacement angle that is the maximum of Eq. 2.21
within that patch:

$\displaystyle\mathcal{S}^{\rm CR\,CR}_{\ell}\rightarrow{\cal
B}_{\ell}^{2}\,\mathcal{S}^{\rm CR\,CR}_{\ell}\;,~{}~{}$
$\displaystyle~{}~{}\mathcal{S}^{\rm g\,CR}_{\ell}\rightarrow{\cal
B}_{\ell}\,\mathcal{S}^{\rm g\,CR}_{\ell}\;,$ (2.23)

for the AC and XC, respectively. Therefore, since the magnetic deflections
only depend on $|b|$, we separate the sky into two regions: a band around the
Galactic plane up to a given latitude $|b|<b_{\text{cut}}$ for which the
displacement is $\sigma(0)$; and the two polar caps $|b|\geq b_{\text{cut}}$
where we use instead $\sigma(b_{\text{cut}})$. We then can add the two SNR
(once their respective sky fractions have been accounted for) as
$\text{SNR}^{ab}_{\ell}=\sqrt{\left(\text{SNR}^{ab,>}_{\ell}\right)^{2}+\left(\text{SNR}^{ab,<}_{\ell}\right)^{2}}$
where the $>$ ($<$) superscript stands for $|b|\geq b_{\text{cut}}$
($|b|<b_{\text{cut}}$).555Using more bands is possible, but the improvement is
minimal at the expense of a significant increase in complexity due to the
“leaking” (see Figure 1). For instance, the maximal displacement for 40 EeV
protons are of approximately 4.7 deg when $b=0$, but if we cut off the galaxy
and retain only $|b|\geq 40\,\mathrm{deg}$ we find $\sigma\approx
1.3\,\mathrm{deg}$ in the polar caps covering about 36% of the sky (for Si28
we find respectively $66\,\mathrm{deg}$ and $18\,\mathrm{deg}$). Notice that
we employ this prescription here simply to have a better estimate of what can
be expected from an actual data analysis.

Since the magnetic deflections depend not only on $Z$ but also on the energy,
we can improve on the analysis by binning the UHECR flux in five logarithmic
energy bins with logarithmic width of 0.1, and compute the total flux by
adding them up proportionally to the UHECR energy spectrum—in the O16 case we
do this for the fractions of oxygen and proton nuclei separately. We then
apply to each bin a magnetic smearing obtained from the lowest energy in the
bin, that is, $E_{\text{cut},\,i}$, where
$E_{\text{cut}}\in[E_{\text{cut},\,i}\;,E_{\text{cut},\,i+1}]$ (for the last
bin we set $E_{\text{cut},\,i+1}=\infty$), as obtained from the energy
spectrum Eq. 2.6 below. The signal is generalised as

$\displaystyle\mathcal{S}^{\rm CR\,CR}_{\ell}$
$\displaystyle\rightarrow\sum_{ij}Q_{i}\,Q_{j}\,{\cal B}_{\ell,i}\,{\cal
B}_{\ell,j}\,\mathcal{S}^{\rm CR\,CR}_{\ell}\;,$ (2.24)
$\displaystyle\mathcal{S}^{\rm g\,CR}_{\ell}$
$\displaystyle\rightarrow\sum_{i}Q_{i}\,{\cal B}_{\ell,i}\,\mathcal{S}^{\rm
g\,CR}_{\ell}\;,$ (2.25)

where $Q_{i}$ are the fractions of flux in the energy bin $i$ and ${\cal
B}_{\ell,i}\coloneqq{\cal B}_{\ell}[\sigma(E_{\text{cut},\,i})]$.

####  2.3.2 Magnetic leaking

Because in our treatment the magnetic smearing Eq. 2.22 is latitude-dependent,
in Section 3 we will assess the performance of the AC and XC in the case we
split the sky in two in order to contain the effects of the smearing.
Physically, smearing the map of sources with a certain angle reproduces the
effects of the deflections experienced by UHECRs while propagating through the
GMF. Therefore, in the vicinity of $|b|=b_{\text{cut}}$ it is possible that
some UHECRs “leak” between the two regions.

In order to assess how many UHECRs do so we have simulated the propagation of
$10^{6}$ UHECRs uniformly distributed on the sphere in a random magnetic field
that follows the latitude-dependence spelt out in Eq. 2.22; the cell size was
chosen to be 10 pc, and each UHECR is propagated until it leaves the sphere
which has a radius of 10 kpc (making on average approximately few$\times
10^{3}$ steps). The overall strength of the magnetic field is normalised to
$1.4\,\text{\textmu}\mathrm{G}$ in order to reproduce a distribution of
displacements that follows Eq. 2.22 for 95% of the UHECRs: this is
conservative in the sense that if we require that _all_ the UHECRs stay below
the upper limit Eq. 2.22 we would have on average smaller deflections. Note
that there are some small deviations from this distribution at very large
deflections above $50\,\mathrm{deg}$: in that case the actual distribution of
displacements is slightly wider with $b$; hence, the choice of keeping the
same formula Eq. 2.22 even in this case is conservative. For each
$b_{\text{cut}}$, we then count what is the fraction of UHECRs that originally
was in the inner, high-displacement region and finished in the polar caps.
This fraction of UHECRs would have to be assigned a larger smearing angle than
$\sigma(b_{\text{cut}})$; to be conservative, we count the leaked fraction of
UHECRs as isotropic, which means that they only contribute to the noise and
not the signal. What we find is that even for the lowest energies
$E_{\text{cut}}=40\,\mathrm{EeV}$ and heavier elements (Si28 with $Z=14$), the
maximum contamination from the inner, high-deflection regions never exceeds
20% (at $b_{\text{cut}}\approx 25\,\mathrm{deg}$), while it is below 10% for
$E_{\text{cut}}=100\,\mathrm{EeV}$ and irrelevant for H1 in all cases—for a
cut at $b_{\text{cut}}=40\,\mathrm{deg}$ we have, for the worst case of Si28,
less than 15%, 7%, 4% for $E_{\text{cut}}=40\,\mathrm{EeV}$,
$E_{\text{cut}}=63\,\mathrm{EeV}$ and $E_{\text{cut}}=100\,\mathrm{EeV}$,
respectively. This is shown in Fig. 1. Moreover, since we bin the UHECR energy
spectrum in energy bins, the amount of leaked UHECRs is in fact even
smaller—notice that we neglect the small fluctuations between energy bins
here, as they are subdominant.

Figure 1: Percentage of synthetic UHECRs in the polar caps that leaked out
from the inner, high-deflection region (around the Galactic plane) for H1
(left), O16 (centre) and Si28 (right).

##  3 Results

The magnetic deflections decrease as we look away from the Milky Way. Within
this model there is therefore a trade-off: on the one hand choosing a region
in the sky where the displacements are small improves the signal because
magnetic smearing suppresses angular anisotropies in proportion to the
displacement angle $\sigma$; on the other hand the smaller the fraction of the
sky $f_{\text{sky}}$ we observe, the larger cosmic variance. In order to
quantify this trade-off, in Fig. 2 we show the total SNR for AC and optimal XC
in the range $\ell=[2,1000]$ for the three injection models H1, O16 and Si28,
each of them for the three energy cuts of $E_{\text{cut}}\approx
40\,\mathrm{EeV}$, $E_{\text{cut}}\approx 63\,\mathrm{EeV}$ and
$E_{\text{cut}}=100\,\mathrm{EeV}$, where we divide the sky in two regions:
the polar caps with $|b|\geq b_{\text{cut}}$ and for which the displacement is
given by $\sigma(b_{\text{cut}})$ are shown in solid, whereas the SNR from the
inner region around the Galaxy (with displacement $\sigma(0)$) is shown as a
dashed line.666In this section we always only consider the optimal XC as the
SNR of the non-optimised XC is always much smaller. The most notable feature
here is that, if UHECRs are heavy nuclei, the SNR in the polar caps peaks at
around $b_{\text{cut}}=40\,\mathrm{deg}$, corresponding to having about $35\%$
of the sky in the caps and $65\%$ around the Galactic plane. This SNR is
larger than the full-sky SNR with maximal deflections. In UHECRs are protons,
separating the sky in a low-deflection and a high-deflection region does not
bring about much improvement—this can be understood because heavy elements are
more affected by the magnetic deflections; nonetheless, when adding in
quadrature the SNR from the two regions, the split sky gives better results.
We will therefore use $b_{\text{cut}}=40\,\mathrm{deg}$ in what follows.

Figure 2: Total SNR for $\ell=[2,1000]$ as a function of the cut
$b_{\text{cut}}$ for a north-south-symmetric split for which the sky fraction
in the polar caps is $f_{\text{sky}}=1-\sin b_{\text{cut}}$ for H1 injection
(bottom panels), O16 injection (middle panels) and Si28 injection (top
panels). The left column shows the AC and the right column shows the XC
(optimised). The solid lines refer to the polar caps with $|b|\geq
b_{\text{cut}}=40\,\text{deg}$. The dashed lines refer to the complementary
region around the Galactic plane.

In Fig. 3 we show the expected per-$\ell$ SNR for the three injection models
H1, O16 and Si28. The most important feature is the shift of the sensitivity
between the AC and the XC, as the latter is more sensitive to smaller angular
scales, i.e. higher multipoles $\ell$. This is in line with the expectations
from our previous work [17], and it holds up for all the three injections
models we consider, as well as after applying the GMF smearing—which, however,
severely suppresses the anisotropy, especially at those small scales. We also
observe how, as expected from Figure 2, the difference between polar caps and
Galactic plane when $b_{\text{cut}}=40\,\mathrm{deg}$ is much more pronounced
fo Si28 and O16 than for H1, the latter being much less sensitive to the GMF.

Figure 3: $\text{SNR}_{\ell}$ for H1, O16, Si28 for a North-South symmetric
mask with $b_{\text{cut}}=40\,\deg$, for the same energy cuts as in Fig. 2.
The left column is the AC, the middle column is the XC (optimised) and the
right column is the AC+XC (optimised).

The total anisotropic power for both the AC and the XC (if optimal weights are
applied) is detectable for all injection models and energy cuts, as is shown
in Fig. 2. However, unless we use the _combined_ power of the AC and the XC,
individual multipoles are below the level of detection (which we quantify at
$3\,\sigma$, approximately $\text{SNR}_{\ell}\approx 3$). Such combined
analysis (rightmost panels, dubbed ‘AC+XC’) is worth being discussed for
another, peculiar feature. Looking at Fig. 3, we observe that the AC+XC shows
an overall enhancement of $\text{SNR}_{\ell}$ for all injection models and
energy cuts particularly on the largest angular scales, i.e. low multipoles,
even as the AC and XC separately may drop as $\ell\rightarrow 1$—notice the
different y-axes ranges in the rightmost panels compared to the AC and XC
alone. This is caused by a cosmic-variance mitigation effect that is known to
occur when multiple tracers of the cosmic large-scale structure are employed
simultaneously [40, see also 41 for a first application to harmonic-space
power spectra]. What happens is that, in order to analyse the AC and XC
jointly, we have to take into account the cross-covariance between the two
signals [see 17, Section 3.1]. It is important to notice that both signals
trace the same, underlying large-scale cosmic structure. Therefore, when
analysing both at the same time, the inverse covariance in the joint SNR
involves ratios of $C^{\rm CR\,CR}$ and $C^{\rm g\,CR}$, in which the common,
stochastic contribution from the matter density distribution effectively
cancels out. While cosmic-variance cancellation happens at all scales, this
effect is much more pronounced at low $\ell$ because cosmic variance scales as
$1/(2\ell+1)$ so it is larger (hence, it pollutes the signal more) at low
$\ell$; moreover, cosmic variance cancellation is effective when sample
variance dominates over shot noise, which is increasingly true as we move
towards low $\ell$. This face raises the exciting possibility that some low
multipoles, e.g. the quadrupole $\ell=2$, could be within reach of current
data already, if the combined AC and XC were used. We leave a full discussion
of this possibility to future work.

In order to assess how a future facility with access to larger exposure could
improve on this result, in Fig. 4 we compare the
$E_{\text{cut}}=100\,\mathrm{EeV}$ results of Fig. 3 with synthetic UHECR data
sets containing approximately three and ten times more events, namely
$N_{\text{ev}}=100$ and $N_{\text{ev}}=300$ events. We note how the AC
improves much faster than XC, as expected given the fact that the AC entirely
relies on $N_{\text{ev}}$ to trace the underlying distribution of sources (the
noise for the AC goes as $1/N_{\text{ev}}$), see Eqs. 2.17, LABEL: and 2.18.

Figure 4: $\text{SNR}_{\ell}$ for H1, O16, Si28 for a north-south-symmetric
mask with $b_{\text{cut}}=40\,\deg$, for $E_{\text{cut}}=100\,\mathrm{EeV}$
with $N_{\text{ev}}=30$ (blue, same as in Fig. 2), $N_{\text{ev}}=100$
(magenta) and $N_{\text{ev}}=300$ (green).

###  3.1 Catalogues

##### Incomplete sky coverage

Galaxy surveys are never able to observe the whole sky. In the simplest case,
a large region around the Galactic plane must be masked to avoid systematics
from dust extinction and star contamination. In addition to this, ground-based
observatories never have access to the whole celestial sphere, and observing
conditions also place constraints on the final survey footprint. From the
point of view of data analysis, a limited sky area has a number of
consequences. First of all, not all angular scales will be accessible, with
the minimum multipole being approximately $\ell_{\rm
min}\sim\pi/(2\,f_{\text{sky}})$. Secondly, a smaller sky area will impact how
noisy the signal will be. An accurate estimate of this effect depends on the
details of the sky mask, but for $f_{\text{sky}}\gtrsim\mathcal{O}(10^{-1})$
we can account for this by multiplying the (Gaussian) error bars on the power
spectrum by a factor $1/\sqrt{f_{\text{sky}}}$. In addition to this, an
incomplete sky coverage effectively corresponds to applying a binary mask
$v(\hat{\bm{n}})$ to the underlying target field $f(\hat{\bm{n}})$—binary in
the sense that $v$ is either $1$ if the corresponding pixel has been observed,
or $0$ if not. However, the simple product $v(\hat{\bm{n}})\,f(\hat{\bm{n}})$
in real space becomes a convolution in harmonic space. This fact has two major
consequences: $i)$ the naïve estimator for the harmonic-space power spectrum
becomes biassed and such a bias has to be corrected for; and $ii)$ different
$\ell$-modes become coupled because of the presence of the mask, and the data
covariance is no longer diagonal in $\ell$-$\ell^{\prime}$, even on linear
scales. Nonetheless, from the point of view of the results presented here,
this last point has no major impact [42], since we are dealing with very large
sky areas, for which the effects of the mask will be small.

Furthermore, it is worth emphasising that a limited sky area for a galaxy
survey will also impact the cross-correlation between galaxies and UHECRs,
since the areas of the sky observed by the two experiments will not in
principle be the same. As a matter of fact, some degree of correlation is
expected even in non-overlapping areas, since both galaxies and UHECR emitters
trace the same underlying cosmic large-scale structure. However, to be
conservative, we only consider the intersection of the sky patches observed by
the two experiments.

##### Flux limitations

The main impact of flux incompleteness in the galaxy catalogue is the
potential for some of the detected UHECRs being sourced by galaxies not
included in the catalogue. The quantitative impact of this on the results
presented here would be relatively mild as long as the galaxy catalogue is
able to cover the same physical volume over which most of the UHECR sources
are distributed. Although the cross-correlation noise bias would change (see
Eq. 2.15), the main noise contribution to the power spectrum covariance comes
from the noise in the auto-correlations, which would stay the same. On
sufficiently large scales (which are the only ones where the cross-correlation
can be measured effectively, anyway), the amplitude of the cross-correlation
is controlled by the spatial overlap between both tracers (i.e. their ability
to trace the same matter fluctuations), regardless of their overlap at the
level of individual sources. Thus, as a rough estimate, if, say, 2% of UHECR
sources are missing from the catalogue this will translate into a maximum of
2% loss of XC power, because of the 2% smaller galaxy kernel.

This picture would change if the UHECR sources were located at redshifts that
are significantly larger than those probed by the galaxy catalogue, but this
is unlikely given the redshift dependence of the attenuation factor for
typical cosmic rays, all the more so for UHECRs.

##### Other catalogues

To date, a rather large amount of galaxy surveys for cosmology has been
conducted, resulting into a likewise large availability of galaxy catalogues
to employ for cross-correlation studies. Moreover, future observational
campaigns plan to observe deeper (in both redshift and magnitude), meaning
that in the next future even more data will be available.

This notwithstanding, to our aim of learning about UHECRs via cross-
correlations with galaxy clustering, we are limited to a small volume around
the Earth, for the radial kernel of UHECRs decreases rapidly with increasing
redshift. This means that we need a catalogue with as many as possible local
galaxies, so that the two signals correlate (technically, this is so to
maximise the support of the integral in Eq. 2.13). For this reason, we have
focussed on 2MRS, which is the largest catalogue available at low redshift,
both in terms of number density and of sky coverage.

However, it is worth keeping in mind that galaxy surveys differ from one
another not only for what concerns redshift range, observed area, and total
number of detected objects, but also because they target different galaxy
populations. Such galaxy populations will have, for instance, different bias,
which may make the cross-correlation signal stronger. Moreover, their
astrophysical properties will also differ, making them better targets for
their rate of emission of UHECRs. Therefore, for an actual data analysis, it
would be interesting to compare UHECR cross-correlations among various
catalogues, as they may teach us something about the properties of UHECR
sources.

##  4 Conclusions and outlook

In this work we have studied the angular, harmonic cross-correlation between
the distribution of galaxies in the local Universe (our synthetic galaxy
catalogue peaks at redshift $z\approx 0.03$) and a full-sky set of UHECRs in
numbers comparable to what has been detected with current facilities. We have
considered three injection models (H1, O16, Si28) which bracket realistic fits
to the observed UHECR spectrum and composition, and three energy cuts at
$40\,\mathrm{EeV}$, $63\,\mathrm{EeV}$ and $100\,\mathrm{EeV}$. Lastly, we
have included the effects of intervening magnetic fields using a data-driven
toy-model for smearing.

We find that, with statistics comparable with current data, both the AC and XC
for individual multipoles will be rather weak and remain below the threshotd
for detection, mostly because of the magnetic deflections. However, if we
combine the AC and the XC in the most optimal statistics for the detection of
the anisotropy, we are able reach detection levels of $3\sigma$ and above in
the O16 and Si28 cases at nearly all energies considered here for very large
scales (low multipoles). This is thanks to a cancellation of cosmic variance
that arises when employing multiple tracers of the same underlying
distribution (in our case the large-scale structure). Cosmic variance
cancellation motivates further studies, aiming at improving on the searches
for anisotropy at low multipoles, e.g., the quadrupole $\ell=2$ (see [12]). In
some cases, a marginal detection at $3\sigma$ for individual, optimised XC
multipoles is within reach with ten times more events than currently
available, as would be the case for instance with the next generation of
detectors such as POEMMA or GRAND. Lastly, the total power
$C^{ab}\coloneqq\sum_{\ell}C^{ab}_{\ell}$ is detectable with SNRs well above 5
for the AC and XC (optimised) for both H1 and O16 at all energies, with peaks
of $\text{SNR}\approx 10$ for the XC for H1 at $100\,\mathrm{EeV}$ and
$\text{SNR}\approx 8$ for the AC and XC of O16 at most energies. Si28 is the
most impacted by the magnetic deflections, as expected, but still has
$\text{SNR}\approx 6$ for the AC and $\text{SNR}\approx 5$ for the optimised
XC (cutting the sky at $b_{\text{cut}}=40\deg$). We remark that all our
results, except for the simulated energy losses of propagating UHECRs, are
analytic; this enables us to quickly asses some properties of the harmonic
observables in different scenarios.

Our findings can serve as further motivation to search for the XC (and the
AC+XC) with existing data from the Telescope Array (located in the Northern
Hemisphere) and the Pierre Auger Observatory (located in the Southern
Hemisphere), following the lead of the AC analyses performed by the Auger-TA
joint working group on arrival directions [12]. Another application is the use
of the angular, harmonic power spectra to discriminate between different UHECR
injection models, in particular to help determine the UHECR composition, see
for example [22].

For the future we plan to move away from the analytic analysis we have kept to
thus far and perform fully-fledged propagation simulations of UHECRs through
realistic models of the GMF. This will enable us to interpret the possible
detection of harmonic correlations from the data. This is especially needed
for the promising AC+XC quadrupole, which we predict could already be
detectable with existing data.

## Acknowledgments

FU thanks Armando di Matteo for help with _SimProp_. For most of the
development of this project FU was supported by the European Regional
Development Fund (ESIF/ERDF) and the Czech Ministry of Education, Youth and
Sports (MEYS) through Project CoGraDS - `CZ.02.1.01/0.0/0.0/15_003/0000437`.
SC is supported by the Italian Ministry of Education, University and Research
(miur) through Rita Levi Montalcini project ‘prometheus – Probing and Relating
Observables with Multi-wavelength Experiments To Help Enlightening the
Universe’s Structure’, and by the ‘Departments of Excellence 2018-2022’ Grant
awarded by miur (L. 232/2016). DA acknowledges support from the Beecroft
Trust, and from the Science and Technology Facilities Council through an
Ernest Rutherford Fellowship, grant reference ST/P004474/1.

##  Appendix A Exposures

The exposures of the Telescope Array Collaboration (TA) and Pierre Auger
Observatory (PAO) experiments are strongly $b$-dependent. This means that
masking certain regions of the sky changes not only cosmic variance, as
quantified by $f_{\text{sky}}$, but also the density of UHECR events within
the unmasked region $\bar{N}_{\varOmega,{\rm
CR}}\rightarrow\bar{N}_{\varOmega,{\rm CR}}(b_{\text{cut}})\coloneqq N_{\rm
CR}/[4\,\pi\,f_{\text{sky}}(b_{\text{cut}})]$, and with it the noise.

Exposures above 10 EeV are purely geometric and, save for very subdominant
sidereal effects, only depend on declination $\delta$ as

$\displaystyle\omega$
$\displaystyle\propto\cos\lambda\,\cos\delta\,\sin\alpha_{m}+\alpha_{m}\,\sin\lambda\,\sin\delta\;,$
(A.1)

where $\lambda$ is the latitude of the detector (for TA $\lambda=39.2\,\deg$
and for Auger $\lambda=-35.2\,\deg$), and the parameter $\alpha_{m}$ is

$\displaystyle\alpha_{m}$ $\displaystyle\coloneqq\begin{cases}0&\xi>1,\\\
\pi&\xi<-1,\\\ \arccos\xi&\,\mathrm{otherwise}\end{cases}\;,$ (A.2)

with

$\displaystyle\xi$
$\displaystyle\coloneqq\frac{\cos\theta_{z}-\sin\lambda\,\sin\delta}{\cos\lambda\,\cos\delta}\;,$
(A.3)

and $\theta_{z}$ the maximum zenith angle (we use $\theta_{z}=55\,\deg$ for TA
and $\theta_{z}=60\,\deg$ for Auger) [18]. The exposure in Galactic
coordinates $(l,b)$ can be computed thanks to
$\sin\delta=\sin\delta_{\text{NGP}}\,\sin{b}+\cos\delta_{\text{NGP}}\,\cos{b}\,\cos(l_{\text{NCP}}-l)$
where the north Galactic pole has $\delta_{\text{NGP}}=27.13\,\deg$ and the
north celestial pole has $l_{\text{NCP}}=122.9\,\deg$ in the J2000 system—here
we can also use $\cos\delta=\sqrt{1-\sin^{2}\delta}$ because
$\delta\in[-\pi/2,\pi/2]$. Because the GMF deflections Eq. 2.22 do not depend
on $l$ we average over it:
$\omega(b)\coloneqq\int\mathrm{d}l\,\omega(b,l)/(2\,\pi)$. In Fig. A.1, left
panel, we show the exposures of TA and Auger normalised such that
$\int\mathrm{d}b\sin{b}\,\omega(b)=1$. In this approximation the exposure
$\omega(b)$ rises nearly linearly with $b$; thence, if we apply a north-south-
symmetric mask the density $\bar{N}_{\varOmega,{\rm CR}}(b_{\text{cut}})$ is
nearly independent of $b_{\text{cut}}$ (the increase in density in one
hemisphere being compensated by the loss in the opposite hemisphere) and, in
practice, we recover the case of uniform exposure.

Figure A.1: Left y-axis (blue colour): normalised exposures $\omega(b)$
averaged over $l$ for TA (solid) and PAO (dashed). Right y-axis (orange
colour): $\bar{N}_{\varOmega,{\rm CR}}(b_{\text{cut}})$, normalised to 1 for a
full sky, when only the north, resp. south, polar cap beyond
$|b_{\text{cut}}|$ are unmasked.

An alternative approach would be to cut the sky from $b=-\pi/2$ to
$b=b_{\text{cut}}$ and keep everything north of that for TA (or any north-
located experiment), and cut the sky from $b=\pi/2$ to $b=b_{\text{cut}}$ and
keep everything south of that for Auger (or any south-located experiment). We
show the UHECR event densities with this choice of cuts in Fig. A.1, right
panel. In this case the expression for the deflection $\sigma(b)$ is

$\displaystyle\sigma=\frac{1}{\sqrt{2}}\left(\frac{40\,\text{EV}}{E/Z}\right)\frac{1\,\text{deg}}{\varTheta(\pm
b)\,\sin^{2}b+0.15}\;,$ (A.4)

where $\varTheta$ is the Heaviside unit step function and the sign $+$ is for
TA and the $-$ is for Auger. We have verified that for all cases under
consideration (H1, O16, Si28) it is best to keep the full sky. If the magnetic
deflections are larger, for example when we consider heavier primaries, lower
energies, or stronger magnetic fields, it might however be advantageous to
keep only the north (south) polar cap for TA (Auger).

##  Appendix B The galaxy noise power spectrum

Let us start by considering the measurement of galaxy number count
fluctuations in real space and focus on a purely Poisson process. The
(weighed) number of galaxies at angular position $\hat{\bm{n}}$ is

$N_{\rm g}^{w}(\hat{\bm{n}})=\int\mathrm{d}\chi\,N_{\rm
g}(\hat{\bm{n}},\chi)\,w(\chi)\;,$ (B.1)

where $N_{\rm g}(\hat{\bm{n}},\chi)\,\mathrm{d}\chi$ is the number of galaxies
in direction $\hat{\bm{n}}$ in the radial-comoving distance interval
$[\chi,\chi+\mathrm{d}\chi]$, and $w(\chi)$ is an arbitrary weight assigned to
it ($w\equiv 1$ being the unweighed case). Note that the second equality comes
from the definition of $\varDelta_{\rm g}$, Eq. 2.7. Then, we have

$\displaystyle\left\langle\left[N_{\rm
g}^{w}(\hat{\bm{n}})\right]^{2}\right\rangle$
$\displaystyle=\int\mathrm{d}\chi\,\int\mathrm{d}\chi^{\prime}\,w(\chi)\,w(\chi^{\prime})\,\left\langle
N_{\rm g}(\hat{\bm{n}},\chi)\,N_{\rm
g}(\hat{\bm{n}},\chi^{\prime})\right\rangle$
$\displaystyle=\int\mathrm{d}\chi\,\int\mathrm{d}\chi^{\prime}\,w(\chi)\,w(\chi^{\prime})\,\left[\left\langle
N_{\rm g}(\hat{\bm{n}},\chi)\right\rangle\,\langle N_{\rm
g}(\hat{\bm{n}},\chi^{\prime})\rangle+\left\langle N_{\rm
g}(\hat{\bm{n}},\chi)\right\rangle\,\delta^{\rm(D)}(\chi-\chi^{\prime})\right]$
$\displaystyle=\left[\int\mathrm{d}\chi\,w(\chi)\,\bar{N}_{\rm
g}(\chi)\right]^{2}+\int\mathrm{d}\chi\,w^{2}(\chi)\,\bar{N}_{\rm g}(\chi)\;,$
(B.2)

with $\delta^{\rm(D)}$ denoting the Dirac distribution and $\bar{N}_{\rm
g}(\chi)$ being the mean number of galaxies per unit comoving distance in
$[\chi,\chi+\mathrm{d}\chi]$. In the second line we have used the fact that:
i) number counts are Poisson processes, for which the variance is equal to the
mean; and ii) there is no correlation unless the radial position is the same.
Thence,

${\rm Var}\left[N_{\rm
g}^{w}(\hat{\bm{n}})\right]=\int\mathrm{d}\chi\,w^{2}(\chi)\,\bar{N}_{\rm
g}(\chi)\;.$ (B.3)

Now, $\varDelta_{\rm g}$ is defined in Eq. 2.7 as the number density contrast,
i.e. the fluctuations in number counts relative to the mean. Therefore, by
dividing Eq. B.3 by Eq. B.1 squared and taking the average over all possible
directions in the sky, we eventually obtain the Poissonian shot noise for the
harmonic-space power spectrum of a weighed galaxy distribution, viz. Eq. 2.16.

## References

  * [1] J. Linsley, L. Scarsi and B. Rossi, _Extremely energetic cosmic-ray event_ , _Phys. Rev. Lett._ 6 (1961) 485.
  * [2] R. Alves Batista et al., _Open Questions in Cosmic-Ray Research at Ultrahigh Energies_ , _Front. Astron. Space Sci._ 6 (2019) 23 [1903.06714].
  * [3] Telescope Array collaboration, _Indications of a Cosmic Ray Source in the Perseus-Pisces Supercluster_ , 2110.14827.
  * [4] Telescope Array, Pierre Auger collaboration, _UHECR arrival directions in the latest data from the original Auger and TA surface detectors and nearby galaxies_ , _PoS_ ICRC2021 (2021) 308 [2111.12366].
  * [5] Telescope Array, Pierre Auger collaboration, _The UHECR dipole and quadrupole in the latest data from the original Auger and TA surface detectors_ , _PoS_ ICRC2021 (2021) 375 [2111.14593].
  * [6] Telescope Array collaboration, _Hotspot Update, and a new Excess of Events on the Sky Seen by the Telescope Array Experiment_ , _PoS_ ICRC2021 (2021) 328.
  * [7] Telescope Array collaboration, _Telescope Array anisotropy summary_ , _PoS_ ICRC2021 (2021) 392.
  * [8] Telescope Array collaboration, _UHECR mass composition from anisotropy of their arrival directions with the Telescope Array SD_ , _PoS_ ICRC2021 (2021) 294.
  * [9] Pierre Auger collaboration, _Large-scale and multipolar anisotropies of cosmic rays detected at the Pierre Auger Observatory with energies above 4 EeV_ , _PoS_ ICRC2021 (2021) 335.
  * [10] Pierre Auger collaboration, _The ultra-high-energy cosmic-ray sky above 32 EeV viewed from the Pierre Auger Observatory_ , _PoS_ ICRC2021 (2021) 307.
  * [11] Pierre Auger collaboration, _Indication of a mass-dependent anisotropy above $10^{18.7}\,$eV in the hybrid data of the Pierre Auger Observatory_, _PoS_ ICRC2021 (2021) 321.
  * [12] Telescope Array, Pierre Auger collaboration, _2022 report from the Auger-TA working group on UHECR arrival directions_ , _EPJ Web Conf._ 283 (2023) 03002 [2302.04502].
  * [13] Telescope Array collaboration, _Cosmic-ray mass composition with the TA SD 12-year data_ , _PoS_ ICRC2021 (2021) 300.
  * [14] Pierre Auger collaboration, _Combined fit of the energy spectrum and mass composition across the ankle with the data measured at the Pierre Auger Observatory_ , _PoS_ ICRC2021 (2021) 311.
  * [15] F. Boulanger et al., _IMAGINE: A comprehensive view of the interstellar medium, Galactic magnetic fields and cosmic rays_ , _JCAP_ 1808 (2018) 049 [1805.02496].
  * [16] Telescope Array, Pierre Auger collaboration, _Joint analysis of the energy spectrum of ultra-high-energy cosmic rays as measured at the Pierre Auger Observatory and the Telescope Array_ , _PoS_ ICRC2021 (2021) 337.
  * [17] F. R. Urban, S. Camera and D. Alonso, _Detecting ultra-high-energy cosmic ray anisotropies through harmonic cross-correlations_ , _Astron. Astrophys._ 652 (2021) A41 [2005.00244].
  * [18] P. Sommers, _Cosmic ray anisotropy analysis with a full-sky observatory_ , _Astropart. Phys._ 14 (2001) 271 [astro-ph/0004016].
  * [19] P. G. Tinyakov and F. R. Urban, _Full sky harmonic analysis hints at large ultra-high energy cosmic ray deflections_ , _J. Exp. Theor. Phys._ 120 (2015) 533 [1411.2486].
  * [20] O. Deligny, _Large-Scale Distribution of Arrival Directions of Cosmic Rays Detected at the Pierre Auger Observatory and the Telescope Array (ICRC2015)_ , _PoS_ ICRC2015 (2015) 395.
  * [21] A. di Matteo, O. Deligny, K. Kawata, R. M. de Almeida, M. Mostafá, E. Moura Santos et al., _Arrival Directions of Cosmic Rays at Ultra-High Energies_ , _JPS Conf. Proc._ 19 (2018) 011020.
  * [22] K. Tanidis, F. R. Urban and S. Camera, _Constraining ultra-high-energy cosmic ray composition through cross-correlations_ , 2203.09538.
  * [23] K. Tanidis, F. R. Urban and S. Camera, _Honing cross-correlation tools for inference on ultra-high-energy cosmic-ray composition_ , 2310.17699.
  * [24] A. di Matteo and P. Tinyakov, _How isotropic can the UHECR flux be?_ , _Mon. Not. Roy. Astron. Soc._ 476 (2018) 715 [1706.02534].
  * [25] M. S. Pshirkov, P. G. Tinyakov and F. R. Urban, _Mapping UHECRs deflections through the turbulent galactic magnetic field with the latest RM data_ , _Mon. Not. Roy. Astron. Soc._ 436 (2013) 2326 [1304.3217].
  * [26] E. Waxman, K. B. Fisher and T. Piran, _The Signature of a correlation between $>$ 10**19-eV cosmic ray sources and large scale structure_, _Astrophys. J._ 483 (1997) 1 [astro-ph/9604005].
  * [27] H. B. J. Koers and P. Tinyakov, _Testing large-scale (an)isotropy of ultra-high energy cosmic rays_ , _JCAP_ 0904 (2009) 003 [0812.0860].
  * [28] N. Globus, A. Fedynitch and R. D. Blandford, _Treasure Maps for Detections of Extreme Energy Cosmic Rays_ , 2210.15885.
  * [29] R. Aloisio, D. Boncioli, A. Di Matteo, A. F. Grillo, S. Petrera and F. Salamida, _SimProp v2r4: Monte Carlo simulation code for UHECR propagation_ , _JCAP_ 1711 (2017) 009 [1705.03729].
  * [30] F. W. Stecker, M. A. Malkan and S. T. Scully, _Intergalactic photon spectra from the far ir to the uv lyman limit for 0 < Z < 6 and the optical depth of the universe to high energy gamma-rays_, _Astrophys. J._ 648 (2006) 774 [astro-ph/0510449].
  * [31] Telescope Array collaboration, _Recent measurement of the Telescope Array energy spectrum and observation of the shoulder feature in the Northern Hemisphere_ , _PoS_ ICRC2021 (2021) 341.
  * [32] Pierre Auger collaboration, _Energy spectrum of cosmic rays measured using the Pierre Auger Observatory_ , _PoS_ ICRC2021 (2021) 324.
  * [33] J. P. Huchra et al., _The 2MASS Redshift Survey - Description and Data Release_ , _Astrophys. J. Suppl._ 199 (2012) 26 [1108.0669].
  * [34] S. Ando, A. Benoit-Lévy and E. Komatsu, _Angular power spectrum of galaxies in the 2MASS Redshift Survey_ , _Mon. Not. Roy. Astron. Soc._ 473 (2018) 4318 [1706.05422].
  * [35] J. A. Peacock and R. E. Smith, _Halo occupation numbers and galaxy bias_ , _Mon. Not. Roy. Astron. Soc._ 318 (2000) 1144 [astro-ph/0005010].
  * [36] M. Haverkorn, _Magnetic Fields in the Milky Way_ , 1406.0283.
  * [37] M. Unger and G. R. Farrar, _Uncertainties in the Magnetic Field of the Milky Way_ , vol. 35, p. 558, 2017, 1707.02339.
  * [38] V. Shaw, A. van Vliet and A. M. Taylor, _Galactic halo magnetic fields and UHECR deflections_ , 2202.06780.
  * [39] D. Harari, S. Mollerach, E. Roulet and F. Sanchez, _Lensing of ultrahigh-energy cosmic rays in turbulent magnetic fields_ , _JHEP_ 03 (2002) 045 [astro-ph/0202362].
  * [40] U. Seljak, _Extracting Primordial Non-Gaussianity without Cosmic Variance_ , _Phys. Rev. Lett._ 102 (2009) 021302 [0807.1770].
  * [41] L. D. Ferramacho, M. G. Santos, M. J. Jarvis and S. Camera, _Radio galaxy populations and the multitracer technique: pushing the limits on primordial non-Gaussianity_ , _MNRAS_ 442 (2014) 2511 [1402.2290].
  * [42] M. Crocce, A. Cabré and E. Gaztañaga, _Modelling the angular correlation function and its full covariance in photometric galaxy surveys_ , _MNRAS_ 414 (2011) 329 [1004.4640].


# ReProbe: An Architecture for Reconfigurable
and Adaptive Probes ††thanks:  This work has been partially supported by the
Centro Nazionale HPC, Big Data e Quantum Computing (PNRR CN1 spoke 9 Digital
Society & Smart Cities); the Engineered MachinE Learning-intensive IoT systems
(EMELIOT) national research project, which has been funded by the MUR under
the PRIN 2020 program (Contract 2020W3A5FY); and the COmmunity-Based Organized
Littering (COBOL) national research project, which has been funded by the MUR
under the PRIN 2022 PNRR program (Contract P20224K9EK).

Federico Alessi1, Alessandro Tundo21, Marco Mobilio1, Oliviero Riganelli1,
Leonardo Mariani1
1University of Milano-Bicocca
Milan, Italy
<EMAIL_ADDRESS><EMAIL_ADDRESS>
2Vienna University of Technology
Vienna, Austria
<EMAIL_ADDRESS>
###### Abstract

Modern distributed systems are highly dynamic and scalable, requiring
monitoring solutions that can adapt to rapid changes. Monitoring systems that
rely on external probes can only achieve adaptation through expensive
operations such as deployment, undeployment, and reconfiguration.

This poster paper introduces ReProbes, a class of adaptive monitoring probes
that can handle rapid changes in data collection strategies. ReProbe offers
controllable and configurable self-adaptive capabilities for data
transmission, collection, and analysis methods. The resulting architecture can
effectively enhance probe adaptability when qualitatively compared to state-
of-the-art monitoring solutions.

###### Index Terms:

reconfigurable probe, adaptive probe, adaptive monitoring

##  I Introduction

The growing usage of microservice-based applications, Systems-of-Systems
(SoS), and Internet-of-Things (IoT) services make effective and highly-dynamic
monitoring of paramount importance to ensure seamless operations in the face
of growing complexity [1]. Despite applications can be enhanced with
monitoring capabilities through code instrumentation techniques (e.g., Kieker
Framework [2], OpenTelemetry [3], Prometheus client libraries [4]), this is
not always possible as in the case of legacy or third-party systems, where
monitoring activities have to be performed by employing external probes111A
probe is a software component responsible for collecting the raw data, such as
sampling the CPU consumption of a service or recording the temperature in a
room from a sensor..

State-of-the-art monitoring systems using external probes can be adapted to
changes by only performing _expensive_ probe-level operations, such as
redeploying or reconfiguring probes, to accommodate any change in the data
collection strategies [5]. For instance, changing the set of collected
indicators or changing the sampling rate of data collection using Prometheus
[6] and its exporters [7] requires: (i) updating the Prometheus configuration
to change the sampling rate (i.e., scraping interval), (ii) updating the
exporter configuration to change the collected indicators, and (iii) at least
restarting the exporter when Prometheus is properly configured for hot reload
via HTTP API request.

This poster paper _presents a hierarchical architecture that can be used to
design reconfigurable and adaptive monitoring probes_ that can _quickly_
change (i) configuration settings (i.e., the sampling rate, the observed
target, and the collected indicators), (ii) the ingestion services, (iii) the
data collection logic, and (iv) the data analysis logic, without requiring any
expensive probe redeployment. The class of adaptive probes presented in this
paper, which we called _ReProbe_ s, can augment probing systems with efficient
_probe-level self-adaptive behaviors_ , that is, adaptive behaviors local to
the components responsible for data collection, in contrast to solutions
designed to facilitate probe (un)deployment [5]. The proposed architecture
exploits the plug-in architectural pattern [8, 9] to achieve the desired level
of flexibility. Our initial qualitative evaluation shows that ReProbe can
effectively cope with dynamic monitoring scenarios by offering flexible
adaptivity.

##  II ReProbe

Figure 1 shows the main modules of ReProbe, namely the _Publishers Manager_ ,
its collection of Publishers (Pub), the _Collectors Manager_ , its collection
of Collectors (Col), and the _Data Manager_. Note that ReProbe defines the
architecture of individual self-adaptive probes, which can be used in the
context of any monitoring system. In fact, ReProbe does not impose any
restriction on the type of collected data, which can be transmitted through
any platform (e.g., Apache Kafka streams [10]) and stored in any time series
database (e.g., Elasticsearch [11]). ReProbe can be used to define probes that
embed self-adaptive behaviors that influence _what_ and _how_ data is
collected, and then transmitted outside the probe.

Figure 1: ReProbe architecture.

The Collectors Manager and Publishers Manager are the components responsible
for handling the life-cycle of the plugin available in the collection of
collectors and publishers, respectively. When an authorized client requests
changes through the API interface, the managers add and remove plugins from
the collections. Moreover, they also handle the run-time operation of the
plugin, namely _instantiating_ (i.e., starting a plugin), _destroying_ (i.e.,
eliminating a previously created instance of a plugin), and _reconfiguring_
(i.e., update the configuration of a running plugin) them.

A Col in the set of the collectors has the responsibility to collect data from
the monitored resource and publish the collected data to the Data Manager. The
architecture supports the presence of multiple Cols since each Col might be in
charge of collecting a different set of indicators. For instance, a Col can be
responsible for the collection of CPU utilization indicators (e.g., idle, I/O
wait, user, system, and total percentages) while another Col can be
responsible for the collection of network-related indicators (e.g., I/O
received packets, I/O dropped packets, and I/O bytes). Conversely, a publisher
has the responsibility of reading data from the Data Manager and publishing
the data to ingestion services. Again, the architecture supports the presence
of multiple publishers able to deal with different ingestion services.

Multiple collectors and publishers can be simultaneously active. The data
collected by the running collectors is forwarded to the data manager and
consumed by the active publishers that send the data to specific external
services. For example, let us consider an instance of ReProbe configured to
monitor a load balancer (e.g., HAProxy). It may collect system-level
indicators by means of two Cols (e.g., CPU utilization and memory
consumption), and application-level indicators by means of another Col (e.g.,
HAProxy performance statistics). Simultaneously, all the collected
observations are sent to Elasticsearch [11] for offline processing. At the
same time, application-level indicators are also sent to Kafka streams [10]
for an online processing pipeline used to tune HAProxy configuration.

Any external actor (e.g., operator or other services) can interact with
ReProbe through the API to _upload_ new plugin, and to control the life-cycle
of the uploaded plugins.

Figure 1 shows how ReProbe can reconfigure itself, responding to requests
received by the API, through the red dashed lines, which represent the control
flow, and the black lines, which represent the data-flow links. Note that the
adaptations triggered through the API are purely _reactive_ , since the
trigger is external to the probe.

The Collector is the component responsible for collecting a given set of
indicators from a target. In the ReProbe architecture, it embeds self-adaptive
behaviors that can be used to proactively adjust the monitoring strategy
according to observations. Figure 2 shows the architecture of a Collector,
which includes a Controller, a set of Metric Samplers (MSs), a set of Data
Analyzers (DAs), and a _configuration_.

Figure 2: Collector architecture.

The Metric Sampler is the component responsible for gathering a given set of
indicators from the monitored resource. For instance, it can be the component
that reads the CPU or the memory utilization according to a given sampling
rate. Each MS inside the plugin encodes a slightly different way of sampling
the same data (e.g., sampling the bandwidth measurements based on the recent
indicator values to increase accuracy or reduce the overhead).

The Data Anlyzer receives the sampled observations from the active MS. It is
responsible for any analysis and manipulation (e.g., filtering data, changing
its format, and aggregating data) before sending the processed data outside
the boundary of the plugin to the Data Manager.

Only a single MS and DA can be active at any given time.

The Configuration stores the information needed by a collector to operate,
such as the parameters that influence the behavior of the running DA and MS,
as well as the configuration of the target resource.

The Controller is the component in charge of concretely updating the
configurations and triggers any reconfiguration of the plugin, if needed.

In addition to processing the collected data, DAs also include the logic to
trigger the self-adaptation through the reconfiguration of the metric
samplers. For example, if the active DA reports that a specific indicator is
stable, it may ask the Controller to update the configuration and reduce the
sampling rate of the active MS to save resources (e.g. less overhead on the
monitored target or less network traffic). On the other hand, if a given
metric is unstable, the same DA may increase the sampling rate. The presence
of multiple DAs and multiple MSs allows to collect the observations in
different ways and self-adapt the monitoring strategy according to different
policies.

Note how each Col is a self-adaptive module itself, running two flows. The
first flow is represented by the data gathered from the target by the MS,
which flows through the DA and finally outside to the Data Manager of ReProbe.
The second flow is the self-adaptive loop triggered by the DA through the
Controller every time active MS needs to be reconfigured.

##  III Qualitative Evaluation

To assess the proposed solution, we qualitatively compared ReProbe to both
research and commercial monitoring solutions along two dimensions, namely,
_features_ and _usage scenarios_. The feature-based evaluation compares the
adaptation capabilities of the probes available in the considered solutions.
The scenario-based evaluation assesses the effectiveness of the probes in the
context of two realistic monitoring scenarios.

For the comparison, we selected six representative monitoring solutions for
the cloud: two research monitoring tools with adaptive behaviors, JCatascopia
[12] and Adaptive FogMon [13]; and four popular commercial tools [14],
Prometheus [6], Datadog [15], Zabbix [16], and ElasticStack [17].

###  III-A Feature-Based Comparison

TABLE I: Feature-Based Comparison

Feature | Probe Technologies
---|---
ReProbe |  | JCatascopia
---
Agent [12]
| Adaptive FogMon
---
Follower [13]
| ElasticStack
---
Beats [18]
| Prometheus
---
Exporter [7]
| Datadog
---
Agent [15]
| Zabbix
---
Agent [16]
Zero-downtime deployment of new collection logic | yes | yes | no | no | no | no | yes
Zero-downtime deployment of new data publishing logic | yes | no | no | no | no | no | no
API-enabled configuration | yes | yes | no | no | no | yes | yes
Self-adaptive collection logic | yes | no | partially | no | no | no | no
Within-probe data analysis | yes | yes | yes | no | no | no | no
Supporting multiple data ingestion services | yes | no | no | yes | no | no | no
Simultaneous data publishing | yes | no | no | no | no | no | no

We identified seven main adaptive features that are desirable for self-
adaptive probes (as summarized in Table I). Two of these features concern with
the deployment of new behaviors, three concern with run-time adaptation, and
two concern with the flexibility of the probe. Regarding the extensibility of
the probes, we have _zero-downtime deployment of new collection logic_ and
_zero-downtime deployment of new data publishing logic_ features that concern
with the capability of adding new ways of collecting data (e.g., to support
new indicators or to run new analyses on the collected indicators) and
publishing data (e.g., to publish data on a new, previously unsupported,
external service). Regarding changes to run-time behaviors, _API-enable
configuration_ is the possibility to reconfigure a probe from the outside
through an API; _self-adaptive collection logic_ is the capability to self-
adapt the configuration of the data collection logic (e.g., to adaptively
modify the sampling rate or logic); and _within-probe data analysis_ is the
capability to locally analyze data and immediately react to any detected
pattern (e.g., changing the data collection logic or reporting an anomaly).
Regarding the flexibility of the probe, _supporting multiple data ingestion
services_ concerns with the capability of a probe to publish data on many
external services, and _simultaneous data publishing_ concerns with the
capability of a probe to publish the data on multiple services simultaneously.

ReProbe is the only class of probes supporting all these capabilities.
Deployment of additional code is only supported by JCatascopia agents and
Zabbix agents, but in both cases extending the set of the supported publishers
is not possible.

Runtime adaptation of the behaviors is very unique to ReProbe. While few other
solutions support reconfiguration through APIs (JCatascopia agent, Datadog
agent, and Zabbix agent) and within-probe analysis (JCatascopia agent and
Adaptive Fogmon Follower), none of the other approaches provide self-adaptive
behaviors. The only exception is Adaptive Fogmon Followers which can be
configured to self-adapt the sampling rate of an indicator or to modify the
set of collected indicators according to an internal analysis of the trends.
Finally, while some approaches can flexibly support multiple data ingestion
services, none of the other probing systems can simultaneously interact with
multiple data ingestion services.

###  III-B Scenario-Based Comparison

TABLE II: Scenario-Based Comparison

Scenario | Monitoring Solutions
---|---
ReProbe | JCatascopia [12] | Adaptive Fogmon [13] | ElasticStack [17] | Prometheus [6] | Datadog [15] | Zabbix [16]
| Monitoring a new data center spun up
---
to deliver services to a new user base
API-driven | API-driven | manual | manual | manual |  | partially
---
API-driven
| partially
---
API-driven
| Adapting the sampling logic
---
in response to indicator trends
self-adaptive |  | partially
---
self-adaptive
| partially
---
self-adaptive
manual | manual |  | partially
---
API-driven
| partially
---
API-driven

We discuss two realistic monitoring scenarios that capture key needs in
service and infrastructure monitoring, providing further evidence of the need
for self-adaptiveness.

#### Monitoring a new data center spun up to deliver services to a new user
base

The spin-up of a new data center requires monitoring the satisfaction of a set
of Service Level Objectives (SLOs) to verify the correct rollout of the
system, which in turn may require both reconfiguring the monitoring system to
collect new indicators and updating the sampling rate of the already collected
indicators on other data centers.

#### Adapting the sampling logic in response to indicator trends

Due to the need to promptly detect anomalies to avoid potential system
failures or SLA violations, it is required to dynamically adapt both the
collection and sampling logic of an indicator according to the analysis of its
trend.

Table II summarizes how each scenario is supported by the monitoring solutions
with the following labels: manual, the scenario can be completed by doing
manual operations; partially API-driven, the scenario can be partially
completed using an API, but some operations still require manual intervention;
API-driven, the scenario can be entirely completed by interacting with APIs;
partially self-adaptive, the scenario can be addressed with self-adaptive
behaviors, but some API interactions or manual operations are still needed;
self-adaptive, the scenarios can be completed using self-adaptive behaviors.

The solution providing the highest level of automation is ReProbe, which
provides probes that can be entirely reconfigured through APIs or configured
with self-adaptive behaviors. The closest solution in terms of automation
based on API is Zabbix, which provides a full spectrum of API-driven
automation, but does not provide self-adaptive capabilities. Some self-
adaptive capabilities are present in JCatascopia and Adaptive Fogmon. However,
self-adaption is limited to some specific aspects, such as specific settings
(e.g., sampling rate) or part of the workflow (e.g., adaptive data analysis
only), and API-driven reconfiguration is partially or not present at all. The
rest of the solutions provide limited reconfiguration capabilities and no
self-adaptation capabilities.

##  IV Related Work

Adaptive monitoring is an approach investigated in many domains to respond to
continuously changing conditions [19]. For example, the probe framework [20]
uses adaptive probes for the dynamic creation of custom detectors, allowing
their run-time insertion and removal. The Netconf-based approach [21]
leverages adaptive monitoring probes for anomaly and attack detection
scenarios. JCatascopia [12] introduces adaptive probes to enable dynamic
adjustments based on the detected workload variance. Tundo et al. [5] proposed
a framework for automated probe life-cycle management within the context of
Monitoring-as-a-Service for cloud systems. A multitude of studies, including
AdapPF [22] and the approach by Mertz and Nunes [23], delve into adaptive
strategies for sampling metrics in dynamic environments. In contrast to these
works, ReProbe introduces an architecture for the design of probe-level self-
adaptive behaviors.

##  V Conclusions

The evolving nature of distributed systems demands for reconfigurable and
adaptive monitoring solutions capable of swiftly responding to dynamic changes
in both the requirements and the observed system. This paper introduces
_ReProbe_ s, an architecture for reconfigurable and adaptive probes with
probe-level self-adaptive behaviors. Unlike traditional systems requiring
probe deployment and undeployment, ReProbe offers flexibility without
disruption.

ReProbe stands as a significant, although preliminary, step towards efficient
and responsive monitoring in the era of complex distributed architectures. Our
future work concerns with the quantitative experimentation of the proposed
approach, beyond the qualitative evaluation presented in this paper.

## References

  * [1] A. Noor, D. N. Jha, K. Mitra, P. P. Jayaraman, A. Souza, R. Ranjan, and S. Dustdar, “A framework for monitoring microservice-oriented cloud applications in heterogeneous virtualization environments,” in _international conference on cloud computing (CLOUD)_ , 2019.
  * [2] W. Hasselbring and A. van Hoorn, “Kieker: A monitoring framework for software engineering research,” _Software Impacts_ , vol. 5, Jun. 2020.
  * [3] The OpenTelemetry Authors . (2024) Openteletry. [Online]. Available: https://opentelemetry.io/
  * [4] Prometheus Authors. (2024) Client libraries. [Online]. Available: https://prometheus.io/docs/instrumenting/clientlibs/
  * [5] A. Tundo, M. Mobilio, O. Riganelli, and L. Mariani, “Automated probe life-cycle management for monitoring-as-a-service,” _IEEE Transactions on Services Computing_ , vol. 16, no. 2, pp. 969–982, 2023.
  * [6] Prometheus Authors. (2023) Prometheus. [Online]. Available: https://prometheus.io
  * [7] ——. (2023) Exporters and integrations. [Online]. Available: https://prometheus.io/docs/instrumenting/exporters/
  * [8] D. Birsan, “On plug-ins and extensible architectures: Extensible application architectures such as eclipse offer many advantages, but one must be careful to avoid “plug-in hell”,” _Queue_ , vol. 3, no. 2, 2005.
  * [9] M. Richards, _Software architecture patterns_. O’Reilly Media, Inc., 2015, ch. 3.
  * [10] The Apache Software Foundation. (2023) Kafka streams. [Online]. Available: https://kafka.apache.org/documentation/streams/
  * [11] Elasticsearch B.V. (2023) Elasticsearch. [Online]. Available: https://www.elastic.co/elasticsearch
  * [12] D. Trihinas, G. Pallis, and M. D. Dikaiakos, “Jcatascopia: Monitoring elastically adaptive applications in the cloud,” in _International Symposium on Cluster, Cloud and Grid Computing_ , 2014, pp. 226–235.
  * [13] V. Colombo, A. Tundo, M. Ciavotta, and L. Mariani, “Towards self-adaptive peer-to-peer monitoring for fog environments,” in _Proceedings of the 17th Symposium on Software Engineering for Adaptive and Self-Managing Systems_ , 2022, p. 156–166.
  * [14] Y. Verginadis, “A review of monitoring probes for cloud computing continuum,” in _International Conference on Advanced Information Networking and Applications_. Springer, 2023, pp. 631–643.
  * [15] Datadog. (2023) Datadog. [Online]. Available: https://www.datadoghq.com
  * [16] Zabbix LLC. (2023) Zabbix. [Online]. Available: https://www.zabbix.com
  * [17] Elasticsearch B.V. (2023) Elasticstack. [Online]. Available: https://www.elastic.co/elastic-stack
  * [18] ——. (2023) Beats: Data shippers for elasticsearch. [Online]. Available: https://www.elastic.co/beats
  * [19] E. Zavala, X. Franch, and J. Marco, “Adaptive monitoring: A systematic mapping,” _Information and software technology_ , vol. 105, 2019.
  * [20] Z. J. Estrada, C. Pham, F. Deng, L. Yan, Z. Kalbarczyk, and R. K. Iyer, “Dynamic vm dependability monitoring using hypervisor probes,” in _European Dependable Computing Conference (EDCC)_ , 2015.
  * [21] G. Munz, A. Antony, F. Dressler, and G. Carle, “Using netconf for configuring monitoring probes,” in _IEEE/IFIP Network Operations and Management Symposium NOMS 2006_ , 2006, pp. 1–4.
  * [22] C. Huang and G. Pierre, “Adappf: Self-adaptive scrape interval for monitoring in geo-distributed cluster federations,” in _IEEE Symposium on Computers and Communications (ISCC)_ , 2023, pp. 417–423.
  * [23] J. Mertz and I. Nunes, “Software runtime monitoring with adaptive sampling rate to collect representative samples of execution traces,” _Journal of Systems and Software_ , p. 111708, 2023.