# Calibration of the underground muon detector of the Pierre Auger Observatory

The Pierre Auger Collaboration

###### Abstract

To obtain direct measurements of the muon content of extensive air showers
with energy above $10^{16.5}$ eV, the Pierre Auger Observatory is currently
being equipped with an underground muon detector (UMD), consisting of 219 10
$\mathrm{m^{2}}$-modules, each segmented into 64 scintillators coupled to
silicon photomultipliers (SiPMs). Direct access to the shower muon content
allows for the study of both of the composition of primary cosmic rays and of
high-energy hadronic interactions in the forward direction. As the muon
density can vary between tens of muons per m2 close to the intersection of the
shower axis with the ground to much less than one per m2 when far away, the
necessary broad dynamic range is achieved by the simultaneous implementation
of two acquisition modes in the read-out electronics: the binary mode, tuned
to count single muons, and the ADC mode, suited to measure a high number of
them. In this work, we present the end-to-end calibration of the muon detector
modules: first, the SiPMs are calibrated by means of the binary channel, and
then, the ADC channel is calibrated using atmospheric muons, detected in
parallel to the shower data acquisition. The laboratory and field measurements
performed to develop the implementation of the full calibration chain of both
binary and ADC channels are presented and discussed. The calibration procedure
is reliable to work with the high amount of channels in the UMD, which will be
operated continuously, in changing environmental conditions, for several
years.

##  1 Introduction

Many questions remain open concerning the origin of high- and ultra-high-
energy cosmic rays. Due to their low flux, cosmic rays with energy above
$10^{15}$ eV can only be indirectly detected by measuring extensive air
showers produced when they interact with molecules in the Earth’s atmosphere.
Therefore, the nature of each primary cosmic ray, or the composition of their
global flux, is difficult to determine, hindering the full understanding of
data. However, the total number of muons produced in air showers scales with
the mass number of the primary particle. Direct measurements of these muons
are thus optimal to understand the chemical composition of primary cosmic
rays.

The aim of the Pierre Auger Observatory [1], located near the town of Malargüe
in Argentina, is to measure cosmic rays with energies above $10^{17}$ eV. A
hybrid technique is employed to do so, in which a fluorescence detector (FD)
and a surface detector (SD) are used in combination to detect extensive air
showers. The FD is composed of 27 fluorescence telescopes distributed at four
sites at the edge of the SD. The SD consists of an array of 1600 water-
Cherenkov detectors (WCDs) with a 1500 m spacing (SD-1500), covering a total
area of 3000 km2, and a denser array of 71 WCDs over 28 km2 with a 750 m
spacing (SD-750). More recently, the SD energy threshold has been extended
down to $10^{16.5}$ eV thanks to the installation of an even denser array of
about 2 km2 in which detectors are spaced by 433 m (SD-433).

To provide a direct measurement of the muon content in air showers, an
underground muon detector (UMD) [2, 3] is being deployed at the Auger site.
The UMD will consist of an array of 219 scintillator modules co-located at 73
positions: 61 positions of the SD-750, forming a compact hexagon, and at each
position of the SD-433. At a distance of at least 7 m from the WCD, three 10
m2 scintillation modules are buried at a depth of 2.3 m. The separation from
the surface detector guarantees that even particles with zenith angles up to
45∘ hit the scintillators without passing through the WCD while probing the
same shower density. The chosen depth, which corresponds to 540 g/cm2 of
overburden as determined by the local soil density, ensures that the
electromagnetic component of extensive air showers is largely absorbed while
vertical muons with energy >1 GeV can reach the buried detectors. Each
underground module is segmented into 64 plastic-scintillation strips of 400
cm$\times$1 cm$\times$4 cm with embedded wavelength-shifting optical fibers
coupled to an array of 64 silicon photomultipliers (SiPMs) [4, 5], Hamamatsu
S13361-2050.

The acquisition of the UMD data is performed in slave-mode, this is, only when
triggered by the associated SD stations. The shower geometry (arrival
direction and intersection of the shower axis with the ground) and the energy
are reconstructed with data from the WCDs alone by analyzing the signal as a
function of their transverse distance to the shower axis [6]. The number of
muons at the ground is, on the other hand, reconstructed with the UMD data
separately. A global estimator of the shower muon density is obtained by
reconstructing the muon lateral distribution [7] from all UMD stations and
obtaining its value at 450 m (the distance at which fluctuations of signals in
individual showers measured with the SD-750 array are minimized with respect
to an average lateral distribution). In a previous work, using data of the UMD
engineering array, we confirmed that the number of muons in data is larger
than those expected from Monte-Carlo simulations of showers between
$10^{17.5}$ eV and $10^{18}$ eV, as reported by other experiments [8, 9],
which shows that the models used in simulations do not describe accurately the
high energy hadronic interactions in the forward direction. After the
engineering-array phase, modifications in the design were implemented to
optimize the detector performance based on the experience gained with the
operation of prototypes. In particular, photomultiplier-tubes were replaced
with SiPMs to improve the detection efficiency, and upgraded electronics with
an additional acquisition mode to extend the dynamic range were implemented
[10].

A key element for the reconstruction of the lateral distribution function is
the conversion of UMD signals in units of muons, which relies on the
calibration described in this work. The density of muons at the ground depends
on the energy, primary composition, zenith angle, and distance from the shower
core: it can vary between tens of muons per m2 close to the core to much less
than one per m2 when far away. Therefore, the UMD needs to work efficiently in
a broad dynamic range and, to achieve this, two complementary working modes
are implemented: one, dubbed binary, is tuned to count single muons, the
other, named ADC, is devoted to measuring a high number of them. We describe
in detail these two modes in section 2, where we also discuss how the signal
acquired in both modes correlates.

The way to convert raw signals into a number of muons is different for the two
acquisition modes. In the binary mode, the identification of particles
crossing the detector relies on its segmentation, and the number of muons is
determined by counting signals, segment-by-segment, with an amplitude higher
than a given threshold. In contrast, the ADC mode relies on the determination
of the charge of all segments simultaneously, converted afterward to a number
of muons through the average charge of the signal produced by a single muon.
The calibration methods for the two modes are explained in Sections 3 and 4,
respectively. Finally, we summarize and conclude in section 5.

##  2 Electronics features of the underground muon detector

The UMD electronics consist, basically, of: (i) an array of SiPM with its
read-out system, (ii) a Field-Programmable Gate Array (FPGA) that houses the
acquisition logic and a soft-core for data transmission and slow control, and
(iii) an interface for the WCD trigger, monitoring data, and communications.
Working as a slave-detector, the event acquisition in the UMD stations is
governed by that of the SD, whose trigger chain has different levels [11],
being the first (T1) and second (T2) ones formed locally at each WCD. The
averaged rate of T1s and T2s is 100 and 20 triggers per second, respectively.
The buried scintillators are synchronized at the first-level trigger: when
such a condition is met, the WCD sends a signal with a timestamp to all the
underground detector modules associated with it. Upon the arrival of such a
signal, the traces of both, binary and ADC, acquisition modes, as well as the
timestamp of the trigger, are stored in an internal memory that can
accommodate data up to 2048 triggers. Therefore, the rate of triggers
determines that, at each position, the data are retrievable up to about 20 s
after its occurrence. Most of the T1s in the WCD are triggered by background
muons and, as such, uncorrelated among the array. On the other hand, all WCDs
passing the second-level triggers send their timestamp to a central data
acquisition server to form a global shower-trigger that initiates the data
acquisition from the array for permanent storage. When a shower trigger takes
place, all first- and second-level triggers from the WCDs, along with the muon
detector traces, are sent to the central data acquisition server.

A broad dynamic range is attained by the implementation of two acquisition
modes, binary and ADC, which work simultaneously. The binary mode benefits
from the detector segmentation: muons can be directly counted as pulses above
a certain threshold. This mode is very robust since it neither relies on
deconvoluting the total number of particles from a single integrated-signal,
nor on the precise optical device gain or its fluctuations, and is almost
completely independent of the hitting position of the particle on the
scintillator strip and the corresponding light attenuation through the fiber.
Another advantage is that it does not require a thick scintillator to control
Poissonian fluctuations in the number of photons per impinging muon.
Nevertheless, the binary acquisition mode is limited by the segmentation
itself: two muons arriving at the same strip simultaneously will be
indistinguishable and, therefore, only counted once. This feature is known as
pile-up effect. As far as strips with a simultaneous signal are less than the
segmentation, the pile-up effect can be statistically treated and corrected
[12], but as soon as the 64 strips in a given module have signal at the same
time, the correction is no longer feasible and the binary mode is saturated.
If the pile-up effect might produce under-counting, knock-on electrons from
the soil or inclined muons might produce signals in two neighboring strips
resulting in an over-count of particles as well as an increased probability of
saturating the detector. Therefore, the binary mode is limited in the number
of muons that can be detected at the same time, which translates into a limit
in the distance to the shower core that can be probed with it. Furthermore,
since the fluctuations in the number of hit segments depend on the number of
impinging particles, the resolution of the binary mode worsen when measuring
closer to the shower core.

The ADC mode extends the saturation point of the detector: it is well-suited
to measure a higher number of particles using the integrated signals of all 64
strips as a single measurement of the total charge in the module. Contrary to
the binary acquisition mode for which fluctuations grow with the number of
impinging particles, the fluctuations in the ADC mode decrease with the number
of detected muons. However, since it relies on estimating the number of
particles from an integrated signal in terms of an average charge, the signal
fluctuations are propagated to uncertainties in the estimation of the number
of muons. The resolution of the ADC channel is 60% for single-muon signals and
decreases inversely to the square root of the number of particles; it reaches
10% at a few tens of muons where it matches the resolution of the binary mode.
At higher muon numbers, the ADC mode has significantly better resolution than
the binary mode, allowing for measurements of particle densities closer to the
core with improved precision. In the binary acquisition mode, the 64 SiPM
signals are handled independently through a pre-amplifier, fast-shaper, and a
discriminator, built within each channel of two 32-channel Application-
Specific Integrated Circuits (ASICs). The discriminator signal is sampled at
320 MHz (3.125 ns sample time) by the FPGA into 64 traces of 2048 bits. In
each trace, a “1”-bit results if the fast-shaper output is above a fixed
discriminator threshold, and a “0”-bit otherwise. As explained in section 3,
this threshold is defined to significantly reject the SiPM noise while
preserving a high detection efficiency.

In the ADC acquisition mode the 64 SiPM signals are added up analogically and
the result is amplified with low- and high-gain amplifiers with a ratio of
about 4 in the amplification factor. The low-gain channel extends the dynamic
range of the UMD to measure a higher number of muons when the binary mode is
saturated. The high-gain channel, initially conceived to work in an
intermediate range of muon numbers, resulted to perform similarly to the
binary channel. The amplifier outputs are sampled at 160 MHz (6.25 ns sample
time) with two 14-bit Analog-to-Digital Converters (ADCs) resulting in two
waveforms of 1024 samples. A schema of the read-out electronics is shown in
the upper panel of Fig. 1. In the lower panel, we show, from top to bottom,
how the SiPM (a), the binary channel (b), and the ADC channel (c) respond to a
simulated single muon.

Figure 1: (Top) Schematics of the UMD electronics. (Bottom-a) Simulated SiPM
response to a single-muon signal. (Bottom-b) Simulation of the binary mode
response to the simulated muon. The discriminator pulse is re-scaled for
illustration. (Bottom-c) Simulation of the ADC mode processing. The two ADC
outputs (low- and high-gains) are shown.

As the signal processing in binary and ADC modes is performed with different
sets of electronics, differences in the response to the same physical event
are expected. The different architectures introduce, in particular, different
read-out delays, so that the start time of the signals in binary and ADC
channels do not coincide: as one can see in the lower panel of Fig. 1, the
start-time of the ADC trace is delayed with respect to the binary one.
Nevertheless, as we shall shortly see in the following, this time shift is
very-well defined, stable in time, and uniform across the different
electronics.

To study in detail how the two electronics relate one to another, we assembled
in the laboratory a scintillator strip, identical to those used in the UMD,
with an optical fiber coupled to a standard SiPM and read-out electronics, and
a muon telescope movable at different positions along the scintillator strip.
The muon telescope consists of two scintillator segments of 4 cm$\times$4
cm$\times$1 cm, with a SiPM coupled to each segment. A schema of the setup is
presented in the upper panel of Fig. 2. When a muon impinges on the telescope,
the SiPM signals are correlated with an AND gate to inject a trigger signal in
the UMD electronics. The muon also produces a signal in the scintillator
strip: a sequence of positive samples in the binary mode and a waveform in the
ADC mode is then obtained.

Figure 2: (Top) Schematics of the laboratory setup used to characterize the
UMD traces. (Bottom-left) Average trace for the binary (solid-blue and left
y-axis), ADC high- (long-dashed-orange and right y-axis), and low-gain (short-
dashed-red) channels measured at different optical-fiber lengths (1 to 4.5 m).
(Bottom-right) Signal start time as a function of the optical-fiber length,
for the binary (blue triangles) and ADC (red triangles) channels. The
difference between the channel start times ($\Delta$t) measured in the
laboratory (black circles) and the field (red square) is shown in the lower
part.

The signals read by the SiPM depend on the optical-fiber length between the
SiPM and the position where the muon impinges on the scintillator. To
characterize these signals, we measure 20000 muons produced at different
positions on the scintillator strip, corresponding to lengths between 1 m and
4.5 m, in intervals of 0.5 m. The average of these signals for each length is
displayed in the bottom-left panel of Fig. 2. For the binary mode, each of the
2048-bit traces is summed and normalized to the total number of events (solid-
blue curves and left y-axis) resulting in a histogram whose peaks correspond
to the most probable bit to have a “1”. For the ADC high- (long-dashed-orange
curves) and low-gain (short-dashed-red curves) modes, the traces are averaged
resulting in the mean waveform (right y-axis) of the muon signal. In both
cases, we indicate the curves corresponding to the measurements at 1 m and 4.5
m. The general time shift between the ADC and the binary channels, introduced
by different time response of the electronics, is apparent.

In the bottom-left panel of Fig. 2, two optical-fiber effects can be inferred.
The light produced after a muon impinges on the scintillator strip is
attenuated in the optical fiber and loses intensity as it propagates to the
SiPM: the histogram peak of the binary channel and the maximum of the ADC
waveform decrease as a function of the optical-fiber length. In addition,
photons produced at different positions of the strip arrive at the SiPM at
different times due to the delay introduced by the photon propagation in the
optical fiber: the time of the histogram and waveform maximums is shifted when
measuring at different positions on the strips. To better illustrate this
fact, we present in the bottom-right panel of Fig. 2 the mean start time of
the binary channel as a function of the optical-fiber length (blue triangles).
We define the start time of each individual trace as the time where the first
“1” in the trace is found. The slope of this curve corresponds to the signal
speed in the optical fiber, which we estimated, by fitting the data, to be
$\mathrm{(5.67\pm 0.05)\,ns/m}$, about 1 / 0.6 c. This indicates that the
photons in the optical fiber propagate at about 60% of the speed of light in
the vacuum111Due to the refractive index of the material (n = 1.60), the speed
of light in the core of the optical fiber is 62.5% of the speed in the vacuum.
However, the speed of the signal is a convolution of the effect from the
refractive index of the material and the path the photons follow when
propagating due to total internal reflection in the fiber cladding. The
selected fiber for the UMD is the Saint-Gobain BCF-99-29AMC..

Finally, we show in the bottom-right panel of Fig. 2 the mean shift,
$\langle\Delta$t$\rangle$, between the start times of the binary and ADC
channels (black circles); the time shift was calculated event by event, and
only averaged values are shown. The ADC start time is defined as the time at
which the signal is five standard deviations above the baseline. We obtained
that the time shift between channels is constant and does not depend on the
position where the muon impinges on the scintillator strip. We also show the
mean time shift calculated with data taken with a module deployed in the field
(red square), as an example. In this case, as it is not possible to identify
the position on the strip where the muons impinge on the scintillator, we
assume the average optical-fiber length corresponding to the center of the
module. The data from the field are well-matched with those taken in the
laboratory. For completeness, the mean start times of the ADC channel, minus
$\langle\Delta$t$\rangle$, are shown as red triangles in the panel above, to
illustrate their consistency with the start times of the binary. As it will be
explained in section 4, characterizing the signal start time and the time
shift between the channels constitutes a key point in the calibration of the
ADC.

##  3 SiPM calibration and setup of the binary acquisition mode

Each SiPM of the UMD is an array of 1584 avalanche photo-diodes (also referred
to as cells) operated in Geiger mode; the cells are distributed over an area
of 2 $\times$ 2 mm2. To set up the SiPM, a reverse bias voltage across the
SiPM terminals ($\mathrm{V_{bias}}$) needs to be set. The minimum voltage for
SiPMs to function corresponds to the breakdown voltage of the silicon
junctions ($\mathrm{V_{br}}$). Then, the SiPM gain, noise and efficiency are
proportional to the difference between $\mathrm{V_{bias}}$ and
$\mathrm{V_{br}}$, commonly referred as over-voltage ($\mathrm{V_{ov}}$). When
$\mathrm{V_{ov}}>0$, photons absorbed in the silicon produce electron-hole
pairs, which trigger a diverging avalanche of carriers. Regardless of the
number of photons impinging on a cell, this avalanche results always in the
same macroscopic current, which only depends on the SiPM gain. Therefore, at
the SiPM output, the signal amplitude and charge are proportional to
$\mathrm{V_{ov}}$ and to the number of triggered cells, which we refer to as
photo-equivalents (PE). Furthermore, since signal fluctuations are rather
small, the amplitude (and charge) distributions for each number of PE are
distinguishable one from another, making it possible to identify the number of
PE in individual signals.

The noise in SiPMs is mainly caused by spurious pulses produced in absence of
light. In silicon, there is a probability for carriers to be generated by
thermal agitation; if an electron-hole pair is produced inside the active
region of a cell, an avalanche may be initiated. We refer to the pulses that
are not originated by incident radiation as dark counts. Furthermore, optical
cross-talk between cells may occur when electrons are recombined during the
avalanche emitting photons [13]. These may escape from the triggered cell and
hit neighbor cells producing simultaneous avalanches. As a result, an output
of several PEs may be obtained, despite having only one cell triggered by
incident radiation or thermal noise. In SiPMs, the dark-count rate and cross-
talk probability are increasing function of the $\mathrm{V_{ov}}$ and of the
temperature222The direct dependency of the cross-talk probability with the
temperature is rather weak, if any, compared to most of the parameters in
SiPMs. [13, 14].

The purpose of the SiPM calibration in the UMD is to equalize the gain (hence
$\mathrm{V_{ov}=V_{bias}-V_{br}}$) of the 64 devices in each 10 m2 unit to
achieve a uniform response in each scintillator module and, therefore, in the
full UMD array. As $\mathrm{V_{br}}$ varies from one SiPM to another, the
first step of the calibration is to determine the $\mathrm{V_{br}}$ of each.
To this aim, we first measure the dark-count rates with different
$\mathrm{V_{bias}}$. At each $\mathrm{V_{bias}}$, we vary the discriminator
threshold of the binary channel, using a 10-bit digital-to-analog converter
(DAC) integrated into the ASICs, and we count the number of SiPM signals per
second above each threshold level. The measurements are performed at a
reference temperature of 25 ∘C. As an example, we show in the top panel of
Fig. 3 the dark counts per second (cps) as a function of the discriminator
threshold for one SiPM measured with different $\mathrm{V_{bias}}$, in which
different plateaus can be clearly identified. The width of the plateau is
related to the separation in amplitude between two consecutive integer numbers
of PEs and corresponds to the transition between these signals.

Since the amplitude fluctuation of signals with a certain number of PEs is
small, the rate of events varies slowly when the discriminator threshold is
moved far from the mean signal amplitude. In contrast, when the discriminator
threshold is near to the mean amplitude of the signal, the number of events
changes rapidly and a transition between plateaus is observed. The first
plateau corresponds to the transition between the baseline and 1 PE signals,
the second to the transition between 1 and 2 PE signals, etc. The 1 PE signal
amplitude, which depends on $\mathrm{V_{bias}}$, can be easily identified in
the middle between the first and second plateau. The sets of amplitude values
for 1 and 2.5 PEs are represented, as an example, in the figure with black
dashed lines. To better illustrate the procedure for finding the PE
amplitudes, we present in the inset a curve with its derivative fitted with a
Gaussian distribution whose mean corresponds to the amplitude of 1 PE for
$\mathrm{V_{bias}}$ = 57.58 V. By repeating this process with all the curves
we can determine the signal amplitude of 1 PE for different
$\mathrm{V_{bias}}$.

Figure 3: (Top) Dark-counts per second (cps) as a function of the
discriminator threshold at different $\mathrm{V_{bias}}$ for an individual
SiPM. The dark-count rates and PE amplitudes shift towards higher values when
rising the $\mathrm{V_{bias}}$. In the inset, we show one curve at the middle
of the displayed range in the main plot and its derivative fitted with a
Gaussian distribution to obtain the corresponding PE amplitude. (Bottom-left)
$\mathrm{V_{br}}$ determination for this specific SiPM. The linear
extrapolation to 0 amplitude yields the value of $\mathrm{V_{br}}$. (Bottom-
right) Dark-count rate (red squares) and cross-talk probability (black dots)
as a function of the $\mathrm{V_{bias}}$.

The dark-count rate thus allows us to build the mean amplitude of the 1 PE
signal as a function of the $\mathrm{V_{bias}}$, as shown in the bottom-left
panel of Fig. 3. The mean amplitude increases linearly with
$\mathrm{V_{bias}}$ as expected, and is zero for
$\mathrm{V_{bias}}=\mathrm{V_{br}}$. With a linear fit to the measured
amplitudes, we thus obtain $\mathrm{V_{br}}$ for each SiPM, which is
$(51.68\pm 0.04)$ V in this example. The $\mathrm{V_{bias}}$ is then adjusted
for each SiPM, so that all of them get the same $\mathrm{V_{ov}}$, making
their response uniform.

Besides equalizing $\mathrm{V_{ov}}$ for the different SiPMs, the dark-count
rate measurements allow us also to quantify the SiPM noise. We show in the
bottom-right panel of Fig. 3 the dark-count rate (red squares and y-axis on
the left) and cross-talk probability (black dots and y-axis on the right) as a
function of $\mathrm{V_{bias}}$. The dark-count rate is estimated as the rate
of signals with amplitude $>0.5$ PE. In turn, the cross-talk probability is
derived from the ratio between the rate of signals with amplitude >2 PE and
that with amplitude >1 PE, due to the fact that events produced by thermal
fluctuations can have more than 1 PE amplitude only if cross-talk signals are
produced effectively. As one can see from the figure, both the dark-count rate
and the cross-talk probability depends on the $\mathrm{V_{bias}}$, given that,
as mentioned before, the SiPM noise depends on the $\mathrm{V_{ov}}$.

Increasing the $\mathrm{V_{ov}}$ thus results, on the one hand, in a higher
gain and efficiency, but also, on the other end, in higher noise. During the
engineering array phase, the $\mathrm{V_{ov}}$ was optimized by balancing
these parameters [5]. With laboratory measurements we have shown that both a
high efficiency and a sufficiently low noise are achieved with
$\mathrm{V_{ov}}=3.5$ V ($\mathrm{V_{bias}}=55.2$ V in the figure). For the
selected $\mathrm{V_{ov}}$, the typical value of cross-talk probability is
about 2% and that of dark-count rate, at 0.5 PE, is between 450$\sim$850 kcps,
depending on the SiPM temperature. The discriminator threshold of the binary
mode is then chosen to reduce the impact of this residual noise. Increasing
the threshold results in a better background reduction but also loss of
signal. 2.5 PE gives the best balance to reduce the dark-count rate
measurements while keeping about 99% of the muons from the muon telescope
studies [15].

The effect of the SiPM equalization can be appreciated in Fig. 4, where we
illustrate the result of the calibration performed on one scintillator module,
as an example. We show the dark-count rate curves for the 64 SiPMs as a
function of the discriminator threshold and, in the insets, the 1 PE amplitude
histograms, before (left) and after (right) calibration. The standard
deviation of the histograms is reduced by a factor about two after
calibration, which reflects the expected better uniformity of the detector
response. Note that lines are displayed to ease the comparison between the
curves, where points correspond to the data taken. After calibration, the
dispersion of the gain is smaller than the minimum step of 1 DAC count, which
corresponds to the DAC resolution. Note that while a large dispersion on the
gain should be avoided, the chosen discriminator threshold in the middle of a
plateau where the dark-count rate is rather constant minimizes the effect it
has on rejecting dark counts. As can be inferred from the top panel of Fig. 3,
variations of a few ADC counts in the gain have no impact in the performance
and are acceptable for the full calibration procedure.

Figure 4: Dark-count rate as a function of the discriminator threshold for the
64 SiPMs of a typical scintillator module, before (left) and after (right) the
calibration. Histograms of the signal amplitude corresponding to 1 PE are also
shown in the insets. Solid lines between data points are shown to ease the
comparison.

The procedure described above was performed in the laboratory but it can be
repeated at any time in the field, with the module removed from normal data
acquisition. In fact, given that the strategy to calibrate the SiPM array and
the binary channel is based on the analysis of thermal noise, performing the
calibration in the laboratory or the field is, for any purpose, indistinct.
This can be seen in the left panel of Fig 5 where we show the histograms of
the PE amplitudes, obtained with dark-count measurements on a sample of 512
SiPMs, in the laboratory (dashed blue line) and in the field (solid-red line),
after calibration. The two distributions are well-superimposed, the difference
between the two measurements being always smaller than 0.5 DAC counts. We
compare the discriminator threshold levels using the mean of these histograms:
in both cases the level of 2.5 PE corresponds to 61 DAC counts.

The calibration procedure has been applied, in the laboratory, to more than
6400 SiPMs: in the right panel of Fig. 5 we show that the distribution of the
PE amplitudes minus the mean of each corresponding module results well-
centered to zero. This shows that the calibration effectively allows us to
obtain a uniform gain, the maximum deviation from the mean being of 3 DAC
counts (4.5 standard deviation), which, well within the acceptable range, does
not have an impact on the detector performance. This calibration procedure
guarantees a sufficient uniformity to work in all SiPMs of the UMD array.

Figure 5: (Left) PE amplitude obtained in the laboratory and in the field for
512 SiPMs. (Right) PE amplitude for 6400 SiPMs; for each set of electronics,
the mean amplitude of each module was centered to zero.

The SiPM gains, determined with the calibration, are however subjected to
drifts, due to the well-known dependence of $\mathrm{V_{br}}$ with the
temperature. The UMD operates in an outdoor ambient: even if it is buried 2.3
m deep, it is exposed to temperature variations of about $20^{\circ}$ every
year. The gain drifts must be compensated by readjusting $\mathrm{V_{bias}}$,
to maintain the value of $\mathrm{V_{ov}}$ constant. Such an adjustment is
done automatically, by exploiting a mechanism of temperature compensation that
is integrated into the high-voltage source (C11204-01 by Hamamatsu). This
mechanism adjusts the high voltage, i.e., $\mathrm{V_{bias}}$, according to
the temperature variation: set at a reference value of
$25\,^{\circ}\mathrm{C}$, it compensates by 54 mV$/^{\circ}\mathrm{C}$, which
is the temperature dependence of $\mathrm{V_{br}}$. In the left panel of Fig.
6, we present the signal amplitude at 1 PE (averaged over 64 SiPMs) as a
function of the temperature of the high-voltage source measured in the
laboratory without (red squares) and with (black circles) the temperature
compensation. When the compensation mechanism is off and the temperature
increases, then the $\mathrm{V_{br}}$ increases and, for a fixed
$\mathrm{V_{bias}}$, the $\mathrm{V_{ov}}$ and the gain decrease. It is
apparent how the PE amplitude is stabilized when the compensation mechanism is
on. The remaining variation of about 2 DAC counts in the whole temperature
range is well within the acceptable range, and does not have an impact on the
detector performance. In the right panel of Fig. 6 we show the average dark-
count rate as a function of the temperature measured with and without the
compensation mechanism. For comparison, the same temperature range is
displayed for both measurements. Since the dark-count rate increases both with
the SiPM temperature and $\mathrm{V_{ov}}$, when the compensation mechanism is
off, two opposite effects contribute to the variation of the total rate: an
increase with the temperature of the silicon and a decrease with the fall of
the $\mathrm{V_{ov}}$. However, in the given ranges, the fluctuation of the
dark-count rate due to the temperature variation (about 12 ∘C) is stronger
than that produced by the variation of the $\mathrm{V_{ov}}$ (about 0.6 V).
Therefore, the dark-count rate fluctuation behaves similarly when measuring
with and without the temperature compensation mechanism.

Figure 6: 1 PE amplitude (left) and dark-count rate (right) as a function of
the temperature of the high-voltage source measured in the laboratory. We
display the results for measurements with and without the compensation
mechanism on. The fitted slopes for the PE amplitude data are (-0.049 $\pm$
0.003) PE / ∘C and (-0.431 $\pm$ 0.002) PE / ∘C, respectively.  Figure 7: Mean
rate per SiPM (left y-axis) and mean electronics temperature (right y-axis)
averaged every hour in an example scintillation module in operation in the
field. The gap between mid-December and January of 2020 corresponds to a
period in which the corresponding position was not in acquisition.

The temperature range probed in the laboratory, between about $30^{\circ}$ and
$42^{\circ}$, is contained within that experienced during operation in the
field, as one can see in Fig. 7. The red squares show the variations of the
temperature of the electronics (right y-axis), averaged every hour, in one
scintillation module during more than three years of operation: the daily and
yearly modulations are apparent. We also show the rate of signals with an
amplitude $>2.5$ PE averaged over all the SiPMs in the same module (black dots
and left y-axis). Since the dark-count rate depends on the SiPM temperature,
independently of the $\mathrm{V_{ov}}$, a modulation of the background rate is
observed, despite the stability of $\mathrm{V_{ov}}$. The factor 2 in noise
level between the warmest and coolest months of the years translates into a
probability between 13% and 25% of having noise producing a “1” in the binary
trace. This will not have a significant impact in the muon number estimation,
since these signals can be rejected by setting a cut, explained in Section
4.1.

##  4 Calibration of the ADC acquisition mode

In the ADC channel, the number of muons is estimated by normalizing the charge
of the recorded signal to the mean charge of a single muon. Therefore, the
goal of the ADC calibration is to obtain the latter for each UMD module. To
this aim, we exploit atmospheric muons, with the same approach used for the
calibration of the WCD [16]. This approach is however more challenging for the
UMD modules. As explained in Section 2, the trigger of each UMD module is not
autonomous, but it depends on the first-level trigger of the associated WCD.
The signals in the WCD and in the UMD are in fact mostly uncorrelated: in the
majority of the events triggered by the WCDs, the signals in the UMD are
largely due to noise. However, we show in this section how it is possible to
select single-muon signals and use them to calibrate the ADC channel, thanks
to the sufficiently high rate of the first-level WCD trigger (100 triggers per
second). The procedure aims at filling, during data acquisition, histograms of
the muon-charge distribution, from which the average charge is extracted. Two
ingredients need to be defined first: the strategy for the selection of muons,
performed with the binary channel, and the calculation of their integrated
charge, performed with the ADC channel.

###  4.1 Muon selection criteria and charge calculation

At the lowest trigger level, the bulk of the signals recorded in the ADC are
due to SiPM dark counts. As no amplitude filter is implemented in the ADC
acquisition mode (such as the discriminator threshold in the binary mode),
each of the 64 SiPMs contributes to the baseline fluctuation with more than
0.5 Mcps of dark-count rate. To reduce such background while selecting signals
due to muons, we make use of the binary channel. When a first-level trigger is
received from the WCD, we build an internal second-level trigger based on the
signal width, defined as the number of “1”s in the binary trace. To define
such a trigger, we first characterized the width of single-muon and noise
signals by using a laboratory setup, including a muon telescope, similar to
that described in section 2 and shown in the upper panel of Figure 2. In this
configuration, only one SiPM in the array is coupled to an optical fiber, with
the other 63 being passive and contributing only with dark counts. The SiPM
array and binary channel were calibrated as explained in Section 3.

We used the muon telescope to collect 16000 signals at different positions of
the scintillator strip, in the same way as explained in section 2. To
characterize the muon signal, we defined a time window around the trigger time
given by the telescope. Signals produced away from this window were classified
as noise. The other 63 SiPMs were also turned on: when dark counts with
amplitude $>2.5$ PE were produced, “1”s in the corresponding binary traces
were obtained. In the left panel of Fig. 8, we display the histogram of the
widths of the single-muon signal (solid-black line), based on the measurements
made at all positions. The spread of the distribution is in part an effect of
the attenuation in the optical fiber, as illustrated in the right panel of
Fig. 8, where we show the behavior of the average width of the single-muon
signal (black circles) as a function of the optical-fiber length. The error
bars represent one standard deviation. One can clearly see that, as expected,
longer (shorter) signals are detected closer to (farther from) the SiPM. In
spite of such a spread, the distribution is well-separated from that due to
noise signals, also shown in the left panel of Fig. 8. The long-dashed red
histogram corresponds to the distribution of the widths of the noise signals
in the 63 SiPMs not coupled to the optical fiber, while the short-dashed blue
one is due to noise-signals in the SiPM coupled to the optical fiber. The
widths of the SiPM noise and of the muon signal are well-distinguished: the
distribution relative to the muon signal has an average width of 7.8 samples
(24.4 ns) with a standard deviation of 1.5 samples (4.7 ns) while that due to
the SiPM-noise has an average width of 3.0 samples (9.4 ns) with standard
deviation 0.7 samples (2.2 ns). Note that the noise in the SiPM coupled to the
optical fiber (blue histogram) is the convolution of two contributions: the
SiPM dark-count noise, the peak of which is at about 3 samples, and a tail,
with a mean at about 6 samples, not related to the SiPMs, which we attribute
to the optical-fiber/scintillator system [17]. Even though the latter
generates signals shorter than those due to muons, it cannot be efficiently
rejected without losing muon signals. The effect of this irreducible
background is however rather small, yielding a 5% probability of over-counting
a muon in a module when considering the whole trace of 6.4 $\mu$s.

Figure 8: Binary channel. (Left) histograms of signal widths due to single
muons (black), SiPM noise (long-dashed red), and SiPM plus optical fiber-
scintillator noise (short-dashed blue). (Right) mean width of single-muon
signals as a function of the optical-fiber length. The one standard deviation
from the mean is displayed as error bars. The red square corresponds to the
result obtained with field data. The widths are indicated in number of samples
(3.125 ns).

Based on these measurements, to select a muon we set a condition on the width
of the signal, requiring more than four “1”s (left arrow in the figure), which
corresponds to about three standard deviation from the average duration. This
cut rejects thus more than 95% of the SiPM noise with less than 1% of muon
signal loss. Similarly, we also set an upper cut of 12 samples (right arrow in
the figure), which allows us to select about 99% of the single-muon signals,
avoiding biases in the muon charge estimation due to shower events with
several particles. The two cuts are also shown as grey dashed lines in the
right panel. To verify the applicability of these cuts to the UMD in the
field, we measured the average width of binary signals in a module as an
example. In the latter, we expect to observe wider signals than the mean found
in the laboratory, since knock-on electrons from the soil also deposit energy
in the scintillators, thus producing more light. Furthermore, the laboratory
setup is more efficient at selecting vertical muons, which leave a smaller
energy deposit than inclined muons as found in the field. The resulting
measurement, shown in the figure as a red square, is still well-consistent
with that obtained in the laboratory.

The second ingredient required for the calibration procedure is the
calculation of the integrated charge for the selected muon, by means of the
ADC channel. To do that, an integration window needs to be optimized to reduce
contributions from the noise due to the SiPM dark-count rate which causes
fluctuations in the ADC baselines. By using the laboratory data acquired with
the ADC channel, we studied the width of the muon signal, defined as the
number of ADC bins with an amplitude above two times the standard deviation of
the baseline. The distribution of the muon-signal widths is displayed in the
left panel of Fig. 9, with the right panel showing the behavior of the signal
width (black circles) as a function of the optical-fiber length (error bars
correspond to one standard deviation). Also, in this case, the signal becomes
narrower as the distance increases, due to the light attenuation of the
optical fiber. Based on this measurement, we defined the integration window
for single-muon signals as three standard deviations from the mean width (grey
arrow in the left panel, grey dashed line in the right one), which corresponds
to 32 samples (200 ns). The red square corresponds to the mean width obtained
from measurements in the field, showing its compatibility with the measurement
in the laboratory.

Figure 9: ADC channel. (Left) histogram of the single-muon signal width
measured with the ADC in the laboratory. (Right) Mean width of the single-muon
signal as a function of the position of the impinging muon. Results from the
laboratory (black circles) and field (red squares) with one standard deviation
as error bars are presented. The widths are indicated in number of samples
(6.25 ns).

###  4.2 Filling of charge histograms

The core of the calibration procedure is to periodically fill histograms of
the signal charge produced by atmospheric muons. To explain the strategy, we
start from Fig. 10, where we show the traces from the binary channel (top) and
the ADC (bottom), recorded following a WCD first-level trigger and summed over
one hour of operation in the field (360000 events). The two histograms show
similar features: a rather uniform distribution of signals along the trace,
due to random detector noise, and an accumulation, due to particles, at a time
that corresponds to that of the trigger from the WCD. Note that in the summed
ADC traces undershoots are observed, after the trigger time, corresponding to
high-amplitude signals, most likely due to air shower events.

To identify single-muon signals in the traces, we first search for “1”s in a
window around the trigger, illustrated with orange lines in the top panel.
Then we apply the selection criterion described above, looking for a sequence
of more than four and less than twelve “1”s within this window. If this
criterion is satisfied, we set as the start time of the binary trace the first
“1” within the window, and then we locate the start-time of the ADC trace by
including the information on the time shift between binary and ADC channels,
presented in section 2. Finally, to extract the signal charge, we use the
integration window of 32 samples in the ADC trace (200 ns), as explained above
and illustrated by the orange lines in the bottom panel. As already mentioned,
this procedure is effective to remove most of the noise from the SiPM, but not
that from the optical fiber/scintillator system. Therefore, we repeat the
above-described procedure, applying it in a time window separated from the
trigger time window (represented by black lines in the bottom panel) where
only noise contributes to the traces: we use this window to estimate the
optical fiber/scintillator noise to subtract from the calibration data.

Figure 10: 360000 background events which correspond to the equivalent of one
hour of first-level triggered events. In orange (black) the time windows to
search for muon (noise) signals are indicated. (Top) sum of binary channel
traces. (Bottom) overlap of ADC channel traces. Note that the x-axis is
shifted by about 170 ns, the time shift between the channels presented in the
bottom-right panel of Fig. 2.

The choice of the window size to search for muons and noise is important.
Narrowing the size would reduce the number of noise signals, but it would also
reduce the number of muon signals. To illustrate this effect, we show, in the
left panel of Fig. 11, the number of events, satisfying the single-muon search
criteria, in the noise window of the traces (black squares) and in the signal
one (orange circles), as a function of the window size. As there are no muon
signals in the noise window, the number of events grows linearly with its
size. For the signal window, the number of events increases more rapidly than
in the case of pure noise, up to about 35 samples. Above this value, the curve
becomes parallel to that due to only noise, thus indicating that no muons
contribute any more to the number of events. This fact can be better observed
in the right panel of Fig. 11, where we show the difference in the number of
events between noise and signal windows (green circles): it increases for
windows up to 35 samples, then it becomes constant. In addition, we present
the ratio between the number of signal and noise events (blue squares) as a
function of the window size. The asymptotic behavior to zero indicates that,
when increasing the window size, the number of noise events grows
proportionally faster than the number of signal events. To implement the
calibration procedure, we thus choose a window size of 20 samples (indicated
with gray-dashed lines), for which 80% of the events are muons and 20% are
noise. Such a choice allows us to keep a large enough number of events, about
1200 for the analyzed period, minimizing the contamination due to noise.

Figure 11: Number of events in noise and signal-with-noise windows (left) and
number of muon events (right) for each window size is displayed. The ratio
between the number of signal and noise events is also presented.

In Fig. 12 we present the muon charge histogram (green curve) obtained with
the described procedure and a window of 20 samples. This histogram has been
obtained by subtracting the noise histogram (dashed black) from the total
charge histogram (dotted orange) that contains both signal and noise. The mean
charge obtained in this case is (5.3 $\pm$ 0.1) pC with a standard deviation
of (3.1 $\pm$ 0.1) pC. This corresponds to the mean charge deposited in the
UMD by omnidirectional muons, and therefore includes the contribution of
secondary knock-on electrons produced in the 540 g/cm2 overburden. The mean
muon energy in showers measured with the UMD varies from about 2 GeV far from
the shower core up to about 30 GeV when close. The contribution of knock-on
electrons to the total charge is not expected to largely depend on the
distance to the shower axis: on average about 10% variation is expected in the
estimation of the particle density for the most energetic muons close to the
core with respect to those, less energetic, far from the impact point. For the
analysis of cosmic-ray showers, dividing the total charge seen in a UMD module
by the mean atmospheric muon charge measured in the same underground module
will give, at first order, an estimation of the muon density at the surface.

Figure 12: Charge histograms for a window size of 20 samples. The calibration
histogram (solid green) is filled by subtracting the charge histogram in the
noise window from the charge histogram in the trigger window.

To extract data for the ADC calibration, an algorithm is set up in each UMD
module in the field, which runs in parallel to the air shower data
acquisition. This algorithm is designed to select the traces that match the
muon selection criteria, both for the signal and noise window. This
information, along with the trigger timestamp, is stored in an external
server, and the subsequent analysis is performed offline: a calibration
histogram can be filled by selecting a time range of these data. To assess the
performance of this calibration method under temperature variations, we have
filled calibration histograms with data obtained with one module during
several weeks in two periods with average electronic temperatures of 32∘C and
43∘C. In both cases, the mean of all histogram was (6.09 $\pm$ 0.01) pC with a
standard deviation of (0.14 $\pm$ 0.01) pC and (0.12 $\pm$ 0.01) pC
respectively, which translates into a fluctuation of the mean charge of about
2%. Furthermore, the uniformity of the calibration has been tested over 21
modules in the field obtaining a mean charge of (6.09 $\pm$ 0.01) pC with a
standard deviation of (0.5$\pm$ 0.1) pC, which indicates that the method is
robust and its output quite uniform.

##  5 Summary

In this work, we have discussed the method developed to calibrate the
underground muon detector of the Pierre Auger Observatory. A key
characteristic of the scintillator-based detector is its high segmentation:
each module is divided into 64 segments, coupled to a set of 64 SiPMs. The aim
of the UMD is the direct measurement of the muon content in air showers with
energy above $10^{16.5}$ eV. This is achieved by implementing two acquisition
modes: a binary mode, suitable for measuring low particle densities as found
far from the shower core, and an ADC mode, tuned to measure high muon
densities, such as those close to the shower core. For both channels, the
ultimate scope of the calibration is the conversion from raw signals to the
number of muons.

To develop the calibration method, correlations between the binary and ADC
channels were first studied using a dedicated laboratory setup, consisting of
a scintillator strip with optical fiber coupled to a SiPM and of a movable
muon telescope. This allowed us, on the one hand, to study the response of the
two channels to vertical muons at different positions, in terms of signal
amplitude and start time. On the other hand, it allowed us to fully
characterize the intrinsic time shift between the two channels.

Having characterized the relevant electronics features, we then proceed to
explain the calibration of the binary channel. The first step is to equalize
the SiPM gains and guarantee a uniform response of the UMD array which is of
utmost importance since, in this channel, the number of muons is obtained by
counting signals above a certain threshold, segment by segment. We implemented
a method based on the measurement of dark-count rates, in which we first
determine the reverse bias voltage for each SiPM, and then we set a threshold
for the binary channel to 2.5 PE, which filters most of the SiPM noise. By
applying the method to more than 6400 SiPMs, we have verified that it allows
us to achieve a PE amplitude with a maximum deviation from the mean of 3 DAC
counts. Also, we have demonstrated that the performance of the calibration is
the same when applied either in the laboratory or the field, thanks to the
chosen strategy based on the analysis of thermal noise. Finally, we showed
that the well-known drifts in the gains, due to temperature variations, are
successfully stabilized by the temperature compensation mechanism integrated
into the electronics, which reduces the temperature dependence by a factor of
about 9.

Next, we presented the calibration of the ADC channel. In this case, the aim
of the calibration is to provide the average charge of the signal of a single
muon, so that the number of muons can be obtained by scaling the total charge
measured in a UMD module. To this aim we use atmospheric muons, detected
underground. Because the UMD is not triggered autonomously, but by the
associated water-Cherenkov detector, only a part of the available signals
correspond to muons and most of them correspond to uncorrelated noise. To
overcome such a large background, we use the information in the binary channel
to select muon signals. The selection criteria, based on the amplitude and the
duration of the binary signals, have been defined after a study in the
laboratory with a muon telescope. We have shown that they reject more than 95%
of the SiPM noise with about 1% of muon signal loss. Once muons are selected
with the binary channel, the start time of the signal in the ADC channel is
identified and the charge is computed using an integration window which
corresponds to the mean width of the single-muon signal plus three standard
deviations, as measured in the laboratory and verified in the field. Finally,
to obtain an unbiased estimation of the mean charge of single muons, we fill a
charge histogram of the signal subtracted of that due to the detector noise.
This method was tested in different UMD modules, also over time, denoting that
the calibration is stable over time and uniform over the array.

In conclusion, we presented the methods to perform the UMD end-to-end
calibration. We proved that these methods are robust to work with the high
amount of channels to operate and stable under the temperature variations
expected in the field. The methods described in this work will be applied to
the 73 stations (219 modules) of the UMD array.

## Acknowledgments

The successful installation, commissioning, and operation of the Pierre Auger
Observatory would not have been possible without the strong commitment and
effort from the technical and administrative staff in Malargüe. We are very
grateful to the following agencies and organizations for financial support:

Argentina – Comisión Nacional de Energía Atómica; Agencia Nacional de
Promoción Científica y Tecnológica (ANPCyT); Consejo Nacional de
Investigaciones Científicas y Técnicas (CONICET); Gobierno de la Provincia de
Mendoza; Municipalidad de Malargüe; NDM Holdings and Valle Las Leñas; in
gratitude for their continuing cooperation over land access; Australia – the
Australian Research Council; Brazil – Conselho Nacional de Desenvolvimento
Científico e Tecnológico (CNPq); Financiadora de Estudos e Projetos (FINEP);
Fundação de Amparo à Pesquisa do Estado de Rio de Janeiro (FAPERJ); São Paulo
Research Foundation (FAPESP) Grants No. 2019/10151-2, No. 2010/07359-6 and No.
1999/05404-3; Ministério da Ciência, Tecnologia, Inovações e Comunicações
(MCTIC); Czech Republic – Grant No. MSMT CR LTT18004, LM2015038, LM2018102,
CZ.02.1.01/0.0/0.0/16_013/0001402, CZ.02.1.01/0.0/0.0/18_046/0016010 and
CZ.02.1.01/0.0/0.0/17_049/0008422; France – Centre de Calcul IN2P3/CNRS;
Centre National de la Recherche Scientifique (CNRS); Conseil Régional Ile-de-
France; Département Physique Nucléaire et Corpusculaire (PNC-IN2P3/CNRS);
Département Sciences de l’Univers (SDU-INSU/CNRS); Institut Lagrange de Paris
(ILP) Grant No. LABEX ANR-10-LABX-63 within the Investissements d’Avenir
Programme Grant No. ANR-11-IDEX-0004-02; Germany – Bundesministerium für
Bildung und Forschung (BMBF); Deutsche Forschungsgemeinschaft (DFG);
Finanzministerium Baden-Württemberg; Helmholtz Alliance for Astroparticle
Physics (HAP); Helmholtz-Gemeinschaft Deutscher Forschungszentren (HGF);
Ministerium für Innovation, Wissenschaft und Forschung des Landes Nordrhein-
Westfalen; Ministerium für Wissenschaft, Forschung und Kunst des Landes Baden-
Württemberg; Italy – Istituto Nazionale di Fisica Nucleare (INFN); Istituto
Nazionale di Astrofisica (INAF); Ministero dell’Istruzione, dell’Universitá e
della Ricerca (MIUR); CETEMPS Center of Excellence; Ministero degli Affari
Esteri (MAE); México – Consejo Nacional de Ciencia y Tecnología (CONACYT) No.
167733; Universidad Nacional Autónoma de México (UNAM); PAPIIT DGAPA-UNAM; The
Netherlands – Ministry of Education, Culture and Science; Netherlands
Organisation for Scientific Research (NWO); Dutch national e-infrastructure
with the support of SURF Cooperative; Poland -Ministry of Science and Higher
Education, grant No. DIR/WK/2018/11; National Science Centre, Grants No.
2013/08/M/ST9/00322, No. 2016/23/B/ST9/01635 and No. HARMONIA
5–2013/10/M/ST9/00062, UMO-2016/22/M/ST9/00198; Portugal – Portuguese national
funds and FEDER funds within Programa Operacional Factores de Competitividade
through Fundação para a Ciência e a Tecnologia (COMPETE); Romania – Romanian
Ministry of Education and Research, the Program Nucleu within MCI
(PN19150201/16N/2019 and PN19060102) and project PN-
III-P1-1.2-PCCDI-2017-0839/19PCCDI/2018 within PNCDI III; Slovenia – Slovenian
Research Agency, grants P1-0031, P1-0385, I0-0033, N1-0111; Spain – Ministerio
de Economía, Industria y Competitividad (FPA2017-85114-P and
PID2019-104676GB-C32, Xunta de Galicia (ED431C 2017/07), Junta de Andalucía
(SOMM17/6104/UGR, P18-FR-4314) Feder Funds, RENATA Red Nacional Temática de
Astropartículas (FPA2015-68783-REDT) and María de Maeztu Unit of Excellence
(MDM-2016-0692); USA – Department of Energy, Contracts No. DE-AC02-07CH11359,
No. DE-FR02-04ER41300, No. DE-FG02-99ER41107 and No. DE-SC0011689; National
Science Foundation, Grant No. 0450696; The Grainger Foundation; Marie Curie-
IRSES/EPLANET; European Particle Physics Latin American Network; and UNESCO.

## References

  * [1] Pierre Auger collaboration, A. Aab et al., _The Pierre Auger Cosmic Ray Observatory_ , _Nucl. Instrum. Meth. A_ 798 (2015) 172–213, [1502.01323].
  * [2] Pierre Auger collaboration, A. Etchegoyen, _AMIGA, Auger Muons and Infill for the Ground Array_ , in _30th International Cosmic Ray Conference_ , vol. 5, pp. 1191–1194, 7, 2007.  0710.1646.
  * [3] Pierre Auger collaboration, A. Aab et al., _The Pierre Auger Observatory Upgrade - Preliminary Design Report_ , 1604.03637.
  * [4] Pierre Auger collaboration, A. Aab et al., _Prototype muon detectors for the AMIGA component of the Pierre Auger Observatory_ , _JINST_ 11 (2016) P02012, [1605.01625].
  * [5] Pierre Auger collaboration, A. Aab et al., _Muon counting using silicon photomultipliers in the AMIGA detector of the Pierre Auger Observatory_ , _JINST_ 12 (2017) P03002, [1703.06193].
  * [6] Pierre Auger collaboration, A. Aab et al., _Reconstruction of Events Recorded with the Surface Detector of the Pierre Auger Observatory_ , _JINST_ 15 (2020) P10021, [2007.09035].
  * [7] D. Ravignani and A. Supanitsky, _A new method for reconstructing the muon lateral distribution with an array of segmented counters_ , _Astropart. Phys._ 65 (2015) 1–10, [1411.7649].
  * [8] EAS-MSU, IceCube, KASCADE Grande, NEVOD-DECOR, Pierre Auger, SUGAR, Telescope Array collaboration, L. Cazon, _Working Group Report on the Combined Analysis of Muon Density Measurements from Eight Air Shower Experiments_ , _PoS_ ICRC2019 (2020) 214, [2001.07508].
  * [9] Pierre Auger collaboration, A. Aab et al., _Direct measurement of the muonic content of extensive air showers between $\mathbf{2\times 10^{17}}$ and $\mathbf{2\times 10^{18}}\leavevmode\nobreak\ $eV at the Pierre Auger Observatory_, _Eur. Phys. J. C_ 80 (2020) 751.
  * [10] Pierre Auger collaboration, A. Aab et al., _Design, upgrade and characterization of the silicon photomultiplier front-end for the AMIGA detector at the Pierre Auger Observatory_ , 2011.06633.
  * [11] Pierre Auger collaboration, J. Abraham et al., _Trigger and Aperture of the Surface Detector Array of the Pierre Auger Observatory_ , _Nucl. Instrum. Meth. A_ 613 (2010) 29–39, [1111.6764].
  * [12] A. Supanitsky, A. Etchegoyen, G. Medina-Tanco, I. Allekotte, M. Berisso and M. C. Medina, _Underground Muon Counters as a Tool for Composition Analyses_ , _Astropart. Phys._ 29 (2008) 461–470, [0804.1068].
  * [13] S. Gundacker and A. Heering, _The silicon photomultiplier: fundamentals and applications of a modern solid-state photon detector_ , _Physics in Medicine & Biology_ 65 (aug, 2020) 17TR01.
  * [14] M. Ramilli, _Characterization of sipm: Temperature dependencies_ , in _2008 IEEE Nuclear Science Symposium Conference Record_ , pp. 2467–2470, 2008\.  DOI.
  * [15] Pierre Auger collaboration, A. M. Botti, _The AMIGA underground muon detector of the Pierre Auger Observatory - performance and event reconstruction_ , _PoS_ ICRC2019 (2020) 202.
  * [16] Pierre Auger collaboration, X. Bertou et al., _Calibration of the surface array of the Pierre Auger Observatory_ , _Nucl. Instrum. Meth. A_ 568 (2006) 839–846.
  * [17] S. Avvakumov et al., _Spontaneous light emission from fibers in MINOS_ , _Nucl. Instrum. Meth. A_ 545 (2005) 145–155.

$\bullet$

## The Pierre Auger Collaboration

A. Aab80, P. Abreu72, M. Aglietta52,50, J.M. Albury12, I. Allekotte1, A.
Almela8,11, J. Alvarez-Muñiz79, R. Alves Batista80, G.A. Anastasi61,50, L.
Anchordoqui87, B. Andrada8, S. Andringa72, C. Aramo48, P.R. Araújo Ferreira40,
J. C. Arteaga Velázquez66, H. Asorey8, P. Assis72, G. Avila10, A.M. Badescu75,
A. Bakalova30, A. Balaceanu73, F. Barbato43,44, R.J. Barreira Luz72, K.H.
Becker36, J.A. Bellido12,68, C. Berat34, M.E. Bertaina61,50, X. Bertou1, P.L.
Biermannb, T. Bister40, J. Biteau35, J. Blazek30, C. Bleve34, M. Boháčová30,
D. Boncioli55,44, C. Bonifazi24, L. Bonneau Arbeletche19, N. Borodai69, A.M.
Botti8, J. Brackf, T. Bretz40, P.G. Brichetto Orchera8, F.L. Briechle40, P.
Buchholz42, A. Bueno78, S. Buitink14, M. Buscemi45, K.S. Caballero-Mora65, L.
Caccianiga57,47, F. Canfora80,82, I. Caracas36, J.M. Carceller78, R.
Caruso56,45, A. Castellina52,50, F. Catalani17, G. Cataldi46, L. Cazon72, M.
Cerda9, J.A. Chinellato20, K. Choi13, J. Chudoba30, L. Chytka31, R.W. Clay12,
A.C. Cobos Cerutti7, R. Colalillo58,48, A. Coleman93, M.R. Coluccia46, R.
Conceição72, A. Condorelli43,44, G. Consolati47,53, F. Contreras10, F.
Convenga54,46, D. Correia dos Santos26, C.E. Covault85, S. Dasso5,3, K.
Daumiller39, B.R. Dawson12, J.A. Day12, R.M. de Almeida26, J. de Jesús8,39,
S.J. de Jong80,82, G. De Mauro80,82, J.R.T. de Mello Neto24,25, I. De
Mitri43,44, J. de Oliveira26, D. de Oliveira Franco20, F. de Palma54,46, V. de
Souza18, E. De Vito54,46, M. del Río10, O. Deligny32, A. Di Matteo50, C.
Dobrigkeit20, J.C. D’Olivo67, R.C. dos Anjos23, M.T. Dova4, J. Ebr30, R.
Engel37,39, I. Epicoco54,46, M. Erdmann40, C.O. Escobara, A. Etchegoyen8,11,
H. Falcke80,83,82, J. Farmer92, G. Farrar90, A.C. Fauth20, N. Fazzinid, F.
Feldbusch38, F. Fenu52,50, B. Fick89, J.M. Figueira8, A. Filipčič77,76, T.
Fodran80, M.M. Freire6, T. Fujii92,g, A. Fuster8,11, C. Galea80, C.
Galelli57,47, B. García7, A.L. Garcia Vegas40, H. Gemmeke38, F. Gesualdi8,39,
A. Gherghel-Lascu73, P.L. Ghia32, U. Giaccari80, M. Giammarchi47, M. Giller70,
J. Glombitza40, F. Gobbi9, F. Gollan8, G. Golup1, M. Gómez Berisso1, P.F.
Gómez Vitale10, J.P. Gongora10, J.M. González1, N. González13, I. Goos1,39, D.
Góra69, A. Gorgi52,50, M. Gottowik36, T.D. Grubb12, F. Guarino58,48, G.P.
Guedes21, E. Guido50,61, S. Hahn39,8, P. Hamal30, M.R. Hampel8, P. Hansen4, D.
Harari1, V.M. Harvey12, A. Haungs39, T. Hebbeker40, D. Heck39, G.C. Hill12, C.
Hojvatd, J.R. Hörandel80,82, P. Horvath31, M. Hrabovský31, T. Huege39,14, J.
Hulsman8,39, A. Insolia56,45, P.G. Isar74, P. Janecek30, J.A. Johnsen86, J.
Jurysek30, A. Kääpä36, K.H. Kampert36, B. Keilhauer39, J. Kemp40, H.O.
Klages39, M. Kleifges38, J. Kleinfeller9, M. Köpke37, N. Kunka38, B.L. Lago16,
R.G. Lang18, N. Langner40, M.A. Leigui de Oliveira22, V. Lenok39, A.
Letessier-Selvon33, I. Lhenry-Yvon32, D. Lo Presti56,45, L. Lopes72, R.
López62, L. Lu94, Q. Luce37, A. Lucero8, J.P. Lundquist76, A. Machado
Payeras20, G. Mancarella54,46, D. Mandat30, B.C. Manning12, J. Manshanden41,
P. Mantschd, S. Marafico32, A.G. Mariazzi4, I.C. Mariş13, G. Marsella59,45, D.
Martello54,46, H. Martinez18, O. Martínez Bravo62, M. Mastrodicasa55,44, H.J.
Mathes39, J. Matthews88, G. Matthiae60,49, E. Mayotte36, P.O. Mazurd, G.
Medina-Tanco67, D. Melo8, A. Menshikov38, K.-D. Merenda86, S. Michal31, M.I.
Micheletti6, L. Miramonti57,47, S. Mollerach1, F. Montanet34, C. Morello52,50,
M. Mostafá91, A.L. Müller8,39, M.A. Muller20, K. Mulrey14, R. Mussa50, M.
Muzio90, W.M. Namasaka36, A. Nasr-Esfahani36, L. Nellen67, M. Niculescu-
Oglinzanu73, M. Niechciol42, D. Nitz89,e, D. Nosek29, V. Novotny29, L.
Nožka31, A Nucita54,46, L.A. Núñez28, M. Palatka30, J. Pallotta2, P.
Papenbreer36, G. Parente79, A. Parra62, M. Pech30, F. Pedreira79, J. Pȩkala69,
R. Pelayo64, J. Peña-Rodriguez28, E.E. Pereira Martins37, J. Perez Armand19,
C. Pérez Bertolli8,39, M. Perlin8,39, L. Perrone54,46, S. Petrera43,44, T.
Pierog39, M. Pimenta72, V. Pirronello56,45, M. Platino8, B. Pont80, M.
Pothast82,80, P. Privitera92, M. Prouza30, A. Puyleart89, S. Querchfeld36, J.
Rautenberg36, D. Ravignani8, M. Reininghaus39,8, J. Ridky30, F. Riehn72, M.
Risse42, V. Rizi55,44, W. Rodrigues de Carvalho19, J. Rodriguez Rojo10, M.J.
Roncoroni8, M. Roth39, E. Roulet1, A.C. Rovero5, P. Ruehl42, S.J. Saffi12, A.
Saftoiu73, F. Salamida55,44, H. Salazar62, G. Salina49, J.D. Sanabria Gomez28,
F. Sánchez8, E.M. Santos19, E. Santos30, F. Sarazin86, R. Sarmento72, C.
Sarmiento-Cano8, R. Sato10, P. Savina54,46,32, C.M. Schäfer39, V. Scherini46,
H. Schieler39, M. Schimassek37,8, M. Schimp36, F. Schlüter39,8, D. Schmidt37,
O. Scholten81,14, P. Schovánek30, F.G. Schröder93,39, S. Schröder36, J.
Schulte40, S.J. Sciutto4, M. Scornavacche8,39, A. Segreto51,45, S. Sehgal36,
R.C. Shellard15, G. Sigl41, G. Silli8,39, O. Sima73,h, R. Šmída92, P.
Sommers91, J.F. Soriano87, J. Souchard34, R. Squartini9, M. Stadelmaier39,8,
D. Stanca73, S. Stanič76, J. Stasielak69, P. Stassi34, A. Streich37,8, M.
Suárez-Durán28, T. Sudholz12, T. Suomijärvi35, A.D. Supanitsky8, J. Šupík31,
Z. Szadkowski71, A. Taboada37, A. Tapia27, C. Taricco61,50, C.
Timmermans82,80, O. Tkachenko39, P. Tobiska30, C.J. Todero Peixoto17, B.
Tomé72, A. Travaini9, P. Travnicek30, C. Trimarelli55,44, M. Trini76, M.
Tueros4, R. Ulrich39, M. Unger39, L. Vaclavek31, M. Vacula31, J.F. Valdés
Galicia67, L. Valore58,48, E. Varela62, V. Varma K.C.8,39, A. Vásquez-
Ramírez28, D. Veberič39, C. Ventura25, I.D. Vergara Quispe4, V. Verzi49, J.
Vicha30, J. Vink84, S. Vorobiov76, H. Wahlberg4, C. Watanabe24, A.A. Watsonc,
M. Weber38, A. Weindl39, L. Wiencke86, H. Wilczyński69, T. Winchen14, M.
Wirtz40, D. Wittkowski36, B. Wundheiler8, A. Yushkov30, O. Zapparrata13, E.
Zas79, D. Zavrtanik76,77, M. Zavrtanik77,76, L. Zehrer76, A. Zepeda63

1

Centro Atómico Bariloche and Instituto Balseiro (CNEA-UNCuyo-CONICET), San
Carlos de Bariloche, Argentina

2

Centro de Investigaciones en Láseres y Aplicaciones, CITEDEF and CONICET,
Villa Martelli, Argentina

3

Departamento de Física and Departamento de Ciencias de la Atmósfera y los
Océanos, FCEyN, Universidad de Buenos Aires and CONICET, Buenos Aires,
Argentina

4

IFLP, Universidad Nacional de La Plata and CONICET, La Plata, Argentina

5

Instituto de Astronomía y Física del Espacio (IAFE, CONICET-UBA), Buenos
Aires, Argentina

6

Instituto de Física de Rosario (IFIR) – CONICET/U.N.R. and Facultad de
Ciencias Bioquímicas y Farmacéuticas U.N.R., Rosario, Argentina

7

Instituto de Tecnologías en Detección y Astropartículas (CNEA, CONICET,
UNSAM), and Universidad Tecnológica Nacional – Facultad Regional Mendoza
(CONICET/CNEA), Mendoza, Argentina

8

Instituto de Tecnologías en Detección y Astropartículas (CNEA, CONICET,
UNSAM), Buenos Aires, Argentina

9

Observatorio Pierre Auger, Malargüe, Argentina

10

Observatorio Pierre Auger and Comisión Nacional de Energía Atómica, Malargüe,
Argentina

11

Universidad Tecnológica Nacional – Facultad Regional Buenos Aires, Buenos
Aires, Argentina

12

University of Adelaide, Adelaide, S.A., Australia

13

Université Libre de Bruxelles (ULB), Brussels, Belgium

14

Vrije Universiteit Brussels, Brussels, Belgium

15

Centro Brasileiro de Pesquisas Fisicas, Rio de Janeiro, RJ, Brazil

16

Centro Federal de Educação Tecnológica Celso Suckow da Fonseca, Nova Friburgo,
Brazil

17

Universidade de São Paulo, Escola de Engenharia de Lorena, Lorena, SP, Brazil

18

Universidade de São Paulo, Instituto de Física de São Carlos, São Carlos, SP,
Brazil

19

Universidade de São Paulo, Instituto de Física, São Paulo, SP, Brazil

20

Universidade Estadual de Campinas, IFGW, Campinas, SP, Brazil

21

Universidade Estadual de Feira de Santana, Feira de Santana, Brazil

22

Universidade Federal do ABC, Santo André, SP, Brazil

23

Universidade Federal do Paraná, Setor Palotina, Palotina, Brazil

24

Universidade Federal do Rio de Janeiro, Instituto de Física, Rio de Janeiro,
RJ, Brazil

25

Universidade Federal do Rio de Janeiro (UFRJ), Observatório do Valongo, Rio de
Janeiro, RJ, Brazil

26

Universidade Federal Fluminense, EEIMVR, Volta Redonda, RJ, Brazil

27

Universidad de Medellín, Medellín, Colombia

28

Universidad Industrial de Santander, Bucaramanga, Colombia

29

Charles University, Faculty of Mathematics and Physics, Institute of Particle
and Nuclear Physics, Prague, Czech Republic

30

Institute of Physics of the Czech Academy of Sciences, Prague, Czech Republic

31

Palacky University, RCPTM, Olomouc, Czech Republic

32

CNRS/IN2P3, IJCLab, Université Paris-Saclay, Orsay, France

33

Laboratoire de Physique Nucléaire et de Hautes Energies (LPNHE), Sorbonne
Université, Université de Paris, CNRS-IN2P3, Paris, France

34

Univ. Grenoble Alpes, CNRS, Grenoble Institute of Engineering Univ. Grenoble
Alpes, LPSC-IN2P3, 38000 Grenoble, France

35

Université Paris-Saclay, CNRS/IN2P3, IJCLab, Orsay, France

36

Bergische Universität Wuppertal, Department of Physics, Wuppertal, Germany

37

Karlsruhe Institute of Technology, Institute for Experimental Particle Physics
(ETP), Karlsruhe, Germany

38

Karlsruhe Institute of Technology, Institut für Prozessdatenverarbeitung und
Elektronik, Karlsruhe, Germany

39

Karlsruhe Institute of Technology, Institute for Astroparticle Physics,
Karlsruhe, Germany

40

RWTH Aachen University, III. Physikalisches Institut A, Aachen, Germany

41

Universität Hamburg, II. Institut für Theoretische Physik, Hamburg, Germany

42

Universität Siegen, Department Physik – Experimentelle Teilchenphysik, Siegen,
Germany

43

Gran Sasso Science Institute, L’Aquila, Italy

44

INFN Laboratori Nazionali del Gran Sasso, Assergi (L’Aquila), Italy

45

INFN, Sezione di Catania, Catania, Italy

46

INFN, Sezione di Lecce, Lecce, Italy

47

INFN, Sezione di Milano, Milano, Italy

48

INFN, Sezione di Napoli, Napoli, Italy

49

INFN, Sezione di Roma “Tor Vergata”, Roma, Italy

50

INFN, Sezione di Torino, Torino, Italy

51

Istituto di Astrofisica Spaziale e Fisica Cosmica di Palermo (INAF), Palermo,
Italy

52

Osservatorio Astrofisico di Torino (INAF), Torino, Italy

53

Politecnico di Milano, Dipartimento di Scienze e Tecnologie Aerospaziali ,
Milano, Italy

54

Università del Salento, Dipartimento di Matematica e Fisica “E. De Giorgi”,
Lecce, Italy

55

Università dell’Aquila, Dipartimento di Scienze Fisiche e Chimiche, L’Aquila,
Italy

56

Università di Catania, Dipartimento di Fisica e Astronomia, Catania, Italy

57

Università di Milano, Dipartimento di Fisica, Milano, Italy

58

Università di Napoli “Federico II”, Dipartimento di Fisica “Ettore Pancini”,
Napoli, Italy

59

Università di Palermo, Dipartimento di Fisica e Chimica ”E. Segrè”, Palermo,
Italy

60

Università di Roma “Tor Vergata”, Dipartimento di Fisica, Roma, Italy

61

Università Torino, Dipartimento di Fisica, Torino, Italy

62

Benemérita Universidad Autónoma de Puebla, Puebla, México

63

Centro de Investigación y de Estudios Avanzados del IPN (CINVESTAV), México,
D.F., México

64

Unidad Profesional Interdisciplinaria en Ingeniería y Tecnologías Avanzadas
del Instituto Politécnico Nacional (UPIITA-IPN), México, D.F., México

65

Universidad Autónoma de Chiapas, Tuxtla Gutiérrez, Chiapas, México

66

Universidad Michoacana de San Nicolás de Hidalgo, Morelia, Michoacán, México

67

Universidad Nacional Autónoma de México, México, D.F., México

68

Universidad Nacional de San Agustin de Arequipa, Facultad de Ciencias
Naturales y Formales, Arequipa, Peru

69

Institute of Nuclear Physics PAN, Krakow, Poland

70

University of Łódź, Faculty of Astrophysics, Łódź, Poland

71

University of Łódź, Faculty of High-Energy Astrophysics,Łódź, Poland

72

Laboratório de Instrumentação e Física Experimental de Partículas – LIP and
Instituto Superior Técnico – IST, Universidade de Lisboa – UL, Lisboa,
Portugal

73

“Horia Hulubei” National Institute for Physics and Nuclear Engineering,
Bucharest-Magurele, Romania

74

Institute of Space Science, Bucharest-Magurele, Romania

75

University Politehnica of Bucharest, Bucharest, Romania

76

Center for Astrophysics and Cosmology (CAC), University of Nova Gorica, Nova
Gorica, Slovenia

77

Experimental Particle Physics Department, J. Stefan Institute, Ljubljana,
Slovenia

78

Universidad de Granada and C.A.F.P.E., Granada, Spain

79

Instituto Galego de Física de Altas Enerxías (IGFAE), Universidade de Santiago
de Compostela, Santiago de Compostela, Spain

80

IMAPP, Radboud University Nijmegen, Nijmegen, The Netherlands

81

KVI – Center for Advanced Radiation Technology, University of Groningen,
Groningen, The Netherlands

82

Nationaal Instituut voor Kernfysica en Hoge Energie Fysica (NIKHEF), Science
Park, Amsterdam, The Netherlands

83

Stichting Astronomisch Onderzoek in Nederland (ASTRON), Dwingeloo, The
Netherlands

84

Universiteit van Amsterdam, Faculty of Science, Amsterdam, The Netherlands

85

Case Western Reserve University, Cleveland, OH, USA

86

Colorado School of Mines, Golden, CO, USA

87

Department of Physics and Astronomy, Lehman College, City University of New
York, Bronx, NY, USA

88

Louisiana State University, Baton Rouge, LA, USA

89

Michigan Technological University, Houghton, MI, USA

90

New York University, New York, NY, USA

91

Pennsylvania State University, University Park, PA, USA

92

University of Chicago, Enrico Fermi Institute, Chicago, IL, USA

93

University of Delaware, Department of Physics and Astronomy, Bartol Research
Institute, Newark, DE, USA

94

University of Wisconsin-Madison, Department of Physics, Madison, WI, USA

—–

a

Fermi National Accelerator Laboratory, USA

b

Max-Planck-Institut für Radioastronomie, Bonn, Germany

c

School of Physics and Astronomy, University of Leeds, Leeds, United Kingdom

d

Fermi National Accelerator Laboratory, Fermilab, Batavia, IL, USA

e

also at Karlsruhe Institute of Technology, Karlsruhe, Germany

f

Colorado State University, Fort Collins, CO, USA

g

now at Hakubi Center for Advanced Research and Graduate School of Science,
Kyoto University, Kyoto, Japan

h

also at University of Bucharest, Physics Department, Bucharest, Romania


# Temporal aggregation of audio-visual modalities for emotion recognition

Andreea Birhala∗, Catalin Nicolae Ristea∗, Anamaria Radoi∗, and Liviu Cristian
Dutu† This work has been partially supported by the Ministry of Innovation and
Research, UEFISCDI, project SPIA-VA, agreement 2SOL/2017, grant PN-
III-P2-2.1-SOL-2016-02-0002. ∗Research Center CAMPUS, University Politehnica
of Bucharest, Bucharest, Romania
†Xperi Corporation, Bucharest, Romania
Email<EMAIL_ADDRESS>
###### Abstract

Emotion recognition has a pivotal role in affective computing and in human-
computer interaction. The current technological developments lead to increased
possibilities of collecting data about the emotional state of a person. In
general, human perception regarding the emotion transmitted by a subject is
based on vocal and visual information collected in the first seconds of
interaction with the subject. As a consequence, the integration of verbal
(i.e., speech) and non-verbal (i.e., image) information seems to be the
preferred choice in most of the current approaches towards emotion
recognition. In this paper, we propose a multimodal fusion technique for
emotion recognition based on combining audio-visual modalities from a temporal
window with different temporal offsets for each modality. We show that our
proposed method outperforms other methods from the literature and human
accuracy rating. The experiments are conducted over the open-access multimodal
dataset CREMA-D.

###### Index Terms:

asynchronous data; convolutional neural network; data fusion; emotion
recognition; multimodal information; spectrogram.

##  I Introduction

Automatic detection of human emotions has become an important area of research
due to the technological development that occurred in the human-computer
interaction domain (e.g., social robots [1, 2], monitoring systems for car
drivers' condition [3]). In order to increase the accuracy of emotion
recognition systems, most of the currently developed methods incorporate
multimodal information (e.g., facial and speech features) [4, 5, 6, 7]. Facial
expressions represent one of the most important modes of communication through
which people express their emotions and intentions. In addition to facial
expressions, people also express their feelings through speech; e.g, speech
inflection, vocal intensity are characteristics that contain information about
the emotional state of a subject.

Each person is unique and can express emotions in their own characteristic
way, depending on their culture, age, gender or previous life experiences [8].
Nevertheless, there are common characteristics that can be exploited in order
to obtain an accurate classification system. In general, most recognition
systems consider only 6 types of emotions (e.g., anger, happiness, surprise,
disgust, contempt, anxiety) [9]. According to the Facial Action Coding System
(FACS), each human emotion can be described through a combination of several
Facial Action Units (FAUs) [10]. More precisely, the FACS refer to a combined
set of facial muscle movements that correspond to a displayed emotion. The
basic element in this coding system is Action Unit (AU) and each AU is related
to the contraction of one or more facial muscles.

Emotion recognition systems using only visual information (i.e., video frames)
can be mainly classified into static and dynamic methods depending on the
feature representations. In static-based methods, the features are encoded
with spatial information from singular frames without taking into
consideration the temporal extent, whilst dynamic-based methods consider the
temporal relation between continuous frames from the input sequence. In the
case of static-based methods, state-of-the-art deep neural networks
architectures (e.g., VGG [11], ResNet [12]) have been proposed for feature
extraction, whilst the classification into emotion categories is performed
using a Support Vector Machine (SVM) module [13].

Due to the increased interest in developing real-world scenarios datasets and
also the increased computer processing capabilities, recent approaches are
based on deep learning techniques that are able to extract both facial and
audio discriminant information. In a recent paper [4], we have shown that not
very deep convolutional neural network (CNN) architectures are able to extract
meaningful information regarding emotion categories from both video frames and
spectrograms of audio signals. By combining the audio-video information, we
managed to achieve an increase of almost 7 % compared to the case when only
video data is considered.

Considering the behavioral differences between people and the diverse modes of
communicating their feelings, methods addressing person-specific affective
understanding have been also developed. In [14], Grow-When-Required Networks
and personalized affective memories are used to learn individualized aspects
of emotions. However, the complexity of the proposed model limits the real-
time usage of the proposed solution.

In order to include temporal dynamic characteristics between video frames,
Beard et. al proposed a recurrent multi-attention (RMA) mechanism with shared
external memory that is updated over multiple iterations of analysis [6]. This
approach allows relevant memories to persists over multiple hops. The method
achieved a maximum accuracy of 65 % on the CREMA-D dataset, comparable to the
human rating accuracy reported for this dataset (i.e., 63.6 % [15]).

In an attempt to exploit the complementary information brought by diverse
modalities (i.e., audio and video), a Multimodal Emotion Recognition Metric
Learning (MERML) was defined in [16]. The learned metric was further used by
SVM with Radial Basis Function (RBF) kernel.

In this paper, we propose a novel multimodal architecture that combines visual
and audio features extracted from random selections of analysis windows within
individual temporal segments of the input video. Thus, the temporal
aggregation of audio and video allows for asynchronous inputs of the two
considered modalities. We tested our solution on the CREMA-D [15], a widely
used audio-visual dataset in the multimodal emotion recognition field.

The rest of the paper is organized as follows. Section II introduces the
multimodal model architecture. Section III describes the dataset used for
experiments, whereas section IV presents the experimental results. Finally,
section V concludes the paper.

##  II Proposed Approach

Inspired by the solution for action recognition presented in [17] and by how
the human brain processes audio-visual information, we propose a temporal
aggregation mechanism in order to combine modalities within a range of
temporal offsets. In this mechanism we explore the fusion between audio and
visual inputs within a temporal window, that will allow the model to be
trained with asynchronous inputs from both modalities. Our proposed temporal
aggregation mechanism is shown in Fig 1. In the following, we denote by
$r_{v}$ the sampling rate of the video sequence and by $r_{a}$ (different from
$r_{v}$) the sampling rate of the audio signal.

The input video sequence is divided into $N$ temporal segments of equal
length. For each temporal segment, we randomly select a video frame $j$ and we
randomly choose the center $c$ of the audio signal window between
$[j/r_{v}-b,j/r_{v}+b]$ seconds. The audio signal used in the analysis of the
current temporal segment is considered between $[c-d/2,c+d/2]$ seconds (i.e.,
samples between $\lceil r_{a}(c-d/2)\rceil$ and $\lfloor
r_{a}(c+d/2)\rfloor$). Further, the video frame and the spectrogram of the
audio signal are fed into an audio-visual network corresponding to the current
temporal segment. It is worth noting that $N$ independent audio-visual
networks are needed to build the entire emotion recognition architecture.

Figure 1: Our proposed temporal aggregation mechanism. FC label is a
6-dimensional vector of class probabilities obtained after passing the audio-
visual features through the Fully Connected (FC) layer.

Following a similar approach to the method that has been recently proposed by
the authors in [4], the core audio-visual network is composed of a sequence of
convolutional blocks, which extracts the audio and visual features. After the
concatenation of the audio and video feature vectors, the resulting feature
vector is considered as input for a Fully Connected (FC) layer, followed by a
SoftMax activation function which yields the class probabilities for the
considered emotion categories. The solution proposed in [4] achieved
approximately the human rating performance accuracy at low computational
costs. However, in order to accelerate the training process and to increase
the stability of the core audio-visual network, we inserted a Batch
Normalization layer after each convolutional layer [18]. The core audio-visual
network architecture, which processes asynchronous multimodal information, is
shown in Fig. 2.

After aggregating the emotion class probabilities for all the temporal
segments composing the video, the class label for the entire video is the one
for which the maximum score is achieved. As shown in Fig. 1, the final score
is obtaining by summing up the emotion class probabilities over all the
temporal segments.

Figure 2: Our proposed audio-visual network used to retrieve emotion class
probabilities for each temporal segment of the analysed video.

We mention that the frames of the original videos are pre-processed using the
MTCNN algorithm [19], whose aim is to perform face detection and to remove the
unnecessary information (i.e., background) with respect to the emotion
recognition task.

##  III Database

Over the past years, several databases for the emotion recognition task have
been proposed and the research in the affective computing domain focused on
mixing different sources of information to achieve better performance. The
CREMA-D multimodal database was published in 2015 [15] and contains 7442 clips
of 91 actors (48 male and 43 female) with different ethnic backgrounds. The
actors were asked to convey particular emotions while producing, with
different intonations, 12 particular sentences that evoke the target emotions.
Six labels have been used to discriminate among different emotion categories
(i.e., neutral, happy, anger, disgust, fear, sad) with four different
intensity levels (i.e., low, medium, high, unspecified). The labels
corresponding to each recording were collected using crowd-sourcing. More
precisely, 2443 participants were asked to label the perceived emotion and its
intensity. The human accuracy achieved for this task was, on average,
$63.6\%$. It is worth mentioning that human training was achieved through
participants' previous experience.

##  IV Experiments

(a) Loss

(b) Accuracy

Figure 3: Performance over the training and validation sets. Figure 4:
Confusion matrix for best performance. Figure 5: Variation of accuracy values
with respect to the number of segments considered in the temporal aggregation.

In this section, the results achieved on the CREMA-D data set, as well as the
experimental setup used in our approach, are provided. For all our
experiments, we use an user-independent 10-fold cross validation technique to
split the dataset into train and validation subsets. We divide the dataset
into 10 different folds (i.e., none of the actors is introduced in more than
one fold for generalization reasons) and compute the mean accuracy over all
the folds.

Various evaluation metrics are used to assess the performance of the proposed
solution for emotion recognition, namely mean overall accuracy, loss values
and confusion matrix. These performance measures are shown in Fig. 3 and Fig.
4 and are achieved for a temporal aggregation of 10 segments extracted from
the video. As shown in Fig. 5, the overall accuracy increases with the number
of segments considered. However, the time required for training a model and
the inference time increase almost linearly with the number of segments, i.e.,
from 3 hours for a model with 3 segments to approximately 5.5 hours for a
model with 10 segments. Moreover, the accuracy does not increase substantially
for more than 8 segments.

Furthermore, the overall accuracy is compared with the performances achieved
by other methods from the literature (e.g., CNN-based approach [4], RMA [6]).
It is worth mentioning that the experiments using the approach proposed in [4]
followed the same 10-fold cross validation technique.

The input frames were cut at $224\times 224$ pixels around the detected faces
[19], whereas the spectrograms were resized to $192\times 120$. The length of
the audio signal $d$ was set to 1.28 seconds, whereas the offset $b$ was set
to 0.01 seconds. In order to train the models, we used the cross entropy loss
and the stochastic gradient descent optimization method with 0.9 momentum. For
each epoch, the learning rate was initially set to 1e-3 and decayed by a
factor of 10 every 50 training steps. The batch size was set to 16.

Our solution outperformed the baseline approach proposed in [4] by $12.6\%$,
human accuracy rating by $4.8\%$ and also methods based on recurrent multi-
attention [6]. The proposed method of early combining features and temporal
aggregation of partial results leads to a better performance, which suggests
that combining information coming from different sources in an asynchronous
manner proves to be beneficial in the process of emotion understanding.

We mention that all the experiments were conducted over an Intel Xeon
E5-1680v3, 8 cores @3.2 GHz, equipped with NVIDIA Quadro M4000 GPU with 8 GB
RAM.

TABLE I: Average accuracy rate on CREMA-D Method | Accuracy [%]
---|---
Human accuracy [15] | 63.6
CNN-based approach [4] | 55.8
RMA [6] | 65.0
MERML [16] | 66.5
Proposed method | 68.4

##  V Conclusion

In this paper, we proposed a new method to incorporate multimodal information
to discriminate among categories of emotions. The methodology benefits from
combining the audio and visual information in an asynchronous manner which
allows a certain degree of flexibility between the analysis of the two
modalities. Using a simple audio-visual neural network as core architecture
for predicting the emotional states, the temporal aggregation of the
multimodal information from various segments leads to a substantial increase
in the accuracy of the recognition system and outperforms other approaches
from the literature.

## References

  * [1] F. Cavallo, F. Semeraro, L. Fiorini, G. Magyar, P. Sinčák, and P. Dario, ``Emotion modelling for social robotics applications: a review,'' _Journal of Bionic Engineering_ , vol. 15, no. 2, pp. 185–203, 2018.
  * [2] L. Grama and C. Rusu, ``Extending assisted audio capabilities of tiago service robot,'' in _2019 International Conference on Speech Technology and Human-Computer Dialogue (SpeD)_ , 2019, pp. 1–8.
  * [3] C. D. Katsis, N. Katertsidis, G. Ganiatsas, and D. I. Fotiadis, ``Toward emotion recognition in car-racing drivers: A biosignal processing approach,'' _IEEE Trans. on Systems, Man, and Cybernetics - Part A: Systems and Humans_ , vol. 38, no. 3, pp. 502–512, May 2008.
  * [4] N.-C. Ristea, L. C. Duţu, and A. Radoi, ``Emotion recognition system from speech and visual information based on convolutional neural networks,'' in _2019 International Conference on Speech Technology and Human-Computer Dialogue (SpeD)_ , 2019, pp. 1–6.
  * [5] M. Valstar, B. Schuller, K. Smith, F. Eyben, B. Jiang, S. Bilakhia, S. Schnieder, R. Cowie, and M. Pantic, ``Avec 2013: The continuous audio/visual emotion and depression recognition challenge,'' in _Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge_ , ser. AVEC ’13, 2013, p. 3–10.
  * [6] R. Beard, R. Das, R. W. M. Ng, P. G. K. Gopalakrishnan, L. Eerens, P. Swietojanski, and O. Miksik, ``Multi-modal sequence fusion via recursive attention for emotion recognition,'' in _Proceedings of the 22nd Conference on Computational Natural Language Learning_. Brussels, Belgium: Association for Computational Linguistics, Oct. 2018, pp. 251–259.
  * [7] E. Ghaleb, M. Popa, and S. Asteriadis, ``Multimodal and temporal perception of audio-visual cues for emotion recognition,'' in _2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)_ , Sep. 2019, pp. 552–558.
  * [8] P. Tzirakis, G. Trigeorgis, M. A. Nicolaou, B. W. Schuller, and S. Zafeiriou, ``End-to-end multimodal emotion recognition using deep neural networks,'' _IEEE Journal of Selected Topics in Signal Processing_ , vol. 11, no. 8, pp. 1301–1309, Dec 2017.
  * [9] P. Ekman, ``An argument for basic emotions,'' _Cognition & emotion_, vol. 6, no. 3-4, pp. 169–200, 1992.
  * [10] Y. . Tian, T. Kanade, and J. F. Cohn, ``Recognizing action units for facial expression analysis,'' _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 23, no. 2, pp. 97–115, Feb 2001.
  * [11] K. Simonyan and A. Zisserman, ``Very deep convolutional networks for large-scale image recognition,'' _arXiv preprint arXiv:1409.1556_ , 2014.
  * [12] K. He, X. Zhang, S. Ren, and J. Sun, ``Deep residual learning for image recognition,'' in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [13] S. Bargal, E. Barsoum, C. Ferrer, and C. Zhang, ``Emotion recognition in the wild from videos using images,'' in _Proceedings of the 18th ACM International Conference on Multimodal Interaction ICMI 2016_ , 10 2016, pp. 433–436.
  * [14] P. V. A. Barros, G. I. Parisi, and S. Wermter, ``A personalized affective memory neural model for improving emotion recognition,'' _ArXiv_ , vol. abs/1904.12632, 2019.
  * [15] H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, ``Crema-d: Crowd-sourced emotional multimodal actors dataset,'' _IEEE Trans. on Affective Computing_ , vol. 5, no. 4, pp. 377–390, 2014.
  * [16] E. Ghaleb, M. Popa, and S. Asteriadis, ``Metric learning based multimodal audio-visual emotion recognition,'' _IEEE Multimedia_ , pp. 1–1, 2019.
  * [17] E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen, ``Epic-fusion: Audio-visual temporal binding for egocentric action recognition,'' in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 5492–5501.
  * [18] S. Ioffe and C. Szegedy, ``Batch normalization: Accelerating deep network training by reducing internal covariate shift,'' _arXiv preprint arXiv:1502.03167_ , 2015.
  * [19] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, ``Joint face detection and alignment using multitask cascaded convolutional networks,'' _IEEE Signal Processing Letters_ , vol. 23, no. 10, pp. 1499–1503, Oct 2016.