.gifpng.pngconvert gif:#1 png: .gif

# The DAQ system of the 12,000 Channel CMS High Granularity Calorimeter
Prototype

B. Acar  G. Adamov  C. Adloff  S. Afanasiev  N. Akchurin  B. Akgün  M.
Alhusseini  J. Alison  G. Altopp  M. Alyari  S. An  S. Anagul  I. Andreev  M.
Andrews  P. Aspell  I. A. Atakisi  O. Bach  A. Baden  G. Bakas  A. Bakshi  P.
Bargassa  D. Barney  E. Becheva  P. Behera  A. Belloni  T. Bergauer  M.
Besancon  S. Bhattacharya  S. Bhattacharya  D. Bhowmik  P. Bloch  A. Bodek  M.
Bonanomi  A. Bonnemaison  S. Bonomally  J. Borg  F. Bouyjou  D. Braga  J.
Brashear  E. Brondolin  P. Bryant  J. Bueghly  B. Bilki  B. Burkle  A. Butler-
Nalin  S. Callier  D. Calvet  X. Cao  B. Caraway  S. Caregari  L. Ceard  Y. C.
Cekmecelioglu  G. Cerminara  N. Charitonidis  R. Chatterjee  Y. M. Chen  Z.
Chen  K. y. Cheng  S. Chernichenko  H. Cheung  C. H. Chien  S. Choudhury  D.
Čoko  G. Collura  F. Couderc  I. Dumanoglu  D. Dannheim  P. Dauncey  A. David
G. Davies  E. Day  P. DeBarbaro  F. De Guio  C. de La Taille  M. De Silva  P.
Debbins  E. Delagnes  J. M. Deltoro  G. Derylo  P.G. Dias de Almeida  D. Diaz
P. Dinaucourt  J. Dittmann  M. Dragicevic  S. Dugad  V. Dutta  S. Dutta  J.
Eckdahl  T. K. Edberg  M. El Berni  S. C. Eno  Yu. Ershov  P. Everaerts  S.
Extier  F. Fahim  C. Fallon  B. A. Fontana Santos Alves  E. Frahm  G. Franzoni
J. Freeman  T. French  E. Gurpinar Guler  Y. Guler  M. Gagnan  P. Gandhi  S.
Ganjour  A. Garcia-Bellido  Z. Gecse  Y. Geerebaert  H. Gerwig  O. Gevin  W.
Gilbert  A. Gilbert  K. Gill  C. Gingu  S. Gninenko  A. Golunov  I. Golutvin
T. Gonzalez  N. Gorbounov  L. Gouskos  Y. Gu  F. Guilloux  E. Gülmez  M.
Hammer  A. Harilal  K. Hatakeyama  A. Heering  V. Hegde  U. Heintz  V. Hinger
N. Hinton  J. Hirschauer  J. Hoff  W. S. Hou  C. Isik  J. Incandela  S. Jain
H. R. Jheng  U. Joshi  O. Kara  V. Kachanov  A. Kalinin  R. Kameshwar  A.
Kaminskiy  A. Karneyeu  O. Kaya  M. Kaya  A. Khukhunaishvili  S. Kim  K. Koetz
T. Kolberg  A. Kristić  M. Krohn  K. Krüger  N. Kulagin  S. Kulis  S. Kunori
C. M. Kuo  V. Kuryatkov  S. Kyre  O. K. Köseyan  Y. Lai  K. Lamichhane  G.
Landsberg  J. Langford  M. Y. Lee  A. Levin  A. Li  B. Li  J.-H. Li  H. Liao
D. Lincoln  L. Linssen  R. Lipton  Y. Liu  A. Lobanov  R. S. Lu  I. Lysova  A.
M. Magnan  F. Magniette  A. A. Maier  A. Malakhov  I. Mandjavize  M. Mannelli
J. Mans  A. Marchioro  A. Martelli  P. Masterson  B. Meng  T. Mengke  A.
Mestvirishvili  I. Mirza  S. Moccia  I. Morrissey  T. Mudholkar  J. Musić  I.
Musienko  S. Nabili  A. Nagar  A. Nikitenko  D. Noonan  M. Noy  K. Nurdan  C.
Ochando  B. Odegard  N. Odell  Y. Onel  W. Ortez  J. Ozegović  L. Pacheco
Rodriguez  E. Paganis  D. Pagenkopf  V. Palladino  S. Pandey  F. Pantaleo  C.
Papageorgakis  I. Papakrivopoulos  J. Parshook  N. Pastika  M. Paulini  P.
Paulitsch  T. Peltola  R. Pereira Gomes  H. Perkins  P. Petiot  F. Pitters  F.
Pitters  H. Prosper  M. Prvan  I. Puljak  T. Quast  R. Quinn  M. Quinnan  K.
Rapacz  L. Raux  G. Reichenbach  M. Reinecke  M. Revering  A. Rodriguez  T.
Romanteau  A. Rose  M. Rovere  A. Roy  P. Rubinov  R. Rusack  A. E. Simsek  U.
Sozbilir  O. M. Sahin  A. Sanchez  R. Saradhy  T. Sarkar  M. A. Sarkisla  J.
B. Sauvan  I. Schmidt  M. Schmitt  E. Scott  C. Seez  F. Sefkow  S. Sharma  I.
Shein  A. Shenai  R. Shukla  E. Sicking  P. Sieberer  Y. Sirois  V. Smirnov
E. Spencer  A. Steen  J. Strait  T. Strebler  N. Strobbe  J. W. Su  E. Sukhov
L. Sun  M. Sun  C. Syal  B. Tali  U. G. Tok  A. Kayis Topaksu  C. L. Tan  I.
Tastan  T. Tatli  R. Thaus  S. Tekten  D. Thienpont  T. Pierre-Emile  E. Tiras
M. Titov  D. Tlisov  J. Troska  Z. Tsamalaidze  G. Tsipolitis  A. Tsirou  N.
Tyurin  S. Undleeb  D. Urbanski  V. Ustinov  A. Uzunian  M. van de Klundert
J. Varela  M. Velasco  M. Vicente Barreto Pinto  P. M. da Silva  T. Virdee  R.
Vizinho de Oliveira  J. Voelker  E. Voirin  Z. Wang  X. Wang  F. Wang  M.
Wayne  S. N. Webb  M. Weinberg  A. Whitbeck  D. White  R. Wickwire  J. S.
Wilson  H. Y. Wu  L. Wu  C. H Yeh  R. Yohay  G. B. Yu  S. S. Yu  D. Yu  F.
Yumiceva  A. Zacharopoulou  N. Zamiatin  A. Zarubin  S. Zenz  H. Zhang  J.
Zhang

###### Abstract

The CMS experiment at the CERN LHC will be upgraded to accommodate the 5-fold
increase in the instantaneous luminosity expected at the High-Luminosity LHC
(HL-LHC) [1]. Concomitant with this increase will be an increase in the number
of interactions in each bunch crossing and a significant increase in the total
ionising dose and fluence. One part of this upgrade is the replacement of the
current endcap calorimeters with a high granularity sampling calorimeter
equipped with silicon sensors, designed to manage the high collision rates
[2]. As part of the development of this calorimeter, a series of beam tests
have been conducted with different sampling configurations using prototype
segmented silicon detectors. In the most recent of these tests, conducted in
late 2018 at the CERN SPS, the performance of a prototype calorimeter equipped
with ${\approx}12,000\rm{~{}channels}$ of silicon sensors was studied with
beams of high-energy electrons, pions and muons. This paper describes the
custom-built scalable data acquisition system that was built with readily
available FPGA mezzanines and low-cost Raspberry PI computers.

##  1 Introduction

The HL-LHC at CERN is planned to operate with an instantaneous luminosity of
$5\times 10^{34}$ cm-2s-1 or higher, delivering up to ten times more
integrated luminosity than is expected in the current LHC programme. This
increase poses significant challenges in the design and operation of the
detectors at the HL-LHC. In particular, in the forward direction the absorbed
dose will be up as much as 2 MGy and the fluence in the innermost region is
expected to reach $10^{16}$ neq/cm2, which is an unprecedented level in high
energy collider experiments. Additionally there will be ${\approx}140$ proton-
proton interactions occurring (pile up) in every bunch crossing, which happens
at a rate of 40 MHz. This considerably complicates the reconstruction of
events. To contend with these conditions, the CMS Collaboration is planning a
series of upgrades to some of the existing detector components, and replacing
others with new detectors designed specifically to mitigate the effect of the
high pile up [1]. As part of this upgrade programme the current
electromagnetic and hadronic calorimeters in the endcaps will be replaced with
a new calorimeter, known as the ‘High-Granularity Calorimeter’ (HGCAL) [2].
This new sampling calorimeter (CE) will be sub-divided into two sections, the
electromagnetic (CE-E) and the hadronic (CE-H), in an arrangement shown in
Figure 1. The CE-E will be equipped with silicon sensors, while the CE-H will
be equipped with both silicon sensors and scintillator tiles read out directly
with SiPMs. In the CE-H the silicon sensors will be at small radii, close to
the beam where the radiation levels are highest, and scintillator tiles at the
larger radii. The absorber of CE-E will be a mixture of lead, copper and
sintered copper-tungsten, while in CE-H the absorber plates will be stainless
steel. The hexagonal silicon sensors will be subdivided into hexagonal cells
with areas of ${\approx}1.1\,\rm{cm}^{2}$ or 0.5 cm2, with the sensors with
smaller cells placed at small radii. The full calorimeter will be operated at
-30∘C to reduce the dark current in the silicon sensors and in the SiPMs.
There will be 28 sampling layers in CE-E and 22 in CE-H. This high-transverse
granularity, combined with the high longitudinal segmentation of the
calorimeter has been selected, within the constraints of cost and available
space, to optimise the identification and measurement of hadronic and
electromagnetic showers in the presence of the high pile up.

Figure 1:  Schematic view of the CMS high granularity Endcap Calorimeter.

The basic detector unit is a silicon module. A module consists of a silicon
sensor, glued to a baseplate on one side and to a printed circuit board (PCB)
for the readout on the opposite side. The individual cells of the sensor are
connected electrically to a readout ASIC on the PCB with wire-bonds that pass
through holes in the PCB. The ASIC amplifies and digitises the analogue
signals and transmits them to the off-detector electronics on receipt of an
external trigger signal. By 2018 more than 100 prototype silicon modules have
been produced using 6-inch hexagonal silicon sensors subdivided into cells
with areas of ${\approx}$1.1 cm2. A module is shown in Fig. 2 (left). Further
details of the construction and assembly of the silicon modules used in these
tests may be found in [3]. The modules were assembled into a prototype of
HGCAL that was tested with beams of electrons, pions and muons at the CERN
SPS.

The ASIC used to read out of the signals from the silicon cells was the
Skiroc2-CMS [4], a custom ASIC designed by the OMEGA group at Ecole
Polytechnique. The function of this 64-channel ASIC was to measure both the
amplitude of the signal and its time of arrival, so not only the energy
response can be measured, but the timing performance can be characterised.
This Skiroc2-CMS ASIC has many of the features of the HGCAL front-end readout
ASIC in development.

In the latest beam test in late 2018 the HGCAL prototype was equipped with
94-hexagonal silicon modules arranged into a 26 radiation length
electromagnetic section and 5 nuclear interaction length hadronic section.
Behind the prototype calorimeter we placed the Analog Hadronic Calorimeter
(AHCAL) prototype, developed by the CALICE Collaboration [5]. This calorimeter
is a scintillator-based sampling calorimeter, similar in design to the
proposed design of the HGCAL [2], but with much finer longitudinal
segmentation. In the final test at the CERN SPS, data were taken with beams of
muons, charged hadrons and electrons with energies ranging from 20 to 300 GeV
at the H2 beam line of the CERN-SPS over a period of two weeks in October
2018.

The data acquisition (DAQ) system for the beam tests needed to be flexible and
scalable to control and read out the increasing number of prototype silicon
modules as they became available. It was designed with readily available FPGA
mezzanines and low-cost Raspberry PIs, and scaled up to work with
${\approx}$12,000 channels for the final test.

This paper describes the DAQ system and is structured as follows: the overall
architecture of the system is described in section 2; the data format and the
back-end DAQ components are described in section 3; the DAQ software is
explained in section 4; in section 5 the detector systems used for system
synchronisation is discussed and the operational experience is discussed in
section 6.

##  2 DAQ System Architecture

Each hexagonal cell of the silicon sensor was connected to the 64-channel
Skiroc2-CMS ASIC. Each channel of this ASIC had a low noise pre-amplifier
followed by high- and low-gain shapers, with a shaping time of 40 ns, and
time-over-threshold (ToT) and time-of-arrival (ToA) circuits. Both of the
shapers had analogue-to-digital converters (ADCs) that sampled the signal
every 25 ns. It also had a circuit to measure the ToA of large amplitude
signals (${}>3\rm{fC}$) with a precision of 50 ps. Further details of the
design can be found in [4].

To simplify routing of the signals in a very dense board, four Skiroc2-CMS
ASICs were used to readout the 128 channels of each silicon sensor, leaving
half of the channels unused. The PCB also had a MAX®10 field programmable gate
array (FPGA) to control the readout of the module. It received the clock,
trigger and busy signals from the off-detector electronics, aggregated the
data from the Skiroc2-CMS ASICs and transmitted it to the off-detector
electronics. Figure 2 (left) is a photograph of a prototype module, with the
Skiroc2-CMS ASICs marked with rectangles, and the MAX®10 FPGA, on the top
left, is indicated with a white circle.

Figure 2:  (left) A prototype module used in beam tests. The Skiroc2-CMS ASICs
are marked with white rectangles. The MAX®10 FPGA is marked with a white
circle. The red grounding wire (on the left side) and, red and violet bias
wires (on the right side) are soldered on the PCB. The micro HDMI (uHDMI)
cable is connected on the top. (right) An interposer board used in beam tests.
The HDMI cable and bias voltage wires, connecting the interposer board to the
prototype module, are connected on the top. The HDMI and RG174 cables,
connecting the interposer board to the DAQ board, are connected on the bottom.

Each of the prototype silicon modules was connected to the off-detector DAQ
boards through an interposer board. These boards regulated the 5 V output from
the DAQ boards to the 3.3 V needed by the prototype modules via HDMI cables.
They also filtered and transmitted the bias voltage coming from the DAQ boards
through RG174 cables to wires soldered to the prototype modules. A photograph
of an interposer board is shown in Figure 2 (right).

The off-detector electronics consisted of a set of custom 9U readout boards,
each of which could receive data from up to eight silicon modules. All the the
readout boards were controlled by a single custom 9U synchronisation board,
the ‘sync board’. The sync board distributed the clock, trigger and busy
signals to all the readout boards. The readout boards communicated with the
data acquisition computer through Ethernet, with a 100 Mbit/s output of the
readout boards connected to a Gigabit Ethernet switch, from which data were
sent to the DAQ computer for processing. In the tests with 94 silicon modules,
one sync board and fourteen readout boards were mounted in two custom air-
cooled crates. The crate that was equipped with one sync board and seven
readout boards is shown in Figure 3.

Figure 3:  A crate, used in HGCAL beam tests, populated with one sync board
and seven readout boards. The sync board (in the leftmost slot) distributed
the clock, trigger and busy signals to the readout boards via HDMI cables, not
shown. The readout boards were used to send control data to the silicon
modules, to receive data from them, and to transmit the data to the
acquisition computer through an Ethernet switch (on the right side of the
readout boards). The readout and sync boards were powered by the power modules
located under the DAQ crate.

Figure 4 shows a schematic view of the inter-connectivity of the DAQ system.
The readout boards were connected to the prototype silicon modules by HDMI
cables both between the readout boards and the interposer boards and between
the interposer boards and the modules. The trigger signal was formed from a
coincidence of signals from two scintillation counters located upstream of the
calorimeter. The 40 MHz system clock, generated on the sync board and the
trigger signals were transmitted to the readout boards with HDMI cables. Since
the beam was not synchronised with the clock, the trigger and the clock were
asynchronous. The bias voltage was distributed to the silicon sensors through
the readout boards with separate RG174 cables.

Figure 4:  Schematic view of the DAQ system used in HGCAL beam tests.

##  3 Data format and back-end DAQ components

###  3.1 Skiroc2-CMS ASIC and prototype module data format

In operation, the analog signals were stored every 25 ns in a switched-
capacitor array (SCA) with a depth of 13 cells. When a trigger was received,
the updating of the SCA was halted and the two values for the ToA were stored,
one referenced to the next falling edge of the 40 MHz clock, and the other to
the next rising clock edge. Two values of the ToT were also kept, one with a
fast ramp time-to-digital converter and another with a slow ramp.

[bitwidth=1.6em]16 []5 & 0,1,2,3,4,15
1924 $\times$ 16-bit
integers []5$\times$64 channels 111 110 110 11$H_{A}$ 1212Low gain ADC (SCA0)
[]5$\times$64 channels 111 110 110 11$H_{A}$ 1212High gain ADC (SCA0)
[]5 1616…$\times$13 SCA cells
[]5$\times$64 channels 111 110 110 11$H_{A}$ 1212Low gain ADC (SCA12)
[]5$\times$64 channels 111 110 110 11$H_{A}$ 1212High gain ADC (SCA12)
[]5$\times$64 channels 111 110 110 11$H_{A}$ 1212ToA (stop falling clk)
[]5$\times$64 channels 111 110 110 11$H_{A}$ 1212ToA (stop rising clk)
[]5$\times$64 channels 111 110 110 11$H_{T}$ 1212ToT (fast ramp)
[]5$\times$64 channels 111 110 110 11$H_{T}$ 1212ToT (slow ramp)
[]5 110 110 110 1313Roll position (13-bit)
[]5 110 110 1414Global timestamp MSB (14-bit)
[]5 110 110 110 1212Global timestamp LSB (12-bit) 110
[]5 111 111 110 110 110 110 110 110 88Chip ID (8-bit)

Figure 5: Data format of the Skiroc2-CMS ASIC. $H_{A}$ is the hit bit for ToA
and is set to ’1’ when ToA is fired. Similarly, $H_{T}$ is the hit bit for ToT
and is set to ’1’ when ToT is fired. The 13-bit roll position is used to
reorder the SCA cells in time.

Figure 5 shows the data format of the Skiroc2-CMS ASIC.

When a trigger was received, by the MAX®10 FPGA of the hexaboard the four
Skiroc2-CMS ASICs converted the data in analogue memory to the digital data
format. These data were then read out by the MAX®10 FPGA and packaged as shown
in Figure 6. The bits $b_{Ai}$ belong to the ASIC "i" which followed the
Skiroc2-CMS data format shown in Fig. 5. For every event 30784 bytes of data
were transmitted from each hexaboard. These data were then gathered, via the
HDMI-uHDMI cables, by the back-end readout boards.

[bitwidth=3.0em]8 0,3,4,7
44HEADER & 441 bit per ASIC
1924 $\times$ 16 bytes 11 0 0 0 $b_{A0}$ $b_{A1}$ $b_{A2}$ $b_{A3}$
88…
11 0 0 0 $b_{A0}$ $b_{A1}$ $b_{A2}$ $b_{A3}$

Figure 6: Hexaboard data format. The bits $b_{Ai}$ belong to the ASIC "i"
which followed the Skiroc2-CMS data format described by Figure 5.

###  3.2 Back-end DAQ electronics

The DAQ system was designed to be easily scalable to provide a readout for
different numbers of silicon modules. To minimise costs, readily available
commercial components were used. Additionally, optical receiver modules (oRMs)
[6], recovered from the CMS level-1 trigger system, when it was upgraded with
faster electronics, were used. Each oRM was equipped with a Kintex-7 FPGA, 4.8
Mbits of block RAM, two 6.6 Gbit/s bi-directional serial ports, and a 128 Mbit
FLASH memory for configuration. The connection from the Kintex-7 FPGA to the
gigabit Ethernet switch was made with SFP to RJ-45 adapters.

####  3.2.1 Readout board

In the final beam test in October 2018, there were 94 silicon modules that
were read out with 14 readout boards mounted in two racks, controlled by a
sync board. The readout boards performed the following tasks:

  * •

Loading firmware on the Max®10 FPGAs and module initialisation and reset.

  * •

Generating and distributing control signals for the prototype silicon modules.

  * •

Accumulating the data received from the prototype silicon modules.

  * •

Distributing the clock, busy and trigger signals.

  * •

Distributing the low voltage power and the bias voltage for the prototype
silicon modules.

The readout board, shown in Figure 7, was a custom PCB equipped with five oRMs
and a Raspberry Pi. Each board had eight HDMI ports on the front panel
connected to the silicon modules through the interposers and one HDMI port on
the back panel for connection to the sync board. On the front of each board
there were eight RG174 connectors that were used to distribute, after
filtering, the bias voltage through the interposers to each detector module.

Figure 7:  The readout board used to send control data to the silicon modules,
to receive data from them and to transmit it to the data acquisition computer.
It readout data from up to eight silicon modules via HDMI connectors. It also
supplied the bias voltage for up to eight prototype silicon modules via
standard RG174 connectors. It was equipped with one control and four data oRMs
and one Raspberry Pi.

The overall readout cycle was controlled via helper processes running on the
Raspberry Pi, which communicated with each oRM through the SPI bus. Each Pi
was connected to the central DAQ server through its Ethernet port, from which
it also received ‘Start’, ‘Stop’, and other commands.

A single readout board was equipped with five oRMs: one control (CTL) oRM, and
four DATA oRMs. The DATA oRMs were responsible for reading the data from up to
two silicon modules, while the CTL oRM received data from the DATA oRMs and
transferred it to the central server. It also managed the communication with
the sync board. The firmware installed on the CTL oRM’s FPGA included the
IPBus firmware [7, 8] for this purpose. The IPBus IP and MAC addresses of the
oRM were set by the Raspberry Pi, as well as other parameters used by the CTL
oRM as it combined the four streams of data.

Before a run started, the helper processes on the Raspberry Pis first
configured the ASICs on the prototype silicon modules, and data collection was
initiated. The Trigger signal was broadcast from the sync board to the readout
boards, from where it was forwarded on to the modules. On the readout boards,
the helper processes running on the Raspberry Pis after receiving the trigger
signal, prompted the ASICs to initiate data transmission. The data from the
ASICs were then sent unprocessed, via the MAX®10 FPGA, to the DATA oRMs, where
it was then merged into a single data stream by the CTL oRM. The 4-bit headers
of Figure 6 were then dropped and 32-bit integers were built with the data
from up to eight modules, corresponding to 32 ASICs. These 32-bit integers
were written to a FIFO to be readout by the central server using the IPBus
protocol over a gigabit Ethernet link. When ready, a flag was set inside the
CTL’s RAM to indicate that the data were ready for transfer. In Fig. 8 the
output data format of the CTL oRM FIFO is shown.

Once the data had been fully read out by the server, the helper processes on
the Pis reset the ASICs, and sent a start acquisition signal. The CTL oRM then
sent a ‘ReadoutDone’ signal to the sync board, indicating the boards had
finished their cycles and were ready to receive the next trigger. The firmware
block diagram of the readout board is shown in Figure 9.

[bitwidth=2.0em,bitformatting=16 0-15
30784 $\times$ 32-bit
integers 1 $b_{0,0}$ $b_{0,1}$ $b_{0,2}$ $b_{0,3}$ 88… 1$b_{7,0}$ $b_{7,1}$
$b_{7,2}$ $b_{7,3}$
1616…
1 $b_{0,0}$ $b_{0,1}$ $b_{0,2}$ $b_{0,3}$ 88… 1$b_{7,0}$ $b_{7,1}$ $b_{7,2}$
$b_{7,3}$

Figure 8: Data format of the CTL oRM FIFO readout by the central server using
the IPBus protocol. The bits $b_{i,j}$ correspond to the data of ASIC "j" of
module "i". Figure 9:  The firmware block diagram of the readout board.

####  3.2.2 Sync board

The function of the sync board, shown in Figure 10, was the distribution of
the common signals to the readout boards and to synchronise the flow of data
from the readout boards. The sync board generated the 40 MHz system clock on a
small mezzanine card mounted at the rear of the board. On the same mezzanine
there were four 50 $\Omega$ RG174 connectors. Two were for the Trigger and a
Veto signal inputs and two for the Clock and a copy of the Trigger signal
outputs. The Veto signal was not used in these tests. Processing on the sync
boards was handled by a Raspberry Pi as well as a central (SYNC) oRM. One sync
board could control up to 15 readout boards through 15 HDMI ports mounted on
the front. An extra HDMI port was mounted on the front to allow for the
possibility of daisy-chaining two or more sync boards together when more than
15 readout boards are to be readout.

Figure 10:  The sync board was used to control up to 15 readout boards. It
received the Veto and the Trigger signals and distributed control signals to
up to 15 readout boards. On top of the 15 HDMI ports for readout board control
there was one extra port for connection to another sync board for daisy-
chaining. It was equipped with a Raspberry Pi and an Kintex-7 FPGA for control
and communication. It was also equipped with a mezzanine card for clock
generation and receiving external signals.

At the start of a readout cycle, the sync board waited for an asynchronous
External Trigger signal. Then this signal was synchronized with the on-board
40 MHz clock and sent to the readout boards to be distributed to the silicon
modules. The sync board then waited for a ‘ReadoutDone’ signal from each
readout board. Once this signal was received from all the readout boards, the
sync board made itself ready to process the next available trigger. The
firmware block diagram of the sync board is shown in Figure 11.

Figure 11:  The firmware block diagram of the sync board.

##  4 Data acquisition software

The DAQ software selected for these tests was based on the EUDAQ [9]
framework. This framework, written in C++, was developed specifically for
small-to-medium scale systems with significantly less overhead than frameworks
used in large scale experiments, like XDAQ [10]. Additionally, in separate
earlier tests of the AHCAL prototype, the EUDAQ system had already been used
successfully.

The EUDAQ framework was designed to be modular and portable. It was structured
so that software for the readout of specific detector components was kept
separate and distinct from the core processes. For this each detector
component that produced data had a ‘Producer’ process running. The functions
performed by the Producer was to initialise, configure, issue stops and starts
to the component, and to collect the data and forward it to the core process.

###  4.1 CMS-CE EUDAQ Producer

A Readout Producer was developed to read out the data from one or more readout
boards in parallel. During the combined beam test of October 2018, the DAQ had
seven of these Readout Producers, each connected to two readout boards. The
$\mu$TCA Hardware Access Library ($\mu$HAL) was used to read the IPBus UDP
transactions from the readout boards. The sequence of operations of the
Readout Producer were as follows:

  1. 1.

Wait until each $\mu$HAL interface is notified that a trigger occurred (by
checking an IPBus register of the CTL oRM boards).

  2. 2.

Read out the FIFO of the CTL oRM board from each readout board and fill raw
data containers. The data format of this FIFO is described in Section 3.2.1. A
time-stamp – the number of 40 MHz clock cycles in a 64-bit integers since the
last configuration – is read out with the data.

  3. 3.

‘ReadoutDone’ signal is sent from CTL oRM to Sync oRM.

  4. 4.

Create an event block containing the raw data from each readout board, the
time-stamp of the readout board and the event ID.

  5. 5.

Forward the event block to the EUDAQ data collector.

  6. 6.

Increment the event ID.

  7. 7.

Return to step 1 and wait for the next trigger.

Once the readout was complete, data from each of the event blocks from each of
the Readout Producers were combined with data from the Producers connected to
other detector components to form a complete event data block.

####  4.1.1 EUDAQ online data monitoring

Part of the EUDAQ framework were tools to monitor the data collection. Online
analyses were developed to monitor in real time the stability of the pedestal
values, noise and occupancies of each of the silicon sensor channels.

####  4.1.2 Data unpacking and first analysis steps

The first step of the data analysis consisted of unpacking the CMS CE EUDAQ
events. For this purpose a C++ library embedded in the EUDAQ framework has
been developed.

An initial data quality check was performed before unpacking the raw data by
comparing the difference in time-stamp with the time-stamp of the previous
event as a check of the synchronisation for all the readout boards. During the
October data taking only a few runs had events with a synchronization failure.

After this test the data were unpacked and the data from each ASIC were sorted
into tables of 16-bit integers with the structure shown in Figure 5 and stored
in a ROOT [11] file. As zero-suppression was not used for simplicity, these
tables contained the data from every channel of the ASICs, including those not
connected to a detector channel. The data for each cell in an event contained
data and pointer information from the 13 SCA cells for both the high- and the
low-gain slow shapers, the ToA and the ToT measurements. The pointer was the
address of first SCA cell data for the event, which allowed the ordering of
the SCA data in time. This was required for the data reconstruction since the
trigger was asynchronous with the 40 MHz clock. The data analysis workflow,
which was developed in the CMSSW framework [12], and transformed the tables of
16-bit integers into a collection of calibarated hits for data analysis used
the ROOT files as input.

##  5 System synchronization

###  5.1 Beam-characterization detectors

The tests of HGCAL and AHCAL prototypes with particle beams in the H2 area at
CERN had been complemented by the readout of various beam-characterization
detectors. Four delay wire chambers (DWC) [13] measured the trajectory and
impact of the particles in the beam, two scintillator detectors served as
external trigger source and two micro-channel plates (MCP) had been used to
provide fast signals for reference timing measurement of the incident
particles [14]. For this purpose, two 16-channel CAEN v1290N TDCs and one
v1742 digitiser were integrated into the HGCAL prototype DAQ.

From each DWC four signals were separately discriminated at a threshold of -30
mV and fed as inputs to the TDC. Since the binning of the time-stamp
digitization should be less than 1 ns corresponding to an optimal resolution
of the position measurement of 200$~{}\mu$m [13], a binning of 25 ps had been
chosen. For proper event synchronization, the trigger for all CAEN modules
stemmed from the duplicated TTL trigger signal issued by the synchronization
board. After conversion to NIM, it was copied three times and fed into each
module individually. After receiving a trigger, events were built, were
labeled with trigger time-stamps and subsequently stored in a local buffer.

Two dedicated EUDAQ producers [9] had been developed. They ran on a separate
computer and communicated to these modules through optical link and VMEbus.
These producers polled the event data from the buffers at a (configurable)
frequency of 500 Hz (HGCAL DAQ rate $\approx 40~{}\text{Hz}$), converted the
raw data into the EUDAQ format and sent it to the main DAQ for storage and
online data monitoring.

###  5.2 AHCAL

The front-end electronics of the AHCAL prototype were designed for low power
operation under a specific ILC accelerator timing with a less than 1 ms long
spill followed by 199 ms idle time. The operation of the Spiroc ASIC [15] was
therefore split in 3 phases: 1) acquisition phase, where self-triggered events
were stored into up to 16 analog memory columns; 2) conversion phase, where up
to 16 events were sequentially digitized by internal ADC; 3) readout phase,
where the digitized data was read out. Detailed timing was described in [16].
For beam test purposes, the acquisition phase length was extended to 16 ms and
the readout phase varied typically between 2 to 20 ms, depending on the hit
occupancy. Any external trigger from the sync board stopped the acquisition
for immediate readout, as shown in Fig 12.

Figure 12:  The time diagram of AHCAL acquisition and readout phases.

Due to the ‘self-trigger’ design of the ASIC, the AHCAL did not require an
external trigger for data taking. All the hits (including noise hits) were
read out and referenced by a number of bunch-crossing clock cycles (4 us
period, called BXID) from the start of the acquisition phase. In order to
assign an external trigger to the hits in the AHCAL, the DAQ internally
samples the external trigger number (with a time-stamp, 48-bit counter with a
25 ns resolution) and the time-stamp of the start of the relevant acquisition
phase. The trigger was assigned to one of the self-triggered events in the
acquisition cycle. Assignment was based on the startup time from the start of
the acquisition phase to the beginning of the first BXID and the additional
delay due to the length of the trigger cable.

Figure 13:  The organization of self-triggered particles in AHCAL BXIDs,
showing the BXID ambiguity for an example of split events due to the time of
arrival with respect to the BXID clock.

The collation of events according to the BXID might have however led to an
existence of incomplete events for particles, that arrived close to the BXID
counter value switching in the ASICs. An example for such a split event is
shown in Figure 13 (particle no. 2). Several factors contribute to the BXID
ambiguity: time walk of the signals, clock skew due to board and ASIC
location, clock tree distribution through the FPGAs and clock jitter. The
internal ASIC TDC [15] had also a region of non-linearity around the BXID
change. Therefore, particles arriving close to the BXID change, within 10 ns,
needed to be excluded from the data analysis.

The AHCAL provided two means of synchronization with the HGCAL data: the
trigger number and the trigger time-stamp, which used the 40 MHz clock from
the HGCAL sync board. Both pieces of information were accessible in the data
file.

##  6 Collected Data

The DAQ was operated at a readout rate of 40 Hz. At this rate the amount of
data readout for a typical 5 second spill of the SPS for the HGCAL prototype
was approx 300 MB, as each of the 94 silicon modules sent 16 KB of data per
event.

In the last run, over a period of two weeks, six million events were collected
with beams of charged hadrons, electrons, and muons with momenta from 20 to
300 GeV/c, with different detector configurations. Figure 14 shows accumulated
events for different detectors over the beam-test campaign.

Figure 14:  The accumulated events for different detectors during beam-test
campaign in October 2018.

##  7 Summary

In the upgrade of the CMS detector for when the HL-LHC is operational, the two
endcap calorimeters will be replaced with high granularity sampling
calorimeters equipped with silicon sensors. As part of the development of this
calorimeter, a series of beam tests have been conducted with different
sampling configurations using prototype segmented silicon detectors readout
with a low-cost custom scalable data acquisition system. The software
framework used for the run control and data collection was the portable
modular EUDAQ framework. In the most recent of the tests conducted in late
2018 at the CERN SPS in 2018, the performance of a prototype calorimeter
equipped with ${\approx}12,000\rm{~{}channels}$ of silicon sensors, in
conjunction with the CALICE prototype analogue hadron calorimeter, was studied
with beams of high-energy electrons, pions and muons, with six million events
collected over a two week period.

## Acknowledgments

We thank the technical and administrative staffs at CERN and at other CMS
institutes for their contributions to the success of the CMS effort. We
acknowledge the enduring support provided by the following funding agencies:
BMBWF and FWF (Austria); CERN; CAS, MoST, and NSFC (China); MSES and CSF
(Croatia); CEA and CNRS/IN2P3 (France); SRNSF (Georgia); BMBF, DFG, and HGF
(Germany); GSRT (Greece); DAE and DST (India); MES (Latvia); MOE and UM
(Malaysia); MOS (Montenegro); PAEC (Pakistan); FCT (Portugal); JINR (Dubna);
MON, RosAtom, RAS, RFBR, and NRC KI (Russia); MST (Taipei); ThEPCenter, IPST,
STAR, and NSTDA (Thailand); TUBITAK and TENMAK (Turkey); STFC (United
Kingdom); DOE (USA).

## References

  * [1] D. Contardo, M. Klute, J. Mans, L. Silvestris, and J. Butler, “Technical Proposal for the Phase-II Upgrade of the CMS Detector,” 2015.
  * [2] C. Collaboration, “The Phase-2 Upgrade of the CMS Endcap Calorimeter,” Tech. Rep. CERN-LHCC-2017-023. CMS-TDR-019, CERN, Geneva, Nov 2017.  Technical Design Report of the endcap calorimeter for the Phase-2 upgrade of the CMS experiment, in view of the HL-LHC run.
  * [3] A. Steen, B. Akgun, et al., “Construction, commissioning and response equalisation of cms ce prototype silicon modules,” In preparation for JINST.
  * [4] J. Borg, S. Callier, D. Coko, F. Dulucq, C. de La Taille, L. Raux, T. Sculac, and D. Thienpont, “SKIROC2-CMS an ASIC for testing CMS HGCAL,” JINST, vol. 12, no. 02, p. C02019, 2017.
  * [5] F. Sefkow et al., “A highly granular SiPM-on-tile calorimeter prototype,” J. Phys.: Conf. Ser., vol. 1162, p. 012012, 2019.
  * [6] M. Baber et al., “Development and testing of an upgrade to the CMS level-1 calorimeter trigger,” JINST, vol. 9, no. 01, pp. C01006–C01006, 2014.
  * [7] C. G. Larrea, K. Harder, D. Newbold, D. Sankey, A. Rose, A. Thea, and T. Williams, “IPbus: a flexible ethernet-based control system for xTCA hardware,” JINST, vol. 10, no. 02, pp. C02019–C02019, 2015.
  * [8] J. Mans and E. Frahm, “Status report on a MicroTCA card for HCAL trigger and readout at SLHC,” JINST, vol. 5, no. 12, pp. C12027–C12027, 2010\.
  * [9] E. D. Team, “EUDAQ User Manual - Last update on October 2016 for EUDAQ version 1.7,” 2016.
  * [10] V. B. et al., “Using xdaq in application scenarios of the cms experiment,” FERMILAB-CONF-03-293, CHEP-2003-MOGT008 (2003).
  * [11] “https://root.cern.ch/guides/reference-guide,”
  * [12] “https://twiki.cern.ch/twiki/bin/view/cmspublic/workbook,”
  * [13] J. Spanggaard, “Delay Wire Chambers - A Users Guide,” SL-Note-98-023, 1998\.
  * [14] N. Akchurin et al., “First beam tests of prototype silicon modules for the CMS High Granularity Endcap Calorimeter,” JINST, vol. 13, p. P10023, 2018.
  * [15] S. Lorenzo, S. Callier, J. Fleury, F. Dulucq, C. de La Taille, G. Martin Chassard, L. Raux, and N. Seguin-Moreau, “SPIROC: design and performances of a dedicated very front-end electronics for an ILC Analog Hadronic CALorimeter (AHCAL) prototype with SiPM read-out,” Journal of Instrumentation, vol. 8, p. C01027, 01 2013.
  * [16] J. Kvasnicka, “Data acquisition system for the CALICE AHCAL calorimeter,” Journal of Instrumentation, vol. 12, no. 03, p. C03043, 2017.