# Rate-capability of the VMM3a front-end in the RD51 Scalable Readout System

D. Pfeiffer<EMAIL_ADDRESS>L. Scharenberg
<EMAIL_ADDRESS>P. Schwäbig<EMAIL_ADDRESS>S. Alcock
F. Brunbauer  M. J. Christensen  K. Desch  K. Flöthner  F. Garcia  R. Hall-
Wilton  M. Hracek  G. Iakovidis  D. Janssens  J. Kaminski  M. Lupberger  H.
Muller  E. Oliveri  L. Ropelewski  A. Rusu  J. Samarati  M. van Stenis  A.
Utrobicic  R. Veenhof  European Spallation Source ERIC (ESS), Box 176, SE-221
00 Lund, Sweden  European Organization for Nuclear Research (CERN), 1211
Geneva 23, Switzerland  University of Milano-Bicocca, Department of Physics,
Piazza della Scienza 3, 20126 Milan, Italy  Physikalisches Institut,
University of Bonn, Nußallee 12, 53115 Bonn, Germany  European Spallation
Source ERIC (ESS), Data Management and Software Centre, Ole Maaløes Vej 3,
2200 Copenhagen, Denmark  Helmholtz-Institut für Strahlen- und Kernphysik,
University of Bonn, Nußallee 14–16, 53115 Bonn, Germany  Helsinki Institute of
Physics, P.O. Box 64, FI-00014 University of Helsinki, Finland  Institute of
Experimental and Applied Physics, Czech Technical University in Prague, Husova
240/5, 110 00 Prague 1, Czech Republic  Brookhaven National Laboratory, PO Box
5000, Upton, NY 11973, United States of America  Physics Department, Vrije
Universiteit Brussel, Pleinlaan 2, 1050 Brussels, Belgium  SRS Technology, 30
Promenade des Artisans, 1217 Meyrin, Switzerland  Bursa Uludağ University,
Görükle Kampusu, 16059 Niüfer/Bursa, Turkey

###### Abstract

The VMM3a is an Application Specific Integrated Circuit (ASIC), specifically
developed for the readout of gaseous detectors. Originally developed within
the ATLAS New Small Wheel (NSW) upgrade, it has been successfully integrated
into the Scalable Readout System (SRS) of the RD51 collaboration. This allows,
to use the VMM3a also in small laboratory set-ups and mid-scale experiments,
which make use of Micro-Pattern Gaseous Detectors (MPGDs). As part of the
integration of the VMM3a into the SRS, the readout and data transfer scheme
was optimised to reach a high rate-capability of the entire readout system and
profit from the VMM3a’s high single-channel rate-capability of
$3.6\text{\,}\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$.
The optimisation focused mainly on the handling of the data output stream of
the VMM3a, but also on the development of a trigger-logic between the front-
end cards and the DAQ computer. In this article, two firmware implementations
of the non-ATLAS continuous readout mode are presented, as well as the
implementation of the trigger-logic. Afterwards, a short overview on X-ray
imaging results is presented, to illustrate the high rate-capability from an
application point-of-view.

###### keywords:

VMM3a ASIC, Scalable Readout System (SRS), Micro-Pattern Gaseous Detector
(MPGD), Readout electronics, Field Programmable Gate Array (FPGA), X-ray
imaging

††journal: Nuclear Instruments and Methods A

##  1 Introduction

A common goal in the development of radiation detectors is to improve their
rate-capability. Prominent examples driving the need for larger statistics,
shorter data acquisition times and better handling of the detector occupancy
are the high luminosity upgrades of the Large Hadron Collider (LHC) in high
energy physics or the European Spallation Source (ESS) for neutron and
material sciences. Alongside the development of new detector technologies
coping with higher event rates comes the development of powerful readout
electronics allowing to deal sufficiently with the new rate challenges.

This high-rate challenge was one of the reasons for the development of the
VMM3a Application Specific Integrated Circuit (ASIC) within the ATLAS New
Small Wheel Upgrade [1]. In parallel to the integration of the VMM3a into the
ATLAS environment, the VMM3a was also integrated [2] into the RD51 [3]
Scalable Readout System (SRS) [4], providing its features for small R&D setups
and mid-sized experiments. Since the readout scheme of the VMM3a in the SRS
differs from that used by ATLAS, the rate-capability of the VMM3a/SRS system
was investigated. The results are presented in this article. In the main part
(section 2), the readout scheme of the VMM3a within the SRS is described. The
focus is set on a description of the firmware, the software to control and
operate the system, and the data acquisition rates that can be achieved with
well-defined test pulses. To confirm the test pulse measurements, and to
demonstrate the rate performance under application conditions, X-ray
measurements were carried out. The results are described in the second part
(section 3).

##  2 The VMM3a ASIC and the RD51 Scalable Readout System

The Scalable Readout System (SRS) is a versatile and multi-purpose readout
system, developed by the RD51 collaboration for the readout of Micro-Pattern
Gaseous Detectors (MPGDs). A typical configuration of the SRS is shown in
figure 1.

(a) 3D drawing of the VMM3a/SRS stages.

(b) Photograph of an R&D set-up.

Figure 1: Example of a typical SRS laboratory set-up. The RD51 VMM Hybrid
contains two VMM3a ASICs (stage A) that are connected (stage B) to a Spartan-6
FPGA (stage C). Each hybrid is connected via HDMI (stage D) to the combination
of DVMM card and FEC card (stage E) that here is shown outside the Minicrate
2k. The FEC is then connected via Ethernet (stage F) to the DAQ computer
(stage G).

The VMM3a ASIC on the hybrid (stage A) records the charge signals from the
detector readout. It is connected (stage B) to the Spartan-6 FPGA on the
hybrid (stage C) which serially reads the digital representation of these
signals from the VMM3a channels. It creates hit data from this information,
and then sends them out via an HDMI cable (stage D) to the combination of the
Digital VMM adapter (DVMM) card and the Front End Concentrator (FEC) card
(stage E). The DVMM card has eight HDMI ports, and can also provide power to
the hybrids, either via HDMI cable or via a power cable plugged into its power
outlet. From the FEC card, the data is sent in the form of UDP frames via
Ethernet (stage F) to a DAQ computer (stage G).

Two different options exist to take data with more than one FEC. Up to 40 FECs
can be connected to the Scalable Readout Unit (SRU), which has one
$10\text{\,}\mathrm{G}\mathrm{b}\mathrm{p}\mathrm{s}$ Ethernet port that
communicates with the network card of the DAQ computer. Alternatively, the
Clock and Trigger Fanout (CTF) card can be used, which provides a common clock
for eight FECs. For the measurements presented here, a CTF card and two FECs
were used. The data from the two FECs were sent via the FEC
$1\text{\,}\mathrm{G}\mathrm{b}\mathrm{p}\mathrm{s}$ network connection to a
$10\text{\,}\mathrm{G}\mathrm{b}\mathrm{p}\mathrm{s}$ switch, and from there
to a PC with a $10\text{\,}\mathrm{G}\mathrm{b}\mathrm{p}\mathrm{s}$ network
card.

In the following sections, a detailed bottom-up description of the VMM3a ASIC
and the SRS is given. The focus lies on the integration of the VMM3a ASIC into
the SRS and the firmware of the different readout stages. Further, the
software tools needed to control the SRS and to acquire and analyse data are
presented. For a more comprehensive technical description of all components,
please refer to [4, 5] (SRS), [6, 1] (VMM) and [2] (VMM3a/SRS).

###  2.1 The VMM3a ASIC

The VMM3a is a $64$-channel readout ASIC, developed by the Brookhaven National
Laboratory for the ATLAS New Small Wheel Upgrade, specifically for the readout
of gaseous detectors [1]. The analogue part of each VMM channel consists of a
charge sensitive amplifier, a shaper, a discriminator and a peak finder. The
gain of the amplifier can be adjusted within a range of
$0.5\text{\,}\mathrm{m}\mathrm{V}\mathrm{/}\mathrm{f}\mathrm{C}$ to
$16.0\text{\,}\mathrm{m}\mathrm{V}\mathrm{/}\mathrm{f}\mathrm{C}$ (8 discrete
settings), and the peaking time of the shaper can be set to values between
$25\text{\,}\mathrm{n}\mathrm{s}$ and $200\text{\,}\mathrm{n}\mathrm{s}$ (4
discrete settings). A test pulse generator optionally generates an internal
test pulse to inject charge into the charge sensitive amplifier. When the
signal amplitude crosses the discriminator threshold and the signal peak has
been found by the peak detector (PDO), the charge and time information of the
signal is subsequently digitised. This digitisation or conversion takes
$250\text{\,}\mathrm{n}\mathrm{s}$. The maximum rate-capability of a VMM3a
channel amounts thus to
$4\text{\,}\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$.

The peak height of the signal is obtained with a
$10\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ (effectively
$\sim$8\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$$) two-stage current-mode
domino ADC [1]. The available time information consists of a
$12\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ coarse time stamp and an
$8\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ fine time stamp. The coarse
timestamp, which is called BCID, is implemented as
$12\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ Gray counter and counts clock
cycles of the bunch crossing clock CKBC111The ATLAS experiment at CERN
measures particles coming from collisions of proton bunches in LHC. Every
$25\text{\,}\mathrm{n}\mathrm{s}$ one proton bunch arrives at the interaction
point, hence the bunch crossing frequency is
$40\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$. The name bunch crossing clock
refers thus to the ATLAS situation, but the clock is in reality just a
timer/counter and can have any chosen frequency. At a CKBC frequency of
$160\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ missing BCIDs were observed. The
maximum CKBC frequency that works reliably lies therefore somewhere between
$44\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ and
$160\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$, but has not been determined by
the authors. The lowest CKBC frequency is around
$2.5\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$. When the peak finder finds the
peak, the ramp of the TDC starts. The next rising edge of CKBC arms the stop
circuit and the next falling edge stops the timing ramp. The TAC slope has
thus to be at least $1.5$ times as long as the period of CKBC. For the longest
available TAC slope of $650\text{\,}\mathrm{n}\mathrm{s}$, the lowest possible
CKBC is thus
$$1.5$/$650\text{\,}\mathrm{n}\mathrm{s}$=$2.31\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$$.
Slower CKBC frequencies therefore mean worse time resolution.. The BCID of the
digitised signal is the last BC clock before the peak of the signal. Depending
on the chosen BC clock frequency ($40\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$
and $44.\overline{4}\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ in the presented
measurements), the resolution of the BCID is $25\text{\,}\mathrm{n}\mathrm{s}$
or $22.5\text{\,}\mathrm{n}\mathrm{s}$. The fine time stamp is provided by the
time detector (TDO), which uses an $8\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$
Time to Amplitude Converter (TAC) with adjustable slopes
($60\text{\,}\mathrm{n}\mathrm{s}$ to $650\text{\,}\mathrm{n}\mathrm{s}$ in 4
discrete settings). The measured value represents the time between the peak of
the signal and the falling edge of the next BC clock. Hence in the case of a
$40\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ BC clock and a TAC slope of
$60\text{\,}\mathrm{n}\mathrm{s}$, the complete time of the peak can be
calculated from the BCID and the TDO with the following formula:

$t_{\textrm{chip}}=\textrm{BCID}\times$25\text{\,}\mathrm{n}\mathrm{s}$+(1.5\times$25\text{\,}\mathrm{n}\mathrm{s}$-\textrm{TDO}\times\frac{$60\text{\,}\mathrm{n}\mathrm{s}$}{255})\
.$ (1)

The VMM3a configuration data consists of
$1728\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ and contains global
and individual channel settings. The available per-channel settings are the
enabling of the internal test pulse, the masking of the channel in case of
malfunctioning, and finally baseline corrections for the PDO and the TDO.
Global settings are gain, peaking time, TAC slope and polarity of the input
signal. Further different modes can be chosen, which are not all relevant for
the non-ATLAS continuous mode used by RD51. Settings that are useful are the
neighbouring-logic (also called ‘neighbour trigger’), the sub-hysteresis
discrimination and the Double Data Rate (DDR) mode for the data clock CKDT.
Channels are normally only read out if the signal amplitude crossed the
discriminator threshold. If the neighbouring-logic (NL) is enabled, the
channels adjacent to channels over threshold are also read out, even when
their signal stayed below the threshold (e.g. channel 8 is over threshold,
with the neighbouring-logic also channels 7 and 9 are read). The DDR mode for
the data clock CKDT means that new bits are available on the data lines _data
0_ and _data 1_ on the rising and the falling edge of the clock, whereas in
single data rate (SDR) mode this happens only on the rising edge of CKDT. The
timing diagram for the data readout of one channel is illustrated in figure 2.

Figure 2: Timing diagram of the VMM3a in the non-ATLAS continuous readout
mode. The Peak Detector (PDO) starts the
$10\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ ADC for the charge information,
and the voltage ramp (TAC slope) for the Time Detector (TDO). This voltage
ramp is stopped by the next falling edge of the bunch crossing clock CKBC. The
$8\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ value of the TDO represents thus
the time between the peak and the subsequent BCID. To obtain the digitised
data, a token is generated with the token clock CKTK, and passed around to
channels with data. The channel that has the token is then bit-wise read out
with the data clock CKDT on the two data lines _data 0_ and _data 1_.

###  2.2 The RD51 VMM3a Hybrid

The RD51 VMM3a hybrid is a small
$$5\text{\,}\mathrm{c}\mathrm{m}$\times$8\text{\,}\mathrm{c}\mathrm{m}$$
Printed Circuit Board (PCB) equipped with two VMM3a ASICs, providing thus 128
charge input channels. A 144 pin Hirose connector on the hybrid connects to
the readout of a gaseous detector. From the Hirose connector, the 128 channels
are routed to the channel inputs of the two VMM3a ASICs, with each channel
having a spark protection circuit on the PCB. The other I/Os on the ASICs
(clock inputs, configuration inputs, data outputs) are presently connected to
a Xilinx Spartan-6 Field Programmable Gate Array (FPGA). The FPGA will be
upgraded to Spartan-7 in the second half of 2021. The FPGA provides the
necessary clocks for the VMM3a (bunch crossing clock CKBC, token clock CKTK,
data clock CKDT, and the test pulse clock CKTP), and reads out the two data
lines _data 0_ and _data 1_. Further, the FPGA takes care of sending the
configuration data to the ASICs.

The format of one $38\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ long VMM3a hit
(a hit is the data read out from one channel) is shown in table 1.

Table 1: Data format of one 38 bit VMM3a hit. Name | Length (bit) | Values
---|---|---
Data flag | 1 | always 1
Over-threshold flag | 1 | over threshold: 1
below threshold (requires NL on): 0
Channel number | 6 | 0 to 63
ADC (PDO) | 10 | 0 to 1023
TDC (TDO) | 8 | 0 to 255
BCID (clock counter) | 12 | 0 to 4095

After reading out the hits from the two VMM3a, they are sent from the hybrid
via the DVMM card to the RD51 Front-End Concentrator (FEC) card. The maximum
possible readout rate that can be obtained for the complete VMM3a ASIC depends
on the passing speed of the readout token, and the speed of the data transfer.
In the following two sections, different firmware approaches to obtain high
rates on the hybrid are presented.

The hybrid is connected via HDMI cable to the RD51 FEC. In the HDMI
cable222The HDMI cable just serves as electrical connection between hybrid and
FEC. The wires of the HDMI cable are not mapped according to the HDMI
standard, and a custom made protocol different from the HDMI protocol is used
for data transmission. four differential pairs are used for high-speed
transmission, and two single wires for I2C. Via the clock differential pair, a
base clock is sent from the FEC to the hybrid. The purpose of the
configuration/trigger pair is to transmit configuration data and to send
control commands like the start and stop of the acquisition from the FEC to
the hybrid. For each of the two VMM3a ASICs on the hybrid, a dedicated
differential pair is available for data transmission from hybrid to FEC. VMM3a
data, as well as configuration/trigger data, are
$8\text{\,}\mathrm{b}$/$10\text{\,}\mathrm{b}$ encoded.

All four differential pairs are connected to serialiser/deserialiser (SERDES)
components in the FEC and the hybrid firmwares. In the hybrid firmware, the
two data pairs use output serialiser/deserialiser (OSERDES) components,
whereas the clock and configuration/trigger pairs are connected to input
serialiser/deserialiser (ISERDES) components. The ISERDES of the
configuration/trigger receiver is used in combination with an input/output
delay (IODELAY) component. The delay of the IODELAY component is automatically
adjusted so that each bit is read exactly at the centre of the data eye. For
the correct word alignment of the $10\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$
word, the bit slip feature of the ISERDES is used.

The timing diagram for the data readout from the VMM3a is shown in figure 3.

Figure 3: The diagram explains the readout of the VMM3a data on the hybrid
FPGA, and displays one full readout cycle (cycle _n_). In case hit data is
available, it is read out in the time between two tokens. In the depicted DDR
mode, the maximum readout frequency is
$9\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$, whereas in SDR mode it is
$6\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$. Since the time for token
generation and readout of the flag remains unchanged, the maximum DDR readout
frequency is not twice as large as the SDR frequency.

With the help of a counter, the readout token clock CKTK is generated from the
readout process clock with a pulse length of about
$28\text{\,}\mathrm{n}\mathrm{s}$ and a frequency of up to
$9\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$. The token clock is always running
and continuously sending tokens. If data is available, the VMM3a pulls the
data line _data 0_ high as a flag. The firmware detects this flag and starts
the data clock CKDT that can have a frequency of up to
$180\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$. Although designed for CKDT
frequencies up to $200\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ [2], no higher
CKDT frequencies than $180\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ could be
reached, due to the observation of occasional data corruption on some ASICs
(see section 2.2.1).

The data clock then pushes out the $38\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$
VMM3a hit. Only after the entire hit has been transferred by the data clock,
the channel is clear and can store a new hit333It should be mentioned that in
the non-ATLAS continuous mode, which is used in the SRS integration of the
VMM3a, the 4-hit deep FIFO mentioned in [1] does not work (it was identified
by the authors together with the designer of the chip [7]). Due to a bug that
occurred when the L0 mode was implemented in the 2016 revision of VMM3, the
FIFO-read advancement does not work. This means the next three hits after a
hit are stored, but cannot be read out. The FIFO is not used in the L0 mode
used by ATLAS.. The diagram depicts the DDR readout mode, with 19 bits coming
out on each of the two data lines _data 0_ and _data 1_ on the rising and
falling edge (when using SDR mode only on the rising edge) of CKDT. When the
complete hit has been read out, the
$19\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ from _data 0_ and
$19\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ from _data 1_ are
combined to a $38\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ hit, which is padded
with two zeros to $40\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$. The
$40\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ hit is first stored in a $1024$
hit deep FIFO. This occurs at the same time as the sending of the next token.
The hit data is then read out from the FIFO, encoded in
$8\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$/$10\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$
format and subsequently sent to the FEC.

The following two chapters describe two different high-speed firmware
implementations, the Bonn version and the ESS version. The structure of both
hybrid firmware versions is shown in figure 4. The versions differ mainly in
the choice of clock frequencies and the use of different components (ISERDES
for Bonn, IDDR for ESS) to read the data from the VMM3a ASIC.

Figure 4: Schematic drawing of the main functional components of the Spartan-6
firmware versions for the RD51 VMM3a hybrid. There are two data readout blocks
on the hybrid (one per VMM3a), and a common configuration and clock part.

####  2.2.1 Bonn firmware for the RD51 VMM3a Hybrid

The Bonn firmware of the hybrid was used for initial research of the high-
speed readout capabilities of the VMM. It serves as a general-purpose firmware
with many configuration possibilities. The base clock of the system is
$40\text{\,}\mathrm{MHz}$. Using two phase-locked loops (PLLs) all remaining
necessary clocks are generated based on the $40\text{\,}\mathrm{MHz}$ clock.
One of the PLLs is used for generating the clocks necessary for data transfers
coming from the VMM while the other PLL is used for generating all remaining
clocks needed by the firmware. To the latter belongs the bunch crossing clock
CKBC which can be chosen from a list of seven different frequencies:
$2.5\text{\,}\mathrm{MHz}5\text{\,}\mathrm{MHz}10\text{\,}\mathrm{MHz}20\text{\,}\mathrm{MHz}40\text{\,}\mathrm{MHz}80\text{\,}\mathrm{MHz}160\text{\,}\mathrm{MHz}$.
The incoming data from the VMM3a ASICS is received by ISERDES components with
a deserialisation factor of five, reducing the clock frequency behind the IOB
by the same factor. This way most of the logic of the firmware uses a lower
clock which relaxes timing. To align the data an IODELAY component with fixed
tap delay is used in front of the ISERDES.

Figure 5: Maximum data transfer clock measurement, results for eight tested
VMM3a ASICS and five different transfer clocks CKDT. The relative amount of
valid data transfers (with a total of about $20\,000$ transfers per VMM3a) is
shown for each channel.

The data transfer clock CKDT has a maximum frequency of
$180\text{\,}\mathrm{MHz}$ which is lower than the design value of
$200\text{\,}\mathrm{MHz}$, as tests showed that frequencies above
$180\text{\,}\mathrm{MHz}$ resulted in incomplete data transfers when using
double data rate. Measurements of this can be seen in figure 5. The lower
readout frequency showed to be a design problem of the VMM that was verified
in simulation. Since all clocks for the data transfers from the VMM3a are
generated by the same PLL from which the remaining firmware is independent, it
is possible to reprogram the PLL using its Dynamic Reconfiguration Port (DRP)
to chose transfer clock frequencies of
$90\text{\,}\mathrm{MHz}45\text{\,}\mathrm{MHz}22.5\text{\,}\mathrm{MHz}$
instead.

When using data transfers with double data rate the token clock CKTK has a
frequency of $8\text{\,}\mathrm{MHz}$ with a duty cycle of roughly
$20\text{\,}\mathrm{\char 37\relax}$ and is generated using a counter. The
resulting token length of roughly $28\text{\,}\mathrm{ns}$ was tested to be
necessary such that all tested VMM3a ASICs recognise the token.

The number of active channels has a significant influence on the highest
possible data rate. This is due to an empty data transfer which occurs when
the token has to be passed to a channel with lower channel number: when there
is a data transfer from channel $n$ but no channel with number $>n$ has data
to transfer, the token must be passed down to a lower channel or has to stay
at $n$. The latter is the case if no other channel has any data to be
transferred. The VMM3a will then require an additional token on CKTK during
which it does not shift out any data, effectively reducing the data rate of
the VMM. Having $N$ active channels and a sufficiently high hit rate the total
transfer time for all channels is given by $(N+1)\times t_{\textrm{d}}$ with
$t_{\mathrm{d}}=1/f_{\textrm{CKTK}}$. In order to improve the data rate, the
firmware will increase the token clock CKTK and send the next token after a
shorter time $t_{\textrm{p}}$ if the VMM3a does not initiate a data transfer
some time after the previous token. The highest expected hit rate considering
the effects of the additional token and influence of active channel count and
varying token clock is given by

$\textrm{hit rate}=\frac{\textrm{active channels}}{\textrm{active
channels}\times t_{\textrm{d}}+t_{\textrm{p}}}\times\frac{\textrm{acquisition
window}}{4095}\ ,$ (2)

with the acquisition window being a configurable amount of counts by the Gray
counter for which the VMM is enabled during a complete BCID cycle.

Measured hit rates in dependence on the number of active channels as well as
the corresponding expected rate calculated using equation 2 are shown in
figure 6. It can be seen that theoretical limit and measurement are in very
good accordance, with the extrapolated maximum hit rate close to
$7.2\text{\,}\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$.
When looking at the rates for one active channel note that the upper limit for
a single channel is
$4\text{\,}\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$
although in tests no VMM performed with rates above
$3.6\text{\,}\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$.
Since this upper limit varied between different VMM ASICs it is assumed that
this is due to the ASIC itself. Tests with different waveforms and added time
offsets between channels did not increase the measured hit rate.

Figure 6: The hit rate received from a single VMM as a function of the number
of active channels for three different Bonn firmware versions (similar
limitations apply to the ESS firmware, but are not shown here). The markers
show the measurement and the lines the expected rate calculated based on the
times $t_{d}$ and $t_{p}$. Note that for a single active channel the
measurements for the upper two curves are identical. The horizontal lines mark
the rates in case the hit rate would be independent of the amount of active
channels. Stimulus for all measurements is an externally generated, periodic
sawtooth waveform.

An additional feature of the firmware is the auto alignment: for updates and
changes of the firmware – especially those regarding the readout and clocking
scheme – the data read from the VMM3a might become misaligned such that bits
are missing at the beginning or end. Therefore the firmware offers the
possibility to automatically align the
$38\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ data word correctly.

####  2.2.2 ESS firmware for the RD51 VMM3a Hybrid

The implementation of the data readout in the ESS firmware is depicted in
figure 4. Via the clock differential pair, a
$44.\overline{4}\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ base clock is sent
from the FEC to the hybrid. The
$44.\overline{4}\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ clock from the FEC is
fed into a phase-locked loop, which creates all the clocks needed in the
firmware444When the hybrid is used as part of the ESS readout and not the SRS,
the basic clock frequency send to the hybrid will be
$44.0265\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$, half of the ESS facility
clock frequency of $88.053\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$.. The
output clocks are distributed via global clock buffers (BUFG) components. The
general firmware logic uses a frequency of
$44.\overline{4}\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$. The clock generator
for the bunch crossing clock works with
$88.\overline{8}\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ and generates a fixed
$44.\overline{4}\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ CKBC. The data
readout procedure and the test pulse clock generator all use
$177.\overline{6}\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$. The fastest clock
is the $444.\overline{4}\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ clock for the
SERDES components.

The clock and configuration part of the firmware is common for both ASICs,
whereas one data readout block has been implemented per VMM3a. The two data
lines (_data 0_ and _data 1_) of the VMM3a are connected to input double data
rate register (IDDR2) components.

Assuming the clock settings described above (the
$44.\overline{4}\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ clock from the FEC is
taken as basic clock for the hybrid), the maximum hit rate is
$8.\overline{8}\text{\,}\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$.
This rate is perfectly matched to the speed of the SERDES. With a clock
frequency of $444.\overline{4}\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ and
$8\text{\,}\mathrm{b}$/$10\text{\,}\mathrm{b}$ encoding, the effective bit
rate of the SERDES amounts to

$\textrm{bit
rate}_{\textrm{max}}=$444.\overline{4}\text{\,}\mathrm{M}\mathrm{b}\mathrm{p}\mathrm{s}$\times
0.8=$355.\overline{5}\text{\,}\mathrm{M}\mathrm{b}\mathrm{p}\mathrm{s}$\ .$
(3)

For a $40\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ hit, the hit rate is hence

$\textrm{hit
rate}_{\textrm{max}}=$355.\overline{5}\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$/$40\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$=$8.\overline{8}\text{\,}\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$\
.$ (4)

This is close to the maximum of
$9\text{\,}\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$,
which can be read from a single VMM3a. This maximum rate would require a
SERDES clock of $450\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ and a CKDT of
$180\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$.

###  2.3 The RD51 SRS FEC

The combination of RD51 FEC and DVMM card can read out up to 8 RD51 VMM3a
hybrids. The hybrids are connected via HDMI cables to the DVMM card, from
which the signals are routed to a Xilinx Virtex 6 FPGA on the FEC. Figure 7
explains the data path of the FEC firmware.

Figure 7: Schematic drawing of the data path in the Virtex 6 firmware of the
RD51 SRS FEC.

The differential pairs in the HDMI are described in chapter 2.2.2. The ESS
version of the FEC firmware operates the four SERDES in DDR mode with a
$222.\overline{2}\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ clock, and sends a
$44.\overline{4}\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ clock to the hybrids,
whereas the Bonn version uses $200\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ and
sends a $40\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ clock. The bit rate of
each SERDES is thus
$444.\overline{4}\text{\,}\mathrm{M}\mathrm{b}\mathrm{p}\mathrm{s}$ for the
ESS version, and $400\text{\,}\mathrm{M}\mathrm{b}\mathrm{p}\mathrm{s}$ for
the Bonn version. Due to $8\text{\,}\mathrm{b}$/$10\text{\,}\mathrm{b}$
decoding, the effective incoming bit rate on each data line on the FEC is
reduced by $20\text{\,}\mathrm{\char 37\relax}$ to
$355.\overline{5}\text{\,}\mathrm{M}\mathrm{b}\mathrm{p}\mathrm{s}$ 3 and
$320\text{\,}\mathrm{M}\mathrm{b}\mathrm{p}\mathrm{s}$, respectively. After
$8\text{\,}\mathrm{b}$/$10\text{\,}\mathrm{b}$ decoding, the hits from the
VMM3a are then time-stamped on the FEC in the latency logic block, and
subsequently as $48\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ hits written into
a hit FIFO. For each hybrid connected to the FEC, the firmware provides two
separate FIFOs (one per VMM3a). Whenever a hit is available, it is written
with a $44.\overline{4}\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ write clock to
the FIFO.

In the readout part of the firmware, there is one common FEC hit FIFO for all
hits. A fair scheduler runs with $125\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$
and reads one hit at a time from all the non-empty VMM3a data FIFOs. Provided
that it is not full, the hits are transferred to the FEC FIFO. To be sent out
via UDP, the $48\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ hits have to be
decomposed into six $8\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ chunks and
written to the UDP FIFO. The 8 bits from the UDP FIFO are read with
$125\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$, as determined by the
$1\text{\,}\mathrm{G}\mathrm{b}\mathrm{p}\mathrm{s}$ Ethernet of the FEC.
Since the maximum data rate per VMM3a is around
$356\text{\,}\mathrm{M}\mathrm{b}\mathrm{p}\mathrm{s}$, with the theoretical
maximum at $360\text{\,}\mathrm{M}\mathrm{b}\mathrm{p}\mathrm{s}$, the FEC can
only support two VMM3a or one RD51 hybrid at maximum rates.

####  2.3.1 Latency logic on the FEC

The coarse time stamp in the hit data, the BCID, is a 12-bit counter and
overflows every 4096 BC clock periods or
$92.16\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{s}$ 555To simplify the
description, in the following the $22.5\text{\,}\mathrm{n}\mathrm{s}$ BC clock
period of the ESS version is taken for the calculations. For the Bonn version,
one simply has to replace the $22.5\text{\,}\mathrm{n}\mathrm{s}$ with
$25\text{\,}\mathrm{n}\mathrm{s}$.. The FEC generates a
$42\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ FEC time stamp with
$22.5\text{\,}\mathrm{n}\mathrm{s}$ resolution, which only overflows every
$1.2\text{\,}\mathrm{d}\mathrm{a}\mathrm{y}\mathrm{s}$. To save bandwidth, the
$42\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ timestamp is only send every 16
BCID overflow periods of
$92.16\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{s}$, i.e. every
$1.47\text{\,}\mathrm{m}\mathrm{s}$ for each VMM. This combination of
$42\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ timestamp,
$5\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ VMM-ID and a
$1\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ flag (always 0) is called _marker_.

In the time between markers, the $5\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$
VMM-ID and a $5\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ overflow cycle or
offset counter are added to the $38\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$
VMM data. The DAQ can distinguish between markers and data by looking at the
$1\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ flag. If the flag is zero it is a
marker, if it is one it is a hit.

The hits from the VMM3a are read out and transmitted in a roughly time ordered
way. The readout token inside the VMM3a is passed around between channels
containing data in a round-robin like fashion, going from lower channel
numbers to higher channel numbers. Therefore slight disturbances of the time
order can occur, as illustrated with the following example: the ASIC
registered three hits that occurred within one BC clock cycle. Channel $2$ has
a hit with BCID $0$, whereas channel $1$ and channel $63$ have a hit with BCID
$4095$. After reading channel $1$ (BCID $4095$), the token will be passed to
channel $2$ (BCID $0$), and subsequently to channel $63$ (BCID $4095$). For
the hit in channel $2$ with BCID $0$, the BCID overflow already happened. The
BCID in channel $1$ and $63$ has not overflown yet, since the hit occurred
slightly earlier. When the FEC generates the higher-order time stamp, the
firmware has thus to consider that the hit with BCID $0$ did not occur $4094$
clock cycles earlier than the hits with BCID $4095$, but instead slightly
later.

To be able to make the decision to which overflow cycle a hit belongs, the FEC
also counts BC clock cycles, similar to the BCID counter in the VMM3a. After
receiving the acquisition start signal from slow control, the FEC starts
incrementing its BC counter and then sends via the hybrid a soft reset command
to the VMM3a. This soft reset command sets the BCID counter on the VMM3a to
zero. After sending the soft reset, the FEC is ready to accept data from the
hybrids. The time at which the FEC sends the soft reset is called _reset
latency_ and can be adjusted via the slow control.

This _reset latency_ time is comprised of two components: the time that it
takes the soft reset command to reach the VMM3a and reset the BCID, and the
time that it takes for a single hit from the VMM3a to reach the FEC. If
correctly set, a single hit with e.g. BCID $100$ will arrive on the FEC at the
moment where the FEC BC counter is also $100$. The aim is thus to have the
same values in BCID and FEC BC. With $2\text{\,}\mathrm{m}$ HDMI cables, the
_reset latency_ latency is of about $47$ clock cycles. That means the soft
reset is sent when the FEC BC counter has a value of $$4096$-$47$=$4049$$.
After setting the reset latency to the correct value, the FEC BC counter is
now identical to the BCID of a single hit upon arrival on the FEC. Figure 8
shows data from one channel on one VMM acquired with the correct latency
settings.

Figure 8: Channel 0 on VMM 2 is pulsed once every 4096 BC clock cycles at BC
clock 100. The Wireshark trace shows the hit data with the offset period and
the BCID. The field _trg_ represents the value of the FEC BC counter. The
difference between the _trg_ and the BCID is the latency, which has values
here between -1 and 1. After all data from offset 15 is transmitted, a marker
with the 42-bit SRS time is sent for every active VMM.

A small jitter of a few clock cycles can nevertheless occur. This parameter is
called _latency jitter_ and is usually set to around $4$ clock cycles. The
third important parameter is the _maximum latency_. If all $64$ channels are
activated at the same time, there is a substantial difference in data arrival
time on the FEC between the first and the last channel that is read out. On
the hybrid one hit can be read out every $5$ clock cycles of the BC clock or
every $112.5\text{\,}\mathrm{n}\mathrm{s}$. So when reading out $64$ hits that
have been digitised at the same time, the last hit will arrive with a maximum
latency of $7.2\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{s}$.

Figure 9 explains how the latency logic is implemented. In addition to
counting BC cycles, the FEC also counts the overflows of its BC counter. After
$16$ overflow cycles, a new marker is generated and written into each VMM
FIFO. The overflow counter is then reset from $15$ to zero. Upon arrival on
the FEC, the BCID of a hit is compared with the FEC BC counter. Based on the
result of the comparison, the FEC firmware can decide to which overflow cycle
a hit belongs. A hit can belong either to the present overflow cycle, the
previous overflow cycle or violate the latency conditions. In the easiest and
most common case, the hit belongs to the present overflow cycle. The present
overflow counter is then added to the data. If the hit belongs to the previous
overflow cycle, the overflow counter is decremented by one. Therefore overflow
counters with values between $-$1$$ and $+$15$$ can be added to the valid
data. Invalid hits that violate the latency conditions are marked with an
overflow counter of $-$16$$. With a correctly configured system and fully
operational hardware, invalid hits do not occur.

Figure 9: The latency logic flow chart explains how the FEC firmware
determines the correct higher order time stamp for the hits that arrive on the
FEC. The BCID of the hit is compared with the FEC BC counter. Depending on the
parameters latency jitter and maximum latency, an overflow counter is added to
the hits. In the most common case the hit belongs to the present overflow
cycle (cases in green boxes). If the FEC BC is a lot smaller than the BCID
(FEC BC counter plus $4096$ is smaller than the BCID plus the maximum
latency), then the FEC BC counter has already overflown while data of the
previous overflow cycle are still read out. The previous overflow cycle can
either belong to the present marker (yellow box), or be the last overflow
cycle of the previous marker (orange box). In case the latency conditions are
violated, the hit is marked as invalid. With correctly set parameters and
working hardware this case does not occur.

The latency logic makes it possible to continuously take data without applying
an acceptance window and without rejecting data. If measuring e.g. X-rays or
white noise, the time distribution of the acquired data is flat. In the DAQ or
the data analysis, the correct FEC timestamp is always calculated from the
last marker that has been sent. The overflow periods are added to the marker
(or subtracted from the marker if the overflow count is $-$1$$) according to
the formula:

$t_{\textrm{FEC}}=t_{\textrm{marker}}\times$22.5\text{\,}\mathrm{n}\mathrm{s}$+\textrm{overflow
counter}\times 4096\times$22.5\text{\,}\mathrm{n}\mathrm{s}$\ .$ (5)

To obtain the complete hit time stamp, the FEC timestamp (equation 5) and the
VMM3a timestamp (equation 1) have to be added together:

$t_{\textrm{hit}}=t_{\textrm{FEC}}+t_{\textrm{VMM3a}}\ .$ (6)

###  2.4 The DAQ PC

The RD51 slow control tool _vmmdcs_ configures FEC, hybrid and VMM3a [8]. The
configuration data is sent in UDP frames via Ethernet cable from the PC to the
FEC. The FEC can determine the destination of the configuration based on the
different UDP ports of the configuration frame. For the data acquisition, two
solutions are available. The _ESS DAQ_ [9] provides event reconstruction and
online monitoring but has limited rate-capabilities when the full non-
reconstructed hit data is written to disk (so-called debug mode).
Alternatively, the _tcpdump_ [10] utility can directly read the data at the
network adapter. The network packages are then written to disk, but online
monitoring and visualisation are not possible. Thus, the solution for high-
rate data-taking is to use _tcpdump_ for the data acquisition, and the _ESS
DAQ_ only for the online monitoring. This combination allows to obtain the
highest possible rates with VMM3a and SRS. It should be noted that when using
standard PC operating systems for high speed UDP network communication several
techniques need to be applied to minimise the processing overhead and reduce
avoidable packet loss [11] The offline data analysis _vmm-sdat_ [12] is able
to cope with the data from _tcpdump_ and _ESS DAQ_. It clusters the hit data,
time matches the clusters in between the readout planes and reconstructs thus
the interaction point of the particle in the gas volume of the detector. This
way the rate of the detected particles can be calculated. The bit rates on the
other hand were directly measured on the network adapter with the Linux
command-line tool _ifstat_. With the DAQ PC666For the measurements (e.g. the
ones presented in section 3), a typical 2.5-inch SATA
($6\text{\,}\mathrm{G}\mathrm{b}\mathrm{p}\mathrm{s}$) SSD combined with a
$10\text{\,}\mathrm{G}\mathrm{b}\mathrm{p}\mathrm{s}$ network card was used.
No specific optimisation/modification of the software or hardware components
was applied. as last stage of the readout scheme, the maximum rates per
readout stage, which could be achieved, are summarised in table 2.

Table 2: Rate-capability of the various stages of VMM3a/SRS in the current implementation of the non-ATLAS continuous readout mode. The values given in number of hits/active channels per second ($\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$) and data bandwidth in megabits per second ($\mathrm{M}\mathrm{b}\mathrm{p}\mathrm{s}$). The rate at the DAQ computer depends on the number $n$ of network ports on the switch and the bandwidth limit of the network card (e.g. $10\text{\,}\mathrm{G}\mathrm{b}\mathrm{p}\mathrm{s}$). Readout stage | Maximum rate
---|---
Unit Quantity | ($\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$) | ($\mathrm{M}\mathrm{b}\mathrm{p}\mathrm{s}$)
(A) | VMM3a channel | $3.6$ | $144$
Rate per channel
(B) | VMM3a to Spartan-6 | $8.\overline{8}$ | $355.\overline{5}$
Rate per VMM3a
(C) | Spartan-6 | $17.\overline{7}$ | $711.\overline{1}$
Rate per HDMI-SerDes
(D) | HDMI | $17.\overline{7}$ | $888.\overline{8}$
Rate per Hybrid
(E) | DVMM and FEC | $71.\overline{1}$ | $7111.\overline{1}$
Rate per DVMM card
(F) | Gigabit Ethernet | 20.8 | $1000$
Rate per FEC
(G) | DAQ computer | $$20.8$\times n$ | $$1000$\times n$
Rate per Switch Port

##  3 Measurements with X-rays

For the rate measurements in the previous descriptions, test pulses have been
used, due to their constant and well-known input rate. In the following, the
results of rate measurements under application conditions are presented.
First, the experimental set-up is described (section 3.1), followed by the
measurements on the rate limitations (section 3.2). As last part, a few
examples of high-rate X-ray imaging are presented, demonstrating the
capabilities of VMM3a/SRS (section 3.3).

###  3.1 Experimental methods

For the measurements, a COMPASS-like triple-GEM detector [13], which was
operated at a gain of around ${10}^{4}$, was uniformly irradiated with X-rays.
The anode featured an $x$-$y$-strip readout with $256$ strips of
$400\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{m}$ pitch in each direction
and an active area of $$10$\times$10\text{\,}\mathrm{c}\mathrm{m}^{2}$$. As
X-ray source, a copper target X-ray tube (Ital Structures Compact 3K5 X-Ray
Generator) was used. The acceleration voltage was set to
$20\text{\,}\mathrm{k}\mathrm{V}$, while the X-ray tube current was varied.

The electronics set-up was chosen in a way that allowed to achieve the highest
readout rates. For this, only a single hybrid was connected to one FEC, with
two FECs in total being used; one for the readout of the $x$-strips of the
detector and one for the readout of the $y$-strips of the detector. Hence, the
actively read out detector area was only
$$5$\times$5\text{\,}\mathrm{c}\mathrm{m}^{2}$$, with the rest of the area
being shielded to absorb the other X-ray photons.

However, most measurements have been performed with the neighbouring-logic of
the VMM enabled, which slightly reduces the rate-capability. As explained in
section 2.1, the neighbouring-logic (NL) allows to read out the adjacent
channels below threshold, if the triggering channel surpassed the threshold.
This feature was enabled because it is beneficial for the position
reconstruction and thus X-ray imaging applications [14].

###  3.2 Results of the rate limitation measurements

The measurement results are shown in figures 10 to 12.

Figure 10: Measured hit rate (number of active channels in the full set-up per
second) depending on the X-ray tube current for the two firmware types.

In figure 10 the measured hit rate, the number of recorded hits (active
channels) per second, is plotted against the X-ray tube current. A larger
X-ray tube current means that more X-ray photons are generated and thus more
hits are expected777The expected hit and cluster rates are illustrated as
dashed lines in figures 10 and 11. The rates are estimated based on the known
linear relation between photon flux and X-ray tube current in the selected
current region for the specific X-ray tube. The linear behaviour is confirmed
with a fit for the first three data points in the low X-ray tube current
region, where both firmware versions lead to similar cluster rates and no
cluster losses (plane clusters) are observed. In addition, for the data point
at $0.6\text{\,}\mathrm{m}\mathrm{A}$ X-ray tube current, the same rate was
measured with an oscilloscope, while reading the induced signals from the
bottom of the last GEM in the detector. Based on this, the linear cluster rate
behaviour was extrapolated from the first three data points towards larger
X-ray tube currents. To get the hit rate, the average cluster size of about 5
strips per readout plane is used (measured at
$0.6\text{\,}\mathrm{m}\mathrm{A}$).. It can be seen that the measured hit
rate saturates at high X-ray currents, indicating a bandwidth limitation. The
cause for the saturation was identified to be the token passing of the VMM3a
itself in combination with the randomly888Random in position and time, but
still uniformly distributed. occurring X-ray interactions (see also section
2.2.1).

In figure 11 and 12 the effect of the saturation of the measured hit rate on
the cluster reconstruction is shown; a cluster contains all hits that belong
to the interaction of a single X-ray photon in the detector. It should be
noted that some of the observed effects (see especially figure 12) are related
to the $x$-$y$-strip readout999Here with a 1:1 mapping, meaning that the strip
number of the detector is the same as the readout channel number of the
electronics. and they may be different for other readout geometries. Due to
the saturation in the hit rate, also a saturation in the cluster rate is
expected. This is observed for the clusters in each individual readout plane
(see figure 11).

Figure 11: Measured cluster rate depending on the X-ray tube current for the
two firmware types. Both, the cluster rates in each readout plane, as well as
the detector cluster rate (where a cluster in the $x$-plane could be matched
with a cluster in the $y$-plane), are shown.

However, for the detector cluster rate (where a cluster in the $x$-plane could
be matched with a cluster in the $y$-plane), a decrease after reaching a
maximum is observed. The saturation of the measured cluster plane rate can be
entirely explained with the bandwidth limits, while the decrease in the
measured cluster detector rate is a combination of the bandwidth limit and the
cluster reconstruction. For its explanation, the cluster reconstruction
procedure is recalled. As mentioned in section 2.4, the clusters are first
reconstructed in each readout-plane individually. For this, the hits in a
certain time interval (here $\Delta
t_{\mathrm{hits}}\leq$150\text{\,}\mathrm{n}\mathrm{s}$$) are considered as a
cluster in time. Then the geometrical component is added, meaning that the
hits in the time cluster occur on adjacent strips (in the present analysis,
one missing strip was allowed). Hence, a saturation in the hit rate leads to a
saturation of the measured cluster rate in the readout plane. Afterwards, the
timestamps of the clusters found in each plane (calculated via centre-of-
gravity from the individual hits, with the charge as weight) are matched in
time again (here $\Delta
t_{\mathrm{plane}}\leq$150\text{\,}\mathrm{n}\mathrm{s}$$). If now, due to
bandwidth limitations, data (meaning clusters) are lost/not processed; for
example the hits of a cluster in the $x$-plane can still be reconstructed to a
cluster, while the corresponding hits in the $y$-plane are lost due to the
bandwidth limits and thus no combined detector cluster can be identified. This
matching procedure gets less efficient, the larger the actual X-ray
interaction rate in the detector gets, because it means that more hits can be
lost. This explains the decrease in the rate of reconstructed clusters and is
illustrated in figure 12, where the fraction of the matched clusters for each
plane is shown.

Figure 12: Fraction of matched clusters in $x$\- and $y$-direction depending
on the X-ray tube current for the two firmware types.

In addition to the quantitative effects (fewer reconstructed clusters due to
the reduced cluster finding efficiency), the bandwidth limitations also affect
the quality of the reconstructed data. Figure 13 displays the X-ray spectra
for each current setting of the X-ray tube.

Figure 13: Measured spectra from the copper target X-ray tube for various
X-ray tube currents.

The quality of the spectra decreases significantly with increasing hit rates
and is particularly bad, as soon as the bandwidth saturation
($\geq$8\text{\,}\mathrm{m}\mathrm{A}$$ X-ray current) is reached. The
bandwidth limitation originates from the readout of the VMM3a on the hybrid
level101010A bandwidth problem of the FEC can be excluded. A hit rate of
maximally
$17.5\text{\,}\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$
per FEC is well below the maximum of
$20.8\text{\,}\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$
derived from the gigabit Ethernet.. The readout of each channel is achieved by
‘looping’ over all $64$ VMM channels and reading one channel after another
that has the active token flag for stored data; as mentioned in section 2.2, a
VMM channel can only register a new hit after the previous hit has been read
out. This can lead to the loss of charge information required to reconstruct
the full cluster, as illustrated in figure 14.

Figure 14: Illustration of two clusters $A$ and $B$ that have a geometrical
overlap of $\Delta x$. The time difference between the occurrence of the
clusters is $\Delta t$. The active readout channels are read out token-by-
token, with the readout time $T_{n}$ for the token on channel $n$.

To illustrate the effect of this VMM3a readout limitation, two charge
clouds/clusters are shown. The two clusters overlap geometrically and are
close in time. Each cluster will generate hits on several readout channels.
The rate limitation occurs, if the time difference $\Delta t=t_{B}-t_{A}$
between the occurrence of two clusters $A$ and $B$ is now smaller than the
time difference between the readout of the channels $\Delta
t^{\prime}=T_{n+1}-T_{n}$. Some of the hits in the first cluster $A$ (in
figure 14 it is channels $n+4$ and $n+5$) have not been read out when the
second cluster $B$ occurs. In this case, the information that is acquired by
the VMM channels $n+4$ and $n+5$ cannot be stored in the channel buffer and is
simply lost. The more channels are activated, the more likely this type of
data loss is to occur due to the token passing scheme and the serial nature of
the data transfer.

This explanation is confirmed by the distributions, shown in figure 15.

(a) $2\text{\,}\mathrm{m}\mathrm{A}$ X-ray tube current

(b) $10\text{\,}\mathrm{m}\mathrm{A}$ X-ray tube current

Figure 15: Distributions of the time differences between two consecutive hits
for a certain number of Activated Channels Per Hybrid (ACPH) and different
X-ray tube currents. The different axis scales between (a) and (b) should be
noted.

There, the distributions of the time difference between two consecutive hits
on a single channel are shown, for two X-ray tube currents (meaning different
interaction rates) and for various numbers of Activated Channels Per Hybrid
(ACPH). The time it takes to read out one channel and jump to the next one
with the raised token flag is constant. Thus, the more channels are active,
the more time it takes to loop over all channels with an active token and read
them out. For a low X-ray current ($2\text{\,}\mathrm{m}\mathrm{A}$), the
shape of the distributions remains the same, independent of the number of
channels that could be read out. By turning the NL off, only the number of
hits changes as expected (less active strips per cluster), but not the overall
behaviour. At higher X-ray tube currents ($10\text{\,}\mathrm{m}\mathrm{A}$),
in the bandwidth limitation, the effect of the token passing can be seen. The
lower the number of channels which can be read out (ACPH), the quicker one
‘readout loop’ of the token is finished and the more hits can be read out in
the same amount of time. This can be seen, as the peak of the time difference
distributions moves towards smaller values for smaller ACPHs, but also for the
same number of ACPH, when the NL is turned off.

###  3.3 X-ray imaging examples

As last part of the X-ray studies, a few measurements highlighting the high-
rate imaging capabilities of VMM3a/SRS are shown. The X-ray tube settings were
slightly changed compared to the previous measurements: the acceleration
voltage was set to $18\text{\,}\mathrm{k}\mathrm{V}$, while the tube current
was kept constant at $5\text{\,}\mathrm{m}\mathrm{A}$.

In figure 16 an X-ray image of a pen is shown, containing
$17\text{\times}{10}^{6}$ clusters.

Figure 16: Image of a pen containing $17\text{\times}{10}^{6}$ clusters. The
full data set contains $50\text{\times}{10}^{6}$ clusters, that have been
recorded in
$30\text{\,}\mathrm{s}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{d}\mathrm{s}$.

The full data set, due to the larger active area, consists of
$50\text{\times}{10}^{6}$ reconstructed clusters, which have been recorded in
$30$ seconds, corresponding to a measured interaction rate of
$1.7\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$. The measured hit rate was on
average
$20.8\text{\,}\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$.
Similar to the image of the pen, an X-ray image of a small mammal was taken
(figure 17).

Figure 17: Image of a dead mammal. The data set for this image contains
$277\text{\times}{10}^{6}$ clusters, that have been recorded in $180$ seconds.

This image was recorded with similar hit and measured cluster rates as the one
of the pen. However, the acquisition time was slightly longer with
$3\text{\,}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{u}\mathrm{t}\mathrm{e}\mathrm{s}$,
allowing to record $277\text{\times}{10}^{6}$ clusters for the image.

Due to the high rate-capability with the self-triggered readout and the
individual event reconstruction, it is also possible to perform imaging of
dynamic processes. To illustrate this, two examples are shown in figure 18.

(a) Frames of the recorded opening of the X-ray tube’s shutter.

(b) Frames of the rotating blades of a fan.

Figure 18: Examples of the continuous data stream, which is sliced into
frames, allowing to reconstruct dynamic processes.

Figure 18a displays the opening of the X-ray tube’s shutter. The length of
each frame was here chosen to be $2\text{\,}\mathrm{m}\mathrm{s}$. In figure
18b, the rotating blades of a fan are shown, again with a frame length of
$2\text{\,}\mathrm{m}\mathrm{s}$. Each frame of the fan contains around $4000$
clusters. To increase the absorption of the X-ray photons, the blades have
been covered with copper tape. Further, the X-ray tube voltage was reduced to
$16\text{\,}\mathrm{k}\mathrm{V}$ in order to decrease the Bremsstrahlung
fraction in the spectrum.

##  4 Conclusion and outlook

In the past years, the VMM3a ASIC, which was specifically designed to read out
gaseous detectors of the ATLAS New Small Wheel, has been successfully
integrated into the RD51 Scalable Readout System. Two VMM3a ASICs are combined
together with a Spartan-6 FPGA on a front-end board, the RD51 VMM hybrid. As
part of the integration process, the non-ATLAS continuous self-triggered
readout scheme between the VMMs and the FPGA, as well as the readout scheme
towards the Front-End Concentrator (FEC) card, were optimised for high rates.
In this article, two firmware implementations for the Spartan-6 FPGA have been
presented. Additionally, the optimised firmware on the FEC is described. It
was shown, that readout rates of up to
$8.\overline{8}\text{\,}\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$
per VMM and
$20.8\text{\,}\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$
per FEC can be achieved. The results were then confirmed by using the
electronics for high-rate X-ray-imaging studies, where in an optimised set-up,
interaction rates of around $2\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ were
measured.

The SRS and specifically the RD51 VMM3a are thus suitable for various
applications that require high data rates. Future studies like the measurement
of fast charge-up effects in detectors in high irradiation environments, or
the multi-channel investigation of space charge effects are now possible. In
the coming productions of the RD51 hybrid, the Spartan-6 FPGA on the RD51 VMM
hybrid will be replaced with a Spartan-7 FPGA. A working prototype of the
Spartan-7 version already exists. This hardware upgrade will also allow the
investigation and implementation of other VMM readout modes, which are e.g.
described here [15]. Implementations of even faster readout rates of about
$10\text{\,}\mathrm{M}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$
can be envisaged.

## Acknowledgements

Parts of this work have been sponsored by the Wolfgang Gentner Programme of
the German Federal Ministry of Education and Research (grant no. 05E18CHA).

Parts of this work have received funding from the European Union’s Horizon
2020 research and innovation programme under the Marie Sklodowska-Curie grant
agreement No. 846674 as well as from the BMBF (Germany) under 05K19PD1.

## References

  * [1] G. Iakovidis, VMM3a, an ASIC for tracking detectors, Journal of Physics: Conference Series 1498 (2020) 012051.  doi:10.1088/1742-6596/1498/1/012051.
  * [2] M. Lupberger, et al., Implementation of the VMM ASIC in the Scalable Readout System, Nuclear Instruments and Methods in Physics Research A 903 (2018) 91–98.  doi:10.1016/j.nima.2018.06.046.
  * [3] RD51 Collaboration, Development of Micro-Pattern Gas Detectors Technologies.
URL http://rd51-public.web.cern.ch/rd51-public/

  * [4] S. Martoiu, et al., Development of the scalable readout system for micro-pattern gas detectors and other applications, Journal of Instrumentation 8 (2013) C03015.  doi:10.1088/1748-0221/8/03/c03015.
  * [5] S. Martoiu, et al., Front-end electronics for the Scalable Readout System of RD51, in: 2011 IEEE Nuclear Science Symposium Conference Record, 2011, pp. 2036–2038.  doi:10.1109/NSSMIC.2011.6154414.
  * [6] G. D. Geronimo, et al., VMM1 - An ASIC for micropattern detectors, in: 2012 IEEE Nuclear Science Symposium and Medical Imaging Conference Record (NSS/MIC), 2012, pp. 633–639.  doi:10.1109/NSSMIC.2012.6551184.
  * [7] G. De Geronimo, personal communication.
  * [8] M. Guth, et al., vmmsc – VMM3a Slow Control and Calibration Software.
URL https://gitlab.cern.ch/rd51-slow-control/vmmsc

  * [9] M. Shetty, et al., ess-dmsc/essdaq: brightness-software-deliverable-MS35.  doi:10.5281/zenodo.1298367.
URL https://doi.org/10.5281/zenodo.1298367

  * [10] The Tcpdump Group, Tcpdump.
URL https://www.tcpdump.org/

  * [11] M. Christensen, T. Richter, Achieving reliable UDP transmission at 10 Gb/s using BSD socket for data acquisition systems, Journal of Instrumentation 15 (2020) T09005.  doi:10.1088/1748-0221/15/09/t09005.
  * [12] D. Pfeiffer, et al., vmm-sdat – VMM3a/SRS Data Analysis Tool: Analysis software for VMM3a data, recorded with the SRS as PCAP or HDF5 files (GdGEM pipeline of the EFU).
URL http://github.com/ess-dmsc/vmm-sdat

  * [13] C. Altunbas, et al., Construction, test and commissioning of the triple-gem tracking detector for compass, Nuclear Instruments and Methods in Physics Research A 490 (2002) 177–203.  doi:10.1016/S0168-9002(02)00910-5.
  * [14] L. Scharenberg, X-ray imaging with gaseous detectors using the VMM3a and the SRS, Nuclear Instruments and Methods in Physics Research A 1011 (2021) 165576\.  doi:https://doi.org/10.1016/j.nima.2021.165576.
  * [15] T. Alexopoulos, et al., The VMM readout system, Nuclear Instruments and Methods in Physics Research Section A 955 (2020) 163306.  doi:https://doi.org/10.1016/j.nima.2019.163306.


# Constructing trading strategy ensembles by
classifying market states

Michal Balcerak<EMAIL_ADDRESS><EMAIL_ADDRESS>Institute for Theoretical Physics,
Georg-August-Universität Göttingen, Germany  Thomas Schmelzer
<EMAIL_ADDRESS><EMAIL_ADDRESS>Faculty of Business and
Economics (HEC Lausanne),
University of Lausanne, Switzerland

Rather than directly predicting future prices or returns, we follow a more
recent trend in asset management and classify the state of a market based on
labels. We use numerous standard labels and even construct our own ones.

The labels rely on future data to be calculated, and can be used a target for
training a market state classifier using an appropriate set of market
features, e.g. moving averages. The construction of those features relies on
their label separation power. Only a set of reasonable distinct features can
approximate the labels.

For each label we use a specific neural network to classify the state using
the market features from our feature space. Each classifier gives a
probability to buy or to sell and combining all their recommendations (here
only done in a linear way) results in what we call a trading strategy. There
are many such strategies and some of them are somewhat dubious and misleading.
We construct our own metric based on past returns but penalizing for a low
number of transactions or small capital involvement. Only top score-
performance-wise trading strategies end up in final ensembles.

Using the Bitcoin market we show that the strategy ensembles outperform both
in returns and risk-adjusted returns in the out-of-sample period. Even more so
we demonstrate that there is a clear correlation between the success achieved
in the past (if measured in our custom metric) and the future.

###### Contents

  1. 1 Introduction
  2. 2 Labeling
  3. 3 Market representation
    1. 3.1 Feature space
    2. 3.2 Fixing feature space parameters
    3. 3.3 Chosen market representation.
  4. 4 Approximation of labels
    1. 4.1 Goal of the approximation
    2. 4.2 Chosen algorithm
    3. 4.3 Classifier evaluation
  5. 5 Trading strategy
  6. 6 Experiment
    1. 6.1 Experimental setup
    2. 6.2 Results - single strategy performances
    3. 6.3 Results - strategy ensemble performances
  7. 7 Conclusions
  8. 8 Future work
  9. A Appendix: selected labels
  10. B Appendix: feature space
  11. C Appendix: strategy space

##  1 Introduction

Using neural networks to predict financial time series data is today widely
regarded as the old unfulfilled dream of quantitative finance. An idea would
be to apply supervised learning and train a neural network with sub-windows of
a time series to predict the next data point(s). So instead of using images of
dogs and cats we use at some time $t$ the last $n$ points of a time series to
predict a point following at some time $t^{\prime}>t$. Given the non-
stationary nature of time series market data and low signal-to-noise ratios,
this is a rather ambitious problem.

For instance, rather than using $n$ prices (or returns), we reduce the
dimensionality of the problem by using $m<<n$ features based on the very same
$n$ points, i.e. an optimal combination of $m$ moving averages. Such questions
have typically been addressed by linear regression. However, linear regression
fails to exploit any non-linear effects between the features.

We do not stop by only modifying the input - we also alter the goals of our
predictions. Rather than aiming for a (noisy) price trajectory we ask simpler
questions more suitable for the machinery of machine learning. Our goal is to
quantify the probability $p$ of a market being in a class or category $c$ or
moving into one within the next hours or minutes. This could be the
probability for a trend reversion or a spike in volatility or volume. We rely
on labels as recently made popular by López de Prado [2] but also create some
on our own. The flexibility of labels allows us to design a strategy by
emphasizing effects we try to cover.

For each label we ask for an optimal set of $m$ features to approximate them.
These features, through a classifier, induce a probability for the market to
be in a particular label-class. We then ask for an optimal linear combination
of those probabilities to execute trades. Rather than looking at a Sharpe
ratio in an out-of-sample period we construct robust variations of this
concept and penalize for a lack of trading activity, etc. Although we don’t
aim directly for it we observe high Sharpe ratios and attractive returns as an
unavoidable collateral side effect.

##  2 Labeling

We describe a market by a time series of datapoints
$p_{t_{0}},p_{t_{1}},\ldots$. Predicting unseen price data is a hard problem
often resulting in the notorious estimate that the next price is just the last
observed price.

Rather than aiming for the next price, we argue that the market is currently
in a particular label-class which we ultimately want to identify without using
any unseen future data.

Throughout this work we distinguish three such label-classes and identify them
with the actions we intend to take:

  * •

Buy. The market may start or continue to rise over the next few periods

  * •

Sell. The market may drop over the next few periods, the volume may drop
significantly or there is a spike in volatility.

  * •

Neutral. We do nothing.

Obviously identifying buy opportunities is trivial with a good level of
hindsight. Looking at a historic time series we can identify numerous buy
opportunities. This process is subject to some constraints we set, e.g. we may
argue that the price at time $t$ was a buy opportunity if we see a significant
rise over the next minutes following $t$.

We call the process of classifying over time the _labeling_ of a time series.
So the particular _label_ is a time series mapping $p_{t}$ to one of the three
classes.

Numerous such labels can and should be used. We use the popular _threshold
label_ to discuss the concept. We define the return over the period $t_{i}$ to
$t_{i+1}$ as

$r_{i,i+1}=\frac{p_{t_{i+1}}}{p_{t_{i}}}-1.$

We introduce a threshold $\tau$ and use

$y_{i}=\left\\{\begin{array}[]{ll}sgn(r_{i,i+1})&|r_{i,i+1}|>\tau\\\
0&\text{otherwise}\end{array}\right.$ (1)

Note that $y_{i}=0$ if $|r_{i,i+1}|\leq\tau$ otherwise $y_{i}=1$ or
$y_{i}=-1$.

We also use a continuous companion of this label function. We use a continuous
interpolation between the two labels $1$ and $-1$. So if $|r_{i,i+1}|<=\tau$
we use instead of $y_{i}=0$ the function

$y_{i}=\left(\frac{r_{i,i+1}}{\tau}\right)^{3}.$

We could use a simpler linear term. However, in our experiments we have made
better experiences with this particular choice.

So given a historic time series with all its price jumps and chaotic behaviour
we reduce it to a time series just oscillating between three label-classes.
Obviously we loose some information in this process but one could also argue
we emphasize the information we really care about. And we can always combine
multiple labels.

Identifying the moments we have missed to make a profit can help to evaluate
the quality of a strategy, however, its inherent delay renders it of limited
use in a live trading setup.

The idea is to approximate the labels with market features (i.e. technical
trading indicators) that do not use any future data. Once in live trading, we
can live update the indicators and therefore talk about label-classes
predictions. The threshold is often made dynamic using estimates for the
current volatility.

We use a variation of this idea where rather than $p_{t_{i+1}}$ in the
definition of the return we use a moving average of prices following $t_{i}$.

The construction of such labels is an exercise only done during the training
phase of the strategies. Running a backtest based on the actions induced by
the labels over this training period would be a severe mistake.

Although it would be possible to have labels based on all sorts of financial
data, e.g. volume, we use here exclusively labels based on price data.

##  3 Market representation

The central idea of this paper is to approximate the labels with a set of
functions, referred to also as market features. The functions we use are
standard technical indicators. The art is to resolve the labels in a small set
of such parametrized functions. Those parameters are chosen in a way to
maximize the _label separation power_ of those functions.

Although the arsenal of orthogonal functions, i.e. a set of sin waves, is
generally a great choice for approximations, we believe it is not suitable to
capture market dynamics. A Fourier transform of the label would learn
everything about the seasonality of this label but is of very limited
generalization in an out-of-sample period.

We present our ideas using a toy example of only two functions with one free
parameter each. In Appendix: feature space we give a complete list of features
we have used.

The set of feature functions we identify as _feature space_. The parameters
are not completely free. They are integer numbers from intervals we define.
Hence we can pick for each label from a finite set of such features.

###  3.1 Feature space

To illustrate a feature space on an relatively simple example, let us define
it as 2 indicators with some possible parameters:
Example feature space:

  * •

feature 1: $A[X]$, VWAP - SMA($X$) where $X\in[2,10]$ [minutes]

  * •

feature 2: $B[Y]$, VWAP - SMA($Y$) where $Y\in[30,60]$ [minutes]

where VWAP stands for volume weighted average price of a givin minute and
SMA($Z$) stands for a moving average of the last $Z$ minutes.

This set of two features has one feature that looks at relatively short term
time horizon and one feature with relatively long term. We normalise their
values to $[-1,1]$ using local scaling by standard deviation and arctangent
function.

Let us define a label as one of the threshold labels: 1.5 % price change in 5
minute window. We now face a dilemma - which features from the feature space
should we use? There are 279 candidates (9 different feature 1 and 31
different feature 2).

Let us fix parameters to acquire 2 possible feature sets from the feature
space and solve the dilemma there:

  * •

feature set 1:

$A[5],B[50]$ (2)

  * •

feature set 2:

$A[10],B[30]$ (3)

An approximator sees market states only through their market representation.
It is essential that features used in the market representation will differ in
values if they encounter different classes of our label of choice. To measure
these differences we use the L1 distance between corresponding vectors of
features values. The market representations are illustrated on Figures 1 and 2

BTC/USDT market and its representation

Figure 1: Market VWMP, threshold label with 1.2% price change, 5 minutes time
horizon and its Eq. 2 market representation. Zoomed out Fig. 2.

BTC/USDT market and its representation - zoomed in.

Figure 2:  Fig. 1 \- zoomed in. The red and the blue star indicate two
different market states label-class-wise but with similar feature values in
Eq. 2 market representation. The two features fail to resolve the cross-label-
classes which is the central problem of the market representation through
feature selection.

Frequent low values of cross-label-class distances in a given market
representation may cause severe problems for an approximator to correctly
classify different market states as different label classes. Let us look at
feature-wise market representation distances across a time period through a
histogram of cross-label-class distances - Fig. 3

Cross-label-class distances histogram.

Figure 3: Histogram of distances in Eq. 2 (feature set 1) and Eq. 3 (feature
set 2) market representations. Both representations contain a lot of cross-
label-class pairs with distances close to zero, however, the feature set 1
should be slightly better for an approximator than the feature set 2 because
the feature set 1 distances histogram is skewed to the right. 3000 points from
each class so 9 million points per histogram

Based on Fig. 3 we conclude this subsection saying that the Eq. 2 market
representation is better for the threshold label with 1.2% price change, 5
minutes time horizon, than the Eq. 3 market representation. In it important to
point out that both representations contain a lot of cross-label-class pairs
with distances close to zero, so one should either search for a different
feature set from the feature space or change the feature set altogether.

###  3.2 Fixing feature space parameters

We need a way to quantify goodness of a particular features set to represent a
market for a particular label across a time period. Let us define a following
metric for it:
Label separation power of a feature set: inverse of an area under a cross-
label-class distances histogram (like in Fig. 3) weighted by a function to
only select values relatively close to zero. Choice of the weighting function
depends on the label and the numbers of feature in the feature space.

Choosing a particular feature set from a feature space for a given label is
done through maximising their label separation power with Bayesian
Optimization [5] and HyperBand [1].

###  3.3 Chosen market representation.

Feature space used in later parts of the paper contains 28 standard price and
volume indicators and is formally defined in Appendix: feature space.

Threshold label with 1.2% price change and 5 minutes time horizon is one of
the labels which we used for the analysis. The selected feature set has the
following cross-label-class distance histogram:

Cross-label-class distances histogram.

Figure 4: Histogram of distances of the selected features for the threshold
label with 1.2% price change and 5 minutes time horizon. 3000 points from each
class so 9 million points per histogram. Note fundamental differences between
Fig. 3 and this one. Only 0.005% of cross-label-class distances is below 3 on
the same dataset as Fig. 3 histogram.

Cross-label-class distances histogram - log scale.

Figure 5: Log scaled Fig. 4 Y-axis-wise. class0 (buy) and class2 (sell) are
the easiest to separate - this is in agreement with our intuition.

##  4 Approximation of labels

###  4.1 Goal of the approximation

There are eight labels to approximate (defined in Appendix: selected labels)
using eight different market representations (from a feature space defined in
Appendix: feature space). Instead of approximating the continuous labels, we
choose to classify discrete-label-classes. This way we can focus on
identification of the most important three regimes of the labels. In addition,
this approach gives us a probabilistic way to determine confidences of our
predictions. Each discrete-label-classes will be assigned a probability of
occurring at a given time. The process of training a label classifier using
historical market data is illustrated on the Fig. 6, whereas getting the label
approximation is illustrated on the Fig. 7:

Process of label classifier preparation (training)

Figure 6: Illustration of training a label classifier using historical market
data. Calculating the features requires knowledge of what happened in recent
past, however, calculating true labels requires also knowledge of the near
future.

Process of live label classification

Figure 7: Illustration of label classification using current market data.
There is no need to know near future in this process, which makes it possible
to perform live.

###  4.2 Chosen algorithm

Based on Fig. 4 we see that an accurate approximator for this label and this
feature set is possible to built but has to be non-linear. We conclude the
same for the other seven labels. Because of a high number of datapoints in our
training dataset (exact numbers in Experimental setup) and the requested non-
linear behaviour we have decided to use a neural network classifier and a
supervised learning algorithm. For hyper-parameter optimisation we used
previously mentioned Bayesian Optimization [5] and HyperBand [1]. The loss
calculator, which appears on Fig. 6, is built based on a concept called loss
scaling which scales loss based on continuous labels. The central idea is to
make class0 (buy) and class2 (sell) prediction accuracy more significant than
class1 (do nothing) in the feedback loop to the label classifier during the
training. This is an essential step because of heavy class-unbalance in the
labels we have chosen. We construct the scaling in such a way that the sum of
loss scale factors associated with class0 (buy) and class2 (sell) is equal to
the sum of loss scale factors for class1 (do nothing). In addition, we reduce
the loss scaling in-between 0-1 and 1-2 continuous label to make the training
focus on clearer buy/do nothing/sell signals {0,1,2}.

Figure 8: Continuous label based loss scaling factors used during the training
process (Fig. 6). The sum of loss scale factors associated with class0 (buy)
and class2 (sell) is equal to the sum of loss scale factors for class1 (do
nothing). We reduce the loss scaling in-between 0-1 and 1-2 continuous label
to make the training focus on clearer buy/do nothing/sell signals (0,1,2).

###  4.3 Classifier evaluation

A trained classifier acting on unseen data is illustrated on Fig. 9. Apart
from industry-standard metrics like generalisation and confusion matrix
coefficients, we also study our classifiers through Shapley Values [3][4][6].
This approach enables to see impact of a particular feature on the model
output. If at this point, the data would not comply with our intuitions, we
would not have chosen this particular feature space and the algorithm for
label approximation for further experiments.

Neural network output

Figure 9: Prices, corresponding true labels and neural network model output
predictions. Class0 (buy) in red, class1 (do nothing) in green, and
class2(sell) in blue for the threshold label with 1.2% price change and 5
minutes time horizon. Out-of-sample data fragment.

##  5 Trading strategy

Our strategies are based on linear combinations of 16 model outputs -
class0(buy) and class2(sell) of each 8 labels. We map it to $[0,1]$ using a
map:

$\phi(x)=\left\\{\begin{array}[]{ll}(x+1)/2&|x|<1\\\
sgn(x)&\text{otherwise}\end{array}\right.$ (4)

to acquire a trading signal $y$:

$y=\phi(W^{T}X)$ (5)

where $y$ is a trading signal, $W$ is a column of weights, $X$ is a column of
16 model outputs - class0(buy) and class2(sell) of each 8 labels. We interpret
the output value $y$ as a desired long position on an asset. To execute
trades, we use three thresholds: $y_{buy}$ to buy, $y_{sell}$ to sell and
$y_{width}$ to prevent execution of relatively small transactions. 19
constrained parameters in total. The parameter space are formally defined in
Appendix: strategy space. Fig. 10 illustrates threshold-based trading
execution. We increase (decrease) the long position if the trading signal $y$
is above (below) $y_{buy}$ ($y_{sell}$) and a distance between our previous
position and the desired position is greater than $y_{width}$.

Figure 10: Illustration of the way our trading strategy changes the long
position size based on a trading signal $y$. Thresholds for the illustration:
$y_{buy}$ = 0.75, $y_{sell}$ = 0.25, $y_{width}$ = 0.10.

##  6 Experiment

We ran an experiment of backtesting Trading strategy with 20 thousand
different weights columns. The goal was to check performances in the past
dataset and check their generalisation on the future dataset. Exact
definitions of the used datasets are in the Experimental setup section. To
acquire the weights we ran Bayesian Optimization [5] and HyperBand [1] with a
task of producing weights with high performance in the past dataset. Because
of random nature of the HyperBand algorithm we have acquired a full spectrum
of strategies - from bad to good performance-wise.

###  6.1 Experimental setup

We based our experiment on BTC/USDT tick-by-tick transaction data recorded on
the Binance exchange. We divided our dataset into three parts
(dd-mm-yyyy):

  * •

01-01-2018 : 31-12-2019: Classifier train&evaluate dataset

  * •

04-01-2020 : 05-05-2020: Strategy past dataset

  * •

09-05-2020 : 19-09-2020: Strategy future dataset (out-of-sample)

Gaps in-between datasets were designed to prevent the look ahead bias. We
aggregate the data into 1 minute datapoints. Transactions in our backtesting
use the next open price to execute orders and carry a flat transaction fee
equal to 0.05%. This is a realistic estimation of a transaction cost which can
be achieved on an exchange.

###  6.2 Results - single strategy performances

We have to find a selection metric for our strategies. First, let us talk
about cross-datasets returns, where we put past dataset strategy returns
against corresponding future dataset strategy returns. The comparisons are
illustrated on the Fig. 11.

Figure 11: Cross-dataset returns of strategies. Returns on the past and future
datasets are on the X and Y axis respectively. Statistically insignificant
future red label corresponds to strategies that resulted in less than 5 trades
per month on the future dataset

As illustrated on Fig. 11, a past dataset return cannot be used as a metric to
select strategies with promising results in the future. The positive
correlation brakes down around 5% montly return on the past dataset and the
region with highest future returns is filled with statistically insignificant
future performances. We need a better strategy selection method.

Let us introduce score function as follows:

$S=MP-TP-MCIP$ (6)

where:

  * •

S: strategy score

  * •

MR: montly return

  * •

TP: transaction penalty - if number of transaction per month on the past
dataset is lower than 30, then it is equal to the missing transactions per
month

  * •

MCIP: mean capital involvement penalty - if a mean capital involvement (mean
long position) is lower than $25$%, then it is equal to a half of missing
percentages

The goal of the score function is to map the problematic low-return or
statistically insignificant regions (illustrated on the Fig. 11) to low scores
but preserving the positive performance correlation structure. Parameter
values of the score function were chosen intuitively, before the cross-
datasets studies. The number 30 in the transaction penalty was chosen simply
because for Bitcoin it is the number of trading days in a month, and all of
the used features can drastically change intra-day. An idea how to modify the
score function, to make it less accidental is presented in Future work.

Cross-score performances are illustrated on the Fig. 12 down below:

Figure 12: Cross-dataset performances of strategies. Scores on the past
dataset and returns on the future dataset are on the X and Y axis
respectively. Statistically insignificant future red label corresponds to
strategies that resulted in less than 5 trades per month on the future
dataset.

The higher the past dataset score the higher (on average) the future dataset
monthly returns. The statistically insignificant regions were mapped out of
the illustrated on Fig. 12 region. The score can now be used as a selection
metric for our trading strategies.

###  6.3 Results - strategy ensemble performances

Strategy ensemble backtests using Top100, Top20, Top10, and Top5 past score-
performance-wise models are illustrated on Fig. 13, Fig. 14, Fig. 15, and Fig.
16 respectively.

Figure 13: Performance of top 100 score-wise strategies and a corresponding
strategy ensemble on the future dataset. Risk adjustment is performed through
scaling the strategy ensemble returns by
$\frac{\sigma_{BTC/USDT}}{\sigma_{ensemble}}$ where $\sigma$ is the standard
deviation of returns.  Figure 14: Performance of top 20 score-wise strategies
and a corresponding strategy ensemble on the future dataset. Risk adjustment
is performed through scaling the strategy ensemble returns by
$\frac{\sigma_{BTC/USDT}}{\sigma_{ensemble}}$ where $\sigma$ is the standard
deviation of returns.  Figure 15: Performance of top 10 score-wise strategies
and a corresponding strategy ensemble on the future dataset. Risk adjustment
is performed through scaling the strategy ensemble returns by
$\frac{\sigma_{BTC/USDT}}{\sigma_{ensemble}}$ where $\sigma$ is the standard
deviation of returns.  Figure 16: Performance of top 5 score-wise strategies
and a corresponding strategy ensemble on the future dataset. Risk adjustment
is performed through scaling the strategy ensemble returns by
$\frac{\sigma_{BTC/USDT}}{\sigma_{ensemble}}$ where $\sigma$ is the standard
deviation of returns.

Both the returns and risk-adjusted returns are increasing as the average past
score-performance increases.

##  7 Conclusions

Performances of created strategies increase in terms of return and risk-
adjusted return on the out-of-sample future dataset as the past score-
performance increases. Using top score-performance-wise strategies we achieved
exceptional market-beating results. As of right now, using a framework which
we have described can lead to further improvements of capital allocations of
institutional investors with access to market data and computational power.

##  8 Future work

Making transaction rates in the score function definition dataset dependent,
because static rates lead to ruling out potentially high performance
strategies if they do not comply with dataset’s dynamics. The optimal
transaction rate should be based on characteristics of the past dataset - i.e.
average volatility - and in general should not be hard-coded.

Changing strategies on the fly should further increase performance by swapping
under-performing single model strategies with more promising substitutes. In
this approach, the strategies are ranked for selection based on their up-to-
date past performances. The effective past dataset would change periodically.

More sophisticated feature space would potentially lead to better classifiers
and enable detection of sub-minute movements.

More sophisticated trading strategies \- our linear combination was selected
to reduce complexity. We now look into more complex solutions which are still
relatively easy to interpret.

Running computations for longer to find higher past score-performance-wise
strategies should further increase out-of-sample performances.

## Acknowledgement

We would like to thank David Klemm for his support, discussions, and the
access to his computational grid.

##  Appendix A Appendix: selected labels

The 8 chosen labels can be categorised into 2 subcategories: threshold labels
(see Labeling) and local extrema labels.

Threshold labels short descriptions:

  * •

1.2% price change in the next 5 minutes

  * •

1.2% price change in the next 60 minutes

  * •

2.2% price change in the next 2 minutes

  * •

3% price change in the next 5 minutes

  * •

3% price change in the next 60 minutes

The remaining 3 local extrema labels are a custom construct and are a material
for a separate paper. Visualise and explore them all through our repository:
GitHub Labels

##  Appendix B Appendix: feature space

The feature space (see Market representation) we use consists of 28 functions
and is illustrated on the Fig. 17 below:

Definition of our feature space .

Figure 17: Definition of each of the 28 features and their corresponding
parameter ranges that were used during our search for optimal market
representations in Market representation. Parameters are integer only and
represent minutes.

Ranges of possible parameters and types of indicators are based on our domain
knowledge.

##  Appendix C Appendix: strategy space

The strategy space (see Trading strategy) consists of 19 parameters: 3
thresholds and 16 weights.
Strategy thresholds: $y_{buy}\in[0.7,1]$, $y_{sell}\in[0,0.3]$,
$y_{width}\in[0.02,0.1]$.
Weights: $w_{n}\in[-1,1]$.

## References

  * [1] Lisha Li et al.  “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization”  In _J. Mach. Learn. Res._ 18 JMLR.org, 2017, pp. 6765–6816
  * [2] Marcos Lopez Prado  “Advances in Financial Machine Learning”  Wiley Publishing, 2018
  * [3] Marcos López Prado  “Interpretable Machine Learning: Shapley Values (Seminar Slides” Available at SSRN:, 2020  DOI: 10.2139/ssrn.3637020
  * [4] L.S. Shapley  “A value for n-person games, vol II of Contributions to the theory of games”  Princeton: Princeton University Press, 1953
  * [5] Jasper Snoek, Hugo Larochelle and Ryan P Adams  “Practical Bayesian Optimization of Machine Learning Algorithms”  In _Advances in Neural Information Processing Systems 25_ Curran Associates, Inc., 2012, pp. 2951–2959  URL: http://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf
  * [6] Erik Štrumbelj and Igor Kononenko  “Explaining prediction models and individual predictions with feature contributions”  In _Knowledge and Information Systems_ 41.3, 2014, pp. 647–665  DOI: 10.1007/s10115-013-0679-x