# Testbeam results of irradiated SiGe BiCMOS monolithic silicon pixel detector
without internal gain layer

T. Moretti  M. Milanesio  R. Cardella  T. Kugathasan  A. Picardi  I.
Semendyaev  M. Elviretti  H. Rücker  K. Nakamura  Y. Takubo  M. Togawa  F.
Cadoux  R. Cardarelli  L. Cecconi  S. Débieux  Y. Favre  C. A. Fenoglio  D.
Ferrere  S. Gonzalez-Sevilla  L. Iodice  R. Kotitsa  C. Magliocca  M. Nessi
A. Pizarro-Medina  J. Sabater Iglesias  J. Saidi  M. Vicente Barreto Pinto  S.
Zambito  L. Paolozzi  and G. Iacobucci11footnotetext: Corresponding author.

###### Abstract

Samples of the monolithic silicon pixel ASIC prototype produced in 2022 within
the framework of the Horizon 2020 MONOLITH ERC Advanced project were
irradiated with 70 MeV protons up to a fluence of $1\times 10^{16}$
n${}_{\text{eq}}$/cm2, and then tested using a beam of 120 GeV/c pions. The
ASIC contains a matrix of 100 $\mu$m pitch hexagonal pixels, readout out by
low noise and very fast frontend electronics produced in a 130 nm SiGe BiCMOS
technology process. The dependence on the proton fluence of the efficiency and
the time resolution of this prototype was measured with the frontend
electronics operated at a power density between 0.13 and 0.9 W/cm2. The
testbeam data show that the detection efficiency of 99.8% measured at sensor
bias voltage of 200 V before irradiation becomes 96.5% after a fluence of
$1\times 10^{16}$ n${}_{\text{eq}}$/cm2. An increase of the sensor bias
voltage to 300 V provides an efficiency to 99.5% at that proton fluence. The
timing resolution of 20 ps measured before irradiation rises for a proton
fluence of $1\times 10^{16}$ n${}_{\text{eq}}$/cm2 to 48 and 44 ps at HV = 200
and 300 V, respectively.

##  1 Introduction

To cope with the large event pile up foreseen during the CERN LHC High-
Luminosity data-taking period, the experiments showed evidence for the need to
upgrade the present detectors with layers with timing capability of the order
of tens of picoseconds. The major LHC Collaborations foresee the addition of
timing layers [1, 2] to match the required performance. One choice is to build
these timing layers with mm2 silicon pads based on the low-gain avalanche
detectors (LGAD) [3], which feature an internal gain layer under the pixel.

The particle-physics community is presently attempting to develop a new
generation of silicon sensors able to achieve both high spatial granularity
and excellent timing capabilities [4, 5], although the radiation tolerance of
the gain layer still constitutes a problem to place these sensors at small
radii in present and future high-energy hadron colliders, where radiation will
be very high. Studies to overcome the limited radiation tolerance of the gain
layer are ongoing [6, 7, 8]. A particularly interesting approach is to use a
resistive layer that spreads the signal among four adjacent pixels and, by
reconstructing the hit position from those signals, reduces drastically the
number of detection channels needed to have spatial resolutions at the level
of 10 $\mu$m, at the price of a reduction of the timing performance by
approximately a factor of two [9, 10, 11].

The MONOLITH Horizon 2020 ERC Advanced project utilises the SG13G2 130 nm SiGe
BiCMOS process by IHP to produce low noise, low power and very fast frontend
electronics, implemented in a fully efficient high granularity monolithic
sensor able to provide excellent timing. The foundry masks of the first
prototype of the MONOLITH project [12] were used to produce a proof-of-concept
picosecond avalanche detector (PicoAD) [13], a novel detector that implements
a continuous deep gain layer [14]. At a power density of 2.7 W/cm2, this
proof-of-concept monolithic ASIC provided full efficiency and an average time
resolution of 17 ps, varying between 13 ps at the center of the pixel and 25
ps in the inter-pixel region [15].

A second prototype of the MONOLITH project containing an improved electronics
[16] was produced in 2022. Before the production of the next generation PicoAD
prototypes using special wafers with deep gain-layer implant, the foundry
masks of this second prototype were used to realise a detector without
internal gain layer, implementing the sensor on a 50 $\mu$m thick epilayer of
350 $\Omega$cm resistivity. Several samples of this ASIC were irradiated at
the CYRIC facility [17] in Japan with 70 MeV protons up to a fluence of
$1\times 10^{16}$ n${}_{\text{eq}}$/cm2. The pixel matrix contains four analog
pixels consisting of a fast charge amplifier in SiGe HBT and a two-stages
analog driver. The time jitter of the single-ended output of the analog pixels
was measured using a 90Sr radioactive source [18]. In this paper, we present
the efficiency and time resolution before and after proton irradiation
measured with this prototype without gain layer using a beam of pions of 120
GeV/c at the CERN SPS.

##  2 Data samples and experimental set-up

Given the limited time availability during the testbeam experiment, data were
taken only with four irradiated boards out of the seven characterised in [18].
In addition, data were taken also with the board not irradiated previously
characterised in [16]. Table 1 gives a summary of the power density of the
frontend and of the sensor bias voltage of the 18 datasets taken at the
testbeam. All boards were operated at a temperature of -10 ∘C and at a
frontend feedback current of 2.0 $\mu$A.

| Proton Fluence
---
[1 MeV n${}_{\text{eq}}$/cm2]
$P_{\it{density}}$ [W/cm2] | High Voltage [V]
0 | 0.13 | 0.54 | 0.9 | 200
$9\times 10^{13}$ | 0.13 | 0.54 | 0.9
$6\times 10^{14}$ | 0.13 | 0.54 | 0.9
$3\times 10^{15}$ | 0.13 | 0.54 | 0.9
$1\times 10^{16}$ |  | 0.13 | 0.54
---|---
250
0.9 | 150 | 200 | 250 | 300
Table 1:  Power density and sensor bias voltage of the 18 data samples taken
at the CERN SPS testbeam with the four irradiated boards and with the board
not irradiated. The proton-fluence values are reported in the first column in
1 MeV n${}_{\text{eq}}$/cm2 values.

The UNIGE FE-I4 telescope [19] was used to provide external tracking. The
device under test (DUT) was mounted at the center of the telescope as
explained in [16]. Two MCP detectors were positioned downstream of the
telescope and used as time reference. The analog outputs of the two MCP
detectors and of the DUT were read by two fast oscilloscopes. The first one
with a sampling rate of 20 GS/s and an analog bandwidth set to 4 GHz for the
channels connected to the MCPs and 1 GHz for the channels connected to the
pair of single-ended analog output of the pixel used to measure the DUT time
resolution. Two other pairs of single-ended output of adjacent pixels were
connected to the second oscilloscope with a sampling rate of 10 GS/s and an
analog bandwidth set to 1 GHz. The DUT was mounted on a cooling plate and
covered by an insulating box to keep the ASIC at a constant temperature of -10
∘C.

The waveforms acquired for the positive and negative single-ended signals of
the DUT were subtracted offline, sample point by sample point, to construct
differential signals. Throughout the analysis, only the differential signals
were used since they provide better signal-to-noise ratio.

##  3 Detection efficiency

The detection efficiency was computed using pion tracks reconstructed by the
FE-I4 telescope that produced hits in the six telescope planes and with a
$\chi^{2}$/NDF $\leq$ 1.5. Since the number of channels available in the two
oscilloscopes allowed recording of the signals from only two out of six
neighboring analog pixels, the calculation of the efficiency was restricted to
the telescope tracks whose extrapolation on the pixel surface was inside the
region defined by the triangle connecting the center of the three pixels (the
pixel under study and the two adjacent pixels that were read out). This
triangle represents in the correct proportions all the areas of a pixel, and
provide a result unbiased by the pointing resolution of the telescope which
have been observed to show artificial inefficiency at the edge of pixels if
the adjacent pixel is not read out [16].

Figure 1:  Detection efficiency measured at a power density $P_{\it{density}}$
= 0.9 W/cm2 as a function of the voltage threshold. The data refer to the
board not irradiated operated at HV = 200 V (open red squares), and to the
board irradiated at $1\times 10^{16}$ n${}_{\text{eq}}$/cm2 operated at HV =
300 V (full red squares). The voltage threshold is given in units of the
voltage noise $\sigma_{\it{V}}$. The plot reports also the noise-hit rate
(values reported on the right-hand-axis scale) measured with the two boards.

Figure 1 shows the efficiency as a function of the voltage noise
$\sigma_{\it{V}}$, in the case of the board not irradiated operated at a
sensor bias voltage of 200 V, and of the board irradiated at $1\times 10^{16}$
n${}_{\text{eq}}$/cm2 operated at 300 V. The voltage noise $\sigma_{\it{V}}$
was measured event-by-event using the waveform samplings recorded in a time
interval of 200 ns preceding the signal. At the nominal threshold value
$V_{\it{th}}$ = 7 $\sigma_{\it{V}}$, the ASIC not irradiated provides a
detection efficiency of (99.96${}^{~{}\\!+0.01}_{-0.02}$)% and, given the
large signal-to-noise ratio, it maintains an efficiency at the level of 99.8%
even with a threshold value of 14 $\sigma_{\it{V}}$. On the contrary, in the
case of the ASIC that was exposed to $1\times 10^{16}$ n${}_{\text{eq}}$/cm2,
since the signal-to-noise ratio diminishes, the efficiency was found to depend
more on the threshold, and at HV = 300 V varies from 99.7% at a threshold
$V_{\it{th}}$ = 7 $\sigma_{\it{V}}$ to approximately 97.0% already at
$V_{\it{th}}$ = 10 $\sigma_{\it{V}}$. Figure 1 also reports the noise-hit rate
as a function of the threshold. The noise-hit rate shows the expected
exponential drop. It increases by a factor of approximately five after the
ASIC has received a proton fluence of $1\times 10^{16}$ n${}_{\text{eq}}$/cm2.

| Proton Fluence
---
[1 MeV n${}_{\text{eq}}$/cm2]
| $P_{\it{density}}$
---
[W/cm2]
| High
---
Voltage
[V]
| Detection
---
Efficiency
[%]
| Time
---
Resolution
[ps]
0 | 0.13 | 200 | 99.94${}^{~{}\\!+0.03}_{-0.05}$ | 31.0 $\pm$ 1.6
0.54 | 99.96${}^{~{}\\!+0.02}_{-0.03}$ | 29.0 $\pm$ 1.3
0.9 | 99.96${}^{~{}\\!+0.01}_{-0.02}$ | 20.0 $\pm$ 1.0
$9\times 10^{13}$ | 0.13 | 200 | 99.12${}^{~{}\\!+0.10}_{-0.19}$ | 37.8 $\pm$ 2.8
0.54 | 99.63${}^{~{}\\!+0.07}_{-0.13}$ | 29.0 $\pm$ 1.3
0.9 | 99.78${}^{~{}\\!+0.04}_{-0.11}$ | 24.7 $\pm$ 1.0
$6\times 10^{14}$ | 0.13 | 200 | 98.38${}^{~{}\\!+0.17}_{-0.19}$ | 42.4 $\pm$ 3.0
0.54 | 99.13${}^{~{}\\!+0.12}_{-0.13}$ | 29.9 $\pm$ 2.2
0.9 | 99.24${}^{~{}\\!+0.10}_{-0.11}$ | 26.4 $\pm$ 1.5
$3\times 10^{15}$ | 0.13 | 200 | 94.06${}^{~{}\\!+0.22}_{-0.23}$ | 75.9 $\pm$ 4.5
0.54 | 97.22${}^{~{}\\!+0.17}_{-0.18}$ | 59.4 $\pm$ 0.9
0.9 | 97.97${}^{~{}\\!+0.11}_{-0.11}$ | 48.3 $\pm$ 1.9
$1\times 10^{16}$ | 0.13 | 250 | 94.74${}^{~{}\\!+0.18}_{-0.19}$ | 74.0 $\pm$ 5.4
0.54 | 98.33${}^{~{}\\!+0.10}_{-0.11}$ | 48.3 $\pm$ 1.9
0.9 | 150 | 84.11${}^{~{}\\!+0.53}_{-0.54}$ | 60.4 $\pm$ 3.3
200 | 96.18${}^{~{}\\!+0.31}_{-0.33}$ | 53.1 $\pm$ 3.4
250 | 98.75${}^{~{}\\!+0.09}_{-0.10}$ | 50.2 $\pm$ 1.6
300 | 99.69${}^{~{}\\!+0.06}_{-0.07}$ | 45.3 $\pm$ 1.6
Table 2: Detection efficiency and time resolution measured for the 18 data
samples taken at the CERN SPS testbeam with the four irradiated boards and
with the board not irradiated. The values were obtained for a signal amplitude
threshold $V_{\it{th}}$ > 7 $\sigma_{\it{V}}$.

Table 2 reports the detection efficiencies obtained at a threshold
$V_{\it{th}}$ = 7 $\sigma_{\it{V}}$ for the 18 datasets listed in table 1.

Figure 2 shows the detection efficiency of the ASIC irradiated at $1\times
10^{16}$ n${}_{\text{eq}}$/cm2 as a function of the sensor bias voltage. Only
one data set at HV = 200 V was acquired at the testbeam with the board not
irradiated. The data taken with the board not irradiated in a previous
testbeam [16] are superimposed in figure 2 as open black squares for
comparison222It should be noted that, in addition to the different power
density, the data from [16] were acquired at feedback current $i_{\it
feedback}$ = 0.1 $\mu$A and at a temperature of 20∘C in contrast with the 2.0
$\mu$A and -10∘C of the present data. This change in operating conditions
results in an increase of the signal-to-noise ratio, that leads to better
efficiency and time resolution.. The measurement shows that after $1\times
10^{16}$ n${}_{\text{eq}}$/cm2 a bias voltage of 300 V is just enough to reach
the detection efficiency plateau. This behaviour is completely different from
that of the unirradiated board, for which the efficiency plateau is reached
already at HV = 120 V.

Figure 2:  Detection efficiency measured as a function of the sensor bias
voltage at a threshold value $V_{\it{th}}$ = 7 $\sigma_{\it{V}}$. The red dots
show the results obtained with the board irradiated at $1\times 10^{16}$
n${}_{\text{eq}}$/cm2, while the blue square that obtained with the board not
irradiated, all operated at $P_{\it{density}}$ = 0.9 W/cm2. The results
obtained with the board not irradiated at $P_{\it{density}}$ = 2.7 W/cm2
reported in [16] are superimposed as black open squares.

This measurement, as well as the drop in detection efficiency shown in figure
1, could in large part be attributed to the change in resistivity of the
silicon active volume, that we estimated to vary from the initial 350
$\Omega$cm to approximately 50 $\Omega$cm extrapolating the data of [20, 21].
At such resistivity, the 50 $\mu$m thick sensor would not be fully depleted. A
concurrent radiation damage of the frontend electronics cannot be excluded,
and should be measured irradiating the SiGe HBT alone.

Figure 3: Detection efficiency measured for power density between 0.13 and 0.9
W/cm2 for a threshold value of $V_{\it{th}}$ = 7 $\sigma_{\it{V}}$. The
results obtained with the sample not irradiated (black empty squares) are
presented with those of the four samples irradiated with fluences between
$9\times 10^{13}$ n${}_{\text{eq}}$/cm2and $1\times 10^{16}$
n${}_{\text{eq}}$/cm2, all operated with a sensor bias voltage of HV = 200 V.
In the case of the board irradiated at $1\times 10^{16}$ n${}_{\text{eq}}$/cm2
the data taken at sensor bias voltage of 250 and 300 V are also shown. The
marker relative to the data point at HV = 300 V (red star) is displayed with a
small horizontal offset to avoid overlapping with other markers.

Figure 3 shows the impact on the detection efficiency of the power density at
which the frontend was operated. At a sensor bias voltage HV= 200 V, a steady
increase of the efficiency is measured as $P_{\it{density}}$ increases. This
effect is more pronounced at the highest proton fluences.

In the case of the board irradiated to $1\times 10^{16}$
n${}_{\text{eq}}$/cm2, at HV = 200 V data were taken only at
$P_{\it{density}}$ = 0.9 W/cm2, which result in a detection efficiency of
(96.2 $\pm$ 0.3)% (red circle in the figure). For this board the scan in
$P_{\it{density}}$ was performed at HV = 250 V, and the results are displayed
in figure 3 by the red crosses. At HV = 300 V, only a dataset at
$P_{\it{density}}$ = 0.9 W/cm2 was acquired (red star in the figure), which
provides a detection efficiency of (99.69${}^{~{}\\!+0.06}_{-0.07}$)%.

Finally, the detection efficiency is shown as a function of the distance from
the centre of the pixel in figure 4. All the data points shown refer to a
sensor bias voltage of 200 V, with the exception of the dataset at $1\times
10^{16}$ n${}_{\text{eq}}$/cm2, which was obtained at 300 V. While at no or
small irradiation values the efficiency does not depend on the position within
the pixel area, starting from a fluence of 6 $\times$ 1014
n${}_{\text{eq}}$/cm2 the efficiency was measured to drop in the inter-pixel
area, although an increase of the sensor bias voltage allows recovering of the
efficiency far from the pixel center, as the data at HV = 300 V (red stars)
show.

Figure 4: Detection efficiency as a function of the distance from the pixel
center. The data refer to power density $P_{\it{density}}$ = 0.9 W/cm2,
voltage threshold $V_{\it{th}}$ = 7 $\sigma_{\it{V}}$ and sensor bias voltage
HV = 200 V, with the exception of the data taken at $1\times 10^{16}$
n${}_{\text{eq}}$/cm2, for which the sensor bias voltage was 300 V. The
efficiency was computed in the following bins of distance from the pixel
center in microns: $[0-21],[21-30],[30-37],[37-44],[44-51],[51-65]$. In each
bin, the data points are plotted at the mean value of the track distance from
the pixel center.

##  4 Time resolution

The time resolution of the DUT was measured using as reference the timestamp
of two precise MCP detectors that were located outside the FE-I4 telescope
downstream the pion beam. The sample of telescope tracks used for the
measurement of the time resolution was the same used for the detection
efficiency, with the only exception that all the tracks within the pixel were
considered here, and not exclusively the tracks within the aforementioned
triangle defined by the three analog pixel centers.

The time of arrival (TOA) was measured for the DUT and the MCPs making offline
a linear interpolation between the oscilloscope samplings and taking the time
at which the signals reached 50% of the maximum value of the signal amplitude.
The difference in TOA between the three pairs of detectors was then used to
extract the time resolution of the DUT, as explained in [16].

Taking the TOA at a fixed percentage of the signal amplitude has the advantage
to correct for most of the time walk between signals with different
amplitudes, under the assumption that signals of all amplitudes have
consistent rise times. It was observed that in our case this method carries
the drawback of creating a small distortion of the distribution of the TOA
difference, which deviates slightly from a Gaussian distribution and generates
a tail at large values of ${\rm TOA}_{\it DUT}$ \- ${\rm TOA}_{\it MCP}$.

Figure 5:  Difference in TOA between the DUT and one of the MCP detectors used
to provide a precise reference time. The left-hand panels show the results for
the board not irradiated, while the right-hand panels those for the board
irradiated to $1\times 10^{16}$ n${}_{\text{eq}}$/cm2. The top panels show the
data with superimposed the Gaussian fits extending to the first bin containing
20% of the maximum value of the distribution; the bottom panels show the same
data with superimposed the Gaussian fits extending to all the bins of the
distributions.

This effect is shown in figure 5 in the case of the board not irradiated (left
panels) and of the board irradiated at $1\times 10^{16}$ n${}_{\text{eq}}$/cm2
(right panels).

To account for the observed distortion in the TOA difference, the $\sigma$
values from the Gaussian fits obtained using only the bins of the distribution
larger than 20% of the bin with the highest number of entries (two top panels
in the figure 5) were used as central values of the time resolution. The
difference between these values and those obtained by fitting a Gaussian
functional form in the entire TOA-difference distributions (bottom panels in
figure 5) were used as systematic uncertainty. These uncertainties were summed
in quadrature with the statistical uncertainty coming from the Gaussian fits
that were used as central values, and are shown as vertical error bars in the
time resolution plots.

Figure 6:  Time resolution measured as a function of the sensor bias voltage.
The results obtained with the board irradiated to $1\times 10^{16}$
n${}_{\text{eq}}$/cm2 at $P_{\it{density}}$ = 0.9 W/cm2 (red circles) are
presented together with the result obtained with the board not irradiated at
HV = 200 V (blue square). The plot results obtained in a previous testbeam
[16] with the board not irradiated at $P_{\it{density}}$ = 2.7 W/cm2 and
$i_{\it feedback}$ = 0.1 $\mu$A are also superimposed (black open squares).

The last column of table 2 reports the time resolution obtained for the 18
datasets acquired at $V_{\it{th}}$ = 7 $\sigma_{\it{V}}$.

Figure 6 shows the time resolution obtained at $P_{\it{density}}$ = 0.9 W/cm2
for the board irradiated to $1\times 10^{16}$ n${}_{\text{eq}}$/cm2 for a
sensor bias voltage varying between 150 and 300 V, and the result obtained
with the unirradiated board at HV = 200 V. For comparison, the figure reports
as well the time resolution measured at $P_{\it{density}}$ = 2.7 W/cm2 with
the board not irradiated in previous testbeam experiment [16]. While the board
not irradiated reaches the plateau of time resolution already at HV = 160 V,
the time resolution measured with the irradiated board does not seem to have
yet reached a plateau value at 300 V.

Figure 7: Time resolution measured for three values of the power density
supplied to the preamplifier between 0.13 and 0.9 W/cm2. The plot shows the
results obtained at sensor bias voltage HV = 200 V with the five boards which
were subject to proton fluence between zero and $1\times 10^{16}$
n${}_{\text{eq}}$/cm2. The data taken with the board most irradiated at HV =
250 and 300 V are also shown.

The time resolution of the DUT was also measured as a function of the power
density supplied to the frontend electronics. The results are presented in
figure 7 at a sensor bias voltage HV = 200 V for all the boards, and for the
most irradiated board also at HV = 250 and 300 V. In all cases, the time
resolution was found to steadily improve with increasing $P_{\it{density}}$,
and degrade with increasing proton fluence. In the $P_{\it{density}}$ range
studied, it varies between 30 and 20 ps in the case of the board not
irradiated, and between 75 and 50 ps in the case of the board irradiated to
$1\times 10^{16}$ n${}_{\text{eq}}$/cm2.

It should be noted that, for each of the irradiation values studied, the ratio
of the time resolutions at $P_{\it{density}}$ = 0.13 and 0.9 W/cm2 is
constant, showing a relative improvement of approximately 35% independently of
the fluence, as reported in table 3.

Fluence [1 MeV n${}_{\text{eq}}$/cm2] |  | Time Resolution [ps]
---
Ratio
$P_{\it{density}}$ = 0.13 W/cm2 |  $P_{\it{density}}$ = 0.9 W/cm2
0 | 31.0 $\pm$ 1.6 | 20.0 $\pm$ 1.0 | 1.55 $\pm$ 0.15
$9\times 10^{13}$ | 37.8 $\pm$ 2.8 | 24.7 $\pm$ 1.0 | 1.53 $\pm$ 0.21
$6\times 10^{14}$ | 42.4 $\pm$ 3.0 | 26.4 $\pm$ 1.5 | 1.61 $\pm$ 0.21
$3\times 10^{15}$ | 75.9 $\pm$ 4.5 | 48.3 $\pm$ 1.9 | 1.57 $\pm$ 0.17
$1\times 10^{16}$ | 74.0 $\pm$ 5.4 | 50.2 $\pm$ 1.6 | 1.57 $\pm$ 0.19
Table 3:  Time resolution measured with the five boards at $P_{\it{density}}$
= 0.13 and 0.9 W/cm2 and their ratio. The data were taken at sensor bias
voltage HV = 200 V for all boards except the board irradiated to $1\times
10^{16}$ n${}_{\text{eq}}$/cm2 for which the sensor bias voltage was 250 V.

This is an indication that the dependence of the transition frequency from the
collector current, which is observed in [22] not to change up to 1 $\times$
1014 n${}_{\text{eq}}$/cm2, might not be affected by radiation damage even at
$1\times 10^{16}$ n${}_{\text{eq}}$/cm2.

Figure 8:  Time resolution as a function of the proton fluence for
$P_{\it{density}}$ = 0.9 W/cm2. The blue dots show the results obtained at a
sensor bias voltage HV = 200 V, while the red star shows the result obtained
at HV = 300 V with the board irradiated to $1\times 10^{16}$
n${}_{\text{eq}}$/cm2.

Figure 8 shows the time resolutions as a function of the proton fluence at
$P_{\it{density}}$ = 0.9 W/cm2. The results obtained with the datasets taken
at sensor bias voltage HV = 200 V vary between 20 ps for the board not
irradiated and 53 ps for the board irradiated to $1\times 10^{16}$
n${}_{\text{eq}}$/cm2. For this last board, an increase of the sensor bias
voltage to 300 V provides a time resolution of 45 ps, thus improving it by
approximately 10%.

The time resolution is shown as a function of the distance from the centre of
the pixel in figure 9. All the data refer to a sensor bias voltage of 200 V,
with the exception of the dataset at $1\times 10^{16}$ n${}_{\text{eq}}$/cm2,
which was acquired at 300 V. For all fluences the time resolution is smaller
in the central region of the pixel than in the inter-pixel region, where it
increases approximately 5 ps.

Figure 9: Time resolution as a function of the distance from the pixel center.
The data refer to power density $P_{\it{density}}$ = 0.9 W/cm2, voltage
threshold $V_{\it{th}}$ = 7 $\sigma_{\it{V}}$ and sensor bias voltage HV = 200
V, with the exception of the data taken at $1\times 10^{16}$
n${}_{\text{eq}}$/cm2, for which the sensor bias voltage was 300 V. The time
resolution was computed in the same bins of figure 4.

##  5 Summary and Conclusions

A monolithic silicon pixel prototype produced for the Horizon 2020 MONOLITH
ERC Advanced project was irradiated up to a proton fluence of $1\times
10^{16}$ n${}_{\text{eq}}$/cm2. The prototype contains a matrix of hexagonal
pixels with 100 $\mu$m pitch, read by a SiGe BiCMOS 130 nm frontend
electronics. The sensor was produced on a 50 $\mu$m epilayer of a resistivity
of 350 $\Omega$cm. The analog channels present in the pixel matrix were read
by a fast oscilloscope and tested at the CERN SPS using a beam of 120 GeV/c
pions. Only signals with an amplitude larger than a voltage threshold
$V_{\it{th}}$ = 7 $\sigma_{\it{V}}$ were considered for the measurement of the
detection efficiency and time resolution.

Before proton irradiation, the detection efficiency at a sensor bias voltage
HV = 200 V and frontend power density $P_{\it{density}}$ = 0.9 W/cm2 was
measured to be (99.96${}^{~{}\\!+0.01}_{-0.02}$)%. For a proton fluence
$1\times 10^{16}$ n${}_{\text{eq}}$/cm2, the detection efficiency in the same
conditions was measured to be (96.2 $\pm$ 0.3)%; an increase of the bias
voltage to 300 V provided (99.69${}^{~{}\\!+0.06}_{-0.07}$)%. These efficiency
values include also the inter-pixel regions. Therefore there is no inactive
area in between adjacent pixels within the sensor matrix.

The time of arrival of the signals used in the data analysis was defined as
the time at 50% of the signal amplitude, and was computed by a linear
interpolation of the oscilloscope samplings. No further time-walk correction
was applied to the data. The time resolution at $P_{\it{density}}$ = 0.9 W/cm2
and HV = 200 V was measured to be (20.0 $\pm$ 1.0) ps before proton
irradiation. At the highest proton fluence studied of $1\times 10^{16}$
n${}_{\text{eq}}$/cm2, the time resolution became (53.1 $\pm$ 3.4) ps. An
increase of the sensor bias voltage to HV = 300 V improved the result to (45.3
$\pm$ 1.6) ps.

The scan in sensor bias voltage of the board exposed to $1\times 10^{16}$
n${}_{\text{eq}}$/cm2, showed that the plateau in detection efficiency was
just reached at 300 V, while the time resolution did not yet reach its plateau
value. These observations, supported by an estimation of the change in
resistivity after $1\times 10^{16}$ n${}_{\text{eq}}$/cm2, imply that the
sensor was not yet fully depleted at 300 V and suggest that the nominal
working point at that proton fluence is beyond 300 V.

We conclude that very high efficiency, very high time resolution and radiation
tolerance can be obtained by monolithic pixelated silicon detectors without
internal gain layer that exploit low noise, low power and fast response SiGe
HBTs. Commercial SiGe BiCMOS foundry processes therefore represent a very
strong candidate to build monolithic silicon detectors for 4D tracking systems
at future colliders at a contained cost.

## Acknowledgments

This research is supported by the Horizon 2020 MONOLITH ERC Advanced Grant ID:
884447. The authors wish to thank Coralie Husi, Javier Mesa, Gabriel
Pelleriti, and the technical staff of the University of Geneva and IHP
Microelectronics. The authors acknowledge the support of EUROPRACTICE in
providing design tools and MPW fabrication services, as well as the CERN SPS
testbeam team.

## References

  * [1] ATLAS Collaboration.  Technical Design Report: A High-Granularity Timing Detector for the ATLAS Phase-II Upgrade.  CERN-LHCC-2020-007; ATLAS-TDR-031, June 2020.
  * [2] CMS Collaboration.  A MIP Timing Detector for the CMS Phase-2 Upgrade.  CERN-LHCC-2019-003; CMS-TDR-020, March 2019.
  * [3] G. Pellegrini et al.  Technology developments and first measurements of Low Gain Avalanche Detectors (LGAD) for high energy physics applications.  Nuclear Instruments and Methods Section A, 765:12–16, 2014.  Proceedings of the HSTD-9 2013 "Hiroshima" Symposium.
  * [4] Hartmut F-W Sadrozinski, Abraham Seiden, and Nicolò Cartiglia.  4D tracking with ultra-fast silicon detectors.  Reports on Progress in Physics, 81(2):026101, dec 2017.
  * [5] N. Cartiglia et al.  4D tracking: present status and perspectives.  Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 1040:167228, 2022.
  * [6] V. Sola et al.  A compensated design of the LGAD gain layer.  Nuclear Instruments and Methods Section A, 1040:167232, 2022.
  * [7] P. Asenov et al.  Tcad modeling of bulk radiation damage effects in silicon devices with the perugia radiation damage model.  Nuclear Instruments and Methods Section A, 1040:167180, 2022.
  * [8] T. Croci et al.  TCAD optimization of LGAD sensors for extremely high fluence applications.  Journal of Instrumentation, 18(01):C01008, jan 2023.
  * [9] R. Arcidiacono et al.  High-Precision 4D Tracking with Large Pixels using Thin Resistive Silicon Detectors.  submitted to NIMA, nov 2022.
  * [10] S. Kita et al.  Optimization of capacitively coupled Low Gain Avalanche Diode (AC-LGAD) sensors for precise time and spatial resolution.  Nuclear Instruments and Methods Section A, 1048:168009, 2023.
  * [11] T. Imamura et al.  Development of HPK Capacitive Coupled LGAD (AC-LGAD) detectors.  Proceedings of the 32nd International Workshop on Vertex Detectors (VERTEX2023) - New developments: Timing, accepted for publication, 448, 2023.
  * [12] G. Iacobucci et al.  Efficiency and Time Resolution of Monolithic Silicon Pixel Detectors in SiGe BiCMOS Technology.  Journal of Instrumentation, 17(02):P02019, Feb. 2022.
  * [13] G. Iacobucci, L. Paolozzi, and P. Valerio.  Multi-Junction Pico-Avalanche Detector.  European Patent EP3654376A1, US Patent US2021280734A1, Nov. 2018\.
  * [14] L. Paolozzi et al.  Picosecond Avalanche Detector – Working Principle and Gain Measurement With a Proof-of-Concept Prototype.  Journal of Instrumentation, 17(10):P10032, Oct. 2022.
  * [15] G. Iacobucci et al.  Testbeam Results of the Picosecond Avalanche Detector Proof-of-Concept Prototype.  Journal of Instrumentation, 17(10):P10040, Oct. 2022.
  * [16] S. Zambito et al.  20 ps time resolution with a fully-efficient monolithic silicon pixel detector without internal gain layer.  Journal of Instrumentation, 18(03):P03047, March 2023.
  * [17] K. Nakamura et al.  Irradiation and testbeam of kek/hpk planar p-type pixel modules for hl-lhc.  10(06):C06008, jun 2015.
  * [18] M. Milanesio et al.  Radiation tolerance of sige bicmos monolithic silicon pixel detectors without internal gain layer.  Journal of Instrumentation, 18(03):P03047, oct 2023.
  * [19] M. Benoit et al.  The FE-I4 telescope for particle tracking in testbeam experiments.  Journal of Instrumentation, 11(07):P07003–P07003, jul 2016.
  * [20] M. Bruzzi.  Radiation damage in silicon detectors for high-energy physics experiments.  IEEE Transactions on Nuclear Science, 48(4):960–971, 2001.
  * [21] Michael Moll.  Displacement damage in silicon detectors for high energy physics.  IEEE Transactions on Nuclear Science, 65(8):1561–1582, 2018.
  * [22] John D. Cressler.  Radiation effects in sige technology.  IEEE Transactions on Nuclear Science, 60(3):1992–2014, 2013.


11institutetext: Center for Computational Engineering and Networking, Amrita
School Of Engineering, Amrita vishwa vidyapeetham, Coimbatore, India. 11email:
<EMAIL_ADDRESS>22institutetext: Graduate School, Computer Science
Department,
Texas State University. 22email<EMAIL_ADDRESS>Division of Biomedical Informatics, Cincinnati Children’s Hospital Medical
Centre, Cincinnati, OH, United States.
33email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Deep Learning Approach for Enhanced Cyber Threat Indicators in Twitter
Stream

Simran K 11 Prathiksha Balakrishna 22 Vinayakumar R 33 1 1 Soman KP 11

###### Abstract

In recent days, the amount of Cyber Security text data shared via social media
resources mainly Twitter has increased. An accurate analysis of this data can
help to develop cyber threat situational awareness framework for a cyber
threat. This work proposes a deep learning based approach for tweet data
analysis. To convert the tweets into numerical representations, various text
representations are employed. These features are feed into deep learning
architecture for optimal feature extraction as well as classification. Various
hyperparameter tuning approaches are used for identifying optimal text
representation method as well as optimal network parameters and network
structures for deep learning models. For comparative analysis, the classical
text representation method with classical machine learning algorithm is
employed. From the detailed analysis of experiments, we found that the deep
learning architecture with advanced text representation methods performed
better than the classical text representation and classical machine learning
algorithms. The primary reason for this is that the advanced text
representation methods have the capability to learn sequential properties
which exist among the textual data and deep learning architectures learns the
optimal features along with decreasing the feature size.

###### Keywords:

Information Extraction Twitter Cyber Security Deep learning.

##  1 Introduction

As social media is an interactive platform where individuals share thoughts,
information, professional interests and different types of expression via
virtual communities and systems, it introduces a rich and timely source of
information on events occurring everywhere throughout the world [13]. Social
media giants like Facebook, Twitter, WhatsApp, etc enable a lot of
applications like recognizing the area of missing people during catastrophic
events or earthquake detection. Past work on event extraction has depended on
a large amount of labeled information or taken an open-domain approach in
which general events are extracted without a particular core interest.
Information analyst is often interested in tracking a very specific type of
event, for example, data breaches or account hijacking and probably won’t have
time or expertise to build an information extraction framework from scratch in
response to emerging incidents. To address this challenge we introduce a deep
learning approach for rapid training cyber threat indicators for the Twitter
stream.

Open-Source Intelligence (OSINT) provides a vital source of information and
has proven to be an important asset for Cyber Threat Intelligence (CTI). One
of OSINT rich sources is Twitter. Twitter’s popularity in the society provides
an environment for defensive and offensive Cyber Security practitioners to
debate, and promote timely indicators of different type of cyber events such
as attacks, malware, vulnerabilities, etc. Various initial reports of recent
major cyber events like the exposure of multiple “0-day”Microsoft Windows
vulnerabilities, exposure of ransomware campaigns [1] and user reports on DDoS
attacks [2] exhibits the value of Twitter data to CTI analysts.

Multiple frameworks for detecting as well as analysing the treat indicators in
Twitter stream have come from the research on Twitter-based OSINT collection.
However, most of these proposals have a high false-positive rate in detecting
the relevant tweets as they are using heavily manual heuristics like keyword
lists that are relevant to Cyber Security are used to detect and filter
tweets. Furthermore, potentially valuable information in tweets is getting
neglected by the emergence of new terminology and flexible typography. In
recent days, the applications of deep learning with natural language
processing methods leveraged in various Cyber Security tasks [13, 14, 15, 16,
17, 18, 19, 20]. These methods have obtained good performance and most
importantly, these methods performed well compared to the classical machine
learning classifiers.

The major contribution of this proposed work are given below:

  1. 1.

This work proposes a deep learning based framework for cyber threat indicators
in the Twitter stream. The framework is highly scalable on using commodity
hardware.

  2. 2.

To identify a proper tweet representation, various state-of-the-art text
representation exists in the domain of natural language processing (NLP) are
leveraged for cyber threat indicators in Twitter Stream.

  3. 3.

To identify an optimal machine learning approach, we have carried out a
comprehensive and in-depth study of the application of classical machine
learning and deep learning theory in the context of cyber threat indicators in
Twitter stream.

  4. 4.

In particular, we discuss several parameterization options for classical
machine learning, deep learning, and tweet representation and we present a
large variety of benchmarks which have been used to either experimentally
validate our choices or to help us to take the adequate decision.

The remaining of this paper is arranged in the following order: Section 2
documents a survey of the related literature, followed by background related
to NLP and deep learning concepts in Section 3. Section 4 provides a
description of Cyber Security related tweets data set used in this work.
Section 5 describes the details of the proposed architecture. Section 6
reports the experiments and observations made by the proposed architecture.
Section 7 concludes the paper as well as tells the remakes on future work of
research.

##  2 Literature Survey

Classification and detection of CTI extraction from Twitter are less
investigated compared to the other area, for example, crime prevention [3],
identification of cyber-bullies [4], and disaster response [5]. To distinguish
three sorts of threats and events ie., account hijacking, data breaches, and
Distributed Denial of Service (DDoS) attacks, Khandpur et al. [6] proposed an
architecture to separate cyber threat as well as security information from the
Tweets. Target domain generation, event extraction, and dynamically typed
query expansion are the three major segments of this framework. This
methodology is powerful as it abuses syntactic, semantic analysis and
dependency tree graph yet it requires the persistent tracking of features for
each type of threat. It likewise requests a high computational overhead to
produce as well as keep the focus on corpus space of tweet content for query
extension. Also, this architecture can’t flawlessly stretch out to more
classifications of threats and events.

Furthermore, categorizing Cyber Security events from tweets was proposed by Le
Sceller et al. [7]. The detection of events uses the taxonomy of Cyber
Security and a set of keywords that describes the event type. Expanding of the
set of seed keywords are performed by not only identifying but also attaching
new words with comparative meaning with regards to word embeddings utilizing a
physically indicated edge in the cosine similarity distance between word
vectors. Term frequency - inverse document frequency (TF-IDF) method which
produced events as groups of tweets was used in this framework. Inadvertent
biasing effects of the initial seed keywords caused this algorithm to give a
high false-positive rate.

Security Vulnerability Concept Extractor (SVCE) was utilized to process tweets
in the structure proposed in [8]. SVSE is trained on a data set containing
reports of the national vulnerability database to recognize as well as label
the terms and ideas identified with CTI, for example, affected software,
consequences of the attack and so forth. To additionally improve the extracted
information, the ideas and substances extracted by SVCE are examined dependent
on outside freely accessible semantic learning bases such as DBPedia. The
client needs to specify a target framework profile included data about
installed software or hardware, as this system is produced for client-based
applications. As per the information provided, an ontology is produced and
utilized alongside SWRL rules to address as well as organize time-delicate CTI
entries. Later conversion from separated and labeled CTIs to RDF triple
proclamations is finished. The ready alert system can reason over the
information as RDF connected information portrayal is put away in the
knowledge base. This framework is incapable of distinguishing novel threat
types and indicators.

In [9], ontology-based technique and Named Entity Recognition (NER) were
utilized to classify tweets as related events or not related events. This
framework performs topic identification by means of cross-referencing NER
results with other external knowledge bases, for example, DBPedia utilizing
Wikipedia’s Current Event Portal just as human info gathered using Amazon
Mechanical Turk, produced an annotated data set of tweet event type and CTI.
Different machine learning algorithms, for example, naive bayes, support
vector machines (SVM) and deep learning architectures such as long short term
memory (LSTM), recurrent neural network (RNN) used this annotated data set and
the best outcome was delivered by LSTM architecture with word embedding. They
additionally show that particular classifications of NER are useful in
classifying the classes as well as event type, though the nonexclusive class
of NER is useful in binary classification. Pagerank algorithm was used in this
work for topic recognizable proof.

[10] proposed a framework which recognizes influential user or community of
people to prioritize CTI information. This was finished utilizing a scoring
strategy that is scores were given to the user and community who produced CTI-
related tweets. This work has four segments. For gathering information from
the Twitter platform, a social media connector is referred as the principal
segment. The second segment is a module for recognizing and stretching out the
rundown of specialists to discover developing themes. The third segment
comprises of weight contribution and fitness calculation. Lastly, to recognize
emerging threats the author proposed a topic detection algorithm. Anyhow,
threat indicators are not adequately referred by the specialists in this work.

The framework proposed in [11] is a weekly supervised learning approach that
trains a model for extracting new classes of Cyber Security events. This
framework does extraction by seeding a little amount of positive event tests
over a fundamentally amount of unlabeled data. The target to learn in this
work is done by regularizing the label distribution over the unlabeled
distribution. This work is vigorously reliant on historical seed and neglects
to give the details of coordinating named entities into an event category.

##  3 Background

###  3.1 Text representation

To represent the tweet into numeric form, we used various text representations
in this works. The basic idea behind these text representations is discussed
below.

####  3.1.1 Bag-of-Words (TDM, TF-IDF):

Bag-of-words is basically a collection of words. So the texts (tweets) are
represented as a bag of its words. Every unique word passed as an input will
have a position in this bag (vector). The vector records the frequency of the
words in the tweets. Term document matrix (TDM) and Term frequency-inverse
document frequency (TF-IDF) are features extracted from the documents. They
are the measures used to understand the similarities between the tweets. TDM
will have each corpus word as rows and document (tweet) as columns. The matrix
represents the frequencies of the words occurring in that particular tweet.
The most used words are highlighted because of their high frequency. TF-IDF
tells how frequently a word occurs in a specific record contrasted with the
whole corpus. The uncommon words are featured to demonstrate their relative
significance.

####  3.1.2 N-gram:

N-gram is a contiguous order of n items from a given sample of content
(tweets). N-gram with $N=1$ is known as a unigram and it takes one
word/character at once. $N=2$ and $N=3$ are called bigram and trigram
respectively and will take two and three words/characters at a time. If n
words/characters are to be taken at once then N will be equal to n.

####  3.1.3 Keras Embedding:

Word Embedding basically converts words into a dense vector of real numbers in
such a way that sequence and word similarities are additionally safeguarded.
Keras offers an Embedding layer which is initialized with random weights. It
will learn embedding of all the words in the training set but the input word
should be represented by a unique integer. Keras is an open-source neural
network library which contains various executions of generally utilized neural
network building blocks. It also supports convolutional, recurrent neural
networks and other common utility layers like pooling, batch normalization,
and dropout.

####  3.1.4 fastText:

fastText chips away at character n-gram level instead of just word level
(word2vec) and it is better for morphologically rich dialects. To convert
words into vectors it utilizes skip-gram and subword model. Given the present
word, skip-gram model predicts the surrounding words. In the event that window
size is 2, at that point we see just 5 vectors at once. The subword model will
see the internal structure of the words. In this model n-grams per word are
extracted. For example, ‘her’ will have distinctive vector and n-gram ‘her’
from the word ‘where’ will have a different vector.

###  3.2 Deep learning

To understand which deep learning approach works for enhanced cyber threat
indicators in the Twitter stream, we used various deep learning architectures.
The basic idea behind different deep learning approaches is given below.

####  3.2.1 Deep Neural Network:

A deep neural network (DNN) is a neural network with multiple layers which
makes it somewhat mind-boggling. DNN contains one input layer, at least one
hidden layer, and one output layer. Each hidden layer contains a rectified
linear unit (ReLU). ReLU is an activation function which characterized the
positive piece of its argument. It has less vanishing gradient problems and
computationally efficient. Hidden layer is also called a fully connected layer
since every neuron in one layer is associated with every neuron in the
following layer.

####  3.2.2 Convolutional Neural Network:

A convolutional neural network (CNN) otherwise called ConvNet is a deep neural
network which is based on shared-loads architecture. It lessens the number of
free parameters enabling the network to be deeper with fewer parameters.
Generally, CNN architecture contains convolution, pooling, and fully connected
layers. The convolution operation is performed using a number of filters which
slide through the input and learns the features of the input data. Pooling
layer is used to decrease the size of the feature matrix. The pooling can be
min, max, or average. At the end of the CNN, there will be at least one fully
connected layer where all the neurons are connected to all the neurons of its
previous layer. Also in between these layers batchnomralization and dropout
can be used. Batch normalization layer allows the network to learn by itself a
little bit more independently of other layers and in turn reduces overfitting
as it has slight regularization effects. Dropout is a regularization technique
in which some neurons are randomly ignored during training the model. This
method is treated like a layer and makes neural networks with different
architectures to train in parallel.

####  3.2.3 Recurrent Structures (RNN, LSTM, GRU):

A recurrent neural network (RNN) is a recurrent structure where associations
between nodes form a directed graph along a sequence. This enables RNN to
display temporal dynamic behavior for a time sequence that is applied to
natural language processing (NLP). RNNs can utilize their internal state to
process arrangements of inputs yet can do it for just a short amount of time
i.e., they can not remember long term data.

Long short-term memory (LSTM) network is another recurrent structure that
contains a cell, and three gates namely, input, output, and forget gate. A
cell recalls esteems over discretionary time intervals and the three gates
direct the stream of data in and out of the cell. This makes LSTM remember
long term information. LSTMs were created to manage the vanishing and
exploding gradient problems that can be experienced when training conventional
RNNs.

Gated recurrent unit (GRU) is an enhanced version of standard RNN and is also
considered as a minor variation from LSTM. To tackle the disappearing gradient
problem of a standard RNN, GRU utilizes update gate and reset gate. These two
vectors choose what information ought to be passed to the output. They are
exceptional in light of the fact that they can be trained to keep information
from a long prior time, without washing it through time or evacuate
information which is superfluous to the expectation.

##  4 Description of the Data set

The data set for data analysis of tweets from Twitter social media resource is
provided by [12]. The authors used a stream listener to listen to the
streaming of tweets from Twitter. They selected a set of keywords in order to
filter as well as narrow down the results of the stream listener. The words
like “0day” and “vulnerability” were selected for applicability to CTI. For
producing more targeted filters, words related to a particular type of threat
were selected. Preprocessing of the data set is also performed in [12]. The
detailed statistics are tabulated in Table 1 and 2.

Table 1: Binary class Twitter data samples.

Data set | Relevant | Irrelevant
---|---|---
Train | 11,781 | 5,313
Test | 2,989 | 1,285

Table 2: Multiclass Twitter data samples.

Category | Train data set | Test data set
---|---|---
Vulnerability | 5,926 | 1,428
Ransomware | 2,549 | 654
DDoS | 1,776 | 469
Data leak | 106 | 30
General | 5,588 | 1,410
Day | 585 | 145
Botnet | 564 | 138

##  5 Proposed Architecture

The proposed architecture is shown in Figure 1. The preprocessed tweets are
sent to Keras embedding layer where the words are converted into dense
vectors. These numerical features are passed into CNN and then to GRU layer
for feature generation. Finally, the output from GRU is sent to a fully
connected layer for classification.

Figure 1: Proposed Architecture.

##  6 Experiments, Results and Observations

Scikit-learn111https://scikit-learn.org/ and
TensorFlow222https://www.tensorflow.org/ with Keras333https://keras.io/
framework were utilized to implement classical machine learning algorithms and
deep learning architectures respectively. All the models are trained on GPU
enabled TensorFlow. Various statistical measures are utilized in order to
evaluate the performance of the proposed framework.

Table 3: Average performance metrics. Model | Accuracy (%) | Precision (%) | Recall (%) | F1-Score (%)
---|---|---|---|---
Binary class classification
SVM-TDM | 81.9 | 68.8 | 72.8 | 70.7
SVM-TF-IDF | 82.2 | 69.2 | 73.6 | 71.3
DNN-3gram | 82.9 | 73.5 | 67.6 | 70.4
CNN-Keras word embedding [12] | 83.6 | 71.4 | 75.9 | 73.6
RNN-Keras word embedding | 83.1 | 71.7 | 72.1 | 71.9
LSTM-Keras word embedding | 84.3 | 70.1 | 83.1 | 76.0
GRU-Keras word embedding | 84.7 | 73.9 | 76.0 | 74.9
CNN-GRU-Keras word embedding | 85.8 | 73.7 | 82.3 | 77.8
fastText | 84.4 | 74.6 | 73.2 | 73.9
Multiclass classification
SVM-TDM | 86.2 | 86.2 | 86.2 | 86.2
SVM-TF-IDF | 86.3 | 86.4 | 86.3 | 86.3
DNN-3gram | 86.9 | 87.0 | 86.9 | 86.9
CNN-Keras word embedding [12] | 87.5 | 87.8 | 87.5 | 87.6
RNN-Keras word embedding | 87.0 | 87.1 | 87.0 | 87.0
LSTM-Keras word embedding | 88.0 | 88.1 | 88.0 | 88.0
GRU-Keras word embedding | 88.4 | 88.8 | 88.4 | 88.5
CNN-GRU-Keras word embedding | 89.3 | 90.3 | 89.3 | 89.3
fastText | 87.9 | 88.0 | 87.9 | 87.9

Preprocessing steps given in the proposed architecture section is followed for
the data set to convert the unstructured format into a structured format. In
this work, various text representation methods such as TDM, TF-IDF, 3-gram,
and embedding are employed. SVM is implemented along with TDM and TF-IDF. SVM
uses rbf kernel and c value of 100. Scikit-learn default parameters of TDM and
TF-IDF are used. As the tweet length is not huge and there are a lot of
important keywords used in tweets that might be the reason why TF-IDF has
performed better than TDM. We followed n-gram representation specifically
3-gram is employed and we constructed a feature vector whose length will very
huge. So in order to decrease the dimension be employed featurization
technique to decrease the length of the sequence. This 1,000 length vector is
passed into a deep neural network (DNN). DNN contains three layers, the first
layer contains 1,024 neurons, the second layer contains 512 neurons and the
third layer contains 128 neurons. In a sequential model, initially random
weights are given to the model and these random values will be updated based
on the loss of the function while backpropagation. When Keras embedding is
employed along with the deep learning model, updation of weight will take
place upto the embedding layer during backpropogation and not just stop at the
deep neural layers. Since the data set is not huge, word embedding like
word2vec is not followed in this work. Various deep learning classifiers like
CNN, RNN, LSTM, GRU, CNN-GRU are used along with Keras word embedding in order
to find the best deep learning model. Embedding size of 128, batch-size of 32,
learning rate of 0.01, 128 hidden units, and Adam optimizer are hyperparameter
value used by RNN, LSTM, GRU, CNN, and CNN-GRU classifiers. The output layer
consists of 1 neuron in binary classification and 7 neurons for multiclass
classification. In CNN, the number of filters used is 64 and the filter length
is 3. In CNN-GRU as well as GRU, the number of hidden units used is 50.
Finally, fastText is employed as fastText has given better performance in
recent day applications. The value of parameters for fastText are learning
rate of 0.1, dimension of 128, minimum word count of 1, 100 epochs, and 2
N-grams. The average performance metrics of all the models for binary and
multiclass data set are reported in Table 3. Among all, CNN-GRU along with
Keras word embedding has performed better in both binary and multiclass
classification. For all the models, the training data set is used for training
the models and testing data set is used to test the trained models.

##  7 Conclusion and Future Work

Twitter is one of the most popular social networks, where users share their
opinions on various topics. The tweet could be related to security. This work
evaluates the performance of various text representation techniques along with
various deep learning models for cyber threat indicators in the Twitter
stream. CNN-GRU with Keras embedding performed better than any other
architecture in both binary as well as multiclass classification. The best
part about the proposed architecture is that it does not require any feature
engineering technique to be employed. Present and future work focus on event
tracking and event detection of cyber threats using social media resources
like Twitter, Facebook, etc.

## Acknowledgements

This research was supported in part by Paramount Computer Systems and Lakhshya
Cyber Security Labs. We are grateful to NVIDIA India, for the GPU hardware
support to research grant. We are also grateful to Computational Engineering
and Networking (CEN) department for encouraging the research.

## References

  * [1] A. Sapienza, A. Bessi, S. Damodaran, P. Shakarian, K. Lerman, and E. Ferrara, “Early warnings of cyber threats in online discussions,” in Data Mining Workshops (ICDMW), 2017 IEEE International Conference on. IEEE, 2017, pp. 667–674
  * [2] C. Sabottke, O. Suciu, and T. Dumitras, “Vulnerability disclosure in the age of social media: Exploiting twitter for predicting real-world exploits.” in USENIX Security Symposium, 2015, pp. 1041–1056.
  * [3] T. Mackey, J. Kalyanam, J. Klugman, E. Kuzmenko, and R. Gupta, “Solution to detect, classify, and report illicit online marketing and sales of controlled substances via twitter: Using machine learning and web forensics to combat digital opioid access,” Journal of medical Internet research, vol. 20, no. 4, 2018.
  * [4] P. Gal´an-Garc´ıa, J. G. d. l. Puerta, C. L. G´omez, I. Santos, and P. G. Bringas, “Supervised machine learning for the detection of troll profiles in twitter social network: Application to a real case of cyberbullying,” Logic Journal of the IGPL, vol. 24, no. 1, pp. 42–53, 2016.
  * [5] Z. Ashktorab, C. Brown, M. Nandi, and A. Culotta, “Tweedr: Mining twitter to inform disaster response.” in ISCRAM, 2014.
  * [6] R. P. Khandpur, T. Ji, S. Jan, G. Wang, C.-T. Lu, and N. Ramakrishnan, “Crowdsourcing cybersecurity: Cyber attack detection using social media,” in Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 2017, pp. 1049– 1057\.
  * [7] Q. Le Sceller, E. B. Karbab, M. Debbabi, and F. Iqbal, “Sonar: Automatic detection of cyber security events over the twitter stream,” in Proceedings of the 12th International Conference on Availability, Reliability and Security. ACM, 2017, p. 23.
  * [8] S. Mittal, P. K. Das, V. Mulwad, A. Joshi, and T. Finin, “Cybertwitter: Using twitter to generate alerts for cybersecurity threats and vulnerabilities,” in Proceedings of the 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining. IEEE Press, 2016, pp. 860–867.
  * [9] A. Edouard, “Event detection and analysis on short text messages,” Ph.D. dissertation, Universit´e Cˆoted’Azur, 2017.
  * [10] K.-C. Lee, C.-H. Hsieh, L.-J. Wei, C.-H. Mao, J.- H. Dai, and Y.-T. Kuang, “Sec-buzzer: cyber security emerging topic mining with open threat intelligence retrieval and timeline event annotation,” Soft Computing, vol. 21, no. 11, pp. 2883–2896, 2017.
  * [11] A. Ritter, E. Wright, W. Casey, and T. Mitchell, “Weakly supervised extraction of computer security events from twitter,” in Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2015, pp. 896–905.
  * [12] Behzadan, V., Aguirre, C., Bose, A., & Hsu, W. (2018). Corpus and Deep Learning Classifier for Collection of Cyber Threat Indicators in Twitter Stream. 2018 IEEE International Conference on Big Data (Big Data), 5002-5007.
  * [13] Vinayakumar, R., Alazab, M., Jolfaei, A., Soman, K. P., & Poornachandran, P. (2019, May). Ransomware triage using deep learning: twitter as a case study. In 2019 Cybersecurity and Cyberforensics Conference (CCC) (pp. 67-73). IEEE.
  * [14] Vinayakumar, R., Soman, K. P., Poornachandran, P., & Menon, V. K. (2019). A Deep-dive on Machine Learning for Cyber Security Use Cases. In Machine Learning for Computer and Cyber Security (pp. 122-158). CRC Press.
  * [15] Vinayakumar, R., Soman, K. P., Poornachandran, P., Alazab, M., & Jolfaei, A. (2019). DBD: Deep Learning DGA-Based Botnet Detection. In Deep Learning Applications for Cyber Security (pp. 127-149). Springer, Cham.
  * [16] Vinayakumar, R., Soman, K. P., Poornachandran, P., Akarsh, S., & Elhoseny, M. (2019). Deep Learning Framework for Cyber Threat Situational Awareness Based on Email and URL Data Analysis. In Cybersecurity and Secure Information Systems (pp. 87-124). Springer, Cham.
  * [17] Vinayakumar, R., Soman, K. P., Poornachandran, P., Akarsh, S., & Elhoseny, M. (2019). Improved DGA Domain Names Detection and Categorization Using Deep Learning Architectures with Classical Machine Learning Algorithms. In Cybersecurity and Secure Information Systems (pp. 161-192). Springer, Cham.
  * [18] Vinayakumar, R., Alazab, M., Srinivasan, S., Pham, Q. V., Padannayil, S. K., & Simran, K. (2020). A Visualized Botnet Detection System based Deep Learning for the Internet of Things Networks of Smart Cities. IEEE Transactions on Industry Applications.
  * [19] Vinayakumar, R., Alazab, M., Soman, K. P., Poornachandran, P., & Venkatraman, S. (2019). Robust intelligent malware detection using deep learning. IEEE Access, 7, 46717-46738.
  * [20] Vinayakumar, R., & Soman, K. P. (2020). Siamese neural network architecture for homoglyph attacks detection. ICT Express, 6(1), 16-19.


* [Hir64] Heisuke Hironaka.  Resolution of singularities of an algebraic variety over a field of characteristic zero. I, II.  Ann. of Math. (2), 79:109–203; 79 (1964), 205–326, ISSN = 0003–486X, MRCLASS = 14.18, MRNUMBER = 199184, MRREVIEWER = Jean Giraud, DOI = 10.2307/1970547, URL = https://doi.org/10.2307/1970547,, 1964\.
  * [Jon15] Mattias Jonsson.  Dynamics of Berkovich spaces in low dimensions.  In Berkovich spaces and applications, volume 2119 of Lecture Notes in Math., pages 205–366. Springer, Cham, 2015.
  * [KP04] Tzee-Char Kuo and Adam Parusiński.  Newton-Puiseux roots of Jacobian determinants.  J. Algebraic Geom., 13(3):579–601, 2004.
  * [Lê09] Dũng Tráng Lê.  Linear systems and multiplicity of ideals.  São Paulo J. Math. Sci., 3(2):161–178, 2009.
  * [LJT08] Monique Lejeune-Jalabert and Bernard Teissier.  Clôture intégrale des idéaux et équisingularité.  Ann. Fac. Sci. Toulouse Math. (6), 17(4):781–859, 2008.  With an appendix by Jean-Jacques Risler.
  * [Mic08] Françoise Michel.  Jacobian curves for normal complex surfaces.  In Singularities II, volume 475 of Contemp. Math., pages 135–150. Amer. Math. Soc., Providence, RI, 2008.
  * [MM20] Hélène Maugendre and Françoise Michel.  On the growth behaviour of Hironaka quotients.  J. Singul., 20:31–53, 2020.
  * [Mos85] Tadeusz Mostowski.  Lipschitz equisingularity.  Dissertationes Math. (Rozprawy Mat.), 243:46, 1985.
  * [NP14] Walter D. Neumann and Anne Pichon.  Lipschitz geometry of complex curves.  J. Singul., 10:225–234, 2014.
  * [NP16] Walter D. Neumann and Anne Pichon.  Lipschitz geometry of complex surfaces: Analytic invariants and equisingularity.  https://arxiv.org/pdf/1211.4897.pdf, page 61, 2016.
  * [NV16] Nhan Nguyen and Guillaume Valette.  Lipschitz stratifications in o-minimal structures.  Ann. Sci. Éc. Norm. Supér. (4), 49(2):399–421, 2016.
  * [Par94] Adam Parusiński.  Lipschitz stratification of subanalytic sets.  Ann. Sci. École Norm. Sup. (4), 27(6):661–696, 1994.
  * [Pic20] Anne Pichon.  An introduction to Lipschitz geometry of complex singularities.  In Introduction to Lipschitz geometry of singularities, volume 2280 of Lecture Notes in Math., pages 167–216. Springer, Cham, [2020] ©2020.
  * [PT20] Frédéric Pham and Bernard Teissier.  Lipschitz fractions of a complex analytic algebra and Zariski saturation.  In Introduction to Lipschitz geometry of singularities, volume 2280 of Lecture Notes in Math., pages 309–337. Springer, Cham, [2020] ©2020.
  * [Sha13] Igor R. Shafarevich.  Basic algebraic geometry. 1.  Springer, Heidelberg, third edition, 2013.  Varieties in projective space.
  * [Spi90] Mark Spivakovsky.  Sandwiched singularities and desingularization of surfaces by normalized Nash transformations.  Ann. of Math. (2), 131(3):411–491, 1990.
  * [Tei82] Bernard Teissier.  Variétés polaires. II. Multiplicités polaires, sections planes, et conditions de Whitney.  In Algebraic geometry (La Rábida, 1981), volume 961 of Lecture Notes in Math., pages 314–491. Springer, Berlin, 1982.
  * [Tei04] Bernard Teissier.  Monomial ideals, binomial ideals, polynomial ideals.  In Trends in commutative algebra, volume 51 of Math. Sci. Res. Inst. Publ., pages 211–246. Cambridge Univ. Press, Cambridge, 2004.