#  On the role of soft gluons in collinear parton densities

M. Mendizabal  Deutsches Elektronen-Synchrotron DESY, Hamburg, Germany
0000-0002-6506-5177 F. Guzman  InSTEC, Universidad de La Habana, Havanna, Cuba
0000-0002-7612-1488 H. Jung  Deutsches Elektronen-Synchrotron DESY, Hamburg,
Germany  II. Institut für Theoretische Physik, Universität Hamburg, Hamburg,
Germany  Elementary Particle Physics, University of Antwerp, Belgium
0000-0002-2964-9845 S. Taheri Monfared  Deutsches Elektronen-Synchrotron DESY,
Hamburg, Germany  0000-0003-2988-7859

###### Abstract

The role of soft (non-perturbative) gluons in collinear parton densities is
investigated with the Parton Branching method as a solution to the DGLAP
evolution equations. It is found that soft gluons contribute significantly to
collinear parton densities.

Within the Parton Branching frame, the Sudakov form factor can be split into a
perturbative and non-perturbative part. The non-perturbative part can be
calculated analytically under certain conditions. It is shown that the
inclusion of soft (non-perturbative) gluons to the parton density evolution is
essential for the proper cancellation of divergent terms.

It is argued that the non-perturbative part of the Sudakov form factor has its
correspondence in Transverse Momentum Dependent parton distributions. Within
the Parton Branching approach, this non-perturbative Sudakov form factor is
constrained by fits of inclusive, collinear parton densities.

version 1.0

##  1 Introduction

The DGLAP [1, 2, 3, 4] evolution of parton densities together with hard
scattering coefficient functions (or matrix elements) at next-to-leading (NLO)
and next-to-next-to-leading order (NNLO) in the strong coupling provides a
very successful description of experimental measurements from small to rather
larger scales $\mu$.

The Parton-Branching (PB) approach [5, 6] gives a solution to the DGLAP
equations, based on an iterative solution of the integral evolution equation.
The PB-approach also allows to study in detail each of the branching vertices,
and in particular the contribution of perturbative and non-perturbative
emissions. In the frame of Transverse Momentum Dependent (TMD) parton
densities [7] and especially in the CSS formalism [8], a non-perturbative
Sudakov form factor is introduced in addition. In the PB-approach this non-
perturbative Sudakov form factor already appears in inclusive (collinear)
parton densities, and with the determination of collinear parton densities by
fits to inclusive experimental data, the non-perturbative Sudakov form factor
is fixed also when applied to TMD parton densities.

In this paper we show explicitly how the Sudakov from factor is obtained from
the DGLAP evolution equation and how this form factor can be split into a
perturbative and non-perturbative part. We argue, that both parts are
essential for collinear parton distributions, as neglecting soft gluons would
lead to non-cancellation of divergencies for cross section calculations at NLO
and beyond.

##  2 PB approach as a solution of DGLAP

The PB method [5, 6] provides a solution of the DGLAP [1, 2, 3, 4] evolution
equations. The DGLAP evolution equation for the parton density of parton $a$
with momentum fraction $x$ at the scale $\mu$ reads:

$\mu^{2}\frac{{\partial}{f}_{a}(x,\mu^{2})}{{\partial}\mu^{2}}=\sum_{b}\int_{x}^{1}{\frac{dz}{z}}\;{P}_{ab}\left(\alpha_{s}(\mu^{2}),z\right)\;{f}_{b}\left({\frac{x}{z}},\mu^{2}\right)\;,$
(1)

with the regularized DGLAP splitting functions $P_{ab}$ describing the
splitting of parton $b$ into a parton $a$. The splitting functions $P_{ab}$
can be decomposed as (in the notation of Ref. [5]):

${P}_{ab}(z,\alpha_{s})=D_{ab}(\alpha_{s})\delta(1-z)+K_{ab}(\alpha_{s})\frac{1}{(1-z)_{+}}+R_{ab}(z,\alpha_{s})\;.$
(2)

The coefficients $D$ and $K$ can be written as
$D_{ab}(\alpha_{s})=\delta_{ab}d_{a}(\alpha_{s})$,
$K_{ab}(\alpha_{s})=\delta_{ab}k_{a}(\alpha_{s})$ and the coefficients
$R_{ab}$ contain only terms which are not singular for $z\rightarrow 1$. Each
of those three coefficients can be expanded in powers of $\alpha_{s}$:

$d_{a}(\alpha_{s})=\sum_{n=1}^{\infty}\left(\frac{\alpha_{s}}{2\pi}\right)^{n}d_{a}^{(n-1)},\;k_{a}(\alpha_{s})=\sum_{n=1}^{\infty}\left(\frac{\alpha_{s}}{2\pi}\right)^{n}k_{a}^{(n-1)},\;R_{ab}(z,\alpha_{s})=\sum_{n=1}^{\infty}\left(\frac{\alpha_{s}}{2\pi}\right)^{n}R_{ab}^{(n-1)}(z)$
(3)

The plus-prescription and the $D$ part in eq.(2) can be expanded in eq.(1) by
introducing a Sudakov form factor $\Delta^{S}_{a}(\mu^{2})$ (for details on
the calculation see appendix) which is defined as:

$\Delta^{S}_{a}(\mu^{2},\mu^{2}_{0})=\exp\left(-\int_{\mu_{0}^{2}}^{\mu^{2}}\frac{\textrm{d}\mu^{\prime
2}}{\mu^{\prime
2}}\left[\int_{0}^{z_{M}}k_{a}(\alpha_{s})\frac{1}{1-z}\textrm{d}z-d_{a}(\alpha_{s})\right]\right)\;,$
(4)

where an upper limit $z_{M}=1-\epsilon\;$ is introduced to allow numerical
integration over $z$. In order to reproduce DGLAP, $\epsilon\to 0$ is
required. Note, that the expression of $\Delta^{S}_{a}$ is different from the
one used in Ref. [5], since only the large $z$ part of the splitting function,
including virtual contributions, is used. The evolution equation for the
parton density ${f}_{a}(x,\mu^{2})$ at scale $\mu$ is then given by (as a
solution of eq.(1), see also [9]):

${f}_{a}(x,\mu^{2})=\Delta^{S}_{a}(\mu^{2})\
{f}_{a}(x,\mu^{2}_{0})+\sum_{b}\int^{\mu^{2}}_{\mu^{2}_{0}}{{d\mu^{\prime
2}}\over\mu^{\prime
2}}{{\Delta^{S}_{a}(\mu^{2})}\over{\Delta^{S}_{a}(\mu^{\prime
2})}}\int_{x}^{z_{M}}{\frac{dz}{z}}\;\hat{P}_{ab}(\alpha_{s}(\mu^{\prime
2}),z)\;{f}_{b}\left({\frac{x}{z}},\mu^{\prime 2}\right)$ (5)

with the unregularized splitting functions $\hat{P}_{ab}$ (without the
$D_{ab}$ piece, and replacing $1/(1-z)_{+}$ by $1/(1-z)$) and $\mu_{0}$ being
the starting scale. Since the evolution equation is solved iteratively, on has
access to every individual branching vertex, and thus one can calculate also
the transverse momenta ($q_{\rm t}$) of the emitted partons. Details on the
formulation for TMD parton distributions are given in Ref. [5]).

On a collinear level, it was shown in Ref. [5] that the PB approach reproduces
exactly the DGLAP evolution of parton densities [10], if the renormalization
scale (the argument in $\alpha_{s}$) is set to the evolution scale $\mu$ and
if $z_{M}\to 1$. In Ref. [11] the PB parton distributions are obtained from a
fit [12, 13] of the $x$-dependent starting distributions to describe high-
precision deep-inelastic scattering data [14] . Two different parton
distribution sets (we use PB-NLO-2018 as a shorthand notation for PB-NLO-
HERAI+II-2018) were obtained, PB-NLO-2018 Set1, which for collinear
distributions agrees exactly with HERAPDF2.0NLO [14], and another set, PB-
NLO-2018 Set2, which uses $q_{\rm t}$ as the argument in $\alpha_{s}$,
inspired by angular ordering conditions. All PB parton distributions (and many
others) are accessible in TMDlib and via the graphical interface TMDplotter
[15, 16].

In the following we concentrate on the PB-NLO-2018 Set1 scenario because of
its direct correspondence to standard DGLAP solutions, a discussion on PB-
NLO-2018 Set2 is given in Ref. [17].

###  2.1 The PB Sudakov form factor

The concept of resolvable and non-resolvable branchings with Sudakov form
factors allows for an intuitive interpretation of the parton evolution
pattern. The Sudakov form factors give the probability to evolve from one
scale to another scale without resolvable branching. While the concept of the
PB method is similar to a parton shower approach, the method is used here to
solve the DGLAP evolution equation.

In order to illustrate the importance of resolvable and non-resolvable
branchings we separate the Sudakov form factor $\Delta^{S}_{a}$ into a
perturbative ($q_{\rm t}>q_{0}$) and non-perturbative ($q_{\rm t}<q_{0}$) part
by introducing a resolution scale $z_{\rm dyn}=1-q_{0}/\mu^{\prime}$ (see Ref.
[18]). This scale is motivated by angular ordering and the requirement to
resolve an emitted parton with $q_{\rm t}>q_{0}$.

The Sudakov form factor is then given by***It can be shown, that
$\Delta_{a}^{(\text{P})}$ coincides with the Sudakov form factor used in CSS
[8] up to next-to-leading and even partially next-to-next-to-leading
logarithms (see [19, 20]). The non-perturbative Sudakov form factor
$\Delta_{a}^{(\text{NP})}$ has a similar structure as the non-perturbative
Sudakov form factor in CSS with the typical $\log(\mu^{2}/\mu_{0}^{2})$
dependence. :

$\displaystyle\Delta_{a}^{S}(\mu^{2},\mu^{2}_{0})=$
$\displaystyle\exp\left(-\int^{\mu^{2}}_{\mu^{2}_{0}}{{d\mu^{\prime
2}}\over\mu^{\prime 2}}\left[\int_{0}^{z_{\rm
dyn}(\mu^{\prime})}dz\frac{k_{a}(\alpha_{s})}{1-z}-d_{a}(\alpha_{s})\right]\right)$
(6) $\displaystyle\times\exp\left(-\int^{\mu^{2}}_{\mu^{2}_{0}}{{d\mu^{\prime
2}}\over\mu^{\prime 2}}\int_{z_{\rm
dyn}(\mu^{\prime})}^{z_{M}}dz\frac{k_{a}(\alpha_{s})}{1-z}\right)$
$\displaystyle=$
$\displaystyle\Delta_{a}^{(\text{P})}\left(\mu^{2},\mu_{0}^{2},q^{2}_{0}\right)\cdot\Delta_{a}^{(\text{NP})}\left(\mu^{2},\mu_{0}^{2},\epsilon,q_{0}^{2}\right)\;.$

Figure 1: Integrated down-quark distributions at $\mu=10$ GeV and $\mu=100$
GeV obtained from the PB approach for different values of $z_{M}$: PB-NLO-
HERAI+II-set1 applies $z_{M}\to 1$ and PB-NLO-set1 applies $z_{M}=z_{\rm dyn}$
with $q_{0}=1$ GeV and without intrinsic -$k_{\rm T}$ distribution
($q_{s}=0$). The ratio plots show the ratios to the one for $z_{M}\to 1$.

In Fig. 1 we show parton distributions obtained with the PB approach using the
starting distributions from PB-NLO-2018 Set1 from Ref. [11]. We show
distributions for down quark parton densities for different values of $z_{M}$:
$z_{M}\to 1$ (default) and $z_{M}=z_{\rm dyn}=1-q_{0}/\mu^{\prime}$, with
$q_{0}=1$ GeV†††The value of $q_{0}$ is arbitrary, and chosen as $q_{0}=1$ GeV
for illustration only. The starting parameters are the same as for PB-NLO-2018
Set1. and without any intrinsic -$k_{\rm T}$ distribution ($q_{s}=0$), which
by definition does not matter for the integrated parton densities. The
distributions obtained from PB-NLO-2018 set1 with $z_{M}\to 1$ are
significantly different from those applying $z_{M}=z_{\rm dyn}$, illustrating
the importance of soft contributions even for collinear distributions. It is
obvious that limiting the $z$-integration by $z_{\rm dyn}$ (and neglecting
$\Delta_{a}^{(\text{NP})}$) leads to distributions which are no longer
consistent with the collinear $\overline{\rm MS}$ factorization scheme. A
similar conclusion was found in Ref. [21].

Since the PB-approach can also be used to determine TMD parton distributions
(see Ref. [5]) we illustrate in Fig. 2 the effect of the $z_{M}$ cut-off in
the transverse momentum distribution. We show results obtained with the PB-
approach for down quarks for PB-NLO-2018 Set1 with a default Gaussian width
$q_{s}=0.5$ GeV for the intrinsic $k_{\rm T}$-distribution. Since we want to
focus only on the evolution (as given in eq.(6)), we show also the results
when no intrinsic-$k_{\rm T}$ distribution applied $q_{s}=0$ GeV (practically
we use a Gauss distribution with $q_{s}=0.00001$ GeV) and we illustrate the
effect of neglecting $\Delta_{a}^{(\text{NP})}$ (by applying $z_{\rm
dyn}=1-q_{0}/\mu^{\prime}$, as in Fig. 1).

Figure 2: Transverse momentum distributions of down quarks at $\mu=10$ GeV
(left column) and $\mu=100$ GeV (right column) obtained from the PB approach
for $z_{M}\to 1$ as well as $z_{M}=z_{\rm dyn}=1-q_{0}/q$. The red curve shows
the PB-NLO-2018 Set1 (including intrinsic-$k_{\rm T}$), the blue curve shows
PB-NLO-2018 Set1 without including any intrinsic-$k_{\rm T}$ distribution
($q_{s}=0$), and the magenta curve shows the prediction applying $z_{M}=z_{\rm
dyn}$ with $q_{0}=1.0$ GeV without including intrinsic-$k_{\rm T}$
distributions.

The transverse momentum distributions show very clearly the effect of
$\Delta_{a}^{(\text{NP})}$. Applying the cutoff-scale, $z_{M}=z_{\rm
dyn}=1-q_{0}/\mu^{\prime}$, removes emissions with $q_{\rm t}<q_{0}$ (there
are still low-$k_{\rm T}$ contributions, which come from adding vectorially
all intermediate emissions). However, very soft emissions are automatically
included with $z_{M}\to 1$. Please note, that if $\alpha_{s}(q_{\rm t})$ is
chosen a special treatment is needed for $q_{\rm t}<q_{0}$.

##  3 Cross Section calculation

Physical cross sections are calculated as a convolution of the scale dependent
parton densities with the hard matrix elements (coefficient functions). For
simplicity we show only the calculation of deep inelastic scattering with NLO
parton densities and coefficient functions, the expressions are given in text
books (i.e. Ref. [9] eq. 4.80 ):

$F_{2}(x,Q^{2})=x\sum_{q,\bar{q}}e_{q}^{2}\int_{x}^{1}\frac{d\xi}{\xi}q(\xi,Q^{2})\left[\delta\left(1-\frac{x}{\xi}\right)+\frac{\alpha_{s}}{2\pi}C_{\over{\rm
MS}}\left(\frac{x}{\xi}\right)+\cdots\right]$ (7)

with the coefficient function $C_{\over{\rm MS}}$ in $\overline{\rm
MS}$-scheme. The $\overline{\rm MS}$ coefficient function for quarks at ${\cal
O}(\alpha_{s})$ read:

$\displaystyle C^{\over{\rm MS}}_{q}(z)$ $\displaystyle=$ $\displaystyle
C_{F}\left[2\left(\frac{\log(1-z)}{1-z}\right)_{+}-\frac{3}{2}\left(\frac{1}{1-z}\right)_{+}-(1+z)\log(1-z)\right.$
(8) $\displaystyle\left.-\frac{1+z^{2}}{1-z}\log
z+3+2z-\left(\frac{\pi^{2}}{3}+\frac{9}{2}\right)\delta(1-z)\right]$

For a consistent formulation, the integral over $\xi$ in eq.(7) has to extend
up to one, both in the expression for the cross section, as well as in the
expression for the parton density, otherwise singular pieces remain un-
cancelled.

It becomes clear, that the contribution of soft gluon emissions is important
both in the parton densities as well as in the cross section calculations.
Approaches, where the integral is limited by $z_{\rm dyn}$ lead to a different
factorization scheme and the coefficient functions obtained in collinear
$\overline{\rm MS}$-scheme are no longer appropriate.

From the above considerations, we can summarize, that the region of soft gluon
emissions is very important in the evolution of the parton densities, as well
as in the calculation of the hard cross section. The separation of the Sudakov
form factor into a perturbative and non-perturbative part helps the
interpretation of TMD parton distributions essential e.g. for the Drell-Yan pt
spectrum. Within the PB-approach, the non-perturbative Sudakov form factor is
constrained by the fit to inclusive measurements to determine inclusive parton
distributions: once the evolution frame is specified (depending on the scale
choice for $\alpha_{s}$), the non-perturbative Sudakov form factor is fixed by
the fit to inclusive measurements. The PB-TMD distributions are then
calculated without any further assumptions (in contrast to approaches like
CSS, where the non-perturbative Sudakov form factor has to be determined
independent of the collinear distributions).

##  4 Summary and Conclusion

We have investigated the perturbative and non-perturbative regions of
collinear parton densities, by making use of the formulation of the DGLAP
evolution equation in terms of Sudakov form factors applied in the PB- method.
The separation of the Sudakov form factor in a perturbative and non-
perturbative part is motived by investigations of the Drell-Yan transverse
momentum spectrum within the CSS approach.

We find, that soft (non-perturbative) gluon emissions play a significant role
in inclusive parton distributions, and those emissions are an essential part
of the $\overline{\rm MS}$-scheme: neglecting those would lead to non-
cancellation of important singular pieces.

With the requirement to describe and fit inclusive distributions like the
inclusive DIS cross-section, the non-perturbative Sudakov form factor is
constrained and determined, once the factorization and evolution scheme for
collinear parton densities is fixed (i.e. the choice of scale for the
evolution of the strong coupling $\alpha_{s}$, which implies also a different
form of the non-perturbative Sudakov form factor). The non-perturbative
Sudakov form factor, fixed from inclusive distributions, plays an important
role in the description of the Drell-Yan transverse momentum spectrum.

##  5 Appendix: The Sudakov form factor

The DGLAP splitting functions in evolution equation eq.(1) can be written in
different forms, where the plus-prescription acts on different terms‡‡‡In the
derivation we only consider $\alpha_{s}(\mu)$:

$\displaystyle{P}_{ab}(z,\alpha_{s})$ $\displaystyle=$
$\displaystyle{P}(z,\alpha_{s})_{+}$ (9) $\displaystyle=$ $\displaystyle
D_{ab}(\alpha_{s})\delta(1-z)+K_{ab}(\alpha_{s})\frac{1}{(1-z)_{+}}+R_{ab}(z,\alpha_{s})\;$
(10)

The plus prescription is given by:

$\displaystyle\int_{0}^{1}dz\frac{\phi(z)}{(1-z)_{+}}$ $\displaystyle=$
$\displaystyle\int_{0}^{1}dz\frac{\phi(z)-\phi(1)}{(1-z)}$ (11)
$\displaystyle=$ $\displaystyle\lim_{\epsilon\to
0}\int_{0}^{1-\epsilon}dz\frac{\phi(z)}{(1-z)}-\lim_{\epsilon\to
0}\int_{0}^{1-\epsilon}dz\frac{\phi(1)}{(1-z)}\;,$ (12)

where the last line is used for expanding the plus-prescription. Applying
eq.(9) and eq.(12) leads to:

$\displaystyle\mu^{2}\frac{{\partial}{f}_{a}(x,\mu^{2})}{{\partial}\mu^{2}}$
$\displaystyle=$
$\displaystyle\sum_{b}\int_{x}^{1}{\frac{dz}{z}}\;{P}_{ab}\left(z\right)\;{f}_{b}\left({\frac{x}{z}},\mu^{2}\right)$
(13) $\displaystyle=$
$\displaystyle\sum_{b}\int_{0}^{1}\frac{dz}{z}\;P_{ab}\left(z\right)\;f_{b}\left({\frac{x}{z}}\right)-\sum_{b}\int_{0}^{x}\frac{dz}{z}\;P_{ab}(z)\;f_{b}\left(\frac{x}{z}\right)$
(14) $\displaystyle=$
$\displaystyle\sum_{b}\int_{0}^{1}dz\;\hat{P}_{ab}\left(z\right)\left(\frac{1}{z}\;f_{b}\left({\frac{x}{z}}\right)-f_{b}(z)\right)-\sum_{b}\int_{0}^{x}\frac{dz}{z}\;\hat{P}_{ab}\left(z\right)\;f_{b}\left(\frac{x}{z}\right)$
(15) $\displaystyle=$
$\displaystyle\sum_{b}\int_{x}^{1}\frac{dz}{z}\;\hat{P}_{ab}\left(z\right)\;f_{b}\left({\frac{x}{z}}\right)-f_{a}(x)\sum_{b}\int_{0}^{1}dz\;\hat{P}_{ab}\left(z\right)$
(16) $\displaystyle=$
$\displaystyle\sum_{b}\int_{x}^{1}\frac{dz}{z}\;\hat{P}_{ab}\left(z\right)\;f_{b}\left({\frac{x}{z}}\right)+f_{a}(x)\frac{\mu^{2}}{\Delta_{a}}\frac{\partial\Delta_{a}}{\partial\mu^{2}}$
(17)

where we dropped $\alpha_{s}$ and $\mu^{2}$ dependence for better reading. The
unregularized splitting function is denoted by $\hat{P}$ (without the $D_{ab}$
piece, and replacing $1/(1-z)_{+}$ by $1/(1-z)$), and the Sudakov form factor
$\Delta_{a}(\mu^{2})$ is given by:

$\Delta_{a}(\mu^{2})=\exp\left(-\sum_{b}\int_{\mu_{0}^{2}}^{\mu^{2}}\frac{d\mu^{\prime
2}}{\mu^{\prime 2}}\int_{0}^{1-\epsilon}dz\hat{P}_{ab}(\alpha_{s},z)\right)$
(18)

With this, the evolution equation for
${f_{a}\left(x,\mu^{2}\right)}/{\Delta_{a}(\mu^{2})}$ is given by (see [9]):

$\mu^{2}\frac{\partial}{\partial\mu^{2}}\frac{f_{a}(x,\mu^{2})}{\Delta_{a}(\mu^{2})}=\sum_{b}\int_{x}^{1}\frac{dz}{z}\;\hat{P}_{ab}\left(z\right)\;\frac{f_{b}\left({\frac{x}{z},\mu^{2}}\right)}{\Delta_{a}(\mu^{2})}$
(19)

If instead we start from eq.(10), and apply the same steps as above, we
obtain:

$\displaystyle\mu^{2}\frac{{\partial}{f}_{a}(x,\mu^{2})}{{\partial}\mu^{2}}$
$\displaystyle=$
$\displaystyle\sum_{b}\int_{x}^{1}{\frac{dz}{z}}\;{P}_{ab}\left(z\right)\;{f}_{b}\left({\frac{x}{z}},\mu^{2}\right)$
(20) $\displaystyle=$
$\displaystyle\sum_{b}\int_{0}^{1}\frac{dz}{z}\;\left(D_{ab}\delta(1-z)+K_{ab}\frac{1}{(1-z)_{+}}+R_{ab}(z)\right)\;f_{b}\left(\frac{x}{z}\right)$
(22) $\displaystyle=$
$\displaystyle\sum_{b}\int_{0}^{1}\frac{dz}{z}\left(\frac{K_{ab}}{1-z}+R_{ab}(z)\right)f_{b}\left(\frac{x}{z}\right)$
$\displaystyle-
f_{a}(z)\int_{0}^{1}dz\left(\frac{k_{a}}{(1-z)}-d_{a}\delta(1-z)\right)\;$

The Sudakov form factor $\Delta_{S}(\mu^{2})$ is now defined as (see eq.(4)):

$\Delta_{a}^{S}(\mu^{2})=\exp\left(-\int_{\mu_{0}^{2}}^{\mu^{2}}\frac{d\mu^{\prime
2}}{\mu^{\prime
2}}\int_{0}^{1-\epsilon}dz\left[\frac{k_{a}}{(1-z)}-d_{a}\delta(1-z)\right]\right)\;,$
(24)

leading to an evolution equation for
${f_{a}\left(x,\mu^{2}\right)}/{\Delta^{S}_{a}(\mu^{2})}$.

## References

  * [1] V. N. Gribov and L. N. Lipatov, “Deep inelastic $ep$ scattering in perturbation theory”,  Sov. J. Nucl. Phys. 15 (1972) 438. [Yad. Fiz.15,781(1972)].
  * [2] L. N. Lipatov, “The parton model and perturbation theory”,  Sov. J. Nucl. Phys. 20 (1975) 94. [Yad. Fiz.20,181(1974)].
  * [3] G. Altarelli and G. Parisi, “Asymptotic freedom in parton language”,  Nucl. Phys. B 126 (1977) 298\.
  * [4] Y. L. Dokshitzer, “Calculation of the structure functions for Deep Inelastic Scattering and $e^{+}e^{-}$ annihilation by perturbation theory in Quantum Chromodynamics.”,  Sov. Phys. JETP 46 (1977) 641\. [Zh. Eksp. Teor. Fiz.73,1216(1977)].
  * [5] F. Hautmann et al., “Collinear and TMD quark and gluon densities from Parton Branching solution of QCD evolution equations”,  JHEP 01 (2018) 070, arXiv:1708.03279.
  * [6] F. Hautmann et al., “Soft-gluon resolution scale in QCD evolution equations”,  Phys. Lett. B 772 (2017) 446, arXiv:1704.01757.
  * [7] J. Collins and T. C. Rogers, “Connecting Different TMD Factorization Formalisms in QCD”,  Phys. Rev. D 96 (2017) 054011, arXiv:1705.07167.
  * [8] J. C. Collins, D. E. Soper, and G. F. Sterman, “Transverse Momentum Distribution in Drell-Yan Pair and W and Z Boson production”,  Nucl. Phys. B 250 (1985) 199\.
  * [9] R. K. Ellis, W. J. Stirling, and B. R. Webber, “QCD and collider physics”,  Camb. Monogr. Part. Phys. Nucl. Phys. Cosmol. 8 (1996) 1\.
  * [10] M. Botje, “QCDNUM: fast QCD evolution and convolution”,  Comput.Phys.Commun. 182 (2011) 490, arXiv:1005.1481.
  * [11] A. Bermudez Martinez et al., “Collinear and TMD parton densities from fits to precision DIS measurements in the parton branching method”,  Phys. Rev. D 99 (2019) 074008, arXiv:1804.11152.
  * [12] xFitter Developers’ Team Collaboration, H. Abdolmaleki et al., “xFitter: An Open Source QCD Analysis Framework. A resource and reference document for the Snowmass study”,  6, 2022.  arXiv:2206.12465.
  * [13] S. Alekhin et al., “HERAFitter”,  Eur. Phys. J. C 75 (2015) 304, arXiv:1410.4412.
  * [14] ZEUS, H1 Collaboration, “Combination of measurements of inclusive deep inelastic ${e^{\pm}p}$ scattering cross sections and QCD analysis of HERA data”,  Eur. Phys. J. C 75 (2015) 580, arXiv:1506.06042.
  * [15] N. A. Abdulov et al., “TMDlib2 and TMDplotter: a platform for 3D hadron structure studies”,  Eur. Phys. J. C 81 (2021) 752, arXiv:2103.09741.
  * [16] F. Hautmann et al., “TMDlib and TMDplotter: library and plotting tools for transverse-momentum-dependent parton distributions”,  Eur. Phys. J. C 74 (2014), no. 12, 3220, arXiv:1408.3015.
  * [17] I. Bubanja et al., “The small $k_{T}$ \- region in Drell-Yan production at next-to-leading order with the Parton Branching Method”. to be published.
  * [18] F. Hautmann, L. Keersmaekers, A. Lelek, and A. M. Van Kampen, “Dynamical resolution scale in transverse momentum distributions at the LHC”,  Nucl. Phys. B 949 (2019) 114795, arXiv:1908.08524.
  * [19] A. M. van Kampen, “Drell-Yan transverse spectra at the LHC: a comparison of parton branching and analytical resummation approaches”,  SciPost Phys. Proc. 8 (2022) 151, arXiv:2108.04099.
  * [20] A. Bermudez Martinez et al., “The TMD Parton Branching Sudakov Form Factor in the context of TMD factorization”. to be published.
  * [21] Z. Nagy and D. E. Soper, “Evolution of parton showers and parton distribution functions”,  Phys. Rev. D 102 (2020), no. 1, 014025, arXiv:2002.04125.


# SeACo-Paraformer: A Non-Autoregressive ASR System with Flexible and
Effective Hotword Customization Ability

###### Abstract

Hotword customization is one of the important issues remained in ASR field -
it is of value to enable users of ASR systems to customize names of entities,
persons and other phrases. The past few years have seen both implicit and
explicit modeling strategies for ASR contextualization developed. While these
approaches have performed adequately, they still exhibit certain shortcomings
such as instability in effectiveness. In this paper we propose Semantic-
augmented Contextual-Paraformer (SeACo-Paraformer) a novel NAR based ASR
system with flexible and effective hotword customization ability. It combines
the accuracy of the AED-based model, the efficiency of the NAR model, and the
excellent performance in contextualization. In 50,000 hours industrial big
data experiments, our proposed model outperforms strong baselines in
customization and general ASR tasks. Besides, we explore an efficient way to
filter large scale incoming hotwords for further improvement. The source codes
and industrial models proposed and compared are all opened as well as two
hotword test sets.

Index Terms—  end-to-end ASR, non-autoregressive ASR, contextualized ASR,
hotword customization

##  1 Introduction

End-to-end speech recognition has made significant progress in the past
decade, with several classic and high-performance ASR backbone models
including Transducer [1], listen-attend-and-spell (LAS) [2] and Transformer
[3] receiving considerable attention and spawning numerous variants to address
various issues in the field of ASR like streaming ASR [4, 5, 6], multi-lingual
ASR [7, 8], and non-autoregressive ASR [9, 10], among others. In commercial
ASR systems, especially those applied in vertical domains, enabling users to
input personal names, place names, named entities and other words as hotwords
to obtain personalized recognition results, in short supporting hotword
customization, is a question of both academic and commercial value.

At the age of conventional ASR systems, as the acoustic model and language
model focus on the phonetic and linguistic information separately, the
personalized and biased information are introduced by adjusting the weights in
weighted finite state transducer (WFST) decoder [11, 12]. For E2E ASR systems
without WFST decoder, several works are explored to enable users to customize
their own hotwords [13, 14, 15]. Contextual listen, attend and spell (CLAS)
[13] is an E2E contextual ASR modeling method proposed by Golan Pundak et al.
They first proposed to introduce a multi-headed attention (MHA) which is
jointly trained with randomly sampled phrases in LAS model. Intuitively, the
MHA is trained to capture the relationship between hotword embedding and
decoder output in each step. leading the decoder to generate biased
probabilities once matched. Such strategy is proved an efficient way for
introducing bias information to ASR models without damaging the recognition
performance, and then becomes a widely adopted contextual ASR solution.

Building upon the fundamental strategy of CLAS: randomly sampling hotwords and
adopting MHA, several extensions are explored to achieve hotword
customization. These methods can be categorized as explicit and implicit ones.
Huang et al. recently proposed contextual phrase prediction network (CPP
Network) [15] which is similiar with CLAS but an extra connectionist temporal
classification (CTC) head is introduced for predicting hotwords only. By
jointly optimizing the CTC module, the encoder output is forced to obtain
hotword information thus the ASR probabilities are biased with given hotwords.
By adopting an extra criterion, CPP network enables CTC, Transducer and
Transformer models with hotword customization capacity. Han et al. proposed
Continuous integrate-and-fire based collaborative decoding (CIF ColDec) [14].
It is an implicit method based on CIF ASR backbone to achieve hotword
customization, predicting an external hotword probabilities which is
synchronous with ASR prediction. In the training stage, MHA is conducted
between CIF output $\mathbf{E}$ and hotword embeddings. Each step of the MHA
output carries hotword attended semantic information $\mathbf{g}$, then a
feed-forward layer turns it to probabilities for computing cross-entropy (CE)
loss with the hotword-position-aware target (e.g. ‘A B C D E F’ for ASR
target, and ‘A B # # # #’ for bias target when ‘A B’ is sampled as a hotword
and ‘#’ for non-bias label). The introduced modules actually work as a hotword
detector to generate biased probabilities
$\mathbf{P_{b}}\left(y_{i}\mid\mathbf{g}_{\leq i}\right)$ which is synchronous
with ASR predictions $\mathbf{P_{ASR}}\left(y_{i}\mid\mathbf{E}_{\leq
i},y_{\leq i-1}\right)$. As for the inference stage, ASR output layer and
biased output layer conduct collaborative decoding by merging the
probabilities.

However, each of the aforementioned methods has its own respective
shortcomings. The vanilla CLAS does not demonstrate consistent effectiveness -
the label insertion strategy proposed in [13] is not applicable to all E2E
systems. Additionally, implicit modeling make it hard to decouple the general
ASR modeling and contextual modeling when the system encounters bad cases.
Apparently, comparing to CLAS and its extensions, CIF ColDec has a more
intuitive and controllable hotword predicting process and training for the
external parameters is decoupled from ASR model. On the other hand, the ASR
backbone it employs does not attain the high accuracy of attention-encoder-
decoder (AED) models. Moreover, all of these methods suffer from significant
decreases in hotword recall ability as the input hot word list expands. Such
factors motivate us to propose a novel hotword customization enabled ASR
system: Semantic-Augmented Contextual Paraformer (SeACo-Paraformer). We adopt
Paraformer [10] as our backbone model, which serves as a highly efficient NAR
model while also maintaining the high accuracy of mainstream AED models. More
importantly, CIF is used in the modeling, which makes it possible to conduct a
more complex CIF based hotword prediction comparing to CIF ColDec. To address
the issue of performance decrease resulting from an expanding hotword list, we
propose attention score filtering (ASF) to filter the large scale incoming
hotwords. We conduct a series of experiments over large scale industrial data
to validate the performace of SeACo-Paraformer and ASF. The paper is
structured as follows: Section 2 provides a brief introduction to the adopted
CIF and Paraformer. Section 3 presents a detailed introduction to SeACo-
Paraformer, including training, inference, and other techniques. Section 4
presents the experimental setup and results, while Section 5 analyzes the
performance of the models. Finally, Section 6 concludes the paper.

##  2 PRELIMINARIES

###  2.1 Continuous Integrate-and-Fire

Continuous integrate-and-fire (CIF) a mechanism which utilizes the monotonic
character of ASR task to obtain acoustic embedding with encoder output. CIF
predicts weights for each frame and accumulates them. Once the weights added
up to 1.0, the encoder output of the former periods are integrated to generate
a step in acoustic embedding. Considering encoder output $\mathbf{e}_{1:T}$
with predicted weights $\alpha=(0.4,0.8,0.3,0.5,0.2)$, The integrated acoustic
embedding $\mathbf{E}_{1}=0.4\times\mathbf{e}_{1}+0.6\times\mathbf{e}_{2}$,
$\mathbf{E}_{2}=0.2\times\mathbf{e}_{2}+0.3\times\mathbf{e}_{3}+0.5\times\mathbf{e}_{4}$.
The sum of weights $\alpha$ is $L^{{}^{\prime}}$ which stands for the length
of output sequence.The most important features of CIF is that the embedding it
generates has the same length as the target sequence, which means it’s a
entirely acoustic based representation but synchronous with predictions.

###  2.2 Paraformer

In this work, we adopt Paraformer [10], a novel NAR ASR model as our backbone.
Paraformer stands for Parallel Transformer which contains a decoder receiving
acoustic embedding or semantic embedding at the same time in training.
Briefly, Paraformer achieves non-autoregressive decoding capacity by utilizing
CIF [16] and two-pass training strategy as illustrated in Fig 1 lower dotted
box. A CIF predictor is trained to predict the number of tokens and generate
acoustic embedding $\mathbf{E_{1:L^{{}^{\prime}}}}$ for parallel decoder,
which makes up Pass1 in training (w/o gradient). The steps in char embedding
$\mathbf{c_{1:L}}$ will be gradually replaced by
$\mathbf{E_{1:L^{{}^{\prime}}}}$ as accuracy raises in Pass1 and the so called
semantic embedding $\mathbf{S_{1:L}}$ is generated according to the correctly
recognized positions. Getting rid of the massive computation overhead
introduced by auto-regressive decoding and beam-search, Paraformer gains more
than 10x speedup with even lower error rate. More implementation details and
experiment results can be found in [10, 17].

##  3 SeACo-Paraformer

###  3.1 Architecture and training

Taking the advantages and shortcomings of the previous works discussed in
Section 1 into account, we propose a novel hotword customization system called
Semantic-augmented Contextual-Paraformer (SeACo-Paraformer). Based on
Paraformer which is a strong ASR backbone, 3 modules are introduced as
illustrated in Fig 1. Intuitively, SeACo-Paraformer utilize the modeling
characteristic of CIF predictor to conduct hotword prediction, while the CIF
output and parallel decoder output are sent to bias decoder separately.

Formally, considering an speech feature $\mathbf{x}_{1:T}$ and corresponding
text $\mathbf{y}_{1:L}$, we retain the CIF output
$\mathbf{E}_{1:L^{{}^{\prime}}}$ and parallel decoder hidden state (before
output layer) $\mathbf{D}_{1:L^{{}^{\prime}}}$ in Paraformer inference (batch
dimension omitted):

$\displaystyle\mathbf{e}_{1:T}$
$\displaystyle=\operatorname{Encoder}\left(\mathbf{x}_{1:T}\right)$ (1)
$\displaystyle\mathbf{E}_{1:L^{{}^{\prime}}}$
$\displaystyle=\operatorname{CIF}(\mathbf{e}_{1:T})$
$\displaystyle\mathbf{s}_{1:L^{{}^{\prime}}}$
$\displaystyle=\operatorname{Sampler}\left(\mathbf{E}_{1:L^{{}^{\prime}}},\operatorname{EMB}\left(\mathbf{y}_{1:L}\right)\right)$
$\displaystyle\mathbf{D}_{1:L^{{}^{\prime}}}$
$\displaystyle=\operatorname{ParallelDecoder}\left(\mathbf{s}_{1:L^{{}^{\prime}}};\mathbf{e}_{1:T}\right)$

Then $n$ hotwords are randomly sampled out of batches of $\mathbf{y}_{1:L}$
with batch size $bs$, denoted as $\mathbf{H}_{1:n}$. We use 4 hyper-parameter
here to control the sampling process: $r_{b}$ for the ratio of batches to
conduct sampling, the forward of the other batches will be conducted with a
default hotword <blank>; $r_{u}$ is similar with $r_{b}$ but in utterance
level inside an active batch, the average number of hotwords sampled for
active batch is $r_{u}\times bs+1$ (one for the default hotword); $l_{min}$
and $l_{max}$ for the minimum and maximum lengths of sampled hotwords.

Fig. 1: Illustration of SeACo-Paraformer.

The character sequences in hotword list are then embedded with bias encoder
which contains a embedding layer (shared with ASR embedding) and a LSTM layer:

$\displaystyle\mathbf{Z}_{1:n}=\operatorname{LSTM}\left(\operatorname{EMB}\left(\mathbf{H}_{1:n}\right)\right)$
(2)

$\mathbf{Z}_{1:n}\in R^{n\times d}$ is unsqueezed and repeated on the 0th
dimension for batch calculation. Then comes to the main part of SeACo-
Paraformer. Inside bias decoder, the bias information of hotwords is
introduced to acoustic embedding $\mathbf{E}_{1:L^{{}^{\prime}}}$ and decoder
hidden state $\mathbf{D}_{1:L^{{}^{\prime}}}$ through the attention mechanism:

$\displaystyle\mathbf{D^{+}}_{1:L^{{}^{\prime}}}$
$\displaystyle=\operatorname{MultiHeadedAttention}\left(\mathbf{D}_{1:L^{{}^{\prime}}},\mathbf{Z}_{1:n},\mathbf{Z}_{1:n}\right)$
(3) $\displaystyle\mathbf{E^{+}}_{1:L^{{}^{\prime}}}$
$\displaystyle=\operatorname{MultiHeadedAttention}\left(\mathbf{E}_{1:L^{{}^{\prime}}},\mathbf{Z}_{1:n},\mathbf{Z}_{1:n}\right)$

Bias decoder is composed of several multi-headed attention and feed-forward
layer. With biased acoustic embedding and biased decoder hidden state, biased
probabilities $\mathbf{P}_{ASR_{1:L^{{}^{\prime}}}}$ can be conducted with an
output layer. Note that an additional token (counted as #, indexed as $i_{nh}$
in vocabulary) has been added to the ASR output vocabulary to mark non-hotword
outputs.

$\mathbf{P_{b_{i}}}=\operatorname{BiasOutLayer}\left(\mathbf{D^{+}}_{1:L^{{}^{\prime}}}+\mathbf{E^{+}}_{1:L^{{}^{\prime}}}\right)$
(4)

Given the biased probabilities $\mathbf{P}_{b_{1:L^{{}^{\prime}}}}$, the
parameters of bias-related can be updated with hotword-position-aware target
in which tokens in non-hotword positions is replaced by <no-bias> (as shown in
$L_{bias}$ of Fig 1) and cross-entropy (CE) loss.

With a well-trained Paraformer model freezing, we enable contextualization of
hotwords for an ASR system by introducing bias out layer, bias decoder and
bias encoder, and training them with randomly sampled hotwords and their
corresponding targets. Notably, the training of bias-related parameters is
separate from the ASR training, thereby allowing for the use of specialized
hotword data (e.g. low frequency language phrases) and training strategies
without affecting the general ASR performance.

###  3.2 Inference and Auxiliary Technics

For $i$-th step in SeACo-Paraformer inference with a given hotword list, we
get the final merged probabilities for contextualized ASR as

$\mathbf{P}_{\mathbf{m}_{\mathbf{i}}}=\left\\{\begin{array}[]{l}\mathbf{P}_{\mathbf{ASR}_{\mathbf{i}}},\text{
if }\arg\max\mathbf{P}_{\mathbf{b}_{\mathbf{i}}}=\textless\text{no-
bias}\textgreater\\\
\lambda\mathbf{P}_{\mathbf{b}_{\mathbf{i}}}+(1-\lambda)\mathbf{P}_{\mathbf{ASR}},\text{
otherwise }\end{array}\right.$ (5)

When there is no hotword incoming or no hotword detected, SeACo-Paraformer
uses $\mathbf{P}_{ASR_{i}}$ only. $\lambda$ is a tunable parameter to adjust
the degree of trust bias decoder output.

In practical applications, as the number of incoming hotwords expands, the
performance of hotword activation decreases accordingly - it becomes difficult
for the cross-attention inside bias decoder to establish correct connection
between ASR decoder output $\mathbf{D}_{1:L^{{}^{\prime}}}$ and large-sacle
sparse hotword embedding $\mathbf{Z}_{1:n}$. For enabling SeACo-Paraformer to
conduct hotword customization with large scale hotword list, we propose
attention score filtering (ASF) strategy. The bias decoder inference is
conducted with full hotword list first to obtain the attention score matrix
$\mathbf{A}\in R^{L\times n}$, where $L$ is the length of output tokens and
$n$ is the number of hotwords. Then we sum up the scores of steps in $L$ and
get the attention scores for each hotword. According to attention scores, we
can pick the most active $k$ hotwords to conduct truly effective bias decoder
inference.

##  4 Experiments

###  4.1 Data Introduction

We conducted a series of experiments using large-scale industrial data from
speech lab of DAMO Academy, Alibaba - the random sampling based hotword models
requires sufficient diversity of data, otherwise it will easily overfit to
limited semantic information. The total amount of training data we use is
around 50,000 hours. For evaluating the performance of models in hotword
customization as well as general ASR, we use different test sets as shown in
Table1. For all of the hotword test sets, we distinguish some hotwords as
R1-hotwords, which means that their recall rate in the recognition of the
basic ASR model is less than 40%. The recall difficulty of these hot words is
higher, which can better demonstrate the customization ability of the hot word
model. Test-General is a large collection of several general ASR test sets,
covering various domains and scenarios. Test-Commercial, Test-Term and Test-
Entity are three hotword test sets of different domain.

To demonstrate reproducible experiment results and model comparison, we
construct and open two subsets for testing hotword customization based on the
open-source Aishell-1 dataset. First we use part-of-speech tagging [18] to
generate name entity list from transcription of Aishell-1 test and development
sets. Then treating the name entities as hotwords, we use all of the
R1-hotwords and a random part of the remains to compose two hotword lists.
Finally, Test-Aishell1-NE and Dev-Aishell1-NE are filtered out based on the
hotword lists111Find the opened data sets, source codes, detailed experiment
results and models at https://github.com/R1ckShi/SeACo-Paraformer.

Table 1: Information of test sets for customization and general ASR. | #utt | #hotword | #R1-hotword
---|---|---|---
Test-Gerneral | 40603 | - | -
Test-Commercial | 2000 | 693 | 72
Test-Term | 1639 | 969 | 258
Test-Entity | 1308 | 231 | 54
Test-Aishell1-NE | 808 | 400 | 226
Dev-Aishell1-NE | 1334 | 600 | 371

###  4.2 Experimental Setup and Evaluation Metrics

We implement the models with FunASR222https://github.com/alibaba-damo-
academy/FunASR and upload model files to
Modelscope333https://modelscope.cn/models, where models can be downloaded or
experienced through online demo service. The Paraformer model, Paraformer-CLAS
model and the proposed SeACo-Paraformer are totally opened including source
code, configuration files and models trained with industrial
data11footnotemark: 1. Paraformer base model is composed of 50-layers SAN-M
[19] encoder and 16layer NAR Transformer decoder, with 2048 hidden units.
Compared to the models discussed in [17], the Paraformer-CLAS presented in
this paper has undergone an upgrade and is guaranteed to possess a strong and
finely-tuned customization baseline. As for SeACo-Paraformer, the model is
trained with ASR backbone freezing using the same 50,000 hours training data
for 6 epochs. We set sampling ratio $r_{u}$ and $r_{b}$ to 0.75, $l_{min}=2$
and $l_{max}=8$. The introduced modules have following architectures: bias
encoder is a 2-layer LSTM with 512 hidden size, bias decoder is a 4-layer
Transformer decoder with 1024 hidden units. Only the introduced modules are
trained so the training batch is relatively larger, 6000 speech frames each
batch ($\times 16$ GPU).

Table 2: Experiment results over all test sets.

Models are compared according to 3 metrics: character error rate (CER) over
total test sets, recall rate/precision/F1 score (R/P/F) of the all hotwords in
average and R/P/F of R1-hotwords in average. A hotword is only considered a
successful recall when it is fully predicted, so we abandon calculating the
error rate of the hotword part as one of the evaluation indicators.

###  4.3 Results

Table 2 shows the experiment results over all test sets of the four models.
Note that we use $\lambda=1.0$ for Equation (5) in inference and $k=50$ for
ASF. In the general test, Paraformer-CLAS demonstrated similar CER to the
Paraformer-Base model. As the SeACo-Paraformer was trained with a frozen ASR
backbone, the four models exhibited minimal differences in general ASR
performance. In the internal hotword customization test, CLAS was shown to be
an efficient method, with the average recall of R1-hotwords increasing from
12% to 51%. However, SeACo-Paraformer still outperformed CLAS with a recall
rate of 60%. The addition of ASF results in a further enhancement, the recall
rate increased to 65%. And SeACo-Paraformer achieved a 5% relative CER
reduction compared to Paraformer-CLAS. We get similar results in open-source
test sets while Paraformer-CLAS, SeACo-Paraformer and SeACo-Paraformer+ASF get
69%, 79% and 87% recall rates respectively. The proposed method achieves
relative 30.2% CER reducion and 26.1% recall improvement comparing to
Paraformer-CLAS.

Fig. 2: Results of increasing hotword number experiments.

The experiment results of increasing hotword number are shown in Table 2.
Test-Entity has 231 hotwords in total and we enlarge it with irrelevant
hotwords to size 400, 800, 1200, 2000 and 4000. The curves in the figure
indicate that the performance of all three models deteriorates as the hotword
list expands. In terms of CER, only SeACo-Paraformer+ASF achieves a reduction
in CER when the list expands to 4000 hotwords. From the perspective of recall
rate, the SeACo-Paraformer model performs equally with CLAS when the number of
hotwords reaches 1200. However, with the support of ASF, the proposed model
maintains its advantage over CLAS.

Table 3: Ablation study results (CER and R1-Hotword Recall) | Test -C | Test -T | Test-E | Avg.
---|---|---|---|---
Seaco-Paraformer | 3.16/67 | 4.51/59 | 8.53/55 | 5.40/60
A1: +merge first | 3.28/65 | 4.72/61 | 8.67/53 | 5.56/60
A2: -cif attn | 3.86/64 | 5.29/62 | 8.66/58 | 5.94/61
A3: -dec attn | 3.25/63 | 4.59/50 | 8.60/51 | 5.49/55

Table 3 shows the results of ablation study over calculation of bias decoder.
Focus on Equation (3), there are several ways to introduce bias information
before bias output layer, merging $\mathbf{E}_{1:L^{{}^{\prime}}}$ and
$\mathbf{D}_{1:L^{{}^{\prime}}}$ before MHA rather thar after also makes sense
(A1). Besides, it’s essential to conduct ablation experiment for using
$\mathbf{E}_{1:L^{{}^{\prime}}}$ or $\mathbf{D}_{1:L^{{}^{\prime}}}$ only (A2
and A3). The result indicates that using CIF output alone to calculate hotword
loss leads to higher recall but worse in CER, which means that predicting
hotword with acoustic embedding only is not a stable method and decoder output
is indispensable. On the other hand, conducting MHA with
$\mathbf{E}_{1:L^{{}^{\prime}}}$ and $\mathbf{D}_{1:L^{{}^{\prime}}}$ twice
and separately is better than merging them first.

##  5 Analysis

The attention in bias decoder plays the most critical role in hotword
customization, in this section we go deep into the attention score matrix.
Figure 3 reveals how the bias decoder establishes the connection between
hotwords and semantic information (attention score matrix of the last layer in
bias decoder), but the phenomenon contradicts our initial hypothesis. In the
recognition of this utterance, SeACo-Paraformer predicts two hotwords
correctly. Two red dotted box shows the attention score pattern once matched
and it turns out that semantic information attends to non-blank phrases only
when relevant hotwords show up and end (yellow arrows), which is different
from [14]. Besides, we found the attention scores of <blank> at each step
imply the probability of hotwords to appear. And the sum of scores for all
steps of each hotword indicate the probability of the hotword to appear, which
is actually the basic idea of ASF.

Fig. 3: Attention score matirx $\mathbf{A}\in R^{L\times n}$.

##  6 Conclusion

In this paper we propose a flexible and effective hotword customization model
based on NAR ASR backbone called SeACo-Paraformer. A series of experiments
with industrial data proved that the proposed model outperforms CLAS which is
a classical hotword customization solution, and the ASF strategy for inference
is proved an effective method to improve ASR accuracy and hotwords recall
rate. Besides, we make further analysis over the performance with increasing
number of hotwords, ablation study of bias decoder calculation and the
attention score matrix. In the future, we will focus on large-scale incoming
hotword list and structure of bias encoder.

## References

  * [1] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton,  “Speech recognition with deep recurrent neural networks,”  in 2013 IEEE international conference on acoustics, speech and signal processing. Ieee, 2013, pp. 6645–6649.
  * [2] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals,  “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,”  in 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2016, pp. 4960–4964.
  * [3] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [4] Kanishka Rao, Haşim Sak, and Rohit Prabhavalkar,  “Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer,”  in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 193–199.
  * [5] Niko Moritz, Takaaki Hori, and Jonathan Le,  “Streaming automatic speech recognition with the transformer model,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6074–6078.
  * [6] Zhifu Gao, Shiliang Zhang, Ming Lei, and Ian McLoughlin,  “Universal asr: Unifying streaming and non-streaming asr using a single encoder-decoder model,”  arXiv preprint arXiv:2010.14099, 2020.
  * [7] Shiyu Zhou, Shuang Xu, and Bo Xu,  “Multilingual end-to-end speech recognition with a single transformer on low-resource languages,”  arXiv preprint arXiv:1806.05059, 2018.
  * [8] Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, et al.,  “A comparative study on transformer vs rnn in speech applications,”  in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 449–456.
  * [9] Fan Yu, Haoneng Luo, Pengcheng Guo, Yuhao Liang, Zhuoyuan Yao, Lei Xie, Yingying Gao, Leijing Hou, and Shilei Zhang,  “Boundary and context aware training for cif-based non-autoregressive end-to-end asr,”  in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 328–334.
  * [10] Zhifu Gao, Shiliang Zhang, Ian McLoughlin, and Zhijie Yan,  “Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition,”  arXiv preprint arXiv:2206.08317, 2022.
  * [11] Petar Aleksic, Cyril Allauzen, David Elson, Aleksandar Kracun, Diego Melendo Casado, and Pedro J Moreno,  “Improved recognition of contact names in voice commands,”  in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, pp. 5172–5175.
  * [12] Keith Hall, Eunjoon Cho, Cyril Allauzen, Francoise Beaufays, Noah Coccaro, Kaisuke Nakajima, Michael Riley, Brian Roark, David Rybach, and Linda Zhang,  “Composition-based on-the-fly rescoring for salient n-gram biasing,”  2015\.
  * [13] Golan Pundak, Tara N Sainath, Rohit Prabhavalkar, Anjuli Kannan, and Ding Zhao,  “Deep context: end-to-end contextual speech recognition,”  in 2018 IEEE spoken language technology workshop (SLT). IEEE, 2018, pp. 418–425.
  * [14] Minglun Han, Linhao Dong, Shiyu Zhou, and Bo Xu,  “Cif-based collaborative decoding for end-to-end contextual speech recognition,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6528–6532.
  * [15] Kaixun Huang, Ao Zhang, Zhanheng Yang, Pengcheng Guo, Bingshen Mu, Tianyi Xu, and Lei Xie,  “Contextualized end-to-end speech recognition with contextual phrase prediction network,”  arXiv preprint arXiv:2305.12493, 2023.
  * [16] Linhao Dong and Bo Xu,  “Cif: Continuous integrate-and-fire for end-to-end speech recognition,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6079–6083.
  * [17] Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, Zhangyu Xiao, et al.,  “Funasr: A fundamental end-to-end speech recognition toolkit,”  arXiv preprint arXiv:2305.11013, 2023.
  * [18] Zhenyu Jiao, Shuqi Sun, and Ke Sun,  “Chinese lexical analysis with deep bi-gru-crf network,”  arXiv preprint arXiv:1807.01882, 2018.
  * [19] Zhifu Gao, Shiliang Zhang, Ming Lei, and Ian McLoughlin,  “San-m: Memory equipped self-attention for end-to-end speech recognition,”  arXiv preprint arXiv:2006.01713, 2020.


# Achieving Data Dissemination with Security using FIWARE and Intel Software
Guard Extensions (SGX)

Dalton Cézane Gomes Valadares1,2, Matteus Sthefano Leite da Silva2,
Andrey Elísio Monteiro Brito2 and Ewerton Monteiro Salvador3 *This research
was partially funded by EU-BRA SecureCloud project (EC, MCTIC/RNP, and SERI,
3rd Coordinated Call, H2020-ICT-2015 Grant agreement no. 690111)1Federal
University of Campina Grande, Informatics and Electrical Engineering Center,
Computer Science, Campina Grande, Paraíba, Brazil<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Institute
of Pernambuco, Mechanical Engineering Department, Caruaru, Pernambuco, Brazil
<EMAIL_ADDRESS>University of Paraíba, João
Pessoa, Paraíba, Brazil<EMAIL_ADDRESS>IEEE. Personal use of
this material is permitted. Permission from IEEE must be obtained for all
other uses, in any current or future media, including reprinting/republishing
this material for advertising or promotional purposes, creating new collective
works, for resale or redistribution to servers or lists, or reuse of any
copyrighted component of this work in other works.

###### Abstract

The Internet of Things (IoT) field has gained much attention from industry and
academia, being the main subject for numerous research and development
projects. Frequently, the dense amount of generated data from IoT applications
is sent to a cloud service, that is responsible for processing and storage.
Many of these applications demand security and privacy for their data because
of their sensitive nature. This is specially true when such data must be
processed in entities hosted in public clouds, where the environment in which
applications run may not be trusted. Some concerns are then raised since it is
not trivial to provide the needed protection for these sensitive data. We
present a solution that considers the security components of FIWARE and the
Intel SGX capabilities. FIWARE is a platform created to support the
development of Smart Applications, including IoT systems, and SGX is the Intel
solution for Trusted Execution Environment (TEE). We propose a new component
for key management that, together with other FIWARE components, can be used to
provide privacy, confidentiality, and integrity guarantees for IoT data. A
case study illustrates how this proposed solution can be employed in a
realistic scenario, which allows the dissemination of sensitive data through
public clouds without risking privacy issues. The results of the experiments
provide evidence that our approach does not harm scalability or availability
of the system. In addition, it presents acceptable memory costs when
considering the benefit of the privacy guarantees achieved.

##  I Introduction

The Internet of Things (IoT) concept has gained considerable importance in
industry and academia in the last years. The possibility of interconnecting a
huge number of devices allows the creation of various new applications. This
is possible thanks to the advances in devices technologies and communication
protocols that enable such interconnection. IoT sensors and actuators are
increasingly becoming smarter and, even when they are not so smart, they can
connect to other gateway devices that may add additional functionality.

IoT devices often have some common constraints like limited storage and
processing capacities, limited energy, or connectivity. These constraints are
dealt by cloud-based IoT. The idea, in general, is to send all collected data
to the cloud in a way they can be analyzed and processed for use in other
systems [1]. Among the cloud-based IoT benefits, the following can be
highlighted: long-term storage and processing of collected data, data reuse in
multiple services, data integration of different users/things, user mobility
support and granted availability for intermittent connection [2].

Frequently, the IoT applications will be generating sensitive data related to
specific businesses that can involve large financial transactions, or even
related to personally identifiable information (PII) of their users. Both
cases, among many others, require special attention in order to provide
security and privacy capabilities for these kind of applications. It is
mandatory to ensure an acceptable level of trust for processing and storing
these data and this is an actual challenge nowadays (privacy and security are
still concerns related to IoT applications [1, 3]).

A common scenario for an IoT application considers a publish/subscribe system
that is responsible for registering the interest of some entities in specific
data or measurements produced by another and for sending notifications to the
interested entities whenever changes occur in the measured values. When the
notifications are received, the entities can take important decisions as
turning on/off machines, triggering alarms, performing complex processing,
etc.

An instance of this previous scenario can be an energy consumption monitoring
application, considering two perspectives to facilitate the understanding:
firstly, there are many smart meters, named here Producers, collecting energy
consumption from different buildings along a smart city; secondly, there are
applications responsible for doing the data aggregation for customers’ billing
purposes. Some techniques that hit the customers’ privacy, like NIALM [4],
enable attackers to estimate the number of people in buildings, as well as to
gain information on what kinds of activities are being performed. The same can
be applied to an industrial environment, in which attackers can discover
secrets related to production, estimating how many machines are working, how
many workers, etc. Because of this, it is highly recommended to keep a good
level of security and privacy in architectures that support such scenarios.

To avoid such kind of attacks, the data can be encrypted in the producers in
order to be processed only in a secure way. This can be achieved by
homomorphic cryptography [5], which performs common operations over encrypted
data, but its great overhead makes it unpractical to use with complex
operations [6]. Another approach is the use of a trusted execution environment
(TEE), that enables, by using specific hardware instructions, the creation of
a shielded space in the memory in a way data can be processed securely inside
this space. Encrypted data enters the protected area, being decrypted only
inside it and encrypted again before leaving. TEE use is more feasible than
homomorphic cryptography, because, it takes considerably less time to process
data [5].

Related to this problem, two research questions were identified for guiding
this research:

  1. 1.

How to limit the need to trust the storage provider and the consumer of the
data?

  2. 2.

How to control access allowing only authorized entities to consume data?

In this paper we present a solution that prevent users’ data from unauthorized
access (e.g., NIALM attacks), keeping them private and secure. For this, we
considered both previous questions. First, we use the Intel TEE technology,
named Software Guard Extensions (SGX). We propose a component responsible for
key generation, storage and management, named Key Vault. Key Vault also
provides data security by using Intel SGX to protect its execution. The
producers send the encrypted measurements for a publish/subscribe component
and the consumers, that are also SGX applications, receive these measurements
through notifications. Thus, the measurements are protected because they are
only decrypted and processed in a secure space, inside an SGX application.

Second, we have used some components from FIWARE, which is a software
framework that provides modules to ease the development of smart applications.
For authentication and authorization of producers and consumers, we have used
the components Keyrock Identity Management and Wilma PEP Proxy. For the
publish/subscribe system we have used the FIWARE Orion Context Broker, which
receives the encrypted measurements from producers and triggers notifications
to consumers.

This paper is organized as follows: in Section II we present a background,
concisely explaining FIWARE, Intel SGX and our case study; the proposed
solution is presented in Section III including principles, architecture,
threat model and the definition of Key Vault; the evaluation of our solution
as well as the results and a discussion of them are presented in Section IV;
in Section V, we discuss the related works and finish this paper with Section
VI, presenting the conclusions and suggesting future works.

##  II Background

In this Section we present some basic concepts in order to ease the
understanding of our proposal. FIWARE and Intel Software Guard Extensions
(SGX) are shortly covered next. In addition, our case study is briefly
described.

###  II-A FIWARE

FIWARE (Future Internet-WARE) is a software platform created to ease the
development of smart applications for the Future Internet in multiple sectors
and following open standards [7]. For this, it provides a wide set of APIs
(Application Programming Interfaces). FIWARE is composed by many components,
called Generic Enablers (GEs), with some of them being based in the well-
accepted cloud platform OpenStack. OpenStack is also used to establish a cloud
that provides FIWARE services. For each FIWARE component (GE) specification,
there is an open source reference implementation. Some of the domains covered
by FIWARE GEs are the following [8]: Data and Context Management, Internet of
Things Services Enablement, Security, Cloud Hosting, Advanced Web-based User
Interface, etc.

This work uses three FIWARE GEs for our proof of concept: Orion Context Broker
(CB), Keyrock Identity Manager and Wilma PEP (Policy Enforcement Point) Proxy.
Orion CB is a context data management system, allowing the operations of
registering, updating and querying context data [9]. It also works with the
publish/subscribe communication pattern, allowing the sending of notifications
to interested parties when changes occur in the entities of interest. The
Keyrock Identity Manager is an identity and access management system, being
responsible for registering and querying identity data, providing access
tokens to users and validating them in order to authenticate users and
allowing access to protected services [10]. Wilma PEP Proxy is a policy
enforcement point responsible for basic authorization, controlling the access
to protected services by validating received tokens with the Keyrock Identity
Manager [11].

For our application, Orion CB receives the energy consumption data from
producers (smart meters) and sends notifications to consumers (aggregators);
Keyrock and Wilma are used to protect Orion CB and our key manager (Key
Vault), doing the authentication and authorization processes.

###  II-B Intel Software Guard Extensions (SGX)

Intel SGX is a Trusted Execution Environment (TEE) technology that protects
code and data from disclosure or modification [12]. Applications intended to
be safe are executed on special protected memory regions such that their code
and data are isolated from other software running in the system, even with
higher privilege, like operating system (OS). These special regions in the
memory are called enclaves, which are created and manipulated through a
distinct set of processor instructions, with help of a software development
kit (SDK) provided by Intel. With these hardware-based capabilities, Intel
promises that code and data remain protected even when drivers, OSs or BIOS
are compromised.

SGX works with a small attack surface, that is the CPU boundary, preventing
direct attacks on executing code or sensitive data in memory. The enclaves
work in this boundary, shielding the data and code inside them, and encrypting
data when they need to leave the enclave. When data return to enclaves,
integrity checking is performed.

Intel SGX also provides a way to enable remote parties to check if an
application really executes in a valid enclave of a real Intel SGX processor.
This mechanism ensures the authenticity of an enclave, validating, for the
remote party, the application enclave’s identity. This is possible due to a
remote attestation process that is performed following a specific protocol, in
which both parts exchange some information in a way the remote application can
verify the authenticity of the supposed SGX application by accessing the Intel
Attestation Service and checking some information generated in the enclave. At
the end of the remote attestation, both sides will have a symmetric shared key
that can be used to exchange sensitive information between the remote
application and the application running in the attested enclave. The
authenticity of the code can also be checked, but the description of this
process is out of scope for this paper.

###  II-C Case Study

This case study is based on a residential environment in which the residences
have smart meters to measure their energy consumption for billing purposes. A
basic scenario for this application considers the following entities: smart
meters, a broker acting as a publish/subscribe system, and some consumers
responsible for processing the data.

In this scenario, the smart meters are the data producers and the processing
applications are the consumers. A common architecture for this scenario can be
seen in the Figure 1. The communication flow between components follows:

  1. 1.

The Producers (Smart Meters) send energy consumption data to the IoT Gateway;

  2. 2.

The IoT Gateway preprocesses these energy consumption data and sends them to a
Broker (Publish/Subscribe System);

  3. 3.

The Consumers register interest on the specific Producers, in the Broker, in
order to receive notifications about their energy consumptions;

  4. 4.

The Broker sends notifications with the Producers’ energy consumption data for
the Consumers that previously registered interests.

This IoT Gateway can be any device more powerful than the smart meters, for
example, capable of processing more complex operations on their generated data
or having better communication capabilities. This scenario is often known as
fog/edge computing, since the IoT devices are connected to another device with
more processing capacity and which is closest to the network edge [13].

Figure 1: Basic Scenario for Smart Metering Application.

####  II-C1 Aggregation of Smart Meter Data

For this case study, it was considered that the consumers are data aggregators
responsible to process the energy consumption data for billing purposes. This
data aggregation is made by summing all the energy consumption data collected
for each residence in a specific region.

To make this aggregation, it is necessary to read all the energy consumption
data for all the residences in a region. As stated before, there are
approaches to non-invasive estimating what kind of electronic devices or
household items are used at specific time and for how long. This technique is
called NIALM (Non Intrusive Appliance Load Monitoring) and it is considered a
privacy problem since an evil-minded adversary can discover what is done by
people living in a specific residence (the time in which people have shower,
watch TV or use the microwave, for instance).

####  II-C2 Need for data protection

Due to the privacy problem mentioned, it is necessary to apply some mechanisms
to protect the data generated at producers, avoiding the discovery of the
users’ behavior in the monitored residences. As the FIWARE does not provide
such mechanisms in any security components (Generic Enablers), we had to
elaborate a solution to protect the sensitive data using some techniques and
our key manager proposed, named Key Vault Generic Enabler, described in the
next Section.

##  III Trusted Data Dissemination with FIWARE and Intel SGX

Among the current FIWARE Generic Enablers (GEs), there is no specific ones for
data protection, enabling users to hide or shield sensitive information and
preserving users’ privacy. The FIWARE Security modules (GEs) available are
useful only for authentication and authorization operations, managing
identities and providing access control mechanisms. It was necessary, then, to
design and implement a solution that protects the sensitive data against
untrusted or unauthorized third parties, like attackers or even cloud
administrators.

###  III-A Threat Model

Our study application is composed mainly of three parts: the producers (smart
meters), the Orion CB (a publish/subscribe system), and the consumers (data
aggregators). The attack surface for this application considers the
possibility of an attacker gaining access while the data are being transmitted
between the parts and the possibility of an attacker gaining complete access
to Orion Context Broker and to consumers hosts.

An attacker can intercept the messages sent from the producers (smart meters)
to the Orion CB or even the notifications sent from the Orion CB to the
consumers (aggregator). Besides, an attacker can get data having access to a
consumer host or to our publish/subscribe system (Orion CB). In any case, the
sensitive data are compromised. Then, our solution must deal with these
threats, avoiding that any attacker in any of these situations can read the
sensitive data.

Our solution considers that all the producers (smart meters) are secure, thus,
we do not deal with the possible existence of fake/crook producers. In any
case, it is worth mentioning this possibility could cause only some noise to
generated data by valid producers and this would not be a privacy problem for
the customers.

###  III-B Principles of the Trusted Solution

In order to protect the sensitive data, considering the specified threat
model, we have used cryptography for data security and privacy along with an
identity manager (IdM) with basic access control. Our trusted solution is
based on the fact that all the generated data are encrypted in the producers
(source), before being transmitted to the publish/subscribe system, and they
are only decrypted by a trusted consumer, running on a trusted execution
environment (TEE) and with access to the correct key. Furthermore, all the
producers and consumers are identified with credentials through the OAuth2
protocol [14], that is an industry-standard for authorization.

With this architecture, the data encryption in the producers guarantees a
secure transmission of the data allowing only a secure processing in a trusted
machine, since the consumers must run on a TEE, like SGX. Therefore, we have
the data privacy preserved because the sensitive data are decrypted and
processed only within a TEE application.

We have designed and implemented a component responsible for generating and
storing the keys used by producers and consumers for encryption and
decryption: the Key Vault. It also runs on a TEE-enabled machine and is
detailed in the next subsection.

In our case study, to validate that the trusted components (Consumers and Key
Vault) are running inside SGX enclaves, we perform the remote attestation (RA)
process that checks with the Intel if these enclaves are valid and if they are
running on a real Intel SGX. This process is done by producers, attesting Key
Vault whenever it is necessary to get a public key, and by Key Vault,
attesting the consumer whenever it receives a private key request.

###  III-C The Keyvault Generic Enabler

A Public Key Infrastructure (PKI) was needed to supply appropriated keys to
the appropriated entities in a secure way to support our solution. On the one
hand, producers must encrypt their data before sending them to Orion Context
Broker. On the other hand, consumers must have the appropriate key to decrypt
producers’ data. Symmetric cryptography is not suitable for this scenario,
because an attacker can get access to some producer credentials, be
authenticated and request the symmetric key. With this symmetric key, the
attacker can decrypt all interested data from the producers.

In order to generate, maintain and distribute keys securely, we developed the
Key Vault, which runs in an SGX environment and provides public and private
keys accordingly and respectively to producers and consumers from our
scenario. The communication with Key Vault is done through Secure Socket Layer
(SSL) [15], using HTTPS. The Key Vault provides public keys for authenticated
producers and private key for authenticated and attested consumers.

The Key Vault treats the requests from producers and consumers, after being
authenticated, and sends the appropriate key (public for producers, private
for consumers). As it runs in an SGX enclave, the keys are handled in a secure
way. The Key Vault must be attested by producers before sending public keys as
well as it must attest the consumers before sending private keys. The private
key is only sent to SGX consumers, after the remote attestation properly
successful. The private key is encrypted with a shared key that is generated
in the remote attestation process. If the the remote attestation of the
consumers fails, the private keys are not sent.

###  III-D Trusted Architecture

Our trusted architecture proposed comprises the following components:

  * •

Producers - the smart meters, responsible for measuring the energy
consumption, with or without some IoT gateway or another device acting as a
gateway to process the generated data (Fog/Edge Computing) before sending them
to a publish/subscribe system. In the Figure 2 they are represented as SM, for
smart meter, and BBB, for BeagleBone Black acting as a gateway;

  * •

FIWARE Keyrock IdM - the system responsible for the Identity Management,
providing valid OAuth2 tokens for authentication purpose;

  * •

FIWARE Wilma PEP Proxy - the Policy Enforcement Point responsible for
allowing/disallowing the access to protected services after receiving an
OAuth2 token and validates it with Keyrock IdM. This architecture considers
two of this component: one to protect Key Vault and another to protect Orion
Context Broker;

  * •

FIWARE Orion CB (Context Broker) - the publish/subscribe system responsible
for receiving the producers’ data and send them to consumers through
notifications;

  * •

Key Vault - our proposed system responsible for securely managing, storing and
distributing the cryptographic keys to producers and consumers;

  * •

Consumers - the data aggregators responsible to process the energy consumption
measures for billing purposes.

The communication flow for producers and consumers is detailed next. As
mentioned before, the data are encrypted in the producers, which are the smart
meters. This is performed as a privacy technique in order to achieve the data
privacy. The consumer is responsible for decrypting the sensitive data and
processing them in a secure way. All the steps in the communication flow can
be seen with a respective number (1 to 5 for producers and 6 to 13 for
consumers) in the Figure 2 and are described bellow:

Figure 2: Producer and Consumer Communication Flow.

  1. 1.

The producer (smart meter or its gateway) accesses the Keyrock IdM passing
valid credentials in order to be authenticated and get an OAuth2 valid token;

  2. 2.

The producer attests the Key Vault, sends it a request asking for a public key
and passing the OAuth2 token obtained with the Keyrock IdM;

  3. 3.

The Wilma PEP Proxy intercepts the request, checks the validity of the token
with Keyrock IdM and forwards the request to the Key Vault if the token is
valid. Key Vault then sends back, to the producer, the asked public key ($PuK$
in the Figure 2);

  4. 4.

With the public key ($PuK$), the producer can encrypt the generated data
(energy consumption) and send them to the Orion CB;

  5. 5.

Same as step 3 (Wilma PEP Proxy checks the validity of the token with Keyrock
IdM and forwards the request to Orion CB if the token is valid).

  6. 6.

The consumer, an SGX application, accesses the Keyrock IdM passing valid
credentials in order to be authenticated and get an OAuth2 valid token;

  7. 7.

The consumer sends a request to Key Vault asking for the private key ($PrK$ in
the Figure 2) related to the public key sent to producers and passing the
OAuth2 token obtained with the Keyrock IdM. Besides the OAuth2 token, the
request also informs an endpoint in order to enable the Key Vault to attest
that the consumer is a real SGX application (remote attestation);

  8. 8.

The Wilma PEP Proxy intercepts the request, checks the validity of the token
with Keyrock IdM and forwards the request to the Key Vault if the token is
valid;

  9. 9.

Key Vault starts the remote attestation process with the endpoint received and
once the consumer is attested, the private key ($PrK$) is sent to the
consumer. This $PrK$ is encrypted with the symmetric shared key ($ShK$ in the
Figure 2), before being sent;

  10. 10.

The consumer sends subscription requests to the Orion CB in order to receive
notifications (energy consumptions) from the producers;

  11. 11.

Wilma PEP Proxy intercepts the request and validates the OAuth2 token with
Keyrock IdM. Once the token is valid, the subscriptions are stored in the
Orion CB;

  12. 12.

Consumer receives notifications whenever a producer sends new data (energy
consumptions) to Orion CB. As the consumer has the private key ($PrK$), it can
decrypt data and process them in a secure SGX enclave.

With these steps, the data are transmitted securely from producers to Orion
CB. An attacker can not discover what sensitive data are transmitted and
stored in Orion since they are encrypted. The messages exchanged with Key
Vault are performed through HTTPS.

##  IV Evaluation

In this section we present the set of experiments we considered to assess the
performance of our solution.

###  IV-A Experiments Setup

To evaluate our solution, we deployed all the components of our architecture
in two machines equipped with Intel Skylake CPU model i7-6700, 3.4GHz, 8MB
cache and 8GB RAM (Dell OptPlex 5040). To have a more realistic scenario, we
distributed all the components as follows: one machine with the Producers and
the other with Orion CB deployed at a virtual machine, Consumer and Key Vault
at separated containers, as well as KeyRock IdM and Wilma PEP Proxy.

After this setup, we have run a set of experiments involving the communication
flow explained previously: data generated in the Producers were sent to the
Orion CB and it sent the information to the Consumer through notifications.

###  IV-B Scenarios, Metrics, and Parameters

To analyze the cost of our solution, we considered two scenarios:

  * •

Without data security, considering just Producers, Consumer and Orion CB
(without any security mechanism);

  * •

With data security, including the Key Vault for key management and
distribution, encryption/decryption enabled by these keys, and the
authentication/authorization enabled by Keyrock and Wilma PEP Proxy.

These scenarios were considered in order to compare our solution with a case
using the regular architecture, without employing any security mechanisms. The
idea was to verify the cost of the protection added by our proposal.

We have decided to measure latency for both scenarios. This metric was
obtained as follows: the elapsed time for some data to leave the Producer and
reach the Consumer; as seen before, to accomplish this, the data must leave
the Producer and arrive in the Consumer through notifications from Orion CB,
in the first scenario; this flow is repeated for the second scenario,
considering the key distribution and the
encryption/decryption/authentication/authorization processes.

For both scenarios, we have executed 500 publication cycles, to obtain a
reliable sample. For each execution, we have marked the time when the data
were generated, at the Producer, and time when the data were processed, at the
Consumer. Besides the communication flow latency for both scenarios, we also
have measured the remote attestation latency, since the remote attestation is
an important process performed in the communication with our Intel SGX
applications: the Key Vault and the Consumer.

Finally, we also have carried out tests related to the scalability of our
solution, in respect to the number of possible producers sending measurements
concurrently. We have executed tests with 100, 200, 300 and 400 producers.

###  IV-C Results and Discussion

According to achieved results, seen in Fig. 3, our proposed solution for data
security presents low overhead related to a solution that does not use any
security mechanism: approximately 10.5 milliseconds against 7.5 milliseconds
on average. The remote attestation process had an average time of 2.5 seconds,
but this process is executed only once to get the keys (once producers and
consumer have the respective keys, they do not need to communicate with Key
Vault).

Figure 3: Latency

This solution also presents a good level of scalability, since it has succeed
to keep working even with different loads of requests from different numbers
of producers at same time: 100, 200, 300 and 400. However, as seen in Fig. 4,
the higher the number of producers at same time, the higher the latency to
process the requests and the lower the number of processed requests.

Figure 4: Latency considering many producers at same time (stress)

Considering the threat model applied and experiments, we can say that our
solution provides the following features:

  * •

Integrity – the produced data are preserved during all the communication flow;
they cannot be modified without being detected;

  * •

Privacy – the privacy of the customers can be preserved, since the produced
data can not be read in any of the communication flow steps, even if the Orion
CB is under control of an attacker;

  * •

Confidentiality – the confidentiality of the data is preserved because they
can not be read/accessed, except by authenticated and authorized producers and
consumers;

  * •

Authentication – all producers and consumers must use their credentials in
order to get access to the protected services and thus successfully perform
the communication flow;

  * •

Secure communication – data in transit and channel are encrypted.

These properties are achieved thanks to: cryptography techniques, TLS/HTTPS
protocol and Intel SGX technology. Assuming that the producers credentials are
safe/protected, our solution also satisfy the non-repudiation property, in a
way any producer can not deny any previous sending of data.

##  V Related Work

In this section we present some works related to security and privacy in IoT,
involving general aspects, and Trusted Execution Environments, such as Intel
SGX.

Data privacy is always a concern when applications deal with sensitive data.
Weber [16] and Perera et al. [17] present concerns related to data privacy in
IoT, with main challenges and some efforts performed by EU as an attempt to
regulate privacy and data protection. For Weber, the main concern is the
management of great quantity of generated data in order to secure storage and
communication. Skarmeta et al. [18] present requirements that need to be
addressed regarding security and privacy challenges in IoT. They propose a
distributed capability-based access control that uses public key cryptography
and is viable, according to carried out experiments, to deal with complex
scenarios in the IoT.

Werner et al. [19] present a survey on privacy strategies, relating cloud
identity management and strategies to achieve good levels of privacy, and
listing main features and challenges. The authors proposed a user-centered
approach, giving PII (Personally Identifiable Information) control to data
owners and encrypting these data to prevent unauthorized access. Henze et al.
[2] present an approach that allows users to set their privacy requirements
before sending sensitive data to cloud, with an adaptable interface in a
transparent way. This is performed during the development process and is also
an user-centered approach, with each user owning and operating one or more
smart objects (IoT devices).

An end-to-end security architecture for IoT applications is proposed by
Vucinic et al [20]. They consider a trusted Authorization Server, responsible
for providing access secrets to clients (constrained nodes using CoAP
protocol) and use encryption and signing mechanisms. Experiments resulted in
low energy consumption and latency. Li et al. [1] propose a secure
communication channel for IoT devices based in a heterogeneous ring
signcryption scheme, with identity-based cryptography and a public key
infrastructure. Its performance was compared to four other techniques, being
more efficient than others and, then, suitable for data transmission in IoT.

Intel SGX is used to implement a Secure Content-Based Routing (SCBR) system
[6], providing privacy preserving to messages, since they are not exposed to
unauthorized parties and they are filtered only in a secure enclave. Extensive
experiments concluded that SGX adds a limited overhead providing much better
performance when compared to other alternatives of Secure CBR. Silva et al.
[5] presents a solution for privacy and security preservation, also using
Intel SGX and considering a smart metering application. Their experiments
resulted in a relative low overhead. Guan et al. [13] proposed TrustShadow, a
system to shield legacy applications, running on multiprogramming IoT devices,
from untrusted OSes. It uses ARM TrustZone technology (TEE) to secure critical
applications and it has presented negligible or moderate overhead running real
applications.

Sotiriadis et al. [21] used FIWARE Generic Enablers to implement an
architecture that covers the cloud management and the operational features of
an application in the health area. This architecture integrates many software
modules in order to achieve all the application requirements, including
modules for resources and data management, privacy, devices and identity
management, cloud resources, etc.

Security and privacy, as seen in some related works, are still challenges for
the IoT systems [3, 18, 1, 17]. Although there are good solutions being
developed, there is still space for complete solutions end to end, considering
all the points we made in our proposal, such as privacy, confidentiality,
security, authentication and authorization.

##  VI Conclusion

As highlighted in the related work, data security in the IoT field is still a
challenge. In this paper we presented an example that demands such data
security: a smart metering system that generates sensitive data. Due to NIALM
techniques, this system can be target of privacy attacks, since an attacker
can infer behavior from a house by just analyzing its energy consumption
measurements.

We then have proposed a solution based on the Key Vault: a new Generic Enabler
(GE) responsible for key management and distribution. Key Vault is secure and
trusted because it runs on an Intel SGX. In our architecture, Key Vault
substitutes the use of a Public Key Infrastructure (PKI) and, thanks to its
key distribution and the consequent cryptography used, we achieved data
integrity and confidentiality, enabling the development of privacy-friendly
solutions.

As seen in the Section IV, our solution has a low overhead compared to an
architecture that does not consider any data security. Using Key Vault and its
capabilities, together with authentication/authorization, our solution had an
increase of just 3 milliseconds in the latency of the messages flow, i.e., the
elapsed time between producers and consumers considering all the communication
flow with the operations of getting keys, encrypting, sending data and
decrypting.

Moreover, our solution does not impose scalability constraints additional to
the underlying infrastructure. The Key Vault component and the needed
attestation tasks can be easily made scalable due to its simplicity. In
addition, our stress test has reached a number of 400 producers sending data
almost at same time. We also measured the time spent with the remote
attestation (RA) process, that validates if a system really runs on a valid
Intel SGX. The RA process presented a mean time of 2.5 seconds, what is
considered acceptable for our application, since it is commonly performed just
one time for each new producer or consumer, to validate the system or the new
member, respectively.

In our solution we used two FIWARE Security GEs: KeyRock Identity Management
and Wilma PEP Proxy. Both can be considered as untrusted since they run
unprotected in cloud and can be abused by adversaries. As future work, we
envisage that these GEs can be implemented with some TEE technology.
Furthermore, we can implement Attribute Based Encryption on the Key Vault to
control access to keys using intrinsic attributes of the IoT devices. In this
case, the access to keys would be controlled by entities’ attributes.

## References

  * [1] F. Li, Z. Zheng, and C. Jin, “Secure and efficient data transmission in the internet of things,” _Telecommunication Systems_ , vol. 62, no. 1, pp. 111–122, May 2016.
  * [2] M. Henze, L. Hermerschmidt, D. Kerpen, R. Häußling, B. Rumpe, and K. Wehrle, “A comprehensive approach to privacy in the cloud-based internet of things,” _Future Gener. Comput. Syst._ , 2016.
  * [3] J. Wilson, R. S. Wahby, H. Corrigan-Gibbs, D. Boneh, P. Levis, and K. Winstein, “Trust but verify: Auditing the secure internet of things,” in _Proc. of the 15th Annual International Conf. on Mobile Systems, Applications, and Services_ , ser. MobiSys ’17. ACM, 2017\.
  * [4] G. Eibl and D. Engel, “Influence of data granularity on nonintrusive appliance load monitoring,” in _Proc. of the 2nd ACM Workshop on Information Hiding and Multimedia Security_ , ser. IHMMSec ’14. ACM, 2014.
  * [5] L. V. Silva, R. Marinho, J. L. Vivas, and A. Brito, “Security and privacy preserving data aggregation in cloud computing,” in _Proc. of the Symposium on Applied Computing_ , ser. SAC ’17. ACM, 2017.
  * [6] R. Pires, M. Pasin, P. Felber, and C. Fetzer, “Secure content-based routing using intel software guard extensions,” in _Proc. of the 17th International Middleware Conf._ , ser. Middleware ’16. ACM, 2016.
  * [7] F. Community. Fiware. [Online]. Available: https://www.fiware.org/
  * [8] ——. The fiware catalogue. [Online]. Available: https://catalogue.fiware.org/
  * [9] ——. Fiware-orion. [Online]. Available: https://fiware-orion.readthedocs.io/en/master/
  * [10] ——. Keyrock. [Online]. Available: http://fiware-idm.readthedocs.io/en/latest/introduction.html
  * [11] ——. Fiware-pep-proxy. [Online]. Available: http://fiware-pep-proxy.readthedocs.io/en/latest/user_guide/
  * [12] Intel. Intel® software guard extensions (intel® sgx). [Online]. Available: https://software.intel.com/en-us/sgx
  * [13] L. Guan, P. Liu, X. Xing, X. Ge, S. Zhang, M. Yu, and T. Jaeger, “Trustshadow: Secure execution of unmodified applications with arm trustzone,” in _Proc. of the 15th Annual International Conf. on Mobile Systems, Applications, and Services_ , ser. MobiSys ’17. ACM, 2017.
  * [14] A. Parecki and O. Community. Oauth 2.0. [Online]. Available: https://oauth.net/2/
  * [15] S. Shopper. What is ssl? ssl certificate basics. [Online]. Available: https://www.sslshopper.com/what-is-ssl.html
  * [16] R. Weber, “Internet of things: Privacy issues revisited,” _Computer Law and Security Review_ , vol. 31, 2015.
  * [17] C. Perera, R. Ranjan, L. Wang, S. U. Khan, and A. Y. Zomaya, “Big data privacy in the internet of things era,” _IT Professional_ , vol. 17, 2015.
  * [18] A. F. Skarmeta, J. L. Hernández-Ramos, and M. V. Moreno, “A decentralized approach for security and privacy challenges in the internet of things,” in _IEEE World Forum on IoT (WF-IoT)_ , 2014.
  * [19] J. Werner, C. Merkle Westphall, and C. Westphall, “Cloud identity management: A survey on privacy strategies,” _Computer Networks_ , vol. 122, 2017.
  * [20] M. Vucinic, B. Tourancheau, F. Rousseau, A. Duda, L. Damon, and R. Guizzetti, “Oscar: Object security architecture for the internet of things,” _Ad Hoc Networks_ , vol. 32, 2015.
  * [21] S. Sotiriadis, E. G. Petrakis, S. Covaci, P. Zampognaro, E. Georga, and C. Thuemmler, “An architecture for designing future internet applications in sensitive domains: Expressing the software to data paradigm by utilizing hybrid cloud technology,” in _13th IEEE International Conf. on BioInformatics and BioEngineering(BIBE)_ , 2014.